Pipelines

Overview  

A pipeline is the path your data takes from a source to your data platform. You do not write one. You declare the sources, the target, and the jobs that connect them, and Amfelt derives the rest from the data contract it discovered.

Everything a pipeline needs lives in the Project file: which objects to move, which layer they land in, and how often.

Jobs Are the Unit of Work  

A job moves data between one named source and one named target. It selects objects from the source contract with a glob, names the data platform layer it populates, and sets a default writeMode (how records land) for them. What the history layer then keeps is a separate setting, historyMode, which can be overridden per object. Write Modes and History covers both.

Scheduling belongs to the job rather than to the project. Its runs are strategy-and-schedule pairs (a full load nightly and another on a weekend schedule are two runs of the same job; incremental runs are on the roadmap), and maxParallel bounds how many of the job’s tables load at once.

Layers  

Data does not land in one place. Each job populates a named layer, and the layers build on each other:

The Layering Model covers the history modes in more detail, including full SCD Type 2 history.

What a Run Does  

A run executes in dependency order (staging before history, history before bridge), so a layer is never built from a half-written one beneath it.

The source plugin reads records one at a time and packs them into Apache Arrow record batches. The agent relays each batch unchanged to the target plugin, which writes its records one at a time into the staging table through the data platform’s bulk-load interface. The SQL that builds each layer is rendered from Templates for your data platform’s own dialect. The same inputs produce the same pipeline every time, so a run can be reproduced.

When the Source Changes  

Sources change without warning. Every schema change is classified as a patch, minor, or major change and surfaced against the source for you to act on. The project’s driftPolicy declares what should happen at each level; applying it automatically at run time is on the roadmap. Individual objects can override that policy where one table needs different handling.

Next  

• Projects : where jobs, sources, and targets are declared
• Plugins : the connectors a pipeline runs on
• How It Works : the lifecycle end to end
• Data Quality Checks : what is verified while a run moves data