# Pipelines

How data travels from a source to your data platform, covering the jobs that move it, the layers it lands in, and the order a run executes them.

## Overview

A pipeline is the path your data takes from a source to your data platform. You do not write one. You declare the sources, the target, and the jobs that connect them, and Amfelt derives the rest from the data contract it discovered.

Everything a pipeline needs lives in the [project](/docs/concepts/projects/) file: which objects to move, which layer they land in, and how often.

## Jobs are the unit of work

A job moves data between one named source and one named target. It selects objects from the source contract with a glob, names the data platform `layer` it populates, and sets a default `writeMode` (how records land) for them. What the history layer then *keeps* is a separate setting, `historyMode`, which can be overridden per object. [Write modes and history](/docs/concepts/write-modes-and-history/) covers both.

Scheduling belongs to the job rather than to the project. Its `runs` are strategy-and-schedule pairs (a full load nightly and another on a weekend schedule are two runs of the same job; incremental runs are on the roadmap), and `maxParallel` bounds how many of the job's tables load at once.

## Layers

Data does not land in one place. Each job populates a named layer, and the layers build on each other:

| Layer | What it holds |
| --- | --- |
| `staging` | The source data as extracted |
| `history` | Change over time, tracked according to the history mode you chose |
| `bridge` | The views your analytics consume, with business-friendly names |
| `business` | Reserved for your own models on top of the bridge; Amfelt builds nothing in it today |

The [layering model](/platform/layering/) covers the history modes in more detail, including full SCD Type 2 history.

## What a run does

A run executes in dependency order (staging before history, history before bridge), so a layer is never built from a half-written one beneath it.

The source plugin reads records one at a time and packs them into Apache Arrow record batches. The agent relays each batch unchanged to the target plugin, which writes its records one at a time into the staging table through the data platform's bulk-load interface. The SQL (Structured Query Language) that builds each layer is rendered from [templates](/docs/concepts/templates/) for your data platform's own dialect. The same inputs produce the same pipeline every time, so a run can be reproduced.

## When the source changes

Sources change without warning. Every schema change is classified as a patch, minor, or major change and surfaced against the source for you to act on. The project's `driftPolicy` declares what should happen at each level; applying it automatically at run time is on the roadmap. Individual objects can override that policy where one table needs different handling.

## Next

- [Projects](/docs/concepts/projects/): where jobs, sources, and targets are declared
- [Plugins](/docs/concepts/plugins/): the connectors a pipeline runs on
- [How it works](/platform/how-it-works/): the lifecycle end to end
- [Data quality checks](/docs/concepts/data-quality/): what is verified while a run moves data