Jobs

The consolidated job model: glob-based object selection, run strategies and schedules, and per-column axes (extract, compare, expose) plus per-column bridge config.

Job

type: Job

A pipeline job: which source and target to move data between, how to select objects, and when to run.

Example

name: crm-sync
source: crm
target: warehouse
layer: staging
select:
  include: ["public.*"]
  objects:
    public.orders:
      historyMode: scd2
      columns:
        config:
          total:
            transform: "value / 100"
            target: totalDollars
runs:
  - strategy: full
    schedule: "0 * * * *"

Property	Type	Required	Description
`layer`	string	No	Layer is the data platform layer this job populates: one of “staging”, “history”, “bridge”, “business”, or “all”. Used to namespace generated schemas/tables.
Layer is the data platform layer this job populates: one of “staging”, “history”, “bridge”, “business”, or “all”. Used to namespace generated schemas/tables.
`maxParallel`	integer	No	MaxParallel is the number of objects this job processes concurrently. 0 or unset runs sequentially (one object at a time); values above 1 enable concurrency, further capped by the target’s connection limit.
MaxParallel is the number of objects this job processes concurrently. 0 or unset runs sequentially (one object at a time); values above 1 enable concurrency, further capped by the target’s connection limit.
`name`	string	Yes	Name identifies this job within the project. Must be unique among jobs.
Name identifies this job within the project. Must be unique among jobs.
`runs`	Run []	No	Runs are the (strategy, schedule) executions of this job’s selection.
Runs are the (strategy, schedule) executions of this job’s selection.
`schemaPrefix`	string	No	SchemaPrefix overrides the generated schema/dataset name prefix for this job’s objects. Defaults to the project’s naming convention when empty.
SchemaPrefix overrides the generated schema/dataset name prefix for this job’s objects. Defaults to the project’s naming convention when empty.
`select`	Select	No	Select is the glob-based object selection over the source contract.
Select is the glob-based object selection over the source contract.
`source`	string	Yes	Source is the name of the source this job reads from. Must match the name of an entry in the project’s sources list.
Source is the name of the source this job reads from. Must match the name of an entry in the project’s sources list.
`target`	string	Yes	Target is the name of the target this job writes to. Must match the name of an entry in the project’s targets list.
Target is the name of the target this job writes to. Must match the name of an entry in the project’s targets list.
`writeMode`	string	No	WriteMode records the intended load strategy for objects in this job’s selection: “append”, “upsert” or “replace”. The value is recorded but not acted on. The staging layer always replaces (it is truncated and reloaded on every run), so all three values land identically; how records accumulate across runs is decided by historyMode at the history layer. Omitting the key is the correct authoring shape. The agent warns once per job when the writeMode it resolves is set to anything other than “replace”; removing the key or spelling it “replace” both silence that warning, because both name what the staging layer already does.
WriteMode records the intended load strategy for objects in this job’s selection: “append”, “upsert” or “replace”. The value is recorded but not acted on. The staging layer always replaces (it is truncated and reloaded on every run), so all three values land identically; how records accumulate across runs is decided by historyMode at the history layer. Omitting the key is the correct authoring shape. The agent warns once per job when the writeMode it resolves is set to anything other than “replace”; removing the key or spelling it “replace” both silence that warning, because both name what the staging layer already does.

Object Override

type: ObjectOverride

Per-object operational config.

Property	Type	Required	Description
`columns`	ColumnAxes	No	Columns configures the extract/compare/expose glob axes and per-column bridge-view overrides for this object.
Columns configures the extract/compare/expose glob axes and per-column bridge-view overrides for this object.
`cursor`	string	No	Cursor is the column used to detect new/changed records for incremental loads.
Cursor is the column used to detect new/changed records for incremental loads.
`cursorType`	string	No	CursorType says how the cursor column orders records: “timestamp” or “sequence”. Required when cursor is set and a run is incremental; it is never guessed. Valid values are: `timestamp`, `sequence`.
CursorType says how the cursor column orders records: “timestamp” or “sequence”. Required when cursor is set and a run is incremental; it is never guessed. Valid values are: `timestamp`, `sequence`.
`dedupKey`	string	No	DedupKey is the column (or comma-separated columns) that uniquely identifies a record, used to deduplicate on load.
DedupKey is the column (or comma-separated columns) that uniquely identifies a record, used to deduplicate on load.
`deleteDetection`	string	No	DeleteDetection selects how deleted source records are found: “none”, “fullRun” (a full run reconciles deletes), “keyScan” (compare the source keys), or “softDelete” (a column marks deleted records; needs softDeleteColumn and softDeleteValue). Defaults to “fullRun”. Valid values are: `none`, `fullRun`, `keyScan`, `softDelete`.
DeleteDetection selects how deleted source records are found: “none”, “fullRun” (a full run reconciles deletes), “keyScan” (compare the source keys), or “softDelete” (a column marks deleted records; needs softDeleteColumn and softDeleteValue). Defaults to “fullRun”. Valid values are: `none`, `fullRun`, `keyScan`, `softDelete`.
`eventTimeCol`	string	No	EventTimeCol is the column holding the event’s business timestamp, used for event-time ordering and late-arrival handling.
EventTimeCol is the column holding the event’s business timestamp, used for event-time ordering and late-arrival handling.
`historyMode`	string	No	HistoryMode overrides the selection default for this object: one of “scd2”, “scd2_event_time”, “scd1”, “snapshot”, “append”, “dedup_append”, or “replace”.
HistoryMode overrides the selection default for this object: one of “scd2”, “scd2_event_time”, “scd1”, “snapshot”, “append”, “dedup_append”, or “replace”.
`lateArrivalThreshold`	string	No	LateArrivalThreshold is the maximum delay tolerated for late-arriving events before they are dropped or flagged, expressed as a duration string (e.g. “24h”).
LateArrivalThreshold is the maximum delay tolerated for late-arriving events before they are dropped or flagged, expressed as a duration string (e.g. “24h”).
`lookback`	string, integer	No	Lookback re-reads records just behind the stored cursor on each incremental run: a duration (e.g. “5m”) for a timestamp cursor, a non-negative integer for a sequence cursor. Defaults to “5m” for a timestamp cursor and 0 for a sequence cursor.
Lookback re-reads records just behind the stored cursor on each incremental run: a duration (e.g. “5m”) for a timestamp cursor, a non-negative integer for a sequence cursor. Defaults to “5m” for a timestamp cursor and 0 for a sequence cursor.
`softDeleteColumn`	string	No	SoftDeleteColumn is the column that marks a record as deleted when deleteDetection is “softDelete”.
SoftDeleteColumn is the column that marks a record as deleted when deleteDetection is “softDelete”.
`softDeleteValue`	string, integer, boolean	No	SoftDeleteValue is the value of softDeleteColumn that marks a record as deleted when deleteDetection is “softDelete”.
SoftDeleteValue is the value of softDeleteColumn that marks a record as deleted when deleteDetection is “softDelete”.
`tableType`	string	No	TableType is the business classification of this object: one of “master”, “transaction”, or “reference”. It picks the default history mode when HistoryMode is unset. Values from before the taxonomy was reduced still load and are mapped onto it.
TableType is the business classification of this object: one of “master”, “transaction”, or “reference”. It picks the default history mode when HistoryMode is unset. Values from before the taxonomy was reduced still load and are mapped onto it.
`window`	Window	No	Window reloads a trailing time window on each incremental run. Its size defaults to lateArrivalThreshold.
Window reloads a trailing time window on each incremental run. Its size defaults to lateArrivalThreshold.

Window

type: Window

Window reload settings: the records whose column value falls inside the trailing window are read again on each incremental run.

Property	Type	Required	Description
`column`	string	Yes	Column is the timestamp column the window is measured on.
Column is the timestamp column the window is measured on.
`size`	string	No	Size is the window length as a duration (e.g. “24h”). Defaults to the object’s lateArrivalThreshold; without either, the window is invalid.
Size is the window length as a duration (e.g. “24h”). Defaults to the object’s lateArrivalThreshold; without either, the window is invalid.

Select

type: Select

Glob-based selection of objects from the source contract, with optional per-object overrides.

Example

include: ["public.*", "sales.*"]
exclude: ["public.audit_*"]
defaults:
  historyMode: scd2
objects:
  public.orders:
    tableType: transaction
    columns:
      extract:
        exclude: ["_raw_blob"]

Property	Type	Required	Description
`defaults`	SelectDefaults	No	Defaults holds selection settings applied to every matched object unless a per-object entry in Objects overrides them.
Defaults holds selection settings applied to every matched object unless a per-object entry in Objects overrides them.
`exclude`	string[]	No	Exclude lists glob patterns for object names to drop from the selection, applied after Include.
Exclude lists glob patterns for object names to drop from the selection, applied after Include.
`include`	string[]	No	Include lists glob patterns for object names to select from the source contract. Defaults to ["*"] (all objects) when empty.
Include lists glob patterns for object names to select from the source contract. Defaults to ["*"] (all objects) when empty.
`objects`	object	No	Objects maps an object name or glob to per-object operational overrides (history mode, cursor, column axes, etc.).
Objects maps an object name or glob to per-object operational overrides (history mode, cursor, column axes, etc.).

Select Defaults

type: SelectDefaults

Selection defaults applied to every matched object unless a per-object override sets its own value.

Property	Type	Required	Description
`historyMode`	string	No	HistoryMode is the default history-tracking mode for matched objects: one of “scd2”, “scd2_event_time”, “scd1”, “snapshot”, “append”, “dedup_append”, or “replace”.
HistoryMode is the default history-tracking mode for matched objects: one of “scd2”, “scd2_event_time”, “scd1”, “snapshot”, “append”, “dedup_append”, or “replace”.
`writeMode`	string	No	WriteMode records the intended load strategy for matched objects: “append”, “upsert” or “replace”. The value is recorded but not acted on. The staging layer always replaces, so all three values land identically; use historyMode to control how records accumulate across runs. Omitting the key is the correct authoring shape. This default is what the agent resolves first, and it warns when the resolved writeMode is set to anything other than “replace”; removing the key or spelling it “replace” both silence that warning.
WriteMode records the intended load strategy for matched objects: “append”, “upsert” or “replace”. The value is recorded but not acted on. The staging layer always replaces, so all three values land identically; use historyMode to control how records accumulate across runs. Omitting the key is the correct authoring shape. This default is what the agent resolves first, and it warns when the resolved writeMode is set to anything other than “replace”; removing the key or spelling it “replace” both silence that warning.

Column Selector

type: ColumnSelector

An include/exclude glob-list over column names.

Property	Type	Required	Description
`exclude`	string[]	No	Exclude lists glob patterns for column names to drop, applied after Include.
Exclude lists glob patterns for column names to drop, applied after Include.
`include`	string[]	No	Include lists glob patterns for column names to select. Defaults to ["*"] (all columns) when empty.
Include lists glob patterns for column names to select. Defaults to ["*"] (all columns) when empty.

Column Axes

type: ColumnAxes

Per-column configuration for an object: the extract/compare/expose glob axes plus per-column bridge-view overrides.

Example

extract:
  exclude: ["_raw_blob"]         # never read this column from the source
compare:
  exclude: ["updated_at"]        # ignore this column when detecting changes
expose:
  include: ["order_id", "total"] # only these appear in the bridge view
config:
  total:
    transform: "value / 100"     # SQL expression; "value" = the source column
    target: totalDollars         # rename in the bridge view

Property	Type	Required	Description
`compare`	ColumnSelector	No	Compare selects which columns participate in change detection (drift and history comparisons). Defaults to all extracted columns when unset.
Compare selects which columns participate in change detection (drift and history comparisons). Defaults to all extracted columns when unset.
`config`	object	No	Config holds per-column bridge-view overrides, keyed by source column name.
Config holds per-column bridge-view overrides, keyed by source column name.
`expose`	ColumnSelector	No	Expose selects which columns are surfaced in the bridge view. Defaults to all extracted columns when unset.
Expose selects which columns are surfaced in the bridge view. Defaults to all extracted columns when unset.
`extract`	ColumnSelector	No	Extract selects which columns are read from the source. Defaults to all columns when unset.
Extract selects which columns are read from the source. Defaults to all columns when unset.

Column Config

type: ColumnConfig

Per-column bridge-VIEW configuration (keyed by the source column name in ColumnAxes.config).

Property	Type	Required	Description
`bridge`	boolean	No	Bridge controls whether the column appears in the bridge VIEW. Default: true.
Bridge controls whether the column appears in the bridge VIEW. Default: true.
`target`	string	No	Target column name in the bridge VIEW. Defaults to the source column when empty.
Target column name in the bridge VIEW. Defaults to the source column when empty.
`transform`	string	No	Transform is a SQL expression for the bridge VIEW SELECT list; “value” is the placeholder for the source column reference.
Transform is a SQL expression for the bridge VIEW SELECT list; “value” is the placeholder for the source column reference.

Run

type: Run

One scheduled execution of a job’s selection under one strategy.

Example

name: hourly
strategy: full                   # incremental and cdc are on the roadmap
schedule: "0 * * * *"            # cron; omit for on-demand runs

Property	Type	Required	Description
`name`	string	No	Name identifies this run within the job’s Runs list. Optional when a job has only one run.
Name identifies this run within the job’s Runs list. Optional when a job has only one run.
`schedule`	string	No	Schedule is the cron expression controlling when this run executes. Unset means the run is triggered manually or by the job’s default schedule.
Schedule is the cron expression controlling when this run executes. Unset means the run is triggered manually or by the job’s default schedule.
`strategy`	string	Yes	Strategy is the load strategy of this run. “full” is the strategy that ships; “incremental” and “cdc” are accepted values for loads on the roadmap, and an agent refuses “cdc”. Valid values are: `full`, `incremental`, `cdc`.
Strategy is the load strategy of this run. “full” is the strategy that ships; “incremental” and “cdc” are accepted values for loads on the roadmap, and an agent refuses “cdc”. Valid values are: `full`, `incremental`, `cdc`.

Last updated on November 1, 2026

Docs

Jobs

Job

Object Override

Window

Select

Select Defaults

Column Selector

Column Axes

Column Config

Run

On this page