Choose a Load Strategy

A job describes what moves for one source. Each job has one or more runs, and each run carries a strategy and a schedule. The strategy is the decision on this page: how a given execution loads the source.

Full Load  

A full load reads everything the object holds, every time.

It is simple, it is self-correcting (a bad previous run is fixed by the next one), and it is the right default for small and medium objects, for reference and dimension data that changes rarely, and for the first load of anything. What it costs you is the same work on every execution, whether or not anything changed.

Incremental Load (On the Roadmap)  

An incremental load will read only the records that changed since the previous run, using a cursor: a column that reliably moves forward whenever a record changes, such as a modification timestamp or a monotonic sequence.

It is not available yet. Today the platform runs full loads only, and a job that names an incremental run is refused when you save it.

When it arrives, choose the cursor with care, because it is where incremental loads go wrong. A column that a bulk update does not touch, or that can be set backwards, would silently skip records, and silently is the problem. Nothing fails; you simply stop receiving some of the data.

Choosing  

Use a full load. If a large object takes too long, schedule it less often rather than more. Once incremental loads arrive, switch an object only when its cursor is one you actually trust: a full load on a less frequent schedule is a better trade than an incremental load that misses records.

Classify Your Objects  

Each object also carries a table type telling Amfelt how to treat it: reference data, master data, or transactional data. That classification drives how the object is historized (every change kept as a dated version) in the history layer, so it is worth setting rather than leaving at the default.

Let the Advisor Propose It  

Recommend patterns profiles the source’s objects and suggests a table type for each one, based on what the data actually looks like rather than on what the table is called.

Suggestions are proposals, not changes. Each one is shown against the object with the option to accept it, so you review the advisor’s reasoning object by object and take the ones you agree with.

Next  

Decide when it runs: Schedule a Job .