scd1
systems: crm: type: csv path: data/customer.csv
bronze: tables: customer: source: { system: crm, object: account } business_key: [id] strategy: scd1 merge: sequence_by: [seq] delete_when: "op = 'D'" delete_mode: softOne row per business_key. A new key is inserted, a changed row is updated in place, no history is kept. Five runs of this table, seq is the sequence_by column:
| Run | Source rows | Table after the run |
|---|---|---|
| 1 | 1 Berlin seq 1, 2 Bonn seq 1 |
1 Berlin, 2 Bonn |
| 2 | 1 Hamburg seq 2, 3 Köln seq 1 |
1 Hamburg, 2 Bonn, 3 Köln |
| 3 | 1 Paris seq 1 (older) |
unchanged: seq 1 is lower than the stored seq 2 |
| 4 | 2 D seq 2 (delete) |
2 Bonn with _is_deleted = true, other values kept |
| 5 | 2 Bonn seq 3 |
2 Bonn with _is_deleted = false |
Rules:
- Duplicates in a batch: the row with the highest
sequence_bywins, the rest are dropped and noted. Withoutsequence_bythe winner is arbitrary. - Across batches: with
sequence_by, a row older than the stored one is ignored, delete rows included. Equal or newer wins, so on a tie the later batch wins. NULL sorts lowest. Withoutsequence_bythe last batch wins. A soft delete stores its ownsequence_byvalues, so a later row with a lower sequence is ignored and the key stays deleted; an equal or higher one revives it. - Same batch twice: no change, no
_execution_idbump. - Deletes:
softmarks the row deleted,hardremoves it,ignoredrops delete rows. A delete for an unknown key does nothing. A soft-deleted key that comes back is active again, even with identical values. A hard-deleted key that comes back is a new insert, and a late older row for it simply inserts too: a hard delete removes the row and with it the sequence, so there is nothing to compare against. Withmerge.snapshot: true, a key missing from the batch is deleted as well, see Snapshot merge. ignore_columns: if only those columns changed, the row is updated in place and_execution_idstays.- NULL keys: a row whose key is NULL (all key columns NULL for a composite key) is dropped and noted. See NULL business keys.
- Schema:
schema_policyapplies.
Options in detail: Merge options.