YAML reference

Generated from src/pipetree/model.py. For editor completion, use the JSON Schema.

Top level

Key	Type	Description
defaults	Defaults	Optional. Values for every table.
systems	map[string, System]	Declared systems.
notifications	Notifications	Optional. Who is told about a failed run.
any other key	Layer	A layer with a tables map of name to Table. The layer name is the default schema of its tables.

Defaults

Under defaults.

Field	Type	Default	Description
system_columns	list[string]	["_inserted_at", "_updated_at", "_is_deleted", "_execution_id", "_source_system"]	Names of the audit columns. Read from the config but not used to select columns: pipetree stamps its fixed audit set on every table.
schema_policy	“evolve” | “fail” | “ignore”	"evolve"	Schema drift policy for every table that does not set its own.

System

Under systems.<name>. Which other keys apply depends on type.

Field	Type	Default	Description
type	string	required	Source type: csv, json, parquet, sqlserver, kusto, storage_stream, d365_export, synapse_link or custom. Other keys are passed to the source reader.
any other key	string | {secret: name}	-	Passed to the source reader, for example path, host or class. A {secret: name} value is resolved by the platform at run time.

Table

Under <layer>.tables.<name>.

Field	Type	Default	Description
table_schema	string	-	Schema the table is created in. Defaults to the layer name.
source	Source | null	null	Read from a declared system. Exactly one of source and logic.
logic	string | null	null	Path to a .sql or .py file, relative to the config. Exactly one of source and logic.
business_key	list[string]	[]	Columns that identify a row. Required for scd1, scd2 and unknown_member.
strategy	“scd2” | “scd1” | “replace” | “append”	required	How the result is written to the table.
merge	Merge	-	Options for the scd1 and scd2 strategies.
depends_on	“auto” | list[string]	[]	auto reads the logic file; a list names parent tables (fqn or unambiguous bare name).
unknown_member	boolean	false	Seed one row with a sentinel business key for facts to resolve a missing key to.
schema_policy	“evolve” | “fail” | “ignore”	-	Schema drift policy for this table. Defaults to defaults.schema_policy.
surrogate_key	string | null	null	Name of a generated integer key column (scd1 and scd2 only). The unknown member gets -1.
tags	list[string]	[]	Labels for selecting tables with pipetree run --tag. Lowercase letters, digits, - and _, starting with a letter or digit, at most 40 characters, no duplicates within a table.

Notifications

Under notifications. See Notifications.

Field	Type	Default	Description
email	EmailNotification | null	null	Email a failed run through Azure Communication Services, authenticated with a token credential (no connection string).

Email

Under notifications.email.

Field	Type	Default	Description
endpoint	string | SecretRef	required	Endpoint of the Azure Communication Services resource, https://<resource>.communication.azure.com. A literal or {secret: name}.
sender	string	required	Sender address of a domain connected to the resource, for example DoNotReply@<guid>.azurecomm.net.
to	list[string]	required	Recipient addresses. At least one.
on	“failure”	"failure"	When to send. failure mails a run in which a table failed or was marked upstream_failed. No mail for a successful run.

Source

Under a table’s source.

Field	Type	Default	Description
system	string	required	Name of a system declared under systems.
object	string	required	What to read from the system: a table, entity or file name the reader understands.

Merge

Under a table’s merge. See Merge options.

Field	Type	Default	Description
sequence_by	list[string]	[]	Source columns that order the versions of a key. The highest value wins.
delete_when	string | null	null	Spark SQL expression, true for source rows that are deletes.
delete_mode	“soft” | “hard” | “ignore”	"soft"	What a delete row does: soft marks the row deleted, hard removes it (scd1 only), ignore drops delete rows from the batch.
ignore_columns	list[string]	[]	Columns whose changes alone are written in place: no new scd2 version and no audit column update.
snapshot	boolean	false	The batch is the complete current state of the source (scd1 and scd2 only). An active key of the table that is not in the batch is handled like a delete row, following delete_mode.
snapshot_max_missing_ratio	number	0.5	With snapshot: the table fails and nothing is written when more than this share of its active keys is missing from the batch.
snapshot_allow_empty	boolean	false	With snapshot: an empty batch is an error unless this is true (then every active key counts as missing, subject to snapshot_max_missing_ratio).