Spec schemas¶

Field-by-field reference for the five spec kinds. Parsing is strict: unknown keys, duplicate YAML keys, and grammar violations are hard SpecParseErrors, batched per document with a source path per failure. Parse validates shape and grammar only — whether references exist is checked at resolution.

Each project document self-identifies by its version key: spec_version (EntityModel), mapping_version (Mapping), metrics_version (MetricSet), marts_version (MartSet). A project holds exactly one EntityModel, any number of Mappings, and at most one MetricSet and one MartSet. The Catalog (catalog_version) is not part of a project — load it with load_catalog and pass it separately.

Shared grammars¶

Reserved names. metric_time may not be used as a field, dimension, or role name — the planner owns it as the canonical query-time dimension. The generated data-quality and ingestion-metadata columns are reserved on the same terms: _quality_flags, _quality_ok, _load_id, _ingested_at, _source_row_id, has_quality_flags.

Retention units. Months and years are deliberately absent — they are not fixed durations, and a retention window that means something different in February is a legal problem rather than a convenience. Minutes are absent because m would read as either.

Catalog (catalog_version)¶

One per vertical, loaded via load_catalog(text).

CanonicalField¶

Recipe¶

CanonicalRelationship¶

MetricTemplate¶

Same shape as a Metric minus template/cumulative, with additivity required. A project metric that names it via template: merges values with the metric's own winning.

DateDimension¶

One definition emits both the gold.dim_date model and the MetricFlow time-spine declaration — bounds are calendar years, never a clock.

EntityModel (spec_version)¶

Exactly one per project.

Entity¶

Field¶

AssertClause¶

All members optional; at least one should be present for the clause to do anything.

not_null and enum lower to builtin audits/tests; min, max, and regex lower to custom audit artifacts.

Relationship¶

Entity quality rules¶

Row rules, discriminated on rule. Both read more than one column, which is why they sit on the entity rather than on a mapping field.

referential carries on_missing, not on_fail: unknown_member is a disposition no other rule has (the row passes with its fk rewritten to the reserved '__unknown__' member), and fail is deliberately unavailable — a pipeline-stopping orphan gate is a reconcile check instead. unknown_member requires a string-typed fk; the relationship's to side may not be the declaring entity.

Dedupe¶

Partitions by the entity's key. tie_break is optional in the grammar and mandatory in a model: its absence under keep: latest_by is the compile error DedupeTieBreakMissing. The order finishes with _source_row_id, with NULLS LAST pinned on every sort key, so the winner is unique by construction.

Quarantine¶

The block is required whenever any rule can quarantine — QuarantineRetentionMissing otherwise, never a default, because reject rows hold raw source payloads. A redact: path may not intersect a path the mapping reads (RedactionConflict): replay re-runs the mapping against raw, and a redacted path is gone by then.

Reconcile¶

Document-level, a sibling of entities: — a check relates two entities and belongs to neither.

A side is one of two closed shapes:

left:  "sum(order_item.line_total) by order_id"   # <agg>(<entity>.<column>) by <columns>
right: "order.total_amount"                       # <entity>.<column>, keyed by that entity's key

Both sides must key on the same columns, since they join on their keys. Anything outside the grammar is a GuardrailError — a reconcile side is a declared shape, not SQL.

Mapping (mapping_version)¶

One document per (source, target entity) pair.

An entity that uses quarantine: or dedupe: requires the bronze ingestion-metadata columns _load_id, _ingested_at, and _source_row_id. They are reserved names, so a mapping states they exist by listing them in unmapped:; their absence is IngestionMetadataMissing. Their values are checked at run time by a generated blocking audit: none of the three may be null, _source_row_id must be unique per source row, and _ingested_at must cast to timestamp. The audit is emitted only for dialects with TRY_CAST; on the others the entity is UnsupportedByTarget.

KeyField¶

FieldMapping — two forms¶

A field mapping is simple or recipe, discriminated on the presence of recipe:

# simple: one source path plus a transform chain
quantity: {from: "$.qty", transform: [to_int]}

# recipe: a recorded catalog recipe id plus alias → path bindings
unit_price:
  recipe: from_total
  from: {line_total: "$.total", quantity: "$.qty"}

Transform steps¶

A step is a bare name or a single-key mapping; args normalize to a tuple:

transform: [trim, to_string]              # zero-arg steps
transform: [{parse_ts: ISO8601}]          # one arg
transform: [{to_decimal: [12, 4]}]        # multiple args

The available names are the closed whitelist in the transforms reference; existence is checked at typecheck, not parse.

Field quality rules¶

A closed catalogue, discriminated on rule. New rules are RFC amendments, never config. Every rule requires on_fail ∈ flag | quarantine | fail — there is no project-wide default, and there is deliberately no drop and no repair in v1.

Notes that change how you write them:

• coercible is implicit and opt-in per entity. An entity joins the quality system by declaring quality:, quarantine:, or any field-level quality: — dedupe: alone does not. Every mapped field of a quality-carrying entity then gets a coercible rule at quarantine unless you declare one explicitly to override the disposition. It is forced to fail on any field named by dedupe.field/tie_break (DedupeDispositionConflict otherwise).
• Bounds are separate rules. range/length take one or both bounds, and two bounds needing different dispositions are simply two rules.
• range bounds are exact. Write an int, or quote the value: a quoted bound must be an exact decimal ("10000000000", "0.01" — sign and fraction allowed, no exponent) or an ISO date/timestamp. "nan", "inf", and "1e10" are refused at parse: the first two make a comparison that is never true on one engine and always true on another, and the third renders as a float literal, which no emission path may carry.
• in_enum takes no values. The admissible set is the chain's enum_map targets; restating it here would let the two drift.
• unique is per partition slice, in both full and incremental runs — for an unpartitioned entity the slice is the whole table. Cross-partition duplicates are key-based dedupe's job, in every mode.
• Nulls belong to not_null and coercible only. Every other rule's violation predicate must be definitively TRUE to fire; a NULL-involved comparison evaluates to SQL UNKNOWN and stays silent.

assert: on an entity field and quality: on a mapping field are different tools: assert: is "alert me" (an audit that observes), quality: is "act on the row" (the disposition system that routes). A field may carry both.

The portable regex subset¶

pattern regexes are restricted to what the shipped dialect ports agree on — DuckDB and Trino run RE2, Postgres runs POSIX ARE. The subset is an allowlist: a pattern is scanned left to right and anything the list below does not name is refused at parse, with the construct named in the error.

Escapes and character classes are scanned, not substring-matched, so \(?= and [(?=] are literals rather than lookahead.

Patterns must be anchored, and you write the anchors. [0-9]{5} is refused; ^[0-9]{5}$ is accepted. A SQL regex predicate matches a substring, so the unanchored form would accept abc12345xyz. Each top-level alternative carries its own pair: ^a$|^b$, not ^a|b$.

Capturing groups are refused in favour of (?:…): a rule is a boolean match and captures nothing, and numbered groups are what backreferences read.

Two divergences are accepted and documented rather than refused: . excludes newline on RE2 and includes it on Postgres ARE, and \d/\w/\s are ASCII on RE2 but locale-defined on ARE.

At the guardrail stage each pattern is additionally checked against the shipped dialect ports (duckdb, postgres, trino) for a regex surface — a dialect that declares no regex support, or through which the pattern literal cannot be rendered, is a GuardrailError. That check is deliberately narrow: bloomery never executes SQL, so nothing at compile time can prove an engine's regex engine accepts a pattern. The subset above is what carries the portability claim.

MetricSet (metrics_version)¶

At most one per project.

Metric¶

A metric is a template instantiation (template: plus overrides) or fully inline.

RatioSpec¶

SemiAdditivePolicy¶

MartSet (marts_version)¶

At most one per project; a project without marts compiles silver only.

Mart¶

Flatten steps — two forms¶

Discriminated on via vs date:

flatten:
  - {via: item_of_order, prefix: order_}    # flatten a relationship
  - {date: order_date, role: ordered}       # declare a date role