Spec schemas¶

Field-by-field reference for eight of the ten spec kinds; the two composition documents, exports_version and imports_version, are described where they are consumed, under Composing across projects. Parsing is strict: unknown keys, duplicate YAML keys, and grammar violations are hard SpecParseErrors, batched per document with a source path per failure. Parse validates shape and grammar only — whether references exist is checked at resolution.

Each project document self-identifies by its version key: spec_version (EntityModel), mapping_version (Mapping), metrics_version (MetricSet), marts_version (MartSet), steps_version (StepSet), exposures_version (ExposureSet), retrieval_version (RetrievalSpec). A project holds exactly one EntityModel, any number of Mappings, and at most one of every other kind. The Catalog (catalog_version) is not part of a project — load it with load_catalog and pass it separately.

A version key is required, and it is also how bloomery tells one kind of document from another. Each is pinned to 1: a document declaring a version bloomery does not implement is refused, never read as one it does.

Shared grammars¶

timestamp is UTC. Not "a timestamp in whatever zone the source used" — the type means an instant, and two doors lead into it: {to_utc: <zone>}, which converts a wall clock, and zone_in: UTC, which declares that the source's wall clocks already are one. A wall clock read for its position with neither behind it is refused — see zone_in:. date carries no instant and asks nothing.

Reserved names. metric_time may not be used as a field, dimension, or role name — the planner owns it as the canonical query-time dimension. The generated data-quality and ingestion-metadata columns are reserved on the same terms: _quality_flags, _quality_ok, _quality_repairs, _load_id, _ingested_at, _source_row_id, has_quality_flags. So is _source, the union merge's provenance column — reserved unconditionally rather than only on merged entities, because a name that is legal until a second mapping arrives is a trap laid for the change that adds one.

Reserved entity names. An entity may not be called canonical, exposure, mart, metric, source or step. Those are the six lineage node-id prefixes: a metric is metric.<name>, a canonical field canonical.<name>, a gold relation mart.<name>, and an entity field is <entity>.<field> with no prefix at all — so an entity named metric with a field revenue produces the same id as a metric named revenue, and bloomery lineage --node metric.revenue can no longer say which one you meant. Reserved unconditionally, for the same reason _source is: a name that is legal until someone adds a metric in another file is a trap laid for them. This binds a step output too, which is named after the last segment of the relation its wiring binds.

Retention units. Months and years are deliberately absent — they are not fixed durations, and a retention window that means something different in February is a legal problem rather than a convenience. Minutes are absent because m would read as either.

Catalog (catalog_version)¶

One per vertical, loaded via load_catalog(text).

FxRates¶

The shape of a rate table the operator supplies; bloomery reads it and never builds it. See transforms for what it is for.

No two roles may name the same column: each reads a different column of the rate relation, and a shared name makes the emitted predicate compare a column against itself.

CanonicalField¶

Recipe¶

CanonicalRelationship¶

MetricTemplate¶

Same shape as a Metric minus template, with additivity required. A project metric that names it via template: merges values with the metric's own winning.

DateDimension¶

One definition emits both the gold.dim_date model and the MetricFlow time-spine declaration — bounds are calendar years, never a clock.

EntityModel (spec_version)¶

Exactly one per project.

Entity¶

arrival_lag reprocesses past days on SQLMesh. The model pins interval_unit 'day' and sets lookback to the lag in whole days, rounded up (30h is lookback 2), so a late row lands on a later run rather than never. A cron more frequent than daily on such a model is refused by SQLMesh at load unless you also set allow_partials. Unset, neither is emitted. dbt carries nothing: its incremental entity merges its whole select on the key every run, so a late row already lands.

materialization is derived, never inferred, and that distinction is the rule. An entity with a partition key gets incremental_by_partition without anyone writing it — that is a default: visible in the compiled output, and overridable by saying otherwise. What bloomery will not do is infer the strategy from grain and partition key in the ambiguous cases, because an inference is a guess the author cannot see, and a wrong guess here produces a wrong backfill scope that surfaces as missing rows.

bloomery derives defaults; it does not infer intent.

The same line runs through recipes: the compiler validates the recipe a mapping recorded and never chooses one itself. If a specific inference is later shown to be both safe and visible in the emitted artifact, it can be argued as a change — a default carries no such burden, which is why one is cheap and the other is not.

Field¶

AssertClause¶

All members optional; at least one should be present for the clause to do anything.

not_null and enum lower to builtin audits/tests; min, max, and regex lower to custom audit artifacts.

Relationship¶

Entity quality rules¶

Row rules, discriminated on rule. Both read more than one column, which is why they sit on the entity rather than on a mapping field.

referential carries on_missing, not on_fail: unknown_member is a disposition no other rule has (the row passes with its fk rewritten to the reserved '__unknown__' member), and fail is deliberately unavailable — a pipeline-stopping orphan gate is a reconcile check instead. unknown_member requires a string-typed fk; the relationship's to side may not be the declaring entity.

Dedupe¶

Partitions by the entity's key. tie_break is optional in the grammar and mandatory in a model: its absence under keep: latest_by is the compile error DedupeTieBreakMissing. The order finishes with _source_row_id, with NULLS LAST pinned on every sort key, so the winner is unique by construction.

Quarantine¶

The block is required whenever any rule can quarantine — QuarantineRetentionMissing otherwise, never a default, because reject rows hold raw source payloads. A redact: path may not intersect a path the mapping reads (RedactionConflict): replay re-runs the mapping against raw, and a redacted path is gone by then.

Reconcile¶

Document-level, a sibling of entities: — a check relates two entities and belongs to neither.

A side is one of two closed shapes:

left:  "sum(order_item.line_total) by order_id"   # <agg>(<entity>.<column>) by <columns>
right: "order.total_amount"                       # <entity>.<column>, keyed by that entity's key

Both sides must key on the same columns, since they join on their keys. Anything outside the grammar is a GuardrailError — a reconcile side is a declared shape, not SQL.

A NULL key is a group, not an absence. The comparison is null-safe throughout, because the aggregate side's GROUP BY already collects every NULL key into one group and everything below it has to read that column by the same rule. The emitted model is a _keys CTE — both sides' key columns, UNIONed — with each side attached by LEFT JOIN ... IS NOT DISTINCT FROM. A key present on one side only still fails, which is what the outer join is for, but a NULL group that agrees on both sides passes, as one row. Neither the entity key nor an aggregate's by columns are forced non-nullable, so this is reachable from an ordinary project; if a NULL key is itself a data-quality problem for you, say so with a not_null rule.

The long spelling is not a stylistic choice. FULL OUTER JOIN ... ON a IS NOT DISTINCT FROM b says the same thing in one line and renders identically on all three dialects, and PostgreSQL then refuses to run it — its full join is planned as a merge or hash join, and a null-safe condition is neither. One shape is emitted everywhere rather than a check that compares NULL keys on two engines out of three.

Mapping (mapping_version)¶

One document per (source, target entity) pair.

An entity that uses quarantine: or dedupe: requires the bronze ingestion-metadata columns _load_id, _ingested_at, and _source_row_id. They are reserved names, so a mapping states they exist by listing them in unmapped:; their absence is IngestionMetadataMissing. Their values are checked at run time by a generated blocking audit: none of the three may be null, _source_row_id must be unique per source row, and _ingested_at must cast to timestamp. The audit is emitted only for dialects with TRY_CAST; on the others the entity is UnsupportedByTarget.

Freshness¶

A staleness threshold for the mapping's bronze relation. bloomery emits it; the framework measures it — dbt source freshness runs SELECT MAX(_ingested_at), and nothing in the compiler reads a clock or touches data.

Durations are the same closed grammar as quarantine.retention — 6h, 24h, 90d, 2w. Both keys are required and neither is defaulted: a source with no freshness: block gets no threshold at all, because a guessed one would emit an assertion nobody made.

error_after earlier than warn_after is refused at parse — the warning could never fire. Equal values are accepted; both thresholds then trip together.

Two more refusals come from the guardrail stage, and both follow from one sentence: a threshold is a statement about the relation, not about the mapping that carries it.

• A freshness: block on a mapping whose target entity declares neither quarantine: nor dedupe: is refused. Only those blocks make _ingested_at mandatory, so without one the emitted loaded_at_field would name a column that may not exist — the project compiles clean and the framework errors when it runs.
• Two mappings giving different thresholds to one bronze relation are refused, naming both. The emitted sources.yml holds one entry per relation, so one of them would be silently dropped. Equal thresholds collapse and are not a conflict; a mapping that declares none is not disagreeing with one, so a relation read by a declaring mapping and a silent one is fine.

Only dbt emits anything for a threshold. SQLMesh and Cube model no source object, so there is nothing there to attach one to and nothing is refused.

KeyField¶

FieldMapping — two forms¶

A field mapping is simple or recipe, discriminated on the presence of recipe:

# simple: one source path plus a transform chain
quantity: {from: "$.qty", transform: [to_int]}

# recipe: a recorded catalog recipe id plus alias → path bindings
unit_price:
  recipe: from_total
  from: {line_total: "$.total", quantity: "$.qty"}

Transform steps¶

A step is a bare name or a single-key mapping; args normalize to a tuple:

transform: [trim, to_string]              # zero-arg steps
transform: [{parse_ts: ISO8601}]          # one arg
transform: [{to_decimal: [12, 4]}]        # multiple args

The available names are the closed whitelist in the transforms reference; existence is checked at typecheck, not parse.

Field quality rules¶

A closed catalogue, discriminated on rule. New rules are RFC amendments, never config. Every rule requires on_fail ∈ flag | quarantine | fail — there is no project-wide default, and there is deliberately no drop and no repair in v1.

Notes that change how you write them:

• coercible is implicit and opt-in per entity. An entity joins the quality system by declaring quality:, quarantine:, or any field-level quality: — dedupe: alone does not. Every mapped field of a quality-carrying entity then gets a coercible rule at quarantine unless you declare one explicitly to override the disposition. It is forced to fail on any field named by dedupe.field/tie_break (DedupeDispositionConflict otherwise).
• Bounds are separate rules. range/length take one or both bounds, and two bounds needing different dispositions are simply two rules.
• range bounds are exact. Write an int, or quote the value: a quoted bound must be an exact decimal ("10000000000", "0.01" — sign and fraction allowed, no exponent) or an ISO date/timestamp. "nan", "inf", and "1e10" are refused at parse: the first two make a comparison that is never true on one engine and always true on another, and the third renders as a float literal, which no emission path may carry.
• in_enum takes no values. The admissible set is the chain's enum_map targets; restating it here would let the two drift.
• unique is per partition slice, in both full and incremental runs — for an unpartitioned entity the slice is the whole table. Cross-partition duplicates are key-based dedupe's job, in every mode.
• Nulls belong to not_null and coercible only. Every other rule's violation predicate must be definitively TRUE to fire; a NULL-involved comparison evaluates to SQL UNKNOWN and stays silent.

assert: on an entity field and quality: on a mapping field are different tools: assert: is "alert me" (an audit that observes), quality: is "act on the row" (the disposition system that routes). A field may carry both.

The portable regex subset¶

pattern regexes are restricted to what the shipped dialect ports agree on — DuckDB and Trino run RE2, Postgres runs POSIX ARE. The subset is an allowlist: a pattern is scanned left to right and anything the list below does not name is refused at parse, with the construct named in the error.

Escapes and character classes are scanned, not substring-matched, so \(?= and [(?=] are literals rather than lookahead.

Patterns must be anchored, and you write the anchors. [0-9]{5} is refused; ^[0-9]{5}$ is accepted. A SQL regex predicate matches a substring, so the unanchored form would accept abc12345xyz. Each top-level alternative carries its own pair: ^a$|^b$, not ^a|b$.

Capturing groups are refused in favour of (?:…): a rule is a boolean match and captures nothing, and numbered groups are what backreferences read.

Two divergences are accepted and documented rather than refused: . excludes newline on RE2 and includes it on Postgres ARE, and \d/\w/\s are ASCII on RE2 but locale-defined on ARE.

At the guardrail stage each pattern is additionally checked against the shipped dialect ports (duckdb, postgres, trino) for a regex surface — a dialect that declares no regex support, or through which the pattern literal cannot be rendered, is a GuardrailError. That check is deliberately narrow: bloomery never executes SQL, so nothing at compile time can prove an engine's regex engine accepts a pattern. The subset above is what carries the portability claim.

MetricSet (metrics_version)¶

At most one per project.

Metric¶

A metric is a template instantiation (template: plus overrides) or fully inline.

RatioSpec¶

DerivedSpec¶

A metric with no measure of its own, computed by an expression over others. The alias is the mapping key rather than a field, because the expression references it.

The expression must reference exactly the declared aliases — an unknown one and an unread one are both refused (InvalidMetricShape). The inputs are also the metric's dependency edges; they need not be repeated in requires_metrics.

MetricInputSpec¶

MetricOffset¶

Exactly one of the two.

CumulativeSpec¶

Exactly one of the two. The metric keeps its own agg/expr and its own additivity — those describe the measure, this describes how it accumulates, and a metric with neither agg nor expr has nothing to accumulate (InvalidMetricShape). semi_additive is refused here: its over: dimension is a date role and a window accumulates along that same axis.

last rather than MetricFlow's own first, and it is this project's only deliberate divergence from it: on a month totalling 257 a grain_to_date: month metric reported 100 under first — the running total on the first day, which is not month-to-date by any reading. The key exists so the default is a default rather than a decision taken from the author.

MetricFilter¶

A typed predicate, never a SQL string: values are validated against the flattened column's declared logical type and are never cast.

A filter compares its dimension against literals (values:) or against another dimension (column:) — exactly one of the two. The comparison is admitted only when both columns are proven to be roles of one dimension member: both flatten steps declare role_of: naming the same dimension, and both columns carry the same member.

filter:
  - {dimension: billing_region, op: eq, column: shipping_region}

Without the declarations the metric is refused with MetricFilterInvalid, naming the role_of: to add. bloomery explain shows the facts that admitted the comparison beside the filter. A target that has no spelling for a column comparison refuses it with UnsupportedByTarget; Cube and MetricFlow both spell it.

Deliberately narrower than the query vocabulary a request may use: no like/ilike, whose escape language and case-folding portability buy nothing where the author already knows the values. A date-role dimension is refused — a metric pinned to one period is a constant, and the time relation belongs in cumulative: or an offset:.

The name is pattern-constrained where a member name elsewhere is not, and that is load-bearing rather than tidy: this is the one place a member name reaches a template that does not quote it — {{ Dimension('<entity>__<name>') }} on MetricFlow, {CUBE}.<name> on Cube. Every other field name reaches SQL through SQLGlot, which quotes it.

SemiAdditivePolicy¶

MartSet (marts_version)¶

At most one per project; a project without marts compiles silver only.

Mart¶

Rollup¶

A mart at a coarser grain than one above it: SELECT keep…, agg(measure) … FROM <parent> GROUP BY keep…, built once instead of computed per query. It names what it keeps; what it drops is every other column of the parent, derived rather than listed, because two statements of one fact will disagree.

rollups:
  order_items_monthly:
    of: order_items
    keep: [order_customer_id, ordered_month]
    measures: [gross_revenue]

There is no grain, no base and no cost_hint. The first two name an entity and a rollup's rows are identified by keep; the third breaks ties between marts that can serve a metric, and a rollup is never one of them.

A rollup is never chosen for you. No request is answered from it and no query is redirected to it — reading a rollup instead of the detail is a decision bloomery does not yet make, so the table is there for whatever reads it directly.

An unprovable rollup is refused, not warned about (UnprovableRollup). A rollup is read instead of the detail table, so a wrong one does not fail — it answers, quickly and plausibly. Refused: a measure that may not be re-aggregated over what the rollup drops (distinct_count, semi_additive, non_additive, snapshot), one the parent does not store, one whose filter: or cumulative: the single aggregate would drop, and a rollup that keeps every column and so drops nothing. A ratio measure is carried as its operands and recomputed, which is what it is on every other target too.

Flatten steps — two forms¶

Discriminated on via vs date:

flatten:
  - {via: item_of_order, prefix: order_}    # flatten a relationship
  - {via: order_of_customer, prefix: customer_, as_of: order_date}  # ... as of a date
  - {via: order_of_customer, prefix: customer_now_, reading: current}  # ... or as it is now
  - {date: order_date, role: ordered}       # declare a date role

ExposureSet (exposures_version)¶

At most one per project, and optional. An exposure is something outside the project that reads it — a dashboard, a notebook, a reverse-ETL sync — declared so the lineage graph has a sink and plan() can name the consumers a change reaches. Nothing is built for one, and nothing about one is discovered: see Declare what reads your project.

Exposure¶

DependsOn¶

At least one entry across the two: an exposure that names nothing answers neither "what does this read" nor "who does this change reach", while still reporting clean. A name repeated within a list is refused for the same reason — it would build one graph edge twice and report one consumer twice against a single change.

Shape versus resolution. Parse checks the grammar above. Whether a named metric or mart exists is a guardrail question, and a name that resolves to nothing is refused there (DanglingExposure).

exposures_version: 1
exposures:
  weekly_revenue_review:
    kind: dashboard
    owner: analytics@example.com
    url: https://bi.example.com/dash/17
    depends_on:
      metrics: [gross_revenue, order_count]
      marts: [order_items]

StepSet (steps_version)¶

At most one per project, and optional — most projects wire no steps. A step is a referenced implementation: the document names platform-owned code by ref@version and binds it into the pipeline. It cannot carry a body and cannot name a file to load, which is what keeps an authored spec from becoming an arbitrary-code-execution surface — the property comes from the absence of a field, not from validating one.

The manifest describing what a step is — its tier, its declared inputs, outputs and parameter bounds — is not a spec document. It reaches the compiler as a StepRegistry passed to compile_project(..., steps=…), so bloomery reads no step files at all. See Steps: referenced implementations for the manifest side and the three tiers.

StepWiring¶

A bound relation is an optionally-namespaced identifier and nothing else (^[A-Za-z_][A-Za-z0-9_]*(?:\.[A-Za-z_][A-Za-z0-9_]*)*$) — these strings reach generated source, so the grammar is the second of two locks.

steps_version: 1
steps:
  - use: resolve_customers@2
    inputs: {shopify: bronze.shopify_customers, stripe: bronze.stripe_customers}
    outputs: {customer: silver.customer, links: silver.customer_link}
    parameters: {threshold: 0.92}
    canonical:
      customer: {customer_id: customer_id}
    quality:
      - {name: confidence_is_sane, rule: expression, expr: "confidence BETWEEN 0 AND 1", on_fail: fail}
    applies_to: {confidence_is_sane: links}

Shape versus resolution. Parse checks that canonical and applies_to name outputs this wiring binds and that every declared rule says which output it applies to. Whether the manifest actually declares that output, that column or that parameter is a resolution question — the spec layer has never seen a manifest.

RetrievalSpec (retrieval_version)¶

At most one per project, and optional. A retrieval document declares the logical retrieval surface over relations the project already builds: which relation is a corpus, which column holds its vectors, what space those vectors live in, and what a query may filter on and get back. Nothing is built for one — the artifact is retrieval_manifest.json, emitted by the retrieval target — and nothing about one is discovered. See Retrieval for what this is and is not.

No float appears anywhere in the grammar, and no embedding ever reaches the compiler. A space declares a shape (dimensions, scalar, distance) and an identity (which encoder writes the corpus, which reads the query) — the identity is opaque strings compared to strings, never resolved against a provider. Fusion is rank-based, so there is no weight to tune.

SemanticSpace¶

A space is what makes two vectors comparable. Two corpora in one space are searchable together; two corpora that merely have the same dimensions are not.

Both sides are declared because they are two different calls, and a corpus written by one model and queried by another is a silently different space. An encoder under query_encoder declaring input_kind: document is refused at parse: the document would read as symmetric while saying it is not.

Encoder¶

RetrievalProfile¶

CorpusRelation¶

Two keys rather than one dotted string, because the two namespaces are separate: a name that resolves in both would otherwise be ambiguous, and neither prefix is a namespace a runtime can use.

VectorSide¶

LexicalSide¶

Analyser¶

An analyser tokenises a query the way it tokenised the corpus, so there is no input_kind: two strings, and nothing else.

Fusion¶

A method name is not a ranking: two consumers honouring rrf with a different constant, or a different candidate depth, return different orders for one query. The manifest carries the effective pair — the defaults written out where the document did not name them — so a consumer that supplies its own is the one that diverged.

retrieval_version: 1
semantic_spaces:
  chunk_text:
    dimensions: 1536
    scalar: float32
    distance: cosine
    document_encoder: {family: openai, model: text-embedding-3-small, input_kind: document}
    query_encoder: {family: openai, model: text-embedding-3-small, input_kind: query}
profiles:
  chunk_hybrid:
    relation: {entity: chunk, analyser: {family: lucene, model: english}}
    grain: [chunk_id]
    vector:
      field: embedding
      space: chunk_text
      producer: {family: openai, model: text-embedding-3-small, input_kind: document}
    lexical: {fields: [body], analyser: {family: lucene, model: english}}
    fusion: {method: rrf, k: 60, depth: 100}
    filterable: [document_id]
    return: [chunk_id, document_id, body]

Where a vector column comes from. A vector(...) column is assignable only from an identical vector and is produced by no transform, so no mapping chain can populate one: it arrives from a step's declared output (produces: {embedding: {type: "vector(float32, 1536)"}}) and nowhere else. Asking a SQL dialect for such a relation is refused — UnsupportedByTarget, "no physical type for vector(float32, 1536)" — because a declared vector is not a column a warehouse port emits DDL for. The relation is written by the platform's step; what bloomery contributes is the contract for querying it.

Shape versus resolution. Parse checks the grammar above, that every profile names a space this document declares, and that lexical/fusion are both present or both absent. Whether the corpus relation exists, whether its vector column is a vector of the right length and scalar, whether the grain is the relation's key, whether the lexical side's analyser is the one the corpus declares and whether every named column is there are guardrail questions — see Retrieval refusals.