Search

Full-text and vector search over a SearchSpec. It splits into a query port (run ranked searches) and a command port (maintain the index); the conceptual surface is Reading data → Searching.

q = ctx.search.query(spec)    # search
c = ctx.search.command(spec)  # index maintenance

ctx.search also exposes .hub(spec) and .federated(spec) (a query port over composed indexes, declared with HubSearchSpec / FederatedSearchSpec) and .snapshot(spec) for result-set snapshots.

Spec¶

SearchSpec[M] — the searchable model, its indexed fields, and ranking/encryption policy:

HubSearchSpec carries the same read_conformity / lenient_read_fields / materialized over its hub-row model (a hub has no index fields of its own), and takes extra_aggregatable_fields: hub-relation columns outside that model an aggregate may group by or measure (see Aggregating matches). FederatedSearchSpec inherits these from each member spec. A federated search merges thin by default (thin_merge=True): each member returns its ids (and the fields of any requested sort), and just the page is re-read from its member. thin_merge=False fuses full hits in memory instead.

materialized is for filtering and sorting search results by a derived value — the column must already exist (typically written by the document side over the same table). Returning a computed field needs no materialized: it recomputes from the row on decode.

Query port¶

Same search / project_search / select_search flavors and _page / _cursor containers as the document query port. The query text is the first argument; everything else mirrors the document side:

query is a string (or a sequence of strings); filters and sorts use the query DSL. A page from a single index is ordered by relevance, then by sorts (or the spec's default_sort when the request has none), then by id, in the sort's direction when it has a single one. A sort that names id ends there. A blank query has no relevance, so the sort alone orders it, and the id gives rows that tie on every other key one place, so offset pages neither repeat nor skip them. Meilisearch is the exception for a query with search text: it sorts by the request's sorts only, and leaves relevance ties in the engine's order. options: SearchOptions is the backend- and topology-agnostic per-request surface — relevance weights, fuzzy matching, a per-request fields narrowing, multi-term combination (phrase_combine: "any" / "all"), the count policy (search_count), an advisory candidate cap (max_candidates), the facet / highlight requests below (facets, facet_size, highlight), and a precomputed query_vector. A query_vector ranks a vector index (pgvector, Atlas Vector Search) in place of embedding the query. It must come from the same embedding model as the indexed documents, its length must match the index's dimensions, and query must be blank; None reads as no vector. Every other backend, and hub and federated search, refuse it with query_feature_unsupported. Hub and federated searches resolve to a MultiSourceSearchOptions port that also carries member selection (member_weights / members), a post-merge cap (merge_candidates), and the fusion strategy ("rrf" / "weighted"); passing those keys to a single-index query(...) port is a type error. A federated search also takes member_filters: a filter per member name, joined with $and to filters on that member's leg alone. A hub refuses it, since a hub filter applies to the hub rows. A key neither type defines is refused as a precondition naming it, at a request and at the port alike.

An authorization hook on a hub or federated search operation authorizes the operation, not each member it reads. Attach AuthzSearchMembers to check every member against its own read action. It binds a SearchMemberGrant, which every hub and federated adapter applies where it reads its options, after any request mapper. The registry refuses to freeze such an operation without the hook (see Enforce on operations).

A sort on a field the read model lacks is refused as the caller's error, a key after id included. On Postgres, a ranked page reads a capped pool of candidates (max_candidates, or the route's candidate_limit; on a hub, its merge cap), always widened to cover the page plus a margin of 50 rows (offset + limit + 50). The pool is the first rows of the page order itself, so a capped offset page holds the rows the uncapped cursor puts there; a cursor walk is uncapped, except on a vector index, whose cap bounds it too. Where FTS or a PGroonga filter_first plan applies its filters before the cap, the pool is cut to exactly offset + limit rows (the spec's max_results stands in for a missing limit), since the page cannot reach further. That does not happen when the page writes a snapshot or asks for an approximate total, which counts the pool. A vector search keeps its cap as its top-k bound.

Page totals¶

On Postgres, search_count (or the spec's default_search_count) picks how a _page call counts its total:

The planner can't estimate a text match: it gives a PGroonga predicate the same estimate whatever the query. A hub's cap is its merged-rows cap (combo_limit, or per_leg_limit by default), and each leg brings at most per_leg_limit rows, so a leg cut there makes the approximate total a lower bound. The cap widens with a deeper page, so that page's approximate total can be larger, never past the exact one. A hub usually counts it in the page's own query, so it costs about what none does; exact runs the whole match again.

Streaming exports¶

search_stream (and the project_ / select_ variants) iterate the whole matching set in bounded-memory keyset chunks — peak memory is one chunk, there is no total count. Use it to export a ranked result set without loading it all at once:

async for chunk in ctx.search.query(spec).search_stream("annual report", chunk_size=1000):
    await write_rows(chunk)

Streaming is capability-gated (spec's adapter must advertise SearchCapabilities.supports_stream): Postgres FTS / PGroonga and Mongo text / Atlas stream; Meilisearch (offset-only), the top-k vector engines, and hub search (each leg is capped at per_leg_limit, so a full walk isn't guaranteed) refuse rather than truncate. Pick the right export tool for the shape:

• Ranked live export → search_stream. A concurrent write may shift a hit between chunks.
• Filter-only export (no query terms) → the document port's find_stream — it's a plain keyset read with no ranking overhead (server-side cursor on Postgres).
• Stable / point-in-time / Meilisearch / very deep → a result snapshot (build the ordered-id pool once, page it by id); works where a live cursor cannot.

Aggregating matches¶

aggregate_search groups and measures the rows a search matches, with the same AggregatesExpression ($groups, $computed, $having) the document port's aggregate_many takes over the rows a filter matches:

page = await ctx.search.query(spec).aggregate_search_page(
    {"$groups": {"status": "status"}, "$computed": {"n": {"$count": None}, "total": {"$sum": "amount"}}},
    "overdue invoice",
)

It reads the rows an exact page total counts: every match, never a candidate cap, or for a blank query the rows the page counts by its filters alone. sorts name output aliases, and the group keys order the groups after them, so pages of groups neither repeat nor skip one; without a limit every group comes back, as aggregate_many returns them. aggregate_search_page adds the number of groups, after $having. Groups, measures and per-metric filters may name only the spec's aggregatable_fields — stored, not field-encrypted — and the facets and highlight options are refused, as they describe hits, as are max_candidates and a hub's merge_candidates, which would cut the set.

A hub aggregates its merged rows with every leg uncapped: per_leg_limit and the merge cap bound a page, not an aggregate, so the groups add up to the exact page total. Its measures often live in columns of the hub relation that its read model leaves out, such as a view's computed totals; name them in HubSearchSpec.extra_aggregatable_fields and an aggregate may group by, measure and metric-filter on them too. Postgres checks they exist on the first aggregate. The search's own filters stay on the read model. A query that reaches a vector leg is refused, since an uncapped vector leg matches every embedded row.

An aggregate names an extra column by its own value: a dotted path into one, in a group, a measure or a per-metric filter, is refused, as is a filter's element quantifier over one. Postgres checks a metric filter's operators against the column's type, as it checks a search filter against the read model, so $null, ranges and $in work on a numeric column, and $like or $empty on one is refused. A declared precision (numeric(12,2)) keeps the type, and a column of a type with no value rule (jsonb, time, …) takes only $null. The in-memory hub types an extra column by the values its matched rows hold, so a string range bound casts there as it does on Postgres:

await hub.aggregate_search(
    {
        "$computed": {
            "total": {"$sum": "upd_or_quantity"},
            "missing": {"$count": {"filter": {"$values": {"upd_or_quantity": {"$null": True}}}}},
        }
    },
    "catheter",
)

Aggregation is capability-gated (SearchCapabilities.supports_aggregates): Postgres FTS / PGroonga, hub search (Postgres and in-memory) and the in-memory adapter aggregate; vector search (every row is some embedding's neighbour), federated search, Mongo and Meilisearch refuse with query_feature_unsupported rather than measure a partial set.

Facets and highlights¶

A query requests term facet distributions with options={"facets": [...]} and per-hit match snippets with options={"highlight": True} (or a HighlightOptions mapping to narrow fields / customize the <em> markers), over the spec's facetable_fields / highlightable_fields. Results ride the page as optional page.facets (one set per query, over the full matching set) and page.highlights (per hit, index-aligned with hits), None when not requested. (Pages also carry optional per-hit relevance page.scores, index-aligned the same way; on pgvector a score is the negated distance, so higher is nearer.) A field or backend that cannot serve a request fails closed with query_feature_unsupported.

Support is per backend (single-index) and per topology (hub / federated). fail-closed raises query_feature_unsupported; — means not applicable.

• PGroonga / hub highlights are marked in process (case-insensitive substring over the raw field text), so they fold case for any script and keep the original casing; FTS uses ts_headline for language-aware stemming.
• Vector ranks by distance with no lexical match, so there is no snippet to highlight (facets still group over the ranked set).
• Hub highlights apply in both sql and parallel execution; hub facets run as a companion GROUP BY over the merged set under sql execution and fail closed under parallel (the Python merge can't dedup-count the facet field).
• Federated facets are deferred — per-member distributions don't compose under the reciprocal-rank merge, so every federated search fails closed on facets. Highlights survive only the RRF-merge path (each merged hit keeps its originating leg's snippet), not Meilisearch native federation.
• Requesting facets or highlights bypasses result-snapshot replay (snapshots store hit ids only), so a sidecar request always runs against a live query.

Command port¶

Data-plane document writes — ctx.search.command(spec).

Management port¶

Control-plane index provisioning — ctx.search.management(spec). Kept separate from the command port (provisioning mutates shared topology / wipes are destructive admin, run outside the request path); acquired via the command path, so a read-only operation cannot provision or wipe an index.

Both are postconditions rather than operations on an existing thing, so both are safe to repeat and safe to call first: delete_all() on a surface nothing has created yet succeeds, because an index that does not exist already holds no documents.

Rebuilding an index¶

An external index is kept in step by the index-sync bindings (AggregateKit(search=…), or bind_search_sync / bind_search_sync_outbox directly), and those are incremental: they carry a row into the index when that row is written. Nothing carries a row that was never written since the index existed. So an aggregate that gained search=… after it already held rows, an index provisioned fresh, one restored onto new infrastructure, and one that drifted while its sync was broken all end up the same way — correct-looking, and empty or stale for every untouched row.

rebuild_search_index is the backfill. It streams the document plane and applies each row to the index under the same rule the incremental syncs apply, so a rebuilt index holds exactly what an unbroken sync would have produced:

from forze_kits.integrations.search import rebuild_search_index

report = await rebuild_search_index(
    ctx.doc.query(CUSTOMER_SPEC),
    ctx.search.command(CUSTOMER_INDEX),
    document=CUSTOMER_SPEC,
    search=CUSTOMER_INDEX,
)
report.indexed, report.removed, report.scanned   # live rows, soft-deleted rows, total

An AggregateKit already holds both specs, so it exposes the same sweep as await kit.rebuild_search(ctx).

Keyset-paged, so memory is bounded by chunk_size whatever the collection's size; idempotent, so an interrupted sweep is re-run rather than repaired. A soft-deleted row is removed from the index rather than upserted — a sweep that merely upserted everything it read would resurrect every soft-deleted row as a hit that GET then 404s. Run it once per tenant (under bind_identity(tenant=…)) on a tenant-aware route.

Implemented by¶

An external index seals encrypted fields in the index document; in-place search decrypts out of the document table's results — which is why the encryption policy must be shared with the DocumentSpec.