Skip to content

Deadlines

An operation without a time budget can outlive the caller that wanted it: the client gave up seconds ago, but the handler is still holding a connection, a bulkhead slot, and a transaction. Forze uses gRPC-style deadlines: a budget is bound once at operation entry, and everything downstream — hooks, the transaction, dispatched operations, resilience strategies, outbound calls — inherits it for free.

The tightest of the plan and caller budgets is bound at operation entry, then inherited by hooks, the transaction, the resilience chain, and outbound HTTP calls The tightest of the plan and caller budgets is bound at operation entry, then inherited by hooks, the transaction, the resilience chain, and outbound HTTP calls

Declare the budget on the plan

A deadline is a property of the operation, not of the route or the caller. Declare it where the operation is composed:

registry = (
    registry
    .bind("orders.create")
    .with_deadline(timedelta(seconds=5))
    .finish()
    .freeze()
)

Patch mode sets a default across many operations at once — registry.patch(selector).with_deadline(timedelta(seconds=10)).finish() gives every matched operation a budget in one line. Scope it to a namespace with patch(selector, namespace=ns), or settle it into per-operation plans with registry.materialize_patches() so it can't leak to a sibling registry on merge — see Cross-cutting patches.

Merging is restrictive: across patches, explicit plans, and any caller-bound deadline, the tightest budget wins. A layer can shorten an operation's budget, never extend it — to give one operation more room than a broad patch default, narrow the patch selector instead.

Bind a caller budget at the boundary

A boundary (HTTP middleware, a worker loop) may additionally bind the caller's own budget:

from forze.application.execution import bind_deadline

with bind_deadline(timeout_s):   # None is a no-op passthrough
    result = await resolved_op(args)

Binding is task-scoped and tighten-only — a nested bind can shorten the budget but never extend past the enclosing deadline. bind_deadline(None) passes through, so an optional per-request timeout forwards without branching. Ports that want to derive per-call budgets can read remaining_time() — seconds left, or None when no deadline is bound.

What enforcement looks like

  • Entry fail-fast — an operation invoked with its budget already spent fails immediately, before any hook runs.
  • The whole plan is bounded — the budget covers hooks, the transaction, and dispatch chains, and propagates into dispatched operations.
  • Resilience cooperates — the resilience executor gates its entire strategy chain from the outside, and a retry abandons a backoff sleep that would outlive the deadline, surfacing the real error instead.

Expiry raises exc.timeout (code="deadline_exceeded") — 504 at the FastAPI edge, details suppressed. The kind is deliberately non-retryable: within the same invocation the budget is spent, and a fresh invocation carries a fresh deadline.

Deadline ≠ per-attempt timeout

The resilience TimeoutStrategy bounds a single attempt and keeps raising retryable infrastructure, so a retry can take another shot. The invocation deadline bounds the whole call and is final. They compose: per-attempt timeouts inside, the deadline outside.

When no deadline is declared or bound, nothing changes — the unbound path is one ContextVar read, with no timeout machinery.

The budget is visible

A declared deadline projects through the operation catalog like every other plan-derived fact: OperationCatalogEntry.deadline, an x-deadline-seconds vendor extension plus a "Time budget" line on generated FastAPI routes, and a sentence in each MCP tool description — so clients and agents can set their own timeouts instead of retrying a call that died of budget exhaustion.

Budgets cross service boundaries

When one Forze service calls another, the outbound HTTP adapter attaches the caller's remaining budget as an X-Forze-Deadline-Budget header automatically (opt out per service with HttpServiceConfig(propagate_deadline=False)). The receiving side honors it only when asked:

app.add_middleware(
    InvocationMetadataMiddleware,
    ctx_dep=runtime.get_context,
    bind_deadline_from_header=True,
)

The header carries a duration, not an instant, so clock skew between hosts doesn't matter; and because binding is tighten-only, a forged or stale value can only shorten the sender's own request — never extend it.

The budget is deliberately not a message-envelope header: deadlines belong to the synchronous call chain. A queued event consumed after a backlog must not inherit its producer's leftover budget.

Cancellation never skips committed work

A deadline firing — or a client disconnecting — cancels the operation's task. One window must survive that: after the root transaction commits, the deferred after-commit work (idempotency records, event dispatch) is a cancellation-protected critical section. It runs to completion even while cancellation is pending, and the cancellation re-raises afterwards — so a committed transaction is never left half-announced.

A late cancel is never reported as safe

A deadline or a cancel that lands at or after the commit point leaves the outcome unknown. The commit may have landed, so a retry could apply the operation twice. What happens depends on where it lands:

  • In the commit or the after-commit work, a deadline or a cancel raises exc.internal with code="commit_ambiguous" instead of deadline_exceeded or a bare cancellation. The kind is non-retryable.
  • A deadline after the transaction has closed — in an outer hook or a dispatched operation — raises the same error.
  • A cancel after the transaction has closed stays a cancellation, because server shutdown and task groups rely on it propagating. The operation logs a warning with code="commit_ambiguous" and operation, carrying the invocation's ids, and adds a forze.commit_ambiguous event to the active span.

What each surface does with that late cancel:

Surface What the caller sees What records it
forze ops call a commit_ambiguous error, exit 1 the error document and the warning
HTTP routes, MCP tools, Socket.IO and WebSocket commands nothing: the client went away or the server is stopping the warning and the span event
Durable runs, Inngest functions, your own Temporal activities the runner's cancel outcome; recovery or a retry may run it again the warning and the span event
Saga steps saga.step_ambiguous, and nothing is compensated the saga's error or its journal row

Reconcile from the warning before running the operation again. Where a client retries, an idempotency key makes the retry safe.

A query's read-only transaction is a commit point only where the backend could have let a write through it:

Transaction manager A query interrupted after its read reports
Postgres, and the mock's journal and strict managers, when the query's read-only transaction is the only one it entered deadline_exceeded (CLI exit 6), or a plain cancellation (CLI exit 130 or 143): the store refused writes inside it
The same managers, when the query's transaction is nested in one opened outside forze commit_ambiguous: it became a savepoint, which cannot be read-only
The same managers, when a nested scope on another route finished its body commit_ambiguous: that route opened, and may have committed, a transaction of its own
The same managers, when the query queued after-commit work (run_or_defer) commit_ambiguous: that work runs after the transaction closes and may write
Postgres, after a detached() statement, another client's bound_connection(), or another client's transaction or statement outside a transaction commit_ambiguous: those commit outside the read-only transaction, so even an autocommit read gives up the skip
Mongo, Firestore, Neo4j, and the mock's none manager commit_ambiguous: they accept read_only without enforcing it

The kernel asks the manager about the transaction it actually opened, through ReadOnlyAware.read_only_enforced(), and only while the read-only root is the only route that may have committed and no after-commit work is queued. A manager that does not implement it counts as not enforcing. Where a query reports commit_ambiguous but wrote nothing, it is still safe to run again.

A deadline is one of several controls over how an operation behaves under stress; the full set — retries, bulkheads, breakers, hedging — composes in Resilience.