Shutdown & fleets

Processes get told to stop — deploys, autoscaling, a node going away. And in production there is rarely one process: the same app runs as N replicas behind a load balancer. The runtime treats both as first-class: shutdown drains instead of dropping work, and a declared fleet posture catches the mistakes that only N replicas can make.

Graceful drain¶

ExecutionRuntime.shutdown() — and therefore scope() exit and runtime_lifespan — does not tear infrastructure down under in-flight work. It drains first:

1. The scope stops admitting new top-level invocations. They fail with a retryable throttled (code="draining") — a 429 at the FastAPI edge, a requeue-worthy nack for a queue consumer.
2. In-flight operations get a bounded window — drain_timeout, default 10 seconds — to finish before lifecycle teardown closes the clients they depend on.
3. Lifecycle shutdown runs as usual, in reverse wave order.

runtime = build_runtime(..., drain_timeout=timedelta(seconds=20))

An operation that is already running keeps all of its machinery: nested dispatch deliberately rides the outer invocation's slot, so draining never starves an admitted operation of its own dispatch chains. Zero in-flight work exits immediately; a window that expires logs the leftover count and proceeds — shutdown is never blocked indefinitely.

Readiness¶

The load balancer should stop routing before the drain window starts. The runtime exposes its state — runtime.ready and runtime.draining — and the FastAPI integration turns it into a probe:

from forze_fastapi.routes import attach_readiness_route

attach_readiness_route(router, runtime)   # GET /readyz

200 while a scope is active and not draining; 503 draining once shutdown flips the gate, 503 unavailable before the scope exists. Point your orchestrator's readiness check here and the rollout sequence takes care of itself: routing stops, in-flight work drains, teardown runs.

Point liveness somewhere else — at attach_liveness_route(router), which answers 200 unconditionally. A draining pod is alive but not ready; one endpoint answering both questions gets the pod killed mid-drain instead of letting it finish.

Processes that are not HTTP servers — the outbox relay, inbox and commit-stream consumers, the socket.io gateway, durable runners — get both probes from one lifecycle step:

from forze.application.execution.background import probe_listener_step

lifecycle_steps.append(probe_listener_step(runtime, port=8079))

Stdlib asyncio, no HTTP framework, same runtime state, same drain semantics: it reports draining for the whole window and closes only after. See The Grafana stack for the Kubernetes mapping.

Are the dependencies reachable?¶

The drain gate answers whether the process is willing to take traffic. It says nothing about whether it can serve one: an un-drained process whose database is gone answers ready, the load balancer keeps routing, and every request 500s.

Name the clients you cannot serve without and each is resolved from the deps container and asked health() — the signature every forze client port already declares:

from datetime import timedelta

from forze_postgres import PostgresClientDepKey
from forze_redis import RedisClientDepKey

attach_readiness_route(
    router,
    runtime,
    probes={"postgres": PostgresClientDepKey, "redis": RedisClientDepKey},
    probe_timeout=timedelta(seconds=2),
)

The response then carries a per-dependency breakdown, and any failed check makes the whole probe a 503:

{"status": "degraded",
 "checks": {"postgres": {"ok": true,  "detail": "ok"},
            "redis":    {"ok": false, "detail": "timed out after 2.0s"}}}

probe_timeout is the budget for each probe, not for the sweep — deliberately. The two look equivalent, since the probes run concurrently and the wall clock is the same either way. They are not: when a dependency is unreachable its driver retries past the deadline, and a sweep-wide timeout then cancels every probe and answers 503 with checks: {} — no breakdown, in exactly the situation the breakdown exists for. Per probe, a hanging dependency is one failed check with a name on it.

A probe reports failures and never becomes one. A client whose health() raises is reported as {"ok": false, "detail": "<ExceptionType>"} — the type and not the message, because a driver's connection error routinely carries the DSN and this body is served to anything that can reach the probe. Nothing is auto-discovered: the integrations satisfy health() structurally, with no base port to enumerate, and a sweep of every registered dependency would probe things readiness does not depend on. A probe_timeout that is not positive is refused at attach time.

Keep the endpoint internal, as you would /metrics. detail is the client's own summary, and a failing driver's is its exception text — which routinely names the host, the port and the user it failed to authenticate as. That is exactly what the breakdown is for and exactly what should not reach the public internet. Every request also runs every probe, so the probe period is a real query rate against each dependency.

No probes (the default) leaves the drain-gate-only behaviour untouched.

Declare the fleet posture¶

Some startup work is safe in one process and a stampede in twenty — N replicas all running CREATE INDEX or a data migration at the same moment. Declaring the posture makes that a composition-time error instead of a 3 a.m. incident:

from forze.application.execution import DeploymentProfile

runtime = build_runtime(..., deployment=DeploymentProfile.FLEET)

Under FLEET, building the runtime fails for any lifecycle step marked mutates_shared_state=True that is not also singleton_guarded. The markers are declared by the step author — mutation can't be detected structurally, so the validation is honest-by-declaration: mark the steps that touch shared backends, and the profile enforces that each one is guarded.

Singleton lifecycle steps¶

The guard itself ships in forze_kits: wrap a step in a distributed lock so one replica runs it and the rest skip —

from forze.application.contracts.dlock import DistributedLockSpec
from forze_kits.lifecycle import singleton_lifecycle_step

step = singleton_lifecycle_step(
    ensure_indexes_step,
    spec=DistributedLockSpec(name="ensure-indexes"),  # resolved from the scope
    owner=instance_id,
)

You pass the lock spec, not a live port: the guard resolves the command port from the execution context (ctx.dlock.command(spec)) at startup, so it slots into a lifecycle plan that's assembled before any scope exists.

The first replica to acquire the lock runs the startup and releases it; replicas that find the lock held skip — the holder is doing the work. Shutdown later runs only on the replica whose startup actually executed. The step must be idempotent ("ensure"-style): a replica that starts after the holder released will acquire and run it again. Size the lock's TTL to comfortably exceed the step's duration — no heartbeat extends it here.

What's shared, what's per-process¶

Most coordination state already lives in your backends (idempotency records, distributed locks, the outbox). The resilience layer's state is process-local by default, and each piece has a deliberate fleet answer:

Wiring for the shared stores is on the resilience page. The framework's periodic loops — the outbox relay tick, consumer crash-restart backoff — are jittered out of the box, so N replicas don't synchronize into a thundering herd against the same claim query.

Knowing a fleet drains cleanly — or spots one thundering against a claim query — is something you confirm by watching it; that's Observability.