Skip to content

Observability

Forze is already observability-rich inside — a runtime tracer, a transaction tracer, structured logs. One call pushes that out to OpenTelemetry: a span and metrics for every operation, tagged with identity and correlation, so your operations show up in any tracing or metrics backend.

OpenTelemetry is a core dependency — the logging layer already uses it — so this is built in. You only bring an exporter.

Instrument every operation

Wrap the registry once, before you freeze it:

from forze.application.execution import instrument_operations

registry = instrument_operations(build_orders_registry())
frozen = registry.freeze()

That's the whole integration. It emits through the global OTel providers by default; pass tracer= / meter= to target your own.

What you get

Every operation produces a span, named by its operation key and nested under whatever span is already active (an incoming HTTP request, say). A server-side failure — an internal, infrastructure, configuration or timeout error, or any exception that is not a CoreException — records the exception and sets the span status to ERROR; a failure the caller can handle, such as not-found, validation or unavailable, leaves the span clean. Either way the exception is re-raised unchanged. The span carries:

Attribute Value
forze.operation / forze.operation.kind the operation key and query / command
forze.execution_id / forze.correlation_id / forze.causation_id the invocation metadata
forze.invocation_surface the boundary the invocation entered through, or unknown
forze.tenant_id / forze.principal_id the bound tenant and principal

Alongside each span, two metrics — forze.operations (a counter) and forze.operation.duration (a histogram, in ms) — labelled by operation, kind, and outcome: success, failed for a failure the caller can handle, or error.

An unavailable error — something an operator switched off — counts as failed, not error, and no transport logs it as a server error, so an error-rate alert on forze.outcome="error" leaves it out. The operation metrics carry no error code, so if switch-offs matter to you, track them at the edge: count responses whose X-Error-Code is the code your app raises with exc.unavailable, or 503s at your proxy.

See which boundary an invocation came through

Every boundary that starts an invocation names itself on the invocation metadata, so a log line or span says whether the work arrived over HTTP, from forze ops, or from a recovered durable run. Each line logged inside an operation carries the invocation_surface field, and the operation span carries forze.invocation_surface. An operation run outside any boundary, such as a test harness calling run_operation, records unknown.

Value Boundary
http forze_fastapi's InvocationMetadataMiddleware: generated routes and SSE
websocket a forze_fastapi realtime WebSocket command frame
socketio a forze_socketio command event
mcp a forze_mcp tool call
cli forze ops call
inbox process_with_inbox, once per delivery
temporal a workflow or activity behind forze_temporal's interceptors
inngest a forze_inngest function call
durable one attempt of a self-hosted durable run
unknown no boundary bound metadata

The values are InvocationSurface members in forze.application.execution. A nested call keeps the surface of the invocation around it: an agent tool call made while serving an HTTP request records http. A hop names itself and carries nothing: a Temporal activity started from an HTTP request records temporal, and the request is found through the shared correlation_id.

The field is a log field, not a metric label and not a Loki label, so it adds no series. The key is invocation_surface and not surface because surface is the port-call key that with_port_logging() writes, and both appear on one port-call line.

Your own boundary. A gRPC server or a queue worker of your own passes its own token: mint_invocation_metadata(surface="grpc"), from forze.application.execution.ingress, or InvocationMetadata(..., surface="grpc"). A token must match ^[a-z][a-z0-9_]{0,31}$, or construction raises configuration with code invocation_surface_invalid. Pass a literal from your boundary's source:

  • never a value read from a header, a frame, a message or an event, which a caller would then choose;
  • never a token that names a person, a tenant or a customer, since it reaches every log line and span.

Attribution, not authority

No framework hook, policy or guard reads the surface, and your code should not either. Any code in the process can build metadata with any valid token, so a check such as "only if the surface is cli" proves nothing. Restrict which operations a surface may call with its exposure allowlist instead.

Trace every outbound call

The operation span tells you that an operation ran and how long it took; to see where the time went — which database read, which cache call, which outbound request — span the ports too. In a hexagonal app every external call is a port, so opting the resolved ports into a span turns that seam into a complete outbound-I/O trace under each operation. It rides on the dependency registry, alongside instrument_operations on the operation registry:

from forze.application.execution import DepsRegistry

deps = DepsRegistry.from_modules(postgres_module, redis_module).with_otel_port_spans()

Freeze deps into the runtime as usual. Every document / cache / queue / search / HTTP call now opens a CLIENT span named surface.op (document_command.create, cache_query.get), nested under the operation span:

Attribute Value
forze.port.domain / forze.port.surface the port family and its dependency surface
forze.port.op the method called — create, get, publish
forze.port.route / forze.port.phase the specification name and phase, when present

The name stays low-cardinality (surface.op); the specification name rides as the route attribute, never in the name. The span sits inside the resilience policy, so a retried call is one span per attempt and a call shed by an open breaker or a full bulkhead emits none — the trace shows the work that actually reached the backend. Failure status follows the exception kind: an infrastructure, internal, or configuration fault reds the span, while a domain failure the caller can handle — not-found, conflict, precondition — leaves it clean, exactly as a 404 does not red an HTTP client span. Streaming methods (a cursor, a consume loop, a subscription) pass through un-spanned — a lone span around a long-lived stream is the wrong shape; per-message spans belong in the consumer.

For a log line per outbound call instead of (or alongside) a span, add deps.with_port_logging(). It logs every port call uniformly — surface, route, op, and duration_ms under forze.integrations.<domain> — at trace on success (so it costs nothing in production unless you turn trace on), debug on an expected domain failure, and warning with a traceback on an unexpected one.

Both wrap the port a context resolves, so a @runtime_checkable protocol check of it is false even when the backend implements the protocol. Ask implements(port, SomeAware), from forze.application.contracts.deps, instead: it answers for the backend behind the wrappers. Keep calling the port itself, so the span and the log line still happen.

Follow the trace across boundaries

Spans and port spans cover one operation inside one process. When the work crosses a boundary the trace would normally break — so Forze carries the W3C traceparent across the two crossings OpenTelemetry's transport instrumentation can't bridge on its own.

An event published now and consumed later keeps its trace when you set propagate_trace=True on the outbox integration: the publishing operation's trace context is captured as the event is staged, travels with it through the broker, and the consumer rebuilds it so the handler's spans link back to the publish span. On a relational backend, add a nullable traceparent column first — exactly as you do for hlc_ordering.

An outbound HTTP request carries it automatically: the HTTP adapter injects the active trace context into every request (a no-op when no span is active), honouring your application's configured propagator. Nothing to enable.

The traceparent is untrusted metadata

It arrives over the broker or the wire, so it influences trace parenting only — never identity, tenancy, or deduplication. Treat it as a hint for the tracing backend, nothing more.

Inbound HTTP runs the other way: a request arriving at FastAPI is picked up by the standard opentelemetry-instrumentation-fastapi, which creates the server span your operation span nests under — so Forze does not re-extract it.

Resilience metrics

The resilience layer makes decisions worth watching — retries, rejections, breaker trips, bulkhead backoff. instrument_resilience exports them as always-on metrics, independent of any tracing gate, so a production process with tracing off still reports them:

from forze.application.execution import instrument_resilience

instrument_resilience(ctx.resilience())  # once, when the scope is up
Metric What it carries
forze.resilience.events (counter) every event — retry attempts, timeouts, rate-limit and bulkhead rejections, breaker transitions — labelled by event, policy, and route
forze.resilience.breaker.state (gauge) breaker phase per policy/route: 0 closed, 1 half-open, 2 open
forze.resilience.bulkhead.queue_depth (gauge) calls queued behind each bulkhead, sampled at collection
forze.resilience.bulkhead.in_use (gauge) calls holding a slot in each bulkhead right now, sampled at collection
forze.resilience.bulkhead.limit (gauge) each bulkhead's concurrency limit: the fixed cap, or the current AIMD or Gradient2 limit
forze.resilience.hedge.delay (gauge) the effective adaptive hedge delay (P² quantile estimate), in seconds

Two reading notes: breaker_open counts the open transition and every admission shed while open, so its rate tracks shed load; and a breaker that never tripped reports no state at all — closed by absence.

For bulkheads, in_use / limit is occupancy for every bulkhead kind. It shows a bulkhead filling up before the queue grows, and a slot that is taken and never released shows as in_use that stays up when traffic stops.

Tenant pool metrics

Routed clients keep one connection pool per tenant in a bounded LRU, and evicting a pool is expensive — the next request rebuilds the connection from scratch. instrument_tenant_pools exports the churn counters:

from forze.application.execution import instrument_tenant_pools

instrument_tenant_pools({"postgres": pg, "redis": redis})

Per client (labelled forze.client): forze.tenancy.pool.size and ….capacity gauges, plus cumulative ….created, ….disposed, and ….evicted_explicit counters. The alert worth setting: a sustained creation rate while size sits at capacity means the LRU is thrashing — hot tenants' pools evicted by cold one-off traffic, each rebuild paying full connection establishment. The fix is usually a larger max_cached_tenants; the metric tells you when.

Connection pool metrics

Pool state is the first thing to check when an API slows under load, and a connection that is never returned is the leak a soak test exists to catch. instrument_connection_pools exports the pool of every client you name:

from forze.application.execution import instrument_connection_pools

instrument_connection_pools({"postgres": pg, "redis": redis, "mongo": mongo})
Metric What it carries
forze.connection_pool.in_use (gauge) connections checked out, or being opened or reopened by the pool
forze.connection_pool.idle (gauge) connections in the pool not checked out
forze.connection_pool.max (gauge) the configured cap, absent for a pool without one
forze.connection_pool.waiting (gauge) callers queued for a connection, Postgres only

One series per client, labelled forze.client with the name you chose. A client counts if it implements SupportsConnectionPoolStats (forze.application.contracts.pools) with an in-memory snapshot. The callback runs on every scrape, so it never does I/O. PostgresClient, RedisClient and MongoClient do, each from public driver numbers only; the metric catalog says where each one comes from. A MongoClient reports its busiest server's pool, so on a replica set max is the per-server cap. A routed client is refused, because it keeps one pool per tenant and a sum would hide the saturated one. Its connections are not exported: the tenant pool gauges above report only its pool cache. Alert on in_use pinned at max: the pool is exhausted. What callers do next depends on the pool. Postgres, MongoDB, and a Redis client with pool_timeout set queue them until they time out; a Redis client without it refuses the next checkout at once. Only Postgres exports waiting, so there waiting above zero across scrapes shows the queue itself. On Postgres, a database that is down shows the same pattern, because the pool's reconnect attempts hold in_use at max while callers wait. Pair the alert with a connectivity signal, such as the health check.

httpx exposes no pool numbers, so outbound HTTP gets a count of requests instead: instrument_http_clients from forze_http exports forze.http.client.in_flight per HttpClient: requests sent and not finished, until request has read the response or until a stream or send block exits. A count that never falls back is a block that never exits; it may or may not still hold its connection.

from forze_http import instrument_http_clients

instrument_http_clients({"payments": payments})

Document L1 metrics

The in-process L1 exports its counters the same way:

from forze.application.integrations.document import instrument_document_l1

instrument_document_l1()

Per document (labelled forze.document): forze.cache.l1.size / ….capacity gauges and cumulative ….hits / ….misses / ….evictions counters. The hit rate validates that the L1 is earning its staleness budget, and sustained evictions at full capacity with a sagging hit rate is the scan-pollution signature — the signal to switch the eviction policy to the in-box W-TinyLFU store or raise capacity.

Process metrics

A slow memory leak or CPU creep only shows in a soak if the server's own memory and CPU arrive in the same scrape as everything else. instrument_process exports them, standard library only:

from forze.application.execution import instrument_process

instrument_process()  # once per process — in every worker of a pre-fork server
Metric What it carries
process.cpu.time (counter) CPU seconds labelled cpu.mode — user / system; rate() gives the cores used
process.memory.usage (up-down counter) resident memory in bytes, from /proc/self/statm — Linux only

The names are OpenTelemetry's process semantic conventions. Prometheus sees them as process_cpu_time_seconds_total and process_memory_usage_bytes, not prometheus_client's process_cpu_seconds_total / process_resident_memory_bytes. A default PrometheusMetricReader() renders the default registry, whose own process collector reports that pair beside these. Outside Linux the standard library offers only the peak resident size, so the memory instrument is skipped rather than reporting a peak as current. Each collection costs one syscall and a small /proc read.

Every worker reports its own series, but one Prometheus scrape of a pre-fork server reaches one worker. To see them all, give each worker worker_metrics_reader() and serve MetricsSnapshots, which labels each worker service_instance_id, or push every worker to one collector, which labels it instance.

For more — open file descriptors, threads, disk and network I/O — use opentelemetry-instrumentation-system-metrics, which depends on psutil.

Realtime metrics

Each realtime plane has its own opt-in call; the catalog lists what each exports. Two cover what a soak needs to tell a stuck node from a busy one:

from forze_fastapi.realtime import instrument_realtime_hub
from forze_kits.integrations.realtime import instrument_realtime_mailbox

instrument_realtime_hub(hub)  # the hub the SSE and WebSocket routes share
instrument_realtime_mailbox()  # every mailbox built in the process, per spec

forze.realtime.hub.subscribers counts the streams on this node's live tail; a count that only rises means streams that never leave. forze.realtime.hub.dropped counts live signals lost to a full subscriber queue. forze.realtime.mailbox.replay_size is a histogram of each connect's unacked backlog: the entries past the device's acked cursor that its replay read. A tail that keeps growing means devices fall behind on acks. It is a trend, not the backlog's size: counting mailbox rows takes a store query, and no scrape callback runs one. Only a replay that finishes records; one the client leaves or the store fails does not.

Configure logging in one call

bootstrap_logging wires the whole logging surface: the framework's own loggers (forze.*, forze_kits.*, and the forze.integrations.* adapters), any integration loggers you name, third-party stdlib loggers routed through the same formatter, and the uncaught-exception hook.

from forze import bootstrap_logging

bootstrap_logging(
    level="info",
    render_mode="json",  # "console" for local dev
    logger_names=["forze_postgres", "forze_redis"],
    third_party=["uvicorn", "uvicorn.access", "sqlalchemy.engine"],
)

Name a package root and the stdlib logger hierarchy carries the setting to every logger under it (forze_postgres.kernel, forze_postgres.adapters, …), so there is nothing per-subpackage to enumerate.

Naming loggers narrows capture, not redaction: a logger you did not list still reaches a scrubbing handler on the root at warning and above (or your level, if higher), and prints scrubbed on the configured stream where Python's last-resort handler printed it raw on stderr. With level above warning, forze's handler stops printing unlisted warnings. A root handler you installed yourself stays, so each unlisted line then appears twice; an app that manages the root itself passes redact_unnamed=False, which declines the root handler with a warning and leaves unlisted loggers to Python's default handling, unscrubbed. A later logging.basicConfig() does nothing, because the root already has a handler: call it with force=True, or before bootstrap_logging. Uvicorn sets propagate=False on uvicorn and on uvicorn.access, so neither reaches the root: name both in third_party.

Until structlog is configured (configure_logging, bootstrap_logging or your own structlog.configure), forze does not print to stdout itself: it hands each event to the stdlib logger of the same name at that logger's level, so warnings and above reach stderr by default, or your own logging handlers at the level you set.

configure_logging remains the lower-level entry point when you want to wire the pieces yourself. Either way, pass otel_config=... and the active span's trace_id and span_id are injected into every log line — because instrument_operations starts that span, your structured logs line up with the operation trace automatically, no extra wiring.

Log levels and the verbosity budget

The framework holds one rule: at level="info" it emits almost nothing per request. A quiet default is a feature — you turn detail up when investigating, rather than filtering noise out in steady state. Each level has a fixed meaning:

Level What the framework logs here Steady-state volume
trace per-row / per-message / per-port-call detail none in production — the trace gate is one integer compare unless you configure level="trace"
debug per-operation internals, cache hits, dedup skips opt-in — at a higher level, a debug call returns after one cheap check
info lifecycle events only: startup, shutdown, saga pivot, relay batch summaries rare
warning degraded-but-continuing: retries exhausted, breaker open, a callback failed rare, deduped
error an unhandled server-side fault (a bug) should be ~zero
critical data loss or an unrecoverable condition ~zero

A domain failure is never an error. A validation, not-found, conflict, or precondition outcome is the application working as designed — it logs at debug or not at all, exactly as it leaves an HTTP span clean. Only an unhandled fault reaches error.

Tame high-volume logs

Two per-event controls collapse the events that would otherwise flood a log, without dropping the first occurrence you actually need to see. They are a no-op for events that don't opt in:

# keep 1 in 100 of a uniformly high-volume event
logger.debug("cache miss", _sample=100)

# emit a flapping condition at most once per window (default 60s)
logger.warning("upstream degraded", _dedup_key="upstream-degraded")

_sample=N keeps one in every N events sharing the same logger and message; _dedup_key emits at most one event per key per window (_dedup_window= overrides the seconds). The control keys are stripped before rendering. This is on by default (configure_logging(enable_sampling=True)).

For per-request access logs — the largest steady-state source — the FastAPI and MCP middlewares are quiet by default: successful requests are sampled 1-in-N and error responses are always logged. The FastAPI middleware additionally drops health and readiness probe paths (DEFAULT_HEALTH_PATHS); MCP messages have no such path, so its default sampler applies no path exclusion. Configure either with access_log=AccessLogSampler(...): mode="full" logs every request, mode="off" disables them, and sample_rate / exclude tune the rate and the excluded subjects (request paths for FastAPI, method names for MCP).

Sensitive data is scrubbed

Log output runs through a redaction pass that masks sensitive keys (password, token, secret, api-key, cookie, authorization, …) and secret-shaped values (bearer tokens, JWTs, connection DSNs) in both extras and the message text. Extend it for deployment-specific patterns once at startup:

from forze.base.scrubbing.policy import register_sensitive_patterns

register_sensitive_patterns(keys=["x_internal_token"])

Disable with configure_logging(sanitize_logs=False) only when you fully control the sink and its retention.

Naming: where a log line comes from

Every logger is namespaced so you can raise or lower detail per area without touching the rest. Core framework logs sit under forze.* (forze.application, forze.domain); pre-built wiring under forze_kits.*; each integration under its own forze_<name>.* (forze_postgres.adapters, forze_redis.kernel). Generic adapter machinery shared across integrations logs under forze.integrations.<domain> (forze.integrations.cache, forze.integrations.document) — filter the whole group with forze.integrations.*, or a single domain on its own.

One-call SDK setup

Everything above emits through the global OTel providers. Somebody still has to install those providers and point them at a collector — and that is the fifteen lines every deployment hand-writes and gets wrong in the same four ways. bootstrap_telemetry is the bootstrap_logging twin that owns them:

from forze.base.logging import bootstrap_logging
from forze.base.telemetry import bootstrap_telemetry

bootstrap_logging(render_mode="json", otel_config={"enable": True})

telemetry = bootstrap_telemetry(
    service_name="orders-api",
    service_version=APP_VERSION,
)

# ... assemble: instrument_operations / with_otel_port_spans / instrument_* ...

await telemetry.shutdown()   # flush is part of drain, not an afterthought

Endpoint, headers and the head sampler come from the standard OTEL_EXPORTER_OTLP_* / OTEL_TRACES_SAMPLER environment variables — no parallel vocabulary. Install the exporter with the observability extra (uv add "forze[observability]"); it is OTLP http/protobuf, so no grpc.

Four things it gets right that hand-written setup usually does not:

Why it matters
a fresh service.instance.id per process without it, every worker's cumulative counters collide into one series and rate() flaps
millisecond histogram views the SDK's default buckets are second-oriented; Forze records ms
shutdown() wired into drain otherwise the last export interval of every deploy dies with the pod
defers to an existing provider an application that owns its SDK keeps it, untouched

Call it before assembly. The API's proxies do late-bind instruments created earlier (pinned by test, not assumed), but ordering it this way stops the first spans of a slow startup being dropped.

Bringing your own SDK is still fine and still supported — bootstrap_telemetry never replaces a provider you installed; pass on_existing_provider="error" if you would rather find out loudly than have it quietly do nothing.

Where the signals go

The Grafana stack is the prescriptive other half: one docker compose up for Alloy + Prometheus + Loki + Tempo + Grafana, four provisioned dashboards, a starter alert pack, and the label discipline that keeps both indexes affordable. Every metric is listed in the metric catalog.

The signals you watch in production are also what you assert against before shipping — see Testing.