Observability
Forze is already observability-rich inside — a runtime tracer, a transaction tracer, structured logs. One call pushes that out to OpenTelemetry: a span and metrics for every operation, tagged with identity and correlation, so your operations show up in any tracing or metrics backend.
OpenTelemetry is a core dependency — the logging layer already uses it — so this is built in. You only bring an exporter.
Instrument every operation¶
Wrap the registry once, before you freeze it:
from forze.application.execution import instrument_operations
registry = instrument_operations(build_orders_registry())
frozen = registry.freeze()
That's the whole integration. It emits through the global OTel providers by
default; pass tracer= / meter= to target your own.
What you get¶
Every operation produces a span, named by its operation key and nested under
whatever span is already active (an incoming HTTP request, say). A server-side
failure — an internal, infrastructure, configuration or timeout error, or any
exception that is not a CoreException — records the exception and sets the span
status to ERROR; a failure the caller can handle, such as not-found, validation or
unavailable, leaves the span clean. Either way the exception is re-raised
unchanged. The span carries:
| Attribute | Value |
|---|---|
forze.operation / forze.operation.kind |
the operation key and query / command |
forze.execution_id / forze.correlation_id / forze.causation_id |
the invocation metadata |
forze.invocation_surface |
the boundary the invocation entered through, or unknown |
forze.tenant_id / forze.principal_id |
the bound tenant and principal |
Alongside each span, two metrics — forze.operations (a counter) and
forze.operation.duration (a histogram, in ms) — labelled by operation, kind,
and outcome: success, failed for a failure the caller can handle, or error.
An unavailable error — something an operator switched off — counts as failed, not
error, and no transport logs it as a server error, so an error-rate alert on
forze.outcome="error" leaves it out. The operation metrics carry no error code, so if
switch-offs matter to you, track them at the edge: count responses whose X-Error-Code is
the code your app raises with exc.unavailable, or 503s at your proxy.
See which boundary an invocation came through¶
Every boundary that starts an invocation names itself on the invocation metadata, so a
log line or span says whether the work arrived over HTTP, from forze ops, or from a
recovered durable run. Each line logged inside an operation carries the
invocation_surface field, and the operation span carries forze.invocation_surface.
An operation run outside any boundary, such as a test harness calling run_operation,
records unknown.
| Value | Boundary |
|---|---|
http |
forze_fastapi's InvocationMetadataMiddleware: generated routes and SSE |
websocket |
a forze_fastapi realtime WebSocket command frame |
socketio |
a forze_socketio command event |
mcp |
a forze_mcp tool call |
cli |
forze ops call |
inbox |
process_with_inbox, once per delivery |
temporal |
a workflow or activity behind forze_temporal's interceptors |
inngest |
a forze_inngest function call |
durable |
one attempt of a self-hosted durable run |
unknown |
no boundary bound metadata |
The values are InvocationSurface members in forze.application.execution. A nested
call keeps the surface of the invocation around it: an agent tool call made while serving
an HTTP request records http. A hop names itself and carries nothing: a Temporal
activity started from an HTTP request records temporal, and the request is found
through the shared correlation_id.
The field is a log field, not a metric label and not a Loki label, so it adds no series.
The key is invocation_surface and not surface because surface is the port-call key
that with_port_logging() writes, and both appear on one port-call line.
Your own boundary. A gRPC server or a queue worker of your own passes its own token:
mint_invocation_metadata(surface="grpc"), from forze.application.execution.ingress,
or InvocationMetadata(..., surface="grpc"). A token must match
^[a-z][a-z0-9_]{0,31}$, or construction raises configuration with code
invocation_surface_invalid. Pass a literal from your boundary's source:
- never a value read from a header, a frame, a message or an event, which a caller would then choose;
- never a token that names a person, a tenant or a customer, since it reaches every log line and span.
Attribution, not authority
No framework hook, policy or guard reads the surface, and your code should not either.
Any code in the process can build metadata with any valid token, so a check such as
"only if the surface is cli" proves nothing. Restrict which operations a surface may
call with its exposure allowlist instead.
Trace every outbound call¶
The operation span tells you that an operation ran and how long it took; to see
where the time went — which database read, which cache call, which outbound
request — span the ports too. In a hexagonal app every external call is a port, so
opting the resolved ports into a span turns that seam into a complete outbound-I/O
trace under each operation. It rides on the dependency registry, alongside
instrument_operations on the operation registry:
from forze.application.execution import DepsRegistry
deps = DepsRegistry.from_modules(postgres_module, redis_module).with_otel_port_spans()
Freeze deps into the runtime as usual. Every document / cache / queue / search /
HTTP call now opens a CLIENT span named surface.op (document_command.create,
cache_query.get), nested under the operation span:
| Attribute | Value |
|---|---|
forze.port.domain / forze.port.surface |
the port family and its dependency surface |
forze.port.op |
the method called — create, get, publish |
forze.port.route / forze.port.phase |
the specification name and phase, when present |
The name stays low-cardinality (surface.op); the specification name rides as the
route attribute, never in the name. The span sits inside the resilience
policy, so a retried call is one span per attempt and a call shed by an open breaker
or a full bulkhead emits none — the trace shows the work that actually reached the
backend. Failure status follows the exception kind: an infrastructure, internal,
or configuration fault reds the span, while a domain failure the caller can handle —
not-found, conflict, precondition — leaves it clean, exactly as a 404 does not red an
HTTP client span. Streaming methods (a cursor, a consume loop, a subscription) pass
through un-spanned — a lone span around a long-lived stream is the wrong shape;
per-message spans belong in the consumer.
For a log line per outbound call instead of (or alongside) a span, add
deps.with_port_logging(). It logs every port call uniformly — surface, route,
op, and duration_ms under forze.integrations.<domain> — at trace on success
(so it costs nothing in production unless you turn trace on), debug on an expected
domain failure, and warning with a traceback on an unexpected one.
Both wrap the port a context resolves, so a @runtime_checkable protocol check of it
is false even when the backend implements the protocol. Ask
implements(port, SomeAware), from forze.application.contracts.deps, instead: it
answers for the backend behind the wrappers. Keep calling the port itself, so the span
and the log line still happen.
Follow the trace across boundaries¶
Spans and port spans cover one operation inside one process. When the work crosses a
boundary the trace would normally break — so Forze carries the W3C traceparent
across the two crossings OpenTelemetry's transport instrumentation can't bridge on
its own.
An event published now and consumed later keeps its trace when you set
propagate_trace=True on the outbox integration:
the publishing operation's trace context is captured as the event is staged, travels
with it through the broker, and the consumer rebuilds it so the handler's spans link
back to the publish span. On a relational backend, add a nullable traceparent
column first — exactly as you do for hlc_ordering.
An outbound HTTP request carries it automatically: the HTTP adapter injects the active trace context into every request (a no-op when no span is active), honouring your application's configured propagator. Nothing to enable.
The traceparent is untrusted metadata
It arrives over the broker or the wire, so it influences trace parenting only — never identity, tenancy, or deduplication. Treat it as a hint for the tracing backend, nothing more.
Inbound HTTP runs the other way: a request arriving at FastAPI is picked up by the
standard opentelemetry-instrumentation-fastapi, which creates the server span your
operation span nests under — so Forze does not re-extract it.
Resilience metrics¶
The resilience layer makes decisions worth watching —
retries, rejections, breaker trips, bulkhead backoff. instrument_resilience
exports them as always-on metrics, independent of any tracing gate, so a
production process with tracing off still reports them:
from forze.application.execution import instrument_resilience
instrument_resilience(ctx.resilience()) # once, when the scope is up
| Metric | What it carries |
|---|---|
forze.resilience.events (counter) |
every event — retry attempts, timeouts, rate-limit and bulkhead rejections, breaker transitions — labelled by event, policy, and route |
forze.resilience.breaker.state (gauge) |
breaker phase per policy/route: 0 closed, 1 half-open, 2 open |
forze.resilience.bulkhead.queue_depth (gauge) |
calls queued behind each bulkhead, sampled at collection |
forze.resilience.bulkhead.in_use (gauge) |
calls holding a slot in each bulkhead right now, sampled at collection |
forze.resilience.bulkhead.limit (gauge) |
each bulkhead's concurrency limit: the fixed cap, or the current AIMD or Gradient2 limit |
forze.resilience.hedge.delay (gauge) |
the effective adaptive hedge delay (P² quantile estimate), in seconds |
Two reading notes: breaker_open counts the open transition and every
admission shed while open, so its rate tracks shed load; and a breaker that
never tripped reports no state at all — closed by absence.
For bulkheads, in_use / limit is occupancy for every bulkhead kind. It shows a
bulkhead filling up before the queue grows, and a slot that is taken and never
released shows as in_use that stays up when traffic stops.
Tenant pool metrics¶
Routed clients keep one connection pool per tenant in a
bounded LRU, and evicting a pool is expensive — the next request rebuilds the
connection from scratch. instrument_tenant_pools exports the churn
counters:
from forze.application.execution import instrument_tenant_pools
instrument_tenant_pools({"postgres": pg, "redis": redis})
Per client (labelled forze.client): forze.tenancy.pool.size and
….capacity gauges, plus cumulative ….created, ….disposed, and
….evicted_explicit counters. The alert worth setting: a sustained
creation rate while size sits at capacity means the LRU is thrashing —
hot tenants' pools evicted by cold one-off traffic, each rebuild paying full
connection establishment. The fix is usually a larger max_cached_tenants;
the metric tells you when.
Connection pool metrics¶
Pool state is the first thing to check when an API slows under load, and a
connection that is never returned is the leak a soak test exists to catch.
instrument_connection_pools exports the pool of every client you name:
from forze.application.execution import instrument_connection_pools
instrument_connection_pools({"postgres": pg, "redis": redis, "mongo": mongo})
| Metric | What it carries |
|---|---|
forze.connection_pool.in_use (gauge) |
connections checked out, or being opened or reopened by the pool |
forze.connection_pool.idle (gauge) |
connections in the pool not checked out |
forze.connection_pool.max (gauge) |
the configured cap, absent for a pool without one |
forze.connection_pool.waiting (gauge) |
callers queued for a connection, Postgres only |
One series per client, labelled forze.client with the name you chose. A
client counts if it implements SupportsConnectionPoolStats
(forze.application.contracts.pools) with an in-memory snapshot. The callback
runs on every scrape, so it never does I/O. PostgresClient, RedisClient
and MongoClient do, each from public driver numbers only; the
metric catalog says where each one
comes from. A MongoClient reports its busiest server's pool, so on a replica
set max is the per-server cap. A routed client is refused, because it keeps one
pool per tenant and a sum would hide the saturated one. Its connections are
not exported: the tenant pool gauges above report only its pool cache. Alert
on in_use pinned at max: the pool is exhausted. What callers do next
depends on the pool. Postgres, MongoDB, and a Redis client with
pool_timeout set queue them until they time out; a Redis client without it
refuses the next checkout at once. Only Postgres exports waiting, so there
waiting above zero across scrapes shows the queue itself. On Postgres, a
database that is down shows the same pattern, because the pool's reconnect
attempts hold in_use at max while callers wait. Pair the alert with a
connectivity signal, such as the health check.
httpx exposes no pool numbers, so outbound HTTP gets a count of requests
instead: instrument_http_clients from forze_http exports
forze.http.client.in_flight per HttpClient: requests sent and not
finished, until request has read the response or until a stream or
send block exits. A count that never falls back is a block that never
exits; it may or may not still hold its connection.
from forze_http import instrument_http_clients
instrument_http_clients({"payments": payments})
Document L1 metrics¶
The in-process L1 exports its counters the same way:
from forze.application.integrations.document import instrument_document_l1
instrument_document_l1()
Per document (labelled forze.document): forze.cache.l1.size /
….capacity gauges and cumulative ….hits / ….misses / ….evictions
counters. The hit rate validates that the L1 is earning its staleness budget,
and sustained evictions at full capacity with a sagging hit rate is the
scan-pollution signature — the signal to switch the eviction policy to the
in-box W-TinyLFU store or raise capacity.
Process metrics¶
A slow memory leak or CPU creep only shows in a soak if the server's own memory
and CPU arrive in the same scrape as everything else. instrument_process
exports them, standard library only:
from forze.application.execution import instrument_process
instrument_process() # once per process — in every worker of a pre-fork server
| Metric | What it carries |
|---|---|
process.cpu.time (counter) |
CPU seconds labelled cpu.mode — user / system; rate() gives the cores used |
process.memory.usage (up-down counter) |
resident memory in bytes, from /proc/self/statm — Linux only |
The names are OpenTelemetry's process semantic conventions. Prometheus sees
them as process_cpu_time_seconds_total and process_memory_usage_bytes, not
prometheus_client's process_cpu_seconds_total /
process_resident_memory_bytes. A default PrometheusMetricReader() renders
the default registry, whose own process collector reports that pair beside
these. Outside Linux the standard library offers only the peak resident
size, so the memory instrument is skipped rather than reporting a peak as
current. Each collection costs one syscall and a small /proc read.
Every worker reports its own series, but one Prometheus scrape of a pre-fork
server reaches one worker. To see them all, give each worker
worker_metrics_reader() and serve
MetricsSnapshots, which labels
each worker service_instance_id, or push every worker to
one collector, which labels it
instance.
For more — open file descriptors, threads, disk and network I/O — use
opentelemetry-instrumentation-system-metrics, which depends on psutil.
Realtime metrics¶
Each realtime plane has its own opt-in call; the catalog lists what each exports. Two cover what a soak needs to tell a stuck node from a busy one:
from forze_fastapi.realtime import instrument_realtime_hub
from forze_kits.integrations.realtime import instrument_realtime_mailbox
instrument_realtime_hub(hub) # the hub the SSE and WebSocket routes share
instrument_realtime_mailbox() # every mailbox built in the process, per spec
forze.realtime.hub.subscribers counts the streams on this node's live tail; a count that only
rises means streams that never leave. forze.realtime.hub.dropped counts live signals lost to
a full subscriber queue. forze.realtime.mailbox.replay_size is a histogram of each
connect's unacked backlog: the entries past the device's acked cursor that its replay read.
A tail that keeps growing means devices fall behind on acks. It is a trend, not the
backlog's size: counting mailbox rows takes a store query, and no scrape callback runs one.
Only a replay that finishes records; one the client leaves or the store fails does not.
Configure logging in one call¶
bootstrap_logging wires the whole logging surface: the framework's own loggers
(forze.*, forze_kits.*, and the forze.integrations.* adapters), any
integration loggers you name, third-party stdlib loggers routed through the same
formatter, and the uncaught-exception hook.
from forze import bootstrap_logging
bootstrap_logging(
level="info",
render_mode="json", # "console" for local dev
logger_names=["forze_postgres", "forze_redis"],
third_party=["uvicorn", "uvicorn.access", "sqlalchemy.engine"],
)
Name a package root and the stdlib logger hierarchy carries the setting to every
logger under it (forze_postgres.kernel, forze_postgres.adapters, …), so there
is nothing per-subpackage to enumerate.
Naming loggers narrows capture, not redaction: a logger you did not list still reaches a
scrubbing handler on the root at warning and above (or your level, if higher), and
prints scrubbed on the configured stream where Python's last-resort handler printed it raw
on stderr. With level above warning, forze's handler stops printing unlisted warnings. A root
handler you installed yourself stays, so each unlisted line then appears twice; an app that
manages the root itself passes redact_unnamed=False, which declines the root handler with
a warning and leaves unlisted loggers to Python's default handling, unscrubbed. A later
logging.basicConfig() does nothing, because the root already has a handler: call it with
force=True, or before bootstrap_logging. Uvicorn sets propagate=False on uvicorn and
on uvicorn.access, so neither reaches the root: name both in third_party.
Until structlog is configured (configure_logging, bootstrap_logging or your own
structlog.configure), forze does not print to stdout itself: it hands each event to
the stdlib logger of the same name at that logger's level, so warnings and above reach
stderr by default, or your own logging handlers at the level you set.
configure_logging remains the lower-level entry point when you want to wire the
pieces yourself. Either way, pass otel_config=... and the active span's
trace_id and span_id are injected into every log line — because
instrument_operations starts that span, your structured logs line up with the
operation trace automatically, no extra wiring.
Log levels and the verbosity budget¶
The framework holds one rule: at level="info" it emits almost nothing per
request. A quiet default is a feature — you turn detail up when investigating,
rather than filtering noise out in steady state. Each level has a fixed meaning:
| Level | What the framework logs here | Steady-state volume |
|---|---|---|
trace |
per-row / per-message / per-port-call detail | none in production — the trace gate is one integer compare unless you configure level="trace" |
debug |
per-operation internals, cache hits, dedup skips | opt-in — at a higher level, a debug call returns after one cheap check |
info |
lifecycle events only: startup, shutdown, saga pivot, relay batch summaries | rare |
warning |
degraded-but-continuing: retries exhausted, breaker open, a callback failed | rare, deduped |
error |
an unhandled server-side fault (a bug) | should be ~zero |
critical |
data loss or an unrecoverable condition | ~zero |
A domain failure is never an error. A validation, not-found, conflict, or
precondition outcome is the application working as designed — it logs at debug
or not at all, exactly as it leaves an HTTP span clean.
Only an unhandled fault reaches error.
Tame high-volume logs¶
Two per-event controls collapse the events that would otherwise flood a log, without dropping the first occurrence you actually need to see. They are a no-op for events that don't opt in:
# keep 1 in 100 of a uniformly high-volume event
logger.debug("cache miss", _sample=100)
# emit a flapping condition at most once per window (default 60s)
logger.warning("upstream degraded", _dedup_key="upstream-degraded")
_sample=N keeps one in every N events sharing the same logger and message;
_dedup_key emits at most one event per key per window (_dedup_window= overrides
the seconds). The control keys are stripped before rendering. This is on by default
(configure_logging(enable_sampling=True)).
For per-request access logs — the largest steady-state source — the FastAPI and MCP
middlewares are quiet by default: successful requests are sampled 1-in-N and error
responses are always logged. The FastAPI middleware additionally drops health and
readiness probe paths (DEFAULT_HEALTH_PATHS); MCP messages have no such path, so
its default sampler applies no path exclusion. Configure either with
access_log=AccessLogSampler(...): mode="full" logs every request, mode="off"
disables them, and sample_rate / exclude tune the rate and the excluded subjects
(request paths for FastAPI, method names for MCP).
Sensitive data is scrubbed¶
Log output runs through a redaction pass that masks sensitive keys (password, token, secret, api-key, cookie, authorization, …) and secret-shaped values (bearer tokens, JWTs, connection DSNs) in both extras and the message text. Extend it for deployment-specific patterns once at startup:
from forze.base.scrubbing.policy import register_sensitive_patterns
register_sensitive_patterns(keys=["x_internal_token"])
Disable with configure_logging(sanitize_logs=False) only when you fully control
the sink and its retention.
Naming: where a log line comes from¶
Every logger is namespaced so you can raise or lower detail per area without
touching the rest. Core framework logs sit under forze.* (forze.application,
forze.domain); pre-built wiring under forze_kits.*; each integration under its
own forze_<name>.* (forze_postgres.adapters, forze_redis.kernel). Generic
adapter machinery shared across integrations logs under forze.integrations.<domain>
(forze.integrations.cache, forze.integrations.document) — filter the whole group
with forze.integrations.*, or a single domain on its own.
One-call SDK setup¶
Everything above emits through the global OTel providers. Somebody still has to
install those providers and point them at a collector — and that is the fifteen lines
every deployment hand-writes and gets wrong in the same four ways. bootstrap_telemetry
is the bootstrap_logging twin that owns them:
from forze.base.logging import bootstrap_logging
from forze.base.telemetry import bootstrap_telemetry
bootstrap_logging(render_mode="json", otel_config={"enable": True})
telemetry = bootstrap_telemetry(
service_name="orders-api",
service_version=APP_VERSION,
)
# ... assemble: instrument_operations / with_otel_port_spans / instrument_* ...
await telemetry.shutdown() # flush is part of drain, not an afterthought
Endpoint, headers and the head sampler come from the standard OTEL_EXPORTER_OTLP_* /
OTEL_TRACES_SAMPLER environment variables — no parallel vocabulary. Install the exporter
with the observability extra (uv add "forze[observability]"); it is OTLP
http/protobuf, so no grpc.
Four things it gets right that hand-written setup usually does not:
| Why it matters | |
|---|---|
a fresh service.instance.id per process |
without it, every worker's cumulative counters collide into one series and rate() flaps |
| millisecond histogram views | the SDK's default buckets are second-oriented; Forze records ms |
shutdown() wired into drain |
otherwise the last export interval of every deploy dies with the pod |
| defers to an existing provider | an application that owns its SDK keeps it, untouched |
Call it before assembly. The API's proxies do late-bind instruments created earlier (pinned by test, not assumed), but ordering it this way stops the first spans of a slow startup being dropped.
Bringing your own SDK is still fine and still supported — bootstrap_telemetry never
replaces a provider you installed; pass on_existing_provider="error" if you would rather
find out loudly than have it quietly do nothing.
Where the signals go¶
The Grafana stack is the prescriptive other half: one
docker compose up for Alloy + Prometheus + Loki + Tempo + Grafana, four provisioned
dashboards, a starter alert pack, and the label discipline that keeps both indexes
affordable. Every metric is listed in the metric catalog.
The signals you watch in production are also what you assert against before shipping — see Testing.