Skip to content

Metric catalog

Every metric below is emitted through the global OpenTelemetry meter named forze, by an instrument_* call you make once at assembly. Nothing appears until you make that call — the instrumentation surface is opt-in per plane. Point an SDK at a collector with bootstrap_telemetry and see them in Grafana with the stack recipe.

Names are stable. This table is checked against the constants in src/ by a unit test, in both directions: a metric that exists but is not documented fails, and so does a documented metric that no longer exists.

Conventions that hold throughout:

  • Unit 1 means a plain count. ms is milliseconds, s seconds, By bytes.
  • Observable instruments are sampled by the SDK at collection time from a live object (pool, keyring, cache, mailbox). They are cumulative per process, so a restart resets them — rate() handles that correctly only if each process carries a distinct service.instance.id, which is why bootstrap_telemetry mints one by default.
  • tenant_id is never a metric label, anywhere, by design. Per-tenant questions are answered from traces and logs, which both carry it. See the cardinality doctrine.

Operations

instrument_operations(registry) — the outermost middleware on every operation.

Metric Instrument Unit Labels Notes
forze.operations counter 1 forze.operation, forze.operation.kind, forze.outcome kind is command / query; outcome is success / failed / error
forze.operation.duration histogram ms same as above ms-ladder buckets installed by bootstrap_telemetry

The distinction that matters: outcome="failed" is a client-class domain failure (validation, not-found, conflict, precondition — a 4xx the caller may well handle) and leaves the span clean. outcome="error" is a genuine fault. Alert on error, chart both.

Resilience

instrument_resilience(executor) — independent of the tracing gate; a production process with tracing off still reports these.

Metric Instrument Unit Labels Notes
forze.resilience.events counter 1 forze.event, forze.policy, forze.route retries, timeouts, rate-limit/bulkhead rejections, budget exhaustion, breaker transitions
forze.resilience.breaker.state gauge 1 forze.policy, forze.route 0 closed, 1 half-open, 2 open. A breaker that never tripped reports nothing — closed by absence
forze.resilience.bulkhead.queue_depth observable gauge 1 forze.policy, forze.route calls queued behind the semaphore
forze.resilience.bulkhead.in_use observable gauge 1 forze.policy, forze.route calls holding a slot right now, for every bulkhead kind
forze.resilience.bulkhead.limit observable gauge 1 forze.policy, forze.route concurrency limit: a fixed bulkhead's cap, or the current AIMD / Gradient2 limit. in_use / limit is occupancy
forze.resilience.hedge.delay observable gauge s forze.policy, forze.route effective adaptive hedge delay (windowed P² estimate)

breaker_open in the events counter covers both the open transition and every admission rejected while open, so its rate tracks shed load rather than flap count.

Tenant pools

instrument_tenant_pools({"postgres": client, ...}) — one entry per routed client.

Metric Instrument Unit Labels Notes
forze.tenancy.pool.size observable gauge 1 forze.client live tenant pools
forze.tenancy.pool.capacity observable gauge 1 forze.client max_cached_tenants
forze.tenancy.pool.created observable counter 1 forze.client cumulative pool creations
forze.tenancy.pool.disposed observable counter 1 forze.client cumulative pool disposals
forze.tenancy.pool.evicted_explicit observable counter 1 forze.client explicit evictions (rotation signals)

The alert that matters: a sustained created rate while size == capacity is LRU thrash — hot tenants' pools evicted by cold one-off traffic, each rebuild paying full connection establishment.

Connection pools

instrument_connection_pools({"postgres": client, ...}) — one entry per client that implements SupportsConnectionPoolStats: PostgresClient, RedisClient and MongoClient.

Metric Instrument Unit Labels Notes
forze.connection_pool.in_use observable gauge 1 forze.client connections checked out, or being opened or reopened by the pool
forze.connection_pool.idle observable gauge 1 forze.client connections in the pool not checked out
forze.connection_pool.max observable gauge 1 forze.client configured cap (max_size); absent for a pool with none, such as MongoDB with max_pool_size=0
forze.connection_pool.waiting observable gauge 1 forze.client callers queued for a connection; Postgres only, absent for Redis and MongoDB

Where each number comes from:

  • Postgres: psycopg_pool's public get_stats().
  • Redis: redis-py's public get_connection_count(), split by its db.client.connection.state label. redis-py keeps a connection it disconnected, or failed to connect on checkout, among the idle ones, so idle can include closed connections. A pubsub subscription holds a connection in use while it runs.
  • MongoDB: pymongo's public ConnectionPoolListener events, which the client registers when it is built. pymongo keeps one pool per server, and the gauges report the pool with the most connections in use, so max is the per-server max_pool_size. A sum would hide a saturated primary behind idle secondaries. A connection still opening counts as in use for at most connect_timeout plus a second: pymongo sends no close event for one whose open is cancelled at the wrong moment, so it would otherwise count until its pool closes.

waiting is absent where it cannot be counted honestly. redis-py exposes no count of callers queued on a BlockingConnectionPool. pymongo publishes no event when a queued caller is cancelled, so a count would only climb.

A client with no open pool reports nothing. A routed client is refused: summing its per-tenant pools would hide the one that is saturated. Its connections are not exported; the tenant pool gauges above report only its pool cache (size and churn).

The alert that matters: in_use pinned at max means the pool is exhausted. What callers do next depends on the pool. psycopg_pool, pymongo, and redis-py's BlockingConnectionPool (used when RedisConfig.pool_timeout is set) queue them until they time out. A plain redis-py ConnectionPool refuses the next checkout at once. Only Postgres exports waiting, so there waiting above zero across scrapes shows the queue itself. in_use that climbs under steady load and never falls back is a leaked connection. On Postgres, a database that is down shows the same pattern: the pool's reconnect attempts hold in_use at max with waiting above zero while no caller saturates the pool, so pair the alert with a connectivity signal. A Postgres caller that timed out still counts in waiting until a connection is returned to the pool, so waiting stays up for as long as every connection is held or the database is unreachable.

Outbound HTTP

instrument_http_clients({"payments": client, ...}) from forze_http — one entry per HttpClient. httpx exposes no pool numbers, so outbound HTTP has no connection pool gauges. The client counts requests at its own methods instead.

Metric Instrument Unit Labels Notes
forze.http.client.in_flight observable gauge 1 forze.client requests sent and not finished: request until it has read the response, stream and send until their block exits; queued ones count

It counts requests, not connections. Leaving a stream or send block closes its response, but closing the response inside the block does not end the count. A client before initialize or after close reports nothing. A RoutedHttpClient is refused.

The alert that matters: a count that climbs under steady load and never falls back is a stream or send block that never exits. It may still hold its connection, or may have closed its response already; the count alone does not tell which. For a service transport, it is usually an SDK response nobody closes.

Crypto and KMS

instrument_crypto({"default": keyring, ...}).

Metric Instrument Unit Labels Notes
forze.crypto.data_keys.generated observable counter 1 forze.keyring KMS round-trips on the encrypt path
forze.crypto.data_keys.unwrapped observable counter 1 forze.keyring KMS round-trips on the decrypt path
forze.crypto.cache.hits observable counter 1 forze.keyring, forze.crypto.path path is encrypt / decrypt; data keys reused without a KMS call
forze.crypto.cold_miss observable counter 1 forze.keyring synchronous crypt that hit a cold cache and raised cipher_not_warm

Hit ratio is hits / (hits + the matching generated|unwrapped) — composed at query time, never precomputed, so it aggregates correctly across processes.

The alert that matters: cold_miss should sit at ~0. A sustained rate means a read/write path is skipping warm / ensure_unwrapped.

Document L1 cache

instrument_document_l1() — reads a process-wide registry of live L1 stores.

Metric Instrument Unit Labels Notes
forze.cache.l1.size observable gauge 1 forze.document live entries
forze.cache.l1.capacity observable gauge 1 forze.document configured capacity
forze.cache.l1.hits observable counter 1 forze.document cumulative hits
forze.cache.l1.misses observable counter 1 forze.document cumulative misses
forze.cache.l1.evictions observable counter 1 forze.document includes rejected admissions

Sustained evictions while size == capacity is the scan-pollution signature that justifies W-TinyLFU (L1Spec(store_factory=tiny_lfu_l1_store)) or a bigger capacity.

Process

instrument_process() — this process's own CPU and memory, from the standard library. The names are OpenTelemetry's process semantic conventions, not forze.*.

Metric Instrument Unit Labels Notes
process.cpu.time observable counter s cpu.mode user / system, from os.times(); rate() is the cores this process uses
process.memory.usage observable up-down counter By — resident memory from /proc/self/statm. Linux only: absent elsewhere

Outside Linux the standard library has only the peak resident size, and a peak exported as the current value would hide the leak it is meant to show, so the instrument is skipped.

The OpenTelemetry Prometheus exporter renders these as process_cpu_time_seconds_total and process_memory_usage_bytes, not prometheus_client's process_cpu_seconds_total / process_resident_memory_bytes, so panels built on those need new queries. Through the Grafana stack, with suffixes off, they arrive as process_cpu_time and process_memory_usage, which its data planes dashboard queries. A default PrometheusMetricReader() renders the default registry, whose own process collector reports that pair beside these; worker_metrics_reader() renders a registry without it.

Each process reports its own series. A pre-fork server scraped through one port answers from whichever worker takes the scrape, unless every worker writes a snapshot through worker_metrics_reader() and MetricsSnapshots, which label each series service_instance_id. See Several workers, one scrape.

Durable execution

DurableTelemetry.create(), wired into the durable runner.

Metric Instrument Unit Labels Notes
forze.durable.runs counter 1 forze.durable.name, forze.durable.outcome outcome is completed / failed / retrying / forward_incomplete / cancelled / timed_out / reclaimed / interrupted / unrecorded — alert on failed, not on the total: cancelled means somebody pressed Stop, and retrying is an attempt that failed and sent its run back for another under the runner's retry strategy. unrecorded describes the attempt, not the run: the body finished and its terminal write went unacknowledged, so the row is either already terminal (the write committed, the ack was lost) or still RUNNING for recovery — the worker cannot tell. A sustained rate of it means the run store is unreachable. interrupted is the neighbouring case where the body never finished at all (drain, shutdown) and no terminal state was written — expected during a deploy
forze.durable.run.duration histogram ms forze.durable.name, forze.durable.outcome ms-ladder buckets installed by bootstrap_telemetry
forze.durable.recovered counter 1 — runs a recovery sweep claimed: reclaimed from a process that died holding them, or simply due (enqueued, back for a retry, requeued)
forze.durable.schedule.fires counter 1 forze.durable.name one per schedule fire

A recovered spike means runs are being reclaimed from processes that died holding them — normal in small numbers after a deploy, a signal when it is sustained. The counter cannot tell those from runs that were simply due, so enqueued runs, retries under a retry strategy and requeues set its baseline: alert above it.

Leadership

instrument_leadership() — reads every leadership_lifecycle_step in the process.

Metric Instrument Unit Labels Notes
forze.leadership.held observable gauge 1 forze.leadership.key 1 while this process holds the lease, through its term's grace and until the release
forze.leadership.terms_ended observable counter 1 forze.leadership.key, forze.leadership.reason reason is lost / expiring / stopped / yielded / returned / raised / unclaimed

Summed across processes, held should be 1 per key. A sustained rate of lost or expiring means the lock backend is slow or unreachable from the leader; raised is the body's own failure.

Job progress

instrument_job_staleness(monitor) — see Operation progress.

Metric Instrument Unit Labels Notes
forze.jobs.stalled observable gauge 1 forze.job.kind started, unfinished, silent past the window
forze.jobs.stalled.oldest_silence observable gauge s forze.job.kind seconds since the quietest stuck job reported; computed at scrape
forze.jobs.staleness.scan_age observable gauge s — seconds since the last successful sweep; -1 before the first

Undeclared kinds land in the __other__ bucket rather than vanishing, so a kind added later is still counted.

Alarm on scan_age too, not only on stalled. The counts are a cache: if the sweep loop dies they freeze at their last value — almost always zero — and a rule watching only the count goes green at the exact moment it stops knowing anything.

Realtime gateway

instrument_realtime_gateway(stats) — the live-emit path. All labelled forze.realtime.channel.

Metric Instrument Unit Notes
forze.realtime.gateway.emitted observable counter 1 frames delivered to Socket.IO
forze.realtime.gateway.emit_failed observable counter 1 sio.emit raised, including emit-timeout expiries
forze.realtime.gateway.presence_skipped observable counter 1 empty principal room; recoverable via the mailbox
forze.realtime.gateway.dedup_skipped observable counter 1 durable signals already seen
forze.realtime.gateway.admission_rejected observable counter 1 rejected at the catalog admission gate
forze.realtime.gateway.untenanted_dropped observable counter 1 no tenant resolved on a require_tenant gateway
forze.realtime.gateway.mailboxed observable counter 1 stored for offline replay
forze.realtime.gateway.bridge_failed observable counter 1 redelivered if durable, dropped if ephemeral
forze.realtime.gateway.poisoned observable counter 1 dropped at the delivery ceiling — bounded loss

The two to alarm on: poisoned (every increment is a dropped durable delivery) and emit_failed climbing while emitted is flat (Socket.IO, or its Redis backplane, stopped taking frames).

Several channels in one process: call once per channel. The gateway, backplane, connections and hub calls share one set of instruments per meter, so each channel exports its own series. A repeat call for the same object and channel adds nothing. Give every call the same meter, or none: a default call made before the SDK is configured and a later call passing the SDK's forze meter register under different meters, and the later one's channels are dropped.

from forze.application.execution.realtime import (
    RealtimeGatewayStats,
    instrument_realtime_gateway,
)

orders, chat = RealtimeGatewayStats(), RealtimeGatewayStats()
instrument_realtime_gateway(orders, channel="orders")
instrument_realtime_gateway(chat, channel="chat")

Realtime backplane

instrument_realtime_backplane(health), fed by the heartbeat lifecycle step. Labelled forze.realtime.channel.

Metric Instrument Unit Notes
forze.realtime.backplane.seconds_since_ok observable gauge s seconds since the backplane last accepted a probe; -1 means never
forze.realtime.backplane.consecutive_failures observable gauge 1 failed probes since the last success

A dead AsyncRedisManager listener silently stops every cross-node emit and nothing in python-socketio surfaces it. Alarm on seconds_since_ok exceeding a few heartbeat intervals; -1 is a wiring problem, not an outage.

Realtime connections

instrument_realtime_connections(stats), with the same RealtimeConnectionStats handed to attach_realtime_connection and connection_drain_lifecycle_step. Labelled forze.realtime.channel.

Metric Instrument Unit Labels Notes
forze.realtime.connections observable up-down counter 1 forze.realtime.namespace connections open on this node
forze.realtime.connect_refused observable counter 1 forze.realtime.namespace, forze.realtime.reason protocol, auth, draining, admission, internal; a draining connect is turned away by closing its transport
forze.realtime.drain_closed observable counter 1 forze.realtime.outcome closes the drain started: closed, failed, timed_out

What to watch: connect_refused{reason="draining"} that keeps climbing long after a deploy means a balancer still routes to a replica it should have dropped, and any drain_closed{outcome="timed_out"} means the drain's tail is too short for this node's closes.

Realtime hub

instrument_realtime_hub(hub) from forze_fastapi.realtime, with the RealtimeSseHub the SSE and WebSocket routes share. Labelled forze.realtime.channel.

Metric Instrument Unit Notes
forze.realtime.hub.subscribers observable up-down counter 1 SSE and WebSocket streams subscribed to the live tail on this node
forze.realtime.hub.dropped observable counter 1 live signals dropped on a full subscriber queue; each is a lost live frame

These are not folded into forze.realtime.connections, which only Socket.IO feeds: the FastAPI WebSocket route reports to the hub alone, and no Socket.IO connection subscribes to it. The two never overlap, so their sum is the node's live realtime connections. A catch-up-only SSE stream or a replay-only WebSocket never joins the live tail, so it is in neither. What to watch: a dropped rate means some client reads slower than signals arrive. With a mailbox wired, its durable signals arrive on its next reconnect; ephemeral ones are gone.

Realtime mailbox

instrument_realtime_mailbox() — reads a process-wide count per mailbox and cursor spec, so it covers every mailbox built in the process: a per-connect mailbox_factory, the Socket.IO gateway, RealtimeMailboxWriter and the retention sweep. Labelled forze.document, the spec's name: the mailbox spec's, or the cursor spec's for acked.

Metric Instrument Unit Notes
forze.realtime.mailbox.stored observable counter 1 durable principal signals stored for replay
forze.realtime.mailbox.replayed observable counter 1 entries fetched on connect-time replay
forze.realtime.mailbox.trimmed observable counter 1 dropped by retention/ack trimming
forze.realtime.mailbox.acked observable counter 1 per-device cursor advances
forze.realtime.mailbox.overflowed observable counter 1 replays that lost their oldest backlog to the cap
forze.realtime.mailbox.replay_size histogram 1 per connect, the entries past the device's acked cursor that its replay read; recorded only when the replay finishes; the SDK's default buckets cover the default cap of 1000

The alert that matters: every overflowed increment is a device that fell more than cap entries behind and lost signals it will never see. replay_size warns earlier: a tail that keeps growing means devices fall behind on acks. It is a trend, not the backlog's size: no gauge counts mailbox rows, since that takes a store query a scrape must not run.

replay_size counts the unacked backlog, not the frames sent. A resume with a last-seen id (SSE Last-Event-ID, WebSocket last_event_id) skips the entries delivered before the tear, and they still count. A replay the client leaves, or the store fails, records nothing. Only a drain through iter_backlog records, as every forze transport does: a custom transport that drains iter_replay itself counts replayed but records no replay_size. The counters are process totals, so the first scrape after the call includes what came before it; replay_size records only the replays that finish after it.

trimmed counts the ids a trim or sweep handed to the document port's kill_many, once it returns. The shipped adapters raise unless every id was deleted, so a custom port that accepts absent ids can over-count, and only when two cleanups delete the same rows at once.

Access-token signing

instrument_signing({"default": service, ...}) — the identity plane. Labelled forze.signer, forze.signer.algorithm, and forze.signer.kid when set.

Metric Instrument Unit Notes
forze.authn.tokens.signed observable counter 1 access tokens issued; for a KMS-held key this tracks sign round-trips
forze.authn.tokens.verified observable counter 1 verified successfully
forze.authn.tokens.verify_failed observable counter 1 rejected as expired/invalid

A rising verify_failed rate is the signal that matters: key-rotation gaps, clock skew, or forgeries.

Revocation states

instrument_revocation_states({"default": revocation_states}) — the identity plane's fast path for sessions and API keys. Labelled forze.revocation_states and forze.credential (session or api_key).

Metric Instrument Unit Notes
forze.authn.revocation_states.hits observable counter 1 verifications answered from Redis alone, refusals included
forze.authn.revocation_states.misses observable counter 1 verifications that read the database for a state Redis lacked or could not trust
forze.authn.revocation_states.fallbacks observable counter 1 verifications whose Redis read failed, timed out or met an open breaker
forze.authn.revocation_states.write_failures observable counter 1 revocations whose write-through failed every attempt
forze.authn.revocation_states.clock_skew observable counter 1 API-key states refused because the application's clock runs two ttl or more ahead of Redis's

The alert that matters: every write_failures increment is a revoked credential that stays admitted for up to the states' ttl. See Authentication for every cause of that window.

Grants invalidation

instrument_grants_invalidation({"default": grants_invalidation}) — the broadcast that keeps every process's grants cache current. Labelled forze.grants_invalidation.

Metric Instrument Unit Notes
forze.authz.grants_invalidation.publish_failures observable counter 1 committed invalidations whose marker write failed
forze.authz.grants_invalidation.flushes observable counter 1 whole grants caches dropped on a gap in the stream or a catalog change
forze.authz.grants_invalidation.bypasses observable counter 1 effective-grants reads (decisions, scoping, grant queries) that skipped the cache because no subscription kept it current

Every publish_failures increment leaves other processes on the old grants for up to the cache's TTL. A steady flushes rate is a flapping connection: the cache stays correct and stops helping. A rising bypasses count in a process that should be subscribed means the lifecycle step is not registered there, or its stream has ended. See Identity.

Worker snapshots

Rendered by attach_metrics_route(..., snapshots=...) itself, not by an instrument_* call: one sample per worker the scrape serves, labelled service_instance_id. Prometheus sees it as forze_metrics_snapshot_age_seconds. See Several workers, one scrape.

Metric Instrument Unit Notes
forze.metrics.snapshot.age gauge s seconds since that worker wrote the snapshot served; never above the ttl

It describes a worker's whole snapshot, not each series in it. count() of it is the number of workers served: alert when it falls below the worker count, because up stays 1 while a worker is missing.