Metric catalog

Every metric below is emitted through the global OpenTelemetry meter named forze, by an instrument_* call you make once at assembly. Nothing appears until you make that call — the instrumentation surface is opt-in per plane. Point an SDK at a collector with bootstrap_telemetry and see them in Grafana with the stack recipe.

Names are stable. This table is checked against the constants in src/ by a unit test, in both directions: a metric that exists but is not documented fails, and so does a documented metric that no longer exists.

Conventions that hold throughout:

• Unit 1 means a plain count. ms is milliseconds, s seconds.
• Observable instruments are sampled by the SDK at collection time from a live object (pool, keyring, cache, mailbox). They are cumulative per process, so a restart resets them — rate() handles that correctly only if each process carries a distinct service.instance.id, which is why bootstrap_telemetry mints one by default.
• tenant_id is never a metric label, anywhere, by design. Per-tenant questions are answered from traces and logs, which both carry it. See the cardinality doctrine.

Operations¶

instrument_operations(registry) — the outermost middleware on every operation.

The distinction that matters: outcome="failed" is a client-class domain failure (validation, not-found, conflict, precondition — a 4xx the caller may well handle) and leaves the span clean. outcome="error" is a genuine fault. Alert on error, chart both.

Resilience¶

instrument_resilience(executor) — independent of the tracing gate; a production process with tracing off still reports these.

breaker_open in the events counter covers both the open transition and every admission rejected while open, so its rate tracks shed load rather than flap count.

Tenant pools¶

instrument_tenant_pools({"postgres": client, ...}) — one entry per routed client.

The alert that matters: a sustained created rate while size == capacity is LRU thrash — hot tenants' pools evicted by cold one-off traffic, each rebuild paying full connection establishment.

Crypto and KMS¶

instrument_crypto({"default": keyring, ...}).

Hit ratio is hits / (hits + the matching generated|unwrapped) — composed at query time, never precomputed, so it aggregates correctly across processes.

The alert that matters: cold_miss should sit at ~0. A sustained rate means a read/write path is skipping warm / ensure_unwrapped.

Document L1 cache¶

instrument_document_l1() — reads a process-wide registry of live L1 stores.

Sustained evictions while size == capacity is the scan-pollution signature that justifies W-TinyLFU (L1Spec(store_factory=tiny_lfu_l1_store)) or a bigger capacity.

Durable execution¶

DurableTelemetry.create(), wired into the durable runner.

A recovered spike means runs are being reclaimed from processes that died holding them — normal in small numbers after a deploy, a signal when it is sustained. The counter cannot tell those from runs that were simply due, so enqueued runs, retries under a retry strategy and requeues set its baseline: alert above it.

Leadership¶

instrument_leadership() — reads every leadership_lifecycle_step in the process.

Summed across processes, held should be 1 per key. A sustained rate of lost or expiring means the lock backend is slow or unreachable from the leader; raised is the body's own failure.

Job progress¶

instrument_job_staleness(monitor) — see Operation progress.

Undeclared kinds land in the __other__ bucket rather than vanishing, so a kind added later is still counted.

Alarm on scan_age too, not only on stalled. The counts are a cache: if the sweep loop dies they freeze at their last value — almost always zero — and a rule watching only the count goes green at the exact moment it stops knowing anything.

Realtime gateway¶

instrument_realtime_gateway(stats) — the live-emit path. All labelled forze.realtime.channel.

The two to alarm on: poisoned (every increment is a dropped durable delivery) and emit_failed climbing while emitted is flat (Socket.IO, or its Redis backplane, stopped taking frames).

Realtime backplane¶

instrument_realtime_backplane(health), fed by the heartbeat lifecycle step. Labelled forze.realtime.channel.

A dead AsyncRedisManager listener silently stops every cross-node emit and nothing in python-socketio surfaces it. Alarm on seconds_since_ok exceeding a few heartbeat intervals; -1 is a wiring problem, not an outage.

Realtime connections¶

instrument_realtime_connections(stats), with the same RealtimeConnectionStats handed to attach_realtime_connection and connection_drain_lifecycle_step. Labelled forze.realtime.channel.

What to watch: connect_refused{reason="draining"} that keeps climbing long after a deploy means a balancer still routes to a replica it should have dropped, and any drain_closed{outcome="timed_out"} means the drain's tail is too short for this node's closes.

Realtime mailbox¶

instrument_realtime_mailbox(mailbox, cursors). Labelled forze.realtime.channel.

The alert that matters: every overflowed increment is a device that fell more than cap entries behind and lost signals it will never see.

Access-token signing¶

instrument_signing({"default": service, ...}) — the identity plane. Labelled forze.signer, forze.signer.algorithm, and forze.signer.kid when set.

A rising verify_failed rate is the signal that matters: key-rotation gaps, clock skew, or forgeries.

Revocation states¶

instrument_revocation_states({"default": revocation_states}) — the identity plane's fast path for sessions and API keys. Labelled forze.revocation_states and forze.credential (session or api_key).

The alert that matters: every write_failures increment is a revoked credential that stays admitted for up to the states' ttl. See Authentication for every cause of that window.