Skip to content

The Grafana stack

Observability gets your application emitting. This page is the other half: where those signals go, and what you look at when something is wrong. It is a prescription, not an abstraction — the framework ships contracts (JSON on stdout, OTLP, probes) plus these assets, and never ships deployment code.

The wiring picture

Your application talks to exactly one thing: Alloy. Everything else is behind it, which is what lets you swap Loki for Elasticsearch or Prometheus for Mimir without touching a service.

  your app ──stdout JSON──► Alloy ──► Loki    ─┐
           ──OTLP/http────► Alloy ──► Tempo   ─┼──► Grafana
                                  └─► Prometheus (remote-write)
  node exporter ─────────────────────► Prometheus     (infrastructure; framework-uninvolved)

Three facts make this work without any glue code of yours:

  • Metrics and traces already leave through the global OTel providers, so pointing them somewhere is a matter of installing an SDK — that is all bootstrap_telemetry does.
  • Logs are already JSON on stdout with trace_id and span_id injected from the active span, so a log-to-trace jump in Grafana is a datasource setting, not a code change.
  • traceparent already crosses the outbox → broker → inbox boundary, so a trace that starts in an HTTP request continues in the consumer that handles its event.

One docker compose up

Download the stack and start it:

Keep the layout — the compose file mounts ./provisioning and ./dashboards by path:

grafana/
  docker-compose.yml   config.alloy   prometheus.yml   tempo.yml   alerts.yml
  dashboards/          forze-*.json
  provisioning/        datasources/datasources.yml   dashboards/dashboards.yml
docker compose up -d
open http://localhost:3000

Grafana comes up with the datasources wired, the four dashboards in a Forze folder, and Prometheus already evaluating the alert rules. Nothing is populated yet, because nothing is pointed at it.

Point your application at it

Two calls at process start, before assembly:

from forze.base.logging import bootstrap_logging
from forze.base.telemetry import bootstrap_telemetry

bootstrap_logging(render_mode="json", otel_config={"enable": True})

telemetry = bootstrap_telemetry(
    service_name="orders-api",
    service_version=APP_VERSION,
)

and one environment variable:

OTEL_EXPORTER_OTLP_ENDPOINT=http://alloy:4318

Endpoint, headers, timeouts and the head sampler all come from the standard OTEL_* variables; bootstrap_telemetry deliberately adds no second vocabulary for them. Install the exporter with the observability extra:

uv add "forze[observability]"

Both argument lists are also a settings model, so a deployment configures them from the environment instead of retyping the defaults:

from forze.base.settings import RuntimeSettings

rt = RuntimeSettings(version=APP_VERSION, build_id=BUILD_ID, telemetry="otlp")

bootstrap_logging(level=rt.log_level, render_mode=rt.log_render)
bootstrap_telemetry(
    service_name="orders-api",
    service_version=rt.full_version,
    exporter=rt.telemetry,
)

RuntimeSettings is a plain BaseModel — mount it on your own BaseSettings root, which owns the environment prefix and delimiter. It defaults log_render to json, unlike bootstrap_logging itself: a settings object exists because something is being deployed.

Then flush on the way out, after the drain gate flips:

@asynccontextmanager
async def lifespan(app: FastAPI):
    async with runtime.scope():
        yield

    # After the scope drained and tore down, but before the process exits: the last export
    # interval is exactly the one that says why the pod went away.
    await telemetry.shutdown()

Order matters at shutdown

Shut telemetry down before closing clients and pools. The final metric collection runs observable callbacks, and those read live objects — pool stats, keyring stats, bulkhead depths. Tear the clients down first and the last collection reports numbers from a half-disposed object, or raises inside the exporter thread.

The dashboards stay empty until something is actually instrumented — bootstrap_telemetry installs the SDK, it does not emit. Pair it with the instrument_* calls from Observability, and the operations dashboard fills in within one export interval of the first request.

Metric names in Prometheus

The shipped Alloy config sets add_metric_suffixes = false on the Prometheus exporter, which makes the mapping trivial: the OTel name with dots turned into underscores.

OpenTelemetry Prometheus
forze.operations forze_operations
forze.operation.duration forze_operation_duration_bucket / _sum / _count
forze.crypto.cold_miss forze_crypto_cold_miss
process.cpu.time process_cpu_time

Leave suffixes on — the collector default — and the same metrics arrive as forze_operations_total, forze_operation_duration_milliseconds_bucket and process_cpu_time_seconds_total. Both are fine; they are just different, and the shipped dashboards and alert rules assume the first. A unit test checks every expression in those files against the metric constants in src/, which is only possible because the mapping is pinned — and an integration test puts a real Alloy in front of a real Prometheus and asserts the names that actually arrive, so the pinning is verified rather than assumed.

Attribute names follow the same rule: forze.outcome becomes the label forze_outcome. Alloy also derives job from service.name and instance from service.instance.id.

Leave the temporality preference alone

Every rate() in the shipped dashboards and rules assumes cumulative temporality, which is the OTLP exporter's default. Setting OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=delta makes each export carry the interval's delta instead of a running total, and rate() over that is wrong — quietly, with plausible-looking numbers. Delta is the right choice for some backends; it is not the one these assets were written against.

Label discipline

This is the one section worth reading twice, because both indexes here are priced by cardinality and both are easy to ruin from application code.

Loki labels come from container metadata only. tenant_id, trace_id, correlation_id, principal_id stay inside the JSON line and are queried with | json:

{service="orders-api"} | json | tenant_id="acme" | level="error"

Promote tenant_id to a label and Loki's stream count multiplies by your tenant count — an unbounded dimension in the index, for a field you can filter on perfectly well at query time. trace_id is handled separately, as structured metadata, which is what makes the Grafana log-to-trace link work without indexing it.

tenant_id is never a metric label. Not in Forze, not in your own instruments. It is a span attribute and a log field, both of which already carry it, and per-tenant questions are answered from those. A per-tenant metric is a time series per tenant per metric per process, forever — including for tenants that churned out a year ago.

Alert on fields, not on message text. The log sanitizer rewrites message strings to scrub secrets and PII. A LogQL rule matching a substring of a message is matching something the scrubber is allowed to change; match structured fields instead.

Sampling

Two places, and they are not interchangeable.

Head sampling is OTEL_TRACES_SAMPLER in the application. It is cheap and it is blind: the decision happens before anyone knows whether the trace was interesting.

Tail sampling is a commented-out block in config.alloy. The collector has seen the whole trace, so it can keep every error and every slow request and 5% of the rest. That is almost always the policy you want; it costs the collector memory for the decision window.

One volume note specific to Forze: outbound port spans are emitted per retry attempt. A dependency that flaps does not add spans linearly — a policy with five attempts quintuples that route's span volume exactly when the system is least healthy. Tail sampling handles this well (those traces are the interesting ones); a flat head sample does not.

Histogram buckets

bootstrap_telemetry installs views on forze.operation.duration and forze.durable.run.duration with an explicit millisecond ladder — 1, 2, 5, 10, 25, 50, 100, 250, 500, 1 000, 2 500, 5 000, 10 000, 30 000, 60 000.

The SDK's default boundaries are second-oriented. Applied to millisecond values they put every sub-5 ms operation in one bucket and everything over 10 s in the overflow, which makes both the p50 of a fast handler and the p99 of a slow one unreadable.

For Prometheus native histograms, pass exponential_histograms=True. To add your own views without losing these, compose them:

from forze.base.telemetry import bootstrap_telemetry, millisecond_histogram_views

bootstrap_telemetry(
    service_name="orders-api",
    histogram_views=[*millisecond_histogram_views(), *my_views],
)

The views apply only to the provider bootstrap_telemetry creates. An application that installed its own SDK owns its aggregation, and the bootstrap defers to it entirely.

Counter resets

Most of the framework's gauges and counters are observable and cumulative per process — pool stats, keyring stats, L1 stats, realtime counters, signing counters. A restart resets them to zero.

rate() handles resets correctly if it can tell one process from another. That is what service.instance.id is for, and why bootstrap_telemetry mints a fresh UUID per process by default. Override it only with something equally unique: a pod name is not unique across a pre-fork server's workers; a pod name plus worker index is.

This is also why the blessed path is push, not scrape — see below.

Pull metrics, and why not

There is a pull endpoint:

from opentelemetry.exporter.prometheus import PrometheusMetricReader
from forze_fastapi.routes import attach_metrics_route

reader = PrometheusMetricReader()
bootstrap_telemetry(service_name="orders-api", extra_metric_readers=[reader])
attach_metrics_route(router, reader)

As written, it is correct for a single-process deployment and wrong for any other. Under a pre-fork server the scrape is answered by whichever worker accepted the connection, so consecutive scrapes sample different processes: cumulative counters interleave, rate() sees phantom resets, gauges flap between workers. Nothing inside one process can fix that — it is a property of one socket in front of N independent processes. Two setups give one scrape that covers every process, and both keep each process its own series:

Snapshots A collector
Reaches the workers of one host every process that can reach the collector, on any host
Runs inside your FastAPI app as one more container
Names and labels the pull path's: forze_operations_total, service_instance_id the push path's: forze_operations, job, instance
Shipped dashboards and alerts do not apply as they stand apply

Pick snapshots for a pre-fork API on one host that should not grow a service. Pick the collector for replicas on several hosts, for processes that are not web servers, or to use the shipped dashboards without the rest of the Grafana stack.

prometheus_client is deliberately not part of the observability extra; install opentelemetry-exporter-prometheus yourself if you want this route.

Several workers, one scrape

For uvicorn --workers N or gunicorn's uvicorn worker, with a Prometheus and no collector: every worker leaves a snapshot of its own metrics in a directory they all share, and whichever worker takes the scrape answers for all of them. Each series keeps its worker's service.instance.id as the label service_instance_id, and nothing is summed.

from pathlib import Path

from fastapi import APIRouter, FastAPI

from forze.application.execution import ExecutionRuntime, build_runtime
from forze.base.telemetry import bootstrap_telemetry
from forze_fastapi import runtime_lifespan
from forze_fastapi.routes import MetricsSnapshots, attach_metrics_route, worker_metrics_reader


def create_app() -> FastAPI:  # uvicorn orders.app:create_app --factory --workers 4
    reader = worker_metrics_reader()
    snapshots = MetricsSnapshots(directory=Path("/dev/shm/forze-metrics"))

    def build() -> ExecutionRuntime:  # in each worker, after any fork
        bootstrap_telemetry(
            service_name="orders-api", exporter="none", extra_metric_readers=[reader]
        )
        return build_runtime(lifecycle_steps=snapshots.writer_step(reader))

    lifespan = runtime_lifespan(build=build)

    router = APIRouter()
    attach_metrics_route(router, reader, snapshots=snapshots)

    app = FastAPI(lifespan=lifespan)
    app.include_router(router)

    return app

Three pieces, one per process:

  • worker_metrics_reader() builds the PrometheusMetricReader the mode needs: its own registry, and service.instance.id copied onto every series. It needs opentelemetry-exporter-prometheus 0.66b0 or newer and refuses an older one.
  • bootstrap_telemetry runs inside the per-worker build, never at import: it mints the worker's service.instance.id. Gunicorn's --preload runs import-time code once, before the fork, so every worker would share one id, and the route withholds them all. The reader and the MetricsSnapshots value carry no identity, so the factory builds them.
  • snapshots.writer_step(reader) is a lifecycle step, so each worker runs its own. At startup it creates the directory (0700), opens it without following a symlink, checks the open directory — owned by the app's user and writable by no one else — and renders once. From then on every write, sweep and read goes through that open directory, so nobody can swap the path afterwards. It refuses to start on an unsafe directory, or on a reader whose sample lines do not all carry one service_instance_id — a plain PrometheusMetricReader() renders the process collectors, which carry none. It then writes every interval, atomically, from the first render that holds a series: a worker that has recorded nothing yet is not served. On shutdown it writes once more. It shares one render at a time with any plain metrics route on the same reader.
  • attach_metrics_route(..., snapshots=snapshots) serves every fresh snapshot owned by the app's user, merged family by family, and renders nothing itself.
MetricsSnapshots Default
directory required One directory per worker group. A tmpfs path such as /dev/shm/... keeps writes in memory; in a container it is private to that container.
interval 10 s How often each worker writes. Each write renders that worker's metrics once, the cost one scrape has today.
ttl 3 × interval How old a snapshot may be and still be served, by both the host's monotonic clock and the wall clock. Below 2 × interval is refused.
max_snapshot_bytes 16 MiB A larger snapshot is neither written nor served.

Every scrape also carries forze_metrics_snapshot_age_seconds{service_instance_id}, one sample per worker it served: how old that worker's numbers are, on the host's monotonic clock, which a wall-clock step cannot move. A step withholds the old snapshots until each worker writes again, within one interval. The route owns that name: a snapshot family called the same is left out. Query per worker, and sum on the query side:

sum without (service_instance_id) (rate(forze_operations_total[5m]))
count(forze_metrics_snapshot_age_seconds)   # workers served; alert when below the count

Prometheus's own up stays 1 while one worker is missing, so the count is the alert. The route answers 503 when it has nothing to serve: no fresh snapshot at all, or only snapshots it withholds. It withholds two processes that claim the same service.instance.id and logs both pids, because serving either would interleave them into one series. That happens when bootstrap_telemetry runs before the fork, under gunicorn's --preload.

What to know before you rely on it:

  • A restart is a new series. A worker that dies drops out once its snapshot is older than ttl, and its replacement comes back under a new service_instance_id. That is what keeps rate() honest, and it costs one fresh set of series per restart. Worker recycling, such as gunicorn's --max-requests, multiplies it. OTLP push pays the same.
  • One host. The directory joins the workers of one host. Several replicas are several scrape targets, which Prometheus discovers; across hosts without that, push to one collector.
  • The response grows with the workers. About 1 MB per worker for 200 operation label sets, measured. Raise scrape_timeout, or put GZipMiddleware in front of the route.
  • The names differ from the push path's. The pull path renders suffixed names (forze_operations_total, forze_operation_duration_milliseconds_bucket), labels the worker service_instance_id rather than instance, and job is your scrape job. The shipped dashboards and alert rules assume the push names above, so they do not apply as they stand.
  • No exponential histograms. The exporter does not render them: with bootstrap_telemetry(exponential_histograms=True), the duration histograms are missing from every pull scrape, this mode's included. Keep the default millisecond ladder.
  • No process collectors. process_* and python_gc_* carry no instance label, so the reader's own registry leaves them out.
  • A blocked observable callback delays exit. The final render on shutdown gets one interval, then is abandoned, but its thread runs on, and Python waits for that thread before the process exits. A callback that blocks on I/O therefore holds its worker until it returns or the supervisor kills it. A plain route's render behaves the same way. Give a callback that does I/O its own timeout.

One collector, one scrape

Every process pushes OTLP to one OpenTelemetry Collector, and Prometheus scrapes the collector. That reaches processes on any host, and ones that are not web servers, such as an outbox relay. It is the push path above with only its metrics kept: one container, and no Alloy, Loki, Tempo or Grafana.

Download config.yaml, the collector's whole pipeline, and add the collector next to your app:

services:
  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.130.0
    volumes:
      - ./config.yaml:/etc/otelcol-contrib/config.yaml:ro
    ports:
      - "4318:4318"   # OTLP http/protobuf, for processes on other hosts

The receiver takes unauthenticated plaintext: publish 4318 only on a private network, or put TLS and an authenticator in front of it. Prometheus on the same Compose network scrapes otel-collector:8889 without a published port; publish 8889 only for a Prometheus outside it, on the same terms.

Point every process at it, with traces off, since the recipe carries metrics only:

bootstrap_telemetry(service_name="orders-api", traces=False)
# On the Compose network; from another host, use the address 4318 is published on.
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318

Under a pre-fork server, call bootstrap_telemetry in each worker, never at import before the fork, exactly as for snapshots. Workers that share one service.instance.id interleave into one series. Then scrape the collector, and keep the labels it sets:

scrape_configs:
  - job_name: otel-collector
    honor_labels: true
    static_configs:
      - targets: ["otel-collector:8889"]

What to know before you rely on it:

  • The names and labels are the push path's. The config turns name suffixes off, as config.alloy does, so the metric names above hold. Each series carries job from service.name and instance from service.instance.id. honor_labels: true is what keeps them. Without it, Prometheus renames them exported_job and exported_instance, and every by (job) in the shipped dashboards and alert rules lumps all services under the collector's job. With it, those assets apply as they stand.
  • Each process is its own series. Nothing is summed in the collector, and a restart comes back under a new instance. Sum on the query side: sum without (instance) (rate(forze_operations[5m])).
  • up describes the collector. A process that stops pushing keeps its last values until metric_expiration (5 minutes in the config) passes, then drops out. count by (job) (target_info) counts the processes the collector serves. Alert when it falls below the count you run, and when it is absent, which is what a full outage looks like once every process has expired: count by (job) (target_info{job="orders-api"}) < 3 or absent(target_info{job="orders-api"}).
  • Freshness is the export interval plus the scrape interval. Each process pushes every metric_export_interval (15 s by default), and Prometheus reads the latest push on its next scrape. Keep metric_export_interval well under metric_expiration, or a process drops out between pushes.
  • Traces need a pipeline of their own. Without one, the collector answers 404 on /v1/traces. To keep traces, add a traces pipeline to an exporter you run, or use the full stack above.

Probes

Every process gets both probes, including the ones that are not HTTP servers.

from forze_fastapi.routes import attach_liveness_route, attach_readiness_route

attach_liveness_route(router)              # /livez
attach_readiness_route(router, runtime)    # /readyz
from forze.application.execution.background import probe_listener_step

lifecycle_steps.append(probe_listener_step(port=8079))

Stdlib asyncio only — an outbox relay does not grow an HTTP framework to answer two endpoints. Serves the same paths, and answers through the same evaluator.

Kubernetes doctrine:

Probe Path Why
livenessProbe /livez answers 200 while the event loop schedules work — reaching the handler is the check
readinessProbe /readyz ready / draining / unavailable: ready only once startup finished and until a drain begins
startupProbe /readyz same endpoint, generous failureThreshold for slow assembly

The separation is the point. A draining pod is alive but not ready: readiness takes it out of rotation, liveness keeps the kubelet from killing it mid-drain. One endpoint forced to answer both questions cannot express that, and a slow drain gets restarted instead of finished.

Probes are the restart signal. They cannot see a loop that is running but making no progress — for that you keep alerting on the gauges: forze.realtime.backplane .seconds_since_ok, forze.jobs.staleness.scan_age, forze.realtime.mailbox.overflowed. Both, not either.

Dashboards

Four, provisioned into a Forze folder:

Dashboard What it answers
Operations throughput, the error vs failed split, latency distribution, durable runs, job progress
Resilience breaker states, event rates by kind, bulkhead slots in use and queue depth against the limit, hedge delay
Data planes tenant-pool size/capacity/churn, connection-pool occupancy and waiting callers, outbound HTTP requests in flight, KMS round-trips and cache hit ratio, document L1, token signing, process CPU and resident memory
Realtime gateway delivery outcomes, backplane freshness, live-hub subscribers and drops, offline mailbox and its replay size

A row fills once its instrument_* call runs; the metric catalog names the call for each plane. Every panel queries a metric that exists — checked by a unit test against the constants in src/, so a renamed metric fails CI instead of silently emptying a panel.

Alerts

alerts.yml codifies the alarms that until now lived only in docstrings. Each rule carries its reasoning in the annotation, because a threshold without a "what this means, what to do" is a page nobody can action.

Alert The signal
ForzeOperationErrorRateHigh genuine faults only — 4xx-class domain failures are excluded by design
ForzeDurableRecoverySustained runs keep being reclaimed from processes that died holding them
ForzeDurableRunsFailing over 5% of a function's finished runs failed for 5 minutes, so one failed run of a quiet function is enough; cancelled, timed-out and retrying runs are excluded
ForzeJobsStalled stuck jobs or a dead staleness sweep
ForzeCircuitBreakerOpen a route has been shedding for five minutes
ForzeBulkheadSaturated continuous queueing plus rejections
ForzeCryptoColdMiss a path is skipping warm / ensure_unwrapped
ForzeTenantPoolThrash pool creation while the cache sits at capacity
ForzeConnectionPoolExhausted a process has held every connection of a pool for ten minutes
ForzeTokenVerificationFailures rotation gaps, clock skew, or forgeries
ForzeRevocationWriteFailures a revoked credential stays admitted for up to the revocation states' ttl
ForzeRealtimeBackplaneStale cross-node emit is silently down
ForzeRealtimeEmitFailing every Socket.IO emit fails and none is delivered
ForzeRealtimeDeliveriesDropped poisoned or untenanted deliveries
ForzeRealtimeMailboxOverflow a device lost signals it will never see

The or in ForzeJobsStalled is load-bearing

sum by (job, forze_job_kind) (forze_jobs_stalled) > 0
  or max by (job) (forze_jobs_staleness_scan_age) > 300

The stalled gauge reads a cache the sweep fills. If the sweep loop dies, the gauge freezes at its last value — almost always zero — and a rule watching only the count goes green at the exact moment it stops knowing anything. The second clause is what makes silence loud.

Thresholds are starting points. Tune them against your own traffic; the annotations tell you what you are trading.

No rule ships for a leaked HTTP stream: no fixed threshold tells a leak from load. Watch the floor of the Requests in flight panel instead. A floor that rises under steady load and never falls back is a stream or send block that never exits.

Exemplars

Grafana can jump from a histogram bucket straight to a trace, if the SDK attached an exemplar. The shipped datasource leaves exemplarTraceIdDestinations empty on purpose: exemplar behavior at the pinned OpenTelemetry version is not something the framework promises, and a derived link that never resolves is worse than no link at all. Verify it in your own stack first, then enable it.

Log-to-trace correlation needs no such caveat — trace_id is on every JSON line, and the provisioned Loki datasource already links it to Tempo.

Next