The Grafana stack
Observability gets your application emitting. This page is the other half: where those signals go, and what you look at when something is wrong. It is a prescription, not an abstraction — the framework ships contracts (JSON on stdout, OTLP, probes) plus these assets, and never ships deployment code.
The wiring picture¶
Your application talks to exactly one thing: Alloy. Everything else is behind it, which is what lets you swap Loki for Elasticsearch or Prometheus for Mimir without touching a service.
your app ──stdout JSON──► Alloy ──► Loki ─┐
──OTLP/http────► Alloy ──► Tempo ─┼──► Grafana
└─► Prometheus (remote-write)
node exporter ─────────────────────► Prometheus (infrastructure; framework-uninvolved)
Three facts make this work without any glue code of yours:
- Metrics and traces already leave through the global OTel providers, so pointing them
somewhere is a matter of installing an SDK — that is all
bootstrap_telemetrydoes. - Logs are already JSON on stdout with
trace_idandspan_idinjected from the active span, so a log-to-trace jump in Grafana is a datasource setting, not a code change. traceparentalready crosses the outbox → broker → inbox boundary, so a trace that starts in an HTTP request continues in the consumer that handles its event.
One docker compose up¶
Download the stack and start it:
- docker-compose.yml — Alloy, Prometheus, Loki, Tempo, Grafana
- config.alloy — the whole ingest pipeline
- prometheus.yml · tempo.yml — backend config
- alerts.yml — the starter rule pack
- provisioning: datasources · dashboards
- dashboards: operations · resilience · data planes · realtime
Keep the layout — the compose file mounts ./provisioning and ./dashboards by path:
grafana/
docker-compose.yml config.alloy prometheus.yml tempo.yml alerts.yml
dashboards/ forze-*.json
provisioning/ datasources/datasources.yml dashboards/dashboards.yml
docker compose up -d
open http://localhost:3000
Grafana comes up with the datasources wired, the four dashboards in a Forze folder, and Prometheus already evaluating the alert rules. Nothing is populated yet, because nothing is pointed at it.
Point your application at it¶
Two calls at process start, before assembly:
from forze.base.logging import bootstrap_logging
from forze.base.telemetry import bootstrap_telemetry
bootstrap_logging(render_mode="json", otel_config={"enable": True})
telemetry = bootstrap_telemetry(
service_name="orders-api",
service_version=APP_VERSION,
)
and one environment variable:
OTEL_EXPORTER_OTLP_ENDPOINT=http://alloy:4318
Endpoint, headers, timeouts and the head sampler all come from the standard OTEL_*
variables; bootstrap_telemetry deliberately adds no second vocabulary for them. Install
the exporter with the observability extra:
uv add "forze[observability]"
Both argument lists are also a settings model, so a deployment configures them from the environment instead of retyping the defaults:
from forze.base.settings import RuntimeSettings
rt = RuntimeSettings(version=APP_VERSION, build_id=BUILD_ID, telemetry="otlp")
bootstrap_logging(level=rt.log_level, render_mode=rt.log_render)
bootstrap_telemetry(
service_name="orders-api",
service_version=rt.full_version,
exporter=rt.telemetry,
)
RuntimeSettings is a plain BaseModel — mount it on your own BaseSettings root, which
owns the environment prefix and delimiter. It defaults log_render to json, unlike
bootstrap_logging itself: a settings object exists because something is being deployed.
Then flush on the way out, after the drain gate flips:
@asynccontextmanager
async def lifespan(app: FastAPI):
async with runtime.scope():
yield
# After the scope drained and tore down, but before the process exits: the last export
# interval is exactly the one that says why the pod went away.
await telemetry.shutdown()
Order matters at shutdown
Shut telemetry down before closing clients and pools. The final metric collection runs observable callbacks, and those read live objects — pool stats, keyring stats, bulkhead depths. Tear the clients down first and the last collection reports numbers from a half-disposed object, or raises inside the exporter thread.
The dashboards stay empty until something is actually instrumented — bootstrap_telemetry
installs the SDK, it does not emit. Pair it with the instrument_* calls from
Observability, and the operations dashboard fills in within one export
interval of the first request.
Metric names in Prometheus¶
The shipped Alloy config sets add_metric_suffixes = false on the Prometheus exporter,
which makes the mapping trivial: the OTel name with dots turned into underscores.
| OpenTelemetry | Prometheus |
|---|---|
forze.operations |
forze_operations |
forze.operation.duration |
forze_operation_duration_bucket / _sum / _count |
forze.crypto.cold_miss |
forze_crypto_cold_miss |
process.cpu.time |
process_cpu_time |
Leave suffixes on — the collector default — and the same metrics arrive as
forze_operations_total, forze_operation_duration_milliseconds_bucket and
process_cpu_time_seconds_total. Both are fine; they are just different, and the shipped
dashboards and alert rules assume the first. A unit test checks every expression in those
files against the metric constants in src/, which is only possible because the mapping is
pinned — and an integration test
puts a real Alloy in front of a real Prometheus and asserts the names that actually
arrive, so the pinning is verified rather than assumed.
Attribute names follow the same rule: forze.outcome becomes the label forze_outcome.
Alloy also derives job from service.name and instance from service.instance.id.
Leave the temporality preference alone
Every rate() in the shipped dashboards and rules assumes cumulative temporality,
which is the OTLP exporter's default. Setting
OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=delta makes each export carry the
interval's delta instead of a running total, and rate() over that is wrong — quietly,
with plausible-looking numbers. Delta is the right choice for some backends; it is not
the one these assets were written against.
Label discipline¶
This is the one section worth reading twice, because both indexes here are priced by cardinality and both are easy to ruin from application code.
Loki labels come from container metadata only. tenant_id, trace_id,
correlation_id, principal_id stay inside the JSON line and are queried with | json:
{service="orders-api"} | json | tenant_id="acme" | level="error"
Promote tenant_id to a label and Loki's stream count multiplies by your tenant count —
an unbounded dimension in the index, for a field you can filter on perfectly well at query
time. trace_id is handled separately, as structured metadata, which is what makes the
Grafana log-to-trace link work without indexing it.
tenant_id is never a metric label. Not in Forze, not in your own instruments. It is
a span attribute and a log field, both of which already carry it, and per-tenant questions
are answered from those. A per-tenant metric is a time series per tenant per metric per
process, forever — including for tenants that churned out a year ago.
Alert on fields, not on message text. The log sanitizer rewrites message strings to scrub secrets and PII. A LogQL rule matching a substring of a message is matching something the scrubber is allowed to change; match structured fields instead.
Sampling¶
Two places, and they are not interchangeable.
Head sampling is OTEL_TRACES_SAMPLER in the application. It is cheap and it is blind:
the decision happens before anyone knows whether the trace was interesting.
Tail sampling is a commented-out block in config.alloy. The collector has seen the
whole trace, so it can keep every error and every slow request and 5% of the rest. That is
almost always the policy you want; it costs the collector memory for the decision window.
One volume note specific to Forze: outbound port spans are emitted per retry attempt. A dependency that flaps does not add spans linearly — a policy with five attempts quintuples that route's span volume exactly when the system is least healthy. Tail sampling handles this well (those traces are the interesting ones); a flat head sample does not.
Histogram buckets¶
bootstrap_telemetry installs views on forze.operation.duration and
forze.durable.run.duration with an explicit millisecond ladder — 1, 2, 5, 10, 25, 50,
100, 250, 500, 1 000, 2 500, 5 000, 10 000, 30 000, 60 000.
The SDK's default boundaries are second-oriented. Applied to millisecond values they put every sub-5 ms operation in one bucket and everything over 10 s in the overflow, which makes both the p50 of a fast handler and the p99 of a slow one unreadable.
For Prometheus native histograms, pass exponential_histograms=True. To add your own
views without losing these, compose them:
from forze.base.telemetry import bootstrap_telemetry, millisecond_histogram_views
bootstrap_telemetry(
service_name="orders-api",
histogram_views=[*millisecond_histogram_views(), *my_views],
)
The views apply only to the provider bootstrap_telemetry creates. An application that
installed its own SDK owns its aggregation, and the bootstrap defers to it entirely.
Counter resets¶
Most of the framework's gauges and counters are observable and cumulative per process — pool stats, keyring stats, L1 stats, realtime counters, signing counters. A restart resets them to zero.
rate() handles resets correctly if it can tell one process from another. That is what
service.instance.id is for, and why bootstrap_telemetry mints a fresh UUID per process
by default. Override it only with something equally unique: a pod name is not unique
across a pre-fork server's workers; a pod name plus worker index is.
This is also why the blessed path is push, not scrape — see below.
Pull metrics, and why not¶
There is a pull endpoint:
from opentelemetry.exporter.prometheus import PrometheusMetricReader
from forze_fastapi.routes import attach_metrics_route
reader = PrometheusMetricReader()
bootstrap_telemetry(service_name="orders-api", extra_metric_readers=[reader])
attach_metrics_route(router, reader)
As written, it is correct for a single-process deployment and wrong for any other.
Under a pre-fork server the scrape is answered by whichever worker accepted the connection,
so consecutive scrapes sample different processes: cumulative counters interleave, rate()
sees phantom resets, gauges flap between workers. Nothing inside one process can fix that —
it is a property of one socket in front of N independent processes. Two setups give one
scrape that covers every process, and both keep each process its own series:
| Snapshots | A collector | |
|---|---|---|
| Reaches | the workers of one host | every process that can reach the collector, on any host |
| Runs | inside your FastAPI app | as one more container |
| Names and labels | the pull path's: forze_operations_total, service_instance_id |
the push path's: forze_operations, job, instance |
| Shipped dashboards and alerts | do not apply as they stand | apply |
Pick snapshots for a pre-fork API on one host that should not grow a service. Pick the collector for replicas on several hosts, for processes that are not web servers, or to use the shipped dashboards without the rest of the Grafana stack.
prometheus_client is deliberately not part of the observability extra; install
opentelemetry-exporter-prometheus yourself if you want this route.
Several workers, one scrape¶
For uvicorn --workers N or gunicorn's uvicorn worker, with a Prometheus and no
collector: every worker leaves a snapshot of its own metrics in a directory they all share,
and whichever worker takes the scrape answers for all of them. Each series keeps its
worker's service.instance.id as the label service_instance_id, and nothing is summed.
from pathlib import Path
from fastapi import APIRouter, FastAPI
from forze.application.execution import ExecutionRuntime, build_runtime
from forze.base.telemetry import bootstrap_telemetry
from forze_fastapi import runtime_lifespan
from forze_fastapi.routes import MetricsSnapshots, attach_metrics_route, worker_metrics_reader
def create_app() -> FastAPI: # uvicorn orders.app:create_app --factory --workers 4
reader = worker_metrics_reader()
snapshots = MetricsSnapshots(directory=Path("/dev/shm/forze-metrics"))
def build() -> ExecutionRuntime: # in each worker, after any fork
bootstrap_telemetry(
service_name="orders-api", exporter="none", extra_metric_readers=[reader]
)
return build_runtime(lifecycle_steps=snapshots.writer_step(reader))
lifespan = runtime_lifespan(build=build)
router = APIRouter()
attach_metrics_route(router, reader, snapshots=snapshots)
app = FastAPI(lifespan=lifespan)
app.include_router(router)
return app
Three pieces, one per process:
worker_metrics_reader()builds thePrometheusMetricReaderthe mode needs: its own registry, andservice.instance.idcopied onto every series. It needsopentelemetry-exporter-prometheus0.66b0 or newer and refuses an older one.bootstrap_telemetryruns inside the per-worker build, never at import: it mints the worker'sservice.instance.id. Gunicorn's--preloadruns import-time code once, before the fork, so every worker would share one id, and the route withholds them all. The reader and theMetricsSnapshotsvalue carry no identity, so the factory builds them.snapshots.writer_step(reader)is a lifecycle step, so each worker runs its own. At startup it creates the directory (0700), opens it without following a symlink, checks the open directory — owned by the app's user and writable by no one else — and renders once. From then on every write, sweep and read goes through that open directory, so nobody can swap the path afterwards. It refuses to start on an unsafe directory, or on a reader whose sample lines do not all carry oneservice_instance_id— a plainPrometheusMetricReader()renders the process collectors, which carry none. It then writes everyinterval, atomically, from the first render that holds a series: a worker that has recorded nothing yet is not served. On shutdown it writes once more. It shares one render at a time with any plain metrics route on the same reader.attach_metrics_route(..., snapshots=snapshots)serves every fresh snapshot owned by the app's user, merged family by family, and renders nothing itself.
MetricsSnapshots |
Default | |
|---|---|---|
directory |
required | One directory per worker group. A tmpfs path such as /dev/shm/... keeps writes in memory; in a container it is private to that container. |
interval |
10 s | How often each worker writes. Each write renders that worker's metrics once, the cost one scrape has today. |
ttl |
3 × interval |
How old a snapshot may be and still be served, by both the host's monotonic clock and the wall clock. Below 2 × interval is refused. |
max_snapshot_bytes |
16 MiB | A larger snapshot is neither written nor served. |
Every scrape also carries forze_metrics_snapshot_age_seconds{service_instance_id}, one
sample per worker it served: how old that worker's numbers are, on the host's monotonic
clock, which a wall-clock step cannot move. A step withholds the old snapshots until each
worker writes again, within one interval. The route owns that name: a snapshot family
called the same is left out. Query per worker, and sum on the query side:
sum without (service_instance_id) (rate(forze_operations_total[5m]))
count(forze_metrics_snapshot_age_seconds) # workers served; alert when below the count
Prometheus's own up stays 1 while one worker is missing, so the count is the alert. The
route answers 503 when it has nothing to serve: no fresh snapshot at all, or only
snapshots it withholds. It withholds two processes that claim the same service.instance.id
and logs both pids, because serving either would interleave them into one series. That
happens when bootstrap_telemetry runs before the fork, under gunicorn's --preload.
What to know before you rely on it:
- A restart is a new series. A worker that dies drops out once its snapshot is older
than
ttl, and its replacement comes back under a newservice_instance_id. That is what keepsrate()honest, and it costs one fresh set of series per restart. Worker recycling, such as gunicorn's--max-requests, multiplies it. OTLP push pays the same. - One host. The directory joins the workers of one host. Several replicas are several scrape targets, which Prometheus discovers; across hosts without that, push to one collector.
- The response grows with the workers. About 1 MB per worker for 200 operation label
sets, measured. Raise
scrape_timeout, or putGZipMiddlewarein front of the route. - The names differ from the push path's. The pull path renders suffixed names
(
forze_operations_total,forze_operation_duration_milliseconds_bucket), labels the workerservice_instance_idrather thaninstance, andjobis your scrape job. The shipped dashboards and alert rules assume the push names above, so they do not apply as they stand. - No exponential histograms. The exporter does not render them: with
bootstrap_telemetry(exponential_histograms=True), the duration histograms are missing from every pull scrape, this mode's included. Keep the default millisecond ladder. - No process collectors.
process_*andpython_gc_*carry no instance label, so the reader's own registry leaves them out. - A blocked observable callback delays exit. The final render on shutdown gets one
interval, then is abandoned, but its thread runs on, and Python waits for that thread before the process exits. A callback that blocks on I/O therefore holds its worker until it returns or the supervisor kills it. A plain route's render behaves the same way. Give a callback that does I/O its own timeout.
One collector, one scrape¶
Every process pushes OTLP to one OpenTelemetry Collector, and Prometheus scrapes the collector. That reaches processes on any host, and ones that are not web servers, such as an outbox relay. It is the push path above with only its metrics kept: one container, and no Alloy, Loki, Tempo or Grafana.
Download config.yaml, the collector's whole pipeline, and add the collector next to your app:
services:
otel-collector:
image: otel/opentelemetry-collector-contrib:0.130.0
volumes:
- ./config.yaml:/etc/otelcol-contrib/config.yaml:ro
ports:
- "4318:4318" # OTLP http/protobuf, for processes on other hosts
The receiver takes unauthenticated plaintext: publish 4318 only on a private network, or
put TLS and an authenticator in front of it. Prometheus on the same Compose network scrapes
otel-collector:8889 without a published port; publish 8889 only for a Prometheus outside
it, on the same terms.
Point every process at it, with traces off, since the recipe carries metrics only:
bootstrap_telemetry(service_name="orders-api", traces=False)
# On the Compose network; from another host, use the address 4318 is published on.
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
Under a pre-fork server, call bootstrap_telemetry in each worker, never at import before
the fork, exactly as for snapshots. Workers that share one service.instance.id interleave
into one series. Then scrape the collector, and keep the labels it sets:
scrape_configs:
- job_name: otel-collector
honor_labels: true
static_configs:
- targets: ["otel-collector:8889"]
What to know before you rely on it:
- The names and labels are the push path's. The config turns name suffixes off, as
config.alloydoes, so the metric names above hold. Each series carriesjobfromservice.nameandinstancefromservice.instance.id.honor_labels: trueis what keeps them. Without it, Prometheus renames themexported_jobandexported_instance, and everyby (job)in the shipped dashboards and alert rules lumps all services under the collector's job. With it, those assets apply as they stand. - Each process is its own series. Nothing is summed in the collector, and a restart
comes back under a new
instance. Sum on the query side:sum without (instance) (rate(forze_operations[5m])). updescribes the collector. A process that stops pushing keeps its last values untilmetric_expiration(5 minutes in the config) passes, then drops out.count by (job) (target_info)counts the processes the collector serves. Alert when it falls below the count you run, and when it is absent, which is what a full outage looks like once every process has expired:count by (job) (target_info{job="orders-api"}) < 3 or absent(target_info{job="orders-api"}).- Freshness is the export interval plus the scrape interval. Each process pushes every
metric_export_interval(15 s by default), and Prometheus reads the latest push on its next scrape. Keepmetric_export_intervalwell undermetric_expiration, or a process drops out between pushes. - Traces need a pipeline of their own. Without one, the collector answers
404on/v1/traces. To keep traces, add a traces pipeline to an exporter you run, or use the full stack above.
Probes¶
Every process gets both probes, including the ones that are not HTTP servers.
from forze_fastapi.routes import attach_liveness_route, attach_readiness_route
attach_liveness_route(router) # /livez
attach_readiness_route(router, runtime) # /readyz
from forze.application.execution.background import probe_listener_step
lifecycle_steps.append(probe_listener_step(port=8079))
Stdlib asyncio only — an outbox relay does not grow an HTTP framework to answer two
endpoints. Serves the same paths, and answers through the same evaluator.
Kubernetes doctrine:
| Probe | Path | Why |
|---|---|---|
livenessProbe |
/livez |
answers 200 while the event loop schedules work — reaching the handler is the check |
readinessProbe |
/readyz |
ready / draining / unavailable: ready only once startup finished and until a drain begins |
startupProbe |
/readyz |
same endpoint, generous failureThreshold for slow assembly |
The separation is the point. A draining pod is alive but not ready: readiness takes it out of rotation, liveness keeps the kubelet from killing it mid-drain. One endpoint forced to answer both questions cannot express that, and a slow drain gets restarted instead of finished.
Probes are the restart signal. They cannot see a loop that is running but making no
progress — for that you keep alerting on the gauges: forze.realtime.backplane
.seconds_since_ok, forze.jobs.staleness.scan_age, forze.realtime.mailbox.overflowed.
Both, not either.
Dashboards¶
Four, provisioned into a Forze folder:
| Dashboard | What it answers |
|---|---|
| Operations | throughput, the error vs failed split, latency distribution, durable runs, job progress |
| Resilience | breaker states, event rates by kind, bulkhead slots in use and queue depth against the limit, hedge delay |
| Data planes | tenant-pool size/capacity/churn, connection-pool occupancy and waiting callers, outbound HTTP requests in flight, KMS round-trips and cache hit ratio, document L1, token signing, process CPU and resident memory |
| Realtime | gateway delivery outcomes, backplane freshness, live-hub subscribers and drops, offline mailbox and its replay size |
A row fills once its instrument_* call runs; the metric catalog
names the call for each plane. Every panel queries a metric that exists — checked by a unit
test against the constants in src/, so a renamed metric fails CI instead of silently
emptying a panel.
Alerts¶
alerts.yml codifies the alarms that until now lived only in
docstrings. Each rule carries its reasoning in the annotation, because a threshold without
a "what this means, what to do" is a page nobody can action.
| Alert | The signal |
|---|---|
ForzeOperationErrorRateHigh |
genuine faults only — 4xx-class domain failures are excluded by design |
ForzeDurableRecoverySustained |
runs keep being reclaimed from processes that died holding them |
ForzeDurableRunsFailing |
over 5% of a function's finished runs failed for 5 minutes, so one failed run of a quiet function is enough; cancelled, timed-out and retrying runs are excluded |
ForzeJobsStalled |
stuck jobs or a dead staleness sweep |
ForzeCircuitBreakerOpen |
a route has been shedding for five minutes |
ForzeBulkheadSaturated |
continuous queueing plus rejections |
ForzeCryptoColdMiss |
a path is skipping warm / ensure_unwrapped |
ForzeTenantPoolThrash |
pool creation while the cache sits at capacity |
ForzeConnectionPoolExhausted |
a process has held every connection of a pool for ten minutes |
ForzeTokenVerificationFailures |
rotation gaps, clock skew, or forgeries |
ForzeRevocationWriteFailures |
a revoked credential stays admitted for up to the revocation states' ttl |
ForzeRealtimeBackplaneStale |
cross-node emit is silently down |
ForzeRealtimeEmitFailing |
every Socket.IO emit fails and none is delivered |
ForzeRealtimeDeliveriesDropped |
poisoned or untenanted deliveries |
ForzeRealtimeMailboxOverflow |
a device lost signals it will never see |
The or in ForzeJobsStalled is load-bearing
sum by (job, forze_job_kind) (forze_jobs_stalled) > 0
or max by (job) (forze_jobs_staleness_scan_age) > 300
The stalled gauge reads a cache the sweep fills. If the sweep loop dies, the gauge freezes at its last value — almost always zero — and a rule watching only the count goes green at the exact moment it stops knowing anything. The second clause is what makes silence loud.
Thresholds are starting points. Tune them against your own traffic; the annotations tell you what you are trading.
No rule ships for a leaked HTTP stream: no fixed threshold tells a leak from load. Watch
the floor of the Requests in flight panel instead. A floor that rises under steady
load and never falls back is a stream or send block that never exits.
Exemplars¶
Grafana can jump from a histogram bucket straight to a trace, if the SDK attached an
exemplar. The shipped datasource leaves exemplarTraceIdDestinations empty on purpose:
exemplar behavior at the pinned OpenTelemetry version is not something the framework
promises, and a derived link that never resolves is worse than no link at all. Verify it
in your own stack first, then enable it.
Log-to-trace correlation needs no such caveat — trace_id is on every JSON line, and the
provisioned Loki datasource already links it to Tempo.
Next¶
- Metric catalog — every metric, its labels, and the alert that matters
- Observability — what is emitted and how to instrument it
- Shutdown and fleets — the drain window the readiness probe reports on