Faults, latency & time

A handler that passes every happy-path test still meets a hostile world in production: downstreams error, calls time out, the slow tail of a latency distribution lands mid-transaction. DST lets you declare that environment as data on the config, and applies it at the port boundary — over any resolved port, seeded from the master seed, with the app untouched. The handler makes its ordinary await port calls; the simulator decides what happens at each one.

Faults¶

A FaultPolicy is a set of rules, each matching (surface, route, op) and rolling — per matched call — one of the failure behaviours. Declare it on the config and every matching port call becomes a place a fault can strike:

from forze_dst import SimulationConfig
from forze_dst.faults import FaultPolicy, FaultRule

report = simulation.run(SimulationConfig(
    seeds=range(200),
    faults=FaultPolicy(rules=(
        FaultRule(surface="document_command", error=0.2),   # 20% transient failures
    )),
))

A FaultRule rolls, per matched call: error (a retryable failure), timeout, and crash, plus the transport behaviours drop (silent loss), duplicate (redelivery), and delay (advance virtual time before the call). Faults are seeded by construction — you never pass an RNG, so a 20%-error sweep reproduces exactly. This is how you prove a retry loop actually converges, or that a consumer stays idempotent when the broker redelivers.

Latency¶

A LatencyProfile samples a per-route distribution and advances the virtual clock by the result before each matched call — modelling the time a real downstream takes:

from forze_dst.latency import LatencyProfile, LatencyRule, Exponential

latency = LatencyProfile(rules=(
    LatencyRule(dist=Exponential(mean=0.05), surface="document_command"),
))

The distributions are Constant, Uniform, Exponential, and the heavy-tailed LogNormal and Pareto. The heavy tails matter: their long right edge produces the p99 blow-ups that expose timeout and deadline bugs a fixed delay never reaches — the one call in five hundred that takes ten times the mean is exactly the one that outlives a TTL.

Virtual time¶

Latency and delay advance a virtual clock, not a real one. A workload spanning minutes of simulated time runs in real-wall milliseconds, and time-dependent bugs surface with no sleep anywhere in your handlers.

The reservation example is a pure-time bug — no concurrency at all. confirm checks the hold is still valid, then charges through a slow payment downstream; the charge takes longer than the reservation's TTL, so the hold expires mid-flight yet the confirmation is still written:

@attrs.define(slots=True, kw_only=True)
class _Confirm(Handler[ConfirmCmd, None]):
    ctx: ExecutionContext

    async def __call__(self, args: ConfirmCmd) -> None:
        reservation = await self.ctx.document.query(RESERVATION_SPEC).get(
            args.reservation_id
        )
        if reservation.confirmed_at is not None:  # already confirmed
            return

        # Checked valid *now* — but the decision is made before the charge below.
        if utcnow() >= reservation.expires_at:
            return

        # Charge through the payment downstream — an ordinary port call. The simulator models
        # this call's latency (the payment processor's round-trip), so the virtual clock
        # advances here; in production it is wall-clock minutes. The bug: the hold can expire
        # during the charge, yet we still write the confirmation afterwards.
        await self.ctx.document.command(PAYMENT_SPEC).create(
            PaymentCreate(reservation_id=args.reservation_id)
        )
        await self.ctx.document.command(RESERVATION_SPEC).update(
            args.reservation_id, reservation.rev, ReservationUpdate(confirmed_at=utcnow())
        )

The handler has no idea time is passing unusually. The simulation simply says the payment route is slow, and DST fast-forwards its clock through the charge:

def _latency(surface: str | None, route: str | None, op: str) -> float:
    # The payment downstream is slow; everything else is instant.
    del surface, op
    return AUTH_LATENCY.total_seconds() if route == "payments" else 0.0


async def _observe(ctx: ExecutionContext) -> None:
    page = await ctx.document.query(RESERVATION_SPEC).find_many()
    for reservation in page.hits:
        if reservation.confirmed_at is not None:
            record_event(
                "confirm",
                on_time=reservation.confirmed_at <= reservation.expires_at,
            )


simulation = Simulation(
    operations=registry,
    deps=lambda: MockDepsModule(),
    observe=_observe,
    latency=_latency,
    invariants=[
        expect(
            "confirm",
            lambda event: event.fields["on_time"],
            message="a reservation was confirmed after it expired",
        )
    ],
)

DST drives confirm, advances ten virtual minutes through the charge against a five-minute hold, and the observe invariant catches the confirmation written after expiry — a classic check-then-act-across-time mistake, found without waiting and without flakiness.

Faults and latency perturb a single running process. The next step removes the process entirely — crashes and partitions, where the runtime dies or the network splits.