Overview

A concurrency bug reproduces once in a thousand runs, never in the debugger, and never the same way twice. The race that double-charges a customer, the retry that applies a message twice, the timeout that fires mid-transaction — they live in the interleavings and failures a test suite never schedules. You cannot fix what you cannot reproduce.

Deterministic Simulation Testing (DST) makes those runs reproducible. It takes your real operations, runs them concurrently on a virtual-time event loop, and explores the orderings, faults, and delays a production system would hit — then, when an invariant breaks, hands back the smallest workload that still breaks it, replayable forever from a single integer.

Nothing in your app changes. Handlers talk to ports exactly as they do in production; the simulation lives entirely on the test side.

Point it at your app¶

A Simulation needs three things: your operation registry, a deps factory (one MockDepsModule() auto-mocks every port, built fresh per run so each starts clean), and the invariants that must hold. An optional observe hook records the domain facts the invariants read.

async def _observe(ctx: ExecutionContext) -> None:
    payments = await ctx.document.query(PAYMENT_SPEC).count()
    record_event("payments", total=payments)


simulation = Simulation(
    operations=registry,
    deps=lambda: MockDepsModule(domain_events=_EVENTS),
    observe=_observe,
    invariants=[
        expect(
            "payments",
            lambda event: event.fields["total"] <= 1,
            message="an order was charged more than once",
        )
    ],
)

Then sweep seeds:

from forze_dst import SimulationConfig

report = simulation.run(SimulationConfig(seeds=range(64)))

if report is not None:
    print(report.format())  # a readable, reproducible counterexample

run builds a meaningful workload from your operation catalog (an arrange→act scenario — forze dst derive prints it), runs it under perturbed interleavings, and checks the invariants. On the first violating seed it minimises the workload to a 1-minimal set that still fails and returns a ViolationReport; a clean sweep returns None. There is nothing to assert about how — point it at the app and go.

Reach for a preset instead of hand-tuning the config — SimulationConfig.quick() while iterating, .thorough() before you ship (see Exploration strategies).

The app under test here is ordinary Forze code. pay_order charges, then flips the order to paid — but it charges before the optimistic-concurrency-guarded transition, so two concurrent payments both charge:

@attrs.define(slots=True, kw_only=True)
class _PayOrder(Handler[PayCmd, None]):
    ctx: ExecutionContext

    async def __call__(self, args: PayCmd) -> None:
        order = await self.ctx.document.query(ORDER_SPEC).get(args.order_id)
        if order is None or order.paid:  # pyright: ignore[reportUnnecessaryComparison]
            return

        # BUG: the charge (a side effect) happens *before* the optimistic-concurrency-guarded
        # transition. Two payments that interleave at the port boundaries above both read
        # rev=0 and both create a payment row; the rev guard then lets only one ``update``
        # win (the other conflicts) — but both already charged. The fix is to update first
        # and charge only once the guarded write succeeds. (No artificial yield — the real
        # ``await`` port calls are the interleaving points under simulation.)
        await self.ctx.document.command(PAYMENT_SPEC).create(
            PaymentCreate(order_id=args.order_id)
        )
        await self.ctx.document.command(ORDER_SPEC).update(
            args.order_id, order.rev, OrderUpdate(paid=True)
        )
        # Emitting a domain event triggers the registered handler (a saga-style cascade).
        await DomainDeps(ctx=self.ctx)().dispatch([OrderPaid(order_id=args.order_id)])

DST finds the race, shrinks it to two contending payments, and reports the seed that reproduces it — no sleep, no thread choreography, no luck.

What DST gives you¶

The harness is one small facade — run, coverage, audit, coverage_guided — over a layered engine. Each page below takes one capability from "I pointed it at my app" to "I understand and can operate it":

Forze passes its own simulation¶

DST is judged by the bugs it finds in real systems — so Forze runs its own distributed machinery through it. Each scenario pairs a safety invariant (must always hold) with a reachability target (must sometimes fire), so a green result means the property was tested against the hard case, not a quiet run:

Each scenario also keeps a broken twin — drop the lock, ignore the remote stamp — that the oracle catches, minimises, and reproduces, so the test proves it can still fail. That is the bar: the framework's own concurrency code is continuously simulation-tested, and app authors inherit the same harness for free.

The mock matches the real engine¶

Every invariant DST proves holds against the mock port, not against Postgres or Mongo — so a sweep is only as trustworthy as the mock's fidelity. For the property that worries people most, transactional isolation, Forze closes that gap with a differential conformance battery.

forze_dst.conformance ships the classic isolation anomalies — dirty read, read skew, write skew, the three-transaction read-only anomaly, predicate phantoms — as deterministic forced interleavings, each with a known verdict per isolation level. The same battery runs against the in-memory mock and, over testcontainers, against real Postgres (every level) and Mongo (snapshot):

from forze.application.contracts.transaction import IsolationLevel
from forze_dst.conformance import BATTERY, expected_verdict

# `backend` is a ConformanceBackend — N independent sessions over one shared store
level = IsolationLevel.SERIALIZABLE
for case in BATTERY:
    assert await case.run(backend, level) == expected_verdict(case, level)

Each anomaly's outcome is normalised to permitted or prevented, so the differential compares the behaviour at the declared level — never the mechanism, the error code, or which transaction lost. The handful of expected differences (Forze's revision guard prevents lost update at every level, for one) live in a reviewed CONTRACT_STRENGTHENINGS / MECHANISM_DIVERGENCES catalog, so a real divergence stands out instead of drowning in noise. A green battery means the mock and the real engine agree on the anomaly — which is what lets "it passed on the mock" mean "it matches the real engine."

Start with what must hold — invariants are the lens through which every other capability reports a bug.