feat: checkpoint batching and release-gate stories

.scratch/distributed-gguf-runtime/evidence/DGR-013/README.md

@@ -0,0 +1,223 @@
+# DGR-013 — Harden failure, cancellation, and restart semantics: evidence
+
+Status: done
+Date: 2026-07-16
+Evidence kind: **synthetic-unit** (pure-numpy KV-cached dense-Llama reference +
+node-local hardened stream). No model download, no GPU, no torch, no network, no
+API credit.
+
+## Summary
+
+Implemented bounded, explicit failure/cancellation/restart semantics for the
+per-Route-Session decode stream, layered on the DGR-007 Hot KV State manager
+(isolated `(session, epoch)` KV) and the DGR-012 continuous-batch scheduler. The
+goal (RALPH product objective) is that distributed speed never comes with hanging
+or corrupted generations: every blocked op is bounded, every cancel frees state,
+duplicate steps are idempotent, uncertain mutations are never silently replayed,
+alpha failover restarts from token zero, and billing distinguishes what actually
+completed.
+
+Everything runs against the same deterministic numpy dense-Llama reference the
+default gate uses (`tests/test_hot_kv_state.py::_KvDenseLlama` / `_KvReferenceShard`),
+so the whole failure matrix is deterministic, download-free, GPU-free, and
+API-credit-free while exercising the **real** KV isolation path
+(`KvBoundaryAdapter` + `HotKvStateManager`). The pinned llama.cpp worker (DGR-008)
+implements the identical adapter contract, so the semantics carry over to native
+execution unchanged.
+
+### What was built (`packages/node/meshnet_node/failure_semantics.py`, new)
+
+- **`DeadlineGuard` + `StreamTerminated`** — bounds every step against an absolute
+  deadline and a heartbeat-timeout on an injected clock. A reached deadline or a
+  lost heartbeat (peer health loss) raises `StreamTerminated(kind)` so a blocked
+  stream terminates instead of hanging. (**AC: deadlines/heartbeat terminate
+  blocked ops.**)
+- **`CancellationToken`, `ShardCancellationGroup`, `CancellationOutcome`** — one
+  cancel fans across **every** node-local Shard of a Route Session, releasing the
+  `(session, epoch)` KV on each shard's manager and invoking every queued-buffer
+  release callback (the pending activation bundles). Idempotent. The DGR-012
+  scheduler also gains a `cancel()` that drops queued/active work on this node and
+  frees its KV. (**AC: cancellation propagates across every Shard, releases KV +
+  queued buffers.**)
+- **`IdempotencyLedger`, `StepKey`, `StepDisposition`, `UncertainMutationError`** —
+  records each committed `(session, epoch, step)`; a duplicate delivery returns the
+  recorded token with no re-mutation. A step whose mutation outcome is *uncertain*
+  (worker died mid-step) is marked uncertain and can **never** be replayed
+  silently — `begin()` on an uncertain (or still in-flight) step raises
+  `UncertainMutationError`, forcing verify-or-restart. (**AC: duplicate steps
+  idempotent; uncertain mutations never replayed silently.**)
+- **`RestartController`** — alpha failover: opens the *next* route epoch, releases
+  every shard's prior-epoch KV, and `assert_fresh_start` fails closed if any shard
+  still holds new-epoch KV. The restart re-prefills the whole prompt from token
+  zero; the failed epoch becomes stale (KV manager rejects it). Unverified KV is
+  never migrated (RALPH runtime decision #14). (**AC: alpha failover restarts from
+  token zero rather than importing unverified KV.**)
+- **`WorkStatus`, `WorkRecord`, `WorkLedger`** — a typed per-attempt work record
+  with four distinct statuses: `completed`, `cancelled`, `failed`, `unverified`.
+  Only `completed` records are billable; cancelled/failed/unverified tokens are
+  recorded for observability but never charged. JSON-safe for the tracker billing
+  handoff (`packages/tracker/meshnet_tracker/billing.py` charges only completed,
+  verified work). (**AC: billing/work records distinguish completed/cancelled/
+  failed/unverified.**)
+- **`HardenedSessionRunner`** — composes all of the above to drive one session's
+  prefill+decode through the adapter under a deadline/heartbeat guard + cancel
+  token, records the typed outcome, and `run_with_failover` restarts a transient
+  failure from token zero on a fresh epoch.
+- **`FailureKind` + `classify_exception` + `work_status_for`** — stable-string
+  classification of worker death, stream reset, malformed bundle, stale epoch,
+  cache miss, deadline, heartbeat loss, and cancel, plus the failure→billing-status
+  mapping. Suitable for the native protocol's structured status.
+
+### Scheduler extension (`packages/node/meshnet_node/batch_scheduler.py`, DGR-012 file, additive)
+
+Purely additive so the DGR-012 gate stays green (16/16):
+- `DoneReason.CANCELLED` / `DoneReason.FAILED` terminal reasons.
+- `ContinuousBatchScheduler.cancel(session_id, *, reason)` — drops a queued
+  session from the bounded queue or releases an active session's KV, moving it to
+  the done set with a non-completed reason (never counted as completed work).
+- `SchedulerTelemetry.cancelled_sessions` / `failed_sessions` counters.
+
+## Files changed
+
+- `packages/node/meshnet_node/failure_semantics.py` — new module (the whole
+  failure/cancel/restart layer above).
+- `packages/node/meshnet_node/batch_scheduler.py` — additive `cancel()` + two
+  `DoneReason` members + two telemetry counters (DGR-012 file; its 16 tests still
+  pass unchanged).
+- `tests/test_failure_semantics.py` — new, 22 tests (matrix below); reuses the
+  DGR-007 numpy reference via `from test_hot_kv_state import _KvDenseLlama,
+  _KvReferenceShard`.
+- `.scratch/distributed-gguf-runtime/evidence/DGR-013/` — this README,
+  `commands.txt`, `generate_evidence.py`, `results.json`.
+- `.ralph-tui/progress.md` — appended the DGR-013 note.
+- `.scratch/distributed-gguf-runtime/issues/13-...md` — set `Status: done`.
+
+## Acceptance criteria → evidence
+
+| Criterion | Tests (`tests/test_failure_semantics.py`) |
+|---|---|
+| Deadlines/heartbeat loss terminate blocked stream ops | `test_deadline_terminates_a_blocked_stream_and_releases_kv`, `test_heartbeat_loss_terminates_a_blocked_stream`, `test_deadline_guard_reports_remaining_and_resets_on_heartbeat` |
+| Cancellation propagates across every Shard, releases KV + queued buffers | `test_cancellation_token_terminates_stream_and_releases_kv`, `test_shard_cancellation_group_releases_every_shard_and_queued_buffers`, `test_scheduler_cancel_drains_queue_and_releases_active_kv`, `test_scheduler_cancel_rejects_a_completed_reason` |
+| Duplicate steps idempotent; uncertain mutations never replayed silently | `test_duplicate_step_delivery_is_idempotent_no_remutation`, `test_idempotent_run_replays_tokens_without_advancing_kv`, `test_uncertain_mutation_is_never_replayed_silently`, `test_in_flight_duplicate_is_treated_as_uncertain` |
+| Alpha failover restarts from token zero, no unverified KV import | `test_alpha_failover_restarts_from_token_zero_and_completes`, `test_failover_refuses_to_import_unverified_kv`, `test_non_restartable_failure_is_not_retried` |
+| Worker death, stream reset, malformed bundle, stale epoch, cache miss | `test_worker_death_midstream_is_unverified_and_marks_step_uncertain`, `test_stream_reset_is_restartable_failure`, `test_malformed_bundle_is_classified_and_does_not_corrupt_kv`, `test_stale_epoch_reference_is_rejected_and_classified`, `test_cache_miss_midstream_is_restartable` |
+| Billing/work records distinguish completed/cancelled/failed/unverified | `test_work_ledger_distinguishes_all_four_statuses`, `test_work_status_and_classification_mapping`, plus the clean-run billability check `test_clean_run_matches_stateless_reference_and_is_billable` |
+
+## Failure matrix (real, deterministic — `results.json`)
+
+Generated by `generate_evidence.py` against the numpy dense-Llama (prompt `[7,3,9,1]`,
+8 new tokens):
+
+| scenario | status | failure_kind | tokens | restartable | KV released |
+|---|---|---|---|---|---|
+| clean | completed | — | 8 | — | (held, then reaped) |
+| deadline | failed | deadline-exceeded | 2 | no | yes |
+| heartbeat_loss | failed | heartbeat-lost | 3 | no | yes |
+| cancel | cancelled | cancelled | 3 | no | yes |
+| worker_death | unverified | worker-death | 3 | yes | yes |
+| stream_reset | failed | stream-reset | — | yes | yes |
+| stale_epoch | failed | stale-epoch | — | no | (never opened) |
+| cache_miss | failed | cache-miss | 4 | yes | (already evicted) |
+| alpha_failover | **completed** (epoch 1) | — | 8 | — | old epoch stale |
+
+Alpha failover: attempt 0 (epoch 0) dies mid-step → `unverified`; the controller
+advances to epoch 1, drops epoch-0 KV, and the restart re-prefills from token zero
+→ `completed`, reproducing the byte-identical stateless reference. The old epoch is
+now stale (a reference to it raises `StaleRouteEpochError`). Work ledger:
+`{completed: 2, cancelled: 1, failed: 0, unverified: 2}`, `billable_tokens = 16`
+(only the two completed streams — the failover restart and the clean run — are
+billed; the cancelled and the two unverified attempts are not).
+
+## Commands and real results
+
+See `commands.txt`. Key results:
+
+```
+tests/test_failure_semantics.py     -> 22 passed
+tests/test_batch_scheduler.py       -> 16 passed (DGR-012 unchanged)
+tests/test_hot_kv_state.py          -> 22 passed (DGR-007)
+tests/test_gguf_backend.py          ->  2 passed (DGR-009)
+python -m compileall -q packages tests -> exit 0
+git diff --check                       -> exit 0
+python -m pytest -q                    -> 16 failed, 792 passed, 14 skipped in 253.93s
+```
+
+## Full-suite baseline (pre-existing, unrelated failures)
+
+The 16 failures are **pre-existing and unrelated to DGR-013**. None import
+`failure_semantics` or `batch_scheduler`; they live in the tracker/control-plane,
+node-startup, doctor, calibration, and route-benchmark suites and fail on the
+model-download / control-plane / recipe-admission paths (e.g.
+`UnsupportedRecipeParam: worker_transport` from the DGR-009 native recipe against
+the Torch backend, and Torch/HF-model startup that this deterministic sandbox does
+not provide). Removing the two DGR-013 files and re-running the failing tests
+reproduces the identical failures (see `commands.txt`, 4-test spot check → same
+4 failures), so DGR-013 introduces no new failure.
+
+Exact failing set (16):
+
+```
+tests/test_dynamic_routing.py::test_admin_can_replace_a_served_model_and_release_it
+tests/test_manual_route_benchmark.py::test_pinned_route_uses_named_node
+tests/test_manual_route_benchmark.py::test_unknown_route_node_is_400
+tests/test_manual_route_benchmark.py::test_invalid_route_shape_is_400
+tests/test_manual_route_benchmark.py::test_clients_without_route_are_unaffected
+tests/test_manual_route_benchmark.py::test_benchmark_records_one_and_two_node_routes
+tests/test_node_doctor.py::test_the_shipped_recipes_are_all_applicable_by_the_backend
+tests/test_node_doctor.py::test_cli_doctor_flags_select_what_is_validated
+tests/test_node_startup.py::test_preset_model_with_hf_repo_loads_torch_backend
+tests/test_node_startup.py::test_real_model_startup_registers_downloaded_inventory_without_checksum
+tests/test_toploc_calibration_dispatch.py::test_calibration_run_dispatches_only_solo_capable_nodes
+tests/test_toploc_calibration_dispatch.py::test_calibration_run_persists_corpus_and_results_endpoint_reports_it
+tests/test_toploc_calibration_dispatch.py::test_calibration_run_node_without_commitment_endpoint_is_skipped_not_failed
+tests/test_tracker_capability_admission.py::test_an_enforcing_tracker_never_routes_a_node_whose_proof_does_not_cover_it[invalid]
+tests/test_tracker_routing.py::test_torch_node_applies_tracker_load_shard_directive
+tests/test_tracker_routing.py::test_shard_heal_cycle_surviving_node_covers_dead_peers_gap
+```
+
+## Limitations and deferred work
+
+- **Synthetic-unit, not real weights.** Semantics are exercised against the
+  deterministic numpy dense-Llama, not a downloaded GGUF, to keep the default gate
+  deterministic/download-free/GPU-free. Real worker-death/stream-reset behavior on
+  a live llama.cpp worker over gRPC belongs to DGR-008/DGR-010 (DGR-010 is blocked
+  — no certified dense-Llama artifact on this machine; see
+  `evidence/DGR-010/BLOCKED.md`).
+- **Single-node per-session stream.** `HardenedSessionRunner` drives one full-shard
+  session (the node-local case); multi-node cancellation is modelled by
+  `ShardCancellationGroup` fanning across each node's KV manager. The cross-node
+  propagation *transport* (cancel frames over gRPC/relay) is the native protocol's
+  job (DGR-002/008); this story owns the local release + record semantics the
+  transport triggers.
+- **Fault injection is deterministic.** Worker death is a shard that raises on the
+  Nth step; stream reset / deadline / heartbeat are injected via an explicit clock
+  and hook. This is what makes the matrix reproducible; live fault behavior is a
+  native/real-hardware property.
+- **Greedy sampling only.** Reuses the DGR-006 greedy `SamplingContract`; the
+  idempotent-replay equality check depends on order-independent greedy decode.
+- **Native / llama.cpp gates N/A.** No native code, CMake, or llama.cpp patch was
+  touched (same as DGR-005/006/007/012), so those gates do not apply.
+
+## Compatibility / migration notes
+
+- `failure_semantics.py` is a new, additive module — no existing behavior changes.
+- `batch_scheduler.py` changes are additive (new enum members, one method, two
+  telemetry fields); the DGR-012 contract and its 16 tests are unchanged.
+- `WorkRecord.to_dict()` / `WorkLedger.to_dict()` are JSON-safe and map cleanly to
+  the tracker `BillingLedger.charge_request` inputs: report `node_work` only for
+  `billable` (completed) records so cancelled/failed/unverified work is never
+  charged. `FailureKind` / `WorkStatus` are stable strings suitable for the native
+  protocol's structured status and the capability/heartbeat report.
+
+## Handoff for dependent stories
+
+- **DGR-008 (C++ gRPC worker):** implement the same contract natively — the worker
+  maps a transport deadline/heartbeat to `StreamTerminated`, a dropped stream to a
+  restartable failure, and a mid-`llama_decode` crash to an *uncertain* step
+  (mark-uncertain, never silent replay). `RestartController.failover` maps to
+  opening a fresh llama sequence under the new `(session, epoch)`; the failed
+  sequence's KV is dropped, never migrated.
+- **DGR-010/DGR-014 (real acceptance / release gate):** drive the same failure
+  scenarios against the live worker to produce real cleanup/latency numbers, and
+  feed the `WorkLedger` status split into the billing/attribution comparison —
+  only `completed` work is charged.

.scratch/distributed-gguf-runtime/evidence/DGR-013/commands.txt

@@ -0,0 +1,36 @@
+# DGR-013 — exact commands and real results (worktree venv)
+VP=/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python
+
+# Targeted story tests (this story)
+$VP -m pytest -q tests/test_failure_semantics.py
+# -> 22 passed
+
+# Dependency gates stay green
+$VP -m pytest -q tests/test_batch_scheduler.py        # DGR-012
+# -> 16 passed
+$VP -m pytest -q tests/test_hot_kv_state.py           # DGR-007
+# -> 22 passed
+$VP -m pytest -q tests/test_gguf_backend.py           # DGR-009
+# -> 2 passed
+
+# Quality gates
+$VP -m compileall -q packages tests
+# -> exit 0
+git diff --check
+# -> exit 0
+
+# Machine-readable evidence
+$VP .scratch/distributed-gguf-runtime/evidence/DGR-013/generate_evidence.py
+# -> wrote results.json; work statuses {'completed':2,'cancelled':1,'failed':0,'unverified':2} billable_tokens=16
+
+# Full deterministic suite
+$VP -m pytest -q -p no:cacheprovider
+# -> 16 failed, 792 passed, 14 skipped in 253.93s
+
+# Clean-tree reproduction of the 16 pre-existing failures (DGR-013 files removed)
+# rm packages/node/meshnet_node/failure_semantics.py tests/test_failure_semantics.py
+$VP -m pytest -q tests/test_dynamic_routing.py::test_admin_can_replace_a_served_model_and_release_it \
+  tests/test_node_doctor.py::test_the_shipped_recipes_are_all_applicable_by_the_backend \
+  tests/test_tracker_routing.py::test_torch_node_applies_tracker_load_shard_directive \
+  tests/test_node_startup.py::test_preset_model_with_hf_repo_loads_torch_backend
+# -> 4 failed (same failures reproduce without any DGR-013 change)

.scratch/distributed-gguf-runtime/evidence/DGR-013/generate_evidence.py

@@ -0,0 +1,234 @@
+#!/usr/bin/env python
+"""Generate deterministic DGR-013 failure/cancel/restart evidence (results.json).
+
+Runs the real hardened per-session stream (``HardenedSessionRunner`` over the
+DGR-007 ``KvBoundaryAdapter`` + ``HotKvStateManager``) through each failure mode
+with the same pure-numpy dense-Llama reference the default gate uses. No model
+download, no GPU, no torch, no network, no API credit.
+
+Run from the repo root with the worktree venv:
+
+    .venv/bin/python .scratch/distributed-gguf-runtime/evidence/DGR-013/generate_evidence.py
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import sys
+
+import numpy as np
+
+# Make the worktree packages and the DGR-007 numpy reference importable, exactly
+# as pytest's prepend-import + conftest do.
+ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "..", "..", ".."))
+sys.path.insert(0, os.path.join(ROOT, "packages", "node"))
+sys.path.insert(0, os.path.join(ROOT, "tests"))
+
+from meshnet_node.hot_kv_state import (  # noqa: E402
+    HotKvStateConfig,
+    HotKvStateManager,
+    KvBoundaryAdapter,
+    StaleRouteEpochError,
+    kv_recipe_for,
+)
+from meshnet_node.batch_scheduler import GenerationRequest  # noqa: E402
+from meshnet_node.failure_semantics import (  # noqa: E402
+    CancellationToken,
+    FailureKind,
+    HardenedSessionRunner,
+    RestartController,
+    StreamTerminated,
+    WorkLedger,
+    WorkStatus,
+)
+
+from test_hot_kv_state import _KvDenseLlama, _KvReferenceShard  # noqa: E402
+
+
+class _FaultyShard(_KvReferenceShard):
+    def __init__(self, model, start, end, *, fail_at_call=None):
+        super().__init__(model, start, end)
+        self._fail_at_call = fail_at_call
+        self.calls = 0
+
+    def run_layers_cached(self, hidden, *, positions, past_kv):
+        self.calls += 1
+        if self._fail_at_call is not None and self.calls == self._fail_at_call:
+            raise RuntimeError("worker died mid-step")
+        return super().run_layers_cached(hidden, positions=positions, past_kv=past_kv)
+
+
+class _Clock:
+    def __init__(self):
+        self.now = 0.0
+
+    def __call__(self):
+        return self.now
+
+    def advance(self, d):
+        self.now += d
+
+
+def _adapter(model, *, config=None, shard=None):
+    shard = shard or _KvReferenceShard(model, 0, model.n_layers - 1)
+    manager = HotKvStateManager(kv_recipe_for(shard), config=config)
+    return KvBoundaryAdapter(shard, manager)
+
+
+def _gen(sid, prompt, n, epoch=0):
+    return GenerationRequest(
+        session_id=sid, route_epoch=epoch,
+        prompt_token_ids=tuple(prompt), max_new_tokens=n,
+    )
+
+
+def _kv_released(manager, sid, epoch):
+    from meshnet_node.hot_kv_state import CacheMiss
+    return isinstance(manager.resolve(sid, epoch), CacheMiss)
+
+
+def main() -> None:
+    model = _KvDenseLlama()
+    prompt = [7, 3, 9, 1]
+    n_new = 8
+    ledger = WorkLedger()
+    scenarios = []
+
+    # 1. Clean baseline.
+    ad = _adapter(model)
+    r = HardenedSessionRunner(ad, work_ledger=ledger).run(_gen("clean", prompt, n_new))
+    scenarios.append({
+        "scenario": "clean",
+        "status": r.status.value,
+        "tokens": r.token_count,
+        "matches_reference": list(r.tokens) == model.stateless_greedy(prompt, n_new),
+        "kv_released": _kv_released(ad.manager, "clean", 0),
+    })
+
+    # 2. Deadline terminates a blocked stream.
+    clk = _Clock()
+    ad = _adapter(model)
+    r = HardenedSessionRunner(ad, clock=clk).run(
+        _gen("deadline", prompt, 50), deadline=3.0,
+        before_step=lambda _s: clk.advance(1.0),
+    )
+    scenarios.append({
+        "scenario": "deadline", "status": r.status.value,
+        "failure_kind": r.failure_kind.value, "tokens": r.token_count,
+        "kv_released": _kv_released(ad.manager, "deadline", 0),
+    })
+
+    # 3. Heartbeat/health loss terminates a blocked stream.
+    clk = _Clock()
+    ad = _adapter(model)
+    r = HardenedSessionRunner(ad, clock=clk).run(
+        _gen("heartbeat", prompt, 50), heartbeat_timeout=1.5,
+        heartbeat=lambda step: step < 2,
+        before_step=lambda _s: clk.advance(1.0),
+    )
+    scenarios.append({
+        "scenario": "heartbeat_loss", "status": r.status.value,
+        "failure_kind": r.failure_kind.value, "tokens": r.token_count,
+        "kv_released": _kv_released(ad.manager, "heartbeat", 0),
+    })
+
+    # 4. Explicit client cancellation releases KV.
+    ad = _adapter(model)
+    tok = CancellationToken()
+    r = HardenedSessionRunner(ad, work_ledger=ledger).run(
+        _gen("cancel", prompt, 50), cancel_token=tok,
+        before_step=lambda step: tok.cancel("client-hangup") if step == 3 else None,
+    )
+    scenarios.append({
+        "scenario": "cancel", "status": r.status.value,
+        "failure_kind": r.failure_kind.value, "tokens": r.token_count,
+        "kv_released": _kv_released(ad.manager, "cancel", 0),
+    })
+
+    # 5. Worker death mid-step -> unverified.
+    ad = _adapter(model, shard=_FaultyShard(model, 0, model.n_layers - 1, fail_at_call=4))
+    r = HardenedSessionRunner(ad, work_ledger=ledger).run(_gen("worker", prompt, n_new))
+    scenarios.append({
+        "scenario": "worker_death", "status": r.status.value,
+        "failure_kind": r.failure_kind.value, "tokens": r.token_count,
+        "restartable": r.restartable, "kv_released": _kv_released(ad.manager, "worker", 0),
+    })
+
+    # 6. Stream reset -> failed, restartable.
+    ad = _adapter(model)
+    def reset(step):
+        if step == 2:
+            raise StreamTerminated(FailureKind.STREAM_RESET, "peer reset")
+    r = HardenedSessionRunner(ad).run(_gen("reset", prompt, n_new), before_step=reset)
+    scenarios.append({
+        "scenario": "stream_reset", "status": r.status.value,
+        "failure_kind": r.failure_kind.value, "restartable": r.restartable,
+    })
+
+    # 7. Stale epoch -> failed.
+    ad = _adapter(model)
+    ad.manager.open("stale", 5)
+    r = HardenedSessionRunner(ad).run(_gen("stale", prompt, n_new, epoch=3))
+    scenarios.append({
+        "scenario": "stale_epoch", "status": r.status.value,
+        "failure_kind": r.failure_kind.value,
+    })
+
+    # 8. Cache miss mid-stream -> restartable.
+    ad = _adapter(model)
+    mgr = ad.manager
+    r = HardenedSessionRunner(ad).run(
+        _gen("miss", prompt, 12),
+        before_step=lambda step: mgr.release("miss", 0) if step == 4 else None,
+    )
+    scenarios.append({
+        "scenario": "cache_miss", "status": r.status.value,
+        "failure_kind": r.failure_kind.value, "tokens": r.token_count,
+        "restartable": r.restartable,
+    })
+
+    # 9. Alpha failover: restart from token zero, no unverified KV import.
+    faulty = _FaultyShard(model, 0, model.n_layers - 1, fail_at_call=3)
+    ad = _adapter(model, shard=faulty)
+    runner = HardenedSessionRunner(ad, work_ledger=ledger)
+    controller = RestartController([ad.manager])
+    fo = runner.run_with_failover(_gen("failover", prompt, n_new, epoch=0), controller,
+                                  max_restarts=2)
+    old_epoch_stale = False
+    try:
+        ad.manager.resolve("failover", 0)
+    except StaleRouteEpochError:
+        old_epoch_stale = True
+    scenarios.append({
+        "scenario": "alpha_failover",
+        "final_status": fo.outcome.status.value,
+        "final_epoch": fo.outcome.route_epoch,
+        "restarts": fo.restarts,
+        "restarted_from_token_zero": list(fo.outcome.tokens) == model.stateless_greedy(prompt, n_new),
+        "old_epoch_stale": old_epoch_stale,
+        "attempt_statuses": [a.status.value for a in fo.attempts],
+    })
+
+    result = {
+        "schema_version": 1,
+        "evidence_kind": "synthetic-unit",
+        "model": {
+            "architecture": model.architecture_adapter,
+            "n_layers": model.n_layers, "vocab": model.vocab, "hidden": model.hidden,
+        },
+        "scenarios": scenarios,
+        "work_ledger": ledger.to_dict(),
+    }
+
+    out_path = os.path.join(os.path.dirname(__file__), "results.json")
+    with open(out_path, "w") as fh:
+        json.dump(result, fh, indent=2)
+        fh.write("\n")
+    counts = ledger.counts_by_status()
+    print(f"wrote {out_path}")
+    print(f"work statuses: {counts}  billable_tokens={ledger.billable_tokens()}")
+
+
+if __name__ == "__main__":
+    main()

.scratch/distributed-gguf-runtime/evidence/DGR-013/results.json

@@ -0,0 +1,135 @@
+{
+  "schema_version": 1,
+  "evidence_kind": "synthetic-unit",
+  "model": {
+    "architecture": "dense-llama",
+    "n_layers": 6,
+    "vocab": 48,
+    "hidden": 32
+  },
+  "scenarios": [
+    {
+      "scenario": "clean",
+      "status": "completed",
+      "tokens": 8,
+      "matches_reference": true,
+      "kv_released": false
+    },
+    {
+      "scenario": "deadline",
+      "status": "failed",
+      "failure_kind": "deadline-exceeded",
+      "tokens": 2,
+      "kv_released": true
+    },
+    {
+      "scenario": "heartbeat_loss",
+      "status": "failed",
+      "failure_kind": "heartbeat-lost",
+      "tokens": 3,
+      "kv_released": true
+    },
+    {
+      "scenario": "cancel",
+      "status": "cancelled",
+      "failure_kind": "cancelled",
+      "tokens": 3,
+      "kv_released": true
+    },
+    {
+      "scenario": "worker_death",
+      "status": "unverified",
+      "failure_kind": "worker-death",
+      "tokens": 3,
+      "restartable": true,
+      "kv_released": true
+    },
+    {
+      "scenario": "stream_reset",
+      "status": "failed",
+      "failure_kind": "stream-reset",
+      "restartable": true
+    },
+    {
+      "scenario": "stale_epoch",
+      "status": "failed",
+      "failure_kind": "stale-epoch"
+    },
+    {
+      "scenario": "cache_miss",
+      "status": "failed",
+      "failure_kind": "cache-miss",
+      "tokens": 4,
+      "restartable": true
+    },
+    {
+      "scenario": "alpha_failover",
+      "final_status": "completed",
+      "final_epoch": 1,
+      "restarts": 1,
+      "restarted_from_token_zero": true,
+      "old_epoch_stale": true,
+      "attempt_statuses": [
+        "unverified",
+        "completed"
+      ]
+    }
+  ],
+  "work_ledger": {
+    "schema_version": 1,
+    "records": [
+      {
+        "session_id": "clean",
+        "route_epoch": 0,
+        "status": "completed",
+        "tokens": 8,
+        "failure_kind": null,
+        "detail": "",
+        "billable": true
+      },
+      {
+        "session_id": "cancel",
+        "route_epoch": 0,
+        "status": "cancelled",
+        "tokens": 3,
+        "failure_kind": "cancelled",
+        "detail": "operation cancelled: client-hangup",
+        "billable": false
+      },
+      {
+        "session_id": "worker",
+        "route_epoch": 0,
+        "status": "unverified",
+        "tokens": 3,
+        "failure_kind": "worker-death",
+        "detail": "worker died mid-step",
+        "billable": false
+      },
+      {
+        "session_id": "failover",
+        "route_epoch": 0,
+        "status": "unverified",
+        "tokens": 2,
+        "failure_kind": "worker-death",
+        "detail": "worker died mid-step",
+        "billable": false
+      },
+      {
+        "session_id": "failover",
+        "route_epoch": 1,
+        "status": "completed",
+        "tokens": 8,
+        "failure_kind": null,
+        "detail": "",
+        "billable": true
+      }
+    ],
+    "counts_by_status": {
+      "completed": 2,
+      "cancelled": 1,
+      "failed": 0,
+      "unverified": 2
+    },
+    "billable_tokens": 16
+  }
+}