feat: checkpoint batching and release-gate stories

.scratch/distributed-gguf-runtime/evidence/DGR-013/README.md

@@ -0,0 +1,223 @@
+# DGR-013 — Harden failure, cancellation, and restart semantics: evidence
+
+Status: done
+Date: 2026-07-16
+Evidence kind: **synthetic-unit** (pure-numpy KV-cached dense-Llama reference +
+node-local hardened stream). No model download, no GPU, no torch, no network, no
+API credit.
+
+## Summary
+
+Implemented bounded, explicit failure/cancellation/restart semantics for the
+per-Route-Session decode stream, layered on the DGR-007 Hot KV State manager
+(isolated `(session, epoch)` KV) and the DGR-012 continuous-batch scheduler. The
+goal (RALPH product objective) is that distributed speed never comes with hanging
+or corrupted generations: every blocked op is bounded, every cancel frees state,
+duplicate steps are idempotent, uncertain mutations are never silently replayed,
+alpha failover restarts from token zero, and billing distinguishes what actually
+completed.
+
+Everything runs against the same deterministic numpy dense-Llama reference the
+default gate uses (`tests/test_hot_kv_state.py::_KvDenseLlama` / `_KvReferenceShard`),
+so the whole failure matrix is deterministic, download-free, GPU-free, and
+API-credit-free while exercising the **real** KV isolation path
+(`KvBoundaryAdapter` + `HotKvStateManager`). The pinned llama.cpp worker (DGR-008)
+implements the identical adapter contract, so the semantics carry over to native
+execution unchanged.
+
+### What was built (`packages/node/meshnet_node/failure_semantics.py`, new)
+
+- **`DeadlineGuard` + `StreamTerminated`** — bounds every step against an absolute
+  deadline and a heartbeat-timeout on an injected clock. A reached deadline or a
+  lost heartbeat (peer health loss) raises `StreamTerminated(kind)` so a blocked
+  stream terminates instead of hanging. (**AC: deadlines/heartbeat terminate
+  blocked ops.**)
+- **`CancellationToken`, `ShardCancellationGroup`, `CancellationOutcome`** — one
+  cancel fans across **every** node-local Shard of a Route Session, releasing the
+  `(session, epoch)` KV on each shard's manager and invoking every queued-buffer
+  release callback (the pending activation bundles). Idempotent. The DGR-012
+  scheduler also gains a `cancel()` that drops queued/active work on this node and
+  frees its KV. (**AC: cancellation propagates across every Shard, releases KV +
+  queued buffers.**)
+- **`IdempotencyLedger`, `StepKey`, `StepDisposition`, `UncertainMutationError`** —
+  records each committed `(session, epoch, step)`; a duplicate delivery returns the
+  recorded token with no re-mutation. A step whose mutation outcome is *uncertain*
+  (worker died mid-step) is marked uncertain and can **never** be replayed
+  silently — `begin()` on an uncertain (or still in-flight) step raises
+  `UncertainMutationError`, forcing verify-or-restart. (**AC: duplicate steps
+  idempotent; uncertain mutations never replayed silently.**)
+- **`RestartController`** — alpha failover: opens the *next* route epoch, releases
+  every shard's prior-epoch KV, and `assert_fresh_start` fails closed if any shard
+  still holds new-epoch KV. The restart re-prefills the whole prompt from token
+  zero; the failed epoch becomes stale (KV manager rejects it). Unverified KV is
+  never migrated (RALPH runtime decision #14). (**AC: alpha failover restarts from
+  token zero rather than importing unverified KV.**)
+- **`WorkStatus`, `WorkRecord`, `WorkLedger`** — a typed per-attempt work record
+  with four distinct statuses: `completed`, `cancelled`, `failed`, `unverified`.
+  Only `completed` records are billable; cancelled/failed/unverified tokens are
+  recorded for observability but never charged. JSON-safe for the tracker billing
+  handoff (`packages/tracker/meshnet_tracker/billing.py` charges only completed,
+  verified work). (**AC: billing/work records distinguish completed/cancelled/
+  failed/unverified.**)
+- **`HardenedSessionRunner`** — composes all of the above to drive one session's
+  prefill+decode through the adapter under a deadline/heartbeat guard + cancel
+  token, records the typed outcome, and `run_with_failover` restarts a transient
+  failure from token zero on a fresh epoch.
+- **`FailureKind` + `classify_exception` + `work_status_for`** — stable-string
+  classification of worker death, stream reset, malformed bundle, stale epoch,
+  cache miss, deadline, heartbeat loss, and cancel, plus the failure→billing-status
+  mapping. Suitable for the native protocol's structured status.
+
+### Scheduler extension (`packages/node/meshnet_node/batch_scheduler.py`, DGR-012 file, additive)
+
+Purely additive so the DGR-012 gate stays green (16/16):
+- `DoneReason.CANCELLED` / `DoneReason.FAILED` terminal reasons.
+- `ContinuousBatchScheduler.cancel(session_id, *, reason)` — drops a queued
+  session from the bounded queue or releases an active session's KV, moving it to
+  the done set with a non-completed reason (never counted as completed work).
+- `SchedulerTelemetry.cancelled_sessions` / `failed_sessions` counters.
+
+## Files changed
+
+- `packages/node/meshnet_node/failure_semantics.py` — new module (the whole
+  failure/cancel/restart layer above).
+- `packages/node/meshnet_node/batch_scheduler.py` — additive `cancel()` + two
+  `DoneReason` members + two telemetry counters (DGR-012 file; its 16 tests still
+  pass unchanged).
+- `tests/test_failure_semantics.py` — new, 22 tests (matrix below); reuses the
+  DGR-007 numpy reference via `from test_hot_kv_state import _KvDenseLlama,
+  _KvReferenceShard`.
+- `.scratch/distributed-gguf-runtime/evidence/DGR-013/` — this README,
+  `commands.txt`, `generate_evidence.py`, `results.json`.
+- `.ralph-tui/progress.md` — appended the DGR-013 note.
+- `.scratch/distributed-gguf-runtime/issues/13-...md` — set `Status: done`.
+
+## Acceptance criteria → evidence
+
+| Criterion | Tests (`tests/test_failure_semantics.py`) |
+|---|---|
+| Deadlines/heartbeat loss terminate blocked stream ops | `test_deadline_terminates_a_blocked_stream_and_releases_kv`, `test_heartbeat_loss_terminates_a_blocked_stream`, `test_deadline_guard_reports_remaining_and_resets_on_heartbeat` |
+| Cancellation propagates across every Shard, releases KV + queued buffers | `test_cancellation_token_terminates_stream_and_releases_kv`, `test_shard_cancellation_group_releases_every_shard_and_queued_buffers`, `test_scheduler_cancel_drains_queue_and_releases_active_kv`, `test_scheduler_cancel_rejects_a_completed_reason` |
+| Duplicate steps idempotent; uncertain mutations never replayed silently | `test_duplicate_step_delivery_is_idempotent_no_remutation`, `test_idempotent_run_replays_tokens_without_advancing_kv`, `test_uncertain_mutation_is_never_replayed_silently`, `test_in_flight_duplicate_is_treated_as_uncertain` |
+| Alpha failover restarts from token zero, no unverified KV import | `test_alpha_failover_restarts_from_token_zero_and_completes`, `test_failover_refuses_to_import_unverified_kv`, `test_non_restartable_failure_is_not_retried` |
+| Worker death, stream reset, malformed bundle, stale epoch, cache miss | `test_worker_death_midstream_is_unverified_and_marks_step_uncertain`, `test_stream_reset_is_restartable_failure`, `test_malformed_bundle_is_classified_and_does_not_corrupt_kv`, `test_stale_epoch_reference_is_rejected_and_classified`, `test_cache_miss_midstream_is_restartable` |
+| Billing/work records distinguish completed/cancelled/failed/unverified | `test_work_ledger_distinguishes_all_four_statuses`, `test_work_status_and_classification_mapping`, plus the clean-run billability check `test_clean_run_matches_stateless_reference_and_is_billable` |
+
+## Failure matrix (real, deterministic — `results.json`)
+
+Generated by `generate_evidence.py` against the numpy dense-Llama (prompt `[7,3,9,1]`,
+8 new tokens):
+
+| scenario | status | failure_kind | tokens | restartable | KV released |
+|---|---|---|---|---|---|
+| clean | completed | — | 8 | — | (held, then reaped) |
+| deadline | failed | deadline-exceeded | 2 | no | yes |
+| heartbeat_loss | failed | heartbeat-lost | 3 | no | yes |
+| cancel | cancelled | cancelled | 3 | no | yes |
+| worker_death | unverified | worker-death | 3 | yes | yes |
+| stream_reset | failed | stream-reset | — | yes | yes |
+| stale_epoch | failed | stale-epoch | — | no | (never opened) |
+| cache_miss | failed | cache-miss | 4 | yes | (already evicted) |
+| alpha_failover | **completed** (epoch 1) | — | 8 | — | old epoch stale |
+
+Alpha failover: attempt 0 (epoch 0) dies mid-step → `unverified`; the controller
+advances to epoch 1, drops epoch-0 KV, and the restart re-prefills from token zero
+→ `completed`, reproducing the byte-identical stateless reference. The old epoch is
+now stale (a reference to it raises `StaleRouteEpochError`). Work ledger:
+`{completed: 2, cancelled: 1, failed: 0, unverified: 2}`, `billable_tokens = 16`
+(only the two completed streams — the failover restart and the clean run — are
+billed; the cancelled and the two unverified attempts are not).
+
+## Commands and real results
+
+See `commands.txt`. Key results:
+
+```
+tests/test_failure_semantics.py     -> 22 passed
+tests/test_batch_scheduler.py       -> 16 passed (DGR-012 unchanged)
+tests/test_hot_kv_state.py          -> 22 passed (DGR-007)
+tests/test_gguf_backend.py          ->  2 passed (DGR-009)
+python -m compileall -q packages tests -> exit 0
+git diff --check                       -> exit 0
+python -m pytest -q                    -> 16 failed, 792 passed, 14 skipped in 253.93s
+```
+
+## Full-suite baseline (pre-existing, unrelated failures)
+
+The 16 failures are **pre-existing and unrelated to DGR-013**. None import
+`failure_semantics` or `batch_scheduler`; they live in the tracker/control-plane,
+node-startup, doctor, calibration, and route-benchmark suites and fail on the
+model-download / control-plane / recipe-admission paths (e.g.
+`UnsupportedRecipeParam: worker_transport` from the DGR-009 native recipe against
+the Torch backend, and Torch/HF-model startup that this deterministic sandbox does
+not provide). Removing the two DGR-013 files and re-running the failing tests
+reproduces the identical failures (see `commands.txt`, 4-test spot check → same
+4 failures), so DGR-013 introduces no new failure.
+
+Exact failing set (16):
+
+```
+tests/test_dynamic_routing.py::test_admin_can_replace_a_served_model_and_release_it
+tests/test_manual_route_benchmark.py::test_pinned_route_uses_named_node
+tests/test_manual_route_benchmark.py::test_unknown_route_node_is_400
+tests/test_manual_route_benchmark.py::test_invalid_route_shape_is_400
+tests/test_manual_route_benchmark.py::test_clients_without_route_are_unaffected
+tests/test_manual_route_benchmark.py::test_benchmark_records_one_and_two_node_routes
+tests/test_node_doctor.py::test_the_shipped_recipes_are_all_applicable_by_the_backend
+tests/test_node_doctor.py::test_cli_doctor_flags_select_what_is_validated
+tests/test_node_startup.py::test_preset_model_with_hf_repo_loads_torch_backend
+tests/test_node_startup.py::test_real_model_startup_registers_downloaded_inventory_without_checksum
+tests/test_toploc_calibration_dispatch.py::test_calibration_run_dispatches_only_solo_capable_nodes
+tests/test_toploc_calibration_dispatch.py::test_calibration_run_persists_corpus_and_results_endpoint_reports_it
+tests/test_toploc_calibration_dispatch.py::test_calibration_run_node_without_commitment_endpoint_is_skipped_not_failed
+tests/test_tracker_capability_admission.py::test_an_enforcing_tracker_never_routes_a_node_whose_proof_does_not_cover_it[invalid]
+tests/test_tracker_routing.py::test_torch_node_applies_tracker_load_shard_directive
+tests/test_tracker_routing.py::test_shard_heal_cycle_surviving_node_covers_dead_peers_gap
+```
+
+## Limitations and deferred work
+
+- **Synthetic-unit, not real weights.** Semantics are exercised against the
+  deterministic numpy dense-Llama, not a downloaded GGUF, to keep the default gate
+  deterministic/download-free/GPU-free. Real worker-death/stream-reset behavior on
+  a live llama.cpp worker over gRPC belongs to DGR-008/DGR-010 (DGR-010 is blocked
+  — no certified dense-Llama artifact on this machine; see
+  `evidence/DGR-010/BLOCKED.md`).
+- **Single-node per-session stream.** `HardenedSessionRunner` drives one full-shard
+  session (the node-local case); multi-node cancellation is modelled by
+  `ShardCancellationGroup` fanning across each node's KV manager. The cross-node
+  propagation *transport* (cancel frames over gRPC/relay) is the native protocol's
+  job (DGR-002/008); this story owns the local release + record semantics the
+  transport triggers.
+- **Fault injection is deterministic.** Worker death is a shard that raises on the
+  Nth step; stream reset / deadline / heartbeat are injected via an explicit clock
+  and hook. This is what makes the matrix reproducible; live fault behavior is a
+  native/real-hardware property.
+- **Greedy sampling only.** Reuses the DGR-006 greedy `SamplingContract`; the
+  idempotent-replay equality check depends on order-independent greedy decode.
+- **Native / llama.cpp gates N/A.** No native code, CMake, or llama.cpp patch was
+  touched (same as DGR-005/006/007/012), so those gates do not apply.
+
+## Compatibility / migration notes
+
+- `failure_semantics.py` is a new, additive module — no existing behavior changes.
+- `batch_scheduler.py` changes are additive (new enum members, one method, two
+  telemetry fields); the DGR-012 contract and its 16 tests are unchanged.
+- `WorkRecord.to_dict()` / `WorkLedger.to_dict()` are JSON-safe and map cleanly to
+  the tracker `BillingLedger.charge_request` inputs: report `node_work` only for
+  `billable` (completed) records so cancelled/failed/unverified work is never
+  charged. `FailureKind` / `WorkStatus` are stable strings suitable for the native
+  protocol's structured status and the capability/heartbeat report.
+
+## Handoff for dependent stories
+
+- **DGR-008 (C++ gRPC worker):** implement the same contract natively — the worker
+  maps a transport deadline/heartbeat to `StreamTerminated`, a dropped stream to a
+  restartable failure, and a mid-`llama_decode` crash to an *uncertain* step
+  (mark-uncertain, never silent replay). `RestartController.failover` maps to
+  opening a fresh llama sequence under the new `(session, epoch)`; the failed
+  sequence's KV is dropped, never migrated.
+- **DGR-010/DGR-014 (real acceptance / release gate):** drive the same failure
+  scenarios against the live worker to produce real cleanup/latency numbers, and
+  feed the `WorkLedger` status split into the billing/attribution comparison —
+  only `completed` work is charged.