# DGR-013 — Harden failure, cancellation, and restart semantics: evidence Status: done Date: 2026-07-16 Evidence kind: **synthetic-unit** (pure-numpy KV-cached dense-Llama reference + node-local hardened stream). No model download, no GPU, no torch, no network, no API credit. ## Summary Implemented bounded, explicit failure/cancellation/restart semantics for the per-Route-Session decode stream, layered on the DGR-007 Hot KV State manager (isolated `(session, epoch)` KV) and the DGR-012 continuous-batch scheduler. The goal (RALPH product objective) is that distributed speed never comes with hanging or corrupted generations: every blocked op is bounded, every cancel frees state, duplicate steps are idempotent, uncertain mutations are never silently replayed, alpha failover restarts from token zero, and billing distinguishes what actually completed. Everything runs against the same deterministic numpy dense-Llama reference the default gate uses (`tests/test_hot_kv_state.py::_KvDenseLlama` / `_KvReferenceShard`), so the whole failure matrix is deterministic, download-free, GPU-free, and API-credit-free while exercising the **real** KV isolation path (`KvBoundaryAdapter` + `HotKvStateManager`). The pinned llama.cpp worker (DGR-008) implements the identical adapter contract, so the semantics carry over to native execution unchanged. ### What was built (`packages/node/meshnet_node/failure_semantics.py`, new) - **`DeadlineGuard` + `StreamTerminated`** — bounds every step against an absolute deadline and a heartbeat-timeout on an injected clock. A reached deadline or a lost heartbeat (peer health loss) raises `StreamTerminated(kind)` so a blocked stream terminates instead of hanging. (**AC: deadlines/heartbeat terminate blocked ops.**) - **`CancellationToken`, `ShardCancellationGroup`, `CancellationOutcome`** — one cancel fans across **every** node-local Shard of a Route Session, releasing the `(session, epoch)` KV on each shard's manager and invoking every queued-buffer release callback (the pending activation bundles). Idempotent. The DGR-012 scheduler also gains a `cancel()` that drops queued/active work on this node and frees its KV. (**AC: cancellation propagates across every Shard, releases KV + queued buffers.**) - **`IdempotencyLedger`, `StepKey`, `StepDisposition`, `UncertainMutationError`** — records each committed `(session, epoch, step)`; a duplicate delivery returns the recorded token with no re-mutation. A step whose mutation outcome is *uncertain* (worker died mid-step) is marked uncertain and can **never** be replayed silently — `begin()` on an uncertain (or still in-flight) step raises `UncertainMutationError`, forcing verify-or-restart. (**AC: duplicate steps idempotent; uncertain mutations never replayed silently.**) - **`RestartController`** — alpha failover: opens the *next* route epoch, releases every shard's prior-epoch KV, and `assert_fresh_start` fails closed if any shard still holds new-epoch KV. The restart re-prefills the whole prompt from token zero; the failed epoch becomes stale (KV manager rejects it). Unverified KV is never migrated (RALPH runtime decision #14). (**AC: alpha failover restarts from token zero rather than importing unverified KV.**) - **`WorkStatus`, `WorkRecord`, `WorkLedger`** — a typed per-attempt work record with four distinct statuses: `completed`, `cancelled`, `failed`, `unverified`. Only `completed` records are billable; cancelled/failed/unverified tokens are recorded for observability but never charged. JSON-safe for the tracker billing handoff (`packages/tracker/meshnet_tracker/billing.py` charges only completed, verified work). (**AC: billing/work records distinguish completed/cancelled/ failed/unverified.**) - **`HardenedSessionRunner`** — composes all of the above to drive one session's prefill+decode through the adapter under a deadline/heartbeat guard + cancel token, records the typed outcome, and `run_with_failover` restarts a transient failure from token zero on a fresh epoch. - **`FailureKind` + `classify_exception` + `work_status_for`** — stable-string classification of worker death, stream reset, malformed bundle, stale epoch, cache miss, deadline, heartbeat loss, and cancel, plus the failure→billing-status mapping. Suitable for the native protocol's structured status. ### Scheduler extension (`packages/node/meshnet_node/batch_scheduler.py`, DGR-012 file, additive) Purely additive so the DGR-012 gate stays green (16/16): - `DoneReason.CANCELLED` / `DoneReason.FAILED` terminal reasons. - `ContinuousBatchScheduler.cancel(session_id, *, reason)` — drops a queued session from the bounded queue or releases an active session's KV, moving it to the done set with a non-completed reason (never counted as completed work). - `SchedulerTelemetry.cancelled_sessions` / `failed_sessions` counters. ## Files changed - `packages/node/meshnet_node/failure_semantics.py` — new module (the whole failure/cancel/restart layer above). - `packages/node/meshnet_node/batch_scheduler.py` — additive `cancel()` + two `DoneReason` members + two telemetry counters (DGR-012 file; its 16 tests still pass unchanged). - `tests/test_failure_semantics.py` — new, 22 tests (matrix below); reuses the DGR-007 numpy reference via `from test_hot_kv_state import _KvDenseLlama, _KvReferenceShard`. - `.scratch/distributed-gguf-runtime/evidence/DGR-013/` — this README, `commands.txt`, `generate_evidence.py`, `results.json`. - `.ralph-tui/progress.md` — appended the DGR-013 note. - `.scratch/distributed-gguf-runtime/issues/13-...md` — set `Status: done`. ## Acceptance criteria → evidence | Criterion | Tests (`tests/test_failure_semantics.py`) | |---|---| | Deadlines/heartbeat loss terminate blocked stream ops | `test_deadline_terminates_a_blocked_stream_and_releases_kv`, `test_heartbeat_loss_terminates_a_blocked_stream`, `test_deadline_guard_reports_remaining_and_resets_on_heartbeat` | | Cancellation propagates across every Shard, releases KV + queued buffers | `test_cancellation_token_terminates_stream_and_releases_kv`, `test_shard_cancellation_group_releases_every_shard_and_queued_buffers`, `test_scheduler_cancel_drains_queue_and_releases_active_kv`, `test_scheduler_cancel_rejects_a_completed_reason` | | Duplicate steps idempotent; uncertain mutations never replayed silently | `test_duplicate_step_delivery_is_idempotent_no_remutation`, `test_idempotent_run_replays_tokens_without_advancing_kv`, `test_uncertain_mutation_is_never_replayed_silently`, `test_in_flight_duplicate_is_treated_as_uncertain` | | Alpha failover restarts from token zero, no unverified KV import | `test_alpha_failover_restarts_from_token_zero_and_completes`, `test_failover_refuses_to_import_unverified_kv`, `test_non_restartable_failure_is_not_retried` | | Worker death, stream reset, malformed bundle, stale epoch, cache miss | `test_worker_death_midstream_is_unverified_and_marks_step_uncertain`, `test_stream_reset_is_restartable_failure`, `test_malformed_bundle_is_classified_and_does_not_corrupt_kv`, `test_stale_epoch_reference_is_rejected_and_classified`, `test_cache_miss_midstream_is_restartable` | | Billing/work records distinguish completed/cancelled/failed/unverified | `test_work_ledger_distinguishes_all_four_statuses`, `test_work_status_and_classification_mapping`, plus the clean-run billability check `test_clean_run_matches_stateless_reference_and_is_billable` | ## Failure matrix (real, deterministic — `results.json`) Generated by `generate_evidence.py` against the numpy dense-Llama (prompt `[7,3,9,1]`, 8 new tokens): | scenario | status | failure_kind | tokens | restartable | KV released | |---|---|---|---|---|---| | clean | completed | — | 8 | — | (held, then reaped) | | deadline | failed | deadline-exceeded | 2 | no | yes | | heartbeat_loss | failed | heartbeat-lost | 3 | no | yes | | cancel | cancelled | cancelled | 3 | no | yes | | worker_death | unverified | worker-death | 3 | yes | yes | | stream_reset | failed | stream-reset | — | yes | yes | | stale_epoch | failed | stale-epoch | — | no | (never opened) | | cache_miss | failed | cache-miss | 4 | yes | (already evicted) | | alpha_failover | **completed** (epoch 1) | — | 8 | — | old epoch stale | Alpha failover: attempt 0 (epoch 0) dies mid-step → `unverified`; the controller advances to epoch 1, drops epoch-0 KV, and the restart re-prefills from token zero → `completed`, reproducing the byte-identical stateless reference. The old epoch is now stale (a reference to it raises `StaleRouteEpochError`). Work ledger: `{completed: 2, cancelled: 1, failed: 0, unverified: 2}`, `billable_tokens = 16` (only the two completed streams — the failover restart and the clean run — are billed; the cancelled and the two unverified attempts are not). ## Commands and real results See `commands.txt`. Key results: ``` tests/test_failure_semantics.py -> 22 passed tests/test_batch_scheduler.py -> 16 passed (DGR-012 unchanged) tests/test_hot_kv_state.py -> 22 passed (DGR-007) tests/test_gguf_backend.py -> 2 passed (DGR-009) python -m compileall -q packages tests -> exit 0 git diff --check -> exit 0 python -m pytest -q -> 16 failed, 792 passed, 14 skipped in 253.93s ``` ## Full-suite baseline (pre-existing, unrelated failures) The 16 failures are **pre-existing and unrelated to DGR-013**. None import `failure_semantics` or `batch_scheduler`; they live in the tracker/control-plane, node-startup, doctor, calibration, and route-benchmark suites and fail on the model-download / control-plane / recipe-admission paths (e.g. `UnsupportedRecipeParam: worker_transport` from the DGR-009 native recipe against the Torch backend, and Torch/HF-model startup that this deterministic sandbox does not provide). Removing the two DGR-013 files and re-running the failing tests reproduces the identical failures (see `commands.txt`, 4-test spot check → same 4 failures), so DGR-013 introduces no new failure. Exact failing set (16): ``` tests/test_dynamic_routing.py::test_admin_can_replace_a_served_model_and_release_it tests/test_manual_route_benchmark.py::test_pinned_route_uses_named_node tests/test_manual_route_benchmark.py::test_unknown_route_node_is_400 tests/test_manual_route_benchmark.py::test_invalid_route_shape_is_400 tests/test_manual_route_benchmark.py::test_clients_without_route_are_unaffected tests/test_manual_route_benchmark.py::test_benchmark_records_one_and_two_node_routes tests/test_node_doctor.py::test_the_shipped_recipes_are_all_applicable_by_the_backend tests/test_node_doctor.py::test_cli_doctor_flags_select_what_is_validated tests/test_node_startup.py::test_preset_model_with_hf_repo_loads_torch_backend tests/test_node_startup.py::test_real_model_startup_registers_downloaded_inventory_without_checksum tests/test_toploc_calibration_dispatch.py::test_calibration_run_dispatches_only_solo_capable_nodes tests/test_toploc_calibration_dispatch.py::test_calibration_run_persists_corpus_and_results_endpoint_reports_it tests/test_toploc_calibration_dispatch.py::test_calibration_run_node_without_commitment_endpoint_is_skipped_not_failed tests/test_tracker_capability_admission.py::test_an_enforcing_tracker_never_routes_a_node_whose_proof_does_not_cover_it[invalid] tests/test_tracker_routing.py::test_torch_node_applies_tracker_load_shard_directive tests/test_tracker_routing.py::test_shard_heal_cycle_surviving_node_covers_dead_peers_gap ``` ## Limitations and deferred work - **Synthetic-unit, not real weights.** Semantics are exercised against the deterministic numpy dense-Llama, not a downloaded GGUF, to keep the default gate deterministic/download-free/GPU-free. Real worker-death/stream-reset behavior on a live llama.cpp worker over gRPC belongs to DGR-008/DGR-010 (DGR-010 is blocked — no certified dense-Llama artifact on this machine; see `evidence/DGR-010/BLOCKED.md`). - **Single-node per-session stream.** `HardenedSessionRunner` drives one full-shard session (the node-local case); multi-node cancellation is modelled by `ShardCancellationGroup` fanning across each node's KV manager. The cross-node propagation *transport* (cancel frames over gRPC/relay) is the native protocol's job (DGR-002/008); this story owns the local release + record semantics the transport triggers. - **Fault injection is deterministic.** Worker death is a shard that raises on the Nth step; stream reset / deadline / heartbeat are injected via an explicit clock and hook. This is what makes the matrix reproducible; live fault behavior is a native/real-hardware property. - **Greedy sampling only.** Reuses the DGR-006 greedy `SamplingContract`; the idempotent-replay equality check depends on order-independent greedy decode. - **Native / llama.cpp gates N/A.** No native code, CMake, or llama.cpp patch was touched (same as DGR-005/006/007/012), so those gates do not apply. ## Compatibility / migration notes - `failure_semantics.py` is a new, additive module — no existing behavior changes. - `batch_scheduler.py` changes are additive (new enum members, one method, two telemetry fields); the DGR-012 contract and its 16 tests are unchanged. - `WorkRecord.to_dict()` / `WorkLedger.to_dict()` are JSON-safe and map cleanly to the tracker `BillingLedger.charge_request` inputs: report `node_work` only for `billable` (completed) records so cancelled/failed/unverified work is never charged. `FailureKind` / `WorkStatus` are stable strings suitable for the native protocol's structured status and the capability/heartbeat report. ## Handoff for dependent stories - **DGR-008 (C++ gRPC worker):** implement the same contract natively — the worker maps a transport deadline/heartbeat to `StreamTerminated`, a dropped stream to a restartable failure, and a mid-`llama_decode` crash to an *uncertain* step (mark-uncertain, never silent replay). `RestartController.failover` maps to opening a fresh llama sequence under the new `(session, epoch)`; the failed sequence's KV is dropped, never migrated. - **DGR-010/DGR-014 (real acceptance / release gate):** drive the same failure scenarios against the live worker to produce real cleanup/latency numbers, and feed the `WorkLedger` status split into the billing/attribution comparison — only `completed` work is charged.