neuron-tai/.scratch/distributed-gguf-runtime/evidence/DGR-013

DGR-013 — Harden failure, cancellation, and restart semantics: evidence

Status: done Date: 2026-07-16 Evidence kind: synthetic-unit (pure-numpy KV-cached dense-Llama reference + node-local hardened stream). No model download, no GPU, no torch, no network, no API credit.

Summary

Implemented bounded, explicit failure/cancellation/restart semantics for the per-Route-Session decode stream, layered on the DGR-007 Hot KV State manager (isolated (session, epoch) KV) and the DGR-012 continuous-batch scheduler. The goal (RALPH product objective) is that distributed speed never comes with hanging or corrupted generations: every blocked op is bounded, every cancel frees state, duplicate steps are idempotent, uncertain mutations are never silently replayed, alpha failover restarts from token zero, and billing distinguishes what actually completed.

Everything runs against the same deterministic numpy dense-Llama reference the default gate uses (tests/test_hot_kv_state.py::_KvDenseLlama / _KvReferenceShard), so the whole failure matrix is deterministic, download-free, GPU-free, and API-credit-free while exercising the real KV isolation path (KvBoundaryAdapter + HotKvStateManager). The pinned llama.cpp worker (DGR-008) implements the identical adapter contract, so the semantics carry over to native execution unchanged.

What was built (packages/node/meshnet_node/failure_semantics.py, new)

• DeadlineGuard + StreamTerminated — bounds every step against an absolute deadline and a heartbeat-timeout on an injected clock. A reached deadline or a lost heartbeat (peer health loss) raises StreamTerminated(kind) so a blocked stream terminates instead of hanging. (AC: deadlines/heartbeat terminate blocked ops.)
• CancellationToken, ShardCancellationGroup, CancellationOutcome — one cancel fans across every node-local Shard of a Route Session, releasing the (session, epoch) KV on each shard's manager and invoking every queued-buffer release callback (the pending activation bundles). Idempotent. The DGR-012 scheduler also gains a cancel() that drops queued/active work on this node and frees its KV. (AC: cancellation propagates across every Shard, releases KV + queued buffers.)
• IdempotencyLedger, StepKey, StepDisposition, UncertainMutationError — records each committed (session, epoch, step); a duplicate delivery returns the recorded token with no re-mutation. A step whose mutation outcome is uncertain (worker died mid-step) is marked uncertain and can never be replayed silently — begin() on an uncertain (or still in-flight) step raises UncertainMutationError, forcing verify-or-restart. (AC: duplicate steps idempotent; uncertain mutations never replayed silently.)
• RestartController — alpha failover: opens the next route epoch, releases every shard's prior-epoch KV, and assert_fresh_start fails closed if any shard still holds new-epoch KV. The restart re-prefills the whole prompt from token zero; the failed epoch becomes stale (KV manager rejects it). Unverified KV is never migrated (RALPH runtime decision #14). (AC: alpha failover restarts from token zero rather than importing unverified KV.)
• WorkStatus, WorkRecord, WorkLedger — a typed per-attempt work record with four distinct statuses: completed, cancelled, failed, unverified. Only completed records are billable; cancelled/failed/unverified tokens are recorded for observability but never charged. JSON-safe for the tracker billing handoff (packages/tracker/meshnet_tracker/billing.py charges only completed, verified work). (AC: billing/work records distinguish completed/cancelled/ failed/unverified.)
• HardenedSessionRunner — composes all of the above to drive one session's prefill+decode through the adapter under a deadline/heartbeat guard + cancel token, records the typed outcome, and run_with_failover restarts a transient failure from token zero on a fresh epoch.
• FailureKind + classify_exception + work_status_for — stable-string classification of worker death, stream reset, malformed bundle, stale epoch, cache miss, deadline, heartbeat loss, and cancel, plus the failure→billing-status mapping. Suitable for the native protocol's structured status.

Scheduler extension (packages/node/meshnet_node/batch_scheduler.py, DGR-012 file, additive)

Purely additive so the DGR-012 gate stays green (16/16):

• DoneReason.CANCELLED / DoneReason.FAILED terminal reasons.
• ContinuousBatchScheduler.cancel(session_id, *, reason) — drops a queued session from the bounded queue or releases an active session's KV, moving it to the done set with a non-completed reason (never counted as completed work).
• SchedulerTelemetry.cancelled_sessions / failed_sessions counters.

Files changed

• packages/node/meshnet_node/failure_semantics.py — new module (the whole failure/cancel/restart layer above).
• packages/node/meshnet_node/batch_scheduler.py — additive cancel() + two DoneReason members + two telemetry counters (DGR-012 file; its 16 tests still pass unchanged).
• tests/test_failure_semantics.py — new, 22 tests (matrix below); reuses the DGR-007 numpy reference via from test_hot_kv_state import _KvDenseLlama, _KvReferenceShard.
• .scratch/distributed-gguf-runtime/evidence/DGR-013/ — this README, commands.txt, generate_evidence.py, results.json.
• .ralph-tui/progress.md — appended the DGR-013 note.
• .scratch/distributed-gguf-runtime/issues/13-...md — set Status: done.

Acceptance criteria → evidence

Criterion	Tests (tests/test_failure_semantics.py)
Deadlines/heartbeat loss terminate blocked stream ops	test_deadline_terminates_a_blocked_stream_and_releases_kv, test_heartbeat_loss_terminates_a_blocked_stream, test_deadline_guard_reports_remaining_and_resets_on_heartbeat
Cancellation propagates across every Shard, releases KV + queued buffers	test_cancellation_token_terminates_stream_and_releases_kv, test_shard_cancellation_group_releases_every_shard_and_queued_buffers, test_scheduler_cancel_drains_queue_and_releases_active_kv, test_scheduler_cancel_rejects_a_completed_reason
Duplicate steps idempotent; uncertain mutations never replayed silently	test_duplicate_step_delivery_is_idempotent_no_remutation, test_idempotent_run_replays_tokens_without_advancing_kv, test_uncertain_mutation_is_never_replayed_silently, test_in_flight_duplicate_is_treated_as_uncertain
Alpha failover restarts from token zero, no unverified KV import	test_alpha_failover_restarts_from_token_zero_and_completes, test_failover_refuses_to_import_unverified_kv, test_non_restartable_failure_is_not_retried
Worker death, stream reset, malformed bundle, stale epoch, cache miss	test_worker_death_midstream_is_unverified_and_marks_step_uncertain, test_stream_reset_is_restartable_failure, test_malformed_bundle_is_classified_and_does_not_corrupt_kv, test_stale_epoch_reference_is_rejected_and_classified, test_cache_miss_midstream_is_restartable
Billing/work records distinguish completed/cancelled/failed/unverified	test_work_ledger_distinguishes_all_four_statuses, test_work_status_and_classification_mapping, plus the clean-run billability check test_clean_run_matches_stateless_reference_and_is_billable

Failure matrix (real, deterministic — results.json)

Generated by generate_evidence.py against the numpy dense-Llama (prompt [7,3,9,1], 8 new tokens):

scenario	status	failure_kind	tokens	restartable	KV released
clean	completed	—	8	—	(held, then reaped)
deadline	failed	deadline-exceeded	2	no	yes
heartbeat_loss	failed	heartbeat-lost	3	no	yes
cancel	cancelled	cancelled	3	no	yes
worker_death	unverified	worker-death	3	yes	yes
stream_reset	failed	stream-reset	—	yes	yes
stale_epoch	failed	stale-epoch	—	no	(never opened)
cache_miss	failed	cache-miss	4	yes	(already evicted)
alpha_failover	completed (epoch 1)	—	8	—	old epoch stale

Alpha failover: attempt 0 (epoch 0) dies mid-step → unverified; the controller advances to epoch 1, drops epoch-0 KV, and the restart re-prefills from token zero → completed, reproducing the byte-identical stateless reference. The old epoch is now stale (a reference to it raises StaleRouteEpochError). Work ledger: {completed: 2, cancelled: 1, failed: 0, unverified: 2}, billable_tokens = 16 (only the two completed streams — the failover restart and the clean run — are billed; the cancelled and the two unverified attempts are not).

Commands and real results

See commands.txt. Key results:

tests/test_failure_semantics.py     -> 22 passed
tests/test_batch_scheduler.py       -> 16 passed (DGR-012 unchanged)
tests/test_hot_kv_state.py          -> 22 passed (DGR-007)
tests/test_gguf_backend.py          ->  2 passed (DGR-009)
python -m compileall -q packages tests -> exit 0
git diff --check                       -> exit 0
python -m pytest -q                    -> 16 failed, 792 passed, 14 skipped in 253.93s

Full-suite baseline (pre-existing, unrelated failures)

The 16 failures are pre-existing and unrelated to DGR-013. None import failure_semantics or batch_scheduler; they live in the tracker/control-plane, node-startup, doctor, calibration, and route-benchmark suites and fail on the model-download / control-plane / recipe-admission paths (e.g. UnsupportedRecipeParam: worker_transport from the DGR-009 native recipe against the Torch backend, and Torch/HF-model startup that this deterministic sandbox does not provide). Removing the two DGR-013 files and re-running the failing tests reproduces the identical failures (see commands.txt, 4-test spot check → same 4 failures), so DGR-013 introduces no new failure.

Exact failing set (16):

tests/test_dynamic_routing.py::test_admin_can_replace_a_served_model_and_release_it
tests/test_manual_route_benchmark.py::test_pinned_route_uses_named_node
tests/test_manual_route_benchmark.py::test_unknown_route_node_is_400
tests/test_manual_route_benchmark.py::test_invalid_route_shape_is_400
tests/test_manual_route_benchmark.py::test_clients_without_route_are_unaffected
tests/test_manual_route_benchmark.py::test_benchmark_records_one_and_two_node_routes
tests/test_node_doctor.py::test_the_shipped_recipes_are_all_applicable_by_the_backend
tests/test_node_doctor.py::test_cli_doctor_flags_select_what_is_validated
tests/test_node_startup.py::test_preset_model_with_hf_repo_loads_torch_backend
tests/test_node_startup.py::test_real_model_startup_registers_downloaded_inventory_without_checksum
tests/test_toploc_calibration_dispatch.py::test_calibration_run_dispatches_only_solo_capable_nodes
tests/test_toploc_calibration_dispatch.py::test_calibration_run_persists_corpus_and_results_endpoint_reports_it
tests/test_toploc_calibration_dispatch.py::test_calibration_run_node_without_commitment_endpoint_is_skipped_not_failed
tests/test_tracker_capability_admission.py::test_an_enforcing_tracker_never_routes_a_node_whose_proof_does_not_cover_it[invalid]
tests/test_tracker_routing.py::test_torch_node_applies_tracker_load_shard_directive
tests/test_tracker_routing.py::test_shard_heal_cycle_surviving_node_covers_dead_peers_gap

Limitations and deferred work

• Synthetic-unit, not real weights. Semantics are exercised against the deterministic numpy dense-Llama, not a downloaded GGUF, to keep the default gate deterministic/download-free/GPU-free. Real worker-death/stream-reset behavior on a live llama.cpp worker over gRPC belongs to DGR-008/DGR-010 (DGR-010 is blocked — no certified dense-Llama artifact on this machine; see evidence/DGR-010/BLOCKED.md).
• Single-node per-session stream. HardenedSessionRunner drives one full-shard session (the node-local case); multi-node cancellation is modelled by ShardCancellationGroup fanning across each node's KV manager. The cross-node propagation transport (cancel frames over gRPC/relay) is the native protocol's job (DGR-002/008); this story owns the local release + record semantics the transport triggers.
• Fault injection is deterministic. Worker death is a shard that raises on the Nth step; stream reset / deadline / heartbeat are injected via an explicit clock and hook. This is what makes the matrix reproducible; live fault behavior is a native/real-hardware property.
• Greedy sampling only. Reuses the DGR-006 greedy SamplingContract; the idempotent-replay equality check depends on order-independent greedy decode.
• Native / llama.cpp gates N/A. No native code, CMake, or llama.cpp patch was touched (same as DGR-005/006/007/012), so those gates do not apply.

Compatibility / migration notes

• failure_semantics.py is a new, additive module — no existing behavior changes.
• batch_scheduler.py changes are additive (new enum members, one method, two telemetry fields); the DGR-012 contract and its 16 tests are unchanged.
• WorkRecord.to_dict() / WorkLedger.to_dict() are JSON-safe and map cleanly to the tracker BillingLedger.charge_request inputs: report node_work only for billable (completed) records so cancelled/failed/unverified work is never charged. FailureKind / WorkStatus are stable strings suitable for the native protocol's structured status and the capability/heartbeat report.

Handoff for dependent stories

• DGR-008 (C++ gRPC worker): implement the same contract natively — the worker maps a transport deadline/heartbeat to StreamTerminated, a dropped stream to a restartable failure, and a mid-llama_decode crash to an uncertain step (mark-uncertain, never silent replay). RestartController.failover maps to opening a fresh llama sequence under the new (session, epoch); the failed sequence's KV is dropped, never migrated.
• DGR-010/DGR-014 (real acceptance / release gate): drive the same failure scenarios against the live worker to produce real cleanup/latency numbers, and feed the WorkLedger status split into the billing/attribution comparison — only completed work is charged.