224 lines
14 KiB
Markdown
224 lines
14 KiB
Markdown
# DGR-013 — Harden failure, cancellation, and restart semantics: evidence
|
|
|
|
Status: done
|
|
Date: 2026-07-16
|
|
Evidence kind: **synthetic-unit** (pure-numpy KV-cached dense-Llama reference +
|
|
node-local hardened stream). No model download, no GPU, no torch, no network, no
|
|
API credit.
|
|
|
|
## Summary
|
|
|
|
Implemented bounded, explicit failure/cancellation/restart semantics for the
|
|
per-Route-Session decode stream, layered on the DGR-007 Hot KV State manager
|
|
(isolated `(session, epoch)` KV) and the DGR-012 continuous-batch scheduler. The
|
|
goal (RALPH product objective) is that distributed speed never comes with hanging
|
|
or corrupted generations: every blocked op is bounded, every cancel frees state,
|
|
duplicate steps are idempotent, uncertain mutations are never silently replayed,
|
|
alpha failover restarts from token zero, and billing distinguishes what actually
|
|
completed.
|
|
|
|
Everything runs against the same deterministic numpy dense-Llama reference the
|
|
default gate uses (`tests/test_hot_kv_state.py::_KvDenseLlama` / `_KvReferenceShard`),
|
|
so the whole failure matrix is deterministic, download-free, GPU-free, and
|
|
API-credit-free while exercising the **real** KV isolation path
|
|
(`KvBoundaryAdapter` + `HotKvStateManager`). The pinned llama.cpp worker (DGR-008)
|
|
implements the identical adapter contract, so the semantics carry over to native
|
|
execution unchanged.
|
|
|
|
### What was built (`packages/node/meshnet_node/failure_semantics.py`, new)
|
|
|
|
- **`DeadlineGuard` + `StreamTerminated`** — bounds every step against an absolute
|
|
deadline and a heartbeat-timeout on an injected clock. A reached deadline or a
|
|
lost heartbeat (peer health loss) raises `StreamTerminated(kind)` so a blocked
|
|
stream terminates instead of hanging. (**AC: deadlines/heartbeat terminate
|
|
blocked ops.**)
|
|
- **`CancellationToken`, `ShardCancellationGroup`, `CancellationOutcome`** — one
|
|
cancel fans across **every** node-local Shard of a Route Session, releasing the
|
|
`(session, epoch)` KV on each shard's manager and invoking every queued-buffer
|
|
release callback (the pending activation bundles). Idempotent. The DGR-012
|
|
scheduler also gains a `cancel()` that drops queued/active work on this node and
|
|
frees its KV. (**AC: cancellation propagates across every Shard, releases KV +
|
|
queued buffers.**)
|
|
- **`IdempotencyLedger`, `StepKey`, `StepDisposition`, `UncertainMutationError`** —
|
|
records each committed `(session, epoch, step)`; a duplicate delivery returns the
|
|
recorded token with no re-mutation. A step whose mutation outcome is *uncertain*
|
|
(worker died mid-step) is marked uncertain and can **never** be replayed
|
|
silently — `begin()` on an uncertain (or still in-flight) step raises
|
|
`UncertainMutationError`, forcing verify-or-restart. (**AC: duplicate steps
|
|
idempotent; uncertain mutations never replayed silently.**)
|
|
- **`RestartController`** — alpha failover: opens the *next* route epoch, releases
|
|
every shard's prior-epoch KV, and `assert_fresh_start` fails closed if any shard
|
|
still holds new-epoch KV. The restart re-prefills the whole prompt from token
|
|
zero; the failed epoch becomes stale (KV manager rejects it). Unverified KV is
|
|
never migrated (RALPH runtime decision #14). (**AC: alpha failover restarts from
|
|
token zero rather than importing unverified KV.**)
|
|
- **`WorkStatus`, `WorkRecord`, `WorkLedger`** — a typed per-attempt work record
|
|
with four distinct statuses: `completed`, `cancelled`, `failed`, `unverified`.
|
|
Only `completed` records are billable; cancelled/failed/unverified tokens are
|
|
recorded for observability but never charged. JSON-safe for the tracker billing
|
|
handoff (`packages/tracker/meshnet_tracker/billing.py` charges only completed,
|
|
verified work). (**AC: billing/work records distinguish completed/cancelled/
|
|
failed/unverified.**)
|
|
- **`HardenedSessionRunner`** — composes all of the above to drive one session's
|
|
prefill+decode through the adapter under a deadline/heartbeat guard + cancel
|
|
token, records the typed outcome, and `run_with_failover` restarts a transient
|
|
failure from token zero on a fresh epoch.
|
|
- **`FailureKind` + `classify_exception` + `work_status_for`** — stable-string
|
|
classification of worker death, stream reset, malformed bundle, stale epoch,
|
|
cache miss, deadline, heartbeat loss, and cancel, plus the failure→billing-status
|
|
mapping. Suitable for the native protocol's structured status.
|
|
|
|
### Scheduler extension (`packages/node/meshnet_node/batch_scheduler.py`, DGR-012 file, additive)
|
|
|
|
Purely additive so the DGR-012 gate stays green (16/16):
|
|
- `DoneReason.CANCELLED` / `DoneReason.FAILED` terminal reasons.
|
|
- `ContinuousBatchScheduler.cancel(session_id, *, reason)` — drops a queued
|
|
session from the bounded queue or releases an active session's KV, moving it to
|
|
the done set with a non-completed reason (never counted as completed work).
|
|
- `SchedulerTelemetry.cancelled_sessions` / `failed_sessions` counters.
|
|
|
|
## Files changed
|
|
|
|
- `packages/node/meshnet_node/failure_semantics.py` — new module (the whole
|
|
failure/cancel/restart layer above).
|
|
- `packages/node/meshnet_node/batch_scheduler.py` — additive `cancel()` + two
|
|
`DoneReason` members + two telemetry counters (DGR-012 file; its 16 tests still
|
|
pass unchanged).
|
|
- `tests/test_failure_semantics.py` — new, 22 tests (matrix below); reuses the
|
|
DGR-007 numpy reference via `from test_hot_kv_state import _KvDenseLlama,
|
|
_KvReferenceShard`.
|
|
- `.scratch/distributed-gguf-runtime/evidence/DGR-013/` — this README,
|
|
`commands.txt`, `generate_evidence.py`, `results.json`.
|
|
- `.ralph-tui/progress.md` — appended the DGR-013 note.
|
|
- `.scratch/distributed-gguf-runtime/issues/13-...md` — set `Status: done`.
|
|
|
|
## Acceptance criteria → evidence
|
|
|
|
| Criterion | Tests (`tests/test_failure_semantics.py`) |
|
|
|---|---|
|
|
| Deadlines/heartbeat loss terminate blocked stream ops | `test_deadline_terminates_a_blocked_stream_and_releases_kv`, `test_heartbeat_loss_terminates_a_blocked_stream`, `test_deadline_guard_reports_remaining_and_resets_on_heartbeat` |
|
|
| Cancellation propagates across every Shard, releases KV + queued buffers | `test_cancellation_token_terminates_stream_and_releases_kv`, `test_shard_cancellation_group_releases_every_shard_and_queued_buffers`, `test_scheduler_cancel_drains_queue_and_releases_active_kv`, `test_scheduler_cancel_rejects_a_completed_reason` |
|
|
| Duplicate steps idempotent; uncertain mutations never replayed silently | `test_duplicate_step_delivery_is_idempotent_no_remutation`, `test_idempotent_run_replays_tokens_without_advancing_kv`, `test_uncertain_mutation_is_never_replayed_silently`, `test_in_flight_duplicate_is_treated_as_uncertain` |
|
|
| Alpha failover restarts from token zero, no unverified KV import | `test_alpha_failover_restarts_from_token_zero_and_completes`, `test_failover_refuses_to_import_unverified_kv`, `test_non_restartable_failure_is_not_retried` |
|
|
| Worker death, stream reset, malformed bundle, stale epoch, cache miss | `test_worker_death_midstream_is_unverified_and_marks_step_uncertain`, `test_stream_reset_is_restartable_failure`, `test_malformed_bundle_is_classified_and_does_not_corrupt_kv`, `test_stale_epoch_reference_is_rejected_and_classified`, `test_cache_miss_midstream_is_restartable` |
|
|
| Billing/work records distinguish completed/cancelled/failed/unverified | `test_work_ledger_distinguishes_all_four_statuses`, `test_work_status_and_classification_mapping`, plus the clean-run billability check `test_clean_run_matches_stateless_reference_and_is_billable` |
|
|
|
|
## Failure matrix (real, deterministic — `results.json`)
|
|
|
|
Generated by `generate_evidence.py` against the numpy dense-Llama (prompt `[7,3,9,1]`,
|
|
8 new tokens):
|
|
|
|
| scenario | status | failure_kind | tokens | restartable | KV released |
|
|
|---|---|---|---|---|---|
|
|
| clean | completed | — | 8 | — | (held, then reaped) |
|
|
| deadline | failed | deadline-exceeded | 2 | no | yes |
|
|
| heartbeat_loss | failed | heartbeat-lost | 3 | no | yes |
|
|
| cancel | cancelled | cancelled | 3 | no | yes |
|
|
| worker_death | unverified | worker-death | 3 | yes | yes |
|
|
| stream_reset | failed | stream-reset | — | yes | yes |
|
|
| stale_epoch | failed | stale-epoch | — | no | (never opened) |
|
|
| cache_miss | failed | cache-miss | 4 | yes | (already evicted) |
|
|
| alpha_failover | **completed** (epoch 1) | — | 8 | — | old epoch stale |
|
|
|
|
Alpha failover: attempt 0 (epoch 0) dies mid-step → `unverified`; the controller
|
|
advances to epoch 1, drops epoch-0 KV, and the restart re-prefills from token zero
|
|
→ `completed`, reproducing the byte-identical stateless reference. The old epoch is
|
|
now stale (a reference to it raises `StaleRouteEpochError`). Work ledger:
|
|
`{completed: 2, cancelled: 1, failed: 0, unverified: 2}`, `billable_tokens = 16`
|
|
(only the two completed streams — the failover restart and the clean run — are
|
|
billed; the cancelled and the two unverified attempts are not).
|
|
|
|
## Commands and real results
|
|
|
|
See `commands.txt`. Key results:
|
|
|
|
```
|
|
tests/test_failure_semantics.py -> 22 passed
|
|
tests/test_batch_scheduler.py -> 16 passed (DGR-012 unchanged)
|
|
tests/test_hot_kv_state.py -> 22 passed (DGR-007)
|
|
tests/test_gguf_backend.py -> 2 passed (DGR-009)
|
|
python -m compileall -q packages tests -> exit 0
|
|
git diff --check -> exit 0
|
|
python -m pytest -q -> 16 failed, 792 passed, 14 skipped in 253.93s
|
|
```
|
|
|
|
## Full-suite baseline (pre-existing, unrelated failures)
|
|
|
|
The 16 failures are **pre-existing and unrelated to DGR-013**. None import
|
|
`failure_semantics` or `batch_scheduler`; they live in the tracker/control-plane,
|
|
node-startup, doctor, calibration, and route-benchmark suites and fail on the
|
|
model-download / control-plane / recipe-admission paths (e.g.
|
|
`UnsupportedRecipeParam: worker_transport` from the DGR-009 native recipe against
|
|
the Torch backend, and Torch/HF-model startup that this deterministic sandbox does
|
|
not provide). Removing the two DGR-013 files and re-running the failing tests
|
|
reproduces the identical failures (see `commands.txt`, 4-test spot check → same
|
|
4 failures), so DGR-013 introduces no new failure.
|
|
|
|
Exact failing set (16):
|
|
|
|
```
|
|
tests/test_dynamic_routing.py::test_admin_can_replace_a_served_model_and_release_it
|
|
tests/test_manual_route_benchmark.py::test_pinned_route_uses_named_node
|
|
tests/test_manual_route_benchmark.py::test_unknown_route_node_is_400
|
|
tests/test_manual_route_benchmark.py::test_invalid_route_shape_is_400
|
|
tests/test_manual_route_benchmark.py::test_clients_without_route_are_unaffected
|
|
tests/test_manual_route_benchmark.py::test_benchmark_records_one_and_two_node_routes
|
|
tests/test_node_doctor.py::test_the_shipped_recipes_are_all_applicable_by_the_backend
|
|
tests/test_node_doctor.py::test_cli_doctor_flags_select_what_is_validated
|
|
tests/test_node_startup.py::test_preset_model_with_hf_repo_loads_torch_backend
|
|
tests/test_node_startup.py::test_real_model_startup_registers_downloaded_inventory_without_checksum
|
|
tests/test_toploc_calibration_dispatch.py::test_calibration_run_dispatches_only_solo_capable_nodes
|
|
tests/test_toploc_calibration_dispatch.py::test_calibration_run_persists_corpus_and_results_endpoint_reports_it
|
|
tests/test_toploc_calibration_dispatch.py::test_calibration_run_node_without_commitment_endpoint_is_skipped_not_failed
|
|
tests/test_tracker_capability_admission.py::test_an_enforcing_tracker_never_routes_a_node_whose_proof_does_not_cover_it[invalid]
|
|
tests/test_tracker_routing.py::test_torch_node_applies_tracker_load_shard_directive
|
|
tests/test_tracker_routing.py::test_shard_heal_cycle_surviving_node_covers_dead_peers_gap
|
|
```
|
|
|
|
## Limitations and deferred work
|
|
|
|
- **Synthetic-unit, not real weights.** Semantics are exercised against the
|
|
deterministic numpy dense-Llama, not a downloaded GGUF, to keep the default gate
|
|
deterministic/download-free/GPU-free. Real worker-death/stream-reset behavior on
|
|
a live llama.cpp worker over gRPC belongs to DGR-008/DGR-010 (DGR-010 is blocked
|
|
— no certified dense-Llama artifact on this machine; see
|
|
`evidence/DGR-010/BLOCKED.md`).
|
|
- **Single-node per-session stream.** `HardenedSessionRunner` drives one full-shard
|
|
session (the node-local case); multi-node cancellation is modelled by
|
|
`ShardCancellationGroup` fanning across each node's KV manager. The cross-node
|
|
propagation *transport* (cancel frames over gRPC/relay) is the native protocol's
|
|
job (DGR-002/008); this story owns the local release + record semantics the
|
|
transport triggers.
|
|
- **Fault injection is deterministic.** Worker death is a shard that raises on the
|
|
Nth step; stream reset / deadline / heartbeat are injected via an explicit clock
|
|
and hook. This is what makes the matrix reproducible; live fault behavior is a
|
|
native/real-hardware property.
|
|
- **Greedy sampling only.** Reuses the DGR-006 greedy `SamplingContract`; the
|
|
idempotent-replay equality check depends on order-independent greedy decode.
|
|
- **Native / llama.cpp gates N/A.** No native code, CMake, or llama.cpp patch was
|
|
touched (same as DGR-005/006/007/012), so those gates do not apply.
|
|
|
|
## Compatibility / migration notes
|
|
|
|
- `failure_semantics.py` is a new, additive module — no existing behavior changes.
|
|
- `batch_scheduler.py` changes are additive (new enum members, one method, two
|
|
telemetry fields); the DGR-012 contract and its 16 tests are unchanged.
|
|
- `WorkRecord.to_dict()` / `WorkLedger.to_dict()` are JSON-safe and map cleanly to
|
|
the tracker `BillingLedger.charge_request` inputs: report `node_work` only for
|
|
`billable` (completed) records so cancelled/failed/unverified work is never
|
|
charged. `FailureKind` / `WorkStatus` are stable strings suitable for the native
|
|
protocol's structured status and the capability/heartbeat report.
|
|
|
|
## Handoff for dependent stories
|
|
|
|
- **DGR-008 (C++ gRPC worker):** implement the same contract natively — the worker
|
|
maps a transport deadline/heartbeat to `StreamTerminated`, a dropped stream to a
|
|
restartable failure, and a mid-`llama_decode` crash to an *uncertain* step
|
|
(mark-uncertain, never silent replay). `RestartController.failover` maps to
|
|
opening a fresh llama sequence under the new `(session, epoch)`; the failed
|
|
sequence's KV is dropped, never migrated.
|
|
- **DGR-010/DGR-014 (real acceptance / release gate):** drive the same failure
|
|
scenarios against the live worker to produce real cleanup/latency numbers, and
|
|
feed the `WorkLedger` status split into the billing/attribution comparison —
|
|
only `completed` work is charged.
|