feat: checkpoint distributed gguf runtime stories

.scratch/distributed-gguf-runtime/evidence/DGR-007/README.md

@@ -0,0 +1,229 @@
+# DGR-007 — Isolated concurrent local Hot KV State: evidence
+
+Status: done
+Date: 2026-07-15
+Evidence kind: **synthetic-unit** (pure-numpy KV-cached dense-Llama reference +
+session/KV manager). No model download, no GPU, no torch, no network, no API
+credit.
+
+## Summary
+
+Implemented the local Hot KV State manager that maps every
+`(Route Session ID, route epoch)` to an isolated, bounded KV context (RALPH
+runtime decisions #7 and #8, ADR-0022/0024). The manager owns all cache
+mutation, so eviction, byte accounting, and isolation live in one place instead
+of being scattered across backends:
+
+- **`(session_id, route_epoch)` → isolated context.** Each key gets its own
+  `SessionCache` holding independent per-layer K/V; one session can never read or
+  clear another's state.
+- **KV allocated only for owned layers.** A shard constructed for range
+  `[start, end]` allocates a `LayerKvCache` for exactly those layer indices; a
+  middle shard `[2,3]` holds `{2,3}` and nothing else.
+- **Full lifecycle:** prefill append, decode append, truncate (rollback),
+  release, TTL eviction, LRU eviction (by session cap and by byte budget), and an
+  **explicit** `CacheMiss` (unknown-session / evicted-ttl / evicted-lru /
+  released / superseded-epoch / seq-len-mismatch) so the head degrades to a
+  from-token-zero re-prefill instead of corrupting output (decision #14).
+- **Fails closed on identity.** Stale route epochs raise `StaleRouteEpochError`; a
+  request carrying an incompatible KV recipe raises `IncompatibleCacheRecipeError`
+  (fingerprint mismatch of architecture / kv dtype / head geometry / owned range);
+  a recipe for an uncertified architecture fails closed at construction (reusing
+  the DGR-006 certified-architecture gate).
+- **KV-aware boundary driver.** `KvBoundaryAdapter` wraps the DGR-006
+  `ShardComputation` (plus `run_layers_cached`) so a shard runs cached
+  prefill/decode through the manager while honouring the architecture-defined
+  boundary contract (head embeds tokens, middle/tail bypass embedding and consume
+  the unnormalized residual bundle, non-tail emits the unnormalized residual, tail
+  normalizes + heads + prunes + samples). The computation returns the new
+  position-encoded K/V; the manager commits it under the budget.
+
+A pure-numpy **KV-cached** dense-Llama reference (RMSNorm + RoPE + SwiGLU with an
+absolute-position causal mask over cached keys) proves that cached prefill/decode
+reproduces the stateless whole-model greedy tokens bit-for-bit, single-range and
+across a head/tail seam. torch/transformers are not installed in the default
+`.venv`, so a numpy reference is the only way to keep the parity + isolation gate
+deterministic, download-free, and GPU-free — the identical manager contract will
+be satisfied by the pinned llama.cpp worker (DGR-008), where the KV context maps
+onto a llama sequence.
+
+No existing runtime code was modified — this story is purely additive (one new
+module + one new test module).
+
+## Files changed (all new)
+
+- `packages/node/meshnet_node/hot_kv_state.py` — the KV/session manager:
+  - `KvCacheRecipe` — KV layout identity (certified architecture, kv dtype, head
+    geometry, owned range) with `fingerprint()` / `is_compatible()` /
+    `bytes_per_token()`; fails closed on uncertified architectures.
+  - `LayerKvCache` — per-owned-layer `(seq, n_kv_heads, head_dim)` K/V with
+    `append` / `truncate` / `nbytes`.
+  - `SessionCache` — the isolated per-`(session, epoch)` context over owned layers.
+  - `CacheMiss` / `CacheMissReason` — the explicit, serializable miss response.
+  - `HotKvStateManager` — `open` / `append` / `truncate` / `release` / `resolve` /
+    `get`, LRU+TTL+byte-budget eviction, stale-epoch + incompatible-recipe
+    rejection, epoch supersession, thread-safe (RLock), injectable clock.
+  - `KvBoundaryAdapter` + `kv_recipe_for()` — KV-aware boundary driver.
+- `tests/test_hot_kv_state.py` — pure-numpy KV-cached dense-Llama reference and 22
+  tests (see below).
+
+## Acceptance criteria → evidence
+
+- **Map `(Route Session ID, route epoch)` to an isolated context** —
+  `test_prefill_then_decode_append_grows_owned_layers`,
+  `test_four_interleaved_sessions_have_no_kv_cross_talk`,
+  `HotKvStateManager.open` keys sessions on `(session_id, route_epoch)`.
+- **Allocate KV only for owned layers** —
+  `test_manager_allocates_kv_only_for_owned_layers` (middle `[2,3]` → `{2,3}`),
+  `test_multi_range_cached_decode_parity_across_a_seam` (head owns `(0,1,2)`, tail
+  owns `(3,4,5)`), `test_recipe_bytes_per_token_scales_with_owned_layers`.
+- **Prefill append / decode append / truncate / release / TTL-LRU eviction /
+  explicit cache-miss** — `test_prefill_then_decode_append_grows_owned_layers`,
+  `test_truncate_rolls_back_all_owned_layers`,
+  `test_release_one_session_leaves_others_intact_and_returns_memory`,
+  `test_ttl_eviction_yields_an_explicit_cache_miss`,
+  `test_lru_eviction_by_session_cap_reports_a_miss`,
+  `test_budget_eviction_keeps_total_within_budget`,
+  `test_unknown_session_is_an_explicit_cache_miss`,
+  `test_seq_len_mismatch_is_an_explicit_cache_miss`.
+- **Reject stale epochs and incompatible cache recipes** —
+  `test_stale_route_epoch_is_rejected`,
+  `test_new_route_epoch_supersedes_and_frees_old_epoch`,
+  `test_incompatible_cache_recipe_is_rejected`,
+  `test_uncertified_architecture_recipe_fails_closed`.
+- **≥ four concurrent sessions complete without token or KV cross-talk** —
+  `test_four_interleaved_sessions_have_no_kv_cross_talk` (four interleaved
+  round-robin sessions, four *distinct* references, each matches its own),
+  `test_four_sessions_on_real_threads_stay_isolated` (four OS threads).
+- **Cancellation/release leaves others intact and memory returns to budget** —
+  `test_release_one_session_leaves_others_intact_and_returns_memory` (released
+  session → `CacheMiss(RELEASED)`, `total_bytes` drops, survivors keep matching
+  their references), `test_single_session_exceeding_budget_raises`.
+- **Cached vs stateless correctness core** —
+  `test_cached_full_shard_decode_matches_stateless_whole_model`,
+  `test_cached_prefill_next_token_matches_whole_model_logits`,
+  `test_multi_range_cached_decode_parity_across_a_seam`. Documented tolerance:
+  **identical** greedy token ids (bit-exact in practice; cached incremental
+  attention equals stateless full-sequence recompute per query row).
+- **Targeted pytest** — `22 passed`.
+- **compileall packages tests** — exit 0.
+- **git diff --check** — clean.
+- **Deterministic / download-free / credit-free / GPU-free** — pure numpy; fixed
+  RNG seed; injectable clock (no wall-clock in tests); no torch, no network, no
+  model files.
+- **Full deterministic pytest** — `13 failed, 755 passed, 14 skipped in 254.50s`.
+  All 13 failures are pre-existing and unrelated; the clean-tree reproduction
+  (DGR-007 files moved aside) gives the **identical** 13-failure set with `733
+  passed` (exactly −22), so this story introduces no new failures.
+- **Native C++ / CTest / llama.cpp patch stack** — **not touched by this story.**
+  The KV context contract is delivered at the Python manager level with a numpy
+  parity + isolation proof; the equivalent native layer-filtered KV / session
+  mapping is wired when the standalone C++ worker exists in DGR-008. No native
+  code, CMake, or llama.cpp patch was modified, so those gates are N/A here (same
+  as DGR-005/006).
+
+## Commands and real results
+
+```bash
+VP=/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python
+
+$VP -m pytest -q tests/test_hot_kv_state.py
+# -> 22 passed in ~0.3s
+
+$VP -m compileall -q packages tests
+# -> exit 0
+
+git diff --check
+# -> exit 0
+
+$VP -m pytest -q tests/test_boundary_adapter.py tests/test_gguf_ownership.py
+# -> 25 passed
+
+$VP -m pytest -q -rfE
+# -> 13 failed, 755 passed, 14 skipped in 254.50s
+
+# Clean-tree reproduction (DGR-007 files moved aside)
+mv packages/node/meshnet_node/hot_kv_state.py /tmp/ && mv tests/test_hot_kv_state.py /tmp/
+$VP -m pytest -q -rfE
+# -> 13 failed, 733 passed, 14 skipped in 252.12s  (identical FAILED set; passed -22)
+```
+
+`commands.txt` beside this README captures the exact commands.
+
+## Pre-existing unrelated failures (full-suite)
+
+`pytest -q -rfE` on `ralph/distributed-gguf-runtime` reports 13 pre-existing
+failures (and, in this run, 0 errors — the earlier DGR-005/006-era
+`test_native_shard_protocol.py` protobuf errors no longer appear in this
+environment). None touch the KV manager. Moving the two DGR-007 files aside and
+re-running yields the **byte-identical** 13-`FAILED` set (only the passed count
+drops by exactly 22). The exact set (all tracker/routing/benchmark/toploc/doctor,
+i.e. socket-bind / control-plane env, not KV):
+
+```
+tests/test_dynamic_routing.py::test_admin_can_replace_a_served_model_and_release_it
+tests/test_manual_route_benchmark.py::test_benchmark_records_one_and_two_node_routes
+tests/test_manual_route_benchmark.py::test_clients_without_route_are_unaffected
+tests/test_manual_route_benchmark.py::test_invalid_route_shape_is_400
+tests/test_manual_route_benchmark.py::test_pinned_route_uses_named_node
+tests/test_manual_route_benchmark.py::test_unknown_route_node_is_400
+tests/test_node_doctor.py::test_cli_doctor_flags_select_what_is_validated
+tests/test_toploc_calibration_dispatch.py::test_calibration_run_dispatches_only_solo_capable_nodes
+tests/test_toploc_calibration_dispatch.py::test_calibration_run_node_without_commitment_endpoint_is_skipped_not_failed
+tests/test_toploc_calibration_dispatch.py::test_calibration_run_persists_corpus_and_results_endpoint_reports_it
+tests/test_tracker_capability_admission.py::test_an_enforcing_tracker_never_routes_a_node_whose_proof_does_not_cover_it[invalid]
+tests/test_tracker_routing.py::test_shard_heal_cycle_surviving_node_covers_dead_peers_gap
+tests/test_tracker_routing.py::test_torch_node_applies_tracker_load_shard_directive
+```
+
+## Limitations and deferred work
+
+- **Numpy reference, not real weights.** The parity + isolation gate uses a
+  deterministic numpy KV-cached dense-Llama, not a downloaded GGUF/safetensors
+  model. Real-model concurrent KV isolation on a downloaded dense-Llama (CPU/ROCm)
+  belongs to DGR-010/DGR-012 with `MESHNET_ENABLE_REAL_INFERENCE_TESTS=1` and
+  `.venv-rocm`.
+- **Manager-owned storage, native mapping deferred.** The KV bytes are numpy
+  arrays managed in-process. The llama.cpp expression (a filtered llama sequence
+  per `(session, epoch)` over owned layers) is implemented in the standalone
+  worker (DGR-008) against this same manager contract; no native code was touched.
+- **Continuous batching is DGR-012.** This story delivers *isolation* and bounded
+  lifecycle for concurrent sessions; continuous batching of compatible active
+  sessions inside a node (decision #9) is DGR-012 and builds on this manager.
+- **Greedy-only sampling.** Reuses the DGR-006 `SamplingContract` (greedy
+  certified). Stochastic sampling is out of scope for the deterministic gate.
+- **Coexists with legacy `SessionCacheStore`.** The older AH-25
+  `model_backend.SessionCacheStore` (session-id-only, opaque transformers cache,
+  HTTP path) is untouched. `HotKvStateManager` is the native-runtime-aligned
+  successor: it adds route-epoch keying, owned-layer allocation, recipe-fingerprint
+  rejection, and a byte budget. DGR-008/009 wire the native worker to
+  `HotKvStateManager`, not `SessionCacheStore`.
+
+## Compatibility / migration notes
+
+- `KvCacheRecipe.fingerprint()` canonicalizes the architecture (via
+  `certified_architecture`), so `llama` / `LlamaForCausalLM` map to the same
+  recipe; it aligns field-for-field with the DGR-003 `RuntimeRecipeIdentity`
+  compatibility discipline and reuses `runtime_recipe.compatibility_fingerprint`.
+- `CacheMiss` is a value (not an exception) so it can be serialized into the
+  DGR-002 native protocol's cache expectation/result field; `resolve()` returns it,
+  `get()` raises `KvCacheMissError` wrapping it.
+- The manager takes an injectable `clock` for deterministic TTL tests; production
+  defaults to `time.monotonic`.
+
+## Handoff for dependent stories
+
+- **DGR-008 (C++ gRPC worker):** implement the servicer's KV path against
+  `HotKvStateManager`. Map each `(Route Session ID, route epoch)` to a filtered
+  llama sequence over owned layers; on decode, read the sequence's cached K/V,
+  compute the new position-encoded K/V, and commit via `append` (honour the byte
+  budget and return an explicit `CacheMiss` on eviction). Enforce
+  `KvCacheRecipe.is_compatible` before activation and reject stale epochs.
+- **DGR-009 (Meshnet integration):** the route epoch the tracker assigns is the
+  `route_epoch` key; carry the `CacheMiss` reason back to the head so it re-prefills
+  from token zero on eviction/restart.
+- **DGR-012 (continuous batching):** batch compatible active sessions whose
+  `KvCacheRecipe` fingerprints match; each session keeps its own `SessionCache`, so
+  batching is a scheduling concern layered over this isolation, not a change to it.
+- **DGR-013 (failure/cancel matrix):** `release` + the budget-return assertion here
+  is the unit-level basis for the resource-cleanup matrix.

.scratch/distributed-gguf-runtime/evidence/DGR-007/commands.txt

@@ -0,0 +1,31 @@
+# DGR-007 — exact commands (run from the worktree root).
+# Python: /run/media/popov/d/DEV/repos/d-popov.com/AI/.venv (Python 3.14.6, numpy 2.4.4).
+# Root conftest.py adds packages/* to sys.path, so `meshnet_node` imports work.
+
+VP=/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python
+
+# Targeted tests for this story.
+$VP -m pytest -q tests/test_hot_kv_state.py
+# -> 22 passed
+
+# Python compile check for the changed packages/tests.
+$VP -m compileall -q packages tests
+# -> exit 0
+
+# Diff hygiene.
+git diff --check
+# -> exit 0
+
+# Dependency (DGR-006) + range-ownership (DGR-005) tests still green.
+$VP -m pytest -q tests/test_boundary_adapter.py tests/test_gguf_ownership.py
+# -> 25 passed
+
+# Full deterministic suite (with DGR-007 files present).
+$VP -m pytest -q -rfE
+# -> see README (pre-existing unrelated failure set, +22 passed vs baseline)
+
+# Clean-tree reproduction (DGR-007 files moved aside).
+mv packages/node/meshnet_node/hot_kv_state.py /tmp/ && mv tests/test_hot_kv_state.py /tmp/
+$VP -m pytest -q -rfE
+# -> identical failure/error set, passed count drops by exactly 22
+mv /tmp/hot_kv_state.py packages/node/meshnet_node/ && mv /tmp/test_hot_kv_state.py tests/

.scratch/distributed-gguf-runtime/evidence/DGR-007/results.json

@@ -0,0 +1,47 @@
+{
+  "task_id": "DGR-007",
+  "title": "Add isolated concurrent local Hot KV State",
+  "status": "done",
+  "date": "2026-07-15",
+  "evidence_kind": "synthetic-unit",
+  "python": "/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv (Python 3.14.6, numpy 2.4.4)",
+  "files_changed": [
+    "packages/node/meshnet_node/hot_kv_state.py",
+    "tests/test_hot_kv_state.py"
+  ],
+  "gates": {
+    "targeted_pytest": {"command": "pytest -q tests/test_hot_kv_state.py", "result": "22 passed"},
+    "compileall": {"command": "python -m compileall -q packages tests", "exit": 0},
+    "git_diff_check": {"command": "git diff --check", "exit": 0},
+    "dependency_tests": {"command": "pytest -q tests/test_boundary_adapter.py tests/test_gguf_ownership.py", "result": "25 passed"},
+    "full_suite_with_files": {"command": "pytest -q -rfE", "result": "13 failed, 755 passed, 14 skipped", "seconds": 254.50},
+    "full_suite_clean_tree": {"command": "pytest -q -rfE (DGR-007 files moved aside)", "result": "13 failed, 733 passed, 14 skipped", "seconds": 252.12}
+  },
+  "no_new_failures": true,
+  "failure_set_identical": true,
+  "passed_delta": 22,
+  "preexisting_failures": [
+    "tests/test_dynamic_routing.py::test_admin_can_replace_a_served_model_and_release_it",
+    "tests/test_manual_route_benchmark.py::test_benchmark_records_one_and_two_node_routes",
+    "tests/test_manual_route_benchmark.py::test_clients_without_route_are_unaffected",
+    "tests/test_manual_route_benchmark.py::test_invalid_route_shape_is_400",
+    "tests/test_manual_route_benchmark.py::test_pinned_route_uses_named_node",
+    "tests/test_manual_route_benchmark.py::test_unknown_route_node_is_400",
+    "tests/test_node_doctor.py::test_cli_doctor_flags_select_what_is_validated",
+    "tests/test_toploc_calibration_dispatch.py::test_calibration_run_dispatches_only_solo_capable_nodes",
+    "tests/test_toploc_calibration_dispatch.py::test_calibration_run_node_without_commitment_endpoint_is_skipped_not_failed",
+    "tests/test_toploc_calibration_dispatch.py::test_calibration_run_persists_corpus_and_results_endpoint_reports_it",
+    "tests/test_tracker_capability_admission.py::test_an_enforcing_tracker_never_routes_a_node_whose_proof_does_not_cover_it[invalid]",
+    "tests/test_tracker_routing.py::test_shard_heal_cycle_surviving_node_covers_dead_peers_gap",
+    "tests/test_tracker_routing.py::test_torch_node_applies_tracker_load_shard_directive"
+  ],
+  "native_gates_touched": false,
+  "acceptance": {
+    "session_epoch_isolated_context": true,
+    "kv_only_owned_layers": true,
+    "prefill_decode_truncate_release_ttl_lru_cachemiss": true,
+    "reject_stale_epoch_and_incompatible_recipe": true,
+    "four_concurrent_sessions_no_crosstalk": true,
+    "release_leaves_others_and_returns_memory": true
+  }
+}