# DGR-007 — Isolated concurrent local Hot KV State: evidence

Status: done
Date: 2026-07-15
Evidence kind: **synthetic-unit** (pure-numpy KV-cached dense-Llama reference +
session/KV manager). No model download, no GPU, no torch, no network, no API
credit.

## Summary

Implemented the local Hot KV State manager that maps every
`(Route Session ID, route epoch)` to an isolated, bounded KV context (RALPH
runtime decisions #7 and #8, ADR-0022/0024). The manager owns all cache
mutation, so eviction, byte accounting, and isolation live in one place instead
of being scattered across backends:

- **`(session_id, route_epoch)` → isolated context.** Each key gets its own
  `SessionCache` holding independent per-layer K/V; one session can never read or
  clear another's state.
- **KV allocated only for owned layers.** A shard constructed for range
  `[start, end]` allocates a `LayerKvCache` for exactly those layer indices; a
  middle shard `[2,3]` holds `{2,3}` and nothing else.
- **Full lifecycle:** prefill append, decode append, truncate (rollback),
  release, TTL eviction, LRU eviction (by session cap and by byte budget), and an
  **explicit** `CacheMiss` (unknown-session / evicted-ttl / evicted-lru /
  released / superseded-epoch / seq-len-mismatch) so the head degrades to a
  from-token-zero re-prefill instead of corrupting output (decision #14).
- **Fails closed on identity.** Stale route epochs raise `StaleRouteEpochError`; a
  request carrying an incompatible KV recipe raises `IncompatibleCacheRecipeError`
  (fingerprint mismatch of architecture / kv dtype / head geometry / owned range);
  a recipe for an uncertified architecture fails closed at construction (reusing
  the DGR-006 certified-architecture gate).
- **KV-aware boundary driver.** `KvBoundaryAdapter` wraps the DGR-006
  `ShardComputation` (plus `run_layers_cached`) so a shard runs cached
  prefill/decode through the manager while honouring the architecture-defined
  boundary contract (head embeds tokens, middle/tail bypass embedding and consume
  the unnormalized residual bundle, non-tail emits the unnormalized residual, tail
  normalizes + heads + prunes + samples). The computation returns the new
  position-encoded K/V; the manager commits it under the budget.

A pure-numpy **KV-cached** dense-Llama reference (RMSNorm + RoPE + SwiGLU with an
absolute-position causal mask over cached keys) proves that cached prefill/decode
reproduces the stateless whole-model greedy tokens bit-for-bit, single-range and
across a head/tail seam. torch/transformers are not installed in the default
`.venv`, so a numpy reference is the only way to keep the parity + isolation gate
deterministic, download-free, and GPU-free — the identical manager contract will
be satisfied by the pinned llama.cpp worker (DGR-008), where the KV context maps
onto a llama sequence.

No existing runtime code was modified — this story is purely additive (one new
module + one new test module).

## Files changed (all new)

- `packages/node/meshnet_node/hot_kv_state.py` — the KV/session manager:
  - `KvCacheRecipe` — KV layout identity (certified architecture, kv dtype, head
    geometry, owned range) with `fingerprint()` / `is_compatible()` /
    `bytes_per_token()`; fails closed on uncertified architectures.
  - `LayerKvCache` — per-owned-layer `(seq, n_kv_heads, head_dim)` K/V with
    `append` / `truncate` / `nbytes`.
  - `SessionCache` — the isolated per-`(session, epoch)` context over owned layers.
  - `CacheMiss` / `CacheMissReason` — the explicit, serializable miss response.
  - `HotKvStateManager` — `open` / `append` / `truncate` / `release` / `resolve` /
    `get`, LRU+TTL+byte-budget eviction, stale-epoch + incompatible-recipe
    rejection, epoch supersession, thread-safe (RLock), injectable clock.
  - `KvBoundaryAdapter` + `kv_recipe_for()` — KV-aware boundary driver.
- `tests/test_hot_kv_state.py` — pure-numpy KV-cached dense-Llama reference and 22
  tests (see below).

## Acceptance criteria → evidence

- **Map `(Route Session ID, route epoch)` to an isolated context** —
  `test_prefill_then_decode_append_grows_owned_layers`,
  `test_four_interleaved_sessions_have_no_kv_cross_talk`,
  `HotKvStateManager.open` keys sessions on `(session_id, route_epoch)`.
- **Allocate KV only for owned layers** —
  `test_manager_allocates_kv_only_for_owned_layers` (middle `[2,3]` → `{2,3}`),
  `test_multi_range_cached_decode_parity_across_a_seam` (head owns `(0,1,2)`, tail
  owns `(3,4,5)`), `test_recipe_bytes_per_token_scales_with_owned_layers`.
- **Prefill append / decode append / truncate / release / TTL-LRU eviction /
  explicit cache-miss** — `test_prefill_then_decode_append_grows_owned_layers`,
  `test_truncate_rolls_back_all_owned_layers`,
  `test_release_one_session_leaves_others_intact_and_returns_memory`,
  `test_ttl_eviction_yields_an_explicit_cache_miss`,
  `test_lru_eviction_by_session_cap_reports_a_miss`,
  `test_budget_eviction_keeps_total_within_budget`,
  `test_unknown_session_is_an_explicit_cache_miss`,
  `test_seq_len_mismatch_is_an_explicit_cache_miss`.
- **Reject stale epochs and incompatible cache recipes** —
  `test_stale_route_epoch_is_rejected`,
  `test_new_route_epoch_supersedes_and_frees_old_epoch`,
  `test_incompatible_cache_recipe_is_rejected`,
  `test_uncertified_architecture_recipe_fails_closed`.
- **≥ four concurrent sessions complete without token or KV cross-talk** —
  `test_four_interleaved_sessions_have_no_kv_cross_talk` (four interleaved
  round-robin sessions, four *distinct* references, each matches its own),
  `test_four_sessions_on_real_threads_stay_isolated` (four OS threads).
- **Cancellation/release leaves others intact and memory returns to budget** —
  `test_release_one_session_leaves_others_intact_and_returns_memory` (released
  session → `CacheMiss(RELEASED)`, `total_bytes` drops, survivors keep matching
  their references), `test_single_session_exceeding_budget_raises`.
- **Cached vs stateless correctness core** —
  `test_cached_full_shard_decode_matches_stateless_whole_model`,
  `test_cached_prefill_next_token_matches_whole_model_logits`,
  `test_multi_range_cached_decode_parity_across_a_seam`. Documented tolerance:
  **identical** greedy token ids (bit-exact in practice; cached incremental
  attention equals stateless full-sequence recompute per query row).
- **Targeted pytest** — `22 passed`.
- **compileall packages tests** — exit 0.
- **git diff --check** — clean.
- **Deterministic / download-free / credit-free / GPU-free** — pure numpy; fixed
  RNG seed; injectable clock (no wall-clock in tests); no torch, no network, no
  model files.
- **Full deterministic pytest** — `13 failed, 755 passed, 14 skipped in 254.50s`.
  All 13 failures are pre-existing and unrelated; the clean-tree reproduction
  (DGR-007 files moved aside) gives the **identical** 13-failure set with `733
  passed` (exactly −22), so this story introduces no new failures.
- **Native C++ / CTest / llama.cpp patch stack** — **not touched by this story.**
  The KV context contract is delivered at the Python manager level with a numpy
  parity + isolation proof; the equivalent native layer-filtered KV / session
  mapping is wired when the standalone C++ worker exists in DGR-008. No native
  code, CMake, or llama.cpp patch was modified, so those gates are N/A here (same
  as DGR-005/006).

## Commands and real results

```bash
VP=/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python

$VP -m pytest -q tests/test_hot_kv_state.py
# -> 22 passed in ~0.3s

$VP -m compileall -q packages tests
# -> exit 0

git diff --check
# -> exit 0

$VP -m pytest -q tests/test_boundary_adapter.py tests/test_gguf_ownership.py
# -> 25 passed

$VP -m pytest -q -rfE
# -> 13 failed, 755 passed, 14 skipped in 254.50s

# Clean-tree reproduction (DGR-007 files moved aside)
mv packages/node/meshnet_node/hot_kv_state.py /tmp/ && mv tests/test_hot_kv_state.py /tmp/
$VP -m pytest -q -rfE
# -> 13 failed, 733 passed, 14 skipped in 252.12s  (identical FAILED set; passed -22)
```

`commands.txt` beside this README captures the exact commands.

## Pre-existing unrelated failures (full-suite)

`pytest -q -rfE` on `ralph/distributed-gguf-runtime` reports 13 pre-existing
failures (and, in this run, 0 errors — the earlier DGR-005/006-era
`test_native_shard_protocol.py` protobuf errors no longer appear in this
environment). None touch the KV manager. Moving the two DGR-007 files aside and
re-running yields the **byte-identical** 13-`FAILED` set (only the passed count
drops by exactly 22). The exact set (all tracker/routing/benchmark/toploc/doctor,
i.e. socket-bind / control-plane env, not KV):

```
tests/test_dynamic_routing.py::test_admin_can_replace_a_served_model_and_release_it
tests/test_manual_route_benchmark.py::test_benchmark_records_one_and_two_node_routes
tests/test_manual_route_benchmark.py::test_clients_without_route_are_unaffected
tests/test_manual_route_benchmark.py::test_invalid_route_shape_is_400
tests/test_manual_route_benchmark.py::test_pinned_route_uses_named_node
tests/test_manual_route_benchmark.py::test_unknown_route_node_is_400
tests/test_node_doctor.py::test_cli_doctor_flags_select_what_is_validated
tests/test_toploc_calibration_dispatch.py::test_calibration_run_dispatches_only_solo_capable_nodes
tests/test_toploc_calibration_dispatch.py::test_calibration_run_node_without_commitment_endpoint_is_skipped_not_failed
tests/test_toploc_calibration_dispatch.py::test_calibration_run_persists_corpus_and_results_endpoint_reports_it
tests/test_tracker_capability_admission.py::test_an_enforcing_tracker_never_routes_a_node_whose_proof_does_not_cover_it[invalid]
tests/test_tracker_routing.py::test_shard_heal_cycle_surviving_node_covers_dead_peers_gap
tests/test_tracker_routing.py::test_torch_node_applies_tracker_load_shard_directive
```

## Limitations and deferred work

- **Numpy reference, not real weights.** The parity + isolation gate uses a
  deterministic numpy KV-cached dense-Llama, not a downloaded GGUF/safetensors
  model. Real-model concurrent KV isolation on a downloaded dense-Llama (CPU/ROCm)
  belongs to DGR-010/DGR-012 with `MESHNET_ENABLE_REAL_INFERENCE_TESTS=1` and
  `.venv-rocm`.
- **Manager-owned storage, native mapping deferred.** The KV bytes are numpy
  arrays managed in-process. The llama.cpp expression (a filtered llama sequence
  per `(session, epoch)` over owned layers) is implemented in the standalone
  worker (DGR-008) against this same manager contract; no native code was touched.
- **Continuous batching is DGR-012.** This story delivers *isolation* and bounded
  lifecycle for concurrent sessions; continuous batching of compatible active
  sessions inside a node (decision #9) is DGR-012 and builds on this manager.
- **Greedy-only sampling.** Reuses the DGR-006 `SamplingContract` (greedy
  certified). Stochastic sampling is out of scope for the deterministic gate.
- **Coexists with legacy `SessionCacheStore`.** The older AH-25
  `model_backend.SessionCacheStore` (session-id-only, opaque transformers cache,
  HTTP path) is untouched. `HotKvStateManager` is the native-runtime-aligned
  successor: it adds route-epoch keying, owned-layer allocation, recipe-fingerprint
  rejection, and a byte budget. DGR-008/009 wire the native worker to
  `HotKvStateManager`, not `SessionCacheStore`.

## Compatibility / migration notes

- `KvCacheRecipe.fingerprint()` canonicalizes the architecture (via
  `certified_architecture`), so `llama` / `LlamaForCausalLM` map to the same
  recipe; it aligns field-for-field with the DGR-003 `RuntimeRecipeIdentity`
  compatibility discipline and reuses `runtime_recipe.compatibility_fingerprint`.
- `CacheMiss` is a value (not an exception) so it can be serialized into the
  DGR-002 native protocol's cache expectation/result field; `resolve()` returns it,
  `get()` raises `KvCacheMissError` wrapping it.
- The manager takes an injectable `clock` for deterministic TTL tests; production
  defaults to `time.monotonic`.

## Handoff for dependent stories

- **DGR-008 (C++ gRPC worker):** implement the servicer's KV path against
  `HotKvStateManager`. Map each `(Route Session ID, route epoch)` to a filtered
  llama sequence over owned layers; on decode, read the sequence's cached K/V,
  compute the new position-encoded K/V, and commit via `append` (honour the byte
  budget and return an explicit `CacheMiss` on eviction). Enforce
  `KvCacheRecipe.is_compatible` before activation and reject stale epochs.
- **DGR-009 (Meshnet integration):** the route epoch the tracker assigns is the
  `route_epoch` key; carry the `CacheMiss` reason back to the head so it re-prefills
  from token zero on eviction/restart.
- **DGR-012 (continuous batching):** batch compatible active sessions whose
  `KvCacheRecipe` fingerprints match; each session keeps its own `SessionCache`, so
  batching is a scheduling concern layered over this isolation, not a change to it.
- **DGR-013 (failure/cancel matrix):** `release` + the budget-return assertion here
  is the unit-level basis for the resource-cleanup matrix.