DGR-007 — Isolated concurrent local Hot KV State: evidence

Status: done Date: 2026-07-15 Evidence kind: synthetic-unit (pure-numpy KV-cached dense-Llama reference + session/KV manager). No model download, no GPU, no torch, no network, no API credit.

Summary

Implemented the local Hot KV State manager that maps every (Route Session ID, route epoch) to an isolated, bounded KV context (RALPH runtime decisions #7 and #8, ADR-0022/0024). The manager owns all cache mutation, so eviction, byte accounting, and isolation live in one place instead of being scattered across backends:

(session_id, route_epoch) → isolated context. Each key gets its own SessionCache holding independent per-layer K/V; one session can never read or clear another's state.
KV allocated only for owned layers. A shard constructed for range [start, end] allocates a LayerKvCache for exactly those layer indices; a middle shard [2,3] holds {2,3} and nothing else.
Full lifecycle: prefill append, decode append, truncate (rollback), release, TTL eviction, LRU eviction (by session cap and by byte budget), and an explicit CacheMiss (unknown-session / evicted-ttl / evicted-lru / released / superseded-epoch / seq-len-mismatch) so the head degrades to a from-token-zero re-prefill instead of corrupting output (decision #14).
Fails closed on identity. Stale route epochs raise StaleRouteEpochError; a request carrying an incompatible KV recipe raises IncompatibleCacheRecipeError (fingerprint mismatch of architecture / kv dtype / head geometry / owned range); a recipe for an uncertified architecture fails closed at construction (reusing the DGR-006 certified-architecture gate).
KV-aware boundary driver. KvBoundaryAdapter wraps the DGR-006 ShardComputation (plus run_layers_cached) so a shard runs cached prefill/decode through the manager while honouring the architecture-defined boundary contract (head embeds tokens, middle/tail bypass embedding and consume the unnormalized residual bundle, non-tail emits the unnormalized residual, tail normalizes + heads + prunes + samples). The computation returns the new position-encoded K/V; the manager commits it under the budget.

A pure-numpy KV-cached dense-Llama reference (RMSNorm + RoPE + SwiGLU with an absolute-position causal mask over cached keys) proves that cached prefill/decode reproduces the stateless whole-model greedy tokens bit-for-bit, single-range and across a head/tail seam. torch/transformers are not installed in the default .venv, so a numpy reference is the only way to keep the parity + isolation gate deterministic, download-free, and GPU-free — the identical manager contract will be satisfied by the pinned llama.cpp worker (DGR-008), where the KV context maps onto a llama sequence.

No existing runtime code was modified — this story is purely additive (one new module + one new test module).

Files changed (all new)

packages/node/meshnet_node/hot_kv_state.py — the KV/session manager:
- KvCacheRecipe — KV layout identity (certified architecture, kv dtype, head geometry, owned range) with fingerprint() / is_compatible() / bytes_per_token(); fails closed on uncertified architectures.
- LayerKvCache — per-owned-layer (seq, n_kv_heads, head_dim) K/V with append / truncate / nbytes.
- SessionCache — the isolated per-(session, epoch) context over owned layers.
- CacheMiss / CacheMissReason — the explicit, serializable miss response.
- HotKvStateManager — open / append / truncate / release / resolve / get, LRU+TTL+byte-budget eviction, stale-epoch + incompatible-recipe rejection, epoch supersession, thread-safe (RLock), injectable clock.
- KvBoundaryAdapter + kv_recipe_for() — KV-aware boundary driver.
tests/test_hot_kv_state.py — pure-numpy KV-cached dense-Llama reference and 22 tests (see below).

Acceptance criteria → evidence

Map (Route Session ID, route epoch) to an isolated context — test_prefill_then_decode_append_grows_owned_layers, test_four_interleaved_sessions_have_no_kv_cross_talk, HotKvStateManager.open keys sessions on (session_id, route_epoch).
Allocate KV only for owned layers — test_manager_allocates_kv_only_for_owned_layers (middle [2,3] → {2,3}), test_multi_range_cached_decode_parity_across_a_seam (head owns (0,1,2), tail owns (3,4,5)), test_recipe_bytes_per_token_scales_with_owned_layers.
Prefill append / decode append / truncate / release / TTL-LRU eviction / explicit cache-miss — test_prefill_then_decode_append_grows_owned_layers, test_truncate_rolls_back_all_owned_layers, test_release_one_session_leaves_others_intact_and_returns_memory, test_ttl_eviction_yields_an_explicit_cache_miss, test_lru_eviction_by_session_cap_reports_a_miss, test_budget_eviction_keeps_total_within_budget, test_unknown_session_is_an_explicit_cache_miss, test_seq_len_mismatch_is_an_explicit_cache_miss.
Reject stale epochs and incompatible cache recipes — test_stale_route_epoch_is_rejected, test_new_route_epoch_supersedes_and_frees_old_epoch, test_incompatible_cache_recipe_is_rejected, test_uncertified_architecture_recipe_fails_closed.
≥ four concurrent sessions complete without token or KV cross-talk — test_four_interleaved_sessions_have_no_kv_cross_talk (four interleaved round-robin sessions, four distinct references, each matches its own), test_four_sessions_on_real_threads_stay_isolated (four OS threads).
Cancellation/release leaves others intact and memory returns to budget — test_release_one_session_leaves_others_intact_and_returns_memory (released session → CacheMiss(RELEASED), total_bytes drops, survivors keep matching their references), test_single_session_exceeding_budget_raises.
Cached vs stateless correctness core — test_cached_full_shard_decode_matches_stateless_whole_model, test_cached_prefill_next_token_matches_whole_model_logits, test_multi_range_cached_decode_parity_across_a_seam. Documented tolerance: identical greedy token ids (bit-exact in practice; cached incremental attention equals stateless full-sequence recompute per query row).
Targeted pytest — 22 passed.
compileall packages tests — exit 0.
git diff --check — clean.
Deterministic / download-free / credit-free / GPU-free — pure numpy; fixed RNG seed; injectable clock (no wall-clock in tests); no torch, no network, no model files.
Full deterministic pytest — 13 failed, 755 passed, 14 skipped in 254.50s. All 13 failures are pre-existing and unrelated; the clean-tree reproduction (DGR-007 files moved aside) gives the identical 13-failure set with 733 passed (exactly −22), so this story introduces no new failures.
Native C++ / CTest / llama.cpp patch stack — not touched by this story. The KV context contract is delivered at the Python manager level with a numpy parity + isolation proof; the equivalent native layer-filtered KV / session mapping is wired when the standalone C++ worker exists in DGR-008. No native code, CMake, or llama.cpp patch was modified, so those gates are N/A here (same as DGR-005/006).

Commands and real results

VP=/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python

$VP -m pytest -q tests/test_hot_kv_state.py
# -> 22 passed in ~0.3s

$VP -m compileall -q packages tests
# -> exit 0

git diff --check
# -> exit 0

$VP -m pytest -q tests/test_boundary_adapter.py tests/test_gguf_ownership.py
# -> 25 passed

$VP -m pytest -q -rfE
# -> 13 failed, 755 passed, 14 skipped in 254.50s

# Clean-tree reproduction (DGR-007 files moved aside)
mv packages/node/meshnet_node/hot_kv_state.py /tmp/ && mv tests/test_hot_kv_state.py /tmp/
$VP -m pytest -q -rfE
# -> 13 failed, 733 passed, 14 skipped in 252.12s  (identical FAILED set; passed -22)

commands.txt beside this README captures the exact commands.

Pre-existing unrelated failures (full-suite)

pytest -q -rfE on ralph/distributed-gguf-runtime reports 13 pre-existing failures (and, in this run, 0 errors — the earlier DGR-005/006-era test_native_shard_protocol.py protobuf errors no longer appear in this environment). None touch the KV manager. Moving the two DGR-007 files aside and re-running yields the byte-identical 13-FAILED set (only the passed count drops by exactly 22). The exact set (all tracker/routing/benchmark/toploc/doctor, i.e. socket-bind / control-plane env, not KV):

tests/test_dynamic_routing.py::test_admin_can_replace_a_served_model_and_release_it
tests/test_manual_route_benchmark.py::test_benchmark_records_one_and_two_node_routes
tests/test_manual_route_benchmark.py::test_clients_without_route_are_unaffected
tests/test_manual_route_benchmark.py::test_invalid_route_shape_is_400
tests/test_manual_route_benchmark.py::test_pinned_route_uses_named_node
tests/test_manual_route_benchmark.py::test_unknown_route_node_is_400
tests/test_node_doctor.py::test_cli_doctor_flags_select_what_is_validated
tests/test_toploc_calibration_dispatch.py::test_calibration_run_dispatches_only_solo_capable_nodes
tests/test_toploc_calibration_dispatch.py::test_calibration_run_node_without_commitment_endpoint_is_skipped_not_failed
tests/test_toploc_calibration_dispatch.py::test_calibration_run_persists_corpus_and_results_endpoint_reports_it
tests/test_tracker_capability_admission.py::test_an_enforcing_tracker_never_routes_a_node_whose_proof_does_not_cover_it[invalid]
tests/test_tracker_routing.py::test_shard_heal_cycle_surviving_node_covers_dead_peers_gap
tests/test_tracker_routing.py::test_torch_node_applies_tracker_load_shard_directive

Limitations and deferred work

Numpy reference, not real weights. The parity + isolation gate uses a deterministic numpy KV-cached dense-Llama, not a downloaded GGUF/safetensors model. Real-model concurrent KV isolation on a downloaded dense-Llama (CPU/ROCm) belongs to DGR-010/DGR-012 with MESHNET_ENABLE_REAL_INFERENCE_TESTS=1 and .venv-rocm.
Manager-owned storage, native mapping deferred. The KV bytes are numpy arrays managed in-process. The llama.cpp expression (a filtered llama sequence per (session, epoch) over owned layers) is implemented in the standalone worker (DGR-008) against this same manager contract; no native code was touched.
Continuous batching is DGR-012. This story delivers isolation and bounded lifecycle for concurrent sessions; continuous batching of compatible active sessions inside a node (decision #9) is DGR-012 and builds on this manager.
Greedy-only sampling. Reuses the DGR-006 SamplingContract (greedy certified). Stochastic sampling is out of scope for the deterministic gate.
Coexists with legacy SessionCacheStore. The older AH-25 model_backend.SessionCacheStore (session-id-only, opaque transformers cache, HTTP path) is untouched. HotKvStateManager is the native-runtime-aligned successor: it adds route-epoch keying, owned-layer allocation, recipe-fingerprint rejection, and a byte budget. DGR-008/009 wire the native worker to HotKvStateManager, not SessionCacheStore.

Compatibility / migration notes

KvCacheRecipe.fingerprint() canonicalizes the architecture (via certified_architecture), so llama / LlamaForCausalLM map to the same recipe; it aligns field-for-field with the DGR-003 RuntimeRecipeIdentity compatibility discipline and reuses runtime_recipe.compatibility_fingerprint.
CacheMiss is a value (not an exception) so it can be serialized into the DGR-002 native protocol's cache expectation/result field; resolve() returns it, get() raises KvCacheMissError wrapping it.
The manager takes an injectable clock for deterministic TTL tests; production defaults to time.monotonic.

Handoff for dependent stories

DGR-008 (C++ gRPC worker): implement the servicer's KV path against HotKvStateManager. Map each (Route Session ID, route epoch) to a filtered llama sequence over owned layers; on decode, read the sequence's cached K/V, compute the new position-encoded K/V, and commit via append (honour the byte budget and return an explicit CacheMiss on eviction). Enforce KvCacheRecipe.is_compatible before activation and reject stale epochs.
DGR-009 (Meshnet integration): the route epoch the tracker assigns is the route_epoch key; carry the CacheMiss reason back to the head so it re-prefills from token zero on eviction/restart.
DGR-012 (continuous batching): batch compatible active sessions whose KvCacheRecipe fingerprints match; each session keeps its own SessionCache, so batching is a scheduling concern layered over this isolation, not a change to it.
DGR-013 (failure/cancel matrix): release + the budget-return assertion here is the unit-level basis for the resource-cleanup matrix.

13 KiB Raw Blame History Unescape Escape