230 lines
13 KiB
Markdown
230 lines
13 KiB
Markdown
# DGR-007 — Isolated concurrent local Hot KV State: evidence
|
||
|
||
Status: done
|
||
Date: 2026-07-15
|
||
Evidence kind: **synthetic-unit** (pure-numpy KV-cached dense-Llama reference +
|
||
session/KV manager). No model download, no GPU, no torch, no network, no API
|
||
credit.
|
||
|
||
## Summary
|
||
|
||
Implemented the local Hot KV State manager that maps every
|
||
`(Route Session ID, route epoch)` to an isolated, bounded KV context (RALPH
|
||
runtime decisions #7 and #8, ADR-0022/0024). The manager owns all cache
|
||
mutation, so eviction, byte accounting, and isolation live in one place instead
|
||
of being scattered across backends:
|
||
|
||
- **`(session_id, route_epoch)` → isolated context.** Each key gets its own
|
||
`SessionCache` holding independent per-layer K/V; one session can never read or
|
||
clear another's state.
|
||
- **KV allocated only for owned layers.** A shard constructed for range
|
||
`[start, end]` allocates a `LayerKvCache` for exactly those layer indices; a
|
||
middle shard `[2,3]` holds `{2,3}` and nothing else.
|
||
- **Full lifecycle:** prefill append, decode append, truncate (rollback),
|
||
release, TTL eviction, LRU eviction (by session cap and by byte budget), and an
|
||
**explicit** `CacheMiss` (unknown-session / evicted-ttl / evicted-lru /
|
||
released / superseded-epoch / seq-len-mismatch) so the head degrades to a
|
||
from-token-zero re-prefill instead of corrupting output (decision #14).
|
||
- **Fails closed on identity.** Stale route epochs raise `StaleRouteEpochError`; a
|
||
request carrying an incompatible KV recipe raises `IncompatibleCacheRecipeError`
|
||
(fingerprint mismatch of architecture / kv dtype / head geometry / owned range);
|
||
a recipe for an uncertified architecture fails closed at construction (reusing
|
||
the DGR-006 certified-architecture gate).
|
||
- **KV-aware boundary driver.** `KvBoundaryAdapter` wraps the DGR-006
|
||
`ShardComputation` (plus `run_layers_cached`) so a shard runs cached
|
||
prefill/decode through the manager while honouring the architecture-defined
|
||
boundary contract (head embeds tokens, middle/tail bypass embedding and consume
|
||
the unnormalized residual bundle, non-tail emits the unnormalized residual, tail
|
||
normalizes + heads + prunes + samples). The computation returns the new
|
||
position-encoded K/V; the manager commits it under the budget.
|
||
|
||
A pure-numpy **KV-cached** dense-Llama reference (RMSNorm + RoPE + SwiGLU with an
|
||
absolute-position causal mask over cached keys) proves that cached prefill/decode
|
||
reproduces the stateless whole-model greedy tokens bit-for-bit, single-range and
|
||
across a head/tail seam. torch/transformers are not installed in the default
|
||
`.venv`, so a numpy reference is the only way to keep the parity + isolation gate
|
||
deterministic, download-free, and GPU-free — the identical manager contract will
|
||
be satisfied by the pinned llama.cpp worker (DGR-008), where the KV context maps
|
||
onto a llama sequence.
|
||
|
||
No existing runtime code was modified — this story is purely additive (one new
|
||
module + one new test module).
|
||
|
||
## Files changed (all new)
|
||
|
||
- `packages/node/meshnet_node/hot_kv_state.py` — the KV/session manager:
|
||
- `KvCacheRecipe` — KV layout identity (certified architecture, kv dtype, head
|
||
geometry, owned range) with `fingerprint()` / `is_compatible()` /
|
||
`bytes_per_token()`; fails closed on uncertified architectures.
|
||
- `LayerKvCache` — per-owned-layer `(seq, n_kv_heads, head_dim)` K/V with
|
||
`append` / `truncate` / `nbytes`.
|
||
- `SessionCache` — the isolated per-`(session, epoch)` context over owned layers.
|
||
- `CacheMiss` / `CacheMissReason` — the explicit, serializable miss response.
|
||
- `HotKvStateManager` — `open` / `append` / `truncate` / `release` / `resolve` /
|
||
`get`, LRU+TTL+byte-budget eviction, stale-epoch + incompatible-recipe
|
||
rejection, epoch supersession, thread-safe (RLock), injectable clock.
|
||
- `KvBoundaryAdapter` + `kv_recipe_for()` — KV-aware boundary driver.
|
||
- `tests/test_hot_kv_state.py` — pure-numpy KV-cached dense-Llama reference and 22
|
||
tests (see below).
|
||
|
||
## Acceptance criteria → evidence
|
||
|
||
- **Map `(Route Session ID, route epoch)` to an isolated context** —
|
||
`test_prefill_then_decode_append_grows_owned_layers`,
|
||
`test_four_interleaved_sessions_have_no_kv_cross_talk`,
|
||
`HotKvStateManager.open` keys sessions on `(session_id, route_epoch)`.
|
||
- **Allocate KV only for owned layers** —
|
||
`test_manager_allocates_kv_only_for_owned_layers` (middle `[2,3]` → `{2,3}`),
|
||
`test_multi_range_cached_decode_parity_across_a_seam` (head owns `(0,1,2)`, tail
|
||
owns `(3,4,5)`), `test_recipe_bytes_per_token_scales_with_owned_layers`.
|
||
- **Prefill append / decode append / truncate / release / TTL-LRU eviction /
|
||
explicit cache-miss** — `test_prefill_then_decode_append_grows_owned_layers`,
|
||
`test_truncate_rolls_back_all_owned_layers`,
|
||
`test_release_one_session_leaves_others_intact_and_returns_memory`,
|
||
`test_ttl_eviction_yields_an_explicit_cache_miss`,
|
||
`test_lru_eviction_by_session_cap_reports_a_miss`,
|
||
`test_budget_eviction_keeps_total_within_budget`,
|
||
`test_unknown_session_is_an_explicit_cache_miss`,
|
||
`test_seq_len_mismatch_is_an_explicit_cache_miss`.
|
||
- **Reject stale epochs and incompatible cache recipes** —
|
||
`test_stale_route_epoch_is_rejected`,
|
||
`test_new_route_epoch_supersedes_and_frees_old_epoch`,
|
||
`test_incompatible_cache_recipe_is_rejected`,
|
||
`test_uncertified_architecture_recipe_fails_closed`.
|
||
- **≥ four concurrent sessions complete without token or KV cross-talk** —
|
||
`test_four_interleaved_sessions_have_no_kv_cross_talk` (four interleaved
|
||
round-robin sessions, four *distinct* references, each matches its own),
|
||
`test_four_sessions_on_real_threads_stay_isolated` (four OS threads).
|
||
- **Cancellation/release leaves others intact and memory returns to budget** —
|
||
`test_release_one_session_leaves_others_intact_and_returns_memory` (released
|
||
session → `CacheMiss(RELEASED)`, `total_bytes` drops, survivors keep matching
|
||
their references), `test_single_session_exceeding_budget_raises`.
|
||
- **Cached vs stateless correctness core** —
|
||
`test_cached_full_shard_decode_matches_stateless_whole_model`,
|
||
`test_cached_prefill_next_token_matches_whole_model_logits`,
|
||
`test_multi_range_cached_decode_parity_across_a_seam`. Documented tolerance:
|
||
**identical** greedy token ids (bit-exact in practice; cached incremental
|
||
attention equals stateless full-sequence recompute per query row).
|
||
- **Targeted pytest** — `22 passed`.
|
||
- **compileall packages tests** — exit 0.
|
||
- **git diff --check** — clean.
|
||
- **Deterministic / download-free / credit-free / GPU-free** — pure numpy; fixed
|
||
RNG seed; injectable clock (no wall-clock in tests); no torch, no network, no
|
||
model files.
|
||
- **Full deterministic pytest** — `13 failed, 755 passed, 14 skipped in 254.50s`.
|
||
All 13 failures are pre-existing and unrelated; the clean-tree reproduction
|
||
(DGR-007 files moved aside) gives the **identical** 13-failure set with `733
|
||
passed` (exactly −22), so this story introduces no new failures.
|
||
- **Native C++ / CTest / llama.cpp patch stack** — **not touched by this story.**
|
||
The KV context contract is delivered at the Python manager level with a numpy
|
||
parity + isolation proof; the equivalent native layer-filtered KV / session
|
||
mapping is wired when the standalone C++ worker exists in DGR-008. No native
|
||
code, CMake, or llama.cpp patch was modified, so those gates are N/A here (same
|
||
as DGR-005/006).
|
||
|
||
## Commands and real results
|
||
|
||
```bash
|
||
VP=/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python
|
||
|
||
$VP -m pytest -q tests/test_hot_kv_state.py
|
||
# -> 22 passed in ~0.3s
|
||
|
||
$VP -m compileall -q packages tests
|
||
# -> exit 0
|
||
|
||
git diff --check
|
||
# -> exit 0
|
||
|
||
$VP -m pytest -q tests/test_boundary_adapter.py tests/test_gguf_ownership.py
|
||
# -> 25 passed
|
||
|
||
$VP -m pytest -q -rfE
|
||
# -> 13 failed, 755 passed, 14 skipped in 254.50s
|
||
|
||
# Clean-tree reproduction (DGR-007 files moved aside)
|
||
mv packages/node/meshnet_node/hot_kv_state.py /tmp/ && mv tests/test_hot_kv_state.py /tmp/
|
||
$VP -m pytest -q -rfE
|
||
# -> 13 failed, 733 passed, 14 skipped in 252.12s (identical FAILED set; passed -22)
|
||
```
|
||
|
||
`commands.txt` beside this README captures the exact commands.
|
||
|
||
## Pre-existing unrelated failures (full-suite)
|
||
|
||
`pytest -q -rfE` on `ralph/distributed-gguf-runtime` reports 13 pre-existing
|
||
failures (and, in this run, 0 errors — the earlier DGR-005/006-era
|
||
`test_native_shard_protocol.py` protobuf errors no longer appear in this
|
||
environment). None touch the KV manager. Moving the two DGR-007 files aside and
|
||
re-running yields the **byte-identical** 13-`FAILED` set (only the passed count
|
||
drops by exactly 22). The exact set (all tracker/routing/benchmark/toploc/doctor,
|
||
i.e. socket-bind / control-plane env, not KV):
|
||
|
||
```
|
||
tests/test_dynamic_routing.py::test_admin_can_replace_a_served_model_and_release_it
|
||
tests/test_manual_route_benchmark.py::test_benchmark_records_one_and_two_node_routes
|
||
tests/test_manual_route_benchmark.py::test_clients_without_route_are_unaffected
|
||
tests/test_manual_route_benchmark.py::test_invalid_route_shape_is_400
|
||
tests/test_manual_route_benchmark.py::test_pinned_route_uses_named_node
|
||
tests/test_manual_route_benchmark.py::test_unknown_route_node_is_400
|
||
tests/test_node_doctor.py::test_cli_doctor_flags_select_what_is_validated
|
||
tests/test_toploc_calibration_dispatch.py::test_calibration_run_dispatches_only_solo_capable_nodes
|
||
tests/test_toploc_calibration_dispatch.py::test_calibration_run_node_without_commitment_endpoint_is_skipped_not_failed
|
||
tests/test_toploc_calibration_dispatch.py::test_calibration_run_persists_corpus_and_results_endpoint_reports_it
|
||
tests/test_tracker_capability_admission.py::test_an_enforcing_tracker_never_routes_a_node_whose_proof_does_not_cover_it[invalid]
|
||
tests/test_tracker_routing.py::test_shard_heal_cycle_surviving_node_covers_dead_peers_gap
|
||
tests/test_tracker_routing.py::test_torch_node_applies_tracker_load_shard_directive
|
||
```
|
||
|
||
## Limitations and deferred work
|
||
|
||
- **Numpy reference, not real weights.** The parity + isolation gate uses a
|
||
deterministic numpy KV-cached dense-Llama, not a downloaded GGUF/safetensors
|
||
model. Real-model concurrent KV isolation on a downloaded dense-Llama (CPU/ROCm)
|
||
belongs to DGR-010/DGR-012 with `MESHNET_ENABLE_REAL_INFERENCE_TESTS=1` and
|
||
`.venv-rocm`.
|
||
- **Manager-owned storage, native mapping deferred.** The KV bytes are numpy
|
||
arrays managed in-process. The llama.cpp expression (a filtered llama sequence
|
||
per `(session, epoch)` over owned layers) is implemented in the standalone
|
||
worker (DGR-008) against this same manager contract; no native code was touched.
|
||
- **Continuous batching is DGR-012.** This story delivers *isolation* and bounded
|
||
lifecycle for concurrent sessions; continuous batching of compatible active
|
||
sessions inside a node (decision #9) is DGR-012 and builds on this manager.
|
||
- **Greedy-only sampling.** Reuses the DGR-006 `SamplingContract` (greedy
|
||
certified). Stochastic sampling is out of scope for the deterministic gate.
|
||
- **Coexists with legacy `SessionCacheStore`.** The older AH-25
|
||
`model_backend.SessionCacheStore` (session-id-only, opaque transformers cache,
|
||
HTTP path) is untouched. `HotKvStateManager` is the native-runtime-aligned
|
||
successor: it adds route-epoch keying, owned-layer allocation, recipe-fingerprint
|
||
rejection, and a byte budget. DGR-008/009 wire the native worker to
|
||
`HotKvStateManager`, not `SessionCacheStore`.
|
||
|
||
## Compatibility / migration notes
|
||
|
||
- `KvCacheRecipe.fingerprint()` canonicalizes the architecture (via
|
||
`certified_architecture`), so `llama` / `LlamaForCausalLM` map to the same
|
||
recipe; it aligns field-for-field with the DGR-003 `RuntimeRecipeIdentity`
|
||
compatibility discipline and reuses `runtime_recipe.compatibility_fingerprint`.
|
||
- `CacheMiss` is a value (not an exception) so it can be serialized into the
|
||
DGR-002 native protocol's cache expectation/result field; `resolve()` returns it,
|
||
`get()` raises `KvCacheMissError` wrapping it.
|
||
- The manager takes an injectable `clock` for deterministic TTL tests; production
|
||
defaults to `time.monotonic`.
|
||
|
||
## Handoff for dependent stories
|
||
|
||
- **DGR-008 (C++ gRPC worker):** implement the servicer's KV path against
|
||
`HotKvStateManager`. Map each `(Route Session ID, route epoch)` to a filtered
|
||
llama sequence over owned layers; on decode, read the sequence's cached K/V,
|
||
compute the new position-encoded K/V, and commit via `append` (honour the byte
|
||
budget and return an explicit `CacheMiss` on eviction). Enforce
|
||
`KvCacheRecipe.is_compatible` before activation and reject stale epochs.
|
||
- **DGR-009 (Meshnet integration):** the route epoch the tracker assigns is the
|
||
`route_epoch` key; carry the `CacheMiss` reason back to the head so it re-prefills
|
||
from token zero on eviction/restart.
|
||
- **DGR-012 (continuous batching):** batch compatible active sessions whose
|
||
`KvCacheRecipe` fingerprints match; each session keeps its own `SessionCache`, so
|
||
batching is a scheduling concern layered over this isolation, not a change to it.
|
||
- **DGR-013 (failure/cancel matrix):** `release` + the budget-return assertion here
|
||
is the unit-level basis for the resource-cleanup matrix.
|