distributd cache

.scratch/alpha-hardening/issues/25-per-node-kv-cache-distributed.md

@@ -1,4 +1,14 @@
-Status: ready-for-agent
+Status: implemented 2026-07-08 — pending live 2-node GPU verification
+
+Implemented in `packages/node/meshnet_node/model_backend.py` + `torch_server.py`; design in
+[ADR-0022](../../../docs/adr/0022-sharded-per-node-kv-cache.md); tests in
+`tests/test_kv_cache_distributed.py` (11 fast tests + env-gated golden test,
+`MESHNET_REAL_MODEL_TESTS=1`).
+
+**Measured (two-shard Qwen2.5-0.5B 0-11/12-23, CPU, 44-token prompt, 40 steps):**
+stateless 7.05 tps decaying 32% (8.09 → 5.50 first-10 vs last-10); cached 18.93 tps and
+FLAT (17.21 → 19.28) — 2.68× overall, gap grows quadratically with length. Remaining
+acceptance item: re-measure on the live 2-node GPU topology (needs both machines).
 
 Scoped 2026-07-08 from a live two-machine distributed-inference debugging session (Qwen2.5-0.5B GPU+GPU pipeline, and Qwen3.6-35B-A3B mixed GPU/CPU). The ADR-0020 mixed-topology `start_layer` bug is fixed (`518c259`, `e44abc9`, `1ecc599`); this issue is the next performance blocker in the same code path.
 
@@ -33,15 +43,15 @@ Observed cost of this on a live 2-node Qwen2.5-0.5B GPU pipeline (layers 0-20 /
 
 ## Acceptance criteria
 
-- [ ] A session ID is stable across all steps of one chat generation (not re-minted per token)
-- [ ] Steps after the first prefill send only the new token's activation, not the full sequence, over the wire between nodes
-- [ ] Each node caches `past_key_values`/recurrent state only for its own shard's layer range; no node ever holds another node's cache
-- [ ] Cache works correctly for both standard-attention shards and Qwen3.6-style hybrid linear-attention/recurrent shards (cache abstraction is not K/V-shaped-only)
-- [ ] Bounded memory: TTL + LRU eviction; eviction/restart triggers a documented cache-miss response, not silent corruption or an unhandled exception
-- [ ] Golden-output regression test proves cached and uncached distributed generation produce equivalent output for a fixed prompt
-- [ ] Measured tps improvement recorded on the same 2-node Qwen2.5-0.5B topology used to observe the regression (target: flat tps across generation length, not decaying)
-- [ ] `tests/test_two_node_pipeline.py` and `tests/test_dynamic_routing.py` still pass
-- [ ] Design captured in a new ADR (or an amendment to ADR-0020/0021) covering the cache-miss/route-change interaction
+- [x] A session ID is stable across all steps of one chat generation (not re-minted per token) — minted once in `_do_chat_completions`, asserted in `test_session_is_stable_and_decode_payloads_are_single_token`
+- [x] Steps after the first prefill send only the new token's activation (`[1, 1, hidden]` via `encode_next_token`) with `X-Meshnet-Cache: decode` + `X-Meshnet-Past-Len`
+- [x] Each node caches state only for its own shard's layer range (`TorchModelShard.kv_sessions`; sharding falls out of per-node layer execution)
+- [x] Cache abstraction is not K/V-shaped-only: `DynamicCache(config=model.config)` — the same construction Qwen3.6-Next's own forward uses for hybrid linear-attention conv/delta state; store treats it as opaque; `TypeError` fallback disables caching per-backend
+- [x] Bounded memory: TTL (600 s, `MESHNET_KV_TTL_SECONDS`) + LRU (8, `MESHNET_KV_MAX_SESSIONS`); miss → HTTP 409 `{"error": "cache_miss"}` → head re-prefills (tested)
+- [x] Golden-output test: cached and stateless produce identical token ids on real two-shard Qwen2.5-0.5B (`test_cached_distributed_generation_matches_stateless_golden`, passed)
+- [x] Measured (CPU two-shard proxy, 40 steps): stateless 7.05 tps w/ 32% decay → cached 18.93 tps flat, 2.68×. ⚠️ still to run on the live 2-node GPU topology
+- [x] `tests/test_two_node_pipeline.py` and `tests/test_dynamic_routing.py` pass (30 passed; 6 tmp-dir fixture errors are a pre-existing Windows temp-permission env issue, identical on clean tree)
+- [x] Design captured in [ADR-0022](../../../docs/adr/0022-sharded-per-node-kv-cache.md) incl. cache-miss/route-change interaction with ADR-0021
 
 ## Notes