story: DGR-038 Implement isolated shard-local Hot KV State

.scratch/distributed-gguf-runtime/evidence/DGR-038/README.md

@@ -0,0 +1,66 @@
+# DGR-038 evidence — isolated shard-local Hot KV State
+
+**Date:** 2026-08-01  
+**Authority:** `.scratch/distributed-gguf-runtime/prd.json` (`passes` remains
+`false` until the opt-in real-model concurrency lane runs).
+
+## Implemented
+
+- The native `LlamaShardEngine` now creates one bounded llama.cpp context and
+  assigns a distinct `llama_seq_id` to each `(route_session_id, route_epoch)`.
+  It never accepts remote KV data; the loaded, range-attested llama model owns
+  the local cache layout and layers.
+- Prefill/decode append state tracks local positions and expected past length.
+  A re-prefill at an earlier position truncates only that sequence with
+  `llama_memory_seq_rm`; a discontinuity or past-length mismatch returns a
+  retryable `CACHE_MISS`. Older route epochs return `EPOCH_STALE`.
+- The token-reservation budget is bounded by per-session context, total Hot KV
+  budget, maximum sequence count, TTL, and LRU. Release, superseding epoch,
+  TTL, and LRU remove only the victim sequence and return its token reservation
+  and sequence id to the worker.
+- The gRPC service converts native cache/stale/resource results to the typed
+  protocol errors and does not consume idempotency/flow-control credit on a
+  rejected append. Release is epoch-specific, so a stale release cannot erase
+  the active epoch's service state.
+- Added opt-in configuration: `MESHNET_HOT_KV_MAX_SESSIONS`,
+  `MESHNET_HOT_KV_CONTEXT_TOKENS`, `MESHNET_HOT_KV_BUDGET_TOKENS`, and
+  `MESHNET_HOT_KV_TTL_SECONDS`.
+
+## Changed files
+
+- `packages/node/native/worker/llama_shard_engine.{h,cpp}`
+- `packages/node/native/worker/shard_service.cpp`
+- `packages/node/native/worker/shard_worker_main.cpp`
+- `tests/test_llama_shard_worker_binding.py`
+
+## Commands and results
+
+```text
+PYTHONPATH=packages/node /home/popov/.hermes/hermes-agent/venv/bin/python -m pytest -q \
+  tests/test_llama_shard_worker_binding.py tests/test_native_shard_protocol.py
+54 passed, 2 skipped
+
+/home/popov/.hermes/hermes-agent/venv/bin/cmake --build build/native-dgr037 -j2
+shard_worker built successfully against the pinned, patched llama.cpp source.
+
+/home/popov/.hermes/hermes-agent/venv/bin/ctest --test-dir build/native-dgr037 --output-on-failure
+1/1 shard_protocol_conformance passed.
+
+PYTHONPATH=packages/node /home/popov/.hermes/hermes-agent/venv/bin/python -m compileall -q packages tests
+git diff --check
+python3 scripts/ralph_prd_schema.py validate .scratch/distributed-gguf-runtime/prd.json
+compileall and diff check passed; OK: 55 stories validated.
+```
+
+## Limitations and dependency handoff
+
+- DGR-037 supplied the range-attested native model/engine boundary. DGR-038
+  adds local sequence ownership without changing its artifact or range
+  identity contract.
+- No mounted GGUF artifact was selected. Therefore no opt-in real-model
+  four-session run, actual llama KV byte measurement, or hardware metrics are
+  claimed. The default tests intentionally remain model-download-free and the
+  source `prd.json` remains `passes: false`.
+- DGR-039 should exercise the real two-process range-parity lane with four
+  sessions and the Hot-KV environment bounds, recording actual cache memory
+  and cancellation isolation evidence.