neuron-tai/.scratch/distributed-gguf-runtime/evidence/DGR-038/README.md

DGR-038 evidence — isolated shard-local Hot KV State

Date: 2026-08-01
Authority: .scratch/distributed-gguf-runtime/prd.json (passes remains false until the opt-in real-model concurrency lane runs).

Implemented

• The native LlamaShardEngine now creates one bounded llama.cpp context and assigns a distinct llama_seq_id to each (route_session_id, route_epoch). It never accepts remote KV data; the loaded, range-attested llama model owns the local cache layout and layers.
• Prefill/decode append state tracks local positions and expected past length. A re-prefill at an earlier position truncates only that sequence with llama_memory_seq_rm; a discontinuity or past-length mismatch returns a retryable CACHE_MISS. Older route epochs return EPOCH_STALE.
• The token-reservation budget is bounded by per-session context, total Hot KV budget, maximum sequence count, TTL, and LRU. Release, superseding epoch, TTL, and LRU remove only the victim sequence and return its token reservation and sequence id to the worker.
• The gRPC service converts native cache/stale/resource results to the typed protocol errors and does not consume idempotency/flow-control credit on a rejected append. Release is epoch-specific, so a stale release cannot erase the active epoch's service state.
• Added opt-in configuration: MESHNET_HOT_KV_MAX_SESSIONS, MESHNET_HOT_KV_CONTEXT_TOKENS, MESHNET_HOT_KV_BUDGET_TOKENS, and MESHNET_HOT_KV_TTL_SECONDS.

Changed files

• packages/node/native/worker/llama_shard_engine.{h,cpp}
• packages/node/native/worker/shard_service.cpp
• packages/node/native/worker/shard_worker_main.cpp
• tests/test_llama_shard_worker_binding.py

Commands and results

PYTHONPATH=packages/node /home/popov/.hermes/hermes-agent/venv/bin/python -m pytest -q \
  tests/test_llama_shard_worker_binding.py tests/test_native_shard_protocol.py
54 passed, 2 skipped

/home/popov/.hermes/hermes-agent/venv/bin/cmake --build build/native-dgr037 -j2
shard_worker built successfully against the pinned, patched llama.cpp source.

/home/popov/.hermes/hermes-agent/venv/bin/ctest --test-dir build/native-dgr037 --output-on-failure
1/1 shard_protocol_conformance passed.

PYTHONPATH=packages/node /home/popov/.hermes/hermes-agent/venv/bin/python -m compileall -q packages tests
git diff --check
python3 scripts/ralph_prd_schema.py validate .scratch/distributed-gguf-runtime/prd.json
compileall and diff check passed; OK: 55 stories validated.

Limitations and dependency handoff

• DGR-037 supplied the range-attested native model/engine boundary. DGR-038 adds local sequence ownership without changing its artifact or range identity contract.
• No mounted GGUF artifact was selected. Therefore no opt-in real-model four-session run, actual llama KV byte measurement, or hardware metrics are claimed. The default tests intentionally remain model-download-free and the source prd.json remains passes: false.
• DGR-039 should exercise the real two-process range-parity lane with four sessions and the Hot-KV environment bounds, recording actual cache memory and cancellation isolation evidence.