tasks

.scratch/distributed-gguf-runtime/issues/04-pytorch-distributed-kv-reference.md

@@ -0,0 +1,23 @@
+# 04 — PyTorch distributed KV reference route
+
+Status: ready-for-agent
+
+## What to build
+
+Fix the existing distributed PyTorch route so it uses the Route Session and prefill/decode protocol to keep Hot KV State local to each Shard node. The visible behavior is that prefill processes the prompt once, and decode no longer recomputes or resends the full growing prompt for every token.
+
+## Acceptance criteria
+
+- [ ] Distributed PyTorch prefill stores per-session cache/state on each Shard node.
+- [ ] Distributed PyTorch decode-step reads and appends local per-shard cache/state.
+- [ ] Decode activation seam payload is one token/hidden-state step after prefill.
+- [ ] The old full-growing-prompt decode loop is not used for models that support the reference cache path.
+- [ ] Unsupported model/cache APIs fail with an explicit backend capability error.
+- [ ] Session close or TTL cleanup releases per-shard cache.
+- [ ] Regression tests prove decode does not call the full prompt encoder for every generated token.
+
+## Blocked by
+
+- 01 — Route Session lifecycle.
+- 02 — Prefill/decode binary HTTP protocol.
+- 03 — Generation Telemetry and streaming response contract.