1.2 KiB
1.2 KiB
08 — Networked distributed GGUF route
Status: pending
What to build
Run a GGUF-backed model over a real multi-node Inference Route using the resolved Route Session, binary HTTP prefill/decode protocol, local Hot KV State, Generation Telemetry, and alpha fail-fast behavior.
Acceptance criteria
- Two machines can form one GGUF-backed Inference Route over contiguous Shards.
- Prefill builds local per-shard cache/state and decode-step uses one-step seam payloads.
- The client receives streamed token deltas when supported by the GGUF path.
- The client receives Generation Telemetry for phase, generated tokens, tokens/sec, route health, and failure reason.
- Route-node loss fails the Route Session cleanly; no automatic repair is attempted in alpha.
- Tracker metrics show prefill tokens/sec, decode tokens/sec, seam latency, queue depth, and cache memory by node.
- Billing/audit records identify route membership and layer/token work for the completed or failed session.
Blocked by
- 03 — Generation Telemetry and streaming response contract.
- 04 — PyTorch distributed KV reference route.
- 06 — Model Artifact manifest and Shard advertisement.
- 07 — llama.cpp layer-boundary prototype.