25 lines
1.2 KiB
Markdown
25 lines
1.2 KiB
Markdown
# 08 — Networked distributed GGUF route
|
|
|
|
Status: pending
|
|
|
|
## What to build
|
|
|
|
Run a GGUF-backed model over a real multi-node Inference Route using the resolved Route Session, binary HTTP prefill/decode protocol, local Hot KV State, Generation Telemetry, and alpha fail-fast behavior.
|
|
|
|
## Acceptance criteria
|
|
|
|
- [ ] Two machines can form one GGUF-backed Inference Route over contiguous Shards.
|
|
- [ ] Prefill builds local per-shard cache/state and decode-step uses one-step seam payloads.
|
|
- [ ] The client receives streamed token deltas when supported by the GGUF path.
|
|
- [ ] The client receives Generation Telemetry for phase, generated tokens, tokens/sec, route health, and failure reason.
|
|
- [ ] Route-node loss fails the Route Session cleanly; no automatic repair is attempted in alpha.
|
|
- [ ] Tracker metrics show prefill tokens/sec, decode tokens/sec, seam latency, queue depth, and cache memory by node.
|
|
- [ ] Billing/audit records identify route membership and layer/token work for the completed or failed session.
|
|
|
|
## Blocked by
|
|
|
|
- 03 — Generation Telemetry and streaming response contract.
|
|
- 04 — PyTorch distributed KV reference route.
|
|
- 06 — Model Artifact manifest and Shard advertisement.
|
|
- 07 — llama.cpp layer-boundary prototype.
|