25 lines
620 B
Markdown
25 lines
620 B
Markdown
# 06 — Networked distributed GGUF route
|
|
|
|
Status: pending
|
|
|
|
Depends on: 01, 03, 04, 05
|
|
|
|
## Goal
|
|
|
|
Run a GGUF-backed model over a real multi-node route using the tracker-selected route and per-shard local KV.
|
|
|
|
## Scope
|
|
|
|
- Extend node backend registry with GGUF layer ranges.
|
|
- Add route selection for GGUF nodes.
|
|
- Use the prefill/decode protocol.
|
|
- Track route health and queue depth.
|
|
- Bill by layer work and token work.
|
|
|
|
## Acceptance
|
|
|
|
- Two physical machines can serve one model route.
|
|
- Node dropout during alpha fails request cleanly.
|
|
- Tracker metrics show prefill TPS, decode TPS, seam latency, and cache memory.
|
|
|