4.6 KiB
4.6 KiB
Distributed GGUF runtime — planning index
Status: draft scratch package.
Goal: make the node network capable of serving large, high-quality open models by distributing GGUF/model artifacts over a torrent-style swarm while executing inference over a sticky multi-node route with per-shard local KV cache.
This scratch supersedes the old assumption in ADR-0001 that llama.cpp is only a single-node leaf backend. That assumption was correct for the original llama.cpp RPC shape, but the target is now different: torrent-distributed GGUF artifacts plus an explicit route/KV protocol owned by this platform, ideally developed in collaboration with upstream llama.cpp.
Artifacts
| Path | Purpose |
|---|---|
| architecture.md | Proposed runtime architecture, data flow, session state, and failure model |
| technical-challenges.md | Detailed challenge/solution register with acceptance tests |
| decision-framework.md | Grilling framework for open decisions and recommended answers |
| research-prior-art.md | Prior-art notes for Petals, exo, Distributed Llama, prima.cpp, llama.cpp, DeepSeek-V4-Flash, GLM-5.2, and Ornith |
| ADR-0020-distributed-gguf-runtime.md | Draft decision record for the GGUF/llama.cpp distributed runtime |
| PRD.md | Product/runtime requirements and acceptance criteria |
| milestones.md | Dependency-ordered implementation milestones |
| issues/ | Implementation-ready tracer-bullet issue briefs |
Decision Summary
Adopt a hybrid runtime:
- Weights and artifacts: distributed by torrent / content-addressed storage / optional CDN.
- Hot KV cache: local to the node that owns the corresponding layer range.
- Prefix snapshots: optionally persisted to cache servers for reuse, retry, and failover.
- Active route: sticky for one request/session.
- Context cap: 128K hard product limit for large models unless explicitly revised.
- Backends: keep PyTorch for fast model-architecture coverage and validation; add llama.cpp/GGUF as the performance path for supported models.
- Client feedback: stream token deltas when feasible; always expose Generation Telemetry.
- First serious target model: DeepSeek-V4-Flash after a smaller GGUF protocol smoke test.
What We Learned
- Our current full-model PyTorch path uses Transformers
generate()and gets local KV cache. - Our current distributed PyTorch path disables cache and recomputes the full growing sequence per token.
- The seam today carries hidden activations, not KV cache; at 128K this becomes impossible for serious models if repeated every decode token.
- The missing capability is not "send KV across the network"; it is stable per-session local KV cache per shard.
- GGUF distribution is solved enough at the artifact layer, but GGUF/llama.cpp needs explicit layer-boundary execution APIs for our route model.
Recommended Order
See milestones.md for the full dependency map.
- 01 — Route Session lifecycle
- 02 — Prefill/decode binary HTTP protocol
- 03 — Generation Telemetry and streaming response contract
- 04 — PyTorch distributed KV reference route
- 05 — Local llama.cpp/GGUF backend
- 06 — Model Artifact manifest and Shard advertisement
- 07 — llama.cpp layer-boundary prototype
- 08 — Networked distributed GGUF route
- 09 — DeepSeek-V4-Flash support audit
- 10 — GLM-5.2 and Ornith follow-up support audit
Open Questions
- Does upstream llama.cpp already expose enough internal API for arbitrary layer-range execution and hidden-state boundary I/O, or do we need an extension?
- Can GGUF split metadata be made layer/tensor semantic enough for torrent placement and partial loading?
- What is the minimum protocol needed for compressed KV formats such as GLM-5.2 DSA/MLA without exposing model-specific internals to the tracker?
- How much reliability do we need in alpha: fail request on route loss, or support route repair with KV snapshots?