64 lines
4.6 KiB
Markdown
64 lines
4.6 KiB
Markdown
# Distributed GGUF runtime — planning index
|
|
|
|
Status: draft scratch package.
|
|
|
|
Goal: make the node network capable of serving large, high-quality open models by distributing GGUF/model artifacts over a torrent-style swarm while executing inference over a sticky multi-node route with per-shard local KV cache.
|
|
|
|
This scratch supersedes the old assumption in [ADR-0001](../../docs/adr/0001-pytorch-over-llama-cpp.md) that llama.cpp is only a single-node leaf backend. That assumption was correct for the original llama.cpp RPC shape, but the target is now different: torrent-distributed GGUF artifacts plus an explicit route/KV protocol owned by this platform, ideally developed in collaboration with upstream llama.cpp.
|
|
|
|
## Artifacts
|
|
|
|
| Path | Purpose |
|
|
|---|---|
|
|
| [architecture.md](./architecture.md) | Proposed runtime architecture, data flow, session state, and failure model |
|
|
| [technical-challenges.md](./technical-challenges.md) | Detailed challenge/solution register with acceptance tests |
|
|
| [decision-framework.md](./decision-framework.md) | Grilling framework for open decisions and recommended answers |
|
|
| [research-prior-art.md](./research-prior-art.md) | Prior-art notes for Petals, exo, Distributed Llama, prima.cpp, llama.cpp, DeepSeek-V4-Flash, GLM-5.2, and Ornith |
|
|
| [ADR-0020-distributed-gguf-runtime.md](./ADR-0020-distributed-gguf-runtime.md) | Draft decision record for the GGUF/llama.cpp distributed runtime |
|
|
| [PRD.md](./PRD.md) | Product/runtime requirements and acceptance criteria |
|
|
| [milestones.md](./milestones.md) | Dependency-ordered implementation milestones |
|
|
| [issues/](./issues/) | Implementation-ready tracer-bullet issue briefs |
|
|
|
|
## Decision Summary
|
|
|
|
Adopt a hybrid runtime:
|
|
|
|
- **Weights and artifacts**: distributed by torrent / content-addressed storage / optional CDN.
|
|
- **Hot KV cache**: local to the node that owns the corresponding layer range.
|
|
- **Prefix snapshots**: optionally persisted to cache servers for reuse, retry, and failover.
|
|
- **Active route**: sticky for one request/session.
|
|
- **Context cap**: 128K hard product limit for large models unless explicitly revised.
|
|
- **Backends**: keep PyTorch for fast model-architecture coverage and validation; add llama.cpp/GGUF as the performance path for supported models.
|
|
- **Client feedback**: stream token deltas when feasible; always expose Generation Telemetry.
|
|
- **First serious target model**: DeepSeek-V4-Flash after a smaller GGUF protocol smoke test.
|
|
|
|
## What We Learned
|
|
|
|
- Our current full-model PyTorch path uses Transformers `generate()` and gets local KV cache.
|
|
- Our current distributed PyTorch path disables cache and recomputes the full growing sequence per token.
|
|
- The seam today carries hidden activations, not KV cache; at 128K this becomes impossible for serious models if repeated every decode token.
|
|
- The missing capability is not "send KV across the network"; it is **stable per-session local KV cache per shard**.
|
|
- GGUF distribution is solved enough at the artifact layer, but GGUF/llama.cpp needs explicit layer-boundary execution APIs for our route model.
|
|
|
|
## Recommended Order
|
|
|
|
See [milestones.md](./milestones.md) for the full dependency map.
|
|
|
|
1. [01 — Route Session lifecycle](./issues/01-route-session-lifecycle.md)
|
|
2. [02 — Prefill/decode binary HTTP protocol](./issues/02-prefill-decode-binary-http.md)
|
|
3. [03 — Generation Telemetry and streaming response contract](./issues/03-generation-telemetry-and-streaming.md)
|
|
4. [04 — PyTorch distributed KV reference route](./issues/04-pytorch-distributed-kv-reference.md)
|
|
5. [05 — Local llama.cpp/GGUF backend](./issues/05-local-llamacpp-gguf-backend.md)
|
|
6. [06 — Model Artifact manifest and Shard advertisement](./issues/06-model-artifact-manifest.md)
|
|
7. [07 — llama.cpp layer-boundary prototype](./issues/07-llamacpp-layer-boundary-prototype.md)
|
|
8. [08 — Networked distributed GGUF route](./issues/08-networked-distributed-gguf-route.md)
|
|
9. [09 — DeepSeek-V4-Flash support audit](./issues/09-deepseek-v4-flash-support-audit.md)
|
|
10. [10 — GLM-5.2 and Ornith follow-up support audit](./issues/10-glm52-ornith-followup-audit.md)
|
|
|
|
## Open Questions
|
|
|
|
- Does upstream llama.cpp already expose enough internal API for arbitrary layer-range execution and hidden-state boundary I/O, or do we need an extension?
|
|
- Can GGUF split metadata be made layer/tensor semantic enough for torrent placement and partial loading?
|
|
- What is the minimum protocol needed for compressed KV formats such as GLM-5.2 DSA/MLA without exposing model-specific internals to the tracker?
|
|
- How much reliability do we need in alpha: fail request on route loss, or support route repair with KV snapshots?
|