feature-gguf-distributed

.scratch/distributed-gguf-runtime/README.md

@@ -0,0 +1,58 @@
+# Distributed GGUF runtime — planning index
+
+Status: draft scratch package.
+
+Goal: make the node network capable of serving large, high-quality open models by distributing GGUF/model artifacts over a torrent-style swarm while executing inference over a sticky multi-node route with per-shard local KV cache.
+
+This scratch supersedes the old assumption in [ADR-0001](../../docs/adr/0001-pytorch-over-llama-cpp.md) that llama.cpp is only a single-node leaf backend. That assumption was correct for the original llama.cpp RPC shape, but the target is now different: torrent-distributed GGUF artifacts plus an explicit route/KV protocol owned by this platform, ideally developed in collaboration with upstream llama.cpp.
+
+## Artifacts
+
+| Path | Purpose |
+|---|---|
+| [architecture.md](./architecture.md) | Proposed runtime architecture, data flow, session state, and failure model |
+| [technical-challenges.md](./technical-challenges.md) | Detailed challenge/solution register with acceptance tests |
+| [decision-framework.md](./decision-framework.md) | Grilling framework for open decisions and recommended answers |
+| [research-prior-art.md](./research-prior-art.md) | Prior-art notes for Petals, exo, Distributed Llama, prima.cpp, llama.cpp, DeepSeek-V4-Flash, GLM-5.2, and Ornith |
+| [ADR-0020-distributed-gguf-runtime.md](./ADR-0020-distributed-gguf-runtime.md) | Draft decision record for the GGUF/llama.cpp distributed runtime |
+| [issues/](./issues/) | Implementation slices in dependency order |
+
+## Decision Summary
+
+Adopt a hybrid runtime:
+
+- **Weights and artifacts**: distributed by torrent / content-addressed storage / optional CDN.
+- **Hot KV cache**: local to the node that owns the corresponding layer range.
+- **Prefix snapshots**: optionally persisted to cache servers for reuse, retry, and failover.
+- **Active route**: sticky for one request/session.
+- **Context cap**: 128K hard product limit for large models unless explicitly revised.
+- **Backends**: keep PyTorch for fast model-architecture coverage and validation; add llama.cpp/GGUF as the performance path for supported models.
+- **Client feedback**: stream token deltas when feasible; always expose Generation Telemetry.
+- **First serious target model**: DeepSeek-V4-Flash after a smaller GGUF protocol smoke test.
+
+## What We Learned
+
+- Our current full-model PyTorch path uses Transformers `generate()` and gets local KV cache.
+- Our current distributed PyTorch path disables cache and recomputes the full growing sequence per token.
+- The seam today carries hidden activations, not KV cache; at 128K this becomes impossible for serious models if repeated every decode token.
+- The missing capability is not "send KV across the network"; it is **stable per-session local KV cache per shard**.
+- GGUF distribution is solved enough at the artifact layer, but GGUF/llama.cpp needs explicit layer-boundary execution APIs for our route model.
+
+## Recommended Order
+
+1. Local llama.cpp/GGUF backend for full-model serving.
+2. Stable distributed session ID and per-shard KV cache in the existing PyTorch path.
+3. Binary prefill/decode protocol split: chunked prefill, one-step decode.
+4. Route-session Generation Telemetry and streaming response support where feasible.
+5. GGUF artifact manifest and torrent seeding.
+6. llama.cpp layer-boundary prototype on localhost.
+7. Networked distributed GGUF route.
+8. DeepSeek-V4-Flash as first serious large-model target.
+9. GLM-5.2 / DSA / MLA and Ornith support once runtime support is confirmed.
+
+## Open Questions
+
+- Does upstream llama.cpp already expose enough internal API for arbitrary layer-range execution and hidden-state boundary I/O, or do we need an extension?
+- Can GGUF split metadata be made layer/tensor semantic enough for torrent placement and partial loading?
+- What is the minimum protocol needed for compressed KV formats such as GLM-5.2 DSA/MLA without exposing model-specific internals to the tracker?
+- How much reliability do we need in alpha: fail request on route loss, or support route repair with KV snapshots?