# Distributed GGUF runtime — planning index

Status: draft scratch package.

Goal: make the node network capable of serving large, high-quality open models by distributing GGUF/model artifacts over a torrent-style swarm while executing inference over a sticky multi-node route with per-shard local KV cache.

This scratch supersedes the old assumption in [ADR-0001](../../docs/adr/0001-pytorch-over-llama-cpp.md) that llama.cpp is only a single-node leaf backend. That assumption was correct for the original llama.cpp RPC shape, but the target is now different: torrent-distributed GGUF artifacts plus an explicit route/KV protocol owned by this platform, ideally developed in collaboration with upstream llama.cpp.

## Artifacts

| Path | Purpose |
|---|---|
| [architecture.md](./architecture.md) | Proposed runtime architecture, data flow, session state, and failure model |
| [technical-challenges.md](./technical-challenges.md) | Detailed challenge/solution register with acceptance tests |
| [decision-framework.md](./decision-framework.md) | Grilling framework for open decisions and recommended answers |
| [research-prior-art.md](./research-prior-art.md) | Prior-art notes for Petals, exo, Distributed Llama, prima.cpp, llama.cpp, DeepSeek-V4-Flash, GLM-5.2, and Ornith |
| [ADR-0020-distributed-gguf-runtime.md](./ADR-0020-distributed-gguf-runtime.md) | Draft decision record for the GGUF/llama.cpp distributed runtime |
| [issues/](./issues/) | Implementation slices in dependency order |

## Decision Summary

Adopt a hybrid runtime:

- **Weights and artifacts**: distributed by torrent / content-addressed storage / optional CDN.
- **Hot KV cache**: local to the node that owns the corresponding layer range.
- **Prefix snapshots**: optionally persisted to cache servers for reuse, retry, and failover.
- **Active route**: sticky for one request/session.
- **Context cap**: 128K hard product limit for large models unless explicitly revised.
- **Backends**: keep PyTorch for fast model-architecture coverage and validation; add llama.cpp/GGUF as the performance path for supported models.
- **Client feedback**: stream token deltas when feasible; always expose Generation Telemetry.
- **First serious target model**: DeepSeek-V4-Flash after a smaller GGUF protocol smoke test.

## What We Learned

- Our current full-model PyTorch path uses Transformers `generate()` and gets local KV cache.
- Our current distributed PyTorch path disables cache and recomputes the full growing sequence per token.
- The seam today carries hidden activations, not KV cache; at 128K this becomes impossible for serious models if repeated every decode token.
- The missing capability is not "send KV across the network"; it is **stable per-session local KV cache per shard**.
- GGUF distribution is solved enough at the artifact layer, but GGUF/llama.cpp needs explicit layer-boundary execution APIs for our route model.

## Recommended Order

1. Local llama.cpp/GGUF backend for full-model serving.
2. Stable distributed session ID and per-shard KV cache in the existing PyTorch path.
3. Binary prefill/decode protocol split: chunked prefill, one-step decode.
4. Route-session Generation Telemetry and streaming response support where feasible.
5. GGUF artifact manifest and torrent seeding.
6. llama.cpp layer-boundary prototype on localhost.
7. Networked distributed GGUF route.
8. DeepSeek-V4-Flash as first serious large-model target.
9. GLM-5.2 / DSA / MLA and Ornith support once runtime support is confirmed.

## Open Questions

- Does upstream llama.cpp already expose enough internal API for arbitrary layer-range execution and hidden-state boundary I/O, or do we need an extension?
- Can GGUF split metadata be made layer/tensor semantic enough for torrent placement and partial loading?
- What is the minimum protocol needed for compressed KV formats such as GLM-5.2 DSA/MLA without exposing model-specific internals to the tracker?
- How much reliability do we need in alpha: fail request on route loss, or support route repair with KV snapshots?