Goal: make the node network capable of serving large, high-quality open models by distributing GGUF/model artifacts over a torrent-style swarm while executing inference over a sticky multi-node route with per-shard local KV cache.

This scratch supersedes the old assumption in ADR-0001 that llama.cpp is only a single-node leaf backend. That assumption was correct for the original llama.cpp RPC shape, but the target is now different: torrent-distributed GGUF artifacts plus an explicit route/KV protocol owned by this platform, ideally developed in collaboration with upstream llama.cpp.

Artifacts

Path	Purpose
architecture.md	Proposed runtime architecture, data flow, session state, and failure model
technical-challenges.md	Detailed challenge/solution register with acceptance tests
decision-framework.md	Grilling framework for open decisions and recommended answers
research-prior-art.md	Prior-art notes for Petals, exo, Distributed Llama, prima.cpp, llama.cpp, DeepSeek-V4-Flash, GLM-5.2, and Ornith
ADR-0020-distributed-gguf-runtime.md	Draft decision record for the GGUF/llama.cpp distributed runtime
issues/	Implementation slices in dependency order

Decision Summary

Adopt a hybrid runtime:

Weights and artifacts: distributed by torrent / content-addressed storage / optional CDN.
Hot KV cache: local to the node that owns the corresponding layer range.
Prefix snapshots: optionally persisted to cache servers for reuse, retry, and failover.
Active route: sticky for one request/session.
Context cap: 128K hard product limit for large models unless explicitly revised.
Backends: keep PyTorch for fast model-architecture coverage and validation; add llama.cpp/GGUF as the performance path for supported models.
Client feedback: stream token deltas when feasible; always expose Generation Telemetry.
First serious target model: DeepSeek-V4-Flash after a smaller GGUF protocol smoke test.

What We Learned

Our current full-model PyTorch path uses Transformers generate() and gets local KV cache.
Our current distributed PyTorch path disables cache and recomputes the full growing sequence per token.
The seam today carries hidden activations, not KV cache; at 128K this becomes impossible for serious models if repeated every decode token.
The missing capability is not "send KV across the network"; it is stable per-session local KV cache per shard.
GGUF distribution is solved enough at the artifact layer, but GGUF/llama.cpp needs explicit layer-boundary execution APIs for our route model.

Recommended Order

Local llama.cpp/GGUF backend for full-model serving.
Stable distributed session ID and per-shard KV cache in the existing PyTorch path.
Binary prefill/decode protocol split: chunked prefill, one-step decode.
Route-session Generation Telemetry and streaming response support where feasible.
GGUF artifact manifest and torrent seeding.
llama.cpp layer-boundary prototype on localhost.
Networked distributed GGUF route.
DeepSeek-V4-Flash as first serious large-model target.
GLM-5.2 / DSA / MLA and Ornith support once runtime support is confirmed.

Open Questions

Does upstream llama.cpp already expose enough internal API for arbitrary layer-range execution and hidden-state boundary I/O, or do we need an extension?
Can GGUF split metadata be made layer/tensor semantic enough for torrent placement and partial loading?
What is the minimum protocol needed for compressed KV formats such as GLM-5.2 DSA/MLA without exposing model-specific internals to the tracker?
How much reliability do we need in alpha: fail request on route loss, or support route repair with KV snapshots?