neuron-tai/.scratch/distributed-gguf-runtime/ADR-0020-distributed-gguf-runtime.md

ADR-0020: Distributed GGUF/llama.cpp Runtime With Per-Shard Local KV

Status: Proposed

Context

The project currently uses PyTorch/Transformers for real model shards. That decision was captured in ADR-0001 because llama.cpp RPC at the time required the primary node to load the full model and distribute weights to workers, which conflicted with the desired model where nodes independently hold shards.

We now want to serve very large open models, including GLM-5.2 and Ornith-class MoE models, over a torrent-like inference marketplace. CPU and mixed consumer hardware matter. LM Studio and llama.cpp demonstrate much better CPU/GGUF performance than our current PyTorch CPU path. The user also has a personal relationship with Georgi Gerganov, making upstream collaboration plausible.

The current distributed PyTorch path is not yet production-grade: it recomputes the full growing sequence for every output token and disables KV cache inside manual layer calls. It sends hidden activations across seams, not KV, but those activations currently cover the full sequence every decode step.

Decision

Adopt a distributed GGUF/llama.cpp runtime track while keeping PyTorch as the reference and fast-architecture backend.

The runtime model is:

• GGUF/model artifacts are distributed through torrent/content-addressed storage.
• Nodes independently acquire and verify artifacts; no root node streams model weights to workers at session start.
• Tracker chooses a sticky route covering all layers.
• Each node owns hot KV/state for the layers it executes.
• Prefill sends chunked activations through the route and builds local per-shard KV.
• Decode sends one-step activations through the route and appends local KV at every shard.
• Cache/CDN servers store cold artifacts and optional prefix/session snapshots, not hot per-token KV.
• Context is capped at 128K for the first serious product path.

Technical Framework

The design separates five planes:

• Control plane: tracker registry, coverage map, route selection, session lifecycle, telemetry, billing, and audit.
• Artifact plane: Shard Swarms, GGUF/safetensors/tokenizer files, manifests, hashes, and local node storage.
• Execution plane: active Inference Route, chunked prefill, one-step decode, and hidden-state movement across activation seams.
• Session state plane: per-shard Hot KV State on route nodes, plus optional Prefix Snapshots outside the hot loop.
• Economics/trust plane: reward accounting, validation events, slash proofs, public/private route policy.

Hard invariants:

1. Public-network Shards are contiguous layer ranges.
2. Hot KV State is local to the node serving that Shard in that Route Session.
3. Artifact distribution and route execution are separate systems.
4. Decode seam payload must be O(hidden_size).
5. Prefill may be O(sequence_length * hidden_size), but only in bounded chunks.
6. The tracker chooses routes; nodes do not negotiate route topology peer-to-peer.
7. Model/backend-specific cache internals stay behind backend capability reports.
8. PyTorch remains the correctness/reference backend while llama.cpp/GGUF becomes the performance backend.
9. Streaming responses are preferred when feasible; Generation Telemetry is always required.

The full challenge register is in technical-challenges.md. The open decision gates are in decision-framework.md.

Resolved gate:

• Public-network Shards are layer ranges. Tensor-parallel/ring execution belongs inside a trusted node, colocated pod, or future composite node abstraction, not as the v1 public routing primitive.
• Hot KV State is local to each route node for the Shard it serves. Cache servers may store Prefix Snapshots, but they are not part of the per-token decode path.
• Distributed Route Session and Hot KV State semantics will be proven in the PyTorch route before llama.cpp/GGUF is extended for layer-boundary execution.
• Streaming responses are preferred when feasible. Realtime Generation Telemetry is required so clients can see phase, generated token count, and tokens/sec even during prefill or non-streaming fallback paths.
• llama.cpp/GGUF work targets upstreamable libllama/ggml hooks. A prototype fork is acceptable for exploration, but a permanent fork is not the plan.
• Model targeting is two-tiered: use a small llama.cpp-supported GGUF model for the first protocol smoke test, then use deepseek-ai/DeepSeek-V4-Flash as the first serious large-model target. GLM-5.2 and Ornith remain later support audits.
• Alpha fails Route Sessions on route-node loss instead of attempting automatic route repair. Repair requires compatible Prefix Snapshots and is a later capability.
• v1 activation transfer stays on binary HTTP as defined by ADR-0008. QUIC/WebRTC/custom transport can be introduced later behind the same activation protocol.

Non-Goals

• Do not put remote cache servers in the per-token hot KV path.
• Do not require every node to hold the full model.
• Do not fork llama.cpp long-term if upstream APIs can support the needed layer-boundary hooks.
• Do not target GLM-5.2 or Ornith first; prove the route/KV protocol on a simpler well-supported GGUF model, then target DeepSeek-V4-Flash as the first serious large model.

Options Considered

A. Keep PyTorch-only distributed inference

Pros:

• Easy access to new Hugging Face architectures.
• Transformers has mature single-process KV semantics.
• Existing code already loads shards.

Cons:

• CPU inference is much slower than llama.cpp/GGUF.
• Current distributed path bypasses generate() and disables cache.
• Quantized GGUF ecosystem and LM Studio users are outside the runtime.

B. Use llama.cpp only as a full local model backend

Pros:

• Quick performance win for nodes with enough RAM/VRAM.
• Minimal coordination with distributed protocol.

Cons:

• Does not unlock 397B/753B-class models for ordinary nodes.
• Does not solve marketplace layer routing.

C. Distributed GGUF with per-shard local KV (chosen)

Pros:

• Aligns with torrent artifact distribution.
• Avoids root streaming weights to workers.
• Uses llama.cpp/GGUF performance where supported.
• Compatible with public node rewards by layer/work contribution.
• Scales KV memory by layer range.

Cons:

• Requires new runtime APIs around layer-boundary hidden states and per-session KV.
• Requires model-specific cache metadata for DSA/MLA/hybrid attention.
• Harder to debug than single-process generate().

D. Centralized KV cache servers

Pros:

• Easier apparent session failover.
• Central accounting of active cache.

Cons:

• Puts remote storage in the per-token hot path.
• Adds bandwidth and latency at the worst possible point.
• Creates consistency and privacy problems.

Rejected for hot decode. Accepted only for cold prefix snapshots and failover checkpoints.

Consequences

• ADR-0001 should eventually be amended: PyTorch remains valid, but llama.cpp/GGUF becomes a first-class backend.
• The activation protocol must split prefill and decode explicitly.
• Session IDs must be stable across the full request. The current fresh UUID-per-hop-call behavior must change.
• Backends must report cache budget and cache compatibility.
• Tracker route selection must include disk, memory pressure, cache warmth, and network latency.
• Billing can be based on layer work, prefill tokens, decode tokens, and observed route participation.
• Client UX should stream token deltas when feasible and must include route-session progress telemetry even when token deltas are not streamed.

Required Runtime Capabilities

PyTorch path:

• manual layer calls with past_key_values / model-specific cache object
• per-shard session cache store
• prefill chunk append
• decode step append
• stable session lifecycle endpoints

llama.cpp/GGUF path:

• full local GGUF serving
• layer/tensor map extraction from GGUF
• optional partial layer loading or mmap-backed selected execution
• inbound hidden-state execution from arbitrary start layer
• outbound hidden-state return at stop layer
• per-session KV ownership for loaded layers
• cache budget/compatibility introspection
• GLM-5.2 DSA support when upstream/runtime supports it

Implementation Plan

1. Add full-model LlamaCppBackend using llama-server or libllama.
2. Implement distributed KV in the PyTorch path to prove semantics.
3. Add session lifecycle and prefill/decode wire protocol.
4. Add model artifact manifest and torrent seeding metadata.
5. Prototype localhost two-process llama.cpp layer boundary execution.
6. Generalize to network route.
7. Bring in GLM-5.2/Ornith once backend support and cache accounting are verified.

Acceptance Criteria

• A two-node localhost route can prefill once and decode N tokens without recomputing the full prompt.
• Seam payload during decode is O(hidden_size), not O(sequence_length * hidden_size).
• Per-node KV memory grows with owned layer count and context length.
• Route loss during alpha fails cleanly with explicit reason.
• Full local GGUF backend outperforms PyTorch CPU on a supported model.
• Artifact manifest can identify exactly which files/chunks a node must seed for its advertised layer range.