# ADR-0024: Lean Native Distributed GGUF Runtime

Status: Accepted
Date: 2026-07-13

> **Numbering note:** ADR-0020 is reserved for dashboard chat streaming and mixed-topology routing (`docs/adr/0020-chat-streaming-live-progress-and-mixed-topology-routing.md`). This record was originally drafted as ADR-0020 in `.scratch/distributed-gguf-runtime/` and renumbered to avoid the collision.
>
> **Relation to ADR-0001:** PyTorch/safetensors remains the correctness reference and current production backend. This ADR defines a benchmark-gated native GGUF path; it does not revoke ADR-0001 until release gates pass.

## Context

The project currently uses Transformers/safetensors as its real model execution backend. This provides broad architecture coverage and a correctness reference, but reported and observed consumer CPU/GPU inference performance motivates evaluating llama.cpp/GGML and quantized GGUF.

The product objective is not merely local GGUF serving. It is performant concurrent inference for top open models whose weights do not fit on one consumer node. The project already owns the Tracker, Inference Route, Route Session, Activation Seam, local Hot KV State, relay/direct transport, cancellation, telemetry, billing, and capability admission.

Research audited llama.cpp RPC, GPUStack/llama-box, Nakshatra, prima.cpp, llama-gguf, LiGGUF, vLLM and its GGUF plugin, Petals, exo, and related projects. No repository provides the complete public-network contract. llama.cpp is the strongest GGUF execution substrate. vLLM has mature managed-cluster parallelism and scheduling concepts but its PP/TP/EP runtime assumes a static trusted distributed world and is unsuitable as the public Shard runtime.

The project must remain lean and avoid combining several half-integrated inference control planes.

## Decision

### Primary native runtime

Use llama.cpp/GGML through one standalone C++ Shard worker and a small exact-commit patch stack.

The patch scope is limited to:

- Range-aware GGUF tensor ownership/loading.
- Architecture-defined intermediate boundary input/output.
- Intermediate output before tail normalization/head.
- Layer-filtered KV and external session-to-sequence mapping.

Meshnet networking, routing, admission, billing, telemetry, and work evidence stay outside llama.cpp.

Nakshatra, prima.cpp, llama-gguf, LiGGUF, and historical GPUStack are source/test donors only. Their repositories are not runtime dependencies.

### Distributed parallelism

The first public-network primitive is layer/pipeline parallelism through contiguous Shards in an Inference Route.

Per-node continuous batching combines decode steps from compatible active Route Sessions. Multiple complete routes provide data parallelism.

Tensor and expert parallel collectives may later operate inside one trusted composite node or managed cluster represented as one provider. They are not public WAN routing primitives.

### Transport

Use gRPC over HTTP/2 with Protocol Buffers for the native Python/C++ Shard data plane.

- One long-lived bidirectional stream per Route Session Activation Seam.
- Deadlines, cancellation, flow control, TLS/authentication hooks, structured status, and generated schemas.
- Bounded chunks for prefill and a small decode fast path.
- Existing relay infrastructure may carry the same versioned protobuf frames as opaque binary when direct connectivity is unavailable.
- OpenAI client APIs remain HTTP/SSE; existing Tracker APIs remain unchanged.

The boundary payload is a versioned named-tensor bundle because architecture boundaries may require more than one tensor.

### vLLM

Do not fork vLLM for public distributed Shards and do not transplant PagedAttention, Torch process groups, or the vLLM GGUF plugin into the llama.cpp worker.

Allow unmodified vLLM as an optional whole-model backend or managed TP/PP/EP cluster represented as one logical provider.

Adapt only small control-plane concepts:

- Named intermediate bundles.
- Continuous batching and request ownership.
- Versioned cache-transfer compatibility fingerprints.
- Explicit transfer failure/abort lifecycle.
- Load telemetry and fair tie-breaking.

### Benchmark gate

GGUF performance is a hypothesis. Before expensive native work, compare the current Transformers/safetensors recipe with whole-model llama.cpp on controlled model, hardware, prompt, context, output, sampling, concurrency, memory, and quality lanes.

Later distributed release gates use thresholds locked before implementation results are known. The native track stops if llama.cpp/GGUF offers neither a meaningful performance benefit nor a meaningful model-fit benefit at useful speed.

### Concurrency

A native worker must isolate `(Route Session ID, route epoch)` through a llama sequence or bounded context and must not serialize all generations behind one global serving sequence.

The node admits sessions against weight/KV/scratch budgets, batches compatible decode steps, prevents prefill starvation, applies backpressure, and exposes queue/batch/KV telemetry.

### Architecture certification

Dense Llama-family is first. Qwen3/Qwen3-MoE is a separate explicit adapter. Every architecture/backend/recipe remains registered-but-dark until a real distributed forward, parity test, concurrency test, and capability admission pass.

## Alternatives rejected

### Fork vLLM for the public mesh

Rejected because extracting its PP/TP/EP stages requires replacing static process groups, rank lifecycle, scheduler, request ownership, cache layout, failure behavior, and hardware assumptions. This would create a large difficult fork while discarding much of vLLM's core architecture.

### llama.cpp RPC as the public protocol

Rejected because it exposes coordinator-owned raw GGML devices, not independent Shards. Its trust, security, failure, cache, and per-node accounting model is unsuitable for arbitrary volunteer nodes.

### Adopt Nakshatra or prima.cpp wholesale

Rejected because their repositories, build reproducibility, session/concurrency semantics, architecture coverage, protocol identity, and control planes do not satisfy the project contract. Their partial-loading and boundary work remains valuable evidence.

### Build a custom GGUF engine

Rejected because llama.cpp already provides the parser, kernels, architecture graphs, KV, tokenizer, and heterogeneous backends. Reimplementing these would spread effort and increase correctness risk.

### Invent a custom transport

Rejected. gRPC/HTTP2 already provides mature streaming, flow control, deadlines, cancellation, TLS, and cross-language schema generation.

## Consequences

- The critical path contains Meshnet, one standalone worker, and one small pinned llama.cpp patch stack.
- Transformers/safetensors remains the correctness reference and fallback for unsupported architectures.
- Whole-model llama.cpp and vLLM managed clusters remain useful optional provider types.
- The first milestone emphasizes controlled benchmark, parity, concurrent KV, and real two-machine evidence rather than a large-model demo.
- Upstream collaboration with llama.cpp targets generic local hooks only; the project remains able to ship a narrow pinned fork if upstream acceptance takes time.
- QUIC, public tensor parallelism, disaggregated prefill, speculative decode, route repair, and KV migration remain deferred until the core route passes release gates.

## Verification gates

1. Controlled safetensors-versus-GGUF performance contract.
2. Two-process local range parity.
3. Four-session concurrent KV isolation.
4. Real two-machine execution using both Shards.
5. End-to-end performance/fit advantage over the current distributed route.
6. Separate Qwen3-family architecture certification.

## Relationship to US-042 (whole-model GGUF shortcut)

[US-042](../issues/42-gguf-llamacpp-node-backend.md) **phase C** ships first: a node with enough RAM serves a **full** GGUF via llama.cpp on a single-hop Inference Route using the existing HTTP activation seam and PyTorch-era tracker integration. That is intentionally small and does not require this ADR's gRPC worker or llama.cpp patch stack.

This ADR's track starts only after **DGR-001** (controlled safetensors-vs-GGUF benchmark) shows a meaningful speed or fit benefit. Then implement the native worker (DGR-002+) — which subsumes US-042 direction A (layer-range GGUF + boundary tensors) if the benchmark warrants it.

Do not run US-042 phase C and DGR-008+ in parallel on the same node backend without an explicit integration plan; phase C uses llama-cpp-python (or equivalent) whole-model path; ADR-0024 uses the standalone C++ worker.
