# Performant Concurrent Distributed GGUF Architecture Status: current target architecture Last updated: 2026-07-13 ## Product invariant The system exists to serve high-quality models that exceed one consumer node's memory while retaining useful interactive speed and aggregate concurrency. A feature that only produces a distributed demo but is slower, globally serialized, or impossible to operate on consumer hardware is not complete. ## Existing control plane Meshnet remains the only public control plane: - Tracker registration, Coverage Map, route scoring and assignment. - Contiguous Shards and overlap-safe effective starts. - Stable Route Sessions and route epochs. - Local per-Shard Hot KV State in the reference backend. - Direct/relay transport, cancellation and backpressure. - Generation Telemetry, billing, validation and per-node attribution. - Model-agnostic capability admission. No external engine replaces these responsibilities. ## Runtime topology ```text OpenAI-compatible client | Gateway / Tracker Node | ordered Inference Route | +-- head Shard: tokenizer/embedding + early layers | local weights and Hot KV State | +-- middle Shard(s): architecture boundary + owned layers | local weights and Hot KV State | +-- tail Shard: final layers + norm/head/sampling local weights and Hot KV State ``` Weights never move in the per-request hot path. Every node opens and verifies its local Model Artifact before becoming routable. ## Primary execution substrate ```text project-owned C++ Shard worker | small exact-commit llama.cpp patch stack | GGUF mmap, quantized kernels, architecture graphs, KV/sequence operations, CPU/CUDA/HIP/Vulkan/Metal backends ``` The patch stack adds only the missing local execution seam: 1. Range-aware tensor registration/loading. 2. Endpoint-specific embedding and final head ownership. 3. Architecture-defined intermediate input. 4. Architecture-defined pre-tail boundary output. 5. Layer-filtered KV and external session mapping. The worker owns protocol translation and process lifecycle. llama.cpp never receives Tracker, relay, billing or volunteer-network code. ## Shard data plane Use Protocol Buffers and gRPC over HTTP/2. ### Service shape - Unary capability and health. - Bidirectional Route Session stream. - Explicit release and cancellation. - Metrics suitable for capability admission and route scoring. ### Session stream One long-lived stream represents one Route Session Activation Seam. It amortizes connection setup and inherits HTTP/2 flow control. Every message carries enough identity to reject stale or incompatible work. ```text schema version request/work id Route Session id route epoch Model Artifact hash runtime recipe fingerprint Shard begin/end and effective start prefill/decode/release/cancel phase position and token range idempotency step id cache expectation/result named tensor bundle compression/checksum ``` Prefill tensors are split into bounded ordered frames. Decode messages carry one-step architecture boundary bundles and remain small. Direct nodes use gRPC. Nodes requiring the existing relay carry the same protobuf frames as opaque binary through the relay session. This preserves one semantic protocol instead of maintaining separate direct and relay payload contracts. ## Architecture boundary The public boundary is a versioned named-tensor bundle: ```text bundle schema/version architecture adapter and boundary point named tensors per-tensor shape, dtype and byte order payload fragments compression/checksum ``` Dense Llama may use one residual tensor. Other adapters may require more. vLLM's Llama and Qwen3-MoE PP paths demonstrate a boundary with both `hidden_states` and `residual`; therefore the generic protocol must not assume one anonymous tensor. Only the head owns token embedding. Only the tail owns final normalization, LM head and sampling. Middle Shards exchange the architecture-defined pre-tail boundary, not final normalized embeddings. ## Hot KV State and concurrency ```text (Route Session id, route epoch) -> local llama sequence or bounded context -> KV for owned layers only -> lease, memory accounting and lifecycle ``` Required operations: - Prefill append. - Decode append. - Truncate after rejected speculative positions if later enabled. - Explicit release. - TTL/LRU eviction. - Cache-miss response. - Stale-epoch rejection. A node must not clear global KV on a new stream or serialize all requests behind one logical serving sequence. ## Continuous batching Autoregressive dependencies remain sequential inside one Route Session. Aggregate throughput comes from batching compatible decode steps across active sessions: ```text time 0: session A token 1 + session B token 8 + session C token 3 -> one llama batch for this Shard time 1: next ready positions from active sessions -> next llama batch ``` The node scheduler: - Admits work against weight, KV, scratch and queue budgets. - Keeps per-session token positions and outputs separate. - Prevents long prefill from starving decode. - Applies bounded backpressure. - Reports active sessions, queue depth, batch occupancy, KV pressure and throughput. The initial deterministic gate is four concurrent sessions on a small model without cross-talk. Hardware-specific limits are measured and advertised through capability admission. ## Parallelism boundaries | Mechanism | First-runtime use | |---|---| | Layer/pipeline parallelism | Public Inference Route across contiguous Shards | | Continuous batching | Inside every node across active Route Sessions | | Data parallelism | Multiple complete routes for independent requests | | Tensor parallelism | Deferred to a trusted composite node/managed cluster | | Expert parallelism | Deferred to a trusted composite node/managed cluster | | Disaggregated prefill | Deferred until core route performance passes | | Speculative decoding | Deferred optimization | Public WAN tensor/expert collectives are rejected for the first runtime because their per-layer communication and static rank assumptions conflict with heterogeneous volunteer nodes. ## Optional providers ### Transformers/safetensors Remains: - Correctness/reference backend. - Fallback for unsupported architectures. - Baseline for performance and output quality. ### vLLM May run unmodified as a complete model or managed TP/PP/EP cluster represented as one logical provider. Its internal ranks are not independently routed or rewarded. Borrow only concepts such as named bundles, continuous batching, typed compatibility fingerprints, explicit transfer lifecycle and load telemetry. ### Whole-model llama.cpp Provides a local proxy backend, correctness oracle and performance baseline. It is not the native distributed milestone. ## Artifact and recipe compatibility A routable recipe identifies separately: - Source Model Artifact hash and optional derivative/slice hash. - Architecture and adapter version. - Tokenizer revision and vocabulary. - Weight quantization. - Activation interchange dtype/schema. - Backend compute dtype and backend implementation. - KV dtype/layout. - RoPE/context parameters. - llama.cpp commit and project patch version. - Shard range and endpoint ownership. Compatibility fails closed. Similar quantization labels or model names are not enough. ## Admission and failure A recipe becomes routable only after a real local and distributed forward passes. Synthetic tests remain unit coverage. Alpha failure behavior: - Deadline or node loss cancels the Route Session. - Every node releases KV and queued buffers. - Uncertain mutations are not replayed silently. - Retry starts from token zero on a newly compatible route. - No cross-node KV import is trusted until a later signed/compatible snapshot protocol exists. ## Performance release contract Before native development proceeds, compare the current Transformers/safetensors backend with whole-model llama.cpp under controlled model/hardware/quality lanes. Final release compares distributed GGUF with distributed safetensors using thresholds locked before seeing final results. Required measurements: - TTFT. - Prefill and decode tokens/sec. - Aggregate concurrency throughput. - p50/p95 latency. - Seam bytes and latency. - Queue/batch occupancy. - RSS, VRAM and KV pressure. - Output-quality drift. - Cancellation/failure cleanup. The GGUF path ships only if it is faster at acceptable quality or enables a larger otherwise-unroutable model at useful measured speed. ## Implementation sequence 1. Lock benchmark/performance contract. 2. Define gRPC/protobuf and exact recipe identity. 3. Pin llama.cpp and create the minimal patch stack. 4. Implement dense-Llama range loading and boundary parity. 5. Implement concurrent local KV. 6. Build and integrate the standalone worker. 7. Pass local two-process real-model acceptance. 8. Pass real heterogeneous two-machine acceptance. 9. Add continuous batching and failure hardening. 10. Enforce the GGUF-versus-safetensors release gate. 11. Add Qwen3/Qwen3-MoE as a separately certified adapter. 12. Prepare narrow upstream collaboration patches/tests. See [the Ralph backlog](prd.json) and [implementation strategy](implementation-strategy.md).