neuron-tai/.scratch/distributed-gguf-runtime/architecture.md

Performant Concurrent Distributed GGUF Architecture

Status: current target architecture Last updated: 2026-07-13

Product invariant

The system exists to serve high-quality models that exceed one consumer node's memory while retaining useful interactive speed and aggregate concurrency. A feature that only produces a distributed demo but is slower, globally serialized, or impossible to operate on consumer hardware is not complete.

Existing control plane

Meshnet remains the only public control plane:

• Tracker registration, Coverage Map, route scoring and assignment.
• Contiguous Shards and overlap-safe effective starts.
• Stable Route Sessions and route epochs.
• Local per-Shard Hot KV State in the reference backend.
• Direct/relay transport, cancellation and backpressure.
• Generation Telemetry, billing, validation and per-node attribution.
• Model-agnostic capability admission.

No external engine replaces these responsibilities.

Runtime topology

OpenAI-compatible client
        |
Gateway / Tracker Node
        |
ordered Inference Route
        |
        +-- head Shard: tokenizer/embedding + early layers
        |       local weights and Hot KV State
        |
        +-- middle Shard(s): architecture boundary + owned layers
        |       local weights and Hot KV State
        |
        +-- tail Shard: final layers + norm/head/sampling
                local weights and Hot KV State

Weights never move in the per-request hot path. Every node opens and verifies its local Model Artifact before becoming routable.

Primary execution substrate

project-owned C++ Shard worker
        |
small exact-commit llama.cpp patch stack
        |
GGUF mmap, quantized kernels, architecture graphs,
KV/sequence operations, CPU/CUDA/HIP/Vulkan/Metal backends

The patch stack adds only the missing local execution seam:

1. Range-aware tensor registration/loading.
2. Endpoint-specific embedding and final head ownership.
3. Architecture-defined intermediate input.
4. Architecture-defined pre-tail boundary output.
5. Layer-filtered KV and external session mapping.

The worker owns protocol translation and process lifecycle. llama.cpp never receives Tracker, relay, billing or volunteer-network code.

Shard data plane

Use Protocol Buffers and gRPC over HTTP/2.

Service shape

• Unary capability and health.
• Bidirectional Route Session stream.
• Explicit release and cancellation.
• Metrics suitable for capability admission and route scoring.

Session stream

One long-lived stream represents one Route Session Activation Seam. It amortizes connection setup and inherits HTTP/2 flow control. Every message carries enough identity to reject stale or incompatible work.

schema version
request/work id
Route Session id
route epoch
Model Artifact hash
runtime recipe fingerprint
Shard begin/end and effective start
prefill/decode/release/cancel phase
position and token range
idempotency step id
cache expectation/result
named tensor bundle
compression/checksum

Prefill tensors are split into bounded ordered frames. Decode messages carry one-step architecture boundary bundles and remain small.

Direct nodes use gRPC. Nodes requiring the existing relay carry the same protobuf frames as opaque binary through the relay session. This preserves one semantic protocol instead of maintaining separate direct and relay payload contracts.

Architecture boundary

The public boundary is a versioned named-tensor bundle:

bundle schema/version
architecture adapter and boundary point
named tensors
per-tensor shape, dtype and byte order
payload fragments
compression/checksum

Dense Llama may use one residual tensor. Other adapters may require more. vLLM's Llama and Qwen3-MoE PP paths demonstrate a boundary with both hidden_states and residual; therefore the generic protocol must not assume one anonymous tensor.

Only the head owns token embedding. Only the tail owns final normalization, LM head and sampling. Middle Shards exchange the architecture-defined pre-tail boundary, not final normalized embeddings.

Hot KV State and concurrency

(Route Session id, route epoch)
       -> local llama sequence or bounded context
       -> KV for owned layers only
       -> lease, memory accounting and lifecycle

Required operations:

• Prefill append.
• Decode append.
• Truncate after rejected speculative positions if later enabled.
• Explicit release.
• TTL/LRU eviction.
• Cache-miss response.
• Stale-epoch rejection.

A node must not clear global KV on a new stream or serialize all requests behind one logical serving sequence.

Continuous batching

Autoregressive dependencies remain sequential inside one Route Session. Aggregate throughput comes from batching compatible decode steps across active sessions:

time 0: session A token 1 + session B token 8 + session C token 3
          -> one llama batch for this Shard

time 1: next ready positions from active sessions
          -> next llama batch

The node scheduler:

• Admits work against weight, KV, scratch and queue budgets.
• Keeps per-session token positions and outputs separate.
• Prevents long prefill from starving decode.
• Applies bounded backpressure.
• Reports active sessions, queue depth, batch occupancy, KV pressure and throughput.

The initial deterministic gate is four concurrent sessions on a small model without cross-talk. Hardware-specific limits are measured and advertised through capability admission.

Parallelism boundaries

Mechanism	First-runtime use
Layer/pipeline parallelism	Public Inference Route across contiguous Shards
Continuous batching	Inside every node across active Route Sessions
Data parallelism	Multiple complete routes for independent requests
Tensor parallelism	Deferred to a trusted composite node/managed cluster
Expert parallelism	Deferred to a trusted composite node/managed cluster
Disaggregated prefill	Deferred until core route performance passes
Speculative decoding	Deferred optimization

Public WAN tensor/expert collectives are rejected for the first runtime because their per-layer communication and static rank assumptions conflict with heterogeneous volunteer nodes.

Optional providers

Transformers/safetensors

Remains:

• Correctness/reference backend.
• Fallback for unsupported architectures.
• Baseline for performance and output quality.

vLLM

May run unmodified as a complete model or managed TP/PP/EP cluster represented as one logical provider. Its internal ranks are not independently routed or rewarded.

Borrow only concepts such as named bundles, continuous batching, typed compatibility fingerprints, explicit transfer lifecycle and load telemetry.

Whole-model llama.cpp

Provides a local proxy backend, correctness oracle and performance baseline. It is not the native distributed milestone.

Artifact and recipe compatibility

A routable recipe identifies separately:

• Source Model Artifact hash and optional derivative/slice hash.
• Architecture and adapter version.
• Tokenizer revision and vocabulary.
• Weight quantization.
• Activation interchange dtype/schema.
• Backend compute dtype and backend implementation.
• KV dtype/layout.
• RoPE/context parameters.
• llama.cpp commit and project patch version.
• Shard range and endpoint ownership.

Compatibility fails closed. Similar quantization labels or model names are not enough.

Admission and failure

A recipe becomes routable only after a real local and distributed forward passes. Synthetic tests remain unit coverage.

Alpha failure behavior:

• Deadline or node loss cancels the Route Session.
• Every node releases KV and queued buffers.
• Uncertain mutations are not replayed silently.
• Retry starts from token zero on a newly compatible route.
• No cross-node KV import is trusted until a later signed/compatible snapshot protocol exists.

Performance release contract

Before native development proceeds, compare the current Transformers/safetensors backend with whole-model llama.cpp under controlled model/hardware/quality lanes.

Final release compares distributed GGUF with distributed safetensors using thresholds locked before seeing final results.

Required measurements:

• TTFT.
• Prefill and decode tokens/sec.
• Aggregate concurrency throughput.
• p50/p95 latency.
• Seam bytes and latency.
• Queue/batch occupancy.
• RSS, VRAM and KV pressure.
• Output-quality drift.
• Cancellation/failure cleanup.

The GGUF path ships only if it is faster at acceptable quality or enables a larger otherwise-unroutable model at useful measured speed.

Implementation sequence

1. Lock benchmark/performance contract.
2. Define gRPC/protobuf and exact recipe identity.
3. Pin llama.cpp and create the minimal patch stack.
4. Implement dense-Llama range loading and boundary parity.
5. Implement concurrent local KV.
6. Build and integrate the standalone worker.
7. Pass local two-process real-model acceptance.
8. Pass real heterogeneous two-machine acceptance.
9. Add continuous batching and failure hardening.
10. Enforce the GGUF-versus-safetensors release gate.
11. Add Qwen3/Qwen3-MoE as a separately certified adapter.
12. Prepare narrow upstream collaboration patches/tests.

See the Ralph backlog and implementation strategy.