neuron-tai/.scratch/distributed-gguf-runtime/architecture.md

Performant Concurrent Distributed GGUF Architecture

Status: current target architecture Last updated: 2026-07-13

Product invariant

The system exists to serve high-quality models that exceed one consumer node's memory while retaining useful interactive speed and aggregate concurrency. A feature that only produces a distributed demo but is slower, globally serialized, or impossible to operate on consumer hardware is not complete.

The alpha target is the exact pinned GLM-5.2 UD-IQ1_S artifact in reasoning_effort=max mode. Its target-specific architecture/resource/acceptance contract is GLM-5.2-MAX-ALPHA-ROADMAP.md. Dense Llama is a structural fixture, not the product target.

Existing control plane

Meshnet remains the only public control plane:

• Tracker registration, Coverage Map, route scoring and assignment.
• Contiguous Shards and overlap-safe effective starts.
• Stable Route Sessions and route epochs.
• Local per-Shard Hot KV State in the reference backend.
• Direct/relay transport, cancellation and backpressure.
• Generation Telemetry, billing, validation and per-node attribution.
• Model-agnostic capability admission.

No external engine replaces these responsibilities.

Runtime topology

OpenAI-compatible client
        |
Gateway / Tracker Node
        |
ordered Inference Route
        |
        +-- head Shard: tokenizer/embedding + early layers
        |       local weights and Hot KV State
        |
        +-- middle Shard(s): architecture boundary + owned layers
        |       local weights and Hot KV State
        |
        +-- tail Shard: final layers + norm/head/sampling
                local weights and Hot KV State

Weights never move in the per-request hot path. Every node opens and verifies its local Model Artifact before becoming routable.

Primary execution substrate

project-owned C++ Shard worker
        |
small exact-commit llama.cpp patch stack
        |
GGUF mmap, quantized kernels, architecture graphs,
KV/sequence operations, CPU/CUDA/HIP/Vulkan/Metal backends

The patch stack adds only the missing local execution seam:

1. Range-aware tensor registration/loading.
2. Endpoint-specific embedding and final head ownership.
3. Architecture-defined intermediate input.
4. Architecture-defined pre-tail boundary output.
5. Layer-filtered KV and external session mapping.

The worker owns protocol translation and process lifecycle. llama.cpp never receives Tracker, relay, billing or volunteer-network code.

Shard data plane

Use Protocol Buffers and gRPC over HTTP/2.

Service shape

• Unary capability and health.
• Bidirectional Route Session stream.
• Explicit release and cancellation.
• Metrics suitable for capability admission and route scoring.

Session stream

One long-lived stream represents one Route Session Activation Seam. It amortizes connection setup and inherits HTTP/2 flow control. Every message carries enough identity to reject stale or incompatible work.

schema version
request/work id
Route Session id
route epoch
Model Artifact hash
runtime recipe fingerprint
Shard begin/end and effective start
prefill/decode/release/cancel phase
position and token range
idempotency step id
cache expectation/result
named tensor bundle
compression/checksum

Prefill tensors are split into bounded ordered frames. Decode messages carry one-step architecture boundary bundles and remain small. DGR-006 amends the current v1 decode fast path—which carries only one NamedTensor—to carry a versioned TensorBundle, while preserving compact one-tensor encoding and explicit compatibility behavior.

Tail completion is not inferred from an activation tensor name. The protocol exposes a typed logits and/or sampled-token result, and exact sampling parameters plus chat-template/reasoning mode are bound to request/runtime identity.

Direct nodes use gRPC. Nodes requiring the existing relay carry the same protobuf frames as opaque binary through the relay session. This preserves one semantic protocol instead of maintaining separate direct and relay payload contracts.

Architecture boundary

The public boundary is a versioned named-tensor bundle:

bundle schema/version
architecture adapter and boundary point
named tensors
per-tensor shape, dtype and byte order
payload fragments
compression/checksum

Dense Llama may use one residual tensor. Other adapters may require more. vLLM's Llama and Qwen3-MoE PP paths demonstrate a boundary with both hidden_states and residual; therefore the generic protocol must not assume one anonymous tensor.

GLM-5.2 normally exchanges a 6,144-element hidden state. If a memory-balanced Shard boundary splits an IndexShare Full producer from Shared consumers, the bundle also carries the typed top-k index sideband. The planner prefers boundaries that keep an IndexShare ownership group local, but the protocol validates the sideband rather than assuming it never crosses a seam.

Only the head owns token embedding. Only the tail owns final normalization, LM head and sampling. Middle Shards exchange the architecture-defined pre-tail boundary, not final normalized embeddings.

Hot KV State and concurrency

(Route Session id, route epoch)
       -> local llama sequence or bounded context
       -> KV for owned layers only
       -> lease, memory accounting and lifecycle

Required operations:

• Prefill append.
• Decode append.
• Truncate after rejected speculative positions if later enabled.
• Explicit release.
• TTL/LRU eviction.
• Cache-miss response.
• Stale-epoch rejection.

A node must not clear global KV on a new stream or serialize all requests behind one logical serving sequence.

Continuous batching

Autoregressive dependencies remain sequential inside one Route Session. Aggregate throughput comes from batching compatible decode steps across active sessions:

time 0: session A token 1 + session B token 8 + session C token 3
          -> one llama batch for this Shard

time 1: next ready positions from active sessions
          -> next llama batch

The node scheduler:

• Admits work against weight, KV, scratch and queue budgets.
• Keeps per-session token positions and outputs separate.
• Prevents long prefill from starving decode.
• Applies bounded backpressure.
• Reports active sessions, queue depth, batch occupancy, KV pressure and throughput.

The initial deterministic gate is four concurrent sessions on a small model without cross-talk. Hardware-specific limits are measured and advertised through capability admission.

Parallelism boundaries

Mechanism	First-runtime use
Layer/pipeline parallelism	Public Inference Route across contiguous Shards
Continuous batching	Inside every node across active Route Sessions
Data parallelism	Multiple complete routes for independent requests
Tensor parallelism	Deferred to a trusted composite node/managed cluster
Expert parallelism	Deferred to a trusted composite node/managed cluster
Disaggregated prefill	Deferred until core route performance passes
Speculative decoding	Deferred optimization

Public WAN tensor/expert collectives are rejected for the first runtime because their per-layer communication and static rank assumptions conflict with heterogeneous volunteer nodes.

Optional providers

Transformers/safetensors

Remains:

• Correctness/reference backend.
• Fallback for unsupported architectures.
• Baseline for performance and output quality.

vLLM

May run unmodified as a complete model or managed TP/PP/EP cluster represented as one logical provider. Its internal ranks are not independently routed or rewarded.

Borrow only concepts such as named bundles, continuous batching, typed compatibility fingerprints, explicit transfer lifecycle and load telemetry.

Whole-model llama.cpp

Provides a local proxy backend, correctness oracle and performance baseline. It is not the native distributed milestone.

Artifact and recipe compatibility

A routable recipe identifies separately:

• Source Model Artifact hash and optional derivative/slice hash.
• Architecture and adapter version.
• Tokenizer revision and vocabulary.
• Weight quantization.
• Activation interchange dtype/schema.
• Backend compute dtype and backend implementation.
• KV dtype/layout.
• RoPE/context parameters.
• llama.cpp commit and project patch version.
• Shard range and endpoint ownership.

Compatibility fails closed. Similar quantization labels or model names are not enough.

Admission and failure

A recipe becomes routable only after a real local and distributed forward passes. Synthetic tests remain unit coverage.

Alpha failure behavior:

• Deadline or node loss cancels the Route Session.
• Every node releases KV and queued buffers.
• Uncertain mutations are not replayed silently.
• Retry starts from token zero on a newly compatible route.
• No cross-node KV import is trusted until a later signed/compatible snapshot protocol exists.

Performance release contract

Before native development proceeds, compare the current Transformers/safetensors backend with whole-model llama.cpp under controlled model/hardware/quality lanes.

Final release compares distributed GGUF with distributed safetensors using thresholds locked before seeing final results.

Required measurements:

• TTFT.
• Prefill and decode tokens/sec.
• Aggregate concurrency throughput.
• p50/p95 latency.
• Seam bytes and latency.
• Queue/batch occupancy.
• RSS, VRAM and KV pressure.
• Output-quality drift.
• Cancellation/failure cleanup.

The GGUF path ships only if it is faster at acceptable quality or enables a larger otherwise-unroutable model at useful measured speed.

Implementation sequence

1. Preserve completed DGR-001 performance and DGR-002 protocol contracts.
2. DGR-017 locks exact GLM-5.2 Max artifact, resource, and alpha acceptance identity.
3. Define exact recipe identity and pin one reproducible llama.cpp boundary.
4. Run two lanes in parallel: DGR-018 establishes the whole-model UD-IQ1_S oracle on 224+ GiB usable memory, while DGR-005/DGR-006 implement range loading and named boundary parity with a cheap dense fixture.
5. DGR-019 adds explicit GLM-5.2 MoE/MLA/DSA/IndexShare semantics after both lanes pass.
6. Implement local KV; build and integrate the standalone worker.
7. Pass local two-process and real two-physical-machine execution.
8. Harden cancellation, node loss, restart, and cleanup required by alpha.
9. DGR-020 executes the exact multi-node target and emits immutable alpha or stop.
10. Post-alpha: continuous batching, final comparison, longer context, MTP, and package optimization.
11. Prepare narrow upstream patches/tests; add Qwen as later architecture expansion.

See the Ralph backlog and implementation strategy.