neuron-tai/.scratch/distributed-gguf-runtime/PRD.md

PRD: Performant Concurrent Distributed GGUF Runtime

Overview

Build one lean native GGUF execution path that lets an Inference Route combine consumer machines to serve models larger than any one node can hold. Reuse the existing Meshnet control plane and llama.cpp/GGML execution engine. Adopt gRPC/HTTP2 and Protocol Buffers for the native Shard worker data plane rather than inventing a transport.

The program is benchmark-gated. GGUF is not assumed faster merely because it is quantized or uses a different file format. The first story compares the current Transformers/safetensors backend against whole-model llama.cpp on controlled model/hardware/quality lanes and locks a performance contract. Native distributed work proceeds only when GGUF provides a meaningful speed or fit benefit.

The alpha target is now exact: zai-org/GLM-5.2 at a pinned revision, the lowest published Unsloth UD-IQ1_S GGUF, and reasoning_effort=max, distributed across physical consumer machines. See GLM-5.2-MAX-ALPHA-ROADMAP.md. Dense Llama is a cheap structural fixture; Qwen is post-alpha expansion.

Goals

• Execute one GGUF model across independently addressable contiguous Shards.
• Retain Hot KV State locally for each Shard and isolate concurrent Route Sessions.
• Batch compatible decode steps across active sessions for aggregate throughput.
• Use consumer CPU, AMD, NVIDIA, Vulkan, Metal, and mixed routes only where a real certified forward passes.
• Beat the current distributed safetensors route under a controlled performance contract or enable a larger otherwise-unroutable model at useful measured speed.
• Keep the critical path to Meshnet plus a small pinned llama.cpp fork and standalone C++ worker.
• Produce narrow upstream collaboration material for llama.cpp without placing Meshnet networking or economics inside upstream.
• Pass an immutable GLM-5.2 Max alpha/stop gate with native MoE, DSA, IndexShare, parity, usefulness, speed, failure, and cleanup evidence.

Quality Gates

Every story must:

• Run its targeted pytest tests.
• Run python -m compileall packages tests for Python changes.
• Run git diff --check.
• Keep default tests deterministic, model-download-free, API-credit-free, and GPU-free.
• Preserve existing Transformers/safetensors behavior unless the story explicitly changes a versioned compatibility contract.

Stories touching the native worker must also:

• Build the pinned C++ target with CMake.
• Run focused C++/protocol tests through CTest or the documented equivalent.
• Verify the llama.cpp patch stack applies cleanly to the exact pinned commit.

Real-model/hardware stories must:

• Require MESHNET_ENABLE_REAL_INFERENCE_TESTS=1.
• Use the machine-specific mounted-drive model path and the certified runtime environment; never place model artifacts under /home.
• Record exact model revision, artifact hash, runtime recipe, hardware, driver/backend, commands, raw JSON metrics, and output-quality result.
• Label synthetic tests as unit coverage rather than distributed acceptance.

Before a story is marked complete, run the full deterministic pytest -q suite or record the exact pre-existing unrelated failure with a clean-tree reproduction.

Dependency Graph and Status

Status as of 2026-07-14 (MAINT-003). Authoritative per-story status is passes in prd.json; closed issues live in docs/issues/distributed-gguf-runtime/, open and blocked issues in issues/.

graph TD
    classDef done fill:#c8e6c9,stroke:#2e7d32;
    classDef blocked fill:#ffcdd2,stroke:#c62828;

    DGR001[DGR-001 perf contract]:::done
    DGR002[DGR-002 gRPC Shard protocol]:::done
    DGR003[DGR-003 artifact/recipe identity]:::done
    DGR004[DGR-004 pinned llama.cpp patch stack]:::done
    DGR005[DGR-005 dense-Llama range ownership]:::done
    DGR006[DGR-006 boundary input/output]:::done
    DGR017[DGR-017 GLM-5.2 target/alpha contract]:::done
    DGR018[DGR-018 whole-model GLM oracle]:::blocked
    DGR019[DGR-019 GLM range/DSA/IndexShare]:::blocked
    DGR020[DGR-020 distributed GLM alpha]:::blocked
    DGR007[DGR-007 Hot KV State]
    DGR008[DGR-008 C++ gRPC worker]
    DGR009[DGR-009 Meshnet integration]
    DGR010[DGR-010 local two-process acceptance]
    DGR011[DGR-011 two-machine route]
    DGR012[DGR-012 continuous batching]
    DGR013[DGR-013 failure/cancel/restart]
    DGR014[DGR-014 release gate]
    DGR015[DGR-015 Qwen3 adapter]
    DGR016[DGR-016 upstream package]

    DGR002 --> DGR003
    DGR017 --> DGR003
    DGR001 --> DGR004
    DGR017 --> DGR004
    DGR003 --> DGR005
    DGR004 --> DGR005
    DGR002 --> DGR006
    DGR005 --> DGR006
    DGR001 --> DGR017
    DGR002 --> DGR017
    DGR003 --> DGR018
    DGR004 --> DGR018
    DGR017 --> DGR018
    DGR005 --> DGR019
    DGR006 --> DGR019
    DGR018 --> DGR019
    DGR006 --> DGR007
    DGR019 --> DGR007
    DGR002 --> DGR008
    DGR003 --> DGR008
    DGR004 --> DGR008
    DGR006 --> DGR008
    DGR007 --> DGR008
    DGR003 --> DGR009
    DGR008 --> DGR009
    DGR009 --> DGR010
    DGR010 --> DGR011
    DGR007 --> DGR012
    DGR009 --> DGR012
    DGR010 --> DGR012
    DGR008 --> DGR013
    DGR009 --> DGR013
    DGR001 --> DGR014
    DGR011 --> DGR014
    DGR012 --> DGR014
    DGR013 --> DGR014
    DGR014 --> DGR015
    DGR010 --> DGR016
    DGR007 --> DGR020
    DGR008 --> DGR020
    DGR009 --> DGR020
    DGR011 --> DGR020
    DGR013 --> DGR020
    DGR017 --> DGR020
    DGR018 --> DGR020
    DGR019 --> DGR020

• Done (passes: true): DGR-001, DGR-002, DGR-003, DGR-004, DGR-005, DGR-006, DGR-017.
• Blocked on hardware: DGR-018 requires a 256-GiB-class host with at least 224 GiB runtime-accessible memory and 250 GB free storage outside /home; no such host is currently available (development host: 124.9 GiB MemTotal). Exact preflight output: evidence/DGR-018/BLOCKED.md. DGR-019 (needs the DGR-018 oracle) and DGR-020 (needs DGR-018/DGR-019 plus multiple physical consumer nodes) are blocked transitively.
• Consequence of the graph as written: DGR-007 depends on DGR-019, so every remaining story (DGR-007 through DGR-016) is transitively blocked on the 256-GiB host. Unblocking the generic dense pipeline without that host would require an explicit re-planning decision to relax the DGR-007 → DGR-019 dependency; that decision is out of scope for maintenance and has not been made.

User Stories

DGR-001: Lock the safetensors-versus-GGUF performance contract

Description: As a runtime engineer, I need a controlled baseline so that GGUF work proceeds from measured speed, memory, and quality rather than reputation.

Acceptance Criteria:

• Benchmark the same model architecture/revision, machine, prompts, context lengths, output lengths, sampling policy, and concurrency across the current Transformers/safetensors recipe and whole-model llama.cpp recipes.
• Separate correctness/quality lanes from quantized performance/fit lanes instead of claiming BF16 and Q4 are numerically equivalent.
• Report TTFT, prefill tok/s, decode tok/s, p50/p95 latency, aggregate throughput, RSS, VRAM, artifact size, failures, and output drift in machine-readable JSON.
• Add concurrency levels 1 and 4 where memory permits.
• Write a versioned performance contract consumed by later release gates, including an explicit stop condition when llama.cpp/GGUF has no meaningful speed or fit benefit.

DGR-002: Adopt the versioned gRPC Shard protocol

Description: As a node developer, I need a battle-proven streaming protocol so that Python and C++ Shards communicate without a custom socket protocol.

Acceptance Criteria:

• Add a Protocol Buffers schema for capability, health, session stream, release, and cancellation operations.
• Define one long-lived bidirectional gRPC stream per Route Session Activation Seam with deadlines, cancellation, flow control, and structured errors.
• Define bounded chunking for prefill and a small decode fast path.
• Carry schema version, request/work ID, Route Session ID, route epoch, artifact/recipe fingerprint, Shard range/effective start, phase, position, idempotency step, cache expectation, compression, and checksum.
• Define a versioned named-tensor bundle with per-tensor name, shape, dtype, byte order, and payload fragments.
• Add generated-schema round-trip and compatibility tests in Python and C++.

DGR-003: Define exact Artifact and runtime recipe identity

Description: As the Tracker, I need exact compatibility identity so that only numerically and operationally compatible Shards form an Inference Route.

Acceptance Criteria:

• Separate weight quantization, activation dtype, compute dtype, KV dtype/layout, tokenizer revision, architecture adapter, backend, and runtime version.
• Bind derivative or split artifacts to an exact source Model Artifact hash and Shard range.
• Produce a stable compatibility fingerprint used by capability admission and the gRPC handshake.
• Fail closed on mismatched artifact, tokenizer, architecture, range, boundary schema, activation recipe, or cache layout.
• Keep unsupported recipes registered-but-dark until a real distributed forward certifies them.

DGR-004: Create the reproducible pinned llama.cpp patch stack

Description: As a maintainer, I need a small auditable fork boundary so that upstream updates do not turn the runtime into an unmaintainable stitched codebase.

Acceptance Criteria:

• Pin one exact llama.cpp commit through a reproducible source dependency mechanism.
• Store a numbered minimal patch stack separately from Meshnet networking code.
• Add a build script that applies/checks patches and builds the standalone worker without manual source copying.
• Record upstream file/ABI assumptions and fail clearly when the pin changes.
• Preserve upstream license and attribution notices.
• Add a clean rebuild smoke test that does not download a model.

DGR-005: Implement dense-Llama range-aware GGUF ownership

Description: As a node, I need to map only my assigned dense-Llama Shard so that aggregate consumer memory can hold a model larger than one node.

Acceptance Criteria:

• Register and allocate only blk.N.* tensors in the assigned range.
• Load embeddings only for the head and final norm/LM head only for the tail, including tied embeddings.
• Prefer range-aware mapping from one exact source GGUF; if derivative sub-GGUFs are used temporarily, verify source/slice hashes and avoid claiming final artifact semantics.
• Report authoritative loaded range and endpoint ownership from the model, not operator CLI claims.
• Demonstrate mapped/resident memory scales with owned tensors rather than full model size.

DGR-006: Implement architecture-defined boundary input/output

Description: As a Shard, I need to consume and emit the correct transformer boundary state so that disjoint processes reproduce whole-model execution.

Acceptance Criteria:

• Head accepts token IDs and owns token embedding.
• Middle/tail bypass token embedding and accept the named boundary bundle.
• Amend the decode fast path from one NamedTensor to a versioned TensorBundle, preserving compact one-tensor compatibility and regenerating Python/C++ protocol goldens.
• Define a typed tail logits/token result with sampling and chat-template/reasoning identity.
• Non-tail emits the unnormalized architecture-defined residual/boundary before final norm/head and before tail-only row pruning.
• Tail emits logits or token output through an explicit sampling contract.
• Dense-Llama whole-model versus two-range prefill and greedy-decode parity passes the documented tolerance.
• The adapter interface fails closed for uncertified architectures.

DGR-007: Add isolated concurrent local Hot KV State

Description: As a client, I need concurrent Route Sessions to retain independent per-Shard cache so that one request cannot clear or corrupt another.

Acceptance Criteria:

• Map (Route Session ID, route epoch) to an isolated llama sequence or bounded context.
• Allocate KV only for owned layers.
• Support prefill append, decode append, truncate, release, TTL/LRU eviction, and explicit cache-miss response.
• Reject stale epochs and incompatible cache recipes.
• At least four concurrent sessions on a small model complete without token or KV cross-talk.
• Cancellation/release of one session leaves other sessions intact and memory returns to the configured budget.

DGR-008: Build the standalone C++ gRPC Shard worker

Description: As a node runtime, I need one supervised native process so that llama.cpp internals remain behind a stable project-owned protocol.

Acceptance Criteria:

• Worker exposes capability, health, session stream, release, cancellation, and metrics services from DGR-002.
• Worker loads one exact Artifact/recipe/Shard identity and refuses mismatched requests.
• Streaming path enforces bounded messages, flow control, deadlines, idempotency, and independent session cancellation.
• Worker does not expose raw llama.cpp RPC or arbitrary GGML graph execution.
• Graceful shutdown releases sessions; crash behavior is bounded and observable.
• Python integration tests run against a fake model mode without model downloads.

DGR-009: Integrate the native worker with Meshnet

Description: As the existing node service, I need a GGUF Shard backend adapter so that the Tracker, relay, billing, telemetry, and capability admission remain the sole control plane.

Acceptance Criteria:

• Implement the existing model-backend surface without changing Transformers behavior.
• Registration carries exact validated GGUF recipe, Shard, backend and concurrency/KV capacity.
• Tracker forms only complete compatible routes and keeps uncertified recipes dark.
• Direct routes use gRPC streams; relayed routes carry the same versioned protobuf frames as opaque binary through the existing relay seam.
• Existing request/work IDs, cancellation, Generation Telemetry, billing, and per-node attribution remain correlated.
• No vLLM, Nakshatra, prima.cpp, or custom-engine control plane becomes a core dependency.

DGR-010: Pass local real-model two-process acceptance

Description: As a release engineer, I need real local distributed parity before involving network variability.

Acceptance Criteria:

• Two local worker processes open disjoint dense-Llama ranges from the certified Artifact.
• Prefill and at least 32 greedy decode tokens match whole-model llama.cpp within the certified tolerance.
• Each worker retains only its own tensors and Hot KV State.
• Four concurrent Route Sessions pass isolation and cleanup checks.
• Report TTFT, prefill/decode throughput, seam bytes/latency, worker RSS/VRAM, KV memory, batch size, and queue time.
• Killing one worker produces a bounded structured failure rather than a deadlock.

DGR-011: Pass a real heterogeneous two-machine route

Description: As a consumer-hardware operator, I need two physical machines to execute one GGUF model so that the distributed claim is real.

Acceptance Criteria:

• Tracker selects two physical nodes with disjoint Shards and one exact certified recipe/compatibility class.
• Actual CPU/GPU execution occurs on both nodes; synthetic workers do not satisfy acceptance.
• Prefill/decode, concurrent-session isolation, telemetry, cancellation, and cleanup pass over the real transport/relay path.
• Exact hardware, network, backend, model hash, route, commands, and raw metrics are recorded.
• A model or recipe larger than one participating node's admitted memory is exercised when available.
• Output drift is measured and incompatible mixed backends fail closed.

DGR-012: Implement continuous batching and bounded admission

Description: As a node operator, I need active sessions batched safely so that concurrency increases aggregate throughput rather than serializing every request.

Acceptance Criteria:

• Node scheduler admits sessions against weight, KV, scratch, and queue budgets.
• Compatible decode steps from multiple sessions form llama.cpp batches while preserving per-session positions and outputs.
• Prefill does not starve decode; scheduling policy and bounds are explicit.
• Backpressure prevents unbounded queued activations or KV growth.
• Capability telemetry reports active sessions, queue depth, batch occupancy, KV pressure, prefill/decode rates, and rejected admissions.
• Concurrency 1/2/4/8 benchmark identifies saturation and shows no cross-session corruption.

DGR-013: Harden failure, cancellation, and restart semantics

Description: As a client, I need failures to be bounded and explicit so that distributed speed does not come with hanging or corrupted generations.

Acceptance Criteria:

• Deadlines and heartbeat/health loss terminate blocked stream operations.
• Cancellation propagates across every Shard and releases local KV and queued buffers.
• Duplicate steps are idempotent; uncertain mutations are never replayed silently.
• Alpha failover restarts from token zero on a newly compatible route rather than importing unverified KV.
• Worker death, stream reset, malformed bundle, stale epoch, and cache miss tests pass.
• Billing/work records distinguish completed, cancelled, failed, and unverified work.

DGR-014: Enforce the GGUF-versus-safetensors release gate

Description: As the product owner, I need an end-to-end comparison so that the native runtime ships only if it advances model access or performance.

Acceptance Criteria:

• Run current distributed safetensors and distributed GGUF routes on the same certified model/hardware/network scenario where technically comparable.
• Report quality, TTFT, prefill/decode throughput, aggregate concurrency throughput, p95 latency, seam cost, memory, KV pressure, failures, and cleanup.
• Evaluate against the DGR-001 performance contract without changing thresholds after seeing results.
• Ship recommendation is one of: promote GGUF, optimize a measured bottleneck with a new bounded task, or stop the native track.
• Results clearly separate quantization gains from transport/runtime gains.

DGR-015: Add and certify a Qwen3/Qwen3-MoE adapter

Description: As a client seeking top models, I need a separately certified MoE-capable architecture after the dense runtime proves stable.

Acceptance Criteria:

• Implement explicit tensor ownership, router/top-k, expert/shared-expert, Q/K normalization, boundary bundle, and cache semantics for the selected Qwen3 family recipe.
• Do not reuse the dense-Llama adapter through unchecked name substitutions.
• Whole-model versus distributed prefill/decode parity passes the architecture-specific tolerance.
• Expert memory ownership and communication are measured.
• Real consumer-hardware acceptance and capability admission pass before the recipe becomes routable.

DGR-016: Produce the upstream llama.cpp collaboration package

Description: As a maintainer, I need narrow upstreamable proposals so that our patch burden can shrink without asking llama.cpp to own Meshnet networking.

Acceptance Criteria:

• Separate generic llama.cpp hooks from Meshnet protocol/control-plane code.
• Prepare minimal reproducible examples and tests for range-aware loading, boundary input/output, and layer-filtered KV.
• Compare the proposal with Nakshatra and prima.cpp evidence and explain why the API is generally useful.
• Preserve one scoped commit/patch per concern against the exact upstream pin.
• Produce an outreach document suitable for Georgi/llama.cpp maintainers; actual sending remains a human action.

DGR-017: Lock the GLM-5.2 Max target and alpha contract

Description: Pin exact official/GGUF revisions, UD-IQ1_S files and hashes, Max-mode semantics, resource accounting, and immutable target thresholds before implementation results exist.

Acceptance Criteria:

• Produce machine-readable target, resource, upstream-status, and immutable acceptance contracts without downloading full weights.
• Distinguish the 224-GiB hard-fit floor from the recommended 5×64 or 3×96/128 topology, using Q8 KV, 20%/8-GiB node reserves, and a wired 2.5-GbE minimum.
• Count unified RAM/VRAM once and test revision/shard/byte/threshold mutation failures.

DGR-018: Certify whole-model GLM-5.2 runtime semantics

Description: Establish the exact IQ1_S oracle on a 256-GiB-class host with at least 224 GiB runtime-accessible memory; lock Q8_0 MLA/indexer KV and native target semantics before distributed parity work.

Acceptance Criteria:

• Verify the complete artifact and prove native MoE/shared expert, DSA, IndexShare, KV, NextN policy, and Max-template behavior.
• Dense/replicated compatibility fallback cannot become the oracle merely because it emits text.

DGR-019: Implement and certify GLM-5.2 range, DSA, and IndexShare semantics

Description: Add explicit target-model tensor, graph, boundary, sideband, and local-KV ownership after the generic dense seam.

Acceptance Criteria:

• Preserve MoE/shared expert, DSA lightning indexer/sparse attention, and IndexShare Full/Shared semantics across contiguous Shards.
• Pass locked fixture/target parity and measured per-Shard memory ownership; never claim full-target parity from a reduced fixture.

DGR-020: Pass real distributed GLM-5.2 Max alpha acceptance

Description: Run the exact lowest-quant target through Meshnet on enough physical consumer nodes that no node can admit the whole recipe.

Acceptance Criteria:

• Pass the immutable identity, semantic, parity, Max-mode usefulness, 0.5 token/s, TTFT, reliability, and mounted-storage gates.
• Preserve signed raw evidence and emit alpha only if every target criterion passes; otherwise emit stop.

Functional Requirements

1. The public distributed primitive is an ordered Inference Route of contiguous Shards.
2. The native runtime uses llama.cpp/GGML; vLLM remains optional as a complete managed provider.
3. Native worker communication uses gRPC/HTTP2 and Protocol Buffers with one stable stream per Route Session Activation Seam.
4. Artifact identity, runtime recipe, boundary schema, activation dtype and cache layout must match exactly before routing.
5. Hot KV State remains local to the node serving the Shard.
6. Multiple Route Sessions must execute concurrently without shared-cache corruption.
7. Nodes batch compatible active decode steps and enforce bounded admission/backpressure.
8. Unsupported architectures and hardware recipes remain non-routable until real certification passes.
9. Default tests never download models or require GPUs; real tests are explicit and preserve artifacts off /home.
10. The release decision is based on measured performance, fit, quality, concurrency, and reliability relative to the safetensors baseline.

Non-Goals

• Forking vLLM or importing its PagedAttention/Torch distributed runtime.
• Adopting Nakshatra, prima.cpp, llama-gguf, LiGGUF, or GPUStack as the control plane.
• Public WAN tensor/expert parallel collectives.
• QUIC, WebRTC, or a custom socket protocol.
• Automatic KV migration or mid-generation route repair in the first release.
• Speculative decoding or disaggregated prefill before the core release gate.
• Supporting every GGUF architecture before the exact GLM-5.2 target; Qwen3-family certification is post-alpha.
• A marketing-scale model demo that bypasses parity, concurrency, admission, or performance gates.

Success Metrics

• A real model larger than one admitted node can execute across consumer machines when suitable hardware/artifacts are available.
• Four or more concurrent sessions complete without cross-talk; hardware-specific saturation is measured.
• Distributed GGUF passes the locked performance/fit contract against the existing safetensors route.
• Worker and Tracker recover all resources after completion, cancellation, malformed input, and node failure.
• The critical runtime remains Meshnet plus one standalone worker and a small auditable llama.cpp patch stack.

Open Questions

• DGR-001 remains immutable. DGR-017 locks the exact GLM-5.2/UD-IQ1_S target and alpha thresholds without rewriting DGR-001 evidence.
• Final hardware-specific concurrency and useful-speed thresholds are locked by measured baselines rather than guessed globally.
• Upstream llama.cpp acceptance is desirable but not a prerequisite for the first narrow pinned fork.