Distributed GGUF Runtime architecture

Specification status: planning artifacts only. No distributed GGUF runtime is implemented by this materialization, no story has completion credit, and legacy files remain for the DGR-017 audit. prd.json is authoritative.

Locked scope

Existing Meshnet Tracker routing, load balancing, billing, telemetry, relay, and provider semantics are backend-agnostic and are not redesigned. GGUF contributes exact compatibility, range/capacity, queue/load, seam-cost, health/reliability, and certification inputs only.
The data plane is a standalone project-owned C++ Shard worker with gRPC/Protobuf and a project-owned ShardEngine boundary.
llama.cpp is fetched at one exact commit into an ignored workspace from an in-repo manifest, then a numbered minimal patch stack is applied. There is no submodule, vendored tree, or permanent-fork dependency.
llama.cpp owns DeepSeek V4 graphs, mHC, MoE, attention, hash routing, and kernels. Meshnet adds only range-ownership hooks, typed boundary/local-state adapters, worker integration, and parity/certification.
Quantization and placement are dynamic recipe inputs. The 2–4 and 10+ stage layouts are certification scenarios, never product constants.
Per-shard Hot KV and V4 CSA/HCA/SWA/indexer/compressor state remain local and keyed by route session/epoch. The WAN seam carries the typed mHC 4×4096 residual boundary, positions, token-ID sideband where required, and schema/cache expectations—not per-layer caches.
Route changes use cache miss plus re-prefill/restart. There is no WAN KV or V4 auxiliary-cache migration.
CPU/CUDA/ROCm/Vulkan/Metal compile lanes are planned; only exact real-hardware-certified backend/model/recipe lanes may be advertised.
Alpha requires correctness and the pre-locked useful-speed gate. MTP is reserved and off for alpha; its ownership contract, implementation, and benchmark are required before beta.

Target identities

DeepSeek V4 official target SHA: 60d8d70770c6776ff598c94bb586a859a38244f1.
llama.cpp V4 support lineage began at PR 24162 / merge 8c146a8366304c871efc26057cc90370ccf58dad; DGR-027 later pins one exact validated current commit.
V4 scope: 43 main layers plus MTP; mHC 4×4096 boundary; 256 routed + 1 shared experts with six routed active; token IDs required for the first three hash-routed layers.
Exact split-GGUF artifacts are provisioned to mounted-drive storage with a complete hashed manifest and resumable verification; no model artifact may be placed under /home.

Topology

existing Meshnet Tracker/control plane
  -> existing backend-agnostic route/load-balancing decision
  -> direct gRPC or existing opaque relay
  -> project-owned standalone C++ Shard worker
  -> project-owned ShardEngine
  -> pinned upstream llama.cpp + numbered range/boundary/state hook patches
  -> GGUF mmap, upstream V4 graph/kernels, local per-shard state

A route is ordered contiguous half-open ranges. Head owns token embedding; tail owns final norm/head/sampling. Compatibility fingerprints bind source/split hashes, tokenizer, architecture adapter, typed boundary, runtime pin/patches, backend, quant, activation/compute/KV layout, range, and certification.

V4 boundary and state

The inter-stage boundary is semantic and versioned: mHC 4×4096 residual, positions, token IDs only where the first three hash-routed layers require them, and cache/schema expectations. CSA/HCA/SWA/indexer/compressor/KV state belongs to upstream layer execution on the owning worker and is isolated by (route_session_id, route_epoch). On loss, return cache miss and re-prefill/restart. Never serialize those caches into the WAN bundle.

Concurrency, failure, and admission

Compatible sessions may be continuously batched within a worker while retaining isolated positions/state. Admission bounds weights, local state/KV, scratch, fragments, and queues. Uncertain cross-route mutation is not replayed. Registration can show an uncertified lane, but existing admission keeps it unroutable until signed/versioned real-hardware evidence exists.

4.0 KiB Raw Blame History Unescape Escape