neuron-tai/.scratch/distributed-gguf-runtime/decision-framework.md

Distributed GGUF Runtime decision framework

Specification status: planning artifacts only. No distributed GGUF runtime is implemented. DGR-017 cleanup is complete; no runtime implementation story has completion credit. prd.json is authoritative.

Decision order

1. DGR-019 locks comparable lanes and thresholds before results.
2. DGR-020 runs safetensors and whole-model llama.cpp only, then returns go, optimize baseline, or stop.
3. Dense and V4 work must prove parity, independent per-stage execution, local-state isolation, bounded failure, and measured resources.
4. DGR-054 returns alpha, optimize measured bottleneck, or stop; MTP is explicitly off.
5. Post-alpha optimizations must be selected from profiles, not assumptions.
6. DGR-070 returns beta, targeted optimization, or stop/rollback, and requires MTP and the exact certified hardware/recipe matrix.

Interpretation rules

• Quant/model-fit gains are separate from runtime/kernel/transport gains.
• Fixture, real-model, real-hardware, and release evidence are never interchangeable.
• 2–4 and 10+ stages are certification scenarios only.
• Existing routing policy is certified, not redesigned.
• Build success is not hardware certification; dark lanes remain unroutable.
• Route loss uses cache miss and re-prefill/restart, never WAN cache migration.

Locked scope

• Existing Meshnet Tracker routing, load balancing, billing, telemetry, relay, and provider semantics are backend-agnostic and are not redesigned. GGUF contributes exact compatibility, range/capacity, queue/load, seam-cost, health/reliability, and certification inputs only.
• The data plane is a standalone project-owned C++ Shard worker with gRPC/Protobuf and a project-owned ShardEngine boundary.
• llama.cpp is fetched at one exact commit into an ignored workspace from an in-repo manifest, then a numbered minimal patch stack is applied. There is no submodule, vendored tree, or permanent-fork dependency.
• llama.cpp owns DeepSeek V4 graphs, mHC, MoE, attention, hash routing, and kernels. Meshnet adds only range-ownership hooks, typed boundary/local-state adapters, worker integration, and parity/certification.
• Quantization and placement are dynamic recipe inputs. The 2–4 and 10+ stage layouts are certification scenarios, never product constants.
• Per-shard Hot KV and V4 CSA/HCA/SWA/indexer/compressor state remain local and keyed by route session/epoch. The WAN seam carries the typed mHC 4×4096 residual boundary, positions, token-ID sideband where required, and schema/cache expectations—not per-layer caches.
• Route changes use cache miss plus re-prefill/restart. There is no WAN KV or V4 auxiliary-cache migration.
• CPU/CUDA/ROCm/Vulkan/Metal compile lanes are planned; only exact real-hardware-certified backend/model/recipe lanes may be advertised.
• Alpha requires correctness and the pre-locked useful-speed gate. MTP is reserved and off for alpha; its ownership contract, implementation, and benchmark are required before beta.

Target identities

• DeepSeek V4 official target SHA: 60d8d70770c6776ff598c94bb586a859a38244f1.
• llama.cpp V4 support lineage began at PR 24162 / merge 8c146a8366304c871efc26057cc90370ccf58dad; DGR-027 later pins one exact validated current commit.
• V4 scope: 43 main layers plus MTP; mHC 4×4096 boundary; 256 routed + 1 shared experts with six routed active; token IDs required for the first three hash-routed layers.
• Exact split-GGUF artifacts are provisioned to mounted-drive storage with a complete hashed manifest and resumable verification; no model artifact may be placed under /home.