Distributed GGUF Runtime — Project Milestone Map
What this project is
We're building a system to run a giant AI model (DeepSeek V4 Flash, 671B params) split across multiple machines. Instead of one machine needing one huge GPU, we chop the model's layer stack into ranges (shards), run each range on a different machine, and pipe data between them over the network.
Key components: Tracker (matchmaker that assigns shards to machines), Nodes (worker machines running a shard), Gateway (entry point that receives user requests and routes them through shards), all built on top of llama.cpp (the C++ engine that actually runs the model).
Where we are (July 22, 2026)
13 of 55 tasks complete. M1 is 90% done — the protocol, build system, and scaffolding are in place. The next 4 tasks finish M1, then M2 begins the real engine work.
Milestone structure
Completed tasks (13/55)
M0 — Foundation & Cleanup (DGR-017..020)
- DGR-017 — Reconcile superseded backlog (clean slate)
- DGR-018 — Define canonical Ralph/Gitea metadata schema
- DGR-019 — Lock alpha/beta performance contracts (needs human review)
- DGR-020 — Run controlled whole-model GGUF baseline (needs human review)
M1 — Protocol & Build System (DGR-021..029)
- DGR-021 — Define versioned named-tensor stream envelope
- DGR-022 — Define Shard lifecycle and structured status RPCs
- DGR-023 — Make Python and C++ protobuf generation reproducible
- DGR-024 — Implement real generated-gRPC protocol harness
- DGR-025 — Define exact artifact/runtime recipe identity
- DGR-026 — Provision exact split-GGUF artifacts outside /home
- DGR-027 — Add exact llama.cpp provenance manifest + fetch workspace
- DGR-028 — Implement numbered patch-stack apply and verification
- DGR-029 — Create native CMake skeleton + deterministic CPU lane
Remaining tasks by milestone
M1: Protocol & Build System (4 remaining)
| Task |
What it means |
| DGR-030 |
Build presets + CI matrix — make the C++ build work with CUDA/ROCm/CPU, add CI tests |
| DGR-031 |
ShardEngine interface — define the contract every shard must implement |
| DGR-032 |
Fake ShardEngine — a pretend shard that returns correct-shaped fake data for testing |
| DGR-033 |
Fake C++ gRPC worker — wrap that fake shard in a real gRPC server (first end-to-end network test) |
M2: Shard Engine & Native Worker (DGR-034..043)
| Task |
What it means |
| DGR-034 |
Range-aware GGUF ownership — teach the shard to load only its slice of layers |
| DGR-035 |
Boundary I/O — define exact tensor shapes crossing between shards |
| DGR-036 |
Fixture vs real-model parity — prove fake shard matches real shard outputs |
| DGR-037 |
Bind llama.cpp to the worker — plug real llama.cpp engine into gRPC worker |
| DGR-038 |
Hot KV State — keep each shard's piece of conversation memory hot and accessible |
| DGR-039 |
Two-process acceptance — run 2 shards on one machine, verify output matches whole model |
| DGR-040 |
Worker supervision — start/monitor/restart native workers (like container orchestrator for shards) |
| DGR-041 |
Register capabilities — tell the Tracker "I can run layers 10-20 on this GPU" |
| DGR-042 |
Carry frames through seams — tensor data travels over direct connections and relay |
| DGR-043 |
Cost inputs to routing — tell Tracker "this shard takes X ms per token, Y GB bandwidth" |
M3: DeepSeek V4 Flash Integration (DGR-044..054)
| Task |
What it means |
| DGR-044 |
Pin the target contract — document exactly what V4 Flash needs (layers, tensors, memory) |
| DGR-045 |
Inventory V4 tensors — open the model file, list every tensor, assign to layers |
| DGR-046 |
V4 architecture boundary — define data crossing shard boundaries for MoE model |
| DGR-047 |
Adapt V4 for ranged ownership — modify upstream code so each machine runs only its range |
| DGR-048 |
Token-ID sideband — pass token ID alongside data between shards for expert routing |
| DGR-049 |
Shard-local attention state — keep attention/auxiliary state local per shard |
| DGR-050 |
Validate MoE routing — verify expert routing works when experts are on different machines |
| DGR-051 |
V4 ShardEngine adapter — the big integration: make V4 fit into the ShardEngine interface |
| DGR-052 |
V4 local vs distributed parity — run V4 on one machine vs split across two, verify same output |
| DGR-053 |
Certify real 2-4 stage route — run V4 across 2-4 machines with real GPUs (human review) |
| DGR-054 |
Enforce V4 alpha gate — alpha-quality checkpoint (human review) |
M4: Hardening & Performance (DGR-055..067)
| Task |
What it means |
| DGR-055 |
Continuous batching — handle multiple user requests simultaneously |
| DGR-056 |
Admission and backpressure — don't pile up requests, slow down gracefully |
| DGR-057 |
Benchmark batching — measure max simultaneous users before slowdown |
| DGR-058 |
Failure hardening — handle shard crashes mid-request gracefully |
| DGR-059 |
Route recovery — reroute around dead shards automatically |
| DGR-060 |
Long-context correctness — verify distributed version handles 128K token conversations |
| DGR-061 |
10+ stage routing — test routing across 10+ machines |
| DGR-062 |
Dynamic 10+ stage V4 scenario — real-world test across 10+ machines (human review) |
| DGR-063 |
Profile and optimize — find and fix the slowest part of the pipeline |
| DGR-064 |
Activation compression — compress data between machines to save bandwidth |
| DGR-065 |
MTP ownership — define multi-token prediction across shards |
| DGR-066 |
Implement MTP — build and benchmark distributed multi-token prediction |
| DGR-067 |
Certify capability matrix — final: what hardware, what models, what performance (human review) |
M5: Release & Upstream (DGR-068..071)
| Task |
What it means |
| DGR-068 |
Package releases — reproducible release binaries for others to install |
| DGR-069 |
Upstream patches — clean patches to submit to llama.cpp project (human review) |
| DGR-070 |
Beta gate certification — final beta-quality checkpoint (human review) |
| DGR-071 |
Maintenance docs — playbook for updating pin, reapplying patches, certifying releases |
Current state
- Branch:
ralph/distributed-gguf-runtime
- Progress: 13/55 tasks complete, 42 remaining
- Next task: DGR-030 (build presets + CI matrix)
- Last session stopped: Ralph hit Claude session limit at 09:44 on July 22. Reset at 13:30 Europe/Sofia. Use
ralph-tui resume to continue.
- 26 files committed from the last Ralph run (DGR-019..029 work). Branch pushed to origin.
Working conventions
- Ralph runs headless: reads backlog, spawns fresh Claude Code per ticket, verifies, reports
- DGR-019/020 marked
ready-for-human — needs review before certifying
- Changes left uncommitted for review per Ralph policy (unless explicitly pushed)
ralph-tui resume picks up where it left off
