docs: define implementation-ready distributed GGUF roadmap
This commit is contained in:
Dobromir Popov
@@ -1,46 +1,32 @@
|
||||
# Performant concurrent distributed GGUF runtime
|
||||
# Distributed GGUF Runtime planning workspace
|
||||
|
||||
Status: active benchmark-gated implementation program.
|
||||
> **Specification status:** planning artifacts only. No distributed GGUF runtime is implemented by this materialization, no story has completion credit, and legacy files remain for the DGR-017 audit. `prd.json` is authoritative.
|
||||
|
||||
## Objective
|
||||
|
||||
Serve top open models across consumer machines with useful performance and concurrent Route Sessions while keeping the runtime lean.
|
||||
## Locked scope
|
||||
|
||||
## Critical path
|
||||
- Existing Meshnet Tracker routing, load balancing, billing, telemetry, relay, and provider semantics are backend-agnostic and are **not redesigned**. GGUF contributes exact compatibility, range/capacity, queue/load, seam-cost, health/reliability, and certification inputs only.
|
||||
- The data plane is a standalone project-owned C++ Shard worker with gRPC/Protobuf and a project-owned `ShardEngine` boundary.
|
||||
- llama.cpp is fetched at one exact commit into an ignored workspace from an in-repo manifest, then a numbered minimal patch stack is applied. There is no submodule, vendored tree, or permanent-fork dependency.
|
||||
- llama.cpp owns DeepSeek V4 graphs, mHC, MoE, attention, hash routing, and kernels. Meshnet adds only range-ownership hooks, typed boundary/local-state adapters, worker integration, and parity/certification.
|
||||
- Quantization and placement are dynamic recipe inputs. The 2–4 and 10+ stage layouts are certification scenarios, never product constants.
|
||||
- Per-shard Hot KV and V4 CSA/HCA/SWA/indexer/compressor state remain local and keyed by route session/epoch. The WAN seam carries the typed mHC 4×4096 residual boundary, positions, token-ID sideband where required, and schema/cache expectations—not per-layer caches.
|
||||
- Route changes use cache miss plus re-prefill/restart. There is no WAN KV or V4 auxiliary-cache migration.
|
||||
- CPU/CUDA/ROCm/Vulkan/Metal compile lanes are planned; only exact real-hardware-certified backend/model/recipe lanes may be advertised.
|
||||
- Alpha requires correctness and the pre-locked useful-speed gate. MTP is reserved and off for alpha; its ownership contract, implementation, and benchmark are required before beta.
|
||||
|
||||
```text
|
||||
Meshnet control plane
|
||||
-> versioned gRPC/Protobuf Shard protocol
|
||||
-> project-owned standalone C++ worker
|
||||
-> small pinned llama.cpp patch stack
|
||||
```
|
||||
## Target identities
|
||||
|
||||
Transformers/safetensors remains the correctness baseline. vLLM remains an optional complete managed provider and a design donor; it is not forked into the public mesh.
|
||||
- DeepSeek V4 official target SHA: `60d8d70770c6776ff598c94bb586a859a38244f1`.
|
||||
- llama.cpp V4 support lineage began at PR 24162 / merge `8c146a8366304c871efc26057cc90370ccf58dad`; DGR-027 later pins one exact validated current commit.
|
||||
- V4 scope: 43 main layers plus MTP; mHC 4×4096 boundary; 256 routed + 1 shared experts with six routed active; token IDs required for the first three hash-routed layers.
|
||||
- Exact split-GGUF artifacts are provisioned to mounted-drive storage with a complete hashed manifest and resumable verification; no model artifact may be placed under `/home`.
|
||||
|
||||
## Planning artifacts
|
||||
## Navigation
|
||||
|
||||
- **[Mandatory Ralph context](RALPH-CONTEXT.md)** — read first in every fresh iteration
|
||||
- [Task evidence contract](evidence/README.md)
|
||||
- [Implementation strategy](implementation-strategy.md)
|
||||
- [Current architecture](architecture.md)
|
||||
- [PRD](PRD.md)
|
||||
- [Ralph backlog](prd.json)
|
||||
- [ADR-0024](../../docs/adr/0024-distributed-gguf-runtime.md)
|
||||
- [Milestones](milestones.md)
|
||||
- [Issues](issues/)
|
||||
- [Distributed GGUF research](../../docs/research/distributed-gguf-landscape.md)
|
||||
- [GitHub follow-up](../../docs/research/distributed-gguf-github-followup.md)
|
||||
- [vLLM assessment](../../docs/research/vllm-distributed-gguf-assessment.md)
|
||||
|
||||
## Ralph execution
|
||||
|
||||
Use supervised one-story iterations for this high-risk runtime:
|
||||
|
||||
```bash
|
||||
ralph-tui run \
|
||||
--prd .scratch/distributed-gguf-runtime/prd.json \
|
||||
--agent claude --model opus \
|
||||
--iterations 1 --no-tui --no-setup --verify
|
||||
```
|
||||
|
||||
Inspect the diff, run the story gates, and commit one verified story before the next iteration. Real-model stories require the explicit environment gate and mounted-drive model storage.
|
||||
- [`prd.json`](prd.json) — sole authoritative 55-story backlog, DGR-017..071.
|
||||
- [`PRD.md`](PRD.md) — human-readable projection of goals, gates, and all stories.
|
||||
- [`RALPH-CONTEXT.md`](RALPH-CONTEXT.md) — mandatory fresh-session context.
|
||||
- [`architecture.md`](architecture.md), [`implementation-strategy.md`](implementation-strategy.md), [`milestones.md`](milestones.md) — design and execution sequence.
|
||||
- [`issues/`](issues/) — generated story specs; files 01..16 are retained legacy artifacts pending DGR-017.
|
||||
- [`evidence/`](evidence/) — provenance and future per-story handoffs.
|
||||
|
||||
Reference in New Issue
Block a user