docs: define distributed GGUF runtime plan

This commit is contained in:
Dobromir Popov
2026-07-13 15:09:27 +03:00
parent b5fa7245df
commit 4cae4a6c5c
42 changed files with 4913 additions and 691 deletions

View File

@@ -1,32 +1,35 @@
# Distributed GGUF Runtime Milestones
# Distributed GGUF runtime milestones
## Proposed Breakdown
## Gate A — measured runtime value
| Order | Issue | Title | Blocked by | User-visible proof |
|---:|---|---|---|---|
| 1 | [01](./issues/01-route-session-lifecycle.md) | Route Session lifecycle | None | Stable route/session status and cleanup |
| 2 | [02](./issues/02-prefill-decode-binary-http.md) | Prefill/decode binary HTTP protocol | 01 | Stub route proves prefill chunks and one-step decode payloads |
| 3 | [03](./issues/03-generation-telemetry-and-streaming.md) | Generation Telemetry and streaming response contract | 01 | Client sees route progress and streamed deltas when available |
| 4 | [04](./issues/04-pytorch-distributed-kv-reference.md) | PyTorch distributed KV reference route | 01, 02, 03 | Distributed PyTorch decode stops full-prompt recompute |
| 5 | [05](./issues/05-local-llamacpp-gguf-backend.md) | Local llama.cpp/GGUF backend | None | Local GGUF model serves through node API |
| 6 | [06](./issues/06-model-artifact-manifest.md) | Model Artifact manifest and Shard advertisement | 01 | Node verifies artifacts and advertises serveable Shards |
| 7 | [07](./issues/07-llamacpp-layer-boundary-prototype.md) | llama.cpp layer-boundary prototype | 02, 05, 06 | Local two-process GGUF route identifies upstream API |
| 8 | [08](./issues/08-networked-distributed-gguf-route.md) | Networked distributed GGUF route | 03, 04, 06, 07 | Two machines serve one GGUF route with telemetry |
| 9 | [09](./issues/09-deepseek-v4-flash-support-audit.md) | DeepSeek-V4-Flash support audit | None | Runtime recommendation for first serious large model |
| 10 | [10](./issues/10-glm52-ornith-followup-audit.md) | GLM-5.2 and Ornith follow-up support audit | 09 | Follow-up compatibility matrix and upstream blockers |
- DGR-001 locks the safetensors-versus-GGUF performance/fit/quality contract.
- DGR-002 can proceed independently and defines the battle-proven backend-neutral wire protocol.
- DGR-003 builds exact recipe identity on DGR-002.
- Expensive native llama.cpp work remains gated by DGR-001.
## First Three To Implement
## Gate B — minimal native execution seam
1. **01 — Route Session lifecycle**: makes every later cache, telemetry, and route decision concrete.
2. **02 — Prefill/decode binary HTTP protocol**: proves the payload shape and route/session headers before model internals.
3. **03 — Generation Telemetry and streaming response contract**: gives every later long-running route a visible user experience and failure surface.
- DGR-004 creates the reproducible pinned fork boundary.
- DGR-005 implements dense-Llama range ownership.
- DGR-006 proves architecture-defined boundary parity.
## Parallel Work
## Gate C — concurrent production worker
- **05 — Local llama.cpp/GGUF backend** can run in parallel with 0103 because it is a full-model local backend.
- **09 — DeepSeek-V4-Flash support audit** can run in parallel because it is research/compatibility work.
- DGR-007 isolates concurrent Hot KV State.
- DGR-008 exposes the native worker over gRPC.
- DGR-009 integrates the worker without replacing Meshnet's control plane.
- DGR-010 passes local real-model two-process acceptance.
## Human-Gated Work
## Gate D — real consumer-hardware route
- **07 — llama.cpp layer-boundary prototype** is the collaboration point with Georgi/upstream llama.cpp.
- **08 — Networked distributed GGUF route** should wait until the PyTorch reference route proves the cache/session contract.
- DGR-011 passes two-physical-machine execution.
- DGR-012 adds continuous batching and bounded admission.
- DGR-013 hardens failure and cancellation.
## Gate E — product release decision
- DGR-014 compares distributed GGUF against the current distributed safetensors route under locked thresholds.
- DGR-015 adds Qwen3/Qwen3-MoE only after the dense runtime passes.
- DGR-016 prepares narrow upstream llama.cpp collaboration material.
No later gate may be claimed from synthetic workers or documentation-only evidence.