docs: define distributed GGUF runtime plan
This commit is contained in:
@@ -1,32 +1,35 @@
|
||||
# Distributed GGUF Runtime Milestones
|
||||
# Distributed GGUF runtime milestones
|
||||
|
||||
## Proposed Breakdown
|
||||
## Gate A — measured runtime value
|
||||
|
||||
| Order | Issue | Title | Blocked by | User-visible proof |
|
||||
|---:|---|---|---|---|
|
||||
| 1 | [01](./issues/01-route-session-lifecycle.md) | Route Session lifecycle | None | Stable route/session status and cleanup |
|
||||
| 2 | [02](./issues/02-prefill-decode-binary-http.md) | Prefill/decode binary HTTP protocol | 01 | Stub route proves prefill chunks and one-step decode payloads |
|
||||
| 3 | [03](./issues/03-generation-telemetry-and-streaming.md) | Generation Telemetry and streaming response contract | 01 | Client sees route progress and streamed deltas when available |
|
||||
| 4 | [04](./issues/04-pytorch-distributed-kv-reference.md) | PyTorch distributed KV reference route | 01, 02, 03 | Distributed PyTorch decode stops full-prompt recompute |
|
||||
| 5 | [05](./issues/05-local-llamacpp-gguf-backend.md) | Local llama.cpp/GGUF backend | None | Local GGUF model serves through node API |
|
||||
| 6 | [06](./issues/06-model-artifact-manifest.md) | Model Artifact manifest and Shard advertisement | 01 | Node verifies artifacts and advertises serveable Shards |
|
||||
| 7 | [07](./issues/07-llamacpp-layer-boundary-prototype.md) | llama.cpp layer-boundary prototype | 02, 05, 06 | Local two-process GGUF route identifies upstream API |
|
||||
| 8 | [08](./issues/08-networked-distributed-gguf-route.md) | Networked distributed GGUF route | 03, 04, 06, 07 | Two machines serve one GGUF route with telemetry |
|
||||
| 9 | [09](./issues/09-deepseek-v4-flash-support-audit.md) | DeepSeek-V4-Flash support audit | None | Runtime recommendation for first serious large model |
|
||||
| 10 | [10](./issues/10-glm52-ornith-followup-audit.md) | GLM-5.2 and Ornith follow-up support audit | 09 | Follow-up compatibility matrix and upstream blockers |
|
||||
- DGR-001 locks the safetensors-versus-GGUF performance/fit/quality contract.
|
||||
- DGR-002 can proceed independently and defines the battle-proven backend-neutral wire protocol.
|
||||
- DGR-003 builds exact recipe identity on DGR-002.
|
||||
- Expensive native llama.cpp work remains gated by DGR-001.
|
||||
|
||||
## First Three To Implement
|
||||
## Gate B — minimal native execution seam
|
||||
|
||||
1. **01 — Route Session lifecycle**: makes every later cache, telemetry, and route decision concrete.
|
||||
2. **02 — Prefill/decode binary HTTP protocol**: proves the payload shape and route/session headers before model internals.
|
||||
3. **03 — Generation Telemetry and streaming response contract**: gives every later long-running route a visible user experience and failure surface.
|
||||
- DGR-004 creates the reproducible pinned fork boundary.
|
||||
- DGR-005 implements dense-Llama range ownership.
|
||||
- DGR-006 proves architecture-defined boundary parity.
|
||||
|
||||
## Parallel Work
|
||||
## Gate C — concurrent production worker
|
||||
|
||||
- **05 — Local llama.cpp/GGUF backend** can run in parallel with 01–03 because it is a full-model local backend.
|
||||
- **09 — DeepSeek-V4-Flash support audit** can run in parallel because it is research/compatibility work.
|
||||
- DGR-007 isolates concurrent Hot KV State.
|
||||
- DGR-008 exposes the native worker over gRPC.
|
||||
- DGR-009 integrates the worker without replacing Meshnet's control plane.
|
||||
- DGR-010 passes local real-model two-process acceptance.
|
||||
|
||||
## Human-Gated Work
|
||||
## Gate D — real consumer-hardware route
|
||||
|
||||
- **07 — llama.cpp layer-boundary prototype** is the collaboration point with Georgi/upstream llama.cpp.
|
||||
- **08 — Networked distributed GGUF route** should wait until the PyTorch reference route proves the cache/session contract.
|
||||
- DGR-011 passes two-physical-machine execution.
|
||||
- DGR-012 adds continuous batching and bounded admission.
|
||||
- DGR-013 hardens failure and cancellation.
|
||||
|
||||
## Gate E — product release decision
|
||||
|
||||
- DGR-014 compares distributed GGUF against the current distributed safetensors route under locked thresholds.
|
||||
- DGR-015 adds Qwen3/Qwen3-MoE only after the dense runtime passes.
|
||||
- DGR-016 prepares narrow upstream llama.cpp collaboration material.
|
||||
|
||||
No later gate may be claimed from synthetic workers or documentation-only evidence.
|
||||
|
||||
Reference in New Issue
Block a user