33 lines
2.6 KiB
Markdown
33 lines
2.6 KiB
Markdown
# Distributed GGUF Runtime Milestones
|
||
|
||
## Proposed Breakdown
|
||
|
||
| Order | Issue | Title | Blocked by | User-visible proof |
|
||
|---:|---|---|---|---|
|
||
| 1 | [01](./issues/01-route-session-lifecycle.md) | Route Session lifecycle | None | Stable route/session status and cleanup |
|
||
| 2 | [02](./issues/02-prefill-decode-binary-http.md) | Prefill/decode binary HTTP protocol | 01 | Stub route proves prefill chunks and one-step decode payloads |
|
||
| 3 | [03](./issues/03-generation-telemetry-and-streaming.md) | Generation Telemetry and streaming response contract | 01 | Client sees route progress and streamed deltas when available |
|
||
| 4 | [04](./issues/04-pytorch-distributed-kv-reference.md) | PyTorch distributed KV reference route | 01, 02, 03 | Distributed PyTorch decode stops full-prompt recompute |
|
||
| 5 | [05](./issues/05-local-llamacpp-gguf-backend.md) | Local llama.cpp/GGUF backend | None | Local GGUF model serves through node API |
|
||
| 6 | [06](./issues/06-model-artifact-manifest.md) | Model Artifact manifest and Shard advertisement | 01 | Node verifies artifacts and advertises serveable Shards |
|
||
| 7 | [07](./issues/07-llamacpp-layer-boundary-prototype.md) | llama.cpp layer-boundary prototype | 02, 05, 06 | Local two-process GGUF route identifies upstream API |
|
||
| 8 | [08](./issues/08-networked-distributed-gguf-route.md) | Networked distributed GGUF route | 03, 04, 06, 07 | Two machines serve one GGUF route with telemetry |
|
||
| 9 | [09](./issues/09-deepseek-v4-flash-support-audit.md) | DeepSeek-V4-Flash support audit | None | Runtime recommendation for first serious large model |
|
||
| 10 | [10](./issues/10-glm52-ornith-followup-audit.md) | GLM-5.2 and Ornith follow-up support audit | 09 | Follow-up compatibility matrix and upstream blockers |
|
||
|
||
## First Three To Implement
|
||
|
||
1. **01 — Route Session lifecycle**: makes every later cache, telemetry, and route decision concrete.
|
||
2. **02 — Prefill/decode binary HTTP protocol**: proves the payload shape and route/session headers before model internals.
|
||
3. **03 — Generation Telemetry and streaming response contract**: gives every later long-running route a visible user experience and failure surface.
|
||
|
||
## Parallel Work
|
||
|
||
- **05 — Local llama.cpp/GGUF backend** can run in parallel with 01–03 because it is a full-model local backend.
|
||
- **09 — DeepSeek-V4-Flash support audit** can run in parallel because it is research/compatibility work.
|
||
|
||
## Human-Gated Work
|
||
|
||
- **07 — llama.cpp layer-boundary prototype** is the collaboration point with Georgi/upstream llama.cpp.
|
||
- **08 — Networked distributed GGUF route** should wait until the PyTorch reference route proves the cache/session contract.
|