popov/neuron-tai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

tasks

Browse Source
This commit is contained in:
Dobromir Popov
2026-07-07 15:56:38 +03:00
parent ca49675f50
commit eac852a515
22 changed files with 343 additions and 262 deletions
Download Patch File Download Diff File Expand all files Collapse all files

25
.scratch/distributed-gguf-runtime/README.md
View File

@@ -15,7 +15,9 @@ This scratch supersedes the old assumption in [ADR-0001](../../docs/adr/0001-pyt
| [decision-framework.md](./decision-framework.md) | Grilling framework for open decisions and recommended answers |
| [research-prior-art.md](./research-prior-art.md) | Prior-art notes for Petals, exo, Distributed Llama, prima.cpp, llama.cpp, DeepSeek-V4-Flash, GLM-5.2, and Ornith |
| [ADR-0020-distributed-gguf-runtime.md](./ADR-0020-distributed-gguf-runtime.md) | Draft decision record for the GGUF/llama.cpp distributed runtime |
| [issues/](./issues/) | Implementation slices in dependency order |
| [PRD.md](./PRD.md) | Product/runtime requirements and acceptance criteria |
| [milestones.md](./milestones.md) | Dependency-ordered implementation milestones |
| [issues/](./issues/) | Implementation-ready tracer-bullet issue briefs |
## Decision Summary
@@ -40,15 +42,18 @@ Adopt a hybrid runtime:
## Recommended Order
1. Local llama.cpp/GGUF backend for full-model serving.
2. Stable distributed session ID and per-shard KV cache in the existing PyTorch path.
3. Binary prefill/decode protocol split: chunked prefill, one-step decode.
4. Route-session Generation Telemetry and streaming response support where feasible.
5. GGUF artifact manifest and torrent seeding.
6. llama.cpp layer-boundary prototype on localhost.
7. Networked distributed GGUF route.
8. DeepSeek-V4-Flash as first serious large-model target.
9. GLM-5.2 / DSA / MLA and Ornith support once runtime support is confirmed.
See [milestones.md](./milestones.md) for the full dependency map.
1. [01 — Route Session lifecycle](./issues/01-route-session-lifecycle.md)
2. [02 — Prefill/decode binary HTTP protocol](./issues/02-prefill-decode-binary-http.md)
3. [03 — Generation Telemetry and streaming response contract](./issues/03-generation-telemetry-and-streaming.md)
4. [04 — PyTorch distributed KV reference route](./issues/04-pytorch-distributed-kv-reference.md)
5. [05 — Local llama.cpp/GGUF backend](./issues/05-local-llamacpp-gguf-backend.md)
6. [06 — Model Artifact manifest and Shard advertisement](./issues/06-model-artifact-manifest.md)
7. [07 — llama.cpp layer-boundary prototype](./issues/07-llamacpp-layer-boundary-prototype.md)
8. [08 — Networked distributed GGUF route](./issues/08-networked-distributed-gguf-route.md)
9. [09 — DeepSeek-V4-Flash support audit](./issues/09-deepseek-v4-flash-support-audit.md)
10. [10 — GLM-5.2 and Ornith follow-up support audit](./issues/10-glm52-ornith-followup-audit.md)
## Open Questions