neuron-tai/.scratch/distributed-gguf-runtime/milestones.md

Distributed GGUF Runtime Milestones

Proposed Breakdown

Order	Issue	Title	Blocked by	User-visible proof
1	01	Route Session lifecycle	None	Stable route/session status and cleanup
2	02	Prefill/decode binary HTTP protocol	01	Stub route proves prefill chunks and one-step decode payloads
3	03	Generation Telemetry and streaming response contract	01	Client sees route progress and streamed deltas when available
4	04	PyTorch distributed KV reference route	01, 02, 03	Distributed PyTorch decode stops full-prompt recompute
5	05	Local llama.cpp/GGUF backend	None	Local GGUF model serves through node API
6	06	Model Artifact manifest and Shard advertisement	01	Node verifies artifacts and advertises serveable Shards
7	07	llama.cpp layer-boundary prototype	02, 05, 06	Local two-process GGUF route identifies upstream API
8	08	Networked distributed GGUF route	03, 04, 06, 07	Two machines serve one GGUF route with telemetry
9	09	DeepSeek-V4-Flash support audit	None	Runtime recommendation for first serious large model
10	10	GLM-5.2 and Ornith follow-up support audit	09	Follow-up compatibility matrix and upstream blockers

First Three To Implement

1. 01 — Route Session lifecycle: makes every later cache, telemetry, and route decision concrete.
2. 02 — Prefill/decode binary HTTP protocol: proves the payload shape and route/session headers before model internals.
3. 03 — Generation Telemetry and streaming response contract: gives every later long-running route a visible user experience and failure surface.

Parallel Work

• 05 — Local llama.cpp/GGUF backend can run in parallel with 01–03 because it is a full-model local backend.
• 09 — DeepSeek-V4-Flash support audit can run in parallel because it is research/compatibility work.

Human-Gated Work

• 07 — llama.cpp layer-boundary prototype is the collaboration point with Georgi/upstream llama.cpp.
• 08 — Networked distributed GGUF route should wait until the PyTorch reference route proves the cache/session contract.