neuron-tai/.scratch/distributed-gguf-runtime/PRD.md

PRD: Distributed GGUF Runtime

Summary

Build a distributed inference runtime that can serve large, quality-first open models by combining torrent-style model artifact distribution with sticky multi-node Inference Routes and per-shard local Hot KV State.

The first runtime proof uses the existing PyTorch route because it exposes model internals and cache semantics more directly. GGUF/llama.cpp becomes the performance path after the route-session contract is proven.

Goals

• Eliminate full-prompt recompute in distributed decode.
• Keep decode activation seams proportional to hidden_size, not context_length * hidden_size.
• Keep Hot KV State local to the node serving the relevant Shard.
• Stream token deltas when feasible and always expose Generation Telemetry.
• Add a local full-model GGUF backend for immediate CPU performance wins.
• Define Model Artifact manifests so nodes can verify, seed, and advertise artifacts without depending on Hugging Face at request time.
• Prototype an upstreamable llama.cpp/libllama layer-boundary API.
• Use DeepSeek-V4-Flash as the first serious large-model target after smaller protocol smoke tests.

Non-Goals

• No centralized hot KV cache in the per-token decode path.
• No automatic route repair in alpha.
• No permanent llama.cpp fork as the intended architecture.
• No GLM-5.2 or Ornith first; they remain follow-up support audits.
• No transport rewrite to QUIC/WebRTC before route/session semantics are proven.

Resolved Decisions

• Public-network Shards are contiguous transformer layer ranges.
• Tensor/ring parallelism belongs inside one trusted node, one colocated pod, or a future composite node abstraction.
• Hot KV State is local to route nodes; Prefix Snapshots are optional cold recovery/reuse artifacts.
• PyTorch distributed KV/session semantics are proven before llama.cpp distributed execution.
• Streaming responses are preferred; Generation Telemetry is mandatory.
• llama.cpp/GGUF work targets upstreamable libllama/ggml hooks.
• Alpha fails Route Sessions on route-node loss.
• v1 activation transfer stays on binary HTTP.

Target User Experience

A client sends an OpenAI-compatible request. The Gateway or Tracker Node accepts the request, creates a Route Session, and streams token deltas when supported. The client receives live Generation Telemetry for route phase, prefill progress, generated token count, rolling tokens/sec, route health, and failure reason.

If a route node drops in alpha, the request fails clearly. A retry starts a new Route Session from scratch.

Runtime Shape

client request
  -> Gateway / Tracker Node creates Route Session
  -> Tracker selects sticky Inference Route
  -> prefill:
       prompt chunks move through Shards
       each node appends local Hot KV State
  -> decode:
       one-step activation moves through Shards
       each node reads/appends local Hot KV State
       tail returns token/logits
  -> client receives streamed token deltas where possible
  -> Generation Telemetry continues until complete or failed

Milestones

Milestone	Outcome	Issues
M1 — Session protocol proof	Stub route has stable Route Sessions, prefill/decode split, telemetry, and streaming contract	01, 02, 03
M2 — PyTorch reference route	Distributed PyTorch decode uses local per-shard cache and stops full-prompt recompute	04
M3 — Local GGUF performance path	Single-node GGUF backend serves through the node API and reports backend metadata	05
M4 — Artifact plane	Model Artifact manifest supports verification, layer mapping, and node advertisement	06
M5 — llama.cpp collaboration proof	Localhost layer-boundary prototype identifies upstreamable llama.cpp/libllama API	07
M6 — Networked GGUF route	Multi-node GGUF route uses the resolved protocol and fails cleanly on node loss	08
M7 — First large model	DeepSeek-V4-Flash support path is audited and converted into follow-up runtime tasks	09

Acceptance Criteria

• A two-node route can prefill once and decode without resending full prompt activations.
• Decode seam payload is one token/hidden-state step after prefill.
• Route Session telemetry is visible before first token and during decode.
• Streaming token deltas work where the backend supports them.
• Route-node loss produces a structured alpha failure and does not attempt unsafe repair.
• A local GGUF model can serve via the node API.
• A Model Artifact manifest can prove which Shards a node can serve.
• DeepSeek-V4-Flash has a written support recommendation: PyTorch, vLLM/SGLang, llama.cpp/GGUF, or blocked.