# PRD: Distributed GGUF Runtime

## Summary

Build a distributed inference runtime that can serve large, quality-first open models by combining torrent-style model artifact distribution with sticky multi-node Inference Routes and per-shard local Hot KV State.

The first runtime proof uses the existing PyTorch route because it exposes model internals and cache semantics more directly. GGUF/llama.cpp becomes the performance path after the route-session contract is proven.

## Goals

- Eliminate full-prompt recompute in distributed decode.
- Keep decode activation seams proportional to `hidden_size`, not `context_length * hidden_size`.
- Keep Hot KV State local to the node serving the relevant Shard.
- Stream token deltas when feasible and always expose Generation Telemetry.
- Add a local full-model GGUF backend for immediate CPU performance wins.
- Define Model Artifact manifests so nodes can verify, seed, and advertise artifacts without depending on Hugging Face at request time.
- Prototype an upstreamable llama.cpp/libllama layer-boundary API.
- Use DeepSeek-V4-Flash as the first serious large-model target after smaller protocol smoke tests.

## Non-Goals

- No centralized hot KV cache in the per-token decode path.
- No automatic route repair in alpha.
- No permanent llama.cpp fork as the intended architecture.
- No GLM-5.2 or Ornith first; they remain follow-up support audits.
- No transport rewrite to QUIC/WebRTC before route/session semantics are proven.

## Resolved Decisions

- Public-network Shards are contiguous transformer layer ranges.
- Tensor/ring parallelism belongs inside one trusted node, one colocated pod, or a future composite node abstraction.
- Hot KV State is local to route nodes; Prefix Snapshots are optional cold recovery/reuse artifacts.
- PyTorch distributed KV/session semantics are proven before llama.cpp distributed execution.
- Streaming responses are preferred; Generation Telemetry is mandatory.
- llama.cpp/GGUF work targets upstreamable `libllama`/ggml hooks.
- Alpha fails Route Sessions on route-node loss.
- v1 activation transfer stays on binary HTTP.

## Target User Experience

A client sends an OpenAI-compatible request. The Gateway or Tracker Node accepts the request, creates a Route Session, and streams token deltas when supported. The client receives live Generation Telemetry for route phase, prefill progress, generated token count, rolling tokens/sec, route health, and failure reason.

If a route node drops in alpha, the request fails clearly. A retry starts a new Route Session from scratch.

## Runtime Shape

```text
client request
  -> Gateway / Tracker Node creates Route Session
  -> Tracker selects sticky Inference Route
  -> prefill:
       prompt chunks move through Shards
       each node appends local Hot KV State
  -> decode:
       one-step activation moves through Shards
       each node reads/appends local Hot KV State
       tail returns token/logits
  -> client receives streamed token deltas where possible
  -> Generation Telemetry continues until complete or failed
```

## Milestones

| Milestone | Outcome | Issues |
|---|---|---|
| M1 — Session protocol proof | Stub route has stable Route Sessions, prefill/decode split, telemetry, and streaming contract | 01, 02, 03 |
| M2 — PyTorch reference route | Distributed PyTorch decode uses local per-shard cache and stops full-prompt recompute | 04 |
| M3 — Local GGUF performance path | Single-node GGUF backend serves through the node API and reports backend metadata | 05 |
| M4 — Artifact plane | Model Artifact manifest supports verification, layer mapping, and node advertisement | 06 |
| M5 — llama.cpp collaboration proof | Localhost layer-boundary prototype identifies upstreamable llama.cpp/libllama API | 07 |
| M6 — Networked GGUF route | Multi-node GGUF route uses the resolved protocol and fails cleanly on node loss | 08 |
| M7 — First large model | DeepSeek-V4-Flash support path is audited and converted into follow-up runtime tasks | 09 |

## Acceptance Criteria

- A two-node route can prefill once and decode without resending full prompt activations.
- Decode seam payload is one token/hidden-state step after prefill.
- Route Session telemetry is visible before first token and during decode.
- Streaming token deltas work where the backend supports them.
- Route-node loss produces a structured alpha failure and does not attempt unsafe repair.
- A local GGUF model can serve via the node API.
- A Model Artifact manifest can prove which Shards a node can serve.
- DeepSeek-V4-Flash has a written support recommendation: PyTorch, vLLM/SGLang, llama.cpp/GGUF, or blocked.