84 lines
4.6 KiB
Markdown
84 lines
4.6 KiB
Markdown
# PRD: Distributed GGUF Runtime
|
|
|
|
## Summary
|
|
|
|
Build a distributed inference runtime that can serve large, quality-first open models by combining torrent-style model artifact distribution with sticky multi-node Inference Routes and per-shard local Hot KV State.
|
|
|
|
The first runtime proof uses the existing PyTorch route because it exposes model internals and cache semantics more directly. GGUF/llama.cpp becomes the performance path after the route-session contract is proven.
|
|
|
|
## Goals
|
|
|
|
- Eliminate full-prompt recompute in distributed decode.
|
|
- Keep decode activation seams proportional to `hidden_size`, not `context_length * hidden_size`.
|
|
- Keep Hot KV State local to the node serving the relevant Shard.
|
|
- Stream token deltas when feasible and always expose Generation Telemetry.
|
|
- Add a local full-model GGUF backend for immediate CPU performance wins.
|
|
- Define Model Artifact manifests so nodes can verify, seed, and advertise artifacts without depending on Hugging Face at request time.
|
|
- Prototype an upstreamable llama.cpp/libllama layer-boundary API.
|
|
- Use DeepSeek-V4-Flash as the first serious large-model target after smaller protocol smoke tests.
|
|
|
|
## Non-Goals
|
|
|
|
- No centralized hot KV cache in the per-token decode path.
|
|
- No automatic route repair in alpha.
|
|
- No permanent llama.cpp fork as the intended architecture.
|
|
- No GLM-5.2 or Ornith first; they remain follow-up support audits.
|
|
- No transport rewrite to QUIC/WebRTC before route/session semantics are proven.
|
|
|
|
## Resolved Decisions
|
|
|
|
- Public-network Shards are contiguous transformer layer ranges.
|
|
- Tensor/ring parallelism belongs inside one trusted node, one colocated pod, or a future composite node abstraction.
|
|
- Hot KV State is local to route nodes; Prefix Snapshots are optional cold recovery/reuse artifacts.
|
|
- PyTorch distributed KV/session semantics are proven before llama.cpp distributed execution.
|
|
- Streaming responses are preferred; Generation Telemetry is mandatory.
|
|
- llama.cpp/GGUF work targets upstreamable `libllama`/ggml hooks.
|
|
- Alpha fails Route Sessions on route-node loss.
|
|
- v1 activation transfer stays on binary HTTP.
|
|
|
|
## Target User Experience
|
|
|
|
A client sends an OpenAI-compatible request. The Gateway or Tracker Node accepts the request, creates a Route Session, and streams token deltas when supported. The client receives live Generation Telemetry for route phase, prefill progress, generated token count, rolling tokens/sec, route health, and failure reason.
|
|
|
|
If a route node drops in alpha, the request fails clearly. A retry starts a new Route Session from scratch.
|
|
|
|
## Runtime Shape
|
|
|
|
```text
|
|
client request
|
|
-> Gateway / Tracker Node creates Route Session
|
|
-> Tracker selects sticky Inference Route
|
|
-> prefill:
|
|
prompt chunks move through Shards
|
|
each node appends local Hot KV State
|
|
-> decode:
|
|
one-step activation moves through Shards
|
|
each node reads/appends local Hot KV State
|
|
tail returns token/logits
|
|
-> client receives streamed token deltas where possible
|
|
-> Generation Telemetry continues until complete or failed
|
|
```
|
|
|
|
## Milestones
|
|
|
|
| Milestone | Outcome | Issues |
|
|
|---|---|---|
|
|
| M1 — Session protocol proof | Stub route has stable Route Sessions, prefill/decode split, telemetry, and streaming contract | 01, 02, 03 |
|
|
| M2 — PyTorch reference route | Distributed PyTorch decode uses local per-shard cache and stops full-prompt recompute | 04 |
|
|
| M3 — Local GGUF performance path | Single-node GGUF backend serves through the node API and reports backend metadata | 05 |
|
|
| M4 — Artifact plane | Model Artifact manifest supports verification, layer mapping, and node advertisement | 06 |
|
|
| M5 — llama.cpp collaboration proof | Localhost layer-boundary prototype identifies upstreamable llama.cpp/libllama API | 07 |
|
|
| M6 — Networked GGUF route | Multi-node GGUF route uses the resolved protocol and fails cleanly on node loss | 08 |
|
|
| M7 — First large model | DeepSeek-V4-Flash support path is audited and converted into follow-up runtime tasks | 09 |
|
|
|
|
## Acceptance Criteria
|
|
|
|
- A two-node route can prefill once and decode without resending full prompt activations.
|
|
- Decode seam payload is one token/hidden-state step after prefill.
|
|
- Route Session telemetry is visible before first token and during decode.
|
|
- Streaming token deltas work where the backend supports them.
|
|
- Route-node loss produces a structured alpha failure and does not attempt unsafe repair.
|
|
- A local GGUF model can serve via the node API.
|
|
- A Model Artifact manifest can prove which Shards a node can serve.
|
|
- DeepSeek-V4-Flash has a written support recommendation: PyTorch, vLLM/SGLang, llama.cpp/GGUF, or blocked.
|