Merge branch 'master' of https://git.d-popov.com/popov/neuron-tai

.scratch/distributed-gguf-runtime/PRD.md

@@ -0,0 +1,83 @@
+# PRD: Distributed GGUF Runtime
+
+## Summary
+
+Build a distributed inference runtime that can serve large, quality-first open models by combining torrent-style model artifact distribution with sticky multi-node Inference Routes and per-shard local Hot KV State.
+
+The first runtime proof uses the existing PyTorch route because it exposes model internals and cache semantics more directly. GGUF/llama.cpp becomes the performance path after the route-session contract is proven.
+
+## Goals
+
+- Eliminate full-prompt recompute in distributed decode.
+- Keep decode activation seams proportional to `hidden_size`, not `context_length * hidden_size`.
+- Keep Hot KV State local to the node serving the relevant Shard.
+- Stream token deltas when feasible and always expose Generation Telemetry.
+- Add a local full-model GGUF backend for immediate CPU performance wins.
+- Define Model Artifact manifests so nodes can verify, seed, and advertise artifacts without depending on Hugging Face at request time.
+- Prototype an upstreamable llama.cpp/libllama layer-boundary API.
+- Use DeepSeek-V4-Flash as the first serious large-model target after smaller protocol smoke tests.
+
+## Non-Goals
+
+- No centralized hot KV cache in the per-token decode path.
+- No automatic route repair in alpha.
+- No permanent llama.cpp fork as the intended architecture.
+- No GLM-5.2 or Ornith first; they remain follow-up support audits.
+- No transport rewrite to QUIC/WebRTC before route/session semantics are proven.
+
+## Resolved Decisions
+
+- Public-network Shards are contiguous transformer layer ranges.
+- Tensor/ring parallelism belongs inside one trusted node, one colocated pod, or a future composite node abstraction.
+- Hot KV State is local to route nodes; Prefix Snapshots are optional cold recovery/reuse artifacts.
+- PyTorch distributed KV/session semantics are proven before llama.cpp distributed execution.
+- Streaming responses are preferred; Generation Telemetry is mandatory.
+- llama.cpp/GGUF work targets upstreamable `libllama`/ggml hooks.
+- Alpha fails Route Sessions on route-node loss.
+- v1 activation transfer stays on binary HTTP.
+
+## Target User Experience
+
+A client sends an OpenAI-compatible request. The Gateway or Tracker Node accepts the request, creates a Route Session, and streams token deltas when supported. The client receives live Generation Telemetry for route phase, prefill progress, generated token count, rolling tokens/sec, route health, and failure reason.
+
+If a route node drops in alpha, the request fails clearly. A retry starts a new Route Session from scratch.
+
+## Runtime Shape
+
+```text
+client request
+  -> Gateway / Tracker Node creates Route Session
+  -> Tracker selects sticky Inference Route
+  -> prefill:
+       prompt chunks move through Shards
+       each node appends local Hot KV State
+  -> decode:
+       one-step activation moves through Shards
+       each node reads/appends local Hot KV State
+       tail returns token/logits
+  -> client receives streamed token deltas where possible
+  -> Generation Telemetry continues until complete or failed
+```
+
+## Milestones
+
+| Milestone | Outcome | Issues |
+|---|---|---|
+| M1 — Session protocol proof | Stub route has stable Route Sessions, prefill/decode split, telemetry, and streaming contract | 01, 02, 03 |
+| M2 — PyTorch reference route | Distributed PyTorch decode uses local per-shard cache and stops full-prompt recompute | 04 |
+| M3 — Local GGUF performance path | Single-node GGUF backend serves through the node API and reports backend metadata | 05 |
+| M4 — Artifact plane | Model Artifact manifest supports verification, layer mapping, and node advertisement | 06 |
+| M5 — llama.cpp collaboration proof | Localhost layer-boundary prototype identifies upstreamable llama.cpp/libllama API | 07 |
+| M6 — Networked GGUF route | Multi-node GGUF route uses the resolved protocol and fails cleanly on node loss | 08 |
+| M7 — First large model | DeepSeek-V4-Flash support path is audited and converted into follow-up runtime tasks | 09 |
+
+## Acceptance Criteria
+
+- A two-node route can prefill once and decode without resending full prompt activations.
+- Decode seam payload is one token/hidden-state step after prefill.
+- Route Session telemetry is visible before first token and during decode.
+- Streaming token deltas work where the backend supports them.
+- Route-node loss produces a structured alpha failure and does not attempt unsafe repair.
+- A local GGUF model can serve via the node API.
+- A Model Artifact manifest can prove which Shards a node can serve.
+- DeepSeek-V4-Flash has a written support recommendation: PyTorch, vLLM/SGLang, llama.cpp/GGUF, or blocked.

.scratch/distributed-gguf-runtime/README.md

@@ -15,7 +15,9 @@ This scratch supersedes the old assumption in [ADR-0001](../../docs/adr/0001-pyt
 | [decision-framework.md](./decision-framework.md) | Grilling framework for open decisions and recommended answers |
 | [research-prior-art.md](./research-prior-art.md) | Prior-art notes for Petals, exo, Distributed Llama, prima.cpp, llama.cpp, DeepSeek-V4-Flash, GLM-5.2, and Ornith |
 | [ADR-0020-distributed-gguf-runtime.md](./ADR-0020-distributed-gguf-runtime.md) | Draft decision record for the GGUF/llama.cpp distributed runtime |
-| [issues/](./issues/) | Implementation slices in dependency order |
+| [PRD.md](./PRD.md) | Product/runtime requirements and acceptance criteria |
+| [milestones.md](./milestones.md) | Dependency-ordered implementation milestones |
+| [issues/](./issues/) | Implementation-ready tracer-bullet issue briefs |
 
 ## Decision Summary
 
@@ -40,15 +42,18 @@ Adopt a hybrid runtime:
 
 ## Recommended Order
 
-1. Local llama.cpp/GGUF backend for full-model serving.
+See [milestones.md](./milestones.md) for the full dependency map.
-2. Stable distributed session ID and per-shard KV cache in the existing PyTorch path.
+
-3. Binary prefill/decode protocol split: chunked prefill, one-step decode.
+1. [01 — Route Session lifecycle](./issues/01-route-session-lifecycle.md)
-4. Route-session Generation Telemetry and streaming response support where feasible.
+2. [02 — Prefill/decode binary HTTP protocol](./issues/02-prefill-decode-binary-http.md)
-5. GGUF artifact manifest and torrent seeding.
+3. [03 — Generation Telemetry and streaming response contract](./issues/03-generation-telemetry-and-streaming.md)
-6. llama.cpp layer-boundary prototype on localhost.
+4. [04 — PyTorch distributed KV reference route](./issues/04-pytorch-distributed-kv-reference.md)
-7. Networked distributed GGUF route.
+5. [05 — Local llama.cpp/GGUF backend](./issues/05-local-llamacpp-gguf-backend.md)
-8. DeepSeek-V4-Flash as first serious large-model target.
+6. [06 — Model Artifact manifest and Shard advertisement](./issues/06-model-artifact-manifest.md)
-9. GLM-5.2 / DSA / MLA and Ornith support once runtime support is confirmed.
+7. [07 — llama.cpp layer-boundary prototype](./issues/07-llamacpp-layer-boundary-prototype.md)
+8. [08 — Networked distributed GGUF route](./issues/08-networked-distributed-gguf-route.md)
+9. [09 — DeepSeek-V4-Flash support audit](./issues/09-deepseek-v4-flash-support-audit.md)
+10. [10 — GLM-5.2 and Ornith follow-up support audit](./issues/10-glm52-ornith-followup-audit.md)
 
 ## Open Questions
 

.scratch/distributed-gguf-runtime/issues/01-local-llamacpp-gguf-backend.md

@@ -1,29 +0,0 @@
-# 01 — Local llama.cpp/GGUF backend
-
-Status: ready-for-agent
-
-## Goal
-
-Add a local full-model llama.cpp/GGUF backend to the node so a machine that can hold a GGUF model can serve it through the existing OpenAI-compatible node API.
-
-## Scope
-
-- Add backend selection for `llama.cpp` / GGUF.
-- Support launching or calling `llama-server` first; direct `libllama` bindings may come later.
-- Register model metadata and hardware profile with tracker.
-- Preserve current PyTorch path.
-- Add a local benchmark comparing PyTorch CPU vs llama.cpp/GGUF for the same supported small model.
-
-## Non-Goals
-
-- No distributed GGUF route yet.
-- No partial layer loading yet.
-- No torrent seeding yet.
-
-## Acceptance
-
-- A local GGUF model can answer `/v1/chat/completions`.
-- Startup output clearly says backend=`llama.cpp`.
-- Node registration includes backend and artifact metadata.
-- Test or smoke script verifies the backend wiring without requiring a huge model.
-

.scratch/distributed-gguf-runtime/issues/01-route-session-lifecycle.md

@@ -0,0 +1,19 @@
+# 01 — Route Session lifecycle
+
+Status: ready-for-agent
+
+## What to build
+
+Add the narrowest end-to-end Route Session lifecycle that can be used by distributed inference routes: create a session, bind it to a selected Inference Route, expose status, and close it cleanly. This slice does not need real model cache yet; it proves stable session identity across the control plane and activation plane.
+
+## Acceptance criteria
+
+- [ ] A request can create a Route Session with a stable `session_id`, `route_id`, model preset, backend id, and route membership.
+- [ ] Every downstream activation request carries the same session identity and fails clearly if the session or route id does not match.
+- [ ] Session status reports phase, route nodes, model preset, backend id, created time, and last activity time.
+- [ ] Closing a session releases all registered per-session state.
+- [ ] Tests cover create, status, close, stale-session rejection, and wrong-route rejection.
+
+## Blocked by
+
+None - can start immediately.

.scratch/distributed-gguf-runtime/issues/02-prefill-decode-binary-http.md

@@ -0,0 +1,20 @@
+# 02 — Prefill/decode binary HTTP protocol
+
+Status: ready-for-agent
+
+## What to build
+
+Split the activation protocol into explicit prefill and decode-step calls using the existing binary HTTP direction from ADR-0008. The completed slice should work against a stub backend so payload shape, route/session headers, relay preservation, and failure behavior are testable before real KV cache work begins.
+
+## Acceptance criteria
+
+- [ ] Prefill accepts chunked binary activations with route/session metadata and forwards them through the selected route.
+- [ ] Decode-step accepts a one-step binary activation and forwards a one-step activation through the selected route.
+- [ ] Decode-step payload size is independent of prompt length in protocol tests.
+- [ ] Relay forwarding preserves route/session headers, shape, dtype, position, and wire version.
+- [ ] Legacy `/forward` either remains as a compatibility wrapper or fails with a clear wire-version error.
+- [ ] Tests cover prefill chunking, decode-step shape validation, relay preservation, and malformed header rejection.
+
+## Blocked by
+
+- 01 — Route Session lifecycle.

.scratch/distributed-gguf-runtime/issues/02-stable-session-and-distributed-kv-pytorch.md

@@ -1,33 +0,0 @@
-# 02 — Stable session and distributed KV in PyTorch path
-
-Status: ready-for-agent
-
-## Goal
-
-Fix the existing distributed PyTorch path so it does not recompute the full growing prompt for every output token.
-
-## Scope
-
-- Introduce stable `session_id` for one request/session.
-- Add per-node session cache keyed by `session_id`.
-- Split `/forward` semantics into prefill and decode-step.
-- Use model cache objects / `past_key_values` where supported.
-- Keep hot KV local to each shard node.
-- Add cleanup/TTL for abandoned sessions.
-
-## Current Problem
-
-The current distributed path:
-
-- calls `encode_prompt(current_text)` for every generated token
-- sends full-sequence activations through the route
-- calls layers with `use_cache=False`
-- creates a fresh UUID inside `_run_downstream_pipeline()`
-
-## Acceptance
-
-- Decode seam payload is one token / one hidden state after prefill.
-- Per-shard cache grows locally with generated tokens.
-- Regression test proves layer calls use cache after prefill.
-- Fallback error is explicit for models whose manual cache API is unsupported.
-

.scratch/distributed-gguf-runtime/issues/03-generation-telemetry-and-streaming.md

@@ -0,0 +1,21 @@
+# 03 — Generation Telemetry and streaming response contract
+
+Status: ready-for-agent
+
+## What to build
+
+Expose realtime Generation Telemetry for active Route Sessions and stream token deltas when the serving path can produce them. This slice should make long distributed requests observable before real large-model work begins.
+
+## Acceptance criteria
+
+- [ ] A client can observe route-session phase changes: queued, loading, prefill, decode, finalizing, completed, failed.
+- [ ] Telemetry includes prefill progress, generated token count, rolling tokens/sec, average tokens/sec, active route nodes, and failure reason.
+- [ ] Telemetry is available before the first output token.
+- [ ] A streaming response can include token deltas while telemetry remains available.
+- [ ] A non-streaming fallback still exposes telemetry until final answer or failure.
+- [ ] Route-node failure reports the last known phase and reason.
+- [ ] Tests cover telemetry updates, streaming token deltas, non-streaming fallback, and structured failure closeout.
+
+## Blocked by
+
+- 01 — Route Session lifecycle.

.scratch/distributed-gguf-runtime/issues/03-prefill-decode-wire-protocol.md

@@ -1,29 +0,0 @@
-# 03 — Prefill/decode wire protocol
-
-Status: ready-for-agent
-
-## Goal
-
-Define and implement the activation protocol needed by both PyTorch and future GGUF backends.
-
-## Scope
-
-- Add route/session lifecycle headers.
-- Separate `prefill` from `decode-step`.
-- Keep binary bfloat16 activation bodies.
-- Preserve relay compatibility.
-- Add route id and model artifact hash validation.
-
-## Draft Endpoints
-
-- `POST /sessions/{session_id}/prefill`
-- `POST /sessions/{session_id}/decode-step`
-- `DELETE /sessions/{session_id}`
-- `GET /sessions/{session_id}/status`
-
-## Acceptance
-
-- Old `/forward` remains temporarily or fails with clear version message.
-- Tests cover relay preservation of session headers.
-- Decode-step payload is independent of prompt length.
-

.scratch/distributed-gguf-runtime/issues/04-model-artifact-manifest-and-torrent.md

@@ -1,24 +0,0 @@
-# 04 — Model artifact manifest and torrent distribution
-
-Status: ready-for-agent
-
-## Goal
-
-Represent model artifacts independently from runtime routes so nodes can seed, verify, and advertise model files without relying on Hugging Face at runtime.
-
-## Scope
-
-- Define manifest schema.
-- Include file/chunk hashes.
-- Include tensor/layer map where available.
-- Include tokenizer and chat template hashes.
-- Include backend compatibility.
-- Add torrent/magnet URI fields and HTTP fallback URLs.
-- Extend node registration with artifact availability.
-
-## Acceptance
-
-- A model can be registered from a manifest without contacting Hugging Face.
-- Tracker can show coverage by artifact and layer range.
-- Node refuses to advertise corrupt artifacts.
-

.scratch/distributed-gguf-runtime/issues/04-pytorch-distributed-kv-reference.md

@@ -0,0 +1,23 @@
+# 04 — PyTorch distributed KV reference route
+
+Status: ready-for-agent
+
+## What to build
+
+Fix the existing distributed PyTorch route so it uses the Route Session and prefill/decode protocol to keep Hot KV State local to each Shard node. The visible behavior is that prefill processes the prompt once, and decode no longer recomputes or resends the full growing prompt for every token.
+
+## Acceptance criteria
+
+- [ ] Distributed PyTorch prefill stores per-session cache/state on each Shard node.
+- [ ] Distributed PyTorch decode-step reads and appends local per-shard cache/state.
+- [ ] Decode activation seam payload is one token/hidden-state step after prefill.
+- [ ] The old full-growing-prompt decode loop is not used for models that support the reference cache path.
+- [ ] Unsupported model/cache APIs fail with an explicit backend capability error.
+- [ ] Session close or TTL cleanup releases per-shard cache.
+- [ ] Regression tests prove decode does not call the full prompt encoder for every generated token.
+
+## Blocked by
+
+- 01 — Route Session lifecycle.
+- 02 — Prefill/decode binary HTTP protocol.
+- 03 — Generation Telemetry and streaming response contract.

.scratch/distributed-gguf-runtime/issues/05-llamacpp-layer-boundary-prototype.md

@@ -1,32 +0,0 @@
-# 05 — llama.cpp layer-boundary prototype
-
-Status: ready-for-human
-
-## Goal
-
-Prototype whether llama.cpp can execute only a selected layer range and accept/return hidden activations at model layer boundaries.
-
-## Scope
-
-- Start with a small model already supported by llama.cpp.
-- Run two local processes: head and tail.
-- Head owns embeddings + early layers.
-- Tail owns later layers + norm/lm_head.
-- Prefill once, then decode using local per-process KV.
-
-## Collaboration Point
-
-This is the best place to collaborate with Georgi/upstream llama.cpp. The desired upstream API shape:
-
-- load layer range or mmap full GGUF but execute layer range
-- run prefill chunk from inbound hidden states
-- run decode step from inbound hidden state
-- expose per-session KV/state handles
-- report cache memory budget
-
-## Acceptance
-
-- Localhost two-process decode does not recompute full prompt per token.
-- Seam payload after prefill is one hidden state per token.
-- No long-lived fork-only hooks unless upstream path is infeasible.
-

.scratch/distributed-gguf-runtime/issues/05-local-llamacpp-gguf-backend.md

@@ -0,0 +1,20 @@
+# 05 — Local llama.cpp/GGUF backend
+
+Status: ready-for-agent
+
+## What to build
+
+Add a local full-model GGUF backend so a node that can hold a GGUF model can serve it through the existing node API. This is the immediate CPU-performance path and the baseline for later distributed llama.cpp work.
+
+## Acceptance criteria
+
+- [ ] A node can start with backend `llama.cpp` or `gguf` for a local full-model GGUF artifact.
+- [ ] The node can answer an OpenAI-compatible chat completion through the existing API.
+- [ ] Startup and registration clearly report backend, quantization/artifact metadata, context cap, and local model path.
+- [ ] The PyTorch backend remains unchanged and selectable.
+- [ ] A smoke test or script validates backend wiring with a small GGUF model or a stubbed llama.cpp process.
+- [ ] A benchmark command can compare local PyTorch CPU and local GGUF CPU for the same small supported model when both are available.
+
+## Blocked by
+
+None - can start immediately.

.scratch/distributed-gguf-runtime/issues/06-model-artifact-manifest.md

@@ -0,0 +1,20 @@
+# 06 — Model Artifact manifest and Shard advertisement
+
+Status: ready-for-agent
+
+## What to build
+
+Introduce a Model Artifact manifest that separates storage distribution from route execution. A node should be able to verify local model files, determine which Shards it can serve, and advertise artifact/layer availability to the Tracker without contacting Hugging Face at request time.
+
+## Acceptance criteria
+
+- [ ] Manifest records model preset, upstream revision, license, backend support, quantization, context cap, tokenizer artifacts, file hashes, piece hashes, and tensor/layer mapping where available.
+- [ ] A node can verify local artifacts against the manifest and reject corrupt or incomplete artifacts.
+- [ ] A node can derive advertised Shard ranges from the manifest and local files.
+- [ ] Tracker registration can include artifact id, backend id, Shard range, and verification status.
+- [ ] Tracker coverage can distinguish model-layer coverage from artifact availability.
+- [ ] Tests cover valid manifest registration, corrupt artifact rejection, and missing layer/tensor metadata.
+
+## Blocked by
+
+- 01 — Route Session lifecycle.

.scratch/distributed-gguf-runtime/issues/06-networked-distributed-gguf-route.md

@@ -1,24 +0,0 @@
-# 06 — Networked distributed GGUF route
-
-Status: pending
-
-Depends on: 01, 03, 04, 05
-
-## Goal
-
-Run a GGUF-backed model over a real multi-node route using the tracker-selected route and per-shard local KV.
-
-## Scope
-
-- Extend node backend registry with GGUF layer ranges.
-- Add route selection for GGUF nodes.
-- Use the prefill/decode protocol.
-- Track route health and queue depth.
-- Bill by layer work and token work.
-
-## Acceptance
-
-- Two physical machines can serve one model route.
-- Node dropout during alpha fails request cleanly.
-- Tracker metrics show prefill TPS, decode TPS, seam latency, and cache memory.
-

.scratch/distributed-gguf-runtime/issues/07-glm52-ornith-support-audit.md

@@ -1,29 +0,0 @@
-# 07 — Large-model support audit
-
-Status: pending
-
-Depends on: 01, 05
-
-## Goal
-
-Determine which large target models can run through the distributed path and what upstream runtime work remains.
-
-The first serious large-model target is `deepseek-ai/DeepSeek-V4-Flash`. GLM-5.2 and Ornith remain follow-up targets.
-
-## Scope
-
-- Verify PyTorch/Transformers load semantics for DeepSeek-V4-Flash.
-- Verify vLLM/SGLang serving support for DeepSeek-V4-Flash.
-- Verify whether a GGUF/llama.cpp quantization path exists for DeepSeek-V4-Flash.
-- Estimate artifact size and 128K KV/cache memory by layer range for DeepSeek-V4-Flash.
-- Verify llama.cpp/GGUF support for `glm_moe_dsa`.
-- Verify cache accounting for GLM-5.2 DSA/MLA.
-- Verify Ornith/Qwen3.5-MoE hybrid attention support.
-- Identify smallest viable quantization for quality-first use.
-
-## Acceptance
-
-- Written compatibility matrix.
-- Clear "supported now / upstream needed / not viable" status per model.
-- DeepSeek-V4-Flash has a recommended first-runtime path: PyTorch, vLLM/SGLang, llama.cpp/GGUF, or blocked.
-- Runtime blockers converted into issues or upstream collaboration notes.

.scratch/distributed-gguf-runtime/issues/07-llamacpp-layer-boundary-prototype.md

@@ -0,0 +1,25 @@
+# 07 — llama.cpp layer-boundary prototype
+
+Status: ready-for-human
+
+## What to build
+
+Build a local prototype that proves whether llama.cpp/libllama can support the platform's distributed execution contract: execute a selected layer range, accept inbound hidden states, emit outbound hidden states, and own per-session cache for only the loaded/served range.
+
+This is the collaboration package for upstream llama.cpp. The target is an upstreamable API shape, not a permanent fork.
+
+## Acceptance criteria
+
+- [ ] A small llama.cpp-supported GGUF model can be split into a two-process localhost head/tail prototype.
+- [ ] The head process runs embeddings and early layers, then emits hidden states at an Activation Seam.
+- [ ] The tail process accepts hidden states, runs later layers plus output head, and produces logits/tokens comparable to single-process execution.
+- [ ] Prefill is performed once and decode-step seam payload is one hidden-state step per generated token.
+- [ ] Each process owns only its own per-session cache/state.
+- [ ] The prototype records the minimum upstream API needed for layer-range execution, hidden-state I/O, partial loading/introspection, and per-session KV ownership.
+- [ ] If upstream support is unavailable, the issue ends with a concrete recommendation: upstream proposal, narrow adapter fork, or keep GGUF distribution local-only for now.
+
+## Blocked by
+
+- 02 — Prefill/decode binary HTTP protocol.
+- 05 — Local llama.cpp/GGUF backend.
+- 06 — Model Artifact manifest and Shard advertisement.

.scratch/distributed-gguf-runtime/issues/08-generation-telemetry.md

@@ -1,28 +0,0 @@
-# Issue 08: Route-Session Generation Telemetry
-
-## Goal
-
-Expose realtime progress for long-running distributed inference requests. This is required whether or not token output is streamed.
-
-## Background
-
-Streaming token deltas is the preferred client experience when the backend and transport support it. Users still need realtime confidence that the route is alive and useful speed feedback during prefill, queueing, and any non-streaming fallback path.
-
-## Scope
-
-- Define a route-session telemetry schema.
-- Track phase: queued, loading, prefill, decode, finalizing, failed.
-- Track prefill token progress.
-- Track generated token count.
-- Track rolling and average tokens/sec.
-- Track active route nodes and failure reason.
-- Expose telemetry by SSE, WebSocket, or polling.
-- Ensure telemetry can coexist with streamed token deltas.
-
-## Acceptance Criteria
-
-- A client can display live route progress before the first output token is available.
-- During decode, the client sees rolling tokens/sec.
-- A streaming response can include token deltas and telemetry.
-- A non-streaming fallback still provides progress telemetry until final answer or failure.
-- Route failures include the last known phase and reason.

.scratch/distributed-gguf-runtime/issues/08-networked-distributed-gguf-route.md

@@ -0,0 +1,24 @@
+# 08 — Networked distributed GGUF route
+
+Status: pending
+
+## What to build
+
+Run a GGUF-backed model over a real multi-node Inference Route using the resolved Route Session, binary HTTP prefill/decode protocol, local Hot KV State, Generation Telemetry, and alpha fail-fast behavior.
+
+## Acceptance criteria
+
+- [ ] Two machines can form one GGUF-backed Inference Route over contiguous Shards.
+- [ ] Prefill builds local per-shard cache/state and decode-step uses one-step seam payloads.
+- [ ] The client receives streamed token deltas when supported by the GGUF path.
+- [ ] The client receives Generation Telemetry for phase, generated tokens, tokens/sec, route health, and failure reason.
+- [ ] Route-node loss fails the Route Session cleanly; no automatic repair is attempted in alpha.
+- [ ] Tracker metrics show prefill tokens/sec, decode tokens/sec, seam latency, queue depth, and cache memory by node.
+- [ ] Billing/audit records identify route membership and layer/token work for the completed or failed session.
+
+## Blocked by
+
+- 03 — Generation Telemetry and streaming response contract.
+- 04 — PyTorch distributed KV reference route.
+- 06 — Model Artifact manifest and Shard advertisement.
+- 07 — llama.cpp layer-boundary prototype.

.scratch/distributed-gguf-runtime/issues/09-deepseek-v4-flash-support-audit.md

@@ -0,0 +1,21 @@
+# 09 — DeepSeek-V4-Flash support audit
+
+Status: ready-for-agent
+
+## What to build
+
+Audit `deepseek-ai/DeepSeek-V4-Flash` as the first serious large-model target after the small GGUF protocol smoke test. The output is a compatibility matrix and a recommended runtime path, not full production support.
+
+## Acceptance criteria
+
+- [ ] Verify current PyTorch/Transformers load and generation semantics for DeepSeek-V4-Flash from primary model documentation.
+- [ ] Verify vLLM and SGLang support status from primary runtime documentation or release notes.
+- [ ] Verify whether a GGUF/llama.cpp quantization path exists or would need upstream work.
+- [ ] Estimate artifact size, active parameter behavior, and 128K cache memory by Shard range.
+- [ ] Identify required backend capability flags for the Tracker.
+- [ ] Produce a compatibility matrix: PyTorch, vLLM, SGLang, llama.cpp/GGUF.
+- [ ] End with one recommendation: first runtime path, blocked pending upstream, or defer.
+
+## Blocked by
+
+None - can start immediately.

.scratch/distributed-gguf-runtime/issues/09-streaming-response.md

@@ -1,24 +0,0 @@
-# Issue 09: Streaming Response Support
-
-## Goal
-
-Stream generated token deltas to clients when the backend and transport support it, while preserving Generation Telemetry as an independent progress channel.
-
-## Background
-
-The preferred client experience is streamed output plus live tokens/sec feedback. Some early route proofs or backend integrations may only support a final response, so telemetry remains mandatory even when token deltas are unavailable.
-
-## Scope
-
-- Define an OpenAI-compatible streaming response shape.
-- Decide whether token deltas and telemetry travel over the same SSE stream or separate channels.
-- Preserve non-streaming final-response mode for simple clients.
-- Ensure prefill progress is visible before first token delta.
-- Ensure route failures close streams with a structured error and last known telemetry.
-
-## Acceptance Criteria
-
-- A client can request streamed token deltas.
-- A client can receive Generation Telemetry before and during streamed decode.
-- Non-streaming clients still receive telemetry through the route-session telemetry endpoint.
-- Stream failure includes session id, phase, and failure reason.

.scratch/distributed-gguf-runtime/issues/10-glm52-ornith-followup-audit.md

@@ -0,0 +1,20 @@
+# 10 — GLM-5.2 and Ornith follow-up support audit
+
+Status: pending
+
+## What to build
+
+Audit GLM-5.2 and Ornith after the smaller protocol smoke path and DeepSeek-V4-Flash audit. The output is a follow-up compatibility matrix focused on architecture/runtime blockers: DSA/MLA, hybrid attention, cache accounting, and GGUF/llama.cpp support.
+
+## Acceptance criteria
+
+- [ ] Verify GLM-5.2 PyTorch/Transformers serving requirements and cache semantics from primary model documentation.
+- [ ] Verify llama.cpp/GGUF support status for `glm_moe_dsa` or equivalent architecture support.
+- [ ] Verify Ornith/Qwen3.5-MoE and hybrid attention support status in the candidate runtimes.
+- [ ] Estimate artifact size and 128K cache memory by Shard range for each model.
+- [ ] Identify smallest quality-preserving quantization worth testing.
+- [ ] Convert each runtime blocker into a follow-up issue or upstream collaboration note.
+
+## Blocked by
+
+- 09 — DeepSeek-V4-Flash support audit.

.scratch/distributed-gguf-runtime/milestones.md

@@ -0,0 +1,32 @@
+# Distributed GGUF Runtime Milestones
+
+## Proposed Breakdown
+
+| Order | Issue | Title | Blocked by | User-visible proof |
+|---:|---|---|---|---|
+| 1 | [01](./issues/01-route-session-lifecycle.md) | Route Session lifecycle | None | Stable route/session status and cleanup |
+| 2 | [02](./issues/02-prefill-decode-binary-http.md) | Prefill/decode binary HTTP protocol | 01 | Stub route proves prefill chunks and one-step decode payloads |
+| 3 | [03](./issues/03-generation-telemetry-and-streaming.md) | Generation Telemetry and streaming response contract | 01 | Client sees route progress and streamed deltas when available |
+| 4 | [04](./issues/04-pytorch-distributed-kv-reference.md) | PyTorch distributed KV reference route | 01, 02, 03 | Distributed PyTorch decode stops full-prompt recompute |
+| 5 | [05](./issues/05-local-llamacpp-gguf-backend.md) | Local llama.cpp/GGUF backend | None | Local GGUF model serves through node API |
+| 6 | [06](./issues/06-model-artifact-manifest.md) | Model Artifact manifest and Shard advertisement | 01 | Node verifies artifacts and advertises serveable Shards |
+| 7 | [07](./issues/07-llamacpp-layer-boundary-prototype.md) | llama.cpp layer-boundary prototype | 02, 05, 06 | Local two-process GGUF route identifies upstream API |
+| 8 | [08](./issues/08-networked-distributed-gguf-route.md) | Networked distributed GGUF route | 03, 04, 06, 07 | Two machines serve one GGUF route with telemetry |
+| 9 | [09](./issues/09-deepseek-v4-flash-support-audit.md) | DeepSeek-V4-Flash support audit | None | Runtime recommendation for first serious large model |
+| 10 | [10](./issues/10-glm52-ornith-followup-audit.md) | GLM-5.2 and Ornith follow-up support audit | 09 | Follow-up compatibility matrix and upstream blockers |
+
+## First Three To Implement
+
+1. **01 — Route Session lifecycle**: makes every later cache, telemetry, and route decision concrete.
+2. **02 — Prefill/decode binary HTTP protocol**: proves the payload shape and route/session headers before model internals.
+3. **03 — Generation Telemetry and streaming response contract**: gives every later long-running route a visible user experience and failure surface.
+
+## Parallel Work
+
+- **05 — Local llama.cpp/GGUF backend** can run in parallel with 01–03 because it is a full-model local backend.
+- **09 — DeepSeek-V4-Flash support audit** can run in parallel because it is research/compatibility work.
+
+## Human-Gated Work
+
+- **07 — llama.cpp layer-boundary prototype** is the collaboration point with Georgi/upstream llama.cpp.
+- **08 — Networked distributed GGUF route** should wait until the PyTorch reference route proves the cache/session contract.