feature-gguf-distributed

.scratch/distributed-gguf-runtime/issues/01-local-llamacpp-gguf-backend.md

@@ -0,0 +1,29 @@
+# 01 — Local llama.cpp/GGUF backend
+
+Status: ready-for-agent
+
+## Goal
+
+Add a local full-model llama.cpp/GGUF backend to the node so a machine that can hold a GGUF model can serve it through the existing OpenAI-compatible node API.
+
+## Scope
+
+- Add backend selection for `llama.cpp` / GGUF.
+- Support launching or calling `llama-server` first; direct `libllama` bindings may come later.
+- Register model metadata and hardware profile with tracker.
+- Preserve current PyTorch path.
+- Add a local benchmark comparing PyTorch CPU vs llama.cpp/GGUF for the same supported small model.
+
+## Non-Goals
+
+- No distributed GGUF route yet.
+- No partial layer loading yet.
+- No torrent seeding yet.
+
+## Acceptance
+
+- A local GGUF model can answer `/v1/chat/completions`.
+- Startup output clearly says backend=`llama.cpp`.
+- Node registration includes backend and artifact metadata.
+- Test or smoke script verifies the backend wiring without requiring a huge model.
+

.scratch/distributed-gguf-runtime/issues/02-stable-session-and-distributed-kv-pytorch.md

@@ -0,0 +1,33 @@
+# 02 — Stable session and distributed KV in PyTorch path
+
+Status: ready-for-agent
+
+## Goal
+
+Fix the existing distributed PyTorch path so it does not recompute the full growing prompt for every output token.
+
+## Scope
+
+- Introduce stable `session_id` for one request/session.
+- Add per-node session cache keyed by `session_id`.
+- Split `/forward` semantics into prefill and decode-step.
+- Use model cache objects / `past_key_values` where supported.
+- Keep hot KV local to each shard node.
+- Add cleanup/TTL for abandoned sessions.
+
+## Current Problem
+
+The current distributed path:
+
+- calls `encode_prompt(current_text)` for every generated token
+- sends full-sequence activations through the route
+- calls layers with `use_cache=False`
+- creates a fresh UUID inside `_run_downstream_pipeline()`
+
+## Acceptance
+
+- Decode seam payload is one token / one hidden state after prefill.
+- Per-shard cache grows locally with generated tokens.
+- Regression test proves layer calls use cache after prefill.
+- Fallback error is explicit for models whose manual cache API is unsupported.
+

.scratch/distributed-gguf-runtime/issues/03-prefill-decode-wire-protocol.md

@@ -0,0 +1,29 @@
+# 03 — Prefill/decode wire protocol
+
+Status: ready-for-agent
+
+## Goal
+
+Define and implement the activation protocol needed by both PyTorch and future GGUF backends.
+
+## Scope
+
+- Add route/session lifecycle headers.
+- Separate `prefill` from `decode-step`.
+- Keep binary bfloat16 activation bodies.
+- Preserve relay compatibility.
+- Add route id and model artifact hash validation.
+
+## Draft Endpoints
+
+- `POST /sessions/{session_id}/prefill`
+- `POST /sessions/{session_id}/decode-step`
+- `DELETE /sessions/{session_id}`
+- `GET /sessions/{session_id}/status`
+
+## Acceptance
+
+- Old `/forward` remains temporarily or fails with clear version message.
+- Tests cover relay preservation of session headers.
+- Decode-step payload is independent of prompt length.
+

.scratch/distributed-gguf-runtime/issues/04-model-artifact-manifest-and-torrent.md

@@ -0,0 +1,24 @@
+# 04 — Model artifact manifest and torrent distribution
+
+Status: ready-for-agent
+
+## Goal
+
+Represent model artifacts independently from runtime routes so nodes can seed, verify, and advertise model files without relying on Hugging Face at runtime.
+
+## Scope
+
+- Define manifest schema.
+- Include file/chunk hashes.
+- Include tensor/layer map where available.
+- Include tokenizer and chat template hashes.
+- Include backend compatibility.
+- Add torrent/magnet URI fields and HTTP fallback URLs.
+- Extend node registration with artifact availability.
+
+## Acceptance
+
+- A model can be registered from a manifest without contacting Hugging Face.
+- Tracker can show coverage by artifact and layer range.
+- Node refuses to advertise corrupt artifacts.
+

.scratch/distributed-gguf-runtime/issues/05-llamacpp-layer-boundary-prototype.md

@@ -0,0 +1,32 @@
+# 05 — llama.cpp layer-boundary prototype
+
+Status: ready-for-human
+
+## Goal
+
+Prototype whether llama.cpp can execute only a selected layer range and accept/return hidden activations at model layer boundaries.
+
+## Scope
+
+- Start with a small model already supported by llama.cpp.
+- Run two local processes: head and tail.
+- Head owns embeddings + early layers.
+- Tail owns later layers + norm/lm_head.
+- Prefill once, then decode using local per-process KV.
+
+## Collaboration Point
+
+This is the best place to collaborate with Georgi/upstream llama.cpp. The desired upstream API shape:
+
+- load layer range or mmap full GGUF but execute layer range
+- run prefill chunk from inbound hidden states
+- run decode step from inbound hidden state
+- expose per-session KV/state handles
+- report cache memory budget
+
+## Acceptance
+
+- Localhost two-process decode does not recompute full prompt per token.
+- Seam payload after prefill is one hidden state per token.
+- No long-lived fork-only hooks unless upstream path is infeasible.
+

.scratch/distributed-gguf-runtime/issues/06-networked-distributed-gguf-route.md

@@ -0,0 +1,24 @@
+# 06 — Networked distributed GGUF route
+
+Status: pending
+
+Depends on: 01, 03, 04, 05
+
+## Goal
+
+Run a GGUF-backed model over a real multi-node route using the tracker-selected route and per-shard local KV.
+
+## Scope
+
+- Extend node backend registry with GGUF layer ranges.
+- Add route selection for GGUF nodes.
+- Use the prefill/decode protocol.
+- Track route health and queue depth.
+- Bill by layer work and token work.
+
+## Acceptance
+
+- Two physical machines can serve one model route.
+- Node dropout during alpha fails request cleanly.
+- Tracker metrics show prefill TPS, decode TPS, seam latency, and cache memory.
+

.scratch/distributed-gguf-runtime/issues/07-glm52-ornith-support-audit.md

@@ -0,0 +1,29 @@
+# 07 — Large-model support audit
+
+Status: pending
+
+Depends on: 01, 05
+
+## Goal
+
+Determine which large target models can run through the distributed path and what upstream runtime work remains.
+
+The first serious large-model target is `deepseek-ai/DeepSeek-V4-Flash`. GLM-5.2 and Ornith remain follow-up targets.
+
+## Scope
+
+- Verify PyTorch/Transformers load semantics for DeepSeek-V4-Flash.
+- Verify vLLM/SGLang serving support for DeepSeek-V4-Flash.
+- Verify whether a GGUF/llama.cpp quantization path exists for DeepSeek-V4-Flash.
+- Estimate artifact size and 128K KV/cache memory by layer range for DeepSeek-V4-Flash.
+- Verify llama.cpp/GGUF support for `glm_moe_dsa`.
+- Verify cache accounting for GLM-5.2 DSA/MLA.
+- Verify Ornith/Qwen3.5-MoE hybrid attention support.
+- Identify smallest viable quantization for quality-first use.
+
+## Acceptance
+
+- Written compatibility matrix.
+- Clear "supported now / upstream needed / not viable" status per model.
+- DeepSeek-V4-Flash has a recommended first-runtime path: PyTorch, vLLM/SGLang, llama.cpp/GGUF, or blocked.
+- Runtime blockers converted into issues or upstream collaboration notes.

.scratch/distributed-gguf-runtime/issues/08-generation-telemetry.md

@@ -0,0 +1,28 @@
+# Issue 08: Route-Session Generation Telemetry
+
+## Goal
+
+Expose realtime progress for long-running distributed inference requests. This is required whether or not token output is streamed.
+
+## Background
+
+Streaming token deltas is the preferred client experience when the backend and transport support it. Users still need realtime confidence that the route is alive and useful speed feedback during prefill, queueing, and any non-streaming fallback path.
+
+## Scope
+
+- Define a route-session telemetry schema.
+- Track phase: queued, loading, prefill, decode, finalizing, failed.
+- Track prefill token progress.
+- Track generated token count.
+- Track rolling and average tokens/sec.
+- Track active route nodes and failure reason.
+- Expose telemetry by SSE, WebSocket, or polling.
+- Ensure telemetry can coexist with streamed token deltas.
+
+## Acceptance Criteria
+
+- A client can display live route progress before the first output token is available.
+- During decode, the client sees rolling tokens/sec.
+- A streaming response can include token deltas and telemetry.
+- A non-streaming fallback still provides progress telemetry until final answer or failure.
+- Route failures include the last known phase and reason.

.scratch/distributed-gguf-runtime/issues/09-streaming-response.md

@@ -0,0 +1,24 @@
+# Issue 09: Streaming Response Support
+
+## Goal
+
+Stream generated token deltas to clients when the backend and transport support it, while preserving Generation Telemetry as an independent progress channel.
+
+## Background
+
+The preferred client experience is streamed output plus live tokens/sec feedback. Some early route proofs or backend integrations may only support a final response, so telemetry remains mandatory even when token deltas are unavailable.
+
+## Scope
+
+- Define an OpenAI-compatible streaming response shape.
+- Decide whether token deltas and telemetry travel over the same SSE stream or separate channels.
+- Preserve non-streaming final-response mode for simple clients.
+- Ensure prefill progress is visible before first token delta.
+- Ensure route failures close streams with a structured error and last known telemetry.
+
+## Acceptance Criteria
+
+- A client can request streamed token deltas.
+- A client can receive Generation Telemetry before and during streamed decode.
+- Non-streaming clients still receive telemetry through the route-session telemetry endpoint.
+- Stream failure includes session id, phase, and failure reason.