feature-gguf-distributed

.scratch/distributed-gguf-runtime/issues/09-streaming-response.md

@@ -0,0 +1,24 @@
+# Issue 09: Streaming Response Support
+
+## Goal
+
+Stream generated token deltas to clients when the backend and transport support it, while preserving Generation Telemetry as an independent progress channel.
+
+## Background
+
+The preferred client experience is streamed output plus live tokens/sec feedback. Some early route proofs or backend integrations may only support a final response, so telemetry remains mandatory even when token deltas are unavailable.
+
+## Scope
+
+- Define an OpenAI-compatible streaming response shape.
+- Decide whether token deltas and telemetry travel over the same SSE stream or separate channels.
+- Preserve non-streaming final-response mode for simple clients.
+- Ensure prefill progress is visible before first token delta.
+- Ensure route failures close streams with a structured error and last known telemetry.
+
+## Acceptance Criteria
+
+- A client can request streamed token deltas.
+- A client can receive Generation Telemetry before and during streamed decode.
+- Non-streaming clients still receive telemetry through the route-session telemetry endpoint.
+- Stream failure includes session id, phase, and failure reason.