1.1 KiB
1.1 KiB
Issue 09: Streaming Response Support
Goal
Stream generated token deltas to clients when the backend and transport support it, while preserving Generation Telemetry as an independent progress channel.
Background
The preferred client experience is streamed output plus live tokens/sec feedback. Some early route proofs or backend integrations may only support a final response, so telemetry remains mandatory even when token deltas are unavailable.
Scope
- Define an OpenAI-compatible streaming response shape.
- Decide whether token deltas and telemetry travel over the same SSE stream or separate channels.
- Preserve non-streaming final-response mode for simple clients.
- Ensure prefill progress is visible before first token delta.
- Ensure route failures close streams with a structured error and last known telemetry.
Acceptance Criteria
- A client can request streamed token deltas.
- A client can receive Generation Telemetry before and during streamed decode.
- Non-streaming clients still receive telemetry through the route-session telemetry endpoint.
- Stream failure includes session id, phase, and failure reason.