25 lines
1.1 KiB
Markdown
25 lines
1.1 KiB
Markdown
# Issue 09: Streaming Response Support
|
|
|
|
## Goal
|
|
|
|
Stream generated token deltas to clients when the backend and transport support it, while preserving Generation Telemetry as an independent progress channel.
|
|
|
|
## Background
|
|
|
|
The preferred client experience is streamed output plus live tokens/sec feedback. Some early route proofs or backend integrations may only support a final response, so telemetry remains mandatory even when token deltas are unavailable.
|
|
|
|
## Scope
|
|
|
|
- Define an OpenAI-compatible streaming response shape.
|
|
- Decide whether token deltas and telemetry travel over the same SSE stream or separate channels.
|
|
- Preserve non-streaming final-response mode for simple clients.
|
|
- Ensure prefill progress is visible before first token delta.
|
|
- Ensure route failures close streams with a structured error and last known telemetry.
|
|
|
|
## Acceptance Criteria
|
|
|
|
- A client can request streamed token deltas.
|
|
- A client can receive Generation Telemetry before and during streamed decode.
|
|
- Non-streaming clients still receive telemetry through the route-session telemetry endpoint.
|
|
- Stream failure includes session id, phase, and failure reason.
|