1.1 KiB
1.1 KiB
03 — Generation Telemetry and streaming response contract
Status: ready-for-agent
What to build
Expose realtime Generation Telemetry for active Route Sessions and stream token deltas when the serving path can produce them. This slice should make long distributed requests observable before real large-model work begins.
Acceptance criteria
- A client can observe route-session phase changes: queued, loading, prefill, decode, finalizing, completed, failed.
- Telemetry includes prefill progress, generated token count, rolling tokens/sec, average tokens/sec, active route nodes, and failure reason.
- Telemetry is available before the first output token.
- A streaming response can include token deltas while telemetry remains available.
- A non-streaming fallback still exposes telemetry until final answer or failure.
- Route-node failure reports the last known phase and reason.
- Tests cover telemetry updates, streaming token deltas, non-streaming fallback, and structured failure closeout.
Blocked by
- 01 — Route Session lifecycle.