1.2 KiB
1.2 KiB
Issue 08: Route-Session Generation Telemetry
Goal
Expose realtime progress for long-running distributed inference requests. This is required whether or not token output is streamed.
Background
Streaming token deltas is the preferred client experience when the backend and transport support it. Users still need realtime confidence that the route is alive and useful speed feedback during prefill, queueing, and any non-streaming fallback path.
Scope
- Define a route-session telemetry schema.
- Track phase: queued, loading, prefill, decode, finalizing, failed.
- Track prefill token progress.
- Track generated token count.
- Track rolling and average tokens/sec.
- Track active route nodes and failure reason.
- Expose telemetry by SSE, WebSocket, or polling.
- Ensure telemetry can coexist with streamed token deltas.
Acceptance Criteria
- A client can display live route progress before the first output token is available.
- During decode, the client sees rolling tokens/sec.
- A streaming response can include token deltas and telemetry.
- A non-streaming fallback still provides progress telemetry until final answer or failure.
- Route failures include the last known phase and reason.