neuron-tai/docs/adr/0020-chat-streaming-live-progress-and-mixed-topology-routing.md

ADR-0020: Dashboard chat streaming, live request progress, and the mixed-topology routing flaw

Status: Accepted (chat/streaming/styles implemented); routing flaw documented, fix pending

Context

Live alpha testing (2026-07-07) with Qwen3.6-35B-A3B split across two LAN nodes surfaced three UX gaps and one routing correctness flaw:

1. No visibility while a request is processing. The Call wall showed "no in-flight requests" during a 52-second generation. Cause: the dashboard chat sent stream: false, and the tracker only emits proxy progress console events (the Call wall's live-status source, _tracker_log_proxy_progress, server.py ~2199) for streamed requests. Non-streamed proxying produces only route selected → connected → complete, and short requests complete inside the dashboard's 4-second poll window.
2. Chat did not stream. The nodes support SSE token-by-token generation (generate_text_streaming, hardened earlier for split shards), and the tracker proxy passes text/event-stream through (server.py ~3256), but the chat panel blocked on full JSON and showed nothing until completion.
3. Chat panel styles drifted. The "new chat layout" redesign left hardcoded one-off colors (#1f4788, #2563b8, #10151d, #1a1012, #5c2020, #ffb4b4) mixed with the CSS custom-property palette.

Decisions

1. Chat streams by default (SSE)

dashboard.html sendChat() now sends stream: true and consumes the SSE body with a ReadableStream reader:

• Assistant tokens render incrementally into the last bubble (direct DOM update, full re-render only at boundaries), with a blinking ▍ cursor while streaming.
• Chat status shows live progress: generating… N tokens · X tok/s.
• The send button becomes a stop button (■) during generation, backed by an AbortController; a stopped generation keeps the partial text.
• Non-SSE responses (JSON fallback, errors) are still handled; data: {"error": ...} stream events surface as error bubbles.
• streaming flags are stripped when loading persisted sessions so an interrupted generation never leaves a stuck cursor.

2. Live in-flight visibility rides on streaming

No tracker change was needed: because chat now streams, the tracker emits proxy progress events (throttled to stdout, updated in place in the console ring via update_console_key), and the existing Call wall state machine (buildCallWallStates) renders processing rows with live tokens/TPS/queue.

Known limitation (accepted): non-streamed API requests still show no progress between proxy connected and proxy complete — there is nothing to report until the node returns. Callers wanting live visibility should use stream: true.

3. Chat style tokens

All chat colors route through :root custom properties (--hover-bg, --chat-user-bg #1f6feb, --chat-user-border, --chat-error-bg/border/fg). No hardcoded hex values remain in chat rules, so future palette changes are single-line edits.

Documented flaw: mixed-topology routing (partial GPU head + full CPU node)

Observed (2026-07-07, tracker 192.168.0.179:8080)

Two nodes registered for qwen3.6-35b-a3b:

node	hardware	shard	benchmark
5gMLrmyB-ec3afe6f1a03 (192.168.0.20)	RTX 4060, CUDA	0–21 (partial, fast)	11,164
7j77FsPY-55249b0583e5 (192.168.0.179)	CPU	0–39 (full, slow)	425

When the tracker selected the GPU node as head, it injected:

downstream=[{"endpoint": "http://192.168.0.179:7000", "start_layer": 0}]

start_layer: 0 — not 22. The downstream full node re-ran all 40 layers from layer 0 on hidden states that had already passed through the head's layers 0–21, producing garbage logits. Evidence from the logs:

• GPU-headed requests: generation complete tokens=1 and billed out=0/out=1/out=3 — near-instant EOS from corrupt activations.
• The same prompt routed directly to the CPU full node: 209 tokens over 52 s (healthy).
• Observed TPS for GPU-headed requests was meaningless (2.5–19.0 "tok/s" on 0–3 token outputs), and those samples now pollute the rolling per-(node, model) throughput stats used for routing preference.
• Clients were billed for these broken 1-token responses.

Root cause

The route planner treats the full-coverage node as a standalone complete route (route=7j77FsPY…[0-39]) but still injects it as the head's downstream with the downstream node's own shard_start (0) instead of head.shard_end + 1 (22). A partial head + full-model downstream is a topology the planner never had to handle before — prior split tests used disjoint shards (0–11 + 12–23) where shard_start happened to equal the correct continuation layer.

Required fix (not yet implemented)

1. Correct continuation layer: when hop N ends at layer e, hop N+1 must execute from start_layer = e + 1 regardless of the downstream node's own shard_start (the X-Meshnet-Start-Layer overlapping-shard mechanism from ADR-0012 exists for exactly this; the planner must set it for full-model downstream nodes too).
2. Route preference sanity: with a healthy single-node full route available, prefer it over a multi-hop route unless the pipeline is estimated faster; a fast head that forces a slow full-model tail wins nothing (every token still crosses the CPU node).
3. Stat hygiene: exclude or flag throughput samples from responses with ≤ a few output tokens, so broken routes don't skew routing preference.
4. Billing guard (consider): suspiciously short completions from multi-hop routes during this window were billed; a minimum-viability check (or refund path) may be warranted once audits land.

Verification for the fix

Reproduce with a partial GPU head (0–21) + full CPU node (0–39): a chat request routed through the GPU head must produce output equivalent to the direct CPU route, with downstream start_layer=22 visible in proxy route selected, and multi-token streamed output on the Call wall.

Verification of this ADR's implemented changes

• pytest tests/test_dashboard.py — 5 passed (stale "Chat / inference" panel assertion updated to the tabbed layout).
• Embedded dashboard JS parses (new Function(script) under Node 22).
• Live check: open /dashboard → Chat, send a prompt to qwen3.6-35b-a3b — tokens must appear incrementally with live tok/s in the status line, the Call wall must show the request as processing with live TPS, and the send button must stop generation mid-stream keeping partial text.