128 lines
6.6 KiB
Markdown
128 lines
6.6 KiB
Markdown
# ADR-0020: Dashboard chat streaming, live request progress, and the mixed-topology routing flaw
|
||
|
||
## Status: Accepted (chat/streaming/styles and mixed-topology routing fix implemented)
|
||
|
||
## Context
|
||
|
||
Live alpha testing (2026-07-07) with `Qwen3.6-35B-A3B` split across two LAN nodes surfaced
|
||
three UX gaps and one routing correctness flaw:
|
||
|
||
1. **No visibility while a request is processing.** The Call wall showed
|
||
"no in-flight requests" during a 52-second generation. Cause: the dashboard chat sent
|
||
`stream: false`, and the tracker only emits `proxy progress` console events (the Call
|
||
wall's live-status source, `_tracker_log_proxy_progress`, `server.py` ~2199) for
|
||
**streamed** requests. Non-streamed proxying produces only
|
||
`route selected → connected → complete`, and short requests complete inside the
|
||
dashboard's 4-second poll window.
|
||
2. **Chat did not stream.** The nodes support SSE token-by-token generation
|
||
(`generate_text_streaming`, hardened earlier for split shards), and the tracker proxy
|
||
passes `text/event-stream` through (`server.py` ~3256), but the chat panel blocked on
|
||
full JSON and showed nothing until completion.
|
||
3. **Chat panel styles drifted.** The "new chat layout" redesign left hardcoded one-off
|
||
colors (`#1f4788`, `#2563b8`, `#10151d`, `#1a1012`, `#5c2020`, `#ffb4b4`) mixed with
|
||
the CSS custom-property palette.
|
||
|
||
## Decisions
|
||
|
||
### 1. Chat streams by default (SSE)
|
||
|
||
`dashboard.html` `sendChat()` now sends `stream: true` and consumes the SSE body with a
|
||
`ReadableStream` reader:
|
||
|
||
- Assistant tokens render incrementally into the last bubble (direct DOM update, full
|
||
re-render only at boundaries), with a blinking `▍` cursor while streaming.
|
||
- Chat status shows live progress: `generating… N tokens · X tok/s`.
|
||
- The send button becomes a stop button (`■`) during generation, backed by an
|
||
`AbortController`; a stopped generation keeps the partial text.
|
||
- Non-SSE responses (JSON fallback, errors) are still handled; `data: {"error": ...}`
|
||
stream events surface as error bubbles.
|
||
- `streaming` flags are stripped when loading persisted sessions so an interrupted
|
||
generation never leaves a stuck cursor.
|
||
|
||
### 2. Live in-flight visibility rides on streaming
|
||
|
||
No tracker change was needed: because chat now streams, the tracker emits `proxy progress`
|
||
events (throttled to stdout, updated in place in the console ring via
|
||
`update_console_key`), and the existing Call wall state machine
|
||
(`buildCallWallStates`) renders processing rows with live tokens/TPS/queue.
|
||
|
||
**Known limitation (accepted):** non-streamed API requests still show no progress between
|
||
`proxy connected` and `proxy complete` — there is nothing to report until the node
|
||
returns. Callers wanting live visibility should use `stream: true`.
|
||
|
||
### 3. Chat style tokens
|
||
|
||
All chat colors route through `:root` custom properties (`--hover-bg`, `--chat-user-bg`
|
||
`#1f6feb`, `--chat-user-border`, `--chat-error-bg/border/fg`). No hardcoded hex values
|
||
remain in chat rules, so future palette changes are single-line edits.
|
||
|
||
## Documented flaw: mixed-topology routing (partial GPU head + full CPU node)
|
||
|
||
### Observed (2026-07-07, tracker 192.168.0.179:8080)
|
||
|
||
Two nodes registered for `qwen3.6-35b-a3b`:
|
||
|
||
| node | hardware | shard | benchmark |
|
||
|---|---|---|---|
|
||
| `5gMLrmyB-ec3afe6f1a03` (192.168.0.20) | RTX 4060, CUDA | 0–21 (partial, fast) | 11,164 |
|
||
| `7j77FsPY-55249b0583e5` (192.168.0.179) | CPU | 0–39 (full, slow) | 425 |
|
||
|
||
When the tracker selected the GPU node as head, it injected:
|
||
|
||
```
|
||
downstream=[{"endpoint": "http://192.168.0.179:7000", "start_layer": 0}]
|
||
```
|
||
|
||
`start_layer: 0` — not 22. The downstream full node re-ran **all 40 layers from layer 0
|
||
on hidden states that had already passed through the head's layers 0–21**, producing
|
||
garbage logits. Evidence from the logs:
|
||
|
||
- GPU-headed requests: `generation complete tokens=1` and billed `out=0`/`out=1`/`out=3`
|
||
— near-instant EOS from corrupt activations.
|
||
- The same prompt routed directly to the CPU full node: 209 tokens over 52 s (healthy).
|
||
- Observed TPS for GPU-headed requests was meaningless (2.5–19.0 "tok/s" on 0–3 token
|
||
outputs), and those samples now pollute the rolling per-`(node, model)` throughput
|
||
stats used for routing preference.
|
||
- Clients were **billed** for these broken 1-token responses.
|
||
|
||
### Root cause
|
||
|
||
The route planner treats the full-coverage node as a standalone complete route
|
||
(`route=7j77FsPY…[0-39]`) but still injects it as the head's downstream with the
|
||
downstream node's own `shard_start` (0) instead of `head.shard_end + 1` (22). A partial
|
||
head + full-model downstream is a topology the planner never had to handle before —
|
||
prior split tests used disjoint shards (0–11 + 12–23) where `shard_start` happened to
|
||
equal the correct continuation layer.
|
||
|
||
### Required fix (implemented 2026-07-07 — commits `518c259`, `e44abc9`, `1ecc599`; see ADR-0021)
|
||
|
||
1. **Correct continuation layer:** when hop N ends at layer `e`, hop N+1 must execute
|
||
from `start_layer = e + 1` regardless of the downstream node's own `shard_start`
|
||
(the `X-Meshnet-Start-Layer` overlapping-shard mechanism from ADR-0012 exists for
|
||
exactly this; the planner must set it for full-model downstream nodes too).
|
||
2. **Route preference sanity:** with a healthy single-node full route available, prefer
|
||
it over a multi-hop route unless the pipeline is estimated faster; a fast head that
|
||
forces a slow full-model tail wins nothing (every token still crosses the CPU node).
|
||
3. **Stat hygiene:** exclude or flag throughput samples from responses with ≤ a few
|
||
output tokens, so broken routes don't skew routing preference.
|
||
4. **Billing guard (consider):** suspiciously short completions from multi-hop routes
|
||
during this window were billed; a minimum-viability check (or refund path) may be
|
||
warranted once audits land.
|
||
|
||
### Verification for the fix
|
||
|
||
Reproduce with a partial GPU head (0–21) + full CPU node (0–39): a chat request routed
|
||
through the GPU head must produce output equivalent to the direct CPU route, with
|
||
`downstream start_layer=22` visible in `proxy route selected`, and multi-token streamed
|
||
output on the Call wall.
|
||
|
||
## Verification of this ADR's implemented changes
|
||
|
||
- `pytest tests/test_dashboard.py` — 5 passed (stale "Chat / inference" panel assertion
|
||
updated to the tabbed layout).
|
||
- Embedded dashboard JS parses (`new Function(script)` under Node 22).
|
||
- Live check: open `/dashboard` → Chat, send a prompt to `qwen3.6-35b-a3b` — tokens
|
||
must appear incrementally with live tok/s in the status line, the Call wall must show
|
||
the request as `processing` with live TPS, and the send button must stop generation
|
||
mid-stream keeping partial text.
|