diff --git a/docs/adr/0020-chat-streaming-live-progress-and-mixed-topology-routing.md b/docs/adr/0020-chat-streaming-live-progress-and-mixed-topology-routing.md new file mode 100644 index 0000000..62fe4da --- /dev/null +++ b/docs/adr/0020-chat-streaming-live-progress-and-mixed-topology-routing.md @@ -0,0 +1,127 @@ +# ADR-0020: Dashboard chat streaming, live request progress, and the mixed-topology routing flaw + +## Status: Accepted (chat/streaming/styles implemented); routing flaw documented, fix pending + +## Context + +Live alpha testing (2026-07-07) with `Qwen3.6-35B-A3B` split across two LAN nodes surfaced +three UX gaps and one routing correctness flaw: + +1. **No visibility while a request is processing.** The Call wall showed + "no in-flight requests" during a 52-second generation. Cause: the dashboard chat sent + `stream: false`, and the tracker only emits `proxy progress` console events (the Call + wall's live-status source, `_tracker_log_proxy_progress`, `server.py` ~2199) for + **streamed** requests. Non-streamed proxying produces only + `route selected → connected → complete`, and short requests complete inside the + dashboard's 4-second poll window. +2. **Chat did not stream.** The nodes support SSE token-by-token generation + (`generate_text_streaming`, hardened earlier for split shards), and the tracker proxy + passes `text/event-stream` through (`server.py` ~3256), but the chat panel blocked on + full JSON and showed nothing until completion. +3. **Chat panel styles drifted.** The "new chat layout" redesign left hardcoded one-off + colors (`#1f4788`, `#2563b8`, `#10151d`, `#1a1012`, `#5c2020`, `#ffb4b4`) mixed with + the CSS custom-property palette. + +## Decisions + +### 1. Chat streams by default (SSE) + +`dashboard.html` `sendChat()` now sends `stream: true` and consumes the SSE body with a +`ReadableStream` reader: + +- Assistant tokens render incrementally into the last bubble (direct DOM update, full + re-render only at boundaries), with a blinking `▍` cursor while streaming. +- Chat status shows live progress: `generating… N tokens · X tok/s`. +- The send button becomes a stop button (`■`) during generation, backed by an + `AbortController`; a stopped generation keeps the partial text. +- Non-SSE responses (JSON fallback, errors) are still handled; `data: {"error": ...}` + stream events surface as error bubbles. +- `streaming` flags are stripped when loading persisted sessions so an interrupted + generation never leaves a stuck cursor. + +### 2. Live in-flight visibility rides on streaming + +No tracker change was needed: because chat now streams, the tracker emits `proxy progress` +events (throttled to stdout, updated in place in the console ring via +`update_console_key`), and the existing Call wall state machine +(`buildCallWallStates`) renders processing rows with live tokens/TPS/queue. + +**Known limitation (accepted):** non-streamed API requests still show no progress between +`proxy connected` and `proxy complete` — there is nothing to report until the node +returns. Callers wanting live visibility should use `stream: true`. + +### 3. Chat style tokens + +All chat colors route through `:root` custom properties (`--hover-bg`, `--chat-user-bg` +`#1f6feb`, `--chat-user-border`, `--chat-error-bg/border/fg`). No hardcoded hex values +remain in chat rules, so future palette changes are single-line edits. + +## Documented flaw: mixed-topology routing (partial GPU head + full CPU node) + +### Observed (2026-07-07, tracker 192.168.0.179:8080) + +Two nodes registered for `qwen3.6-35b-a3b`: + +| node | hardware | shard | benchmark | +|---|---|---|---| +| `5gMLrmyB-ec3afe6f1a03` (192.168.0.20) | RTX 4060, CUDA | 0–21 (partial, fast) | 11,164 | +| `7j77FsPY-55249b0583e5` (192.168.0.179) | CPU | 0–39 (full, slow) | 425 | + +When the tracker selected the GPU node as head, it injected: + +``` +downstream=[{"endpoint": "http://192.168.0.179:7000", "start_layer": 0}] +``` + +`start_layer: 0` — not 22. The downstream full node re-ran **all 40 layers from layer 0 +on hidden states that had already passed through the head's layers 0–21**, producing +garbage logits. Evidence from the logs: + +- GPU-headed requests: `generation complete tokens=1` and billed `out=0`/`out=1`/`out=3` + — near-instant EOS from corrupt activations. +- The same prompt routed directly to the CPU full node: 209 tokens over 52 s (healthy). +- Observed TPS for GPU-headed requests was meaningless (2.5–19.0 "tok/s" on 0–3 token + outputs), and those samples now pollute the rolling per-`(node, model)` throughput + stats used for routing preference. +- Clients were **billed** for these broken 1-token responses. + +### Root cause + +The route planner treats the full-coverage node as a standalone complete route +(`route=7j77FsPY…[0-39]`) but still injects it as the head's downstream with the +downstream node's own `shard_start` (0) instead of `head.shard_end + 1` (22). A partial +head + full-model downstream is a topology the planner never had to handle before — +prior split tests used disjoint shards (0–11 + 12–23) where `shard_start` happened to +equal the correct continuation layer. + +### Required fix (not yet implemented) + +1. **Correct continuation layer:** when hop N ends at layer `e`, hop N+1 must execute + from `start_layer = e + 1` regardless of the downstream node's own `shard_start` + (the `X-Meshnet-Start-Layer` overlapping-shard mechanism from ADR-0012 exists for + exactly this; the planner must set it for full-model downstream nodes too). +2. **Route preference sanity:** with a healthy single-node full route available, prefer + it over a multi-hop route unless the pipeline is estimated faster; a fast head that + forces a slow full-model tail wins nothing (every token still crosses the CPU node). +3. **Stat hygiene:** exclude or flag throughput samples from responses with ≤ a few + output tokens, so broken routes don't skew routing preference. +4. **Billing guard (consider):** suspiciously short completions from multi-hop routes + during this window were billed; a minimum-viability check (or refund path) may be + warranted once audits land. + +### Verification for the fix + +Reproduce with a partial GPU head (0–21) + full CPU node (0–39): a chat request routed +through the GPU head must produce output equivalent to the direct CPU route, with +`downstream start_layer=22` visible in `proxy route selected`, and multi-token streamed +output on the Call wall. + +## Verification of this ADR's implemented changes + +- `pytest tests/test_dashboard.py` — 5 passed (stale "Chat / inference" panel assertion + updated to the tabbed layout). +- Embedded dashboard JS parses (`new Function(script)` under Node 22). +- Live check: open `/dashboard` → Chat, send a prompt to `qwen3.6-35b-a3b` — tokens + must appear incrementally with live tok/s in the status line, the Call wall must show + the request as `processing` with live TPS, and the send button must stop generation + mid-stream keeping partial text. diff --git a/packages/tracker/meshnet_tracker/dashboard.html b/packages/tracker/meshnet_tracker/dashboard.html index faa07fb..72e16a2 100644 --- a/packages/tracker/meshnet_tracker/dashboard.html +++ b/packages/tracker/meshnet_tracker/dashboard.html @@ -7,7 +7,10 @@