# ADR-0020: Dashboard chat streaming, live request progress, and the mixed-topology routing flaw

## Status: Accepted (chat/streaming/styles and mixed-topology routing fix implemented)

## Context

Live alpha testing (2026-07-07) with `Qwen3.6-35B-A3B` split across two LAN nodes surfaced
three UX gaps and one routing correctness flaw:

1. **No visibility while a request is processing.** The Call wall showed
   "no in-flight requests" during a 52-second generation. Cause: the dashboard chat sent
   `stream: false`, and the tracker only emits `proxy progress` console events (the Call
   wall's live-status source, `_tracker_log_proxy_progress`, `server.py` ~2199) for
   **streamed** requests. Non-streamed proxying produces only
   `route selected → connected → complete`, and short requests complete inside the
   dashboard's 4-second poll window.
2. **Chat did not stream.** The nodes support SSE token-by-token generation
   (`generate_text_streaming`, hardened earlier for split shards), and the tracker proxy
   passes `text/event-stream` through (`server.py` ~3256), but the chat panel blocked on
   full JSON and showed nothing until completion.
3. **Chat panel styles drifted.** The "new chat layout" redesign left hardcoded one-off
   colors (`#1f4788`, `#2563b8`, `#10151d`, `#1a1012`, `#5c2020`, `#ffb4b4`) mixed with
   the CSS custom-property palette.

## Decisions

### 1. Chat streams by default (SSE)

`dashboard.html` `sendChat()` now sends `stream: true` and consumes the SSE body with a
`ReadableStream` reader:

- Assistant tokens render incrementally into the last bubble (direct DOM update, full
  re-render only at boundaries), with a blinking `▍` cursor while streaming.
- Chat status shows live progress: `generating… N tokens · X tok/s`.
- The send button becomes a stop button (`■`) during generation, backed by an
  `AbortController`; a stopped generation keeps the partial text.
- Non-SSE responses (JSON fallback, errors) are still handled; `data: {"error": ...}`
  stream events surface as error bubbles.
- `streaming` flags are stripped when loading persisted sessions so an interrupted
  generation never leaves a stuck cursor.

### 2. Live in-flight visibility rides on streaming

No tracker change was needed: because chat now streams, the tracker emits `proxy progress`
events (throttled to stdout, updated in place in the console ring via
`update_console_key`), and the existing Call wall state machine
(`buildCallWallStates`) renders processing rows with live tokens/TPS/queue.

**Known limitation (accepted):** non-streamed API requests still show no progress between
`proxy connected` and `proxy complete` — there is nothing to report until the node
returns. Callers wanting live visibility should use `stream: true`.

### 3. Chat style tokens

All chat colors route through `:root` custom properties (`--hover-bg`, `--chat-user-bg`
`#1f6feb`, `--chat-user-border`, `--chat-error-bg/border/fg`). No hardcoded hex values
remain in chat rules, so future palette changes are single-line edits.

## Documented flaw: mixed-topology routing (partial GPU head + full CPU node)

### Observed (2026-07-07, tracker 192.168.0.179:8080)

Two nodes registered for `qwen3.6-35b-a3b`:

| node | hardware | shard | benchmark |
|---|---|---|---|
| `5gMLrmyB-ec3afe6f1a03` (192.168.0.20) | RTX 4060, CUDA | 0–21 (partial, fast) | 11,164 |
| `7j77FsPY-55249b0583e5` (192.168.0.179) | CPU | 0–39 (full, slow) | 425 |

When the tracker selected the GPU node as head, it injected:

```
downstream=[{"endpoint": "http://192.168.0.179:7000", "start_layer": 0}]
```

`start_layer: 0` — not 22. The downstream full node re-ran **all 40 layers from layer 0
on hidden states that had already passed through the head's layers 0–21**, producing
garbage logits. Evidence from the logs:

- GPU-headed requests: `generation complete tokens=1` and billed `out=0`/`out=1`/`out=3`
  — near-instant EOS from corrupt activations.
- The same prompt routed directly to the CPU full node: 209 tokens over 52 s (healthy).
- Observed TPS for GPU-headed requests was meaningless (2.5–19.0 "tok/s" on 0–3 token
  outputs), and those samples now pollute the rolling per-`(node, model)` throughput
  stats used for routing preference.
- Clients were **billed** for these broken 1-token responses.

### Root cause

The route planner treats the full-coverage node as a standalone complete route
(`route=7j77FsPY…[0-39]`) but still injects it as the head's downstream with the
downstream node's own `shard_start` (0) instead of `head.shard_end + 1` (22). A partial
head + full-model downstream is a topology the planner never had to handle before —
prior split tests used disjoint shards (0–11 + 12–23) where `shard_start` happened to
equal the correct continuation layer.

### Required fix (implemented 2026-07-07 — commits `518c259`, `e44abc9`, `1ecc599`; see ADR-0021)

1. **Correct continuation layer:** when hop N ends at layer `e`, hop N+1 must execute
   from `start_layer = e + 1` regardless of the downstream node's own `shard_start`
   (the `X-Meshnet-Start-Layer` overlapping-shard mechanism from ADR-0012 exists for
   exactly this; the planner must set it for full-model downstream nodes too).
2. **Route preference sanity:** with a healthy single-node full route available, prefer
   it over a multi-hop route unless the pipeline is estimated faster; a fast head that
   forces a slow full-model tail wins nothing (every token still crosses the CPU node).
3. **Stat hygiene:** exclude or flag throughput samples from responses with ≤ a few
   output tokens, so broken routes don't skew routing preference.
4. **Billing guard (consider):** suspiciously short completions from multi-hop routes
   during this window were billed; a minimum-viability check (or refund path) may be
   warranted once audits land.

### Verification for the fix

Reproduce with a partial GPU head (0–21) + full CPU node (0–39): a chat request routed
through the GPU head must produce output equivalent to the direct CPU route, with
`downstream start_layer=22` visible in `proxy route selected`, and multi-token streamed
output on the Call wall.

## Verification of this ADR's implemented changes

- `pytest tests/test_dashboard.py` — 5 passed (stale "Chat / inference" panel assertion
  updated to the tabbed layout).
- Embedded dashboard JS parses (`new Function(script)` under Node 22).
- Live check: open `/dashboard` → Chat, send a prompt to `qwen3.6-35b-a3b` — tokens
  must appear incrementally with live tok/s in the status line, the Call wall must show
  the request as `processing` with live TPS, and the send button must stop generation
  mid-stream keeping partial text.