# ADR-0021: Dynamic statistical routing (bandit-style route selection)

## Status: Accepted, implemented

## Context

ADR-0020 documented the mixed-topology flaw: with a fast GPU node serving layers 0–21 and
a slow CPU node serving 0–39 of `Qwen3.6-35B-A3B`, the tracker picked the GPU node as
proxy head *independently* of route planning, injecting a downstream hop with the wrong
`start_layer` (0 instead of 22) and corrupting generation.

Beyond the bug, the deeper issue is that the tracker **cannot know a priori** which route
is faster. Is one CPU node running all 40 layers faster than a GPU running 0–21 plus a
CPU hop for 22–39? Benchmarks don't answer that — network hops, MoE expert loading, and
queue dynamics only show up in real end-to-end requests. The router must *measure*.

## Decision

Route selection is a **multi-armed bandit** over enumerated candidate routes, implemented
in `packages/tracker/meshnet_tracker/routing_stats.py` and wired into the chat proxy in
`server.py`.

### Arms: route signatures

A route's identity is `model_key | node_id[shard] -> node_id[shard] -> …`. Node ids embed
wallet + shard, so a node re-registering with a different shard produces a new arm
automatically. The proxy target is **always the route's own head** (`route_nodes[0]`),
and each hop's `start_layer` is `previous_hop.shard_end + 1` — this fixes ADR-0020's flaw
structurally: head choice and route planning can no longer disagree.

### Candidate enumeration (`_enumerate_routes`)

One candidate per distinct head (a node whose `shard_start` equals the model's first
layer — it must tokenize/embed), greedily completed with longest-advancing hops. Each
candidate carries a `prior_tps`: its bottleneck hop's queue-adjusted effective throughput
× reputation. Capped at 8 candidates ranked by prior.

### Statistics: decayed EWMA + topology epochs

Per (model, signature), `RouteStatsStore` keeps an EWMA of observed end-to-end tokens/sec
with **time-decayed sample mass** (half-life default 600 s). Two staleness mechanisms
handle the morphing network:

- **Continuous**: sample mass decays; a route unproven for a while (mass < 0.5) drops out
  of the exploit pool and gets re-scouted.
- **Abrupt**: any node join/leave/shard-change bumps the model's *topology epoch*. Stats
  from an older epoch keep their EWMA as a display prior but are demoted to the scout
  pool ("stale") until re-measured under the new topology.

Sample hygiene: completions below `min_sample_tokens` (default 8) are rejected — the
1-token garbage responses from the ADR-0020 bug would otherwise poison arms with
meaningless tps values. Routes with no samples for 24 h are pruned.

### Selection policy (`choose_route`)

1. **Scout** (probability `explore_share`, default 0.3): if any candidate is unproven /
   stale / decayed, route the request there — least-measured first, tiebreak on prior.
   These are the user's "discovery/scout routes". With *no* proven arms at all, selection
   is deterministic best-prior (matches the old benchmark-based behavior, keeps cold
   start sane and tests deterministic).
2. **Exploit** (otherwise): weighted random among proven arms with
   `P(route) ∝ tps^alpha`, `alpha` default 1.0 — a 1.5×-faster route gets 1.5× the
   traffic. `alpha` is a config knob: >1 shifts toward winner-takes-most as the network
   matures, without redesign. (Proportional split is not throughput-optimal in queueing
   terms, but it keeps every arm warm with fresh samples; tune alpha up when traffic
   justifies it.)

Pinned routes (`"route": [...]` in the request body) bypass the bandit but still record
samples.

### Configuration

| CLI flag | env var | default |
|---|---|---|
| `--route-explore-share` | `MESHNET_ROUTE_EXPLORE_SHARE` | 0.3 |
| `--route-weight-alpha` | `MESHNET_ROUTE_WEIGHT_ALPHA` | 1.0 |
| `--route-stats-half-life` | `MESHNET_ROUTE_STATS_HALF_LIFE` | 600 |
| — | `MESHNET_ROUTE_MIN_SAMPLE_TOKENS` | 8 |

High explore share now (development, few requests); drop toward 0.05–0.1 once real
traffic provides passive coverage.

### Visibility

- **`GET /v1/routing`** (optionally `?model=`): per model — topology epoch and the full
  candidate table: hops, learned tps, **coefficient** (tps ÷ best proven route's tps),
  **expected traffic share**, sample count, decayed weight, status
  (proven / unsampled / stale / decayed).
- **Dashboard → Overview → "Routing (learned)"**: renders that table live (4 s poll),
  with the active config in the header line.
- **Console/`proxy route selected`** events now include the routing decision
  (`{"mode": "scout"|"exploit"|"pinned"|"greedy-fallback", "signature": …}`), so the Call
  wall history shows which arm served each request.

## Storage considerations

Stats are **in-memory per tracker** for alpha: they are cheap to relearn (a few requests
per route), and gossiping them would import ADR-0019's consistency questions for data
that is intentionally ephemeral. If multi-tracker route learning is needed later, ship
route samples over the existing stats gossip and merge EWMAs by decayed weight — the
store's (value, mass, timestamp) representation merges cleanly.

## Consequences

- The GPU(0–21)+CPU(0–39) topology now works: both routes get measured, the coefficient
  is visible on the dashboard, and traffic shifts to whichever is actually faster.
- Routing is no longer deterministic once samples exist. Tests needing determinism seed
  `server.route_rng` or rely on the cold-start deterministic path.
- The billing-relevant fix: heads are always part of the planned route, so per-hop
  `start_layer` and work-unit spans are consistent.

## Verification

`tests/test_dynamic_routing.py` (11 tests): EWMA/decay/epoch semantics, near-empty sample
rejection, traffic split ≈ tps ratio at alpha=1 (0.6/0.4 over 4000 seeded draws), scout
rate ≈ explore share, mixed-topology enumeration (both routes, hybrid prior = bottleneck),
head-is-route-head regression with `start_layer=22` on the hybrid route, and `/v1/routing`
table shape. Live: start both nodes, run several chats, open the dashboard "Routing
(learned)" panel and watch coefficients converge.