ADR-0021: Dynamic statistical routing (bandit-style route selection)

Status: Accepted, implemented

Context

ADR-0020 documented the mixed-topology flaw: with a fast GPU node serving layers 0–21 and a slow CPU node serving 0–39 of Qwen3.6-35B-A3B, the tracker picked the GPU node as proxy head independently of route planning, injecting a downstream hop with the wrong start_layer (0 instead of 22) and corrupting generation.

Beyond the bug, the deeper issue is that the tracker cannot know a priori which route is faster. Is one CPU node running all 40 layers faster than a GPU running 0–21 plus a CPU hop for 22–39? Benchmarks don't answer that — network hops, MoE expert loading, and queue dynamics only show up in real end-to-end requests. The router must measure.

Decision

Route selection is a multi-armed bandit over enumerated candidate routes, implemented in packages/tracker/meshnet_tracker/routing_stats.py and wired into the chat proxy in server.py.

Arms: route signatures

A route's identity is model_key | node_id[shard] -> node_id[shard] -> …. Node ids embed wallet + shard, so a node re-registering with a different shard produces a new arm automatically. The proxy target is always the route's own head (route_nodes[0]), and each hop's start_layer is previous_hop.shard_end + 1 — this fixes ADR-0020's flaw structurally: head choice and route planning can no longer disagree.

Candidate enumeration (`_enumerate_routes`)

One candidate per distinct head (a node whose shard_start equals the model's first layer — it must tokenize/embed), greedily completed with longest-advancing hops. Each candidate carries a prior_tps: its bottleneck hop's queue-adjusted effective throughput × reputation. Capped at 8 candidates ranked by prior.

Statistics: decayed EWMA + topology epochs

Per (model, signature), RouteStatsStore keeps an EWMA of observed end-to-end tokens/sec with time-decayed sample mass (half-life default 600 s). Two staleness mechanisms handle the morphing network:

Continuous: sample mass decays; a route unproven for a while (mass < 0.5) drops out of the exploit pool and gets re-scouted.
Abrupt: any node join/leave/shard-change bumps the model's topology epoch. Stats from an older epoch keep their EWMA as a display prior but are demoted to the scout pool ("stale") until re-measured under the new topology.

Sample hygiene: completions below min_sample_tokens (default 8) are rejected — the 1-token garbage responses from the ADR-0020 bug would otherwise poison arms with meaningless tps values. Routes with no samples for 24 h are pruned.

Selection policy (`choose_route`)

Scout (probability explore_share, default 0.3): if any candidate is unproven / stale / decayed, route the request there — least-measured first, tiebreak on prior. These are the user's "discovery/scout routes". With no proven arms at all, selection is deterministic best-prior (matches the old benchmark-based behavior, keeps cold start sane and tests deterministic).
Exploit (otherwise): weighted random among proven arms with P(route) ∝ tps^alpha, alpha default 1.0 — a 1.5×-faster route gets 1.5× the traffic. alpha is a config knob: >1 shifts toward winner-takes-most as the network matures, without redesign. (Proportional split is not throughput-optimal in queueing terms, but it keeps every arm warm with fresh samples; tune alpha up when traffic justifies it.)

Pinned routes ("route": [...] in the request body) bypass the bandit but still record samples.

Configuration

CLI flag	env var	default
`--route-explore-share`	`MESHNET_ROUTE_EXPLORE_SHARE`	0.3
`--route-weight-alpha`	`MESHNET_ROUTE_WEIGHT_ALPHA`	1.0
`--route-stats-half-life`	`MESHNET_ROUTE_STATS_HALF_LIFE`	600
—	`MESHNET_ROUTE_MIN_SAMPLE_TOKENS`	8

High explore share now (development, few requests); drop toward 0.05–0.1 once real traffic provides passive coverage.

Visibility

GET /v1/routing (optionally ?model=): per model — topology epoch and the full candidate table: hops, learned tps, coefficient (tps ÷ best proven route's tps), expected traffic share, sample count, decayed weight, status (proven / unsampled / stale / decayed).
Dashboard → Overview → "Routing (learned)": renders that table live (4 s poll), with the active config in the header line.
Console/proxy route selected events now include the routing decision ({"mode": "scout"|"exploit"|"pinned"|"greedy-fallback", "signature": …}), so the Call wall history shows which arm served each request.

Storage considerations

Stats are in-memory per tracker for alpha: they are cheap to relearn (a few requests per route), and gossiping them would import ADR-0019's consistency questions for data that is intentionally ephemeral. If multi-tracker route learning is needed later, ship route samples over the existing stats gossip and merge EWMAs by decayed weight — the store's (value, mass, timestamp) representation merges cleanly.

Consequences

The GPU(0–21)+CPU(0–39) topology now works: both routes get measured, the coefficient is visible on the dashboard, and traffic shifts to whichever is actually faster.
Routing is no longer deterministic once samples exist. Tests needing determinism seed server.route_rng or rely on the cold-start deterministic path.
The billing-relevant fix: heads are always part of the planned route, so per-hop start_layer and work-unit spans are consistent.

Verification

tests/test_dynamic_routing.py (11 tests): EWMA/decay/epoch semantics, near-empty sample rejection, traffic split ≈ tps ratio at alpha=1 (0.6/0.4 over 4000 seeded draws), scout rate ≈ explore share, mixed-topology enumeration (both routes, hybrid prior = bottleneck), head-is-route-head regression with start_layer=22 on the hybrid route, and /v1/routing table shape. Live: start both nodes, run several chats, open the dashboard "Routing (learned)" panel and watch coefficients converge.

6.0 KiB Raw Blame History Unescape Escape