120 lines
6.0 KiB
Markdown
120 lines
6.0 KiB
Markdown
# ADR-0021: Dynamic statistical routing (bandit-style route selection)
|
||
|
||
## Status: Accepted, implemented
|
||
|
||
## Context
|
||
|
||
ADR-0020 documented the mixed-topology flaw: with a fast GPU node serving layers 0–21 and
|
||
a slow CPU node serving 0–39 of `Qwen3.6-35B-A3B`, the tracker picked the GPU node as
|
||
proxy head *independently* of route planning, injecting a downstream hop with the wrong
|
||
`start_layer` (0 instead of 22) and corrupting generation.
|
||
|
||
Beyond the bug, the deeper issue is that the tracker **cannot know a priori** which route
|
||
is faster. Is one CPU node running all 40 layers faster than a GPU running 0–21 plus a
|
||
CPU hop for 22–39? Benchmarks don't answer that — network hops, MoE expert loading, and
|
||
queue dynamics only show up in real end-to-end requests. The router must *measure*.
|
||
|
||
## Decision
|
||
|
||
Route selection is a **multi-armed bandit** over enumerated candidate routes, implemented
|
||
in `packages/tracker/meshnet_tracker/routing_stats.py` and wired into the chat proxy in
|
||
`server.py`.
|
||
|
||
### Arms: route signatures
|
||
|
||
A route's identity is `model_key | node_id[shard] -> node_id[shard] -> …`. Node ids embed
|
||
wallet + shard, so a node re-registering with a different shard produces a new arm
|
||
automatically. The proxy target is **always the route's own head** (`route_nodes[0]`),
|
||
and each hop's `start_layer` is `previous_hop.shard_end + 1` — this fixes ADR-0020's flaw
|
||
structurally: head choice and route planning can no longer disagree.
|
||
|
||
### Candidate enumeration (`_enumerate_routes`)
|
||
|
||
One candidate per distinct head (a node whose `shard_start` equals the model's first
|
||
layer — it must tokenize/embed), greedily completed with longest-advancing hops. Each
|
||
candidate carries a `prior_tps`: its bottleneck hop's queue-adjusted effective throughput
|
||
× reputation. Capped at 8 candidates ranked by prior.
|
||
|
||
### Statistics: decayed EWMA + topology epochs
|
||
|
||
Per (model, signature), `RouteStatsStore` keeps an EWMA of observed end-to-end tokens/sec
|
||
with **time-decayed sample mass** (half-life default 600 s). Two staleness mechanisms
|
||
handle the morphing network:
|
||
|
||
- **Continuous**: sample mass decays; a route unproven for a while (mass < 0.5) drops out
|
||
of the exploit pool and gets re-scouted.
|
||
- **Abrupt**: any node join/leave/shard-change bumps the model's *topology epoch*. Stats
|
||
from an older epoch keep their EWMA as a display prior but are demoted to the scout
|
||
pool ("stale") until re-measured under the new topology.
|
||
|
||
Sample hygiene: completions below `min_sample_tokens` (default 8) are rejected — the
|
||
1-token garbage responses from the ADR-0020 bug would otherwise poison arms with
|
||
meaningless tps values. Routes with no samples for 24 h are pruned.
|
||
|
||
### Selection policy (`choose_route`)
|
||
|
||
1. **Scout** (probability `explore_share`, default 0.3): if any candidate is unproven /
|
||
stale / decayed, route the request there — least-measured first, tiebreak on prior.
|
||
These are the user's "discovery/scout routes". With *no* proven arms at all, selection
|
||
is deterministic best-prior (matches the old benchmark-based behavior, keeps cold
|
||
start sane and tests deterministic).
|
||
2. **Exploit** (otherwise): weighted random among proven arms with
|
||
`P(route) ∝ tps^alpha`, `alpha` default 1.0 — a 1.5×-faster route gets 1.5× the
|
||
traffic. `alpha` is a config knob: >1 shifts toward winner-takes-most as the network
|
||
matures, without redesign. (Proportional split is not throughput-optimal in queueing
|
||
terms, but it keeps every arm warm with fresh samples; tune alpha up when traffic
|
||
justifies it.)
|
||
|
||
Pinned routes (`"route": [...]` in the request body) bypass the bandit but still record
|
||
samples.
|
||
|
||
### Configuration
|
||
|
||
| CLI flag | env var | default |
|
||
|---|---|---|
|
||
| `--route-explore-share` | `MESHNET_ROUTE_EXPLORE_SHARE` | 0.3 |
|
||
| `--route-weight-alpha` | `MESHNET_ROUTE_WEIGHT_ALPHA` | 1.0 |
|
||
| `--route-stats-half-life` | `MESHNET_ROUTE_STATS_HALF_LIFE` | 600 |
|
||
| — | `MESHNET_ROUTE_MIN_SAMPLE_TOKENS` | 8 |
|
||
|
||
High explore share now (development, few requests); drop toward 0.05–0.1 once real
|
||
traffic provides passive coverage.
|
||
|
||
### Visibility
|
||
|
||
- **`GET /v1/routing`** (optionally `?model=`): per model — topology epoch and the full
|
||
candidate table: hops, learned tps, **coefficient** (tps ÷ best proven route's tps),
|
||
**expected traffic share**, sample count, decayed weight, status
|
||
(proven / unsampled / stale / decayed).
|
||
- **Dashboard → Overview → "Routing (learned)"**: renders that table live (4 s poll),
|
||
with the active config in the header line.
|
||
- **Console/`proxy route selected`** events now include the routing decision
|
||
(`{"mode": "scout"|"exploit"|"pinned"|"greedy-fallback", "signature": …}`), so the Call
|
||
wall history shows which arm served each request.
|
||
|
||
## Storage considerations
|
||
|
||
Stats are **in-memory per tracker** for alpha: they are cheap to relearn (a few requests
|
||
per route), and gossiping them would import ADR-0019's consistency questions for data
|
||
that is intentionally ephemeral. If multi-tracker route learning is needed later, ship
|
||
route samples over the existing stats gossip and merge EWMAs by decayed weight — the
|
||
store's (value, mass, timestamp) representation merges cleanly.
|
||
|
||
## Consequences
|
||
|
||
- The GPU(0–21)+CPU(0–39) topology now works: both routes get measured, the coefficient
|
||
is visible on the dashboard, and traffic shifts to whichever is actually faster.
|
||
- Routing is no longer deterministic once samples exist. Tests needing determinism seed
|
||
`server.route_rng` or rely on the cold-start deterministic path.
|
||
- The billing-relevant fix: heads are always part of the planned route, so per-hop
|
||
`start_layer` and work-unit spans are consistent.
|
||
|
||
## Verification
|
||
|
||
`tests/test_dynamic_routing.py` (11 tests): EWMA/decay/epoch semantics, near-empty sample
|
||
rejection, traffic split ≈ tps ratio at alpha=1 (0.6/0.4 over 4000 seeded draws), scout
|
||
rate ≈ explore share, mixed-topology enumeration (both routes, hybrid prior = bottleneck),
|
||
head-is-route-head regression with `start_layer=22` on the hybrid route, and `/v1/routing`
|
||
table shape. Live: start both nodes, run several chats, open the dashboard "Routing
|
||
(learned)" panel and watch coefficients converge.
|