neuron-tai/docs/adr/0013-rolling-stats-smart-routing.md

ADR-0013: Rolling RPM statistics, smart assignment scoring, and throughput routing

Status: Accepted

Context

The tracker made routing and assignment decisions blind to actual network traffic. Three related improvements were needed and designed together:

1. Model usage statistics — how many requests per model, so the tracker knows demand
2. Smart assignment — assign new nodes to where demand × unmet coverage is highest
3. Throughput routing — when multiple nodes can complete a route, pick the faster ones

Decisions

1. Rolling RPM counters

_RollingCounter is a circular-bucket structure where each slot covers a fixed time epoch. Recording a value for the current epoch increments that slot; an expired slot is silently reset on the next write. Three windows per model:

Window	Buckets	Bucket size	Total span
per_minute	60	60 s	1 hour
per_hour	24	3600 s	1 day
per_day	30	86400 s	~1 month

rpm() sums all non-stale buckets and divides by total window minutes.

Alternative: exponential moving average (simpler, single float). Rejected because EMA cannot be persisted and restored without loss, and cannot be accurately merged from peer slices (each tracker runs its own requests, so merging EMA values doesn't give the true combined rate).

2. Per-tracker stat slices + additive gossip

Each tracker keeps only its own request slice. Gossip exchanges these slices and each tracker stores a {tracker_url → {model → rpms}} map. get_combined_stats() sums all slices.

This is additive: if tracker A sees 10 RPM for model X and tracker B sees 5 RPM, combined is 15 RPM. Slices are keyed by tracker URL so a stale peer update simply overwrites its own key without corrupting other peers' data.

Alternative: one global aggregator. Rejected — single point of failure, contradicts the distributed model.

3. Assignment scoring formula

score(model) = (demand_rpm + 1.0) × (coverage_deficit + 0.01)

• demand_rpm = get_combined_stats()[model]["rpm_last_hour"]
• coverage_deficit = fraction of model layers with zero-node coverage ∈ [0.0, 1.0]
• +1.0 floor: zero-traffic models still compete by coverage
• +0.01 floor: fully-covered models can still attract nodes if they have high demand

The product ensures both dimensions matter: high demand but full coverage scores lower than high demand with partial coverage. Pure coverage deficits without traffic score lower than even modest traffic combined with any gap.

price_per_token: 0.0 is returned in the assignment response, reserved for future billing.

4. Throughput tiebreak in route selection

effective_throughput(node) = benchmark_tokens_per_sec / (queue_depth + 1)

_select_route uses this as a tiebreak only: when two candidates reach the same maximum shard_end, the one with higher effective throughput is preferred. Coverage maximization remains the primary objective.

benchmark_tokens_per_sec comes from the hardware profile at registration. queue_depth comes from the most recent heartbeat. A busy node (high queue) is deprioritized without being excluded.

5. SQLite persistence

Stats are saved to SQLite (configurable via --stats-db PATH) every 60 seconds and on clean shutdown. Schema: model_rpm_buckets(model, window, bucket_idx, bucket_epoch, count). The circular-bucket structure maps directly — each slot is one row.

Consequences

• Tracker startup is slightly slower when loading a large stats DB (sub-second for typical sizes)
• Peer gossip adds one round-trip per gossip interval per peer
• price_per_token is a stable wire field; future billing sets it to a real value
• effective_throughput depends on benchmark_tokens_per_sec being set correctly at registration; nodes that don't set it get the default 1.0 and are treated as slowest