neuron-tai/docs/adr/0009-coverage-first-shard-assignment-and-tracker-as-node.md

Coverage-first shard assignment and tracker-routed inference

The tracker assigns shard ranges to worker nodes using a coverage-first, speed-weighted bin-packing algorithm. The tracker is a control-plane service and public inference API endpoint: it stores registry state, selects routes, enforces billing, and proxies OpenAI-compatible requests to the selected head worker. It does not download or load model weights.

Problem

A 700B model sharded across volunteer nodes of wildly different VRAM (8GB to 128GB+) and different quantization levels (NF4 / INT8 / bfloat16) must be fully covered at all times for inference to work at all. 199 copies of 99% of a model are worthless if the last 1% of layers has zero coverage. Replication only matters after full coverage is achieved.

Decision

Coverage-first bin-packing

The tracker maintains a coverage map: a sorted list of (start_layer, end_layer, node_count) for each model. A layer range with node_count=0 is a coverage gap — the model is unservable until the gap is filled. The tracker's shard assignment algorithm:

1. Identify uncovered layer ranges (gaps) first
2. Assign new or idle nodes to fill gaps, largest VRAM-fit first
3. Once fully covered, assign additional nodes to the most-replicated-needed ranges (latency hotspots) before adding redundancy elsewhere
4. Speed-weight: give faster nodes wider shard ranges — fill their VRAM completely, then assign the remainder to the next-fastest available node

Example: 700B NF4 model (~350GB weights). Node A has 128GB, Node B and C each have 80GB and are idle.

• Node A gets layers[0..k_a] filling ~120GB (reserve 8GB for KV cache)
• Node B gets layers[k_a..k_b] filling ~72GB
• Node C gets layers[k_b..N] (the remainder)
• If Node B benchmarks 2× faster than Node C, the tracker shifts the B/C boundary so B carries more layers

Tracker-routed head worker

A worker that serves layers[0..k] is the head worker for that model. The tracker forwards /v1/chat/completions to a live head worker and injects the remaining downstream route. The worker, not the tracker process:

When a client request arrives, the tracker:

1. Authenticates/bills the request
2. Selects a live head worker and full downstream route from the coverage map
3. Proxies the request to that head worker
4. Records usage and credits node shares after completion

The head worker:

1. Tokenizes the input (owns the tokenizer)
2. Runs model.embed_tokens + model.layers[0..k]
3. Forwards activations to the next node in the route
4. Receives the final hidden state back and streams tokens to the client

This keeps the public tracker lightweight: a standalone HTTP proxy/load-balancer may sit in front to pick which tracker handles the request, but neither proxy nor tracker carries model weights.

Multiple head workers for the same model = multiple inference entry points = horizontal scale for first-layer compute. Multiple trackers scale routing and billing, not model execution.

Last-layer node (tail)

The node assigned layers[N-k..N] also runs model.norm and model.lm_head. It returns decoded token IDs (not hidden states) to the head worker, which assembles the response. The tail shard assignment is marked is_tail: true in the shard registry.

Adaptive quantization

Nodes declare their capabilities at registration:

{
  "vram_bytes": 25769803776,
  "quantizations": ["nf4", "int8", "bfloat16"],
  "benchmark_tokens_per_sec": 12.4,
  "benchmark_model": "meta-llama/Llama-3-8B"
}

The tracker uses declared VRAM + quantization to compute max assignable layers:

max_layers = floor(available_vram / bytes_per_layer_at_declared_quant)

where bytes_per_layer_at_declared_quant is a per-model constant stored in the model preset metadata. KV cache headroom (configurable, default 20% of VRAM) is subtracted before the calculation.

A node may serve different models at different quantizations. The tracker tracks (model, quantization, shard_range) triples per node.

Continuous rebalancing

The tracker monitors coverage every 30 seconds and on every node join/leave event. When a coverage gap appears (node drops offline), the tracker issues rebalance directives to idle nodes or nodes with redundant coverage:

• LOAD_SHARD(model, start_layer, end_layer, quantization) — download and begin serving
• DROP_SHARD(model, start_layer, end_layer) — stop serving (safe only when coverage > 1 for that range)

Nodes obey directives asynchronously; the tracker waits up to a configurable timeout before marking the gap critical.

Considered Options

• Static shard assignment (assign once at registration): simple, breaks when nodes leave — rejected
• Replication-first (maximize redundancy before optimizing speed): wastes resources on popular shards, leaves rare shards uncovered — rejected (user: "we have no use of 199 copies of 99% of the model")
• Coverage-first, speed-weighted bin-packing with continuous rebalancing: chosen

Consequences

• The standalone meshnet-gateway service from US-005 becomes a thin compatibility proxy/load-balancer; the public tracker can also serve the OpenAI-compatible endpoint directly
• Tracker processes do not download or load model data. Only worker nodes load model shards.
• Benchmark data is self-reported by nodes at registration; the validator can detect fraudulent benchmarks (a node claiming 100 tokens/sec but delivering 2 gets slashed for under-performance)
• VRAM reservation for KV cache means nodes can host fewer layers than their raw VRAM suggests — this is intentional; running out of KV cache during inference causes OOM crashes
• New CONTEXT.md terms: Head Worker (worker node serving first-layer shard for a model), Coverage Map (tracker's per-model layer-range → node-count mapping), Rebalance Directive (tracker instruction to a node to load or drop a shard)