neuron-tai/docs/adr/0009-coverage-first-shard-assignment-and-tracker-as-node.md

Coverage-first shard assignment and tracker-as-first-layer-node

The tracker assigns shard ranges to nodes using a coverage-first, speed-weighted bin-packing algorithm. Tracker nodes must host at least the first layer shard of every model they coordinate, making them the natural inference entry point. Any node serving layers[0..k] can become a tracker node for that model.

Problem

A 700B model sharded across volunteer nodes of wildly different VRAM (8GB to 128GB+) and different quantization levels (NF4 / INT8 / bfloat16) must be fully covered at all times for inference to work at all. 199 copies of 99% of a model are worthless if the last 1% of layers has zero coverage. Replication only matters after full coverage is achieved.

Decision

Coverage-first bin-packing

The tracker maintains a coverage map: a sorted list of (start_layer, end_layer, node_count) for each model. A layer range with node_count=0 is a coverage gap — the model is unservable until the gap is filled. The tracker's shard assignment algorithm:

1. Identify uncovered layer ranges (gaps) first
2. Assign new or idle nodes to fill gaps, largest VRAM-fit first
3. Once fully covered, assign additional nodes to the most-replicated-needed ranges (latency hotspots) before adding redundancy elsewhere
4. Speed-weight: give faster nodes wider shard ranges — fill their VRAM completely, then assign the remainder to the next-fastest available node

Example: 700B NF4 model (~350GB weights). Node A has 128GB, Node B and C each have 80GB and are idle.

• Node A gets layers[0..k_a] filling ~120GB (reserve 8GB for KV cache)
• Node B gets layers[k_a..k_b] filling ~72GB
• Node C gets layers[k_b..N] (the remainder)
• If Node B benchmarks 2× faster than Node C, the tracker shifts the B/C boundary so B carries more layers

Tracker-as-first-layer-node

Any node that advertises a new model to the network becomes a tracker node for that model. Tracker nodes have one hard requirement: they must hold and serve layers[0..k] (the first-layer shard) for every model they coordinate.

The reason is functional: a tracker node is also the inference entry point. When a client request arrives, the tracker node:

1. Tokenizes the input (owns the tokenizer)
2. Runs model.embed_tokens + model.layers[0..k]
3. Selects the optimal onward route from the coverage map
4. Forwards activations to the next node in the route
5. Receives the final hidden state back and streams tokens to the client

This collapses the separate "gateway" role: the tracker node that starts an inference request IS the gateway for that request. A standalone HTTP proxy/load-balancer may sit in front to pick which tracker node handles the request, but it carries no model weights.

Multiple tracker nodes for the same model = multiple entry points = horizontal scale for both routing decisions and the first-layer compute.

Last-layer node (tail)

The node assigned layers[N-k..N] also runs model.norm and model.lm_head. It returns decoded token IDs (not hidden states) to the tracker node, which assembles the response. The tail shard assignment is marked is_tail: true in the shard registry.

Adaptive quantization

Nodes declare their capabilities at registration:

{
  "vram_bytes": 25769803776,
  "quantizations": ["nf4", "int8", "bfloat16"],
  "benchmark_tokens_per_sec": 12.4,
  "benchmark_model": "meta-llama/Llama-3-8B"
}

The tracker uses declared VRAM + quantization to compute max assignable layers:

max_layers = floor(available_vram / bytes_per_layer_at_declared_quant)

where bytes_per_layer_at_declared_quant is a per-model constant stored in the model preset metadata. KV cache headroom (configurable, default 20% of VRAM) is subtracted before the calculation.

A node may serve different models at different quantizations. The tracker tracks (model, quantization, shard_range) triples per node.

Continuous rebalancing

The tracker monitors coverage every 30 seconds and on every node join/leave event. When a coverage gap appears (node drops offline), the tracker issues rebalance directives to idle nodes or nodes with redundant coverage:

• LOAD_SHARD(model, start_layer, end_layer, quantization) — download and begin serving
• DROP_SHARD(model, start_layer, end_layer) — stop serving (safe only when coverage > 1 for that range)

Nodes obey directives asynchronously; the tracker waits up to a configurable timeout before marking the gap critical.

Considered Options

• Static shard assignment (assign once at registration): simple, breaks when nodes leave — rejected
• Replication-first (maximize redundancy before optimizing speed): wastes resources on popular shards, leaves rare shards uncovered — rejected (user: "we have no use of 199 copies of 99% of the model")
• Coverage-first, speed-weighted bin-packing with continuous rebalancing: chosen

Consequences

• The standalone meshnet-gateway service from US-005 becomes a thin load-balancer that routes to tracker nodes; tracker nodes do the actual inference orchestration
• Tracker nodes must download more model data (tokenizer + first-layer shard) — this is the price of being an entry point
• Benchmark data is self-reported by nodes at registration; the validator can detect fraudulent benchmarks (a node claiming 100 tokens/sec but delivering 2 gets slashed for under-performance)
• VRAM reservation for KV cache means nodes can host fewer layers than their raw VRAM suggests — this is intentional; running out of KV cache during inference causes OOM crashes
• New CONTEXT.md terms: Tracker Node (node serving first-layer shard + inference routing for a model), Coverage Map (tracker's per-model layer-range → node-count mapping), Rebalance Directive (tracker instruction to a node to load or drop a shard)