91 lines
5.8 KiB
Markdown
91 lines
5.8 KiB
Markdown
# Coverage-first shard assignment and tracker-routed inference
|
||
|
||
The tracker assigns shard ranges to worker nodes using a coverage-first, speed-weighted bin-packing algorithm. The tracker is a control-plane service and public inference API endpoint: it stores registry state, selects routes, enforces billing, and proxies OpenAI-compatible requests to the selected head worker. It does not download or load model weights.
|
||
|
||
## Problem
|
||
|
||
A 700B model sharded across volunteer nodes of wildly different VRAM (8GB to 128GB+) and different quantization levels (NF4 / INT8 / bfloat16) must be fully covered at all times for inference to work at all. 199 copies of 99% of a model are worthless if the last 1% of layers has zero coverage. Replication only matters after full coverage is achieved.
|
||
|
||
## Decision
|
||
|
||
### Coverage-first bin-packing
|
||
|
||
The tracker maintains a **coverage map**: a sorted list of (start_layer, end_layer, node_count) for each model. A layer range with node_count=0 is a coverage gap — the model is unservable until the gap is filled. The tracker's shard assignment algorithm:
|
||
|
||
1. Identify uncovered layer ranges (gaps) first
|
||
2. Assign new or idle nodes to fill gaps, largest VRAM-fit first
|
||
3. Once fully covered, assign additional nodes to the most-replicated-needed ranges (latency hotspots) before adding redundancy elsewhere
|
||
4. Speed-weight: give faster nodes wider shard ranges — fill their VRAM completely, then assign the remainder to the next-fastest available node
|
||
|
||
Example: 700B NF4 model (~350GB weights). Node A has 128GB, Node B and C each have 80GB and are idle.
|
||
- Node A gets layers[0..k_a] filling ~120GB (reserve 8GB for KV cache)
|
||
- Node B gets layers[k_a..k_b] filling ~72GB
|
||
- Node C gets layers[k_b..N] (the remainder)
|
||
- If Node B benchmarks 2× faster than Node C, the tracker shifts the B/C boundary so B carries more layers
|
||
|
||
### Tracker-routed head worker
|
||
|
||
A worker that serves `layers[0..k]` is the **head worker** for that model. The tracker forwards `/v1/chat/completions` to a live head worker and injects the remaining downstream route. The worker, not the tracker process:
|
||
|
||
When a client request arrives, the tracker:
|
||
1. Authenticates/bills the request
|
||
2. Selects a live head worker and full downstream route from the coverage map
|
||
3. Proxies the request to that head worker
|
||
4. Records usage and credits node shares after completion
|
||
|
||
The head worker:
|
||
1. Tokenizes the input (owns the tokenizer)
|
||
2. Runs `model.embed_tokens` + `model.layers[0..k]`
|
||
3. Forwards activations to the next node in the route
|
||
4. Receives the final hidden state back and streams tokens to the client
|
||
|
||
This keeps the public tracker lightweight: a standalone HTTP proxy/load-balancer may sit in front to pick which tracker handles the request, but neither proxy nor tracker carries model weights.
|
||
|
||
Multiple head workers for the same model = multiple inference entry points = horizontal scale for first-layer compute. Multiple trackers scale routing and billing, not model execution.
|
||
|
||
### Last-layer node (tail)
|
||
|
||
The node assigned `layers[N-k..N]` also runs `model.norm` and `model.lm_head`. It returns decoded token IDs (not hidden states) to the head worker, which assembles the response. The tail shard assignment is marked `is_tail: true` in the shard registry.
|
||
|
||
### Adaptive quantization
|
||
|
||
Nodes declare their capabilities at registration:
|
||
```json
|
||
{
|
||
"vram_bytes": 25769803776,
|
||
"quantizations": ["nf4", "int8", "bfloat16"],
|
||
"benchmark_tokens_per_sec": 12.4,
|
||
"benchmark_model": "meta-llama/Llama-3-8B"
|
||
}
|
||
```
|
||
|
||
The tracker uses declared VRAM + quantization to compute max assignable layers:
|
||
```
|
||
max_layers = floor(available_vram / bytes_per_layer_at_declared_quant)
|
||
```
|
||
where `bytes_per_layer_at_declared_quant` is a per-model constant stored in the model preset metadata. KV cache headroom (configurable, default 20% of VRAM) is subtracted before the calculation.
|
||
|
||
A node may serve different models at different quantizations. The tracker tracks (model, quantization, shard_range) triples per node.
|
||
|
||
### Continuous rebalancing
|
||
|
||
The tracker monitors coverage every 30 seconds and on every node join/leave event. When a coverage gap appears (node drops offline), the tracker issues **rebalance directives** to idle nodes or nodes with redundant coverage:
|
||
- `LOAD_SHARD(model, start_layer, end_layer, quantization)` — download and begin serving
|
||
- `DROP_SHARD(model, start_layer, end_layer)` — stop serving (safe only when coverage > 1 for that range)
|
||
|
||
Nodes obey directives asynchronously; the tracker waits up to a configurable timeout before marking the gap critical.
|
||
|
||
## Considered Options
|
||
|
||
- **Static shard assignment** (assign once at registration): simple, breaks when nodes leave — rejected
|
||
- **Replication-first** (maximize redundancy before optimizing speed): wastes resources on popular shards, leaves rare shards uncovered — rejected (user: "we have no use of 199 copies of 99% of the model")
|
||
- **Coverage-first, speed-weighted bin-packing with continuous rebalancing**: chosen
|
||
|
||
## Consequences
|
||
|
||
- The standalone `meshnet-gateway` service from US-005 becomes a thin compatibility proxy/load-balancer; the public tracker can also serve the OpenAI-compatible endpoint directly
|
||
- Tracker processes do not download or load model data. Only worker nodes load model shards.
|
||
- Benchmark data is self-reported by nodes at registration; the validator can detect fraudulent benchmarks (a node claiming 100 tokens/sec but delivering 2 gets slashed for under-performance)
|
||
- VRAM reservation for KV cache means nodes can host fewer layers than their raw VRAM suggests — this is intentional; running out of KV cache during inference causes OOM crashes
|
||
- New CONTEXT.md terms: **Head Worker** (worker node serving first-layer shard for a model), **Coverage Map** (tracker's per-model layer-range → node-count mapping), **Rebalance Directive** (tracker instruction to a node to load or drop a shard)
|