dash works !!! good data. billing seems to work
This commit is contained in:
@@ -1,6 +1,6 @@
|
||||
# Coverage-first shard assignment and tracker-as-first-layer-node
|
||||
# Coverage-first shard assignment and tracker-routed inference
|
||||
|
||||
The tracker assigns shard ranges to nodes using a coverage-first, speed-weighted bin-packing algorithm. Tracker nodes must host at least the first layer shard of every model they coordinate, making them the natural inference entry point. Any node serving layers[0..k] can become a tracker node for that model.
|
||||
The tracker assigns shard ranges to worker nodes using a coverage-first, speed-weighted bin-packing algorithm. The tracker is a control-plane service and public inference API endpoint: it stores registry state, selects routes, enforces billing, and proxies OpenAI-compatible requests to the selected head worker. It does not download or load model weights.
|
||||
|
||||
## Problem
|
||||
|
||||
@@ -23,24 +23,29 @@ Example: 700B NF4 model (~350GB weights). Node A has 128GB, Node B and C each ha
|
||||
- Node C gets layers[k_b..N] (the remainder)
|
||||
- If Node B benchmarks 2× faster than Node C, the tracker shifts the B/C boundary so B carries more layers
|
||||
|
||||
### Tracker-as-first-layer-node
|
||||
### Tracker-routed head worker
|
||||
|
||||
Any node that advertises a new model to the network becomes a **tracker node** for that model. Tracker nodes have one hard requirement: they must hold and serve `layers[0..k]` (the first-layer shard) for every model they coordinate.
|
||||
A worker that serves `layers[0..k]` is the **head worker** for that model. The tracker forwards `/v1/chat/completions` to a live head worker and injects the remaining downstream route. The worker, not the tracker process:
|
||||
|
||||
The reason is functional: a tracker node is also the inference entry point. When a client request arrives, the tracker node:
|
||||
When a client request arrives, the tracker:
|
||||
1. Authenticates/bills the request
|
||||
2. Selects a live head worker and full downstream route from the coverage map
|
||||
3. Proxies the request to that head worker
|
||||
4. Records usage and credits node shares after completion
|
||||
|
||||
The head worker:
|
||||
1. Tokenizes the input (owns the tokenizer)
|
||||
2. Runs `model.embed_tokens` + `model.layers[0..k]`
|
||||
3. Selects the optimal onward route from the coverage map
|
||||
4. Forwards activations to the next node in the route
|
||||
5. Receives the final hidden state back and streams tokens to the client
|
||||
3. Forwards activations to the next node in the route
|
||||
4. Receives the final hidden state back and streams tokens to the client
|
||||
|
||||
This collapses the separate "gateway" role: the tracker node that starts an inference request IS the gateway for that request. A standalone HTTP proxy/load-balancer may sit in front to pick which tracker node handles the request, but it carries no model weights.
|
||||
This keeps the public tracker lightweight: a standalone HTTP proxy/load-balancer may sit in front to pick which tracker handles the request, but neither proxy nor tracker carries model weights.
|
||||
|
||||
Multiple tracker nodes for the same model = multiple entry points = horizontal scale for both routing decisions and the first-layer compute.
|
||||
Multiple head workers for the same model = multiple inference entry points = horizontal scale for first-layer compute. Multiple trackers scale routing and billing, not model execution.
|
||||
|
||||
### Last-layer node (tail)
|
||||
|
||||
The node assigned `layers[N-k..N]` also runs `model.norm` and `model.lm_head`. It returns decoded token IDs (not hidden states) to the tracker node, which assembles the response. The tail shard assignment is marked `is_tail: true` in the shard registry.
|
||||
The node assigned `layers[N-k..N]` also runs `model.norm` and `model.lm_head`. It returns decoded token IDs (not hidden states) to the head worker, which assembles the response. The tail shard assignment is marked `is_tail: true` in the shard registry.
|
||||
|
||||
### Adaptive quantization
|
||||
|
||||
@@ -78,8 +83,8 @@ Nodes obey directives asynchronously; the tracker waits up to a configurable tim
|
||||
|
||||
## Consequences
|
||||
|
||||
- The standalone `meshnet-gateway` service from US-005 becomes a thin load-balancer that routes to tracker nodes; tracker nodes do the actual inference orchestration
|
||||
- Tracker nodes must download more model data (tokenizer + first-layer shard) — this is the price of being an entry point
|
||||
- The standalone `meshnet-gateway` service from US-005 becomes a thin compatibility proxy/load-balancer; the public tracker can also serve the OpenAI-compatible endpoint directly
|
||||
- Tracker processes do not download or load model data. Only worker nodes load model shards.
|
||||
- Benchmark data is self-reported by nodes at registration; the validator can detect fraudulent benchmarks (a node claiming 100 tokens/sec but delivering 2 gets slashed for under-performance)
|
||||
- VRAM reservation for KV cache means nodes can host fewer layers than their raw VRAM suggests — this is intentional; running out of KV cache during inference causes OOM crashes
|
||||
- New CONTEXT.md terms: **Tracker Node** (node serving first-layer shard + inference routing for a model), **Coverage Map** (tracker's per-model layer-range → node-count mapping), **Rebalance Directive** (tracker instruction to a node to load or drop a shard)
|
||||
- New CONTEXT.md terms: **Head Worker** (worker node serving first-layer shard for a model), **Coverage Map** (tracker's per-model layer-range → node-count mapping), **Rebalance Directive** (tracker instruction to a node to load or drop a shard)
|
||||
|
||||
Reference in New Issue
Block a user