dash works !!! good data. billing seems to work

This commit is contained in:
Dobromir Popov
2026-07-02 23:07:41 +02:00
parent 1e0aa6ea8f
commit a938c19a82
14 changed files with 617 additions and 49 deletions

View File

@@ -1,6 +1,6 @@
# Coverage-first shard assignment and tracker-as-first-layer-node
# Coverage-first shard assignment and tracker-routed inference
The tracker assigns shard ranges to nodes using a coverage-first, speed-weighted bin-packing algorithm. Tracker nodes must host at least the first layer shard of every model they coordinate, making them the natural inference entry point. Any node serving layers[0..k] can become a tracker node for that model.
The tracker assigns shard ranges to worker nodes using a coverage-first, speed-weighted bin-packing algorithm. The tracker is a control-plane service and public inference API endpoint: it stores registry state, selects routes, enforces billing, and proxies OpenAI-compatible requests to the selected head worker. It does not download or load model weights.
## Problem
@@ -23,24 +23,29 @@ Example: 700B NF4 model (~350GB weights). Node A has 128GB, Node B and C each ha
- Node C gets layers[k_b..N] (the remainder)
- If Node B benchmarks 2× faster than Node C, the tracker shifts the B/C boundary so B carries more layers
### Tracker-as-first-layer-node
### Tracker-routed head worker
Any node that advertises a new model to the network becomes a **tracker node** for that model. Tracker nodes have one hard requirement: they must hold and serve `layers[0..k]` (the first-layer shard) for every model they coordinate.
A worker that serves `layers[0..k]` is the **head worker** for that model. The tracker forwards `/v1/chat/completions` to a live head worker and injects the remaining downstream route. The worker, not the tracker process:
The reason is functional: a tracker node is also the inference entry point. When a client request arrives, the tracker node:
When a client request arrives, the tracker:
1. Authenticates/bills the request
2. Selects a live head worker and full downstream route from the coverage map
3. Proxies the request to that head worker
4. Records usage and credits node shares after completion
The head worker:
1. Tokenizes the input (owns the tokenizer)
2. Runs `model.embed_tokens` + `model.layers[0..k]`
3. Selects the optimal onward route from the coverage map
4. Forwards activations to the next node in the route
5. Receives the final hidden state back and streams tokens to the client
3. Forwards activations to the next node in the route
4. Receives the final hidden state back and streams tokens to the client
This collapses the separate "gateway" role: the tracker node that starts an inference request IS the gateway for that request. A standalone HTTP proxy/load-balancer may sit in front to pick which tracker node handles the request, but it carries no model weights.
This keeps the public tracker lightweight: a standalone HTTP proxy/load-balancer may sit in front to pick which tracker handles the request, but neither proxy nor tracker carries model weights.
Multiple tracker nodes for the same model = multiple entry points = horizontal scale for both routing decisions and the first-layer compute.
Multiple head workers for the same model = multiple inference entry points = horizontal scale for first-layer compute. Multiple trackers scale routing and billing, not model execution.
### Last-layer node (tail)
The node assigned `layers[N-k..N]` also runs `model.norm` and `model.lm_head`. It returns decoded token IDs (not hidden states) to the tracker node, which assembles the response. The tail shard assignment is marked `is_tail: true` in the shard registry.
The node assigned `layers[N-k..N]` also runs `model.norm` and `model.lm_head`. It returns decoded token IDs (not hidden states) to the head worker, which assembles the response. The tail shard assignment is marked `is_tail: true` in the shard registry.
### Adaptive quantization
@@ -78,8 +83,8 @@ Nodes obey directives asynchronously; the tracker waits up to a configurable tim
## Consequences
- The standalone `meshnet-gateway` service from US-005 becomes a thin load-balancer that routes to tracker nodes; tracker nodes do the actual inference orchestration
- Tracker nodes must download more model data (tokenizer + first-layer shard) — this is the price of being an entry point
- The standalone `meshnet-gateway` service from US-005 becomes a thin compatibility proxy/load-balancer; the public tracker can also serve the OpenAI-compatible endpoint directly
- Tracker processes do not download or load model data. Only worker nodes load model shards.
- Benchmark data is self-reported by nodes at registration; the validator can detect fraudulent benchmarks (a node claiming 100 tokens/sec but delivering 2 gets slashed for under-performance)
- VRAM reservation for KV cache means nodes can host fewer layers than their raw VRAM suggests — this is intentional; running out of KV cache during inference causes OOM crashes
- New CONTEXT.md terms: **Tracker Node** (node serving first-layer shard + inference routing for a model), **Coverage Map** (tracker's per-model layer-range → node-count mapping), **Rebalance Directive** (tracker instruction to a node to load or drop a shard)
- New CONTEXT.md terms: **Head Worker** (worker node serving first-layer shard for a model), **Coverage Map** (tracker's per-model layer-range → node-count mapping), **Rebalance Directive** (tracker instruction to a node to load or drop a shard)