popov/neuron-tai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: add ADRs and user stories for real model inference stack (US-011–014)

Browse Source 
ADR-0008: binary activation wire format — raw bfloat16 over HTTP, zstd compression,
128-token chunked prefill; replaces base64 JSON (~33% overhead removed).

ADR-0009: coverage-first shard assignment and tracker-as-first-layer-node —
any node serving layers[0..k] becomes the inference entry point for that model;
bin-packing fills all coverage gaps before adding redundancy; tracker issues
LOAD_SHARD/DROP_SHARD rebalance directives; nodes declare VRAM + quantization.

US-011: binary wire format migration
US-012: real PyTorch layer execution (transformers + bitsandbytes, test on GPT-2)
US-013: coverage-first tracker bin-packing with VRAM-aware shard assignment
US-014: tracker-as-node (tracker node serves first layers + handles client requests)

CONTEXT.md: Tracker Node, Coverage Map, Rebalance Directive terms added.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Dobromir Popov
2026-06-29 14:43:54 +03:00
parent 3286f77e27
commit b02e07d308
8 changed files with 471 additions and 56 deletions
Download Patch File Download Diff File Expand all files Collapse all files

50
.scratch/distributed-inference-network/issues/11-binary-wire-format.md Normal file
View File

@@ -0,0 +1,50 @@
# US-011 — Binary activation wire format
Replace the base64-encoded JSON activation payload from US-002 with a binary HTTP body, zstd compression, and chunked prefill. This is a protocol migration that must be applied to all nodes and the gateway before US-012 (real model backend) can be built on a sane foundation.
## Context
The current wire format encodes activation tensors as base64 strings inside a JSON dict:
```json
{"shape": [1, 1, 64], "dtype": "float32", "data": "<base64>", "context": {"prompt": "..."}}
```
This works for stub nodes (tiny tensors) but is unsuitable for real models:
- base64 adds 33% size overhead
- A 2048-token prefill at hidden_dim=16384 generates 64MB of activations per boundary — 85MB base64
- JSON parsing overhead grows with payload size
The new format per ADR-0008:
```
POST /forward HTTP/1.1
Content-Type: application/octet-stream
X-Meshnet-Shape: 1,128,16384
X-Meshnet-Dtype: bfloat16
X-Meshnet-Session: <uuid>
X-Meshnet-Chunk-Index: 0
X-Meshnet-Chunk-Total: 16
X-Meshnet-Encoding: zstd
Content-Length: <n>
<raw zstd-compressed little-endian bfloat16 bytes>
```
Stub nodes still emit zeroed tensors — just in binary now. No real model required for this story.
## Acceptance Criteria
- `packages/node` `/forward` endpoint reads `X-Meshnet-Shape`, `X-Meshnet-Dtype`, `X-Meshnet-Session`, `X-Meshnet-Chunk-Index`, `X-Meshnet-Chunk-Total`, and `X-Meshnet-Encoding` headers; reads raw body bytes; decompresses if `zstd`; reconstructs tensor as numpy array
- Node forward handler returns a binary response with the same header set reflecting output shape
- The gateway sends binary chunked activations to the first node; reassembles binary responses from the last node
- Chunked prefill: the gateway splits input sequences longer than `MESHNET_CHUNK_TOKENS` (default 128) into N chunks and sends them sequentially through the pipeline; stub nodes pass through each chunk unchanged
- An integration test sends a 512-token stub activation (4 chunks of 128) through a two-node pipeline and asserts all 4 chunk responses are received with correct headers
- `zstd` Python package added as a dependency to `packages/node` and `packages/gateway`
- The old `_make_stub_activations` function in `server.py` is replaced with `_make_stub_binary_activation(shape, dtype) -> bytes`
- `python -m pytest` passes from repo root
- Commit only this story's changes
## Implementation Notes
- Use `numpy` to pack/unpack tensor bytes: `np.frombuffer(body, dtype=np.float16).reshape(shape)` (bfloat16 → numpy uses `np.dtype('bfloat16')` in numpy >= 1.20, or load as uint16 and reinterpret)
- zstd: `import zstandard as zstd; cctx = zstd.ZstdCompressor(level=1); compressed = cctx.compress(raw_bytes)`
- Chunk index + total allow the receiving node to know whether this is a mid-sequence chunk (no special handling needed for stub; real model in US-012 will need to manage KV cache across chunks)
- Wire version header `X-Meshnet-Wire: 2` can be added for debugging mismatches between old and new nodes

61
.scratch/distributed-inference-network/issues/12-real-pytorch-model-backend.md Normal file
View File

@@ -0,0 +1,61 @@
# US-012 — Real PyTorch model backend
Replace stub node inference with actual `transformers` layer execution. A node downloads a HuggingFace SafeTensors model, loads `model.layers[start:end]` into GPU/CPU memory at the declared quantization level, and runs real forward passes. The first-layer node additionally runs `embed_tokens`; the last-layer node additionally runs `model.norm` and `lm_head`.
## Context
Per ADR-0001 (PyTorch over llama.cpp) and the grilling session decisions:
- Model format: HuggingFace SafeTensors (GGUF rejected — no per-layer hidden state API)
- Layer extraction: `model.model.layers[start:end]` from a loaded `AutoModelForCausalLM`
- Quantization: bitsandbytes NF4 / INT8 / bfloat16, declared by node at registration
- Activation dtype at boundaries: always `bfloat16` regardless of weight quantization
- First-layer node owns: tokenizer, `model.embed_tokens`, `model.layers[0..k]`
- Last-layer node owns: `model.layers[N-k..N]`, `model.norm`, `model.lm_head`
Start with a small real model for testing: `openai-community/gpt2` (117M, no special license, SafeTensors available). The production target is 700B+ models — the layer extraction pattern is identical regardless of size.
## Acceptance Criteria
- `packages/node` can be started with `--model-id openai-community/gpt2 --shard-start 0 --shard-end 6` and loads the corresponding transformer blocks
- Node detects whether it is a head shard (`shard_start == 0`) and loads tokenizer + `model.embed_tokens`
- Node detects whether it is a tail shard (`shard_end == total_layers`) and loads `model.norm + model.lm_head`
- A `/forward` request with a text prompt (head node) or binary bfloat16 tensor (mid/tail node) runs a real forward pass and returns binary bfloat16 activations
- A two-node integration test with GPT-2 sharded across two nodes returns a coherent (non-random, deterministic) text completion
- Quantization level is read from `--quantization [bfloat16|int8|nf4]` flag; `bfloat16` is default; `int8` and `nf4` use `bitsandbytes.BitsAndBytesConfig`
- A node with insufficient VRAM for the assigned shard at the declared quantization prints a clear error and exits (no silent OOM)
- `transformers`, `bitsandbytes`, `safetensors`, and `accelerate` added as dependencies to `packages/node`
- `python -m pytest` passes (GPT-2 test downloads model once, cached; mark with `@pytest.mark.integration` and skip in CI if no GPU)
- Commit only this story's changes
## Implementation Notes
```python
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch
def load_shard(model_id, start, end, quantization):
quant_config = None
if quantization == "nf4":
quant_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16)
elif quantization == "int8":
quant_config = BitsAndBytesConfig(load_in_8bit=True)
# Load full model metadata without weights, then extract layers
model = AutoModelForCausalLM.from_pretrained(
model_id,
quantization_config=quant_config,
device_map="auto",
torch_dtype=torch.bfloat16,
)
layers = model.model.layers[start:end]
return model, layers
```
- `device_map="auto"` handles GPU/CPU offload automatically based on available VRAM
- For large models, use `low_cpu_mem_usage=True` to avoid doubling peak RAM during load
- The forward pass for a mid-shard node: `hidden_states = layer(hidden_states)[0]` (most HF models return a tuple, take index 0)
- Head node forward: `input_ids → embed_tokens → position_embed → layers[0..k] → bfloat16 binary out`
- Tail node forward: `bfloat16 binary in → layers[N-k..N] → norm → lm_head → argmax/sample → token_id → back to head`
- Attention mask and position IDs must be forwarded alongside hidden states for models that need them (pack them in additional headers or a small JSON sidecar in the same HTTP request)
- Keep attention/position passing simple for v1: assume left-padded inputs, pass `attention_mask` as a separate binary blob with header `X-Meshnet-Attn-Mask: <base64-uint8>` (small enough for base64 — it's [B, S] not [B, S, D])

49
.scratch/distributed-inference-network/issues/13-coverage-first-shard-assignment.md Normal file
View File

@@ -0,0 +1,49 @@
# US-013 — Coverage-first tracker shard assignment
Upgrade the tracker's route selection to a coverage-first, speed-weighted bin-packing algorithm. The tracker maintains a live coverage map per model, issues rebalance directives when gaps appear, and assigns shard ranges to nodes based on declared VRAM, quantization, and benchmark throughput.
## Context
Per ADR-0009:
- **Coverage gap** = any layer range in a model with zero serving nodes → model is unroutable
- **Coverage-first**: fill all gaps before adding redundancy to covered ranges
- **Speed-weighted bin-packing**: fill fastest nodes to their VRAM limit first; cascade to next-fastest
- **Continuous rebalancing**: triggered every 30s and on every node join/leave event
- **Rebalance directive**: `LOAD_SHARD` or `DROP_SHARD` instruction sent from tracker to node
Node registration now includes capability data:
```json
{
"wallet": "...",
"endpoint": "http://10.0.0.5:7000",
"vram_bytes": 25769803776,
"ram_bytes": 137438953472,
"quantizations": ["nf4", "int8", "bfloat16"],
"benchmark_tokens_per_sec": 12.4,
"benchmark_model": "openai-community/gpt2"
}
```
Model preset metadata (stored in tracker config) includes `bytes_per_layer` at each quantization level and `total_layers`. The tracker can compute max assignable layers for any node.
## Acceptance Criteria
- Tracker accepts `vram_bytes`, `ram_bytes`, `quantizations`, `benchmark_tokens_per_sec` in the node registration payload
- `GET /v1/coverage/<model_preset>` returns the coverage map: list of `{start_layer, end_layer, node_count}` for each assigned range
- A model is only routable when all layer ranges have `node_count >= 1`
- When a new node registers, the tracker assigns it to the highest-priority uncovered range (or expands the most-loaded range if fully covered)
- When a node disconnects (heartbeat timeout), the tracker marks affected ranges as reduced-coverage and, if any reach 0, issues `LOAD_SHARD` directives to idle nodes within 30 seconds
- The shard assignment respects VRAM: `assigned_layers <= floor((vram_bytes * 0.8) / bytes_per_layer_at_quant)`
- Speed weighting: given two idle nodes both capable of covering a gap, the tracker assigns the wider sub-range to the faster node
- An integration test: start tracker with a model preset, register three nodes with different VRAM, assert the coverage map shows 100% coverage and the largest VRAM node received the widest shard range
- A second integration test: kill the node covering the middle layer range; assert tracker issues a `LOAD_SHARD` directive to an idle node and coverage recovers
- `python -m pytest` passes
- Commit only this story's changes
## Implementation Notes
- Coverage map data structure: `dict[str, list[tuple[int, int, int]]]` keyed by model preset
- Bin-packing is run in-process on the tracker (not a separate service); it's triggered by registration/heartbeat events and a 30s periodic timer
- Rebalance directives are delivered as responses to node heartbeat POSTs (node polls tracker every 10s anyway) — no new push channel needed
- `bytes_per_layer` is read from a `model_presets.json` config file; add GPT-2 (12 layers, ~30MB/layer bfloat16) as the test preset
- The current tracker `_select_route` function is extended, not replaced — backward compat with stub nodes that don't send VRAM data (default to 8GB / bfloat16 if omitted)

34
.scratch/distributed-inference-network/issues/14-tracker-as-node.md Normal file
View File

@@ -0,0 +1,34 @@
# US-014 — Tracker-as-first-layer-node (inference entry point)
Merge the inference orchestration role of the gateway into tracker nodes that serve the first-layer shard. A tracker node for a model receives client requests directly, runs the head shard, routes activations through the network, collects the tail output, and streams tokens back to the client. The standalone gateway becomes a thin load-balancer that routes to whichever tracker node is least loaded.
## Context
Per ADR-0009:
- Any node serving `layers[0..k]` for a model can act as its tracker node
- Tracker nodes own: tokenizer, `embed_tokens`, `layers[0..k]`
- Tracker nodes select the optimal onward route from the live coverage map (they already maintain it)
- Multiple tracker nodes for the same model = horizontal scale at both the routing and the first-layer compute level
- The standalone `meshnet-gateway` process from US-005 becomes a dumb round-robin load-balancer; it no longer orchestrates shard pipelines
This collapses a network hop and a process boundary: previously gateway → tracker → node; now client → tracker-node (which is both tracker and first-layer node).
## Acceptance Criteria
- A node started with `--tracker-mode` (or automatically when assigned `shard_start=0`) exposes both its existing `/forward` endpoint and a new `/v1/chat/completions` OpenAI-compatible endpoint
- The tracker-node's `/v1/chat/completions` handler: tokenizes input, embeds, runs its own layers, selects onward route from coverage map, forwards binary activations to next node, receives tail output, decodes tokens, streams SSE back to client
- Multiple tracker nodes for the same model can each independently handle requests (verified by sending requests to both and getting valid responses)
- The existing `meshnet-gateway` is updated to proxy requests to tracker nodes (round-robin or least-connections) rather than orchestrating the pipeline itself
- The gateway can discover tracker nodes from the tracker registry (`GET /v1/tracker-nodes/<model_preset>` returns the list of endpoints for tracker nodes for that model)
- An integration test: register two tracker nodes and two mid-shard nodes for GPT-2; send 10 requests to the gateway; assert both tracker nodes received roughly equal load and all responses are valid
- Existing US-005 OpenAI-compatible tests still pass (gateway still exposes `/v1/chat/completions`, just proxies to tracker nodes now)
- `python -m pytest` passes
- Commit only this story's changes
## Implementation Notes
- Tracker-mode detection: `shard_start == 0` OR `--tracker-mode` CLI flag
- The tracker-node process runs two HTTP servers on different ports: port 7000 for node-to-node (`/forward`), port 8080 for client-facing (`/v1/chat/completions`) — or a single server with both route prefixes
- Route selection inside the tracker node: same `GET /v1/route/<model_preset>` call to the central tracker registry, but the tracker node itself is already the first hop — the returned route starts from the *second* node
- Streaming: tail node returns token IDs one by one (or in batches) back to the tracker node via chunked HTTP; tracker node converts to SSE and streams to client
- The gateway's existing pipeline orchestration code (`_run_pipeline`) can be extracted into a shared `meshnet_common.pipeline` module reused by both tracker-node and the legacy gateway path

188
.scratch/distributed-inference-network/prd.json
View File

@@ -5,8 +5,8 @@
"userStories": [ "userStories": [
{ {
"id": "US-001", "id": "US-001",
"title": "01 — Monorepo scaffold + single-node smoke test", "title": "01 \u2014 Monorepo scaffold + single-node smoke test",
"description": "Stand up the monorepo package layout and prove that a client HTTP request can travel end-to-end through the stack — gateway → one node serving all layers → valid response — before any real model or distributed logic is wired in. The six top-level packages should exist with minimal stubs: - `packages/node` — node client CLI (`meshnet-node`) - `packages/gateway` — HTTP gateway + route orchestration - `packages/tracker` — node registry and route selection - `packages/sdk` — `meshnet` Python SDK - `packages/contracts` — Solana smart contract wrappers - `packages/p2p` — gossip and shard swarm The smoke test uses a tiny in-process stub model (not a real LLM) so the test is fast and has no external dependencies. The node runs in the same process as the test. The gateway is a real HTTP server. The client sends a real `POST /v1/chat/completions` request and receives a valid OpenAI-format response.", "description": "Stand up the monorepo package layout and prove that a client HTTP request can travel end-to-end through the stack \u2014 gateway \u2192 one node serving all layers \u2192 valid response \u2014 before any real model or distributed logic is wired in. The six top-level packages should exist with minimal stubs: - `packages/node` \u2014 node client CLI (`meshnet-node`) - `packages/gateway` \u2014 HTTP gateway + route orchestration - `packages/tracker` \u2014 node registry and route selection - `packages/sdk` \u2014 `meshnet` Python SDK - `packages/contracts` \u2014 Solana smart contract wrappers - `packages/p2p` \u2014 gossip and shard swarm The smoke test uses a tiny in-process stub model (not a real LLM) so the test is fast and has no external dependencies. The node runs in the same process as the test. The gateway is a real HTTP server. The client sends a real `POST /v1/chat/completions` request and receives a valid OpenAI-format response.",
"acceptanceCriteria": [ "acceptanceCriteria": [
"`pip install -e packages/node packages/gateway packages/tracker` works from repo root", "`pip install -e packages/node packages/gateway packages/tracker` works from repo root",
"`meshnet-node`, `meshnet-gateway`, and `meshnet-tracker` CLI entry points exist", "`meshnet-node`, `meshnet-gateway`, and `meshnet-tracker` CLI entry points exist",
@@ -27,14 +27,14 @@
}, },
{ {
"id": "US-002", "id": "US-002",
"title": "02 — Two-node shard pipeline", "title": "02 \u2014 Two-node shard pipeline",
"description": "Extend the single-node smoke test so that inference is routed through exactly two nodes, each serving half of a stub model's layers. Activation tensors must travel from the gateway to node A, then from node A to node B, and the final output returns to the gateway. This proves the distributed shard pipeline works end-to-end before real models or a real tracker are introduced. The two nodes run in the same process as the test (no real network required). The gateway hardcodes the two-node route for now — dynamic route selection comes in issue 03. The activation tensor protocol (shape, dtype, serialisation format) must be established here, as all later issues depend on it. Use the domain vocabulary from `CONTEXT.md`: the ordered list of nodes is an **inference route**; each node's layer range is a **shard**; the tensors passed between nodes are activations (not \"data\" or \"chunks\").", "description": "Extend the single-node smoke test so that inference is routed through exactly two nodes, each serving half of a stub model's layers. Activation tensors must travel from the gateway to node A, then from node A to node B, and the final output returns to the gateway. This proves the distributed shard pipeline works end-to-end before real models or a real tracker are introduced. The two nodes run in the same process as the test (no real network required). The gateway hardcodes the two-node route for now \u2014 dynamic route selection comes in issue 03. The activation tensor protocol (shape, dtype, serialisation format) must be established here, as all later issues depend on it. Use the domain vocabulary from `CONTEXT.md`: the ordered list of nodes is an **inference route**; each node's layer range is a **shard**; the tensors passed between nodes are activations (not \"data\" or \"chunks\").",
"acceptanceCriteria": [ "acceptanceCriteria": [
"The integration test spins up two stub nodes, each configured with a non-overlapping shard range", "The integration test spins up two stub nodes, each configured with a non-overlapping shard range",
"A `POST /v1/chat/completions` request results in activation tensors flowing node-A → node-B (verifiable via test assertions or logs)", "A `POST /v1/chat/completions` request results in activation tensors flowing node-A \u2192 node-B (verifiable via test assertions or logs)",
"The gateway assembles the final response correctly from node-B's output", "The gateway assembles the final response correctly from node-B's output",
"The test passes with no external services running", "The test passes with no external services running",
"The activation tensor serialisation format is documented in a short inline comment (shape, dtype, wire format) — this becomes the contract all future nodes implement", "The activation tensor serialisation format is documented in a short inline comment (shape, dtype, wire format) \u2014 this becomes the contract all future nodes implement",
"Follow TDD where code is added: add/adjust an observable behavior test before implementation when practical", "Follow TDD where code is added: add/adjust an observable behavior test before implementation when practical",
"Run the task-specific tests and ensure they pass", "Run the task-specific tests and ensure they pass",
"Run `python -m pytest` from repo root and ensure it passes, or document why no test suite exists yet", "Run `python -m pytest` from repo root and ensure it passes, or document why no test suite exists yet",
@@ -51,8 +51,8 @@
}, },
{ {
"id": "US-003", "id": "US-003",
"title": "03 — Tracker: node registration + route selection", "title": "03 \u2014 Tracker: node registration + route selection",
"description": "Replace the hardcoded two-node route from issue 02 with a real tracker. Nodes register themselves with the tracker on startup (endpoint, shard range, hardware profile, node score). The gateway queries the tracker for an inference route for a given model preset instead of using a static list. The tracker returns an ordered list of node endpoints whose shards collectively cover all layers. The tracker runs as a lightweight HTTP service. Node score is initially a simple placeholder (e.g. fixed value or random) — real throughput/latency scoring comes later. The tracker must correctly reject route requests when no registered nodes cover a required shard range and return an appropriate error. This issue establishes the tracker's registration and routing API contract, which issues 04 (node startup) and 05 (OpenAI gateway) both depend on.", "description": "Replace the hardcoded two-node route from issue 02 with a real tracker. Nodes register themselves with the tracker on startup (endpoint, shard range, hardware profile, node score). The gateway queries the tracker for an inference route for a given model preset instead of using a static list. The tracker returns an ordered list of node endpoints whose shards collectively cover all layers. The tracker runs as a lightweight HTTP service. Node score is initially a simple placeholder (e.g. fixed value or random) \u2014 real throughput/latency scoring comes later. The tracker must correctly reject route requests when no registered nodes cover a required shard range and return an appropriate error. This issue establishes the tracker's registration and routing API contract, which issues 04 (node startup) and 05 (OpenAI gateway) both depend on.",
"acceptanceCriteria": [ "acceptanceCriteria": [
"A node can register with the tracker via HTTP, providing its endpoint, shard range, and a hardware profile", "A node can register with the tracker via HTTP, providing its endpoint, shard range, and a hardware profile",
"The gateway queries the tracker with a model preset name and receives an ordered list of node endpoints forming a complete inference route", "The gateway queries the tracker with a model preset name and receives an ordered list of node endpoints forming a complete inference route",
@@ -76,8 +76,8 @@
}, },
{ {
"id": "US-004", "id": "US-004",
"title": "04 — Node client startup flow (`meshnet-node start`)", "title": "04 \u2014 Node client startup flow (`meshnet-node start`)",
"description": "Make `meshnet-node start` self-configuring from scratch on a machine with a CUDA-capable GPU (or CPU fallback). The full startup sequence must complete without any manual configuration: 1. Detect GPU model and available VRAM 2. Load an existing Solana wallet from disk, or generate and save a new one 3. Query the tracker for the optimal shard assignment given the hardware profile 4. Download the assigned shard from HuggingFace (`huggingface_hub`) 5. Register with the tracker (wallet address, endpoint, shard range, hardware profile) 6. Begin accepting inference connections The node prints a short status summary on startup: wallet address, assigned shard, model preset, and download progress. No interactive prompts — the entire flow is non-interactive. This is the primary viral growth vector. Every second of friction in this flow costs node operators. The startup sequence must work on Linux with CUDA, and degrade gracefully to CPU on machines without a GPU.", "description": "Make `meshnet-node start` self-configuring from scratch on a machine with a CUDA-capable GPU (or CPU fallback). The full startup sequence must complete without any manual configuration: 1. Detect GPU model and available VRAM 2. Load an existing Solana wallet from disk, or generate and save a new one 3. Query the tracker for the optimal shard assignment given the hardware profile 4. Download the assigned shard from HuggingFace (`huggingface_hub`) 5. Register with the tracker (wallet address, endpoint, shard range, hardware profile) 6. Begin accepting inference connections The node prints a short status summary on startup: wallet address, assigned shard, model preset, and download progress. No interactive prompts \u2014 the entire flow is non-interactive. This is the primary viral growth vector. Every second of friction in this flow costs node operators. The startup sequence must work on Linux with CUDA, and degrade gracefully to CPU on machines without a GPU.",
"acceptanceCriteria": [ "acceptanceCriteria": [
"`meshnet-node start --tracker http://localhost:8080` completes startup without prompts on a machine with a CUDA GPU", "`meshnet-node start --tracker http://localhost:8080` completes startup without prompts on a machine with a CUDA GPU",
"A Solana wallet keypair is generated and saved to `~/.config/meshnet/wallet.json` if none exists", "A Solana wallet keypair is generated and saved to `~/.config/meshnet/wallet.json` if none exists",
@@ -102,8 +102,8 @@
}, },
{ {
"id": "US-005", "id": "US-005",
"title": "05 — OpenAI-compatible gateway", "title": "05 \u2014 OpenAI-compatible gateway",
"description": "Expose a production-shape HTTP API from the gateway so that any client using the OpenAI Python SDK (or any OpenAI-compatible tool) works by changing only the `base_url`. The gateway translates OpenAI-format requests into inference route execution and streams responses back in OpenAI-format. Endpoints to implement: - `POST /v1/chat/completions` — streaming (`text/event-stream`) and non-streaming - `GET /v1/models` — returns the list of model presets currently routable on the network - `GET /v1/health` — liveness check The gateway selects an inference route from the tracker, executes the shard pipeline, and assembles the streamed response. If the tracker returns no route for the requested model, the gateway responds with a standard OpenAI-format error (`model_not_available`). Authentication (API key → SOL/USDC balance) is a stub in this issue — return 200 for any non-empty `Authorization` header. Real payment gating comes in issue 06.", "description": "Expose a production-shape HTTP API from the gateway so that any client using the OpenAI Python SDK (or any OpenAI-compatible tool) works by changing only the `base_url`. The gateway translates OpenAI-format requests into inference route execution and streams responses back in OpenAI-format. Endpoints to implement: - `POST /v1/chat/completions` \u2014 streaming (`text/event-stream`) and non-streaming - `GET /v1/models` \u2014 returns the list of model presets currently routable on the network - `GET /v1/health` \u2014 liveness check The gateway selects an inference route from the tracker, executes the shard pipeline, and assembles the streamed response. If the tracker returns no route for the requested model, the gateway responds with a standard OpenAI-format error (`model_not_available`). Authentication (API key \u2192 SOL/USDC balance) is a stub in this issue \u2014 return 200 for any non-empty `Authorization` header. Real payment gating comes in issue 06.",
"acceptanceCriteria": [ "acceptanceCriteria": [
"`openai.OpenAI(base_url=\"http://localhost:8080/v1\", api_key=\"test\").chat.completions.create(model=\"stub-model\", messages=[...])` returns a valid response", "`openai.OpenAI(base_url=\"http://localhost:8080/v1\", api_key=\"test\").chat.completions.create(model=\"stub-model\", messages=[...])` returns a valid response",
"Streaming works: `stream=True` returns `text/event-stream` chunks in OpenAI SSE format", "Streaming works: `stream=True` returns `text/event-stream` chunks in OpenAI SSE format",
@@ -127,8 +127,8 @@
}, },
{ {
"id": "US-006", "id": "US-006",
"title": "06 — Solana stake + settlement contracts", "title": "06 \u2014 Solana stake + settlement contracts",
"description": "Deploy and integrate the Solana smart contracts that make node staking, client payment, and token reward settlement trustless. All development and testing targets **Solana testnet** — never devnet or mainnet during development, to avoid real costs. Three contracts are needed: **Registry contract**: records stake balances, strike counts, and ban status per wallet. The gateway and tracker read from this contract to exclude banned wallets from route selection. **Payment contract**: clients pre-fund an API key account with SOL or USDC. The gateway records per-request compute attribution (which node served which layer range, for how many tokens). **Settlement contract**: called once per epoch. Debits client accounts proportional to compute consumed. Credits node operator wallets with our native token proportional to layers served. Distributes a validator reward share. The `packages/contracts` package provides Python wrappers for reading and writing to all three contracts. The gateway uses these wrappers to check stake before routing to a node and to record attribution after each request.", "description": "Deploy and integrate the Solana smart contracts that make node staking, client payment, and token reward settlement trustless. All development and testing targets **Solana testnet** \u2014 never devnet or mainnet during development, to avoid real costs.",
"acceptanceCriteria": [ "acceptanceCriteria": [
"All contracts deploy successfully to Solana testnet", "All contracts deploy successfully to Solana testnet",
"A node can submit a stake transaction and have its balance reflected in the registry contract", "A node can submit a stake transaction and have its balance reflected in the registry contract",
@@ -136,13 +136,10 @@
"After a completed inference session, compute attribution is recorded on-chain with correct node/layer attribution", "After a completed inference session, compute attribution is recorded on-chain with correct node/layer attribution",
"The epoch settlement transaction correctly distributes token rewards to node operators and deducts client balances", "The epoch settlement transaction correctly distributes token rewards to node operators and deducts client balances",
"The gateway refuses to route to a node whose stake balance is below the minimum threshold", "The gateway refuses to route to a node whose stake balance is below the minimum threshold",
"All contract interactions in tests run against a local Solana test validator (via `solana-test-validator`) — no live testnet required for CI", "All contract interactions in tests run against a local Solana test validator (via `solana-test-validator`) \u2014 no live testnet required for CI",
"A `.env.testnet` config points to Solana testnet RPC for manual end-to-end testing", "A `.env.testnet` config points to Solana testnet RPC for manual end-to-end testing",
"Follow TDD where code is added: add/adjust an observable behavior test before implementation when practical", "python -m pytest passes from repo root",
"Run the task-specific tests and ensure they pass", "Commit only this story's changes"
"Run `python -m pytest` from repo root and ensure it passes, or document why no test suite exists yet",
"Keep the implementation scoped to this story; do not implement downstream stories early",
"Commit only this story's code changes with a focused conventional commit message"
], ],
"priority": 6, "priority": 6,
"passes": true, "passes": true,
@@ -154,8 +151,8 @@
}, },
{ {
"id": "US-007", "id": "US-007",
"title": "07 — Fraud detection: validator + on-chain slash", "title": "07 \u2014 Fraud detection: validator + on-chain slash",
"description": "Implement the optimistic fraud detection loop. After each inference request completes, the validator process independently decides whether to re-run the request on a reference node (~5% sample rate). If the reference output diverges from the node's output beyond a configurable floating-point tolerance, the validator submits a slash transaction on-chain. The validator is a separate process (not the gateway or tracker). It subscribes to completed inference events, selects a random sample, re-runs those requests against a trusted reference node, compares outputs, and submits slash proofs when fraud is detected. On slash: the offending node's stake balance decreases by the configured slash amount; its strike count increments. The gateway reads strike state from the registry contract on each route selection and excludes nodes that exceed the strike threshold. Node operators receive a push notification (logged to stdout and, if configured, a webhook) when their node is slashed.", "description": "Implement the optimistic fraud detection loop. After each inference request completes, the validator process independently decides whether to re-run the request on a reference node (~5% sample rate).",
"acceptanceCriteria": [ "acceptanceCriteria": [
"The validator process samples ~5% of completed inference requests (configurable)", "The validator process samples ~5% of completed inference requests (configurable)",
"A node returning a deliberately wrong output is detected and slashed within one validation cycle", "A node returning a deliberately wrong output is detected and slashed within one validation cycle",
@@ -164,11 +161,8 @@
"A node that reaches the strike threshold is excluded from route selection on the next gateway request", "A node that reaches the strike threshold is excluded from route selection on the next gateway request",
"A slashed node logs a clear warning to stdout", "A slashed node logs a clear warning to stdout",
"An integration test: run a deliberately-bad node, send 20 requests, assert at least one slash transaction is submitted and the node is eventually excluded from routes", "An integration test: run a deliberately-bad node, send 20 requests, assert at least one slash transaction is submitted and the node is eventually excluded from routes",
"Follow TDD where code is added: add/adjust an observable behavior test before implementation when practical", "python -m pytest passes from repo root",
"Run the task-specific tests and ensure they pass", "Commit only this story's changes"
"Run `python -m pytest` from repo root and ensure it passes, or document why no test suite exists yet",
"Keep the implementation scoped to this story; do not implement downstream stories early",
"Commit only this story's code changes with a focused conventional commit message"
], ],
"priority": 7, "priority": 7,
"passes": true, "passes": true,
@@ -181,21 +175,18 @@
}, },
{ {
"id": "US-008", "id": "US-008",
"title": "08 — Node probationary period + ban enforcement", "title": "08 \u2014 Node probationary period + ban enforcement",
"description": "Implement the two anti-sybil mechanisms that make re-entering the network after a ban economically costly. **Probationary period**: a new wallet's first N completed inference jobs earn no token rewards. N is a configurable on-chain parameter (default: 50 jobs). The settlement contract reads the wallet's job count and skips reward distribution until the threshold is met. The node client displays the remaining probationary job count on startup. **Ban enforcement**: when a wallet's strike count reaches the configured threshold, the registry contract marks it as banned. The tracker and gateway must check ban status on every route selection and exclude banned wallets. Ban status is read exclusively from the Solana registry contract — the tracker cannot set or unset bans. Both mechanisms together mean a banned node operator must: create a new wallet, fund a new stake, and complete N free jobs before earning again — a meaningful economic barrier.", "description": "Implement the two anti-sybil mechanisms that make re-entering the network after a ban economically costly.",
"acceptanceCriteria": [ "acceptanceCriteria": [
"A node with a new wallet receives no token rewards for its first N jobs (verified via settlement contract state)", "A node with a new wallet receives no token rewards for its first N jobs (verified via settlement contract state)",
"The node client prints remaining probationary jobs on startup (e.g. \"Probationary period: 38 jobs remaining before earning\")", "The node client prints remaining probationary jobs on startup",
"After N jobs, the next epoch settlement correctly credits the node with token rewards", "After N jobs, the next epoch settlement correctly credits the node with token rewards",
"A wallet with strike count at the threshold is marked banned in the registry contract", "A wallet with strike count at the threshold is marked banned in the registry contract",
"The tracker excludes banned wallets from route selection", "The tracker excludes banned wallets from route selection",
"A banned wallet that attempts to register with the tracker is rejected", "A banned wallet that attempts to register with the tracker is rejected",
"An integration test covers: new wallet → N jobs → earning begins; and: strike threshold reached → banned → excluded from routes", "An integration test covers: new wallet \u2192 N jobs \u2192 earning begins; and: strike threshold reached \u2192 banned \u2192 excluded from routes",
"Follow TDD where code is added: add/adjust an observable behavior test before implementation when practical", "python -m pytest passes from repo root",
"Run the task-specific tests and ensure they pass", "Commit only this story's changes"
"Run `python -m pytest` from repo root and ensure it passes, or document why no test suite exists yet",
"Keep the implementation scoped to this story; do not implement downstream stories early",
"Commit only this story's code changes with a focused conventional commit message"
], ],
"priority": 8, "priority": 8,
"passes": true, "passes": true,
@@ -207,20 +198,17 @@
}, },
{ {
"id": "US-009", "id": "US-009",
"title": "09 — P2P shard swarm", "title": "09 \u2014 P2P shard swarm",
"description": "Once a node has downloaded a shard, it seeds that shard to other nodes that are assigned the same shard. New nodes with the same shard assignment should download from the swarm (peer nodes) rather than HuggingFace when peers are available, reducing HuggingFace load and speeding up propagation of popular shards. The shard swarm is keyed by `(model_preset, shard_index)`. Nodes announce their shard availability to the tracker on registration. When a new node requests a shard download, the tracker returns a list of peer endpoints that hold the shard alongside the HuggingFace fallback URL. The node tries peers first; falls back to HuggingFace if no peers respond within a timeout. Peers serve shard data over a simple chunked HTTP endpoint — no custom P2P protocol needed in v1. The node client displays download source (peer or HuggingFace) during shard download.", "description": "Once a node has downloaded a shard, it seeds that shard to other nodes that are assigned the same shard.",
"acceptanceCriteria": [ "acceptanceCriteria": [
"A node that has downloaded a shard is listed as a peer for that shard in the tracker", "A node that has downloaded a shard is listed as a peer for that shard in the tracker",
"A second node assigned the same shard downloads it from the first node (peer) rather than HuggingFace (verified by asserting no HuggingFace HTTP calls are made in the test)", "A second node assigned the same shard downloads it from the first node (peer) rather than HuggingFace",
"If no peers are available, the node falls back to HuggingFace without error", "If no peers are available, the node falls back to HuggingFace without error",
"The node client logs whether the shard was downloaded from a peer or HuggingFace", "The node client logs whether the shard was downloaded from a peer or HuggingFace",
"Shard chunks are verified against a checksum before being marked complete", "Shard chunks are verified against a checksum before being marked complete",
"An integration test: node A downloads shard from HuggingFace stub; node B with same assignment downloads from node A", "An integration test: node A downloads shard from HuggingFace stub; node B with same assignment downloads from node A",
"Follow TDD where code is added: add/adjust an observable behavior test before implementation when practical", "python -m pytest passes from repo root",
"Run the task-specific tests and ensure they pass", "Commit only this story's changes"
"Run `python -m pytest` from repo root and ensure it passes, or document why no test suite exists yet",
"Keep the implementation scoped to this story; do not implement downstream stories early",
"Commit only this story's code changes with a focused conventional commit message"
], ],
"priority": 9, "priority": 9,
"passes": true, "passes": true,
@@ -232,23 +220,20 @@
}, },
{ {
"id": "US-010", "id": "US-010",
"title": "10 — `meshnet` Python SDK", "title": "10 \u2014 `meshnet` Python SDK",
"description": "A Python SDK (`packages/sdk`) that wraps the OpenAI-compatible gateway and exposes network-specific controls that the raw OpenAI SDK cannot express. The SDK is the recommended interface for developers who want more than basic inference. The SDK surface: ```python from meshnet import Client client = Client(api_key=\"...\", base_url=\"https://api.meshnet.ai\") # Basic inference (identical to OpenAI SDK) response = client.chat.completions.create(model=\"llama-3-70b\", messages=[...]) # Network-specific features client.wallet.balance() # SOL/USDC balance remaining client.wallet.top_up(amount_sol=1.0) # returns a Solana payment address + QR client.models.available() # list of routable model presets with shard coverage % client.request(redundancy=2) # route through 2 independent chains, return majority client.estimate_cost(model=\"llama-3-70b\", tokens=1000) # cost in SOL before sending ``` The SDK must not require clients to hold or know about our native token. All wallet operations are in SOL or USDC.", "description": "A Python SDK (`packages/sdk`) that wraps the OpenAI-compatible gateway and exposes network-specific controls.",
"acceptanceCriteria": [ "acceptanceCriteria": [
"`pip install meshnet` installs the SDK", "`pip install meshnet` installs the SDK",
"`Client.chat.completions.create(...)` works identically to the OpenAI SDK (same parameters, same response shape)", "`Client.chat.completions.create(...)` works identically to the OpenAI SDK",
"`client.wallet.balance()` returns the current SOL/USDC balance for the API key", "`client.wallet.balance()` returns the current SOL/USDC balance for the API key",
"`client.wallet.top_up()` returns a valid Solana payment address", "`client.wallet.top_up()` returns a valid Solana payment address",
"`client.models.available()` returns model presets with shard coverage percentage", "`client.models.available()` returns model presets with shard coverage percentage",
"`client.estimate_cost(model, tokens)` returns a cost estimate in SOL before the request is sent", "`client.estimate_cost(model, tokens)` returns a cost estimate in SOL",
"`client.request(redundancy=2)` sends the same prompt to two independent inference routes and returns the majority response", "`client.request(redundancy=2)` sends to two independent inference routes and returns majority response",
"The SDK is typed (py.typed, full type stubs)", "The SDK is typed (py.typed, full type stubs)",
"An integration test covers each SDK method against a local gateway + tracker + stub nodes", "An integration test covers each SDK method against a local gateway + tracker + stub nodes",
"Follow TDD where code is added: add/adjust an observable behavior test before implementation when practical", "python -m pytest passes from repo root",
"Run the task-specific tests and ensure they pass", "Commit only this story's changes"
"Run `python -m pytest` from repo root and ensure it passes, or document why no test suite exists yet",
"Keep the implementation scoped to this story; do not implement downstream stories early",
"Commit only this story's code changes with a focused conventional commit message"
], ],
"priority": 10, "priority": 10,
"passes": true, "passes": true,
@@ -258,9 +243,100 @@
"US-006" "US-006"
], ],
"completionNotes": "Completed by fresh Ralph/Codex session ef4eea0e; verified by pytest, compileall, editable SDK install, and diff check." "completionNotes": "Completed by fresh Ralph/Codex session ef4eea0e; verified by pytest, compileall, editable SDK install, and diff check."
},
{
"id": "US-011",
"title": "11 \u2014 Binary activation wire format",
"description": "Replace the base64 JSON activation payload with raw binary HTTP bodies, zstd compression, and chunked prefill (128 tokens/chunk). All nodes and the gateway must be migrated. Stub nodes continue to emit zeroed tensors, just in binary. This is a protocol prerequisite for US-012 (real model backend).",
"acceptanceCriteria": [
"Node /forward endpoint reads shape/dtype/session/chunk from HTTP headers; body is raw binary (optionally zstd-compressed)",
"Node /forward response is raw binary with the same header set",
"Gateway splits prompts > MESHNET_CHUNK_TOKENS (default 128) into sequential chunks sent through the pipeline",
"Integration test: 512-token stub activation (4 chunks) through a two-node pipeline returns 4 valid binary chunk responses",
"zstd Python package added as a dependency to packages/node and packages/gateway",
"_make_stub_activations replaced with _make_stub_binary_activation(shape, dtype) -> bytes",
"python -m pytest passes from repo root",
"Commit only this story's changes"
],
"priority": 11,
"passes": false,
"notes": "Source issue: .scratch/distributed-inference-network/issues/11-binary-wire-format.md",
"dependsOn": [
"US-002"
]
},
{
"id": "US-012",
"title": "12 \u2014 Real PyTorch model backend",
"description": "Replace stub node inference with actual transformers layer execution. Node loads HuggingFace SafeTensors model shard (model.model.layers[start:end]), runs real forward passes in bfloat16, handles head (embed_tokens) and tail (lm_head) responsibilities, and supports bitsandbytes NF4/INT8/bfloat16 quantization via --quantization flag. Test model: openai-community/gpt2.",
"acceptanceCriteria": [
"meshnet-node start --model-id openai-community/gpt2 --shard-start 0 --shard-end 6 loads the shard without error",
"Head node (shard_start==0) loads tokenizer and embed_tokens",
"Tail node (shard_end==total_layers) loads model.norm and lm_head",
"Two-node GPT-2 integration test returns deterministic coherent text completion",
"--quantization [bfloat16|int8|nf4] flag supported; bfloat16 default",
"Node with insufficient VRAM prints clear error and exits",
"transformers, bitsandbytes, safetensors, accelerate added as node dependencies",
"Integration tests marked @pytest.mark.integration, skipped in CI without GPU",
"python -m pytest passes from repo root",
"Commit only this story's changes"
],
"priority": 12,
"passes": false,
"notes": "Source issue: .scratch/distributed-inference-network/issues/12-real-pytorch-model-backend.md",
"dependsOn": [
"US-011"
]
},
{
"id": "US-013",
"title": "13 \u2014 Coverage-first tracker shard assignment",
"description": "Upgrade tracker route selection to coverage-first, speed-weighted bin-packing. Tracker maintains a live coverage map per model, issues LOAD_SHARD/DROP_SHARD rebalance directives when coverage drops, and assigns shard ranges using declared VRAM, quantization, and benchmark throughput. A model is only routable when all layer ranges have node_count >= 1.",
"acceptanceCriteria": [
"Node registration accepts vram_bytes, ram_bytes, quantizations[], benchmark_tokens_per_sec",
"GET /v1/coverage/<model_preset> returns list of {start_layer, end_layer, node_count}",
"Model is unroutable when any layer range has node_count=0",
"New node gets assigned to highest-priority uncovered range first",
"Node disconnection triggers LOAD_SHARD directive to idle node within 30 seconds",
"Shard assignment respects VRAM: assigned_layers <= floor((vram_bytes * 0.8) / bytes_per_layer_at_quant)",
"Faster node receives wider shard range when both can cover the same gap",
"Integration test: three nodes with different VRAM show widest range on largest-VRAM node with 100% coverage",
"Integration test: kill middle-range node \u2192 tracker issues LOAD_SHARD \u2192 coverage recovers",
"python -m pytest passes from repo root",
"Commit only this story's changes"
],
"priority": 13,
"passes": false,
"notes": "Source issue: .scratch/distributed-inference-network/issues/13-coverage-first-shard-assignment.md",
"dependsOn": [
"US-003",
"US-012"
]
},
{
"id": "US-014",
"title": "14 \u2014 Tracker-as-first-layer-node (inference entry point)",
"description": "Merge inference orchestration into tracker nodes that serve the first-layer shard. A tracker node exposes /v1/chat/completions directly: tokenizes input, runs embed_tokens + layers[0..k], selects onward route from coverage map, forwards binary activations, receives tail output, and streams tokens back. The standalone gateway becomes a thin load-balancer routing to tracker nodes.",
"acceptanceCriteria": [
"Node with shard_start==0 (or --tracker-mode flag) exposes /v1/chat/completions alongside /forward",
"Tracker-node /v1/chat/completions: tokenize \u2192 embed \u2192 own layers \u2192 route selection \u2192 forward binary \u2192 receive tail \u2192 stream SSE",
"Two tracker nodes for same model each handle requests independently",
"Gateway proxies /v1/chat/completions to tracker nodes (round-robin), discovered via GET /v1/tracker-nodes/<model_preset>",
"Integration test: two tracker nodes + two mid-shard nodes for GPT-2; 10 requests via gateway; both tracker nodes receive load; all responses valid",
"Existing US-005 OpenAI-compatible tests still pass",
"python -m pytest passes from repo root",
"Commit only this story's changes"
],
"priority": 14,
"passes": false,
"notes": "Source issue: .scratch/distributed-inference-network/issues/14-tracker-as-node.md",
"dependsOn": [
"US-012",
"US-013"
]
} }
], ],
"metadata": { "metadata": {
"updatedAt": "2026-06-28T22:37:54.858Z" "updatedAt": "2026-06-29T00:00:00.000Z"
} }
} }

12
CONTEXT.md
View File

@@ -32,6 +32,18 @@ _Avoid_: proxy, relay, orchestrator, primary
The coordinator service that maintains the node registry, scores nodes by throughput/latency, and assigns inference routes. Runs as a centralized service with a P2P gossip fallback. The coordinator service that maintains the node registry, scores nodes by throughput/latency, and assigns inference routes. Runs as a centralized service with a P2P gossip fallback.
_Avoid_: coordinator, scheduler, director _Avoid_: coordinator, scheduler, director
**Tracker Node**:
A node that serves at least the first-layer shard (`layers[0..k]`) for a model and acts as the inference entry point for that model. Tracker nodes own the tokenizer and `embed_tokens`, receive client requests directly, select the onward route from the coverage map, and stream results back. Any node advertising a new model to the network becomes its tracker node.
_Avoid_: primary node, master node, gateway node
**Coverage Map**:
The tracker's per-model mapping of layer ranges to node counts: `[(start_layer, end_layer, node_count), ...]`. A layer range with `node_count=0` is a coverage gap — the model is unroutable until the gap is filled. Coverage-first bin-packing fills all gaps before adding redundancy.
_Avoid_: shard map, assignment table, coverage report
**Rebalance Directive**:
A `LOAD_SHARD` or `DROP_SHARD` instruction the tracker issues to a node when the coverage map changes (node joins, node leaves, or load-balance reoptimization). Delivered as part of the node's heartbeat response.
_Avoid_: rebalance command, shard instruction, migration order
**Node Score**: **Node Score**:
A throughput/latency rating the tracker maintains per node, used for route selection. Updated continuously from inference telemetry. A throughput/latency rating the tracker maintains per node, used for route selection. Updated continuously from inference telemetry.
_Avoid_: reputation, rating, rank _Avoid_: reputation, rating, rank

48
docs/adr/0008-binary-activation-wire-format.md Normal file
View File

@@ -0,0 +1,48 @@
# Binary activation wire format with zstd compression and chunked prefill
Activation tensors between shard nodes are sent as raw binary (little-endian bfloat16 bytes) over HTTP with metadata in request headers and optional zstd body compression. The previous base64-encoded JSON format is retired.
## Problem
Base64 adds 33% size overhead plus CPU cost on both ends. For a 700B model at bfloat16 (hidden_dim=16384), a 2048-token prefill generates 64MB of activation data per shard boundary. At 10 boundaries that is 640MB of transfer just for prefill. Encoding that as base64 JSON makes it ~860MB plus repeated parse overhead.
## Decision
**Control plane** (registration, heartbeat, route selection, health) stays JSON over HTTP — these are small and infrequent.
**Activation plane** (node-to-node tensor transfer) uses a binary HTTP body:
```
POST /forward HTTP/1.1
Content-Type: application/octet-stream
X-Meshnet-Shape: 1,128,16384
X-Meshnet-Dtype: bfloat16
X-Meshnet-Session: <uuid>
X-Meshnet-Chunk-Index: 0
X-Meshnet-Chunk-Total: 16
X-Meshnet-Encoding: zstd
Content-Length: <compressed_byte_count>
<raw zstd-compressed little-endian bfloat16 tensor bytes>
```
**Chunked prefill**: long prompts are split into 128-token chunks before the first forward pass. Each chunk is sent and forwarded independently through the pipeline. This keeps peak transfer per boundary at ~4MB (128 × 16384 × 2 bytes) rather than 64MB+ for a full 2048-token prefill. Generation (one token at a time) is already naturally chunked.
**Compression**: zstd level 1 (near-lz4 speed, better ratio). bfloat16 hidden states post-LayerNorm have high compressibility (many near-zero values). Expected 2–4× reduction on typical activations. `X-Meshnet-Encoding` is optional — omit for uncompressed binary; receiving nodes must handle both.
**Activation dtype**: always bfloat16 at every shard boundary, regardless of the weight quantization a node uses internally. Weights may be NF4 or INT8; computation upcasts to bfloat16; the result crosses the wire in bfloat16. This avoids dtype negotiation between nodes with different quantization tiers.
## Considered Options
- **base64 JSON**: existing format, works everywhere, ~33% overhead, no streaming, high CPU cost — retired
- **msgpack with embedded binary**: compact metadata + binary blob, requires msgpack dep on all nodes — rejected (HTTP headers are sufficient for metadata)
- **Raw TCP with length-prefixed frames**: maximally efficient, no HTTP overhead — rejected (breaks existing HTTP server infrastructure and complicates firewall traversal)
- **Binary HTTP body with headers**: chosen — no new protocol, standard HTTP chunked transfer, works with existing server framework, easy to debug with curl
## Consequences
- Node forward endpoint changes from `POST /forward` with JSON body to `POST /forward` with binary body and header metadata
- All nodes must be updated together (wire format version mismatch = 400 error with `X-Meshnet-Wire-Version` header for debugging)
- The stub wire format (`base64 JSON dict`) used in US-001 through US-010 is replaced; stub nodes can emit zeroed binary tensors trivially
- Chunk boundaries in prefill must align to token positions (no mid-token splits)
- KV cache invalidation across chunks is the calling node's responsibility; the receiving node treats each chunk as an independent forward pass segment

85
docs/adr/0009-coverage-first-shard-assignment-and-tracker-as-node.md Normal file
View File

@@ -0,0 +1,85 @@
# Coverage-first shard assignment and tracker-as-first-layer-node
The tracker assigns shard ranges to nodes using a coverage-first, speed-weighted bin-packing algorithm. Tracker nodes must host at least the first layer shard of every model they coordinate, making them the natural inference entry point. Any node serving layers[0..k] can become a tracker node for that model.
## Problem
A 700B model sharded across volunteer nodes of wildly different VRAM (8GB to 128GB+) and different quantization levels (NF4 / INT8 / bfloat16) must be fully covered at all times for inference to work at all. 199 copies of 99% of a model are worthless if the last 1% of layers has zero coverage. Replication only matters after full coverage is achieved.
## Decision
### Coverage-first bin-packing
The tracker maintains a **coverage map**: a sorted list of (start_layer, end_layer, node_count) for each model. A layer range with node_count=0 is a coverage gap — the model is unservable until the gap is filled. The tracker's shard assignment algorithm:
1. Identify uncovered layer ranges (gaps) first
2. Assign new or idle nodes to fill gaps, largest VRAM-fit first
3. Once fully covered, assign additional nodes to the most-replicated-needed ranges (latency hotspots) before adding redundancy elsewhere
4. Speed-weight: give faster nodes wider shard ranges — fill their VRAM completely, then assign the remainder to the next-fastest available node
Example: 700B NF4 model (~350GB weights). Node A has 128GB, Node B and C each have 80GB and are idle.
- Node A gets layers[0..k_a] filling ~120GB (reserve 8GB for KV cache)
- Node B gets layers[k_a..k_b] filling ~72GB
- Node C gets layers[k_b..N] (the remainder)
- If Node B benchmarks 2× faster than Node C, the tracker shifts the B/C boundary so B carries more layers
### Tracker-as-first-layer-node
Any node that advertises a new model to the network becomes a **tracker node** for that model. Tracker nodes have one hard requirement: they must hold and serve `layers[0..k]` (the first-layer shard) for every model they coordinate.
The reason is functional: a tracker node is also the inference entry point. When a client request arrives, the tracker node:
1. Tokenizes the input (owns the tokenizer)
2. Runs `model.embed_tokens` + `model.layers[0..k]`
3. Selects the optimal onward route from the coverage map
4. Forwards activations to the next node in the route
5. Receives the final hidden state back and streams tokens to the client
This collapses the separate "gateway" role: the tracker node that starts an inference request IS the gateway for that request. A standalone HTTP proxy/load-balancer may sit in front to pick which tracker node handles the request, but it carries no model weights.
Multiple tracker nodes for the same model = multiple entry points = horizontal scale for both routing decisions and the first-layer compute.
### Last-layer node (tail)
The node assigned `layers[N-k..N]` also runs `model.norm` and `model.lm_head`. It returns decoded token IDs (not hidden states) to the tracker node, which assembles the response. The tail shard assignment is marked `is_tail: true` in the shard registry.
### Adaptive quantization
Nodes declare their capabilities at registration:
```json
{
"vram_bytes": 25769803776,
"quantizations": ["nf4", "int8", "bfloat16"],
"benchmark_tokens_per_sec": 12.4,
"benchmark_model": "meta-llama/Llama-3-8B"
}
```
The tracker uses declared VRAM + quantization to compute max assignable layers:
```
max_layers = floor(available_vram / bytes_per_layer_at_declared_quant)
```
where `bytes_per_layer_at_declared_quant` is a per-model constant stored in the model preset metadata. KV cache headroom (configurable, default 20% of VRAM) is subtracted before the calculation.
A node may serve different models at different quantizations. The tracker tracks (model, quantization, shard_range) triples per node.
### Continuous rebalancing
The tracker monitors coverage every 30 seconds and on every node join/leave event. When a coverage gap appears (node drops offline), the tracker issues **rebalance directives** to idle nodes or nodes with redundant coverage:
- `LOAD_SHARD(model, start_layer, end_layer, quantization)` — download and begin serving
- `DROP_SHARD(model, start_layer, end_layer)` — stop serving (safe only when coverage > 1 for that range)
Nodes obey directives asynchronously; the tracker waits up to a configurable timeout before marking the gap critical.
## Considered Options
- **Static shard assignment** (assign once at registration): simple, breaks when nodes leave — rejected
- **Replication-first** (maximize redundancy before optimizing speed): wastes resources on popular shards, leaves rare shards uncovered — rejected (user: "we have no use of 199 copies of 99% of the model")
- **Coverage-first, speed-weighted bin-packing with continuous rebalancing**: chosen
## Consequences
- The standalone `meshnet-gateway` service from US-005 becomes a thin load-balancer that routes to tracker nodes; tracker nodes do the actual inference orchestration
- Tracker nodes must download more model data (tokenizer + first-layer shard) — this is the price of being an entry point
- Benchmark data is self-reported by nodes at registration; the validator can detect fraudulent benchmarks (a node claiming 100 tokens/sec but delivering 2 gets slashed for under-performance)
- VRAM reservation for KV cache means nodes can host fewer layers than their raw VRAM suggests — this is intentional; running out of KV cache during inference causes OOM crashes
- New CONTEXT.md terms: **Tracker Node** (node serving first-layer shard + inference routing for a model), **Coverage Map** (tracker's per-model layer-range → node-count mapping), **Rebalance Directive** (tracker instruction to a node to load or drop a shard)