docs: add ADRs and user stories for real model inference stack (US-011–014)

ADR-0008: binary activation wire format — raw bfloat16 over HTTP, zstd compression,
128-token chunked prefill; replaces base64 JSON (~33% overhead removed).

ADR-0009: coverage-first shard assignment and tracker-as-first-layer-node —
any node serving layers[0..k] becomes the inference entry point for that model;
bin-packing fills all coverage gaps before adding redundancy; tracker issues
LOAD_SHARD/DROP_SHARD rebalance directives; nodes declare VRAM + quantization.

US-011: binary wire format migration
US-012: real PyTorch layer execution (transformers + bitsandbytes, test on GPT-2)
US-013: coverage-first tracker bin-packing with VRAM-aware shard assignment
US-014: tracker-as-node (tracker node serves first layers + handles client requests)

CONTEXT.md: Tracker Node, Coverage Map, Rebalance Directive terms added.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Dobromir Popov
2026-06-29 14:43:54 +03:00
parent 3286f77e27
commit b02e07d308
8 changed files with 471 additions and 56 deletions

View File

@@ -32,6 +32,18 @@ _Avoid_: proxy, relay, orchestrator, primary
The coordinator service that maintains the node registry, scores nodes by throughput/latency, and assigns inference routes. Runs as a centralized service with a P2P gossip fallback.
_Avoid_: coordinator, scheduler, director
**Tracker Node**:
A node that serves at least the first-layer shard (`layers[0..k]`) for a model and acts as the inference entry point for that model. Tracker nodes own the tokenizer and `embed_tokens`, receive client requests directly, select the onward route from the coverage map, and stream results back. Any node advertising a new model to the network becomes its tracker node.
_Avoid_: primary node, master node, gateway node
**Coverage Map**:
The tracker's per-model mapping of layer ranges to node counts: `[(start_layer, end_layer, node_count), ...]`. A layer range with `node_count=0` is a coverage gap — the model is unroutable until the gap is filled. Coverage-first bin-packing fills all gaps before adding redundancy.
_Avoid_: shard map, assignment table, coverage report
**Rebalance Directive**:
A `LOAD_SHARD` or `DROP_SHARD` instruction the tracker issues to a node when the coverage map changes (node joins, node leaves, or load-balance reoptimization). Delivered as part of the node's heartbeat response.
_Avoid_: rebalance command, shard instruction, migration order
**Node Score**:
A throughput/latency rating the tracker maintains per node, used for route selection. Updated continuously from inference telemetry.
_Avoid_: reputation, rating, rank