docs: add ADRs and user stories for real model inference stack (US-011–014)
ADR-0008: binary activation wire format — raw bfloat16 over HTTP, zstd compression, 128-token chunked prefill; replaces base64 JSON (~33% overhead removed). ADR-0009: coverage-first shard assignment and tracker-as-first-layer-node — any node serving layers[0..k] becomes the inference entry point for that model; bin-packing fills all coverage gaps before adding redundancy; tracker issues LOAD_SHARD/DROP_SHARD rebalance directives; nodes declare VRAM + quantization. US-011: binary wire format migration US-012: real PyTorch layer execution (transformers + bitsandbytes, test on GPT-2) US-013: coverage-first tracker bin-packing with VRAM-aware shard assignment US-014: tracker-as-node (tracker node serves first layers + handles client requests) CONTEXT.md: Tracker Node, Coverage Map, Rebalance Directive terms added. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
12
CONTEXT.md
12
CONTEXT.md
@@ -32,6 +32,18 @@ _Avoid_: proxy, relay, orchestrator, primary
|
||||
The coordinator service that maintains the node registry, scores nodes by throughput/latency, and assigns inference routes. Runs as a centralized service with a P2P gossip fallback.
|
||||
_Avoid_: coordinator, scheduler, director
|
||||
|
||||
**Tracker Node**:
|
||||
A node that serves at least the first-layer shard (`layers[0..k]`) for a model and acts as the inference entry point for that model. Tracker nodes own the tokenizer and `embed_tokens`, receive client requests directly, select the onward route from the coverage map, and stream results back. Any node advertising a new model to the network becomes its tracker node.
|
||||
_Avoid_: primary node, master node, gateway node
|
||||
|
||||
**Coverage Map**:
|
||||
The tracker's per-model mapping of layer ranges to node counts: `[(start_layer, end_layer, node_count), ...]`. A layer range with `node_count=0` is a coverage gap — the model is unroutable until the gap is filled. Coverage-first bin-packing fills all gaps before adding redundancy.
|
||||
_Avoid_: shard map, assignment table, coverage report
|
||||
|
||||
**Rebalance Directive**:
|
||||
A `LOAD_SHARD` or `DROP_SHARD` instruction the tracker issues to a node when the coverage map changes (node joins, node leaves, or load-balance reoptimization). Delivered as part of the node's heartbeat response.
|
||||
_Avoid_: rebalance command, shard instruction, migration order
|
||||
|
||||
**Node Score**:
|
||||
A throughput/latency rating the tracker maintains per node, used for route selection. Updated continuously from inference telemetry.
|
||||
_Avoid_: reputation, rating, rank
|
||||
|
||||
Reference in New Issue
Block a user