Files

Dobromir Popov b02e07d308 docs: add ADRs and user stories for real model inference stack (US-011–014)

ADR-0008: binary activation wire format — raw bfloat16 over HTTP, zstd compression,
128-token chunked prefill; replaces base64 JSON (~33% overhead removed).

ADR-0009: coverage-first shard assignment and tracker-as-first-layer-node —
any node serving layers[0..k] becomes the inference entry point for that model;
bin-packing fills all coverage gaps before adding redundancy; tracker issues
LOAD_SHARD/DROP_SHARD rebalance directives; nodes declare VRAM + quantization.

US-011: binary wire format migration
US-012: real PyTorch layer execution (transformers + bitsandbytes, test on GPT-2)
US-013: coverage-first tracker bin-packing with VRAM-aware shard assignment
US-014: tracker-as-node (tracker node serves first layers + handles client requests)

CONTEXT.md: Tracker Node, Coverage Map, Rebalance Directive terms added.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

2026-06-29 14:43:54 +03:00

3.3 KiB

Raw Blame History

US-014 — Tracker-as-first-layer-node (inference entry point)

Merge the inference orchestration role of the gateway into tracker nodes that serve the first-layer shard. A tracker node for a model receives client requests directly, runs the head shard, routes activations through the network, collects the tail output, and streams tokens back to the client. The standalone gateway becomes a thin load-balancer that routes to whichever tracker node is least loaded.

Context

Per ADR-0009:

Any node serving layers[0..k] for a model can act as its tracker node
Tracker nodes own: tokenizer, embed_tokens, layers[0..k]
Tracker nodes select the optimal onward route from the live coverage map (they already maintain it)
Multiple tracker nodes for the same model = horizontal scale at both the routing and the first-layer compute level
The standalone meshnet-gateway process from US-005 becomes a dumb round-robin load-balancer; it no longer orchestrates shard pipelines

This collapses a network hop and a process boundary: previously gateway → tracker → node; now client → tracker-node (which is both tracker and first-layer node).

Acceptance Criteria

A node started with --tracker-mode (or automatically when assigned shard_start=0) exposes both its existing /forward endpoint and a new /v1/chat/completions OpenAI-compatible endpoint
The tracker-node's /v1/chat/completions handler: tokenizes input, embeds, runs its own layers, selects onward route from coverage map, forwards binary activations to next node, receives tail output, decodes tokens, streams SSE back to client
Multiple tracker nodes for the same model can each independently handle requests (verified by sending requests to both and getting valid responses)
The existing meshnet-gateway is updated to proxy requests to tracker nodes (round-robin or least-connections) rather than orchestrating the pipeline itself
The gateway can discover tracker nodes from the tracker registry (GET /v1/tracker-nodes/<model_preset> returns the list of endpoints for tracker nodes for that model)
An integration test: register two tracker nodes and two mid-shard nodes for GPT-2; send 10 requests to the gateway; assert both tracker nodes received roughly equal load and all responses are valid
Existing US-005 OpenAI-compatible tests still pass (gateway still exposes /v1/chat/completions, just proxies to tracker nodes now)
python -m pytest passes
Commit only this story's changes

Implementation Notes

Tracker-mode detection: shard_start == 0 OR --tracker-mode CLI flag
The tracker-node process runs two HTTP servers on different ports: port 7000 for node-to-node (/forward), port 8080 for client-facing (/v1/chat/completions) — or a single server with both route prefixes
Route selection inside the tracker node: same GET /v1/route/<model_preset> call to the central tracker registry, but the tracker node itself is already the first hop — the returned route starts from the second node
Streaming: tail node returns token IDs one by one (or in batches) back to the tracker node via chunked HTTP; tracker node converts to SSE and streams to client
The gateway's existing pipeline orchestration code (_run_pipeline) can be extracted into a shared meshnet_common.pipeline module reused by both tracker-node and the legacy gateway path

3.3 KiB Raw Blame History

US-014 — Tracker-as-first-layer-node (inference entry point)

Context

Acceptance Criteria

Implementation Notes

3.3 KiB

Raw Blame History