neuron-tai/docs/issues/14-tracker-as-node.md

US-014 — Tracker-as-first-layer-node (inference entry point)

Merge the inference orchestration role of the gateway into tracker nodes that serve the first-layer shard. A tracker node for a model receives client requests directly, runs the head shard, routes activations through the network, collects the tail output, and streams tokens back to the client. The standalone gateway becomes a thin load-balancer that routes to whichever tracker node is least loaded.

Context

Per ADR-0009:

• Any node serving layers[0..k] for a model can act as its tracker node
• Tracker nodes own: tokenizer, embed_tokens, layers[0..k]
• Tracker nodes select the optimal onward route from the live coverage map (they already maintain it)
• Multiple tracker nodes for the same model = horizontal scale at both the routing and the first-layer compute level
• The standalone meshnet-gateway process from US-005 becomes a dumb round-robin load-balancer; it no longer orchestrates shard pipelines

This collapses a network hop and a process boundary: previously gateway → tracker → node; now client → tracker-node (which is both tracker and first-layer node).

Acceptance Criteria

• A node started with --tracker-mode (or automatically when assigned shard_start=0) exposes both its existing /forward endpoint and a new /v1/chat/completions OpenAI-compatible endpoint
• The tracker-node's /v1/chat/completions handler: tokenizes input, embeds, runs its own layers, selects onward route from coverage map, forwards binary activations to next node, receives tail output, decodes tokens, streams SSE back to client
• Multiple tracker nodes for the same model can each independently handle requests (verified by sending requests to both and getting valid responses)
• The existing meshnet-gateway is updated to proxy requests to tracker nodes (round-robin or least-connections) rather than orchestrating the pipeline itself
• The gateway can discover tracker nodes from the tracker registry (GET /v1/tracker-nodes/<model_preset> returns the list of endpoints for tracker nodes for that model)
• An integration test: register two tracker nodes and two mid-shard nodes for GPT-2; send 10 requests to the gateway; assert both tracker nodes received roughly equal load and all responses are valid
• Existing US-005 OpenAI-compatible tests still pass (gateway still exposes /v1/chat/completions, just proxies to tracker nodes now)
• python -m pytest passes
• Commit only this story's changes

Implementation Notes

• Tracker-mode detection: shard_start == 0 OR --tracker-mode CLI flag
• The tracker-node process runs two HTTP servers on different ports: port 7000 for node-to-node (/forward), port 8080 for client-facing (/v1/chat/completions) — or a single server with both route prefixes
• Route selection inside the tracker node: same GET /v1/route/<model_preset> call to the central tracker registry, but the tracker node itself is already the first hop — the returned route starts from the second node
• Streaming: tail node returns token IDs one by one (or in batches) back to the tracker node via chunked HTTP; tracker node converts to SSE and streams to client
• The gateway's existing pipeline orchestration code (_run_pipeline) can be extracted into a shared meshnet_common.pipeline module reused by both tracker-node and the legacy gateway path