docs: consolidate all docs under docs/ — single source of truth
Move issues (01–29) and PRD from .scratch/distributed-inference-network/ into docs/issues/ and docs/. Update ralph_progress.py DEFAULT_PRD path and rewrite docs/agents/issue-tracker.md to reflect the new layout. The distributed_inference_network.egg-info/docs/ mirror is a build artifact already covered by *.egg-info/ in .gitignore — not committed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
34
docs/issues/14-tracker-as-node.md
Normal file
34
docs/issues/14-tracker-as-node.md
Normal file
@@ -0,0 +1,34 @@
|
||||
# US-014 — Tracker-as-first-layer-node (inference entry point)
|
||||
|
||||
Merge the inference orchestration role of the gateway into tracker nodes that serve the first-layer shard. A tracker node for a model receives client requests directly, runs the head shard, routes activations through the network, collects the tail output, and streams tokens back to the client. The standalone gateway becomes a thin load-balancer that routes to whichever tracker node is least loaded.
|
||||
|
||||
## Context
|
||||
|
||||
Per ADR-0009:
|
||||
- Any node serving `layers[0..k]` for a model can act as its tracker node
|
||||
- Tracker nodes own: tokenizer, `embed_tokens`, `layers[0..k]`
|
||||
- Tracker nodes select the optimal onward route from the live coverage map (they already maintain it)
|
||||
- Multiple tracker nodes for the same model = horizontal scale at both the routing and the first-layer compute level
|
||||
- The standalone `meshnet-gateway` process from US-005 becomes a dumb round-robin load-balancer; it no longer orchestrates shard pipelines
|
||||
|
||||
This collapses a network hop and a process boundary: previously gateway → tracker → node; now client → tracker-node (which is both tracker and first-layer node).
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- A node started with `--tracker-mode` (or automatically when assigned `shard_start=0`) exposes both its existing `/forward` endpoint and a new `/v1/chat/completions` OpenAI-compatible endpoint
|
||||
- The tracker-node's `/v1/chat/completions` handler: tokenizes input, embeds, runs its own layers, selects onward route from coverage map, forwards binary activations to next node, receives tail output, decodes tokens, streams SSE back to client
|
||||
- Multiple tracker nodes for the same model can each independently handle requests (verified by sending requests to both and getting valid responses)
|
||||
- The existing `meshnet-gateway` is updated to proxy requests to tracker nodes (round-robin or least-connections) rather than orchestrating the pipeline itself
|
||||
- The gateway can discover tracker nodes from the tracker registry (`GET /v1/tracker-nodes/<model_preset>` returns the list of endpoints for tracker nodes for that model)
|
||||
- An integration test: register two tracker nodes and two mid-shard nodes for GPT-2; send 10 requests to the gateway; assert both tracker nodes received roughly equal load and all responses are valid
|
||||
- Existing US-005 OpenAI-compatible tests still pass (gateway still exposes `/v1/chat/completions`, just proxies to tracker nodes now)
|
||||
- `python -m pytest` passes
|
||||
- Commit only this story's changes
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
- Tracker-mode detection: `shard_start == 0` OR `--tracker-mode` CLI flag
|
||||
- The tracker-node process runs two HTTP servers on different ports: port 7000 for node-to-node (`/forward`), port 8080 for client-facing (`/v1/chat/completions`) — or a single server with both route prefixes
|
||||
- Route selection inside the tracker node: same `GET /v1/route/<model_preset>` call to the central tracker registry, but the tracker node itself is already the first hop — the returned route starts from the *second* node
|
||||
- Streaming: tail node returns token IDs one by one (or in batches) back to the tracker node via chunked HTTP; tracker node converts to SSE and streams to client
|
||||
- The gateway's existing pipeline orchestration code (`_run_pipeline`) can be extracted into a shared `meshnet_common.pipeline` module reused by both tracker-node and the legacy gateway path
|
||||
Reference in New Issue
Block a user