docs: consolidate all docs under docs/ — single source of truth
Move issues (01–29) and PRD from .scratch/distributed-inference-network/ into docs/issues/ and docs/. Update ralph_progress.py DEFAULT_PRD path and rewrite docs/agents/issue-tracker.md to reflect the new layout. The distributed_inference_network.egg-info/docs/ mirror is a build artifact already covered by *.egg-info/ in .gitignore — not committed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Dobromir Popov
29
docs/issues/01-monorepo-scaffold.md
Normal file
29
docs/issues/01-monorepo-scaffold.md
Normal file
@@ -0,0 +1,29 @@
|
||||
Status: ready-for-agent
|
||||
|
||||
# 01 — Monorepo scaffold + single-node smoke test
|
||||
|
||||
## What to build
|
||||
|
||||
Stand up the monorepo package layout and prove that a client HTTP request can travel end-to-end through the stack — gateway → one node serving all layers → valid response — before any real model or distributed logic is wired in.
|
||||
|
||||
The six top-level packages should exist with minimal stubs:
|
||||
- `packages/node` — node client CLI (`meshnet-node`)
|
||||
- `packages/gateway` — HTTP gateway + route orchestration
|
||||
- `packages/tracker` — node registry and route selection
|
||||
- `packages/sdk` — `meshnet` Python SDK
|
||||
- `packages/contracts` — Solana smart contract wrappers
|
||||
- `packages/p2p` — gossip and shard swarm
|
||||
|
||||
The smoke test uses a tiny in-process stub model (not a real LLM) so the test is fast and has no external dependencies. The node runs in the same process as the test. The gateway is a real HTTP server. The client sends a real `POST /v1/chat/completions` request and receives a valid OpenAI-format response.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- [ ] `pip install -e packages/node packages/gateway packages/tracker` works from repo root
|
||||
- [ ] `meshnet-node`, `meshnet-gateway`, and `meshnet-tracker` CLI entry points exist
|
||||
- [ ] A single integration test starts a gateway and one stub node in-process, sends a `POST /v1/chat/completions` request, and asserts a valid OpenAI-format response is returned
|
||||
- [ ] The test passes with `pytest` from repo root with no external services running
|
||||
- [ ] All six package directories exist with `pyproject.toml` and an importable top-level module
|
||||
|
||||
## Blocked by
|
||||
|
||||
None — can start immediately.
|
||||
23
docs/issues/02-two-node-shard-pipeline.md
Normal file
23
docs/issues/02-two-node-shard-pipeline.md
Normal file
@@ -0,0 +1,23 @@
|
||||
Status: ready-for-agent
|
||||
|
||||
# 02 — Two-node shard pipeline
|
||||
|
||||
## What to build
|
||||
|
||||
Extend the single-node smoke test so that inference is routed through exactly two nodes, each serving half of a stub model's layers. Activation tensors must travel from the gateway to node A, then from node A to node B, and the final output returns to the gateway. This proves the distributed shard pipeline works end-to-end before real models or a real tracker are introduced.
|
||||
|
||||
The two nodes run in the same process as the test (no real network required). The gateway hardcodes the two-node route for now — dynamic route selection comes in issue 03. The activation tensor protocol (shape, dtype, serialisation format) must be established here, as all later issues depend on it.
|
||||
|
||||
Use the domain vocabulary from `CONTEXT.md`: the ordered list of nodes is an **inference route**; each node's layer range is a **shard**; the tensors passed between nodes are activations (not "data" or "chunks").
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- [ ] The integration test spins up two stub nodes, each configured with a non-overlapping shard range
|
||||
- [ ] A `POST /v1/chat/completions` request results in activation tensors flowing node-A → node-B (verifiable via test assertions or logs)
|
||||
- [ ] The gateway assembles the final response correctly from node-B's output
|
||||
- [ ] The test passes with no external services running
|
||||
- [ ] The activation tensor serialisation format is documented in a short inline comment (shape, dtype, wire format) — this becomes the contract all future nodes implement
|
||||
|
||||
## Blocked by
|
||||
|
||||
- `01-monorepo-scaffold.md`
|
||||
24
docs/issues/03-tracker-registration-and-routing.md
Normal file
24
docs/issues/03-tracker-registration-and-routing.md
Normal file
@@ -0,0 +1,24 @@
|
||||
Status: ready-for-agent
|
||||
|
||||
# 03 — Tracker: node registration + route selection
|
||||
|
||||
## What to build
|
||||
|
||||
Replace the hardcoded two-node route from issue 02 with a real tracker. Nodes register themselves with the tracker on startup (endpoint, shard range, hardware profile, node score). The gateway queries the tracker for an inference route for a given model preset instead of using a static list. The tracker returns an ordered list of node endpoints whose shards collectively cover all layers.
|
||||
|
||||
The tracker runs as a lightweight HTTP service. Node score is initially a simple placeholder (e.g. fixed value or random) — real throughput/latency scoring comes later. The tracker must correctly reject route requests when no registered nodes cover a required shard range and return an appropriate error.
|
||||
|
||||
This issue establishes the tracker's registration and routing API contract, which issues 04 (node startup) and 05 (OpenAI gateway) both depend on.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- [ ] A node can register with the tracker via HTTP, providing its endpoint, shard range, and a hardware profile
|
||||
- [ ] The gateway queries the tracker with a model preset name and receives an ordered list of node endpoints forming a complete inference route
|
||||
- [ ] The end-to-end integration test from issue 02 still passes with the tracker now in the loop (no hardcoded routes)
|
||||
- [ ] The tracker returns a clear error response when no route is available for a requested model preset
|
||||
- [ ] Nodes that fail to heartbeat within a configurable window are removed from the registry
|
||||
- [ ] The tracker's registration and route-selection HTTP API is defined (paths, request/response shapes) in the tracker package
|
||||
|
||||
## Blocked by
|
||||
|
||||
- `02-two-node-shard-pipeline.md`
|
||||
32
docs/issues/04-node-client-startup.md
Normal file
32
docs/issues/04-node-client-startup.md
Normal file
@@ -0,0 +1,32 @@
|
||||
Status: ready-for-agent
|
||||
|
||||
# 04 — Node client startup flow (`meshnet-node start`)
|
||||
|
||||
## What to build
|
||||
|
||||
Make `meshnet-node start` self-configuring from scratch on a machine with a CUDA-capable GPU (or CPU fallback). The full startup sequence must complete without any manual configuration:
|
||||
|
||||
1. Detect GPU model and available VRAM
|
||||
2. Load an existing Solana wallet from disk, or generate and save a new one
|
||||
3. Query the tracker for the optimal shard assignment given the hardware profile
|
||||
4. Download the assigned shard from HuggingFace (`huggingface_hub`)
|
||||
5. Register with the tracker (wallet address, endpoint, shard range, hardware profile)
|
||||
6. Begin accepting inference connections
|
||||
|
||||
The node prints a short status summary on startup: wallet address, assigned shard, model preset, and download progress. No interactive prompts — the entire flow is non-interactive.
|
||||
|
||||
This is the primary viral growth vector. Every second of friction in this flow costs node operators. The startup sequence must work on Linux with CUDA, and degrade gracefully to CPU on machines without a GPU.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- [ ] `meshnet-node start --tracker http://localhost:8080` completes startup without prompts on a machine with a CUDA GPU
|
||||
- [ ] A Solana wallet keypair is generated and saved to `~/.config/meshnet/wallet.json` if none exists
|
||||
- [ ] The assigned shard is downloaded from HuggingFace and cached to `~/.cache/meshnet/shards/`
|
||||
- [ ] The node registers with the tracker and appears in the tracker's node registry
|
||||
- [ ] The node accepts a live inference connection and processes activation tensors correctly after startup
|
||||
- [ ] On a CPU-only machine, the node starts with a warning and serves a CPU-appropriate shard
|
||||
- [ ] An integration test covers the full startup sequence against a local tracker stub
|
||||
|
||||
## Blocked by
|
||||
|
||||
- `03-tracker-registration-and-routing.md`
|
||||
29
docs/issues/05-openai-compatible-gateway.md
Normal file
29
docs/issues/05-openai-compatible-gateway.md
Normal file
@@ -0,0 +1,29 @@
|
||||
Status: ready-for-agent
|
||||
|
||||
# 05 — OpenAI-compatible gateway
|
||||
|
||||
## What to build
|
||||
|
||||
Expose a production-shape HTTP API from the gateway so that any client using the OpenAI Python SDK (or any OpenAI-compatible tool) works by changing only the `base_url`. The gateway translates OpenAI-format requests into inference route execution and streams responses back in OpenAI-format.
|
||||
|
||||
Endpoints to implement:
|
||||
- `POST /v1/chat/completions` — streaming (`text/event-stream`) and non-streaming
|
||||
- `GET /v1/models` — returns the list of model presets currently routable on the network
|
||||
- `GET /v1/health` — liveness check
|
||||
|
||||
The gateway selects an inference route from the tracker, executes the shard pipeline, and assembles the streamed response. If the tracker returns no route for the requested model, the gateway responds with a standard OpenAI-format error (`model_not_available`).
|
||||
|
||||
Authentication (API key → SOL/USDC balance) is a stub in this issue — return 200 for any non-empty `Authorization` header. Real payment gating comes in issue 06.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- [ ] `openai.OpenAI(base_url="http://localhost:8080/v1", api_key="test").chat.completions.create(model="stub-model", messages=[...])` returns a valid response
|
||||
- [ ] Streaming works: `stream=True` returns `text/event-stream` chunks in OpenAI SSE format
|
||||
- [ ] `GET /v1/models` returns a JSON array of available model preset names
|
||||
- [ ] A request for an unavailable model returns an OpenAI-format error response with HTTP 503
|
||||
- [ ] LangChain `ChatOpenAI(base_url=..., api_key=...)` works against the gateway
|
||||
- [ ] An integration test covers streaming and non-streaming paths end-to-end through a real tracker and two stub nodes
|
||||
|
||||
## Blocked by
|
||||
|
||||
- `03-tracker-registration-and-routing.md`
|
||||
32
docs/issues/06-solana-stake-and-settlement.md
Normal file
32
docs/issues/06-solana-stake-and-settlement.md
Normal file
@@ -0,0 +1,32 @@
|
||||
Status: ready-for-agent
|
||||
|
||||
# 06 — Solana stake + settlement contracts
|
||||
|
||||
## What to build
|
||||
|
||||
Deploy and integrate the Solana smart contracts that make node staking, client payment, and token reward settlement trustless. All development and testing targets **Solana testnet** — never devnet or mainnet during development, to avoid real costs.
|
||||
|
||||
Three contracts are needed:
|
||||
|
||||
**Registry contract**: records stake balances, strike counts, and ban status per wallet. The gateway and tracker read from this contract to exclude banned wallets from route selection.
|
||||
|
||||
**Payment contract**: clients pre-fund an API key account with SOL or USDC. The gateway records per-request compute attribution (which node served which layer range, for how many tokens).
|
||||
|
||||
**Settlement contract**: called once per epoch. Debits client accounts proportional to compute consumed. Credits node operator wallets with our native token proportional to layers served. Distributes a validator reward share.
|
||||
|
||||
The `packages/contracts` package provides Python wrappers for reading and writing to all three contracts. The gateway uses these wrappers to check stake before routing to a node and to record attribution after each request.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- [ ] All contracts deploy successfully to Solana testnet
|
||||
- [ ] A node can submit a stake transaction and have its balance reflected in the registry contract
|
||||
- [ ] A client can fund an API key account with testnet SOL
|
||||
- [ ] After a completed inference session, compute attribution is recorded on-chain with correct node/layer attribution
|
||||
- [ ] The epoch settlement transaction correctly distributes token rewards to node operators and deducts client balances
|
||||
- [ ] The gateway refuses to route to a node whose stake balance is below the minimum threshold
|
||||
- [ ] All contract interactions in tests run against a local Solana test validator (via `solana-test-validator`) — no live testnet required for CI
|
||||
- [ ] A `.env.testnet` config points to Solana testnet RPC for manual end-to-end testing
|
||||
|
||||
## Blocked by
|
||||
|
||||
- `03-tracker-registration-and-routing.md`
|
||||
28
docs/issues/07-fraud-detection-slash.md
Normal file
28
docs/issues/07-fraud-detection-slash.md
Normal file
@@ -0,0 +1,28 @@
|
||||
Status: ready-for-agent
|
||||
|
||||
# 07 — Fraud detection: validator + on-chain slash
|
||||
|
||||
## What to build
|
||||
|
||||
Implement the optimistic fraud detection loop. After each inference request completes, the validator process independently decides whether to re-run the request on a reference node (~5% sample rate). If the reference output diverges from the node's output beyond a configurable floating-point tolerance, the validator submits a slash transaction on-chain.
|
||||
|
||||
The validator is a separate process (not the gateway or tracker). It subscribes to completed inference events, selects a random sample, re-runs those requests against a trusted reference node, compares outputs, and submits slash proofs when fraud is detected.
|
||||
|
||||
On slash: the offending node's stake balance decreases by the configured slash amount; its strike count increments. The gateway reads strike state from the registry contract on each route selection and excludes nodes that exceed the strike threshold.
|
||||
|
||||
Node operators receive a push notification (logged to stdout and, if configured, a webhook) when their node is slashed.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- [ ] The validator process samples ~5% of completed inference requests (configurable)
|
||||
- [ ] A node returning a deliberately wrong output is detected and slashed within one validation cycle
|
||||
- [ ] The on-chain stake balance of the slashed node decreases by the correct slash amount
|
||||
- [ ] The strike count for the slashed node increments on-chain
|
||||
- [ ] A node that reaches the strike threshold is excluded from route selection on the next gateway request
|
||||
- [ ] A slashed node logs a clear warning to stdout
|
||||
- [ ] An integration test: run a deliberately-bad node, send 20 requests, assert at least one slash transaction is submitted and the node is eventually excluded from routes
|
||||
|
||||
## Blocked by
|
||||
|
||||
- `05-openai-compatible-gateway.md`
|
||||
- `06-solana-stake-and-settlement.md`
|
||||
27
docs/issues/08-probationary-period-and-bans.md
Normal file
27
docs/issues/08-probationary-period-and-bans.md
Normal file
@@ -0,0 +1,27 @@
|
||||
Status: ready-for-agent
|
||||
|
||||
# 08 — Node probationary period + ban enforcement
|
||||
|
||||
## What to build
|
||||
|
||||
Implement the two anti-sybil mechanisms that make re-entering the network after a ban economically costly.
|
||||
|
||||
**Probationary period**: a new wallet's first N completed inference jobs earn no token rewards. N is a configurable on-chain parameter (default: 50 jobs). The settlement contract reads the wallet's job count and skips reward distribution until the threshold is met. The node client displays the remaining probationary job count on startup.
|
||||
|
||||
**Ban enforcement**: when a wallet's strike count reaches the configured threshold, the registry contract marks it as banned. The tracker and gateway must check ban status on every route selection and exclude banned wallets. Ban status is read exclusively from the Solana registry contract — the tracker cannot set or unset bans.
|
||||
|
||||
Both mechanisms together mean a banned node operator must: create a new wallet, fund a new stake, and complete N free jobs before earning again — a meaningful economic barrier.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- [ ] A node with a new wallet receives no token rewards for its first N jobs (verified via settlement contract state)
|
||||
- [ ] The node client prints remaining probationary jobs on startup (e.g. "Probationary period: 38 jobs remaining before earning")
|
||||
- [ ] After N jobs, the next epoch settlement correctly credits the node with token rewards
|
||||
- [ ] A wallet with strike count at the threshold is marked banned in the registry contract
|
||||
- [ ] The tracker excludes banned wallets from route selection
|
||||
- [ ] A banned wallet that attempts to register with the tracker is rejected
|
||||
- [ ] An integration test covers: new wallet → N jobs → earning begins; and: strike threshold reached → banned → excluded from routes
|
||||
|
||||
## Blocked by
|
||||
|
||||
- `07-fraud-detection-slash.md`
|
||||
24
docs/issues/09-p2p-shard-swarm.md
Normal file
24
docs/issues/09-p2p-shard-swarm.md
Normal file
@@ -0,0 +1,24 @@
|
||||
Status: ready-for-agent
|
||||
|
||||
# 09 — P2P shard swarm
|
||||
|
||||
## What to build
|
||||
|
||||
Once a node has downloaded a shard, it seeds that shard to other nodes that are assigned the same shard. New nodes with the same shard assignment should download from the swarm (peer nodes) rather than HuggingFace when peers are available, reducing HuggingFace load and speeding up propagation of popular shards.
|
||||
|
||||
The shard swarm is keyed by `(model_preset, shard_index)`. Nodes announce their shard availability to the tracker on registration. When a new node requests a shard download, the tracker returns a list of peer endpoints that hold the shard alongside the HuggingFace fallback URL. The node tries peers first; falls back to HuggingFace if no peers respond within a timeout.
|
||||
|
||||
Peers serve shard data over a simple chunked HTTP endpoint — no custom P2P protocol needed in v1. The node client displays download source (peer or HuggingFace) during shard download.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- [ ] A node that has downloaded a shard is listed as a peer for that shard in the tracker
|
||||
- [ ] A second node assigned the same shard downloads it from the first node (peer) rather than HuggingFace (verified by asserting no HuggingFace HTTP calls are made in the test)
|
||||
- [ ] If no peers are available, the node falls back to HuggingFace without error
|
||||
- [ ] The node client logs whether the shard was downloaded from a peer or HuggingFace
|
||||
- [ ] Shard chunks are verified against a checksum before being marked complete
|
||||
- [ ] An integration test: node A downloads shard from HuggingFace stub; node B with same assignment downloads from node A
|
||||
|
||||
## Blocked by
|
||||
|
||||
- `04-node-client-startup.md`
|
||||
44
docs/issues/10-meshnet-sdk.md
Normal file
44
docs/issues/10-meshnet-sdk.md
Normal file
@@ -0,0 +1,44 @@
|
||||
Status: ready-for-agent
|
||||
|
||||
# 10 — `meshnet` Python SDK
|
||||
|
||||
## What to build
|
||||
|
||||
A Python SDK (`packages/sdk`) that wraps the OpenAI-compatible gateway and exposes network-specific controls that the raw OpenAI SDK cannot express. The SDK is the recommended interface for developers who want more than basic inference.
|
||||
|
||||
The SDK surface:
|
||||
|
||||
```python
|
||||
from meshnet import Client
|
||||
|
||||
client = Client(api_key="...", base_url="https://api.meshnet.ai")
|
||||
|
||||
# Basic inference (identical to OpenAI SDK)
|
||||
response = client.chat.completions.create(model="llama-3-70b", messages=[...])
|
||||
|
||||
# Network-specific features
|
||||
client.wallet.balance() # SOL/USDC balance remaining
|
||||
client.wallet.top_up(amount_sol=1.0) # returns a Solana payment address + QR
|
||||
client.models.available() # list of routable model presets with shard coverage %
|
||||
client.request(redundancy=2) # route through 2 independent chains, return majority
|
||||
client.estimate_cost(model="llama-3-70b", tokens=1000) # cost in SOL before sending
|
||||
```
|
||||
|
||||
The SDK must not require clients to hold or know about our native token. All wallet operations are in SOL or USDC.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- [ ] `pip install meshnet` installs the SDK
|
||||
- [ ] `Client.chat.completions.create(...)` works identically to the OpenAI SDK (same parameters, same response shape)
|
||||
- [ ] `client.wallet.balance()` returns the current SOL/USDC balance for the API key
|
||||
- [ ] `client.wallet.top_up()` returns a valid Solana payment address
|
||||
- [ ] `client.models.available()` returns model presets with shard coverage percentage
|
||||
- [ ] `client.estimate_cost(model, tokens)` returns a cost estimate in SOL before the request is sent
|
||||
- [ ] `client.request(redundancy=2)` sends the same prompt to two independent inference routes and returns the majority response
|
||||
- [ ] The SDK is typed (py.typed, full type stubs)
|
||||
- [ ] An integration test covers each SDK method against a local gateway + tracker + stub nodes
|
||||
|
||||
## Blocked by
|
||||
|
||||
- `05-openai-compatible-gateway.md`
|
||||
- `06-solana-stake-and-settlement.md`
|
||||
50
docs/issues/11-binary-wire-format.md
Normal file
50
docs/issues/11-binary-wire-format.md
Normal file
@@ -0,0 +1,50 @@
|
||||
# US-011 — Binary activation wire format
|
||||
|
||||
Replace the base64-encoded JSON activation payload from US-002 with a binary HTTP body, zstd compression, and chunked prefill. This is a protocol migration that must be applied to all nodes and the gateway before US-012 (real model backend) can be built on a sane foundation.
|
||||
|
||||
## Context
|
||||
|
||||
The current wire format encodes activation tensors as base64 strings inside a JSON dict:
|
||||
```json
|
||||
{"shape": [1, 1, 64], "dtype": "float32", "data": "<base64>", "context": {"prompt": "..."}}
|
||||
```
|
||||
This works for stub nodes (tiny tensors) but is unsuitable for real models:
|
||||
- base64 adds 33% size overhead
|
||||
- A 2048-token prefill at hidden_dim=16384 generates 64MB of activations per boundary — 85MB base64
|
||||
- JSON parsing overhead grows with payload size
|
||||
|
||||
The new format per ADR-0008:
|
||||
```
|
||||
POST /forward HTTP/1.1
|
||||
Content-Type: application/octet-stream
|
||||
X-Meshnet-Shape: 1,128,16384
|
||||
X-Meshnet-Dtype: bfloat16
|
||||
X-Meshnet-Session: <uuid>
|
||||
X-Meshnet-Chunk-Index: 0
|
||||
X-Meshnet-Chunk-Total: 16
|
||||
X-Meshnet-Encoding: zstd
|
||||
Content-Length: <n>
|
||||
|
||||
<raw zstd-compressed little-endian bfloat16 bytes>
|
||||
```
|
||||
|
||||
Stub nodes still emit zeroed tensors — just in binary now. No real model required for this story.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- `packages/node` `/forward` endpoint reads `X-Meshnet-Shape`, `X-Meshnet-Dtype`, `X-Meshnet-Session`, `X-Meshnet-Chunk-Index`, `X-Meshnet-Chunk-Total`, and `X-Meshnet-Encoding` headers; reads raw body bytes; decompresses if `zstd`; reconstructs tensor as numpy array
|
||||
- Node forward handler returns a binary response with the same header set reflecting output shape
|
||||
- The gateway sends binary chunked activations to the first node; reassembles binary responses from the last node
|
||||
- Chunked prefill: the gateway splits input sequences longer than `MESHNET_CHUNK_TOKENS` (default 128) into N chunks and sends them sequentially through the pipeline; stub nodes pass through each chunk unchanged
|
||||
- An integration test sends a 512-token stub activation (4 chunks of 128) through a two-node pipeline and asserts all 4 chunk responses are received with correct headers
|
||||
- `zstd` Python package added as a dependency to `packages/node` and `packages/gateway`
|
||||
- The old `_make_stub_activations` function in `server.py` is replaced with `_make_stub_binary_activation(shape, dtype) -> bytes`
|
||||
- `python -m pytest` passes from repo root
|
||||
- Commit only this story's changes
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
- Use `numpy` to pack/unpack tensor bytes: `np.frombuffer(body, dtype=np.float16).reshape(shape)` (bfloat16 → numpy uses `np.dtype('bfloat16')` in numpy >= 1.20, or load as uint16 and reinterpret)
|
||||
- zstd: `import zstandard as zstd; cctx = zstd.ZstdCompressor(level=1); compressed = cctx.compress(raw_bytes)`
|
||||
- Chunk index + total allow the receiving node to know whether this is a mid-sequence chunk (no special handling needed for stub; real model in US-012 will need to manage KV cache across chunks)
|
||||
- Wire version header `X-Meshnet-Wire: 2` can be added for debugging mismatches between old and new nodes
|
||||
61
docs/issues/12-real-pytorch-model-backend.md
Normal file
61
docs/issues/12-real-pytorch-model-backend.md
Normal file
@@ -0,0 +1,61 @@
|
||||
# US-012 — Real PyTorch model backend
|
||||
|
||||
Replace stub node inference with actual `transformers` layer execution. A node downloads a HuggingFace SafeTensors model, loads `model.layers[start:end]` into GPU/CPU memory at the declared quantization level, and runs real forward passes. The first-layer node additionally runs `embed_tokens`; the last-layer node additionally runs `model.norm` and `lm_head`.
|
||||
|
||||
## Context
|
||||
|
||||
Per ADR-0001 (PyTorch over llama.cpp) and the grilling session decisions:
|
||||
- Model format: HuggingFace SafeTensors (GGUF rejected — no per-layer hidden state API)
|
||||
- Layer extraction: `model.model.layers[start:end]` from a loaded `AutoModelForCausalLM`
|
||||
- Quantization: bitsandbytes NF4 / INT8 / bfloat16, declared by node at registration
|
||||
- Activation dtype at boundaries: always `bfloat16` regardless of weight quantization
|
||||
- First-layer node owns: tokenizer, `model.embed_tokens`, `model.layers[0..k]`
|
||||
- Last-layer node owns: `model.layers[N-k..N]`, `model.norm`, `model.lm_head`
|
||||
|
||||
Start with a small real model for testing: `openai-community/gpt2` (117M, no special license, SafeTensors available). The production target is 700B+ models — the layer extraction pattern is identical regardless of size.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- `packages/node` can be started with `--model-id openai-community/gpt2 --shard-start 0 --shard-end 6` and loads the corresponding transformer blocks
|
||||
- Node detects whether it is a head shard (`shard_start == 0`) and loads tokenizer + `model.embed_tokens`
|
||||
- Node detects whether it is a tail shard (`shard_end == total_layers`) and loads `model.norm + model.lm_head`
|
||||
- A `/forward` request with a text prompt (head node) or binary bfloat16 tensor (mid/tail node) runs a real forward pass and returns binary bfloat16 activations
|
||||
- A two-node integration test with GPT-2 sharded across two nodes returns a coherent (non-random, deterministic) text completion
|
||||
- Quantization level is read from `--quantization [bfloat16|int8|nf4]` flag; `bfloat16` is default; `int8` and `nf4` use `bitsandbytes.BitsAndBytesConfig`
|
||||
- A node with insufficient VRAM for the assigned shard at the declared quantization prints a clear error and exits (no silent OOM)
|
||||
- `transformers`, `bitsandbytes`, `safetensors`, and `accelerate` added as dependencies to `packages/node`
|
||||
- `python -m pytest` passes (GPT-2 test downloads model once, cached; mark with `@pytest.mark.integration` and skip in CI if no GPU)
|
||||
- Commit only this story's changes
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
```python
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
|
||||
import torch
|
||||
|
||||
def load_shard(model_id, start, end, quantization):
|
||||
quant_config = None
|
||||
if quantization == "nf4":
|
||||
quant_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
|
||||
bnb_4bit_compute_dtype=torch.bfloat16)
|
||||
elif quantization == "int8":
|
||||
quant_config = BitsAndBytesConfig(load_in_8bit=True)
|
||||
|
||||
# Load full model metadata without weights, then extract layers
|
||||
model = AutoModelForCausalLM.from_pretrained(
|
||||
model_id,
|
||||
quantization_config=quant_config,
|
||||
device_map="auto",
|
||||
torch_dtype=torch.bfloat16,
|
||||
)
|
||||
layers = model.model.layers[start:end]
|
||||
return model, layers
|
||||
```
|
||||
|
||||
- `device_map="auto"` handles GPU/CPU offload automatically based on available VRAM
|
||||
- For large models, use `low_cpu_mem_usage=True` to avoid doubling peak RAM during load
|
||||
- The forward pass for a mid-shard node: `hidden_states = layer(hidden_states)[0]` (most HF models return a tuple, take index 0)
|
||||
- Head node forward: `input_ids → embed_tokens → position_embed → layers[0..k] → bfloat16 binary out`
|
||||
- Tail node forward: `bfloat16 binary in → layers[N-k..N] → norm → lm_head → argmax/sample → token_id → back to head`
|
||||
- Attention mask and position IDs must be forwarded alongside hidden states for models that need them (pack them in additional headers or a small JSON sidecar in the same HTTP request)
|
||||
- Keep attention/position passing simple for v1: assume left-padded inputs, pass `attention_mask` as a separate binary blob with header `X-Meshnet-Attn-Mask: <base64-uint8>` (small enough for base64 — it's [B, S] not [B, S, D])
|
||||
53
docs/issues/13-coverage-first-shard-assignment.md
Normal file
53
docs/issues/13-coverage-first-shard-assignment.md
Normal file
@@ -0,0 +1,53 @@
|
||||
# US-013 — Coverage-first tracker shard assignment
|
||||
|
||||
Upgrade the tracker's route selection to a coverage-first, speed-weighted bin-packing algorithm. The tracker maintains a live coverage map per model, issues rebalance directives when gaps appear, and assigns shard ranges to nodes based on declared VRAM, quantization, and benchmark throughput.
|
||||
|
||||
## Context
|
||||
|
||||
Per ADR-0009:
|
||||
- **Coverage gap** = any layer range in a model with zero serving nodes → model is unroutable
|
||||
- **Coverage-first**: fill all gaps before adding redundancy to covered ranges
|
||||
- **Speed-weighted bin-packing**: fill fastest nodes to their VRAM limit first; cascade to next-fastest
|
||||
- **Continuous rebalancing**: triggered every 30s and on every node join/leave event
|
||||
- **Rebalance directive**: `LOAD_SHARD` or `DROP_SHARD` instruction sent from tracker to node
|
||||
|
||||
Node registration now includes capability data:
|
||||
```json
|
||||
{
|
||||
"wallet": "...",
|
||||
"endpoint": "http://10.0.0.5:7000",
|
||||
"vram_bytes": 25769803776,
|
||||
"ram_bytes": 137438953472,
|
||||
"quantizations": ["nf4", "int8", "bfloat16"],
|
||||
"benchmark_tokens_per_sec": 12.4,
|
||||
"benchmark_model": "openai-community/gpt2"
|
||||
}
|
||||
```
|
||||
|
||||
Model preset metadata (stored in tracker config) includes `bytes_per_layer` at each quantization level and `total_layers`. The tracker can compute max assignable layers for any node.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- Tracker accepts `vram_bytes`, `ram_bytes`, `quantizations`, `benchmark_tokens_per_sec` in the node registration payload
|
||||
- `GET /v1/coverage/<model_preset>` returns the coverage map: list of `{start_layer, end_layer, node_count}` for each assigned range
|
||||
- A model is only routable when all layer ranges have `node_count >= 1`
|
||||
- When a new node registers, the tracker assigns it to the highest-priority uncovered range (or expands the most-loaded range if fully covered)
|
||||
- When a node disconnects (heartbeat timeout), the tracker marks affected ranges as reduced-coverage and, if any reach 0, issues `LOAD_SHARD` directives to idle nodes within 30 seconds
|
||||
- The shard assignment respects VRAM: `assigned_layers <= floor((vram_bytes * 0.8) / bytes_per_layer_at_quant)`
|
||||
- Speed weighting: given two idle nodes both capable of covering a gap, the tracker assigns the wider sub-range to the faster node
|
||||
- An integration test: start tracker with a model preset, register three nodes with different VRAM, assert the coverage map shows 100% coverage and the largest VRAM node received the widest shard range
|
||||
- A second integration test: kill the node covering the middle layer range; assert tracker issues a `LOAD_SHARD` directive to an idle node and coverage recovers
|
||||
- `python -m pytest` passes
|
||||
- Commit only this story's changes
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
- Coverage map data structure: `dict[str, list[tuple[int, int, int]]]` keyed by model preset
|
||||
- Bin-packing is run in-process on the tracker (not a separate service); it's triggered by registration/heartbeat events and a 30s periodic timer
|
||||
- Rebalance directives are delivered as responses to node heartbeat POSTs (node polls tracker every 10s anyway) — no new push channel needed
|
||||
- `bytes_per_layer` is read from a `model_presets.json` config file; add GPT-2 (12 layers, ~30MB/layer bfloat16) as the test preset
|
||||
- The current tracker `_select_route` function is extended, not replaced — backward compat with stub nodes that don't send VRAM data (default to 8GB / bfloat16 if omitted)
|
||||
|
||||
## Comments
|
||||
|
||||
- 2026-06-30: Follow-up capacity hardening lives in `20-memory-budget-shard-slots-and-dropout-relocation.md`. US-013 remains the base coverage-first assignment and dropout rebalance story; US-020 owns operator `--memory`, `--max-shards`, shard-slot enforcement, and relocation limit hardening so the two scopes do not conflict.
|
||||
34
docs/issues/14-tracker-as-node.md
Normal file
34
docs/issues/14-tracker-as-node.md
Normal file
@@ -0,0 +1,34 @@
|
||||
# US-014 — Tracker-as-first-layer-node (inference entry point)
|
||||
|
||||
Merge the inference orchestration role of the gateway into tracker nodes that serve the first-layer shard. A tracker node for a model receives client requests directly, runs the head shard, routes activations through the network, collects the tail output, and streams tokens back to the client. The standalone gateway becomes a thin load-balancer that routes to whichever tracker node is least loaded.
|
||||
|
||||
## Context
|
||||
|
||||
Per ADR-0009:
|
||||
- Any node serving `layers[0..k]` for a model can act as its tracker node
|
||||
- Tracker nodes own: tokenizer, `embed_tokens`, `layers[0..k]`
|
||||
- Tracker nodes select the optimal onward route from the live coverage map (they already maintain it)
|
||||
- Multiple tracker nodes for the same model = horizontal scale at both the routing and the first-layer compute level
|
||||
- The standalone `meshnet-gateway` process from US-005 becomes a dumb round-robin load-balancer; it no longer orchestrates shard pipelines
|
||||
|
||||
This collapses a network hop and a process boundary: previously gateway → tracker → node; now client → tracker-node (which is both tracker and first-layer node).
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- A node started with `--tracker-mode` (or automatically when assigned `shard_start=0`) exposes both its existing `/forward` endpoint and a new `/v1/chat/completions` OpenAI-compatible endpoint
|
||||
- The tracker-node's `/v1/chat/completions` handler: tokenizes input, embeds, runs its own layers, selects onward route from coverage map, forwards binary activations to next node, receives tail output, decodes tokens, streams SSE back to client
|
||||
- Multiple tracker nodes for the same model can each independently handle requests (verified by sending requests to both and getting valid responses)
|
||||
- The existing `meshnet-gateway` is updated to proxy requests to tracker nodes (round-robin or least-connections) rather than orchestrating the pipeline itself
|
||||
- The gateway can discover tracker nodes from the tracker registry (`GET /v1/tracker-nodes/<model_preset>` returns the list of endpoints for tracker nodes for that model)
|
||||
- An integration test: register two tracker nodes and two mid-shard nodes for GPT-2; send 10 requests to the gateway; assert both tracker nodes received roughly equal load and all responses are valid
|
||||
- Existing US-005 OpenAI-compatible tests still pass (gateway still exposes `/v1/chat/completions`, just proxies to tracker nodes now)
|
||||
- `python -m pytest` passes
|
||||
- Commit only this story's changes
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
- Tracker-mode detection: `shard_start == 0` OR `--tracker-mode` CLI flag
|
||||
- The tracker-node process runs two HTTP servers on different ports: port 7000 for node-to-node (`/forward`), port 8080 for client-facing (`/v1/chat/completions`) — or a single server with both route prefixes
|
||||
- Route selection inside the tracker node: same `GET /v1/route/<model_preset>` call to the central tracker registry, but the tracker node itself is already the first hop — the returned route starts from the *second* node
|
||||
- Streaming: tail node returns token IDs one by one (or in batches) back to the tracker node via chunked HTTP; tracker node converts to SSE and streams to client
|
||||
- The gateway's existing pipeline orchestration code (`_run_pipeline`) can be extracted into a shared `meshnet_common.pipeline` module reused by both tracker-node and the legacy gateway path
|
||||
178
docs/issues/15-ralph-agent-agnostic-status-aware.md
Normal file
178
docs/issues/15-ralph-agent-agnostic-status-aware.md
Normal file
@@ -0,0 +1,178 @@
|
||||
# US-015 — Ralph: agent-agnostic runner + status-field-aware dashboard
|
||||
|
||||
Two tightly coupled improvements to the Ralph workflow tooling:
|
||||
|
||||
1. **Status-field awareness** — `ralph_progress.py` currently reads `passes: true/false` everywhere. Replace all `passes` references with the rich `status` field introduced in the prd.json schema migration. Surface `to-revise` and `needs-review` stories as an attention list rather than treating them like failed or blocked tasks.
|
||||
|
||||
2. **Agent-agnostic runner** — Ralph currently hardcodes `agentPlugin: "codex"` and always calls `ralph-tui run --agent codex`. Make the agent selectable per session with first-class support for Claude Code CLI, OpenRouter (unified API covering GPT-4, Mistral, DeepSeek, Llama, etc.), and the existing Codex plugin.
|
||||
|
||||
## Part 1: Status-field awareness in ralph_progress.py
|
||||
|
||||
### What to replace
|
||||
|
||||
Every occurrence of `story.get("passes")` in `scripts/ralph_progress.py` must be replaced with a helper that reads the `status` field with a `passes` fallback for backward compatibility:
|
||||
|
||||
```python
|
||||
# Status → done mapping (backward-compat: passes=True counts as done if no status field)
|
||||
_DONE_STATUSES = {"done"}
|
||||
_ATTENTION_STATUSES = {"to-revise", "needs-review"}
|
||||
_ACTIVE_STATUSES = {"in-progress"}
|
||||
_DESIGN_STATUSES = {"in-design"}
|
||||
|
||||
def _is_done(story: dict) -> bool:
|
||||
status = story.get("status")
|
||||
if status:
|
||||
return status in _DONE_STATUSES
|
||||
return bool(story.get("passes")) # fallback
|
||||
|
||||
def _needs_attention(story: dict) -> bool:
|
||||
return story.get("status") in _ATTENTION_STATUSES
|
||||
|
||||
def _is_active(story: dict) -> bool:
|
||||
return story.get("status") in _ACTIVE_STATUSES
|
||||
|
||||
def _is_in_design(story: dict) -> bool:
|
||||
return story.get("status") in _DESIGN_STATUSES
|
||||
```
|
||||
|
||||
### `_story_sets` rewrite
|
||||
|
||||
Current behaviour lumps `to-revise` into the "done" bucket (because `passes=True`) or the "open" bucket. New behaviour:
|
||||
|
||||
```
|
||||
done → status == "done" (or passes=True fallback)
|
||||
attention → status in {to-revise, needs-review} ← new bucket
|
||||
active → status == "in-progress"
|
||||
in-design → status == "in-design" (treated as blocked — needs human before agent)
|
||||
ready → status == "open" AND all deps done
|
||||
blocked → status == "open" AND some dep not done
|
||||
```
|
||||
|
||||
### `_deps_done` rewrite
|
||||
|
||||
Dependency is satisfied when the dep story `_is_done()` — not when `passes=True`.
|
||||
|
||||
### Dashboard additions
|
||||
|
||||
```
|
||||
Ralph progress: Distributed Inference Network
|
||||
PRD: .scratch/distributed-inference-network/prd.json
|
||||
[############################] 13/15 complete (86%)
|
||||
Done: 13 Ready: 1 Attention: 2 Blocked: 0
|
||||
|
||||
⚠ Attention required (review before running):
|
||||
⚠ US-002 02 — Two-node shard pipeline
|
||||
Wire format replaced by US-011 binary protocol; verify tests.
|
||||
⚠ US-005 05 — OpenAI-compatible gateway
|
||||
Gateway orchestration superseded by US-014; defer until US-014 lands.
|
||||
|
||||
→ US-014 14 — Tracker-as-first-layer-node (inference entry point)
|
||||
```
|
||||
|
||||
### `auto` command behaviour
|
||||
|
||||
`auto` skips `to-revise` and `needs-review` stories — they need human review before an agent re-runs them. Add `--include-revise` flag to override this for unattended runs.
|
||||
|
||||
### `_review_report` additions
|
||||
|
||||
Include a dedicated **Attention Required** section in the generated review brief listing all `to-revise` and `needs-review` stories with their `status_reason`.
|
||||
|
||||
---
|
||||
|
||||
## Part 2: Agent-agnostic runner
|
||||
|
||||
### Agent selection
|
||||
|
||||
Add `--agent` option with these values:
|
||||
|
||||
| Value | CLI invoked | Config required |
|
||||
|-------|-------------|-----------------|
|
||||
| `codex` | `ralph-tui run --agent codex ...` | `OPENAI_API_KEY` (existing) |
|
||||
| `claude` | `ralph-tui run --agent claude ...` | `ANTHROPIC_API_KEY` |
|
||||
| `openrouter` | `ralph-tui run --agent openrouter ...` | `OPENROUTER_API_KEY` + `--model` |
|
||||
| `custom` | `--agent-cmd <path>` | user-defined |
|
||||
|
||||
If `ralph-tui` does not natively support a requested agent plugin, `ralph_progress.py` falls back to invoking the agent CLI directly with the task prompt file, bypassing `ralph-tui run` entirely.
|
||||
|
||||
### Persistent agent config
|
||||
|
||||
Save the last-used agent choice to `.ralph-tui/agent-config.json` so it doesn't have to be passed on every command:
|
||||
|
||||
```json
|
||||
{
|
||||
"agent": "openrouter",
|
||||
"model": "anthropic/claude-opus-4",
|
||||
"updatedAt": "2026-06-29T..."
|
||||
}
|
||||
```
|
||||
|
||||
`--agent` on the CLI always overrides the saved config. `ralph_progress.py set-agent --agent openrouter --model openai/gpt-4o` writes the config file.
|
||||
|
||||
### OpenRouter adapter
|
||||
|
||||
When `agent == "openrouter"` and `ralph-tui` doesn't natively support it, `ralph_progress.py` implements a thin adapter:
|
||||
|
||||
1. Read the task prompt from the issue file + prd.json story
|
||||
2. POST to `https://openrouter.ai/api/v1/chat/completions` with `OPENROUTER_API_KEY`
|
||||
3. Stream response to stdout
|
||||
4. Watch prd.json for `status` change to detect task completion
|
||||
5. Timeout after configurable duration (default: 10 minutes per task)
|
||||
|
||||
The OpenRouter model is set via `--model` (e.g. `openai/gpt-4o`, `mistralai/mistral-large`, `meta-llama/llama-3-70b-instruct`). Default: `anthropic/claude-opus-4`.
|
||||
|
||||
### `custom` agent
|
||||
|
||||
`--agent custom --agent-cmd ./my-agent.sh` — Ralph passes the task prompt file path as `$1`. The script exits 0 on success. This makes Ralph compatible with any future agent tool without code changes.
|
||||
|
||||
---
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- All `passes` reads in `ralph_progress.py` replaced with `_is_done()` helper (with fallback)
|
||||
- `_story_sets` returns five buckets: done, attention, active, in-design, ready, blocked
|
||||
- Dashboard shows `⚠ Attention required` section with `status_reason` for each affected story
|
||||
- `auto` skips `to-revise` / `needs-review` by default; `--include-revise` overrides
|
||||
- `ralph_progress.py set-agent --agent <name>` writes `.ralph-tui/agent-config.json`
|
||||
- `--agent codex|claude|openrouter|custom` accepted by all subcommands that invoke Ralph
|
||||
- `ralph_progress.py run-next --agent openrouter --model openai/gpt-4o` runs a task via OpenRouter adapter
|
||||
- `ralph_progress.py run-next --agent custom --agent-cmd ./my-agent.sh` runs a task via custom script
|
||||
- `python -m pytest` passes from repo root
|
||||
- Commit only this story's changes
|
||||
|
||||
---
|
||||
|
||||
## Part 4: Per-story metadata in dashboard
|
||||
|
||||
Every story in the dashboard gets an optional second line showing what is known about it:
|
||||
|
||||
```
|
||||
✓ US-012 12 — Real PyTorch model backend
|
||||
codex · done · "Added model_backend.py with bitsandbytes NF4 support"
|
||||
⚡ US-014 14 — Tracker-as-first-layer-node
|
||||
codex · in-progress · worktree: ../AI-worktree-us-014
|
||||
→ US-015 15 — Ralph agent-agnostic runner
|
||||
claude · in-progress · worktree: ../AI-worktree-us-015
|
||||
```
|
||||
|
||||
### Sources
|
||||
|
||||
| Data | Source |
|
||||
|------|--------|
|
||||
| Active worktree path | `git worktree list --porcelain` — match branches `feat/<story-id>` |
|
||||
| Agent (active story) | `.ralph-tui/agent-config.json` → `agent` field |
|
||||
| Agent (completed story) | `.ralph-tui/session.json` → `agentPlugin`; or `completionNotes` text |
|
||||
| Last output summary | `story.completionNotes` (done); latest `git log -1 --format=%s feat/<id>` (worktree); last lines of most recent `.ralph-tui/iterations/<hash>_<date>_<id>.log` (in-progress) |
|
||||
|
||||
### Behaviour
|
||||
|
||||
- Metadata line is omitted if there is nothing to show (no agent, no worktree, no notes)
|
||||
- Default: shown (verbose). `--compact` flag suppresses it for a tighter one-line-per-story view
|
||||
- `_active_worktrees() -> dict[story_id, rel_path]` helper, called once per dashboard render
|
||||
- `_story_meta(story, worktrees, session) -> str | None` helper returns the joined metadata string
|
||||
|
||||
### Additional acceptance criteria
|
||||
|
||||
- `ralph_progress.py show` displays worktree path for any story whose branch exists as `feat/<story-id>`
|
||||
- Agent name appears next to in-progress and recently-completed stories
|
||||
- `completionNotes` from prd.json appears as the summary for done stories
|
||||
- `--compact` suppresses metadata lines
|
||||
206
docs/issues/16-mining-cli-ux.md
Normal file
206
docs/issues/16-mining-cli-ux.md
Normal file
@@ -0,0 +1,206 @@
|
||||
# US-016 — Mining-style node startup CLI + live dashboard
|
||||
|
||||
## Goal
|
||||
|
||||
Replace the bare flag-driven `meshnet-node start` with a wizard-guided first-run experience modelled on GPU mining clients (like PhoenixMiner, lolMiner, etc.). After the wizard, the terminal switches to a live status dashboard showing real-time node health and earnings.
|
||||
|
||||
## Wizard flow (first run only)
|
||||
|
||||
```
|
||||
╔══════════════════════════════════════════════════════════╗
|
||||
║ meshnet-node v0.1.0 ║
|
||||
║ Distributed AI Inference — Node Setup ║
|
||||
╚══════════════════════════════════════════════════════════╝
|
||||
|
||||
Detecting hardware...
|
||||
GPU 0: NVIDIA RTX 4090 24 GB VRAM ✓
|
||||
GPU 1: NVIDIA RTX 3090 24 GB VRAM ✓
|
||||
|
||||
Select a model to serve:
|
||||
|
||||
# Model Layers NF4 INT8 BF16
|
||||
1 Llama-3-70B-Instruct 80 ✓18GB ✓40GB ✗80GB
|
||||
2 Qwen-2.5-72B-Instruct 80 ✓19GB ✗41GB ✗81GB
|
||||
3 Mixtral-8x7B-Instruct-v0.1 32 ✓ 7GB ✓14GB ✓27GB
|
||||
4 Phi-3-medium-128k-instruct 40 ✓ 4GB ✓ 8GB ✓15GB
|
||||
5 [Browse HuggingFace…]
|
||||
|
||||
Enter number [1]: _
|
||||
|
||||
Quantization [nf4/int8/bf16] (nf4 recommended for 24GB): _
|
||||
|
||||
Download directory [~/.meshnet/models]: _
|
||||
|
||||
Tracker URL [http://localhost:8080]: _
|
||||
|
||||
Wallet path [~/.config/meshnet/wallet.json] (new wallet will be created): _
|
||||
|
||||
Config saved to ~/.config/meshnet/config.json
|
||||
Starting node…
|
||||
```
|
||||
|
||||
Second run with existing config:
|
||||
|
||||
```
|
||||
meshnet-node
|
||||
Reading config from ~/.config/meshnet/config.json
|
||||
Model: Llama-3-70B-Instruct Quant: nf4 Shard: layers 0–15
|
||||
Tracker: http://192.168.1.10:8080
|
||||
Starting…
|
||||
```
|
||||
|
||||
## Live dashboard (once running)
|
||||
|
||||
Renders every 2 seconds using `rich.live`. Fallback: plain-text status line if `rich` is unavailable or terminal is not a TTY (important for WSL2 / SSH).
|
||||
|
||||
```
|
||||
meshnet-node Llama-3-70B-Instruct [nf4] shard 0–15/80 up 00:03:22
|
||||
|
||||
GPU 0 RTX 4090 GPU ████████░░ 73% VRAM 18.2/24.0 GB 45°C
|
||||
GPU 1 RTX 3090 GPU ███░░░░░░░ 28% VRAM 8.7/24.0 GB 38°C
|
||||
|
||||
Tokens/sec ▁▂▃▄▅▆▇█ 42.3 t/s (EMA 30s)
|
||||
Requests 1,247 served 3 active
|
||||
Peers 8 connected (tracker: ✓ relay: ✓)
|
||||
TAI earned 0.00 TAI (payments active after US-006)
|
||||
Uptime 00:03:22
|
||||
|
||||
[q] quit [r] reset stats [c] compact view
|
||||
```
|
||||
|
||||
Compact mode (`--compact` or pressing `c`) shows a single status line:
|
||||
```
|
||||
[43t/s VRAM18.2GB req1247 peers8 up3m22s]
|
||||
```
|
||||
|
||||
## Implementation notes
|
||||
|
||||
### Hardware detection
|
||||
|
||||
```python
|
||||
import torch
|
||||
|
||||
def detect_gpus() -> list[dict]:
|
||||
gpus = []
|
||||
if torch.cuda.is_available():
|
||||
for i in range(torch.cuda.device_count()):
|
||||
props = torch.cuda.get_device_properties(i)
|
||||
gpus.append({
|
||||
"index": i,
|
||||
"name": props.name,
|
||||
"vram_gb": props.total_memory / 1e9,
|
||||
"backend": "cuda"
|
||||
})
|
||||
# ROCm / Apple Silicon stubs for later
|
||||
return gpus
|
||||
```
|
||||
|
||||
### Curated model list
|
||||
|
||||
`packages/node/meshnet_node/model_catalog.py` — a hardcoded list of `ModelPreset` dataclasses:
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class ModelPreset:
|
||||
name: str # display name
|
||||
hf_repo: str # HuggingFace repo ID
|
||||
num_layers: int
|
||||
vram_gb: dict # {"nf4": 18, "int8": 40, "bf16": 80}
|
||||
description: str # one-line description
|
||||
```
|
||||
|
||||
Initial list (expand over time):
|
||||
- `meta-llama/Meta-Llama-3-70B-Instruct` — 80L, NF4 18GB, INT8 40GB, BF16 80GB
|
||||
- `Qwen/Qwen2.5-72B-Instruct` — 80L, NF4 19GB, INT8 41GB, BF16 81GB
|
||||
- `mistralai/Mixtral-8x7B-Instruct-v0.1` — 32L, NF4 7GB, INT8 14GB, BF16 27GB
|
||||
- `microsoft/Phi-3-medium-128k-instruct` — 40L, NF4 4GB, INT8 8GB, BF16 15GB
|
||||
- `google/gemma-2-27b-it` — 46L, NF4 10GB, INT8 20GB, BF16 40GB
|
||||
|
||||
### HuggingFace Browse
|
||||
|
||||
```python
|
||||
from huggingface_hub import list_models
|
||||
|
||||
def browse_hf(top_n=20) -> list[dict]:
|
||||
models = list_models(
|
||||
pipeline_tag="text-generation",
|
||||
library="transformers",
|
||||
sort="downloads",
|
||||
direction=-1,
|
||||
limit=top_n,
|
||||
cardData=True,
|
||||
)
|
||||
return [{"repo": m.modelId, "downloads": m.downloads} for m in models]
|
||||
```
|
||||
|
||||
### Persistent config
|
||||
|
||||
`~/.config/meshnet/config.json`:
|
||||
```json
|
||||
{
|
||||
"model_hf_repo": "meta-llama/Meta-Llama-3-70B-Instruct",
|
||||
"quantization": "nf4",
|
||||
"download_dir": "~/.meshnet/models",
|
||||
"tracker_url": "http://192.168.1.10:8080",
|
||||
"wallet_path": "~/.config/meshnet/wallet.json",
|
||||
"shard_start": null,
|
||||
"shard_end": null,
|
||||
"updatedAt": "2026-06-29T..."
|
||||
}
|
||||
```
|
||||
|
||||
`shard_start`/`shard_end`: null means tracker auto-assigns. User can pin a range for dedicated partial-model nodes.
|
||||
|
||||
### CLI flags
|
||||
|
||||
All wizard answers are overridable without re-running the wizard:
|
||||
|
||||
```
|
||||
meshnet-node [start]
|
||||
--model <hf-repo-id> # e.g. meta-llama/Meta-Llama-3-70B-Instruct
|
||||
--quantization [bf16|int8|nf4]
|
||||
--download-dir <path>
|
||||
--tracker <url>
|
||||
--wallet <path>
|
||||
--shard-start <int> # pin shard range (optional)
|
||||
--shard-end <int>
|
||||
--reset-config # ignore saved config, re-run wizard
|
||||
--no-tui # plain-text output (for CI / headless)
|
||||
--compact # single-line status instead of full dashboard
|
||||
|
||||
meshnet-node models # list curated models and exit
|
||||
meshnet-node models --browse # list HF Hub top-20 and exit
|
||||
meshnet-node config # print current config and exit
|
||||
```
|
||||
|
||||
### WSL2 / non-TTY fallback
|
||||
|
||||
```python
|
||||
import sys, os
|
||||
|
||||
def is_interactive_tty() -> bool:
|
||||
return sys.stdout.isatty() and os.environ.get("TERM") not in ("dumb", "")
|
||||
|
||||
if not is_interactive_tty():
|
||||
# fall back to plain-text periodic status
|
||||
run_plain_status_loop(node)
|
||||
else:
|
||||
run_rich_dashboard(node)
|
||||
```
|
||||
|
||||
Do NOT use `termios`, `fcntl`, or `/dev/tty` — these break in Windows cmd.exe and some WSL2 terminal emulators.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- `meshnet-node` with no args and no config → wizard starts
|
||||
- Wizard detects GPU and marks `[too large]` for models that exceed available VRAM
|
||||
- `meshnet-node models` prints curated list and exits
|
||||
- `meshnet-node models --browse` calls HF Hub API, prints top-20, exits
|
||||
- Second run (config exists) → skips wizard, starts immediately
|
||||
- `--reset-config` re-runs wizard even with config present
|
||||
- All wizard inputs override-able via CLI flags
|
||||
- Live rich dashboard renders and updates every 2s when running in a TTY
|
||||
- Falls back to plain-text when not a TTY (CI / WSL2 without TERM set)
|
||||
- Ctrl-C prints a clean summary line and exits 0
|
||||
- `python -m pytest` passes from repo root
|
||||
- Commit only this story's changes
|
||||
205
docs/issues/17-p2p-gossip-relay-ssl.md
Normal file
205
docs/issues/17-p2p-gossip-relay-ssl.md
Normal file
@@ -0,0 +1,205 @@
|
||||
# US-017 — P2P gossip, NAT-traversal relay node, and SSL/TLS
|
||||
|
||||
## Goal
|
||||
|
||||
Nodes must work behind NAT (home routers, cloud VMs without public IPs) and must communicate securely. Implement:
|
||||
|
||||
1. **SSL/TLS everywhere** — all HTTP between nodes/tracker is HTTPS; all WebSocket gossip is WSS
|
||||
2. **mDNS peer discovery** — nodes on the same LAN find each other automatically (no config)
|
||||
3. **WebSocket gossip PubSub** — nodes propagate join/leave/coverage-update events in near-real-time
|
||||
4. **Circuit relay node** — team-run public relay (`packages/relay`) that enables NAT traversal and bootstraps new nodes joining from the internet
|
||||
|
||||
Architecture is designed to migrate to libp2p GossipSub + Kademlia DHT without breaking the message schema (topic names and payload formats are stable contracts).
|
||||
|
||||
## Gossip protocol
|
||||
|
||||
### Transport
|
||||
|
||||
WebSocket (`wss://`) using the `websockets` Python library. Each node maintains persistent WSS connections to:
|
||||
- The relay node (always, bootstraps peer list)
|
||||
- Up to 8 direct peers (Kademlia-style target fanout; peers discovered via mDNS + relay peer list)
|
||||
|
||||
### Topics
|
||||
|
||||
All messages are JSON with an envelope:
|
||||
|
||||
```json
|
||||
{
|
||||
"topic": "node-join",
|
||||
"version": 1,
|
||||
"from_peer": "<peer_id>",
|
||||
"timestamp": "<iso8601>",
|
||||
"payload": { ... }
|
||||
}
|
||||
```
|
||||
|
||||
| Topic | Direction | Payload |
|
||||
|-------|-----------|---------|
|
||||
| `node-join` | broadcast | `{peer_id, addr, models: [{model_preset, shard_start, shard_end}], vram_gb, quant}` |
|
||||
| `node-leave` | broadcast | `{peer_id, reason}` |
|
||||
| `coverage-update` | broadcast | `{model_preset, coverage: [{start, end, count}]}` |
|
||||
| `heartbeat` | peer→relay | `{peer_id, addr, uptime_s, tokens_per_sec}` |
|
||||
| `peer-list` | relay→peer | `{peers: [{peer_id, addr}]}` |
|
||||
| `relay-announce` | relay→all | `{relay_id, relay_url, capacity}` |
|
||||
|
||||
Gossip fanout: each node re-broadcasts received messages to all its peers (simple flooding with `seen_ids` dedup, TTL=3 hops). Migration to GossipSub mesh routing is a later ADR.
|
||||
|
||||
### Peer ID
|
||||
|
||||
`peer_id = sha256(public_key)[:16].hex()` — generated on first run, stored in `~/.config/meshnet/identity.json`. The same keypair is used for TLS client certificates (mTLS) in future work.
|
||||
|
||||
## mDNS LAN discovery
|
||||
|
||||
Use Python `zeroconf` library. Service type: `_meshnet._tcp.local.`
|
||||
|
||||
```python
|
||||
from zeroconf import ServiceInfo, Zeroconf
|
||||
|
||||
info = ServiceInfo(
|
||||
"_meshnet._tcp.local.",
|
||||
f"{peer_id}._meshnet._tcp.local.",
|
||||
addresses=[socket.inet_aton(local_ip)],
|
||||
port=node_port,
|
||||
properties={"peer_id": peer_id, "version": "1"},
|
||||
)
|
||||
zc = Zeroconf()
|
||||
zc.register_service(info)
|
||||
```
|
||||
|
||||
On startup, nodes also browse for `_meshnet._tcp.local.` to discover existing nodes. mDNS is LAN-only (does not traverse routers), which is correct for LAN discovery.
|
||||
|
||||
## NAT traversal: circuit relay
|
||||
|
||||
### How it works
|
||||
|
||||
1. Node A (behind NAT) cannot accept inbound TCP connections
|
||||
2. Node A connects outbound to the public relay via WSS
|
||||
3. Node A tells the tracker: `"effective_addr": "wss://relay.meshnet.ai/relay/{peer_id_A}"`
|
||||
4. Node B (wants to call A) connects to the relay at the above URL
|
||||
5. Relay proxies the TCP stream between A and B
|
||||
|
||||
Hole-punching (direct connection via STUN) is attempted first (future work). Relay is the fallback.
|
||||
|
||||
### meshnet-relay
|
||||
|
||||
`packages/relay/meshnet_relay/server.py` — a standalone aiohttp server:
|
||||
|
||||
```
|
||||
GET /health → {status: ok}
|
||||
GET /v1/peers → [{peer_id, addr, last_seen}]
|
||||
POST /v1/gossip → receive a gossip message, fan out to connected peers
|
||||
WSS /ws → persistent gossip connection (subscribe to all topics)
|
||||
WSS /relay/{peer_id} → circuit relay proxy to that peer_id
|
||||
GET /v1/relay/capacity → {connected_peers: N, max_peers: 500}
|
||||
```
|
||||
|
||||
CLI:
|
||||
|
||||
```
|
||||
meshnet-relay [--port 8443] [--cert path/to/cert.pem] [--key path/to/key.pem]
|
||||
[--tracker-url http://...] [--max-peers 500]
|
||||
```
|
||||
|
||||
The relay can optionally proxy to the tracker (so `relay.meshnet.ai` is the single internet-visible endpoint).
|
||||
|
||||
## SSL/TLS setup
|
||||
|
||||
### Node certificate (self-signed, auto-generated)
|
||||
|
||||
On first run, `meshnet-node` generates a self-signed RSA-2048 cert valid for 10 years:
|
||||
|
||||
```python
|
||||
from cryptography import x509
|
||||
from cryptography.hazmat.primitives import hashes, serialization
|
||||
from cryptography.hazmat.primitives.asymmetric import rsa
|
||||
```
|
||||
|
||||
Cert saved to `~/.config/meshnet/node_cert.pem` + `node_key.pem`. Fingerprint stored in config and shared with tracker via heartbeat. Nodes connecting to each other validate the fingerprint (TOFU — trust on first use), not the CA chain.
|
||||
|
||||
### Relay certificate
|
||||
|
||||
The relay uses a real Let's Encrypt cert (cert-bot or acme.sh). The relay cert is pinned in `packages/p2p/relay_bootstrap.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"relays": [
|
||||
{
|
||||
"url": "wss://relay.meshnet.ai:8443",
|
||||
"cert_fingerprint": "sha256:<hex>",
|
||||
"operator": "meshnet-team"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### All HTTP switched to HTTPS
|
||||
|
||||
`meshnet-node` starts an HTTPS server using `ssl.SSLContext`. `meshnet-tracker` similarly. All outbound `httpx` / `aiohttp` calls use TLS verification against pinned fingerprints (not the system CA store — too many corporate proxies break this).
|
||||
|
||||
## Tracker changes
|
||||
|
||||
Heartbeat payload gains new fields:
|
||||
|
||||
```json
|
||||
{
|
||||
"peer_id": "a1b2c3d4e5f6a1b2",
|
||||
"effective_addr": "https://192.168.1.42:8001",
|
||||
"relay_addr": "wss://relay.meshnet.ai:8443/relay/a1b2c3d4e5f6a1b2",
|
||||
"cert_fingerprint": "sha256:...",
|
||||
"gossip_peers": ["peer_id_1", "peer_id_2"]
|
||||
}
|
||||
```
|
||||
|
||||
Tracker uses `effective_addr` (direct) or `relay_addr` (fallback) when building inference routes.
|
||||
|
||||
## Integration test
|
||||
|
||||
```
|
||||
tests/test_gossip_and_relay.py
|
||||
|
||||
scenario:
|
||||
1. Start a local relay (localhost:18443)
|
||||
2. Start node A (no inbound port — simulate NAT by binding to 127.0.0.1 only)
|
||||
3. Start node B (public-reachable on localhost)
|
||||
4. Both register with relay; relay peer-list includes both
|
||||
5. Node B sends a gossip node-join message
|
||||
6. Assert node A receives it within 500ms
|
||||
7. Start tracker; confirm tracker's node registry includes node A via relay_addr
|
||||
8. Send inference request; assert it routes through relay to node A
|
||||
```
|
||||
|
||||
## Package layout
|
||||
|
||||
```
|
||||
packages/relay/
|
||||
pyproject.toml
|
||||
meshnet_relay/
|
||||
__init__.py
|
||||
server.py # aiohttp relay + gossip hub + circuit relay proxy
|
||||
cli.py # meshnet-relay entrypoint
|
||||
peer_registry.py # in-memory {peer_id: {addr, last_seen, ...}}
|
||||
circuit_relay.py # WSS proxy between two peers
|
||||
|
||||
packages/p2p/
|
||||
meshnet_p2p/
|
||||
gossip.py # GossipClient — connect to relay + peers, pub/sub
|
||||
mdns.py # ZeroconfDiscovery — mDNS announce + browse
|
||||
identity.py # PeerIdentity — generate/load peer_id + keypair
|
||||
tls.py # cert generation, fingerprint, SSLContext helpers
|
||||
|
||||
packages/node/meshnet_node/
|
||||
gossip_integration.py # wires GossipClient into node lifecycle
|
||||
```
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- All node↔node and node↔tracker HTTP uses HTTPS; self-signed cert auto-generated on first run
|
||||
- `cert_fingerprint` included in heartbeat; tracker stores and logs it
|
||||
- mDNS: two nodes on the same LAN discover each other without manual tracker URL (test with two localhost processes using different mDNS names)
|
||||
- Relay: `meshnet-relay` starts, accepts WSS connections, fans out gossip messages to all connected peers
|
||||
- Circuit relay: node A (127.0.0.1-only) can receive a gossip message via the relay from node B
|
||||
- Tracker routes inference to node A using `relay_addr` when direct addr not reachable
|
||||
- `relay_bootstrap.json` exists in `packages/p2p/` with at least one entry (localhost for tests)
|
||||
- ADR-0010 documents the gossip architecture and libp2p migration path
|
||||
- `python -m pytest` passes from repo root
|
||||
- Commit only this story's changes
|
||||
157
docs/issues/18-two-machine-lan-test.md
Normal file
157
docs/issues/18-two-machine-lan-test.md
Normal file
@@ -0,0 +1,157 @@
|
||||
# US-018 — End-to-end two-machine LAN inference test
|
||||
|
||||
## Goal
|
||||
|
||||
Run real distributed inference across two physical machines: the Linux rig and a Windows 11 rig running WSL2. Document every setup step, firewall rule, and gotcha so this is repeatable. The test script exits 0 with token output and timing, proving the network works.
|
||||
|
||||
## Network topology for LAN test
|
||||
|
||||
```
|
||||
[Linux machine] [Windows 11 / WSL2]
|
||||
meshnet-tracker :8080 meshnet-node (shard B)
|
||||
meshnet-node :8001 (shard A, tracker-mode)
|
||||
meshnet-gateway :8000 (optional, for OpenAI-compat)
|
||||
|
||||
Client (either machine):
|
||||
scripts/test_lan_inference.py --tracker http://192.168.1.10:8080
|
||||
```
|
||||
|
||||
The Linux machine runs the tracker + the first-shard node (tracker-mode). The Windows/WSL2 machine runs the second-shard node. A small model (e.g. Phi-3-medium at BF16, fits on one GPU each) is split across both.
|
||||
|
||||
## WSL2 setup (Windows side)
|
||||
|
||||
`docs/INSTALL_WINDOWS.md` covers:
|
||||
|
||||
1. Enable WSL2: `wsl --install -d Ubuntu-24.04`
|
||||
2. CUDA in WSL2: install NVIDIA driver on Windows (NOT inside WSL); WSL2 gets CUDA automatically
|
||||
- Verify: `nvidia-smi` inside WSL2 should show GPU
|
||||
3. Install Python 3.11+ and pip inside WSL2
|
||||
4. `pip install -e packages/node packages/p2p` (clone repo first)
|
||||
5. Firewall: Windows Defender must allow inbound WSL2 → LAN on node port
|
||||
- PowerShell: `New-NetFirewallRule -DisplayName "meshnet-node" -Direction Inbound -Protocol TCP -LocalPort 8001 -Action Allow`
|
||||
6. WSL2 IP: WSL2 has its own NAT'd IP (172.x.x.x); to expose to LAN, either:
|
||||
- Option A: `netsh interface portproxy add v4tov4 listenport=8001 listenaddress=0.0.0.0 connectport=8001 connectaddress=$(wsl hostname -I)`
|
||||
- Option B: use the relay node (US-017) — no port forwarding needed
|
||||
|
||||
## Linux setup
|
||||
|
||||
Standard install (already done after US-016). Firewall:
|
||||
|
||||
```bash
|
||||
# If using ufw
|
||||
sudo ufw allow 8080/tcp # tracker
|
||||
sudo ufw allow 8001/tcp # node
|
||||
sudo ufw allow 8000/tcp # gateway (optional)
|
||||
```
|
||||
|
||||
## Model split
|
||||
|
||||
For the test, use a model that has enough layers to split meaningfully but fits comfortably in memory. Phi-3-medium-128k-instruct (40 layers, BF16 15GB) works on a single 24GB GPU on each machine:
|
||||
|
||||
- Linux node: layers 0–19 (tracker-mode, owns tokenizer + embed_tokens)
|
||||
- Windows/WSL2 node: layers 20–39
|
||||
|
||||
Start sequence:
|
||||
```bash
|
||||
# Terminal 1 (Linux) — tracker
|
||||
meshnet-tracker --port 8080
|
||||
|
||||
# Terminal 2 (Linux) — first-shard node (tracker-mode auto-detected because shard_start=0)
|
||||
meshnet-node --model microsoft/Phi-3-medium-128k-instruct \
|
||||
--quantization bf16 \
|
||||
--shard-start 0 --shard-end 19 \
|
||||
--tracker http://localhost:8080 \
|
||||
--port 8001
|
||||
|
||||
# Terminal 3 (Windows WSL2) — second-shard node
|
||||
meshnet-node --model microsoft/Phi-3-medium-128k-instruct \
|
||||
--quantization bf16 \
|
||||
--shard-start 20 --shard-end 39 \
|
||||
--tracker http://192.168.1.10:8080 \
|
||||
--port 8001
|
||||
```
|
||||
|
||||
## Test script
|
||||
|
||||
`scripts/test_lan_inference.py`:
|
||||
|
||||
```python
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
End-to-end LAN inference test.
|
||||
Usage: python scripts/test_lan_inference.py --tracker http://192.168.1.10:8080
|
||||
"""
|
||||
import argparse, time, httpx, json
|
||||
|
||||
MESSAGES = [
|
||||
{"role": "user", "content": "What is 7 × 8? Answer in one word."},
|
||||
{"role": "user", "content": "Name the capital of France in one word."},
|
||||
{"role": "user", "content": "Complete the sequence: 1, 1, 2, 3, 5, ___"},
|
||||
]
|
||||
|
||||
def run_test(tracker_url: str, gateway_url: str | None):
|
||||
# Discover inference entry point via tracker if gateway not given
|
||||
if not gateway_url:
|
||||
r = httpx.get(f"{tracker_url}/v1/tracker-nodes/phi-3-medium", timeout=5)
|
||||
r.raise_for_status()
|
||||
nodes = r.json()
|
||||
assert nodes, "No tracker-mode nodes registered — is the first-shard node running?"
|
||||
gateway_url = nodes[0]["url"]
|
||||
|
||||
print(f"Inference endpoint: {gateway_url}")
|
||||
print(f"Tracker: {tracker_url}")
|
||||
print()
|
||||
|
||||
for i, msg in enumerate(MESSAGES):
|
||||
t0 = time.monotonic()
|
||||
r = httpx.post(
|
||||
f"{gateway_url}/v1/chat/completions",
|
||||
json={"model": "phi-3-medium", "messages": [msg], "stream": False},
|
||||
timeout=60,
|
||||
)
|
||||
r.raise_for_status()
|
||||
data = r.json()
|
||||
elapsed = time.monotonic() - t0
|
||||
|
||||
content = data["choices"][0]["message"]["content"]
|
||||
tokens = data["usage"]["completion_tokens"]
|
||||
tps = tokens / elapsed if elapsed > 0 else 0
|
||||
|
||||
print(f"[{i+1}] Q: {msg['content']}")
|
||||
print(f" A: {content}")
|
||||
print(f" {tokens} tokens {elapsed:.2f}s {tps:.1f} t/s")
|
||||
print()
|
||||
|
||||
print("✓ All 3 requests completed successfully")
|
||||
|
||||
if __name__ == "__main__":
|
||||
p = argparse.ArgumentParser()
|
||||
p.add_argument("--tracker", required=True)
|
||||
p.add_argument("--gateway", default=None)
|
||||
args = p.parse_args()
|
||||
run_test(args.tracker, args.gateway)
|
||||
```
|
||||
|
||||
## Docs: TWO_MACHINE_TEST.md
|
||||
|
||||
`docs/TWO_MACHINE_TEST.md` must cover:
|
||||
|
||||
1. Prerequisites (models downloaded on both machines, same model ID, complementary shard ranges)
|
||||
2. Start order: tracker first, then nodes, then test script
|
||||
3. How to verify nodes are registered: `GET /v1/nodes` on tracker
|
||||
4. How to verify coverage: `GET /v1/coverage/phi-3-medium` — all 40 layers must show node_count ≥ 1
|
||||
5. How to run the test script
|
||||
6. Expected output
|
||||
7. Latency breakdown: how to read per-hop latency from node logs
|
||||
8. **Known Issues** section — updated during actual test run with real gotchas
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- `docs/INSTALL_WINDOWS.md` covers WSL2 + CUDA + meshnet-node install end-to-end
|
||||
- `docs/TWO_MACHINE_TEST.md` covers the full two-machine setup and test procedure
|
||||
- `scripts/test_lan_inference.py` exists and is executable
|
||||
- When run against a real two-machine LAN setup: script exits 0, prints 3 valid answers with timing
|
||||
- Coverage map shows 100% coverage (no gap) after both nodes register
|
||||
- Known Issues section in TWO_MACHINE_TEST.md contains at least the issues encountered during this test run
|
||||
- No new pytest failures from repo root (this story adds docs + a script, not new Python packages)
|
||||
- Commit only this story's changes
|
||||
68
docs/issues/19-binary-data-plane-and-peer-weight-transfer.md
Normal file
68
docs/issues/19-binary-data-plane-and-peer-weight-transfer.md
Normal file
@@ -0,0 +1,68 @@
|
||||
# US-019 — Binary data plane and optional peer weight transfer
|
||||
|
||||
Status: needs-triage
|
||||
Priority: Low
|
||||
Stage: Design parking lot
|
||||
|
||||
## Context
|
||||
|
||||
The current project focus is inference democratization: let small GPU owners contribute useful compute and let users run inference on models larger than one host can serve alone. Weight distribution is useful, but it is secondary to low-latency distributed inference.
|
||||
|
||||
Recent findings:
|
||||
|
||||
- HuggingFace already handles initial model origin distribution well enough for the first working version.
|
||||
- The inference-critical path is activation transfer between shard nodes, not torrenting model files.
|
||||
- Torrent/content-addressed transfer is a good future fit for model weights, shard cache replication, fine-tuned models, and offline/local swarm behavior.
|
||||
- Torrenting is not a good fit for activation traffic because activations are latency-sensitive, ordered, session-specific binary streams.
|
||||
|
||||
## Design note
|
||||
|
||||
Keep the tracker as the control plane:
|
||||
|
||||
- node registration
|
||||
- heartbeats
|
||||
- route selection
|
||||
- model/shard manifests
|
||||
- peer/checksum metadata
|
||||
|
||||
Keep binary payloads on the data plane:
|
||||
|
||||
- direct node-to-node activation transfer where reachable
|
||||
- relay/QUIC/WSS fallback where direct transport is unavailable
|
||||
- future peer weight transfer as content-addressed blobs or pieces
|
||||
|
||||
## Potential future direction
|
||||
|
||||
For inference traffic:
|
||||
|
||||
- Prefer direct binary transport with backpressure.
|
||||
- Use raw binary activation frames rather than JSON/base64.
|
||||
- Preserve tensor metadata out-of-band: shape, dtype, session, chunk index, encoding.
|
||||
- Consider QUIC or a mature NAT-friendly transport for direct-when-possible, relay-when-needed behavior.
|
||||
|
||||
For model weights:
|
||||
|
||||
- Keep HuggingFace as default origin and fallback.
|
||||
- Add peer cache transfer only as an optimization.
|
||||
- Consider a real content-addressed/torrent-like library or sidecar for weight blobs.
|
||||
- Store manifests and checksums in tracker state so peers can verify exact shard contents.
|
||||
- Prefer piece/chunk transfer with resume and hash verification over one giant tarball.
|
||||
|
||||
## Not in scope now
|
||||
|
||||
- Replacing HuggingFace as the primary model origin.
|
||||
- Building a full BitTorrent/IPFS/libp2p subsystem.
|
||||
- Routing activation traffic through a torrent protocol.
|
||||
- Making peer weight transfer mandatory for node startup.
|
||||
|
||||
## Acceptance criteria for a future implementation issue
|
||||
|
||||
- [ ] Activation transfer remains binary end-to-end and avoids JSON/base64 payloads.
|
||||
- [ ] Tracker does not proxy large binary payloads except as an explicit fallback path.
|
||||
- [ ] Weight transfer, if added, is optional and falls back to HuggingFace.
|
||||
- [ ] Weight pieces are content-addressed and checksum-verified.
|
||||
- [ ] The design preserves low-latency inference as the primary objective.
|
||||
|
||||
## Comments
|
||||
|
||||
Created as a low-priority design parking-lot item after discussing inference democratization versus weight distribution. Do not pick up for implementation until the core public tracker, relay, and binary activation path are stable.
|
||||
@@ -0,0 +1,66 @@
|
||||
Status: ready-for-agent
|
||||
|
||||
# US-020 - Memory budget, shard slots, and dropout relocation hardening
|
||||
|
||||
## Goal
|
||||
|
||||
Make node capacity limits explicit and enforce them consistently when the tracker assigns, rebalances, and relocates shards after a node dropout.
|
||||
|
||||
This is a follow-up to US-013, not a replacement. US-013 owns the coverage-first assignment and rebalance algorithm. This issue hardens the capacity contract around that algorithm: operator memory budget, maximum loaded shard slots, and relocation behavior when one node must absorb or split ranges after another node disappears.
|
||||
|
||||
## Context
|
||||
|
||||
Recent work added the first part of the contract:
|
||||
|
||||
- `meshnet-node --memory MB` is registered with the tracker as `vram_bytes` when explicitly set.
|
||||
- CPU nodes without `--memory` keep the tracker default capacity, preserving old behavior.
|
||||
- `meshnet-node --max-shards N` is accepted and registered as `max_loaded_shards`.
|
||||
- Tracker registration validates `max_loaded_shards >= 1`.
|
||||
|
||||
The current runtime still effectively has one active backend shard per node. A node may advertise `max_loaded_shards`, but the tracker does not yet use multiple shard slots in bin-packing, and the node does not yet host multiple concurrently loaded shard ranges.
|
||||
|
||||
## Scope
|
||||
|
||||
- Make tracker rebalance logic account for `max_loaded_shards` as a capacity multiplier or explicit shard-slot list.
|
||||
- Ensure a node is never assigned more total layers than its memory budget can support across all loaded shard slots.
|
||||
- Decide and implement the runtime behavior for multiple loaded shards:
|
||||
- either support multiple concurrently loaded shard backends on one node, or
|
||||
- keep one backend active and treat `max_loaded_shards` as future metadata, with tracker enforcement preventing multi-range assignment for now.
|
||||
- On heartbeat timeout, relocate the dropped node's uncovered layer range to eligible managed nodes while respecting both memory and shard-slot limits.
|
||||
- Surface the effective memory budget and shard slot count in tracker/network inspection output so operators can diagnose why a node did or did not receive a range.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Do not redesign the US-013 coverage-first algorithm from scratch.
|
||||
- Do not change relay, `/ws`, or `/rpc` behavior.
|
||||
- Do not change the token/reward model.
|
||||
- Do not require public internet verification; all behavior must be locally testable.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- Tracker stores and exposes `max_loaded_shards` for registered nodes.
|
||||
- Assignment/rebalance never exceeds:
|
||||
- `assigned_layers_total <= floor((vram_bytes * 0.8) / bytes_per_layer_at_quant)`
|
||||
- `assigned_range_count <= max_loaded_shards`
|
||||
- A managed node with `max_loaded_shards=1` only receives one active shard range.
|
||||
- A managed node with `max_loaded_shards=2` can absorb two non-contiguous uncovered ranges only if the node runtime supports serving both; otherwise tracker must keep assigning at most one range and document `max_loaded_shards` as reserved.
|
||||
- Dropout test: register nodes covering a model, let a middle/tail node heartbeat-expire, and assert the tracker queues `LOAD_SHARD` directives that restore full coverage without violating memory or shard-slot limits.
|
||||
- CLI test: `--memory` and `--max-shards` are reflected in the registration payload.
|
||||
- `python -m pytest tests/test_tracker_routing.py tests/test_node_startup.py` passes in the project virtualenv, aside from any pre-existing platform-specific wallet permission assertion documented in the final notes.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
- Existing files likely involved:
|
||||
- `packages/node/meshnet_node/cli.py`
|
||||
- `packages/node/meshnet_node/startup.py`
|
||||
- `packages/node/meshnet_node/torch_server.py`
|
||||
- `packages/tracker/meshnet_tracker/server.py`
|
||||
- `tests/test_tracker_routing.py`
|
||||
- `tests/test_node_startup.py`
|
||||
- Keep backward compatibility: nodes that omit `vram_bytes` default to tracker defaults; nodes that omit `max_loaded_shards` default to `1`.
|
||||
- Prefer a small internal representation for assigned ranges if multiple ranges become real, for example `assigned_shards: list[tuple[int, int]]`, while preserving `shard_start`/`shard_end` in public responses for single-range nodes.
|
||||
|
||||
## Comments
|
||||
|
||||
- 2026-06-30: Created after implementing the initial registration plumbing in commit `f1e4ed6` (`--memory`, `--max-shards`, tracker validation). This issue captures the remaining end-to-end behavior so it does not conflict with US-013.
|
||||
- 2026-06-30: Implementation decision: `max_loaded_shards` is currently a validated and exposed capacity field, but multi-range assignment remains reserved because `TorchNodeServer` serves one active backend shard. The tracker therefore emits at most one active range per node while exposing `vram_bytes`, `ram_bytes`, `max_loaded_shards`, quantization, throughput, and computed `max_assignable_layers` in inspection endpoints.
|
||||
20
docs/issues/20-tracker-node-hardening.md
Normal file
20
docs/issues/20-tracker-node-hardening.md
Normal file
@@ -0,0 +1,20 @@
|
||||
# US-020 — Tracker + node hardening: BrokenPipe fix, deterministic node IDs, HF coverage
|
||||
|
||||
Status: done
|
||||
Priority: High
|
||||
Stage: Maintenance
|
||||
|
||||
## Context
|
||||
|
||||
First two-machine LAN test (US-018) exposed three reliability issues:
|
||||
|
||||
1. `BrokenPipeError` crash in tracker `_send_json` when a slow-inference client disconnected mid-response
|
||||
2. Random UUID node IDs meant every re-registration (after tracker restart) created a phantom entry
|
||||
3. `GET /v1/coverage/<model>` returned no results when called with a short name (`Qwen2.5-0.5B`) instead of the full HF repo ID
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- [ ] `BrokenPipeError` in tracker and node `_send_json` is silently swallowed
|
||||
- [ ] Node IDs are deterministic: `sha256(wallet_address + str(port))[:16]`
|
||||
- [ ] `GET /v1/coverage/<model>` accepts both short names and full `owner/repo` IDs
|
||||
- [ ] `python -m pytest` passes from repo root
|
||||
19
docs/issues/21-route-timeout-flag.md
Normal file
19
docs/issues/21-route-timeout-flag.md
Normal file
@@ -0,0 +1,19 @@
|
||||
# US-021 — `--route-timeout` CLI flag for node tracker route lookup
|
||||
|
||||
Status: done
|
||||
Priority: Medium
|
||||
Stage: Implemented
|
||||
|
||||
## Context
|
||||
|
||||
The node's slow-path tracker route lookup (`/v1/route`) used a hard-coded 30-second HTTP timeout.
|
||||
On high-latency links (relay, satellite, 5G) or when the tracker is under load, legitimate route
|
||||
lookups were failing prematurely. The timeout is deployment-specific and should be tunable.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- [ ] `meshnet-node start` accepts `--route-timeout <seconds>` (float, default 30.0)
|
||||
- [ ] Value is passed through to `TorchNodeServer` and used in the `/v1/route` HTTP call
|
||||
- [ ] `TorchNodeServer` exposes `route_timeout` as a readable property
|
||||
- [ ] Test: setting `--route-timeout 45` is reflected as `45.0` on the running server object
|
||||
- [ ] `python -m pytest` passes
|
||||
29
docs/issues/22-start-layer-protocol.md
Normal file
29
docs/issues/22-start-layer-protocol.md
Normal file
@@ -0,0 +1,29 @@
|
||||
# US-022 — X-Meshnet-Start-Layer: overlapping shard execution protocol
|
||||
|
||||
Status: done
|
||||
Priority: High
|
||||
Stage: Implemented
|
||||
|
||||
## Context
|
||||
|
||||
Two nodes may register overlapping shard ranges (node A: 0–15, node B: 12–23) to increase
|
||||
redundancy or to enable partial-model nodes to form a complete route. Without coordination,
|
||||
node B would re-run layers 12–15 on top of activations that already include them, producing
|
||||
wrong output.
|
||||
|
||||
The `X-Meshnet-Start-Layer` header tells each downstream node which model layer the incoming
|
||||
activation tensor represents, so the node skips layers it has already been told to skip.
|
||||
|
||||
## Decision
|
||||
|
||||
Option A: tracker injects `start_layer` into `X-Meshnet-Route` hops at proxy time. The head
|
||||
node passes it per-hop as `X-Meshnet-Start-Layer`. No peer-to-peer negotiation needed.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- [ ] Tracker `_handle_proxy_chat` builds route hops with `start_layer` = `covered_up_to + 1`
|
||||
- [ ] `_handle_binary_forward` reads `X-Meshnet-Start-Layer` and passes it to `backend.forward_bytes`
|
||||
- [ ] `_get_remaining_route` parses `start_layer` from injected header and from `/v1/route` slow-path
|
||||
- [ ] `TorchModelShard.forward_bytes` accepts optional `start_layer` and skips layers below it
|
||||
- [ ] Test: overlapping two-node route produces correct output without double-computing layers
|
||||
- [ ] `python -m pytest` passes
|
||||
26
docs/issues/23-heartbeat-stats.md
Normal file
26
docs/issues/23-heartbeat-stats.md
Normal file
@@ -0,0 +1,26 @@
|
||||
# US-023 — Heartbeat stats payload: request counters + dynamic reassignment response
|
||||
|
||||
Status: done
|
||||
Priority: Medium
|
||||
Stage: Implemented
|
||||
|
||||
## Context
|
||||
|
||||
Node heartbeats are currently empty POSTs. The tracker has no visibility into per-node load,
|
||||
making load balancing and assignment decisions blind. Heartbeats should carry cumulative stats.
|
||||
The heartbeat response channel is also the natural place for the tracker to deliver reassignment
|
||||
instructions without requiring a node restart.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- [ ] Heartbeat POST body includes: `total_requests`, `failed_requests`, `queue_depth`, `uptime_seconds`, `status`
|
||||
- [ ] `TorchNodeServer` tracks the three counters with a `threading.Lock`
|
||||
- [ ] Tracker stores the last heartbeat payload per node
|
||||
- [ ] Heartbeat response may include `new_assignment: {model, shard_start, shard_end}`; node logs it
|
||||
- [ ] Stats survive tracker outage: buffered locally, flushed on next successful heartbeat
|
||||
- [ ] `python -m pytest` passes
|
||||
|
||||
## Notes
|
||||
|
||||
Hot-reload (loading a new shard without restart) is deferred to a future story. The response
|
||||
field is wired so trackers can send the signal; nodes log it but don't act yet.
|
||||
24
docs/issues/24-availability-map.md
Normal file
24
docs/issues/24-availability-map.md
Normal file
@@ -0,0 +1,24 @@
|
||||
# US-024 — Enhanced availability map with per-node health details
|
||||
|
||||
Status: done
|
||||
Priority: Medium
|
||||
Stage: Implemented
|
||||
|
||||
## Context
|
||||
|
||||
`GET /v1/coverage/<model>` returns band-level coverage (start_layer, end_layer, node_count)
|
||||
but gives no visibility into which specific nodes are in each band or whether they are alive.
|
||||
Operators debugging a split-model deployment need to see node-level health at a glance.
|
||||
|
||||
## Format decision
|
||||
|
||||
Both: band metadata + node list grouped under each band. Dead nodes included with `healthy: false`
|
||||
so operators can see them.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- [ ] Each band in the response includes `nodes: [{node_id, endpoint, healthy, queue_depth, last_seen_s}]`
|
||||
- [ ] `healthy` is `true` iff last heartbeat < `heartbeat_timeout` seconds ago
|
||||
- [ ] Existing band fields (`start_layer`, `end_layer`, `node_count`) preserved
|
||||
- [ ] Tests updated to check band fields individually (not exact dict comparison)
|
||||
- [ ] `python -m pytest` passes
|
||||
30
docs/issues/25-rolling-rpm-stats.md
Normal file
30
docs/issues/25-rolling-rpm-stats.md
Normal file
@@ -0,0 +1,30 @@
|
||||
# US-025 — Model usage statistics: rolling RPM windows + SQLite persistence
|
||||
|
||||
Status: done
|
||||
Priority: Medium
|
||||
Stage: Implemented
|
||||
|
||||
## Context
|
||||
|
||||
Trackers need per-model request-rate data to drive smart shard assignment (US-026) and
|
||||
to give operators visibility into what the network is actually serving. Stats must survive
|
||||
tracker restarts and should be shareable across a tracker cluster.
|
||||
|
||||
## Design
|
||||
|
||||
- `_RollingCounter`: circular-bucket counter, epoch-indexed, stale buckets auto-reset
|
||||
- Three windows: 60×1-min buckets (last hour), 24×1-hr (last day), 30×1-day (last month)
|
||||
- `_StatsCollector`: `record_request()`, `get_local_rpms()`, `merge_peer_rpms()`, `get_combined_stats()`
|
||||
- SQLite persistence via `--stats-db PATH`
|
||||
- Gossip: each tracker keeps its own slice; merge is additive (not averaged)
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- [ ] `_RollingCounter` passes unit tests (record, rpm, stale bucket reset)
|
||||
- [ ] `_StatsCollector` accumulates and merges peer slices additively
|
||||
- [ ] SQLite round-trip: buckets saved and restored across restart
|
||||
- [ ] `GET /v1/stats` returns combined stats JSON
|
||||
- [ ] `POST /v1/stats/gossip` accepts peer slice and merges
|
||||
- [ ] `_handle_proxy_chat` calls `stats.record_request()` after model is resolved
|
||||
- [ ] `--stats-db PATH` CLI flag added to tracker
|
||||
- [ ] 6 unit tests pass; `python -m pytest` passes
|
||||
32
docs/issues/26-smart-model-assignment.md
Normal file
32
docs/issues/26-smart-model-assignment.md
Normal file
@@ -0,0 +1,32 @@
|
||||
# US-026 — Smart model assignment via demand×coverage scoring
|
||||
|
||||
Status: done
|
||||
Priority: Medium
|
||||
Stage: Implemented
|
||||
|
||||
## Context
|
||||
|
||||
`/v1/network/assign` currently picks the model with the largest uncovered shard gap,
|
||||
ignoring traffic. A model serving 1000 RPM at 60% coverage is far more valuable to fill
|
||||
than a zero-traffic model at 50% coverage.
|
||||
|
||||
## Scoring formula
|
||||
|
||||
```
|
||||
score = (demand_rpm + 1.0) × (coverage_deficit + 0.01)
|
||||
```
|
||||
|
||||
- `demand_rpm`: combined RPM from `_StatsCollector.get_combined_stats()`
|
||||
- `coverage_deficit`: fraction of model layers with zero node coverage, in [0.0, 1.0]
|
||||
- `+1.0` floor: models with no traffic still compete by coverage
|
||||
- `+0.01` floor: fully-covered models still have a non-zero score if they have traffic
|
||||
|
||||
`price_per_token: 0.0` reserved in the response for future billing integration.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- [ ] `_handle_network_assign` computes score per model and returns the highest
|
||||
- [ ] Demand uses combined stats (local + peer slices)
|
||||
- [ ] `price_per_token: 0.0` present in response
|
||||
- [ ] Test: high-demand low-coverage model beats low-demand high-coverage model
|
||||
- [ ] `python -m pytest` passes
|
||||
28
docs/issues/27-throughput-routing.md
Normal file
28
docs/issues/27-throughput-routing.md
Normal file
@@ -0,0 +1,28 @@
|
||||
# US-027 — Throughput-optimized routing: effective throughput as tiebreak
|
||||
|
||||
Status: done
|
||||
Priority: Medium
|
||||
Stage: Implemented
|
||||
|
||||
## Context
|
||||
|
||||
The greedy max-reach route selection picks nodes by shard coverage but ignores node speed.
|
||||
When two nodes cover the same remaining layer range, we should prefer the faster one.
|
||||
This is a tiebreak only — coverage maximization remains the primary objective.
|
||||
|
||||
## Effective throughput formula
|
||||
|
||||
```
|
||||
effective_throughput = benchmark_tokens_per_sec / (queue_depth + 1)
|
||||
```
|
||||
|
||||
`benchmark_tokens_per_sec` comes from the hardware profile at registration time.
|
||||
`queue_depth` comes from the last heartbeat.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- [ ] `_effective_throughput(node)` helper in `server.py`
|
||||
- [ ] `_select_route` uses throughput as tiebreak when `shard_end` is equal
|
||||
- [ ] Test: two nodes, same shard range, different throughput → faster node selected
|
||||
- [ ] Existing coverage tests still pass unchanged
|
||||
- [ ] `python -m pytest` passes
|
||||
26
docs/issues/28-routing-correctness-tests.md
Normal file
26
docs/issues/28-routing-correctness-tests.md
Normal file
@@ -0,0 +1,26 @@
|
||||
# US-028 — Routing correctness tests: three-node, overlap, and throughput scenarios
|
||||
|
||||
Status: done
|
||||
Priority: Medium
|
||||
Stage: Implemented
|
||||
|
||||
## Context
|
||||
|
||||
Route selection logic (`_select_route`) is the core of the inference network. Without
|
||||
locked-down tests, routing regressions are silent. This story adds a comprehensive
|
||||
scenario suite that locks in the routing contract.
|
||||
|
||||
## Test scenarios
|
||||
|
||||
1. **No-overlap three nodes**: greedy picks in layer-start order (A→C→B for ranges 0–7, 8–15, 16–23)
|
||||
Note: the algorithm picks by earliest uncovered layer, not by node label — so if C.shard_start < B.shard_start, C comes first.
|
||||
2. **Overlapping shards**: correct resolution without double-computing layers
|
||||
3. **Throughput tiebreak**: faster node wins when shard_end is equal
|
||||
4. **Gap detection**: partial coverage returns a 503 error with description
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- [ ] 7 routing tests covering the above scenarios
|
||||
- [ ] Tests use in-process tracker (no mocking)
|
||||
- [ ] `_make_node` helper for concise test setup
|
||||
- [ ] All tests pass; `python -m pytest` passes
|
||||
44
docs/issues/29-relay-outbound-client.md
Normal file
44
docs/issues/29-relay-outbound-client.md
Normal file
@@ -0,0 +1,44 @@
|
||||
# US-029 — Outbound relay client: NAT/internet pipeline hops
|
||||
|
||||
Status: done
|
||||
Priority: Critical
|
||||
Stage: Implemented
|
||||
|
||||
## Context
|
||||
|
||||
Nodes behind NAT (WSL2 with 172.x.x.x addresses, 5G mobile, home routers) register with the
|
||||
tracker and include a `relay_addr` (`wss://relay/rpc/{peer_id}`). When the head node needs to
|
||||
forward activations to such a peer, it currently fails because the direct HTTP endpoint is
|
||||
unreachable.
|
||||
|
||||
The relay server (US-017) is already running and the node already opens a persistent outbound
|
||||
WebSocket (`RelayHttpBridge`). What is missing is the *outbound caller side*: given a `relay_addr`,
|
||||
open a per-hop WebSocket to the relay's `/rpc/{peer_id}` endpoint and send the activation through it.
|
||||
|
||||
## Protocol
|
||||
|
||||
```
|
||||
Node A → WS connect wss://relay/rpc/{peer_id_B}
|
||||
→ send JSON: {request_id, method, path, headers, body_base64}
|
||||
Relay → forward as relay-http-request envelope to Node B's persistent WS
|
||||
Node B → process /forward locally
|
||||
→ send relay-http-response envelope back
|
||||
Relay → resolve future, send response JSON to Node A
|
||||
Node A ← {request_id, status, headers, body_base64}
|
||||
```
|
||||
|
||||
Binary activations (bfloat16) are base64-encoded. No precision loss.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- [ ] `_relay_hop(relay_addr, path, body, headers, timeout)` in `torch_server.py` — opens WS, sends, receives, returns `(status, headers_lower, body_bytes)`
|
||||
- [ ] `_get_remaining_route` returns `list[dict]` with `relay_addr` field (was `list[tuple]`)
|
||||
- [ ] `_run_downstream_pipeline` dispatches via `_relay_hop` when hop has `relay_addr`; falls back to direct HTTP if relay connection fails
|
||||
- [ ] Tracker `_handle_proxy_chat` includes `relay_addr` in downstream hop dicts when node has one
|
||||
- [ ] `relay_bridge._handle_request` decodes `body_base64`; response uses `body_base64` for `octet-stream` content
|
||||
- [ ] All 157 tests pass (`python -m pytest`)
|
||||
- [ ] QUICKSTART.md updated with relay NAT/internet architecture and test scenario
|
||||
|
||||
## WSL2 test scenario
|
||||
|
||||
Start two nodes in WSL2 pointing at the public tracker. Both get `172.x.x.x` endpoints (unreachable from outside). Both connect to relay automatically. Send an inference request through the tracker — activations flow via relay. No `--advertise-host` needed.
|
||||
Reference in New Issue
Block a user