- [ ] QUICKSTART.md updated with relay NAT/internet architecture and test scenario
## WSL2 test scenario
Start two nodes in WSL2 pointing at the public tracker. Both get `172.x.x.x` endpoints (unreachable from outside). Both connect to relay automatically. Send an inference request through the tracker — activations flow via relay. No `--advertise-host` needed.
"description":"Stand up the monorepo package layout and prove that a client HTTP request can travel end-to-end through the stack \u2014 gateway \u2192 one node serving all layers \u2192 valid response \u2014 before any real model or distributed logic is wired in. The six top-level packages should exist with minimal stubs: - `packages/node` \u2014 node client CLI (`meshnet-node`) - `packages/gateway` \u2014 HTTP gateway + route orchestration - `packages/tracker` \u2014 node registry and route selection - `packages/sdk` \u2014 `meshnet` Python SDK - `packages/contracts` \u2014 Solana smart contract wrappers - `packages/p2p` \u2014 gossip and shard swarm The smoke test uses a tiny in-process stub model (not a real LLM) so the test is fast and has no external dependencies. The node runs in the same process as the test. The gateway is a real HTTP server. The client sends a real `POST /v1/chat/completions` request and receives a valid OpenAI-format response.",
"description":"Stand up the monorepo package layout and prove that a client HTTP request can travel end-to-end through the stack — gateway → one node serving all layers → valid response — before any real model or distributed logic is wired in. The six top-level packages should exist with minimal stubs: - `packages/node` — node client CLI (`meshnet-node`) - `packages/gateway` — HTTP gateway + route orchestration - `packages/tracker` — node registry and route selection - `packages/sdk` — `meshnet` Python SDK - `packages/contracts` — Solana smart contract wrappers - `packages/p2p` — gossip and shard swarm The smoke test uses a tiny in-process stub model (not a real LLM) so the test is fast and has no external dependencies. The node runs in the same process as the test. The gateway is a real HTTP server. The client sends a real `POST /v1/chat/completions` request and receives a valid OpenAI-format response.",
"acceptanceCriteria":[
"acceptanceCriteria":[
"`pip install -e packages/node packages/gateway packages/tracker` works from repo root",
"`pip install -e packages/node packages/gateway packages/tracker` works from repo root",
"`meshnet-node`, `meshnet-gateway`, and `meshnet-tracker` CLI entry points exist",
"`meshnet-node`, `meshnet-gateway`, and `meshnet-tracker` CLI entry points exist",
@@ -28,14 +28,14 @@
},
},
{
{
"id":"US-002",
"id":"US-002",
"title":"02 \u2014 Two-node shard pipeline",
"title":"02 — Two-node shard pipeline",
"description":"Extend the single-node smoke test so that inference is routed through exactly two nodes, each serving half of a stub model's layers. Activation tensors must travel from the gateway to node A, then from node A to node B, and the final output returns to the gateway. This proves the distributed shard pipeline works end-to-end before real models or a real tracker are introduced. The two nodes run in the same process as the test (no real network required). The gateway hardcodes the two-node route for now \u2014 dynamic route selection comes in issue 03. The activation tensor protocol (shape, dtype, serialisation format) must be established here, as all later issues depend on it. Use the domain vocabulary from `CONTEXT.md`: the ordered list of nodes is an **inference route**; each node's layer range is a **shard**; the tensors passed between nodes are activations (not \"data\" or \"chunks\").",
"description":"Extend the single-node smoke test so that inference is routed through exactly two nodes, each serving half of a stub model's layers. Activation tensors must travel from the gateway to node A, then from node A to node B, and the final output returns to the gateway. This proves the distributed shard pipeline works end-to-end before real models or a real tracker are introduced. The two nodes run in the same process as the test (no real network required). The gateway hardcodes the two-node route for now — dynamic route selection comes in issue 03. The activation tensor protocol (shape, dtype, serialisation format) must be established here, as all later issues depend on it. Use the domain vocabulary from `CONTEXT.md`: the ordered list of nodes is an **inference route**; each node's layer range is a **shard**; the tensors passed between nodes are activations (not \"data\" or \"chunks\").",
"acceptanceCriteria":[
"acceptanceCriteria":[
"The integration test spins up two stub nodes, each configured with a non-overlapping shard range",
"The integration test spins up two stub nodes, each configured with a non-overlapping shard range",
"A `POST /v1/chat/completions` request results in activation tensors flowing node-A \u2192 node-B (verifiable via test assertions or logs)",
"A `POST /v1/chat/completions` request results in activation tensors flowing node-A → node-B (verifiable via test assertions or logs)",
"The gateway assembles the final response correctly from node-B's output",
"The gateway assembles the final response correctly from node-B's output",
"The test passes with no external services running",
"The test passes with no external services running",
"The activation tensor serialisation format is documented in a short inline comment (shape, dtype, wire format) \u2014 this becomes the contract all future nodes implement",
"The activation tensor serialisation format is documented in a short inline comment (shape, dtype, wire format) — this becomes the contract all future nodes implement",
"Follow TDD where code is added: add/adjust an observable behavior test before implementation when practical",
"Follow TDD where code is added: add/adjust an observable behavior test before implementation when practical",
"Run the task-specific tests and ensure they pass",
"Run the task-specific tests and ensure they pass",
"Run `python -m pytest` from repo root and ensure it passes, or document why no test suite exists yet",
"Run `python -m pytest` from repo root and ensure it passes, or document why no test suite exists yet",
"description":"Replace the hardcoded two-node route from issue 02 with a real tracker. Nodes register themselves with the tracker on startup (endpoint, shard range, hardware profile, node score). The gateway queries the tracker for an inference route for a given model preset instead of using a static list. The tracker returns an ordered list of node endpoints whose shards collectively cover all layers. The tracker runs as a lightweight HTTP service. Node score is initially a simple placeholder (e.g. fixed value or random) \u2014 real throughput/latency scoring comes later. The tracker must correctly reject route requests when no registered nodes cover a required shard range and return an appropriate error. This issue establishes the tracker's registration and routing API contract, which issues 04 (node startup) and 05 (OpenAI gateway) both depend on.",
"description":"Replace the hardcoded two-node route from issue 02 with a real tracker. Nodes register themselves with the tracker on startup (endpoint, shard range, hardware profile, node score). The gateway queries the tracker for an inference route for a given model preset instead of using a static list. The tracker returns an ordered list of node endpoints whose shards collectively cover all layers. The tracker runs as a lightweight HTTP service. Node score is initially a simple placeholder (e.g. fixed value or random) — real throughput/latency scoring comes later. The tracker must correctly reject route requests when no registered nodes cover a required shard range and return an appropriate error. This issue establishes the tracker's registration and routing API contract, which issues 04 (node startup) and 05 (OpenAI gateway) both depend on.",
"acceptanceCriteria":[
"acceptanceCriteria":[
"A node can register with the tracker via HTTP, providing its endpoint, shard range, and a hardware profile",
"A node can register with the tracker via HTTP, providing its endpoint, shard range, and a hardware profile",
"The gateway queries the tracker with a model preset name and receives an ordered list of node endpoints forming a complete inference route",
"The gateway queries the tracker with a model preset name and receives an ordered list of node endpoints forming a complete inference route",
"description":"Make `meshnet-node start` self-configuring from scratch on a machine with a CUDA-capable GPU (or CPU fallback). The full startup sequence must complete without any manual configuration: 1. Detect GPU model and available VRAM 2. Load an existing Solana wallet from disk, or generate and save a new one 3. Query the tracker for the optimal shard assignment given the hardware profile 4. Download the assigned shard from HuggingFace (`huggingface_hub`) 5. Register with the tracker (wallet address, endpoint, shard range, hardware profile) 6. Begin accepting inference connections The node prints a short status summary on startup: wallet address, assigned shard, model preset, and download progress. No interactive prompts \u2014 the entire flow is non-interactive. This is the primary viral growth vector. Every second of friction in this flow costs node operators. The startup sequence must work on Linux with CUDA, and degrade gracefully to CPU on machines without a GPU.",
"description":"Make `meshnet-node start` self-configuring from scratch on a machine with a CUDA-capable GPU (or CPU fallback). The full startup sequence must complete without any manual configuration: 1. Detect GPU model and available VRAM 2. Load an existing Solana wallet from disk, or generate and save a new one 3. Query the tracker for the optimal shard assignment given the hardware profile 4. Download the assigned shard from HuggingFace (`huggingface_hub`) 5. Register with the tracker (wallet address, endpoint, shard range, hardware profile) 6. Begin accepting inference connections The node prints a short status summary on startup: wallet address, assigned shard, model preset, and download progress. No interactive prompts — the entire flow is non-interactive. This is the primary viral growth vector. Every second of friction in this flow costs node operators. The startup sequence must work on Linux with CUDA, and degrade gracefully to CPU on machines without a GPU.",
"acceptanceCriteria":[
"acceptanceCriteria":[
"`meshnet-node start --tracker http://localhost:8080` completes startup without prompts on a machine with a CUDA GPU",
"`meshnet-node start --tracker http://localhost:8080` completes startup without prompts on a machine with a CUDA GPU",
"A Solana wallet keypair is generated and saved to `~/.config/meshnet/wallet.json` if none exists",
"A Solana wallet keypair is generated and saved to `~/.config/meshnet/wallet.json` if none exists",
@@ -107,8 +107,8 @@
},
},
{
{
"id":"US-005",
"id":"US-005",
"title":"05 \u2014 OpenAI-compatible gateway",
"title":"05 — OpenAI-compatible gateway",
"description":"Expose a production-shape HTTP API from the gateway so that any client using the OpenAI Python SDK (or any OpenAI-compatible tool) works by changing only the `base_url`. The gateway translates OpenAI-format requests into inference route execution and streams responses back in OpenAI-format. Endpoints to implement: - `POST /v1/chat/completions` \u2014 streaming (`text/event-stream`) and non-streaming - `GET /v1/models` \u2014 returns the list of model presets currently routable on the network - `GET /v1/health` \u2014 liveness check The gateway selects an inference route from the tracker, executes the shard pipeline, and assembles the streamed response. If the tracker returns no route for the requested model, the gateway responds with a standard OpenAI-format error (`model_not_available`). Authentication (API key \u2192 SOL/USDC balance) is a stub in this issue \u2014 return 200 for any non-empty `Authorization` header. Real payment gating comes in issue 06.",
"description":"Expose a production-shape HTTP API from the gateway so that any client using the OpenAI Python SDK (or any OpenAI-compatible tool) works by changing only the `base_url`. The gateway translates OpenAI-format requests into inference route execution and streams responses back in OpenAI-format. Endpoints to implement: - `POST /v1/chat/completions` — streaming (`text/event-stream`) and non-streaming - `GET /v1/models` — returns the list of model presets currently routable on the network - `GET /v1/health` — liveness check The gateway selects an inference route from the tracker, executes the shard pipeline, and assembles the streamed response. If the tracker returns no route for the requested model, the gateway responds with a standard OpenAI-format error (`model_not_available`). Authentication (API key → SOL/USDC balance) is a stub in this issue — return 200 for any non-empty `Authorization` header. Real payment gating comes in issue 06.",
"acceptanceCriteria":[
"acceptanceCriteria":[
"`openai.OpenAI(base_url=\"http://localhost:8080/v1\", api_key=\"test\").chat.completions.create(model=\"stub-model\", messages=[...])` returns a valid response",
"`openai.OpenAI(base_url=\"http://localhost:8080/v1\", api_key=\"test\").chat.completions.create(model=\"stub-model\", messages=[...])` returns a valid response",
"Streaming works: `stream=True` returns `text/event-stream` chunks in OpenAI SSE format",
"Streaming works: `stream=True` returns `text/event-stream` chunks in OpenAI SSE format",
@@ -130,12 +130,12 @@
],
],
"completionNotes":"All 6 gateway tests pass including OpenAI SDK, LangChain, streaming, and 503 for unknown model.",
"completionNotes":"All 6 gateway tests pass including OpenAI SDK, LangChain, streaming, and 503 for unknown model.",
"status":"done",
"status":"done",
"status_reason":"Gateway is currently the pipeline orchestrator. US-014 moves orchestration to tracker-nodes; gateway becomes a thin load-balancer proxy. Implementation will be superseded \u2014 defer rework until US-014 lands."
"status_reason":"Gateway is currently the pipeline orchestrator. US-014 moves orchestration to tracker-nodes; gateway becomes a thin load-balancer proxy. Implementation will be superseded — defer rework until US-014 lands."
"description":"Deploy and integrate the Solana smart contracts that make node staking, client payment, and token reward settlement trustless. All development and testing targets **Solana testnet** \u2014 never devnet or mainnet during development, to avoid real costs.",
"description":"Deploy and integrate the Solana smart contracts that make node staking, client payment, and token reward settlement trustless. All development and testing targets **Solana testnet** — never devnet or mainnet during development, to avoid real costs.",
"acceptanceCriteria":[
"acceptanceCriteria":[
"All contracts deploy successfully to Solana testnet",
"All contracts deploy successfully to Solana testnet",
"A node can submit a stake transaction and have its balance reflected in the registry contract",
"A node can submit a stake transaction and have its balance reflected in the registry contract",
@@ -143,7 +143,7 @@
"After a completed inference session, compute attribution is recorded on-chain with correct node/layer attribution",
"After a completed inference session, compute attribution is recorded on-chain with correct node/layer attribution",
"The epoch settlement transaction correctly distributes token rewards to node operators and deducts client balances",
"The epoch settlement transaction correctly distributes token rewards to node operators and deducts client balances",
"The gateway refuses to route to a node whose stake balance is below the minimum threshold",
"The gateway refuses to route to a node whose stake balance is below the minimum threshold",
"All contract interactions in tests run against a local Solana test validator (via `solana-test-validator`) \u2014 no live testnet required for CI",
"All contract interactions in tests run against a local Solana test validator (via `solana-test-validator`) — no live testnet required for CI",
"A `.env.testnet` config points to Solana testnet RPC for manual end-to-end testing",
"A `.env.testnet` config points to Solana testnet RPC for manual end-to-end testing",
"description":"Implement the optimistic fraud detection loop. After each inference request completes, the validator process independently decides whether to re-run the request on a reference node (~5% sample rate).",
"description":"Implement the optimistic fraud detection loop. After each inference request completes, the validator process independently decides whether to re-run the request on a reference node (~5% sample rate).",
"acceptanceCriteria":[
"acceptanceCriteria":[
"The validator process samples ~5% of completed inference requests (configurable)",
"The validator process samples ~5% of completed inference requests (configurable)",
@@ -184,7 +184,7 @@
},
},
{
{
"id":"US-008",
"id":"US-008",
"title":"08 \u2014 Node probationary period + ban enforcement",
"title":"08 — Node probationary period + ban enforcement",
"description":"Implement the two anti-sybil mechanisms that make re-entering the network after a ban economically costly.",
"description":"Implement the two anti-sybil mechanisms that make re-entering the network after a ban economically costly.",
"acceptanceCriteria":[
"acceptanceCriteria":[
"A node with a new wallet receives no token rewards for its first N jobs (verified via settlement contract state)",
"A node with a new wallet receives no token rewards for its first N jobs (verified via settlement contract state)",
@@ -193,7 +193,7 @@
"A wallet with strike count at the threshold is marked banned in the registry contract",
"A wallet with strike count at the threshold is marked banned in the registry contract",
"The tracker excludes banned wallets from route selection",
"The tracker excludes banned wallets from route selection",
"A banned wallet that attempts to register with the tracker is rejected",
"A banned wallet that attempts to register with the tracker is rejected",
"An integration test covers: new wallet \u2192 N jobs \u2192 earning begins; and: strike threshold reached \u2192 banned \u2192 excluded from routes",
"An integration test covers: new wallet → N jobs → earning begins; and: strike threshold reached → banned → excluded from routes",
"python -m pytest passes from repo root",
"python -m pytest passes from repo root",
"Commit only this story's changes"
"Commit only this story's changes"
],
],
@@ -208,7 +208,7 @@
},
},
{
{
"id":"US-009",
"id":"US-009",
"title":"09 \u2014 P2P shard swarm",
"title":"09 — P2P shard swarm",
"description":"Once a node has downloaded a shard, it seeds that shard to other nodes that are assigned the same shard.",
"description":"Once a node has downloaded a shard, it seeds that shard to other nodes that are assigned the same shard.",
"acceptanceCriteria":[
"acceptanceCriteria":[
"A node that has downloaded a shard is listed as a peer for that shard in the tracker",
"A node that has downloaded a shard is listed as a peer for that shard in the tracker",
@@ -231,7 +231,7 @@
},
},
{
{
"id":"US-010",
"id":"US-010",
"title":"10 \u2014 `meshnet` Python SDK",
"title":"10 — `meshnet` Python SDK",
"description":"A Python SDK (`packages/sdk`) that wraps the OpenAI-compatible gateway and exposes network-specific controls.",
"description":"A Python SDK (`packages/sdk`) that wraps the OpenAI-compatible gateway and exposes network-specific controls.",
"description":"Replace the base64 JSON activation payload with raw binary HTTP bodies, zstd compression, and chunked prefill (128 tokens/chunk). All nodes and the gateway must be migrated. Stub nodes continue to emit zeroed tensors, just in binary. This is a protocol prerequisite for US-012 (real model backend).",
"description":"Replace the base64 JSON activation payload with raw binary HTTP bodies, zstd compression, and chunked prefill (128 tokens/chunk). All nodes and the gateway must be migrated. Stub nodes continue to emit zeroed tensors, just in binary. This is a protocol prerequisite for US-012 (real model backend).",
"acceptanceCriteria":[
"acceptanceCriteria":[
"Node /forward endpoint reads shape/dtype/session/chunk from HTTP headers; body is raw binary (optionally zstd-compressed)",
"Node /forward endpoint reads shape/dtype/session/chunk from HTTP headers; body is raw binary (optionally zstd-compressed)",
@@ -281,7 +281,7 @@
},
},
{
{
"id":"US-012",
"id":"US-012",
"title":"12 \u2014 Real PyTorch model backend",
"title":"12 — Real PyTorch model backend",
"description":"Replace stub node inference with actual transformers layer execution. Node loads HuggingFace SafeTensors model shard (model.model.layers[start:end]), runs real forward passes in bfloat16, handles head (embed_tokens) and tail (lm_head) responsibilities, and supports bitsandbytes NF4/INT8/bfloat16 quantization via --quantization flag. Test model: openai-community/gpt2.",
"description":"Replace stub node inference with actual transformers layer execution. Node loads HuggingFace SafeTensors model shard (model.model.layers[start:end]), runs real forward passes in bfloat16, handles head (embed_tokens) and tail (lm_head) responsibilities, and supports bitsandbytes NF4/INT8/bfloat16 quantization via --quantization flag. Test model: openai-community/gpt2.",
"acceptanceCriteria":[
"acceptanceCriteria":[
"meshnet-node start --model-id openai-community/gpt2 --shard-start 0 --shard-end 6 loads the shard without error",
"meshnet-node start --model-id openai-community/gpt2 --shard-start 0 --shard-end 6 loads the shard without error",
"description":"Upgrade tracker route selection to coverage-first, speed-weighted bin-packing. Tracker maintains a live coverage map per model, issues LOAD_SHARD/DROP_SHARD rebalance directives when coverage drops, and assigns shard ranges using declared VRAM, quantization, and benchmark throughput. A model is only routable when all layer ranges have node_count >= 1.",
"description":"Upgrade tracker route selection to coverage-first, speed-weighted bin-packing. Tracker maintains a live coverage map per model, issues LOAD_SHARD/DROP_SHARD rebalance directives when coverage drops, and assigns shard ranges using declared VRAM, quantization, and benchmark throughput. A model is only routable when all layer ranges have node_count >= 1.",
"description":"Two improvements to the Ralph workflow tooling. (1) Status-field awareness: replace all passes:true/false reads in ralph_progress.py with the rich status field. Surface to-revise and needs-review stories as an attention list in the dashboard; auto command skips them by default. (2) Agent agnosticism: make the agent selectable per session \u2014 codex (existing), claude (Claude Code CLI), openrouter (unified API for GPT-4/Mistral/Llama/DeepSeek via OPENROUTER_API_KEY + --model), or custom (--agent-cmd path to any script). Persist agent choice to .ralph-tui/agent-config.json. When ralph-tui does not natively support a requested agent, ralph_progress.py falls back to a thin adapter that calls the agent API directly with the task prompt. (3) Worktree parallelism: ralph_progress.py auto --parallel [N] creates one git worktree per ready task (git worktree add ../AI-worktree-<id> -b feat/<id>), runs up to N agent sessions concurrently, and merges each branch back to master after the agent exits 0 and tests pass. Non-blocking stories that share no file overlap can safely run in parallel. ralph_progress.py list-parallel shows which open stories have no overlapping dependsOn chains and are safe to run concurrently.",
"description":"Two improvements to the Ralph workflow tooling. (1) Status-field awareness: replace all passes:true/false reads in ralph_progress.py with the rich status field. Surface to-revise and needs-review stories as an attention list in the dashboard; auto command skips them by default. (2) Agent agnosticism: make the agent selectable per session — codex (existing), claude (Claude Code CLI), openrouter (unified API for GPT-4/Mistral/Llama/DeepSeek via OPENROUTER_API_KEY + --model), or custom (--agent-cmd path to any script). Persist agent choice to .ralph-tui/agent-config.json. When ralph-tui does not natively support a requested agent, ralph_progress.py falls back to a thin adapter that calls the agent API directly with the task prompt. (3) Worktree parallelism: ralph_progress.py auto --parallel [N] creates one git worktree per ready task (git worktree add ../AI-worktree-<id> -b feat/<id>), runs up to N agent sessions concurrently, and merges each branch back to master after the agent exits 0 and tests pass. Non-blocking stories that share no file overlap can safely run in parallel. ralph_progress.py list-parallel shows which open stories have no overlapping dependsOn chains and are safe to run concurrently.",
"acceptanceCriteria":[
"acceptanceCriteria":[
"All story.get('passes') reads in ralph_progress.py replaced with _is_done() helper that reads status field with passes fallback",
"All story.get('passes') reads in ralph_progress.py replaced with _is_done() helper that reads status field with passes fallback",
"_story_sets() returns six buckets: done, attention (to-revise/needs-review), active (in-progress), in-design, ready, blocked",
"_story_sets() returns six buckets: done, attention (to-revise/needs-review), active (in-progress), in-design, ready, blocked",
"title":"16 \u2014 Mining-style node startup CLI + live dashboard",
"title":"16 — Mining-style node startup CLI + live dashboard",
"description":"Replace the bare flag-driven node CLI with a wizard-guided first-run experience (like a GPU mining client) followed by a live terminal dashboard once the node is running. On first run, the wizard auto-detects GPU VRAM, presents a curated list of compatible models with VRAM requirements at each quantization level, lets the user pick a download location, and writes a persistent config file so subsequent starts are one command. Once the node is running, the wizard gives way to a rich live status panel showing: GPU temp + VRAM used, tokens/sec, requests served, peers connected, TAI earned (stub until US-006 is live). A Browse HuggingFace option calls the HF Hub API so users can load any HF model beyond the curated list.",
"description":"Replace the bare flag-driven node CLI with a wizard-guided first-run experience (like a GPU mining client) followed by a live terminal dashboard once the node is running. On first run, the wizard auto-detects GPU VRAM, presents a curated list of compatible models with VRAM requirements at each quantization level, lets the user pick a download location, and writes a persistent config file so subsequent starts are one command. Once the node is running, the wizard gives way to a rich live status panel showing: GPU temp + VRAM used, tokens/sec, requests served, peers connected, TAI earned (stub until US-006 is live). A Browse HuggingFace option calls the HF Hub API so users can load any HF model beyond the curated list.",
"acceptanceCriteria":[
"acceptanceCriteria":[
" with no args and no config file enters the interactive setup wizard",
" with no args and no config file enters the interactive setup wizard",
"Wizard step 1: auto-detect GPU(s) via torch.cuda / torch.version.hip; print GPU name + total VRAM",
"Wizard step 1: auto-detect GPU(s) via torch.cuda / torch.version.hip; print GPU name + total VRAM",
"Wizard step 2: show curated model list (name, HF repo, layers, VRAM@NF4/INT8/BF16); mark models that do NOT fit available VRAM as [too large]",
"Wizard step 2: show curated model list (name, HF repo, layers, VRAM@NF4/INT8/BF16); mark models that do NOT fit available VRAM as [too large]",
"Wizard step 3: offer [B] Browse HuggingFace \u2014 calls HF Hub API (huggingface_hub.list_models filtered by pipeline_tag=text-generation, sorted by downloads, top 20) and lets user enter a custom HF repo ID",
"Wizard step 3: offer [B] Browse HuggingFace — calls HF Hub API (huggingface_hub.list_models filtered by pipeline_tag=text-generation, sorted by downloads, top 20) and lets user enter a custom HF repo ID",
"Wizard step 4: prompt for download directory (default ~/.meshnet/models/); validate writable; show estimated disk usage for chosen model+quantization",
"Wizard step 4: prompt for download directory (default ~/.meshnet/models/); validate writable; show estimated disk usage for chosen model+quantization",
"Wizard writes ~/.config/meshnet/config.json; second run skips wizard and starts directly",
"Wizard writes ~/.config/meshnet/config.json; second run skips wizard and starts directly",
@@ -413,7 +413,7 @@
},
},
{
{
"id":"US-017",
"id":"US-017",
"title":"17 \u2014 P2P gossip, NAT-traversal relay node, and SSL/TLS",
"title":"17 — P2P gossip, NAT-traversal relay node, and SSL/TLS",
"description":"Add a gossip layer so nodes discover each other and propagate coverage-map changes without polling the tracker continuously. Introduce a publicly-hosted relay node (run by the team) that solves NAT traversal using circuit relay (Petals-style) and serves as the bootstrap peer list for new nodes. Encrypt all node-to-node and node-to-tracker communications with TLS. Gossip protocol: WebSocket-based PubSub over wss://, topics: node-join / node-leave / coverage-update / heartbeat. Peer discovery: mDNS (zeroconf) for LAN, public relay bootstrap list for internet. NAT traversal: relay node acts as TCP-level circuit relay when direct connection fails (hole-punching first, relay second). Architecture is designed to migrate to libp2p GossipSub + Kademlia DHT in a future story without breaking the message schema.",
"description":"Add a gossip layer so nodes discover each other and propagate coverage-map changes without polling the tracker continuously. Introduce a publicly-hosted relay node (run by the team) that solves NAT traversal using circuit relay (Petals-style) and serves as the bootstrap peer list for new nodes. Encrypt all node-to-node and node-to-tracker communications with TLS. Gossip protocol: WebSocket-based PubSub over wss://, topics: node-join / node-leave / coverage-update / heartbeat. Peer discovery: mDNS (zeroconf) for LAN, public relay bootstrap list for internet. NAT traversal: relay node acts as TCP-level circuit relay when direct connection fails (hole-punching first, relay second). Architecture is designed to migrate to libp2p GossipSub + Kademlia DHT in a future story without breaking the message schema.",
"acceptanceCriteria":[
"acceptanceCriteria":[
"All HTTP between nodes and tracker uses HTTPS (TLS 1.3); self-signed cert generated on first run and fingerprint pinned in config; relay node uses Let's Encrypt",
"All HTTP between nodes and tracker uses HTTPS (TLS 1.3); self-signed cert generated on first run and fingerprint pinned in config; relay node uses Let's Encrypt",
@@ -439,7 +439,7 @@
},
},
{
{
"id":"US-018",
"id":"US-018",
"title":"18 \u2014 End-to-end two-machine LAN inference test",
"title":"18 — End-to-end two-machine LAN inference test",
"description":"Prove the network works across two real machines: the Linux rig (this machine) and a Windows 11 rig running WSL2. One machine runs the tracker + first-shard node (inference entry point). The other machine runs a second-shard node. A client sends a real inference request and receives a streamed response. This story is primarily a test plan + setup guide + test execution script; it produces documented evidence (logs, timing, token output) that real distributed inference works. It also surfaces any real-world issues (port forwarding, CUDA driver version mismatches, WSL2 CUDA passthrough, model download paths) that need fixing.",
"description":"Prove the network works across two real machines: the Linux rig (this machine) and a Windows 11 rig running WSL2. One machine runs the tracker + first-shard node (inference entry point). The other machine runs a second-shard node. A client sends a real inference request and receives a streamed response. This story is primarily a test plan + setup guide + test execution script; it produces documented evidence (logs, timing, token output) that real distributed inference works. It also surfaces any real-world issues (port forwarding, CUDA driver version mismatches, WSL2 CUDA passthrough, model download paths) that need fixing.",
"acceptanceCriteria":[
"acceptanceCriteria":[
"docs/INSTALL_WINDOWS.md exists: step-by-step WSL2 + CUDA + meshnet-node install on Windows 11",
"docs/INSTALL_WINDOWS.md exists: step-by-step WSL2 + CUDA + meshnet-node install on Windows 11",
@@ -447,7 +447,7 @@
"A test script scripts/test_lan_inference.py: given --tracker-url, --gateway-url, sends 3 chat completion requests, asserts valid OpenAI format, prints token count + latency + which nodes served each request",
"A test script scripts/test_lan_inference.py: given --tracker-url, --gateway-url, sends 3 chat completion requests, asserts valid OpenAI format, prints token count + latency + which nodes served each request",
"Both machines can reach each other on LAN (documented: firewall rules, port list)",
"Both machines can reach each other on LAN (documented: firewall rules, port list)",
"At least one successful inference recorded: the test script exits 0 with output showing tokens generated and node IDs",
"At least one successful inference recorded: the test script exits 0 with output showing tokens generated and node IDs",
"Latency breakdown logged: gateway\u2192node-A, node-A\u2192node-B, node-B\u2192gateway (approximate, from server logs)",
"Latency breakdown logged: gateway→node-A, node-A→node-B, node-B→gateway (approximate, from server logs)",
"Known issues during test documented in docs/TWO_MACHINE_TEST.md under a Known Issues section",
"Known issues during test documented in docs/TWO_MACHINE_TEST.md under a Known Issues section",
"description":"Replace the single-point-of-failure tracker with a fault-tolerant cluster. Tracker nodes elect a leader via Raft and commit shard assignments as log entries \u2014 all tracker nodes agree on who owns what. Node liveness (heartbeats) uses CRDT gossip (eventual consistency, high frequency OK). A node registers with any tracker node; the write is forwarded to the leader and replicated to followers. A 3-node tracker cluster survives one tracker failure without losing assignment state. The relay/gossip layer already built in US-017 handles peer heartbeats; this story wires Raft on top for authoritative assignments.",
"description":"Replace the single-point-of-failure tracker with a fault-tolerant cluster. Tracker nodes elect a leader via Raft and commit shard assignments as log entries — all tracker nodes agree on who owns what. Node liveness (heartbeats) uses CRDT gossip (eventual consistency, high frequency OK). A node registers with any tracker node; the write is forwarded to the leader and replicated to followers. A 3-node tracker cluster survives one tracker failure without losing assignment state. The relay/gossip layer already built in US-017 handles peer heartbeats; this story wires Raft on top for authoritative assignments.",
"acceptanceCriteria":[
"acceptanceCriteria":[
"3 tracker nodes can be started and form a Raft cluster (leader election, log replication)",
"3 tracker nodes can be started and form a Raft cluster (leader election, log replication)",
"A node registers with any follower \u2014 the registration is forwarded to the leader and replicated",
"A node registers with any follower — the registration is forwarded to the leader and replicated",
"Killing the leader causes a new election within 5 seconds; registrations continue working",
"Killing the leader causes a new election within 5 seconds; registrations continue working",
"Shard assignments returned by any tracker node are identical (strong consistency)",
"Shard assignments returned by any tracker node are identical (strong consistency)",
"description":"Polish pass on tracker and node after real multi-host LAN testing. Three concrete issues: (1) tracker _send_json crashed with BrokenPipeError when a slow-inference client disconnected mid-stream; (2) node IDs were random UUIDs, making re-registration after tracker restart create phantom entries; (3) HF-repo model coverage endpoint did not handle short-name vs full-repo lookups consistently.",
"acceptanceCriteria":[
"BrokenPipeError in tracker _send_json is silently swallowed (client disconnected is not an error)",
"Node IDs are deterministic (wallet-address + port hash) so re-registration after tracker restart reuses the same ID",
"GET /v1/coverage/<model> accepts both short names and full HF repo IDs",
"python -m pytest passes from repo root"
],
"priority":20,
"status":"done",
"notes":"Discovered during first two-machine LAN test (US-018 follow-up).",
"dependsOn":[
"US-018"
],
"completionNotes":"BrokenPipeError wrapped in try/except in tracker and node send paths. Node IDs now sha256(wallet+port)[:16]. Coverage endpoint normalises short/full HF names."
},
{
"id":"US-021",
"title":"21 — --route-timeout CLI flag for node tracker route lookup",
"description":"The node's slow-path tracker route lookup used a hard-coded 30-second timeout. On high-latency networks (relay, 5G) or when the tracker is slow to respond, this caused premature failures. Expose it as a CLI flag so operators can tune it per deployment.",
"description":"When two nodes register overlapping shard ranges (e.g. node A covers 0-15 and node B covers 12-23), the naive pipeline re-runs layers 12-15 on node B. The X-Meshnet-Start-Layer header tells each downstream node which layer index to start from, skipping already-computed layers. The tracker injects start_layer into X-Meshnet-Route hops at proxy time.",
"acceptanceCriteria":[
"Tracker _handle_proxy_chat builds route hops with start_layer computed from covered_up_to",
"Node _handle_binary_forward reads X-Meshnet-Start-Layer and passes it to backend.forward_bytes",
"Node _get_remaining_route parses start_layer from both injected header and /v1/route slow-path",
"TorchModelShard.forward_bytes accepts optional start_layer and skips layers below it",
"Test: two-node route with overlapping shards produces correct output without double-computing layers",
"python -m pytest passes"
],
"priority":22,
"status":"done",
"notes":"Protocol decision: option A (start_layer injected by tracker, not negotiated peer-to-peer). Approved in design session.",
"dependsOn":[
"US-014",
"US-019"
],
"completionNotes":"X-Meshnet-Start-Layer header added to forward protocol. Tracker computes covered_up_to as it builds route_hops. model_backend.forward_bytes(start_layer=) implemented. Tests added for overlapping shard scenarios."
"description":"Node heartbeats are currently empty POSTs. Extend them to carry cumulative stats (total_requests, failed_requests, queue_depth, uptime_seconds) so the tracker can make informed load-balancing decisions. The heartbeat response may include a new_assignment field, enabling the tracker to redirect a node to a different shard without a restart.",
"acceptanceCriteria":[
"Heartbeat POST body includes total_requests, failed_requests, queue_depth, uptime_seconds, status",
"TorchNodeServer tracks total_requests, failed_requests, queue_depth with a threading.Lock",
"Tracker stores the last heartbeat payload per node and exposes it in /v1/nodes (list endpoint)",
"Heartbeat response may include new_assignment: {model, shard_start, shard_end}; node logs it",
"Stats survive tracker outage: buffered locally, flushed on next successful heartbeat",
"python -m pytest passes"
],
"priority":23,
"status":"done",
"notes":"Reassignment is logged only for now; hot-reload (load new shard without restart) is a future story.",
"dependsOn":[
"US-016"
],
"completionNotes":"torch_server.py: total_requests, failed_requests, queue_depth added with _stats_lock. startup.py _start_heartbeat sends stats dict. Tracker stores last_heartbeat per node. new_assignment field in heartbeat response logged by node."
},
{
"id":"US-024",
"title":"24 — Enhanced availability map with per-node health details",
"description":"GET /v1/coverage/<model> returns coarse band-level coverage. Operators need to know which specific nodes are in each band and whether they are healthy (last heartbeat time, queue depth). Extend the response to include a nodes array per band with per-node health details, and add a separate detailed endpoint.",
"acceptanceCriteria":[
"GET /v1/coverage/<model> response includes nodes: [{node_id, endpoint, healthy, queue_depth, last_seen_s}] per band",
"healthy is true iff last heartbeat < heartbeat_timeout seconds ago",
"Existing band fields (start_layer, end_layer, node_count) are preserved",
"Tests updated: band assertions check nodes array not just node_count",
"python -m pytest passes"
],
"priority":24,
"status":"done",
"notes":"Grouped-by-band format chosen (both node list and band metadata in same object). Dead nodes included in response with healthy=false.",
"dependsOn":[
"US-013"
],
"completionNotes":"_coverage_map_detailed added to tracker. Each band now includes nodes list. Tests updated to check band fields individually rather than exact dict comparison."
},
{
"id":"US-025",
"title":"25 — Model usage statistics: rolling RPM windows + SQLite persistence",
"description":"Trackers need to know which models are being requested most often to make smart load-balancing and assignment decisions. Add per-model rolling request-rate counters (last hour, last day, last month) with a circular-bucket implementation. Persist buckets to SQLite so stats survive tracker restarts. Support gossip-based stat sharing between tracker peers (additive merge, per-tracker slices).",
"acceptanceCriteria":[
"_RollingCounter: circular buckets, epoch-indexed, stale buckets auto-reset on record()",
"notes":"Option B chosen: stats stored locally per tracker and synced via gossip (not aggregated centrally). Rolling windows: 60×1min buckets (1 hour), 24×1hr (1 day), 30×1day (~1 month).",
"dependsOn":[
"US-017"
],
"completionNotes":"_RollingCounter and _ModelStats classes in server.py. _StatsCollector with SQLite save/load. /v1/stats and /v1/stats/gossip endpoints. --stats-db CLI flag. Stats gossip in _stats_loop thread."
},
{
"id":"US-026",
"title":"26 — Smart model assignment via demand×coverage scoring",
"description":"The /v1/network/assign endpoint assigns new nodes to whichever model and shard range covers the biggest uncovered gap. This ignores demand: a model with 1000 RPM and 60% coverage should attract more nodes than a zero-traffic model with 50% coverage. Add a scoring formula: score = (demand_rpm + 1.0) × (coverage_deficit + 0.01) so high-demand under-covered models win. price_per_token is reserved in the protocol response at 0.0.",
"acceptanceCriteria":[
"/v1/network/assign returns the highest-scoring model+shard using demand×coverage formula",
"demand_rpm uses combined stats from _StatsCollector (local + peer slices)",
"coverage_deficit is fraction of layers with zero coverage [0.0, 1.0]",
"price_per_token: 0.0 included in response (reserved field)",
"Models with no traffic still compete by coverage (floor: demand_rpm + 1.0)",
"Test: two models, one high-demand low-coverage wins over low-demand high-coverage",
"python -m pytest passes"
],
"priority":26,
"status":"done",
"notes":"Score formula chosen after grilling: (demand_rpm + 1.0) × (coverage_deficit + 0.01). The +1.0 floor means coverage alone drives assignment when demand is zero.",
"dependsOn":[
"US-025",
"US-024"
],
"completionNotes":"_handle_network_assign updated with scoring. _effective_throughput helper. price_per_token: 0.0 in response. Tests in test_tracker_routing.py."
},
{
"id":"US-027",
"title":"27 — Throughput-optimized routing: effective throughput as tiebreak",
"description":"The greedy max-reach route selection picks nodes by shard coverage but does not consider node speed. When two nodes cover the same remaining range, prefer the one with higher effective throughput (benchmark_tokens_per_sec / (queue_depth + 1)). This is a tiebreak only — coverage maximization remains the primary objective.",
"_select_route uses throughput as tiebreak when shard_end is equal",
"Test: two nodes with same shard range, different throughput — faster node is picked",
"Existing greedy coverage tests still pass (throughput does not change primary selection)",
"python -m pytest passes"
],
"priority":27,
"status":"done",
"notes":"Throughput tiebreak only — coverage-maximizing greedy stays primary. queue_depth from last heartbeat.",
"dependsOn":[
"US-023"
],
"completionNotes":"_effective_throughput added to server.py. _select_route updated: when shard_end equal, max throughput wins. Tests added and existing tests corrected for new tiebreak order."
},
{
"id":"US-028",
"title":"28 — Routing correctness tests: three-node, overlap, and throughput scenarios",
"description":"Add a comprehensive test suite for route selection covering: greedy three-node no-overlap ordering, overlapping shard handling, throughput tiebreak, and gap detection. These tests lock in the routing contract so refactors don't silently regress.",
"acceptanceCriteria":[
"test_select_route_no_overlap_three_nodes: greedy picks nodes in layer order (A→C→B for ranges 0-7, 8-15, 16-23)",
"All tests pass without mocking (in-process tracker server)",
"python -m pytest passes"
],
"priority":28,
"status":"done",
"notes":"Tests revealed the greedy picks A→C→B (not A→B→C) for non-overlapping ranges when C starts at a lower layer than B. Test expectation corrected to match algorithm.",
"dependsOn":[
"US-027"
],
"completionNotes":"7 routing tests added to test_tracker_routing.py. _make_node helper. test_select_route_no_overlap_three_nodes corrected: greedy outputs [A, C, B] when C.shard_start < B.shard_start."
"description":"Nodes behind NAT (WSL2, 5G, home routers) register with the tracker including a relay_addr (wss://relay/rpc/{peer_id}). When the head node needs to forward activations to a behind-NAT peer, it must use the relay instead of direct HTTP. Add _relay_hop() to torch_server.py that opens a per-hop WebSocket to the relay's /rpc/{peer_id} endpoint, sends the binary activation (base64-encoded), and returns the response. If relay fails, fall back to direct HTTP.",
"_get_remaining_route returns list[dict] with relay_addr field (was list[tuple])",
"_run_downstream_pipeline dispatches via _relay_hop when hop has relay_addr",
"Falls back to direct HTTP if relay connection fails (logs warning)",
"Tracker _handle_proxy_chat includes relay_addr in downstream hop dicts",
"relay_bridge._handle_request decodes body_base64; response uses body_base64 for binary (octet-stream)",
"All 157 tests pass",
"QUICKSTART.md updated with relay NAT/internet section"
],
"priority":29,
"status":"done",
"notes":"relay_addr format: wss://relay.../rpc/{peer_id}. Binary activations (bfloat16) base64-encoded through relay JSON protocol — no precision loss. WSL2 nodes now work behind NAT without --advertise-host.",
"dependsOn":[
"US-017",
"US-022"
],
"completionNotes":"_relay_hop() added to torch_server.py. _get_remaining_route returns list[dict]. relay_bridge.py updated with body_base64 support. Tracker injects relay_addr into downstream hop dicts. 157 tests pass."
`_select_route` uses this as a tiebreak only: when two candidates reach the same
maximum `shard_end`, the one with higher effective throughput is preferred.
Coverage maximization remains the primary objective.
`benchmark_tokens_per_sec` comes from the hardware profile at registration.
`queue_depth` comes from the most recent heartbeat. A busy node (high queue)
is deprioritized without being excluded.
### 5. SQLite persistence
Stats are saved to SQLite (configurable via `--stats-db PATH`) every 60 seconds and
on clean shutdown. Schema: `model_rpm_buckets(model, window, bucket_idx, bucket_epoch, count)`.
The circular-bucket structure maps directly — each slot is one row.
## Consequences
- Tracker startup is slightly slower when loading a large stats DB (sub-second for typical sizes)
- Peer gossip adds one round-trip per gossip interval per peer
-`price_per_token` is a stable wire field; future billing sets it to a real value
-`effective_throughput` depends on `benchmark_tokens_per_sec` being set correctly at registration; nodes that don't set it get the default `1.0` and are treated as slowest
The head node reads `relay_addr` from the injected `X-Meshnet-Route` header and calls
`_relay_hop` instead of direct HTTP.
## Consequences
- Nodes behind NAT (WSL2, 5G, home routers) can now participate in distributed pipeline inference without opening firewall ports
-`relay_addr` is a stable registration field; nodes without a relay omit it and receive direct HTTP hops
- Per-hop WebSocket setup adds latency proportional to relay RTT; acceptable for prototype, optimize later with persistent tunnels
- Base64 encoding increases payload size by ~33%; acceptable for prototype
- The relay server remains stateless and horizontally scalable; only the persistent per-peer `/ws` connections are stateful
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Dobromir Popov