dash works !!! good data. billing seems to work

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

.agents/skills/close-feature/SKILL.md

@@ -0,0 +1,115 @@
+---
+name: close-feature
+description: Graduate a completed feature from .scratch/<slug>/ into the permanent docs/ structure. Run when all issues in the feature are done, or when the agent notices a feature looks complete. Always confirm with the user before moving files.
+---
+
+# Close Feature
+
+Merge a completed `.scratch/<slug>/` feature into the permanent `docs/` structure and delete the scratch directory.
+
+## When to run
+
+Run this skill when:
+- The user explicitly calls `/close-feature <slug>`
+- You notice during any task that ALL issues in a `.scratch/<slug>/issues/` directory have `Status: done` — surface this proactively and offer to close ("It looks like <slug> is fully done. Want me to graduate it to docs/?")
+
+Always confirm before moving any files.
+
+## Process
+
+### 1. Identify the slug
+
+If called with an argument (`/close-feature <slug>`), use that. Otherwise list all `.scratch/` directories and ask the user which one to close.
+
+### 2. Verify all issues are done
+
+Read every file in `.scratch/<slug>/issues/`. Check for a `Status:` line.
+
+If any issue does NOT have `Status: done`, list the incomplete ones and stop — ask the user whether to mark them wontfix or wait.
+
+### 3. Show the user what will move
+
+Present a summary:
+- N issues → `docs/issues/` (renumbered from next available)
+- PRD.md → `docs/PRD.md` (or merged if one already exists)
+- prd.json → `docs/prd.json` (merged into existing user stories array)
+- Any ADRs written during the feature → `docs/adr/` (renumbered from next available)
+- `.scratch/<slug>/` deleted
+
+Ask: "Proceed?" before touching anything.
+
+### 4. Move issues
+
+Find the highest existing number in `docs/issues/`. Issues from `.scratch/<slug>/issues/` take the next slots in dependency order (lowest numbered first).
+
+Use `git mv` for each file so history is preserved:
+```
+git mv .scratch/<slug>/issues/01-foo.md docs/issues/<next>-foo.md
+```
+
+### 5. Merge PRD
+
+If `docs/PRD.md` does not exist: `git mv .scratch/<slug>/PRD.md docs/PRD.md`
+
+If `docs/PRD.md` already exists: append the feature's Problem Statement and Solution sections under a `## <Feature Name>` heading. Do not overwrite the existing file.
+
+### 6. Merge prd.json
+
+If `docs/prd.json` does not exist: `git mv .scratch/<slug>/prd.json docs/prd.json`
+
+If `docs/prd.json` already exists: merge the `userStories` array from the scratch prd.json into the main one, assigning new sequential ids to avoid collisions. Update all merged stories to `"status": "done"`.
+
+### 7. Move ADRs
+
+If any `.md` files exist directly in `.scratch/<slug>/` (not in `issues/`) that look like ADRs (contain `## Status` and `## Decision`), move them to `docs/adr/` renumbered from the next available slot.
+
+### 8. Delete scratch directory
+
+```
+git rm -r .scratch/<slug>/
+```
+
+If `.scratch/` is now empty, remove it too (it has no meaning without sub-features).
+
+### 9. Commit
+
+```
+git add docs/
+git commit -m "docs: close feature <slug> — graduate to docs/"
+```
+
+### 10. Update ralph default (if needed)
+
+If `scripts/ralph_progress.py` exists and its `DEFAULT_PRD` still points at `.scratch/<slug>/prd.json`, update it to `docs/prd.json`.
+
+## Machine setup reference
+
+Skills are stored in `.agents/skills/` and symlinked into `.claude/skills/`. Both directories are git-tracked so cloning the repo gives you all skills automatically.
+
+**On a new machine (Linux/Mac):**
+```bash
+git clone <repo>
+# Skills work immediately — .claude/skills/ symlinks resolve automatically.
+# .claude/settings.local.json is machine-local (gitignored).
+# Recreate it if you need custom permission allowlists.
+```
+
+**On Windows:**
+```bash
+git clone -c core.symlinks=true <repo>
+# Requires Developer Mode or admin rights for symlink creation.
+# If symlinks didn't resolve, run: scripts/link-skills.sh (or re-run /setup-matt-pocock-skills)
+```
+
+**Codex:** reads `AGENTS.md` at the repo root — already tracked, no setup needed.
+
+**Hermes / other local LLM tools:** add their context files to the repo root or `docs/` and commit. Point the tool at that file in its config.
+
+**To add a new skill to the repo:**
+```bash
+mkdir .agents/skills/<name>
+# write .agents/skills/<name>/SKILL.md
+ln -s ../../.agents/skills/<name> .claude/skills/<name>
+git add .agents/skills/<name> .claude/skills/<name>
+git commit -m "skill: add <name>"
+```

.claude/memory/MEMORY.md

@@ -0,0 +1,5 @@
+# Memory Index
+
+- [Product selling points](product-selling-points.md) — key differentiators and landing page angles for neuron-tai
+- [User profile](user-profile.md) — who Dobromir is and how to work with him
+- [Project status](project-status.md) — 29/30 done; US-030 (manual route + hop benchmark) is the only open story

.claude/memory/autonomous-work-style.md

@@ -0,0 +1,13 @@
+---
+name: autonomous-work-style
+description: Dobromir wants autonomous batch execution — ask only for architecture decisions, never permissions
+metadata:
+  node_type: memory
+  type: feedback
+---
+
+When given a backlog, work through all open tasks autonomously and report back when done. Ask questions only for implementation/architecture decisions that genuinely need his input (grilling-style, one at a time with a recommendation) — never for permission to proceed, and don't checkpoint between tasks. Running tests is ALWAYS allowed (allowlisted in settings).
+
+**Why:** he said "work on all the tasks and come back when done. ask only for implementation or architecture decisions and not for permissions" and "I ALWAYS ALLOW running tests! stop asking" (2026-07-02, reward-system session).
+
+**How to apply:** default to acting; batch the full backlog from docs/prd.json ([[project-status]]); surface completed-work summaries at the end, not between stories.

.claude/memory/product-selling-points.md

@@ -0,0 +1,38 @@
+---
+name: product-selling-points
+description: Key differentiators and landing page angles for neuron-tai distributed inference network
+metadata: 
+  node_type: memory
+  type: project
+  originSessionId: 8fb120ee-7b8e-45be-98c0-b5ae9c64d1ec
+---
+
+# neuron-tai — Product Selling Points
+
+## Core pitch
+Volunteer GPU network for distributed LLM inference. Small GPU owners contribute compute and earn TAI tokens. Clients get inference on models larger than any single machine can serve.
+
+## Confirmed technical differentiators (verified working)
+
+### Mixed hardware inference routes
+The tracker can chain CPU nodes and GPU nodes into a single inference route. Shard A on a CPU node → Shard B on a GPU node → valid streamed response. Each participant in the route only needs to fit *their shard* in memory, not the whole model.
+
+**Angle for landing page:** "Run a 70B model across three laptops and a gaming PC. Each machine only holds the layers it can fit."
+
+**Nuance to acknowledge:** PyTorch/HuggingFace `device_map="auto"` already does CPU+GPU mixing on a single machine. Our value-add is doing this *across machines over the network*, democratizing access to models that no single volunteer machine could serve alone.
+
+### Hardware-aware routing
+Tracker scores nodes by `benchmark_tokens_per_sec / (queue_depth + 1)` and always routes to the fastest available node per shard range. A GPU node at 11,200 throughput index beats a CPU node at 626 automatically — no user configuration needed.
+
+### Zero port-forwarding required
+Nodes connect outbound to the relay via WebSocket. Works from behind NAT, WSL2, 5G, or a home router with no config. The public tracker at ai.neuron.d-popov.com handles discovery.
+
+### OpenAI-compatible API
+Any app using the OpenAI Python SDK works by changing only `base_url`. No code changes for the client.
+
+## Landing page content TODO
+- User asked to capture these points for the landing page copy (2026-07-01)
+- No landing page file exists in the repo yet
+- When writing copy, lead with the "run models bigger than your GPU" angle, then support with mixed-hardware routing, relay, and OpenAI compat
+
+**How to apply:** When writing product descriptions, pitches, or landing page copy, use these as the primary hooks. The mixed-network inference route (CPU+GPU across machines) is the biggest differentiator vs. single-machine solutions.

.claude/memory/project-status.md

@@ -0,0 +1,30 @@
+---
+name: project-status
+description: Current state of neuron-tai development as of 2026-07-02
+metadata:
+  node_type: memory
+  type: project
+---
+
+# Project Status (2026-07-02)
+
+All 35 user stories in docs/prd.json are done (35/35), including the reward-system arc US-030…US-035 completed 2026-07-02:
+
+- **BillingLedger** (packages/tracker/meshnet_tracker/billing.py): event-sourced USDT ledger, gossip-replicated across the hive (id-deduped events), SQLite-persisted. 90/10 split by work units, per-model per-1K-token pricing, 402 before routing.
+- **Solana custodial adapter** (packages/contracts/meshnet_contracts/solana_adapter.py): urllib JSON-RPC + solders signing. NOTE: installed solana-py 0.40 has NO sync client — don't import solana.rpc.api / spl.token.client.
+- **scripts/devnet_setup.py**: creates mock-USDT mint + treasury, writes .env.devnet; --mint-to funds test clients.
+- **TrackerServer threads**: deposit watcher (exactly-once via deposit-<sig> event ids) + leader-only settlement loop (threshold OR max-period, dust floor, resend-by-settlement-id → no double-pay).
+- **Forfeiture penalty**: validator forfeits pending balance + strike; 3 strikes ban; probation redirects shares to protocol cut. Math in packages/validator/README.md.
+- **Web dashboard**: GET /dashboard on any tracker, embedded dashboard.html, 4s polling.
+
+Suite: 222 passed, 3 skipped (openai/langchain packages missing in .venv — pre-existing).
+
+**Why:** design locked in ADR-0015 (USDT custodial settlement; TAI deferred, protocol cut = future TAI liquidity).
+**How to apply:** next steps are live devnet verification (run devnet_setup.py, start tracker with --solana-rpc-url/--usdt-mint/--treasury-keypair --billing-db), then the TAI mint when volume justifies it. Work not yet committed to git as of session end — check git status.
+
+## Windows CUDA node (working as of 2026-07-01)
+- miniforge3 base env, torch 2.7.1+cu118, torchvision 0.22.x+cu118
+- RTX 4060 Laptop GPU, 8 GB VRAM, benchmark index ~11,200
+- Run: `meshnet-node start --tracker https://ai.neuron.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct`
+- Known: tracker registration fails with `http://` — must use `https://`
+- pynvml deprecation warning is harmless (use nvidia-ml-py to silence it)

.claude/memory/user-profile.md

@@ -0,0 +1,15 @@
+---
+name: user-profile
+description: Who Dobromir is and how to collaborate effectively
+metadata: 
+  node_type: memory
+  type: user
+  originSessionId: 8fb120ee-7b8e-45be-98c0-b5ae9c64d1ec
+---
+
+# Dobromir Popov
+
+- Building neuron-tai: a distributed LLM inference network with volunteer GPU nodes, tracker, relay, and token rewards
+- Works across Linux (AMD Ryzen AI Max APU, 124 GB RAM) and Windows 11 (RTX 4060 Laptop GPU, 8 GB VRAM, miniforge3 Python env)
+- Uses ralph for project management (prd.json + issues in .scratch/)
+- Iterates quickly — prefers short, direct answers and learns from real output/errors rather than pre-emptive explanations

.claude/settings.json

@@ -0,0 +1,27 @@
+{
+  "hooks": {
+    "PreToolUse": [
+      {
+        "matcher": ".*",
+        "hooks": [
+          {
+            "type": "command",
+            "command": "bash -c 'SRC=\"/mnt/d/DEV/workspace/REPOS/git.d-popov.com/neuron-tai/.claude/memory\" && DST=\"/home/dev/.claude/projects/-mnt-d-DEV-workspace-REPOS-git-d-popov-com-neuron-tai/memory\" && mkdir -p \"$DST\" && rsync -a --update \"$SRC/\" \"$DST/\" 2>/dev/null; true'",
+            "runOncePerSession": true
+          }
+        ]
+      }
+    ],
+    "PostToolUse": [
+      {
+        "matcher": "Write|Edit",
+        "hooks": [
+          {
+            "type": "command",
+            "command": "bash -c 'SRC=\"/mnt/d/DEV/workspace/REPOS/git.d-popov.com/neuron-tai/.claude/memory\" && DST=\"/home/dev/.claude/projects/-mnt-d-DEV-workspace-REPOS-git-d-popov-com-neuron-tai/memory\" && mkdir -p \"$DST\" && rsync -a \"$SRC/\" \"$DST/\" 2>/dev/null; true'"
+          }
+        ]
+      }
+    ]
+  }
+}

.claude/skills/close-feature

@@ -0,0 +1 @@
+../../.agents/skills/close-feature

.dockerignore

@@ -0,0 +1,12 @@
+.git
+.venv
+__pycache__
+*.py[cod]
+.pytest_cache
+*.egg-info
+build
+dist
+.ralph-tui
+.scratch
+.claude
+.env*

.scratch/distributed-inference-network/prd.json

@@ -1,399 +0,0 @@
-{
-  "name": "Distributed Inference Network",
-  "description": "Build a distributed inference network with node, gateway, tracker, SDK, contracts, and P2P shard distribution components from the grill session PRD.",
-  "branchName": "ralph/distributed-inference-network",
-  "userStories": [
-    {
-      "id": "US-001",
-      "title": "01 \u2014 Monorepo scaffold + single-node smoke test",
-      "description": "Stand up the monorepo package layout and prove that a client HTTP request can travel end-to-end through the stack \u2014 gateway \u2192 one node serving all layers \u2192 valid response \u2014 before any real model or distributed logic is wired in. The six top-level packages should exist with minimal stubs: - `packages/node` \u2014 node client CLI (`meshnet-node`) - `packages/gateway` \u2014 HTTP gateway + route orchestration - `packages/tracker` \u2014 node registry and route selection - `packages/sdk` \u2014 `meshnet` Python SDK - `packages/contracts` \u2014 Solana smart contract wrappers - `packages/p2p` \u2014 gossip and shard swarm The smoke test uses a tiny in-process stub model (not a real LLM) so the test is fast and has no external dependencies. The node runs in the same process as the test. The gateway is a real HTTP server. The client sends a real `POST /v1/chat/completions` request and receives a valid OpenAI-format response.",
-      "acceptanceCriteria": [
-        "`pip install -e packages/node packages/gateway packages/tracker` works from repo root",
-        "`meshnet-node`, `meshnet-gateway`, and `meshnet-tracker` CLI entry points exist",
-        "A single integration test starts a gateway and one stub node in-process, sends a `POST /v1/chat/completions` request, and asserts a valid OpenAI-format response is returned",
-        "The test passes with `pytest` from repo root with no external services running",
-        "All six package directories exist with `pyproject.toml` and an importable top-level module",
-        "Follow TDD where code is added: add/adjust an observable behavior test before implementation when practical",
-        "Run the task-specific tests and ensure they pass",
-        "Run `python -m pytest` from repo root and ensure it passes, or document why no test suite exists yet",
-        "Keep the implementation scoped to this story; do not implement downstream stories early",
-        "Commit only this story's code changes with a focused conventional commit message"
-      ],
-      "priority": 1,
-      "passes": true,
-      "notes": "Source issue: .scratch/distributed-inference-network/issues/01-monorepo-scaffold.md",
-      "dependsOn": [],
-      "completionNotes": "Completed by Ralph iteration 7b260695; verified by pytest and editable package installs.",
-      "status": "done"
-    },
-    {
-      "id": "US-002",
-      "title": "02 \u2014 Two-node shard pipeline",
-      "description": "Extend the single-node smoke test so that inference is routed through exactly two nodes, each serving half of a stub model's layers. Activation tensors must travel from the gateway to node A, then from node A to node B, and the final output returns to the gateway. This proves the distributed shard pipeline works end-to-end before real models or a real tracker are introduced. The two nodes run in the same process as the test (no real network required). The gateway hardcodes the two-node route for now \u2014 dynamic route selection comes in issue 03. The activation tensor protocol (shape, dtype, serialisation format) must be established here, as all later issues depend on it. Use the domain vocabulary from `CONTEXT.md`: the ordered list of nodes is an **inference route**; each node's layer range is a **shard**; the tensors passed between nodes are activations (not \"data\" or \"chunks\").",
-      "acceptanceCriteria": [
-        "The integration test spins up two stub nodes, each configured with a non-overlapping shard range",
-        "A `POST /v1/chat/completions` request results in activation tensors flowing node-A \u2192 node-B (verifiable via test assertions or logs)",
-        "The gateway assembles the final response correctly from node-B's output",
-        "The test passes with no external services running",
-        "The activation tensor serialisation format is documented in a short inline comment (shape, dtype, wire format) \u2014 this becomes the contract all future nodes implement",
-        "Follow TDD where code is added: add/adjust an observable behavior test before implementation when practical",
-        "Run the task-specific tests and ensure they pass",
-        "Run `python -m pytest` from repo root and ensure it passes, or document why no test suite exists yet",
-        "Keep the implementation scoped to this story; do not implement downstream stories early",
-        "Commit only this story's code changes with a focused conventional commit message"
-      ],
-      "priority": 2,
-      "passes": true,
-      "notes": "Source issue: .scratch/distributed-inference-network/issues/02-two-node-shard-pipeline.md",
-      "dependsOn": [
-        "US-001"
-      ],
-      "completionNotes": "Completed by Ralph iteration 126384e5; verified by pytest two-node pipeline.",
-      "status": "to-revise",
-      "status_reason": "Base64 JSON wire format established here was replaced by binary HTTP protocol in US-011. tests/test_two_node_pipeline.py needs verification it exercises the new binary format end-to-end."
-    },
-    {
-      "id": "US-003",
-      "title": "03 \u2014 Tracker: node registration + route selection",
-      "description": "Replace the hardcoded two-node route from issue 02 with a real tracker. Nodes register themselves with the tracker on startup (endpoint, shard range, hardware profile, node score). The gateway queries the tracker for an inference route for a given model preset instead of using a static list. The tracker returns an ordered list of node endpoints whose shards collectively cover all layers. The tracker runs as a lightweight HTTP service. Node score is initially a simple placeholder (e.g. fixed value or random) \u2014 real throughput/latency scoring comes later. The tracker must correctly reject route requests when no registered nodes cover a required shard range and return an appropriate error. This issue establishes the tracker's registration and routing API contract, which issues 04 (node startup) and 05 (OpenAI gateway) both depend on.",
-      "acceptanceCriteria": [
-        "A node can register with the tracker via HTTP, providing its endpoint, shard range, and a hardware profile",
-        "The gateway queries the tracker with a model preset name and receives an ordered list of node endpoints forming a complete inference route",
-        "The end-to-end integration test from issue 02 still passes with the tracker now in the loop (no hardcoded routes)",
-        "The tracker returns a clear error response when no route is available for a requested model preset",
-        "Nodes that fail to heartbeat within a configurable window are removed from the registry",
-        "The tracker's registration and route-selection HTTP API is defined (paths, request/response shapes) in the tracker package",
-        "Follow TDD where code is added: add/adjust an observable behavior test before implementation when practical",
-        "Run the task-specific tests and ensure they pass",
-        "Run `python -m pytest` from repo root and ensure it passes, or document why no test suite exists yet",
-        "Keep the implementation scoped to this story; do not implement downstream stories early",
-        "Commit only this story's code changes with a focused conventional commit message"
-      ],
-      "priority": 3,
-      "passes": true,
-      "notes": "Source issue: .scratch/distributed-inference-network/issues/03-tracker-registration-and-routing.md",
-      "dependsOn": [
-        "US-002"
-      ],
-      "completionNotes": "Completed by Ralph iteration 79796dd2; verified by pytest, compileall, editable installs, CLI help, and spec/code reviews.",
-      "status": "done"
-    },
-    {
-      "id": "US-004",
-      "title": "04 \u2014 Node client startup flow (`meshnet-node start`)",
-      "description": "Make `meshnet-node start` self-configuring from scratch on a machine with a CUDA-capable GPU (or CPU fallback). The full startup sequence must complete without any manual configuration: 1. Detect GPU model and available VRAM 2. Load an existing Solana wallet from disk, or generate and save a new one 3. Query the tracker for the optimal shard assignment given the hardware profile 4. Download the assigned shard from HuggingFace (`huggingface_hub`) 5. Register with the tracker (wallet address, endpoint, shard range, hardware profile) 6. Begin accepting inference connections The node prints a short status summary on startup: wallet address, assigned shard, model preset, and download progress. No interactive prompts \u2014 the entire flow is non-interactive. This is the primary viral growth vector. Every second of friction in this flow costs node operators. The startup sequence must work on Linux with CUDA, and degrade gracefully to CPU on machines without a GPU.",
-      "acceptanceCriteria": [
-        "`meshnet-node start --tracker http://localhost:8080` completes startup without prompts on a machine with a CUDA GPU",
-        "A Solana wallet keypair is generated and saved to `~/.config/meshnet/wallet.json` if none exists",
-        "The assigned shard is downloaded from HuggingFace and cached to `~/.cache/meshnet/shards/`",
-        "The node registers with the tracker and appears in the tracker's node registry",
-        "The node accepts a live inference connection and processes activation tensors correctly after startup",
-        "On a CPU-only machine, the node starts with a warning and serves a CPU-appropriate shard",
-        "An integration test covers the full startup sequence against a local tracker stub",
-        "Follow TDD where code is added: add/adjust an observable behavior test before implementation when practical",
-        "Run the task-specific tests and ensure they pass",
-        "Run `python -m pytest` from repo root and ensure it passes, or document why no test suite exists yet",
-        "Keep the implementation scoped to this story; do not implement downstream stories early",
-        "Commit only this story's code changes with a focused conventional commit message"
-      ],
-      "priority": 4,
-      "passes": true,
-      "notes": "Source issue: .scratch/distributed-inference-network/issues/04-node-client-startup.md",
-      "dependsOn": [
-        "US-003"
-      ],
-      "completionNotes": "Completed by Ralph iteration 86510a10; verified by pytest, compileall, editable installs, CLI help, and final review.",
-      "status": "done"
-    },
-    {
-      "id": "US-005",
-      "title": "05 \u2014 OpenAI-compatible gateway",
-      "description": "Expose a production-shape HTTP API from the gateway so that any client using the OpenAI Python SDK (or any OpenAI-compatible tool) works by changing only the `base_url`. The gateway translates OpenAI-format requests into inference route execution and streams responses back in OpenAI-format. Endpoints to implement: - `POST /v1/chat/completions` \u2014 streaming (`text/event-stream`) and non-streaming - `GET /v1/models` \u2014 returns the list of model presets currently routable on the network - `GET /v1/health` \u2014 liveness check The gateway selects an inference route from the tracker, executes the shard pipeline, and assembles the streamed response. If the tracker returns no route for the requested model, the gateway responds with a standard OpenAI-format error (`model_not_available`). Authentication (API key \u2192 SOL/USDC balance) is a stub in this issue \u2014 return 200 for any non-empty `Authorization` header. Real payment gating comes in issue 06.",
-      "acceptanceCriteria": [
-        "`openai.OpenAI(base_url=\"http://localhost:8080/v1\", api_key=\"test\").chat.completions.create(model=\"stub-model\", messages=[...])` returns a valid response",
-        "Streaming works: `stream=True` returns `text/event-stream` chunks in OpenAI SSE format",
-        "`GET /v1/models` returns a JSON array of available model preset names",
-        "A request for an unavailable model returns an OpenAI-format error response with HTTP 503",
-        "LangChain `ChatOpenAI(base_url=..., api_key=...)` works against the gateway",
-        "An integration test covers streaming and non-streaming paths end-to-end through a real tracker and two stub nodes",
-        "Follow TDD where code is added: add/adjust an observable behavior test before implementation when practical",
-        "Run the task-specific tests and ensure they pass",
-        "Run `python -m pytest` from repo root and ensure it passes, or document why no test suite exists yet",
-        "Keep the implementation scoped to this story; do not implement downstream stories early",
-        "Commit only this story's code changes with a focused conventional commit message"
-      ],
-      "priority": 5,
-      "passes": true,
-      "notes": "Source issue: .scratch/distributed-inference-network/issues/05-openai-compatible-gateway.md",
-      "dependsOn": [
-        "US-003"
-      ],
-      "completionNotes": "Completed by agent",
-      "status": "to-revise",
-      "status_reason": "Gateway is currently the pipeline orchestrator. US-014 moves orchestration to tracker-nodes; gateway becomes a thin load-balancer proxy. Implementation will be superseded \u2014 defer rework until US-014 lands."
-    },
-    {
-      "id": "US-006",
-      "title": "06 \u2014 Solana stake + settlement contracts",
-      "description": "Deploy and integrate the Solana smart contracts that make node staking, client payment, and token reward settlement trustless. All development and testing targets **Solana testnet** \u2014 never devnet or mainnet during development, to avoid real costs.",
-      "acceptanceCriteria": [
-        "All contracts deploy successfully to Solana testnet",
-        "A node can submit a stake transaction and have its balance reflected in the registry contract",
-        "A client can fund an API key account with testnet SOL",
-        "After a completed inference session, compute attribution is recorded on-chain with correct node/layer attribution",
-        "The epoch settlement transaction correctly distributes token rewards to node operators and deducts client balances",
-        "The gateway refuses to route to a node whose stake balance is below the minimum threshold",
-        "All contract interactions in tests run against a local Solana test validator (via `solana-test-validator`) \u2014 no live testnet required for CI",
-        "A `.env.testnet` config points to Solana testnet RPC for manual end-to-end testing",
-        "python -m pytest passes from repo root",
-        "Commit only this story's changes"
-      ],
-      "priority": 6,
-      "passes": true,
-      "notes": "Source issue: .scratch/distributed-inference-network/issues/06-solana-stake-and-settlement.md",
-      "dependsOn": [
-        "US-003"
-      ],
-      "completionNotes": "Completed by fresh Ralph/Codex session c257ffde with controller patches for clarified economics; verified by pytest, compileall, and diff check.",
-      "status": "done"
-    },
-    {
-      "id": "US-007",
-      "title": "07 \u2014 Fraud detection: validator + on-chain slash",
-      "description": "Implement the optimistic fraud detection loop. After each inference request completes, the validator process independently decides whether to re-run the request on a reference node (~5% sample rate).",
-      "acceptanceCriteria": [
-        "The validator process samples ~5% of completed inference requests (configurable)",
-        "A node returning a deliberately wrong output is detected and slashed within one validation cycle",
-        "The on-chain stake balance of the slashed node decreases by the correct slash amount",
-        "The strike count for the slashed node increments on-chain",
-        "A node that reaches the strike threshold is excluded from route selection on the next gateway request",
-        "A slashed node logs a clear warning to stdout",
-        "An integration test: run a deliberately-bad node, send 20 requests, assert at least one slash transaction is submitted and the node is eventually excluded from routes",
-        "python -m pytest passes from repo root",
-        "Commit only this story's changes"
-      ],
-      "priority": 7,
-      "passes": true,
-      "notes": "Source issue: .scratch/distributed-inference-network/issues/07-fraud-detection-slash.md",
-      "dependsOn": [
-        "US-005",
-        "US-006"
-      ],
-      "completionNotes": "Completed by fresh Ralph/Codex session 04475912 with controller fix for duplicate slash suppression; verified by pytest, compileall, and diff check.",
-      "status": "done"
-    },
-    {
-      "id": "US-008",
-      "title": "08 \u2014 Node probationary period + ban enforcement",
-      "description": "Implement the two anti-sybil mechanisms that make re-entering the network after a ban economically costly.",
-      "acceptanceCriteria": [
-        "A node with a new wallet receives no token rewards for its first N jobs (verified via settlement contract state)",
-        "The node client prints remaining probationary jobs on startup",
-        "After N jobs, the next epoch settlement correctly credits the node with token rewards",
-        "A wallet with strike count at the threshold is marked banned in the registry contract",
-        "The tracker excludes banned wallets from route selection",
-        "A banned wallet that attempts to register with the tracker is rejected",
-        "An integration test covers: new wallet \u2192 N jobs \u2192 earning begins; and: strike threshold reached \u2192 banned \u2192 excluded from routes",
-        "python -m pytest passes from repo root",
-        "Commit only this story's changes"
-      ],
-      "priority": 8,
-      "passes": true,
-      "notes": "Source issue: .scratch/distributed-inference-network/issues/08-probationary-period-and-bans.md",
-      "dependsOn": [
-        "US-007"
-      ],
-      "completionNotes": "Completed by fresh Ralph/Codex session db3f5c10 with controller test fix for banned registration semantics; verified by pytest, compileall, and diff check.",
-      "status": "done"
-    },
-    {
-      "id": "US-009",
-      "title": "09 \u2014 P2P shard swarm",
-      "description": "Once a node has downloaded a shard, it seeds that shard to other nodes that are assigned the same shard.",
-      "acceptanceCriteria": [
-        "A node that has downloaded a shard is listed as a peer for that shard in the tracker",
-        "A second node assigned the same shard downloads it from the first node (peer) rather than HuggingFace",
-        "If no peers are available, the node falls back to HuggingFace without error",
-        "The node client logs whether the shard was downloaded from a peer or HuggingFace",
-        "Shard chunks are verified against a checksum before being marked complete",
-        "An integration test: node A downloads shard from HuggingFace stub; node B with same assignment downloads from node A",
-        "python -m pytest passes from repo root",
-        "Commit only this story's changes"
-      ],
-      "priority": 9,
-      "passes": true,
-      "notes": "Source issue: .scratch/distributed-inference-network/issues/09-p2p-shard-swarm.md",
-      "dependsOn": [
-        "US-004"
-      ],
-      "completionNotes": "Completed by fresh Ralph/Codex session 243fae88 with controller fix for streaming tar archives; verified by pytest, compileall, and diff check.",
-      "status": "done"
-    },
-    {
-      "id": "US-010",
-      "title": "10 \u2014 `meshnet` Python SDK",
-      "description": "A Python SDK (`packages/sdk`) that wraps the OpenAI-compatible gateway and exposes network-specific controls.",
-      "acceptanceCriteria": [
-        "`pip install meshnet` installs the SDK",
-        "`Client.chat.completions.create(...)` works identically to the OpenAI SDK",
-        "`client.wallet.balance()` returns the current SOL/USDC balance for the API key",
-        "`client.wallet.top_up()` returns a valid Solana payment address",
-        "`client.models.available()` returns model presets with shard coverage percentage",
-        "`client.estimate_cost(model, tokens)` returns a cost estimate in SOL",
-        "`client.request(redundancy=2)` sends to two independent inference routes and returns majority response",
-        "The SDK is typed (py.typed, full type stubs)",
-        "An integration test covers each SDK method against a local gateway + tracker + stub nodes",
-        "python -m pytest passes from repo root",
-        "Commit only this story's changes"
-      ],
-      "priority": 10,
-      "passes": true,
-      "notes": "Source issue: .scratch/distributed-inference-network/issues/10-meshnet-sdk.md",
-      "dependsOn": [
-        "US-005",
-        "US-006"
-      ],
-      "completionNotes": "Completed by fresh Ralph/Codex session ef4eea0e; verified by pytest, compileall, editable SDK install, and diff check.",
-      "status": "done"
-    },
-    {
-      "id": "US-011",
-      "title": "11 \u2014 Binary activation wire format",
-      "description": "Replace the base64 JSON activation payload with raw binary HTTP bodies, zstd compression, and chunked prefill (128 tokens/chunk). All nodes and the gateway must be migrated. Stub nodes continue to emit zeroed tensors, just in binary. This is a protocol prerequisite for US-012 (real model backend).",
-      "acceptanceCriteria": [
-        "Node /forward endpoint reads shape/dtype/session/chunk from HTTP headers; body is raw binary (optionally zstd-compressed)",
-        "Node /forward response is raw binary with the same header set",
-        "Gateway splits prompts > MESHNET_CHUNK_TOKENS (default 128) into sequential chunks sent through the pipeline",
-        "Integration test: 512-token stub activation (4 chunks) through a two-node pipeline returns 4 valid binary chunk responses",
-        "zstd Python package added as a dependency to packages/node and packages/gateway",
-        "_make_stub_activations replaced with _make_stub_binary_activation(shape, dtype) -> bytes",
-        "python -m pytest passes from repo root",
-        "Commit only this story's changes"
-      ],
-      "priority": 11,
-      "passes": true,
-      "notes": "Source issue: .scratch/distributed-inference-network/issues/11-binary-wire-format.md",
-      "dependsOn": [
-        "US-002"
-      ],
-      "completionNotes": "Completed by fresh Ralph/Codex session 3f3bed75; verified by pytest, compileall, and diff check.",
-      "status": "done"
-    },
-    {
-      "id": "US-012",
-      "title": "12 \u2014 Real PyTorch model backend",
-      "description": "Replace stub node inference with actual transformers layer execution. Node loads HuggingFace SafeTensors model shard (model.model.layers[start:end]), runs real forward passes in bfloat16, handles head (embed_tokens) and tail (lm_head) responsibilities, and supports bitsandbytes NF4/INT8/bfloat16 quantization via --quantization flag. Test model: openai-community/gpt2.",
-      "acceptanceCriteria": [
-        "meshnet-node start --model-id openai-community/gpt2 --shard-start 0 --shard-end 6 loads the shard without error",
-        "Head node (shard_start==0) loads tokenizer and embed_tokens",
-        "Tail node (shard_end==total_layers) loads model.norm and lm_head",
-        "Two-node GPT-2 integration test returns deterministic coherent text completion",
-        "--quantization [bfloat16|int8|nf4] flag supported; bfloat16 default",
-        "Node with insufficient VRAM prints clear error and exits",
-        "transformers, bitsandbytes, safetensors, accelerate added as node dependencies",
-        "Integration tests marked @pytest.mark.integration, skipped in CI without GPU",
-        "python -m pytest passes from repo root",
-        "Commit only this story's changes"
-      ],
-      "priority": 12,
-      "passes": true,
-      "notes": "Source issue: .scratch/distributed-inference-network/issues/12-real-pytorch-model-backend.md",
-      "dependsOn": [
-        "US-011"
-      ],
-      "completionNotes": "Completed by agent",
-      "status": "done"
-    },
-    {
-      "id": "US-013",
-      "title": "13 \u2014 Coverage-first tracker shard assignment",
-      "description": "Upgrade tracker route selection to coverage-first, speed-weighted bin-packing. Tracker maintains a live coverage map per model, issues LOAD_SHARD/DROP_SHARD rebalance directives when coverage drops, and assigns shard ranges using declared VRAM, quantization, and benchmark throughput. A model is only routable when all layer ranges have node_count >= 1.",
-      "acceptanceCriteria": [
-        "Node registration accepts vram_bytes, ram_bytes, quantizations[], benchmark_tokens_per_sec",
-        "GET /v1/coverage/<model_preset> returns list of {start_layer, end_layer, node_count}",
-        "Model is unroutable when any layer range has node_count=0",
-        "New node gets assigned to highest-priority uncovered range first",
-        "Node disconnection triggers LOAD_SHARD directive to idle node within 30 seconds",
-        "Shard assignment respects VRAM: assigned_layers <= floor((vram_bytes * 0.8) / bytes_per_layer_at_quant)",
-        "Faster node receives wider shard range when both can cover the same gap",
-        "Integration test: three nodes with different VRAM show widest range on largest-VRAM node with 100% coverage",
-        "Integration test: kill middle-range node \u2192 tracker issues LOAD_SHARD \u2192 coverage recovers",
-        "python -m pytest passes from repo root",
-        "Commit only this story's changes"
-      ],
-      "priority": 13,
-      "passes": true,
-      "notes": "Source issue: .scratch/distributed-inference-network/issues/13-coverage-first-shard-assignment.md",
-      "dependsOn": [
-        "US-003",
-        "US-012"
-      ],
-      "completionNotes": "Completed by agent",
-      "status": "done"
-    },
-    {
-      "id": "US-014",
-      "title": "14 \u2014 Tracker-as-first-layer-node (inference entry point)",
-      "description": "Merge inference orchestration into tracker nodes that serve the first-layer shard. A tracker node exposes /v1/chat/completions directly: tokenizes input, runs embed_tokens + layers[0..k], selects onward route from coverage map, forwards binary activations, receives tail output, and streams tokens back. The standalone gateway becomes a thin load-balancer routing to tracker nodes.",
-      "acceptanceCriteria": [
-        "Node with shard_start==0 (or --tracker-mode flag) exposes /v1/chat/completions alongside /forward",
-        "Tracker-node /v1/chat/completions: tokenize \u2192 embed \u2192 own layers \u2192 route selection \u2192 forward binary \u2192 receive tail \u2192 stream SSE",
-        "Two tracker nodes for same model each handle requests independently",
-        "Gateway proxies /v1/chat/completions to tracker nodes (round-robin), discovered via GET /v1/tracker-nodes/<model_preset>",
-        "Integration test: two tracker nodes + two mid-shard nodes for GPT-2; 10 requests via gateway; both tracker nodes receive load; all responses valid",
-        "Existing US-005 OpenAI-compatible tests still pass",
-        "python -m pytest passes from repo root",
-        "Commit only this story's changes"
-      ],
-      "priority": 14,
-      "passes": true,
-      "notes": "Source issue: .scratch/distributed-inference-network/issues/14-tracker-as-node.md",
-      "dependsOn": [
-        "US-012",
-        "US-013"
-      ],
-      "status": "done",
-      "completionNotes": "Implemented: (1) Tracker GET /v1/tracker-nodes/<model> endpoint returning nodes registered with tracker_mode=true at shard_start==0. (2) StubNodeServer and TorchNodeServer now accept tracker_mode/tracker_url params; when tracker_mode=True, /v1/chat/completions is served alongside /forward. (3) TorchNodeServer auto-detects tracker mode when shard_start==0. (4) Gateway _handle_chat_completions checks for tracker-nodes first via _get_tracker_nodes(), proxies round-robin if found, falls back to existing direct pipeline if none (backward compat). (5) CLI --tracker-mode and --tracker-url flags added. (6) Integration test: 2 tracker-nodes + 2 mid-shard nodes for gpt2; 10 requests; round-robin verified (5/5 split); all responses valid OpenAI format. All 78 tests pass."
-    },
-    {
-      "id": "US-015",
-      "title": "15 \u2014 Ralph: agent-agnostic runner + status-field-aware dashboard",
-      "description": "Two improvements to the Ralph workflow tooling. (1) Status-field awareness: replace all passes:true/false reads in ralph_progress.py with the rich status field. Surface to-revise and needs-review stories as an attention list in the dashboard; auto command skips them by default. (2) Agent agnosticism: make the agent selectable per session \u2014 codex (existing), claude (Claude Code CLI), openrouter (unified API for GPT-4/Mistral/Llama/DeepSeek via OPENROUTER_API_KEY + --model), or custom (--agent-cmd path to any script). Persist agent choice to .ralph-tui/agent-config.json. When ralph-tui does not natively support a requested agent, ralph_progress.py falls back to a thin adapter that calls the agent API directly with the task prompt. (3) Worktree parallelism: ralph_progress.py auto --parallel [N] creates one git worktree per ready task (git worktree add ../AI-worktree-<id> -b feat/<id>), runs up to N agent sessions concurrently, and merges each branch back to master after the agent exits 0 and tests pass. Non-blocking stories that share no file overlap can safely run in parallel. ralph_progress.py list-parallel shows which open stories have no overlapping dependsOn chains and are safe to run concurrently.",
-      "acceptanceCriteria": [
-        "All story.get('passes') reads in ralph_progress.py replaced with _is_done() helper that reads status field with passes fallback",
-        "_story_sets() returns six buckets: done, attention (to-revise/needs-review), active (in-progress), in-design, ready, blocked",
-        "Dashboard shows \u26a0 Attention required section listing to-revise/needs-review stories with status_reason",
-        "auto command skips to-revise and needs-review stories; --include-revise flag overrides",
-        "_review_report includes Attention Required section with status_reason for all affected stories",
-        "ralph_progress.py set-agent --agent <name> [--model <model>] writes .ralph-tui/agent-config.json",
-        "--agent codex|claude|openrouter|custom accepted by run-next, auto, review subcommands",
-        "run-next --agent custom --agent-cmd ./my-agent.sh runs a task via custom script (prompt file as $1)",
-        "Saved agent config is loaded as default when --agent is not passed on the CLI",
-        "python -m pytest passes from repo root",
-        "ralph_progress.py auto --parallel 2 starts two worktrees concurrently for the two highest-priority ready tasks",
-        "Each worktree is created at ../AI-worktree-<story-id> on branch feat/<story-id>",
-        "After agent exits 0 and pytest passes in the worktree, the branch is merged to master and the worktree removed",
-        "ralph_progress.py list-parallel prints the set of open stories with no shared dependency chain (safe to parallelize)",
-        "If a worktree merge fails (conflict), the worktree is preserved for manual resolution and reported clearly"
-      ],
-      "priority": 15,
-      "passes": true,
-      "status": "done",
-      "notes": "Source issue: .scratch/distributed-inference-network/issues/15-ralph-agent-agnostic-status-aware.md",
-      "dependsOn": [],
-      "completionNotes": "Implemented by agent: status-aware helpers (_is_done, _needs_attention, _is_active, _is_in_design), 6-bucket _story_sets, attention dashboard section, _review_report Attention Required block, auto --include-revise, set-agent subcommand with persistent agent-config.json, _run_openrouter stub, custom agent support, list-parallel subcommand, and auto --parallel N worktree orchestration. All 65 tests pass."
-    }
-  ],
-  "metadata": {
-    "updatedAt": "2026-06-29T13:30:00.000Z",
-    "statusVocabulary": {
-      "open": "Not started",
-      "in-design": "Decisions pending before implementation can begin",
-      "in-progress": "Agent actively working on it",
-      "needs-review": "Implemented, awaiting human review or verification",
-      "to-revise": "Previously done, needs rework due to design or architecture changes",
-      "done": "Implemented, tested, and verified",
-      "blocked": "Waiting on unresolved external dependency"
-    }
-  }
-}

.vscode/launch.json

@@ -0,0 +1,104 @@
+{
+  "version": "0.2.0",
+  "configurations": [
+    {
+      "name": "Tracker: start local (8080)",
+      "type": "debugpy",
+      "request": "launch",
+      "module": "meshnet_tracker.cli",
+      "args": [
+        "start",
+        "--host",
+        "0.0.0.0",
+        "--port",
+        "8080"
+      ],
+      "console": "integratedTerminal",
+      "justMyCode": false,
+      "env": {
+        "PYTHONPATH": "${workspaceFolder}/packages/tracker:${workspaceFolder}/packages/node:${workspaceFolder}/packages/relay:${workspaceFolder}/packages/gateway:${workspaceFolder}/packages/p2p:${workspaceFolder}/packages/sdk:${workspaceFolder}/packages/validator:${env:PYTHONPATH}"
+      }
+    },
+    {
+      "name": "Tracker: start public + relay (8081)",
+      "type": "debugpy",
+      "request": "launch",
+      "module": "meshnet_tracker.cli",
+      "args": [
+        "start",
+        "--host",
+        "0.0.0.0",
+        "--port",
+        "8081",
+        "--relay-url",
+        "wss://ai.neuron.d-popov.com/ws"
+      ],
+      "console": "integratedTerminal",
+      "justMyCode": false,
+      "env": {
+        "PYTHONPATH": "${workspaceFolder}/packages/tracker:${workspaceFolder}/packages/node:${workspaceFolder}/packages/relay:${workspaceFolder}/packages/gateway:${workspaceFolder}/packages/p2p:${workspaceFolder}/packages/sdk:${workspaceFolder}/packages/validator:${env:PYTHONPATH}"
+      }
+    },
+    {
+      "name": "Node: dashboard UI (saved config)",
+      "type": "debugpy",
+      "request": "launch",
+      "module": "meshnet_node.cli",
+      "args": [
+        "--tracker",
+        "http://localhost:8080",
+        "--model",
+        "stub-model",
+        "--port",
+        "7000",
+        "--debug"
+      ],
+      "console": "integratedTerminal",
+      "justMyCode": false,
+      "env": {
+        "PYTHONPATH": "${workspaceFolder}/packages/tracker:${workspaceFolder}/packages/node:${workspaceFolder}/packages/relay:${workspaceFolder}/packages/gateway:${workspaceFolder}/packages/p2p:${workspaceFolder}/packages/sdk:${workspaceFolder}/packages/validator:${env:PYTHONPATH}"
+      }
+    },
+    {
+      "name": "Node: start local stub (no dashboard)",
+      "type": "debugpy",
+      "request": "launch",
+      "module": "meshnet_node.cli",
+      "args": [
+        "start",
+        "--tracker",
+        "http://localhost:8080",
+        "--model",
+        "stub-model",
+        "--host",
+        "0.0.0.0",
+        "--port",
+        "7001",
+        "--debug"
+      ],
+      "console": "integratedTerminal",
+      "justMyCode": false,
+      "env": {
+        "PYTHONPATH": "${workspaceFolder}/packages/tracker:${workspaceFolder}/packages/node:${workspaceFolder}/packages/relay:${workspaceFolder}/packages/gateway:${workspaceFolder}/packages/p2p:${workspaceFolder}/packages/sdk:${workspaceFolder}/packages/validator:${env:PYTHONPATH}"
+      }
+    }
+  ],
+  "compounds": [
+    {
+      "name": "Local mesh: tracker + node UI",
+      "configurations": [
+        "Tracker: start local (8080)",
+        "Node: dashboard UI (saved config)"
+      ],
+      "stopAll": true
+    },
+    {
+      "name": "Local mesh: tracker + stub node",
+      "configurations": [
+        "Tracker: start local (8080)",
+        "Node: start local stub (no dashboard)"
+      ],
+      "stopAll": true
+    }
+  ]
+}

AGENTS.md

@@ -1,8 +1,12 @@
+## Memory
+
+Persistent memory lives in `.claude/memory/`. Read `MEMORY.md` there at the start of every session for project context, user preferences, and open work. Write updates back to those files so knowledge carries across devices and sessions.
+
 ## Agent skills
 
 ### Issue tracker
 
-Issues live as local markdown files under `.scratch/<feature-slug>/`. See `docs/agents/issue-tracker.md`.
+Active feature work lives under `.scratch/<feature-slug>/`. Completed features are merged into `docs/issues/` via `/close-feature`. See `docs/agents/issue-tracker.md`.
 
 ### Triage labels
 

CONTEXT.md

@@ -51,9 +51,37 @@ _Avoid_: reputation, rating, rank
 ### Payments & fraud
 
 **Stake**:
-Tokens a node locks as collateral that can be slashed for fraud. Stake protects the network economically, but route selection is not based on a node's token balance.
+Collateral a node stands to lose for fraud. In the current design the node's Pending Balance serves as stake — no upfront deposit is required. An optional USDT/TAI deposit may return later for routing priority.
 _Avoid_: deposit, bond, escrow
 
+**Treasury**:
+The single project-owned Solana wallet that custodially holds client deposits, pays node payouts, and accumulates the Protocol Cut. Its keypair is loaded only on settlement-capable trackers.
+_Avoid_: escrow, vault, hot wallet
+
+**Pending Balance**:
+A node's accrued, not-yet-paid USDT earnings on the tracker ledger. Doubles as the node's fraud collateral: it is forfeited in full when a validator catches a divergent output.
+_Avoid_: unpaid rewards, accrual, balance due
+
+**Settlement Period**:
+The dynamic interval driving on-chain payouts: a node is paid when its Pending Balance exceeds the Payout Threshold or the period elapses, whichever comes first. Short in development (seconds), long in production (daily), configurable to grow with volume.
+_Avoid_: epoch, payout cycle, billing cycle
+
+**Payout Threshold**:
+The minimum Pending Balance that triggers an immediate payout before the Settlement Period elapses. Includes a dust floor so payouts are never smaller than they are worth.
+_Avoid_: minimum payout, dust limit
+
+**Protocol Cut**:
+The 10% of inference fees retained by the project for infrastructure; the remaining 90% goes to the nodes that served the request. Accumulates in the Treasury as the future TAI liquidity reserve.
+_Avoid_: spread, commission, house fee
+
+**Deposit Watcher**:
+The tracker component that observes the Treasury's on-chain USDT deposits and credits the sending client's API-key ledger balance.
+_Avoid_: payment listener, chain scanner
+
+**Mock USDT**:
+The self-created 6-decimal SPL mint that stands in for USDT on devnet, where real USDT does not exist. The mint address is configuration, so mainnet cutover is a config change.
+_Avoid_: test token, fake USDT, devnet dollar
+
 **Tax**:
 The share of caller payments distributed to compute nodes as rewards. Taxes are weighted by completed work and historical node speed so faster, larger nodes earn proportionally more.
 _Avoid_: fee, toll, commission
@@ -67,8 +95,8 @@ Work a compute node performs without earning immediate rewards, usually during p
 _Avoid_: unpaid labor, warmup request
 
 **Slash**:
-The act of reducing a node's stake as a penalty for a proven fraud incident.
+The penalty for a proven fraud incident: the node's entire Pending Balance is forfeited to the Treasury and a Strike is recorded.
-_Avoid_: penalize, burn, fine
+_Avoid_: penalize, burn, fine, forfeit
 
 **Strike**:
 A fraud incident recorded on-chain against a node. Enough strikes result in a ban.
@@ -83,7 +111,7 @@ The first N jobs a new wallet must complete without earning, to raise the cost o
 _Avoid_: trial period, warmup, grace period
 
 **Token**:
-Our native Solana L2 token. Used by nodes for staking and received as inference rewards. Clients never need to hold it.
+TAI, our native Solana SPL token. Deferred (ADR-0015): nodes are currently paid directly in USDT; TAI returns as the reward/upside layer once volume exists, funded by the accumulated Protocol Cut. Clients never need to hold it.
 _Avoid_: coin, reward token, native token
 
 **Contract Boundary**:
@@ -105,7 +133,7 @@ _Avoid_: accusation, report, claim
 ### Client-facing
 
 **Client**:
-Any application or user that sends inference requests to the gateway. Pays in SOL or USDC.
+Any application or user that sends inference requests to the gateway. Prepays USDT into the Treasury; each request is metered against the resulting ledger balance at a per-1K-tokens price set per model.
 _Avoid_: user, caller, consumer
 
 **Model Preset**:

QUICKSTART.md

@@ -0,0 +1,717 @@
+# Quickstart — Running a node and testing inference
+
+This guide gets you from zero to a live inference request in three terminals.
+Tested on: AMD Ryzen AI Max (Strix Halo APU), 124 GB RAM, Linux, CPU inference.
+
+---
+
+## Prerequisites
+
+```bash
+# Clone and enter repo
+cd /run/media/popov/d/DEV/repos/d-popov.com/AI
+
+# Create the virtualenv if it does not exist yet
+python3 -m venv .venv
+
+# Keep packaging tools current enough for editable installs
+.venv/bin/python -m pip install --upgrade pip setuptools wheel
+
+# Install Python packages (editable — picks up code changes immediately)
+.venv/bin/pip install -e packages/tracker -e packages/node -e packages/p2p -e packages/gateway -e packages/relay
+
+# CPU-only PyTorch (skip if you have CUDA/ROCm already)
+.venv/bin/pip install torch --index-url https://download.pytorch.org/whl/cpu
+
+# HuggingFace model libraries
+.venv/bin/pip install transformers accelerate
+```
+
+> **NVIDIA GPU (CUDA):** replace the torch line with `pip install torch` (default index).  
+> **AMD GPU (ROCm):** `pip install torch --index-url https://download.pytorch.org/whl/rocm6.2`
+
+## Bootstrap a tracker on a new machine
+
+Use this when provisioning a fresh LAN/public tracker host. The tracker itself is
+lightweight; install the relay too if nodes will connect from NAT, WSL2, mobile,
+or other networks where inbound node ports are not reachable.
+
+```bash
+# 1. Get the repo onto the tracker host
+git clone https://git.d-popov.com/popov/neuron-tai.git AI
+cd AI
+
+# 2. Create an isolated Python environment
+python3 -m venv .venv
+.venv/bin/python -m pip install --upgrade pip setuptools wheel
+
+# 3. Install only the services needed by the tracker host
+.venv/bin/pip install -e packages/tracker -e packages/relay -e packages/gateway
+```
+
+For a private LAN tracker, start only the tracker and open the selected TCP port
+on the host firewall if other machines will join:
+
+```bash
+.venv/bin/meshnet-tracker start --host 0.0.0.0 --port 8080
+```
+
+Verify from the tracker host:
+
+```bash
+curl -s http://localhost:8080/v1/network/map | python3 -m json.tool
+```
+
+Verify from another LAN machine, replacing the IP with the tracker host's LAN IP:
+
+```bash
+curl -s http://192.168.0.179:8080/v1/network/map | python3 -m json.tool
+```
+
+For a public tracker with relay support, run both services. The relay listens on
+`8765`; the tracker below listens on `8081` and advertises the public WebSocket
+URL that nodes should use for outbound relay connections:
+
+```bash
+# Terminal 1 — relay
+.venv/bin/meshnet-relay --host 0.0.0.0 --port 8765
+
+# Terminal 2 — tracker
+.venv/bin/meshnet-tracker start \
+  --host 0.0.0.0 \
+  --port 8081 \
+  --relay-url wss://ai.neuron.d-popov.com/ws
+```
+
+If this host sits behind Nginx Proxy Manager, point `/` and `/v1/*` at tracker
+port `8081`, and point `/ws` plus `/rpc` at relay port `8765` as shown in the
+public tracker section below. After the proxy is configured, verify the public
+bootstrap endpoint:
+
+```bash
+curl -s https://ai.neuron.d-popov.com/v1/network/map | python3 -m json.tool
+```
+
+Nodes can then join with either the LAN tracker URL or the public URL:
+
+```bash
+.venv/bin/meshnet-node start --tracker http://192.168.0.179:8080 --model Qwen/Qwen2.5-0.5B-Instruct
+.venv/bin/meshnet-node start --tracker https://ai.neuron.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct
+```
+
+### Windows / WSL2
+
+Run the Linux commands from WSL, not Git Bash. From the repo opened in Git Bash:
+
+```bash
+wsl
+cd /mnt/d/DEV/workspace/REPOS/git.d-popov.com/neuron-tai
+python3 -m venv .venv
+.venv/bin/python -m pip install --upgrade pip setuptools wheel
+.venv/bin/pip install -e packages/tracker -e packages/node -e packages/p2p -e packages/gateway -e packages/relay
+.venv/bin/pip install torch --index-url https://download.pytorch.org/whl/cpu
+.venv/bin/pip install transformers accelerate
+.venv/bin/meshnet-node --help
+```
+
+If `.venv/bin/meshnet-node` is missing, the editable install step did not finish
+successfully. Re-run the `.venv/bin/pip install -e ...` command above inside WSL.
+
+WSL2 sits behind Windows NAT and is **not directly reachable** from other LAN machines.
+Direct cross-host hops time out. The relay path (see below) solves this: the WSL2 node
+opens an outbound WebSocket to the relay server and all traffic flows through that tunnel.
+No firewall rules, no `--advertise-host` needed — just point at the public tracker URL.
+
+### Native Windows PowerShell node (not WSL)
+
+Use this when the tracker is on another machine and you want Windows to host a
+reachable node on the LAN.
+
+#### Option A — existing conda/miniforge environment with CUDA torch (recommended if you already have it)
+
+First, make sure the conda base environment is active so that `python` and `pip` both
+resolve to the same miniforge installation:
+
+```powershell
+conda activate base
+deactivate   # drop any .venv that may be layered on top; safe no-op if none active
+```
+
+Install project packages into the active conda/miniforge env:
+
+```powershell
+cd D:\DEV\workspace\REPOS\git.d-popov.com\neuron-tai
+
+pip install -e packages\tracker -e packages\node -e packages\p2p -e packages\gateway -e packages\relay
+pip install transformers accelerate safetensors   # torch is already present
+```
+
+Verify torch is importable and CUDA is live **before** starting the node:
+
+```powershell
+python -c "import torch; print(torch.__version__, torch.cuda.is_available())"
+# Expected: 2.x.x+cuXXX  True
+```
+
+If you get `ModuleNotFoundError: No module named 'torch'` even though `pip install torch`
+says "already satisfied", the `torch/` package directory is missing while the metadata
+stub remains (can happen after a conda-managed install). Force-reinstall all three
+PyTorch packages together so their versions stay in sync:
+
+```powershell
+pip install --force-reinstall torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
+```
+
+> **Important:** always reinstall `torch`, `torchvision`, and `torchaudio` as a group.
+> Upgrading only `torch` leaves `torchvision` on an incompatible version, which causes
+> `RuntimeError: operator torchvision::nms does not exist` and makes transformers fail
+> to import any model class (the error surfaces as `Could not import module 'Qwen2ForCausalLM'`).
+
+Then re-run the verify step above.
+
+If that prints `True` but `meshnet-node` still can't find torch, the venv entry point
+is shadowing the conda one. Check which binary wins:
+
+```powershell
+(Get-Command meshnet-node).Source
+# Should show: C:\Users\<you>\miniforge3\Scripts\meshnet-node.exe
+# If it shows .venv\Scripts\meshnet-node.exe, use the full path below instead
+```
+
+To start a node:
+
+```powershell
+$env:HF_HOME = "D:\DEV\models"
+meshnet-node start --tracker https://ai.neuron.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct
+```
+
+If the wrong entry point is shadowing, invoke via the full conda path:
+
+```powershell
+C:\Users\popov\miniforge3\Scripts\meshnet-node.exe start `
+  --tracker https://ai.neuron.d-popov.com `
+  --model Qwen/Qwen2.5-0.5B-Instruct
+```
+
+#### Option B — isolated virtualenv (fresh machine, no existing torch)
+
+1. Install prerequisites on Windows:
+   - Python 3.11 or 3.12 from <https://www.python.org/downloads/windows/>
+   - Git for Windows from <https://git-scm.com/download/win>
+
+2. Open **PowerShell** in the cloned repo and install the node packages:
+
+```powershell
+# Example repo path; adjust to wherever you cloned it
+cd D:\DEV\workspace\REPOS\git.d-popov.com\neuron-tai
+
+python -m venv .venv
+.\.venv\Scripts\python.exe -m pip install --upgrade pip setuptools wheel
+.\.venv\Scripts\pip.exe install -e packages\tracker -e packages\node -e packages\p2p -e packages\gateway -e packages\relay
+
+# CPU-only PyTorch. For NVIDIA CUDA, use `pip install torch` instead.
+.\.venv\Scripts\pip.exe install torch --index-url https://download.pytorch.org/whl/cpu
+.\.venv\Scripts\pip.exe install transformers accelerate
+
+.\.venv\Scripts\meshnet-node.exe --help
+```
+
+For `start`-specific flags, run:
+
+```powershell
+.\.venv\Scripts\meshnet-node.exe start --help
+```
+
+3. Find the Windows LAN IP address:
+
+```powershell
+ipconfig
+```
+
+Use the IPv4 address on the active Ethernet/Wi-Fi adapter, for example
+`192.168.0.42`. Avoid WSL/Docker/Hyper-V adapter addresses like `172.16.x.x`,
+`172.17.x.x`, or other virtual adapter IPs.
+
+4. Allow inbound traffic for the node port in Windows Firewall. Run PowerShell as
+Administrator once:
+
+```powershell
+New-NetFirewallRule `
+  -DisplayName "Meshnet node 8005" `
+  -Direction Inbound `
+  -Action Allow `
+  -Protocol TCP `
+  -LocalPort 8005
+```
+
+5. Start the Windows node from normal PowerShell. Replace the tracker and
+advertised host values with your actual LAN addresses:
+
+```powershell
+$env:HF_HOME = "D:\DEV\models"
+
+.\.venv\Scripts\meshnet-node.exe start `
+  --tracker http://192.168.0.179:8081 `
+  --model Qwen/Qwen2.5-0.5B-Instruct `
+  --shard-start 12 --shard-end 23 `
+  --quantization bfloat16 `
+  --host 0.0.0.0 `
+  --advertise-host 192.168.0.42 `
+  --port 8005
+```
+
+One-line variants (direct LAN — node must be reachable by IP from other machines):
+
+```powershell
+.\.venv\Scripts\meshnet-node.exe start --tracker http://192.168.0.179:8081 --model Qwen/Qwen2.5-0.5B-Instruct --advertise-host 192.168.0.20
+```
+
+Via public hostname with relay (works from behind NAT, WSL2, 5G — no `--advertise-host` needed):
+
+```powershell
+.\.venv\Scripts\meshnet-node.exe start --tracker https://ai.neuron.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct
+```
+
+WSL (same relay path — no `--advertise-host`):
+
+```bash
+.venv/bin/meshnet-node start --tracker https://ai.neuron.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct
+.venv/bin/meshnet-node start --tracker http://192.168.0.179:8081 --model Qwen/Qwen2.5-0.5B-Instruct
+```
+
+`--host 0.0.0.0` binds the node to all Windows interfaces. `--advertise-host`
+is what the tracker gives to other nodes for direct connections; omit it when using
+the relay path since all traffic flows through the relay tunnel instead.
+
+If you want verbose per-hop pipeline logs while debugging a split model, add
+`--debug`. Leave it off for normal runs; otherwise every generated token logs
+lines like:
+
+```text
+  [node] pipeline hop 0: http://127.0.0.1:8005 start_layer=22
+  [node] pipeline hop 0 returned text=' token'
+  [node] pipeline hop 1: wss://ai.neuron.d-popov.com/rpc/abc123 relay start_layer=12
+```
+
+6. From the tracker machine, verify Windows is reachable:
+
+```bash
+curl http://192.168.0.42:8005/v1/health
+```
+
+If that endpoint returns 404 or 501, that is okay: it still proves the TCP
+connection reached the node process. If it times out or connection-refuses, check
+the Windows Firewall rule, `--host 0.0.0.0`, the selected LAN IP, and that the
+node is still running.
+
+---
+
+## Public tracker + relay (internet / NAT nodes)
+
+This setup lets nodes connect from anywhere — behind home NAT, 5G, WSL2, or
+on a different continent — without opening firewall ports.
+
+### Architecture
+
+```
+Client → HTTPS → ai.neuron.d-popov.com (nginx)
+                  ├─ /v1/*   → meshnet-tracker :8081
+                  ├─ /ws     → meshnet-relay :8765  (node persistent outbound WS)
+                  └─ /rpc/*  → meshnet-relay :8765  (caller opens WS per hop)
+```
+
+### Nginx Proxy Manager (Docker)
+
+Use **one** proxy host for the domain. Do not create a second host for the same
+domain to reach another port — path routing is done with **Custom locations** on
+that same host.
+
+**1. Details tab** (default `/` → tracker)
+
+| Field | Value |
+|-------|--------|
+| Domain Names | `ai.neuron.d-popov.com` |
+| Scheme | `http` |
+| Forward Hostname / IP | LAN IP of the tracker machine (e.g. `192.168.0.179`) |
+| Forward Port | `8081` |
+| Websockets Support | ON |
+
+This serves `/v1/network/map`, `/v1/chat/completions`, and the rest of the tracker API.
+
+**2. Custom locations tab** (sub-paths → relay)
+
+The Custom locations form has **no separate Websockets toggle** — only location,
+scheme, forward host, optional sub-folder path, and port. Add **two** locations
+(both pointing at the relay process on port `8765`). Leave **“Add a path for
+sub-folder forwarding”** empty so the full URI reaches the relay
+(`/ws`, `/rpc/<peer_id>`).
+
+Location A — persistent node connections:
+
+| Field | Value |
+|-------|--------|
+| Define location | `/ws` |
+| Scheme | `http` |
+| Forward Hostname / IP | `192.168.0.179` |
+| Forward Port | `8765` |
+| Sub-folder path | *(leave empty)* |
+
+Location B — per-hop RPC:
+
+| Field | Value |
+|-------|--------|
+| Define location | `/rpc` |
+| Scheme | `http` |
+| Forward Hostname / IP | `192.168.0.179` |
+| Forward Port | `8765` |
+| Sub-folder path | *(leave empty)* |
+
+Nginx matches the longer prefixes first: `/ws` and `/rpc/…` go to relay; everything
+else stays on `8081`.
+
+**3. SSL tab**
+
+Use your existing Let’s Encrypt certificate (unchanged).
+
+**4. Advanced tab** (only if WebSocket upgrade fails on `/ws` or `/rpc`)
+
+Custom locations do not expose a Websockets checkbox. If nodes show
+`Relay configured but not connected yet` while `/v1/network/map` works, add this
+snippet on the **proxy host** Advanced tab:
+
+```nginx
+proxy_http_version 1.1;
+proxy_set_header Upgrade $http_upgrade;
+proxy_set_header Connection $http_connection;
+proxy_read_timeout 3600s;
+proxy_send_timeout 3600s;
+```
+
+**5. Verify routing**
+
+```bash
+# Tracker (8081 via default location)
+curl -s https://ai.neuron.d-popov.com/v1/network/map | python3 -m json.tool
+
+# Relay paths should not 502/404 from the tracker — check response headers/status
+curl -sI https://ai.neuron.d-popov.com/ws
+curl -sI https://ai.neuron.d-popov.com/rpc/test-peer
+```
+
+After NPM is correct, start relay and tracker on the LAN machine:
+
+```bash
+# Terminal 1 — relay
+.venv/bin/meshnet-relay --host 0.0.0.0 --port 8765
+
+# Terminal 2 — tracker (advertises relay to nodes)
+.venv/bin/meshnet-tracker start \
+  --host 0.0.0.0 \
+  --port 8081 \
+  --relay-url wss://ai.neuron.d-popov.com/ws
+```
+
+Nodes using `https://ai.neuron.d-popov.com` should then log:
+
+```text
+Relay advertised by tracker — using outbound tunnel wss://ai.neuron.d-popov.com/ws
+  Relay connected — wss://ai.neuron.d-popov.com/rpc/<peer_id>
+```
+
+The `--relay-url` flag embeds the relay address in `/v1/network/map`. Every node
+queries that endpoint on startup and auto-connects if a relay URL is present.
+
+### Start a node (any machine, any network)
+
+No `--advertise-host`, firewall rule, port forwarding, relay URL, or peer URL is
+needed on the node. The public tracker is the only bootstrap URL the user types.
+The node queries the tracker for `/v1/network/map`, discovers the relay URL, and
+opens a persistent outbound WebSocket. If the relay connection drops, the node
+keeps retrying it.
+
+```bash
+.venv/bin/meshnet-node start \
+  --tracker https://ai.neuron.d-popov.com \
+  --model Qwen/Qwen2.5-0.5B-Instruct
+```
+
+No authentication is required in the prototype. The first public node for a model
+must still choose that model with `--model` or a saved wizard config. After at
+least one HF model node is registered, later nodes can join the public network
+with only the tracker URL; the tracker assigns an uncovered shard if one exists:
+
+```bash
+.venv/bin/meshnet-node start --tracker https://ai.neuron.d-popov.com
+```
+
+Use the public tracker to verify registration and routing:
+
+```bash
+curl -s "https://ai.neuron.d-popov.com/v1/network/map" | python3 -m json.tool
+curl -s "https://ai.neuron.d-popov.com/v1/route?model=qwen2.5-0.5b" | python3 -m json.tool
+```
+
+Expected startup output (relay path):
+
+```
+  Auto-detected 24 layers → shard 0–23
+  Relay connected — wss://ai.neuron.d-popov.com/rpc/abc1def2ef3f4567
+================================
+meshnet-node ready
+  Wallet:       <address>
+  Model ID:     Qwen/Qwen2.5-0.5B-Instruct
+  Shard:        layers 0–23; 24 of 24
+  Quantization: bfloat16
+  Endpoint:     http://172.29.104.23:7001
+  Node ID:      <id>
+  Hardware:     CPU
+================================
+```
+
+The `Endpoint` shown is the local IP (unreachable from outside). Other nodes reach
+this one via `wss://ai.neuron.d-popov.com/rpc/<peer_id>` instead.
+
+### How relay hops work
+
+When node A needs to forward activations to node B (behind NAT):
+
+1. Tracker injects `X-Meshnet-Route` with `relay_addr` for each behind-NAT hop.
+2. Node A opens a WebSocket to `wss://relay/rpc/{peer_id_B}`.
+3. Relay forwards the `relay-http-request` envelope to Node B's persistent connection.
+4. Node B processes `/forward` locally, returns `relay-http-response`.
+5. Relay sends the response back to Node A over the same WebSocket.
+6. Node A closes the WebSocket and continues the pipeline.
+
+Binary activation tensors (bfloat16) are Base64-encoded through the relay JSON
+protocol and decoded on both sides — no precision loss.
+
+If the relay hop fails (relay down, peer disconnected), the node logs a warning and
+falls back to a direct HTTP attempt before returning an error.
+
+### Test from WSL2 using the public tracker
+
+In WSL2 (which gets a `172.x.x.x` virtual IP — unreachable from other machines):
+
+```bash
+# WSL2 Terminal 1 — head node (layers 0–11, handles chat requests)
+.venv/bin/meshnet-node start \
+  --tracker https://ai.neuron.d-popov.com \
+  --model Qwen/Qwen2.5-0.5B-Instruct \
+  --shard-start 0 --shard-end 11
+
+# WSL2 Terminal 2 — tail node (layers 12–23)
+.venv/bin/meshnet-node start \
+  --tracker https://ai.neuron.d-popov.com \
+  --model Qwen/Qwen2.5-0.5B-Instruct \
+  --shard-start 12 --shard-end 23
+```
+
+Both nodes connect to the relay automatically. When a chat request arrives at Node A,
+it forwards activations to Node B via `wss://ai.neuron.d-popov.com/rpc/{peer_id_B}`.
+
+Send inference through the tracker (which picks the head node and injects the route):
+
+```bash
+curl -s https://ai.neuron.d-popov.com/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "Qwen/Qwen2.5-0.5B-Instruct",
+    "messages": [{"role": "user", "content": "What is 7 times 8?"}],
+    "stream": false
+  }' | python3 -m json.tool
+```
+
+Or send directly to Node A's local port (within WSL):
+
+```bash
+curl -s http://localhost:7001/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{"model": "Qwen/Qwen2.5-0.5B-Instruct", "messages": [{"role": "user", "content": "Hi"}]}'
+```
+
+---
+
+## Step 1 — Start the tracker (Terminal 1)
+
+```bash
+cd /run/media/popov/d/DEV/repos/d-popov.com/AI
+.venv/bin/meshnet-tracker start --port 8080
+```
+
+Expected output:
+```
+Tracker listening on 0.0.0.0:8080
+```
+
+Keep this terminal open.
+
+---
+
+## Step 2 — Start a node (Terminal 2)
+
+### Recommended model: Qwen2.5-0.5B-Instruct
+
+- 0.5B parameters, ~1 GB in BF16
+- No HuggingFace account or license required
+- Downloads once to `~/.meshnet/models/`, cached for future runs
+- 24 transformer layers (auto-detected — no need to specify)
+
+```bash
+cd /run/media/popov/d/DEV/repos/d-popov.com/AI
+HF_HOME=/run/media/popov/d/DEV/models \
+.venv/bin/meshnet-node start \
+  --model Qwen/Qwen2.5-0.5B-Instruct \
+  --quantization bfloat16 \
+  --tracker http://localhost:8080 \
+  --port 8001
+```
+
+Shard range is **auto-detected** from the curated catalog (no network call for known
+models). For unknown repos, the node fetches only `config.json` (~1 KB) to read
+`num_hidden_layers`. You can still pass `--shard-start` / `--shard-end` explicitly
+to run a partial shard on one machine.
+
+Expected output (after model loads):
+```
+  Auto-detected 24 layers → shard 0–23
+================================
+meshnet-node ready
+  Wallet:       <address>
+  Model ID:     Qwen/Qwen2.5-0.5B-Instruct
+  Shard:        layers 0–23
+  Quantization: bfloat16
+  Endpoint:     http://<host>:8001
+  Hardware:     CPU
+================================
+```
+
+### Other model options (all CPU-friendly)
+
+| Model | HF repo | Layers | BF16 size | Notes |
+|-------|---------|--------|-----------|-------|
+| Qwen2.5-0.5B | `Qwen/Qwen2.5-0.5B-Instruct` | 24 | ~1 GB | Fastest, no gating |
+| Qwen2.5-1.5B | `Qwen/Qwen2.5-1.5B-Instruct` | 28 | ~3 GB | Better quality |
+| Phi-3-mini | `microsoft/Phi-3-mini-4k-instruct` | 32 | ~7.5 GB | Best CPU quality |
+| Llama-3.2-1B | `meta-llama/Llama-3.2-1B-Instruct` | 16 | ~2 GB | Requires HF login |
+| Llama-3.2-3B | `meta-llama/Llama-3.2-3B-Instruct` | 28 | ~6 GB | Requires HF login |
+
+For gated models (Llama), run `huggingface-cli login` first.
+
+---
+
+## Step 3 — Send an inference request (Terminal 3)
+
+If you started the node with `--port 8001`, send the request directly to that
+head node:
+
+```bash Qwen2.5-0.5B-Instruct
+curl -s http://localhost:8001/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "qwen2.5-0.5b",
+    "messages": [{"role": "user", "content": "What is 7 times 8? Answer in one word."}],
+    "stream": false
+  }' | python3 -m json.tool
+```
+
+If you did not pass `--port`, `meshnet-node start` uses the first free port at
+or above `7000`. Use the `Endpoint:` printed by the node instead of `8001`.
+
+To test tracker routing/proxying, send the same OpenAI-compatible request to the
+tracker, using either the full HuggingFace repo or the quick alias:
+
+```bash
+curl -s http://localhost:8080/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "qwen2.5-0.5b",
+    "messages": [{"role": "user", "content": "What is 7 times 8? Answer in one word."}],
+    "stream": false
+  }' | python3 -m json.tool
+```
+
+Or use the test script:
+
+```bash
+.venv/bin/python scripts/test_lan_inference.py \
+  --tracker http://localhost:8080 \
+  --gateway http://localhost:8001
+```
+
+---
+
+## Two-node split (same machine, two terminals)
+
+Split Qwen2.5-0.5B's 24 layers across two node processes to test the sharded pipeline:
+
+**Node A — layers 0–11 (tracker mode, serves chat completions):**
+```bash
+HF_HOME=/run/media/popov/d/DEV/models \
+.venv/bin/meshnet-node start \
+  --model Qwen/Qwen2.5-0.5B-Instruct \
+  --shard-start 0 --shard-end 11 \
+  --quantization bfloat16 \
+  --tracker http://localhost:8080 \
+  --port 8001
+```
+
+**Node B — layers 12–23:**
+```bash
+HF_HOME=/run/media/popov/d/DEV/models \
+.venv/bin/meshnet-node start \
+  --model Qwen/Qwen2.5-0.5B-Instruct \
+  --shard-start 12 --shard-end 23 \
+  --quantization bfloat16 \
+  --tracker http://localhost:8080 \
+  --port 8002
+```
+
+Send the request to Node A — it tokenizes, runs layers 0–11, passes binary
+activations to Node B, and streams the final response back.
+
+---
+
+## Two-machine LAN test (Linux + Windows/WSL2)
+
+See `docs/TWO_MACHINE_TEST.md` (created by US-018).
+
+For WSL2 nodes, registration only proves the node can reach the tracker
+outbound. Tracker-routed inference also requires the tracker to reach the node's
+advertised endpoint inbound. Either run the node in native Windows PowerShell,
+configure Windows port forwarding into WSL for the node port, or start the
+tracker with a relay URL so the node registers a `relay_addr`.
+
+---
+
+## Browse available models
+
+```bash
+# Show curated list with VRAM requirements
+.venv/bin/meshnet-node models
+
+# Browse HuggingFace Hub top-20 text-generation models
+.venv/bin/meshnet-node models --browse
+```
+
+---
+
+## Start with the interactive wizard
+
+```bash
+# First run: wizard detects GPU, shows model list, saves config
+.venv/bin/meshnet-node
+
+# Subsequent runs: starts directly from saved config
+.venv/bin/meshnet-node
+
+# Re-run wizard even with saved config
+.venv/bin/meshnet-node --reset-config
+```
+
+---
+
+## Run all tests
+
+```bash
+.venv/bin/python -m pytest -q
+```

_DEV_NOTES.md

@@ -0,0 +1,38 @@
+# tracker
+.venv/bin/meshnet-tracker start --host 0.0.0.0 --port 8080
+  .\.venv\Scripts\python.exe -m meshnet_tracker.cli start --host 127.0.0.1 --port 8080 --billing-db .\billing.sqlite      
+
+
+# node
+  .\.venv\Scripts\python.exe -m meshnet_node.cli start --tracker http://localhost:8080 --model  Qwen/Qwen2.5-0.5B-Instruct --port 7000 --debug   
+
+ @win
+ .venv/bin/meshnet-node start --tracker http://192.168.0.179:8081  --model Qwen/Qwen2.5-0.5B-Instruct --shard-start 20 --advertise-host 192.168.0.20
+
+.\.venv\Scripts\meshnet-node.exe start http://192.168.0.179:8081 --model-id Qwen/Qwen2.5-0.5B-Instruct  --advertise-host 192.168.0.20
+ .\.venv\Scripts\meshnet-node.exe start --tracker http://ai.neuron.d-popov.com  --model-id Qwen/Qwen2.5-0.5B-Instruct  --advertise-host 192.168.0.20      
+
+ we .\.venv\Scripts\meshnet-node.exe start `                                                                                                                                            
+    --tracker http://192.168.0.179:8081 `                                                                                                                                             
+    --model Qwen/Qwen2.5-0.5B-Instruct `                                                                                                                                              
+    --advertise-host 192.168.0.20                                                                                                                                                     
+
+# linux                     
+HF_HOME=/run/media/popov/d/DEV/models     .venv/bin/meshnet-node start       --model-id Qwen/Qwen2.5-0.5B-Instruct       --shard-start 0       --shard-end 21       --quantization bfloat16       --tracker http://localhost:8081  
+# win
+ meshnet-node start --tracker http://ai.neuron.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct 
+
+
+
+
+  Then test tracker API:                                                                   
+                                                                                           
+  curl http://localhost:8080/v1/health                                                     
+  curl http://localhost:8080/v1/models                                                     
+                                                                                           
+  Because billing is enabled, chat calls need a Bearer key:                                
+                                                                                           
+  curl http://localhost:8080/v1/chat/completions `                                         
+    -H "Content-Type: application/json" `                                                  
+    -H "Authorization: Bearer test-key" `                                                  
+    -d '{"model":"stub-model","messages":[{"role":"user","content":"hi"}]}'   

deploy/docker/Dockerfile

@@ -0,0 +1,27 @@
+FROM python:3.12-slim
+
+ENV PYTHONDONTWRITEBYTECODE=1 \
+    PYTHONUNBUFFERED=1 \
+    PIP_NO_CACHE_DIR=1
+
+WORKDIR /app
+
+RUN useradd --create-home --uid 10001 meshnet \
+    && mkdir -p /var/lib/meshnet \
+    && chown meshnet:meshnet /var/lib/meshnet
+
+COPY packages/contracts /app/packages/contracts
+COPY packages/tracker /app/packages/tracker
+COPY packages/relay /app/packages/relay
+
+RUN python -m pip install --upgrade pip setuptools wheel \
+    && python -m pip install \
+      -e /app/packages/contracts \
+      -e /app/packages/tracker \
+      -e /app/packages/relay
+
+USER meshnet
+
+EXPOSE 8081 8765
+
+CMD ["meshnet-tracker", "start", "--host", "0.0.0.0", "--port", "8081"]

deploy/portainer/meshnet-tracker-nobuild-stack.yml

@@ -0,0 +1,153 @@
+# Meshnet public tracker + relay stack for Portainer without a custom image.
+#
+# This stack does NOT use deploy/docker/Dockerfile and does NOT require pushing an
+# image to a registry. Each service starts from the public python:3.12-slim image,
+# downloads a source tarball, installs the tracker/relay packages into a named
+# venv volume, then starts the service.
+#
+# Required Portainer variables:
+#   SOURCE_TARBALL_URL     URL to a .tar.gz archive of this repo
+#   PUBLIC_TRACKER_URL     e.g. https://cloud.neuron.d-popov.com
+#   PUBLIC_PROXY_NETWORK   Docker network shared with nginx/NPM, e.g. npm_proxy
+#
+# Optional:
+#   CLUSTER_PEERS          e.g. https://ai.neuron.d-popov.com
+#   PUBLIC_RELAY_URL       defaults to PUBLIC_TRACKER_URL with wss/ws + /ws
+#   SOURCE_STRIP_COMPONENTS defaults to 1 for GitHub/Gitea archive tarballs
+
+services:
+  meshnet-tracker:
+    image: python:3.12-slim
+    container_name: meshnet-tracker
+    restart: unless-stopped
+    environment:
+      SOURCE_TARBALL_URL: ${SOURCE_TARBALL_URL:?set SOURCE_TARBALL_URL}
+      SOURCE_STRIP_COMPONENTS: ${SOURCE_STRIP_COMPONENTS:-1}
+      PUBLIC_TRACKER_URL: ${PUBLIC_TRACKER_URL:?set PUBLIC_TRACKER_URL, e.g. https://cloud.neuron.d-popov.com}
+      CLUSTER_PEERS: ${CLUSTER_PEERS:-}
+      PUBLIC_RELAY_URL: ${PUBLIC_RELAY_URL:-}
+      HEARTBEAT_TIMEOUT: ${HEARTBEAT_TIMEOUT:-30}
+      ENABLE_BILLING_DB: ${ENABLE_BILLING_DB:-1}
+      SOLANA_RPC_URL: ${SOLANA_RPC_URL:-}
+      USDT_MINT: ${USDT_MINT:-}
+      MESHNET_TREASURY_KEYPAIR_B64: ${MESHNET_TREASURY_KEYPAIR_B64:-}
+      SETTLE_PERIOD: ${SETTLE_PERIOD:-86400}
+      PAYOUT_THRESHOLD: ${PAYOUT_THRESHOLD:-5}
+      PAYOUT_DUST_FLOOR: ${PAYOUT_DUST_FLOOR:-0.01}
+    command:
+      - /bin/sh
+      - -lc
+      - |
+        set -eu
+        apt-get update
+        apt-get install -y --no-install-recommends ca-certificates curl tar
+        rm -rf /var/lib/apt/lists/*
+
+        rm -rf /opt/meshnet-src
+        mkdir -p /opt/meshnet-src
+        curl -fsSL "$${SOURCE_TARBALL_URL}" -o /tmp/meshnet-src.tar.gz
+        tar -xzf /tmp/meshnet-src.tar.gz -C /opt/meshnet-src --strip-components "$${SOURCE_STRIP_COMPONENTS:-1}"
+
+        python -m venv /opt/meshnet-venv
+        /opt/meshnet-venv/bin/python -m pip install --upgrade pip setuptools wheel
+        /opt/meshnet-venv/bin/pip install \
+          -e /opt/meshnet-src/packages/contracts \
+          -e /opt/meshnet-src/packages/tracker \
+          -e /opt/meshnet-src/packages/relay
+
+        PEER_ARGS=""
+        if [ -n "$${CLUSTER_PEERS:-}" ]; then
+          PEER_ARGS="--cluster-peers $${CLUSTER_PEERS}"
+        fi
+        RELAY_URL="$${PUBLIC_RELAY_URL:-}"
+        if [ -z "$${RELAY_URL}" ]; then
+          RELAY_URL="$$(printf '%s' "$${PUBLIC_TRACKER_URL}" | sed 's#^https://#wss://#; s#^http://#ws://#')/ws"
+        fi
+        BILLING_ARGS=""
+        if [ "$${ENABLE_BILLING_DB:-1}" = "1" ]; then
+          BILLING_ARGS="--billing-db /var/lib/meshnet/billing.sqlite"
+        fi
+        TREASURY_ARGS=""
+        if [ -n "$${SOLANA_RPC_URL:-}" ] || [ -n "$${USDT_MINT:-}" ] || [ -n "$${MESHNET_TREASURY_KEYPAIR_B64:-}" ]; then
+          if [ -z "$${SOLANA_RPC_URL:-}" ] || [ -z "$${USDT_MINT:-}" ] || [ -z "$${MESHNET_TREASURY_KEYPAIR_B64:-}" ]; then
+            echo "SOLANA_RPC_URL, USDT_MINT, and MESHNET_TREASURY_KEYPAIR_B64 must all be set together" >&2
+            exit 1
+          fi
+          printf '%s' "$${MESHNET_TREASURY_KEYPAIR_B64}" | base64 -d > /var/lib/meshnet/treasury-keypair.json
+          chmod 600 /var/lib/meshnet/treasury-keypair.json
+          TREASURY_ARGS="--solana-rpc-url $${SOLANA_RPC_URL} --usdt-mint $${USDT_MINT} --treasury-keypair /var/lib/meshnet/treasury-keypair.json --settle-period $${SETTLE_PERIOD} --payout-threshold $${PAYOUT_THRESHOLD} --payout-dust-floor $${PAYOUT_DUST_FLOOR}"
+        fi
+        exec /opt/meshnet-venv/bin/meshnet-tracker start \
+          --host 0.0.0.0 \
+          --port 8081 \
+          --heartbeat-timeout "$${HEARTBEAT_TIMEOUT}" \
+          --self-url "$${PUBLIC_TRACKER_URL}" \
+          --relay-url "$${RELAY_URL}" \
+          --stats-db /var/lib/meshnet/tracker-stats.sqlite \
+          $${BILLING_ARGS} \
+          $${TREASURY_ARGS} \
+          $${PEER_ARGS}
+    volumes:
+      - meshnet-tracker-data:/var/lib/meshnet
+      - meshnet-tracker-venv:/opt/meshnet-venv
+    expose:
+      - "8081"
+    healthcheck:
+      test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8081/v1/health', timeout=3).read()"]
+      interval: 30s
+      timeout: 5s
+      retries: 3
+      start_period: 60s
+    networks:
+      - public-proxy
+
+  meshnet-relay:
+    image: python:3.12-slim
+    container_name: meshnet-relay
+    restart: unless-stopped
+    environment:
+      SOURCE_TARBALL_URL: ${SOURCE_TARBALL_URL:?set SOURCE_TARBALL_URL}
+      SOURCE_STRIP_COMPONENTS: ${SOURCE_STRIP_COMPONENTS:-1}
+    command:
+      - /bin/sh
+      - -lc
+      - |
+        set -eu
+        apt-get update
+        apt-get install -y --no-install-recommends ca-certificates curl tar
+        rm -rf /var/lib/apt/lists/*
+
+        rm -rf /opt/meshnet-src
+        mkdir -p /opt/meshnet-src
+        curl -fsSL "$${SOURCE_TARBALL_URL}" -o /tmp/meshnet-src.tar.gz
+        tar -xzf /tmp/meshnet-src.tar.gz -C /opt/meshnet-src --strip-components "$${SOURCE_STRIP_COMPONENTS:-1}"
+
+        python -m venv /opt/meshnet-venv
+        /opt/meshnet-venv/bin/python -m pip install --upgrade pip setuptools wheel
+        /opt/meshnet-venv/bin/pip install \
+          -e /opt/meshnet-src/packages/tracker \
+          -e /opt/meshnet-src/packages/relay
+
+        exec /opt/meshnet-venv/bin/meshnet-relay --host 0.0.0.0 --port 8765 --log-level INFO
+    volumes:
+      - meshnet-relay-venv:/opt/meshnet-venv
+    expose:
+      - "8765"
+    healthcheck:
+      test: ["CMD", "python", "-c", "import socket; s=socket.create_connection(('127.0.0.1', 8765), 3); s.close()"]
+      interval: 30s
+      timeout: 5s
+      retries: 3
+      start_period: 60s
+    networks:
+      - public-proxy
+
+volumes:
+  meshnet-tracker-data:
+  meshnet-tracker-venv:
+  meshnet-relay-venv:
+
+networks:
+  public-proxy:
+    external: true
+    name: ${PUBLIC_PROXY_NETWORK:-npm_proxy}

deploy/portainer/meshnet-tracker-stack.yml

@@ -0,0 +1,107 @@
+# Meshnet public tracker + relay stack for Portainer.
+#
+# Intended topology when Nginx Proxy Manager (or another nginx reverse proxy)
+# runs on the same Docker host:
+#   https://YOUR_DOMAIN/v1/*  -> meshnet-tracker:8081
+#   https://YOUR_DOMAIN/ws    -> meshnet-relay:8765  (WebSocket)
+#   https://YOUR_DOMAIN/rpc/* -> meshnet-relay:8765  (WebSocket)
+#
+# Before deploying, create or identify the Docker network shared with nginx/NPM,
+# then set PUBLIC_PROXY_NETWORK to its name in Portainer environment variables.
+
+services:
+  meshnet-tracker:
+    build:
+      context: ../..
+      dockerfile: deploy/docker/Dockerfile
+    image: meshnet-tracker-relay:local
+    container_name: meshnet-tracker
+    restart: unless-stopped
+    environment:
+      PUBLIC_TRACKER_URL: ${PUBLIC_TRACKER_URL:?set PUBLIC_TRACKER_URL, e.g. https://cloud.neuron.d-popov.com}
+      CLUSTER_PEERS: ${CLUSTER_PEERS:-}
+      PUBLIC_RELAY_URL: ${PUBLIC_RELAY_URL:-}
+      HEARTBEAT_TIMEOUT: ${HEARTBEAT_TIMEOUT:-30}
+      ENABLE_BILLING_DB: ${ENABLE_BILLING_DB:-1}
+      SOLANA_RPC_URL: ${SOLANA_RPC_URL:-}
+      USDT_MINT: ${USDT_MINT:-}
+      MESHNET_TREASURY_KEYPAIR_B64: ${MESHNET_TREASURY_KEYPAIR_B64:-}
+      SETTLE_PERIOD: ${SETTLE_PERIOD:-86400}
+      PAYOUT_THRESHOLD: ${PAYOUT_THRESHOLD:-5}
+      PAYOUT_DUST_FLOOR: ${PAYOUT_DUST_FLOOR:-0.01}
+    command:
+      - /bin/sh
+      - -lc
+      - |
+        set -eu
+        PEER_ARGS=""
+        if [ -n "$${CLUSTER_PEERS:-}" ]; then
+          PEER_ARGS="--cluster-peers $${CLUSTER_PEERS}"
+        fi
+        RELAY_URL="$${PUBLIC_RELAY_URL:-}"
+        if [ -z "$${RELAY_URL}" ]; then
+          RELAY_URL="$$(printf '%s' "$${PUBLIC_TRACKER_URL}" | sed 's#^https://#wss://#; s#^http://#ws://#')/ws"
+        fi
+        BILLING_ARGS=""
+        if [ "$${ENABLE_BILLING_DB:-1}" = "1" ]; then
+          BILLING_ARGS="--billing-db /var/lib/meshnet/billing.sqlite"
+        fi
+        TREASURY_ARGS=""
+        if [ -n "$${SOLANA_RPC_URL:-}" ] || [ -n "$${USDT_MINT:-}" ] || [ -n "$${MESHNET_TREASURY_KEYPAIR_B64:-}" ]; then
+          if [ -z "$${SOLANA_RPC_URL:-}" ] || [ -z "$${USDT_MINT:-}" ] || [ -z "$${MESHNET_TREASURY_KEYPAIR_B64:-}" ]; then
+            echo "SOLANA_RPC_URL, USDT_MINT, and MESHNET_TREASURY_KEYPAIR_B64 must all be set together" >&2
+            exit 1
+          fi
+          printf '%s' "$${MESHNET_TREASURY_KEYPAIR_B64}" | base64 -d > /var/lib/meshnet/treasury-keypair.json
+          chmod 600 /var/lib/meshnet/treasury-keypair.json
+          TREASURY_ARGS="--solana-rpc-url $${SOLANA_RPC_URL} --usdt-mint $${USDT_MINT} --treasury-keypair /var/lib/meshnet/treasury-keypair.json --settle-period $${SETTLE_PERIOD} --payout-threshold $${PAYOUT_THRESHOLD} --payout-dust-floor $${PAYOUT_DUST_FLOOR}"
+        fi
+        exec meshnet-tracker start \
+          --host 0.0.0.0 \
+          --port 8081 \
+          --heartbeat-timeout "$${HEARTBEAT_TIMEOUT}" \
+          --self-url "$${PUBLIC_TRACKER_URL}" \
+          --relay-url "$${RELAY_URL}" \
+          --stats-db /var/lib/meshnet/tracker-stats.sqlite \
+          $${BILLING_ARGS} \
+          $${TREASURY_ARGS} \
+          $${PEER_ARGS}
+    volumes:
+      - meshnet-tracker-data:/var/lib/meshnet
+    expose:
+      - "8081"
+    healthcheck:
+      test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8081/v1/health', timeout=3).read()"]
+      interval: 30s
+      timeout: 5s
+      retries: 3
+      start_period: 10s
+    networks:
+      - public-proxy
+
+  meshnet-relay:
+    image: meshnet-tracker-relay:local
+    container_name: meshnet-relay
+    restart: unless-stopped
+    depends_on:
+      meshnet-tracker:
+        condition: service_started
+    command: ["meshnet-relay", "--host", "0.0.0.0", "--port", "8765", "--log-level", "INFO"]
+    expose:
+      - "8765"
+    healthcheck:
+      test: ["CMD", "python", "-c", "import socket; s=socket.create_connection(('127.0.0.1', 8765), 3); s.close()"]
+      interval: 30s
+      timeout: 5s
+      retries: 3
+      start_period: 10s
+    networks:
+      - public-proxy
+
+volumes:
+  meshnet-tracker-data:
+
+networks:
+  public-proxy:
+    external: true
+    name: ${PUBLIC_PROXY_NETWORK:-npm_proxy}

docs/DEPLOY_PORTAINER.md

@@ -0,0 +1,171 @@
+# One-shot Portainer deploy: public tracker + relay
+
+No Dockerfile, no custom image, no registry. Portainer runs `python:3.12-slim`, downloads this repo tarball, installs tracker/relay, and starts both services.
+
+## 1. One-time host setup
+
+DNS points to the Docker host, e.g. `cloud.neuron.d-popov.com`.
+Firewall exposes only `80`, `443`, and admin SSH.
+Nginx Proxy Manager and this stack share a Docker network, e.g. `npm_proxy`.
+
+If needed:
+
+```bash
+docker network create npm_proxy
+docker network connect npm_proxy <nginx-proxy-manager-container>
+```
+
+## 2. Portainer stack
+
+Portainer → Stacks → Add stack → Repository.
+
+Stack file:
+
+```text
+deploy/portainer/meshnet-tracker-nobuild-stack.yml
+```
+
+Deploy only after the current branch has been pushed. The stack passes the
+current tracker billing flags (`--billing-db`, and optional treasury flags), so
+an older remote `master.tar.gz` will crash on startup with unrecognized
+arguments.
+
+Variables:
+
+```text
+SOURCE_TARBALL_URL=https://git.d-popov.com/<owner>/<repo>/archive/master.tar.gz
+PUBLIC_TRACKER_URL=https://cloud.neuron.d-popov.com
+PUBLIC_PROXY_NETWORK=npm_proxy
+CLUSTER_PEERS=https://ai.neuron.d-popov.com
+PUBLIC_RELAY_URL=wss://cloud.neuron.d-popov.com/ws
+HEARTBEAT_TIMEOUT=30
+ENABLE_BILLING_DB=1
+```
+
+For first cloud-only test, use `CLUSTER_PEERS=`. Click **Deploy the stack**.
+
+`ENABLE_BILLING_DB=1` makes billing public behavior active: `/v1/chat/completions`
+requires `Authorization: Bearer <api-key>`. Any key is accepted and starts with
+the configured starter credit; calls return `402` after that balance is
+exhausted. Set `ENABLE_BILLING_DB=0` if existing unauthenticated clients must
+keep working during the first redeploy.
+
+Optional Solana treasury settlement variables:
+
+```text
+SOLANA_RPC_URL=https://api.devnet.solana.com
+USDT_MINT=<mock-usdt-mint-from-scripts/devnet_setup.py>
+MESHNET_TREASURY_KEYPAIR_B64=<base64 of solana keypair JSON>
+SETTLE_PERIOD=86400
+PAYOUT_THRESHOLD=5
+PAYOUT_DUST_FLOOR=0.01
+```
+
+Set all three treasury identity values together (`SOLANA_RPC_URL`, `USDT_MINT`,
+`MESHNET_TREASURY_KEYPAIR_B64`) or leave all three empty. The stack writes the
+decoded keypair into the tracker data volume at startup and passes it to
+`meshnet-tracker`; do not put this key on relay-only hosts or non-settlement
+trackers.
+
+Expected containers:
+
+```text
+meshnet-tracker  internal 8081
+meshnet-relay    internal 8765
+```
+
+## 3. Nginx Proxy Manager
+
+One Proxy Host for `cloud.neuron.d-popov.com`:
+
+```text
+Forward Hostname/IP: meshnet-tracker
+Forward Port:        8081
+Websockets Support:  ON
+SSL:                 Let's Encrypt + Force SSL
+```
+
+Custom locations on the same proxy host:
+
+```text
+/ws   -> http://meshnet-relay:8765
+/rpc  -> http://meshnet-relay:8765
+```
+
+Leave sub-folder forwarding empty.
+
+If WebSockets fail, Advanced:
+
+```nginx
+proxy_http_version 1.1;
+proxy_set_header Upgrade $http_upgrade;
+proxy_set_header Connection $http_connection;
+proxy_read_timeout 3600s;
+proxy_send_timeout 3600s;
+```
+
+## 4. Smoke test
+
+```bash
+curl -s https://cloud.neuron.d-popov.com/v1/health
+curl -s https://cloud.neuron.d-popov.com/v1/network/map | python3 -m json.tool
+curl -s https://cloud.neuron.d-popov.com/v1/raft/status | python3 -m json.tool
+```
+
+Plain curl to `/ws` or `/rpc/test-peer` may show `426 Upgrade Required`; OK. It must not show nginx `502`.
+
+Dashboard:
+
+```bash
+curl -s https://cloud.neuron.d-popov.com/dashboard | head
+```
+
+Then open:
+
+```text
+https://cloud.neuron.d-popov.com/dashboard
+```
+
+The dashboard is served by the tracker through the same proxy host; no extra NPM
+location is required.
+
+If you previously deployed the build-image variant before `/var/lib/meshnet`
+was created as the `meshnet` user, the named volume may already be root-owned.
+Recreate that volume or chown it once before retrying the fixed image.
+
+## 5. Start a node
+
+`meshnet-node` is the worker/miner process. It will ask for a model and load a
+model shard. Do not use it to test the tracker dashboard.
+
+```bash
+meshnet-node start --tracker https://cloud.neuron.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct
+```
+
+Expected:
+
+```text
+Relay advertised by tracker — using outbound tunnel wss://cloud.neuron.d-popov.com/ws
+Relay connected — wss://cloud.neuron.d-popov.com/rpc/<peer_id>
+```
+
+## 6. Two tracker sync
+
+Cloud stack:
+
+```text
+PUBLIC_TRACKER_URL=https://cloud.neuron.d-popov.com
+CLUSTER_PEERS=https://ai.neuron.d-popov.com
+PUBLIC_RELAY_URL=wss://cloud.neuron.d-popov.com/ws
+```
+
+Local tracker:
+
+```bash
+meshnet-tracker start --host 0.0.0.0 --port 8081 \
+  --self-url https://ai.neuron.d-popov.com \
+  --cluster-peers https://cloud.neuron.d-popov.com \
+  --relay-url wss://ai.neuron.d-popov.com/ws
+```
+
+Two Raft peers are enough for sync testing; real HA needs three trackers.

docs/INSTALL_WINDOWS.md

@@ -0,0 +1,212 @@
+# Installing meshnet-node on Windows 11 with WSL2
+
+This guide covers setting up a meshnet-node on a Windows 11 machine using WSL2 with CUDA passthrough so it can join an existing inference network over LAN.
+
+## Prerequisites
+
+- Windows 11 with WSL2 support (most systems with Windows 10 version 2004+ qualify)
+- NVIDIA GPU with CUDA support (driver ≥ 525.x recommended for WSL2 CUDA)
+- At least 8 GB RAM + enough VRAM for the model shard you intend to serve
+- The Linux machine (other node) is reachable on your LAN
+
+---
+
+## Step 1 — Enable WSL2 and install Ubuntu
+
+Open **PowerShell as Administrator** and run:
+
+```powershell
+wsl --install -d Ubuntu-24.04
+```
+
+This installs WSL2 with Ubuntu 24.04. Reboot when prompted.
+
+After reboot, Ubuntu starts and asks you to create a UNIX username/password. Choose anything convenient.
+
+Verify WSL version:
+
+```powershell
+wsl -l -v
+```
+
+Output should show `VERSION 2`.
+
+---
+
+## Step 2 — Install NVIDIA GPU driver on Windows (NOT inside WSL)
+
+WSL2 CUDA passthrough works through the Windows host driver. **Do not install CUDA inside WSL2.**
+
+1. Download the latest Game Ready or Studio driver for your GPU from https://www.nvidia.com/drivers
+2. Install on Windows normally (standard installer).
+3. Inside WSL2 (Ubuntu terminal), verify:
+
+```bash
+nvidia-smi
+```
+
+Expected output: your GPU name, driver version, CUDA version. If this command fails, the Windows driver is too old — update it.
+
+> **Note:** The `cuda-toolkit` package inside WSL2 is optional and only needed if you compile CUDA kernels. For inference with `torch`, the Windows host driver is sufficient.
+
+---
+
+## Step 3 — Install Python 3.11+ inside WSL2
+
+Ubuntu 24.04 ships Python 3.12. Confirm:
+
+```bash
+python3 --version
+```
+
+If it shows 3.10 or older:
+
+```bash
+sudo add-apt-repository ppa:deadsnakes/ppa
+sudo apt update
+sudo apt install python3.12 python3.12-venv python3.12-dev
+```
+
+Install pip:
+
+```bash
+curl -sS https://bootstrap.pypa.io/get-pip.py | python3
+```
+
+---
+
+## Step 4 — Clone the repository
+
+Inside WSL2:
+
+```bash
+# Store the repo in the Linux filesystem (faster I/O than /mnt/c)
+cd ~
+git clone https://github.com/YOUR_ORG/d-popov.com.git
+cd d-popov.com/AI
+```
+
+---
+
+## Step 5 — Create a virtualenv and install meshnet-node
+
+```bash
+python3 -m venv .venv
+source .venv/bin/activate
+
+# Install node + PyTorch (CUDA build)
+pip install torch --index-url https://download.pytorch.org/whl/cu124
+pip install -e "packages/node[torch]"
+```
+
+Verify the install:
+
+```bash
+meshnet-node --help
+```
+
+---
+
+## Step 6 — Pre-download the model shard
+
+Download the model before starting the node so the startup process doesn't time out on the tracker side:
+
+```bash
+python3 - <<'EOF'
+from transformers import AutoConfig
+AutoConfig.from_pretrained("microsoft/Phi-3-medium-128k-instruct")
+EOF
+```
+
+For the full model weights (needed at runtime), `transformers` downloads them automatically on first `meshnet-node` start. If you want to pre-fetch:
+
+```bash
+python3 -c "
+from transformers import AutoModelForCausalLM
+AutoModelForCausalLM.from_pretrained('microsoft/Phi-3-medium-128k-instruct', device_map='cpu')
+"
+```
+
+This can take 10–30 minutes on first run.
+
+---
+
+## Step 7 — Expose the node port to your LAN
+
+WSL2 runs behind a NAT with a virtual IP (typically `172.x.x.x`). Your LAN sees the Windows host IP. You need to forward the node port.
+
+**Option A — Windows port proxy (recommended for simple setups):**
+
+In **PowerShell as Administrator**:
+
+```powershell
+# Get the current WSL2 IP (changes on each WSL restart)
+$wslIp = (wsl hostname -I).Trim()
+
+# Forward Windows host port 8001 → WSL2 port 8001
+netsh interface portproxy add v4tov4 `
+  listenport=8001 listenaddress=0.0.0.0 `
+  connectport=8001 connectaddress=$wslIp
+
+# Allow inbound on Windows Firewall
+New-NetFirewallRule -DisplayName "meshnet-node" `
+  -Direction Inbound -Protocol TCP -LocalPort 8001 -Action Allow
+```
+
+Verify: from the Linux machine, `curl http://WINDOWS_LAN_IP:8001/v1/health` should return a response once the node is running.
+
+**Redo this after every WSL2 restart** — the WSL2 IP changes.
+
+**Option B — P2P relay (US-017, no port forwarding needed):**
+
+Start a relay node on the Linux machine. The WSL2 node connects outbound through the relay. No firewall rules needed. See `docs/TWO_MACHINE_TEST.md` for details.
+
+---
+
+## Step 8 — Start the node
+
+Replace `192.168.1.10` with the actual LAN IP of the Linux machine running the tracker.
+Replace shard range with the complementary range to what the Linux node is serving.
+
+```bash
+source .venv/bin/activate
+
+meshnet-node \
+  --model microsoft/Phi-3-medium-128k-instruct \
+  --quantization bf16 \
+  --shard-start 20 --shard-end 39 \
+  --tracker http://192.168.1.10:8080 \
+  --port 8001 \
+  --host 0.0.0.0 \
+  --advertise-host WINDOWS_LAN_IP
+```
+
+The `--advertise-host` flag tells the tracker what IP the Linux machine should use to reach this node. Use your Windows machine's LAN IP (e.g. `192.168.1.20`), **not** the WSL2 internal IP.
+
+Expected startup output:
+
+```
+Detecting hardware...
+  GPU: NVIDIA GeForce RTX 3080 (10240 MB VRAM)
+Loading wallet...
+  Wallet: 5K7r...
+Loading real PyTorch model shard...
+  Auto-detected 40 layers → shard 20–39
+================================
+meshnet-node ready
+  Model ID:     microsoft/Phi-3-medium-128k-instruct
+  Shard:        layers 20–39; 20 of 40
+  Endpoint:     http://192.168.1.20:8001
+  Hardware:     CUDA
+================================
+```
+
+---
+
+## Known issues
+
+- **WSL2 IP changes on restart.** Always re-run the `netsh` port-proxy command after restarting WSL2 or Windows.
+- **CUDA not visible in WSL2.** If `nvidia-smi` fails inside WSL2, update the Windows host GPU driver to ≥ 525.x. Installing CUDA inside WSL2 will not fix it.
+- **Model download is slow.** HuggingFace downloads happen over HTTPS. Pre-fetch the model before a timed test (see Step 6).
+- **Port 8001 already in use.** Change `--port` to another value and update the firewall/portproxy rules accordingly.
+- **`bf16` not supported on older GPUs.** Use `--quantization int8` on Turing (RTX 20xx) cards or earlier if bfloat16 ops fail.

docs/TWO_MACHINE_TEST.md

@@ -0,0 +1,200 @@
+# Two-machine LAN inference test
+
+This guide proves that distributed inference works across two physical machines: a Linux rig (tracker + first shard) and a Windows 11 / WSL2 rig (second shard). A test script sends real inference requests and validates the output.
+
+## Network topology
+
+```
+[Linux machine — 192.168.1.10]
+  meshnet-tracker  :8080
+  meshnet-node A   :8001   shard 0–19  (tracker-mode, entry point)
+
+[Windows 11 / WSL2 — 192.168.1.20]
+  meshnet-node B   :8001   shard 20–39
+
+[Client — either machine]
+  scripts/test_lan_inference.py --tracker http://192.168.1.10:8080
+```
+
+Adjust the IPs and shard ranges to match your hardware. Use a model that fits (sharded) in both GPUs combined. The example uses `microsoft/Phi-3-medium-128k-instruct` (40 layers, BF16 ~15 GB each shard ~7.5 GB).
+
+---
+
+## Prerequisites
+
+**Both machines:**
+- Python 3.11+ with `meshnet-node` installed (see `docs/INSTALL_WINDOWS.md` for Windows)
+- Model weights already downloaded (pre-fetch prevents timeout on first startup)
+- LAN connectivity verified: `ping 192.168.1.10` from Windows, `ping 192.168.1.20` from Linux
+
+**Linux machine ports open:**
+
+```bash
+# ufw (skip if firewall is off)
+sudo ufw allow 8080/tcp   # tracker
+sudo ufw allow 8001/tcp   # node A
+```
+
+**Windows machine port forwarded (WSL2 only):**
+
+```powershell
+# Run in PowerShell as Administrator — redo after every WSL restart
+$wsl = (wsl hostname -I).Trim()
+netsh interface portproxy add v4tov4 listenport=8001 listenaddress=0.0.0.0 connectport=8001 connectaddress=$wsl
+New-NetFirewallRule -DisplayName "meshnet-node" -Direction Inbound -Protocol TCP -LocalPort 8001 -Action Allow
+```
+
+---
+
+## Start sequence
+
+**Always start in this order: tracker → node A → node B → test.**
+
+### Terminal 1 — Linux: tracker
+
+```bash
+meshnet-tracker --port 8080
+```
+
+Expected:
+
+```
+[tracker] listening on 0.0.0.0:8080
+```
+
+### Terminal 2 — Linux: node A (shard 0–19, tracker-mode)
+
+```bash
+meshnet-node \
+  --model microsoft/Phi-3-medium-128k-instruct \
+  --quantization bf16 \
+  --shard-start 0 --shard-end 19 \
+  --tracker http://localhost:8080 \
+  --port 8001 \
+  --host 0.0.0.0
+```
+
+`shard_start=0` auto-sets `tracker_mode=True` — this node accepts inference requests.
+
+Wait until you see `meshnet-node ready` before continuing.
+
+### Terminal 3 — Windows WSL2: node B (shard 20–39)
+
+```bash
+meshnet-node \
+  --model microsoft/Phi-3-medium-128k-instruct \
+  --quantization bf16 \
+  --shard-start 20 --shard-end 39 \
+  --tracker http://192.168.1.10:8080 \
+  --port 8001 \
+  --host 0.0.0.0 \
+  --advertise-host 192.168.1.20
+```
+
+`--advertise-host` must be the Windows **LAN IP** (not the WSL2 internal 172.x.x.x IP) so the Linux node can reach it.
+
+---
+
+## Verify nodes are registered
+
+From any machine with `curl`:
+
+```bash
+# List all registered nodes
+curl http://192.168.1.10:8080/v1/nodes
+
+# Check route for the model — should list both node endpoints in order
+curl "http://192.168.1.10:8080/v1/route?model=microsoft/Phi-3-medium-128k-instruct"
+```
+
+Expected route response:
+
+```json
+{
+  "route": [
+    "http://192.168.1.10:8001",
+    "http://192.168.1.20:8001"
+  ]
+}
+```
+
+If only one endpoint appears, node B hasn't registered yet — wait a few seconds and retry.
+
+---
+
+## Run the test script
+
+```bash
+# From any machine that can reach the tracker
+python3 scripts/test_lan_inference.py \
+  --tracker http://192.168.1.10:8080 \
+  --gateway http://192.168.1.10:8001
+```
+
+Expected output:
+
+```
+Inference endpoint: http://192.168.1.10:8001
+Tracker: http://192.168.1.10:8080
+
+Route: ['http://192.168.1.10:8001', 'http://192.168.1.20:8001']
+
+[1] Q: What is 7 × 8? Answer in one word.
+     A: 56
+     3 tokens  2.41s  1.2 t/s
+
+[2] Q: Name the capital of France in one word.
+     A: Paris
+     2 tokens  1.87s  1.1 t/s
+
+[3] Q: Complete the sequence: 1, 1, 2, 3, 5, ___
+     A: 8
+     2 tokens  1.93s  1.0 t/s
+
+All 3 requests completed successfully.
+Exit code: 0
+```
+
+The script exits 0 if all 3 requests complete with valid OpenAI-format responses.
+
+---
+
+## Reading latency from node logs
+
+The node logs show per-hop timing. On node A terminal look for:
+
+```
+[node] forwarding to downstream: http://192.168.1.20:8001  (took 1.23s)
+```
+
+Approximate breakdown:
+- **client → node A (encode + first shard):** full request latency minus the downstream time
+- **node A → node B (pipeline):** the `forwarding to downstream` duration
+- **node B → node A (tail decode + token):** included in downstream duration
+
+Full end-to-end latency = prompt encode + shard A forward + network transfer + shard B forward + decode.
+
+With LAN latency < 1 ms, the network transfer is negligible. Bottleneck is GPU compute.
+
+---
+
+## Known Issues
+
+**WSL2 IP changes after restart.**
+The `netsh portproxy` forwarding rule uses a fixed WSL2 IP. If Windows or WSL2 restarts, the IP changes and the rule breaks. Redo the `netsh` and `New-NetFirewallRule` commands. To automate this, add a Task Scheduler job on WSL start.
+
+**Node B registers with internal WSL2 IP (172.x.x.x) instead of LAN IP.**
+Symptom: route response lists `172.x.x.x` and node A cannot reach it.
+Fix: always pass `--advertise-host 192.168.1.20` (your Windows LAN IP) when starting node B.
+
+**Model download times out node registration.**
+If the model hasn't been pre-fetched, `transformers` downloads it during node startup, which can take 20+ minutes. The tracker heartbeat timeout (90s) will expire, and node A will deregister node B. Pre-download the model weights before starting the node (see `docs/INSTALL_WINDOWS.md` Step 6). Node B re-registers automatically via the heartbeat re-registration loop once it's up.
+
+**`bf16` unsupported on older NVIDIA GPUs.**
+GPUs before Ampere (RTX 30xx) have limited bfloat16 support. Use `--quantization int8` on RTX 20xx and earlier.
+
+**Windows Defender blocks inbound connection on WSL2.**
+Even with the firewall rule added, Windows Defender SmartScreen or a corporate security policy can block the connection. Verify by checking Windows Event Viewer → Security → Filtering Platform Connection for blocked connections on port 8001.
+
+**Route returns only one node.**
+If node B registers but the route only returns one endpoint, check that both nodes use the same `--model` string (full HuggingFace repo path). Route lookup matches on `hf_repo` — a short name vs. full path mismatch causes the node to be excluded.

docs/adr/0009-coverage-first-shard-assignment-and-tracker-as-node.md

@@ -1,6 +1,6 @@
-# Coverage-first shard assignment and tracker-as-first-layer-node
+# Coverage-first shard assignment and tracker-routed inference
 
-The tracker assigns shard ranges to nodes using a coverage-first, speed-weighted bin-packing algorithm. Tracker nodes must host at least the first layer shard of every model they coordinate, making them the natural inference entry point. Any node serving layers[0..k] can become a tracker node for that model.
+The tracker assigns shard ranges to worker nodes using a coverage-first, speed-weighted bin-packing algorithm. The tracker is a control-plane service and public inference API endpoint: it stores registry state, selects routes, enforces billing, and proxies OpenAI-compatible requests to the selected head worker. It does not download or load model weights.
 
 ## Problem
 
@@ -23,24 +23,29 @@ Example: 700B NF4 model (~350GB weights). Node A has 128GB, Node B and C each ha
 - Node C gets layers[k_b..N] (the remainder)
 - If Node B benchmarks 2× faster than Node C, the tracker shifts the B/C boundary so B carries more layers
 
-### Tracker-as-first-layer-node
+### Tracker-routed head worker
 
-Any node that advertises a new model to the network becomes a **tracker node** for that model. Tracker nodes have one hard requirement: they must hold and serve `layers[0..k]` (the first-layer shard) for every model they coordinate.
+A worker that serves `layers[0..k]` is the **head worker** for that model. The tracker forwards `/v1/chat/completions` to a live head worker and injects the remaining downstream route. The worker, not the tracker process:
 
-The reason is functional: a tracker node is also the inference entry point. When a client request arrives, the tracker node:
+When a client request arrives, the tracker:
+1. Authenticates/bills the request
+2. Selects a live head worker and full downstream route from the coverage map
+3. Proxies the request to that head worker
+4. Records usage and credits node shares after completion
+
+The head worker:
 1. Tokenizes the input (owns the tokenizer)
 2. Runs `model.embed_tokens` + `model.layers[0..k]`
-3. Selects the optimal onward route from the coverage map
+3. Forwards activations to the next node in the route
-4. Forwards activations to the next node in the route
+4. Receives the final hidden state back and streams tokens to the client
-5. Receives the final hidden state back and streams tokens to the client
 
-This collapses the separate "gateway" role: the tracker node that starts an inference request IS the gateway for that request. A standalone HTTP proxy/load-balancer may sit in front to pick which tracker node handles the request, but it carries no model weights.
+This keeps the public tracker lightweight: a standalone HTTP proxy/load-balancer may sit in front to pick which tracker handles the request, but neither proxy nor tracker carries model weights.
 
-Multiple tracker nodes for the same model = multiple entry points = horizontal scale for both routing decisions and the first-layer compute.
+Multiple head workers for the same model = multiple inference entry points = horizontal scale for first-layer compute. Multiple trackers scale routing and billing, not model execution.
 
 ### Last-layer node (tail)
 
-The node assigned `layers[N-k..N]` also runs `model.norm` and `model.lm_head`. It returns decoded token IDs (not hidden states) to the tracker node, which assembles the response. The tail shard assignment is marked `is_tail: true` in the shard registry.
+The node assigned `layers[N-k..N]` also runs `model.norm` and `model.lm_head`. It returns decoded token IDs (not hidden states) to the head worker, which assembles the response. The tail shard assignment is marked `is_tail: true` in the shard registry.
 
 ### Adaptive quantization
 
@@ -78,8 +83,8 @@ Nodes obey directives asynchronously; the tracker waits up to a configurable tim
 
 ## Consequences
 
-- The standalone `meshnet-gateway` service from US-005 becomes a thin load-balancer that routes to tracker nodes; tracker nodes do the actual inference orchestration
+- The standalone `meshnet-gateway` service from US-005 becomes a thin compatibility proxy/load-balancer; the public tracker can also serve the OpenAI-compatible endpoint directly
-- Tracker nodes must download more model data (tokenizer + first-layer shard) — this is the price of being an entry point
+- Tracker processes do not download or load model data. Only worker nodes load model shards.
 - Benchmark data is self-reported by nodes at registration; the validator can detect fraudulent benchmarks (a node claiming 100 tokens/sec but delivering 2 gets slashed for under-performance)
 - VRAM reservation for KV cache means nodes can host fewer layers than their raw VRAM suggests — this is intentional; running out of KV cache during inference causes OOM crashes
-- New CONTEXT.md terms: **Tracker Node** (node serving first-layer shard + inference routing for a model), **Coverage Map** (tracker's per-model layer-range → node-count mapping), **Rebalance Directive** (tracker instruction to a node to load or drop a shard)
+- New CONTEXT.md terms: **Head Worker** (worker node serving first-layer shard for a model), **Coverage Map** (tracker's per-model layer-range → node-count mapping), **Rebalance Directive** (tracker instruction to a node to load or drop a shard)

docs/adr/0010-p2p-gossip-and-nat-relay.md

@@ -0,0 +1,67 @@
+# ADR-0010: P2P gossip, NAT-traversal relay, and TLS
+
+## Status: Accepted
+
+## Context
+
+All node-to-node and node-to-tracker communication in the prototype is plain HTTP over a LAN or direct-IP internet connection. This has three problems:
+
+1. **NAT blocking**: Most home and cloud nodes cannot accept inbound TCP connections.
+2. **No encryption**: Activations and heartbeats are in plaintext.
+3. **Polling overhead**: Nodes poll the tracker for coverage changes every 30s. This is slow to react to node churn and does not scale past a few hundred nodes.
+
+The reference implementation (Petals) solves this with libp2p — GossipSub for pub/sub and Kademlia DHT for peer discovery. We adopt the same goals but start with simpler, more stable building blocks that can be swapped for libp2p later without changing the message schema.
+
+## Decisions
+
+### 1. TLS everywhere
+
+All HTTP between nodes, tracker, and gateway uses HTTPS (TLS 1.3). Self-signed certificates are auto-generated on first node start and stored in `~/.config/meshnet/`. The certificate fingerprint is included in every heartbeat and gossip envelope. Nodes use TOFU (trust on first use) — they accept a peer's cert on first contact and pin the fingerprint; connections from the same peer with a different fingerprint are rejected.
+
+The relay node uses a real CA-signed certificate (Let's Encrypt) because it is the internet-facing bootstrap point.
+
+### 2. mDNS for LAN peer discovery
+
+Python `zeroconf` library. Service type: `_meshnet._tcp.local.`. A node announces itself on startup and browses for existing peers. This is zero-config discovery for home and lab networks. mDNS does not traverse routers, which is correct — LAN discovery should not bleed into the internet.
+
+### 3. WebSocket PubSub for gossip
+
+Each node maintains persistent WSS connections to the relay and up to 8 direct peers. Messages use a stable JSON envelope with a `topic`, `version`, `from_peer`, and `payload`. Topics: `node-join`, `node-leave`, `coverage-update`, `heartbeat`, `peer-list`, `relay-announce`.
+
+Simple flooding with `seen_ids` dedup and TTL=3 is good enough for the prototype. The message schema is stable; the fanout mechanism can be replaced with GossipSub mesh routing without changing the schema.
+
+### 4. Circuit relay node for NAT traversal
+
+A team-operated public relay (`packages/relay`, CLI: `meshnet-relay`) is the internet bootstrap point. A node behind NAT:
+
+1. Connects outbound to the relay via WSS
+2. Advertises `relay_addr = wss://relay.meshnet.ai:8443/relay/{peer_id}` to the tracker
+3. Other nodes proxy connections through the relay when the direct addr is not reachable
+
+Hole-punching (STUN + simultaneous TCP open) is deferred to a future story. Circuit relay is the reliable fallback.
+
+The relay is stateless in terms of inference — it only proxies bytes. It does not decrypt activations.
+
+### 5. Bootstrap peer list
+
+`packages/p2p/relay_bootstrap.json` contains the team-operated relay endpoints with their TLS fingerprints. New nodes load this file on startup to find their first peer. The list is bundled with the package and updated via pip upgrades.
+
+### Migration path to libp2p
+
+When the network has enough volume to justify the complexity:
+
+1. Replace the WebSocket gossip layer with libp2p GossipSub (same topics and payload schemas, different transport)
+2. Replace mDNS + relay peer list with Kademlia DHT
+3. Replace circuit relay with libp2p circuit relay v2
+
+The gossip envelope schema (`topic`, `version`, `from_peer`, `payload`) is the stable contract. As long as messages on the wire are identical, the transport layer can be swapped without touching node business logic.
+
+## Alternatives rejected
+
+**libp2p from the start**: `py-libp2p` is experimental and not production-ready. A Go libp2p sidecar is operationally complex. The benefits of real libp2p (mesh routing, Kademlia DHT, hole-punching) are not needed until we have hundreds of nodes.
+
+**NATS**: Stable and fast but requires a central NATS server. Adds operational dependency and contradicts the P2P goal.
+
+**ZeroMQ**: No NAT traversal built in. Requires manual topology management.
+
+**No gossip (keep polling)**: Does not scale; slow to react to node churn; misses the relay/NAT requirement.

docs/adr/0011-auto-shard-and-network-assignment.md

@@ -0,0 +1,73 @@
+# ADR-0011: Auto-shard from memory budget and tracker-managed network assignment
+
+## Status: Accepted
+
+## Context
+
+Early node startup required explicit `--shard-start` and `--shard-end` flags. This is
+fine for expert operators but a barrier to new participants who don't know how many layers
+their GPU can hold. Two improvements were needed:
+
+1. **Auto-detect shard range**: fetch `num_hidden_layers` from the model's `config.json`
+   and compute how many layers fit in available VRAM.
+2. **Network-aware assignment**: instead of each node picking its own shard, the tracker
+   knows the current coverage map and can tell the node which gap to fill.
+
+## Decisions
+
+### 1. Layer count from HuggingFace config
+
+`AutoConfig.from_pretrained(model_id)` downloads only `config.json` (~1 KB, no weights).
+`cfg.num_hidden_layers` gives the total layer count. The node uses this to set
+`shard_end = num_layers - 1` when no explicit range is given.
+
+A curated `MODEL_CATALOG` in `model_catalog.py` provides layer counts for common models
+without any network call — HuggingFace is only hit for uncatalogued repos.
+
+### 2. VRAM-aware shard sizing
+
+`hardware.detect_hardware()` returns `vram_mb`. The node sends this to
+`/v1/network/assign?device=cuda&vram_mb=<n>&hf_repo=<repo>`. The tracker responds with
+a `{shard_start, shard_end}` gap that fits within the reported VRAM budget using the
+`bytes_per_layer` table from the model preset.
+
+When the tracker has no registered nodes for the model yet, `gap_found: false` is
+returned and the node defaults to the full model.
+
+### 3. --memory override
+
+`--memory MB` allows overriding the detected VRAM. Useful for CPU nodes (which report 0
+VRAM) that want to serve a specific slice using system RAM.
+
+### 4. Tracker network assignment endpoint
+
+`GET /v1/network/assign` replaces the old `GET /v1/nodes/assign`. It accepts
+`device`, `vram_mb`, and optionally `hf_repo`. It returns:
+
+```json
+{
+  "hf_repo": "Qwen/Qwen2.5-0.5B-Instruct",
+  "shard_start": 12,
+  "shard_end": 23,
+  "num_layers": 24,
+  "gap_found": true,
+  "price_per_token": 0.0
+}
+```
+
+`price_per_token` is reserved at 0.0 for future billing integration.
+
+## Alternatives rejected
+
+**Fixed shard table per model**: would require updating the code for every new model.
+HuggingFace config fetch is more general.
+
+**Node computes its own gap**: requires the node to know the full coverage map. The
+tracker already has this; having the tracker compute the assignment is cleaner.
+
+## Consequences
+
+- Nodes can join the network with a single command: `meshnet-node start --tracker <url>`
+- The tracker is now the authoritative source for shard assignment
+- VRAM budgets are advisory — nodes can still pin a range with explicit flags
+- `price_per_token: 0.0` is a stable protocol field; future billing sets it to a real value

docs/adr/0012-start-layer-overlapping-shards.md

@@ -0,0 +1,67 @@
+# ADR-0012: X-Meshnet-Start-Layer protocol for overlapping shard execution
+
+## Status: Accepted
+
+## Context
+
+The greedy route-selection algorithm picks a minimal set of nodes whose shard ranges
+collectively cover all model layers. This is exact when shard ranges are disjoint
+(node A: 0–11, node B: 12–23). But two nodes with overlapping ranges can also cover
+the full model (node A: 0–15, node B: 10–23).
+
+Without coordination, node B would re-run layers 10–15 on top of an activation tensor
+that already has those layers applied — producing silently wrong output.
+
+The question is: who resolves the overlap, and how?
+
+## Options considered
+
+**A. Tracker injects start_layer per hop (chosen)**  
+The tracker knows the full route when it builds `X-Meshnet-Route`. It computes
+`covered_up_to` as it walks the route and sets `start_layer = covered_up_to + 1`
+for each subsequent hop. The head node forwards this per-hop in
+`X-Meshnet-Start-Layer`. No peer-to-peer negotiation needed.
+
+**B. Each node negotiates with the next**  
+Node A would tell node B "I ran layers 0–15, you start from 16". This requires
+node A to know node B's shard range, which means an extra tracker lookup or
+exposing shard metadata in the activation wire protocol.
+
+**C. Strict non-overlapping enforcement**  
+Reject any route that contains overlapping nodes. Simpler but limits redundancy:
+two nodes with the same shard can't form a route even if their combined coverage
+is complete.
+
+## Decision
+
+Option A. The tracker is already the central coordinator; it already knows every
+node's shard range. Injecting `start_layer` at route-build time costs nothing and
+keeps the node implementation simple.
+
+## Wire protocol
+
+`X-Meshnet-Route` (JSON array, injected by tracker into the first-hop request):
+
+```json
+[
+  {"endpoint": "http://node-b:7002", "start_layer": 12, "relay_addr": null},
+  {"endpoint": "http://node-c:7003", "start_layer": 20}
+]
+```
+
+`X-Meshnet-Start-Layer` (integer header, forwarded by head node to each downstream hop):
+
+```
+X-Meshnet-Start-Layer: 12
+```
+
+The receiving node passes `start_layer` to `backend.forward_bytes(start_layer=12)`.
+The model shard skips transformer blocks below index 12.
+
+## Consequences
+
+- Overlapping shard registrations are valid and useful for redundancy
+- Route selection does not need to enforce disjoint ranges
+- The tracker carries the full route context; nodes are stateless w.r.t. routing
+- `start_layer` must be preserved through the relay path (included in hop dict)
+- Backward compatibility: if `start_layer` is absent, the node runs from its registered `shard_start`

docs/adr/0013-rolling-stats-smart-routing.md

@@ -0,0 +1,89 @@
+# ADR-0013: Rolling RPM statistics, smart assignment scoring, and throughput routing
+
+## Status: Accepted
+
+## Context
+
+The tracker made routing and assignment decisions blind to actual network traffic.
+Three related improvements were needed and designed together:
+
+1. **Model usage statistics** — how many requests per model, so the tracker knows demand
+2. **Smart assignment** — assign new nodes to where demand × unmet coverage is highest
+3. **Throughput routing** — when multiple nodes can complete a route, pick the faster ones
+
+## Decisions
+
+### 1. Rolling RPM counters
+
+`_RollingCounter` is a circular-bucket structure where each slot covers a fixed time epoch.
+Recording a value for the current epoch increments that slot; an expired slot is silently reset
+on the next write. Three windows per model:
+
+| Window | Buckets | Bucket size | Total span |
+|--------|---------|-------------|------------|
+| per_minute | 60 | 60 s | 1 hour |
+| per_hour | 24 | 3600 s | 1 day |
+| per_day | 30 | 86400 s | ~1 month |
+
+`rpm()` sums all non-stale buckets and divides by total window minutes.
+
+Alternative: exponential moving average (simpler, single float). Rejected because EMA
+cannot be persisted and restored without loss, and cannot be accurately merged from peer
+slices (each tracker runs its own requests, so merging EMA values doesn't give the true
+combined rate).
+
+### 2. Per-tracker stat slices + additive gossip
+
+Each tracker keeps only its own request slice. Gossip exchanges these slices and each tracker
+stores a `{tracker_url → {model → rpms}}` map. `get_combined_stats()` sums all slices.
+
+This is additive: if tracker A sees 10 RPM for model X and tracker B sees 5 RPM, combined
+is 15 RPM. Slices are keyed by tracker URL so a stale peer update simply overwrites its
+own key without corrupting other peers' data.
+
+Alternative: one global aggregator. Rejected — single point of failure, contradicts the
+distributed model.
+
+### 3. Assignment scoring formula
+
+```
+score(model) = (demand_rpm + 1.0) × (coverage_deficit + 0.01)
+```
+
+- `demand_rpm` = `get_combined_stats()[model]["rpm_last_hour"]`
+- `coverage_deficit` = fraction of model layers with zero-node coverage ∈ [0.0, 1.0]
+- `+1.0` floor: zero-traffic models still compete by coverage
+- `+0.01` floor: fully-covered models can still attract nodes if they have high demand
+
+The product ensures both dimensions matter: high demand but full coverage scores lower
+than high demand with partial coverage. Pure coverage deficits without traffic score
+lower than even modest traffic combined with any gap.
+
+`price_per_token: 0.0` is returned in the assignment response, reserved for future billing.
+
+### 4. Throughput tiebreak in route selection
+
+```
+effective_throughput(node) = benchmark_tokens_per_sec / (queue_depth + 1)
+```
+
+`_select_route` uses this as a tiebreak only: when two candidates reach the same
+maximum `shard_end`, the one with higher effective throughput is preferred.
+Coverage maximization remains the primary objective.
+
+`benchmark_tokens_per_sec` comes from the hardware profile at registration.
+`queue_depth` comes from the most recent heartbeat. A busy node (high queue)
+is deprioritized without being excluded.
+
+### 5. SQLite persistence
+
+Stats are saved to SQLite (configurable via `--stats-db PATH`) every 60 seconds and
+on clean shutdown. Schema: `model_rpm_buckets(model, window, bucket_idx, bucket_epoch, count)`.
+The circular-bucket structure maps directly — each slot is one row.
+
+## Consequences
+
+- Tracker startup is slightly slower when loading a large stats DB (sub-second for typical sizes)
+- Peer gossip adds one round-trip per gossip interval per peer
+- `price_per_token` is a stable wire field; future billing sets it to a real value
+- `effective_throughput` depends on `benchmark_tokens_per_sec` being set correctly at registration; nodes that don't set it get the default `1.0` and are treated as slowest

docs/adr/0014-relay-outbound-client.md

@@ -0,0 +1,120 @@
+# ADR-0014: Relay outbound client for NAT/internet pipeline hops
+
+## Status: Accepted
+
+## Context
+
+ADR-0010 describes the relay server: a public WebSocket hub where nodes behind NAT
+connect outbound and register as reachable peers. That ADR focused on the *inbound*
+side: how the tracker reaches a behind-NAT node for the initial chat request.
+
+The *pipeline hop* problem is different: when node A has the head shard and node B
+(behind NAT) has the tail shard, node A must forward binary activations to node B
+for *every generated token*. Direct HTTP from A to B is blocked. The relay must
+carry this per-hop activation traffic.
+
+### Why this is harder than tracker → node
+
+The tracker-to-node relay (ADR-0010) proxies a single JSON request. The activation
+hop carries raw bfloat16 tensors — binary data that must survive round-tripping
+through the relay's JSON message envelope without precision loss.
+
+Also, the relay `/rpc/{peer_id}` endpoint (one WebSocket connection per request)
+must be opened and closed for every token in the autoregressive loop. Latency
+of connection setup matters.
+
+## Options considered
+
+**A. Relay hop (WebSocket per hop, chosen)**  
+Node A opens a WebSocket to `wss://relay/rpc/{peer_id_B}`, sends the activation,
+receives the response, closes. The relay's `_handle_rpc` forwards it to B's persistent
+connection via the existing `relay-http-request` envelope mechanism.
+
+Pros: reuses the existing relay server unchanged. Each hop is independent; failures don't
+affect other requests.
+
+Cons: WebSocket connection setup adds ~50–150 ms per hop on a fast relay. For
+autoregressive inference (N tokens × M hops), this adds up.
+
+**B. Persistent per-session tunnel**  
+Node A opens a persistent WebSocket to the relay for the duration of an inference session
+and multiplexes all token hops over it.
+
+Pros: amortises connection setup across tokens.
+
+Cons: requires session-level state on the relay; complicates relay shutdown/failover;
+the current relay is stateless by design. Deferred for a future optimization.
+
+**C. Tracker-proxied activations**  
+Route all activation traffic through the tracker's HTTP proxy.
+
+Cons: the tracker is the control plane, not the data plane. High-volume binary tensor
+traffic through the tracker would saturate it. Rejected.
+
+## Decision
+
+Option A — per-hop WebSocket relay. Simple, reuses existing infrastructure, correct.
+Option B is noted as a future optimization when activation-path latency becomes the
+bottleneck.
+
+## Protocol
+
+```
+Node A opens WS → wss://relay/rpc/{peer_id_B}
+Node A sends:
+  {
+    "request_id": "<hex>",
+    "method": "POST",
+    "path": "/forward",
+    "headers": { "X-Meshnet-Shape": "...", "X-Meshnet-Start-Layer": "12", ... },
+    "body_base64": "<base64(bfloat16 tensor)>"
+  }
+
+Relay forwards to Node B as relay-http-request envelope.
+Node B's RelayHttpBridge decodes body_base64, calls POST /forward locally.
+Response:
+  {
+    "request_id": "<hex>",
+    "status": 200,
+    "headers": { "x-meshnet-shape": "...", "content-type": "application/octet-stream" },
+    "body_base64": "<base64(output tensor)>"   ← for binary responses
+    # OR
+    "body": "<json string>"                    ← for text (last-hop decode)
+  }
+Relay sends response JSON back to Node A.
+Node A decodes body_base64, continues pipeline.
+```
+
+### Binary data through JSON: base64
+
+Raw bfloat16 bytes cannot safely transit JSON (no UTF-8 guarantee, lossy decode).
+`body_base64` carries the tensor as base64; the bridge decodes it before calling
+the local HTTP endpoint, and re-encodes the response. No precision loss.
+
+Text responses (final hop, `application/json`) use `body` (plain string) for efficiency.
+
+### Fallback
+
+If `_relay_hop` raises (relay unreachable, peer disconnected), `_run_downstream_pipeline`
+logs a warning and retries via direct HTTP. If both fail, the hop returns a pipeline error
+string and the token is skipped.
+
+### Tracker injection
+
+The tracker's `_handle_proxy_chat` includes `relay_addr` in each downstream hop dict
+when the node has one registered:
+
+```json
+{"endpoint": "http://172.29.x.x:7002", "start_layer": 12, "relay_addr": "wss://relay/rpc/abc123"}
+```
+
+The head node reads `relay_addr` from the injected `X-Meshnet-Route` header and calls
+`_relay_hop` instead of direct HTTP.
+
+## Consequences
+
+- Nodes behind NAT (WSL2, 5G, home routers) can now participate in distributed pipeline inference without opening firewall ports
+- `relay_addr` is a stable registration field; nodes without a relay omit it and receive direct HTTP hops
+- Per-hop WebSocket setup adds latency proportional to relay RTT; acceptable for prototype, optimize later with persistent tunnels
+- Base64 encoding increases payload size by ~33%; acceptable for prototype
+- The relay server remains stateless and horizontally scalable; only the persistent per-peer `/ws` connections are stateful

docs/adr/0015-usdt-custodial-settlement.md

@@ -0,0 +1,33 @@
+# USDT-direct custodial settlement with pending-balance collateral
+
+Nodes are paid directly in USDT — 90% of inference fees, with 10% retained as the protocol cut. TAI (ADR-0002) is **deferred, not cancelled**: the reward token, revenue-backed floor, and halving curve remain the roadmap once volume exists, and the accumulated protocol cut is the future TAI liquidity source. Nothing in this design blocks minting TAI later (an SPL mint is a one-command operation); this ADR amends ADR-0002's settlement mechanics only.
+
+## The design
+
+**Custodial treasury.** A single project-owned Solana wallet holds all funds. Clients deposit USDT into it; a deposit watcher credits their API-key ledger balance off-chain. Node payouts are batched SPL transfers from the treasury. No Anchor/Rust programs are required — the entire on-chain surface is plain SPL token transfers, extending ADR-0007's contract-boundary approach with a real Solana adapter behind the same interface.
+
+**Off-chain ledger, periodic on-chain settlement.** Trackers meter every request against the client's ledger balance and accrue node earnings as a pending balance. Settlement follows the mining-pool standard — pay a node when `pending ≥ payout_threshold` OR `time_since_last_payout ≥ max_period`, whichever fires first, with a dust floor. Dev defaults: period 60s, threshold ≈ 0 (so every run is verifiable on-chain). Prod defaults: period 24h, threshold a few USDT. Both are dynamic config so the period can grow with volume to avoid chain overhead.
+
+**Raft leader settles.** In the tracker hive, only the current Raft leader runs the settlement loop; followers replicate the ledger. The treasury keypair is loaded only on settlement-capable trackers (operator-designated). Third-party trackers can join the mesh for routing but never hold the key.
+
+**Pricing.** Clients are charged a per-1K-tokens USDT price set per model in tracker config. Revenue per request is split among the nodes that served it proportional to work units (layers × tokens), reusing existing attribution.
+
+**Penalty = pending-balance forfeiture (amends ADR-0003).** The validator still re-runs ~5% of jobs. A caught cheater forfeits their entire pending balance to the treasury and receives a strike; three strikes bans the wallet. Because settlement is periodic, the pending balance itself is the collateral — no upfront stake deposit, preserving zero-friction node onboarding. With daily settlement, pending ≈ a day's earnings ≫ 20× the per-job gain at a 5% check rate, so cheating has negative expected value. The probationary period (ADR-0003 / issue 08) is retained as the anti-sybil re-entry cost.
+
+**Cluster.** Development targets **devnet** (supersedes the "testnet, never devnet" note in issue 06 — both are free; devnet is the ecosystem standard for app development with reliable faucets). Real USDT exists only on mainnet, so devnet uses a self-created mock-USDT SPL mint (6 decimals); the mint address is config, making mainnet cutover a config change. CI uses `solana-test-validator` or the local deterministic adapter.
+
+## Considered options
+
+- **Anchor escrow program now**: trustless, but requires the Rust/Anchor toolchain and can't ship today; the custodial trust assumption is acceptable pre-mainnet — deferred.
+- **Any tracker settles via lease**: most decentralized, but every mesh member would need the treasury key — rejected while custodial.
+- **USDT stake deposits for nodes**: stronger collateral, but adds onboarding friction that contradicts the mining-style UX; may return later as optional stake-for-priority — deferred.
+- **Keep TAI settlement (ADR-0002 as written)**: backing-price calc and buyback endpoint are too much machinery before there is volume — deferred.
+- **USDT-direct custodial settlement with pending-balance collateral**: shippable immediately, verifiable on-chain end-to-end on devnet — **chosen**.
+
+## Consequences
+
+- The entire payment system works with zero smart contracts; the first Anchor program can replace the custodial treasury later behind the same `packages/contracts` boundary.
+- Clients and nodes must trust the treasury key until then; this is explicit and acceptable pre-mainnet.
+- Node collateral scales with the settlement period: shortening the period in prod weakens the fraud deterrent, so period changes must consider both chain overhead and collateral size.
+- A node caught cheating immediately after a payout has little pending balance to forfeit — the strike/ban system covers this window.
+- The 10% protocol cut accumulates in the treasury as the future TAI liquidity reserve.

docs/agents/issue-tracker.md

@@ -1,19 +1,47 @@
 # Issue tracker: Local Markdown
 
-Issues and PRDs for this repo live as markdown files in `.scratch/`.
+## Lifecycle
 
-## Conventions
+Active feature work lives in `.scratch/<feature-slug>/`. When a feature is complete, `/close-feature` graduates it into `docs/`. This keeps in-progress work isolated from the permanent record.
 
-- One feature per directory: `.scratch/<feature-slug>/`
+```
-- The PRD is `.scratch/<feature-slug>/PRD.md`
+/grilling → /to-prd → /to-issues → .scratch/<slug>/
-- Implementation issues are `.scratch/<feature-slug>/issues/<NN>-<slug>.md`, numbered from `01`
+                                         ↓ all Status: done
-- Triage state is recorded as a `Status:` line near the top of each issue file (see `triage-labels.md` for the role strings)
+                                    /close-feature
-- Comments and conversation history append to the bottom of the file under a `## Comments` heading
+                                         ↓
+                                      docs/
+```
+
+## During a feature (in .scratch/)
+
+- `.scratch/<slug>/PRD.md` — narrative PRD
+- `.scratch/<slug>/prd.json` — machine-readable story list (used by ralph)
+- `.scratch/<slug>/issues/<NN>-<slug>.md` — user stories, numbered from `01`
+- Triage state is a `Status:` line near the top of each issue (see `triage-labels.md`)
+- Comments and conversation history append under a `## Comments` heading
+
+## After close (in docs/)
+
+- `docs/PRD.md` — narrative PRD (appended under a feature heading)
+- `docs/prd.json` — merged story list
+- `docs/issues/<NN>-<slug>.md` — renumbered from next available slot
+- `docs/adr/<NNNN>-<slug>.md` — any ADRs from the feature, renumbered
+
+## Closing a feature
+
+Run `/close-feature <slug>` when all issues have `Status: done`. The skill will:
+1. Verify all issues are done (stops if any are incomplete)
+2. Show what will move and ask for confirmation
+3. `git mv` everything into `docs/` with correct numbering
+4. Merge prd.json user stories into `docs/prd.json`
+5. Delete `.scratch/<slug>/` and commit
+
+The agent should also **proactively offer** to run `/close-feature` when it notices all issues in a `.scratch/<slug>/` are done.
 
 ## When a skill says "publish to the issue tracker"
 
-Create a new file under `.scratch/<feature-slug>/` (creating the directory if needed).
+Create a new file under `.scratch/<feature-slug>/issues/` following the numbering convention. Create the directory if it doesn't exist.
 
 ## When a skill says "fetch the relevant ticket"
 
-Read the file at the referenced path. The user will normally pass the path or the issue number directly.
+Read the file at the referenced path. The user will normally pass the path or issue number directly. Check both `docs/issues/` (closed features) and `.scratch/*/issues/` (active features).

docs/issues/13-coverage-first-shard-assignment.md

@@ -47,3 +47,7 @@ Model preset metadata (stored in tracker config) includes `bytes_per_layer` at e
 - Rebalance directives are delivered as responses to node heartbeat POSTs (node polls tracker every 10s anyway) — no new push channel needed
 - `bytes_per_layer` is read from a `model_presets.json` config file; add GPT-2 (12 layers, ~30MB/layer bfloat16) as the test preset
 - The current tracker `_select_route` function is extended, not replaced — backward compat with stub nodes that don't send VRAM data (default to 8GB / bfloat16 if omitted)
+
+## Comments
+
+- 2026-06-30: Follow-up capacity hardening lives in `20-memory-budget-shard-slots-and-dropout-relocation.md`. US-013 remains the base coverage-first assignment and dropout rebalance story; US-020 owns operator `--memory`, `--max-shards`, shard-slot enforcement, and relocation limit hardening so the two scopes do not conflict.

docs/issues/16-mining-cli-ux.md

@@ -0,0 +1,206 @@
+# US-016 — Mining-style node startup CLI + live dashboard
+
+## Goal
+
+Replace the bare flag-driven `meshnet-node start` with a wizard-guided first-run experience modelled on GPU mining clients (like PhoenixMiner, lolMiner, etc.). After the wizard, the terminal switches to a live status dashboard showing real-time node health and earnings.
+
+## Wizard flow (first run only)
+
+```
+╔══════════════════════════════════════════════════════════╗
+║              meshnet-node  v0.1.0                        ║
+║      Distributed AI Inference — Node Setup               ║
+╚══════════════════════════════════════════════════════════╝
+
+Detecting hardware...
+  GPU 0: NVIDIA RTX 4090  24 GB VRAM  ✓
+  GPU 1: NVIDIA RTX 3090  24 GB VRAM  ✓
+
+Select a model to serve:
+
+ #   Model                        Layers   NF4    INT8   BF16
+ 1   Llama-3-70B-Instruct         80       ✓18GB  ✓40GB  ✗80GB
+ 2   Qwen-2.5-72B-Instruct        80       ✓19GB  ✗41GB  ✗81GB
+ 3   Mixtral-8x7B-Instruct-v0.1   32       ✓ 7GB  ✓14GB  ✓27GB
+ 4   Phi-3-medium-128k-instruct   40       ✓ 4GB  ✓ 8GB  ✓15GB
+ 5   [Browse HuggingFace…]
+
+Enter number [1]: _
+
+Quantization [nf4/int8/bf16] (nf4 recommended for 24GB): _
+
+Download directory [~/.meshnet/models]: _
+
+Tracker URL [http://localhost:8080]: _
+
+Wallet path [~/.config/meshnet/wallet.json] (new wallet will be created): _
+
+Config saved to ~/.config/meshnet/config.json
+Starting node…
+```
+
+Second run with existing config:
+
+```
+meshnet-node
+Reading config from ~/.config/meshnet/config.json
+Model: Llama-3-70B-Instruct  Quant: nf4  Shard: layers 0–15
+Tracker: http://192.168.1.10:8080
+Starting…
+```
+
+## Live dashboard (once running)
+
+Renders every 2 seconds using `rich.live`. Fallback: plain-text status line if `rich` is unavailable or terminal is not a TTY (important for WSL2 / SSH).
+
+```
+meshnet-node  Llama-3-70B-Instruct [nf4]  shard 0–15/80  up 00:03:22
+
+GPU 0  RTX 4090       GPU ████████░░  73%   VRAM 18.2/24.0 GB   45°C
+GPU 1  RTX 3090       GPU ███░░░░░░░  28%   VRAM  8.7/24.0 GB   38°C
+
+Tokens/sec   ▁▂▃▄▅▆▇█  42.3 t/s (EMA 30s)
+Requests     1,247 served   3 active
+Peers        8 connected   (tracker: ✓  relay: ✓)
+TAI earned   0.00 TAI  (payments active after US-006)
+Uptime       00:03:22
+
+[q] quit   [r] reset stats   [c] compact view
+```
+
+Compact mode (`--compact` or pressing `c`) shows a single status line:
+```
+[43t/s  VRAM18.2GB  req1247  peers8  up3m22s]
+```
+
+## Implementation notes
+
+### Hardware detection
+
+```python
+import torch
+
+def detect_gpus() -> list[dict]:
+    gpus = []
+    if torch.cuda.is_available():
+        for i in range(torch.cuda.device_count()):
+            props = torch.cuda.get_device_properties(i)
+            gpus.append({
+                "index": i,
+                "name": props.name,
+                "vram_gb": props.total_memory / 1e9,
+                "backend": "cuda"
+            })
+    # ROCm / Apple Silicon stubs for later
+    return gpus
+```
+
+### Curated model list
+
+`packages/node/meshnet_node/model_catalog.py` — a hardcoded list of `ModelPreset` dataclasses:
+
+```python
+@dataclass
+class ModelPreset:
+    name: str               # display name
+    hf_repo: str            # HuggingFace repo ID
+    num_layers: int
+    vram_gb: dict           # {"nf4": 18, "int8": 40, "bf16": 80}
+    description: str        # one-line description
+```
+
+Initial list (expand over time):
+- `meta-llama/Meta-Llama-3-70B-Instruct` — 80L, NF4 18GB, INT8 40GB, BF16 80GB
+- `Qwen/Qwen2.5-72B-Instruct` — 80L, NF4 19GB, INT8 41GB, BF16 81GB
+- `mistralai/Mixtral-8x7B-Instruct-v0.1` — 32L, NF4 7GB, INT8 14GB, BF16 27GB
+- `microsoft/Phi-3-medium-128k-instruct` — 40L, NF4 4GB, INT8 8GB, BF16 15GB
+- `google/gemma-2-27b-it` — 46L, NF4 10GB, INT8 20GB, BF16 40GB
+
+### HuggingFace Browse
+
+```python
+from huggingface_hub import list_models
+
+def browse_hf(top_n=20) -> list[dict]:
+    models = list_models(
+        pipeline_tag="text-generation",
+        library="transformers",
+        sort="downloads",
+        direction=-1,
+        limit=top_n,
+        cardData=True,
+    )
+    return [{"repo": m.modelId, "downloads": m.downloads} for m in models]
+```
+
+### Persistent config
+
+`~/.config/meshnet/config.json`:
+```json
+{
+  "model_hf_repo": "meta-llama/Meta-Llama-3-70B-Instruct",
+  "quantization": "nf4",
+  "download_dir": "~/.meshnet/models",
+  "tracker_url": "http://192.168.1.10:8080",
+  "wallet_path": "~/.config/meshnet/wallet.json",
+  "shard_start": null,
+  "shard_end": null,
+  "updatedAt": "2026-06-29T..."
+}
+```
+
+`shard_start`/`shard_end`: null means tracker auto-assigns. User can pin a range for dedicated partial-model nodes.
+
+### CLI flags
+
+All wizard answers are overridable without re-running the wizard:
+
+```
+meshnet-node [start]
+  --model <hf-repo-id>            # e.g. meta-llama/Meta-Llama-3-70B-Instruct
+  --quantization [bf16|int8|nf4]
+  --download-dir <path>
+  --tracker <url>
+  --wallet <path>
+  --shard-start <int>             # pin shard range (optional)
+  --shard-end <int>
+  --reset-config                  # ignore saved config, re-run wizard
+  --no-tui                        # plain-text output (for CI / headless)
+  --compact                       # single-line status instead of full dashboard
+
+meshnet-node models                # list curated models and exit
+meshnet-node models --browse       # list HF Hub top-20 and exit
+meshnet-node config                # print current config and exit
+```
+
+### WSL2 / non-TTY fallback
+
+```python
+import sys, os
+
+def is_interactive_tty() -> bool:
+    return sys.stdout.isatty() and os.environ.get("TERM") not in ("dumb", "")
+
+if not is_interactive_tty():
+    # fall back to plain-text periodic status
+    run_plain_status_loop(node)
+else:
+    run_rich_dashboard(node)
+```
+
+Do NOT use `termios`, `fcntl`, or `/dev/tty` — these break in Windows cmd.exe and some WSL2 terminal emulators.
+
+## Acceptance criteria
+
+- `meshnet-node` with no args and no config → wizard starts
+- Wizard detects GPU and marks `[too large]` for models that exceed available VRAM
+- `meshnet-node models` prints curated list and exits
+- `meshnet-node models --browse` calls HF Hub API, prints top-20, exits
+- Second run (config exists) → skips wizard, starts immediately
+- `--reset-config` re-runs wizard even with config present
+- All wizard inputs override-able via CLI flags
+- Live rich dashboard renders and updates every 2s when running in a TTY
+- Falls back to plain-text when not a TTY (CI / WSL2 without TERM set)
+- Ctrl-C prints a clean summary line and exits 0
+- `python -m pytest` passes from repo root
+- Commit only this story's changes

docs/issues/17-p2p-gossip-relay-ssl.md

@@ -0,0 +1,205 @@
+# US-017 — P2P gossip, NAT-traversal relay node, and SSL/TLS
+
+## Goal
+
+Nodes must work behind NAT (home routers, cloud VMs without public IPs) and must communicate securely. Implement:
+
+1. **SSL/TLS everywhere** — all HTTP between nodes/tracker is HTTPS; all WebSocket gossip is WSS
+2. **mDNS peer discovery** — nodes on the same LAN find each other automatically (no config)
+3. **WebSocket gossip PubSub** — nodes propagate join/leave/coverage-update events in near-real-time
+4. **Circuit relay node** — team-run public relay (`packages/relay`) that enables NAT traversal and bootstraps new nodes joining from the internet
+
+Architecture is designed to migrate to libp2p GossipSub + Kademlia DHT without breaking the message schema (topic names and payload formats are stable contracts).
+
+## Gossip protocol
+
+### Transport
+
+WebSocket (`wss://`) using the `websockets` Python library. Each node maintains persistent WSS connections to:
+- The relay node (always, bootstraps peer list)
+- Up to 8 direct peers (Kademlia-style target fanout; peers discovered via mDNS + relay peer list)
+
+### Topics
+
+All messages are JSON with an envelope:
+
+```json
+{
+  "topic": "node-join",
+  "version": 1,
+  "from_peer": "<peer_id>",
+  "timestamp": "<iso8601>",
+  "payload": { ... }
+}
+```
+
+| Topic | Direction | Payload |
+|-------|-----------|---------|
+| `node-join` | broadcast | `{peer_id, addr, models: [{model_preset, shard_start, shard_end}], vram_gb, quant}` |
+| `node-leave` | broadcast | `{peer_id, reason}` |
+| `coverage-update` | broadcast | `{model_preset, coverage: [{start, end, count}]}` |
+| `heartbeat` | peer→relay | `{peer_id, addr, uptime_s, tokens_per_sec}` |
+| `peer-list` | relay→peer | `{peers: [{peer_id, addr}]}` |
+| `relay-announce` | relay→all | `{relay_id, relay_url, capacity}` |
+
+Gossip fanout: each node re-broadcasts received messages to all its peers (simple flooding with `seen_ids` dedup, TTL=3 hops). Migration to GossipSub mesh routing is a later ADR.
+
+### Peer ID
+
+`peer_id = sha256(public_key)[:16].hex()` — generated on first run, stored in `~/.config/meshnet/identity.json`. The same keypair is used for TLS client certificates (mTLS) in future work.
+
+## mDNS LAN discovery
+
+Use Python `zeroconf` library. Service type: `_meshnet._tcp.local.`
+
+```python
+from zeroconf import ServiceInfo, Zeroconf
+
+info = ServiceInfo(
+    "_meshnet._tcp.local.",
+    f"{peer_id}._meshnet._tcp.local.",
+    addresses=[socket.inet_aton(local_ip)],
+    port=node_port,
+    properties={"peer_id": peer_id, "version": "1"},
+)
+zc = Zeroconf()
+zc.register_service(info)
+```
+
+On startup, nodes also browse for `_meshnet._tcp.local.` to discover existing nodes. mDNS is LAN-only (does not traverse routers), which is correct for LAN discovery.
+
+## NAT traversal: circuit relay
+
+### How it works
+
+1. Node A (behind NAT) cannot accept inbound TCP connections
+2. Node A connects outbound to the public relay via WSS
+3. Node A tells the tracker: `"effective_addr": "wss://relay.meshnet.ai/relay/{peer_id_A}"`
+4. Node B (wants to call A) connects to the relay at the above URL
+5. Relay proxies the TCP stream between A and B
+
+Hole-punching (direct connection via STUN) is attempted first (future work). Relay is the fallback.
+
+### meshnet-relay
+
+`packages/relay/meshnet_relay/server.py` — a standalone aiohttp server:
+
+```
+GET  /health                        → {status: ok}
+GET  /v1/peers                      → [{peer_id, addr, last_seen}]
+POST /v1/gossip                     → receive a gossip message, fan out to connected peers
+WSS  /ws                            → persistent gossip connection (subscribe to all topics)
+WSS  /relay/{peer_id}               → circuit relay proxy to that peer_id
+GET  /v1/relay/capacity             → {connected_peers: N, max_peers: 500}
+```
+
+CLI:
+
+```
+meshnet-relay [--port 8443] [--cert path/to/cert.pem] [--key path/to/key.pem]
+              [--tracker-url http://...] [--max-peers 500]
+```
+
+The relay can optionally proxy to the tracker (so `relay.meshnet.ai` is the single internet-visible endpoint).
+
+## SSL/TLS setup
+
+### Node certificate (self-signed, auto-generated)
+
+On first run, `meshnet-node` generates a self-signed RSA-2048 cert valid for 10 years:
+
+```python
+from cryptography import x509
+from cryptography.hazmat.primitives import hashes, serialization
+from cryptography.hazmat.primitives.asymmetric import rsa
+```
+
+Cert saved to `~/.config/meshnet/node_cert.pem` + `node_key.pem`. Fingerprint stored in config and shared with tracker via heartbeat. Nodes connecting to each other validate the fingerprint (TOFU — trust on first use), not the CA chain.
+
+### Relay certificate
+
+The relay uses a real Let's Encrypt cert (cert-bot or acme.sh). The relay cert is pinned in `packages/p2p/relay_bootstrap.json`:
+
+```json
+{
+  "relays": [
+    {
+      "url": "wss://relay.meshnet.ai:8443",
+      "cert_fingerprint": "sha256:<hex>",
+      "operator": "meshnet-team"
+    }
+  ]
+}
+```
+
+### All HTTP switched to HTTPS
+
+`meshnet-node` starts an HTTPS server using `ssl.SSLContext`. `meshnet-tracker` similarly. All outbound `httpx` / `aiohttp` calls use TLS verification against pinned fingerprints (not the system CA store — too many corporate proxies break this).
+
+## Tracker changes
+
+Heartbeat payload gains new fields:
+
+```json
+{
+  "peer_id": "a1b2c3d4e5f6a1b2",
+  "effective_addr": "https://192.168.1.42:8001",
+  "relay_addr": "wss://relay.meshnet.ai:8443/relay/a1b2c3d4e5f6a1b2",
+  "cert_fingerprint": "sha256:...",
+  "gossip_peers": ["peer_id_1", "peer_id_2"]
+}
+```
+
+Tracker uses `effective_addr` (direct) or `relay_addr` (fallback) when building inference routes.
+
+## Integration test
+
+```
+tests/test_gossip_and_relay.py
+
+scenario:
+  1. Start a local relay (localhost:18443)
+  2. Start node A (no inbound port — simulate NAT by binding to 127.0.0.1 only)
+  3. Start node B (public-reachable on localhost)
+  4. Both register with relay; relay peer-list includes both
+  5. Node B sends a gossip node-join message
+  6. Assert node A receives it within 500ms
+  7. Start tracker; confirm tracker's node registry includes node A via relay_addr
+  8. Send inference request; assert it routes through relay to node A
+```
+
+## Package layout
+
+```
+packages/relay/
+  pyproject.toml
+  meshnet_relay/
+    __init__.py
+    server.py          # aiohttp relay + gossip hub + circuit relay proxy
+    cli.py             # meshnet-relay entrypoint
+    peer_registry.py   # in-memory {peer_id: {addr, last_seen, ...}}
+    circuit_relay.py   # WSS proxy between two peers
+
+packages/p2p/
+  meshnet_p2p/
+    gossip.py          # GossipClient — connect to relay + peers, pub/sub
+    mdns.py            # ZeroconfDiscovery — mDNS announce + browse
+    identity.py        # PeerIdentity — generate/load peer_id + keypair
+    tls.py             # cert generation, fingerprint, SSLContext helpers
+
+packages/node/meshnet_node/
+  gossip_integration.py  # wires GossipClient into node lifecycle
+```
+
+## Acceptance criteria
+
+- All node↔node and node↔tracker HTTP uses HTTPS; self-signed cert auto-generated on first run
+- `cert_fingerprint` included in heartbeat; tracker stores and logs it
+- mDNS: two nodes on the same LAN discover each other without manual tracker URL (test with two localhost processes using different mDNS names)
+- Relay: `meshnet-relay` starts, accepts WSS connections, fans out gossip messages to all connected peers
+- Circuit relay: node A (127.0.0.1-only) can receive a gossip message via the relay from node B
+- Tracker routes inference to node A using `relay_addr` when direct addr not reachable
+- `relay_bootstrap.json` exists in `packages/p2p/` with at least one entry (localhost for tests)
+- ADR-0010 documents the gossip architecture and libp2p migration path
+- `python -m pytest` passes from repo root
+- Commit only this story's changes

docs/issues/18-two-machine-lan-test.md

@@ -0,0 +1,157 @@
+# US-018 — End-to-end two-machine LAN inference test
+
+## Goal
+
+Run real distributed inference across two physical machines: the Linux rig and a Windows 11 rig running WSL2. Document every setup step, firewall rule, and gotcha so this is repeatable. The test script exits 0 with token output and timing, proving the network works.
+
+## Network topology for LAN test
+
+```
+[Linux machine]                           [Windows 11 / WSL2]
+  meshnet-tracker  :8080                    meshnet-node (shard B)
+  meshnet-node     :8001  (shard A, tracker-mode)
+  meshnet-gateway  :8000  (optional, for OpenAI-compat)
+
+Client (either machine):
+  scripts/test_lan_inference.py --tracker http://192.168.1.10:8080
+```
+
+The Linux machine runs the tracker + the first-shard node (tracker-mode). The Windows/WSL2 machine runs the second-shard node. A small model (e.g. Phi-3-medium at BF16, fits on one GPU each) is split across both.
+
+## WSL2 setup (Windows side)
+
+`docs/INSTALL_WINDOWS.md` covers:
+
+1. Enable WSL2: `wsl --install -d Ubuntu-24.04`
+2. CUDA in WSL2: install NVIDIA driver on Windows (NOT inside WSL); WSL2 gets CUDA automatically
+   - Verify: `nvidia-smi` inside WSL2 should show GPU
+3. Install Python 3.11+ and pip inside WSL2
+4. `pip install -e packages/node packages/p2p` (clone repo first)
+5. Firewall: Windows Defender must allow inbound WSL2 → LAN on node port
+   - PowerShell: `New-NetFirewallRule -DisplayName "meshnet-node" -Direction Inbound -Protocol TCP -LocalPort 8001 -Action Allow`
+6. WSL2 IP: WSL2 has its own NAT'd IP (172.x.x.x); to expose to LAN, either:
+   - Option A: `netsh interface portproxy add v4tov4 listenport=8001 listenaddress=0.0.0.0 connectport=8001 connectaddress=$(wsl hostname -I)`
+   - Option B: use the relay node (US-017) — no port forwarding needed
+
+## Linux setup
+
+Standard install (already done after US-016). Firewall:
+
+```bash
+# If using ufw
+sudo ufw allow 8080/tcp  # tracker
+sudo ufw allow 8001/tcp  # node
+sudo ufw allow 8000/tcp  # gateway (optional)
+```
+
+## Model split
+
+For the test, use a model that has enough layers to split meaningfully but fits comfortably in memory. Phi-3-medium-128k-instruct (40 layers, BF16 15GB) works on a single 24GB GPU on each machine:
+
+- Linux node: layers 0–19 (tracker-mode, owns tokenizer + embed_tokens)
+- Windows/WSL2 node: layers 20–39
+
+Start sequence:
+```bash
+# Terminal 1 (Linux) — tracker
+meshnet-tracker --port 8080
+
+# Terminal 2 (Linux) — first-shard node (tracker-mode auto-detected because shard_start=0)
+meshnet-node --model microsoft/Phi-3-medium-128k-instruct \
+             --quantization bf16 \
+             --shard-start 0 --shard-end 19 \
+             --tracker http://localhost:8080 \
+             --port 8001
+
+# Terminal 3 (Windows WSL2) — second-shard node
+meshnet-node --model microsoft/Phi-3-medium-128k-instruct \
+             --quantization bf16 \
+             --shard-start 20 --shard-end 39 \
+             --tracker http://192.168.1.10:8080 \
+             --port 8001
+```
+
+## Test script
+
+`scripts/test_lan_inference.py`:
+
+```python
+#!/usr/bin/env python3
+"""
+End-to-end LAN inference test.
+Usage: python scripts/test_lan_inference.py --tracker http://192.168.1.10:8080
+"""
+import argparse, time, httpx, json
+
+MESSAGES = [
+    {"role": "user", "content": "What is 7 × 8? Answer in one word."},
+    {"role": "user", "content": "Name the capital of France in one word."},
+    {"role": "user", "content": "Complete the sequence: 1, 1, 2, 3, 5, ___"},
+]
+
+def run_test(tracker_url: str, gateway_url: str | None):
+    # Discover inference entry point via tracker if gateway not given
+    if not gateway_url:
+        r = httpx.get(f"{tracker_url}/v1/tracker-nodes/phi-3-medium", timeout=5)
+        r.raise_for_status()
+        nodes = r.json()
+        assert nodes, "No tracker-mode nodes registered — is the first-shard node running?"
+        gateway_url = nodes[0]["url"]
+
+    print(f"Inference endpoint: {gateway_url}")
+    print(f"Tracker: {tracker_url}")
+    print()
+
+    for i, msg in enumerate(MESSAGES):
+        t0 = time.monotonic()
+        r = httpx.post(
+            f"{gateway_url}/v1/chat/completions",
+            json={"model": "phi-3-medium", "messages": [msg], "stream": False},
+            timeout=60,
+        )
+        r.raise_for_status()
+        data = r.json()
+        elapsed = time.monotonic() - t0
+
+        content = data["choices"][0]["message"]["content"]
+        tokens = data["usage"]["completion_tokens"]
+        tps = tokens / elapsed if elapsed > 0 else 0
+
+        print(f"[{i+1}] Q: {msg['content']}")
+        print(f"     A: {content}")
+        print(f"     {tokens} tokens  {elapsed:.2f}s  {tps:.1f} t/s")
+        print()
+
+    print("✓ All 3 requests completed successfully")
+
+if __name__ == "__main__":
+    p = argparse.ArgumentParser()
+    p.add_argument("--tracker", required=True)
+    p.add_argument("--gateway", default=None)
+    args = p.parse_args()
+    run_test(args.tracker, args.gateway)
+```
+
+## Docs: TWO_MACHINE_TEST.md
+
+`docs/TWO_MACHINE_TEST.md` must cover:
+
+1. Prerequisites (models downloaded on both machines, same model ID, complementary shard ranges)
+2. Start order: tracker first, then nodes, then test script
+3. How to verify nodes are registered: `GET /v1/nodes` on tracker
+4. How to verify coverage: `GET /v1/coverage/phi-3-medium` — all 40 layers must show node_count ≥ 1
+5. How to run the test script
+6. Expected output
+7. Latency breakdown: how to read per-hop latency from node logs
+8. **Known Issues** section — updated during actual test run with real gotchas
+
+## Acceptance criteria
+
+- `docs/INSTALL_WINDOWS.md` covers WSL2 + CUDA + meshnet-node install end-to-end
+- `docs/TWO_MACHINE_TEST.md` covers the full two-machine setup and test procedure
+- `scripts/test_lan_inference.py` exists and is executable
+- When run against a real two-machine LAN setup: script exits 0, prints 3 valid answers with timing
+- Coverage map shows 100% coverage (no gap) after both nodes register
+- Known Issues section in TWO_MACHINE_TEST.md contains at least the issues encountered during this test run
+- No new pytest failures from repo root (this story adds docs + a script, not new Python packages)
+- Commit only this story's changes

docs/issues/19-binary-data-plane-and-peer-weight-transfer.md

@@ -0,0 +1,68 @@
+# US-019 — Binary data plane and optional peer weight transfer
+
+Status: needs-triage
+Priority: Low
+Stage: Design parking lot
+
+## Context
+
+The current project focus is inference democratization: let small GPU owners contribute useful compute and let users run inference on models larger than one host can serve alone. Weight distribution is useful, but it is secondary to low-latency distributed inference.
+
+Recent findings:
+
+- HuggingFace already handles initial model origin distribution well enough for the first working version.
+- The inference-critical path is activation transfer between shard nodes, not torrenting model files.
+- Torrent/content-addressed transfer is a good future fit for model weights, shard cache replication, fine-tuned models, and offline/local swarm behavior.
+- Torrenting is not a good fit for activation traffic because activations are latency-sensitive, ordered, session-specific binary streams.
+
+## Design note
+
+Keep the tracker as the control plane:
+
+- node registration
+- heartbeats
+- route selection
+- model/shard manifests
+- peer/checksum metadata
+
+Keep binary payloads on the data plane:
+
+- direct node-to-node activation transfer where reachable
+- relay/QUIC/WSS fallback where direct transport is unavailable
+- future peer weight transfer as content-addressed blobs or pieces
+
+## Potential future direction
+
+For inference traffic:
+
+- Prefer direct binary transport with backpressure.
+- Use raw binary activation frames rather than JSON/base64.
+- Preserve tensor metadata out-of-band: shape, dtype, session, chunk index, encoding.
+- Consider QUIC or a mature NAT-friendly transport for direct-when-possible, relay-when-needed behavior.
+
+For model weights:
+
+- Keep HuggingFace as default origin and fallback.
+- Add peer cache transfer only as an optimization.
+- Consider a real content-addressed/torrent-like library or sidecar for weight blobs.
+- Store manifests and checksums in tracker state so peers can verify exact shard contents.
+- Prefer piece/chunk transfer with resume and hash verification over one giant tarball.
+
+## Not in scope now
+
+- Replacing HuggingFace as the primary model origin.
+- Building a full BitTorrent/IPFS/libp2p subsystem.
+- Routing activation traffic through a torrent protocol.
+- Making peer weight transfer mandatory for node startup.
+
+## Acceptance criteria for a future implementation issue
+
+- [ ] Activation transfer remains binary end-to-end and avoids JSON/base64 payloads.
+- [ ] Tracker does not proxy large binary payloads except as an explicit fallback path.
+- [ ] Weight transfer, if added, is optional and falls back to HuggingFace.
+- [ ] Weight pieces are content-addressed and checksum-verified.
+- [ ] The design preserves low-latency inference as the primary objective.
+
+## Comments
+
+Created as a low-priority design parking-lot item after discussing inference democratization versus weight distribution. Do not pick up for implementation until the core public tracker, relay, and binary activation path are stable.

docs/issues/20-memory-budget-shard-slots-and-dropout-relocation.md

@@ -0,0 +1,66 @@
+Status: ready-for-agent
+
+# US-020 - Memory budget, shard slots, and dropout relocation hardening
+
+## Goal
+
+Make node capacity limits explicit and enforce them consistently when the tracker assigns, rebalances, and relocates shards after a node dropout.
+
+This is a follow-up to US-013, not a replacement. US-013 owns the coverage-first assignment and rebalance algorithm. This issue hardens the capacity contract around that algorithm: operator memory budget, maximum loaded shard slots, and relocation behavior when one node must absorb or split ranges after another node disappears.
+
+## Context
+
+Recent work added the first part of the contract:
+
+- `meshnet-node --memory MB` is registered with the tracker as `vram_bytes` when explicitly set.
+- CPU nodes without `--memory` keep the tracker default capacity, preserving old behavior.
+- `meshnet-node --max-shards N` is accepted and registered as `max_loaded_shards`.
+- Tracker registration validates `max_loaded_shards >= 1`.
+
+The current runtime still effectively has one active backend shard per node. A node may advertise `max_loaded_shards`, but the tracker does not yet use multiple shard slots in bin-packing, and the node does not yet host multiple concurrently loaded shard ranges.
+
+## Scope
+
+- Make tracker rebalance logic account for `max_loaded_shards` as a capacity multiplier or explicit shard-slot list.
+- Ensure a node is never assigned more total layers than its memory budget can support across all loaded shard slots.
+- Decide and implement the runtime behavior for multiple loaded shards:
+  - either support multiple concurrently loaded shard backends on one node, or
+  - keep one backend active and treat `max_loaded_shards` as future metadata, with tracker enforcement preventing multi-range assignment for now.
+- On heartbeat timeout, relocate the dropped node's uncovered layer range to eligible managed nodes while respecting both memory and shard-slot limits.
+- Surface the effective memory budget and shard slot count in tracker/network inspection output so operators can diagnose why a node did or did not receive a range.
+
+## Non-Goals
+
+- Do not redesign the US-013 coverage-first algorithm from scratch.
+- Do not change relay, `/ws`, or `/rpc` behavior.
+- Do not change the token/reward model.
+- Do not require public internet verification; all behavior must be locally testable.
+
+## Acceptance Criteria
+
+- Tracker stores and exposes `max_loaded_shards` for registered nodes.
+- Assignment/rebalance never exceeds:
+  - `assigned_layers_total <= floor((vram_bytes * 0.8) / bytes_per_layer_at_quant)`
+  - `assigned_range_count <= max_loaded_shards`
+- A managed node with `max_loaded_shards=1` only receives one active shard range.
+- A managed node with `max_loaded_shards=2` can absorb two non-contiguous uncovered ranges only if the node runtime supports serving both; otherwise tracker must keep assigning at most one range and document `max_loaded_shards` as reserved.
+- Dropout test: register nodes covering a model, let a middle/tail node heartbeat-expire, and assert the tracker queues `LOAD_SHARD` directives that restore full coverage without violating memory or shard-slot limits.
+- CLI test: `--memory` and `--max-shards` are reflected in the registration payload.
+- `python -m pytest tests/test_tracker_routing.py tests/test_node_startup.py` passes in the project virtualenv, aside from any pre-existing platform-specific wallet permission assertion documented in the final notes.
+
+## Implementation Notes
+
+- Existing files likely involved:
+  - `packages/node/meshnet_node/cli.py`
+  - `packages/node/meshnet_node/startup.py`
+  - `packages/node/meshnet_node/torch_server.py`
+  - `packages/tracker/meshnet_tracker/server.py`
+  - `tests/test_tracker_routing.py`
+  - `tests/test_node_startup.py`
+- Keep backward compatibility: nodes that omit `vram_bytes` default to tracker defaults; nodes that omit `max_loaded_shards` default to `1`.
+- Prefer a small internal representation for assigned ranges if multiple ranges become real, for example `assigned_shards: list[tuple[int, int]]`, while preserving `shard_start`/`shard_end` in public responses for single-range nodes.
+
+## Comments
+
+- 2026-06-30: Created after implementing the initial registration plumbing in commit `f1e4ed6` (`--memory`, `--max-shards`, tracker validation). This issue captures the remaining end-to-end behavior so it does not conflict with US-013.
+- 2026-06-30: Implementation decision: `max_loaded_shards` is currently a validated and exposed capacity field, but multi-range assignment remains reserved because `TorchNodeServer` serves one active backend shard. The tracker therefore emits at most one active range per node while exposing `vram_bytes`, `ram_bytes`, `max_loaded_shards`, quantization, throughput, and computed `max_assignable_layers` in inspection endpoints.

docs/issues/20-tracker-node-hardening.md

@@ -0,0 +1,20 @@
+# US-020 — Tracker + node hardening: BrokenPipe fix, deterministic node IDs, HF coverage
+
+Status: done
+Priority: High
+Stage: Maintenance
+
+## Context
+
+First two-machine LAN test (US-018) exposed three reliability issues:
+
+1. `BrokenPipeError` crash in tracker `_send_json` when a slow-inference client disconnected mid-response
+2. Random UUID node IDs meant every re-registration (after tracker restart) created a phantom entry
+3. `GET /v1/coverage/<model>` returned no results when called with a short name (`Qwen2.5-0.5B`) instead of the full HF repo ID
+
+## Acceptance criteria
+
+- [ ] `BrokenPipeError` in tracker and node `_send_json` is silently swallowed
+- [ ] Node IDs are deterministic: `sha256(wallet_address + str(port))[:16]`
+- [ ] `GET /v1/coverage/<model>` accepts both short names and full `owner/repo` IDs
+- [ ] `python -m pytest` passes from repo root

docs/issues/21-route-timeout-flag.md

@@ -0,0 +1,19 @@
+# US-021 — `--route-timeout` CLI flag for node tracker route lookup
+
+Status: done
+Priority: Medium
+Stage: Implemented
+
+## Context
+
+The node's slow-path tracker route lookup (`/v1/route`) used a hard-coded 30-second HTTP timeout.
+On high-latency links (relay, satellite, 5G) or when the tracker is under load, legitimate route
+lookups were failing prematurely. The timeout is deployment-specific and should be tunable.
+
+## Acceptance criteria
+
+- [ ] `meshnet-node start` accepts `--route-timeout <seconds>` (float, default 30.0)
+- [ ] Value is passed through to `TorchNodeServer` and used in the `/v1/route` HTTP call
+- [ ] `TorchNodeServer` exposes `route_timeout` as a readable property
+- [ ] Test: setting `--route-timeout 45` is reflected as `45.0` on the running server object
+- [ ] `python -m pytest` passes

docs/issues/22-start-layer-protocol.md

@@ -0,0 +1,29 @@
+# US-022 — X-Meshnet-Start-Layer: overlapping shard execution protocol
+
+Status: done
+Priority: High
+Stage: Implemented
+
+## Context
+
+Two nodes may register overlapping shard ranges (node A: 0–15, node B: 12–23) to increase
+redundancy or to enable partial-model nodes to form a complete route. Without coordination,
+node B would re-run layers 12–15 on top of activations that already include them, producing
+wrong output.
+
+The `X-Meshnet-Start-Layer` header tells each downstream node which model layer the incoming
+activation tensor represents, so the node skips layers it has already been told to skip.
+
+## Decision
+
+Option A: tracker injects `start_layer` into `X-Meshnet-Route` hops at proxy time. The head
+node passes it per-hop as `X-Meshnet-Start-Layer`. No peer-to-peer negotiation needed.
+
+## Acceptance criteria
+
+- [ ] Tracker `_handle_proxy_chat` builds route hops with `start_layer` = `covered_up_to + 1`
+- [ ] `_handle_binary_forward` reads `X-Meshnet-Start-Layer` and passes it to `backend.forward_bytes`
+- [ ] `_get_remaining_route` parses `start_layer` from injected header and from `/v1/route` slow-path
+- [ ] `TorchModelShard.forward_bytes` accepts optional `start_layer` and skips layers below it
+- [ ] Test: overlapping two-node route produces correct output without double-computing layers
+- [ ] `python -m pytest` passes

docs/issues/23-heartbeat-stats.md

@@ -0,0 +1,26 @@
+# US-023 — Heartbeat stats payload: request counters + dynamic reassignment response
+
+Status: done
+Priority: Medium
+Stage: Implemented
+
+## Context
+
+Node heartbeats are currently empty POSTs. The tracker has no visibility into per-node load,
+making load balancing and assignment decisions blind. Heartbeats should carry cumulative stats.
+The heartbeat response channel is also the natural place for the tracker to deliver reassignment
+instructions without requiring a node restart.
+
+## Acceptance criteria
+
+- [ ] Heartbeat POST body includes: `total_requests`, `failed_requests`, `queue_depth`, `uptime_seconds`, `status`
+- [ ] `TorchNodeServer` tracks the three counters with a `threading.Lock`
+- [ ] Tracker stores the last heartbeat payload per node
+- [ ] Heartbeat response may include `new_assignment: {model, shard_start, shard_end}`; node logs it
+- [ ] Stats survive tracker outage: buffered locally, flushed on next successful heartbeat
+- [ ] `python -m pytest` passes
+
+## Notes
+
+Hot-reload (loading a new shard without restart) is deferred to a future story. The response
+field is wired so trackers can send the signal; nodes log it but don't act yet.

docs/issues/24-availability-map.md

@@ -0,0 +1,24 @@
+# US-024 — Enhanced availability map with per-node health details
+
+Status: done
+Priority: Medium
+Stage: Implemented
+
+## Context
+
+`GET /v1/coverage/<model>` returns band-level coverage (start_layer, end_layer, node_count)
+but gives no visibility into which specific nodes are in each band or whether they are alive.
+Operators debugging a split-model deployment need to see node-level health at a glance.
+
+## Format decision
+
+Both: band metadata + node list grouped under each band. Dead nodes included with `healthy: false`
+so operators can see them.
+
+## Acceptance criteria
+
+- [ ] Each band in the response includes `nodes: [{node_id, endpoint, healthy, queue_depth, last_seen_s}]`
+- [ ] `healthy` is `true` iff last heartbeat < `heartbeat_timeout` seconds ago
+- [ ] Existing band fields (`start_layer`, `end_layer`, `node_count`) preserved
+- [ ] Tests updated to check band fields individually (not exact dict comparison)
+- [ ] `python -m pytest` passes

docs/issues/25-rolling-rpm-stats.md

@@ -0,0 +1,30 @@
+# US-025 — Model usage statistics: rolling RPM windows + SQLite persistence
+
+Status: done
+Priority: Medium
+Stage: Implemented
+
+## Context
+
+Trackers need per-model request-rate data to drive smart shard assignment (US-026) and
+to give operators visibility into what the network is actually serving. Stats must survive
+tracker restarts and should be shareable across a tracker cluster.
+
+## Design
+
+- `_RollingCounter`: circular-bucket counter, epoch-indexed, stale buckets auto-reset
+- Three windows: 60×1-min buckets (last hour), 24×1-hr (last day), 30×1-day (last month)
+- `_StatsCollector`: `record_request()`, `get_local_rpms()`, `merge_peer_rpms()`, `get_combined_stats()`
+- SQLite persistence via `--stats-db PATH`
+- Gossip: each tracker keeps its own slice; merge is additive (not averaged)
+
+## Acceptance criteria
+
+- [ ] `_RollingCounter` passes unit tests (record, rpm, stale bucket reset)
+- [ ] `_StatsCollector` accumulates and merges peer slices additively
+- [ ] SQLite round-trip: buckets saved and restored across restart
+- [ ] `GET /v1/stats` returns combined stats JSON
+- [ ] `POST /v1/stats/gossip` accepts peer slice and merges
+- [ ] `_handle_proxy_chat` calls `stats.record_request()` after model is resolved
+- [ ] `--stats-db PATH` CLI flag added to tracker
+- [ ] 6 unit tests pass; `python -m pytest` passes

docs/issues/26-smart-model-assignment.md

@@ -0,0 +1,32 @@
+# US-026 — Smart model assignment via demand×coverage scoring
+
+Status: done
+Priority: Medium
+Stage: Implemented
+
+## Context
+
+`/v1/network/assign` currently picks the model with the largest uncovered shard gap,
+ignoring traffic. A model serving 1000 RPM at 60% coverage is far more valuable to fill
+than a zero-traffic model at 50% coverage.
+
+## Scoring formula
+
+```
+score = (demand_rpm + 1.0) × (coverage_deficit + 0.01)
+```
+
+- `demand_rpm`: combined RPM from `_StatsCollector.get_combined_stats()`
+- `coverage_deficit`: fraction of model layers with zero node coverage, in [0.0, 1.0]
+- `+1.0` floor: models with no traffic still compete by coverage
+- `+0.01` floor: fully-covered models still have a non-zero score if they have traffic
+
+`price_per_token: 0.0` reserved in the response for future billing integration.
+
+## Acceptance criteria
+
+- [ ] `_handle_network_assign` computes score per model and returns the highest
+- [ ] Demand uses combined stats (local + peer slices)
+- [ ] `price_per_token: 0.0` present in response
+- [ ] Test: high-demand low-coverage model beats low-demand high-coverage model
+- [ ] `python -m pytest` passes

docs/issues/27-throughput-routing.md

@@ -0,0 +1,28 @@
+# US-027 — Throughput-optimized routing: effective throughput as tiebreak
+
+Status: done
+Priority: Medium
+Stage: Implemented
+
+## Context
+
+The greedy max-reach route selection picks nodes by shard coverage but ignores node speed.
+When two nodes cover the same remaining layer range, we should prefer the faster one.
+This is a tiebreak only — coverage maximization remains the primary objective.
+
+## Effective throughput formula
+
+```
+effective_throughput = benchmark_tokens_per_sec / (queue_depth + 1)
+```
+
+`benchmark_tokens_per_sec` comes from the hardware profile at registration time.
+`queue_depth` comes from the last heartbeat.
+
+## Acceptance criteria
+
+- [ ] `_effective_throughput(node)` helper in `server.py`
+- [ ] `_select_route` uses throughput as tiebreak when `shard_end` is equal
+- [ ] Test: two nodes, same shard range, different throughput → faster node selected
+- [ ] Existing coverage tests still pass unchanged
+- [ ] `python -m pytest` passes

docs/issues/28-routing-correctness-tests.md

@@ -0,0 +1,26 @@
+# US-028 — Routing correctness tests: three-node, overlap, and throughput scenarios
+
+Status: done
+Priority: Medium
+Stage: Implemented
+
+## Context
+
+Route selection logic (`_select_route`) is the core of the inference network. Without
+locked-down tests, routing regressions are silent. This story adds a comprehensive
+scenario suite that locks in the routing contract.
+
+## Test scenarios
+
+1. **No-overlap three nodes**: greedy picks in layer-start order (A→C→B for ranges 0–7, 8–15, 16–23)
+   Note: the algorithm picks by earliest uncovered layer, not by node label — so if C.shard_start < B.shard_start, C comes first.
+2. **Overlapping shards**: correct resolution without double-computing layers
+3. **Throughput tiebreak**: faster node wins when shard_end is equal
+4. **Gap detection**: partial coverage returns a 503 error with description
+
+## Acceptance criteria
+
+- [ ] 7 routing tests covering the above scenarios
+- [ ] Tests use in-process tracker (no mocking)
+- [ ] `_make_node` helper for concise test setup
+- [ ] All tests pass; `python -m pytest` passes

docs/issues/29-relay-outbound-client.md

@@ -0,0 +1,44 @@
+# US-029 — Outbound relay client: NAT/internet pipeline hops
+
+Status: done
+Priority: Critical
+Stage: Implemented
+
+## Context
+
+Nodes behind NAT (WSL2 with 172.x.x.x addresses, 5G mobile, home routers) register with the
+tracker and include a `relay_addr` (`wss://relay/rpc/{peer_id}`). When the head node needs to
+forward activations to such a peer, it currently fails because the direct HTTP endpoint is
+unreachable.
+
+The relay server (US-017) is already running and the node already opens a persistent outbound
+WebSocket (`RelayHttpBridge`). What is missing is the *outbound caller side*: given a `relay_addr`,
+open a per-hop WebSocket to the relay's `/rpc/{peer_id}` endpoint and send the activation through it.
+
+## Protocol
+
+```
+Node A  →  WS connect wss://relay/rpc/{peer_id_B}
+       →  send JSON: {request_id, method, path, headers, body_base64}
+Relay   →  forward as relay-http-request envelope to Node B's persistent WS
+Node B  →  process /forward locally
+       →  send relay-http-response envelope back
+Relay   →  resolve future, send response JSON to Node A
+Node A  ←  {request_id, status, headers, body_base64}
+```
+
+Binary activations (bfloat16) are base64-encoded. No precision loss.
+
+## Acceptance criteria
+
+- [ ] `_relay_hop(relay_addr, path, body, headers, timeout)` in `torch_server.py` — opens WS, sends, receives, returns `(status, headers_lower, body_bytes)`
+- [ ] `_get_remaining_route` returns `list[dict]` with `relay_addr` field (was `list[tuple]`)
+- [ ] `_run_downstream_pipeline` dispatches via `_relay_hop` when hop has `relay_addr`; falls back to direct HTTP if relay connection fails
+- [ ] Tracker `_handle_proxy_chat` includes `relay_addr` in downstream hop dicts when node has one
+- [ ] `relay_bridge._handle_request` decodes `body_base64`; response uses `body_base64` for `octet-stream` content
+- [ ] All 157 tests pass (`python -m pytest`)
+- [ ] QUICKSTART.md updated with relay NAT/internet architecture and test scenario
+
+## WSL2 test scenario
+
+Start two nodes in WSL2 pointing at the public tracker. Both get `172.x.x.x` endpoints (unreachable from outside). Both connect to relay automatically. Send an inference request through the tracker — activations flow via relay. No `--advertise-host` needed.

docs/issues/30-manual-route-and-hop-benchmark.md

@@ -0,0 +1,48 @@
+# US-020 — Manual route selection + hop-penalty benchmarking
+
+## Context
+
+The tracker auto-selects inference routes based on synthetic benchmark scores. To measure
+the real cost of adding hops (latency per node boundary), we need:
+
+1. A way to pin a request to a specific route so we control the variable.
+2. A benchmark endpoint that runs the same prompt through 1-node, 2-node, and 3-node
+   routes and records per-hop latency.
+
+Results are stored to disk. Routing algorithm is **not** changed in this story — this is
+data collection only. The data will inform a future routing optimisation story.
+
+## Design decisions (grilled 2026-07-01)
+
+| Decision | Choice |
+|---|---|
+| Route spec | Optional `route` field in JSON request body (list of node IDs) |
+| Trigger | Explicit only — `POST /v1/benchmark/hop-penalty` endpoint |
+| Auth | Header-presence stub (`Authorization` must be non-empty); real auth in future story |
+| Routing integration | Store data only; routing algorithm unchanged |
+| Persistence | Append to `benchmark_results.json` in tracker working dir; in-memory queryable |
+
+## Acceptance criteria
+
+- `POST /v1/chat/completions` accepts optional `"route": ["<node_id>", ...]` in the
+  request body. If present, the tracker uses those nodes in order instead of auto-selecting.
+  If absent, existing routing is unchanged (no breaking change for unaware clients).
+- Missing or invalid node IDs in `route` return HTTP 400 with a descriptive error.
+- `POST /v1/benchmark/hop-penalty` is auth-gated: requests without a non-empty
+  `Authorization` header return HTTP 401. Body: `{"model": "...", "prompt": "...",
+  "max_new_tokens": 64}`.
+- Benchmark fans out to up to three routes: 1-node (single node covering all layers),
+  2-node (two consecutive shard nodes), 3-node (three nodes) — using whatever is
+  currently registered. Routes with insufficient coverage are skipped, not errored.
+- Response includes per-route breakdown: `total_ms`, `per_hop_ms: [...]`,
+  `tokens_generated`, `route: [node_id, ...]`.
+- Results are appended to `<tracker_working_dir>/benchmark_results.json` (created if
+  absent) as a JSON array. Each entry includes timestamp, model, prompt hash, and the
+  per-route breakdown.
+- `GET /v1/benchmark/results` returns the stored results array. Also auth-gated.
+- Clients that never send `route` or call `/v1/benchmark/*` are completely unaffected.
+- Integration test: send the same prompt via a pinned 1-node route and a pinned 2-node
+  route; assert 2-node result has 2 entries in `per_hop_ms`; assert both records appear
+  in `benchmark_results.json`.
+- `python -m pytest` passes from repo root.
+- Commit only this story's changes.

docs/issues/31-billing-ledger.md

@@ -0,0 +1,25 @@
+Status: done
+
+# 31 — Billing ledger: per-token pricing, 90/10 split, pending balances
+
+## What to build
+
+Tracker-side off-chain billing per ADR-0015. Each model preset gets a USDT `price_per_1k_tokens` in tracker config. When a request completes, the tracker debits the client's API-key ledger balance (`price × total_tokens / 1000`) and credits 90% to the serving nodes' pending balances proportional to work units (layers × tokens, reusing the existing `ComputeAttribution`), with 10% accruing to the protocol cut. API keys with insufficient balance are rejected with HTTP 402 before any routing happens — no free work.
+
+The ledger persists in the existing tracker SQLite store and replicates across the tracker hive so any follower can serve balance reads.
+
+This story is pure off-chain accounting — no Solana calls.
+
+## Acceptance criteria
+
+- [ ] Per-model `price_per_1k_tokens` in tracker config with sane defaults
+- [ ] Completed request debits client ledger: `price × total_tokens / 1000`
+- [ ] 90% split across serving nodes by work_units; 10% accrues to `protocol_cut`
+- [ ] Insufficient balance → HTTP 402 before routing
+- [ ] Balances survive tracker restart (SQLite) and replicate to hive followers
+- [ ] Unit tests: single-node route, 3-node split, exhausted balance, restart persistence
+
+## Blocked by
+
+- `23-heartbeat-stats.md`
+- `25-rolling-rpm-stats.md`

docs/issues/32-devnet-treasury-deposits.md

@@ -0,0 +1,28 @@
+Status: done
+
+# 32 — Devnet custodial treasury: mock-USDT mint + deposit watcher
+
+## What to build
+
+The real Solana adapter behind the `packages/contracts` boundary, custodial model per ADR-0015. All on-chain surface is plain SPL transfers — no Anchor programs.
+
+**Setup script** (`scripts/devnet_setup.py`): creates the mock-USDT SPL mint (6 decimals, matching real USDT) and the treasury token account on devnet, airdropping SOL for fees, and prints the `.env.devnet` values (mint address, RPC URL, treasury keypair path).
+
+**Wallet binding**: `POST /v1/wallet/register` binds a client wallet pubkey to an API key.
+
+**Deposit watcher**: polls the treasury token account for confirmed incoming USDT transfers and credits the sending wallet's bound API-key ledger balance. Transaction signatures are deduplicated so replayed/re-observed transfers credit exactly once.
+
+Note: this supersedes the "testnet, never devnet" note in issue 06 (see ADR-0015) — devnet is the ecosystem standard for app development and real USDT exists on neither cluster.
+
+## Acceptance criteria
+
+- [ ] `scripts/devnet_setup.py` creates mint + treasury ATA and prints `.env.devnet` values
+- [ ] `POST /v1/wallet/register` binds client wallet pubkey to API key
+- [ ] Deposit watcher credits ledger within one poll interval of a confirmed transfer
+- [ ] Duplicate/replayed transactions credit exactly once (signature dedupe)
+- [ ] Local `solana-test-validator` integration test covers mint → deposit → credit
+- [ ] Deterministic local adapter still works for CI without any validator
+
+## Blocked by
+
+- `31-billing-ledger.md`

docs/issues/33-settlement-loop.md

@@ -0,0 +1,27 @@
+Status: done
+
+# 33 — Settlement loop: leader-only batched USDT payouts
+
+## What to build
+
+The on-chain settlement loop per ADR-0015, following the mining-pool standard: pay a node when `pending ≥ payout_threshold` OR `time_since_last_payout ≥ max_period`, whichever fires first, with a dust floor so no payout is smaller than it is worth.
+
+Only the current Raft leader runs the loop; followers replicate the ledger but never sign. The treasury keypair is loaded only on settlement-capable trackers. Payouts are batched SPL transfers treasury → node wallets on devnet.
+
+Config (all dynamic, so the period can grow with volume): dev defaults `period=60s, threshold≈0` so every run is verifiable on-chain; prod defaults `period=24h, threshold=a few USDT`.
+
+Settlement history (settlement id, tx signature, node wallet, amount, timestamp) persists in SQLite, replicates across the hive, and is queryable over HTTP. Retries are idempotent by settlement id — a failed or timed-out transaction never double-pays.
+
+## Acceptance criteria
+
+- [ ] Only the Raft leader settles; followers never sign (asserted in a 3-tracker test)
+- [ ] Trigger: `pending ≥ threshold` OR `elapsed ≥ max_period`; dust floor respected
+- [ ] Batched SPL transfers land on devnet; pending balances zeroed atomically with recorded tx signature
+- [ ] Failed/timeout transactions retry without double-pay (idempotent by settlement id)
+- [ ] Settlement history queryable via tracker HTTP endpoint
+- [ ] End-to-end devnet test: fund client → run inference → node wallet USDT balance increases
+
+## Blocked by
+
+- `19-binary-data-plane-and-peer-weight-transfer.md`
+- `32-devnet-treasury-deposits.md`

docs/issues/34-forfeiture-penalty.md

@@ -0,0 +1,28 @@
+Status: done
+
+# 34 — Hardened proof-of-work: pending-balance forfeiture penalty
+
+## What to build
+
+Wire the validator's ~5% sampling (issue 07) to the new penalty per ADR-0015. On confirmed output divergence:
+
+1. The node's **entire pending balance is forfeited** to the protocol cut, in the same ledger transaction as the strike (no window where a payout can race the penalty).
+2. A **strike** is recorded; the third strike **bans** the wallet — registration rejected, excluded from all routes, and any unpaid pending balance is never paid out.
+
+The probationary period (first N jobs unpaid, default 50) is retained as the anti-sybil re-entry cost, and the node CLI shows remaining probation jobs.
+
+Why this deters cheating: settlement is periodic, so the pending balance itself is the collateral — no upfront stake deposit, zero onboarding friction. At a 5% check rate a cheater is caught once per ~20 fraudulent jobs, so the penalty must exceed ~20× the per-job gain; with daily settlement, pending ≈ a full day's earnings, well above that bar. Document this math in the validator README.
+
+## Acceptance criteria
+
+- [ ] Divergence → pending balance forfeited to `protocol_cut` atomically with the strike
+- [ ] 3rd strike bans wallet: registration rejected, excluded from all routes
+- [ ] Banned wallet's unpaid pending balance is not paid at next settlement
+- [ ] Probation: first N jobs (default 50) accrue no pending balance; node CLI shows remaining
+- [ ] Integration test: deliberately-bad node loses pending, accrues strikes, banned within 60 requests
+- [ ] Forfeiture events visible in tracker logs and settlement history
+
+## Blocked by
+
+- `07-fraud-detection-slash.md`
+- `31-billing-ledger.md`

docs/issues/35-tracker-web-dashboard.md

@@ -0,0 +1,34 @@
+Status: done
+
+# 35 — Tracker web dashboard
+
+## What to build
+
+A read-only web dashboard served by the tracker itself at `GET /dashboard` — single page, plain HTML/JS polling the tracker's HTTP endpoints, static assets embedded in the tracker package. No new build toolchain.
+
+Because the tracker hive replicates ledger and registry state, **any tracker in the mesh — leader or follower — can serve the dashboard** from its own state.
+
+Panels:
+
+- Hive membership and current Raft leader
+- Node registry: health, scores, coverage map per model
+- Client ledger balances
+- Node pending balances and next-payout ETA
+- Settlement history with devnet explorer links (tx signatures)
+- Strikes / bans / forfeiture events
+- Rolling RPM stats per model
+
+Write operations (editing prices, manual settlement trigger) are deferred to a later story.
+
+## Acceptance criteria
+
+- [ ] `GET /dashboard` serves the UI from any tracker (leader or follower)
+- [ ] All seven panels render with live data
+- [ ] Auto-refresh ≤5s without page reload
+- [ ] No new build toolchain — static assets embedded in the tracker package
+- [ ] Works against a 3-tracker hive in the two-machine LAN test setup
+
+## Blocked by
+
+- `31-billing-ledger.md`
+- `33-settlement-loop.md`

docs/prd.json

@@ -0,0 +1,830 @@
+{
+  "name": "Distributed Inference Network",
+  "description": "Build a distributed inference network with node, gateway, tracker, SDK, contracts, and P2P shard distribution components from the grill session PRD.",
+  "branchName": "ralph/distributed-inference-network",
+  "userStories": [
+    {
+      "id": "US-001",
+      "title": "01 — Monorepo scaffold + single-node smoke test",
+      "description": "Stand up the monorepo package layout and prove that a client HTTP request can travel end-to-end through the stack — gateway → one node serving all layers → valid response — before any real model or distributed logic is wired in. The six top-level packages should exist with minimal stubs: - `packages/node` — node client CLI (`meshnet-node`) - `packages/gateway` — HTTP gateway + route orchestration - `packages/tracker` — node registry and route selection - `packages/sdk` — `meshnet` Python SDK - `packages/contracts` — Solana smart contract wrappers - `packages/p2p` — gossip and shard swarm The smoke test uses a tiny in-process stub model (not a real LLM) so the test is fast and has no external dependencies. The node runs in the same process as the test. The gateway is a real HTTP server. The client sends a real `POST /v1/chat/completions` request and receives a valid OpenAI-format response.",
+      "acceptanceCriteria": [
+        "`pip install -e packages/node packages/gateway packages/tracker` works from repo root",
+        "`meshnet-node`, `meshnet-gateway`, and `meshnet-tracker` CLI entry points exist",
+        "A single integration test starts a gateway and one stub node in-process, sends a `POST /v1/chat/completions` request, and asserts a valid OpenAI-format response is returned",
+        "The test passes with `pytest` from repo root with no external services running",
+        "All six package directories exist with `pyproject.toml` and an importable top-level module",
+        "Follow TDD where code is added: add/adjust an observable behavior test before implementation when practical",
+        "Run the task-specific tests and ensure they pass",
+        "Run `python -m pytest` from repo root and ensure it passes, or document why no test suite exists yet",
+        "Keep the implementation scoped to this story; do not implement downstream stories early",
+        "Commit only this story's code changes with a focused conventional commit message"
+      ],
+      "priority": 1,
+      "passes": true,
+      "notes": "Source issue: .scratch/distributed-inference-network/issues/01-monorepo-scaffold.md",
+      "dependsOn": [],
+      "completionNotes": "Completed by Ralph iteration 7b260695; verified by pytest and editable package installs.",
+      "status": "done"
+    },
+    {
+      "id": "US-002",
+      "title": "02 — Two-node shard pipeline",
+      "description": "Extend the single-node smoke test so that inference is routed through exactly two nodes, each serving half of a stub model's layers. Activation tensors must travel from the gateway to node A, then from node A to node B, and the final output returns to the gateway. This proves the distributed shard pipeline works end-to-end before real models or a real tracker are introduced. The two nodes run in the same process as the test (no real network required). The gateway hardcodes the two-node route for now — dynamic route selection comes in issue 03. The activation tensor protocol (shape, dtype, serialisation format) must be established here, as all later issues depend on it. Use the domain vocabulary from `CONTEXT.md`: the ordered list of nodes is an **inference route**; each node's layer range is a **shard**; the tensors passed between nodes are activations (not \"data\" or \"chunks\").",
+      "acceptanceCriteria": [
+        "The integration test spins up two stub nodes, each configured with a non-overlapping shard range",
+        "A `POST /v1/chat/completions` request results in activation tensors flowing node-A → node-B (verifiable via test assertions or logs)",
+        "The gateway assembles the final response correctly from node-B's output",
+        "The test passes with no external services running",
+        "The activation tensor serialisation format is documented in a short inline comment (shape, dtype, wire format) — this becomes the contract all future nodes implement",
+        "Follow TDD where code is added: add/adjust an observable behavior test before implementation when practical",
+        "Run the task-specific tests and ensure they pass",
+        "Run `python -m pytest` from repo root and ensure it passes, or document why no test suite exists yet",
+        "Keep the implementation scoped to this story; do not implement downstream stories early",
+        "Commit only this story's code changes with a focused conventional commit message"
+      ],
+      "priority": 2,
+      "passes": true,
+      "notes": "Source issue: .scratch/distributed-inference-network/issues/02-two-node-shard-pipeline.md",
+      "dependsOn": [
+        "US-001"
+      ],
+      "completionNotes": "Tests pass; activation tensor flow via stub nodes verified. New HF-model path tested in test_node_startup.py.",
+      "status": "done",
+      "status_reason": "Base64 JSON wire format established here was replaced by binary HTTP protocol in US-011. tests/test_two_node_pipeline.py needs verification it exercises the new binary format end-to-end."
+    },
+    {
+      "id": "US-003",
+      "title": "03 — Tracker: node registration + route selection",
+      "description": "Replace the hardcoded two-node route from issue 02 with a real tracker. Nodes register themselves with the tracker on startup (endpoint, shard range, hardware profile, node score). The gateway queries the tracker for an inference route for a given model preset instead of using a static list. The tracker returns an ordered list of node endpoints whose shards collectively cover all layers. The tracker runs as a lightweight HTTP service. Node score is initially a simple placeholder (e.g. fixed value or random) — real throughput/latency scoring comes later. The tracker must correctly reject route requests when no registered nodes cover a required shard range and return an appropriate error. This issue establishes the tracker's registration and routing API contract, which issues 04 (node startup) and 05 (OpenAI gateway) both depend on.",
+      "acceptanceCriteria": [
+        "A node can register with the tracker via HTTP, providing its endpoint, shard range, and a hardware profile",
+        "The gateway queries the tracker with a model preset name and receives an ordered list of node endpoints forming a complete inference route",
+        "The end-to-end integration test from issue 02 still passes with the tracker now in the loop (no hardcoded routes)",
+        "The tracker returns a clear error response when no route is available for a requested model preset",
+        "Nodes that fail to heartbeat within a configurable window are removed from the registry",
+        "The tracker's registration and route-selection HTTP API is defined (paths, request/response shapes) in the tracker package",
+        "Follow TDD where code is added: add/adjust an observable behavior test before implementation when practical",
+        "Run the task-specific tests and ensure they pass",
+        "Run `python -m pytest` from repo root and ensure it passes, or document why no test suite exists yet",
+        "Keep the implementation scoped to this story; do not implement downstream stories early",
+        "Commit only this story's code changes with a focused conventional commit message"
+      ],
+      "priority": 3,
+      "passes": true,
+      "notes": "Source issue: .scratch/distributed-inference-network/issues/03-tracker-registration-and-routing.md",
+      "dependsOn": [
+        "US-002"
+      ],
+      "completionNotes": "Completed by Ralph iteration 79796dd2; verified by pytest, compileall, editable installs, CLI help, and spec/code reviews.",
+      "status": "done"
+    },
+    {
+      "id": "US-004",
+      "title": "04 — Node client startup flow (`meshnet-node start`)",
+      "description": "Make `meshnet-node start` self-configuring from scratch on a machine with a CUDA-capable GPU (or CPU fallback). The full startup sequence must complete without any manual configuration: 1. Detect GPU model and available VRAM 2. Load an existing Solana wallet from disk, or generate and save a new one 3. Query the tracker for the optimal shard assignment given the hardware profile 4. Download the assigned shard from HuggingFace (`huggingface_hub`) 5. Register with the tracker (wallet address, endpoint, shard range, hardware profile) 6. Begin accepting inference connections The node prints a short status summary on startup: wallet address, assigned shard, model preset, and download progress. No interactive prompts — the entire flow is non-interactive. This is the primary viral growth vector. Every second of friction in this flow costs node operators. The startup sequence must work on Linux with CUDA, and degrade gracefully to CPU on machines without a GPU.",
+      "acceptanceCriteria": [
+        "`meshnet-node start --tracker http://localhost:8080` completes startup without prompts on a machine with a CUDA GPU",
+        "A Solana wallet keypair is generated and saved to `~/.config/meshnet/wallet.json` if none exists",
+        "The assigned shard is downloaded from HuggingFace and cached to `~/.cache/meshnet/shards/`",
+        "The node registers with the tracker and appears in the tracker's node registry",
+        "The node accepts a live inference connection and processes activation tensors correctly after startup",
+        "On a CPU-only machine, the node starts with a warning and serves a CPU-appropriate shard",
+        "An integration test covers the full startup sequence against a local tracker stub",
+        "Follow TDD where code is added: add/adjust an observable behavior test before implementation when practical",
+        "Run the task-specific tests and ensure they pass",
+        "Run `python -m pytest` from repo root and ensure it passes, or document why no test suite exists yet",
+        "Keep the implementation scoped to this story; do not implement downstream stories early",
+        "Commit only this story's code changes with a focused conventional commit message"
+      ],
+      "priority": 4,
+      "passes": true,
+      "notes": "Source issue: .scratch/distributed-inference-network/issues/04-node-client-startup.md",
+      "dependsOn": [
+        "US-003"
+      ],
+      "completionNotes": "Completed by Ralph iteration 86510a10; verified by pytest, compileall, editable installs, CLI help, and final review.",
+      "status": "done"
+    },
+    {
+      "id": "US-005",
+      "title": "05 — OpenAI-compatible gateway",
+      "description": "Expose a production-shape HTTP API from the gateway so that any client using the OpenAI Python SDK (or any OpenAI-compatible tool) works by changing only the `base_url`. The gateway translates OpenAI-format requests into inference route execution and streams responses back in OpenAI-format. Endpoints to implement: - `POST /v1/chat/completions` — streaming (`text/event-stream`) and non-streaming - `GET /v1/models` — returns the list of model presets currently routable on the network - `GET /v1/health` — liveness check The gateway selects an inference route from the tracker, executes the shard pipeline, and assembles the streamed response. If the tracker returns no route for the requested model, the gateway responds with a standard OpenAI-format error (`model_not_available`). Authentication (API key → SOL/USDC balance) is a stub in this issue — return 200 for any non-empty `Authorization` header. Real payment gating comes in issue 06.",
+      "acceptanceCriteria": [
+        "`openai.OpenAI(base_url=\"http://localhost:8080/v1\", api_key=\"test\").chat.completions.create(model=\"stub-model\", messages=[...])` returns a valid response",
+        "Streaming works: `stream=True` returns `text/event-stream` chunks in OpenAI SSE format",
+        "`GET /v1/models` returns a JSON array of available model preset names",
+        "A request for an unavailable model returns an OpenAI-format error response with HTTP 503",
+        "LangChain `ChatOpenAI(base_url=..., api_key=...)` works against the gateway",
+        "An integration test covers streaming and non-streaming paths end-to-end through a real tracker and two stub nodes",
+        "Follow TDD where code is added: add/adjust an observable behavior test before implementation when practical",
+        "Run the task-specific tests and ensure they pass",
+        "Run `python -m pytest` from repo root and ensure it passes, or document why no test suite exists yet",
+        "Keep the implementation scoped to this story; do not implement downstream stories early",
+        "Commit only this story's code changes with a focused conventional commit message"
+      ],
+      "priority": 5,
+      "passes": true,
+      "notes": "Source issue: .scratch/distributed-inference-network/issues/05-openai-compatible-gateway.md",
+      "dependsOn": [
+        "US-003"
+      ],
+      "completionNotes": "All 6 gateway tests pass including OpenAI SDK, LangChain, streaming, and 503 for unknown model.",
+      "status": "done",
+      "status_reason": "Gateway is currently the pipeline orchestrator. US-014 moves orchestration to tracker-nodes; gateway becomes a thin load-balancer proxy. Implementation will be superseded — defer rework until US-014 lands."
+    },
+    {
+      "id": "US-006",
+      "title": "06 — Solana stake + settlement contracts",
+      "description": "Deploy and integrate the Solana smart contracts that make node staking, client payment, and token reward settlement trustless. All development and testing targets **Solana testnet** — never devnet or mainnet during development, to avoid real costs.",
+      "acceptanceCriteria": [
+        "All contracts deploy successfully to Solana testnet",
+        "A node can submit a stake transaction and have its balance reflected in the registry contract",
+        "A client can fund an API key account with testnet SOL",
+        "After a completed inference session, compute attribution is recorded on-chain with correct node/layer attribution",
+        "The epoch settlement transaction correctly distributes token rewards to node operators and deducts client balances",
+        "The gateway refuses to route to a node whose stake balance is below the minimum threshold",
+        "All contract interactions in tests run against a local Solana test validator (via `solana-test-validator`) — no live testnet required for CI",
+        "A `.env.testnet` config points to Solana testnet RPC for manual end-to-end testing",
+        "python -m pytest passes from repo root",
+        "Commit only this story's changes"
+      ],
+      "priority": 6,
+      "passes": true,
+      "notes": "Source issue: .scratch/distributed-inference-network/issues/06-solana-stake-and-settlement.md",
+      "dependsOn": [
+        "US-003"
+      ],
+      "completionNotes": "Completed by fresh Ralph/Codex session c257ffde with controller patches for clarified economics; verified by pytest, compileall, and diff check.",
+      "status": "done"
+    },
+    {
+      "id": "US-007",
+      "title": "07 — Fraud detection: validator + on-chain slash",
+      "description": "Implement the optimistic fraud detection loop. After each inference request completes, the validator process independently decides whether to re-run the request on a reference node (~5% sample rate).",
+      "acceptanceCriteria": [
+        "The validator process samples ~5% of completed inference requests (configurable)",
+        "A node returning a deliberately wrong output is detected and slashed within one validation cycle",
+        "The on-chain stake balance of the slashed node decreases by the correct slash amount",
+        "The strike count for the slashed node increments on-chain",
+        "A node that reaches the strike threshold is excluded from route selection on the next gateway request",
+        "A slashed node logs a clear warning to stdout",
+        "An integration test: run a deliberately-bad node, send 20 requests, assert at least one slash transaction is submitted and the node is eventually excluded from routes",
+        "python -m pytest passes from repo root",
+        "Commit only this story's changes"
+      ],
+      "priority": 7,
+      "passes": true,
+      "notes": "Source issue: .scratch/distributed-inference-network/issues/07-fraud-detection-slash.md",
+      "dependsOn": [
+        "US-005",
+        "US-006"
+      ],
+      "completionNotes": "Completed by fresh Ralph/Codex session 04475912 with controller fix for duplicate slash suppression; verified by pytest, compileall, and diff check.",
+      "status": "done"
+    },
+    {
+      "id": "US-008",
+      "title": "08 — Node probationary period + ban enforcement",
+      "description": "Implement the two anti-sybil mechanisms that make re-entering the network after a ban economically costly.",
+      "acceptanceCriteria": [
+        "A node with a new wallet receives no token rewards for its first N jobs (verified via settlement contract state)",
+        "The node client prints remaining probationary jobs on startup",
+        "After N jobs, the next epoch settlement correctly credits the node with token rewards",
+        "A wallet with strike count at the threshold is marked banned in the registry contract",
+        "The tracker excludes banned wallets from route selection",
+        "A banned wallet that attempts to register with the tracker is rejected",
+        "An integration test covers: new wallet → N jobs → earning begins; and: strike threshold reached → banned → excluded from routes",
+        "python -m pytest passes from repo root",
+        "Commit only this story's changes"
+      ],
+      "priority": 8,
+      "passes": true,
+      "notes": "Source issue: .scratch/distributed-inference-network/issues/08-probationary-period-and-bans.md",
+      "dependsOn": [
+        "US-007"
+      ],
+      "completionNotes": "Completed by fresh Ralph/Codex session db3f5c10 with controller test fix for banned registration semantics; verified by pytest, compileall, and diff check.",
+      "status": "done"
+    },
+    {
+      "id": "US-009",
+      "title": "09 — P2P shard swarm",
+      "description": "Once a node has downloaded a shard, it seeds that shard to other nodes that are assigned the same shard.",
+      "acceptanceCriteria": [
+        "A node that has downloaded a shard is listed as a peer for that shard in the tracker",
+        "A second node assigned the same shard downloads it from the first node (peer) rather than HuggingFace",
+        "If no peers are available, the node falls back to HuggingFace without error",
+        "The node client logs whether the shard was downloaded from a peer or HuggingFace",
+        "Shard chunks are verified against a checksum before being marked complete",
+        "An integration test: node A downloads shard from HuggingFace stub; node B with same assignment downloads from node A",
+        "python -m pytest passes from repo root",
+        "Commit only this story's changes"
+      ],
+      "priority": 9,
+      "passes": true,
+      "notes": "Source issue: .scratch/distributed-inference-network/issues/09-p2p-shard-swarm.md",
+      "dependsOn": [
+        "US-004"
+      ],
+      "completionNotes": "Completed by fresh Ralph/Codex session 243fae88 with controller fix for streaming tar archives; verified by pytest, compileall, and diff check.",
+      "status": "done"
+    },
+    {
+      "id": "US-010",
+      "title": "10 — `meshnet` Python SDK",
+      "description": "A Python SDK (`packages/sdk`) that wraps the OpenAI-compatible gateway and exposes network-specific controls.",
+      "acceptanceCriteria": [
+        "`pip install meshnet` installs the SDK",
+        "`Client.chat.completions.create(...)` works identically to the OpenAI SDK",
+        "`client.wallet.balance()` returns the current SOL/USDC balance for the API key",
+        "`client.wallet.top_up()` returns a valid Solana payment address",
+        "`client.models.available()` returns model presets with shard coverage percentage",
+        "`client.estimate_cost(model, tokens)` returns a cost estimate in SOL",
+        "`client.request(redundancy=2)` sends to two independent inference routes and returns majority response",
+        "The SDK is typed (py.typed, full type stubs)",
+        "An integration test covers each SDK method against a local gateway + tracker + stub nodes",
+        "python -m pytest passes from repo root",
+        "Commit only this story's changes"
+      ],
+      "priority": 10,
+      "passes": true,
+      "notes": "Source issue: .scratch/distributed-inference-network/issues/10-meshnet-sdk.md",
+      "dependsOn": [
+        "US-005",
+        "US-006"
+      ],
+      "completionNotes": "Completed by fresh Ralph/Codex session ef4eea0e; verified by pytest, compileall, editable SDK install, and diff check.",
+      "status": "done"
+    },
+    {
+      "id": "US-011",
+      "title": "11 — Binary activation wire format",
+      "description": "Replace the base64 JSON activation payload with raw binary HTTP bodies, zstd compression, and chunked prefill (128 tokens/chunk). All nodes and the gateway must be migrated. Stub nodes continue to emit zeroed tensors, just in binary. This is a protocol prerequisite for US-012 (real model backend).",
+      "acceptanceCriteria": [
+        "Node /forward endpoint reads shape/dtype/session/chunk from HTTP headers; body is raw binary (optionally zstd-compressed)",
+        "Node /forward response is raw binary with the same header set",
+        "Gateway splits prompts > MESHNET_CHUNK_TOKENS (default 128) into sequential chunks sent through the pipeline",
+        "Integration test: 512-token stub activation (4 chunks) through a two-node pipeline returns 4 valid binary chunk responses",
+        "zstd Python package added as a dependency to packages/node and packages/gateway",
+        "_make_stub_activations replaced with _make_stub_binary_activation(shape, dtype) -> bytes",
+        "python -m pytest passes from repo root",
+        "Commit only this story's changes"
+      ],
+      "priority": 11,
+      "passes": true,
+      "notes": "Source issue: .scratch/distributed-inference-network/issues/11-binary-wire-format.md",
+      "dependsOn": [
+        "US-002"
+      ],
+      "completionNotes": "Completed by fresh Ralph/Codex session 3f3bed75; verified by pytest, compileall, and diff check.",
+      "status": "done"
+    },
+    {
+      "id": "US-012",
+      "title": "12 — Real PyTorch model backend",
+      "description": "Replace stub node inference with actual transformers layer execution. Node loads HuggingFace SafeTensors model shard (model.model.layers[start:end]), runs real forward passes in bfloat16, handles head (embed_tokens) and tail (lm_head) responsibilities, and supports bitsandbytes NF4/INT8/bfloat16 quantization via --quantization flag. Test model: openai-community/gpt2.",
+      "acceptanceCriteria": [
+        "meshnet-node start --model-id openai-community/gpt2 --shard-start 0 --shard-end 6 loads the shard without error",
+        "Head node (shard_start==0) loads tokenizer and embed_tokens",
+        "Tail node (shard_end==total_layers) loads model.norm and lm_head",
+        "Two-node GPT-2 integration test returns deterministic coherent text completion",
+        "--quantization [bfloat16|int8|nf4] flag supported; bfloat16 default",
+        "Node with insufficient VRAM prints clear error and exits",
+        "transformers, bitsandbytes, safetensors, accelerate added as node dependencies",
+        "Integration tests marked @pytest.mark.integration, skipped in CI without GPU",
+        "python -m pytest passes from repo root",
+        "Commit only this story's changes"
+      ],
+      "priority": 12,
+      "passes": true,
+      "notes": "Source issue: .scratch/distributed-inference-network/issues/12-real-pytorch-model-backend.md",
+      "dependsOn": [
+        "US-011"
+      ],
+      "completionNotes": "Completed by agent",
+      "status": "done"
+    },
+    {
+      "id": "US-013",
+      "title": "13 — Coverage-first tracker shard assignment",
+      "description": "Upgrade tracker route selection to coverage-first, speed-weighted bin-packing. Tracker maintains a live coverage map per model, issues LOAD_SHARD/DROP_SHARD rebalance directives when coverage drops, and assigns shard ranges using declared VRAM, quantization, and benchmark throughput. A model is only routable when all layer ranges have node_count >= 1.",
+      "acceptanceCriteria": [
+        "Node registration accepts vram_bytes, ram_bytes, quantizations[], benchmark_tokens_per_sec",
+        "GET /v1/coverage/<model_preset> returns list of {start_layer, end_layer, node_count}",
+        "Model is unroutable when any layer range has node_count=0",
+        "New node gets assigned to highest-priority uncovered range first",
+        "Node disconnection triggers LOAD_SHARD directive to idle node within 30 seconds",
+        "Shard assignment respects VRAM: assigned_layers <= floor((vram_bytes * 0.8) / bytes_per_layer_at_quant)",
+        "Faster node receives wider shard range when both can cover the same gap",
+        "Integration test: three nodes with different VRAM show widest range on largest-VRAM node with 100% coverage",
+        "Integration test: kill middle-range node → tracker issues LOAD_SHARD → coverage recovers",
+        "python -m pytest passes from repo root",
+        "Commit only this story's changes"
+      ],
+      "priority": 13,
+      "passes": true,
+      "notes": "Source issue: .scratch/distributed-inference-network/issues/13-coverage-first-shard-assignment.md",
+      "dependsOn": [
+        "US-003",
+        "US-012"
+      ],
+      "completionNotes": "Completed by agent",
+      "status": "done"
+    },
+    {
+      "id": "US-014",
+      "title": "14 — Tracker-as-first-layer-node (inference entry point)",
+      "description": "Merge inference orchestration into tracker nodes that serve the first-layer shard. A tracker node exposes /v1/chat/completions directly: tokenizes input, runs embed_tokens + layers[0..k], selects onward route from coverage map, forwards binary activations, receives tail output, and streams tokens back. The standalone gateway becomes a thin load-balancer routing to tracker nodes.",
+      "acceptanceCriteria": [
+        "Node with shard_start==0 (or --tracker-mode flag) exposes /v1/chat/completions alongside /forward",
+        "Tracker-node /v1/chat/completions: tokenize → embed → own layers → route selection → forward binary → receive tail → stream SSE",
+        "Two tracker nodes for same model each handle requests independently",
+        "Gateway proxies /v1/chat/completions to tracker nodes (round-robin), discovered via GET /v1/tracker-nodes/<model_preset>",
+        "Integration test: two tracker nodes + two mid-shard nodes for GPT-2; 10 requests via gateway; both tracker nodes receive load; all responses valid",
+        "Existing US-005 OpenAI-compatible tests still pass",
+        "python -m pytest passes from repo root",
+        "Commit only this story's changes"
+      ],
+      "priority": 14,
+      "passes": true,
+      "notes": "Source issue: .scratch/distributed-inference-network/issues/14-tracker-as-node.md",
+      "dependsOn": [
+        "US-012",
+        "US-013"
+      ],
+      "status": "done",
+      "completionNotes": "Implemented: (1) Tracker GET /v1/tracker-nodes/<model> endpoint returning nodes registered with tracker_mode=true at shard_start==0. (2) StubNodeServer and TorchNodeServer now accept tracker_mode/tracker_url params; when tracker_mode=True, /v1/chat/completions is served alongside /forward. (3) TorchNodeServer auto-detects tracker mode when shard_start==0. (4) Gateway _handle_chat_completions checks for tracker-nodes first via _get_tracker_nodes(), proxies round-robin if found, falls back to existing direct pipeline if none (backward compat). (5) CLI --tracker-mode and --tracker-url flags added. (6) Integration test: 2 tracker-nodes + 2 mid-shard nodes for gpt2; 10 requests; round-robin verified (5/5 split); all responses valid OpenAI format. All 78 tests pass."
+    },
+    {
+      "id": "US-015",
+      "title": "15 — Ralph: agent-agnostic runner + status-field-aware dashboard",
+      "description": "Two improvements to the Ralph workflow tooling. (1) Status-field awareness: replace all passes:true/false reads in ralph_progress.py with the rich status field. Surface to-revise and needs-review stories as an attention list in the dashboard; auto command skips them by default. (2) Agent agnosticism: make the agent selectable per session — codex (existing), claude (Claude Code CLI), openrouter (unified API for GPT-4/Mistral/Llama/DeepSeek via OPENROUTER_API_KEY + --model), or custom (--agent-cmd path to any script). Persist agent choice to .ralph-tui/agent-config.json. When ralph-tui does not natively support a requested agent, ralph_progress.py falls back to a thin adapter that calls the agent API directly with the task prompt. (3) Worktree parallelism: ralph_progress.py auto --parallel [N] creates one git worktree per ready task (git worktree add ../AI-worktree-<id> -b feat/<id>), runs up to N agent sessions concurrently, and merges each branch back to master after the agent exits 0 and tests pass. Non-blocking stories that share no file overlap can safely run in parallel. ralph_progress.py list-parallel shows which open stories have no overlapping dependsOn chains and are safe to run concurrently.",
+      "acceptanceCriteria": [
+        "All story.get('passes') reads in ralph_progress.py replaced with _is_done() helper that reads status field with passes fallback",
+        "_story_sets() returns six buckets: done, attention (to-revise/needs-review), active (in-progress), in-design, ready, blocked",
+        "Dashboard shows ⚠ Attention required section listing to-revise/needs-review stories with status_reason",
+        "auto command skips to-revise and needs-review stories; --include-revise flag overrides",
+        "_review_report includes Attention Required section with status_reason for all affected stories",
+        "ralph_progress.py set-agent --agent <name> [--model <model>] writes .ralph-tui/agent-config.json",
+        "--agent codex|claude|openrouter|custom accepted by run-next, auto, review subcommands",
+        "run-next --agent custom --agent-cmd ./my-agent.sh runs a task via custom script (prompt file as $1)",
+        "Saved agent config is loaded as default when --agent is not passed on the CLI",
+        "python -m pytest passes from repo root",
+        "ralph_progress.py auto --parallel 2 starts two worktrees concurrently for the two highest-priority ready tasks",
+        "Each worktree is created at ../AI-worktree-<story-id> on branch feat/<story-id>",
+        "After agent exits 0 and pytest passes in the worktree, the branch is merged to master and the worktree removed",
+        "ralph_progress.py list-parallel prints the set of open stories with no shared dependency chain (safe to parallelize)",
+        "If a worktree merge fails (conflict), the worktree is preserved for manual resolution and reported clearly"
+      ],
+      "priority": 15,
+      "passes": true,
+      "status": "done",
+      "notes": "Source issue: .scratch/distributed-inference-network/issues/15-ralph-agent-agnostic-status-aware.md",
+      "dependsOn": [],
+      "completionNotes": "Implemented by agent: status-aware helpers (_is_done, _needs_attention, _is_active, _is_in_design), 6-bucket _story_sets, attention dashboard section, _review_report Attention Required block, auto --include-revise, set-agent subcommand with persistent agent-config.json, _run_openrouter stub, custom agent support, list-parallel subcommand, and auto --parallel N worktree orchestration. All 65 tests pass."
+    },
+    {
+      "id": "US-016",
+      "title": "16 — Mining-style node startup CLI + live dashboard",
+      "description": "Replace the bare flag-driven node CLI with a wizard-guided first-run experience (like a GPU mining client) followed by a live terminal dashboard once the node is running. On first run, the wizard auto-detects GPU VRAM, presents a curated list of compatible models with VRAM requirements at each quantization level, lets the user pick a download location, and writes a persistent config file so subsequent starts are one command. Once the node is running, the wizard gives way to a rich live status panel showing: GPU temp + VRAM used, tokens/sec, requests served, peers connected, TAI earned (stub until US-006 is live). A Browse HuggingFace option calls the HF Hub API so users can load any HF model beyond the curated list.",
+      "acceptanceCriteria": [
+        " with no args and no config file enters the interactive setup wizard",
+        "Wizard step 1: auto-detect GPU(s) via torch.cuda / torch.version.hip; print GPU name + total VRAM",
+        "Wizard step 2: show curated model list (name, HF repo, layers, VRAM@NF4/INT8/BF16); mark models that do NOT fit available VRAM as [too large]",
+        "Wizard step 3: offer [B] Browse HuggingFace — calls HF Hub API (huggingface_hub.list_models filtered by pipeline_tag=text-generation, sorted by downloads, top 20) and lets user enter a custom HF repo ID",
+        "Wizard step 4: prompt for download directory (default ~/.meshnet/models/); validate writable; show estimated disk usage for chosen model+quantization",
+        "Wizard step 5: prompt for tracker URL (default http://localhost:8080); validate connection",
+        "Wizard writes ~/.config/meshnet/config.json; second run skips wizard and starts directly",
+        "All wizard values overridable via CLI flags: --model, --download-dir, --quantization [bf16|int8|nf4], --tracker, --wallet, --reset-config",
+        "Once node is running, wizard clears and a live dashboard renders every 2s (rich.live): GPU util%, VRAM used/total, tokens/sec (EMA), requests served, TAI earned (stub 0.0), peers connected, uptime, current model/shard range",
+        "Dashboard exits cleanly on Ctrl-C with a summary line",
+        "Works inside WSL2 (no termios/ioctl calls that fail on Windows terminal; fall back to plain-text status if rich is not available)",
+        " passes from repo root",
+        "Commit only this story changes"
+      ],
+      "priority": 16,
+      "status": "done",
+      "notes": "Source issue: .scratch/distributed-inference-network/issues/16-mining-cli-ux.md",
+      "dependsOn": [
+        "US-004",
+        "US-012"
+      ],
+      "completionNotes": "Implemented: mining-style wizard with GPU detection, curated model list (7 models with NF4/INT8/BF16 VRAM requirements), HF Hub browse, persistent config, rich live dashboard with plain-text WSL2 fallback. 19 tests, 97 passed total."
+    },
+    {
+      "id": "US-017",
+      "title": "17 — P2P gossip, NAT-traversal relay node, and SSL/TLS",
+      "description": "Add a gossip layer so nodes discover each other and propagate coverage-map changes without polling the tracker continuously. Introduce a publicly-hosted relay node (run by the team) that solves NAT traversal using circuit relay (Petals-style) and serves as the bootstrap peer list for new nodes. Encrypt all node-to-node and node-to-tracker communications with TLS. Gossip protocol: WebSocket-based PubSub over wss://, topics: node-join / node-leave / coverage-update / heartbeat. Peer discovery: mDNS (zeroconf) for LAN, public relay bootstrap list for internet. NAT traversal: relay node acts as TCP-level circuit relay when direct connection fails (hole-punching first, relay second). Architecture is designed to migrate to libp2p GossipSub + Kademlia DHT in a future story without breaking the message schema.",
+      "acceptanceCriteria": [
+        "All HTTP between nodes and tracker uses HTTPS (TLS 1.3); self-signed cert generated on first run and fingerprint pinned in config; relay node uses Let's Encrypt",
+        "Nodes broadcast node-join / node-leave events over wss:// to known peers within 1s of registration",
+        "mDNS peer discovery (Python zeroconf) finds other meshnet nodes on the same LAN segment without manual tracker URL entry",
+        "Public relay bootstrap list (hardcoded relay URL + ) is consulted when no LAN peers found",
+        "Relay node is a standalone meshnet package () with  CLI: starts a WebSocket relay server + circuit relay + optional tracker proxy",
+        "When a node behind NAT cannot accept inbound connections, the relay forwards its traffic; node advertises relay address (relay_url/node_id) to tracker as its effective endpoint",
+        "Tracker accepts both direct node URLs and relay-proxied URLs in heartbeat payloads",
+        "Integration test: two nodes in separate processes on localhost (simulating NAT) communicate via a local relay process; inference request routes correctly",
+        "ADR-0010 documents the gossip protocol, relay architecture, and migration path to libp2p",
+        " passes from repo root",
+        "Commit only this story changes"
+      ],
+      "priority": 17,
+      "status": "done",
+      "notes": "Source issue: .scratch/distributed-inference-network/issues/17-p2p-gossip-relay-ssl.md",
+      "dependsOn": [
+        "US-013",
+        "US-014"
+      ],
+      "completionNotes": "Implemented: packages/p2p (identity, TLS cert+fingerprint, GossipClient WSS PubSub, MdnsDiscovery with zeroconf optional), packages/relay (RelayServer gossip hub + circuit relay proxy, meshnet-relay CLI), tracker extended with relay_addr/cert_fingerprint/peer_id, relay_bootstrap.json, ADR-0010. 18 new tests; 115 total passed."
+    },
+    {
+      "id": "US-018",
+      "title": "18 — End-to-end two-machine LAN inference test",
+      "description": "Prove the network works across two real machines: the Linux rig (this machine) and a Windows 11 rig running WSL2. One machine runs the tracker + first-shard node (inference entry point). The other machine runs a second-shard node. A client sends a real inference request and receives a streamed response. This story is primarily a test plan + setup guide + test execution script; it produces documented evidence (logs, timing, token output) that real distributed inference works. It also surfaces any real-world issues (port forwarding, CUDA driver version mismatches, WSL2 CUDA passthrough, model download paths) that need fixing.",
+      "acceptanceCriteria": [
+        "docs/INSTALL_WINDOWS.md exists: step-by-step WSL2 + CUDA + meshnet-node install on Windows 11",
+        "docs/TWO_MACHINE_TEST.md exists: how to start tracker on machine A, node on machine B, run inference, interpret output",
+        "A test script scripts/test_lan_inference.py: given --tracker-url, --gateway-url, sends 3 chat completion requests, asserts valid OpenAI format, prints token count + latency + which nodes served each request",
+        "Both machines can reach each other on LAN (documented: firewall rules, port list)",
+        "At least one successful inference recorded: the test script exits 0 with output showing tokens generated and node IDs",
+        "Latency breakdown logged: gateway→node-A, node-A→node-B, node-B→gateway (approximate, from server logs)",
+        "Known issues during test documented in docs/TWO_MACHINE_TEST.md under a Known Issues section",
+        "Commit only this story changes"
+      ],
+      "priority": 18,
+      "status": "done",
+      "notes": "Source issue: .scratch/distributed-inference-network/issues/18-two-machine-lan-test.md",
+      "dependsOn": [
+        "US-016",
+        "US-017"
+      ],
+      "completionNotes": "docs/INSTALL_WINDOWS.md: WSL2+CUDA+meshnet-node install guide. docs/TWO_MACHINE_TEST.md: two-machine LAN test procedure with known issues. scripts/test_lan_inference.py: stdlib-only test script, 3 requests, exit 0 on success, auto-discovers gateway from tracker."
+    },
+    {
+      "id": "US-019",
+      "title": "19 — Distributed tracker consensus (Raft assignments + CRDT heartbeats)",
+      "description": "Replace the single-point-of-failure tracker with a fault-tolerant cluster. Tracker nodes elect a leader via Raft and commit shard assignments as log entries — all tracker nodes agree on who owns what. Node liveness (heartbeats) uses CRDT gossip (eventual consistency, high frequency OK). A node registers with any tracker node; the write is forwarded to the leader and replicated to followers. A 3-node tracker cluster survives one tracker failure without losing assignment state. The relay/gossip layer already built in US-017 handles peer heartbeats; this story wires Raft on top for authoritative assignments.",
+      "acceptanceCriteria": [
+        "3 tracker nodes can be started and form a Raft cluster (leader election, log replication)",
+        "A node registers with any follower — the registration is forwarded to the leader and replicated",
+        "Killing the leader causes a new election within 5 seconds; registrations continue working",
+        "Shard assignments returned by any tracker node are identical (strong consistency)",
+        "Node heartbeats use CRDT gossip (not Raft) — high-frequency, eventual consistency",
+        "meshnet-tracker CLI gains --cluster-peers flag to specify peer tracker URLs",
+        "Integration test: 3 tracker nodes, kill leader mid-test, verify assignment still works",
+        "QUICKSTART.md updated with multi-tracker setup section"
+      ],
+      "priority": 19,
+      "status": "done",
+      "notes": "Architecture decision: Raft for assignments (strong consistency) + CRDT gossip for liveness (eventual consistency). User approved 2026-06-29.",
+      "dependsOn": [
+        "US-017"
+      ],
+      "completionNotes": "raft.py: minimal Raft consensus (leader election, log replication, AppendEntries, RequestVote). gossip.py: LWW CRDT gossip for node heartbeats. TrackerServer gains cluster_peers + cluster_self_url params. --cluster-peers and --self-url CLI flags added. 6 integration tests: leader election <1s, follower registration propagation, leader kill + re-election <5s, gossip table."
+    },
+    {
+      "id": "US-020",
+      "title": "20 — Tracker + node hardening: BrokenPipe fix, deterministic node IDs, HF coverage",
+      "description": "Polish pass on tracker and node after real multi-host LAN testing. Three concrete issues: (1) tracker _send_json crashed with BrokenPipeError when a slow-inference client disconnected mid-stream; (2) node IDs were random UUIDs, making re-registration after tracker restart create phantom entries; (3) HF-repo model coverage endpoint did not handle short-name vs full-repo lookups consistently.",
+      "acceptanceCriteria": [
+        "BrokenPipeError in tracker _send_json is silently swallowed (client disconnected is not an error)",
+        "Node IDs are deterministic (wallet-address + port hash) so re-registration after tracker restart reuses the same ID",
+        "GET /v1/coverage/<model> accepts both short names and full HF repo IDs",
+        "python -m pytest passes from repo root"
+      ],
+      "priority": 20,
+      "status": "done",
+      "notes": "Discovered during first two-machine LAN test (US-018 follow-up).",
+      "dependsOn": [
+        "US-018"
+      ],
+      "completionNotes": "BrokenPipeError wrapped in try/except in tracker and node send paths. Node IDs now sha256(wallet+port)[:16]. Coverage endpoint normalises short/full HF names."
+    },
+    {
+      "id": "US-021",
+      "title": "21 — --route-timeout CLI flag for node tracker route lookup",
+      "description": "The node's slow-path tracker route lookup used a hard-coded 30-second timeout. On high-latency networks (relay, 5G) or when the tracker is slow to respond, this caused premature failures. Expose it as a CLI flag so operators can tune it per deployment.",
+      "acceptanceCriteria": [
+        "meshnet-node start accepts --route-timeout <seconds> (float, default 30.0)",
+        "route_timeout is passed through to TorchNodeServer and used in tracker /v1/route HTTP call",
+        "TorchNodeServer exposes route_timeout as a readable property",
+        "Test: setting a non-default timeout is reflected on the running server object",
+        "python -m pytest passes"
+      ],
+      "priority": 21,
+      "status": "done",
+      "notes": "Small QoL story; acceptance criteria driven by a failing test.",
+      "dependsOn": [
+        "US-016"
+      ],
+      "completionNotes": "cli.py: --route-timeout added to start_cmd. TorchNodeServer._route_timeout stored, route_timeout property exposed. Test: test_route_timeout_config_is_exposed_on_server."
+    },
+    {
+      "id": "US-022",
+      "title": "22 — X-Meshnet-Start-Layer: overlapping shard execution protocol",
+      "description": "When two nodes register overlapping shard ranges (e.g. node A covers 0-15 and node B covers 12-23), the naive pipeline re-runs layers 12-15 on node B. The X-Meshnet-Start-Layer header tells each downstream node which layer index to start from, skipping already-computed layers. The tracker injects start_layer into X-Meshnet-Route hops at proxy time.",
+      "acceptanceCriteria": [
+        "Tracker _handle_proxy_chat builds route hops with start_layer computed from covered_up_to",
+        "Node _handle_binary_forward reads X-Meshnet-Start-Layer and passes it to backend.forward_bytes",
+        "Node _get_remaining_route parses start_layer from both injected header and /v1/route slow-path",
+        "TorchModelShard.forward_bytes accepts optional start_layer and skips layers below it",
+        "Test: two-node route with overlapping shards produces correct output without double-computing layers",
+        "python -m pytest passes"
+      ],
+      "priority": 22,
+      "status": "done",
+      "notes": "Protocol decision: option A (start_layer injected by tracker, not negotiated peer-to-peer). Approved in design session.",
+      "dependsOn": [
+        "US-014",
+        "US-019"
+      ],
+      "completionNotes": "X-Meshnet-Start-Layer header added to forward protocol. Tracker computes covered_up_to as it builds route_hops. model_backend.forward_bytes(start_layer=) implemented. Tests added for overlapping shard scenarios."
+    },
+    {
+      "id": "US-023",
+      "title": "23 — Heartbeat stats payload: request counters + dynamic reassignment response",
+      "description": "Node heartbeats are currently empty POSTs. Extend them to carry cumulative stats (total_requests, failed_requests, queue_depth, uptime_seconds) so the tracker can make informed load-balancing decisions. The heartbeat response may include a new_assignment field, enabling the tracker to redirect a node to a different shard without a restart.",
+      "acceptanceCriteria": [
+        "Heartbeat POST body includes total_requests, failed_requests, queue_depth, uptime_seconds, status",
+        "TorchNodeServer tracks total_requests, failed_requests, queue_depth with a threading.Lock",
+        "Tracker stores the last heartbeat payload per node and exposes it in /v1/nodes (list endpoint)",
+        "Heartbeat response may include new_assignment: {model, shard_start, shard_end}; node logs it",
+        "Stats survive tracker outage: buffered locally, flushed on next successful heartbeat",
+        "python -m pytest passes"
+      ],
+      "priority": 23,
+      "status": "done",
+      "notes": "Reassignment is logged only for now; hot-reload (load new shard without restart) is a future story.",
+      "dependsOn": [
+        "US-016"
+      ],
+      "completionNotes": "torch_server.py: total_requests, failed_requests, queue_depth added with _stats_lock. startup.py _start_heartbeat sends stats dict. Tracker stores last_heartbeat per node. new_assignment field in heartbeat response logged by node."
+    },
+    {
+      "id": "US-024",
+      "title": "24 — Enhanced availability map with per-node health details",
+      "description": "GET /v1/coverage/<model> returns coarse band-level coverage. Operators need to know which specific nodes are in each band and whether they are healthy (last heartbeat time, queue depth). Extend the response to include a nodes array per band with per-node health details, and add a separate detailed endpoint.",
+      "acceptanceCriteria": [
+        "GET /v1/coverage/<model> response includes nodes: [{node_id, endpoint, healthy, queue_depth, last_seen_s}] per band",
+        "healthy is true iff last heartbeat < heartbeat_timeout seconds ago",
+        "Existing band fields (start_layer, end_layer, node_count) are preserved",
+        "Tests updated: band assertions check nodes array not just node_count",
+        "python -m pytest passes"
+      ],
+      "priority": 24,
+      "status": "done",
+      "notes": "Grouped-by-band format chosen (both node list and band metadata in same object). Dead nodes included in response with healthy=false.",
+      "dependsOn": [
+        "US-013"
+      ],
+      "completionNotes": "_coverage_map_detailed added to tracker. Each band now includes nodes list. Tests updated to check band fields individually rather than exact dict comparison."
+    },
+    {
+      "id": "US-025",
+      "title": "25 — Model usage statistics: rolling RPM windows + SQLite persistence",
+      "description": "Trackers need to know which models are being requested most often to make smart load-balancing and assignment decisions. Add per-model rolling request-rate counters (last hour, last day, last month) with a circular-bucket implementation. Persist buckets to SQLite so stats survive tracker restarts. Support gossip-based stat sharing between tracker peers (additive merge, per-tracker slices).",
+      "acceptanceCriteria": [
+        "_RollingCounter: circular buckets, epoch-indexed, stale buckets auto-reset on record()",
+        "_StatsCollector: record_request(), get_local_rpms(), merge_peer_rpms(), get_combined_stats()",
+        "Stats persisted to SQLite (--stats-db PATH); loaded on startup",
+        "GET /v1/stats returns combined stats (local + peer slices) as {model: {rpm_last_hour, rpm_last_day, rpm_last_month}}",
+        "POST /v1/stats/gossip accepts a peer's local slice and merges it additively",
+        "_handle_proxy_chat records a stat after model is determined",
+        "6 unit tests covering counter, collector, merge, SQLite round-trip, gossip endpoint",
+        "python -m pytest passes"
+      ],
+      "priority": 25,
+      "status": "done",
+      "notes": "Option B chosen: stats stored locally per tracker and synced via gossip (not aggregated centrally). Rolling windows: 60×1min buckets (1 hour), 24×1hr (1 day), 30×1day (~1 month).",
+      "dependsOn": [
+        "US-017"
+      ],
+      "completionNotes": "_RollingCounter and _ModelStats classes in server.py. _StatsCollector with SQLite save/load. /v1/stats and /v1/stats/gossip endpoints. --stats-db CLI flag. Stats gossip in _stats_loop thread."
+    },
+    {
+      "id": "US-026",
+      "title": "26 — Smart model assignment via demand×coverage scoring",
+      "description": "The /v1/network/assign endpoint assigns new nodes to whichever model and shard range covers the biggest uncovered gap. This ignores demand: a model with 1000 RPM and 60% coverage should attract more nodes than a zero-traffic model with 50% coverage. Add a scoring formula: score = (demand_rpm + 1.0) × (coverage_deficit + 0.01) so high-demand under-covered models win. price_per_token is reserved in the protocol response at 0.0.",
+      "acceptanceCriteria": [
+        "/v1/network/assign returns the highest-scoring model+shard using demand×coverage formula",
+        "demand_rpm uses combined stats from _StatsCollector (local + peer slices)",
+        "coverage_deficit is fraction of layers with zero coverage [0.0, 1.0]",
+        "price_per_token: 0.0 included in response (reserved field)",
+        "Models with no traffic still compete by coverage (floor: demand_rpm + 1.0)",
+        "Test: two models, one high-demand low-coverage wins over low-demand high-coverage",
+        "python -m pytest passes"
+      ],
+      "priority": 26,
+      "status": "done",
+      "notes": "Score formula chosen after grilling: (demand_rpm + 1.0) × (coverage_deficit + 0.01). The +1.0 floor means coverage alone drives assignment when demand is zero.",
+      "dependsOn": [
+        "US-025",
+        "US-024"
+      ],
+      "completionNotes": "_handle_network_assign updated with scoring. _effective_throughput helper. price_per_token: 0.0 in response. Tests in test_tracker_routing.py."
+    },
+    {
+      "id": "US-027",
+      "title": "27 — Throughput-optimized routing: effective throughput as tiebreak",
+      "description": "The greedy max-reach route selection picks nodes by shard coverage but does not consider node speed. When two nodes cover the same remaining range, prefer the one with higher effective throughput (benchmark_tokens_per_sec / (queue_depth + 1)). This is a tiebreak only — coverage maximization remains the primary objective.",
+      "acceptanceCriteria": [
+        "_effective_throughput(node) = benchmark_tokens_per_sec / (queue_depth + 1)",
+        "_select_route uses throughput as tiebreak when shard_end is equal",
+        "Test: two nodes with same shard range, different throughput — faster node is picked",
+        "Existing greedy coverage tests still pass (throughput does not change primary selection)",
+        "python -m pytest passes"
+      ],
+      "priority": 27,
+      "status": "done",
+      "notes": "Throughput tiebreak only — coverage-maximizing greedy stays primary. queue_depth from last heartbeat.",
+      "dependsOn": [
+        "US-023"
+      ],
+      "completionNotes": "_effective_throughput added to server.py. _select_route updated: when shard_end equal, max throughput wins. Tests added and existing tests corrected for new tiebreak order."
+    },
+    {
+      "id": "US-028",
+      "title": "28 — Routing correctness tests: three-node, overlap, and throughput scenarios",
+      "description": "Add a comprehensive test suite for route selection covering: greedy three-node no-overlap ordering, overlapping shard handling, throughput tiebreak, and gap detection. These tests lock in the routing contract so refactors don't silently regress.",
+      "acceptanceCriteria": [
+        "test_select_route_no_overlap_three_nodes: greedy picks nodes in layer order (A→C→B for ranges 0-7, 8-15, 16-23)",
+        "test_select_route_with_overlap: overlapping nodes correctly resolved",
+        "test_select_route_throughput_tiebreak: faster node wins when reach is equal",
+        "test_select_route_gap_leaves_error: partial coverage returns error",
+        "All tests pass without mocking (in-process tracker server)",
+        "python -m pytest passes"
+      ],
+      "priority": 28,
+      "status": "done",
+      "notes": "Tests revealed the greedy picks A→C→B (not A→B→C) for non-overlapping ranges when C starts at a lower layer than B. Test expectation corrected to match algorithm.",
+      "dependsOn": [
+        "US-027"
+      ],
+      "completionNotes": "7 routing tests added to test_tracker_routing.py. _make_node helper. test_select_route_no_overlap_three_nodes corrected: greedy outputs [A, C, B] when C.shard_start < B.shard_start."
+    },
+    {
+      "id": "US-029",
+      "title": "29 — Outbound relay client: NAT/internet pipeline hops",
+      "description": "Nodes behind NAT (WSL2, 5G, home routers) register with the tracker including a relay_addr (wss://relay/rpc/{peer_id}). When the head node needs to forward activations to a behind-NAT peer, it must use the relay instead of direct HTTP. Add _relay_hop() to torch_server.py that opens a per-hop WebSocket to the relay's /rpc/{peer_id} endpoint, sends the binary activation (base64-encoded), and returns the response. If relay fails, fall back to direct HTTP.",
+      "acceptanceCriteria": [
+        "_relay_hop(relay_addr, path, body, headers, timeout) opens WS to relay, sends body_base64, returns (status, headers, body)",
+        "_get_remaining_route returns list[dict] with relay_addr field (was list[tuple])",
+        "_run_downstream_pipeline dispatches via _relay_hop when hop has relay_addr",
+        "Falls back to direct HTTP if relay connection fails (logs warning)",
+        "Tracker _handle_proxy_chat includes relay_addr in downstream hop dicts",
+        "relay_bridge._handle_request decodes body_base64; response uses body_base64 for binary (octet-stream)",
+        "All 157 tests pass",
+        "QUICKSTART.md updated with relay NAT/internet section"
+      ],
+      "priority": 29,
+      "status": "done",
+      "notes": "relay_addr format: wss://relay.../rpc/{peer_id}. Binary activations (bfloat16) base64-encoded through relay JSON protocol — no precision loss. WSL2 nodes now work behind NAT without --advertise-host.",
+      "dependsOn": [
+        "US-017",
+        "US-022"
+      ],
+      "completionNotes": "_relay_hop() added to torch_server.py. _get_remaining_route returns list[dict]. relay_bridge.py updated with body_base64 support. Tracker injects relay_addr into downstream hop dicts. 157 tests pass."
+    },
+    {
+      "id": "US-030",
+      "title": "30 — Manual route selection + hop-penalty benchmarking",
+      "description": "Two additions to the tracker. (1) Callers can pin an explicit inference route by passing an optional \"route\": [\"<node_id>\", ...] field in the POST /v1/chat/completions body. The tracker uses those nodes in order instead of auto-selecting; clients that omit the field are unaffected. (2) A new privileged POST /v1/benchmark/hop-penalty endpoint runs the same prompt through up to three routes (1-node, 2-node, 3-node) using whatever nodes are registered, records per-hop latency, and appends results to benchmark_results.json in the tracker's working directory. The routing algorithm is not changed — this story is data collection only. Auth is a header-presence stub (non-empty Authorization header required for benchmark endpoints).",
+      "acceptanceCriteria": [
+        "POST /v1/chat/completions accepts optional \"route\": [node_id, ...] in the request body; if present, tracker routes to those nodes in order; if absent, existing auto-routing is unchanged",
+        "Missing or invalid node IDs in route return HTTP 400 with a descriptive error message",
+        "POST /v1/benchmark/hop-penalty requires a non-empty Authorization header; missing/empty returns HTTP 401",
+        "Benchmark body: {\"model\": \"...\", \"prompt\": \"...\", \"max_new_tokens\": 64 (optional)}",
+        "Benchmark fans out to up to 3 routes (1-node, 2-node, 3-node) using currently registered nodes; routes with insufficient coverage are skipped, not errored",
+        "Benchmark response includes per-route entries: {\"route\": [node_id, ...], \"total_ms\": float, \"per_hop_ms\": [float, ...], \"tokens_generated\": int}",
+        "Results appended to <tracker_cwd>/benchmark_results.json (created if absent) as a JSON array; each entry includes ISO timestamp, model, sha256 of prompt, and per-route breakdown",
+        "GET /v1/benchmark/results returns the stored results array; also requires non-empty Authorization header",
+        "Integration test: pin a 1-node route and a 2-node route for the same prompt; assert 2-node result has 2 per_hop_ms entries; assert both records appear in benchmark_results.json",
+        "Clients that never send route or call /v1/benchmark/* are completely unaffected (existing tests pass unchanged)",
+        "python -m pytest passes from repo root",
+        "Commit only this story's changes"
+      ],
+      "priority": 30,
+      "status": "done",
+      "notes": "Source issue: .scratch/distributed-inference-network/issues/30-manual-route-and-hop-benchmark.md. Design decisions grilled 2026-07-01: route via body field, explicit-only benchmark trigger, auth stub, routing algorithm unchanged, persist to JSON file.",
+      "dependsOn": [
+        "US-014",
+        "US-019"
+      ],
+      "completionNotes": "Pinned routes: optional \"route\": [node_id,...] in POST /v1/chat/completions (400 on unknown/invalid ids; absent field unchanged). POST /v1/benchmark/hop-penalty + GET /v1/benchmark/results, both auth-gated (non-empty Authorization). Fans out to 1/2/3-node routes via _find_pinned_route; per-hop latency derived incrementally (k-node total minus (k-1)-node total); appends to benchmark_results.json (path configurable). 6 tests in tests/test_manual_route_benchmark.py."
+    },
+    {
+      "id": "US-031",
+      "title": "31 — Billing ledger: per-token pricing, 90/10 split, pending balances",
+      "description": "Tracker-side off-chain billing per ADR-0015. Each model preset has a USDT price per 1K tokens in tracker config. On request completion the tracker debits the client's API-key ledger balance and credits 90% to the serving nodes' pending balances proportional to work_units (layers × tokens), 10% to the protocol cut. Requests from API keys with insufficient balance are rejected with 402. Ledger persists in the existing tracker SQLite and replicates across the tracker hive.",
+      "acceptanceCriteria": [
+        "Per-model price_per_1k_tokens in tracker config with sane defaults",
+        "Completed request debits client ledger: price × total_tokens / 1000",
+        "90% split across serving nodes by work_units; 10% accrues to protocol_cut",
+        "Insufficient balance → HTTP 402 before routing (no free work)",
+        "Balances survive tracker restart (SQLite) and replicate to hive followers",
+        "Unit tests: single-node route, 3-node split, exhausted balance, restart persistence"
+      ],
+      "priority": 31,
+      "status": "done",
+      "notes": "Pure off-chain — no Solana calls in this story. Reuses ComputeAttribution/work_units from packages/contracts.",
+      "dependsOn": [
+        "US-023",
+        "US-025"
+      ],
+      "completionNotes": "BillingLedger in packages/tracker/meshnet_tracker/billing.py: event-sourced USDT ledger (credit/charge/payout/forfeit events, id-deduped), SQLite persistence, per-model price_per_1k_tokens (preset key or prices dict), 90/10 split by work units, walletless shares to protocol cut. server.py: 401/402 gate before routing, billing on non-streaming/streaming/relayed completion paths, GET /v1/billing/summary, POST /v1/billing/gossip, event push in _stats_loop (cursor advances only when all peers reached). CLI --billing-db. settle_node_payout/forfeit_pending are the US-033/034 hooks. 11 new tests in tests/test_billing_ledger.py; suite 195 passed (3 pre-existing ModuleNotFoundError: openai failures unrelated)."
+    },
+    {
+      "id": "US-032",
+      "title": "32 — Devnet custodial treasury: mock-USDT mint + deposit watcher",
+      "description": "Real Solana adapter behind the packages/contracts boundary, custodial model per ADR-0015. A setup script creates the mock-USDT SPL mint (6 decimals) and treasury token account on devnet. Clients register a wallet pubkey against their API key; the deposit watcher polls the treasury token account and credits the sender's API-key ledger balance on confirmed USDT transfers. Mint address, RPC URL, and treasury keypair path are config (.env.devnet).",
+      "acceptanceCriteria": [
+        "scripts/devnet_setup.py creates mint + treasury ATA and prints .env.devnet values",
+        "POST /v1/wallet/register binds client wallet pubkey to API key",
+        "Deposit watcher credits ledger within one poll interval of a confirmed transfer",
+        "Duplicate/replayed transactions are credited exactly once (signature dedupe)",
+        "Local solana-test-validator integration test covers mint→deposit→credit flow",
+        "Deterministic local adapter still works for CI without any validator"
+      ],
+      "priority": 32,
+      "status": "done",
+      "notes": "Supersedes 'testnet never devnet' note in issue 06. solana-py + spl-token client libs.",
+      "dependsOn": [
+        "US-031"
+      ],
+      "completionNotes": "SolanaCustodialTreasury in packages/contracts/meshnet_contracts/solana_adapter.py: urllib JSON-RPC client + solders signing (installed solana-py 0.40 has no sync client), deposit parsing from jsonParsed token balances, batched payouts, devnet helpers (create_mock_usdt_mint, mint_mock_usdt). scripts/devnet_setup.py writes .env.devnet. POST /v1/wallet/register binds wallet→api_key (replicated ledger 'bind' events). Deposit watcher thread credits exactly-once via deposit-<signature> event ids. Tests: fake-treasury watcher tests + skipif-gated solana-test-validator e2e."
+    },
+    {
+      "id": "US-033",
+      "title": "33 — Settlement loop: leader-only batched USDT payouts",
+      "description": "The Raft leader runs the settlement loop per ADR-0015: pay a node when pending ≥ payout_threshold OR time since its last payout ≥ max_period, whichever fires first, with a dust floor. Payouts are batched SPL transfers treasury→node wallet on devnet. Dev defaults: period 60s, threshold ~0; prod defaults: period 24h, threshold few USDT — all dynamic config. Settlement history (tx signature, node, amount, epoch) persists and replicates.",
+      "acceptanceCriteria": [
+        "Only the Raft leader settles; followers never sign (asserted in a 3-tracker test)",
+        "Trigger: pending ≥ threshold OR elapsed ≥ max_period; dust floor respected",
+        "Batched SPL transfers land on devnet; pending balances zeroed atomically with recorded tx signature",
+        "Failed/timeout transactions retry without double-pay (idempotent by settlement id)",
+        "Settlement history queryable via tracker HTTP endpoint",
+        "End-to-end devnet test: fund client → run inference → observe node wallet USDT balance increase"
+      ],
+      "priority": 33,
+      "status": "done",
+      "notes": "Mining-pool standard: threshold + max-interval, dust floor. Treasury key only on settlement-capable trackers.",
+      "dependsOn": [
+        "US-019",
+        "US-032"
+      ],
+      "completionNotes": "Settlement loop in TrackerServer: leader-only (raft.is_leader; standalone settles), trigger pending≥threshold OR pending age≥settle_period, dust floor. Pending debited via payout events referencing settlement id BEFORE send; unconfirmed batches resent with same id → no double-pay. confirm via 'settlement' event with tx signature. GET /v1/billing/settlements. CLI: --settle-period/--payout-threshold/--payout-dust-floor (prod 24h/5/0.01; dev 60/0). Banned wallets skipped. 6 tests in tests/test_settlement_loop.py incl. non-leader-never-signs."
+    },
+    {
+      "id": "US-034",
+      "title": "34 — Hardened proof-of-work: pending-balance forfeiture penalty",
+      "description": "Wire the validator's ~5% sampling to the new penalty per ADR-0015: on confirmed output divergence the node's entire pending balance is forfeited to the protocol cut, a strike is recorded, and three strikes ban the wallet (tracker rejects registration and excludes from routes). Probationary period (first N jobs unpaid) is retained as the re-entry cost. Penalty math documented: at 5% sampling, forfeiting ~a day's pending earnings ≫ 20× per-job cheat gain.",
+      "acceptanceCriteria": [
+        "Validator divergence → pending balance forfeited to protocol_cut in the same ledger transaction as the strike",
+        "3rd strike bans wallet: registration rejected, excluded from all routes",
+        "Banned wallet's unpaid pending balance is not paid out at next settlement",
+        "Probation: first N jobs (default 50) accrue no pending balance; node CLI shows remaining",
+        "Integration test: deliberately-bad node loses pending, accrues strikes, gets banned within 60 requests",
+        "Slash/forfeiture events visible in tracker logs and settlement history"
+      ],
+      "priority": 34,
+      "status": "done",
+      "notes": "No stake deposit — pending balance IS the collateral. Amends ADR-0003 penalty; sampling mechanics unchanged.",
+      "dependsOn": [
+        "US-031"
+      ],
+      "completionNotes": "ValidatorProcess(billing=ledger) forfeits entire pending balance on divergence (same cycle as strike). Tracker POST /v1/billing/forfeit (auth stub) for remote validators: forfeit + strike + ban state. Probation wired into _bill_completed: wallets with probationary_jobs_remaining>0 have their share redirected to protocol cut; record_completed_job per request. Penalty math in packages/validator/README.md. 5 tests in tests/test_forfeiture_penalty.py; banned-never-paid asserted in settlement tests."
+    },
+    {
+      "id": "US-035",
+      "title": "35 — Tracker web dashboard",
+      "description": "Web dashboard served by the tracker (single-page, no build step — plain HTML/JS polling tracker HTTP endpoints). Shows: hive membership and Raft leader, node registry with health/scores/coverage map, client ledger balances, node pending balances, settlement history with devnet explorer links, strikes/bans, and rolling RPM stats. Read-only in this story; every tracker in the mesh can serve it from its replicated state.",
+      "acceptanceCriteria": [
+        "GET /dashboard serves the UI from any tracker (leader or follower)",
+        "Panels: hive/leader, nodes+coverage, client balances, pending payouts, settlement history (with tx links), strikes/bans, RPM stats",
+        "Auto-refresh ≤5s without page reload",
+        "No new build toolchain — static assets embedded in the tracker package",
+        "Works against a 3-tracker hive in the two-machine LAN test setup"
+      ],
+      "priority": 35,
+      "status": "done",
+      "notes": "CLI dashboard already exists from US-016; this is the web counterpart. Write ops (price config, manual settle) deferred.",
+      "dependsOn": [
+        "US-031",
+        "US-033"
+      ],
+      "completionNotes": "GET /dashboard served from embedded dashboard.html (package-data, no build step) by any tracker. Panels: hive/leader (raft status), nodes+coverage grouped by model, client balances, node pending + protocol cut, settlement history with devnet explorer links, strikes/bans/forfeitures (GET /v1/registry/wallets + snapshot forfeits), RPM stats. 4s auto-refresh via fetch polling. 3 tests in tests/test_dashboard.py."
+    }
+  ],
+  "metadata": {
+    "updatedAt": "2026-07-01T00:00:00.000Z",
+    "statusVocabulary": {
+      "open": "Not started",
+      "in-design": "Decisions pending before implementation can begin",
+      "in-progress": "Agent actively working on it",
+      "needs-review": "Implemented, awaiting human review or verification",
+      "to-revise": "Previously done, needs rework due to design or architecture changes",
+      "done": "Implemented, tested, and verified",
+      "blocked": "Waiting on unresolved external dependency"
+    }
+  }
+}

packages/contracts/meshnet_contracts/__init__.py

@@ -190,6 +190,10 @@ class RegistryContract:
         wallet = self.get_wallet(wallet_address)
         return wallet.stake_balance >= minimum_stake
 
+    def list_wallets(self) -> dict[str, RegistryWallet]:
+        """Snapshot of all known wallets (dashboard / monitoring)."""
+        return dict(self._state.registry)
+
     def record_completed_job(self, wallet_address: str) -> RegistryWallet:
         current = self.get_wallet(wallet_address)
         updated = RegistryWallet(

packages/contracts/meshnet_contracts/solana_adapter.py

@@ -0,0 +1,297 @@
+"""Custodial Solana treasury adapter (ADR-0015, US-032/US-033).
+
+The entire on-chain surface is plain SPL token operations against a single
+project-owned treasury wallet: read confirmed incoming USDT transfers, send
+batched payout transfers, plus devnet helpers to create the mock-USDT mint.
+No Anchor programs.
+
+RPC transport is a minimal urllib JSON-RPC client (the installed solana-py
+0.40 removed its sync client); signing and instruction building use solders /
+spl.token. Import this module only where a cluster is actually configured —
+the deterministic local boundary in ``meshnet_contracts`` stays
+dependency-free for CI.
+"""
+
+from __future__ import annotations
+
+import base64
+import json
+import os
+import urllib.request
+from dataclasses import dataclass
+
+from solders.hash import Hash
+from solders.keypair import Keypair
+from solders.pubkey import Pubkey
+from solders.system_program import CreateAccountParams, create_account
+from solders.transaction import Transaction
+from spl.token.constants import TOKEN_PROGRAM_ID
+from spl.token.instructions import (
+    create_associated_token_account,
+    get_associated_token_address,
+    initialize_mint,
+    mint_to,
+    transfer_checked,
+)
+from spl.token.models import (
+    InitializeMintParams,
+    MintToParams,
+    TransferCheckedParams,
+)
+
+USDT_DECIMALS = 6  # matches real USDT on Solana mainnet
+_MINT_ACCOUNT_SIZE = 82
+
+
+@dataclass(frozen=True)
+class Deposit:
+    """A confirmed incoming USDT transfer into the treasury token account."""
+
+    signature: str
+    sender_wallet: str
+    amount_usdt: float
+
+
+def load_keypair(path: str) -> Keypair:
+    """Load a keypair from a solana-cli style JSON byte-array file."""
+    with open(os.path.expanduser(path), encoding="utf-8") as fh:
+        secret = json.load(fh)
+    return Keypair.from_bytes(bytes(secret))
+
+
+class RpcClient:
+    """Minimal synchronous Solana JSON-RPC client."""
+
+    def __init__(self, url: str, timeout: float = 30.0) -> None:
+        self.url = url
+        self._timeout = timeout
+        self._request_id = 0
+
+    def call(self, method: str, params: list | None = None):
+        self._request_id += 1
+        body = json.dumps({
+            "jsonrpc": "2.0",
+            "id": self._request_id,
+            "method": method,
+            "params": params or [],
+        }).encode()
+        request = urllib.request.Request(
+            self.url,
+            data=body,
+            headers={"Content-Type": "application/json"},
+            method="POST",
+        )
+        with urllib.request.urlopen(request, timeout=self._timeout) as response:
+            payload = json.loads(response.read())
+        if "error" in payload:
+            raise RuntimeError(f"RPC {method} failed: {payload['error']}")
+        return payload.get("result")
+
+
+class SolanaCustodialTreasury:
+    """Read deposits into / send payouts from the custodial treasury wallet."""
+
+    def __init__(
+        self,
+        rpc_url: str,
+        usdt_mint: str,
+        treasury_keypair: Keypair | str,
+        *,
+        decimals: int = USDT_DECIMALS,
+    ) -> None:
+        self.rpc = RpcClient(rpc_url)
+        self._mint = Pubkey.from_string(usdt_mint)
+        self._treasury = (
+            load_keypair(treasury_keypair)
+            if isinstance(treasury_keypair, str)
+            else treasury_keypair
+        )
+        self._decimals = decimals
+        self._treasury_ata = get_associated_token_address(
+            self._treasury.pubkey(), self._mint
+        )
+
+    @property
+    def treasury_wallet(self) -> str:
+        return str(self._treasury.pubkey())
+
+    @property
+    def treasury_token_account(self) -> str:
+        return str(self._treasury_ata)
+
+    # ---- deposits (US-032) ----
+
+    def list_new_deposits(self, is_seen, *, limit: int = 200) -> list[Deposit]:
+        """Confirmed incoming USDT transfers whose signature is not yet seen.
+
+        ``is_seen(signature) -> bool`` lets the caller (the deposit watcher)
+        dedupe against its ledger, so replayed and re-observed transactions
+        are credited exactly once.
+        """
+        infos = self.rpc.call(
+            "getSignaturesForAddress",
+            [str(self._treasury_ata), {"limit": limit}],
+        ) or []
+        deposits: list[Deposit] = []
+        for info in infos:
+            signature = info.get("signature", "")
+            if not signature or info.get("err") is not None or is_seen(signature):
+                continue
+            deposit = self._parse_deposit(signature)
+            if deposit is not None:
+                deposits.append(deposit)
+        return deposits
+
+    def _parse_deposit(self, signature: str) -> Deposit | None:
+        result = self.rpc.call("getTransaction", [
+            signature,
+            {"encoding": "jsonParsed", "maxSupportedTransactionVersion": 0},
+        ])
+        meta = (result or {}).get("meta")
+        if not meta:
+            return None
+        pre = {b["accountIndex"]: b for b in meta.get("preTokenBalances") or []}
+        post = {b["accountIndex"]: b for b in meta.get("postTokenBalances") or []}
+
+        treasury_delta = 0
+        sender_wallet: str | None = None
+        for index, balance in post.items():
+            if balance.get("mint") != str(self._mint):
+                continue
+            before = pre.get(index)
+            before_amount = int(before["uiTokenAmount"]["amount"]) if before else 0
+            delta = int(balance["uiTokenAmount"]["amount"]) - before_amount
+            owner = balance.get("owner")
+            if owner == self.treasury_wallet and delta > 0:
+                treasury_delta = delta
+            elif delta < 0 and owner:
+                sender_wallet = owner
+        if sender_wallet is None:
+            # a token account emptied and closed may appear only in pre balances
+            for index, balance in pre.items():
+                if balance.get("mint") != str(self._mint):
+                    continue
+                if index not in post and balance.get("owner"):
+                    sender_wallet = balance["owner"]
+        if treasury_delta <= 0 or sender_wallet is None:
+            return None
+        return Deposit(
+            signature=signature,
+            sender_wallet=sender_wallet,
+            amount_usdt=treasury_delta / (10 ** self._decimals),
+        )
+
+    # ---- payouts (US-033) ----
+
+    def send_payouts(self, payouts: list[tuple[str, float]]) -> str:
+        """Send one batched transaction of USDT transfers treasury → wallets.
+
+        Creates the recipient's associated token account when missing (fee
+        paid by the treasury). Returns the transaction signature.
+        """
+        if not payouts:
+            raise ValueError("payouts must be non-empty")
+        instructions = []
+        for wallet, amount in payouts:
+            recipient = Pubkey.from_string(wallet)
+            recipient_ata = get_associated_token_address(recipient, self._mint)
+            if not self._account_exists(recipient_ata):
+                instructions.append(create_associated_token_account(
+                    payer=self._treasury.pubkey(),
+                    owner=recipient,
+                    mint=self._mint,
+                ))
+            instructions.append(transfer_checked(TransferCheckedParams(
+                program_id=TOKEN_PROGRAM_ID,
+                source=self._treasury_ata,
+                mint=self._mint,
+                dest=recipient_ata,
+                owner=self._treasury.pubkey(),
+                amount=int(round(amount * (10 ** self._decimals))),
+                decimals=self._decimals,
+            )))
+        return self.send_instructions(instructions)
+
+    # ---- shared plumbing + devnet helpers (scripts/devnet_setup.py) ----
+
+    def _account_exists(self, pubkey: Pubkey) -> bool:
+        result = self.rpc.call("getAccountInfo", [str(pubkey), {"encoding": "base64"}])
+        return bool(result and result.get("value"))
+
+    def send_instructions(self, instructions: list, extra_signers: list | None = None) -> str:
+        blockhash_result = self.rpc.call("getLatestBlockhash")
+        blockhash = Hash.from_string(blockhash_result["value"]["blockhash"])
+        transaction = Transaction.new_signed_with_payer(
+            instructions,
+            self._treasury.pubkey(),
+            [self._treasury, *(extra_signers or [])],
+            blockhash,
+        )
+        encoded = base64.b64encode(bytes(transaction)).decode()
+        return self.rpc.call("sendTransaction", [
+            encoded,
+            {"encoding": "base64", "preflightCommitment": "confirmed"},
+        ])
+
+    def get_sol_balance(self, wallet: str | None = None) -> float:
+        result = self.rpc.call("getBalance", [wallet or self.treasury_wallet])
+        return result["value"] / 1e9
+
+    def request_airdrop(self, sol: float = 2.0) -> str:
+        return self.rpc.call(
+            "requestAirdrop", [self.treasury_wallet, int(sol * 1e9)]
+        )
+
+    def create_mock_usdt_mint(self) -> tuple["SolanaCustodialTreasury", str]:
+        """Create a fresh 6-decimal mock-USDT mint (treasury = mint authority).
+
+        Returns a new adapter bound to the created mint plus its address.
+        """
+        mint_keypair = Keypair()
+        rent = self.rpc.call(
+            "getMinimumBalanceForRentExemption", [_MINT_ACCOUNT_SIZE]
+        )
+        instructions = [
+            create_account(CreateAccountParams(
+                from_pubkey=self._treasury.pubkey(),
+                to_pubkey=mint_keypair.pubkey(),
+                lamports=rent,
+                space=_MINT_ACCOUNT_SIZE,
+                owner=TOKEN_PROGRAM_ID,
+            )),
+            initialize_mint(InitializeMintParams(
+                program_id=TOKEN_PROGRAM_ID,
+                mint=mint_keypair.pubkey(),
+                decimals=self._decimals,
+                mint_authority=self._treasury.pubkey(),
+                freeze_authority=None,
+            )),
+        ]
+        self.send_instructions(instructions, extra_signers=[mint_keypair])
+        fresh = SolanaCustodialTreasury(
+            self.rpc.url,
+            str(mint_keypair.pubkey()),
+            self._treasury,
+            decimals=self._decimals,
+        )
+        return fresh, str(mint_keypair.pubkey())
+
+    def ensure_token_account(self, wallet: str) -> str:
+        owner = Pubkey.from_string(wallet)
+        ata = get_associated_token_address(owner, self._mint)
+        if not self._account_exists(ata):
+            self.send_instructions([create_associated_token_account(
+                payer=self._treasury.pubkey(), owner=owner, mint=self._mint,
+            )])
+        return str(ata)
+
+    def mint_mock_usdt(self, wallet: str, amount: float) -> str:
+        """Mint mock USDT to a wallet (devnet only — treasury is mint authority)."""
+        ata = self.ensure_token_account(wallet)
+        return self.send_instructions([mint_to(MintToParams(
+            program_id=TOKEN_PROGRAM_ID,
+            mint=self._mint,
+            dest=Pubkey.from_string(ata),
+            mint_authority=self._treasury.pubkey(),
+            amount=int(round(amount * (10 ** self._decimals))),
+        ))])

packages/contracts/pyproject.toml

@@ -7,6 +7,10 @@ name = "meshnet-contracts"
 version = "0.1.0"
 description = "Distributed Inference Network Solana smart contract wrappers"
 requires-python = ">=3.10"
+dependencies = [
+    "solana>=0.36",
+    "solders>=0.23",
+]
 
 [tool.setuptools.packages.find]
 where = ["."]

packages/gateway/meshnet_gateway/server.py

@@ -159,7 +159,9 @@ class _GatewayHandler(http.server.BaseHTTPRequestHandler):
                     "shard_coverage_percentage": float(model.get("shard_coverage_percentage", 0.0)),
                 }
                 for model in models_resp.get("data", [])
-                if isinstance(model, dict) and isinstance(model.get("id"), str)
+                if isinstance(model, dict)
+                and isinstance(model.get("id"), str)
+                and float(model.get("shard_coverage_percentage", 0.0)) > 0.0
             ]
             self._send_json(200, {"data": data})
             return
@@ -244,7 +246,7 @@ class _GatewayHandler(http.server.BaseHTTPRequestHandler):
 
     def _handle_chat_completions(self):
         server: _GatewayHTTPServer = self.server  # type: ignore[assignment]
-        # Read raw bytes first so we can proxy them if tracker-nodes are available
+        # Read raw bytes first so we can proxy them if head workers are available.
         length = int(self.headers.get("Content-Length", 0))
         raw_body = self.rfile.read(length)
         try:
@@ -257,12 +259,11 @@ class _GatewayHandler(http.server.BaseHTTPRequestHandler):
             return
 
         model = str(body.get("model", "stub-model"))
-        tracker_nodes = _get_tracker_nodes(server, model)
+        head_workers = _get_head_workers(server, model)
-        if tracker_nodes:
+        if head_workers:
-            # Proxy to a tracker-node (round-robin by request count)
+            target = head_workers[server.request_count % len(head_workers)]
-            target = tracker_nodes[server.request_count % len(tracker_nodes)]
             server.request_count += 1
-            return self._proxy_to_tracker_node(target, raw_body)
+            return self._proxy_to_head_worker(target, raw_body)
 
         # Fallback: use existing direct pipeline (backward compat)
         streaming = bool(body.get("stream", False))
@@ -284,8 +285,8 @@ class _GatewayHandler(http.server.BaseHTTPRequestHandler):
         else:
             self._send_json(200, completion)
 
-    def _proxy_to_tracker_node(self, url: str, body_bytes: bytes) -> None:
+    def _proxy_to_head_worker(self, url: str, body_bytes: bytes) -> None:
-        """Forward a raw request body to a tracker-node and stream the response back."""
+        """Forward a raw request body to a head worker and stream the response back."""
         target_url = f"{url}/v1/chat/completions"
         req = urllib.request.Request(
             target_url,
@@ -307,7 +308,7 @@ class _GatewayHandler(http.server.BaseHTTPRequestHandler):
             self.wfile.write(resp_body)
             return
         except Exception as exc:
-            self._send_json_error(503, f"tracker-node unreachable: {exc}")
+            self._send_json_error(503, f"head worker unreachable: {exc}")
             return
         self.send_response(status)
         self.send_header("Content-Type", content_type)
@@ -816,13 +817,13 @@ def _get_json(url: str, timeout: float = 5.0) -> dict:
         return json.loads(r.read())
 
 
-def _get_tracker_nodes(server: _GatewayHTTPServer, model: str) -> list[str]:
+def _get_head_workers(server: _GatewayHTTPServer, model: str) -> list[str]:
-    """Return tracker-node endpoint URLs for this model, or empty list on any failure."""
+    """Return head-worker endpoint URLs for this model, or empty list on failure."""
     if server.tracker_url is None:
         return []
     try:
-        resp = _get_json(f"{server.tracker_url}/v1/tracker-nodes/{urllib.parse.quote(model)}")
+        resp = _get_json(f"{server.tracker_url}/v1/head-workers/{urllib.parse.quote(model)}")
-        return [n["endpoint"] for n in resp.get("tracker_nodes", [])]
+        return [n["endpoint"] for n in resp.get("head_workers", [])]
     except Exception:
         return []
 

packages/node/meshnet_node/cli.py

@@ -3,6 +3,7 @@
 from __future__ import annotations
 
 import argparse
+import socket
 import sys
 import time
 from pathlib import Path
@@ -22,10 +23,15 @@ def _run_node(cfg: dict) -> None:
             model_id=cfg.get("model_hf_repo") or None,
             shard_start=cfg.get("shard_start"),
             shard_end=cfg.get("shard_end"),
-            quantization=cfg.get("quantization", "bfloat16").replace("bf16", "bfloat16"),
+            quantization=cfg.get("quantization", "int8").replace("bf16", "bfloat16"),
             wallet_path=Path(cfg["wallet_path"]) if cfg.get("wallet_path") else None,
             cache_dir=Path(cfg["download_dir"]) if cfg.get("download_dir") else None,
             host=cfg.get("host", "0.0.0.0"),
+            advertise_host=cfg.get("advertise_host"),
+            route_timeout=float(cfg.get("route_timeout", 30.0)),
+            vram_mb_override=cfg.get("vram_mb_override"),
+            max_loaded_shards=int(cfg.get("max_loaded_shards", 1)),
+            debug=bool(cfg.get("debug", False)),
         )
     except Exception as exc:
         print(f"\nERROR: {exc}", file=sys.stderr, flush=True)
@@ -48,6 +54,19 @@ def _run_node(cfg: dict) -> None:
         )
 
 
+def _first_available_port(host: str, start: int = 7000, attempts: int = 100) -> int:
+    """Return the first TCP port bindable on host, starting at start."""
+    bind_host = "" if host == "0.0.0.0" else host
+    for port in range(start, start + attempts):
+        with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
+            try:
+                sock.bind((bind_host, port))
+            except OSError:
+                continue
+            return port
+    raise OSError(f"no available TCP port in range {start}-{start + attempts - 1}")
+
+
 def _cmd_default(args) -> int:
     """No subcommand: wizard if no config, else start with saved config."""
     from .config import load_config, save_config, merge_cli_overrides
@@ -86,6 +105,16 @@ def _cmd_default(args) -> int:
         overrides["port"] = args.port
     if args.host:
         overrides["host"] = args.host
+    if args.advertise_host:
+        overrides["advertise_host"] = args.advertise_host
+    if args.route_timeout != 30.0:
+        overrides["route_timeout"] = args.route_timeout
+    if getattr(args, "memory", None) is not None:
+        overrides["vram_mb_override"] = args.memory
+    if getattr(args, "max_shards", None) is not None:
+        overrides["max_loaded_shards"] = args.max_shards
+    if args.debug:
+        overrides["debug"] = True
 
     if overrides:
         cfg = merge_cli_overrides(cfg, **overrides)
@@ -142,8 +171,12 @@ def _cmd_start(args) -> int:
     # Build a transient config from flags (don't write to disk)
     cfg = dict(DEFAULTS)
     cfg["tracker_url"] = args.tracker
-    cfg["port"] = args.port
+    cfg["port"] = args.port if args.port is not None else _first_available_port(args.host)
-    cfg["model_name"] = args.model
+    if args.model_id is None and "/" in args.model:
+        cfg["model_hf_repo"] = args.model
+        cfg["model_name"] = args.model.split("/")[-1]
+    else:
+        cfg["model_name"] = args.model
     cfg["quantization"] = args.quantization
     cfg["host"] = args.host
     if args.model_id:
@@ -173,6 +206,10 @@ def _cmd_start(args) -> int:
             cache_dir=Path(cfg["download_dir"]) if cfg.get("download_dir") else None,
             host=cfg["host"],
             advertise_host=getattr(args, "advertise_host", None),
+            route_timeout=getattr(args, "route_timeout", 30.0),
+            vram_mb_override=getattr(args, "memory", None),
+            max_loaded_shards=getattr(args, "max_shards", 1),
+            debug=getattr(args, "debug", False),
         )
     except Exception as exc:
         print(f"ERROR: {exc}", file=sys.stderr, flush=True)
@@ -212,6 +249,14 @@ def main() -> None:
     parser.add_argument("--shard-end", type=int, metavar="N", help="Pin shard end layer")
     parser.add_argument("--port", type=int, metavar="N", help="Port to listen on")
     parser.add_argument("--host", metavar="ADDR", help="Interface to bind (default 0.0.0.0)")
+    parser.add_argument("--advertise-host", metavar="ADDR", help="Host/IP advertised to the tracker")
+    parser.add_argument("--route-timeout", type=float, metavar="SEC", default=30.0,
+                        help="Seconds to wait for tracker route lookup (default 30)")
+    parser.add_argument("--memory", type=int, metavar="MB", default=None,
+                        help="Override autodetected VRAM/RAM budget in MB used for shard assignment")
+    parser.add_argument("--max-shards", type=int, metavar="N", default=None,
+                        help="Maximum shard slots this node advertises to the tracker (default 1)")
+    parser.add_argument("--debug", action="store_true", help="Enable verbose node debug logging")
     parser.add_argument("--no-tui", action="store_true", help="Plain-text output (no rich dashboard)")
     parser.add_argument("--compact", action="store_true", help="Single-line status output")
     parser.add_argument("--reset-config", action="store_true", help="Re-run wizard even if config exists")
@@ -228,18 +273,25 @@ def main() -> None:
     # start subcommand (legacy / backward-compat)
     start_cmd = subparsers.add_parser("start", help="Start node (legacy flags)")
     start_cmd.add_argument("--tracker", default="http://localhost:8080")
-    start_cmd.add_argument("--port", type=int, default=7000)
+    start_cmd.add_argument("--port", type=int)
     start_cmd.add_argument("--model", default="stub-model")
     start_cmd.add_argument("--model-id", help="HuggingFace repo ID")
     start_cmd.add_argument("--shard-start", type=int)
     start_cmd.add_argument("--shard-end", type=int)
-    start_cmd.add_argument("--quantization", choices=["bfloat16", "int8", "nf4", "bf16"], default="bfloat16")
+    start_cmd.add_argument("--quantization", choices=["bfloat16", "int8", "nf4", "bf16"], default="int8")
     start_cmd.add_argument("--host", default="0.0.0.0")
     start_cmd.add_argument("--advertise-host")
     start_cmd.add_argument("--tracker-mode", action="store_true")
     start_cmd.add_argument("--tracker-url", default=None)
     start_cmd.add_argument("--wallet")
     start_cmd.add_argument("--download-dir")
+    start_cmd.add_argument("--route-timeout", type=float, default=30.0,
+                           help="Seconds to wait for tracker route lookup (default 30)")
+    start_cmd.add_argument("--memory", type=int, default=None, metavar="MB",
+                           help="Override autodetected VRAM/RAM budget in MB used for shard assignment")
+    start_cmd.add_argument("--max-shards", type=int, default=1, metavar="N",
+                           help="Maximum shard slots this node advertises to the tracker (default 1)")
+    start_cmd.add_argument("--debug", action="store_true", help="Enable verbose node debug logging")
 
     args = parser.parse_args()
 

packages/node/meshnet_node/config.py

@@ -25,6 +25,7 @@ DEFAULTS = {
     "shard_end": None,
     "port": 7000,
     "host": "0.0.0.0",
+    "debug": False,
 }
 
 

packages/node/meshnet_node/dashboard.py

@@ -94,6 +94,19 @@ def _make_bar(pct: float, width: int = 10) -> str:
     return "█" * filled + "░" * (width - filled)
 
 
+def _node_stats(node) -> dict:
+    total = int(getattr(node, "total_requests", getattr(node, "chat_completion_count", 0)) or 0)
+    failed = int(getattr(node, "failed_requests", 0) or 0)
+    queue_depth = int(getattr(node, "queue_depth", 0) or 0)
+    success_rate = ((total - failed) / total * 100.0) if total else 100.0
+    return {
+        "total_requests": total,
+        "failed_requests": failed,
+        "queue_depth": queue_depth,
+        "success_rate": success_rate,
+    }
+
+
 def run_dashboard(node, config: dict, start_time: float) -> None:
     """Start the live dashboard. Blocks until Ctrl-C. Returns cleanly."""
     if not is_interactive_tty():
@@ -117,7 +130,8 @@ def _build_rich_renderable(
     from rich.text import Text  # type: ignore[import]
 
     uptime = time.monotonic() - start_time
-    req_count = getattr(node, "chat_completion_count", 0)
+    stats = _node_stats(node)
+    req_count = stats["total_requests"]
 
     # Tokens/sec EMA (approximate: 20 tokens per request heuristic when no real counter)
     delta_req = req_count - prev_req[0]
@@ -163,6 +177,7 @@ def _build_rich_renderable(
     stats_lines = [
         f"Tokens/sec   {tps_bar}  {tps:.1f} t/s (EMA)",
         f"Requests     {req_count:,} served",
+        f"Success      {stats['success_rate']:.1f}%  failed {stats['failed_requests']:,}  queue {stats['queue_depth']}",
         f"Peers        0 connected  (gossip: US-017)",
         f"TAI earned   0.00 TAI  (payments: US-006)",
         f"Uptime       {_format_uptime(uptime)}",
@@ -205,14 +220,17 @@ def _run_plain_loop(node, config: dict, start_time: float) -> None:
     try:
         while True:
             uptime = time.monotonic() - start_time
-            req = getattr(node, "chat_completion_count", 0)
+            stats = _node_stats(node)
+            req = stats["total_requests"]
             gpu_stats = _gpu_stats()
             vram_str = ""
             if gpu_stats:
                 g = gpu_stats[0]
                 vram_str = f"  VRAM{g['used_gb']:.1f}GB"
             print(
-                f"[{model_name}  req{req}{vram_str}  up{_format_uptime(uptime)}]",
+                f"[{model_name}  req{req}  ok{stats['success_rate']:.1f}%  "
+                f"fail{stats['failed_requests']}  q{stats['queue_depth']}"
+                f"{vram_str}  up{_format_uptime(uptime)}]",
                 flush=True,
             )
             time.sleep(2)

packages/node/meshnet_node/hardware.py

@@ -1,21 +1,112 @@
 """GPU hardware detection with graceful CPU fallback."""
 
+import json
+import os
 import subprocess
+import time
 
 
-def detect_hardware() -> dict:
+def _detect_ram_mb() -> int:
-    """Detect GPU model and available VRAM. Returns hardware profile dict."""
+    """Return host physical RAM in MB, or 0 if unavailable."""
     try:
-        import torch  # type: ignore[import]
+        pages = os.sysconf("SC_PHYS_PAGES")
-        if torch.cuda.is_available():
+        page_size = os.sysconf("SC_PAGE_SIZE")
-            idx = torch.cuda.current_device()
+        return int((pages * page_size) // (1024 * 1024))
-            name = torch.cuda.get_device_name(idx)
+    except (AttributeError, OSError, ValueError):
-            props = torch.cuda.get_device_properties(idx)
+        pass
-            vram_mb = props.total_memory // (1024 * 1024)
+    return _detect_windows_ram_mb()
-            return {"device": "cuda", "gpu_name": name, "vram_mb": vram_mb}
+
-    except ImportError:
+
+def _detect_windows_ram_mb() -> int:
+    """Return Windows physical RAM in MB, or 0."""
+    try:
+        import ctypes
+
+        class _MemoryStatusEx(ctypes.Structure):
+            _fields_ = [
+                ("dwLength", ctypes.c_ulong),
+                ("dwMemoryLoad", ctypes.c_ulong),
+                ("ullTotalPhys", ctypes.c_ulonglong),
+                ("ullAvailPhys", ctypes.c_ulonglong),
+                ("ullTotalPageFile", ctypes.c_ulonglong),
+                ("ullAvailPageFile", ctypes.c_ulonglong),
+                ("ullTotalVirtual", ctypes.c_ulonglong),
+                ("ullAvailVirtual", ctypes.c_ulonglong),
+                ("ullAvailExtendedVirtual", ctypes.c_ulonglong),
+            ]
+
+        status = _MemoryStatusEx()
+        status.dwLength = ctypes.sizeof(_MemoryStatusEx)
+        if ctypes.windll.kernel32.GlobalMemoryStatusEx(ctypes.byref(status)):
+            return int(status.ullTotalPhys // (1024 * 1024))
+    except (AttributeError, OSError, ValueError):
         pass
 
+    try:
+        result = subprocess.run(
+            [
+                "powershell",
+                "-NoProfile",
+                "-Command",
+                "(Get-CimInstance Win32_ComputerSystem).TotalPhysicalMemory",
+            ],
+            capture_output=True,
+            text=True,
+            timeout=5,
+        )
+        if result.returncode == 0 and result.stdout.strip():
+            return int(result.stdout.strip()) // (1024 * 1024)
+    except (FileNotFoundError, subprocess.TimeoutExpired, ValueError):
+        pass
+    return 0
+
+
+def _detect_windows_gpu_memory() -> dict | None:
+    """Return Windows GPU memory metadata from Win32_VideoController, if available."""
+    try:
+        result = subprocess.run(
+            [
+                "powershell",
+                "-NoProfile",
+                "-Command",
+                (
+                    "Get-CimInstance Win32_VideoController | "
+                    "Select-Object Name,AdapterRAM | ConvertTo-Json -Compress"
+                ),
+            ],
+            capture_output=True,
+            text=True,
+            timeout=5,
+        )
+    except (FileNotFoundError, subprocess.TimeoutExpired):
+        return None
+
+    if result.returncode != 0 or not result.stdout.strip():
+        return None
+    try:
+        raw = json.loads(result.stdout)
+    except json.JSONDecodeError:
+        return None
+    entries = raw if isinstance(raw, list) else [raw]
+    best: dict | None = None
+    for entry in entries:
+        if not isinstance(entry, dict):
+            continue
+        name = str(entry.get("Name") or "").strip()
+        if not name:
+            continue
+        try:
+            adapter_ram = int(entry.get("AdapterRAM") or 0)
+        except (TypeError, ValueError):
+            adapter_ram = 0
+        vram_mb = max(0, adapter_ram // (1024 * 1024))
+        if best is None or vram_mb > best["vram_mb"]:
+            best = {"gpu_name": name, "vram_mb": vram_mb}
+    return best
+
+
+def _detect_nvidia_smi_gpu_memory() -> dict | None:
+    """Return NVIDIA GPU memory metadata from nvidia-smi, if available."""
     try:
         result = subprocess.run(
             ["nvidia-smi", "--query-gpu=name,memory.total", "--format=csv,noheader,nounits"],
@@ -26,8 +117,129 @@ def detect_hardware() -> dict:
             parts = line.split(",", 1)
             gpu_name = parts[0].strip()
             vram_mb = int(parts[1].strip()) if len(parts) > 1 else 0
-            return {"device": "cuda", "gpu_name": gpu_name, "vram_mb": vram_mb}
+            return {"gpu_name": gpu_name, "vram_mb": max(0, vram_mb)}
     except (FileNotFoundError, subprocess.TimeoutExpired, ValueError, IndexError):
         pass
+    return None
 
-    return {"device": "cpu", "gpu_name": None, "vram_mb": 0}
+
+def _torch_cuda_is_executable(torch_module) -> bool:
+    """Return True only if this Python process can execute a CUDA tensor op."""
+    try:
+        if not torch_module.cuda.is_available():
+            return False
+        probe = torch_module.empty((1,), device="cuda")
+        probe += 1
+        torch_module.cuda.synchronize()
+        return True
+    except Exception:
+        return False
+
+
+def _gpu_inventory_profile(gpu: dict | None, ram_mb: int) -> dict | None:
+    if gpu is None:
+        return None
+    return {
+        "device": "cpu",
+        "gpu_name": gpu["gpu_name"],
+        "vram_mb": gpu["vram_mb"],
+        "dedicated_vram_mb": gpu["vram_mb"],
+        "shared_vram_mb": max(0, ram_mb // 2),
+        "ram_mb": ram_mb,
+        "cuda_available": False,
+    }
+
+
+def detect_hardware() -> dict:
+    """Detect GPU model and available VRAM. Returns hardware profile dict."""
+    ram_mb = _detect_ram_mb()
+    try:
+        import torch  # type: ignore[import]
+        if _torch_cuda_is_executable(torch):
+            idx = torch.cuda.current_device()
+            name = torch.cuda.get_device_name(idx)
+            props = torch.cuda.get_device_properties(idx)
+            vram_mb = props.total_memory // (1024 * 1024)
+            shared_vram_mb = max(0, ram_mb // 2)
+            return {
+                "device": "cuda",
+                "gpu_name": name,
+                "vram_mb": vram_mb,
+                "dedicated_vram_mb": vram_mb,
+                "shared_vram_mb": shared_vram_mb,
+                "ram_mb": ram_mb,
+                "cuda_available": True,
+            }
+    except ImportError:
+        pass
+
+    nvidia_gpu = _gpu_inventory_profile(_detect_nvidia_smi_gpu_memory(), ram_mb)
+    if nvidia_gpu is not None:
+        return nvidia_gpu
+
+    windows_gpu = _gpu_inventory_profile(_detect_windows_gpu_memory(), ram_mb)
+    if windows_gpu is not None:
+        return windows_gpu
+
+    return {
+        "device": "cpu",
+        "gpu_name": None,
+        "vram_mb": 0,
+        "dedicated_vram_mb": 0,
+        "shared_vram_mb": 0,
+        "ram_mb": ram_mb,
+        "cuda_available": False,
+    }
+
+
+def benchmark_throughput_checked(device_str: str = "cpu") -> tuple[float, bool, str | None]:
+    """
+    Estimate compute throughput via a synthetic transformer GEMM benchmark.
+
+    Runs hidden_size × (hidden_size*4) matmul — the dominant op in FFN layers —
+    and returns iterations/second as a relative speed index.  Higher = faster.
+    The value is used as benchmark_tokens_per_sec in tracker registration for
+    routing tiebreaks; it is not an absolute token rate.
+
+    Returns (score, ok, error). Score falls back to 1.0 when the requested
+    device cannot run the benchmark.
+    """
+    try:
+        import torch  # type: ignore[import]
+
+        device = torch.device(device_str)
+        # bfloat16 on CUDA matches real inference dtype; float32 on CPU avoids
+        # precision-downcast surprises on older hardware without bfloat16 support.
+        dtype = torch.bfloat16 if device_str == "cuda" else torch.float32
+
+        # hidden_size=2048 is representative of a mid-sized model; large enough
+        # that BLAS finds an efficient kernel on both GPU and CPU.
+        hidden_size = 2048
+        a = torch.randn(1, hidden_size, dtype=dtype, device=device)
+        b = torch.randn(hidden_size, hidden_size * 4, dtype=dtype, device=device)
+
+        def _sync() -> None:
+            if device_str == "cuda":
+                torch.cuda.synchronize()
+
+        # Warmup: prime caches and JIT compilation.
+        for _ in range(10):
+            torch.matmul(a, b)
+        _sync()
+
+        n_iters = 50
+        t0 = time.perf_counter()
+        for _ in range(n_iters):
+            torch.matmul(a, b)
+        _sync()
+        elapsed = time.perf_counter() - t0
+
+        return round(n_iters / max(elapsed, 1e-9), 2), True, None
+    except Exception as exc:
+        return 1.0, False, f"{type(exc).__name__}: {exc}"
+
+
+def benchmark_throughput(device_str: str = "cpu") -> float:
+    """Return only the numeric throughput index, preserving the legacy API."""
+    score, _ok, _error = benchmark_throughput_checked(device_str)
+    return score

packages/node/meshnet_node/model_backend.py

@@ -4,6 +4,7 @@ from __future__ import annotations
 
 import base64
 from dataclasses import dataclass
+from pathlib import Path
 from typing import Any, Literal
 
 Quantization = Literal["bfloat16", "int8", "nf4"]
@@ -65,6 +66,7 @@ class TorchModelShard:
         shard_start: int,
         shard_end: int,
         quantization: Quantization = "bfloat16",
+        cache_dir: Path | None = None,
     ) -> None:
         if shard_start < 0 or shard_end < 0 or shard_start > shard_end:
             raise ValueError("shard_start must be <= shard_end and non-negative")
@@ -89,9 +91,9 @@ class TorchModelShard:
                 model_id,
                 quantization_config=quant_config,
                 device_map="auto" if quant_config is not None else None,
-                torch_dtype=torch.bfloat16,
+                dtype=torch.bfloat16,
                 low_cpu_mem_usage=True,
-                use_safetensors=True,
+                cache_dir=str(cache_dir) if cache_dir is not None else None,
             )
             if quant_config is None:
                 self.model.to(self.device)
@@ -104,15 +106,19 @@ class TorchModelShard:
             raise
 
         self.model.eval()
-        self.tokenizer = AutoTokenizer.from_pretrained(model_id)
+        self.tokenizer = AutoTokenizer.from_pretrained(
+            model_id,
+            cache_dir=str(cache_dir) if cache_dir is not None else None,
+        )
         self.layers = _model_layers(self.model)
         self.total_layers = len(self.layers)
-        if shard_end > self.total_layers:
+        # shard_end is INCLUSIVE (last layer index, 0-based), matching the CLI convention.
+        if shard_end >= self.total_layers:
             raise ValueError(
-                f"shard_end {shard_end} exceeds total layer count {self.total_layers}"
+                f"shard_end {shard_end} exceeds last layer index {self.total_layers - 1}"
             )
         self.is_head = shard_start == 0
-        self.is_tail = shard_end == self.total_layers
+        self.is_tail = shard_end >= self.total_layers - 1
         self.hidden_size = int(
             getattr(self.model.config, "hidden_size", 0)
             or getattr(self.model.config, "n_embd", 0)
@@ -144,6 +150,7 @@ class TorchModelShard:
         shape: list[int],
         attention_mask_header: str | None,
         position_ids_header: str | None,
+        start_layer: int | None = None,
     ) -> TensorPayload | str:
         hidden_states = _tensor_from_bfloat16_bytes(body, shape, self.torch).to(
             self.device
@@ -154,7 +161,9 @@ class TorchModelShard:
         position_ids = _tensor_from_int64_header(
             position_ids_header, self.torch, self.device
         )
-        hidden_states = self._run_layers(hidden_states, attention_mask, position_ids)
+        hidden_states = self._run_layers(
+            hidden_states, attention_mask, position_ids, start_layer=start_layer
+        )
         if self.is_tail:
             return self.decode_tail(hidden_states)
         return self._payload(hidden_states, attention_mask, position_ids)
@@ -168,10 +177,149 @@ class TorchModelShard:
         token_id = int(self.torch.argmax(logits[:, -1, :], dim=-1)[0].item())
         return self.tokenizer.decode([token_id], skip_special_tokens=True)
 
-    def _run_layers(self, hidden_states: Any, attention_mask: Any, position_ids: Any) -> Any:
+    def generate_text(
+        self,
+        messages: list[dict],
+        max_new_tokens: int = 256,
+        temperature: float = 1.0,
+        top_p: float = 1.0,
+    ) -> str:
+        """Autoregressive generation using HF generate() — single-node (head+tail) mode."""
+        if not self.is_head or not self.is_tail:
+            raise ModelBackendError("local generation requires a full head+tail shard")
+        encoded = self._encode_messages(messages)
+        input_ids = encoded["input_ids"].to(self.device)
+        attention_mask = encoded.get("attention_mask")
+        if attention_mask is not None:
+            attention_mask = attention_mask.to(self.device)
+        pad_token_id = getattr(self.tokenizer, "pad_token_id", None) or getattr(self.tokenizer, "eos_token_id", None)
+        do_sample = temperature != 1.0 or top_p != 1.0
         with self.torch.inference_mode():
-            for layer in self.layers[self.shard_start:self.shard_end]:
+            generated = self.model.generate(
-                hidden_states = _call_layer(layer, hidden_states, attention_mask, position_ids)
+                input_ids=input_ids,
+                attention_mask=attention_mask,
+                max_new_tokens=max(1, int(max_new_tokens)),
+                do_sample=do_sample,
+                temperature=temperature if do_sample else None,
+                top_p=top_p if do_sample else None,
+                pad_token_id=pad_token_id,
+            )
+        new_tokens = generated[0, input_ids.shape[-1]:]
+        return self.tokenizer.decode(new_tokens, skip_special_tokens=True).strip()
+
+    def generate_text_streaming(
+        self,
+        messages: list[dict],
+        max_new_tokens: int = 256,
+        temperature: float = 1.0,
+        top_p: float = 1.0,
+    ):
+        """Yield decoded token strings one at a time using HF TextIteratorStreamer."""
+        if not self.is_head or not self.is_tail:
+            raise ModelBackendError("streaming generation requires a full head+tail shard")
+        import threading
+        try:
+            from transformers import TextIteratorStreamer  # type: ignore[import]
+        except ImportError:
+            yield self.generate_text(messages, max_new_tokens, temperature, top_p)
+            return
+
+        encoded = self._encode_messages(messages)
+        input_ids = encoded["input_ids"].to(self.device)
+        attention_mask = encoded.get("attention_mask")
+        if attention_mask is not None:
+            attention_mask = attention_mask.to(self.device)
+        pad_token_id = getattr(self.tokenizer, "pad_token_id", None) or getattr(self.tokenizer, "eos_token_id", None)
+        do_sample = temperature != 1.0 or top_p != 1.0
+
+        streamer = TextIteratorStreamer(self.tokenizer, skip_prompt=True, skip_special_tokens=True)
+        gen_kwargs = dict(
+            input_ids=input_ids,
+            attention_mask=attention_mask,
+            max_new_tokens=max(1, int(max_new_tokens)),
+            do_sample=do_sample,
+            temperature=temperature if do_sample else None,
+            top_p=top_p if do_sample else None,
+            pad_token_id=pad_token_id,
+            streamer=streamer,
+        )
+        t = threading.Thread(target=self.model.generate, kwargs=gen_kwargs, daemon=True)
+        t.start()
+        for token_text in streamer:
+            yield token_text
+        t.join()
+
+    def count_prompt_tokens(self, messages: list[dict]) -> int:
+        """Return tokenizer-backed prompt token count for OpenAI usage metadata."""
+        encoded = self._encode_messages(messages)
+        input_ids = encoded["input_ids"]
+        return int(input_ids.shape[-1])
+
+    def count_text_tokens(self, text: str) -> int:
+        """Return tokenizer-backed completion token count for OpenAI usage metadata."""
+        try:
+            encoded = self.tokenizer(
+                text,
+                return_tensors="pt",
+                add_special_tokens=False,
+            )
+        except TypeError:
+            encoded = self.tokenizer(text, return_tensors="pt")
+        return int(encoded["input_ids"].shape[-1])
+
+    def _encode_messages(self, messages: list[dict]) -> dict:
+        """Format messages with chat template (if available) and tokenize."""
+        if hasattr(self.tokenizer, "apply_chat_template"):
+            try:
+                prompt_str = self.tokenizer.apply_chat_template(
+                    messages,
+                    add_generation_prompt=True,
+                    tokenize=False,
+                )
+                return dict(self.tokenizer(prompt_str, return_tensors="pt"))
+            except Exception:
+                pass
+        prompt = " ".join(
+            str(m.get("content", ""))
+            for m in messages
+            if isinstance(m, dict) and m.get("role") == "user"
+        )
+        return dict(self.tokenizer(prompt, return_tensors="pt"))
+
+    def _run_layers(
+        self,
+        hidden_states: Any,
+        attention_mask: Any,
+        position_ids: Any,
+        start_layer: int | None = None,
+    ) -> Any:
+        # start_layer overrides shard_start for overlapping-shard routing
+        # (X-Meshnet-Start-Layer header). Clamped to shard_start to prevent
+        # indexing outside the loaded weights.
+        effective_start = (
+            max(self.shard_start, start_layer)
+            if start_layer is not None
+            else self.shard_start
+        )
+        position_embeddings = _rotary_position_embeddings(
+            self.model,
+            hidden_states,
+            position_ids,
+        )
+        layer_attention_mask = _decoder_attention_mask(
+            attention_mask,
+            hidden_states,
+            self.torch,
+        )
+        with self.torch.inference_mode():
+            for layer in self.layers[effective_start:self.shard_end + 1]:
+                hidden_states = _call_layer(
+                    layer,
+                    hidden_states,
+                    layer_attention_mask,
+                    position_ids,
+                    position_embeddings,
+                )
             return hidden_states.to(self.torch.bfloat16)
 
     def _payload(self, hidden_states: Any, attention_mask: Any, position_ids: Any) -> TensorPayload:
@@ -193,8 +341,9 @@ def load_torch_shard(
     shard_start: int,
     shard_end: int,
     quantization: Quantization = "bfloat16",
+    cache_dir: Path | None = None,
 ) -> TorchModelShard:
-    return TorchModelShard(model_id, shard_start, shard_end, quantization)
+    return TorchModelShard(model_id, shard_start, shard_end, quantization, cache_dir)
 
 
 def _model_layers(model: Any) -> Any:
@@ -236,8 +385,60 @@ def _position_ids(attention_mask: Any, torch: Any) -> Any:
     return position_ids.masked_fill(attention_mask == 0, 0).to(torch.long)
 
 
-def _call_layer(layer: Any, hidden_states: Any, attention_mask: Any, position_ids: Any) -> Any:
+def _decoder_attention_mask(attention_mask: Any, hidden_states: Any, torch: Any) -> Any:
+    """Build a causal additive mask for decoder layers called outside model.forward."""
+    if attention_mask is None:
+        return None
+    if len(getattr(attention_mask, "shape", ())) != 2:
+        return attention_mask
+    batch_size, seq_len = attention_mask.shape
+    if seq_len <= 1:
+        return None if bool(attention_mask.all()) else attention_mask.to(hidden_states.dtype)
+
+    min_value = torch.finfo(hidden_states.dtype).min
+    causal = torch.full(
+        (seq_len, seq_len),
+        min_value,
+        dtype=hidden_states.dtype,
+        device=hidden_states.device,
+    )
+    causal = torch.triu(causal, diagonal=1)
+    causal = causal[None, None, :, :].expand(batch_size, 1, seq_len, seq_len).clone()
+
+    padding = attention_mask.to(device=hidden_states.device)
+    if not bool(padding.all()):
+        causal = causal.masked_fill(padding[:, None, None, :] == 0, min_value)
+    return causal
+
+
+def _rotary_position_embeddings(model: Any, hidden_states: Any, position_ids: Any) -> Any | None:
+    """Return model-level rotary embeddings required by newer HF decoder layers."""
+    if position_ids is None:
+        return None
+    rotary = None
+    if hasattr(model, "model") and hasattr(model.model, "rotary_emb"):
+        rotary = model.model.rotary_emb
+    elif hasattr(model, "transformer") and hasattr(model.transformer, "rotary_emb"):
+        rotary = model.transformer.rotary_emb
+    if rotary is None:
+        return None
+    return rotary(hidden_states, position_ids)
+
+
+def _call_layer(
+    layer: Any,
+    hidden_states: Any,
+    attention_mask: Any,
+    position_ids: Any,
+    position_embeddings: Any | None = None,
+) -> Any:
     attempts = (
+        {
+            "attention_mask": attention_mask,
+            "position_ids": position_ids,
+            "position_embeddings": position_embeddings,
+            "use_cache": False,
+        },
         {
             "attention_mask": attention_mask,
             "position_ids": position_ids,
@@ -272,7 +473,7 @@ def _tensor_from_bfloat16_bytes(body: bytes, shape: list[int], torch: Any) -> An
 
 
 def _int_tensor_header(tensor: Any) -> str:
-    data = tensor.detach().cpu().to(tensor.int64).contiguous()
+    data = tensor.detach().cpu().long().contiguous()
     raw = data.numpy().tobytes()
     shape = ",".join(str(dim) for dim in data.shape)
     encoded = base64.b64encode(raw).decode("ascii")

packages/node/meshnet_node/model_catalog.py

@@ -2,7 +2,10 @@
 
 from __future__ import annotations
 
+import json
 from dataclasses import dataclass
+from importlib.resources import files
+from pathlib import Path
 
 
 @dataclass
@@ -15,6 +18,7 @@ class ModelPreset:
     vram_int8: float
     vram_bf16: float
     description: str
+    metadata: dict | None = None
 
     def vram_for_quant(self, quant: str) -> float:
         """Return VRAM requirement in GB for the given quantization."""
@@ -41,6 +45,25 @@ class ModelPreset:
         return None
 
 
+def _load_model_metadata() -> dict[str, dict]:
+    try:
+        raw = files("meshnet_node").joinpath("model_metadata.json").read_text()
+        data = json.loads(raw)
+    except Exception:
+        return {}
+    models = data.get("models", {})
+    if not isinstance(models, dict):
+        return {}
+    return {
+        str(repo): metadata
+        for repo, metadata in models.items()
+        if isinstance(metadata, dict)
+    }
+
+
+_MODEL_METADATA = _load_model_metadata()
+
+
 CURATED_MODELS: list[ModelPreset] = [
     ModelPreset(
         name="Qwen2.5-0.5B-Instruct",
@@ -123,6 +146,16 @@ CURATED_MODELS: list[ModelPreset] = [
         vram_bf16=16.0,
         description="DeepSeek's efficient MoE — strong coding + reasoning",
     ),
+    ModelPreset(
+        name="Kimi-K2.7-Code",
+        hf_repo="unsloth/Kimi-K2.7-Code",
+        num_layers=61,
+        vram_nf4=500.0,
+        vram_int8=1000.0,
+        vram_bf16=2000.0,
+        description="Large coding-focused MoE model",
+        metadata=_MODEL_METADATA.get("unsloth/Kimi-K2.7-Code"),
+    ),
 ]
 
 
@@ -140,6 +173,44 @@ def detect_num_layers(hf_repo: str) -> int | None:
         return None
 
 
+def model_metadata_for(
+    hf_repo: str,
+    num_layers: int | None = None,
+    cache_dir: Path | None = None,
+) -> dict:
+    """Return operator-facing model metadata for a HuggingFace repo."""
+    for model in CURATED_MODELS:
+        if model.hf_repo == hf_repo:
+            metadata = dict(model.metadata or {})
+            metadata.setdefault("num_layers", model.num_layers)
+            return metadata
+
+    metadata: dict = {}
+    if num_layers is not None:
+        metadata["num_layers"] = num_layers
+    try:
+        from transformers import AutoConfig  # type: ignore[import]
+
+        cfg = AutoConfig.from_pretrained(
+            hf_repo,
+            cache_dir=str(cache_dir) if cache_dir is not None else None,
+        )
+        for attr, key in (
+            ("model_type", "architecture"),
+            ("num_hidden_layers", "num_layers"),
+            ("hidden_size", "hidden_size"),
+            ("num_attention_heads", "attention_heads"),
+            ("vocab_size", "vocabulary_size"),
+            ("max_position_embeddings", "context_length"),
+        ):
+            value = getattr(cfg, attr, None)
+            if value is not None:
+                metadata[key] = value
+    except Exception:
+        pass
+    return metadata
+
+
 def browse_hf_hub(top_n: int = 20) -> list[dict]:
     """Fetch top downloaded text-generation models from HuggingFace Hub."""
     try:

packages/node/meshnet_node/model_metadata.json

@@ -0,0 +1,32 @@
+{
+  "models": {
+    "unsloth/Kimi-K2.7-Code": {
+      "architecture": "Mixture-of-Experts (MoE)",
+      "total_parameters": "1T",
+      "activated_parameters": "32B",
+      "num_layers": 61,
+      "dense_layers": 1,
+      "attention_hidden_dimension": 7168,
+      "moe_hidden_dimension_per_expert": 2048,
+      "attention_heads": 64,
+      "experts": 384,
+      "selected_experts_per_token": 8,
+      "shared_experts": 1,
+      "vocabulary_size": 160000,
+      "context_length": 256000,
+      "attention_mechanism": "MLA",
+      "activation_function": "SwiGLU",
+      "vision_encoder": "MoonViT",
+      "vision_encoder_parameters": "400M",
+      "license": "modified-mit",
+      "native_quantization": "int4",
+      "download_size_gb": 595,
+      "recommended_short_name": "kimi-k2.7",
+      "recommended_engines": [
+        "vLLM",
+        "SGLang",
+        "KTransformers"
+      ]
+    }
+  }
+}

packages/node/meshnet_node/relay_bridge.py

@@ -0,0 +1,164 @@
+"""Outbound relay bridge for NAT-safe node HTTP requests."""
+
+from __future__ import annotations
+
+import base64
+import json
+import logging
+import threading
+import time
+import urllib.error
+import urllib.request
+from dataclasses import dataclass
+
+log = logging.getLogger(__name__)
+
+
+@dataclass(frozen=True)
+class RelayBridgeInfo:
+    peer_id: str
+    relay_addr: str
+
+
+def _make_envelope(topic: str, payload: dict, peer_id: str) -> dict:
+    return {
+        "topic": topic,
+        "version": 1,
+        "from_peer": peer_id,
+        "timestamp": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
+        "msg_id": f"{peer_id}-{time.time_ns():x}",
+        "ttl": 1,
+        "payload": payload,
+    }
+
+
+class RelayHttpBridge:
+    """Connect outbound to a relay and proxy relay HTTP requests to localhost."""
+
+    def __init__(
+        self,
+        relay_url: str,
+        peer_id: str,
+        local_base_url: str,
+        advertised_addr: str,
+        reconnect_interval: float = 3.0,
+    ) -> None:
+        self.relay_url = relay_url.rstrip("/")
+        self.peer_id = peer_id
+        self.local_base_url = local_base_url.rstrip("/")
+        self.advertised_addr = advertised_addr
+        self.reconnect_interval = reconnect_interval
+        self._running = False
+        self._thread: threading.Thread | None = None
+        self._connected = threading.Event()
+
+    @property
+    def relay_addr(self) -> str:
+        base = self.relay_url
+        if base.endswith("/ws"):
+            base = base[:-3]
+        return f"{base}/rpc/{self.peer_id}"
+
+    def start(self) -> RelayBridgeInfo:
+        self._running = True
+        self._thread = threading.Thread(target=self._run, daemon=True, name="relay-http-bridge")
+        self._thread.start()
+        return RelayBridgeInfo(peer_id=self.peer_id, relay_addr=self.relay_addr)
+
+    def wait_connected(self, timeout: float = 5.0) -> bool:
+        return self._connected.wait(timeout)
+
+    def stop(self) -> None:
+        self._running = False
+        if self._thread:
+            self._thread.join(timeout=3.0)
+
+    def _run(self) -> None:
+        import websockets.sync.client as wsc  # type: ignore[import]
+
+        while self._running:
+            try:
+                with wsc.connect(self.relay_url, open_timeout=5) as ws:
+                    self._connected.set()
+                    ws.send(json.dumps(_make_envelope(
+                        "peer-register",
+                        {"peer_id": self.peer_id, "addr": self.advertised_addr},
+                        self.peer_id,
+                    )))
+                    while self._running:
+                        try:
+                            raw = ws.recv(timeout=1)
+                        except TimeoutError:
+                            continue
+                        try:
+                            envelope = json.loads(raw)
+                        except (TypeError, json.JSONDecodeError):
+                            continue
+                        if envelope.get("topic") != "relay-http-request":
+                            continue
+                        payload = envelope.get("payload", {})
+                        if payload.get("target_peer") not in {None, self.peer_id}:
+                            continue
+                        response = self._handle_request(payload)
+                        ws.send(json.dumps(_make_envelope(
+                            "relay-http-response",
+                            response,
+                            self.peer_id,
+                        )))
+            except Exception as exc:
+                self._connected.clear()
+                if self._running:
+                    log.debug("relay bridge disconnected: %s", exc)
+                    time.sleep(self.reconnect_interval)
+
+    def _handle_request(self, payload: dict) -> dict:
+        request_id = str(payload.get("request_id") or "")
+        method = str(payload.get("method") or "POST").upper()
+        path = str(payload.get("path") or "/")
+        headers = payload.get("headers") if isinstance(payload.get("headers"), dict) else {}
+
+        # body_base64 carries binary data (e.g. bfloat16 activation tensors) safely.
+        # Fallback to text "body" for backward-compat with non-binary requests.
+        body_b64 = payload.get("body_base64")
+        if body_b64:
+            data = base64.b64decode(body_b64)
+        else:
+            body_text = payload.get("body") or ""
+            data = body_text.encode() if isinstance(body_text, str) else bytes(body_text)
+
+        url = f"{self.local_base_url}{path}"
+        req = urllib.request.Request(url, data=data, headers=headers, method=method)
+        try:
+            with urllib.request.urlopen(req, timeout=300.0) as resp:
+                resp_bytes = resp.read()
+                resp_headers = dict(resp.headers)
+                # Forward all X-Meshnet-* headers so the caller can reconstruct the activation.
+                is_binary = "octet-stream" in resp.headers.get("Content-Type", "")
+                result: dict = {
+                    "request_id": request_id,
+                    "status": resp.status,
+                    "headers": resp_headers,
+                }
+                if is_binary:
+                    result["body_base64"] = base64.b64encode(resp_bytes).decode()
+                else:
+                    result["body"] = resp_bytes.decode(errors="replace")
+                return result
+        except urllib.error.HTTPError as exc:
+            return {
+                "request_id": request_id,
+                "status": exc.code,
+                "headers": {"Content-Type": exc.headers.get("Content-Type", "application/json")},
+                "body": exc.read().decode(errors="replace"),
+            }
+        except Exception as exc:
+            return {
+                "request_id": request_id,
+                "status": 503,
+                "headers": {"Content-Type": "application/json"},
+                "body": json.dumps({"error": f"relay bridge local request failed: {exc}"}),
+            }
+
+
+def peer_id_from_wallet(wallet_address: str) -> str:
+    return wallet_address[:16] if len(wallet_address) >= 16 else wallet_address

packages/node/meshnet_node/startup.py

@@ -5,6 +5,8 @@ from __future__ import annotations
 import json
 import socket
 import sys
+import threading
+import time
 import urllib.error
 import urllib.parse
 import urllib.request
@@ -12,12 +14,58 @@ from pathlib import Path
 from typing import Any
 
 from .downloader import compute_shard_checksum, download_shard
-from .hardware import detect_hardware
+from .hardware import detect_hardware, benchmark_throughput_checked
+from .model_catalog import model_metadata_for
+from .relay_bridge import RelayHttpBridge, peer_id_from_wallet
 from .server import StubNodeServer
 from .torch_server import TorchNodeServer
 from .wallet import load_or_create_wallet
 
 
+_DEFAULT_BYTES_PER_LAYER = 30 * 1024 * 1024
+
+
+def _memory_budget(device: str, vram_mb: int, ram_mb: int, shared_vram_mb: int = 0) -> tuple[int, str]:
+    """Return the capacity budget in MB and whether it came from VRAM or RAM."""
+    if device == "cuda" and vram_mb > 0:
+        if shared_vram_mb > 0:
+            return vram_mb + shared_vram_mb, "VRAM + shared RAM"
+        return vram_mb, "VRAM"
+    return max(0, ram_mb), "RAM"
+
+
+def _hardware_label(device: str, gpu_name: str | None = None) -> str:
+    if device == "cuda":
+        return "CUDA"
+    if gpu_name:
+        return "CPU (CUDA inactive)"
+    return "CPU"
+
+
+def _max_assignable_layers(memory_mb: int, total_layers: int | None) -> int:
+    if total_layers is None or total_layers <= 0 or memory_mb <= 0:
+        return 0
+    budget_bytes = memory_mb * 1024 * 1024
+    return min(total_layers, int((budget_bytes * 0.8) // _DEFAULT_BYTES_PER_LAYER))
+
+
+def _shard_budget_line(memory_mb: int, memory_source: str, total_layers: int | None, quantization: str) -> str:
+    memory_gb = memory_mb / 1024
+    gb_str = f"{memory_gb:.1f} GB"
+    if total_layers is None or total_layers <= 0:
+        return f"Memory budget: {gb_str} {memory_source}; shard budget: unknown model layer count"
+    max_layers = _max_assignable_layers(memory_mb, total_layers)
+    # Remaining capacity after one full model load (rough estimate)
+    shard_bytes = max_layers * _DEFAULT_BYTES_PER_LAYER
+    remaining_gb = (memory_mb * 1024 * 1024 - shard_bytes) / (1024 ** 3)
+    remaining_str = f"; {remaining_gb:.1f} GB remaining after full load" if remaining_gb > 1 else ""
+    return (
+        f"Memory budget: {gb_str} {memory_source}; "
+        f"Shard budget: up to {max_layers}/{total_layers} layers at {quantization}"
+        f"{remaining_str}"
+    )
+
+
 def _post_json(url: str, payload: dict, timeout: float = 10.0) -> dict:
     data = json.dumps(payload).encode()
     req = urllib.request.Request(
@@ -32,6 +80,225 @@ def _get_json(url: str, timeout: float = 10.0) -> dict:
         return json.loads(r.read())
 
 
+def _infer_relay_url_from_tracker(tracker_url: str) -> str | None:
+    """Infer relay WebSocket URL from a public HTTPS tracker origin.
+
+    Public deployments colocate relay at /ws on the same host as the tracker API
+    (see QUICKSTART nginx layout).  Local LAN trackers use a separate relay port
+    and must advertise relay_url explicitly via /v1/network/map.
+    """
+    parsed = urllib.parse.urlparse(tracker_url)
+    if parsed.scheme != "https":
+        return None
+    host = parsed.hostname
+    if not host or host in ("127.0.0.1", "localhost"):
+        return None
+    return f"wss://{parsed.netloc}/ws"
+
+
+def _discover_relay_url(tracker_url: str) -> str | None:
+    relay_url: str | None = None
+    try:
+        network_map = _get_json(f"{tracker_url}/v1/network/map", timeout=5.0)
+        raw = network_map.get("relay_url")
+        if isinstance(raw, str) and raw:
+            relay_url = raw
+    except Exception:
+        pass
+    return relay_url or _infer_relay_url_from_tracker(tracker_url)
+
+
+def _start_relay_bridge_if_available(
+    tracker_url: str,
+    wallet_address: str,
+    local_base_url: str,
+    advertised_endpoint: str,
+    relay_url: str | None = None,
+) -> tuple[RelayHttpBridge | None, dict]:
+    relay_url = relay_url or _discover_relay_url(tracker_url)
+    if not relay_url:
+        return None, {}
+    peer_id = peer_id_from_wallet(wallet_address)
+    bridge = RelayHttpBridge(
+        relay_url=relay_url,
+        peer_id=peer_id,
+        local_base_url=local_base_url,
+        advertised_addr=advertised_endpoint,
+    )
+    info = bridge.start()
+    if bridge.wait_connected(timeout=5.0):
+        print(f"  Relay connected — {info.relay_addr}", flush=True)
+    else:
+        print(f"  Relay configured but not connected yet — {info.relay_addr}", flush=True)
+    return bridge, {
+        "relay_addr": info.relay_addr,
+        "peer_id": info.peer_id,
+    }
+
+
+def _attach_relay_bridge(node: StubNodeServer | TorchNodeServer, bridge: RelayHttpBridge | None) -> None:
+    setattr(node, "relay_bridge", bridge)
+    if bridge is None:
+        return
+    original_stop = node.stop
+
+    def _stop_with_bridge() -> None:
+        try:
+            bridge.stop()
+        finally:
+            original_stop()
+
+    node.stop = _stop_with_bridge  # type: ignore[method-assign]
+
+
+def _start_heartbeat(
+    tracker_url: str,
+    node_id: str,
+    register_payload: dict,
+    interval: float = 20.0,
+    node_ref: Any | None = None,
+    start_time: float | None = None,
+) -> threading.Thread:
+    """Daemon thread: sends heartbeats and re-registers automatically after tracker restarts.
+
+    Heartbeat body carries cumulative stats (total_requests, failed_requests,
+    queue_depth, uptime_seconds, status). Stats are buffered locally during
+    outage and flushed on next successful heartbeat.
+
+    Heartbeat response may include new_assignment: {model, shard_start, shard_end}
+    which is logged for now (hot-reload implemented in US-026).
+    """
+    _start_time = start_time or time.monotonic()
+
+    def _get_stats() -> dict:
+        uptime = time.monotonic() - _start_time
+        stats: dict = {"uptime_seconds": round(uptime, 1), "status": "ready"}
+        if node_ref is not None:
+            stats["total_requests"] = getattr(
+                node_ref,
+                "total_requests",
+                getattr(node_ref, "chat_completion_count", 0),
+            )
+            stats["failed_requests"] = getattr(node_ref, "failed_requests", 0)
+            stats["queue_depth"] = getattr(node_ref, "queue_depth", 0)
+        return stats
+
+    def _reregister() -> bool:
+        nonlocal node_id
+        try:
+            resp = _post_json(f"{tracker_url}/v1/nodes/register", register_payload)
+            node_id = resp.get("node_id", node_id)
+            return True
+        except Exception:
+            return False
+
+    def _apply_directives(directives: list[dict]) -> None:
+        if not directives:
+            return
+        if node_ref is None or not hasattr(node_ref, "apply_tracker_directives"):
+            print(f"  [node] tracker directives received: {directives}", flush=True)
+            return
+        try:
+            applied = node_ref.apply_tracker_directives(directives)
+        except Exception as exc:
+            print(f"  [node] WARNING: failed to apply tracker directives: {exc}", flush=True)
+            return
+        if applied:
+            model_id = applied.get("model", register_payload.get("hf_repo") or register_payload.get("model"))
+            register_payload["model"] = str(model_id).split("/")[-1]
+            register_payload["hf_repo"] = model_id
+            register_payload["shard_start"] = applied["shard_start"]
+            register_payload["shard_end"] = applied["shard_end"]
+            register_payload["quantization"] = applied.get("quantization", register_payload.get("quantization"))
+            register_payload["tracker_mode"] = bool(applied.get("tracker_mode", False))
+
+    def _loop() -> None:
+        nonlocal node_id
+        hb_url = f"{tracker_url}/v1/nodes/{node_id}/heartbeat"
+        outage_streak = 0  # consecutive intervals where tracker was unreachable
+
+        while True:
+            time.sleep(interval)
+
+            if outage_streak > 0:
+                # Tracker was down — attempt re-registration first (it may have restarted
+                # with a clean slate and won't know this node).
+                if _reregister():
+                    hb_url = f"{tracker_url}/v1/nodes/{node_id}/heartbeat"
+                    print(f"  [node] re-registered after outage — node ID: {node_id}", flush=True)
+                    outage_streak = 0
+                else:
+                    outage_streak += 1
+                    if outage_streak <= 3 or outage_streak % 10 == 0:
+                        print(
+                            f"  [node] WARNING: tracker still unreachable "
+                            f"({outage_streak * interval:.0f}s)",
+                            flush=True,
+                        )
+                continue
+
+            try:
+                resp = _post_json(hb_url, _get_stats())
+                _apply_directives(resp.get("directives", []))
+                new_asgn = resp.get("new_assignment")
+                if new_asgn:
+                    print(
+                        f"  [node] tracker reassignment received: "
+                        f"model={new_asgn.get('model')!r} "
+                        f"shards={new_asgn.get('shard_start')}-{new_asgn.get('shard_end')}",
+                        flush=True,
+                    )
+            except urllib.error.HTTPError as exc:
+                if exc.code == 404:
+                    # Node was purged (e.g. long gap before restart noticed) — re-register now.
+                    print("  [node] tracker lost registration — re-registering...", flush=True)
+                    if _reregister():
+                        hb_url = f"{tracker_url}/v1/nodes/{node_id}/heartbeat"
+                        print(f"  [node] re-registered — node ID: {node_id}", flush=True)
+                    else:
+                        print("  [node] WARNING: re-registration failed", flush=True)
+                        outage_streak = 1
+                else:
+                    print(f"  [node] WARNING: heartbeat failed ({exc.code}): {exc}", flush=True)
+            except Exception as exc:
+                outage_streak = 1
+                print(f"  [node] WARNING: tracker unreachable: {exc}", flush=True)
+
+    t = threading.Thread(target=_loop, daemon=True, name="heartbeat")
+    t.start()
+    return t
+
+
+def _warn_virtual_network_ip(ip: str | None) -> None:
+    """Print a warning when the auto-detected advertise IP is in a known virtual-network range.
+
+    172.16.0.0/12 is used by Docker, WSL2, and most hypervisors.  Nodes behind these
+    adapters are NOT directly reachable from other physical machines on the LAN, so
+    cross-host pipeline hops will time out.  The user must pass --advertise-host with
+    their actual LAN IP (e.g. 192.168.x.x) to fix this.
+    """
+    if ip is None:
+        return
+    try:
+        parts = [int(p) for p in ip.split(".")]
+        if len(parts) != 4:
+            return
+        a, b = parts[0], parts[1]
+        # 172.16.0.0/12 → 172.16–31.x.x
+        if a == 172 and 16 <= b <= 31:
+            print(
+                f"\n  WARNING: auto-detected endpoint IP {ip} is in 172.16.0.0/12.\n"
+                f"  This range is used by Docker, WSL2, and virtual machines and is\n"
+                f"  NOT reachable from other physical machines on your LAN.\n"
+                f"  Cross-host pipeline hops WILL time out.\n"
+                f"  Fix: use a public tracker with relay (wss://…/ws), or pass\n"
+                f"  --advertise-host <your-LAN-ip> (e.g. 192.168.x.x).\n",
+                flush=True,
+            )
+    except Exception:
+        pass
+
+
 def run_startup(
     tracker_url: str,
     port: int = 0,
@@ -45,6 +312,10 @@ def run_startup(
     host: str = "127.0.0.1",
     advertise_host: str | None = None,
     contracts: Any | None = None,
+    route_timeout: float = 30.0,
+    vram_mb_override: int | None = None,
+    max_loaded_shards: int = 1,
+    debug: bool = False,
 ) -> StubNodeServer | TorchNodeServer:
     """Execute the full startup sequence and return a running node server.
 
@@ -60,19 +331,83 @@ def run_startup(
     """
 
     tracker_url = tracker_url.rstrip("/")
+    relay_url = _discover_relay_url(tracker_url)
+    if max_loaded_shards < 1:
+        raise ValueError("--max-shards must be at least 1")
 
     # 1. Hardware detection
+    if advertise_host is None and host == "0.0.0.0":
+        # socket.getfqdn() returns an mDNS name (.local / .localdomain) that remote
+        # machines on a different OS or subnet often can't resolve. Instead, probe the
+        # outbound IP by opening a UDP socket toward the tracker — no data is sent.
+        try:
+            _tracker_host = urllib.parse.urlparse(tracker_url).hostname or "8.8.8.8"
+            _s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
+            _s.connect((_tracker_host, 80))
+            advertise_host = _s.getsockname()[0]
+            _s.close()
+        except Exception:
+            advertise_host = socket.getfqdn()
+
+    if relay_url:
+        print(f"Relay advertised by tracker — using outbound tunnel {relay_url}", flush=True)
+    else:
+        _warn_virtual_network_ip(advertise_host)
+
     print("Detecting hardware...", flush=True)
     hw = detect_hardware()
     device: str = hw["device"]
     gpu_name: str | None = hw.get("gpu_name")
     vram_mb: int = hw.get("vram_mb", 0)
+    shared_vram_mb: int = hw.get("shared_vram_mb", 0)
+    ram_mb: int = hw.get("ram_mb", 16 * 1024)
 
-    if device == "cpu":
+    if vram_mb_override is not None:
-        print("  WARNING: No CUDA GPU detected — running in CPU mode", flush=True)
+        vram_mb = vram_mb_override
+        shared_vram_mb = 0
+        print(f"  Memory budget overridden to {vram_mb / 1024:.1f} GB via --memory", flush=True)
+    elif device == "cpu":
+        gpu_suffix = ""
+        if gpu_name and vram_mb > 0:
+            gpu_suffix = (
+                f"; CUDA inactive; detected {gpu_name} "
+                f"({vram_mb / 1024:.1f} GB dedicated VRAM, {shared_vram_mb / 1024:.1f} GB shared)"
+            )
+        print(f"  WARNING: No CUDA GPU detected — running in CPU mode ({ram_mb / 1024:.1f} GB RAM{gpu_suffix})", flush=True)
     else:
-        print(f"  GPU: {gpu_name} ({vram_mb} MB VRAM)", flush=True)
+        shared_suffix = f", {shared_vram_mb / 1024:.1f} GB shared" if shared_vram_mb > 0 else ""
+        print(f"  GPU: {gpu_name} ({vram_mb / 1024:.1f} GB dedicated VRAM{shared_suffix}, {ram_mb / 1024:.1f} GB RAM)", flush=True)
 
+    if vram_mb_override is not None:
+        memory_budget_mb = vram_mb
+        memory_budget_source = "memory override"
+    else:
+        memory_budget_mb, memory_budget_source = _memory_budget(device, vram_mb, ram_mb, shared_vram_mb)
+    assignment_vram_mb = memory_budget_mb if device == "cuda" or vram_mb_override is not None else 0
+    print(f"  Memory budget: {memory_budget_mb / 1024:.1f} GB {memory_budget_source}", flush=True)
+
+    print("Benchmarking compute...", flush=True)
+    if device != "cuda" and gpu_name:
+        _cuda_score, cuda_ok, cuda_error = benchmark_throughput_checked("cuda")
+        hw["cuda_benchmark_ok"] = cuda_ok
+        if cuda_error:
+            hw["cuda_benchmark_error"] = cuda_error
+        if not cuda_ok:
+            print(f"  CUDA benchmark unavailable: {cuda_error}; using CPU benchmark", flush=True)
+    bench_tps, bench_ok, bench_error = benchmark_throughput_checked(device)
+    hw["benchmark_device"] = device
+    hw["benchmark_ok"] = bench_ok
+    if bench_error:
+        hw["benchmark_error"] = bench_error
+    device_label = "GPU" if device == "cuda" else "CPU"
+    print(f"  {device_label} throughput index: {bench_tps:,.0f}", flush=True)
+
+    registration_capabilities = {
+        "vram_bytes": max(0, int(assignment_vram_mb)) * 1024 * 1024,
+        "ram_bytes": max(0, int(ram_mb)) * 1024 * 1024,
+        "max_loaded_shards": max_loaded_shards,
+        "benchmark_tokens_per_sec": bench_tps,
+    }
     # 2. Wallet
     print("Loading wallet...", flush=True)
     wallet_kwargs: dict = {}
@@ -84,15 +419,36 @@ def run_startup(
     if probationary_line is not None:
         print(f"  {probationary_line}", flush=True)
 
-    if model_id is not None:
+    if model_id:  # treat "" the same as None — no explicit model given
+        user_pinned_shard = shard_start is not None or shard_end is not None
         # Auto-detect shard range from model config if not explicitly provided
         if shard_start is None or shard_end is None:
-            detected = _detect_num_layers(model_id)
+            try:
+                detected = _detect_num_layers(model_id, cache_dir=cache_dir)
+            except TypeError:
+                detected = _detect_num_layers(model_id)
             if detected is None:
                 raise ValueError(
                     f"Could not read num_hidden_layers from {model_id} config. "
                     "Pass --shard-start and --shard-end explicitly."
                 )
+            # When no explicit shard range given, ask the tracker if there's a gap for this model.
+            if shard_start is None and shard_end is None:
+                try:
+                    qs = urllib.parse.urlencode({
+                        "device": device, "vram_mb": assignment_vram_mb, "ram_mb": ram_mb, "hf_repo": model_id,
+                    })
+                    net_asgn = _get_json(f"{tracker_url}/v1/network/assign?{qs}", timeout=5.0)
+                    if net_asgn.get("hf_repo") == model_id and net_asgn.get("gap_found"):
+                        shard_start = net_asgn["shard_start"]
+                        shard_end = net_asgn["shard_end"]
+                        print(
+                            f"  Tracker found uncovered shard: "
+                            f"layers {shard_start}–{shard_end} (of {detected})",
+                            flush=True,
+                        )
+                except Exception:
+                    pass  # No other nodes registered yet — default to full model below
             shard_start = shard_start if shard_start is not None else 0
             shard_end = shard_end if shard_end is not None else detected - 1
             print(f"  Auto-detected {detected} layers → shard {shard_start}–{shard_end}", flush=True)
@@ -105,19 +461,72 @@ def run_startup(
             shard_start=shard_start,
             shard_end=shard_end,
             quantization=quantization,
+            tracker_url=tracker_url,
+            route_timeout=route_timeout,
+            cache_dir=cache_dir,
+            debug=debug,
         )
+        _node_start_time = time.monotonic()
         actual_port = node.start()
+        total_layers = getattr(getattr(node, "backend", None), "total_layers", None)
+        if isinstance(total_layers, int) and total_layers > 0:
+            layer_count = shard_end - shard_start + 1
+            shard_label = f"layers {shard_start}–{shard_end}; {layer_count} of {total_layers}"
+        else:
+            shard_label = f"layers {shard_start}–{shard_end}"
         public_host = advertise_host or (socket.getfqdn() if host == "0.0.0.0" else host)
         endpoint = f"http://{public_host}:{actual_port}"
+        local_base_url = f"http://127.0.0.1:{actual_port}"
+        relay_bridge, relay_fields = _start_relay_bridge_if_available(
+            tracker_url,
+            address,
+            local_base_url,
+            endpoint,
+            relay_url=relay_url,
+        )
+        _attach_relay_bridge(node, relay_bridge)
+        # Register with tracker so other nodes can auto-join this model.
+        total_layers = getattr(getattr(node, "backend", None), "total_layers", None)
+        reg_payload = {
+            "endpoint": endpoint,
+            "model": model_id.split("/")[-1],
+            "hf_repo": model_id,
+            "num_layers": total_layers,
+            "shard_start": shard_start,
+            "shard_end": shard_end,
+            "hardware_profile": hw,
+            "wallet_address": address,
+            "quantization": quantization,
+            "score": 1.0,
+            "tracker_mode": (shard_start == 0),
+            "managed_assignment": not user_pinned_shard,
+            "model_metadata": model_metadata_for(model_id, total_layers, cache_dir=cache_dir),
+            **registration_capabilities,
+            **relay_fields,
+        }
+        tracker_node_id: str | None = None
+        try:
+            reg_resp = _post_json(f"{tracker_url}/v1/nodes/register", reg_payload)
+            tracker_node_id = str(reg_resp.get("node_id") or "?")
+            setattr(node, "tracker_node_id", tracker_node_id)
+            print(f"  Registered with tracker — node ID: {tracker_node_id}", flush=True)
+            _start_heartbeat(tracker_url, tracker_node_id, reg_payload, node_ref=node, start_time=_node_start_time)
+        except Exception as exc:
+            setattr(node, "tracker_node_id", None)
+            print(f"  Warning: tracker registration failed: {exc}", flush=True)
+
         print(
             f"\n{'=' * 32}\n"
             f"meshnet-node ready\n"
             f"  Wallet:       {address}\n"
             f"  Model ID:     {model_id}\n"
-            f"  Shard:        layers {shard_start}–{shard_end}\n"
+            f"  Shard:        {shard_label}\n"
+            f"  {_shard_budget_line(memory_budget_mb, memory_budget_source, total_layers, quantization)}\n"
             f"  Quantization: {quantization}\n"
             f"  Endpoint:     {endpoint}\n"
-            f"  Hardware:     {device.upper()}\n"
+            f"  Node ID:      {tracker_node_id or 'unregistered'}\n"
+            f"  Hardware:     {_hardware_label(device, gpu_name)}\n"
+            f"  Benchmark:    {bench_tps:,.0f} (throughput index)\n"
             f"{'=' * 32}",
             flush=True,
         )
@@ -125,12 +534,106 @@ def run_startup(
     if shard_start is not None or shard_end is not None:
         raise ValueError("--shard-start / --shard-end require --model-id")
 
-    # 3. Shard assignment from tracker
+    # 3a. Auto-join: query tracker for network-wide HF model assignment.
+    print("Querying tracker for network assignment...", flush=True)
+    assign_qs = urllib.parse.urlencode({"device": device, "vram_mb": assignment_vram_mb, "ram_mb": ram_mb})
+    net_assignment: dict = {}
+    try:
+        net_assignment = _get_json(f"{tracker_url}/v1/network/assign?{assign_qs}")
+    except Exception as exc:
+        print(f"  (auto-join unavailable: {exc})", flush=True)
+    assigned_hf_repo: str | None = net_assignment.get("hf_repo")
+    _gap_found: bool = bool(net_assignment.get("gap_found", False))
+
+    if assigned_hf_repo and _gap_found:
+        assigned_shard_start: int = net_assignment["shard_start"]
+        assigned_shard_end: int = net_assignment["shard_end"]
+        assigned_num_layers: int = net_assignment["num_layers"]
+        print(
+            f"  Assigned: {assigned_hf_repo} "
+            f"layers {assigned_shard_start}–{assigned_shard_end} "
+            f"(of {assigned_num_layers})",
+            flush=True,
+        )
+        print("Loading real PyTorch model shard...", flush=True)
+        node = TorchNodeServer(
+            host=host,
+            port=port,
+            model_id=assigned_hf_repo,
+            shard_start=assigned_shard_start,
+            shard_end=assigned_shard_end,
+            quantization=quantization,
+            tracker_url=tracker_url,
+            route_timeout=route_timeout,
+            cache_dir=cache_dir,
+            debug=debug,
+        )
+        _node_start_time = time.monotonic()
+        actual_port = node.start()
+        public_host = advertise_host or (socket.getfqdn() if host == "0.0.0.0" else host)
+        endpoint = f"http://{public_host}:{actual_port}"
+        local_base_url = f"http://127.0.0.1:{actual_port}"
+        relay_bridge, relay_fields = _start_relay_bridge_if_available(
+            tracker_url,
+            address,
+            local_base_url,
+            endpoint,
+            relay_url=relay_url,
+        )
+        _attach_relay_bridge(node, relay_bridge)
+        auto_reg_payload = {
+            "endpoint": endpoint,
+            "model": assigned_hf_repo.split("/")[-1],
+            "hf_repo": assigned_hf_repo,
+            "num_layers": assigned_num_layers,
+            "shard_start": assigned_shard_start,
+            "shard_end": assigned_shard_end,
+            "hardware_profile": hw,
+            "wallet_address": address,
+            "quantization": quantization,
+            "score": 1.0,
+            "tracker_mode": (assigned_shard_start == 0),
+            "managed_assignment": True,
+            "model_metadata": model_metadata_for(assigned_hf_repo, assigned_num_layers, cache_dir=cache_dir),
+            **registration_capabilities,
+            **relay_fields,
+        }
+        tracker_node_id = None
+        try:
+            reg_resp = _post_json(f"{tracker_url}/v1/nodes/register", auto_reg_payload)
+            tracker_node_id = str(reg_resp.get("node_id") or "?")
+            setattr(node, "tracker_node_id", tracker_node_id)
+            print(f"  Registered with tracker — node ID: {tracker_node_id}", flush=True)
+            _start_heartbeat(tracker_url, tracker_node_id, auto_reg_payload, node_ref=node, start_time=_node_start_time)
+        except Exception as exc:
+            setattr(node, "tracker_node_id", None)
+            print(f"  Warning: tracker registration failed: {exc}", flush=True)
+        shard_count = assigned_shard_end - assigned_shard_start + 1
+        print(
+            f"\n{'=' * 32}\n"
+            f"meshnet-node ready (auto-joined)\n"
+            f"  Wallet:       {address}\n"
+            f"  Model ID:     {assigned_hf_repo}\n"
+            f"  Shard:        layers {assigned_shard_start}–{assigned_shard_end} "
+            f"({shard_count} of {assigned_num_layers})\n"
+            f"  {_shard_budget_line(memory_budget_mb, memory_budget_source, assigned_num_layers, quantization)}\n"
+            f"  Quantization: {quantization}\n"
+            f"  Endpoint:     {endpoint}\n"
+            f"  Node ID:      {tracker_node_id or 'unregistered'}\n"
+            f"  Hardware:     {_hardware_label(device, gpu_name)}\n"
+            f"  Benchmark:    {bench_tps:,.0f} (throughput index)\n"
+            f"{'=' * 32}",
+            flush=True,
+        )
+        return node
+
+    # 3b. Shard assignment from tracker (stub-model / preset-based path)
     print("Querying tracker for shard assignment...", flush=True)
     assign_qs = urllib.parse.urlencode({
         "model": model,
         "device": device,
         "vram_mb": vram_mb,
+        "ram_mb": ram_mb,
     })
     try:
         assignment = _get_json(f"{tracker_url}/v1/nodes/assign?{assign_qs}")
@@ -172,6 +675,15 @@ def run_startup(
     actual_port = node.start()
     public_host = advertise_host or (socket.getfqdn() if host == "0.0.0.0" else host)
     endpoint = f"http://{public_host}:{actual_port}"
+    local_base_url = f"http://127.0.0.1:{actual_port}"
+    relay_bridge, relay_fields = _start_relay_bridge_if_available(
+        tracker_url,
+        address,
+        local_base_url,
+        endpoint,
+        relay_url=relay_url,
+    )
+    _attach_relay_bridge(node, relay_bridge)
 
     # 6. Register with tracker
     print("Registering with tracker...", flush=True)
@@ -187,9 +699,12 @@ def run_startup(
                 "hardware_profile": hw,
                 "wallet_address": address,
                 "score": 1.0,
+                **registration_capabilities,
+                **relay_fields,
             },
         )
-        node_id: str = reg_resp["node_id"]
+        node_id = str(reg_resp["node_id"])
+        setattr(node, "tracker_node_id", node_id)
     except Exception:
         node.stop()
         raise
@@ -197,15 +712,17 @@ def run_startup(
     # Status summary
     hw_str = device.upper()
     if gpu_name:
-        hw_str += f" ({gpu_name}, {vram_mb} MB)"
+        hw_str += f" ({gpu_name}, {vram_mb / 1024:.1f} GB)"
     print(
         f"\n{'=' * 32}\n"
         f"meshnet-node ready\n"
         f"  Wallet:   {address}\n"
         f"  Shard:    layers {shard_start}-{shard_end} ({assigned_model})\n"
+        f"  {_shard_budget_line(memory_budget_mb, memory_budget_source, assignment.get('model_layers_end', shard_end) + 1, quantization)}\n"
         f"  Endpoint: {endpoint}\n"
         f"  Node ID:  {node_id}\n"
         f"  Hardware: {hw_str}\n"
+        f"  Benchmark: {bench_tps:,.0f} (throughput index)\n"
         f"{'=' * 32}",
         flush=True,
     )
@@ -213,11 +730,14 @@ def run_startup(
     return node
 
 
-def _detect_num_layers(model_id: str) -> int | None:
+def _detect_num_layers(model_id: str, cache_dir: Path | None = None) -> int | None:
     """Fetch num_hidden_layers from HuggingFace model config (downloads ~1 KB config.json only)."""
     try:
         from transformers import AutoConfig  # type: ignore[import]
-        cfg = AutoConfig.from_pretrained(model_id)
+        cfg = AutoConfig.from_pretrained(
+            model_id,
+            cache_dir=str(cache_dir) if cache_dir is not None else None,
+        )
         return int(cfg.num_hidden_layers)
     except Exception as exc:
         print(f"  Warning: could not read model config from HF: {exc}", flush=True)

packages/node/meshnet_node/torch_server.py

@@ -2,6 +2,7 @@
 
 from __future__ import annotations
 
+import base64
 import http.server
 import json
 import sys
@@ -11,6 +12,8 @@ import urllib.error
 import urllib.parse
 import urllib.request
 import uuid
+from pathlib import Path
+from typing import Any
 
 from .model_backend import (
     InsufficientVRAMError,
@@ -28,6 +31,40 @@ from .server import (
 )
 
 
+def _relay_hop(
+    relay_addr: str,
+    path: str,
+    body: bytes,
+    headers: dict[str, str],
+    timeout: float = 120.0,
+) -> tuple[int, dict[str, str], bytes]:
+    """Send a single HTTP-shaped request through a relay RPC WebSocket.
+
+    relay_addr is the wss://relay.../rpc/{peer_id} URL.
+    Returns (status, response_headers_lower, response_body).
+    Raises on connection failure so callers can fall back to direct.
+    """
+    import websockets.sync.client as wsc  # type: ignore[import]
+
+    request_id = f"{time.time_ns():x}"
+    payload = json.dumps({
+        "request_id": request_id,
+        "method": "POST",
+        "path": path,
+        "headers": headers,
+        "body_base64": base64.b64encode(body).decode(),
+    })
+    with wsc.connect(relay_addr, open_timeout=timeout) as ws:
+        ws.send(payload)
+        raw = ws.recv(timeout=timeout)
+    resp = json.loads(raw)
+    status = int(resp.get("status", 503))
+    resp_headers = {k.lower(): v for k, v in (resp.get("headers") or {}).items()}
+    body_b64 = resp.get("body_base64")
+    resp_body = base64.b64decode(body_b64) if body_b64 else (resp.get("body") or "").encode()
+    return status, resp_headers, resp_body
+
+
 class _TorchHTTPServer(http.server.HTTPServer):
     def __init__(
         self,
@@ -36,6 +73,8 @@ class _TorchHTTPServer(http.server.HTTPServer):
         backend: TorchModelShard,
         tracker_mode: bool = False,
         tracker_url: str | None = None,
+        route_timeout: float = 30.0,
+        debug: bool = False,
     ):
         super().__init__(addr, handler)
         self.backend = backend
@@ -43,6 +82,12 @@ class _TorchHTTPServer(http.server.HTTPServer):
         self.forward_chunk_count = 0
         self.tracker_mode = tracker_mode
         self.tracker_url = tracker_url
+        self.route_timeout = route_timeout
+        self.debug = debug
+        self.total_requests: int = 0
+        self.failed_requests: int = 0
+        self.queue_depth: int = 0
+        self._stats_lock = threading.Lock()
 
 
 class _TorchHandler(http.server.BaseHTTPRequestHandler):
@@ -138,12 +183,16 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
         if int(self.headers.get("X-Meshnet-Hop-Index", "0")) > 0:
             server.received_activations = True
 
+        start_layer_header = self.headers.get("X-Meshnet-Start-Layer")
+        start_layer = int(start_layer_header) if start_layer_header else None
+
         try:
             result = server.backend.forward_bytes(
                 raw_body,
                 shape,
                 self.headers.get("X-Meshnet-Attn-Mask"),
                 self.headers.get("X-Meshnet-Position-Ids"),
+                start_layer=start_layer,
             )
         except Exception as exc:
             self._send_json(500, {"error": str(exc)})
@@ -205,48 +254,181 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
         self.send_header("Content-Type", "application/json")
         self.send_header("Content-Length", str(len(payload)))
         self.end_headers()
-        self.wfile.write(payload)
+        try:
+            self.wfile.write(payload)
+        except BrokenPipeError:
+            pass  # client disconnected before we could respond — not an error
 
     def _handle_chat_completions(self) -> None:
         server: _TorchHTTPServer = self.server  # type: ignore[assignment]
+        with server._stats_lock:
+            server.total_requests += 1
+            server.queue_depth += 1
+        try:
+            self._do_chat_completions(server)
+        finally:
+            with server._stats_lock:
+                server.queue_depth -= 1
+
+    def _record_failed_request(self) -> None:
+        server: _TorchHTTPServer = self.server  # type: ignore[assignment]
+        with server._stats_lock:
+            server.failed_requests += 1
+
+    def _do_chat_completions(self, server: "_TorchHTTPServer") -> None:
         body = self._read_json_body()
         if body is None:
             return
         messages = body.get("messages", [])
+        if not isinstance(messages, list):
+            messages = []
         stream = bool(body.get("stream", False))
-        model = str(body.get("model", ""))
+        model_name = str(body.get("model", ""))
-        prompt = " ".join(
+        max_tokens = int(body.get("max_tokens") or body.get("max_new_tokens") or 256)
-            str(m.get("content", ""))
+        temperature = float(body.get("temperature") or 1.0)
-            for m in messages
+        top_p = float(body.get("top_p") or 1.0)
-            if isinstance(m, dict) and m.get("role") == "user"
-        )
-        try:
-            payload = server.backend.encode_prompt(prompt)
-        except Exception as exc:
-            self._send_json(500, {"error": f"encode_prompt failed: {exc}"})
-            return
-        remaining_route = self._get_remaining_route(model)
-        result_text = self._run_downstream_pipeline(payload, remaining_route)
-        self._send_openai_response(result_text, model, stream)
 
-    def _get_remaining_route(self, model: str) -> list[str]:
+        # Fast path: this node owns the complete model — use HF generate() with KV cache.
+        # Avoids the single-token-per-forward-pass limitation of the distributed path.
+        if server.backend.is_head and server.backend.is_tail:
+            try:
+                if stream:
+                    self._stream_openai_response(
+                        server.backend.generate_text_streaming(messages, max_tokens, temperature, top_p),
+                        model_name,
+                    )
+                else:
+                    text = server.backend.generate_text(messages, max_tokens, temperature, top_p)
+                    self._send_openai_response(text, model_name, False, messages)
+            except Exception as exc:
+                self._record_failed_request()
+                self._send_json(500, {"error": f"generation failed: {exc}"})
+            return
+
+        # Distributed path: autoregressive generation across shards.
+        # We do N single-step forward passes (no cross-node KV cache), which is slow
+        # but correct. Each step: head encodes current sequence → forwards through route
+        # → tail returns the next token string → append → repeat.
+        remaining_route = self._get_remaining_route(model_name)
+        print(
+            f"  [node] chat route model={model_name!r} max_tokens={max_tokens} "
+            f"downstream={remaining_route}",
+            flush=True,
+        )
+        if not remaining_route:
+            self._send_openai_response(
+                "error: no downstream route — check tracker connectivity",
+                model_name, False, messages,
+            )
+            return
+
+        backend = server.backend
+        # Format with chat template so the model knows it's in assistant mode.
+        try:
+            if hasattr(backend.tokenizer, "apply_chat_template"):
+                prompt_text: str = backend.tokenizer.apply_chat_template(
+                    messages, add_generation_prompt=True, tokenize=False,
+                )
+            else:
+                raise AttributeError("no apply_chat_template")
+        except Exception:
+            prompt_text = " ".join(
+                str(m.get("content", ""))
+                for m in messages
+                if isinstance(m, dict) and m.get("role") == "user"
+            )
+
+        eos_token: str = getattr(backend.tokenizer, "eos_token", "") or ""
+        generated: list[str] = []
+        current_text = prompt_text
+
+        for _ in range(max_tokens):
+            try:
+                payload = backend.encode_prompt(current_text)
+            except Exception as exc:
+                print(f"  [node] distributed encode error: {exc}", flush=True)
+                break
+            token_str = self._run_downstream_pipeline(payload, remaining_route)
+            if not token_str:
+                break
+            # Stop on error responses or EOS.
+            if token_str.startswith(("pipeline error", "decode error", "no downstream", "error:")):
+                break
+            if eos_token and token_str == eos_token:
+                break
+            generated.append(token_str)
+            current_text = current_text + token_str
+
+        result_text = "".join(generated)
+        self._send_openai_response(result_text, model_name, stream, messages)
+
+    def _get_remaining_route(self, model: str) -> list[dict]:
+        """Return downstream hops as dicts with endpoint, start_layer, and optional relay_addr.
+
+        Fast path reads X-Meshnet-Route header injected by the tracker.
+        Slow path queries the tracker's /v1/route endpoint as a fallback.
+        start_layer tells each downstream node which layer to begin from,
+        enabling correct execution when shard ranges overlap.
+        """
+        # Fast path: tracker pre-resolved the downstream route and injected it as a header.
+        injected = self.headers.get("X-Meshnet-Route")
+        if injected:
+            try:
+                route = json.loads(injected)
+                if isinstance(route, list):
+                    hops: list[dict] = []
+                    for item in route:
+                        if isinstance(item, dict):
+                            hop = {
+                                "endpoint": str(item["endpoint"]),
+                                "start_layer": int(item.get("start_layer", 0)),
+                            }
+                            if item.get("relay_addr"):
+                                hop["relay_addr"] = str(item["relay_addr"])
+                            hops.append(hop)
+                        elif isinstance(item, str):
+                            hops.append({"endpoint": item, "start_layer": 0})
+                    print(f"  [node] using injected downstream route: {[h['endpoint'] for h in hops]}", flush=True)
+                    return hops
+            except (json.JSONDecodeError, TypeError, KeyError):
+                pass
+
+        # Slow path: query the tracker (direct node-to-node calls, or tracker didn't inject).
         server: _TorchHTTPServer = self.server  # type: ignore[assignment]
         if server.tracker_url is None:
             return []
+        route_model = getattr(server.backend, "model_id", None) or model
         try:
-            url = f"{server.tracker_url}/v1/route?model={urllib.parse.quote(model)}"
+            url = f"{server.tracker_url}/v1/route?model={urllib.parse.quote(route_model)}"
-            with urllib.request.urlopen(url, timeout=5.0) as r:
+            with urllib.request.urlopen(url, timeout=server.route_timeout) as r:
                 route_resp = json.loads(r.read())
-            route = route_resp.get("route", [])
+            own_port = server.server_address[1]
-            # Skip the first node in the route (self) since we're already the head
+            nodes_info = route_resp.get("nodes", [])
-            return list(route[1:])
+            hops = []
-        except Exception:
+            covered_up_to: int | None = None
+            for node_info in nodes_info:
+                ep = node_info.get("endpoint", "")
+                if ep.rstrip("/").endswith(f":{own_port}"):
+                    covered_up_to = node_info.get("shard_end")
+                    continue
+                if covered_up_to is None:
+                    covered_up_to = (node_info.get("shard_start") or 1) - 1
+                hop = {"endpoint": ep, "start_layer": covered_up_to + 1}
+                if node_info.get("relay_addr"):
+                    hop["relay_addr"] = str(node_info["relay_addr"])
+                hops.append(hop)
+                covered_up_to = node_info.get("shard_end", covered_up_to)
+            print(f"  [node] tracker downstream route: {[h['endpoint'] for h in hops]}", flush=True)
+            return hops
+        except Exception as exc:
+            print(f"  [node] WARNING: route lookup failed for {route_model!r}: {exc}", flush=True)
             return []
 
-    def _run_downstream_pipeline(self, payload: object, route: list[str]) -> str:
+    def _run_downstream_pipeline(self, payload: object, route: list[dict]) -> str:
         server: _TorchHTTPServer = self.server  # type: ignore[assignment]
         if not route:
-            # Single-node mode: decode tail locally if we're the tail
+            # Partial shard at tail: decode the activation from the previous node.
+            # Full single-node (head+tail) is handled before entering this method.
             if server.backend.is_tail:
                 try:
                     tensor = server.backend.torch.frombuffer(
@@ -256,7 +438,7 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
                     return server.backend.decode_tail(tensor)
                 except Exception as exc:
                     return f"decode error: {exc}"
-            return ""
+            return "no downstream route available for non-tail shard"
 
         session = str(uuid.uuid4())
         shape = payload.shape  # type: ignore[union-attr]
@@ -267,7 +449,16 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
         current_attn = attn_mask
         current_pos = pos_ids
 
-        for hop_index, node_url in enumerate(route):
+        for hop_index, hop in enumerate(route):
+            node_url = hop["endpoint"]
+            start_layer = hop.get("start_layer", 0)
+            relay_addr = hop.get("relay_addr")
+            if server.debug:
+                print(
+                    f"  [node] pipeline hop {hop_index}: {node_url} start_layer={start_layer}"
+                    + (f" relay={relay_addr}" if relay_addr else ""),
+                    flush=True,
+                )
             headers: dict[str, str] = {
                 "Content-Type": "application/octet-stream",
                 "X-Meshnet-Wire": _WIRE_VERSION,
@@ -277,28 +468,52 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
                 "X-Meshnet-Chunk-Index": "0",
                 "X-Meshnet-Chunk-Total": "1",
                 "X-Meshnet-Hop-Index": str(hop_index),
+                "X-Meshnet-Start-Layer": str(start_layer),
             }
             if current_attn:
                 headers["X-Meshnet-Attn-Mask"] = current_attn
             if current_pos:
                 headers["X-Meshnet-Position-Ids"] = current_pos
-            req = urllib.request.Request(
+            if relay_addr:
-                f"{node_url}/forward",
+                try:
-                data=current_body,
+                    status, resp_headers, resp_body = _relay_hop(
-                headers=headers,
+                        relay_addr, "/forward", current_body, headers, timeout=120.0,
-                method="POST",
+                    )
-            )
+                    if status >= 400:
-            try:
+                        print(
-                with urllib.request.urlopen(req, timeout=10.0) as r:
+                            f"  [node] relay hop {hop_index} returned {status} from {relay_addr}",
-                    resp_body = r.read()
+                            flush=True,
-                    resp_headers = {k.lower(): v for k, v in r.headers.items()}
+                        )
-            except Exception as exc:
+                        return f"pipeline error at {node_url} via relay: status {status}"
-                return f"pipeline error at {node_url}: {exc}"
+                except Exception as exc:
+                    print(
+                        f"  [node] relay hop {hop_index} failed at {relay_addr}: {exc}; "
+                        f"falling back to direct {node_url}",
+                        flush=True,
+                    )
+                    relay_addr = None  # fall through to direct
+            if not relay_addr:
+                req = urllib.request.Request(
+                    f"{node_url}/forward",
+                    data=current_body,
+                    headers=headers,
+                    method="POST",
+                )
+                try:
+                    with urllib.request.urlopen(req, timeout=120.0) as r:
+                        resp_body = r.read()
+                        resp_headers = {k.lower(): v for k, v in r.headers.items()}
+                except Exception as exc:
+                    print(f"  [node] pipeline hop {hop_index} failed at {node_url}: {exc}", flush=True)
+                    return f"pipeline error at {node_url}: {exc}"
             content_type = resp_headers.get("content-type", "")
             if "application/json" in content_type:
                 try:
                     data = json.loads(resp_body)
-                    return str(data.get("text", ""))
+                    text = str(data.get("text", ""))
+                    if server.debug:
+                        print(f"  [node] pipeline hop {hop_index} returned text={text!r}", flush=True)
+                    return text
                 except json.JSONDecodeError:
                     return resp_body.decode("utf-8", errors="replace")
             # Binary activation — update and forward to next node
@@ -309,10 +524,57 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
             current_pos = resp_headers.get("x-meshnet-position-ids")
         return ""
 
-    def _send_openai_response(self, text: str, model: str, stream: bool) -> None:
+    def _stream_openai_response(self, token_iter, model: str) -> None:
+        """Stream tokens from an iterator as SSE chunks."""
+        chunk_id = "chatcmpl-node"
+        created = int(time.time())
+        self.send_response(200)
+        self.send_header("Content-Type", "text/event-stream; charset=utf-8")
+        self.send_header("Cache-Control", "no-cache")
+        self.end_headers()
+
+        def _emit(data: str) -> None:
+            try:
+                self.wfile.write(f"data: {data}\n\n".encode())
+                self.wfile.flush()
+            except BrokenPipeError:
+                pass
+
+        _emit(json.dumps({
+            "id": chunk_id, "object": "chat.completion.chunk", "created": created,
+            "model": model,
+            "choices": [{"index": 0, "delta": {"role": "assistant", "content": ""}, "finish_reason": None}],
+        }))
+        for token_text in token_iter:
+            if not token_text:
+                continue
+            _emit(json.dumps({
+                "id": chunk_id, "object": "chat.completion.chunk", "created": created,
+                "model": model,
+                "choices": [{"index": 0, "delta": {"content": token_text}, "finish_reason": None}],
+            }))
+        _emit(json.dumps({
+            "id": chunk_id, "object": "chat.completion.chunk", "created": created,
+            "model": model,
+            "choices": [{"index": 0, "delta": {}, "finish_reason": "stop"}],
+        }))
+        try:
+            self.wfile.write(b"data: [DONE]\n\n")
+            self.wfile.flush()
+        except BrokenPipeError:
+            pass
+
+    def _send_openai_response(
+        self,
+        text: str,
+        model: str,
+        stream: bool,
+        messages: list[dict] | None = None,
+    ) -> None:
         chunk_id = "chatcmpl-node"
         created = int(time.time())
         if not stream:
+            usage = _usage_for_response(self.server.backend, messages or [], text)  # type: ignore[attr-defined]
             self._send_json(200, {
                 "id": chunk_id,
                 "object": "chat.completion",
@@ -323,7 +585,7 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
                     "message": {"role": "assistant", "content": text},
                     "finish_reason": "stop",
                 }],
-                "usage": {"prompt_tokens": 0, "completion_tokens": 0, "total_tokens": 0},
+                "usage": usage,
             })
             return
         self.send_response(200)
@@ -332,8 +594,11 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
         self.end_headers()
 
         def _emit(data: str) -> None:
-            self.wfile.write(f"data: {data}\n\n".encode())
+            try:
-            self.wfile.flush()
+                self.wfile.write(f"data: {data}\n\n".encode())
+                self.wfile.flush()
+            except BrokenPipeError:
+                pass
 
         _emit(json.dumps({
             "id": chunk_id, "object": "chat.completion.chunk", "created": created,
@@ -350,8 +615,57 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
             "model": model,
             "choices": [{"index": 0, "delta": {}, "finish_reason": "stop"}],
         }))
-        self.wfile.write(b"data: [DONE]\n\n")
+        try:
-        self.wfile.flush()
+            self.wfile.write(b"data: [DONE]\n\n")
+            self.wfile.flush()
+        except BrokenPipeError:
+            pass
+
+
+def _usage_for_response(backend: object, messages: list[dict], completion_text: str) -> dict[str, int]:
+    prompt_tokens = _backend_token_count(
+        backend,
+        "count_prompt_tokens",
+        messages,
+        fallback=_fallback_message_token_count(messages),
+    )
+    completion_tokens = _backend_token_count(
+        backend,
+        "count_text_tokens",
+        completion_text,
+        fallback=_fallback_text_token_count(completion_text),
+    )
+    return {
+        "prompt_tokens": prompt_tokens,
+        "completion_tokens": completion_tokens,
+        "total_tokens": prompt_tokens + completion_tokens,
+    }
+
+
+def _backend_token_count(backend: object, method_name: str, value: object, fallback: int) -> int:
+    method: Any = getattr(backend, method_name, None)
+    if callable(method):
+        try:
+            return max(0, int(method(value)))
+        except Exception:
+            pass
+    return max(0, int(fallback))
+
+
+def _fallback_message_token_count(messages: list[dict]) -> int:
+    text = " ".join(
+        str(message.get("content", ""))
+        for message in messages
+        if isinstance(message, dict)
+    )
+    return _fallback_text_token_count(text)
+
+
+def _fallback_text_token_count(text: str) -> int:
+    parts = text.split()
+    if parts:
+        return len(parts)
+    return 1 if text else 0
 
 
 class TorchNodeServer:
@@ -368,6 +682,9 @@ class TorchNodeServer:
         backend: TorchModelShard | None = None,
         tracker_mode: bool | None = None,
         tracker_url: str | None = None,
+        route_timeout: float = 30.0,
+        cache_dir: Path | None = None,
+        debug: bool = False,
     ) -> None:
         self._host = host
         self._requested_port = port
@@ -376,14 +693,22 @@ class TorchNodeServer:
             shard_start,
             shard_end,
             quantization,
+            cache_dir,
         )
         # Auto-detect tracker mode: enabled when shard_start == 0 or explicitly set
         self._tracker_mode = tracker_mode if tracker_mode is not None else (shard_start == 0)
         self._tracker_url = tracker_url
+        self._route_timeout = route_timeout
+        self._cache_dir = cache_dir
+        self._debug = debug
         self._server: _TorchHTTPServer | None = None
         self._thread: threading.Thread | None = None
         self.port: int | None = None
 
+    @property
+    def route_timeout(self) -> float:
+        return self._route_timeout
+
     @property
     def backend(self) -> TorchModelShard:
         return self._backend
@@ -396,6 +721,55 @@ class TorchNodeServer:
     def forward_chunk_count(self) -> int:
         return self._server.forward_chunk_count if self._server is not None else 0
 
+    @property
+    def total_requests(self) -> int:
+        return self._server.total_requests if self._server is not None else 0
+
+    @property
+    def failed_requests(self) -> int:
+        return self._server.failed_requests if self._server is not None else 0
+
+    @property
+    def queue_depth(self) -> int:
+        return self._server.queue_depth if self._server is not None else 0
+
+    def apply_tracker_directives(self, directives: list[dict]) -> dict | None:
+        """Apply tracker LOAD_SHARD directives by hot-swapping the loaded backend."""
+        load_directive = next(
+            (directive for directive in reversed(directives) if directive.get("action") == "LOAD_SHARD"),
+            None,
+        )
+        if load_directive is None:
+            return None
+        shard_start = int(load_directive["shard_start"])
+        shard_end = int(load_directive["shard_end"])
+        quantization = str(load_directive.get("quantization") or self._backend.quantization)
+        model_id = str(load_directive.get("model") or self._backend.model_id)
+        print(
+            f"  [node] loading reassigned shard: {model_id} layers {shard_start}-{shard_end}",
+            flush=True,
+        )
+        try:
+            new_backend = _load_backend(model_id, shard_start, shard_end, quantization, self._cache_dir)
+        except TypeError:
+            new_backend = _load_backend(model_id, shard_start, shard_end, quantization)
+        self._backend = new_backend
+        self._tracker_mode = shard_start == 0
+        if self._server is not None:
+            self._server.backend = new_backend
+            self._server.tracker_mode = self._tracker_mode
+        print(
+            f"  [node] loaded reassigned shard: {model_id} layers {shard_start}-{shard_end}",
+            flush=True,
+        )
+        return {
+            "model": model_id,
+            "shard_start": shard_start,
+            "shard_end": shard_end,
+            "quantization": quantization,
+            "tracker_mode": self._tracker_mode,
+        }
+
     def start(self) -> int:
         if self._server is not None:
             raise RuntimeError("TorchNodeServer is already running")
@@ -405,6 +779,8 @@ class TorchNodeServer:
             self._backend,
             self._tracker_mode,
             self._tracker_url,
+            self._route_timeout,
+            self._debug,
         )
         self.port = self._server.server_address[1]
         self._thread = threading.Thread(target=self._server.serve_forever, daemon=True)
@@ -428,12 +804,13 @@ def _load_backend(
     shard_start: int,
     shard_end: int,
     quantization: str,
+    cache_dir: Path | None = None,
 ) -> TorchModelShard:
     from .model_backend import load_torch_shard
 
     quant = validate_quantization(quantization)
     try:
-        return load_torch_shard(model_id, shard_start, shard_end, quant)
+        return load_torch_shard(model_id, shard_start, shard_end, quant, cache_dir)
     except MissingModelDependencyError:
         raise
     except InsufficientVRAMError as exc:

packages/node/pyproject.toml

@@ -17,6 +17,7 @@ dependencies = [
     "safetensors>=0.4",
     "torch>=2.1",
     "transformers>=4.39",
+    "websockets>=13",
     "zstandard>=0.22",
 ]
 
@@ -26,3 +27,6 @@ meshnet-node = "meshnet_node.cli:main"
 [tool.setuptools.packages.find]
 where = ["."]
 include = ["meshnet_node*"]
+
+[tool.setuptools.package-data]
+meshnet_node = ["*.json"]

packages/relay/meshnet_relay/server.py

@@ -16,6 +16,7 @@ import asyncio
 import json
 import logging
 import threading
+import uuid
 from pathlib import Path
 
 from .peer_registry import PeerRegistry
@@ -56,6 +57,7 @@ class RelayServer:
         self._ready = threading.Event()
         self._running = False
         self._stop_event: asyncio.Event | None = None
+        self._pending_rpc: dict[str, asyncio.Future] = {}
 
     @property
     def registry(self) -> PeerRegistry:
@@ -121,6 +123,9 @@ class RelayServer:
         if path.startswith("/relay/"):
             peer_id = path[len("/relay/"):]
             await self._handle_circuit_relay(ws, peer_id)
+        elif path.startswith("/rpc/"):
+            peer_id = path[len("/rpc/"):]
+            await self._handle_rpc(ws, peer_id)
         elif path == "/health":
             await ws.send(json.dumps({"status": "ok", "peers": len(self._registry)}))
             await ws.close()
@@ -164,6 +169,14 @@ class RelayServer:
                     }))
                     continue
 
+                if topic == "relay-http-response":
+                    payload = envelope.get("payload", {})
+                    request_id = payload.get("request_id")
+                    fut = self._pending_rpc.pop(request_id, None)
+                    if fut is not None and not fut.done():
+                        fut.set_result(payload)
+                    continue
+
                 # Fan out to all other registered peers
                 if peer_id:
                     self._registry.touch(peer_id)
@@ -205,6 +218,50 @@ class RelayServer:
             return_exceptions=True,
         )
 
+    async def _handle_rpc(self, ws_requester, target_peer_id: str) -> None:
+        """Send one HTTP-shaped request to a connected peer and relay its response."""
+        target = self._registry.get(target_peer_id)
+        if target is None:
+            await ws_requester.send(json.dumps({
+                "status": 503,
+                "headers": {"Content-Type": "application/json"},
+                "body": json.dumps({"error": f"peer {target_peer_id!r} not connected to relay"}),
+            }))
+            await ws_requester.close()
+            return
+
+        try:
+            raw = await asyncio.wait_for(ws_requester.recv(), timeout=30.0)
+            payload = json.loads(raw)
+        except Exception:
+            await ws_requester.close(1003, "invalid relay rpc request")
+            return
+
+        request_id = str(payload.get("request_id") or uuid.uuid4())
+        payload["request_id"] = request_id
+        payload["target_peer"] = target_peer_id
+        fut = self._loop.create_future() if self._loop is not None else asyncio.get_running_loop().create_future()
+        self._pending_rpc[request_id] = fut
+        try:
+            await target.ws.send(json.dumps({
+                "topic": "relay-http-request",
+                "version": 1,
+                "from_peer": "relay",
+                "payload": payload,
+            }))
+            response = await asyncio.wait_for(fut, timeout=310.0)
+            await ws_requester.send(json.dumps(response))
+        except asyncio.TimeoutError:
+            await ws_requester.send(json.dumps({
+                "request_id": request_id,
+                "status": 504,
+                "headers": {"Content-Type": "application/json"},
+                "body": json.dumps({"error": "relay rpc timed out"}),
+            }))
+        finally:
+            self._pending_rpc.pop(request_id, None)
+            await ws_requester.close()
+
 
 async def _broadcast(raw: str | bytes, peers: list) -> None:
     """Send raw message to all peers; ignore individual send failures."""

packages/tracker/meshnet_tracker/billing.py

@@ -0,0 +1,430 @@
+"""Off-chain USDT billing ledger (ADR-0015, US-031).
+
+Tracks client API-key balances, node pending balances, and the protocol cut.
+All mutations are expressed as append-only billing events with unique ids so
+the ledger converges when events are gossiped across the tracker hive: every
+event is a commutative balance delta, and each tracker applies an event at
+most once (dedupe by event id).
+
+No Solana calls live here — on-chain deposit/settlement adapters (US-032/033)
+consume this ledger through `snapshot()` and `settle_node_payout()`.
+"""
+
+from __future__ import annotations
+
+import json
+import sqlite3
+import threading
+import time
+import uuid
+
+DEFAULT_PRICE_PER_1K_TOKENS = 0.02  # USDT
+DEFAULT_STARTING_CREDIT = 1.0  # USDT of Caller Credit for a new API key
+NODE_REVENUE_SHARE = 0.90  # nodes 90% / protocol cut 10% (ADR-0015)
+
+
+class BillingLedger:
+    """Thread-safe USDT ledger with SQLite persistence and event replication."""
+
+    SAVE_INTERVAL = 10.0
+
+    def __init__(
+        self,
+        db_path: str | None = None,
+        *,
+        prices: dict[str, float] | None = None,
+        default_price_per_1k: float = DEFAULT_PRICE_PER_1K_TOKENS,
+        starting_credit: float = DEFAULT_STARTING_CREDIT,
+        node_share: float = NODE_REVENUE_SHARE,
+    ) -> None:
+        self._db_path = db_path
+        self._prices = dict(prices) if prices else {}
+        self._default_price_per_1k = default_price_per_1k
+        self._starting_credit = starting_credit
+        self._node_share = node_share
+        self._lock = threading.Lock()
+        self._client_balances: dict[str, float] = {}
+        self._node_pending: dict[str, float] = {}
+        self._wallet_bindings: dict[str, str] = {}  # client wallet pubkey -> api_key
+        self._protocol_cut: float = 0.0
+        self._pending_since: dict[str, float] = {}  # wallet -> ts pending became > 0
+        self._last_payout_ts: dict[str, float] = {}
+        # settlement id -> {"payouts": [(wallet, amount)], "signature": str|None, "ts": float}
+        self._settlements: dict[str, dict] = {}
+        self._seen_event_ids: set[str] = set()
+        self._event_log: list[dict] = []  # full log, order of local application
+        self._dirty = False
+        if self._db_path:
+            self._init_db()
+            self._load_from_db()
+
+    # ---- pricing ----
+
+    def price_for(self, model: str) -> float:
+        return self._prices.get(model, self._default_price_per_1k)
+
+    def set_price(self, model: str, price_per_1k: float) -> None:
+        with self._lock:
+            self._prices[model] = price_per_1k
+
+    # ---- local operations (create + apply + log an event) ----
+
+    def ensure_client(self, api_key: str) -> float:
+        """Return the client's balance, granting Caller Credit on first touch."""
+        with self._lock:
+            if api_key not in self._client_balances:
+                self._apply_locked({
+                    "id": f"credit-{uuid.uuid4().hex}",
+                    "type": "credit",
+                    "api_key": api_key,
+                    "amount": self._starting_credit,
+                    "ts": time.time(),
+                    "note": "caller-credit",
+                })
+            return self._client_balances[api_key]
+
+    def has_funds(self, api_key: str) -> bool:
+        return self.ensure_client(api_key) > 0.0
+
+    def credit_client(self, api_key: str, amount: float, *, note: str = "deposit") -> float:
+        if amount <= 0:
+            raise ValueError("credit amount must be positive")
+        with self._lock:
+            self._apply_locked({
+                "id": f"credit-{uuid.uuid4().hex}",
+                "type": "credit",
+                "api_key": api_key,
+                "amount": amount,
+                "ts": time.time(),
+                "note": note,
+            })
+            return self._client_balances[api_key]
+
+    def charge_request(
+        self,
+        api_key: str,
+        model: str,
+        total_tokens: int,
+        node_work: list[tuple[str | None, int]],
+    ) -> dict:
+        """Debit the client and split the fee 90/10.
+
+        ``node_work`` is ``[(wallet_address | None, work_units), ...]`` for the
+        nodes that served the request. Work units of nodes without a wallet
+        accrue to the protocol cut — there is nowhere to pay them out.
+        The client balance may dip below zero on the final request; the next
+        request is then rejected by ``has_funds`` (post-pay drift, standard
+        metered-billing behavior).
+        """
+        cost = self.price_for(model) * max(0, total_tokens) / 1000.0
+        total_work = sum(max(0, w) for _, w in node_work)
+        node_pool = cost * self._node_share
+        shares: dict[str, float] = {}
+        if total_work > 0:
+            for wallet, work in node_work:
+                if wallet and work > 0:
+                    shares[wallet] = shares.get(wallet, 0.0) + node_pool * work / total_work
+        protocol_amount = cost - sum(shares.values())
+        with self._lock:
+            if api_key not in self._client_balances:
+                self._apply_locked({
+                    "id": f"credit-{uuid.uuid4().hex}",
+                    "type": "credit",
+                    "api_key": api_key,
+                    "amount": self._starting_credit,
+                    "ts": time.time(),
+                    "note": "caller-credit",
+                })
+            event = {
+                "id": f"charge-{uuid.uuid4().hex}",
+                "type": "charge",
+                "api_key": api_key,
+                "model": model,
+                "total_tokens": total_tokens,
+                "cost": cost,
+                "shares": shares,
+                "protocol_amount": protocol_amount,
+                "ts": time.time(),
+            }
+            self._apply_locked(event)
+            return event
+
+    def bind_wallet(self, api_key: str, wallet: str) -> dict:
+        """Bind a client wallet pubkey to an API key (US-032 deposits)."""
+        with self._lock:
+            event = {
+                "id": f"bind-{uuid.uuid4().hex}",
+                "type": "bind",
+                "api_key": api_key,
+                "wallet": wallet,
+                "ts": time.time(),
+            }
+            self._apply_locked(event)
+            return event
+
+    def api_key_for_wallet(self, wallet: str) -> str | None:
+        with self._lock:
+            return self._wallet_bindings.get(wallet)
+
+    def credit_deposit(self, api_key: str, amount: float, signature: str) -> dict | None:
+        """Credit an on-chain deposit exactly once.
+
+        The event id embeds the transaction signature, so replayed or
+        re-observed transfers — locally or via hive gossip — are no-ops.
+        Returns None when the signature was already credited.
+        """
+        if amount <= 0:
+            raise ValueError("deposit amount must be positive")
+        event_id = f"deposit-{signature}"
+        with self._lock:
+            if event_id in self._seen_event_ids:
+                return None
+            event = {
+                "id": event_id,
+                "type": "credit",
+                "api_key": api_key,
+                "amount": amount,
+                "signature": signature,
+                "ts": time.time(),
+                "note": "onchain-deposit",
+            }
+            self._apply_locked(event)
+            return event
+
+    def has_event(self, event_id: str) -> bool:
+        with self._lock:
+            return event_id in self._seen_event_ids
+
+    # ---- settlement (US-033) ----
+
+    def payables(
+        self,
+        *,
+        threshold: float,
+        max_period: float,
+        dust_floor: float,
+        now: float | None = None,
+        exclude: set[str] | None = None,
+    ) -> list[tuple[str, float]]:
+        """Wallets due a payout: pending ≥ threshold OR pending age ≥ max_period,
+        never below the dust floor (mining-pool standard, ADR-0015)."""
+        now = now if now is not None else time.time()
+        exclude = exclude or set()
+        due: list[tuple[str, float]] = []
+        with self._lock:
+            for wallet, pending in self._node_pending.items():
+                if wallet in exclude or pending < max(dust_floor, 1e-9):
+                    continue
+                if pending >= threshold:
+                    due.append((wallet, pending))
+                    continue
+                since = self._pending_since.get(wallet)
+                if since is not None and now - since >= max_period:
+                    due.append((wallet, pending))
+        return due
+
+    def confirm_settlement(self, settlement_id: str, signature: str) -> dict:
+        """Record the on-chain transaction signature for a settlement batch."""
+        with self._lock:
+            event = {
+                "id": f"settlement-{settlement_id}",
+                "type": "settlement",
+                "settlement_id": settlement_id,
+                "signature": signature,
+                "ts": time.time(),
+            }
+            self._apply_locked(event)
+            return event
+
+    def unconfirmed_settlements(self) -> dict[str, list[tuple[str, float]]]:
+        """Settlement batches whose pending was debited but whose transaction
+        has not been confirmed — resend these (idempotent by settlement id)."""
+        with self._lock:
+            return {
+                sid: list(s["payouts"])
+                for sid, s in self._settlements.items()
+                if s["signature"] is None and s["payouts"]
+            }
+
+    def settlement_history(self) -> list[dict]:
+        with self._lock:
+            return [
+                {
+                    "settlement_id": sid,
+                    "signature": s["signature"],
+                    "payouts": [
+                        {"wallet": w, "amount": a} for w, a in s["payouts"]
+                    ],
+                    "ts": s["ts"],
+                }
+                for sid, s in sorted(self._settlements.items(), key=lambda kv: kv[1]["ts"])
+            ]
+
+    def settle_node_payout(self, wallet: str, amount: float, *, reference: str = "") -> dict:
+        """Deduct a paid-out amount from a node's pending balance (US-033 hook)."""
+        if amount <= 0:
+            raise ValueError("payout amount must be positive")
+        with self._lock:
+            event = {
+                "id": f"payout-{uuid.uuid4().hex}",
+                "type": "payout",
+                "wallet": wallet,
+                "amount": amount,
+                "reference": reference,
+                "ts": time.time(),
+            }
+            self._apply_locked(event)
+            return event
+
+    def forfeit_pending(self, wallet: str, *, reason: str = "fraud") -> dict:
+        """Forfeit a node's entire pending balance to the protocol cut (US-034 hook)."""
+        with self._lock:
+            event = {
+                "id": f"forfeit-{uuid.uuid4().hex}",
+                "type": "forfeit",
+                "wallet": wallet,
+                "amount": self._node_pending.get(wallet, 0.0),
+                "reason": reason,
+                "ts": time.time(),
+            }
+            self._apply_locked(event)
+            return event
+
+    # ---- replication ----
+
+    def events_since(self, index: int) -> tuple[list[dict], int]:
+        """Return events after ``index`` in the local log and the new cursor."""
+        with self._lock:
+            return list(self._event_log[index:]), len(self._event_log)
+
+    def apply_events(self, events: list[dict]) -> int:
+        """Apply peer events not yet seen locally. Returns how many applied."""
+        applied = 0
+        with self._lock:
+            for event in events:
+                event_id = event.get("id")
+                if not event_id or event_id in self._seen_event_ids:
+                    continue
+                self._apply_locked(event)
+                applied += 1
+        return applied
+
+    def _apply_locked(self, event: dict) -> None:
+        etype = event.get("type")
+        if etype == "credit":
+            key = event["api_key"]
+            self._client_balances[key] = self._client_balances.get(key, 0.0) + float(event["amount"])
+        elif etype == "charge":
+            key = event["api_key"]
+            self._client_balances[key] = self._client_balances.get(key, 0.0) - float(event["cost"])
+            for wallet, amount in (event.get("shares") or {}).items():
+                self._node_pending[wallet] = self._node_pending.get(wallet, 0.0) + float(amount)
+                if amount > 0:
+                    self._pending_since.setdefault(wallet, float(event.get("ts", time.time())))
+            self._protocol_cut += float(event.get("protocol_amount", 0.0))
+        elif etype == "payout":
+            wallet = event["wallet"]
+            amount = float(event["amount"])
+            ts = float(event.get("ts", time.time()))
+            self._node_pending[wallet] = self._node_pending.get(wallet, 0.0) - amount
+            self._pending_since.pop(wallet, None)
+            self._last_payout_ts[wallet] = ts
+            reference = event.get("reference") or ""
+            if reference:
+                settlement = self._settlements.setdefault(
+                    reference, {"payouts": [], "signature": None, "ts": ts}
+                )
+                settlement["payouts"].append((wallet, amount))
+        elif etype == "settlement":
+            settlement = self._settlements.setdefault(
+                event["settlement_id"],
+                {"payouts": [], "signature": None, "ts": float(event.get("ts", 0.0))},
+            )
+            settlement["signature"] = event.get("signature")
+        elif etype == "forfeit":
+            wallet = event["wallet"]
+            amount = float(event.get("amount", 0.0))
+            self._node_pending[wallet] = self._node_pending.get(wallet, 0.0) - amount
+            self._protocol_cut += amount
+        elif etype == "bind":
+            self._wallet_bindings[event["wallet"]] = event["api_key"]
+        else:
+            return
+        self._seen_event_ids.add(event["id"])
+        self._event_log.append(event)
+        self._dirty = True
+
+    # ---- views ----
+
+    def get_client_balance(self, api_key: str) -> float:
+        with self._lock:
+            return self._client_balances.get(api_key, 0.0)
+
+    def get_node_pending(self, wallet: str) -> float:
+        with self._lock:
+            return self._node_pending.get(wallet, 0.0)
+
+    def snapshot(self) -> dict:
+        with self._lock:
+            return {
+                "clients": dict(self._client_balances),
+                "node_pending": dict(self._node_pending),
+                "wallet_bindings": dict(self._wallet_bindings),
+                "protocol_cut": self._protocol_cut,
+                "prices": dict(self._prices),
+                "default_price_per_1k_tokens": self._default_price_per_1k,
+                "node_share": self._node_share,
+                "event_count": len(self._event_log),
+                "forfeits": [
+                    {
+                        "wallet": e["wallet"],
+                        "amount": e.get("amount", 0.0),
+                        "reason": e.get("reason", ""),
+                        "ts": e.get("ts", 0.0),
+                    }
+                    for e in self._event_log if e.get("type") == "forfeit"
+                ][-20:],
+            }
+
+    # ---- persistence ----
+
+    def _init_db(self) -> None:
+        con = sqlite3.connect(self._db_path)  # type: ignore[arg-type]
+        con.execute(
+            "CREATE TABLE IF NOT EXISTS billing_events "
+            "(event_id TEXT PRIMARY KEY, payload TEXT NOT NULL, ts REAL NOT NULL)"
+        )
+        con.commit()
+        con.close()
+
+    def _load_from_db(self) -> None:
+        con = sqlite3.connect(self._db_path)  # type: ignore[arg-type]
+        rows = con.execute(
+            "SELECT payload FROM billing_events ORDER BY ts, event_id"
+        ).fetchall()
+        con.close()
+        with self._lock:
+            for (payload,) in rows:
+                try:
+                    event = json.loads(payload)
+                except json.JSONDecodeError:
+                    continue
+                if event.get("id") not in self._seen_event_ids:
+                    self._apply_locked(event)
+            self._dirty = False
+
+    def save_to_db(self) -> None:
+        if not self._db_path:
+            return
+        with self._lock:
+            if not self._dirty:
+                return
+            events = list(self._event_log)
+            self._dirty = False
+        con = sqlite3.connect(self._db_path)  # type: ignore[arg-type]
+        con.executemany(
+            "INSERT OR IGNORE INTO billing_events (event_id, payload, ts) VALUES (?, ?, ?)",
+            [(e["id"], json.dumps(e), float(e.get("ts", 0.0))) for e in events],
+        )
+        con.commit()
+        con.close()

packages/tracker/meshnet_tracker/cli.py

@@ -4,33 +4,115 @@ import argparse
 import sys
 import time
 
-from .server import TrackerServer
+from .server import TrackerServer, derive_relay_url_from_public_tracker_url
 
 
 def main() -> None:
-    parser = argparse.ArgumentParser(
+    common = argparse.ArgumentParser(add_help=False)
-        prog="meshnet-tracker",
+    common.add_argument("--host", default="0.0.0.0", help="Host interface to listen on")
-        description="Distributed Inference Network node registry and route selection",
+    common.add_argument("--port", type=int, default=8080, help="Port to listen on")
-    )
+    common.add_argument(
-    subparsers = parser.add_subparsers(dest="command")
-
-    start_cmd = subparsers.add_parser("start", help="Start the tracker server")
-    start_cmd.add_argument("--host", default="127.0.0.1", help="Host interface to listen on")
-    start_cmd.add_argument("--port", type=int, default=8080, help="Port to listen on")
-    start_cmd.add_argument(
         "--heartbeat-timeout",
         type=float,
         default=30.0,
         help="Seconds before a node is removed from the registry after missed heartbeat",
     )
+    common.add_argument(
+        "--cluster-peers",
+        default="",
+        help="Comma-separated URLs of peer tracker nodes (enables Raft cluster mode)",
+    )
+    common.add_argument(
+        "--self-url",
+        default=None,
+        help="This tracker's own URL as seen by peers (auto-derived from --host/--port if omitted)",
+    )
+    common.add_argument(
+        "--stats-db",
+        default=None,
+        metavar="PATH",
+        help="SQLite database path for persistent model usage statistics",
+    )
+    common.add_argument(
+        "--relay-url",
+        default=None,
+        help="Public ws(s):// relay URL advertised to nodes, for example wss://ai.neuron.d-popov.com/ws",
+    )
+    common.add_argument(
+        "--billing-db",
+        default=None,
+        metavar="PATH",
+        help="SQLite database path for the USDT billing ledger (enables billing; ADR-0015)",
+    )
+    common.add_argument(
+        "--solana-rpc-url",
+        default=None,
+        help="Solana RPC URL (e.g. https://api.devnet.solana.com); enables the on-chain treasury",
+    )
+    common.add_argument(
+        "--usdt-mint",
+        default=None,
+        help="SPL mint address of (mock) USDT — see scripts/devnet_setup.py",
+    )
+    common.add_argument(
+        "--treasury-keypair",
+        default=None,
+        metavar="PATH",
+        help="Treasury keypair JSON path (only on settlement-capable trackers)",
+    )
+    common.add_argument(
+        "--settle-period",
+        type=float,
+        default=86400.0,
+        help="Max seconds between payouts to a node (dev: 60, prod: 86400)",
+    )
+    common.add_argument(
+        "--payout-threshold",
+        type=float,
+        default=5.0,
+        help="Pending USDT that triggers an immediate payout (dev: 0)",
+    )
+    common.add_argument(
+        "--payout-dust-floor",
+        type=float,
+        default=0.01,
+        help="Never pay out less than this many USDT",
+    )
+
+    parser = argparse.ArgumentParser(
+        prog="meshnet-tracker",
+        description="Distributed Inference Network node registry and route selection",
+        parents=[common],
+    )
+    subparsers = parser.add_subparsers(dest="command")
+
+    subparsers.add_parser("start", help="Start the tracker server", parents=[common])
 
     args = parser.parse_args()
 
-    if args.command == "start":
+    if args.command in {None, "start"}:
+        cluster_peers = [u.strip() for u in args.cluster_peers.split(",") if u.strip()]
+        relay_url = args.relay_url or derive_relay_url_from_public_tracker_url(args.self_url)
+        treasury = None
+        if args.solana_rpc_url and args.usdt_mint and args.treasury_keypair:
+            from meshnet_contracts.solana_adapter import SolanaCustodialTreasury
+
+            treasury = SolanaCustodialTreasury(
+                args.solana_rpc_url, args.usdt_mint, args.treasury_keypair,
+            )
         server = TrackerServer(
             host=args.host,
             port=args.port,
             heartbeat_timeout=args.heartbeat_timeout,
+            cluster_peers=cluster_peers or None,
+            cluster_self_url=args.self_url,
+            stats_db=getattr(args, "stats_db", None),
+            relay_url=relay_url,
+            billing_db=getattr(args, "billing_db", None),
+            treasury=treasury,
+            settle_period=args.settle_period,
+            payout_threshold=args.payout_threshold,
+            payout_dust_floor=args.payout_dust_floor,
         )
         port = server.start()
         print(f"meshnet-tracker listening on http://{args.host}:{port}", flush=True)

packages/tracker/meshnet_tracker/dashboard.html

@@ -0,0 +1,197 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+<meta charset="utf-8">
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<title>meshnet tracker</title>
+<style>
+  :root { --bg:#0d1117; --panel:#161b22; --border:#30363d; --fg:#c9d1d9;
+          --dim:#8b949e; --accent:#58a6ff; --ok:#3fb950; --bad:#f85149; --warn:#d29922; }
+  * { box-sizing:border-box; }
+  body { margin:0; background:var(--bg); color:var(--fg);
+         font:13px/1.5 ui-monospace,SFMono-Regular,Menlo,Consolas,monospace; }
+  header { display:flex; align-items:baseline; gap:14px; padding:14px 20px;
+           border-bottom:1px solid var(--border); }
+  header h1 { font-size:16px; margin:0; color:var(--accent); }
+  header .meta { color:var(--dim); font-size:12px; }
+  main { display:grid; grid-template-columns:repeat(auto-fit,minmax(340px,1fr));
+         gap:14px; padding:14px 20px; }
+  section { background:var(--panel); border:1px solid var(--border);
+            border-radius:8px; padding:12px 14px; min-height:80px; }
+  section h2 { margin:0 0 8px; font-size:12px; text-transform:uppercase;
+               letter-spacing:.08em; color:var(--dim); }
+  table { width:100%; border-collapse:collapse; font-size:12px; }
+  th { text-align:left; color:var(--dim); font-weight:normal;
+       border-bottom:1px solid var(--border); padding:2px 6px 4px 0; }
+  td { padding:3px 6px 3px 0; border-bottom:1px solid #21262d; }
+  .ok { color:var(--ok); } .bad { color:var(--bad); } .warn { color:var(--warn); }
+  .dim { color:var(--dim); } .num { text-align:right; }
+  a { color:var(--accent); text-decoration:none; }
+  .empty { color:var(--dim); font-style:italic; }
+  .pill { display:inline-block; padding:0 7px; border-radius:9px;
+          border:1px solid var(--border); font-size:11px; }
+</style>
+</head>
+<body>
+<header>
+  <h1>meshnet tracker</h1>
+  <span class="meta" id="self-url"></span>
+  <span class="meta" id="refreshed"></span>
+</header>
+<main>
+  <section><h2>Tracker hive</h2><div id="hive" class="empty">loading…</div></section>
+  <section><h2>Nodes &amp; coverage</h2><div id="nodes" class="empty">loading…</div></section>
+  <section><h2>Client balances</h2><div id="clients" class="empty">loading…</div></section>
+  <section><h2>Node pending payouts</h2><div id="pending" class="empty">loading…</div></section>
+  <section><h2>Settlement history</h2><div id="settlements" class="empty">loading…</div></section>
+  <section><h2>Strikes / bans / forfeitures</h2><div id="fraud" class="empty">loading…</div></section>
+  <section><h2>Model usage (RPM)</h2><div id="stats" class="empty">loading…</div></section>
+</main>
+<script>
+"use strict";
+const $ = id => document.getElementById(id);
+const esc = s => String(s).replace(/[&<>"]/g,
+  c => ({"&":"&amp;","<":"&lt;",">":"&gt;",'"':"&quot;"}[c]));
+const usdt = v => (Math.round(v * 1e6) / 1e6).toFixed(6);
+const short = (s, n=14) => { s = String(s); return s.length > n ? s.slice(0, 6) + "…" + s.slice(-5) : s; };
+
+async function fetchJson(path) {
+  try {
+    const r = await fetch(path);
+    if (!r.ok) return null;
+    return await r.json();
+  } catch { return null; }
+}
+
+function table(headers, rows) {
+  if (!rows.length) return '<div class="empty">nothing yet</div>';
+  return '<table><tr>' + headers.map(h => `<th>${esc(h)}</th>`).join("") + '</tr>'
+    + rows.map(r => '<tr>' + r.map(c => `<td>${c}</td>`).join("") + '</tr>').join("")
+    + '</table>';
+}
+
+function renderHive(raft) {
+  const role = raft && (raft.state || raft.role);
+  if (!raft || role === "standalone") {
+    $("hive").innerHTML = '<span class="pill">standalone</span> single tracker, no cluster';
+    return;
+  }
+  const cls = role === "leader" ? "ok" : "warn";
+  $("hive").innerHTML =
+    `role <span class="${cls}">${esc(role || "?")}</span> · term ${esc(raft.term ?? "?")}` +
+    `<br>leader: ${raft.leader ? esc(raft.leader) : '<span class="dim">unknown</span>'}` +
+    (raft.peers ? `<br>peers: ${esc(Array.isArray(raft.peers) ? raft.peers.join(", ") : raft.peers)}` : "");
+}
+
+function renderNodes(map) {
+  const nodes = (map && map.nodes) || [];
+  if (!nodes.length) {
+    $("nodes").innerHTML = '<div class="empty">no nodes registered</div>'; return;
+  }
+  const byModel = {};
+  for (const n of nodes) {
+    const key = n.model || n.hf_repo || "?";
+    (byModel[key] = byModel[key] || []).push(n);
+  }
+  let html = "";
+  for (const [model, group] of Object.entries(byModel)) {
+    html += `<div><b>${esc(model)}</b> <span class="dim">(${group.length} node${group.length===1?"":"s"})</span></div>`;
+    html += table(["node", "shard", "mode", "health"], group.map(n => [
+      esc(short(n.node_id || "?")),
+      esc(`${n.shard_start ?? "?"}-${n.shard_end ?? "?"}`),
+      n.tracker_mode ? '<span class="pill">tracker</span>' : '<span class="dim">worker</span>',
+      (n.stats && (n.stats.alive === false || n.stats.healthy === false))
+        ? '<span class="bad">down</span>' : '<span class="ok">up</span>',
+    ]));
+  }
+  $("nodes").innerHTML = html;
+}
+
+function renderBilling(summary) {
+  if (!summary) {
+    $("clients").innerHTML = '<div class="empty">billing disabled</div>';
+    $("pending").innerHTML = '<div class="empty">billing disabled</div>';
+    return summary;
+  }
+  const clients = Object.entries(summary.clients || {});
+  $("clients").innerHTML = table(["api key", "balance (USDT)"],
+    clients.map(([k, v]) => [esc(short(k)),
+      `<span class="num ${v > 0 ? "ok" : "bad"}">${usdt(v)}</span>`]));
+  const pending = Object.entries(summary.node_pending || {});
+  const cut = summary.protocol_cut || 0;
+  $("pending").innerHTML =
+    table(["node wallet", "pending (USDT)"],
+      pending.map(([w, v]) => [esc(short(w)), `<span class="num">${usdt(v)}</span>`]))
+    + `<div style="margin-top:6px" class="dim">protocol cut: <b>${usdt(cut)}</b> USDT</div>`;
+  return summary;
+}
+
+function renderSettlements(data) {
+  if (!data || !data.settlements) {
+    $("settlements").innerHTML = '<div class="empty">billing disabled</div>'; return;
+  }
+  const rows = data.settlements.slice(-15).reverse().map(s => {
+    const total = (s.payouts || []).reduce((a, p) => a + p.amount, 0);
+    const sig = s.signature;
+    const link = sig && !sig.startsWith("fake-") && !sig.startsWith("local-")
+      ? `<a target="_blank" href="https://explorer.solana.com/tx/${encodeURIComponent(sig)}?cluster=devnet">${esc(short(sig))}</a>`
+      : (sig ? esc(short(sig)) : '<span class="warn">unconfirmed</span>');
+    return [new Date(s.ts * 1000).toLocaleTimeString(), String((s.payouts || []).length),
+            `<span class="num">${usdt(total)}</span>`, link];
+  });
+  $("settlements").innerHTML = table(["time", "payouts", "total (USDT)", "tx"], rows);
+}
+
+function renderFraud(wallets, summary) {
+  const rows = Object.entries((wallets && wallets.wallets) || {})
+    .filter(([, w]) => w.strike_count > 0 || w.banned)
+    .map(([addr, w]) => [esc(short(addr)), String(w.strike_count),
+      w.banned ? '<span class="bad">banned</span>' : '<span class="ok">active</span>']);
+  let html = rows.length ? table(["wallet", "strikes", "status"], rows)
+                         : '<div class="empty">no strikes recorded</div>';
+  const forfeits = (summary && summary.forfeits) || [];
+  if (forfeits.length) {
+    html += '<div style="margin-top:6px"><b class="dim">recent forfeitures</b></div>'
+      + table(["wallet", "amount", "reason"], forfeits.slice(-8).reverse().map(f =>
+        [esc(short(f.wallet)), `<span class="num bad">-${usdt(f.amount)}</span>`, esc(f.reason)]));
+  }
+  $("fraud").innerHTML = html;
+}
+
+function renderStats(stats) {
+  const models = (stats && (stats.models || stats.stats)) || stats;
+  if (!models || typeof models !== "object" || !Object.keys(models).length) {
+    $("stats").innerHTML = '<div class="empty">no requests yet</div>'; return;
+  }
+  const rows = Object.entries(models).map(([m, s]) => [
+    esc(m),
+    esc(String(s.rpm_last_hour ?? "?")),
+    esc(String(s.rpm_last_day ?? "?")),
+    esc(String(s.rpm_last_month ?? "?")),
+  ]);
+  $("stats").innerHTML = table(["model", "rpm (1h)", "rpm (24h)", "rpm (30d)"], rows);
+}
+
+async function refresh() {
+  $("self-url").textContent = location.host;
+  const [raft, map, summary, settlements, wallets, stats] = await Promise.all([
+    fetchJson("/v1/raft/status"),
+    fetchJson("/v1/network/map"),
+    fetchJson("/v1/billing/summary"),
+    fetchJson("/v1/billing/settlements"),
+    fetchJson("/v1/registry/wallets"),
+    fetchJson("/v1/stats"),
+  ]);
+  renderHive(raft);
+  renderNodes(map);
+  renderBilling(summary);
+  renderSettlements(settlements);
+  renderFraud(wallets, summary);
+  renderStats(stats);
+  $("refreshed").textContent = "refreshed " + new Date().toLocaleTimeString();
+}
+refresh();
+setInterval(refresh, 4000);
+</script>
+</body>
+</html>

packages/tracker/meshnet_tracker/gossip.py

@@ -0,0 +1,88 @@
+"""CRDT gossip for node liveness heartbeats.
+
+Uses a last-write-wins (LWW) register per inference node: each tracker node
+keeps its own copy of {node_id → last_seen_monotonic_timestamp} and merges
+incoming gossip by taking the max per key.  This is eventually consistent —
+a heartbeat received by one tracker propagates to all others within a few
+gossip intervals.
+
+Monotonic timestamps are local-clock-relative; for cross-machine gossip the
+caller should use wall-clock seconds (time.time()).  The tracker converts
+monotonic to wall-clock when recording and back when comparing.
+"""
+
+from __future__ import annotations
+
+import json
+import random
+import threading
+import time
+import urllib.request
+
+
+class NodeGossip:
+    """LWW gossip table for inference-node heartbeat timestamps.
+
+    ``record(node_id)`` is called when a node sends a heartbeat to *this*
+    tracker.  ``merge(remote)`` is called when gossip arrives from a peer
+    tracker.  The table is periodically pushed to one random peer.
+    """
+
+    PUSH_INTERVAL = 3.0  # seconds between gossip pushes to a random peer
+
+    def __init__(self, peers: list[str]) -> None:
+        self.peers = list(peers)
+        # Maps node_id → wall-clock seconds of last known heartbeat.
+        self._table: dict[str, float] = {}
+        self._lock = threading.Lock()
+        self._running = False
+
+    def start(self) -> None:
+        self._running = True
+        threading.Thread(target=self._push_loop, daemon=True, name="gossip").start()
+
+    def stop(self) -> None:
+        self._running = False
+
+    def record(self, node_id: str, wall_ts: float | None = None) -> None:
+        """Record a heartbeat for *node_id* at *wall_ts* (default: now)."""
+        ts = wall_ts if wall_ts is not None else time.time()
+        with self._lock:
+            if ts > self._table.get(node_id, 0.0):
+                self._table[node_id] = ts
+
+    def merge(self, remote: dict[str, float]) -> None:
+        """Merge a gossip snapshot from a peer tracker (LWW per key)."""
+        with self._lock:
+            for node_id, ts in remote.items():
+                if ts > self._table.get(node_id, 0.0):
+                    self._table[node_id] = ts
+
+    def last_seen(self, node_id: str) -> float | None:
+        """Return wall-clock timestamp of last known heartbeat, or None."""
+        with self._lock:
+            return self._table.get(node_id)
+
+    def snapshot(self) -> dict[str, float]:
+        with self._lock:
+            return dict(self._table)
+
+    def _push_loop(self) -> None:
+        while self._running:
+            time.sleep(self.PUSH_INTERVAL)
+            if not self.peers:
+                continue
+            peer = random.choice(self.peers)
+            try:
+                snap = self.snapshot()
+                body = json.dumps(snap).encode()
+                req = urllib.request.Request(
+                    f"{peer}/v1/gossip",
+                    data=body,
+                    headers={"Content-Type": "application/json"},
+                    method="POST",
+                )
+                with urllib.request.urlopen(req, timeout=2.0) as r:
+                    r.read()
+            except Exception:
+                pass

packages/tracker/meshnet_tracker/model_presets.json

@@ -0,0 +1,37 @@
+{
+  "models": {
+    "kimi-k2.7": {
+      "layers_start": 0,
+      "layers_end": 60,
+      "hf_repo": "unsloth/Kimi-K2.7-Code",
+      "aliases": [
+        "kimi-k2.7",
+        "Kimi-K2.7-Code",
+        "unsloth/Kimi-K2.7-Code"
+      ],
+      "recommended": true,
+      "deployment_status": "recommended",
+      "required_model_bytes": 638876385280,
+      "download_size_bytes": 638876385280,
+      "native_quantization": "int4",
+      "bytes_per_layer": {
+        "int4": 10473383366
+      },
+      "metadata": {
+        "architecture": "Mixture-of-Experts (MoE)",
+        "total_parameters": "1T",
+        "activated_parameters": "32B",
+        "num_layers": 61,
+        "context_length": 256000,
+        "native_quantization": "int4",
+        "download_size_gb": 595,
+        "recommended_short_name": "kimi-k2.7",
+        "recommended_engines": [
+          "vLLM",
+          "SGLang",
+          "KTransformers"
+        ]
+      }
+    }
+  }
+}

packages/tracker/meshnet_tracker/raft.py

@@ -0,0 +1,372 @@
+"""Minimal Raft consensus for tracker shard assignments.
+
+Only shard-assignment commands (register/deregister) go through the log.
+Node liveness (heartbeats) is handled separately via CRDT gossip — these
+are high-frequency writes where eventual consistency is fine.
+
+Election timeout: random 150–300 ms (tight, suits in-process tests).
+Leader heartbeat interval: 50 ms.
+"""
+
+from __future__ import annotations
+
+import json
+import random
+import threading
+import time
+import urllib.error
+import urllib.request
+from dataclasses import dataclass, field
+from typing import Any, Callable
+
+
+@dataclass
+class LogEntry:
+    term: int
+    command: str   # "register" | "deregister"
+    payload: dict
+
+
+class RaftNode:
+    """Single Raft participant.
+
+    ``apply_fn(command, payload)`` is called (under no external lock) when an
+    entry is committed.  Implementors must apply the command atomically.
+    """
+
+    ELECTION_MIN = 0.15   # seconds
+    ELECTION_MAX = 0.30
+    HB_INTERVAL  = 0.05   # leader heartbeat interval
+
+    # Role constants
+    FOLLOWER  = "follower"
+    CANDIDATE = "candidate"
+    LEADER    = "leader"
+
+    def __init__(
+        self,
+        self_url: str,
+        peers: list[str],
+        apply_fn: Callable[[str, dict], None],
+    ) -> None:
+        self.self_url = self_url
+        self.peers = list(peers)
+        self._apply_fn = apply_fn
+
+        self._lock = threading.Lock()
+        self.role: str = self.FOLLOWER
+        self.current_term: int = 0
+        self.voted_for: str | None = None
+        self.log: list[LogEntry] = []
+        self.commit_index: int = -1
+        self.last_applied: int = -1
+        self.leader_url: str | None = None
+
+        # Leader bookkeeping per peer
+        self._next_index:  dict[str, int] = {}
+        self._match_index: dict[str, int] = {}
+
+        self._election_deadline: float = self._fresh_deadline()
+        self._running = False
+
+    # ------------------------------------------------------------------ start/stop
+
+    def start(self) -> None:
+        self._running = True
+        threading.Thread(target=self._tick_loop, daemon=True, name="raft-tick").start()
+
+    def stop(self) -> None:
+        self._running = False
+
+    # ------------------------------------------------------------------ public API
+
+    @property
+    def is_leader(self) -> bool:
+        with self._lock:
+            return self.role == self.LEADER
+
+    def leader(self) -> str | None:
+        with self._lock:
+            return self.leader_url
+
+    def status(self) -> dict:
+        with self._lock:
+            return {
+                "role": self.role,
+                "term": self.current_term,
+                "leader": self.leader_url,
+                "log_length": len(self.log),
+                "commit_index": self.commit_index,
+            }
+
+    def propose(self, command: str, payload: dict) -> bool:
+        """Leader: append and replicate an entry. Returns True when committed.
+
+        Blocks until majority replication or failure.  Must be called only on
+        the leader; returns False immediately if this node is not the leader.
+        """
+        with self._lock:
+            if self.role != self.LEADER:
+                return False
+            entry = LogEntry(self.current_term, command, payload)
+            self.log.append(entry)
+            entry_index = len(self.log) - 1
+            term = self.current_term
+
+        self._replicate_to_peers(term)
+
+        with self._lock:
+            return self.commit_index >= entry_index
+
+    # ------------------------------------------------------------------ RPC handlers
+
+    def handle_request_vote(self, req: dict) -> dict:
+        with self._lock:
+            term        = int(req["term"])
+            candidate   = req["candidate_url"]
+            last_li     = int(req["last_log_index"])
+            last_lt     = int(req["last_log_term"])
+
+            if term > self.current_term:
+                self._step_down(term)
+
+            if term < self.current_term:
+                return {"term": self.current_term, "vote_granted": False}
+
+            my_li = len(self.log) - 1
+            my_lt = self.log[-1].term if self.log else 0
+            log_ok = (last_lt > my_lt) or (last_lt == my_lt and last_li >= my_li)
+
+            if (self.voted_for is None or self.voted_for == candidate) and log_ok:
+                self.voted_for = candidate
+                self._reset_deadline()
+                return {"term": self.current_term, "vote_granted": True}
+            return {"term": self.current_term, "vote_granted": False}
+
+    def handle_append_entries(self, req: dict) -> dict:
+        with self._lock:
+            term         = int(req["term"])
+            leader_url   = req["leader_url"]
+            prev_li      = int(req["prev_log_index"])
+            prev_lt      = int(req["prev_log_term"])
+            entries_raw  = req.get("entries", [])
+            ldr_commit   = int(req["leader_commit"])
+
+            if term > self.current_term:
+                self._step_down(term)
+
+            if term < self.current_term:
+                return {"term": self.current_term, "success": False}
+
+            # Valid AppendEntries from current leader
+            self._reset_deadline()
+            self.leader_url = leader_url
+            if self.role == self.CANDIDATE:
+                self.role = self.FOLLOWER
+
+            # Consistency check on prev entry
+            if prev_li >= 0:
+                if prev_li >= len(self.log):
+                    return {"term": self.current_term, "success": False}
+                if self.log[prev_li].term != prev_lt:
+                    self.log = self.log[:prev_li]
+                    return {"term": self.current_term, "success": False}
+
+            # Append new entries (detect and overwrite conflicts)
+            for i, raw in enumerate(entries_raw):
+                idx = prev_li + 1 + i
+                entry = LogEntry(int(raw["term"]), raw["command"], raw["payload"])
+                if idx < len(self.log):
+                    if self.log[idx].term != entry.term:
+                        self.log = self.log[:idx]
+                        self.log.append(entry)
+                else:
+                    self.log.append(entry)
+
+            if ldr_commit > self.commit_index:
+                self.commit_index = min(ldr_commit, len(self.log) - 1)
+                self._apply_up_to_commit()
+
+            return {"term": self.current_term, "success": True}
+
+    # ------------------------------------------------------------------ internals
+
+    def _tick_loop(self) -> None:
+        while self._running:
+            time.sleep(0.01)
+            with self._lock:
+                role     = self.role
+                deadline = self._election_deadline
+
+            if role == self.LEADER:
+                self._send_heartbeats()
+                time.sleep(self.HB_INTERVAL)
+            elif time.monotonic() > deadline:
+                self._start_election()
+
+    def _send_heartbeats(self) -> None:
+        with self._lock:
+            term         = self.current_term
+            ldr_commit   = self.commit_index
+            log_snapshot = list(self.log)
+
+        for peer in self.peers:
+            try:
+                self._send_append_entries(peer, term, ldr_commit, log_snapshot)
+            except Exception:
+                pass
+
+    def _replicate_to_peers(self, term: int) -> None:
+        """Send AppendEntries to all peers and update commit_index on majority ack."""
+        with self._lock:
+            ldr_commit   = self.commit_index
+            log_snapshot = list(self.log)
+
+        acks = 1  # leader counts as 1
+        for peer in self.peers:
+            try:
+                ok = self._send_append_entries(peer, term, ldr_commit, log_snapshot)
+                if ok:
+                    acks += 1
+            except Exception:
+                pass
+
+        # Advance commit if majority replicated
+        with self._lock:
+            if self.role != self.LEADER or self.current_term != term:
+                return
+            majority_index = len(self.log) - 1
+            while majority_index > self.commit_index:
+                if self.log[majority_index].term == self.current_term:
+                    count = 1 + sum(
+                        1 for p in self.peers
+                        if self._match_index.get(p, -1) >= majority_index
+                    )
+                    if count > (len(self.peers) + 1) / 2:
+                        self.commit_index = majority_index
+                        self._apply_up_to_commit()
+                        break
+                majority_index -= 1
+
+    def _send_append_entries(
+        self, peer: str, term: int, ldr_commit: int, log_snapshot: list[LogEntry]
+    ) -> bool:
+        with self._lock:
+            next_idx = self._next_index.get(peer, len(log_snapshot))
+
+        prev_li = next_idx - 1
+        prev_lt = log_snapshot[prev_li].term if 0 <= prev_li < len(log_snapshot) else 0
+        entries = [
+            {"term": e.term, "command": e.command, "payload": e.payload}
+            for e in log_snapshot[next_idx:]
+        ]
+
+        body = json.dumps({
+            "term": term,
+            "leader_url": self.self_url,
+            "prev_log_index": prev_li,
+            "prev_log_term": prev_lt,
+            "entries": entries,
+            "leader_commit": ldr_commit,
+        }).encode()
+        req = urllib.request.Request(
+            f"{peer}/v1/raft/append",
+            data=body,
+            headers={"Content-Type": "application/json"},
+            method="POST",
+        )
+        with urllib.request.urlopen(req, timeout=0.15) as r:
+            resp = json.loads(r.read())
+
+        with self._lock:
+            if resp.get("success"):
+                new_match = len(log_snapshot) - 1
+                self._match_index[peer] = max(self._match_index.get(peer, -1), new_match)
+                self._next_index[peer]  = new_match + 1
+                return True
+            else:
+                if resp.get("term", 0) > self.current_term:
+                    self._step_down(resp["term"])
+                else:
+                    self._next_index[peer] = max(0, self._next_index.get(peer, 1) - 1)
+                return False
+
+    def _start_election(self) -> None:
+        with self._lock:
+            self.current_term += 1
+            self.role = self.CANDIDATE
+            self.voted_for = self.self_url
+            term  = self.current_term
+            my_li = len(self.log) - 1
+            my_lt = self.log[-1].term if self.log else 0
+            self._reset_deadline()
+
+        votes = 1
+        for peer in self.peers:
+            try:
+                body = json.dumps({
+                    "term": term,
+                    "candidate_url": self.self_url,
+                    "last_log_index": my_li,
+                    "last_log_term": my_lt,
+                }).encode()
+                req = urllib.request.Request(
+                    f"{peer}/v1/raft/vote",
+                    data=body,
+                    headers={"Content-Type": "application/json"},
+                    method="POST",
+                )
+                with urllib.request.urlopen(req, timeout=0.1) as r:
+                    resp = json.loads(r.read())
+                if resp.get("vote_granted"):
+                    votes += 1
+                elif resp.get("term", 0) > term:
+                    with self._lock:
+                        self._step_down(resp["term"])
+                    return
+            except Exception:
+                pass
+
+        with self._lock:
+            if self.role == self.CANDIDATE and self.current_term == term:
+                total = len(self.peers) + 1
+                if votes > total / 2:
+                    self._become_leader()
+                else:
+                    self.role = self.FOLLOWER
+                    self._reset_deadline()
+
+    def _become_leader(self) -> None:
+        """Must be called with _lock held."""
+        self.role = self.LEADER
+        self.leader_url = self.self_url
+        for peer in self.peers:
+            self._next_index[peer]  = len(self.log)
+            self._match_index[peer] = -1
+        print(f"[raft] {self.self_url} became leader (term {self.current_term})", flush=True)
+
+    def _step_down(self, new_term: int) -> None:
+        """Must be called with _lock held."""
+        self.current_term = new_term
+        self.role = self.FOLLOWER
+        self.voted_for = None
+        self._reset_deadline()
+
+    def _apply_up_to_commit(self) -> None:
+        """Must be called with _lock held."""
+        while self.last_applied < self.commit_index:
+            self.last_applied += 1
+            entry = self.log[self.last_applied]
+            try:
+                self._apply_fn(entry.command, entry.payload)
+            except Exception:
+                pass
+
+    def _reset_deadline(self) -> None:
+        self._election_deadline = self._fresh_deadline()
+
+    @staticmethod
+    def _fresh_deadline() -> float:
+        return time.monotonic() + random.uniform(
+            RaftNode.ELECTION_MIN, RaftNode.ELECTION_MAX
+        )

packages/tracker/pyproject.toml

@@ -8,9 +8,16 @@ version = "0.1.0"
 description = "Distributed Inference Network node registry and route selection"
 requires-python = ">=3.10"
 
+dependencies = [
+    "websockets>=13",
+]
+
 [project.scripts]
 meshnet-tracker = "meshnet_tracker.cli:main"
 
 [tool.setuptools.packages.find]
 where = ["."]
 include = ["meshnet_tracker*"]
+
+[tool.setuptools.package-data]
+meshnet_tracker = ["*.json", "*.html"]

packages/validator/README.md

@@ -0,0 +1,46 @@
+# meshnet-validator
+
+Optimistic fraud detection (ADR-0003, penalty amended by ADR-0015): the
+validator re-runs a random ~5% sample of completed inference requests against
+a trusted reference node and, on divergence, submits a slash proof and
+forfeits the node's pending balance.
+
+## Why the penalty deters cheating
+
+There is no upfront stake. Settlement is periodic (US-033), so a node always
+has an unpaid **pending balance** — that balance *is* the collateral.
+
+At a sampling rate `p`, a cheater is caught on average once every `1/p`
+fraudulent jobs, so cheating is unprofitable when:
+
+```
+penalty > per_job_gain / p        # p = 0.05  →  penalty > 20 × per_job_gain
+```
+
+With the production settlement period of 24h, the pending balance at any
+moment approximates a full day's earnings — hundreds to thousands of jobs —
+which is far above the 20× bar. Each catch also records a strike; three
+strikes ban the wallet (registration rejected, excluded from routes, unpaid
+pending never settled), and the probationary period (first N jobs unpaid)
+makes re-entry with a fresh wallet costly.
+
+Two operational notes:
+
+- Shortening the settlement period shrinks the collateral. Period changes
+  must weigh chain overhead against deterrence.
+- A cheater immediately after a payout has little to forfeit — the
+  strike/ban ladder covers that window.
+
+## Usage
+
+```python
+ValidatorProcess(
+    contracts=contracts,          # registry/validation boundary
+    billing=ledger,               # BillingLedger — enables forfeiture
+    reference_node_url="http://...",
+    sample_rate=0.05,
+)
+```
+
+Remote validators can instead call the tracker's privileged
+`POST /v1/billing/forfeit` endpoint (non-empty Authorization header).

packages/validator/meshnet_validator/__init__.py

@@ -26,6 +26,7 @@ class ValidatorProcess:
         random_seed: int | None = None,
         webhook_url: str | None = None,
         interval_seconds: float = 1.0,
+        billing: Any | None = None,
     ) -> None:
         if not 0.0 <= sample_rate <= 1.0:
             raise ValueError("sample_rate must be between 0 and 1")
@@ -39,6 +40,7 @@ class ValidatorProcess:
             raise ValueError("interval_seconds must be positive")
 
         self._contracts = contracts
+        self._billing = billing
         self._reference_node_url = reference_node_url.rstrip("/")
         self._sample_rate = sample_rate
         self._tolerance = tolerance
@@ -120,6 +122,15 @@ class ValidatorProcess:
             ),
             webhook_url=self._webhook_url,
         ))
+        # ADR-0015: the pending balance is the collateral — forfeit it in the
+        # same validation cycle as the strike.
+        if self._billing is not None:
+            forfeit = self._billing.forfeit_pending(wallet_address, reason="fraud-divergence")
+            print(
+                f"[validator] forfeited pending balance of {wallet_address}: "
+                f"{forfeit['amount']:.6f} USDT (fraud-divergence)",
+                flush=True,
+            )
         return receipts
 
 

pyproject.toml

@@ -11,6 +11,9 @@ requires-python = ">=3.10"
 [project.optional-dependencies]
 dev = ["pytest>=8", "openai>=1", "langchain-openai>=0.1"]
 
+[tool.setuptools]
+packages = []
+
 [tool.pytest.ini_options]
 testpaths = ["tests"]
 markers = [

scripts/devnet_setup.py

@@ -0,0 +1,108 @@
+#!/usr/bin/env python3
+"""Devnet custodial treasury setup (ADR-0015, US-032).
+
+Creates the mock-USDT SPL mint (6 decimals, matching real USDT) and the
+treasury associated token account on Solana devnet, then prints/writes the
+.env.devnet values. Real USDT exists only on mainnet — the mint address is
+config, so mainnet cutover is a config change.
+
+Usage:
+  python scripts/devnet_setup.py                                  # full setup
+  python scripts/devnet_setup.py --mint <ADDR>                    # reuse existing mint
+  python scripts/devnet_setup.py --mint <ADDR> --mint-to <WALLET> --amount 25
+"""
+
+import argparse
+import json
+import os
+import sys
+import time
+
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "packages", "contracts"))
+
+from solders.keypair import Keypair  # noqa: E402
+
+from meshnet_contracts.solana_adapter import SolanaCustodialTreasury  # noqa: E402
+
+DEFAULT_KEYPAIR = os.path.expanduser("~/.config/solana/meshnet-treasury.json")
+DEFAULT_RPC = "https://api.devnet.solana.com"
+# placeholder mint for bootstrapping the adapter before the real mint exists
+_SYSTEM_PROGRAM = "11111111111111111111111111111111"
+
+
+def _load_or_create_keypair(path: str) -> Keypair:
+    if os.path.exists(path):
+        with open(path, encoding="utf-8") as fh:
+            return Keypair.from_bytes(bytes(json.load(fh)))
+    keypair = Keypair()
+    os.makedirs(os.path.dirname(path), exist_ok=True)
+    with open(path, "w", encoding="utf-8") as fh:
+        json.dump(list(bytes(keypair)), fh)
+    os.chmod(path, 0o600)
+    print(f"generated treasury keypair: {path}")
+    return keypair
+
+
+def _ensure_sol(treasury: SolanaCustodialTreasury, minimum_sol: float = 0.5) -> None:
+    balance = treasury.get_sol_balance()
+    if balance >= minimum_sol:
+        print(f"treasury SOL balance: {balance:.4f}")
+        return
+    print(f"airdropping 2 SOL to {treasury.treasury_wallet} (balance {balance:.4f})…")
+    signature = treasury.request_airdrop(2.0)
+    for _ in range(30):
+        time.sleep(2)
+        if treasury.get_sol_balance() >= minimum_sol:
+            print("airdrop confirmed")
+            return
+    sys.exit(
+        f"airdrop {signature} not confirmed — devnet faucet may be rate-limited; retry later"
+    )
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--rpc-url", default=DEFAULT_RPC)
+    parser.add_argument("--keypair", default=DEFAULT_KEYPAIR, help="Treasury keypair path")
+    parser.add_argument("--mint", default=None, help="Reuse an existing mock-USDT mint address")
+    parser.add_argument("--mint-to", default=None, metavar="WALLET",
+                        help="Mint mock USDT to a client wallet (creates their token account)")
+    parser.add_argument("--amount", type=float, default=100.0, help="Amount for --mint-to")
+    parser.add_argument("--env-out", default=".env.devnet", help="Env file to write")
+    args = parser.parse_args()
+
+    keypair = _load_or_create_keypair(args.keypair)
+    bootstrap = SolanaCustodialTreasury(args.rpc_url, args.mint or _SYSTEM_PROGRAM, keypair)
+    print(f"treasury wallet: {bootstrap.treasury_wallet}")
+    _ensure_sol(bootstrap)
+
+    if args.mint:
+        treasury = SolanaCustodialTreasury(args.rpc_url, args.mint, keypair)
+        mint_address = args.mint
+        print(f"using existing mock-USDT mint: {mint_address}")
+    else:
+        treasury, mint_address = bootstrap.create_mock_usdt_mint()
+        print(f"created mock-USDT mint (6 decimals): {mint_address}")
+        time.sleep(2)  # let the mint transaction confirm before creating ATAs
+
+    treasury.ensure_token_account(treasury.treasury_wallet)
+    print(f"treasury token account: {treasury.treasury_token_account}")
+
+    if args.mint_to:
+        signature = treasury.mint_mock_usdt(args.mint_to, args.amount)
+        print(f"minted {args.amount} mock USDT to {args.mint_to} (tx {signature})")
+
+    env = (
+        f"MESHNET_SOLANA_RPC_URL={args.rpc_url}\n"
+        f"MESHNET_USDT_MINT={mint_address}\n"
+        f"MESHNET_TREASURY_KEYPAIR={args.keypair}\n"
+        f"MESHNET_TREASURY_WALLET={treasury.treasury_wallet}\n"
+        f"MESHNET_TREASURY_TOKEN_ACCOUNT={treasury.treasury_token_account}\n"
+    )
+    with open(args.env_out, "w", encoding="utf-8") as fh:
+        fh.write(env)
+    print(f"\nwrote {args.env_out}:\n{env}")
+
+
+if __name__ == "__main__":
+    main()

scripts/ralph_progress.py

@@ -33,7 +33,7 @@ from typing import Any
 
 
 REPO_ROOT = Path(__file__).resolve().parents[1]
-DEFAULT_PRD = REPO_ROOT / ".scratch/distributed-inference-network/prd.json"
+DEFAULT_PRD = REPO_ROOT / "docs/prd.json"
 DEFAULT_AGENT = "codex"
 DEFAULT_INTERVAL = 10.0
 AGENT_CONFIG_PATH = REPO_ROOT / ".ralph-tui" / "agent-config.json"
@@ -152,7 +152,7 @@ def _story_meta(
         parts.append(f"worktree: {wt}")
 
     # summary -------------------------------------------------------------
-    notes = story.get("completionNotes", "").strip()
+    notes = (story.get("completionNotes") or "").strip()
     if not notes and wt:
         notes = _story_last_commit(sid)
     if notes:

scripts/test_lan_inference.py

@@ -0,0 +1,163 @@
+#!/usr/bin/env python3
+"""
+End-to-end LAN inference test for meshnet distributed inference.
+
+Sends 3 chat-completion requests to a meshnet node, validates OpenAI-format
+responses, and prints token counts + latency per request.
+
+Usage:
+    python scripts/test_lan_inference.py \\
+        --tracker http://192.168.1.10:8080 \\
+        --gateway http://192.168.1.10:8001
+
+Exit 0 on success, 1 on any failure.
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+import time
+import urllib.error
+import urllib.parse
+import urllib.request
+
+
+PROMPTS = [
+    {"role": "user", "content": "What is 7 × 8? Answer in one word."},
+    {"role": "user", "content": "Name the capital of France in one word."},
+    {"role": "user", "content": "Complete the sequence: 1, 1, 2, 3, 5, ___. Answer in one word."},
+]
+
+MODEL = "microsoft/Phi-3-medium-128k-instruct"
+
+
+def _get(url: str, timeout: float = 10.0) -> dict:
+    with urllib.request.urlopen(url, timeout=timeout) as r:
+        return json.loads(r.read())
+
+
+def _post(url: str, payload: dict, timeout: float = 60.0) -> dict:
+    data = json.dumps(payload).encode()
+    req = urllib.request.Request(
+        url, data=data, headers={"Content-Type": "application/json"}, method="POST"
+    )
+    with urllib.request.urlopen(req, timeout=timeout) as r:
+        return json.loads(r.read())
+
+
+def discover_gateway(tracker_url: str) -> str:
+    """Return the first tracker-mode node endpoint for MODEL."""
+    nodes = _get(f"{tracker_url}/v1/nodes", timeout=5.0)
+    if isinstance(nodes, dict):
+        nodes = list(nodes.values())
+    tracker_nodes = [
+        n for n in nodes
+        if n.get("tracker_mode") and (
+            n.get("hf_repo") == MODEL or n.get("model") == MODEL.split("/")[-1]
+        )
+    ]
+    if not tracker_nodes:
+        raise RuntimeError(
+            f"No tracker-mode nodes found for {MODEL!r}. "
+            "Is the first-shard node running and registered?"
+        )
+    endpoint: str = tracker_nodes[0]["endpoint"]
+    return endpoint.rstrip("/")
+
+
+def check_route(tracker_url: str, gateway_url: str) -> list[str]:
+    """Return the full inference route for MODEL."""
+    url = f"{tracker_url}/v1/route?model={urllib.parse.quote(MODEL)}"
+    try:
+        resp = _get(url, timeout=5.0)
+        return resp.get("route", [])
+    except Exception as exc:
+        print(f"  Warning: could not fetch route: {exc}", file=sys.stderr)
+        return [gateway_url]
+
+
+def run_inference(gateway_url: str, messages: list[dict]) -> tuple[str, int, float]:
+    """Send one chat-completion request. Returns (content, tokens, elapsed_s)."""
+    t0 = time.monotonic()
+    resp = _post(
+        f"{gateway_url}/v1/chat/completions",
+        {"model": MODEL, "messages": messages, "stream": False},
+        timeout=120.0,
+    )
+    elapsed = time.monotonic() - t0
+
+    choices = resp.get("choices")
+    if not choices:
+        raise ValueError(f"No choices in response: {resp}")
+    content: str = choices[0].get("message", {}).get("content", "")
+    if not isinstance(content, str):
+        raise TypeError(f"Expected string content, got {type(content)}: {content}")
+
+    usage = resp.get("usage", {})
+    tokens: int = usage.get("completion_tokens", len(content.split()))
+
+    return content, tokens, elapsed
+
+
+def main(argv: list[str] | None = None) -> int:
+    p = argparse.ArgumentParser(description=__doc__)
+    p.add_argument("--tracker", required=True, help="Tracker URL, e.g. http://192.168.1.10:8080")
+    p.add_argument(
+        "--gateway",
+        default=None,
+        help="Inference entry point URL. Auto-discovered from tracker if omitted.",
+    )
+    args = p.parse_args(argv)
+
+    tracker_url = args.tracker.rstrip("/")
+
+    print(f"Tracker: {tracker_url}")
+
+    # Resolve gateway
+    gateway_url = args.gateway.rstrip("/") if args.gateway else None
+    if gateway_url is None:
+        try:
+            gateway_url = discover_gateway(tracker_url)
+            print(f"Gateway (auto-discovered): {gateway_url}")
+        except Exception as exc:
+            print(f"ERROR: {exc}", file=sys.stderr)
+            return 1
+    else:
+        print(f"Gateway: {gateway_url}")
+
+    # Show route
+    route = check_route(tracker_url, gateway_url)
+    print(f"Route: {route}")
+    if len(route) < 2:
+        print("  Warning: only one node in route — is the second-shard node registered?")
+    print()
+
+    failures = 0
+    for i, msg in enumerate(PROMPTS, start=1):
+        print(f"[{i}] Q: {msg['content']}")
+        try:
+            content, tokens, elapsed = run_inference(gateway_url, [msg])
+            tps = tokens / elapsed if elapsed > 0 else 0.0
+            print(f"     A: {content.strip()}")
+            print(f"     {tokens} tokens  {elapsed:.2f}s  {tps:.1f} t/s")
+        except urllib.error.HTTPError as exc:
+            body = exc.read().decode(errors="replace")
+            print(f"     ERROR {exc.code}: {body}", file=sys.stderr)
+            failures += 1
+        except Exception as exc:
+            print(f"     ERROR: {exc}", file=sys.stderr)
+            failures += 1
+        print()
+
+    if failures == 0:
+        print(f"All {len(PROMPTS)} requests completed successfully.")
+        print("Exit code: 0")
+        return 0
+    else:
+        print(f"{failures}/{len(PROMPTS)} requests failed.", file=sys.stderr)
+        return 1
+
+
+if __name__ == "__main__":
+    sys.exit(main())

tests/conftest.py

@@ -0,0 +1,19 @@
+"""Shared pytest fixtures for the meshnet test suite."""
+
+import pytest
+
+
+@pytest.fixture(autouse=True)
+def _stub_benchmark_throughput(monkeypatch):
+    """Replace the GEMM benchmark with a fixed value in all tests.
+
+    The benchmark runs 60 matmuls (warmup + measure) which adds ~100ms per test
+    on CPU. Tests verify registration flow, not hardware speed — stub it out.
+    Tests that specifically exercise benchmark_throughput import it directly from
+    meshnet_node.hardware and are not affected by this patch.
+    """
+    try:
+        import meshnet_node.startup as startup_mod
+        monkeypatch.setattr(startup_mod, "benchmark_throughput_checked", lambda _device: (999.0, True, None))
+    except ImportError:
+        pass

tests/test_billing_ledger.py

@@ -0,0 +1,294 @@
+"""US-031: billing ledger — per-token pricing, 90/10 split, pending balances.
+
+Unit tests for BillingLedger math/persistence/replication, plus HTTP
+integration: 401 without an API key, billed 200 with one, 402 once the
+Caller Credit is exhausted (rejected before routing — no free work).
+"""
+
+import http.server
+import json
+import socketserver
+import threading
+import urllib.error
+import urllib.request
+
+import pytest
+
+from meshnet_tracker.billing import BillingLedger
+from meshnet_tracker.server import TrackerServer
+
+MODEL = "openai-community/gpt2"
+
+
+# ---------------------------------------------------------------- unit tests
+
+
+def test_charge_single_node_gets_90_percent():
+    ledger = BillingLedger(starting_credit=1.0, default_price_per_1k=0.02)
+    ledger.ensure_client("key-a")
+    event = ledger.charge_request("key-a", MODEL, total_tokens=1000, node_work=[("wallet-1", 12)])
+    assert event["cost"] == pytest.approx(0.02)
+    assert ledger.get_client_balance("key-a") == pytest.approx(1.0 - 0.02)
+    assert ledger.get_node_pending("wallet-1") == pytest.approx(0.02 * 0.90)
+    assert ledger.snapshot()["protocol_cut"] == pytest.approx(0.02 * 0.10)
+
+
+def test_charge_three_node_split_by_work_units():
+    ledger = BillingLedger(starting_credit=1.0, default_price_per_1k=0.02)
+    ledger.ensure_client("key-a")
+    ledger.charge_request(
+        "key-a", MODEL, total_tokens=1000,
+        node_work=[("wallet-1", 6), ("wallet-2", 3), ("wallet-3", 3)],
+    )
+    pool = 0.02 * 0.90
+    assert ledger.get_node_pending("wallet-1") == pytest.approx(pool * 6 / 12)
+    assert ledger.get_node_pending("wallet-2") == pytest.approx(pool * 3 / 12)
+    assert ledger.get_node_pending("wallet-3") == pytest.approx(pool * 3 / 12)
+    assert ledger.snapshot()["protocol_cut"] == pytest.approx(0.02 * 0.10)
+
+
+def test_walletless_node_share_accrues_to_protocol_cut():
+    ledger = BillingLedger(starting_credit=1.0, default_price_per_1k=0.02)
+    ledger.ensure_client("key-a")
+    ledger.charge_request(
+        "key-a", MODEL, total_tokens=1000,
+        node_work=[("wallet-1", 6), (None, 6)],
+    )
+    pool = 0.02 * 0.90
+    assert ledger.get_node_pending("wallet-1") == pytest.approx(pool / 2)
+    # walletless half of the pool + the 10% cut both land in protocol_cut
+    assert ledger.snapshot()["protocol_cut"] == pytest.approx(pool / 2 + 0.02 * 0.10)
+
+
+def test_per_model_price_override():
+    ledger = BillingLedger(
+        starting_credit=1.0,
+        default_price_per_1k=0.02,
+        prices={MODEL: 0.10},
+    )
+    ledger.ensure_client("key-a")
+    event = ledger.charge_request("key-a", MODEL, 500, [("wallet-1", 1)])
+    assert event["cost"] == pytest.approx(0.10 * 500 / 1000)
+    event = ledger.charge_request("key-a", "other-model", 500, [("wallet-1", 1)])
+    assert event["cost"] == pytest.approx(0.02 * 500 / 1000)
+
+
+def test_payout_and_forfeit_hooks():
+    ledger = BillingLedger(starting_credit=1.0, default_price_per_1k=0.02)
+    ledger.charge_request("key-a", MODEL, 1000, [("wallet-1", 1)])
+    pending = ledger.get_node_pending("wallet-1")
+    ledger.settle_node_payout("wallet-1", pending, reference="tx-sig-1")
+    assert ledger.get_node_pending("wallet-1") == pytest.approx(0.0)
+
+    ledger.charge_request("key-a", MODEL, 1000, [("wallet-1", 1)])
+    cut_before = ledger.snapshot()["protocol_cut"]
+    forfeited = ledger.forfeit_pending("wallet-1")["amount"]
+    assert forfeited == pytest.approx(0.02 * 0.90)
+    assert ledger.get_node_pending("wallet-1") == pytest.approx(0.0)
+    assert ledger.snapshot()["protocol_cut"] == pytest.approx(cut_before + forfeited)
+
+
+def test_restart_persistence(tmp_path):
+    db = str(tmp_path / "billing.db")
+    ledger = BillingLedger(db_path=db, starting_credit=1.0, default_price_per_1k=0.02)
+    ledger.credit_client("key-a", 5.0)
+    ledger.charge_request("key-a", MODEL, 1000, [("wallet-1", 12)])
+    ledger.save_to_db()
+
+    reloaded = BillingLedger(db_path=db, starting_credit=1.0, default_price_per_1k=0.02)
+    assert reloaded.get_client_balance("key-a") == pytest.approx(
+        ledger.get_client_balance("key-a")
+    )
+    assert reloaded.get_node_pending("wallet-1") == pytest.approx(0.02 * 0.90)
+    assert reloaded.snapshot()["protocol_cut"] == pytest.approx(0.02 * 0.10)
+
+
+def test_event_replication_converges_and_dedupes():
+    leader = BillingLedger(starting_credit=1.0, default_price_per_1k=0.02)
+    follower = BillingLedger(starting_credit=1.0, default_price_per_1k=0.02)
+
+    leader.credit_client("key-a", 10.0)
+    leader.charge_request("key-a", MODEL, 2000, [("wallet-1", 8), ("wallet-2", 4)])
+
+    events, cursor = leader.events_since(0)
+    assert follower.apply_events(events) == len(events)
+    # replaying the same batch must be a no-op
+    assert follower.apply_events(events) == 0
+
+    assert follower.get_client_balance("key-a") == pytest.approx(
+        leader.get_client_balance("key-a")
+    )
+    assert follower.get_node_pending("wallet-1") == pytest.approx(
+        leader.get_node_pending("wallet-1")
+    )
+    assert follower.snapshot()["protocol_cut"] == pytest.approx(
+        leader.snapshot()["protocol_cut"]
+    )
+    # incremental cursor: nothing new after full sync
+    more, _ = leader.events_since(cursor)
+    assert more == []
+
+
+# ---------------------------------------------------------- HTTP integration
+
+
+class _UsageStubNode:
+    """Minimal head node returning a chat completion with real usage numbers."""
+
+    def __init__(self, total_tokens: int = 1000):
+        self.total_tokens = total_tokens
+        outer = self
+
+        class Handler(http.server.BaseHTTPRequestHandler):
+            def log_message(self, fmt, *args):
+                pass
+
+            def do_POST(self):
+                length = int(self.headers.get("Content-Length", 0))
+                self.rfile.read(length)
+                body = json.dumps({
+                    "id": "chatcmpl-stub",
+                    "object": "chat.completion",
+                    "model": MODEL,
+                    "choices": [{
+                        "index": 0,
+                        "message": {"role": "assistant", "content": "ok"},
+                        "finish_reason": "stop",
+                    }],
+                    "usage": {
+                        "prompt_tokens": 0,
+                        "completion_tokens": outer.total_tokens,
+                        "total_tokens": outer.total_tokens,
+                    },
+                }).encode()
+                self.send_response(200)
+                self.send_header("Content-Type", "application/json")
+                self.send_header("Content-Length", str(len(body)))
+                self.end_headers()
+                self.wfile.write(body)
+
+        self._server = socketserver.ThreadingTCPServer(("127.0.0.1", 0), Handler)
+        self._server.daemon_threads = True
+        self._thread: threading.Thread | None = None
+
+    def start(self) -> int:
+        self._thread = threading.Thread(target=self._server.serve_forever, daemon=True)
+        self._thread.start()
+        return self._server.server_address[1]
+
+    def stop(self):
+        self._server.shutdown()
+        self._server.server_close()
+
+
+@pytest.fixture
+def billed_tracker():
+    ledger = BillingLedger(starting_credit=0.03, default_price_per_1k=0.02)
+    tracker = TrackerServer(
+        model_presets={
+            MODEL: {
+                "layers_start": 0,
+                "layers_end": 11,
+                "bytes_per_layer": {"bfloat16": 30 * 1024 * 1024},
+            }
+        },
+        billing=ledger,
+    )
+    port = tracker.start()
+    tracker_url = f"http://127.0.0.1:{port}"
+
+    stub = _UsageStubNode(total_tokens=1000)
+    stub_port = stub.start()
+    reg = json.dumps({
+        "endpoint": f"http://127.0.0.1:{stub_port}",
+        "shard_start": 0,
+        "shard_end": 11,
+        "model": MODEL,
+        "hardware_profile": {},
+        "score": 1.0,
+        "tracker_mode": True,
+        "wallet_address": "wallet-head",
+    }).encode()
+    req = urllib.request.Request(
+        f"{tracker_url}/v1/nodes/register",
+        data=reg,
+        headers={"Content-Type": "application/json"},
+        method="POST",
+    )
+    with urllib.request.urlopen(req) as r:
+        r.read()
+
+    yield tracker_url, ledger
+
+    stub.stop()
+    tracker.stop()
+
+
+def _chat(tracker_url: str, api_key: str | None):
+    data = json.dumps({
+        "model": MODEL,
+        "messages": [{"role": "user", "content": "hi"}],
+    }).encode()
+    headers = {"Content-Type": "application/json"}
+    if api_key:
+        headers["Authorization"] = f"Bearer {api_key}"
+    req = urllib.request.Request(
+        f"{tracker_url}/v1/chat/completions",
+        data=data,
+        headers=headers,
+        method="POST",
+    )
+    with urllib.request.urlopen(req) as r:
+        return json.loads(r.read())
+
+
+def test_proxy_chat_requires_api_key_when_billing_enabled(billed_tracker):
+    tracker_url, _ = billed_tracker
+    with pytest.raises(urllib.error.HTTPError) as exc_info:
+        _chat(tracker_url, api_key=None)
+    assert exc_info.value.code == 401
+
+
+def test_proxy_chat_bills_client_and_credits_node(billed_tracker):
+    tracker_url, ledger = billed_tracker
+    _chat(tracker_url, api_key="client-1")
+    # 1000 tokens at 0.02/1K = 0.02 USDT; starting credit 0.03
+    assert ledger.get_client_balance("client-1") == pytest.approx(0.03 - 0.02)
+    assert ledger.get_node_pending("wallet-head") == pytest.approx(0.02 * 0.90)
+
+    summary = json.loads(
+        urllib.request.urlopen(f"{tracker_url}/v1/billing/summary").read()
+    )
+    assert summary["node_pending"]["wallet-head"] == pytest.approx(0.02 * 0.90)
+    assert summary["protocol_cut"] == pytest.approx(0.02 * 0.10)
+
+
+def test_proxy_chat_402_when_balance_exhausted(billed_tracker):
+    tracker_url, ledger = billed_tracker
+    _chat(tracker_url, api_key="client-2")  # 0.03 -> 0.01
+    _chat(tracker_url, api_key="client-2")  # 0.01 -> -0.01 (post-pay drift)
+    assert ledger.get_client_balance("client-2") == pytest.approx(-0.01)
+    with pytest.raises(urllib.error.HTTPError) as exc_info:
+        _chat(tracker_url, api_key="client-2")
+    assert exc_info.value.code == 402
+    # rejected before routing: nothing further was billed
+    assert ledger.get_client_balance("client-2") == pytest.approx(-0.01)
+
+
+def test_billing_gossip_endpoint_applies_events(billed_tracker):
+    tracker_url, ledger = billed_tracker
+    peer = BillingLedger(starting_credit=0.03, default_price_per_1k=0.02)
+    peer.credit_client("remote-client", 7.0)
+    events, _ = peer.events_since(0)
+    body = json.dumps({"events": events}).encode()
+    req = urllib.request.Request(
+        f"{tracker_url}/v1/billing/gossip",
+        data=body,
+        headers={"Content-Type": "application/json"},
+        method="POST",
+    )
+    with urllib.request.urlopen(req) as r:
+        assert json.loads(r.read())["applied"] == len(events)
+    # only the replicated credit event lands here — Caller Credit was granted
+    # on the peer that first saw the key, and its event replicates separately
+    assert ledger.get_client_balance("remote-client") == pytest.approx(7.0)