popov/neuron-tai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: consolidate all docs under docs/ — single source of truth

Browse Source 
Move issues (01–29) and PRD from .scratch/distributed-inference-network/
into docs/issues/ and docs/. Update ralph_progress.py DEFAULT_PRD path
and rewrite docs/agents/issue-tracker.md to reflect the new layout.

The distributed_inference_network.egg-info/docs/ mirror is a build
artifact already covered by *.egg-info/ in .gitignore — not committed.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Dobromir Popov
2026-07-01 14:18:26 +03:00
parent 78834e5045
commit d1e75ddded
34 changed files with 6 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files

138
.scratch/distributed-inference-network/PRD.md
View File

@@ -1,138 +0,0 @@
Status: ready-for-agent
# Distributed Inference Network — PRD
## Problem Statement
Running large language models requires expensive dedicated hardware that most people can't afford. At the same time, millions of GPUs sit idle in homes and offices. There is no dead-simple, incentivised way for GPU owners to share their hardware and for developers to access distributed inference — without depending on a single company, without paying cloud GPU prices, and without setting up complex infrastructure.
## Solution
A volunteer GPU network where anyone can share their GPU by running a single command and immediately start earning tokens. Nodes each load a shard of a large model; a tracker routes inference requests through the optimal chain of nodes whose shards collectively cover all layers. Developers access the network through an OpenAI-compatible API — a one-line change from any existing LLM integration. Clients pay in SOL or USDC; node operators earn our native token. Everything is auto-configured: GPU detection, shard download, wallet creation, and network registration happen automatically on first start.
## User Stories
### Node Operator
1. As a node operator, I want to install the node client with a single command (`pip install meshnet-node`), so that I can start contributing without reading documentation.
2. As a node operator, I want the node client to auto-detect my GPU model and VRAM on first start, so that I don't have to specify my hardware manually.
3. As a node operator, I want the node client to create a Solana wallet for me automatically on first start, so that I can start earning without prior crypto knowledge.
4. As a node operator, I want the tracker to assign me the most-needed shard for my hardware automatically, so that my contribution has the highest possible impact on the network.
5. As a node operator, I want my assigned shard to download automatically from HuggingFace on first start, so that I don't have to manually find or download model weights.
6. As a node operator, I want to seed my shard to other nodes via P2P once I have it, so that new nodes with the same shard assignment don't need to download from HuggingFace.
7. As a node operator, I want the node client to register with the tracker automatically and begin serving inference requests, so that I start earning as soon as setup is complete.
8. As a node operator, I want to see my current node score, shard assignment, and token earnings in the terminal, so that I can verify my node is contributing correctly.
9. As a node operator, I want to stake tokens before serving paid inference, so that I have skin in the game and the network can trust my outputs.
10. As a node operator, I want my first N jobs to run without earning (probationary period), so that the network can establish trust before paying me.
11. As a node operator, I want to be notified immediately if my stake is slashed due to a fraud detection event, so that I can investigate and fix the issue.
12. As a node operator, I want to receive a strike and a warning before being banned, so that accidental failures don't immediately end my participation.
13. As a node operator, I want to be automatically reassigned to a different shard when the tracker determines another shard is more in demand, so that my hardware is always optimally used.
14. As a node operator, I want the node client to reconnect automatically if the tracker is temporarily unavailable, so that transient network issues don't stop me from earning.
15. As a node operator, I want the node client to fall back to P2P gossip for route discovery if the centralized tracker is down, so that inference serving continues during outages.
16. As a node operator, I want to run the node client on a CPU-only machine with smaller shards, so that I can contribute even without a dedicated GPU.
### Client Developer
17. As a client developer, I want to send `POST /v1/chat/completions` requests to the gateway in the same format as the OpenAI API, so that I can switch to the network with a one-line code change.
18. As a client developer, I want to authenticate with an API key funded by SOL or USDC, so that I never need to acquire or hold our native token.
19. As a client developer, I want to top up my API key balance by sending SOL or USDC to a Solana address, so that payment is simple and familiar.
20. As a client developer, I want to see a per-request cost estimate before sending a request, so that I can budget inference costs accurately.
21. As a client developer, I want to receive streaming responses (`text/event-stream`) in OpenAI-compatible format, so that I can build low-latency user experiences.
22. As a client developer, I want `GET /v1/models` to return the list of available model presets on the network, so that I know what I can request.
23. As a client developer, I want to receive a clear error response when no inference route is available for a requested model preset, so that I can handle degraded availability gracefully.
24. As a client developer, I want to use the `meshnet` Python SDK to access network-specific features (redundancy level, wallet top-up, node selection hints), so that I can optimise for my use case beyond basic inference.
25. As a client developer, I want the gateway to be compatible with LangChain, LlamaIndex, and Open WebUI out of the box, so that I can integrate with existing tooling immediately.
26. As a client developer, I want to request redundant execution (same request routed to multiple independent node chains) for high-stakes queries, so that I can trade cost for reliability.
### End User (via a client app)
27. As an end user, I want to buy SOL on any exchange and use it to pay for inference, so that I don't need to understand blockchain technology to use the service.
28. As an end user, I want responses of equivalent quality to centralised providers, so that I don't have to trade quality for cost savings.
29. As an end user, I want low latency on first token, so that conversational applications feel responsive.
### Validator
30. As a validator, I want to automatically re-run a random sample (~5%) of completed inference requests on a reference node, so that I can detect nodes returning fraudulent outputs.
31. As a validator, I want to submit a fraud proof on-chain when a node's output diverges beyond tolerance, so that the slash event is recorded trustlessly.
32. As a validator, I want to earn a reward for each successful fraud detection, so that there is an economic incentive to run validation.
### Network (tracker / system)
33. As the tracker, I want to score nodes continuously by throughput and latency so that I can select the fastest inference route for each request.
34. As the tracker, I want to rebalance shard assignments across nodes when demand for a model preset changes, so that the network always covers the most-requested models.
35. As the tracker, I want to instruct a node to download a new shard when no other node covers it, so that model preset coverage is maintained automatically.
36. As the tracker, I want to exclude banned wallets from route selection, so that fraudulent nodes cannot serve paid inference.
37. As the tracker, I want to read stake, slash, strike, and ban state exclusively from Solana smart contracts, so that I cannot manipulate payouts even with full control of the routing layer.
38. As the network, I want new model presets to be addable by submitting a HuggingFace model ID and shard count, so that the set of available models can grow without code changes.
## Implementation Decisions
### Monorepo structure
The codebase is organized as a Python monorepo with the following top-level packages:
- `packages/node` — node client CLI (`meshnet-node`)
- `packages/gateway` — OpenAI-compatible HTTP gateway and route orchestration
- `packages/tracker` — centralized tracker service (node registry, scoring, route selection)
- `packages/sdk` — `meshnet` Python SDK wrapping gateway + wallet controls
- `packages/contracts` — Solana L2 smart contracts (stake, slash, strike, ban, settlement)
- `packages/p2p` — P2P gossip layer and shard swarm seeding
### Inference engine (ADR-0001)
PyTorch with a Petals-style shard pipeline. Each node independently loads its assigned shard from local disk. At inference time, only activation tensors (~8 KB per layer boundary per token) travel between nodes — no model weights cross the network during serving.
### Inference route execution
The gateway receives a client request, asks the tracker for an inference route (ordered list of node endpoints covering all layers), opens a persistent TCP session to the first node in the route, streams activation tensors through each node in sequence, and returns the final logits as a streaming chat completion response.
### Node client startup sequence
1. Detect GPU/VRAM
2. Load or create Solana wallet
3. Query tracker for shard assignment given hardware profile
4. Download shard from HuggingFace (falling back to shard swarm if available)
5. Join shard swarm as a seeder
6. Register with tracker (wallet, hardware profile, shard, endpoint)
7. Begin accepting inference connections
### Payment flow
Clients pre-fund an API key with SOL/USDC. The gateway records per-request compute attribution. A settlement transaction runs on Solana L2 at the end of each epoch: client balance is debited, node operators receive our native token proportional to layers served, validators receive a reward share. Solana contracts are the authoritative source for all stake, slash, strike, and ban state (ADR-0002).
### Fraud detection (ADR-0003)
Validators re-run ~5% of completed requests. If a node's output diverges beyond floating-point tolerance from the reference, the validator submits a slash transaction on-chain. Strike count increments. At the configured strike threshold, the wallet is banned on-chain. New wallets complete N unpaid jobs before earning begins.
### Tracker architecture (ADR-0004)
Centralized tracker service (HTTP + WebSocket) for fast routing. Nodes gossip state via a lightweight P2P layer so the node client can discover routes during tracker outages. Solana is the authoritative source of truth for all incentive-relevant state.
### Shard distribution (ADR-0005)
Shards are identified by `(model_preset, shard_index)`. On assignment, the node downloads the shard layers from HuggingFace using `huggingface_hub`. Once downloaded, the node joins the P2P shard swarm and seeds to other nodes requesting the same shard. Popular shards propagate entirely via P2P; cold shards fall back to HuggingFace.
### Client API (ADR-0006)
The gateway exposes OpenAI-compatible endpoints (`/v1/chat/completions`, `/v1/models`, `/v1/completions`). The `meshnet` SDK wraps these and adds: `client.wallet.top_up()`, `client.request(redundancy=2)`, `client.models.available()`, per-request cost estimation.
## Testing Decisions
**What makes a good test:** test observable behavior at the highest seam possible — the `POST /v1/chat/completions` → streamed response boundary. Tests should not assert on internal state (which node was chosen, how tensors were split) but on what the client observes: a valid, coherent response arrives within a latency bound, and a payment record is written on-chain.
**Primary test seam:** spin up N in-process mock nodes each serving a known shard range, register them with a local tracker instance, send a real chat completion request through the gateway, and assert the response is a valid OpenAI-format streamed response whose content matches the expected model output for the given prompt.
**Per-component seams:**
- **Tracker**: given a set of registered nodes with known shard coverage and node scores, assert `select_route(model_preset)` returns an optimal ordered list of node endpoints.
- **Node shard serving**: given an activation tensor for the node's layer range, assert the output tensor shape and dtype are correct.
- **Fraud detection**: given a validator that re-runs a known-bad node response, assert a slash transaction is submitted on-chain with correct attribution.
- **Shard swarm**: given a node that has a shard, assert a second node with the same assignment downloads it via P2P rather than HuggingFace.
- **Payment settlement**: given a completed inference session with known compute attribution, assert token balances change by the expected amounts after epoch settlement.
## Out of Scope
- Desktop GUI app (v2)
- DHT as primary tracker mechanism (v2 — P2P gossip is a resilience fallback only in v1)
- zkML / zero-knowledge fraud proofs (research stage, not production-ready for large models)
- TEE attestation (excludes consumer GPUs, defeats viral growth goal)
- Distributed training / fine-tuning (post-v1)
- Mobile node apps (post-v1)
- Cross-chain payments beyond Solana L2 (post-v1)
- The existing `scripts/run_distributed_llama.py` (superseded by ADR-0001)
## Further Notes
- The `meshnet-node` CLI is the primary viral growth vector. Every friction point in the install/start sequence costs node operators. The startup sequence must complete without any manual configuration on a machine with a CUDA-capable GPU.
- The name "meshnet" is a working name. The actual package and token names are TBD.
- The Solana L2 chain selection (vs Base/Arbitrum) is not yet finalised — both are cheap, EVM-compatible fallbacks. The contracts package should abstract chain-specific details.
- The probationary period length (N free jobs) and slash amounts are economic parameters that will need tuning once the network has real usage data. Hardcode sensible defaults; make them on-chain governable.

29
.scratch/distributed-inference-network/issues/01-monorepo-scaffold.md
View File

@@ -1,29 +0,0 @@
Status: ready-for-agent
# 01 — Monorepo scaffold + single-node smoke test
## What to build
Stand up the monorepo package layout and prove that a client HTTP request can travel end-to-end through the stack — gateway → one node serving all layers → valid response — before any real model or distributed logic is wired in.
The six top-level packages should exist with minimal stubs:
- `packages/node` — node client CLI (`meshnet-node`)
- `packages/gateway` — HTTP gateway + route orchestration
- `packages/tracker` — node registry and route selection
- `packages/sdk` — `meshnet` Python SDK
- `packages/contracts` — Solana smart contract wrappers
- `packages/p2p` — gossip and shard swarm
The smoke test uses a tiny in-process stub model (not a real LLM) so the test is fast and has no external dependencies. The node runs in the same process as the test. The gateway is a real HTTP server. The client sends a real `POST /v1/chat/completions` request and receives a valid OpenAI-format response.
## Acceptance criteria
- [ ] `pip install -e packages/node packages/gateway packages/tracker` works from repo root
- [ ] `meshnet-node`, `meshnet-gateway`, and `meshnet-tracker` CLI entry points exist
- [ ] A single integration test starts a gateway and one stub node in-process, sends a `POST /v1/chat/completions` request, and asserts a valid OpenAI-format response is returned
- [ ] The test passes with `pytest` from repo root with no external services running
- [ ] All six package directories exist with `pyproject.toml` and an importable top-level module
## Blocked by
None — can start immediately.

23
.scratch/distributed-inference-network/issues/02-two-node-shard-pipeline.md
View File

@@ -1,23 +0,0 @@
Status: ready-for-agent
# 02 — Two-node shard pipeline
## What to build
Extend the single-node smoke test so that inference is routed through exactly two nodes, each serving half of a stub model's layers. Activation tensors must travel from the gateway to node A, then from node A to node B, and the final output returns to the gateway. This proves the distributed shard pipeline works end-to-end before real models or a real tracker are introduced.
The two nodes run in the same process as the test (no real network required). The gateway hardcodes the two-node route for now — dynamic route selection comes in issue 03. The activation tensor protocol (shape, dtype, serialisation format) must be established here, as all later issues depend on it.
Use the domain vocabulary from `CONTEXT.md`: the ordered list of nodes is an **inference route**; each node's layer range is a **shard**; the tensors passed between nodes are activations (not "data" or "chunks").
## Acceptance criteria
- [ ] The integration test spins up two stub nodes, each configured with a non-overlapping shard range
- [ ] A `POST /v1/chat/completions` request results in activation tensors flowing node-A → node-B (verifiable via test assertions or logs)
- [ ] The gateway assembles the final response correctly from node-B's output
- [ ] The test passes with no external services running
- [ ] The activation tensor serialisation format is documented in a short inline comment (shape, dtype, wire format) — this becomes the contract all future nodes implement
## Blocked by
- `01-monorepo-scaffold.md`

24
.scratch/distributed-inference-network/issues/03-tracker-registration-and-routing.md
View File

@@ -1,24 +0,0 @@
Status: ready-for-agent
# 03 — Tracker: node registration + route selection
## What to build
Replace the hardcoded two-node route from issue 02 with a real tracker. Nodes register themselves with the tracker on startup (endpoint, shard range, hardware profile, node score). The gateway queries the tracker for an inference route for a given model preset instead of using a static list. The tracker returns an ordered list of node endpoints whose shards collectively cover all layers.
The tracker runs as a lightweight HTTP service. Node score is initially a simple placeholder (e.g. fixed value or random) — real throughput/latency scoring comes later. The tracker must correctly reject route requests when no registered nodes cover a required shard range and return an appropriate error.
This issue establishes the tracker's registration and routing API contract, which issues 04 (node startup) and 05 (OpenAI gateway) both depend on.
## Acceptance criteria
- [ ] A node can register with the tracker via HTTP, providing its endpoint, shard range, and a hardware profile
- [ ] The gateway queries the tracker with a model preset name and receives an ordered list of node endpoints forming a complete inference route
- [ ] The end-to-end integration test from issue 02 still passes with the tracker now in the loop (no hardcoded routes)
- [ ] The tracker returns a clear error response when no route is available for a requested model preset
- [ ] Nodes that fail to heartbeat within a configurable window are removed from the registry
- [ ] The tracker's registration and route-selection HTTP API is defined (paths, request/response shapes) in the tracker package
## Blocked by
- `02-two-node-shard-pipeline.md`

32
.scratch/distributed-inference-network/issues/04-node-client-startup.md
View File

@@ -1,32 +0,0 @@
Status: ready-for-agent
# 04 — Node client startup flow (`meshnet-node start`)
## What to build
Make `meshnet-node start` self-configuring from scratch on a machine with a CUDA-capable GPU (or CPU fallback). The full startup sequence must complete without any manual configuration:
1. Detect GPU model and available VRAM
2. Load an existing Solana wallet from disk, or generate and save a new one
3. Query the tracker for the optimal shard assignment given the hardware profile
4. Download the assigned shard from HuggingFace (`huggingface_hub`)
5. Register with the tracker (wallet address, endpoint, shard range, hardware profile)
6. Begin accepting inference connections
The node prints a short status summary on startup: wallet address, assigned shard, model preset, and download progress. No interactive prompts — the entire flow is non-interactive.
This is the primary viral growth vector. Every second of friction in this flow costs node operators. The startup sequence must work on Linux with CUDA, and degrade gracefully to CPU on machines without a GPU.
## Acceptance criteria
- [ ] `meshnet-node start --tracker http://localhost:8080` completes startup without prompts on a machine with a CUDA GPU
- [ ] A Solana wallet keypair is generated and saved to `~/.config/meshnet/wallet.json` if none exists
- [ ] The assigned shard is downloaded from HuggingFace and cached to `~/.cache/meshnet/shards/`
- [ ] The node registers with the tracker and appears in the tracker's node registry
- [ ] The node accepts a live inference connection and processes activation tensors correctly after startup
- [ ] On a CPU-only machine, the node starts with a warning and serves a CPU-appropriate shard
- [ ] An integration test covers the full startup sequence against a local tracker stub
## Blocked by
- `03-tracker-registration-and-routing.md`

29
.scratch/distributed-inference-network/issues/05-openai-compatible-gateway.md
View File

@@ -1,29 +0,0 @@
Status: ready-for-agent
# 05 — OpenAI-compatible gateway
## What to build
Expose a production-shape HTTP API from the gateway so that any client using the OpenAI Python SDK (or any OpenAI-compatible tool) works by changing only the `base_url`. The gateway translates OpenAI-format requests into inference route execution and streams responses back in OpenAI-format.
Endpoints to implement:
- `POST /v1/chat/completions` — streaming (`text/event-stream`) and non-streaming
- `GET /v1/models` — returns the list of model presets currently routable on the network
- `GET /v1/health` — liveness check
The gateway selects an inference route from the tracker, executes the shard pipeline, and assembles the streamed response. If the tracker returns no route for the requested model, the gateway responds with a standard OpenAI-format error (`model_not_available`).
Authentication (API key → SOL/USDC balance) is a stub in this issue — return 200 for any non-empty `Authorization` header. Real payment gating comes in issue 06.
## Acceptance criteria
- [ ] `openai.OpenAI(base_url="http://localhost:8080/v1", api_key="test").chat.completions.create(model="stub-model", messages=[...])` returns a valid response
- [ ] Streaming works: `stream=True` returns `text/event-stream` chunks in OpenAI SSE format
- [ ] `GET /v1/models` returns a JSON array of available model preset names
- [ ] A request for an unavailable model returns an OpenAI-format error response with HTTP 503
- [ ] LangChain `ChatOpenAI(base_url=..., api_key=...)` works against the gateway
- [ ] An integration test covers streaming and non-streaming paths end-to-end through a real tracker and two stub nodes
## Blocked by
- `03-tracker-registration-and-routing.md`

32
.scratch/distributed-inference-network/issues/06-solana-stake-and-settlement.md
View File

@@ -1,32 +0,0 @@
Status: ready-for-agent
# 06 — Solana stake + settlement contracts
## What to build
Deploy and integrate the Solana smart contracts that make node staking, client payment, and token reward settlement trustless. All development and testing targets **Solana testnet** — never devnet or mainnet during development, to avoid real costs.
Three contracts are needed:
**Registry contract**: records stake balances, strike counts, and ban status per wallet. The gateway and tracker read from this contract to exclude banned wallets from route selection.
**Payment contract**: clients pre-fund an API key account with SOL or USDC. The gateway records per-request compute attribution (which node served which layer range, for how many tokens).
**Settlement contract**: called once per epoch. Debits client accounts proportional to compute consumed. Credits node operator wallets with our native token proportional to layers served. Distributes a validator reward share.
The `packages/contracts` package provides Python wrappers for reading and writing to all three contracts. The gateway uses these wrappers to check stake before routing to a node and to record attribution after each request.
## Acceptance criteria
- [ ] All contracts deploy successfully to Solana testnet
- [ ] A node can submit a stake transaction and have its balance reflected in the registry contract
- [ ] A client can fund an API key account with testnet SOL
- [ ] After a completed inference session, compute attribution is recorded on-chain with correct node/layer attribution
- [ ] The epoch settlement transaction correctly distributes token rewards to node operators and deducts client balances
- [ ] The gateway refuses to route to a node whose stake balance is below the minimum threshold
- [ ] All contract interactions in tests run against a local Solana test validator (via `solana-test-validator`) — no live testnet required for CI
- [ ] A `.env.testnet` config points to Solana testnet RPC for manual end-to-end testing
## Blocked by
- `03-tracker-registration-and-routing.md`

28
.scratch/distributed-inference-network/issues/07-fraud-detection-slash.md
View File

@@ -1,28 +0,0 @@
Status: ready-for-agent
# 07 — Fraud detection: validator + on-chain slash
## What to build
Implement the optimistic fraud detection loop. After each inference request completes, the validator process independently decides whether to re-run the request on a reference node (~5% sample rate). If the reference output diverges from the node's output beyond a configurable floating-point tolerance, the validator submits a slash transaction on-chain.
The validator is a separate process (not the gateway or tracker). It subscribes to completed inference events, selects a random sample, re-runs those requests against a trusted reference node, compares outputs, and submits slash proofs when fraud is detected.
On slash: the offending node's stake balance decreases by the configured slash amount; its strike count increments. The gateway reads strike state from the registry contract on each route selection and excludes nodes that exceed the strike threshold.
Node operators receive a push notification (logged to stdout and, if configured, a webhook) when their node is slashed.
## Acceptance criteria
- [ ] The validator process samples ~5% of completed inference requests (configurable)
- [ ] A node returning a deliberately wrong output is detected and slashed within one validation cycle
- [ ] The on-chain stake balance of the slashed node decreases by the correct slash amount
- [ ] The strike count for the slashed node increments on-chain
- [ ] A node that reaches the strike threshold is excluded from route selection on the next gateway request
- [ ] A slashed node logs a clear warning to stdout
- [ ] An integration test: run a deliberately-bad node, send 20 requests, assert at least one slash transaction is submitted and the node is eventually excluded from routes
## Blocked by
- `05-openai-compatible-gateway.md`
- `06-solana-stake-and-settlement.md`

27
.scratch/distributed-inference-network/issues/08-probationary-period-and-bans.md
View File

@@ -1,27 +0,0 @@
Status: ready-for-agent
# 08 — Node probationary period + ban enforcement
## What to build
Implement the two anti-sybil mechanisms that make re-entering the network after a ban economically costly.
**Probationary period**: a new wallet's first N completed inference jobs earn no token rewards. N is a configurable on-chain parameter (default: 50 jobs). The settlement contract reads the wallet's job count and skips reward distribution until the threshold is met. The node client displays the remaining probationary job count on startup.
**Ban enforcement**: when a wallet's strike count reaches the configured threshold, the registry contract marks it as banned. The tracker and gateway must check ban status on every route selection and exclude banned wallets. Ban status is read exclusively from the Solana registry contract — the tracker cannot set or unset bans.
Both mechanisms together mean a banned node operator must: create a new wallet, fund a new stake, and complete N free jobs before earning again — a meaningful economic barrier.
## Acceptance criteria
- [ ] A node with a new wallet receives no token rewards for its first N jobs (verified via settlement contract state)
- [ ] The node client prints remaining probationary jobs on startup (e.g. "Probationary period: 38 jobs remaining before earning")
- [ ] After N jobs, the next epoch settlement correctly credits the node with token rewards
- [ ] A wallet with strike count at the threshold is marked banned in the registry contract
- [ ] The tracker excludes banned wallets from route selection
- [ ] A banned wallet that attempts to register with the tracker is rejected
- [ ] An integration test covers: new wallet → N jobs → earning begins; and: strike threshold reached → banned → excluded from routes
## Blocked by
- `07-fraud-detection-slash.md`

24
.scratch/distributed-inference-network/issues/09-p2p-shard-swarm.md
View File

@@ -1,24 +0,0 @@
Status: ready-for-agent
# 09 — P2P shard swarm
## What to build
Once a node has downloaded a shard, it seeds that shard to other nodes that are assigned the same shard. New nodes with the same shard assignment should download from the swarm (peer nodes) rather than HuggingFace when peers are available, reducing HuggingFace load and speeding up propagation of popular shards.
The shard swarm is keyed by `(model_preset, shard_index)`. Nodes announce their shard availability to the tracker on registration. When a new node requests a shard download, the tracker returns a list of peer endpoints that hold the shard alongside the HuggingFace fallback URL. The node tries peers first; falls back to HuggingFace if no peers respond within a timeout.
Peers serve shard data over a simple chunked HTTP endpoint — no custom P2P protocol needed in v1. The node client displays download source (peer or HuggingFace) during shard download.
## Acceptance criteria
- [ ] A node that has downloaded a shard is listed as a peer for that shard in the tracker
- [ ] A second node assigned the same shard downloads it from the first node (peer) rather than HuggingFace (verified by asserting no HuggingFace HTTP calls are made in the test)
- [ ] If no peers are available, the node falls back to HuggingFace without error
- [ ] The node client logs whether the shard was downloaded from a peer or HuggingFace
- [ ] Shard chunks are verified against a checksum before being marked complete
- [ ] An integration test: node A downloads shard from HuggingFace stub; node B with same assignment downloads from node A
## Blocked by
- `04-node-client-startup.md`

44
.scratch/distributed-inference-network/issues/10-meshnet-sdk.md
View File

@@ -1,44 +0,0 @@
Status: ready-for-agent
# 10 — `meshnet` Python SDK
## What to build
A Python SDK (`packages/sdk`) that wraps the OpenAI-compatible gateway and exposes network-specific controls that the raw OpenAI SDK cannot express. The SDK is the recommended interface for developers who want more than basic inference.
The SDK surface:
```python
from meshnet import Client
client = Client(api_key="...", base_url="https://api.meshnet.ai")
# Basic inference (identical to OpenAI SDK)
response = client.chat.completions.create(model="llama-3-70b", messages=[...])
# Network-specific features
client.wallet.balance() # SOL/USDC balance remaining
client.wallet.top_up(amount_sol=1.0) # returns a Solana payment address + QR
client.models.available() # list of routable model presets with shard coverage %
client.request(redundancy=2) # route through 2 independent chains, return majority
client.estimate_cost(model="llama-3-70b", tokens=1000) # cost in SOL before sending
```
The SDK must not require clients to hold or know about our native token. All wallet operations are in SOL or USDC.
## Acceptance criteria
- [ ] `pip install meshnet` installs the SDK
- [ ] `Client.chat.completions.create(...)` works identically to the OpenAI SDK (same parameters, same response shape)
- [ ] `client.wallet.balance()` returns the current SOL/USDC balance for the API key
- [ ] `client.wallet.top_up()` returns a valid Solana payment address
- [ ] `client.models.available()` returns model presets with shard coverage percentage
- [ ] `client.estimate_cost(model, tokens)` returns a cost estimate in SOL before the request is sent
- [ ] `client.request(redundancy=2)` sends the same prompt to two independent inference routes and returns the majority response
- [ ] The SDK is typed (py.typed, full type stubs)
- [ ] An integration test covers each SDK method against a local gateway + tracker + stub nodes
## Blocked by
- `05-openai-compatible-gateway.md`
- `06-solana-stake-and-settlement.md`

50
.scratch/distributed-inference-network/issues/11-binary-wire-format.md
View File

@@ -1,50 +0,0 @@
# US-011 — Binary activation wire format
Replace the base64-encoded JSON activation payload from US-002 with a binary HTTP body, zstd compression, and chunked prefill. This is a protocol migration that must be applied to all nodes and the gateway before US-012 (real model backend) can be built on a sane foundation.
## Context
The current wire format encodes activation tensors as base64 strings inside a JSON dict:
```json
{"shape": [1, 1, 64], "dtype": "float32", "data": "<base64>", "context": {"prompt": "..."}}
```
This works for stub nodes (tiny tensors) but is unsuitable for real models:
- base64 adds 33% size overhead
- A 2048-token prefill at hidden_dim=16384 generates 64MB of activations per boundary — 85MB base64
- JSON parsing overhead grows with payload size
The new format per ADR-0008:
```
POST /forward HTTP/1.1
Content-Type: application/octet-stream
X-Meshnet-Shape: 1,128,16384
X-Meshnet-Dtype: bfloat16
X-Meshnet-Session: <uuid>
X-Meshnet-Chunk-Index: 0
X-Meshnet-Chunk-Total: 16
X-Meshnet-Encoding: zstd
Content-Length: <n>
<raw zstd-compressed little-endian bfloat16 bytes>
```
Stub nodes still emit zeroed tensors — just in binary now. No real model required for this story.
## Acceptance Criteria
- `packages/node` `/forward` endpoint reads `X-Meshnet-Shape`, `X-Meshnet-Dtype`, `X-Meshnet-Session`, `X-Meshnet-Chunk-Index`, `X-Meshnet-Chunk-Total`, and `X-Meshnet-Encoding` headers; reads raw body bytes; decompresses if `zstd`; reconstructs tensor as numpy array
- Node forward handler returns a binary response with the same header set reflecting output shape
- The gateway sends binary chunked activations to the first node; reassembles binary responses from the last node
- Chunked prefill: the gateway splits input sequences longer than `MESHNET_CHUNK_TOKENS` (default 128) into N chunks and sends them sequentially through the pipeline; stub nodes pass through each chunk unchanged
- An integration test sends a 512-token stub activation (4 chunks of 128) through a two-node pipeline and asserts all 4 chunk responses are received with correct headers
- `zstd` Python package added as a dependency to `packages/node` and `packages/gateway`
- The old `_make_stub_activations` function in `server.py` is replaced with `_make_stub_binary_activation(shape, dtype) -> bytes`
- `python -m pytest` passes from repo root
- Commit only this story's changes
## Implementation Notes
- Use `numpy` to pack/unpack tensor bytes: `np.frombuffer(body, dtype=np.float16).reshape(shape)` (bfloat16 → numpy uses `np.dtype('bfloat16')` in numpy >= 1.20, or load as uint16 and reinterpret)
- zstd: `import zstandard as zstd; cctx = zstd.ZstdCompressor(level=1); compressed = cctx.compress(raw_bytes)`
- Chunk index + total allow the receiving node to know whether this is a mid-sequence chunk (no special handling needed for stub; real model in US-012 will need to manage KV cache across chunks)
- Wire version header `X-Meshnet-Wire: 2` can be added for debugging mismatches between old and new nodes

61
.scratch/distributed-inference-network/issues/12-real-pytorch-model-backend.md
View File

@@ -1,61 +0,0 @@
# US-012 — Real PyTorch model backend
Replace stub node inference with actual `transformers` layer execution. A node downloads a HuggingFace SafeTensors model, loads `model.layers[start:end]` into GPU/CPU memory at the declared quantization level, and runs real forward passes. The first-layer node additionally runs `embed_tokens`; the last-layer node additionally runs `model.norm` and `lm_head`.
## Context
Per ADR-0001 (PyTorch over llama.cpp) and the grilling session decisions:
- Model format: HuggingFace SafeTensors (GGUF rejected — no per-layer hidden state API)
- Layer extraction: `model.model.layers[start:end]` from a loaded `AutoModelForCausalLM`
- Quantization: bitsandbytes NF4 / INT8 / bfloat16, declared by node at registration
- Activation dtype at boundaries: always `bfloat16` regardless of weight quantization
- First-layer node owns: tokenizer, `model.embed_tokens`, `model.layers[0..k]`
- Last-layer node owns: `model.layers[N-k..N]`, `model.norm`, `model.lm_head`
Start with a small real model for testing: `openai-community/gpt2` (117M, no special license, SafeTensors available). The production target is 700B+ models — the layer extraction pattern is identical regardless of size.
## Acceptance Criteria
- `packages/node` can be started with `--model-id openai-community/gpt2 --shard-start 0 --shard-end 6` and loads the corresponding transformer blocks
- Node detects whether it is a head shard (`shard_start == 0`) and loads tokenizer + `model.embed_tokens`
- Node detects whether it is a tail shard (`shard_end == total_layers`) and loads `model.norm + model.lm_head`
- A `/forward` request with a text prompt (head node) or binary bfloat16 tensor (mid/tail node) runs a real forward pass and returns binary bfloat16 activations
- A two-node integration test with GPT-2 sharded across two nodes returns a coherent (non-random, deterministic) text completion
- Quantization level is read from `--quantization [bfloat16|int8|nf4]` flag; `bfloat16` is default; `int8` and `nf4` use `bitsandbytes.BitsAndBytesConfig`
- A node with insufficient VRAM for the assigned shard at the declared quantization prints a clear error and exits (no silent OOM)
- `transformers`, `bitsandbytes`, `safetensors`, and `accelerate` added as dependencies to `packages/node`
- `python -m pytest` passes (GPT-2 test downloads model once, cached; mark with `@pytest.mark.integration` and skip in CI if no GPU)
- Commit only this story's changes
## Implementation Notes
```python
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch
def load_shard(model_id, start, end, quantization):
quant_config = None
if quantization == "nf4":
quant_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16)
elif quantization == "int8":
quant_config = BitsAndBytesConfig(load_in_8bit=True)
# Load full model metadata without weights, then extract layers
model = AutoModelForCausalLM.from_pretrained(
model_id,
quantization_config=quant_config,
device_map="auto",
torch_dtype=torch.bfloat16,
)
layers = model.model.layers[start:end]
return model, layers
```
- `device_map="auto"` handles GPU/CPU offload automatically based on available VRAM
- For large models, use `low_cpu_mem_usage=True` to avoid doubling peak RAM during load
- The forward pass for a mid-shard node: `hidden_states = layer(hidden_states)[0]` (most HF models return a tuple, take index 0)
- Head node forward: `input_ids → embed_tokens → position_embed → layers[0..k] → bfloat16 binary out`
- Tail node forward: `bfloat16 binary in → layers[N-k..N] → norm → lm_head → argmax/sample → token_id → back to head`
- Attention mask and position IDs must be forwarded alongside hidden states for models that need them (pack them in additional headers or a small JSON sidecar in the same HTTP request)
- Keep attention/position passing simple for v1: assume left-padded inputs, pass `attention_mask` as a separate binary blob with header `X-Meshnet-Attn-Mask: <base64-uint8>` (small enough for base64 — it's [B, S] not [B, S, D])

53
.scratch/distributed-inference-network/issues/13-coverage-first-shard-assignment.md
View File

@@ -1,53 +0,0 @@
# US-013 — Coverage-first tracker shard assignment
Upgrade the tracker's route selection to a coverage-first, speed-weighted bin-packing algorithm. The tracker maintains a live coverage map per model, issues rebalance directives when gaps appear, and assigns shard ranges to nodes based on declared VRAM, quantization, and benchmark throughput.
## Context
Per ADR-0009:
- **Coverage gap** = any layer range in a model with zero serving nodes → model is unroutable
- **Coverage-first**: fill all gaps before adding redundancy to covered ranges
- **Speed-weighted bin-packing**: fill fastest nodes to their VRAM limit first; cascade to next-fastest
- **Continuous rebalancing**: triggered every 30s and on every node join/leave event
- **Rebalance directive**: `LOAD_SHARD` or `DROP_SHARD` instruction sent from tracker to node
Node registration now includes capability data:
```json
{
"wallet": "...",
"endpoint": "http://10.0.0.5:7000",
"vram_bytes": 25769803776,
"ram_bytes": 137438953472,
"quantizations": ["nf4", "int8", "bfloat16"],
"benchmark_tokens_per_sec": 12.4,
"benchmark_model": "openai-community/gpt2"
}
```
Model preset metadata (stored in tracker config) includes `bytes_per_layer` at each quantization level and `total_layers`. The tracker can compute max assignable layers for any node.
## Acceptance Criteria
- Tracker accepts `vram_bytes`, `ram_bytes`, `quantizations`, `benchmark_tokens_per_sec` in the node registration payload
- `GET /v1/coverage/<model_preset>` returns the coverage map: list of `{start_layer, end_layer, node_count}` for each assigned range
- A model is only routable when all layer ranges have `node_count >= 1`
- When a new node registers, the tracker assigns it to the highest-priority uncovered range (or expands the most-loaded range if fully covered)
- When a node disconnects (heartbeat timeout), the tracker marks affected ranges as reduced-coverage and, if any reach 0, issues `LOAD_SHARD` directives to idle nodes within 30 seconds
- The shard assignment respects VRAM: `assigned_layers <= floor((vram_bytes * 0.8) / bytes_per_layer_at_quant)`
- Speed weighting: given two idle nodes both capable of covering a gap, the tracker assigns the wider sub-range to the faster node
- An integration test: start tracker with a model preset, register three nodes with different VRAM, assert the coverage map shows 100% coverage and the largest VRAM node received the widest shard range
- A second integration test: kill the node covering the middle layer range; assert tracker issues a `LOAD_SHARD` directive to an idle node and coverage recovers
- `python -m pytest` passes
- Commit only this story's changes
## Implementation Notes
- Coverage map data structure: `dict[str, list[tuple[int, int, int]]]` keyed by model preset
- Bin-packing is run in-process on the tracker (not a separate service); it's triggered by registration/heartbeat events and a 30s periodic timer
- Rebalance directives are delivered as responses to node heartbeat POSTs (node polls tracker every 10s anyway) — no new push channel needed
- `bytes_per_layer` is read from a `model_presets.json` config file; add GPT-2 (12 layers, ~30MB/layer bfloat16) as the test preset
- The current tracker `_select_route` function is extended, not replaced — backward compat with stub nodes that don't send VRAM data (default to 8GB / bfloat16 if omitted)
## Comments
- 2026-06-30: Follow-up capacity hardening lives in `20-memory-budget-shard-slots-and-dropout-relocation.md`. US-013 remains the base coverage-first assignment and dropout rebalance story; US-020 owns operator `--memory`, `--max-shards`, shard-slot enforcement, and relocation limit hardening so the two scopes do not conflict.

34
.scratch/distributed-inference-network/issues/14-tracker-as-node.md
View File

@@ -1,34 +0,0 @@
# US-014 — Tracker-as-first-layer-node (inference entry point)
Merge the inference orchestration role of the gateway into tracker nodes that serve the first-layer shard. A tracker node for a model receives client requests directly, runs the head shard, routes activations through the network, collects the tail output, and streams tokens back to the client. The standalone gateway becomes a thin load-balancer that routes to whichever tracker node is least loaded.
## Context
Per ADR-0009:
- Any node serving `layers[0..k]` for a model can act as its tracker node
- Tracker nodes own: tokenizer, `embed_tokens`, `layers[0..k]`
- Tracker nodes select the optimal onward route from the live coverage map (they already maintain it)
- Multiple tracker nodes for the same model = horizontal scale at both the routing and the first-layer compute level
- The standalone `meshnet-gateway` process from US-005 becomes a dumb round-robin load-balancer; it no longer orchestrates shard pipelines
This collapses a network hop and a process boundary: previously gateway → tracker → node; now client → tracker-node (which is both tracker and first-layer node).
## Acceptance Criteria
- A node started with `--tracker-mode` (or automatically when assigned `shard_start=0`) exposes both its existing `/forward` endpoint and a new `/v1/chat/completions` OpenAI-compatible endpoint
- The tracker-node's `/v1/chat/completions` handler: tokenizes input, embeds, runs its own layers, selects onward route from coverage map, forwards binary activations to next node, receives tail output, decodes tokens, streams SSE back to client
- Multiple tracker nodes for the same model can each independently handle requests (verified by sending requests to both and getting valid responses)
- The existing `meshnet-gateway` is updated to proxy requests to tracker nodes (round-robin or least-connections) rather than orchestrating the pipeline itself
- The gateway can discover tracker nodes from the tracker registry (`GET /v1/tracker-nodes/<model_preset>` returns the list of endpoints for tracker nodes for that model)
- An integration test: register two tracker nodes and two mid-shard nodes for GPT-2; send 10 requests to the gateway; assert both tracker nodes received roughly equal load and all responses are valid
- Existing US-005 OpenAI-compatible tests still pass (gateway still exposes `/v1/chat/completions`, just proxies to tracker nodes now)
- `python -m pytest` passes
- Commit only this story's changes
## Implementation Notes
- Tracker-mode detection: `shard_start == 0` OR `--tracker-mode` CLI flag
- The tracker-node process runs two HTTP servers on different ports: port 7000 for node-to-node (`/forward`), port 8080 for client-facing (`/v1/chat/completions`) — or a single server with both route prefixes
- Route selection inside the tracker node: same `GET /v1/route/<model_preset>` call to the central tracker registry, but the tracker node itself is already the first hop — the returned route starts from the *second* node
- Streaming: tail node returns token IDs one by one (or in batches) back to the tracker node via chunked HTTP; tracker node converts to SSE and streams to client
- The gateway's existing pipeline orchestration code (`_run_pipeline`) can be extracted into a shared `meshnet_common.pipeline` module reused by both tracker-node and the legacy gateway path

178
.scratch/distributed-inference-network/issues/15-ralph-agent-agnostic-status-aware.md
View File

@@ -1,178 +0,0 @@
# US-015 — Ralph: agent-agnostic runner + status-field-aware dashboard
Two tightly coupled improvements to the Ralph workflow tooling:
1. **Status-field awareness** — `ralph_progress.py` currently reads `passes: true/false` everywhere. Replace all `passes` references with the rich `status` field introduced in the prd.json schema migration. Surface `to-revise` and `needs-review` stories as an attention list rather than treating them like failed or blocked tasks.
2. **Agent-agnostic runner** — Ralph currently hardcodes `agentPlugin: "codex"` and always calls `ralph-tui run --agent codex`. Make the agent selectable per session with first-class support for Claude Code CLI, OpenRouter (unified API covering GPT-4, Mistral, DeepSeek, Llama, etc.), and the existing Codex plugin.
## Part 1: Status-field awareness in ralph_progress.py
### What to replace
Every occurrence of `story.get("passes")` in `scripts/ralph_progress.py` must be replaced with a helper that reads the `status` field with a `passes` fallback for backward compatibility:
```python
# Status → done mapping (backward-compat: passes=True counts as done if no status field)
_DONE_STATUSES = {"done"}
_ATTENTION_STATUSES = {"to-revise", "needs-review"}
_ACTIVE_STATUSES = {"in-progress"}
_DESIGN_STATUSES = {"in-design"}
def _is_done(story: dict) -> bool:
status = story.get("status")
if status:
return status in _DONE_STATUSES
return bool(story.get("passes")) # fallback
def _needs_attention(story: dict) -> bool:
return story.get("status") in _ATTENTION_STATUSES
def _is_active(story: dict) -> bool:
return story.get("status") in _ACTIVE_STATUSES
def _is_in_design(story: dict) -> bool:
return story.get("status") in _DESIGN_STATUSES
```
### `_story_sets` rewrite
Current behaviour lumps `to-revise` into the "done" bucket (because `passes=True`) or the "open" bucket. New behaviour:
```
done → status == "done" (or passes=True fallback)
attention → status in {to-revise, needs-review} ← new bucket
active → status == "in-progress"
in-design → status == "in-design" (treated as blocked — needs human before agent)
ready → status == "open" AND all deps done
blocked → status == "open" AND some dep not done
```
### `_deps_done` rewrite
Dependency is satisfied when the dep story `_is_done()` — not when `passes=True`.
### Dashboard additions
```
Ralph progress: Distributed Inference Network
PRD: .scratch/distributed-inference-network/prd.json
[############################] 13/15 complete (86%)
Done: 13 Ready: 1 Attention: 2 Blocked: 0
⚠ Attention required (review before running):
⚠ US-002 02 — Two-node shard pipeline
Wire format replaced by US-011 binary protocol; verify tests.
⚠ US-005 05 — OpenAI-compatible gateway
Gateway orchestration superseded by US-014; defer until US-014 lands.
→ US-014 14 — Tracker-as-first-layer-node (inference entry point)
```
### `auto` command behaviour
`auto` skips `to-revise` and `needs-review` stories — they need human review before an agent re-runs them. Add `--include-revise` flag to override this for unattended runs.
### `_review_report` additions
Include a dedicated **Attention Required** section in the generated review brief listing all `to-revise` and `needs-review` stories with their `status_reason`.
---
## Part 2: Agent-agnostic runner
### Agent selection
Add `--agent` option with these values:
| Value | CLI invoked | Config required |
|-------|-------------|-----------------|
| `codex` | `ralph-tui run --agent codex ...` | `OPENAI_API_KEY` (existing) |
| `claude` | `ralph-tui run --agent claude ...` | `ANTHROPIC_API_KEY` |
| `openrouter` | `ralph-tui run --agent openrouter ...` | `OPENROUTER_API_KEY` + `--model` |
| `custom` | `--agent-cmd <path>` | user-defined |
If `ralph-tui` does not natively support a requested agent plugin, `ralph_progress.py` falls back to invoking the agent CLI directly with the task prompt file, bypassing `ralph-tui run` entirely.
### Persistent agent config
Save the last-used agent choice to `.ralph-tui/agent-config.json` so it doesn't have to be passed on every command:
```json
{
"agent": "openrouter",
"model": "anthropic/claude-opus-4",
"updatedAt": "2026-06-29T..."
}
```
`--agent` on the CLI always overrides the saved config. `ralph_progress.py set-agent --agent openrouter --model openai/gpt-4o` writes the config file.
### OpenRouter adapter
When `agent == "openrouter"` and `ralph-tui` doesn't natively support it, `ralph_progress.py` implements a thin adapter:
1. Read the task prompt from the issue file + prd.json story
2. POST to `https://openrouter.ai/api/v1/chat/completions` with `OPENROUTER_API_KEY`
3. Stream response to stdout
4. Watch prd.json for `status` change to detect task completion
5. Timeout after configurable duration (default: 10 minutes per task)
The OpenRouter model is set via `--model` (e.g. `openai/gpt-4o`, `mistralai/mistral-large`, `meta-llama/llama-3-70b-instruct`). Default: `anthropic/claude-opus-4`.
### `custom` agent
`--agent custom --agent-cmd ./my-agent.sh` — Ralph passes the task prompt file path as `$1`. The script exits 0 on success. This makes Ralph compatible with any future agent tool without code changes.
---
## Acceptance Criteria
- All `passes` reads in `ralph_progress.py` replaced with `_is_done()` helper (with fallback)
- `_story_sets` returns five buckets: done, attention, active, in-design, ready, blocked
- Dashboard shows `⚠ Attention required` section with `status_reason` for each affected story
- `auto` skips `to-revise` / `needs-review` by default; `--include-revise` overrides
- `ralph_progress.py set-agent --agent <name>` writes `.ralph-tui/agent-config.json`
- `--agent codex|claude|openrouter|custom` accepted by all subcommands that invoke Ralph
- `ralph_progress.py run-next --agent openrouter --model openai/gpt-4o` runs a task via OpenRouter adapter
- `ralph_progress.py run-next --agent custom --agent-cmd ./my-agent.sh` runs a task via custom script
- `python -m pytest` passes from repo root
- Commit only this story's changes
---
## Part 4: Per-story metadata in dashboard
Every story in the dashboard gets an optional second line showing what is known about it:
```
✓ US-012 12 — Real PyTorch model backend
codex · done · "Added model_backend.py with bitsandbytes NF4 support"
⚡ US-014 14 — Tracker-as-first-layer-node
codex · in-progress · worktree: ../AI-worktree-us-014
→ US-015 15 — Ralph agent-agnostic runner
claude · in-progress · worktree: ../AI-worktree-us-015
```
### Sources
| Data | Source |
|------|--------|
| Active worktree path | `git worktree list --porcelain` — match branches `feat/<story-id>` |
| Agent (active story) | `.ralph-tui/agent-config.json` → `agent` field |
| Agent (completed story) | `.ralph-tui/session.json` → `agentPlugin`; or `completionNotes` text |
| Last output summary | `story.completionNotes` (done); latest `git log -1 --format=%s feat/<id>` (worktree); last lines of most recent `.ralph-tui/iterations/<hash>_<date>_<id>.log` (in-progress) |
### Behaviour
- Metadata line is omitted if there is nothing to show (no agent, no worktree, no notes)
- Default: shown (verbose). `--compact` flag suppresses it for a tighter one-line-per-story view
- `_active_worktrees() -> dict[story_id, rel_path]` helper, called once per dashboard render
- `_story_meta(story, worktrees, session) -> str | None` helper returns the joined metadata string
### Additional acceptance criteria
- `ralph_progress.py show` displays worktree path for any story whose branch exists as `feat/<story-id>`
- Agent name appears next to in-progress and recently-completed stories
- `completionNotes` from prd.json appears as the summary for done stories
- `--compact` suppresses metadata lines

206
.scratch/distributed-inference-network/issues/16-mining-cli-ux.md
View File

@@ -1,206 +0,0 @@
# US-016 — Mining-style node startup CLI + live dashboard
## Goal
Replace the bare flag-driven `meshnet-node start` with a wizard-guided first-run experience modelled on GPU mining clients (like PhoenixMiner, lolMiner, etc.). After the wizard, the terminal switches to a live status dashboard showing real-time node health and earnings.
## Wizard flow (first run only)
```
╔══════════════════════════════════════════════════════════╗
║ meshnet-node v0.1.0 ║
║ Distributed AI Inference — Node Setup ║
╚══════════════════════════════════════════════════════════╝
Detecting hardware...
GPU 0: NVIDIA RTX 4090 24 GB VRAM ✓
GPU 1: NVIDIA RTX 3090 24 GB VRAM ✓
Select a model to serve:
# Model Layers NF4 INT8 BF16
1 Llama-3-70B-Instruct 80 ✓18GB ✓40GB ✗80GB
2 Qwen-2.5-72B-Instruct 80 ✓19GB ✗41GB ✗81GB
3 Mixtral-8x7B-Instruct-v0.1 32 ✓ 7GB ✓14GB ✓27GB
4 Phi-3-medium-128k-instruct 40 ✓ 4GB ✓ 8GB ✓15GB
5 [Browse HuggingFace…]
Enter number [1]: _
Quantization [nf4/int8/bf16] (nf4 recommended for 24GB): _
Download directory [~/.meshnet/models]: _
Tracker URL [http://localhost:8080]: _
Wallet path [~/.config/meshnet/wallet.json] (new wallet will be created): _
Config saved to ~/.config/meshnet/config.json
Starting node…
```
Second run with existing config:
```
meshnet-node
Reading config from ~/.config/meshnet/config.json
Model: Llama-3-70B-Instruct Quant: nf4 Shard: layers 0–15
Tracker: http://192.168.1.10:8080
Starting…
```
## Live dashboard (once running)
Renders every 2 seconds using `rich.live`. Fallback: plain-text status line if `rich` is unavailable or terminal is not a TTY (important for WSL2 / SSH).
```
meshnet-node Llama-3-70B-Instruct [nf4] shard 0–15/80 up 00:03:22
GPU 0 RTX 4090 GPU ████████░░ 73% VRAM 18.2/24.0 GB 45°C
GPU 1 RTX 3090 GPU ███░░░░░░░ 28% VRAM 8.7/24.0 GB 38°C
Tokens/sec ▁▂▃▄▅▆▇█ 42.3 t/s (EMA 30s)
Requests 1,247 served 3 active
Peers 8 connected (tracker: ✓ relay: ✓)
TAI earned 0.00 TAI (payments active after US-006)
Uptime 00:03:22
[q] quit [r] reset stats [c] compact view
```
Compact mode (`--compact` or pressing `c`) shows a single status line:
```
[43t/s VRAM18.2GB req1247 peers8 up3m22s]
```
## Implementation notes
### Hardware detection
```python
import torch
def detect_gpus() -> list[dict]:
gpus = []
if torch.cuda.is_available():
for i in range(torch.cuda.device_count()):
props = torch.cuda.get_device_properties(i)
gpus.append({
"index": i,
"name": props.name,
"vram_gb": props.total_memory / 1e9,
"backend": "cuda"
})
# ROCm / Apple Silicon stubs for later
return gpus
```
### Curated model list
`packages/node/meshnet_node/model_catalog.py` — a hardcoded list of `ModelPreset` dataclasses:
```python
@dataclass
class ModelPreset:
name: str # display name
hf_repo: str # HuggingFace repo ID
num_layers: int
vram_gb: dict # {"nf4": 18, "int8": 40, "bf16": 80}
description: str # one-line description
```
Initial list (expand over time):
- `meta-llama/Meta-Llama-3-70B-Instruct` — 80L, NF4 18GB, INT8 40GB, BF16 80GB
- `Qwen/Qwen2.5-72B-Instruct` — 80L, NF4 19GB, INT8 41GB, BF16 81GB
- `mistralai/Mixtral-8x7B-Instruct-v0.1` — 32L, NF4 7GB, INT8 14GB, BF16 27GB
- `microsoft/Phi-3-medium-128k-instruct` — 40L, NF4 4GB, INT8 8GB, BF16 15GB
- `google/gemma-2-27b-it` — 46L, NF4 10GB, INT8 20GB, BF16 40GB
### HuggingFace Browse
```python
from huggingface_hub import list_models
def browse_hf(top_n=20) -> list[dict]:
models = list_models(
pipeline_tag="text-generation",
library="transformers",
sort="downloads",
direction=-1,
limit=top_n,
cardData=True,
)
return [{"repo": m.modelId, "downloads": m.downloads} for m in models]
```
### Persistent config
`~/.config/meshnet/config.json`:
```json
{
"model_hf_repo": "meta-llama/Meta-Llama-3-70B-Instruct",
"quantization": "nf4",
"download_dir": "~/.meshnet/models",
"tracker_url": "http://192.168.1.10:8080",
"wallet_path": "~/.config/meshnet/wallet.json",
"shard_start": null,
"shard_end": null,
"updatedAt": "2026-06-29T..."
}
```
`shard_start`/`shard_end`: null means tracker auto-assigns. User can pin a range for dedicated partial-model nodes.
### CLI flags
All wizard answers are overridable without re-running the wizard:
```
meshnet-node [start]
--model <hf-repo-id> # e.g. meta-llama/Meta-Llama-3-70B-Instruct
--quantization [bf16|int8|nf4]
--download-dir <path>
--tracker <url>
--wallet <path>
--shard-start <int> # pin shard range (optional)
--shard-end <int>
--reset-config # ignore saved config, re-run wizard
--no-tui # plain-text output (for CI / headless)
--compact # single-line status instead of full dashboard
meshnet-node models # list curated models and exit
meshnet-node models --browse # list HF Hub top-20 and exit
meshnet-node config # print current config and exit
```
### WSL2 / non-TTY fallback
```python
import sys, os
def is_interactive_tty() -> bool:
return sys.stdout.isatty() and os.environ.get("TERM") not in ("dumb", "")
if not is_interactive_tty():
# fall back to plain-text periodic status
run_plain_status_loop(node)
else:
run_rich_dashboard(node)
```
Do NOT use `termios`, `fcntl`, or `/dev/tty` — these break in Windows cmd.exe and some WSL2 terminal emulators.
## Acceptance criteria
- `meshnet-node` with no args and no config → wizard starts
- Wizard detects GPU and marks `[too large]` for models that exceed available VRAM
- `meshnet-node models` prints curated list and exits
- `meshnet-node models --browse` calls HF Hub API, prints top-20, exits
- Second run (config exists) → skips wizard, starts immediately
- `--reset-config` re-runs wizard even with config present
- All wizard inputs override-able via CLI flags
- Live rich dashboard renders and updates every 2s when running in a TTY
- Falls back to plain-text when not a TTY (CI / WSL2 without TERM set)
- Ctrl-C prints a clean summary line and exits 0
- `python -m pytest` passes from repo root
- Commit only this story's changes

205
.scratch/distributed-inference-network/issues/17-p2p-gossip-relay-ssl.md
View File

@@ -1,205 +0,0 @@
# US-017 — P2P gossip, NAT-traversal relay node, and SSL/TLS
## Goal
Nodes must work behind NAT (home routers, cloud VMs without public IPs) and must communicate securely. Implement:
1. **SSL/TLS everywhere** — all HTTP between nodes/tracker is HTTPS; all WebSocket gossip is WSS
2. **mDNS peer discovery** — nodes on the same LAN find each other automatically (no config)
3. **WebSocket gossip PubSub** — nodes propagate join/leave/coverage-update events in near-real-time
4. **Circuit relay node** — team-run public relay (`packages/relay`) that enables NAT traversal and bootstraps new nodes joining from the internet
Architecture is designed to migrate to libp2p GossipSub + Kademlia DHT without breaking the message schema (topic names and payload formats are stable contracts).
## Gossip protocol
### Transport
WebSocket (`wss://`) using the `websockets` Python library. Each node maintains persistent WSS connections to:
- The relay node (always, bootstraps peer list)
- Up to 8 direct peers (Kademlia-style target fanout; peers discovered via mDNS + relay peer list)
### Topics
All messages are JSON with an envelope:
```json
{
"topic": "node-join",
"version": 1,
"from_peer": "<peer_id>",
"timestamp": "<iso8601>",
"payload": { ... }
}
```
| Topic | Direction | Payload |
|-------|-----------|---------|
| `node-join` | broadcast | `{peer_id, addr, models: [{model_preset, shard_start, shard_end}], vram_gb, quant}` |
| `node-leave` | broadcast | `{peer_id, reason}` |
| `coverage-update` | broadcast | `{model_preset, coverage: [{start, end, count}]}` |
| `heartbeat` | peer→relay | `{peer_id, addr, uptime_s, tokens_per_sec}` |
| `peer-list` | relay→peer | `{peers: [{peer_id, addr}]}` |
| `relay-announce` | relay→all | `{relay_id, relay_url, capacity}` |
Gossip fanout: each node re-broadcasts received messages to all its peers (simple flooding with `seen_ids` dedup, TTL=3 hops). Migration to GossipSub mesh routing is a later ADR.
### Peer ID
`peer_id = sha256(public_key)[:16].hex()` — generated on first run, stored in `~/.config/meshnet/identity.json`. The same keypair is used for TLS client certificates (mTLS) in future work.
## mDNS LAN discovery
Use Python `zeroconf` library. Service type: `_meshnet._tcp.local.`
```python
from zeroconf import ServiceInfo, Zeroconf
info = ServiceInfo(
"_meshnet._tcp.local.",
f"{peer_id}._meshnet._tcp.local.",
addresses=[socket.inet_aton(local_ip)],
port=node_port,
properties={"peer_id": peer_id, "version": "1"},
)
zc = Zeroconf()
zc.register_service(info)
```
On startup, nodes also browse for `_meshnet._tcp.local.` to discover existing nodes. mDNS is LAN-only (does not traverse routers), which is correct for LAN discovery.
## NAT traversal: circuit relay
### How it works
1. Node A (behind NAT) cannot accept inbound TCP connections
2. Node A connects outbound to the public relay via WSS
3. Node A tells the tracker: `"effective_addr": "wss://relay.meshnet.ai/relay/{peer_id_A}"`
4. Node B (wants to call A) connects to the relay at the above URL
5. Relay proxies the TCP stream between A and B
Hole-punching (direct connection via STUN) is attempted first (future work). Relay is the fallback.
### meshnet-relay
`packages/relay/meshnet_relay/server.py` — a standalone aiohttp server:
```
GET /health → {status: ok}
GET /v1/peers → [{peer_id, addr, last_seen}]
POST /v1/gossip → receive a gossip message, fan out to connected peers
WSS /ws → persistent gossip connection (subscribe to all topics)
WSS /relay/{peer_id} → circuit relay proxy to that peer_id
GET /v1/relay/capacity → {connected_peers: N, max_peers: 500}
```
CLI:
```
meshnet-relay [--port 8443] [--cert path/to/cert.pem] [--key path/to/key.pem]
[--tracker-url http://...] [--max-peers 500]
```
The relay can optionally proxy to the tracker (so `relay.meshnet.ai` is the single internet-visible endpoint).
## SSL/TLS setup
### Node certificate (self-signed, auto-generated)
On first run, `meshnet-node` generates a self-signed RSA-2048 cert valid for 10 years:
```python
from cryptography import x509
from cryptography.hazmat.primitives import hashes, serialization
from cryptography.hazmat.primitives.asymmetric import rsa
```
Cert saved to `~/.config/meshnet/node_cert.pem` + `node_key.pem`. Fingerprint stored in config and shared with tracker via heartbeat. Nodes connecting to each other validate the fingerprint (TOFU — trust on first use), not the CA chain.
### Relay certificate
The relay uses a real Let's Encrypt cert (cert-bot or acme.sh). The relay cert is pinned in `packages/p2p/relay_bootstrap.json`:
```json
{
"relays": [
{
"url": "wss://relay.meshnet.ai:8443",
"cert_fingerprint": "sha256:<hex>",
"operator": "meshnet-team"
}
]
}
```
### All HTTP switched to HTTPS
`meshnet-node` starts an HTTPS server using `ssl.SSLContext`. `meshnet-tracker` similarly. All outbound `httpx` / `aiohttp` calls use TLS verification against pinned fingerprints (not the system CA store — too many corporate proxies break this).
## Tracker changes
Heartbeat payload gains new fields:
```json
{
"peer_id": "a1b2c3d4e5f6a1b2",
"effective_addr": "https://192.168.1.42:8001",
"relay_addr": "wss://relay.meshnet.ai:8443/relay/a1b2c3d4e5f6a1b2",
"cert_fingerprint": "sha256:...",
"gossip_peers": ["peer_id_1", "peer_id_2"]
}
```
Tracker uses `effective_addr` (direct) or `relay_addr` (fallback) when building inference routes.
## Integration test
```
tests/test_gossip_and_relay.py
scenario:
1. Start a local relay (localhost:18443)
2. Start node A (no inbound port — simulate NAT by binding to 127.0.0.1 only)
3. Start node B (public-reachable on localhost)
4. Both register with relay; relay peer-list includes both
5. Node B sends a gossip node-join message
6. Assert node A receives it within 500ms
7. Start tracker; confirm tracker's node registry includes node A via relay_addr
8. Send inference request; assert it routes through relay to node A
```
## Package layout
```
packages/relay/
pyproject.toml
meshnet_relay/
__init__.py
server.py # aiohttp relay + gossip hub + circuit relay proxy
cli.py # meshnet-relay entrypoint
peer_registry.py # in-memory {peer_id: {addr, last_seen, ...}}
circuit_relay.py # WSS proxy between two peers
packages/p2p/
meshnet_p2p/
gossip.py # GossipClient — connect to relay + peers, pub/sub
mdns.py # ZeroconfDiscovery — mDNS announce + browse
identity.py # PeerIdentity — generate/load peer_id + keypair
tls.py # cert generation, fingerprint, SSLContext helpers
packages/node/meshnet_node/
gossip_integration.py # wires GossipClient into node lifecycle
```
## Acceptance criteria
- All node↔node and node↔tracker HTTP uses HTTPS; self-signed cert auto-generated on first run
- `cert_fingerprint` included in heartbeat; tracker stores and logs it
- mDNS: two nodes on the same LAN discover each other without manual tracker URL (test with two localhost processes using different mDNS names)
- Relay: `meshnet-relay` starts, accepts WSS connections, fans out gossip messages to all connected peers
- Circuit relay: node A (127.0.0.1-only) can receive a gossip message via the relay from node B
- Tracker routes inference to node A using `relay_addr` when direct addr not reachable
- `relay_bootstrap.json` exists in `packages/p2p/` with at least one entry (localhost for tests)
- ADR-0010 documents the gossip architecture and libp2p migration path
- `python -m pytest` passes from repo root
- Commit only this story's changes

157
.scratch/distributed-inference-network/issues/18-two-machine-lan-test.md
View File

@@ -1,157 +0,0 @@
# US-018 — End-to-end two-machine LAN inference test
## Goal
Run real distributed inference across two physical machines: the Linux rig and a Windows 11 rig running WSL2. Document every setup step, firewall rule, and gotcha so this is repeatable. The test script exits 0 with token output and timing, proving the network works.
## Network topology for LAN test
```
[Linux machine] [Windows 11 / WSL2]
meshnet-tracker :8080 meshnet-node (shard B)
meshnet-node :8001 (shard A, tracker-mode)
meshnet-gateway :8000 (optional, for OpenAI-compat)
Client (either machine):
scripts/test_lan_inference.py --tracker http://192.168.1.10:8080
```
The Linux machine runs the tracker + the first-shard node (tracker-mode). The Windows/WSL2 machine runs the second-shard node. A small model (e.g. Phi-3-medium at BF16, fits on one GPU each) is split across both.
## WSL2 setup (Windows side)
`docs/INSTALL_WINDOWS.md` covers:
1. Enable WSL2: `wsl --install -d Ubuntu-24.04`
2. CUDA in WSL2: install NVIDIA driver on Windows (NOT inside WSL); WSL2 gets CUDA automatically
- Verify: `nvidia-smi` inside WSL2 should show GPU
3. Install Python 3.11+ and pip inside WSL2
4. `pip install -e packages/node packages/p2p` (clone repo first)
5. Firewall: Windows Defender must allow inbound WSL2 → LAN on node port
- PowerShell: `New-NetFirewallRule -DisplayName "meshnet-node" -Direction Inbound -Protocol TCP -LocalPort 8001 -Action Allow`
6. WSL2 IP: WSL2 has its own NAT'd IP (172.x.x.x); to expose to LAN, either:
- Option A: `netsh interface portproxy add v4tov4 listenport=8001 listenaddress=0.0.0.0 connectport=8001 connectaddress=$(wsl hostname -I)`
- Option B: use the relay node (US-017) — no port forwarding needed
## Linux setup
Standard install (already done after US-016). Firewall:
```bash
# If using ufw
sudo ufw allow 8080/tcp # tracker
sudo ufw allow 8001/tcp # node
sudo ufw allow 8000/tcp # gateway (optional)
```
## Model split
For the test, use a model that has enough layers to split meaningfully but fits comfortably in memory. Phi-3-medium-128k-instruct (40 layers, BF16 15GB) works on a single 24GB GPU on each machine:
- Linux node: layers 0–19 (tracker-mode, owns tokenizer + embed_tokens)
- Windows/WSL2 node: layers 20–39
Start sequence:
```bash
# Terminal 1 (Linux) — tracker
meshnet-tracker --port 8080
# Terminal 2 (Linux) — first-shard node (tracker-mode auto-detected because shard_start=0)
meshnet-node --model microsoft/Phi-3-medium-128k-instruct \
--quantization bf16 \
--shard-start 0 --shard-end 19 \
--tracker http://localhost:8080 \
--port 8001
# Terminal 3 (Windows WSL2) — second-shard node
meshnet-node --model microsoft/Phi-3-medium-128k-instruct \
--quantization bf16 \
--shard-start 20 --shard-end 39 \
--tracker http://192.168.1.10:8080 \
--port 8001
```
## Test script
`scripts/test_lan_inference.py`:
```python
#!/usr/bin/env python3
"""
End-to-end LAN inference test.
Usage: python scripts/test_lan_inference.py --tracker http://192.168.1.10:8080
"""
import argparse, time, httpx, json
MESSAGES = [
{"role": "user", "content": "What is 7 × 8? Answer in one word."},
{"role": "user", "content": "Name the capital of France in one word."},
{"role": "user", "content": "Complete the sequence: 1, 1, 2, 3, 5, ___"},
]
def run_test(tracker_url: str, gateway_url: str | None):
# Discover inference entry point via tracker if gateway not given
if not gateway_url:
r = httpx.get(f"{tracker_url}/v1/tracker-nodes/phi-3-medium", timeout=5)
r.raise_for_status()
nodes = r.json()
assert nodes, "No tracker-mode nodes registered — is the first-shard node running?"
gateway_url = nodes[0]["url"]
print(f"Inference endpoint: {gateway_url}")
print(f"Tracker: {tracker_url}")
print()
for i, msg in enumerate(MESSAGES):
t0 = time.monotonic()
r = httpx.post(
f"{gateway_url}/v1/chat/completions",
json={"model": "phi-3-medium", "messages": [msg], "stream": False},
timeout=60,
)
r.raise_for_status()
data = r.json()
elapsed = time.monotonic() - t0
content = data["choices"][0]["message"]["content"]
tokens = data["usage"]["completion_tokens"]
tps = tokens / elapsed if elapsed > 0 else 0
print(f"[{i+1}] Q: {msg['content']}")
print(f" A: {content}")
print(f" {tokens} tokens {elapsed:.2f}s {tps:.1f} t/s")
print()
print("✓ All 3 requests completed successfully")
if __name__ == "__main__":
p = argparse.ArgumentParser()
p.add_argument("--tracker", required=True)
p.add_argument("--gateway", default=None)
args = p.parse_args()
run_test(args.tracker, args.gateway)
```
## Docs: TWO_MACHINE_TEST.md
`docs/TWO_MACHINE_TEST.md` must cover:
1. Prerequisites (models downloaded on both machines, same model ID, complementary shard ranges)
2. Start order: tracker first, then nodes, then test script
3. How to verify nodes are registered: `GET /v1/nodes` on tracker
4. How to verify coverage: `GET /v1/coverage/phi-3-medium` — all 40 layers must show node_count ≥ 1
5. How to run the test script
6. Expected output
7. Latency breakdown: how to read per-hop latency from node logs
8. **Known Issues** section — updated during actual test run with real gotchas
## Acceptance criteria
- `docs/INSTALL_WINDOWS.md` covers WSL2 + CUDA + meshnet-node install end-to-end
- `docs/TWO_MACHINE_TEST.md` covers the full two-machine setup and test procedure
- `scripts/test_lan_inference.py` exists and is executable
- When run against a real two-machine LAN setup: script exits 0, prints 3 valid answers with timing
- Coverage map shows 100% coverage (no gap) after both nodes register
- Known Issues section in TWO_MACHINE_TEST.md contains at least the issues encountered during this test run
- No new pytest failures from repo root (this story adds docs + a script, not new Python packages)
- Commit only this story's changes

68
.scratch/distributed-inference-network/issues/19-binary-data-plane-and-peer-weight-transfer.md
View File

@@ -1,68 +0,0 @@
# US-019 — Binary data plane and optional peer weight transfer
Status: needs-triage
Priority: Low
Stage: Design parking lot
## Context
The current project focus is inference democratization: let small GPU owners contribute useful compute and let users run inference on models larger than one host can serve alone. Weight distribution is useful, but it is secondary to low-latency distributed inference.
Recent findings:
- HuggingFace already handles initial model origin distribution well enough for the first working version.
- The inference-critical path is activation transfer between shard nodes, not torrenting model files.
- Torrent/content-addressed transfer is a good future fit for model weights, shard cache replication, fine-tuned models, and offline/local swarm behavior.
- Torrenting is not a good fit for activation traffic because activations are latency-sensitive, ordered, session-specific binary streams.
## Design note
Keep the tracker as the control plane:
- node registration
- heartbeats
- route selection
- model/shard manifests
- peer/checksum metadata
Keep binary payloads on the data plane:
- direct node-to-node activation transfer where reachable
- relay/QUIC/WSS fallback where direct transport is unavailable
- future peer weight transfer as content-addressed blobs or pieces
## Potential future direction
For inference traffic:
- Prefer direct binary transport with backpressure.
- Use raw binary activation frames rather than JSON/base64.
- Preserve tensor metadata out-of-band: shape, dtype, session, chunk index, encoding.
- Consider QUIC or a mature NAT-friendly transport for direct-when-possible, relay-when-needed behavior.
For model weights:
- Keep HuggingFace as default origin and fallback.
- Add peer cache transfer only as an optimization.
- Consider a real content-addressed/torrent-like library or sidecar for weight blobs.
- Store manifests and checksums in tracker state so peers can verify exact shard contents.
- Prefer piece/chunk transfer with resume and hash verification over one giant tarball.
## Not in scope now
- Replacing HuggingFace as the primary model origin.
- Building a full BitTorrent/IPFS/libp2p subsystem.
- Routing activation traffic through a torrent protocol.
- Making peer weight transfer mandatory for node startup.
## Acceptance criteria for a future implementation issue
- [ ] Activation transfer remains binary end-to-end and avoids JSON/base64 payloads.
- [ ] Tracker does not proxy large binary payloads except as an explicit fallback path.
- [ ] Weight transfer, if added, is optional and falls back to HuggingFace.
- [ ] Weight pieces are content-addressed and checksum-verified.
- [ ] The design preserves low-latency inference as the primary objective.
## Comments
Created as a low-priority design parking-lot item after discussing inference democratization versus weight distribution. Do not pick up for implementation until the core public tracker, relay, and binary activation path are stable.

66
.scratch/distributed-inference-network/issues/20-memory-budget-shard-slots-and-dropout-relocation.md
View File

@@ -1,66 +0,0 @@
Status: ready-for-agent
# US-020 - Memory budget, shard slots, and dropout relocation hardening
## Goal
Make node capacity limits explicit and enforce them consistently when the tracker assigns, rebalances, and relocates shards after a node dropout.
This is a follow-up to US-013, not a replacement. US-013 owns the coverage-first assignment and rebalance algorithm. This issue hardens the capacity contract around that algorithm: operator memory budget, maximum loaded shard slots, and relocation behavior when one node must absorb or split ranges after another node disappears.
## Context
Recent work added the first part of the contract:
- `meshnet-node --memory MB` is registered with the tracker as `vram_bytes` when explicitly set.
- CPU nodes without `--memory` keep the tracker default capacity, preserving old behavior.
- `meshnet-node --max-shards N` is accepted and registered as `max_loaded_shards`.
- Tracker registration validates `max_loaded_shards >= 1`.
The current runtime still effectively has one active backend shard per node. A node may advertise `max_loaded_shards`, but the tracker does not yet use multiple shard slots in bin-packing, and the node does not yet host multiple concurrently loaded shard ranges.
## Scope
- Make tracker rebalance logic account for `max_loaded_shards` as a capacity multiplier or explicit shard-slot list.
- Ensure a node is never assigned more total layers than its memory budget can support across all loaded shard slots.
- Decide and implement the runtime behavior for multiple loaded shards:
- either support multiple concurrently loaded shard backends on one node, or
- keep one backend active and treat `max_loaded_shards` as future metadata, with tracker enforcement preventing multi-range assignment for now.
- On heartbeat timeout, relocate the dropped node's uncovered layer range to eligible managed nodes while respecting both memory and shard-slot limits.
- Surface the effective memory budget and shard slot count in tracker/network inspection output so operators can diagnose why a node did or did not receive a range.
## Non-Goals
- Do not redesign the US-013 coverage-first algorithm from scratch.
- Do not change relay, `/ws`, or `/rpc` behavior.
- Do not change the token/reward model.
- Do not require public internet verification; all behavior must be locally testable.
## Acceptance Criteria
- Tracker stores and exposes `max_loaded_shards` for registered nodes.
- Assignment/rebalance never exceeds:
- `assigned_layers_total <= floor((vram_bytes * 0.8) / bytes_per_layer_at_quant)`
- `assigned_range_count <= max_loaded_shards`
- A managed node with `max_loaded_shards=1` only receives one active shard range.
- A managed node with `max_loaded_shards=2` can absorb two non-contiguous uncovered ranges only if the node runtime supports serving both; otherwise tracker must keep assigning at most one range and document `max_loaded_shards` as reserved.
- Dropout test: register nodes covering a model, let a middle/tail node heartbeat-expire, and assert the tracker queues `LOAD_SHARD` directives that restore full coverage without violating memory or shard-slot limits.
- CLI test: `--memory` and `--max-shards` are reflected in the registration payload.
- `python -m pytest tests/test_tracker_routing.py tests/test_node_startup.py` passes in the project virtualenv, aside from any pre-existing platform-specific wallet permission assertion documented in the final notes.
## Implementation Notes
- Existing files likely involved:
- `packages/node/meshnet_node/cli.py`
- `packages/node/meshnet_node/startup.py`
- `packages/node/meshnet_node/torch_server.py`
- `packages/tracker/meshnet_tracker/server.py`
- `tests/test_tracker_routing.py`
- `tests/test_node_startup.py`
- Keep backward compatibility: nodes that omit `vram_bytes` default to tracker defaults; nodes that omit `max_loaded_shards` default to `1`.
- Prefer a small internal representation for assigned ranges if multiple ranges become real, for example `assigned_shards: list[tuple[int, int]]`, while preserving `shard_start`/`shard_end` in public responses for single-range nodes.
## Comments
- 2026-06-30: Created after implementing the initial registration plumbing in commit `f1e4ed6` (`--memory`, `--max-shards`, tracker validation). This issue captures the remaining end-to-end behavior so it does not conflict with US-013.
- 2026-06-30: Implementation decision: `max_loaded_shards` is currently a validated and exposed capacity field, but multi-range assignment remains reserved because `TorchNodeServer` serves one active backend shard. The tracker therefore emits at most one active range per node while exposing `vram_bytes`, `ram_bytes`, `max_loaded_shards`, quantization, throughput, and computed `max_assignable_layers` in inspection endpoints.

20
.scratch/distributed-inference-network/issues/20-tracker-node-hardening.md
View File

@@ -1,20 +0,0 @@
# US-020 — Tracker + node hardening: BrokenPipe fix, deterministic node IDs, HF coverage
Status: done
Priority: High
Stage: Maintenance
## Context
First two-machine LAN test (US-018) exposed three reliability issues:
1. `BrokenPipeError` crash in tracker `_send_json` when a slow-inference client disconnected mid-response
2. Random UUID node IDs meant every re-registration (after tracker restart) created a phantom entry
3. `GET /v1/coverage/<model>` returned no results when called with a short name (`Qwen2.5-0.5B`) instead of the full HF repo ID
## Acceptance criteria
- [ ] `BrokenPipeError` in tracker and node `_send_json` is silently swallowed
- [ ] Node IDs are deterministic: `sha256(wallet_address + str(port))[:16]`
- [ ] `GET /v1/coverage/<model>` accepts both short names and full `owner/repo` IDs
- [ ] `python -m pytest` passes from repo root

19
.scratch/distributed-inference-network/issues/21-route-timeout-flag.md
View File

@@ -1,19 +0,0 @@
# US-021 — `--route-timeout` CLI flag for node tracker route lookup
Status: done
Priority: Medium
Stage: Implemented
## Context
The node's slow-path tracker route lookup (`/v1/route`) used a hard-coded 30-second HTTP timeout.
On high-latency links (relay, satellite, 5G) or when the tracker is under load, legitimate route
lookups were failing prematurely. The timeout is deployment-specific and should be tunable.
## Acceptance criteria
- [ ] `meshnet-node start` accepts `--route-timeout <seconds>` (float, default 30.0)
- [ ] Value is passed through to `TorchNodeServer` and used in the `/v1/route` HTTP call
- [ ] `TorchNodeServer` exposes `route_timeout` as a readable property
- [ ] Test: setting `--route-timeout 45` is reflected as `45.0` on the running server object
- [ ] `python -m pytest` passes

29
.scratch/distributed-inference-network/issues/22-start-layer-protocol.md
View File

@@ -1,29 +0,0 @@
# US-022 — X-Meshnet-Start-Layer: overlapping shard execution protocol
Status: done
Priority: High
Stage: Implemented
## Context
Two nodes may register overlapping shard ranges (node A: 0–15, node B: 12–23) to increase
redundancy or to enable partial-model nodes to form a complete route. Without coordination,
node B would re-run layers 12–15 on top of activations that already include them, producing
wrong output.
The `X-Meshnet-Start-Layer` header tells each downstream node which model layer the incoming
activation tensor represents, so the node skips layers it has already been told to skip.
## Decision
Option A: tracker injects `start_layer` into `X-Meshnet-Route` hops at proxy time. The head
node passes it per-hop as `X-Meshnet-Start-Layer`. No peer-to-peer negotiation needed.
## Acceptance criteria
- [ ] Tracker `_handle_proxy_chat` builds route hops with `start_layer` = `covered_up_to + 1`
- [ ] `_handle_binary_forward` reads `X-Meshnet-Start-Layer` and passes it to `backend.forward_bytes`
- [ ] `_get_remaining_route` parses `start_layer` from injected header and from `/v1/route` slow-path
- [ ] `TorchModelShard.forward_bytes` accepts optional `start_layer` and skips layers below it
- [ ] Test: overlapping two-node route produces correct output without double-computing layers
- [ ] `python -m pytest` passes

26
.scratch/distributed-inference-network/issues/23-heartbeat-stats.md
View File

@@ -1,26 +0,0 @@
# US-023 — Heartbeat stats payload: request counters + dynamic reassignment response
Status: done
Priority: Medium
Stage: Implemented
## Context
Node heartbeats are currently empty POSTs. The tracker has no visibility into per-node load,
making load balancing and assignment decisions blind. Heartbeats should carry cumulative stats.
The heartbeat response channel is also the natural place for the tracker to deliver reassignment
instructions without requiring a node restart.
## Acceptance criteria
- [ ] Heartbeat POST body includes: `total_requests`, `failed_requests`, `queue_depth`, `uptime_seconds`, `status`
- [ ] `TorchNodeServer` tracks the three counters with a `threading.Lock`
- [ ] Tracker stores the last heartbeat payload per node
- [ ] Heartbeat response may include `new_assignment: {model, shard_start, shard_end}`; node logs it
- [ ] Stats survive tracker outage: buffered locally, flushed on next successful heartbeat
- [ ] `python -m pytest` passes
## Notes
Hot-reload (loading a new shard without restart) is deferred to a future story. The response
field is wired so trackers can send the signal; nodes log it but don't act yet.

24
.scratch/distributed-inference-network/issues/24-availability-map.md
View File

@@ -1,24 +0,0 @@
# US-024 — Enhanced availability map with per-node health details
Status: done
Priority: Medium
Stage: Implemented
## Context
`GET /v1/coverage/<model>` returns band-level coverage (start_layer, end_layer, node_count)
but gives no visibility into which specific nodes are in each band or whether they are alive.
Operators debugging a split-model deployment need to see node-level health at a glance.
## Format decision
Both: band metadata + node list grouped under each band. Dead nodes included with `healthy: false`
so operators can see them.
## Acceptance criteria
- [ ] Each band in the response includes `nodes: [{node_id, endpoint, healthy, queue_depth, last_seen_s}]`
- [ ] `healthy` is `true` iff last heartbeat < `heartbeat_timeout` seconds ago
- [ ] Existing band fields (`start_layer`, `end_layer`, `node_count`) preserved
- [ ] Tests updated to check band fields individually (not exact dict comparison)
- [ ] `python -m pytest` passes

30
.scratch/distributed-inference-network/issues/25-rolling-rpm-stats.md
View File

@@ -1,30 +0,0 @@
# US-025 — Model usage statistics: rolling RPM windows + SQLite persistence
Status: done
Priority: Medium
Stage: Implemented
## Context
Trackers need per-model request-rate data to drive smart shard assignment (US-026) and
to give operators visibility into what the network is actually serving. Stats must survive
tracker restarts and should be shareable across a tracker cluster.
## Design
- `_RollingCounter`: circular-bucket counter, epoch-indexed, stale buckets auto-reset
- Three windows: 60×1-min buckets (last hour), 24×1-hr (last day), 30×1-day (last month)
- `_StatsCollector`: `record_request()`, `get_local_rpms()`, `merge_peer_rpms()`, `get_combined_stats()`
- SQLite persistence via `--stats-db PATH`
- Gossip: each tracker keeps its own slice; merge is additive (not averaged)
## Acceptance criteria
- [ ] `_RollingCounter` passes unit tests (record, rpm, stale bucket reset)
- [ ] `_StatsCollector` accumulates and merges peer slices additively
- [ ] SQLite round-trip: buckets saved and restored across restart
- [ ] `GET /v1/stats` returns combined stats JSON
- [ ] `POST /v1/stats/gossip` accepts peer slice and merges
- [ ] `_handle_proxy_chat` calls `stats.record_request()` after model is resolved
- [ ] `--stats-db PATH` CLI flag added to tracker
- [ ] 6 unit tests pass; `python -m pytest` passes

32
.scratch/distributed-inference-network/issues/26-smart-model-assignment.md
View File

@@ -1,32 +0,0 @@
# US-026 — Smart model assignment via demand×coverage scoring
Status: done
Priority: Medium
Stage: Implemented
## Context
`/v1/network/assign` currently picks the model with the largest uncovered shard gap,
ignoring traffic. A model serving 1000 RPM at 60% coverage is far more valuable to fill
than a zero-traffic model at 50% coverage.
## Scoring formula
```
score = (demand_rpm + 1.0) × (coverage_deficit + 0.01)
```
- `demand_rpm`: combined RPM from `_StatsCollector.get_combined_stats()`
- `coverage_deficit`: fraction of model layers with zero node coverage, in [0.0, 1.0]
- `+1.0` floor: models with no traffic still compete by coverage
- `+0.01` floor: fully-covered models still have a non-zero score if they have traffic
`price_per_token: 0.0` reserved in the response for future billing integration.
## Acceptance criteria
- [ ] `_handle_network_assign` computes score per model and returns the highest
- [ ] Demand uses combined stats (local + peer slices)
- [ ] `price_per_token: 0.0` present in response
- [ ] Test: high-demand low-coverage model beats low-demand high-coverage model
- [ ] `python -m pytest` passes

28
.scratch/distributed-inference-network/issues/27-throughput-routing.md
View File

@@ -1,28 +0,0 @@
# US-027 — Throughput-optimized routing: effective throughput as tiebreak
Status: done
Priority: Medium
Stage: Implemented
## Context
The greedy max-reach route selection picks nodes by shard coverage but ignores node speed.
When two nodes cover the same remaining layer range, we should prefer the faster one.
This is a tiebreak only — coverage maximization remains the primary objective.
## Effective throughput formula
```
effective_throughput = benchmark_tokens_per_sec / (queue_depth + 1)
```
`benchmark_tokens_per_sec` comes from the hardware profile at registration time.
`queue_depth` comes from the last heartbeat.
## Acceptance criteria
- [ ] `_effective_throughput(node)` helper in `server.py`
- [ ] `_select_route` uses throughput as tiebreak when `shard_end` is equal
- [ ] Test: two nodes, same shard range, different throughput → faster node selected
- [ ] Existing coverage tests still pass unchanged
- [ ] `python -m pytest` passes

26
.scratch/distributed-inference-network/issues/28-routing-correctness-tests.md
View File

@@ -1,26 +0,0 @@
# US-028 — Routing correctness tests: three-node, overlap, and throughput scenarios
Status: done
Priority: Medium
Stage: Implemented
## Context
Route selection logic (`_select_route`) is the core of the inference network. Without
locked-down tests, routing regressions are silent. This story adds a comprehensive
scenario suite that locks in the routing contract.
## Test scenarios
1. **No-overlap three nodes**: greedy picks in layer-start order (A→C→B for ranges 0–7, 8–15, 16–23)
Note: the algorithm picks by earliest uncovered layer, not by node label — so if C.shard_start < B.shard_start, C comes first.
2. **Overlapping shards**: correct resolution without double-computing layers
3. **Throughput tiebreak**: faster node wins when shard_end is equal
4. **Gap detection**: partial coverage returns a 503 error with description
## Acceptance criteria
- [ ] 7 routing tests covering the above scenarios
- [ ] Tests use in-process tracker (no mocking)
- [ ] `_make_node` helper for concise test setup
- [ ] All tests pass; `python -m pytest` passes

44
.scratch/distributed-inference-network/issues/29-relay-outbound-client.md
View File

@@ -1,44 +0,0 @@
# US-029 — Outbound relay client: NAT/internet pipeline hops
Status: done
Priority: Critical
Stage: Implemented
## Context
Nodes behind NAT (WSL2 with 172.x.x.x addresses, 5G mobile, home routers) register with the
tracker and include a `relay_addr` (`wss://relay/rpc/{peer_id}`). When the head node needs to
forward activations to such a peer, it currently fails because the direct HTTP endpoint is
unreachable.
The relay server (US-017) is already running and the node already opens a persistent outbound
WebSocket (`RelayHttpBridge`). What is missing is the *outbound caller side*: given a `relay_addr`,
open a per-hop WebSocket to the relay's `/rpc/{peer_id}` endpoint and send the activation through it.
## Protocol
```
Node A → WS connect wss://relay/rpc/{peer_id_B}
→ send JSON: {request_id, method, path, headers, body_base64}
Relay → forward as relay-http-request envelope to Node B's persistent WS
Node B → process /forward locally
→ send relay-http-response envelope back
Relay → resolve future, send response JSON to Node A
Node A ← {request_id, status, headers, body_base64}
```
Binary activations (bfloat16) are base64-encoded. No precision loss.
## Acceptance criteria
- [ ] `_relay_hop(relay_addr, path, body, headers, timeout)` in `torch_server.py` — opens WS, sends, receives, returns `(status, headers_lower, body_bytes)`
- [ ] `_get_remaining_route` returns `list[dict]` with `relay_addr` field (was `list[tuple]`)
- [ ] `_run_downstream_pipeline` dispatches via `_relay_hop` when hop has `relay_addr`; falls back to direct HTTP if relay connection fails
- [ ] Tracker `_handle_proxy_chat` includes `relay_addr` in downstream hop dicts when node has one
- [ ] `relay_bridge._handle_request` decodes `body_base64`; response uses `body_base64` for `octet-stream` content
- [ ] All 157 tests pass (`python -m pytest`)
- [ ] QUICKSTART.md updated with relay NAT/internet architecture and test scenario
## WSL2 test scenario
Start two nodes in WSL2 pointing at the public tracker. Both get `172.x.x.x` endpoints (unreachable from outside). Both connect to relay automatically. Send an inference request through the tracker — activations flow via relay. No `--advertise-host` needed.

701
.scratch/distributed-inference-network/prd.json
View File

@@ -1,701 +0,0 @@
{
"name": "Distributed Inference Network",
"description": "Build a distributed inference network with node, gateway, tracker, SDK, contracts, and P2P shard distribution components from the grill session PRD.",
"branchName": "ralph/distributed-inference-network",
"userStories": [
{
"id": "US-001",
"title": "01 — Monorepo scaffold + single-node smoke test",
"description": "Stand up the monorepo package layout and prove that a client HTTP request can travel end-to-end through the stack — gateway → one node serving all layers → valid response — before any real model or distributed logic is wired in. The six top-level packages should exist with minimal stubs: - `packages/node` — node client CLI (`meshnet-node`) - `packages/gateway` — HTTP gateway + route orchestration - `packages/tracker` — node registry and route selection - `packages/sdk` — `meshnet` Python SDK - `packages/contracts` — Solana smart contract wrappers - `packages/p2p` — gossip and shard swarm The smoke test uses a tiny in-process stub model (not a real LLM) so the test is fast and has no external dependencies. The node runs in the same process as the test. The gateway is a real HTTP server. The client sends a real `POST /v1/chat/completions` request and receives a valid OpenAI-format response.",
"acceptanceCriteria": [
"`pip install -e packages/node packages/gateway packages/tracker` works from repo root",
"`meshnet-node`, `meshnet-gateway`, and `meshnet-tracker` CLI entry points exist",
"A single integration test starts a gateway and one stub node in-process, sends a `POST /v1/chat/completions` request, and asserts a valid OpenAI-format response is returned",
"The test passes with `pytest` from repo root with no external services running",
"All six package directories exist with `pyproject.toml` and an importable top-level module",
"Follow TDD where code is added: add/adjust an observable behavior test before implementation when practical",
"Run the task-specific tests and ensure they pass",
"Run `python -m pytest` from repo root and ensure it passes, or document why no test suite exists yet",
"Keep the implementation scoped to this story; do not implement downstream stories early",
"Commit only this story's code changes with a focused conventional commit message"
],
"priority": 1,
"passes": true,
"notes": "Source issue: .scratch/distributed-inference-network/issues/01-monorepo-scaffold.md",
"dependsOn": [],
"completionNotes": "Completed by Ralph iteration 7b260695; verified by pytest and editable package installs.",
"status": "done"
},
{
"id": "US-002",
"title": "02 — Two-node shard pipeline",
"description": "Extend the single-node smoke test so that inference is routed through exactly two nodes, each serving half of a stub model's layers. Activation tensors must travel from the gateway to node A, then from node A to node B, and the final output returns to the gateway. This proves the distributed shard pipeline works end-to-end before real models or a real tracker are introduced. The two nodes run in the same process as the test (no real network required). The gateway hardcodes the two-node route for now — dynamic route selection comes in issue 03. The activation tensor protocol (shape, dtype, serialisation format) must be established here, as all later issues depend on it. Use the domain vocabulary from `CONTEXT.md`: the ordered list of nodes is an **inference route**; each node's layer range is a **shard**; the tensors passed between nodes are activations (not \"data\" or \"chunks\").",
"acceptanceCriteria": [
"The integration test spins up two stub nodes, each configured with a non-overlapping shard range",
"A `POST /v1/chat/completions` request results in activation tensors flowing node-A → node-B (verifiable via test assertions or logs)",
"The gateway assembles the final response correctly from node-B's output",
"The test passes with no external services running",
"The activation tensor serialisation format is documented in a short inline comment (shape, dtype, wire format) — this becomes the contract all future nodes implement",
"Follow TDD where code is added: add/adjust an observable behavior test before implementation when practical",
"Run the task-specific tests and ensure they pass",
"Run `python -m pytest` from repo root and ensure it passes, or document why no test suite exists yet",
"Keep the implementation scoped to this story; do not implement downstream stories early",
"Commit only this story's code changes with a focused conventional commit message"
],
"priority": 2,
"passes": true,
"notes": "Source issue: .scratch/distributed-inference-network/issues/02-two-node-shard-pipeline.md",
"dependsOn": [
"US-001"
],
"completionNotes": "Tests pass; activation tensor flow via stub nodes verified. New HF-model path tested in test_node_startup.py.",
"status": "done",
"status_reason": "Base64 JSON wire format established here was replaced by binary HTTP protocol in US-011. tests/test_two_node_pipeline.py needs verification it exercises the new binary format end-to-end."
},
{
"id": "US-003",
"title": "03 — Tracker: node registration + route selection",
"description": "Replace the hardcoded two-node route from issue 02 with a real tracker. Nodes register themselves with the tracker on startup (endpoint, shard range, hardware profile, node score). The gateway queries the tracker for an inference route for a given model preset instead of using a static list. The tracker returns an ordered list of node endpoints whose shards collectively cover all layers. The tracker runs as a lightweight HTTP service. Node score is initially a simple placeholder (e.g. fixed value or random) — real throughput/latency scoring comes later. The tracker must correctly reject route requests when no registered nodes cover a required shard range and return an appropriate error. This issue establishes the tracker's registration and routing API contract, which issues 04 (node startup) and 05 (OpenAI gateway) both depend on.",
"acceptanceCriteria": [
"A node can register with the tracker via HTTP, providing its endpoint, shard range, and a hardware profile",
"The gateway queries the tracker with a model preset name and receives an ordered list of node endpoints forming a complete inference route",
"The end-to-end integration test from issue 02 still passes with the tracker now in the loop (no hardcoded routes)",
"The tracker returns a clear error response when no route is available for a requested model preset",
"Nodes that fail to heartbeat within a configurable window are removed from the registry",
"The tracker's registration and route-selection HTTP API is defined (paths, request/response shapes) in the tracker package",
"Follow TDD where code is added: add/adjust an observable behavior test before implementation when practical",
"Run the task-specific tests and ensure they pass",
"Run `python -m pytest` from repo root and ensure it passes, or document why no test suite exists yet",
"Keep the implementation scoped to this story; do not implement downstream stories early",
"Commit only this story's code changes with a focused conventional commit message"
],
"priority": 3,
"passes": true,
"notes": "Source issue: .scratch/distributed-inference-network/issues/03-tracker-registration-and-routing.md",
"dependsOn": [
"US-002"
],
"completionNotes": "Completed by Ralph iteration 79796dd2; verified by pytest, compileall, editable installs, CLI help, and spec/code reviews.",
"status": "done"
},
{
"id": "US-004",
"title": "04 — Node client startup flow (`meshnet-node start`)",
"description": "Make `meshnet-node start` self-configuring from scratch on a machine with a CUDA-capable GPU (or CPU fallback). The full startup sequence must complete without any manual configuration: 1. Detect GPU model and available VRAM 2. Load an existing Solana wallet from disk, or generate and save a new one 3. Query the tracker for the optimal shard assignment given the hardware profile 4. Download the assigned shard from HuggingFace (`huggingface_hub`) 5. Register with the tracker (wallet address, endpoint, shard range, hardware profile) 6. Begin accepting inference connections The node prints a short status summary on startup: wallet address, assigned shard, model preset, and download progress. No interactive prompts — the entire flow is non-interactive. This is the primary viral growth vector. Every second of friction in this flow costs node operators. The startup sequence must work on Linux with CUDA, and degrade gracefully to CPU on machines without a GPU.",
"acceptanceCriteria": [
"`meshnet-node start --tracker http://localhost:8080` completes startup without prompts on a machine with a CUDA GPU",
"A Solana wallet keypair is generated and saved to `~/.config/meshnet/wallet.json` if none exists",
"The assigned shard is downloaded from HuggingFace and cached to `~/.cache/meshnet/shards/`",
"The node registers with the tracker and appears in the tracker's node registry",
"The node accepts a live inference connection and processes activation tensors correctly after startup",
"On a CPU-only machine, the node starts with a warning and serves a CPU-appropriate shard",
"An integration test covers the full startup sequence against a local tracker stub",
"Follow TDD where code is added: add/adjust an observable behavior test before implementation when practical",
"Run the task-specific tests and ensure they pass",
"Run `python -m pytest` from repo root and ensure it passes, or document why no test suite exists yet",
"Keep the implementation scoped to this story; do not implement downstream stories early",
"Commit only this story's code changes with a focused conventional commit message"
],
"priority": 4,
"passes": true,
"notes": "Source issue: .scratch/distributed-inference-network/issues/04-node-client-startup.md",
"dependsOn": [
"US-003"
],
"completionNotes": "Completed by Ralph iteration 86510a10; verified by pytest, compileall, editable installs, CLI help, and final review.",
"status": "done"
},
{
"id": "US-005",
"title": "05 — OpenAI-compatible gateway",
"description": "Expose a production-shape HTTP API from the gateway so that any client using the OpenAI Python SDK (or any OpenAI-compatible tool) works by changing only the `base_url`. The gateway translates OpenAI-format requests into inference route execution and streams responses back in OpenAI-format. Endpoints to implement: - `POST /v1/chat/completions` — streaming (`text/event-stream`) and non-streaming - `GET /v1/models` — returns the list of model presets currently routable on the network - `GET /v1/health` — liveness check The gateway selects an inference route from the tracker, executes the shard pipeline, and assembles the streamed response. If the tracker returns no route for the requested model, the gateway responds with a standard OpenAI-format error (`model_not_available`). Authentication (API key → SOL/USDC balance) is a stub in this issue — return 200 for any non-empty `Authorization` header. Real payment gating comes in issue 06.",
"acceptanceCriteria": [
"`openai.OpenAI(base_url=\"http://localhost:8080/v1\", api_key=\"test\").chat.completions.create(model=\"stub-model\", messages=[...])` returns a valid response",
"Streaming works: `stream=True` returns `text/event-stream` chunks in OpenAI SSE format",
"`GET /v1/models` returns a JSON array of available model preset names",
"A request for an unavailable model returns an OpenAI-format error response with HTTP 503",
"LangChain `ChatOpenAI(base_url=..., api_key=...)` works against the gateway",
"An integration test covers streaming and non-streaming paths end-to-end through a real tracker and two stub nodes",
"Follow TDD where code is added: add/adjust an observable behavior test before implementation when practical",
"Run the task-specific tests and ensure they pass",
"Run `python -m pytest` from repo root and ensure it passes, or document why no test suite exists yet",
"Keep the implementation scoped to this story; do not implement downstream stories early",
"Commit only this story's code changes with a focused conventional commit message"
],
"priority": 5,
"passes": true,
"notes": "Source issue: .scratch/distributed-inference-network/issues/05-openai-compatible-gateway.md",
"dependsOn": [
"US-003"
],
"completionNotes": "All 6 gateway tests pass including OpenAI SDK, LangChain, streaming, and 503 for unknown model.",
"status": "done",
"status_reason": "Gateway is currently the pipeline orchestrator. US-014 moves orchestration to tracker-nodes; gateway becomes a thin load-balancer proxy. Implementation will be superseded — defer rework until US-014 lands."
},
{
"id": "US-006",
"title": "06 — Solana stake + settlement contracts",
"description": "Deploy and integrate the Solana smart contracts that make node staking, client payment, and token reward settlement trustless. All development and testing targets **Solana testnet** — never devnet or mainnet during development, to avoid real costs.",
"acceptanceCriteria": [
"All contracts deploy successfully to Solana testnet",
"A node can submit a stake transaction and have its balance reflected in the registry contract",
"A client can fund an API key account with testnet SOL",
"After a completed inference session, compute attribution is recorded on-chain with correct node/layer attribution",
"The epoch settlement transaction correctly distributes token rewards to node operators and deducts client balances",
"The gateway refuses to route to a node whose stake balance is below the minimum threshold",
"All contract interactions in tests run against a local Solana test validator (via `solana-test-validator`) — no live testnet required for CI",
"A `.env.testnet` config points to Solana testnet RPC for manual end-to-end testing",
"python -m pytest passes from repo root",
"Commit only this story's changes"
],
"priority": 6,
"passes": true,
"notes": "Source issue: .scratch/distributed-inference-network/issues/06-solana-stake-and-settlement.md",
"dependsOn": [
"US-003"
],
"completionNotes": "Completed by fresh Ralph/Codex session c257ffde with controller patches for clarified economics; verified by pytest, compileall, and diff check.",
"status": "done"
},
{
"id": "US-007",
"title": "07 — Fraud detection: validator + on-chain slash",
"description": "Implement the optimistic fraud detection loop. After each inference request completes, the validator process independently decides whether to re-run the request on a reference node (~5% sample rate).",
"acceptanceCriteria": [
"The validator process samples ~5% of completed inference requests (configurable)",
"A node returning a deliberately wrong output is detected and slashed within one validation cycle",
"The on-chain stake balance of the slashed node decreases by the correct slash amount",
"The strike count for the slashed node increments on-chain",
"A node that reaches the strike threshold is excluded from route selection on the next gateway request",
"A slashed node logs a clear warning to stdout",
"An integration test: run a deliberately-bad node, send 20 requests, assert at least one slash transaction is submitted and the node is eventually excluded from routes",
"python -m pytest passes from repo root",
"Commit only this story's changes"
],
"priority": 7,
"passes": true,
"notes": "Source issue: .scratch/distributed-inference-network/issues/07-fraud-detection-slash.md",
"dependsOn": [
"US-005",
"US-006"
],
"completionNotes": "Completed by fresh Ralph/Codex session 04475912 with controller fix for duplicate slash suppression; verified by pytest, compileall, and diff check.",
"status": "done"
},
{
"id": "US-008",
"title": "08 — Node probationary period + ban enforcement",
"description": "Implement the two anti-sybil mechanisms that make re-entering the network after a ban economically costly.",
"acceptanceCriteria": [
"A node with a new wallet receives no token rewards for its first N jobs (verified via settlement contract state)",
"The node client prints remaining probationary jobs on startup",
"After N jobs, the next epoch settlement correctly credits the node with token rewards",
"A wallet with strike count at the threshold is marked banned in the registry contract",
"The tracker excludes banned wallets from route selection",
"A banned wallet that attempts to register with the tracker is rejected",
"An integration test covers: new wallet → N jobs → earning begins; and: strike threshold reached → banned → excluded from routes",
"python -m pytest passes from repo root",
"Commit only this story's changes"
],
"priority": 8,
"passes": true,
"notes": "Source issue: .scratch/distributed-inference-network/issues/08-probationary-period-and-bans.md",
"dependsOn": [
"US-007"
],
"completionNotes": "Completed by fresh Ralph/Codex session db3f5c10 with controller test fix for banned registration semantics; verified by pytest, compileall, and diff check.",
"status": "done"
},
{
"id": "US-009",
"title": "09 — P2P shard swarm",
"description": "Once a node has downloaded a shard, it seeds that shard to other nodes that are assigned the same shard.",
"acceptanceCriteria": [
"A node that has downloaded a shard is listed as a peer for that shard in the tracker",
"A second node assigned the same shard downloads it from the first node (peer) rather than HuggingFace",
"If no peers are available, the node falls back to HuggingFace without error",
"The node client logs whether the shard was downloaded from a peer or HuggingFace",
"Shard chunks are verified against a checksum before being marked complete",
"An integration test: node A downloads shard from HuggingFace stub; node B with same assignment downloads from node A",
"python -m pytest passes from repo root",
"Commit only this story's changes"
],
"priority": 9,
"passes": true,
"notes": "Source issue: .scratch/distributed-inference-network/issues/09-p2p-shard-swarm.md",
"dependsOn": [
"US-004"
],
"completionNotes": "Completed by fresh Ralph/Codex session 243fae88 with controller fix for streaming tar archives; verified by pytest, compileall, and diff check.",
"status": "done"
},
{
"id": "US-010",
"title": "10 — `meshnet` Python SDK",
"description": "A Python SDK (`packages/sdk`) that wraps the OpenAI-compatible gateway and exposes network-specific controls.",
"acceptanceCriteria": [
"`pip install meshnet` installs the SDK",
"`Client.chat.completions.create(...)` works identically to the OpenAI SDK",
"`client.wallet.balance()` returns the current SOL/USDC balance for the API key",
"`client.wallet.top_up()` returns a valid Solana payment address",
"`client.models.available()` returns model presets with shard coverage percentage",
"`client.estimate_cost(model, tokens)` returns a cost estimate in SOL",
"`client.request(redundancy=2)` sends to two independent inference routes and returns majority response",
"The SDK is typed (py.typed, full type stubs)",
"An integration test covers each SDK method against a local gateway + tracker + stub nodes",
"python -m pytest passes from repo root",
"Commit only this story's changes"
],
"priority": 10,
"passes": true,
"notes": "Source issue: .scratch/distributed-inference-network/issues/10-meshnet-sdk.md",
"dependsOn": [
"US-005",
"US-006"
],
"completionNotes": "Completed by fresh Ralph/Codex session ef4eea0e; verified by pytest, compileall, editable SDK install, and diff check.",
"status": "done"
},
{
"id": "US-011",
"title": "11 — Binary activation wire format",
"description": "Replace the base64 JSON activation payload with raw binary HTTP bodies, zstd compression, and chunked prefill (128 tokens/chunk). All nodes and the gateway must be migrated. Stub nodes continue to emit zeroed tensors, just in binary. This is a protocol prerequisite for US-012 (real model backend).",
"acceptanceCriteria": [
"Node /forward endpoint reads shape/dtype/session/chunk from HTTP headers; body is raw binary (optionally zstd-compressed)",
"Node /forward response is raw binary with the same header set",
"Gateway splits prompts > MESHNET_CHUNK_TOKENS (default 128) into sequential chunks sent through the pipeline",
"Integration test: 512-token stub activation (4 chunks) through a two-node pipeline returns 4 valid binary chunk responses",
"zstd Python package added as a dependency to packages/node and packages/gateway",
"_make_stub_activations replaced with _make_stub_binary_activation(shape, dtype) -> bytes",
"python -m pytest passes from repo root",
"Commit only this story's changes"
],
"priority": 11,
"passes": true,
"notes": "Source issue: .scratch/distributed-inference-network/issues/11-binary-wire-format.md",
"dependsOn": [
"US-002"
],
"completionNotes": "Completed by fresh Ralph/Codex session 3f3bed75; verified by pytest, compileall, and diff check.",
"status": "done"
},
{
"id": "US-012",
"title": "12 — Real PyTorch model backend",
"description": "Replace stub node inference with actual transformers layer execution. Node loads HuggingFace SafeTensors model shard (model.model.layers[start:end]), runs real forward passes in bfloat16, handles head (embed_tokens) and tail (lm_head) responsibilities, and supports bitsandbytes NF4/INT8/bfloat16 quantization via --quantization flag. Test model: openai-community/gpt2.",
"acceptanceCriteria": [
"meshnet-node start --model-id openai-community/gpt2 --shard-start 0 --shard-end 6 loads the shard without error",
"Head node (shard_start==0) loads tokenizer and embed_tokens",
"Tail node (shard_end==total_layers) loads model.norm and lm_head",
"Two-node GPT-2 integration test returns deterministic coherent text completion",
"--quantization [bfloat16|int8|nf4] flag supported; bfloat16 default",
"Node with insufficient VRAM prints clear error and exits",
"transformers, bitsandbytes, safetensors, accelerate added as node dependencies",
"Integration tests marked @pytest.mark.integration, skipped in CI without GPU",
"python -m pytest passes from repo root",
"Commit only this story's changes"
],
"priority": 12,
"passes": true,
"notes": "Source issue: .scratch/distributed-inference-network/issues/12-real-pytorch-model-backend.md",
"dependsOn": [
"US-011"
],
"completionNotes": "Completed by agent",
"status": "done"
},
{
"id": "US-013",
"title": "13 — Coverage-first tracker shard assignment",
"description": "Upgrade tracker route selection to coverage-first, speed-weighted bin-packing. Tracker maintains a live coverage map per model, issues LOAD_SHARD/DROP_SHARD rebalance directives when coverage drops, and assigns shard ranges using declared VRAM, quantization, and benchmark throughput. A model is only routable when all layer ranges have node_count >= 1.",
"acceptanceCriteria": [
"Node registration accepts vram_bytes, ram_bytes, quantizations[], benchmark_tokens_per_sec",
"GET /v1/coverage/<model_preset> returns list of {start_layer, end_layer, node_count}",
"Model is unroutable when any layer range has node_count=0",
"New node gets assigned to highest-priority uncovered range first",
"Node disconnection triggers LOAD_SHARD directive to idle node within 30 seconds",
"Shard assignment respects VRAM: assigned_layers <= floor((vram_bytes * 0.8) / bytes_per_layer_at_quant)",
"Faster node receives wider shard range when both can cover the same gap",
"Integration test: three nodes with different VRAM show widest range on largest-VRAM node with 100% coverage",
"Integration test: kill middle-range node → tracker issues LOAD_SHARD → coverage recovers",
"python -m pytest passes from repo root",
"Commit only this story's changes"
],
"priority": 13,
"passes": true,
"notes": "Source issue: .scratch/distributed-inference-network/issues/13-coverage-first-shard-assignment.md",
"dependsOn": [
"US-003",
"US-012"
],
"completionNotes": "Completed by agent",
"status": "done"
},
{
"id": "US-014",
"title": "14 — Tracker-as-first-layer-node (inference entry point)",
"description": "Merge inference orchestration into tracker nodes that serve the first-layer shard. A tracker node exposes /v1/chat/completions directly: tokenizes input, runs embed_tokens + layers[0..k], selects onward route from coverage map, forwards binary activations, receives tail output, and streams tokens back. The standalone gateway becomes a thin load-balancer routing to tracker nodes.",
"acceptanceCriteria": [
"Node with shard_start==0 (or --tracker-mode flag) exposes /v1/chat/completions alongside /forward",
"Tracker-node /v1/chat/completions: tokenize → embed → own layers → route selection → forward binary → receive tail → stream SSE",
"Two tracker nodes for same model each handle requests independently",
"Gateway proxies /v1/chat/completions to tracker nodes (round-robin), discovered via GET /v1/tracker-nodes/<model_preset>",
"Integration test: two tracker nodes + two mid-shard nodes for GPT-2; 10 requests via gateway; both tracker nodes receive load; all responses valid",
"Existing US-005 OpenAI-compatible tests still pass",
"python -m pytest passes from repo root",
"Commit only this story's changes"
],
"priority": 14,
"passes": true,
"notes": "Source issue: .scratch/distributed-inference-network/issues/14-tracker-as-node.md",
"dependsOn": [
"US-012",
"US-013"
],
"status": "done",
"completionNotes": "Implemented: (1) Tracker GET /v1/tracker-nodes/<model> endpoint returning nodes registered with tracker_mode=true at shard_start==0. (2) StubNodeServer and TorchNodeServer now accept tracker_mode/tracker_url params; when tracker_mode=True, /v1/chat/completions is served alongside /forward. (3) TorchNodeServer auto-detects tracker mode when shard_start==0. (4) Gateway _handle_chat_completions checks for tracker-nodes first via _get_tracker_nodes(), proxies round-robin if found, falls back to existing direct pipeline if none (backward compat). (5) CLI --tracker-mode and --tracker-url flags added. (6) Integration test: 2 tracker-nodes + 2 mid-shard nodes for gpt2; 10 requests; round-robin verified (5/5 split); all responses valid OpenAI format. All 78 tests pass."
},
{
"id": "US-015",
"title": "15 — Ralph: agent-agnostic runner + status-field-aware dashboard",
"description": "Two improvements to the Ralph workflow tooling. (1) Status-field awareness: replace all passes:true/false reads in ralph_progress.py with the rich status field. Surface to-revise and needs-review stories as an attention list in the dashboard; auto command skips them by default. (2) Agent agnosticism: make the agent selectable per session — codex (existing), claude (Claude Code CLI), openrouter (unified API for GPT-4/Mistral/Llama/DeepSeek via OPENROUTER_API_KEY + --model), or custom (--agent-cmd path to any script). Persist agent choice to .ralph-tui/agent-config.json. When ralph-tui does not natively support a requested agent, ralph_progress.py falls back to a thin adapter that calls the agent API directly with the task prompt. (3) Worktree parallelism: ralph_progress.py auto --parallel [N] creates one git worktree per ready task (git worktree add ../AI-worktree-<id> -b feat/<id>), runs up to N agent sessions concurrently, and merges each branch back to master after the agent exits 0 and tests pass. Non-blocking stories that share no file overlap can safely run in parallel. ralph_progress.py list-parallel shows which open stories have no overlapping dependsOn chains and are safe to run concurrently.",
"acceptanceCriteria": [
"All story.get('passes') reads in ralph_progress.py replaced with _is_done() helper that reads status field with passes fallback",
"_story_sets() returns six buckets: done, attention (to-revise/needs-review), active (in-progress), in-design, ready, blocked",
"Dashboard shows ⚠ Attention required section listing to-revise/needs-review stories with status_reason",
"auto command skips to-revise and needs-review stories; --include-revise flag overrides",
"_review_report includes Attention Required section with status_reason for all affected stories",
"ralph_progress.py set-agent --agent <name> [--model <model>] writes .ralph-tui/agent-config.json",
"--agent codex|claude|openrouter|custom accepted by run-next, auto, review subcommands",
"run-next --agent custom --agent-cmd ./my-agent.sh runs a task via custom script (prompt file as $1)",
"Saved agent config is loaded as default when --agent is not passed on the CLI",
"python -m pytest passes from repo root",
"ralph_progress.py auto --parallel 2 starts two worktrees concurrently for the two highest-priority ready tasks",
"Each worktree is created at ../AI-worktree-<story-id> on branch feat/<story-id>",
"After agent exits 0 and pytest passes in the worktree, the branch is merged to master and the worktree removed",
"ralph_progress.py list-parallel prints the set of open stories with no shared dependency chain (safe to parallelize)",
"If a worktree merge fails (conflict), the worktree is preserved for manual resolution and reported clearly"
],
"priority": 15,
"passes": true,
"status": "done",
"notes": "Source issue: .scratch/distributed-inference-network/issues/15-ralph-agent-agnostic-status-aware.md",
"dependsOn": [],
"completionNotes": "Implemented by agent: status-aware helpers (_is_done, _needs_attention, _is_active, _is_in_design), 6-bucket _story_sets, attention dashboard section, _review_report Attention Required block, auto --include-revise, set-agent subcommand with persistent agent-config.json, _run_openrouter stub, custom agent support, list-parallel subcommand, and auto --parallel N worktree orchestration. All 65 tests pass."
},
{
"id": "US-016",
"title": "16 — Mining-style node startup CLI + live dashboard",
"description": "Replace the bare flag-driven node CLI with a wizard-guided first-run experience (like a GPU mining client) followed by a live terminal dashboard once the node is running. On first run, the wizard auto-detects GPU VRAM, presents a curated list of compatible models with VRAM requirements at each quantization level, lets the user pick a download location, and writes a persistent config file so subsequent starts are one command. Once the node is running, the wizard gives way to a rich live status panel showing: GPU temp + VRAM used, tokens/sec, requests served, peers connected, TAI earned (stub until US-006 is live). A Browse HuggingFace option calls the HF Hub API so users can load any HF model beyond the curated list.",
"acceptanceCriteria": [
" with no args and no config file enters the interactive setup wizard",
"Wizard step 1: auto-detect GPU(s) via torch.cuda / torch.version.hip; print GPU name + total VRAM",
"Wizard step 2: show curated model list (name, HF repo, layers, VRAM@NF4/INT8/BF16); mark models that do NOT fit available VRAM as [too large]",
"Wizard step 3: offer [B] Browse HuggingFace — calls HF Hub API (huggingface_hub.list_models filtered by pipeline_tag=text-generation, sorted by downloads, top 20) and lets user enter a custom HF repo ID",
"Wizard step 4: prompt for download directory (default ~/.meshnet/models/); validate writable; show estimated disk usage for chosen model+quantization",
"Wizard step 5: prompt for tracker URL (default http://localhost:8080); validate connection",
"Wizard writes ~/.config/meshnet/config.json; second run skips wizard and starts directly",
"All wizard values overridable via CLI flags: --model, --download-dir, --quantization [bf16|int8|nf4], --tracker, --wallet, --reset-config",
"Once node is running, wizard clears and a live dashboard renders every 2s (rich.live): GPU util%, VRAM used/total, tokens/sec (EMA), requests served, TAI earned (stub 0.0), peers connected, uptime, current model/shard range",
"Dashboard exits cleanly on Ctrl-C with a summary line",
"Works inside WSL2 (no termios/ioctl calls that fail on Windows terminal; fall back to plain-text status if rich is not available)",
" passes from repo root",
"Commit only this story changes"
],
"priority": 16,
"status": "done",
"notes": "Source issue: .scratch/distributed-inference-network/issues/16-mining-cli-ux.md",
"dependsOn": [
"US-004",
"US-012"
],
"completionNotes": "Implemented: mining-style wizard with GPU detection, curated model list (7 models with NF4/INT8/BF16 VRAM requirements), HF Hub browse, persistent config, rich live dashboard with plain-text WSL2 fallback. 19 tests, 97 passed total."
},
{
"id": "US-017",
"title": "17 — P2P gossip, NAT-traversal relay node, and SSL/TLS",
"description": "Add a gossip layer so nodes discover each other and propagate coverage-map changes without polling the tracker continuously. Introduce a publicly-hosted relay node (run by the team) that solves NAT traversal using circuit relay (Petals-style) and serves as the bootstrap peer list for new nodes. Encrypt all node-to-node and node-to-tracker communications with TLS. Gossip protocol: WebSocket-based PubSub over wss://, topics: node-join / node-leave / coverage-update / heartbeat. Peer discovery: mDNS (zeroconf) for LAN, public relay bootstrap list for internet. NAT traversal: relay node acts as TCP-level circuit relay when direct connection fails (hole-punching first, relay second). Architecture is designed to migrate to libp2p GossipSub + Kademlia DHT in a future story without breaking the message schema.",
"acceptanceCriteria": [
"All HTTP between nodes and tracker uses HTTPS (TLS 1.3); self-signed cert generated on first run and fingerprint pinned in config; relay node uses Let's Encrypt",
"Nodes broadcast node-join / node-leave events over wss:// to known peers within 1s of registration",
"mDNS peer discovery (Python zeroconf) finds other meshnet nodes on the same LAN segment without manual tracker URL entry",
"Public relay bootstrap list (hardcoded relay URL + ) is consulted when no LAN peers found",
"Relay node is a standalone meshnet package () with CLI: starts a WebSocket relay server + circuit relay + optional tracker proxy",
"When a node behind NAT cannot accept inbound connections, the relay forwards its traffic; node advertises relay address (relay_url/node_id) to tracker as its effective endpoint",
"Tracker accepts both direct node URLs and relay-proxied URLs in heartbeat payloads",
"Integration test: two nodes in separate processes on localhost (simulating NAT) communicate via a local relay process; inference request routes correctly",
"ADR-0010 documents the gossip protocol, relay architecture, and migration path to libp2p",
" passes from repo root",
"Commit only this story changes"
],
"priority": 17,
"status": "done",
"notes": "Source issue: .scratch/distributed-inference-network/issues/17-p2p-gossip-relay-ssl.md",
"dependsOn": [
"US-013",
"US-014"
],
"completionNotes": "Implemented: packages/p2p (identity, TLS cert+fingerprint, GossipClient WSS PubSub, MdnsDiscovery with zeroconf optional), packages/relay (RelayServer gossip hub + circuit relay proxy, meshnet-relay CLI), tracker extended with relay_addr/cert_fingerprint/peer_id, relay_bootstrap.json, ADR-0010. 18 new tests; 115 total passed."
},
{
"id": "US-018",
"title": "18 — End-to-end two-machine LAN inference test",
"description": "Prove the network works across two real machines: the Linux rig (this machine) and a Windows 11 rig running WSL2. One machine runs the tracker + first-shard node (inference entry point). The other machine runs a second-shard node. A client sends a real inference request and receives a streamed response. This story is primarily a test plan + setup guide + test execution script; it produces documented evidence (logs, timing, token output) that real distributed inference works. It also surfaces any real-world issues (port forwarding, CUDA driver version mismatches, WSL2 CUDA passthrough, model download paths) that need fixing.",
"acceptanceCriteria": [
"docs/INSTALL_WINDOWS.md exists: step-by-step WSL2 + CUDA + meshnet-node install on Windows 11",
"docs/TWO_MACHINE_TEST.md exists: how to start tracker on machine A, node on machine B, run inference, interpret output",
"A test script scripts/test_lan_inference.py: given --tracker-url, --gateway-url, sends 3 chat completion requests, asserts valid OpenAI format, prints token count + latency + which nodes served each request",
"Both machines can reach each other on LAN (documented: firewall rules, port list)",
"At least one successful inference recorded: the test script exits 0 with output showing tokens generated and node IDs",
"Latency breakdown logged: gateway→node-A, node-A→node-B, node-B→gateway (approximate, from server logs)",
"Known issues during test documented in docs/TWO_MACHINE_TEST.md under a Known Issues section",
"Commit only this story changes"
],
"priority": 18,
"status": "done",
"notes": "Source issue: .scratch/distributed-inference-network/issues/18-two-machine-lan-test.md",
"dependsOn": [
"US-016",
"US-017"
],
"completionNotes": "docs/INSTALL_WINDOWS.md: WSL2+CUDA+meshnet-node install guide. docs/TWO_MACHINE_TEST.md: two-machine LAN test procedure with known issues. scripts/test_lan_inference.py: stdlib-only test script, 3 requests, exit 0 on success, auto-discovers gateway from tracker."
},
{
"id": "US-019",
"title": "19 — Distributed tracker consensus (Raft assignments + CRDT heartbeats)",
"description": "Replace the single-point-of-failure tracker with a fault-tolerant cluster. Tracker nodes elect a leader via Raft and commit shard assignments as log entries — all tracker nodes agree on who owns what. Node liveness (heartbeats) uses CRDT gossip (eventual consistency, high frequency OK). A node registers with any tracker node; the write is forwarded to the leader and replicated to followers. A 3-node tracker cluster survives one tracker failure without losing assignment state. The relay/gossip layer already built in US-017 handles peer heartbeats; this story wires Raft on top for authoritative assignments.",
"acceptanceCriteria": [
"3 tracker nodes can be started and form a Raft cluster (leader election, log replication)",
"A node registers with any follower — the registration is forwarded to the leader and replicated",
"Killing the leader causes a new election within 5 seconds; registrations continue working",
"Shard assignments returned by any tracker node are identical (strong consistency)",
"Node heartbeats use CRDT gossip (not Raft) — high-frequency, eventual consistency",
"meshnet-tracker CLI gains --cluster-peers flag to specify peer tracker URLs",
"Integration test: 3 tracker nodes, kill leader mid-test, verify assignment still works",
"QUICKSTART.md updated with multi-tracker setup section"
],
"priority": 19,
"status": "done",
"notes": "Architecture decision: Raft for assignments (strong consistency) + CRDT gossip for liveness (eventual consistency). User approved 2026-06-29.",
"dependsOn": [
"US-017"
],
"completionNotes": "raft.py: minimal Raft consensus (leader election, log replication, AppendEntries, RequestVote). gossip.py: LWW CRDT gossip for node heartbeats. TrackerServer gains cluster_peers + cluster_self_url params. --cluster-peers and --self-url CLI flags added. 6 integration tests: leader election <1s, follower registration propagation, leader kill + re-election <5s, gossip table."
},
{
"id": "US-020",
"title": "20 — Tracker + node hardening: BrokenPipe fix, deterministic node IDs, HF coverage",
"description": "Polish pass on tracker and node after real multi-host LAN testing. Three concrete issues: (1) tracker _send_json crashed with BrokenPipeError when a slow-inference client disconnected mid-stream; (2) node IDs were random UUIDs, making re-registration after tracker restart create phantom entries; (3) HF-repo model coverage endpoint did not handle short-name vs full-repo lookups consistently.",
"acceptanceCriteria": [
"BrokenPipeError in tracker _send_json is silently swallowed (client disconnected is not an error)",
"Node IDs are deterministic (wallet-address + port hash) so re-registration after tracker restart reuses the same ID",
"GET /v1/coverage/<model> accepts both short names and full HF repo IDs",
"python -m pytest passes from repo root"
],
"priority": 20,
"status": "done",
"notes": "Discovered during first two-machine LAN test (US-018 follow-up).",
"dependsOn": [
"US-018"
],
"completionNotes": "BrokenPipeError wrapped in try/except in tracker and node send paths. Node IDs now sha256(wallet+port)[:16]. Coverage endpoint normalises short/full HF names."
},
{
"id": "US-021",
"title": "21 — --route-timeout CLI flag for node tracker route lookup",
"description": "The node's slow-path tracker route lookup used a hard-coded 30-second timeout. On high-latency networks (relay, 5G) or when the tracker is slow to respond, this caused premature failures. Expose it as a CLI flag so operators can tune it per deployment.",
"acceptanceCriteria": [
"meshnet-node start accepts --route-timeout <seconds> (float, default 30.0)",
"route_timeout is passed through to TorchNodeServer and used in tracker /v1/route HTTP call",
"TorchNodeServer exposes route_timeout as a readable property",
"Test: setting a non-default timeout is reflected on the running server object",
"python -m pytest passes"
],
"priority": 21,
"status": "done",
"notes": "Small QoL story; acceptance criteria driven by a failing test.",
"dependsOn": [
"US-016"
],
"completionNotes": "cli.py: --route-timeout added to start_cmd. TorchNodeServer._route_timeout stored, route_timeout property exposed. Test: test_route_timeout_config_is_exposed_on_server."
},
{
"id": "US-022",
"title": "22 — X-Meshnet-Start-Layer: overlapping shard execution protocol",
"description": "When two nodes register overlapping shard ranges (e.g. node A covers 0-15 and node B covers 12-23), the naive pipeline re-runs layers 12-15 on node B. The X-Meshnet-Start-Layer header tells each downstream node which layer index to start from, skipping already-computed layers. The tracker injects start_layer into X-Meshnet-Route hops at proxy time.",
"acceptanceCriteria": [
"Tracker _handle_proxy_chat builds route hops with start_layer computed from covered_up_to",
"Node _handle_binary_forward reads X-Meshnet-Start-Layer and passes it to backend.forward_bytes",
"Node _get_remaining_route parses start_layer from both injected header and /v1/route slow-path",
"TorchModelShard.forward_bytes accepts optional start_layer and skips layers below it",
"Test: two-node route with overlapping shards produces correct output without double-computing layers",
"python -m pytest passes"
],
"priority": 22,
"status": "done",
"notes": "Protocol decision: option A (start_layer injected by tracker, not negotiated peer-to-peer). Approved in design session.",
"dependsOn": [
"US-014",
"US-019"
],
"completionNotes": "X-Meshnet-Start-Layer header added to forward protocol. Tracker computes covered_up_to as it builds route_hops. model_backend.forward_bytes(start_layer=) implemented. Tests added for overlapping shard scenarios."
},
{
"id": "US-023",
"title": "23 — Heartbeat stats payload: request counters + dynamic reassignment response",
"description": "Node heartbeats are currently empty POSTs. Extend them to carry cumulative stats (total_requests, failed_requests, queue_depth, uptime_seconds) so the tracker can make informed load-balancing decisions. The heartbeat response may include a new_assignment field, enabling the tracker to redirect a node to a different shard without a restart.",
"acceptanceCriteria": [
"Heartbeat POST body includes total_requests, failed_requests, queue_depth, uptime_seconds, status",
"TorchNodeServer tracks total_requests, failed_requests, queue_depth with a threading.Lock",
"Tracker stores the last heartbeat payload per node and exposes it in /v1/nodes (list endpoint)",
"Heartbeat response may include new_assignment: {model, shard_start, shard_end}; node logs it",
"Stats survive tracker outage: buffered locally, flushed on next successful heartbeat",
"python -m pytest passes"
],
"priority": 23,
"status": "done",
"notes": "Reassignment is logged only for now; hot-reload (load new shard without restart) is a future story.",
"dependsOn": [
"US-016"
],
"completionNotes": "torch_server.py: total_requests, failed_requests, queue_depth added with _stats_lock. startup.py _start_heartbeat sends stats dict. Tracker stores last_heartbeat per node. new_assignment field in heartbeat response logged by node."
},
{
"id": "US-024",
"title": "24 — Enhanced availability map with per-node health details",
"description": "GET /v1/coverage/<model> returns coarse band-level coverage. Operators need to know which specific nodes are in each band and whether they are healthy (last heartbeat time, queue depth). Extend the response to include a nodes array per band with per-node health details, and add a separate detailed endpoint.",
"acceptanceCriteria": [
"GET /v1/coverage/<model> response includes nodes: [{node_id, endpoint, healthy, queue_depth, last_seen_s}] per band",
"healthy is true iff last heartbeat < heartbeat_timeout seconds ago",
"Existing band fields (start_layer, end_layer, node_count) are preserved",
"Tests updated: band assertions check nodes array not just node_count",
"python -m pytest passes"
],
"priority": 24,
"status": "done",
"notes": "Grouped-by-band format chosen (both node list and band metadata in same object). Dead nodes included in response with healthy=false.",
"dependsOn": [
"US-013"
],
"completionNotes": "_coverage_map_detailed added to tracker. Each band now includes nodes list. Tests updated to check band fields individually rather than exact dict comparison."
},
{
"id": "US-025",
"title": "25 — Model usage statistics: rolling RPM windows + SQLite persistence",
"description": "Trackers need to know which models are being requested most often to make smart load-balancing and assignment decisions. Add per-model rolling request-rate counters (last hour, last day, last month) with a circular-bucket implementation. Persist buckets to SQLite so stats survive tracker restarts. Support gossip-based stat sharing between tracker peers (additive merge, per-tracker slices).",
"acceptanceCriteria": [
"_RollingCounter: circular buckets, epoch-indexed, stale buckets auto-reset on record()",
"_StatsCollector: record_request(), get_local_rpms(), merge_peer_rpms(), get_combined_stats()",
"Stats persisted to SQLite (--stats-db PATH); loaded on startup",
"GET /v1/stats returns combined stats (local + peer slices) as {model: {rpm_last_hour, rpm_last_day, rpm_last_month}}",
"POST /v1/stats/gossip accepts a peer's local slice and merges it additively",
"_handle_proxy_chat records a stat after model is determined",
"6 unit tests covering counter, collector, merge, SQLite round-trip, gossip endpoint",
"python -m pytest passes"
],
"priority": 25,
"status": "done",
"notes": "Option B chosen: stats stored locally per tracker and synced via gossip (not aggregated centrally). Rolling windows: 60×1min buckets (1 hour), 24×1hr (1 day), 30×1day (~1 month).",
"dependsOn": [
"US-017"
],
"completionNotes": "_RollingCounter and _ModelStats classes in server.py. _StatsCollector with SQLite save/load. /v1/stats and /v1/stats/gossip endpoints. --stats-db CLI flag. Stats gossip in _stats_loop thread."
},
{
"id": "US-026",
"title": "26 — Smart model assignment via demand×coverage scoring",
"description": "The /v1/network/assign endpoint assigns new nodes to whichever model and shard range covers the biggest uncovered gap. This ignores demand: a model with 1000 RPM and 60% coverage should attract more nodes than a zero-traffic model with 50% coverage. Add a scoring formula: score = (demand_rpm + 1.0) × (coverage_deficit + 0.01) so high-demand under-covered models win. price_per_token is reserved in the protocol response at 0.0.",
"acceptanceCriteria": [
"/v1/network/assign returns the highest-scoring model+shard using demand×coverage formula",
"demand_rpm uses combined stats from _StatsCollector (local + peer slices)",
"coverage_deficit is fraction of layers with zero coverage [0.0, 1.0]",
"price_per_token: 0.0 included in response (reserved field)",
"Models with no traffic still compete by coverage (floor: demand_rpm + 1.0)",
"Test: two models, one high-demand low-coverage wins over low-demand high-coverage",
"python -m pytest passes"
],
"priority": 26,
"status": "done",
"notes": "Score formula chosen after grilling: (demand_rpm + 1.0) × (coverage_deficit + 0.01). The +1.0 floor means coverage alone drives assignment when demand is zero.",
"dependsOn": [
"US-025",
"US-024"
],
"completionNotes": "_handle_network_assign updated with scoring. _effective_throughput helper. price_per_token: 0.0 in response. Tests in test_tracker_routing.py."
},
{
"id": "US-027",
"title": "27 — Throughput-optimized routing: effective throughput as tiebreak",
"description": "The greedy max-reach route selection picks nodes by shard coverage but does not consider node speed. When two nodes cover the same remaining range, prefer the one with higher effective throughput (benchmark_tokens_per_sec / (queue_depth + 1)). This is a tiebreak only — coverage maximization remains the primary objective.",
"acceptanceCriteria": [
"_effective_throughput(node) = benchmark_tokens_per_sec / (queue_depth + 1)",
"_select_route uses throughput as tiebreak when shard_end is equal",
"Test: two nodes with same shard range, different throughput — faster node is picked",
"Existing greedy coverage tests still pass (throughput does not change primary selection)",
"python -m pytest passes"
],
"priority": 27,
"status": "done",
"notes": "Throughput tiebreak only — coverage-maximizing greedy stays primary. queue_depth from last heartbeat.",
"dependsOn": [
"US-023"
],
"completionNotes": "_effective_throughput added to server.py. _select_route updated: when shard_end equal, max throughput wins. Tests added and existing tests corrected for new tiebreak order."
},
{
"id": "US-028",
"title": "28 — Routing correctness tests: three-node, overlap, and throughput scenarios",
"description": "Add a comprehensive test suite for route selection covering: greedy three-node no-overlap ordering, overlapping shard handling, throughput tiebreak, and gap detection. These tests lock in the routing contract so refactors don't silently regress.",
"acceptanceCriteria": [
"test_select_route_no_overlap_three_nodes: greedy picks nodes in layer order (A→C→B for ranges 0-7, 8-15, 16-23)",
"test_select_route_with_overlap: overlapping nodes correctly resolved",
"test_select_route_throughput_tiebreak: faster node wins when reach is equal",
"test_select_route_gap_leaves_error: partial coverage returns error",
"All tests pass without mocking (in-process tracker server)",
"python -m pytest passes"
],
"priority": 28,
"status": "done",
"notes": "Tests revealed the greedy picks A→C→B (not A→B→C) for non-overlapping ranges when C starts at a lower layer than B. Test expectation corrected to match algorithm.",
"dependsOn": [
"US-027"
],
"completionNotes": "7 routing tests added to test_tracker_routing.py. _make_node helper. test_select_route_no_overlap_three_nodes corrected: greedy outputs [A, C, B] when C.shard_start < B.shard_start."
},
{
"id": "US-029",
"title": "29 — Outbound relay client: NAT/internet pipeline hops",
"description": "Nodes behind NAT (WSL2, 5G, home routers) register with the tracker including a relay_addr (wss://relay/rpc/{peer_id}). When the head node needs to forward activations to a behind-NAT peer, it must use the relay instead of direct HTTP. Add _relay_hop() to torch_server.py that opens a per-hop WebSocket to the relay's /rpc/{peer_id} endpoint, sends the binary activation (base64-encoded), and returns the response. If relay fails, fall back to direct HTTP.",
"acceptanceCriteria": [
"_relay_hop(relay_addr, path, body, headers, timeout) opens WS to relay, sends body_base64, returns (status, headers, body)",
"_get_remaining_route returns list[dict] with relay_addr field (was list[tuple])",
"_run_downstream_pipeline dispatches via _relay_hop when hop has relay_addr",
"Falls back to direct HTTP if relay connection fails (logs warning)",
"Tracker _handle_proxy_chat includes relay_addr in downstream hop dicts",
"relay_bridge._handle_request decodes body_base64; response uses body_base64 for binary (octet-stream)",
"All 157 tests pass",
"QUICKSTART.md updated with relay NAT/internet section"
],
"priority": 29,
"status": "done",
"notes": "relay_addr format: wss://relay.../rpc/{peer_id}. Binary activations (bfloat16) base64-encoded through relay JSON protocol — no precision loss. WSL2 nodes now work behind NAT without --advertise-host.",
"dependsOn": [
"US-017",
"US-022"
],
"completionNotes": "_relay_hop() added to torch_server.py. _get_remaining_route returns list[dict]. relay_bridge.py updated with body_base64 support. Tracker injects relay_addr into downstream hop dicts. 157 tests pass."
}
],
"metadata": {
"updatedAt": "2026-06-29T15:35:00.000Z",
"statusVocabulary": {
"open": "Not started",
"in-design": "Decisions pending before implementation can begin",
"in-progress": "Agent actively working on it",
"needs-review": "Implemented, awaiting human review or verification",
"to-revise": "Previously done, needs rework due to design or architecture changes",
"done": "Implemented, tested, and verified",
"blocked": "Waiting on unresolved external dependency"
}
}
}