neuron-tai/docs/PRD.md

Status: done (US-001…US-035 complete; friends-test arc US-036…US-049 in `docs/prd.json`; US-048/050 tracked. See ADRs 0015–0018, 0023, 0025–0026.)

# Distributed Inference Network — PRD

## Problem Statement

Running large language models requires expensive dedicated hardware that most people can't afford. At the same time, millions of GPUs sit idle in homes and offices. There is no dead-simple, incentivised way for GPU owners to share their hardware and for developers to access distributed inference — without depending on a single company, without paying cloud GPU prices, and without setting up complex infrastructure.

## Solution

A volunteer GPU network where anyone can share their GPU by running a single command and immediately start earning **USDT**. Nodes each load a shard of a large model; a tracker routes inference requests through the optimal chain of nodes whose shards collectively cover all layers. Developers access the network through an OpenAI-compatible API — a one-line change from any existing LLM integration. Clients pay in **USDT** (alpha: devnet mock-USDT; production: mainnet USDT). Node operators earn USDT payouts from the custodial treasury (ADR-0015); the TAI reward token (ADR-0002) remains deferred. Everything is auto-configured: GPU detection, shard download, wallet creation, and network registration happen automatically on first start.

## User Stories

> **Status (2026-07-13):** Stories below are the original product intent. **Shipped behavior** is in [Implementation Decisions](#implementation-decisions) and ADRs 0015–0018, 0023, 0025–0026. Superseded lines are marked inline.

### Node Operator

1. As a node operator, I want to install the node client with a single command (`pip install meshnet-node`), so that I can start contributing without reading documentation.
2. As a node operator, I want the node client to auto-detect my GPU model and VRAM on first start, so that I don't have to specify my hardware manually.
3. As a node operator, I want the node client to create a Solana wallet for me automatically on first start, so that I can start earning without prior crypto knowledge.
4. As a node operator, I want the tracker to assign me the most-needed shard for my hardware automatically, so that my contribution has the highest possible impact on the network.
5. As a node operator, I want my assigned shard to download automatically from HuggingFace on first start, so that I don't have to manually find or download model weights.
6. As a node operator, I want to seed my shard to other nodes via P2P once I have it, so that new nodes with the same shard assignment don't need to download from HuggingFace.
7. As a node operator, I want the node client to register with the tracker automatically and begin serving inference requests, so that I start earning as soon as setup is complete.
8. As a node operator, I want to see my current node score, shard assignment, and USDT earnings in the terminal, so that I can verify my node is contributing correctly.
9. As a node operator, I want to serve paid inference without upfront stake deposits, with my accrued USDT pending balance as fraud collateral and probation as the anti-sybil cost, so that onboarding stays frictionless. *(Supersedes stake-before-serving; ADR-0015/0018.)*
10. As a node operator, I want my first N jobs to run without earning (probationary period), so that the network can establish trust before paying me.
11. As a node operator, I want to be notified when my pending balance is forfeited due to a failed audit, so that I can investigate and fix the issue. *(Supersedes stake slash; ADR-0018 forfeiture.)*
12. As a node operator, I want to receive a strike and a warning before being banned, so that accidental failures don't immediately end my participation.
13. As a node operator, I want to be automatically reassigned to a different shard when the tracker determines another shard is more in demand, so that my hardware is always optimally used.
14. As a node operator, I want the node client to reconnect automatically if the tracker is temporarily unavailable, so that transient network issues don't stop me from earning.
15. As a node operator, I want the node client to fall back to P2P gossip for route discovery if the centralized tracker is down, so that inference serving continues during outages.
16. As a node operator, I want to run the node client on a CPU-only machine with smaller shards, so that I can contribute even without a dedicated GPU.

### Client Developer

17. As a client developer, I want to send `POST /v1/chat/completions` requests to the gateway in the same format as the OpenAI API, so that I can switch to the network with a one-line code change.
18. As a client developer, I want to authenticate with an API key funded by USDT, so that I never need to acquire or hold our native token. *(ADR-0015.)*
19. As a client developer, I want to top up my API key balance by sending USDT to the treasury Solana address, so that payment is simple and familiar. *(ADR-0015; wallet binding US-039/041.)*
20. As a client developer, I want to see a per-request cost estimate before sending a request, so that I can budget inference costs accurately.
21. As a client developer, I want to receive streaming responses (`text/event-stream`) in OpenAI-compatible format, so that I can build low-latency user experiences.
22. As a client developer, I want `GET /v1/models` to return the list of available model presets on the network, so that I know what I can request.
23. As a client developer, I want to receive a clear error response when no inference route is available for a requested model preset, so that I can handle degraded availability gracefully.
24. As a client developer, I want to use the `meshnet` Python SDK to access network-specific features (redundancy level, wallet top-up, node selection hints), so that I can optimise for my use case beyond basic inference.
25. As a client developer, I want the gateway to be compatible with LangChain, LlamaIndex, and Open WebUI out of the box, so that I can integrate with existing tooling immediately.
26. As a client developer, I want to request redundant execution (same request routed to multiple independent node chains) for high-stakes queries, so that I can trade cost for reliability.

### End User (via a client app)

27. As an end user, I want to buy USDT on an exchange and use it to pay for inference via Solana, so that I don't need deep crypto knowledge to use the service. *(Clients pay USDT; SOL is only for network fees if they self-custody.)*
28. As an end user, I want responses of equivalent quality to centralised providers, so that I don't have to trade quality for cost savings.
29. As an end user, I want low latency on first token, so that conversational applications feel responsive.

### Validator

30. As a validator, I want to automatically re-run a random sample (~5%) of completed inference requests on a reference node with TOPLOC activation verification, so that I can detect nodes returning fraudulent outputs. *(ADR-0018.)*
31. As a validator, I want the tracker to record forfeiture and strikes when an audit fails, so that penalties are applied consistently. *(Supersedes on-chain fraud proof in alpha; ADR-0018.)*
32. As a validator, I want economic incentive to run validation, so that fraud detection is not purely altruistic. *(Validator reward share deferred; forfeiture to protocol cut today.)*

### Network (tracker / system)

33. As the tracker, I want to score nodes continuously by throughput and latency so that I can select the fastest inference route for each request.
34. As the tracker, I want to rebalance shard assignments across nodes when demand for a model preset changes, so that the network always covers the most-requested models.
35. As the tracker, I want to instruct a node to download a new shard when no other node covers it, so that model preset coverage is maintained automatically.
36. As the tracker, I want to exclude banned wallets from route selection, so that fraudulent nodes cannot serve paid inference.
37. As the tracker, I want strike and ban state persisted in the registry and enforced on route selection, so that fraudulent wallets cannot serve paid inference. *(Supersedes on-chain-only stake/slash registry; ADR-0018; on-chain deferred per ADR-0007/0015.)*
38. As the network, I want new model presets to be addable by submitting a HuggingFace model ID and shard count, so that the set of available models can grow without code changes.

## Implementation Decisions

### Monorepo structure
The codebase is organized as a Python monorepo with the following top-level packages:
- `packages/node` — node client CLI (`meshnet-node`)
- `packages/gateway` — OpenAI-compatible HTTP gateway and route orchestration
- `packages/tracker` — centralized tracker service (node registry, scoring, route selection)
- `packages/sdk` — `meshnet` Python SDK wrapping gateway + wallet controls
- `packages/contracts` — Solana adapter boundary (custodial USDT treasury, local registry prototype)
- `packages/p2p` — P2P gossip layer and shard swarm seeding

### Inference engine (ADR-0001; native GGUF path ADR-0024)
PyTorch with a Petals-style shard pipeline remains the current production backend. A benchmark-gated llama.cpp/GGUF native path is planned in ADR-0024. Each node independently loads its assigned shard from local disk. At inference time, only activation tensors (~8 KB per layer boundary per token) travel between nodes — no model weights cross the network during serving.

### Inference route execution
The gateway receives a client request, asks the tracker for an inference route (ordered list of node endpoints covering all layers), opens a persistent TCP session to the first node in the route, streams activation tensors through each node in sequence, and returns the final logits as a streaming chat completion response.

### Node client startup sequence
1. Detect GPU/VRAM
2. Load or create Solana wallet
3. Query tracker for shard assignment given hardware profile
4. Download shard from HuggingFace (falling back to shard swarm if available)
5. Join shard swarm as a seeder
6. Register with tracker (wallet, hardware profile, shard, endpoint)
7. Begin accepting inference connections

### Payment flow (ADR-0015 supersedes ADR-0002 settlement mechanics)
Clients pre-fund an API key with USDT. The tracker meters each request against the off-chain ledger. Periodic settlement batches USDT payouts from the custodial treasury to node operators proportional to work units (default: every 24 h or when pending ≥ 5 USDT). Fraud penalties forfeit pending balance (ADR-0018); strike/ban state persists in the tracker registry. TAI reward accrual is deferred — see ADR-0025 for reserved-mint / off-chain phase B/C; ADR-0002 roadmap for public listing.

### Fraud detection (ADR-0018; historical ADR-0003)
Validators re-run ~5% of completed requests with TOPLOC activation verification. Caught cheaters forfeit pending balance and receive strikes; three strikes bans the wallet. Probation (first N unpaid jobs) remains the anti-sybil re-entry cost.

### Tracker architecture (ADR-0004)
Centralized tracker service (HTTP + WebSocket) for fast routing. Nodes gossip state via a lightweight P2P layer so the node client can discover routes during tracker outages. **Alpha:** strike/ban/forfeiture state lives in the tracker registry (ADR-0018); USDT settlement via custodial treasury (ADR-0015). On-chain programs deferred (ADR-0007).

### Shard distribution (ADR-0005)
Shards are identified by `(model_preset, shard_index)`. On assignment, the node downloads the shard layers from HuggingFace using `huggingface_hub`. Once downloaded, the node joins the P2P shard swarm and seeds to other nodes requesting the same shard. Popular shards propagate entirely via P2P; cold shards fall back to HuggingFace.

### Client API (ADR-0006)
The gateway exposes OpenAI-compatible endpoints (`/v1/chat/completions`, `/v1/models`, `/v1/completions`). The `meshnet` SDK wraps these and adds: `client.wallet.top_up()`, `client.request(redundancy=2)`, `client.models.available()`, per-request cost estimation.

## Testing Decisions

**What makes a good test:** test observable behavior at the highest seam possible — the `POST /v1/chat/completions` → streamed response boundary. Tests should not assert on internal state (which node was chosen, how tensors were split) but on what the client observes: a valid, coherent response arrives within a latency bound, and a payment record is written on-chain.

**Primary test seam:** spin up N in-process mock nodes each serving a known shard range, register them with a local tracker instance, send a real chat completion request through the gateway, and assert the response is a valid OpenAI-format streamed response whose content matches the expected model output for the given prompt.

**Per-component seams:**
- **Tracker**: given a set of registered nodes with known shard coverage and node scores, assert `select_route(model_preset)` returns an optimal ordered list of node endpoints.
- **Node shard serving**: given an activation tensor for the node's layer range, assert the output tensor shape and dtype are correct.
- **Fraud detection**: given a validator that re-runs a known-bad node response, assert strike/forfeiture state updates with correct attribution (ADR-0018; on-chain slash deferred).
- **Shard swarm**: given a node that has a shard, assert a second node with the same assignment downloads it via P2P rather than HuggingFace.
- **Payment settlement**: given a completed inference session with known compute attribution, assert USDT ledger balances change by the expected amounts after epoch settlement (ADR-0015).

## Out of Scope

- Desktop GUI app (v2)
- DHT as primary tracker mechanism (v2 — P2P gossip is a resilience fallback only in v1)
- zkML / zero-knowledge fraud proofs (research stage, not production-ready for large models)
- TEE attestation (excludes consumer GPUs, defeats viral growth goal)
- Distributed training / fine-tuning (post-v1)
- Mobile node apps (post-v1)
- Cross-chain payments beyond Solana L2 (post-v1)
- The existing `scripts/run_distributed_llama.py` (superseded by ADR-0001)

## Further Notes

- The `meshnet-node` CLI is the primary viral growth vector. Every friction point in the install/start sequence costs node operators. The startup sequence must complete without any manual configuration on a machine with a CUDA-capable GPU.
- The name "meshnet" is a working name. The actual package and token names are TBD.
- The Solana L2 chain selection (vs Base/Arbitrum) is not yet finalised — both are cheap, EVM-compatible fallbacks. The contracts package should abstract chain-specific details.
- The probationary period length (N free jobs) and forfeiture amounts are economic parameters that will need tuning once the network has real usage data. Hardcode sensible defaults; governance TBD (ADR-0018).