139 lines
14 KiB
Markdown
139 lines
14 KiB
Markdown
Status: done (base program US-001…US-035 complete; friends-test arc US-036…US-047 in `docs/prd.json`. Payment/settlement superseded by ADR-0015; fraud by ADR-0018.)
|
|
|
|
# Distributed Inference Network — PRD
|
|
|
|
## Problem Statement
|
|
|
|
Running large language models requires expensive dedicated hardware that most people can't afford. At the same time, millions of GPUs sit idle in homes and offices. There is no dead-simple, incentivised way for GPU owners to share their hardware and for developers to access distributed inference — without depending on a single company, without paying cloud GPU prices, and without setting up complex infrastructure.
|
|
|
|
## Solution
|
|
|
|
A volunteer GPU network where anyone can share their GPU by running a single command and immediately start earning tokens. Nodes each load a shard of a large model; a tracker routes inference requests through the optimal chain of nodes whose shards collectively cover all layers. Developers access the network through an OpenAI-compatible API — a one-line change from any existing LLM integration. Clients pay in **USDT** (alpha: devnet mock-USDT; production: mainnet USDT). Node operators earn USDT payouts from the custodial treasury (ADR-0015); the TAI reward token (ADR-0002) remains deferred. Everything is auto-configured: GPU detection, shard download, wallet creation, and network registration happen automatically on first start.
|
|
|
|
## User Stories
|
|
|
|
### Node Operator
|
|
|
|
1. As a node operator, I want to install the node client with a single command (`pip install meshnet-node`), so that I can start contributing without reading documentation.
|
|
2. As a node operator, I want the node client to auto-detect my GPU model and VRAM on first start, so that I don't have to specify my hardware manually.
|
|
3. As a node operator, I want the node client to create a Solana wallet for me automatically on first start, so that I can start earning without prior crypto knowledge.
|
|
4. As a node operator, I want the tracker to assign me the most-needed shard for my hardware automatically, so that my contribution has the highest possible impact on the network.
|
|
5. As a node operator, I want my assigned shard to download automatically from HuggingFace on first start, so that I don't have to manually find or download model weights.
|
|
6. As a node operator, I want to seed my shard to other nodes via P2P once I have it, so that new nodes with the same shard assignment don't need to download from HuggingFace.
|
|
7. As a node operator, I want the node client to register with the tracker automatically and begin serving inference requests, so that I start earning as soon as setup is complete.
|
|
8. As a node operator, I want to see my current node score, shard assignment, and token earnings in the terminal, so that I can verify my node is contributing correctly.
|
|
9. As a node operator, I want to stake tokens before serving paid inference, so that I have skin in the game and the network can trust my outputs.
|
|
10. As a node operator, I want my first N jobs to run without earning (probationary period), so that the network can establish trust before paying me.
|
|
11. As a node operator, I want to be notified immediately if my stake is slashed due to a fraud detection event, so that I can investigate and fix the issue.
|
|
12. As a node operator, I want to receive a strike and a warning before being banned, so that accidental failures don't immediately end my participation.
|
|
13. As a node operator, I want to be automatically reassigned to a different shard when the tracker determines another shard is more in demand, so that my hardware is always optimally used.
|
|
14. As a node operator, I want the node client to reconnect automatically if the tracker is temporarily unavailable, so that transient network issues don't stop me from earning.
|
|
15. As a node operator, I want the node client to fall back to P2P gossip for route discovery if the centralized tracker is down, so that inference serving continues during outages.
|
|
16. As a node operator, I want to run the node client on a CPU-only machine with smaller shards, so that I can contribute even without a dedicated GPU.
|
|
|
|
### Client Developer
|
|
|
|
17. As a client developer, I want to send `POST /v1/chat/completions` requests to the gateway in the same format as the OpenAI API, so that I can switch to the network with a one-line code change.
|
|
18. As a client developer, I want to authenticate with an API key funded by SOL or USDC, so that I never need to acquire or hold our native token.
|
|
19. As a client developer, I want to top up my API key balance by sending SOL or USDC to a Solana address, so that payment is simple and familiar.
|
|
20. As a client developer, I want to see a per-request cost estimate before sending a request, so that I can budget inference costs accurately.
|
|
21. As a client developer, I want to receive streaming responses (`text/event-stream`) in OpenAI-compatible format, so that I can build low-latency user experiences.
|
|
22. As a client developer, I want `GET /v1/models` to return the list of available model presets on the network, so that I know what I can request.
|
|
23. As a client developer, I want to receive a clear error response when no inference route is available for a requested model preset, so that I can handle degraded availability gracefully.
|
|
24. As a client developer, I want to use the `meshnet` Python SDK to access network-specific features (redundancy level, wallet top-up, node selection hints), so that I can optimise for my use case beyond basic inference.
|
|
25. As a client developer, I want the gateway to be compatible with LangChain, LlamaIndex, and Open WebUI out of the box, so that I can integrate with existing tooling immediately.
|
|
26. As a client developer, I want to request redundant execution (same request routed to multiple independent node chains) for high-stakes queries, so that I can trade cost for reliability.
|
|
|
|
### End User (via a client app)
|
|
|
|
27. As an end user, I want to buy SOL on any exchange and use it to pay for inference, so that I don't need to understand blockchain technology to use the service.
|
|
28. As an end user, I want responses of equivalent quality to centralised providers, so that I don't have to trade quality for cost savings.
|
|
29. As an end user, I want low latency on first token, so that conversational applications feel responsive.
|
|
|
|
### Validator
|
|
|
|
30. As a validator, I want to automatically re-run a random sample (~5%) of completed inference requests on a reference node, so that I can detect nodes returning fraudulent outputs.
|
|
31. As a validator, I want to submit a fraud proof on-chain when a node's output diverges beyond tolerance, so that the slash event is recorded trustlessly.
|
|
32. As a validator, I want to earn a reward for each successful fraud detection, so that there is an economic incentive to run validation.
|
|
|
|
### Network (tracker / system)
|
|
|
|
33. As the tracker, I want to score nodes continuously by throughput and latency so that I can select the fastest inference route for each request.
|
|
34. As the tracker, I want to rebalance shard assignments across nodes when demand for a model preset changes, so that the network always covers the most-requested models.
|
|
35. As the tracker, I want to instruct a node to download a new shard when no other node covers it, so that model preset coverage is maintained automatically.
|
|
36. As the tracker, I want to exclude banned wallets from route selection, so that fraudulent nodes cannot serve paid inference.
|
|
37. As the tracker, I want to read stake, slash, strike, and ban state exclusively from Solana smart contracts, so that I cannot manipulate payouts even with full control of the routing layer.
|
|
38. As the network, I want new model presets to be addable by submitting a HuggingFace model ID and shard count, so that the set of available models can grow without code changes.
|
|
|
|
## Implementation Decisions
|
|
|
|
### Monorepo structure
|
|
The codebase is organized as a Python monorepo with the following top-level packages:
|
|
- `packages/node` — node client CLI (`meshnet-node`)
|
|
- `packages/gateway` — OpenAI-compatible HTTP gateway and route orchestration
|
|
- `packages/tracker` — centralized tracker service (node registry, scoring, route selection)
|
|
- `packages/sdk` — `meshnet` Python SDK wrapping gateway + wallet controls
|
|
- `packages/contracts` — Solana L2 smart contracts (stake, slash, strike, ban, settlement)
|
|
- `packages/p2p` — P2P gossip layer and shard swarm seeding
|
|
|
|
### Inference engine (ADR-0001; native GGUF path ADR-0024)
|
|
PyTorch with a Petals-style shard pipeline remains the current production backend. A benchmark-gated llama.cpp/GGUF native path is planned in ADR-0024. Each node independently loads its assigned shard from local disk. At inference time, only activation tensors (~8 KB per layer boundary per token) travel between nodes — no model weights cross the network during serving.
|
|
|
|
### Inference route execution
|
|
The gateway receives a client request, asks the tracker for an inference route (ordered list of node endpoints covering all layers), opens a persistent TCP session to the first node in the route, streams activation tensors through each node in sequence, and returns the final logits as a streaming chat completion response.
|
|
|
|
### Node client startup sequence
|
|
1. Detect GPU/VRAM
|
|
2. Load or create Solana wallet
|
|
3. Query tracker for shard assignment given hardware profile
|
|
4. Download shard from HuggingFace (falling back to shard swarm if available)
|
|
5. Join shard swarm as a seeder
|
|
6. Register with tracker (wallet, hardware profile, shard, endpoint)
|
|
7. Begin accepting inference connections
|
|
|
|
### Payment flow (ADR-0015 supersedes ADR-0002 settlement mechanics)
|
|
Clients pre-fund an API key with USDT. The tracker meters each request against the off-chain ledger. Periodic settlement batches USDT payouts from the custodial treasury to node operators proportional to work units (default: every 24 h or when pending ≥ 5 USDT). Fraud penalties forfeit pending balance (ADR-0018); strike/ban state persists in the tracker registry. TAI reward accrual is deferred — see ADR-0025 for reserved-mint / off-chain phase B/C; ADR-0002 roadmap for public listing.
|
|
|
|
### Fraud detection (ADR-0018; historical ADR-0003)
|
|
Validators re-run ~5% of completed requests with TOPLOC activation verification. Caught cheaters forfeit pending balance and receive strikes; three strikes bans the wallet. Probation (first N unpaid jobs) remains the anti-sybil re-entry cost.
|
|
|
|
### Tracker architecture (ADR-0004)
|
|
Centralized tracker service (HTTP + WebSocket) for fast routing. Nodes gossip state via a lightweight P2P layer so the node client can discover routes during tracker outages. Solana is the authoritative source of truth for all incentive-relevant state.
|
|
|
|
### Shard distribution (ADR-0005)
|
|
Shards are identified by `(model_preset, shard_index)`. On assignment, the node downloads the shard layers from HuggingFace using `huggingface_hub`. Once downloaded, the node joins the P2P shard swarm and seeds to other nodes requesting the same shard. Popular shards propagate entirely via P2P; cold shards fall back to HuggingFace.
|
|
|
|
### Client API (ADR-0006)
|
|
The gateway exposes OpenAI-compatible endpoints (`/v1/chat/completions`, `/v1/models`, `/v1/completions`). The `meshnet` SDK wraps these and adds: `client.wallet.top_up()`, `client.request(redundancy=2)`, `client.models.available()`, per-request cost estimation.
|
|
|
|
## Testing Decisions
|
|
|
|
**What makes a good test:** test observable behavior at the highest seam possible — the `POST /v1/chat/completions` → streamed response boundary. Tests should not assert on internal state (which node was chosen, how tensors were split) but on what the client observes: a valid, coherent response arrives within a latency bound, and a payment record is written on-chain.
|
|
|
|
**Primary test seam:** spin up N in-process mock nodes each serving a known shard range, register them with a local tracker instance, send a real chat completion request through the gateway, and assert the response is a valid OpenAI-format streamed response whose content matches the expected model output for the given prompt.
|
|
|
|
**Per-component seams:**
|
|
- **Tracker**: given a set of registered nodes with known shard coverage and node scores, assert `select_route(model_preset)` returns an optimal ordered list of node endpoints.
|
|
- **Node shard serving**: given an activation tensor for the node's layer range, assert the output tensor shape and dtype are correct.
|
|
- **Fraud detection**: given a validator that re-runs a known-bad node response, assert a slash transaction is submitted on-chain with correct attribution.
|
|
- **Shard swarm**: given a node that has a shard, assert a second node with the same assignment downloads it via P2P rather than HuggingFace.
|
|
- **Payment settlement**: given a completed inference session with known compute attribution, assert token balances change by the expected amounts after epoch settlement.
|
|
|
|
## Out of Scope
|
|
|
|
- Desktop GUI app (v2)
|
|
- DHT as primary tracker mechanism (v2 — P2P gossip is a resilience fallback only in v1)
|
|
- zkML / zero-knowledge fraud proofs (research stage, not production-ready for large models)
|
|
- TEE attestation (excludes consumer GPUs, defeats viral growth goal)
|
|
- Distributed training / fine-tuning (post-v1)
|
|
- Mobile node apps (post-v1)
|
|
- Cross-chain payments beyond Solana L2 (post-v1)
|
|
- The existing `scripts/run_distributed_llama.py` (superseded by ADR-0001)
|
|
|
|
## Further Notes
|
|
|
|
- The `meshnet-node` CLI is the primary viral growth vector. Every friction point in the install/start sequence costs node operators. The startup sequence must complete without any manual configuration on a machine with a CUDA-capable GPU.
|
|
- The name "meshnet" is a working name. The actual package and token names are TBD.
|
|
- The Solana L2 chain selection (vs Base/Arbitrum) is not yet finalised — both are cheap, EVM-compatible fallbacks. The contracts package should abstract chain-specific details.
|
|
- The probationary period length (N free jobs) and slash amounts are economic parameters that will need tuning once the network has real usage data. Hardcode sensible defaults; make them on-chain governable.
|