popov/neuron-tai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

tasks

Browse Source
This commit is contained in:
Dobromir Popov
2026-06-29 00:10:21 +03:00
parent 0f24a1d4f9
commit 2f1f9717be
16 changed files with 478 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

138
.scratch/distributed-inference-network/PRD.md Normal file
View File

@@ -0,0 +1,138 @@
Status: ready-for-agent
# Distributed Inference Network — PRD
## Problem Statement
Running large language models requires expensive dedicated hardware that most people can't afford. At the same time, millions of GPUs sit idle in homes and offices. There is no dead-simple, incentivised way for GPU owners to share their hardware and for developers to access distributed inference — without depending on a single company, without paying cloud GPU prices, and without setting up complex infrastructure.
## Solution
A volunteer GPU network where anyone can share their GPU by running a single command and immediately start earning tokens. Nodes each load a shard of a large model; a tracker routes inference requests through the optimal chain of nodes whose shards collectively cover all layers. Developers access the network through an OpenAI-compatible API — a one-line change from any existing LLM integration. Clients pay in SOL or USDC; node operators earn our native token. Everything is auto-configured: GPU detection, shard download, wallet creation, and network registration happen automatically on first start.
## User Stories
### Node Operator
1. As a node operator, I want to install the node client with a single command (`pip install meshnet-node`), so that I can start contributing without reading documentation.
2. As a node operator, I want the node client to auto-detect my GPU model and VRAM on first start, so that I don't have to specify my hardware manually.
3. As a node operator, I want the node client to create a Solana wallet for me automatically on first start, so that I can start earning without prior crypto knowledge.
4. As a node operator, I want the tracker to assign me the most-needed shard for my hardware automatically, so that my contribution has the highest possible impact on the network.
5. As a node operator, I want my assigned shard to download automatically from HuggingFace on first start, so that I don't have to manually find or download model weights.
6. As a node operator, I want to seed my shard to other nodes via P2P once I have it, so that new nodes with the same shard assignment don't need to download from HuggingFace.
7. As a node operator, I want the node client to register with the tracker automatically and begin serving inference requests, so that I start earning as soon as setup is complete.
8. As a node operator, I want to see my current node score, shard assignment, and token earnings in the terminal, so that I can verify my node is contributing correctly.
9. As a node operator, I want to stake tokens before serving paid inference, so that I have skin in the game and the network can trust my outputs.
10. As a node operator, I want my first N jobs to run without earning (probationary period), so that the network can establish trust before paying me.
11. As a node operator, I want to be notified immediately if my stake is slashed due to a fraud detection event, so that I can investigate and fix the issue.
12. As a node operator, I want to receive a strike and a warning before being banned, so that accidental failures don't immediately end my participation.
13. As a node operator, I want to be automatically reassigned to a different shard when the tracker determines another shard is more in demand, so that my hardware is always optimally used.
14. As a node operator, I want the node client to reconnect automatically if the tracker is temporarily unavailable, so that transient network issues don't stop me from earning.
15. As a node operator, I want the node client to fall back to P2P gossip for route discovery if the centralized tracker is down, so that inference serving continues during outages.
16. As a node operator, I want to run the node client on a CPU-only machine with smaller shards, so that I can contribute even without a dedicated GPU.
### Client Developer
17. As a client developer, I want to send `POST /v1/chat/completions` requests to the gateway in the same format as the OpenAI API, so that I can switch to the network with a one-line code change.
18. As a client developer, I want to authenticate with an API key funded by SOL or USDC, so that I never need to acquire or hold our native token.
19. As a client developer, I want to top up my API key balance by sending SOL or USDC to a Solana address, so that payment is simple and familiar.
20. As a client developer, I want to see a per-request cost estimate before sending a request, so that I can budget inference costs accurately.
21. As a client developer, I want to receive streaming responses (`text/event-stream`) in OpenAI-compatible format, so that I can build low-latency user experiences.
22. As a client developer, I want `GET /v1/models` to return the list of available model presets on the network, so that I know what I can request.
23. As a client developer, I want to receive a clear error response when no inference route is available for a requested model preset, so that I can handle degraded availability gracefully.
24. As a client developer, I want to use the `meshnet` Python SDK to access network-specific features (redundancy level, wallet top-up, node selection hints), so that I can optimise for my use case beyond basic inference.
25. As a client developer, I want the gateway to be compatible with LangChain, LlamaIndex, and Open WebUI out of the box, so that I can integrate with existing tooling immediately.
26. As a client developer, I want to request redundant execution (same request routed to multiple independent node chains) for high-stakes queries, so that I can trade cost for reliability.
### End User (via a client app)
27. As an end user, I want to buy SOL on any exchange and use it to pay for inference, so that I don't need to understand blockchain technology to use the service.
28. As an end user, I want responses of equivalent quality to centralised providers, so that I don't have to trade quality for cost savings.
29. As an end user, I want low latency on first token, so that conversational applications feel responsive.
### Validator
30. As a validator, I want to automatically re-run a random sample (~5%) of completed inference requests on a reference node, so that I can detect nodes returning fraudulent outputs.
31. As a validator, I want to submit a fraud proof on-chain when a node's output diverges beyond tolerance, so that the slash event is recorded trustlessly.
32. As a validator, I want to earn a reward for each successful fraud detection, so that there is an economic incentive to run validation.
### Network (tracker / system)
33. As the tracker, I want to score nodes continuously by throughput and latency so that I can select the fastest inference route for each request.
34. As the tracker, I want to rebalance shard assignments across nodes when demand for a model preset changes, so that the network always covers the most-requested models.
35. As the tracker, I want to instruct a node to download a new shard when no other node covers it, so that model preset coverage is maintained automatically.
36. As the tracker, I want to exclude banned wallets from route selection, so that fraudulent nodes cannot serve paid inference.
37. As the tracker, I want to read stake, slash, strike, and ban state exclusively from Solana smart contracts, so that I cannot manipulate payouts even with full control of the routing layer.
38. As the network, I want new model presets to be addable by submitting a HuggingFace model ID and shard count, so that the set of available models can grow without code changes.
## Implementation Decisions
### Monorepo structure
The codebase is organized as a Python monorepo with the following top-level packages:
- `packages/node` — node client CLI (`meshnet-node`)
- `packages/gateway` — OpenAI-compatible HTTP gateway and route orchestration
- `packages/tracker` — centralized tracker service (node registry, scoring, route selection)
- `packages/sdk` — `meshnet` Python SDK wrapping gateway + wallet controls
- `packages/contracts` — Solana L2 smart contracts (stake, slash, strike, ban, settlement)
- `packages/p2p` — P2P gossip layer and shard swarm seeding
### Inference engine (ADR-0001)
PyTorch with a Petals-style shard pipeline. Each node independently loads its assigned shard from local disk. At inference time, only activation tensors (~8 KB per layer boundary per token) travel between nodes — no model weights cross the network during serving.
### Inference route execution
The gateway receives a client request, asks the tracker for an inference route (ordered list of node endpoints covering all layers), opens a persistent TCP session to the first node in the route, streams activation tensors through each node in sequence, and returns the final logits as a streaming chat completion response.
### Node client startup sequence
1. Detect GPU/VRAM
2. Load or create Solana wallet
3. Query tracker for shard assignment given hardware profile
4. Download shard from HuggingFace (falling back to shard swarm if available)
5. Join shard swarm as a seeder
6. Register with tracker (wallet, hardware profile, shard, endpoint)
7. Begin accepting inference connections
### Payment flow
Clients pre-fund an API key with SOL/USDC. The gateway records per-request compute attribution. A settlement transaction runs on Solana L2 at the end of each epoch: client balance is debited, node operators receive our native token proportional to layers served, validators receive a reward share. Solana contracts are the authoritative source for all stake, slash, strike, and ban state (ADR-0002).
### Fraud detection (ADR-0003)
Validators re-run ~5% of completed requests. If a node's output diverges beyond floating-point tolerance from the reference, the validator submits a slash transaction on-chain. Strike count increments. At the configured strike threshold, the wallet is banned on-chain. New wallets complete N unpaid jobs before earning begins.
### Tracker architecture (ADR-0004)
Centralized tracker service (HTTP + WebSocket) for fast routing. Nodes gossip state via a lightweight P2P layer so the node client can discover routes during tracker outages. Solana is the authoritative source of truth for all incentive-relevant state.
### Shard distribution (ADR-0005)
Shards are identified by `(model_preset, shard_index)`. On assignment, the node downloads the shard layers from HuggingFace using `huggingface_hub`. Once downloaded, the node joins the P2P shard swarm and seeds to other nodes requesting the same shard. Popular shards propagate entirely via P2P; cold shards fall back to HuggingFace.
### Client API (ADR-0006)
The gateway exposes OpenAI-compatible endpoints (`/v1/chat/completions`, `/v1/models`, `/v1/completions`). The `meshnet` SDK wraps these and adds: `client.wallet.top_up()`, `client.request(redundancy=2)`, `client.models.available()`, per-request cost estimation.
## Testing Decisions
**What makes a good test:** test observable behavior at the highest seam possible — the `POST /v1/chat/completions` → streamed response boundary. Tests should not assert on internal state (which node was chosen, how tensors were split) but on what the client observes: a valid, coherent response arrives within a latency bound, and a payment record is written on-chain.
**Primary test seam:** spin up N in-process mock nodes each serving a known shard range, register them with a local tracker instance, send a real chat completion request through the gateway, and assert the response is a valid OpenAI-format streamed response whose content matches the expected model output for the given prompt.
**Per-component seams:**
- **Tracker**: given a set of registered nodes with known shard coverage and node scores, assert `select_route(model_preset)` returns an optimal ordered list of node endpoints.
- **Node shard serving**: given an activation tensor for the node's layer range, assert the output tensor shape and dtype are correct.
- **Fraud detection**: given a validator that re-runs a known-bad node response, assert a slash transaction is submitted on-chain with correct attribution.
- **Shard swarm**: given a node that has a shard, assert a second node with the same assignment downloads it via P2P rather than HuggingFace.
- **Payment settlement**: given a completed inference session with known compute attribution, assert token balances change by the expected amounts after epoch settlement.
## Out of Scope
- Desktop GUI app (v2)
- DHT as primary tracker mechanism (v2 — P2P gossip is a resilience fallback only in v1)
- zkML / zero-knowledge fraud proofs (research stage, not production-ready for large models)
- TEE attestation (excludes consumer GPUs, defeats viral growth goal)
- Distributed training / fine-tuning (post-v1)
- Mobile node apps (post-v1)
- Cross-chain payments beyond Solana L2 (post-v1)
- The existing `scripts/run_distributed_llama.py` (superseded by ADR-0001)
## Further Notes
- The `meshnet-node` CLI is the primary viral growth vector. Every friction point in the install/start sequence costs node operators. The startup sequence must complete without any manual configuration on a machine with a CUDA-capable GPU.
- The name "meshnet" is a working name. The actual package and token names are TBD.
- The Solana L2 chain selection (vs Base/Arbitrum) is not yet finalised — both are cheap, EVM-compatible fallbacks. The contracts package should abstract chain-specific details.
- The probationary period length (N free jobs) and slash amounts are economic parameters that will need tuning once the network has real usage data. Hardcode sensible defaults; make them on-chain governable.