diff --git a/.scratch/distributed-inference-network/PRD.md b/.scratch/distributed-inference-network/PRD.md new file mode 100644 index 0000000..4275333 --- /dev/null +++ b/.scratch/distributed-inference-network/PRD.md @@ -0,0 +1,138 @@ +Status: ready-for-agent + +# Distributed Inference Network — PRD + +## Problem Statement + +Running large language models requires expensive dedicated hardware that most people can't afford. At the same time, millions of GPUs sit idle in homes and offices. There is no dead-simple, incentivised way for GPU owners to share their hardware and for developers to access distributed inference — without depending on a single company, without paying cloud GPU prices, and without setting up complex infrastructure. + +## Solution + +A volunteer GPU network where anyone can share their GPU by running a single command and immediately start earning tokens. Nodes each load a shard of a large model; a tracker routes inference requests through the optimal chain of nodes whose shards collectively cover all layers. Developers access the network through an OpenAI-compatible API — a one-line change from any existing LLM integration. Clients pay in SOL or USDC; node operators earn our native token. Everything is auto-configured: GPU detection, shard download, wallet creation, and network registration happen automatically on first start. + +## User Stories + +### Node Operator + +1. As a node operator, I want to install the node client with a single command (`pip install meshnet-node`), so that I can start contributing without reading documentation. +2. As a node operator, I want the node client to auto-detect my GPU model and VRAM on first start, so that I don't have to specify my hardware manually. +3. As a node operator, I want the node client to create a Solana wallet for me automatically on first start, so that I can start earning without prior crypto knowledge. +4. As a node operator, I want the tracker to assign me the most-needed shard for my hardware automatically, so that my contribution has the highest possible impact on the network. +5. As a node operator, I want my assigned shard to download automatically from HuggingFace on first start, so that I don't have to manually find or download model weights. +6. As a node operator, I want to seed my shard to other nodes via P2P once I have it, so that new nodes with the same shard assignment don't need to download from HuggingFace. +7. As a node operator, I want the node client to register with the tracker automatically and begin serving inference requests, so that I start earning as soon as setup is complete. +8. As a node operator, I want to see my current node score, shard assignment, and token earnings in the terminal, so that I can verify my node is contributing correctly. +9. As a node operator, I want to stake tokens before serving paid inference, so that I have skin in the game and the network can trust my outputs. +10. As a node operator, I want my first N jobs to run without earning (probationary period), so that the network can establish trust before paying me. +11. As a node operator, I want to be notified immediately if my stake is slashed due to a fraud detection event, so that I can investigate and fix the issue. +12. As a node operator, I want to receive a strike and a warning before being banned, so that accidental failures don't immediately end my participation. +13. As a node operator, I want to be automatically reassigned to a different shard when the tracker determines another shard is more in demand, so that my hardware is always optimally used. +14. As a node operator, I want the node client to reconnect automatically if the tracker is temporarily unavailable, so that transient network issues don't stop me from earning. +15. As a node operator, I want the node client to fall back to P2P gossip for route discovery if the centralized tracker is down, so that inference serving continues during outages. +16. As a node operator, I want to run the node client on a CPU-only machine with smaller shards, so that I can contribute even without a dedicated GPU. + +### Client Developer + +17. As a client developer, I want to send `POST /v1/chat/completions` requests to the gateway in the same format as the OpenAI API, so that I can switch to the network with a one-line code change. +18. As a client developer, I want to authenticate with an API key funded by SOL or USDC, so that I never need to acquire or hold our native token. +19. As a client developer, I want to top up my API key balance by sending SOL or USDC to a Solana address, so that payment is simple and familiar. +20. As a client developer, I want to see a per-request cost estimate before sending a request, so that I can budget inference costs accurately. +21. As a client developer, I want to receive streaming responses (`text/event-stream`) in OpenAI-compatible format, so that I can build low-latency user experiences. +22. As a client developer, I want `GET /v1/models` to return the list of available model presets on the network, so that I know what I can request. +23. As a client developer, I want to receive a clear error response when no inference route is available for a requested model preset, so that I can handle degraded availability gracefully. +24. As a client developer, I want to use the `meshnet` Python SDK to access network-specific features (redundancy level, wallet top-up, node selection hints), so that I can optimise for my use case beyond basic inference. +25. As a client developer, I want the gateway to be compatible with LangChain, LlamaIndex, and Open WebUI out of the box, so that I can integrate with existing tooling immediately. +26. As a client developer, I want to request redundant execution (same request routed to multiple independent node chains) for high-stakes queries, so that I can trade cost for reliability. + +### End User (via a client app) + +27. As an end user, I want to buy SOL on any exchange and use it to pay for inference, so that I don't need to understand blockchain technology to use the service. +28. As an end user, I want responses of equivalent quality to centralised providers, so that I don't have to trade quality for cost savings. +29. As an end user, I want low latency on first token, so that conversational applications feel responsive. + +### Validator + +30. As a validator, I want to automatically re-run a random sample (~5%) of completed inference requests on a reference node, so that I can detect nodes returning fraudulent outputs. +31. As a validator, I want to submit a fraud proof on-chain when a node's output diverges beyond tolerance, so that the slash event is recorded trustlessly. +32. As a validator, I want to earn a reward for each successful fraud detection, so that there is an economic incentive to run validation. + +### Network (tracker / system) + +33. As the tracker, I want to score nodes continuously by throughput and latency so that I can select the fastest inference route for each request. +34. As the tracker, I want to rebalance shard assignments across nodes when demand for a model preset changes, so that the network always covers the most-requested models. +35. As the tracker, I want to instruct a node to download a new shard when no other node covers it, so that model preset coverage is maintained automatically. +36. As the tracker, I want to exclude banned wallets from route selection, so that fraudulent nodes cannot serve paid inference. +37. As the tracker, I want to read stake, slash, strike, and ban state exclusively from Solana smart contracts, so that I cannot manipulate payouts even with full control of the routing layer. +38. As the network, I want new model presets to be addable by submitting a HuggingFace model ID and shard count, so that the set of available models can grow without code changes. + +## Implementation Decisions + +### Monorepo structure +The codebase is organized as a Python monorepo with the following top-level packages: +- `packages/node` — node client CLI (`meshnet-node`) +- `packages/gateway` — OpenAI-compatible HTTP gateway and route orchestration +- `packages/tracker` — centralized tracker service (node registry, scoring, route selection) +- `packages/sdk` — `meshnet` Python SDK wrapping gateway + wallet controls +- `packages/contracts` — Solana L2 smart contracts (stake, slash, strike, ban, settlement) +- `packages/p2p` — P2P gossip layer and shard swarm seeding + +### Inference engine (ADR-0001) +PyTorch with a Petals-style shard pipeline. Each node independently loads its assigned shard from local disk. At inference time, only activation tensors (~8 KB per layer boundary per token) travel between nodes — no model weights cross the network during serving. + +### Inference route execution +The gateway receives a client request, asks the tracker for an inference route (ordered list of node endpoints covering all layers), opens a persistent TCP session to the first node in the route, streams activation tensors through each node in sequence, and returns the final logits as a streaming chat completion response. + +### Node client startup sequence +1. Detect GPU/VRAM +2. Load or create Solana wallet +3. Query tracker for shard assignment given hardware profile +4. Download shard from HuggingFace (falling back to shard swarm if available) +5. Join shard swarm as a seeder +6. Register with tracker (wallet, hardware profile, shard, endpoint) +7. Begin accepting inference connections + +### Payment flow +Clients pre-fund an API key with SOL/USDC. The gateway records per-request compute attribution. A settlement transaction runs on Solana L2 at the end of each epoch: client balance is debited, node operators receive our native token proportional to layers served, validators receive a reward share. Solana contracts are the authoritative source for all stake, slash, strike, and ban state (ADR-0002). + +### Fraud detection (ADR-0003) +Validators re-run ~5% of completed requests. If a node's output diverges beyond floating-point tolerance from the reference, the validator submits a slash transaction on-chain. Strike count increments. At the configured strike threshold, the wallet is banned on-chain. New wallets complete N unpaid jobs before earning begins. + +### Tracker architecture (ADR-0004) +Centralized tracker service (HTTP + WebSocket) for fast routing. Nodes gossip state via a lightweight P2P layer so the node client can discover routes during tracker outages. Solana is the authoritative source of truth for all incentive-relevant state. + +### Shard distribution (ADR-0005) +Shards are identified by `(model_preset, shard_index)`. On assignment, the node downloads the shard layers from HuggingFace using `huggingface_hub`. Once downloaded, the node joins the P2P shard swarm and seeds to other nodes requesting the same shard. Popular shards propagate entirely via P2P; cold shards fall back to HuggingFace. + +### Client API (ADR-0006) +The gateway exposes OpenAI-compatible endpoints (`/v1/chat/completions`, `/v1/models`, `/v1/completions`). The `meshnet` SDK wraps these and adds: `client.wallet.top_up()`, `client.request(redundancy=2)`, `client.models.available()`, per-request cost estimation. + +## Testing Decisions + +**What makes a good test:** test observable behavior at the highest seam possible — the `POST /v1/chat/completions` → streamed response boundary. Tests should not assert on internal state (which node was chosen, how tensors were split) but on what the client observes: a valid, coherent response arrives within a latency bound, and a payment record is written on-chain. + +**Primary test seam:** spin up N in-process mock nodes each serving a known shard range, register them with a local tracker instance, send a real chat completion request through the gateway, and assert the response is a valid OpenAI-format streamed response whose content matches the expected model output for the given prompt. + +**Per-component seams:** +- **Tracker**: given a set of registered nodes with known shard coverage and node scores, assert `select_route(model_preset)` returns an optimal ordered list of node endpoints. +- **Node shard serving**: given an activation tensor for the node's layer range, assert the output tensor shape and dtype are correct. +- **Fraud detection**: given a validator that re-runs a known-bad node response, assert a slash transaction is submitted on-chain with correct attribution. +- **Shard swarm**: given a node that has a shard, assert a second node with the same assignment downloads it via P2P rather than HuggingFace. +- **Payment settlement**: given a completed inference session with known compute attribution, assert token balances change by the expected amounts after epoch settlement. + +## Out of Scope + +- Desktop GUI app (v2) +- DHT as primary tracker mechanism (v2 — P2P gossip is a resilience fallback only in v1) +- zkML / zero-knowledge fraud proofs (research stage, not production-ready for large models) +- TEE attestation (excludes consumer GPUs, defeats viral growth goal) +- Distributed training / fine-tuning (post-v1) +- Mobile node apps (post-v1) +- Cross-chain payments beyond Solana L2 (post-v1) +- The existing `scripts/run_distributed_llama.py` (superseded by ADR-0001) + +## Further Notes + +- The `meshnet-node` CLI is the primary viral growth vector. Every friction point in the install/start sequence costs node operators. The startup sequence must complete without any manual configuration on a machine with a CUDA-capable GPU. +- The name "meshnet" is a working name. The actual package and token names are TBD. +- The Solana L2 chain selection (vs Base/Arbitrum) is not yet finalised — both are cheap, EVM-compatible fallbacks. The contracts package should abstract chain-specific details. +- The probationary period length (N free jobs) and slash amounts are economic parameters that will need tuning once the network has real usage data. Hardcode sensible defaults; make them on-chain governable. diff --git a/.scratch/distributed-inference-network/issues/01-monorepo-scaffold.md b/.scratch/distributed-inference-network/issues/01-monorepo-scaffold.md new file mode 100644 index 0000000..83e8047 --- /dev/null +++ b/.scratch/distributed-inference-network/issues/01-monorepo-scaffold.md @@ -0,0 +1,29 @@ +Status: ready-for-agent + +# 01 — Monorepo scaffold + single-node smoke test + +## What to build + +Stand up the monorepo package layout and prove that a client HTTP request can travel end-to-end through the stack — gateway → one node serving all layers → valid response — before any real model or distributed logic is wired in. + +The six top-level packages should exist with minimal stubs: +- `packages/node` — node client CLI (`meshnet-node`) +- `packages/gateway` — HTTP gateway + route orchestration +- `packages/tracker` — node registry and route selection +- `packages/sdk` — `meshnet` Python SDK +- `packages/contracts` — Solana smart contract wrappers +- `packages/p2p` — gossip and shard swarm + +The smoke test uses a tiny in-process stub model (not a real LLM) so the test is fast and has no external dependencies. The node runs in the same process as the test. The gateway is a real HTTP server. The client sends a real `POST /v1/chat/completions` request and receives a valid OpenAI-format response. + +## Acceptance criteria + +- [ ] `pip install -e packages/node packages/gateway packages/tracker` works from repo root +- [ ] `meshnet-node`, `meshnet-gateway`, and `meshnet-tracker` CLI entry points exist +- [ ] A single integration test starts a gateway and one stub node in-process, sends a `POST /v1/chat/completions` request, and asserts a valid OpenAI-format response is returned +- [ ] The test passes with `pytest` from repo root with no external services running +- [ ] All six package directories exist with `pyproject.toml` and an importable top-level module + +## Blocked by + +None — can start immediately. diff --git a/.scratch/distributed-inference-network/issues/02-two-node-shard-pipeline.md b/.scratch/distributed-inference-network/issues/02-two-node-shard-pipeline.md new file mode 100644 index 0000000..943d0d2 --- /dev/null +++ b/.scratch/distributed-inference-network/issues/02-two-node-shard-pipeline.md @@ -0,0 +1,23 @@ +Status: ready-for-agent + +# 02 — Two-node shard pipeline + +## What to build + +Extend the single-node smoke test so that inference is routed through exactly two nodes, each serving half of a stub model's layers. Activation tensors must travel from the gateway to node A, then from node A to node B, and the final output returns to the gateway. This proves the distributed shard pipeline works end-to-end before real models or a real tracker are introduced. + +The two nodes run in the same process as the test (no real network required). The gateway hardcodes the two-node route for now — dynamic route selection comes in issue 03. The activation tensor protocol (shape, dtype, serialisation format) must be established here, as all later issues depend on it. + +Use the domain vocabulary from `CONTEXT.md`: the ordered list of nodes is an **inference route**; each node's layer range is a **shard**; the tensors passed between nodes are activations (not "data" or "chunks"). + +## Acceptance criteria + +- [ ] The integration test spins up two stub nodes, each configured with a non-overlapping shard range +- [ ] A `POST /v1/chat/completions` request results in activation tensors flowing node-A → node-B (verifiable via test assertions or logs) +- [ ] The gateway assembles the final response correctly from node-B's output +- [ ] The test passes with no external services running +- [ ] The activation tensor serialisation format is documented in a short inline comment (shape, dtype, wire format) — this becomes the contract all future nodes implement + +## Blocked by + +- `01-monorepo-scaffold.md` diff --git a/.scratch/distributed-inference-network/issues/03-tracker-registration-and-routing.md b/.scratch/distributed-inference-network/issues/03-tracker-registration-and-routing.md new file mode 100644 index 0000000..224efce --- /dev/null +++ b/.scratch/distributed-inference-network/issues/03-tracker-registration-and-routing.md @@ -0,0 +1,24 @@ +Status: ready-for-agent + +# 03 — Tracker: node registration + route selection + +## What to build + +Replace the hardcoded two-node route from issue 02 with a real tracker. Nodes register themselves with the tracker on startup (endpoint, shard range, hardware profile, node score). The gateway queries the tracker for an inference route for a given model preset instead of using a static list. The tracker returns an ordered list of node endpoints whose shards collectively cover all layers. + +The tracker runs as a lightweight HTTP service. Node score is initially a simple placeholder (e.g. fixed value or random) — real throughput/latency scoring comes later. The tracker must correctly reject route requests when no registered nodes cover a required shard range and return an appropriate error. + +This issue establishes the tracker's registration and routing API contract, which issues 04 (node startup) and 05 (OpenAI gateway) both depend on. + +## Acceptance criteria + +- [ ] A node can register with the tracker via HTTP, providing its endpoint, shard range, and a hardware profile +- [ ] The gateway queries the tracker with a model preset name and receives an ordered list of node endpoints forming a complete inference route +- [ ] The end-to-end integration test from issue 02 still passes with the tracker now in the loop (no hardcoded routes) +- [ ] The tracker returns a clear error response when no route is available for a requested model preset +- [ ] Nodes that fail to heartbeat within a configurable window are removed from the registry +- [ ] The tracker's registration and route-selection HTTP API is defined (paths, request/response shapes) in the tracker package + +## Blocked by + +- `02-two-node-shard-pipeline.md` diff --git a/.scratch/distributed-inference-network/issues/04-node-client-startup.md b/.scratch/distributed-inference-network/issues/04-node-client-startup.md new file mode 100644 index 0000000..28020c2 --- /dev/null +++ b/.scratch/distributed-inference-network/issues/04-node-client-startup.md @@ -0,0 +1,32 @@ +Status: ready-for-agent + +# 04 — Node client startup flow (`meshnet-node start`) + +## What to build + +Make `meshnet-node start` self-configuring from scratch on a machine with a CUDA-capable GPU (or CPU fallback). The full startup sequence must complete without any manual configuration: + +1. Detect GPU model and available VRAM +2. Load an existing Solana wallet from disk, or generate and save a new one +3. Query the tracker for the optimal shard assignment given the hardware profile +4. Download the assigned shard from HuggingFace (`huggingface_hub`) +5. Register with the tracker (wallet address, endpoint, shard range, hardware profile) +6. Begin accepting inference connections + +The node prints a short status summary on startup: wallet address, assigned shard, model preset, and download progress. No interactive prompts — the entire flow is non-interactive. + +This is the primary viral growth vector. Every second of friction in this flow costs node operators. The startup sequence must work on Linux with CUDA, and degrade gracefully to CPU on machines without a GPU. + +## Acceptance criteria + +- [ ] `meshnet-node start --tracker http://localhost:8080` completes startup without prompts on a machine with a CUDA GPU +- [ ] A Solana wallet keypair is generated and saved to `~/.config/meshnet/wallet.json` if none exists +- [ ] The assigned shard is downloaded from HuggingFace and cached to `~/.cache/meshnet/shards/` +- [ ] The node registers with the tracker and appears in the tracker's node registry +- [ ] The node accepts a live inference connection and processes activation tensors correctly after startup +- [ ] On a CPU-only machine, the node starts with a warning and serves a CPU-appropriate shard +- [ ] An integration test covers the full startup sequence against a local tracker stub + +## Blocked by + +- `03-tracker-registration-and-routing.md` diff --git a/.scratch/distributed-inference-network/issues/05-openai-compatible-gateway.md b/.scratch/distributed-inference-network/issues/05-openai-compatible-gateway.md new file mode 100644 index 0000000..41db5a7 --- /dev/null +++ b/.scratch/distributed-inference-network/issues/05-openai-compatible-gateway.md @@ -0,0 +1,29 @@ +Status: ready-for-agent + +# 05 — OpenAI-compatible gateway + +## What to build + +Expose a production-shape HTTP API from the gateway so that any client using the OpenAI Python SDK (or any OpenAI-compatible tool) works by changing only the `base_url`. The gateway translates OpenAI-format requests into inference route execution and streams responses back in OpenAI-format. + +Endpoints to implement: +- `POST /v1/chat/completions` — streaming (`text/event-stream`) and non-streaming +- `GET /v1/models` — returns the list of model presets currently routable on the network +- `GET /v1/health` — liveness check + +The gateway selects an inference route from the tracker, executes the shard pipeline, and assembles the streamed response. If the tracker returns no route for the requested model, the gateway responds with a standard OpenAI-format error (`model_not_available`). + +Authentication (API key → SOL/USDC balance) is a stub in this issue — return 200 for any non-empty `Authorization` header. Real payment gating comes in issue 06. + +## Acceptance criteria + +- [ ] `openai.OpenAI(base_url="http://localhost:8080/v1", api_key="test").chat.completions.create(model="stub-model", messages=[...])` returns a valid response +- [ ] Streaming works: `stream=True` returns `text/event-stream` chunks in OpenAI SSE format +- [ ] `GET /v1/models` returns a JSON array of available model preset names +- [ ] A request for an unavailable model returns an OpenAI-format error response with HTTP 503 +- [ ] LangChain `ChatOpenAI(base_url=..., api_key=...)` works against the gateway +- [ ] An integration test covers streaming and non-streaming paths end-to-end through a real tracker and two stub nodes + +## Blocked by + +- `03-tracker-registration-and-routing.md` diff --git a/.scratch/distributed-inference-network/issues/06-solana-stake-and-settlement.md b/.scratch/distributed-inference-network/issues/06-solana-stake-and-settlement.md new file mode 100644 index 0000000..0c759cf --- /dev/null +++ b/.scratch/distributed-inference-network/issues/06-solana-stake-and-settlement.md @@ -0,0 +1,32 @@ +Status: ready-for-agent + +# 06 — Solana stake + settlement contracts + +## What to build + +Deploy and integrate the Solana smart contracts that make node staking, client payment, and token reward settlement trustless. All development and testing targets **Solana testnet** — never devnet or mainnet during development, to avoid real costs. + +Three contracts are needed: + +**Registry contract**: records stake balances, strike counts, and ban status per wallet. The gateway and tracker read from this contract to exclude banned wallets from route selection. + +**Payment contract**: clients pre-fund an API key account with SOL or USDC. The gateway records per-request compute attribution (which node served which layer range, for how many tokens). + +**Settlement contract**: called once per epoch. Debits client accounts proportional to compute consumed. Credits node operator wallets with our native token proportional to layers served. Distributes a validator reward share. + +The `packages/contracts` package provides Python wrappers for reading and writing to all three contracts. The gateway uses these wrappers to check stake before routing to a node and to record attribution after each request. + +## Acceptance criteria + +- [ ] All contracts deploy successfully to Solana testnet +- [ ] A node can submit a stake transaction and have its balance reflected in the registry contract +- [ ] A client can fund an API key account with testnet SOL +- [ ] After a completed inference session, compute attribution is recorded on-chain with correct node/layer attribution +- [ ] The epoch settlement transaction correctly distributes token rewards to node operators and deducts client balances +- [ ] The gateway refuses to route to a node whose stake balance is below the minimum threshold +- [ ] All contract interactions in tests run against a local Solana test validator (via `solana-test-validator`) — no live testnet required for CI +- [ ] A `.env.testnet` config points to Solana testnet RPC for manual end-to-end testing + +## Blocked by + +- `03-tracker-registration-and-routing.md` diff --git a/.scratch/distributed-inference-network/issues/07-fraud-detection-slash.md b/.scratch/distributed-inference-network/issues/07-fraud-detection-slash.md new file mode 100644 index 0000000..e673b5f --- /dev/null +++ b/.scratch/distributed-inference-network/issues/07-fraud-detection-slash.md @@ -0,0 +1,28 @@ +Status: ready-for-agent + +# 07 — Fraud detection: validator + on-chain slash + +## What to build + +Implement the optimistic fraud detection loop. After each inference request completes, the validator process independently decides whether to re-run the request on a reference node (~5% sample rate). If the reference output diverges from the node's output beyond a configurable floating-point tolerance, the validator submits a slash transaction on-chain. + +The validator is a separate process (not the gateway or tracker). It subscribes to completed inference events, selects a random sample, re-runs those requests against a trusted reference node, compares outputs, and submits slash proofs when fraud is detected. + +On slash: the offending node's stake balance decreases by the configured slash amount; its strike count increments. The gateway reads strike state from the registry contract on each route selection and excludes nodes that exceed the strike threshold. + +Node operators receive a push notification (logged to stdout and, if configured, a webhook) when their node is slashed. + +## Acceptance criteria + +- [ ] The validator process samples ~5% of completed inference requests (configurable) +- [ ] A node returning a deliberately wrong output is detected and slashed within one validation cycle +- [ ] The on-chain stake balance of the slashed node decreases by the correct slash amount +- [ ] The strike count for the slashed node increments on-chain +- [ ] A node that reaches the strike threshold is excluded from route selection on the next gateway request +- [ ] A slashed node logs a clear warning to stdout +- [ ] An integration test: run a deliberately-bad node, send 20 requests, assert at least one slash transaction is submitted and the node is eventually excluded from routes + +## Blocked by + +- `05-openai-compatible-gateway.md` +- `06-solana-stake-and-settlement.md` diff --git a/.scratch/distributed-inference-network/issues/08-probationary-period-and-bans.md b/.scratch/distributed-inference-network/issues/08-probationary-period-and-bans.md new file mode 100644 index 0000000..8a22394 --- /dev/null +++ b/.scratch/distributed-inference-network/issues/08-probationary-period-and-bans.md @@ -0,0 +1,27 @@ +Status: ready-for-agent + +# 08 — Node probationary period + ban enforcement + +## What to build + +Implement the two anti-sybil mechanisms that make re-entering the network after a ban economically costly. + +**Probationary period**: a new wallet's first N completed inference jobs earn no token rewards. N is a configurable on-chain parameter (default: 50 jobs). The settlement contract reads the wallet's job count and skips reward distribution until the threshold is met. The node client displays the remaining probationary job count on startup. + +**Ban enforcement**: when a wallet's strike count reaches the configured threshold, the registry contract marks it as banned. The tracker and gateway must check ban status on every route selection and exclude banned wallets. Ban status is read exclusively from the Solana registry contract — the tracker cannot set or unset bans. + +Both mechanisms together mean a banned node operator must: create a new wallet, fund a new stake, and complete N free jobs before earning again — a meaningful economic barrier. + +## Acceptance criteria + +- [ ] A node with a new wallet receives no token rewards for its first N jobs (verified via settlement contract state) +- [ ] The node client prints remaining probationary jobs on startup (e.g. "Probationary period: 38 jobs remaining before earning") +- [ ] After N jobs, the next epoch settlement correctly credits the node with token rewards +- [ ] A wallet with strike count at the threshold is marked banned in the registry contract +- [ ] The tracker excludes banned wallets from route selection +- [ ] A banned wallet that attempts to register with the tracker is rejected +- [ ] An integration test covers: new wallet → N jobs → earning begins; and: strike threshold reached → banned → excluded from routes + +## Blocked by + +- `07-fraud-detection-slash.md` diff --git a/.scratch/distributed-inference-network/issues/09-p2p-shard-swarm.md b/.scratch/distributed-inference-network/issues/09-p2p-shard-swarm.md new file mode 100644 index 0000000..37e20e8 --- /dev/null +++ b/.scratch/distributed-inference-network/issues/09-p2p-shard-swarm.md @@ -0,0 +1,24 @@ +Status: ready-for-agent + +# 09 — P2P shard swarm + +## What to build + +Once a node has downloaded a shard, it seeds that shard to other nodes that are assigned the same shard. New nodes with the same shard assignment should download from the swarm (peer nodes) rather than HuggingFace when peers are available, reducing HuggingFace load and speeding up propagation of popular shards. + +The shard swarm is keyed by `(model_preset, shard_index)`. Nodes announce their shard availability to the tracker on registration. When a new node requests a shard download, the tracker returns a list of peer endpoints that hold the shard alongside the HuggingFace fallback URL. The node tries peers first; falls back to HuggingFace if no peers respond within a timeout. + +Peers serve shard data over a simple chunked HTTP endpoint — no custom P2P protocol needed in v1. The node client displays download source (peer or HuggingFace) during shard download. + +## Acceptance criteria + +- [ ] A node that has downloaded a shard is listed as a peer for that shard in the tracker +- [ ] A second node assigned the same shard downloads it from the first node (peer) rather than HuggingFace (verified by asserting no HuggingFace HTTP calls are made in the test) +- [ ] If no peers are available, the node falls back to HuggingFace without error +- [ ] The node client logs whether the shard was downloaded from a peer or HuggingFace +- [ ] Shard chunks are verified against a checksum before being marked complete +- [ ] An integration test: node A downloads shard from HuggingFace stub; node B with same assignment downloads from node A + +## Blocked by + +- `04-node-client-startup.md` diff --git a/.scratch/distributed-inference-network/issues/10-meshnet-sdk.md b/.scratch/distributed-inference-network/issues/10-meshnet-sdk.md new file mode 100644 index 0000000..9def6a5 --- /dev/null +++ b/.scratch/distributed-inference-network/issues/10-meshnet-sdk.md @@ -0,0 +1,44 @@ +Status: ready-for-agent + +# 10 — `meshnet` Python SDK + +## What to build + +A Python SDK (`packages/sdk`) that wraps the OpenAI-compatible gateway and exposes network-specific controls that the raw OpenAI SDK cannot express. The SDK is the recommended interface for developers who want more than basic inference. + +The SDK surface: + +```python +from meshnet import Client + +client = Client(api_key="...", base_url="https://api.meshnet.ai") + +# Basic inference (identical to OpenAI SDK) +response = client.chat.completions.create(model="llama-3-70b", messages=[...]) + +# Network-specific features +client.wallet.balance() # SOL/USDC balance remaining +client.wallet.top_up(amount_sol=1.0) # returns a Solana payment address + QR +client.models.available() # list of routable model presets with shard coverage % +client.request(redundancy=2) # route through 2 independent chains, return majority +client.estimate_cost(model="llama-3-70b", tokens=1000) # cost in SOL before sending +``` + +The SDK must not require clients to hold or know about our native token. All wallet operations are in SOL or USDC. + +## Acceptance criteria + +- [ ] `pip install meshnet` installs the SDK +- [ ] `Client.chat.completions.create(...)` works identically to the OpenAI SDK (same parameters, same response shape) +- [ ] `client.wallet.balance()` returns the current SOL/USDC balance for the API key +- [ ] `client.wallet.top_up()` returns a valid Solana payment address +- [ ] `client.models.available()` returns model presets with shard coverage percentage +- [ ] `client.estimate_cost(model, tokens)` returns a cost estimate in SOL before the request is sent +- [ ] `client.request(redundancy=2)` sends the same prompt to two independent inference routes and returns the majority response +- [ ] The SDK is typed (py.typed, full type stubs) +- [ ] An integration test covers each SDK method against a local gateway + tracker + stub nodes + +## Blocked by + +- `05-openai-compatible-gateway.md` +- `06-solana-stake-and-settlement.md` diff --git a/docs/adr/0002-dual-token-payment-model.md b/docs/adr/0002-dual-token-payment-model.md new file mode 100644 index 0000000..85ff9e8 --- /dev/null +++ b/docs/adr/0002-dual-token-payment-model.md @@ -0,0 +1,11 @@ +# Dual token payment model: own token for nodes, SOL/USDC for clients + +Clients pay in SOL or USDC — familiar, easy to buy, no new token required. Node operators stake and earn our native Solana L2 token, which creates speculative upside for early contributors and a staking mechanism for fraud prevention (slashing). The two are decoupled: client payments are auto-converted to partially fund token rewards; clients never need to hold or know about our token. + +Using SOL alone would give node operators no early-adopter upside ("why run my GPU for market-rate SOL?"), killing the viral growth mechanic. Launching a token-only system requires clients to acquire our token, adding friction that kills adoption. The dual model solves both. + +## Considered Options + +- **SOL only**: simplest, no token launch risk, but no node incentive beyond spot market rates +- **Own token only**: maximum node incentive, but clients must acquire it — high friction +- **Own token (stake/rewards) + SOL/USDC (client pay)**: clean separation of concerns, chosen diff --git a/docs/adr/0003-optimistic-fraud-detection.md b/docs/adr/0003-optimistic-fraud-detection.md new file mode 100644 index 0000000..50458f6 --- /dev/null +++ b/docs/adr/0003-optimistic-fraud-detection.md @@ -0,0 +1,14 @@ +# Optimistic trust with stake slashing and strike-based bans + +All inference responses are trusted by default. Validators re-run a random sample (~5%) of requests on reference nodes and compare outputs. Nodes that fail are slashed (stake reduced). Enough strikes result in a permanent on-chain ban. + +New wallets must complete N jobs without earning (probationary period) to raise the economic cost of re-entering after a ban — a banned node can't just create a new wallet and immediately cheat again; it must fund a new stake and contribute N free jobs first. + +zkML (zero-knowledge proofs of inference) would give cryptographic guarantees but is 1000–10000× slower than inference for large models and is not production-ready. Redundant execution consensus (Gensyn's approach) gives stronger guarantees but costs 2–3× compute per request. TEE (trusted hardware attestation) is cryptographically strong but excludes most consumer GPUs, defeating the viral GPU-sharing goal. Optimistic + slash is the pragmatic choice that ships and can be calibrated economically. + +## Considered Options + +- **zkML**: cryptographically perfect, not production-ready for large models in 2025 +- **Redundant consensus**: strong guarantees, 2-3× compute cost per request +- **TEE attestation**: strong guarantees, excludes consumer GPUs +- **Optimistic + slash + strike ban**: chosen — ships, works, economically tunable diff --git a/docs/adr/0004-hybrid-tracker.md b/docs/adr/0004-hybrid-tracker.md new file mode 100644 index 0000000..acdac5a --- /dev/null +++ b/docs/adr/0004-hybrid-tracker.md @@ -0,0 +1,11 @@ +# Hybrid tracker: centralized routing + P2P gossip, Solana as source of truth + +The tracker runs as a centralized service (we operate it) for fast, optimizable inference routing and node scoring. Nodes also gossip state via a lightweight P2P layer so the network degrades gracefully if the tracker is unavailable. Critically, Solana smart contracts are the authoritative source for stake balances, slash events, strike counts, and bans — the tracker can read this state but cannot modify it, preventing us from manipulating payouts even if we control the routing layer. + +A pure DHT (like Petals uses) makes latency-aware routing and ban enforcement significantly harder to implement correctly. A purely centralized tracker is a single point of failure and a regulatory target. The hybrid gives us fast routing now with a clear path to decentralization as the network scales. + +## Considered Options + +- **Pure DHT**: resilient, harder to build good routing and enforce bans +- **Centralized only**: simple, single point of failure, we could manipulate payouts +- **Hybrid (centralized + P2P gossip, Solana authoritative)**: chosen diff --git a/docs/adr/0005-shard-distribution.md b/docs/adr/0005-shard-distribution.md new file mode 100644 index 0000000..eb68993 --- /dev/null +++ b/docs/adr/0005-shard-distribution.md @@ -0,0 +1,5 @@ +# HuggingFace bootstrap + P2P propagation for shard distribution + +When the tracker assigns a node a new shard, the node downloads it from HuggingFace (or our seed servers as fallback). Once downloaded, the node joins the P2P shard swarm for that shard and seeds to other nodes that need it. Popular shards propagate virally; new shards bootstrap from HF. + +This mirrors how BitTorrent works in practice: a seed server provides the first copy, P2P takes over distribution as the swarm grows. Having nodes bring their own weights (manual download) makes tracker-driven shard rebalancing impossible — you can't instruct a node to switch shards if it doesn't have the weights. Pure P2P with no bootstrap source means the first node for any shard has nowhere to download from. diff --git a/docs/adr/0006-openai-compatible-api.md b/docs/adr/0006-openai-compatible-api.md new file mode 100644 index 0000000..cf8a9b8 --- /dev/null +++ b/docs/adr/0006-openai-compatible-api.md @@ -0,0 +1,7 @@ +# OpenAI-compatible REST API as the primary client interface + +The gateway exposes an OpenAI-compatible REST API (`/v1/chat/completions`, `/v1/models`, etc.) so any existing application using the OpenAI SDK can switch to our network by changing one line (`base_url`). LangChain, LlamaIndex, Open WebUI, and thousands of other tools work immediately with zero code changes. + +A `meshnet` SDK wraps the OpenAI-compatible interface and exposes network-specific controls: wallet top-up, redundancy level, node selection hints, cost estimates, and streaming earnings. Clients who don't need these features never interact with the SDK. + +A custom-protocol-only API would require every adopter to rewrite their existing LLM integrations, killing day-one adoption.