docs: add ADRs and user stories for real model inference stack (US-011–014)
ADR-0008: binary activation wire format — raw bfloat16 over HTTP, zstd compression, 128-token chunked prefill; replaces base64 JSON (~33% overhead removed). ADR-0009: coverage-first shard assignment and tracker-as-first-layer-node — any node serving layers[0..k] becomes the inference entry point for that model; bin-packing fills all coverage gaps before adding redundancy; tracker issues LOAD_SHARD/DROP_SHARD rebalance directives; nodes declare VRAM + quantization. US-011: binary wire format migration US-012: real PyTorch layer execution (transformers + bitsandbytes, test on GPT-2) US-013: coverage-first tracker bin-packing with VRAM-aware shard assignment US-014: tracker-as-node (tracker node serves first layers + handles client requests) CONTEXT.md: Tracker Node, Coverage Map, Rebalance Directive terms added. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Dobromir Popov
@@ -5,8 +5,8 @@
|
||||
"userStories": [
|
||||
{
|
||||
"id": "US-001",
|
||||
"title": "01 — Monorepo scaffold + single-node smoke test",
|
||||
"description": "Stand up the monorepo package layout and prove that a client HTTP request can travel end-to-end through the stack — gateway → one node serving all layers → valid response — before any real model or distributed logic is wired in. The six top-level packages should exist with minimal stubs: - `packages/node` — node client CLI (`meshnet-node`) - `packages/gateway` — HTTP gateway + route orchestration - `packages/tracker` — node registry and route selection - `packages/sdk` — `meshnet` Python SDK - `packages/contracts` — Solana smart contract wrappers - `packages/p2p` — gossip and shard swarm The smoke test uses a tiny in-process stub model (not a real LLM) so the test is fast and has no external dependencies. The node runs in the same process as the test. The gateway is a real HTTP server. The client sends a real `POST /v1/chat/completions` request and receives a valid OpenAI-format response.",
|
||||
"title": "01 \u2014 Monorepo scaffold + single-node smoke test",
|
||||
"description": "Stand up the monorepo package layout and prove that a client HTTP request can travel end-to-end through the stack \u2014 gateway \u2192 one node serving all layers \u2192 valid response \u2014 before any real model or distributed logic is wired in. The six top-level packages should exist with minimal stubs: - `packages/node` \u2014 node client CLI (`meshnet-node`) - `packages/gateway` \u2014 HTTP gateway + route orchestration - `packages/tracker` \u2014 node registry and route selection - `packages/sdk` \u2014 `meshnet` Python SDK - `packages/contracts` \u2014 Solana smart contract wrappers - `packages/p2p` \u2014 gossip and shard swarm The smoke test uses a tiny in-process stub model (not a real LLM) so the test is fast and has no external dependencies. The node runs in the same process as the test. The gateway is a real HTTP server. The client sends a real `POST /v1/chat/completions` request and receives a valid OpenAI-format response.",
|
||||
"acceptanceCriteria": [
|
||||
"`pip install -e packages/node packages/gateway packages/tracker` works from repo root",
|
||||
"`meshnet-node`, `meshnet-gateway`, and `meshnet-tracker` CLI entry points exist",
|
||||
@@ -27,14 +27,14 @@
|
||||
},
|
||||
{
|
||||
"id": "US-002",
|
||||
"title": "02 — Two-node shard pipeline",
|
||||
"description": "Extend the single-node smoke test so that inference is routed through exactly two nodes, each serving half of a stub model's layers. Activation tensors must travel from the gateway to node A, then from node A to node B, and the final output returns to the gateway. This proves the distributed shard pipeline works end-to-end before real models or a real tracker are introduced. The two nodes run in the same process as the test (no real network required). The gateway hardcodes the two-node route for now — dynamic route selection comes in issue 03. The activation tensor protocol (shape, dtype, serialisation format) must be established here, as all later issues depend on it. Use the domain vocabulary from `CONTEXT.md`: the ordered list of nodes is an **inference route**; each node's layer range is a **shard**; the tensors passed between nodes are activations (not \"data\" or \"chunks\").",
|
||||
"title": "02 \u2014 Two-node shard pipeline",
|
||||
"description": "Extend the single-node smoke test so that inference is routed through exactly two nodes, each serving half of a stub model's layers. Activation tensors must travel from the gateway to node A, then from node A to node B, and the final output returns to the gateway. This proves the distributed shard pipeline works end-to-end before real models or a real tracker are introduced. The two nodes run in the same process as the test (no real network required). The gateway hardcodes the two-node route for now \u2014 dynamic route selection comes in issue 03. The activation tensor protocol (shape, dtype, serialisation format) must be established here, as all later issues depend on it. Use the domain vocabulary from `CONTEXT.md`: the ordered list of nodes is an **inference route**; each node's layer range is a **shard**; the tensors passed between nodes are activations (not \"data\" or \"chunks\").",
|
||||
"acceptanceCriteria": [
|
||||
"The integration test spins up two stub nodes, each configured with a non-overlapping shard range",
|
||||
"A `POST /v1/chat/completions` request results in activation tensors flowing node-A → node-B (verifiable via test assertions or logs)",
|
||||
"A `POST /v1/chat/completions` request results in activation tensors flowing node-A \u2192 node-B (verifiable via test assertions or logs)",
|
||||
"The gateway assembles the final response correctly from node-B's output",
|
||||
"The test passes with no external services running",
|
||||
"The activation tensor serialisation format is documented in a short inline comment (shape, dtype, wire format) — this becomes the contract all future nodes implement",
|
||||
"The activation tensor serialisation format is documented in a short inline comment (shape, dtype, wire format) \u2014 this becomes the contract all future nodes implement",
|
||||
"Follow TDD where code is added: add/adjust an observable behavior test before implementation when practical",
|
||||
"Run the task-specific tests and ensure they pass",
|
||||
"Run `python -m pytest` from repo root and ensure it passes, or document why no test suite exists yet",
|
||||
@@ -51,8 +51,8 @@
|
||||
},
|
||||
{
|
||||
"id": "US-003",
|
||||
"title": "03 — Tracker: node registration + route selection",
|
||||
"description": "Replace the hardcoded two-node route from issue 02 with a real tracker. Nodes register themselves with the tracker on startup (endpoint, shard range, hardware profile, node score). The gateway queries the tracker for an inference route for a given model preset instead of using a static list. The tracker returns an ordered list of node endpoints whose shards collectively cover all layers. The tracker runs as a lightweight HTTP service. Node score is initially a simple placeholder (e.g. fixed value or random) — real throughput/latency scoring comes later. The tracker must correctly reject route requests when no registered nodes cover a required shard range and return an appropriate error. This issue establishes the tracker's registration and routing API contract, which issues 04 (node startup) and 05 (OpenAI gateway) both depend on.",
|
||||
"title": "03 \u2014 Tracker: node registration + route selection",
|
||||
"description": "Replace the hardcoded two-node route from issue 02 with a real tracker. Nodes register themselves with the tracker on startup (endpoint, shard range, hardware profile, node score). The gateway queries the tracker for an inference route for a given model preset instead of using a static list. The tracker returns an ordered list of node endpoints whose shards collectively cover all layers. The tracker runs as a lightweight HTTP service. Node score is initially a simple placeholder (e.g. fixed value or random) \u2014 real throughput/latency scoring comes later. The tracker must correctly reject route requests when no registered nodes cover a required shard range and return an appropriate error. This issue establishes the tracker's registration and routing API contract, which issues 04 (node startup) and 05 (OpenAI gateway) both depend on.",
|
||||
"acceptanceCriteria": [
|
||||
"A node can register with the tracker via HTTP, providing its endpoint, shard range, and a hardware profile",
|
||||
"The gateway queries the tracker with a model preset name and receives an ordered list of node endpoints forming a complete inference route",
|
||||
@@ -76,8 +76,8 @@
|
||||
},
|
||||
{
|
||||
"id": "US-004",
|
||||
"title": "04 — Node client startup flow (`meshnet-node start`)",
|
||||
"description": "Make `meshnet-node start` self-configuring from scratch on a machine with a CUDA-capable GPU (or CPU fallback). The full startup sequence must complete without any manual configuration: 1. Detect GPU model and available VRAM 2. Load an existing Solana wallet from disk, or generate and save a new one 3. Query the tracker for the optimal shard assignment given the hardware profile 4. Download the assigned shard from HuggingFace (`huggingface_hub`) 5. Register with the tracker (wallet address, endpoint, shard range, hardware profile) 6. Begin accepting inference connections The node prints a short status summary on startup: wallet address, assigned shard, model preset, and download progress. No interactive prompts — the entire flow is non-interactive. This is the primary viral growth vector. Every second of friction in this flow costs node operators. The startup sequence must work on Linux with CUDA, and degrade gracefully to CPU on machines without a GPU.",
|
||||
"title": "04 \u2014 Node client startup flow (`meshnet-node start`)",
|
||||
"description": "Make `meshnet-node start` self-configuring from scratch on a machine with a CUDA-capable GPU (or CPU fallback). The full startup sequence must complete without any manual configuration: 1. Detect GPU model and available VRAM 2. Load an existing Solana wallet from disk, or generate and save a new one 3. Query the tracker for the optimal shard assignment given the hardware profile 4. Download the assigned shard from HuggingFace (`huggingface_hub`) 5. Register with the tracker (wallet address, endpoint, shard range, hardware profile) 6. Begin accepting inference connections The node prints a short status summary on startup: wallet address, assigned shard, model preset, and download progress. No interactive prompts \u2014 the entire flow is non-interactive. This is the primary viral growth vector. Every second of friction in this flow costs node operators. The startup sequence must work on Linux with CUDA, and degrade gracefully to CPU on machines without a GPU.",
|
||||
"acceptanceCriteria": [
|
||||
"`meshnet-node start --tracker http://localhost:8080` completes startup without prompts on a machine with a CUDA GPU",
|
||||
"A Solana wallet keypair is generated and saved to `~/.config/meshnet/wallet.json` if none exists",
|
||||
@@ -102,8 +102,8 @@
|
||||
},
|
||||
{
|
||||
"id": "US-005",
|
||||
"title": "05 — OpenAI-compatible gateway",
|
||||
"description": "Expose a production-shape HTTP API from the gateway so that any client using the OpenAI Python SDK (or any OpenAI-compatible tool) works by changing only the `base_url`. The gateway translates OpenAI-format requests into inference route execution and streams responses back in OpenAI-format. Endpoints to implement: - `POST /v1/chat/completions` — streaming (`text/event-stream`) and non-streaming - `GET /v1/models` — returns the list of model presets currently routable on the network - `GET /v1/health` — liveness check The gateway selects an inference route from the tracker, executes the shard pipeline, and assembles the streamed response. If the tracker returns no route for the requested model, the gateway responds with a standard OpenAI-format error (`model_not_available`). Authentication (API key → SOL/USDC balance) is a stub in this issue — return 200 for any non-empty `Authorization` header. Real payment gating comes in issue 06.",
|
||||
"title": "05 \u2014 OpenAI-compatible gateway",
|
||||
"description": "Expose a production-shape HTTP API from the gateway so that any client using the OpenAI Python SDK (or any OpenAI-compatible tool) works by changing only the `base_url`. The gateway translates OpenAI-format requests into inference route execution and streams responses back in OpenAI-format. Endpoints to implement: - `POST /v1/chat/completions` \u2014 streaming (`text/event-stream`) and non-streaming - `GET /v1/models` \u2014 returns the list of model presets currently routable on the network - `GET /v1/health` \u2014 liveness check The gateway selects an inference route from the tracker, executes the shard pipeline, and assembles the streamed response. If the tracker returns no route for the requested model, the gateway responds with a standard OpenAI-format error (`model_not_available`). Authentication (API key \u2192 SOL/USDC balance) is a stub in this issue \u2014 return 200 for any non-empty `Authorization` header. Real payment gating comes in issue 06.",
|
||||
"acceptanceCriteria": [
|
||||
"`openai.OpenAI(base_url=\"http://localhost:8080/v1\", api_key=\"test\").chat.completions.create(model=\"stub-model\", messages=[...])` returns a valid response",
|
||||
"Streaming works: `stream=True` returns `text/event-stream` chunks in OpenAI SSE format",
|
||||
@@ -127,8 +127,8 @@
|
||||
},
|
||||
{
|
||||
"id": "US-006",
|
||||
"title": "06 — Solana stake + settlement contracts",
|
||||
"description": "Deploy and integrate the Solana smart contracts that make node staking, client payment, and token reward settlement trustless. All development and testing targets **Solana testnet** — never devnet or mainnet during development, to avoid real costs. Three contracts are needed: **Registry contract**: records stake balances, strike counts, and ban status per wallet. The gateway and tracker read from this contract to exclude banned wallets from route selection. **Payment contract**: clients pre-fund an API key account with SOL or USDC. The gateway records per-request compute attribution (which node served which layer range, for how many tokens). **Settlement contract**: called once per epoch. Debits client accounts proportional to compute consumed. Credits node operator wallets with our native token proportional to layers served. Distributes a validator reward share. The `packages/contracts` package provides Python wrappers for reading and writing to all three contracts. The gateway uses these wrappers to check stake before routing to a node and to record attribution after each request.",
|
||||
"title": "06 \u2014 Solana stake + settlement contracts",
|
||||
"description": "Deploy and integrate the Solana smart contracts that make node staking, client payment, and token reward settlement trustless. All development and testing targets **Solana testnet** \u2014 never devnet or mainnet during development, to avoid real costs.",
|
||||
"acceptanceCriteria": [
|
||||
"All contracts deploy successfully to Solana testnet",
|
||||
"A node can submit a stake transaction and have its balance reflected in the registry contract",
|
||||
@@ -136,13 +136,10 @@
|
||||
"After a completed inference session, compute attribution is recorded on-chain with correct node/layer attribution",
|
||||
"The epoch settlement transaction correctly distributes token rewards to node operators and deducts client balances",
|
||||
"The gateway refuses to route to a node whose stake balance is below the minimum threshold",
|
||||
"All contract interactions in tests run against a local Solana test validator (via `solana-test-validator`) — no live testnet required for CI",
|
||||
"All contract interactions in tests run against a local Solana test validator (via `solana-test-validator`) \u2014 no live testnet required for CI",
|
||||
"A `.env.testnet` config points to Solana testnet RPC for manual end-to-end testing",
|
||||
"Follow TDD where code is added: add/adjust an observable behavior test before implementation when practical",
|
||||
"Run the task-specific tests and ensure they pass",
|
||||
"Run `python -m pytest` from repo root and ensure it passes, or document why no test suite exists yet",
|
||||
"Keep the implementation scoped to this story; do not implement downstream stories early",
|
||||
"Commit only this story's code changes with a focused conventional commit message"
|
||||
"python -m pytest passes from repo root",
|
||||
"Commit only this story's changes"
|
||||
],
|
||||
"priority": 6,
|
||||
"passes": true,
|
||||
@@ -154,8 +151,8 @@
|
||||
},
|
||||
{
|
||||
"id": "US-007",
|
||||
"title": "07 — Fraud detection: validator + on-chain slash",
|
||||
"description": "Implement the optimistic fraud detection loop. After each inference request completes, the validator process independently decides whether to re-run the request on a reference node (~5% sample rate). If the reference output diverges from the node's output beyond a configurable floating-point tolerance, the validator submits a slash transaction on-chain. The validator is a separate process (not the gateway or tracker). It subscribes to completed inference events, selects a random sample, re-runs those requests against a trusted reference node, compares outputs, and submits slash proofs when fraud is detected. On slash: the offending node's stake balance decreases by the configured slash amount; its strike count increments. The gateway reads strike state from the registry contract on each route selection and excludes nodes that exceed the strike threshold. Node operators receive a push notification (logged to stdout and, if configured, a webhook) when their node is slashed.",
|
||||
"title": "07 \u2014 Fraud detection: validator + on-chain slash",
|
||||
"description": "Implement the optimistic fraud detection loop. After each inference request completes, the validator process independently decides whether to re-run the request on a reference node (~5% sample rate).",
|
||||
"acceptanceCriteria": [
|
||||
"The validator process samples ~5% of completed inference requests (configurable)",
|
||||
"A node returning a deliberately wrong output is detected and slashed within one validation cycle",
|
||||
@@ -164,11 +161,8 @@
|
||||
"A node that reaches the strike threshold is excluded from route selection on the next gateway request",
|
||||
"A slashed node logs a clear warning to stdout",
|
||||
"An integration test: run a deliberately-bad node, send 20 requests, assert at least one slash transaction is submitted and the node is eventually excluded from routes",
|
||||
"Follow TDD where code is added: add/adjust an observable behavior test before implementation when practical",
|
||||
"Run the task-specific tests and ensure they pass",
|
||||
"Run `python -m pytest` from repo root and ensure it passes, or document why no test suite exists yet",
|
||||
"Keep the implementation scoped to this story; do not implement downstream stories early",
|
||||
"Commit only this story's code changes with a focused conventional commit message"
|
||||
"python -m pytest passes from repo root",
|
||||
"Commit only this story's changes"
|
||||
],
|
||||
"priority": 7,
|
||||
"passes": true,
|
||||
@@ -181,21 +175,18 @@
|
||||
},
|
||||
{
|
||||
"id": "US-008",
|
||||
"title": "08 — Node probationary period + ban enforcement",
|
||||
"description": "Implement the two anti-sybil mechanisms that make re-entering the network after a ban economically costly. **Probationary period**: a new wallet's first N completed inference jobs earn no token rewards. N is a configurable on-chain parameter (default: 50 jobs). The settlement contract reads the wallet's job count and skips reward distribution until the threshold is met. The node client displays the remaining probationary job count on startup. **Ban enforcement**: when a wallet's strike count reaches the configured threshold, the registry contract marks it as banned. The tracker and gateway must check ban status on every route selection and exclude banned wallets. Ban status is read exclusively from the Solana registry contract — the tracker cannot set or unset bans. Both mechanisms together mean a banned node operator must: create a new wallet, fund a new stake, and complete N free jobs before earning again — a meaningful economic barrier.",
|
||||
"title": "08 \u2014 Node probationary period + ban enforcement",
|
||||
"description": "Implement the two anti-sybil mechanisms that make re-entering the network after a ban economically costly.",
|
||||
"acceptanceCriteria": [
|
||||
"A node with a new wallet receives no token rewards for its first N jobs (verified via settlement contract state)",
|
||||
"The node client prints remaining probationary jobs on startup (e.g. \"Probationary period: 38 jobs remaining before earning\")",
|
||||
"The node client prints remaining probationary jobs on startup",
|
||||
"After N jobs, the next epoch settlement correctly credits the node with token rewards",
|
||||
"A wallet with strike count at the threshold is marked banned in the registry contract",
|
||||
"The tracker excludes banned wallets from route selection",
|
||||
"A banned wallet that attempts to register with the tracker is rejected",
|
||||
"An integration test covers: new wallet → N jobs → earning begins; and: strike threshold reached → banned → excluded from routes",
|
||||
"Follow TDD where code is added: add/adjust an observable behavior test before implementation when practical",
|
||||
"Run the task-specific tests and ensure they pass",
|
||||
"Run `python -m pytest` from repo root and ensure it passes, or document why no test suite exists yet",
|
||||
"Keep the implementation scoped to this story; do not implement downstream stories early",
|
||||
"Commit only this story's code changes with a focused conventional commit message"
|
||||
"An integration test covers: new wallet \u2192 N jobs \u2192 earning begins; and: strike threshold reached \u2192 banned \u2192 excluded from routes",
|
||||
"python -m pytest passes from repo root",
|
||||
"Commit only this story's changes"
|
||||
],
|
||||
"priority": 8,
|
||||
"passes": true,
|
||||
@@ -207,20 +198,17 @@
|
||||
},
|
||||
{
|
||||
"id": "US-009",
|
||||
"title": "09 — P2P shard swarm",
|
||||
"description": "Once a node has downloaded a shard, it seeds that shard to other nodes that are assigned the same shard. New nodes with the same shard assignment should download from the swarm (peer nodes) rather than HuggingFace when peers are available, reducing HuggingFace load and speeding up propagation of popular shards. The shard swarm is keyed by `(model_preset, shard_index)`. Nodes announce their shard availability to the tracker on registration. When a new node requests a shard download, the tracker returns a list of peer endpoints that hold the shard alongside the HuggingFace fallback URL. The node tries peers first; falls back to HuggingFace if no peers respond within a timeout. Peers serve shard data over a simple chunked HTTP endpoint — no custom P2P protocol needed in v1. The node client displays download source (peer or HuggingFace) during shard download.",
|
||||
"title": "09 \u2014 P2P shard swarm",
|
||||
"description": "Once a node has downloaded a shard, it seeds that shard to other nodes that are assigned the same shard.",
|
||||
"acceptanceCriteria": [
|
||||
"A node that has downloaded a shard is listed as a peer for that shard in the tracker",
|
||||
"A second node assigned the same shard downloads it from the first node (peer) rather than HuggingFace (verified by asserting no HuggingFace HTTP calls are made in the test)",
|
||||
"A second node assigned the same shard downloads it from the first node (peer) rather than HuggingFace",
|
||||
"If no peers are available, the node falls back to HuggingFace without error",
|
||||
"The node client logs whether the shard was downloaded from a peer or HuggingFace",
|
||||
"Shard chunks are verified against a checksum before being marked complete",
|
||||
"An integration test: node A downloads shard from HuggingFace stub; node B with same assignment downloads from node A",
|
||||
"Follow TDD where code is added: add/adjust an observable behavior test before implementation when practical",
|
||||
"Run the task-specific tests and ensure they pass",
|
||||
"Run `python -m pytest` from repo root and ensure it passes, or document why no test suite exists yet",
|
||||
"Keep the implementation scoped to this story; do not implement downstream stories early",
|
||||
"Commit only this story's code changes with a focused conventional commit message"
|
||||
"python -m pytest passes from repo root",
|
||||
"Commit only this story's changes"
|
||||
],
|
||||
"priority": 9,
|
||||
"passes": true,
|
||||
@@ -232,23 +220,20 @@
|
||||
},
|
||||
{
|
||||
"id": "US-010",
|
||||
"title": "10 — `meshnet` Python SDK",
|
||||
"description": "A Python SDK (`packages/sdk`) that wraps the OpenAI-compatible gateway and exposes network-specific controls that the raw OpenAI SDK cannot express. The SDK is the recommended interface for developers who want more than basic inference. The SDK surface: ```python from meshnet import Client client = Client(api_key=\"...\", base_url=\"https://api.meshnet.ai\") # Basic inference (identical to OpenAI SDK) response = client.chat.completions.create(model=\"llama-3-70b\", messages=[...]) # Network-specific features client.wallet.balance() # SOL/USDC balance remaining client.wallet.top_up(amount_sol=1.0) # returns a Solana payment address + QR client.models.available() # list of routable model presets with shard coverage % client.request(redundancy=2) # route through 2 independent chains, return majority client.estimate_cost(model=\"llama-3-70b\", tokens=1000) # cost in SOL before sending ``` The SDK must not require clients to hold or know about our native token. All wallet operations are in SOL or USDC.",
|
||||
"title": "10 \u2014 `meshnet` Python SDK",
|
||||
"description": "A Python SDK (`packages/sdk`) that wraps the OpenAI-compatible gateway and exposes network-specific controls.",
|
||||
"acceptanceCriteria": [
|
||||
"`pip install meshnet` installs the SDK",
|
||||
"`Client.chat.completions.create(...)` works identically to the OpenAI SDK (same parameters, same response shape)",
|
||||
"`Client.chat.completions.create(...)` works identically to the OpenAI SDK",
|
||||
"`client.wallet.balance()` returns the current SOL/USDC balance for the API key",
|
||||
"`client.wallet.top_up()` returns a valid Solana payment address",
|
||||
"`client.models.available()` returns model presets with shard coverage percentage",
|
||||
"`client.estimate_cost(model, tokens)` returns a cost estimate in SOL before the request is sent",
|
||||
"`client.request(redundancy=2)` sends the same prompt to two independent inference routes and returns the majority response",
|
||||
"`client.estimate_cost(model, tokens)` returns a cost estimate in SOL",
|
||||
"`client.request(redundancy=2)` sends to two independent inference routes and returns majority response",
|
||||
"The SDK is typed (py.typed, full type stubs)",
|
||||
"An integration test covers each SDK method against a local gateway + tracker + stub nodes",
|
||||
"Follow TDD where code is added: add/adjust an observable behavior test before implementation when practical",
|
||||
"Run the task-specific tests and ensure they pass",
|
||||
"Run `python -m pytest` from repo root and ensure it passes, or document why no test suite exists yet",
|
||||
"Keep the implementation scoped to this story; do not implement downstream stories early",
|
||||
"Commit only this story's code changes with a focused conventional commit message"
|
||||
"python -m pytest passes from repo root",
|
||||
"Commit only this story's changes"
|
||||
],
|
||||
"priority": 10,
|
||||
"passes": true,
|
||||
@@ -258,9 +243,100 @@
|
||||
"US-006"
|
||||
],
|
||||
"completionNotes": "Completed by fresh Ralph/Codex session ef4eea0e; verified by pytest, compileall, editable SDK install, and diff check."
|
||||
},
|
||||
{
|
||||
"id": "US-011",
|
||||
"title": "11 \u2014 Binary activation wire format",
|
||||
"description": "Replace the base64 JSON activation payload with raw binary HTTP bodies, zstd compression, and chunked prefill (128 tokens/chunk). All nodes and the gateway must be migrated. Stub nodes continue to emit zeroed tensors, just in binary. This is a protocol prerequisite for US-012 (real model backend).",
|
||||
"acceptanceCriteria": [
|
||||
"Node /forward endpoint reads shape/dtype/session/chunk from HTTP headers; body is raw binary (optionally zstd-compressed)",
|
||||
"Node /forward response is raw binary with the same header set",
|
||||
"Gateway splits prompts > MESHNET_CHUNK_TOKENS (default 128) into sequential chunks sent through the pipeline",
|
||||
"Integration test: 512-token stub activation (4 chunks) through a two-node pipeline returns 4 valid binary chunk responses",
|
||||
"zstd Python package added as a dependency to packages/node and packages/gateway",
|
||||
"_make_stub_activations replaced with _make_stub_binary_activation(shape, dtype) -> bytes",
|
||||
"python -m pytest passes from repo root",
|
||||
"Commit only this story's changes"
|
||||
],
|
||||
"priority": 11,
|
||||
"passes": false,
|
||||
"notes": "Source issue: .scratch/distributed-inference-network/issues/11-binary-wire-format.md",
|
||||
"dependsOn": [
|
||||
"US-002"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "US-012",
|
||||
"title": "12 \u2014 Real PyTorch model backend",
|
||||
"description": "Replace stub node inference with actual transformers layer execution. Node loads HuggingFace SafeTensors model shard (model.model.layers[start:end]), runs real forward passes in bfloat16, handles head (embed_tokens) and tail (lm_head) responsibilities, and supports bitsandbytes NF4/INT8/bfloat16 quantization via --quantization flag. Test model: openai-community/gpt2.",
|
||||
"acceptanceCriteria": [
|
||||
"meshnet-node start --model-id openai-community/gpt2 --shard-start 0 --shard-end 6 loads the shard without error",
|
||||
"Head node (shard_start==0) loads tokenizer and embed_tokens",
|
||||
"Tail node (shard_end==total_layers) loads model.norm and lm_head",
|
||||
"Two-node GPT-2 integration test returns deterministic coherent text completion",
|
||||
"--quantization [bfloat16|int8|nf4] flag supported; bfloat16 default",
|
||||
"Node with insufficient VRAM prints clear error and exits",
|
||||
"transformers, bitsandbytes, safetensors, accelerate added as node dependencies",
|
||||
"Integration tests marked @pytest.mark.integration, skipped in CI without GPU",
|
||||
"python -m pytest passes from repo root",
|
||||
"Commit only this story's changes"
|
||||
],
|
||||
"priority": 12,
|
||||
"passes": false,
|
||||
"notes": "Source issue: .scratch/distributed-inference-network/issues/12-real-pytorch-model-backend.md",
|
||||
"dependsOn": [
|
||||
"US-011"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "US-013",
|
||||
"title": "13 \u2014 Coverage-first tracker shard assignment",
|
||||
"description": "Upgrade tracker route selection to coverage-first, speed-weighted bin-packing. Tracker maintains a live coverage map per model, issues LOAD_SHARD/DROP_SHARD rebalance directives when coverage drops, and assigns shard ranges using declared VRAM, quantization, and benchmark throughput. A model is only routable when all layer ranges have node_count >= 1.",
|
||||
"acceptanceCriteria": [
|
||||
"Node registration accepts vram_bytes, ram_bytes, quantizations[], benchmark_tokens_per_sec",
|
||||
"GET /v1/coverage/<model_preset> returns list of {start_layer, end_layer, node_count}",
|
||||
"Model is unroutable when any layer range has node_count=0",
|
||||
"New node gets assigned to highest-priority uncovered range first",
|
||||
"Node disconnection triggers LOAD_SHARD directive to idle node within 30 seconds",
|
||||
"Shard assignment respects VRAM: assigned_layers <= floor((vram_bytes * 0.8) / bytes_per_layer_at_quant)",
|
||||
"Faster node receives wider shard range when both can cover the same gap",
|
||||
"Integration test: three nodes with different VRAM show widest range on largest-VRAM node with 100% coverage",
|
||||
"Integration test: kill middle-range node \u2192 tracker issues LOAD_SHARD \u2192 coverage recovers",
|
||||
"python -m pytest passes from repo root",
|
||||
"Commit only this story's changes"
|
||||
],
|
||||
"priority": 13,
|
||||
"passes": false,
|
||||
"notes": "Source issue: .scratch/distributed-inference-network/issues/13-coverage-first-shard-assignment.md",
|
||||
"dependsOn": [
|
||||
"US-003",
|
||||
"US-012"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "US-014",
|
||||
"title": "14 \u2014 Tracker-as-first-layer-node (inference entry point)",
|
||||
"description": "Merge inference orchestration into tracker nodes that serve the first-layer shard. A tracker node exposes /v1/chat/completions directly: tokenizes input, runs embed_tokens + layers[0..k], selects onward route from coverage map, forwards binary activations, receives tail output, and streams tokens back. The standalone gateway becomes a thin load-balancer routing to tracker nodes.",
|
||||
"acceptanceCriteria": [
|
||||
"Node with shard_start==0 (or --tracker-mode flag) exposes /v1/chat/completions alongside /forward",
|
||||
"Tracker-node /v1/chat/completions: tokenize \u2192 embed \u2192 own layers \u2192 route selection \u2192 forward binary \u2192 receive tail \u2192 stream SSE",
|
||||
"Two tracker nodes for same model each handle requests independently",
|
||||
"Gateway proxies /v1/chat/completions to tracker nodes (round-robin), discovered via GET /v1/tracker-nodes/<model_preset>",
|
||||
"Integration test: two tracker nodes + two mid-shard nodes for GPT-2; 10 requests via gateway; both tracker nodes receive load; all responses valid",
|
||||
"Existing US-005 OpenAI-compatible tests still pass",
|
||||
"python -m pytest passes from repo root",
|
||||
"Commit only this story's changes"
|
||||
],
|
||||
"priority": 14,
|
||||
"passes": false,
|
||||
"notes": "Source issue: .scratch/distributed-inference-network/issues/14-tracker-as-node.md",
|
||||
"dependsOn": [
|
||||
"US-012",
|
||||
"US-013"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"updatedAt": "2026-06-28T22:37:54.858Z"
|
||||
"updatedAt": "2026-06-29T00:00:00.000Z"
|
||||
}
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user