documentation revision

docs/PRD.md

@@ -1,4 +1,4 @@
-Status: ready-for-agent
+Status: done (base program US-001…US-035 complete; see `docs/prd.json`. Post-035 work lives in `docs/issues/36+` and `.scratch/`. Payment/settlement superseded by ADR-0015; fraud by ADR-0018.)
 
 # Distributed Inference Network — PRD
 
@@ -8,7 +8,7 @@ Running large language models requires expensive dedicated hardware that most pe
 
 ## Solution
 
-A volunteer GPU network where anyone can share their GPU by running a single command and immediately start earning tokens. Nodes each load a shard of a large model; a tracker routes inference requests through the optimal chain of nodes whose shards collectively cover all layers. Developers access the network through an OpenAI-compatible API — a one-line change from any existing LLM integration. Clients pay in SOL or USDC; node operators earn our native token. Everything is auto-configured: GPU detection, shard download, wallet creation, and network registration happen automatically on first start.
+A volunteer GPU network where anyone can share their GPU by running a single command and immediately start earning tokens. Nodes each load a shard of a large model; a tracker routes inference requests through the optimal chain of nodes whose shards collectively cover all layers. Developers access the network through an OpenAI-compatible API — a one-line change from any existing LLM integration. Clients pay in **USDT** (alpha: devnet mock-USDT; production: mainnet USDT). Node operators earn USDT payouts from the custodial treasury (ADR-0015); the TAI reward token (ADR-0002) remains deferred. Everything is auto-configured: GPU detection, shard download, wallet creation, and network registration happen automatically on first start.
 
 ## User Stories
 
@@ -76,8 +76,8 @@ The codebase is organized as a Python monorepo with the following top-level pack
 - `packages/contracts` — Solana L2 smart contracts (stake, slash, strike, ban, settlement)
 - `packages/p2p` — P2P gossip layer and shard swarm seeding
 
-### Inference engine (ADR-0001)
-PyTorch with a Petals-style shard pipeline. Each node independently loads its assigned shard from local disk. At inference time, only activation tensors (~8 KB per layer boundary per token) travel between nodes — no model weights cross the network during serving.
+### Inference engine (ADR-0001; native GGUF path ADR-0024)
+PyTorch with a Petals-style shard pipeline remains the current production backend. A benchmark-gated llama.cpp/GGUF native path is planned in ADR-0024. Each node independently loads its assigned shard from local disk. At inference time, only activation tensors (~8 KB per layer boundary per token) travel between nodes — no model weights cross the network during serving.
 
 ### Inference route execution
 The gateway receives a client request, asks the tracker for an inference route (ordered list of node endpoints covering all layers), opens a persistent TCP session to the first node in the route, streams activation tensors through each node in sequence, and returns the final logits as a streaming chat completion response.
@@ -91,11 +91,11 @@ The gateway receives a client request, asks the tracker for an inference route (
 6. Register with tracker (wallet, hardware profile, shard, endpoint)
 7. Begin accepting inference connections
 
-### Payment flow
-Clients pre-fund an API key with SOL/USDC. The gateway records per-request compute attribution. A settlement transaction runs on Solana L2 at the end of each epoch: client balance is debited, node operators receive our native token proportional to layers served, validators receive a reward share. Solana contracts are the authoritative source for all stake, slash, strike, and ban state (ADR-0002).
+### Payment flow (ADR-0015 supersedes ADR-0002 settlement mechanics)
+Clients pre-fund an API key with USDT. The tracker meters each request against the off-chain ledger. Periodic settlement batches USDT payouts from the custodial treasury to node operators proportional to work units. Fraud penalties forfeit pending balance (ADR-0018); strike/ban state persists in the tracker registry. TAI token emission remains deferred (ADR-0002 roadmap).
 
-### Fraud detection (ADR-0003)
-Validators re-run ~5% of completed requests. If a node's output diverges beyond floating-point tolerance from the reference, the validator submits a slash transaction on-chain. Strike count increments. At the configured strike threshold, the wallet is banned on-chain. New wallets complete N unpaid jobs before earning begins.
+### Fraud detection (ADR-0018; historical ADR-0003)
+Validators re-run ~5% of completed requests with TOPLOC activation verification. Caught cheaters forfeit pending balance and receive strikes; three strikes bans the wallet. Probation (first N unpaid jobs) remains the anti-sybil re-entry cost.
 
 ### Tracker architecture (ADR-0004)
 Centralized tracker service (HTTP + WebSocket) for fast routing. Nodes gossip state via a lightweight P2P layer so the node client can discover routes during tracker outages. Solana is the authoritative source of truth for all incentive-relevant state.

docs/adr/0001-pytorch-over-llama-cpp.md

@@ -1,5 +1,7 @@
 # PyTorch over llama.cpp for the inference engine
 
+> **Runtime direction update (2026-07-13):** PyTorch/safetensors remains the current production backend and correctness reference. A benchmark-gated native GGUF path is defined in [ADR-0024](0024-distributed-gguf-runtime.md); it does not replace this ADR until release gates pass.
+
 We started with llama.cpp RPC as the distributed backend (following kyuz0/amd-strix-halo-toolboxes), but switched to PyTorch with a Petals-style shard pipeline. llama.cpp RPC requires the primary node to load the full model and distribute weights over the network at every session start — for a 70B model that's ~70GB over LAN per launch, making tracker-driven node rebalancing prohibitively expensive. PyTorch/Petals lets each node load its shard independently from local disk; only activations (~8KB per layer boundary per token) cross the network at inference time. PyTorch also has same-day support for new model architectures, training support (required for the planned torrent-style fine-tuning feature), and is the engine Petals itself uses for this exact use case.
 
 ## Considered Options

docs/adr/0003-optimistic-fraud-detection.md

@@ -1,5 +1,7 @@
 # Optimistic trust with stake slashing and strike-based bans
 
+> **Settlement update (2026-07-04):** Alpha uses pending-balance forfeiture instead of stake slashing ([ADR-0015](0015-usdt-custodial-settlement.md)). Fraud detection, TOPLOC audits, and persisted reputation are specified in [ADR-0018](0018-fraud-detection-verification-and-reputation.md). The text below is the historical prototype design.
+
 All inference responses are trusted by default. Validators re-run a random sample (~5%) of requests on reference nodes and compare outputs. Nodes that fail are slashed (stake reduced). Enough strikes result in a permanent on-chain ban.
 
 For the prototype, the gateway emits validation events after completed requests. A validation event records the session id, model preset, request messages, observed output, and the route metadata for each node that served the request. The validator samples events with a configurable rate and deterministic seed for tests. Sampled events are re-run against a trusted reference node/reference function; string outputs must match exactly for stub models, while future tensor/model outputs use a configurable floating-point tolerance.

docs/adr/0020-chat-streaming-live-progress-and-mixed-topology-routing.md

@@ -1,6 +1,6 @@
 # ADR-0020: Dashboard chat streaming, live request progress, and the mixed-topology routing flaw
 
-## Status: Accepted (chat/streaming/styles implemented); routing flaw documented, fix pending
+## Status: Accepted (chat/streaming/styles and mixed-topology routing fix implemented)
 
 ## Context
 
@@ -94,7 +94,7 @@ head + full-model downstream is a topology the planner never had to handle befor
 prior split tests used disjoint shards (0–11 + 12–23) where `shard_start` happened to
 equal the correct continuation layer.
 
-### Required fix (not yet implemented)
+### Required fix (implemented 2026-07-07 — commits `518c259`, `e44abc9`, `1ecc599`; see ADR-0021)
 
 1. **Correct continuation layer:** when hop N ends at layer `e`, hop N+1 must execute
    from `start_layer = e + 1` regardless of the downstream node's own `shard_start`

docs/adr/0022-sharded-per-node-generation-cache.md

@@ -1,6 +1,8 @@
 # ADR-0022: Sharded per-node generation cache for distributed PyTorch routes
 
-## Status: Accepted
+## Status: Superseded — see [0022-sharded-per-node-kv-cache.md](0022-sharded-per-node-kv-cache.md)
+
+> Draft alternate header names (`X-Meshnet-Cache-Mode`, `X-Meshnet-Seq-Len`) were not implemented. The accepted wire protocol and implementation use `X-Meshnet-Cache` and `X-Meshnet-Past-Len` per the linked ADR.
 
 ## Context
 

docs/adr/0024-distributed-gguf-runtime.md

@@ -0,0 +1,125 @@
+# ADR-0024: Lean Native Distributed GGUF Runtime
+
+Status: Accepted
+Date: 2026-07-13
+
+> **Numbering note:** ADR-0020 is reserved for dashboard chat streaming and mixed-topology routing (`docs/adr/0020-chat-streaming-live-progress-and-mixed-topology-routing.md`). This record was originally drafted as ADR-0020 in `.scratch/distributed-gguf-runtime/` and renumbered to avoid the collision.
+>
+> **Relation to ADR-0001:** PyTorch/safetensors remains the correctness reference and current production backend. This ADR defines a benchmark-gated native GGUF path; it does not revoke ADR-0001 until release gates pass.
+
+## Context
+
+The project currently uses Transformers/safetensors as its real model execution backend. This provides broad architecture coverage and a correctness reference, but reported and observed consumer CPU/GPU inference performance motivates evaluating llama.cpp/GGML and quantized GGUF.
+
+The product objective is not merely local GGUF serving. It is performant concurrent inference for top open models whose weights do not fit on one consumer node. The project already owns the Tracker, Inference Route, Route Session, Activation Seam, local Hot KV State, relay/direct transport, cancellation, telemetry, billing, and capability admission.
+
+Research audited llama.cpp RPC, GPUStack/llama-box, Nakshatra, prima.cpp, llama-gguf, LiGGUF, vLLM and its GGUF plugin, Petals, exo, and related projects. No repository provides the complete public-network contract. llama.cpp is the strongest GGUF execution substrate. vLLM has mature managed-cluster parallelism and scheduling concepts but its PP/TP/EP runtime assumes a static trusted distributed world and is unsuitable as the public Shard runtime.
+
+The project must remain lean and avoid combining several half-integrated inference control planes.
+
+## Decision
+
+### Primary native runtime
+
+Use llama.cpp/GGML through one standalone C++ Shard worker and a small exact-commit patch stack.
+
+The patch scope is limited to:
+
+- Range-aware GGUF tensor ownership/loading.
+- Architecture-defined intermediate boundary input/output.
+- Intermediate output before tail normalization/head.
+- Layer-filtered KV and external session-to-sequence mapping.
+
+Meshnet networking, routing, admission, billing, telemetry, and work evidence stay outside llama.cpp.
+
+Nakshatra, prima.cpp, llama-gguf, LiGGUF, and historical GPUStack are source/test donors only. Their repositories are not runtime dependencies.
+
+### Distributed parallelism
+
+The first public-network primitive is layer/pipeline parallelism through contiguous Shards in an Inference Route.
+
+Per-node continuous batching combines decode steps from compatible active Route Sessions. Multiple complete routes provide data parallelism.
+
+Tensor and expert parallel collectives may later operate inside one trusted composite node or managed cluster represented as one provider. They are not public WAN routing primitives.
+
+### Transport
+
+Use gRPC over HTTP/2 with Protocol Buffers for the native Python/C++ Shard data plane.
+
+- One long-lived bidirectional stream per Route Session Activation Seam.
+- Deadlines, cancellation, flow control, TLS/authentication hooks, structured status, and generated schemas.
+- Bounded chunks for prefill and a small decode fast path.
+- Existing relay infrastructure may carry the same versioned protobuf frames as opaque binary when direct connectivity is unavailable.
+- OpenAI client APIs remain HTTP/SSE; existing Tracker APIs remain unchanged.
+
+The boundary payload is a versioned named-tensor bundle because architecture boundaries may require more than one tensor.
+
+### vLLM
+
+Do not fork vLLM for public distributed Shards and do not transplant PagedAttention, Torch process groups, or the vLLM GGUF plugin into the llama.cpp worker.
+
+Allow unmodified vLLM as an optional whole-model backend or managed TP/PP/EP cluster represented as one logical provider.
+
+Adapt only small control-plane concepts:
+
+- Named intermediate bundles.
+- Continuous batching and request ownership.
+- Versioned cache-transfer compatibility fingerprints.
+- Explicit transfer failure/abort lifecycle.
+- Load telemetry and fair tie-breaking.
+
+### Benchmark gate
+
+GGUF performance is a hypothesis. Before expensive native work, compare the current Transformers/safetensors recipe with whole-model llama.cpp on controlled model, hardware, prompt, context, output, sampling, concurrency, memory, and quality lanes.
+
+Later distributed release gates use thresholds locked before implementation results are known. The native track stops if llama.cpp/GGUF offers neither a meaningful performance benefit nor a meaningful model-fit benefit at useful speed.
+
+### Concurrency
+
+A native worker must isolate `(Route Session ID, route epoch)` through a llama sequence or bounded context and must not serialize all generations behind one global serving sequence.
+
+The node admits sessions against weight/KV/scratch budgets, batches compatible decode steps, prevents prefill starvation, applies backpressure, and exposes queue/batch/KV telemetry.
+
+### Architecture certification
+
+Dense Llama-family is first. Qwen3/Qwen3-MoE is a separate explicit adapter. Every architecture/backend/recipe remains registered-but-dark until a real distributed forward, parity test, concurrency test, and capability admission pass.
+
+## Alternatives rejected
+
+### Fork vLLM for the public mesh
+
+Rejected because extracting its PP/TP/EP stages requires replacing static process groups, rank lifecycle, scheduler, request ownership, cache layout, failure behavior, and hardware assumptions. This would create a large difficult fork while discarding much of vLLM's core architecture.
+
+### llama.cpp RPC as the public protocol
+
+Rejected because it exposes coordinator-owned raw GGML devices, not independent Shards. Its trust, security, failure, cache, and per-node accounting model is unsuitable for arbitrary volunteer nodes.
+
+### Adopt Nakshatra or prima.cpp wholesale
+
+Rejected because their repositories, build reproducibility, session/concurrency semantics, architecture coverage, protocol identity, and control planes do not satisfy the project contract. Their partial-loading and boundary work remains valuable evidence.
+
+### Build a custom GGUF engine
+
+Rejected because llama.cpp already provides the parser, kernels, architecture graphs, KV, tokenizer, and heterogeneous backends. Reimplementing these would spread effort and increase correctness risk.
+
+### Invent a custom transport
+
+Rejected. gRPC/HTTP2 already provides mature streaming, flow control, deadlines, cancellation, TLS, and cross-language schema generation.
+
+## Consequences
+
+- The critical path contains Meshnet, one standalone worker, and one small pinned llama.cpp patch stack.
+- Transformers/safetensors remains the correctness reference and fallback for unsupported architectures.
+- Whole-model llama.cpp and vLLM managed clusters remain useful optional provider types.
+- The first milestone emphasizes controlled benchmark, parity, concurrent KV, and real two-machine evidence rather than a large-model demo.
+- Upstream collaboration with llama.cpp targets generic local hooks only; the project remains able to ship a narrow pinned fork if upstream acceptance takes time.
+- QUIC, public tensor parallelism, disaggregated prefill, speculative decode, route repair, and KV migration remain deferred until the core route passes release gates.
+
+## Verification gates
+
+1. Controlled safetensors-versus-GGUF performance contract.
+2. Two-process local range parity.
+3. Four-session concurrent KV isolation.
+4. Real two-machine execution using both Shards.
+5. End-to-end performance/fit advantage over the current distributed route.
+6. Separate Qwen3-family architecture certification.

docs/issues/01-monorepo-scaffold.md

@@ -1,4 +1,4 @@
-Status: ready-for-agent
+Status: done
 
 # 01 — Monorepo scaffold + single-node smoke test
 

docs/issues/02-two-node-shard-pipeline.md

@@ -1,4 +1,4 @@
-Status: ready-for-agent
+Status: done
 
 # 02 — Two-node shard pipeline
 

docs/issues/03-tracker-registration-and-routing.md

@@ -1,4 +1,4 @@
-Status: ready-for-agent
+Status: done
 
 # 03 — Tracker: node registration + route selection
 

docs/issues/04-node-client-startup.md

@@ -1,4 +1,4 @@
-Status: ready-for-agent
+Status: done
 
 # 04 — Node client startup flow (`meshnet-node start`)
 

docs/issues/05-openai-compatible-gateway.md

@@ -1,4 +1,4 @@
-Status: ready-for-agent
+Status: done
 
 # 05 — OpenAI-compatible gateway
 

docs/issues/08-probationary-period-and-bans.md

@@ -1,4 +1,4 @@
-Status: ready-for-agent
+Status: done (on-chain registry mechanics superseded — probation/ban enforcement uses tracker registry + ADR-0015/0018)
 
 # 08 — Node probationary period + ban enforcement
 

docs/issues/09-p2p-shard-swarm.md

@@ -1,4 +1,4 @@
-Status: ready-for-agent
+Status: done
 
 # 09 — P2P shard swarm
 

docs/issues/10-meshnet-sdk.md

@@ -1,4 +1,4 @@
-Status: ready-for-agent
+Status: done
 
 # 10 — `meshnet` Python SDK
 

docs/issues/19-binary-data-plane-and-peer-weight-transfer.md

@@ -1,8 +1,6 @@
 # US-019 — Binary data plane and optional peer weight transfer
 
-Status: needs-triage
-Priority: Low
-Stage: Design parking lot
+Status: done (design parking lot; binary activation path shipped in US-011/US-019)
 
 ## Context
 

docs/issues/36-relay-streamed-chat.md

@@ -1,8 +1,6 @@
 # US-036 — Streamed chat completions over the relay RPC path
 
-Status: planned
-Priority: Critical (blocks public friends-test deployment)
-Stage: Designed
+Status: done (implemented — `_stream_relayed_frames` in `server.py`; verify on public NAT relay before friends-test)
 
 ## Context
 

docs/issues/42-gguf-llamacpp-node-backend.md

@@ -1,7 +1,7 @@
 # US-042 — GGUF/llama.cpp node backend
 
 Status: planned
-Priority: High (unlocks big MoE models on volunteer hardware — the pool's core value)
+Priority: High (whole-model GGUF shortcut; distributed path in [ADR-0024](../adr/0024-distributed-gguf-runtime.md))
 Stage: Draft design
 
 ## Context

docs/issues/48-memory-budget-shard-slots-and-dropout-relocation.md

@@ -1,6 +1,8 @@
-Status: ready-for-agent
+Status: planned
 
-# US-020 - Memory budget, shard slots, and dropout relocation hardening
+# US-048 — Memory budget, shard slots, and dropout relocation hardening
+
+> Renumbered from duplicate slot `20` (which belongs to tracker-node-hardening / US-020 in `docs/prd.json`).
 
 ## Goal
 
@@ -64,3 +66,4 @@ The current runtime still effectively has one active backend shard per node. A n
 
 - 2026-06-30: Created after implementing the initial registration plumbing in commit `f1e4ed6` (`--memory`, `--max-shards`, tracker validation). This issue captures the remaining end-to-end behavior so it does not conflict with US-013.
 - 2026-06-30: Implementation decision: `max_loaded_shards` is currently a validated and exposed capacity field, but multi-range assignment remains reserved because `TorchNodeServer` serves one active backend shard. The tracker therefore emits at most one active range per node while exposing `vram_bytes`, `ram_bytes`, `max_loaded_shards`, quantization, throughput, and computed `max_assignable_layers` in inspection endpoints.
+- 2026-07-13: Renumbered from `docs/issues/20-memory-budget-…` to resolve duplicate issue slot 20.