Add alpha-hardening ADRs and issue plan from pre-release audit.
Lock alpha scope, tracker auth, TOPLOC fraud verification, and deferred multi-tracker money-path work; supersede legacy fraud issues with ADR-0018. Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
@@ -1,5 +1,9 @@
|
||||
# TAI token: revenue-backed rewards for nodes, USDT/SOL payments for clients
|
||||
|
||||
## Status: Accepted (settlement mechanics superseded by ADR-0015)
|
||||
|
||||
> **Settlement update (2026-07-04):** Alpha and near-term production use **USDT-direct custodial settlement** with pending-balance forfeiture penalties (ADR-0015). TAI emission, backing-price buyback, and stake deposits described below remain the **roadmap tokenomics** — not the live payment path. When TAI ships, it will mint from accumulated protocol cut; this ADR's incentive structure still governs long-term design.
|
||||
|
||||
## Core principle
|
||||
|
||||
Nodes must get paid reliably for inference work. This is the primary growth engine. All tokenomics decisions serve this goal first. The model is intentionally simple and sound — parameters can be tweaked later, but the structure must have no fatal flaws from day one.
|
||||
|
||||
@@ -16,10 +16,12 @@ The reference implementation (Petals) solves this with libp2p — GossipSub for
|
||||
|
||||
### 1. TLS everywhere
|
||||
|
||||
All HTTP between nodes, tracker, and gateway uses HTTPS (TLS 1.3). Self-signed certificates are auto-generated on first node start and stored in `~/.config/meshnet/`. The certificate fingerprint is included in every heartbeat and gossip envelope. Nodes use TOFU (trust on first use) — they accept a peer's cert on first contact and pin the fingerprint; connections from the same peer with a different fingerprint are rejected.
|
||||
**Intent:** All HTTP between nodes, tracker, and gateway uses HTTPS (TLS 1.3). Self-signed certificates are auto-generated on first node start and stored in `~/.config/meshnet/`. The certificate fingerprint is included in every heartbeat and gossip envelope. Nodes use TOFU (trust on first use) — they accept a peer's cert on first contact and pin the fingerprint; connections from the same peer with a different fingerprint are rejected.
|
||||
|
||||
The relay node uses a real CA-signed certificate (Let's Encrypt) because it is the internet-facing bootstrap point.
|
||||
|
||||
> **Alpha reality correction (2026-07-04, ADR-0016):** As implemented, **TLS applies to the relay path** (`packages/relay`, WSS circuit proxy) and P2P packages that generate certs — **not** to plain HTTP on tracker ↔ node inference and registration in typical LAN/dev deployments. Tracker registration already requires `https://` URLs for production trackers, but node-to-tracker proxy traffic may still be plaintext on alpha. Full end-to-end TLS remains the target; alpha documents this as a **known limitation**, not a silent guarantee. Hive gossip auth (ADR-0017) is the immediate mitigation for tracker replication integrity.
|
||||
|
||||
### 2. mDNS for LAN peer discovery
|
||||
|
||||
Python `zeroconf` library. Service type: `_meshnet._tcp.local.`. A node announces itself on startup and browses for existing peers. This is zero-config discovery for home and lab networks. mDNS does not traverse routers, which is correct — LAN discovery should not bleed into the internet.
|
||||
|
||||
52
docs/adr/0016-alpha-scope-and-known-limitations.md
Normal file
52
docs/adr/0016-alpha-scope-and-known-limitations.md
Normal file
@@ -0,0 +1,52 @@
|
||||
# ADR-0016: Alpha scope and known limitations
|
||||
|
||||
## Status: Accepted
|
||||
|
||||
## Context
|
||||
|
||||
Pre-release audits found the routing layer is solid (ADR-0013) but the money and trust paths are not alpha-ready: unauthenticated gossip, free starting credit, node-reported accounting, ephemeral strike/ban state, and plaintext tracker HTTP undermine release. The owner locked an **alpha scope** that ships real inference and devnet settlement while explicitly deferring multi-tracker trustlessness and cryptographic verification.
|
||||
|
||||
## Decisions
|
||||
|
||||
### 1. Single settlement tracker
|
||||
|
||||
One operator-designated tracker holds the treasury keypair and runs the settlement loop (ADR-0015). Third-party trackers may join the hive for routing and state replication but **never** sign payouts. Raft elects a leader for shard-assignment commands only; settlement leadership is operator-configured, not democratic.
|
||||
|
||||
### 2. Open node join
|
||||
|
||||
Any wallet may register a node without invite, stake deposit, or hardware attestation. Anti-sybil cost comes from probation (first N jobs unpaid), carried-forward reputation, and fraud penalties — not from closed admission.
|
||||
|
||||
### 3. Devnet mock-USDT
|
||||
|
||||
Development and alpha target **Solana devnet** with a self-created mock-USDT SPL mint (ADR-0015). Mainnet USDT is a config change, not an architecture change. TAI settlement (ADR-0002) remains deferred; the 10% protocol cut accumulates as future TAI liquidity.
|
||||
|
||||
### 4. Reputation carries forward
|
||||
|
||||
Strike count, ban status, graduated reputation, and audit history **persist across tracker restarts** and are intended to survive alpha → mainnet cutover. A banned operator cannot evade penalties by restarting the tracker or re-registering under a fresh node id with the same wallet.
|
||||
|
||||
### 5. Known alpha limitations (explicit, not bugs)
|
||||
|
||||
| Limitation | Alpha posture | Post-alpha path |
|
||||
|---|---|---|
|
||||
| Tracker is trusted referee for audits and blame | Acceptable — see ADR-0018 | Decentralized verifier market; Verde-style on-chain games |
|
||||
| Plaintext HTTP on tracker/node paths | Acceptable on LAN/dev; relay TLS only (ADR-0010 amended) | Full TLS + cert pinning per ADR-0010 intent |
|
||||
| Unauthenticated gossip replication | **Blocker** — fix before alpha (ADR-0017) | HMAC/mTLS between hive members |
|
||||
| zkML / GPU TEE verification | Roadmap-only | ADR-0018 §9; re-evaluate yearly |
|
||||
| Multi-tracker money-path safety | Design now (ADR-0019), implement later | Consensus-gated settlement, idempotent payouts |
|
||||
| Bitwise output equality across GPUs | Not required — logit/activation tolerance (TOPLOC) | Optional Gensyn RepOps tier for referee |
|
||||
|
||||
### 6. Fraud must be bounded
|
||||
|
||||
Because reputation carries forward, correctness fraud (bad outputs) and accounting fraud (inflated tokens/shard spans) must be **economically irrational** at the locked ~5% audit budget. Detection, blame, and penalty design are ADR-0018; implementation is Bucket 1 in `.scratch/alpha-hardening/`.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Alpha release criteria are Bucket 1 blockers in `.scratch/alpha-hardening/README.md`, not "all 35 user stories done."
|
||||
- Documentation and runbooks must state trust assumptions plainly — custodial treasury, trusted tracker, devnet funds.
|
||||
- Bucket 2 (multi-tracker) issues are tracked but not alpha-blocking.
|
||||
|
||||
## Related
|
||||
|
||||
- Supersedes the spirit of prototype-only trust in ADR-0003 for production alpha behavior — see ADR-0018.
|
||||
- Settlement mechanics: ADR-0015.
|
||||
- Research grounding: `.scratch/alpha-hardening/research-verifiable-inference.md`.
|
||||
75
docs/adr/0017-tracker-authentication-and-authorization.md
Normal file
75
docs/adr/0017-tracker-authentication-and-authorization.md
Normal file
@@ -0,0 +1,75 @@
|
||||
# ADR-0017: Tracker authentication and authorization
|
||||
|
||||
## Status: Accepted
|
||||
|
||||
## Context
|
||||
|
||||
The tracker exposes three overlapping trust domains:
|
||||
|
||||
1. **Client API access** — Bearer API keys for inference and billing (partially gated today).
|
||||
2. **Operator/admin access** — session-based accounts in `packages/tracker/meshnet_tracker/accounts.py` (registration, login, admin listing).
|
||||
3. **Hive replication** — gossip endpoints that mutate billing, accounts, and stats without authentication today.
|
||||
|
||||
Financial and registry endpoints leak through inconsistent checks: some handlers require only a non-empty `Authorization` header (header-presence stub), while gossip accepts arbitrary peer events. Alpha requires a **single auth boundary** before any state mutation or privileged read.
|
||||
|
||||
## Decisions
|
||||
|
||||
### 1. Account subsystem (`accounts.py`)
|
||||
|
||||
- First registered account is **admin**; subsequent accounts are `user`.
|
||||
- Login identifiers: email **or** wallet address + PBKDF2-SHA256 password.
|
||||
- API keys (`sk-mesh-…`) are scoped to an account; revocation is event-sourced and gossip-replicated.
|
||||
- **Sessions are local** (bearer tokens in memory, 7-day TTL) — not replicated across the hive. Each tracker validates its own sessions.
|
||||
- Account mutations persist to SQLite and replicate via `/v1/accounts/gossip` events.
|
||||
|
||||
### 2. Client inference auth
|
||||
|
||||
- `POST /v1/chat/completions` requires a valid, non-revoked API key and positive ledger balance (402 if broke).
|
||||
- **Starting credit is 0** for new API keys (ADR-0016 / issue C5). No implicit faucet; clients must deposit mock-USDT via bound wallet (US-032) or admin credit.
|
||||
|
||||
### 3. Gossip auth (alpha blocker)
|
||||
|
||||
All hive mutation endpoints require **authenticated peer identity**:
|
||||
|
||||
- `/v1/billing/gossip` — billing event replication
|
||||
- `/v1/accounts/gossip` — account/key replication
|
||||
- `/v1/stats/gossip` — rolling RPM merge
|
||||
|
||||
Acceptable alpha mechanisms (pick one in implementation):
|
||||
|
||||
- **Shared hive secret** (HMAC over request body + timestamp, configured on all trackers), or
|
||||
- **Mutual TLS** between tracker peers (extends ADR-0010 relay TLS pattern).
|
||||
|
||||
Followers must reject events from unauthenticated callers. Read-only endpoints (`/dashboard`, `/v1/stats` GET) may remain public on LAN alpha deployments.
|
||||
|
||||
**Out of scope (alpha):** `POST /v1/gossip` — node throughput / peer fan-out gossip (`server.py` ~1331), distinct from hive mutation endpoints above. Document as **unauthenticated alpha limitation**; authenticate in a future ADR amendment when node identity binding exists (issue 01 tracks hive endpoints only).
|
||||
|
||||
### 4. Privileged operator endpoints
|
||||
|
||||
These require **real** auth — not header presence alone:
|
||||
|
||||
| Endpoint | Required role |
|
||||
|---|---|
|
||||
| `POST /v1/billing/forfeit` | Validator service identity or admin session |
|
||||
| `POST /v1/benchmark/hop-penalty` | Admin session or service token |
|
||||
| `GET /v1/benchmark/results` | Admin session or service token |
|
||||
| `GET /v1/admin/accounts` | Admin session |
|
||||
| `GET /v1/billing/summary`, `/v1/billing/settlements`, `/v1/registry/wallets` | Admin session (alpha: restrict financial reads) |
|
||||
|
||||
The unified auth middleware resolves: API key → account; session token → account + role; service token → validator; hive HMAC → peer.
|
||||
|
||||
### 5. Node registration
|
||||
|
||||
Node `POST /v1/register` is open (ADR-0016) but **banned wallets are rejected** at registration and excluded from routing. Wallet binding for client deposits (`POST /v1/wallet/register`) requires a valid API key and cryptographic ownership proof (issue C6).
|
||||
|
||||
## Consequences
|
||||
|
||||
- Gossip authentication (C1) and unified auth boundary (A2) are prerequisites for fraud penalties and billing fixes — an attacker can otherwise forge credits, strikes, or forfeit events.
|
||||
- Session locality means admin login is per-tracker URL; document in runbooks.
|
||||
- Multi-tracker auth evolution (client-signed commands) is deferred to ADR-0019.
|
||||
|
||||
## Related
|
||||
|
||||
- ADR-0016 (alpha scope)
|
||||
- ADR-0015 (billing gate on inference)
|
||||
- `.scratch/alpha-hardening/issues/01-c1-gossip-auth.md`, `02-a2-unified-auth-boundary.md`
|
||||
92
docs/adr/0018-fraud-detection-verification-and-reputation.md
Normal file
92
docs/adr/0018-fraud-detection-verification-and-reputation.md
Normal file
@@ -0,0 +1,92 @@
|
||||
# ADR-0018: Fraud detection, verification, and reputation
|
||||
|
||||
## Status: Accepted
|
||||
|
||||
## Context
|
||||
|
||||
ADR-0003 established optimistic sampling with stake slashing; ADR-0015 replaced stake with **pending-balance forfeiture** as collateral. Pre-alpha audits identified two distinct fraud types:
|
||||
|
||||
1. **Correctness fraud** — wrong model, layer-skipping, garbage outputs.
|
||||
2. **Accounting fraud** — inflated token counts or shard-span work units reported by nodes.
|
||||
|
||||
The validator today compares final text only and always blames the last hop (`_final_text_node` in `packages/validator/meshnet_validator/__init__.py` ~137–140) — wrong for multi-hop pipelines.
|
||||
|
||||
Research (`.scratch/alpha-hardening/research-verifiable-inference.md`, 2026-07-04) grounds the alpha design in deployed patterns (Prime Intellect TOPLOC, Hyperbolic PoSP, Gensyn Verde blame **patterns**). This ADR is the flagship decision record for alpha hardening.
|
||||
|
||||
## Decisions
|
||||
|
||||
### 1. Anchor technique: optimistic accept + teacher-forced audit
|
||||
|
||||
- Default audit probability **p ≈ 5%** — a **budget target**, not a hard cap. Anomalies, low reputation, and disputes escalate rate; veterans floor at ≥ 2% (research §6, §8).
|
||||
- **Deterrence condition:** at p = 0.05, penalty L must exceed **L > g·(1−p)/p ≈ 19×** per-job gain g (research §1.1). **Full pending forfeiture** is the primary penalty; three strikes → ban. The ×0.8-per-strike multiplier applies to **routing/payout weight** (reputation decay), not to the forfeiture amount.
|
||||
- Single-tracker alpha: the tracker (or a designated reference node) is the auditor — no verifier market, no verifier's dilemma (research §1.1).
|
||||
|
||||
### 2. Detection primitive: ADOPT TOPLOC
|
||||
|
||||
- **`pip install toploc`** (MIT, [PrimeIntellect-ai/toploc](https://github.com/PrimeIntellect-ai/toploc)) for activation fingerprint commit + verify (research build-vs-adopt table).
|
||||
- Teacher-forced prefill re-verification — compare in **logit/activation space with tolerances**, never free-running token equality (research §2).
|
||||
- Pin **one canonical precision/quantization** per served model; TOPLOC detects precision drift by design.
|
||||
- Per-hop boundary fingerprints extend TOPLOC's final-hidden-state encoding for **multi-hop blame** (research §1.2, §8 layer 1).
|
||||
|
||||
### 3. Commit layer: on-demand activation commitments
|
||||
|
||||
- Nodes commit compact TOPLOC-style fingerprints of **output boundary activations** per hop when selected for audit (on-demand, not every request — lower serving latency; brief retention window for recent activations).
|
||||
- Commitments are **audit pins, not proofs** — correctness requires independent recomputation (research §4).
|
||||
|
||||
### 4. Blame layer: hop-boundary bisection (adapt Verde pattern)
|
||||
|
||||
On audit failure:
|
||||
|
||||
1. Referee (tracker) teacher-forces claimed token sequence through reference model.
|
||||
2. Compare committed hop-boundary fingerprints to reference at each cut-point.
|
||||
3. **First divergent hop** is the culprit — fixes `_final_text_node` last-hop-only bug.
|
||||
4. Full interactive Truebit/Verde on-chain game and bitwise RepOps kernels: **roadmap-only** (research §1.2, §9).
|
||||
|
||||
### 5. Accounting fraud: tracker-authoritative metering
|
||||
|
||||
- **Token counts** come from the tracker's proxied stream/non-stream response parsing (`server.py` ~1890–1943), not node self-reports.
|
||||
- **Work units** derive from **tracker-assigned shard span** at route construction (`server.py` ~1776–1782), not node-declared ranges at billing time.
|
||||
- See issue H2.
|
||||
|
||||
### 6. Reputation model (graduated, persisted)
|
||||
|
||||
Reputation derives **only from tracker-verified audit outcomes** + uptime/latency — never peer ratings (research §6, collusion surface).
|
||||
|
||||
| Signal | Effect |
|
||||
|---|---|
|
||||
| Clean audits | Slow reputation build; higher routing weight |
|
||||
| Strike | ×0.8 routing multiplier per strike (graduated decay) |
|
||||
| Failed audit | Full pending forfeiture + strike; audit rate → maximum |
|
||||
| Ban (3 strikes) | Registration rejected; excluded from routes; pending never paid |
|
||||
| New/low reputation | Elevated audit rate (20–30% target for newcomers) |
|
||||
| Inactivity | Reputation decay |
|
||||
|
||||
Persist strike/ban/reputation in SQLite alongside billing (issue A1/A5). Probation (first N jobs unpaid) retained as re-entry cost.
|
||||
|
||||
### 7. Passive tripwires
|
||||
|
||||
Perplexity/repetition/truncation heuristics on all traffic raise audit rate without direct punishment (research §8 layer 5).
|
||||
|
||||
### 8. Roadmap-only (explicitly NOT alpha)
|
||||
|
||||
- zkML proofs of LLM inference (research §1.3)
|
||||
- GPU TEE attestation on consumer cards (research §1.4)
|
||||
- Fully trustless Verde interactive games + RepOps bitwise kernels (research §9)
|
||||
- Decentralized verifier markets
|
||||
- Peer-rating reputation (EigenTrust)
|
||||
- PoW as correctness proof — registration-time hardware attestation only, optional (research §4)
|
||||
|
||||
## Consequences
|
||||
|
||||
- Validator must be rewired: TOPLOC verify, hop blame, tracker-authoritative events — not string compare on final text alone.
|
||||
- Threshold calibration requires an **honest-noise corpus** across the volunteer fleet before production thresholds (research §8).
|
||||
- ADR-0003 remains historical; penalty mechanics follow ADR-0015 + this ADR.
|
||||
- Implementation order: auth + persistence → accounting → TOPLOC → bisection → reputation routing (`.scratch/alpha-hardening/README.md`).
|
||||
|
||||
## Related
|
||||
|
||||
- Research: `.scratch/alpha-hardening/research-verifiable-inference.md` (§8 recommended scheme, §9 roadmap, build-vs-adopt table)
|
||||
- ADR-0015 (forfeiture collateral)
|
||||
- ADR-0016 (alpha scope)
|
||||
- ADR-0017 (validator/forfeit auth)
|
||||
- Issues: `06-fraud-toploc-integration.md` through `10-fraud-penalty-calibration-wiring.md`
|
||||
66
docs/adr/0019-money-path-consistency-multi-tracker.md
Normal file
66
docs/adr/0019-money-path-consistency-multi-tracker.md
Normal file
@@ -0,0 +1,66 @@
|
||||
# ADR-0019: Money-path consistency (multi-tracker)
|
||||
|
||||
## Status: Accepted (design); implementation deferred post-alpha
|
||||
|
||||
## Context
|
||||
|
||||
ADR-0015 assumes a **single settlement tracker** with Raft-replicated ledger events. The hive already gossips billing, accounts, and stats between trackers. When multiple trackers participate, several money-path races appear:
|
||||
|
||||
- Settlement retries may double-pay if idempotency keys diverge from on-chain reality.
|
||||
- Any tracker follower can apply `forfeit` / `charge` events — fine under one trusted operator, dangerous with adversarial peers.
|
||||
- Raft term/vote state is in-memory — restart can duplicate leadership epochs.
|
||||
- `forfeit` events are not commutative with `charge`/`payout` ordering under concurrent replication.
|
||||
|
||||
Alpha ships with **one settlement-capable tracker** (ADR-0016). This ADR records the accepted multi-tracker design so Bucket 2 issues do not block alpha while preventing silent architectural drift.
|
||||
|
||||
## Decisions
|
||||
|
||||
### 1. Settlement idempotency (C2)
|
||||
|
||||
- Every on-chain payout batch carries a **stable `settlement_id`** already debiting pending before broadcast (`server.py` ~3353–3356, `_send_settlement` ~3358).
|
||||
- **Required hardening:** `solana_adapter.send_payouts` (~186–213) and the settlement loop must treat the Solana transaction signature as the idempotency anchor — retries reuse the same `settlement_id`; confirmed signatures must never spawn a second debit.
|
||||
- Deposit watcher already uses `deposit-<sig>` event ids — extend the same pattern to payouts.
|
||||
|
||||
### 2. Consensus-gated money mutations (C3/C4)
|
||||
|
||||
Money-affecting events (`charge`, `payout`, `forfeit`, `credit`, `settlement`, `bind`) must commit through **Raft** (or successor consensus), not best-effort gossip alone:
|
||||
|
||||
- Only the Raft leader appends money commands to the replicated log.
|
||||
- Followers apply committed entries in order; gossip becomes a transport, not the source of truth.
|
||||
- Settlement signing remains **leader-only** and **treasury-key-bearing** (ADR-0015).
|
||||
|
||||
Shard-assignment commands already use Raft (`raft.py`); extend the log command set.
|
||||
|
||||
### 3. Durable Raft metadata (A3)
|
||||
|
||||
Persist `currentTerm`, `votedFor`, and log index to disk (`raft.py` ~26 — today term lives only in memory). Without this, restart can elect split leaders and duplicate settlement epochs.
|
||||
|
||||
### 4. Commutative forfeit semantics (H1)
|
||||
|
||||
Define explicit ordering rules when replicating `forfeit` vs `charge`/`payout`:
|
||||
|
||||
- Forfeit amount is snapshotted at event creation time (`billing.py` ~287).
|
||||
- Apply forfeit **after** pending balance reflects all prior committed charges in the same epoch, or use signed pending snapshots in the event payload.
|
||||
- `setdefault` on `_pending_since` (~324) and wallet bindings (~351) must not resurrect stale state on out-of-order apply — use last-writer-wins with `(term, index)` tie-break.
|
||||
|
||||
### 5. Alpha posture
|
||||
|
||||
| Item | Alpha | Post-alpha |
|
||||
|---|---|---|
|
||||
| Single treasury holder | Required | Optional multi-sig |
|
||||
| Gossip-authenticated billing | Required (ADR-0017) | Superseded by Raft commit for money |
|
||||
| On-chain idempotency | Best-effort resend | Hard guarantee (C2) |
|
||||
| Consensus-gated settlement | Leader check only (~3331) | Full Raft gate (C3/C4) |
|
||||
|
||||
## Consequences
|
||||
|
||||
- Bucket 2 issues in `.scratch/alpha-hardening/` are **tracked but not alpha-blocking**.
|
||||
- Implementing C3/C4 before alpha would delay release without security benefit while only one operator runs settlement.
|
||||
- When enabling third-party settlement trackers, **all four** hardening items must ship together — partial deployment is worse than single-tracker alpha.
|
||||
|
||||
## Related
|
||||
|
||||
- ADR-0015 (settlement loop, pending collateral)
|
||||
- ADR-0016 (single settlement tracker for alpha)
|
||||
- ADR-0017 (gossip auth prerequisite)
|
||||
- Issues: `12-c2-on-chain-idempotency.md` through `15-h1-commutative-forfeit.md`
|
||||
Reference in New Issue
Block a user