popov/neuron-tai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add alpha-hardening ADRs and issue plan from pre-release audit.

Browse Source 
Lock alpha scope, tracker auth, TOPLOC fraud verification, and deferred multi-tracker money-path work; supersede legacy fraud issues with ADR-0018.

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
D.Popov
2026-07-04 23:12:09 +03:00
parent 7caf12980a
commit 68e057209c
35 changed files with 1550 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
.claude/memory/MEMORY.md
View File

@@ -2,4 +2,5 @@
- [Product selling points](product-selling-points.md) — key differentiators and landing page angles for neuron-tai
- [User profile](user-profile.md) — who Dobromir is and how to work with him
- [Project status](project-status.md) — 29/30 done; US-030 (manual route + hop benchmark) is the only open story
- [Project status](project-status.md) — 35/35 stories done; alpha hardening next
- **Alpha hardening** — `.scratch/alpha-hardening/` (22 issues, ADRs 0016–0019, [README](../.scratch/alpha-hardening/README.md), [handoff](../.scratch/alpha-hardening/handoff.md))

4
.claude/memory/project-status.md
View File

@@ -23,6 +23,10 @@ Suite: 222 passed, 3 skipped (openai/langchain packages missing in .venv — pre
**Why:** design locked in ADR-0015 (USDT custodial settlement; TAI deferred, protocol cut = future TAI liquidity).
**How to apply:** next steps are live devnet verification (run devnet_setup.py, start tracker with --solana-rpc-url/--usdt-mint/--treasury-keypair --billing-db), then the TAI mint when volume justifies it. Work not yet committed to git as of session end — check git status.
## Alpha hardening (2026-07-04)
Planning complete in `.scratch/alpha-hardening/`: research report, ADRs 0016–0019, 22 issue files, README + handoff. **Bucket 1 trust-boundary blockers** (auth, persistence, starting credit, accounting, wallet binding) are next before fraud arc (TOPLOC, bisection, reputation routing). Prod audit thresholds gated on issue 21 (honest-noise calibration corpus).
## Windows CUDA node (working as of 2026-07-01)
- miniforge3 base env, torch 2.7.1+cu118, torchvision 0.22.x+cu118
- RTX 4060 Laptop GPU, 8 GB VRAM, benchmark index ~11,200

93
.scratch/alpha-hardening/README.md Normal file
View File

@@ -0,0 +1,93 @@
# Alpha hardening — planning index
Pre-release alpha audit + grilling (2026-07-04). **Research complete; planning complete; Bucket 1 blockers next.**
Locked scope: one settlement tracker, open node join, devnet mock-USDT, reputation carries forward → fraud must be bounded. See [ADR-0016](../../docs/adr/0016-alpha-scope-and-known-limitations.md).
## Artifacts
| Path | Status |
|---|---|
| [research-verifiable-inference.md](./research-verifiable-inference.md) | Complete — SOTA research, §8 layered scheme, TOPLOC adopt |
| [handoff.md](./handoff.md) | Session handoff — locked decisions, env notes |
| [docs/adr/0016–0019](../../docs/adr/) | Alpha scope, auth, fraud, multi-tracker design |
| [issues/](./issues/) | 22 work items (Buckets 1–3) |
## ADRs (this feature)
| ADR | Title |
|---|---|
| [0016](../../docs/adr/0016-alpha-scope-and-known-limitations.md) | Alpha scope & known limitations |
| [0017](../../docs/adr/0017-tracker-authentication-and-authorization.md) | Tracker authentication & authorization |
| [0018](../../docs/adr/0018-fraud-detection-verification-and-reputation.md) | Fraud detection, verification & reputation (flagship) |
| [0019](../../docs/adr/0019-money-path-consistency-multi-tracker.md) | Money-path consistency — design accepted, impl deferred |
| [0002](../../docs/adr/0002-dual-token-payment-model.md) | Amended — settlement superseded by 0015 |
| [0010](../../docs/adr/0010-p2p-gossip-and-nat-relay.md) | Amended — TLS alpha reality (relay only) |
## Recommended implementation order
**Implement Bucket 1 first.** Fraud arc depends on **auth (01–02, 20)** and **persistence (05)**.
### Phase 1 — Trust boundary (alpha blockers)
| Order | Issue | ID | Depends on |
|---|---|---|---|
| 1 | [Unified auth + gossip auth](./issues/02-a2-unified-auth-boundary.md) + [C1 gossip](./issues/01-c1-gossip-auth.md) + [Validator service token](./issues/20-validator-service-token.md) | A2, C1, — | — |
| 2 | [Persist strike/ban/reputation](./issues/05-a1-a5-persist-strike-ban-reputation.md) | A1/A5 | 02 |
| 3 | [Starting credit 0 + spend cap](./issues/03-c5-starting-credit-zero.md) | C5, M1 | 02 |
| 4 | [Tracker-authoritative accounting](./issues/04-h2-tracker-authoritative-accounting.md) | H2 | 02 |
| 5 | [Wallet binding proof](./issues/11-c6-wallet-binding-proof.md) | C6 | 02, 03 |
### Phase 2 — Fraud arc (after Phase 1)
| Order | Issue | Depends on |
|---|---|---|
| 6 | [TOPLOC integration](./issues/06-fraud-toploc-integration.md) | 05 |
| 7 | [Commitment + bisection blame](./issues/07-fraud-commitment-bisection-blame.md) | 06 |
| 8 | [Reputation model](./issues/08-fraud-reputation-model-persistence.md) | 05, 07 |
| 9 | [Routing + adaptive audit](./issues/09-fraud-reputation-routing-adaptive-audit.md) | 08 |
| 10 | [Penalty calibration wiring](./issues/10-fraud-penalty-calibration-wiring.md) | 07, 08, 02 |
**Prod gate:** [21 honest-noise calibration corpus](./issues/21-honest-noise-calibration-corpus.md) must complete before enabling production TOPLOC audit thresholds (issues 09–10 in prod). Dev/staging TOPLOC wiring (06–08) may proceed in parallel.
### Phase 3 — Bucket 2 (post-alpha, design tracked)
| Issue | ADR |
|---|---|
| [12 C2 on-chain idempotency](./issues/12-c2-on-chain-idempotency.md) | 0019 §1 |
| [13 C3/C4 consensus-gated settlement](./issues/13-c3-c4-consensus-gated-settlement.md) | 0019 §2 |
| [14 A3 durable Raft term/vote](./issues/14-a3-raft-durable-term-vote.md) | 0019 §3 |
| [15 H1 commutative forfeit](./issues/15-h1-commutative-forfeit.md) | 0019 §4 |
### Phase 4 — Doc hygiene (parallel anytime)
| Issue |
|---|
| [16 US-006 + fraud issue reconciliation](./issues/16-doc-us006-reconciliation.md) |
| [17 Duplicate US-020 dedup](./issues/17-doc-duplicate-us020-dedup.md) |
| [18 Operational runbooks](./issues/18-doc-operational-runbooks.md) |
| [19 Cryptography + test env](./issues/19-doc-cryptography-test-env.md) |
| [22 MEMORY + project-status index](./issues/22-doc-memory-project-status.md) |
| [21 Honest-noise calibration corpus](./issues/21-honest-noise-calibration-corpus.md) (ops; prod gate for audits) |
## First 3 to implement
1. **01 + 02 + 20** — Gossip auth + unified auth boundary + validator service token (unblocks financial endpoints)
2. **05** — Persist strike/ban/reputation (penalties must survive restart)
3. **03** — Starting credit 0 + funded-account gate (closes free-credit faucet)
## Research anchor
Fraud design cites [.scratch/alpha-hardening/research-verifiable-inference.md](./research-verifiable-inference.md):
- **ADOPT** TOPLOC (§8–9, build-vs-adopt table)
- **On-demand** commitments, not every request (§8 layer 1 footnote; ADR-0018 §3)
- **5% audit budget** as target, not cap (§1.1, §6)
- **19× deterrence** via full pending forfeiture (§1.1)
- **Hop bisection** blame pattern (§1.2, §8 layer 3)
- **Honest-noise corpus** before prod thresholds (§8 layer 3; issue 21)
- **Roadmap-only:** zkML, TEE, Gensyn RepOps (§9)
## Comments
<!-- Append triage / implementation notes below -->

111
.scratch/alpha-hardening/handoff.md Normal file
View File

@@ -0,0 +1,111 @@
# neuron-tai — Alpha Hardening Handoff
**Date:** 2026-07-04
**Repo:** `D:/DEV/git.d-popov.com/neuron-tai`
**Prior session:** Pre-release alpha audit + grilling (design locked; planning artifacts complete)
---
## 1. Mission / where we are
neuron-tai is a volunteer-GPU, pipeline-parallel LLM inference network with a working routing layer and a **broken money/trust path**. Three independent audits agreed: unauthenticated gossip, free-credit faucet, double-pay risks, ephemeral bans, and node self-reported accounting undermine alpha release. The owner locked alpha scope (single settlement tracker, open node join, devnet mock-USDT, carried-forward reputation) and a fraud/verification design (TOPLOC adoption, adaptive audits, on-demand hop bisection, persisted graduated reputation, tracker-authoritative accounting). **Research and planning artifacts are complete** (ADRs 0016–0019, 22 issue files, README index). Next: implement Bucket 1 blockers test-first.
---
## 2. Locked decisions
Point to artifacts — do not re-derive from this handoff.
| Decision | Status | Reference |
|---|---|---|
| Alpha scope: one settlement tracker, open join, devnet mock-USDT, reputation carries forward | Locked | `docs/adr/0016-alpha-scope-and-known-limitations.md` |
| Two fraud types: correctness (bad output) vs accounting (inflated tokens/shard span) | Locked | Session grilling; research §8 |
| Detection: **ADOPT TOPLOC** (MIT, `pip install toploc`); teacher-forced prefill; pin one canonical precision per model | Locked | `.scratch/alpha-hardening/research-verifiable-inference.md` §8 |
| Audit rate: **5% default, not a cap**; escalate on anomaly, low reputation, disputes | Locked | Research §1.1, §6, §8 |
| Blame: **on-demand** per-hop activation commitments + bisection to first divergent hop (Verde/Truebit **pattern only**) | Locked | Research §1.2; ADR-0018 §3–4 |
| Reputation: graduated multiplier (×0.8-per-strike shape), persisted, affects routing + audit rate | Locked | ADR-0018 §6; `packages/validator/README.md` |
| Penalty: **full pending forfeiture** is primary hammer (19× deterrence at 5% audit); ×0.8 is routing/payout decay | Locked | Research §1.1; ADR-0018 §1 |
| Accounting: tracker authoritative — count tokens from proxied stream; work units from **tracker-assigned** shard span | Locked | ADR-0018 §5; issue 04 |
| Persistence prerequisite: strike/ban/reputation/probation must survive tracker restart | Locked | Issue 05 |
| Validator service token for forfeit | Locked | ADR-0017 §4; issue 20 |
| Honest-noise corpus before prod audit thresholds | Locked | ADR-0018 consequences; issue 21 |
| Build vs adopt: TOPLOC **ADOPT**; Verde bisection **ADAPT**; zkML/TEE **roadmap-only** | Locked | Research §9 |
| Multi-tracker money-path (C2/C3/C4/A3/H1): **design now, implement later** | Locked | `docs/adr/0019-money-path-consistency-multi-tracker.md` |
| Routing layer | **Solid** — no redesign needed | ADR-0013 |
**Existing ADRs still relevant:** ADR-0003 (historical prototype), ADR-0015 (USDT custodial settlement).
---
## 3. Artifact index (read first)
| Path | What it contains |
|---|---|
| `.scratch/alpha-hardening/README.md` | Issue/ADR index + implementation order |
| `.scratch/alpha-hardening/issues/` | 22 work items (Buckets 1–3) |
| `.scratch/alpha-hardening/research-verifiable-inference.md` | SOTA research, layered alpha scheme (§8), build-vs-adopt (§9) |
| `docs/adr/0016–0019` | Alpha scope, auth, fraud, multi-tracker design |
| `docs/agents/issue-tracker.md` | Issue file conventions |
| `packages/validator/meshnet_validator/__init__.py` | Current validator; `_final_text_node` blame bug |
| `packages/tracker/meshnet_tracker/server.py` | Auth gaps, gossip handlers, proxy accounting |
| `.claude/memory/MEMORY.md` | Agent memory index |
| Agent transcript (grilling session) | [Alpha audit grilling](4406ccbb-011a-4157-851d-b5b242bba7f7) |
### Bucket summaries
**Bucket 1 — Alpha blockers:** C1 gossip auth; A2 unified auth + issue 20 validator token; C5 starting credit; H2 tracker accounting; A1/A5 persistence; C6 wallet binding; fraud arc 06–10.
**Bucket 2 — Design deferred:** C2, C3/C4, A3, H1 (issues 12–15).
**Bucket 3 — Doc hygiene:** US-006/07/34 reconciliation (16), MEMORY index (22), runbooks (18), cryptography test env (19).
---
## 4. Recommended implementation order
See `.scratch/alpha-hardening/README.md` for full table.
**First 3:**
1. **01 + 02 + 20** — Gossip auth + unified auth boundary + validator service token
2. **05** — Persist strike/ban/reputation
3. **03** — Starting credit 0 + funded-account gate
**Prod gate:** issue **21** (honest-noise corpus) before enabling production audit thresholds.
---
## 5. Open questions / deferred
| Topic | State |
|---|---|
| Multi-tracker consensus & settlement idempotency | Bucket 2 (ADR-0019) |
| `/v1/gossip` node throughput auth | Out of scope alpha — ADR-0017 §3 note |
| Seed-synchronized exact-token audits | Optional complement; depends on sampler control |
| ADR-0010 TLS everywhere | Relay TLS only in alpha; ADR amended |
---
## 6. Environment notes
| Item | Detail |
|---|---|
| OS | Windows 10 |
| Repo path | `D:/DEV/git.d-popov.com/neuron-tai` |
| Python venv | `.venv/Scripts/python.exe` |
| CUDA node | RTX 4060 8 GB; tracker registration requires `https://` |
| Secrets | Do not commit `.env.devnet`, keypairs, treasury material |
---
## 7. Suggested skills
| Skill | When |
|---|---|
| **implement** | Bucket 1 code work |
| **tdd** | All Bucket 1 fixes |
| **diagnosing-bugs** | Auth/gossip/accounting failures |
| **domain-modeling** | ADR cross-links |
| **code-review** | After each blocker milestone |
Read `.claude/memory/MEMORY.md` at session start.

41
.scratch/alpha-hardening/issues/01-c1-gossip-auth.md Normal file
View File

@@ -0,0 +1,41 @@
Status: ready-for-agent
# 01 — C1: Authenticate hive gossip endpoints
## What to build
Add authenticated peer identity to all tracker gossip mutation endpoints. Today any caller can push billing, account, and stats events without verification.
**Code refs:**
- `packages/tracker/meshnet_tracker/server.py` — `_handle_billing_gossip` (~2414–2427)
- `packages/tracker/meshnet_tracker/server.py` — `_handle_accounts_gossip` (~2610–2623)
- `packages/tracker/meshnet_tracker/server.py` — `_handle_stats_gossip` (~2355–2364)
- `packages/tracker/meshnet_tracker/billing.py` — `apply_events` (~301–311)
- `packages/tracker/meshnet_tracker/accounts.py` — `apply_events` (~220–226)
Implement per ADR-0017 §3: shared hive HMAC (body + timestamp) or mutual TLS between configured tracker peers. Reject unauthenticated gossip with 401.
**Note:** `/v1/gossip` (node throughput fan-out, `server.py` ~1331) is **not** in scope for this issue — see ADR-0017 §3 out-of-scope note.
## Test-first
1. Red: unauthenticated POST to `/v1/billing/gossip` applies a credit event today — test must fail after fix.
2. Red: authenticated peer with valid HMAC applies events; invalid/missing auth returns 401 and `applied: 0`.
3. Green: implement verifier + config (`--hive-secret` or peer cert paths).
## Acceptance criteria
- [ ] `/v1/billing/gossip`, `/v1/accounts/gossip`, `/v1/stats/gossip` reject requests without valid hive auth
- [ ] Authenticated peers replicate events as today (id-dedup preserved)
- [ ] Config documented for multi-tracker dev setups
- [ ] Tests cover reject + accept paths without live network
## ADR links
- [ADR-0017](../../docs/adr/0017-tracker-authentication-and-authorization.md)
- [ADR-0016](../../docs/adr/0016-alpha-scope-and-known-limitations.md)
## Blocked by
None — implement alongside or immediately before `02-a2-unified-auth-boundary.md`.

45
.scratch/alpha-hardening/issues/02-a2-unified-auth-boundary.md Normal file
View File

@@ -0,0 +1,45 @@
Status: ready-for-agent
# 02 — A2: Unified auth boundary for privileged and financial reads
## What to build
Replace header-presence stubs with a single auth middleware that resolves API keys, admin sessions, validator service tokens, and hive peer identity. Close leaks on financial and operator endpoints.
**Code refs:**
- `packages/tracker/meshnet_tracker/server.py` — `_handle_billing_forfeit` (~2429–2464) — H3: non-empty `Authorization` only
- `packages/tracker/meshnet_tracker/server.py` — `_handle_benchmark_hop_penalty` (~2650–2658), `_handle_benchmark_results` (~2745–2748) — H3
- `packages/tracker/meshnet_tracker/server.py` — `_handle_billing_summary` (~2366–2371) — H4
- `packages/tracker/meshnet_tracker/server.py` — `_handle_billing_settlements` (~2407–2412) — H4
- `packages/tracker/meshnet_tracker/server.py` — `_handle_registry_wallets` (~2391–2405) — H4
- `packages/tracker/meshnet_tracker/server.py` — `_session_account` (~2468+), `_handle_admin_accounts` (~2588–2608) — H4
- `packages/tracker/meshnet_tracker/accounts.py` — `session_account()`, `create_session()` only (session store; not handler wiring)
Per ADR-0017 §4: forfeit → validator or admin; benchmark → admin; billing summary/settlements/registry wallets → admin session. Forfeit validator identity: see `20-validator-service-token.md`.
## Test-first
1. Red: POST `/v1/billing/forfeit` with `Authorization: Bearer garbage` succeeds today — must require validator/admin identity.
2. Red: GET `/v1/billing/summary` without admin session returns 401/403.
3. Green: middleware + role checks; existing inference API-key path unchanged.
## Acceptance criteria
- [ ] Single `_require_auth(role=...)` (or equivalent) used by all privileged handlers
- [ ] Forfeit accepts only validator service token or admin session — not arbitrary Bearer strings
- [ ] Financial read endpoints require admin session (alpha posture)
- [ ] Benchmark write/read require admin or service token
- [ ] Integration tests for each endpoint class (reject unauth, accept valid)
## ADR links
- [ADR-0017](../../docs/adr/0017-tracker-authentication-and-authorization.md)
## Related
- `20-validator-service-token.md` — validator service token format, rotation, forfeit auth
## Blocked by
- `01-c1-gossip-auth.md` (shared auth config)

38
.scratch/alpha-hardening/issues/03-c5-starting-credit-zero.md Normal file
View File

@@ -0,0 +1,38 @@
Status: ready-for-agent
# 03 — C5 + M1: Starting credit 0, funded-account gate, spend cap
## What to build
Close the free-credit faucet. New API keys start at **0 USDT**; inference requires a real deposit or admin credit. Add a configurable per-request spend cap (M1) to limit runaway charges on compromised keys.
**Code refs:**
- `packages/tracker/meshnet_tracker/billing.py` — `DEFAULT_STARTING_CREDIT = 1.0` (~22), `ensure_client` (~73–85), `has_funds` (~87–88), duplicate credit on charge (~130–138)
- `packages/tracker/meshnet_tracker/server.py` — billing gate before routing (~1667–1690)
Per ADR-0017 §2 and ADR-0016 §3.
## Test-first
1. Red: new API key gets 1.0 USDT implicit credit — test expects 0 balance until deposit.
2. Red: first inference without deposit returns 402.
3. Green: `DEFAULT_STARTING_CREDIT = 0.0`; optional `--max-charge-per-request` config.
## Acceptance criteria
- [ ] `DEFAULT_STARTING_CREDIT` is 0.0; no automatic caller credit on first touch
- [ ] `has_funds` false for fresh keys; 402 before routing (server.py ~1684)
- [ ] Admin `credit_client` or bound-wallet deposit still funds accounts
- [ ] Configurable max charge per request (M1) rejects oversize completions with clear error
- [ ] Tests: fresh key blocked; after credit/deposit, inference proceeds
## ADR links
- [ADR-0017](../../docs/adr/0017-tracker-authentication-and-authorization.md)
- [ADR-0016](../../docs/adr/0016-alpha-scope-and-known-limitations.md)
- [ADR-0015](../../docs/adr/0015-usdt-custodial-settlement.md)
## Blocked by
- `02-a2-unified-auth-boundary.md` (admin credit path secured)

37
.scratch/alpha-hardening/issues/04-h2-tracker-authoritative-accounting.md Normal file
View File

@@ -0,0 +1,37 @@
Status: ready-for-agent
# 04 — H2: Tracker-authoritative token and work-unit accounting
## What to build
Stop trusting node-reported usage for billing. The tracker already proxies responses — count tokens from the proxied stream/body and compute work units from the **route it constructed**, not node declarations.
**Code refs:**
- `packages/tracker/meshnet_tracker/server.py` — `node_work` from route construction (~1776–1782, ~1781–1782)
- `packages/tracker/meshnet_tracker/server.py` — streaming token/chunk billing (~1890–1921)
- `packages/tracker/meshnet_tracker/server.py` — non-streaming `_usage_total_tokens` (~1938–1943)
- `packages/tracker/meshnet_tracker/billing.py` — `charge_request` node_work split (~104–151)
Accounting fraud = inflating tokens or shard span. Per ADR-0018 §5.
## Test-first
1. Red: mock upstream returns inflated `usage.total_tokens` in body but tracker bills that value — test expects tracker-measured count.
2. Red: node registers false `shard_end`; billing uses tracker route span, not registration field alone.
3. Green: authoritative counters; ignore node-reported work units on charge path.
## Acceptance criteria
- [ ] Token count for billing derived from tracker-parsed proxy response — **prefer tracker-measured stream chunk count**; when upstream `usage.total_tokens` is present, use the **minimum** of tracker count and node-reported value (cap inflated node usage)
- [ ] Work units = tracker-computed layer span per hop at route build time (~1781–1782)
- [ ] Nodes cannot increase payout by lying about shard range mid-request
- [ ] Integration test: malicious node metadata does not inflate `charge_request` shares
## ADR links
- [ADR-0018](../../docs/adr/0018-fraud-detection-verification-and-reputation.md) §5
## Blocked by
- `02-a2-unified-auth-boundary.md`

40
.scratch/alpha-hardening/issues/05-a1-a5-persist-strike-ban-reputation.md Normal file
View File

@@ -0,0 +1,40 @@
Status: ready-for-agent
# 05 — A1/A5: Persist strike, ban, and reputation state
## What to build
Registry strike/ban/reputation state today lives in RAM-only `_LocalContractState` — tracker restart wipes penalties. Persist to SQLite (same pattern as `BillingLedger` and `AccountStore`) so reputation carries forward per ADR-0016 §4.
**Code refs:**
- `packages/contracts/meshnet_contracts/__init__.py` — `RegistryContract`, `RegistryWallet`, in-memory `_state.registry` (~103–206)
- `packages/tracker/meshnet_tracker/billing.py` — SQLite persistence pattern (~60, event log)
- `packages/tracker/meshnet_tracker/accounts.py` — SQLite + event replication (~40–56)
Include fields for: `strike_count`, `banned`, `completed_job_count`, graduated **reputation score** (float, default 1.0), `last_audit_ts`, probation tracking.
**Scope split:** this issue owns **schema + persistence + load/reload** only. Reputation **scoring deltas** (audit pass/fail adjustments, decay rules) belong in issue 08.
## Test-first
1. Red: record strike, restart tracker process, strike count is 0 — must fail.
2. Green: persist + reload; gossip replicates strike events if multi-tracker.
3. Red: banned wallet registers node — must reject (wire to routing).
## Acceptance criteria
- [ ] Strike/ban/reputation survive tracker restart (SQLite or equivalent)
- [ ] `RegistryContract.list_wallets` reflects persisted state
- [ ] Banned wallet rejected at registration and excluded from routes
- [ ] Reputation score field present for routing/audit issues (08–09)
- [ ] Event-sourced mutations compatible with future Raft (ADR-0019)
## ADR links
- [ADR-0016](../../docs/adr/0016-alpha-scope-and-known-limitations.md) §4
- [ADR-0018](../../docs/adr/0018-fraud-detection-verification-and-reputation.md) §6
## Blocked by
- `02-a2-unified-auth-boundary.md`

47
.scratch/alpha-hardening/issues/06-fraud-toploc-integration.md Normal file
View File

@@ -0,0 +1,47 @@
Status: ready-for-agent
# 06 — FRAUD: TOPLOC integration (teacher-forced audit primitive)
## What to build
Adopt [TOPLOC](https://github.com/PrimeIntellect-ai/toploc) (MIT, `pip install toploc`) for activation fingerprint commit and verify. Replace string-equality validator checks with teacher-forced prefill + TOPLOC tolerance matching.
**Estimated effort:** 2+ sessions — split into subtasks below without separate issue files.
| Subtask | Owner package | Deliverable |
|---|---|---|
| Validator audit primitive | `packages/validator/` | Teacher-forced prefill, TOPLOC verify, unit tests with stub tensors |
| Node runtime commitments | `packages/node/` (if prover-side) | On-demand activation fingerprint generation on audit-selected requests |
**Code refs:**
- `packages/validator/meshnet_validator/__init__.py` — `_run_reference`, `_outputs_match` (~92–148)
- `packages/validator/README.md` — deterrence math (update for 19× at p=0.05)
- Research: `.scratch/alpha-hardening/research-verifiable-inference.md` §8 layers 1–2, build-vs-adopt table
Pin one canonical precision/quantization per model preset. Add `toploc` to validator (and node if prover-side) dependencies.
## Test-first
1. Red: validator compares final text strings — fails on cross-GPU honest divergence (document expected).
2. Green: stub activation tensors + TOPLOC proofs round-trip in unit test.
3. Integration: reference node teacher-forces tokens; verify accepts honest proof, rejects swapped precision.
## Acceptance criteria
- [ ] `toploc` dependency declared; `build_proofs_*` / `verify_proofs_*` wired
- [ ] Validator re-runs claimed token sequence as prefill, not free generation
- [ ] Model preset documents canonical dtype/quantization
- [ ] README updated: 19× deterrence at 5% audit (research §1.1)
- [ ] Tests with deterministic stub tensors (no GPU required in CI)
## ADR links
- [ADR-0018](../../docs/adr/0018-fraud-detection-verification-and-reputation.md) §2
- Research: [research-verifiable-inference.md](../research-verifiable-inference.md) §8, §9 build-vs-adopt
## Blocked by
- `05-a1-a5-persist-strike-ban-reputation.md`
**Prod gate:** do not enable production audit thresholds until `21-honest-noise-calibration-corpus.md` completes (see README Phase 2 note).

35
.scratch/alpha-hardening/issues/07-fraud-commitment-bisection-blame.md Normal file
View File

@@ -0,0 +1,35 @@
Status: ready-for-agent
# 07 — FRAUD: On-demand commitment + hop bisection blame
## What to build
On audit selection, require nodes to supply TOPLOC-style fingerprints of **output boundary activations** per hop (on-demand, brief retention). On verify failure, referee identifies the **first divergent hop** — not always the last text node.
**Code refs:**
- `packages/validator/meshnet_validator/__init__.py` — `_slash_route`, `_final_text_node` bug (~102–140) — blames `max(shard_end)` only
- `packages/tracker/meshnet_tracker/server.py` — route hop construction (~1774–1783) — cut-points for bisection
- Research: `.scratch/alpha-hardening/research-verifiable-inference.md` §1.2, §8 layer 3 (Verde **pattern**, not on-chain game)
## Test-first
1. Red: two-hop route, corrupt hop-0 activations — `_final_text_node` blames hop-1 — test must fail.
2. Green: bisection selects hop-0; forfeit targets hop-0 wallet.
3. On-demand: commitment requested only when audit flag set on proxied request.
## Acceptance criteria
- [ ] Audit requests carry tracker RNG/VRF flag indistinguishable from normal traffic (research §6)
- [ ] Nodes retain recent boundary activations for on-demand commit window (configurable TTL)
- [ ] Validator/tracker compares fingerprints at each hop cut-point; first mismatch = culprit
- [ ] `_final_text_node` removed or limited to text-only fallback
- [ ] Integration test: multi-hop pipeline, fault injected at known hop
## ADR links
- [ADR-0018](../../docs/adr/0018-fraud-detection-verification-and-reputation.md) §3–4
## Blocked by
- `06-fraud-toploc-integration.md`

39
.scratch/alpha-hardening/issues/08-fraud-reputation-model-persistence.md Normal file
View File

@@ -0,0 +1,39 @@
Status: ready-for-agent
# 08 — FRAUD: Reputation model + persistence
## What to build
Implement graduated reputation per ADR-0018 §6: score derives only from tracker audit outcomes + uptime/latency. Slow build, instant loss, inactivity decay. ×0.8 routing multiplier per strike (not whole penalty — forfeiture stays full pending).
**Scope split:** issue 05 owns **schema + SQLite persistence**; this issue owns **scoring rules** (deltas, decay, strike→multiplier wiring) on top of persisted fields.
**Code refs:**
- `packages/contracts/meshnet_contracts/__init__.py` — extend `RegistryWallet` / persistence from issue 05
- `packages/validator/meshnet_validator/__init__.py` — `_slash_route` forfeiture path (~125–133)
- `packages/tracker/meshnet_tracker/billing.py` — `forfeit_pending` (~280–292)
- Research: `.scratch/alpha-hardening/research-verifiable-inference.md` §6
## Test-first
1. Red: no reputation field — add schema + default 1.0 for new wallets.
2. Green: clean audit +0.05 (tunable); failed audit −0.3 and strike; three strikes → ban persisted.
3. Inactivity decay after N days without completed jobs.
## Acceptance criteria
- [ ] `reputation_score` persisted in registry SQLite (issue 05)
- [ ] Audit pass/fail updates score with documented deltas
- [ ] Strike applies ×0.8 multiplier to routing weight (separate from forfeiture amount)
- [ ] Ban at 3 strikes; probation job count still enforced
- [ ] No peer-to-peer reputation inputs
## ADR links
- [ADR-0018](../../docs/adr/0018-fraud-detection-verification-and-reputation.md) §6
## Blocked by
- `05-a1-a5-persist-strike-ban-reputation.md`
- `07-fraud-commitment-bisection-blame.md` (audit outcomes feed reputation)

38
.scratch/alpha-hardening/issues/09-fraud-reputation-routing-adaptive-audit.md Normal file
View File

@@ -0,0 +1,38 @@
Status: ready-for-agent
# 09 — FRAUD: Reputation-weighted routing + adaptive audit rate
## What to build
Wire reputation into route selection and audit sampling. Default network audit budget ≈5% — **not a cap**. New/low-reputation nodes: 20–30% audit rate; veterans: 2–3% floor ≥2%. Tripwires escalate rate without direct punishment.
**Code refs:**
- `packages/tracker/meshnet_tracker/server.py` — route selection `_select_route`, `_effective_throughput` (~1747, routing helpers)
- `packages/validator/meshnet_validator/__init__.py` — `sample_rate=0.05`
- Research: `.scratch/alpha-hardening/research-verifiable-inference.md` §1.1, §6, §8 layers 2–4
Audit selection must be unpredictable at request time (tracker RNG after commitment window opens).
## Test-first
1. Red: uniform 5% sample regardless of reputation — test expects higher rate for low-reputation wallet.
2. Green: budget balancer keeps fleet-wide average ≈ configured target.
3. Routing prefers higher reputation among equal throughput candidates.
## Acceptance criteria
- [ ] Per-wallet audit probability function of reputation (newcomer high, veteran low, floor ≥2%)
- [ ] Fleet-wide audit budget configurable (~5% default target); over ≥1000 requests with fixed seed, measured fleet audit rate within **±1.0 percentage point** of configured target (e.g. 4.0–6.0% at 5% default)
- [ ] Route scoring includes reputation multiplier (earnings scale with tenure)
- [ ] Passive tripwire flags (perplexity/repetition) bump audit rate only
- [ ] Tests: deterministic seed for sampling distribution checks
## ADR links
- [ADR-0018](../../docs/adr/0018-fraud-detection-verification-and-reputation.md) §1, §6–7
- [ADR-0013](../../docs/adr/0013-rolling-stats-smart-routing.md)
## Blocked by
- `08-fraud-reputation-model-persistence.md`

42
.scratch/alpha-hardening/issues/10-fraud-penalty-calibration-wiring.md Normal file
View File

@@ -0,0 +1,42 @@
Status: ready-for-agent
# 10 — FRAUD: Penalty calibration wiring (forfeit + strike + ban)
## What to build
End-to-end wiring: confirmed audit failure → atomic pending forfeiture + strike + reputation decay + audit-rate snap to max. Ensure payout cannot race penalty (ADR-0015). Document 19× deterrence math in validator README.
**Code refs:**
- `packages/validator/meshnet_validator/__init__.py` — `_slash_route` (~102–134)
- `packages/tracker/meshnet_tracker/server.py` — `_handle_billing_forfeit` (~2429–2464)
- `packages/tracker/meshnet_tracker/billing.py` — `forfeit_pending` (~280–292), payout exclusion for banned (~3337–3344 in settlement loop)
- `packages/validator/README.md` — update 20× → 19× at p=0.05
Per ADR-0018: **full pending forfeiture** is primary penalty; ×0.8 is routing decay per strike, not partial forfeit.
## Test-first
1. Red: integration from issue 34 — extend with multi-hop blame wallet from issue 07.
2. Green: node with pending balance → audit fail → pending zero, strike++, banned on 3rd, excluded from next settlement.
3. Settlement loop skips banned wallets (~3337–3344).
## Acceptance criteria
- [ ] Audit failure triggers forfeiture + strike in one tracker transaction
- [ ] Banned nodes excluded from `payables` / settlement
- [ ] Validator uses authenticated forfeit endpoint (issue 02)
- [ ] README: `L > 19× g` at p=0.05; pending balance = collateral
- [ ] Integration test: 60-request fraud scenario → ban within threshold
## ADR links
- [ADR-0018](../../docs/adr/0018-fraud-detection-verification-and-reputation.md)
- [ADR-0015](../../docs/adr/0015-usdt-custodial-settlement.md)
- Research: [research-verifiable-inference.md](../research-verifiable-inference.md) §1.1
## Blocked by
- `07-fraud-commitment-bisection-blame.md`
- `08-fraud-reputation-model-persistence.md`
- `02-a2-unified-auth-boundary.md`

37
.scratch/alpha-hardening/issues/11-c6-wallet-binding-proof.md Normal file
View File

@@ -0,0 +1,37 @@
Status: ready-for-agent
# 11 — C6: Wallet binding ownership proof + binding overwrite safety
## What to build
`POST /v1/wallet/register` binds a client Solana wallet to an API key for deposit attribution. Today any Bearer key can bind any wallet string without proving ownership. Prevent hijack and accidental overwrite.
**Code refs:**
- `packages/tracker/meshnet_tracker/server.py` — `_handle_wallet_register` (~2625–2648)
- `packages/tracker/meshnet_tracker/billing.py` — `bind_wallet` (~153+), `_wallet_bindings` / direct overwrite on apply (~351)
Require signed message from wallet pubkey (ed25519 via `cryptography` / solders). Reject rebinding without admin or signed release. Use explicit overwrite policy — today `~351` overwrites binding directly; gossip apply must reject conflicting binds instead of silently clobbering.
## Test-first
1. Red: bind wallet A with only API key, no signature — must fail after fix.
2. Red: wallet already bound to key1; key2 cannot steal without proof.
3. Green: valid signature binds; deposit watcher credits correct API key.
## Acceptance criteria
- [ ] Wallet binding requires cryptographic proof of pubkey ownership
- [ ] One wallet → one API key (or documented admin override)
- [ ] Gossip `bind` events cannot overwrite existing binding via direct overwrite at `~351`
- [ ] Tests with deterministic keypairs (local adapter)
## ADR links
- [ADR-0017](../../docs/adr/0017-tracker-authentication-and-authorization.md) §5
- [ADR-0015](../../docs/adr/0015-usdt-custodial-settlement.md)
## Blocked by
- `02-a2-unified-auth-boundary.md`
- `03-c5-starting-credit-zero.md`

29
.scratch/alpha-hardening/issues/12-c2-on-chain-idempotency.md Normal file
View File

@@ -0,0 +1,29 @@
Status: ready-for-human
# 12 — C2: On-chain settlement idempotency (deferred)
## What to build
Harden payout idempotency so Solana transaction retries never double-pay. Design accepted in ADR-0019 §1; **implementation deferred post-alpha**.
**Code refs:**
- `packages/tracker/meshnet_tracker/server.py` — `_settlement_loop` resend (~3331–3356), `_send_settlement` (~3358–3376)
- `packages/contracts/meshnet_contracts/solana_adapter.py` — `send_payouts` (~186–213)
Today: pending debited before broadcast with stable `settlement_id`; unconfirmed batches resent. Gap: on-chain confirmation vs ledger state if tx succeeds but confirm fails.
## Acceptance criteria
- [ ] `confirm_settlement` only after RPC finalized confirmation
- [ ] Retry path reuses same `settlement_id` and detects already-confirmed signature
- [ ] Property test: N retries → single on-chain transfer per wallet per settlement_id
- [ ] Document recovery procedure for stuck unconfirmed batches
## ADR links
- [ADR-0019](../../docs/adr/0019-money-path-consistency-multi-tracker.md) §1
## Blocked by
Alpha release (ADR-0016 single settlement tracker)

31
.scratch/alpha-hardening/issues/13-c3-c4-consensus-gated-settlement.md Normal file
View File

@@ -0,0 +1,31 @@
Status: ready-for-human
# 13 — C3/C4: Consensus-gated money mutations (deferred)
## What to build
Route money-affecting ledger events through Raft commit, not gossip-only apply. Extend `raft.py` command set beyond register/deregister. Settlement remains leader-only with treasury key.
**Code refs:**
- `packages/tracker/meshnet_tracker/server.py` — settlement leader gate (~3331–3332), payout batch (~3353–3356)
- `packages/tracker/meshnet_tracker/raft.py` — log entry types (~26–27)
- `packages/tracker/meshnet_tracker/billing.py` — `apply_events` (~301–311)
Design: ADR-0019 §2. **Deferred post-alpha** while single operator holds settlement.
## Acceptance criteria
- [ ] `charge`, `payout`, `forfeit`, `credit`, `settlement`, `bind` commit via Raft log
- [ ] Followers reject direct gossip money mutations
- [ ] Leader-only `_settlement_loop` unchanged in semantics
- [ ] Migration plan from gossip-only billing to Raft-backed log
## ADR links
- [ADR-0019](../../docs/adr/0019-money-path-consistency-multi-tracker.md) §2
## Blocked by
- `12-c2-on-chain-idempotency.md`
- `14-a3-raft-durable-term-vote.md`

25
.scratch/alpha-hardening/issues/14-a3-raft-durable-term-vote.md Normal file
View File

@@ -0,0 +1,25 @@
Status: ready-for-human
# 14 — A3: Durable Raft term and vote state (deferred)
## What to build
Persist Raft `currentTerm`, `votedFor`, and log metadata to disk. In-memory-only term (~26) risks split leadership after tracker restart → duplicate settlement epochs.
**Code refs:**
- `packages/tracker/meshnet_tracker/raft.py` — `LogEntry.term` (~25–27), election state in `RaftNode`
## Acceptance criteria
- [ ] Term/vote persisted alongside tracker data dir
- [ ] Restart resumes as follower/candidate with monotonic term
- [ ] Test: kill leader mid-settlement, restart, no duplicate payout batch
## ADR links
- [ADR-0019](../../docs/adr/0019-money-path-consistency-multi-tracker.md) §3
## Blocked by
Alpha single-settlement posture

27
.scratch/alpha-hardening/issues/15-h1-commutative-forfeit.md Normal file
View File

@@ -0,0 +1,27 @@
Status: ready-for-human
# 15 — H1: Commutative forfeit event ordering (deferred)
## What to build
Define deterministic ordering when `forfeit`, `charge`, and `payout` events replicate concurrently. Forfeit snapshots amount at creation (~287) but apply order can desync pending balances under gossip.
**Code refs:**
- `packages/tracker/meshnet_tracker/billing.py` — `forfeit_pending` (~280–292), `_apply_locked` forfeit branch (~345–349)
- `packages/tracker/meshnet_tracker/billing.py` — `_pending_since.setdefault` (~324), wallet bind direct overwrite (~351)
## Acceptance criteria
- [ ] Documented commit order: charges before forfeit before payout for same wallet epoch
- [ ] Forfeit events carry pending snapshot or `(term, index)` for tie-break
- [ ] `setdefault` replaced with explicit merge rules on out-of-order apply
- [ ] Property tests under shuffled event delivery
## ADR links
- [ADR-0019](../../docs/adr/0019-money-path-consistency-multi-tracker.md) §4
## Blocked by
- `13-c3-c4-consensus-gated-settlement.md`

30
.scratch/alpha-hardening/issues/16-doc-us006-reconciliation.md Normal file
View File

@@ -0,0 +1,30 @@
Status: ready-for-agent
# 16 — DOC: US-006 reconciliation note
## What to build
Reconcile stale US-006 (Solana testnet stake contracts) with ADR-0015/0016 devnet custodial settlement. Issue `docs/issues/06-solana-stake-and-settlement.md` says "never devnet"; ADR-0015 explicitly targets devnet mock-USDT.
Also reconcile legacy fraud issues with the alpha-hardening fraud arc:
- `docs/issues/07-fraud-detection-slash.md` — on-chain stake slash model superseded by pending-balance forfeiture + TOPLOC (ADR-0018)
- `docs/issues/34-forfeiture-penalty.md` — partially implemented; remaining fraud work lives in `.scratch/alpha-hardening/issues/06-fraud-toploc-integration.md` through `10-fraud-penalty-calibration-wiring.md`
## Acceptance criteria
- [ ] Add reconciliation comment atop `docs/issues/06-solana-stake-and-settlement.md` (Status: superseded for alpha — see ADR-0015, issue 33/34)
- [ ] Add **superseded** banner atop `docs/issues/07-fraud-detection-slash.md` → ADR-0018 + issues 06–10
- [ ] Add **superseded for remaining scope** banner atop `docs/issues/34-forfeiture-penalty.md` → ADR-0018 + issues 06–10 (note done items: basic forfeiture wired)
- [ ] Update `docs/prd.json` US-006 description footnote if present
- [ ] Cross-link ADR-0015 devnet decision
- [ ] No production code changes
## ADR links
- [ADR-0015](../../docs/adr/0015-usdt-custodial-settlement.md)
- [ADR-0016](../../docs/adr/0016-alpha-scope-and-known-limitations.md)
## Blocked by
None

23
.scratch/alpha-hardening/issues/17-doc-duplicate-us020-dedup.md Normal file
View File

@@ -0,0 +1,23 @@
Status: ready-for-agent
# 17 — DOC: Duplicate US-020 issue dedup
## What to build
Two files share the US-020 number with different slugs:
- `docs/issues/20-memory-budget-shard-slots-and-dropout-relocation.md` (ready-for-agent)
- `docs/issues/20-tracker-node-hardening.md` (done)
Resolve numbering collision without losing history.
## Acceptance criteria
- [ ] Document canonical mapping in this issue's Comments or a short `docs/issues/README.md` note
- [ ] Renumber or prefix disambiguation (e.g. keep done item as US-020a, renumber memory-budget to next slot) — **human approval before git mv**
- [ ] Update any prd.json / cross-links that reference US-020 ambiguously
- [ ] No production code changes
## Blocked by
Human approval for renumbering

27
.scratch/alpha-hardening/issues/18-doc-operational-runbooks.md Normal file
View File

@@ -0,0 +1,27 @@
Status: ready-for-agent
# 18 — DOC: Operational runbooks (stubs)
## What to build
Add operational runbook stubs for alpha operators under `docs/runbooks/` (or `.scratch/alpha-hardening/runbooks/` until close-feature):
1. **Ledger backup** — billing SQLite, accounts SQLite, registry DB paths; gossip pause procedure
2. **Treasury key rotation** — devnet mock-USDT mint + treasury keypair rotation without double-credit
3. **Upgrade path** — tracker rolling restart with persisted strike/reputation (post issue 05)
## Acceptance criteria
- [ ] Three markdown runbook stubs with prerequisites, steps, rollback
- [ ] Reference ADR-0015 settlement loop and ADR-0016 trust assumptions
- [ ] Secrets handling: never commit `.env.devnet`, keypairs
- [ ] No production code changes
## ADR links
- [ADR-0015](../../docs/adr/0015-usdt-custodial-settlement.md)
- [ADR-0016](../../docs/adr/0016-alpha-scope-and-known-limitations.md)
## Blocked by
None (stubs can land before issue 05; update after persistence ships)

24
.scratch/alpha-hardening/issues/19-doc-cryptography-test-env.md Normal file
View File

@@ -0,0 +1,24 @@
Status: ready-for-agent
# 19 — DOC: Cryptography dependency + test environment note
## What to build
Document and verify test/dev environment setup for wallet crypto paths. `packages/node/meshnet_node/wallet.py` uses `cryptography`; failures occur when `.venv` lacks deps.
**Code refs:**
- `packages/node/pyproject.toml` — `cryptography>=41` (verify declared)
- `packages/node/meshnet_node/wallet.py`
- Handoff: tests fail without `cryptography`, `openai`, `langchain` in `.venv`
## Acceptance criteria
- [ ] Confirm `cryptography` in node package deps; add to root/dev extras if tests import wallet without node install
- [ ] Add short **Test environment** section to `CONTRIBUTING.md` or `docs/dev/test-env.md`: use `.venv/Scripts/python.exe`, `pip install -e packages/node ...`, optional dep skips
- [ ] Note which tests require optional deps (`--ignore=test_openai_gateway,...`)
- [ ] No unrelated production code changes
## Blocked by
None

52
.scratch/alpha-hardening/issues/20-validator-service-token.md Normal file
View File

@@ -0,0 +1,52 @@
Status: ready-for-agent
# 20 — Validator service token for `/v1/billing/forfeit`
## What to build
Define and implement a **validator service token** distinct from client API keys and admin sessions. The validator process must authenticate when calling `POST /v1/billing/forfeit`; arbitrary Bearer strings and client API keys must be rejected.
Per [ADR-0017 §4](../../docs/adr/0017-tracker-authentication-and-authorization.md): forfeit accepts **validator service identity or admin session** only.
## Configuration
| Item | Alpha default |
|---|---|
| Env var | `MESHNET_VALIDATOR_SERVICE_TOKEN` (tracker + validator) |
| Config flag | `--validator-service-token` / tracker config file equivalent |
| Header format | `Authorization: Bearer <service-token>` with a dedicated prefix or separate header scheme documented in runbooks (e.g. `Authorization: Service <token>` — pick one and test consistently) |
| Rotation | Manual: set new token on tracker + validator, restart both; document zero-downtime rotation as post-alpha |
## Rejection rules
- Client API keys (`sk-mesh-…`) → **403** on forfeit (even if valid for inference)
- Non-empty garbage Bearer → **401/403**
- Missing auth → **401**
- Valid validator service token → **200** (existing forfeit semantics)
- Admin session → **200** (operator override)
## Test-first
1. Red: validator (or test client) posts forfeit with a valid API key — must fail after fix.
2. Red: `Authorization: Bearer garbage` — must fail (covered by issue 02; this issue defines the accepted token).
3. Green: configured service token succeeds; wrong token fails.
## Acceptance criteria
- [ ] Service token configurable via env/flag on tracker and validator
- [ ] Unified auth middleware resolves service token → `validator` role (issue 02)
- [ ] API keys explicitly rejected on forfeit path
- [ ] Integration test: validator client with service token forfeit succeeds; API key forfeit fails
- [ ] Runbook stub: rotation procedure (manual alpha)
## ADR links
- [ADR-0017](../../docs/adr/0017-tracker-authentication-and-authorization.md) §4
## Related
- `02-a2-unified-auth-boundary.md` — middleware + role checks
## Blocked by
- `02-a2-unified-auth-boundary.md`

37
.scratch/alpha-hardening/issues/21-honest-noise-calibration-corpus.md Normal file
View File

@@ -0,0 +1,37 @@
Status: ready-for-human
# 21 — Honest-noise TOPLOC calibration corpus
## What to build
Before enabling production TOPLOC audit thresholds, collect an **honest-noise baseline** across the heterogeneous volunteer fleet. Run identical inference jobs on every active node/GPU combo; measure the divergence envelope (TOPLOC exponent/mantissa deltas, logprob-rank spread) under real hardware variance.
Per [ADR-0018 consequences](../../docs/adr/0018-fraud-detection-verification-and-reputation.md): threshold calibration requires an honest-noise corpus across the fleet before production thresholds.
Research anchor: `.scratch/alpha-hardening/research-verifiable-inference.md` §8 layer 3 — "collect this first — run identical jobs across the current node fleet to measure the honest divergence envelope before setting thresholds."
## Deliverables
- [ ] Scripted benchmark job (fixed prompt, model preset, seed policy) runnable on all nodes
- [ ] Aggregated corpus artifact (per node: GPU model, dtype, TOPLOC deltas vs reference)
- [ ] Recommended tolerance thresholds documented (p99 honest envelope + safety margin)
- [ ] Gate checklist: production audit enable blocked until corpus covers ≥N distinct hardware profiles (define N in runbook, suggest ≥3)
## Acceptance criteria
- [ ] Corpus collected from current fleet (or documented subset + extrapolation note)
- [ ] Threshold constants in validator config derived from corpus, not guessed
- [ ] False-positive rate estimate documented at chosen thresholds
- [ ] README / runbook cross-link: **do not enable production audits** until this issue closes
## ADR links
- [ADR-0018](../../docs/adr/0018-fraud-detection-verification-and-reputation.md) — Consequences (honest-noise corpus)
## Blocked by
- `06-fraud-toploc-integration.md` (TOPLOC wired; calibration uses same primitive)
## Blocks (prod gate)
- Production enable of adaptive audit thresholds (issues 09–10 in prod)

21
.scratch/alpha-hardening/issues/22-doc-memory-project-status.md Normal file
View File

@@ -0,0 +1,21 @@
Status: ready-for-agent
# 22 — DOC: MEMORY.md + project-status alpha-hardening index
## What to build
Update persistent memory files so agents and humans find the alpha-hardening feature without stale handoff paths.
## Acceptance criteria
- [ ] `.claude/memory/MEMORY.md` — index entry for alpha-hardening (`.scratch/alpha-hardening/`, ADRs 0016–0019, issue count)
- [ ] `.claude/memory/project-status.md` — brief alpha-hardening section: planning complete, Bucket 1 blockers next, link README
- [ ] Cross-link `.scratch/alpha-hardening/handoff.md` from README (not temp path)
## ADR links
- [ADR-0016](../../docs/adr/0016-alpha-scope-and-known-limitations.md)
## Blocked by
None — parallel with other Bucket 3 doc issues

281
.scratch/alpha-hardening/research-verifiable-inference.md Normal file
View File

@@ -0,0 +1,281 @@
# Verifiable Inference & Fraud Layer — Research Findings
**Date:** 2026-07-04
**Scope:** Ground the neuron-tai fraud/verification layer in 2026 state-of-the-art and in what comparable decentralized compute networks actually do.
**Alpha context assumed throughout:** volunteer consumer GPUs (8 GB class, e.g. RTX 4060), pipeline-parallel multi-hop inference where different nodes hold different layer ranges, a single trusted tracker (coordinator), devnet mock-USDT payments but reputation that carries forward, and an owner-approved verification budget of roughly 5% redundant recomputation.
---
## Executive summary
1. **The proven, deployed pattern for exactly our setting is: optimistic acceptance + random teacher-forced re-verification + slashing/reputation penalties.** This is what Prime Intellect runs in production for INTELLECT-2 (TOPLOC validators, random sampling, slash-and-evict) and what Hyperbolic formalized game-theoretically (Proof of Sampling). At a 5% audit rate the math works if the penalty for a caught cheat exceeds ~19× the per-job gain — which carried-forward reputation plus forfeited pending balance easily provides.
2. **Teacher-forced logit/activation re-verification is a recognized, robust technique** — it is the core of TOPLOC (ICML 2025), Token-DiFR (2025), and log-probability auditing papers. It sidesteps autoregressive divergence entirely and is up to ~100× cheaper than the original generation because verification prefill is compute-bound while generation is memory-bound.
3. **Bitwise output equality is unattainable across heterogeneous volunteer GPUs**; honest nodes on different hardware/batch conditions produce different tokens even at temperature 0. All robust schemes compare in *logit/activation space with tolerances* (TOPLOC exponent/mantissa thresholds, DiFR divergence scores) or force bitwise reproducibility with special kernels (Gensyn RepOps) at a performance cost.
4. **zkML and GPU TEEs are roadmap-only.** zkML in 2025–2026 proves GPT-2-scale models in tens of seconds per inference (100–10,000× overhead); nothing near serving speed for multi-billion-parameter models. GPU confidential computing exists only on H100/Blackwell-class datacenter hardware paired with server CPUs (SEV-SNP/TDX) — categorically unavailable on consumer volunteer cards.
5. **Layer-skipping is NOT reliably detectable by output quality.** Careful pruning of ~25% of layers keeps ~90% of benchmark scores (ShortGPT); models degrade gracefully until a sharp collapse around 20–55% removal depending on family. But any layer skipping changes hidden states and logits drastically, so per-position activation/logit comparison catches it essentially always (TOPLOC reports 100% detection of model modifications in its evals). Verify in logit space, not by eyeballing text quality.
6. **Classic PoW is Sybil/hardware admission control, not correctness proof** — io.net's hourly PoW is the cautionary tale (it verifies a GPU exists and has claimed VRAM, nothing about job correctness). Self-computed activation checksums are *binding commitments*, not correctness proofs: a cheater simply commits to its wrong values. Commitments earn their keep only when a referee later recomputes and compares — i.e., for audit pinning and dispute bisection.
---
## 1. Technique-by-technique review
### 1.1 Optimistic verification + spot-checking (probabilistic deterrence)
**Mechanism.** Accept results by default; re-run a random, unpredictable subset with an independent replica; punish divergences. Hyperbolic's Proof of Sampling (PoSP) paper proves a pure-strategy Nash equilibrium in which rational nodes are honest, and claims the spot-check approach adds well under 1% overhead when everyone is rational ([PoSP, arXiv:2405.00295](https://arxiv.org/html/2405.00295); [Hyperbolic spML breakdown](https://www.hyperbolic.ai/blog/spml-breakdown)). VeriLLM adds VRF-selected audit indices over Merkle-committed hidden states so verifiers can't be lazy or predictable ([VeriLLM, arXiv:2509.24257](https://arxiv.org/html/2509.24257v3)). Prime Intellect's production deployment: validators randomly sample committed batches, and "since the Inference Provider does not know which generations will be checked, they are incentivized to be honest on all generations" ([INTELLECT-2, arXiv:2505.07291](https://arxiv.org/html/2505.07291)).
**Sampling math.** With independent audit probability *p* per job:
- A node that cheats on every job survives *N* jobs undetected with probability (1−p)^N. At p = 0.05: ~36% survive 20 jobs, ~4.6% survive 60 jobs, ~0.6% survive 100 jobs. Detection is near-certain over any meaningful volume.
- A node that cheats on a fraction *f* of jobs is caught per job with probability *p·f*; expected jobs until caught = 1/(p·f). At p = 0.05, f = 0.2: caught within ~100 jobs on average. Low-rate cheating stretches time-to-detection linearly — this is why the penalty must scale with accumulated stake/reputation, not per-incident.
- **Deterrence condition (rational node):** cheating gains *g* per job and loses *L* when caught, so expected value of cheating is (1−p)·g − p·L. Honesty dominates iff **L > g·(1−p)/p ≈ 19·g at p = 0.05**. If a caught node forfeits its pending balance and a reputation that gates future routing (i.e., discounted future earnings), L is orders of magnitude above 19× a single job's margin. This is exactly the structure PoSP formalizes and the existing neuron-tai forfeiture design (pending-balance forfeiture + strikes + ban) already implements.
**The verifier-incentive caveat.** Truebit identified the classic failure mode: if the system works and nobody cheats, verifiers never earn anything and stop checking; Truebit's answer was "forced errors" with jackpot payouts, and it estimated the verification tax at 500–5000% of task cost to keep independent verifiers attentive ([Truebit whitepaper §2, §4](http://people.cs.uchicago.edu/~teutsch/papers/truebit.pdf)). **This whole problem disappears in a single-tracker alpha**: the tracker audits as a protocol function paid from the protocol cut, not as a profit-seeking third party. It becomes relevant again only when verification is decentralized.
**Feasibility for alpha: HIGH.** This is the anchor technique. 5% audit budget is in the same range as deployed systems (OTR proposes ρ ≈ 1% spot-checks as sufficient deterrent in a TEE-hybrid design — [arXiv:2512.20176](https://doi.org/10.48550/arxiv.2512.20176)); 5% gives comfortable margin for a young network with thin reputational stakes.
### 1.2 Interactive verification / refereed delegation (Truebit-style bisection)
**Mechanism.** Solver commits Merkle roots of intermediate execution states. On dispute, challenger and solver binary-search over the committed states to the first step where they diverge; a referee (smart contract or trusted party) recomputes only that single step and rules. One honest challenger suffices; the game costs O(log n) rounds ([Truebit whitepaper](http://people.cs.uchicago.edu/~teutsch/papers/truebit.pdf); [Truebit explainer, Medium](https://medium.com/truebit/truebit-the-marketplace-for-verifiable-computation-f51d1726798f)).
**ML adaptation.** Gensyn's Verde adapts this to neural networks with a two-phase bisection: first narrow to the training/inference *step*, then to the single *operator* in the computational graph; the referee recomputes one operator, needing ~two orders of magnitude less compute than the full job. Crucially, Verde only works because RepOps makes execution **bitwise reproducible across heterogeneous hardware** by fixing floating-point operation order — otherwise honest nodes diverge and bisection finds spurious "fraud" ([Verde paper, arXiv:2502.19405](https://arxiv.org/html/2502.19405v1); [Gensyn Verde blog](https://blog.gensyn.ai/verde-a-verification-system-for-machine-learning-over-untrusted-nodes/)).
**Mapping to our multi-hop pipeline.** Our pipeline is naturally pre-bisected: each hop's boundary activations are the "intermediate states." A workable scheme:
1. Each node commits a compact fingerprint (TOPLOC-style top-k encoding, not a raw hash — see §2) of its input and output boundary activations per request.
2. When an end-to-end audit fails, the referee (tracker) teacher-forces the full claimed token sequence through a reference model and computes reference boundary activations at each hop cut-point in one forward pass.
3. The first hop whose committed output fingerprint diverges from the reference (beyond tolerance) while its input fingerprint matched is the culprit. No interactive game is needed at hop granularity because the number of hops is small — the referee can check all cut-points in a single replay.
4. Bisection *within* a hop (to a layer or operator, Verde-style) is only needed if you must prove fault to a third party at fine granularity; for internal blame assignment, hop granularity is enough since payment and reputation are per node.
The main adaptation cost is the referee needing enough VRAM to run the full model (or to re-run hop layer-ranges one at a time, which any 8 GB card can do for its own range).
**Feasibility for alpha: MEDIUM-HIGH** in the simplified "referee replays and compares committed hop boundaries" form (no on-chain game, tracker as referee). The full trustless interactive game with bitwise-reproducible kernels is roadmap.
### 1.3 zkML (zero-knowledge proofs of inference)
**2025–2026 state.** The frontier moved fast but remains far from LLM serving speed:
- EZKL (ONNX→Halo2) is the most-used toolkit; generic overhead is characterized as 100–10,000× native execution, and much real usage is verifiable data science rather than LLMs ([Equilibrium Labs survey](https://equilibrium.co/writing/state-of-verifiable-inference)).
- zkGPT (USENIX Security 2025) proves a GPT-2 inference in **under 25 seconds** on a CPU server — 185× faster than the prior ZKML system ([zkGPT](https://www.usenix.org/conference/usenixsecurity25/presentation/qu-zkgpt)).
- NanoZK (2026) proves GPT-2-scale transformer blocks in ~43 s with 6.9 KB proofs via layerwise decomposition, 52–228× faster than EZKL ([NanoZK, arXiv:2603.18046](https://arxiv.org/html/2603.18046v1)).
- For a 7B model, one estimate puts pure-zkML verification at **over 20 minutes per inference** ([Optimistic TEE-Rollups, arXiv:2512.20176](https://doi.org/10.48550/arxiv.2512.20176)).
- Project reality check: Modulus Labs (authored the field-defining benchmark ["The Cost of Intelligence"](https://eprint.iacr.org/2026/1063), demonstrated proving of a multi-billion-parameter LLM as a feat, not a service) was acquired by Tools for Humanity in 2024 and now does World ID cryptography ([announcement](https://world.org/blog/announcements/modulus-labs-joins-tfh-support-applied-research-world)). Giza uses ZK proofs for *small, simple* DeFi-agent models on Starknet ([Equilibrium survey](https://equilibrium.co/writing/state-of-verifiable-inference)). EZKL itself has drifted toward verifiable data science and co-SNARKs.
**Honest read:** proving is ~GPT-2 scale (~1.5B params max in tens of seconds), on beefy servers, per short inference. For a volunteer network serving multi-billion-parameter models interactively, zkML is 3+ orders of magnitude away, and proof generation would crush an 8 GB consumer card regardless.
**Feasibility for alpha: NONE (roadmap-only).**
### 1.4 TEE / confidential computing on GPUs
**What it gives.** NVIDIA H100 is the first GPU with a hardware TEE anchored in an on-die root of trust: measured/secure boot, SPDM session with a driver inside a CPU confidential VM, and a signed **attestation report** covering GPU firmware/state that a remote party verifies against NVIDIA's attestation service before releasing work ([NVIDIA blog](https://developer.nvidia.com/blog/confidential-computing-on-h100-gpus-for-secure-and-trustworthy-ai/); [CACM: Creating the First Confidential GPUs](https://cacm.acm.org/practice/creating-the-first-confidential-gpus/)). Combined with CPU TEE attestation (AMD SEV-SNP or Intel TDX measuring the VM image), you get a verifiable claim that *a specific measured software stack loaded a specific model and ran on genuine hardware*, with only ~2–6% overhead (Blackwell figures, [GPUYard setup guide](https://www.gpuyard.com/tutorials/howto/nvidia-blackwell-confidential-computing-setup/)) or ~5–10% generally ([Equilibrium survey](https://equilibrium.co/writing/state-of-verifiable-inference)). Phala runs exactly this to sell "verifiable LLMs" through OpenRouter ([Phala blog](https://phala.com/posts/GPU-TEEs-is-Alive-on-OpenRouter)).
**Caveats.** Attestation proves the *environment*, not the mathematics — you trust NVIDIA/AMD/Intel hardware and are exposed to side channels and firmware bugs (the OTR paper layers ZK spot-checks on top of TEEs precisely because of compromised-TEE risk). And critically for us: CC mode requires Hopper/Blackwell datacenter GPUs plus server platforms with SEV-SNP/TDX enabled in BIOS and the open kernel modules ([NVIDIA deployment guide](https://docs.nvidia.com/cc-deployment-guide-tdx-snp.pdf)). **No RTX consumer card supports it.** Volunteer nodes on 4060-class hardware categorically cannot provide GPU TEE attestation; Intel SGX on client CPUs is likewise not a path (deprecated on consumer parts, and it would only cover CPU-side code anyway).
**Feasibility for alpha: NONE for volunteer nodes.** Roadmap option: a *TEE tier* — if datacenter H100/Blackwell operators ever join, attest them and route confidentiality-sensitive or high-value jobs there; also usable for the tracker's own reference validator.
---
## 2. The non-determinism problem for output comparison
### 2.1 Why honest nodes disagree
- **Floating-point non-associativity** is the root cause, but the *mechanism* that surfaces it is batch-variant kernels: inference servers pick different kernels/split strategies depending on batch size and load, changing reduction order and hence results — the same request returns different logits depending on what else is in the batch ([Thinking Machines: Defeating Nondeterminism in LLM Inference](https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/)).
- **Even greedy (temperature-0) decoding diverges across configurations**: changing batch size, GPU count, GPU architecture, or tensor-parallel size measurably changes outputs under greedy decoding, and small numerical differences get **amplified autoregressively** — one flipped token early in a long generation cascades into a completely different continuation ([arXiv:2506.09501](https://arxiv.org/html/2506.09501v2); [arXiv:2511.17826](https://arxiv.org/html/2511.17826v2)).
- Temperature > 0 sampling adds outright randomness on top unless seeds and the sampling algorithm (e.g. Gumbel-max in vLLM) are synchronized.
**Consequence:** naive "re-run the prompt, compare the text" audits will flag honest heterogeneous nodes as cheaters. Never compare free-running generations token-for-token across different hardware.
### 2.2 Techniques that make honest-vs-honest comparison robust
1. **Teacher-forced re-verification (per-position logit/activation check) — yes, this is a recognized, state-of-the-art technique.**
- **TOPLOC** (Prime Intellect, ICML 2025): the prover commits top-k values/indices of the last hidden state per decode chunk (258 bytes per 32 tokens, ~1000× smaller than raw activations); the validator *re-runs the claimed token sequence as a single prefill* (teacher forcing) and checks the committed top-k against recomputed values using exponent/mantissa error thresholds. Reported: 100% detection of model, prompt, or precision substitutions with zero false positives across different GPUs, tensor-parallel layouts, and attention implementations, and validation up to **100× faster than the original inference** because prefill is compute-bound while decode is memory-bound ([TOPLOC, arXiv:2501.16007](https://arxiv.org/html/2501.16007v1); [GitHub](https://github.com/PrimeIntellect-ai/toploc)).
- **Token-DiFR / Activation-DiFR** (2025): synchronize the sampling seed (Gumbel-max), then compare the provider's tokens against a reference re-generation — with a shared seed, >98% of tokens match exactly for honest providers, so token divergence itself becomes the audit signal; detects 4-bit quantization with AUC > 0.999 within 300 tokens. Activation-DiFR compresses activations with random orthogonal projections and detects the same with 2 tokens. Released as a vLLM integration ([DiFR, arXiv:2511.20621](https://arxiv.org/pdf/2511.20621); [author explainer](https://technicallyprivate.substack.com/p/token-difr-llm-inference-verification)).
- **Log-probability auditing**: comparing per-token logprob distributions against a reference model detects even single-step fine-tuning or quantization changes; simple statistical tests (permutation test on per-token mean logprobs, KS tests) overcome logprob noise ([Logprob Tracking, arXiv:2512.03816](https://arxiv.org/html/2512.03816v1); [model-substitution audit, arXiv:2504.04715](https://arxiv.org/html/2504.04715v1)).
2. **Tolerance-based matching, not bitwise equality.** TOPLOC accepts bounded exponent-intersection and mantissa-error deviations; DiFR scores divergence-from-reference against an honest-noise baseline. Both are explicitly designed so cross-GPU numerical noise passes while quantization/model swaps fail.
3. **Beware scalar aggregate thresholds alone.** The DiFR authors note that a single statistic like mean cross-entropy can be gamed — a malicious provider can tune its sampling temperature until the aggregate matches expectation. Seed-synchronized token matching or per-position top-k activation checks leave far fewer degrees of freedom ([DiFR explainer](https://technicallyprivate.substack.com/p/token-difr-llm-inference-verification)).
4. **Bitwise-deterministic kernels** (Gensyn RepOps; Thinking Machines' batch-invariant kernels; TBIK for cross-tensor-parallel invariance) make exact comparison possible but require replacing the whole operator stack and sacrifice performance — practical for a controlled verification environment, unrealistic to impose on volunteer nodes running stock llama.cpp/transformers ([Gensyn](https://blog.gensyn.ai/verde-a-verification-system-for-machine-learning-over-untrusted-nodes/); [Thinking Machines](https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/); [arXiv:2511.17826](https://arxiv.org/html/2511.17826v2)).
**Practical recipe for our audits:** record prompt, claimed output tokens, sampling params, and per-hop activation fingerprints. Audit = one teacher-forced forward pass of the claimed tokens on a reference node; compare per-position: (a) hop-boundary fingerprints TOPLOC-style, and/or (b) whether each claimed token is plausible under the reference distribution (rank/logprob within tolerance; exact match if seeds are synchronized). This is cheap (single prefill), robust to hardware noise, and immune to autoregressive divergence because the token sequence is fixed.
---
## 3. Layer-skipping detectability
**How gracefully do transformers degrade when layers are dropped?**
- **Careful pruning degrades subtly.** ShortGPT removes ~25% of layers (chosen by Block Influence, training-free) and keeps ~90% of benchmark performance — e.g. LLaMA2-13B MMLU drops only 55.0→52.2 after removing 10 of 40 layers ([ShortGPT, arXiv:2403.03853](https://arxiv.org/pdf/2403.03853)). Gromov et al. find a "characteristic flat region of robust performance" on QA benchmarks followed by a **sharp collapse to random accuracy** at a model-dependent threshold: ~45–55% of layers for Llama-2 family, ~35% for Mistral-7B, ~25% for Phi-2, ~20% for Qwen — and up to ~half of Llama-2-70B's layers with healing finetuning ([The Unreasonable Ineffectiveness of the Deeper Layers, arXiv:2403.17887](https://arxiv.org/html/2403.17887v1)). LayerDrop showed a decade of precedent: networks *trained* with layer dropout can be pruned at inference with modest loss ([arXiv:1909.11556](https://arxiv.org/abs/1909.11556)).
- **So: would skipping ~36 of 100 layers produce gibberish?** Probably not gibberish. For a Llama-like model, 36% removal of *well-chosen* layers sits below the collapse threshold — output would be degraded but often fluent, and on easy prompts might look acceptable. A lazy cheater skipping a contiguous middle block *without* choosing layers carefully does worse — Gromov et al. show the naive heuristic's loss "begins to increase very rapidly even with small amounts of pruning" before healing — but "worse" still means plausible-looking text on many prompts, not obvious garbage. **Human-eyeball or coarse quality checks are not a reliable fraud detector.**
- **Statistical detection: yes, and easily.** Without healing, C4 validation loss rises sharply with pruning fraction (Gromov et al. Fig. 3), so perplexity of the node's outputs under a reference model shifts detectably given enough tokens. But the far stronger signal is direct: skipping any layer changes the hidden states and output logits at *every* position by amounts vastly exceeding floating-point noise. TOPLOC's evaluation explicitly includes detecting modified models with 100% accuracy; DiFR detects even 4-bit quantization (a much subtler perturbation than deleting layers) with AUC > 0.999 in ≤300 tokens. **A single teacher-forced audit pass catches layer-skipping essentially deterministically.**
- **Pipeline nuance:** in multi-hop inference, a node that skips layers inside its assigned range corrupts its output boundary activations; the hop-boundary fingerprint comparison in §1.2 localizes exactly which node did it.
---
## 4. Why classic PoW does not solve verifiable useful computation
- **PoW's defining property is verification asymmetry over an *arbitrary* puzzle**: work is hard to produce, trivially cheap to check (hash preimage with leading zeros), and the puzzle content is irrelevant — it exists purely to price Sybil identities and block production ([Wikipedia: Proof of work](https://en.wikipedia.org/wiki/Proof_of_work)). Useful computation generally lacks this asymmetry: checking an ML result naively requires re-executing it, which is the whole problem. This is the central, repeatedly confirmed obstacle in the Proof-of-Useful-Work literature — "the main weakness that all presented PoUW approaches have in common is the verification of results" ([Challenges of PoUW, arXiv:2209.03865](https://doi.org/10.48550/arxiv.2209.03865); [SoK: Is Proof-of-Useful-Work Really Useful?, IACR 2025/1814](https://eprint.iacr.org/2025/1814.pdf)). Expensive verification then triggers the **verifier's dilemma**: rational verifiers skip checking, and security collapses ([Truebit whitepaper §1](http://people.cs.uchicago.edu/~teutsch/papers/truebit.pdf)).
- **Live case study:** io.net's hourly "Proof-of-Work" makes GPUs solve hash puzzles and proves *the hardware exists, has the claimed VRAM, and is online* ([io.net PoW docs](https://io.net/docs/guides/workers/proof-of-work)). It was bolted on after the April 2024 Sybil attack (~1.8M fake GPUs spoofed via a shared auth token to farm airdrops) and says nothing about whether any customer job was computed correctly ([io.net 2026 analysis](https://cryptoaianalysis.com/io-net-io-analysis-2026/)). PoW = admission control and capacity attestation; correctness needs a separate mechanism.
- **Self-computed checksums/commitments: confirmed — no standalone correctness guarantee.** A hash or Merkle commitment over activations is *binding* (the node can't later change its story) and possibly *hiding*, but the committed values are whatever the node computed — a malicious node honestly commits to its dishonest activations, and the checksum verifies perfectly. Correctness only enters when an independent party recomputes and compares against the commitment. That is precisely how the serious systems use commitments: Truebit's Merkle state roots and Verde's checkpoint hashes exist to make **dispute bisection** possible and to **pin claims before an unpredictable audit** (so a node can't retroactively fix its answer once it learns it's being checked), never as proof by themselves ([Truebit](http://people.cs.uchicago.edu/~teutsch/papers/truebit.pdf); [Verde, arXiv:2502.19405](https://arxiv.org/html/2502.19405v1); TOPLOC commitments are verified by validator recomputation — [arXiv:2501.16007](https://arxiv.org/html/2501.16007v1)).
---
## 5. What comparable networks actually do
| Network | Verification of outputs | Sybil / fraud handling | Notes |
|---|---|---|---|
| **Prime Intellect** | TOPLOC activation commitments per sequence; validators teacher-force-recompute (sampled, up to 100× faster than generation); plus sampling-sanity and data-sanity checks | Invalid batches → node **slashed and evicted** from compute pool; hardware checks at registration | Deployed at scale for INTELLECT-2 (32B RL run over permissionless nodes) ([arXiv:2505.07291](https://arxiv.org/html/2505.07291)) |
| **Gensyn** | Verde refereed delegation: bisect disputes to a single operator; referee recomputes it; requires RepOps bitwise-reproducible kernels | Correct result guaranteed if ≥1 of the assigned providers is honest; economic penalties on losers of disputes | The most rigorous trustless design; reproducibility stack is the price ([Verde](https://arxiv.org/html/2502.19405v1); [docs](https://docs.gensyn.ai/core-components)) |
| **Bittensor** | No direct output verification: subnet **validators score miners**; Yuma Consensus aggregates stake-weighted scores into emissions | Registration costs + stake; the notorious **weight-copying** problem (validators free-riding by copying consensus weights) countered by Commit-Reveal v3: timelock-encrypted weights revealed epochs later so copiers only get stale data ([weight copying](https://docs.learnbittensor.org/concepts/weight-copying-in-bittensor); [commit-reveal](https://docs.learnbittensor.org/concepts/commit-reveal)) | Verification quality is per-subnet and heuristic; consensus rewards conformity, not proven correctness |
| **io.net** | None for job outputs | Hourly hash-puzzle PoW + VRAM check + Proof of Time-Lock (uptime), added after the Apr 2024 fake-GPU Sybil attack; failed checks → excluded from rewards/hiring ([PoW docs](https://io.net/docs/guides/workers/proof-of-work); [Messari overview](https://messari.io/report/understanding-io-net-a-comprehensive-overview)) | Marketplace model: hardware attestation only, correctness left to the customer |
| **Ritual (Infernet)** | Modular: nodes may attach ZK, optimistic, or TEE proofs; consumer contracts opt in ([Ritual blog](https://ritual.net/blog/celestia)) | ChainLight's security review: base Infernet has **no aggregation of multiple nodes, no reputation, no penalty for malicious nodes returning wrong results** ([ChainLight](https://blog.chainlight.io/ecosystem-explorer-exploring-security-risks-in-ai-blockchain-projects-2c490a726d13)) | "Verification-optional" in practice |
| **Hyperbolic** | PoSP/spML: random spot-check by a second node; disagreement → arbitration; slashing of the dishonest party ([PoSP, arXiv:2405.00295](https://arxiv.org/html/2405.00295)) | Nash-equilibrium argument that rational nodes stay honest; validator anonymity until finalization to resist collusion | Closest published formalization of our intended scheme |
| **Akash** | None (general compute marketplace) | **Auditor-attested provider tiers**: governance-approved auditors post on-chain attestations; no provider staking/slashing — ChainLight flags malicious-provider risk as an accepted gap ([ChainLight](https://blog.chainlight.io/ecosystem-explorer-exploring-security-risks-in-ai-blockchain-projects-2c490a726d13); [Akash docs](https://akash.network/docs/providers/getting-started/should-i-run-a-provider/)) | Reputation = audit badge + market history |
| **Petals** (closest architectural cousin: volunteer, pipeline-parallel, consumer GPUs) | **None** — the paper and docs openly state malicious peers can alter outputs; mitigation is "use a private swarm of people you trust"; devs planned a centralized points system for nodes that "consistently return correct results" ([GitHub](http://github.com/bigscience-workshop/petals); [dev comments on HN](https://news.ycombinator.com/item?id=34215665)) | Proof that the architecture works, and that the fraud layer is the missing piece we're building |
| **Together AI** | Centralized provider; no public trustless verification protocol — trust is contractual/reputational | n/a | Included for contrast: the "trusted provider" baseline |
| **Atoma / Ambient** (emerging) | Atoma: TEE where available, else user-configurable redundant sampling; Ambient: "Proof of Logits" fingerprinting with ~0.1% claimed overhead but no open implementation ([Equilibrium survey](https://equilibrium.co/writing/state-of-verifiable-inference)) | — | Signal of where the field is converging: logit fingerprints + sampling |
**Pattern:** nobody in production uses zkML for LLM serving; nobody offers TEE on consumer GPUs; every serious inference-verification deployment converges on *commit → sampled teacher-forced recompute → slash/reputation*, with refereed delegation as the trustless escalation path.
---
## 6. Reputation systems for routing and audit-rate weighting
**Foundations.** EigenTrust computes global trust as the principal eigenvector of normalized local trust ratings and remains the canonical design; in simulation it suppresses malicious collectives up to large fractions of the network, but it depends on pre-trusted peers and majority-honest aggregation ([EigenTrust, WWW'03](https://nlp.stanford.edu/pubs/eigentrust.pdf)).
**Documented pitfalls** (from EigenTrust's own threat models and the P2P reputation-attack literature — [survey](https://cnitarot.github.io/papers/p2p-reputation-survey.pdf)):
- **Sybil flooding / whitewashing:** free identities let attackers dominate the "unknown newcomer" pool and shed bad reputations by rejoining. Defense: entry cost (stake, invite, proof-of-hardware), newcomer probation with low routing weight and *elevated* audit rates.
- **Reputation farming / milking:** behave honestly on cheap jobs to build score, then cash in on expensive ones ("moles" that interleave honest work are documented in real systems like Maze). Defense: value-weighted reputation (a job's reputation contribution ∝ its audited value), and audit probability that never floors at zero for anyone.
- **Collusion rings:** peers mutually inflating ratings. Defense: in our alpha, reputation should derive **only from tracker-verified audit outcomes**, never from peer ratings — this deletes the collusion surface EigenTrust suffers from. (EigenTrust++ hardens propagation with feedback-similarity for when peer ratings do get introduced — [EigenTrust++](https://doi.org/10.4108/icst.collaboratecom.2012.250420).)
- **Weight-copying (Bittensor's lesson):** any published evaluation signal will be free-ridden by lazy evaluators; the fix is delaying/encrypting evaluations (commit-reveal timelock) until they're stale ([Bittensor docs](https://docs.learnbittensor.org/concepts/weight-copying-in-bittensor)). Relevant to us only when third-party validators exist; a single tracker has no one to copy from.
**Reputation-weighted routing + audit rates in practice.** io.net gates hiring and block rewards on rolling verification status; Akash routes via audit-tier filters; Bittensor routes emissions via consensus-weighted scores. The clean design for us, consistent with PoSP-style deterrence:
- **Routing weight** increases with reputation (and observed TPS, already tracked), so honest nodes earn more — making L (loss-on-catch, §1.1) grow with tenure.
- **Audit rate** decreases with reputation but has a hard floor (e.g. new node 20–30% → veteran 2–3%, floor ≥ 2%), keeping the overall budget ≈ 5% while making cheating-as-a-newcomer maximally risky and reputation-then-defect still irrational.
- **Asymmetric dynamics:** slow to build, instant to lose (existing 3-strike forfeiture design already matches this), plus decay on inactivity so parked identities can't hoard trust.
- **Unpredictability is load-bearing:** audit selection must be indistinguishable from normal traffic at request time (same request format, decided by tracker RNG/VRF after commitment), or nodes cheat only on non-audited requests — the exact property both PoSP and INTELLECT-2 rely on.
---
## 7. Feasibility table for the neuron-tai alpha
Alpha constraints: volunteer 8 GB consumer GPUs, single trusted tracker, mock-USDT but real carried-forward reputation, ~5% audit budget, small models today (0.5B-class) scaling up.
| Technique | Alpha feasibility | Overhead | What it buys | Verdict |
|---|---|---|---|---|
| Optimistic accept + random teacher-forced audit (~5%) | **High** — tracker (or a trusted validator node) re-runs claimed tokens as one prefill | ~5% of serving compute; audits themselves up to ~100× cheaper than original generation | Detects wrong model, quantization, layer-skipping, garbage outputs; rational-cheating deterrence at L > 19× per-job gain | **Do now (anchor)** |
| Per-hop activation fingerprints (TOPLOC-style top-k, tolerance matching) | **High** — few hundred bytes per response per hop; nodes already stream activations | Negligible at inference; storage ~258 B/32 tokens/hop | Pins claims before audit; localizes blame to the exact hop on audit failure | **Do now** |
| Hop-boundary replay "bisection" (referee = tracker, non-interactive) | **Medium-high** — needs a reference node able to run the full model or per-hop ranges | Only on audit failure / dispute | Culprit identification in multi-hop pipelines; fair slashing | **Do now (simplified form)** |
| Seed-synchronized sampling (Token-DiFR-style exact-token audits) | **Medium** — requires controlling the sampler (Gumbel-max w/ shared seed) in node runtime | Zero at serve time | Strongest per-token evidence; >98% exact match for honest nodes | **Do if node runtime is ours; else logprob-rank tolerance** |
| Statistical logprob/perplexity monitoring (passive, on all traffic) | **High** — reference-free heuristics (output perplexity, repetition, truncation) + periodic logprob tests | Tiny | Cheap tripwire that raises a node's audit rate when outputs look off | **Do now (cheap complement)** |
| Reputation-weighted routing + reputation-weighted audit rate with floor | **High** — extends existing tracker reputation/strikes | None | Makes deterrence math work; concentrates audits on new/suspect nodes | **Do now** |
| Full interactive verification game (Truebit/Verde) with bitwise RepOps kernels | Low — requires replacing node operator stacks, decentralized referees | High engineering; runtime penalty for reproducible kernels | Trustless dispute resolution without a trusted tracker | **Roadmap (multi-tracker era)** |
| zkML proofs of inference | None — GPT-2-scale, tens of seconds per proof, server-class hardware ([zkGPT](https://www.usenix.org/conference/usenixsecurity25/presentation/qu-zkgpt); [NanoZK](https://arxiv.org/html/2603.18046v1)) | 100–10,000× | Cryptographic soundness | **Roadmap-only; re-evaluate yearly** |
| GPU TEE attestation | None on consumer cards (H100/Blackwell + SEV-SNP/TDX servers only — [NVIDIA guide](https://docs.nvidia.com/cc-deployment-guide-tdx-snp.pdf)) | ~2–10% where available | Hardware-rooted "right model+code ran" | **Roadmap: optional attested tier for datacenter contributors** |
| PoW-style hardware challenges (io.net-like) | Medium — trivial to add | Idle-time only | Sybil/VRAM/capacity attestation at registration; NOT output correctness | **Optional, registration-time only; don't confuse with fraud layer** |
---
## 8. Recommended layered scheme for alpha
1. **Commit layer (on-demand, audit-selected):** when a request is selected for audit, each hop signs and reports a TOPLOC-style top-k fingerprint of its output boundary activations plus the sampling metadata (params, seed if controlled). Client-visible response carries the full claimed token sequence. Commitments are cheap and make retroactive lying impossible — they are *audit pins, not proofs* (§4).[^alpha-on-demand]
[^alpha-on-demand]: Alpha implements **on-demand** commitments per [ADR-0018 §3](../../docs/adr/0018-fraud-detection-verification-and-reputation.md) — not every request. Nodes retain recent activations briefly; serving path stays uncommitted until audit selection.
2. **Audit layer (~5% of requests, VRF/tracker-RNG selected, indistinguishable ex ante):** a reference executor teacher-forces the claimed tokens in one prefill and checks (a) per-position token plausibility under the reference distribution (exact match if seed-synced; logprob-rank tolerance otherwise) and (b) hop-boundary fingerprints within TOPLOC-style exponent/mantissa tolerances. New/low-reputation nodes get 20–30% audit rates, veterans 2–3%, floor ≥ 2%, budget-balanced to ≈5% overall.
3. **Blame layer (on audit failure):** replay comparison across hop cut-points identifies the first divergent hop; that node eats the penalty (forfeit pending balance + strike, per the existing forfeiture design), and its audit rate snaps to maximum. Honest-noise false positives are handled by tolerance calibration against an honest-node baseline corpus (collect this first — run identical jobs across the current node fleet to measure the honest divergence envelope before setting thresholds).
4. **Reputation layer:** reputation derives exclusively from audit outcomes + uptime/latency, never peer ratings (no collusion surface). It weights routing (earnings) and inversely weights audit probability. Slow build, instant loss, decay on inactivity, entry probation for Sybil/whitewash resistance.
5. **Tripwire layer (all traffic, passive):** perplexity/repetition/truncation anomaly scoring on outputs; anomalies don't punish directly, they escalate the node's audit rate.
This is, deliberately, the Prime Intellect / Hyperbolic pattern adapted to multi-hop pipelines with the tracker as referee — every component has a deployed or peer-reviewed precedent.
## 9. Explicitly NOT feasible for alpha (roadmap-only)
- **zkML proofs of LLM inference** — 3+ orders of magnitude too slow for multi-billion-parameter serving; provers won't run on 8 GB cards. Revisit if layerwise/GKR provers (NanoZK, zkGPT lineage) reach real-time on 7B+ models.
- **GPU TEE / confidential-computing attestation on volunteer nodes** — hardware doesn't exist on consumer GPUs. Keep as an *attested datacenter tier* concept.
- **Fully trustless interactive verification games** (Truebit/Verde with on-chain referees and bitwise-reproducible kernels) — pointless while a single trusted tracker exists; becomes the correct escalation path when trackers decentralize. Requires shipping a reproducible-operator runtime to nodes.
- **Decentralized third-party verifier markets** (and their incentive pathologies: verifier's dilemma, forced-error jackpots, Truebit's 500–5000% verification tax) — single-tracker alpha sidesteps all of it; re-enters with multi-tracker.
- **Peer-rating reputation (EigenTrust-style transitive trust)** — collusion/Sybil surface with no benefit while the tracker is the sole verifier.
- **Commit-reveal timelock machinery for evaluator honesty** (Bittensor CRV3) — solves weight-copying among many validators; irrelevant until independent validators exist.
- **PoW puzzles as a correctness mechanism** — category error (§4); acceptable only as optional registration-time hardware attestation.
---
## Sources
**Papers / peer-reviewed:**
- Truebit: [A scalable verification solution for blockchains](http://people.cs.uchicago.edu/~teutsch/papers/truebit.pdf) (Teutsch & Reitwießner)
- Gensyn Verde: [arXiv:2502.19405](https://arxiv.org/html/2502.19405v1); [Gensyn blog](https://blog.gensyn.ai/verde-a-verification-system-for-machine-learning-over-untrusted-nodes/); [Gensyn docs](https://docs.gensyn.ai/core-components)
- TOPLOC: [arXiv:2501.16007](https://arxiv.org/html/2501.16007v1) (ICML 2025); [GitHub](https://github.com/PrimeIntellect-ai/toploc)
- INTELLECT-2: [arXiv:2505.07291](https://arxiv.org/html/2505.07291)
- Proof of Sampling (Hyperbolic): [arXiv:2405.00295](https://arxiv.org/html/2405.00295); [spML blog](https://www.hyperbolic.ai/blog/spml-breakdown)
- DiFR (Token/Activation): [arXiv:2511.20621](https://arxiv.org/pdf/2511.20621); [explainer](https://technicallyprivate.substack.com/p/token-difr-llm-inference-verification)
- Logprob tracking: [arXiv:2512.03816](https://arxiv.org/html/2512.03816v1); Model-substitution auditing: [arXiv:2504.04715](https://arxiv.org/html/2504.04715v1)
- VeriLLM: [arXiv:2509.24257](https://arxiv.org/html/2509.24257v3); Optimistic TEE-Rollups: [arXiv:2512.20176](https://doi.org/10.48550/arxiv.2512.20176)
- zkGPT: [USENIX Security 2025](https://www.usenix.org/conference/usenixsecurity25/presentation/qu-zkgpt); NanoZK: [arXiv:2603.18046](https://arxiv.org/html/2603.18046v1); Modulus "Cost of Intelligence": [IACR 2026/1063](https://eprint.iacr.org/2026/1063)
- Nondeterminism: [Thinking Machines blog](https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/); [arXiv:2506.09501](https://arxiv.org/html/2506.09501v2); [arXiv:2511.17826](https://arxiv.org/html/2511.17826v2); [arXiv:2511.00025](https://arxiv.org/html/2511.00025v1)
- Layer pruning: [arXiv:2403.17887](https://arxiv.org/html/2403.17887v1) (Gromov et al., ICLR 2025); [ShortGPT, arXiv:2403.03853](https://arxiv.org/pdf/2403.03853) (ACL 2025); LayerDrop: [arXiv:1909.11556](https://arxiv.org/abs/1909.11556)
- PoUW: [SoK, IACR 2025/1814](https://eprint.iacr.org/2025/1814.pdf); [arXiv:2209.03865](https://doi.org/10.48550/arxiv.2209.03865); [Wikipedia: Proof of work](https://en.wikipedia.org/wiki/Proof_of_work)
- EigenTrust: [WWW'03](https://nlp.stanford.edu/pubs/eigentrust.pdf); [EigenTrust++](https://doi.org/10.4108/icst.collaboratecom.2012.250420); [P2P reputation attack survey](https://cnitarot.github.io/papers/p2p-reputation-survey.pdf)
- Petals: [GitHub](http://github.com/bigscience-workshop/petals); [petals.dev](https://petals.dev/); [dev statements on incentives/correctness, HN](https://news.ycombinator.com/item?id=34215665)
**Official docs / engineering blogs:**
- NVIDIA CC: [H100 CC blog](https://developer.nvidia.com/blog/confidential-computing-on-h100-gpus-for-secure-and-trustworthy-ai/); [deployment guide (TDX/SNP)](https://docs.nvidia.com/cc-deployment-guide-tdx-snp.pdf); [CACM article](https://cacm.acm.org/practice/creating-the-first-confidential-gpus/); [Phala GPU-TEE on OpenRouter](https://phala.com/posts/GPU-TEEs-is-Alive-on-OpenRouter)
- Bittensor: [weight copying](https://docs.learnbittensor.org/concepts/weight-copying-in-bittensor); [commit reveal](https://docs.learnbittensor.org/concepts/commit-reveal); [OTF blog](https://blog.bittensor.com/weight-copying-in-bittensor-422585ab8fa5)
- io.net: [PoW docs](https://io.net/docs/guides/workers/proof-of-work); [Messari overview](https://messari.io/report/understanding-io-net-a-comprehensive-overview); [2026 risk analysis incl. Sybil attack history](https://cryptoaianalysis.com/io-net-io-analysis-2026/)
- Ritual: [ritual.net blog](https://ritual.net/blog/celestia); [ChainLight security review of AI-chain projects (Ritual, Akash)](https://blog.chainlight.io/ecosystem-explorer-exploring-security-risks-in-ai-blockchain-projects-2c490a726d13)
- Akash: [provider docs](https://akash.network/docs/providers/getting-started/should-i-run-a-provider/)
- Landscape survey: [Equilibrium Labs — State of Verifiable Inference](https://equilibrium.co/writing/state-of-verifiable-inference)
- Modulus Labs acquisition: [world.org announcement](https://world.org/blog/announcements/modulus-labs-joins-tfh-support-applied-research-world)
---
## Open-source reusability & build-vs-adopt
**Added 2026-07-04.** Focused follow-up: which of the projects above ship code we can actually reuse, and is verified decentralized inference a "solved problem"? All repo facts below checked against the GitHub repos/APIs on 2026-07-04.
### TOPLOC (Prime Intellect) — a real, adoptable library
- **Repo & license:** [PrimeIntellect-ai/toploc](https://github.com/PrimeIntellect-ai/toploc), **MIT** (both the repo license and the `pyproject.toml` declaration). Published on PyPI as `pip install toploc`. Separate [toploc-experiments](https://github.com/PrimeIntellect-ai/toploc-experiments) repo holds the paper's vLLM-integrated experiment code.
- **Maintenance:** actively maintained — created 2025-01-28, last push **2026-07-02** (two days before this writing), 56 stars / 12 forks / 12 open issues. Small single-purpose codebase (Python + a compiled extension; ships wheels, depends only on `torch`/`numpy`).
- **What it actually is:** a *library*, not a framework. Two function families ([README](https://github.com/PrimeIntellect-ai/toploc)):
- `build_proofs_bytes/base64(activations, decode_batching_size, topk, skip_prefill)` — prover side: takes a list of activation tensors (prefill tensor + one last-hidden-state tensor per generated token, bf16 in the examples) and returns compact proofs (~10 bytes each; 258 bytes per 32 tokens per the paper).
- `verify_proofs_bytes/base64(recomputed_activations, proofs, ...)` — verifier side: returns per-chunk `VerificationResult(exp_intersections, mant_err_mean, mant_err_median)`; **the accept/reject thresholds are ours to set and calibrate** (the paper gives reference thresholds validated across GPU types).
- **Assumptions:** model- and runtime-agnostic — it operates on tensors you extract yourself. It does *not* hook into vLLM/transformers for you; capturing the last hidden state per decode step in the node runtime, and recomputing activations via teacher-forced prefill on the validator, is our integration work. Detects precision changes by design, so **we must pin one canonical precision/quantization per served model** or honest quantized volunteers will fail verification.
- **Integration estimate for our pipeline-parallel case:** modest. (1) Node runtime: capture final-hidden-state per token, call `build_proofs_*`, attach to response — days, not weeks. (2) Tracker/validator: teacher-forced prefill re-run + `verify_proofs_*` — days. (3) The genuinely new part is *per-hop* commitments: TOPLOC as published commits only the **final** hidden state (whole-pipeline check, catches that fraud happened); applying the same encoding to hop-boundary activations for per-node blame is our own straightforward extension since the functions are tensor-generic, but its thresholds across heterogeneous 8 GB cards must be calibrated by us — that empirical honest-noise calibration is the real work. (4) Threshold calibration corpus across the volunteer fleet — the long pole, but required for any tolerance-based scheme regardless of library.
### Gensyn — papers and proprietary binaries, not a reusable verification library
What is actually public in [github.com/gensyn-ai](https://github.com/gensyn-ai) (org listing checked 2026-07-04):
- **[rl-swarm](https://github.com/gensyn-ai/rl-swarm)** — MIT, 1.7k stars: a framework for *RL training swarms*, plus [rl-swarm-contracts](https://github.com/gensyn-ai/rl-swarm-contracts) (coordination contracts). Not inference verification; not relevant to our fraud layer.
- **Verde (dispute resolution): paper only.** No `verde` repository exists in the org. The bisection/arbitration protocol from [arXiv:2502.19405](https://arxiv.org/html/2502.19405v1) has no published implementation. If we want it, we implement it from the paper.
- **RepOps / reproducible execution: shipping, but proprietary.** [repops-demo](https://github.com/gensyn-ai/repops-demo) is demo scripts with **no OSS license file** (only a `LICENSE-LLAMA` for the model weights) driving a prebuilt Docker image; supported targets include consumer RTX 3090/4070/4090 (CC 7.5–9, CUDA 12.6+). The successor [ree](https://github.com/gensyn-ai/ree) (Reproducible Execution Environment) is explicit in its README: the SDK wrapper is MIT, but **"the REE compiler binary and the REE reproducible-operators binary … are not licensed under the MIT License"** — they fall under a proprietary REE Binary License. REE does reproducible *LLM inference* with run/verify receipts, pipeline parallelism up to 72B (v0.2.0 notes), and needs Docker + NVIDIA driver ≥570. So bitwise cross-GPU determinism is real and demonstrably achievable on consumer GPUs — but as a closed-source appliance, not an embeddable library.
- **"Judge"** (verifiable AI evaluator, [docs](https://docs.gensyn.ai/core-components)) — no public library repo found in the org.
- **Assessment of the RepOps requirement:** deterministic cross-GPU execution means reimplementing/fixing the reduction order of every operator (matmul, attention, norms) for every hardware target — a compiler/kernel-engineering effort Gensyn keeps proprietary, and Thinking Machines' open batch-invariant kernels ([blog](https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/)) only address batch-invariance on one platform, not cross-GPU bitwise equality. **This is a large lift with no off-the-shelf OSS solution; it is exactly what TOPLOC-style tolerance matching lets us avoid.** Adopting REE would mean forcing volunteers into Gensyn's Docker appliance (proprietary binaries, driver floor, perf overhead) — a non-starter for our alpha; a possible future option for the *validator/referee* environment only.
### Bittensor — an incentive market, not verification code (confirmed)
- **Confirmed from primary sources:** the [whitepaper](https://bittensor.com/whitepaper) frames Bittensor as peers ranking peers ("intelligence measured by intelligence"), and the [Yuma Consensus docs](https://docs.learnbittensor.org/yuma-consensus) describe the on-chain mechanism precisely: each subnet validator submits a **weight vector ranking miners**; YC resolves the stake-weighted matrix into emissions, **clipping** any weight above the level supported by κ (default 0.5) of stake to punish out-of-consensus/collusive over-evaluation, and paying validators via EMA **bonds** that reward staying near consensus. There is **no cryptographic verification of any computation anywhere in the mechanism** — correctness of miner outputs is whatever each subnet's own validator code decides to score. The chain ([opentensor/subtensor](https://github.com/opentensor/subtensor), Rust, The Unlicense, actively developed) implements the consensus/emissions math, not inference checking.
- **What subnets actually do:** subnet owners write off-chain validator logic (Python, typically on the MIT-licensed [bittensor SDK](https://github.com/opentensor/bittensor)) that queries miners with tasks and scores responses — reference-model comparison, heuristics, or LLM-judging, entirely subnet-specific. Quality of validation is therefore uneven by construction.
- **Known weaknesses & mitigations (documented by Bittensor itself):** validator **weight-copying** (free-riding on the public weight matrix) — mitigated by Commit-Reveal v3 timelock-encrypted weights revealed epochs later ([weight copying](https://docs.learnbittensor.org/concepts/weight-copying-in-bittensor); [commit reveal](https://docs.learnbittensor.org/concepts/commit-reveal)); lazy/colluding validators — mitigated only economically via clipping and bond penalties; and the docs concede commit-reveal fails if miner rankings are too static.
- **Reusable for us:** the *design patterns* — stake-weighted score aggregation with clipping, EMA-smoothed trust, commit-reveal for future multi-validator honesty — not code. Subtensor's Rust consensus math is chain-embedded and solves a multi-validator problem we don't have while the single tracker is the only scorer.
### Other OSS in this niche
- **EZKL** ([zkonduit/ezkl](https://github.com/zkonduit/ezkl)) — **real, mature library** (Rust with Python/JS bindings, **Apache-2.0** per its [npm package](https://registry.npmjs.org/ezkl) and README, ~1.2k stars, last push 2026-02). It proves ONNX-graph inference in ZK and genuinely works — for *small* models (MLPs, small CNNs/regressors; §1.3 overheads apply). Irrelevant to serving multi-billion-parameter LLMs, but adoptable later if we ever need to prove a tiny model (e.g., a routing/scoring model) on-chain.
- **Petals** ([bigscience-workshop/petals](https://github.com/bigscience-workshop/petals)) — MIT, 10.3k stars, but **effectively dormant (last push Sep 2024)** and contains **zero trust machinery**: its own docs say malicious peers can alter outputs and recommend private swarms (§5). Its `hivemind` DHT/networking stack (MIT, also mirrored by Gensyn) is reusable *infrastructure*, not verification. Petals is validation of our architecture and a warning, not a component.
- **Hyperbolic PoSP/spML** — **paper + marketing only**. The [HyperbolicLabs GitHub org](https://github.com/HyperbolicLabs) contains agent kits, MCP servers, and a k8s OS — no PoSP/spML implementation is published. Adapt the pattern from [arXiv:2405.00295](https://arxiv.org/html/2405.00295); there is nothing to adopt.
- **Ritual Infernet** — the node ([ritual-net/infernet-node](https://github.com/ritual-net/infernet-node), Python, BSD-3-Clause-Clear per its community port; direct repo access was flaky at check time) plus [infernet-deploy](https://github.com/ritual-net/infernet-deploy)/[infernet-sdk](https://github.com/ritual-net/infernet-sdk) are open **coordination/oracle plumbing** (request routing, container orchestration, on-chain delivery). Proofs are a pluggable slot, not shipped verification logic, and ChainLight's review found no output aggregation, reputation, or penalties in the base system (§5). Nothing here advances our fraud layer.
- **DiFR** — the Token/Activation-DiFR authors state they released an **open-source vLLM integration** ([arXiv:2511.20621](https://arxiv.org/pdf/2511.20621)); worth tracking as a second adoptable audit primitive alongside TOPLOC, especially the seed-synchronized exact-token variant.
### Bottom line: is it solved?
**The audit primitive is solved and adoptable; the system around it is not.** No OSS project ships an end-to-end "verified decentralized inference network in a box" — every network either built theirs in-house (Prime Intellect), kept the hard part proprietary (Gensyn REE binaries), or doesn't verify at all (Petals, Akash, base Infernet, Bittensor's chain). What *is* genuinely reusable is small, high-quality, and MIT-licensed: TOPLOC. Everything else we need is a pattern to adapt or thin logic to build on our tracker, which is appropriately bespoke (it's our economics).
| Capability | Verdict | Component / reference | Rationale |
|---|---|---|---|
| **Detection / audit** (teacher-forced re-verification) | **ADOPT** | [`toploc`](https://github.com/PrimeIntellect-ai/toploc) (MIT, PyPI, maintained) — track [DiFR's vLLM integration](https://arxiv.org/pdf/2511.20621) as a complement | Proven encoding + verifier with cross-GPU tolerance semantics; our work is runtime hooks + threshold calibration, not algorithm development |
| **Blame attribution / bisection** (per-hop) | **ADAPT + BUILD** | Verde's commit-then-recompute pattern ([arXiv:2502.19405](https://arxiv.org/html/2502.19405v1)); reuse TOPLOC's encoding on hop-boundary activations | No published Verde code; full bisection is overkill for a handful of hops — a single referee replay over committed hop boundaries suffices (§1.2) |
| **Reputation / incentive** | **ADAPT + BUILD** | PoSP deterrence math ([arXiv:2405.00295](https://arxiv.org/html/2405.00295)); Yuma patterns (clipping, EMA trust, [commit-reveal](https://docs.learnbittensor.org/concepts/commit-reveal)) for the future multi-validator era; EigenTrust pitfalls list (§6) | No code to adopt (PoSP unpublished; Yuma is chain-embedded Rust for a different topology); our tracker-side logic is small and economics-specific |
| **Cryptographic proof** (zkML / TEE / bitwise-reproducible execution) | **NEITHER (roadmap)** | EZKL (Apache-2.0) if a tiny provable model ever needs it; Gensyn REE as a proprietary-appliance option for a future referee tier; GPU TEE per §1.4 | All three are unusable on 8 GB volunteer cards today (§1.3–1.4); REE's reproducible operators are closed-source binaries |
**Direct answer to the owner:** Gensyn and Bittensor are the right things to read but the wrong things to build on — Gensyn's verification core is a paper plus proprietary binaries aimed at training-grade trustlessness we don't need under a single trusted tracker, and Bittensor ships an incentive market that deliberately contains no computation verification at all. The one shelf-ready piece for our exact audit step is TOPLOC (MIT, pip-installable, updated this week), and the recommended alpha scheme in §8 remains: adopt TOPLOC for the audit primitive, adapt PoSP/Verde patterns for deterrence and blame, and build the thin tracker-side reputation/audit-rate logic ourselves.

4
docs/adr/0002-dual-token-payment-model.md
View File

@@ -1,5 +1,9 @@
# TAI token: revenue-backed rewards for nodes, USDT/SOL payments for clients
## Status: Accepted (settlement mechanics superseded by ADR-0015)
> **Settlement update (2026-07-04):** Alpha and near-term production use **USDT-direct custodial settlement** with pending-balance forfeiture penalties (ADR-0015). TAI emission, backing-price buyback, and stake deposits described below remain the **roadmap tokenomics** — not the live payment path. When TAI ships, it will mint from accumulated protocol cut; this ADR's incentive structure still governs long-term design.
## Core principle
Nodes must get paid reliably for inference work. This is the primary growth engine. All tokenomics decisions serve this goal first. The model is intentionally simple and sound — parameters can be tweaked later, but the structure must have no fatal flaws from day one.

4
docs/adr/0010-p2p-gossip-and-nat-relay.md
View File

@@ -16,10 +16,12 @@ The reference implementation (Petals) solves this with libp2p — GossipSub for
### 1. TLS everywhere
All HTTP between nodes, tracker, and gateway uses HTTPS (TLS 1.3). Self-signed certificates are auto-generated on first node start and stored in `~/.config/meshnet/`. The certificate fingerprint is included in every heartbeat and gossip envelope. Nodes use TOFU (trust on first use) — they accept a peer's cert on first contact and pin the fingerprint; connections from the same peer with a different fingerprint are rejected.
**Intent:** All HTTP between nodes, tracker, and gateway uses HTTPS (TLS 1.3). Self-signed certificates are auto-generated on first node start and stored in `~/.config/meshnet/`. The certificate fingerprint is included in every heartbeat and gossip envelope. Nodes use TOFU (trust on first use) — they accept a peer's cert on first contact and pin the fingerprint; connections from the same peer with a different fingerprint are rejected.
The relay node uses a real CA-signed certificate (Let's Encrypt) because it is the internet-facing bootstrap point.
> **Alpha reality correction (2026-07-04, ADR-0016):** As implemented, **TLS applies to the relay path** (`packages/relay`, WSS circuit proxy) and P2P packages that generate certs — **not** to plain HTTP on tracker ↔ node inference and registration in typical LAN/dev deployments. Tracker registration already requires `https://` URLs for production trackers, but node-to-tracker proxy traffic may still be plaintext on alpha. Full end-to-end TLS remains the target; alpha documents this as a **known limitation**, not a silent guarantee. Hive gossip auth (ADR-0017) is the immediate mitigation for tracker replication integrity.
### 2. mDNS for LAN peer discovery
Python `zeroconf` library. Service type: `_meshnet._tcp.local.`. A node announces itself on startup and browses for existing peers. This is zero-config discovery for home and lab networks. mDNS does not traverse routers, which is correct — LAN discovery should not bleed into the internet.

52
docs/adr/0016-alpha-scope-and-known-limitations.md Normal file
View File

@@ -0,0 +1,52 @@
# ADR-0016: Alpha scope and known limitations
## Status: Accepted
## Context
Pre-release audits found the routing layer is solid (ADR-0013) but the money and trust paths are not alpha-ready: unauthenticated gossip, free starting credit, node-reported accounting, ephemeral strike/ban state, and plaintext tracker HTTP undermine release. The owner locked an **alpha scope** that ships real inference and devnet settlement while explicitly deferring multi-tracker trustlessness and cryptographic verification.
## Decisions
### 1. Single settlement tracker
One operator-designated tracker holds the treasury keypair and runs the settlement loop (ADR-0015). Third-party trackers may join the hive for routing and state replication but **never** sign payouts. Raft elects a leader for shard-assignment commands only; settlement leadership is operator-configured, not democratic.
### 2. Open node join
Any wallet may register a node without invite, stake deposit, or hardware attestation. Anti-sybil cost comes from probation (first N jobs unpaid), carried-forward reputation, and fraud penalties — not from closed admission.
### 3. Devnet mock-USDT
Development and alpha target **Solana devnet** with a self-created mock-USDT SPL mint (ADR-0015). Mainnet USDT is a config change, not an architecture change. TAI settlement (ADR-0002) remains deferred; the 10% protocol cut accumulates as future TAI liquidity.
### 4. Reputation carries forward
Strike count, ban status, graduated reputation, and audit history **persist across tracker restarts** and are intended to survive alpha → mainnet cutover. A banned operator cannot evade penalties by restarting the tracker or re-registering under a fresh node id with the same wallet.
### 5. Known alpha limitations (explicit, not bugs)
| Limitation | Alpha posture | Post-alpha path |
|---|---|---|
| Tracker is trusted referee for audits and blame | Acceptable — see ADR-0018 | Decentralized verifier market; Verde-style on-chain games |
| Plaintext HTTP on tracker/node paths | Acceptable on LAN/dev; relay TLS only (ADR-0010 amended) | Full TLS + cert pinning per ADR-0010 intent |
| Unauthenticated gossip replication | **Blocker** — fix before alpha (ADR-0017) | HMAC/mTLS between hive members |
| zkML / GPU TEE verification | Roadmap-only | ADR-0018 §9; re-evaluate yearly |
| Multi-tracker money-path safety | Design now (ADR-0019), implement later | Consensus-gated settlement, idempotent payouts |
| Bitwise output equality across GPUs | Not required — logit/activation tolerance (TOPLOC) | Optional Gensyn RepOps tier for referee |
### 6. Fraud must be bounded
Because reputation carries forward, correctness fraud (bad outputs) and accounting fraud (inflated tokens/shard spans) must be **economically irrational** at the locked ~5% audit budget. Detection, blame, and penalty design are ADR-0018; implementation is Bucket 1 in `.scratch/alpha-hardening/`.
## Consequences
- Alpha release criteria are Bucket 1 blockers in `.scratch/alpha-hardening/README.md`, not "all 35 user stories done."
- Documentation and runbooks must state trust assumptions plainly — custodial treasury, trusted tracker, devnet funds.
- Bucket 2 (multi-tracker) issues are tracked but not alpha-blocking.
## Related
- Supersedes the spirit of prototype-only trust in ADR-0003 for production alpha behavior — see ADR-0018.
- Settlement mechanics: ADR-0015.
- Research grounding: `.scratch/alpha-hardening/research-verifiable-inference.md`.

75
docs/adr/0017-tracker-authentication-and-authorization.md Normal file
View File

@@ -0,0 +1,75 @@
# ADR-0017: Tracker authentication and authorization
## Status: Accepted
## Context
The tracker exposes three overlapping trust domains:
1. **Client API access** — Bearer API keys for inference and billing (partially gated today).
2. **Operator/admin access** — session-based accounts in `packages/tracker/meshnet_tracker/accounts.py` (registration, login, admin listing).
3. **Hive replication** — gossip endpoints that mutate billing, accounts, and stats without authentication today.
Financial and registry endpoints leak through inconsistent checks: some handlers require only a non-empty `Authorization` header (header-presence stub), while gossip accepts arbitrary peer events. Alpha requires a **single auth boundary** before any state mutation or privileged read.
## Decisions
### 1. Account subsystem (`accounts.py`)
- First registered account is **admin**; subsequent accounts are `user`.
- Login identifiers: email **or** wallet address + PBKDF2-SHA256 password.
- API keys (`sk-mesh-…`) are scoped to an account; revocation is event-sourced and gossip-replicated.
- **Sessions are local** (bearer tokens in memory, 7-day TTL) — not replicated across the hive. Each tracker validates its own sessions.
- Account mutations persist to SQLite and replicate via `/v1/accounts/gossip` events.
### 2. Client inference auth
- `POST /v1/chat/completions` requires a valid, non-revoked API key and positive ledger balance (402 if broke).
- **Starting credit is 0** for new API keys (ADR-0016 / issue C5). No implicit faucet; clients must deposit mock-USDT via bound wallet (US-032) or admin credit.
### 3. Gossip auth (alpha blocker)
All hive mutation endpoints require **authenticated peer identity**:
- `/v1/billing/gossip` — billing event replication
- `/v1/accounts/gossip` — account/key replication
- `/v1/stats/gossip` — rolling RPM merge
Acceptable alpha mechanisms (pick one in implementation):
- **Shared hive secret** (HMAC over request body + timestamp, configured on all trackers), or
- **Mutual TLS** between tracker peers (extends ADR-0010 relay TLS pattern).
Followers must reject events from unauthenticated callers. Read-only endpoints (`/dashboard`, `/v1/stats` GET) may remain public on LAN alpha deployments.
**Out of scope (alpha):** `POST /v1/gossip` — node throughput / peer fan-out gossip (`server.py` ~1331), distinct from hive mutation endpoints above. Document as **unauthenticated alpha limitation**; authenticate in a future ADR amendment when node identity binding exists (issue 01 tracks hive endpoints only).
### 4. Privileged operator endpoints
These require **real** auth — not header presence alone:
| Endpoint | Required role |
|---|---|
| `POST /v1/billing/forfeit` | Validator service identity or admin session |
| `POST /v1/benchmark/hop-penalty` | Admin session or service token |
| `GET /v1/benchmark/results` | Admin session or service token |
| `GET /v1/admin/accounts` | Admin session |
| `GET /v1/billing/summary`, `/v1/billing/settlements`, `/v1/registry/wallets` | Admin session (alpha: restrict financial reads) |
The unified auth middleware resolves: API key → account; session token → account + role; service token → validator; hive HMAC → peer.
### 5. Node registration
Node `POST /v1/register` is open (ADR-0016) but **banned wallets are rejected** at registration and excluded from routing. Wallet binding for client deposits (`POST /v1/wallet/register`) requires a valid API key and cryptographic ownership proof (issue C6).
## Consequences
- Gossip authentication (C1) and unified auth boundary (A2) are prerequisites for fraud penalties and billing fixes — an attacker can otherwise forge credits, strikes, or forfeit events.
- Session locality means admin login is per-tracker URL; document in runbooks.
- Multi-tracker auth evolution (client-signed commands) is deferred to ADR-0019.
## Related
- ADR-0016 (alpha scope)
- ADR-0015 (billing gate on inference)
- `.scratch/alpha-hardening/issues/01-c1-gossip-auth.md`, `02-a2-unified-auth-boundary.md`

92
docs/adr/0018-fraud-detection-verification-and-reputation.md Normal file
View File

@@ -0,0 +1,92 @@
# ADR-0018: Fraud detection, verification, and reputation
## Status: Accepted
## Context
ADR-0003 established optimistic sampling with stake slashing; ADR-0015 replaced stake with **pending-balance forfeiture** as collateral. Pre-alpha audits identified two distinct fraud types:
1. **Correctness fraud** — wrong model, layer-skipping, garbage outputs.
2. **Accounting fraud** — inflated token counts or shard-span work units reported by nodes.
The validator today compares final text only and always blames the last hop (`_final_text_node` in `packages/validator/meshnet_validator/__init__.py` ~137–140) — wrong for multi-hop pipelines.
Research (`.scratch/alpha-hardening/research-verifiable-inference.md`, 2026-07-04) grounds the alpha design in deployed patterns (Prime Intellect TOPLOC, Hyperbolic PoSP, Gensyn Verde blame **patterns**). This ADR is the flagship decision record for alpha hardening.
## Decisions
### 1. Anchor technique: optimistic accept + teacher-forced audit
- Default audit probability **p ≈ 5%** — a **budget target**, not a hard cap. Anomalies, low reputation, and disputes escalate rate; veterans floor at ≥ 2% (research §6, §8).
- **Deterrence condition:** at p = 0.05, penalty L must exceed **L > g·(1−p)/p ≈ 19×** per-job gain g (research §1.1). **Full pending forfeiture** is the primary penalty; three strikes → ban. The ×0.8-per-strike multiplier applies to **routing/payout weight** (reputation decay), not to the forfeiture amount.
- Single-tracker alpha: the tracker (or a designated reference node) is the auditor — no verifier market, no verifier's dilemma (research §1.1).
### 2. Detection primitive: ADOPT TOPLOC
- **`pip install toploc`** (MIT, [PrimeIntellect-ai/toploc](https://github.com/PrimeIntellect-ai/toploc)) for activation fingerprint commit + verify (research build-vs-adopt table).
- Teacher-forced prefill re-verification — compare in **logit/activation space with tolerances**, never free-running token equality (research §2).
- Pin **one canonical precision/quantization** per served model; TOPLOC detects precision drift by design.
- Per-hop boundary fingerprints extend TOPLOC's final-hidden-state encoding for **multi-hop blame** (research §1.2, §8 layer 1).
### 3. Commit layer: on-demand activation commitments
- Nodes commit compact TOPLOC-style fingerprints of **output boundary activations** per hop when selected for audit (on-demand, not every request — lower serving latency; brief retention window for recent activations).
- Commitments are **audit pins, not proofs** — correctness requires independent recomputation (research §4).
### 4. Blame layer: hop-boundary bisection (adapt Verde pattern)
On audit failure:
1. Referee (tracker) teacher-forces claimed token sequence through reference model.
2. Compare committed hop-boundary fingerprints to reference at each cut-point.
3. **First divergent hop** is the culprit — fixes `_final_text_node` last-hop-only bug.
4. Full interactive Truebit/Verde on-chain game and bitwise RepOps kernels: **roadmap-only** (research §1.2, §9).
### 5. Accounting fraud: tracker-authoritative metering
- **Token counts** come from the tracker's proxied stream/non-stream response parsing (`server.py` ~1890–1943), not node self-reports.
- **Work units** derive from **tracker-assigned shard span** at route construction (`server.py` ~1776–1782), not node-declared ranges at billing time.
- See issue H2.
### 6. Reputation model (graduated, persisted)
Reputation derives **only from tracker-verified audit outcomes** + uptime/latency — never peer ratings (research §6, collusion surface).
| Signal | Effect |
|---|---|
| Clean audits | Slow reputation build; higher routing weight |
| Strike | ×0.8 routing multiplier per strike (graduated decay) |
| Failed audit | Full pending forfeiture + strike; audit rate → maximum |
| Ban (3 strikes) | Registration rejected; excluded from routes; pending never paid |
| New/low reputation | Elevated audit rate (20–30% target for newcomers) |
| Inactivity | Reputation decay |
Persist strike/ban/reputation in SQLite alongside billing (issue A1/A5). Probation (first N jobs unpaid) retained as re-entry cost.
### 7. Passive tripwires
Perplexity/repetition/truncation heuristics on all traffic raise audit rate without direct punishment (research §8 layer 5).
### 8. Roadmap-only (explicitly NOT alpha)
- zkML proofs of LLM inference (research §1.3)
- GPU TEE attestation on consumer cards (research §1.4)
- Fully trustless Verde interactive games + RepOps bitwise kernels (research §9)
- Decentralized verifier markets
- Peer-rating reputation (EigenTrust)
- PoW as correctness proof — registration-time hardware attestation only, optional (research §4)
## Consequences
- Validator must be rewired: TOPLOC verify, hop blame, tracker-authoritative events — not string compare on final text alone.
- Threshold calibration requires an **honest-noise corpus** across the volunteer fleet before production thresholds (research §8).
- ADR-0003 remains historical; penalty mechanics follow ADR-0015 + this ADR.
- Implementation order: auth + persistence → accounting → TOPLOC → bisection → reputation routing (`.scratch/alpha-hardening/README.md`).
## Related
- Research: `.scratch/alpha-hardening/research-verifiable-inference.md` (§8 recommended scheme, §9 roadmap, build-vs-adopt table)
- ADR-0015 (forfeiture collateral)
- ADR-0016 (alpha scope)
- ADR-0017 (validator/forfeit auth)
- Issues: `06-fraud-toploc-integration.md` through `10-fraud-penalty-calibration-wiring.md`

66
docs/adr/0019-money-path-consistency-multi-tracker.md Normal file
View File

@@ -0,0 +1,66 @@
# ADR-0019: Money-path consistency (multi-tracker)
## Status: Accepted (design); implementation deferred post-alpha
## Context
ADR-0015 assumes a **single settlement tracker** with Raft-replicated ledger events. The hive already gossips billing, accounts, and stats between trackers. When multiple trackers participate, several money-path races appear:
- Settlement retries may double-pay if idempotency keys diverge from on-chain reality.
- Any tracker follower can apply `forfeit` / `charge` events — fine under one trusted operator, dangerous with adversarial peers.
- Raft term/vote state is in-memory — restart can duplicate leadership epochs.
- `forfeit` events are not commutative with `charge`/`payout` ordering under concurrent replication.
Alpha ships with **one settlement-capable tracker** (ADR-0016). This ADR records the accepted multi-tracker design so Bucket 2 issues do not block alpha while preventing silent architectural drift.
## Decisions
### 1. Settlement idempotency (C2)
- Every on-chain payout batch carries a **stable `settlement_id`** already debiting pending before broadcast (`server.py` ~3353–3356, `_send_settlement` ~3358).
- **Required hardening:** `solana_adapter.send_payouts` (~186–213) and the settlement loop must treat the Solana transaction signature as the idempotency anchor — retries reuse the same `settlement_id`; confirmed signatures must never spawn a second debit.
- Deposit watcher already uses `deposit-<sig>` event ids — extend the same pattern to payouts.
### 2. Consensus-gated money mutations (C3/C4)
Money-affecting events (`charge`, `payout`, `forfeit`, `credit`, `settlement`, `bind`) must commit through **Raft** (or successor consensus), not best-effort gossip alone:
- Only the Raft leader appends money commands to the replicated log.
- Followers apply committed entries in order; gossip becomes a transport, not the source of truth.
- Settlement signing remains **leader-only** and **treasury-key-bearing** (ADR-0015).
Shard-assignment commands already use Raft (`raft.py`); extend the log command set.
### 3. Durable Raft metadata (A3)
Persist `currentTerm`, `votedFor`, and log index to disk (`raft.py` ~26 — today term lives only in memory). Without this, restart can elect split leaders and duplicate settlement epochs.
### 4. Commutative forfeit semantics (H1)
Define explicit ordering rules when replicating `forfeit` vs `charge`/`payout`:
- Forfeit amount is snapshotted at event creation time (`billing.py` ~287).
- Apply forfeit **after** pending balance reflects all prior committed charges in the same epoch, or use signed pending snapshots in the event payload.
- `setdefault` on `_pending_since` (~324) and wallet bindings (~351) must not resurrect stale state on out-of-order apply — use last-writer-wins with `(term, index)` tie-break.
### 5. Alpha posture
| Item | Alpha | Post-alpha |
|---|---|---|
| Single treasury holder | Required | Optional multi-sig |
| Gossip-authenticated billing | Required (ADR-0017) | Superseded by Raft commit for money |
| On-chain idempotency | Best-effort resend | Hard guarantee (C2) |
| Consensus-gated settlement | Leader check only (~3331) | Full Raft gate (C3/C4) |
## Consequences
- Bucket 2 issues in `.scratch/alpha-hardening/` are **tracked but not alpha-blocking**.
- Implementing C3/C4 before alpha would delay release without security benefit while only one operator runs settlement.
- When enabling third-party settlement trackers, **all four** hardening items must ship together — partial deployment is worse than single-tracker alpha.
## Related
- ADR-0015 (settlement loop, pending collateral)
- ADR-0016 (single settlement tracker for alpha)
- ADR-0017 (gossip auth prerequisite)
- Issues: `12-c2-on-chain-idempotency.md` through `15-h1-commutative-forfeit.md`

2
docs/issues/07-fraud-detection-slash.md
View File

@@ -1,4 +1,4 @@
Status: ready-for-agent
Status: superseded for alpha — see [ADR-0018](../adr/0018-fraud-detection-verification-and-reputation.md) and `.scratch/alpha-hardening/issues/06-fraud-toploc-integration.md` through `10-fraud-penalty-calibration-wiring.md`. On-chain stake slashing replaced by pending-balance forfeiture (ADR-0015).
# 07 — Fraud detection: validator + on-chain slash

2
docs/issues/34-forfeiture-penalty.md
View File

@@ -1,4 +1,4 @@
Status: done
Status: done (core forfeiture wired); remaining fraud scope superseded — see [ADR-0018](../adr/0018-fraud-detection-verification-and-reputation.md) and `.scratch/alpha-hardening/issues/06-fraud-toploc-integration.md` through `10-fraud-penalty-calibration-wiring.md`.
# 34 — Hardened proof-of-work: pending-balance forfeiture penalty