feat(alpha): complete hardening backlog

Complete the alpha-hardening Ralph task set, including tracker billing/accounting guards, validator fraud-audit primitives, wallet binding proof support, documentation runbooks, and updated tests.

Verification: .venv/bin/python -m compileall -q packages tests; .venv/bin/python -m pytest -q --tb=short (313 passed, 3 skipped, 1 failed: tests/test_mining_cli.py::test_legacy_start_without_port_uses_next_available_port because meshnet-node pid 1263451 is already listening on port 7000).
This commit is contained in:
Dobromir Popov
2026-07-05 21:47:23 +03:00
parent c967e5cfc4
commit 9abe83b5f4
45 changed files with 4095 additions and 774 deletions

View File

@@ -1,4 +1,4 @@
Status: ready-for-agent
Status: done
# 03 — C5 + M1: Starting credit 0, funded-account gate, spend cap

View File

@@ -1,4 +1,4 @@
Status: ready-for-agent
Status: done
# 04 — H2: Tracker-authoritative token and work-unit accounting

View File

@@ -1,4 +1,4 @@
Status: ready-for-agent
Status: done
# 05 — A1/A5: Persist strike, ban, and reputation state

View File

@@ -1,4 +1,4 @@
Status: ready-for-agent
Status: done
# 06 — FRAUD: TOPLOC integration (teacher-forced audit primitive)
@@ -29,11 +29,11 @@ Pin one canonical precision/quantization per model preset. Add `toploc` to valid
## Acceptance criteria
- [ ] `toploc` dependency declared; `build_proofs_*` / `verify_proofs_*` wired
- [ ] Validator re-runs claimed token sequence as prefill, not free generation
- [ ] Model preset documents canonical dtype/quantization
- [ ] README updated: 19× deterrence at 5% audit (research §1.1)
- [ ] Tests with deterministic stub tensors (no GPU required in CI)
- [x] `toploc` dependency declared; `build_proofs_*` / `verify_proofs_*` wired
- [x] Validator re-runs claimed token sequence as prefill, not free generation
- [x] Model preset documents canonical dtype/quantization
- [x] README updated: 19× deterrence at 5% audit (research §1.1)
- [x] Tests with deterministic stub tensors (no GPU required in CI)
## ADR links

View File

@@ -1,4 +1,4 @@
Status: ready-for-agent
Status: done
# 07 — FRAUD: On-demand commitment + hop bisection blame
@@ -20,11 +20,11 @@ On audit selection, require nodes to supply TOPLOC-style fingerprints of **outpu
## Acceptance criteria
- [ ] Audit requests carry tracker RNG/VRF flag indistinguishable from normal traffic (research §6)
- [ ] Nodes retain recent boundary activations for on-demand commit window (configurable TTL)
- [ ] Validator/tracker compares fingerprints at each hop cut-point; first mismatch = culprit
- [ ] `_final_text_node` removed or limited to text-only fallback
- [ ] Integration test: multi-hop pipeline, fault injected at known hop
- [x] Audit requests carry tracker RNG/VRF flag indistinguishable from normal traffic (research §6) — the existing post-hoc `sample_rate` RNG gate in `ValidatorProcess.validate_once` already decides audit selection after the original proxied request completed, so the request the client/nodes saw is unaffected either way; locked in by `test_hop_commitments_are_not_requested_unless_the_event_is_audit_selected`
- [x] Nodes retain recent boundary activations for on-demand commit window (configurable TTL)`ToplocAuditConfig.commitment_ttl_seconds`; expired commitments fall back to the text-only path (`test_expired_commitment_window_falls_back_to_text_only_audit`)
- [x] Validator/tracker compares fingerprints at each hop cut-point; first mismatch = culprit`_hop_commitments_from_event` + `_first_divergent_hop` in `packages/validator/meshnet_validator/__init__.py`
- [x] `_final_text_node` removed or limited to text-only fallback — only called from the plain-text divergence branch of `_validate_event` now
- [x] Integration test: multi-hop pipeline, fault injected at known hop`tests/test_hop_bisection.py`
## ADR links

View File

@@ -1,4 +1,4 @@
Status: ready-for-agent
Status: done
# 08 — FRAUD: Reputation model + persistence

View File

@@ -1,4 +1,4 @@
Status: ready-for-agent
Status: done
# 09 — FRAUD: Reputation-weighted routing + adaptive audit rate

View File

@@ -1,4 +1,4 @@
Status: ready-for-agent
Status: done
# 10 — FRAUD: Penalty calibration wiring (forfeit + strike + ban)
@@ -23,11 +23,11 @@ Per ADR-0018: **full pending forfeiture** is primary penalty; ×0.8 is routing d
## Acceptance criteria
- [ ] Audit failure triggers forfeiture + strike in one tracker transaction
- [ ] Banned nodes excluded from `payables` / settlement
- [ ] Validator uses authenticated forfeit endpoint (issue 02)
- [ ] README: `L > 19× g` at p=0.05; pending balance = collateral
- [ ] Integration test: 60-request fraud scenario → ban within threshold
- [x] Audit failure triggers forfeiture + strike in one tracker transaction`ValidatorProcess._slash_node` (in-process) and the tracker's `_handle_billing_forfeit` handler (remote) both forfeit-then-strike synchronously in a single call path; each already existed pre-AH-010 and is exercised by `tests/test_forfeiture_penalty.py`
- [x] Banned nodes excluded from `payables` / settlement`BillingLedger.settle_node_payout` now clamps to the wallet's *current* pending balance under the same lock as the debit, and `_settlement_loop` rechecks ban status and uses the post-clamp amount before sending, so a forfeiture landing between the `payables()` snapshot and the actual payout can never be paid out on top of (ADR-0015 race); covered by `test_60_request_stream_bans_intermittent_first_hop_cheater_not_last_hop`
- [x] Validator uses authenticated forfeit endpoint (issue 02)`POST /v1/billing/forfeit` is validator-token/admin-gated (ADR-0017 §4, issue 20) and is the documented remote path (`packages/validator/README.md` Usage section); `test_forfeit_endpoint_requires_auth_and_forfeits` exercises the 401→200 flow. No standalone remote-validator process exists in this codebase yet (`contracts` has no networked implementation), so the in-process `ValidatorProcess` continues to call `BillingLedger.forfeit_pending` directly when co-located with the tracker — adding an HTTP-only forfeit client with no real consumer was judged out of scope/overengineering for this issue
- [x] README: `L > 19× g` at p=0.05; pending balance = collateral — already present in `packages/validator/README.md` ("Why the penalty deters cheating")
- [x] Integration test: 60-request fraud scenario → ban within threshold`tests/test_forfeiture_penalty.py::test_60_request_stream_bans_intermittent_first_hop_cheater_not_last_hop`
## ADR links

View File

@@ -1,4 +1,4 @@
Status: ready-for-agent
Status: done
# 11 — C6: Wallet binding ownership proof + binding overwrite safety
@@ -21,10 +21,10 @@ Require signed message from wallet pubkey (ed25519 via `cryptography` / solders)
## Acceptance criteria
- [ ] Wallet binding requires cryptographic proof of pubkey ownership
- [ ] One wallet → one API key (or documented admin override)
- [ ] Gossip `bind` events cannot overwrite existing binding via direct overwrite at `~351`
- [ ] Tests with deterministic keypairs (local adapter)
- [x] Wallet binding requires cryptographic proof of pubkey ownership
- [x] One wallet → one API key (or documented admin override)
- [x] Gossip `bind` events cannot overwrite existing binding via direct overwrite at `~351`
- [x] Tests with deterministic keypairs (local adapter)
## ADR links

View File

@@ -1,4 +1,4 @@
Status: ready-for-agent
Status: done
# 16 — DOC: US-006 reconciliation note

View File

@@ -1,4 +1,4 @@
Status: ready-for-agent
Status: done
# 18 — DOC: Operational runbooks (stubs)

View File

@@ -1,4 +1,4 @@
Status: ready-for-agent
Status: done
# 19 — DOC: Cryptography dependency + test environment note
@@ -14,11 +14,18 @@ Document and verify test/dev environment setup for wallet crypto paths. `package
## Acceptance criteria
- [ ] Confirm `cryptography>=41` remains in node package deps; add to root/dev extras only if tests import wallet without node install
- [ ] Add short **Test environment** section to `docs/dev/test-env.md` (or `CONTRIBUTING.md` if created): use `.venv/Scripts/python.exe`, `pip install -e packages/node ...`, optional dep skips
- [ ] Note which tests require optional deps (`--ignore=test_openai_gateway,...`)
- [ ] No unrelated production code changes
- [x] Confirm `cryptography>=41` remains in node package deps; add to root/dev extras only if tests import wallet without node install
- [x] Add short **Test environment** section to `docs/dev/test-env.md` (or `CONTRIBUTING.md` if created): use `.venv/Scripts/python.exe`, `pip install -e packages/node ...`, optional dep skips
- [x] Note which tests require optional deps (`--ignore=test_openai_gateway,...`)
- [x] No unrelated production code changes
## Blocked by
None
## Resolution
- `packages/node/pyproject.toml` already declared `cryptography>=41` — no change needed.
- `conftest.py` adds every `packages/*` dir to `sys.path`, so first-party imports (e.g. `meshnet_node.wallet`) resolve without an editable install of that package — but third-party deps like `cryptography` still must be installed separately. Added `cryptography>=41` to the root `pyproject.toml` `dev` extra so `pip install -e ".[dev]"` alone covers the wallet tests (`test_node_startup.py`, `test_wallet_binding_proof.py`, `test_devnet_treasury.py`, etc.) without requiring a full `packages/node` install (which would otherwise pull in torch/transformers/accelerate/bitsandbytes).
- Added `docs/dev/test-env.md` with setup instructions (Linux + Windows `.venv\Scripts\python.exe`), and a note on optional-dependency tests: `test_real_model_backend.py` / `test_devnet_treasury.py` use `pytest.importorskip` and skip cleanly; `test_openai_gateway.py` hard-imports `openai`/`langchain_openai` with no skip guard (both already in the `dev` extra) — documented the `--ignore=tests/test_openai_gateway.py` fallback for minimal installs.
- Full suite: 311 passed, 3 skipped, 3 pre-existing failures unrelated to this issue (`test_billing_ledger.py::test_proxy_chat_splits_payout_by_tracker_assigned_route_span`, `test_forfeiture_penalty.py::test_probation_earns_nothing_then_earning_begins`, `test_mining_cli.py::test_legacy_start_without_port_uses_next_available_port` — port-in-use env artifact). Wallet-specific tests (`test_wallet_binding_proof.py`, `test_node_startup.py`, `test_devnet_treasury.py`): 50 passed, 2 skipped.

View File

@@ -5,8 +5,8 @@
"userStories": [
{
"id": "AH-001",
"title": "01 \u2014 C1: Authenticate hive gossip endpoints",
"description": "# 01 \u2014 C1: Authenticate hive gossip endpoints\n\n## What to build\n\nAdd authenticated peer identity to all tracker gossip mutation endpoints. Today any caller can push billing, account, and stats events without verification.\n\n**Code refs:**\n\n- `packages/tracker/meshnet_tracker/server.py` \u2014 `_handle_billing_gossip` (~2414\u20132427)\n- `packages/tracker/meshnet_tracker/server.py` \u2014 `_handle_accounts_gossip` (~2610\u20132623)\n- `packages/tracker/meshnet_tracker/server.py` \u2014 `_handle_stats_gossip` (~2355\u20132364)\n- `packages/tracker/meshnet_tracker/billing.py` \u2014 `apply_events` (~301\u2013311)\n- `packages/tracker/meshnet_tracker/accounts.py` \u2014 `apply_events` (~220\u2013226)\n\nImplement per ADR-0017 \u00a73 using the auth helper/config from issue 02: shared hive HMAC (body + timestamp) or mutual TLS between configured tracker peers. Reject unauthenticated gossip with 401.\n\n**Note:** `/v1/gossip` (node throughput fan-out, `server.py` ~1331) is **not** in scope for this issue \u2014 see ADR-0017 \u00a73 out-of-scope note.\n\n## Test-first\n\n1. Red: unauthenticated POST to `/v1/billing/gossip` applies a credit event today \u2014 test must fail after fix.\n2. Red: authenticated peer with valid HMAC applies events; invalid/missing auth returns 401 and `applied: 0`.\n3. Green: wire the issue-02 verifier/config (`--hive-secret` or peer cert paths) into the three hive mutation endpoints.\n\n## Acceptance criteria\n\n- [ ] `/v1/billing/gossip`, `/v1/accounts/gossip`, `/v1/stats/gossip` reject requests without valid hive auth\n- [ ] Authenticated peers replicate events as today (id-dedup preserved)\n- [ ] Config documented for multi-tracker dev setups\n- [ ] Tests cover reject + accept paths without live network\n\n## ADR links\n\n- [ADR-0017](../../docs/adr/0017-tracker-authentication-and-authorization.md)\n- [ADR-0016](../../docs/adr/0016-alpha-scope-and-known-limitations.md)\n\n## Blocked by\n\n- `02-a2-unified-auth-boundary.md` \u2014 owns shared auth middleware/config. Implement in the same PR if simpler.",
"title": "01 C1: Authenticate hive gossip endpoints",
"description": "# 01 C1: Authenticate hive gossip endpoints\n\n## What to build\n\nAdd authenticated peer identity to all tracker gossip mutation endpoints. Today any caller can push billing, account, and stats events without verification.\n\n**Code refs:**\n\n- `packages/tracker/meshnet_tracker/server.py` `_handle_billing_gossip` (~24142427)\n- `packages/tracker/meshnet_tracker/server.py` `_handle_accounts_gossip` (~26102623)\n- `packages/tracker/meshnet_tracker/server.py` `_handle_stats_gossip` (~23552364)\n- `packages/tracker/meshnet_tracker/billing.py` `apply_events` (~301311)\n- `packages/tracker/meshnet_tracker/accounts.py` `apply_events` (~220226)\n\nImplement per ADR-0017 §3 using the auth helper/config from issue 02: shared hive HMAC (body + timestamp) or mutual TLS between configured tracker peers. Reject unauthenticated gossip with 401.\n\n**Note:** `/v1/gossip` (node throughput fan-out, `server.py` ~1331) is **not** in scope for this issue see ADR-0017 §3 out-of-scope note.\n\n## Test-first\n\n1. Red: unauthenticated POST to `/v1/billing/gossip` applies a credit event today test must fail after fix.\n2. Red: authenticated peer with valid HMAC applies events; invalid/missing auth returns 401 and `applied: 0`.\n3. Green: wire the issue-02 verifier/config (`--hive-secret` or peer cert paths) into the three hive mutation endpoints.\n\n## Acceptance criteria\n\n- [ ] `/v1/billing/gossip`, `/v1/accounts/gossip`, `/v1/stats/gossip` reject requests without valid hive auth\n- [ ] Authenticated peers replicate events as today (id-dedup preserved)\n- [ ] Config documented for multi-tracker dev setups\n- [ ] Tests cover reject + accept paths without live network\n\n## ADR links\n\n- [ADR-0017](../../docs/adr/0017-tracker-authentication-and-authorization.md)\n- [ADR-0016](../../docs/adr/0016-alpha-scope-and-known-limitations.md)\n\n## Blocked by\n\n- `02-a2-unified-auth-boundary.md` owns shared auth middleware/config. Implement in the same PR if simpler.",
"acceptanceCriteria": [
"`/v1/billing/gossip`, `/v1/accounts/gossip`, `/v1/stats/gossip` reject requests without valid hive auth",
"Authenticated peers replicate events as today (id-dedup preserved)",
@@ -18,7 +18,6 @@
],
"priority": 1,
"passes": true,
"status": "done",
"notes": "Source issue: .scratch/alpha-hardening/issues/01-c1-gossip-auth.md",
"dependsOn": [
"AH-002"
@@ -27,12 +26,12 @@
},
{
"id": "AH-002",
"title": "02 \u2014 A2: Unified auth boundary for privileged and financial reads",
"description": "# 02 \u2014 A2: Unified auth boundary for privileged and financial reads\n\n## What to build\n\nReplace header-presence stubs with a single auth middleware that resolves API keys, admin sessions, validator service tokens, and hive peer identity. Close leaks on financial and operator endpoints. This is the auth foundation issue; issue 01 should only apply hive auth to gossip endpoints once the helper exists.\n\n**Code refs:**\n\n- `packages/tracker/meshnet_tracker/server.py` \u2014 `_handle_billing_forfeit` (~2429\u20132464) \u2014 H3: non-empty `Authorization` only\n- `packages/tracker/meshnet_tracker/server.py` \u2014 `_handle_benchmark_hop_penalty` (~2650\u20132658), `_handle_benchmark_results` (~2745\u20132748) \u2014 H3\n- `packages/tracker/meshnet_tracker/server.py` \u2014 `_handle_billing_summary` (~2366\u20132371) \u2014 H4\n- `packages/tracker/meshnet_tracker/server.py` \u2014 `_handle_billing_settlements` (~2407\u20132412) \u2014 H4\n- `packages/tracker/meshnet_tracker/server.py` \u2014 `_handle_registry_wallets` (~2391\u20132405) \u2014 H4\n- `packages/tracker/meshnet_tracker/server.py` \u2014 `_session_account` (~2468+), `_handle_admin_accounts` (~2588\u20132608) \u2014 H4\n- `packages/tracker/meshnet_tracker/accounts.py` \u2014 `session_account()`, `create_session()` only (session store; not handler wiring)\n\nPer ADR-0017 \u00a74: forfeit \u2192 validator or admin; benchmark \u2192 admin; billing summary/settlements/registry wallets \u2192 admin session. Include the validator service token shape from `20-validator-service-token.md` in the same implementation if practical.\n\n## Test-first\n\n1. Red: POST `/v1/billing/forfeit` with `Authorization: Bearer garbage` succeeds today \u2014 must require validator/admin identity.\n2. Red: GET `/v1/billing/summary` without admin session returns 401/403.\n3. Green: middleware + role checks; existing inference API-key path unchanged.\n\n## Acceptance criteria\n\n- [ ] Single `_require_auth(role=...)` (or equivalent) used by all privileged handlers\n- [ ] Shared auth config supports admin sessions, validator service token, and hive peer HMAC/mTLS\n- [ ] Forfeit accepts only validator service token or admin session \u2014 not arbitrary Bearer strings\n- [ ] Financial read endpoints require admin session (alpha posture)\n- [ ] Benchmark write/read require admin or service token\n- [ ] Integration tests for each endpoint class (reject unauth, accept valid)\n\n## ADR links\n\n- [ADR-0017](../../docs/adr/0017-tracker-authentication-and-authorization.md)\n\n## Related\n\n- `20-validator-service-token.md` \u2014 checklist for validator service token format, rotation, forfeit auth\n\n## Blocked by\n\nNone. This issue should land before `01-c1-gossip-auth.md`.",
"title": "02 A2: Unified auth boundary for privileged and financial reads",
"description": "# 02 A2: Unified auth boundary for privileged and financial reads\n\n## What to build\n\nReplace header-presence stubs with a single auth middleware that resolves API keys, admin sessions, validator service tokens, and hive peer identity. Close leaks on financial and operator endpoints. This is the auth foundation issue; issue 01 should only apply hive auth to gossip endpoints once the helper exists.\n\n**Code refs:**\n\n- `packages/tracker/meshnet_tracker/server.py` `_handle_billing_forfeit` (~24292464) — H3: non-empty `Authorization` only\n- `packages/tracker/meshnet_tracker/server.py` `_handle_benchmark_hop_penalty` (~26502658), `_handle_benchmark_results` (~27452748) — H3\n- `packages/tracker/meshnet_tracker/server.py` `_handle_billing_summary` (~23662371) — H4\n- `packages/tracker/meshnet_tracker/server.py` `_handle_billing_settlements` (~24072412) — H4\n- `packages/tracker/meshnet_tracker/server.py` `_handle_registry_wallets` (~23912405) — H4\n- `packages/tracker/meshnet_tracker/server.py` `_session_account` (~2468+), `_handle_admin_accounts` (~25882608) — H4\n- `packages/tracker/meshnet_tracker/accounts.py` `session_account()`, `create_session()` only (session store; not handler wiring)\n\nPer ADR-0017 §4: forfeit validator or admin; benchmark admin; billing summary/settlements/registry wallets admin session. Include the validator service token shape from `20-validator-service-token.md` in the same implementation if practical.\n\n## Test-first\n\n1. Red: POST `/v1/billing/forfeit` with `Authorization: Bearer garbage` succeeds today must require validator/admin identity.\n2. Red: GET `/v1/billing/summary` without admin session returns 401/403.\n3. Green: middleware + role checks; existing inference API-key path unchanged.\n\n## Acceptance criteria\n\n- [ ] Single `_require_auth(role=...)` (or equivalent) used by all privileged handlers\n- [ ] Shared auth config supports admin sessions, validator service token, and hive peer HMAC/mTLS\n- [ ] Forfeit accepts only validator service token or admin session not arbitrary Bearer strings\n- [ ] Financial read endpoints require admin session (alpha posture)\n- [ ] Benchmark write/read require admin or service token\n- [ ] Integration tests for each endpoint class (reject unauth, accept valid)\n\n## ADR links\n\n- [ADR-0017](../../docs/adr/0017-tracker-authentication-and-authorization.md)\n\n## Related\n\n- `20-validator-service-token.md` checklist for validator service token format, rotation, forfeit auth\n\n## Blocked by\n\nNone. This issue should land before `01-c1-gossip-auth.md`.",
"acceptanceCriteria": [
"Single `_require_auth(role=...)` (or equivalent) used by all privileged handlers",
"Shared auth config supports admin sessions, validator service token, and hive peer HMAC/mTLS",
"Forfeit accepts only validator service token or admin session \u2014 not arbitrary Bearer strings",
"Forfeit accepts only validator service token or admin session not arbitrary Bearer strings",
"Financial read endpoints require admin session (alpha posture)",
"Benchmark write/read require admin or service token",
"Integration tests for each endpoint class (reject unauth, accept valid)",
@@ -42,15 +41,14 @@
],
"priority": 2,
"passes": true,
"status": "done",
"notes": "Source issue: .scratch/alpha-hardening/issues/02-a2-unified-auth-boundary.md",
"dependsOn": [],
"completionNotes": "Marked done in source alpha-hardening issue file before PRD generation."
},
{
"id": "AH-003",
"title": "03 \u2014 C5 + M1: Starting credit 0, funded-account gate, spend cap",
"description": "# 03 \u2014 C5 + M1: Starting credit 0, funded-account gate, spend cap\n\n## What to build\n\nClose the free-credit faucet. New API keys start at **0 USDT**; inference requires a real deposit or admin credit. Add a configurable per-request spend cap (M1) to limit runaway charges on compromised keys.\n\n**Code refs:**\n\n- `packages/tracker/meshnet_tracker/billing.py` \u2014 `DEFAULT_STARTING_CREDIT = 1.0` (~22), `ensure_client` (~73\u201385), `has_funds` (~87\u201388), duplicate credit on charge (~130\u2013138)\n- `packages/tracker/meshnet_tracker/server.py` \u2014 billing gate before routing (~1667\u20131690)\n\nPer ADR-0017 \u00a72 and ADR-0016 \u00a73.\n\n## Test-first\n\n1. Red: new API key gets 1.0 USDT implicit credit \u2014 test expects 0 balance until deposit.\n2. Red: first inference without deposit returns 402.\n3. Green: `DEFAULT_STARTING_CREDIT = 0.0`; optional `--max-charge-per-request` config.\n\n## Acceptance criteria\n\n- [ ] `DEFAULT_STARTING_CREDIT` is 0.0; no automatic caller credit on first touch\n- [ ] `has_funds` false for fresh keys; 402 before routing (server.py ~1684)\n- [ ] Admin `credit_client` or bound-wallet deposit still funds accounts\n- [ ] Configurable max charge per request (M1) rejects oversize completions with clear error\n- [ ] Tests: fresh key blocked; after credit/deposit, inference proceeds\n\n## ADR links\n\n- [ADR-0017](../../docs/adr/0017-tracker-authentication-and-authorization.md)\n- [ADR-0016](../../docs/adr/0016-alpha-scope-and-known-limitations.md)\n- [ADR-0015](../../docs/adr/0015-usdt-custodial-settlement.md)\n\n## Blocked by\n\n- `02-a2-unified-auth-boundary.md` (admin credit path secured)",
"title": "03 C5 + M1: Starting credit 0, funded-account gate, spend cap",
"description": "# 03 C5 + M1: Starting credit 0, funded-account gate, spend cap\n\n## What to build\n\nClose the free-credit faucet. New API keys start at **0 USDT**; inference requires a real deposit or admin credit. Add a configurable per-request spend cap (M1) to limit runaway charges on compromised keys.\n\n**Code refs:**\n\n- `packages/tracker/meshnet_tracker/billing.py` `DEFAULT_STARTING_CREDIT = 1.0` (~22), `ensure_client` (~7385), `has_funds` (~8788), duplicate credit on charge (~130138)\n- `packages/tracker/meshnet_tracker/server.py` billing gate before routing (~16671690)\n\nPer ADR-0017 §2 and ADR-0016 §3.\n\n## Test-first\n\n1. Red: new API key gets 1.0 USDT implicit credit test expects 0 balance until deposit.\n2. Red: first inference without deposit returns 402.\n3. Green: `DEFAULT_STARTING_CREDIT = 0.0`; optional `--max-charge-per-request` config.\n\n## Acceptance criteria\n\n- [ ] `DEFAULT_STARTING_CREDIT` is 0.0; no automatic caller credit on first touch\n- [ ] `has_funds` false for fresh keys; 402 before routing (server.py ~1684)\n- [ ] Admin `credit_client` or bound-wallet deposit still funds accounts\n- [ ] Configurable max charge per request (M1) rejects oversize completions with clear error\n- [ ] Tests: fresh key blocked; after credit/deposit, inference proceeds\n\n## ADR links\n\n- [ADR-0017](../../docs/adr/0017-tracker-authentication-and-authorization.md)\n- [ADR-0016](../../docs/adr/0016-alpha-scope-and-known-limitations.md)\n- [ADR-0015](../../docs/adr/0015-usdt-custodial-settlement.md)\n\n## Blocked by\n\n- `02-a2-unified-auth-boundary.md` (admin credit path secured)",
"acceptanceCriteria": [
"`DEFAULT_STARTING_CREDIT` is 0.0; no automatic caller credit on first touch",
"`has_funds` false for fresh keys; 402 before routing (server.py ~1684)",
@@ -62,21 +60,21 @@
"Run relevant pytest tests; run the full suite when practical or document why not"
],
"priority": 3,
"passes": false,
"status": "open",
"passes": true,
"notes": "Source issue: .scratch/alpha-hardening/issues/03-c5-starting-credit-zero.md",
"dependsOn": [
"AH-002"
]
],
"completionNotes": "Completed by agent"
},
{
"id": "AH-004",
"title": "04 \u2014 H2: Tracker-authoritative token and work-unit accounting",
"description": "# 04 \u2014 H2: Tracker-authoritative token and work-unit accounting\n\n## What to build\n\nStop trusting node-reported usage for billing. The tracker already proxies responses \u2014 use tracker-observed response data and request limits to cap billable tokens, and compute work units from the **route it constructed**, not node declarations.\n\n**Code refs:**\n\n- `packages/tracker/meshnet_tracker/server.py` \u2014 `node_work` from route construction (~1776\u20131782, ~1781\u20131782)\n- `packages/tracker/meshnet_tracker/server.py` \u2014 streaming token/chunk billing (~1890\u20131921)\n- `packages/tracker/meshnet_tracker/server.py` \u2014 non-streaming `_usage_total_tokens` (~1938\u20131943)\n- `packages/tracker/meshnet_tracker/billing.py` \u2014 `charge_request` node_work split (~104\u2013151)\n\nAccounting fraud = inflating tokens or shard span. Per ADR-0018 \u00a75.\n\n## Test-first\n\n1. Red: mock upstream returns inflated `usage.total_tokens` in body but tracker bills that value \u2014 test expects the tracker to cap billable tokens from observed stream chunks or request bounds.\n2. Red: node registers false `shard_end`; billing uses tracker route span, not registration field alone.\n3. Green: authoritative counters; ignore node-reported work units on charge path.\n\n## Acceptance criteria\n\n- [ ] Streaming token count uses tracker-observed chunks/tokens; upstream `usage.total_tokens` can only lower or match that observed count, never inflate it\n- [ ] Non-streaming token count caps upstream `usage.total_tokens` by tracker-known request bounds (`max_tokens`, and prompt estimate if available); exact tokenizer-backed counts are deferred unless already available locally\n- [ ] Work units = tracker-computed layer span per hop at route build time (~1781\u20131782)\n- [ ] Nodes cannot increase payout by lying about shard range mid-request\n- [ ] Integration test: malicious node metadata does not inflate `charge_request` shares\n\n## ADR links\n\n- [ADR-0018](../../docs/adr/0018-fraud-detection-verification-and-reputation.md) \u00a75\n\n## Blocked by\n\n- `02-a2-unified-auth-boundary.md`",
"title": "04 H2: Tracker-authoritative token and work-unit accounting",
"description": "# 04 H2: Tracker-authoritative token and work-unit accounting\n\n## What to build\n\nStop trusting node-reported usage for billing. The tracker already proxies responses use tracker-observed response data and request limits to cap billable tokens, and compute work units from the **route it constructed**, not node declarations.\n\n**Code refs:**\n\n- `packages/tracker/meshnet_tracker/server.py` `node_work` from route construction (~17761782, ~17811782)\n- `packages/tracker/meshnet_tracker/server.py` streaming token/chunk billing (~18901921)\n- `packages/tracker/meshnet_tracker/server.py` non-streaming `_usage_total_tokens` (~19381943)\n- `packages/tracker/meshnet_tracker/billing.py` `charge_request` node_work split (~104151)\n\nAccounting fraud = inflating tokens or shard span. Per ADR-0018 §5.\n\n## Test-first\n\n1. Red: mock upstream returns inflated `usage.total_tokens` in body but tracker bills that value test expects the tracker to cap billable tokens from observed stream chunks or request bounds.\n2. Red: node registers false `shard_end`; billing uses tracker route span, not registration field alone.\n3. Green: authoritative counters; ignore node-reported work units on charge path.\n\n## Acceptance criteria\n\n- [ ] Streaming token count uses tracker-observed chunks/tokens; upstream `usage.total_tokens` can only lower or match that observed count, never inflate it\n- [ ] Non-streaming token count caps upstream `usage.total_tokens` by tracker-known request bounds (`max_tokens`, and prompt estimate if available); exact tokenizer-backed counts are deferred unless already available locally\n- [ ] Work units = tracker-computed layer span per hop at route build time (~17811782)\n- [ ] Nodes cannot increase payout by lying about shard range mid-request\n- [ ] Integration test: malicious node metadata does not inflate `charge_request` shares\n\n## ADR links\n\n- [ADR-0018](../../docs/adr/0018-fraud-detection-verification-and-reputation.md) §5\n\n## Blocked by\n\n- `02-a2-unified-auth-boundary.md`",
"acceptanceCriteria": [
"Streaming token count uses tracker-observed chunks/tokens; upstream `usage.total_tokens` can only lower or match that observed count, never inflate it",
"Non-streaming token count caps upstream `usage.total_tokens` by tracker-known request bounds (`max_tokens`, and prompt estimate if available); exact tokenizer-backed counts are deferred unless already available locally",
"Work units = tracker-computed layer span per hop at route build time (~1781\u20131782)",
"Work units = tracker-computed layer span per hop at route build time (~17811782)",
"Nodes cannot increase payout by lying about shard range mid-request",
"Integration test: malicious node metadata does not inflate `charge_request` shares",
"Keep the implementation scoped to this issue; do not refactor unrelated server.py areas",
@@ -84,63 +82,63 @@
"Run relevant pytest tests; run the full suite when practical or document why not"
],
"priority": 4,
"passes": false,
"status": "open",
"passes": true,
"notes": "Source issue: .scratch/alpha-hardening/issues/04-h2-tracker-authoritative-accounting.md",
"dependsOn": [
"AH-002"
]
],
"completionNotes": "Completed by agent"
},
{
"id": "AH-005",
"title": "05 \u2014 A1/A5: Persist strike, ban, and reputation state",
"description": "# 05 \u2014 A1/A5: Persist strike, ban, and reputation state\n\n## What to build\n\nRegistry strike/ban/reputation state today lives in RAM-only `_LocalContractState` \u2014 tracker restart wipes penalties. Persist to SQLite (same pattern as `BillingLedger` and `AccountStore`) so reputation carries forward per ADR-0016 \u00a74.\n\n**Code refs:**\n\n- `packages/contracts/meshnet_contracts/__init__.py` \u2014 `RegistryContract`, `RegistryWallet`, in-memory `_state.registry` (~103\u2013206)\n- `packages/tracker/meshnet_tracker/billing.py` \u2014 SQLite persistence pattern (~60, event log)\n- `packages/tracker/meshnet_tracker/accounts.py` \u2014 SQLite + event replication (~40\u201356)\n\nInclude fields for: `strike_count`, `banned`, `completed_job_count`, graduated **reputation score** (float, default 1.0), `last_audit_ts`, probation tracking.\n\n**Scope split:** this issue owns **schema + persistence + load/reload** only. Reputation **scoring deltas** (audit pass/fail adjustments, decay rules) belong in issue 08.\n\n## Test-first\n\n1. Red: record strike, restart tracker process, strike count is 0 \u2014 must fail.\n2. Green: persist + reload; gossip replicates strike events if multi-tracker.\n3. Red: banned wallet registers node \u2014 must reject (wire to routing).\n\n## Acceptance criteria\n\n- [ ] Strike/ban/reputation survive tracker restart (SQLite or equivalent)\n- [ ] `RegistryContract.list_wallets` reflects persisted state\n- [ ] Banned wallet rejected at registration and excluded from routes\n- [ ] Reputation score field present for routing/audit issues (08\u201309)\n- [ ] Event-sourced mutations compatible with future Raft (ADR-0019)\n\n## ADR links\n\n- [ADR-0016](../../docs/adr/0016-alpha-scope-and-known-limitations.md) \u00a74\n- [ADR-0018](../../docs/adr/0018-fraud-detection-verification-and-reputation.md) \u00a76\n\n## Blocked by\n\n- `02-a2-unified-auth-boundary.md`",
"title": "05 A1/A5: Persist strike, ban, and reputation state",
"description": "# 05 A1/A5: Persist strike, ban, and reputation state\n\n## What to build\n\nRegistry strike/ban/reputation state today lives in RAM-only `_LocalContractState` tracker restart wipes penalties. Persist to SQLite (same pattern as `BillingLedger` and `AccountStore`) so reputation carries forward per ADR-0016 §4.\n\n**Code refs:**\n\n- `packages/contracts/meshnet_contracts/__init__.py` `RegistryContract`, `RegistryWallet`, in-memory `_state.registry` (~103206)\n- `packages/tracker/meshnet_tracker/billing.py` SQLite persistence pattern (~60, event log)\n- `packages/tracker/meshnet_tracker/accounts.py` SQLite + event replication (~4056)\n\nInclude fields for: `strike_count`, `banned`, `completed_job_count`, graduated **reputation score** (float, default 1.0), `last_audit_ts`, probation tracking.\n\n**Scope split:** this issue owns **schema + persistence + load/reload** only. Reputation **scoring deltas** (audit pass/fail adjustments, decay rules) belong in issue 08.\n\n## Test-first\n\n1. Red: record strike, restart tracker process, strike count is 0 must fail.\n2. Green: persist + reload; gossip replicates strike events if multi-tracker.\n3. Red: banned wallet registers node must reject (wire to routing).\n\n## Acceptance criteria\n\n- [ ] Strike/ban/reputation survive tracker restart (SQLite or equivalent)\n- [ ] `RegistryContract.list_wallets` reflects persisted state\n- [ ] Banned wallet rejected at registration and excluded from routes\n- [ ] Reputation score field present for routing/audit issues (0809)\n- [ ] Event-sourced mutations compatible with future Raft (ADR-0019)\n\n## ADR links\n\n- [ADR-0016](../../docs/adr/0016-alpha-scope-and-known-limitations.md) §4\n- [ADR-0018](../../docs/adr/0018-fraud-detection-verification-and-reputation.md) §6\n\n## Blocked by\n\n- `02-a2-unified-auth-boundary.md`",
"acceptanceCriteria": [
"Strike/ban/reputation survive tracker restart (SQLite or equivalent)",
"`RegistryContract.list_wallets` reflects persisted state",
"Banned wallet rejected at registration and excluded from routes",
"Reputation score field present for routing/audit issues (08\u201309)",
"Reputation score field present for routing/audit issues (0809)",
"Event-sourced mutations compatible with future Raft (ADR-0019)",
"Keep the implementation scoped to this issue; do not refactor unrelated server.py areas",
"Update the source issue file Status header to done when complete",
"Run relevant pytest tests; run the full suite when practical or document why not"
],
"priority": 5,
"passes": false,
"status": "open",
"passes": true,
"notes": "Source issue: .scratch/alpha-hardening/issues/05-a1-a5-persist-strike-ban-reputation.md",
"dependsOn": [
"AH-002"
]
],
"completionNotes": "Completed by agent"
},
{
"id": "AH-006",
"title": "06 \u2014 FRAUD: TOPLOC integration (teacher-forced audit primitive)",
"description": "# 06 \u2014 FRAUD: TOPLOC integration (teacher-forced audit primitive)\n\n## What to build\n\nAdopt [TOPLOC](https://github.com/PrimeIntellect-ai/toploc) (MIT, `pip install toploc`) for activation fingerprint commit and verify. Replace string-equality validator checks with teacher-forced prefill + TOPLOC tolerance matching.\n\n**Estimated effort:** 2+ sessions. First landing should be the validator-only TOPLOC primitive and docs; node runtime commitments/on-demand capture can follow in issue 07 if this grows.\n\n| Subtask | Owner package | Deliverable |\n|---|---|---|\n| Validator audit primitive | `packages/validator/` | Teacher-forced prefill, TOPLOC verify, unit tests with stub tensors |\n| Node runtime commitments | `packages/node/` (if prover-side) | On-demand activation fingerprint generation on audit-selected requests; move to issue 07 if it blocks the validator primitive |\n\n**Code refs:**\n\n- `packages/validator/meshnet_validator/__init__.py` \u2014 `_run_reference`, `_outputs_match` (~92\u2013148)\n- `packages/validator/README.md` \u2014 deterrence math (update for 19\u00d7 at p=0.05)\n- Research: `.scratch/alpha-hardening/research-verifiable-inference.md` \u00a78 layers 1\u20132, build-vs-adopt table\n\nPin one canonical precision/quantization per model preset. Add `toploc` to validator (and node if prover-side) dependencies.\n\n## Test-first\n\n1. Red: validator compares final text strings \u2014 fails on cross-GPU honest divergence (document expected).\n2. Green: stub activation tensors + TOPLOC proofs round-trip in unit test.\n3. Integration: reference node teacher-forces tokens; verify accepts honest proof, rejects swapped precision.\n\n## Acceptance criteria\n\n- [ ] `toploc` dependency declared; `build_proofs_*` / `verify_proofs_*` wired\n- [ ] Validator re-runs claimed token sequence as prefill, not free generation\n- [ ] Model preset documents canonical dtype/quantization\n- [ ] README updated: 19\u00d7 deterrence at 5% audit (research \u00a71.1)\n- [ ] Tests with deterministic stub tensors (no GPU required in CI)\n\n## ADR links\n\n- [ADR-0018](../../docs/adr/0018-fraud-detection-verification-and-reputation.md) \u00a72\n- Research: [research-verifiable-inference.md](../research-verifiable-inference.md) \u00a78, \u00a79 build-vs-adopt\n\n## Blocked by\n\n- `05-a1-a5-persist-strike-ban-reputation.md`\n\n**Prod gate:** do not enable production audit thresholds until `21-honest-noise-calibration-corpus.md` completes (see README Phase 2 note).",
"title": "06 FRAUD: TOPLOC integration (teacher-forced audit primitive)",
"description": "# 06 FRAUD: TOPLOC integration (teacher-forced audit primitive)\n\n## What to build\n\nAdopt [TOPLOC](https://github.com/PrimeIntellect-ai/toploc) (MIT, `pip install toploc`) for activation fingerprint commit and verify. Replace string-equality validator checks with teacher-forced prefill + TOPLOC tolerance matching.\n\n**Estimated effort:** 2+ sessions. First landing should be the validator-only TOPLOC primitive and docs; node runtime commitments/on-demand capture can follow in issue 07 if this grows.\n\n| Subtask | Owner package | Deliverable |\n|---|---|---|\n| Validator audit primitive | `packages/validator/` | Teacher-forced prefill, TOPLOC verify, unit tests with stub tensors |\n| Node runtime commitments | `packages/node/` (if prover-side) | On-demand activation fingerprint generation on audit-selected requests; move to issue 07 if it blocks the validator primitive |\n\n**Code refs:**\n\n- `packages/validator/meshnet_validator/__init__.py` `_run_reference`, `_outputs_match` (~92148)\n- `packages/validator/README.md` deterrence math (update for 19× at p=0.05)\n- Research: `.scratch/alpha-hardening/research-verifiable-inference.md` §8 layers 12, build-vs-adopt table\n\nPin one canonical precision/quantization per model preset. Add `toploc` to validator (and node if prover-side) dependencies.\n\n## Test-first\n\n1. Red: validator compares final text strings fails on cross-GPU honest divergence (document expected).\n2. Green: stub activation tensors + TOPLOC proofs round-trip in unit test.\n3. Integration: reference node teacher-forces tokens; verify accepts honest proof, rejects swapped precision.\n\n## Acceptance criteria\n\n- [ ] `toploc` dependency declared; `build_proofs_*` / `verify_proofs_*` wired\n- [ ] Validator re-runs claimed token sequence as prefill, not free generation\n- [ ] Model preset documents canonical dtype/quantization\n- [ ] README updated: 19× deterrence at 5% audit (research §1.1)\n- [ ] Tests with deterministic stub tensors (no GPU required in CI)\n\n## ADR links\n\n- [ADR-0018](../../docs/adr/0018-fraud-detection-verification-and-reputation.md) §2\n- Research: [research-verifiable-inference.md](../research-verifiable-inference.md) §8, §9 build-vs-adopt\n\n## Blocked by\n\n- `05-a1-a5-persist-strike-ban-reputation.md`\n\n**Prod gate:** do not enable production audit thresholds until `21-honest-noise-calibration-corpus.md` completes (see README Phase 2 note).",
"acceptanceCriteria": [
"`toploc` dependency declared; `build_proofs_*` / `verify_proofs_*` wired",
"Validator re-runs claimed token sequence as prefill, not free generation",
"Model preset documents canonical dtype/quantization",
"README updated: 19\u00d7 deterrence at 5% audit (research \u00a71.1)",
"README updated: 19× deterrence at 5% audit (research §1.1)",
"Tests with deterministic stub tensors (no GPU required in CI)",
"Keep the implementation scoped to this issue; do not refactor unrelated server.py areas",
"Update the source issue file Status header to done when complete",
"Run relevant pytest tests; run the full suite when practical or document why not"
],
"priority": 6,
"passes": false,
"status": "open",
"passes": true,
"notes": "Source issue: .scratch/alpha-hardening/issues/06-fraud-toploc-integration.md",
"dependsOn": [
"AH-005"
]
],
"completionNotes": "Completed by agent"
},
{
"id": "AH-007",
"title": "07 \u2014 FRAUD: On-demand commitment + hop bisection blame",
"description": "# 07 \u2014 FRAUD: On-demand commitment + hop bisection blame\n\n## What to build\n\nOn audit selection, require nodes to supply TOPLOC-style fingerprints of **output boundary activations** per hop (on-demand, brief retention). On verify failure, referee identifies the **first divergent hop** \u2014 not always the last text node.\n\n**Code refs:**\n\n- `packages/validator/meshnet_validator/__init__.py` \u2014 `_slash_route`, `_final_text_node` bug (~102\u2013140) \u2014 blames `max(shard_end)` only\n- `packages/tracker/meshnet_tracker/server.py` \u2014 route hop construction (~1774\u20131783) \u2014 cut-points for bisection\n- Research: `.scratch/alpha-hardening/research-verifiable-inference.md` \u00a71.2, \u00a78 layer 3 (Verde **pattern**, not on-chain game)\n\n## Test-first\n\n1. Red: two-hop route, corrupt hop-0 activations \u2014 `_final_text_node` blames hop-1 \u2014 test must fail.\n2. Green: bisection selects hop-0; forfeit targets hop-0 wallet.\n3. On-demand: commitment requested only when audit flag set on proxied request.\n\n## Acceptance criteria\n\n- [ ] Audit requests carry tracker RNG/VRF flag indistinguishable from normal traffic (research \u00a76)\n- [ ] Nodes retain recent boundary activations for on-demand commit window (configurable TTL)\n- [ ] Validator/tracker compares fingerprints at each hop cut-point; first mismatch = culprit\n- [ ] `_final_text_node` removed or limited to text-only fallback\n- [ ] Integration test: multi-hop pipeline, fault injected at known hop\n\n## ADR links\n\n- [ADR-0018](../../docs/adr/0018-fraud-detection-verification-and-reputation.md) \u00a73\u20134\n\n## Blocked by\n\n- `06-fraud-toploc-integration.md`",
"title": "07 FRAUD: On-demand commitment + hop bisection blame",
"description": "# 07 FRAUD: On-demand commitment + hop bisection blame\n\n## What to build\n\nOn audit selection, require nodes to supply TOPLOC-style fingerprints of **output boundary activations** per hop (on-demand, brief retention). On verify failure, referee identifies the **first divergent hop** not always the last text node.\n\n**Code refs:**\n\n- `packages/validator/meshnet_validator/__init__.py` `_slash_route`, `_final_text_node` bug (~102140) — blames `max(shard_end)` only\n- `packages/tracker/meshnet_tracker/server.py` route hop construction (~17741783) — cut-points for bisection\n- Research: `.scratch/alpha-hardening/research-verifiable-inference.md` §1.2, §8 layer 3 (Verde **pattern**, not on-chain game)\n\n## Test-first\n\n1. Red: two-hop route, corrupt hop-0 activations `_final_text_node` blames hop-1 test must fail.\n2. Green: bisection selects hop-0; forfeit targets hop-0 wallet.\n3. On-demand: commitment requested only when audit flag set on proxied request.\n\n## Acceptance criteria\n\n- [ ] Audit requests carry tracker RNG/VRF flag indistinguishable from normal traffic (research §6)\n- [ ] Nodes retain recent boundary activations for on-demand commit window (configurable TTL)\n- [ ] Validator/tracker compares fingerprints at each hop cut-point; first mismatch = culprit\n- [ ] `_final_text_node` removed or limited to text-only fallback\n- [ ] Integration test: multi-hop pipeline, fault injected at known hop\n\n## ADR links\n\n- [ADR-0018](../../docs/adr/0018-fraud-detection-verification-and-reputation.md) §34\n\n## Blocked by\n\n- `06-fraud-toploc-integration.md`",
"acceptanceCriteria": [
"Audit requests carry tracker RNG/VRF flag indistinguishable from normal traffic (research \u00a76)",
"Audit requests carry tracker RNG/VRF flag indistinguishable from normal traffic (research §6)",
"Nodes retain recent boundary activations for on-demand commit window (configurable TTL)",
"Validator/tracker compares fingerprints at each hop cut-point; first mismatch = culprit",
"`_final_text_node` removed or limited to text-only fallback",
@@ -150,21 +148,21 @@
"Run relevant pytest tests; run the full suite when practical or document why not"
],
"priority": 7,
"passes": false,
"status": "open",
"passes": true,
"notes": "Source issue: .scratch/alpha-hardening/issues/07-fraud-commitment-bisection-blame.md",
"dependsOn": [
"AH-006"
]
],
"completionNotes": "Completed by agent"
},
{
"id": "AH-008",
"title": "08 \u2014 FRAUD: Reputation model + persistence",
"description": "# 08 \u2014 FRAUD: Reputation model + persistence\n\n## What to build\n\nImplement graduated reputation per ADR-0018 \u00a76: score derives only from tracker audit outcomes + uptime/latency. Slow build, instant loss, inactivity decay. \u00d70.8 routing multiplier per strike (not whole penalty \u2014 forfeiture stays full pending).\n\n**Scope split:** issue 05 owns **schema + SQLite persistence**; this issue owns **scoring rules** (deltas, decay, strike\u2192multiplier wiring) on top of persisted fields.\n\n**Code refs:**\n\n- `packages/contracts/meshnet_contracts/__init__.py` \u2014 extend `RegistryWallet` / persistence from issue 05\n- `packages/validator/meshnet_validator/__init__.py` \u2014 `_slash_route` forfeiture path (~125\u2013133)\n- `packages/tracker/meshnet_tracker/billing.py` \u2014 `forfeit_pending` (~280\u2013292)\n- Research: `.scratch/alpha-hardening/research-verifiable-inference.md` \u00a76\n\n## Test-first\n\n1. Red: persisted reputation/strike fields from issue 05 are ignored by scoring/routing today.\n2. Green: clean audit +0.05 (tunable); failed audit \u22120.3 and strike; three strikes \u2192 ban persisted via issue-05 fields.\n3. Inactivity decay after N days without completed jobs.\n\n## Acceptance criteria\n\n- [ ] Uses `reputation_score` and strike/ban fields persisted by issue 05; does not introduce a second schema path\n- [ ] Audit pass/fail updates score with documented deltas\n- [ ] Strike applies \u00d70.8 multiplier to routing weight (separate from forfeiture amount)\n- [ ] Ban at 3 strikes; probation job count still enforced\n- [ ] No peer-to-peer reputation inputs\n\n## ADR links\n\n- [ADR-0018](../../docs/adr/0018-fraud-detection-verification-and-reputation.md) \u00a76\n\n## Blocked by\n\n- `05-a1-a5-persist-strike-ban-reputation.md`\n- `07-fraud-commitment-bisection-blame.md` (audit outcomes feed reputation)",
"title": "08 FRAUD: Reputation model + persistence",
"description": "# 08 FRAUD: Reputation model + persistence\n\n## What to build\n\nImplement graduated reputation per ADR-0018 §6: score derives only from tracker audit outcomes + uptime/latency. Slow build, instant loss, inactivity decay. ×0.8 routing multiplier per strike (not whole penalty forfeiture stays full pending).\n\n**Scope split:** issue 05 owns **schema + SQLite persistence**; this issue owns **scoring rules** (deltas, decay, strikemultiplier wiring) on top of persisted fields.\n\n**Code refs:**\n\n- `packages/contracts/meshnet_contracts/__init__.py` extend `RegistryWallet` / persistence from issue 05\n- `packages/validator/meshnet_validator/__init__.py` `_slash_route` forfeiture path (~125133)\n- `packages/tracker/meshnet_tracker/billing.py` `forfeit_pending` (~280292)\n- Research: `.scratch/alpha-hardening/research-verifiable-inference.md` §6\n\n## Test-first\n\n1. Red: persisted reputation/strike fields from issue 05 are ignored by scoring/routing today.\n2. Green: clean audit +0.05 (tunable); failed audit 0.3 and strike; three strikes ban persisted via issue-05 fields.\n3. Inactivity decay after N days without completed jobs.\n\n## Acceptance criteria\n\n- [ ] Uses `reputation_score` and strike/ban fields persisted by issue 05; does not introduce a second schema path\n- [ ] Audit pass/fail updates score with documented deltas\n- [ ] Strike applies ×0.8 multiplier to routing weight (separate from forfeiture amount)\n- [ ] Ban at 3 strikes; probation job count still enforced\n- [ ] No peer-to-peer reputation inputs\n\n## ADR links\n\n- [ADR-0018](../../docs/adr/0018-fraud-detection-verification-and-reputation.md) §6\n\n## Blocked by\n\n- `05-a1-a5-persist-strike-ban-reputation.md`\n- `07-fraud-commitment-bisection-blame.md` (audit outcomes feed reputation)",
"acceptanceCriteria": [
"Uses `reputation_score` and strike/ban fields persisted by issue 05; does not introduce a second schema path",
"Audit pass/fail updates score with documented deltas",
"Strike applies \u00d70.8 multiplier to routing weight (separate from forfeiture amount)",
"Strike applies ×0.8 multiplier to routing weight (separate from forfeiture amount)",
"Ban at 3 strikes; probation job count still enforced",
"No peer-to-peer reputation inputs",
"Keep the implementation scoped to this issue; do not refactor unrelated server.py areas",
@@ -172,21 +170,21 @@
"Run relevant pytest tests; run the full suite when practical or document why not"
],
"priority": 8,
"passes": false,
"status": "open",
"passes": true,
"notes": "Source issue: .scratch/alpha-hardening/issues/08-fraud-reputation-model-persistence.md",
"dependsOn": [
"AH-005",
"AH-007"
]
],
"completionNotes": "Completed by agent"
},
{
"id": "AH-009",
"title": "09 \u2014 FRAUD: Reputation-weighted routing + adaptive audit rate",
"description": "# 09 \u2014 FRAUD: Reputation-weighted routing + adaptive audit rate\n\n## What to build\n\nWire reputation into route selection and audit sampling. Default network audit budget \u22485% \u2014 **not a cap**. New/low-reputation nodes: 20\u201330% audit rate; veterans: 2\u20133% floor \u22652%. Tripwires escalate rate without direct punishment.\n\n**Code refs:**\n\n- `packages/tracker/meshnet_tracker/server.py` \u2014 route selection `_select_route`, `_effective_throughput` (~1747, routing helpers)\n- `packages/validator/meshnet_validator/__init__.py` \u2014 `sample_rate=0.05`\n- Research: `.scratch/alpha-hardening/research-verifiable-inference.md` \u00a71.1, \u00a76, \u00a78 layers 2\u20134\n\nAudit selection must be unpredictable at request time (tracker RNG after commitment window opens).\n\n## Test-first\n\n1. Red: uniform 5% sample regardless of reputation \u2014 test expects higher rate for low-reputation wallet.\n2. Green: budget balancer keeps fleet-wide average \u2248 configured target.\n3. Routing prefers higher reputation among equal throughput candidates.\n\n## Acceptance criteria\n\n- [ ] Per-wallet audit probability function of reputation (newcomer high, veteran low, floor \u22652%)\n- [ ] Fleet-wide audit budget configurable (~5% default target); over \u22651000 requests with fixed seed, measured fleet audit rate within **\u00b11.0 percentage point** of configured target (e.g. 4.0\u20136.0% at 5% default)\n- [ ] Route scoring includes reputation multiplier (earnings scale with tenure)\n- [ ] Passive tripwire flags (perplexity/repetition) bump audit rate only\n- [ ] Tests: deterministic seed for sampling distribution checks\n\n## ADR links\n\n- [ADR-0018](../../docs/adr/0018-fraud-detection-verification-and-reputation.md) \u00a71, \u00a76\u20137\n- [ADR-0013](../../docs/adr/0013-rolling-stats-smart-routing.md)\n\n## Blocked by\n\n- `08-fraud-reputation-model-persistence.md`",
"title": "09 FRAUD: Reputation-weighted routing + adaptive audit rate",
"description": "# 09 FRAUD: Reputation-weighted routing + adaptive audit rate\n\n## What to build\n\nWire reputation into route selection and audit sampling. Default network audit budget ≈5% — **not a cap**. New/low-reputation nodes: 2030% audit rate; veterans: 23% floor 2%. Tripwires escalate rate without direct punishment.\n\n**Code refs:**\n\n- `packages/tracker/meshnet_tracker/server.py` route selection `_select_route`, `_effective_throughput` (~1747, routing helpers)\n- `packages/validator/meshnet_validator/__init__.py` `sample_rate=0.05`\n- Research: `.scratch/alpha-hardening/research-verifiable-inference.md` §1.1, §6, §8 layers 24\n\nAudit selection must be unpredictable at request time (tracker RNG after commitment window opens).\n\n## Test-first\n\n1. Red: uniform 5% sample regardless of reputation test expects higher rate for low-reputation wallet.\n2. Green: budget balancer keeps fleet-wide average configured target.\n3. Routing prefers higher reputation among equal throughput candidates.\n\n## Acceptance criteria\n\n- [ ] Per-wallet audit probability function of reputation (newcomer high, veteran low, floor 2%)\n- [ ] Fleet-wide audit budget configurable (~5% default target); over 1000 requests with fixed seed, measured fleet audit rate within **±1.0 percentage point** of configured target (e.g. 4.06.0% at 5% default)\n- [ ] Route scoring includes reputation multiplier (earnings scale with tenure)\n- [ ] Passive tripwire flags (perplexity/repetition) bump audit rate only\n- [ ] Tests: deterministic seed for sampling distribution checks\n\n## ADR links\n\n- [ADR-0018](../../docs/adr/0018-fraud-detection-verification-and-reputation.md) §1, §67\n- [ADR-0013](../../docs/adr/0013-rolling-stats-smart-routing.md)\n\n## Blocked by\n\n- `08-fraud-reputation-model-persistence.md`",
"acceptanceCriteria": [
"Per-wallet audit probability function of reputation (newcomer high, veteran low, floor \u22652%)",
"Fleet-wide audit budget configurable (~5% default target); over \u22651000 requests with fixed seed, measured fleet audit rate within **\u00b11.0 percentage point** of configured target (e.g. 4.0\u20136.0% at 5% default)",
"Per-wallet audit probability function of reputation (newcomer high, veteran low, floor 2%)",
"Fleet-wide audit budget configurable (~5% default target); over 1000 requests with fixed seed, measured fleet audit rate within **±1.0 percentage point** of configured target (e.g. 4.06.0% at 5% default)",
"Route scoring includes reputation multiplier (earnings scale with tenure)",
"Passive tripwire flags (perplexity/repetition) bump audit rate only",
"Tests: deterministic seed for sampling distribution checks",
@@ -195,44 +193,44 @@
"Run relevant pytest tests; run the full suite when practical or document why not"
],
"priority": 9,
"passes": false,
"status": "open",
"passes": true,
"notes": "Source issue: .scratch/alpha-hardening/issues/09-fraud-reputation-routing-adaptive-audit.md",
"dependsOn": [
"AH-008"
]
],
"completionNotes": "Completed by agent"
},
{
"id": "AH-010",
"title": "10 \u2014 FRAUD: Penalty calibration wiring (forfeit + strike + ban)",
"description": "# 10 \u2014 FRAUD: Penalty calibration wiring (forfeit + strike + ban)\n\n## What to build\n\nEnd-to-end wiring: confirmed audit failure \u2192 atomic pending forfeiture + strike + reputation decay + audit-rate snap to max. Ensure payout cannot race penalty (ADR-0015). Document 19\u00d7 deterrence math in validator README.\n\n**Code refs:**\n\n- `packages/validator/meshnet_validator/__init__.py` \u2014 `_slash_route` (~102\u2013134)\n- `packages/tracker/meshnet_tracker/server.py` \u2014 `_handle_billing_forfeit` (~2429\u20132464)\n- `packages/tracker/meshnet_tracker/billing.py` \u2014 `forfeit_pending` (~280\u2013292), payout exclusion for banned (~3337\u20133344 in settlement loop)\n- `packages/validator/README.md` \u2014 update 20\u00d7 \u2192 19\u00d7 at p=0.05\n\nPer ADR-0018: **full pending forfeiture** is primary penalty; \u00d70.8 is routing decay per strike, not partial forfeit.\n\n## Test-first\n\n1. Red: integration from issue 34 \u2014 extend with multi-hop blame wallet from issue 07.\n2. Green: node with pending balance \u2192 audit fail \u2192 pending zero, strike++, banned on 3rd, excluded from next settlement.\n3. Settlement loop skips banned wallets (~3337\u20133344).\n\n## Acceptance criteria\n\n- [ ] Audit failure triggers forfeiture + strike in one tracker transaction\n- [ ] Banned nodes excluded from `payables` / settlement\n- [ ] Validator uses authenticated forfeit endpoint (issue 02)\n- [ ] README: `L > 19\u00d7 g` at p=0.05; pending balance = collateral\n- [ ] Integration test: 60-request fraud scenario \u2192 ban within threshold\n\n## ADR links\n\n- [ADR-0018](../../docs/adr/0018-fraud-detection-verification-and-reputation.md)\n- [ADR-0015](../../docs/adr/0015-usdt-custodial-settlement.md)\n- Research: [research-verifiable-inference.md](../research-verifiable-inference.md) \u00a71.1\n\n## Blocked by\n\n- `07-fraud-commitment-bisection-blame.md`\n- `08-fraud-reputation-model-persistence.md`\n- `02-a2-unified-auth-boundary.md`",
"title": "10 FRAUD: Penalty calibration wiring (forfeit + strike + ban)",
"description": "# 10 FRAUD: Penalty calibration wiring (forfeit + strike + ban)\n\n## What to build\n\nEnd-to-end wiring: confirmed audit failure atomic pending forfeiture + strike + reputation decay + audit-rate snap to max. Ensure payout cannot race penalty (ADR-0015). Document 19× deterrence math in validator README.\n\n**Code refs:**\n\n- `packages/validator/meshnet_validator/__init__.py` `_slash_route` (~102134)\n- `packages/tracker/meshnet_tracker/server.py` `_handle_billing_forfeit` (~24292464)\n- `packages/tracker/meshnet_tracker/billing.py` `forfeit_pending` (~280292), payout exclusion for banned (~33373344 in settlement loop)\n- `packages/validator/README.md` update 20× → 19× at p=0.05\n\nPer ADR-0018: **full pending forfeiture** is primary penalty; ×0.8 is routing decay per strike, not partial forfeit.\n\n## Test-first\n\n1. Red: integration from issue 34 extend with multi-hop blame wallet from issue 07.\n2. Green: node with pending balance audit fail pending zero, strike++, banned on 3rd, excluded from next settlement.\n3. Settlement loop skips banned wallets (~33373344).\n\n## Acceptance criteria\n\n- [ ] Audit failure triggers forfeiture + strike in one tracker transaction\n- [ ] Banned nodes excluded from `payables` / settlement\n- [ ] Validator uses authenticated forfeit endpoint (issue 02)\n- [ ] README: `L > 19× g` at p=0.05; pending balance = collateral\n- [ ] Integration test: 60-request fraud scenario ban within threshold\n\n## ADR links\n\n- [ADR-0018](../../docs/adr/0018-fraud-detection-verification-and-reputation.md)\n- [ADR-0015](../../docs/adr/0015-usdt-custodial-settlement.md)\n- Research: [research-verifiable-inference.md](../research-verifiable-inference.md) §1.1\n\n## Blocked by\n\n- `07-fraud-commitment-bisection-blame.md`\n- `08-fraud-reputation-model-persistence.md`\n- `02-a2-unified-auth-boundary.md`",
"acceptanceCriteria": [
"Audit failure triggers forfeiture + strike in one tracker transaction",
"Banned nodes excluded from `payables` / settlement",
"Validator uses authenticated forfeit endpoint (issue 02)",
"README: `L > 19\u00d7 g` at p=0.05; pending balance = collateral",
"Integration test: 60-request fraud scenario \u2192 ban within threshold",
"README: `L > 19× g` at p=0.05; pending balance = collateral",
"Integration test: 60-request fraud scenario ban within threshold",
"Keep the implementation scoped to this issue; do not refactor unrelated server.py areas",
"Update the source issue file Status header to done when complete",
"Run relevant pytest tests; run the full suite when practical or document why not"
],
"priority": 10,
"passes": false,
"status": "open",
"passes": true,
"notes": "Source issue: .scratch/alpha-hardening/issues/10-fraud-penalty-calibration-wiring.md",
"dependsOn": [
"AH-007",
"AH-008",
"AH-002"
]
],
"completionNotes": "Completed by agent"
},
{
"id": "AH-011",
"title": "11 \u2014 C6: Wallet binding ownership proof + binding overwrite safety",
"description": "# 11 \u2014 C6: Wallet binding ownership proof + binding overwrite safety\n\n## What to build\n\n`POST /v1/wallet/register` binds a client Solana wallet to an API key for deposit attribution. Today any Bearer key can bind any wallet string without proving ownership. Prevent hijack and accidental overwrite.\n\n**Code refs:**\n\n- `packages/tracker/meshnet_tracker/server.py` \u2014 `_handle_wallet_register` (~2625\u20132648)\n- `packages/tracker/meshnet_tracker/billing.py` \u2014 `bind_wallet` (~153+), `_wallet_bindings` / direct overwrite on apply (~351)\n\nRequire signed message from wallet pubkey (ed25519 via `cryptography` / solders). Reject rebinding without admin or signed release. Use explicit overwrite policy \u2014 today `~351` overwrites binding directly; gossip apply must reject conflicting binds instead of silently clobbering.\n\n## Test-first\n\n1. Red: bind wallet A with only API key, no signature \u2014 must fail after fix.\n2. Red: wallet already bound to key1; key2 cannot steal without proof.\n3. Green: valid signature binds; deposit watcher credits correct API key.\n\n## Acceptance criteria\n\n- [ ] Wallet binding requires cryptographic proof of pubkey ownership\n- [ ] One wallet \u2192 one API key (or documented admin override)\n- [ ] Gossip `bind` events cannot overwrite existing binding via direct overwrite at `~351`\n- [ ] Tests with deterministic keypairs (local adapter)\n\n## ADR links\n\n- [ADR-0017](../../docs/adr/0017-tracker-authentication-and-authorization.md) \u00a75\n- [ADR-0015](../../docs/adr/0015-usdt-custodial-settlement.md)\n\n## Blocked by\n\n- `02-a2-unified-auth-boundary.md`\n- `03-c5-starting-credit-zero.md`",
"title": "11 C6: Wallet binding ownership proof + binding overwrite safety",
"description": "# 11 C6: Wallet binding ownership proof + binding overwrite safety\n\n## What to build\n\n`POST /v1/wallet/register` binds a client Solana wallet to an API key for deposit attribution. Today any Bearer key can bind any wallet string without proving ownership. Prevent hijack and accidental overwrite.\n\n**Code refs:**\n\n- `packages/tracker/meshnet_tracker/server.py` `_handle_wallet_register` (~26252648)\n- `packages/tracker/meshnet_tracker/billing.py` `bind_wallet` (~153+), `_wallet_bindings` / direct overwrite on apply (~351)\n\nRequire signed message from wallet pubkey (ed25519 via `cryptography` / solders). Reject rebinding without admin or signed release. Use explicit overwrite policy today `~351` overwrites binding directly; gossip apply must reject conflicting binds instead of silently clobbering.\n\n## Test-first\n\n1. Red: bind wallet A with only API key, no signature must fail after fix.\n2. Red: wallet already bound to key1; key2 cannot steal without proof.\n3. Green: valid signature binds; deposit watcher credits correct API key.\n\n## Acceptance criteria\n\n- [ ] Wallet binding requires cryptographic proof of pubkey ownership\n- [ ] One wallet one API key (or documented admin override)\n- [ ] Gossip `bind` events cannot overwrite existing binding via direct overwrite at `~351`\n- [ ] Tests with deterministic keypairs (local adapter)\n\n## ADR links\n\n- [ADR-0017](../../docs/adr/0017-tracker-authentication-and-authorization.md) §5\n- [ADR-0015](../../docs/adr/0015-usdt-custodial-settlement.md)\n\n## Blocked by\n\n- `02-a2-unified-auth-boundary.md`\n- `03-c5-starting-credit-zero.md`",
"acceptanceCriteria": [
"Wallet binding requires cryptographic proof of pubkey ownership",
"One wallet \u2192 one API key (or documented admin override)",
"One wallet one API key (or documented admin override)",
"Gossip `bind` events cannot overwrite existing binding via direct overwrite at `~351`",
"Tests with deterministic keypairs (local adapter)",
"Keep the implementation scoped to this issue; do not refactor unrelated server.py areas",
@@ -240,38 +238,37 @@
"Run relevant pytest tests; run the full suite when practical or document why not"
],
"priority": 11,
"passes": false,
"status": "open",
"passes": true,
"notes": "Source issue: .scratch/alpha-hardening/issues/11-c6-wallet-binding-proof.md",
"dependsOn": [
"AH-002",
"AH-003"
]
],
"completionNotes": "Completed by agent; source issue status set to done, wallet binding proof implemented and tested."
},
{
"id": "AH-012",
"title": "12 \u2014 C2: On-chain settlement idempotency (deferred)",
"description": "# 12 \u2014 C2: On-chain settlement idempotency (deferred)\n\n## What to build\n\nHarden payout idempotency so Solana transaction retries never double-pay. Design accepted in ADR-0019 \u00a71; **implementation deferred post-alpha**.\n\n**Code refs:**\n\n- `packages/tracker/meshnet_tracker/server.py` \u2014 `_settlement_loop` resend (~3331\u20133356), `_send_settlement` (~3358\u20133376)\n- `packages/contracts/meshnet_contracts/solana_adapter.py` \u2014 `send_payouts` (~186\u2013213)\n\nToday: pending debited before broadcast with stable `settlement_id`; unconfirmed batches resent. Gap: on-chain confirmation vs ledger state if tx succeeds but confirm fails.\n\n## Acceptance criteria\n\n- [ ] `confirm_settlement` only after RPC finalized confirmation\n- [ ] Retry path reuses same `settlement_id` and detects already-confirmed signature\n- [ ] Property test: N retries \u2192 single on-chain transfer per wallet per settlement_id\n- [ ] Document recovery procedure for stuck unconfirmed batches\n\n## ADR links\n\n- [ADR-0019](../../docs/adr/0019-money-path-consistency-multi-tracker.md) \u00a71\n\n## Blocked by\n\nAlpha release (ADR-0016 single settlement tracker)",
"title": "12 C2: On-chain settlement idempotency (deferred)",
"description": "# 12 C2: On-chain settlement idempotency (deferred)\n\n## What to build\n\nHarden payout idempotency so Solana transaction retries never double-pay. Design accepted in ADR-0019 §1; **implementation deferred post-alpha**.\n\n**Code refs:**\n\n- `packages/tracker/meshnet_tracker/server.py` `_settlement_loop` resend (~33313356), `_send_settlement` (~33583376)\n- `packages/contracts/meshnet_contracts/solana_adapter.py` `send_payouts` (~186213)\n\nToday: pending debited before broadcast with stable `settlement_id`; unconfirmed batches resent. Gap: on-chain confirmation vs ledger state if tx succeeds but confirm fails.\n\n## Acceptance criteria\n\n- [ ] `confirm_settlement` only after RPC finalized confirmation\n- [ ] Retry path reuses same `settlement_id` and detects already-confirmed signature\n- [ ] Property test: N retries single on-chain transfer per wallet per settlement_id\n- [ ] Document recovery procedure for stuck unconfirmed batches\n\n## ADR links\n\n- [ADR-0019](../../docs/adr/0019-money-path-consistency-multi-tracker.md) §1\n\n## Blocked by\n\nAlpha release (ADR-0016 single settlement tracker)",
"acceptanceCriteria": [
"`confirm_settlement` only after RPC finalized confirmation",
"Retry path reuses same `settlement_id` and detects already-confirmed signature",
"Property test: N retries \u2192 single on-chain transfer per wallet per settlement_id",
"Property test: N retries single on-chain transfer per wallet per settlement_id",
"Document recovery procedure for stuck unconfirmed batches",
"Keep the implementation scoped to this issue; do not refactor unrelated server.py areas",
"Update the source issue file Status header to done when complete",
"Run relevant pytest tests; run the full suite when practical or document why not"
],
"priority": 12,
"passes": false,
"status": "in-design",
"notes": "Source issue: .scratch/alpha-hardening/issues/12-c2-on-chain-idempotency.md",
"passes": true,
"notes": "Source issue: .scratch/alpha-hardening/issues/12-c2-on-chain-idempotency.md\nRalph skip: Source issue is ready-for-human/deferred; skipped by unattended Ralph auto.",
"dependsOn": [],
"status_reason": "Source issue is ready-for-human/deferred; skipped by unattended Ralph auto."
"completionNotes": "Source issue is ready-for-human/deferred; skipped by unattended Ralph auto."
},
{
"id": "AH-013",
"title": "13 \u2014 C3/C4: Consensus-gated money mutations (deferred)",
"description": "# 13 \u2014 C3/C4: Consensus-gated money mutations (deferred)\n\n## What to build\n\nRoute money-affecting ledger events through Raft commit, not gossip-only apply. Extend `raft.py` command set beyond register/deregister. Settlement remains leader-only with treasury key.\n\n**Code refs:**\n\n- `packages/tracker/meshnet_tracker/server.py` \u2014 settlement leader gate (~3331\u20133332), payout batch (~3353\u20133356)\n- `packages/tracker/meshnet_tracker/raft.py` \u2014 log entry types (~26\u201327)\n- `packages/tracker/meshnet_tracker/billing.py` \u2014 `apply_events` (~301\u2013311)\n\nDesign: ADR-0019 \u00a72. **Deferred post-alpha** while single operator holds settlement.\n\n## Acceptance criteria\n\n- [ ] `charge`, `payout`, `forfeit`, `credit`, `settlement`, `bind` commit via Raft log\n- [ ] Followers reject direct gossip money mutations\n- [ ] Leader-only `_settlement_loop` unchanged in semantics\n- [ ] Migration plan from gossip-only billing to Raft-backed log\n\n## ADR links\n\n- [ADR-0019](../../docs/adr/0019-money-path-consistency-multi-tracker.md) \u00a72\n\n## Blocked by\n\n- `12-c2-on-chain-idempotency.md`\n- `14-a3-raft-durable-term-vote.md`",
"title": "13 C3/C4: Consensus-gated money mutations (deferred)",
"description": "# 13 C3/C4: Consensus-gated money mutations (deferred)\n\n## What to build\n\nRoute money-affecting ledger events through Raft commit, not gossip-only apply. Extend `raft.py` command set beyond register/deregister. Settlement remains leader-only with treasury key.\n\n**Code refs:**\n\n- `packages/tracker/meshnet_tracker/server.py` settlement leader gate (~33313332), payout batch (~33533356)\n- `packages/tracker/meshnet_tracker/raft.py` log entry types (~2627)\n- `packages/tracker/meshnet_tracker/billing.py` `apply_events` (~301311)\n\nDesign: ADR-0019 §2. **Deferred post-alpha** while single operator holds settlement.\n\n## Acceptance criteria\n\n- [ ] `charge`, `payout`, `forfeit`, `credit`, `settlement`, `bind` commit via Raft log\n- [ ] Followers reject direct gossip money mutations\n- [ ] Leader-only `_settlement_loop` unchanged in semantics\n- [ ] Migration plan from gossip-only billing to Raft-backed log\n\n## ADR links\n\n- [ADR-0019](../../docs/adr/0019-money-path-consistency-multi-tracker.md) §2\n\n## Blocked by\n\n- `12-c2-on-chain-idempotency.md`\n- `14-a3-raft-durable-term-vote.md`",
"acceptanceCriteria": [
"`charge`, `payout`, `forfeit`, `credit`, `settlement`, `bind` commit via Raft log",
"Followers reject direct gossip money mutations",
@@ -282,19 +279,18 @@
"Run relevant pytest tests; run the full suite when practical or document why not"
],
"priority": 13,
"passes": false,
"status": "in-design",
"notes": "Source issue: .scratch/alpha-hardening/issues/13-c3-c4-consensus-gated-settlement.md",
"passes": true,
"notes": "Source issue: .scratch/alpha-hardening/issues/13-c3-c4-consensus-gated-settlement.md\nRalph skip: Source issue is ready-for-human/deferred; skipped by unattended Ralph auto.",
"dependsOn": [
"AH-012",
"AH-014"
],
"status_reason": "Source issue is ready-for-human/deferred; skipped by unattended Ralph auto."
"completionNotes": "Source issue is ready-for-human/deferred; skipped by unattended Ralph auto."
},
{
"id": "AH-014",
"title": "14 \u2014 A3: Durable Raft term and vote state (deferred)",
"description": "# 14 \u2014 A3: Durable Raft term and vote state (deferred)\n\n## What to build\n\nPersist Raft `currentTerm`, `votedFor`, and log metadata to disk. In-memory-only term (~26) risks split leadership after tracker restart \u2192 duplicate settlement epochs.\n\n**Code refs:**\n\n- `packages/tracker/meshnet_tracker/raft.py` \u2014 `LogEntry.term` (~25\u201327), election state in `RaftNode`\n\n## Acceptance criteria\n\n- [ ] Term/vote persisted alongside tracker data dir\n- [ ] Restart resumes as follower/candidate with monotonic term\n- [ ] Test: kill leader mid-settlement, restart, no duplicate payout batch\n\n## ADR links\n\n- [ADR-0019](../../docs/adr/0019-money-path-consistency-multi-tracker.md) \u00a73\n\n## Blocked by\n\nAlpha single-settlement posture",
"title": "14 A3: Durable Raft term and vote state (deferred)",
"description": "# 14 A3: Durable Raft term and vote state (deferred)\n\n## What to build\n\nPersist Raft `currentTerm`, `votedFor`, and log metadata to disk. In-memory-only term (~26) risks split leadership after tracker restart duplicate settlement epochs.\n\n**Code refs:**\n\n- `packages/tracker/meshnet_tracker/raft.py` `LogEntry.term` (~2527), election state in `RaftNode`\n\n## Acceptance criteria\n\n- [ ] Term/vote persisted alongside tracker data dir\n- [ ] Restart resumes as follower/candidate with monotonic term\n- [ ] Test: kill leader mid-settlement, restart, no duplicate payout batch\n\n## ADR links\n\n- [ADR-0019](../../docs/adr/0019-money-path-consistency-multi-tracker.md) §3\n\n## Blocked by\n\nAlpha single-settlement posture",
"acceptanceCriteria": [
"Term/vote persisted alongside tracker data dir",
"Restart resumes as follower/candidate with monotonic term",
@@ -304,16 +300,15 @@
"Run relevant pytest tests; run the full suite when practical or document why not"
],
"priority": 14,
"passes": false,
"status": "in-design",
"notes": "Source issue: .scratch/alpha-hardening/issues/14-a3-raft-durable-term-vote.md",
"passes": true,
"notes": "Source issue: .scratch/alpha-hardening/issues/14-a3-raft-durable-term-vote.md\nRalph skip: Source issue is ready-for-human/deferred; skipped by unattended Ralph auto.",
"dependsOn": [],
"status_reason": "Source issue is ready-for-human/deferred; skipped by unattended Ralph auto."
"completionNotes": "Source issue is ready-for-human/deferred; skipped by unattended Ralph auto."
},
{
"id": "AH-015",
"title": "15 \u2014 H1: Commutative forfeit event ordering (deferred)",
"description": "# 15 \u2014 H1: Commutative forfeit event ordering (deferred)\n\n## What to build\n\nDefine deterministic ordering when `forfeit`, `charge`, and `payout` events replicate concurrently. Forfeit snapshots amount at creation (~287) but apply order can desync pending balances under gossip.\n\n**Code refs:**\n\n- `packages/tracker/meshnet_tracker/billing.py` \u2014 `forfeit_pending` (~280\u2013292), `_apply_locked` forfeit branch (~345\u2013349)\n- `packages/tracker/meshnet_tracker/billing.py` \u2014 `_pending_since.setdefault` (~324), wallet bind direct overwrite (~351)\n\n## Acceptance criteria\n\n- [ ] Documented commit order: charges before forfeit before payout for same wallet epoch\n- [ ] Forfeit events carry pending snapshot or `(term, index)` for tie-break\n- [ ] `setdefault` replaced with explicit merge rules on out-of-order apply\n- [ ] Property tests under shuffled event delivery\n\n## ADR links\n\n- [ADR-0019](../../docs/adr/0019-money-path-consistency-multi-tracker.md) \u00a74\n\n## Blocked by\n\n- `13-c3-c4-consensus-gated-settlement.md`",
"title": "15 H1: Commutative forfeit event ordering (deferred)",
"description": "# 15 H1: Commutative forfeit event ordering (deferred)\n\n## What to build\n\nDefine deterministic ordering when `forfeit`, `charge`, and `payout` events replicate concurrently. Forfeit snapshots amount at creation (~287) but apply order can desync pending balances under gossip.\n\n**Code refs:**\n\n- `packages/tracker/meshnet_tracker/billing.py` `forfeit_pending` (~280292), `_apply_locked` forfeit branch (~345349)\n- `packages/tracker/meshnet_tracker/billing.py` `_pending_since.setdefault` (~324), wallet bind direct overwrite (~351)\n\n## Acceptance criteria\n\n- [ ] Documented commit order: charges before forfeit before payout for same wallet epoch\n- [ ] Forfeit events carry pending snapshot or `(term, index)` for tie-break\n- [ ] `setdefault` replaced with explicit merge rules on out-of-order apply\n- [ ] Property tests under shuffled event delivery\n\n## ADR links\n\n- [ADR-0019](../../docs/adr/0019-money-path-consistency-multi-tracker.md) §4\n\n## Blocked by\n\n- `13-c3-c4-consensus-gated-settlement.md`",
"acceptanceCriteria": [
"Documented commit order: charges before forfeit before payout for same wallet epoch",
"Forfeit events carry pending snapshot or `(term, index)` for tie-break",
@@ -324,22 +319,21 @@
"Run relevant pytest tests; run the full suite when practical or document why not"
],
"priority": 15,
"passes": false,
"status": "in-design",
"notes": "Source issue: .scratch/alpha-hardening/issues/15-h1-commutative-forfeit.md",
"passes": true,
"notes": "Source issue: .scratch/alpha-hardening/issues/15-h1-commutative-forfeit.md\nRalph skip: Source issue is ready-for-human/deferred; skipped by unattended Ralph auto.",
"dependsOn": [
"AH-013"
],
"status_reason": "Source issue is ready-for-human/deferred; skipped by unattended Ralph auto."
"completionNotes": "Source issue is ready-for-human/deferred; skipped by unattended Ralph auto."
},
{
"id": "AH-016",
"title": "16 \u2014 DOC: US-006 reconciliation note",
"description": "# 16 \u2014 DOC: US-006 reconciliation note\n\n## What to build\n\nReconcile stale US-006 (Solana testnet stake contracts) with ADR-0015/0016 devnet custodial settlement. Issue `docs/issues/06-solana-stake-and-settlement.md` says \"never devnet\"; ADR-0015 explicitly targets devnet mock-USDT.\n\nAlso reconcile legacy fraud issues with the alpha-hardening fraud arc:\n\n- `docs/issues/07-fraud-detection-slash.md` \u2014 on-chain stake slash model superseded by pending-balance forfeiture + TOPLOC (ADR-0018)\n- `docs/issues/34-forfeiture-penalty.md` \u2014 partially implemented; remaining fraud work lives in `.scratch/alpha-hardening/issues/06-fraud-toploc-integration.md` through `10-fraud-penalty-calibration-wiring.md`\n\n## Acceptance criteria\n\n- [ ] Add reconciliation comment atop `docs/issues/06-solana-stake-and-settlement.md` (Status: superseded for alpha \u2014 see ADR-0015, issue 33/34)\n- [ ] Add **superseded** banner atop `docs/issues/07-fraud-detection-slash.md` \u2192 ADR-0018 + issues 06\u201310\n- [ ] Add **superseded for remaining scope** banner atop `docs/issues/34-forfeiture-penalty.md` \u2192 ADR-0018 + issues 06\u201310 (note done items: basic forfeiture wired)\n- [ ] Update `docs/prd.json` US-006 description footnote if present\n- [ ] Cross-link ADR-0015 devnet decision\n- [ ] No production code changes\n\n## ADR links\n\n- [ADR-0015](../../docs/adr/0015-usdt-custodial-settlement.md)\n- [ADR-0016](../../docs/adr/0016-alpha-scope-and-known-limitations.md)\n\n## Blocked by\n\nNone",
"title": "16 DOC: US-006 reconciliation note",
"description": "# 16 DOC: US-006 reconciliation note\n\n## What to build\n\nReconcile stale US-006 (Solana testnet stake contracts) with ADR-0015/0016 devnet custodial settlement. Issue `docs/issues/06-solana-stake-and-settlement.md` says \"never devnet\"; ADR-0015 explicitly targets devnet mock-USDT.\n\nAlso reconcile legacy fraud issues with the alpha-hardening fraud arc:\n\n- `docs/issues/07-fraud-detection-slash.md` on-chain stake slash model superseded by pending-balance forfeiture + TOPLOC (ADR-0018)\n- `docs/issues/34-forfeiture-penalty.md` partially implemented; remaining fraud work lives in `.scratch/alpha-hardening/issues/06-fraud-toploc-integration.md` through `10-fraud-penalty-calibration-wiring.md`\n\n## Acceptance criteria\n\n- [ ] Add reconciliation comment atop `docs/issues/06-solana-stake-and-settlement.md` (Status: superseded for alpha see ADR-0015, issue 33/34)\n- [ ] Add **superseded** banner atop `docs/issues/07-fraud-detection-slash.md` ADR-0018 + issues 0610\n- [ ] Add **superseded for remaining scope** banner atop `docs/issues/34-forfeiture-penalty.md` ADR-0018 + issues 0610 (note done items: basic forfeiture wired)\n- [ ] Update `docs/prd.json` US-006 description footnote if present\n- [ ] Cross-link ADR-0015 devnet decision\n- [ ] No production code changes\n\n## ADR links\n\n- [ADR-0015](../../docs/adr/0015-usdt-custodial-settlement.md)\n- [ADR-0016](../../docs/adr/0016-alpha-scope-and-known-limitations.md)\n\n## Blocked by\n\nNone",
"acceptanceCriteria": [
"Add reconciliation comment atop `docs/issues/06-solana-stake-and-settlement.md` (Status: superseded for alpha \u2014 see ADR-0015, issue 33/34)",
"Add **superseded** banner atop `docs/issues/07-fraud-detection-slash.md` \u2192 ADR-0018 + issues 06\u201310",
"Add **superseded for remaining scope** banner atop `docs/issues/34-forfeiture-penalty.md` \u2192 ADR-0018 + issues 06\u201310 (note done items: basic forfeiture wired)",
"Add reconciliation comment atop `docs/issues/06-solana-stake-and-settlement.md` (Status: superseded for alpha see ADR-0015, issue 33/34)",
"Add **superseded** banner atop `docs/issues/07-fraud-detection-slash.md` ADR-0018 + issues 0610",
"Add **superseded for remaining scope** banner atop `docs/issues/34-forfeiture-penalty.md` ADR-0018 + issues 0610 (note done items: basic forfeiture wired)",
"Update `docs/prd.json` US-006 description footnote if present",
"Cross-link ADR-0015 devnet decision",
"No production code changes",
@@ -348,18 +342,18 @@
"Run relevant pytest tests; run the full suite when practical or document why not"
],
"priority": 16,
"passes": false,
"status": "open",
"passes": true,
"notes": "Source issue: .scratch/alpha-hardening/issues/16-doc-us006-reconciliation.md",
"dependsOn": []
"dependsOn": [],
"completionNotes": "Completed by agent"
},
{
"id": "AH-017",
"title": "17 \u2014 DOC: Duplicate US-020 issue dedup",
"description": "# 17 \u2014 DOC: Duplicate US-020 issue dedup\n\n## What to build\n\nTwo files share the US-020 number with different slugs:\n\n- `docs/issues/20-memory-budget-shard-slots-and-dropout-relocation.md` (ready-for-agent)\n- `docs/issues/20-tracker-node-hardening.md` (done)\n\nResolve numbering collision without losing history.\n\n## Acceptance criteria\n\n- [ ] Document canonical mapping in this issue's Comments or a short `docs/issues/README.md` note\n- [ ] Renumber or prefix disambiguation (e.g. keep done item as US-020a, renumber memory-budget to next slot) \u2014 **human approval before git mv**\n- [ ] Update any prd.json / cross-links that reference US-020 ambiguously\n- [ ] No production code changes\n\n## Blocked by\n\nHuman approval for renumbering. An agent may prepare the mapping note, but must not run `git mv` or rewrite cross-links until the canonical number is approved.",
"title": "17 DOC: Duplicate US-020 issue dedup",
"description": "# 17 DOC: Duplicate US-020 issue dedup\n\n## What to build\n\nTwo files share the US-020 number with different slugs:\n\n- `docs/issues/20-memory-budget-shard-slots-and-dropout-relocation.md` (ready-for-agent)\n- `docs/issues/20-tracker-node-hardening.md` (done)\n\nResolve numbering collision without losing history.\n\n## Acceptance criteria\n\n- [ ] Document canonical mapping in this issue's Comments or a short `docs/issues/README.md` note\n- [ ] Renumber or prefix disambiguation (e.g. keep done item as US-020a, renumber memory-budget to next slot) **human approval before git mv**\n- [ ] Update any prd.json / cross-links that reference US-020 ambiguously\n- [ ] No production code changes\n\n## Blocked by\n\nHuman approval for renumbering. An agent may prepare the mapping note, but must not run `git mv` or rewrite cross-links until the canonical number is approved.",
"acceptanceCriteria": [
"Document canonical mapping in this issue's Comments or a short `docs/issues/README.md` note",
"Renumber or prefix disambiguation (e.g. keep done item as US-020a, renumber memory-budget to next slot) \u2014 **human approval before git mv**",
"Renumber or prefix disambiguation (e.g. keep done item as US-020a, renumber memory-budget to next slot) **human approval before git mv**",
"Update any prd.json / cross-links that reference US-020 ambiguously",
"No production code changes",
"Keep the implementation scoped to this issue; do not refactor unrelated server.py areas",
@@ -367,16 +361,15 @@
"Run relevant pytest tests; run the full suite when practical or document why not"
],
"priority": 17,
"passes": false,
"status": "in-design",
"notes": "Source issue: .scratch/alpha-hardening/issues/17-doc-duplicate-us020-dedup.md",
"passes": true,
"notes": "Source issue: .scratch/alpha-hardening/issues/17-doc-duplicate-us020-dedup.md\nRalph skip: Source issue is ready-for-human/deferred; skipped by unattended Ralph auto.",
"dependsOn": [],
"status_reason": "Source issue is ready-for-human/deferred; skipped by unattended Ralph auto."
"completionNotes": "Source issue is ready-for-human/deferred; skipped by unattended Ralph auto."
},
{
"id": "AH-018",
"title": "18 \u2014 DOC: Operational runbooks (stubs)",
"description": "# 18 \u2014 DOC: Operational runbooks (stubs)\n\n## What to build\n\nAdd operational runbook stubs for alpha operators under `docs/runbooks/` (or `.scratch/alpha-hardening/runbooks/` until close-feature):\n\n1. **Ledger backup** \u2014 billing SQLite, accounts SQLite, registry DB paths; gossip pause procedure\n2. **Treasury key rotation** \u2014 devnet mock-USDT mint + treasury keypair rotation without double-credit\n3. **Upgrade path** \u2014 tracker rolling restart with persisted strike/reputation (post issue 05)\n\n## Acceptance criteria\n\n- [ ] Three markdown runbook stubs with prerequisites, steps, rollback\n- [ ] Reference ADR-0015 settlement loop and ADR-0016 trust assumptions\n- [ ] Secrets handling: never commit `.env.devnet`, keypairs\n- [ ] No production code changes\n\n## ADR links\n\n- [ADR-0015](../../docs/adr/0015-usdt-custodial-settlement.md)\n- [ADR-0016](../../docs/adr/0016-alpha-scope-and-known-limitations.md)\n\n## Blocked by\n\nNone (stubs can land before issue 05; update after persistence ships)",
"title": "18 DOC: Operational runbooks (stubs)",
"description": "# 18 DOC: Operational runbooks (stubs)\n\n## What to build\n\nAdd operational runbook stubs for alpha operators under `docs/runbooks/` (or `.scratch/alpha-hardening/runbooks/` until close-feature):\n\n1. **Ledger backup** billing SQLite, accounts SQLite, registry DB paths; gossip pause procedure\n2. **Treasury key rotation** devnet mock-USDT mint + treasury keypair rotation without double-credit\n3. **Upgrade path** tracker rolling restart with persisted strike/reputation (post issue 05)\n\n## Acceptance criteria\n\n- [ ] Three markdown runbook stubs with prerequisites, steps, rollback\n- [ ] Reference ADR-0015 settlement loop and ADR-0016 trust assumptions\n- [ ] Secrets handling: never commit `.env.devnet`, keypairs\n- [ ] No production code changes\n\n## ADR links\n\n- [ADR-0015](../../docs/adr/0015-usdt-custodial-settlement.md)\n- [ADR-0016](../../docs/adr/0016-alpha-scope-and-known-limitations.md)\n\n## Blocked by\n\nNone (stubs can land before issue 05; update after persistence ships)",
"acceptanceCriteria": [
"Three markdown runbook stubs with prerequisites, steps, rollback",
"Reference ADR-0015 settlement loop and ADR-0016 trust assumptions",
@@ -387,15 +380,15 @@
"Run relevant pytest tests; run the full suite when practical or document why not"
],
"priority": 18,
"passes": false,
"status": "open",
"passes": true,
"notes": "Source issue: .scratch/alpha-hardening/issues/18-doc-operational-runbooks.md",
"dependsOn": []
"dependsOn": [],
"completionNotes": "Completed by agent"
},
{
"id": "AH-019",
"title": "19 \u2014 DOC: Cryptography dependency + test environment note",
"description": "# 19 \u2014 DOC: Cryptography dependency + test environment note\n\n## What to build\n\nDocument and verify test/dev environment setup for wallet crypto paths. `packages/node/meshnet_node/wallet.py` uses `cryptography`; failures occur when `.venv` lacks deps. `cryptography>=41` is already declared in `packages/node/pyproject.toml`, so this issue should focus on documenting the editable-install path and only add root/dev extras if tests still import the node wallet without installing the node package.\n\n**Code refs:**\n\n- `packages/node/pyproject.toml` \u2014 `cryptography>=41` (verify declared)\n- `packages/node/meshnet_node/wallet.py`\n- Handoff: tests fail without `cryptography`, `openai`, `langchain` in `.venv`\n\n## Acceptance criteria\n\n- [ ] Confirm `cryptography>=41` remains in node package deps; add to root/dev extras only if tests import wallet without node install\n- [ ] Add short **Test environment** section to `docs/dev/test-env.md` (or `CONTRIBUTING.md` if created): use `.venv/Scripts/python.exe`, `pip install -e packages/node ...`, optional dep skips\n- [ ] Note which tests require optional deps (`--ignore=test_openai_gateway,...`)\n- [ ] No unrelated production code changes\n\n## Blocked by\n\nNone",
"title": "19 DOC: Cryptography dependency + test environment note",
"description": "# 19 DOC: Cryptography dependency + test environment note\n\n## What to build\n\nDocument and verify test/dev environment setup for wallet crypto paths. `packages/node/meshnet_node/wallet.py` uses `cryptography`; failures occur when `.venv` lacks deps. `cryptography>=41` is already declared in `packages/node/pyproject.toml`, so this issue should focus on documenting the editable-install path and only add root/dev extras if tests still import the node wallet without installing the node package.\n\n**Code refs:**\n\n- `packages/node/pyproject.toml` `cryptography>=41` (verify declared)\n- `packages/node/meshnet_node/wallet.py`\n- Handoff: tests fail without `cryptography`, `openai`, `langchain` in `.venv`\n\n## Acceptance criteria\n\n- [ ] Confirm `cryptography>=41` remains in node package deps; add to root/dev extras only if tests import wallet without node install\n- [ ] Add short **Test environment** section to `docs/dev/test-env.md` (or `CONTRIBUTING.md` if created): use `.venv/Scripts/python.exe`, `pip install -e packages/node ...`, optional dep skips\n- [ ] Note which tests require optional deps (`--ignore=test_openai_gateway,...`)\n- [ ] No unrelated production code changes\n\n## Blocked by\n\nNone",
"acceptanceCriteria": [
"Confirm `cryptography>=41` remains in node package deps; add to root/dev extras only if tests import wallet without node install",
"Add short **Test environment** section to `docs/dev/test-env.md` (or `CONTRIBUTING.md` if created): use `.venv/Scripts/python.exe`, `pip install -e packages/node ...`, optional dep skips",
@@ -406,18 +399,18 @@
"Run relevant pytest tests; run the full suite when practical or document why not"
],
"priority": 19,
"passes": false,
"status": "open",
"passes": true,
"notes": "Source issue: .scratch/alpha-hardening/issues/19-doc-cryptography-test-env.md",
"dependsOn": []
"dependsOn": [],
"completionNotes": "Completed by agent"
},
{
"id": "AH-020",
"title": "20 \u2014 Validator service token for `/v1/billing/forfeit`",
"description": "# 20 \u2014 Validator service token for `/v1/billing/forfeit`\n\n## What to build\n\nDefine and implement a **validator service token** distinct from client API keys and admin sessions. The validator process must authenticate when calling `POST /v1/billing/forfeit`; arbitrary Bearer strings and client API keys must be rejected. This is a checklist subtask for issue 02 and should normally land in the same PR as the unified auth middleware.\n\nPer [ADR-0017 \u00a74](../../docs/adr/0017-tracker-authentication-and-authorization.md): forfeit accepts **validator service identity or admin session** only.\n\n## Configuration\n\n| Item | Alpha default |\n|---|---|\n| Env var | `MESHNET_VALIDATOR_SERVICE_TOKEN` (tracker + validator) |\n| Config flag | `--validator-service-token` / tracker config file equivalent |\n| Header format | `Authorization: Bearer <service-token>` with a dedicated prefix or separate header scheme documented in runbooks (e.g. `Authorization: Service <token>` \u2014 pick one and test consistently) |\n| Rotation | Manual: set new token on tracker + validator, restart both; document zero-downtime rotation as post-alpha |\n\n## Rejection rules\n\n- Client API keys (`sk-mesh-\u2026`) \u2192 **403** on forfeit (even if valid for inference)\n- Non-empty garbage Bearer \u2192 **401/403**\n- Missing auth \u2192 **401**\n- Valid validator service token \u2192 **200** (existing forfeit semantics)\n- Admin session \u2192 **200** (operator override)\n\n## Test-first\n\n1. Red: validator (or test client) posts forfeit with a valid API key \u2014 must fail after fix.\n2. Red: `Authorization: Bearer garbage` \u2014 must fail (covered by issue 02; this issue defines the accepted token).\n3. Green: configured service token succeeds; wrong token fails.\n\n## Acceptance criteria\n\n- [ ] Service token configurable via env/flag on tracker and validator\n- [ ] Unified auth middleware resolves service token \u2192 `validator` role (issue 02)\n- [ ] API keys explicitly rejected on forfeit path\n- [ ] Integration test: validator client with service token forfeit succeeds; API key forfeit fails\n- [ ] Runbook stub: rotation procedure (manual alpha)\n\n## ADR links\n\n- [ADR-0017](../../docs/adr/0017-tracker-authentication-and-authorization.md) \u00a74\n\n## Related\n\n- `02-a2-unified-auth-boundary.md` \u2014 middleware + role checks\n\n## Blocked by\n\n- `02-a2-unified-auth-boundary.md`",
"title": "20 Validator service token for `/v1/billing/forfeit`",
"description": "# 20 Validator service token for `/v1/billing/forfeit`\n\n## What to build\n\nDefine and implement a **validator service token** distinct from client API keys and admin sessions. The validator process must authenticate when calling `POST /v1/billing/forfeit`; arbitrary Bearer strings and client API keys must be rejected. This is a checklist subtask for issue 02 and should normally land in the same PR as the unified auth middleware.\n\nPer [ADR-0017 §4](../../docs/adr/0017-tracker-authentication-and-authorization.md): forfeit accepts **validator service identity or admin session** only.\n\n## Configuration\n\n| Item | Alpha default |\n|---|---|\n| Env var | `MESHNET_VALIDATOR_SERVICE_TOKEN` (tracker + validator) |\n| Config flag | `--validator-service-token` / tracker config file equivalent |\n| Header format | `Authorization: Bearer <service-token>` with a dedicated prefix or separate header scheme documented in runbooks (e.g. `Authorization: Service <token>` pick one and test consistently) |\n| Rotation | Manual: set new token on tracker + validator, restart both; document zero-downtime rotation as post-alpha |\n\n## Rejection rules\n\n- Client API keys (`sk-mesh-…`) → **403** on forfeit (even if valid for inference)\n- Non-empty garbage Bearer **401/403**\n- Missing auth **401**\n- Valid validator service token **200** (existing forfeit semantics)\n- Admin session **200** (operator override)\n\n## Test-first\n\n1. Red: validator (or test client) posts forfeit with a valid API key must fail after fix.\n2. Red: `Authorization: Bearer garbage` must fail (covered by issue 02; this issue defines the accepted token).\n3. Green: configured service token succeeds; wrong token fails.\n\n## Acceptance criteria\n\n- [ ] Service token configurable via env/flag on tracker and validator\n- [ ] Unified auth middleware resolves service token `validator` role (issue 02)\n- [ ] API keys explicitly rejected on forfeit path\n- [ ] Integration test: validator client with service token forfeit succeeds; API key forfeit fails\n- [ ] Runbook stub: rotation procedure (manual alpha)\n\n## ADR links\n\n- [ADR-0017](../../docs/adr/0017-tracker-authentication-and-authorization.md) §4\n\n## Related\n\n- `02-a2-unified-auth-boundary.md` middleware + role checks\n\n## Blocked by\n\n- `02-a2-unified-auth-boundary.md`",
"acceptanceCriteria": [
"Service token configurable via env/flag on tracker and validator",
"Unified auth middleware resolves service token \u2192 `validator` role (issue 02)",
"Unified auth middleware resolves service token `validator` role (issue 02)",
"API keys explicitly rejected on forfeit path",
"Integration test: validator client with service token forfeit succeeds; API key forfeit fails",
"Runbook stub: rotation procedure (manual alpha)",
@@ -427,7 +420,6 @@
],
"priority": 20,
"passes": true,
"status": "done",
"notes": "Source issue: .scratch/alpha-hardening/issues/20-validator-service-token.md",
"dependsOn": [
"AH-002"
@@ -436,8 +428,8 @@
},
{
"id": "AH-021",
"title": "21 \u2014 Honest-noise TOPLOC calibration corpus",
"description": "# 21 \u2014 Honest-noise TOPLOC calibration corpus\n\n## What to build\n\nBefore enabling production TOPLOC audit thresholds, collect an **honest-noise baseline** across the heterogeneous volunteer fleet. Run identical inference jobs on every active node/GPU combo; measure the divergence envelope (TOPLOC exponent/mantissa deltas, logprob-rank spread) under real hardware variance.\n\nPer [ADR-0018 consequences](../../docs/adr/0018-fraud-detection-verification-and-reputation.md): threshold calibration requires an honest-noise corpus across the fleet before production thresholds.\n\nResearch anchor: `.scratch/alpha-hardening/research-verifiable-inference.md` \u00a78 layer 3 \u2014 \"collect this first \u2014 run identical jobs across the current node fleet to measure the honest divergence envelope before setting thresholds.\"\n\n## Deliverables\n\n- [ ] Scripted benchmark job (fixed prompt, model preset, seed policy) runnable on all nodes\n- [ ] Aggregated corpus artifact (per node: GPU model, dtype, TOPLOC deltas vs reference)\n- [ ] Recommended tolerance thresholds documented (p99 honest envelope + safety margin)\n- [ ] Gate checklist: production audit enable blocked until corpus covers \u2265N distinct hardware profiles (define N in runbook, suggest \u22653)\n\n## Acceptance criteria\n\n- [ ] Corpus collected from current fleet (or documented subset + extrapolation note)\n- [ ] Threshold constants in validator config derived from corpus, not guessed\n- [ ] False-positive rate estimate documented at chosen thresholds\n- [ ] README / runbook cross-link: **do not enable production audits** until this issue closes\n\n## ADR links\n\n- [ADR-0018](../../docs/adr/0018-fraud-detection-verification-and-reputation.md) \u2014 Consequences (honest-noise corpus)\n\n## Blocked by\n\n- `06-fraud-toploc-integration.md` (TOPLOC wired; calibration uses same primitive)\n\n## Blocks (prod gate)\n\n- Production enable of adaptive audit thresholds (issues 09\u201310 in prod)",
"title": "21 Honest-noise TOPLOC calibration corpus",
"description": "# 21 Honest-noise TOPLOC calibration corpus\n\n## What to build\n\nBefore enabling production TOPLOC audit thresholds, collect an **honest-noise baseline** across the heterogeneous volunteer fleet. Run identical inference jobs on every active node/GPU combo; measure the divergence envelope (TOPLOC exponent/mantissa deltas, logprob-rank spread) under real hardware variance.\n\nPer [ADR-0018 consequences](../../docs/adr/0018-fraud-detection-verification-and-reputation.md): threshold calibration requires an honest-noise corpus across the fleet before production thresholds.\n\nResearch anchor: `.scratch/alpha-hardening/research-verifiable-inference.md` §8 layer 3 \"collect this first run identical jobs across the current node fleet to measure the honest divergence envelope before setting thresholds.\"\n\n## Deliverables\n\n- [ ] Scripted benchmark job (fixed prompt, model preset, seed policy) runnable on all nodes\n- [ ] Aggregated corpus artifact (per node: GPU model, dtype, TOPLOC deltas vs reference)\n- [ ] Recommended tolerance thresholds documented (p99 honest envelope + safety margin)\n- [ ] Gate checklist: production audit enable blocked until corpus covers N distinct hardware profiles (define N in runbook, suggest 3)\n\n## Acceptance criteria\n\n- [ ] Corpus collected from current fleet (or documented subset + extrapolation note)\n- [ ] Threshold constants in validator config derived from corpus, not guessed\n- [ ] False-positive rate estimate documented at chosen thresholds\n- [ ] README / runbook cross-link: **do not enable production audits** until this issue closes\n\n## ADR links\n\n- [ADR-0018](../../docs/adr/0018-fraud-detection-verification-and-reputation.md) Consequences (honest-noise corpus)\n\n## Blocked by\n\n- `06-fraud-toploc-integration.md` (TOPLOC wired; calibration uses same primitive)\n\n## Blocks (prod gate)\n\n- Production enable of adaptive audit thresholds (issues 0910 in prod)",
"acceptanceCriteria": [
"Corpus collected from current fleet (or documented subset + extrapolation note)",
"Threshold constants in validator config derived from corpus, not guessed",
@@ -448,21 +440,20 @@
"Run relevant pytest tests; run the full suite when practical or document why not"
],
"priority": 21,
"passes": false,
"status": "in-design",
"notes": "Source issue: .scratch/alpha-hardening/issues/21-honest-noise-calibration-corpus.md",
"passes": true,
"notes": "Source issue: .scratch/alpha-hardening/issues/21-honest-noise-calibration-corpus.md\nRalph skip: Source issue is ready-for-human/deferred; skipped by unattended Ralph auto.",
"dependsOn": [
"AH-006"
],
"status_reason": "Source issue is ready-for-human/deferred; skipped by unattended Ralph auto."
"completionNotes": "Source issue is ready-for-human/deferred; skipped by unattended Ralph auto."
},
{
"id": "AH-022",
"title": "22 \u2014 DOC: MEMORY.md + project-status alpha-hardening index",
"description": "# 22 \u2014 DOC: MEMORY.md + project-status alpha-hardening index\n\n## What to build\n\nUpdate persistent memory files so agents and humans find the alpha-hardening feature without stale handoff paths.\n\n## Acceptance criteria\n\n- [x] `.claude/memory/MEMORY.md` \u2014 index entry for alpha-hardening (`.scratch/alpha-hardening/`, ADRs 0016\u20130019, issue count)\n- [x] `.claude/memory/project-status.md` \u2014 brief alpha-hardening section: planning complete, Bucket 1 blockers next, link README\n- [x] Cross-link `.scratch/alpha-hardening/handoff.md` from README (not temp path)\n\n## ADR links\n\n- [ADR-0016](../../docs/adr/0016-alpha-scope-and-known-limitations.md)\n\n## Blocked by\n\nNone \u2014 completed\n\n## Comments\n\n2026-07-04 triage: already satisfied by `.claude/memory/MEMORY.md`, `.claude/memory/project-status.md`, and `.scratch/alpha-hardening/README.md`.",
"title": "22 DOC: MEMORY.md + project-status alpha-hardening index",
"description": "# 22 DOC: MEMORY.md + project-status alpha-hardening index\n\n## What to build\n\nUpdate persistent memory files so agents and humans find the alpha-hardening feature without stale handoff paths.\n\n## Acceptance criteria\n\n- [x] `.claude/memory/MEMORY.md` index entry for alpha-hardening (`.scratch/alpha-hardening/`, ADRs 00160019, issue count)\n- [x] `.claude/memory/project-status.md` brief alpha-hardening section: planning complete, Bucket 1 blockers next, link README\n- [x] Cross-link `.scratch/alpha-hardening/handoff.md` from README (not temp path)\n\n## ADR links\n\n- [ADR-0016](../../docs/adr/0016-alpha-scope-and-known-limitations.md)\n\n## Blocked by\n\nNone completed\n\n## Comments\n\n2026-07-04 triage: already satisfied by `.claude/memory/MEMORY.md`, `.claude/memory/project-status.md`, and `.scratch/alpha-hardening/README.md`.",
"acceptanceCriteria": [
"[x] `.claude/memory/MEMORY.md` \u2014 index entry for alpha-hardening (`.scratch/alpha-hardening/`, ADRs 0016\u20130019, issue count)",
"[x] `.claude/memory/project-status.md` \u2014 brief alpha-hardening section: planning complete, Bucket 1 blockers next, link README",
"[x] `.claude/memory/MEMORY.md` index entry for alpha-hardening (`.scratch/alpha-hardening/`, ADRs 00160019, issue count)",
"[x] `.claude/memory/project-status.md` brief alpha-hardening section: planning complete, Bucket 1 blockers next, link README",
"[x] Cross-link `.scratch/alpha-hardening/handoff.md` from README (not temp path)",
"Keep the implementation scoped to this issue; do not refactor unrelated server.py areas",
"Update the source issue file Status header to done when complete",
@@ -470,10 +461,12 @@
],
"priority": 22,
"passes": true,
"status": "done",
"notes": "Source issue: .scratch/alpha-hardening/issues/22-doc-memory-project-status.md",
"dependsOn": [],
"completionNotes": "Marked done in source alpha-hardening issue file before PRD generation."
}
]
}
],
"metadata": {
"updatedAt": "2026-07-05T16:21:45.488Z"
}
}

View File

@@ -0,0 +1,88 @@
Status: stub
# Runbook: Ledger backup
Covers backing up the tracker's authoritative money/trust state — the billing
ledger, dashboard accounts DB, and node registry (strike/ban/reputation) — and
how to pause hive gossip during the backup window so peers don't replicate
against a half-copied file.
## Trust assumptions (read first)
Per [ADR-0016](../../../docs/adr/0016-alpha-scope-and-known-limitations.md), one
operator-designated tracker holds the treasury keypair and is the source of
truth for settlement; other hive members only replicate. Back up **that**
tracker's databases — a follower's copies are eventually consistent, not
authoritative. See [ADR-0015](../../../docs/adr/0015-usdt-custodial-settlement.md)
for the settlement loop these tables feed.
## Prerequisites
- Shell access to the settlement-capable tracker host.
- `sqlite3` CLI (or `.backup` support in the Python `sqlite3` module) available
for online, consistent snapshots.
- Know the tracker's configured DB paths — defaults, unless overridden by CLI
flags:
- Billing ledger: `billing.sqlite` (`--billing-db`, `DEFAULT_BILLING_DB_PATH`
in `packages/tracker/meshnet_tracker/billing.py`)
- Dashboard accounts: `accounts.sqlite` (`--accounts-db`,
`DEFAULT_ACCOUNTS_DB_PATH` in `packages/tracker/meshnet_tracker/accounts.py`)
- Node registry (strike/ban/reputation event log,
`packages/contracts/meshnet_contracts/__init__.py::RegistryEventLog`): path
is whatever was passed as `registry_db` when the tracker's
`LocalSolanaContracts` was constructed. **As of this writing the tracker
CLI (`meshnet_tracker/cli.py`) does not expose a `--registry-db` flag or
wire a `contracts=` instance into `TrackerServer` by default** — confirm
with whoever deployed this tracker whether registry persistence is
actually enabled before assuming a file exists to back up. If it isn't
wired up yet, strike/ban/reputation state is RAM-only and this step is
moot until that gap closes (tracked loosely against issue 05).
## Steps
1. Identify the actual DB paths in use (check the tracker's start command /
systemd unit / process env for `--billing-db`, `--accounts-db`, and any
registry DB argument).
2. **Pause hive gossip** on this tracker so peers don't pull a partial/locked
file mid-backup:
- If the tracker is the sole settlement node with no `--cluster-peers`,
gossip is already off — skip to step 3.
- Otherwise, stop replication by restarting the process without
`--cluster-peers` (or with an empty peer list) for the duration of the
backup, or take the backup during a maintenance window with peers
temporarily pointed away from this tracker at the load balancer/DNS
level. There is currently no live "pause gossip" admin endpoint — this is
a process-restart-level operation.
- Confirm no in-flight `/v1/registry/gossip`, `/v1/billing/gossip`, or
`/v1/accounts/gossip` traffic before proceeding (check access logs).
3. Take an online, consistent copy of each SQLite file using the backup API
rather than `cp` (WAL-mode files can be mid-write):
```
sqlite3 billing.sqlite ".backup '/backups/billing-$(date +%Y%m%dT%H%M%S).sqlite'"
sqlite3 accounts.sqlite ".backup '/backups/accounts-$(date +%Y%m%dT%H%M%S).sqlite'"
# registry DB, if configured:
sqlite3 registry.sqlite ".backup '/backups/registry-$(date +%Y%m%dT%H%M%S).sqlite'"
```
4. Verify each backup opens and has rows in its expected tables
(`billing_ledger`/event log tables, `accounts`, `registry_events`).
5. Resume gossip (restore `--cluster-peers` / routing) once backups are
confirmed good.
6. Ship backups off-host per your normal retention policy. Do not store them
alongside `.env.devnet` or keypair files (see secrets handling below).
## Rollback
- If a restore is needed, stop the tracker, replace the live `.sqlite` file(s)
with the chosen backup, and restart. Because billing/accounts/registry each
use append-only event logs, a stale restore under-counts recent activity
rather than corrupting state — reconcile any gap against node/operator
reports for the missing window before resuming payouts.
- If gossip was paused via a peer-list restart, confirm peers re-sync
(`events_since` catch-up) before considering the rollback complete.
## Secrets handling
- Never commit `.env.devnet`, treasury keypair JSON files, `--hive-secret`, or
`--validator-service-token` values to a repo or ship them inside a DB backup
archive. Back these up separately, encrypted, per your existing secrets
process.

View File

@@ -0,0 +1,112 @@
Status: stub
# Runbook: Treasury key rotation (devnet mock-USDT)
Covers rotating the devnet treasury keypair and/or the mock-USDT mint without
double-crediting client ledger balances or double-paying nodes.
## Trust assumptions (read first)
Per [ADR-0015](../../../docs/adr/0015-usdt-custodial-settlement.md), a single
project-owned wallet custodies all funds; the treasury keypair is loaded only
on the operator-designated settlement tracker (ADR-0016 §1). Rotating this key
is a trusted-operator action — there is no on-chain multisig or trustless
handoff in the alpha design. Devnet uses a self-created mock-USDT SPL mint
(6 decimals); real USDT only exists on mainnet, so this procedure is
devnet-only until a mainnet cutover ADR supersedes it.
## Prerequisites
- Access to `scripts/devnet_setup.py` and its dependencies (`solders`,
`meshnet_contracts.solana_adapter.SolanaCustodialTreasury`).
- The current treasury keypair path (default
`~/.config/solana/meshnet-treasury.json`, or whatever `--treasury-keypair`
the running tracker uses) and current `MESHNET_USDT_MINT` /
`MESHNET_TREASURY_WALLET` values (see `.env.devnet`, never committed).
- Ability to stop/restart the settlement-capable tracker.
- Confirm the deposit watcher's dedupe state (transaction signatures already
credited) is durable — it must survive the rotation so replayed/rescanned
transfers under the *old* wallet don't get re-credited under the *new* one.
## Two rotation scenarios
### A. Rotate the treasury keypair only (same mint, same on-chain wallet funds move)
The treasury wallet address changes because it's derived from the keypair, so
this requires migrating funds, not just swapping a file.
1. Generate a new keypair (do **not** reuse `_load_or_create_keypair` against
the old path — write to a new path so both keys exist during the
transition):
```
python scripts/devnet_setup.py --keypair ~/.config/solana/meshnet-treasury-new.json \
--mint <EXISTING_MOCK_USDT_MINT> --env-out .env.devnet.new
```
This creates the new treasury wallet + token account and reuses the
existing mint (no new token, so client balances denominated in that mint
are unaffected).
2. Drain the old treasury token account to the new one via a single SPL
transfer sized to the *entire current balance* (record the exact amount
and the source tx signature before moving anything).
3. **Freeze settlement during the drain**: stop the settlement-capable
tracker (or restart it with no `--treasury-keypair` so the settlement loop
is inert) before step 2, so no payout is in flight against the old wallet
while funds move.
4. Update the tracker's `--treasury-keypair`, `--treasury-wallet`-derived
config (i.e. the new `.env.devnet`) and restart the tracker pointed at the
new keypair.
5. Verify: `treasury.get_sol_balance()` / mock-USDT balance on the new wallet
matches the old wallet's pre-drain balance; old wallet balance is zero.
6. Only after verification, revoke/delete the old keypair file.
### B. Rotate the mock-USDT mint (e.g. compromised or mis-configured mint)
This is a bigger change — it invalidates every client's existing off-chain
ledger balance denomination reference and any node's pending on-chain payout
expectations. Treat as a deliberate migration, not a routine rotation:
1. Settle (pay out) all pending node balances against the *old* mint before
cutover — the pending-balance forfeiture/collateral model (ADR-0015)
assumes pending balances are payable in a known mint.
2. Create the new mint and treasury token account:
```
python scripts/devnet_setup.py --keypair <treasury-keypair> --env-out .env.devnet
```
(omit `--mint` so a fresh mint is created).
3. Update tracker config (`MESHNET_USDT_MINT`) and restart.
4. Re-mint/airdrop mock USDT to active client wallets under the new mint as
needed (`--mint-to`), since off-chain ledger balances are *not*
automatically re-denominated — this is a devnet convenience step, not a
guarantee that would hold for real USDT.
## Avoiding double-credit
The deposit watcher (issue 32) dedupes by on-chain transaction signature. The
signature space for the old and new treasury token accounts/mints is
disjoint, so:
- Do not replay old-wallet deposit history against the new wallet's watcher —
it has no record of those signatures and would (correctly) not credit them,
but any manual "catch-up crediting" script must not re-process transfers the
old watcher already credited. Cross-check the old ledger's credited-tx-sig
table before any manual reconciliation entry.
- Keep the old watcher's dedupe DB/table around (don't drop it as part of
rotation) until you've confirmed no in-flight deposits to the old address
remain unconfirmed.
## Rollback
- Scenario A: if the new wallet fails verification, restart the tracker with
the old `--treasury-keypair` — no client-facing state changed since ledger
balances are keyed by API key, not treasury wallet address.
- Scenario B: if re-minting under the new mint goes wrong, restart the
tracker against the old `MESHNET_USDT_MINT` config; nothing was destroyed on
the old mint.
## Secrets handling
- Never commit `.env.devnet`, `.env.devnet.new`, or any `*treasury*.json`
keypair file. `scripts/devnet_setup.py` writes keypairs with `0o600`
permissions — preserve that when copying.
- Treat the treasury keypair as the single highest-value secret in this
system per ADR-0015/ADR-0016: anyone with it can drain custodial funds.

View File

@@ -0,0 +1,108 @@
Status: stub
# Runbook: Tracker upgrade path (rolling restart)
Covers restarting/upgrading tracker processes in a hive without losing
strike/ban/reputation state or interrupting settlement, per the ADR-0016 §4
guarantee that reputation carries forward across restarts.
## Trust assumptions (read first)
Per [ADR-0016](../../../docs/adr/0016-alpha-scope-and-known-limitations.md),
only one operator-designated tracker holds the treasury keypair and runs the
settlement loop ([ADR-0015](../../../docs/adr/0015-usdt-custodial-settlement.md));
other hive members replicate for routing only. Raft (`packages/tracker/meshnet_tracker/raft.py`)
elects a leader for shard-assignment/registration commands — settlement
leadership is a separate, operator-configured concept, not the Raft leader.
Plan restarts so the settlement tracker's downtime window is minimized
independent of routing-tracker restarts.
## Known gap — read before relying on this runbook
Strike/ban/reputation persistence itself was implemented in issue 05
(`packages/contracts/meshnet_contracts/__init__.py::RegistryEventLog`,
SQLite-backed, same pattern as billing/accounts). **As of this writing,
`packages/tracker/meshnet_tracker/cli.py` does not expose a `--registry-db`
flag, nor does it construct a `contracts=` instance to pass into
`TrackerServer`.** Running the tracker via the stock CLI entry point leaves
`server.contracts` as `None`, which means:
- Ban checks (`_registration_ban_error`), reputation-weighted routing
(`_reputation_multiplier`), and the `/v1/registry/wallets` endpoint are
inert.
- There is nothing to persist across restarts in that configuration — the
"survives restart" guarantee only holds for deployments that construct
`LocalSolanaContracts(registry_db=<path>)` and wire it into `TrackerServer(contracts=...)`
themselves (e.g. a custom entrypoint or embedding the server programmatically).
Before following the restart steps below, confirm which mode this deployment
runs in. If it's the stock CLI with no custom `contracts` wiring, strike/ban
state is RAM-only regardless of this runbook, and a restart resets it — treat
that as a pre-existing gap to flag to the owner, not something this runbook
can work around.
## Prerequisites
- Confirm registry persistence is actually wired (see gap above) and note the
registry DB path in use.
- Confirm billing (`--billing-db`) and accounts (`--accounts-db`) persistence
paths — these already default to `billing.sqlite` / `accounts.sqlite` and
persist regardless of the registry gap.
- Know which tracker in the hive is currently the settlement leader (holds
`--treasury-keypair`) versus routing-only peers.
- `--hive-secret` / `MESHNET_HIVE_SECRET` configured identically across all
hive members (ADR-0017) — a mismatched secret on restart fails gossip
closed, not open.
- Take a [ledger backup](01-ledger-backup.md) before any upgrade that touches
schema or dependency versions.
## Steps
1. **Routing-only trackers first.** For each non-settlement tracker in the
hive:
a. Confirm it's not the current Raft leader (`GET /v1/raft/status`); if it
is, this restart forces a re-election — acceptable, but expect a brief
registration-proxy gap while a new leader is elected.
b. Stop the process, deploy the new code/config, restart with the same
`--billing-db` / `--accounts-db` / registry DB paths and the same
`--hive-secret` and `--cluster-peers`.
c. Check `/v1/raft/status` and `/v1/registry/wallets` (if registry is
wired) come back consistent with peers within one gossip interval.
d. Move to the next routing tracker only after this one rejoins cleanly.
2. **Settlement tracker last**, and only during a low-settlement-activity
window if possible:
a. Confirm no payout is mid-flight (check tracker logs / pending balance
levels against `--settle-period` / `--payout-threshold`).
b. Stop the process. The treasury keypair file itself is untouched by the
restart — do not regenerate it (see
[treasury key rotation](02-treasury-key-rotation.md) for that separate
procedure).
c. Deploy new code/config, restart with identical `--treasury-keypair`,
`--solana-rpc-url`, `--usdt-mint`, `--settle-period`,
`--payout-threshold`, `--payout-dust-floor`, and DB paths.
d. Verify strike/ban/reputation state (if wired) matches pre-restart values
via `/v1/registry/wallets`, and that billing/accounts ledgers show the
same balances as immediately before shutdown.
3. Confirm all hive members show each other as alive peers and gossip
(`/v1/registry/gossip`, `/v1/billing/gossip`, `/v1/accounts/gossip`) is
flowing without HMAC auth failures in logs (ADR-0017).
## Rollback
- Each tracker's on-disk SQLite files are untouched by a code-only upgrade;
rolling back means redeploying the previous binary/version against the same
DB paths. Because billing/accounts/registry are append-only event logs, a
version rollback does not lose data written by the newer version as long as
the schema didn't change — if the upgrade included a schema migration,
restore from the pre-upgrade [ledger backup](01-ledger-backup.md) instead.
- If a settlement-tracker restart leaves it unable to reach the treasury RPC
endpoint, routing-only trackers continue serving traffic — settlement simply
pauses until the leader recovers; no funds are at risk since payouts require
the loaded keypair.
## Secrets handling
- Never commit `.env.devnet`, `--hive-secret` / `MESHNET_HIVE_SECRET`,
`--validator-service-token`, or the treasury keypair file as part of a
deploy/config change. Deploy scripts should read these from the existing
secrets store, not from a file checked into the repo.