popov/neuron-tai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(alpha): complete hardening backlog

Browse Source 
Complete the alpha-hardening Ralph task set, including tracker billing/accounting guards, validator fraud-audit primitives, wallet binding proof support, documentation runbooks, and updated tests.

Verification: .venv/bin/python -m compileall -q packages tests; .venv/bin/python -m pytest -q --tb=short (313 passed, 3 skipped, 1 failed: tests/test_mining_cli.py::test_legacy_start_without_port_uses_next_available_port because meshnet-node pid 1263451 is already listening on port 7000).
This commit is contained in:
Dobromir Popov
2026-07-05 21:47:23 +03:00
parent c967e5cfc4
commit 9abe83b5f4
45 changed files with 4095 additions and 774 deletions
Download Patch File Download Diff File Expand all files Collapse all files

88
.scratch/alpha-hardening/runbooks/01-ledger-backup.md Normal file
View File

@@ -0,0 +1,88 @@
Status: stub
# Runbook: Ledger backup
Covers backing up the tracker's authoritative money/trust state — the billing
ledger, dashboard accounts DB, and node registry (strike/ban/reputation) — and
how to pause hive gossip during the backup window so peers don't replicate
against a half-copied file.
## Trust assumptions (read first)
Per [ADR-0016](../../../docs/adr/0016-alpha-scope-and-known-limitations.md), one
operator-designated tracker holds the treasury keypair and is the source of
truth for settlement; other hive members only replicate. Back up **that**
tracker's databases — a follower's copies are eventually consistent, not
authoritative. See [ADR-0015](../../../docs/adr/0015-usdt-custodial-settlement.md)
for the settlement loop these tables feed.
## Prerequisites
- Shell access to the settlement-capable tracker host.
- `sqlite3` CLI (or `.backup` support in the Python `sqlite3` module) available
for online, consistent snapshots.
- Know the tracker's configured DB paths — defaults, unless overridden by CLI
flags:
- Billing ledger: `billing.sqlite` (`--billing-db`, `DEFAULT_BILLING_DB_PATH`
in `packages/tracker/meshnet_tracker/billing.py`)
- Dashboard accounts: `accounts.sqlite` (`--accounts-db`,
`DEFAULT_ACCOUNTS_DB_PATH` in `packages/tracker/meshnet_tracker/accounts.py`)
- Node registry (strike/ban/reputation event log,
`packages/contracts/meshnet_contracts/__init__.py::RegistryEventLog`): path
is whatever was passed as `registry_db` when the tracker's
`LocalSolanaContracts` was constructed. **As of this writing the tracker
CLI (`meshnet_tracker/cli.py`) does not expose a `--registry-db` flag or
wire a `contracts=` instance into `TrackerServer` by default** — confirm
with whoever deployed this tracker whether registry persistence is
actually enabled before assuming a file exists to back up. If it isn't
wired up yet, strike/ban/reputation state is RAM-only and this step is
moot until that gap closes (tracked loosely against issue 05).
## Steps
1. Identify the actual DB paths in use (check the tracker's start command /
systemd unit / process env for `--billing-db`, `--accounts-db`, and any
registry DB argument).
2. **Pause hive gossip** on this tracker so peers don't pull a partial/locked
file mid-backup:
- If the tracker is the sole settlement node with no `--cluster-peers`,
gossip is already off — skip to step 3.
- Otherwise, stop replication by restarting the process without
`--cluster-peers` (or with an empty peer list) for the duration of the
backup, or take the backup during a maintenance window with peers
temporarily pointed away from this tracker at the load balancer/DNS
level. There is currently no live "pause gossip" admin endpoint — this is
a process-restart-level operation.
- Confirm no in-flight `/v1/registry/gossip`, `/v1/billing/gossip`, or
`/v1/accounts/gossip` traffic before proceeding (check access logs).
3. Take an online, consistent copy of each SQLite file using the backup API
rather than `cp` (WAL-mode files can be mid-write):
```
sqlite3 billing.sqlite ".backup '/backups/billing-$(date +%Y%m%dT%H%M%S).sqlite'"
sqlite3 accounts.sqlite ".backup '/backups/accounts-$(date +%Y%m%dT%H%M%S).sqlite'"
# registry DB, if configured:
sqlite3 registry.sqlite ".backup '/backups/registry-$(date +%Y%m%dT%H%M%S).sqlite'"
```
4. Verify each backup opens and has rows in its expected tables
(`billing_ledger`/event log tables, `accounts`, `registry_events`).
5. Resume gossip (restore `--cluster-peers` / routing) once backups are
confirmed good.
6. Ship backups off-host per your normal retention policy. Do not store them
alongside `.env.devnet` or keypair files (see secrets handling below).
## Rollback
- If a restore is needed, stop the tracker, replace the live `.sqlite` file(s)
with the chosen backup, and restart. Because billing/accounts/registry each
use append-only event logs, a stale restore under-counts recent activity
rather than corrupting state — reconcile any gap against node/operator
reports for the missing window before resuming payouts.
- If gossip was paused via a peer-list restart, confirm peers re-sync
(`events_since` catch-up) before considering the rollback complete.
## Secrets handling
- Never commit `.env.devnet`, treasury keypair JSON files, `--hive-secret`, or
`--validator-service-token` values to a repo or ship them inside a DB backup
archive. Back these up separately, encrypted, per your existing secrets
process.

112
.scratch/alpha-hardening/runbooks/02-treasury-key-rotation.md Normal file
View File

@@ -0,0 +1,112 @@
Status: stub
# Runbook: Treasury key rotation (devnet mock-USDT)
Covers rotating the devnet treasury keypair and/or the mock-USDT mint without
double-crediting client ledger balances or double-paying nodes.
## Trust assumptions (read first)
Per [ADR-0015](../../../docs/adr/0015-usdt-custodial-settlement.md), a single
project-owned wallet custodies all funds; the treasury keypair is loaded only
on the operator-designated settlement tracker (ADR-0016 §1). Rotating this key
is a trusted-operator action — there is no on-chain multisig or trustless
handoff in the alpha design. Devnet uses a self-created mock-USDT SPL mint
(6 decimals); real USDT only exists on mainnet, so this procedure is
devnet-only until a mainnet cutover ADR supersedes it.
## Prerequisites
- Access to `scripts/devnet_setup.py` and its dependencies (`solders`,
`meshnet_contracts.solana_adapter.SolanaCustodialTreasury`).
- The current treasury keypair path (default
`~/.config/solana/meshnet-treasury.json`, or whatever `--treasury-keypair`
the running tracker uses) and current `MESHNET_USDT_MINT` /
`MESHNET_TREASURY_WALLET` values (see `.env.devnet`, never committed).
- Ability to stop/restart the settlement-capable tracker.
- Confirm the deposit watcher's dedupe state (transaction signatures already
credited) is durable — it must survive the rotation so replayed/rescanned
transfers under the *old* wallet don't get re-credited under the *new* one.
## Two rotation scenarios
### A. Rotate the treasury keypair only (same mint, same on-chain wallet funds move)
The treasury wallet address changes because it's derived from the keypair, so
this requires migrating funds, not just swapping a file.
1. Generate a new keypair (do **not** reuse `_load_or_create_keypair` against
the old path — write to a new path so both keys exist during the
transition):
```
python scripts/devnet_setup.py --keypair ~/.config/solana/meshnet-treasury-new.json \
--mint <EXISTING_MOCK_USDT_MINT> --env-out .env.devnet.new
```
This creates the new treasury wallet + token account and reuses the
existing mint (no new token, so client balances denominated in that mint
are unaffected).
2. Drain the old treasury token account to the new one via a single SPL
transfer sized to the *entire current balance* (record the exact amount
and the source tx signature before moving anything).
3. **Freeze settlement during the drain**: stop the settlement-capable
tracker (or restart it with no `--treasury-keypair` so the settlement loop
is inert) before step 2, so no payout is in flight against the old wallet
while funds move.
4. Update the tracker's `--treasury-keypair`, `--treasury-wallet`-derived
config (i.e. the new `.env.devnet`) and restart the tracker pointed at the
new keypair.
5. Verify: `treasury.get_sol_balance()` / mock-USDT balance on the new wallet
matches the old wallet's pre-drain balance; old wallet balance is zero.
6. Only after verification, revoke/delete the old keypair file.
### B. Rotate the mock-USDT mint (e.g. compromised or mis-configured mint)
This is a bigger change — it invalidates every client's existing off-chain
ledger balance denomination reference and any node's pending on-chain payout
expectations. Treat as a deliberate migration, not a routine rotation:
1. Settle (pay out) all pending node balances against the *old* mint before
cutover — the pending-balance forfeiture/collateral model (ADR-0015)
assumes pending balances are payable in a known mint.
2. Create the new mint and treasury token account:
```
python scripts/devnet_setup.py --keypair <treasury-keypair> --env-out .env.devnet
```
(omit `--mint` so a fresh mint is created).
3. Update tracker config (`MESHNET_USDT_MINT`) and restart.
4. Re-mint/airdrop mock USDT to active client wallets under the new mint as
needed (`--mint-to`), since off-chain ledger balances are *not*
automatically re-denominated — this is a devnet convenience step, not a
guarantee that would hold for real USDT.
## Avoiding double-credit
The deposit watcher (issue 32) dedupes by on-chain transaction signature. The
signature space for the old and new treasury token accounts/mints is
disjoint, so:
- Do not replay old-wallet deposit history against the new wallet's watcher —
it has no record of those signatures and would (correctly) not credit them,
but any manual "catch-up crediting" script must not re-process transfers the
old watcher already credited. Cross-check the old ledger's credited-tx-sig
table before any manual reconciliation entry.
- Keep the old watcher's dedupe DB/table around (don't drop it as part of
rotation) until you've confirmed no in-flight deposits to the old address
remain unconfirmed.
## Rollback
- Scenario A: if the new wallet fails verification, restart the tracker with
the old `--treasury-keypair` — no client-facing state changed since ledger
balances are keyed by API key, not treasury wallet address.
- Scenario B: if re-minting under the new mint goes wrong, restart the
tracker against the old `MESHNET_USDT_MINT` config; nothing was destroyed on
the old mint.
## Secrets handling
- Never commit `.env.devnet`, `.env.devnet.new`, or any `*treasury*.json`
keypair file. `scripts/devnet_setup.py` writes keypairs with `0o600`
permissions — preserve that when copying.
- Treat the treasury keypair as the single highest-value secret in this
system per ADR-0015/ADR-0016: anyone with it can drain custodial funds.

108
.scratch/alpha-hardening/runbooks/03-upgrade-path.md Normal file
View File

@@ -0,0 +1,108 @@
Status: stub
# Runbook: Tracker upgrade path (rolling restart)
Covers restarting/upgrading tracker processes in a hive without losing
strike/ban/reputation state or interrupting settlement, per the ADR-0016 §4
guarantee that reputation carries forward across restarts.
## Trust assumptions (read first)
Per [ADR-0016](../../../docs/adr/0016-alpha-scope-and-known-limitations.md),
only one operator-designated tracker holds the treasury keypair and runs the
settlement loop ([ADR-0015](../../../docs/adr/0015-usdt-custodial-settlement.md));
other hive members replicate for routing only. Raft (`packages/tracker/meshnet_tracker/raft.py`)
elects a leader for shard-assignment/registration commands — settlement
leadership is a separate, operator-configured concept, not the Raft leader.
Plan restarts so the settlement tracker's downtime window is minimized
independent of routing-tracker restarts.
## Known gap — read before relying on this runbook
Strike/ban/reputation persistence itself was implemented in issue 05
(`packages/contracts/meshnet_contracts/__init__.py::RegistryEventLog`,
SQLite-backed, same pattern as billing/accounts). **As of this writing,
`packages/tracker/meshnet_tracker/cli.py` does not expose a `--registry-db`
flag, nor does it construct a `contracts=` instance to pass into
`TrackerServer`.** Running the tracker via the stock CLI entry point leaves
`server.contracts` as `None`, which means:
- Ban checks (`_registration_ban_error`), reputation-weighted routing
(`_reputation_multiplier`), and the `/v1/registry/wallets` endpoint are
inert.
- There is nothing to persist across restarts in that configuration — the
"survives restart" guarantee only holds for deployments that construct
`LocalSolanaContracts(registry_db=<path>)` and wire it into `TrackerServer(contracts=...)`
themselves (e.g. a custom entrypoint or embedding the server programmatically).
Before following the restart steps below, confirm which mode this deployment
runs in. If it's the stock CLI with no custom `contracts` wiring, strike/ban
state is RAM-only regardless of this runbook, and a restart resets it — treat
that as a pre-existing gap to flag to the owner, not something this runbook
can work around.
## Prerequisites
- Confirm registry persistence is actually wired (see gap above) and note the
registry DB path in use.
- Confirm billing (`--billing-db`) and accounts (`--accounts-db`) persistence
paths — these already default to `billing.sqlite` / `accounts.sqlite` and
persist regardless of the registry gap.
- Know which tracker in the hive is currently the settlement leader (holds
`--treasury-keypair`) versus routing-only peers.
- `--hive-secret` / `MESHNET_HIVE_SECRET` configured identically across all
hive members (ADR-0017) — a mismatched secret on restart fails gossip
closed, not open.
- Take a [ledger backup](01-ledger-backup.md) before any upgrade that touches
schema or dependency versions.
## Steps
1. **Routing-only trackers first.** For each non-settlement tracker in the
hive:
a. Confirm it's not the current Raft leader (`GET /v1/raft/status`); if it
is, this restart forces a re-election — acceptable, but expect a brief
registration-proxy gap while a new leader is elected.
b. Stop the process, deploy the new code/config, restart with the same
`--billing-db` / `--accounts-db` / registry DB paths and the same
`--hive-secret` and `--cluster-peers`.
c. Check `/v1/raft/status` and `/v1/registry/wallets` (if registry is
wired) come back consistent with peers within one gossip interval.
d. Move to the next routing tracker only after this one rejoins cleanly.
2. **Settlement tracker last**, and only during a low-settlement-activity
window if possible:
a. Confirm no payout is mid-flight (check tracker logs / pending balance
levels against `--settle-period` / `--payout-threshold`).
b. Stop the process. The treasury keypair file itself is untouched by the
restart — do not regenerate it (see
[treasury key rotation](02-treasury-key-rotation.md) for that separate
procedure).
c. Deploy new code/config, restart with identical `--treasury-keypair`,
`--solana-rpc-url`, `--usdt-mint`, `--settle-period`,
`--payout-threshold`, `--payout-dust-floor`, and DB paths.
d. Verify strike/ban/reputation state (if wired) matches pre-restart values
via `/v1/registry/wallets`, and that billing/accounts ledgers show the
same balances as immediately before shutdown.
3. Confirm all hive members show each other as alive peers and gossip
(`/v1/registry/gossip`, `/v1/billing/gossip`, `/v1/accounts/gossip`) is
flowing without HMAC auth failures in logs (ADR-0017).
## Rollback
- Each tracker's on-disk SQLite files are untouched by a code-only upgrade;
rolling back means redeploying the previous binary/version against the same
DB paths. Because billing/accounts/registry are append-only event logs, a
version rollback does not lose data written by the newer version as long as
the schema didn't change — if the upgrade included a schema migration,
restore from the pre-upgrade [ledger backup](01-ledger-backup.md) instead.
- If a settlement-tracker restart leaves it unable to reach the treasury RPC
endpoint, routing-only trackers continue serving traffic — settlement simply
pauses until the leader recovers; no funds are at risk since payouts require
the loaded keypair.
## Secrets handling
- Never commit `.env.devnet`, `--hive-secret` / `MESHNET_HIVE_SECRET`,
`--validator-service-token`, or the treasury keypair file as part of a
deploy/config change. Deploy scripts should read these from the existing
secrets store, not from a file checked into the repo.