feat(alpha): complete hardening backlog
Complete the alpha-hardening Ralph task set, including tracker billing/accounting guards, validator fraud-audit primitives, wallet binding proof support, documentation runbooks, and updated tests. Verification: .venv/bin/python -m compileall -q packages tests; .venv/bin/python -m pytest -q --tb=short (313 passed, 3 skipped, 1 failed: tests/test_mining_cli.py::test_legacy_start_without_port_uses_next_available_port because meshnet-node pid 1263451 is already listening on port 7000).
This commit is contained in:
108
.scratch/alpha-hardening/runbooks/03-upgrade-path.md
Normal file
108
.scratch/alpha-hardening/runbooks/03-upgrade-path.md
Normal file
@@ -0,0 +1,108 @@
|
||||
Status: stub
|
||||
|
||||
# Runbook: Tracker upgrade path (rolling restart)
|
||||
|
||||
Covers restarting/upgrading tracker processes in a hive without losing
|
||||
strike/ban/reputation state or interrupting settlement, per the ADR-0016 §4
|
||||
guarantee that reputation carries forward across restarts.
|
||||
|
||||
## Trust assumptions (read first)
|
||||
|
||||
Per [ADR-0016](../../../docs/adr/0016-alpha-scope-and-known-limitations.md),
|
||||
only one operator-designated tracker holds the treasury keypair and runs the
|
||||
settlement loop ([ADR-0015](../../../docs/adr/0015-usdt-custodial-settlement.md));
|
||||
other hive members replicate for routing only. Raft (`packages/tracker/meshnet_tracker/raft.py`)
|
||||
elects a leader for shard-assignment/registration commands — settlement
|
||||
leadership is a separate, operator-configured concept, not the Raft leader.
|
||||
Plan restarts so the settlement tracker's downtime window is minimized
|
||||
independent of routing-tracker restarts.
|
||||
|
||||
## Known gap — read before relying on this runbook
|
||||
|
||||
Strike/ban/reputation persistence itself was implemented in issue 05
|
||||
(`packages/contracts/meshnet_contracts/__init__.py::RegistryEventLog`,
|
||||
SQLite-backed, same pattern as billing/accounts). **As of this writing,
|
||||
`packages/tracker/meshnet_tracker/cli.py` does not expose a `--registry-db`
|
||||
flag, nor does it construct a `contracts=` instance to pass into
|
||||
`TrackerServer`.** Running the tracker via the stock CLI entry point leaves
|
||||
`server.contracts` as `None`, which means:
|
||||
|
||||
- Ban checks (`_registration_ban_error`), reputation-weighted routing
|
||||
(`_reputation_multiplier`), and the `/v1/registry/wallets` endpoint are
|
||||
inert.
|
||||
- There is nothing to persist across restarts in that configuration — the
|
||||
"survives restart" guarantee only holds for deployments that construct
|
||||
`LocalSolanaContracts(registry_db=<path>)` and wire it into `TrackerServer(contracts=...)`
|
||||
themselves (e.g. a custom entrypoint or embedding the server programmatically).
|
||||
|
||||
Before following the restart steps below, confirm which mode this deployment
|
||||
runs in. If it's the stock CLI with no custom `contracts` wiring, strike/ban
|
||||
state is RAM-only regardless of this runbook, and a restart resets it — treat
|
||||
that as a pre-existing gap to flag to the owner, not something this runbook
|
||||
can work around.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Confirm registry persistence is actually wired (see gap above) and note the
|
||||
registry DB path in use.
|
||||
- Confirm billing (`--billing-db`) and accounts (`--accounts-db`) persistence
|
||||
paths — these already default to `billing.sqlite` / `accounts.sqlite` and
|
||||
persist regardless of the registry gap.
|
||||
- Know which tracker in the hive is currently the settlement leader (holds
|
||||
`--treasury-keypair`) versus routing-only peers.
|
||||
- `--hive-secret` / `MESHNET_HIVE_SECRET` configured identically across all
|
||||
hive members (ADR-0017) — a mismatched secret on restart fails gossip
|
||||
closed, not open.
|
||||
- Take a [ledger backup](01-ledger-backup.md) before any upgrade that touches
|
||||
schema or dependency versions.
|
||||
|
||||
## Steps
|
||||
|
||||
1. **Routing-only trackers first.** For each non-settlement tracker in the
|
||||
hive:
|
||||
a. Confirm it's not the current Raft leader (`GET /v1/raft/status`); if it
|
||||
is, this restart forces a re-election — acceptable, but expect a brief
|
||||
registration-proxy gap while a new leader is elected.
|
||||
b. Stop the process, deploy the new code/config, restart with the same
|
||||
`--billing-db` / `--accounts-db` / registry DB paths and the same
|
||||
`--hive-secret` and `--cluster-peers`.
|
||||
c. Check `/v1/raft/status` and `/v1/registry/wallets` (if registry is
|
||||
wired) come back consistent with peers within one gossip interval.
|
||||
d. Move to the next routing tracker only after this one rejoins cleanly.
|
||||
2. **Settlement tracker last**, and only during a low-settlement-activity
|
||||
window if possible:
|
||||
a. Confirm no payout is mid-flight (check tracker logs / pending balance
|
||||
levels against `--settle-period` / `--payout-threshold`).
|
||||
b. Stop the process. The treasury keypair file itself is untouched by the
|
||||
restart — do not regenerate it (see
|
||||
[treasury key rotation](02-treasury-key-rotation.md) for that separate
|
||||
procedure).
|
||||
c. Deploy new code/config, restart with identical `--treasury-keypair`,
|
||||
`--solana-rpc-url`, `--usdt-mint`, `--settle-period`,
|
||||
`--payout-threshold`, `--payout-dust-floor`, and DB paths.
|
||||
d. Verify strike/ban/reputation state (if wired) matches pre-restart values
|
||||
via `/v1/registry/wallets`, and that billing/accounts ledgers show the
|
||||
same balances as immediately before shutdown.
|
||||
3. Confirm all hive members show each other as alive peers and gossip
|
||||
(`/v1/registry/gossip`, `/v1/billing/gossip`, `/v1/accounts/gossip`) is
|
||||
flowing without HMAC auth failures in logs (ADR-0017).
|
||||
|
||||
## Rollback
|
||||
|
||||
- Each tracker's on-disk SQLite files are untouched by a code-only upgrade;
|
||||
rolling back means redeploying the previous binary/version against the same
|
||||
DB paths. Because billing/accounts/registry are append-only event logs, a
|
||||
version rollback does not lose data written by the newer version as long as
|
||||
the schema didn't change — if the upgrade included a schema migration,
|
||||
restore from the pre-upgrade [ledger backup](01-ledger-backup.md) instead.
|
||||
- If a settlement-tracker restart leaves it unable to reach the treasury RPC
|
||||
endpoint, routing-only trackers continue serving traffic — settlement simply
|
||||
pauses until the leader recovers; no funds are at risk since payouts require
|
||||
the loaded keypair.
|
||||
|
||||
## Secrets handling
|
||||
|
||||
- Never commit `.env.devnet`, `--hive-secret` / `MESHNET_HIVE_SECRET`,
|
||||
`--validator-service-token`, or the treasury keypair file as part of a
|
||||
deploy/config change. Deploy scripts should read these from the existing
|
||||
secrets store, not from a file checked into the repo.
|
||||
Reference in New Issue
Block a user