9abe83b5f4
Complete the alpha-hardening Ralph task set, including tracker billing/accounting guards, validator fraud-audit primitives, wallet binding proof support, documentation runbooks, and updated tests. Verification: .venv/bin/python -m compileall -q packages tests; .venv/bin/python -m pytest -q --tb=short (313 passed, 3 skipped, 1 failed: tests/test_mining_cli.py::test_legacy_start_without_port_uses_next_available_port because meshnet-node pid 1263451 is already listening on port 7000).
109 lines
5.6 KiB
Markdown
109 lines
5.6 KiB
Markdown
Status: stub
|
|
|
|
# Runbook: Tracker upgrade path (rolling restart)
|
|
|
|
Covers restarting/upgrading tracker processes in a hive without losing
|
|
strike/ban/reputation state or interrupting settlement, per the ADR-0016 §4
|
|
guarantee that reputation carries forward across restarts.
|
|
|
|
## Trust assumptions (read first)
|
|
|
|
Per [ADR-0016](../../../docs/adr/0016-alpha-scope-and-known-limitations.md),
|
|
only one operator-designated tracker holds the treasury keypair and runs the
|
|
settlement loop ([ADR-0015](../../../docs/adr/0015-usdt-custodial-settlement.md));
|
|
other hive members replicate for routing only. Raft (`packages/tracker/meshnet_tracker/raft.py`)
|
|
elects a leader for shard-assignment/registration commands — settlement
|
|
leadership is a separate, operator-configured concept, not the Raft leader.
|
|
Plan restarts so the settlement tracker's downtime window is minimized
|
|
independent of routing-tracker restarts.
|
|
|
|
## Known gap — read before relying on this runbook
|
|
|
|
Strike/ban/reputation persistence itself was implemented in issue 05
|
|
(`packages/contracts/meshnet_contracts/__init__.py::RegistryEventLog`,
|
|
SQLite-backed, same pattern as billing/accounts). **As of this writing,
|
|
`packages/tracker/meshnet_tracker/cli.py` does not expose a `--registry-db`
|
|
flag, nor does it construct a `contracts=` instance to pass into
|
|
`TrackerServer`.** Running the tracker via the stock CLI entry point leaves
|
|
`server.contracts` as `None`, which means:
|
|
|
|
- Ban checks (`_registration_ban_error`), reputation-weighted routing
|
|
(`_reputation_multiplier`), and the `/v1/registry/wallets` endpoint are
|
|
inert.
|
|
- There is nothing to persist across restarts in that configuration — the
|
|
"survives restart" guarantee only holds for deployments that construct
|
|
`LocalSolanaContracts(registry_db=<path>)` and wire it into `TrackerServer(contracts=...)`
|
|
themselves (e.g. a custom entrypoint or embedding the server programmatically).
|
|
|
|
Before following the restart steps below, confirm which mode this deployment
|
|
runs in. If it's the stock CLI with no custom `contracts` wiring, strike/ban
|
|
state is RAM-only regardless of this runbook, and a restart resets it — treat
|
|
that as a pre-existing gap to flag to the owner, not something this runbook
|
|
can work around.
|
|
|
|
## Prerequisites
|
|
|
|
- Confirm registry persistence is actually wired (see gap above) and note the
|
|
registry DB path in use.
|
|
- Confirm billing (`--billing-db`) and accounts (`--accounts-db`) persistence
|
|
paths — these already default to `billing.sqlite` / `accounts.sqlite` and
|
|
persist regardless of the registry gap.
|
|
- Know which tracker in the hive is currently the settlement leader (holds
|
|
`--treasury-keypair`) versus routing-only peers.
|
|
- `--hive-secret` / `MESHNET_HIVE_SECRET` configured identically across all
|
|
hive members (ADR-0017) — a mismatched secret on restart fails gossip
|
|
closed, not open.
|
|
- Take a [ledger backup](01-ledger-backup.md) before any upgrade that touches
|
|
schema or dependency versions.
|
|
|
|
## Steps
|
|
|
|
1. **Routing-only trackers first.** For each non-settlement tracker in the
|
|
hive:
|
|
a. Confirm it's not the current Raft leader (`GET /v1/raft/status`); if it
|
|
is, this restart forces a re-election — acceptable, but expect a brief
|
|
registration-proxy gap while a new leader is elected.
|
|
b. Stop the process, deploy the new code/config, restart with the same
|
|
`--billing-db` / `--accounts-db` / registry DB paths and the same
|
|
`--hive-secret` and `--cluster-peers`.
|
|
c. Check `/v1/raft/status` and `/v1/registry/wallets` (if registry is
|
|
wired) come back consistent with peers within one gossip interval.
|
|
d. Move to the next routing tracker only after this one rejoins cleanly.
|
|
2. **Settlement tracker last**, and only during a low-settlement-activity
|
|
window if possible:
|
|
a. Confirm no payout is mid-flight (check tracker logs / pending balance
|
|
levels against `--settle-period` / `--payout-threshold`).
|
|
b. Stop the process. The treasury keypair file itself is untouched by the
|
|
restart — do not regenerate it (see
|
|
[treasury key rotation](02-treasury-key-rotation.md) for that separate
|
|
procedure).
|
|
c. Deploy new code/config, restart with identical `--treasury-keypair`,
|
|
`--solana-rpc-url`, `--usdt-mint`, `--settle-period`,
|
|
`--payout-threshold`, `--payout-dust-floor`, and DB paths.
|
|
d. Verify strike/ban/reputation state (if wired) matches pre-restart values
|
|
via `/v1/registry/wallets`, and that billing/accounts ledgers show the
|
|
same balances as immediately before shutdown.
|
|
3. Confirm all hive members show each other as alive peers and gossip
|
|
(`/v1/registry/gossip`, `/v1/billing/gossip`, `/v1/accounts/gossip`) is
|
|
flowing without HMAC auth failures in logs (ADR-0017).
|
|
|
|
## Rollback
|
|
|
|
- Each tracker's on-disk SQLite files are untouched by a code-only upgrade;
|
|
rolling back means redeploying the previous binary/version against the same
|
|
DB paths. Because billing/accounts/registry are append-only event logs, a
|
|
version rollback does not lose data written by the newer version as long as
|
|
the schema didn't change — if the upgrade included a schema migration,
|
|
restore from the pre-upgrade [ledger backup](01-ledger-backup.md) instead.
|
|
- If a settlement-tracker restart leaves it unable to reach the treasury RPC
|
|
endpoint, routing-only trackers continue serving traffic — settlement simply
|
|
pauses until the leader recovers; no funds are at risk since payouts require
|
|
the loaded keypair.
|
|
|
|
## Secrets handling
|
|
|
|
- Never commit `.env.devnet`, `--hive-secret` / `MESHNET_HIVE_SECRET`,
|
|
`--validator-service-token`, or the treasury keypair file as part of a
|
|
deploy/config change. Deploy scripts should read these from the existing
|
|
secrets store, not from a file checked into the repo.
|