Files

Dobromir Popov 9abe83b5f4 feat(alpha): complete hardening backlog

Complete the alpha-hardening Ralph task set, including tracker billing/accounting guards, validator fraud-audit primitives, wallet binding proof support, documentation runbooks, and updated tests.

Verification: .venv/bin/python -m compileall -q packages tests; .venv/bin/python -m pytest -q --tb=short (313 passed, 3 skipped, 1 failed: tests/test_mining_cli.py::test_legacy_start_without_port_uses_next_available_port because meshnet-node pid 1263451 is already listening on port 7000).

2026-07-05 21:47:23 +03:00

5.6 KiB

Raw Blame History

Status: stub

Runbook: Tracker upgrade path (rolling restart)

Covers restarting/upgrading tracker processes in a hive without losing strike/ban/reputation state or interrupting settlement, per the ADR-0016 §4 guarantee that reputation carries forward across restarts.

Trust assumptions (read first)

Per ADR-0016, only one operator-designated tracker holds the treasury keypair and runs the settlement loop (ADR-0015); other hive members replicate for routing only. Raft (packages/tracker/meshnet_tracker/raft.py) elects a leader for shard-assignment/registration commands — settlement leadership is a separate, operator-configured concept, not the Raft leader. Plan restarts so the settlement tracker's downtime window is minimized independent of routing-tracker restarts.

Known gap — read before relying on this runbook

Strike/ban/reputation persistence itself was implemented in issue 05 (packages/contracts/meshnet_contracts/__init__.py::RegistryEventLog, SQLite-backed, same pattern as billing/accounts). As of this writing, packages/tracker/meshnet_tracker/cli.py does not expose a --registry-db flag, nor does it construct a contracts= instance to pass into TrackerServer. Running the tracker via the stock CLI entry point leaves server.contracts as None, which means:

Ban checks (_registration_ban_error), reputation-weighted routing (_reputation_multiplier), and the /v1/registry/wallets endpoint are inert.
There is nothing to persist across restarts in that configuration — the "survives restart" guarantee only holds for deployments that construct LocalSolanaContracts(registry_db=<path>) and wire it into TrackerServer(contracts=...) themselves (e.g. a custom entrypoint or embedding the server programmatically).

Before following the restart steps below, confirm which mode this deployment runs in. If it's the stock CLI with no custom contracts wiring, strike/ban state is RAM-only regardless of this runbook, and a restart resets it — treat that as a pre-existing gap to flag to the owner, not something this runbook can work around.

Prerequisites

Confirm registry persistence is actually wired (see gap above) and note the registry DB path in use.
Confirm billing (--billing-db) and accounts (--accounts-db) persistence paths — these already default to billing.sqlite / accounts.sqlite and persist regardless of the registry gap.
Know which tracker in the hive is currently the settlement leader (holds --treasury-keypair) versus routing-only peers.
--hive-secret / MESHNET_HIVE_SECRET configured identically across all hive members (ADR-0017) — a mismatched secret on restart fails gossip closed, not open.
Take a ledger backup before any upgrade that touches schema or dependency versions.

Steps

Routing-only trackers first. For each non-settlement tracker in the hive: a. Confirm it's not the current Raft leader (GET /v1/raft/status); if it is, this restart forces a re-election — acceptable, but expect a brief registration-proxy gap while a new leader is elected. b. Stop the process, deploy the new code/config, restart with the same --billing-db / --accounts-db / registry DB paths and the same --hive-secret and --cluster-peers. c. Check /v1/raft/status and /v1/registry/wallets (if registry is wired) come back consistent with peers within one gossip interval. d. Move to the next routing tracker only after this one rejoins cleanly.
Settlement tracker last, and only during a low-settlement-activity window if possible: a. Confirm no payout is mid-flight (check tracker logs / pending balance levels against --settle-period / --payout-threshold). b. Stop the process. The treasury keypair file itself is untouched by the restart — do not regenerate it (see treasury key rotation for that separate procedure). c. Deploy new code/config, restart with identical --treasury-keypair, --solana-rpc-url, --usdt-mint, --settle-period, --payout-threshold, --payout-dust-floor, and DB paths. d. Verify strike/ban/reputation state (if wired) matches pre-restart values via /v1/registry/wallets, and that billing/accounts ledgers show the same balances as immediately before shutdown.
Confirm all hive members show each other as alive peers and gossip (/v1/registry/gossip, /v1/billing/gossip, /v1/accounts/gossip) is flowing without HMAC auth failures in logs (ADR-0017).

Rollback

Each tracker's on-disk SQLite files are untouched by a code-only upgrade; rolling back means redeploying the previous binary/version against the same DB paths. Because billing/accounts/registry are append-only event logs, a version rollback does not lose data written by the newer version as long as the schema didn't change — if the upgrade included a schema migration, restore from the pre-upgrade ledger backup instead.
If a settlement-tracker restart leaves it unable to reach the treasury RPC endpoint, routing-only trackers continue serving traffic — settlement simply pauses until the leader recovers; no funds are at risk since payouts require the loaded keypair.

Secrets handling

Never commit .env.devnet, --hive-secret / MESHNET_HIVE_SECRET, --validator-service-token, or the treasury keypair file as part of a deploy/config change. Deploy scripts should read these from the existing secrets store, not from a file checked into the repo.

5.6 KiB Raw Blame History