neuron-tai/.scratch/alpha-hardening/runbooks/01-ledger-backup.md

Status: stub

Runbook: Ledger backup

Covers backing up the tracker's authoritative money/trust state — the billing ledger, dashboard accounts DB, and node registry (strike/ban/reputation) — and how to pause hive gossip during the backup window so peers don't replicate against a half-copied file.

Trust assumptions (read first)

Per ADR-0016, one operator-designated tracker holds the treasury keypair and is the source of truth for settlement; other hive members only replicate. Back up that tracker's databases — a follower's copies are eventually consistent, not authoritative. See ADR-0015 for the settlement loop these tables feed.

Prerequisites

• Shell access to the settlement-capable tracker host.
• sqlite3 CLI (or .backup support in the Python sqlite3 module) available for online, consistent snapshots.
• Know the tracker's configured DB paths — defaults, unless overridden by CLI flags:
  • Billing ledger: billing.sqlite (--billing-db, DEFAULT_BILLING_DB_PATH in packages/tracker/meshnet_tracker/billing.py)
  • Dashboard accounts: accounts.sqlite (--accounts-db, DEFAULT_ACCOUNTS_DB_PATH in packages/tracker/meshnet_tracker/accounts.py)
  • Node registry (strike/ban/reputation event log, packages/contracts/meshnet_contracts/__init__.py::RegistryEventLog): path is whatever was passed as registry_db when the tracker's LocalSolanaContracts was constructed. As of this writing the tracker CLI (meshnet_tracker/cli.py) does not expose a --registry-db flag or wire a contracts= instance into TrackerServer by default — confirm with whoever deployed this tracker whether registry persistence is actually enabled before assuming a file exists to back up. If it isn't wired up yet, strike/ban/reputation state is RAM-only and this step is moot until that gap closes (tracked loosely against issue 05).

Steps

1. Identify the actual DB paths in use (check the tracker's start command / systemd unit / process env for --billing-db, --accounts-db, and any registry DB argument).
2. Pause hive gossip on this tracker so peers don't pull a partial/locked file mid-backup:
   • If the tracker is the sole settlement node with no --cluster-peers, gossip is already off — skip to step 3.
   • Otherwise, stop replication by restarting the process without --cluster-peers (or with an empty peer list) for the duration of the backup, or take the backup during a maintenance window with peers temporarily pointed away from this tracker at the load balancer/DNS level. There is currently no live "pause gossip" admin endpoint — this is a process-restart-level operation.
   • Confirm no in-flight /v1/registry/gossip, /v1/billing/gossip, or /v1/accounts/gossip traffic before proceeding (check access logs).
3. Take an online, consistent copy of each SQLite file using the backup API rather than cp (WAL-mode files can be mid-write):

   sqlite3 billing.sqlite ".backup '/backups/billing-$(date +%Y%m%dT%H%M%S).sqlite'"
   sqlite3 accounts.sqlite ".backup '/backups/accounts-$(date +%Y%m%dT%H%M%S).sqlite'"
   # registry DB, if configured:
   sqlite3 registry.sqlite ".backup '/backups/registry-$(date +%Y%m%dT%H%M%S).sqlite'"
   

4. Verify each backup opens and has rows in its expected tables (billing_ledger/event log tables, accounts, registry_events).
5. Resume gossip (restore --cluster-peers / routing) once backups are confirmed good.
6. Ship backups off-host per your normal retention policy. Do not store them alongside .env.devnet or keypair files (see secrets handling below).

Rollback

• If a restore is needed, stop the tracker, replace the live .sqlite file(s) with the chosen backup, and restart. Because billing/accounts/registry each use append-only event logs, a stale restore under-counts recent activity rather than corrupting state — reconcile any gap against node/operator reports for the missing window before resuming payouts.
• If gossip was paused via a peer-list restart, confirm peers re-sync (events_since catch-up) before considering the rollback complete.

Secrets handling

• Never commit .env.devnet, treasury keypair JSON files, --hive-secret, or --validator-service-token values to a repo or ship them inside a DB backup archive. Back these up separately, encrypted, per your existing secrets process.