Status: stub # Runbook: Ledger backup Covers backing up the tracker's authoritative money/trust state — the billing ledger, dashboard accounts DB, and node registry (strike/ban/reputation) — and how to pause hive gossip during the backup window so peers don't replicate against a half-copied file. ## Trust assumptions (read first) Per [ADR-0016](../../../docs/adr/0016-alpha-scope-and-known-limitations.md), one operator-designated tracker holds the treasury keypair and is the source of truth for settlement; other hive members only replicate. Back up **that** tracker's databases — a follower's copies are eventually consistent, not authoritative. See [ADR-0015](../../../docs/adr/0015-usdt-custodial-settlement.md) for the settlement loop these tables feed. ## Prerequisites - Shell access to the settlement-capable tracker host. - `sqlite3` CLI (or `.backup` support in the Python `sqlite3` module) available for online, consistent snapshots. - Know the tracker's configured DB paths — defaults, unless overridden by CLI flags: - Billing ledger: `billing.sqlite` (`--billing-db`, `DEFAULT_BILLING_DB_PATH` in `packages/tracker/meshnet_tracker/billing.py`) - Dashboard accounts: `accounts.sqlite` (`--accounts-db`, `DEFAULT_ACCOUNTS_DB_PATH` in `packages/tracker/meshnet_tracker/accounts.py`) - Node registry (strike/ban/reputation event log, `packages/contracts/meshnet_contracts/__init__.py::RegistryEventLog`): path is whatever was passed as `registry_db` when the tracker's `LocalSolanaContracts` was constructed. **As of this writing the tracker CLI (`meshnet_tracker/cli.py`) does not expose a `--registry-db` flag or wire a `contracts=` instance into `TrackerServer` by default** — confirm with whoever deployed this tracker whether registry persistence is actually enabled before assuming a file exists to back up. If it isn't wired up yet, strike/ban/reputation state is RAM-only and this step is moot until that gap closes (tracked loosely against issue 05). ## Steps 1. Identify the actual DB paths in use (check the tracker's start command / systemd unit / process env for `--billing-db`, `--accounts-db`, and any registry DB argument). 2. **Pause hive gossip** on this tracker so peers don't pull a partial/locked file mid-backup: - If the tracker is the sole settlement node with no `--cluster-peers`, gossip is already off — skip to step 3. - Otherwise, stop replication by restarting the process without `--cluster-peers` (or with an empty peer list) for the duration of the backup, or take the backup during a maintenance window with peers temporarily pointed away from this tracker at the load balancer/DNS level. There is currently no live "pause gossip" admin endpoint — this is a process-restart-level operation. - Confirm no in-flight `/v1/registry/gossip`, `/v1/billing/gossip`, or `/v1/accounts/gossip` traffic before proceeding (check access logs). 3. Take an online, consistent copy of each SQLite file using the backup API rather than `cp` (WAL-mode files can be mid-write): ``` sqlite3 billing.sqlite ".backup '/backups/billing-$(date +%Y%m%dT%H%M%S).sqlite'" sqlite3 accounts.sqlite ".backup '/backups/accounts-$(date +%Y%m%dT%H%M%S).sqlite'" # registry DB, if configured: sqlite3 registry.sqlite ".backup '/backups/registry-$(date +%Y%m%dT%H%M%S).sqlite'" ``` 4. Verify each backup opens and has rows in its expected tables (`billing_ledger`/event log tables, `accounts`, `registry_events`). 5. Resume gossip (restore `--cluster-peers` / routing) once backups are confirmed good. 6. Ship backups off-host per your normal retention policy. Do not store them alongside `.env.devnet` or keypair files (see secrets handling below). ## Rollback - If a restore is needed, stop the tracker, replace the live `.sqlite` file(s) with the chosen backup, and restart. Because billing/accounts/registry each use append-only event logs, a stale restore under-counts recent activity rather than corrupting state — reconcile any gap against node/operator reports for the missing window before resuming payouts. - If gossip was paused via a peer-list restart, confirm peers re-sync (`events_since` catch-up) before considering the rollback complete. ## Secrets handling - Never commit `.env.devnet`, treasury keypair JSON files, `--hive-secret`, or `--validator-service-token` values to a repo or ship them inside a DB backup archive. Back these up separately, encrypted, per your existing secrets process.