9abe83b5f4
Complete the alpha-hardening Ralph task set, including tracker billing/accounting guards, validator fraud-audit primitives, wallet binding proof support, documentation runbooks, and updated tests. Verification: .venv/bin/python -m compileall -q packages tests; .venv/bin/python -m pytest -q --tb=short (313 passed, 3 skipped, 1 failed: tests/test_mining_cli.py::test_legacy_start_without_port_uses_next_available_port because meshnet-node pid 1263451 is already listening on port 7000).
89 lines
4.5 KiB
Markdown
89 lines
4.5 KiB
Markdown
Status: stub
|
|
|
|
# Runbook: Ledger backup
|
|
|
|
Covers backing up the tracker's authoritative money/trust state — the billing
|
|
ledger, dashboard accounts DB, and node registry (strike/ban/reputation) — and
|
|
how to pause hive gossip during the backup window so peers don't replicate
|
|
against a half-copied file.
|
|
|
|
## Trust assumptions (read first)
|
|
|
|
Per [ADR-0016](../../../docs/adr/0016-alpha-scope-and-known-limitations.md), one
|
|
operator-designated tracker holds the treasury keypair and is the source of
|
|
truth for settlement; other hive members only replicate. Back up **that**
|
|
tracker's databases — a follower's copies are eventually consistent, not
|
|
authoritative. See [ADR-0015](../../../docs/adr/0015-usdt-custodial-settlement.md)
|
|
for the settlement loop these tables feed.
|
|
|
|
## Prerequisites
|
|
|
|
- Shell access to the settlement-capable tracker host.
|
|
- `sqlite3` CLI (or `.backup` support in the Python `sqlite3` module) available
|
|
for online, consistent snapshots.
|
|
- Know the tracker's configured DB paths — defaults, unless overridden by CLI
|
|
flags:
|
|
- Billing ledger: `billing.sqlite` (`--billing-db`, `DEFAULT_BILLING_DB_PATH`
|
|
in `packages/tracker/meshnet_tracker/billing.py`)
|
|
- Dashboard accounts: `accounts.sqlite` (`--accounts-db`,
|
|
`DEFAULT_ACCOUNTS_DB_PATH` in `packages/tracker/meshnet_tracker/accounts.py`)
|
|
- Node registry (strike/ban/reputation event log,
|
|
`packages/contracts/meshnet_contracts/__init__.py::RegistryEventLog`): path
|
|
is whatever was passed as `registry_db` when the tracker's
|
|
`LocalSolanaContracts` was constructed. **As of this writing the tracker
|
|
CLI (`meshnet_tracker/cli.py`) does not expose a `--registry-db` flag or
|
|
wire a `contracts=` instance into `TrackerServer` by default** — confirm
|
|
with whoever deployed this tracker whether registry persistence is
|
|
actually enabled before assuming a file exists to back up. If it isn't
|
|
wired up yet, strike/ban/reputation state is RAM-only and this step is
|
|
moot until that gap closes (tracked loosely against issue 05).
|
|
|
|
## Steps
|
|
|
|
1. Identify the actual DB paths in use (check the tracker's start command /
|
|
systemd unit / process env for `--billing-db`, `--accounts-db`, and any
|
|
registry DB argument).
|
|
2. **Pause hive gossip** on this tracker so peers don't pull a partial/locked
|
|
file mid-backup:
|
|
- If the tracker is the sole settlement node with no `--cluster-peers`,
|
|
gossip is already off — skip to step 3.
|
|
- Otherwise, stop replication by restarting the process without
|
|
`--cluster-peers` (or with an empty peer list) for the duration of the
|
|
backup, or take the backup during a maintenance window with peers
|
|
temporarily pointed away from this tracker at the load balancer/DNS
|
|
level. There is currently no live "pause gossip" admin endpoint — this is
|
|
a process-restart-level operation.
|
|
- Confirm no in-flight `/v1/registry/gossip`, `/v1/billing/gossip`, or
|
|
`/v1/accounts/gossip` traffic before proceeding (check access logs).
|
|
3. Take an online, consistent copy of each SQLite file using the backup API
|
|
rather than `cp` (WAL-mode files can be mid-write):
|
|
```
|
|
sqlite3 billing.sqlite ".backup '/backups/billing-$(date +%Y%m%dT%H%M%S).sqlite'"
|
|
sqlite3 accounts.sqlite ".backup '/backups/accounts-$(date +%Y%m%dT%H%M%S).sqlite'"
|
|
# registry DB, if configured:
|
|
sqlite3 registry.sqlite ".backup '/backups/registry-$(date +%Y%m%dT%H%M%S).sqlite'"
|
|
```
|
|
4. Verify each backup opens and has rows in its expected tables
|
|
(`billing_ledger`/event log tables, `accounts`, `registry_events`).
|
|
5. Resume gossip (restore `--cluster-peers` / routing) once backups are
|
|
confirmed good.
|
|
6. Ship backups off-host per your normal retention policy. Do not store them
|
|
alongside `.env.devnet` or keypair files (see secrets handling below).
|
|
|
|
## Rollback
|
|
|
|
- If a restore is needed, stop the tracker, replace the live `.sqlite` file(s)
|
|
with the chosen backup, and restart. Because billing/accounts/registry each
|
|
use append-only event logs, a stale restore under-counts recent activity
|
|
rather than corrupting state — reconcile any gap against node/operator
|
|
reports for the missing window before resuming payouts.
|
|
- If gossip was paused via a peer-list restart, confirm peers re-sync
|
|
(`events_since` catch-up) before considering the rollback complete.
|
|
|
|
## Secrets handling
|
|
|
|
- Never commit `.env.devnet`, treasury keypair JSON files, `--hive-secret`, or
|
|
`--validator-service-token` values to a repo or ship them inside a DB backup
|
|
archive. Back these up separately, encrypted, per your existing secrets
|
|
process.
|