Status: stub

# Runbook: Tracker upgrade path (rolling restart)

Covers restarting/upgrading tracker processes in a hive without losing
strike/ban/reputation state or interrupting settlement, per the ADR-0016 §4
guarantee that reputation carries forward across restarts.

## Trust assumptions (read first)

Per [ADR-0016](../../../docs/adr/0016-alpha-scope-and-known-limitations.md),
only one operator-designated tracker holds the treasury keypair and runs the
settlement loop ([ADR-0015](../../../docs/adr/0015-usdt-custodial-settlement.md));
other hive members replicate for routing only. Raft (`packages/tracker/meshnet_tracker/raft.py`)
elects a leader for shard-assignment/registration commands — settlement
leadership is a separate, operator-configured concept, not the Raft leader.
Plan restarts so the settlement tracker's downtime window is minimized
independent of routing-tracker restarts.

## Known gap — read before relying on this runbook

Strike/ban/reputation persistence itself was implemented in issue 05
(`packages/contracts/meshnet_contracts/__init__.py::RegistryEventLog`,
SQLite-backed, same pattern as billing/accounts). **As of this writing,
`packages/tracker/meshnet_tracker/cli.py` does not expose a `--registry-db`
flag, nor does it construct a `contracts=` instance to pass into
`TrackerServer`.** Running the tracker via the stock CLI entry point leaves
`server.contracts` as `None`, which means:

- Ban checks (`_registration_ban_error`), reputation-weighted routing
  (`_reputation_multiplier`), and the `/v1/registry/wallets` endpoint are
  inert.
- There is nothing to persist across restarts in that configuration — the
  "survives restart" guarantee only holds for deployments that construct
  `LocalSolanaContracts(registry_db=<path>)` and wire it into `TrackerServer(contracts=...)`
  themselves (e.g. a custom entrypoint or embedding the server programmatically).

Before following the restart steps below, confirm which mode this deployment
runs in. If it's the stock CLI with no custom `contracts` wiring, strike/ban
state is RAM-only regardless of this runbook, and a restart resets it — treat
that as a pre-existing gap to flag to the owner, not something this runbook
can work around.

## Prerequisites

- Confirm registry persistence is actually wired (see gap above) and note the
  registry DB path in use.
- Confirm billing (`--billing-db`) and accounts (`--accounts-db`) persistence
  paths — these already default to `billing.sqlite` / `accounts.sqlite` and
  persist regardless of the registry gap.
- Know which tracker in the hive is currently the settlement leader (holds
  `--treasury-keypair`) versus routing-only peers.
- `--hive-secret` / `MESHNET_HIVE_SECRET` configured identically across all
  hive members (ADR-0017) — a mismatched secret on restart fails gossip
  closed, not open.
- Take a [ledger backup](01-ledger-backup.md) before any upgrade that touches
  schema or dependency versions.

## Steps

1. **Routing-only trackers first.** For each non-settlement tracker in the
   hive:
   a. Confirm it's not the current Raft leader (`GET /v1/raft/status`); if it
      is, this restart forces a re-election — acceptable, but expect a brief
      registration-proxy gap while a new leader is elected.
   b. Stop the process, deploy the new code/config, restart with the same
      `--billing-db` / `--accounts-db` / registry DB paths and the same
      `--hive-secret` and `--cluster-peers`.
   c. Check `/v1/raft/status` and `/v1/registry/wallets` (if registry is
      wired) come back consistent with peers within one gossip interval.
   d. Move to the next routing tracker only after this one rejoins cleanly.
2. **Settlement tracker last**, and only during a low-settlement-activity
   window if possible:
   a. Confirm no payout is mid-flight (check tracker logs / pending balance
      levels against `--settle-period` / `--payout-threshold`).
   b. Stop the process. The treasury keypair file itself is untouched by the
      restart — do not regenerate it (see
      [treasury key rotation](02-treasury-key-rotation.md) for that separate
      procedure).
   c. Deploy new code/config, restart with identical `--treasury-keypair`,
      `--solana-rpc-url`, `--usdt-mint`, `--settle-period`,
      `--payout-threshold`, `--payout-dust-floor`, and DB paths.
   d. Verify strike/ban/reputation state (if wired) matches pre-restart values
      via `/v1/registry/wallets`, and that billing/accounts ledgers show the
      same balances as immediately before shutdown.
3. Confirm all hive members show each other as alive peers and gossip
   (`/v1/registry/gossip`, `/v1/billing/gossip`, `/v1/accounts/gossip`) is
   flowing without HMAC auth failures in logs (ADR-0017).

## Rollback

- Each tracker's on-disk SQLite files are untouched by a code-only upgrade;
  rolling back means redeploying the previous binary/version against the same
  DB paths. Because billing/accounts/registry are append-only event logs, a
  version rollback does not lose data written by the newer version as long as
  the schema didn't change — if the upgrade included a schema migration,
  restore from the pre-upgrade [ledger backup](01-ledger-backup.md) instead.
- If a settlement-tracker restart leaves it unable to reach the treasury RPC
  endpoint, routing-only trackers continue serving traffic — settlement simply
  pauses until the leader recovers; no funds are at risk since payouts require
  the loaded keypair.

## Secrets handling

- Never commit `.env.devnet`, `--hive-secret` / `MESHNET_HIVE_SECRET`,
  `--validator-service-token`, or the treasury keypair file as part of a
  deploy/config change. Deploy scripts should read these from the existing
  secrets store, not from a file checked into the repo.