neuron-tai/.scratch/alpha-hardening/runbooks/02-treasury-key-rotation.md

Status: stub

Runbook: Treasury key rotation (devnet mock-USDT)

Covers rotating the devnet treasury keypair and/or the mock-USDT mint without double-crediting client ledger balances or double-paying nodes.

Trust assumptions (read first)

Per ADR-0015, a single project-owned wallet custodies all funds; the treasury keypair is loaded only on the operator-designated settlement tracker (ADR-0016 §1). Rotating this key is a trusted-operator action — there is no on-chain multisig or trustless handoff in the alpha design. Devnet uses a self-created mock-USDT SPL mint (6 decimals); real USDT only exists on mainnet, so this procedure is devnet-only until a mainnet cutover ADR supersedes it.

Prerequisites

• Access to scripts/devnet_setup.py and its dependencies (solders, meshnet_contracts.solana_adapter.SolanaCustodialTreasury).
• The current treasury keypair path (default ~/.config/solana/meshnet-treasury.json, or whatever --treasury-keypair the running tracker uses) and current MESHNET_USDT_MINT / MESHNET_TREASURY_WALLET values (see .env.devnet, never committed).
• Ability to stop/restart the settlement-capable tracker.
• Confirm the deposit watcher's dedupe state (transaction signatures already credited) is durable — it must survive the rotation so replayed/rescanned transfers under the old wallet don't get re-credited under the new one.

Two rotation scenarios

A. Rotate the treasury keypair only (same mint, same on-chain wallet funds move)

The treasury wallet address changes because it's derived from the keypair, so this requires migrating funds, not just swapping a file.

1. Generate a new keypair (do not reuse _load_or_create_keypair against the old path — write to a new path so both keys exist during the transition):

   python scripts/devnet_setup.py --keypair ~/.config/solana/meshnet-treasury-new.json \
     --mint <EXISTING_MOCK_USDT_MINT> --env-out .env.devnet.new
   

   This creates the new treasury wallet + token account and reuses the existing mint (no new token, so client balances denominated in that mint are unaffected).
2. Drain the old treasury token account to the new one via a single SPL transfer sized to the entire current balance (record the exact amount and the source tx signature before moving anything).
3. Freeze settlement during the drain: stop the settlement-capable tracker (or restart it with no --treasury-keypair so the settlement loop is inert) before step 2, so no payout is in flight against the old wallet while funds move.
4. Update the tracker's --treasury-keypair, --treasury-wallet-derived config (i.e. the new .env.devnet) and restart the tracker pointed at the new keypair.
5. Verify: treasury.get_sol_balance() / mock-USDT balance on the new wallet matches the old wallet's pre-drain balance; old wallet balance is zero.
6. Only after verification, revoke/delete the old keypair file.

B. Rotate the mock-USDT mint (e.g. compromised or mis-configured mint)

This is a bigger change — it invalidates every client's existing off-chain ledger balance denomination reference and any node's pending on-chain payout expectations. Treat as a deliberate migration, not a routine rotation:

1. Settle (pay out) all pending node balances against the old mint before cutover — the pending-balance forfeiture/collateral model (ADR-0015) assumes pending balances are payable in a known mint.
2. Create the new mint and treasury token account:

   python scripts/devnet_setup.py --keypair <treasury-keypair> --env-out .env.devnet
   

   (omit --mint so a fresh mint is created).
3. Update tracker config (MESHNET_USDT_MINT) and restart.
4. Re-mint/airdrop mock USDT to active client wallets under the new mint as needed (--mint-to), since off-chain ledger balances are not automatically re-denominated — this is a devnet convenience step, not a guarantee that would hold for real USDT.

Avoiding double-credit

The deposit watcher (issue 32) dedupes by on-chain transaction signature. The signature space for the old and new treasury token accounts/mints is disjoint, so:

• Do not replay old-wallet deposit history against the new wallet's watcher — it has no record of those signatures and would (correctly) not credit them, but any manual "catch-up crediting" script must not re-process transfers the old watcher already credited. Cross-check the old ledger's credited-tx-sig table before any manual reconciliation entry.
• Keep the old watcher's dedupe DB/table around (don't drop it as part of rotation) until you've confirmed no in-flight deposits to the old address remain unconfirmed.

Rollback

• Scenario A: if the new wallet fails verification, restart the tracker with the old --treasury-keypair — no client-facing state changed since ledger balances are keyed by API key, not treasury wallet address.
• Scenario B: if re-minting under the new mint goes wrong, restart the tracker against the old MESHNET_USDT_MINT config; nothing was destroyed on the old mint.

Secrets handling

• Never commit .env.devnet, .env.devnet.new, or any *treasury*.json keypair file. scripts/devnet_setup.py writes keypairs with 0o600 permissions — preserve that when copying.
• Treat the treasury keypair as the single highest-value secret in this system per ADR-0015/ADR-0016: anyone with it can drain custodial funds.