Runbook 04 — Honest-noise TOPLOC calibration (issue 21)

Status: engineering complete; operator action required before production audit thresholds.

Blocks: enabling calibrated TOPLOC thresholds on a mainnet / friends-test fleet (issue 21, ADR-0018).

When to run

Before first real-money traffic with audit enforcement enabled.
Again whenever the fleet’s hardware mix changes materially (new GPU generation, CPU-only nodes added, precision/recipe change per model).

Alpha exception: with a small hired-VPS-only fleet, gate_status.ready may mean “covers every node we operate today” (--toploc-calibration-gate-min-hardware-profiles 1).

Prerequisites

Tracker running with billing + registry + --toploc-calibration-db PATH (or default under tracker cwd).
At least one solo-capable node per hardware profile you want in the corpus (full model coverage — partial shards are skipped).
Admin or validator credentials (Authorization header or validator service token per ADR-0017).
Reference validator can replay the fixed calibration prompt (same model/seed as dispatch uses).

Steps

Register the fleet — all nodes you intend to pay on mainnet should be up, admitted (NCA when enabled), and solo-serving the calibration model.

Dispatch the job (admin/validator only):

curl -X POST "https://<tracker>/v1/calibration/toploc/run" \
  -H "Authorization: Bearer <admin-or-validator-token>" \
  -H "Content-Type: application/json" \
  -d '{}'

Partial-shard nodes appear under skipped_partial_shard_node_ids. Per-node failures appear under skipped with reasons.

Wait for completion — watch tracker logs and node consoles until every solo-capable node has a row in the corpus.
Fetch results:
```
curl "https://<tracker>/v1/calibration/toploc/results" \
  -H "Authorization: Bearer <admin-or-validator-token>"
```
Record:
- envelope — p99 metrics + 20% safety margin (recommended tolerances).
- gate_status.ready and gate_status.hardware_profiles.
- estimated_false_positive_rate (in-sample sanity check only).
Write up thresholds — paste envelope values into operator notes / issue 21 comment. Do not wire into production ToplocAuditConfig until you have reviewed FPR on this fleet.
Mark issue 21 done — when corpus covers the launch fleet and thresholds are documented.

Two-wallet / minimal pilot variant

If your “fleet” is one node machine + one client:

Run calibration against the node profile only (one hardware row is enough for gate_status with min profiles = 1).
Client wallet is irrelevant to calibration — it never serves inference.

Do not

Enable stricter production audit thresholds before this completes.
Reuse a corpus collected on devnet/mock hardware for a different mainnet GPU mix without re-running.

References

Issue: .scratch/alpha-hardening/issues/21-honest-noise-calibration-corpus.md
Code: packages/tracker/meshnet_tracker/calibration.py, POST /v1/calibration/toploc/run
Validator: packages/validator/README.md — TOPLOC audit contract

3.1 KiB Raw Permalink Blame History Unescape Escape