71 lines
3.1 KiB
Markdown
71 lines
3.1 KiB
Markdown
# Runbook 04 — Honest-noise TOPLOC calibration (issue 21)
|
||
|
||
**Status:** engineering complete; **operator action required** before production audit thresholds.
|
||
|
||
**Blocks:** enabling calibrated TOPLOC thresholds on a mainnet / friends-test fleet (issue 21, ADR-0018).
|
||
|
||
## When to run
|
||
|
||
- Before first real-money traffic with audit enforcement enabled.
|
||
- Again whenever the fleet’s **hardware mix** changes materially (new GPU generation, CPU-only nodes added, precision/recipe change per model).
|
||
|
||
Alpha exception: with a **small hired-VPS-only** fleet, `gate_status.ready` may mean “covers every node we operate today” (`--toploc-calibration-gate-min-hardware-profiles 1`).
|
||
|
||
## Prerequisites
|
||
|
||
- Tracker running with billing + registry + `--toploc-calibration-db PATH` (or default under tracker cwd).
|
||
- At least one **solo-capable** node per hardware profile you want in the corpus (full model coverage — partial shards are skipped).
|
||
- Admin or validator credentials (`Authorization` header or validator service token per ADR-0017).
|
||
- Reference validator can replay the fixed calibration prompt (same model/seed as dispatch uses).
|
||
|
||
## Steps
|
||
|
||
1. **Register the fleet** — all nodes you intend to pay on mainnet should be up, admitted (NCA when enabled), and solo-serving the calibration model.
|
||
|
||
2. **Dispatch the job** (admin/validator only):
|
||
|
||
```bash
|
||
curl -X POST "https://<tracker>/v1/calibration/toploc/run" \
|
||
-H "Authorization: Bearer <admin-or-validator-token>" \
|
||
-H "Content-Type: application/json" \
|
||
-d '{}'
|
||
```
|
||
|
||
Partial-shard nodes appear under `skipped_partial_shard_node_ids`. Per-node failures appear under `skipped` with reasons.
|
||
|
||
3. **Wait for completion** — watch tracker logs and node consoles until every solo-capable node has a row in the corpus.
|
||
|
||
4. **Fetch results**:
|
||
|
||
```bash
|
||
curl "https://<tracker>/v1/calibration/toploc/results" \
|
||
-H "Authorization: Bearer <admin-or-validator-token>"
|
||
```
|
||
|
||
Record:
|
||
- `envelope` — p99 metrics + 20% safety margin (recommended tolerances).
|
||
- `gate_status.ready` and `gate_status.hardware_profiles`.
|
||
- `estimated_false_positive_rate` (in-sample sanity check only).
|
||
|
||
5. **Write up thresholds** — paste envelope values into operator notes / issue 21 comment. Do **not** wire into production `ToplocAuditConfig` until you have reviewed FPR on this fleet.
|
||
|
||
6. **Mark issue 21 done** — when corpus covers the launch fleet and thresholds are documented.
|
||
|
||
## Two-wallet / minimal pilot variant
|
||
|
||
If your “fleet” is one node machine + one client:
|
||
|
||
- Run calibration against the **node** profile only (one hardware row is enough for `gate_status` with min profiles = 1).
|
||
- Client wallet is irrelevant to calibration — it never serves inference.
|
||
|
||
## Do not
|
||
|
||
- Enable stricter production audit thresholds before this completes.
|
||
- Reuse a corpus collected on devnet/mock hardware for a different mainnet GPU mix without re-running.
|
||
|
||
## References
|
||
|
||
- Issue: `.scratch/alpha-hardening/issues/21-honest-noise-calibration-corpus.md`
|
||
- Code: `packages/tracker/meshnet_tracker/calibration.py`, `POST /v1/calibration/toploc/run`
|
||
- Validator: `packages/validator/README.md` — TOPLOC audit contract
|