feat: harden node placement and partial model loading

This commit is contained in:
Dobromir Popov
2026-07-13 21:58:08 +02:00
parent a6bcc69288
commit 5d87e81bc9
21 changed files with 497 additions and 55 deletions

View File

@@ -0,0 +1,70 @@
# Runbook 04 — Honest-noise TOPLOC calibration (issue 21)
**Status:** engineering complete; **operator action required** before production audit thresholds.
**Blocks:** enabling calibrated TOPLOC thresholds on a mainnet / friends-test fleet (issue 21, ADR-0018).
## When to run
- Before first real-money traffic with audit enforcement enabled.
- Again whenever the fleets **hardware mix** changes materially (new GPU generation, CPU-only nodes added, precision/recipe change per model).
Alpha exception: with a **small hired-VPS-only** fleet, `gate_status.ready` may mean “covers every node we operate today” (`--toploc-calibration-gate-min-hardware-profiles 1`).
## Prerequisites
- Tracker running with billing + registry + `--toploc-calibration-db PATH` (or default under tracker cwd).
- At least one **solo-capable** node per hardware profile you want in the corpus (full model coverage — partial shards are skipped).
- Admin or validator credentials (`Authorization` header or validator service token per ADR-0017).
- Reference validator can replay the fixed calibration prompt (same model/seed as dispatch uses).
## Steps
1. **Register the fleet** — all nodes you intend to pay on mainnet should be up, admitted (NCA when enabled), and solo-serving the calibration model.
2. **Dispatch the job** (admin/validator only):
```bash
curl -X POST "https://<tracker>/v1/calibration/toploc/run" \
-H "Authorization: Bearer <admin-or-validator-token>" \
-H "Content-Type: application/json" \
-d '{}'
```
Partial-shard nodes appear under `skipped_partial_shard_node_ids`. Per-node failures appear under `skipped` with reasons.
3. **Wait for completion** — watch tracker logs and node consoles until every solo-capable node has a row in the corpus.
4. **Fetch results**:
```bash
curl "https://<tracker>/v1/calibration/toploc/results" \
-H "Authorization: Bearer <admin-or-validator-token>"
```
Record:
- `envelope` — p99 metrics + 20% safety margin (recommended tolerances).
- `gate_status.ready` and `gate_status.hardware_profiles`.
- `estimated_false_positive_rate` (in-sample sanity check only).
5. **Write up thresholds** — paste envelope values into operator notes / issue 21 comment. Do **not** wire into production `ToplocAuditConfig` until you have reviewed FPR on this fleet.
6. **Mark issue 21 done** — when corpus covers the launch fleet and thresholds are documented.
## Two-wallet / minimal pilot variant
If your “fleet” is one node machine + one client:
- Run calibration against the **node** profile only (one hardware row is enough for `gate_status` with min profiles = 1).
- Client wallet is irrelevant to calibration — it never serves inference.
## Do not
- Enable stricter production audit thresholds before this completes.
- Reuse a corpus collected on devnet/mock hardware for a different mainnet GPU mix without re-running.
## References
- Issue: `.scratch/alpha-hardening/issues/21-honest-noise-calibration-corpus.md`
- Code: `packages/tracker/meshnet_tracker/calibration.py`, `POST /v1/calibration/toploc/run`
- Validator: `packages/validator/README.md` — TOPLOC audit contract