neuron-tai/.scratch/alpha-hardening/issues/21-honest-noise-calibration-corpus.md

Status: ready-for-human

**BLOCKS ALPHA RELEASE.** Scoped 2026-07-06 during alpha-launch-readiness grilling session — must complete before real-money mainnet USDT traffic goes live for the friends + hired-VPS-host launch. Loose/uncalibrated thresholds + manual admin slash-reversal are the stopgap only until this closes.

**Engineering complete 2026-07-06; blocked on a human running it against the real hired-VPS fleet before launch.** The three code gaps below are closed and unit-tested (see Deliverables), but nothing in a dev session can stand in for actually dispatching the job at real hardware — that step, plus the threshold/FPR write-up that depends on its output, needs an operator with the live fleet. See the validator README's "Honest-noise calibration corpus" section for the operational how-to.

# 21 — Honest-noise TOPLOC calibration corpus

## What to build

Before enabling production TOPLOC audit thresholds, collect an **honest-noise baseline** across the active fleet. Run identical inference jobs on every active node/GPU combo; measure the divergence envelope (TOPLOC exponent/mantissa deltas, logprob-rank spread) under real hardware variance. This must be driven by the tracker (scheduled/dispatched job), not a manual one-off script, so it can be re-run as the fleet's hardware mix changes.

Per [ADR-0018 consequences](../../docs/adr/0018-fraud-detection-verification-and-reputation.md): threshold calibration requires an honest-noise corpus across the fleet before production thresholds.

Research anchor: `.scratch/alpha-hardening/research-verifiable-inference.md` §8 layer 3 — "collect this first — run identical jobs across the current node fleet to measure the honest divergence envelope before setting thresholds."

**Launch context (why this is buildable now, not a research project):** first-launch nodes are hired VPS/VPC hosts under our own direct control (test infrastructure we pay for, not third-party volunteers) — not a long-term topology, but risk-free for calibration purposes since there's no external party to dispute a bad reading. Friends are client-side users of the API in this phase, not node operators. Run the calibration pass against this small, fully-controlled fleet first; hired hosts stay on probation (no upfront stake) until it's done, then move to paid USDT serving once thresholds derive from their own hardware.

**Current gap (historical — closed 2026-07-06):** the three engineering pieces below were missing when this issue was filed; all are now implemented and unit-tested. Remaining work is the human calibration run on the live hired-VPS fleet.

1. `verify_activation_proofs()` (`packages/validator/meshnet_validator/audit.py:94-127`) returns a **plain bool** — no raw TOPLOC divergence/distance value is ever computed or surfaced. Every "done" fraud-detection issue (06–10) currently runs on a guessed threshold baked into that bool, not a calibrated one.
2. Fleet dispatch exists but is the wrong shape: `_handle_benchmark_hop_penalty` / `_handle_benchmark_results` (`packages/tracker/meshnet_tracker/server.py:2998-3104`, from the old US-030 latency work) targets pinned 1–3-node *routes* and measures latency, not TOPLOC divergence across *every* registered node.
3. Storage is the wrong shape: `record_audit_outcome` (`packages/contracts/meshnet_contracts/__init__.py:416`) persists only `strike_count`/`banned`/`passed` to `registry_events` — no divergence value, no GPU/dtype/hardware-profile column anywhere. Benchmark results otherwise land in a flat JSON file (`server.benchmark_results_path`), not a queryable per-node/hardware schema.

## Deliverables

- [x] Extend the TOPLOC verify call path (`audit.py`) to return the raw distance/divergence metric alongside the existing bool — `verify_activation_proofs_detailed()` / `ToplocVerificationResult` in `packages/validator/meshnet_validator/audit.py`; `verify_activation_proofs()` kept as a thin bool-only wrapper for existing callers. Also fixes a real bug this issue's code-read surfaced: the old code did `bool(_call_toploc(...))`, which is always `True` for the real `toploc` library's non-empty per-chunk `VerificationResult` list regardless of divergence — `tests/test_toploc_audit.py::test_verify_activation_proofs_detailed_aggregates_per_chunk_divergence` exercises this directly.
- [x] Extend the existing fleet-dispatch pattern (`server.py:2998+`) from pinned-route benchmarking to a tracker-scheduled job that hits **every currently registered node** with a fixed prompt/model/seed — `POST /v1/calibration/toploc/run` (admin/validator-gated, same shape as `POST /v1/benchmark/hop-penalty`) in `packages/tracker/meshnet_tracker/server.py`. Dispatches to every node that can solo-serve the full model range (single-hop pinned route, isolating one node's hardware noise from route-composition effects); partial-shard nodes are reported under `skipped_partial_shard_node_ids`, and nodes that don't answer the on-demand TOPLOC commitment fetch are reported per-node under `"skipped": "..."` rather than counted as pass or fail. See `tests/test_toploc_calibration_dispatch.py`.
- [x] Add a small SQLite table (same pattern as `billing.py`/`accounts.py`) keyed by node wallet + GPU model + dtype, storing the divergence value per calibration run — `packages/tracker/meshnet_tracker/calibration.py::ToplocCalibrationStore`, `toploc_calibration_runs` table.
- [x] Aggregation: p99 honest envelope + safety margin computed from that table, written as the recommended tolerance constants — `ToplocCalibrationStore.envelope()`, exposed via `GET /v1/calibration/toploc/results`.
- [x] Gate checklist: production audit enable blocked until corpus covers ≥N distinct hardware profiles — `ToplocCalibrationStore.gate_status(min_hardware_profiles=N)`; N is `--toploc-calibration-gate-min-hardware-profiles` (default 1) on the tracker CLI, documented alpha exception in the validator README.

## Acceptance criteria

- [ ] Corpus collected from the current hired-VPS fleet (documented as a small-fleet alpha corpus, not the eventual volunteer-fleet corpus) — **not done: needs a human to run `POST /v1/calibration/toploc/run` against the live hired-VPS fleet before launch; no such fleet exists in a dev session.**
- [ ] Threshold constants in validator config derived from corpus, not guessed — mechanically ready (`envelope()` returns them) but depends on the real corpus above; not yet wired into `ToplocAuditConfig` as enforced thresholds (deliberately — enforcing unvalidated thresholds would be worse than today's guessed bool).
- [ ] False-positive rate estimate documented at chosen thresholds — `envelope()` returns `estimated_false_positive_rate` (in-sample: fraction of the recorded corpus the recommended thresholds would themselves flag); needs the real corpus to be a meaningful number, and should be written up in the runbook once collected.
- [x] README / runbook cross-link: **do not enable production audits** until this issue closes — `packages/validator/README.md` "TOPLOC audit contract" section, updated with the full operational how-to.
- [x] Note in the runbook that this alpha corpus must be re-run once the fleet grows beyond the hired-VPS set (different hardware mix invalidates the envelope) — same README section; [runbook 04](../runbooks/04-toploc-calibration-run.md).

## ADR links

- [ADR-0018](../../docs/adr/0018-fraud-detection-verification-and-reputation.md) — Consequences (honest-noise corpus)

## Blocked by

- `06-fraud-toploc-integration_completed.md` (TOPLOC wired; calibration uses same primitive) — done

## Blocks (prod gate)

- Alpha release to real-money friends+hired-VPS launch (raised from "production adaptive audit thresholds" to a hard alpha-release gate during 2026-07-06 grilling)
- Production enable of adaptive audit thresholds (issues 09–10 in prod)