neuron-tai/.scratch/alpha-hardening/issues/21-honest-noise-calibration-corpus.md

Status: ready-for-human

BLOCKS ALPHA RELEASE. Scoped 2026-07-06 during alpha-launch-readiness grilling session — must complete before real-money (mainnet SOL/USDT) traffic goes live for the friends + hired-VPS-host launch. Loose/uncalibrated thresholds + manual admin slash-reversal are the stopgap only until this closes.

Engineering complete 2026-07-06; blocked on a human running it against the real hired-VPS fleet before launch. The three code gaps below are closed and unit-tested (see Deliverables), but nothing in a dev session can stand in for actually dispatching the job at real hardware — that step, plus the threshold/FPR write-up that depends on its output, needs an operator with the live fleet. See the validator README's "Honest-noise calibration corpus" section for the operational how-to.

21 — Honest-noise TOPLOC calibration corpus

What to build

Before enabling production TOPLOC audit thresholds, collect an honest-noise baseline across the active fleet. Run identical inference jobs on every active node/GPU combo; measure the divergence envelope (TOPLOC exponent/mantissa deltas, logprob-rank spread) under real hardware variance. This must be driven by the tracker (scheduled/dispatched job), not a manual one-off script, so it can be re-run as the fleet's hardware mix changes.

Per ADR-0018 consequences: threshold calibration requires an honest-noise corpus across the fleet before production thresholds.

Research anchor: .scratch/alpha-hardening/research-verifiable-inference.md §8 layer 3 — "collect this first — run identical jobs across the current node fleet to measure the honest divergence envelope before setting thresholds."

Launch context (why this is buildable now, not a research project): first-launch nodes are hired VPS/VPC hosts under our own direct control (test infrastructure we pay for, not third-party volunteers) — not a long-term topology, but risk-free for calibration purposes since there's no external party to dispute a bad reading. Friends are client-side users of the API in this phase, not node operators. Run the calibration pass against this small, fully-controlled fleet first; hired hosts stay stake-free until it's done, then move to real staking once thresholds derive from their own hardware.

Current gap (confirmed 2026-07-06 by code read): none of the three pieces below exist yet.

1. verify_activation_proofs() (packages/validator/meshnet_validator/audit.py:94-127) returns a plain bool — no raw TOPLOC divergence/distance value is ever computed or surfaced. Every "done" fraud-detection issue (06–10) currently runs on a guessed threshold baked into that bool, not a calibrated one.
2. Fleet dispatch exists but is the wrong shape: _handle_benchmark_hop_penalty / _handle_benchmark_results (packages/tracker/meshnet_tracker/server.py:2998-3104, from the old US-030 latency work) targets pinned 1–3-node routes and measures latency, not TOPLOC divergence across every registered node.
3. Storage is the wrong shape: record_audit_outcome (packages/contracts/meshnet_contracts/__init__.py:416) persists only strike_count/banned/passed to registry_events — no divergence value, no GPU/dtype/hardware-profile column anywhere. Benchmark results otherwise land in a flat JSON file (server.benchmark_results_path), not a queryable per-node/hardware schema.

Deliverables

• Extend the TOPLOC verify call path (audit.py) to return the raw distance/divergence metric alongside the existing bool — verify_activation_proofs_detailed() / ToplocVerificationResult in packages/validator/meshnet_validator/audit.py; verify_activation_proofs() kept as a thin bool-only wrapper for existing callers. Also fixes a real bug this issue's code-read surfaced: the old code did bool(_call_toploc(...)), which is always True for the real toploc library's non-empty per-chunk VerificationResult list regardless of divergence — tests/test_toploc_audit.py::test_verify_activation_proofs_detailed_aggregates_per_chunk_divergence exercises this directly.
• Extend the existing fleet-dispatch pattern (server.py:2998+) from pinned-route benchmarking to a tracker-scheduled job that hits every currently registered node with a fixed prompt/model/seed — POST /v1/calibration/toploc/run (admin/validator-gated, same shape as POST /v1/benchmark/hop-penalty) in packages/tracker/meshnet_tracker/server.py. Dispatches to every node that can solo-serve the full model range (single-hop pinned route, isolating one node's hardware noise from route-composition effects); partial-shard nodes are reported under skipped_partial_shard_node_ids, and nodes that don't answer the on-demand TOPLOC commitment fetch are reported per-node under "skipped": "..." rather than counted as pass or fail. See tests/test_toploc_calibration_dispatch.py.
• Add a small SQLite table (same pattern as billing.py/accounts.py) keyed by node wallet + GPU model + dtype, storing the divergence value per calibration run — packages/tracker/meshnet_tracker/calibration.py::ToplocCalibrationStore, toploc_calibration_runs table.
• Aggregation: p99 honest envelope + safety margin computed from that table, written as the recommended tolerance constants — ToplocCalibrationStore.envelope(), exposed via GET /v1/calibration/toploc/results.
• Gate checklist: production audit enable blocked until corpus covers ≥N distinct hardware profiles — ToplocCalibrationStore.gate_status(min_hardware_profiles=N); N is --toploc-calibration-gate-min-hardware-profiles (default 1) on the tracker CLI, documented alpha exception in the validator README.

Acceptance criteria

• Corpus collected from the current hired-VPS fleet (documented as a small-fleet alpha corpus, not the eventual volunteer-fleet corpus) — not done: needs a human to run POST /v1/calibration/toploc/run against the live hired-VPS fleet before launch; no such fleet exists in a dev session.
• Threshold constants in validator config derived from corpus, not guessed — mechanically ready (envelope() returns them) but depends on the real corpus above; not yet wired into ToplocAuditConfig as enforced thresholds (deliberately — enforcing unvalidated thresholds would be worse than today's guessed bool).
• False-positive rate estimate documented at chosen thresholds — envelope() returns estimated_false_positive_rate (in-sample: fraction of the recorded corpus the recommended thresholds would themselves flag); needs the real corpus to be a meaningful number, and should be written up in the runbook once collected.
• README / runbook cross-link: do not enable production audits until this issue closes — packages/validator/README.md "TOPLOC audit contract" section, updated with the full operational how-to.
• Note in the runbook that this alpha corpus must be re-run once the fleet grows beyond the hired-VPS set (different hardware mix invalidates the envelope) — same README section.

ADR links

• ADR-0018 — Consequences (honest-noise corpus)

Blocked by

• 06-fraud-toploc-integration_completed.md (TOPLOC wired; calibration uses same primitive) — done

Blocks (prod gate)

• Alpha release to real-money friends+hired-VPS launch (raised from "production adaptive audit thresholds" to a hard alpha-release gate during 2026-07-06 grilling)
• Production enable of adaptive audit thresholds (issues 09–10 in prod)