68e057209c
Lock alpha scope, tracker auth, TOPLOC fraud verification, and deferred multi-tracker money-path work; supersede legacy fraud issues with ADR-0018. Co-authored-by: Cursor <cursoragent@cursor.com>
1.9 KiB
1.9 KiB
Status: ready-for-human
21 — Honest-noise TOPLOC calibration corpus
What to build
Before enabling production TOPLOC audit thresholds, collect an honest-noise baseline across the heterogeneous volunteer fleet. Run identical inference jobs on every active node/GPU combo; measure the divergence envelope (TOPLOC exponent/mantissa deltas, logprob-rank spread) under real hardware variance.
Per ADR-0018 consequences: threshold calibration requires an honest-noise corpus across the fleet before production thresholds.
Research anchor: .scratch/alpha-hardening/research-verifiable-inference.md §8 layer 3 — "collect this first — run identical jobs across the current node fleet to measure the honest divergence envelope before setting thresholds."
Deliverables
- Scripted benchmark job (fixed prompt, model preset, seed policy) runnable on all nodes
- Aggregated corpus artifact (per node: GPU model, dtype, TOPLOC deltas vs reference)
- Recommended tolerance thresholds documented (p99 honest envelope + safety margin)
- Gate checklist: production audit enable blocked until corpus covers ≥N distinct hardware profiles (define N in runbook, suggest ≥3)
Acceptance criteria
- Corpus collected from current fleet (or documented subset + extrapolation note)
- Threshold constants in validator config derived from corpus, not guessed
- False-positive rate estimate documented at chosen thresholds
- README / runbook cross-link: do not enable production audits until this issue closes
ADR links
- ADR-0018 — Consequences (honest-noise corpus)
Blocked by
06-fraud-toploc-integration.md(TOPLOC wired; calibration uses same primitive)
Blocks (prod gate)
- Production enable of adaptive audit thresholds (issues 09–10 in prod)