68e057209c
Lock alpha scope, tracker auth, TOPLOC fraud verification, and deferred multi-tracker money-path work; supersede legacy fraud issues with ADR-0018. Co-authored-by: Cursor <cursoragent@cursor.com>
38 lines
1.9 KiB
Markdown
38 lines
1.9 KiB
Markdown
Status: ready-for-human
|
||
|
||
# 21 — Honest-noise TOPLOC calibration corpus
|
||
|
||
## What to build
|
||
|
||
Before enabling production TOPLOC audit thresholds, collect an **honest-noise baseline** across the heterogeneous volunteer fleet. Run identical inference jobs on every active node/GPU combo; measure the divergence envelope (TOPLOC exponent/mantissa deltas, logprob-rank spread) under real hardware variance.
|
||
|
||
Per [ADR-0018 consequences](../../docs/adr/0018-fraud-detection-verification-and-reputation.md): threshold calibration requires an honest-noise corpus across the fleet before production thresholds.
|
||
|
||
Research anchor: `.scratch/alpha-hardening/research-verifiable-inference.md` §8 layer 3 — "collect this first — run identical jobs across the current node fleet to measure the honest divergence envelope before setting thresholds."
|
||
|
||
## Deliverables
|
||
|
||
- [ ] Scripted benchmark job (fixed prompt, model preset, seed policy) runnable on all nodes
|
||
- [ ] Aggregated corpus artifact (per node: GPU model, dtype, TOPLOC deltas vs reference)
|
||
- [ ] Recommended tolerance thresholds documented (p99 honest envelope + safety margin)
|
||
- [ ] Gate checklist: production audit enable blocked until corpus covers ≥N distinct hardware profiles (define N in runbook, suggest ≥3)
|
||
|
||
## Acceptance criteria
|
||
|
||
- [ ] Corpus collected from current fleet (or documented subset + extrapolation note)
|
||
- [ ] Threshold constants in validator config derived from corpus, not guessed
|
||
- [ ] False-positive rate estimate documented at chosen thresholds
|
||
- [ ] README / runbook cross-link: **do not enable production audits** until this issue closes
|
||
|
||
## ADR links
|
||
|
||
- [ADR-0018](../../docs/adr/0018-fraud-detection-verification-and-reputation.md) — Consequences (honest-noise corpus)
|
||
|
||
## Blocked by
|
||
|
||
- `06-fraud-toploc-integration.md` (TOPLOC wired; calibration uses same primitive)
|
||
|
||
## Blocks (prod gate)
|
||
|
||
- Production enable of adaptive audit thresholds (issues 09–10 in prod)
|