feat: harden node placement and partial model loading
This commit is contained in:
Dobromir Popov
@@ -2,9 +2,9 @@
|
||||
|
||||
Pre-release alpha audit + grilling (2026-07-04). Bucket 1 trust-boundary blockers + fraud arc: **done** (16/22 original issues). Bucket 2 (12-15, multi-tracker) and 17 (doc dedup) remain deferred/human-gated — not launch blockers.
|
||||
|
||||
**Launch-readiness grilling (2026-07-06):** locked plan is devnet dev/test run now, then real mainnet SOL/USDT for the first cohort — friends (API clients) + hired VPS/VPC hosts (own test infra, not third-party volunteers, stake-free). No new public token; TAI stays dormant per ADR-0002's existing volume/legal gates. Two new issues came out of this session:
|
||||
**Launch-readiness grilling (2026-07-06):** locked plan is devnet dev/test run now, then real mainnet USDT for the first cohort — friends (API clients) + hired VPS/VPC hosts (own test infra, not third-party volunteers; no upfront stake, probation only). No new public token; TAI stays dormant per ADR-0002's existing volume/legal gates. Two new issues came out of this session:
|
||||
|
||||
- **[21 — Honest-noise calibration corpus](./issues/21-honest-noise-calibration-corpus.md)** — rescoped from "prod gate" to a hard **alpha-release blocker**. `Status: ready-for-human` — engineering (audit.py raw divergence, tracker dispatch endpoint, SQLite corpus, p99 envelope) done 2026-07-06; blocked on a human running the calibration job against the real hired-VPS fleet before launch.
|
||||
- **[21 — Honest-noise calibration corpus](./issues/21-honest-noise-calibration-corpus.md)** — rescoped from "prod gate" to a hard **alpha-release blocker**. `Status: ready-for-human` — engineering (audit.py raw divergence, tracker dispatch endpoint, SQLite corpus, p99 envelope) done 2026-07-06; blocked on a human running the calibration job against the real hired-VPS fleet before launch. Runbook: [04-toploc-calibration-run](./runbooks/04-toploc-calibration-run.md).
|
||||
- **[23 — Dynamic HF-benchmarked pricing](./issues/23-dynamic-hf-pricing_completed.md)** — new, high priority but not a release blocker. `Status: done` — engineering complete 2026-07-06 (hf_pricing.py, opt-in daily refresh loop, GET /v1/pricing/hf/history); real `hf_aliases` curation per model is a follow-up human sign-off, not a completion blocker.
|
||||
|
||||
Locked scope: one settlement tracker, open node join, devnet mock-USDT, reputation carries forward → fraud must be bounded. See [ADR-0016](../../docs/adr/0016-alpha-scope-and-known-limitations.md).
|
||||
@@ -77,6 +77,7 @@ Locked scope: one settlement tracker, open node join, devnet mock-USDT, reputati
|
||||
| [17 Duplicate US-020 dedup](./issues/17-doc-duplicate-us020-dedup.md) |
|
||||
| [18 Operational runbooks](./issues/18-doc-operational-runbooks_completed.md) |
|
||||
| [19 Cryptography + test env](./issues/19-doc-cryptography-test-env_completed.md) |
|
||||
| [04 TOPLOC calibration run](./runbooks/04-toploc-calibration-run.md) (issue 21 ops) |
|
||||
| [22 MEMORY + project-status index](./issues/22-doc-memory-project-status_completed.md) (done) |
|
||||
| [21 Honest-noise calibration corpus](./issues/21-honest-noise-calibration-corpus.md) (ops; prod gate for audits) |
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
Status: ready-for-human
|
||||
|
||||
**BLOCKS ALPHA RELEASE.** Scoped 2026-07-06 during alpha-launch-readiness grilling session — must complete before real-money (mainnet SOL/USDT) traffic goes live for the friends + hired-VPS-host launch. Loose/uncalibrated thresholds + manual admin slash-reversal are the stopgap only until this closes.
|
||||
**BLOCKS ALPHA RELEASE.** Scoped 2026-07-06 during alpha-launch-readiness grilling session — must complete before real-money mainnet USDT traffic goes live for the friends + hired-VPS-host launch. Loose/uncalibrated thresholds + manual admin slash-reversal are the stopgap only until this closes.
|
||||
|
||||
**Engineering complete 2026-07-06; blocked on a human running it against the real hired-VPS fleet before launch.** The three code gaps below are closed and unit-tested (see Deliverables), but nothing in a dev session can stand in for actually dispatching the job at real hardware — that step, plus the threshold/FPR write-up that depends on its output, needs an operator with the live fleet. See the validator README's "Honest-noise calibration corpus" section for the operational how-to.
|
||||
|
||||
@@ -14,7 +14,7 @@ Per [ADR-0018 consequences](../../docs/adr/0018-fraud-detection-verification-and
|
||||
|
||||
Research anchor: `.scratch/alpha-hardening/research-verifiable-inference.md` §8 layer 3 — "collect this first — run identical jobs across the current node fleet to measure the honest divergence envelope before setting thresholds."
|
||||
|
||||
**Launch context (why this is buildable now, not a research project):** first-launch nodes are hired VPS/VPC hosts under our own direct control (test infrastructure we pay for, not third-party volunteers) — not a long-term topology, but risk-free for calibration purposes since there's no external party to dispute a bad reading. Friends are client-side users of the API in this phase, not node operators. Run the calibration pass against this small, fully-controlled fleet first; hired hosts stay stake-free until it's done, then move to real staking once thresholds derive from their own hardware.
|
||||
**Launch context (why this is buildable now, not a research project):** first-launch nodes are hired VPS/VPC hosts under our own direct control (test infrastructure we pay for, not third-party volunteers) — not a long-term topology, but risk-free for calibration purposes since there's no external party to dispute a bad reading. Friends are client-side users of the API in this phase, not node operators. Run the calibration pass against this small, fully-controlled fleet first; hired hosts stay on probation (no upfront stake) until it's done, then move to paid USDT serving once thresholds derive from their own hardware.
|
||||
|
||||
**Current gap (historical — closed 2026-07-06):** the three engineering pieces below were missing when this issue was filed; all are now implemented and unit-tested. Remaining work is the human calibration run on the live hired-VPS fleet.
|
||||
|
||||
@@ -36,7 +36,7 @@ Research anchor: `.scratch/alpha-hardening/research-verifiable-inference.md` §8
|
||||
- [ ] Threshold constants in validator config derived from corpus, not guessed — mechanically ready (`envelope()` returns them) but depends on the real corpus above; not yet wired into `ToplocAuditConfig` as enforced thresholds (deliberately — enforcing unvalidated thresholds would be worse than today's guessed bool).
|
||||
- [ ] False-positive rate estimate documented at chosen thresholds — `envelope()` returns `estimated_false_positive_rate` (in-sample: fraction of the recorded corpus the recommended thresholds would themselves flag); needs the real corpus to be a meaningful number, and should be written up in the runbook once collected.
|
||||
- [x] README / runbook cross-link: **do not enable production audits** until this issue closes — `packages/validator/README.md` "TOPLOC audit contract" section, updated with the full operational how-to.
|
||||
- [x] Note in the runbook that this alpha corpus must be re-run once the fleet grows beyond the hired-VPS set (different hardware mix invalidates the envelope) — same README section.
|
||||
- [x] Note in the runbook that this alpha corpus must be re-run once the fleet grows beyond the hired-VPS set (different hardware mix invalidates the envelope) — same README section; [runbook 04](../runbooks/04-toploc-calibration-run.md).
|
||||
|
||||
## ADR links
|
||||
|
||||
|
||||
@@ -440,12 +440,12 @@
|
||||
"Run relevant pytest tests; run the full suite when practical or document why not"
|
||||
],
|
||||
"priority": 21,
|
||||
"passes": true,
|
||||
"notes": "Source issue: .scratch/alpha-hardening/issues/21-honest-noise-calibration-corpus.md. BLOCKS ALPHA RELEASE (real-money friends+hired-VPS launch) — rescoped 2026-07-06, no longer a Ralph-skip.",
|
||||
"passes": false,
|
||||
"notes": "Source issue: .scratch/alpha-hardening/issues/21-honest-noise-calibration-corpus.md. BLOCKS ALPHA RELEASE (real-money mainnet USDT). Operator runbook: .scratch/alpha-hardening/runbooks/04-toploc-calibration-run.md",
|
||||
"dependsOn": [
|
||||
"AH-006"
|
||||
],
|
||||
"completionNotes": "Engineering complete and unit-tested (validator audit.py detailed-verify aggregation, tracker calibration.py corpus store, calibration dispatch endpoints). Marked ready-for-human, not done: real corpus collection against the live hired-VPS fleet, and the threshold/FPR write-up that depends on its output, need a human operator — see .ralph-tui/progress.md and packages/validator/README.md."
|
||||
"completionNotes": "Engineering complete and unit-tested. Remaining: human runs POST /v1/calibration/toploc/run on live hired-VPS fleet, records envelope/FPR, wires thresholds — see runbook 04 and packages/validator/README.md."
|
||||
},
|
||||
{
|
||||
"id": "AH-022",
|
||||
|
||||
@@ -0,0 +1,70 @@
|
||||
# Runbook 04 — Honest-noise TOPLOC calibration (issue 21)
|
||||
|
||||
**Status:** engineering complete; **operator action required** before production audit thresholds.
|
||||
|
||||
**Blocks:** enabling calibrated TOPLOC thresholds on a mainnet / friends-test fleet (issue 21, ADR-0018).
|
||||
|
||||
## When to run
|
||||
|
||||
- Before first real-money traffic with audit enforcement enabled.
|
||||
- Again whenever the fleet’s **hardware mix** changes materially (new GPU generation, CPU-only nodes added, precision/recipe change per model).
|
||||
|
||||
Alpha exception: with a **small hired-VPS-only** fleet, `gate_status.ready` may mean “covers every node we operate today” (`--toploc-calibration-gate-min-hardware-profiles 1`).
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Tracker running with billing + registry + `--toploc-calibration-db PATH` (or default under tracker cwd).
|
||||
- At least one **solo-capable** node per hardware profile you want in the corpus (full model coverage — partial shards are skipped).
|
||||
- Admin or validator credentials (`Authorization` header or validator service token per ADR-0017).
|
||||
- Reference validator can replay the fixed calibration prompt (same model/seed as dispatch uses).
|
||||
|
||||
## Steps
|
||||
|
||||
1. **Register the fleet** — all nodes you intend to pay on mainnet should be up, admitted (NCA when enabled), and solo-serving the calibration model.
|
||||
|
||||
2. **Dispatch the job** (admin/validator only):
|
||||
|
||||
```bash
|
||||
curl -X POST "https://<tracker>/v1/calibration/toploc/run" \
|
||||
-H "Authorization: Bearer <admin-or-validator-token>" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{}'
|
||||
```
|
||||
|
||||
Partial-shard nodes appear under `skipped_partial_shard_node_ids`. Per-node failures appear under `skipped` with reasons.
|
||||
|
||||
3. **Wait for completion** — watch tracker logs and node consoles until every solo-capable node has a row in the corpus.
|
||||
|
||||
4. **Fetch results**:
|
||||
|
||||
```bash
|
||||
curl "https://<tracker>/v1/calibration/toploc/results" \
|
||||
-H "Authorization: Bearer <admin-or-validator-token>"
|
||||
```
|
||||
|
||||
Record:
|
||||
- `envelope` — p99 metrics + 20% safety margin (recommended tolerances).
|
||||
- `gate_status.ready` and `gate_status.hardware_profiles`.
|
||||
- `estimated_false_positive_rate` (in-sample sanity check only).
|
||||
|
||||
5. **Write up thresholds** — paste envelope values into operator notes / issue 21 comment. Do **not** wire into production `ToplocAuditConfig` until you have reviewed FPR on this fleet.
|
||||
|
||||
6. **Mark issue 21 done** — when corpus covers the launch fleet and thresholds are documented.
|
||||
|
||||
## Two-wallet / minimal pilot variant
|
||||
|
||||
If your “fleet” is one node machine + one client:
|
||||
|
||||
- Run calibration against the **node** profile only (one hardware row is enough for `gate_status` with min profiles = 1).
|
||||
- Client wallet is irrelevant to calibration — it never serves inference.
|
||||
|
||||
## Do not
|
||||
|
||||
- Enable stricter production audit thresholds before this completes.
|
||||
- Reuse a corpus collected on devnet/mock hardware for a different mainnet GPU mix without re-running.
|
||||
|
||||
## References
|
||||
|
||||
- Issue: `.scratch/alpha-hardening/issues/21-honest-noise-calibration-corpus.md`
|
||||
- Code: `packages/tracker/meshnet_tracker/calibration.py`, `POST /v1/calibration/toploc/run`
|
||||
- Validator: `packages/validator/README.md` — TOPLOC audit contract
|
||||
Reference in New Issue
Block a user