Files
neuron-tai/.scratch/alpha-hardening/README.md
Dobromir Popov daddbaa4a3 distributd cache
2026-07-08 22:53:03 +02:00

7.5 KiB
Raw Blame History

Alpha hardening — planning index

Pre-release alpha audit + grilling (2026-07-04). Bucket 1 trust-boundary blockers + fraud arc: done (16/22 original issues). Bucket 2 (12-15, multi-tracker) and 17 (doc dedup) remain deferred/human-gated — not launch blockers.

Launch-readiness grilling (2026-07-06): locked plan is devnet dev/test run now, then real mainnet SOL/USDT for the first cohort — friends (API clients) + hired VPS/VPC hosts (own test infra, not third-party volunteers, stake-free). No new public token; TAI stays dormant per ADR-0002's existing volume/legal gates. Two new issues came out of this session:

  • 21 — Honest-noise calibration corpus — rescoped from "prod gate" to a hard alpha-release blocker. Status: ready-for-human — engineering (audit.py raw divergence, tracker dispatch endpoint, SQLite corpus, p99 envelope) done 2026-07-06; blocked on a human running the calibration job against the real hired-VPS fleet before launch.
  • 23 — Dynamic HF-benchmarked pricing — new, high priority but not a release blocker. Status: done — engineering complete 2026-07-06 (hf_pricing.py, opt-in daily refresh loop, GET /v1/pricing/hf/history); real hf_aliases curation per model is a follow-up human sign-off, not a completion blocker.

Locked scope: one settlement tracker, open node join, devnet mock-USDT, reputation carries forward → fraud must be bounded. See ADR-0016.

Resume task (2026-07-07): 24 - Routing telemetry resume is ready-for-agent. Learned-routing commit 518c259 is already present; dirty tree contains current-request heartbeat/dashboard telemetry and a known import-time annotation crash in server.py:1490.

Perf follow-up (2026-07-08): 25 — Sharded per-node KV cache for distributed generation is implemented (ADR-0022): per-generation session ids, prefill/decode wire protocol (X-Meshnet-Cache/X-Meshnet-Past-Len), per-node sharded DynamicCache(config=…) (hybrid-attention-aware), TTL+LRU eviction with 409 cache-miss → full re-prefill fallback. Golden test proves token-identical output vs the stateless path; CPU two-shard measurement: 7.05 tps decaying 32% → 18.93 tps flat (2.68×). Remaining: re-measure on the live 2-node GPU topology and the Qwen3.6-35B-A3B mixed topology.

Artifacts

Path Status
research-verifiable-inference.md Complete — SOTA research, §8 layered scheme, TOPLOC adopt
handoff.md Session handoff — locked decisions, env notes
docs/adr/00160019 Alpha scope, auth, fraud, multi-tracker design
issues/ 25 work items (Buckets 13 + perf follow-ups)

ADRs (this feature)

ADR Title
0016 Alpha scope & known limitations
0017 Tracker authentication & authorization
0018 Fraud detection, verification & reputation (flagship)
0019 Money-path consistency — design accepted, impl deferred
0002 Amended — settlement superseded by 0015
0010 Amended — TLS alpha reality (relay only)

Implement Bucket 1 first. Fraud arc depends on auth foundation (02 + 20), hive gossip enforcement (01), and persistence (05).

Phase 1 — Trust boundary (alpha blockers)

Order Issue ID Depends on
1 Unified auth boundary + Validator service token A2, —
2 C1 hive gossip auth enforcement C1 02
3 Persist strike/ban/reputation A1/A5 02
4 Starting credit 0 + spend cap C5, M1 02
5 Tracker-authoritative accounting H2 02
6 Wallet binding proof C6 02, 03

Phase 2 — Fraud arc (after Phase 1)

Order Issue Depends on
6 TOPLOC integration 05
7 Commitment + bisection blame 06
8 Reputation model 05, 07
9 Routing + adaptive audit 08
10 Penalty calibration wiring 07, 08, 02

Prod gate: 21 honest-noise calibration corpus must complete before enabling production TOPLOC audit thresholds (issues 0910 in prod). Dev/staging TOPLOC wiring (0608) may proceed in parallel.

Phase 3 — Bucket 2 (post-alpha, design tracked)

Issue ADR
12 C2 on-chain idempotency 0019 §1
13 C3/C4 consensus-gated settlement 0019 §2
14 A3 durable Raft term/vote 0019 §3
15 H1 commutative forfeit 0019 §4

Phase 4 — Doc hygiene (parallel anytime)

Issue
16 US-006 + fraud issue reconciliation
17 Duplicate US-020 dedup
18 Operational runbooks
19 Cryptography + test env
22 MEMORY + project-status index (done)
21 Honest-noise calibration corpus (ops; prod gate for audits)

Phase 5 — Distributed-inference performance (post-routing-fix)

Issue Depends on
25 Sharded per-node KV cache ADR-0020 routing fix (done), 24 routing telemetry resume

First 3 to implement

  1. 02 + 20 — Unified auth boundary + validator service token (shared helper and roles)
  2. 01 — Apply hive auth to billing/accounts/stats gossip endpoints
  3. 05 — Persist strike/ban/reputation (penalties must survive restart)

Research anchor

Fraud design cites .scratch/alpha-hardening/research-verifiable-inference.md:

  • ADOPT TOPLOC (§89, build-vs-adopt table)
  • On-demand commitments, not every request (§8 layer 1 footnote; ADR-0018 §3)
  • 5% audit budget as target, not cap (§1.1, §6)
  • 19× deterrence via full pending forfeiture (§1.1)
  • Hop bisection blame pattern (§1.2, §8 layer 3)
  • Honest-noise corpus before prod thresholds (§8 layer 3; issue 21)
  • Roadmap-only: zkML, TEE, Gensyn RepOps (§9)

Comments