popov/neuron-tai
1
0
Fork 0
Code Issues 42 Pull Requests Actions Packages Projects Releases Wiki Activity

chore: archive historical task programs

Browse Source
This commit is contained in:
Dobromir Popov
2026-07-17 12:41:46 +03:00
parent 9b257d9a1b
commit a35d86f343
65 changed files with 15 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

41
docs/archived/alpha-hardening/issues/01-c1-gossip-auth_completed.md Normal file
View File

@@ -0,0 +1,41 @@
Status: done
# 01 — C1: Authenticate hive gossip endpoints
## What to build
Add authenticated peer identity to all tracker gossip mutation endpoints. Today any caller can push billing, account, and stats events without verification.
**Code refs:**
- `packages/tracker/meshnet_tracker/server.py` — `_handle_billing_gossip` (~2414–2427)
- `packages/tracker/meshnet_tracker/server.py` — `_handle_accounts_gossip` (~2610–2623)
- `packages/tracker/meshnet_tracker/server.py` — `_handle_stats_gossip` (~2355–2364)
- `packages/tracker/meshnet_tracker/billing.py` — `apply_events` (~301–311)
- `packages/tracker/meshnet_tracker/accounts.py` — `apply_events` (~220–226)
Implement per ADR-0017 §3 using the auth helper/config from issue 02: shared hive HMAC (body + timestamp) or mutual TLS between configured tracker peers. Reject unauthenticated gossip with 401.
**Note:** `/v1/gossip` (node throughput fan-out, `server.py` ~1331) is **not** in scope for this issue — see ADR-0017 §3 out-of-scope note.
## Test-first
1. Red: unauthenticated POST to `/v1/billing/gossip` applies a credit event today — test must fail after fix.
2. Red: authenticated peer with valid HMAC applies events; invalid/missing auth returns 401 and `applied: 0`.
3. Green: wire the issue-02 verifier/config (`--hive-secret` or peer cert paths) into the three hive mutation endpoints.
## Acceptance criteria
- [ ] `/v1/billing/gossip`, `/v1/accounts/gossip`, `/v1/stats/gossip` reject requests without valid hive auth
- [ ] Authenticated peers replicate events as today (id-dedup preserved)
- [ ] Config documented for multi-tracker dev setups
- [ ] Tests cover reject + accept paths without live network
## ADR links
- [ADR-0017](../../docs/adr/0017-tracker-authentication-and-authorization.md)
- [ADR-0016](../../docs/adr/0016-alpha-scope-and-known-limitations.md)
## Blocked by
- `02-a2-unified-auth-boundary_completed.md` — owns shared auth middleware/config. Implement in the same PR if simpler.

46
docs/archived/alpha-hardening/issues/02-a2-unified-auth-boundary_completed.md Normal file
View File

@@ -0,0 +1,46 @@
Status: done
# 02 — A2: Unified auth boundary for privileged and financial reads
## What to build
Replace header-presence stubs with a single auth middleware that resolves API keys, admin sessions, validator service tokens, and hive peer identity. Close leaks on financial and operator endpoints. This is the auth foundation issue; issue 01 should only apply hive auth to gossip endpoints once the helper exists.
**Code refs:**
- `packages/tracker/meshnet_tracker/server.py` — `_handle_billing_forfeit` (~2429–2464) — H3: non-empty `Authorization` only
- `packages/tracker/meshnet_tracker/server.py` — `_handle_benchmark_hop_penalty` (~2650–2658), `_handle_benchmark_results` (~2745–2748) — H3
- `packages/tracker/meshnet_tracker/server.py` — `_handle_billing_summary` (~2366–2371) — H4
- `packages/tracker/meshnet_tracker/server.py` — `_handle_billing_settlements` (~2407–2412) — H4
- `packages/tracker/meshnet_tracker/server.py` — `_handle_registry_wallets` (~2391–2405) — H4
- `packages/tracker/meshnet_tracker/server.py` — `_session_account` (~2468+), `_handle_admin_accounts` (~2588–2608) — H4
- `packages/tracker/meshnet_tracker/accounts.py` — `session_account()`, `create_session()` only (session store; not handler wiring)
Per ADR-0017 §4: forfeit → validator or admin; benchmark → admin; billing summary/settlements/registry wallets → admin session. Include the validator service token shape from `20-validator-service-token_completed.md` in the same implementation if practical.
## Test-first
1. Red: POST `/v1/billing/forfeit` with `Authorization: Bearer garbage` succeeds today — must require validator/admin identity.
2. Red: GET `/v1/billing/summary` without admin session returns 401/403.
3. Green: middleware + role checks; existing inference API-key path unchanged.
## Acceptance criteria
- [ ] Single `_require_auth(role=...)` (or equivalent) used by all privileged handlers
- [ ] Shared auth config supports admin sessions, validator service token, and hive peer HMAC/mTLS
- [ ] Forfeit accepts only validator service token or admin session — not arbitrary Bearer strings
- [ ] Financial read endpoints require admin session (alpha posture)
- [ ] Benchmark write/read require admin or service token
- [ ] Integration tests for each endpoint class (reject unauth, accept valid)
## ADR links
- [ADR-0017](../../docs/adr/0017-tracker-authentication-and-authorization.md)
## Related
- `20-validator-service-token_completed.md` — checklist for validator service token format, rotation, forfeit auth
## Blocked by
None. This issue should land before `01-c1-gossip-auth_completed.md`.

38
docs/archived/alpha-hardening/issues/03-c5-starting-credit-zero_completed.md Normal file
View File

@@ -0,0 +1,38 @@
Status: done
# 03 — C5 + M1: Starting credit 0, funded-account gate, spend cap
## What to build
Close the free-credit faucet. New API keys start at **0 USDT**; inference requires a real deposit or admin credit. Add a configurable per-request spend cap (M1) to limit runaway charges on compromised keys.
**Code refs:**
- `packages/tracker/meshnet_tracker/billing.py` — `DEFAULT_STARTING_CREDIT = 1.0` (~22), `ensure_client` (~73–85), `has_funds` (~87–88), duplicate credit on charge (~130–138)
- `packages/tracker/meshnet_tracker/server.py` — billing gate before routing (~1667–1690)
Per ADR-0017 §2 and ADR-0016 §3.
## Test-first
1. Red: new API key gets 1.0 USDT implicit credit — test expects 0 balance until deposit.
2. Red: first inference without deposit returns 402.
3. Green: `DEFAULT_STARTING_CREDIT = 0.0`; optional `--max-charge-per-request` config.
## Acceptance criteria
- [ ] `DEFAULT_STARTING_CREDIT` is 0.0; no automatic caller credit on first touch
- [ ] `has_funds` false for fresh keys; 402 before routing (server.py ~1684)
- [ ] Admin `credit_client` or bound-wallet deposit still funds accounts
- [ ] Configurable max charge per request (M1) rejects oversize completions with clear error
- [ ] Tests: fresh key blocked; after credit/deposit, inference proceeds
## ADR links
- [ADR-0017](../../docs/adr/0017-tracker-authentication-and-authorization.md)
- [ADR-0016](../../docs/adr/0016-alpha-scope-and-known-limitations.md)
- [ADR-0015](../../docs/adr/0015-usdt-custodial-settlement.md)
## Blocked by
- `02-a2-unified-auth-boundary_completed.md` (admin credit path secured)

38
docs/archived/alpha-hardening/issues/04-h2-tracker-authoritative-accounting_completed.md Normal file
View File

@@ -0,0 +1,38 @@
Status: done
# 04 — H2: Tracker-authoritative token and work-unit accounting
## What to build
Stop trusting node-reported usage for billing. The tracker already proxies responses — use tracker-observed response data and request limits to cap billable tokens, and compute work units from the **route it constructed**, not node declarations.
**Code refs:**
- `packages/tracker/meshnet_tracker/server.py` — `node_work` from route construction (~1776–1782, ~1781–1782)
- `packages/tracker/meshnet_tracker/server.py` — streaming token/chunk billing (~1890–1921)
- `packages/tracker/meshnet_tracker/server.py` — non-streaming `_usage_total_tokens` (~1938–1943)
- `packages/tracker/meshnet_tracker/billing.py` — `charge_request` node_work split (~104–151)
Accounting fraud = inflating tokens or shard span. Per ADR-0018 §5.
## Test-first
1. Red: mock upstream returns inflated `usage.total_tokens` in body but tracker bills that value — test expects the tracker to cap billable tokens from observed stream chunks or request bounds.
2. Red: node registers false `shard_end`; billing uses tracker route span, not registration field alone.
3. Green: authoritative counters; ignore node-reported work units on charge path.
## Acceptance criteria
- [ ] Streaming token count uses tracker-observed chunks/tokens; upstream `usage.total_tokens` can only lower or match that observed count, never inflate it
- [ ] Non-streaming token count caps upstream `usage.total_tokens` by tracker-known request bounds (`max_tokens`, and prompt estimate if available); exact tokenizer-backed counts are deferred unless already available locally
- [ ] Work units = tracker-computed layer span per hop at route build time (~1781–1782)
- [ ] Nodes cannot increase payout by lying about shard range mid-request
- [ ] Integration test: malicious node metadata does not inflate `charge_request` shares
## ADR links
- [ADR-0018](../../docs/adr/0018-fraud-detection-verification-and-reputation.md) §5
## Blocked by
- `02-a2-unified-auth-boundary_completed.md`

40
docs/archived/alpha-hardening/issues/05-a1-a5-persist-strike-ban-reputation_completed.md Normal file
View File

@@ -0,0 +1,40 @@
Status: done
# 05 — A1/A5: Persist strike, ban, and reputation state
## What to build
Registry strike/ban/reputation state today lives in RAM-only `_LocalContractState` — tracker restart wipes penalties. Persist to SQLite (same pattern as `BillingLedger` and `AccountStore`) so reputation carries forward per ADR-0016 §4.
**Code refs:**
- `packages/contracts/meshnet_contracts/__init__.py` — `RegistryContract`, `RegistryWallet`, in-memory `_state.registry` (~103–206)
- `packages/tracker/meshnet_tracker/billing.py` — SQLite persistence pattern (~60, event log)
- `packages/tracker/meshnet_tracker/accounts.py` — SQLite + event replication (~40–56)
Include fields for: `strike_count`, `banned`, `completed_job_count`, graduated **reputation score** (float, default 1.0), `last_audit_ts`, probation tracking.
**Scope split:** this issue owns **schema + persistence + load/reload** only. Reputation **scoring deltas** (audit pass/fail adjustments, decay rules) belong in issue 08.
## Test-first
1. Red: record strike, restart tracker process, strike count is 0 — must fail.
2. Green: persist + reload; gossip replicates strike events if multi-tracker.
3. Red: banned wallet registers node — must reject (wire to routing).
## Acceptance criteria
- [ ] Strike/ban/reputation survive tracker restart (SQLite or equivalent)
- [ ] `RegistryContract.list_wallets` reflects persisted state
- [ ] Banned wallet rejected at registration and excluded from routes
- [ ] Reputation score field present for routing/audit issues (08–09)
- [ ] Event-sourced mutations compatible with future Raft (ADR-0019)
## ADR links
- [ADR-0016](../../docs/adr/0016-alpha-scope-and-known-limitations.md) §4
- [ADR-0018](../../docs/adr/0018-fraud-detection-verification-and-reputation.md) §6
## Blocked by
- `02-a2-unified-auth-boundary_completed.md`

47
docs/archived/alpha-hardening/issues/06-fraud-toploc-integration_completed.md Normal file
View File

@@ -0,0 +1,47 @@
Status: done
# 06 — FRAUD: TOPLOC integration (teacher-forced audit primitive)
## What to build
Adopt [TOPLOC](https://github.com/PrimeIntellect-ai/toploc) (MIT, `pip install toploc`) for activation fingerprint commit and verify. Replace string-equality validator checks with teacher-forced prefill + TOPLOC tolerance matching.
**Estimated effort:** 2+ sessions. First landing should be the validator-only TOPLOC primitive and docs; node runtime commitments/on-demand capture can follow in issue 07 if this grows.
| Subtask | Owner package | Deliverable |
|---|---|---|
| Validator audit primitive | `packages/validator/` | Teacher-forced prefill, TOPLOC verify, unit tests with stub tensors |
| Node runtime commitments | `packages/node/` (if prover-side) | On-demand activation fingerprint generation on audit-selected requests; move to issue 07 if it blocks the validator primitive |
**Code refs:**
- `packages/validator/meshnet_validator/__init__.py` — `_run_reference`, `_outputs_match` (~92–148)
- `packages/validator/README.md` — deterrence math (update for 19× at p=0.05)
- Research: `.scratch/alpha-hardening/research-verifiable-inference.md` §8 layers 1–2, build-vs-adopt table
Pin one canonical precision/quantization per model preset. Add `toploc` to validator (and node if prover-side) dependencies.
## Test-first
1. Red: validator compares final text strings — fails on cross-GPU honest divergence (document expected).
2. Green: stub activation tensors + TOPLOC proofs round-trip in unit test.
3. Integration: reference node teacher-forces tokens; verify accepts honest proof, rejects swapped precision.
## Acceptance criteria
- [x] `toploc` dependency declared; `build_proofs_*` / `verify_proofs_*` wired
- [x] Validator re-runs claimed token sequence as prefill, not free generation
- [x] Model preset documents canonical dtype/quantization
- [x] README updated: 19× deterrence at 5% audit (research §1.1)
- [x] Tests with deterministic stub tensors (no GPU required in CI)
## ADR links
- [ADR-0018](../../docs/adr/0018-fraud-detection-verification-and-reputation.md) §2
- Research: [research-verifiable-inference.md](../research-verifiable-inference.md) §8, §9 build-vs-adopt
## Blocked by
- `05-a1-a5-persist-strike-ban-reputation_completed.md`
**Prod gate:** do not enable production audit thresholds until `21-honest-noise-calibration-corpus.md` completes (see README Phase 2 note).

35
docs/archived/alpha-hardening/issues/07-fraud-commitment-bisection-blame_completed.md Normal file
View File

@@ -0,0 +1,35 @@
Status: done
# 07 — FRAUD: On-demand commitment + hop bisection blame
## What to build
On audit selection, require nodes to supply TOPLOC-style fingerprints of **output boundary activations** per hop (on-demand, brief retention). On verify failure, referee identifies the **first divergent hop** — not always the last text node.
**Code refs:**
- `packages/validator/meshnet_validator/__init__.py` — `_slash_route`, `_final_text_node` bug (~102–140) — blames `max(shard_end)` only
- `packages/tracker/meshnet_tracker/server.py` — route hop construction (~1774–1783) — cut-points for bisection
- Research: `.scratch/alpha-hardening/research-verifiable-inference.md` §1.2, §8 layer 3 (Verde **pattern**, not on-chain game)
## Test-first
1. Red: two-hop route, corrupt hop-0 activations — `_final_text_node` blames hop-1 — test must fail.
2. Green: bisection selects hop-0; forfeit targets hop-0 wallet.
3. On-demand: commitment requested only when audit flag set on proxied request.
## Acceptance criteria
- [x] Audit requests carry tracker RNG/VRF flag indistinguishable from normal traffic (research §6) — the existing post-hoc `sample_rate` RNG gate in `ValidatorProcess.validate_once` already decides audit selection after the original proxied request completed, so the request the client/nodes saw is unaffected either way; locked in by `test_hop_commitments_are_not_requested_unless_the_event_is_audit_selected`
- [x] Nodes retain recent boundary activations for on-demand commit window (configurable TTL) — `ToplocAuditConfig.commitment_ttl_seconds`; expired commitments fall back to the text-only path (`test_expired_commitment_window_falls_back_to_text_only_audit`)
- [x] Validator/tracker compares fingerprints at each hop cut-point; first mismatch = culprit — `_hop_commitments_from_event` + `_first_divergent_hop` in `packages/validator/meshnet_validator/__init__.py`
- [x] `_final_text_node` removed or limited to text-only fallback — only called from the plain-text divergence branch of `_validate_event` now
- [x] Integration test: multi-hop pipeline, fault injected at known hop — `tests/test_hop_bisection.py`
## ADR links
- [ADR-0018](../../docs/adr/0018-fraud-detection-verification-and-reputation.md) §3–4
## Blocked by
- `06-fraud-toploc-integration_completed.md`

39
docs/archived/alpha-hardening/issues/08-fraud-reputation-model-persistence_completed.md Normal file
View File

@@ -0,0 +1,39 @@
Status: done
# 08 — FRAUD: Reputation model + persistence
## What to build
Implement graduated reputation per ADR-0018 §6: score derives only from tracker audit outcomes + uptime/latency. Slow build, instant loss, inactivity decay. ×0.8 routing multiplier per strike (not whole penalty — forfeiture stays full pending).
**Scope split:** issue 05 owns **schema + SQLite persistence**; this issue owns **scoring rules** (deltas, decay, strike→multiplier wiring) on top of persisted fields.
**Code refs:**
- `packages/contracts/meshnet_contracts/__init__.py` — extend `RegistryWallet` / persistence from issue 05
- `packages/validator/meshnet_validator/__init__.py` — `_slash_route` forfeiture path (~125–133)
- `packages/tracker/meshnet_tracker/billing.py` — `forfeit_pending` (~280–292)
- Research: `.scratch/alpha-hardening/research-verifiable-inference.md` §6
## Test-first
1. Red: persisted reputation/strike fields from issue 05 are ignored by scoring/routing today.
2. Green: clean audit +0.05 (tunable); failed audit −0.3 and strike; three strikes → ban persisted via issue-05 fields.
3. Inactivity decay after N days without completed jobs.
## Acceptance criteria
- [ ] Uses `reputation_score` and strike/ban fields persisted by issue 05; does not introduce a second schema path
- [ ] Audit pass/fail updates score with documented deltas
- [ ] Strike applies ×0.8 multiplier to routing weight (separate from forfeiture amount)
- [ ] Ban at 3 strikes; probation job count still enforced
- [ ] No peer-to-peer reputation inputs
## ADR links
- [ADR-0018](../../docs/adr/0018-fraud-detection-verification-and-reputation.md) §6
## Blocked by
- `05-a1-a5-persist-strike-ban-reputation_completed.md`
- `07-fraud-commitment-bisection-blame_completed.md` (audit outcomes feed reputation)

38
docs/archived/alpha-hardening/issues/09-fraud-reputation-routing-adaptive-audit_completed.md Normal file
View File

@@ -0,0 +1,38 @@
Status: done
# 09 — FRAUD: Reputation-weighted routing + adaptive audit rate
## What to build
Wire reputation into route selection and audit sampling. Default network audit budget ≈5% — **not a cap**. New/low-reputation nodes: 20–30% audit rate; veterans: 2–3% floor ≥2%. Tripwires escalate rate without direct punishment.
**Code refs:**
- `packages/tracker/meshnet_tracker/server.py` — route selection `_select_route`, `_effective_throughput` (~1747, routing helpers)
- `packages/validator/meshnet_validator/__init__.py` — `sample_rate=0.05`
- Research: `.scratch/alpha-hardening/research-verifiable-inference.md` §1.1, §6, §8 layers 2–4
Audit selection must be unpredictable at request time (tracker RNG after commitment window opens).
## Test-first
1. Red: uniform 5% sample regardless of reputation — test expects higher rate for low-reputation wallet.
2. Green: budget balancer keeps fleet-wide average ≈ configured target.
3. Routing prefers higher reputation among equal throughput candidates.
## Acceptance criteria
- [ ] Per-wallet audit probability function of reputation (newcomer high, veteran low, floor ≥2%)
- [ ] Fleet-wide audit budget configurable (~5% default target); over ≥1000 requests with fixed seed, measured fleet audit rate within **±1.0 percentage point** of configured target (e.g. 4.0–6.0% at 5% default)
- [ ] Route scoring includes reputation multiplier (earnings scale with tenure)
- [ ] Passive tripwire flags (perplexity/repetition) bump audit rate only
- [ ] Tests: deterministic seed for sampling distribution checks
## ADR links
- [ADR-0018](../../docs/adr/0018-fraud-detection-verification-and-reputation.md) §1, §6–7
- [ADR-0013](../../docs/adr/0013-rolling-stats-smart-routing.md)
## Blocked by
- `08-fraud-reputation-model-persistence_completed.md`

42
docs/archived/alpha-hardening/issues/10-fraud-penalty-calibration-wiring_completed.md Normal file
View File

@@ -0,0 +1,42 @@
Status: done
# 10 — FRAUD: Penalty calibration wiring (forfeit + strike + ban)
## What to build
End-to-end wiring: confirmed audit failure → atomic pending forfeiture + strike + reputation decay + audit-rate snap to max. Ensure payout cannot race penalty (ADR-0015). Document 19× deterrence math in validator README.
**Code refs:**
- `packages/validator/meshnet_validator/__init__.py` — `_slash_route` (~102–134)
- `packages/tracker/meshnet_tracker/server.py` — `_handle_billing_forfeit` (~2429–2464)
- `packages/tracker/meshnet_tracker/billing.py` — `forfeit_pending` (~280–292), payout exclusion for banned (~3337–3344 in settlement loop)
- `packages/validator/README.md` — update 20× → 19× at p=0.05
Per ADR-0018: **full pending forfeiture** is primary penalty; ×0.8 is routing decay per strike, not partial forfeit.
## Test-first
1. Red: integration from issue 34 — extend with multi-hop blame wallet from issue 07.
2. Green: node with pending balance → audit fail → pending zero, strike++, banned on 3rd, excluded from next settlement.
3. Settlement loop skips banned wallets (~3337–3344).
## Acceptance criteria
- [x] Audit failure triggers forfeiture + strike in one tracker transaction — `ValidatorProcess._slash_node` (in-process) and the tracker's `_handle_billing_forfeit` handler (remote) both forfeit-then-strike synchronously in a single call path; each already existed pre-AH-010 and is exercised by `tests/test_forfeiture_penalty.py`
- [x] Banned nodes excluded from `payables` / settlement — `BillingLedger.settle_node_payout` now clamps to the wallet's *current* pending balance under the same lock as the debit, and `_settlement_loop` rechecks ban status and uses the post-clamp amount before sending, so a forfeiture landing between the `payables()` snapshot and the actual payout can never be paid out on top of (ADR-0015 race); covered by `test_60_request_stream_bans_intermittent_first_hop_cheater_not_last_hop`
- [x] Validator uses authenticated forfeit endpoint (issue 02) — `POST /v1/billing/forfeit` is validator-token/admin-gated (ADR-0017 §4, issue 20) and is the documented remote path (`packages/validator/README.md` Usage section); `test_forfeit_endpoint_requires_auth_and_forfeits` exercises the 401→200 flow. No standalone remote-validator process exists in this codebase yet (`contracts` has no networked implementation), so the in-process `ValidatorProcess` continues to call `BillingLedger.forfeit_pending` directly when co-located with the tracker — adding an HTTP-only forfeit client with no real consumer was judged out of scope/overengineering for this issue
- [x] README: `L > 19× g` at p=0.05; pending balance = collateral — already present in `packages/validator/README.md` ("Why the penalty deters cheating")
- [x] Integration test: 60-request fraud scenario → ban within threshold — `tests/test_forfeiture_penalty.py::test_60_request_stream_bans_intermittent_first_hop_cheater_not_last_hop`
## ADR links
- [ADR-0018](../../docs/adr/0018-fraud-detection-verification-and-reputation.md)
- [ADR-0015](../../docs/adr/0015-usdt-custodial-settlement.md)
- Research: [research-verifiable-inference.md](../research-verifiable-inference.md) §1.1
## Blocked by
- `07-fraud-commitment-bisection-blame_completed.md`
- `08-fraud-reputation-model-persistence_completed.md`
- `02-a2-unified-auth-boundary_completed.md`

37
docs/archived/alpha-hardening/issues/11-c6-wallet-binding-proof_completed.md Normal file
View File

@@ -0,0 +1,37 @@
Status: done
# 11 — C6: Wallet binding ownership proof + binding overwrite safety
## What to build
`POST /v1/wallet/register` binds a client Solana wallet to an API key for deposit attribution. Today any Bearer key can bind any wallet string without proving ownership. Prevent hijack and accidental overwrite.
**Code refs:**
- `packages/tracker/meshnet_tracker/server.py` — `_handle_wallet_register` (~2625–2648)
- `packages/tracker/meshnet_tracker/billing.py` — `bind_wallet` (~153+), `_wallet_bindings` / direct overwrite on apply (~351)
Require signed message from wallet pubkey (ed25519 via `cryptography` / solders). Reject rebinding without admin or signed release. Use explicit overwrite policy — today `~351` overwrites binding directly; gossip apply must reject conflicting binds instead of silently clobbering.
## Test-first
1. Red: bind wallet A with only API key, no signature — must fail after fix.
2. Red: wallet already bound to key1; key2 cannot steal without proof.
3. Green: valid signature binds; deposit watcher credits correct API key.
## Acceptance criteria
- [x] Wallet binding requires cryptographic proof of pubkey ownership
- [x] One wallet → one API key (or documented admin override)
- [x] Gossip `bind` events cannot overwrite existing binding via direct overwrite at `~351`
- [x] Tests with deterministic keypairs (local adapter)
## ADR links
- [ADR-0017](../../docs/adr/0017-tracker-authentication-and-authorization.md) §5
- [ADR-0015](../../docs/adr/0015-usdt-custodial-settlement.md)
## Blocked by
- `02-a2-unified-auth-boundary_completed.md`
- `03-c5-starting-credit-zero_completed.md`

29
docs/archived/alpha-hardening/issues/12-c2-on-chain-idempotency.md Normal file
View File

@@ -0,0 +1,29 @@
Status: ready-for-human
# 12 — C2: On-chain settlement idempotency (deferred)
## What to build
Harden payout idempotency so Solana transaction retries never double-pay. Design accepted in ADR-0019 §1; **implementation deferred post-alpha**.
**Code refs:**
- `packages/tracker/meshnet_tracker/server.py` — `_settlement_loop` resend (~3331–3356), `_send_settlement` (~3358–3376)
- `packages/contracts/meshnet_contracts/solana_adapter.py` — `send_payouts` (~186–213)
Today: pending debited before broadcast with stable `settlement_id`; unconfirmed batches resent. Gap: on-chain confirmation vs ledger state if tx succeeds but confirm fails.
## Acceptance criteria
- [ ] `confirm_settlement` only after RPC finalized confirmation
- [ ] Retry path reuses same `settlement_id` and detects already-confirmed signature
- [ ] Property test: N retries → single on-chain transfer per wallet per settlement_id
- [ ] Document recovery procedure for stuck unconfirmed batches
## ADR links
- [ADR-0019](../../docs/adr/0019-money-path-consistency-multi-tracker.md) §1
## Blocked by
Alpha release (ADR-0016 single settlement tracker)

31
docs/archived/alpha-hardening/issues/13-c3-c4-consensus-gated-settlement.md Normal file
View File

@@ -0,0 +1,31 @@
Status: ready-for-human
# 13 — C3/C4: Consensus-gated money mutations (deferred)
## What to build
Route money-affecting ledger events through Raft commit, not gossip-only apply. Extend `raft.py` command set beyond register/deregister. Settlement remains leader-only with treasury key.
**Code refs:**
- `packages/tracker/meshnet_tracker/server.py` — settlement leader gate (~3331–3332), payout batch (~3353–3356)
- `packages/tracker/meshnet_tracker/raft.py` — log entry types (~26–27)
- `packages/tracker/meshnet_tracker/billing.py` — `apply_events` (~301–311)
Design: ADR-0019 §2. **Deferred post-alpha** while single operator holds settlement.
## Acceptance criteria
- [ ] `charge`, `payout`, `forfeit`, `credit`, `settlement`, `bind` commit via Raft log
- [ ] Followers reject direct gossip money mutations
- [ ] Leader-only `_settlement_loop` unchanged in semantics
- [ ] Migration plan from gossip-only billing to Raft-backed log
## ADR links
- [ADR-0019](../../docs/adr/0019-money-path-consistency-multi-tracker.md) §2
## Blocked by
- `12-c2-on-chain-idempotency.md`
- `14-a3-raft-durable-term-vote.md`

25
docs/archived/alpha-hardening/issues/14-a3-raft-durable-term-vote.md Normal file
View File

@@ -0,0 +1,25 @@
Status: ready-for-human
# 14 — A3: Durable Raft term and vote state (deferred)
## What to build
Persist Raft `currentTerm`, `votedFor`, and log metadata to disk. In-memory-only term (~26) risks split leadership after tracker restart → duplicate settlement epochs.
**Code refs:**
- `packages/tracker/meshnet_tracker/raft.py` — `LogEntry.term` (~25–27), election state in `RaftNode`
## Acceptance criteria
- [ ] Term/vote persisted alongside tracker data dir
- [ ] Restart resumes as follower/candidate with monotonic term
- [ ] Test: kill leader mid-settlement, restart, no duplicate payout batch
## ADR links
- [ADR-0019](../../docs/adr/0019-money-path-consistency-multi-tracker.md) §3
## Blocked by
Alpha single-settlement posture

27
docs/archived/alpha-hardening/issues/15-h1-commutative-forfeit.md Normal file
View File

@@ -0,0 +1,27 @@
Status: ready-for-human
# 15 — H1: Commutative forfeit event ordering (deferred)
## What to build
Define deterministic ordering when `forfeit`, `charge`, and `payout` events replicate concurrently. Forfeit snapshots amount at creation (~287) but apply order can desync pending balances under gossip.
**Code refs:**
- `packages/tracker/meshnet_tracker/billing.py` — `forfeit_pending` (~280–292), `_apply_locked` forfeit branch (~345–349)
- `packages/tracker/meshnet_tracker/billing.py` — `_pending_since.setdefault` (~324), wallet bind direct overwrite (~351)
## Acceptance criteria
- [ ] Documented commit order: charges before forfeit before payout for same wallet epoch
- [ ] Forfeit events carry pending snapshot or `(term, index)` for tie-break
- [ ] `setdefault` replaced with explicit merge rules on out-of-order apply
- [ ] Property tests under shuffled event delivery
## ADR links
- [ADR-0019](../../docs/adr/0019-money-path-consistency-multi-tracker.md) §4
## Blocked by
- `13-c3-c4-consensus-gated-settlement.md`

30
docs/archived/alpha-hardening/issues/16-doc-us006-reconciliation_completed.md Normal file
View File

@@ -0,0 +1,30 @@
Status: done
# 16 — DOC: US-006 reconciliation note
## What to build
Reconcile stale US-006 (Solana testnet stake contracts) with ADR-0015/0016 devnet custodial settlement. Issue `docs/issues/06-solana-stake-and-settlement.md` says "never devnet"; ADR-0015 explicitly targets devnet mock-USDT.
Also reconcile legacy fraud issues with the alpha-hardening fraud arc:
- `docs/issues/07-fraud-detection-slash.md` — on-chain stake slash model superseded by pending-balance forfeiture + TOPLOC (ADR-0018)
- `docs/issues/34-forfeiture-penalty.md` — partially implemented; remaining fraud work lives in `.scratch/alpha-hardening/issues/06-fraud-toploc-integration_completed.md` through `10-fraud-penalty-calibration-wiring_completed.md`
## Acceptance criteria
- [ ] Add reconciliation comment atop `docs/issues/06-solana-stake-and-settlement.md` (Status: superseded for alpha — see ADR-0015, issue 33/34)
- [ ] Add **superseded** banner atop `docs/issues/07-fraud-detection-slash.md` → ADR-0018 + issues 06–10
- [ ] Add **superseded for remaining scope** banner atop `docs/issues/34-forfeiture-penalty.md` → ADR-0018 + issues 06–10 (note done items: basic forfeiture wired)
- [ ] Update `docs/prd.json` US-006 description footnote if present
- [ ] Cross-link ADR-0015 devnet decision
- [ ] No production code changes
## ADR links
- [ADR-0015](../../docs/adr/0015-usdt-custodial-settlement.md)
- [ADR-0016](../../docs/adr/0016-alpha-scope-and-known-limitations.md)
## Blocked by
None

23
docs/archived/alpha-hardening/issues/17-doc-duplicate-us020-dedup.md Normal file
View File

@@ -0,0 +1,23 @@
Status: ready-for-human
# 17 — DOC: Duplicate US-020 issue dedup
## What to build
Two files share the US-020 number with different slugs:
- `docs/issues/20-memory-budget-shard-slots-and-dropout-relocation.md` (ready-for-agent)
- `docs/issues/20-tracker-node-hardening.md` (done)
Resolve numbering collision without losing history.
## Acceptance criteria
- [ ] Document canonical mapping in this issue's Comments or a short `docs/issues/README.md` note
- [ ] Renumber or prefix disambiguation (e.g. keep done item as US-020a, renumber memory-budget to next slot) — **human approval before git mv**
- [ ] Update any prd.json / cross-links that reference US-020 ambiguously
- [ ] No production code changes
## Blocked by
Human approval for renumbering. An agent may prepare the mapping note, but must not run `git mv` or rewrite cross-links until the canonical number is approved.

27
docs/archived/alpha-hardening/issues/18-doc-operational-runbooks_completed.md Normal file
View File

@@ -0,0 +1,27 @@
Status: done
# 18 — DOC: Operational runbooks (stubs)
## What to build
Add operational runbook stubs for alpha operators under `docs/runbooks/` (or `.scratch/alpha-hardening/runbooks/` until close-feature):
1. **Ledger backup** — billing SQLite, accounts SQLite, registry DB paths; gossip pause procedure
2. **Treasury key rotation** — devnet mock-USDT mint + treasury keypair rotation without double-credit
3. **Upgrade path** — tracker rolling restart with persisted strike/reputation (post issue 05)
## Acceptance criteria
- [ ] Three markdown runbook stubs with prerequisites, steps, rollback
- [ ] Reference ADR-0015 settlement loop and ADR-0016 trust assumptions
- [ ] Secrets handling: never commit `.env.devnet`, keypairs
- [ ] No production code changes
## ADR links
- [ADR-0015](../../docs/adr/0015-usdt-custodial-settlement.md)
- [ADR-0016](../../docs/adr/0016-alpha-scope-and-known-limitations.md)
## Blocked by
None (stubs can land before issue 05; update after persistence ships)

31
docs/archived/alpha-hardening/issues/19-doc-cryptography-test-env_completed.md Normal file
View File

@@ -0,0 +1,31 @@
Status: done
# 19 — DOC: Cryptography dependency + test environment note
## What to build
Document and verify test/dev environment setup for wallet crypto paths. `packages/node/meshnet_node/wallet.py` uses `cryptography`; failures occur when `.venv` lacks deps. `cryptography>=41` is already declared in `packages/node/pyproject.toml`, so this issue should focus on documenting the editable-install path and only add root/dev extras if tests still import the node wallet without installing the node package.
**Code refs:**
- `packages/node/pyproject.toml` — `cryptography>=41` (verify declared)
- `packages/node/meshnet_node/wallet.py`
- Handoff: tests fail without `cryptography`, `openai`, `langchain` in `.venv`
## Acceptance criteria
- [x] Confirm `cryptography>=41` remains in node package deps; add to root/dev extras only if tests import wallet without node install
- [x] Add short **Test environment** section to `docs/dev/test-env.md` (or `CONTRIBUTING.md` if created): use `.venv/Scripts/python.exe`, `pip install -e packages/node ...`, optional dep skips
- [x] Note which tests require optional deps (`--ignore=test_openai_gateway,...`)
- [x] No unrelated production code changes
## Blocked by
None
## Resolution
- `packages/node/pyproject.toml` already declared `cryptography>=41` — no change needed.
- `conftest.py` adds every `packages/*` dir to `sys.path`, so first-party imports (e.g. `meshnet_node.wallet`) resolve without an editable install of that package — but third-party deps like `cryptography` still must be installed separately. Added `cryptography>=41` to the root `pyproject.toml` `dev` extra so `pip install -e ".[dev]"` alone covers the wallet tests (`test_node_startup.py`, `test_wallet_binding_proof.py`, `test_devnet_treasury.py`, etc.) without requiring a full `packages/node` install (which would otherwise pull in torch/transformers/accelerate/bitsandbytes).
- Added `docs/dev/test-env.md` with setup instructions (Linux + Windows `.venv\Scripts\python.exe`), and a note on optional-dependency tests: `test_real_model_backend.py` / `test_devnet_treasury.py` use `pytest.importorskip` and skip cleanly; `test_openai_gateway.py` hard-imports `openai`/`langchain_openai` with no skip guard (both already in the `dev` extra) — documented the `--ignore=tests/test_openai_gateway.py` fallback for minimal installs.
- Full suite: 311 passed, 3 skipped, 3 pre-existing failures unrelated to this issue (`test_billing_ledger.py::test_proxy_chat_splits_payout_by_tracker_assigned_route_span`, `test_forfeiture_penalty.py::test_probation_earns_nothing_then_earning_begins`, `test_mining_cli.py::test_legacy_start_without_port_uses_next_available_port` — port-in-use env artifact). Wallet-specific tests (`test_wallet_binding_proof.py`, `test_node_startup.py`, `test_devnet_treasury.py`): 50 passed, 2 skipped.

52
docs/archived/alpha-hardening/issues/20-validator-service-token_completed.md Normal file
View File

@@ -0,0 +1,52 @@
Status: done
# 20 — Validator service token for `/v1/billing/forfeit`
## What to build
Define and implement a **validator service token** distinct from client API keys and admin sessions. The validator process must authenticate when calling `POST /v1/billing/forfeit`; arbitrary Bearer strings and client API keys must be rejected. This is a checklist subtask for issue 02 and should normally land in the same PR as the unified auth middleware.
Per [ADR-0017 §4](../../docs/adr/0017-tracker-authentication-and-authorization.md): forfeit accepts **validator service identity or admin session** only.
## Configuration
| Item | Alpha default |
|---|---|
| Env var | `MESHNET_VALIDATOR_SERVICE_TOKEN` (tracker + validator) |
| Config flag | `--validator-service-token` / tracker config file equivalent |
| Header format | `Authorization: Bearer <service-token>` with a dedicated prefix or separate header scheme documented in runbooks (e.g. `Authorization: Service <token>` — pick one and test consistently) |
| Rotation | Manual: set new token on tracker + validator, restart both; document zero-downtime rotation as post-alpha |
## Rejection rules
- Client API keys (`sk-mesh-…`) → **403** on forfeit (even if valid for inference)
- Non-empty garbage Bearer → **401/403**
- Missing auth → **401**
- Valid validator service token → **200** (existing forfeit semantics)
- Admin session → **200** (operator override)
## Test-first
1. Red: validator (or test client) posts forfeit with a valid API key — must fail after fix.
2. Red: `Authorization: Bearer garbage` — must fail (covered by issue 02; this issue defines the accepted token).
3. Green: configured service token succeeds; wrong token fails.
## Acceptance criteria
- [ ] Service token configurable via env/flag on tracker and validator
- [ ] Unified auth middleware resolves service token → `validator` role (issue 02)
- [ ] API keys explicitly rejected on forfeit path
- [ ] Integration test: validator client with service token forfeit succeeds; API key forfeit fails
- [ ] Runbook stub: rotation procedure (manual alpha)
## ADR links
- [ADR-0017](../../docs/adr/0017-tracker-authentication-and-authorization.md) §4
## Related
- `02-a2-unified-auth-boundary_completed.md` — middleware + role checks
## Blocked by
- `02-a2-unified-auth-boundary_completed.md`

52
docs/archived/alpha-hardening/issues/21-honest-noise-calibration-corpus.md Normal file
View File

@@ -0,0 +1,52 @@
Status: ready-for-human
**BLOCKS ALPHA RELEASE.** Scoped 2026-07-06 during alpha-launch-readiness grilling session — must complete before real-money mainnet USDT traffic goes live for the friends + hired-VPS-host launch. Loose/uncalibrated thresholds + manual admin slash-reversal are the stopgap only until this closes.
**Engineering complete 2026-07-06; blocked on a human running it against the real hired-VPS fleet before launch.** The three code gaps below are closed and unit-tested (see Deliverables), but nothing in a dev session can stand in for actually dispatching the job at real hardware — that step, plus the threshold/FPR write-up that depends on its output, needs an operator with the live fleet. See the validator README's "Honest-noise calibration corpus" section for the operational how-to.
# 21 — Honest-noise TOPLOC calibration corpus
## What to build
Before enabling production TOPLOC audit thresholds, collect an **honest-noise baseline** across the active fleet. Run identical inference jobs on every active node/GPU combo; measure the divergence envelope (TOPLOC exponent/mantissa deltas, logprob-rank spread) under real hardware variance. This must be driven by the tracker (scheduled/dispatched job), not a manual one-off script, so it can be re-run as the fleet's hardware mix changes.
Per [ADR-0018 consequences](../../docs/adr/0018-fraud-detection-verification-and-reputation.md): threshold calibration requires an honest-noise corpus across the fleet before production thresholds.
Research anchor: `.scratch/alpha-hardening/research-verifiable-inference.md` §8 layer 3 — "collect this first — run identical jobs across the current node fleet to measure the honest divergence envelope before setting thresholds."
**Launch context (why this is buildable now, not a research project):** first-launch nodes are hired VPS/VPC hosts under our own direct control (test infrastructure we pay for, not third-party volunteers) — not a long-term topology, but risk-free for calibration purposes since there's no external party to dispute a bad reading. Friends are client-side users of the API in this phase, not node operators. Run the calibration pass against this small, fully-controlled fleet first; hired hosts stay on probation (no upfront stake) until it's done, then move to paid USDT serving once thresholds derive from their own hardware.
**Current gap (historical — closed 2026-07-06):** the three engineering pieces below were missing when this issue was filed; all are now implemented and unit-tested. Remaining work is the human calibration run on the live hired-VPS fleet.
1. `verify_activation_proofs()` (`packages/validator/meshnet_validator/audit.py:94-127`) returns a **plain bool** — no raw TOPLOC divergence/distance value is ever computed or surfaced. Every "done" fraud-detection issue (06–10) currently runs on a guessed threshold baked into that bool, not a calibrated one.
2. Fleet dispatch exists but is the wrong shape: `_handle_benchmark_hop_penalty` / `_handle_benchmark_results` (`packages/tracker/meshnet_tracker/server.py:2998-3104`, from the old US-030 latency work) targets pinned 1–3-node *routes* and measures latency, not TOPLOC divergence across *every* registered node.
3. Storage is the wrong shape: `record_audit_outcome` (`packages/contracts/meshnet_contracts/__init__.py:416`) persists only `strike_count`/`banned`/`passed` to `registry_events` — no divergence value, no GPU/dtype/hardware-profile column anywhere. Benchmark results otherwise land in a flat JSON file (`server.benchmark_results_path`), not a queryable per-node/hardware schema.
## Deliverables
- [x] Extend the TOPLOC verify call path (`audit.py`) to return the raw distance/divergence metric alongside the existing bool — `verify_activation_proofs_detailed()` / `ToplocVerificationResult` in `packages/validator/meshnet_validator/audit.py`; `verify_activation_proofs()` kept as a thin bool-only wrapper for existing callers. Also fixes a real bug this issue's code-read surfaced: the old code did `bool(_call_toploc(...))`, which is always `True` for the real `toploc` library's non-empty per-chunk `VerificationResult` list regardless of divergence — `tests/test_toploc_audit.py::test_verify_activation_proofs_detailed_aggregates_per_chunk_divergence` exercises this directly.
- [x] Extend the existing fleet-dispatch pattern (`server.py:2998+`) from pinned-route benchmarking to a tracker-scheduled job that hits **every currently registered node** with a fixed prompt/model/seed — `POST /v1/calibration/toploc/run` (admin/validator-gated, same shape as `POST /v1/benchmark/hop-penalty`) in `packages/tracker/meshnet_tracker/server.py`. Dispatches to every node that can solo-serve the full model range (single-hop pinned route, isolating one node's hardware noise from route-composition effects); partial-shard nodes are reported under `skipped_partial_shard_node_ids`, and nodes that don't answer the on-demand TOPLOC commitment fetch are reported per-node under `"skipped": "..."` rather than counted as pass or fail. See `tests/test_toploc_calibration_dispatch.py`.
- [x] Add a small SQLite table (same pattern as `billing.py`/`accounts.py`) keyed by node wallet + GPU model + dtype, storing the divergence value per calibration run — `packages/tracker/meshnet_tracker/calibration.py::ToplocCalibrationStore`, `toploc_calibration_runs` table.
- [x] Aggregation: p99 honest envelope + safety margin computed from that table, written as the recommended tolerance constants — `ToplocCalibrationStore.envelope()`, exposed via `GET /v1/calibration/toploc/results`.
- [x] Gate checklist: production audit enable blocked until corpus covers ≥N distinct hardware profiles — `ToplocCalibrationStore.gate_status(min_hardware_profiles=N)`; N is `--toploc-calibration-gate-min-hardware-profiles` (default 1) on the tracker CLI, documented alpha exception in the validator README.
## Acceptance criteria
- [ ] Corpus collected from the current hired-VPS fleet (documented as a small-fleet alpha corpus, not the eventual volunteer-fleet corpus) — **not done: needs a human to run `POST /v1/calibration/toploc/run` against the live hired-VPS fleet before launch; no such fleet exists in a dev session.**
- [ ] Threshold constants in validator config derived from corpus, not guessed — mechanically ready (`envelope()` returns them) but depends on the real corpus above; not yet wired into `ToplocAuditConfig` as enforced thresholds (deliberately — enforcing unvalidated thresholds would be worse than today's guessed bool).
- [ ] False-positive rate estimate documented at chosen thresholds — `envelope()` returns `estimated_false_positive_rate` (in-sample: fraction of the recorded corpus the recommended thresholds would themselves flag); needs the real corpus to be a meaningful number, and should be written up in the runbook once collected.
- [x] README / runbook cross-link: **do not enable production audits** until this issue closes — `packages/validator/README.md` "TOPLOC audit contract" section, updated with the full operational how-to.
- [x] Note in the runbook that this alpha corpus must be re-run once the fleet grows beyond the hired-VPS set (different hardware mix invalidates the envelope) — same README section; [runbook 04](../runbooks/04-toploc-calibration-run.md).
## ADR links
- [ADR-0018](../../docs/adr/0018-fraud-detection-verification-and-reputation.md) — Consequences (honest-noise corpus)
## Blocked by
- `06-fraud-toploc-integration_completed.md` (TOPLOC wired; calibration uses same primitive) — done
## Blocks (prod gate)
- Alpha release to real-money friends+hired-VPS launch (raised from "production adaptive audit thresholds" to a hard alpha-release gate during 2026-07-06 grilling)
- Production enable of adaptive audit thresholds (issues 09–10 in prod)

25
docs/archived/alpha-hardening/issues/22-doc-memory-project-status_completed.md Normal file
View File

@@ -0,0 +1,25 @@
Status: done
# 22 — DOC: MEMORY.md + project-status alpha-hardening index
## What to build
Update persistent memory files so agents and humans find the alpha-hardening feature without stale handoff paths.
## Acceptance criteria
- [x] `.claude/memory/MEMORY.md` — index entry for alpha-hardening (`.scratch/alpha-hardening/`, ADRs 0016–0019, issue count)
- [x] `.claude/memory/project-status.md` — brief alpha-hardening section: planning complete, Bucket 1 blockers next, link README
- [x] Cross-link `.scratch/alpha-hardening/handoff.md` from README (not temp path)
## ADR links
- [ADR-0016](../../docs/adr/0016-alpha-scope-and-known-limitations.md)
## Blocked by
None — completed
## Comments
2026-07-04 triage: already satisfied by `.claude/memory/MEMORY.md`, `.claude/memory/project-status.md`, and `.scratch/alpha-hardening/README.md`.

53
docs/archived/alpha-hardening/issues/23-dynamic-hf-pricing_completed.md Normal file
View File

@@ -0,0 +1,53 @@
Status: done
Scoped 2026-07-06 during alpha-launch-readiness grilling session. High priority, ship-soon for launch — **not** an alpha-release blocker (unlike issue 21): a stale/static price is a revenue/business-model risk, not a safety risk, so the friends + hired-VPS launch may proceed on the current static default while this lands in parallel.
# 23 — Dynamic per-model pricing benchmarked against HuggingFace inference rates
## What to build
Client-facing price per model should track the market: **80% of the cheapest comparable provider rate on HuggingFace's inference marketplace** (`https://huggingface.co/inference/models`), refreshed daily, auto-adjusting so served models stay competitively priced as the market moves. Nodes are unaffected by this loop (per launch design: clients are the only party spending real money; node payouts come from the 90/10 split of whatever price is charged, per ADR-0015/`packages/validator/README.md`).
**Current state (confirmed by code read 2026-07-06):** pricing is 100% static today. `DEFAULT_PRICE_PER_1K_TOKENS = 0.02` (`packages/tracker/meshnet_tracker/billing.py:21`) is the fallback nearly every model hits, since `model_presets.json` currently has no `price_per_1k_tokens` key for any preset. `BillingLedger.set_price(model, price)` (`billing.py:67-69`) is the only write path and already exists — no CLI/admin route calls it yet. No external HTTP/market-data integration exists anywhere in the tracker.
**Data source:** `https://huggingface.co/inference/models` aggregates multiple providers (novita, together, fireworks-ai, deepinfra, etc.) with per-model, per-provider $/1M input and output token pricing; the "cheapest" badge already identifies the lowest-cost provider per model on the page itself. It supports a GET query param for filtering, e.g. `?search=GLM`. **No confirmed public JSON API was found** during this session's fetch — the page reads as a rendered table. Owner's suggestion: try a plain `requests` + BeautifulSoup scrape first; if the pricing table turns out to be client-rendered (not present in the initial HTML), that's the fallback signal to escalate to a headless-browser fetch (e.g. Playwright) — confirm which is needed during implementation before building the full pipeline around it. Another data source is acceptable if more convenient/stable, owner is not wedded to this specific page.
## Deliverables
- [x] Live-fetch attempt (requests + BeautifulSoup against the HF page with `?search=<model-family>`, or an equivalent stable source) as the primary path — confirm during implementation whether the pricing table is present in the raw HTML or requires a headless-browser fetch, and note which in the PR
- [x] Extend `model_presets.json` per model with: `hf_aliases` (curated list of comparable HF model+provider IDs — **human-verified, not auto-discovered**), `hf_verified_match_note` (free text: params count + quantization confirmation, so a human signs off once per alias that it is a fair comparable before it's used for auto-pricing), `hf_last_price_per_1k` (derived from the $/1M rate), `hf_last_updated` (ISO date)
- [x] Daily refresh job reusing the tracker's existing daemon-thread pattern (`_settlement_loop`/`_deposit_loop` in `server.py`, `threading.Event().wait(interval)` loop) — for each preset with a non-empty `hf_aliases` list, fetch current pricing for those aliases, compute `0.8 × cheapest matched alias price`, call `set_price()`, and update `hf_last_price_per_1k`/`hf_last_updated`
- [x] Every price change logged (old price, new price, source alias, timestamp) — needed for dispute auditability if a client questions a charge
- [x] Fallback behavior: empty/missing `hf_aliases`, fetch failure, or no verified match → silently keep the existing static default price. Never error the pricing path, never zero-price a model
## Acceptance criteria
- [x] At least one model preset has a working end-to-end refresh (alias → live fetch → 80% computed price → `set_price()` called → metadata updated) demonstrated in a test
- [x] Models without a curated/verified alias continue to use the static default, unaffected by this feature
- [x] Fetch failures (network error, page structure change, no match found) degrade gracefully — logged, not raised to the request path
- [x] Price-change log is queryable/inspectable (doesn't need a UI yet — a log line or table row is sufficient for alpha)
- [x] Note in the runbook/issue on which fetch mechanism (plain HTTP scrape vs. headless browser) was actually required, so the next person doesn't have to rediscover it
## Implementation notes (2026-07-06)
**Fetch mechanism confirmed: plain HTTP scrape, no headless browser needed.** Live-fetched `https://huggingface.co/inference/models?search=GLM` this session — the pricing table is server-rendered into the initial HTML response (SvelteKit SSR), confirmed by grepping the raw response for `cheapest`/`$`-prefixed price cells before any JS runs. A stdlib `urllib.request` GET + `html.parser.HTMLParser`-based table walk is sufficient; no `requests`/`bs4`/Playwright dependency was added, matching this package's existing zero-new-HTTP-dependency convention (`gossip.py`/`raft.py`/`server.py` all use `urllib.request` only). Each row's most stable extraction anchor turned out to be the `<a href="/<org>/<repo>/?inference_api=true&inference_provider=<provider>">` link, not the display text (which duplicates the repo id at two responsive breakpoints and is easy to mis-parse).
**What shipped:** new `packages/tracker/meshnet_tracker/hf_pricing.py` — pure HTML parser (`parse_hf_pricing_table`), alias matching (`cheapest_matching_quote`, supports both `org/repo` and `org/repo::provider` forms so a human can pin a specific provider's deployment), a pure per-preset computation function (`refresh_preset_price`, never raises), and `HfPricingLog` (SQLite-backed change log, same shape as `billing.py`/`calibration.py`). `TrackerServer` gained an opt-in (`enable_hf_pricing=True` / `--enable-hf-pricing`) daily daemon thread (`_hf_pricing_loop`, same `threading.Event().wait(interval)` shape as `_settlement_loop`) and `GET /v1/pricing/hf/history` (admin/validator-gated, mirrors `/v1/calibration/toploc/results`). `model_presets.json`'s `kimi-k2.7` preset now carries the `hf_aliases`/`hf_verified_match_note` schema fields, left as an empty list pending a human sign-off on a genuinely comparable HF listing (params count + quantization) — per this issue's own "human-verified, not auto-discovered" requirement, an agent should not fabricate that sign-off. This also means the shipped default config demonstrates the required "no alias → static price, unaffected" fallback for a real production preset; the alias→live-fetch→80%→set_price() path is demonstrated end-to-end against an injected fetch backend in `tests/test_hf_pricing_dispatch.py` (the `fetch_html=`/`hf_pricing_fetch_html=` injection point mirrors this codebase's `backend=` convention for anything that would otherwise hit the network in tests).
**Bug caught and fixed while wiring this in:** `TrackerServer` previously did `dict(DEFAULT_MODEL_PRESETS)` when no explicit `model_presets` was passed — a shallow copy that aliases every preset's inner dict to the shared module-level global. Writing `hf_last_price_per_1k`/`hf_last_updated` in place would have leaked across every other `TrackerServer` instance in the same process (real risk in the test suite, and in any future multi-tracker-in-one-process embedding). Fixed with a `_clone_model_presets()` helper that also shallow-copies each preset dict.
**Follow-up for a human (not a completion blocker):** populate real `hf_aliases`/`hf_verified_match_note` entries for production presets once someone has confirmed a genuinely comparable HF-listed deployment (params + quantization) — that activates dynamic pricing for that model on the next refresh tick. Until then every preset safely stays on its static price.
Tests: `tests/test_hf_pricing.py` (11 tests: parsing, blended-price math, alias matching incl. provider-scoped aliases, all three fallback paths, log persistence) + `tests/test_hf_pricing_dispatch.py` (5 tests: full TrackerServer end-to-end refresh, unaffected-without-alias, history auth gating, history content, history model filter). Full suite (`pytest tests/ -q -k "not integration"`): 346 passed, 2 skipped.
## ADR links
- [ADR-0015](../../docs/adr/0015-usdt-custodial-settlement.md) — settlement/pricing this touches (90/10 split, per-model pricing)
## Blocked by
None — independent of the alpha-hardening trust-boundary work; touches `billing.py`/`server.py` pricing paths only.
## Blocks
None — ship-soon for launch quality, not a release gate (see status note above).

92
docs/archived/alpha-hardening/issues/24-routing-telemetry-resume.md Normal file
View File

@@ -0,0 +1,92 @@
Status: ready-for-agent
Scoped 2026-07-07 from an interrupted Claude session. This is a resume/cleanup task for routing and live-progress work that is partly committed and partly left dirty in the working tree.
# 24 - Finish learned-routing telemetry and live-progress cleanup
## Current state
The main dynamic routing feature is already committed at `518c259` (`routing improvements - dynamic (wip)`):
- `packages/tracker/meshnet_tracker/routing_stats.py` - decayed-EWMA route stats store, epsilon-greedy route selection, diagnostics.
- `packages/tracker/meshnet_tracker/server.py` - route enumeration per head, bandit selection in the chat proxy, epoch bumps on node join/leave, `/v1/routing`, route sample recording with 8-token hygiene.
- `packages/tracker/meshnet_tracker/cli.py` - `--route-explore-share`, `--route-weight-alpha`, `--route-stats-half-life` and env vars.
- `packages/tracker/meshnet_tracker/dashboard.html` - "Routing (learned)" panel.
- `docs/adr/0021-dynamic-statistical-routing.md` - design record.
- `tests/test_dynamic_routing.py` - includes the exact GPU(0-21)+CPU(0-39) topology, hybrid downstream `start_layer=22`, 0.6/0.4 traffic split for a 1.5 TPS ratio, and scout-rate behavior.
The current working tree still has uncommitted follow-up work:
- `packages/node/meshnet_node/torch_server.py` - tracks in-flight chat requests, exposes `TorchNodeServer.current_requests`, prints generation progress with TPS.
- `packages/node/meshnet_node/startup.py` - sends `current_requests` in heartbeat payloads and increases heartbeat cadence while busy.
- `packages/tracker/meshnet_tracker/server.py` - accepts heartbeat `current_requests`, includes them in `/v1/network/map`, and logs `proxy connecting` before upstream connection.
- `packages/tracker/meshnet_tracker/dashboard.html` - enriches the call wall from heartbeat `current_requests` so active requests remain visible even before terminal proxy events.
- `tests/test_real_model_backend.py` and `tests/test_tracker_routing.py` - targeted coverage for current-request snapshots, heartbeat sanitization/storage, and TPS progress logging.
- `QUICKSTART.md` - documents optional linear-attention fast-path packages for Qwen3.5/3.6 GPU nodes.
There is also an untracked local file, `.claude/settings.local.json`, which should not be included unless the owner explicitly wants local Claude settings committed.
## Known blocker found during resume
Targeted pytest currently fails during import before reaching the new tests:
```text
TypeError: unsupported operand type(s) for |: 'builtin_function_or_method' and 'NoneType'
```
Immediate cause: `packages/tracker/meshnet_tracker/server.py:1490` annotates `ws_lock: threading.Lock | None = None`. `threading.Lock` is a factory function at runtime, not a type, so `| None` evaluates eagerly and crashes. This exists on `HEAD` too, not just in the dirty telemetry changes.
Fix options:
- Add `from __future__ import annotations` at the top of `server.py`, then run enough tests to catch any annotation side effects.
- Or change that annotation to a safe runtime type such as `Any | None` / remove the union annotation. Keep the change minimal.
## What to do next
1. Fix the import-time `threading.Lock | None` crash.
2. Re-run the targeted tests:
```bash
.\.venv\Scripts\python.exe -m pytest tests/test_tracker_routing.py::test_tracker_heartbeat_stores_current_requests tests/test_tracker_routing.py::test_normalize_current_requests_sanitizes_payload tests/test_real_model_backend.py::test_current_requests_snapshot_while_generating tests/test_real_model_backend.py::test_distributed_generating_log_includes_tps -q
```
3. Run the relevant routing regression tests:
```bash
.\.venv\Scripts\python.exe -m pytest tests/test_dynamic_routing.py tests/test_tracker_routing.py -q
```
4. If practical, run the non-integration suite:
```bash
.\.venv\Scripts\python.exe -m pytest tests/ -q -m "not integration"
```
5. Confirm or document the pre-existing failure from the interrupted session: `test_proxy_chat_splits_payout_by_tracker_assigned_route_span` reportedly failed on `HEAD` too and was unrelated.
6. Commit the intentional work in two commits if it remains naturally split:
- learned routing is already committed in `518c259`; leave it alone unless fixing regressions there.
- commit the live-progress/current-request telemetry cleanup separately after tests pass.
## Acceptance criteria
- [ ] Importing `meshnet_tracker.server` no longer crashes on the lock annotation.
- [ ] Current-request heartbeat payloads are sanitized and surfaced in `/v1/network/map`.
- [ ] Node-side in-flight chat snapshots report request id, model, token count, elapsed seconds, tokens/sec, and routing completion.
- [ ] Dashboard call wall can show active requests from heartbeat data, not only tracker console terminal events.
- [ ] Targeted telemetry tests pass.
- [ ] Dynamic routing tests still pass, including GPU(0-21)+CPU(0-39) hybrid-route enumeration and traffic split behavior.
- [ ] Full or non-integration suite result is recorded; unrelated pre-existing failures are named explicitly.
- [ ] `.claude/settings.local.json` remains uncommitted unless intentionally approved.
## ADR links
- [ADR-0020](../../docs/adr/0020-chat-streaming-live-progress-and-mixed-topology-routing.md)
- [ADR-0021](../../docs/adr/0021-dynamic-statistical-routing.md)
## Blocked by
None. The import-time annotation crash is the first fix.
## Blocks
Clean handoff/commit of the interrupted live routing progress work.

58
docs/archived/alpha-hardening/issues/25-per-node-kv-cache-distributed.md Normal file
View File

@@ -0,0 +1,58 @@
Status: implemented 2026-07-08 — pending live 2-node GPU verification
Implemented in `packages/node/meshnet_node/model_backend.py` + `torch_server.py`; design in
[ADR-0022](../../../docs/adr/0022-sharded-per-node-kv-cache.md); tests in
`tests/test_kv_cache_distributed.py` (11 fast tests + env-gated golden test,
`MESHNET_REAL_MODEL_TESTS=1`).
**Measured (two-shard Qwen2.5-0.5B 0-11/12-23, CPU, 44-token prompt, 40 steps):**
stateless 7.05 tps decaying 32% (8.09 → 5.50 first-10 vs last-10); cached 18.93 tps and
FLAT (17.21 → 19.28) — 2.68× overall, gap grows quadratically with length. Remaining
acceptance item: re-measure on the live 2-node GPU topology (needs both machines).
Scoped 2026-07-08 from a live two-machine distributed-inference debugging session (Qwen2.5-0.5B GPU+GPU pipeline, and Qwen3.6-35B-A3B mixed GPU/CPU). The ADR-0020 mixed-topology `start_layer` bug is fixed (`518c259`, `e44abc9`, `1ecc599`); this issue is the next performance blocker in the same code path.
# 25 — Sharded per-node KV cache for distributed generation (MoE/hybrid-attention aware)
## What to build
The distributed generation loop (`torch_server.py:515-612`, `_do_chat_completions` distributed path) currently has **no KV cache at all**: `model_backend.py` passes `use_cache: False` in every layer-forward call (lines 763, 768, 770-771), and each autoregressive step re-encodes the *entire* prompt-so-far from scratch (`backend.encode_prompt(current_text)`), re-running every layer on every node in the route for every generated token.
Observed cost of this on a live 2-node Qwen2.5-0.5B GPU pipeline (layers 0-20 / 21-23): tps decayed from 22.3 (at 235 output tokens) to 12.6 (at 449 tokens) within a single generation — the expected quadratic-cost signature. On the Qwen3.6-35B-A3B mixed-topology case this collapses to ~0.07 tps even after the routing fix, partly for this reason.
`X-Meshnet-Session` already exists on the wire (`torch_server.py:707`, minted fresh **per token**, not per generation) but today only labels one activation transfer for chunk reassembly/logging — it is not used to key any cached state.
| Subtask | Owner package | Deliverable |
|---|---|---|
| Session lifecycle | `packages/node/meshnet_node/torch_server.py` | Mint session ID once per chat request (not per token); reuse across all steps of that generation; add `X-Meshnet-Seq-Len` / position header so a node can tell prefill from decode steps |
| Per-node sharded cache | `packages/node/meshnet_node/model_backend.py` | `TorchModelShard` holds a `session_id → cache_state` map scoped to *its own* layer range only (naturally sharded — no node stores another node's KV); `forward_bytes` takes `use_cache=True` and returns/reuses `past_key_values` (or `use_cache=False` for the prefill token to keep failure/eviction simple) |
| Prefill vs. decode split | `packages/node/meshnet_node/torch_server.py` | Step 0 sends the full prompt activation (current behavior); steps 1+ send only the newest token's hidden state (`[1, 1, hidden]`) with correct `position_ids`, cutting per-step payload from O(seq_len) to O(1) |
| MoE / hybrid-attention state | `packages/node/meshnet_node/model_backend.py` | Cache abstraction must hold "whatever `use_cache=True` returns for this layer range," not assume standard K/V tensors — Qwen3.6's linear-attention/hybrid layers (see `[transformers] The fast path is not available...` warning already logged at startup) cache **recurrent conv/delta state**, not K/V pairs. MoE expert routing itself is layer-local and needs no cross-token cache, but confirm no expert-choice state leaks across the stateless-vs-cached boundary when `use_cache` toggles between prefill and decode |
| Cache lifecycle | `packages/node/meshnet_node/torch_server.py` | TTL + LRU eviction per node (bounded by `max_loaded_shards`/memory budget); explicit "cache miss" response so a restarted/evicted node causes the head to fall back to a full re-prefill instead of a hard error — keep today's fully-stateless path as the recovery mode |
| Correctness parity | `tests/` | Golden-output test: distributed multi-token output with caching enabled must match the existing stateless path token-for-token (or within sampling tolerance) for a fixed prompt/seed |
**Non-goals for first landing:** cross-node cache migration/rebalancing on route change (evict + re-prefill is acceptable initially); speculative decoding; batching multiple concurrent sessions' KV within one node beyond what eviction already requires.
**Code refs:**
- `packages/node/meshnet_node/torch_server.py:515-612` — distributed generation loop (`current_text = current_text + token_str`, full re-encode every step)
- `packages/node/meshnet_node/torch_server.py:690-789` — `_run_downstream_pipeline`, session minting, `X-Meshnet-Session`/`X-Meshnet-Hop-Index`/`X-Meshnet-Start-Layer` headers
- `packages/node/meshnet_node/model_backend.py:189-201, 330-351, 763-771` — `use_cache: False` call sites, `effective_start` layer-slicing logic that any cache keying must respect
- `docs/adr/0020-chat-streaming-live-progress-and-mixed-topology-routing.md` — prerequisite routing fix this issue builds on
- `docs/adr/0021-dynamic-statistical-routing.md` — route selection this cache must stay compatible with (a route change mid-generation should trigger cache-miss fallback, not corruption)
## Acceptance criteria
- [x] A session ID is stable across all steps of one chat generation (not re-minted per token) — minted once in `_do_chat_completions`, asserted in `test_session_is_stable_and_decode_payloads_are_single_token`
- [x] Steps after the first prefill send only the new token's activation (`[1, 1, hidden]` via `encode_next_token`) with `X-Meshnet-Cache: decode` + `X-Meshnet-Past-Len`
- [x] Each node caches state only for its own shard's layer range (`TorchModelShard.kv_sessions`; sharding falls out of per-node layer execution)
- [x] Cache abstraction is not K/V-shaped-only: `DynamicCache(config=model.config)` — the same construction Qwen3.6-Next's own forward uses for hybrid linear-attention conv/delta state; store treats it as opaque; `TypeError` fallback disables caching per-backend
- [x] Bounded memory: TTL (600 s, `MESHNET_KV_TTL_SECONDS`) + LRU (8, `MESHNET_KV_MAX_SESSIONS`); miss → HTTP 409 `{"error": "cache_miss"}` → head re-prefills (tested)
- [x] Golden-output test: cached and stateless produce identical token ids on real two-shard Qwen2.5-0.5B (`test_cached_distributed_generation_matches_stateless_golden`, passed)
- [x] Measured (CPU two-shard proxy, 40 steps): stateless 7.05 tps w/ 32% decay → cached 18.93 tps flat, 2.68×. ⚠️ still to run on the live 2-node GPU topology
- [x] `tests/test_two_node_pipeline.py` and `tests/test_dynamic_routing.py` pass (30 passed; 6 tmp-dir fixture errors are a pre-existing Windows temp-permission env issue, identical on clean tree)
- [x] Design captured in [ADR-0022](../../../docs/adr/0022-sharded-per-node-kv-cache.md) incl. cache-miss/route-change interaction with ADR-0021
## Notes
MoE routing (router + expert FFN) is layer-local per token and does not itself need a cross-token cache — it was ruled out as the cause of the earlier Qwen3.6 garbage-output bug (that was the ADR-0020 `start_layer` double-execution). The MoE angle that *does* matter here is architecture-awareness in the cache design: don't hardcode a K/V tensor shape assumption that breaks on Qwen3.6's hybrid attention layers.