feat: USDT reward system — billing ledger, devnet treasury, settlement loop, forfeiture PoW, tracker dashboard (US-030…US-035, ADR-0015)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Dobromir Popov
2026-07-02 22:31:02 +02:00
parent 4d4ab607ca
commit 1e0aa6ea8f
24 changed files with 3168 additions and 994 deletions

View File

@@ -1,48 +1,48 @@
# US-020 — Manual route selection + hop-penalty benchmarking
## Context
The tracker auto-selects inference routes based on synthetic benchmark scores. To measure
the real cost of adding hops (latency per node boundary), we need:
1. A way to pin a request to a specific route so we control the variable.
2. A benchmark endpoint that runs the same prompt through 1-node, 2-node, and 3-node
routes and records per-hop latency.
Results are stored to disk. Routing algorithm is **not** changed in this story — this is
data collection only. The data will inform a future routing optimisation story.
## Design decisions (grilled 2026-07-01)
| Decision | Choice |
|---|---|
| Route spec | Optional `route` field in JSON request body (list of node IDs) |
| Trigger | Explicit only — `POST /v1/benchmark/hop-penalty` endpoint |
| Auth | Header-presence stub (`Authorization` must be non-empty); real auth in future story |
| Routing integration | Store data only; routing algorithm unchanged |
| Persistence | Append to `benchmark_results.json` in tracker working dir; in-memory queryable |
## Acceptance criteria
- `POST /v1/chat/completions` accepts optional `"route": ["<node_id>", ...]` in the
request body. If present, the tracker uses those nodes in order instead of auto-selecting.
If absent, existing routing is unchanged (no breaking change for unaware clients).
- Missing or invalid node IDs in `route` return HTTP 400 with a descriptive error.
- `POST /v1/benchmark/hop-penalty` is auth-gated: requests without a non-empty
`Authorization` header return HTTP 401. Body: `{"model": "...", "prompt": "...",
"max_new_tokens": 64}`.
- Benchmark fans out to up to three routes: 1-node (single node covering all layers),
2-node (two consecutive shard nodes), 3-node (three nodes) — using whatever is
currently registered. Routes with insufficient coverage are skipped, not errored.
- Response includes per-route breakdown: `total_ms`, `per_hop_ms: [...]`,
`tokens_generated`, `route: [node_id, ...]`.
- Results are appended to `<tracker_working_dir>/benchmark_results.json` (created if
absent) as a JSON array. Each entry includes timestamp, model, prompt hash, and the
per-route breakdown.
- `GET /v1/benchmark/results` returns the stored results array. Also auth-gated.
- Clients that never send `route` or call `/v1/benchmark/*` are completely unaffected.
- Integration test: send the same prompt via a pinned 1-node route and a pinned 2-node
route; assert 2-node result has 2 entries in `per_hop_ms`; assert both records appear
in `benchmark_results.json`.
- `python -m pytest` passes from repo root.
- Commit only this story's changes.
# US-020 — Manual route selection + hop-penalty benchmarking
## Context
The tracker auto-selects inference routes based on synthetic benchmark scores. To measure
the real cost of adding hops (latency per node boundary), we need:
1. A way to pin a request to a specific route so we control the variable.
2. A benchmark endpoint that runs the same prompt through 1-node, 2-node, and 3-node
routes and records per-hop latency.
Results are stored to disk. Routing algorithm is **not** changed in this story — this is
data collection only. The data will inform a future routing optimisation story.
## Design decisions (grilled 2026-07-01)
| Decision | Choice |
|---|---|
| Route spec | Optional `route` field in JSON request body (list of node IDs) |
| Trigger | Explicit only — `POST /v1/benchmark/hop-penalty` endpoint |
| Auth | Header-presence stub (`Authorization` must be non-empty); real auth in future story |
| Routing integration | Store data only; routing algorithm unchanged |
| Persistence | Append to `benchmark_results.json` in tracker working dir; in-memory queryable |
## Acceptance criteria
- `POST /v1/chat/completions` accepts optional `"route": ["<node_id>", ...]` in the
request body. If present, the tracker uses those nodes in order instead of auto-selecting.
If absent, existing routing is unchanged (no breaking change for unaware clients).
- Missing or invalid node IDs in `route` return HTTP 400 with a descriptive error.
- `POST /v1/benchmark/hop-penalty` is auth-gated: requests without a non-empty
`Authorization` header return HTTP 401. Body: `{"model": "...", "prompt": "...",
"max_new_tokens": 64}`.
- Benchmark fans out to up to three routes: 1-node (single node covering all layers),
2-node (two consecutive shard nodes), 3-node (three nodes) — using whatever is
currently registered. Routes with insufficient coverage are skipped, not errored.
- Response includes per-route breakdown: `total_ms`, `per_hop_ms: [...]`,
`tokens_generated`, `route: [node_id, ...]`.
- Results are appended to `<tracker_working_dir>/benchmark_results.json` (created if
absent) as a JSON array. Each entry includes timestamp, model, prompt hash, and the
per-route breakdown.
- `GET /v1/benchmark/results` returns the stored results array. Also auth-gated.
- Clients that never send `route` or call `/v1/benchmark/*` are completely unaffected.
- Integration test: send the same prompt via a pinned 1-node route and a pinned 2-node
route; assert 2-node result has 2 entries in `per_hop_ms`; assert both records appear
in `benchmark_results.json`.
- `python -m pytest` passes from repo root.
- Commit only this story's changes.

View File

@@ -1,4 +1,4 @@
Status: ready-for-agent
Status: done
# 32 — Devnet custodial treasury: mock-USDT mint + deposit watcher

View File

@@ -1,4 +1,4 @@
Status: ready-for-agent
Status: done
# 33 — Settlement loop: leader-only batched USDT payouts

View File

@@ -1,4 +1,4 @@
Status: ready-for-agent
Status: done
# 34 — Hardened proof-of-work: pending-balance forfeiture penalty

View File

@@ -1,4 +1,4 @@
Status: ready-for-agent
Status: done
# 35 — Tracker web dashboard

View File

@@ -704,12 +704,13 @@
"Commit only this story's changes"
],
"priority": 30,
"status": "open",
"status": "done",
"notes": "Source issue: .scratch/distributed-inference-network/issues/30-manual-route-and-hop-benchmark.md. Design decisions grilled 2026-07-01: route via body field, explicit-only benchmark trigger, auth stub, routing algorithm unchanged, persist to JSON file.",
"dependsOn": [
"US-014",
"US-019"
]
],
"completionNotes": "Pinned routes: optional \"route\": [node_id,...] in POST /v1/chat/completions (400 on unknown/invalid ids; absent field unchanged). POST /v1/benchmark/hop-penalty + GET /v1/benchmark/results, both auth-gated (non-empty Authorization). Fans out to 1/2/3-node routes via _find_pinned_route; per-hop latency derived incrementally (k-node total minus (k-1)-node total); appends to benchmark_results.json (path configurable). 6 tests in tests/test_manual_route_benchmark.py."
},
{
"id": "US-031",
@@ -745,11 +746,12 @@
"Deterministic local adapter still works for CI without any validator"
],
"priority": 32,
"status": "open",
"status": "done",
"notes": "Supersedes 'testnet never devnet' note in issue 06. solana-py + spl-token client libs.",
"dependsOn": [
"US-031"
]
],
"completionNotes": "SolanaCustodialTreasury in packages/contracts/meshnet_contracts/solana_adapter.py: urllib JSON-RPC client + solders signing (installed solana-py 0.40 has no sync client), deposit parsing from jsonParsed token balances, batched payouts, devnet helpers (create_mock_usdt_mint, mint_mock_usdt). scripts/devnet_setup.py writes .env.devnet. POST /v1/wallet/register binds wallet→api_key (replicated ledger 'bind' events). Deposit watcher thread credits exactly-once via deposit-<signature> event ids. Tests: fake-treasury watcher tests + skipif-gated solana-test-validator e2e."
},
{
"id": "US-033",
@@ -764,12 +766,13 @@
"End-to-end devnet test: fund client → run inference → observe node wallet USDT balance increase"
],
"priority": 33,
"status": "open",
"status": "done",
"notes": "Mining-pool standard: threshold + max-interval, dust floor. Treasury key only on settlement-capable trackers.",
"dependsOn": [
"US-019",
"US-032"
]
],
"completionNotes": "Settlement loop in TrackerServer: leader-only (raft.is_leader; standalone settles), trigger pending≥threshold OR pending age≥settle_period, dust floor. Pending debited via payout events referencing settlement id BEFORE send; unconfirmed batches resent with same id → no double-pay. confirm via 'settlement' event with tx signature. GET /v1/billing/settlements. CLI: --settle-period/--payout-threshold/--payout-dust-floor (prod 24h/5/0.01; dev 60/0). Banned wallets skipped. 6 tests in tests/test_settlement_loop.py incl. non-leader-never-signs."
},
{
"id": "US-034",
@@ -784,11 +787,12 @@
"Slash/forfeiture events visible in tracker logs and settlement history"
],
"priority": 34,
"status": "open",
"status": "done",
"notes": "No stake deposit — pending balance IS the collateral. Amends ADR-0003 penalty; sampling mechanics unchanged.",
"dependsOn": [
"US-031"
]
],
"completionNotes": "ValidatorProcess(billing=ledger) forfeits entire pending balance on divergence (same cycle as strike). Tracker POST /v1/billing/forfeit (auth stub) for remote validators: forfeit + strike + ban state. Probation wired into _bill_completed: wallets with probationary_jobs_remaining>0 have their share redirected to protocol cut; record_completed_job per request. Penalty math in packages/validator/README.md. 5 tests in tests/test_forfeiture_penalty.py; banned-never-paid asserted in settlement tests."
},
{
"id": "US-035",
@@ -802,12 +806,13 @@
"Works against a 3-tracker hive in the two-machine LAN test setup"
],
"priority": 35,
"status": "open",
"status": "done",
"notes": "CLI dashboard already exists from US-016; this is the web counterpart. Write ops (price config, manual settle) deferred.",
"dependsOn": [
"US-031",
"US-033"
]
],
"completionNotes": "GET /dashboard served from embedded dashboard.html (package-data, no build step) by any tracker. Panels: hive/leader (raft status), nodes+coverage grouped by model, client balances, node pending + protocol cut, settlement history with devnet explorer links, strikes/bans/forfeitures (GET /v1/registry/wallets + snapshot forfeits), RPM stats. 4s auto-refresh via fetch polling. 3 tests in tests/test_dashboard.py."
}
],
"metadata": {