23 Commits

Author SHA1 Message Date
Dobromir Popov
47b243cd98 model loading, dash 2026-07-15 13:55:38 +02:00
Dobromir Popov
2852b1f80b loading more 2026-07-15 12:54:51 +02:00
Dobromir Popov
22f28bd69a fix model load/unload 2026-07-15 12:35:32 +02:00
Dobromir Popov
97e2784b37 node registration fixes 2026-07-15 10:34:41 +02:00
Dobromir Popov
ba7c656364 node metrics 2026-07-14 20:33:02 +02:00
Dobromir Popov
b661590ac7 log window bigger 2026-07-14 17:47:20 +02:00
Dobromir Popov
21e6c86147 fix: let admin placement recover joined nodes 2026-07-14 16:37:42 +02:00
Dobromir Popov
def47f1a42 Merge branch 'master' of https://git.d-popov.com/popov/neuron-tai 2026-07-14 16:11:26 +02:00
Dobromir Popov
8cb00e951f feat: show admin node pool capacity 2026-07-14 16:11:18 +02:00
Dobromir Popov
7b3399760e chore: wrap up completed story metadata 2026-07-14 17:09:04 +03:00
Dobromir Popov
22467f145c merge: distributed performance baseline benchmark 2026-07-14 17:01:08 +03:00
Dobromir Popov
35af1e21de fix: make model placement controls observable 2026-07-14 16:00:37 +02:00
Dobromir Popov
905ea16ce0 feat: complete route session baseline benchmark 2026-07-14 16:55:52 +03:00
Dobromir Popov
348b003d6e fix: restore responsive dashboard panel grid 2026-07-14 15:55:24 +02:00
Dobromir Popov
1e64a5b2b9 new dash update 2026-07-14 15:29:11 +02:00
Dobromir Popov
e2f3ae32b8 feat: let admins manage model placement 2026-07-14 15:16:23 +02:00
Dobromir Popov
29351d6217 chore: ignore local model cache 2026-07-14 14:05:37 +02:00
Dobromir Popov
5c9a2f6c97 dash style fix 2026-07-14 13:29:51 +02:00
Dobromir Popov
13d82f8032 dash, tests 2026-07-14 12:26:10 +02:00
Dobromir Popov
d1a1400db9 Move tracker hive to admin and expand nodes panel.
Give Nodes & coverage full width on overview with inference prices and live speed, and expose model pricing on /v1/models.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-14 12:19:25 +02:00
Dobromir Popov
5d87e81bc9 feat: harden node placement and partial model loading 2026-07-13 21:58:08 +02:00
Dobromir Popov
a6bcc69288 sol mainnet payouts tasks 2026-07-13 18:51:40 +02:00
Dobromir Popov
c938d38031 more docs review 2026-07-13 18:37:07 +02:00
51 changed files with 2141 additions and 236 deletions

View File

@@ -2,9 +2,9 @@
- [Product selling points](product-selling-points.md) — key differentiators and landing page angles for neuron-tai - [Product selling points](product-selling-points.md) — key differentiators and landing page angles for neuron-tai
- [User profile](user-profile.md) — who Dobromir is and how to work with him - [User profile](user-profile.md) — who Dobromir is and how to work with him
- [Project status](project-status.md) — 35/35 stories done; alpha hardening next - [Project status](project-status.md) — US-001…US-035 done; US-036…US-050 in docs/prd.json; alpha hardening + scratch features next
- **Alpha hardening** — `.scratch/alpha-hardening/` (22 issues, ADRs 00160019, [README](../../.scratch/alpha-hardening/README.md), [handoff](../../.scratch/alpha-hardening/handoff.md)) - **Alpha hardening** — `.scratch/alpha-hardening/` (22 issues, ADRs 00160019, [README](../../.scratch/alpha-hardening/README.md), [handoff](../../.scratch/alpha-hardening/handoff.md))
- [Alpha hardening navigation](alpha-hardening-navigation.md) — locked fraud/auth decisions, Bucket-1 order, handoff pointers - [Alpha hardening navigation](alpha-hardening-navigation.md) — locked fraud/auth decisions, Bucket-1 order, handoff pointers
- **Node capability admission** — `.scratch/node-capability-admission/` (P0 plan: generic doctor/real-forward validation, fail-closed readiness, tracker admission gate; [PRD](../../.scratch/node-capability-admission/PRD.md), [README](../../.scratch/node-capability-admission/README.md), ADR-0023) - **Node capability admission** — `.scratch/node-capability-admission/` (P0 plan; [ADR-0023](../../docs/adr/0023-model-agnostic-node-capability-admission.md), [ADR-0026](../../docs/adr/0026-node-assignment-ownership-and-managed-placement.md))
- **Distributed relay performance** — relay `/rpc` requester sockets are persistent per Route Session and Activation Seam as of 2026-07-10; `request_id` remains unique per activation while `X-Meshnet-Session` remains stable for KV state. Next low-risk priorities: persistent direct/loopback HTTP, seam byte/latency telemetry, then trace-driven zstd tuning. - **Distributed relay performance** — relay `/rpc` requester sockets are persistent per Route Session and Activation Seam as of 2026-07-10; `request_id` remains unique per activation while `X-Meshnet-Session` remains stable for KV state. Next low-risk priorities: persistent direct/loopback HTTP, seam byte/latency telemetry, then trace-driven zstd tuning.
- **Distributed GGUF direction** — benchmark-gated native runtime: compare controlled Transformers/safetensors and whole-model llama.cpp lanes before expensive work; ship only for measured speed or model-fit advantage. Public parallelism is contiguous Shards in an Inference Route; concurrency comes from per-node continuous batching across isolated Route Sessions, while tensor/expert collectives stay inside optional trusted composite providers. Native data plane uses versioned Protobuf over long-lived gRPC/HTTP2 seam streams, with existing relay carrying the same opaque frames when needed. llama.cpp/GGML remains the substrate behind a project-owned standalone worker and small pinned fork; vLLM is an optional complete managed provider and concept donor, not a fork. Nakshatra, `prima.cpp`, `llama-gguf`, LiGGUF and historical GPUStack are source/test donors only. Active plan: [README](../../.scratch/distributed-gguf-runtime/README.md), [architecture](../../.scratch/distributed-gguf-runtime/architecture.md), [PRD](../../.scratch/distributed-gguf-runtime/PRD.md), [Ralph backlog](../../.scratch/distributed-gguf-runtime/prd.json). ADR: [0024](../../docs/adr/0024-distributed-gguf-runtime.md). Research: [landscape](../../docs/research/distributed-gguf-landscape.md), [GitHub follow-up](../../docs/research/distributed-gguf-github-followup.md), [vLLM](../../docs/research/vllm-distributed-gguf-assessment.md). - **Distributed GGUF direction** — benchmark-gated native runtime: compare controlled Transformers/safetensors and whole-model llama.cpp lanes before expensive work; ship only for measured speed or model-fit advantage. Public parallelism is contiguous Shards in an Inference Route; concurrency comes from per-node continuous batching across isolated Route Sessions, while tensor/expert collectives stay inside optional trusted composite providers. Native data plane uses versioned Protobuf over long-lived gRPC/HTTP2 seam streams, with existing relay carrying the same opaque frames when needed. llama.cpp/GGML remains the substrate behind a project-owned standalone worker and small pinned fork; vLLM is an optional complete managed provider and concept donor, not a fork. Nakshatra, `prima.cpp`, `llama-gguf`, LiGGUF and historical GPUStack are source/test donors only. Active plan: [README](../../.scratch/distributed-gguf-runtime/README.md), [architecture](../../.scratch/distributed-gguf-runtime/architecture.md), [PRD](../../.scratch/distributed-gguf-runtime/PRD.md), [Ralph backlog](../../.scratch/distributed-gguf-runtime/prd.json). ADR: [0024](../../docs/adr/0024-distributed-gguf-runtime.md). Research: [landscape](../../docs/research/distributed-gguf-landscape.md), [GitHub follow-up](../../docs/research/distributed-gguf-github-followup.md), [vLLM](../../docs/research/vllm-distributed-gguf-assessment.md).

View File

@@ -26,7 +26,7 @@ Active workstream (started 2026-07-04): alpha hardening of the money/trust path.
Both are already migrated into `.scratch/alpha-hardening/prd.json` (AH-021 updated, AH-023 added) and the README index — ready for Ralph to pick up unattended. Both are already migrated into `.scratch/alpha-hardening/prd.json` (AH-021 updated, AH-023 added) and the README index — ready for Ralph to pick up unattended.
**Ralph note:** `scripts/ralph_progress.py` tracks `docs/prd.json` (35/35 done) and does NOT see `.scratch/alpha-hardening/issues/`. No ralph loop is running and no `.ralph-tui/` state exists. `.scratch/alpha-hardening/prd.json` now has 23 stories (AH-001…AH-023); point Ralph at that file for the alpha-hardening branch. Do NOT use `ralph auto --parallel` on server.py-touching issues — 21 and 23 both touch `server.py`/`billing.py`/`audit.py`; if run in the same Ralph pass, run them serially, not in parallel (merge-conflict risk, same lesson as 03/04 previously). **Ralph note:** `scripts/ralph_progress.py` tracks `docs/prd.json` (US-001…US-047; base 35/35 done, friends-test arc 3647 open/in-progress). Alpha hardening uses `.scratch/alpha-hardening/prd.json` (AH-001…AH-023). Point Ralph at the prd.json for the branch you're running.
**Why:** three audits agreed the alpha blockers are unauthenticated gossip (anyone can inject billing events), the free-credit faucet, and ephemeral bans. **Why:** three audits agreed the alpha blockers are unauthenticated gossip (anyone can inject billing events), the free-credit faucet, and ephemeral bans.
**How to apply:** work test-first per issue acceptance criteria; use `.venv`; `cryptography` belongs in node deps (wallet.py imports it — causes many of the 24 "failures" in a fresh env). See [[project-status]] and [[autonomous-work-style]]. **How to apply:** work test-first per issue acceptance criteria; use `.venv`; `cryptography` belongs in node deps (wallet.py imports it — causes many of the 24 "failures" in a fresh env). See [[project-status]] and [[autonomous-work-style]].

View File

@@ -8,7 +8,16 @@ metadata:
# Project Status (2026-07-13) # Project Status (2026-07-13)
> Doc reconciliation 2026-07-13: `docs/PRD.md`, ADR numbering (0024 = distributed GGUF), and `docs/issues/` statuses updated to match `docs/prd.json` (US-001…US-035 done). Post-035 work remains in issues 3648 and `.scratch/`. ## Selected-node model placement (2026-07-14)
- Admin Model placement now opens a node selector for load and release; the control-plane accepts optional `node_id` and targets only that registry assignment. Multi-model serving remains supported through `ADD_SHARD` and `max_loaded_shards`.
- Total node pool resource values are rendered from `/v1/network/map`'s `node.capacity` contract. Route selection remains assignment/capability/throughput/queue based; capacity is used for placement and falls back to tracker defaults only if a node truly omits it.
## Distributed inference performance (2026-07-14)
`DIP-001` is done in `.scratch/distributed-inference-performance/`: the deterministic two-node Route Session stub benchmark covers direct/relay plus cached/stateless prefill and decode. Its JSON and concise summary explicitly attribute model execution, activation encode/decode, compression, connection setup, relay queueing, local HTTP forwarding, and end-to-end seam latency. `PYTHONPATH=packages/node pytest -q tests/test_route_session_benchmark.py` passed (7); the fixture assertion checks output-token identity and connection attempts.
> Doc reconciliation 2026-07-13: `docs/prd.json` tracks US-001…US-050 (048 memory budget, 049 mainnet pilot, 050 Qwen demand placement). ADRs 00250026 added (TAI phase B/C, assignment ownership).
All 35 user stories in docs/prd.json are done (35/35), including the reward-system arc US-030…US-035 completed 2026-07-02: All 35 user stories in docs/prd.json are done (35/35), including the reward-system arc US-030…US-035 completed 2026-07-02:

1
.gitignore vendored
View File

@@ -20,6 +20,7 @@ dist/
!.env.testnet !.env.testnet
.rocm-local/* .rocm-local/*
.pytest-tmp/* .pytest-tmp/*
.cache/
# Local tracker/node sqlite databases (never commit runtime state) # Local tracker/node sqlite databases (never commit runtime state)
*.sqlite *.sqlite

View File

@@ -2,9 +2,9 @@
Pre-release alpha audit + grilling (2026-07-04). Bucket 1 trust-boundary blockers + fraud arc: **done** (16/22 original issues). Bucket 2 (12-15, multi-tracker) and 17 (doc dedup) remain deferred/human-gated — not launch blockers. Pre-release alpha audit + grilling (2026-07-04). Bucket 1 trust-boundary blockers + fraud arc: **done** (16/22 original issues). Bucket 2 (12-15, multi-tracker) and 17 (doc dedup) remain deferred/human-gated — not launch blockers.
**Launch-readiness grilling (2026-07-06):** locked plan is devnet dev/test run now, then real mainnet SOL/USDT for the first cohort — friends (API clients) + hired VPS/VPC hosts (own test infra, not third-party volunteers, stake-free). No new public token; TAI stays dormant per ADR-0002's existing volume/legal gates. Two new issues came out of this session: **Launch-readiness grilling (2026-07-06):** locked plan is devnet dev/test run now, then real mainnet USDT for the first cohort — friends (API clients) + hired VPS/VPC hosts (own test infra, not third-party volunteers; no upfront stake, probation only). No new public token; TAI stays dormant per ADR-0002's existing volume/legal gates. Two new issues came out of this session:
- **[21 — Honest-noise calibration corpus](./issues/21-honest-noise-calibration-corpus.md)** — rescoped from "prod gate" to a hard **alpha-release blocker**. `Status: ready-for-human` — engineering (audit.py raw divergence, tracker dispatch endpoint, SQLite corpus, p99 envelope) done 2026-07-06; blocked on a human running the calibration job against the real hired-VPS fleet before launch. - **[21 — Honest-noise calibration corpus](./issues/21-honest-noise-calibration-corpus.md)** — rescoped from "prod gate" to a hard **alpha-release blocker**. `Status: ready-for-human` — engineering (audit.py raw divergence, tracker dispatch endpoint, SQLite corpus, p99 envelope) done 2026-07-06; blocked on a human running the calibration job against the real hired-VPS fleet before launch. Runbook: [04-toploc-calibration-run](./runbooks/04-toploc-calibration-run.md).
- **[23 — Dynamic HF-benchmarked pricing](./issues/23-dynamic-hf-pricing_completed.md)** — new, high priority but not a release blocker. `Status: done` — engineering complete 2026-07-06 (hf_pricing.py, opt-in daily refresh loop, GET /v1/pricing/hf/history); real `hf_aliases` curation per model is a follow-up human sign-off, not a completion blocker. - **[23 — Dynamic HF-benchmarked pricing](./issues/23-dynamic-hf-pricing_completed.md)** — new, high priority but not a release blocker. `Status: done` — engineering complete 2026-07-06 (hf_pricing.py, opt-in daily refresh loop, GET /v1/pricing/hf/history); real `hf_aliases` curation per model is a follow-up human sign-off, not a completion blocker.
Locked scope: one settlement tracker, open node join, devnet mock-USDT, reputation carries forward → fraud must be bounded. See [ADR-0016](../../docs/adr/0016-alpha-scope-and-known-limitations.md). Locked scope: one settlement tracker, open node join, devnet mock-USDT, reputation carries forward → fraud must be bounded. See [ADR-0016](../../docs/adr/0016-alpha-scope-and-known-limitations.md).
@@ -77,6 +77,7 @@ Locked scope: one settlement tracker, open node join, devnet mock-USDT, reputati
| [17 Duplicate US-020 dedup](./issues/17-doc-duplicate-us020-dedup.md) | | [17 Duplicate US-020 dedup](./issues/17-doc-duplicate-us020-dedup.md) |
| [18 Operational runbooks](./issues/18-doc-operational-runbooks_completed.md) | | [18 Operational runbooks](./issues/18-doc-operational-runbooks_completed.md) |
| [19 Cryptography + test env](./issues/19-doc-cryptography-test-env_completed.md) | | [19 Cryptography + test env](./issues/19-doc-cryptography-test-env_completed.md) |
| [04 TOPLOC calibration run](./runbooks/04-toploc-calibration-run.md) (issue 21 ops) |
| [22 MEMORY + project-status index](./issues/22-doc-memory-project-status_completed.md) (done) | | [22 MEMORY + project-status index](./issues/22-doc-memory-project-status_completed.md) (done) |
| [21 Honest-noise calibration corpus](./issues/21-honest-noise-calibration-corpus.md) (ops; prod gate for audits) | | [21 Honest-noise calibration corpus](./issues/21-honest-noise-calibration-corpus.md) (ops; prod gate for audits) |

View File

@@ -1,6 +1,6 @@
Status: ready-for-human Status: ready-for-human
**BLOCKS ALPHA RELEASE.** Scoped 2026-07-06 during alpha-launch-readiness grilling session — must complete before real-money (mainnet SOL/USDT) traffic goes live for the friends + hired-VPS-host launch. Loose/uncalibrated thresholds + manual admin slash-reversal are the stopgap only until this closes. **BLOCKS ALPHA RELEASE.** Scoped 2026-07-06 during alpha-launch-readiness grilling session — must complete before real-money mainnet USDT traffic goes live for the friends + hired-VPS-host launch. Loose/uncalibrated thresholds + manual admin slash-reversal are the stopgap only until this closes.
**Engineering complete 2026-07-06; blocked on a human running it against the real hired-VPS fleet before launch.** The three code gaps below are closed and unit-tested (see Deliverables), but nothing in a dev session can stand in for actually dispatching the job at real hardware — that step, plus the threshold/FPR write-up that depends on its output, needs an operator with the live fleet. See the validator README's "Honest-noise calibration corpus" section for the operational how-to. **Engineering complete 2026-07-06; blocked on a human running it against the real hired-VPS fleet before launch.** The three code gaps below are closed and unit-tested (see Deliverables), but nothing in a dev session can stand in for actually dispatching the job at real hardware — that step, plus the threshold/FPR write-up that depends on its output, needs an operator with the live fleet. See the validator README's "Honest-noise calibration corpus" section for the operational how-to.
@@ -14,7 +14,7 @@ Per [ADR-0018 consequences](../../docs/adr/0018-fraud-detection-verification-and
Research anchor: `.scratch/alpha-hardening/research-verifiable-inference.md` §8 layer 3 — "collect this first — run identical jobs across the current node fleet to measure the honest divergence envelope before setting thresholds." Research anchor: `.scratch/alpha-hardening/research-verifiable-inference.md` §8 layer 3 — "collect this first — run identical jobs across the current node fleet to measure the honest divergence envelope before setting thresholds."
**Launch context (why this is buildable now, not a research project):** first-launch nodes are hired VPS/VPC hosts under our own direct control (test infrastructure we pay for, not third-party volunteers) — not a long-term topology, but risk-free for calibration purposes since there's no external party to dispute a bad reading. Friends are client-side users of the API in this phase, not node operators. Run the calibration pass against this small, fully-controlled fleet first; hired hosts stay stake-free until it's done, then move to real staking once thresholds derive from their own hardware. **Launch context (why this is buildable now, not a research project):** first-launch nodes are hired VPS/VPC hosts under our own direct control (test infrastructure we pay for, not third-party volunteers) — not a long-term topology, but risk-free for calibration purposes since there's no external party to dispute a bad reading. Friends are client-side users of the API in this phase, not node operators. Run the calibration pass against this small, fully-controlled fleet first; hired hosts stay on probation (no upfront stake) until it's done, then move to paid USDT serving once thresholds derive from their own hardware.
**Current gap (historical — closed 2026-07-06):** the three engineering pieces below were missing when this issue was filed; all are now implemented and unit-tested. Remaining work is the human calibration run on the live hired-VPS fleet. **Current gap (historical — closed 2026-07-06):** the three engineering pieces below were missing when this issue was filed; all are now implemented and unit-tested. Remaining work is the human calibration run on the live hired-VPS fleet.
@@ -36,7 +36,7 @@ Research anchor: `.scratch/alpha-hardening/research-verifiable-inference.md` §8
- [ ] Threshold constants in validator config derived from corpus, not guessed — mechanically ready (`envelope()` returns them) but depends on the real corpus above; not yet wired into `ToplocAuditConfig` as enforced thresholds (deliberately — enforcing unvalidated thresholds would be worse than today's guessed bool). - [ ] Threshold constants in validator config derived from corpus, not guessed — mechanically ready (`envelope()` returns them) but depends on the real corpus above; not yet wired into `ToplocAuditConfig` as enforced thresholds (deliberately — enforcing unvalidated thresholds would be worse than today's guessed bool).
- [ ] False-positive rate estimate documented at chosen thresholds — `envelope()` returns `estimated_false_positive_rate` (in-sample: fraction of the recorded corpus the recommended thresholds would themselves flag); needs the real corpus to be a meaningful number, and should be written up in the runbook once collected. - [ ] False-positive rate estimate documented at chosen thresholds — `envelope()` returns `estimated_false_positive_rate` (in-sample: fraction of the recorded corpus the recommended thresholds would themselves flag); needs the real corpus to be a meaningful number, and should be written up in the runbook once collected.
- [x] README / runbook cross-link: **do not enable production audits** until this issue closes — `packages/validator/README.md` "TOPLOC audit contract" section, updated with the full operational how-to. - [x] README / runbook cross-link: **do not enable production audits** until this issue closes — `packages/validator/README.md` "TOPLOC audit contract" section, updated with the full operational how-to.
- [x] Note in the runbook that this alpha corpus must be re-run once the fleet grows beyond the hired-VPS set (different hardware mix invalidates the envelope) — same README section. - [x] Note in the runbook that this alpha corpus must be re-run once the fleet grows beyond the hired-VPS set (different hardware mix invalidates the envelope) — same README section; [runbook 04](../runbooks/04-toploc-calibration-run.md).
## ADR links ## ADR links

View File

@@ -440,12 +440,12 @@
"Run relevant pytest tests; run the full suite when practical or document why not" "Run relevant pytest tests; run the full suite when practical or document why not"
], ],
"priority": 21, "priority": 21,
"passes": true, "passes": false,
"notes": "Source issue: .scratch/alpha-hardening/issues/21-honest-noise-calibration-corpus.md. BLOCKS ALPHA RELEASE (real-money friends+hired-VPS launch) — rescoped 2026-07-06, no longer a Ralph-skip.", "notes": "Source issue: .scratch/alpha-hardening/issues/21-honest-noise-calibration-corpus.md. BLOCKS ALPHA RELEASE (real-money mainnet USDT). Operator runbook: .scratch/alpha-hardening/runbooks/04-toploc-calibration-run.md",
"dependsOn": [ "dependsOn": [
"AH-006" "AH-006"
], ],
"completionNotes": "Engineering complete and unit-tested (validator audit.py detailed-verify aggregation, tracker calibration.py corpus store, calibration dispatch endpoints). Marked ready-for-human, not done: real corpus collection against the live hired-VPS fleet, and the threshold/FPR write-up that depends on its output, need a human operator — see .ralph-tui/progress.md and packages/validator/README.md." "completionNotes": "Engineering complete and unit-tested. Remaining: human runs POST /v1/calibration/toploc/run on live hired-VPS fleet, records envelope/FPR, wires thresholds — see runbook 04 and packages/validator/README.md."
}, },
{ {
"id": "AH-022", "id": "AH-022",

View File

@@ -0,0 +1,70 @@
# Runbook 04 — Honest-noise TOPLOC calibration (issue 21)
**Status:** engineering complete; **operator action required** before production audit thresholds.
**Blocks:** enabling calibrated TOPLOC thresholds on a mainnet / friends-test fleet (issue 21, ADR-0018).
## When to run
- Before first real-money traffic with audit enforcement enabled.
- Again whenever the fleets **hardware mix** changes materially (new GPU generation, CPU-only nodes added, precision/recipe change per model).
Alpha exception: with a **small hired-VPS-only** fleet, `gate_status.ready` may mean “covers every node we operate today” (`--toploc-calibration-gate-min-hardware-profiles 1`).
## Prerequisites
- Tracker running with billing + registry + `--toploc-calibration-db PATH` (or default under tracker cwd).
- At least one **solo-capable** node per hardware profile you want in the corpus (full model coverage — partial shards are skipped).
- Admin or validator credentials (`Authorization` header or validator service token per ADR-0017).
- Reference validator can replay the fixed calibration prompt (same model/seed as dispatch uses).
## Steps
1. **Register the fleet** — all nodes you intend to pay on mainnet should be up, admitted (NCA when enabled), and solo-serving the calibration model.
2. **Dispatch the job** (admin/validator only):
```bash
curl -X POST "https://<tracker>/v1/calibration/toploc/run" \
-H "Authorization: Bearer <admin-or-validator-token>" \
-H "Content-Type: application/json" \
-d '{}'
```
Partial-shard nodes appear under `skipped_partial_shard_node_ids`. Per-node failures appear under `skipped` with reasons.
3. **Wait for completion** — watch tracker logs and node consoles until every solo-capable node has a row in the corpus.
4. **Fetch results**:
```bash
curl "https://<tracker>/v1/calibration/toploc/results" \
-H "Authorization: Bearer <admin-or-validator-token>"
```
Record:
- `envelope` — p99 metrics + 20% safety margin (recommended tolerances).
- `gate_status.ready` and `gate_status.hardware_profiles`.
- `estimated_false_positive_rate` (in-sample sanity check only).
5. **Write up thresholds** — paste envelope values into operator notes / issue 21 comment. Do **not** wire into production `ToplocAuditConfig` until you have reviewed FPR on this fleet.
6. **Mark issue 21 done** — when corpus covers the launch fleet and thresholds are documented.
## Two-wallet / minimal pilot variant
If your “fleet” is one node machine + one client:
- Run calibration against the **node** profile only (one hardware row is enough for `gate_status` with min profiles = 1).
- Client wallet is irrelevant to calibration — it never serves inference.
## Do not
- Enable stricter production audit thresholds before this completes.
- Reuse a corpus collected on devnet/mock hardware for a different mainnet GPU mix without re-running.
## References
- Issue: `.scratch/alpha-hardening/issues/21-honest-noise-calibration-corpus.md`
- Code: `packages/tracker/meshnet_tracker/calibration.py`, `POST /v1/calibration/toploc/run`
- Validator: `packages/validator/README.md` — TOPLOC audit contract

View File

@@ -12,4 +12,10 @@ Provide an opt-in, admin-only tracker Dashboard Testing tab that dynamically dis
- One active run. - One active run.
- Real inference stays separately environment-gated and excluded from default suites. - Real inference stays separately environment-gated and excluded from default suites.
## Operator workflow
See [`docs/dev/dashboard-test-runner.md`](../../docs/dev/dashboard-test-runner.md)
for launch configuration, default safe suites vs the gated real-inference suite,
and required environment variables.
See `prd.json` for executable Ralph user stories and acceptance criteria. See `prd.json` for executable Ralph user stories and acceptance criteria.

View File

@@ -51,15 +51,16 @@
"uv run pytest tests/test_dashboard.py tests/test_dynamic_routing.py -q passes." "uv run pytest tests/test_dashboard.py tests/test_dynamic_routing.py -q passes."
], ],
"priority": 3, "priority": 3,
"passes": false, "passes": true,
"notes": "Do not reintroduce --enable-test-runner without implementing its CLI argument in US-001.", "notes": "Do not reintroduce --enable-test-runner without implementing its CLI argument in US-001.",
"dependsOn": [ "dependsOn": [
"US-001", "US-001",
"US-002" "US-002"
] ],
"completionNotes": "Completed by agent"
} }
], ],
"metadata": { "metadata": {
"updatedAt": "2026-07-11T17:02:30.520Z" "updatedAt": "2026-07-12T01:58:06.286Z"
} }
} }

View File

@@ -1,4 +1,4 @@
Status: ready-for-agent Status: done (2026-07-14)
# 01 — Baseline and profiling harness # 01 — Baseline and profiling harness
@@ -12,16 +12,15 @@ sizes and connection counts without requiring a real model or external host.
## Acceptance criteria ## Acceptance criteria
- [ ] The harness runs a fixed prompt and fixed generated-token count through a - [x] The harness runs a fixed prompt and fixed generated-token count through a
two-node route in direct and relay modes. two-node route in direct and relay modes.
- [ ] It reports p50/p95 per-token latency, per-hop latency, payload bytes, - [x] It reports p50/p95 per-token latency, per-hop latency, payload bytes,
compression ratio, connection attempts, and queue wait. compression ratio, connection attempts, and queue wait.
- [ ] It distinguishes prefill from decode and cached from stateless mode. - [x] It distinguishes prefill from decode and cached from stateless mode.
- [ ] It emits machine-readable JSON suitable for CI artifacts and a concise - [x] It emits machine-readable JSON suitable for CI artifacts and a concise
human-readable summary. human-readable summary.
- [ ] A test fixture can assert connection attempts and output token identity. - [x] A test fixture can assert connection attempts and output token identity.
## Blocked by ## Blocked by
None - can start immediately. None - completed. Verified with `PYTHONPATH=packages/node pytest -q tests/test_route_session_benchmark.py` (7 passed).

View File

@@ -15,9 +15,10 @@
"Can assert connection count and output token identity" "Can assert connection count and output token identity"
], ],
"priority": 1, "priority": 1,
"passes": false, "passes": true,
"notes": "Source issue: .scratch/distributed-inference-performance/issues/01-baseline-profiling-harness.md", "notes": "Source issue: .scratch/distributed-inference-performance/issues/01-baseline-profiling-harness.md",
"dependsOn": [] "dependsOn": [],
"completionNotes": "Completed by agent"
}, },
{ {
"id": "DIP-002", "id": "DIP-002",
@@ -31,9 +32,12 @@
"Tests cover binary, JSON, timeout, disconnect, cancellation, and cleanup" "Tests cover binary, JSON, timeout, disconnect, cancellation, and cleanup"
], ],
"priority": 2, "priority": 2,
"passes": false, "passes": true,
"notes": "Source issue: .scratch/distributed-inference-performance/issues/02-relay-session-compatibility.md", "notes": "Source issue: .scratch/distributed-inference-performance/issues/02-relay-session-compatibility.md",
"dependsOn": ["DIP-001"] "dependsOn": [
"DIP-001"
],
"completionNotes": "Completed by agent"
}, },
{ {
"id": "DIP-003", "id": "DIP-003",
@@ -47,9 +51,12 @@
"Benchmark shows healthy-session connection count independent of token count" "Benchmark shows healthy-session connection count independent of token count"
], ],
"priority": 3, "priority": 3,
"passes": false, "passes": true,
"notes": "Source issue: .scratch/distributed-inference-performance/issues/03-http-keepalive.md", "notes": "Source issue: .scratch/distributed-inference-performance/issues/03-http-keepalive.md",
"dependsOn": ["DIP-001"] "dependsOn": [
"DIP-001"
],
"completionNotes": "Completed by agent"
}, },
{ {
"id": "DIP-004", "id": "DIP-004",
@@ -63,9 +70,12 @@
"Tests verify cadence and cleanup" "Tests verify cadence and cleanup"
], ],
"priority": 4, "priority": 4,
"passes": false, "passes": true,
"notes": "Source issue: .scratch/distributed-inference-performance/issues/04-seam-telemetry.md", "notes": "Source issue: .scratch/distributed-inference-performance/issues/04-seam-telemetry.md",
"dependsOn": ["DIP-001"] "dependsOn": [
"DIP-001"
],
"completionNotes": "Completed by agent"
}, },
{ {
"id": "DIP-005", "id": "DIP-005",
@@ -79,9 +89,12 @@
"Tests cover compressible, incompressible, threshold, malformed, and legacy bodies" "Tests cover compressible, incompressible, threshold, malformed, and legacy bodies"
], ],
"priority": 5, "priority": 5,
"passes": false, "passes": true,
"notes": "Source issue: .scratch/distributed-inference-performance/issues/05-adaptive-compression.md", "notes": "Source issue: .scratch/distributed-inference-performance/issues/05-adaptive-compression.md",
"dependsOn": ["DIP-001"] "dependsOn": [
"DIP-001"
],
"completionNotes": "Completed by agent"
}, },
{ {
"id": "DIP-006", "id": "DIP-006",
@@ -95,9 +108,12 @@
"Wire and token-output regression tests pass" "Wire and token-output regression tests pass"
], ],
"priority": 6, "priority": 6,
"passes": false, "passes": true,
"notes": "Source issue: .scratch/distributed-inference-performance/issues/06-activation-framing-copies.md", "notes": "Source issue: .scratch/distributed-inference-performance/issues/06-activation-framing-copies.md",
"dependsOn": ["DIP-001"] "dependsOn": [
"DIP-001"
],
"completionNotes": "Completed by agent"
}, },
{ {
"id": "DIP-007", "id": "DIP-007",
@@ -111,9 +127,13 @@
"Tests cover chunking, slow consumers, failure, and legacy peers" "Tests cover chunking, slow consumers, failure, and legacy peers"
], ],
"priority": 7, "priority": 7,
"passes": false, "passes": true,
"notes": "Source issue: .scratch/distributed-inference-performance/issues/07-prefill-backpressure.md", "notes": "Source issue: .scratch/distributed-inference-performance/issues/07-prefill-backpressure.md",
"dependsOn": ["DIP-001", "DIP-004"] "dependsOn": [
"DIP-001",
"DIP-004"
],
"completionNotes": "Completed by agent"
}, },
{ {
"id": "DIP-008", "id": "DIP-008",
@@ -127,9 +147,20 @@
"Gate verifies token identity, session stability, and resource cleanup" "Gate verifies token identity, session stability, and resource cleanup"
], ],
"priority": 8, "priority": 8,
"passes": false, "passes": true,
"notes": "Source issue: .scratch/distributed-inference-performance/issues/08-end-to-end-performance-gate.md", "notes": "Source issue: .scratch/distributed-inference-performance/issues/08-end-to-end-performance-gate.md",
"dependsOn": ["DIP-002", "DIP-003", "DIP-004", "DIP-005", "DIP-006", "DIP-007"] "dependsOn": [
"DIP-002",
"DIP-003",
"DIP-004",
"DIP-005",
"DIP-006",
"DIP-007"
],
"completionNotes": "Completed by agent"
}
],
"metadata": {
"updatedAt": "2026-07-12T02:35:28.752Z"
} }
]
} }

View File

@@ -71,6 +71,8 @@ As an operator and release engineer, I need clear doctor output and opt-in hardw
Add a small generic capability domain object in the node package. `doctor` loads the requested generic model path through the same backend startup uses, executes a bounded real forward at the assigned Shard, and emits the report. Startup gates routable registration on the successful report. Registration carries validated capabilities; the tracker persists/exposes them and filters route candidates at the model/shard/recipe seam. Add a small generic capability domain object in the node package. `doctor` loads the requested generic model path through the same backend startup uses, executes a bounded real forward at the assigned Shard, and emits the report. Startup gates routable registration on the successful report. Registration carries validated capabilities; the tracker persists/exposes them and filters route candidates at the model/shard/recipe seam.
**Assignment ownership:** NCA validates whatever the node loads; it does not assign models. Pinned vs tracker-managed assignment rules are in [ADR-0026](../../docs/adr/0026-node-assignment-ownership-and-managed-placement.md). Demand-driven managed placement (Qwen scratch PRD) may only consume spare capacity; admission applies equally to pinned and managed loads.
The future signed-update contract is represented only by a local manifest version and generic schema in P0. A future Tracker Model Artifact Manifest may be signed data, but Node executable behavior remains supplied by signed Node releases. The future signed-update contract is represented only by a local manifest version and generic schema in P0. A future Tracker Model Artifact Manifest may be signed data, but Node executable behavior remains supplied by signed Node releases.
## Success measures ## Success measures

View File

@@ -7,6 +7,7 @@ This P0 makes a Node prove it can serve its selected Model Artifact and Shard be
## Locked decisions ## Locked decisions
- A Node explicitly asked to serve a Model Preset fails closed when no validated recipe can execute it; it must not register as ready or accept paid inference. - A Node explicitly asked to serve a Model Preset fails closed when no validated recipe can execute it; it must not register as ready or accept paid inference.
- **Assignment ownership:** startup/`--model` loads are **pinned**; tracker-managed demand placement (Qwen US-050) may use **spare capacity only** — [ADR-0026](../../docs/adr/0026-node-assignment-ownership-and-managed-placement.md).
- Default validation covers the selected model/shard only. `meshnet-node doctor --all-recipes` is reserved for support and CI. - Default validation covers the selected model/shard only. `meshnet-node doctor --all-recipes` is reserved for support and CI.
- A Model Preset may have multiple named recipes. Each independently proves a real forward; the Tracker schedules only validated recipes while considering measured performance. - A Model Preset may have multiple named recipes. Each independently proves a real forward; the Tracker schedules only validated recipes while considering measured performance.
- Compatibility schemas are generic. A future Tracker may publish signed, data-only Model Artifact Manifests, but executable recipes arrive only through signed Node releases. - Compatibility schemas are generic. A future Tracker may publish signed, data-only Model Artifact Manifests, but executable recipes arrive only through signed Node releases.

View File

@@ -35,11 +35,12 @@
"Full pytest passes or an exact unrelated blocker is recorded" "Full pytest passes or an exact unrelated blocker is recorded"
], ],
"priority": 2, "priority": 2,
"passes": false, "passes": true,
"notes": "Source issue: .scratch/node-capability-admission/issues/02-doctor-real-forward.md", "notes": "Source issue: .scratch/node-capability-admission/issues/02-doctor-real-forward.md",
"dependsOn": [ "dependsOn": [
"NCA-001" "NCA-001"
] ],
"completionNotes": "Completed by agent"
}, },
{ {
"id": "NCA-003", "id": "NCA-003",
@@ -54,12 +55,13 @@
"Full pytest passes or an exact unrelated blocker is recorded" "Full pytest passes or an exact unrelated blocker is recorded"
], ],
"priority": 3, "priority": 3,
"passes": false, "passes": true,
"notes": "Source issue: .scratch/node-capability-admission/issues/03-fail-closed-startup-admission.md", "notes": "Source issue: .scratch/node-capability-admission/issues/03-fail-closed-startup-admission.md",
"dependsOn": [ "dependsOn": [
"NCA-001", "NCA-001",
"NCA-002" "NCA-002"
] ],
"completionNotes": "Completed by agent"
}, },
{ {
"id": "NCA-004", "id": "NCA-004",
@@ -76,12 +78,13 @@
"Full pytest passes or an exact unrelated blocker is recorded" "Full pytest passes or an exact unrelated blocker is recorded"
], ],
"priority": 4, "priority": 4,
"passes": false, "passes": true,
"notes": "Source issue: .scratch/node-capability-admission/issues/04-tracker-validated-capability-routing.md", "notes": "Source issue: .scratch/node-capability-admission/issues/04-tracker-validated-capability-routing.md",
"dependsOn": [ "dependsOn": [
"NCA-001", "NCA-001",
"NCA-003" "NCA-003"
] ],
"completionNotes": "Completed by agent"
}, },
{ {
"id": "NCA-005", "id": "NCA-005",
@@ -96,15 +99,16 @@
"Full pytest passes or an exact unrelated blocker is recorded" "Full pytest passes or an exact unrelated blocker is recorded"
], ],
"priority": 5, "priority": 5,
"passes": false, "passes": true,
"notes": "Source issue: .scratch/node-capability-admission/issues/05-docs-hardware-lane-contract.md", "notes": "Source issue: .scratch/node-capability-admission/issues/05-docs-hardware-lane-contract.md",
"dependsOn": [ "dependsOn": [
"NCA-002", "NCA-002",
"NCA-004" "NCA-004"
] ],
"completionNotes": "Completed by agent"
} }
], ],
"metadata": { "metadata": {
"updatedAt": "2026-07-11T19:16:52.768Z" "updatedAt": "2026-07-12T01:54:03.030Z"
} }
} }

View File

@@ -46,13 +46,12 @@ model rather than waiting for an operator to request a load.
## Node ownership ## Node ownership
- A startup-assigned `(model, shard range, quantization)` is pinned and never Reconciled with [ADR-0026](../../docs/adr/0026-node-assignment-ownership-and-managed-placement.md) and NCA (ADR-0023):
changed by the tracker.
- Spare capacity on a pinned node, and all capacity on a model-less node, is - A **startup-assigned** `(model, shard range, quantization)` from explicit `--model` or accepted bootstrap assign is **pinned** until the operator restarts.
available for tracker-managed assignments. - **Tracker-managed** assignments (this feature) use only **spare capacity** — model-less nodes or (future, US-048) unused shard slots — and are marked `managed: true`.
- Tracker-added assignments are explicitly marked managed and may be moved or - The tracker may move or remove managed assignments under the safety policy below; it must not retarget a pinned serving assignment to satisfy demand.
removed by the tracker under the safety policy. Runtime UI controls are a - Every assignment, pinned or managed, must pass NCA `doctor` before becoming routable when admission is enabled.
later feature.
## Pricing ## Pricing

View File

@@ -16,12 +16,9 @@
.\.venv\Scripts\meshnet-node.exe start http://192.168.0.179:8081 --model-id Qwen/Qwen2.5-0.5B-Instruct --advertise-host 192.168.0.20 .\.venv\Scripts\meshnet-node.exe start http://192.168.0.179:8081 --model-id Qwen/Qwen2.5-0.5B-Instruct --advertise-host 192.168.0.20
.\.venv\Scripts\meshnet-node.exe start --tracker http://ai.neuron.d-popov.com --model-id Qwen/Qwen2.5-0.5B-Instruct --advertise-host 192.168.0.20 .\.venv\Scripts\meshnet-node.exe start --tracker http://ai.neuron.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct --advertise-host 192.168.0.20
we .\.venv\Scripts\meshnet-node.exe start ` we .\.venv\Scripts\meshnet-node.exe start --tracker http://192.168.0.179:8081 --model Qwen/Qwen2.5-0.5B-Instruct
--tracker http://192.168.0.179:8081 `
--model Qwen/Qwen2.5-0.5B-Instruct `
--advertise-host 192.168.0.20
# trackers: # trackers:
https://meshnet.2.d-popov.com https://meshnet.2.d-popov.com
https://ai.neuron.d-popov.com https://ai.neuron.d-popov.com

View File

@@ -1,4 +1,4 @@
Status: done (base program US-001…US-035 complete; see `docs/prd.json`. Post-035 work lives in `docs/issues/36+` and `.scratch/`. Payment/settlement superseded by ADR-0015; fraud by ADR-0018.) Status: done (US-001…US-035 complete; friends-test arc US-036…US-049 in `docs/prd.json`; US-048/050 tracked. See ADRs 00150018, 0023, 00250026.)
# Distributed Inference Network — PRD # Distributed Inference Network — PRD
@@ -8,10 +8,12 @@ Running large language models requires expensive dedicated hardware that most pe
## Solution ## Solution
A volunteer GPU network where anyone can share their GPU by running a single command and immediately start earning tokens. Nodes each load a shard of a large model; a tracker routes inference requests through the optimal chain of nodes whose shards collectively cover all layers. Developers access the network through an OpenAI-compatible API — a one-line change from any existing LLM integration. Clients pay in **USDT** (alpha: devnet mock-USDT; production: mainnet USDT). Node operators earn USDT payouts from the custodial treasury (ADR-0015); the TAI reward token (ADR-0002) remains deferred. Everything is auto-configured: GPU detection, shard download, wallet creation, and network registration happen automatically on first start. A volunteer GPU network where anyone can share their GPU by running a single command and immediately start earning **USDT**. Nodes each load a shard of a large model; a tracker routes inference requests through the optimal chain of nodes whose shards collectively cover all layers. Developers access the network through an OpenAI-compatible API — a one-line change from any existing LLM integration. Clients pay in **USDT** (alpha: devnet mock-USDT; production: mainnet USDT). Node operators earn USDT payouts from the custodial treasury (ADR-0015); the TAI reward token (ADR-0002) remains deferred. Everything is auto-configured: GPU detection, shard download, wallet creation, and network registration happen automatically on first start.
## User Stories ## User Stories
> **Status (2026-07-13):** Stories below are the original product intent. **Shipped behavior** is in [Implementation Decisions](#implementation-decisions) and ADRs 00150018, 0023, 00250026. Superseded lines are marked inline.
### Node Operator ### Node Operator
1. As a node operator, I want to install the node client with a single command (`pip install meshnet-node`), so that I can start contributing without reading documentation. 1. As a node operator, I want to install the node client with a single command (`pip install meshnet-node`), so that I can start contributing without reading documentation.
@@ -21,10 +23,10 @@ A volunteer GPU network where anyone can share their GPU by running a single com
5. As a node operator, I want my assigned shard to download automatically from HuggingFace on first start, so that I don't have to manually find or download model weights. 5. As a node operator, I want my assigned shard to download automatically from HuggingFace on first start, so that I don't have to manually find or download model weights.
6. As a node operator, I want to seed my shard to other nodes via P2P once I have it, so that new nodes with the same shard assignment don't need to download from HuggingFace. 6. As a node operator, I want to seed my shard to other nodes via P2P once I have it, so that new nodes with the same shard assignment don't need to download from HuggingFace.
7. As a node operator, I want the node client to register with the tracker automatically and begin serving inference requests, so that I start earning as soon as setup is complete. 7. As a node operator, I want the node client to register with the tracker automatically and begin serving inference requests, so that I start earning as soon as setup is complete.
8. As a node operator, I want to see my current node score, shard assignment, and token earnings in the terminal, so that I can verify my node is contributing correctly. 8. As a node operator, I want to see my current node score, shard assignment, and USDT earnings in the terminal, so that I can verify my node is contributing correctly.
9. As a node operator, I want to stake tokens before serving paid inference, so that I have skin in the game and the network can trust my outputs. 9. As a node operator, I want to serve paid inference without upfront stake deposits, with my accrued USDT pending balance as fraud collateral and probation as the anti-sybil cost, so that onboarding stays frictionless. *(Supersedes stake-before-serving; ADR-0015/0018.)*
10. As a node operator, I want my first N jobs to run without earning (probationary period), so that the network can establish trust before paying me. 10. As a node operator, I want my first N jobs to run without earning (probationary period), so that the network can establish trust before paying me.
11. As a node operator, I want to be notified immediately if my stake is slashed due to a fraud detection event, so that I can investigate and fix the issue. 11. As a node operator, I want to be notified when my pending balance is forfeited due to a failed audit, so that I can investigate and fix the issue. *(Supersedes stake slash; ADR-0018 forfeiture.)*
12. As a node operator, I want to receive a strike and a warning before being banned, so that accidental failures don't immediately end my participation. 12. As a node operator, I want to receive a strike and a warning before being banned, so that accidental failures don't immediately end my participation.
13. As a node operator, I want to be automatically reassigned to a different shard when the tracker determines another shard is more in demand, so that my hardware is always optimally used. 13. As a node operator, I want to be automatically reassigned to a different shard when the tracker determines another shard is more in demand, so that my hardware is always optimally used.
14. As a node operator, I want the node client to reconnect automatically if the tracker is temporarily unavailable, so that transient network issues don't stop me from earning. 14. As a node operator, I want the node client to reconnect automatically if the tracker is temporarily unavailable, so that transient network issues don't stop me from earning.
@@ -34,8 +36,8 @@ A volunteer GPU network where anyone can share their GPU by running a single com
### Client Developer ### Client Developer
17. As a client developer, I want to send `POST /v1/chat/completions` requests to the gateway in the same format as the OpenAI API, so that I can switch to the network with a one-line code change. 17. As a client developer, I want to send `POST /v1/chat/completions` requests to the gateway in the same format as the OpenAI API, so that I can switch to the network with a one-line code change.
18. As a client developer, I want to authenticate with an API key funded by SOL or USDC, so that I never need to acquire or hold our native token. 18. As a client developer, I want to authenticate with an API key funded by USDT, so that I never need to acquire or hold our native token. *(ADR-0015.)*
19. As a client developer, I want to top up my API key balance by sending SOL or USDC to a Solana address, so that payment is simple and familiar. 19. As a client developer, I want to top up my API key balance by sending USDT to the treasury Solana address, so that payment is simple and familiar. *(ADR-0015; wallet binding US-039/041.)*
20. As a client developer, I want to see a per-request cost estimate before sending a request, so that I can budget inference costs accurately. 20. As a client developer, I want to see a per-request cost estimate before sending a request, so that I can budget inference costs accurately.
21. As a client developer, I want to receive streaming responses (`text/event-stream`) in OpenAI-compatible format, so that I can build low-latency user experiences. 21. As a client developer, I want to receive streaming responses (`text/event-stream`) in OpenAI-compatible format, so that I can build low-latency user experiences.
22. As a client developer, I want `GET /v1/models` to return the list of available model presets on the network, so that I know what I can request. 22. As a client developer, I want `GET /v1/models` to return the list of available model presets on the network, so that I know what I can request.
@@ -46,15 +48,15 @@ A volunteer GPU network where anyone can share their GPU by running a single com
### End User (via a client app) ### End User (via a client app)
27. As an end user, I want to buy SOL on any exchange and use it to pay for inference, so that I don't need to understand blockchain technology to use the service. 27. As an end user, I want to buy USDT on an exchange and use it to pay for inference via Solana, so that I don't need deep crypto knowledge to use the service. *(Clients pay USDT; SOL is only for network fees if they self-custody.)*
28. As an end user, I want responses of equivalent quality to centralised providers, so that I don't have to trade quality for cost savings. 28. As an end user, I want responses of equivalent quality to centralised providers, so that I don't have to trade quality for cost savings.
29. As an end user, I want low latency on first token, so that conversational applications feel responsive. 29. As an end user, I want low latency on first token, so that conversational applications feel responsive.
### Validator ### Validator
30. As a validator, I want to automatically re-run a random sample (~5%) of completed inference requests on a reference node, so that I can detect nodes returning fraudulent outputs. 30. As a validator, I want to automatically re-run a random sample (~5%) of completed inference requests on a reference node with TOPLOC activation verification, so that I can detect nodes returning fraudulent outputs. *(ADR-0018.)*
31. As a validator, I want to submit a fraud proof on-chain when a node's output diverges beyond tolerance, so that the slash event is recorded trustlessly. 31. As a validator, I want the tracker to record forfeiture and strikes when an audit fails, so that penalties are applied consistently. *(Supersedes on-chain fraud proof in alpha; ADR-0018.)*
32. As a validator, I want to earn a reward for each successful fraud detection, so that there is an economic incentive to run validation. 32. As a validator, I want economic incentive to run validation, so that fraud detection is not purely altruistic. *(Validator reward share deferred; forfeiture to protocol cut today.)*
### Network (tracker / system) ### Network (tracker / system)
@@ -62,7 +64,7 @@ A volunteer GPU network where anyone can share their GPU by running a single com
34. As the tracker, I want to rebalance shard assignments across nodes when demand for a model preset changes, so that the network always covers the most-requested models. 34. As the tracker, I want to rebalance shard assignments across nodes when demand for a model preset changes, so that the network always covers the most-requested models.
35. As the tracker, I want to instruct a node to download a new shard when no other node covers it, so that model preset coverage is maintained automatically. 35. As the tracker, I want to instruct a node to download a new shard when no other node covers it, so that model preset coverage is maintained automatically.
36. As the tracker, I want to exclude banned wallets from route selection, so that fraudulent nodes cannot serve paid inference. 36. As the tracker, I want to exclude banned wallets from route selection, so that fraudulent nodes cannot serve paid inference.
37. As the tracker, I want to read stake, slash, strike, and ban state exclusively from Solana smart contracts, so that I cannot manipulate payouts even with full control of the routing layer. 37. As the tracker, I want strike and ban state persisted in the registry and enforced on route selection, so that fraudulent wallets cannot serve paid inference. *(Supersedes on-chain-only stake/slash registry; ADR-0018; on-chain deferred per ADR-0007/0015.)*
38. As the network, I want new model presets to be addable by submitting a HuggingFace model ID and shard count, so that the set of available models can grow without code changes. 38. As the network, I want new model presets to be addable by submitting a HuggingFace model ID and shard count, so that the set of available models can grow without code changes.
## Implementation Decisions ## Implementation Decisions
@@ -73,7 +75,7 @@ The codebase is organized as a Python monorepo with the following top-level pack
- `packages/gateway` — OpenAI-compatible HTTP gateway and route orchestration - `packages/gateway` — OpenAI-compatible HTTP gateway and route orchestration
- `packages/tracker` — centralized tracker service (node registry, scoring, route selection) - `packages/tracker` — centralized tracker service (node registry, scoring, route selection)
- `packages/sdk``meshnet` Python SDK wrapping gateway + wallet controls - `packages/sdk``meshnet` Python SDK wrapping gateway + wallet controls
- `packages/contracts` — Solana L2 smart contracts (stake, slash, strike, ban, settlement) - `packages/contracts` — Solana adapter boundary (custodial USDT treasury, local registry prototype)
- `packages/p2p` — P2P gossip layer and shard swarm seeding - `packages/p2p` — P2P gossip layer and shard swarm seeding
### Inference engine (ADR-0001; native GGUF path ADR-0024) ### Inference engine (ADR-0001; native GGUF path ADR-0024)
@@ -92,13 +94,13 @@ The gateway receives a client request, asks the tracker for an inference route (
7. Begin accepting inference connections 7. Begin accepting inference connections
### Payment flow (ADR-0015 supersedes ADR-0002 settlement mechanics) ### Payment flow (ADR-0015 supersedes ADR-0002 settlement mechanics)
Clients pre-fund an API key with USDT. The tracker meters each request against the off-chain ledger. Periodic settlement batches USDT payouts from the custodial treasury to node operators proportional to work units. Fraud penalties forfeit pending balance (ADR-0018); strike/ban state persists in the tracker registry. TAI token emission remains deferred (ADR-0002 roadmap). Clients pre-fund an API key with USDT. The tracker meters each request against the off-chain ledger. Periodic settlement batches USDT payouts from the custodial treasury to node operators proportional to work units (default: every 24 h or when pending ≥ 5 USDT). Fraud penalties forfeit pending balance (ADR-0018); strike/ban state persists in the tracker registry. TAI reward accrual is deferred — see ADR-0025 for reserved-mint / off-chain phase B/C; ADR-0002 roadmap for public listing.
### Fraud detection (ADR-0018; historical ADR-0003) ### Fraud detection (ADR-0018; historical ADR-0003)
Validators re-run ~5% of completed requests with TOPLOC activation verification. Caught cheaters forfeit pending balance and receive strikes; three strikes bans the wallet. Probation (first N unpaid jobs) remains the anti-sybil re-entry cost. Validators re-run ~5% of completed requests with TOPLOC activation verification. Caught cheaters forfeit pending balance and receive strikes; three strikes bans the wallet. Probation (first N unpaid jobs) remains the anti-sybil re-entry cost.
### Tracker architecture (ADR-0004) ### Tracker architecture (ADR-0004)
Centralized tracker service (HTTP + WebSocket) for fast routing. Nodes gossip state via a lightweight P2P layer so the node client can discover routes during tracker outages. Solana is the authoritative source of truth for all incentive-relevant state. Centralized tracker service (HTTP + WebSocket) for fast routing. Nodes gossip state via a lightweight P2P layer so the node client can discover routes during tracker outages. **Alpha:** strike/ban/forfeiture state lives in the tracker registry (ADR-0018); USDT settlement via custodial treasury (ADR-0015). On-chain programs deferred (ADR-0007).
### Shard distribution (ADR-0005) ### Shard distribution (ADR-0005)
Shards are identified by `(model_preset, shard_index)`. On assignment, the node downloads the shard layers from HuggingFace using `huggingface_hub`. Once downloaded, the node joins the P2P shard swarm and seeds to other nodes requesting the same shard. Popular shards propagate entirely via P2P; cold shards fall back to HuggingFace. Shards are identified by `(model_preset, shard_index)`. On assignment, the node downloads the shard layers from HuggingFace using `huggingface_hub`. Once downloaded, the node joins the P2P shard swarm and seeds to other nodes requesting the same shard. Popular shards propagate entirely via P2P; cold shards fall back to HuggingFace.
@@ -115,9 +117,9 @@ The gateway exposes OpenAI-compatible endpoints (`/v1/chat/completions`, `/v1/mo
**Per-component seams:** **Per-component seams:**
- **Tracker**: given a set of registered nodes with known shard coverage and node scores, assert `select_route(model_preset)` returns an optimal ordered list of node endpoints. - **Tracker**: given a set of registered nodes with known shard coverage and node scores, assert `select_route(model_preset)` returns an optimal ordered list of node endpoints.
- **Node shard serving**: given an activation tensor for the node's layer range, assert the output tensor shape and dtype are correct. - **Node shard serving**: given an activation tensor for the node's layer range, assert the output tensor shape and dtype are correct.
- **Fraud detection**: given a validator that re-runs a known-bad node response, assert a slash transaction is submitted on-chain with correct attribution. - **Fraud detection**: given a validator that re-runs a known-bad node response, assert strike/forfeiture state updates with correct attribution (ADR-0018; on-chain slash deferred).
- **Shard swarm**: given a node that has a shard, assert a second node with the same assignment downloads it via P2P rather than HuggingFace. - **Shard swarm**: given a node that has a shard, assert a second node with the same assignment downloads it via P2P rather than HuggingFace.
- **Payment settlement**: given a completed inference session with known compute attribution, assert token balances change by the expected amounts after epoch settlement. - **Payment settlement**: given a completed inference session with known compute attribution, assert USDT ledger balances change by the expected amounts after epoch settlement (ADR-0015).
## Out of Scope ## Out of Scope
@@ -135,4 +137,4 @@ The gateway exposes OpenAI-compatible endpoints (`/v1/chat/completions`, `/v1/mo
- The `meshnet-node` CLI is the primary viral growth vector. Every friction point in the install/start sequence costs node operators. The startup sequence must complete without any manual configuration on a machine with a CUDA-capable GPU. - The `meshnet-node` CLI is the primary viral growth vector. Every friction point in the install/start sequence costs node operators. The startup sequence must complete without any manual configuration on a machine with a CUDA-capable GPU.
- The name "meshnet" is a working name. The actual package and token names are TBD. - The name "meshnet" is a working name. The actual package and token names are TBD.
- The Solana L2 chain selection (vs Base/Arbitrum) is not yet finalised — both are cheap, EVM-compatible fallbacks. The contracts package should abstract chain-specific details. - The Solana L2 chain selection (vs Base/Arbitrum) is not yet finalised — both are cheap, EVM-compatible fallbacks. The contracts package should abstract chain-specific details.
- The probationary period length (N free jobs) and slash amounts are economic parameters that will need tuning once the network has real usage data. Hardcode sensible defaults; make them on-chain governable. - The probationary period length (N free jobs) and forfeiture amounts are economic parameters that will need tuning once the network has real usage data. Hardcode sensible defaults; governance TBD (ADR-0018).

View File

@@ -123,3 +123,11 @@ Rejected. gRPC/HTTP2 already provides mature streaming, flow control, deadlines,
4. Real two-machine execution using both Shards. 4. Real two-machine execution using both Shards.
5. End-to-end performance/fit advantage over the current distributed route. 5. End-to-end performance/fit advantage over the current distributed route.
6. Separate Qwen3-family architecture certification. 6. Separate Qwen3-family architecture certification.
## Relationship to US-042 (whole-model GGUF shortcut)
[US-042](../issues/42-gguf-llamacpp-node-backend.md) **phase C** ships first: a node with enough RAM serves a **full** GGUF via llama.cpp on a single-hop Inference Route using the existing HTTP activation seam and PyTorch-era tracker integration. That is intentionally small and does not require this ADR's gRPC worker or llama.cpp patch stack.
This ADR's track starts only after **DGR-001** (controlled safetensors-vs-GGUF benchmark) shows a meaningful speed or fit benefit. Then implement the native worker (DGR-002+) — which subsumes US-042 direction A (layer-range GGUF + boundary tensors) if the benchmark warrants it.
Do not run US-042 phase C and DGR-008+ in parallel on the same node backend without an explicit integration plan; phase C uses llama-cpp-python (or equivalent) whole-model path; ADR-0024 uses the standalone C++ worker.

View File

@@ -0,0 +1,52 @@
# ADR-0025: TAI reserved mint and off-chain accrual (phase B/C)
## Status: Accepted
## Context
ADR-0015 chose **USDT-direct custodial settlement** for alpha and near-term production. Clients pay USDT; nodes receive batched USDT SPL payouts. ADR-0002's TAI reward token, revenue-backed floor, and open-market listing gates remain the long-term design but are **not** the live payment path.
The owner wants TAI to exist without the cost and legal surface of a public launch: no AMM, no open listing, no client-facing TAI, no on-chain stake machinery.
## Decision
### Phase B — Reserved mainnet mint (cheap, optional early)
- Create a fixed-supply TAI SPL mint on **mainnet** when treasury work happens (~0.002 SOL).
- Entire initial supply sits in a **team-controlled** wallet (same custody posture as the USDT treasury today).
- **No public emission, no market, no client UX.** Mint exists for name reservation and future programmatic rewards only.
- Document mint address in operator config; do not advertise to users.
### Phase C — Off-chain TAI accrual alongside USDT (before automatic on-chain TAI payouts)
- Extend the billing ledger with **`tai_pending[wallet]`** accrued from completed inference work using a simple rule (e.g. USDT node share × configurable TAI-per-USDT rate, or fixed TAI per work unit).
- TAI accrual is **display-only + ledger-persisted** initially; nodes see pending TAI in dashboard/CLI.
- **Clients never pay or hold TAI.** USDT remains the only client-facing asset.
- Optional manual or scheduled **TAI SPL batch transfers** from the team wallet (same batching pattern as USDT `send_payouts`) — operator-triggered until automatic emission is justified by volume.
- The existing **10% protocol USDT cut** continues to accumulate as future TAI liquidity per ADR-0015/0002; do not redirect it until a deliberate liquidity event.
### Explicit non-goals (this ADR)
- Open-market listing, AMM, or DEX liquidity
- Buyback floor endpoint or backing-price oracle (ADR-0002 machinery)
- On-chain stake deposits or slash contracts
- Paying clients rebates or accepting TAI for inference
- Replacing USDT node payouts with TAI-only payouts before volume gates in ADR-0002 pass
## Relation to ADR-0002 listing gates
Public TAI listing stays gated on **$50k cumulative USDT volume** and **25+ nodes / 15+ wallets**. Phase B/C may proceed **below** those gates because they do not create a public market — only reserved supply and off-chain accounting.
Securities review remains required before any **public** distribution or listing; off-chain accrual to hired/known operators with manual SPL transfers is an operator discretion, not a product promise.
## Consequences
- USDT mainnet pilot (two-wallet setup) is unblocked without TAI complexity.
- TAI narrative is preserved at minimal cost (mint + ledger column + optional manual transfers).
- Automatic TAI emission can later reuse the US-033 settlement loop shape with a second mint and separate pending bucket.
- Dashboard and APIs must label TAI balances as **non-withdrawable** until an on-chain payout batch confirms.
## Verification
- USDT settlement tests remain authoritative for production payouts (`tests/test_settlement_loop.py`).
- When phase C lands: ledger tests for `tai_pending` accrual, idempotent gossip replication, and optional TAI batch payout adapter tests mirroring USDT.

View File

@@ -0,0 +1,51 @@
# ADR-0026: Node assignment ownership — pinned startup vs managed demand placement
## Status: Accepted
## Context
Three features define how a node gets its `(model, shard range, recipe/quantization)`:
1. **ADR-0011 / US-013** — tracker suggests a gap from coverage map on startup or auto-join.
2. **Node capability admission (ADR-0023 / NCA)** — a node must pass `doctor` + real forward before becoming routable; startup-assigned work is validated, not blindly trusted.
3. **Qwen demand placement** (`.scratch/qwen3.6-27b-demand-placement/`) — tracker deploys a model when chat demand appears and spare capacity exists.
These looked contradictory: NCA and the Qwen PRD both say startup assignments are "pinned," while demand placement wants the tracker to assign models dynamically.
## Decision
### Three assignment tiers
| Tier | How it is created | Mutable by tracker? | Admission |
|---|---|---|---|
| **Operator-initiated** | Node starts with explicit `--model` / shard flags | **No** — pinned until operator restarts or explicitly reloads | Must pass NCA `doctor` before routable |
| **Network bootstrap** | `/v1/network/assign` or `/v1/nodes/assign` on first join (ADR-0011) | **No** for the active loaded shard — treated as operator-equivalent once accepted at startup | Must pass NCA before routable |
| **Tracker-managed** | Demand-driven placement (Qwen PRD) on spare capacity | **Yes** — marked `managed: true`; subject to cooldown / safety policy | Must pass NCA for the new assignment before routable |
### Spare capacity rule (unifies NCA + Qwen)
- A nodes **active** `(model, shard, recipe)` from startup is **pinned** — the tracker does not silently retarget a serving node to a different model.
- **Spare capacity** — memory/slots not holding the pinned assignment, or a node registered without a model — may receive **tracker-managed** assignments to satisfy demand.
- Until multi-shard runtime exists (US-048), “spare capacity” effectively means **model-less nodes** or nodes explicitly registered for managed placement; do not overload a single-shard node with a second assignment.
### Demand placement interaction
- First chat request for an unrouted model queues **demand**; leader tracker may assign **managed** nodes only when eligible spare capacity exists (Qwen PRD).
- Until complete coverage + validated recipes exist, return retryable `503 model_loading` with coverage metadata.
- Managed assignments must not evict pinned assignments on other nodes without the Qwen safety policy (≥3 copies, 1.5× demand multiplier, cooldown).
### NCA is not optional for any tier
Regardless of assignment source, registration carries **validated capability** only after `doctor` succeeds. The tracker excludes nodes with absent, stale, or failed capability reports (ADR-0023).
## Consequences
- NCA and Qwen demand placement are complementary: NCA gates *quality*; demand placement gates *where new coverage comes from*.
- US-048 (multi-shard slots) extends spare capacity — until then, demand placement primarily targets nodes that join without `--model`.
- Rebalance / dropout relocation (US-013, US-048) applies to **coverage gaps**, not retroactive retargeting of pinned nodes for demand convenience.
## Verification
- NCA tests: unvalidated nodes never routed.
- Demand-placement tests (when implemented): managed flag set; pinned nodes unchanged.
- Documented in Qwen scratch PRD and NCA README cross-links.

View File

@@ -0,0 +1,10 @@
Status: superseded
# Superseded — renumbered to US-048
This issue slot was a duplicate of tracker-node-hardening (US-020). Memory budget / shard slots / dropout relocation work lives at:
- **Issue:** [48-memory-budget-shard-slots-and-dropout-relocation.md](./48-memory-budget-shard-slots-and-dropout-relocation.md)
- **PRD:** `docs/prd.json``US-048`
Do not implement from this file.

View File

@@ -1,9 +1,16 @@
# US-042 — GGUF/llama.cpp node backend # US-042 — GGUF/llama.cpp node backend
Status: planned Status: planned
Priority: High (whole-model GGUF shortcut; distributed path in [ADR-0024](../adr/0024-distributed-gguf-runtime.md)) Priority: High (unlocks DeepSeek-V4-Flash on volunteer hardware — the pool's core value)
Stage: Draft design Stage: Draft design
## Goal
Run **DeepSeek-V4-Flash** as the first real large-model target on volunteer
hardware via GGUF/llama.cpp. This epic is no longer GLM-oriented: the initial
objective is to prove that DeepSeek-V4-Flash can load and serve correctly on
consumer/unified-memory nodes, then expand from there.
## Context ## Context
The node backend is transformers-only (`model_backend.py` The node backend is transformers-only (`model_backend.py`

View File

@@ -86,10 +86,10 @@ What exists already (build on it, don't duplicate):
- [ ] Two-machine test: machine A (tracker + node, holds full snapshot) serves - [ ] Two-machine test: machine A (tracker + node, holds full snapshot) serves
layers 0k; machine B joins with no model and receives **only** the files layers 0k; machine B joins with no model and receives **only** the files
for its assigned range from A — nothing fetched from HF for its assigned range from A — nothing fetched from HF
- [ ] Machine B's resident memory scales with its shard size, not model size - [x] Machine B's resident memory scales with its shard size, not model size
- [ ] Checksums verified end-to-end; corrupted transfer falls back cleanly - [x] Checksums verified end-to-end; corrupted transfer falls back cleanly
- [x] Single-node/full-model flows unchanged - [x] Single-node/full-model flows unchanged
- [ ] `python -m pytest` passes from repo root - [x] `python -m pytest` passes from repo root
## Implementation notes ## Implementation notes
@@ -98,6 +98,13 @@ What exists already (build on it, don't duplicate):
`full_url`; HuggingFace remains fallback-only, and when it is used the node `full_url`; HuggingFace remains fallback-only, and when it is used the node
computes `allow_patterns` from the repo's remote SafeTensors index so it computes `allow_patterns` from the repo's remote SafeTensors index so it
stays layer-filtered even without tracker-cached files. Remaining hard half stays layer-filtered even without tracker-cached files. Remaining hard half
is true partial model materialization: the backend can prefer a downloaded is partial model materialization: the backend can prefer a downloaded
local model directory, but Transformers still needs a `meta`-device load local model directory, but Transformers still needs a `meta`-device load
path that materializes only assigned layers. path that materializes only assigned layers.
- 2026-07-13: Partial LOAD implemented. `_load_partial_model_from_snapshot` builds
on `meta` via `init_empty_weights`, materializes only layer-scoped checkpoint
tensors, and finalizes device placement without copying unmaterialized meta
weights (`_finalize_active_shard_modules_on_device`). Tests cover memory
scaling (`test_partial_snapshot_resident_weight_numel_scales_with_shard`)
and real-torch meta-vs-materialized counts. Remaining: live two-machine LAN
verification.

View File

@@ -0,0 +1,101 @@
Status: ready-for-agent
# US-049 — Mainnet USDT cutover: two-wallet pilot checklist
Priority: High (first real-money friends test)
Stage: Operator runbook + config verification
## Goal
Move from **Solana devnet + mock-USDT** to **Solana mainnet + real USDT** for a minimal pilot: **one client wallet** (inference payer) and **one node-operator wallet** (payout recipient). Treasury holds USDT and pays SOL fees. TAI stays phase B/C per [ADR-0025](../adr/0025-tai-off-chain-accrual-and-reserved-mint.md).
## Wallet roles
| Role | Keypair | On-chain use |
|---|---|---|
| **Treasury** | Operator `treasury-keypair.json` (multisig when ready) | Holds USDT float + SOL for fees; sends batched node payouts |
| **Client** | Your inference-user wallet | SPL USDT → treasury; bound to API key for ledger credit |
| **Node** | Your node-operator wallet | Receives USDT payout batches from treasury |
The node process already creates/loads a Solana wallet at startup; the client wallet is bound via accounts/dashboard (`POST /v1/wallet/register` or US-041 flows).
## Pre-flight (devnet smoke — do not skip)
- [ ] Tracker with `--solana-rpc-url https://api.devnet.solana.com`, mock mint, treasury keypair
- [ ] `--settle-period 60 --payout-threshold 0` — confirm payout appears on dashboard **Settlement history** with explorer link
- [ ] Run `python -m pytest tests/test_settlement_loop.py -q` — includes prod 24h/5 USDT gate tests
- [ ] One inference request → node pending → settlement tx → node wallet balance increases
## Mainnet config change (config-only cutover)
Replace devnet values; **no code deploy required** beyond what is already on the branch.
```bash
# Example — use your mainnet RPC provider
meshnet-tracker start \
--solana-rpc-url https://api.mainnet-beta.solana.com \
--usdt-mint EPjFWdd5AufqSSqeM2qN1xzybapC8G4wEGGkZwyTDt1v \
--treasury-keypair ~/.config/solana/meshnet-treasury-mainnet.json \
--settle-period 86400 \
--payout-threshold 5.0 \
--payout-dust-floor 0.01 \
--starting-credit 0 \
--devnet-topup 0
```
| Flag | Devnet (test) | Mainnet (pilot) |
|---|---|---|
| RPC | `api.devnet.solana.com` | Mainnet RPC (Helius/QuickNode/etc.) |
| `--usdt-mint` | mock mint from `devnet_setup.py` | Real USDT mint (`EPjF…` on Solana) |
| `--settle-period` | `60` (fast verify) | `86400` (24 h) |
| `--payout-threshold` | `0` | `5.0` USDT |
| `--starting-credit` | `1.0` (optional) | `0` |
| `--devnet-topup` | `1.0` | `0` |
## Treasury funding
- [ ] Fund treasury wallet with **SOL** for fees (~0.10.5 SOL to start; ~$0.001 per daily batch + ~$0.30 once per new node ATA)
- [ ] Fund treasury with **USDT** for node payouts (your float — e.g. first week of expected node earnings)
- [ ] Client wallet holds USDT; send a test SPL transfer to treasury ATA; confirm deposit watcher credits API key within one poll interval
## Two-wallet pilot steps
1. **Start tracker** on mainnet config above (single settlement tracker per ADR-0016).
2. **Client path:** register account → create API key → bind **client wallet** → deposit USDT to treasury → verify ledger balance on dashboard.
3. **Node path:** start `meshnet-node` with **node wallet** keypair → register → serve inference.
4. **Inference:** client sends `POST /v1/chat/completions` with API key; verify 402 before deposit, success after.
5. **Accrual:** confirm node **pending USDT** on dashboard rises; client balance debits.
6. **Payout (24 h):** wait for `--settle-period` **or** temporarily lower to `300` for first pilot verification, then restore `86400`.
7. **Threshold path:** alternatively, accumulate ≥ `5` USDT pending in one session to trigger immediate batch without waiting 24 h.
8. **Verify on-chain:** settlement history shows mainnet tx signature; node wallet USDT ATA balance increased; pending zeroed.
## Safety checks
- [ ] `--devnet-topup 0` — no faucet on mainnet
- [ ] `--starting-credit 0` — no free inference credit
- [ ] Treasury keypair not committed to git; file mode 600
- [ ] Plan multisig migration before large float ([alpha runbook](../../.scratch/alpha-hardening/runbooks/02-treasury-key-rotation.md) intent)
- [ ] Issue **21** (TOPLOC calibration) before production audit thresholds on untrusted nodes — runbook: [04-toploc-calibration-run](../../.scratch/alpha-hardening/runbooks/04-toploc-calibration-run.md)
## Cost estimate (this pilot)
- **SOL fees:** pennies per day at 1 batch × 1 node
- **USDT:** whatever clients deposit and nodes earn (treasury is passthrough for payouts)
- **TAI mint (optional, ADR-0025 phase B):** ~0.002 SOL one-time — defer if not needed for pilot
## Acceptance criteria
- [ ] Devnet checklist completed once
- [ ] Mainnet tracker serves dashboard; billing enabled
- [ ] Client wallet deposit → ledger credit → inference → debit
- [ ] Node wallet receives ≥1 confirmed USDT payout batch on mainnet
- [ ] 24 h period enforced: sub-threshold pending not paid before period (covered by `tests/test_settlement_loop.py`)
- [ ] ≥5 USDT pending triggers payout without waiting full period (covered by tests)
- [ ] Rollback documented: switch RPC + mint back to devnet if needed
## Related
- ADR-0015 (USDT custodial settlement)
- ADR-0025 (TAI reserved mint / off-chain accrual — not blocking this pilot)
- US-033 / US-032 (settlement + deposits)
- `scripts/devnet_setup.py` (devnet only)

View File

@@ -0,0 +1,15 @@
Status: in-design
# US-050 — Qwen3.6-27B demand-driven managed placement
> Full spec: [.scratch/qwen3.6-27b-demand-placement/PRD.md](../../.scratch/qwen3.6-27b-demand-placement/PRD.md)
> Assignment rules: [ADR-0026](../adr/0026-node-assignment-ownership-and-managed-placement.md)
> Admission: [ADR-0023](../adr/0023-model-agnostic-node-capability-admission.md)
## Summary
Deploy `Qwen/Qwen3.6-27B` when chat demand appears and **spare** fleet capacity exists. Startup `--model` assignments stay **pinned**; tracker-managed loads fill gaps on model-less or (future US-048) unused slot capacity only.
## Acceptance criteria
See scratch PRD and `docs/prd.json` US-050.

View File

@@ -1,6 +1,6 @@
{ {
"name": "Distributed Inference Network", "name": "Distributed Inference Network",
"description": "Build a distributed inference network with node, gateway, tracker, SDK, contracts, and P2P shard distribution components from the grill session PRD.", "description": "Distributed inference network: base program US-001…US-035 complete; friends-test arc US-036…US-049; capacity/placement US-048/050. Scratch features (alpha hardening AH-001…AH-025, NCA, distributed GGUF, Qwen demand) have separate prd.json files under .scratch/.",
"branchName": "ralph/distributed-inference-network", "branchName": "ralph/distributed-inference-network",
"userStories": [ "userStories": [
{ {
@@ -814,10 +814,307 @@
"US-033" "US-033"
], ],
"completionNotes": "GET /dashboard served from embedded dashboard.html (package-data, no build step) by any tracker. Panels: hive/leader (raft status), nodes+coverage grouped by model, client balances, node pending + protocol cut, settlement history with devnet explorer links, strikes/bans/forfeitures (GET /v1/registry/wallets + snapshot forfeits), RPM stats. 4s auto-refresh via fetch polling. 3 tests in tests/test_dashboard.py." "completionNotes": "GET /dashboard served from embedded dashboard.html (package-data, no build step) by any tracker. Panels: hive/leader (raft status), nodes+coverage grouped by model, client balances, node pending + protocol cut, settlement history with devnet explorer links, strikes/bans/forfeitures (GET /v1/registry/wallets + snapshot forfeits), RPM stats. 4s auto-refresh via fetch polling. 3 tests in tests/test_dashboard.py."
},
{
"id": "US-036",
"title": "36 — Streamed chat completions over the relay RPC path",
"description": "Public NAT deployments proxy every chat request tracker → relay → head node. Implement true multi-frame SSE streaming over the relay WebSocket so clients see live tokens and relayed streams bill through the same SSE accounting loop as direct proxy streams. Inter-node /forward activation hops stay single-frame (ADR-0014).",
"acceptanceCriteria": [
"stream: true chat via relay delivers SSE chunks incrementally (≥2 distinct frame arrivals before [DONE])",
"Relayed streamed request records nonzero billed tokens and node work credit",
"Non-streamed relayed requests and /forward binary hops behave exactly as before (single frame, body_base64 intact)",
"Legacy single-frame response from an old node is accepted as terminal",
"Idle stream (no frame for 120 s) returns 504 and cleans up the relay-side queue",
"Extend tests/test_gossip_and_relay.py alongside test_relay_rpc_round_trips_http_request_to_peer",
"python -m pytest passes from repo root"
],
"priority": 36,
"status": "needs-review",
"notes": "Source issue: docs/issues/36-relay-streamed-chat.md. Implemented via _stream_relayed_frames in server.py; verify on public NAT relay before friends-test.",
"dependsOn": [
"US-029",
"US-031"
],
"completionNotes": "Multi-frame relay-http-response protocol; node relay_bridge line-by-line SSE emit; relay server per-request asyncio.Queue; tracker _stream_relayed_frames with SSE billing parity. Client mid-stream disconnect accepted limitation for alpha."
},
{
"id": "US-037",
"title": "37 — Concurrent request handling in the node relay bridge",
"description": "RelayHttpBridge currently handles relay-http-request envelopes serially, blocking up to 300 s per request. Off-LAN a node can be head of one route and downstream hop of another — overlapping routes through a shared node break. Dispatch on a bounded ThreadPoolExecutor (default 8, configurable) with per-frame WS send locking compatible with US-036 streaming.",
"acceptanceCriteria": [
"While one relayed request is in flight, a second relay-http-request to the same node completes without waiting for the first",
"Responses are correctly matched by request_id when they complete out of order",
"More than N simultaneous requests queue and all eventually complete; thread count never exceeds N workers",
"Bridge survives a relay reconnect with workers still in flight (no crash, no deadlock; orphaned responses dropped)",
"Configurable via meshnet-node start --relay-concurrency N (env MESHNET_RELAY_CONCURRENCY)",
"Extend tests/test_gossip_and_relay.py",
"python -m pytest passes from repo root"
],
"priority": 37,
"status": "done",
"notes": "Source issue: docs/issues/37-relay-bridge-concurrency.md. Critical for public friends-test; blocks concurrent head + hop on same node.",
"dependsOn": [
"US-036"
],
"completionNotes": "ThreadPoolExecutor dispatch in relay_bridge.py; per-frame WS send lock; test_relay_bridge_serves_concurrent_requests; --relay-concurrency CLI flag sets MESHNET_RELAY_CONCURRENCY."
},
{
"id": "US-038",
"title": "38 — Tracker cluster join via a single seed peer",
"description": "Tracker cluster membership is static today — a newcomer configured with only one existing peer is never learned by the rest of the hive and quorum math diverges. A joining tracker configured with any one live seed announces via hive-HMAC-signed POST /v1/cluster/join; membership changes replicate through the Raft log and persist across restarts.",
"acceptanceCriteria": [
"Start trackers A+B; start C with only A as seed → within one election timeout A, B, and C report the same 3-peer membership on GET /v1/cluster/peers, and a value proposed on C commits on A and B",
"Join without a valid hive signature is rejected with 403; join to a follower is forwarded to the leader",
"Restarting C with its seed offline rejoins from persisted membership",
"Standalone tracker (no seeds) behaves exactly as today",
"python -m pytest passes from repo root"
],
"priority": 38,
"status": "open",
"notes": "Source issue: docs/issues/38-tracker-seed-join.md. Out of scope: peer removal, joint consensus, automatic seed retry.",
"dependsOn": [
"US-013",
"US-017"
]
},
{
"id": "US-039",
"title": "39 — Caller Credit granted once per account; chat requires account keys",
"description": "DEFAULT_STARTING_CREDIT=0 and no grant path leaves every fresh public tracker request at 402. Grant Caller Credit once per account on first API key creation via deterministic event id caller-credit-{account_id}; chat on accounts-enabled trackers requires a real active sk-mesh- key (401 for invented bearers).",
"acceptanceCriteria": [
"Fresh account → first key → key has --starting-credit balance; chat succeeds",
"Second key on the same account → no additional credit",
"Revoke-and-recreate keys → still no additional credit (deterministic event id)",
"Random bearer string on an accounts-enabled tracker → 401, never 402/free work",
"Tracker without accounts store: gate behavior unchanged",
"--starting-credit 0 disables the grant entirely (mainnet posture)",
"python -m pytest passes from repo root"
],
"priority": 39,
"status": "done",
"notes": "Source issue: docs/issues/39-caller-credit-account-keys.md. Critical for friends-test inference.",
"dependsOn": [
"US-031",
"US-035"
],
"completionNotes": "Caller credit granted once per account on first API key via deterministic event id; tests/test_accounts.py covers grant, revoke, and invented-bearer rejection."
},
{
"id": "US-040",
"title": "40 — Devnet top-up button on the dashboard",
"description": "After Caller Credit (US-039) is spent, devnet friends need a dashboard faucet refill without on-chain USDT deposits. POST /v1/account/topup (session-authenticated) credits a configured fixed amount per click; flag off returns 404 and hides the button.",
"acceptanceCriteria": [
"Flag off: endpoint 404s, dashboard shows no top-up button",
"Flag on: logged-in user tops up own key, balance rises by exactly N",
"Topping up another account's key → 403",
"python -m pytest passes from repo root"
],
"priority": 40,
"status": "done",
"notes": "Source issue: docs/issues/40-devnet-dashboard-topup.md. Mainnet deployments set --devnet-topup 0.",
"dependsOn": [
"US-039"
],
"completionNotes": "POST /v1/account/topup with session auth and flag gating; tests/test_accounts.py covers flag off/on, own-account credit, and cross-account 403."
},
{
"id": "US-041",
"title": "41 — Account wallet: browser-extension signing, in-browser generation, export-only",
"description": "Accounts need a visible wallet for deposit attribution without the tracker ever holding private keys. Dashboard integrates Solana wallet-adapter connect+nonce proof, or in-browser keypair generation with one-time export; no private-key import endpoint.",
"acceptanceCriteria": [
"Connect-extension flow stores a verified pubkey (rejects unsigned/mismatched nonce proofs)",
"Generate flow: pubkey lands on the account; private key is never sent to the tracker, export works",
"No endpoint or UI accepts a private key",
"Deposits to the shown address credit the account's keys via the existing watcher",
"Address visible on the account panel after either flow",
"python -m pytest passes from repo root"
],
"priority": 41,
"status": "open",
"notes": "Source issue: docs/issues/41-account-wallet-keypair.md. Not needed for devnet friends test; needed before mainnet.",
"dependsOn": [
"US-032",
"US-039"
]
},
{
"id": "US-042",
"title": "42 — GGUF/llama.cpp node backend (phase C whole-model first)",
"description": "Node backend is transformers-only today; large MoE models on consumer hardware require GGUF via llama.cpp. Phase C: whole-model GGUF nodes (single-hop routes) first; partial-layer distributed GGUF deferred to ADR-0024. Also: GGUF catalog entries, Strix Halo/Vulkan hardware detection, download dir applies to GGUF files.",
"acceptanceCriteria": [
"A node with --gguf <repo-or-path> --quant IQ3_XXS serves /v1/chat/completions via llama.cpp with GPU offload where available",
"Tracker treats it as a full-coverage node (single-hop routes, billing works)",
"Streamed responses work through the tracker proxy and the relay (US-036)",
"python -m pytest passes from repo root (llama.cpp behind an optional extra)"
],
"priority": 42,
"status": "in-design",
"notes": "Source issue: docs/issues/42-gguf-llamacpp-node-backend.md. Phase C before ADR-0024 distributed worker; see runtime sequencing in issue file.",
"dependsOn": [
"US-036"
]
},
{
"id": "US-043",
"title": "43 — Dashboard model search and model cards",
"description": "Dashboard lacks model-centric discovery. Add server-side HF search proxy merged with tracker presets and live coverage; model cards show architecture, coverage gaps, pricing, memory per quant, and a request-this-model action. Featured section driven by CURATED_MODELS including GGUF once US-042 lands.",
"acceptanceCriteria": [
"Searching a HF repo id or free text returns results without the browser calling HF directly",
"A served model's card shows live coverage and a working chat-now state",
"An unserved model's card shows the request action and estimated memory per quant",
"python -m pytest passes from repo root"
],
"priority": 43,
"status": "open",
"notes": "Source issue: docs/issues/43-dashboard-model-search-cards.md. Post-deploy polish.",
"dependsOn": [
"US-035"
]
},
{
"id": "US-044",
"title": "44 — Tracker as model-file source; nodes download only their shard",
"description": "Second nodes joining a fleet today download entire HF snapshots even for small shard assignments. Tracker --models-dir advertises layer-scoped safetensors subsets; nodes race tracker/peer sources before HF allow_patterns fallback. Hard half remaining: meta-device partial model materialization so resident memory scales with shard size, not full model size.",
"acceptanceCriteria": [
"Tracker started with --models-dir / MESHNET_MODELS_DIR advertises local model-file sources in assignment responses",
"Tracker serves a tar stream (or per-file API) containing only safetensors files for the assigned layer range plus config/tokenizer/index metadata",
"Node downloader tries exact-shard peers, then tracker/peer file subsets, then HF snapshot_download with allow_patterns — never silently full-repo when layer index is available",
"Two-machine test: machine B receives only its assigned range from machine A — nothing fetched from HF",
"Machine B resident memory scales with its shard size, not model size",
"Checksums verified end-to-end; corrupted transfer falls back cleanly",
"Single-node/full-model flows unchanged",
"python -m pytest passes from repo root"
],
"priority": 44,
"status": "in-progress",
"notes": "Source issue: docs/issues/44-tracker-shard-source-partial-download.md. Download path and partial LOAD implemented; live two-machine LAN verification remains.",
"dependsOn": [
"US-004",
"US-012"
],
"completionNotes": "Tracker models-dir indexing, layer-scoped tar stream, HF allow_patterns client-side from remote index, per-file download API with retries, symlink dereference in tar writers. Partial LOAD via init_empty_weights + layer-scoped safetensors materialization; memory-scaling and checksum fallback tests pass. Remaining: live two-machine test (machine B receives only assigned files from A, no HF)."
},
{
"id": "US-045",
"title": "45 — Dual-rate billing: separate input and output token prices",
"description": "Ledger has one price_per_1k_tokens and stream vs non-stream paths disagree on input vs output counting. Charge both input and output tokens at separate rates per model; HF pricing refresher applies 80% of each marketplace side separately.",
"acceptanceCriteria": [
"Streamed and non-streamed requests for the same exchange bill the same split (input charged in both)",
"A model with asymmetric provider rates bills input and output differently; usage_for / billing events expose the split",
"Old persisted billing events replay byte-identically (balances unchanged)",
"HF refresh sets both rates from the marketplace row, not the average",
"Spend cap (--max-charge-per-request) uses the dual rates",
"python -m pytest passes from repo root"
],
"priority": 45,
"status": "in-progress",
"notes": "Source issue: docs/issues/45-dual-rate-billing.md. Billing correctness before friends test.",
"dependsOn": [
"US-031"
]
},
{
"id": "US-046",
"title": "46 — Tracker .env awareness + first-node auto-join bootstrap",
"description": "Fresh trackers return 503 on auto-join because deployability ignores the joining caller's hardware, and meshnet-tracker ignores .env MESHNET_DOWNLOAD_DIR. Fix empty-registry bootstrap, tracker env loading parity with node CLI, models-dir fallback chain, and tar dereference for HF symlink snapshots.",
"acceptanceCriteria": [
"Fresh tracker (empty registry) + caller with enough memory for a recommended preset → /v1/network/assign returns 200 with model_sources populated when tracker holds a local snapshot",
"Fresh tracker + caller too small for any recommended preset → still 503",
"meshnet-tracker start in a directory with .env setting MESHNET_DOWNLOAD_DIR serves /v1/model-files/download from that dir with no extra flags",
"Explicit --models-dir and MESHNET_MODELS_DIR still take precedence",
"python -m pytest passes from repo root"
],
"priority": 46,
"status": "needs-review",
"notes": "Source issue: docs/issues/46-tracker-env-and-first-node-autojoin.md. Verified live 2026-07-06.",
"dependsOn": [
"US-044"
],
"completionNotes": "Empty-registry synthesizes caller as candidate node; tracker loads .env; models_dir falls back MESHNET_MODELS_DIR → MESHNET_DOWNLOAD_DIR; tar dereference=True. Pytest passes aside from known port-7000 env conflict."
},
{
"id": "US-047",
"title": "47 — Tracker-first model downloads: visibility, sane timeouts, RAM-based sizing",
"description": "Explicit --model startup should skip pointless auto-join; tracker/peer sources preferred over HF with visible progress and 30 s socket timeouts; client abort during tar stream logs one line; CPU nodes size shards from RAM not phantom GPU VRAM; per-file downloads for robustness over fragile multi-GB tar streams.",
"acceptanceCriteria": [
"Node started with explicit --model never queries /v1/network/assign and never prints auto-join unavailable",
"Tracker/peer model source preferred; HF only when no source, all sources fail, or --tracker-source-disabled",
"Tracker-source downloads print progress every 512 MB and print exception + URL on failure",
"A ≥2 s read stall no longer aborts a tracker model-source download (30 s socket timeout)",
"Client disconnect during /v1/model-files/download logs one line on the tracker, no traceback",
"CPU node with big RAM gets a RAM-sized shard: sizing paths ignore VRAM unless device=cuda",
"Live two-machine retest: Windows node downloads from tracker at LAN speed with RAM-sized shard assignment",
"python -m pytest passes from repo root"
],
"priority": 47,
"status": "in-progress",
"notes": "Source issue: docs/issues/47-model-source-download-visibility.md. Engineering largely complete 2026-07-06; live two-machine retest pending.",
"dependsOn": [
"US-044",
"US-046"
],
"completionNotes": "Skip auto-join when model explicit; sequential source try with progress; 30 s model-source timeout; assignment_vram_mb for CPU; per-file /v1/model-files/download with manifest and retries. Remaining: live Windows two-machine retest."
},
{
"id": "US-049",
"title": "49 — Mainnet USDT cutover: two-wallet pilot checklist",
"description": "Operator runbook to move from Solana devnet + mock-USDT to mainnet + real USDT for a minimal pilot: one client wallet (deposits USDT, pays for inference) and one node wallet (receives batched payouts). Treasury holds USDT float and SOL for fees. TAI deferred per ADR-0025.",
"acceptanceCriteria": [
"Devnet smoke completed: settlement loop pays with --settle-period 60 and mock mint",
"python -m pytest tests/test_settlement_loop.py -q passes (includes 24 h / 5 USDT gate tests)",
"Mainnet tracker configured: real USDT mint, --starting-credit 0, --devnet-topup 0, --settle-period 86400",
"Client wallet deposit credits API key ledger; inference debits balance",
"Node wallet receives at least one confirmed mainnet USDT payout batch",
"Sub-threshold pending not paid before 24 h; ≥5 USDT pending triggers immediate payout"
],
"priority": 49,
"status": "open",
"notes": "Source issue: docs/issues/49-mainnet-usdt-cutover-two-wallet-pilot.md. ADR-0025 covers optional TAI mint; not blocking this pilot.",
"dependsOn": [
"US-032",
"US-033",
"US-039"
]
},
{
"id": "US-048",
"title": "48 — Memory budget, shard slots, and dropout relocation hardening",
"description": "Harden the capacity contract around US-013 coverage-first assignment: enforce memory budget and max_loaded_shards in rebalance/dropout relocation, and decide whether one node may host multiple concurrent shard backends or max_loaded_shards remains metadata until runtime support lands.",
"acceptanceCriteria": [
"Assignment/rebalance never exceeds memory budget or max_loaded_shards",
"Dropout test restores full coverage without violating capacity limits",
"CLI --memory and --max-shards reflected in registration payload",
"python -m pytest tests/test_tracker_routing.py tests/test_node_startup.py passes"
],
"priority": 48,
"status": "open",
"notes": "Source issue: docs/issues/48-memory-budget-shard-slots-and-dropout-relocation.md. Renumbered from duplicate slot 20. Enables spare shard slots for ADR-0026 managed placement.",
"dependsOn": [
"US-013"
]
},
{
"id": "US-050",
"title": "50 — Qwen3.6-27B demand-driven managed placement",
"description": "Offer pinned Qwen/Qwen3.6-27B as a recommended text-only chat model. Valid chat requests prove demand; when spare fleet capacity exists, the tracker assigns managed nodes to reach complete coverage. Pinned startup assignments remain immutable per ADR-0026; NCA admission required before routable.",
"acceptanceCriteria": [
"First valid request for an uncovered variant queues demand and returns 503 model_loading until complete validated coverage exists",
"Managed assignments use only spare capacity and carry managed: true",
"Pinned startup assignments are never silently retargeted",
"Optional quantization field (bfloat16/int8/nf4) with coverage-vote UI semantics per scratch PRD",
"python -m pytest passes from repo root"
],
"priority": 50,
"status": "in-design",
"notes": "Source: .scratch/qwen3.6-27b-demand-placement/PRD.md and docs/issues/50-qwen3.6-27b-demand-placement.md. Reconciled with ADR-0026 and ADR-0023.",
"dependsOn": [
"US-035",
"US-048"
]
} }
], ],
"metadata": { "metadata": {
"updatedAt": "2026-07-01T00:00:00.000Z", "updatedAt": "2026-07-13T17:00:00.000Z",
"statusVocabulary": { "statusVocabulary": {
"open": "Not started", "open": "Not started",
"in-design": "Decisions pending before implementation can begin", "in-design": "Decisions pending before implementation can begin",

View File

@@ -36,6 +36,12 @@ def _load_env_file(path: Path) -> None:
os.environ[key] = value os.environ[key] = value
def _apply_relay_concurrency_flag(value: int | None) -> None:
"""Expose relay bridge worker cap via CLI (env MESHNET_RELAY_CONCURRENCY)."""
if value is not None:
os.environ["MESHNET_RELAY_CONCURRENCY"] = str(max(1, value))
def _load_env_defaults() -> None: def _load_env_defaults() -> None:
"""Load machine-specific, local, and user-level node env defaults.""" """Load machine-specific, local, and user-level node env defaults."""
machine = socket.gethostname().strip() machine = socket.gethostname().strip()
@@ -189,6 +195,8 @@ def _cmd_default(args) -> int:
if getattr(args, "cpu", False): if getattr(args, "cpu", False):
overrides["force_cpu"] = True overrides["force_cpu"] = True
_apply_relay_concurrency_flag(getattr(args, "relay_concurrency", None))
if overrides: if overrides:
cfg = merge_cli_overrides(cfg, **overrides) cfg = merge_cli_overrides(cfg, **overrides)
@@ -349,6 +357,8 @@ def _cmd_start(args) -> int:
if getattr(args, "node_name", None): if getattr(args, "node_name", None):
cfg["node_name"] = args.node_name cfg["node_name"] = args.node_name
_apply_relay_concurrency_flag(getattr(args, "relay_concurrency", None))
# Legacy start: just run without the dashboard (keep original blocking loop) # Legacy start: just run without the dashboard (keep original blocking loop)
from .startup import run_startup from .startup import run_startup
@@ -433,6 +443,8 @@ def main() -> None:
help="Set PyTorch inter-op CPU worker threads") help="Set PyTorch inter-op CPU worker threads")
parser.add_argument("--cpu", action="store_true", parser.add_argument("--cpu", action="store_true",
help="Force CPU inference even when a GPU is available") help="Force CPU inference even when a GPU is available")
parser.add_argument("--relay-concurrency", type=int, metavar="N",
help="Max concurrent relay-http-request workers (env MESHNET_RELAY_CONCURRENCY)")
parser.add_argument("--debug", action="store_true", help="Enable verbose node debug logging") parser.add_argument("--debug", action="store_true", help="Enable verbose node debug logging")
parser.add_argument("--no-tui", action="store_true", help="Plain-text output (no rich dashboard)") parser.add_argument("--no-tui", action="store_true", help="Plain-text output (no rich dashboard)")
parser.add_argument("--compact", action="store_true", help="Single-line status output") parser.add_argument("--compact", action="store_true", help="Single-line status output")
@@ -510,6 +522,8 @@ def main() -> None:
help="Set PyTorch inter-op CPU worker threads") help="Set PyTorch inter-op CPU worker threads")
start_cmd.add_argument("--cpu", action="store_true", start_cmd.add_argument("--cpu", action="store_true",
help="Force CPU inference even when a GPU is available") help="Force CPU inference even when a GPU is available")
start_cmd.add_argument("--relay-concurrency", type=int, metavar="N",
help="Max concurrent relay-http-request workers (env MESHNET_RELAY_CONCURRENCY)")
start_cmd.add_argument("--debug", action="store_true", help="Enable verbose node debug logging") start_cmd.add_argument("--debug", action="store_true", help="Enable verbose node debug logging")
start_cmd.add_argument("--tracker-source-disabled", action="store_true", start_cmd.add_argument("--tracker-source-disabled", action="store_true",
help="Skip tracker/peer model-file sources and download from HuggingFace directly") help="Skip tracker/peer model-file sources and download from HuggingFace directly")

View File

@@ -2,6 +2,7 @@
import json import json
import os import os
import shutil
import subprocess import subprocess
import time import time
@@ -183,6 +184,17 @@ def with_forced_cpu(hw: dict) -> dict:
return forced return forced
def _with_model_drive(profile: dict) -> dict:
"""Attach free space for the default model cache drive to tracker diagnostics."""
try:
cache_root = os.path.expanduser("~/.cache/meshnet/shards")
profile["model_drive_free_bytes"] = shutil.disk_usage(os.path.expanduser("~")).free
profile["model_drive_path"] = cache_root
except OSError:
pass
return profile
def detect_hardware() -> dict: def detect_hardware() -> dict:
"""Detect GPU model and available VRAM. Returns hardware profile dict.""" """Detect GPU model and available VRAM. Returns hardware profile dict."""
ram_mb = _detect_ram_mb() ram_mb = _detect_ram_mb()
@@ -208,23 +220,23 @@ def detect_hardware() -> dict:
} }
if torch_gpu is not None and torch_gpu.get("gcn_arch"): if torch_gpu is not None and torch_gpu.get("gcn_arch"):
profile["gcn_arch"] = torch_gpu["gcn_arch"] profile["gcn_arch"] = torch_gpu["gcn_arch"]
return profile return _with_model_drive(profile)
except ImportError: except ImportError:
pass pass
torch_inventory = _gpu_inventory_profile(torch_gpu, ram_mb) torch_inventory = _gpu_inventory_profile(torch_gpu, ram_mb)
if torch_inventory is not None: if torch_inventory is not None:
return torch_inventory return _with_model_drive(torch_inventory)
nvidia_gpu = _gpu_inventory_profile(_detect_nvidia_smi_gpu_memory(), ram_mb) nvidia_gpu = _gpu_inventory_profile(_detect_nvidia_smi_gpu_memory(), ram_mb)
if nvidia_gpu is not None: if nvidia_gpu is not None:
return nvidia_gpu return _with_model_drive(nvidia_gpu)
windows_gpu = _gpu_inventory_profile(_detect_windows_gpu_memory(), ram_mb) windows_gpu = _gpu_inventory_profile(_detect_windows_gpu_memory(), ram_mb)
if windows_gpu is not None: if windows_gpu is not None:
return windows_gpu return _with_model_drive(windows_gpu)
return { return _with_model_drive({
"device": "cpu", "device": "cpu",
"gpu_name": None, "gpu_name": None,
"vram_mb": 0, "vram_mb": 0,
@@ -232,7 +244,7 @@ def detect_hardware() -> dict:
"shared_vram_mb": 0, "shared_vram_mb": 0,
"ram_mb": ram_mb, "ram_mb": ram_mb,
"cuda_available": False, "cuda_available": False,
} })
def benchmark_throughput_checked(device_str: str = "cpu") -> tuple[float, bool, str | None]: def benchmark_throughput_checked(device_str: str = "cpu") -> tuple[float, bool, str | None]:

View File

@@ -899,10 +899,39 @@ def _load_partial_model_from_snapshot(
dtype=dtype, dtype=dtype,
) )
_finalize_active_shard_modules_on_device(model, shard_start, shard_end, device)
return model
def _finalize_active_shard_modules_on_device(
model: Any, shard_start: int, shard_end: int, device: Any
) -> None:
"""Place active shard modules on device without copying unmaterialized meta weights."""
for module in _active_modules_for_shard(model, shard_start, shard_end): for module in _active_modules_for_shard(model, shard_start, shard_end):
parameters = getattr(module, "parameters", None)
if not callable(parameters):
if hasattr(module, "to"): if hasattr(module, "to"):
module.to(device) module.to(device)
return model continue
params = list(parameters(recurse=True))
buffers_fn = getattr(module, "buffers", None)
buffers = list(buffers_fn(recurse=True)) if callable(buffers_fn) else []
tensors = params + buffers
if not tensors:
if hasattr(module, "to"):
module.to(device)
continue
if all(tensor.device.type == "meta" for tensor in tensors):
to_empty = getattr(module, "to_empty", None)
if callable(to_empty):
to_empty(device)
continue
if all(tensor.device.type != "meta" for tensor in tensors):
if hasattr(module, "to"):
module.to(device)
continue
# Partially materialized: set_module_tensor_to_device already placed loaded
# weights on the target device; leave remaining meta parameters untouched.
def _model_load_plan( def _model_load_plan(

View File

@@ -44,6 +44,7 @@ class SeamSample:
cache_mode: CacheMode cache_mode: CacheMode
model_ms: float model_ms: float
encode_ms: float encode_ms: float
activation_decode_ms: float
framing_ms: float framing_ms: float
metadata_ms: float metadata_ms: float
copy_allocation_ms: float copy_allocation_ms: float
@@ -52,6 +53,7 @@ class SeamSample:
decompression_ms: float decompression_ms: float
connection_setup_ms: float connection_setup_ms: float
queue_wait_ms: float queue_wait_ms: float
local_http_forwarding_ms: float
transport_ms: float transport_ms: float
seam_latency_ms: float seam_latency_ms: float
payload_bytes: int payload_bytes: int
@@ -120,6 +122,10 @@ def _summary(samples: list[SeamSample]) -> dict[str, float | int]:
"compression_cpu_ms": round( "compression_cpu_ms": round(
sum(sample.compression_ms + sample.decompression_ms for sample in samples), 4 sum(sample.compression_ms + sample.decompression_ms for sample in samples), 4
), ),
"model_execution_ms": round(sum(sample.model_ms for sample in samples), 4),
"activation_encoding_ms": round(sum(sample.encode_ms for sample in samples), 4),
"activation_decoding_ms": round(sum(sample.activation_decode_ms for sample in samples), 4),
"local_http_forwarding_ms": round(sum(sample.local_http_forwarding_ms for sample in samples), 4),
"peak_buffered_bytes": max((sample.copy_allocation_bytes for sample in samples), default=0), "peak_buffered_bytes": max((sample.copy_allocation_bytes for sample in samples), default=0),
} }
@@ -159,6 +165,7 @@ class _StubTransport:
queue_wait_ms = 0.0 if self.mode == "direct" else 0.18 + (0.05 if token_index is not None and token_index % 2 else 0.0) queue_wait_ms = 0.0 if self.mode == "direct" else 0.18 + (0.05 if token_index is not None and token_index % 2 else 0.0)
model_ms = 1.6 if phase == "prefill" else 0.45 model_ms = 1.6 if phase == "prefill" else 0.45
encode_ms = 0.16 if phase == "prefill" else 0.06 encode_ms = 0.16 if phase == "prefill" else 0.06
activation_decode_ms = 0.055 if phase == "prefill" else 0.02
# Keep framing/metadata/copy costs explicit rather than hiding them in # Keep framing/metadata/copy costs explicit rather than hiding them in
# serialization or transport time. The stub owns one binary frame and # serialization or transport time. The stub owns one binary frame and
# one response body per hop; no base64 body is modeled. # one response body per hop; no base64 body is modeled.
@@ -168,20 +175,26 @@ class _StubTransport:
copy_allocation_bytes = wire_bytes + payload_bytes copy_allocation_bytes = wire_bytes + payload_bytes
compression_ms = 0.09 if self.scenario.compression else 0.0 compression_ms = 0.09 if self.scenario.compression else 0.0
decompression_ms = 0.07 if self.scenario.compression else 0.0 decompression_ms = 0.07 if self.scenario.compression else 0.0
# Both routes finish by forwarding the decoded activation to the local
# tail-node HTTP handler; relay adds its own queue before that hop.
local_http_forwarding_ms = 0.11 if self.mode == "direct" else 0.16
transport_ms = (0.32 if self.mode == "direct" else 0.61) + wire_bytes / 100_000 transport_ms = (0.32 if self.mode == "direct" else 0.61) + wire_bytes / 100_000
seam_latency_ms = round( seam_latency_ms = round(
model_ms + encode_ms + framing_ms + metadata_ms + copy_allocation_ms model_ms + encode_ms + activation_decode_ms + framing_ms + metadata_ms + copy_allocation_ms
+ compression_ms + decompression_ms + connection_setup_ms + queue_wait_ms + transport_ms, + compression_ms + decompression_ms + connection_setup_ms + queue_wait_ms + transport_ms
+ local_http_forwarding_ms,
4, 4,
) )
return SeamSample( return SeamSample(
phase=phase, token_index=token_index, session_id=self.session_id, phase=phase, token_index=token_index, session_id=self.session_id,
activation_id=f"benchmark-activation-{self._activation_count}", seam="head->tail", mode=self.mode, activation_id=f"benchmark-activation-{self._activation_count}", seam="head->tail", mode=self.mode,
cache_mode=self.cache_mode, model_ms=model_ms, encode_ms=encode_ms, cache_mode=self.cache_mode, model_ms=model_ms, encode_ms=encode_ms,
activation_decode_ms=activation_decode_ms,
framing_ms=framing_ms, metadata_ms=metadata_ms, framing_ms=framing_ms, metadata_ms=metadata_ms,
copy_allocation_ms=copy_allocation_ms, copy_allocation_bytes=copy_allocation_bytes, copy_allocation_ms=copy_allocation_ms, copy_allocation_bytes=copy_allocation_bytes,
compression_ms=compression_ms, decompression_ms=decompression_ms, compression_ms=compression_ms, decompression_ms=decompression_ms,
connection_setup_ms=connection_setup_ms, queue_wait_ms=queue_wait_ms, connection_setup_ms=connection_setup_ms, queue_wait_ms=queue_wait_ms,
local_http_forwarding_ms=local_http_forwarding_ms,
transport_ms=round(transport_ms, 4), seam_latency_ms=seam_latency_ms, transport_ms=round(transport_ms, 4), seam_latency_ms=seam_latency_ms,
payload_bytes=payload_bytes, wire_bytes=wire_bytes, payload_bytes=payload_bytes, wire_bytes=wire_bytes,
compression_ratio=round(payload_bytes / wire_bytes, 4), connection_attempted=connection_attempted, compression_ratio=round(payload_bytes / wire_bytes, 4), connection_attempted=connection_attempted,
@@ -329,9 +342,10 @@ def run_real_model_lan_benchmark(url: str, *, model: str, timeout: float = 120.0
sample = SeamSample( sample = SeamSample(
phase="decode", token_index=0, session_id=session_id, activation_id="lan-activation-1", phase="decode", token_index=0, session_id=session_id, activation_id="lan-activation-1",
seam="head->tail", mode="direct", cache_mode="cached", model_ms=0.0, encode_ms=0.0, seam="head->tail", mode="direct", cache_mode="cached", model_ms=0.0, encode_ms=0.0,
activation_decode_ms=0.0,
framing_ms=0.0, metadata_ms=0.0, copy_allocation_ms=0.0, copy_allocation_bytes=0, framing_ms=0.0, metadata_ms=0.0, copy_allocation_ms=0.0, copy_allocation_bytes=0,
compression_ms=0.0, decompression_ms=0.0, connection_setup_ms=elapsed_ms, compression_ms=0.0, decompression_ms=0.0, connection_setup_ms=elapsed_ms,
queue_wait_ms=0.0, transport_ms=elapsed_ms, seam_latency_ms=elapsed_ms, queue_wait_ms=0.0, local_http_forwarding_ms=0.0, transport_ms=elapsed_ms, seam_latency_ms=elapsed_ms,
payload_bytes=len(body), wire_bytes=len(body) + len(response_body), compression_ratio=1.0, payload_bytes=len(body), wire_bytes=len(body) + len(response_body), compression_ratio=1.0,
connection_attempted=True, connection_attempted=True,
) )
@@ -354,6 +368,10 @@ def format_summary(report: dict) -> str:
f"{decode['tokens_per_sec']:.1f} tok/s; {decode['bytes_per_token']:.0f} B/tok; " f"{decode['tokens_per_sec']:.1f} tok/s; {decode['bytes_per_token']:.0f} B/tok; "
f"seam {seam['payload_bytes']}/{seam['wire_bytes']} B " f"seam {seam['payload_bytes']}/{seam['wire_bytes']} B "
f"({seam['compression_ratio']:.2f}x); connections {run['connections']['attempts']}; " f"({seam['compression_ratio']:.2f}x); connections {run['connections']['attempts']}; "
f"model/encode/decode {decode['model_execution_ms']:.2f}/"
f"{decode['activation_encoding_ms']:.2f}/{decode['activation_decoding_ms']:.2f} ms; "
f"compression {decode['compression_cpu_ms']:.2f} ms; "
f"HTTP {decode['local_http_forwarding_ms']:.2f} ms; "
f"queue p95 {decode['p95_queue_wait_ms']:.2f} ms" f"queue p95 {decode['p95_queue_wait_ms']:.2f} ms"
) )
return "\n".join(lines) return "\n".join(lines)

View File

@@ -12,7 +12,7 @@ import urllib.error
import urllib.parse import urllib.parse
import urllib.request import urllib.request
from pathlib import Path from pathlib import Path
from typing import Any from typing import Any, Callable
from .admission import ( from .admission import (
AdmissionRequirement, AdmissionRequirement,
@@ -419,6 +419,7 @@ def _start_heartbeat(
interval: float = _HEARTBEAT_INTERVAL_IDLE, interval: float = _HEARTBEAT_INTERVAL_IDLE,
node_ref: Any | None = None, node_ref: Any | None = None,
start_time: float | None = None, start_time: float | None = None,
refresh_capability: Callable[[dict], dict | None] | None = None,
) -> threading.Thread: ) -> threading.Thread:
"""Daemon thread: sends heartbeats and re-registers automatically after tracker restarts. """Daemon thread: sends heartbeats and re-registers automatically after tracker restarts.
@@ -430,6 +431,7 @@ def _start_heartbeat(
which is logged for now (hot-reload implemented in US-026). which is logged for now (hot-reload implemented in US-026).
""" """
_start_time = start_time or time.monotonic() _start_time = start_time or time.monotonic()
completed_directives: list[dict] = []
def _current_requests_snapshot() -> list[dict]: def _current_requests_snapshot() -> list[dict]:
if node_ref is None: if node_ref is None:
@@ -454,6 +456,8 @@ def _start_heartbeat(
current_requests = _current_requests_snapshot() current_requests = _current_requests_snapshot()
if current_requests: if current_requests:
stats["current_requests"] = current_requests stats["current_requests"] = current_requests
if completed_directives:
stats["completed_directives"] = list(completed_directives)
return stats return stats
def _sleep_interval() -> float: def _sleep_interval() -> float:
@@ -461,9 +465,26 @@ def _start_heartbeat(
return _HEARTBEAT_INTERVAL_BUSY return _HEARTBEAT_INTERVAL_BUSY
return interval return interval
def _refresh_proof(payload: dict) -> None:
"""Re-prove the current shard so a re-registration never presents an aged proof.
The tracker refuses proofs older than its freshness budget: re-sending the
startup-time report after an outage would re-register the node unroutable.
"""
if refresh_capability is None or "capability_report" not in payload:
return
try:
fresh = refresh_capability(payload)
except Exception as exc:
print(f" [node] WARNING: capability re-validation failed: {exc}", flush=True)
return
if fresh:
payload["capability_report"] = fresh
def _reregister() -> bool: def _reregister() -> bool:
nonlocal node_id nonlocal node_id
try: try:
_refresh_proof(register_payload)
resp = _post_json(f"{tracker_url}/v1/nodes/register", register_payload) resp = _post_json(f"{tracker_url}/v1/nodes/register", register_payload)
node_id = resp.get("node_id", node_id) node_id = resp.get("node_id", node_id)
if node_ref is not None: if node_ref is not None:
@@ -485,6 +506,7 @@ def _start_heartbeat(
"managed_assignment": True, "managed_assignment": True,
} }
try: try:
_refresh_proof(extra_payload)
reg_resp = _post_json(f"{tracker_url}/v1/nodes/register", extra_payload) reg_resp = _post_json(f"{tracker_url}/v1/nodes/register", extra_payload)
print( print(
f" [node] registered additional model — node ID: {reg_resp.get('node_id')}", f" [node] registered additional model — node ID: {reg_resp.get('node_id')}",
@@ -493,21 +515,26 @@ def _start_heartbeat(
except Exception as exc: except Exception as exc:
print(f" [node] WARNING: additional model registration failed: {exc}", flush=True) print(f" [node] WARNING: additional model registration failed: {exc}", flush=True)
def _apply_directives(directives: list[dict]) -> None: def _apply_directives(directives: list[dict]) -> dict | None:
if not directives: if not directives:
return return None
if node_ref is None or not hasattr(node_ref, "apply_tracker_directives"): if node_ref is None or not hasattr(node_ref, "apply_tracker_directives"):
print(f" [node] tracker directives received: {directives}", flush=True) print(f" [node] tracker directives received: {directives}", flush=True)
return return None
try: try:
applied = node_ref.apply_tracker_directives(directives) applied = node_ref.apply_tracker_directives(directives)
except Exception as exc: except Exception as exc:
print(f" [node] WARNING: failed to apply tracker directives: {exc}", flush=True) print(f" [node] WARNING: failed to apply tracker directives: {exc}", flush=True)
return return None
if applied: if applied:
completed_directives.append(dict(applied))
if applied.get("action") == "ADD_SHARD": if applied.get("action") == "ADD_SHARD":
_register_additional_assignment(applied) _register_additional_assignment(applied)
return return applied
if applied.get("action") in {"DROP_SHARD", "DROP_ALL_SHARDS"}:
# A release has no replacement range. It is not a failed
# heartbeat and must not re-register the released assignment.
return applied
model_id = applied.get("model", register_payload.get("hf_repo") or register_payload.get("model")) model_id = applied.get("model", register_payload.get("hf_repo") or register_payload.get("model"))
register_payload["model"] = str(model_id).split("/")[-1] register_payload["model"] = str(model_id).split("/")[-1]
register_payload["hf_repo"] = model_id register_payload["hf_repo"] = model_id
@@ -515,6 +542,7 @@ def _start_heartbeat(
register_payload["shard_end"] = applied["shard_end"] register_payload["shard_end"] = applied["shard_end"]
register_payload["quantization"] = applied.get("quantization", register_payload.get("quantization")) register_payload["quantization"] = applied.get("quantization", register_payload.get("quantization"))
register_payload["tracker_mode"] = bool(applied.get("tracker_mode", False)) register_payload["tracker_mode"] = bool(applied.get("tracker_mode", False))
return applied
def _loop() -> None: def _loop() -> None:
nonlocal node_id nonlocal node_id
@@ -542,7 +570,10 @@ def _start_heartbeat(
continue continue
try: try:
resp = _post_json(hb_url, _get_stats()) heartbeat = _get_stats()
resp = _post_json(hb_url, heartbeat)
if heartbeat.get("completed_directives"):
completed_directives.clear()
_apply_directives(resp.get("directives", [])) _apply_directives(resp.get("directives", []))
new_asgn = resp.get("new_assignment") new_asgn = resp.get("new_assignment")
if new_asgn: if new_asgn:
@@ -579,6 +610,7 @@ def _register_with_tracker(
reg_payload: dict, reg_payload: dict,
node: Any, node: Any,
start_time: float, start_time: float,
refresh_capability: Callable[[dict], dict | None] | None = None,
) -> str | None: ) -> str | None:
"""Register with the tracker, or start background retries when it is unreachable.""" """Register with the tracker, or start background retries when it is unreachable."""
try: try:
@@ -586,7 +618,14 @@ def _register_with_tracker(
tracker_node_id = str(reg_resp.get("node_id") or "?") tracker_node_id = str(reg_resp.get("node_id") or "?")
setattr(node, "tracker_node_id", tracker_node_id) setattr(node, "tracker_node_id", tracker_node_id)
print(f" Registered with tracker — node ID: {tracker_node_id}", flush=True) print(f" Registered with tracker — node ID: {tracker_node_id}", flush=True)
_start_heartbeat(tracker_url, tracker_node_id, reg_payload, node_ref=node, start_time=start_time) _start_heartbeat(
tracker_url,
tracker_node_id,
reg_payload,
node_ref=node,
start_time=start_time,
refresh_capability=refresh_capability,
)
return tracker_node_id return tracker_node_id
except Exception as exc: except Exception as exc:
setattr(node, "tracker_node_id", None) setattr(node, "tracker_node_id", None)
@@ -598,6 +637,7 @@ def _register_with_tracker(
reg_payload, reg_payload,
node_ref=node, node_ref=node,
start_time=start_time, start_time=start_time,
refresh_capability=refresh_capability,
) )
return None return None
@@ -718,6 +758,54 @@ def _admit_capability(
return report return report
def _capability_refresher(
node: Any,
*,
manifest: RecipeManifest,
recipe: Recipe,
detected_device: str,
cache_dir: Path | None,
force_cpu: bool,
validator: CapabilityValidator | None = None,
) -> Callable[[dict], dict | None]:
"""A fresh proof for what the node serves *now*, run at re-registration time.
The startup proof ages past the tracker's freshness budget, and directives
can move the node to a shard the startup proof never covered — so every
re-registration re-proves against the currently loaded backend rather than
replaying the report captured at boot.
"""
def refresh(payload: dict) -> dict | None:
target_model = payload.get("hf_repo") or payload.get("model")
backend = None
accessor = getattr(node, "backend_for", None)
if callable(accessor) and target_model:
backend = accessor(str(target_model))
if backend is None:
backend = getattr(node, "backend", None)
if backend is None:
return None
context = CapabilityContext(
backend=backend,
selection=DoctorSelection(
model_id=str(getattr(backend, "model_id", target_model)),
shard_start=int(getattr(backend, "shard_start", 0) or 0),
shard_end=int(getattr(backend, "shard_end", 0) or 0),
quantization=str(getattr(backend, "quantization", None) or "auto"),
cache_dir=cache_dir,
force_cpu=force_cpu,
),
recipe=recipe,
manifest=manifest,
device=_capability_device(backend, detected_device),
)
report = (validator or probe_capability)(context)
setattr(node, "capability_report", report)
return report.to_dict()
return refresh
def run_startup( def run_startup(
tracker_url: str, tracker_url: str,
port: int = 0, port: int = 0,
@@ -1026,6 +1114,15 @@ def run_startup(
} }
tracker_node_id = _register_with_tracker( tracker_node_id = _register_with_tracker(
tracker_url, reg_payload, node, _node_start_time, tracker_url, reg_payload, node, _node_start_time,
refresh_capability=_capability_refresher(
node,
manifest=manifest,
recipe=recipe,
detected_device=device,
cache_dir=cache_dir,
force_cpu=force_cpu,
validator=capability_validator,
),
) )
print( print(
@@ -1197,6 +1294,15 @@ def run_startup(
} }
tracker_node_id = _register_with_tracker( tracker_node_id = _register_with_tracker(
tracker_url, auto_reg_payload, node, _node_start_time, tracker_url, auto_reg_payload, node, _node_start_time,
refresh_capability=_capability_refresher(
node,
manifest=manifest,
recipe=recipe,
detected_device=device,
cache_dir=cache_dir,
force_cpu=force_cpu,
validator=capability_validator,
),
) )
shard_label = _format_shard_label( shard_label = _format_shard_label(
assigned_shard_start, assigned_shard_start,
@@ -1389,6 +1495,15 @@ def run_startup(
} }
tracker_node_id = _register_with_tracker( tracker_node_id = _register_with_tracker(
tracker_url, reg_payload, node, _node_start_time, tracker_url, reg_payload, node, _node_start_time,
refresh_capability=_capability_refresher(
node,
manifest=manifest,
recipe=recipe,
detected_device=device,
cache_dir=cache_dir,
force_cpu=force_cpu,
validator=capability_validator,
),
) )
print( print(
f"\n{'=' * 32}\n" f"\n{'=' * 32}\n"
@@ -1474,7 +1589,22 @@ def run_startup(
) )
node_id = str(reg_resp["node_id"]) node_id = str(reg_resp["node_id"])
setattr(node, "tracker_node_id", node_id) setattr(node, "tracker_node_id", node_id)
_start_heartbeat(tracker_url, node_id, reg_payload, node_ref=node, start_time=_node_start_time) _start_heartbeat(
tracker_url,
node_id,
reg_payload,
node_ref=node,
start_time=_node_start_time,
refresh_capability=_capability_refresher(
node,
manifest=manifest,
recipe=recipe,
detected_device=device,
cache_dir=shard_path,
force_cpu=force_cpu,
validator=capability_validator,
),
)
except Exception: except Exception:
node.stop() node.stop()
raise raise

View File

@@ -1543,8 +1543,52 @@ class TorchNodeServer:
def loaded_model_ids(self) -> list[str]: def loaded_model_ids(self) -> list[str]:
return list(self._backends.keys()) return list(self._backends.keys())
def backend_for(self, model_id: str) -> TorchModelShard | None:
"""The loaded backend serving `model_id` — full repo id or short name."""
backend = self._backends.get(model_id)
if backend is not None:
return backend
short = model_id.split("/")[-1].lower()
for key, candidate in self._backends.items():
if key.split("/")[-1].lower() == short:
return candidate
return None
def apply_tracker_directives(self, directives: list[dict]) -> dict | None: def apply_tracker_directives(self, directives: list[dict]) -> dict | None:
"""Apply tracker shard directives (LOAD_SHARD replace, ADD_SHARD load-more).""" """Apply tracker shard directives (LOAD_SHARD replace, ADD_SHARD load-more)."""
drop_all_directive = next(
(directive for directive in reversed(directives) if directive.get("action") == "DROP_ALL_SHARDS"),
None,
)
if drop_all_directive is not None:
self._backends.clear()
self._backend = None
self._tracker_mode = False
if self._server is not None:
self._server.backends = {}
self._server.backend = None
self._server.tracker_mode = False
return {"action": "DROP_ALL_SHARDS"}
drop_directive = next(
(directive for directive in reversed(directives) if directive.get("action") == "DROP_SHARD"),
None,
)
if drop_directive is not None:
model_id = str(drop_directive.get("model") or "")
removed = self._backends.pop(model_id, None)
if removed is None:
return None
if self._backends:
self._backend = next(iter(self._backends.values()))
self._tracker_mode = self._backend.shard_start == 0
else:
self._backend = None
self._tracker_mode = False
if self._server is not None:
self._server.backends = dict(self._backends)
self._server.backend = self._backend
self._server.tracker_mode = self._tracker_mode
return {"action": "DROP_SHARD", "model": model_id}
add_directive = next( add_directive = next(
(directive for directive in reversed(directives) if directive.get("action") == "ADD_SHARD"), (directive for directive in reversed(directives) if directive.get("action") == "ADD_SHARD"),
None, None,
@@ -1574,6 +1618,8 @@ class TorchNodeServer:
flush=True, flush=True,
) )
try: try:
if replacing:
self._backends.clear()
new_backend = _load_backend(model_id, shard_start, shard_end, quantization, self._cache_dir) new_backend = _load_backend(model_id, shard_start, shard_end, quantization, self._cache_dir)
except TypeError: except TypeError:
new_backend = _load_backend(model_id, shard_start, shard_end, quantization) new_backend = _load_backend(model_id, shard_start, shard_end, quantization)

View File

@@ -21,6 +21,9 @@ import uuid
DEFAULT_PRICE_PER_1K_TOKENS = 0.02 # USDT DEFAULT_PRICE_PER_1K_TOKENS = 0.02 # USDT
DEFAULT_STARTING_CREDIT = 0.0 # USDT; accounts must be funded before inference DEFAULT_STARTING_CREDIT = 0.0 # USDT; accounts must be funded before inference
DEFAULT_BILLING_DB_PATH = "billing.sqlite" DEFAULT_BILLING_DB_PATH = "billing.sqlite"
DEFAULT_SETTLE_PERIOD = 86400.0 # seconds — max time between node payouts (24 h prod)
DEFAULT_PAYOUT_THRESHOLD = 5.0 # USDT — immediate payout when pending exceeds this
DEFAULT_PAYOUT_DUST_FLOOR = 0.01 # USDT — never pay less than this
NODE_REVENUE_SHARE = 0.90 # nodes 90% / protocol cut 10% (ADR-0015) NODE_REVENUE_SHARE = 0.90 # nodes 90% / protocol cut 10% (ADR-0015)

View File

@@ -8,7 +8,12 @@ import time
from pathlib import Path from pathlib import Path
from .accounts import DEFAULT_ACCOUNTS_DB_PATH from .accounts import DEFAULT_ACCOUNTS_DB_PATH
from .billing import DEFAULT_BILLING_DB_PATH from .billing import (
DEFAULT_BILLING_DB_PATH,
DEFAULT_PAYOUT_DUST_FLOOR,
DEFAULT_PAYOUT_THRESHOLD,
DEFAULT_SETTLE_PERIOD,
)
from .capability import ALL_POLICIES as ALL_CAPABILITY_POLICIES from .capability import ALL_POLICIES as ALL_CAPABILITY_POLICIES
from .hf_pricing import DEFAULT_HF_PRICING_LOG_DB_PATH from .hf_pricing import DEFAULT_HF_PRICING_LOG_DB_PATH
from .logging_setup import ( from .logging_setup import (
@@ -237,19 +242,19 @@ def main() -> None:
common.add_argument( common.add_argument(
"--settle-period", "--settle-period",
type=float, type=float,
default=86400.0, default=DEFAULT_SETTLE_PERIOD,
help="Max seconds between payouts to a node (dev: 60, prod: 86400)", help="Max seconds between payouts to a node (dev: 60, prod: 86400)",
) )
common.add_argument( common.add_argument(
"--payout-threshold", "--payout-threshold",
type=float, type=float,
default=5.0, default=DEFAULT_PAYOUT_THRESHOLD,
help="Pending USDT that triggers an immediate payout (dev: 0)", help="Pending USDT that triggers an immediate payout (dev: 0)",
) )
common.add_argument( common.add_argument(
"--payout-dust-floor", "--payout-dust-floor",
type=float, type=float,
default=0.01, default=DEFAULT_PAYOUT_DUST_FLOOR,
help="Never pay out less than this many USDT", help="Never pay out less than this many USDT",
) )
common.add_argument( common.add_argument(

View File

@@ -22,8 +22,9 @@
border-bottom:1px solid var(--border); flex-shrink:0; } border-bottom:1px solid var(--border); flex-shrink:0; }
header h1 { font-size:16px; margin:0; color:var(--accent); } header h1 { font-size:16px; margin:0; color:var(--accent); }
header .meta { color:var(--dim); font-size:12px; } header .meta { color:var(--dim); font-size:12px; }
main { display:grid; grid-template-columns:repeat(auto-fit,minmax(340px,1fr)); main { display:grid; grid-template-columns:1fr;
gap:14px; padding:14px 20px; } gap:14px; padding:14px 20px; }
main > section { width:100%; min-width:0; }
body.chat-tab-active main { body.chat-tab-active main {
flex:1; min-height:0; display:flex; flex-direction:column; flex:1; min-height:0; display:flex; flex-direction:column;
padding:0; gap:0; overflow:hidden; padding:0; gap:0; overflow:hidden;
@@ -32,6 +33,7 @@
border-radius:8px; padding:12px 14px; min-height:80px; } border-radius:8px; padding:12px 14px; min-height:80px; }
section h2 { margin:0 0 8px; font-size:12px; text-transform:uppercase; section h2 { margin:0 0 8px; font-size:12px; text-transform:uppercase;
letter-spacing:.08em; color:var(--dim); } letter-spacing:.08em; color:var(--dim); }
.panel-heading { display:flex; align-items:center; justify-content:space-between; gap:8px; }
table { width:100%; border-collapse:collapse; font-size:12px; } table { width:100%; border-collapse:collapse; font-size:12px; }
th { text-align:left; color:var(--dim); font-weight:normal; th { text-align:left; color:var(--dim); font-weight:normal;
border-bottom:1px solid var(--border); padding:2px 6px 4px 0; } border-bottom:1px solid var(--border); padding:2px 6px 4px 0; }
@@ -42,12 +44,15 @@
.empty { color:var(--dim); font-style:italic; } .empty { color:var(--dim); font-style:italic; }
.pill { display:inline-block; padding:0 7px; border-radius:9px; .pill { display:inline-block; padding:0 7px; border-radius:9px;
border:1px solid var(--border); font-size:11px; } border:1px solid var(--border); font-size:11px; }
input, button { font:inherit; color:var(--fg); background:var(--bg); input, button, select { font:inherit; color:var(--fg); background:var(--bg);
border:1px solid var(--border); border-radius:6px; padding:5px 8px; } border:1px solid var(--border); border-radius:6px; padding:5px 8px; }
input { width:100%; margin-bottom:6px; } input { width:100%; margin-bottom:6px; }
button { cursor:pointer; color:var(--accent); } button { cursor:pointer; color:var(--accent); }
button:hover { border-color:var(--accent); } button:hover { border-color:var(--accent); }
button.small { font-size:11px; padding:1px 7px; } button.small { font-size:11px; padding:1px 7px; }
dialog { color:var(--fg); background:var(--panel); border:1px solid var(--border); border-radius:8px; min-width:min(420px,calc(100vw - 32px)); }
dialog::backdrop { background:rgba(0,0,0,.55); }
.placement-dialog-actions { display:flex; justify-content:flex-end; gap:8px; margin-top:12px; }
.form-row { display:flex; gap:8px; } .form-row { display:flex; gap:8px; }
.form-row button { white-space:nowrap; } .form-row button { white-space:nowrap; }
.error-msg { color:var(--bad); font-size:12px; min-height:16px; } .error-msg { color:var(--bad); font-size:12px; min-height:16px; }
@@ -70,6 +75,12 @@
background:transparent; color:var(--dim); padding:5px 0 8px; } background:transparent; color:var(--dim); padding:5px 0 8px; }
.dashboard-tabs button.active { color:var(--accent); border-bottom-color:var(--accent); } .dashboard-tabs button.active { color:var(--accent); border-bottom-color:var(--accent); }
.wide { grid-column:1 / -1; } .wide { grid-column:1 / -1; }
/* Compact status cards fan out on desktop; tables remain readable at half width. */
@media (min-width:900px) {
main { grid-template-columns:repeat(4,minmax(0,1fr)); }
main > section { grid-column:span 1; }
.wide { grid-column:span 2; }
}
section[hidden] { display:none !important; } section[hidden] { display:none !important; }
section.chat-section { section.chat-section {
padding:0; border:0; border-radius:0; background:var(--bg); min-height:0; padding:0; border:0; border-radius:0; background:var(--bg); min-height:0;
@@ -204,7 +215,7 @@
.chat-compose button:disabled { opacity:.45; cursor:not-allowed; } .chat-compose button:disabled { opacity:.45; cursor:not-allowed; }
.console { .console {
background:var(--bg); border:1px solid var(--border); border-radius:6px; background:var(--bg); border:1px solid var(--border); border-radius:6px;
min-height:160px; max-height:280px; overflow:auto; padding:7px 9px; min-height:160px; max-height:520px; overflow-y:auto; overflow-x:auto; padding:7px 9px;
white-space:pre-wrap; word-break:break-word; font-size:11px; white-space:pre-wrap; word-break:break-word; font-size:11px;
} }
.console-line { padding:1px 0; border-bottom:1px solid #161b22; } .console-line { padding:1px 0; border-bottom:1px solid #161b22; }
@@ -247,8 +258,7 @@
</nav> </nav>
<main> <main>
<section data-tab="overview" id="account-section"><h2>Account</h2><div id="account">loading…</div></section> <section data-tab="overview" id="account-section"><h2>Account</h2><div id="account">loading…</div></section>
<section data-tab="overview"><h2>Tracker hive</h2><div id="hive" class="empty">loading…</div></section> <section data-tab="overview" class="wide"><h2>Nodes &amp; coverage</h2><div id="nodes" class="empty">loading…</div></section>
<section data-tab="overview"><h2>Nodes &amp; coverage</h2><div id="nodes" class="empty">loading…</div></section>
<section data-tab="overview"><h2>Model usage (RPM)</h2><div id="stats" class="empty">loading…</div></section> <section data-tab="overview"><h2>Model usage (RPM)</h2><div id="stats" class="empty">loading…</div></section>
<section data-tab="overview" class="wide"><h2>Routing (learned)</h2><div id="routing" class="empty">loading…</div></section> <section data-tab="overview" class="wide"><h2>Routing (learned)</h2><div id="routing" class="empty">loading…</div></section>
<section data-tab="overview" class="wide"><h2>Model inference speed</h2><div id="model-speed-chart" class="empty">loading…</div></section> <section data-tab="overview" class="wide"><h2>Model inference speed</h2><div id="model-speed-chart" class="empty">loading…</div></section>
@@ -287,10 +297,13 @@
<section data-tab="billing"><h2>Request history</h2><div id="billing-usage" class="empty">login required</div></section> <section data-tab="billing"><h2>Request history</h2><div id="billing-usage" class="empty">login required</div></section>
<section data-tab="billing" data-admin-only><h2>Node pending payouts</h2><div id="pending" class="empty">admin login required</div></section> <section data-tab="billing" data-admin-only><h2>Node pending payouts</h2><div id="pending" class="empty">admin login required</div></section>
<section data-tab="billing" data-admin-only><h2>Settlement history</h2><div id="settlements" class="empty">admin login required</div></section> <section data-tab="billing" data-admin-only><h2>Settlement history</h2><div id="settlements" class="empty">admin login required</div></section>
<section data-tab="admin"><h2>Tracker hive</h2><div id="hive" class="empty">loading…</div></section>
<section data-tab="admin" class="wide"><h2>Model placement</h2><div id="admin-model-placement-status" class="dim">Choose a model to load or release.</div><div id="admin-model-placement" class="empty">admin login required</div></section>
<section data-tab="admin" class="wide"><h2>Total node pool</h2><div id="admin-node-pool" class="empty">admin login required</div></section>
<section data-tab="admin" id="admin-section"><h2>All accounts (admin)</h2><div id="admin" class="empty"></div></section> <section data-tab="admin" id="admin-section"><h2>All accounts (admin)</h2><div id="admin" class="empty"></div></section>
<section data-tab="admin" data-admin-only><h2>Strikes / bans / forfeitures</h2><div id="fraud" class="empty">admin login required</div></section> <section data-tab="admin" data-admin-only><h2>Strikes / bans / forfeitures</h2><div id="fraud" class="empty">admin login required</div></section>
<section data-tab="admin"><h2>Client balances</h2><div id="clients" class="empty">admin login required</div></section> <section data-tab="admin"><h2>Client balances</h2><div id="clients" class="empty">admin login required</div></section>
<section data-tab="admin" class="wide"><h2>Console output</h2><div id="console" class="console empty">admin login required</div></section> <section data-tab="admin" class="wide"><h2 class="panel-heading">Console output <button class="small" type="button" onclick="clearConsole()">clear</button></h2><div id="console" class="console empty">admin login required</div></section>
<section data-tab="testing" data-admin-only> <section data-tab="testing" data-admin-only>
<h2>Test run status</h2> <h2>Test run status</h2>
<div id="testing-error" class="testing-error" hidden></div> <div id="testing-error" class="testing-error" hidden></div>
@@ -314,6 +327,16 @@
<div id="testing-log" class="console empty">no test output yet</div> <div id="testing-log" class="console empty">no test output yet</div>
</section> </section>
</main> </main>
<dialog id="model-placement-dialog">
<form method="dialog">
<div id="model-placement-dialog-title"></div>
<label for="model-placement-node">Node</label>
<select id="model-placement-node"></select>
<label id="model-placement-replace" style="display:none"><input type="checkbox" id="model-placement-replace-confirm"> Unload the currently loaded model before loading this one</label>
<div id="model-placement-replace-error" class="bad" style="display:none"></div>
<div class="placement-dialog-actions"><button value="cancel">Cancel</button><button type="button" id="model-placement-confirm">Confirm</button></div>
</form>
</dialog>
<script> <script>
"use strict"; "use strict";
const $ = id => document.getElementById(id); const $ = id => document.getElementById(id);
@@ -524,7 +547,67 @@ function renderHive(raft) {
(raft.peers ? `<br>peers: ${esc(Array.isArray(raft.peers) ? raft.peers.join(", ") : raft.peers)}` : ""); (raft.peers ? `<br>peers: ${esc(Array.isArray(raft.peers) ? raft.peers.join(", ") : raft.peers)}` : "");
} }
function renderNodes(map) { function nodeLiveTps(node) {
const reqs = (node.stats && node.stats.current_requests) || [];
let best = null;
for (const req of reqs) {
const rate = req.tokens_per_sec;
if (rate == null) continue;
if (best === null || rate > best) best = rate;
}
return best;
}
function buildModelPriceMap(modelsResp) {
const prices = new Map();
for (const model of (modelsResp && modelsResp.data) || []) {
if (!model.pricing) continue;
for (const key of modelLookupKeys(model)) {
prices.set(key, model.pricing);
prices.set(modelAliasKey(key), model.pricing);
}
}
return prices;
}
function lookupModelPricing(modelKey, aliasMap, priceMap) {
const candidates = [modelKey];
const alias = aliasMap.get(modelAliasKey(modelKey));
if (alias) candidates.push(alias);
for (const candidate of candidates) {
const hit = priceMap.get(candidate) || priceMap.get(modelAliasKey(candidate));
if (hit) return hit;
}
return null;
}
function formatInferencePrice(pricing) {
if (!pricing) return '<span class="dim">—</span>';
const input = pricing.input_per_1k_usdt;
const output = pricing.output_per_1k_usdt;
if (input == null && output == null) return '<span class="dim">—</span>';
if (input === output || output == null) {
return `<span class="num">${usdt(input ?? output)}/1k</span>`;
}
return `<span class="num">${usdt(input)}/${usdt(output)}</span> <span class="dim">in/out per 1k</span>`;
}
function modelGroupTypicalTps(modelKey, aliasMap, routing) {
const sampleNode = { hf_repo: modelKey, model: modelKey };
const resolved = resolveModelGroup(sampleNode, aliasMap);
const model = {
id: resolved,
name: resolved,
hf_repo: modelKey.includes("/") ? modelKey : null,
aliases: [modelKey, resolved].filter(Boolean),
};
return chatModelTypicalTps(model, routing);
}
function renderNodes(payload) {
const map = payload && payload.map;
const routing = payload && payload.routing;
const priceMap = payload && payload.priceMap;
const nodes = (map && map.nodes) || []; const nodes = (map && map.nodes) || [];
if (!nodes.length) { if (!nodes.length) {
$("nodes").innerHTML = '<div class="empty">no nodes registered</div>'; return; $("nodes").innerHTML = '<div class="empty">no nodes registered</div>'; return;
@@ -543,16 +626,22 @@ function renderNodes(map) {
.map(n => n.model_supply && n.model_supply.served_model_copies) .map(n => n.model_supply && n.model_supply.served_model_copies)
.filter(v => v !== null && v !== undefined); .filter(v => v !== null && v !== undefined);
const served = servedValues.length ? Math.max(...servedValues) : undefined; const served = servedValues.length ? Math.max(...servedValues) : undefined;
html += `<div><b>${esc(model)}</b> <span class="dim">(${group.length} node${group.length===1?"":"s"} · ${esc(copies(served))} served)</span></div>`; const typicalTps = modelGroupTypicalTps(model, aliasMap, routing);
html += table(["node", "shard", "tps (1h)", "queue", "served"], group.map(n => { const pricing = lookupModelPricing(model, aliasMap, priceMap);
html += `<div style="margin-top:8px"><b>${esc(model)}</b> <span class="dim">(` +
`${group.length} node${group.length === 1 ? "" : "s"} · ${esc(copies(served))} served · ` +
`${esc(tps(typicalTps))} tok/s · price ${formatInferencePrice(pricing)})</span></div>`;
html += table(["node", "shard", "last tps", "tps (1h)", "queue", "served"], group.map(n => {
const modelStats = (n.throughput && (n.throughput[n.hf_repo] || n.throughput[n.model])) || {}; const modelStats = (n.throughput && (n.throughput[n.hf_repo] || n.throughput[n.model])) || {};
return [ return [
nodeDisplayCell(n), nodeDisplayCell(n),
esc(`${n.shard_start ?? "?"}-${n.shard_end ?? "?"}`), esc(`${n.shard_start ?? "?"}-${n.shard_end ?? "?"}`),
`<span class="num">${esc(tps(nodeLiveTps(n)))}</span>`,
`<span class="num">${esc(tps(modelStats.tokens_per_sec_last_hour))}</span>`, `<span class="num">${esc(tps(modelStats.tokens_per_sec_last_hour))}</span>`,
esc(String((n.stats && n.stats.queue_depth) ?? 0)), esc(String((n.stats && n.stats.queue_depth) ?? 0)),
`<span class="num">${esc(copies(n.model_supply && n.model_supply.served_model_copies))}</span>`, `<span class="num">${esc(copies(n.model_supply && n.model_supply.served_model_copies))}</span>`,
]; })); ];
}));
} }
$("nodes").innerHTML = html; $("nodes").innerHTML = html;
} }
@@ -1006,13 +1095,22 @@ function renderBillingUsage(records) {
table(["time", "model", "api key", "tokens", "cost (USDT)"], rows); table(["time", "model", "api key", "tokens", "cost (USDT)"], rows);
} }
let consoleClearedAt = 0;
const CONSOLE_MAX_LINES = 1000;
function clearConsole() {
consoleClearedAt = Date.now() / 1000;
panelSig.console = null;
renderConsole({ events: [] });
}
function renderConsole(data) { function renderConsole(data) {
const events = (data && data.events) || []; const events = ((data && data.events) || []).filter(event => event.ts > consoleClearedAt);
if (!events.length) { if (!events.length) {
$("console").innerHTML = '<div class="empty">no console events</div>'; $("console").innerHTML = '<div class="empty">no console events</div>';
return; return;
} }
$("console").innerHTML = events.slice(-120).map(e => { $("console").innerHTML = events.slice(-CONSOLE_MAX_LINES).map(e => {
const level = String(e.level || "info"); const level = String(e.level || "info");
const cls = level === "error" ? "console-level-error" : level === "warn" ? "console-level-warn" : "console-level-info"; const cls = level === "error" ? "console-level-error" : level === "warn" ? "console-level-warn" : "console-level-info";
const fields = e.fields && Object.keys(e.fields).length ? " " + JSON.stringify(e.fields) : ""; const fields = e.fields && Object.keys(e.fields).length ? " " + JSON.stringify(e.fields) : "";
@@ -1706,7 +1804,7 @@ async function requestSelectedModelLoad() {
if (!selectedChatModel) return; if (!selectedChatModel) return;
const button = $("request-model-load"); const button = $("request-model-load");
if (button) button.disabled = true; if (button) button.disabled = true;
const result = await apiCall("/v1/models/load", "POST", { model: selectedChatModel }); const result = await apiCall("/v1/models/load", "POST", { model: selectedChatModel, force: isAdmin });
if (button) button.disabled = false; if (button) button.disabled = false;
if (!result.ok) { if (!result.ok) {
alert(result.data.error || "model load request failed"); alert(result.data.error || "model load request failed");
@@ -1716,6 +1814,136 @@ async function requestSelectedModelLoad() {
$("chat-status").textContent = `load queued on ${short(assignment.node_id || "node")} for layers ${assignment.shard_start}-${assignment.shard_end}`; $("chat-status").textContent = `load queued on ${short(assignment.node_id || "node")} for layers ${assignment.shard_start}-${assignment.shard_end}`;
} }
async function requestAdminModelLoad(model, nodeId, replacing) {
const result = await apiCall("/v1/models/load", "POST", { model, node_id: nodeId, force: replacing });
if (!result.ok) return showAdminModelPlacementStatus(result.data.error || "model load request failed", true);
const assignment = result.data.assignment || {};
showAdminModelPlacementStatus(`Load queued on ${short(assignment.node_id || "node")} for ${model}.`);
await refreshActiveTab(true);
}
async function releaseAdminModel(model, nodeId) {
const result = await apiCall("/v1/models/release", "POST", { model, node_id: nodeId });
if (!result.ok) return showAdminModelPlacementStatus(result.data.error || "model release request failed", true);
showAdminModelPlacementStatus(`Release queued for ${result.data.released || 0} node(s) serving ${model}.`);
await refreshActiveTab(true);
}
async function releaseAllNodeModels(nodeId) {
if (!confirm("Unload every model from this node?")) return;
const result = await apiCall("/v1/nodes/release-all", "POST", { node_id: nodeId });
if (!result.ok) return showAdminModelPlacementStatus(result.data.error || "node unload failed", true);
showAdminModelPlacementStatus(`Unload queued for ${short(nodeId)}.`);
await refreshActiveTab(true);
}
function showAdminModelPlacementStatus(message, isError) {
const status = $("admin-model-placement-status");
status.textContent = message;
status.className = isError ? "bad" : "ok";
}
function gib(bytes) { return bytes == null ? "not reported" : `${(Number(bytes) / 1073741824).toFixed(1)} GiB`; }
function renderAdminNodePool(map) {
const groups = {};
for (const node of (map && map.nodes) || []) {
const account = node.wallet_address || "unbound account";
(groups[account] = groups[account] || []).push(node);
}
let html = "";
for (const [account, nodes] of Object.entries(groups).sort(([a], [b]) => a.localeCompare(b))) {
html += `<div style="margin-top:10px"><b>${esc(short(account, 20))}</b> <span class="dim">${nodes.length} node(s)</span></div>`;
html += table(["node", "assignment", "state / slots", "model RAM", "RAM", "GPU / VRAM", "model drive", "action"], nodes.map(node => {
const hw = node.hardware_profile || {};
const cap = node.capacity || {};
// The network map keeps reported resource capacity under `capacity`.
node.ram_bytes = cap.ram_bytes ?? node.ram_bytes;
node.vram_bytes = cap.vram_bytes ?? node.vram_bytes;
const disk = hw.model_drive_free_bytes ?? hw.model_path_free_bytes ?? hw.disk_free_bytes;
const gpu = hw.gpu_name || (hw.cuda_available ? "CUDA GPU" : "CPU only");
const row = [nodeDisplayCell(node), esc(node.hf_repo || node.model || "unassigned"),
esc(`${node.stats?.status || "?"} · ${cap.loaded_slots ?? "?"}/${cap.max_loaded_shards ?? node.max_loaded_shards ?? "?"} slots`),
esc(gib(cap.loaded_model_bytes)),
esc(gib(node.ram_bytes || (hw.ram_mb && hw.ram_mb * 1048576))),
esc(`${gpu} · ${gib(node.vram_bytes || (hw.vram_mb && hw.vram_mb * 1048576))}`), esc(gib(disk))];
return row.concat([
node.shard_start == null ? '<span class="dim">empty</span>' :
`<button class="small" data-admin-node-release="${esc(node.node_id)}">unload all</button>`,
]);
}));
}
$("admin-node-pool").innerHTML = html || '<div class="empty">no nodes registered</div>';
}
$("admin-node-pool").addEventListener("click", event => {
const unload = event.target.closest("[data-admin-node-release]");
if (unload) void releaseAllNodeModels(unload.dataset.adminNodeRelease);
});
function renderAdminModelPlacement(models, map) {
const nodes = (map && map.nodes) || [];
const rows = ((models && models.data) || []).map(model => {
const aliases = new Set([model.id, model.hf_repo, ...(model.aliases || [])].filter(Boolean));
const serving = nodes.filter(node => aliases.has(node.model) || aliases.has(node.hf_repo)).length;
const downloaded = nodes.filter(node => aliases.has(node.model) || aliases.has(node.hf_repo) ||
(node.downloaded_models || []).some(item => aliases.has(item.model) || aliases.has(item.hf_repo))).length;
const loadable = model.id !== "stub-model";
const actions = `<button class="small" data-admin-model-load="${esc(model.id)}"${loadable ? "" : " disabled"}>load</button> ` +
`<button class="small" data-admin-model-release="${esc(model.id)}"${serving ? "" : " disabled"}>release</button>`;
return [esc(model.name || model.id), String(serving), String(downloaded), actions];
});
$("admin-model-placement").innerHTML = rows.length
? table(["model", "serving nodes", "downloaded on nodes", "admin action"], rows)
: '<div class="empty">no model presets configured</div>';
}
$("admin-model-placement").addEventListener("click", event => {
const load = event.target.closest("[data-admin-model-load]");
const release = event.target.closest("[data-admin-model-release]");
if (load) void chooseModelPlacementNode("load", load.dataset.adminModelLoad);
if (release) void chooseModelPlacementNode("release", release.dataset.adminModelRelease);
});
function chooseModelPlacementNode(action, model) {
const dialog = $("model-placement-dialog");
const select = $("model-placement-node");
const targetAlias = modelAliasKey(model);
const nodes = (lastNetworkMap?.nodes || []).filter(node => action === "load" ||
modelAliasKey(node.model) === targetAlias || modelAliasKey(node.hf_repo) === targetAlias);
if (!nodes.length) return showAdminModelPlacementStatus(`No node can ${action} ${model}.`, true);
$("model-placement-dialog-title").textContent = `${action === "load" ? "Load" : "Release"} ${model} on a node`;
select.innerHTML = nodes.map(node => `<option value="${esc(node.node_id)}">${esc(short(node.friendly_name || node.node_id, 20))}${esc(node.hf_repo || node.model || "unassigned")}</option>`).join("");
const replace = $("model-placement-replace");
const replaceConfirm = $("model-placement-replace-confirm");
const replaceError = $("model-placement-replace-error");
const confirmButton = $("model-placement-confirm");
const selectedNode = () => nodes.find(node => node.node_id === select.value);
const updateReplacementWarning = () => {
const node = selectedNode();
const occupied = action === "load" && node && node.shard_start != null && node.shard_end != null &&
modelAliasKey(node.hf_repo || node.model) !== targetAlias;
replace.style.display = occupied ? "" : "none";
replaceConfirm.checked = false;
replaceError.style.display = "none";
};
select.onchange = updateReplacementWarning;
updateReplacementWarning();
dialog.onclose = null;
confirmButton.onclick = () => {
const replacing = replace.style.display !== "none";
if (replacing && !replaceConfirm.checked) {
replaceError.textContent = "Tick the box to confirm that this will unload the current model.";
replaceError.style.display = "";
return;
}
dialog.close("confirm");
if (action === "load") void requestAdminModelLoad(model, select.value, replacing);
else void releaseAdminModel(model, select.value);
};
dialog.showModal();
}
function chatAuthToken() { function chatAuthToken() {
if (accountApiKeys.length) return accountApiKeys[0]; if (accountApiKeys.length) return accountApiKeys[0];
return null; return null;
@@ -2293,21 +2521,21 @@ function markRefreshed() {
async function fetchOverviewTab() { async function fetchOverviewTab() {
const fetches = [ const fetches = [
fetchJson("/v1/raft/status"),
fetchJson("/v1/network/map"), fetchJson("/v1/network/map"),
fetchJson("/v1/models"),
fetchJson("/v1/stats"), fetchJson("/v1/stats"),
fetchJson("/v1/routing"), fetchJson("/v1/routing"),
fetchJson("/v1/console"), fetchJson("/v1/console"),
]; ];
if (isLoggedIn) fetches.push(apiCall("/v1/account")); if (isLoggedIn) fetches.push(apiCall("/v1/account"));
const results = await Promise.all(fetches); const results = await Promise.all(fetches);
const [raft, map, stats, routing, consoleData, accountResp] = results; const [map, models, stats, routing, consoleData, accountResp] = results;
lastStats = stats; lastStats = stats;
lastRouting = routing; lastRouting = routing;
lastNetworkMap = map; lastNetworkMap = map;
if (accountResp && accountResp.ok) applyAccountSummary(accountResp.data, false); if (accountResp && accountResp.ok) applyAccountSummary(accountResp.data, false);
renderIfChanged("hive", raft, renderHive); const nodesPayload = { map, routing, priceMap: buildModelPriceMap(models) };
renderIfChanged("nodes", map, renderNodes); renderIfChanged("nodes", nodesPayload, renderNodes);
renderIfChanged("stats", stats, renderStats); renderIfChanged("stats", stats, renderStats);
renderIfChanged("routing", routing, renderRouting); renderIfChanged("routing", routing, renderRouting);
const speedModels = Object.keys((routing && routing.models) || {}); const speedModels = Object.keys((routing && routing.models) || {});
@@ -2348,16 +2576,23 @@ async function fetchBillingTab() {
async function fetchAdminTab() { async function fetchAdminTab() {
const fetches = [ const fetches = [
fetchJson("/v1/raft/status"),
fetchJson("/v1/console"), fetchJson("/v1/console"),
fetchJson("/v1/billing/summary"), fetchJson("/v1/billing/summary"),
fetchJson("/v1/registry/wallets"), fetchJson("/v1/registry/wallets"),
fetchJson("/v1/models"),
fetchJson("/v1/network/map"),
]; ];
if (isAdmin) fetches.push(apiCall("/v1/admin/accounts")); if (isAdmin) fetches.push(apiCall("/v1/admin/accounts"));
const results = await Promise.all(fetches); const results = await Promise.all(fetches);
const [consoleData, summary, wallets, adminResp] = results; const [raft, consoleData, summary, wallets, models, map, adminResp] = results;
if (map) lastNetworkMap = map;
renderIfChanged("hive", raft, renderHive);
renderIfChanged("console", consoleData, renderConsole); renderIfChanged("console", consoleData, renderConsole);
renderIfChanged("billing-summary", summary, data => renderBilling(data)); renderIfChanged("billing-summary", summary, data => renderBilling(data));
renderIfChanged("fraud", { wallets, summary }, data => renderFraud(data.wallets, data.summary)); renderIfChanged("fraud", { wallets, summary }, data => renderFraud(data.wallets, data.summary));
renderIfChanged("admin-model-placement", { models, map }, data => renderAdminModelPlacement(data.models, data.map));
renderIfChanged("admin-node-pool", map, renderAdminNodePool);
if (adminResp && adminResp.ok) { if (adminResp && adminResp.ok) {
renderIfChanged("admin", adminResp.data.accounts || [], accounts => { renderIfChanged("admin", adminResp.data.accounts || [], accounts => {
const rows = accounts.map(a => { const rows = accounts.map(a => {

View File

@@ -86,7 +86,7 @@ from .model_files import files_for_layer_range, snapshot_dir_for_repo
from .raft import RaftNode from .raft import RaftNode
_CONSOLE_LIMIT = 300 _CONSOLE_LIMIT = 1000
_PROXY_PROGRESS_LOG_INTERVAL = 5.0 _PROXY_PROGRESS_LOG_INTERVAL = 5.0
_SESSION_COOKIE_NAME = "meshnet_session" _SESSION_COOKIE_NAME = "meshnet_session"
@@ -1101,12 +1101,15 @@ def _registration_quantization(body: dict, quantizations: list[str]) -> str | No
An absent field predates the protocol adding it: it means "unknown", not An absent field predates the protocol adding it: it means "unknown", not
"unsupported", so the node keeps the best precision it advertises and stays "unsupported", so the node keeps the best precision it advertises and stays
routable. Anything the node states explicitly is taken at its word -- a null, routable. An explicit "auto" means the same thing — the node's CLI default
a non-string, or an unsupported name leaves it with no usable precision and delegates the choice, it does not refuse one. Anything else the node states
routing excludes it. explicitly is taken at its word -- a null, a non-string, or an unsupported
name leaves it with no usable precision and routing excludes it.
""" """
if "quantization" in body: declared = body.get("quantization")
return _normalize_quantization(body["quantization"]) declared_auto = isinstance(declared, str) and declared.strip().lower() == "auto"
if "quantization" in body and not declared_auto:
return _normalize_quantization(declared)
supported = [ supported = [
normalized for value in quantizations normalized for value in quantizations
if (normalized := _normalize_quantization(value)) is not None if (normalized := _normalize_quantization(value)) is not None
@@ -1225,6 +1228,7 @@ def _node_capacity_summary(node: _NodeEntry, preset: dict | None = None) -> dict
"quantization": node.quantization, "quantization": node.quantization,
"benchmark_tokens_per_sec": node.benchmark_tokens_per_sec, "benchmark_tokens_per_sec": node.benchmark_tokens_per_sec,
"effective_throughput": round(_effective_throughput(node), 4), "effective_throughput": round(_effective_throughput(node), 4),
"loaded_model_bytes": _assignment_memory_bytes(node, preset),
} }
if preset is not None: if preset is not None:
summary["max_assignable_layers"] = _node_layer_capacity(node, preset) summary["max_assignable_layers"] = _node_layer_capacity(node, preset)
@@ -1494,7 +1498,9 @@ def _scale_demanded_models_locked(server: "_TrackerHTTPServer") -> None:
break break
def _request_model_load_locked(server: "_TrackerHTTPServer", model_key: str) -> dict | None: def _request_model_load_locked(
server: "_TrackerHTTPServer", model_key: str, node_id: str | None = None,
) -> dict | None:
"""Queue an explicitly requested model on the best available joined node.""" """Queue an explicitly requested model on the best available joined node."""
resolved_name, preset = _resolve_model_preset(server.model_presets, model_key) resolved_name, preset = _resolve_model_preset(server.model_presets, model_key)
if preset is None or not preset.get("hf_repo"): if preset is None or not preset.get("hf_repo"):
@@ -1510,6 +1516,8 @@ def _request_model_load_locked(server: "_TrackerHTTPServer", model_key: str) ->
continue continue
host_nodes = [server.registry[item["node_id"]] for item in host["loaded"] if item["node_id"] in server.registry] host_nodes = [server.registry[item["node_id"]] for item in host["loaded"] if item["node_id"] in server.registry]
placeable = [node for node in host_nodes if _has_usable_quantization(node)] placeable = [node for node in host_nodes if _has_usable_quantization(node)]
if node_id is not None:
placeable = [node for node in placeable if node.node_id == node_id]
if not placeable: if not placeable:
continue continue
anchor = max(placeable, key=lambda node: node.benchmark_tokens_per_sec) anchor = max(placeable, key=lambda node: node.benchmark_tokens_per_sec)
@@ -1528,6 +1536,68 @@ def _request_model_load_locked(server: "_TrackerHTTPServer", model_key: str) ->
return None return None
def _force_model_load_locked(
server: "_TrackerHTTPServer", model_key: str, node_id: str | None = None,
) -> dict | None:
"""Replace the fastest ready assignment after an explicit admin eviction."""
resolved_name, preset = _resolve_model_preset(server.model_presets, model_key)
if preset is None or not preset.get("hf_repo"):
return None
start, end = _preset_layer_bounds(preset)
# An explicit admin eviction is permitted to recover a stuck/loading node
# and to use the preset default precision. It must only avoid a node that
# already has another assignment in flight.
candidates = [
node for node in server.registry.values()
if node.pending_new_assignment is None
and (node_id is None or node.node_id == node_id)
]
if not candidates:
return None
node = max(candidates, key=lambda item: item.benchmark_tokens_per_sec)
shard_end = min(end, start + max(1, min(_node_layer_capacity(node, preset), end - start + 1)) - 1)
quantization = _node_quantization(node, preset)
directive = _load_directive(node, str(preset["hf_repo"]), start, shard_end, quantization)
replaced = node.hf_repo or node.model
node.model, node.hf_repo = resolved_name, str(preset["hf_repo"])
node.shard_start, node.shard_end, node.quantization = start, shard_end, quantization
node.managed_assignment, node.pending_new_assignment = True, directive
node.pending_directives.append(directive)
_tracker_log(server, "warn", "model load forced", node_id=node.node_id,
model=resolved_name, replaced_model=replaced, shard=f"{start}-{shard_end}")
return {"node_id": node.node_id, "model": resolved_name, "hf_repo": preset["hf_repo"],
"shard_start": start, "shard_end": shard_end, "replaced_model": replaced}
def _release_model_locked(
server: "_TrackerHTTPServer", model_key: str, node_id: str | None = None,
) -> int:
"""Queue DROP_SHARD for every served shard and remove it from routing immediately."""
resolved_name, preset = _resolve_model_preset(server.model_presets, model_key)
if preset is None:
return 0
released = 0
for node in server.registry.values():
if node_id is not None and node.node_id != node_id:
continue
if not _node_matches_preset(node, resolved_name, preset) or node.shard_start is None or node.shard_end is None:
continue
node.pending_directives.append(_drop_directive(node, str(preset.get("hf_repo") or resolved_name), node.shard_start, node.shard_end, node.quantization or "bfloat16"))
node.status = "loading"
released += 1
return released
def _release_all_node_models_locked(server: "_TrackerHTTPServer", node_id: str) -> int:
"""Queue removal of every loaded assignment on one node."""
node = server.registry.get(node_id)
if node is None or node.shard_start is None or node.shard_end is None:
return 0
node.pending_directives.append({"action": "DROP_ALL_SHARDS"})
node.status = "loading"
return 1
def _preferred_node_quantization( def _preferred_node_quantization(
node: _NodeEntry, node: _NodeEntry,
preset: dict, preset: dict,
@@ -3043,6 +3113,12 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
if self.path == "/v1/models/load": if self.path == "/v1/models/load":
self._handle_model_load_request() self._handle_model_load_request()
return return
if self.path == "/v1/models/release":
self._handle_model_release_request()
return
if self.path == "/v1/nodes/release-all":
self._handle_node_release_all_request()
return
if self.path == "/v1/models/vote": if self.path == "/v1/models/vote":
self._handle_model_coverage_vote() self._handle_model_coverage_vote()
return return
@@ -3145,6 +3221,16 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
self.send_response(404) self.send_response(404)
self.end_headers() self.end_headers()
def _model_pricing_payload(self, model: str) -> dict | None:
server: _TrackerHTTPServer = self.server # type: ignore[assignment]
if server.billing is None:
return None
in_rate, out_rate = server.billing.prices_for(model)
return {
"input_per_1k_usdt": in_rate,
"output_per_1k_usdt": out_rate,
}
def _handle_models(self): def _handle_models(self):
server: _TrackerHTTPServer = self.server # type: ignore[assignment] server: _TrackerHTTPServer = self.server # type: ignore[assignment]
created = int(time.time()) created = int(time.time())
@@ -3160,8 +3246,6 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
seen_ids: set[str] = set() seen_ids: set[str] = set()
for name, preset in server.model_presets.items(): for name, preset in server.model_presets.items():
model_nodes = [node for node in alive if _node_matches_preset(node, name, preset)] model_nodes = [node for node in alive if _node_matches_preset(node, name, preset)]
if not model_nodes and not preset.get("recommended"):
continue
required_start, required_end = _preset_layer_bounds(preset) required_start, required_end = _preset_layer_bounds(preset)
coverage = _coverage_percentage( coverage = _coverage_percentage(
model_nodes, model_nodes,
@@ -3200,6 +3284,7 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
"shard_coverage_percentage": coverage, "shard_coverage_percentage": coverage,
"served_model_copies": served_copies, "served_model_copies": served_copies,
"quantizations": quantizations, "quantizations": quantizations,
"pricing": self._model_pricing_payload(name),
}) })
seen_ids.add(name) seen_ids.add(name)
# Note: the preset's hf_repo is deliberately NOT added to seen_ids — # Note: the preset's hf_repo is deliberately NOT added to seen_ids —
@@ -3210,7 +3295,12 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
node.hf_repo or node.model node.hf_repo or node.model
for node in alive for node in alive
if node.model is not None if node.model is not None
and node.model not in server.model_presets # The same model can be registered under its HF repository while
# the catalogue exposes its short preset id. Do not emit a second
# repo-keyed entry when either node identifier resolves to a preset.
and _resolve_model_preset(
server.model_presets, node.hf_repo or node.model,
)[1] is None
and node.shard_start is not None and node.shard_start is not None
and node.shard_end is not None and node.shard_end is not None
and node.num_layers is not None and node.num_layers is not None
@@ -3251,6 +3341,7 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
required_start, required_start,
required_end, required_end,
), ),
"pricing": self._model_pricing_payload(model_id),
}) })
seen_ids.add(model_id) seen_ids.add(model_id)
self._send_json(200, {"object": "list", "data": data}) self._send_json(200, {"object": "list", "data": data})
@@ -3310,6 +3401,11 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
"endpoint": node.endpoint, "endpoint": node.endpoint,
"relay_addr": node.relay_addr, "relay_addr": node.relay_addr,
"peer_id": node.peer_id, "peer_id": node.peer_id,
"wallet_address": node.wallet_address,
"hardware_profile": dict(node.hardware_profile),
"ram_bytes": node.ram_bytes,
"vram_bytes": node.vram_bytes,
"max_loaded_shards": node.max_loaded_shards,
} }
for node in tracker_nodes for node in tracker_nodes
], ],
@@ -3333,12 +3429,7 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
memory_pool = _memory_pool_map(server) memory_pool = _memory_pool_map(server)
def capacity_for(node: _NodeEntry) -> dict: def capacity_for(node: _NodeEntry) -> dict:
preset = None return _node_capacity_summary(node, _preset_for_node(server, node))
if node.model:
preset = server.model_presets.get(node.model)
if preset is None and node.hf_repo and node.num_layers:
preset = _hf_rebalance_preset([node])
return _node_capacity_summary(node, preset)
def throughput_for(node: _NodeEntry) -> dict: def throughput_for(node: _NodeEntry) -> dict:
if server.stats is None: if server.stats is None:
@@ -4748,6 +4839,20 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
entry.uptime_seconds = float(body["uptime_seconds"]) entry.uptime_seconds = float(body["uptime_seconds"])
if "status" in body and body["status"] in ("ready", "loading"): if "status" in body and body["status"] in ("ready", "loading"):
entry.status = body["status"] entry.status = body["status"]
completed_directives = body.get("completed_directives", [])
if isinstance(completed_directives, list):
for directive in completed_directives:
if not isinstance(directive, dict) or directive.get("action") not in {"DROP_SHARD", "DROP_ALL_SHARDS"}:
continue
# A node has confirmed the release. Stop advertising its
# old route immediately so the dashboard and routing state
# agree with the runtime.
entry.model = "stub-model"
entry.hf_repo = None
entry.shard_start = None
entry.shard_end = None
entry.tracker_mode = False
entry.status = "ready"
if "friendly_name" in body: if "friendly_name" in body:
try: try:
entry.friendly_name = _normalize_friendly_name(body.get("friendly_name")) entry.friendly_name = _normalize_friendly_name(body.get("friendly_name"))
@@ -4819,14 +4924,68 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
if not isinstance(model, str) or not model.strip(): if not isinstance(model, str) or not model.strip():
self._send_json(400, {"error": "model is required"}) self._send_json(400, {"error": "model is required"})
return return
node_id = body.get("node_id")
if node_id is not None and (not isinstance(node_id, str) or not node_id):
self._send_json(400, {"error": "node_id must be a non-empty string"})
return
_resolved_name, preset = _resolve_model_preset(server.model_presets, model)
if preset is None or str(preset.get("hf_repo") or "").strip().lower() == "stub-model":
self._send_json(400, {"error": "stub-model is a local test backend and cannot be loaded onto a node"})
return
with server.lock: with server.lock:
self._purge_expired_nodes() self._purge_expired_nodes()
assignment = _request_model_load_locked(server, model) assignment = _request_model_load_locked(server, model, node_id)
if assignment is None and body.get("force") is True:
assignment = _force_model_load_locked(server, model, node_id)
if assignment is None: if assignment is None:
self._send_json(409, {"error": "no ready joined node has an available model slot and sufficient capacity"}) self._send_json(409, {"error": "no ready joined node has an available model slot and sufficient capacity"})
return return
self._send_json(202, {"status": "queued", "assignment": assignment}) self._send_json(202, {"status": "queued", "assignment": assignment})
def _handle_model_release_request(self):
server: _TrackerHTTPServer = self.server # type: ignore[assignment]
if not self._require_role("admin", "validator"):
return
body = self._read_json_body()
if body is None:
return
model = body.get("model")
if not isinstance(model, str) or not model.strip():
self._send_json(400, {"error": "model is required"})
return
node_id = body.get("node_id")
if node_id is not None and (not isinstance(node_id, str) or not node_id):
self._send_json(400, {"error": "node_id must be a non-empty string"})
return
with server.lock:
self._purge_expired_nodes()
released = _release_model_locked(server, model, node_id)
if not released:
self._send_json(404, {"error": "no served shards found for model"})
return
self._send_json(202, {"status": "release_queued", "released": released})
def _handle_node_release_all_request(self):
server: _TrackerHTTPServer = self.server # type: ignore[assignment]
if not self._require_role("admin", "validator"):
return
body = self._read_json_body()
if body is None:
return
node_id = body.get("node_id")
if not isinstance(node_id, str) or not node_id:
self._send_json(400, {"error": "node_id must be a non-empty string"})
return
with server.lock:
self._purge_expired_nodes()
released = _release_all_node_models_locked(server, node_id)
if not released:
self._send_json(404, {"error": "no loaded models found for node"})
return
self._send_json(202, {
"status": "release_queued", "released": released, "node_id": node_id,
})
def _handle_model_coverage_vote(self): def _handle_model_coverage_vote(self):
"""Record a rolling wish-list signal for an unavailable precision.""" """Record a rolling wish-list signal for an unavailable precision."""
server: _TrackerHTTPServer = self.server # type: ignore[assignment] server: _TrackerHTTPServer = self.server # type: ignore[assignment]
@@ -6477,7 +6636,7 @@ class TrackerServer:
embedded_relay_port: int = 8765, embedded_relay_port: int = 8765,
embedded_relay_max_peers: int = 500, embedded_relay_max_peers: int = 500,
billing: BillingLedger | None = None, billing: BillingLedger | None = None,
enable_billing: bool = False, enable_billing: bool = True,
billing_db: str | None = None, billing_db: str | None = None,
accounts: AccountStore | None = None, accounts: AccountStore | None = None,
accounts_db: str | None = None, accounts_db: str | None = None,

View File

@@ -396,7 +396,7 @@ def test_accounts_gossip_endpoint_applies_events(account_tracker):
def test_accounts_endpoints_404_when_disabled(): def test_accounts_endpoints_404_when_disabled():
"Accounts endpoints 404 when disabled\n\nTags: accounts, auth, http" "Accounts endpoints 404 when disabled\n\nTags: accounts, auth, http"
tracker = TrackerServer() # no accounts, no billing tracker = TrackerServer(enable_billing=False) # no accounts, no billing
port = tracker.start() port = tracker.start()
try: try:
with pytest.raises(urllib.error.HTTPError) as exc_info: with pytest.raises(urllib.error.HTTPError) as exc_info:

View File

@@ -172,13 +172,13 @@ def test_restart_persistence(tmp_path):
assert reloaded.snapshot()["protocol_cut"] == pytest.approx(0.02 * 0.10) assert reloaded.snapshot()["protocol_cut"] == pytest.approx(0.02 * 0.10)
def test_tracker_enables_billing_with_default_db_when_requested(tmp_path, monkeypatch): def test_tracker_enables_billing_with_default_db(tmp_path, monkeypatch):
"Tracker enables billing with default db when requested\n\nTags: billing, http, payments" "Tracker enables billing with default db by default\n\nTags: billing, http, payments"
from meshnet_tracker.billing import DEFAULT_BILLING_DB_PATH from meshnet_tracker.billing import DEFAULT_BILLING_DB_PATH
monkeypatch.chdir(tmp_path) monkeypatch.chdir(tmp_path)
tracker = TrackerServer(enable_billing=True) tracker = TrackerServer()
port = tracker.start() port = tracker.start()
try: try:
# /v1/billing/summary is admin-gated now; just confirm the server is up. # /v1/billing/summary is admin-gated now; just confirm the server is up.
@@ -329,6 +329,7 @@ def billed_tracker():
} }
}, },
billing=ledger, billing=ledger,
enable_billing=False,
hive_secret=HIVE_SECRET, hive_secret=HIVE_SECRET,
) )
port = tracker.start() port = tracker.start()
@@ -457,6 +458,7 @@ def test_proxy_chat_caps_inflated_streaming_usage_by_observed_chunks():
} }
}, },
billing=ledger, billing=ledger,
enable_billing=False,
) )
port = tracker.start() port = tracker.start()
tracker_url = f"http://127.0.0.1:{port}" tracker_url = f"http://127.0.0.1:{port}"
@@ -506,6 +508,7 @@ def test_proxy_chat_splits_payout_by_tracker_assigned_route_span():
} }
}, },
billing=ledger, billing=ledger,
enable_billing=False,
) )
port = tracker.start() port = tracker.start()
tracker_url = f"http://127.0.0.1:{port}" tracker_url = f"http://127.0.0.1:{port}"
@@ -582,6 +585,7 @@ def test_proxy_chat_rejects_request_above_spend_cap_before_routing():
} }
}, },
billing=ledger, billing=ledger,
enable_billing=False,
max_charge_per_request=0.01, max_charge_per_request=0.01,
) )
port = tracker.start() port = tracker.start()
@@ -656,6 +660,7 @@ def test_proxy_chat_records_validation_event_with_plain_route_metadata():
} }
}, },
contracts=contracts, contracts=contracts,
enable_billing=False,
) )
port = tracker.start() port = tracker.start()
tracker_url = f"http://127.0.0.1:{port}" tracker_url = f"http://127.0.0.1:{port}"

View File

@@ -39,6 +39,14 @@ def test_dashboard_served_with_all_panels():
assert "resolveModelGroup" in html assert "resolveModelGroup" in html
assert "buildModelAliasMap" in html assert "buildModelAliasMap" in html
assert "modelAliasKey(raw)" in html assert "modelAliasKey(raw)" in html
assert "@media (min-width:900px)" in html
assert "grid-template-columns:repeat(4,minmax(0,1fr));" in html
assert ".wide { grid-column:span 2; }" in html
assert 'onclick="clearConsole()"' in html
assert "let consoleClearedAt = 0;" in html
assert "max-height:520px; overflow-y:auto; overflow-x:auto;" in html
assert "const CONSOLE_MAX_LINES = 1000;" in html
assert "events.slice(-CONSOLE_MAX_LINES)" in html
finally: finally:
tracker.stop() tracker.stop()
@@ -97,6 +105,39 @@ def test_dashboard_allows_admin_to_request_selected_model_load():
assert '$("request-model-load").style.display = enabled ? "" : "none"' in html assert '$("request-model-load").style.display = enabled ? "" : "none"' in html
def test_dashboard_exposes_admin_model_inventory_and_release_controls():
"Admin placement controls show the full model inventory and can release capacity."
html = _dashboard_html()
assert 'id="admin-model-placement"' in html
assert "renderAdminModelPlacement" in html
assert '"/v1/models/release"' in html
assert "requestAdminModelLoad" in html
assert "releaseAdminModel" in html
assert 'data-admin-model-load=' in html
assert 'data-admin-model-release=' in html
assert "admin-model-placement-status" in html
assert 'id="admin-node-pool"' in html
assert "renderAdminNodePool" in html
assert "model drive" in html
# RAM and VRAM live in the network-map capacity object, not at node top level.
assert "node.ram_bytes = cap.ram_bytes" in html
assert "node.vram_bytes = cap.vram_bytes" in html
assert 'id="model-placement-dialog"' in html
assert "chooseModelPlacementNode" in html
assert "node_id: nodeId" in html
assert "modelAliasKey(node.model)" in html
assert 'id="model-placement-replace"' in html
assert 'id="model-placement-confirm"' in html
assert 'id="model-placement-replace-error"' in html
assert "force: replacing" in html
assert "Tick the box to confirm" in html
assert "releaseAllNodeModels" in html
assert '"/v1/nodes/release-all"' in html
assert "model RAM" in html
assert "loaded_model_bytes" in html
def test_network_map_includes_node_friendly_name(): def test_network_map_includes_node_friendly_name():
"Network map includes node friendly name\n\nTags: dashboard, http" "Network map includes node friendly name\n\nTags: dashboard, http"
tracker = TrackerServer() tracker = TrackerServer()

View File

@@ -253,7 +253,7 @@ def test_proxy_head_is_route_head_and_routing_endpoint_lists_routes():
threads.append(thread) threads.append(thread)
gpu_stub, cpu_stub = stubs gpu_stub, cpu_stub = stubs
tracker = TrackerServer(model_presets={ tracker = TrackerServer(enable_billing=False, model_presets={
"qwen3.6-35b-a3b": { "qwen3.6-35b-a3b": {
"layers_start": 0, "layers_start": 0,
"layers_end": 39, "layers_end": 39,
@@ -321,7 +321,7 @@ def test_proxy_head_is_route_head_and_routing_endpoint_lists_routes():
def test_admin_model_load_request_queues_directive_on_joined_node(): def test_admin_model_load_request_queues_directive_on_joined_node():
"Admin model load request queues directive on joined node\n\nTags: http, performance, routing, tracker" "Admin model load request queues directive on joined node\n\nTags: http, performance, routing, tracker"
tracker = TrackerServer(validator_service_token="test-admin") tracker = TrackerServer(enable_billing=False, validator_service_token="test-admin")
port = tracker.start() port = tracker.start()
try: try:
node = _post_json( node = _post_json(
@@ -355,6 +355,75 @@ def test_admin_model_load_request_queues_directive_on_joined_node():
assert heartbeat["directives"][0]["model"] == "Qwen/Qwen2.5-0.5B-Instruct" assert heartbeat["directives"][0]["model"] == "Qwen/Qwen2.5-0.5B-Instruct"
def test_admin_can_replace_a_served_model_and_release_it():
"Forced admin placement replaces a served shard; release queues DROP_SHARD."
tracker = TrackerServer(enable_billing=False, validator_service_token="test-admin")
port = tracker.start()
try:
node = _post_json(
f"http://127.0.0.1:{port}/v1/nodes/register",
{"endpoint": "http://127.0.0.1:9912", "model": "stub-model",
"shard_start": 0, "shard_end": 3, "managed_assignment": True,
"max_loaded_shards": 1, "memory_mb": 1,
"hardware_profile": {"host_id": "full-host"}},
)
headers = {"Content-Type": "application/json", "Authorization": "Bearer test-admin"}
load = urllib.request.Request(
f"http://127.0.0.1:{port}/v1/models/load",
data=json.dumps({
"model": "qwen2.5-0.5b-instruct",
"node_id": node["node_id"],
"force": True,
}).encode(),
headers=headers, method="POST")
with urllib.request.urlopen(load) as response:
assert json.loads(response.read())["assignment"]["node_id"] == node["node_id"]
heartbeat = _post_json(f"http://127.0.0.1:{port}/v1/nodes/{node['node_id']}/heartbeat", {})
_post_json(
f"http://127.0.0.1:{port}/v1/nodes/{node['node_id']}/heartbeat",
{"completed_directives": [{"action": "DROP_SHARD", "model": "Qwen/Qwen2.5-0.5B-Instruct"}]},
)
network = _get_json(f"http://127.0.0.1:{port}/v1/network/map")
assert heartbeat["directives"][0]["action"] == "LOAD_SHARD"
release = urllib.request.Request(
f"http://127.0.0.1:{port}/v1/models/release",
data=json.dumps({"model": "qwen2.5-0.5b-instruct"}).encode(), headers=headers, method="POST")
with urllib.request.urlopen(release) as response:
assert json.loads(response.read())["released"] == 1
heartbeat = _post_json(f"http://127.0.0.1:{port}/v1/nodes/{node['node_id']}/heartbeat", {})
finally:
tracker.stop()
assert heartbeat["directives"][0]["action"] == "DROP_SHARD"
released_node = next(item for item in network["nodes"] if item["node_id"] == node["node_id"])
assert released_node["shard_start"] is None
assert released_node["shard_end"] is None
def test_models_list_does_not_duplicate_a_preset_registered_by_hf_repo():
"""A preset and its canonical repository are one selectable model."""
tracker = TrackerServer(enable_billing=False)
port = tracker.start()
try:
_post_json(
f"http://127.0.0.1:{port}/v1/nodes/register",
{
"endpoint": "http://127.0.0.1:9913",
"model": "Qwen2.5-0.5B-Instruct",
"hf_repo": "Qwen/Qwen2.5-0.5B-Instruct",
"num_layers": 24,
"shard_start": 0,
"shard_end": 23,
},
)
models = _get_json(f"http://127.0.0.1:{port}/v1/models")["data"]
finally:
tracker.stop()
assert [model["id"] for model in models].count("qwen2.5-0.5b-instruct") == 1
assert not any(model["id"] == "Qwen/Qwen2.5-0.5B-Instruct" for model in models)
def test_endpoint_key_distinguishes_same_port_different_hosts(): def test_endpoint_key_distinguishes_same_port_different_hosts():
"Endpoint key distinguishes same port different hosts\n\nTags: http, performance, routing, tracker" "Endpoint key distinguishes same port different hosts\n\nTags: http, performance, routing, tracker"
from meshnet_node.torch_server import _clamp_downstream_hops, _endpoint_key from meshnet_node.torch_server import _clamp_downstream_hops, _endpoint_key

View File

@@ -203,6 +203,7 @@ def test_probation_earns_nothing_then_earning_begins():
}, },
contracts=contracts, contracts=contracts,
billing=ledger, billing=ledger,
enable_billing=False,
) )
port = tracker.start() port = tracker.start()
tracker_url = f"http://127.0.0.1:{port}" tracker_url = f"http://127.0.0.1:{port}"

View File

@@ -60,7 +60,7 @@ def test_bad_node_is_slashed_and_excluded_from_gateway_routes(capsys):
contracts = LocalSolanaContracts() contracts = LocalSolanaContracts()
contracts.registry.submit_stake("wallet-good", 500) contracts.registry.submit_stake("wallet-good", 500)
contracts.registry.submit_stake("wallet-bad", 500) contracts.registry.submit_stake("wallet-bad", 500)
tracker = TrackerServer(contracts=contracts) tracker = TrackerServer(contracts=contracts, enable_billing=False)
tracker_port = tracker.start() tracker_port = tracker.start()
tracker_url = f"http://127.0.0.1:{tracker_port}" tracker_url = f"http://127.0.0.1:{tracker_port}"

View File

@@ -76,7 +76,7 @@ def test_tracker_records_increasing_hop_latency_for_model_and_hardware(model, ha
layer_count, hf_repo = MODELS[model] layer_count, hf_repo = MODELS[model]
nodes, threads = _start_latency_nodes(5) nodes, threads = _start_latency_nodes(5)
tracker = TrackerServer(model_presets={ tracker = TrackerServer(enable_billing=False, model_presets={
model: { model: {
"layers_start": 0, "layers_start": 0,
"layers_end": layer_count - 1, "layers_end": layer_count - 1,
@@ -142,7 +142,7 @@ def test_tracker_records_increasing_hop_latency_for_model_and_hardware(model, ha
def test_model_speed_dashboard_includes_visualization_and_route_drilldown(): def test_model_speed_dashboard_includes_visualization_and_route_drilldown():
"Model speed dashboard includes visualization and route drilldown\n\nTags: model, node, routing" "Model speed dashboard includes visualization and route drilldown\n\nTags: model, node, routing"
tracker = TrackerServer() tracker = TrackerServer(enable_billing=False)
port = tracker.start() port = tracker.start()
try: try:
html = urllib.request.urlopen(f"http://127.0.0.1:{port}/dashboard").read().decode() html = urllib.request.urlopen(f"http://127.0.0.1:{port}/dashboard").read().decode()

View File

@@ -358,6 +358,73 @@ def test_a_stale_report_cannot_be_reused_to_register(startup_env):
assert startup_env == [] assert startup_env == []
# ---------------------------------------------------------------------------
# Re-registration: the proof presented is fresh, never the one captured at boot
# ---------------------------------------------------------------------------
def test_run_startup_hands_the_heartbeat_a_refresher_for_the_current_shard(startup_env, monkeypatch):
"The tracker refuses aged proofs, so the heartbeat must be able to re-prove what the node serves now.\n\nTags: node, admission, startup"
import meshnet_node.startup as startup_mod
captured: dict = {}
monkeypatch.setattr(
startup_mod, "_start_heartbeat", lambda *a, **kw: captured.update(kw)
)
_start(capability_validator=capability_stub())
refresh = captured.get("refresh_capability")
assert callable(refresh), "run_startup no longer wires a capability refresher"
fresh = refresh({"hf_repo": MODEL, "model": MODEL.split("/")[-1]})
assert fresh is not None
assert fresh["model"]["model_id"] == MODEL
assert (fresh["shard"]["start"], fresh["shard"]["end"]) == (0, 23)
assert fresh["validated_at"] > time.time() - 60
def test_a_reregistration_presents_a_refreshed_proof(monkeypatch):
"Replaying the boot-time report after an outage re-registers the node unroutable; the re-register path must present a fresh proof.\n\nTags: node, admission, startup"
import json
import meshnet_node.startup as startup_mod
model = "acme/refresh-model-7b"
boot_report = {"validated_at": 1.0, "marker": "boot"}
fresh_report = {"validated_at": 2.0, "marker": "fresh"}
posted: list[dict] = []
def _record(url, payload, timeout=10.0):
if url.endswith("/v1/nodes/register") and payload.get("hf_repo") == model:
posted.append(json.loads(json.dumps(payload)))
return {"node_id": "node-refresh"}
raise SystemExit # first heartbeat POST ends the daemon loop
monkeypatch.setattr(startup_mod, "_post_json", _record)
payload = {
"hf_repo": model,
"model": model.split("/")[-1],
"capability_report": dict(boot_report),
}
thread = startup_mod._start_heartbeat(
"http://tracker.invalid",
startup_mod._PENDING_NODE_ID, # forces a re-registration on the first tick
payload,
interval=0.02,
refresh_capability=lambda _payload: dict(fresh_report),
)
# The loop must be dead before this test returns: once monkeypatch restores
# `_post_json`, a surviving thread would re-register through whatever the
# *next* test patches in and corrupt its call counts.
thread.join(timeout=5.0)
assert not thread.is_alive(), "the heartbeat loop outlived the test"
assert posted, "the heartbeat never re-registered"
assert posted[0]["capability_report"] == fresh_report
def test_a_matching_passing_report_registers_and_travels_with_the_payload(startup_env): def test_a_matching_passing_report_registers_and_travels_with_the_payload(startup_env):
"Registration carries the proof for exactly the model/shard/recipe it advertises.\n\nTags: node, admission, startup" "Registration carries the proof for exactly the model/shard/recipe it advertises.\n\nTags: node, admission, startup"
node = _start() # production validator against a working fake backend node = _start() # production validator against a working fake backend

View File

@@ -287,6 +287,47 @@ def test_configure_torch_threads_applies_explicit_settings(monkeypatch):
assert active == {"torch_threads": 12, "torch_interop_threads": 2} assert active == {"torch_threads": 12, "torch_interop_threads": 2}
def test_heartbeat_applies_release_without_reregistering(monkeypatch):
"""DROP_SHARD has no replacement range and must not look like an outage."""
import meshnet_node.startup as startup_mod
released = threading.Event()
requests: list[tuple[str, dict]] = []
class FakeNode:
def apply_tracker_directives(self, directives):
assert directives == [{"action": "DROP_SHARD", "model": "Qwen/Qwen2.5-0.5B-Instruct"}]
return {"action": "DROP_SHARD", "model": "Qwen/Qwen2.5-0.5B-Instruct"}
def fake_post(url, payload, timeout=10.0):
requests.append((url, dict(payload)))
released.set()
return {"directives": [{"action": "DROP_SHARD", "model": "Qwen/Qwen2.5-0.5B-Instruct"}]}
sleep_calls = 0
def one_heartbeat(_seconds):
nonlocal sleep_calls
sleep_calls += 1
if sleep_calls > 1:
raise SystemExit
monkeypatch.setattr(startup_mod, "_post_json", fake_post)
monkeypatch.setattr(startup_mod.time, "sleep", one_heartbeat)
payload = {
"model": "Qwen2.5-0.5B-Instruct",
"hf_repo": "Qwen/Qwen2.5-0.5B-Instruct",
"shard_start": 0,
"shard_end": 23,
}
startup_mod._start_heartbeat("http://tracker", "node-1", payload, interval=0, node_ref=FakeNode())
assert released.wait(1), "heartbeat did not receive the queued release"
assert len(requests) == 1, "release must not trigger a re-registration"
assert payload["shard_start"] == 0
assert payload["shard_end"] == 23
def test_benchmark_throughput_is_registered_in_payload(monkeypatch, tmp_path): def test_benchmark_throughput_is_registered_in_payload(monkeypatch, tmp_path):
"benchmark_tokens_per_sec from the benchmark is included in the tracker registration.\n\nTags: node, performance, startup" "benchmark_tokens_per_sec from the benchmark is included in the tracker registration.\n\nTags: node, performance, startup"
import meshnet_node.startup as startup_mod import meshnet_node.startup as startup_mod

View File

@@ -32,7 +32,7 @@ def _register_node(tracker_url: str, endpoint: str, shard_start: int, shard_end:
@pytest.fixture @pytest.fixture
def two_node_setup(): def two_node_setup():
"""Start tracker, two stub nodes, and gateway; yield (gateway_url, node_a, node_b).""" """Start tracker, two stub nodes, and gateway; yield (gateway_url, node_a, node_b)."""
tracker = TrackerServer() tracker = TrackerServer(enable_billing=False)
tracker_port = tracker.start() tracker_port = tracker.start()
tracker_url = f"http://127.0.0.1:{tracker_port}" tracker_url = f"http://127.0.0.1:{tracker_port}"

View File

@@ -1075,6 +1075,190 @@ def test_partial_snapshot_loader_materializes_only_assigned_tensors(tmp_path):
assert model.model.rotary_emb.to_calls == ["cpu:0"] assert model.model.rotary_emb.to_calls == ["cpu:0"]
def _build_partial_snapshot_fixture(snapshot_dir: Path, *, num_layers: int = 4) -> dict[str, dict[str, int]]:
"""Minimal HF snapshot with per-tensor numel metadata for memory-scaling tests."""
weight_map: dict[str, str] = {
"model.embed_tokens.weight": "shard-head.safetensors",
"model.norm.weight": "shard-tail.safetensors",
"lm_head.weight": "shard-tail.safetensors",
}
tensor_numel: dict[str, dict[str, int]] = {
"shard-head.safetensors": {"model.embed_tokens.weight": 10_000},
"shard-tail.safetensors": {
"model.norm.weight": 512,
"lm_head.weight": 10_000,
},
}
for layer in range(num_layers):
rel = f"shard-layer-{layer}.safetensors"
weight_map[f"model.layers.{layer}.self_attn.q_proj.weight"] = rel
weight_map[f"model.layers.{layer}.mlp.down_proj.weight"] = rel
per_layer = 1_000 * (layer + 1)
tensor_numel[rel] = {
f"model.layers.{layer}.self_attn.q_proj.weight": per_layer,
f"model.layers.{layer}.mlp.down_proj.weight": per_layer,
}
(snapshot_dir / rel).write_bytes(b"stub")
(snapshot_dir / "config.json").write_text(json.dumps({"num_hidden_layers": num_layers}))
(snapshot_dir / "model.safetensors.index.json").write_text(json.dumps({"weight_map": weight_map}))
(snapshot_dir / "shard-head.safetensors").write_bytes(b"stub")
(snapshot_dir / "shard-tail.safetensors").write_bytes(b"stub")
return tensor_numel
def _partial_load_materialized_numel(
snapshot_dir: Path,
shard_start: int,
shard_end: int,
tensor_numel: dict[str, dict[str, int]],
) -> int:
"""Return the summed numel of checkpoint tensors assigned to one shard load."""
class FakeModule:
def to(self, device):
return self
class FakeModel:
def __init__(self, num_layers: int):
self.model = types.SimpleNamespace(
embed_tokens=FakeModule(),
layers=[FakeModule() for _ in range(num_layers)],
rotary_emb=FakeModule(),
norm=FakeModule(),
)
self.lm_head = FakeModule()
def tie_weights(self):
pass
class AutoConfigStub:
@staticmethod
def from_pretrained(model_id):
assert model_id == str(snapshot_dir)
return types.SimpleNamespace(num_hidden_layers=num_layers)
class AutoModelStub:
@staticmethod
def from_config(cfg, torch_dtype=None):
return FakeModel(cfg.num_hidden_layers)
class UnusedContext:
def __enter__(self):
return None
def __exit__(self, exc_type, exc, tb):
return False
num_layers = json.loads((snapshot_dir / "config.json").read_text())["num_hidden_layers"]
loaded_numel = 0
def fake_set_tensor(module, tensor_name, device, value=None, dtype=None):
nonlocal loaded_numel
loaded_numel += int(getattr(value, "numel", lambda: 0)())
class FakeTensor:
def __init__(self, numel: int):
self._numel = numel
def numel(self) -> int:
return self._numel
class FakeSafeOpen:
def __init__(self, filename, framework, device):
self.filename = Path(filename).name
def __enter__(self):
return self
def __exit__(self, exc_type, exc, tb):
return False
def get_tensor(self, tensor_name):
return FakeTensor(tensor_numel[self.filename][tensor_name])
_load_partial_model_from_snapshot(
AutoConfigStub,
AutoModelStub,
types.SimpleNamespace(),
str(snapshot_dir),
shard_start,
shard_end,
"bf16",
"cpu:0",
init_empty_weights_fn=UnusedContext,
set_tensor_fn=fake_set_tensor,
safe_open_fn=FakeSafeOpen,
)
return loaded_numel
def test_partial_snapshot_resident_weight_numel_scales_with_shard(tmp_path):
"Partial load materializes only assigned-layer weights, not the full checkpoint.\n\nTags: model, node, real-inference"
snapshot_dir = tmp_path / "snapshot"
snapshot_dir.mkdir()
tensor_numel = _build_partial_snapshot_fixture(snapshot_dir, num_layers=4)
middle_numel = _partial_load_materialized_numel(snapshot_dir, 1, 1, tensor_numel)
full_numel = _partial_load_materialized_numel(snapshot_dir, 0, 3, tensor_numel)
# Layer 1 only: two tensors at 2000 numel each.
assert middle_numel == 4_000
# Head embed + four layers (two tensors each, increasing sizes) + tail norm/lm_head.
assert full_numel == 10_000 + 512 + 10_000 + 2_000 + 4_000 + 6_000 + 8_000
assert middle_numel < full_numel // 4
@pytest.mark.integration
def test_partial_snapshot_materialized_param_count_with_real_torch(tmp_path):
"When torch is installed, partial load leaves unassigned params on meta device.\n\nTags: model, node, real-inference"
torch = _require_functional_torch()
pytest.importorskip("transformers")
pytest.importorskip("safetensors")
pytest.importorskip("accelerate")
from safetensors.torch import save_file
from transformers import AutoConfig, AutoModelForCausalLM, GPT2Config
n_layer = 4
n_embd = 16
snapshot_dir = tmp_path / "snapshot"
snapshot_dir.mkdir()
config = GPT2Config(n_layer=n_layer, n_embd=n_embd, n_head=2, n_positions=32)
(snapshot_dir / "config.json").write_text(config.to_json_string())
weight_map: dict[str, str] = {}
for layer in range(n_layer):
key = f"transformer.h.{layer}.attn.c_attn.weight"
rel = f"layer-{layer}.safetensors"
weight_map[key] = rel
save_file({key: torch.ones(n_embd, 3 * n_embd)}, snapshot_dir / rel)
(snapshot_dir / "model.safetensors.index.json").write_text(json.dumps({"weight_map": weight_map}))
def count_materialized(model) -> tuple[int, int]:
materialized = 0
meta = 0
for param in model.parameters():
if param.device.type == "meta":
meta += param.numel()
else:
materialized += param.numel()
return materialized, meta
middle = _load_partial_model_from_snapshot(
AutoConfig,
AutoModelForCausalLM,
torch,
str(snapshot_dir),
1,
1,
torch.float32,
torch.device("cpu"),
)
mid_mat, mid_meta = count_materialized(middle)
assert mid_mat == n_embd * 3 * n_embd
assert mid_meta > mid_mat * 10
def test_partial_snapshot_loader_requires_known_layer_count(tmp_path): def test_partial_snapshot_loader_requires_known_layer_count(tmp_path):
"Partial snapshot loader requires known layer count\n\nTags: model, node, real-inference" "Partial snapshot loader requires known layer count\n\nTags: model, node, real-inference"
snapshot_dir = tmp_path / "snapshot" snapshot_dir = tmp_path / "snapshot"

View File

@@ -32,12 +32,18 @@ def test_matrix_reports_direct_relay_prefill_decode_and_machine_readable_metrics
assert {"p50_latency_ms", "p95_latency_ms", "payload_bytes", "compression_ratio", assert {"p50_latency_ms", "p95_latency_ms", "payload_bytes", "compression_ratio",
"connection_attempts", "p95_queue_wait_ms"} <= set(run["phases"]["decode"]) "connection_attempts", "p95_queue_wait_ms"} <= set(run["phases"]["decode"])
sample = run["samples"][0] sample = run["samples"][0]
assert sample["model_ms"] > 0
assert sample["encode_ms"] > 0
assert sample["activation_decode_ms"] > 0
assert sample["framing_ms"] > 0 assert sample["framing_ms"] > 0
assert sample["metadata_ms"] > 0 assert sample["metadata_ms"] > 0
assert sample["copy_allocation_ms"] > 0 assert sample["copy_allocation_ms"] > 0
assert sample["copy_allocation_bytes"] >= sample["payload_bytes"] assert sample["copy_allocation_bytes"] >= sample["payload_bytes"]
assert sample["local_http_forwarding_ms"] > 0
assert len(run["samples"]) == 1 + len(run["output_tokens"]) assert len(run["samples"]) == 1 + len(run["output_tokens"])
assert {"tokens_per_sec", "bytes_per_token", "compression_cpu_ms", "peak_buffered_bytes"} <= set(run["phases"]["decode"]) assert {"tokens_per_sec", "bytes_per_token", "compression_cpu_ms", "peak_buffered_bytes",
"model_execution_ms", "activation_encoding_ms", "activation_decoding_ms",
"local_http_forwarding_ms"} <= set(run["phases"]["decode"])
def test_cached_sessions_reuse_one_connection_and_preserve_stub_tokens(): def test_cached_sessions_reuse_one_connection_and_preserve_stub_tokens():
@@ -74,7 +80,10 @@ def test_cli_writes_json_artifact_and_human_summary(tmp_path, capsys):
report = json.loads(output.read_text()) report = json.loads(output.read_text())
assert report["schema_version"] == 1 assert report["schema_version"] == 1
assert "Route Session benchmark" in capsys.readouterr().out assert "Route Session benchmark" in capsys.readouterr().out
assert "relay" in format_summary(report) summary = format_summary(report)
assert "relay" in summary
assert "model/encode/decode" in summary
assert "HTTP" in summary
def test_performance_gate_checks_comparison_identity_session_and_cleanup(): def test_performance_gate_checks_comparison_identity_session_and_cleanup():

View File

@@ -12,7 +12,12 @@ import urllib.request
import pytest import pytest
from meshnet_contracts import LocalSolanaContracts from meshnet_contracts import LocalSolanaContracts
from meshnet_tracker.billing import BillingLedger from meshnet_tracker.billing import (
BillingLedger,
DEFAULT_PAYOUT_DUST_FLOOR,
DEFAULT_PAYOUT_THRESHOLD,
DEFAULT_SETTLE_PERIOD,
)
from meshnet_tracker.server import TrackerServer from meshnet_tracker.server import TrackerServer
MODEL = "stub-model" MODEL = "stub-model"
@@ -172,3 +177,115 @@ def test_non_leader_never_signs():
assert ledger.get_node_pending("wallet-a") > 0 assert ledger.get_node_pending("wallet-a") > 0
finally: finally:
tracker.stop() tracker.stop()
# Production defaults — sourced from billing.py (cli.py uses the same constants)
PROD_SETTLE_PERIOD = DEFAULT_SETTLE_PERIOD
PROD_PAYOUT_THRESHOLD = DEFAULT_PAYOUT_THRESHOLD
PROD_DUST_FLOOR = DEFAULT_PAYOUT_DUST_FLOOR
def _set_pending_since(ledger: BillingLedger, wallet: str, ts: float) -> None:
with ledger._lock:
ledger._pending_since[wallet] = ts
def test_prod_settlement_constants():
"Production settlement policy is 24 h period, 5 USDT threshold, 0.01 dust floor.\n\nTags: billing, settlement"
assert DEFAULT_SETTLE_PERIOD == 86400.0
assert DEFAULT_PAYOUT_THRESHOLD == 5.0
assert DEFAULT_PAYOUT_DUST_FLOOR == 0.01
def test_payables_24h_period_not_due_before_elapsed():
"Sub-threshold pending is not payable until the 24 h period elapses.\n\nTags: billing, settlement"
ledger = BillingLedger(starting_credit=100.0, default_price_per_1k=0.02)
ledger.charge_request("client", MODEL, 1000, [("node-wallet", 12)])
assert ledger.get_node_pending("node-wallet") == pytest.approx(0.018)
t0 = 1_700_000_000.0
_set_pending_since(ledger, "node-wallet", t0)
due_early = ledger.payables(
threshold=PROD_PAYOUT_THRESHOLD,
max_period=PROD_SETTLE_PERIOD,
dust_floor=PROD_DUST_FLOOR,
now=t0 + PROD_SETTLE_PERIOD - 1.0,
)
assert due_early == []
def test_payables_24h_period_triggers_at_boundary():
"Sub-threshold pending becomes payable exactly after the 24 h period.\n\nTags: billing, settlement"
ledger = BillingLedger(starting_credit=100.0, default_price_per_1k=0.02)
ledger.charge_request("client", MODEL, 1000, [("node-wallet", 12)])
pending = ledger.get_node_pending("node-wallet")
t0 = 1_700_000_000.0
_set_pending_since(ledger, "node-wallet", t0)
due = ledger.payables(
threshold=PROD_PAYOUT_THRESHOLD,
max_period=PROD_SETTLE_PERIOD,
dust_floor=PROD_DUST_FLOOR,
now=t0 + PROD_SETTLE_PERIOD,
)
assert due == [("node-wallet", pytest.approx(pending))]
def test_payables_threshold_triggers_before_24h_period():
"Pending ≥ 5 USDT pays immediately without waiting the 24 h period.\n\nTags: billing, settlement"
ledger = BillingLedger(starting_credit=1000.0, default_price_per_1k=0.02)
# cost = 300k * 0.02 / 1k = 6.0 USDT → node share 5.4 USDT
ledger.charge_request("client", MODEL, 300_000, [("node-wallet", 12)])
assert ledger.get_node_pending("node-wallet") >= PROD_PAYOUT_THRESHOLD
due = ledger.payables(
threshold=PROD_PAYOUT_THRESHOLD,
max_period=PROD_SETTLE_PERIOD,
dust_floor=PROD_DUST_FLOOR,
now=time.time(),
)
assert len(due) == 1
assert due[0][0] == "node-wallet"
def test_prod_defaults_hold_sub_threshold_pending():
"Tracker with prod 24 h / 5 USDT settings does not pay tiny balances early.\n\nTags: billing, settlement"
ledger = BillingLedger(starting_credit=100.0, default_price_per_1k=0.02)
ledger.charge_request("client", MODEL, 1000, [("node-wallet", 12)])
treasury = _FakePayoutTreasury()
tracker = _make_tracker(
ledger,
treasury,
threshold=PROD_PAYOUT_THRESHOLD,
period=PROD_SETTLE_PERIOD,
dust=PROD_DUST_FLOOR,
)
tracker.start()
try:
time.sleep(0.5)
assert treasury.batches == []
assert ledger.get_node_pending("node-wallet") > 0
finally:
tracker.stop()
def test_prod_threshold_triggers_immediate_payout():
"Tracker with prod 24 h period still pays when pending ≥ 5 USDT.\n\nTags: billing, settlement"
ledger = BillingLedger(starting_credit=1000.0, default_price_per_1k=0.02)
ledger.charge_request("client", MODEL, 300_000, [("node-wallet", 12)])
treasury = _FakePayoutTreasury()
tracker = _make_tracker(
ledger,
treasury,
threshold=PROD_PAYOUT_THRESHOLD,
period=PROD_SETTLE_PERIOD,
dust=PROD_DUST_FLOOR,
)
tracker.start()
try:
assert _wait_for(lambda: treasury.batches)
assert ledger.get_node_pending("node-wallet") == pytest.approx(0.0)
assert treasury.batches[0][0][0] == "node-wallet"
assert treasury.batches[0][0][1] >= PROD_PAYOUT_THRESHOLD
finally:
tracker.stop()

View File

@@ -28,6 +28,11 @@ from meshnet_tracker.server import (
_TEST_HIVE_SECRET = "test-hive-secret" _TEST_HIVE_SECRET = "test-hive-secret"
def _tracker(**kwargs):
kwargs.setdefault("enable_billing", False)
return TrackerServer(**kwargs)
def _post_json(url: str, payload: dict) -> dict: def _post_json(url: str, payload: dict) -> dict:
data = json.dumps(payload).encode() data = json.dumps(payload).encode()
req = urllib.request.Request( req = urllib.request.Request(
@@ -67,7 +72,7 @@ def test_tracker_send_json_ignores_broken_pipe_after_client_disconnect():
def test_tracker_exposes_registered_model_metadata(): def test_tracker_exposes_registered_model_metadata():
"Tracker exposes registered model metadata\n\nTags: http, routing, tracker" "Tracker exposes registered model metadata\n\nTags: http, routing, tracker"
tracker = TrackerServer() tracker = _tracker()
port = tracker.start() port = tracker.start()
url = f"http://127.0.0.1:{port}" url = f"http://127.0.0.1:{port}"
try: try:
@@ -114,7 +119,7 @@ def test_tracker_exposes_registered_model_metadata():
def test_tracker_lists_recommended_kimi_before_nodes_register(): def test_tracker_lists_recommended_kimi_before_nodes_register():
"Tracker lists recommended kimi before nodes register\n\nTags: http, routing, tracker" "Tracker lists recommended kimi before nodes register\n\nTags: http, routing, tracker"
tracker = TrackerServer() tracker = _tracker()
port = tracker.start() port = tracker.start()
url = f"http://127.0.0.1:{port}" url = f"http://127.0.0.1:{port}"
try: try:
@@ -138,7 +143,7 @@ def test_tracker_lists_recommended_qwen_27b_before_nodes_register():
Tags: http, routing, tracker Tags: http, routing, tracker
""" """
tracker = TrackerServer() tracker = _tracker()
port = tracker.start() port = tracker.start()
url = f"http://127.0.0.1:{port}" url = f"http://127.0.0.1:{port}"
try: try:
@@ -177,7 +182,7 @@ def test_quantization_coverage_uses_higher_precision_shards():
def test_network_map_exposes_pool_size_and_speed_summary(): def test_network_map_exposes_pool_size_and_speed_summary():
"Network map exposes pool size and speed summary\n\nTags: http, routing, tracker" "Network map exposes pool size and speed summary\n\nTags: http, routing, tracker"
tracker = TrackerServer() tracker = _tracker()
port = tracker.start() port = tracker.start()
url = f"http://127.0.0.1:{port}" url = f"http://127.0.0.1:{port}"
try: try:
@@ -219,7 +224,7 @@ def test_network_map_exposes_pool_size_and_speed_summary():
def test_network_map_exposes_served_model_copy_count(): def test_network_map_exposes_served_model_copy_count():
"Network map exposes served model copy count\n\nTags: http, routing, tracker" "Network map exposes served model copy count\n\nTags: http, routing, tracker"
tracker = TrackerServer() tracker = _tracker()
port = tracker.start() port = tracker.start()
url = f"http://127.0.0.1:{port}" url = f"http://127.0.0.1:{port}"
try: try:
@@ -273,7 +278,7 @@ def test_network_map_exposes_served_model_copy_count():
def test_recommended_kimi_becomes_deployable_when_pool_is_large_enough(): def test_recommended_kimi_becomes_deployable_when_pool_is_large_enough():
"Recommended kimi becomes deployable when pool is large enough\n\nTags: http, routing, tracker" "Recommended kimi becomes deployable when pool is large enough\n\nTags: http, routing, tracker"
tracker = TrackerServer() tracker = _tracker()
port = tracker.start() port = tracker.start()
url = f"http://127.0.0.1:{port}" url = f"http://127.0.0.1:{port}"
gib = 1024 * 1024 * 1024 gib = 1024 * 1024 * 1024
@@ -302,7 +307,7 @@ def test_recommended_kimi_becomes_deployable_when_pool_is_large_enough():
def test_network_assign_can_start_recommended_kimi_by_short_name(): def test_network_assign_can_start_recommended_kimi_by_short_name():
"Network assign can start recommended kimi by short name\n\nTags: http, routing, tracker" "Network assign can start recommended kimi by short name\n\nTags: http, routing, tracker"
tracker = TrackerServer() tracker = _tracker()
port = tracker.start() port = tracker.start()
url = f"http://127.0.0.1:{port}" url = f"http://127.0.0.1:{port}"
try: try:
@@ -344,7 +349,7 @@ def test_tracker_serves_health_while_proxy_request_is_in_flight():
slow_node = http.server.HTTPServer(("127.0.0.1", 0), SlowChatHandler) slow_node = http.server.HTTPServer(("127.0.0.1", 0), SlowChatHandler)
slow_thread = threading.Thread(target=slow_node.serve_forever, daemon=True) slow_thread = threading.Thread(target=slow_node.serve_forever, daemon=True)
slow_thread.start() slow_thread.start()
tracker = TrackerServer(heartbeat_timeout=60.0) tracker = _tracker(heartbeat_timeout=60.0)
tracker_port = tracker.start() tracker_port = tracker.start()
proxy_error = [] proxy_error = []
try: try:
@@ -410,7 +415,7 @@ def test_tracker_route_log_counts_proxy_inflight_requests():
slow_node = http.server.HTTPServer(("127.0.0.1", 0), SlowChatHandler) slow_node = http.server.HTTPServer(("127.0.0.1", 0), SlowChatHandler)
slow_thread = threading.Thread(target=slow_node.serve_forever, daemon=True) slow_thread = threading.Thread(target=slow_node.serve_forever, daemon=True)
slow_thread.start() slow_thread.start()
tracker = TrackerServer(heartbeat_timeout=60.0) tracker = _tracker(heartbeat_timeout=60.0)
tracker_port = tracker.start() tracker_port = tracker.start()
errors = [] errors = []
try: try:
@@ -498,7 +503,7 @@ def test_tracker_logs_stream_progress_before_request_completes():
node = http.server.HTTPServer(("127.0.0.1", 0), StreamingChatHandler) node = http.server.HTTPServer(("127.0.0.1", 0), StreamingChatHandler)
node_thread = threading.Thread(target=node.serve_forever, daemon=True) node_thread = threading.Thread(target=node.serve_forever, daemon=True)
node_thread.start() node_thread.start()
tracker = TrackerServer(heartbeat_timeout=60.0) tracker = _tracker(heartbeat_timeout=60.0)
tracker_port = tracker.start() tracker_port = tracker.start()
response = None response = None
try: try:
@@ -590,7 +595,7 @@ def test_tracker_stream_survives_idle_gap_between_sse_chunks():
node = http.server.HTTPServer(("127.0.0.1", 0), IdleStreamingChatHandler) node = http.server.HTTPServer(("127.0.0.1", 0), IdleStreamingChatHandler)
node_thread = threading.Thread(target=node.serve_forever, daemon=True) node_thread = threading.Thread(target=node.serve_forever, daemon=True)
node_thread.start() node_thread.start()
tracker = TrackerServer(heartbeat_timeout=60.0) tracker = _tracker(heartbeat_timeout=60.0)
tracker_port = tracker.start() tracker_port = tracker.start()
response = None response = None
try: try:
@@ -657,7 +662,7 @@ def test_tracker_dashboard_can_cancel_inflight_proxy():
node = http.server.HTTPServer(("127.0.0.1", 0), StreamingChatHandler) node = http.server.HTTPServer(("127.0.0.1", 0), StreamingChatHandler)
node_thread = threading.Thread(target=node.serve_forever, daemon=True) node_thread = threading.Thread(target=node.serve_forever, daemon=True)
node_thread.start() node_thread.start()
tracker = TrackerServer(heartbeat_timeout=60.0) tracker = _tracker(heartbeat_timeout=60.0)
tracker_port = tracker.start() tracker_port = tracker.start()
response = None response = None
request_id = None request_id = None
@@ -723,7 +728,7 @@ def test_tracker_dashboard_can_cancel_inflight_proxy():
def test_tracker_routes_hf_model_alias_from_quickstart(): def test_tracker_routes_hf_model_alias_from_quickstart():
"The documented qwen2.5-0.5b alias resolves a full HF repo registration.\n\nTags: http, routing, tracker" "The documented qwen2.5-0.5b alias resolves a full HF repo registration.\n\nTags: http, routing, tracker"
tracker = TrackerServer() tracker = _tracker()
tracker_port = tracker.start() tracker_port = tracker.start()
try: try:
_post_json( _post_json(
@@ -775,7 +780,7 @@ def test_tracker_proxy_accepts_hf_model_alias_from_quickstart():
node = http.server.HTTPServer(("127.0.0.1", 0), ChatHandler) node = http.server.HTTPServer(("127.0.0.1", 0), ChatHandler)
node_thread = threading.Thread(target=node.serve_forever, daemon=True) node_thread = threading.Thread(target=node.serve_forever, daemon=True)
node_thread.start() node_thread.start()
tracker = TrackerServer() tracker = _tracker()
tracker_port = tracker.start() tracker_port = tracker.start()
try: try:
_post_json( _post_json(
@@ -833,7 +838,7 @@ def test_tracker_proxy_routes_split_preset_nodes_by_alias():
head = http.server.HTTPServer(("127.0.0.1", 0), ChatHandler) head = http.server.HTTPServer(("127.0.0.1", 0), ChatHandler)
head_thread = threading.Thread(target=head.serve_forever, daemon=True) head_thread = threading.Thread(target=head.serve_forever, daemon=True)
head_thread.start() head_thread.start()
tracker = TrackerServer(model_presets={ tracker = _tracker(model_presets={
"qwen3.6-35b-a3b": { "qwen3.6-35b-a3b": {
"layers_start": 0, "layers_start": 0,
"layers_end": 39, "layers_end": 39,
@@ -889,7 +894,7 @@ def test_tracker_proxy_routes_split_preset_nodes_by_alias():
def test_tracker_route_endpoint_routes_split_preset_nodes_by_alias(): def test_tracker_route_endpoint_routes_split_preset_nodes_by_alias():
"Tracker route endpoint routes split preset nodes by alias\n\nTags: http, routing, tracker" "Tracker route endpoint routes split preset nodes by alias\n\nTags: http, routing, tracker"
tracker = TrackerServer(model_presets={ tracker = _tracker(model_presets={
"qwen3.6-35b-a3b": { "qwen3.6-35b-a3b": {
"layers_start": 0, "layers_start": 0,
"layers_end": 39, "layers_end": 39,
@@ -936,7 +941,7 @@ def test_tracker_route_endpoint_routes_split_preset_nodes_by_alias():
def test_tracker_route_endpoint_ignores_model_case_and_outer_whitespace(): def test_tracker_route_endpoint_ignores_model_case_and_outer_whitespace():
"Tracker route endpoint ignores model case and outer whitespace\n\nTags: http, routing, tracker" "Tracker route endpoint ignores model case and outer whitespace\n\nTags: http, routing, tracker"
tracker = TrackerServer(model_presets={ tracker = _tracker(model_presets={
"qwen3.6-35b-a3b": { "qwen3.6-35b-a3b": {
"layers_start": 0, "layers_start": 0,
"layers_end": 39, "layers_end": 39,
@@ -970,7 +975,7 @@ def test_tracker_route_endpoint_ignores_model_case_and_outer_whitespace():
def test_tracker_route_prefers_distributed_over_single_full_shard(): def test_tracker_route_prefers_distributed_over_single_full_shard():
"When a full 0-39 node and a partial 0-21 head coexist, /v1/route\n\nTags: http, routing, tracker" "When a full 0-39 node and a partial 0-21 head coexist, /v1/route\n\nTags: http, routing, tracker"
tracker = TrackerServer(model_presets={ tracker = _tracker(model_presets={
"qwen3.6-35b-a3b": { "qwen3.6-35b-a3b": {
"layers_start": 0, "layers_start": 0,
"layers_end": 39, "layers_end": 39,
@@ -1044,7 +1049,7 @@ def test_tracker_proxy_ignores_model_case_and_outer_whitespace():
node = http.server.HTTPServer(("127.0.0.1", 0), ChatHandler) node = http.server.HTTPServer(("127.0.0.1", 0), ChatHandler)
node_thread = threading.Thread(target=node.serve_forever, daemon=True) node_thread = threading.Thread(target=node.serve_forever, daemon=True)
node_thread.start() node_thread.start()
tracker = TrackerServer(model_presets={ tracker = _tracker(model_presets={
"qwen3.6-35b-a3b": { "qwen3.6-35b-a3b": {
"layers_start": 0, "layers_start": 0,
"layers_end": 39, "layers_end": 39,
@@ -1083,7 +1088,7 @@ def test_tracker_proxy_ignores_model_case_and_outer_whitespace():
def test_tracker_registration_node_id_includes_wallet_prefix_and_stable_suffix(): def test_tracker_registration_node_id_includes_wallet_prefix_and_stable_suffix():
"Tracker registration node id includes wallet prefix and stable suffix\n\nTags: http, routing, security, tracker, wallet" "Tracker registration node id includes wallet prefix and stable suffix\n\nTags: http, routing, security, tracker, wallet"
tracker = TrackerServer() tracker = _tracker()
tracker_port = tracker.start() tracker_port = tracker.start()
wallet = "7j77FsPY1evV8tuf4Z73AVrWwxBEW1pvKwi4EvcRD3g" wallet = "7j77FsPY1evV8tuf4Z73AVrWwxBEW1pvKwi4EvcRD3g"
try: try:
@@ -1124,7 +1129,7 @@ def test_tracker_registration_node_id_includes_wallet_prefix_and_stable_suffix()
def test_tracker_node_registration(): def test_tracker_node_registration():
"A node can register with the tracker and receives a node_id.\n\nTags: http, routing, tracker" "A node can register with the tracker and receives a node_id.\n\nTags: http, routing, tracker"
tracker = TrackerServer() tracker = _tracker()
tracker_port = tracker.start() tracker_port = tracker.start()
try: try:
resp = _post_json( resp = _post_json(
@@ -1146,7 +1151,7 @@ def test_tracker_node_registration():
def test_tracker_route_selection(): def test_tracker_route_selection():
"Tracker returns ordered route when nodes collectively cover all layers.\n\nTags: http, routing, tracker" "Tracker returns ordered route when nodes collectively cover all layers.\n\nTags: http, routing, tracker"
tracker = TrackerServer() tracker = _tracker()
tracker_port = tracker.start() tracker_port = tracker.start()
try: try:
_post_json( _post_json(
@@ -1174,7 +1179,7 @@ def test_tracker_route_selection():
def test_tracker_route_error_no_coverage(): def test_tracker_route_error_no_coverage():
"Tracker returns 503 when registered nodes do not cover all required layers.\n\nTags: http, routing, tracker" "Tracker returns 503 when registered nodes do not cover all required layers.\n\nTags: http, routing, tracker"
tracker = TrackerServer() tracker = _tracker()
tracker_port = tracker.start() tracker_port = tracker.start()
try: try:
# Only half the layers covered — route must fail. # Only half the layers covered — route must fail.
@@ -1197,7 +1202,7 @@ def test_tracker_route_error_no_coverage():
def test_tracker_coverage_endpoint_reports_uncovered_ranges(): def test_tracker_coverage_endpoint_reports_uncovered_ranges():
"Coverage endpoint returns compressed layer ranges with node counts.\n\nTags: http, routing, tracker" "Coverage endpoint returns compressed layer ranges with node counts.\n\nTags: http, routing, tracker"
tracker = TrackerServer(model_presets={ tracker = _tracker(model_presets={
"tiny-model": { "tiny-model": {
"total_layers": 6, "total_layers": 6,
"bytes_per_layer": {"bfloat16": 1_000}, "bytes_per_layer": {"bfloat16": 1_000},
@@ -1242,7 +1247,7 @@ def test_tracker_coverage_endpoint_reports_uncovered_ranges():
def test_tracker_coverage_endpoint_accepts_registered_hf_repo_or_short_name(): def test_tracker_coverage_endpoint_accepts_registered_hf_repo_or_short_name():
"Coverage endpoint supports real HF models registered outside preset catalog.\n\nTags: http, routing, tracker" "Coverage endpoint supports real HF models registered outside preset catalog.\n\nTags: http, routing, tracker"
tracker = TrackerServer() tracker = _tracker()
tracker_port = tracker.start() tracker_port = tracker.start()
try: try:
_post_json( _post_json(
@@ -1279,7 +1284,7 @@ def test_tracker_coverage_endpoint_accepts_registered_hf_repo_or_short_name():
def test_tracker_models_endpoint_lists_registered_hf_repo_and_short_name_alias(): def test_tracker_models_endpoint_lists_registered_hf_repo_and_short_name_alias():
"Tracker models endpoint lists registered hf repo and short name alias\n\nTags: http, routing, tracker" "Tracker models endpoint lists registered hf repo and short name alias\n\nTags: http, routing, tracker"
tracker = TrackerServer() tracker = _tracker()
tracker_port = tracker.start() tracker_port = tracker.start()
try: try:
_post_json( _post_json(
@@ -1302,7 +1307,7 @@ def test_tracker_models_endpoint_lists_registered_hf_repo_and_short_name_alias()
def test_tracker_auto_assigns_new_node_to_uncovered_range_first(): def test_tracker_auto_assigns_new_node_to_uncovered_range_first():
"Capability-driven registration fills the first uncovered layer gap.\n\nTags: http, routing, tracker" "Capability-driven registration fills the first uncovered layer gap.\n\nTags: http, routing, tracker"
tracker = TrackerServer(model_presets={ tracker = _tracker(model_presets={
"tiny-model": { "tiny-model": {
"total_layers": 8, "total_layers": 8,
"bytes_per_layer": {"bfloat16": 1_000}, "bytes_per_layer": {"bfloat16": 1_000},
@@ -1331,7 +1336,7 @@ def test_tracker_auto_assigns_new_node_to_uncovered_range_first():
def test_tracker_speed_weighted_vram_assignment_covers_model(): def test_tracker_speed_weighted_vram_assignment_covers_model():
"Three auto-assigned nodes reach 100% coverage with widest range on largest VRAM.\n\nTags: http, routing, tracker" "Three auto-assigned nodes reach 100% coverage with widest range on largest VRAM.\n\nTags: http, routing, tracker"
tracker = TrackerServer(model_presets={ tracker = _tracker(model_presets={
"tiny-model": { "tiny-model": {
"total_layers": 12, "total_layers": 12,
"bytes_per_layer": {"bfloat16": 1_000}, "bytes_per_layer": {"bfloat16": 1_000},
@@ -1375,7 +1380,7 @@ def test_tracker_speed_weighted_vram_assignment_covers_model():
def test_tracker_speed_is_primary_when_both_nodes_can_cover_gap(): def test_tracker_speed_is_primary_when_both_nodes_can_cover_gap():
"Tracker speed is primary when both nodes can cover gap\n\nTags: http, routing, tracker" "Tracker speed is primary when both nodes can cover gap\n\nTags: http, routing, tracker"
tracker = TrackerServer(model_presets={ tracker = _tracker(model_presets={
"tiny-model": { "tiny-model": {
"total_layers": 12, "total_layers": 12,
"bytes_per_layer": {"bfloat16": 1_000}, "bytes_per_layer": {"bfloat16": 1_000},
@@ -1410,7 +1415,7 @@ def test_tracker_speed_is_primary_when_both_nodes_can_cover_gap():
def test_tracker_registration_directive_is_not_replayed_on_heartbeat(): def test_tracker_registration_directive_is_not_replayed_on_heartbeat():
"Tracker registration directive is not replayed on heartbeat\n\nTags: http, routing, tracker" "Tracker registration directive is not replayed on heartbeat\n\nTags: http, routing, tracker"
tracker = TrackerServer(model_presets={ tracker = _tracker(model_presets={
"tiny-model": { "tiny-model": {
"total_layers": 2, "total_layers": 2,
"bytes_per_layer": {"bfloat16": 1_000}, "bytes_per_layer": {"bfloat16": 1_000},
@@ -1434,7 +1439,7 @@ def test_tracker_registration_directive_is_not_replayed_on_heartbeat():
def test_tracker_pool_join_adds_redundant_copy_without_splitting_incumbent(): def test_tracker_pool_join_adds_redundant_copy_without_splitting_incumbent():
"A second managed node with capacity for the full model keeps the first copy intact.\n\nTags: http, routing, tracker" "A second managed node with capacity for the full model keeps the first copy intact.\n\nTags: http, routing, tracker"
tracker = TrackerServer(model_presets={ tracker = _tracker(model_presets={
"tiny-model": { "tiny-model": {
"total_layers": 4, "total_layers": 4,
"bytes_per_layer": {"bfloat16": 1_000}, "bytes_per_layer": {"bfloat16": 1_000},
@@ -1472,7 +1477,7 @@ def test_tracker_pool_join_adds_redundant_copy_without_splitting_incumbent():
def test_tracker_explicit_full_copy_join_preserves_existing_serving_node(): def test_tracker_explicit_full_copy_join_preserves_existing_serving_node():
"--model style joins with explicit shards add redundancy instead of reshuffling.\n\nTags: http, routing, tracker" "--model style joins with explicit shards add redundancy instead of reshuffling.\n\nTags: http, routing, tracker"
tracker = TrackerServer(heartbeat_timeout=10.0) tracker = _tracker(heartbeat_timeout=10.0)
tracker_port = tracker.start() tracker_port = tracker.start()
try: try:
base_reg = { base_reg = {
@@ -1511,7 +1516,7 @@ def test_tracker_explicit_full_copy_join_preserves_existing_serving_node():
def test_tracker_periodic_rebalance_purges_expired_nodes_without_requests(): def test_tracker_periodic_rebalance_purges_expired_nodes_without_requests():
"Tracker periodic rebalance purges expired nodes without requests\n\nTags: http, routing, tracker" "Tracker periodic rebalance purges expired nodes without requests\n\nTags: http, routing, tracker"
tracker = TrackerServer( tracker = _tracker(
heartbeat_timeout=0.05, heartbeat_timeout=0.05,
rebalance_interval=0.02, rebalance_interval=0.02,
model_presets={"tiny-model": {"total_layers": 1, "bytes_per_layer": {"bfloat16": 1_000}}}, model_presets={"tiny-model": {"total_layers": 1, "bytes_per_layer": {"bfloat16": 1_000}}},
@@ -1533,7 +1538,7 @@ def test_tracker_periodic_rebalance_purges_expired_nodes_without_requests():
def test_tracker_faster_node_receives_wider_range_when_capacity_ties(): def test_tracker_faster_node_receives_wider_range_when_capacity_ties():
"Tracker faster node receives wider range when capacity ties\n\nTags: http, routing, tracker" "Tracker faster node receives wider range when capacity ties\n\nTags: http, routing, tracker"
tracker = TrackerServer(model_presets={ tracker = _tracker(model_presets={
"tiny-model": { "tiny-model": {
"total_layers": 12, "total_layers": 12,
"bytes_per_layer": {"bfloat16": 1_000}, "bytes_per_layer": {"bfloat16": 1_000},
@@ -1564,7 +1569,7 @@ def test_tracker_faster_node_receives_wider_range_when_capacity_ties():
def test_tracker_rebalances_after_middle_range_node_timeout(): def test_tracker_rebalances_after_middle_range_node_timeout():
"Killing a middle shard queues LOAD_SHARD and restores coverage.\n\nTags: http, routing, tracker" "Killing a middle shard queues LOAD_SHARD and restores coverage.\n\nTags: http, routing, tracker"
tracker = TrackerServer( tracker = _tracker(
heartbeat_timeout=0.15, heartbeat_timeout=0.15,
model_presets={ model_presets={
"tiny-model": { "tiny-model": {
@@ -1612,7 +1617,7 @@ def test_tracker_rebalances_after_middle_range_node_timeout():
def test_tracker_rebalances_managed_hf_node_after_peer_timeout(): def test_tracker_rebalances_managed_hf_node_after_peer_timeout():
"HF nodes auto-assigned by the tracker receive LOAD_SHARD after a peer dies.\n\nTags: http, routing, tracker" "HF nodes auto-assigned by the tracker receive LOAD_SHARD after a peer dies.\n\nTags: http, routing, tracker"
tracker = TrackerServer(heartbeat_timeout=0.15, rebalance_interval=10.0) tracker = _tracker(heartbeat_timeout=0.15, rebalance_interval=10.0)
tracker_port = tracker.start() tracker_port = tracker.start()
try: try:
managed = _post_json( managed = _post_json(
@@ -1648,7 +1653,7 @@ def test_tracker_rebalances_managed_hf_node_after_peer_timeout():
def test_tracker_route_error_no_nodes(): def test_tracker_route_error_no_nodes():
"Tracker returns 503 with clear error when the registry is empty.\n\nTags: http, routing, tracker" "Tracker returns 503 with clear error when the registry is empty.\n\nTags: http, routing, tracker"
tracker = TrackerServer() tracker = _tracker()
tracker_port = tracker.start() tracker_port = tracker.start()
try: try:
try: try:
@@ -1664,7 +1669,7 @@ def test_tracker_route_error_no_nodes():
def test_tracker_heartbeat_updates_node(): def test_tracker_heartbeat_updates_node():
"Sending a heartbeat for a registered node succeeds.\n\nTags: http, routing, tracker" "Sending a heartbeat for a registered node succeeds.\n\nTags: http, routing, tracker"
tracker = TrackerServer() tracker = _tracker()
tracker_port = tracker.start() tracker_port = tracker.start()
try: try:
reg = _post_json( reg = _post_json(
@@ -1684,7 +1689,7 @@ def test_tracker_heartbeat_updates_node():
def test_tracker_heartbeat_stores_current_requests(): def test_tracker_heartbeat_stores_current_requests():
"Node-reported in-flight request snapshots appear on the network map.\n\nTags: http, routing, tracker" "Node-reported in-flight request snapshots appear on the network map.\n\nTags: http, routing, tracker"
tracker = TrackerServer() tracker = _tracker()
tracker_port = tracker.start() tracker_port = tracker.start()
try: try:
reg = _post_json( reg = _post_json(
@@ -1749,7 +1754,7 @@ def test_normalize_current_requests_sanitizes_payload():
def test_tracker_heartbeat_expiry(): def test_tracker_heartbeat_expiry():
"Nodes that miss their heartbeat window are excluded from routes.\n\nTags: http, routing, tracker" "Nodes that miss their heartbeat window are excluded from routes.\n\nTags: http, routing, tracker"
tracker = TrackerServer(heartbeat_timeout=0.05) # 50 ms tracker = _tracker(heartbeat_timeout=0.05) # 50 ms
tracker_port = tracker.start() tracker_port = tracker.start()
try: try:
_post_json( _post_json(
@@ -1778,7 +1783,7 @@ def test_tracker_heartbeat_expiry():
def test_tracker_heartbeat_expiry_removes_node_from_registry(): def test_tracker_heartbeat_expiry_removes_node_from_registry():
"Expired nodes are removed, not merely hidden from route responses.\n\nTags: http, routing, tracker" "Expired nodes are removed, not merely hidden from route responses.\n\nTags: http, routing, tracker"
tracker = TrackerServer(heartbeat_timeout=0.05) tracker = _tracker(heartbeat_timeout=0.05)
tracker_port = tracker.start() tracker_port = tracker.start()
try: try:
reg = _post_json( reg = _post_json(
@@ -1806,7 +1811,7 @@ def test_tracker_heartbeat_expiry_removes_node_from_registry():
def test_tracker_route_rejects_non_extending_overlap(): def test_tracker_route_rejects_non_extending_overlap():
"Overlapping shards that do not extend coverage cannot form a complete route.\n\nTags: http, routing, tracker" "Overlapping shards that do not extend coverage cannot form a complete route.\n\nTags: http, routing, tracker"
tracker = TrackerServer() tracker = _tracker()
tracker_port = tracker.start() tracker_port = tracker.start()
try: try:
_post_json( _post_json(
@@ -1831,7 +1836,7 @@ def test_tracker_route_rejects_non_extending_overlap():
def test_tracker_console_records_model_not_available(): def test_tracker_console_records_model_not_available():
"Tracker console records model not available\n\nTags: http, routing, tracker" "Tracker console records model not available\n\nTags: http, routing, tracker"
tracker = TrackerServer() tracker = _tracker()
tracker_port = tracker.start() tracker_port = tracker.start()
try: try:
try: try:
@@ -1856,7 +1861,7 @@ def test_tracker_console_records_model_not_available():
def test_tracker_registration_rejects_invalid_payload(): def test_tracker_registration_rejects_invalid_payload():
"Registration errors return a defined JSON 400 response.\n\nTags: http, routing, tracker" "Registration errors return a defined JSON 400 response.\n\nTags: http, routing, tracker"
tracker = TrackerServer() tracker = _tracker()
tracker_port = tracker.start() tracker_port = tracker.start()
try: try:
try: try:
@@ -1876,7 +1881,7 @@ def test_tracker_registration_rejects_invalid_payload():
def test_tracker_registration_rejects_invalid_max_loaded_shards(): def test_tracker_registration_rejects_invalid_max_loaded_shards():
"Tracker registration rejects invalid max loaded shards\n\nTags: http, routing, tracker" "Tracker registration rejects invalid max loaded shards\n\nTags: http, routing, tracker"
tracker = TrackerServer() tracker = _tracker()
tracker_port = tracker.start() tracker_port = tracker.start()
try: try:
try: try:
@@ -1896,7 +1901,7 @@ def test_tracker_registration_rejects_invalid_max_loaded_shards():
def test_network_map_exposes_node_capacity_limits(): def test_network_map_exposes_node_capacity_limits():
"Network map exposes node capacity limits\n\nTags: http, routing, tracker" "Network map exposes node capacity limits\n\nTags: http, routing, tracker"
tracker = TrackerServer(model_presets={ tracker = _tracker(model_presets={
"tiny-model": { "tiny-model": {
"total_layers": 12, "total_layers": 12,
"bytes_per_layer": {"bfloat16": 1_000}, "bytes_per_layer": {"bfloat16": 1_000},
@@ -1927,7 +1932,7 @@ def test_network_map_exposes_node_capacity_limits():
def test_tracker_capacity_uses_ram_when_node_has_no_vram(): def test_tracker_capacity_uses_ram_when_node_has_no_vram():
"CPU-only nodes should expose RAM-backed shard capacity, not default GPU capacity.\n\nTags: http, routing, tracker" "CPU-only nodes should expose RAM-backed shard capacity, not default GPU capacity.\n\nTags: http, routing, tracker"
tracker = TrackerServer(model_presets={ tracker = _tracker(model_presets={
"tiny-model": { "tiny-model": {
"total_layers": 20, "total_layers": 20,
"bytes_per_layer": {"bfloat16": 1_000}, "bytes_per_layer": {"bfloat16": 1_000},
@@ -1956,7 +1961,7 @@ def test_tracker_capacity_uses_ram_when_node_has_no_vram():
def test_rebalance_keeps_one_active_range_even_when_multiple_slots_advertised(): def test_rebalance_keeps_one_active_range_even_when_multiple_slots_advertised():
"max_loaded_shards is exposed but reserved until node runtime supports multi-range serving.\n\nTags: http, routing, tracker" "max_loaded_shards is exposed but reserved until node runtime supports multi-range serving.\n\nTags: http, routing, tracker"
tracker = TrackerServer(model_presets={ tracker = _tracker(model_presets={
"tiny-model": { "tiny-model": {
"total_layers": 8, "total_layers": 8,
"bytes_per_layer": {"bfloat16": 1_000}, "bytes_per_layer": {"bfloat16": 1_000},
@@ -1983,7 +1988,7 @@ def test_rebalance_keeps_one_active_range_even_when_multiple_slots_advertised():
def test_tracker_routes_only_nodes_for_requested_model(): def test_tracker_routes_only_nodes_for_requested_model():
"A node registered for one model cannot satisfy another model's route.\n\nTags: http, routing, tracker" "A node registered for one model cannot satisfy another model's route.\n\nTags: http, routing, tracker"
tracker = TrackerServer(model_presets={ tracker = _tracker(model_presets={
"model-a": {"layers_start": 0, "layers_end": 31}, "model-a": {"layers_start": 0, "layers_end": 31},
"model-b": {"layers_start": 0, "layers_end": 31}, "model-b": {"layers_start": 0, "layers_end": 31},
}) })
@@ -2006,7 +2011,7 @@ def test_tracker_routes_only_nodes_for_requested_model():
def test_tracker_assignment_coverage_is_model_scoped(): def test_tracker_assignment_coverage_is_model_scoped():
"Shard assignment gaps are computed only from nodes serving the same model.\n\nTags: http, routing, tracker" "Shard assignment gaps are computed only from nodes serving the same model.\n\nTags: http, routing, tracker"
tracker = TrackerServer(model_presets={ tracker = _tracker(model_presets={
"model-a": {"layers_start": 0, "layers_end": 31}, "model-a": {"layers_start": 0, "layers_end": 31},
"model-b": {"layers_start": 0, "layers_end": 31}, "model-b": {"layers_start": 0, "layers_end": 31},
}) })
@@ -2031,7 +2036,7 @@ def test_tracker_excludes_banned_wallets_from_routes():
contracts.registry.submit_stake("wallet-a", 1_000) contracts.registry.submit_stake("wallet-a", 1_000)
contracts.registry.submit_stake("wallet-b", 1_000) contracts.registry.submit_stake("wallet-b", 1_000)
contracts.registry.ban_wallet("wallet-b") contracts.registry.ban_wallet("wallet-b")
tracker = TrackerServer(contracts=contracts, minimum_stake=100) tracker = _tracker(contracts=contracts, minimum_stake=100)
tracker_port = tracker.start() tracker_port = tracker.start()
try: try:
_post_json( _post_json(
@@ -2065,7 +2070,7 @@ def test_tracker_rejects_banned_wallet_registration():
contracts = LocalSolanaContracts() contracts = LocalSolanaContracts()
contracts.registry.submit_stake("wallet-b", 1_000) contracts.registry.submit_stake("wallet-b", 1_000)
contracts.registry.ban_wallet("wallet-b") contracts.registry.ban_wallet("wallet-b")
tracker = TrackerServer(contracts=contracts, minimum_stake=100) tracker = _tracker(contracts=contracts, minimum_stake=100)
tracker_port = tracker.start() tracker_port = tracker.start()
try: try:
try: try:
@@ -2111,7 +2116,7 @@ def test_gateway_routes_even_when_compute_node_has_low_stake():
contracts = LocalSolanaContracts() contracts = LocalSolanaContracts()
contracts.registry.submit_stake("wallet-a", 1_000) contracts.registry.submit_stake("wallet-a", 1_000)
contracts.registry.submit_stake("wallet-b", 5) contracts.registry.submit_stake("wallet-b", 5)
tracker = TrackerServer() tracker = _tracker()
tracker_port = tracker.start() tracker_port = tracker.start()
tracker_url = f"http://127.0.0.1:{tracker_port}" tracker_url = f"http://127.0.0.1:{tracker_port}"
@@ -2166,7 +2171,7 @@ def test_gateway_records_compute_attribution_after_inference_session():
contracts.registry.submit_stake("wallet-a", 1_000) contracts.registry.submit_stake("wallet-a", 1_000)
contracts.registry.submit_stake("wallet-b", 1_000) contracts.registry.submit_stake("wallet-b", 1_000)
contracts.payment.fund_api_key("api-key-a", lamports=10_000) contracts.payment.fund_api_key("api-key-a", lamports=10_000)
tracker = TrackerServer(contracts=contracts, minimum_stake=100) tracker = _tracker(contracts=contracts, minimum_stake=100)
tracker_port = tracker.start() tracker_port = tracker.start()
tracker_url = f"http://127.0.0.1:{tracker_port}" tracker_url = f"http://127.0.0.1:{tracker_port}"
@@ -2294,7 +2299,7 @@ def test_gateway_returns_json_503_for_malformed_tracker_route():
def test_two_node_pipeline_via_tracker(): def test_two_node_pipeline_via_tracker():
"End-to-end: nodes register with tracker; gateway discovers route dynamically.\n\nTags: http, routing, tracker" "End-to-end: nodes register with tracker; gateway discovers route dynamically.\n\nTags: http, routing, tracker"
tracker = TrackerServer() tracker = _tracker()
tracker_port = tracker.start() tracker_port = tracker.start()
tracker_url = f"http://127.0.0.1:{tracker_port}" tracker_url = f"http://127.0.0.1:{tracker_port}"
@@ -2351,7 +2356,7 @@ def test_two_node_pipeline_via_tracker():
def test_stats_endpoint_returns_empty_models_initially(): def test_stats_endpoint_returns_empty_models_initially():
"Stats endpoint returns empty models initially\n\nTags: http, routing, tracker" "Stats endpoint returns empty models initially\n\nTags: http, routing, tracker"
tracker = TrackerServer() tracker = _tracker()
port = tracker.start() port = tracker.start()
try: try:
resp = _get_json(f"http://127.0.0.1:{port}/v1/stats") resp = _get_json(f"http://127.0.0.1:{port}/v1/stats")
@@ -2449,7 +2454,7 @@ def test_stats_sqlite_persistence_survives_restart(tmp_path):
def test_stats_gossip_endpoint_merges_peer_slice(): def test_stats_gossip_endpoint_merges_peer_slice():
"Stats gossip endpoint merges peer slice\n\nTags: gossip, http, network, routing, tracker" "Stats gossip endpoint merges peer slice\n\nTags: gossip, http, network, routing, tracker"
tracker = TrackerServer(hive_secret=_TEST_HIVE_SECRET) tracker = _tracker(hive_secret=_TEST_HIVE_SECRET)
port = tracker.start() port = tracker.start()
try: try:
peer_payload = { peer_payload = {
@@ -2610,7 +2615,7 @@ def test_select_route_reputation_never_overrides_coverage():
def test_two_stub_nodes_complete_pipeline_via_tracker(): def test_two_stub_nodes_complete_pipeline_via_tracker():
"Integration: two StubNodeServer instances serving complementary shards\n\nTags: http, routing, tracker" "Integration: two StubNodeServer instances serving complementary shards\n\nTags: http, routing, tracker"
tracker = TrackerServer(model_presets={ tracker = _tracker(model_presets={
"two-node-model": {"layers_start": 0, "layers_end": 3} "two-node-model": {"layers_start": 0, "layers_end": 3}
}) })
tracker_port = tracker.start() tracker_port = tracker.start()
@@ -2732,7 +2737,7 @@ def test_shard_heal_cycle_surviving_node_covers_dead_peers_gap(monkeypatch):
monkeypatch.setattr(torch_server, "_load_backend", fake_load) monkeypatch.setattr(torch_server, "_load_backend", fake_load)
# Use a very short timeout so Node A expires quickly. # Use a very short timeout so Node A expires quickly.
tracker = TrackerServer(heartbeat_timeout=0.15, rebalance_interval=10.0) tracker = _tracker(heartbeat_timeout=0.15, rebalance_interval=10.0)
tracker_port = tracker.start() tracker_port = tracker.start()
node_b = TorchNodeServer(backend=_FakeBackend(shard_start=12, shard_end=23)) node_b = TorchNodeServer(backend=_FakeBackend(shard_start=12, shard_end=23))
@@ -2807,7 +2812,7 @@ def test_shard_heal_cycle_surviving_node_covers_dead_peers_gap(monkeypatch):
def test_network_map_exposes_memory_pool(): def test_network_map_exposes_memory_pool():
"Network map exposes memory pool\n\nTags: http, routing, tracker" "Network map exposes memory pool\n\nTags: http, routing, tracker"
tracker = TrackerServer(model_presets={ tracker = _tracker(model_presets={
"tiny-model": { "tiny-model": {
"total_layers": 8, "total_layers": 8,
"bytes_per_layer": {"bfloat16": 1_000}, "bytes_per_layer": {"bfloat16": 1_000},
@@ -2836,7 +2841,7 @@ def test_network_map_exposes_memory_pool():
def test_same_endpoint_can_register_multiple_models(): def test_same_endpoint_can_register_multiple_models():
"Same endpoint can register multiple models\n\nTags: http, routing, tracker" "Same endpoint can register multiple models\n\nTags: http, routing, tracker"
tracker = TrackerServer() tracker = _tracker()
tracker_port = tracker.start() tracker_port = tracker.start()
try: try:
base = { base = {
@@ -2864,9 +2869,46 @@ def test_same_endpoint_can_register_multiple_models():
tracker.stop() tracker.stop()
def test_explicit_model_placement_targets_only_the_selected_node():
"An admin can add and release a model on one chosen multi-model node.\n\nTags: http, routing, tracker"
from meshnet_tracker.server import _release_model_locked, _request_model_load_locked
tracker = _tracker(model_presets={
"model-a": {"total_layers": 4, "bytes_per_layer": {"bfloat16": 1_000}, "hf_repo": "org/ModelA"},
"model-b": {"total_layers": 4, "bytes_per_layer": {"bfloat16": 1_000}, "hf_repo": "org/ModelB"},
})
tracker_port = tracker.start()
try:
registrations = []
for port in (9062, 9063):
registrations.append(_post_json(
f"http://127.0.0.1:{tracker_port}/v1/nodes/register",
{"endpoint": f"http://127.0.0.1:{port}", "model": "model-a", "hf_repo": "org/ModelA",
"num_layers": 4, "shard_start": 0, "shard_end": 3, "max_loaded_shards": 2,
"vram_bytes": 20_000, "ram_bytes": 20_000, "quantizations": ["bfloat16"],
"benchmark_tokens_per_sec": 1.0, "hardware_profile": {}, "score": 1.0},
))
selected, other = (item["node_id"] for item in registrations)
with tracker._lock:
assignment = _request_model_load_locked(tracker._server, "model-b", selected) # type: ignore[arg-type]
assert assignment is not None
assert assignment["node_id"] == selected
assert tracker._registry[selected].pending_new_assignment is not None
assert tracker._registry[other].pending_new_assignment is None
with tracker._lock:
released = _release_model_locked(tracker._server, "model-a", selected) # type: ignore[arg-type]
assert released == 1
assert len(tracker._registry[selected].pending_directives) == 2
assert tracker._registry[other].pending_directives == []
finally:
tracker.stop()
def test_scale_demanded_models_queues_add_shard_on_spare_host(): def test_scale_demanded_models_queues_add_shard_on_spare_host():
"Scale demanded models queues add shard on spare host\n\nTags: http, routing, tracker" "Scale demanded models queues add shard on spare host\n\nTags: http, routing, tracker"
tracker = TrackerServer(model_presets={ tracker = _tracker(model_presets={
"model-a": { "model-a": {
"total_layers": 4, "total_layers": 4,
"bytes_per_layer": {"bfloat16": 1_000}, "bytes_per_layer": {"bfloat16": 1_000},
@@ -2957,7 +2999,7 @@ def _proxy_chat_status(policy: str, quantization: str | None) -> tuple[int, dict
node = http.server.HTTPServer(("127.0.0.1", 0), _EchoChatHandler) node = http.server.HTTPServer(("127.0.0.1", 0), _EchoChatHandler)
node_thread = threading.Thread(target=node.serve_forever, daemon=True) node_thread = threading.Thread(target=node.serve_forever, daemon=True)
node_thread.start() node_thread.start()
tracker = TrackerServer(capability_policy=policy) tracker = _tracker(capability_policy=policy)
tracker_port = tracker.start() tracker_port = tracker.start()
try: try:
endpoint = f"http://127.0.0.1:{node.server_address[1]}" endpoint = f"http://127.0.0.1:{node.server_address[1]}"
@@ -3000,12 +3042,19 @@ def test_a_node_declaring_an_unsupported_quantization_is_never_routed():
assert status == 503 assert status == 503
def test_a_node_declaring_auto_quantization_serves_a_default_precision_request():
"'auto' is the CLI default that delegates the choice — it is not a refusal, so the node must resolve to its best advertised precision and route.\n\nTags: http, routing, tracker"
status, response = _proxy_chat_status(POLICY_COMPAT, quantization="auto")
assert status == 200
assert response["choices"][0]["message"]["content"] == "ok"
def test_a_node_declaring_a_null_quantization_is_never_routed(): def test_a_node_declaring_a_null_quantization_is_never_routed():
"An explicit null states 'no usable precision' -- only an absent field is legacy.\n\nTags: http, routing, tracker" "An explicit null states 'no usable precision' -- only an absent field is legacy.\n\nTags: http, routing, tracker"
node = http.server.HTTPServer(("127.0.0.1", 0), _EchoChatHandler) node = http.server.HTTPServer(("127.0.0.1", 0), _EchoChatHandler)
node_thread = threading.Thread(target=node.serve_forever, daemon=True) node_thread = threading.Thread(target=node.serve_forever, daemon=True)
node_thread.start() node_thread.start()
tracker = TrackerServer(capability_policy=POLICY_COMPAT) tracker = _tracker(capability_policy=POLICY_COMPAT)
tracker_port = tracker.start() tracker_port = tracker.start()
try: try:
endpoint = f"http://127.0.0.1:{node.server_address[1]}" endpoint = f"http://127.0.0.1:{node.server_address[1]}"
@@ -3028,7 +3077,7 @@ def test_a_node_declaring_a_null_quantization_is_never_routed():
def test_raft_apply_preserves_a_declared_precision_for_follower_routing(): def test_raft_apply_preserves_a_declared_precision_for_follower_routing():
"A follower that drops the declared precision would route unlike the leader.\n\nTags: raft, routing, tracker" "A follower that drops the declared precision would route unlike the leader.\n\nTags: raft, routing, tracker"
tracker = TrackerServer() tracker = _tracker()
tracker.start() tracker.start()
try: try:
tracker._raft_apply("register", { tracker._raft_apply("register", {
@@ -3051,7 +3100,7 @@ def test_raft_apply_preserves_a_declared_precision_for_follower_routing():
def test_raft_apply_grandfathers_a_legacy_registration_that_omits_quantization(): def test_raft_apply_grandfathers_a_legacy_registration_that_omits_quantization():
"The follower must reach the same legacy fallback the leader did, not a bare None.\n\nTags: raft, routing, tracker" "The follower must reach the same legacy fallback the leader did, not a bare None.\n\nTags: raft, routing, tracker"
tracker = TrackerServer() tracker = _tracker()
tracker.start() tracker.start()
try: try:
tracker._raft_apply("register", { tracker._raft_apply("register", {
@@ -3071,7 +3120,7 @@ def test_raft_apply_grandfathers_a_legacy_registration_that_omits_quantization()
def test_rebalancing_does_not_resurrect_a_node_that_declared_an_unusable_precision(): def test_rebalancing_does_not_resurrect_a_node_that_declared_an_unusable_precision():
"Managed assignment picks a node's precision, so it must not overwrite an explicit unusable one.\n\nTags: http, routing, tracker" "Managed assignment picks a node's precision, so it must not overwrite an explicit unusable one.\n\nTags: http, routing, tracker"
tracker = TrackerServer(heartbeat_timeout=0.15, rebalance_interval=10.0) tracker = _tracker(heartbeat_timeout=0.15, rebalance_interval=10.0)
tracker_port = tracker.start() tracker_port = tracker.start()
try: try:
managed = _post_json( managed = _post_json(
@@ -3120,7 +3169,7 @@ def _register_shard(tracker_port: int, port: int, **overrides) -> dict:
def test_a_node_with_an_unusable_precision_covers_no_layers(): def test_a_node_with_an_unusable_precision_covers_no_layers():
"A node no request can reach must not report its shards as served coverage.\n\nTags: http, routing, tracker" "A node no request can reach must not report its shards as served coverage.\n\nTags: http, routing, tracker"
tracker = TrackerServer(rebalance_interval=10.0) tracker = _tracker(rebalance_interval=10.0)
tracker_port = tracker.start() tracker_port = tracker.start()
try: try:
_register_shard( _register_shard(
@@ -3136,7 +3185,7 @@ def test_a_node_with_an_unusable_precision_covers_no_layers():
def test_rebalancing_fills_the_layers_an_unusable_node_only_appears_to_cover(): def test_rebalancing_fills_the_layers_an_unusable_node_only_appears_to_cover():
"Counting a dark node as coverage hides the gap, so the rebalancer never fills it.\n\nTags: http, routing, tracker" "Counting a dark node as coverage hides the gap, so the rebalancer never fills it.\n\nTags: http, routing, tracker"
tracker = TrackerServer(rebalance_interval=10.0) tracker = _tracker(rebalance_interval=10.0)
tracker_port = tracker.start() tracker_port = tracker.start()
try: try:
# The dark node claims the model's tail; the managed node holds the head. # The dark node claims the model's tail; the managed node holds the head.