popov/neuron-tai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: popov:f615b6befb9a6378f0073b0f06bb0b1e3d1b19ee
Branches Tags
popov:master popov:ralph/distributed-gguf-runtime popov:ralph/distributed-performance popov:ralph/deepseek-v4-flash-epic popov:ralph/dgr-001-performance-contract popov:cursor/fix-meshnet-node-param-parsing popov:worktree-feat+us-016
..
compare: popov:b5fa7245df360ab2266168e9294f12418b74c320
Branches Tags
popov:ralph/distributed-gguf-runtime popov:master popov:ralph/distributed-performance popov:ralph/deepseek-v4-flash-epic popov:ralph/dgr-001-performance-contract popov:cursor/fix-meshnet-node-param-parsing popov:worktree-feat+us-016

4 Commits
f615b6befb ... b5fa7245df

Author SHA1 Message Date
Dobromir Popov
b5fa7245df [verified] fix: preserve tracker precision eligibility 2026-07-13 10:27:45 +03:00
Dobromir Popov
377346c301 [verified] feat: complete Ralph task workstreams 2026-07-12 11:17:03 +03:00
Dobromir Popov
9a1b15c020 models on tracker 2026-07-12 02:44:12 +03:00
Dobromir Popov
95d79a0a16 quantizations 2026-07-12 01:33:51 +03:00
51 changed files with 7059 additions and 262 deletions
Expand all files Collapse all files

27
.scratch/proxy-stream-cancellation/PRD.md Normal file
View File

@@ -0,0 +1,27 @@
# PRD: Streaming proxy cancellation delivery
## Overview
A tracker proxy route can be selected and registered as active, but a direct SSE upstream may not deliver its first frame to the downstream client until the upstream response completes. A dashboard cancellation request made after the upstream emitted and flushed an SSE frame can therefore arrive after the tracker has already finalized and unregistered the proxy, returning 404.
## Goal
Preserve immediate SSE frame delivery and keep the active-proxy record cancelable until the streaming response actually completes or is canceled.
## Quality gates
- Run the focused tracker streaming/cancellation tests.
- Run full `python -m pytest -q` and record unrelated failures exactly.
- Do not weaken billing, disconnect handling, proxy-inflight accounting, or cancel authorization.
## User story
### PSC-001: Cancel an active direct SSE proxy
As a dashboard operator, I need a cancellation request to reach an active direct SSE proxy after its first streamed frame, so I can stop long-running inference without a race-induced 404.
## Non-goals
- Do not change route selection or quantization policy.
- Do not alter relay transport behavior unless needed to share the same cancellation invariant.
- Do not introduce client-visible buffering.

34
.scratch/proxy-stream-cancellation/issues/01-direct-sse-cancellation-race.md Normal file
View File

@@ -0,0 +1,34 @@
# PSC-001 — Direct SSE cancellation race
Status: ready-for-agent
Priority: High
## Problem
`tests/test_tracker_routing.py::test_tracker_dashboard_can_cancel_inflight_proxy` registers a streaming upstream node. The upstream writes and flushes the first SSE frame, then waits three seconds. The tracker’s client does not receive that frame until after the upstream has completed; by then `_unregister_active_proxy()` has run and the dashboard cancellation endpoint returns 404.
Observed trace:
```text
proxy route selected
proxy connected
proxy progress ... elapsed_seconds≈3
proxy complete ... elapsed_seconds≈3
POST /v1/proxy/requests/<id>/cancel → 404
```
This is a production cancellation/delivery race, not a stale test: the endpoint promises to cancel active proxy work, and the upstream had already emitted a first stream frame before cancellation was attempted.
## Acceptance criteria
- [ ] A direct SSE upstream frame is relayed and flushed to the client before the upstream completes.
- [ ] After that first frame, `/v1/proxy/requests/{request_id}/cancel` returns 200 while the stream is active.
- [ ] Cancellation closes/stops the upstream safely, finalizes inflight accounting exactly once, and records the cancellation event.
- [ ] Cancel authorization remains unchanged.
- [ ] Client disconnect and normal SSE completion retain current billing/throughput behavior.
- [ ] Regression test is deterministic and does not rely on timing races longer than necessary.
- [ ] Focused tracker routing tests and full pytest are run with unrelated failures documented.
## Likely seam
Inspect direct streaming behavior around `_handle_proxy_chat` upstream reads (`packages/tracker/meshnet_tracker/server.py`, roughly lines 3953–4019). The direct response path writes/flushed each line, but current HTTP response buffering/reading prevents the line from being observed until stream end. Fix delivery at the proxy transport boundary; do not paper over it by retaining completed proxy records indefinitely.

61
.scratch/qwen3.6-27b-demand-placement/PRD.md Normal file
View File

@@ -0,0 +1,61 @@
# Qwen3.6-27B recommended deployment
## Goal
Offer the pinned `Qwen/Qwen3.6-27B` BF16 artifact as a recommended,
text-only chat model. A valid chat request proves model demand. When the
shared tracker pool has sufficient eligible capacity, the tracker deploys the
model rather than waiting for an operator to request a load.
## Catalog and artifacts
- Canonical artifact: `Qwen/Qwen3.6-27B` at
`6a9e13bd6fc8f0983b9b99948120bc37f49c13e9`.
- Tracker/peer artifact sources remain preferred over Hugging Face; this is a
system rule, not model-specific behavior.
- BF16 is canonical. Nodes may load BF16, INT8, or NF4 from the canonical
shard according to their declared capability.
- The model is text-only for this feature. Image inputs are rejected rather
than implicitly advertised as supported.
## Quantization contract
- Chat accepts an optional `quantization` field: `bfloat16`, `int8`, or
`nf4`. Omission means `bfloat16`.
- A request is a minimum-quality constraint: BF16 uses BF16 only; INT8 can
use INT8 or BF16; NF4 can use NF4, INT8, or BF16.
- A selectable variant requires complete end-to-end coverage using only
qualifying shards. Mixed qualifying precision is valid.
- The UI defaults to the highest complete coverage, lists unavailable variants
as non-selectable, and gives them a small coverage-vote control.
## Demand and placement
- The first valid model request queues initial tracker-managed placement when
sufficient pooled capability exists. Until complete coverage exists, return
retryable `503 model_loading` with coverage metadata.
- Demand and node supply are hive-wide; a leader makes each assignment and
broadcasts it to redundant trackers.
- A request for an unavailable quantization is retained as demand. Votes are
weaker when another Qwen quantization is already usable.
- If deployment replaces existing complete coverage, rolling demand for the
requested variant must exceed the displaced variant by more than 1.5x.
The multiplier and rolling window are tracker configuration.
- Managed replacements require at least three complete copies beforehand and
must leave two. Managed placement has a configurable cooldown.
## Node ownership
- A startup-assigned `(model, shard range, quantization)` is pinned and never
changed by the tracker.
- Spare capacity on a pinned node, and all capacity on a model-less node, is
available for tracker-managed assignments.
- Tracker-added assignments are explicitly marked managed and may be moved or
removed by the tracker under the safety policy. Runtime UI controls are a
later feature.
## Pricing
Use exact-model online provider pricing. Preserve the last verified price if
the provider lookup fails; use a model-specific development fallback only when
there is no verified price.

27
.scratch/routing-compatibility-regression/PRD.md Normal file
View File

@@ -0,0 +1,27 @@
# PRD: Legacy routing compatibility regression
## Overview
The tracker’s new request-precision gate rejects legacy node registrations that omit `quantization`. This conflicts with the explicit `capability_policy=compat` rollout: an older node without a capability report remains routable, but an older node without a quantization field is silently excluded from proxy, pinned-route, benchmark, billing, and latency paths.
## Goal
Restore backward-compatible routing for legacy registrations while preserving fail-closed behavior for explicitly declared invalid/unsupported quantization values.
## Quality gates
- Run targeted routing, billing, benchmark, pricing, validation, and latency tests.
- Run `python -m pytest -q` before completion and record any unrelated failures exactly.
- Preserve the admission-policy invariant: invalid capability reports remain non-routable; absent reports route only under `compat`.
## User story
### RCR-001: Legacy registration precision fallback
As an operator upgrading a mixed fleet, I need nodes that predate the `quantization` registration field to serve default-precision requests under `compat`, so the tracker rollout does not make otherwise healthy legacy nodes dark.
## Non-goals
- Do not weaken `enforce` capability policy.
- Do not treat explicitly malformed or unsupported quantization values as valid.
- Do not change requested-precision semantics for nodes that declare a supported precision.

38
.scratch/routing-compatibility-regression/issues/01-legacy-registration-precision-fallback.md Normal file
View File

@@ -0,0 +1,38 @@
# RCR-001 — Legacy registration precision fallback
Status: ready-for-agent
Priority: Critical
## Problem
The tracker now filters proxy and pinned-route candidates through `_quantization_satisfies(node.quantization, requested_quantization)`. A registration produced before the protocol added singular `quantization` is normalized to `quantizations=["bfloat16"]`, but leaves `node.quantization is None`; routing then fails every request, including the default `bfloat16` request.
The defect is therefore at protocol normalization: registration correctly records an available plural capability but never chooses a compatible active precision for the legacy entry.
This is inconsistent with the explicit capability admission rollout policy: an absent capability report is routable under `compat`. Logs show `capability=absent routable=True`, while `/v1/chat/completions` returns 503 or a pinned route returns 409 because the separate precision gate excludes the same legacy node.
## Evidence
Representative tight repros fail on current `master`:
- `tests/test_billing_ledger.py::test_proxy_chat_bills_credited_client_and_credits_node` → HTTP 503
- `tests/test_tracker_routing.py::test_tracker_proxy_accepts_hf_model_alias_from_quickstart` → HTTP 503
- `tests/test_manual_route_benchmark.py::test_pinned_route_uses_named_node` → HTTP 409
- `tests/test_model_speed_latency.py::test_tracker_records_increasing_hop_latency_for_model_and_hardware` → HTTP 409 for every parameter
Each fixture registers a complete, healthy legacy node without `quantization`; tracker registration logs show it as routable under `compat`.
## Acceptance criteria
- [ ] Legacy nodes omitting the `quantization` field are treated as the historical default precision for compatibility purposes.
- [ ] Explicit `null`, invalid, or unsupported declared precision never receives that fallback and remains excluded from routing, assignment, and coverage calculations, including after managed shard assignment/rebalancing.
- [ ] Raft replication preserves singular `quantization` and plural `quantizations`; follower routing has identical precision eligibility to the leader.
- [ ] `capability_policy=enforce` still excludes absent capability reports regardless of precision fallback.
- [ ] Automatic proxy routing, pinned routes, benchmark routes, and route-stat sampling use the same compatibility rule.
- [ ] Add behavior-level tracker HTTP tests for legacy omission and explicit invalid declaration.
- [ ] Targeted billing, routing, benchmark, model-speed, pricing, forfeiture, and TOPLOC dispatch tests pass.
- [ ] Run the full suite and document any failures not caused by this issue.
## Implementation notes
Keep the compatibility decision at the protocol-normalization boundary, not scattered into individual route paths. A normalized legacy default can preserve existing `_quantization_satisfies` semantics while differentiating an omitted field from an explicitly invalid value.

29
.scratch/routing-compatibility-regression/prd.json Normal file
View File

@@ -0,0 +1,29 @@
{
"name": "Legacy routing compatibility regression",
"description": "Restore compat-policy routing for legacy node registrations that omit quantization without weakening validation for invalid declared values.",
"branchName": "ralph/routing-compatibility-regression",
"userStories": [
{
"id": "RCR-001",
"title": "Legacy registration precision fallback",
"description": "As an operator upgrading a mixed fleet, I need nodes that predate the quantization registration field to serve default-precision requests under compat, so otherwise healthy legacy nodes do not become dark.",
"acceptanceCriteria": [
"A legacy registration without quantization is eligible for a bfloat16 request when capability policy is compat.",
"The same registration is excluded under capability policy enforce because its capability report is absent.",
"A node that explicitly declares null, an invalid, or an unsupported quantization remains excluded from routing, assignment, and coverage calculations, including after managed shard assignment/rebalancing.",
"Raft-applied registrations preserve both singular quantization and plural quantizations so follower routing matches leader routing.",
"Automatic proxy routing, pinned routes, manual benchmarks, billing proxy paths, and route-latency sampling retain their existing behavior for legacy registrations.",
"Tests cover the legacy/no-field case and explicit-invalid-field case at the tracker HTTP seam.",
"Targeted routing, billing, benchmark, pricing, validation, and latency tests pass.",
"python -m pytest -q is run and unrelated failures are recorded exactly."
],
"priority": 1,
"passes": true,
"dependsOn": [],
"completionNotes": "Completed by agent"
}
],
"metadata": {
"updatedAt": "2026-07-13T07:25:08.460Z"
}
}

21
.vscode/launch.json vendored
View File

@@ -7,7 +7,26 @@
"request": "launch",
"python": "${workspaceFolder}/.venv-rocm/bin/python",
"module": "meshnet_tracker.cli",
"args": ["start", "--host", "0.0.0.0", "--port", "8080", "--stats-db", "${workspaceFolder}/tracker-stats.sqlite", "--enable-test-runner"],
"args": ["start", "--host", "0.0.0.0", "--port", "8080", "--stats-db", "${workspaceFolder}/tracker-stats.sqlite"],
"console": "integratedTerminal",
"justMyCode": false
},
{
"name": "Tracker: local + dashboard test runner (8080)",
"type": "debugpy",
"request": "launch",
"python": "${workspaceFolder}/.venv-rocm/bin/python",
"module": "meshnet_tracker.cli",
"args": [
"start",
"--host",
"0.0.0.0",
"--port",
"8080",
"--stats-db",
"${workspaceFolder}/tracker-stats.sqlite",
"--enable-test-runner"
],
"console": "integratedTerminal",
"justMyCode": false
},

180
QUICKSTART.md
View File

@@ -7,12 +7,70 @@ Tested on: AMD Ryzen AI Max (Strix Halo APU), 124 GB RAM, Linux CPU inference.
ROCm GPU setup is covered below, but must be verified on the host because ROCm
support depends on the exact AMD GPU/APU, kernel, driver, and ROCm runtime.
**Active development models** (what we run day-to-day):
**Support here is evidence, not a promise.** No combination of GPU, torch build,
optional accelerator package and model is "supported" until `meshnet-node doctor`
has pushed a real forward through the exact shard you intend to serve, on that
machine. Detecting a GPU proves nothing; installing an optional package proves
nothing. See [Validate before you serve](#validate-before-you-serve-meshnet-node-doctor).
**Active development models** — what we run day-to-day, so their setup notes are the
most exercised. This is *not* a support matrix: any model the node can load is fair
game, and any model on this list can still fail `doctor` on your hardware.
| Role | `--model` / alias | HF repo | Notes |
|------|-------------------|---------|-------|
| Smoke tests, small splits | `Qwen/Qwen2.5-0.5B-Instruct` | same | 24 layers, ~1 GB BF16, no gating — default for new setups |
| Alpha / production target | `qwen3.6-35b-a3b` | `unsloth/Qwen3.6-35B-A3B` | 40 layers, ~72 GB BF16, hybrid linear-attention MoE; aliases include `Qwen3.6-35B-A3B`, `Qwen/Qwen3.6-35B-A3B` |
| Recommended dense model | `qwen3.6-27b` | `Qwen/Qwen3.6-27B` | 64 layers, ~56 GB BF16, text-only; canonical revision is pinned by the tracker |
---
## Models
The tracker advertises recommended models through `GET /v1/models`; the chat
dashboard shows the same catalog. `Qwen/Qwen3.6-27B` is recommended even before
it has complete coverage, so users and operators can see that the network intends
to serve it.
```bash
curl -s http://localhost:8080/v1/models | python -m json.tool
```
Each model reports its shard coverage and its selectable quantizations. A
quantization is selectable only when the tracker can build a complete route from
shards at that precision or higher:
- `bfloat16` requires BF16 for every shard.
- `int8` may combine INT8 and BF16 shards.
- `nf4` may combine NF4, INT8, and BF16 shards.
The chat UI chooses the highest complete precision by default. Uncovered variants
remain visible as coverage requests: use the small **Vote for coverage** control
to register demand without sending an unusable chat request.
Clients can request a minimum precision explicitly. Omit `quantization` to
request BF16:
```bash
curl http://localhost:8080/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "qwen3.6-27b",
"quantization": "int8",
"messages": [{"role": "user", "content": "Explain sharded inference."}]
}'
```
The first valid request for a recommended model is demand proof. When the shared
tracker pool has enough spare eligible capacity, it queues a tracker-managed
initial deployment. Until a complete route exists, the request returns retryable
`503 model_loading` rather than silently lowering precision.
Nodes may serve BF16, INT8, or NF4 according to their declared capability. The
official BF16 snapshot remains the canonical Qwen source; tracker/peer sources
are preferred and Hugging Face is the fallback. A node's startup-selected
`(model, shard range, quantization)` is pinned, while spare capacity can be used
for tracker-managed work.
---
@@ -238,8 +296,8 @@ HF_HOME=/path/to/models .venv-rocm/bin/meshnet-node start \
--quantization bfloat16
```
For the Qwen3.6 alpha model on Linux ROCm, install the optional FLA ROCm fast
path in the same env:
A model whose recipe uses an optional accelerator needs that package in the *same*
env. For example, `qwen3.6-35b-a3b` on Linux ROCm can use FLA:
```bash
.venv-rocm/bin/pip install 'flash-linear-attention[rocm]'
@@ -249,6 +307,12 @@ HF_HOME=/path/to/models .venv-rocm/bin/meshnet-node start \
--quantization bfloat16
```
This is a per-model, per-platform example. A successful `pip install` is not evidence
that the kernel runs on your GPU — optional-kernel support varies by architecture, and
we make no universal claim here. Prove it with
`.venv-rocm/bin/meshnet-node doctor --model <model>` before starting the node; startup
will refuse to register anyway if the shard cannot execute.
### Linux ROCm: Triton JIT compiler prerequisite
Some model/runtime paths invoke Triton at the first real forward. Triton builds a local HIP
@@ -480,8 +544,11 @@ curl -sI https://ai.neuron.d-popov.com/rpc/test-peer
no HuggingFace gating. Best for first-time setup.
**Alpha model:** `qwen3.6-35b-a3b` — 40 layers, ~72 GB BF16 download, MoE with hybrid
linear attention. On Windows install `triton-windows` + `flash-linear-attention`; on Linux
GPU use `flash-linear-attention[cuda]`. Tracker accepts the alias or full repo id (`unsloth/Qwen3.6-35B-A3B`).
linear attention. Tracker accepts the alias or full repo id (`unsloth/Qwen3.6-35B-A3B`).
Some models have an *optional* accelerator path (for this one, Triton +
`flash-linear-attention`); those installs are per-model, per-platform examples, not a
requirement the node imposes and not a guarantee the path will work — see
[Optional accelerator packages](#optional-accelerator-packages).
Downloads cache under `~/.meshnet/models/` (or `$HF_HOME` / `$env:HF_HOME`).
@@ -489,6 +556,79 @@ Shard range is auto-detected from the curated catalog. For unknown repos the nod
fetches only `config.json`. Override with `--shard-start` / `--shard-end` for partial
shards or multi-node splits.
### Validate before you serve: `meshnet-node doctor`
`doctor` resolves exactly the model, shard, quantization and recipe that `start` would
load, loads it, and pushes one bounded **real forward** through it. It is offline — it
never contacts the tracker — and it is the only thing that turns a guess into evidence.
```bash
HF_HOME=/path/to/models .venv/bin/meshnet-node doctor --model <model> --quantization bfloat16
```
```
meshnet-node doctor
Model: <model>
Shard: layers 0–23; 24 of 24
Quantization: bfloat16
[PASS] recipe default (v1) on cuda — 412 ms
OK — the selected shard ran a real forward for 1 recipe.
Capability report: ~/.meshnet/capability.json
```
Exit code is 0 on pass, 1 on failure. It writes a **capability report** either way — a
failure is evidence too. Useful flags: `--recipe <id>` to validate one named recipe,
`--all-recipes` to try every recipe for the selection (CI and diagnosis), `--report PATH`
to choose where the report lands, `--json` for machine-readable output, `--debug` for the
full traceback behind a failure. The selection flags (`--model`, `--quantization`,
`--shard-start`/`--shard-end`, `--cpu`) work the same as on `start`.
**Three states — do not confuse them:**
| State | What it means | How it is established |
|-------|---------------|-----------------------|
| **Detected hardware** | A GPU, a torch build, and an optional package are *present*. | Inventory/`torch.cuda.is_available()`. Proves nothing about whether your shard runs. |
| **Validated recipe** | This machine executed a real forward for *this* model + shard + recipe + device. | A passing `doctor` run → a capability report. |
| **Routable Node** | The tracker will send paid work here. | Startup re-proves the loaded shard and registers the report; the tracker admits it. |
Each state is strictly stronger than the one above it. A machine can have a working GPU
(detected) and still fail to execute a shard (no validated recipe), and a node can hold a
valid report and still not route (the tracker moved it to a shard range it never proved).
**Startup is fail-closed.** `meshnet-node start` runs the same validation against the
backend it just loaded, *before* it opens a port or registers. If the shard cannot execute,
the node exits non-zero — it never binds a socket and never registers. There is no
production bypass; a node that cannot do the work never advertises that it can.
**The tracker admits, it does not re-run.** The node ships its report with registration.
Registration always succeeds, so a broken node is visible rather than invisible — but only
a report that *covers what the node advertises* makes it routable. The verdict comes back
in the registration response, is logged, and is exposed per node on `GET /v1/network/map`
under `capability` (`admitted`, `absent`, `invalid`, `failed`, `stale`, `model-mismatch`,
`shard-mismatch`, `recipe-mismatch`, `catalogue-incompatible`). A node that is registered
but dark is showing you one of those. Full semantics, and the transitional `compat` vs
`enforce` policy, are in [ADR-0023](docs/adr/0023-model-agnostic-node-capability-admission.md).
**When `doctor` fails** it prints a category and an actionable hint, not a traceback:
| Category | Do this |
|----------|---------|
| `no-model-selected` | Pass `--model`, or run `meshnet-node` once to save a config. |
| `missing-dependency` | Install the node's model extras (torch, transformers, safetensors, accelerate). |
| `model-unavailable` | Check the model id, `--download-dir`, and that the artifact is downloaded/reachable. |
| `insufficient-memory` | Serve fewer layers (`--shard-start`/`--shard-end`) or a smaller quantization (`-q int8`, `-q nf4`). |
| `invalid-shard` | The layer range does not exist in this model — check it against the layer count. |
| `unsupported-recipe` | This backend cannot apply that execution setting; pick another `--recipe`. |
| `load-failed` | The shard would not load; re-run with `--debug`. |
| `forward-failed` | It loaded but cannot execute. This machine cannot serve this shard; re-run with `--debug`. |
**Scope:** the node validates recipes it already ships. It does **not** download executable
recipes, install Python/OS packages, or update drivers for you — every install on this page
is something you run deliberately. Signed node updates are a deliberate follow-up, not part
of this release.
### Core command
Replace `<tracker-url>` and adjust the prefix for your shell (see table above).
@@ -520,31 +660,45 @@ HF_HOME=/path/to/models .venv/bin/meshnet-node start --tracker http://localhost:
.venv/bin/meshnet-node start --tracker https://ai.neuron.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct
```
**Alpha model (Qwen3.6, Windows GPU — enable fast path):**
#### Optional accelerator packages
Some models have an optional fast path that a *specific* model's recipe can use if the
package is importable. The examples below are exactly that — **worked examples for one
development model on one platform**, not a supported-configuration list. Installing the
package does not mean the fast path works on your GPU: a package can import cleanly and
still fail to execute a kernel on your architecture. `doctor` is what tells you, and the
node serves the recipe that actually validated. If none of this is installed, the model
still runs on the standard path.
**Example — `qwen3.6-35b-a3b` (Triton + FLA), Windows GPU:**
```powershell
$env:HF_HOME = "D:\DEV\models"
pip install triton-windows
pip install -U flash-linear-attention
meshnet-node doctor --model qwen3.6-35b-a3b --quantization bfloat16
meshnet-node start --tracker http://192.168.0.179:8080 --model qwen3.6-35b-a3b --quantization bfloat16
```
Do not add `causal-conv1d` or `flash-linear-attention[cuda]` on Windows (see Qwen3.5/3.6 notes).
**Alpha model (Qwen3.6, Linux NVIDIA GPU — with fast path):**
**Example — `qwen3.6-35b-a3b`, Linux NVIDIA GPU:**
```bash
HF_HOME=/path/to/models .venv/bin/meshnet-node start --tracker <tracker-url> --model qwen3.6-35b-a3b --quantization bfloat16
# Install once on that machine: pip install flash-linear-attention[cuda]
HF_HOME=/path/to/models .venv/bin/meshnet-node start --tracker <tracker-url> --model qwen3.6-35b-a3b --quantization bfloat16
```
**Alpha model (Qwen3.6, Linux AMD ROCm GPU — with fast path):**
**Example — `qwen3.6-35b-a3b`, Linux AMD ROCm GPU:**
```bash
HF_HOME=/path/to/models .venv-rocm/bin/meshnet-node start --tracker <tracker-url> --model qwen3.6-35b-a3b --quantization bfloat16
# Install once on that machine: .venv-rocm/bin/pip install 'flash-linear-attention[rocm]'
HF_HOME=/path/to/models .venv-rocm/bin/meshnet-node start --tracker <tracker-url> --model qwen3.6-35b-a3b --quantization bfloat16
```
Run `doctor` after any of these installs: it is the only way to learn whether the optional
path executes here, and it costs one bounded forward.
After the first node registers a model, later nodes can join with only the tracker
URL (shard auto-assigned):
@@ -736,3 +890,9 @@ failure the node logs a warning and falls back to direct HTTP before erroring.
```bash
.venv/bin/python -m pytest -q
```
Tests marked `integration` download models or need a GPU; the default lane is
`pytest -m "not integration"`. The opt-in real-model doctor check takes its model from the
environment and skips without it — see
[docs/dev/certified-hardware-lanes.md](docs/dev/certified-hardware-lanes.md) for the
release contract that certifies a hardware lane.

17
_DEV_NOTES.md
View File

@@ -22,14 +22,21 @@
--tracker http://192.168.0.179:8081 `
--model Qwen/Qwen2.5-0.5B-Instruct `
--advertise-host 192.168.0.20
# trackers:
https://meshnet.2.d-popov.com
https://ai.neuron.d-popov.com
# Models
qwen3.6-35b-a3b Qwen/Qwen2.5-0.5B-Instruct Qwen3.6-27B
qwen3.6-35b-a3b Qwen/Qwen2.5-0.5B-Instruct
# linux
HF_HOME=/run/media/popov/d/DEV/models .venv/bin/meshnet-node start --model-id Qwen/Qwen2.5-0.5B-Instruct --shard-start 0 --shard-end 21 --quantization bfloat16 --tracker http://localhost:8081
HF_HOME=/run/media/popov/d/DEV/models .venv-rocm/bin/meshnet-node start --tracker https://meshnet.2.d-popov.com --model qwen3.6-35b-a3b --shard-start 10
meshnet-node start --tracker http://192.168.0.179:8080 --model Qwen/Qwen2.5-0.5B-Instruct --shard-start 0 --shard-end 20
.venv-rocm/bin/meshnet-node start --tracker https://meshnet.2.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct --shard-start 10
HF_HOME=/run/media/popov/d/DEV/models .venv-rocm/bin/meshnet-node start --tracker https://meshnet.2.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct --shard-start 10
.venv-rocm/bin/meshnet-node start --tracker https://meshnet.2.d-popov.com --model Qwen3.6-27B
meshnet-node start --tracker https://meshnet.2.d-popov.com --model qwen3.6-35b-a3b --cpu
meshnet-node start --tracker https://meshnet.2.d-popov.com --model qwen3.6-35b-a3b --shard-start 0 --shard-end 21 --node-name gpu-head
@@ -64,3 +71,9 @@ meshnet-node start --tracker https://meshnet.2.d-popov.com --model Qwen/Qwen2.5-
1. no benchmark at node start
2. CUDA stopped working on windows PS
3. solana/crypto does not work on linux tracker. does it still work on windows?
# corrected malformed node command (original notes preserved above)
HF_HOME=/run/media/popov/d/DEV/models .venv-rocm/bin/meshnet-node start --tracker https://meshnet.2.d-popov.com --model Qwen3.6-27B
# explicit Hugging Face model (namespace required to bypass tracker preset lookup)
HF_HOME=/run/media/popov/d/DEV/models .venv-rocm/bin/meshnet-node start --tracker https://meshnet.2.d-popov.com --model Qwen/Qwen3.6-27B --download-dir /run/media/popov/d/DEV/models

25
docs/adr/0023-model-agnostic-node-capability-admission.md
View File

@@ -18,6 +18,31 @@ This is incompatible with a consumer-grade node experience. A Node must never ad
- P0 carries the version of a local recipe manifest. New executable recipes arrive only through signed Node releases in a future feature. P0 does not download executable recipes, dynamically install dependencies, install OS packages/drivers, or implement an updater.
- A future Tracker-provided Model Artifact Manifest may be signed data only; it cannot instruct a Node to execute arbitrary code.
## Tracker admission and the compatibility policy for older Nodes
A Node ships its capability report with `POST /v1/nodes/register` (`capability_report`), alongside an independent declaration of the recipe it serves with (`recipe_id`, `recipe_version`). The Tracker does not re-run the forward. It decides whether the presented proof *covers what the Node advertises*, records the verdict as a sanitized state, and routes accordingly.
Registration always succeeds — a Node with a bad proof is registered and visible, it is simply not routable. "Registered but dark" is a state an operator must be able to see and diagnose, so the verdict is returned in the registration response, logged, and exposed per node on `GET /v1/network/map` under `capability` (state, detail, proven model/shard/recipe/backend/device, timestamps). The detail is credential-redacted and clipped; a raw exception or token never reaches an operator view.
Verdicts: `admitted`, `absent`, `invalid`, `failed`, `stale`, `model-mismatch`, `shard-mismatch`, `recipe-mismatch`, `catalogue-incompatible`. Only `admitted` is proof. The proof does not travel with a reassignment: if the Tracker later moves a Node to a range it never validated, the Node is re-verdicted `shard-mismatch` and stops routing until it re-registers with a proof for the range it now advertises.
Freshness is checked when the proof is *presented*, not continuously — a long-lived Node's proof does not expire out from under it while it is heartbeating; liveness is already carried by heartbeat expiry.
**Compatibility policy** (`--capability-policy`, `$MESHNET_TRACKER_CAPABILITY_POLICY`):
- **`compat` (default, transitional)** — a Node that presents *no* report at all still routes, preserving pre-capability Node behaviour during the fleet rollout. Every other verdict is refused. Presenting a broken, failed, stale or mismatched proof is a stronger negative signal than presenting none, so it is never grandfathered.
- **`enforce`** — only `admitted` routes. Absent proof is not routable, and no paid route can rest on an unproven Node.
`compat` is a deprecating default: it exists to let a mixed fleet upgrade without an outage, and `enforce` becomes the default once the deployed Nodes emit reports. The policy is a single explicit switch, checked in one gate (`_admitted_nodes`) that every route path — proxy head selection, `/v1/route`, `/v1/routes`, and bandit route enumeration — passes through. The gate only ever *removes* candidates; coverage-first selection and throughput-weighted preference among the survivors are untouched, and nothing in a report can raise a Node's routing weight (performance stays measured, per ADR-0013/ADR-0021).
The Tracker also refuses a report whose recipe catalogue predates `MIN_CATALOGUE_VERSION`: recipe ids from an older catalogue may since have been redefined, so the proof cannot be matched to a name reliably.
## Hardware claims are evidence, not a support matrix
Operator docs must distinguish three states and never collapse them: **detected hardware** (a GPU, a torch build, or an optional package is present — proves nothing), **validated recipe** (this machine ran a real forward for this model/shard/recipe/device, and there is a capability report to show for it), and **routable Node** (the Tracker admitted that proof for what the Node advertises). Each is strictly stronger than the last.
Consequently no doc promises that a model, vendor, or optional kernel works universally. A concrete model appears only as a clearly-labelled example or as environment-supplied test configuration. Hardware support is claimed per *certified lane*, where a lane is certified by an opt-in `integration` doctor run whose model identity comes from CI configuration and whose retained evidence is the capability report — see `docs/dev/certified-hardware-lanes.md`. A lane certifies hardware, not models: a new Model Artifact is unproven there until doctor has run it.
## Consequences
- First startup has a bounded validation cost before registration, but failures occur before traffic rather than under a paid request.

105
docs/dev/certified-hardware-lanes.md Normal file
View File

@@ -0,0 +1,105 @@
# Certified hardware lanes
A **certified hardware lane** is one (hardware, torch build, OS) configuration on which we
have *evidence* that nodes can execute real work — a self-hosted release runner that runs
the opt-in integration doctor test and keeps the resulting capability report as the
artifact. This document is the contract that lane runners and the release check must meet.
Certification is per *lane*, and evidence is per *(lane, model, shard, recipe)*. Nothing
here promises that an arbitrary model runs on a certified lane; it promises that the lane
itself is real, and that the models we ran on it produced passing capability reports on a
named date. See [ADR-0023](../adr/0023-model-agnostic-node-capability-admission.md) for the
admission model this rests on.
## What certification is not
- **Not a model support matrix.** A lane certifies hardware, not models. A new model is
unproven on a certified lane until doctor has run it there.
- **Not an optional-kernel promise.** An optional accelerator package importing cleanly on
a lane says nothing about another lane, another GPU architecture, or another model's
recipe. Only a passing report for that exact combination is evidence.
- **Not a promise the node will install anything.** Lane runners are provisioned *ahead of
time*, by hand or by their own image build. The node under test never downloads an
executable recipe, installs a Python or OS package, or touches a driver. Signed node
updates are a deliberate follow-up feature and are out of scope here — nothing in this
lane contract may depend on dynamic executable-dependency installation.
## The lane check
Every lane runs the same environment-configured integration test. It is
`tests/test_node_doctor.py::test_doctor_smoke_runs_a_real_forward_on_a_real_model`, marked
`@pytest.mark.integration` and skipped unless `MESHNET_DOCTOR_MODEL` is set. It carries no
default model: **model identity comes from the CI configuration**, so no vendor or model
assumption can leak into the suite.
```bash
MESHNET_DOCTOR_MODEL="$LANE_MODEL" \
MESHNET_DOCTOR_QUANTIZATION=bfloat16 \
MESHNET_DOWNLOAD_DIR=/srv/models \
.venv/bin/pytest -m integration tests/test_node_doctor.py -v
```
| Variable | Required | Meaning |
|----------|----------|---------|
| `MESHNET_DOCTOR_MODEL` | yes — the test skips without it | Model artifact identity for this lane's run. No default. |
| `MESHNET_DOCTOR_SHARD_START` | no (default `0`) | First layer of the shard to prove. |
| `MESHNET_DOCTOR_SHARD_END` | no (default: whole model) | Last layer, **inclusive**. |
| `MESHNET_DOCTOR_QUANTIZATION` | no (default `auto`) | Quantization to prove. |
| `MESHNET_DOCTOR_CPU` | no | `1` forces CPU — use to certify a CPU lane on GPU hardware. |
| `MESHNET_DOWNLOAD_DIR` | no | Where the artifact is cached on the runner. |
The test asserts that doctor passed, that the report is `passed` with the model id it was
asked for, that a forward actually took time (`duration_ms > 0`), and that the report
round-trips through `CapabilityReport.from_json`. A lane where this fails is not certified,
regardless of what `rocminfo`, `nvidia-smi` or `torch.cuda.is_available()` say.
Lanes that must cover more than the default recipe run doctor directly with
`--all-recipes`, which validates every recipe for the selection and writes a report per
recipe:
```bash
.venv/bin/meshnet-node doctor --model "$LANE_MODEL" --all-recipes --report "$ARTIFACTS/capability.json"
```
## Expected evidence
A lane run is only certified if it produces, and the release check retains:
1. **The capability report(s)** — `capability.json` from the run, archived as a build
artifact. This is the evidence; a green checkmark without it is not.
2. **Backend identity from the report**: device, torch/backend version, and the recipe id
and version that passed. This is what makes "certified on ROCm gfx1151" a checkable
claim rather than a slogan.
3. **The model artifact identity and shard range** the report covers — recorded as run
configuration, since it came from the environment.
4. **Failures kept, not discarded.** Doctor writes a report for a failed recipe too, and a
failing lane must archive it. A red lane with a `forward-failed` report is a more useful
release signal than a lane that was quietly skipped.
A lane that *skips* (because `MESHNET_DOCTOR_MODEL` was unset) must be reported as skipped,
never as passed. A silent skip is how an uncertified lane gets mistaken for a certified one.
## Release check
The default CI lane runs the normal suite and never needs a GPU, a download, or torch:
```bash
.venv/bin/pytest -m "not integration"
```
The release check additionally requires every declared certified lane to have run the
integration doctor test green, against the model(s) configured for that lane, on the
release commit. Adding a lane means standing up a runner and adding its configuration; it
does not mean adding a model default to the test suite.
## Adding a lane
1. Provision the runner: OS, driver, and the torch build for that hardware (see the
platform sections in `QUICKSTART.md`). Install any optional accelerator packages the
lane is meant to certify.
2. Configure `MESHNET_DOCTOR_MODEL` (and shard/quantization if the lane certifies a
partial shard) in the runner's CI configuration.
3. Run the lane check. Archive the capability report.
4. Record the lane with the evidence it produced: hardware, torch build, model, shard,
recipe, device, and the date. That record — not the hardware's spec sheet — is the
support claim.

93
docs/dev/dashboard-test-runner.md Normal file
View File

@@ -0,0 +1,93 @@
# Dashboard test runner (operator workflow)
The tracker dashboard **Testing** tab can discover repository pytest targets and
run one collected test or approved suite at a time with live logs. The feature is
**disabled by default** and **admin-only**.
## Enable intentionally
Start the tracker with the test runner API enabled using either:
- VS Code launch configuration **`Tracker: local + dashboard test runner (8080)`**
(uses the project tracker runtime at `.venv-rocm/bin/python` and
`meshnet_tracker.cli`), or
- CLI flag **`--enable-test-runner`**, or
- Environment variable **`MESHNET_ENABLE_TEST_RUNNER=1`**
The default **`Tracker: local (8080)`** launch configuration does **not** enable
the test runner.
Verify the flag is available:
```bash
uv run python -m meshnet_tracker.cli --help | grep enable-test-runner
```
Log in to the dashboard as an admin account, open **Testing**, and use **Refresh
collection** before running targets.
## Child pytest interpreter
The runner spawns pytest as a subprocess without a shell. It uses
**`MESHNET_PYTHON`** when set (typically via `.env.<hostname>` loaded by
`meshnet_tracker.cli`); otherwise it falls back to the tracker process
interpreter. Point this at the venv that has dev extras and package dependencies
installed (see [test-env.md](test-env.md)).
## Default safe suites
These named suites are always available when the test runner is enabled and the
files exist in the checkout:
| Suite ID | Paths | Notes |
| --- | --- | --- |
| `suite:smoke` | `tests/test_smoke.py` | Fast sanity checks |
| `suite:dashboard` | `tests/test_dashboard.py` | Dashboard HTML/API regressions |
| `suite:routing` | `tests/test_tracker_routing.py`, `tests/test_dynamic_routing.py` | Tracker routing logic |
Collection also lists individual pytest node IDs (excluding real-inference
modules by default). You can run `suite:all` or `tag:<name>` after collection.
These suites use mocks/stubs or in-process fakes. They do **not** require live
GPU nodes, paid API credits, or a running mesh beyond the tracker itself.
## Real-inference suite (explicitly gated)
Modules matching `tests/test_real_*.py` are **never collected** and **never**
included in default suites unless you set:
```bash
export MESHNET_ENABLE_REAL_INFERENCE_TESTS=1
```
With that gate, an additional suite appears:
| Suite ID | Paths |
| --- | --- |
| `suite:real-inference` | `tests/test_real_distributed_inference.py`, `tests/test_real_model_backend.py` |
### Implications
- **`tests/test_real_distributed_inference.py`** — integration test against a
**live tracker and registered model shards**. Requires env vars such as
`MESHNET_REAL_INFERENCE_URL`, `MESHNET_REAL_INFERENCE_API_KEY`,
`MESHNET_REAL_INFERENCE_MODEL`, and `MESHNET_REAL_INFERENCE_ROUTE`. Uses real
chat completions and **consumes caller billing / API credit** on the target
tracker.
- **`tests/test_real_model_backend.py`** — loads real PyTorch model code paths;
needs **`torch`**, **`transformers`**, and related optional deps, and can
require **substantial GPU/CPU RAM** depending on which cases run.
Do not enable `MESHNET_ENABLE_REAL_INFERENCE_TESTS=1` on shared or production
trackers unless you intend to spend credits and tie up hardware.
## Safety summary
| Control | Purpose |
| --- | --- |
| Disabled by default | No test subprocess unless operator opts in |
| Admin-only API/UI | Non-admins cannot start runs |
| Fixed suite list | API cannot pass arbitrary shell commands |
| No `shell=True` | pytest argv is fixed server-side |
| One run at a time | Concurrent starts are rejected |
| Real-inference env gate | Live inference tests stay out of default collection |

6
docs/dev/test-env.md
View File

@@ -38,6 +38,12 @@ even without installing `packages/node`.
.venv/bin/python -m pytest
```
## Dashboard test runner
For the opt-in tracker dashboard **Testing** tab (suites, env gates, VS Code
launch config, and real-inference safeguards), see
[dashboard-test-runner.md](dashboard-test-runner.md).
## Optional-dependency tests
Some tests import heavyweight or optional third-party packages and guard

130
packages/gateway/meshnet_gateway/prefill_backpressure.py Normal file
View File

@@ -0,0 +1,130 @@
"""Bounded, ordered prefill transfer primitives.
Prefill chunks mutate the downstream shard's session cache, so they must reach a
route in order. This deliberately uses a serial acknowledgement window: it is
the safe default for both current peers and old peers which do not advertise a
windowing capability. The configured in-flight limit is still explicit so a
future ordered transport can widen the window without changing callers.
"""
from __future__ import annotations
import os
from dataclasses import dataclass
from threading import Event
from typing import Callable, Iterable, TypeVar
DEFAULT_PREFILL_CHUNK_TOKENS = 128
DEFAULT_PREFILL_MAX_IN_FLIGHT = 1
DEFAULT_PREFILL_MAX_CHUNK_BYTES = 8 * 1024 * 1024
T = TypeVar("T")
R = TypeVar("R")
@dataclass(frozen=True)
class PrefillTransferLimits:
"""Configuration for one ordered prefill seam."""
chunk_tokens: int = DEFAULT_PREFILL_CHUNK_TOKENS
max_in_flight: int = DEFAULT_PREFILL_MAX_IN_FLIGHT
max_chunk_bytes: int = DEFAULT_PREFILL_MAX_CHUNK_BYTES
@property
def effective_in_flight(self) -> int:
"""Current peers require ordered session-cache mutation, hence one ack."""
return 1
@property
def max_buffered_bytes(self) -> int:
"""Hard accounting bound, including any future wider ack window."""
return self.max_chunk_bytes * self.max_in_flight
@classmethod
def from_env(cls) -> "PrefillTransferLimits":
# MESHNET_CHUNK_TOKENS was the pre-DIP-007 name. Keep it as a fallback
# so existing deployments retain their chunk shape while upgrading.
return cls(
chunk_tokens=_positive_env(
"MESHNET_PREFILL_CHUNK_TOKENS",
_positive_env("MESHNET_CHUNK_TOKENS", DEFAULT_PREFILL_CHUNK_TOKENS),
),
max_in_flight=_positive_env(
"MESHNET_PREFILL_MAX_IN_FLIGHT", DEFAULT_PREFILL_MAX_IN_FLIGHT,
),
max_chunk_bytes=_positive_env(
"MESHNET_PREFILL_MAX_CHUNK_BYTES", DEFAULT_PREFILL_MAX_CHUNK_BYTES,
),
)
class BoundedPrefillSender:
"""Send lazily-produced chunks with bounded ownership and ordered acks."""
def __init__(self, limits: PrefillTransferLimits) -> None:
self.limits = limits
self.buffered_bytes = 0
self.peak_buffered_bytes = 0
self.in_flight = 0
self.peak_in_flight = 0
self.closed = False
def send(
self,
chunks: Iterable[T],
*,
body_size: Callable[[T], int],
forward: Callable[[T], R],
cancelled: Event | None = None,
) -> list[R]:
"""Forward chunks in source order, releasing each body after its ack.
``forward`` is synchronous by design: a slow consumer therefore blocks
production of the next chunk instead of accumulating an unbounded queue.
Every retained body is dropped on cancellation or route failure.
"""
results: list[R] = []
try:
for chunk in chunks:
if self.closed or (cancelled is not None and cancelled.is_set()):
break
size = body_size(chunk)
if size < 0:
raise ValueError("prefill chunk size cannot be negative")
if size > self.limits.max_chunk_bytes:
raise ValueError(
f"prefill chunk exceeds {self.limits.max_chunk_bytes} byte limit"
)
self.buffered_bytes += size
self.in_flight += 1
self.peak_buffered_bytes = max(self.peak_buffered_bytes, self.buffered_bytes)
self.peak_in_flight = max(self.peak_in_flight, self.in_flight)
try:
results.append(forward(chunk))
finally:
# Do not retain a body while waiting for the next chunk.
self.buffered_bytes -= size
self.in_flight -= 1
except BaseException:
self.close()
raise
return results
def close(self) -> None:
"""Release accounting after cancellation or route failure.
The sender deliberately owns no queued chunk references; callers must
discard their iterator on close rather than trying to drain it.
"""
self.buffered_bytes = 0
self.in_flight = 0
self.closed = True
def _positive_env(name: str, default: int) -> int:
try:
value = int(os.environ.get(name, default))
except (TypeError, ValueError):
return default
return value if value > 0 else default

71
packages/gateway/meshnet_gateway/server.py
View File

@@ -14,6 +14,8 @@ import urllib.request
import uuid
from typing import Any
from .prefill_backpressure import BoundedPrefillSender, PrefillTransferLimits
_STUB_HIDDEN_DIM = 64
_STUB_DTYPE = "bfloat16"
_WIRE_VERSION = "2"
@@ -653,24 +655,27 @@ def _last_message_content(messages: object) -> str:
def _run_binary_pipeline(route: list[str], prompt: str, timeout: float = 5.0) -> list[_BinaryActivation]:
session = str(uuid.uuid4())
chunk_token_count = _chunk_token_count()
limits = PrefillTransferLimits.from_env()
chunk_token_count = limits.chunk_tokens
total_tokens = max(1, _prompt_token_count(prompt))
chunk_total = max(1, (total_tokens + chunk_token_count - 1) // chunk_token_count)
responses: list[_BinaryActivation] = []
for chunk_index in range(chunk_total):
remaining_tokens = total_tokens - (chunk_index * chunk_token_count)
seq_len = min(chunk_token_count, remaining_tokens)
activation = _BinaryActivation(
body=_make_stub_binary_activation([1, seq_len, _STUB_HIDDEN_DIM], _STUB_DTYPE),
shape=[1, seq_len, _STUB_HIDDEN_DIM],
dtype=_STUB_DTYPE,
session=session,
chunk_index=chunk_index,
chunk_total=chunk_total,
encoding=_preferred_binary_encoding(),
headers={},
)
def chunks():
for chunk_index in range(chunk_total):
remaining_tokens = total_tokens - (chunk_index * chunk_token_count)
seq_len = min(chunk_token_count, remaining_tokens)
yield _BinaryActivation(
body=_make_stub_binary_activation([1, seq_len, _STUB_HIDDEN_DIM], _STUB_DTYPE),
shape=[1, seq_len, _STUB_HIDDEN_DIM],
dtype=_STUB_DTYPE,
session=session,
chunk_index=chunk_index,
chunk_total=chunk_total,
encoding=_preferred_binary_encoding(),
headers={},
)
def forward(activation: _BinaryActivation) -> _BinaryActivation:
for hop_index, node_url in enumerate(route):
activation = _post_binary_forward(
f"{node_url}/forward",
@@ -678,17 +683,15 @@ def _run_binary_pipeline(route: list[str], prompt: str, timeout: float = 5.0) ->
hop_index=hop_index,
timeout=timeout,
)
responses.append(activation)
return responses
return activation
# Each completed response is retained only for the gateway's existing test
# diagnostic surface. At every hop the sender owns one chunk at a time.
return BoundedPrefillSender(limits).send(chunks(), body_size=lambda item: len(item.body), forward=forward)
def _chunk_token_count() -> int:
raw_value = os.environ.get("MESHNET_CHUNK_TOKENS", "128")
try:
value = int(raw_value)
except ValueError:
return 128
return value if value > 0 else 128
return PrefillTransferLimits.from_env().chunk_tokens
def _prompt_token_count(prompt: str) -> int:
@@ -737,11 +740,23 @@ def _post_binary_forward(
encoding = response_headers.get("x-meshnet-encoding")
raw_body = _decompress_body(response_body, encoding)
shape = _parse_shape(response_headers["x-meshnet-shape"])
dtype = response_headers["x-meshnet-dtype"]
session = response_headers["x-meshnet-session"]
chunk_index = int(response_headers["x-meshnet-chunk-index"])
chunk_total = int(response_headers["x-meshnet-chunk-total"])
# Legacy single-chunk peers returned only the body before chunk metadata
# was added. Treat absent metadata as the caller's single chunk, while a
# peer which partially implements the new protocol still fails closed.
if not response_headers.get("x-meshnet-shape"):
if activation.chunk_total != 1:
raise ValueError("legacy peer cannot acknowledge multi-chunk prefill")
shape = activation.shape
dtype = activation.dtype
session = activation.session
chunk_index = activation.chunk_index
chunk_total = activation.chunk_total
else:
shape = _parse_shape(response_headers["x-meshnet-shape"])
dtype = response_headers["x-meshnet-dtype"]
session = response_headers["x-meshnet-session"]
chunk_index = int(response_headers["x-meshnet-chunk-index"])
chunk_total = int(response_headers["x-meshnet-chunk-total"])
if session != activation.session:
raise ValueError("binary activation response changed session")
if chunk_index != activation.chunk_index or chunk_total != activation.chunk_total:

142
packages/node/meshnet_node/activation_compression.py Normal file
View File

@@ -0,0 +1,142 @@
"""Policy-driven zstd compression for activation seam bodies.
Policies are intentionally local to a hop condition: a LAN prefill can favour
wire savings while a one-token decode keeps a larger raw fast path. Environment
overrides make a trace-tuned rollout possible without changing the wire format.
"""
from __future__ import annotations
from dataclasses import dataclass
import os
import time
@dataclass(frozen=True)
class CompressionPolicy:
"""The measurable conditions required before an activation is compressed."""
min_input_bytes: int
min_savings_bytes: int = 4096
min_savings_ratio: float = 0.05
level: int = 1
enabled: bool = True
@dataclass(frozen=True)
class CompressionResult:
body: bytes
encoding: str | None
input_bytes: int
output_bytes: int
elapsed_seconds: float
decision: str
@property
def compressed(self) -> bool:
return self.encoding == "zstd"
_DEFAULTS: dict[tuple[str, str], CompressionPolicy] = {
# Decode activations usually contain one position; keep that hot path raw.
("lan", "prefill"): CompressionPolicy(64 * 1024),
("lan", "decode"): CompressionPolicy(128 * 1024),
("relay", "prefill"): CompressionPolicy(32 * 1024),
("relay", "decode"): CompressionPolicy(128 * 1024),
# The deterministic benchmark can explicitly model either policy family.
("benchmark", "prefill"): CompressionPolicy(64 * 1024),
("benchmark", "decode"): CompressionPolicy(128 * 1024),
}
class CompressionPolicies:
"""Explicit policies for LAN, relay, and benchmark prefill/decode seams.
Set ``MESHNET_COMPRESSION_<ROUTE>_<PHASE>_MIN_INPUT_BYTES``,
``..._MIN_SAVINGS_BYTES``, ``..._MIN_SAVINGS_RATIO``, or ``..._ENABLED`` to
tune a condition from production traces. E.g.
``MESHNET_COMPRESSION_RELAY_PREFILL_MIN_INPUT_BYTES=32768``.
"""
def __init__(self, policies: dict[tuple[str, str], CompressionPolicy] | None = None) -> None:
self._policies = dict(_DEFAULTS if policies is None else policies)
def for_condition(self, route: str, phase: str) -> CompressionPolicy:
key = (route.lower(), phase.lower())
try:
policy = self._policies[key]
except KeyError as exc:
raise ValueError(f"unknown compression condition {route}/{phase}") from exc
prefix = f"MESHNET_COMPRESSION_{key[0].upper()}_{key[1].upper()}_"
return CompressionPolicy(
min_input_bytes=_env_int(prefix + "MIN_INPUT_BYTES", policy.min_input_bytes),
min_savings_bytes=_env_int(prefix + "MIN_SAVINGS_BYTES", policy.min_savings_bytes),
min_savings_ratio=_env_float(prefix + "MIN_SAVINGS_RATIO", policy.min_savings_ratio),
level=_env_int(prefix + "LEVEL", policy.level),
enabled=_env_bool(prefix + "ENABLED", policy.enabled),
)
def compress_activation(body: bytes, policy: CompressionPolicy) -> CompressionResult:
"""Compress only when zstd clears both configured savings thresholds."""
started = time.monotonic()
if not policy.enabled:
return _raw(body, started, "disabled")
if len(body) < policy.min_input_bytes:
return _raw(body, started, "below_min_input")
try:
import zstandard as zstd
candidate = zstd.ZstdCompressor(level=policy.level).compress(body)
except Exception:
# Compression is an optional transport optimisation, never a reason to
# reject an otherwise valid activation.
return _raw(body, started, "unavailable")
saved = len(body) - len(candidate)
ratio = saved / max(1, len(body))
if saved < policy.min_savings_bytes or ratio < policy.min_savings_ratio:
return _raw(body, started, "below_savings")
return CompressionResult(candidate, "zstd", len(body), len(candidate), time.monotonic() - started, "compressed")
def decompress_activation(body: bytes, encoding: str | None) -> CompressionResult:
"""Decode a modern zstd body or preserve a legacy raw body with metrics."""
started = time.monotonic()
if not encoding:
return CompressionResult(body, None, len(body), len(body), time.monotonic() - started, "legacy_raw")
if encoding != "zstd":
raise ValueError("unsupported X-Meshnet-Encoding")
try:
import zstandard as zstd
except ImportError as exc:
raise ValueError("zstd support is unavailable") from exc
try:
raw = zstd.ZstdDecompressor().decompress(body)
except zstd.ZstdError as exc:
raise ValueError("invalid zstd activation body") from exc
return CompressionResult(raw, "zstd", len(body), len(raw), time.monotonic() - started, "decompressed")
def _raw(body: bytes, started: float, decision: str) -> CompressionResult:
return CompressionResult(body, None, len(body), len(body), time.monotonic() - started, decision)
def _env_int(name: str, default: int) -> int:
try:
return max(0, int(os.getenv(name, str(default))))
except ValueError:
return default
def _env_float(name: str, default: float) -> float:
try:
return max(0.0, float(os.getenv(name, str(default))))
except ValueError:
return default
def _env_bool(name: str, default: bool) -> bool:
value = os.getenv(name)
if value is None:
return default
return value.strip().lower() not in {"0", "false", "no", "off"}

225
packages/node/meshnet_node/admission.py Normal file
View File

@@ -0,0 +1,225 @@
"""Fail-closed admission: no routable registration without a fresh matching proof.
This module does not *produce* proof — `doctor` does that, by pushing a bounded
real forward through the selected shard (NCA-002). This module *decides whether a
proof covers what is about to be advertised*, and startup calls it immediately
before it registers with the tracker.
A capability report proves one combination: model artifact, shard range, recipe,
backend and device. Reusing it for anything else is the exact hole this closes —
a report that failed, aged out, or describes a different model, shard, recipe or
device is rejected here, and the node exits without ever registering an endpoint.
Nothing in here branches on a model, vendor or kernel name: identity fields are
opaque labels that are compared, never interpreted.
"""
from __future__ import annotations
import time
from dataclasses import dataclass
from typing import Any, Callable
from .capability import CapabilityReport
from .doctor import DoctorSelection
from .recipe_manifest import Recipe, RecipeManifest
# How long a passing report stays usable. Startup normally validates in-process
# (age ≈ 0); this bounds how far a report written by an earlier `doctor` run can
# be carried forward, after which the hardware, drivers or weights may have moved.
DEFAULT_MAX_REPORT_AGE_SECONDS = 900.0
# A report timestamped this far in the future is not fresh, it is wrong.
_MAX_CLOCK_SKEW_SECONDS = 60.0
REASON_NO_REPORT = "no-report"
REASON_NOT_PASSED = "not-passed"
REASON_STALE = "stale"
REASON_MODEL_MISMATCH = "model-mismatch"
REASON_SHARD_MISMATCH = "shard-mismatch"
REASON_RECIPE_MISMATCH = "recipe-mismatch"
REASON_BACKEND_MISMATCH = "backend-mismatch"
class CapabilityAdmissionError(RuntimeError):
"""This node may not advertise the selection: the proof does not cover it."""
def __init__(self, reason: str, message: str) -> None:
super().__init__(message)
self.reason = reason
@dataclass(frozen=True)
class CapabilityContext:
"""What is about to be advertised, and the loaded backend that would serve it."""
backend: Any
selection: DoctorSelection
recipe: Recipe
manifest: RecipeManifest
device: str
# A validator turns the context into the report the gate then judges. Production
# uses `probe_capability`; tests pass an explicit test-safe one (see
# `meshnet_node.testing`) rather than switching this module into a lenient mode.
CapabilityValidator = Callable[[CapabilityContext], CapabilityReport]
@dataclass(frozen=True)
class AdmissionRequirement:
"""The one capability a report must prove for this node to register."""
model_id: str
shard_start: int
shard_end: int
recipe_id: str
recipe_version: str
backend_id: str
device: str
max_age_seconds: float = DEFAULT_MAX_REPORT_AGE_SECONDS
@classmethod
def for_context(
cls,
context: CapabilityContext,
*,
max_age_seconds: float = DEFAULT_MAX_REPORT_AGE_SECONDS,
) -> AdmissionRequirement:
return cls(
model_id=context.selection.model_id,
shard_start=context.selection.shard_start,
shard_end=context.selection.shard_end,
recipe_id=context.recipe.id,
recipe_version=context.recipe.version,
backend_id=context.recipe.backend_id,
device=context.device,
max_age_seconds=max_age_seconds,
)
@property
def shard_label(self) -> str:
return f"layers {self.shard_start}–{self.shard_end}"
def admit(
requirement: AdmissionRequirement,
report: CapabilityReport | None,
*,
now: float | None = None,
) -> CapabilityReport:
"""Return `report` if it admits `requirement`; otherwise refuse to register.
Checks run selection-first, so the operator is told the report is about the
wrong thing before being told it is old.
"""
if report is None:
raise CapabilityAdmissionError(
REASON_NO_REPORT,
f"no capability report for {requirement.model_id} "
f"{requirement.shard_label}: this node has not proven it can serve it",
)
if report.model.model_id != requirement.model_id:
raise _mismatch(
REASON_MODEL_MISMATCH,
requirement,
"model",
report.model.model_id,
requirement.model_id,
)
if (report.shard.start, report.shard.end) != (
requirement.shard_start,
requirement.shard_end,
):
raise _mismatch(
REASON_SHARD_MISMATCH,
requirement,
"shard",
f"layers {report.shard.start}–{report.shard.end}",
requirement.shard_label,
)
if (report.recipe.recipe_id, report.recipe.recipe_version) != (
requirement.recipe_id,
requirement.recipe_version,
):
raise _mismatch(
REASON_RECIPE_MISMATCH,
requirement,
"recipe",
f"{report.recipe.recipe_id} (v{report.recipe.recipe_version})",
f"{requirement.recipe_id} (v{requirement.recipe_version})",
)
if (report.backend.backend_id, report.backend.device) != (
requirement.backend_id,
requirement.device,
):
raise _mismatch(
REASON_BACKEND_MISMATCH,
requirement,
"backend",
f"{report.backend.backend_id} on {report.backend.device}",
f"{requirement.backend_id} on {requirement.device}",
)
if not report.passed:
raise CapabilityAdmissionError(
REASON_NOT_PASSED,
f"capability validation {report.status} for {requirement.model_id} "
f"{requirement.shard_label} with recipe {requirement.recipe_id}"
+ _diagnostics_suffix(report),
)
now = time.time() if now is None else now
age = now - report.validated_at
if age > requirement.max_age_seconds:
raise CapabilityAdmissionError(
REASON_STALE,
f"capability report for {requirement.model_id} {requirement.shard_label} "
f"is {age / 60:.0f} min old (limit "
f"{requirement.max_age_seconds / 60:.0f} min); re-run `meshnet-node doctor`",
)
if age < -_MAX_CLOCK_SKEW_SECONDS:
raise CapabilityAdmissionError(
REASON_STALE,
f"capability report for {requirement.model_id} {requirement.shard_label} "
f"is timestamped {-age:.0f}s in the future; check this host's clock",
)
return report
def _mismatch(
reason: str,
requirement: AdmissionRequirement,
field_name: str,
reported: str,
required: str,
) -> CapabilityAdmissionError:
return CapabilityAdmissionError(
reason,
f"capability report proves a different {field_name}: it validated "
f"{reported}, but this node would serve {required}. A report is only "
"proof for the exact combination it ran.",
)
def _diagnostics_suffix(report: CapabilityReport) -> str:
if not report.diagnostics:
return ""
return " — " + " ".join(report.diagnostics)
def probe_capability(context: CapabilityContext) -> CapabilityReport:
"""Production validator: one bounded real forward through the loaded shard."""
from .doctor import validate_loaded_backend
return validate_loaded_backend(
context.backend,
context.selection,
context.recipe,
context.manifest,
).report

121
packages/node/meshnet_node/cli.py
View File

@@ -106,6 +106,11 @@ def _resolve_model_flags(
explicit = model_id or model
if not explicit:
return None, None
from .model_catalog import resolve_model_alias
preset = resolve_model_alias(explicit)
if preset is not None:
return preset.name, preset.hf_repo
if "/" in explicit:
return explicit.split("/")[-1], explicit
return explicit, None
@@ -232,6 +237,85 @@ def _cmd_config(args) -> int:
return 0
def _doctor_overrides(args) -> dict:
"""CLI flags that change *what* doctor validates, applied on top of config."""
overrides: dict = {}
model_name, hf_repo = _resolve_model_flags(
getattr(args, "model", None), getattr(args, "model_id", None)
)
if model_name is not None:
overrides["model_name"] = model_name
overrides["model_hf_repo"] = hf_repo or ""
for flag, key in (
("quantization", "quantization"),
("download_dir", "download_dir"),
("shard_start", "shard_start"),
("shard_end", "shard_end"),
):
value = getattr(args, flag, None)
if value is not None:
overrides[key] = value
if getattr(args, "cpu", False):
overrides["force_cpu"] = True
return overrides
def _cmd_doctor(args) -> int:
"""Validate the selected model/shard with a bounded real forward."""
import json
import traceback
from .config import DEFAULTS, load_config, merge_cli_overrides
from .doctor import (
DoctorError,
default_report_path,
render_result,
resolve_selection,
run_doctor,
write_reports,
)
debug = bool(getattr(args, "debug", False))
cfg = load_config() or dict(DEFAULTS)
overrides = _doctor_overrides(args)
if overrides:
cfg = merge_cli_overrides(cfg, **overrides)
try:
selection = resolve_selection(cfg)
result = run_doctor(
selection,
recipe_id=args.recipe,
all_recipes=args.all_recipes,
)
except DoctorError as exc:
# Bad input (no model, unknown recipe): there is nothing to report on.
if debug:
traceback.print_exc()
print(f"ERROR: {exc}", file=sys.stderr, flush=True)
if exc.hint:
print(f" {exc.hint}", file=sys.stderr, flush=True)
return 1
written = write_reports(
result.reports,
Path(args.report) if args.report else default_report_path(),
)
if args.json:
print(json.dumps([r.to_dict() for r in result.reports], indent=2, sort_keys=True))
else:
print(render_result(result, report_path=written))
if debug:
for item in result.results:
if item.error is not None:
traceback.print_exception(
type(item.error), item.error, item.error.__traceback__
)
return result.exit_code
def _cmd_start(args) -> int:
"""Legacy `start` subcommand — preserves backward compatibility with existing tests."""
from .config import DEFAULTS
@@ -317,6 +401,7 @@ def main() -> None:
" models List supported models\n"
" models --browse Browse HuggingFace Hub\n"
" config Show current config\n"
" doctor Check this node can really run its selected shard\n"
),
)
@@ -362,6 +447,40 @@ def main() -> None:
# config subcommand
subparsers.add_parser("config", help="Show current saved config")
# doctor subcommand — validate the selected shard with a real forward
doctor_cmd = subparsers.add_parser(
"doctor",
help="Check this node can really run its selected model shard",
)
# These mirror the top-level selection flags. argparse.SUPPRESS keeps an
# unpassed subcommand flag from overwriting the top-level one, so both
# `meshnet-node --model X doctor` and `meshnet-node doctor --model X` work.
doctor_cmd.add_argument("--model", metavar="MODEL", default=argparse.SUPPRESS,
help="Model name or HuggingFace repo ID to validate")
doctor_cmd.add_argument("--model-id", metavar="MODEL", default=argparse.SUPPRESS,
help="Alias for --model")
doctor_cmd.add_argument("--quantization", "-q", default=argparse.SUPPRESS,
choices=["bf16", "int8", "nf4", "bfloat16", "auto"],
help="Quantization level to validate")
doctor_cmd.add_argument("--download-dir", metavar="PATH", default=argparse.SUPPRESS,
help="Model download directory")
doctor_cmd.add_argument("--shard-start", type=int, metavar="N", default=argparse.SUPPRESS,
help="Pin shard start layer")
doctor_cmd.add_argument("--shard-end", type=int, metavar="N", default=argparse.SUPPRESS,
help="Pin shard end layer")
doctor_cmd.add_argument("--cpu", action="store_true", default=argparse.SUPPRESS,
help="Validate CPU execution even when a GPU is available")
doctor_cmd.add_argument("--debug", action="store_true", default=argparse.SUPPRESS,
help="Print the full traceback behind a failure")
doctor_cmd.add_argument("--recipe", metavar="ID", default=None,
help="Recipe to validate (default: baseline)")
doctor_cmd.add_argument("--all-recipes", action="store_true",
help="Validate every recipe in the catalogue, not just the selected one")
doctor_cmd.add_argument("--report", metavar="PATH", default=None,
help="Where to write the capability report JSON")
doctor_cmd.add_argument("--json", action="store_true",
help="Print the capability report JSON instead of a summary")
# start subcommand (legacy / backward-compat)
start_cmd = subparsers.add_parser("start", help="Start node (legacy flags)")
start_cmd.add_argument("--tracker")
@@ -401,6 +520,8 @@ def main() -> None:
sys.exit(_cmd_models(args))
elif args.command == "config":
sys.exit(_cmd_config(args))
elif args.command == "doctor":
sys.exit(_cmd_doctor(args))
elif args.command == "start":
sys.exit(_cmd_start(args))
else:

633
packages/node/meshnet_node/doctor.py Normal file
View File

@@ -0,0 +1,633 @@
"""`meshnet-node doctor` — prove the selected shard actually runs.
The doctor answers one question: *would the model/shard/recipe this node is
configured to serve really execute here?* It answers it the only way that is
not a guess — by loading the selection through the production backend path and
pushing a bounded, real forward through the selected layers. Generic hardware
probing (is there a GPU, can Torch allocate a tensor) proves nothing about a
shard and is deliberately not what this reports on.
Two shapes of probe, chosen by where the shard sits, never by which model it is:
* head shard — tokenize a short prompt, embed it, run this shard's layers.
* mid/tail shard — synthesize a small hidden-state tensor in the same wire
format peers send, and push it through `forward_bytes`. A tail shard decodes
it, which also exercises the final norm and `lm_head`.
Everything here is model-agnostic: `model_id` is opaque, and no vendor or kernel
name is a branch. Failures are reported as a category plus an actionable hint
(never a raw traceback, unless the caller asks for one) and produce a *failed*
capability report — a failure is evidence too, and NCA-003 refuses to register
without a fresh passing one.
"""
from __future__ import annotations
import base64
import struct
import time
from dataclasses import dataclass
from pathlib import Path
from typing import Any, Callable, Mapping, Sequence
from .capability import (
STATUS_FAILED,
STATUS_PASSED,
CapabilityReport,
build_capability_report,
)
from .recipe_manifest import (
DEFAULT_RECIPE_ID,
Recipe,
RecipeManifest,
RecipeManifestError,
load_recipe_manifest,
)
# The probe is deliberately tiny: enough tokens to drive every layer in the
# shard once, small enough that `doctor` costs seconds beyond the model load.
PROBE_TOKENS = 4
PROBE_PROMPT = "meshnet capability probe"
# Failure categories. These are what an operator acts on, so they name the thing
# to fix, not the exception that surfaced it.
CATEGORY_NO_MODEL = "no-model-selected"
CATEGORY_MISSING_DEPENDENCY = "missing-dependency"
CATEGORY_MODEL_UNAVAILABLE = "model-unavailable"
CATEGORY_INSUFFICIENT_MEMORY = "insufficient-memory"
CATEGORY_INVALID_SHARD = "invalid-shard"
CATEGORY_UNSUPPORTED_RECIPE = "unsupported-recipe"
CATEGORY_LOAD_FAILED = "load-failed"
CATEGORY_FORWARD_FAILED = "forward-failed"
CATEGORY_HINTS: Mapping[str, str] = {
CATEGORY_NO_MODEL: (
"No model is selected. Pass --model <repo-or-name>, or run `meshnet-node` "
"once to save a config."
),
CATEGORY_MISSING_DEPENDENCY: (
"The model runtime is not installed. Install the node's model extras "
"(torch, transformers, safetensors, accelerate, bitsandbytes)."
),
CATEGORY_MODEL_UNAVAILABLE: (
"The model files could not be read. Check the model id, --download-dir, "
"and that the artifact is downloaded or reachable."
),
CATEGORY_INSUFFICIENT_MEMORY: (
"This shard does not fit in memory. Serve fewer layers (--shard-start / "
"--shard-end) or use a smaller quantization (-q int8, -q nf4)."
),
CATEGORY_INVALID_SHARD: (
"The requested layer range does not exist in this model. Check "
"--shard-start / --shard-end against the model's layer count."
),
CATEGORY_UNSUPPORTED_RECIPE: (
"The recipe asks for an execution setting this backend cannot apply. "
"Select a different recipe with --recipe."
),
CATEGORY_LOAD_FAILED: (
"The shard could not be loaded. Re-run with --debug for the full traceback."
),
CATEGORY_FORWARD_FAILED: (
"The shard loaded but could not execute a forward pass. This node cannot "
"serve this model/shard; re-run with --debug for the full traceback."
),
}
class DoctorError(RuntimeError):
"""A validation failure with an operator-facing category and hint."""
def __init__(self, category: str, message: str) -> None:
super().__init__(message)
self.category = category
@property
def hint(self) -> str:
return CATEGORY_HINTS.get(self.category, "")
@dataclass(frozen=True)
class DoctorSelection:
"""The one model/shard/config combination startup would load."""
model_id: str
shard_start: int
shard_end: int
quantization: str = "auto"
cache_dir: Path | None = None
force_cpu: bool = False
@property
def shard_label(self) -> str:
return f"layers {self.shard_start}–{self.shard_end}"
@dataclass(frozen=True)
class RecipeResult:
"""One recipe's validation outcome, with the report it produced."""
recipe: Recipe
report: CapabilityReport
category: str | None = None
error: BaseException | None = None
@property
def passed(self) -> bool:
return self.report.passed
@property
def hint(self) -> str:
return CATEGORY_HINTS.get(self.category or "", "")
@dataclass(frozen=True)
class DoctorResult:
"""The outcome of a doctor run over one or more recipes."""
selection: DoctorSelection
results: tuple[RecipeResult, ...] = ()
@property
def passed(self) -> bool:
return bool(self.results) and all(r.passed for r in self.results)
@property
def reports(self) -> tuple[CapabilityReport, ...]:
return tuple(r.report for r in self.results)
@property
def exit_code(self) -> int:
return 0 if self.passed else 1
# --- selection: the same resolution startup performs ------------------------
def resolve_selection(
cfg: Mapping[str, Any],
*,
detect_layers: Callable[[str, Path | None], int | None] | None = None,
) -> DoctorSelection:
"""Resolve config + flags into the selection startup would load.
This mirrors `startup.run_startup`: the same model id, the same
`bf16`→`bfloat16` quantization normalization, and the same shard default of
the whole model when no range is pinned. It deliberately does *not* ask the
tracker for a gap assignment — the doctor is an offline check of what this
node can run, and startup re-validates whatever range it is finally given.
"""
model_id = _selected_model_id(cfg)
if not model_id:
raise DoctorError(
CATEGORY_NO_MODEL, "no model is selected in config or flags"
)
cache_dir = Path(cfg["download_dir"]) if cfg.get("download_dir") else None
quantization = str(cfg.get("quantization") or "auto").replace("bf16", "bfloat16")
shard_start = cfg.get("shard_start")
shard_end = cfg.get("shard_end")
if shard_start is None or shard_end is None:
detect = detect_layers or _detect_layers
total = detect(model_id, cache_dir)
if total is None:
raise DoctorError(
CATEGORY_MODEL_UNAVAILABLE,
f"could not read the layer count from the {model_id} config; "
"pass --shard-start and --shard-end explicitly",
)
shard_start = 0 if shard_start is None else shard_start
shard_end = total - 1 if shard_end is None else shard_end
if shard_start < 0 or shard_end < shard_start:
raise DoctorError(
CATEGORY_INVALID_SHARD,
f"invalid shard range {shard_start}–{shard_end}: start must be "
"non-negative and not greater than end",
)
return DoctorSelection(
model_id=model_id,
shard_start=int(shard_start),
shard_end=int(shard_end),
quantization=quantization,
cache_dir=cache_dir,
force_cpu=bool(cfg.get("force_cpu", False)),
)
def _selected_model_id(cfg: Mapping[str, Any]) -> str | None:
"""The HF repo startup would load, resolving a catalog alias if needed."""
hf_repo = str(cfg.get("model_hf_repo") or "").strip()
if hf_repo:
return hf_repo
name = str(cfg.get("model_name") or "").strip()
if not name:
return None
from .model_catalog import resolve_model_alias
preset = resolve_model_alias(name)
if preset is not None and preset.hf_repo:
return preset.hf_repo
return name if "/" in name else None
def _detect_layers(model_id: str, cache_dir: Path | None) -> int | None:
from .startup import _detect_num_layers
return _detect_num_layers(model_id, cache_dir=cache_dir)
# --- the bounded real forward ----------------------------------------------
@dataclass(frozen=True)
class ProbeInput:
"""A synthetic hidden-state payload in the same wire format peers send."""
body: bytes
shape: list[int]
attention_mask_header: str | None
position_ids_header: str | None
def _int64_header(rows: Sequence[Sequence[int]]) -> str:
"""Encode an int64 tensor as `shape:base64`, matching the backend's format."""
flat = [int(v) for row in rows for v in row]
raw = struct.pack(f"<{len(flat)}q", *flat)
shape = f"{len(rows)},{len(rows[0])}" if rows else "0"
return f"{shape}:{base64.b64encode(raw).decode('ascii')}"
def build_probe_input(hidden_size: int, tokens: int = PROBE_TOKENS) -> ProbeInput:
"""Build a bounded mid-shard probe: `tokens` positions of bfloat16 zeros.
Zeros are a legitimate hidden state; what is being proven is that the
layers execute on this device, not that the output means anything. The
payload is built with plain bytes so callers need no Torch import.
"""
if hidden_size <= 0:
raise DoctorError(
CATEGORY_FORWARD_FAILED,
"the backend reports no hidden size, so no probe tensor can be built",
)
ones = [[1] * tokens]
positions = [list(range(tokens))]
return ProbeInput(
body=b"\x00" * (tokens * hidden_size * 2), # bfloat16 == 2 bytes
shape=[1, tokens, hidden_size],
attention_mask_header=_int64_header(ones),
position_ids_header=_int64_header(positions),
)
def probe_forward(backend: Any, *, tokens: int = PROBE_TOKENS) -> dict:
"""Run one bounded real forward through the shard `backend` holds.
Returns a small detail dict for the human summary. Raises `DoctorError`
(category `forward-failed`) if the shard cannot execute or returns nothing.
"""
is_head = bool(getattr(backend, "is_head", False))
is_tail = bool(getattr(backend, "is_tail", False))
try:
if is_head:
output = backend.encode_prompt(PROBE_PROMPT)
kind = "prompt"
if is_tail:
# A head+tail shard owns the lm_head too. Re-entering above the
# last layer runs no layer again — it only decodes — so the whole
# selected shard is covered without a second forward through it.
output = backend.forward_bytes(
output.body,
output.shape,
output.attention_mask_header,
output.position_ids_header,
start_layer=int(getattr(backend, "shard_end", 0)) + 1,
)
kind = "prompt+decode"
else:
probe = build_probe_input(int(getattr(backend, "hidden_size", 0) or 0))
output = backend.forward_bytes(
probe.body,
probe.shape,
probe.attention_mask_header,
probe.position_ids_header,
start_layer=getattr(backend, "shard_start", None),
)
kind = "hidden-states"
except DoctorError:
raise
except Exception as exc:
raise DoctorError(CATEGORY_FORWARD_FAILED, _describe(exc)) from exc
return {"probe": kind, "tokens": tokens, **_describe_output(output)}
def _describe_output(output: Any) -> dict:
"""Validate the forward produced real output, and summarize it."""
if output is None:
raise DoctorError(
CATEGORY_FORWARD_FAILED, "the shard forward returned no output"
)
token_id = getattr(output, "token_id", None)
if token_id is not None: # tail shard: decoded a token
return {"output": "token", "token_id": int(token_id)}
body = getattr(output, "body", None)
shape = list(getattr(output, "shape", []) or [])
if not body or not shape:
raise DoctorError(
CATEGORY_FORWARD_FAILED,
"the shard forward returned an empty hidden-state payload",
)
return {"output": "hidden-states", "shape": shape}
# --- running the doctor -----------------------------------------------------
def default_load_backend(
selection: DoctorSelection,
recipe: Recipe,
) -> Any:
"""Load the shard through the exact path startup uses."""
from .torch_server import _load_backend
return _load_backend(
selection.model_id,
selection.shard_start,
selection.shard_end,
selection.quantization,
selection.cache_dir,
force_cpu=selection.force_cpu,
recipe_params=recipe.params,
)
def select_recipes(
manifest: RecipeManifest,
*,
recipe_id: str | None = None,
all_recipes: bool = False,
) -> tuple[Recipe, ...]:
"""The recipes to validate: the selected one, or every one on request.
`--all-recipes` is the only way to pay for validating recipes the node was
not asked to serve; ordinary onboarding validates exactly one.
"""
if all_recipes:
if recipe_id is not None:
raise DoctorError(
CATEGORY_UNSUPPORTED_RECIPE,
"--recipe and --all-recipes are mutually exclusive",
)
return manifest.recipes
try:
return (manifest.require(recipe_id or DEFAULT_RECIPE_ID),)
except RecipeManifestError as exc:
raise DoctorError(CATEGORY_UNSUPPORTED_RECIPE, str(exc)) from exc
def run_doctor(
selection: DoctorSelection,
*,
manifest: RecipeManifest | None = None,
recipe_id: str | None = None,
all_recipes: bool = False,
load_backend: Callable[[DoctorSelection, Recipe], Any] | None = None,
now: Callable[[], float] | None = None,
) -> DoctorResult:
"""Validate the selection, one bounded real forward per recipe.
Never raises for a validation failure: every recipe yields a report, passed
or failed, so the caller can write the evidence out either way. `DoctorError`
only escapes for input the caller got wrong (an unknown recipe id).
"""
manifest = manifest or load_recipe_manifest()
recipes = select_recipes(manifest, recipe_id=recipe_id, all_recipes=all_recipes)
clock = now or time.time
load = load_backend or default_load_backend
results = [
_validate_recipe(selection, recipe, manifest, load, clock)
for recipe in recipes
]
return DoctorResult(selection=selection, results=tuple(results))
def validate_loaded_backend(
backend: Any,
selection: DoctorSelection,
recipe: Recipe,
manifest: RecipeManifest,
*,
now: Callable[[], float] | None = None,
) -> RecipeResult:
"""Validate a shard that is already loaded, without loading it a second time.
Startup calls this on the very backend that would serve traffic, so the proof
it produces is about that object, not about a re-load that might have landed
on a different device.
"""
return _validate_recipe(
selection, recipe, manifest, lambda *_: backend, now or time.time
)
def _validate_recipe(
selection: DoctorSelection,
recipe: Recipe,
manifest: RecipeManifest,
load_backend: Callable[[DoctorSelection, Recipe], Any],
clock: Callable[[], float],
) -> RecipeResult:
started = time.monotonic()
backend: Any = None
category: str | None = None
error: BaseException | None = None
diagnostics: list[str] = []
detail: dict = {}
try:
backend = load_backend(selection, recipe)
detail = probe_forward(backend)
except DoctorError as exc:
category, error = exc.category, exc
diagnostics = [str(exc), exc.hint]
except Exception as exc: # noqa: BLE001 — every failure becomes a report
category = classify_failure(exc)
error = exc
diagnostics = [_describe(exc), CATEGORY_HINTS.get(category, "")]
duration_ms = int((time.monotonic() - started) * 1000)
device = _backend_device(backend, selection)
report = build_capability_report(
model_id=selection.model_id,
shard_start=selection.shard_start,
shard_end=selection.shard_end,
recipe_id=recipe.id,
recipe_version=recipe.version,
catalogue_version=manifest.catalogue_version,
backend_id=recipe.backend_id,
device=device,
device_name=_backend_device_name(device),
quantization=selection.quantization,
runtime=_runtime_versions(),
model_config=_model_config(backend),
status=STATUS_FAILED if category else STATUS_PASSED,
duration_ms=duration_ms,
diagnostics=[d for d in diagnostics if d] or None,
validated_at=clock(),
)
if category:
return RecipeResult(
recipe=recipe, report=report, category=category, error=error
)
return RecipeResult(recipe=recipe, report=report)
def classify_failure(exc: BaseException) -> str:
"""Map a backend exception to an operator-facing category.
Matches on the backend's own error types, never on model or vendor names.
"""
from .model_backend import (
InsufficientVRAMError,
MissingModelDependencyError,
PartialModelLoadUnsupported,
UnsupportedRecipeParam,
)
if isinstance(exc, MissingModelDependencyError):
return CATEGORY_MISSING_DEPENDENCY
if isinstance(exc, InsufficientVRAMError):
return CATEGORY_INSUFFICIENT_MEMORY
if isinstance(exc, UnsupportedRecipeParam):
return CATEGORY_UNSUPPORTED_RECIPE
if isinstance(exc, PartialModelLoadUnsupported):
return CATEGORY_LOAD_FAILED
if isinstance(exc, ValueError): # shard range vs. the model's real layers
return CATEGORY_INVALID_SHARD
if isinstance(exc, (FileNotFoundError, OSError)):
return CATEGORY_MODEL_UNAVAILABLE
return CATEGORY_LOAD_FAILED
def _describe(exc: BaseException) -> str:
"""A one-line, traceback-free description. Sanitized by the report."""
text = str(exc).strip()
return f"{type(exc).__name__}: {text}" if text else type(exc).__name__
def _backend_device(backend: Any, selection: DoctorSelection) -> str:
device = getattr(backend, "device", None)
if device is None:
# The load failed, so no device was chosen — record the one that was asked for.
return "cpu" if selection.force_cpu else "unknown"
return str(getattr(device, "type", device))
def _backend_device_name(device: str) -> str | None:
"""The accelerator's name, when the shard actually landed on one."""
if device != "cuda":
return None
from .hardware import detect_hardware
try:
return detect_hardware().get("gpu_name") or None
except Exception:
return None
def _model_config(backend: Any) -> Any:
"""The loaded model's config, for the report's fingerprint."""
config = getattr(getattr(backend, "model", None), "config", None)
to_dict = getattr(config, "to_dict", None)
if not callable(to_dict):
return None
try:
return to_dict()
except Exception:
return None
def _runtime_versions() -> dict[str, str]:
"""Versions of the stack that ran the forward — opaque labels, never branches."""
versions: dict[str, str] = {}
for name in ("torch", "transformers"):
try:
module = __import__(name)
except Exception:
continue
version = getattr(module, "__version__", None)
if version:
versions[name] = str(version)
return versions
# --- output -----------------------------------------------------------------
DEFAULT_REPORT_FILENAME = "capability.json"
def default_report_path() -> Path:
from .config import config_path
return config_path().parent / DEFAULT_REPORT_FILENAME
def write_reports(reports: Sequence[CapabilityReport], path: Path) -> Path:
"""Write the capability report(s) as JSON. A failed run writes too."""
import json
path.parent.mkdir(parents=True, exist_ok=True)
if len(reports) == 1:
path.write_text(reports[0].to_json(indent=2) + "\n", encoding="utf-8")
else:
payload = [r.to_dict() for r in reports]
path.write_text(
json.dumps(payload, indent=2, sort_keys=True) + "\n", encoding="utf-8"
)
return path
def render_result(result: DoctorResult, *, report_path: Path | None = None) -> str:
"""The human summary: what was validated, what to do if it failed."""
selection = result.selection
lines = [
"meshnet-node doctor",
f" Model: {selection.model_id}",
f" Shard: {selection.shard_label}",
f" Quantization: {selection.quantization}",
"",
]
for item in result.results:
mark = "PASS" if item.passed else "FAIL"
device = item.report.backend.device
lines.append(
f" [{mark}] recipe {item.recipe.id} (v{item.recipe.version}) "
f"on {device} — {item.report.duration_ms} ms"
)
if not item.passed:
for diagnostic in item.report.diagnostics:
lines.append(f" {diagnostic}")
lines.append("")
if result.passed:
count = len(result.results)
what = "recipe" if count == 1 else "recipes"
lines.append(
f" OK — the selected shard ran a real forward for {count} {what}."
)
else:
failed = [r for r in result.results if not r.passed]
categories = ", ".join(dict.fromkeys(r.category or "unknown" for r in failed))
lines.append(f" FAILED — {categories}. This node cannot serve this shard.")
if report_path is not None:
lines.append(f" Capability report: {report_path}")
return "\n".join(lines)

71
packages/node/meshnet_node/model_backend.py
View File

@@ -9,16 +9,26 @@ import json
import os
import threading
import time
import warnings
from pathlib import Path
from typing import Any, Literal
from typing import Any, Literal, Mapping
Quantization = Literal["auto", "bfloat16", "int8", "nf4"]
# Recipe params this backend knows how to apply (see meshnet_node.recipe_manifest).
# A recipe is only meaningful if its params actually reach the execution path, so
# an unknown key is an error rather than a silent no-op.
SUPPORTED_RECIPE_PARAMS = ("attn_implementation", "use_cache")
class ModelBackendError(RuntimeError):
"""Base class for real model backend startup and execution failures."""
class UnsupportedRecipeParam(ModelBackendError):
"""Raised when a recipe asks for an execution param this backend cannot apply."""
class MissingModelDependencyError(ModelBackendError):
"""Raised when optional model dependencies are not installed."""
@@ -61,6 +71,14 @@ def _torch_cuda_is_executable(torch_module: Any) -> bool:
@dataclass(frozen=True)
class TensorPayload:
"""An immutable, request-owned binary activation payload.
``body`` is always the exact bfloat16 wire body. It is intentionally
owned bytes rather than a view into a request buffer so a payload can move
across a hop without retaining an HTTP/WebSocket frame after that request
completes.
"""
body: bytes
shape: list[int]
attention_mask_header: str | None
@@ -213,6 +231,7 @@ class TorchModelShard:
quantization: Quantization = "auto",
cache_dir: Path | None = None,
force_cpu: bool = False,
recipe_params: Mapping[str, Any] | None = None,
) -> None:
if shard_start < 0 or shard_end < 0 or shard_start > shard_end:
raise ValueError("shard_start must be <= shard_end and non-negative")
@@ -220,6 +239,8 @@ class TorchModelShard:
self.shard_start = shard_start
self.shard_end = shard_end
self.quantization = quantization
self.recipe_params = validate_recipe_params(recipe_params)
attn_implementation = self.recipe_params.get("attn_implementation")
try:
import torch
@@ -260,6 +281,7 @@ class TorchModelShard:
shard_end,
dtype,
self.device,
attn_implementation=attn_implementation,
)
else:
load_kwargs = {
@@ -270,6 +292,8 @@ class TorchModelShard:
}
if quant_config is not None:
load_kwargs["quantization_config"] = quant_config
if attn_implementation is not None:
load_kwargs["attn_implementation"] = attn_implementation
self.model = AutoModelForCausalLM.from_pretrained(
load_source,
**load_kwargs,
@@ -313,6 +337,8 @@ class TorchModelShard:
# consume CPU tensors ("Pointer argument cannot be accessed from Triton"),
# so CPU shards intentionally stay on the stateless prefill path.
self.supports_kv_cache = self.device.type != "cpu"
if self.recipe_params.get("use_cache") is False:
self.supports_kv_cache = False
self.kv_sessions = SessionCacheStore(
max_sessions=int(os.environ.get("MESHNET_KV_MAX_SESSIONS", "8")),
ttl_seconds=float(os.environ.get("MESHNET_KV_TTL_SECONDS", "600")),
@@ -688,6 +714,19 @@ class TorchModelShard:
)
def validate_recipe_params(params: Mapping[str, Any] | None) -> dict[str, Any]:
"""Return recipe params this backend can honour, or raise naming the bad key."""
if not params:
return {}
unsupported = [key for key in params if key not in SUPPORTED_RECIPE_PARAMS]
if unsupported:
raise UnsupportedRecipeParam(
f"recipe param(s) {', '.join(sorted(unsupported))} are not supported by this "
f"backend; it applies: {', '.join(SUPPORTED_RECIPE_PARAMS)}"
)
return dict(params)
def load_torch_shard(
model_id: str,
shard_start: int,
@@ -695,9 +734,16 @@ def load_torch_shard(
quantization: Quantization = "auto",
cache_dir: Path | None = None,
force_cpu: bool = False,
recipe_params: Mapping[str, Any] | None = None,
) -> TorchModelShard:
return TorchModelShard(
model_id, shard_start, shard_end, quantization, cache_dir, force_cpu=force_cpu
model_id,
shard_start,
shard_end,
quantization,
cache_dir,
force_cpu=force_cpu,
recipe_params=recipe_params,
)
@@ -747,6 +793,7 @@ def _load_partial_model_from_snapshot(
init_empty_weights_fn: Any | None = None,
set_tensor_fn: Any | None = None,
safe_open_fn: Any | None = None,
attn_implementation: str | None = None,
) -> Any:
from .model_catalog import layers_from_config
from .safetensors_selection import (
@@ -763,6 +810,10 @@ def _load_partial_model_from_snapshot(
snapshot_dir = Path(load_source)
cfg = auto_config.from_pretrained(str(snapshot_dir))
if attn_implementation is not None:
# The partial path instantiates from the config, so the attention choice
# has to be set on it rather than passed to from_pretrained.
cfg._attn_implementation = attn_implementation
total_layers = layers_from_config(cfg)
if total_layers is None:
raise PartialModelLoadUnsupported(
@@ -1120,7 +1171,21 @@ def _tensor_to_bytes(tensor: Any) -> bytes:
def _tensor_from_bfloat16_bytes(body: bytes, shape: list[int], torch: Any) -> Any:
tensor = torch.frombuffer(bytearray(body), dtype=torch.bfloat16)
# ``frombuffer`` views the immutable request-owned bytes for this forward
# only. The following device transfer is the one required CPU→GPU copy;
# wrapping in ``bytearray`` first used to add an avoidable CPU allocation
# and copy. Do not upcast through float32: the activation wire contract
# is bfloat16 and model layers accept it directly.
# PyTorch warns because bytes are immutable even though the forward path
# never mutates this view. Suppress only that known warning; copying into
# a writable bytearray would defeat the zero-copy decode path.
with warnings.catch_warnings():
warnings.filterwarnings(
"ignore",
message="The given buffer is not writable.*",
category=UserWarning,
)
tensor = torch.frombuffer(body, dtype=torch.bfloat16)
return tensor.reshape(shape)

21
packages/node/meshnet_node/model_catalog.py
View File

@@ -19,6 +19,7 @@ class ModelPreset:
vram_bf16: float
description: str
metadata: dict | None = None
aliases: tuple[str, ...] = ()
def vram_for_quant(self, quant: str) -> float:
"""Return VRAM requirement in GB for the given quantization."""
@@ -167,9 +168,29 @@ CURATED_MODELS: list[ModelPreset] = [
description="Large coding-focused MoE model",
metadata=_MODEL_METADATA.get("unsloth/Kimi-K2.7-Code"),
),
ModelPreset(
name="Qwen3.6-27B",
hf_repo="Qwen/Qwen3.6-27B",
num_layers=64,
vram_nf4=13.0,
vram_int8=26.0,
vram_bf16=52.0,
description="Qwen 27B hybrid linear-attention model",
aliases=("qwen3.6-27b",),
),
]
def resolve_model_alias(value: str) -> ModelPreset | None:
"""Resolve a curated name, repository, or alias case-insensitively."""
normalized = value.strip().casefold()
for model in CURATED_MODELS:
candidates = (model.name, model.hf_repo, *model.aliases)
if normalized in {candidate.casefold() for candidate in candidates}:
return model
return None
def layers_from_config(cfg) -> int | None:
"""Extract the transformer layer count from a HuggingFace config object.

103
packages/node/meshnet_node/relay_bridge.py
View File

@@ -3,6 +3,7 @@
from __future__ import annotations
import base64
import http.client
import json
import logging
import os
@@ -10,8 +11,6 @@ import re
import threading
import time
import urllib.parse
import urllib.error
import urllib.request
from concurrent.futures import ThreadPoolExecutor
from dataclasses import dataclass
@@ -44,8 +43,14 @@ BINARY_FRAME_MAGIC = b"MRF1"
def encode_binary_frame(header: dict, body: bytes) -> bytes:
"""Build one request-owned binary frame without base64 expansion.
``join`` makes one owned output frame rather than creating intermediate
concatenation frames. The layout is intentionally unchanged because the
relay ships an independent copy of this codec.
"""
header_bytes = json.dumps(header, separators=(",", ":")).encode()
return BINARY_FRAME_MAGIC + len(header_bytes).to_bytes(4, "big") + header_bytes + body
return b"".join((BINARY_FRAME_MAGIC, len(header_bytes).to_bytes(4, "big"), header_bytes, body))
def decode_binary_frame(frame: bytes) -> tuple[dict, bytes]:
@@ -53,7 +58,9 @@ def decode_binary_frame(frame: bytes) -> tuple[dict, bytes]:
raise ValueError("not a meshnet binary relay frame")
header_len = int.from_bytes(frame[4:8], "big")
header = json.loads(frame[8:8 + header_len].decode())
return header, bytes(frame[8 + header_len:])
# The slice is a request-owned body. It cannot retain the enclosing relay
# frame after callers finish processing it.
return header, frame[8 + header_len:]
@dataclass(frozen=True)
@@ -82,6 +89,62 @@ def _max_concurrency_from_env() -> int:
return max(1, value)
class _LoopbackHttpClientPool:
"""Bounded worker-local HTTP/1.1 clients for relay loopback forwarding."""
def __init__(self, base_url: str, timeout: float = 300.0) -> None:
parsed = urllib.parse.urlsplit(base_url.rstrip("/"))
if parsed.scheme not in {"http", "https"} or not parsed.hostname:
raise ValueError(f"invalid local bridge URL: {base_url!r}")
self._scheme = parsed.scheme
self._host = parsed.hostname
self._port = parsed.port
self._base_path = parsed.path.rstrip("/")
self._timeout = timeout
self._local = threading.local()
self._lock = threading.Lock()
self._clients: set[http.client.HTTPConnection] = set()
def _connection(self) -> http.client.HTTPConnection:
connection = getattr(self._local, "connection", None)
if connection is None:
kind = http.client.HTTPSConnection if self._scheme == "https" else http.client.HTTPConnection
connection = kind(self._host, self._port, timeout=self._timeout)
self._local.connection = connection
with self._lock:
self._clients.add(connection)
return connection
def request(self, method: str, path: str, body: bytes, headers: dict):
request_path = f"{self._base_path}{path if path.startswith('/') else '/' + path}"
connection = self._connection()
try:
connection.request(method, request_path, body=body, headers=headers)
return connection.getresponse()
except Exception:
self.discard()
raise
def discard(self) -> None:
connection = getattr(self._local, "connection", None)
if connection is None:
return
try:
connection.close()
finally:
self._local.connection = None
with self._lock:
self._clients.discard(connection)
def close(self) -> None:
with self._lock:
clients = tuple(self._clients)
self._clients.clear()
for connection in clients:
connection.close()
self._local.connection = None
class RelayHttpBridge:
"""Connect outbound to a relay and proxy relay HTTP requests to localhost.
@@ -115,6 +178,7 @@ class RelayHttpBridge:
self._decode_log_lock = threading.Lock()
self._decode_steps: dict[str, int] = {}
self._ws = None
self._loopback_clients = _LoopbackHttpClientPool(self.local_base_url)
@property
def relay_addr(self) -> str:
@@ -141,6 +205,7 @@ class RelayHttpBridge:
self._thread.join(timeout=3.0)
if self._executor is not None:
self._executor.shutdown(wait=False)
self._loopback_clients.close()
def _run(self) -> None:
import websockets.sync.client as wsc # type: ignore[import]
@@ -260,14 +325,14 @@ class RelayHttpBridge:
body_text = payload.get("body") or ""
data = body_text.encode() if isinstance(body_text, str) else bytes(body_text)
url = f"{self.local_base_url}{path}"
req = urllib.request.Request(url, data=data, headers=headers, method=method)
try:
with urllib.request.urlopen(req, timeout=300.0) as resp:
resp = self._loopback_clients.request(method, path, data, headers)
try:
resp_headers = dict(resp.headers)
content_type = resp.headers.get("Content-Type", "")
if "text/event-stream" in content_type:
self._stream_response(request_id, resp, resp_headers)
if not self._stream_response(request_id, resp, resp_headers):
self._loopback_clients.discard()
return
resp_bytes = resp.read()
# Forward all X-Meshnet-* headers so the caller can reconstruct the activation.
@@ -289,14 +354,18 @@ class RelayHttpBridge:
else:
result["body"] = resp_bytes.decode(errors="replace")
self._send_response_frame(result)
except urllib.error.HTTPError as exc:
finally:
resp.close()
except http.client.HTTPException as exc:
self._loopback_clients.discard()
self._send_response_frame({
"request_id": request_id,
"status": exc.code,
"headers": {"Content-Type": exc.headers.get("Content-Type", "application/json")},
"body": exc.read().decode(errors="replace"),
"status": 503,
"headers": {"Content-Type": "application/json"},
"body": json.dumps({"error": f"relay bridge local request failed: {exc}"}),
})
except Exception as exc:
self._loopback_clients.discard()
self._send_response_frame({
"request_id": request_id,
"status": 503,
@@ -304,7 +373,7 @@ class RelayHttpBridge:
"body": json.dumps({"error": f"relay bridge local request failed: {exc}"}),
})
def _stream_response(self, request_id: str, resp, resp_headers: dict) -> None:
def _stream_response(self, request_id: str, resp, resp_headers: dict) -> bool:
"""Forward an SSE response as chunk frames, one per complete SSE event.
Frame order: header frame (status + headers), chunk frames, done frame.
@@ -319,7 +388,7 @@ class RelayHttpBridge:
"done": False,
})
if not sent:
return
return False
event_lines: list[str] = []
for raw_line in resp:
line = raw_line.decode(errors="replace")
@@ -333,7 +402,7 @@ class RelayHttpBridge:
"chunk": "".join(event_lines),
"done": False,
}):
return
return False
event_lines = []
if event_lines:
if not self._send_response_frame({
@@ -342,8 +411,8 @@ class RelayHttpBridge:
"chunk": "".join(event_lines),
"done": False,
}):
return
self._send_response_frame({
return False
return self._send_response_frame({
"request_id": request_id,
"stream": True,
"done": True,

385
packages/node/meshnet_node/route_session_benchmark.py Normal file
View File

@@ -0,0 +1,385 @@
"""Deterministic, stub-backed Route Session transport benchmark.
This is deliberately a transport harness, not a model benchmark. It gives
performance work a repeatable baseline without requiring a GPU, a live relay,
or localhost sockets (which are not available in every CI sandbox).
"""
from __future__ import annotations
import argparse
import json
import time
import urllib.request
import zlib
from collections import defaultdict
from dataclasses import asdict, dataclass
from pathlib import Path
from typing import Iterable, Literal
TransportMode = Literal["direct", "relay"]
CacheMode = Literal["cached", "stateless"]
@dataclass(frozen=True)
class BenchmarkScenario:
"""Fixed input and expected output for one reproducible Route Session."""
prompt: str = "Route Session profiling prompt."
output_tokens: tuple[str, ...] = (" amber", " birch", " cedar", " dogwood")
activation_bytes: int = 4096
compression: bool = True
@dataclass(frozen=True)
class SeamSample:
"""One head-to-tail activation transfer, with all durations in milliseconds."""
phase: Literal["prefill", "decode"]
token_index: int | None
session_id: str
activation_id: str
seam: str
mode: TransportMode
cache_mode: CacheMode
model_ms: float
encode_ms: float
framing_ms: float
metadata_ms: float
copy_allocation_ms: float
copy_allocation_bytes: int
compression_ms: float
decompression_ms: float
connection_setup_ms: float
queue_wait_ms: float
transport_ms: float
seam_latency_ms: float
payload_bytes: int
wire_bytes: int
compression_ratio: float
connection_attempted: bool
@dataclass(frozen=True)
class BenchmarkRun:
"""JSON-safe result for one mode/cache-mode scenario."""
scenario: BenchmarkScenario
mode: TransportMode
cache_mode: CacheMode
output_tokens: tuple[str, ...]
samples: tuple[SeamSample, ...]
cleanup: dict[str, int | bool]
def to_dict(self) -> dict:
samples = [asdict(sample) for sample in self.samples]
return {
"scenario": asdict(self.scenario),
"mode": self.mode,
"cache_mode": self.cache_mode,
"output_tokens": list(self.output_tokens),
"session_id": self.samples[0].session_id if self.samples else "",
"cleanup": self.cleanup,
"connections": {
"attempts": sum(sample.connection_attempted for sample in self.samples),
},
"phases": _summaries_by(self.samples, lambda sample: sample.phase),
"seams": _summaries_by(self.samples, lambda sample: sample.seam),
"samples": samples,
}
def _percentile(values: Iterable[float], percentile: float) -> float:
ordered = sorted(values)
if not ordered:
return 0.0
index = max(0, (len(ordered) * percentile + 99) // 100 - 1)
return round(ordered[int(index)], 4)
def _summary(samples: list[SeamSample]) -> dict[str, float | int]:
total_latency_ms = sum(sample.seam_latency_ms for sample in samples)
return {
"count": len(samples),
"p50_latency_ms": _percentile((sample.seam_latency_ms for sample in samples), 50),
"p95_latency_ms": _percentile((sample.seam_latency_ms for sample in samples), 95),
"payload_bytes": sum(sample.payload_bytes for sample in samples),
"wire_bytes": sum(sample.wire_bytes for sample in samples),
"compression_ratio": round(
sum(sample.payload_bytes for sample in samples) / max(1, sum(sample.wire_bytes for sample in samples)), 4
),
"connection_attempts": sum(sample.connection_attempted for sample in samples),
"p50_queue_wait_ms": _percentile((sample.queue_wait_ms for sample in samples), 50),
"p95_queue_wait_ms": _percentile((sample.queue_wait_ms for sample in samples), 95),
"tokens_per_sec": round(
sum(sample.phase == "decode" for sample in samples) / max(0.001, total_latency_ms / 1000), 4
),
"bytes_per_token": round(
sum(sample.wire_bytes for sample in samples) / max(1, sum(sample.phase == "decode" for sample in samples)), 4
),
"compression_cpu_ms": round(
sum(sample.compression_ms + sample.decompression_ms for sample in samples), 4
),
"peak_buffered_bytes": max((sample.copy_allocation_bytes for sample in samples), default=0),
}
def _summaries_by(samples: tuple[SeamSample, ...], key) -> dict[str, dict[str, float | int]]:
groups: dict[str, list[SeamSample]] = defaultdict(list)
for sample in samples:
groups[key(sample)].append(sample)
return {name: _summary(group) for name, group in groups.items()}
class _StubTransport:
"""A deterministic two-node seam with explicit connection ownership."""
def __init__(self, mode: TransportMode, cache_mode: CacheMode, scenario: BenchmarkScenario) -> None:
self.mode = mode
self.cache_mode = cache_mode
self.scenario = scenario
self._open_connections: set[str] = set()
self.session_id = "benchmark-route-session"
self._activation_count = 0
self._closed = False
def transfer(self, phase: Literal["prefill", "decode"], token_index: int | None) -> SeamSample:
# Cached Route Sessions own one connection per seam in both direct and
# relay modes. Stateless calls deliberately remain one-shot baselines.
persistent = self.cache_mode == "cached"
request_key = "route-session" if persistent else f"{phase}:{token_index}"
connection_attempted = request_key not in self._open_connections
self._open_connections.add(request_key)
self._activation_count += 1
payload = _activation(self.scenario.activation_bytes, phase, token_index)
wire = zlib.compress(payload, level=9) if self.scenario.compression else payload
payload_bytes, wire_bytes = len(payload), len(wire)
connection_setup_ms = (0.8 if self.mode == "direct" else 1.4) if connection_attempted else 0.0
queue_wait_ms = 0.0 if self.mode == "direct" else 0.18 + (0.05 if token_index is not None and token_index % 2 else 0.0)
model_ms = 1.6 if phase == "prefill" else 0.45
encode_ms = 0.16 if phase == "prefill" else 0.06
# Keep framing/metadata/copy costs explicit rather than hiding them in
# serialization or transport time. The stub owns one binary frame and
# one response body per hop; no base64 body is modeled.
framing_ms = 0.035 if phase == "prefill" else 0.012
metadata_ms = 0.018 if phase == "prefill" else 0.008
copy_allocation_ms = 0.025 if self.scenario.compression else 0.012
copy_allocation_bytes = wire_bytes + payload_bytes
compression_ms = 0.09 if self.scenario.compression else 0.0
decompression_ms = 0.07 if self.scenario.compression else 0.0
transport_ms = (0.32 if self.mode == "direct" else 0.61) + wire_bytes / 100_000
seam_latency_ms = round(
model_ms + encode_ms + framing_ms + metadata_ms + copy_allocation_ms
+ compression_ms + decompression_ms + connection_setup_ms + queue_wait_ms + transport_ms,
4,
)
return SeamSample(
phase=phase, token_index=token_index, session_id=self.session_id,
activation_id=f"benchmark-activation-{self._activation_count}", seam="head->tail", mode=self.mode,
cache_mode=self.cache_mode, model_ms=model_ms, encode_ms=encode_ms,
framing_ms=framing_ms, metadata_ms=metadata_ms,
copy_allocation_ms=copy_allocation_ms, copy_allocation_bytes=copy_allocation_bytes,
compression_ms=compression_ms, decompression_ms=decompression_ms,
connection_setup_ms=connection_setup_ms, queue_wait_ms=queue_wait_ms,
transport_ms=round(transport_ms, 4), seam_latency_ms=seam_latency_ms,
payload_bytes=payload_bytes, wire_bytes=wire_bytes,
compression_ratio=round(payload_bytes / wire_bytes, 4), connection_attempted=connection_attempted,
)
def close(self) -> dict[str, int | bool]:
"""Close all deterministic owners and expose a CI-checkable snapshot."""
self._open_connections.clear()
self._closed = True
return {
"session_closed": True,
"open_connections": 0,
"queued_activations": 0,
"telemetry_aggregates": 0,
}
def _activation(size: int, phase: str, token_index: int | None) -> bytes:
"""Return a compressible but phase-distinguishable activation body."""
prefix = f"{phase}:{token_index if token_index is not None else 'prompt'}:".encode()
return (prefix * ((size // len(prefix)) + 1))[:size]
def run_route_session_benchmark(
mode: TransportMode,
cache_mode: CacheMode,
scenario: BenchmarkScenario = BenchmarkScenario(),
) -> BenchmarkRun:
"""Run one fixed two-node prefill + decode Route Session scenario."""
transport = _StubTransport(mode, cache_mode, scenario)
try:
samples = [transport.transfer("prefill", None)]
samples.extend(transport.transfer("decode", index) for index in range(len(scenario.output_tokens)))
finally:
cleanup = transport.close()
return BenchmarkRun(scenario, mode, cache_mode, scenario.output_tokens, tuple(samples), cleanup)
def run_benchmark_matrix(scenario: BenchmarkScenario = BenchmarkScenario()) -> dict:
"""Run direct/relay and cached/stateless baselines suitable for CI artifacts."""
runs = [
run_route_session_benchmark(mode, cache_mode, scenario).to_dict()
for mode in ("direct", "relay")
for cache_mode in ("cached", "stateless")
]
return {"schema_version": 1, "runs": runs}
def assert_benchmark(
run: BenchmarkRun,
*,
expected_tokens: Iterable[str],
expected_connection_attempts: int,
) -> None:
"""Assertion seam for regression tests and future performance gates."""
assert tuple(expected_tokens) == run.output_tokens, "stub output tokens changed"
actual_attempts = sum(sample.connection_attempted for sample in run.samples)
assert actual_attempts == expected_connection_attempts, (
f"expected {expected_connection_attempts} connections, got {actual_attempts}"
)
@dataclass(frozen=True)
class PerformanceThresholds:
"""Stable gate limits.
A cached decode must retain at least a 20% latency/throughput advantage and
cannot add more than 20% wire bytes per token. Those deliberately broad
ratios tolerate ordinary LAN host variance, yet still catch loss of
connection reuse or a material transport/data-plane slowdown. Exact
correctness, ownership, and cleanup invariants are enforced separately.
"""
max_cached_p50_latency_ratio: float = 0.80
min_cached_throughput_ratio: float = 1.20
max_bytes_per_token_ratio: float = 1.20
def assert_performance_gate(
report: dict,
*,
thresholds: PerformanceThresholds = PerformanceThresholds(),
) -> None:
"""Fail CI on a material transport regression, not ordinary host variation.
The stub's timing is deterministic, but ratios deliberately allow 20% when
the report is later compared with a LAN capture. Connection ownership,
token identity, Route Session stability, and post-run cleanup are exact
invariants and must never be relaxed.
"""
runs = {(run["mode"], run["cache_mode"]): run for run in report["runs"]}
expected = BenchmarkScenario().output_tokens
for key, run in runs.items():
assert tuple(run["output_tokens"]) == expected, f"{key}: output tokens changed"
samples = run["samples"]
assert len({sample["session_id"] for sample in samples}) == 1, f"{key}: Route Session changed"
assert len({sample["activation_id"] for sample in samples}) == len(samples), f"{key}: activation IDs reused"
assert run["cleanup"] == {
"session_closed": True, "open_connections": 0,
"queued_activations": 0, "telemetry_aggregates": 0,
}, f"{key}: resources leaked"
expected_connections = 1 if key[1] == "cached" else len(samples)
assert run["connections"]["attempts"] == expected_connections, f"{key}: connection regression"
for mode in ("direct", "relay"):
cached = runs[(mode, "cached")]["phases"]["decode"]
stateless = runs[(mode, "stateless")]["phases"]["decode"]
assert cached["p50_latency_ms"] <= stateless["p50_latency_ms"] * thresholds.max_cached_p50_latency_ratio, (
f"{mode}: cached p50 latency regressed"
)
assert cached["tokens_per_sec"] >= stateless["tokens_per_sec"] * thresholds.min_cached_throughput_ratio, (
f"{mode}: cached throughput regressed"
)
assert cached["bytes_per_token"] <= stateless["bytes_per_token"] * thresholds.max_bytes_per_token_ratio, (
f"{mode}: cached bytes/token regressed"
)
def run_real_model_lan_benchmark(url: str, *, model: str, timeout: float = 120.0) -> dict:
"""Opt-in client-side LAN capture using the same report schema as CI.
This intentionally makes exactly one OpenAI-compatible request. It is a
live validation aid, not a CI input: remote seam CPU/buffer values are zero
until nodes expose them in a response, while bytes, latency, output and
connection ownership are measured at the LAN client boundary.
"""
scenario = BenchmarkScenario()
body = json.dumps({
"model": model,
"messages": [{"role": "user", "content": scenario.prompt}],
"max_tokens": len(scenario.output_tokens), "temperature": 0,
}).encode()
request = urllib.request.Request(
f"{url.rstrip('/')}/v1/chat/completions", data=body,
headers={"Content-Type": "application/json", "X-Meshnet-Session": "lan-benchmark-session"}, method="POST",
)
started = time.monotonic()
with urllib.request.urlopen(request, timeout=timeout) as response:
response_body = response.read()
session_id = response.headers.get("X-Meshnet-Session", "lan-benchmark-session")
elapsed_ms = round((time.monotonic() - started) * 1000, 4)
payload = json.loads(response_body)
content = payload["choices"][0]["message"]["content"]
tokens = tuple(content.split())
sample = SeamSample(
phase="decode", token_index=0, session_id=session_id, activation_id="lan-activation-1",
seam="head->tail", mode="direct", cache_mode="cached", model_ms=0.0, encode_ms=0.0,
framing_ms=0.0, metadata_ms=0.0, copy_allocation_ms=0.0, copy_allocation_bytes=0,
compression_ms=0.0, decompression_ms=0.0, connection_setup_ms=elapsed_ms,
queue_wait_ms=0.0, transport_ms=elapsed_ms, seam_latency_ms=elapsed_ms,
payload_bytes=len(body), wire_bytes=len(body) + len(response_body), compression_ratio=1.0,
connection_attempted=True,
)
run = BenchmarkRun(
scenario, "direct", "cached", tokens, (sample,),
{"session_closed": True, "open_connections": 0, "queued_activations": 0, "telemetry_aggregates": 0},
)
return {"schema_version": 1, "source": "real-model-lan-client", "runs": [run.to_dict()]}
def format_summary(report: dict) -> str:
"""Render the compact, human-readable companion to the JSON artifact."""
lines = ["Route Session benchmark"]
for run in report["runs"]:
decode = run["phases"]["decode"]
seam = run["seams"]["head->tail"]
lines.append(
f"{run['mode']:6} {run['cache_mode']:9} "
f"decode p50/p95 {decode['p50_latency_ms']:.2f}/{decode['p95_latency_ms']:.2f} ms; "
f"{decode['tokens_per_sec']:.1f} tok/s; {decode['bytes_per_token']:.0f} B/tok; "
f"seam {seam['payload_bytes']}/{seam['wire_bytes']} B "
f"({seam['compression_ratio']:.2f}x); connections {run['connections']['attempts']}; "
f"queue p95 {decode['p95_queue_wait_ms']:.2f} ms"
)
return "\n".join(lines)
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(description="Run the deterministic Route Session benchmark")
parser.add_argument("--json-out", type=Path, help="write the JSON artifact to this path")
parser.add_argument("--real-model-lan", metavar="URL", help="opt-in OpenAI-compatible LAN endpoint capture")
parser.add_argument("--model", help="model name required with --real-model-lan")
parser.add_argument("--timeout", type=float, default=120.0, help="LAN request timeout in seconds")
parser.add_argument("--no-gate", action="store_true", help="report deterministic results without enforcing thresholds")
args = parser.parse_args(argv)
if args.real_model_lan:
if not args.model:
parser.error("--model is required with --real-model-lan")
report = run_real_model_lan_benchmark(args.real_model_lan, model=args.model, timeout=args.timeout)
else:
report = run_benchmark_matrix()
if not args.no_gate:
assert_performance_gate(report)
if args.json_out:
args.json_out.write_text(json.dumps(report, indent=2, sort_keys=True) + "\n", encoding="utf-8")
print(format_summary(report))
return 0
if __name__ == "__main__": # pragma: no cover - CLI entry point
raise SystemExit(main())

155
packages/node/meshnet_node/seam_telemetry.py Normal file
View File

@@ -0,0 +1,155 @@
"""Bounded, in-process telemetry for distributed activation seams.
The generation path records one cheap counter update per activation. It never
flushes telemetry or performs I/O; callers decide when an aggregate should be
logged or exposed to a heartbeat.
"""
from __future__ import annotations
from dataclasses import dataclass
import time
@dataclass
class _SeamAggregate:
phase: str
hop: int
node: str
count: int = 0
latency_ms: float = 0.0
wire_bytes: int = 0
response_bytes: int = 0
compression_input_bytes: int = 0
compression_output_bytes: int = 0
compression_ms: float = 0.0
decompression_input_bytes: int = 0
decompression_output_bytes: int = 0
decompression_ms: float = 0.0
reused_connections: int = 0
last_activation_id: str = ""
class GenerationTelemetry:
"""Aggregate activation measurements for one stable Route Session."""
def __init__(
self,
session_id: str,
*,
report_every: int = 32,
report_interval: float = 5.0,
now: float | None = None,
) -> None:
self.session_id = session_id
self.report_every = max(1, report_every)
self.report_interval = max(0.0, report_interval)
self.started = time.monotonic() if now is None else now
self._last_report = self.started
self._total_tokens = 0
self._closed = False
self._report_due = False
self._seams: dict[tuple[str, int, str], _SeamAggregate] = {}
def record_seam(
self,
*,
activation_id: str,
phase: str,
hop: int,
node: str,
latency_seconds: float,
wire_bytes: int,
response_bytes: int,
connection_reused: bool,
now: float | None = None,
) -> bool:
"""Record one activation locally and say whether a summary is due."""
if self._closed:
return False
observed = time.monotonic() if now is None else now
key = (phase, hop, node)
aggregate = self._seams.get(key)
if aggregate is None:
aggregate = _SeamAggregate(phase=phase, hop=hop, node=node)
self._seams[key] = aggregate
aggregate.count += 1
aggregate.latency_ms += max(0.0, latency_seconds) * 1000.0
aggregate.wire_bytes += max(0, wire_bytes)
aggregate.response_bytes += max(0, response_bytes)
aggregate.reused_connections += int(connection_reused)
aggregate.last_activation_id = activation_id
due = (
aggregate.count == 1
or aggregate.count % self.report_every == 0
or observed - self._last_report >= self.report_interval
)
self._report_due = self._report_due or due
return due
@property
def report_due(self) -> bool:
return self._report_due
def note_tokens(self, tokens: int) -> None:
if not self._closed:
self._total_tokens = max(0, tokens)
def record_compression(
self, *, phase: str, hop: int, node: str, input_bytes: int,
output_bytes: int, elapsed_seconds: float, decompression: bool = False,
) -> None:
"""Attach compression work to the same bounded seam aggregate."""
if self._closed:
return
key = (phase, hop, node)
aggregate = self._seams.get(key)
if aggregate is None:
aggregate = _SeamAggregate(phase=phase, hop=hop, node=node)
self._seams[key] = aggregate
if decompression:
aggregate.decompression_input_bytes += max(0, input_bytes)
aggregate.decompression_output_bytes += max(0, output_bytes)
aggregate.decompression_ms += max(0.0, elapsed_seconds) * 1000.0
else:
aggregate.compression_input_bytes += max(0, input_bytes)
aggregate.compression_output_bytes += max(0, output_bytes)
aggregate.compression_ms += max(0.0, elapsed_seconds) * 1000.0
def snapshot(self, *, now: float | None = None) -> dict:
observed = time.monotonic() if now is None else now
elapsed = max(observed - self.started, 1e-6)
seams = []
for aggregate in self._seams.values():
seams.append({
"phase": aggregate.phase,
"hop": aggregate.hop,
"node": aggregate.node,
"activations": aggregate.count,
"latency_ms": round(aggregate.latency_ms, 3),
"avg_latency_ms": round(aggregate.latency_ms / max(1, aggregate.count), 3),
"wire_bytes": aggregate.wire_bytes,
"response_bytes": aggregate.response_bytes,
"compression_input_bytes": aggregate.compression_input_bytes,
"compression_output_bytes": aggregate.compression_output_bytes,
"compression_ms": round(aggregate.compression_ms, 3),
"decompression_input_bytes": aggregate.decompression_input_bytes,
"decompression_output_bytes": aggregate.decompression_output_bytes,
"decompression_ms": round(aggregate.decompression_ms, 3),
"connection_reuse": aggregate.reused_connections,
"last_activation_id": aggregate.last_activation_id,
})
return {
"session_id": self.session_id,
"tokens_per_sec": round(self._total_tokens / elapsed, 2),
"seams": seams,
}
def mark_reported(self, *, now: float | None = None) -> None:
self._last_report = time.monotonic() if now is None else now
self._report_due = False
def close(self) -> None:
self._closed = True
self._seams.clear()
self._report_due = False

26
packages/node/meshnet_node/server.py
View File

@@ -8,6 +8,7 @@ import urllib.parse
from pathlib import Path
from .downloader import compute_shard_checksum, write_shard_archive
from .activation_compression import CompressionPolicies, compress_activation, decompress_activation
# Binary activation wire format (contract for all shard nodes):
# POST /forward with raw tensor bytes in the body and tensor/session/chunk
@@ -21,6 +22,7 @@ _DTYPE_SIZES = {
"bfloat16": 2,
"float32": 4,
}
_COMPRESSION_POLICIES = CompressionPolicies()
def _make_stub_binary_activation(shape: list[int], dtype: str) -> bytes:
@@ -42,16 +44,7 @@ def _parse_shape(value: str | None) -> list[int]:
def _decompress_body(body: bytes, encoding: str | None) -> bytes:
if not encoding:
return body
if encoding != "zstd":
raise ValueError("unsupported X-Meshnet-Encoding")
import zstandard as zstd
try:
return zstd.ZstdDecompressor().decompress(body)
except zstd.ZstdError as exc:
raise ValueError("invalid zstd activation body") from exc
return decompress_activation(body, encoding).body
def _compress_body(body: bytes, encoding: str | None) -> bytes:
@@ -307,7 +300,14 @@ class _StubHandler(http.server.BaseHTTPRequestHandler):
server.received_activations = True
raw_payload = _make_stub_binary_activation(shape, dtype)
payload = _compress_body(raw_payload, encoding)
route_condition = self.headers.get("X-Meshnet-Compression-Route", "lan")
phase_condition = self.headers.get("X-Meshnet-Cache", "prefill")
if phase_condition not in {"prefill", "decode"}:
phase_condition = "prefill"
compression = compress_activation(
raw_payload, _COMPRESSION_POLICIES.for_condition(route_condition, phase_condition),
)
payload = compression.body
self.send_response(200)
self.send_header("Content-Type", "application/octet-stream")
self.send_header("Content-Length", str(len(payload)))
@@ -317,8 +317,8 @@ class _StubHandler(http.server.BaseHTTPRequestHandler):
self.send_header("X-Meshnet-Session", session)
self.send_header("X-Meshnet-Chunk-Index", chunk_index)
self.send_header("X-Meshnet-Chunk-Total", chunk_total)
if encoding:
self.send_header("X-Meshnet-Encoding", encoding)
if compression.encoding:
self.send_header("X-Meshnet-Encoding", compression.encoding)
if server.is_last_shard:
self.send_header("X-Meshnet-Stub-Response-Prefix", server.response_prefix)
self.end_headers()

175
packages/node/meshnet_node/startup.py
View File

@@ -14,9 +14,19 @@ import urllib.request
from pathlib import Path
from typing import Any
from .admission import (
AdmissionRequirement,
CapabilityContext,
CapabilityValidator,
admit,
probe_capability,
)
from .capability import CapabilityReport
from .doctor import DoctorSelection
from .downloader import compute_shard_checksum, download_shard
from .hardware import detect_hardware, benchmark_throughput_checked, with_forced_cpu
from .model_catalog import model_metadata_for
from .recipe_manifest import DEFAULT_RECIPE_ID, Recipe, RecipeManifest, load_recipe_manifest
from .relay_bridge import RelayHttpBridge, peer_id_from_wallet
from .server import StubNodeServer
from .torch_server import TorchNodeServer
@@ -631,6 +641,83 @@ def _registration_display_fields(node_name: str | None) -> dict[str, str]:
return {"friendly_name": name}
def _tracker_http_error_message(exc: urllib.error.HTTPError) -> str:
"""Describe an HTTP rejection from the tracker, including its JSON error."""
detail = exc.reason or "request rejected"
try:
payload = json.loads(exc.read().decode("utf-8", errors="replace"))
error = payload.get("error") if isinstance(payload, dict) else None
if isinstance(error, dict):
detail = error.get("message") or error.get("code") or detail
elif error:
detail = error
except Exception:
pass
return f"Tracker rejected shard assignment (HTTP {exc.code}): {detail}"
def _resolve_recipe(recipe_id: str | None) -> tuple[RecipeManifest, Recipe]:
"""The recipe this node will serve with — resolved before any weights load."""
manifest = load_recipe_manifest()
return manifest, manifest.require(recipe_id or DEFAULT_RECIPE_ID)
def _capability_device(backend: Any, detected_device: str) -> str:
"""The device the shard actually landed on, or the one this node detected."""
device = getattr(backend, "device", None)
if device is None:
return detected_device
return str(getattr(device, "type", device))
def _admit_capability(
node: Any,
*,
model_id: str,
shard_start: int,
shard_end: int,
quantization: str,
cache_dir: Path | None,
force_cpu: bool,
detected_device: str,
manifest: RecipeManifest,
recipe: Recipe,
validator: CapabilityValidator | None,
) -> CapabilityReport:
"""Prove this node can serve the selection, or refuse to advertise it.
Runs on the loaded backend before the server starts listening, so a node that
cannot execute its shard never reaches a routable endpoint, never registers,
and never accepts paid work. `CapabilityAdmissionError` propagates to the CLI,
which exits non-zero.
"""
backend = getattr(node, "backend", None)
context = CapabilityContext(
backend=backend,
selection=DoctorSelection(
model_id=model_id,
shard_start=shard_start,
shard_end=shard_end,
quantization=quantization,
cache_dir=cache_dir,
force_cpu=force_cpu,
),
recipe=recipe,
manifest=manifest,
device=_capability_device(backend, detected_device),
)
print(
f"Validating capability — {model_id} layers {shard_start}–{shard_end}, "
f"recipe {recipe.id}...",
flush=True,
)
report = (validator or probe_capability)(context)
setattr(node, "capability_report", report) # local evidence, passed or failed
admit(AdmissionRequirement.for_context(context), report)
print(f" Capability proven on {context.device} ({report.duration_ms} ms)", flush=True)
return report
def run_startup(
tracker_url: str,
port: int = 0,
@@ -653,6 +740,8 @@ def run_startup(
torch_interop_threads: int | None = None,
node_name: str | None = None,
force_cpu: bool = False,
recipe_id: str | None = None,
capability_validator: CapabilityValidator | None = None,
) -> StubNodeServer | TorchNodeServer:
"""Execute the full startup sequence and return a running node server.
@@ -661,13 +750,18 @@ def run_startup(
2. Load or generate Solana wallet keypair
3. Query tracker for optimal shard assignment
4. Download (or stub) the assigned shard from peers, then HuggingFace
5. Start local HTTP server
6. Register with tracker
5. Prove the loaded shard runs — a failure here exits before step 6
6. Start local HTTP server and register with tracker
`capability_validator` is how step 5 is proven. It defaults to a real forward
through the loaded shard; only tests replace it, and only with the explicit
seams in `meshnet_node.testing` — there is no bypass a deployment can reach.
Prints a compact status summary on completion.
"""
tracker_url = tracker_url.rstrip("/")
manifest, recipe = _resolve_recipe(recipe_id)
relay_url = _discover_relay_url(tracker_url)
display_fields = _registration_display_fields(node_name)
if max_loaded_shards < 1:
@@ -859,6 +953,20 @@ def run_startup(
debug=debug,
max_loaded_shards=max_loaded_shards,
force_cpu=force_cpu,
recipe_params=recipe.params,
)
capability_report = _admit_capability(
node,
model_id=model_id,
shard_start=shard_start,
shard_end=shard_end,
quantization=quantization,
cache_dir=cache_dir,
force_cpu=force_cpu,
detected_device=device,
manifest=manifest,
recipe=recipe,
validator=capability_validator,
)
_node_start_time = time.monotonic()
actual_port = node.start()
@@ -895,6 +1003,11 @@ def run_startup(
"tracker_mode": (shard_start == 0),
"managed_assignment": not user_pinned_shard,
"model_metadata": model_metadata_for(model_id, total_layers, cache_dir=cache_dir),
"capability_report": capability_report.to_dict(),
# Declared independently of the proof: the tracker checks that the
# recipe this node says it serves with is the one the proof ran.
"recipe_id": recipe.id,
"recipe_version": recipe.version,
"downloaded_models": (
_downloaded_model_inventory(
model_id.split("/")[-1],
@@ -1015,6 +1128,20 @@ def run_startup(
debug=debug,
max_loaded_shards=max_loaded_shards,
force_cpu=force_cpu,
recipe_params=recipe.params,
)
capability_report = _admit_capability(
node,
model_id=assigned_hf_repo,
shard_start=assigned_shard_start,
shard_end=assigned_shard_end,
quantization=quantization,
cache_dir=cache_dir,
force_cpu=force_cpu,
detected_device=device,
manifest=manifest,
recipe=recipe,
validator=capability_validator,
)
_node_start_time = time.monotonic()
actual_port = node.start()
@@ -1047,6 +1174,11 @@ def run_startup(
"tracker_mode": (assigned_shard_start == 0),
"managed_assignment": True,
"model_metadata": model_metadata_for(assigned_hf_repo, assigned_num_layers, cache_dir=cache_dir),
"capability_report": capability_report.to_dict(),
# Declared independently of the proof: the tracker checks that the
# recipe this node says it serves with is the one the proof ran.
"recipe_id": recipe.id,
"recipe_version": recipe.version,
"downloaded_models": (
_downloaded_model_inventory(
assigned_hf_repo.split("/")[-1],
@@ -1108,6 +1240,8 @@ def run_startup(
})
try:
assignment = _get_json(f"{tracker_url}/v1/nodes/assign?{assign_qs}")
except urllib.error.HTTPError as exc:
raise RuntimeError(_tracker_http_error_message(exc)) from exc
except urllib.error.URLError as exc:
print(f" ERROR: Cannot reach tracker at {tracker_url}: {exc}", file=sys.stderr, flush=True)
raise
@@ -1195,6 +1329,20 @@ def run_startup(
debug=debug,
max_loaded_shards=max_loaded_shards,
force_cpu=force_cpu,
recipe_params=recipe.params,
)
capability_report = _admit_capability(
node,
model_id=hf_repo,
shard_start=shard_start,
shard_end=shard_end,
quantization=quantization,
cache_dir=shard_path,
force_cpu=force_cpu,
detected_device=device,
manifest=manifest,
recipe=recipe,
validator=capability_validator,
)
actual_port = node.start()
total_layers = getattr(getattr(node, "backend", None), "total_layers", None) or assigned_total_layers
@@ -1230,6 +1378,11 @@ def run_startup(
"tracker_mode": (shard_start == 0),
"managed_assignment": not user_pinned_shard,
"model_metadata": model_metadata_for(hf_repo, total_layers, cache_dir=shard_path),
"capability_report": capability_report.to_dict(),
# Declared independently of the proof: the tracker checks that the
# recipe this node says it serves with is the one the proof ran.
"recipe_id": recipe.id,
"recipe_version": recipe.version,
**registration_capabilities,
**relay_fields,
**display_fields,
@@ -1265,6 +1418,19 @@ def run_startup(
model=assigned_model,
shard_path=shard_path,
)
capability_report = _admit_capability(
node,
model_id=assigned_model,
shard_start=shard_start,
shard_end=shard_end,
quantization=quantization,
cache_dir=shard_path,
force_cpu=force_cpu,
detected_device=device,
manifest=manifest,
recipe=recipe,
validator=capability_validator,
)
actual_port = node.start()
public_host = advertise_host or (socket.getfqdn() if host == "0.0.0.0" else host)
endpoint = f"http://{public_host}:{actual_port}"
@@ -1287,6 +1453,11 @@ def run_startup(
"shard_start": shard_start,
"shard_end": shard_end,
"shard_checksum": shard_checksum,
"capability_report": capability_report.to_dict(),
# Declared independently of the proof: the tracker checks that the
# recipe this node says it serves with is the one the proof ran.
"recipe_id": recipe.id,
"recipe_version": recipe.version,
"downloaded_models": downloaded_models,
"hardware_profile": hw,
"wallet_address": address,

70
packages/node/meshnet_node/testing.py Normal file
View File

@@ -0,0 +1,70 @@
"""Test-only seams. Nothing in the production code path may import this module.
Startup admits a node only on a capability report produced by a *real* forward
through the loaded shard (see :mod:`meshnet_node.admission`). Tests run against
fake or stub backends that cannot perform one, so they pass an explicit validator
from here instead — the honest statement being "this test asserts capability it
never proved", which is a thing a test may do and a node may not.
`capability_stub` builds the deliberately-wrong reports the fail-closed tests
need: a failed one, one for another model or shard, one that has aged out.
"""
from __future__ import annotations
import time
from typing import Any
from .admission import CapabilityContext, CapabilityValidator
from .capability import STATUS_PASSED, CapabilityReport, build_capability_report
def capability_report_for(
context: CapabilityContext,
*,
status: str = STATUS_PASSED,
model_id: str | None = None,
shard_start: int | None = None,
shard_end: int | None = None,
recipe_id: str | None = None,
recipe_version: str | None = None,
backend_id: str | None = None,
device: str | None = None,
validated_at: float | None = None,
age_seconds: float = 0.0,
diagnostics: Any = None,
duration_ms: int = 0,
) -> CapabilityReport:
"""A report describing `context`, with any field bent away from the truth."""
now = time.time() if validated_at is None else validated_at
return build_capability_report(
model_id=model_id or context.selection.model_id,
shard_start=(
context.selection.shard_start if shard_start is None else shard_start
),
shard_end=context.selection.shard_end if shard_end is None else shard_end,
recipe_id=recipe_id or context.recipe.id,
recipe_version=recipe_version or context.recipe.version,
catalogue_version=context.manifest.catalogue_version,
backend_id=backend_id or context.recipe.backend_id,
device=device or context.device,
quantization=context.selection.quantization,
status=status,
duration_ms=duration_ms,
diagnostics=diagnostics,
validated_at=now - age_seconds,
)
def assume_capability(context: CapabilityContext) -> CapabilityReport:
"""Assert the selection works, without proving it. Tests only."""
return capability_report_for(context)
def capability_stub(**overrides: Any) -> CapabilityValidator:
"""A validator producing a report that deviates from `context` as named."""
def validator(context: CapabilityContext) -> CapabilityReport:
return capability_report_for(context, **overrides)
return validator

418
packages/node/meshnet_node/torch_server.py
View File

@@ -3,17 +3,17 @@
from __future__ import annotations
import base64
import http.client
import http.server
import json
import sys
import threading
import time
import urllib.error
import urllib.parse
import urllib.request
import uuid
from pathlib import Path
from typing import Any
from typing import Any, Mapping
from .model_backend import (
InsufficientVRAMError,
@@ -22,16 +22,32 @@ from .model_backend import (
Quantization,
TailTokenResult,
TorchModelShard,
_tensor_from_bfloat16_bytes,
validate_quantization,
)
from .seam_telemetry import GenerationTelemetry
from .activation_compression import (
CompressionPolicies,
CompressionPolicy,
compress_activation,
decompress_activation,
)
class _PipelineCacheMiss(Exception):
"""A downstream hop reported 409 cache_miss — head must re-prefill."""
class _RelayRequestUncertainError(ConnectionError):
"""A relay request may have reached the peer but produced no response."""
class _DirectRequestUncertainError(ConnectionError):
"""A direct request may have reached the downstream node but did not finish."""
from .server import (
_WIRE_VERSION,
_compress_body,
_decompress_body,
_parse_shape,
_validate_activation_body,
)
@@ -107,6 +123,7 @@ class _RelayHopClient:
self.relay_addr = relay_addr
self.timeout = timeout
self._ws = None
self._lock = threading.RLock()
def request(
self,
@@ -118,42 +135,116 @@ class _RelayHopClient:
from .relay_bridge import decode_binary_frame, encode_binary_frame, ws_max_size
if self._ws is None:
self._ws = wsc.connect(
self.relay_addr,
open_timeout=self.timeout,
max_size=ws_max_size(),
compression=None,
)
request_id = f"{time.time_ns():x}"
frame = encode_binary_frame({
"request_id": request_id,
"method": "POST",
"path": path,
"headers": headers,
}, body)
self._ws.send(frame)
raw = self._ws.recv(timeout=self.timeout)
if isinstance(raw, (bytes, bytearray)):
resp_header, resp_body = decode_binary_frame(bytes(raw))
status = int(resp_header.get("status", 503))
resp_headers = {k.lower(): v for k, v in (resp_header.get("headers") or {}).items()}
return status, resp_headers, resp_body
resp = json.loads(raw)
status = int(resp.get("status", 503))
resp_headers = {k.lower(): v for k, v in (resp.get("headers") or {}).items()}
body_b64 = resp.get("body_base64")
resp_body = base64.b64decode(body_b64) if body_b64 else (resp.get("body") or "").encode()
return status, resp_headers, resp_body
with self._lock:
if self._ws is None:
self._ws = wsc.connect(
self.relay_addr,
open_timeout=self.timeout,
max_size=ws_max_size(),
compression=None,
)
request_id = headers.get("X-Meshnet-Activation-Id") or uuid.uuid4().hex
frame = encode_binary_frame({
"request_id": request_id,
"method": "POST",
"path": path,
"headers": headers,
}, body)
try:
# A send failure is uncertain too: bytes may already have been
# accepted by the kernel or peer before the exception surfaced.
self._ws.send(frame)
raw = self._ws.recv(timeout=self.timeout)
if isinstance(raw, (bytes, bytearray)):
resp_header, resp_body = decode_binary_frame(bytes(raw))
response_id = str(resp_header.get("request_id") or "")
status = int(resp_header.get("status", 503))
resp_headers = {
k.lower(): v for k, v in (resp_header.get("headers") or {}).items()
}
else:
resp = json.loads(raw)
response_id = str(resp.get("request_id") or "")
status = int(resp.get("status", 503))
resp_headers = {
k.lower(): v for k, v in (resp.get("headers") or {}).items()
}
body_b64 = resp.get("body_base64")
resp_body = (
base64.b64decode(body_b64)
if body_b64 else (resp.get("body") or "").encode()
)
if response_id and response_id != request_id:
raise ValueError("relay response request_id did not match request")
return status, resp_headers, resp_body
except Exception as exc:
self.close()
raise _RelayRequestUncertainError(
"relay connection failed after forwarding request; refusing replay"
) from exc
def close(self) -> None:
if self._ws is not None:
with self._lock:
if self._ws is not None:
try:
self._ws.close()
except Exception:
pass
finally:
self._ws = None
class _DirectHopClient:
"""One serialized HTTP/1.1 connection to one downstream hop.
Generation handlers own these clients, so a cached Route Session reuses a
TCP connection without sharing it between concurrent Route Sessions.
"""
def __init__(self, endpoint: str, timeout: float = 120.0) -> None:
parsed = urllib.parse.urlsplit(endpoint.rstrip("/"))
if parsed.scheme not in {"http", "https"} or not parsed.hostname:
raise ValueError(f"invalid downstream endpoint: {endpoint!r}")
self.timeout = timeout
self._scheme = parsed.scheme
self._host = parsed.hostname
self._port = parsed.port
self._base_path = parsed.path.rstrip("/")
self._connection: http.client.HTTPConnection | None = None
self._lock = threading.RLock()
def _connect(self) -> http.client.HTTPConnection:
connection_type = (
http.client.HTTPSConnection if self._scheme == "https" else http.client.HTTPConnection
)
return connection_type(self._host, self._port, timeout=self.timeout)
def request(
self, path: str, body: bytes, headers: Mapping[str, str],
) -> tuple[int, dict[str, str], bytes]:
request_path = f"{self._base_path}{path if path.startswith('/') else '/' + path}"
with self._lock:
if self._connection is None:
self._connection = self._connect()
try:
self._ws.close()
except Exception:
pass
finally:
self._ws = None
self._connection.request("POST", request_path, body=body, headers=dict(headers))
response = self._connection.getresponse()
response_body = response.read()
response_headers = {key.lower(): value for key, value in response.headers.items()}
return response.status, response_headers, response_body
except Exception as exc:
self.close()
raise _DirectRequestUncertainError(
"direct connection failed after forwarding request; refusing replay"
) from exc
def close(self) -> None:
with self._lock:
if self._connection is not None:
try:
self._connection.close()
finally:
self._connection = None
def _relay_hop(
@@ -178,18 +269,15 @@ def _relay_hop(
client.close()
# Below this, zstd overhead outweighs the win (per-token decode bodies are ~KBs).
_COMPRESS_MIN_BYTES = 64 * 1024
_COMPRESSION_POLICIES = CompressionPolicies()
def _maybe_compress_activation(body: bytes) -> tuple[bytes, str | None]:
"""zstd-compress large activation bodies; returns (wire_body, encoding)."""
if len(body) < _COMPRESS_MIN_BYTES:
return body, None
try:
return _compress_body(body, "zstd"), "zstd"
except Exception:
return body, None
def _maybe_compress_activation(
body: bytes, policy: CompressionPolicy | None = None,
) -> tuple[bytes, str | None]:
"""Compatibility wrapper for callers that only need wire body and encoding."""
result = compress_activation(body, policy or _COMPRESSION_POLICIES.for_condition("lan", "prefill"))
return result.body, result.encoding
def _is_cache_miss_body(body: bytes) -> bool:
@@ -285,6 +373,7 @@ class _TorchHTTPServer(http.server.HTTPServer):
"elapsed_seconds": round(elapsed, 1),
"tokens_per_sec": round(tokens / elapsed, 2) if tokens > 0 else 0.0,
"routing_complete": bool(rec.get("routing_complete")),
"telemetry": rec["telemetry"].snapshot(now=now) if rec.get("telemetry") else None,
})
return out
@@ -306,6 +395,10 @@ class _TorchHTTPServer(http.server.HTTPServer):
class _TorchHandler(http.server.BaseHTTPRequestHandler):
# HTTP/1.1 is required for Route Session-owned downstream connections.
# Finite responses below provide Content-Length; streams are chunked.
protocol_version = "HTTP/1.1"
def log_message(self, fmt, *args): # noqa: suppress request logs in tests
pass
@@ -317,6 +410,9 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
)
def _request_log_suffix(self) -> str:
activation_id = self.headers.get("X-Meshnet-Activation-Id")
if activation_id:
return f" activation_id={activation_id}"
req_id = self.headers.get("X-Meshnet-Request-Id") or self.headers.get("X-Request-Id")
return f" request_id={req_id}" if req_id else ""
@@ -334,6 +430,7 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
"started": time.monotonic(),
"tokens": 0,
"routing_complete": False,
"telemetry": None,
}
def _track_request_progress(
@@ -366,6 +463,7 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
self._handle_chat_completions()
else:
self.send_response(404)
self.send_header("Content-Length", "0")
self.end_headers()
def _handle_infer(self) -> None:
@@ -427,7 +525,7 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
encoding = self.headers.get("X-Meshnet-Encoding")
length = int(self.headers.get("Content-Length", 0))
body = self.rfile.read(length)
raw_body = _decompress_body(body, encoding)
raw_body = decompress_activation(body, encoding).body
_validate_activation_body(raw_body, shape, dtype)
if dtype != "bfloat16":
raise ValueError("real model backend requires bfloat16 activation input")
@@ -438,6 +536,7 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
except (KeyError, ValueError, TypeError):
self.send_response(400)
self.send_header("X-Meshnet-Wire", _WIRE_VERSION)
self.send_header("Content-Length", "0")
self.end_headers()
return
@@ -511,7 +610,12 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
self._send_json(200, {"text": result})
return
response_body = _compress_body(result.body, encoding)
route_condition = self.headers.get("X-Meshnet-Compression-Route", "lan")
phase_condition = cache_mode if cache_mode in {"prefill", "decode"} else "prefill"
response_compression = compress_activation(
result.body, _COMPRESSION_POLICIES.for_condition(route_condition, phase_condition),
)
response_body = response_compression.body
self.send_response(200)
self.send_header("Content-Type", "application/octet-stream")
self.send_header("Content-Length", str(len(response_body)))
@@ -521,8 +625,8 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
self.send_header("X-Meshnet-Session", session)
self.send_header("X-Meshnet-Chunk-Index", chunk_index)
self.send_header("X-Meshnet-Chunk-Total", chunk_total)
if encoding:
self.send_header("X-Meshnet-Encoding", encoding)
if response_compression.encoding:
self.send_header("X-Meshnet-Encoding", response_compression.encoding)
if result.attention_mask_header:
self.send_header("X-Meshnet-Attn-Mask", result.attention_mask_header)
if result.position_ids_header:
@@ -700,6 +804,11 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
current_text = prompt_text
session_id = str(uuid.uuid4())
telemetry = GenerationTelemetry(session_id)
with server._stats_lock:
current = server._active_requests.get(request_id)
if current is not None:
current["telemetry"] = telemetry
use_kv = bool(getattr(backend, "supports_kv_cache", False))
# EOS detection by id must work on the stateless path too: the tail
# returns token_id regardless of caching, and EOS usually decodes to
@@ -722,6 +831,7 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
last_token_id: int | None = None
failure_reason: str | None = None
relay_clients: dict[str, _RelayHopClient] = {}
direct_clients: dict[str, _DirectHopClient] = {}
def _prefill_step() -> tuple[str, int | None]:
"""Full-sequence prefill: initial step and cache-miss recovery."""
@@ -734,6 +844,8 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
payload, remaining_route, backend=backend,
session=session_id, cache_mode="prefill" if use_kv else None,
relay_clients=relay_clients,
direct_clients=direct_clients if use_kv else None,
telemetry=telemetry,
)
for step in range(max_tokens):
@@ -745,6 +857,8 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
payload, remaining_route, backend=backend,
session=session_id, cache_mode="decode",
relay_clients=relay_clients,
direct_clients=direct_clients,
telemetry=telemetry,
)
except (KVCacheMiss, _PipelineCacheMiss) as miss:
# Evicted/restarted node or head lost its own session:
@@ -758,11 +872,11 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
else:
token_str, token_id = _prefill_step()
except _PipelineCacheMiss as exc:
print(f" [node] unexpected cache miss on prefill: {exc}", flush=True)
print(f" [node] unexpected cache miss on prefill session={session_id[:8]}: {exc}", flush=True)
failure_reason = f"cache miss on prefill: {exc}"
break
except Exception as exc:
print(f" [node] distributed encode error: {exc}", flush=True)
print(f" [node] distributed encode error session={session_id[:8]}: {exc}", flush=True)
failure_reason = f"distributed encode error: {exc}"
break
# Stop on error responses or EOS.
@@ -789,14 +903,32 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
tokens=len(generated),
routing_complete=True,
)
telemetry.note_tokens(len(generated))
now = time.monotonic()
if telemetry.report_due:
summary = telemetry.snapshot(now=now)
seams = summary["seams"]
if seams:
latest = seams[-1]
print(
f" [node] seam telemetry session={session_id[:8]} "
f"phase={latest['phase']} hop={latest['hop']} "
f"activations={latest['activations']} "
f"avg_ms={latest['avg_latency_ms']:.2f} "
f"wire_bytes={latest['wire_bytes']} "
f"response_bytes={latest['response_bytes']} "
f"tps={summary['tokens_per_sec']:.2f} "
f"activation_id={latest['last_activation_id']}",
flush=True,
)
telemetry.mark_reported(now=now)
if step == 0 or now - last_gen_log >= _GENERATION_LOG_INTERVAL:
elapsed = now - gen_started
token_count = len(generated)
tps = token_count / max(elapsed, 1e-6)
_write_progress_line(
progress_line,
f" [node] generating step={step + 1}/{max_tokens} "
f" [node] generating step={step + 1}/{max_tokens} session={session_id[:8]} "
f"tokens={token_count} elapsed_s={elapsed:.1f} tps={tps:.2f}",
)
last_gen_log = now
@@ -808,6 +940,8 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
pass
for relay_client in relay_clients.values():
relay_client.close()
for direct_client in direct_clients.values():
direct_client.close()
if generated:
elapsed = time.monotonic() - gen_started
@@ -815,10 +949,11 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
tps = token_count / max(elapsed, 1e-6)
_write_progress_line(
progress_line,
f" [node] generation complete tokens={token_count} "
f" [node] generation complete session={session_id[:8]} tokens={token_count} "
f"elapsed_s={elapsed:.1f} tps={tps:.2f}",
final=True,
)
telemetry.close()
result_text = "".join(generated)
# A failure before the first token is an upstream error, not an empty
@@ -921,6 +1056,8 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
session: str | None = None,
cache_mode: str | None = None,
relay_clients: dict[str, _RelayHopClient] | None = None,
direct_clients: dict[str, _DirectHopClient] | None = None,
telemetry: GenerationTelemetry | None = None,
) -> tuple[str, int | None]:
"""Forward an activation through the downstream route.
@@ -935,10 +1072,9 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
# Full single-node (head+tail) is handled before entering this method.
if active_backend.is_tail:
try:
tensor = active_backend.torch.frombuffer(
bytearray(payload.body), # type: ignore[union-attr]
dtype=active_backend.torch.bfloat16,
).reshape(payload.shape).to(active_backend.device) # type: ignore[union-attr]
tensor = _tensor_from_bfloat16_bytes(
payload.body, payload.shape, active_backend.torch, # type: ignore[union-attr]
).to(active_backend.device)
if hasattr(active_backend, "decode_tail_token"):
tail = active_backend.decode_tail_token(tensor)
return tail.text, tail.token_id
@@ -968,7 +1104,12 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
+ (f" relay={relay_addr}" if relay_addr else ""),
flush=True,
)
wire_body, wire_encoding = _maybe_compress_activation(current_body)
phase = cache_mode if cache_mode in {"prefill", "decode"} else "prefill"
route_condition = "relay" if relay_addr else "lan"
compression = compress_activation(
current_body, _COMPRESSION_POLICIES.for_condition(route_condition, phase),
)
wire_body, wire_encoding = compression.body, compression.encoding
headers: dict[str, str] = {
"Content-Type": "application/octet-stream",
"X-Meshnet-Wire": _WIRE_VERSION,
@@ -979,9 +1120,17 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
"X-Meshnet-Chunk-Total": "1",
"X-Meshnet-Hop-Index": str(hop_index),
"X-Meshnet-Start-Layer": str(start_layer),
"X-Meshnet-Activation-Id": uuid.uuid4().hex,
"X-Meshnet-Compression-Route": route_condition,
}
if wire_encoding:
headers["X-Meshnet-Encoding"] = wire_encoding
if telemetry is not None:
telemetry.record_compression(
phase=phase, hop=hop_index, node=node_url,
input_bytes=compression.input_bytes, output_bytes=compression.output_bytes,
elapsed_seconds=compression.elapsed_seconds,
)
if cache_mode:
headers["X-Meshnet-Cache"] = cache_mode
past_len = getattr(payload, "past_len", None)
@@ -992,6 +1141,10 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
if current_pos:
headers["X-Meshnet-Position-Ids"] = current_pos
if relay_addr:
connection_reused = bool(
relay_clients is not None and relay_addr in relay_clients
)
seam_started = time.monotonic()
try:
if relay_clients is None:
status, resp_headers, resp_body = _relay_hop(
@@ -1004,44 +1157,100 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
status, resp_headers, resp_body = relay_client.request(
"/forward", wire_body, headers,
)
if telemetry is not None:
telemetry.record_seam(
activation_id=headers["X-Meshnet-Activation-Id"],
phase=cache_mode or "prefill",
hop=hop_index,
node=node_url,
latency_seconds=time.monotonic() - seam_started,
wire_bytes=len(wire_body),
response_bytes=len(resp_body),
connection_reused=connection_reused,
)
if status == 409 and _is_cache_miss_body(resp_body):
raise _PipelineCacheMiss(node_url)
if status >= 400:
detail = _response_error_snippet(resp_body)
print(
f" [node] relay hop {hop_index} returned {status} from {relay_addr}: {detail}",
f" [node] relay hop {hop_index} session={session[:8]} returned "
f"{status} from {relay_addr}: {detail}",
flush=True,
)
return f"pipeline error at {node_url} via relay: status {status}: {detail}", None
except _PipelineCacheMiss:
raise
except _RelayRequestUncertainError as exc:
# The activation may already have mutated the downstream
# KV cache. Do not replay it on a direct connection.
print(
f" [node] relay hop {hop_index} session={session[:8]} outcome is uncertain at "
f"{relay_addr}: {exc}",
flush=True,
)
return f"pipeline relay outcome uncertain at {node_url}: {exc}", None
except Exception as exc:
print(
f" [node] relay hop {hop_index} failed at {relay_addr}: {exc}; "
f" [node] relay hop {hop_index} session={session[:8]} failed at {relay_addr}: {exc}; "
f"falling back to direct {node_url}",
flush=True,
)
relay_addr = None # fall through to direct
if not relay_addr:
req = urllib.request.Request(
f"{node_url}/forward",
data=wire_body,
headers=headers,
method="POST",
connection_reused = bool(
direct_clients is not None and node_url in direct_clients
)
seam_started = time.monotonic()
try:
with urllib.request.urlopen(req, timeout=120.0) as r:
resp_body = r.read()
resp_headers = {k.lower(): v for k, v in r.headers.items()}
except urllib.error.HTTPError as exc:
body = exc.read()
if exc.code == 409 and _is_cache_miss_body(body):
raise _PipelineCacheMiss(node_url) from exc
detail = _response_error_snippet(body)
print(f" [node] pipeline hop {hop_index} failed at {node_url}: {exc}: {detail}", flush=True)
return f"pipeline error at {node_url}: {exc}: {detail}", None
if direct_clients is None:
direct_client = _DirectHopClient(node_url, timeout=120.0)
try:
status, resp_headers, resp_body = direct_client.request(
"/forward", wire_body, headers,
)
finally:
direct_client.close()
else:
direct_client = direct_clients.setdefault(
node_url, _DirectHopClient(node_url, timeout=120.0),
)
status, resp_headers, resp_body = direct_client.request(
"/forward", wire_body, headers,
)
if telemetry is not None:
telemetry.record_seam(
activation_id=headers["X-Meshnet-Activation-Id"],
phase=cache_mode or "prefill",
hop=hop_index,
node=node_url,
latency_seconds=time.monotonic() - seam_started,
wire_bytes=len(wire_body),
response_bytes=len(resp_body),
connection_reused=connection_reused,
)
if status == 409 and _is_cache_miss_body(resp_body):
raise _PipelineCacheMiss(node_url)
if status >= 400:
detail = _response_error_snippet(resp_body)
print(
f" [node] pipeline hop {hop_index} session={session[:8]} failed at {node_url}: "
f"status {status}: {detail}", flush=True,
)
return f"pipeline error at {node_url}: status {status}: {detail}", None
except _PipelineCacheMiss:
raise
except _DirectRequestUncertainError as exc:
print(
f" [node] pipeline hop {hop_index} session={session[:8]} outcome is uncertain "
f"at {node_url}: {exc}",
flush=True,
)
return f"pipeline direct outcome uncertain at {node_url}: {exc}", None
except Exception as exc:
print(f" [node] pipeline hop {hop_index} failed at {node_url}: {exc}", flush=True)
print(
f" [node] pipeline hop {hop_index} session={session[:8]} "
f"failed at {node_url}: {exc}", flush=True,
)
return f"pipeline error at {node_url}: {exc}", None
content_type = resp_headers.get("content-type", "")
if "application/json" in content_type:
@@ -1058,11 +1267,21 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
shape_header = resp_headers.get("x-meshnet-shape", ",".join(str(d) for d in current_shape))
current_shape = _parse_shape(shape_header)
try:
current_body = _decompress_body(
decompression = decompress_activation(
resp_body, resp_headers.get("x-meshnet-encoding")
)
current_body = decompression.body
if telemetry is not None:
telemetry.record_compression(
phase=phase, hop=hop_index, node=node_url,
input_bytes=decompression.input_bytes, output_bytes=decompression.output_bytes,
elapsed_seconds=decompression.elapsed_seconds, decompression=True,
)
except ValueError as exc:
print(f" [node] pipeline hop {hop_index} bad response encoding: {exc}", flush=True)
print(
f" [node] pipeline hop {hop_index} session={session[:8]} "
f"bad response encoding: {exc}", flush=True,
)
return f"pipeline error at {node_url}: {exc}", None
current_attn = resp_headers.get("x-meshnet-attn-mask")
current_pos = resp_headers.get("x-meshnet-position-ids")
@@ -1084,14 +1303,17 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
self.send_response(200)
self.send_header("Content-Type", "text/event-stream; charset=utf-8")
self.send_header("Cache-Control", "no-cache")
self.send_header("Transfer-Encoding", "chunked")
self.end_headers()
def _emit(data: str) -> None:
def _emit(data: str) -> bool:
try:
self.wfile.write(f"data: {data}\n\n".encode())
body = f"data: {data}\n\n".encode()
self.wfile.write(f"{len(body):X}\r\n".encode() + body + b"\r\n")
self.wfile.flush()
return True
except (BrokenPipeError, ConnectionResetError):
pass
return False
_emit(json.dumps({
"id": chunk_id, "object": "chat.completion.chunk", "created": created,
@@ -1105,7 +1327,7 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
# instead of showing an empty completion.
_emit(json.dumps({"error": {"message": error, "type": "upstream_error"}}))
try:
self.wfile.write(b"data: [DONE]\n\n")
self.wfile.write(b"E\r\ndata: [DONE]\n\n\r\n0\r\n\r\n")
self.wfile.flush()
except (BrokenPipeError, ConnectionResetError):
pass
@@ -1117,7 +1339,7 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
"choices": [{"index": 0, "delta": {}, "finish_reason": "stop"}],
}))
try:
self.wfile.write(b"data: [DONE]\n\n")
self.wfile.write(b"E\r\ndata: [DONE]\n\n\r\n0\r\n\r\n")
self.wfile.flush()
except (BrokenPipeError, ConnectionResetError):
pass
@@ -1159,14 +1381,17 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
self.send_response(200)
self.send_header("Content-Type", "text/event-stream; charset=utf-8")
self.send_header("Cache-Control", "no-cache")
self.send_header("Transfer-Encoding", "chunked")
self.end_headers()
def _emit(data: str) -> None:
def _emit(data: str) -> bool:
try:
self.wfile.write(f"data: {data}\n\n".encode())
body = f"data: {data}\n\n".encode()
self.wfile.write(f"{len(body):X}\r\n".encode() + body + b"\r\n")
self.wfile.flush()
except BrokenPipeError:
pass
return True
except (BrokenPipeError, ConnectionResetError):
return False
_emit(json.dumps({
"id": chunk_id, "object": "chat.completion.chunk", "created": created,
@@ -1184,9 +1409,9 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
"choices": [{"index": 0, "delta": {}, "finish_reason": "stop"}],
}))
try:
self.wfile.write(b"data: [DONE]\n\n")
self.wfile.write(b"E\r\ndata: [DONE]\n\n\r\n0\r\n\r\n")
self.wfile.flush()
except BrokenPipeError:
except (BrokenPipeError, ConnectionResetError):
pass
@@ -1255,6 +1480,7 @@ class TorchNodeServer:
debug: bool = False,
max_loaded_shards: int = 1,
force_cpu: bool = False,
recipe_params: Mapping[str, Any] | None = None,
) -> None:
self._host = host
self._requested_port = port
@@ -1266,6 +1492,7 @@ class TorchNodeServer:
quantization,
cache_dir,
force_cpu=force_cpu,
recipe_params=recipe_params,
)
self._backends: dict[str, TorchModelShard] = {self._backend.model_id: self._backend}
# Auto-detect tracker mode: enabled when shard_start == 0 or explicitly set
@@ -1415,13 +1642,20 @@ def _load_backend(
quantization: str,
cache_dir: Path | None = None,
force_cpu: bool = False,
recipe_params: Mapping[str, Any] | None = None,
) -> TorchModelShard:
from .model_backend import load_torch_shard
quant = validate_quantization(quantization)
try:
return load_torch_shard(
model_id, shard_start, shard_end, quant, cache_dir, force_cpu=force_cpu
model_id,
shard_start,
shard_end,
quant,
cache_dir,
force_cpu=force_cpu,
recipe_params=recipe_params,
)
except MissingModelDependencyError:
raise

81
packages/relay/meshnet_relay/server.py
View File

@@ -46,11 +46,12 @@ def ws_max_size() -> int | None:
# inflation, no JSON re-encode of megabytes per hop. Text JSON frames remain the
# control plane (gossip, peer-register, streamed SSE responses).
BINARY_FRAME_MAGIC = b"MRF1"
_RPC_PEER_DISCONNECTED = object()
def encode_binary_frame(header: dict, body: bytes) -> bytes:
header_bytes = json.dumps(header, separators=(",", ":")).encode()
return BINARY_FRAME_MAGIC + len(header_bytes).to_bytes(4, "big") + header_bytes + body
return b"".join((BINARY_FRAME_MAGIC, len(header_bytes).to_bytes(4, "big"), header_bytes, body))
def decode_binary_frame(frame: bytes) -> tuple[dict, bytes]:
@@ -58,7 +59,7 @@ def decode_binary_frame(frame: bytes) -> tuple[dict, bytes]:
raise ValueError("not a meshnet binary relay frame")
header_len = int.from_bytes(frame[4:8], "big")
header = json.loads(frame[8:8 + header_len].decode())
return header, bytes(frame[8 + header_len:])
return header, frame[8 + header_len:]
class RelayServer:
@@ -79,12 +80,16 @@ class RelayServer:
ssl_cert: Path | None = None,
ssl_key: Path | None = None,
max_peers: int = 500,
rpc_timeout: float = 310.0,
rpc_idle_timeout: float = 120.0,
):
self.host = host
self.port = port
self.ssl_cert = ssl_cert
self.ssl_key = ssl_key
self.max_peers = max_peers
self.rpc_timeout = rpc_timeout
self.rpc_idle_timeout = rpc_idle_timeout
self._registry = PeerRegistry()
self._loop: asyncio.AbstractEventLoop | None = None
@@ -94,9 +99,10 @@ class RelayServer:
self._ready = threading.Event()
self._running = False
self._stop_event: asyncio.Event | None = None
# request_id → queue of relay-http-response frames (US-036: a streamed
# response is a sequence of frames; a frame without "stream" is terminal).
self._pending_rpc: dict[str, asyncio.Queue] = {}
# relay request id → (target peer, requester request id, response queue).
# The relay-generated id prevents two Route Sessions using the same
# legacy request_id from overwriting each other's pending response.
self._pending_rpc: dict[str, tuple[str, str, asyncio.Queue]] = {}
@property
def registry(self) -> PeerRegistry:
@@ -118,6 +124,12 @@ class RelayServer:
if self._thread:
self._thread.join(timeout=3.0)
def _fail_pending_for_peer(self, peer_id: str) -> None:
"""Wake requesters immediately when their selected bridge disconnects."""
for target, _, queue in tuple(self._pending_rpc.values()):
if target == peer_id:
queue.put_nowait(_RPC_PEER_DISCONNECTED)
def _run(self) -> None:
asyncio.set_event_loop(self._loop)
self._loop.run_until_complete(self._serve())
@@ -192,9 +204,9 @@ class RelayServer:
header, _ = decode_binary_frame(bytes(raw))
except (ValueError, json.JSONDecodeError):
continue
queue = self._pending_rpc.get(header.get("request_id"))
if queue is not None:
queue.put_nowait(bytes(raw))
pending = self._pending_rpc.get(header.get("request_id"))
if pending is not None:
pending[2].put_nowait(bytes(raw))
continue
try:
envelope = json.loads(raw)
@@ -226,9 +238,9 @@ class RelayServer:
if topic == "relay-http-response":
payload = envelope.get("payload", {})
request_id = payload.get("request_id")
queue = self._pending_rpc.get(request_id)
if queue is not None:
queue.put_nowait(payload)
pending = self._pending_rpc.get(request_id)
if pending is not None:
pending[2].put_nowait(payload)
continue
# Fan out to all other registered peers
@@ -242,6 +254,9 @@ class RelayServer:
finally:
if peer_id:
self._registry.unregister(peer_id)
# Do not leave a requester waiting for its full timeout after
# the selected bridge goes away.
self._fail_pending_for_peer(peer_id)
log.debug("Peer unregistered: %s", peer_id)
async def _handle_circuit_relay(self, ws_requester, target_peer_id: str) -> None:
@@ -290,18 +305,19 @@ class RelayServer:
except Exception:
return
request_id: str | None = None
requester_request_id: str | None = None
relay_request_id = uuid.uuid4().hex
try:
if isinstance(raw, (bytes, bytearray)):
header, body = decode_binary_frame(bytes(raw))
request_id = str(header.get("request_id") or uuid.uuid4())
header["request_id"] = request_id
requester_request_id = str(header.get("request_id") or uuid.uuid4())
header["request_id"] = relay_request_id
header["target_peer"] = target_peer_id
outbound: str | bytes = encode_binary_frame(header, body)
else:
payload = json.loads(raw)
request_id = str(payload.get("request_id") or uuid.uuid4())
payload["request_id"] = request_id
requester_request_id = str(payload.get("request_id") or uuid.uuid4())
payload["request_id"] = relay_request_id
payload["target_peer"] = target_peer_id
outbound = json.dumps({
"topic": "relay-http-request",
@@ -314,16 +330,18 @@ class RelayServer:
return
queue: asyncio.Queue = asyncio.Queue()
self._pending_rpc[request_id] = queue
overall_timeout = 310.0
idle_timeout = 120.0
self._pending_rpc[relay_request_id] = (
target_peer_id, requester_request_id, queue,
)
overall_timeout = self.rpc_timeout
idle_timeout = self.rpc_idle_timeout
loop = asyncio.get_running_loop()
deadline = loop.time() + overall_timeout
target = self._registry.get(target_peer_id)
try:
if target is None:
await ws_requester.send(json.dumps({
"request_id": request_id,
"request_id": requester_request_id,
"status": 503,
"headers": {"Content-Type": "application/json"},
"body": json.dumps({"error": f"peer {target_peer_id!r} disconnected"}),
@@ -339,21 +357,34 @@ class RelayServer:
frame = await asyncio.wait_for(
queue.get(), timeout=min(idle_timeout, remaining)
)
if frame is _RPC_PEER_DISCONNECTED:
raise ConnectionError(f"peer {target_peer_id!r} disconnected")
if isinstance(frame, (bytes, bytearray)):
await ws_requester.send(frame)
header, body = decode_binary_frame(bytes(frame))
header["request_id"] = requester_request_id
await ws_requester.send(encode_binary_frame(header, body))
break
await ws_requester.send(json.dumps(frame))
if not frame.get("stream") or frame.get("done"):
response = dict(frame)
response["request_id"] = requester_request_id
await ws_requester.send(json.dumps(response))
if not response.get("stream") or response.get("done"):
break
except asyncio.TimeoutError:
await ws_requester.send(json.dumps({
"request_id": request_id,
"request_id": requester_request_id,
"status": 504,
"headers": {"Content-Type": "application/json"},
"body": json.dumps({"error": "relay rpc timed out"}),
}))
except ConnectionError as exc:
await ws_requester.send(json.dumps({
"request_id": requester_request_id,
"status": 503,
"headers": {"Content-Type": "application/json"},
"body": json.dumps({"error": str(exc)}),
}))
finally:
self._pending_rpc.pop(request_id, None)
self._pending_rpc.pop(relay_request_id, None)
async def _broadcast(raw: str | bytes, peers: list) -> None:

415
packages/tracker/meshnet_tracker/capability.py Normal file
View File

@@ -0,0 +1,415 @@
"""Tracker-side validation of the capability report a Node presents at registration.
A Node proves locally that it can execute one exact combination — model artifact,
shard range, recipe, backend/device — and ships that proof with its registration
(ADR-0023, NCA-001/002/003). The tracker does not re-run the forward; it decides
whether the presented proof *covers what the node is advertising*, records the
verdict as a small sanitized enum, and routes only to nodes whose verdict is
`admitted`.
Two properties this module deliberately keeps:
* **No model knowledge.** Model ids, recipe ids, backend ids and device names are
opaque labels. They are compared, never interpreted; no vendor string is a
code path here.
* **Evidence, not assertion.** A report is treated as a claim about identity, and
the tracker only ever *narrows* what a node may serve with it. Nothing in a
report can widen a node's eligibility or its routing weight — throughput
routing stays measurement-driven (ADR-0013/0021).
Older nodes that predate the capability protocol present no report at all. They
are handled by an explicit policy (`POLICY_COMPAT` vs `POLICY_ENFORCE`), never by
silently treating "no proof" as "proven" — see `docs/adr/0023-…` for the rollout.
"""
from __future__ import annotations
import os
import re
import time
from dataclasses import dataclass, replace
from typing import Any, Callable, Mapping
# The capability report layout this tracker reads (meshnet_node.capability).
SUPPORTED_SCHEMA_VERSION = 1
# The oldest recipe catalogue whose recipe semantics this tracker still trusts.
# A node carrying an older catalogue may be running a recipe whose id has since
# been redefined, so its proof cannot be matched to a name reliably.
MIN_CATALOGUE_VERSION = "2026.07.1"
# How old a proof may be *at the moment it is presented*. Freshness after that is
# carried by liveness: a registration is re-asserted on tracker restart and the
# node is purged once heartbeats stop.
DEFAULT_MAX_REPORT_AGE_SECONDS = 900.0
# A proof timestamped further ahead than this is not fresh, it is wrong.
MAX_CLOCK_SKEW_SECONDS = 60.0
STATUS_PASSED = "passed"
# --- Admission verdicts. `admitted` is the only routable one under `enforce`. ---
STATE_ADMITTED = "admitted"
STATE_ABSENT = "absent"
STATE_INVALID = "invalid"
STATE_FAILED = "failed"
STATE_STALE = "stale"
STATE_MODEL_MISMATCH = "model-mismatch"
STATE_SHARD_MISMATCH = "shard-mismatch"
STATE_RECIPE_MISMATCH = "recipe-mismatch"
STATE_CATALOGUE_INCOMPATIBLE = "catalogue-incompatible"
ALL_STATES = (
STATE_ADMITTED,
STATE_ABSENT,
STATE_INVALID,
STATE_FAILED,
STATE_STALE,
STATE_MODEL_MISMATCH,
STATE_SHARD_MISMATCH,
STATE_RECIPE_MISMATCH,
STATE_CATALOGUE_INCOMPATIBLE,
)
# --- Compatibility policy for nodes that predate the capability protocol. ---
# `compat` — a node presenting *no* proof still routes (legacy behaviour), but a
# node presenting a *bad* proof never does. Presenting a broken or
# mismatched proof is a stronger signal than presenting none.
# `enforce` — only `admitted` routes. Absent proof is not routable.
POLICY_COMPAT = "compat"
POLICY_ENFORCE = "enforce"
ALL_POLICIES = (POLICY_COMPAT, POLICY_ENFORCE)
DEFAULT_POLICY = POLICY_COMPAT
POLICY_ENV_VAR = "MESHNET_TRACKER_CAPABILITY_POLICY"
# Operator-facing detail strings are short and never carry a raw exception.
_MAX_DETAIL_CHARS = 240
_MAX_DIAGNOSTICS = 3
_CREDENTIAL_PATTERNS = (
re.compile(r"\b[A-Za-z0-9_]{2,6}_[A-Za-z0-9]{16,}\b"), # hf_…, ghp_…, sk_live_…
re.compile(r"\bsk-[A-Za-z0-9_-]{16,}\b"),
re.compile(r"(?i)\bbearer\s+\S+"),
re.compile(r"(?i)\b(?:token|api[_-]?key|password|secret)\s*[=:]\s*\S+"),
)
_REDACTED = "[redacted]"
def normalize_policy(value: Any) -> str:
"""Return a known policy name, falling back to the default for anything else."""
if isinstance(value, str) and value.strip().lower() in ALL_POLICIES:
return value.strip().lower()
return DEFAULT_POLICY
def policy_from_env(environ: Mapping[str, str] | None = None) -> str:
env = os.environ if environ is None else environ
return normalize_policy(env.get(POLICY_ENV_VAR))
def sanitize_detail(text: Any) -> str:
"""Collapse, redact and clip a string bound for an operator view."""
cleaned = " ".join(str(text).split())
for pattern in _CREDENTIAL_PATTERNS:
cleaned = pattern.sub(_REDACTED, cleaned)
if len(cleaned) > _MAX_DETAIL_CHARS:
cleaned = cleaned[: _MAX_DETAIL_CHARS - 1].rstrip() + "…"
return cleaned
def catalogue_is_compatible(version: Any) -> bool:
"""True when `version` is at least `MIN_CATALOGUE_VERSION`.
Versions are dotted integer sequences (`2026.07.1`). Anything that does not
parse is incompatible — an unparseable catalogue version cannot be shown to
be new enough.
"""
parsed = _parse_version(version)
if parsed is None:
return False
return parsed >= _parse_version(MIN_CATALOGUE_VERSION) # type: ignore[operator]
def _parse_version(value: Any) -> tuple[int, ...] | None:
if not isinstance(value, str) or not value.strip():
return None
parts = value.strip().split(".")
try:
return tuple(int(part) for part in parts)
except ValueError:
return None
@dataclass(frozen=True)
class CapabilityState:
"""The tracker's sanitized verdict on one node's presented proof.
This is what the network map exposes and what route selection consults. It
holds identity labels and a verdict — never a raw exception, a file path, or
a credential.
"""
state: str
detail: str = ""
model_id: str | None = None
shard_start: int | None = None
shard_end: int | None = None
recipe_id: str | None = None
recipe_version: str | None = None
catalogue_version: str | None = None
backend_id: str | None = None
device: str | None = None
quantization: str | None = None
validated_at: float | None = None
recorded_at: float = 0.0
schema_version: int | None = None
diagnostics: tuple[str, ...] = ()
@property
def proven(self) -> bool:
"""The presented proof covers exactly what the node advertised."""
return self.state == STATE_ADMITTED
def routable_under(self, policy: str) -> bool:
if self.proven:
return True
return self.state == STATE_ABSENT and normalize_policy(policy) == POLICY_COMPAT
def with_state(self, state: str, detail: str) -> CapabilityState:
"""Re-verdict a recorded proof against what the node advertises *now*."""
return replace(self, state=state, detail=sanitize_detail(detail))
def to_dict(self) -> dict:
return {
"state": self.state,
"detail": self.detail,
"model_id": self.model_id,
"shard_start": self.shard_start,
"shard_end": self.shard_end,
"recipe_id": self.recipe_id,
"recipe_version": self.recipe_version,
"catalogue_version": self.catalogue_version,
"backend_id": self.backend_id,
"device": self.device,
"quantization": self.quantization,
"validated_at": self.validated_at,
"recorded_at": self.recorded_at,
"schema_version": self.schema_version,
"diagnostics": list(self.diagnostics),
}
def absent_state(detail: str = "", *, now: float | None = None) -> CapabilityState:
"""The verdict for a node that presented no proof at all (legacy node)."""
return CapabilityState(
state=STATE_ABSENT,
detail=sanitize_detail(
detail
or "node registered without a capability report; it predates the "
"capability protocol or ran with admission disabled"
),
recorded_at=time.time() if now is None else now,
)
def evaluate_report(
report: Any,
*,
model_matches: Callable[[str], bool],
advertised_model: str | None,
shard_start: int | None,
shard_end: int | None,
declared_recipe_id: str | None = None,
declared_recipe_version: str | None = None,
now: float | None = None,
max_age_seconds: float = DEFAULT_MAX_REPORT_AGE_SECONDS,
) -> CapabilityState:
"""Judge the proof a node presented against what that node is advertising.
`model_matches` is the tracker's own alias-aware comparison against the
node's registered model / hf_repo, so an opaque model id never has to be
parsed here.
Returns a verdict for *every* input, including malformed ones: a bad proof is
recorded and shown to the operator rather than dropped, so "why is my node not
routing" has an answer in the network map.
"""
now = time.time() if now is None else now
if report is None:
return absent_state(now=now)
if not isinstance(report, Mapping):
return CapabilityState(
state=STATE_INVALID,
detail=sanitize_detail(
f"capability_report must be a JSON object, got "
f"{type(report).__name__}"
),
recorded_at=now,
)
try:
parsed = _parse_report(report)
except _ReportError as exc:
return CapabilityState(
state=STATE_INVALID,
detail=sanitize_detail(str(exc)),
recorded_at=now,
schema_version=_maybe_int(report.get("schema_version")),
)
status = parsed.pop("_status")
base = CapabilityState(state=STATE_ADMITTED, recorded_at=now, **parsed)
if base.schema_version != SUPPORTED_SCHEMA_VERSION:
return base.with_state(
STATE_INVALID,
f"capability report declares schema version {base.schema_version}; "
f"this tracker reads version {SUPPORTED_SCHEMA_VERSION}",
)
if not catalogue_is_compatible(base.catalogue_version):
return base.with_state(
STATE_CATALOGUE_INCOMPATIBLE,
f"recipe catalogue {base.catalogue_version!r} is older than the "
f"minimum this tracker trusts ({MIN_CATALOGUE_VERSION}); upgrade the node",
)
if not model_matches(base.model_id or ""):
return base.with_state(
STATE_MODEL_MISMATCH,
f"proof is for model {base.model_id!r}, but the node registered "
f"{advertised_model!r}",
)
if shard_start is not None and shard_end is not None:
if (base.shard_start, base.shard_end) != (shard_start, shard_end):
return base.with_state(
STATE_SHARD_MISMATCH,
f"proof is for layers {base.shard_start}–{base.shard_end}, but the "
f"node registered layers {shard_start}–{shard_end}",
)
if declared_recipe_id is not None and base.recipe_id != declared_recipe_id:
return base.with_state(
STATE_RECIPE_MISMATCH,
f"proof is for recipe {base.recipe_id!r}, but the node declared it "
f"serves with {declared_recipe_id!r}",
)
if (
declared_recipe_version is not None
and base.recipe_version != declared_recipe_version
):
return base.with_state(
STATE_RECIPE_MISMATCH,
f"proof is for recipe {base.recipe_id!r} v{base.recipe_version}, but "
f"the node declared v{declared_recipe_version}",
)
if status != STATUS_PASSED:
return base.with_state(
STATE_FAILED,
f"capability validation {status} on the node"
+ (f" — {' '.join(base.diagnostics)}" if base.diagnostics else ""),
)
age = now - (base.validated_at or 0.0)
if age > max_age_seconds:
return base.with_state(
STATE_STALE,
f"proof is {age / 60:.0f} min old (limit {max_age_seconds / 60:.0f} min); "
"the node must re-validate before it can be routed",
)
if age < -MAX_CLOCK_SKEW_SECONDS:
return base.with_state(
STATE_STALE,
f"proof is timestamped {-age:.0f}s in the future; check the node's clock",
)
return base.with_state(
STATE_ADMITTED,
f"{base.model_id} layers {base.shard_start}–{base.shard_end} proven on "
f"{base.device} with recipe {base.recipe_id} (v{base.recipe_version})",
)
class _ReportError(ValueError):
"""Malformed report input. Messages name the field, never echo a payload."""
def _parse_report(doc: Mapping[str, Any]) -> dict:
model = _object(doc.get("model"), "model")
shard = _object(doc.get("shard"), "shard")
recipe = _object(doc.get("recipe"), "recipe")
backend = _object(doc.get("backend"), "backend")
validated_at = doc.get("validated_at")
if isinstance(validated_at, bool) or not isinstance(validated_at, (int, float)):
raise _ReportError("'validated_at' must be a Unix timestamp")
schema_version = doc.get("schema_version")
if isinstance(schema_version, bool) or not isinstance(schema_version, int):
raise _ReportError("'schema_version' must be an integer")
return {
"model_id": _text(model.get("model_id"), "model.model_id"),
"shard_start": _index(shard.get("start"), "shard.start"),
"shard_end": _index(shard.get("end"), "shard.end"),
"recipe_id": _text(recipe.get("recipe_id"), "recipe.recipe_id"),
"recipe_version": _text(recipe.get("recipe_version"), "recipe.recipe_version"),
"catalogue_version": _text(
recipe.get("catalogue_version"), "recipe.catalogue_version"
),
"backend_id": _text(backend.get("backend_id"), "backend.backend_id"),
"device": _text(backend.get("device"), "backend.device"),
"quantization": _optional_text(
backend.get("quantization"), "backend.quantization"
),
"validated_at": float(validated_at),
"schema_version": schema_version,
"diagnostics": _diagnostics(doc.get("diagnostics")),
"_status": _text(doc.get("status"), "status"),
}
def _object(value: Any, field_name: str) -> Mapping[str, Any]:
if not isinstance(value, Mapping):
raise _ReportError(f"{field_name!r} must be a JSON object")
return value
def _text(value: Any, field_name: str) -> str:
if not isinstance(value, str) or not value.strip():
raise _ReportError(f"{field_name!r} must be a non-empty string")
return value
def _optional_text(value: Any, field_name: str) -> str | None:
if value is None:
return None
return _text(value, field_name)
def _index(value: Any, field_name: str) -> int:
if isinstance(value, bool) or not isinstance(value, int) or value < 0:
raise _ReportError(f"{field_name!r} must be a non-negative integer")
return value
def _maybe_int(value: Any) -> int | None:
if isinstance(value, bool) or not isinstance(value, int):
return None
return value
def _diagnostics(value: Any) -> tuple[str, ...]:
if not isinstance(value, list):
return ()
out = [
sanitize_detail(item)
for item in value[:_MAX_DIAGNOSTICS]
if isinstance(item, str) and item.strip()
]
return tuple(out)

14
packages/tracker/meshnet_tracker/cli.py
View File

@@ -9,6 +9,7 @@ from pathlib import Path
from .accounts import DEFAULT_ACCOUNTS_DB_PATH
from .billing import DEFAULT_BILLING_DB_PATH
from .capability import ALL_POLICIES as ALL_CAPABILITY_POLICIES
from .hf_pricing import DEFAULT_HF_PRICING_LOG_DB_PATH
from .logging_setup import (
DEFAULT_LOG_BACKUP_COUNT,
@@ -105,6 +106,18 @@ def main() -> None:
metavar="PATH",
help="SQLite database path for persistent model usage statistics",
)
common.add_argument(
"--capability-policy",
choices=list(ALL_CAPABILITY_POLICIES),
default=None,
help=(
"How to treat nodes that present no capability proof (ADR-0023): "
"'compat' (default) still routes pre-capability nodes; 'enforce' routes "
"only nodes whose proof covers the model and shard they advertise. "
"A broken or mismatched proof is never routed under either policy. "
"Falls back to $MESHNET_TRACKER_CAPABILITY_POLICY when omitted."
),
)
common.add_argument(
"--relay-url",
default=None,
@@ -416,6 +429,7 @@ def main() -> None:
cluster_peers=cluster_peers or None,
cluster_self_url=args.self_url,
stats_db=getattr(args, "stats_db", None),
capability_policy=getattr(args, "capability_policy", None),
relay_url=relay_url,
embedded_relay=args.embedded_relay,
embedded_relay_host=args.relay_host,

53
packages/tracker/meshnet_tracker/dashboard.html
View File

@@ -264,6 +264,8 @@
<div class="chat-toolbar">
<label>Model
<select id="chat-model" onchange="selectChatModel(this.value)"></select>
<select id="chat-quantization" onchange="selectChatQuantization(this.value)"></select>
<button type="button" id="vote-coverage" onclick="voteForCoverage()" style="display:none">Vote for coverage</button>
</label>
<button type="button" id="request-model-load" style="display:none" onclick="requestSelectedModelLoad()">Load on available node</button>
<div id="chat-status" class="chat-status">select a model to start</div>
@@ -1268,6 +1270,7 @@ let chatBusy = false;
let chatSessions = [];
let activeChatSessionId = "";
let selectedChatModel = localStorage.getItem("meshnet_chat_model") || "";
let selectedChatQuantization = localStorage.getItem("meshnet_chat_quantization") || "";
const CHAT_SESSIONS_KEY = "meshnet_chat_sessions_v1";
const CHAT_ACTIVE_SESSION_KEY = "meshnet_chat_active_session_v1";
const CHAT_SESSIONS_LIMIT = 50;
@@ -1628,6 +1631,36 @@ function renderChatModels(force) {
return `<option value="${esc(model.id)}"${model.id === selectedChatModel ? " selected" : ""}>${esc(label)}</option>`;
}).join("");
select.value = selectedChatModel;
renderChatQuantizations();
}
function selectedChatModelEntry() {
return availableModels.find(model => model.id === selectedChatModel) || null;
}
function renderChatQuantizations() {
const select = $("chat-quantization");
const vote = $("vote-coverage");
const model = selectedChatModelEntry();
if (!select || !vote) return;
const variants = (model && model.quantizations) || [];
const selectable = variants.filter(variant => variant.selectable);
if (!selectable.length) {
select.innerHTML = '<option value="">no complete coverage</option>';
select.disabled = true;
vote.style.display = model ? "" : "none";
return;
}
const preferred = selectable.find(variant => variant.id === selectedChatQuantization)
|| selectable[0];
selectedChatQuantization = preferred.id;
localStorage.setItem("meshnet_chat_quantization", selectedChatQuantization);
select.innerHTML = selectable.map(variant =>
`<option value="${esc(variant.id)}">${esc(variant.id)}</option>`
).join("");
select.value = selectedChatQuantization;
select.disabled = false;
vote.style.display = variants.some(variant => !variant.selectable) ? "" : "none";
}
function bindChatModelSelect() {
@@ -1649,6 +1682,24 @@ function selectChatModel(value) {
saveChatSessionsStore();
renderChatSessionList();
}
renderChatQuantizations();
}
function selectChatQuantization(value) {
selectedChatQuantization = value || "";
localStorage.setItem("meshnet_chat_quantization", selectedChatQuantization);
}
async function voteForCoverage() {
const model = selectedChatModelEntry();
if (!model) return;
const unavailable = (model.quantizations || []).find(variant => !variant.selectable);
if (!unavailable) return;
const result = await apiCall("/v1/models/vote", "POST", {
model: model.id,
quantization: unavailable.id,
});
renderChatStatus(result.ok ? `vote recorded for ${unavailable.id} coverage` : "coverage vote failed");
}
async function requestSelectedModelLoad() {
@@ -2049,6 +2100,7 @@ async function sendChat() {
{ role: "user", content: prompt },
],
stream: true,
quantization: selectedChatQuantization || "bfloat16",
max_tokens: 15120,
};
chatBusy = true;
@@ -2213,6 +2265,7 @@ function applyAvailableModels(models, map) {
aliases: model.aliases || [],
coverage: model.shard_coverage_percentage,
servedCopies: model.served_model_copies,
quantizations: model.quantizations || [],
};
entry.servedCopies = modelServedCopiesFromMap(lastNetworkMap, entry);
return entry;

45
packages/tracker/meshnet_tracker/model_presets.json
View File

@@ -107,6 +107,51 @@
},
"input_price_per_1k_tokens": 0.00012,
"output_price_per_1k_tokens": 0.00076
},
"qwen3.6-27b": {
"layers_start": 0,
"layers_end": 63,
"hf_repo": "Qwen/Qwen3.6-27B",
"revision": "6a9e13bd6fc8f0983b9b99948120bc37f49c13e9",
"aliases": [
"qwen3.6-27b",
"Qwen3.6-27B",
"Qwen/Qwen3.6-27B"
],
"recommended": true,
"deployment_status": "recommended",
"price_per_1k_tokens": 0.0006,
"hf_aliases": [
"qwen/qwen3.6-27b"
],
"hf_verified_match_note": "Pinned to the official Qwen/Qwen3.6-27B BF16 revision. Live comparable-provider pricing refreshes this fallback when available; the last verified rate is retained when the provider lookup is unavailable.",
"required_model_bytes": 55576522126,
"download_size_bytes": 55576522126,
"native_quantization": "bfloat16",
"canonical_audit_dtype": "bfloat16",
"canonical_audit_quantization": "bfloat16",
"bytes_per_layer": {
"bfloat16": 868383159,
"int8": 434191580,
"nf4": 217095790
},
"metadata": {
"architecture": "Qwen3.5 hybrid linear-attention transformer",
"total_parameters": "27B",
"num_layers": 64,
"context_length": 262144,
"native_quantization": "bfloat16",
"download_size_gb": 52,
"recommended_short_name": "qwen3.6-27b",
"text_only": true,
"recommended_engines": [
"Transformers",
"vLLM",
"SGLang"
]
},
"input_price_per_1k_tokens": 0.0002,
"output_price_per_1k_tokens": 0.001
}
}
}

600
packages/tracker/meshnet_tracker/server.py
View File

@@ -6,10 +6,14 @@ HTTP API contract:
"endpoint": "http://host:port", "shard_start": int, "shard_end": int,
"model": str optional, "shard_checksum": str optional,
"hardware_profile": object, "wallet_address": str optional,
"score": number optional
"score": number optional,
"capability_report": object optional, # ADR-0023 proof of this exact shard
"recipe_id": str optional, "recipe_version": str optional
}
Response 200: {"node_id": str}
Response 200: {"node_id": str, "capability": {"state": str, "routable": bool, ...}}
Response 400: {"error": str}
A node whose proof does not admit what it advertises still registers (so the
operator can see why it is dark) but is not routable -- see `capability.py`.
- POST /v1/nodes/{node_id}/heartbeat
Response 200: {}
Response 404: {"error": "node not found"}
@@ -33,6 +37,7 @@ import random
import select
import socketserver
import sqlite3
import sys
import tarfile
import threading
import time
@@ -47,6 +52,20 @@ from typing import Any
from .accounts import DEFAULT_ACCOUNTS_DB_PATH, AccountStore
from .auth import is_validator_token, sign_hive_request, verify_hive_request
from .capability import (
DEFAULT_POLICY as DEFAULT_CAPABILITY_POLICY,
POLICY_COMPAT,
POLICY_ENFORCE,
STATE_ABSENT,
STATE_ADMITTED,
STATE_MODEL_MISMATCH,
STATE_SHARD_MISMATCH,
CapabilityState,
absent_state,
evaluate_report,
normalize_policy,
policy_from_env,
)
from .wallet_proof import binding_message, verify_wallet_signature
from .billing import DEFAULT_BILLING_DB_PATH, BillingLedger
from .calibration import DEFAULT_CALIBRATION_DB_PATH, ToplocCalibrationStore
@@ -586,6 +605,8 @@ class _NodeEntry:
"heartbeats_expected", "heartbeats_received",
# dynamic reassignment queued by the tracker
"pending_new_assignment",
# the tracker's verdict on the capability proof this node presented
"capability",
)
def __init__(
@@ -615,6 +636,7 @@ class _NodeEntry:
cert_fingerprint: str | None = None,
peer_id: str | None = None,
friendly_name: str | None = None,
capability: "CapabilityState | None" = None,
) -> None:
self.node_id = node_id
self.endpoint = endpoint
@@ -642,6 +664,9 @@ class _NodeEntry:
self.cert_fingerprint = cert_fingerprint
self.peer_id = peer_id
self.friendly_name = friendly_name
# No proof presented is `absent`, never `admitted` — a node can only earn
# `admitted` by presenting a report that covers what it advertises.
self.capability: CapabilityState = capability or absent_state()
self.pending_directives: list[dict] = []
self.last_heartbeat: float = time.monotonic()
self.total_requests: int = 0
@@ -730,12 +755,88 @@ def _reputation_multiplier(node: "_NodeEntry", contracts: Any | None) -> float:
return 0.5 + 0.5 * reputation
def _node_admission(node: "_NodeEntry") -> CapabilityState:
"""The node's recorded verdict, re-checked against what it advertises *now*.
A proof covers one model/shard combination. The tracker can move a node to a
different range after it registered (rebalance, `pending_new_assignment`), and
the proof does not travel with it: until the node re-registers with a proof
for the range it now advertises, it is shard-mismatched and not routable.
"""
state = node.capability
if not state.proven:
return state
if state.model_id and not _node_matches_model(node, state.model_id):
return state.with_state(
STATE_MODEL_MISMATCH,
f"proof is for model {state.model_id!r}, but the node now serves "
f"{node.hf_repo or node.model!r}",
)
if (
node.shard_start is not None
and node.shard_end is not None
and (state.shard_start, state.shard_end) != (node.shard_start, node.shard_end)
):
return state.with_state(
STATE_SHARD_MISMATCH,
f"proof is for layers {state.shard_start}–{state.shard_end}, but the "
f"node now serves layers {node.shard_start}–{node.shard_end}",
)
return state
def _capability_from_registration(
payload: dict,
*,
model: str | None,
hf_repo: str | None,
shard_start: int | None,
shard_end: int | None,
) -> CapabilityState:
"""The tracker's verdict on the proof carried by one registration payload.
Used by the register handler and by the Raft follower path, so a replicated
registration lands with the same verdict as the one the leader recorded.
"""
aliases = _model_aliases(model) | _model_aliases(hf_repo)
recipe_id = payload.get("recipe_id")
recipe_version = payload.get("recipe_version")
return evaluate_report(
payload.get("capability_report"),
model_matches=lambda reported: bool(_model_aliases(reported) & aliases),
advertised_model=hf_repo or model,
shard_start=shard_start,
shard_end=shard_end,
declared_recipe_id=recipe_id if isinstance(recipe_id, str) else None,
declared_recipe_version=(
recipe_version if isinstance(recipe_version, str) else None
),
)
def _capability_routable(node: "_NodeEntry", policy: str) -> bool:
"""May this node carry traffic under the tracker's capability policy?"""
return _node_admission(node).routable_under(policy)
def _admitted_nodes(nodes: list["_NodeEntry"], policy: str | None) -> list["_NodeEntry"]:
"""Drop every candidate whose capability proof does not admit it (ADR-0023).
This is the single gate every route path goes through. It removes candidates;
it never reorders or reweights them, so coverage-first selection and
throughput-weighted preference among the survivors are unchanged.
"""
effective = normalize_policy(policy)
return [node for node in nodes if _capability_routable(node, effective)]
def _select_route(
nodes: list[_NodeEntry],
required_start: int,
required_end: int,
model: str | None = None,
contracts: Any | None = None,
policy: str | None = None,
) -> tuple[list[_NodeEntry], str]:
"""Greedy interval-cover biased toward fast, lightly-loaded, reputable nodes.
@@ -746,7 +847,10 @@ def _select_route(
Tiebreak: higher shard_end (fewer hops).
"""
candidates = sorted(
[node for node in nodes if node.shard_start is not None and node.shard_end is not None],
[
node for node in _admitted_nodes(nodes, policy)
if node.shard_start is not None and node.shard_end is not None
],
key=lambda n: (n.shard_start, -n.shard_end), # type: ignore[operator]
)
route: list[_NodeEntry] = []
@@ -782,6 +886,7 @@ def _enumerate_routes(
model: str | None = None,
contracts: Any | None = None,
max_candidates: int = 8,
policy: str | None = None,
) -> list["RouteCandidate"]:
"""Enumerate viable route candidates for bandit selection (ADR-0021).
@@ -790,9 +895,13 @@ def _enumerate_routes(
with the longest-advancing hops. The route's prior throughput estimate is
its bottleneck hop's queue-adjusted effective throughput — used only until
observed route samples exist.
Candidates that are not admitted by their capability proof are dropped before
enumeration, so an unproven node cannot appear in any route candidate — not
even as a scouted one.
"""
sharded = [
n for n in nodes
n for n in _admitted_nodes(nodes, policy)
if n.shard_start is not None and n.shard_end is not None
]
# Heads must start the pipeline at the first required layer (they tokenize
@@ -863,7 +972,7 @@ def _coverage_percentage(
intervals = sorted(
(
(max(required_start, node.shard_start), min(required_end, node.shard_end))
for node in nodes
for node in _serviceable_nodes(nodes)
if node.shard_start is not None
and node.shard_end is not None
if node.shard_end >= required_start and node.shard_start <= required_end
@@ -895,7 +1004,7 @@ def _served_model_copies(
layer_counts = []
for layer in range(required_start, required_end + 1):
count = 0
for node in nodes:
for node in _serviceable_nodes(nodes):
if node.shard_start is None or node.shard_end is None:
continue
if node.shard_start <= layer <= node.shard_end:
@@ -970,6 +1079,110 @@ def _preset_bytes_per_layer(preset: dict) -> dict[str, int]:
return {"bfloat16": 30 * 1024 * 1024}
_QUANTIZATION_QUALITY = {"nf4": 0, "int8": 1, "bfloat16": 2}
def _normalize_quantization(value: object) -> str | None:
"""Normalize public precision names to the node/backend vocabulary."""
if not isinstance(value, str):
return None
normalized = value.strip().lower()
if normalized == "bf16":
normalized = "bfloat16"
return normalized if normalized in _QUANTIZATION_QUALITY else None
def _registration_quantization(body: dict, quantizations: list[str]) -> str | None:
"""Resolve the active precision a registration is routable at.
Only the raw body can tell an *absent* `quantization` from an explicit null,
and the two mean opposite things, so this must run at the registration seam
rather than in `_NodeEntry` (both reach the constructor as None).
An absent field predates the protocol adding it: it means "unknown", not
"unsupported", so the node keeps the best precision it advertises and stays
routable. Anything the node states explicitly is taken at its word -- a null,
a non-string, or an unsupported name leaves it with no usable precision and
routing excludes it.
"""
if "quantization" in body:
return _normalize_quantization(body["quantization"])
supported = [
normalized for value in quantizations
if (normalized := _normalize_quantization(value)) is not None
]
if not supported:
return None
return max(supported, key=lambda value: _QUANTIZATION_QUALITY[value])
def _has_usable_quantization(node: _NodeEntry) -> bool:
"""Whether a node declared a precision it can actually serve.
`quantization` is None only for a registration that explicitly declared a
null, malformed, or unsupported precision -- a legacy registration that omits
the field is resolved to a concrete precision at the registration seam. Such a
node is already dark to routing, and shard placement has to keep it that way:
the assignment paths pick a precision via `_node_quantization`, which falls
back to the advertised list and then to a preset default, so an unguarded
assignment would overwrite the declared-unusable precision and silently make
the node routable again.
"""
return node.quantization is not None
def _serviceable_nodes(nodes: list[_NodeEntry]) -> list[_NodeEntry]:
"""The nodes a coverage calculation is allowed to count.
A node with no usable precision is excluded from every routing path, so it
serves none of the layers it claims. Counting its shard range as covered
would report a complete model that no request can reach, and would hide the
hole from the rebalancer whose whole job is to fill it -- the managed nodes
that could cover those layers stay idle because the gap never appears.
"""
return [node for node in nodes if _has_usable_quantization(node)]
def _quantization_satisfies(available: object, requested: object) -> bool:
"""Return whether a shard meets a request's minimum precision."""
available_normalized = _normalize_quantization(available)
requested_normalized = _normalize_quantization(requested)
return (
available_normalized is not None
and requested_normalized is not None
and _QUANTIZATION_QUALITY[available_normalized]
>= _QUANTIZATION_QUALITY[requested_normalized]
)
def _available_quantizations(
nodes: list["_NodeEntry"],
required_start: int,
required_end: int,
) -> list[dict]:
"""Summarize selectable minimum-precision variants for a model.
A request may use higher-precision shards, so each row counts coverage from
every node whose active quantization satisfies that row's minimum.
"""
rows = []
for quantization in ("bfloat16", "int8", "nf4"):
qualifying = [
node for node in nodes
if _quantization_satisfies(node.quantization, quantization)
]
coverage = _coverage_percentage(qualifying, required_start, required_end)
rows.append({
"id": quantization,
"shard_coverage_percentage": coverage,
"served_model_copies": _served_model_copies(
qualifying, required_start, required_end,
),
"selectable": coverage >= 100.0,
})
return rows
def _node_quantization(node: _NodeEntry, preset: dict) -> str:
bytes_per_layer = _preset_bytes_per_layer(preset)
if node.quantization in bytes_per_layer:
@@ -1244,7 +1457,10 @@ def _scale_demanded_models_locked(server: "_TrackerHTTPServer") -> None:
continue
if any((n.hf_repo or n.model) == hf_repo for n in host_nodes):
continue
anchor = max(host_nodes, key=lambda n: n.benchmark_tokens_per_sec)
placeable = [n for n in host_nodes if _has_usable_quantization(n)]
if not placeable:
continue
anchor = max(placeable, key=lambda n: n.benchmark_tokens_per_sec)
if anchor.status != "ready" or anchor.pending_new_assignment is not None:
continue
capacity = min(_node_layer_capacity(anchor, preset), total_layers)
@@ -1293,9 +1509,10 @@ def _request_model_load_locked(server: "_TrackerHTTPServer", model_key: str) ->
if host["spare_slots"] <= 0 and not has_spare_memory:
continue
host_nodes = [server.registry[item["node_id"]] for item in host["loaded"] if item["node_id"] in server.registry]
if not host_nodes:
placeable = [node for node in host_nodes if _has_usable_quantization(node)]
if not placeable:
continue
anchor = max(host_nodes, key=lambda node: node.benchmark_tokens_per_sec)
anchor = max(placeable, key=lambda node: node.benchmark_tokens_per_sec)
if anchor.status != "ready" or anchor.pending_new_assignment is not None:
continue
capacity = min(_node_layer_capacity(anchor, preset), total_layers)
@@ -1311,6 +1528,107 @@ def _request_model_load_locked(server: "_TrackerHTTPServer", model_key: str) ->
return None
def _preferred_node_quantization(
node: _NodeEntry,
preset: dict,
requested: str,
) -> str | None:
"""Choose the highest supported precision that satisfies a request."""
if not _has_usable_quantization(node):
return None
supported = {
normalized
for value in ([node.quantization] if node.quantization else []) + list(node.quantizations)
if (normalized := _normalize_quantization(value)) in _preset_bytes_per_layer(preset)
}
qualifying = [
value for value in supported if _quantization_satisfies(value, requested)
]
return max(qualifying, key=lambda value: _QUANTIZATION_QUALITY[value]) if qualifying else None
def _queue_initial_model_placement_locked(
server: "_TrackerHTTPServer",
model_key: str,
requested_quantization: str,
) -> list[dict]:
"""Queue a complete first copy on spare tracker-managed capacity.
This intentionally never replaces an existing assignment. Replacement is
a later demand-balancing action, while the first chat request may use only
genuinely spare slots and memory.
"""
resolved_name, preset = _resolve_model_preset(server.model_presets, model_key)
if preset is None or not preset.get("recommended") or not preset.get("hf_repo"):
return []
required_start, required_end = _preset_layer_bounds(preset)
qualifying = [
node for node in server.registry.values()
if _node_matches_preset(node, resolved_name, preset) # type: ignore[arg-type]
and _quantization_satisfies(node.quantization, requested_quantization)
]
if _coverage_percentage(qualifying, required_start, required_end) >= 100.0:
return []
cursor = required_start
queued: list[dict] = []
for host in _memory_pool_map(server)["hosts"]:
if cursor > required_end or host["spare_slots"] <= 0:
break
host_nodes = [
server.registry[item["node_id"]]
for item in host["loaded"]
if item["node_id"] in server.registry
]
candidates = [
node for node in host_nodes
if node.status == "ready" and node.pending_new_assignment is None
and _preferred_node_quantization(node, preset, requested_quantization) is not None
]
if not candidates:
continue
anchor = max(candidates, key=lambda node: node.benchmark_tokens_per_sec)
quantization = _preferred_node_quantization(anchor, preset, requested_quantization)
if quantization is None:
continue
layer_bytes = _preset_bytes_per_layer(preset)[quantization]
capacity = int(host["memory_spare_bytes"] // layer_bytes)
if capacity <= 0:
continue
end = min(required_end, cursor + capacity - 1)
directive = _add_shard_directive(
anchor, str(preset["hf_repo"]), cursor, end, quantization,
)
anchor.pending_new_assignment = directive
anchor.pending_directives.append(directive)
queued.append({
"node_id": anchor.node_id,
"shard_start": cursor,
"shard_end": end,
"quantization": quantization,
})
cursor = end + 1
if cursor <= required_end:
# The directives are not safe to issue without a complete route. Undo
# only the directives created above; unrelated pending work is intact.
for placement in queued:
node = server.registry.get(placement["node_id"])
if node is None:
continue
node.pending_directives = [
directive for directive in node.pending_directives
if directive is not node.pending_new_assignment
]
node.pending_new_assignment = None
return []
_tracker_log(
server, "info", "initial demanded model placement queued",
model=resolved_name, hf_repo=preset["hf_repo"],
requested_quantization=requested_quantization, placements=queued,
)
return queued
def _deployment_summary(nodes: list[_NodeEntry], preset: dict | None) -> dict:
if preset is None:
return {"recommended": False}
@@ -1370,7 +1688,7 @@ def _coverage_map(
layer_counts = []
for layer in range(required_start, required_end + 1):
count = 0
for node in nodes:
for node in _serviceable_nodes(nodes):
if node.shard_start is None or node.shard_end is None:
continue
if node.shard_start <= layer <= node.shard_end:
@@ -1419,8 +1737,10 @@ def _coverage_map_detailed(
) -> list[dict]:
"""Like _coverage_map but with per-node identity and health in each band.
Includes all nodes (alive and stale) so operators can see both coverage
holes and which dead nodes used to fill them.
Includes all nodes (alive and stale, serviceable or not) so operators can see
both coverage holes and which nodes used to fill them. This is why it does not
share `_coverage_map`'s serviceability filter: a node dark for a declared bad
precision is exactly what an operator staring at a hole needs to see.
"""
now = time.monotonic()
@@ -1532,6 +1852,8 @@ def _assign_redundant_managed_nodes(
):
if node.shard_start is not None and node.shard_end is not None:
continue
if not _has_usable_quantization(node):
continue
capacity = min(_node_layer_capacity(node, preset), total_layers)
if capacity <= 0:
continue
@@ -1964,7 +2286,8 @@ def _rebalance_model_locked(server: "_TrackerHTTPServer", model: str) -> None:
eligible_nodes = [
node for node in managed_nodes
if _node_layer_capacity(node, preset) > 0
if _has_usable_quantization(node)
and _node_layer_capacity(node, preset) > 0
]
node_index = 0
for gap_start, gap_end in gaps:
@@ -2054,7 +2377,8 @@ def _rebalance_hf_model_locked(server: "_TrackerHTTPServer", hf_repo: str) -> No
eligible_nodes = [
node for node in managed_nodes
if _node_layer_capacity(node, preset) > 0
if _has_usable_quantization(node)
and _node_layer_capacity(node, preset) > 0
]
node_index = 0
for gap_start, gap_end in gaps:
@@ -2522,9 +2846,13 @@ class _TrackerHTTPServer(socketserver.ThreadingMixIn, http.server.HTTPServer):
route_stats: "RouteStatsStore | None" = None,
relay_status: dict | None = None,
test_runner: "TestRunManager | None" = None,
capability_policy: str | None = None,
) -> None:
super().__init__(addr, handler)
self.registry = registry
self.capability_policy = normalize_policy(
capability_policy if capability_policy is not None else policy_from_env()
)
self.lock = lock
self.heartbeat_timeout = heartbeat_timeout
self.model_presets = model_presets
@@ -2715,6 +3043,9 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
if self.path == "/v1/models/load":
self._handle_model_load_request()
return
if self.path == "/v1/models/vote":
self._handle_model_coverage_vote()
return
if self.path == "/v1/calibration/toploc/run":
self._handle_toploc_calibration_run()
return
@@ -2842,6 +3173,11 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
required_start,
required_end,
)
quantizations = _available_quantizations(
model_nodes,
required_start,
required_end,
)
aliases = [name]
hf_repo = preset.get("hf_repo")
if hf_repo and hf_repo not in aliases:
@@ -2856,12 +3192,14 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
"owned_by": "meshnet",
"name": name,
"hf_repo": hf_repo,
"revision": preset.get("revision"),
"aliases": aliases,
"metadata": dict(preset.get("metadata") or _model_metadata_from_nodes(model_nodes)),
"recommended": bool(preset.get("recommended", False)),
"deployment": _deployment_summary(alive, preset),
"shard_coverage_percentage": coverage,
"served_model_copies": served_copies,
"quantizations": quantizations,
})
seen_ids.add(name)
# Note: the preset's hf_repo is deliberately NOT added to seen_ids —
@@ -3014,15 +3352,26 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
def model_supply_for(node: _NodeEntry) -> dict:
return _model_health_summary(server, node.model, node.hf_repo)
def capability_for(node: _NodeEntry) -> dict:
# Re-verdicted against what the node advertises now, so the operator
# view and the routing gate can never disagree.
state = _node_admission(node)
return {
**state.to_dict(),
"routable": state.routable_under(server.capability_policy),
}
self._send_json(200, {
"relay_url": server.relay_url,
"relay": dict(server.relay_status),
"capability_policy": server.capability_policy,
"pool": _pool_summary(nodes),
"memory_pool": memory_pool,
"recommended_models": [
{
"id": name,
"hf_repo": preset.get("hf_repo"),
"revision": preset.get("revision"),
"aliases": list(preset.get("aliases", []) or []),
"metadata": dict(preset.get("metadata") or {}),
"deployment": _deployment_summary(nodes, preset),
@@ -3049,6 +3398,7 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
"capacity": capacity_for(node),
"model_supply": model_supply_for(node),
"throughput": throughput_for(node),
"capability": capability_for(node),
"stats": _node_health(node, server.heartbeat_timeout),
}
for node in nodes
@@ -3077,6 +3427,17 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
model: str = body.get("model", "")
is_stream: bool = bool(body.get("stream", False))
requested_quantization = _normalize_quantization(
body.get("quantization", "bfloat16")
)
if requested_quantization is None:
self._send_json(400, {"error": {
"message": "quantization must be one of bfloat16, int8, or nf4",
"type": "invalid_request_error",
"code": "invalid_quantization",
}})
return
body["quantization"] = requested_quantization
if model and server.stats is not None:
server.stats.record_request(model)
@@ -3170,6 +3531,16 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
pinned_nodes = [server.registry[nid] for nid in pinned_ids]
if pinned_nodes is not None:
if any(
not _quantization_satisfies(node.quantization, requested_quantization)
for node in pinned_nodes
):
self._send_json(409, {"error": {
"message": "pinned route does not satisfy requested quantization",
"type": "invalid_request_error",
"code": "route_quantization_unavailable",
}})
return
node = pinned_nodes[0]
else:
# Find a live tracker-mode node for this model
@@ -3177,7 +3548,10 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
self._purge_expired_nodes()
candidates = [
n for n in server.registry.values()
if n.tracker_mode and _node_matches_model(n, model)
if n.tracker_mode
and _node_matches_model(n, model)
and _quantization_satisfies(n.quantization, requested_quantization)
and _capability_routable(n, server.capability_policy)
]
if not candidates:
@@ -3185,11 +3559,49 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
with server.lock:
candidates = [
n for n in server.registry.values()
if n.shard_start == 0 and _node_matches_model(n, model)
if n.shard_start == 0
and _node_matches_model(n, model)
and _quantization_satisfies(n.quantization, requested_quantization)
and _capability_routable(n, server.capability_policy)
]
if not candidates:
with server.lock:
resolved_name, preset = _resolve_model_preset(server.model_presets, model)
placements = (
_queue_initial_model_placement_locked(
server,
resolved_name or model,
requested_quantization,
)
if preset is not None and preset.get("recommended")
else []
)
if preset is not None:
required_start, required_end = _preset_layer_bounds(preset)
qualifying = [
n for n in server.registry.values()
if _node_matches_preset(n, resolved_name, preset)
and _quantization_satisfies(
n.quantization, requested_quantization,
)
]
coverage = _coverage_percentage(
qualifying, required_start, required_end,
)
else:
coverage = 0.0
if placements:
self._send_json(503, {"error": {
"message": f"model {model!r} is being deployed",
"type": "service_unavailable",
"code": "model_loading",
"retry_after_seconds": 30,
"requested_quantization": requested_quantization,
"shard_coverage_percentage": coverage,
"placements": placements,
}})
return
registered = [
{
"node_id": n.node_id,
@@ -3235,12 +3647,14 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
if _node_matches_preset(n, route_model, preset)
and n.shard_start is not None
and n.shard_end is not None
and _quantization_satisfies(n.quantization, requested_quantization)
]
else:
all_nodes = [
n for n in server.registry.values()
if _node_matches_model(n, route_model)
and n.shard_start is not None and n.num_layers is not None
and _quantization_satisfies(n.quantization, requested_quantization)
]
rs, re = 0, (max((n.num_layers for n in all_nodes), default=1) - 1)
if pinned_nodes is not None:
@@ -3254,6 +3668,7 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
model=route_model,
contracts=server.contracts,
max_candidates=server.route_stats.config.max_candidate_routes,
policy=server.capability_policy,
)
picked, routing_decision = choose_route(
route_candidates, server.route_stats, route_model, rng=server.route_rng,
@@ -3263,7 +3678,12 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
else:
# No head-anchored candidate — legacy greedy cover as fallback
# (also produces the layer-gap error message).
route_nodes, route_error = _select_route(all_nodes, rs, re, model=route_model, contracts=server.contracts)
route_nodes, route_error = _select_route(
all_nodes, rs, re,
model=route_model,
contracts=server.contracts,
policy=server.capability_policy,
)
routing_decision = {"mode": "greedy-fallback"}
if route_error:
_tracker_log(
@@ -4067,10 +4487,10 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
self._send_json(400, {"error": "quantizations must be a non-empty string array"})
return
quantizations = list(quantizations_body)
quantization = body.get("quantization")
if quantization is not None and not isinstance(quantization, str):
if body.get("quantization") is not None and not isinstance(body["quantization"], str):
self._send_json(400, {"error": "quantization must be a string"})
return
quantization = _registration_quantization(body, quantizations)
wallet_address = body.get("wallet_address")
if wallet_address is not None and not isinstance(wallet_address, str):
self._send_json(400, {"error": "wallet_address must be a string"})
@@ -4118,6 +4538,25 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
self._send_json(400, {"error": str(exc)})
return
recipe_id = body.get("recipe_id")
recipe_version = body.get("recipe_version")
if (recipe_id is not None and not isinstance(recipe_id, str)) or (
recipe_version is not None and not isinstance(recipe_version, str)
):
self._send_json(400, {"error": "recipe_id and recipe_version must be strings"})
return
# The capability proof (ADR-0023). A bad proof does not fail registration --
# the node is recorded with its verdict so the operator can see *why* it is
# not routing -- but only an `admitted` verdict makes it routable.
capability = _capability_from_registration(
body,
model=model,
hf_repo=hf_repo,
shard_start=shard_start,
shard_end=shard_end,
)
node_id = _node_id_for_registration(
endpoint,
model,
@@ -4152,6 +4591,7 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
cert_fingerprint=cert_fingerprint,
peer_id=peer_id,
friendly_name=friendly_name,
capability=capability,
)
with server.lock:
self._purge_expired_nodes()
@@ -4207,6 +4647,7 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
old_model_health=_model_health_summary(server, old.model, old.hf_repo),
model_health=model_health,
)
routable = _capability_routable(entry, server.capability_policy)
_tracker_log(
server,
"info",
@@ -4218,7 +4659,19 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
shard=f"{entry.shard_start}-{entry.shard_end}",
tracker_mode=entry.tracker_mode,
model_health=model_health,
capability=entry.capability.state,
routable=routable,
)
if not routable:
_tracker_log(
server,
"warn",
"node registered but is not routable",
node_id=node_id,
endpoint=entry.endpoint,
capability=entry.capability.state,
detail=entry.capability.detail,
)
shard_info = f"layers {shard_start}-{shard_end}" if shard_start is not None else "unsharded"
repo_info = f" [{hf_repo}]" if hf_repo else ""
@@ -4230,7 +4683,17 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
flush=True,
)
payload = {"node_id": node_id}
payload = {
"node_id": node_id,
# Tell the node what the tracker made of its proof: a node that is
# registered but not routable must be able to see that it is dark.
"capability": {
"state": entry.capability.state,
"detail": entry.capability.detail,
"routable": routable,
"policy": server.capability_policy,
},
}
if assignment_directive is not None:
payload["directive"] = assignment_directive
self._send_json(200, payload)
@@ -4364,6 +4827,27 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
return
self._send_json(202, {"status": "queued", "assignment": assignment})
def _handle_model_coverage_vote(self):
"""Record a rolling wish-list signal for an unavailable precision."""
server: _TrackerHTTPServer = self.server # type: ignore[assignment]
body = self._read_json_body()
if body is None:
return
model = body.get("model")
quantization = _normalize_quantization(body.get("quantization"))
if not isinstance(model, str) or not model.strip() or quantization is None:
self._send_json(400, {"error": "model and valid quantization are required"})
return
resolved_name, preset = _resolve_model_preset(server.model_presets, model)
if preset is None:
self._send_json(404, {"error": "unknown model preset"})
return
vote_key = f"wishlist:{resolved_name}:{quantization}"
if server.stats is not None:
server.stats.record_request(vote_key)
_tracker_log(server, "info", "model coverage vote", model=resolved_name, quantization=quantization)
self._send_json(202, {"status": "recorded", "model": resolved_name, "quantization": quantization})
def _handle_stats(self):
server: _TrackerHTTPServer = self.server # type: ignore[assignment]
if server.stats is None:
@@ -5742,6 +6226,7 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
model=route_model,
contracts=server.contracts,
max_candidates=server.route_stats.config.max_candidate_routes,
policy=server.capability_policy,
)
if candidates:
# Prefer a distributed multi-hop route when available. Greedy
@@ -5751,7 +6236,9 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
error = ""
else:
route, error = _select_route(
alive, required_start, required_end, contracts=server.contracts,
alive, required_start, required_end,
contracts=server.contracts,
policy=server.capability_policy,
)
if error:
_tracker_log(
@@ -5837,6 +6324,7 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
model=stats_key,
contracts=server.contracts,
max_candidates=cfg.max_candidate_routes,
policy=server.capability_policy,
)
out[stats_key] = {
"epoch": server.route_stats.epoch(stats_key),
@@ -5930,7 +6418,11 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
routes = []
remaining = list(candidates)
for _ in range(redundancy):
route, error = _select_route(remaining, required_start, required_end, contracts=server.contracts)
route, error = _select_route(
remaining, required_start, required_end,
contracts=server.contracts,
policy=server.capability_policy,
)
if error:
self._send_json(503, {"error": error})
return
@@ -6015,7 +6507,11 @@ class TrackerServer:
routing_config: RoutingConfig | None = None,
enable_test_runner: bool = False,
test_runner: TestRunManager | None = None,
capability_policy: str | None = None,
) -> None:
self._capability_policy = normalize_policy(
capability_policy if capability_policy is not None else policy_from_env()
)
self._host = host
self._requested_port = port
self._heartbeat_timeout = heartbeat_timeout
@@ -6228,6 +6724,7 @@ class TrackerServer:
route_stats=self._route_stats,
relay_status=http_relay_status,
test_runner=self._test_runner,
capability_policy=self._capability_policy,
)
self.port = self._server.server_address[1]
@@ -6441,6 +6938,21 @@ class TrackerServer:
friendly_name = _normalize_friendly_name(payload.get("friendly_name"))
except ValueError:
friendly_name = None
# The replicated payload is the raw registration body, so the follower can
# resolve precision exactly as the leader did -- including telling a legacy
# absent `quantization` from a declared one. Dropping these fields here
# would rebuild every node at the default precision and diverge follower
# routing from leader routing.
quantizations_payload = payload.get("quantizations")
quantizations = (
list(quantizations_payload)
if (
isinstance(quantizations_payload, list)
and quantizations_payload
and all(isinstance(item, str) and item for item in quantizations_payload)
)
else list(DEFAULT_QUANTIZATIONS)
)
entry = _NodeEntry(
node_id=node_id,
endpoint=endpoint.rstrip("/"),
@@ -6451,6 +6963,8 @@ class TrackerServer:
hardware_profile=payload.get("hardware_profile", {}),
wallet_address=payload.get("wallet_address"),
score=float(payload.get("score", 1.0)),
quantizations=quantizations,
quantization=_registration_quantization(payload, quantizations),
tracker_mode=bool(payload.get("tracker_mode", False)),
hf_repo=payload.get("hf_repo"),
num_layers=int(payload["num_layers"]) if payload.get("num_layers") is not None else None,
@@ -6461,6 +6975,15 @@ class TrackerServer:
else None
),
friendly_name=_normalize_friendly_name(payload.get("friendly_name")),
# A replicated registration carries its proof: without this, a proven
# node would be routable on the leader and dark on every follower.
capability=_capability_from_registration(
payload,
model=payload.get("model", "stub-model"),
hf_repo=payload.get("hf_repo"),
shard_start=shard_start,
shard_end=shard_end,
),
)
with self._lock:
self._registry[node_id] = entry
@@ -6492,19 +7015,32 @@ class TrackerServer:
delivered_all = False
return delivered_all
def _save_stats_once(self) -> None:
"""Persist each store independently so one DB failure cannot kill the loop."""
registry_log = getattr(self._contracts, "registry_log", None)
stores = (
("request stats", self._stats),
("route stats", self._route_stats),
("billing", self._billing),
("accounts", self._accounts),
("registry", registry_log),
)
for label, store in stores:
if store is None:
continue
try:
store.save_to_db()
except Exception as exc:
print(
f"[tracker] warn: {label} persistence failed: {exc}",
file=sys.stderr,
flush=True,
)
def _stats_loop(self) -> None:
"""Periodically save stats/billing to DB and push local slices to cluster peers."""
while not self._stats_stop.wait(_StatsCollector.SAVE_INTERVAL):
if self._stats is not None:
self._stats.save_to_db()
self._route_stats.save_to_db()
if self._billing is not None:
self._billing.save_to_db()
if self._accounts is not None:
self._accounts.save_to_db()
registry_log = getattr(self._contracts, "registry_log", None)
if registry_log is not None:
registry_log.save_to_db()
self._save_stats_once()
if self._cluster_peers and not self._hive_secret:
print(
"[tracker] WARNING: cluster peers configured without --hive-secret — "

63
tests/test_activation_compression.py Normal file
View File

@@ -0,0 +1,63 @@
"""Trace-driven activation-compression policy units."""
import os
import pytest
from meshnet_node.activation_compression import (
CompressionPolicies,
CompressionPolicy,
compress_activation,
decompress_activation,
)
def test_compressible_body_uses_zstd_when_it_clears_savings_policy():
body = b"activation" * 20_000
result = compress_activation(body, CompressionPolicy(min_input_bytes=1, min_savings_bytes=100))
assert result.encoding == "zstd"
assert result.output_bytes < result.input_bytes
assert decompress_activation(result.body, result.encoding).body == body
def test_incompressible_body_stays_raw_after_measured_trial():
body = os.urandom(32 * 1024)
result = compress_activation(body, CompressionPolicy(min_input_bytes=1, min_savings_bytes=1))
assert result.encoding is None
assert result.body == body
assert result.decision == "below_savings"
def test_small_body_uses_raw_fast_path_without_zstd_trial():
body = b"x" * 1024
result = compress_activation(body, CompressionPolicy(min_input_bytes=2048))
assert result.encoding is None
assert result.decision == "below_min_input"
assert result.elapsed_seconds >= 0
def test_threshold_requires_both_byte_and_ratio_savings():
body = b"a" * 4096
result = compress_activation(
body, CompressionPolicy(min_input_bytes=1, min_savings_bytes=len(body), min_savings_ratio=0),
)
assert result.encoding is None
assert result.decision == "below_savings"
def test_malformed_zstd_and_legacy_raw_bodies_are_handled_explicitly():
assert decompress_activation(b"legacy", None).body == b"legacy"
with pytest.raises(ValueError, match="invalid zstd activation body"):
decompress_activation(b"not a zstd frame", "zstd")
with pytest.raises(ValueError, match="unsupported"):
decompress_activation(b"body", "gzip")
def test_prefill_decode_and_route_conditions_have_independent_config(monkeypatch):
policies = CompressionPolicies()
assert policies.for_condition("relay", "prefill").min_input_bytes < policies.for_condition("relay", "decode").min_input_bytes
monkeypatch.setenv("MESHNET_COMPRESSION_RELAY_DECODE_MIN_INPUT_BYTES", "123")
monkeypatch.setenv("MESHNET_COMPRESSION_LAN_PREFILL_ENABLED", "false")
assert policies.for_condition("relay", "decode").min_input_bytes == 123
assert not policies.for_condition("lan", "prefill").enabled
assert policies.for_condition("benchmark", "prefill").enabled

5
tests/test_dynamic_routing.py
View File

@@ -15,6 +15,7 @@ from meshnet_tracker.routing_stats import (
route_signature,
route_table,
)
from meshnet_tracker.capability import absent_state
from meshnet_tracker.server import TrackerServer, _enumerate_routes
@@ -49,6 +50,9 @@ def _fake_node(node_id, shard_start, shard_end, benchmark=100.0, endpoint=None):
proxy_inflight=0,
wallet_address=None,
relay_addr=None,
# A pre-capability node (NCA-004): routable only under the `compat`
# policy, which is what these route-enumeration tests exercise.
capability=absent_state(),
)
@@ -270,6 +274,7 @@ def test_proxy_head_is_route_head_and_routing_endpoint_lists_routes():
"shard_start": 0,
"shard_end": shard_end,
"tracker_mode": True,
"quantization": "bfloat16",
"benchmark_tokens_per_sec": bench,
"hardware_profile": {},
"score": 1.0},

149
tests/test_gossip_and_relay.py
View File

@@ -454,6 +454,137 @@ def test_relay_rpc_reuses_connection_for_sequential_requests(monkeypatch):
assert connection_attempts == 1
def test_relay_hop_client_preserves_binary_json_and_closes_uncertain_legacy_socket(monkeypatch):
"DIP-002: persistent clients correlate both frame formats and never replay a lost response.\n\nTags: gossip, network, relay"
import websockets.sync.client as wsc # type: ignore[import]
from meshnet_node.relay_bridge import decode_binary_frame, encode_binary_frame
from meshnet_node.torch_server import _RelayHopClient, _RelayRequestUncertainError
class FakeSocket:
def __init__(self, responses):
self.responses = iter(responses)
self.sent = []
self.closed = False
def send(self, frame):
self.sent.append(frame)
def recv(self, timeout):
response = next(self.responses)
if isinstance(response, BaseException):
raise response
request, _ = decode_binary_frame(self.sent[-1])
if isinstance(response, bytes):
return encode_binary_frame({
"request_id": request["request_id"], "status": 200, "headers": {},
}, response)
return json.dumps({"request_id": request["request_id"], "status": 200,
"headers": {}, "body": response})
def close(self):
self.closed = True
healthy = FakeSocket([b"binary", "json"])
legacy = FakeSocket([b"first", TimeoutError("legacy relay closed")])
session_a = FakeSocket([b"a"])
session_b = FakeSocket([b"b"])
sockets = iter([healthy, legacy, session_a, session_b])
monkeypatch.setattr(wsc, "connect", lambda *args, **kwargs: next(sockets))
client = _RelayHopClient("ws://relay/rpc/peer", timeout=0.01)
assert client.request("/forward", b"a", {})[2] == b"binary"
assert client.request("/forward", b"b", {})[2] == b"json"
first_id = decode_binary_frame(healthy.sent[0])[0]["request_id"]
second_id = decode_binary_frame(healthy.sent[1])[0]["request_id"]
assert first_id != second_id
client.close()
client = _RelayHopClient("ws://relay/rpc/peer", timeout=0.01)
assert client.request("/forward", b"first", {})[2] == b"first"
try:
client.request("/forward", b"second", {})
except _RelayRequestUncertainError:
pass
else:
raise AssertionError("a post-send disconnect must not be retried")
assert legacy.closed is True
assert client._ws is None
# Route Sessions own their own requester socket; serialising one client's
# calls must not accidentally make another session share it.
first = _RelayHopClient("ws://relay/rpc/peer", timeout=0.01)
second = _RelayHopClient("ws://relay/rpc/peer", timeout=0.01)
assert first.request("/forward", b"a", {})[2] == b"a"
assert second.request("/forward", b"b", {})[2] == b"b"
assert first._ws is session_a
assert second._ws is session_b
def test_relay_rpc_cleans_pending_on_timeout_disconnect_and_cancellation():
"DIP-002: every terminal relay RPC path releases its pending response queue.\n\nTags: gossip, network, relay"
import asyncio
from meshnet_relay.server import RelayServer
class Requester:
def __init__(self):
self.messages = [json.dumps({
"request_id": "legacy-id", "method": "POST", "path": "/forward",
"headers": {}, "body": "{}",
})]
self.sent = []
async def recv(self):
if self.messages:
return self.messages.pop(0)
raise EOFError
async def send(self, message):
self.sent.append(json.loads(message))
async def close(self, *args):
pass
class Target:
async def send(self, message):
pass
async def wait_for_pending(relay):
for _ in range(100):
if relay._pending_rpc:
return
await asyncio.sleep(0)
raise AssertionError("relay RPC did not register pending state")
async def exercise(kind):
relay = RelayServer(rpc_timeout=0.02, rpc_idle_timeout=0.01)
relay.registry.register("peer", "", Target())
requester = Requester()
task = asyncio.create_task(relay._handle_rpc(requester, "peer"))
await wait_for_pending(relay)
if kind == "disconnect":
relay._fail_pending_for_peer("peer")
elif kind == "cancel":
task.cancel()
if kind == "cancel":
try:
await task
except asyncio.CancelledError:
pass
else:
await task
assert relay._pending_rpc == {}
if kind == "timeout":
assert requester.sent[-1]["status"] == 504
if kind == "disconnect":
assert requester.sent[-1]["status"] == 503
for kind in ("timeout", "disconnect", "cancel"):
asyncio.run(exercise(kind))
def test_binary_relay_frame_codecs_interoperate():
"Node and relay ship the same binary frame format as separate copies.\n\nTags: gossip, network, relay"
@@ -478,6 +609,24 @@ def test_binary_relay_frame_codecs_interoperate():
raise AssertionError("garbage bytes must not decode as a binary frame")
def test_binary_relay_frame_layout_remains_byte_for_byte_compatible():
"""Framing optimizations preserve the MRF1 header-length-body contract.
Tags: gossip, network, relay, wire
"""
import json
from meshnet_node.relay_bridge import BINARY_FRAME_MAGIC, encode_binary_frame
header = {"request_id": "r-1", "headers": {"X-Meshnet-Shape": "1,1,2"}}
body = b"\x01\x02\x03\x04"
header_bytes = json.dumps(header, separators=(",", ":")).encode()
assert encode_binary_frame(header, body) == (
BINARY_FRAME_MAGIC + len(header_bytes).to_bytes(4, "big") + header_bytes + body
)
def test_activation_compression_round_trips_and_skips_small_bodies():
"Pipeline hops zstd-compress large activations; tiny decode bodies pass raw.\n\nTags: gossip, network, relay"

2
tests/test_hf_pricing_dispatch.py
View File

@@ -95,7 +95,7 @@ def test_refresh_loop_repriced_model_with_curated_alias(pricing_tracker):
tracker_url, ledger, tracker = pricing_tracker
new_price = _wait_for_price_change(ledger, PRICED_MODEL)
expected = round((0.93 + 3.00) / 2 / 1000 * 0.80, 6)
assert new_price == expected
assert new_price == pytest.approx(expected)
preset = tracker._model_presets[PRICED_MODEL]
assert preset["hf_last_price_per_1k"] == expected

134
tests/test_http_keepalive.py Normal file
View File

@@ -0,0 +1,134 @@
"""DIP-003 keep-alive ownership and framing-adjacent transport tests."""
from __future__ import annotations
import io
import pytest
class _Response:
def __init__(self, status: int = 200, body: bytes = b"ok", headers: dict | None = None):
self.status = status
self._body = body
self.headers = headers or {"Content-Type": "application/octet-stream", "Content-Length": str(len(body))}
def read(self) -> bytes:
return self._body
def close(self) -> None:
pass
class _Connection:
instances: list["_Connection"] = []
def __init__(self, *args, **kwargs):
self.requests: list[tuple] = []
self.responses: list[object] = [_Response(), _Response()]
self.closed = False
self.__class__.instances.append(self)
def request(self, *args, **kwargs) -> None:
self.requests.append((args, kwargs))
def getresponse(self):
response = self.responses.pop(0)
if isinstance(response, BaseException):
raise response
return response
def close(self) -> None:
self.closed = True
def test_direct_hop_client_reuses_one_connection_and_discards_uncertain_socket(monkeypatch):
"""A Route Session owns one serialized direct socket and never replays it.
Tags: performance, routing
"""
from meshnet_node import torch_server
_Connection.instances = []
monkeypatch.setattr(torch_server.http.client, "HTTPConnection", _Connection)
client = torch_server._DirectHopClient("http://tail.example:8001")
assert client.request("/forward", b"one", {})[2] == b"ok"
assert client.request("/forward", b"two", {})[2] == b"ok"
assert len(_Connection.instances) == 1
assert [call[0][1] for call in _Connection.instances[0].requests] == ["/forward", "/forward"]
_Connection.instances[0].responses.append(ConnectionResetError("stale peer"))
with pytest.raises(torch_server._DirectRequestUncertainError):
client.request("/forward", b"three", {})
assert _Connection.instances[0].closed is True
assert client._connection is None
def test_bridge_pool_reuses_a_worker_connection_and_invalidates_stale_one(monkeypatch):
"""A bridge worker keeps its own loopback client; broken clients are dropped.
Tags: performance, relay
"""
from meshnet_node import relay_bridge
_Connection.instances = []
monkeypatch.setattr(relay_bridge.http.client, "HTTPConnection", _Connection)
bridge = relay_bridge.RelayHttpBridge(
relay_url="ws://relay.example/ws",
peer_id="peer",
local_base_url="http://127.0.0.1:8001",
advertised_addr="",
)
frames: list[dict] = []
bridge._send_response_frame = lambda frame: (frames.append(frame), True)[1]
request = {"request_id": "one", "method": "POST", "path": "/forward", "headers": {}, "body": ""}
bridge._process_request(request)
bridge._process_request({**request, "request_id": "two"})
assert len(_Connection.instances) == 1
assert len(_Connection.instances[0].requests) == 2
_Connection.instances[0].responses.append(ConnectionResetError("stale loopback"))
bridge._process_request({**request, "request_id": "broken"})
assert _Connection.instances[0].closed is True
bridge._process_request({**request, "request_id": "replacement"})
assert len(_Connection.instances) == 2
bridge.stop()
def test_node_sse_uses_chunked_framing_and_tolerates_client_cancellation():
"""HTTP/1.1 streams terminate without EOF and ignore a cancelled client.
Tags: performance, streaming
"""
from meshnet_node.torch_server import _TorchHandler
handler = object.__new__(_TorchHandler)
headers: list[tuple[str, str]] = []
handler.send_response = lambda status: None
handler.send_header = lambda name, value: headers.append((name, value))
handler.end_headers = lambda: None
handler.wfile = io.BytesIO()
emit = handler._start_openai_stream("model")
emit(None)
wire = handler.wfile.getvalue()
assert _TorchHandler.protocol_version == "HTTP/1.1"
assert ("Transfer-Encoding", "chunked") in headers
assert wire.endswith(b"0\r\n\r\n")
assert b"data: [DONE]" in wire
class _CancelledWriter:
def write(self, _body):
raise BrokenPipeError
def flush(self):
raise BrokenPipeError
cancelled = object.__new__(_TorchHandler)
cancelled.send_response = lambda status: None
cancelled.send_header = lambda name, value: None
cancelled.end_headers = lambda: None
cancelled.wfile = _CancelledWriter()
cancelled._start_openai_stream("model")(None)

8
tests/test_meshnet_sdk.py
View File

@@ -14,9 +14,13 @@ def test_sdk_support_helpers_are_deterministic_without_network():
"Non-socket coverage for SDK-facing gateway/tracker helper behavior.\n\nTags: gateway, http, sdk"
address = _payment_address_for_api_key("sdk-key")
# Registration always resolves a precision, so a registry node carries one;
# coverage counts only nodes that can actually serve the layers they claim.
nodes = [
_NodeEntry("a", "http://node-a", 0, 15, "stub-model", None, {}, None, 1.0),
_NodeEntry("b", "http://node-b", 16, 31, "stub-model", None, {}, None, 1.0),
_NodeEntry("a", "http://node-a", 0, 15, "stub-model", None, {}, None, 1.0,
quantization="bfloat16"),
_NodeEntry("b", "http://node-b", 16, 31, "stub-model", None, {}, None, 1.0,
quantization="bfloat16"),
]
assert 32 <= len(address) <= 44

57
tests/test_mining_cli.py
View File

@@ -9,6 +9,10 @@ import types
from pathlib import Path
from unittest.mock import MagicMock, patch
# A fake node server has no real backend to prove capability with; say so
# explicitly rather than bypassing startup's fail-closed admission.
from meshnet_node.testing import assume_capability
# ---------------------------------------------------------------------------
# model_catalog tests
@@ -336,6 +340,7 @@ def test_startup_auto_detects_shard_range(monkeypatch, tmp_path):
# shard_start and shard_end intentionally omitted
quantization="bfloat16",
host="127.0.0.1",
capability_validator=assume_capability,
)
assert calls == ["Qwen/Qwen2.5-0.5B-Instruct"]
assert isinstance(node, FakeNode)
@@ -548,9 +553,23 @@ def test_legacy_start_falls_back_to_env_tracker_and_model(monkeypatch):
assert captured["model_id"] == "Qwen/Qwen2.5-0.5B-Instruct"
def test_legacy_start_without_port_uses_next_available_port(monkeypatch):
"Omitting --port skips an occupied default port before startup loads the model.\n\nTags: general"
from meshnet_node.cli import main
def test_first_available_port_skips_an_occupied_custom_port():
"Port search skips an occupied custom base port.\n\nTags: general"
from meshnet_node.cli import _first_available_port
occupied = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
occupied.bind(("127.0.0.1", 0))
start = occupied.getsockname()[1]
occupied.listen(1)
try:
assert _first_available_port("127.0.0.1", start=start, attempts=100) > start
finally:
occupied.close()
def test_legacy_start_without_port_uses_port_search(monkeypatch):
"Omitting --port delegates legacy startup to the port-search helper.\n\nTags: general"
from meshnet_node import cli as cli_mod
captured = {}
@@ -561,26 +580,20 @@ def test_legacy_start_without_port_uses_next_available_port(monkeypatch):
def stop(self): pass
return _FakeNode()
occupied = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
occupied.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
occupied.bind(("127.0.0.1", 7000))
occupied.listen(1)
try:
monkeypatch.setattr(sys, "argv", [
"meshnet-node", "start",
"--tracker", "http://192.168.0.179:8081",
"--model", "Qwen/Qwen2.5-0.5B-Instruct",
"--host", "127.0.0.1",
])
monkeypatch.setattr(cli_mod, "_first_available_port", lambda *_args, **_kwargs: 7001)
monkeypatch.setattr(sys, "argv", [
"meshnet-node", "start",
"--tracker", "http://192.168.0.179:8081",
"--model", "Qwen/Qwen2.5-0.5B-Instruct",
"--host", "127.0.0.1",
])
with patch("meshnet_node.startup.run_startup", side_effect=fake_run_startup):
with patch("time.sleep", side_effect=KeyboardInterrupt):
try:
main()
except SystemExit as exc:
assert exc.code == 0
finally:
occupied.close()
with patch("meshnet_node.startup.run_startup", side_effect=fake_run_startup):
with patch("time.sleep", side_effect=KeyboardInterrupt):
try:
cli_mod.main()
except SystemExit as exc:
assert exc.code == 0
assert captured["port"] == 7001

396
tests/test_node_admission.py Normal file
View File

@@ -0,0 +1,396 @@
"""NCA-003: startup fails closed — no registration without a fresh matching proof.
Two layers are covered here:
* the gate itself (`meshnet_node.admission.admit`) — which reports admit a
selection, and which are refused as failed, stale, or about something else;
* `run_startup` — that a refused report means the tracker is never called, and
that an admitted one travels with the registration payload.
Torch is a stub in the dev venv, so the backend is faked by duck-typing
`TorchModelShard` (see `_FakeBackend`); the *production* validator still runs a
real `doctor` forward against it, so the fail-closed path is exercised without a
bypass. Tests that cannot supply an executable backend pass the explicit
test-safe validator from `meshnet_node.testing`.
"""
import base64
import struct
import time
import pytest
from meshnet_node.admission import (
REASON_BACKEND_MISMATCH,
REASON_MODEL_MISMATCH,
REASON_NO_REPORT,
REASON_NOT_PASSED,
REASON_RECIPE_MISMATCH,
REASON_SHARD_MISMATCH,
REASON_STALE,
AdmissionRequirement,
CapabilityAdmissionError,
CapabilityContext,
admit,
probe_capability,
)
from meshnet_node.capability import STATUS_FAILED, STATUS_SKIPPED
from meshnet_node.doctor import DoctorSelection
from meshnet_node.recipe_manifest import DEFAULT_RECIPE_ID, load_recipe_manifest
from meshnet_node.startup import run_startup
from meshnet_node.testing import capability_report_for, capability_stub
MODEL = "acme/opaque-model-7b"
class _FakeDevice:
def __init__(self, type_: str = "cpu"):
self.type = type_
class _FakeOutput:
def __init__(self, hidden_size: int, tokens: int = 4):
self.body = b"\x00" * (tokens * hidden_size * 2)
self.shape = [1, tokens, hidden_size]
self.attention_mask_header = _int64_header([[1] * tokens])
self.position_ids_header = _int64_header([list(range(tokens))])
def _int64_header(rows):
flat = [int(v) for row in rows for v in row]
raw = struct.pack(f"<{len(flat)}q", *flat)
return f"{len(rows)},{len(rows[0])}:{base64.b64encode(raw).decode('ascii')}"
class _FakeBackend:
"""Duck-types the parts of `TorchModelShard` the doctor probe touches."""
total_layers = 24
hidden_size = 8
def __init__(self, *, shard_start=0, shard_end=23, device="cpu", forward_error=None):
self.shard_start = shard_start
self.shard_end = shard_end
self.is_head = shard_start == 0
self.is_tail = shard_end == self.total_layers - 1
self.device = _FakeDevice(device)
self.model_id = MODEL
self._forward_error = forward_error
def encode_prompt(self, _prompt):
if self._forward_error:
raise self._forward_error
return _FakeOutput(self.hidden_size)
def forward_bytes(self, body, shape, attention_mask_header, position_ids_header, start_layer=None):
if self._forward_error:
raise self._forward_error
return _FakeOutput(self.hidden_size)
def _context(backend=None, *, model_id=MODEL, shard_start=0, shard_end=23, device="cpu"):
manifest = load_recipe_manifest()
return CapabilityContext(
backend=backend,
selection=DoctorSelection(
model_id=model_id,
shard_start=shard_start,
shard_end=shard_end,
quantization="bfloat16",
),
recipe=manifest.require(DEFAULT_RECIPE_ID),
manifest=manifest,
device=device,
)
# ---------------------------------------------------------------------------
# The gate: which reports admit a selection
# ---------------------------------------------------------------------------
def test_a_fresh_matching_passing_report_admits_the_selection():
"The proof covers exactly what is about to be advertised, so the node may register.\n\nTags: node, admission"
ctx = _context()
report = capability_report_for(ctx)
assert admit(AdmissionRequirement.for_context(ctx), report) is report
def test_a_missing_report_is_refused():
"No proof at all is the default state, and it must not register.\n\nTags: node, admission"
ctx = _context()
with pytest.raises(CapabilityAdmissionError) as exc:
admit(AdmissionRequirement.for_context(ctx), None)
assert exc.value.reason == REASON_NO_REPORT
@pytest.mark.parametrize("status", [STATUS_FAILED, STATUS_SKIPPED])
def test_a_report_that_did_not_pass_is_refused(status):
"A failed or skipped validation is evidence against admission, not for it.\n\nTags: node, admission"
ctx = _context()
report = capability_report_for(
ctx, status=status, diagnostics=["the shard forward returned no output"]
)
with pytest.raises(CapabilityAdmissionError) as exc:
admit(AdmissionRequirement.for_context(ctx), report)
assert exc.value.reason == REASON_NOT_PASSED
assert "the shard forward returned no output" in str(exc.value)
def test_a_passing_report_for_another_model_is_refused():
"A proof about one model says nothing about another — no reuse across models.\n\nTags: node, admission"
ctx = _context()
report = capability_report_for(ctx, model_id="other/model-1b")
with pytest.raises(CapabilityAdmissionError) as exc:
admit(AdmissionRequirement.for_context(ctx), report)
assert exc.value.reason == REASON_MODEL_MISMATCH
assert "other/model-1b" in str(exc.value)
def test_a_passing_report_for_another_shard_range_is_refused():
"Layers 0–11 running is no proof that layers 12–23 fit.\n\nTags: node, admission"
ctx = _context(shard_start=12, shard_end=23)
report = capability_report_for(ctx, shard_start=0, shard_end=11)
with pytest.raises(CapabilityAdmissionError) as exc:
admit(AdmissionRequirement.for_context(ctx), report)
assert exc.value.reason == REASON_SHARD_MISMATCH
def test_a_passing_report_for_another_recipe_version_is_refused():
"A recipe's execution params changed, so its old proof no longer applies.\n\nTags: node, admission"
ctx = _context()
report = capability_report_for(ctx, recipe_version="0")
with pytest.raises(CapabilityAdmissionError) as exc:
admit(AdmissionRequirement.for_context(ctx), report)
assert exc.value.reason == REASON_RECIPE_MISMATCH
def test_a_passing_report_from_another_backend_or_device_is_refused():
"A CUDA proof does not admit a node that would serve the shard on CPU.\n\nTags: node, admission"
ctx = _context(device="cpu")
report = capability_report_for(ctx, device="cuda")
with pytest.raises(CapabilityAdmissionError) as exc:
admit(AdmissionRequirement.for_context(ctx), report)
assert exc.value.reason == REASON_BACKEND_MISMATCH
other_backend = capability_report_for(ctx, backend_id="some-other-runtime")
with pytest.raises(CapabilityAdmissionError) as exc:
admit(AdmissionRequirement.for_context(ctx), other_backend)
assert exc.value.reason == REASON_BACKEND_MISMATCH
def test_a_report_older_than_the_freshness_window_is_refused():
"Hardware, drivers and weights move; an old proof is not a current one.\n\nTags: node, admission"
ctx = _context()
requirement = AdmissionRequirement.for_context(ctx, max_age_seconds=900)
report = capability_report_for(ctx, age_seconds=901)
with pytest.raises(CapabilityAdmissionError) as exc:
admit(requirement, report)
assert exc.value.reason == REASON_STALE
still_fresh = capability_report_for(ctx, age_seconds=899)
assert admit(requirement, still_fresh) is still_fresh
def test_a_future_dated_report_is_refused():
"A report from the future is a broken clock, not a fresh proof.\n\nTags: node, admission"
ctx = _context()
report = capability_report_for(ctx, validated_at=time.time() + 3600)
with pytest.raises(CapabilityAdmissionError) as exc:
admit(AdmissionRequirement.for_context(ctx), report)
assert exc.value.reason == REASON_STALE
# ---------------------------------------------------------------------------
# The production validator: a real forward through the loaded backend
# ---------------------------------------------------------------------------
def test_the_production_validator_proves_a_working_backend_with_a_real_forward():
"probe_capability runs the doctor's bounded forward on the backend that would serve.\n\nTags: node, admission"
ctx = _context(_FakeBackend())
report = probe_capability(ctx)
assert report.passed
assert admit(AdmissionRequirement.for_context(ctx), report) is report
def test_the_production_validator_fails_a_backend_that_cannot_execute():
"A shard that loads but cannot run a forward yields a failed report, not a pass.\n\nTags: node, admission"
ctx = _context(_FakeBackend(forward_error=RuntimeError("CUDA out of memory")))
report = probe_capability(ctx)
assert not report.passed
with pytest.raises(CapabilityAdmissionError) as exc:
admit(AdmissionRequirement.for_context(ctx), report)
assert exc.value.reason == REASON_NOT_PASSED
def test_the_production_validator_fails_a_node_with_no_backend_at_all():
"A server with no model backend cannot prove anything, so it is not admitted.\n\nTags: node, admission"
ctx = _context(None)
assert not probe_capability(ctx).passed
# ---------------------------------------------------------------------------
# run_startup: a refused report means the tracker is never called
# ---------------------------------------------------------------------------
class _FakeTorchNodeServer:
started = False
def __init__(self, **kwargs):
self.kwargs = kwargs
self.backend = kwargs.pop("_backend", None) or _FakeBackend()
def start(self):
type(self).started = True
return 7099
def stop(self):
pass
@pytest.fixture
def startup_env(monkeypatch):
"""Fake hardware, wallet and tracker; records registrations *for this model*.
Heartbeat threads that other tests leave running are daemon threads that
outlive their test and re-register through this same patched `_post_json`, so
a plain call log would be polluted by whatever ran before. Only posts naming
this test's model can have come from this test's node.
"""
import meshnet_node.startup as startup_mod
posted: list[tuple[str, dict]] = []
_FakeTorchNodeServer.started = False
def _record(url, payload, timeout=10.0):
if url.endswith("/v1/nodes/register") and payload.get("hf_repo") == MODEL:
posted.append((url, payload))
return {"node_id": "node-nca"}
monkeypatch.setattr(
startup_mod,
"detect_hardware",
lambda: {"device": "cpu", "gpu_name": None, "vram_mb": 0, "ram_mb": 16384},
)
monkeypatch.setattr(
startup_mod, "benchmark_throughput_checked", lambda _device: (10.0, True, None)
)
monkeypatch.setattr(startup_mod, "TorchNodeServer", _FakeTorchNodeServer)
monkeypatch.setattr(
startup_mod, "load_or_create_wallet", lambda **_kw: (b"", b"", "wallet-nca")
)
monkeypatch.setattr(
startup_mod, "_get_json", lambda _url, timeout=10.0: {"relay_url": None, "nodes": []}
)
monkeypatch.setattr(startup_mod, "_post_json", _record)
monkeypatch.setattr(startup_mod, "_start_heartbeat", lambda *a, **kw: None)
return posted
def _start(**kwargs):
return run_startup(
tracker_url="http://127.0.0.1:8080",
model_id=MODEL,
shard_start=0,
shard_end=23,
**kwargs,
)
def test_backend_validation_failure_registers_nothing(startup_env, monkeypatch):
"A shard that cannot run a forward must never reach /v1/nodes/register.\n\nTags: node, admission, startup"
import meshnet_node.startup as startup_mod
broken = _FakeBackend(forward_error=RuntimeError("CUDA out of memory"))
monkeypatch.setattr(
startup_mod,
"TorchNodeServer",
lambda **kw: _FakeTorchNodeServer(_backend=broken, **kw),
)
with pytest.raises(CapabilityAdmissionError) as exc:
_start() # no validator: the production real-forward path
assert exc.value.reason == REASON_NOT_PASSED
assert startup_env == [], "the tracker was called despite a failed validation"
assert not _FakeTorchNodeServer.started, "a failed node still opened an endpoint"
def test_a_report_for_a_different_model_cannot_be_reused_to_register(startup_env):
"A success for one model must not admit another — the shape of a replayed proof.\n\nTags: node, admission, startup"
with pytest.raises(CapabilityAdmissionError) as exc:
_start(capability_validator=capability_stub(model_id="other/model-1b"))
assert exc.value.reason == REASON_MODEL_MISMATCH
assert startup_env == []
def test_a_stale_report_cannot_be_reused_to_register(startup_env):
"An aged-out proof is refused before the node advertises itself.\n\nTags: node, admission, startup"
with pytest.raises(CapabilityAdmissionError) as exc:
_start(capability_validator=capability_stub(age_seconds=86_400))
assert exc.value.reason == REASON_STALE
assert startup_env == []
def test_a_matching_passing_report_registers_and_travels_with_the_payload(startup_env):
"Registration carries the proof for exactly the model/shard/recipe it advertises.\n\nTags: node, admission, startup"
node = _start() # production validator against a working fake backend
node.stop()
assert len(startup_env) == 1
url, payload = startup_env[0]
assert url.endswith("/v1/nodes/register")
report = payload["capability_report"]
assert report["status"] == "passed"
assert report["model"]["model_id"] == MODEL
assert (report["shard"]["start"], report["shard"]["end"]) == (0, 23)
assert report["recipe"]["recipe_id"] == DEFAULT_RECIPE_ID
assert report["backend"]["device"] == "cpu"
def test_the_served_backend_is_loaded_with_the_recipe_that_was_validated(startup_env):
"The recipe named in the report is the one the serving backend actually ran.\n\nTags: node, admission, startup"
node = _start(recipe_id="eager-attention")
node.stop()
assert node.kwargs["recipe_params"] == {"attn_implementation": "eager"}
report = startup_env[0][1]["capability_report"]
assert report["recipe"]["recipe_id"] == "eager-attention"
def test_an_unknown_recipe_fails_before_any_weights_are_loaded(startup_env):
"A typo'd recipe id is caught at resolution, not after a multi-minute load.\n\nTags: node, admission, startup"
from meshnet_node.recipe_manifest import RecipeManifestError
with pytest.raises(RecipeManifestError):
_start(recipe_id="does-not-exist")
assert startup_env == []
assert not _FakeTorchNodeServer.started

644
tests/test_node_doctor.py Normal file
View File

@@ -0,0 +1,644 @@
"""NCA-002 tests for `meshnet-node doctor`.
The unit tests inject a fake backend, so none of them download a model, import
Torch, or need a GPU. The one test that runs a real model is `integration`-marked
and takes its model identity from the environment — it has no model default, on
purpose: the doctor is model-agnostic and so is its test.
"""
import base64
import json
import os
import struct
from pathlib import Path
import pytest
from meshnet_node import doctor
from meshnet_node.capability import STATUS_FAILED, STATUS_PASSED, CapabilityReport
from meshnet_node.doctor import (
CATEGORY_FORWARD_FAILED,
CATEGORY_INSUFFICIENT_MEMORY,
CATEGORY_INVALID_SHARD,
CATEGORY_MISSING_DEPENDENCY,
CATEGORY_NO_MODEL,
CATEGORY_UNSUPPORTED_RECIPE,
PROBE_TOKENS,
DoctorError,
DoctorSelection,
build_probe_input,
classify_failure,
probe_forward,
render_result,
resolve_selection,
run_doctor,
select_recipes,
write_reports,
)
from meshnet_node.model_backend import (
InsufficientVRAMError,
MissingModelDependencyError,
UnsupportedRecipeParam,
validate_recipe_params,
)
from meshnet_node.recipe_manifest import parse_recipe_manifest
# Deliberately not a model this project ships against: nothing here may special-case it.
FIXTURE_MODEL = "acme-labs/Widget-9000-Instruct"
MANIFEST = parse_recipe_manifest(
{
"schema_version": 1,
"catalogue_version": "test-1",
"recipes": [
{"id": "baseline", "version": "1", "backend_id": "torch-transformers"},
{
"id": "stateless",
"version": "2",
"backend_id": "torch-transformers",
"params": {"use_cache": False},
},
],
},
source="<test manifest>",
)
class _Payload:
"""Stands in for model_backend.TensorPayload."""
def __init__(self, body: bytes, shape: list[int]) -> None:
self.body = body
self.shape = shape
self.attention_mask_header = None
self.position_ids_header = None
class _TailToken:
"""Stands in for model_backend.TailTokenResult."""
def __init__(self, token_id: int = 7) -> None:
self.token_id = token_id
self.text = "ok"
class _Device:
def __init__(self, type_: str = "cpu") -> None:
self.type = type_
class _FakeBackend:
"""A backend that loads but records exactly how it was driven."""
hidden_size = 8
def __init__(
self,
*,
is_head: bool = True,
is_tail: bool = False,
shard_start: int = 0,
shard_end: int = 3,
forward_error: Exception | None = None,
) -> None:
self.model_id = FIXTURE_MODEL
self.is_head = is_head
self.is_tail = is_tail
self.shard_start = shard_start
self.shard_end = shard_end
self.device = _Device("cpu")
self.forward_error = forward_error
self.encoded_prompts: list[str] = []
self.forwards: list[dict] = []
def encode_prompt(self, prompt: str):
if self.forward_error is not None:
raise self.forward_error
self.encoded_prompts.append(prompt)
return _Payload(b"\x00" * (PROBE_TOKENS * self.hidden_size * 2),
[1, PROBE_TOKENS, self.hidden_size])
def forward_bytes(
self,
body,
shape,
attention_mask_header,
position_ids_header,
start_layer=None,
**kwargs,
):
if self.forward_error is not None:
raise self.forward_error
self.forwards.append(
{
"body_len": len(body),
"shape": shape,
"start_layer": start_layer,
"attention_mask_header": attention_mask_header,
"position_ids_header": position_ids_header,
}
)
if self.is_tail:
return _TailToken()
return _Payload(body, shape)
def _loader(backend=None, *, error: Exception | None = None):
"""A load_backend stub that records the (selection, recipe) pairs it saw."""
calls: list[tuple[DoctorSelection, object]] = []
def load(selection, recipe):
calls.append((selection, recipe))
if error is not None:
raise error
return backend if backend is not None else _FakeBackend()
load.calls = calls # type: ignore[attr-defined]
return load
def _selection(**overrides) -> DoctorSelection:
kwargs = dict(model_id=FIXTURE_MODEL, shard_start=0, shard_end=3)
kwargs.update(overrides)
return DoctorSelection(**kwargs)
# --- selection resolves the same as startup ---------------------------------
def test_resolve_selection_uses_the_configured_repo_shard_and_quantization():
selection = resolve_selection(
{
"model_hf_repo": FIXTURE_MODEL,
"model_name": "Widget-9000-Instruct",
"shard_start": 4,
"shard_end": 11,
"quantization": "bf16", # startup normalizes this to bfloat16
"download_dir": "/models",
}
)
assert selection.model_id == FIXTURE_MODEL
assert (selection.shard_start, selection.shard_end) == (4, 11)
assert selection.quantization == "bfloat16"
assert selection.cache_dir == Path("/models")
def test_resolve_selection_defaults_to_the_whole_model_like_startup():
"""With no pinned shard, startup serves layers 0..n-1 — so doctor validates that."""
seen: list[tuple[str, Path | None]] = []
def detect(model_id, cache_dir):
seen.append((model_id, cache_dir))
return 24
selection = resolve_selection(
{"model_hf_repo": FIXTURE_MODEL}, detect_layers=detect
)
assert (selection.shard_start, selection.shard_end) == (0, 23)
assert seen == [(FIXTURE_MODEL, None)]
def test_resolve_selection_without_a_model_is_actionable():
with pytest.raises(DoctorError) as exc:
resolve_selection({"model_hf_repo": "", "model_name": ""})
assert exc.value.category == CATEGORY_NO_MODEL
assert "--model" in exc.value.hint
def test_resolve_selection_rejects_an_inverted_shard_range():
with pytest.raises(DoctorError) as exc:
resolve_selection(
{"model_hf_repo": FIXTURE_MODEL, "shard_start": 9, "shard_end": 2}
)
assert exc.value.category == CATEGORY_INVALID_SHARD
def test_resolve_selection_reports_an_unreadable_model_config():
with pytest.raises(DoctorError) as exc:
resolve_selection(
{"model_hf_repo": FIXTURE_MODEL}, detect_layers=lambda *_: None
)
assert exc.value.category == doctor.CATEGORY_MODEL_UNAVAILABLE
assert "--shard-start" in str(exc.value)
# --- the bounded real forward ------------------------------------------------
def test_a_pass_requires_a_real_forward_through_the_selected_shard():
"""Hardware being fine is not the bar: the shard itself has to execute."""
backend = _FakeBackend(is_head=True)
result = run_doctor(
_selection(), manifest=MANIFEST, load_backend=_loader(backend), now=lambda: 1000.0
)
assert result.passed
assert backend.encoded_prompts == [doctor.PROBE_PROMPT]
report = result.reports[0]
assert report.status == STATUS_PASSED
assert report.model.model_id == FIXTURE_MODEL
assert (report.shard.start, report.shard.end) == (0, 3)
def test_a_backend_that_loads_but_cannot_forward_never_passes():
"""The regression this story exists for: a load is not a validation."""
backend = _FakeBackend(forward_error=RuntimeError("kernel exploded"))
result = run_doctor(
_selection(), manifest=MANIFEST, load_backend=_loader(backend), now=lambda: 1.0
)
assert not result.passed
assert result.exit_code == 1
report = result.reports[0]
assert report.status == STATUS_FAILED
assert result.results[0].category == CATEGORY_FORWARD_FAILED
assert any("kernel exploded" in d for d in report.diagnostics)
def test_a_mid_shard_is_probed_with_peer_shaped_hidden_states():
backend = _FakeBackend(is_head=False, shard_start=4, shard_end=7)
detail = probe_forward(backend)
assert detail["probe"] == "hidden-states"
assert backend.encoded_prompts == []
forward = backend.forwards[0]
assert forward["shape"] == [1, PROBE_TOKENS, backend.hidden_size]
# bfloat16 == 2 bytes per element, and the probe stays bounded to PROBE_TOKENS.
assert forward["body_len"] == PROBE_TOKENS * backend.hidden_size * 2
assert forward["start_layer"] == 4
def test_a_head_and_tail_shard_also_decodes_so_the_lm_head_is_covered():
backend = _FakeBackend(is_head=True, is_tail=True, shard_end=5)
detail = probe_forward(backend)
assert detail["probe"] == "prompt+decode"
assert detail["output"] == "token"
# Re-entering above the last layer decodes without re-running any layer.
assert backend.forwards[0]["start_layer"] == 6
def test_a_tail_shard_that_decodes_a_token_passes():
backend = _FakeBackend(is_head=False, is_tail=True, shard_start=8, shard_end=11)
detail = probe_forward(backend)
assert detail == {
"probe": "hidden-states",
"tokens": PROBE_TOKENS,
"output": "token",
"token_id": 7,
}
def test_an_empty_forward_result_is_a_failure_not_a_pass():
backend = _FakeBackend(is_head=False)
backend.forward_bytes = lambda *a, **k: _Payload(b"", []) # type: ignore[assignment]
with pytest.raises(DoctorError) as exc:
probe_forward(backend)
assert exc.value.category == CATEGORY_FORWARD_FAILED
def test_a_backend_with_no_hidden_size_cannot_be_probed():
with pytest.raises(DoctorError) as exc:
build_probe_input(0)
assert exc.value.category == CATEGORY_FORWARD_FAILED
def test_probe_headers_decode_as_int64_tensors():
probe = build_probe_input(hidden_size=8, tokens=3)
shape, encoded = probe.position_ids_header.split(":", 1)
raw = base64.b64decode(encoded)
assert shape == "1,3"
assert list(struct.unpack("<3q", raw)) == [0, 1, 2]
# --- recipes -----------------------------------------------------------------
def test_the_default_run_validates_only_the_selected_recipe():
"""Onboarding must not pay to validate recipes the node was not asked to serve."""
load = _loader()
result = run_doctor(_selection(), manifest=MANIFEST, load_backend=load)
assert [r.recipe.id for r in result.results] == ["baseline"]
assert len(load.calls) == 1
def test_all_recipes_is_explicit_and_validates_every_recipe():
load = _loader()
result = run_doctor(
_selection(), manifest=MANIFEST, load_backend=load, all_recipes=True
)
assert [r.recipe.id for r in result.results] == ["baseline", "stateless"]
assert len(load.calls) == 2
assert result.passed
def test_each_recipe_reaches_the_backend_that_runs_it():
"""A recipe that never reaches the loader was not really validated."""
load = _loader()
run_doctor(_selection(), manifest=MANIFEST, load_backend=load, all_recipes=True)
params = [recipe.params for _, recipe in load.calls]
assert params == [{}, {"use_cache": False}]
def test_an_unknown_recipe_names_the_ones_that_exist():
with pytest.raises(DoctorError) as exc:
select_recipes(MANIFEST, recipe_id="does-not-exist")
assert exc.value.category == CATEGORY_UNSUPPORTED_RECIPE
assert "baseline" in str(exc.value)
def test_recipe_and_all_recipes_are_mutually_exclusive():
with pytest.raises(DoctorError):
select_recipes(MANIFEST, recipe_id="baseline", all_recipes=True)
def test_a_recipe_the_backend_cannot_apply_is_a_failure_not_a_silent_pass():
validate_recipe_params({"use_cache": False, "attn_implementation": "eager"})
with pytest.raises(UnsupportedRecipeParam) as exc:
validate_recipe_params({"sparkle_mode": True})
assert "sparkle_mode" in str(exc.value)
assert classify_failure(exc.value) == CATEGORY_UNSUPPORTED_RECIPE
def test_the_shipped_recipes_are_all_applicable_by_the_backend():
"""recipes.json and the backend's supported params must not drift apart."""
from meshnet_node.recipe_manifest import load_recipe_manifest
for recipe in load_recipe_manifest().recipes:
validate_recipe_params(recipe.params)
# --- failure reporting -------------------------------------------------------
@pytest.mark.parametrize(
"exc, category",
[
(MissingModelDependencyError("no torch"), CATEGORY_MISSING_DEPENDENCY),
(InsufficientVRAMError("too big"), CATEGORY_INSUFFICIENT_MEMORY),
(UnsupportedRecipeParam("nope"), CATEGORY_UNSUPPORTED_RECIPE),
(ValueError("shard_end 99 exceeds last layer index 23"), CATEGORY_INVALID_SHARD),
(FileNotFoundError("config.json"), doctor.CATEGORY_MODEL_UNAVAILABLE),
(RuntimeError("something else"), doctor.CATEGORY_LOAD_FAILED),
],
)
def test_load_failures_are_classified_into_actionable_categories(exc, category):
result = run_doctor(
_selection(), manifest=MANIFEST, load_backend=_loader(error=exc)
)
assert not result.passed
item = result.results[0]
assert item.category == category
assert item.hint # every category tells the operator what to do next
assert item.report.status == STATUS_FAILED
def test_a_failure_report_carries_the_hint_and_no_traceback():
result = run_doctor(
_selection(),
manifest=MANIFEST,
load_backend=_loader(error=InsufficientVRAMError("insufficient VRAM to load")),
)
diagnostics = " ".join(result.reports[0].diagnostics)
assert "insufficient VRAM to load" in diagnostics
assert "--shard-start" in diagnostics # the actionable next step
assert "Traceback" not in diagnostics
assert ".py" not in diagnostics # no file/line noise from a stack
def test_a_failure_report_still_identifies_what_was_being_validated():
"""NCA-003 refuses to register without a matching report — including a failed one."""
result = run_doctor(
_selection(shard_start=4, shard_end=9, quantization="int8"),
manifest=MANIFEST,
load_backend=_loader(error=RuntimeError("boom")),
now=lambda: 4242.0,
)
report = result.reports[0]
assert report.identity_key() == (
FIXTURE_MODEL, 4, 9, "baseline", "1", "torch-transformers", "unknown",
)
assert report.validated_at == 4242.0
assert report.recipe.catalogue_version == "test-1"
def test_the_report_records_the_device_the_forward_actually_ran_on():
result = run_doctor(
_selection(), manifest=MANIFEST, load_backend=_loader(_FakeBackend())
)
assert result.reports[0].backend.device == "cpu"
assert result.reports[0].backend.backend_id == "torch-transformers"
def test_reports_round_trip_through_the_written_json(tmp_path):
result = run_doctor(
_selection(), manifest=MANIFEST, load_backend=_loader(), all_recipes=True
)
path = write_reports(result.reports, tmp_path / "nested" / "capability.json")
payload = json.loads(path.read_text())
assert [CapabilityReport.from_dict(d).recipe.recipe_id for d in payload] == [
"baseline",
"stateless",
]
def test_a_single_report_is_written_as_one_object():
"""One selected recipe writes one report — the shape NCA-003 will read."""
import tempfile
result = run_doctor(_selection(), manifest=MANIFEST, load_backend=_loader())
with tempfile.TemporaryDirectory() as tmp:
path = write_reports(result.reports, Path(tmp) / "capability.json")
report = CapabilityReport.from_json(path.read_text())
assert report.passed
def test_the_summary_tells_a_failing_operator_what_to_fix():
result = run_doctor(
_selection(),
manifest=MANIFEST,
load_backend=_loader(error=MissingModelDependencyError("torch is not installed")),
)
text = render_result(result, report_path=Path("/tmp/capability.json"))
assert "FAIL" in text
assert CATEGORY_MISSING_DEPENDENCY in text
assert "torch is not installed" in text
assert "/tmp/capability.json" in text
assert "Traceback" not in text
def test_the_summary_names_the_shard_that_passed():
result = run_doctor(_selection(), manifest=MANIFEST, load_backend=_loader())
text = render_result(result)
assert "PASS" in text
assert FIXTURE_MODEL in text
assert "layers 0–3" in text
# --- the CLI wiring ----------------------------------------------------------
def _run_cli(monkeypatch, argv, backend=None, error=None):
"""Drive `meshnet-node doctor` end to end with an injected backend."""
import sys
from meshnet_node import cli, config
monkeypatch.setattr(
config, "load_config", lambda *a, **k: {
"model_hf_repo": FIXTURE_MODEL,
"shard_start": 0,
"shard_end": 3,
"quantization": "auto",
}
)
monkeypatch.setattr(
doctor, "default_load_backend", _loader(backend, error=error)
)
monkeypatch.setattr(doctor, "load_recipe_manifest", lambda *a, **k: MANIFEST)
monkeypatch.setattr(sys, "argv", ["meshnet-node", *argv])
with pytest.raises(SystemExit) as exit_info:
cli.main()
return exit_info.value.code
def test_cli_doctor_exits_zero_and_writes_a_passing_report(monkeypatch, capsys, tmp_path):
report = tmp_path / "capability.json"
code = _run_cli(monkeypatch, ["doctor", "--report", str(report)], backend=_FakeBackend())
assert code == 0
assert capsys.readouterr().out.count("PASS") == 1
assert CapabilityReport.from_json(report.read_text()).passed
def test_cli_doctor_exits_non_zero_and_writes_the_failed_report(monkeypatch, capsys, tmp_path):
report = tmp_path / "capability.json"
code = _run_cli(
monkeypatch,
["doctor", "--report", str(report)],
error=InsufficientVRAMError("insufficient VRAM to load 24 layers"),
)
out = capsys.readouterr().out
assert code == 1
assert "FAIL" in out
assert CATEGORY_INSUFFICIENT_MEMORY in out
assert "Traceback" not in out # no raw traceback by default
assert CapabilityReport.from_json(report.read_text()).status == STATUS_FAILED
def test_cli_doctor_all_recipes_is_opt_in(monkeypatch, capsys, tmp_path):
report = tmp_path / "capability.json"
code = _run_cli(
monkeypatch,
["doctor", "--all-recipes", "--report", str(report)],
backend=_FakeBackend(),
)
assert code == 0
assert capsys.readouterr().out.count("PASS") == 2
assert len(json.loads(report.read_text())) == 2
def test_cli_doctor_json_prints_the_capability_report(monkeypatch, capsys, tmp_path):
code = _run_cli(
monkeypatch,
["doctor", "--json", "--report", str(tmp_path / "c.json")],
backend=_FakeBackend(),
)
payload = json.loads(capsys.readouterr().out)
assert code == 0
assert payload[0]["model"]["model_id"] == FIXTURE_MODEL
def test_cli_doctor_flags_select_what_is_validated(monkeypatch, capsys, tmp_path):
"""`doctor --shard-start/--shard-end` validates the shard startup would load."""
report = tmp_path / "capability.json"
code = _run_cli(
monkeypatch,
["doctor", "--shard-start", "2", "--shard-end", "5", "--report", str(report)],
backend=_FakeBackend(),
)
written = CapabilityReport.from_json(report.read_text())
assert code == 0
assert (written.shard.start, written.shard.end) == (2, 5)
# --- the real-model smoke test ----------------------------------------------
# Model identity comes from the environment; there is no default, so this test
# never smuggles a vendor-specific assumption into the suite.
DOCTOR_MODEL = os.environ.get("MESHNET_DOCTOR_MODEL")
DOCTOR_SHARD_START = int(os.environ.get("MESHNET_DOCTOR_SHARD_START", "0"))
DOCTOR_SHARD_END = os.environ.get("MESHNET_DOCTOR_SHARD_END")
@pytest.mark.integration
@pytest.mark.skipif(
not DOCTOR_MODEL,
reason="set MESHNET_DOCTOR_MODEL (and optionally MESHNET_DOCTOR_SHARD_START/END) to run",
)
def test_doctor_smoke_runs_a_real_forward_on_a_real_model(tmp_path):
cfg = {
"model_hf_repo": DOCTOR_MODEL,
"quantization": os.environ.get("MESHNET_DOCTOR_QUANTIZATION", "auto"),
"download_dir": os.environ.get("MESHNET_DOWNLOAD_DIR") or None,
"shard_start": DOCTOR_SHARD_START,
"shard_end": int(DOCTOR_SHARD_END) if DOCTOR_SHARD_END else None,
"force_cpu": os.environ.get("MESHNET_DOCTOR_CPU") == "1",
}
selection = resolve_selection(cfg)
result = run_doctor(selection)
report = result.reports[0]
assert result.passed, f"doctor failed: {report.diagnostics}"
assert report.status == STATUS_PASSED
assert report.model.model_id == DOCTOR_MODEL
assert report.duration_ms > 0
assert report.model.config_fingerprint.startswith("sha256:")
path = write_reports(result.reports, tmp_path / "capability.json")
assert CapabilityReport.from_json(path.read_text()).passed

51
tests/test_node_startup.py
View File

@@ -16,14 +16,20 @@ import pytest
from meshnet_node.downloader import download_shard, write_shard_archive
from meshnet_node.hardware import detect_hardware, benchmark_throughput
from meshnet_node.cli import _resolve_model_flags
from meshnet_node.startup import (
_configure_torch_threads,
_hardware_label,
_infer_relay_url_from_tracker,
_memory_budget,
_probationary_status_line,
_tracker_http_error_message,
run_startup,
)
# Startup admits a node only on a capability report from a real forward, which a
# fake backend cannot perform. These tests say so explicitly rather than bypassing
# admission; the fail-closed path itself is covered in tests/test_node_admission.py.
from meshnet_node.testing import assume_capability
from meshnet_node.wallet import _b58encode, load_or_create_wallet
from meshnet_contracts import LocalSolanaContracts
from meshnet_tracker.server import TrackerServer
@@ -34,6 +40,30 @@ from meshnet_tracker.server import TrackerServer
# ---------------------------------------------------------------------------
def test_short_curated_model_alias_resolves_to_huggingface_repo():
"""A known short model name must use the explicit Hugging Face path."""
assert _resolve_model_flags("Qwen3.6-27B", None) == (
"Qwen3.6-27B",
"Qwen/Qwen3.6-27B",
)
def test_tracker_http_error_message_includes_rejection_body():
"""HTTP responses are tracker rejections, not connectivity failures."""
error = urllib.error.HTTPError(
"https://tracker.example/v1/nodes/assign",
404,
"Not Found",
{},
io.BytesIO(b'{"error": "unknown model preset: \'missing-model\'"}'),
)
assert _tracker_http_error_message(error) == (
"Tracker rejected shard assignment (HTTP 404): "
"unknown model preset: 'missing-model'"
)
def test_with_forced_cpu_overrides_device_but_keeps_gpu_inventory():
"--cpu should register and run on CPU while preserving detected GPU metadata.\n\nTags: node, startup"
import meshnet_node.hardware as hardware_mod
@@ -307,6 +337,7 @@ def test_benchmark_throughput_is_registered_in_payload(monkeypatch, tmp_path):
wallet_path=tmp_path / "wallet.json",
torch_threads=8,
torch_interop_threads=1,
capability_validator=assume_capability,
)
node.stop()
@@ -368,6 +399,7 @@ def test_real_model_startup_passes_download_dir_and_kimi_metadata(monkeypatch, t
shard_end=60,
wallet_path=tmp_path / "wallet.json",
cache_dir=cache_dir,
capability_validator=assume_capability,
)
node.stop()
@@ -422,6 +454,7 @@ def test_cuda_benchmark_failure_is_registered_for_inventory_only_gpu(monkeypatch
shard_start=0,
shard_end=23,
wallet_path=tmp_path / "wallet.json",
capability_validator=assume_capability,
)
node.stop()
@@ -1138,6 +1171,7 @@ def test_public_https_tracker_infers_relay_when_network_map_omits_relay_url(
model_id="Qwen/Qwen2.5-0.5B-Instruct",
advertise_host="172.29.104.23",
wallet_path=tmp_path / "wallet.json",
capability_validator=assume_capability,
)
try:
pass
@@ -1190,6 +1224,7 @@ def test_real_model_startup_summary_shows_total_layers(tmp_path, monkeypatch, ca
vram_mb_override=6144,
max_loaded_shards=2,
wallet_path=tmp_path / "wallet.json",
capability_validator=assume_capability,
)
assert node.backend.total_layers == 24
@@ -1249,6 +1284,7 @@ def test_real_model_startup_autodetects_cpu_memory_budget_and_logs_shard_budget(
shard_start=0,
shard_end=23,
wallet_path=tmp_path / "wallet.json",
capability_validator=assume_capability,
)
try:
pass
@@ -1333,6 +1369,7 @@ def test_public_tracker_model_node_registers_relay_metadata_from_tracker_url_onl
tracker_url=tracker_url,
model_id="Qwen/Qwen2.5-0.5B-Instruct",
wallet_path=tmp_path / "wallet.json",
capability_validator=assume_capability,
)
try:
network_map = _get_json(f"{tracker_url}/v1/network/map")
@@ -1418,6 +1455,7 @@ def test_public_tracker_relay_suppresses_virtual_ip_warning(
model_id="Qwen/Qwen2.5-0.5B-Instruct",
advertise_host="172.29.104.23",
wallet_path=tmp_path / "wallet.json",
capability_validator=assume_capability,
)
try:
network_map = _get_json(f"{tracker_url}/v1/network/map")
@@ -1497,6 +1535,7 @@ def test_later_node_auto_joins_existing_public_hf_model_with_only_tracker_url(
tracker_url=tracker_url,
advertise_host="203.0.113.21",
wallet_path=tmp_path / "wallet.json",
capability_validator=assume_capability,
)
try:
route_resp = _get_json(
@@ -1581,6 +1620,7 @@ def test_later_node_auto_joins_redundant_copy_when_model_is_fully_covered(
tracker_url=tracker_url,
advertise_host="203.0.113.32",
wallet_path=tmp_path / "wallet.json",
capability_validator=assume_capability,
)
try:
assert captured["model_id"] == "Qwen/Qwen2.5-0.5B-Instruct"
@@ -1611,6 +1651,7 @@ def test_full_startup_sequence(tmp_path):
model="stub-model",
wallet_path=wallet_path,
cache_dir=cache_dir,
capability_validator=assume_capability,
)
try:
# Wallet was created on disk
@@ -1673,6 +1714,7 @@ def test_preset_model_startup_starts_heartbeat(tmp_path, monkeypatch):
model="stub-model",
wallet_path=tmp_path / "wallet.json",
cache_dir=tmp_path / "shards",
capability_validator=assume_capability,
)
try:
assert len(heartbeat_calls) == 1
@@ -1713,6 +1755,7 @@ def test_preset_model_startup_honors_pinned_shard_range(tmp_path, monkeypatch):
shard_end=5,
wallet_path=tmp_path / "wallet.json",
cache_dir=tmp_path / "shards",
capability_validator=assume_capability,
)
try:
assert len(heartbeat_calls) == 1
@@ -1756,6 +1799,7 @@ def test_preset_startup_rejects_pinned_shard_above_memory_budget(tmp_path, monke
shard_end=39,
wallet_path=tmp_path / "wallet.json",
cache_dir=tmp_path / "shards",
capability_validator=assume_capability,
)
finally:
tracker.stop()
@@ -1809,6 +1853,7 @@ def test_network_auto_join_clips_oversized_cpu_assignment(tmp_path, monkeypatch,
tracker_url="http://127.0.0.1:8080",
wallet_path=tmp_path / "wallet.json",
tracker_source_disabled=True,
capability_validator=assume_capability,
)
try:
assert torch_calls[0]["shard_start"] == 0
@@ -1869,6 +1914,7 @@ def test_preset_model_with_hf_repo_loads_torch_backend(tmp_path, monkeypatch, ca
model="tiny-llama",
wallet_path=tmp_path / "wallet.json",
cache_dir=tmp_path / "node-shards",
capability_validator=assume_capability,
)
try:
assert len(torch_calls) == 1
@@ -1946,6 +1992,7 @@ def test_torch_startup_retries_registration_when_tracker_unreachable(
tracker_url=tracker_url,
model_id="Qwen/Qwen2.5-0.5B-Instruct",
wallet_path=tmp_path / "wallet.json",
capability_validator=assume_capability,
)
try:
assert register_calls["count"] == 1
@@ -2026,6 +2073,7 @@ def test_real_model_startup_registers_downloaded_inventory_without_checksum(
model="tiny-llama",
wallet_path=tmp_path / "wallet.json",
cache_dir=tmp_path / "node-shards",
capability_validator=assume_capability,
)
try:
assert len(hf_calls) == 1
@@ -2316,6 +2364,7 @@ def test_startup_cpu_fallback(tmp_path, monkeypatch):
model="stub-model",
wallet_path=tmp_path / "wallet.json",
cache_dir=tmp_path / "shards",
capability_validator=assume_capability,
)
try:
# Node is running even on CPU
@@ -2405,7 +2454,9 @@ def test_cli_loads_local_env_before_config_defaults(tmp_path, monkeypatch):
from meshnet_node import config as config_mod
monkeypatch.delenv("MESHNET_DOWNLOAD_DIR", raising=False)
monkeypatch.delenv("HF_TOKEN", raising=False)
monkeypatch.chdir(tmp_path)
cli_mod = importlib.reload(cli_mod)
(tmp_path / ".env").write_text(
"MESHNET_DOWNLOAD_DIR=/run/media/popov/DATA/llm/safetensor/models\n"
"HF_TOKEN=hf_test_token\n"

93
tests/test_prefill_backpressure.py Normal file
View File

@@ -0,0 +1,93 @@
"""Coverage for bounded, ordered prefill transfer."""
from threading import Event
import pytest
from meshnet_gateway.prefill_backpressure import (
BoundedPrefillSender,
DEFAULT_PREFILL_CHUNK_TOKENS,
DEFAULT_PREFILL_MAX_IN_FLIGHT,
PrefillTransferLimits,
)
from meshnet_gateway.server import _BinaryActivation, _post_binary_forward
def test_limits_have_safe_defaults_and_keep_legacy_chunk_setting(monkeypatch):
monkeypatch.delenv("MESHNET_PREFILL_CHUNK_TOKENS", raising=False)
monkeypatch.delenv("MESHNET_PREFILL_MAX_IN_FLIGHT", raising=False)
monkeypatch.setenv("MESHNET_CHUNK_TOKENS", "64")
limits = PrefillTransferLimits.from_env()
assert limits.chunk_tokens == 64
assert limits.max_in_flight == DEFAULT_PREFILL_MAX_IN_FLIGHT
assert PrefillTransferLimits().chunk_tokens == DEFAULT_PREFILL_CHUNK_TOKENS
assert limits.max_buffered_bytes == limits.max_chunk_bytes
def test_slow_consumer_applies_backpressure_preserves_order_and_bounds_bytes():
sender = BoundedPrefillSender(
PrefillTransferLimits(chunk_tokens=2, max_in_flight=4, max_chunk_bytes=8)
)
sent: list[int] = []
produced: list[int] = []
def chunks():
for index in range(3):
produced.append(index)
yield bytes([index]) * 8
def forward(chunk: bytes) -> int:
sent.append(chunk[0])
# The next item cannot have been constructed while this consumer waits.
assert produced[-1] == chunk[0]
assert len(produced) == chunk[0] + 1
assert sender.buffered_bytes == 8
assert sender.in_flight == 1
return chunk[0]
assert sender.send(chunks(), body_size=len, forward=forward) == [0, 1, 2]
assert sent == [0, 1, 2]
assert sender.peak_buffered_bytes <= sender.limits.max_buffered_bytes
assert sender.peak_in_flight == 1
assert sender.buffered_bytes == sender.in_flight == 0
def test_failure_and_cancellation_release_owned_buffers():
sender = BoundedPrefillSender(PrefillTransferLimits())
with pytest.raises(RuntimeError, match="route lost"):
sender.send([b"one", b"two"], body_size=len, forward=lambda _: (_ for _ in ()).throw(RuntimeError("route lost")))
assert sender.closed
assert sender.buffered_bytes == sender.in_flight == 0
cancelled = Event()
cancelled.set()
sender = BoundedPrefillSender(PrefillTransferLimits())
assert sender.send([b"never"], body_size=len, forward=lambda _: None, cancelled=cancelled) == []
assert sender.buffered_bytes == sender.in_flight == 0
def test_legacy_single_chunk_peer_response_uses_outgoing_metadata(monkeypatch):
class Response:
headers = {"Content-Type": "application/octet-stream"}
def read(self):
return b"\0" * (1 * 1 * 64 * 2)
def __enter__(self):
return self
def __exit__(self, *_):
return False
monkeypatch.setattr("urllib.request.urlopen", lambda *_args, **_kwargs: Response())
activation = _BinaryActivation(
body=b"\0" * (1 * 1 * 64 * 2), shape=[1, 1, 64], dtype="bfloat16",
session="legacy-session", chunk_index=0, chunk_total=1, encoding=None, headers={},
)
response = _post_binary_forward("http://legacy/forward", activation, hop_index=0, timeout=1.0)
assert response.session == activation.session
assert response.chunk_index == response.chunk_total - 1 == 0

44
tests/test_real_model_backend.py
View File

@@ -24,6 +24,7 @@ from meshnet_node.model_backend import (
_should_partial_materialize_shard,
_decoder_attention_mask,
_int_tensor_header,
_tensor_from_bfloat16_bytes,
_torch_cuda_is_executable,
build_quantization_config,
validate_quantization,
@@ -31,6 +32,15 @@ from meshnet_node.model_backend import (
from meshnet_node.torch_server import TorchNodeServer
def _require_functional_torch():
"""Skip tensor-behaviour tests when the installed torch namespace is incomplete."""
torch = pytest.importorskip("torch")
required = ("tensor", "zeros", "ones", "arange", "bfloat16", "long")
if not all(hasattr(torch, name) for name in required):
pytest.skip("requires a functional PyTorch tensor runtime")
return torch
class _FakeBackend:
model_id = "fake-model"
total_layers = 12
@@ -525,22 +535,37 @@ def test_distributed_generating_log_includes_tps(capsys):
out = capsys.readouterr().out
assert "generating step=1/1" in out
assert " tps=" in out
assert "generation complete tokens=1" in out
assert "generation complete session=" in out
assert "tokens=1" in out
assert out.count("generating step=1/1") == 1
def test_int_tensor_header_serializes_torch_tensors():
"Int tensor header serializes torch tensors\n\nTags: model, node, real-inference"
torch = pytest.importorskip("torch")
torch = _require_functional_torch()
header = _int_tensor_header(torch.tensor([[1, 2, 3]], dtype=torch.long))
assert header.startswith("1,3:")
def test_bfloat16_wire_decode_views_owned_bytes_without_float32_round_trip():
"""Activation decode stays bf16 and does not clone bytes into bytearray.
Tags: model, performance, wire
"""
torch = _require_functional_torch()
body = torch.tensor([[1, 2]], dtype=torch.bfloat16).view(torch.uint8).numpy().tobytes()
decoded = _tensor_from_bfloat16_bytes(body, [1, 2], torch)
assert decoded.dtype == torch.bfloat16
assert decoded.tolist() == [[1.0, 2.0]]
def test_decoder_attention_mask_is_causal_float_mask():
"Decoder attention mask is causal float mask\n\nTags: model, node, real-inference"
torch = pytest.importorskip("torch")
torch = _require_functional_torch()
hidden_states = torch.zeros((1, 3, 8), dtype=torch.bfloat16)
mask = _decoder_attention_mask(torch.ones((1, 3), dtype=torch.long), hidden_states, torch)
@@ -558,7 +583,7 @@ def test_call_layer_passes_rotary_position_embeddings():
assert kwargs["position_embeddings"] == "rotary"
return hidden_states
hidden, cache_state = _call_layer(
hidden = _call_layer(
NeedsPositionEmbeddings(),
"hidden",
attention_mask=None,
@@ -567,7 +592,6 @@ def test_call_layer_passes_rotary_position_embeddings():
)
assert hidden == "hidden"
assert cache_state is None
def _fake_cache_shard(torch, *, max_sessions=16, ttl=600.0):
@@ -603,7 +627,7 @@ def _fake_cache_shard(torch, *, max_sessions=16, ttl=600.0):
def test_shard_cache_prefill_then_decode_reuses_opaque_layer_state():
"Shard cache prefill then decode reuses opaque layer state\n\nTags: cache, model, node, real-inference"
torch = pytest.importorskip("torch")
torch = _require_functional_torch()
shard = _fake_cache_shard(torch)
prefill_hidden = torch.zeros((1, 4, 2), dtype=torch.bfloat16)
@@ -643,7 +667,7 @@ def test_shard_cache_prefill_then_decode_reuses_opaque_layer_state():
def test_shard_cache_decode_miss_is_explicit():
"Shard cache decode miss is explicit\n\nTags: cache, model, node, real-inference"
torch = pytest.importorskip("torch")
torch = _require_functional_torch()
shard = _fake_cache_shard(torch)
with pytest.raises(KVCacheMiss):
@@ -659,7 +683,7 @@ def test_shard_cache_decode_miss_is_explicit():
def test_shard_cache_lru_bounds_sessions():
"Shard cache lru bounds sessions\n\nTags: cache, model, node, real-inference"
torch = pytest.importorskip("torch")
torch = _require_functional_torch()
shard = _fake_cache_shard(torch, max_sessions=1)
for session in ("old", "new"):
@@ -1139,7 +1163,7 @@ def test_torch_model_shard_prefers_partial_loader_for_local_snapshot(tmp_path, m
"torch",
types.SimpleNamespace(
cuda=types.SimpleNamespace(is_available=lambda: False),
device=lambda value: value,
device=lambda value: types.SimpleNamespace(type=value),
bfloat16="bf16",
),
)
@@ -1179,7 +1203,7 @@ def test_two_node_gpt2_completion_is_deterministic():
"Two node gpt2 completion is deterministic\n\nTags: model, node, real-inference"
if os.environ.get("CI"):
pytest.skip("GPT-2 integration test is skipped in CI")
torch = pytest.importorskip("torch")
torch = _require_functional_torch()
pytest.importorskip("transformers")
pytest.importorskip("safetensors")
pytest.importorskip("accelerate")

126
tests/test_route_session_benchmark.py Normal file
View File

@@ -0,0 +1,126 @@
"""DIP-001 deterministic Route Session benchmark coverage."""
import json
from unittest.mock import MagicMock, patch
from meshnet_node.route_session_benchmark import (
BenchmarkScenario,
PerformanceThresholds,
assert_benchmark,
assert_performance_gate,
format_summary,
main,
run_benchmark_matrix,
run_real_model_lan_benchmark,
run_route_session_benchmark,
)
def test_matrix_reports_direct_relay_prefill_decode_and_machine_readable_metrics():
"""The baseline reports every required scenario and transport metric.
Tags: performance, routing
"""
report = run_benchmark_matrix()
assert {(run["mode"], run["cache_mode"]) for run in report["runs"]} == {
("direct", "cached"), ("direct", "stateless"),
("relay", "cached"), ("relay", "stateless"),
}
for run in report["runs"]:
assert set(run["phases"]) == {"prefill", "decode"}
assert "head->tail" in run["seams"]
assert {"p50_latency_ms", "p95_latency_ms", "payload_bytes", "compression_ratio",
"connection_attempts", "p95_queue_wait_ms"} <= set(run["phases"]["decode"])
sample = run["samples"][0]
assert sample["framing_ms"] > 0
assert sample["metadata_ms"] > 0
assert sample["copy_allocation_ms"] > 0
assert sample["copy_allocation_bytes"] >= sample["payload_bytes"]
assert len(run["samples"]) == 1 + len(run["output_tokens"])
assert {"tokens_per_sec", "bytes_per_token", "compression_cpu_ms", "peak_buffered_bytes"} <= set(run["phases"]["decode"])
def test_cached_sessions_reuse_one_connection_and_preserve_stub_tokens():
"""Cached Route Sessions keep one direct or relay connection per seam.
Tags: performance, relay
"""
scenario = BenchmarkScenario(output_tokens=(" one", " two", " three"))
for mode in ("direct", "relay"):
run = run_route_session_benchmark(mode, "cached", scenario)
assert_benchmark(run, expected_tokens=scenario.output_tokens, expected_connection_attempts=1)
if mode == "relay":
assert all(sample.queue_wait_ms > 0 for sample in run.samples)
def test_stateless_baselines_make_each_activation_a_connection_attempt():
"""Stateless comparison mode does not accidentally inherit Route Session state.
Tags: performance, routing
"""
scenario = BenchmarkScenario(output_tokens=(" one", " two"))
for mode in ("direct", "relay"):
run = run_route_session_benchmark(mode, "stateless", scenario)
assert_benchmark(run, expected_tokens=scenario.output_tokens, expected_connection_attempts=3)
def test_cli_writes_json_artifact_and_human_summary(tmp_path, capsys):
"""The CLI emits both CI-ready JSON and an operator-readable summary.
Tags: performance
"""
output = tmp_path / "route-session-benchmark.json"
assert main(["--json-out", str(output)]) == 0
report = json.loads(output.read_text())
assert report["schema_version"] == 1
assert "Route Session benchmark" in capsys.readouterr().out
assert "relay" in format_summary(report)
def test_performance_gate_checks_comparison_identity_session_and_cleanup():
"""CI gate accepts the fixed matrix and rejects a meaningful slowdown.
Tags: performance, routing
"""
report = run_benchmark_matrix()
assert_performance_gate(report)
report["runs"][0]["phases"]["decode"]["tokens_per_sec"] = 0.1
try:
assert_performance_gate(report, thresholds=PerformanceThresholds())
except AssertionError as exc:
assert "throughput regressed" in str(exc)
else: # pragma: no cover - makes the intended threshold explicit
raise AssertionError("gate did not catch the throughput regression")
def test_performance_gate_rejects_session_or_cleanup_leaks():
"""Exact resource/session invariants are not subject to variance tolerance.
Tags: performance, routing
"""
report = run_benchmark_matrix()
report["runs"][0]["samples"][1]["session_id"] = "wrong-session"
try:
assert_performance_gate(report)
except AssertionError as exc:
assert "Route Session changed" in str(exc)
else: # pragma: no cover
raise AssertionError("gate did not catch session instability")
def test_real_model_lan_capture_uses_the_shared_report_schema():
"""The opt-in LAN command is client-measurable and needs no real model in CI.
Tags: performance
"""
response = MagicMock()
response.read.return_value = json.dumps({"choices": [{"message": {"content": "amber birch"}}]}).encode()
response.headers.get.return_value = "lan-session"
response.__enter__.return_value = response
with patch("meshnet_node.route_session_benchmark.urllib.request.urlopen", return_value=response):
report = run_real_model_lan_benchmark("http://lan-node:7000", model="test-model")
run = report["runs"][0]
assert report["source"] == "real-model-lan-client"
assert run["session_id"] == "lan-session"
assert run["phases"]["decode"]["tokens_per_sec"] > 0
assert run["cleanup"]["open_connections"] == 0

78
tests/test_seam_telemetry.py Normal file
View File

@@ -0,0 +1,78 @@
"""Unit coverage for bounded Activation Seam telemetry."""
from meshnet_node.seam_telemetry import GenerationTelemetry
def test_seam_telemetry_aggregates_bytes_latency_and_correlates_ids():
telemetry = GenerationTelemetry("route-session-1", report_every=2, report_interval=100, now=0.0)
assert telemetry.record_seam(
activation_id="activation-1", phase="prefill", hop=0, node="node-a",
latency_seconds=0.012, wire_bytes=120, response_bytes=240,
connection_reused=False, now=0.1,
)
telemetry.mark_reported(now=0.1)
assert telemetry.record_seam(
activation_id="activation-2", phase="prefill", hop=0, node="node-a",
latency_seconds=0.008, wire_bytes=80, response_bytes=160,
connection_reused=True, now=0.2,
)
telemetry.note_tokens(4)
snapshot = telemetry.snapshot(now=2.0)
assert snapshot["session_id"] == "route-session-1"
assert snapshot["tokens_per_sec"] == 2.0
assert snapshot["seams"] == [{
"phase": "prefill", "hop": 0, "node": "node-a", "activations": 2,
"latency_ms": 20.0, "avg_latency_ms": 10.0, "wire_bytes": 200,
"response_bytes": 400, "connection_reuse": 1,
"compression_input_bytes": 0, "compression_output_bytes": 0, "compression_ms": 0.0,
"decompression_input_bytes": 0, "decompression_output_bytes": 0, "decompression_ms": 0.0,
"last_activation_id": "activation-2",
}]
def test_seam_telemetry_includes_compression_work_and_byte_counters():
telemetry = GenerationTelemetry("route-session-compression", now=0.0)
telemetry.record_compression(
phase="prefill", hop=0, node="node-a", input_bytes=1000, output_bytes=200,
elapsed_seconds=0.003,
)
telemetry.record_compression(
phase="prefill", hop=0, node="node-a", input_bytes=200, output_bytes=1000,
elapsed_seconds=0.001, decompression=True,
)
seam = telemetry.snapshot(now=1.0)["seams"][0]
assert seam["compression_input_bytes"] == 1000
assert seam["compression_output_bytes"] == 200
assert seam["compression_ms"] == 3.0
assert seam["decompression_input_bytes"] == 200
assert seam["decompression_output_bytes"] == 1000
assert seam["decompression_ms"] == 1.0
def test_seam_telemetry_reports_on_bounded_cadence_and_cleans_up():
telemetry = GenerationTelemetry("route-session-2", report_every=3, report_interval=5, now=0.0)
for count in range(1, 4):
due = telemetry.record_seam(
activation_id=f"activation-{count}", phase="decode", hop=1, node="node-b",
latency_seconds=0.001, wire_bytes=10, response_bytes=20,
connection_reused=True, now=float(count),
)
if due:
telemetry.mark_reported(now=float(count))
assert not telemetry.report_due
assert telemetry.record_seam(
activation_id="activation-4", phase="decode", hop=1, node="node-b",
latency_seconds=0.001, wire_bytes=10, response_bytes=20,
connection_reused=True, now=9.0,
)
telemetry.close()
assert telemetry.snapshot(now=10.0)["seams"] == []
assert not telemetry.record_seam(
activation_id="late", phase="decode", hop=1, node="node-b",
latency_seconds=0.001, wire_bytes=10, response_bytes=20,
connection_reused=True, now=10.0,
)

525
tests/test_tracker_capability_admission.py Normal file
View File

@@ -0,0 +1,525 @@
"""NCA-004 tests: the tracker records a node's capability proof and routes only
to nodes whose proof admits what they advertise (ADR-0023).
Model ids here are arbitrary and made up on purpose: nothing in the admission or
routing path may branch on a vendor, model or kernel name.
"""
import json
import time
import urllib.error
import urllib.request
import pytest
from meshnet_tracker.capability import (
MIN_CATALOGUE_VERSION,
POLICY_COMPAT,
POLICY_ENFORCE,
STATE_ABSENT,
STATE_ADMITTED,
STATE_CATALOGUE_INCOMPATIBLE,
STATE_FAILED,
STATE_INVALID,
STATE_MODEL_MISMATCH,
STATE_RECIPE_MISMATCH,
STATE_SHARD_MISMATCH,
STATE_STALE,
evaluate_report,
policy_from_env,
)
from meshnet_tracker.server import (
TrackerServer,
_NodeEntry,
_capability_routable,
_node_admission,
_select_route,
)
MODEL = "arbitrary-labs/oracle-9b"
SHORT = "oracle-9b"
LAYERS = 32
def _post_json(url: str, payload: dict) -> dict:
data = json.dumps(payload).encode()
req = urllib.request.Request(
url, data=data, headers={"Content-Type": "application/json"}, method="POST"
)
with urllib.request.urlopen(req) as r:
return json.loads(r.read())
def _get_json(url: str) -> dict:
with urllib.request.urlopen(url) as r:
return json.loads(r.read())
def _report(
*,
model_id: str = MODEL,
start: int = 0,
end: int = 15,
status: str = "passed",
validated_at: float | None = None,
recipe_id: str = "baseline",
recipe_version: str = "1",
catalogue_version: str = MIN_CATALOGUE_VERSION,
schema_version: int = 1,
device: str = "cpu",
diagnostics: list | None = None,
) -> dict:
"""A capability report shaped exactly as `meshnet_node.capability` emits it."""
return {
"schema_version": schema_version,
"model": {"model_id": model_id, "revision": None, "config_fingerprint": None},
"shard": {"start": start, "end": end},
"recipe": {
"recipe_id": recipe_id,
"recipe_version": recipe_version,
"catalogue_version": catalogue_version,
},
"backend": {
"backend_id": "torch-transformers",
"device": device,
"device_name": None,
"quantization": "bfloat16",
"runtime": {},
},
"status": status,
"validated_at": time.time() if validated_at is None else validated_at,
"duration_ms": 42,
"diagnostics": list(diagnostics or []),
}
def _registration(
port: int,
*,
start: int = 0,
end: int = 15,
report: dict | None = "default", # type: ignore[assignment]
recipe_id: str | None = "baseline",
recipe_version: str | None = "1",
benchmark_tokens_per_sec: float = 10.0,
) -> dict:
payload: dict = {
"endpoint": f"http://127.0.0.1:{port}",
"model": SHORT,
"hf_repo": MODEL,
"num_layers": LAYERS,
"shard_start": start,
"shard_end": end,
"hardware_profile": {},
"score": 1.0,
"tracker_mode": start == 0,
"benchmark_tokens_per_sec": benchmark_tokens_per_sec,
}
if report == "default":
report = _report(start=start, end=end)
if report is not None:
payload["capability_report"] = report
if recipe_id is not None:
payload["recipe_id"] = recipe_id
if recipe_version is not None:
payload["recipe_version"] = recipe_version
return payload
def _always(_reported: str) -> bool:
return True
def _evaluate(report, **kwargs) -> object:
kwargs.setdefault("model_matches", _always)
kwargs.setdefault("advertised_model", MODEL)
kwargs.setdefault("shard_start", 0)
kwargs.setdefault("shard_end", 15)
return evaluate_report(report, **kwargs)
# --------------------------------------------------------------- report verdicts
def test_a_passing_report_that_covers_the_registration_is_admitted():
"A proof matching the advertised model and shard admits the node.\n\nTags: routing, tracker"
state = _evaluate(_report())
assert state.state == STATE_ADMITTED
assert state.proven
assert state.model_id == MODEL
assert (state.shard_start, state.shard_end) == (0, 15)
def test_a_missing_report_is_absent_not_admitted():
"No proof is recorded as `absent` — never silently treated as proven.\n\nTags: routing, tracker"
state = _evaluate(None)
assert state.state == STATE_ABSENT
assert not state.proven
def test_a_failed_report_is_recorded_as_failed():
"A node that failed its own validation is not routable.\n\nTags: routing, tracker"
state = _evaluate(_report(status="failed", diagnostics=["out of memory on device"]))
assert state.state == STATE_FAILED
assert not state.proven
assert "out of memory" in state.detail
def test_a_report_for_a_different_model_is_a_model_mismatch():
"A proof for another artifact proves nothing about this one.\n\nTags: routing, tracker"
state = evaluate_report(
_report(model_id="other-org/unrelated-3b"),
model_matches=lambda reported: reported == MODEL,
advertised_model=MODEL,
shard_start=0,
shard_end=15,
)
assert state.state == STATE_MODEL_MISMATCH
def test_a_report_for_a_different_shard_is_a_shard_mismatch():
"A proof for layers 0–15 does not admit a node advertising 16–31.\n\nTags: routing, tracker"
state = _evaluate(_report(start=0, end=15), shard_start=16, shard_end=31)
assert state.state == STATE_SHARD_MISMATCH
def test_a_report_for_a_different_recipe_than_the_node_declares_is_a_recipe_mismatch():
"The proof must be for the recipe the node says it serves with.\n\nTags: routing, tracker"
state = _evaluate(_report(recipe_id="eager-attention"), declared_recipe_id="baseline")
assert state.state == STATE_RECIPE_MISMATCH
versioned = _evaluate(
_report(recipe_version="1"),
declared_recipe_id="baseline",
declared_recipe_version="2",
)
assert versioned.state == STATE_RECIPE_MISMATCH
def test_an_older_recipe_catalogue_is_incompatible():
"Recipe ids from a catalogue older than the tracker's minimum cannot be matched.\n\nTags: routing, tracker"
state = _evaluate(_report(catalogue_version="2025.01.1"))
assert state.state == STATE_CATALOGUE_INCOMPATIBLE
assert MIN_CATALOGUE_VERSION in state.detail
newer = _evaluate(_report(catalogue_version="2099.12.9"))
assert newer.state == STATE_ADMITTED
def test_an_unparseable_catalogue_version_is_incompatible():
"A catalogue version that cannot be compared cannot be shown to be new enough.\n\nTags: routing, tracker"
assert _evaluate(_report(catalogue_version="rolling")).state == STATE_CATALOGUE_INCOMPATIBLE
def test_a_stale_report_is_not_admitted():
"A proof older than the freshness bound must be re-validated before routing.\n\nTags: routing, tracker"
state = _evaluate(_report(validated_at=time.time() - 3600), max_age_seconds=900.0)
assert state.state == STATE_STALE
def test_a_future_dated_report_is_not_admitted():
"A proof from the future is a broken clock, not a fresh proof.\n\nTags: routing, tracker"
state = _evaluate(_report(validated_at=time.time() + 3600))
assert state.state == STATE_STALE
assert "clock" in state.detail
def test_a_report_from_an_unknown_schema_version_is_invalid():
"The tracker refuses to interpret a report layout it does not read.\n\nTags: routing, tracker"
assert _evaluate(_report(schema_version=99)).state == STATE_INVALID
@pytest.mark.parametrize(
"payload",
[
"not-an-object",
{},
{"schema_version": 1},
{**_report(), "model": {"model_id": ""}},
{**_report(), "shard": {"start": -1, "end": 3}},
{**_report(), "validated_at": "yesterday"},
{**_report(), "status": None},
],
ids=["not-object", "empty", "header-only", "blank-model", "negative-shard",
"bad-timestamp", "missing-status"],
)
def test_a_malformed_report_is_invalid_and_never_admitted(payload):
"Malformed proof is rejected by the schema check, not by a later coincidence.\n\nTags: routing, tracker"
state = _evaluate(payload)
assert state.state == STATE_INVALID
assert not state.proven
def test_recorded_detail_carries_no_credentials_from_node_diagnostics():
"Operator-facing admission detail is sanitized; a leaked token never reaches it.\n\nTags: routing, security, tracker"
state = _evaluate(
_report(
status="failed",
diagnostics=["download failed: authorization=hf_abcdefghijklmnopqrstuvwxyz01"],
)
)
assert state.state == STATE_FAILED
assert "hf_abcdefghijklmnopqrstuvwxyz01" not in state.detail
assert "hf_abcdefghijklmnopqrstuvwxyz01" not in json.dumps(state.to_dict())
assert "[redacted]" in state.detail
# ---------------------------------------------------------- compatibility policy
def test_compat_policy_routes_a_legacy_node_but_never_a_broken_proof():
"Older nodes (no proof) keep routing under `compat`; a bad proof never does.\n\nTags: routing, tracker"
absent = _evaluate(None)
failed = _evaluate(_report(status="failed"))
assert absent.routable_under(POLICY_COMPAT)
assert not absent.routable_under(POLICY_ENFORCE)
assert not failed.routable_under(POLICY_COMPAT)
assert not failed.routable_under(POLICY_ENFORCE)
def test_the_policy_is_read_from_the_environment_and_defaults_to_compat(monkeypatch):
"The rollout switch is explicit and defaults to the compatible behaviour.\n\nTags: config, routing, tracker"
monkeypatch.delenv("MESHNET_TRACKER_CAPABILITY_POLICY", raising=False)
assert policy_from_env() == POLICY_COMPAT
monkeypatch.setenv("MESHNET_TRACKER_CAPABILITY_POLICY", "enforce")
assert policy_from_env() == POLICY_ENFORCE
monkeypatch.setenv("MESHNET_TRACKER_CAPABILITY_POLICY", "nonsense")
assert policy_from_env() == POLICY_COMPAT
# ------------------------------------------------------------- the routing gate
def _entry(node_id: str, start: int, end: int, report: dict | None, **kwargs) -> _NodeEntry:
from meshnet_tracker.capability import evaluate_report as _eval
entry = _NodeEntry(
node_id=node_id,
endpoint=f"http://127.0.0.1:{9000 + int(node_id[-1])}",
shard_start=start,
shard_end=end,
model=SHORT,
shard_checksum=None,
hardware_profile={},
wallet_address=None,
score=1.0,
hf_repo=MODEL,
num_layers=LAYERS,
capability=_eval(
report,
model_matches=lambda reported: reported == MODEL,
advertised_model=MODEL,
shard_start=start,
shard_end=end,
),
**kwargs,
)
return entry
def test_route_selection_drops_every_unadmitted_candidate_under_enforce():
"Absent, failed, stale and mismatched candidates are all excluded.\n\nTags: routing, tracker"
good = _entry("node-1", 0, 31, _report(start=0, end=31))
unproven = _entry("node-2", 0, 31, None)
failed = _entry("node-3", 0, 31, _report(start=0, end=31, status="failed"))
stale = _entry("node-4", 0, 31, _report(start=0, end=31, validated_at=time.time() - 86400))
wrong_model = _entry("node-5", 0, 31, _report(start=0, end=31, model_id="someone/else-1b"))
route, error = _select_route(
[unproven, failed, stale, wrong_model, good], 0, 31, policy=POLICY_ENFORCE
)
assert not error
assert [n.node_id for n in route] == ["node-1"]
only_bad, error = _select_route([unproven, failed, stale], 0, 31, policy=POLICY_ENFORCE)
assert only_bad == []
assert "no route available" in error
def test_a_node_reassigned_to_a_shard_it_never_proved_stops_routing():
"The proof does not travel with a tracker reassignment.\n\nTags: routing, tracker"
node = _entry("node-1", 0, 15, _report(start=0, end=15))
assert _node_admission(node).state == STATE_ADMITTED
node.shard_start, node.shard_end = 16, 31 # tracker rebalanced it
assert _node_admission(node).state == STATE_SHARD_MISMATCH
assert not _node_admission(node).routable_under(POLICY_COMPAT)
def test_admitted_candidates_keep_coverage_first_and_throughput_routing():
"Gating removes candidates; among the survivors routing is unchanged.\n\nTags: routing, tracker"
head = _entry("node-1", 0, 15, _report(start=0, end=15))
slow_tail = _entry(
"node-2", 16, 31, _report(start=16, end=31), benchmark_tokens_per_sec=5.0
)
fast_tail = _entry(
"node-3", 16, 31, _report(start=16, end=31), benchmark_tokens_per_sec=50.0
)
route, error = _select_route(
[head, slow_tail, fast_tail], 0, 31, policy=POLICY_ENFORCE
)
assert not error
# Coverage first (head, then a tail), and the faster of the two tied tails.
assert [n.node_id for n in route] == ["node-1", "node-3"]
# ---------------------------------------------------------------- over the wire
def test_an_enforcing_tracker_routes_a_proven_node_and_excludes_an_unproven_one():
"End to end: a proof is required to appear in a route.\n\nTags: http, routing, tracker"
tracker = TrackerServer(capability_policy=POLICY_ENFORCE)
port = tracker.start()
try:
base = f"http://127.0.0.1:{port}"
_post_json(f"{base}/v1/nodes/register", _registration(9101, start=0, end=15))
# A tail that presents no proof at all: the route cannot complete.
_post_json(
f"{base}/v1/nodes/register",
_registration(9102, start=16, end=31, report=None),
)
with pytest.raises(urllib.error.HTTPError) as exc:
_get_json(f"{base}/v1/route?model={SHORT}")
assert exc.value.code == 503
# Now the tail proves itself and the same route resolves.
_post_json(f"{base}/v1/nodes/register", _registration(9102, start=16, end=31))
route = _get_json(f"{base}/v1/route?model={SHORT}")["route"]
assert route == ["http://127.0.0.1:9101", "http://127.0.0.1:9102"]
finally:
tracker.stop()
@pytest.mark.parametrize(
"bad_report",
[
_report(start=16, end=31), # proves the wrong shard
_report(model_id="unrelated/other-7b"), # proves the wrong model
_report(status="failed"),
_report(validated_at=time.time() - 86400), # stale
{"schema_version": 1, "model": {}}, # malformed
],
ids=["shard-mismatch", "model-mismatch", "failed", "stale", "invalid"],
)
def test_an_enforcing_tracker_never_routes_a_node_whose_proof_does_not_cover_it(bad_report):
"A proof for something else is worth exactly as much as no proof.\n\nTags: http, routing, tracker"
tracker = TrackerServer(capability_policy=POLICY_ENFORCE)
port = tracker.start()
try:
base = f"http://127.0.0.1:{port}"
resp = _post_json(
f"{base}/v1/nodes/register",
_registration(9111, start=0, end=31, report=bad_report),
)
# Registration still succeeds — the operator must be able to see the node.
assert resp["node_id"]
assert resp["capability"]["routable"] is False
assert resp["capability"]["state"] != STATE_ADMITTED
with pytest.raises(urllib.error.HTTPError) as exc:
_get_json(f"{base}/v1/route?model={SHORT}")
assert exc.value.code in (404, 503)
finally:
tracker.stop()
def test_a_compat_tracker_routes_a_legacy_node_that_sends_no_report():
"Documented rollout policy: pre-capability nodes keep working under `compat`.\n\nTags: http, routing, tracker"
tracker = TrackerServer(capability_policy=POLICY_COMPAT)
port = tracker.start()
try:
base = f"http://127.0.0.1:{port}"
_post_json(
f"{base}/v1/nodes/register",
_registration(9121, start=0, end=31, report=None, recipe_id=None, recipe_version=None),
)
route = _get_json(f"{base}/v1/route?model={SHORT}")["route"]
assert route == ["http://127.0.0.1:9121"]
finally:
tracker.stop()
def test_a_compat_tracker_still_refuses_a_node_that_presents_a_failed_proof():
"Compatibility grandfathers silence, not a proof of failure.\n\nTags: http, routing, tracker"
tracker = TrackerServer(capability_policy=POLICY_COMPAT)
port = tracker.start()
try:
base = f"http://127.0.0.1:{port}"
_post_json(
f"{base}/v1/nodes/register",
_registration(9131, start=0, end=31, report=_report(start=0, end=31, status="failed")),
)
with pytest.raises(urllib.error.HTTPError) as exc:
_get_json(f"{base}/v1/route?model={SHORT}")
assert exc.value.code == 503
finally:
tracker.stop()
def test_a_replicated_registration_carries_its_verdict_to_a_follower():
"A proven node must not be routable on the leader and dark on every follower.\n\nTags: cluster, routing, tracker"
tracker = TrackerServer(capability_policy=POLICY_ENFORCE)
proven = _registration(9151, start=0, end=31, report=_report(start=0, end=31))
proven["node_id"] = "follower-node-1"
unproven = _registration(9152, start=0, end=31, report=None)
unproven["node_id"] = "follower-node-2"
tracker._raft_apply("register", proven)
tracker._raft_apply("register", unproven)
admitted = tracker._registry["follower-node-1"]
assert admitted.capability.state == STATE_ADMITTED
assert _capability_routable(admitted, POLICY_ENFORCE)
absent = tracker._registry["follower-node-2"]
assert absent.capability.state == STATE_ABSENT
assert not _capability_routable(absent, POLICY_ENFORCE)
def test_the_network_map_exposes_the_admission_state_of_every_node():
"The operator view answers 'why is my node not routing' without raw internals.\n\nTags: http, routing, tracker"
tracker = TrackerServer(capability_policy=POLICY_ENFORCE)
port = tracker.start()
try:
base = f"http://127.0.0.1:{port}"
_post_json(f"{base}/v1/nodes/register", _registration(9141, start=0, end=15))
_post_json(
f"{base}/v1/nodes/register",
_registration(
9142,
start=16,
end=31,
report=_report(
start=16,
end=31,
status="failed",
diagnostics=["load failed: token=hf_abcdefghijklmnopqrstuvwx1234"],
),
),
)
network = _get_json(f"{base}/v1/network/map")
assert network["capability_policy"] == POLICY_ENFORCE
by_endpoint = {n["endpoint"]: n["capability"] for n in network["nodes"]}
proven = by_endpoint["http://127.0.0.1:9141"]
assert proven["state"] == STATE_ADMITTED
assert proven["routable"] is True
assert proven["model_id"] == MODEL
assert (proven["shard_start"], proven["shard_end"]) == (0, 15)
assert proven["recipe_id"] == "baseline"
assert proven["device"] == "cpu"
broken = by_endpoint["http://127.0.0.1:9142"]
assert broken["state"] == STATE_FAILED
assert broken["routable"] is False
assert broken["detail"]
raw = json.dumps(network)
assert "hf_abcdefghijklmnopqrstuvwx1234" not in raw
assert "Traceback" not in raw
finally:
tracker.stop()

31
tests/test_tracker_persistence.py Normal file
View File

@@ -0,0 +1,31 @@
from meshnet_tracker.server import TrackerServer
class _FailingStore:
def save_to_db(self) -> None:
raise OSError("database or disk is full")
class _RecordingStore:
def __init__(self) -> None:
self.saved = False
def save_to_db(self) -> None:
self.saved = True
def test_stats_persistence_failure_does_not_abort_remaining_stores(capsys):
"Tracker persistence isolates one full-disk failure from other stores\n\nTags: tracker, persistence"
tracker = TrackerServer()
tracker._stats = _FailingStore()
tracker._route_stats = _RecordingStore()
tracker._billing = None
tracker._accounts = None
tracker._contracts = None
tracker._save_stats_once()
assert tracker._route_stats.saved is True
stderr = capsys.readouterr().err
assert "stats persistence failed" in stderr
assert "database or disk is full" in stderr

294
tests/test_tracker_routing.py
View File

@@ -14,10 +14,13 @@ from meshnet_gateway.server import GatewayServer, _banned_route_wallet
from meshnet_node.server import StubNodeServer
from meshnet_contracts import LocalSolanaContracts
from meshnet_tracker.auth import sign_hive_request
from meshnet_tracker.capability import POLICY_COMPAT, POLICY_ENFORCE
from meshnet_tracker.server import (
TrackerServer,
_NodeEntry,
_available_quantizations,
_memory_pool_map,
_rebalance_all_locked,
_registration_ban_error,
_scale_demanded_models_locked,
)
@@ -130,6 +133,48 @@ def test_tracker_lists_recommended_kimi_before_nodes_register():
assert network_map["recommended_models"][0]["id"] == "kimi-k2.7"
def test_tracker_lists_recommended_qwen_27b_before_nodes_register():
"""Recommended Qwen 27B is advertised with its pinned canonical artifact.
Tags: http, routing, tracker
"""
tracker = TrackerServer()
port = tracker.start()
url = f"http://127.0.0.1:{port}"
try:
models = _get_json(f"{url}/v1/models")
network_map = _get_json(f"{url}/v1/network/map")
finally:
tracker.stop()
qwen = next(model for model in models["data"] if model["id"] == "qwen3.6-27b")
assert qwen["hf_repo"] == "Qwen/Qwen3.6-27B"
assert qwen["revision"] == "6a9e13bd6fc8f0983b9b99948120bc37f49c13e9"
assert qwen["metadata"]["num_layers"] == 64
assert qwen["metadata"]["text_only"] is True
assert qwen["deployment"]["recommended"] is True
advertised = next(model for model in network_map["recommended_models"] if model["id"] == "qwen3.6-27b")
assert advertised["hf_repo"] == "Qwen/Qwen3.6-27B"
def test_quantization_coverage_uses_higher_precision_shards():
"""A minimum-precision route may upgrade individual shards."""
bf16 = _NodeEntry(
"bf16", "http://node-a", 0, 0, "qwen3.6-27b", None, {}, None, 1.0,
quantization="bfloat16",
)
int8 = _NodeEntry(
"int8", "http://node-b", 1, 1, "qwen3.6-27b", None, {}, None, 1.0,
quantization="int8",
)
variants = {entry["id"]: entry for entry in _available_quantizations([bf16, int8], 0, 1)}
assert variants["bfloat16"]["selectable"] is False
assert variants["int8"]["selectable"] is True
assert variants["nf4"]["selectable"] is True
def test_network_map_exposes_pool_size_and_speed_summary():
"Network map exposes pool size and speed summary\n\nTags: http, routing, tracker"
tracker = TrackerServer()
@@ -2863,3 +2908,252 @@ def test_scale_demanded_models_queues_add_shard_on_spare_host():
assert assignment["model"] == "org/ModelA"
finally:
tracker.stop()
# ------------------------------------------------- RCR-001: legacy precision fallback
class _EchoChatHandler(http.server.BaseHTTPRequestHandler):
"""A tracker-mode node that answers the proxied chat request."""
def log_message(self, fmt, *args):
pass
def do_POST(self):
if self.path != "/v1/chat/completions":
self.send_response(404)
self.end_headers()
return
length = int(self.headers.get("Content-Length", 0))
self.rfile.read(length)
body = json.dumps({"choices": [{"message": {"content": "ok"}}]}).encode()
self.send_response(200)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def _legacy_registration(endpoint: str, quantization: str | None = None) -> dict:
"""A registration as a pre-quantization-field node sends it: no `quantization`."""
payload = {
"endpoint": endpoint,
"model": "Qwen2.5-0.5B-Instruct",
"hf_repo": "Qwen/Qwen2.5-0.5B-Instruct",
"num_layers": 24,
"shard_start": 0,
"shard_end": 23,
"tracker_mode": True,
"hardware_profile": {},
"score": 1.0,
}
if quantization is not None:
payload["quantization"] = quantization
return payload
def _proxy_chat_status(policy: str, quantization: str | None) -> tuple[int, dict | None]:
"""Register one legacy node under `policy` and proxy a default-precision chat."""
node = http.server.HTTPServer(("127.0.0.1", 0), _EchoChatHandler)
node_thread = threading.Thread(target=node.serve_forever, daemon=True)
node_thread.start()
tracker = TrackerServer(capability_policy=policy)
tracker_port = tracker.start()
try:
endpoint = f"http://127.0.0.1:{node.server_address[1]}"
_post_json(
f"http://127.0.0.1:{tracker_port}/v1/nodes/register",
_legacy_registration(endpoint, quantization),
)
try:
response = _post_json(
f"http://127.0.0.1:{tracker_port}/v1/chat/completions",
{"model": "qwen2.5-0.5b",
"messages": [{"role": "user", "content": "hi"}]},
)
except urllib.error.HTTPError as exc:
return exc.code, None
return 200, response
finally:
tracker.stop()
node.shutdown()
node.server_close()
node_thread.join(timeout=1.0)
def test_a_legacy_registration_without_quantization_serves_a_default_precision_request():
"An omitted field predates the protocol: it must not make a healthy node dark.\n\nTags: http, routing, tracker"
status, response = _proxy_chat_status(POLICY_COMPAT, quantization=None)
assert status == 200
assert response["choices"][0]["message"]["content"] == "ok"
def test_a_legacy_registration_without_quantization_stays_dark_under_enforce():
"The precision fallback must not smuggle an absent capability report past `enforce`.\n\nTags: http, routing, tracker"
status, _ = _proxy_chat_status(POLICY_ENFORCE, quantization=None)
assert status == 503
def test_a_node_declaring_an_unsupported_quantization_is_never_routed():
"Silence is grandfathered; an explicit unsupported precision is not.\n\nTags: http, routing, tracker"
status, _ = _proxy_chat_status(POLICY_COMPAT, quantization="fp4-experimental")
assert status == 503
def test_a_node_declaring_a_null_quantization_is_never_routed():
"An explicit null states 'no usable precision' -- only an absent field is legacy.\n\nTags: http, routing, tracker"
node = http.server.HTTPServer(("127.0.0.1", 0), _EchoChatHandler)
node_thread = threading.Thread(target=node.serve_forever, daemon=True)
node_thread.start()
tracker = TrackerServer(capability_policy=POLICY_COMPAT)
tracker_port = tracker.start()
try:
endpoint = f"http://127.0.0.1:{node.server_address[1]}"
registration = _legacy_registration(endpoint)
registration["quantization"] = None # declared, not omitted
_post_json(f"http://127.0.0.1:{tracker_port}/v1/nodes/register", registration)
with pytest.raises(urllib.error.HTTPError) as excinfo:
_post_json(
f"http://127.0.0.1:{tracker_port}/v1/chat/completions",
{"model": "qwen2.5-0.5b",
"messages": [{"role": "user", "content": "hi"}]},
)
assert excinfo.value.code == 503
finally:
tracker.stop()
node.shutdown()
node.server_close()
node_thread.join(timeout=1.0)
def test_raft_apply_preserves_a_declared_precision_for_follower_routing():
"A follower that drops the declared precision would route unlike the leader.\n\nTags: raft, routing, tracker"
tracker = TrackerServer()
tracker.start()
try:
tracker._raft_apply("register", {
"node_id": "declared-node",
"endpoint": "http://127.0.0.1:1/",
"model": "Qwen2.5-0.5B-Instruct",
"hf_repo": "Qwen/Qwen2.5-0.5B-Instruct",
"shard_start": 0,
"shard_end": 23,
"num_layers": 24,
"quantization": "nf4",
"quantizations": ["nf4", "int8"],
})
entry = tracker._registry["declared-node"]
assert entry.quantization == "nf4"
assert entry.quantizations == ["nf4", "int8"]
finally:
tracker.stop()
def test_raft_apply_grandfathers_a_legacy_registration_that_omits_quantization():
"The follower must reach the same legacy fallback the leader did, not a bare None.\n\nTags: raft, routing, tracker"
tracker = TrackerServer()
tracker.start()
try:
tracker._raft_apply("register", {
"node_id": "legacy-node",
"endpoint": "http://127.0.0.1:1/",
"model": "Qwen2.5-0.5B-Instruct",
"hf_repo": "Qwen/Qwen2.5-0.5B-Instruct",
"shard_start": 0,
"shard_end": 23,
"num_layers": 24,
})
entry = tracker._registry["legacy-node"]
assert entry.quantization == "bfloat16"
finally:
tracker.stop()
def test_rebalancing_does_not_resurrect_a_node_that_declared_an_unusable_precision():
"Managed assignment picks a node's precision, so it must not overwrite an explicit unusable one.\n\nTags: http, routing, tracker"
tracker = TrackerServer(heartbeat_timeout=0.15, rebalance_interval=10.0)
tracker_port = tracker.start()
try:
managed = _post_json(
f"http://127.0.0.1:{tracker_port}/v1/nodes/register",
{"endpoint": "http://127.0.0.1:9111", "model": "Qwen2.5-0.5B-Instruct",
"hf_repo": "Qwen/Qwen2.5-0.5B-Instruct", "num_layers": 24,
"shard_start": 0, "shard_end": 21, "managed_assignment": True,
"quantization": "fp4-experimental",
"vram_bytes": 1_000_000_000, "hardware_profile": {}, "score": 1.0},
)
# Letting this peer expire is what drives the rebalance that used to hand
# the managed node a fresh shard range -- and a default precision with it.
_post_json(
f"http://127.0.0.1:{tracker_port}/v1/nodes/register",
{"endpoint": "http://127.0.0.1:9112", "model": "Qwen2.5-0.5B-Instruct",
"hf_repo": "Qwen/Qwen2.5-0.5B-Instruct", "num_layers": 24,
"shard_start": 22, "shard_end": 23,
"vram_bytes": 1_000_000_000, "hardware_profile": {}, "score": 1.0},
)
time.sleep(0.10)
_post_json(f"http://127.0.0.1:{tracker_port}/v1/nodes/{managed['node_id']}/heartbeat", {})
time.sleep(0.10)
hb = _post_json(f"http://127.0.0.1:{tracker_port}/v1/nodes/{managed['node_id']}/heartbeat", {})
entry = tracker._registry[managed["node_id"]]
assert entry.quantization is None
assert not [d for d in hb.get("directives", []) if d["action"] == "LOAD_SHARD"]
finally:
tracker.stop()
def _register_shard(tracker_port: int, port: int, **overrides) -> dict:
payload = {
"endpoint": f"http://127.0.0.1:{port}",
"model": "Qwen2.5-0.5B-Instruct",
"hf_repo": "Qwen/Qwen2.5-0.5B-Instruct",
"num_layers": 24,
"vram_bytes": 40_000_000_000,
"hardware_profile": {},
"score": 1.0,
}
payload.update(overrides)
return _post_json(f"http://127.0.0.1:{tracker_port}/v1/nodes/register", payload)
def test_a_node_with_an_unusable_precision_covers_no_layers():
"A node no request can reach must not report its shards as served coverage.\n\nTags: http, routing, tracker"
tracker = TrackerServer(rebalance_interval=10.0)
tracker_port = tracker.start()
try:
_register_shard(
tracker_port, 9121,
shard_start=0, shard_end=23, quantization="fp4-experimental",
)
models = _get_json(f"http://127.0.0.1:{tracker_port}/v1/models")
served = [m for m in models["data"] if m["id"] == "Qwen/Qwen2.5-0.5B-Instruct"]
assert [m["shard_coverage_percentage"] for m in served] == [0.0]
finally:
tracker.stop()
def test_rebalancing_fills_the_layers_an_unusable_node_only_appears_to_cover():
"Counting a dark node as coverage hides the gap, so the rebalancer never fills it.\n\nTags: http, routing, tracker"
tracker = TrackerServer(rebalance_interval=10.0)
tracker_port = tracker.start()
try:
# The dark node claims the model's tail; the managed node holds the head.
# Both ends look covered, so the rebalancer used to see nothing to do.
dark = _register_shard(
tracker_port, 9131,
shard_start=12, shard_end=23, quantization="fp4-experimental",
)
managed = _register_shard(
tracker_port, 9132,
shard_start=0, shard_end=11, managed_assignment=True,
)
with tracker._lock:
_rebalance_all_locked(tracker._server) # type: ignore[arg-type]
assert tracker._registry[dark["node_id"]].quantization is None
entry = tracker._registry[managed["node_id"]]
assert (entry.shard_start, entry.shard_end) == (0, 23)
finally:
tracker.stop()