Merge branch 'master' of https://git.d-popov.com/popov/neuron-tai

relay anounced on node ready

.claude/memory/MEMORY.md

@@ -5,3 +5,4 @@
 - [Project status](project-status.md) — 35/35 stories done; alpha hardening next
 - **Alpha hardening** — `.scratch/alpha-hardening/` (22 issues, ADRs 0016–0019, [README](../.scratch/alpha-hardening/README.md), [handoff](../.scratch/alpha-hardening/handoff.md))
 - [Alpha hardening navigation](alpha-hardening-navigation.md) — locked fraud/auth decisions, Bucket-1 order, handoff pointers
+- **Node capability admission** — `.scratch/node-capability-admission/` (P0 plan: generic doctor/real-forward validation, fail-closed readiness, tracker admission gate; [PRD](../.scratch/node-capability-admission/PRD.md), [README](../.scratch/node-capability-admission/README.md), ADR-0023)

.claude/memory/alpha-hardening-navigation.md

@@ -30,3 +30,7 @@ Both are already migrated into `.scratch/alpha-hardening/prd.json` (AH-021 updat
 
 **Why:** three audits agreed the alpha blockers are unauthenticated gossip (anyone can inject billing events), the free-credit faucet, and ephemeral bans.
 **How to apply:** work test-first per issue acceptance criteria; use `.venv`; `cryptography` belongs in node deps (wallet.py imports it — causes many of the 24 "failures" in a fresh env). See [[project-status]] and [[autonomous-work-style]].
+
+## Routing telemetry resume (2026-07-07)
+
+`.scratch/alpha-hardening/issues/24-routing-telemetry-resume.md` / AH-024 captures the interrupted Claude handoff. Learned routing is already committed at `518c259`; the dirty tree contains live-progress/current-request heartbeat/dashboard telemetry. First known blocker: `packages/tracker/meshnet_tracker/server.py:1490` uses `threading.Lock | None`, which crashes import because `threading.Lock` is a factory function at runtime. Fix that before running the targeted telemetry tests. Keep `.claude/settings.local.json` uncommitted unless explicitly approved.

.claude/memory/project-status.md

@@ -29,6 +29,10 @@ Implementation complete for alpha-scoped blockers in `.scratch/alpha-hardening/`
 
 Historical handoff note: `/mnt/c/Users/popov/Downloads/neuron-tai-alpha-handoff-2026-07-04.md` is useful for navigation and original audit context, but it predates the completed `.scratch/alpha-hardening/` planning artifacts. Treat its "missing ADR/issues/README" statements as stale; prefer `.scratch/alpha-hardening/README.md` and `.scratch/alpha-hardening/handoff.md` for current task order.
 
+## Node capability admission P0 (2026-07-09)
+
+Planning is ready at `.scratch/node-capability-admission/` with five sequential Ralph stories and ADR-0023. The design is model-agnostic: a Node must validate its selected Model Artifact/shard with a bounded real forward before Tracker routing; Qwen3.6 is only an optional development fixture. P0 adds a versioned local recipe-manifest/report contract, `meshnet-node doctor`, fail-closed startup admission, and tracker route gating. It intentionally excludes dynamic recipe/dependency installation and the future signed Node updater.
+
 ## Windows CUDA node (working as of 2026-07-01)
 - miniforge3 base env, torch 2.7.1+cu118, torchvision 0.22.x+cu118
 - RTX 4060 Laptop GPU, 8 GB VRAM, benchmark index ~11,200
@@ -42,7 +46,8 @@ Historical handoff note: `/mnt/c/Users/popov/Downloads/neuron-tai-alpha-handoff-
 - Verification: downloader/startup targeted subset passes (`pytest tests/test_node_startup.py -k "download_shard or same_shard"`). Full `tests/test_node_startup.py` has 46 passed and 4 unrelated Windows chmod/path separator failures.
 - Live Windows confirmation: `meshnet-node start --tracker http://192.168.0.179:8080 --model Qwen3.6-35B-A3B` reuses `F:\_STORAGE\models\qwen3.6-35b-a3b`, prints `Cached at`, registers, and reaches ready as node `5gMLrmyB-26b1f8a4204a`.
 - Follow-up fix: preset-model startup now starts the heartbeat thread after registration; without this, the node appeared briefly on the dashboard and was purged on first inference/route after heartbeat expiry. Tracker dashboard now has a "Console output" panel backed by `/v1/console` for node register/expiry, routing failures, and proxy events.
-- Qwen3.6-35B-A3B reserve-based split is expected: an 79 GB CPU node may be assigned layers 0-36, and a second node fills 37-39. Do not "fix" this by bypassing the 20% assignment reserve unless the shard-planning policy changes.
+- Qwen3.6-35B-A3B CPU runtime cap (2026-07-08): the old reserve-based split could assign an 79 GB CPU node layers 0-36, but real partial loading can exceed that budget and die without a Python traceback. Node startup now clips oversized CPU auto-assignments before loading, and tracker CPU assignment uses a stricter runtime headroom factor; do not revert this to the old 20% reserve-only policy.
 - Route hardening: tracker chat proxy and `/v1/route` diagnostics now use alias-aware preset node matching for split Qwen3.6 routes; dashboard derives grouped inference history from proxy route/complete console events and shows observed TPS after completion.
 - Live proxy hardening: model lookup trims outer whitespace before alias matching (`qwen3.6-35b-a3b ` resolves), and tracker route logs/dashboard queue depth combine heartbeat queue with tracker-local proxy in-flight counts so Postman-style bursts no longer show every selected route as queue `0`.
 - Split-shard streaming hardening: Qwen3.6-style distributed generation now emits SSE chunks token-by-token from the head node instead of buffering all generated text until completion. Tracker direct/relay stream proxy logs `proxy progress` with live tokens/TPS, dashboard Inference history shows currently processing requests with live TPS/tokens/queue, and relay stream completion no longer references an undefined `session_id`.
+- Native Windows Qwen3.6-MoE import fix: `flash-linear-attention` imports `triton`; without `triton-windows`, startup fails with misleading `Could not import module 'Qwen3_5MoeForCausalLM'`. Installed `triton-windows` in `C:\Users\popov\miniforge3` and added it as a Windows-only node dependency.

.gitattributes

@@ -0,0 +1,28 @@
+# Normalize line endings across Windows/Linux checkouts.
+# All text files are stored as LF in the repo and checked out as LF
+# on every OS. Git auto-detects text vs binary.
+* text=auto eol=lf
+
+# Explicitly binary — never touch these bytes.
+*.png binary
+*.jpg binary
+*.jpeg binary
+*.gif binary
+*.ico binary
+*.pdf binary
+*.zip binary
+*.gz binary
+*.tar binary
+*.wasm binary
+*.sqlite binary
+*.sqlite3 binary
+*.safetensors binary
+*.gguf binary
+
+# Scripts that must stay LF even if someone forces CRLF locally.
+*.sh text eol=lf
+*.py text eol=lf
+
+# Windows batch files genuinely need CRLF.
+*.bat text eol=crlf
+*.cmd text eol=crlf

.gitignore

@@ -18,5 +18,12 @@ dist/
 !.env.example
 !.env.testnet
 .rocm-local/*
-billing.sqlite
-.pytest-tmp/*
+.pytest-tmp/*
+
+# Local tracker/node sqlite databases (never commit runtime state)
+*.sqlite
+*.sqlite3
+logs/tracker/error.log
+logs/tracker/info.log
+logs/tracker/warning.log
+.venv*

.scratch/alpha-hardening/README.md

@@ -9,6 +9,10 @@ Pre-release alpha audit + grilling (2026-07-04). Bucket 1 trust-boundary blocker
 
 Locked scope: one settlement tracker, open node join, devnet mock-USDT, reputation carries forward → fraud must be bounded. See [ADR-0016](../../docs/adr/0016-alpha-scope-and-known-limitations.md).
 
+**Resume task (2026-07-07):** [24 - Routing telemetry resume](./issues/24-routing-telemetry-resume.md) is `ready-for-agent`. Learned-routing commit `518c259` is already present; dirty tree contains current-request heartbeat/dashboard telemetry and a known import-time annotation crash in `server.py:1490`.
+
+**Perf follow-up (2026-07-08):** [25 — Sharded per-node KV cache for distributed generation](./issues/25-per-node-kv-cache-distributed.md) is **implemented** ([ADR-0022](../../docs/adr/0022-sharded-per-node-kv-cache.md)): per-generation session ids, prefill/decode wire protocol (`X-Meshnet-Cache`/`X-Meshnet-Past-Len`), per-node sharded `DynamicCache(config=…)` (hybrid-attention-aware), TTL+LRU eviction with 409 cache-miss → full re-prefill fallback. Golden test proves token-identical output vs the stateless path; CPU two-shard measurement: 7.05 tps decaying 32% → 18.93 tps flat (2.68×). Remaining: re-measure on the live 2-node GPU topology and the Qwen3.6-35B-A3B mixed topology.
+
 ## Artifacts
 
 | Path | Status |
@@ -16,7 +20,7 @@ Locked scope: one settlement tracker, open node join, devnet mock-USDT, reputati
 | [research-verifiable-inference.md](./research-verifiable-inference.md) | Complete — SOTA research, §8 layered scheme, TOPLOC adopt |
 | [handoff.md](./handoff.md) | Session handoff — locked decisions, env notes |
 | [docs/adr/0016–0019](../../docs/adr/) | Alpha scope, auth, fraud, multi-tracker design |
-| [issues/](./issues/) | 22 work items (Buckets 1–3) |
+| [issues/](./issues/) | 25 work items (Buckets 1–3 + perf follow-ups) |
 
 ## ADRs (this feature)
 
@@ -76,6 +80,12 @@ Locked scope: one settlement tracker, open node join, devnet mock-USDT, reputati
 | [22 MEMORY + project-status index](./issues/22-doc-memory-project-status.md) (done) |
 | [21 Honest-noise calibration corpus](./issues/21-honest-noise-calibration-corpus.md) (ops; prod gate for audits) |
 
+### Phase 5 — Distributed-inference performance (post-routing-fix)
+
+| Issue | Depends on |
+|---|---|
+| [25 Sharded per-node KV cache](./issues/25-per-node-kv-cache-distributed.md) | ADR-0020 routing fix (done), [24 routing telemetry resume](./issues/24-routing-telemetry-resume.md) |
+
 ## First 3 to implement
 
 1. **02 + 20** — Unified auth boundary + validator service token (shared helper and roles)

.scratch/alpha-hardening/issues/24-routing-telemetry-resume.md

@@ -0,0 +1,92 @@
+Status: ready-for-agent
+
+Scoped 2026-07-07 from an interrupted Claude session. This is a resume/cleanup task for routing and live-progress work that is partly committed and partly left dirty in the working tree.
+
+# 24 - Finish learned-routing telemetry and live-progress cleanup
+
+## Current state
+
+The main dynamic routing feature is already committed at `518c259` (`routing improvements - dynamic (wip)`):
+
+- `packages/tracker/meshnet_tracker/routing_stats.py` - decayed-EWMA route stats store, epsilon-greedy route selection, diagnostics.
+- `packages/tracker/meshnet_tracker/server.py` - route enumeration per head, bandit selection in the chat proxy, epoch bumps on node join/leave, `/v1/routing`, route sample recording with 8-token hygiene.
+- `packages/tracker/meshnet_tracker/cli.py` - `--route-explore-share`, `--route-weight-alpha`, `--route-stats-half-life` and env vars.
+- `packages/tracker/meshnet_tracker/dashboard.html` - "Routing (learned)" panel.
+- `docs/adr/0021-dynamic-statistical-routing.md` - design record.
+- `tests/test_dynamic_routing.py` - includes the exact GPU(0-21)+CPU(0-39) topology, hybrid downstream `start_layer=22`, 0.6/0.4 traffic split for a 1.5 TPS ratio, and scout-rate behavior.
+
+The current working tree still has uncommitted follow-up work:
+
+- `packages/node/meshnet_node/torch_server.py` - tracks in-flight chat requests, exposes `TorchNodeServer.current_requests`, prints generation progress with TPS.
+- `packages/node/meshnet_node/startup.py` - sends `current_requests` in heartbeat payloads and increases heartbeat cadence while busy.
+- `packages/tracker/meshnet_tracker/server.py` - accepts heartbeat `current_requests`, includes them in `/v1/network/map`, and logs `proxy connecting` before upstream connection.
+- `packages/tracker/meshnet_tracker/dashboard.html` - enriches the call wall from heartbeat `current_requests` so active requests remain visible even before terminal proxy events.
+- `tests/test_real_model_backend.py` and `tests/test_tracker_routing.py` - targeted coverage for current-request snapshots, heartbeat sanitization/storage, and TPS progress logging.
+- `QUICKSTART.md` - documents optional linear-attention fast-path packages for Qwen3.5/3.6 GPU nodes.
+
+There is also an untracked local file, `.claude/settings.local.json`, which should not be included unless the owner explicitly wants local Claude settings committed.
+
+## Known blocker found during resume
+
+Targeted pytest currently fails during import before reaching the new tests:
+
+```text
+TypeError: unsupported operand type(s) for |: 'builtin_function_or_method' and 'NoneType'
+```
+
+Immediate cause: `packages/tracker/meshnet_tracker/server.py:1490` annotates `ws_lock: threading.Lock | None = None`. `threading.Lock` is a factory function at runtime, not a type, so `| None` evaluates eagerly and crashes. This exists on `HEAD` too, not just in the dirty telemetry changes.
+
+Fix options:
+
+- Add `from __future__ import annotations` at the top of `server.py`, then run enough tests to catch any annotation side effects.
+- Or change that annotation to a safe runtime type such as `Any | None` / remove the union annotation. Keep the change minimal.
+
+## What to do next
+
+1. Fix the import-time `threading.Lock | None` crash.
+2. Re-run the targeted tests:
+
+   ```bash
+   .\.venv\Scripts\python.exe -m pytest tests/test_tracker_routing.py::test_tracker_heartbeat_stores_current_requests tests/test_tracker_routing.py::test_normalize_current_requests_sanitizes_payload tests/test_real_model_backend.py::test_current_requests_snapshot_while_generating tests/test_real_model_backend.py::test_distributed_generating_log_includes_tps -q
+   ```
+
+3. Run the relevant routing regression tests:
+
+   ```bash
+   .\.venv\Scripts\python.exe -m pytest tests/test_dynamic_routing.py tests/test_tracker_routing.py -q
+   ```
+
+4. If practical, run the non-integration suite:
+
+   ```bash
+   .\.venv\Scripts\python.exe -m pytest tests/ -q -m "not integration"
+   ```
+
+5. Confirm or document the pre-existing failure from the interrupted session: `test_proxy_chat_splits_payout_by_tracker_assigned_route_span` reportedly failed on `HEAD` too and was unrelated.
+6. Commit the intentional work in two commits if it remains naturally split:
+   - learned routing is already committed in `518c259`; leave it alone unless fixing regressions there.
+   - commit the live-progress/current-request telemetry cleanup separately after tests pass.
+
+## Acceptance criteria
+
+- [ ] Importing `meshnet_tracker.server` no longer crashes on the lock annotation.
+- [ ] Current-request heartbeat payloads are sanitized and surfaced in `/v1/network/map`.
+- [ ] Node-side in-flight chat snapshots report request id, model, token count, elapsed seconds, tokens/sec, and routing completion.
+- [ ] Dashboard call wall can show active requests from heartbeat data, not only tracker console terminal events.
+- [ ] Targeted telemetry tests pass.
+- [ ] Dynamic routing tests still pass, including GPU(0-21)+CPU(0-39) hybrid-route enumeration and traffic split behavior.
+- [ ] Full or non-integration suite result is recorded; unrelated pre-existing failures are named explicitly.
+- [ ] `.claude/settings.local.json` remains uncommitted unless intentionally approved.
+
+## ADR links
+
+- [ADR-0020](../../docs/adr/0020-chat-streaming-live-progress-and-mixed-topology-routing.md)
+- [ADR-0021](../../docs/adr/0021-dynamic-statistical-routing.md)
+
+## Blocked by
+
+None. The import-time annotation crash is the first fix.
+
+## Blocks
+
+Clean handoff/commit of the interrupted live routing progress work.

.scratch/alpha-hardening/issues/25-per-node-kv-cache-distributed.md

@@ -0,0 +1,58 @@
+Status: implemented 2026-07-08 — pending live 2-node GPU verification
+
+Implemented in `packages/node/meshnet_node/model_backend.py` + `torch_server.py`; design in
+[ADR-0022](../../../docs/adr/0022-sharded-per-node-kv-cache.md); tests in
+`tests/test_kv_cache_distributed.py` (11 fast tests + env-gated golden test,
+`MESHNET_REAL_MODEL_TESTS=1`).
+
+**Measured (two-shard Qwen2.5-0.5B 0-11/12-23, CPU, 44-token prompt, 40 steps):**
+stateless 7.05 tps decaying 32% (8.09 → 5.50 first-10 vs last-10); cached 18.93 tps and
+FLAT (17.21 → 19.28) — 2.68× overall, gap grows quadratically with length. Remaining
+acceptance item: re-measure on the live 2-node GPU topology (needs both machines).
+
+Scoped 2026-07-08 from a live two-machine distributed-inference debugging session (Qwen2.5-0.5B GPU+GPU pipeline, and Qwen3.6-35B-A3B mixed GPU/CPU). The ADR-0020 mixed-topology `start_layer` bug is fixed (`518c259`, `e44abc9`, `1ecc599`); this issue is the next performance blocker in the same code path.
+
+# 25 — Sharded per-node KV cache for distributed generation (MoE/hybrid-attention aware)
+
+## What to build
+
+The distributed generation loop (`torch_server.py:515-612`, `_do_chat_completions` distributed path) currently has **no KV cache at all**: `model_backend.py` passes `use_cache: False` in every layer-forward call (lines 763, 768, 770-771), and each autoregressive step re-encodes the *entire* prompt-so-far from scratch (`backend.encode_prompt(current_text)`), re-running every layer on every node in the route for every generated token.
+
+Observed cost of this on a live 2-node Qwen2.5-0.5B GPU pipeline (layers 0-20 / 21-23): tps decayed from 22.3 (at 235 output tokens) to 12.6 (at 449 tokens) within a single generation — the expected quadratic-cost signature. On the Qwen3.6-35B-A3B mixed-topology case this collapses to ~0.07 tps even after the routing fix, partly for this reason.
+
+`X-Meshnet-Session` already exists on the wire (`torch_server.py:707`, minted fresh **per token**, not per generation) but today only labels one activation transfer for chunk reassembly/logging — it is not used to key any cached state.
+
+| Subtask | Owner package | Deliverable |
+|---|---|---|
+| Session lifecycle | `packages/node/meshnet_node/torch_server.py` | Mint session ID once per chat request (not per token); reuse across all steps of that generation; add `X-Meshnet-Seq-Len` / position header so a node can tell prefill from decode steps |
+| Per-node sharded cache | `packages/node/meshnet_node/model_backend.py` | `TorchModelShard` holds a `session_id → cache_state` map scoped to *its own* layer range only (naturally sharded — no node stores another node's KV); `forward_bytes` takes `use_cache=True` and returns/reuses `past_key_values` (or `use_cache=False` for the prefill token to keep failure/eviction simple) |
+| Prefill vs. decode split | `packages/node/meshnet_node/torch_server.py` | Step 0 sends the full prompt activation (current behavior); steps 1+ send only the newest token's hidden state (`[1, 1, hidden]`) with correct `position_ids`, cutting per-step payload from O(seq_len) to O(1) |
+| MoE / hybrid-attention state | `packages/node/meshnet_node/model_backend.py` | Cache abstraction must hold "whatever `use_cache=True` returns for this layer range," not assume standard K/V tensors — Qwen3.6's linear-attention/hybrid layers (see `[transformers] The fast path is not available...` warning already logged at startup) cache **recurrent conv/delta state**, not K/V pairs. MoE expert routing itself is layer-local and needs no cross-token cache, but confirm no expert-choice state leaks across the stateless-vs-cached boundary when `use_cache` toggles between prefill and decode |
+| Cache lifecycle | `packages/node/meshnet_node/torch_server.py` | TTL + LRU eviction per node (bounded by `max_loaded_shards`/memory budget); explicit "cache miss" response so a restarted/evicted node causes the head to fall back to a full re-prefill instead of a hard error — keep today's fully-stateless path as the recovery mode |
+| Correctness parity | `tests/` | Golden-output test: distributed multi-token output with caching enabled must match the existing stateless path token-for-token (or within sampling tolerance) for a fixed prompt/seed |
+
+**Non-goals for first landing:** cross-node cache migration/rebalancing on route change (evict + re-prefill is acceptable initially); speculative decoding; batching multiple concurrent sessions' KV within one node beyond what eviction already requires.
+
+**Code refs:**
+
+- `packages/node/meshnet_node/torch_server.py:515-612` — distributed generation loop (`current_text = current_text + token_str`, full re-encode every step)
+- `packages/node/meshnet_node/torch_server.py:690-789` — `_run_downstream_pipeline`, session minting, `X-Meshnet-Session`/`X-Meshnet-Hop-Index`/`X-Meshnet-Start-Layer` headers
+- `packages/node/meshnet_node/model_backend.py:189-201, 330-351, 763-771` — `use_cache: False` call sites, `effective_start` layer-slicing logic that any cache keying must respect
+- `docs/adr/0020-chat-streaming-live-progress-and-mixed-topology-routing.md` — prerequisite routing fix this issue builds on
+- `docs/adr/0021-dynamic-statistical-routing.md` — route selection this cache must stay compatible with (a route change mid-generation should trigger cache-miss fallback, not corruption)
+
+## Acceptance criteria
+
+- [x] A session ID is stable across all steps of one chat generation (not re-minted per token) — minted once in `_do_chat_completions`, asserted in `test_session_is_stable_and_decode_payloads_are_single_token`
+- [x] Steps after the first prefill send only the new token's activation (`[1, 1, hidden]` via `encode_next_token`) with `X-Meshnet-Cache: decode` + `X-Meshnet-Past-Len`
+- [x] Each node caches state only for its own shard's layer range (`TorchModelShard.kv_sessions`; sharding falls out of per-node layer execution)
+- [x] Cache abstraction is not K/V-shaped-only: `DynamicCache(config=model.config)` — the same construction Qwen3.6-Next's own forward uses for hybrid linear-attention conv/delta state; store treats it as opaque; `TypeError` fallback disables caching per-backend
+- [x] Bounded memory: TTL (600 s, `MESHNET_KV_TTL_SECONDS`) + LRU (8, `MESHNET_KV_MAX_SESSIONS`); miss → HTTP 409 `{"error": "cache_miss"}` → head re-prefills (tested)
+- [x] Golden-output test: cached and stateless produce identical token ids on real two-shard Qwen2.5-0.5B (`test_cached_distributed_generation_matches_stateless_golden`, passed)
+- [x] Measured (CPU two-shard proxy, 40 steps): stateless 7.05 tps w/ 32% decay → cached 18.93 tps flat, 2.68×. ⚠️ still to run on the live 2-node GPU topology
+- [x] `tests/test_two_node_pipeline.py` and `tests/test_dynamic_routing.py` pass (30 passed; 6 tmp-dir fixture errors are a pre-existing Windows temp-permission env issue, identical on clean tree)
+- [x] Design captured in [ADR-0022](../../../docs/adr/0022-sharded-per-node-kv-cache.md) incl. cache-miss/route-change interaction with ADR-0021
+
+## Notes
+
+MoE routing (router + expert FFN) is layer-local per token and does not itself need a cross-token cache — it was ruled out as the cause of the earlier Qwen3.6 garbage-output bug (that was the ADR-0020 `start_layer` double-execution). The MoE angle that *does* matter here is architecture-awareness in the cache design: don't hardcode a K/V tensor shape assumption that breaks on Qwen3.6's hybrid attention layers.

.scratch/alpha-hardening/prd.json

@@ -483,9 +483,50 @@
       "notes": "Source issue: .scratch/alpha-hardening/issues/23-dynamic-hf-pricing.md. High priority, ship-soon for launch, NOT an alpha-release blocker (unlike AH-021).",
       "dependsOn": [],
       "completionNotes": "Completed by agent"
+    },
+    {
+      "id": "AH-024",
+      "title": "24 - Finish learned-routing telemetry and live-progress cleanup",
+      "description": "Status: ready-for-agent\n\nScoped 2026-07-07 from an interrupted Claude session. The learned-routing feature is already committed at 518c259 (`routing improvements - dynamic (wip)`): routing_stats.py, tracker route enumeration and bandit selection, CLI routing flags, `/v1/routing`, dashboard Routing (learned), ADR-0021, and tests/test_dynamic_routing.py including the GPU(0-21)+CPU(0-39) hybrid topology. The dirty working tree contains follow-up live-progress/current-request telemetry in torch_server.py, startup.py, tracker server/dashboard, tests, and QUICKSTART. Known blocker found during resume: importing meshnet_tracker.server currently crashes at `server.py:1490` because `ws_lock: threading.Lock | None = None` evaluates `threading.Lock` as a factory function, not a type. Fix that first, then verify and commit the telemetry cleanup separately from the already-committed dynamic-routing work. Leave `.claude/settings.local.json` uncommitted unless explicitly approved.\n\nSource issue has exact file list, commands, and the reported pre-existing unrelated failure (`test_proxy_chat_splits_payout_by_tracker_assigned_route_span`).",
+      "acceptanceCriteria": [
+        "Importing `meshnet_tracker.server` no longer crashes on the lock annotation",
+        "Current-request heartbeat payloads are sanitized and surfaced in `/v1/network/map`",
+        "Node-side in-flight chat snapshots report request id, model, token count, elapsed seconds, tokens/sec, and routing completion",
+        "Dashboard call wall can show active requests from heartbeat data, not only tracker console terminal events",
+        "Targeted telemetry tests pass",
+        "Dynamic routing tests still pass, including GPU(0-21)+CPU(0-39) hybrid-route enumeration and traffic split behavior",
+        "Full or non-integration suite result is recorded; unrelated pre-existing failures are named explicitly",
+        "`.claude/settings.local.json` remains uncommitted unless intentionally approved"
+      ],
+      "priority": 24,
+      "passes": true,
+      "notes": "Source issue: .scratch/alpha-hardening/issues/24-routing-telemetry-resume.md. Resume task for interrupted 2026-07-07 Claude session; first known fix is server.py:1490 annotation crash.",
+      "dependsOn": [],
+      "completionNotes": ""
+    },
+    {
+      "id": "AH-025",
+      "title": "25 — Sharded per-node KV cache for distributed generation (MoE/hybrid-attention aware)",
+      "description": "Status: implemented 2026-07-08 — pending live 2-node GPU verification\n\nScoped 2026-07-08 from a live two-machine distributed-inference debugging session. The ADR-0020 mixed-topology start_layer bug is fixed (518c259, e44abc9, 1ecc599); this is the next performance blocker in the same path. The distributed generation loop has NO KV cache at all: model_backend.py passes use_cache: False in every layer-forward call, and each autoregressive step re-encodes the entire prompt-so-far from scratch, re-running every layer on every node in the route for every generated token. Observed on a live 2-node Qwen2.5-0.5B GPU pipeline: tps decayed from 22.3 (at 235 output tokens) to 12.6 (at 449 tokens) within a single generation, the expected quadratic-cost signature. X-Meshnet-Session already exists on the wire but is minted fresh per token and only labels one activation transfer for chunk reassembly/logging, not keyed to any cached state. Build: (1) stable per-request session lifecycle instead of per-token, (2) per-node sharded cache keyed by session scoped to that node's own layer range only, (3) prefill-vs-decode split so post-prefill steps send only the newest token's activation, (4) cache abstraction that holds whatever use_cache=True returns per layer range (not K/V-shaped-only) because Qwen3.6's hybrid linear-attention layers cache recurrent conv/delta state, not standard K/V, (5) TTL+LRU eviction with an explicit cache-miss fallback to full re-prefill so restarts/route-changes degrade gracefully instead of corrupting output. MoE expert routing itself is layer-local and was already ruled out as the cause of the earlier Qwen3.6 garbage-output bug (that was the start_layer double-execution); the MoE angle that matters here is architecture-awareness so the cache design does not hardcode a K/V shape assumption that breaks on Qwen3.6's hybrid attention layers.\n\nSource issue has full subtask table and code refs.",
+      "acceptanceCriteria": [
+        "A session ID is stable across all steps of one chat generation (not re-minted per token)",
+        "Steps after the first prefill send only the new token's activation, not the full sequence, over the wire between nodes",
+        "Each node caches past_key_values/recurrent state only for its own shard's layer range; no node ever holds another node's cache",
+        "Cache works correctly for both standard-attention shards and Qwen3.6-style hybrid linear-attention/recurrent shards",
+        "Bounded memory: TTL + LRU eviction; eviction/restart triggers a documented cache-miss response, not silent corruption",
+        "Golden-output regression test proves cached and uncached distributed generation produce equivalent output for a fixed prompt",
+        "Measured tps improvement recorded on the same 2-node Qwen2.5-0.5B topology used to observe the regression (target: flat tps across generation length)",
+        "tests/test_two_node_pipeline.py and tests/test_dynamic_routing.py still pass",
+        "Design captured in a new ADR (or an amendment to ADR-0020/0021) covering the cache-miss/route-change interaction"
+      ],
+      "priority": 25,
+      "passes": true,
+      "notes": "Source issue: .scratch/alpha-hardening/issues/25-per-node-kv-cache-distributed.md. Perf follow-up to the ADR-0020 routing fix; no prior story covered KV caching or MoE-specific caching needs.",
+      "dependsOn": [],
+      "completionNotes": "Implemented 2026-07-08 (ADR-0022, docs/adr/0022-sharded-per-node-kv-cache.md). Per-generation session id; X-Meshnet-Cache prefill/decode + X-Meshnet-Past-Len wire headers; decode steps send [1,1,hidden] via encode_next_token (tail now returns token_id so the head never re-tokenizes); per-node SessionCacheStore holds DynamicCache(config=model.config) — hybrid-attention/recurrent-state aware, sharded naturally by each node's own layer range; TTL (600s) + LRU (8) eviction; 409 {\"error\":\"cache_miss\"} -> head re-prefills full sequence under the same session (stateless path kept as recovery mode; legacy nodes without the protocol degrade to per-step prefill). Tests: tests/test_kv_cache_distributed.py — 11 fast tests + env-gated golden test (MESHNET_REAL_MODEL_TESTS=1) proving token-identical cached vs stateless output on a real two-shard Qwen2.5-0.5B split. Measured (CPU two-shard, 40 steps): stateless 7.05 tps decaying 32% -> cached 18.93 tps flat, 2.68x overall. Remaining: re-measure on the live 2-node GPU topology and Qwen3.6-35B-A3B mixed topology (needs both machines)."
     }
   ],
   "metadata": {
-    "updatedAt": "2026-07-06T06:01:25.474Z"
+    "updatedAt": "2026-07-08T23:30:00.000Z"
   }
-}
+}

.scratch/node-capability-admission/PRD.md

@@ -0,0 +1,84 @@
+# PRD: Model-agnostic Node capability admission
+
+## Overview
+
+Make a Node demonstrate that it can execute the selected Model Artifact and assigned Shard before the Tracker exposes it in an Inference Route. The current flow registers a Node after a hardware inventory and synthetic Torch benchmark, but before any real model forward; optional backend/JIT failures can therefore occur on paid traffic.
+
+The solution is generic by design. It must not hardcode Qwen3.6, FLA, Triton, CUDA, ROCm, or any other model/backend name into the capability contract. Qwen3.6 may be used only by opt-in development integration tests.
+
+## Goals
+
+- Provide `meshnet-node doctor` to emit a machine-readable and human-readable capability report for the selected model/shard.
+- Require a successful real forward on the selected execution path before a Node becomes routable.
+- Track named recipes as data, allowing more than one validated implementation for one Model Preset.
+- Let the Tracker schedule only a Node/model/shard/recipe combination that the Node validated locally.
+- Preserve current generic Hugging Face model support and backward-compatible protocol behavior where possible.
+
+## Quality gates
+
+Every user story:
+
+- Runs its targeted `pytest` tests.
+- Runs full `pytest` before completion, or records the exact unrelated failure.
+- Keeps default tests deterministic, model-download-free, and GPU-free.
+
+Release/hardware CI:
+
+- Runs an `integration`-marked real-model doctor smoke test per certified hardware lane.
+- Passes the model ID, source, and expected backend through environment/configuration; no test has a Qwen-specific default.
+
+## User stories
+
+### NCA-001: Generic capability and recipe report
+
+As a Node operator, I need a model-agnostic capability report so that readiness is based on the executable model/shard/backend combination, not a generic GPU claim.
+
+### NCA-002: Doctor selected model/shard
+
+As a Node operator, I need `meshnet-node doctor` to validate the selected model/shard with a real forward before I join the network.
+
+### NCA-003: Fail-closed startup admission
+
+As a Node operator, I need startup to remain non-routable when the selected recipe fails so that a Node never accepts paid work it cannot execute.
+
+### NCA-004: Tracker validated-recipe routing gate
+
+As a client, I need the Tracker to select only validated Node capabilities so that an Inference Route does not include a Node that merely claims compatibility.
+
+### NCA-005: Model-agnostic operations and certified-lane verification
+
+As an operator and release engineer, I need clear doctor output and opt-in hardware-lane test instructions so that failures are actionable without exposing Python/JIT internals to ordinary users.
+
+## Functional requirements
+
+1. The local capability report identifies the Model Artifact by generic model ID/revision/config fingerprint, shard range, selected recipe ID/version, device/backend identity, success/failure status, diagnostics, and measured validation duration.
+2. A recipe is data, not a model-specific code branch. A model may offer multiple recipes; a recipe is valid only after its own real forward succeeds.
+3. `doctor` defaults to the selected model/shard and does not search/download/test unrelated models. `--all-recipes` is explicit.
+4. Startup must execute or consume a fresh matching validation before ready registration. A failed selected recipe exits non-zero before routable registration.
+5. The Tracker records validated capabilities and excludes invalid, absent, stale, model-mismatched, shard-mismatched, or catalogue-version-incompatible capabilities from route selection.
+6. The tracker protocol remains tolerant of old Nodes only during a documented compatibility window; old registrations are not eligible for routes requiring admission proof.
+7. The Node reports a versioned local recipe-manifest version. P0 has no remote executable recipe download, dependency installer, self-updater, driver installer, or GUI.
+
+## Non-goals
+
+- A signed Node auto-updater or dynamic executable recipe delivery.
+- Automatic installation of OS packages, compilers, drivers, or Python dependencies.
+- A native NiceHash-style desktop manager.
+- Supporting or certifying a particular model, GPU vendor, OS, or optional-kernel library.
+- Replacing the existing Model Artifact/assignment protocol.
+
+## Architecture
+
+Add a small generic capability domain object in the node package. `doctor` loads the requested generic model path through the same backend startup uses, executes a bounded real forward at the assigned Shard, and emits the report. Startup gates routable registration on the successful report. Registration carries validated capabilities; the tracker persists/exposes them and filters route candidates at the model/shard/recipe seam.
+
+The future signed-update contract is represented only by a local manifest version and generic schema in P0. A future Tracker Model Artifact Manifest may be signed data, but Node executable behavior remains supplied by signed Node releases.
+
+## Success measures
+
+- A backend failure that formerly appeared on the first `/forward` is caught by `doctor` and prevents ready registration.
+- A Tracker route never includes an unvalidated capability in deterministic tests.
+- The same implementation works for arbitrary test model identifiers supplied by fixtures/configuration, with no Qwen-specific branch.
+
+## Open follow-up
+
+Specify and build the signed Node release/update channel as a separate product feature after the capability contract has proved stable.

.scratch/node-capability-admission/README.md

@@ -0,0 +1,31 @@
+# Node capability admission — planning index
+
+**Status:** ready for supervised Ralph execution.
+
+This P0 makes a Node prove it can serve its selected Model Artifact and Shard before the Tracker treats it as routable. It is deliberately model-agnostic: Qwen3.6 is only a development integration fixture, never a hardcoded runtime target.
+
+## Locked decisions
+
+- A Node explicitly asked to serve a Model Preset fails closed when no validated recipe can execute it; it must not register as ready or accept paid inference.
+- Default validation covers the selected model/shard only. `meshnet-node doctor --all-recipes` is reserved for support and CI.
+- A Model Preset may have multiple named recipes. Each independently proves a real forward; the Tracker schedules only validated recipes while considering measured performance.
+- Compatibility schemas are generic. A future Tracker may publish signed, data-only Model Artifact Manifests, but executable recipes arrive only through signed Node releases.
+- P0 ships a local versioned recipe manifest and reports its version. It does **not** build a self-updater, download executable recipes, or install system dependencies.
+- Every story requires `pytest`; release CI additionally runs an `integration`-marked real-model doctor smoke test on each certified hardware lane.
+
+## Ralph order
+
+1. `NCA-001` generic capability/report contract
+2. `NCA-002` generic doctor command and real-forward validation
+3. `NCA-003` startup admission lifecycle and fail-closed behavior
+4. `NCA-004` tracker registration/routing enforcement
+5. `NCA-005` operator documentation and hardware-lane integration contract
+
+Run serially. Stories 3 and 4 both change registration/startup behavior and must not be executed in parallel.
+
+## Quality gates
+
+- Targeted `pytest` tests named by the issue.
+- Full `pytest` before marking a story done, or record the unrelated blocker.
+- No default test downloads a model or requires a GPU.
+- `pytest -m integration` / the real-model doctor test remains explicit and environment-gated.

.scratch/node-capability-admission/issues/01-generic-capability-report.md

@@ -0,0 +1,34 @@
+Status: ready-for-agent
+
+# 01 — Generic capability and recipe report
+
+## What to build
+
+Create a model-agnostic node capability domain object and local versioned recipe-manifest reader. It must represent a selected Model Artifact identity/revision/config fingerprint, Shard range, named recipe ID/version, device/backend identity, validation timestamp/duration, success/failure state, and sanitized diagnostics.
+
+Do not add Qwen-, FLA-, Triton-, CUDA-, ROCm-, or vendor-specific branches. A recipe is generic data; specific runtime behavior remains in the existing backend.
+
+**Code refs:**
+
+- `packages/node/meshnet_node/model_catalog.py` — existing generic HF config/model metadata helpers
+- `packages/node/meshnet_node/hardware.py` — device identity/executability inventory
+- `packages/node/meshnet_node/model_backend.py` — model/shard loading path
+- `packages/node/pyproject.toml` — package data declarations
+
+## Test-first
+
+1. Write a unit test building reports for two arbitrary fixture model IDs and asserting no model-specific normalization/branch is required.
+2. Write a manifest-version validation test: valid local manifest loads; malformed/unknown schema produces actionable non-secret diagnostics.
+3. Implement the smallest schema, serialization, and local manifest reader needed by later stories.
+
+## Acceptance criteria
+
+- [ ] Capability report has a stable JSON-serializable schema with model identity/fingerprint, shard range, recipe ID/version, backend/device identity, status, timing, and sanitized diagnostic fields
+- [ ] Generic arbitrary model IDs are preserved; no Qwen or optional-kernel name is a product default or code-path discriminator
+- [ ] Local recipe manifest has an explicit schema/catalogue version
+- [ ] Malformed manifest/report input fails locally with actionable diagnostics and never leaks environment secrets
+- [ ] Unit tests cover serialization, schema validation, and model-agnostic behavior
+
+## Blocked by
+
+None.

.scratch/node-capability-admission/issues/02-doctor-real-forward.md

@@ -0,0 +1,35 @@
+Status: ready-for-agent
+
+# 02 — Doctor selected model/shard with a bounded real forward
+
+## What to build
+
+Add `meshnet-node doctor`. By default it validates only the selected Model Artifact and Shard from flags/config. Reuse the production model-loading/backend execution path and execute a bounded real forward through the selected Shard; a generic Torch allocation or synthetic benchmark is insufficient.
+
+It emits concise human output plus capability-report JSON. Add explicit `--all-recipes` plumbing for support/CI without making ordinary startup validate unrelated/downloaded models. The default tests must inject a fake/lightweight backend; a real-model test is integration-marked and environment-gated with model identity supplied externally.
+
+**Code refs:**
+
+- `packages/node/meshnet_node/cli.py` — subcommand parser and config/flag resolution
+- `packages/node/meshnet_node/model_backend.py` — `TorchModelShard`, `encode_prompt`, `forward_bytes`
+- `packages/node/meshnet_node/torch_server.py` — production backend construction
+- `tests/test_node_startup.py`, `tests/test_real_model_backend.py` — startup/backend test patterns
+
+## Test-first
+
+1. Red: `doctor` reports generic hardware availability as ready without invoking model validation.
+2. Red: an injected backend forward failure still produces a success capability.
+3. Green: selected model/shard invokes one bounded generic forward and yields success only on completion.
+4. Add an `integration`-marked, env-gated test whose model ID/source is configurable; it has no model-specific default.
+
+## Acceptance criteria
+
+- [ ] `meshnet-node doctor` resolves the same selected model/shard/config path as startup
+- [ ] Default doctor performs a bounded real forward through the selected shard before reporting success
+- [ ] `--all-recipes` is explicit and does not change default onboarding cost
+- [ ] Failure returns non-zero, writes a failed capability report, and prints a user-actionable category without raw traceback by default
+- [ ] Unit tests require no GPU or model download; a separately marked integration smoke test is model-configurable
+
+## Blocked by
+
+`01-generic-capability-report.md`.

.scratch/node-capability-admission/issues/03-fail-closed-startup-admission.md

@@ -0,0 +1,34 @@
+Status: ready-for-agent
+
+# 03 — Fail-closed startup admission lifecycle
+
+## What to build
+
+Gate `run_startup` on a fresh, matching successful capability report before routable Tracker registration. A Node selected for a Model Preset/shard must fail closed if its recipe cannot perform the doctor forward: no ready/registered endpoint and no paid request acceptance.
+
+Keep local diagnostic behavior useful: a failed report may be persisted/exposed locally, but the Node must not advertise the failed model/shard as ready. Define a bounded freshness/match rule so a report cannot be reused for a different model revision, shard, recipe, or backend identity.
+
+**Code refs:**
+
+- `packages/node/meshnet_node/startup.py` — download/load/start/register sequence
+- `packages/node/meshnet_node/cli.py` — `start` and default startup error paths
+- `packages/node/meshnet_node/torch_server.py` — server lifecycle
+- `tests/test_node_startup.py` — fake startup and registration capture patterns
+
+## Test-first
+
+1. Red: backend validation failure still causes `/v1/nodes/register` to be called.
+2. Red: a success report for one arbitrary model/shard is reused for another.
+3. Green: matching successful validation reaches registration; failed/stale/mismatched validation exits before registration.
+
+## Acceptance criteria
+
+- [ ] Explicit selected model/shard fails closed before routable registration when validation fails
+- [ ] Startup sends only a matching successful capability report with its registration payload
+- [ ] Failed, stale, model-mismatched, shard-mismatched, recipe-mismatched, and backend-mismatched reports are rejected locally
+- [ ] Existing stub/test startup remains usable through an explicit test-safe capability path, not a production bypass
+- [ ] Tests prove the tracker receives no registration on a failed validation
+
+## Blocked by
+
+`01-generic-capability-report.md`, `02-doctor-real-forward.md`.

.scratch/node-capability-admission/issues/04-tracker-validated-capability-routing.md

@@ -0,0 +1,34 @@
+Status: ready-for-agent
+
+# 04 — Tracker validated-capability registration and routing gate
+
+## What to build
+
+Extend Node registration, tracker state, network-map visibility, and route selection so a candidate is eligible only when it presents a successful capability report matching the route Model Artifact and Shard. Treat recipe/backend/capability data as evidence, not a trusted performance assertion. Preserve legacy behavior only through an explicit, documented compatibility policy; no new paid route may rely on an absent proof once admission is enforced.
+
+**Code refs:**
+
+- `packages/tracker/meshnet_tracker/server.py` — `/v1/nodes/register`, tracker node state, route selection, network map
+- `tests/test_tracker_routing.py` — registration and route tests
+- `packages/node/meshnet_node/startup.py` — registration payload producer
+- `docs/adr/0011-auto-shard-and-network-assignment.md` — tracker-owned assignment context
+- `docs/adr/0013-rolling-stats-smart-routing.md` — performance routing context
+
+## Test-first
+
+1. Red: a node with no/failed/mismatched capability report can register as route-eligible for a model/shard.
+2. Red: route selection includes a candidate whose report is for a different arbitrary model or shard.
+3. Green: valid matching candidates route normally; network map exposes only sanitized admission status.
+
+## Acceptance criteria
+
+- [ ] Registration validates the generic capability-report schema and records sanitized capability state
+- [ ] Route selection excludes invalid, absent, failed, stale, model-mismatched, shard-mismatched, recipe-mismatched, or catalogue-version-incompatible candidates
+- [ ] Valid matching candidates retain normal coverage-first and throughput routing behavior
+- [ ] Network map/operator view exposes an actionable admission state without raw exceptions or secrets
+- [ ] Protocol compatibility policy for older Nodes is tested and documented
+- [ ] Deterministic tracker tests cover arbitrary model IDs, not a Qwen fixture
+
+## Blocked by
+
+`01-generic-capability-report.md`, `03-fail-closed-startup-admission.md`.

.scratch/node-capability-admission/issues/05-docs-hardware-lane-contract.md

@@ -0,0 +1,34 @@
+Status: ready-for-agent
+
+# 05 — Model-agnostic operator documentation and hardware-lane contract
+
+## What to build
+
+Document the capability-admission lifecycle, `doctor` usage, failure states, model-agnostic recipe semantics, and the certified hardware-lane release check. Correct setup guidance so it does not imply that an optional accelerator path is universally supported merely because a package can be installed.
+
+Use generic commands/placeholders in primary docs. Any concrete model used for development belongs in a clearly labelled optional example or environment-gated test configuration, never a support guarantee.
+
+**Code refs:**
+
+- `QUICKSTART.md` — node installation/ROCm/optional-backend guidance
+- `packages/node/meshnet_node/cli.py` — doctor user-facing output
+- `docs/adr/0023-model-agnostic-node-capability-admission.md`
+- `tests/test_node_startup.py`, `tests/test_real_model_backend.py` — integration marker conventions
+
+## Test-first / verification
+
+1. Add tests for concise doctor output/category mapping where practical.
+2. Verify documentation commands use the generic selected-model interface and explain the distinction between validated versus merely detected hardware.
+3. Add a release-CI runbook contract for an opt-in `integration` doctor run per certified hardware lane, with model identity supplied by CI configuration.
+
+## Acceptance criteria
+
+- [ ] Docs explain that readiness requires a successful real-forward capability report
+- [ ] Docs distinguish detected hardware, validated recipe, and routable Node states
+- [ ] Docs make no model/vendor/optional-kernel universal support promise
+- [ ] Certified-lane CI contract is documented, including model-configurable integration environment and expected evidence
+- [ ] Signed Node updates are listed as a follow-up; P0 is explicit that it does not dynamically install executable recipes or system dependencies
+
+## Blocked by
+
+`02-doctor-real-forward.md`, `04-tracker-validated-capability-routing.md`.

.scratch/node-capability-admission/prd.json

@@ -0,0 +1,95 @@
+{
+  "name": "Model-agnostic Node capability admission",
+  "branchName": "ralph/node-capability-admission",
+  "description": "Make a Node prove its selected Model Artifact, Shard, and execution recipe work before it becomes routable. Qwen3.6 is only an opt-in development fixture; the implementation and protocol are model-agnostic.",
+  "userStories": [
+    {
+      "id": "NCA-001",
+      "title": "Generic capability and recipe report",
+      "description": "Create a model-agnostic versioned capability report and local recipe-manifest contract without model or vendor code branches.",
+      "acceptanceCriteria": [
+        "Stable JSON-serializable report includes generic model identity/fingerprint, shard range, recipe ID/version, backend/device identity, status, timing, and sanitized diagnostics",
+        "Arbitrary model IDs are preserved without Qwen or optional-kernel code paths",
+        "Local recipe manifest has explicit schema/catalogue version",
+        "Malformed input fails with actionable, secret-safe diagnostics",
+        "Targeted pytest passes",
+        "Full pytest passes or an exact unrelated blocker is recorded"
+      ],
+      "priority": 1,
+      "passes": false,
+      "notes": "Source issue: .scratch/node-capability-admission/issues/01-generic-capability-report.md",
+      "dependsOn": []
+    },
+    {
+      "id": "NCA-002",
+      "title": "Doctor selected model/shard with a bounded real forward",
+      "description": "Add model-agnostic doctor validation using the same backend execution path as startup.",
+      "acceptanceCriteria": [
+        "meshnet-node doctor resolves the same selected model/shard/config as startup",
+        "Default doctor performs one bounded real selected-shard forward before success",
+        "All-recipes mode is explicit",
+        "Failure exits non-zero and writes actionable, non-traceback diagnostics by default",
+        "Unit tests have no GPU/download requirement; integration doctor smoke test is marker- and model-config-gated",
+        "Targeted pytest passes",
+        "Full pytest passes or an exact unrelated blocker is recorded"
+      ],
+      "priority": 2,
+      "passes": false,
+      "notes": "Source issue: .scratch/node-capability-admission/issues/02-doctor-real-forward.md",
+      "dependsOn": ["NCA-001"]
+    },
+    {
+      "id": "NCA-003",
+      "title": "Fail-closed startup admission lifecycle",
+      "description": "Prevent a selected model/shard from registering as routable unless its matching capability report passed.",
+      "acceptanceCriteria": [
+        "Failed selected-recipe validation makes startup exit before tracker registration",
+        "Only a fresh matching model/shard/recipe/backend report can accompany registration",
+        "Stub tests use an explicit test-safe capability path rather than production bypass",
+        "Tests prove tracker registration is not called after validation failure",
+        "Targeted pytest passes",
+        "Full pytest passes or an exact unrelated blocker is recorded"
+      ],
+      "priority": 3,
+      "passes": false,
+      "notes": "Source issue: .scratch/node-capability-admission/issues/03-fail-closed-startup-admission.md",
+      "dependsOn": ["NCA-001", "NCA-002"]
+    },
+    {
+      "id": "NCA-004",
+      "title": "Tracker validated-capability routing gate",
+      "description": "Persist and expose validated generic capability data, then route only matching admitted candidates.",
+      "acceptanceCriteria": [
+        "Tracker validates/records sanitized generic report data",
+        "Route selection excludes invalid, absent, failed, stale, or mismatched capabilities",
+        "Valid candidates retain coverage-first and throughput routing behavior",
+        "Network map exposes safe admission state",
+        "Older-node compatibility policy is documented and tested",
+        "Deterministic tests use arbitrary model IDs",
+        "Targeted pytest passes",
+        "Full pytest passes or an exact unrelated blocker is recorded"
+      ],
+      "priority": 4,
+      "passes": false,
+      "notes": "Source issue: .scratch/node-capability-admission/issues/04-tracker-validated-capability-routing.md",
+      "dependsOn": ["NCA-001", "NCA-003"]
+    },
+    {
+      "id": "NCA-005",
+      "title": "Model-agnostic docs and hardware-lane contract",
+      "description": "Document doctor/admission behavior and the opt-in real-model CI lane without promising model-specific support.",
+      "acceptanceCriteria": [
+        "Docs distinguish detected hardware, validated recipe, and routable Node",
+        "Docs make no universal optional-backend/model/vendor claim",
+        "Certified-lane CI contract includes environment-configured integration doctor test",
+        "Signed updater is explicitly deferred; P0 has no dynamic executable dependency installation",
+        "Targeted pytest passes",
+        "Full pytest passes or an exact unrelated blocker is recorded"
+      ],
+      "priority": 5,
+      "passes": false,
+      "notes": "Source issue: .scratch/node-capability-admission/issues/05-docs-hardware-lane-contract.md",
+      "dependsOn": ["NCA-002", "NCA-004"]
+    }
+  ]
+}

CONTEXT.md

@@ -1,23 +1,23 @@
-# Distributed Inference Network
-
-A volunteer GPU network where nodes independently load model shards, a tracker routes inference through optimal node chains, and contributors earn tokens for serving compute.
-
-## Language
-
-### Nodes & compute
-
-**Node**:
-A volunteer machine that runs the node client, holds one or more shards on disk, and serves inference requests for those shards.
-_Avoid_: worker, peer, miner, server
-
-**Shard**:
-A contiguous range of transformer layers from a model that a node loads and serves. Shards are the unit of storage, assignment, and reward.
-_Avoid_: partition, slice, chunk, segment
-
-**Shard Swarm**:
-The P2P group of nodes that collectively seed and download a specific shard. One swarm exists per shard.
-_Avoid_: torrent, cluster, pool
-
+# Distributed Inference Network
+
+A volunteer GPU network where nodes independently load model shards, a tracker routes inference through optimal node chains, and contributors earn tokens for serving compute.
+
+## Language
+
+### Nodes & compute
+
+**Node**:
+A volunteer machine that runs the node client, holds one or more shards on disk, and serves inference requests for those shards.
+_Avoid_: worker, peer, miner, server
+
+**Shard**:
+A contiguous range of transformer layers from a model that a node loads and serves. Shards are the unit of storage, assignment, and reward.
+_Avoid_: partition, slice, chunk, segment
+
+**Shard Swarm**:
+The P2P group of nodes that collectively seed and download a specific shard. One swarm exists per shard.
+_Avoid_: torrent, cluster, pool
+
 **Inference Route**:
 An ordered sequence of nodes whose shards together cover all layers of a model. The tracker selects the optimal route per request.
 _Avoid_: pipeline, chain, path
@@ -55,115 +55,115 @@ Realtime progress information for an active Route Session, including phase, gene
 _Avoid_: logs, debug output
 
 ### Tracker
-
-**Tracker**:
-The coordinator service that maintains the node registry, scores nodes by throughput/latency, and assigns inference routes. Runs as a centralized service with a P2P gossip fallback.
-_Avoid_: coordinator, scheduler, director
-
+
+**Tracker**:
+The coordinator service that maintains the node registry, scores nodes by throughput/latency, and assigns inference routes. Runs as a centralized service with a P2P gossip fallback.
+_Avoid_: coordinator, scheduler, director
+
 **Tracker Node**:
 A node that serves at least the first-layer shard (`layers[0..k]`) for a model and acts as the inference entry point for that model. Tracker nodes own the tokenizer and `embed_tokens`, receive client requests directly, select the onward route from the coverage map, and stream results and progress when possible. Any node advertising a new model to the network becomes its tracker node.
 _Avoid_: primary node, master node, gateway node
-
-**Coverage Map**:
-The tracker's per-model mapping of layer ranges to node counts: `[(start_layer, end_layer, node_count), ...]`. A layer range with `node_count=0` is a coverage gap — the model is unroutable until the gap is filled. Coverage-first bin-packing fills all gaps before adding redundancy.
-_Avoid_: shard map, assignment table, coverage report
-
-**Rebalance Directive**:
-A `LOAD_SHARD` or `DROP_SHARD` instruction the tracker issues to a node when the coverage map changes (node joins, node leaves, or load-balance reoptimization). Delivered as part of the node's heartbeat response.
-_Avoid_: rebalance command, shard instruction, migration order
-
-**Node Score**:
-A throughput/latency rating the tracker maintains per node, used for route selection. Updated continuously from inference telemetry.
-_Avoid_: reputation, rating, rank
-
-### Payments & fraud
-
-**Stake**:
-Collateral a node stands to lose for fraud. In the current design the node's Pending Balance serves as stake — no upfront deposit is required. An optional USDT/TAI deposit may return later for routing priority.
-_Avoid_: deposit, bond, escrow
-
-**Treasury**:
-The single project-owned Solana wallet that custodially holds client deposits, pays node payouts, and accumulates the Protocol Cut. Its keypair is loaded only on settlement-capable trackers.
-_Avoid_: escrow, vault, hot wallet
-
-**Pending Balance**:
-A node's accrued, not-yet-paid USDT earnings on the tracker ledger. Doubles as the node's fraud collateral: it is forfeited in full when a validator catches a divergent output.
-_Avoid_: unpaid rewards, accrual, balance due
-
-**Settlement Period**:
-The dynamic interval driving on-chain payouts: a node is paid when its Pending Balance exceeds the Payout Threshold or the period elapses, whichever comes first. Short in development (seconds), long in production (daily), configurable to grow with volume.
-_Avoid_: epoch, payout cycle, billing cycle
-
-**Payout Threshold**:
-The minimum Pending Balance that triggers an immediate payout before the Settlement Period elapses. Includes a dust floor so payouts are never smaller than they are worth.
-_Avoid_: minimum payout, dust limit
-
-**Protocol Cut**:
-The 10% of inference fees retained by the project for infrastructure; the remaining 90% goes to the nodes that served the request. Accumulates in the Treasury as the future TAI liquidity reserve.
-_Avoid_: spread, commission, house fee
-
-**Deposit Watcher**:
-The tracker component that observes the Treasury's on-chain USDT deposits and credits the sending client's API-key ledger balance.
-_Avoid_: payment listener, chain scanner
-
-**Mock USDT**:
-The self-created 6-decimal SPL mint that stands in for USDT on devnet, where real USDT does not exist. The mint address is configuration, so mainnet cutover is a config change.
-_Avoid_: test token, fake USDT, devnet dollar
-
-**Tax**:
-The share of caller payments distributed to compute nodes as rewards. Taxes are weighted by completed work and historical node speed so faster, larger nodes earn proportionally more.
-_Avoid_: fee, toll, commission
-
-**Caller Credit**:
-Free starting balance granted to a new caller/API key so they can try the network before topping up.
-_Avoid_: signup bonus, faucet, airdrop
-
-**Free Compute Job**:
-Work a compute node performs without earning immediate rewards, usually during probation or bootstrap phases.
-_Avoid_: unpaid labor, warmup request
-
-**Slash**:
-The penalty for a proven fraud incident: the node's entire Pending Balance is forfeited to the Treasury and a Strike is recorded.
-_Avoid_: penalize, burn, fine, forfeit
-
-**Strike**:
-A fraud incident recorded on-chain against a node. Enough strikes result in a ban.
-_Avoid_: infraction, violation, flag
-
-**Ban**:
-Permanent exclusion of a wallet from the network after exceeding the strike threshold. Recorded on-chain.
-_Avoid_: blacklist, block, suspension
-
-**Probationary Period**:
-The first N jobs a new wallet must complete without earning, to raise the cost of re-entering after a ban.
-_Avoid_: trial period, warmup, grace period
-
-**Token**:
-TAI, our native Solana SPL token. Deferred (ADR-0015): nodes are currently paid directly in USDT; TAI returns as the reward/upside layer once volume exists, funded by the accumulated Protocol Cut. Clients never need to hold it.
-_Avoid_: coin, reward token, native token
-
-**Contract Boundary**:
-The Python interface in `packages/contracts` that represents registry, payment, and settlement behavior. During the prototype it is implemented by deterministic local wrappers; later the same boundary is backed by real Solana programs.
-_Avoid_: mock contract, fake chain, temporary hack
-
-**Validator**:
-A trusted node (or the tracker itself) that re-runs a sample of inference requests to detect fraud.
-_Avoid_: auditor, checker, referee
-
-**Validation Event**:
-A completed inference record that contains enough information for a validator to decide whether to sample and re-run the request: session id, model preset, messages, inference route, node wallets, and observed output.
-_Avoid_: audit log, trace, receipt
-
-**Slash Proof**:
-The record submitted by a validator when a sampled re-run diverges from the observed output beyond tolerance. In the prototype this is deterministic local contract state; later it maps to an on-chain proof transaction.
-_Avoid_: accusation, report, claim
-
-### Client-facing
-
-**Client**:
-Any application or user that sends inference requests to the gateway. Prepays USDT into the Treasury; each request is metered against the resulting ledger balance at a per-1K-tokens price set per model.
-_Avoid_: user, caller, consumer
-
-**Model Preset**:
-A named, versioned model available on the network (e.g. `llama-3-70b`). The tracker knows which nodes hold which shards for each preset.
-_Avoid_: model, checkpoint, version
+
+**Coverage Map**:
+The tracker's per-model mapping of layer ranges to node counts: `[(start_layer, end_layer, node_count), ...]`. A layer range with `node_count=0` is a coverage gap — the model is unroutable until the gap is filled. Coverage-first bin-packing fills all gaps before adding redundancy.
+_Avoid_: shard map, assignment table, coverage report
+
+**Rebalance Directive**:
+A `LOAD_SHARD` or `DROP_SHARD` instruction the tracker issues to a node when the coverage map changes (node joins, node leaves, or load-balance reoptimization). Delivered as part of the node's heartbeat response.
+_Avoid_: rebalance command, shard instruction, migration order
+
+**Node Score**:
+A throughput/latency rating the tracker maintains per node, used for route selection. Updated continuously from inference telemetry.
+_Avoid_: reputation, rating, rank
+
+### Payments & fraud
+
+**Stake**:
+Collateral a node stands to lose for fraud. In the current design the node's Pending Balance serves as stake — no upfront deposit is required. An optional USDT/TAI deposit may return later for routing priority.
+_Avoid_: deposit, bond, escrow
+
+**Treasury**:
+The single project-owned Solana wallet that custodially holds client deposits, pays node payouts, and accumulates the Protocol Cut. Its keypair is loaded only on settlement-capable trackers.
+_Avoid_: escrow, vault, hot wallet
+
+**Pending Balance**:
+A node's accrued, not-yet-paid USDT earnings on the tracker ledger. Doubles as the node's fraud collateral: it is forfeited in full when a validator catches a divergent output.
+_Avoid_: unpaid rewards, accrual, balance due
+
+**Settlement Period**:
+The dynamic interval driving on-chain payouts: a node is paid when its Pending Balance exceeds the Payout Threshold or the period elapses, whichever comes first. Short in development (seconds), long in production (daily), configurable to grow with volume.
+_Avoid_: epoch, payout cycle, billing cycle
+
+**Payout Threshold**:
+The minimum Pending Balance that triggers an immediate payout before the Settlement Period elapses. Includes a dust floor so payouts are never smaller than they are worth.
+_Avoid_: minimum payout, dust limit
+
+**Protocol Cut**:
+The 10% of inference fees retained by the project for infrastructure; the remaining 90% goes to the nodes that served the request. Accumulates in the Treasury as the future TAI liquidity reserve.
+_Avoid_: spread, commission, house fee
+
+**Deposit Watcher**:
+The tracker component that observes the Treasury's on-chain USDT deposits and credits the sending client's API-key ledger balance.
+_Avoid_: payment listener, chain scanner
+
+**Mock USDT**:
+The self-created 6-decimal SPL mint that stands in for USDT on devnet, where real USDT does not exist. The mint address is configuration, so mainnet cutover is a config change.
+_Avoid_: test token, fake USDT, devnet dollar
+
+**Tax**:
+The share of caller payments distributed to compute nodes as rewards. Taxes are weighted by completed work and historical node speed so faster, larger nodes earn proportionally more.
+_Avoid_: fee, toll, commission
+
+**Caller Credit**:
+Free starting balance granted to a new caller/API key so they can try the network before topping up.
+_Avoid_: signup bonus, faucet, airdrop
+
+**Free Compute Job**:
+Work a compute node performs without earning immediate rewards, usually during probation or bootstrap phases.
+_Avoid_: unpaid labor, warmup request
+
+**Slash**:
+The penalty for a proven fraud incident: the node's entire Pending Balance is forfeited to the Treasury and a Strike is recorded.
+_Avoid_: penalize, burn, fine, forfeit
+
+**Strike**:
+A fraud incident recorded on-chain against a node. Enough strikes result in a ban.
+_Avoid_: infraction, violation, flag
+
+**Ban**:
+Permanent exclusion of a wallet from the network after exceeding the strike threshold. Recorded on-chain.
+_Avoid_: blacklist, block, suspension
+
+**Probationary Period**:
+The first N jobs a new wallet must complete without earning, to raise the cost of re-entering after a ban.
+_Avoid_: trial period, warmup, grace period
+
+**Token**:
+TAI, our native Solana SPL token. Deferred (ADR-0015): nodes are currently paid directly in USDT; TAI returns as the reward/upside layer once volume exists, funded by the accumulated Protocol Cut. Clients never need to hold it.
+_Avoid_: coin, reward token, native token
+
+**Contract Boundary**:
+The Python interface in `packages/contracts` that represents registry, payment, and settlement behavior. During the prototype it is implemented by deterministic local wrappers; later the same boundary is backed by real Solana programs.
+_Avoid_: mock contract, fake chain, temporary hack
+
+**Validator**:
+A trusted node (or the tracker itself) that re-runs a sample of inference requests to detect fraud.
+_Avoid_: auditor, checker, referee
+
+**Validation Event**:
+A completed inference record that contains enough information for a validator to decide whether to sample and re-run the request: session id, model preset, messages, inference route, node wallets, and observed output.
+_Avoid_: audit log, trace, receipt
+
+**Slash Proof**:
+The record submitted by a validator when a sampled re-run diverges from the observed output beyond tolerance. In the prototype this is deterministic local contract state; later it maps to an on-chain proof transaction.
+_Avoid_: accusation, report, claim
+
+### Client-facing
+
+**Client**:
+Any application or user that sends inference requests to the gateway. Prepays USDT into the Treasury; each request is metered against the resulting ledger balance at a per-1K-tokens price set per model.
+_Avoid_: user, caller, consumer
+
+**Model Preset**:
+A named, versioned model available on the network (e.g. `llama-3-70b`). The tracker knows which nodes hold which shards for each preset.
+_Avoid_: model, checkpoint, version

MAIN_FEATURES.md

@@ -0,0 +1,255 @@
+# Main Features
+
+High-level product capabilities for neuron-tai. Each section describes the user-facing
+outcome, current status, and how it fits the mass-adoption goal. Implementation detail
+lives in `QUICKSTART.md`, ADRs, and package code; this file is the product map.
+
+**Ralph task sources** (authoritative status lives in source issue headers, not always
+`passes` in JSON):
+
+| Source | Stories | Ralph branch | Notes |
+|--------|---------|--------------|-------|
+| [`docs/prd.json`](docs/prd.json) | US-001…035 | `ralph/distributed-inference-network` | **35/35 done** |
+| [`.scratch/alpha-hardening/prd.json`](.scratch/alpha-hardening/prd.json) | AH-001…025 | `ralph/alpha-hardening` | See status table below — JSON `passes` can be stale |
+| [`docs/issues/`](docs/issues/) US-036+ | 36…47 | not in Ralph yet | Filed after main PRD closed |
+| [`.scratch/distributed-gguf-runtime/`](.scratch/distributed-gguf-runtime/) | 10 milestones | not in Ralph yet | Draft scratch package |
+
+---
+
+## Node bootstrap installer
+
+**Status:** Planned — early development. Manual install (`QUICKSTART.md`) is the
+current path; a unified installer is the next step toward one-click node onboarding.
+
+**Why it matters:** Mass adoption depends on volunteers joining without reading a
+691-line quickstart or guessing which PyTorch wheel matches their GPU. Inspiration:
+[NiceHash](https://www.nicehash.com/) — detect hardware, pick the right runtime,
+install, run. Our version must support heterogeneous fleet hardware (NVIDIA CUDA,
+AMD ROCm including Strix Halo gfx1151, CPU-only laptops) and later wrap the same
+logic in a web-based GUI.
+
+### Scope
+
+| Phase | Boundary | Installer owns | User still does |
+|-------|----------|----------------|-----------------|
+| **v1 (now)** | **B — Python + OS deps** | Clone/update repo, venv, correct PyTorch index, meshnet packages, OS package checks, hardware smoke test, launch setup wizard | GPU driver install (often needs reboot), WSL2 enablement, accepting elevated prompts |
+| **v2 (target)** | **C — NiceHash-style** | Single downloadable artifact; may bundle Python/conda; maximal auto-setup | Almost nothing — accept UAC/reboot where the OS requires it |
+
+v1 explicitly does **not** silently paper over missing drivers. If `--gpu` is set and
+the GPU path cannot be verified, the installer fails with a structured error and a
+wiki slug — it does not fall back to CPU unless `--cpu` was passed.
+
+### Entry points (planned)
+
+```bash
+# Linux / WSL — auto-detect hardware, install, smoke-test, run wizard
+curl -fsSL https://<host>/install.sh | bash
+
+# Explicit device mode (early development — these two flags are enough for v1)
+curl -fsSL https://<host>/install.sh | bash -s -- --gpu
+curl -fsSL https://<host>/install.sh | bash -s -- --cpu
+
+# Non-interactive / GUI-driven (same script, no prompts)
+curl -fsSL https://<host>/install.sh | bash -s -- --gpu --yes
+```
+
+Windows equivalent: `install.ps1` with the same flags.
+
+### `--cpu` / `--gpu` semantics (v1)
+
+| Flag | Meaning |
+|------|---------|
+| *(none)* | Auto-detect hardware, print detected profile, proceed with best match (interactive confirm unless `--yes`) |
+| `--cpu` | Installer: CPU PyTorch wheel. **`meshnet-node --cpu` (implemented):** force CPU inference and CPU shard assignment even if a GPU is present |
+| `--gpu` | Install and verify a GPU runtime; **fail hard** if GPU execution cannot be confirmed after install (installer only — not implemented on `meshnet-node` yet) |
+| `--yes` | Skip interactive confirm; for headless installs and future web GUI orchestration |
+
+Installer flags set install-time intent. At runtime, `meshnet-node` auto-uses GPU when
+CUDA works; pass `--cpu` to ignore it. Hardware metadata (GPU name/VRAM) is still
+detected for diagnostics.
+
+### v1 install pipeline
+
+1. **Preflight** — Python 3.11+ (3.12 recommended for Qwen3.6/FLA), git, disk space,
+   network.
+2. **Hardware probe** — reuse detection logic aligned with
+   `packages/node/meshnet_node/hardware.py` (nvidia-smi, Windows WMI, torch CUDA/HIP
+   inventory, RAM).
+3. **OS dependency checks (boundary B)** — verify or install distro packages where
+   safe (e.g. `python3-venv`, `build-essential`); **check** GPU device nodes
+   (`/dev/kfd`, `/dev/dri/renderD*`) and group membership (`video`, `render`) on
+   Linux AMD; emit fix instructions, do not auto-modify kernel drivers.
+4. **PyTorch variant selection** — one wheel line per detected (or forced) profile:
+
+   | Profile | PyTorch source |
+   |---------|----------------|
+   | NVIDIA CUDA | Default PyPI index |
+   | CPU only | `download.pytorch.org/whl/cpu` |
+   | AMD ROCm (discrete, supported arch) | `download.pytorch.org/whl/rocm6.3` |
+   | AMD Strix Halo / gfx1151 | `rocm.nightlies.amd.com/v2/gfx1151/` |
+
+   See `QUICKSTART.md` § PyTorch variant for host prerequisites and troubleshooting
+   notes already validated on the fleet.
+
+5. **Meshnet packages** — editable install of `packages/node` (+ `p2p` as needed);
+   `transformers`, `accelerate`, and model-specific extras (e.g. `flash-linear-attention`
+   on ROCm for Qwen3.6).
+6. **Smoke test** — short matmul on chosen device (same idea as
+   `benchmark_throughput_checked()`); must pass before declaring success.
+7. **Hand off** — run existing mining-style wizard (`packages/node/meshnet_node/wizard.py`):
+   tracker URL, wallet, model/shard assignment.
+
+Keep ROCm and CPU envs **separate** when probing GPU paths so a failed ROCm attempt
+does not break a known-good CPU venv (`QUICKSTART.md` already documents this pattern).
+
+### Failure telemetry and hardware wiki
+
+Every failed install should report back structured diagnostics so support improves
+with fleet scale:
+
+- **Report payload (planned):** OS, CPU model, RAM, GPU name/VRAM/arch, chosen
+  PyTorch index, failing step, stderr tail, installer version, `--cpu`/`--gpu` flag.
+- **Privacy:** opt-in or anonymous fleet telemetry; no wallet keys or model paths.
+- **Hardware wiki / index:** failed (and successful) profiles accumulate into a
+  searchable support index — e.g. `rocm-missing-kfd`, `gfx1151-wrong-wheel`,
+  `wsl2-nvidia-smi-missing`. Each slug links symptoms, detection rule, fix steps,
+  and "works on" confirmations. Future GUI surfaces the same index when install fails.
+
+This closes the loop NiceHash gets from millions of installs: uncommon hardware
+becomes documented automatically instead of repeating Discord support threads.
+
+### GUI integration (later)
+
+The install script is the **headless API** for a future web-based node manager:
+
+- GUI downloads or invokes `install.sh` / `install.ps1` with `--gpu --yes` and streams
+  log output.
+- Same failure payloads feed the hardware wiki and in-app "your GPU + Fedora 43"
+  fix cards.
+- Post-install, GUI wraps `meshnet-node` dashboard and tracker registration status.
+
+### Related code and docs
+
+| Asset | Role |
+|-------|------|
+| `packages/node/meshnet_node/hardware.py` | Runtime hardware detection and benchmark |
+| `packages/node/meshnet_node/wizard.py` | Post-install interactive setup |
+| `QUICKSTART.md` | Current manual install matrix (source of truth until installer ships) |
+| `docs/INSTALL_WINDOWS.md` | WSL2 + CUDA passthrough path |
+
+### Open decisions (post-v1)
+
+- Exact telemetry endpoint and opt-in UX.
+- Whether v1 ships `install.sh` only or also a pinned release tarball (no git required).
+- Conda vs venv default on Windows (today: both documented; installer should pick one
+  happy path per platform).
+
+---
+
+## Core network (`docs/prd.json` — 35/35 done)
+
+Original distributed-inference Ralph arc. All stories `status: done`.
+
+| Theme | Stories | Status |
+|-------|---------|--------|
+| Scaffold + two-node pipeline | 01–02 | Done |
+| Tracker registration & routing | 03, 13–14, 20–30 | Done |
+| Node client + mining CLI | 04, 16, 21 | Done |
+| OpenAI gateway + SDK | 05, 10 | Done |
+| PyTorch backend + binary wire format | 11–12, 19 | Done |
+| P2P swarm + relay/NAT | 09, 17, 29 | Done |
+| Heartbeat, stats, smart assignment | 23–28 | Done |
+| Billing, devnet treasury, settlement, dashboard | 31–35 | Done |
+| Fraud / stake (superseded) | 06–08 | Done in PRD; alpha path replaced by ADR-0015/0018 + alpha-hardening |
+| Ralph tooling | 15 | Done (`scripts/ralph_progress.py`) |
+| Two-machine LAN test | 18 | Done |
+
+User-facing capabilities this arc delivered: mixed CPU+GPU routes across machines,
+hardware-aware routing, relay (no port-forward), OpenAI-compatible API, mining-style
+`meshnet-node` wizard, billing ledger, devnet USDT, tracker web dashboard.
+
+---
+
+## Alpha hardening (`.scratch/alpha-hardening/` — AH-001…025)
+
+Pre-release trust/money/fraud path. Index:
+[`.scratch/alpha-hardening/README.md`](.scratch/alpha-hardening/README.md).
+
+### Done (engineering complete)
+
+| ID | Feature |
+|----|---------|
+| AH-001…005 | Hive gossip auth, unified auth boundary, zero starting credit, tracker-authoritative accounting, persisted strike/ban/reputation |
+| AH-006…010 | TOPLOC integration, hop bisection, reputation model, adaptive audit routing, penalty wiring |
+| AH-011, AH-020 | Wallet binding proof, validator service token |
+| AH-016, AH-018…019, AH-022 | Doc hygiene: US-006 reconciliation, runbooks, test-env, memory index |
+| AH-023 | Dynamic HF-benchmarked pricing (engineering done; `hf_aliases` curation is human sign-off) |
+
+### Open / not truly done
+
+| ID | Feature | Status | Blocker |
+|----|---------|--------|---------|
+| AH-021 | Honest-noise TOPLOC calibration corpus | **ready-for-human** | **Alpha release blocker** — run calibration job on live hired-VPS fleet; threshold/FPR write-up |
+| AH-024 | Learned-routing telemetry + live-progress cleanup | **ready-for-agent** | `server.py:1490` import crash; dashboard active-request telemetry |
+| AH-025 | Sharded per-node KV cache | **implemented — verify** | Re-measure on live 2-node GPU + Qwen3.6 mixed topology ([ADR-0022](docs/adr/0022-sharded-per-node-kv-cache.md)) |
+
+### Deferred (post-alpha, design tracked — ADR-0019)
+
+| ID | Feature | Status |
+|----|---------|--------|
+| AH-012…015 | On-chain idempotency, consensus-gated settlement, durable Raft term/vote, commutative forfeit | ready-for-human |
+| AH-017 | Duplicate US-020 issue dedup | ready-for-human |
+
+---
+
+## Post-PRD backlog (`docs/issues/` US-036+)
+
+Filed after the main 35-story arc closed. Not yet in a Ralph `prd.json`.
+
+| ID | Feature | Status | Priority note |
+|----|---------|--------|---------------|
+| US-036 | Streamed chat over relay RPC | planned | Critical — blocks public friends-test |
+| US-037 | Relay bridge concurrency | planned | |
+| US-038 | Tracker seed join | planned | |
+| US-039…041 | Caller credit keys, dashboard top-up, account wallet keypair | planned | |
+| US-042 | GGUF / llama.cpp node backend | planned | Pairs with distributed-gguf scratch |
+| US-043 | Dashboard model search cards | planned | |
+| US-044 | Tracker as shard file source (partial download) | **in progress** | High — multi-machine big models |
+| US-045 | Dual-rate billing | **in progress** | |
+| US-046 | Tracker env + first-node autojoin | **in progress** | |
+| US-047 | Model source download visibility | **in progress** | |
+| US-020b | Memory budget, shard slots, dropout relocation | ready-for-agent | Hardens US-013 capacity contract |
+
+---
+
+## Distributed GGUF runtime (draft scratch)
+
+Long-horizon runtime for torrent-distributed GGUF + llama.cpp multi-node routes.
+Not in Ralph yet. See
+[`.scratch/distributed-gguf-runtime/README.md`](.scratch/distributed-gguf-runtime/README.md).
+
+| Milestone | Status |
+|-----------|--------|
+| 01–10 (route session → networked GGUF → model audits) | Planned / not started |
+| PyTorch distributed KV reference (04) | Partially addressed by AH-025 |
+
+---
+
+## Feature status at a glance
+
+| Feature | Status | Ralph / source |
+|---------|--------|----------------|
+| Mixed hardware inference routes | **Working** | US-002+, ADR-0020 |
+| Hardware-aware + learned routing | **Working** (telemetry cleanup open) | US-027+, AH-024 |
+| Zero port-forwarding (relay) | **Working** (streamed relay chat open) | US-017, US-029, US-036 |
+| OpenAI-compatible API | **Working** | US-005 |
+| Mining-style node CLI + wizard | **Working** (`--cpu` forces CPU mode) | US-016 |
+| Billing + devnet USDT | **Working** | US-031…033, alpha-hardening |
+| Fraud / TOPLOC / reputation | **Engineering done** (calibration ops pending) | AH-006…010, AH-021 |
+| Sharded per-node KV cache | **Implemented — GPU verify pending** | AH-025, ADR-0022 |
+| Node bootstrap installer | **Planned** | This doc — not in Ralph yet |
+| Dynamic HF pricing | **Done** (alias curation ongoing) | AH-023 |
+| Distributed GGUF / llama.cpp | **Draft** | `.scratch/distributed-gguf-runtime/` |
+
+Narrative hooks for landing copy:
+[`.claude/memory/product-selling-points.md`](.claude/memory/product-selling-points.md).

_DEV_NOTES.md

@@ -22,9 +22,23 @@
     --tracker http://192.168.0.179:8081 `                                                                                                                                             
     --model Qwen/Qwen2.5-0.5B-Instruct `                                                                                                                                              
     --advertise-host 192.168.0.20                                                                                                                                                     
+ 
 
+ qwen3.6-35b-a3b  Qwen/Qwen2.5-0.5B-Instruct 
 # linux                     
 HF_HOME=/run/media/popov/d/DEV/models     .venv/bin/meshnet-node start       --model-id Qwen/Qwen2.5-0.5B-Instruct       --shard-start 0       --shard-end 21       --quantization bfloat16       --tracker http://localhost:8081  
+HF_HOME=/run/media/popov/d/DEV/models  .venv-rocm/bin/meshnet-node  start --tracker https://meshnet.2.d-popov.com   --model  qwen3.6-35b-a3b   --shard-start 10 
+ meshnet-node start --tracker http://192.168.0.179:8080 --model Qwen/Qwen2.5-0.5B-Instruct  --shard-start 0 --shard-end 20  
+ meshnet-node start --tracker https://meshnet.2.d-popov.com   --model Qwen/Qwen2.5-0.5B-Instruct     
+ meshnet-node start --tracker https://meshnet.2.d-popov.com   --model  qwen3.6-35b-a3b --cpu   
+
+meshnet-node start --tracker https://meshnet.2.d-popov.com --model qwen3.6-35b-a3b --shard-start 0   --shard-end 21 --node-name gpu-head
+meshnet-node start --tracker https://meshnet.2.d-popov.com --model qwen3.6-35b-a3b --shard-start 22   --shard-end 39 --cpu --node-name cpu-tail
+
+
+meshnet-node start --tracker https://meshnet.2.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct  --shard-end 20          --node-name gpu-head
+meshnet-node start --tracker https://meshnet.2.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct  --shard-start 12  --cpu --node-name cpu-tail
+
 # win
  meshnet-node start --tracker http://ai.neuron.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct   --shard-start 10 
  meshnet-node start --tracker http://192.168.0.179:8081  --model Qwen/Qwen2.5-0.5B-Instruct   --shard-start 10 

deploy/portainer/README.md

@@ -0,0 +1,275 @@
+# Portainer deployment
+
+Start here if you want the public alpha tracker online from Portainer.
+
+Recommended first alpha path:
+
+1. If you do **not** have a Docker image in Gitea yet, use `meshnet-tracker-nobuild-stack.yml`.
+2. After that works, create a Gitea **Container package** and switch Portainer to an image-based stack.
+3. Do **not** create an npm package for deployment. This service is Python + Docker. For Portainer, the useful package is a Docker/OCI container image in Gitea Packages.
+
+This folder contains:
+
+| File | Use when |
+| --- | --- |
+| `meshnet-tracker-nobuild-stack.yml` | Easiest first deployment. No Docker registry. Downloads a repo tarball and installs at container start. |
+| `meshnet-tracker-stack.yml` | Cleaner long-term deployment. Uses `deploy/docker/Dockerfile` / a prebuilt container image. |
+| `meshnet-relay-only-stack.yml` | Optional relay-only deployment for a separate relay host/container. Not needed for the default alpha stack because the tracker embeds the relay. |
+| `../docker/Dockerfile` | Builds one image containing tracker + relay + contracts packages. |
+
+Recommended alpha architecture:
+
+- One `meshnet-tracker` container.
+- The tracker runs the relay in-process via `--embedded-relay`.
+- The relay implementation is still the shared `meshnet_relay.RelayServer` class, so a future relay-only node can be split out without changing the protocol.
+- Nginx Proxy Manager (or nginx/Caddy/Traefik) terminates TLS and routes `/v1`, `/dashboard`, `/ws`, and `/rpc` to the container.
+
+Important: the separate `meshnet-relay` container was not dropped as a capability. We removed it from the default alpha stack only to make first deployment simpler. Relay-only deployment remains supported via `meshnet-relay-only-stack.yml` or by running `meshnet-relay` from the same image.
+
+## Option A — easiest today: no registry / no package
+
+Use `meshnet-tracker-nobuild-stack.yml` in Portainer.
+
+It starts from `python:3.12-slim`, downloads a source `.tar.gz`, installs `packages/tracker`, `packages/relay`, and `packages/contracts`, then starts the tracker with embedded relay. First boot is slower, but it avoids creating/pushing a package.
+
+Required Portainer environment variables:
+
+```text
+SOURCE_TARBALL_URL=https://git.d-popov.com/popov/neuron-tai/archive/master.tar.gz
+PUBLIC_TRACKER_URL=https://ai.neuron.d-popov.com
+PUBLIC_PROXY_NETWORK=proxy_net
+```
+
+If your Gitea archive URL requires auth, either make an alpha release tarball downloadable to the Portainer host, or move to Option B and push a container image.
+
+Optional alpha/devnet variables:
+
+```text
+STARTING_CREDIT=1
+DEVNET_TOPUP=1
+HEARTBEAT_TIMEOUT=30
+```
+
+Set `STARTING_CREDIT=0` and `DEVNET_TOPUP=0` before any mainnet / real-money deployment.
+
+## Option B — recommended long-term: Gitea Container package
+
+Gitea Packages supports a Docker/OCI container registry. The package to create is a **Container Registry** package, not npm.
+
+Gitea docs:
+
+- Overview: https://docs.gitea.com/usage/packages/overview/
+- Container Registry: https://docs.gitea.com/usage/packages/container/
+
+For this repo, use an image name like:
+
+```text
+git.d-popov.com/popov/neuron-tai-tracker:alpha
+```
+
+or, if you prefer nested image names:
+
+```text
+git.d-popov.com/popov/neuron-tai/meshnet-tracker-relay:alpha
+```
+
+Gitea image naming rule is:
+
+```text
+{registry}/{owner}/{image}:{tag}
+```
+
+For us:
+
+```text
+registry = git.d-popov.com
+owner    = popov
+image    = neuron-tai-tracker
+label    = alpha
+```
+
+### 1. Create a Gitea token
+
+In Gitea:
+
+1. Open user settings.
+2. Go to Applications / Access Tokens.
+3. Create a token that can write packages for `popov`.
+4. Copy it once and store it safely.
+
+Do not commit the token into this repo or into the Portainer stack file.
+
+### 2. Login to the Gitea container registry
+
+From a machine with Docker and this repo checked out:
+
+```bash
+docker login git.d-popov.com
+```
+
+Username: your Gitea username.
+Password: the Gitea access token.
+
+If using 2FA/OAuth, Gitea docs recommend using a personal access token instead of your password.
+
+### 3. Build the image
+
+Run from repo root:
+
+```bash
+docker build \
+  -f deploy/docker/Dockerfile \
+  -t git.d-popov.com/popov/neuron-tai-tracker:alpha \
+  .
+```
+
+### 4. Push the image package to Gitea
+
+```bash
+docker push git.d-popov.com/popov/neuron-tai-tracker:alpha
+```
+
+After this, Gitea should show the package under the `popov` user/org packages.
+
+### 5. Use the image in Portainer
+
+In `meshnet-tracker-stack.yml`, replace the local build block:
+
+```yaml
+build:
+  context: ../..
+  dockerfile: deploy/docker/Dockerfile
+image: meshnet-tracker-relay:local
+```
+
+with:
+
+```yaml
+image: git.d-popov.com/popov/neuron-tai-tracker:alpha
+```
+
+If the package is private, configure Portainer registry credentials for `git.d-popov.com`:
+
+1. Portainer → Registries → Add registry.
+2. Type: Custom registry.
+3. Registry URL: `git.d-popov.com`.
+4. Username: your Gitea username.
+5. Password/token: the Gitea access token.
+6. Save, then deploy the stack.
+
+## Nginx Proxy Manager routing
+
+Use the Docker bridge network that your reverse proxy is already attached to.
+From the current Portainer network list, use:
+
+```text
+PUBLIC_PROXY_NETWORK=proxy_net
+```
+
+Do **not** use Docker's `host` network for the normal Portainer/Nginx Proxy Manager setup. The stack relies on Docker DNS names such as `meshnet-tracker`, and those work when the tracker and reverse proxy share a bridge network like `proxy_net`. Host networking is only useful for a special manual deployment where the container binds directly on the host and the proxy forwards to `127.0.0.1:<port>` or the host IP; that is less isolated and needs different compose settings (`network_mode: host`, no `networks:` block, and usually no service-name DNS).
+
+Create one Proxy Host for the public tracker domain.
+
+Default location `/`:
+
+```text
+Scheme: http
+Forward Hostname/IP: meshnet-tracker
+Forward Port: 8081
+Websockets Support: ON
+```
+
+Custom locations:
+
+| Location | Forward hostname | Forward port | WebSockets |
+| --- | --- | --- | --- |
+| `/ws` | `meshnet-tracker` | `8765` | ON |
+| `/rpc` | `meshnet-tracker` | `8765` | ON |
+
+Advanced tab if WebSocket upgrades fail:
+
+```nginx
+proxy_http_version 1.1;
+proxy_set_header Upgrade $http_upgrade;
+proxy_set_header Connection $http_connection;
+proxy_read_timeout 3600s;
+proxy_send_timeout 3600s;
+```
+
+## Portainer variables
+
+For both stacks:
+
+```text
+PUBLIC_TRACKER_URL=https://ai.neuron.d-popov.com
+PUBLIC_PROXY_NETWORK=proxy_net
+```
+
+For `meshnet-tracker-nobuild-stack.yml` only:
+
+```text
+SOURCE_TARBALL_URL=https://git.d-popov.com/popov/neuron-tai/archive/master.tar.gz
+SOURCE_STRIP_COMPONENTS=1
+```
+
+Useful optional variables:
+
+```text
+PUBLIC_RELAY_URL=wss://ai.neuron.d-popov.com/ws
+HEARTBEAT_TIMEOUT=30
+ENABLE_BILLING_DB=1
+MESHNET_WS_MAX_BYTES=268435456  # relay WebSocket frame cap (default 256 MiB; <=0 = unlimited)
+STARTING_CREDIT=1
+DEVNET_TOPUP=1
+```
+
+`PUBLIC_RELAY_URL` can usually be omitted; the stack derives it from `PUBLIC_TRACKER_URL` by changing `https://` to `wss://` and appending `/ws`.
+
+## Verify deployment
+
+From outside the Docker host:
+
+```bash
+curl -s https://ai.neuron.d-popov.com/v1/health
+curl -s https://ai.neuron.d-popov.com/v1/network/map | python3 -m json.tool
+```
+
+Expected in `/v1/network/map`:
+
+```json
+{
+  "relay_url": "wss://ai.neuron.d-popov.com/ws",
+  "relay": {
+    "mode": "embedded",
+    "url": "wss://ai.neuron.d-popov.com/ws",
+    "bind_host": "0.0.0.0",
+    "bind_port": 8765
+  }
+}
+```
+
+Then start a node from any NAT/WSL2 machine:
+
+```bash
+meshnet-node start --tracker https://ai.neuron.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct
+```
+
+The node should print:
+
+```text
+Relay advertised by tracker — using outbound tunnel wss://ai.neuron.d-popov.com/ws
+Relay connected — wss://ai.neuron.d-popov.com/rpc/<peer_id>
+```
+
+## Quick answer: npm or Gitea package?
+
+Use a Gitea **Container package** for Portainer.
+
+Do not use npm unless we later ship a JavaScript frontend package or Node.js CLI. It would not help the tracker/relay deployment.
+
+Recommended sequence:
+
+1. Deploy now with `meshnet-tracker-nobuild-stack.yml`.
+2. Build/push `git.d-popov.com/popov/neuron-tai-tracker:alpha` as a Gitea Container package.
+3. Switch Portainer to the image-based stack.
+4. Later automate build/push in CI.

deploy/portainer/meshnet-relay-only-stack.yml

@@ -0,0 +1,35 @@
+# Meshnet relay-only stack for Portainer.
+#
+# Use this when you want to run a relay-only node separately from the tracker.
+# The default alpha tracker stack embeds the same relay implementation in the
+# tracker process, so this file is optional until relay traffic needs its own
+# host/container.
+#
+# Intended topology for a relay-only public host:
+#   https://YOUR_DOMAIN/ws    -> meshnet-relay:8765  (WebSocket)
+#   https://YOUR_DOMAIN/rpc/* -> meshnet-relay:8765  (WebSocket)
+#
+# If the tracker is separate, start it with:
+#   --relay-url wss://YOUR_DOMAIN/ws
+
+services:
+  meshnet-relay:
+    image: ${MESHNET_IMAGE:-git.d-popov.com/popov/neuron-tai-tracker:alpha}
+    container_name: meshnet-relay
+    restart: unless-stopped
+    command: ["meshnet-relay", "--host", "0.0.0.0", "--port", "8765", "--log-level", "INFO"]
+    expose:
+      - "8765"
+    healthcheck:
+      test: ["CMD", "python", "-c", "import socket; s=socket.create_connection(('127.0.0.1', 8765), 3); s.close()"]
+      interval: 30s
+      timeout: 5s
+      retries: 3
+      start_period: 10s
+    networks:
+      - public-proxy
+
+networks:
+  public-proxy:
+    external: true
+    name: ${PUBLIC_PROXY_NETWORK:-proxy_net}

deploy/portainer/meshnet-tracker-nobuild-stack.yml

@@ -1,14 +1,14 @@
-# Meshnet public tracker + relay stack for Portainer without a custom image.
+# Meshnet public tracker stack for Portainer without a custom image.
 #
 # This stack does NOT use deploy/docker/Dockerfile and does NOT require pushing an
 # image to a registry. Each service starts from the public python:3.12-slim image,
 # downloads a source tarball, installs the tracker/relay packages into a named
-# venv volume, then starts the service.
+# venv volume, then starts the tracker with an embedded relay.
 #
 # Required Portainer variables:
 #   SOURCE_TARBALL_URL     URL to a .tar.gz archive of this repo
 #   PUBLIC_TRACKER_URL     e.g. https://cloud.neuron.d-popov.com
-#   PUBLIC_PROXY_NETWORK   Docker network shared with nginx/NPM, e.g. npm_proxy
+#   PUBLIC_PROXY_NETWORK   Docker network shared with nginx/NPM, e.g. proxy_net
 #
 # Optional:
 #   CLUSTER_PEERS          e.g. https://ai.neuron.d-popov.com
@@ -88,6 +88,9 @@ services:
           --heartbeat-timeout "$${HEARTBEAT_TIMEOUT}" \
           --self-url "$${PUBLIC_TRACKER_URL}" \
           --relay-url "$${RELAY_URL}" \
+          --embedded-relay \
+          --relay-host 0.0.0.0 \
+          --relay-port 8765 \
           --stats-db /var/lib/meshnet/tracker-stats.sqlite \
           --accounts-db /var/lib/meshnet/accounts.sqlite \
           --starting-credit "$${STARTING_CREDIT:-1}" \
@@ -100,49 +103,9 @@ services:
       - meshnet-tracker-venv:/opt/meshnet-venv
     expose:
       - "8081"
-    healthcheck:
-      test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8081/v1/health', timeout=3).read()"]
-      interval: 30s
-      timeout: 5s
-      retries: 3
-      start_period: 60s
-    networks:
-      - public-proxy
-
-  meshnet-relay:
-    image: python:3.12-slim
-    container_name: meshnet-relay
-    restart: unless-stopped
-    environment:
-      SOURCE_TARBALL_URL: ${SOURCE_TARBALL_URL:?set SOURCE_TARBALL_URL}
-      SOURCE_STRIP_COMPONENTS: ${SOURCE_STRIP_COMPONENTS:-1}
-    command:
-      - /bin/sh
-      - -lc
-      - |
-        set -eu
-        apt-get update
-        apt-get install -y --no-install-recommends ca-certificates curl tar
-        rm -rf /var/lib/apt/lists/*
-
-        rm -rf /opt/meshnet-src
-        mkdir -p /opt/meshnet-src
-        curl -fsSL "$${SOURCE_TARBALL_URL}" -o /tmp/meshnet-src.tar.gz
-        tar -xzf /tmp/meshnet-src.tar.gz -C /opt/meshnet-src --strip-components "$${SOURCE_STRIP_COMPONENTS:-1}"
-
-        python -m venv /opt/meshnet-venv
-        /opt/meshnet-venv/bin/python -m pip install --upgrade pip setuptools wheel
-        /opt/meshnet-venv/bin/pip install \
-          -e /opt/meshnet-src/packages/tracker \
-          -e /opt/meshnet-src/packages/relay
-
-        exec /opt/meshnet-venv/bin/meshnet-relay --host 0.0.0.0 --port 8765 --log-level INFO
-    volumes:
-      - meshnet-relay-venv:/opt/meshnet-venv
-    expose:
       - "8765"
     healthcheck:
-      test: ["CMD", "python", "-c", "import socket; s=socket.create_connection(('127.0.0.1', 8765), 3); s.close()"]
+      test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8081/v1/health', timeout=3).read()"]
       interval: 30s
       timeout: 5s
       retries: 3
@@ -153,9 +116,8 @@ services:
 volumes:
   meshnet-tracker-data:
   meshnet-tracker-venv:
-  meshnet-relay-venv:
 
 networks:
   public-proxy:
     external: true
-    name: ${PUBLIC_PROXY_NETWORK:-npm_proxy}
+    name: ${PUBLIC_PROXY_NETWORK:-proxy_net}

deploy/portainer/meshnet-tracker-stack.yml

@@ -1,10 +1,10 @@
-# Meshnet public tracker + relay stack for Portainer.
+# Meshnet public tracker stack for Portainer.
 #
 # Intended topology when Nginx Proxy Manager (or another nginx reverse proxy)
 # runs on the same Docker host:
 #   https://YOUR_DOMAIN/v1/*  -> meshnet-tracker:8081
-#   https://YOUR_DOMAIN/ws    -> meshnet-relay:8765  (WebSocket)
-#   https://YOUR_DOMAIN/rpc/* -> meshnet-relay:8765  (WebSocket)
+#   https://YOUR_DOMAIN/ws    -> meshnet-tracker:8765  (embedded relay WebSocket)
+#   https://YOUR_DOMAIN/rpc/* -> meshnet-tracker:8765  (embedded relay WebSocket)
 #
 # Before deploying, create or identify the Docker network shared with nginx/NPM,
 # then set PUBLIC_PROXY_NETWORK to its name in Portainer environment variables.
@@ -64,6 +64,9 @@ services:
           --heartbeat-timeout "$${HEARTBEAT_TIMEOUT}" \
           --self-url "$${PUBLIC_TRACKER_URL}" \
           --relay-url "$${RELAY_URL}" \
+          --embedded-relay \
+          --relay-host 0.0.0.0 \
+          --relay-port 8765 \
           --stats-db /var/lib/meshnet/tracker-stats.sqlite \
           --accounts-db /var/lib/meshnet/accounts.sqlite \
           $${BILLING_ARGS} \
@@ -73,27 +76,9 @@ services:
       - meshnet-tracker-data:/var/lib/meshnet
     expose:
       - "8081"
-    healthcheck:
-      test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8081/v1/health', timeout=3).read()"]
-      interval: 30s
-      timeout: 5s
-      retries: 3
-      start_period: 10s
-    networks:
-      - public-proxy
-
-  meshnet-relay:
-    image: meshnet-tracker-relay:local
-    container_name: meshnet-relay
-    restart: unless-stopped
-    depends_on:
-      meshnet-tracker:
-        condition: service_started
-    command: ["meshnet-relay", "--host", "0.0.0.0", "--port", "8765", "--log-level", "INFO"]
-    expose:
       - "8765"
     healthcheck:
-      test: ["CMD", "python", "-c", "import socket; s=socket.create_connection(('127.0.0.1', 8765), 3); s.close()"]
+      test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8081/v1/health', timeout=3).read()"]
       interval: 30s
       timeout: 5s
       retries: 3
@@ -107,4 +92,4 @@ volumes:
 networks:
   public-proxy:
     external: true
-    name: ${PUBLIC_PROXY_NETWORK:-npm_proxy}
+    name: ${PUBLIC_PROXY_NETWORK:-proxy_net}

docs/INSTALL_WINDOWS.md

@@ -103,8 +103,32 @@ Verify the install:
 
 ```bash
 meshnet-node --help
+python -c "import transformers; print(transformers.__version__)"
 ```
 
+`transformers` must be **≥ 5.12** for Qwen3.5/3.6-MoE models (older versions fail
+with `'Qwen3_5MoeConfig' object has no attribute 'vocab_size'`). If you install
+into an existing conda/miniforge env instead of a fresh venv, run
+`pip install -U transformers` there. The startup warning about
+`flash-linear-attention` / `causal-conv1d` ("fast path is not available") is
+harmless on CPU — those are optional GPU kernels.
+
+If you run the node from native Windows instead of WSL2, install Triton for
+Windows in the same environment:
+
+```powershell
+python -m pip install triton-windows
+```
+
+Without it, Qwen3.5/3.6-MoE startup can fail with the misleading message
+`Could not import module 'Qwen3_5MoeForCausalLM'`.
+
+**NVIDIA GPU on native Windows:** the CUDA fast path works — after
+`triton-windows`, install FLA with plain `pip install flash-linear-attention`
+(no `[cuda]` extra, no `causal-conv1d`; both are Linux-only packaging and fail
+on Windows). No CUDA toolkit / `nvcc` is needed. See the platform table in
+[QUICKSTART.md](../QUICKSTART.md#qwen3536-moe-notes) for details.
+
 ---
 
 ## Step 6 — Pre-download the model shard

docs/adr/0020-chat-streaming-live-progress-and-mixed-topology-routing.md

@@ -0,0 +1,127 @@
+# ADR-0020: Dashboard chat streaming, live request progress, and the mixed-topology routing flaw
+
+## Status: Accepted (chat/streaming/styles implemented); routing flaw documented, fix pending
+
+## Context
+
+Live alpha testing (2026-07-07) with `Qwen3.6-35B-A3B` split across two LAN nodes surfaced
+three UX gaps and one routing correctness flaw:
+
+1. **No visibility while a request is processing.** The Call wall showed
+   "no in-flight requests" during a 52-second generation. Cause: the dashboard chat sent
+   `stream: false`, and the tracker only emits `proxy progress` console events (the Call
+   wall's live-status source, `_tracker_log_proxy_progress`, `server.py` ~2199) for
+   **streamed** requests. Non-streamed proxying produces only
+   `route selected → connected → complete`, and short requests complete inside the
+   dashboard's 4-second poll window.
+2. **Chat did not stream.** The nodes support SSE token-by-token generation
+   (`generate_text_streaming`, hardened earlier for split shards), and the tracker proxy
+   passes `text/event-stream` through (`server.py` ~3256), but the chat panel blocked on
+   full JSON and showed nothing until completion.
+3. **Chat panel styles drifted.** The "new chat layout" redesign left hardcoded one-off
+   colors (`#1f4788`, `#2563b8`, `#10151d`, `#1a1012`, `#5c2020`, `#ffb4b4`) mixed with
+   the CSS custom-property palette.
+
+## Decisions
+
+### 1. Chat streams by default (SSE)
+
+`dashboard.html` `sendChat()` now sends `stream: true` and consumes the SSE body with a
+`ReadableStream` reader:
+
+- Assistant tokens render incrementally into the last bubble (direct DOM update, full
+  re-render only at boundaries), with a blinking `▍` cursor while streaming.
+- Chat status shows live progress: `generating… N tokens · X tok/s`.
+- The send button becomes a stop button (`■`) during generation, backed by an
+  `AbortController`; a stopped generation keeps the partial text.
+- Non-SSE responses (JSON fallback, errors) are still handled; `data: {"error": ...}`
+  stream events surface as error bubbles.
+- `streaming` flags are stripped when loading persisted sessions so an interrupted
+  generation never leaves a stuck cursor.
+
+### 2. Live in-flight visibility rides on streaming
+
+No tracker change was needed: because chat now streams, the tracker emits `proxy progress`
+events (throttled to stdout, updated in place in the console ring via
+`update_console_key`), and the existing Call wall state machine
+(`buildCallWallStates`) renders processing rows with live tokens/TPS/queue.
+
+**Known limitation (accepted):** non-streamed API requests still show no progress between
+`proxy connected` and `proxy complete` — there is nothing to report until the node
+returns. Callers wanting live visibility should use `stream: true`.
+
+### 3. Chat style tokens
+
+All chat colors route through `:root` custom properties (`--hover-bg`, `--chat-user-bg`
+`#1f6feb`, `--chat-user-border`, `--chat-error-bg/border/fg`). No hardcoded hex values
+remain in chat rules, so future palette changes are single-line edits.
+
+## Documented flaw: mixed-topology routing (partial GPU head + full CPU node)
+
+### Observed (2026-07-07, tracker 192.168.0.179:8080)
+
+Two nodes registered for `qwen3.6-35b-a3b`:
+
+| node | hardware | shard | benchmark |
+|---|---|---|---|
+| `5gMLrmyB-ec3afe6f1a03` (192.168.0.20) | RTX 4060, CUDA | 0–21 (partial, fast) | 11,164 |
+| `7j77FsPY-55249b0583e5` (192.168.0.179) | CPU | 0–39 (full, slow) | 425 |
+
+When the tracker selected the GPU node as head, it injected:
+
+```
+downstream=[{"endpoint": "http://192.168.0.179:7000", "start_layer": 0}]
+```
+
+`start_layer: 0` — not 22. The downstream full node re-ran **all 40 layers from layer 0
+on hidden states that had already passed through the head's layers 0–21**, producing
+garbage logits. Evidence from the logs:
+
+- GPU-headed requests: `generation complete tokens=1` and billed `out=0`/`out=1`/`out=3`
+  — near-instant EOS from corrupt activations.
+- The same prompt routed directly to the CPU full node: 209 tokens over 52 s (healthy).
+- Observed TPS for GPU-headed requests was meaningless (2.5–19.0 "tok/s" on 0–3 token
+  outputs), and those samples now pollute the rolling per-`(node, model)` throughput
+  stats used for routing preference.
+- Clients were **billed** for these broken 1-token responses.
+
+### Root cause
+
+The route planner treats the full-coverage node as a standalone complete route
+(`route=7j77FsPY…[0-39]`) but still injects it as the head's downstream with the
+downstream node's own `shard_start` (0) instead of `head.shard_end + 1` (22). A partial
+head + full-model downstream is a topology the planner never had to handle before —
+prior split tests used disjoint shards (0–11 + 12–23) where `shard_start` happened to
+equal the correct continuation layer.
+
+### Required fix (not yet implemented)
+
+1. **Correct continuation layer:** when hop N ends at layer `e`, hop N+1 must execute
+   from `start_layer = e + 1` regardless of the downstream node's own `shard_start`
+   (the `X-Meshnet-Start-Layer` overlapping-shard mechanism from ADR-0012 exists for
+   exactly this; the planner must set it for full-model downstream nodes too).
+2. **Route preference sanity:** with a healthy single-node full route available, prefer
+   it over a multi-hop route unless the pipeline is estimated faster; a fast head that
+   forces a slow full-model tail wins nothing (every token still crosses the CPU node).
+3. **Stat hygiene:** exclude or flag throughput samples from responses with ≤ a few
+   output tokens, so broken routes don't skew routing preference.
+4. **Billing guard (consider):** suspiciously short completions from multi-hop routes
+   during this window were billed; a minimum-viability check (or refund path) may be
+   warranted once audits land.
+
+### Verification for the fix
+
+Reproduce with a partial GPU head (0–21) + full CPU node (0–39): a chat request routed
+through the GPU head must produce output equivalent to the direct CPU route, with
+`downstream start_layer=22` visible in `proxy route selected`, and multi-token streamed
+output on the Call wall.
+
+## Verification of this ADR's implemented changes
+
+- `pytest tests/test_dashboard.py` — 5 passed (stale "Chat / inference" panel assertion
+  updated to the tabbed layout).
+- Embedded dashboard JS parses (`new Function(script)` under Node 22).
+- Live check: open `/dashboard` → Chat, send a prompt to `qwen3.6-35b-a3b` — tokens
+  must appear incrementally with live tok/s in the status line, the Call wall must show
+  the request as `processing` with live TPS, and the send button must stop generation
+  mid-stream keeping partial text.

docs/adr/0021-dynamic-statistical-routing.md

@@ -0,0 +1,119 @@
+# ADR-0021: Dynamic statistical routing (bandit-style route selection)
+
+## Status: Accepted, implemented
+
+## Context
+
+ADR-0020 documented the mixed-topology flaw: with a fast GPU node serving layers 0–21 and
+a slow CPU node serving 0–39 of `Qwen3.6-35B-A3B`, the tracker picked the GPU node as
+proxy head *independently* of route planning, injecting a downstream hop with the wrong
+`start_layer` (0 instead of 22) and corrupting generation.
+
+Beyond the bug, the deeper issue is that the tracker **cannot know a priori** which route
+is faster. Is one CPU node running all 40 layers faster than a GPU running 0–21 plus a
+CPU hop for 22–39? Benchmarks don't answer that — network hops, MoE expert loading, and
+queue dynamics only show up in real end-to-end requests. The router must *measure*.
+
+## Decision
+
+Route selection is a **multi-armed bandit** over enumerated candidate routes, implemented
+in `packages/tracker/meshnet_tracker/routing_stats.py` and wired into the chat proxy in
+`server.py`.
+
+### Arms: route signatures
+
+A route's identity is `model_key | node_id[shard] -> node_id[shard] -> …`. Node ids embed
+wallet + shard, so a node re-registering with a different shard produces a new arm
+automatically. The proxy target is **always the route's own head** (`route_nodes[0]`),
+and each hop's `start_layer` is `previous_hop.shard_end + 1` — this fixes ADR-0020's flaw
+structurally: head choice and route planning can no longer disagree.
+
+### Candidate enumeration (`_enumerate_routes`)
+
+One candidate per distinct head (a node whose `shard_start` equals the model's first
+layer — it must tokenize/embed), greedily completed with longest-advancing hops. Each
+candidate carries a `prior_tps`: its bottleneck hop's queue-adjusted effective throughput
+× reputation. Capped at 8 candidates ranked by prior.
+
+### Statistics: decayed EWMA + topology epochs
+
+Per (model, signature), `RouteStatsStore` keeps an EWMA of observed end-to-end tokens/sec
+with **time-decayed sample mass** (half-life default 600 s). Two staleness mechanisms
+handle the morphing network:
+
+- **Continuous**: sample mass decays; a route unproven for a while (mass < 0.5) drops out
+  of the exploit pool and gets re-scouted.
+- **Abrupt**: any node join/leave/shard-change bumps the model's *topology epoch*. Stats
+  from an older epoch keep their EWMA as a display prior but are demoted to the scout
+  pool ("stale") until re-measured under the new topology.
+
+Sample hygiene: completions below `min_sample_tokens` (default 8) are rejected — the
+1-token garbage responses from the ADR-0020 bug would otherwise poison arms with
+meaningless tps values. Routes with no samples for 24 h are pruned.
+
+### Selection policy (`choose_route`)
+
+1. **Scout** (probability `explore_share`, default 0.3): if any candidate is unproven /
+   stale / decayed, route the request there — least-measured first, tiebreak on prior.
+   These are the user's "discovery/scout routes". With *no* proven arms at all, selection
+   is deterministic best-prior (matches the old benchmark-based behavior, keeps cold
+   start sane and tests deterministic).
+2. **Exploit** (otherwise): weighted random among proven arms with
+   `P(route) ∝ tps^alpha`, `alpha` default 1.0 — a 1.5×-faster route gets 1.5× the
+   traffic. `alpha` is a config knob: >1 shifts toward winner-takes-most as the network
+   matures, without redesign. (Proportional split is not throughput-optimal in queueing
+   terms, but it keeps every arm warm with fresh samples; tune alpha up when traffic
+   justifies it.)
+
+Pinned routes (`"route": [...]` in the request body) bypass the bandit but still record
+samples.
+
+### Configuration
+
+| CLI flag | env var | default |
+|---|---|---|
+| `--route-explore-share` | `MESHNET_ROUTE_EXPLORE_SHARE` | 0.3 |
+| `--route-weight-alpha` | `MESHNET_ROUTE_WEIGHT_ALPHA` | 1.0 |
+| `--route-stats-half-life` | `MESHNET_ROUTE_STATS_HALF_LIFE` | 600 |
+| — | `MESHNET_ROUTE_MIN_SAMPLE_TOKENS` | 8 |
+
+High explore share now (development, few requests); drop toward 0.05–0.1 once real
+traffic provides passive coverage.
+
+### Visibility
+
+- **`GET /v1/routing`** (optionally `?model=`): per model — topology epoch and the full
+  candidate table: hops, learned tps, **coefficient** (tps ÷ best proven route's tps),
+  **expected traffic share**, sample count, decayed weight, status
+  (proven / unsampled / stale / decayed).
+- **Dashboard → Overview → "Routing (learned)"**: renders that table live (4 s poll),
+  with the active config in the header line.
+- **Console/`proxy route selected`** events now include the routing decision
+  (`{"mode": "scout"|"exploit"|"pinned"|"greedy-fallback", "signature": …}`), so the Call
+  wall history shows which arm served each request.
+
+## Storage considerations
+
+Stats are **in-memory per tracker** for alpha: they are cheap to relearn (a few requests
+per route), and gossiping them would import ADR-0019's consistency questions for data
+that is intentionally ephemeral. If multi-tracker route learning is needed later, ship
+route samples over the existing stats gossip and merge EWMAs by decayed weight — the
+store's (value, mass, timestamp) representation merges cleanly.
+
+## Consequences
+
+- The GPU(0–21)+CPU(0–39) topology now works: both routes get measured, the coefficient
+  is visible on the dashboard, and traffic shifts to whichever is actually faster.
+- Routing is no longer deterministic once samples exist. Tests needing determinism seed
+  `server.route_rng` or rely on the cold-start deterministic path.
+- The billing-relevant fix: heads are always part of the planned route, so per-hop
+  `start_layer` and work-unit spans are consistent.
+
+## Verification
+
+`tests/test_dynamic_routing.py` (11 tests): EWMA/decay/epoch semantics, near-empty sample
+rejection, traffic split ≈ tps ratio at alpha=1 (0.6/0.4 over 4000 seeded draws), scout
+rate ≈ explore share, mixed-topology enumeration (both routes, hybrid prior = bottleneck),
+head-is-route-head regression with `start_layer=22` on the hybrid route, and `/v1/routing`
+table shape. Live: start both nodes, run several chats, open the dashboard "Routing
+(learned)" panel and watch coefficients converge.

docs/adr/0022-sharded-per-node-generation-cache.md

@@ -0,0 +1,63 @@
+# ADR-0022: Sharded per-node generation cache for distributed PyTorch routes
+
+## Status: Accepted
+
+## Context
+
+The distributed PyTorch chat path previously recomputed the full prompt-so-far for
+every generated token. The head shard embedded the entire sequence each step, forwarded
+full-sequence activations through every downstream shard, and every shard called its
+decoder layers with `use_cache=False`. On a two-node Qwen2.5-0.5B route this produced
+the expected quadratic slowdown as output length grew.
+
+ADR-0020 and ADR-0021 fixed route construction and `start_layer` semantics. They did not
+define the per-request cache lifecycle needed for efficient decode.
+
+## Decision
+
+Distributed PyTorch generation now uses one stable route session id for an entire chat
+request. The wire protocol marks each activation hop with:
+
+- `X-Meshnet-Session`: stable per generation.
+- `X-Meshnet-Cache-Mode`: `prefill`, `decode`, or `stateless`.
+- `X-Meshnet-Seq-Len`: the total sequence length represented by the step.
+
+Step 0 is prefill: the head sends the full prompt activation through the planned route.
+Each shard stores only the opaque cache state returned by its own executed layer range.
+No shard receives or stores another shard's cache.
+
+Later cached decode steps send only the newest token activation (`[1, 1, hidden]`) with
+the full sequence length and newest position id. The backend deliberately treats layer
+cache state as opaque. Standard K/V tuples, HuggingFace cache objects, and hybrid
+linear-attention recurrent state are stored without shape assumptions.
+
+## Cache lifecycle
+
+Each `TorchModelShard` owns an in-memory LRU map keyed by
+`(session_id, effective_start_layer, shard_end)`. Entries expire by TTL and by a maximum
+session count (`MESHNET_SHARD_CACHE_TTL_SECONDS`, default 600;
+`MESHNET_SHARD_CACHE_MAX_SESSIONS`, default 16).
+
+If a decode step reaches a node after restart, eviction, TTL expiry, or route mismatch,
+the node returns an explicit `cache_miss` response. The head falls back to full prefill
+for the current prompt-so-far using the same session id, rebuilding the shard-local
+caches before continuing. Alpha route repair still does not migrate cache state across
+nodes; a true route change is treated as cache loss and recovered by re-prefill.
+
+## Consequences
+
+- Healthy decode sends O(1) activation payloads per token between nodes instead of
+  O(sequence length).
+- Cache internals stay behind the model backend boundary, which keeps Qwen3.6-style
+  hybrid recurrent cache state compatible with the same route protocol.
+- Restart and eviction degrade to slower stateless/full-prefill work rather than silent
+  output corruption.
+- Cross-node cache migration, batching cache state across sessions, and speculative
+  decoding remain future work.
+
+## Verification
+
+Unit coverage in `tests/test_real_model_backend.py` verifies opaque per-layer cache
+storage, cached one-token decode, explicit cache-miss errors, and LRU eviction. Live
+two-node Qwen2.5-0.5B TPS measurement still requires the physical two-machine topology
+used to observe the regression.

docs/adr/0022-sharded-per-node-kv-cache.md

@@ -0,0 +1,102 @@
+# ADR-0022: Sharded per-node KV cache for distributed generation
+
+## Status: Accepted, implemented (alpha-hardening issue 25)
+
+## Context
+
+The distributed generation loop (`torch_server.py`, `_do_chat_completions` distributed
+path) had **no KV cache**: every layer-forward call passed `use_cache: False`, and each
+autoregressive step re-encoded the entire prompt-so-far from scratch, re-running every
+layer on every node in the route for every generated token. Measured on a live 2-node
+Qwen2.5-0.5B GPU pipeline: tps decayed from 22.3 to 12.6 within a single generation —
+the quadratic-cost signature. On Qwen3.6-35B-A3B mixed GPU/CPU topology this collapsed
+to ~0.07 tps even after the ADR-0020 routing fix.
+
+`X-Meshnet-Session` existed on the wire but was minted fresh **per token** and keyed no
+state.
+
+## Decision
+
+### Session lifecycle
+
+The head mints one session id per chat generation (not per token) and reuses it across
+every step. Two new request headers extend the `/forward` wire protocol:
+
+- `X-Meshnet-Cache: prefill | decode` — absent means legacy stateless (unchanged
+  behavior, and what old nodes send/understand).
+- `X-Meshnet-Past-Len: N` — decode only: the number of tokens the node's session cache
+  must already hold. A mismatch is a cache miss, never silent corruption.
+
+Step 0 (`prefill`) sends the full prompt activation as before; each node creates fresh
+session state for its own layer range. Steps 1+ (`decode`) send only the newest token's
+hidden state — `[1, 1, hidden]`, cutting per-step compute and wire payload from
+O(seq_len) to O(1). The head embeds the next token directly from the `token_id` the tail
+now returns alongside text (`{"text": …, "token_id": …}`), avoiding text
+re-tokenization drift; EOS is detected by id against tokenizer + generation-config eos
+sets.
+
+### Per-node sharded cache
+
+`TorchModelShard.kv_sessions` is a `SessionCacheStore`: `session_id → SessionCacheEntry`
+holding cache state **only for that shard's layer range** — sharding falls out naturally
+because each node only executes (and therefore only caches) its own layers. No node ever
+holds another node's state.
+
+### MoE / hybrid-attention awareness
+
+The cached object is whatever `use_cache=True` produces: a transformers
+`DynamicCache(config=model.config)` — the same construction the model's own `forward()`
+uses. With the config, transformers picks the right per-layer state: K/V tensors for
+standard attention, conv/recurrent delta state for Qwen3.6-style hybrid linear-attention
+layers, sliding-window variants, etc. The store treats it as opaque; nothing assumes a
+K/V tensor shape. Cache slots are indexed by absolute `layer_idx`, so a shard updating
+only layers 12–23 leaves 0–11 empty (verified: sparse `DynamicCache.update` works).
+MoE expert routing is layer-local per token and needs no cross-token state.
+
+Layers are invoked with `past_key_values=<cache>, use_cache=True, cache_position=…`
+(transformers 5.x layer API; the cache is mutated in place). If a model's layers reject
+those kwargs, the backend logs once, sets `supports_kv_cache = False`, and stays on the
+stateless path permanently — exotic architectures degrade to today's behavior instead of
+failing.
+
+### Cache miss and route-change interaction (ADR-0021)
+
+Any decode-mode request that cannot be served — unknown session (evicted, node
+restarted), `past_len` mismatch, `start_layer` mismatch (the route or shard overlap
+changed mid-generation), or caching disabled — raises `KVCacheMiss`, answered as
+**HTTP 409 `{"error": "cache_miss"}`**. The head catches it and falls back to one full
+re-prefill of the accumulated sequence under the same session id, which atomically
+replaces every node's session state, then continues cached. The fully-stateless path is
+therefore still the recovery mode: eviction and restarts cost one prefill, never
+corruption or a failed generation. A decode request against a node whose caching is
+disabled is also a 409 — running a single-token payload statelessly would silently
+produce garbage.
+
+Mixed fleets degrade the same way: if the tail predates the protocol and returns no
+`token_id`, the head simply prefills every step (exactly the old cost).
+
+### Bounded memory
+
+`SessionCacheStore` enforces TTL (default 600 s, `MESHNET_KV_TTL_SECONDS`) plus LRU cap
+(default 8 sessions, `MESHNET_KV_MAX_SESSIONS`), evaluated on every access. The head
+additionally drops its own session explicitly when a generation completes; downstream
+nodes rely on TTL/LRU (an explicit cross-node release RPC was judged not worth the
+failure modes — misses are cheap).
+
+### Non-goals (first landing)
+
+Cross-node cache migration on route change (evict + re-prefill is acceptable),
+speculative decoding, cross-session batching.
+
+## Consequences
+
+- Per-token cost drops from O(seq_len) layer re-execution + O(seq_len) wire transfer per
+  hop to O(1) of both; tps stays flat across generation length instead of decaying.
+- Golden test (`tests/test_kv_cache_distributed.py`, env-gated by
+  `MESHNET_REAL_MODEL_TESTS=1`) proves cached and stateless distributed generation emit
+  identical token ids on a real two-shard Qwen2.5-0.5B split.
+- Nodes now hold per-session GPU/CPU memory between requests (bounded above); operators
+  sizing `max_loaded_shards` should account for ~`sessions × seq_len × kv_bytes_per_token`
+  per resident model.
+- The wire protocol is backward- and forward-compatible: headers are additive, absent
+  headers mean stateless, and 409 is only sent in reply to explicit decode-mode requests.

docs/adr/0023-model-agnostic-node-capability-admission.md

@@ -0,0 +1,26 @@
+# ADR-0023: Model-agnostic Node capability admission
+
+## Status: Accepted (P0 planned)
+
+## Context
+
+A Node currently inventories hardware, benchmarks a generic Torch operation, loads its model, registers with the Tracker, and can be routed before its exact model/backend path has completed a bounded real forward. Optional JIT or model-kernel failures can therefore surface only after a live `/forward` request reaches the Node.
+
+This is incompatible with a consumer-grade node experience. A Node must never advertise a Shard it cannot actually execute. The solution must not be coupled to a development model; model-specific hardcoding would recreate the support burden for every new Model Artifact.
+
+## Decision
+
+- Introduce a generic versioned capability report keyed by Model Artifact identity, Shard range, named recipe, backend/device identity, and local validation result.
+- A recipe is data and can be one of several possible execution paths for the same Model Preset. Every recipe validates itself using a bounded real forward.
+- `meshnet-node doctor` validates the selected model/shard by default. An explicit all-recipes mode supports CI and diagnosis.
+- Startup fails closed for an explicitly selected Model Preset when no matching recipe validates. The Node must not become routable or accept paid work.
+- Nodes register only locally validated capabilities. The Tracker routes only matching validated capabilities and uses measured performance as part of normal route selection.
+- P0 carries the version of a local recipe manifest. New executable recipes arrive only through signed Node releases in a future feature. P0 does not download executable recipes, dynamically install dependencies, install OS packages/drivers, or implement an updater.
+- A future Tracker-provided Model Artifact Manifest may be signed data only; it cannot instruct a Node to execute arbitrary code.
+
+## Consequences
+
+- First startup has a bounded validation cost before registration, but failures occur before traffic rather than under a paid request.
+- The registration and routing protocols gain compatibility/capability fields and require a transition policy for older Nodes.
+- Hardware support claims become evidence-based and can be tested independently of specific development models.
+- The signed Node update channel is deliberately deferred until this capability contract is stable.

docs/dev/_NOTES.md

@@ -0,0 +1,11 @@
+# IDEAS
+
+ - use real torrenting library/ infrastructure
+
+
+
+
+  let's work on the ability to have the tracker on the interet. i want to realease the alpha version to see the first feedbacks. we have to check if we need a  relay
+ node, and is it working. or it will be more practical if the the tracker integrates the relay 
+functionality as well. I'd say keep it separate, as we may have relay only nodes, but a tracker also is a relay -it uses the same code/class for the relay to expose it on it's api as well - 
+a good architecture

docs/issues/30-manual-route-and-hop-benchmark.md

@@ -1,48 +1,48 @@
-# US-020 — Manual route selection + hop-penalty benchmarking
-
-## Context
-
-The tracker auto-selects inference routes based on synthetic benchmark scores. To measure
-the real cost of adding hops (latency per node boundary), we need:
-
-1. A way to pin a request to a specific route so we control the variable.
-2. A benchmark endpoint that runs the same prompt through 1-node, 2-node, and 3-node
-   routes and records per-hop latency.
-
-Results are stored to disk. Routing algorithm is **not** changed in this story — this is
-data collection only. The data will inform a future routing optimisation story.
-
-## Design decisions (grilled 2026-07-01)
-
-| Decision | Choice |
-|---|---|
-| Route spec | Optional `route` field in JSON request body (list of node IDs) |
-| Trigger | Explicit only — `POST /v1/benchmark/hop-penalty` endpoint |
-| Auth | Header-presence stub (`Authorization` must be non-empty); real auth in future story |
-| Routing integration | Store data only; routing algorithm unchanged |
-| Persistence | Append to `benchmark_results.json` in tracker working dir; in-memory queryable |
-
-## Acceptance criteria
-
-- `POST /v1/chat/completions` accepts optional `"route": ["<node_id>", ...]` in the
-  request body. If present, the tracker uses those nodes in order instead of auto-selecting.
-  If absent, existing routing is unchanged (no breaking change for unaware clients).
-- Missing or invalid node IDs in `route` return HTTP 400 with a descriptive error.
-- `POST /v1/benchmark/hop-penalty` is auth-gated: requests without a non-empty
-  `Authorization` header return HTTP 401. Body: `{"model": "...", "prompt": "...",
-  "max_new_tokens": 64}`.
-- Benchmark fans out to up to three routes: 1-node (single node covering all layers),
-  2-node (two consecutive shard nodes), 3-node (three nodes) — using whatever is
-  currently registered. Routes with insufficient coverage are skipped, not errored.
-- Response includes per-route breakdown: `total_ms`, `per_hop_ms: [...]`,
-  `tokens_generated`, `route: [node_id, ...]`.
-- Results are appended to `<tracker_working_dir>/benchmark_results.json` (created if
-  absent) as a JSON array. Each entry includes timestamp, model, prompt hash, and the
-  per-route breakdown.
-- `GET /v1/benchmark/results` returns the stored results array. Also auth-gated.
-- Clients that never send `route` or call `/v1/benchmark/*` are completely unaffected.
-- Integration test: send the same prompt via a pinned 1-node route and a pinned 2-node
-  route; assert 2-node result has 2 entries in `per_hop_ms`; assert both records appear
-  in `benchmark_results.json`.
-- `python -m pytest` passes from repo root.
-- Commit only this story's changes.
+# US-020 — Manual route selection + hop-penalty benchmarking
+
+## Context
+
+The tracker auto-selects inference routes based on synthetic benchmark scores. To measure
+the real cost of adding hops (latency per node boundary), we need:
+
+1. A way to pin a request to a specific route so we control the variable.
+2. A benchmark endpoint that runs the same prompt through 1-node, 2-node, and 3-node
+   routes and records per-hop latency.
+
+Results are stored to disk. Routing algorithm is **not** changed in this story — this is
+data collection only. The data will inform a future routing optimisation story.
+
+## Design decisions (grilled 2026-07-01)
+
+| Decision | Choice |
+|---|---|
+| Route spec | Optional `route` field in JSON request body (list of node IDs) |
+| Trigger | Explicit only — `POST /v1/benchmark/hop-penalty` endpoint |
+| Auth | Header-presence stub (`Authorization` must be non-empty); real auth in future story |
+| Routing integration | Store data only; routing algorithm unchanged |
+| Persistence | Append to `benchmark_results.json` in tracker working dir; in-memory queryable |
+
+## Acceptance criteria
+
+- `POST /v1/chat/completions` accepts optional `"route": ["<node_id>", ...]` in the
+  request body. If present, the tracker uses those nodes in order instead of auto-selecting.
+  If absent, existing routing is unchanged (no breaking change for unaware clients).
+- Missing or invalid node IDs in `route` return HTTP 400 with a descriptive error.
+- `POST /v1/benchmark/hop-penalty` is auth-gated: requests without a non-empty
+  `Authorization` header return HTTP 401. Body: `{"model": "...", "prompt": "...",
+  "max_new_tokens": 64}`.
+- Benchmark fans out to up to three routes: 1-node (single node covering all layers),
+  2-node (two consecutive shard nodes), 3-node (three nodes) — using whatever is
+  currently registered. Routes with insufficient coverage are skipped, not errored.
+- Response includes per-route breakdown: `total_ms`, `per_hop_ms: [...]`,
+  `tokens_generated`, `route: [node_id, ...]`.
+- Results are appended to `<tracker_working_dir>/benchmark_results.json` (created if
+  absent) as a JSON array. Each entry includes timestamp, model, prompt hash, and the
+  per-route breakdown.
+- `GET /v1/benchmark/results` returns the stored results array. Also auth-gated.
+- Clients that never send `route` or call `/v1/benchmark/*` are completely unaffected.
+- Integration test: send the same prompt via a pinned 1-node route and a pinned 2-node
+  route; assert 2-node result has 2 entries in `per_hop_ms`; assert both records appear
+  in `benchmark_results.json`.
+- `python -m pytest` passes from repo root.
+- Commit only this story's changes.

packages/gateway/meshnet_gateway/server.py

@@ -286,7 +286,7 @@ class _GatewayHandler(http.server.BaseHTTPRequestHandler):
             self._send_json(200, completion)
 
     def _proxy_to_head_worker(self, url: str, body_bytes: bytes) -> None:
-        """Forward a raw request body to a head worker and stream the response back."""
+        """Forward a raw request body to a head worker and relay SSE without buffering."""
         target_url = f"{url}/v1/chat/completions"
         req = urllib.request.Request(
             target_url,
@@ -297,6 +297,19 @@ class _GatewayHandler(http.server.BaseHTTPRequestHandler):
         try:
             with urllib.request.urlopen(req, timeout=30.0) as r:
                 content_type = r.headers.get("Content-Type", "application/json")
+                if "text/event-stream" in content_type:
+                    self.send_response(r.status)
+                    self.send_header("Content-Type", content_type)
+                    self.send_header("Cache-Control", "no-cache")
+                    self.send_header("X-Accel-Buffering", "no")
+                    self.end_headers()
+                    while True:
+                        line = r.readline()
+                        if not line:
+                            break
+                        self.wfile.write(line)
+                        self.wfile.flush()
+                    return
                 resp_body = r.read()
                 status = r.status
         except urllib.error.HTTPError as exc:

packages/node/meshnet_node/cli.py

@@ -68,6 +68,8 @@ def _run_node(cfg: dict) -> None:
             tracker_source_disabled=bool(cfg.get("tracker_source_disabled", False)),
             torch_threads=cfg.get("torch_threads"),
             torch_interop_threads=cfg.get("torch_interop_threads"),
+            node_name=cfg.get("node_name"),
+            force_cpu=bool(cfg.get("force_cpu", False)),
         )
     except Exception as exc:
         print(f"\nERROR: {exc}", file=sys.stderr, flush=True)
@@ -157,6 +159,8 @@ def _cmd_default(args) -> int:
         overrides["host"] = args.host
     if args.advertise_host:
         overrides["advertise_host"] = args.advertise_host
+    if getattr(args, "node_name", None):
+        overrides["node_name"] = args.node_name
     if args.route_timeout != 30.0:
         overrides["route_timeout"] = args.route_timeout
     if getattr(args, "memory", None) is not None:
@@ -171,6 +175,8 @@ def _cmd_default(args) -> int:
         overrides["torch_threads"] = args.torch_threads
     if getattr(args, "torch_interop_threads", None) is not None:
         overrides["torch_interop_threads"] = args.torch_interop_threads
+    if getattr(args, "cpu", False):
+        overrides["force_cpu"] = True
 
     if overrides:
         cfg = merge_cli_overrides(cfg, **overrides)
@@ -246,6 +252,8 @@ def _cmd_start(args) -> int:
         cfg["wallet_path"] = args.wallet
     if args.download_dir:
         cfg["download_dir"] = args.download_dir
+    if getattr(args, "node_name", None):
+        cfg["node_name"] = args.node_name
 
     # Legacy start: just run without the dashboard (keep original blocking loop)
     from .startup import run_startup
@@ -270,6 +278,8 @@ def _cmd_start(args) -> int:
             tracker_source_disabled=getattr(args, "tracker_source_disabled", False),
             torch_threads=getattr(args, "torch_threads", None),
             torch_interop_threads=getattr(args, "torch_interop_threads", None),
+            node_name=cfg.get("node_name"),
+            force_cpu=getattr(args, "cpu", False),
         )
     except Exception as exc:
         print(f"ERROR: {exc}", file=sys.stderr, flush=True)
@@ -315,6 +325,7 @@ def main() -> None:
     parser.add_argument("--port", type=int, metavar="N", help="Port to listen on")
     parser.add_argument("--host", metavar="ADDR", help="Interface to bind (default 0.0.0.0)")
     parser.add_argument("--advertise-host", metavar="ADDR", help="Host/IP advertised to the tracker")
+    parser.add_argument("--node-name", metavar="NAME", help="Friendly display name shown on the tracker dashboard")
     parser.add_argument("--route-timeout", type=float, metavar="SEC", default=30.0,
                         help="Seconds to wait for tracker route lookup (default 30)")
     parser.add_argument("--memory", type=int, metavar="MB", default=None,
@@ -325,6 +336,8 @@ def main() -> None:
                         help="Set PyTorch intra-op CPU worker threads")
     parser.add_argument("--torch-interop-threads", type=int, metavar="N",
                         help="Set PyTorch inter-op CPU worker threads")
+    parser.add_argument("--cpu", action="store_true",
+                        help="Force CPU inference even when a GPU is available")
     parser.add_argument("--debug", action="store_true", help="Enable verbose node debug logging")
     parser.add_argument("--no-tui", action="store_true", help="Plain-text output (no rich dashboard)")
     parser.add_argument("--compact", action="store_true", help="Single-line status output")
@@ -350,6 +363,7 @@ def main() -> None:
     start_cmd.add_argument("--quantization", choices=["auto", "bfloat16", "int8", "nf4", "bf16"], default="auto")
     start_cmd.add_argument("--host", default="0.0.0.0")
     start_cmd.add_argument("--advertise-host")
+    start_cmd.add_argument("--node-name", help="Friendly display name shown on the tracker dashboard")
     start_cmd.add_argument("--tracker-mode", action="store_true")
     start_cmd.add_argument("--tracker-url", default=None)
     start_cmd.add_argument("--wallet")
@@ -364,6 +378,8 @@ def main() -> None:
                            help="Set PyTorch intra-op CPU worker threads")
     start_cmd.add_argument("--torch-interop-threads", type=int, metavar="N",
                            help="Set PyTorch inter-op CPU worker threads")
+    start_cmd.add_argument("--cpu", action="store_true",
+                           help="Force CPU inference even when a GPU is available")
     start_cmd.add_argument("--debug", action="store_true", help="Enable verbose node debug logging")
     start_cmd.add_argument("--tracker-source-disabled", action="store_true",
                            help="Skip tracker/peer model-file sources and download from HuggingFace directly")

packages/node/meshnet_node/hardware.py

@@ -123,6 +123,24 @@ def _detect_nvidia_smi_gpu_memory() -> dict | None:
     return None
 
 
+def _detect_torch_cuda_inventory(torch_module) -> dict | None:
+    """Return torch-visible CUDA/HIP GPU metadata without running kernels."""
+    try:
+        if not torch_module.cuda.is_available() or torch_module.cuda.device_count() < 1:
+            return None
+        idx = torch_module.cuda.current_device()
+        name = torch_module.cuda.get_device_name(idx)
+        props = torch_module.cuda.get_device_properties(idx)
+        vram_mb = int(props.total_memory // (1024 * 1024))
+        gpu = {"gpu_name": name, "vram_mb": max(0, vram_mb)}
+        gcn_arch = getattr(props, "gcnArchName", None)
+        if gcn_arch:
+            gpu["gcn_arch"] = str(gcn_arch)
+        return gpu
+    except Exception:
+        return None
+
+
 def _torch_cuda_is_executable(torch_module) -> bool:
     """Return True only if this Python process can execute a CUDA tensor op."""
     try:
@@ -139,7 +157,7 @@ def _torch_cuda_is_executable(torch_module) -> bool:
 def _gpu_inventory_profile(gpu: dict | None, ram_mb: int) -> dict | None:
     if gpu is None:
         return None
-    return {
+    profile = {
         "device": "cpu",
         "gpu_name": gpu["gpu_name"],
         "vram_mb": gpu["vram_mb"],
@@ -148,20 +166,38 @@ def _gpu_inventory_profile(gpu: dict | None, ram_mb: int) -> dict | None:
         "ram_mb": ram_mb,
         "cuda_available": False,
     }
+    if gpu.get("gcn_arch"):
+        profile["gcn_arch"] = gpu["gcn_arch"]
+    return profile
+
+
+def with_forced_cpu(hw: dict) -> dict:
+    """Return a hardware profile forced to CPU execution.
+
+    Keeps detected GPU metadata for diagnostics and tracker registration context,
+    but clears CUDA availability so startup and the model backend stay on CPU.
+    """
+    forced = dict(hw)
+    forced["device"] = "cpu"
+    forced["cuda_available"] = False
+    return forced
 
 
 def detect_hardware() -> dict:
     """Detect GPU model and available VRAM. Returns hardware profile dict."""
     ram_mb = _detect_ram_mb()
+    torch_gpu: dict | None = None
     try:
         import torch  # type: ignore[import]
+
+        torch_gpu = _detect_torch_cuda_inventory(torch)
         if _torch_cuda_is_executable(torch):
-            idx = torch.cuda.current_device()
-            name = torch.cuda.get_device_name(idx)
-            props = torch.cuda.get_device_properties(idx)
-            vram_mb = props.total_memory // (1024 * 1024)
+            if torch_gpu is None:
+                torch_gpu = _detect_torch_cuda_inventory(torch)
+            name = torch_gpu["gpu_name"] if torch_gpu is not None else "CUDA GPU"
+            vram_mb = torch_gpu["vram_mb"] if torch_gpu is not None else 0
             shared_vram_mb = max(0, ram_mb // 2)
-            return {
+            profile = {
                 "device": "cuda",
                 "gpu_name": name,
                 "vram_mb": vram_mb,
@@ -170,9 +206,16 @@ def detect_hardware() -> dict:
                 "ram_mb": ram_mb,
                 "cuda_available": True,
             }
+            if torch_gpu is not None and torch_gpu.get("gcn_arch"):
+                profile["gcn_arch"] = torch_gpu["gcn_arch"]
+            return profile
     except ImportError:
         pass
 
+    torch_inventory = _gpu_inventory_profile(torch_gpu, ram_mb)
+    if torch_inventory is not None:
+        return torch_inventory
+
     nvidia_gpu = _gpu_inventory_profile(_detect_nvidia_smi_gpu_memory(), ram_mb)
     if nvidia_gpu is not None:
         return nvidia_gpu

packages/node/meshnet_node/model_backend.py

@@ -3,8 +3,12 @@
 from __future__ import annotations
 
 import base64
+from collections import OrderedDict
 from dataclasses import dataclass
 import json
+import os
+import threading
+import time
 from pathlib import Path
 from typing import Any, Literal
 
@@ -27,12 +31,148 @@ class PartialModelLoadUnsupported(ModelBackendError):
     """Raised when a shard cannot be materialized from a local snapshot subset."""
 
 
+class KVCacheMiss(ModelBackendError):
+    """Raised when a decode step references session state this node no longer holds.
+
+    The head recovers by re-prefilling the full sequence (the stateless path),
+    so eviction or a node restart degrades throughput instead of corrupting output.
+    """
+
+
+def _torch_cuda_is_executable(torch_module: Any) -> bool:
+    """Return True only when this process can actually execute a CUDA/HIP op.
+
+    On ROCm, ``torch.cuda.is_available()`` can be true for an AMD GPU even when
+    the installed PyTorch wheel has no runnable kernels for that GPU target.
+    Loading weights onto such a device can segfault in native code, so the model
+    backend must use the same executable-device check as startup hardware
+    detection rather than trusting inventory alone.
+    """
+    try:
+        if not torch_module.cuda.is_available():
+            return False
+        probe = torch_module.empty((1,), device="cuda")
+        probe += 1
+        torch_module.cuda.synchronize()
+        return True
+    except Exception:
+        return False
+
+
 @dataclass(frozen=True)
 class TensorPayload:
     body: bytes
     shape: list[int]
     attention_mask_header: str | None
     position_ids_header: str | None
+    # Number of tokens already cached before this payload's tokens (decode steps).
+    past_len: int | None = None
+
+
+@dataclass(frozen=True)
+class TailTokenResult:
+    """Tail-shard decode result: decoded text plus the raw token id.
+
+    The token id lets the head feed the next decode step (and detect EOS)
+    without re-tokenizing text, which is not guaranteed to round-trip.
+    """
+
+    text: str
+    token_id: int
+
+
+@dataclass
+class SessionCacheEntry:
+    """Per-session cached state for one shard's layer range.
+
+    `cache` is whatever `use_cache=True` produces for these layers — a
+    transformers Cache holding K/V tensors for standard attention, or
+    recurrent conv/delta state for hybrid linear-attention layers. The store
+    treats it as opaque.
+    """
+
+    cache: Any
+    seq_len: int
+    effective_start: int
+    last_used: float
+
+
+class SessionCacheStore:
+    """TTL + LRU bounded map of session_id → SessionCacheEntry.
+
+    Each node caches state only for its own layer range; no node ever holds
+    another node's cache. Stale or mismatched entries raise KVCacheMiss so the
+    head falls back to a full re-prefill instead of producing corrupt output.
+    """
+
+    def __init__(
+        self,
+        max_sessions: int = 8,
+        ttl_seconds: float = 600.0,
+        clock: Any = None,
+    ) -> None:
+        self.max_sessions = max(1, int(max_sessions))
+        self.ttl_seconds = float(ttl_seconds)
+        self._clock = clock or time.monotonic
+        self._entries: OrderedDict[str, SessionCacheEntry] = OrderedDict()
+        self._lock = threading.Lock()
+
+    def __len__(self) -> int:
+        with self._lock:
+            return len(self._entries)
+
+    def store(self, session_id: str, cache: Any, seq_len: int, effective_start: int) -> SessionCacheEntry:
+        now = self._clock()
+        with self._lock:
+            self._entries.pop(session_id, None)
+            entry = SessionCacheEntry(cache, seq_len, effective_start, now)
+            self._entries[session_id] = entry
+            self._evict_locked(now)
+            return entry
+
+    def lookup(
+        self,
+        session_id: str,
+        *,
+        expected_seq_len: int | None = None,
+        effective_start: int | None = None,
+    ) -> SessionCacheEntry:
+        now = self._clock()
+        with self._lock:
+            self._evict_locked(now)
+            entry = self._entries.get(session_id)
+            if entry is None:
+                raise KVCacheMiss(f"no cached state for session {session_id[:8]}")
+            if expected_seq_len is not None and entry.seq_len != expected_seq_len:
+                del self._entries[session_id]
+                raise KVCacheMiss(
+                    f"session {session_id[:8]} cache holds {entry.seq_len} tokens, "
+                    f"expected {expected_seq_len}"
+                )
+            if effective_start is not None and entry.effective_start != effective_start:
+                del self._entries[session_id]
+                raise KVCacheMiss(
+                    f"session {session_id[:8]} cached with start_layer "
+                    f"{entry.effective_start}, requested {effective_start}"
+                )
+            entry.last_used = now
+            self._entries.move_to_end(session_id)
+            return entry
+
+    def drop(self, session_id: str) -> None:
+        with self._lock:
+            self._entries.pop(session_id, None)
+
+    def _evict_locked(self, now: float) -> None:
+        if self.ttl_seconds > 0:
+            expired = [
+                sid for sid, entry in self._entries.items()
+                if now - entry.last_used > self.ttl_seconds
+            ]
+            for sid in expired:
+                del self._entries[sid]
+        while len(self._entries) > self.max_sessions:
+            self._entries.popitem(last=False)
 
 
 def validate_quantization(value: str) -> Quantization:
@@ -72,6 +212,7 @@ class TorchModelShard:
         shard_end: int,
         quantization: Quantization = "auto",
         cache_dir: Path | None = None,
+        force_cpu: bool = False,
     ) -> None:
         if shard_start < 0 or shard_end < 0 or shard_start > shard_end:
             raise ValueError("shard_start must be <= shard_end and non-negative")
@@ -89,7 +230,10 @@ class TorchModelShard:
             ) from exc
 
         self.torch = torch
-        self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+        if force_cpu:
+            self.device = torch.device("cpu")
+        else:
+            self.device = torch.device("cuda" if _torch_cuda_is_executable(torch) else "cpu")
         load_source = str(cache_dir) if cache_dir is not None and (cache_dir / "config.json").exists() else model_id
         quant_config, dtype, uses_quantized_weights = _model_load_plan(
             AutoConfig,
@@ -134,8 +278,9 @@ class TorchModelShard:
                     self.model.to(self.device)
         except Exception as exc:
             if _looks_like_oom(exc):
+                memory_kind = "VRAM" if self.device.type == "cuda" else "RAM"
                 raise InsufficientVRAMError(
-                    f"insufficient VRAM to load {model_id} layers {shard_start}:{shard_end} "
+                    f"insufficient {memory_kind} to load {model_id} layers {shard_start}:{shard_end} "
                     f"with {quantization} quantization; choose a smaller shard or lower quantization"
                 ) from exc
             raise
@@ -162,8 +307,18 @@ class TorchModelShard:
         self._position_embeddings = _position_embeddings(self.model)
         self._norm = _final_norm(self.model) if self.is_tail else None
         self._lm_head = getattr(self.model, "lm_head", None) if self.is_tail else None
+        # Per-session KV/recurrent-state cache for this shard's layer range.
+        # Hybrid/linear-attention models such as Qwen3.6 can dispatch Triton
+        # recurrent-cache kernels when use_cache=True. Those kernels cannot
+        # consume CPU tensors ("Pointer argument cannot be accessed from Triton"),
+        # so CPU shards intentionally stay on the stateless prefill path.
+        self.supports_kv_cache = self.device.type != "cpu"
+        self.kv_sessions = SessionCacheStore(
+            max_sessions=int(os.environ.get("MESHNET_KV_MAX_SESSIONS", "8")),
+            ttl_seconds=float(os.environ.get("MESHNET_KV_TTL_SECONDS", "600")),
+        )
 
-    def encode_prompt(self, prompt: str) -> TensorPayload:
+    def encode_prompt(self, prompt: str, session_id: str | None = None) -> TensorPayload:
         if not self.is_head or self._embed_tokens is None:
             raise ModelBackendError("text prompts can only be accepted by the head shard")
         encoded = self.tokenizer(prompt, return_tensors="pt")
@@ -176,9 +331,44 @@ class TorchModelShard:
         hidden_states = self._embed_tokens(input_ids)
         if self._position_embeddings is not None:
             hidden_states = hidden_states + self._position_embeddings(position_ids)
-        hidden_states = self._run_layers(hidden_states, attention_mask, position_ids)
+        hidden_states = self._run_layers_session(
+            hidden_states, attention_mask, position_ids,
+            session_id=session_id, cache_mode="prefill" if session_id else None,
+        )
         return self._payload(hidden_states, attention_mask, position_ids)
 
+    def encode_next_token(self, token_id: int, session_id: str) -> TensorPayload:
+        """Decode step: embed one new token against this head's cached session.
+
+        Raises KVCacheMiss if the session was evicted — callers fall back to a
+        full re-prefill via encode_prompt.
+        """
+        if not self.is_head or self._embed_tokens is None:
+            raise ModelBackendError("decode steps can only start at the head shard")
+        if not self.supports_kv_cache:
+            raise KVCacheMiss("kv cache disabled on this backend")
+        entry = self.kv_sessions.lookup(
+            session_id, effective_start=self._effective_start(None)
+        )
+        past_len = entry.seq_len
+        input_ids = self.torch.tensor([[int(token_id)]], dtype=self.torch.long, device=self.device)
+        position_ids = self.torch.tensor([[past_len]], dtype=self.torch.long, device=self.device)
+        hidden_states = self._embed_tokens(input_ids)
+        if self._position_embeddings is not None:
+            hidden_states = hidden_states + self._position_embeddings(position_ids)
+        hidden_states = self._run_layers(
+            hidden_states, None, position_ids,
+            cache=entry.cache, past_len=past_len,
+        )
+        entry.seq_len = past_len + 1
+        return TensorPayload(
+            body=_tensor_to_bytes(hidden_states.to(self.torch.bfloat16).contiguous()),
+            shape=list(hidden_states.shape),
+            attention_mask_header=None,
+            position_ids_header=_int_tensor_header(position_ids),
+            past_len=past_len,
+        )
+
     def forward_bytes(
         self,
         body: bytes,
@@ -186,7 +376,10 @@ class TorchModelShard:
         attention_mask_header: str | None,
         position_ids_header: str | None,
         start_layer: int | None = None,
-    ) -> TensorPayload | str:
+        session_id: str | None = None,
+        cache_mode: str | None = None,
+        past_len: int | None = None,
+    ) -> TensorPayload | TailTokenResult | str:
         hidden_states = _tensor_from_bfloat16_bytes(body, shape, self.torch).to(
             self.device
         )
@@ -196,26 +389,51 @@ class TorchModelShard:
         position_ids = _tensor_from_int64_header(
             position_ids_header, self.torch, self.device
         )
-        hidden_states = self._run_layers(
-            hidden_states, attention_mask, position_ids, start_layer=start_layer
+        hidden_states = self._run_layers_session(
+            hidden_states, attention_mask, position_ids, start_layer=start_layer,
+            session_id=session_id, cache_mode=cache_mode, past_len=past_len,
         )
         if self.is_tail:
-            return self.decode_tail(hidden_states)
+            return self.decode_tail_token(hidden_states)
         return self._payload(hidden_states, attention_mask, position_ids)
 
     def decode_tail(self, hidden_states: Any) -> str:
+        return self.decode_tail_token(hidden_states).text
+
+    def decode_tail_token(self, hidden_states: Any) -> TailTokenResult:
         if self._norm is not None:
             hidden_states = self._norm(hidden_states)
         if self._lm_head is None:
             raise ModelBackendError("tail shard has no lm_head")
         logits = self._lm_head(hidden_states)
         token_id = int(self.torch.argmax(logits[:, -1, :], dim=-1)[0].item())
-        return self.tokenizer.decode([token_id], skip_special_tokens=True)
+        return TailTokenResult(
+            text=self.tokenizer.decode([token_id], skip_special_tokens=True),
+            token_id=token_id,
+        )
+
+    def eos_token_ids(self) -> list[int]:
+        """All token ids that should terminate generation (tokenizer + generation config)."""
+        ids: set[int] = set()
+        tok_eos = getattr(self.tokenizer, "eos_token_id", None)
+        gen_config = getattr(self.model, "generation_config", None)
+        gen_eos = getattr(gen_config, "eos_token_id", None) if gen_config is not None else None
+        for value in (tok_eos, gen_eos):
+            if value is None:
+                continue
+            if isinstance(value, (list, tuple)):
+                ids.update(int(v) for v in value)
+            else:
+                ids.add(int(value))
+        return sorted(ids)
+
+    def release_session(self, session_id: str) -> None:
+        self.kv_sessions.drop(session_id)
 
     def generate_text(
         self,
         messages: list[dict],
-        max_new_tokens: int = 256,
+        max_new_tokens: int = 5120,
         temperature: float = 1.0,
         top_p: float = 1.0,
     ) -> str:
@@ -245,7 +463,7 @@ class TorchModelShard:
     def generate_text_streaming(
         self,
         messages: list[dict],
-        max_new_tokens: int = 256,
+        max_new_tokens: int = 5000,
         temperature: float = 1.0,
         top_p: float = 1.0,
     ):
@@ -321,21 +539,112 @@ class TorchModelShard:
         )
         return dict(self.tokenizer(prompt, return_tensors="pt"))
 
+    def _effective_start(self, start_layer: int | None) -> int:
+        # start_layer overrides shard_start for overlapping-shard routing
+        # (X-Meshnet-Start-Layer header). Clamped to shard_start to prevent
+        # indexing outside the loaded weights.
+        return (
+            max(self.shard_start, start_layer)
+            if start_layer is not None
+            else self.shard_start
+        )
+
+    def _new_session_cache(self) -> Any | None:
+        """Build the model-appropriate cache object for one session.
+
+        DynamicCache(config=...) lets transformers pick the right per-layer
+        state (K/V for standard attention, conv/recurrent state for hybrid
+        linear-attention layers) — the same construction the model's own
+        forward() uses when use_cache=True.
+        """
+        try:
+            from transformers import DynamicCache
+        except ImportError:
+            return None
+        try:
+            return DynamicCache(config=self.model.config)
+        except TypeError:
+            return DynamicCache()
+
+    def _run_layers_session(
+        self,
+        hidden_states: Any,
+        attention_mask: Any,
+        position_ids: Any,
+        start_layer: int | None = None,
+        session_id: str | None = None,
+        cache_mode: str | None = None,
+        past_len: int | None = None,
+    ) -> Any:
+        """Run this shard's layers, keying cached state by session when requested.
+
+        cache_mode "prefill" creates fresh session state; "decode" requires an
+        existing entry (KVCacheMiss otherwise). None runs fully stateless —
+        today's behavior, kept as the recovery path.
+        """
+        effective_start = self._effective_start(start_layer)
+        if not (session_id and cache_mode and self.supports_kv_cache):
+            if cache_mode == "decode":
+                # A decode payload is one token — running it stateless would
+                # silently produce garbage. Force the head to re-prefill.
+                raise KVCacheMiss("kv cache disabled on this backend")
+            return self._run_layers(
+                hidden_states, attention_mask, position_ids, start_layer=start_layer
+            )
+        if cache_mode == "decode":
+            entry = self.kv_sessions.lookup(
+                session_id,
+                expected_seq_len=past_len,
+                effective_start=effective_start,
+            )
+            seq_len = int(hidden_states.shape[1])
+            # Decode attends over cache + new token; no padding, so no mask needed.
+            hidden_states = self._run_layers(
+                hidden_states, None, position_ids,
+                start_layer=start_layer, cache=entry.cache, past_len=entry.seq_len,
+            )
+            entry.seq_len += seq_len
+            return hidden_states
+        # Prefill: fresh cache for this session (replaces any stale entry).
+        cache = self._new_session_cache()
+        if cache is None:
+            return self._run_layers(
+                hidden_states, attention_mask, position_ids, start_layer=start_layer
+            )
+        try:
+            result = self._run_layers(
+                hidden_states, attention_mask, position_ids,
+                start_layer=start_layer, cache=cache, past_len=0,
+            )
+        except Exception as exc:
+            if not _cache_unsupported_for_shard(exc):
+                raise
+            # Layers reject cache kwargs (exotic architecture) — disable caching
+            # for this backend and stay on the stateless path. Some hybrid
+            # CPU paths also accept cache kwargs but fail at runtime inside
+            # Triton-only kernels; treat those as cache-unsupported too.
+            self.supports_kv_cache = False
+            print(f"  [node] kv cache unsupported by {self.model_id}: {exc}", flush=True)
+            return self._run_layers(
+                hidden_states, attention_mask, position_ids, start_layer=start_layer
+            )
+        self.kv_sessions.store(
+            session_id, cache,
+            seq_len=int(hidden_states.shape[1]),
+            effective_start=effective_start,
+        )
+        return result
+
     def _run_layers(
         self,
         hidden_states: Any,
         attention_mask: Any,
         position_ids: Any,
         start_layer: int | None = None,
+        cache: Any = None,
+        past_len: int = 0,
     ) -> Any:
-        # start_layer overrides shard_start for overlapping-shard routing
-        # (X-Meshnet-Start-Layer header). Clamped to shard_start to prevent
-        # indexing outside the loaded weights.
-        effective_start = (
-            max(self.shard_start, start_layer)
-            if start_layer is not None
-            else self.shard_start
-        )
+        effective_start = self._effective_start(start_layer)
         position_embeddings = _rotary_position_embeddings(
             self.model,
             hidden_states,
@@ -346,6 +655,12 @@ class TorchModelShard:
             hidden_states,
             self.torch,
         )
+        cache_position = None
+        if cache is not None:
+            seq_len = int(hidden_states.shape[1])
+            cache_position = self.torch.arange(
+                past_len, past_len + seq_len, device=hidden_states.device
+            )
         with self.torch.inference_mode():
             for layer in self.layers[effective_start:self.shard_end + 1]:
                 hidden_states = _call_layer(
@@ -354,6 +669,8 @@ class TorchModelShard:
                     layer_attention_mask,
                     position_ids,
                     position_embeddings,
+                    cache=cache,
+                    cache_position=cache_position,
                 )
             return hidden_states.to(self.torch.bfloat16)
 
@@ -377,8 +694,11 @@ def load_torch_shard(
     shard_end: int,
     quantization: Quantization = "auto",
     cache_dir: Path | None = None,
+    force_cpu: bool = False,
 ) -> TorchModelShard:
-    return TorchModelShard(model_id, shard_start, shard_end, quantization, cache_dir)
+    return TorchModelShard(
+        model_id, shard_start, shard_end, quantization, cache_dir, force_cpu=force_cpu
+    )
 
 
 def _total_layers_for_local_snapshot(auto_config: Any, load_source: str) -> int | None:
@@ -411,7 +731,7 @@ def _should_partial_materialize_shard(
         return False
     if total_layers_hint is None:
         return False
-    return not (shard_start == 0 and shard_end >= total_layers_hint - 1)
+    return True
 
 
 def _load_partial_model_from_snapshot(
@@ -476,17 +796,41 @@ def _load_partial_model_from_snapshot(
         )
 
     with init_empty_weights_fn():
-        model = auto_model_for_causal_lm.from_config(cfg, torch_dtype=dtype)
+        model = auto_model_for_causal_lm.from_config(_causal_lm_config(cfg), torch_dtype=dtype)
     tie_weights = getattr(model, "tie_weights", None)
     if callable(tie_weights):
         tie_weights()
 
+    # Multimodal/MTP checkpoints (e.g. Qwen3.5/3.6-MoE) carry vision and
+    # multi-token-prediction tensors the text-only CausalLM never builds;
+    # transformers' from_pretrained drops them via _keys_to_ignore_on_load_unexpected,
+    # so the manual loader must skip them too.
+    expected_keys = _model_state_dict_keys(model)
     tensors_by_file: dict[str, list[str]] = {}
+    skipped: list[str] = []
     for tensor_name in sorted(tensor_names):
         rel_file = weight_map.get(tensor_name)
         if not isinstance(rel_file, str):
             continue
+        if (
+            expected_keys is not None
+            and _checkpoint_tensor_name_for_model(model, tensor_name) not in expected_keys
+        ):
+            skipped.append(tensor_name)
+            continue
         tensors_by_file.setdefault(rel_file, []).append(tensor_name)
+    if skipped:
+        preview = ", ".join(skipped[:3])
+        print(
+            f"  Skipping {len(skipped)} checkpoint tensors absent from the causal LM "
+            f"(e.g. {preview})",
+            flush=True,
+        )
+    if not tensors_by_file:
+        raise PartialModelLoadUnsupported(
+            f"no checkpoint tensors for layers {shard_start}-{shard_end} match the "
+            f"causal LM built from {snapshot_dir}"
+        )
 
     for rel_file, names in tensors_by_file.items():
         checkpoint_file = snapshot_dir / rel_file
@@ -498,7 +842,7 @@ def _load_partial_model_from_snapshot(
             for tensor_name in names:
                 set_tensor_fn(
                     model,
-                    tensor_name,
+                    _checkpoint_tensor_name_for_model(model, tensor_name),
                     device,
                     value=handle.get_tensor(tensor_name),
                     dtype=dtype,
@@ -569,38 +913,85 @@ def _native_torch_dtype(cfg: Any, torch: Any) -> Any:
     return torch.bfloat16
 
 
+def _causal_lm_config(cfg: Any) -> Any:
+    """Use the text decoder config for composite VLM/MoE presets."""
+    get_text_config = getattr(cfg, "get_text_config", None)
+    if callable(get_text_config):
+        try:
+            return get_text_config()
+        except Exception:
+            pass
+    text_config = getattr(cfg, "text_config", None)
+    if text_config is not None:
+        return text_config
+    return cfg
+
+
+def _model_state_dict_keys(model: Any) -> set[str] | None:
+    """Expected parameter/buffer names, or None when the model can't report them."""
+    state_dict = getattr(model, "state_dict", None)
+    if not callable(state_dict):
+        return None
+    try:
+        return set(state_dict().keys())
+    except Exception:
+        return None
+
+
+def _checkpoint_tensor_name_for_model(model: Any, tensor_name: str) -> str:
+    """Map multimodal checkpoint keys onto text-only CausalLM modules when needed."""
+    inner = getattr(model, "model", None)
+    if inner is not None and hasattr(inner, "language_model"):
+        return tensor_name
+    if ".language_model." in tensor_name:
+        return tensor_name.replace(".language_model.", ".")
+    return tensor_name
+
+
+def _transformer_backbone(model: Any) -> Any:
+    if hasattr(model, "model"):
+        inner = model.model
+        language_model = getattr(inner, "language_model", None)
+        if language_model is not None:
+            return language_model
+        return inner
+    if hasattr(model, "transformer"):
+        return model.transformer
+    raise ModelBackendError(
+        "unsupported HuggingFace model architecture: no transformer backbone found"
+    )
+
+
 def _model_layers(model: Any) -> Any:
-    if hasattr(model, "model") and hasattr(model.model, "layers"):
-        return model.model.layers
-    if hasattr(model, "transformer") and hasattr(model.transformer, "h"):
-        return model.transformer.h
+    backbone = _transformer_backbone(model)
+    for attr in ("layers", "h", "blocks"):
+        layers = getattr(backbone, attr, None)
+        if layers is not None:
+            return layers
     raise ModelBackendError(
         "unsupported HuggingFace model architecture: no transformer layers found"
     )
 
 
 def _embed_tokens(model: Any) -> Any:
-    if hasattr(model, "model") and hasattr(model.model, "embed_tokens"):
-        return model.model.embed_tokens
-    if hasattr(model, "transformer") and hasattr(model.transformer, "wte"):
-        return model.transformer.wte
+    backbone = _transformer_backbone(model)
+    for attr in ("embed_tokens", "wte"):
+        embed = getattr(backbone, attr, None)
+        if embed is not None:
+            return embed
     raise ModelBackendError(
         "unsupported HuggingFace model architecture: no token embeddings found"
     )
 
 
 def _position_embeddings(model: Any) -> Any | None:
-    if hasattr(model, "transformer") and hasattr(model.transformer, "wpe"):
-        return model.transformer.wpe
-    return None
+    backbone = _transformer_backbone(model)
+    return getattr(backbone, "wpe", None)
 
 
 def _rotary_embedding_module(model: Any) -> Any | None:
-    if hasattr(model, "model") and hasattr(model.model, "rotary_emb"):
-        return model.model.rotary_emb
-    if hasattr(model, "transformer") and hasattr(model.transformer, "rotary_emb"):
-        return model.transformer.rotary_emb
-    return None
+    backbone = _transformer_backbone(model)
+    return getattr(backbone, "rotary_emb", None)
 
 
 def _active_modules_for_shard(model: Any, shard_start: int, shard_end: int) -> list[Any]:
@@ -627,10 +1018,11 @@ def _active_modules_for_shard(model: Any, shard_start: int, shard_end: int) -> l
 
 
 def _final_norm(model: Any) -> Any | None:
-    if hasattr(model, "model") and hasattr(model.model, "norm"):
-        return model.model.norm
-    if hasattr(model, "transformer") and hasattr(model.transformer, "ln_f"):
-        return model.transformer.ln_f
+    backbone = _transformer_backbone(model)
+    for attr in ("norm", "ln_f", "final_layer_norm"):
+        norm = getattr(backbone, attr, None)
+        if norm is not None:
+            return norm
     return None
 
 
@@ -681,6 +1073,8 @@ def _call_layer(
     attention_mask: Any,
     position_ids: Any,
     position_embeddings: Any | None = None,
+    cache: Any = None,
+    cache_position: Any = None,
 ) -> Any:
     attempts = (
         {
@@ -701,6 +1095,14 @@ def _call_layer(
     last_exc: Exception | None = None
     for kwargs in attempts:
         filtered = {key: value for key, value in kwargs.items() if value is not None}
+        if cache is not None:
+            # transformers 5.x layers take a Cache via past_key_values and
+            # mutate it in place; cache_position is required by sliding-window
+            # and hybrid recurrent layers.
+            filtered["past_key_values"] = cache
+            filtered["use_cache"] = True
+            if cache_position is not None:
+                filtered["cache_position"] = cache_position
         try:
             output = layer(hidden_states, **filtered)
             return output[0] if isinstance(output, tuple) else output
@@ -743,7 +1145,22 @@ def _looks_like_oom(exc: BaseException) -> bool:
     current: BaseException | None = exc
     while current is not None:
         text = str(current).lower()
-        if "out of memory" in text or "cuda error: out of memory" in text:
+        if (
+            "out of memory" in text
+            or "cuda error: out of memory" in text
+            or "paging file is too small" in text
+            or "os error 1455" in text
+        ):
             return True
         current = current.__cause__ or current.__context__
     return False
+
+
+def _cache_unsupported_for_shard(exc: BaseException) -> bool:
+    """True when a layer failure means session cache is unsupported, not fatal."""
+    text = str(exc).lower()
+    return (
+        isinstance(exc, TypeError)
+        or "pointer argument cannot be accessed from triton" in text
+        or ("triton" in text and "cpu tensor" in text)
+    )

packages/node/meshnet_node/relay_bridge.py

@@ -6,8 +6,10 @@ import base64
 import json
 import logging
 import os
+import re
 import threading
 import time
+import urllib.parse
 import urllib.error
 import urllib.request
 from concurrent.futures import ThreadPoolExecutor
@@ -17,6 +19,42 @@ log = logging.getLogger(__name__)
 
 DEFAULT_MAX_CONCURRENCY = 8
 
+# Activation tensors ride the relay as one WebSocket frame per hop, so the
+# websockets default of 1 MiB rejects any real prefill (close code 1009).
+DEFAULT_WS_MAX_BYTES = 256 * 1024 * 1024
+
+
+def ws_max_size() -> int | None:
+    """Max inbound WebSocket frame size; MESHNET_WS_MAX_BYTES<=0 means unlimited."""
+    raw = os.environ.get("MESHNET_WS_MAX_BYTES", "").strip()
+    if not raw:
+        return DEFAULT_WS_MAX_BYTES
+    try:
+        value = int(raw)
+    except ValueError:
+        return DEFAULT_WS_MAX_BYTES
+    return None if value <= 0 else value
+
+
+# Binary relay frame: JSON header + raw body in one WebSocket binary message,
+# so activation bodies travel as bytes instead of base64 inside JSON. Same wire
+# format as meshnet_relay.server — duplicated because node and relay ship as
+# independent distributions.
+BINARY_FRAME_MAGIC = b"MRF1"
+
+
+def encode_binary_frame(header: dict, body: bytes) -> bytes:
+    header_bytes = json.dumps(header, separators=(",", ":")).encode()
+    return BINARY_FRAME_MAGIC + len(header_bytes).to_bytes(4, "big") + header_bytes + body
+
+
+def decode_binary_frame(frame: bytes) -> tuple[dict, bytes]:
+    if len(frame) < 8 or frame[:4] != BINARY_FRAME_MAGIC:
+        raise ValueError("not a meshnet binary relay frame")
+    header_len = int.from_bytes(frame[4:8], "big")
+    header = json.loads(frame[8:8 + header_len].decode())
+    return header, bytes(frame[8 + header_len:])
+
 
 @dataclass(frozen=True)
 class RelayBridgeInfo:
@@ -107,7 +145,9 @@ class RelayHttpBridge:
 
         while self._running:
             try:
-                with wsc.connect(self.relay_url, open_timeout=5) as ws:
+                with wsc.connect(
+                    self.relay_url, open_timeout=5, max_size=ws_max_size(), compression=None,
+                ) as ws:
                     self._ws = ws
                     self._connected.set()
                     ws.send(json.dumps(_make_envelope(
@@ -120,6 +160,17 @@ class RelayHttpBridge:
                             raw = ws.recv(timeout=1)
                         except TimeoutError:
                             continue
+                        if isinstance(raw, (bytes, bytearray)):
+                            try:
+                                payload, body = decode_binary_frame(bytes(raw))
+                            except (ValueError, json.JSONDecodeError):
+                                continue
+                            if payload.get("target_peer") not in {None, self.peer_id}:
+                                continue
+                            if self._executor is None:
+                                break
+                            self._executor.submit(self._process_request, payload, body)
+                            continue
                         try:
                             envelope = json.loads(raw)
                         except (TypeError, json.JSONDecodeError):
@@ -158,20 +209,41 @@ class RelayHttpBridge:
             log.debug("relay bridge send failed (request orphaned): %s", exc)
             return False
 
-    def _process_request(self, payload: dict) -> None:
+    def _send_binary_response_frame(self, header: dict, body: bytes) -> bool:
+        """Send one binary response frame; False if the socket is gone."""
+        ws = self._ws
+        if ws is None:
+            return False
+        frame = encode_binary_frame(header, body)
+        try:
+            with self._send_lock:
+                ws.send(frame)
+            return True
+        except Exception as exc:
+            log.debug("relay bridge binary send failed (request orphaned): %s", exc)
+            return False
+
+    def _process_request(self, payload: dict, binary_body: bytes | None = None) -> None:
         request_id = str(payload.get("request_id") or "")
         method = str(payload.get("method") or "POST").upper()
         path = str(payload.get("path") or "/")
         headers = payload.get("headers") if isinstance(payload.get("headers"), dict) else {}
+        binary_mode = binary_body is not None
 
-        # body_base64 carries binary data (e.g. bfloat16 activation tensors) safely.
-        # Fallback to text "body" for backward-compat with non-binary requests.
-        body_b64 = payload.get("body_base64")
-        if body_b64:
-            data = base64.b64decode(body_b64)
+        req_suffix = f" request_id={request_id}" if request_id else ""
+        print(f"  [node] relay {method} {path}{req_suffix}", flush=True)
+
+        if binary_mode:
+            data = binary_body
         else:
-            body_text = payload.get("body") or ""
-            data = body_text.encode() if isinstance(body_text, str) else bytes(body_text)
+            # Legacy JSON request: body_base64 carries binary data, text "body"
+            # covers non-binary requests.
+            body_b64 = payload.get("body_base64")
+            if body_b64:
+                data = base64.b64decode(body_b64)
+            else:
+                body_text = payload.get("body") or ""
+                data = body_text.encode() if isinstance(body_text, str) else bytes(body_text)
 
         url = f"{self.local_base_url}{path}"
         req = urllib.request.Request(url, data=data, headers=headers, method=method)
@@ -184,6 +256,13 @@ class RelayHttpBridge:
                     return
                 resp_bytes = resp.read()
                 # Forward all X-Meshnet-* headers so the caller can reconstruct the activation.
+                if binary_mode:
+                    self._send_binary_response_frame({
+                        "request_id": request_id,
+                        "status": resp.status,
+                        "headers": resp_headers,
+                    }, resp_bytes)
+                    return
                 is_binary = "octet-stream" in content_type
                 result: dict = {
                     "request_id": request_id,
@@ -256,5 +335,30 @@ class RelayHttpBridge:
         })
 
 
-def peer_id_from_wallet(wallet_address: str) -> str:
-    return wallet_address[:16] if len(wallet_address) >= 16 else wallet_address
+def _peer_id_suffix(value: str) -> str:
+    """Return a relay-safe suffix for a human node name or numeric instance id."""
+    suffix = re.sub(r"[^A-Za-z0-9_.-]+", "-", value.strip()).strip("-._")
+    return suffix[:32]
+
+
+def peer_id_from_wallet(
+    wallet_address: str,
+    *,
+    node_name: str | None = None,
+    advertised_addr: str | None = None,
+) -> str:
+    """Build a per-node relay peer id from the wallet plus node identity.
+
+    Multiple nodes can legitimately share one wallet for payouts, but the relay
+    registry is keyed by peer_id.  Using only the wallet prefix makes those
+    nodes overwrite each other at the relay.  Prefer the operator-provided node
+    name; if absent, use the advertised endpoint port as the stable integer
+    instance suffix (7001, 7002, ... for local multi-node runs).
+    """
+    wallet_prefix = wallet_address[:16] if len(wallet_address) >= 16 else wallet_address
+    suffix = _peer_id_suffix(node_name or "") if node_name else ""
+    if not suffix and advertised_addr:
+        parsed = urllib.parse.urlparse(advertised_addr)
+        if parsed.port is not None:
+            suffix = str(parsed.port)
+    return f"{wallet_prefix}-{suffix}" if suffix else wallet_prefix

packages/node/meshnet_node/startup.py

@@ -15,7 +15,7 @@ from pathlib import Path
 from typing import Any
 
 from .downloader import compute_shard_checksum, download_shard
-from .hardware import detect_hardware, benchmark_throughput_checked
+from .hardware import detect_hardware, benchmark_throughput_checked, with_forced_cpu
 from .model_catalog import model_metadata_for
 from .relay_bridge import RelayHttpBridge, peer_id_from_wallet
 from .server import StubNodeServer
@@ -140,6 +140,13 @@ def _hardware_label(device: str, gpu_name: str | None = None) -> str:
     return "CPU"
 
 
+def _relay_ready_line(relay_fields: dict) -> str:
+    relay_addr = relay_fields.get("relay_addr")
+    if not relay_addr:
+        return ""
+    return f"  Relay:        {relay_addr}\n"
+
+
 def _positive_int(value: int | str | None, name: str) -> int | None:
     if value is None or value == "":
         return None
@@ -197,12 +204,58 @@ def _max_assignable_layers(
     memory_mb: int,
     total_layers: int | None,
     bytes_per_layer: int | None = None,
+    *,
+    safety_fraction: float = 0.8,
 ) -> int:
     if total_layers is None or total_layers <= 0 or memory_mb <= 0:
         return 0
     budget_bytes = memory_mb * 1024 * 1024
     layer_bytes = bytes_per_layer or _DEFAULT_BYTES_PER_LAYER
-    return min(total_layers, int((budget_bytes * 0.8) // layer_bytes))
+    return min(total_layers, int((budget_bytes * safety_fraction) // layer_bytes))
+
+
+def _runtime_shard_safety_fraction(device: str) -> float:
+    """CPU partial loads need room for model skeletons, tokenizer, and allocator peaks."""
+    return 0.55 if device != "cuda" else 0.8
+
+
+def _cap_auto_assigned_shard(
+    shard_start: int,
+    shard_end: int,
+    total_layers: int | None,
+    memory_mb: int,
+    bytes_per_layer: int | None,
+    device: str,
+) -> tuple[int, bool, int]:
+    if bytes_per_layer is None or total_layers is None:
+        return shard_end, False, 0
+    max_layers = _max_assignable_layers(
+        memory_mb,
+        total_layers,
+        bytes_per_layer=bytes_per_layer,
+        safety_fraction=_runtime_shard_safety_fraction(device),
+    )
+    if max_layers <= 0:
+        return shard_end, False, max_layers
+    assigned_layers = shard_end - shard_start + 1
+    if assigned_layers <= max_layers:
+        return shard_end, False, max_layers
+    return min(total_layers - 1, shard_start + max_layers - 1), True, max_layers
+
+
+def _format_shard_label(
+    shard_start: int,
+    shard_end: int,
+    total_layers: int | None = None,
+    *,
+    model_name: str | None = None,
+) -> str:
+    layer_count = shard_end - shard_start + 1
+    if isinstance(total_layers, int) and total_layers > 0:
+        return f"layers {shard_start}–{shard_end} ({layer_count} of {total_layers})"
+    if model_name:
+        return f"layers {shard_start}–{shard_end} ({model_name})"
+    return f"layers {shard_start}–{shard_end}"
 
 
 def _shard_budget_line(
@@ -211,13 +264,19 @@ def _shard_budget_line(
     total_layers: int | None,
     quantization: str,
     bytes_per_layer: int | None = None,
+    safety_fraction: float = 0.8,
 ) -> str:
     memory_gb = memory_mb / 1024
     gb_str = f"{memory_gb:.1f} GB"
     budget_quantization = "bfloat16" if quantization == "auto" else quantization
     if total_layers is None or total_layers <= 0:
         return f"Memory budget: {gb_str} {memory_source}; shard budget: unknown model layer count"
-    max_layers = _max_assignable_layers(memory_mb, total_layers, bytes_per_layer=bytes_per_layer)
+    max_layers = _max_assignable_layers(
+        memory_mb,
+        total_layers,
+        bytes_per_layer=bytes_per_layer,
+        safety_fraction=safety_fraction,
+    )
     # Remaining capacity after one full model load (rough estimate)
     shard_bytes = max_layers * (bytes_per_layer or _DEFAULT_BYTES_PER_LAYER)
     remaining_gb = (memory_mb * 1024 * 1024 - shard_bytes) / (1024 ** 3)
@@ -294,11 +353,16 @@ def _start_relay_bridge_if_available(
     local_base_url: str,
     advertised_endpoint: str,
     relay_url: str | None = None,
+    node_name: str | None = None,
 ) -> tuple[RelayHttpBridge | None, dict]:
     relay_url = relay_url or _discover_relay_url(tracker_url)
     if not relay_url:
         return None, {}
-    peer_id = peer_id_from_wallet(wallet_address)
+    peer_id = peer_id_from_wallet(
+        wallet_address,
+        node_name=node_name,
+        advertised_addr=advertised_endpoint,
+    )
     bridge = RelayHttpBridge(
         relay_url=relay_url,
         peer_id=peer_id,
@@ -334,25 +398,38 @@ def _attach_relay_bridge(node: StubNodeServer | TorchNodeServer, bridge: RelayHt
 _PENDING_NODE_ID = "pending"
 
 
+_HEARTBEAT_INTERVAL_IDLE = 20.0
+_HEARTBEAT_INTERVAL_BUSY = 3.0
+
+
 def _start_heartbeat(
     tracker_url: str,
     node_id: str,
     register_payload: dict,
-    interval: float = 20.0,
+    interval: float = _HEARTBEAT_INTERVAL_IDLE,
     node_ref: Any | None = None,
     start_time: float | None = None,
 ) -> threading.Thread:
     """Daemon thread: sends heartbeats and re-registers automatically after tracker restarts.
 
     Heartbeat body carries cumulative stats (total_requests, failed_requests,
-    queue_depth, uptime_seconds, status). Stats are buffered locally during
-    outage and flushed on next successful heartbeat.
+    queue_depth, current_requests, uptime_seconds, status). Stats are buffered
+    locally during outage and flushed on next successful heartbeat.
 
     Heartbeat response may include new_assignment: {model, shard_start, shard_end}
     which is logged for now (hot-reload implemented in US-026).
     """
     _start_time = start_time or time.monotonic()
 
+    def _current_requests_snapshot() -> list[dict]:
+        if node_ref is None:
+            return []
+        getter = getattr(node_ref, "current_requests", None)
+        if getter is None:
+            return []
+        current = getter() if callable(getter) else getter
+        return list(current) if isinstance(current, list) else []
+
     def _get_stats() -> dict:
         uptime = time.monotonic() - _start_time
         stats: dict = {"uptime_seconds": round(uptime, 1), "status": "ready"}
@@ -364,8 +441,16 @@ def _start_heartbeat(
             )
             stats["failed_requests"] = getattr(node_ref, "failed_requests", 0)
             stats["queue_depth"] = getattr(node_ref, "queue_depth", 0)
+        current_requests = _current_requests_snapshot()
+        if current_requests:
+            stats["current_requests"] = current_requests
         return stats
 
+    def _sleep_interval() -> float:
+        if _current_requests_snapshot() or (node_ref is not None and getattr(node_ref, "queue_depth", 0) > 0):
+            return _HEARTBEAT_INTERVAL_BUSY
+        return interval
+
     def _reregister() -> bool:
         nonlocal node_id
         try:
@@ -427,7 +512,7 @@ def _start_heartbeat(
         outage_streak = 1 if node_id == _PENDING_NODE_ID else 0
 
         while True:
-            time.sleep(interval)
+            time.sleep(_sleep_interval())
 
             if outage_streak > 0:
                 # Tracker was down — attempt re-registration first (it may have restarted
@@ -537,6 +622,15 @@ def _warn_virtual_network_ip(ip: str | None) -> None:
         pass
 
 
+def _registration_display_fields(node_name: str | None) -> dict[str, str]:
+    if not node_name:
+        return {}
+    name = node_name.strip()
+    if not name:
+        return {}
+    return {"friendly_name": name}
+
+
 def run_startup(
     tracker_url: str,
     port: int = 0,
@@ -557,6 +651,8 @@ def run_startup(
     tracker_source_disabled: bool = False,
     torch_threads: int | None = None,
     torch_interop_threads: int | None = None,
+    node_name: str | None = None,
+    force_cpu: bool = False,
 ) -> StubNodeServer | TorchNodeServer:
     """Execute the full startup sequence and return a running node server.
 
@@ -573,6 +669,7 @@ def run_startup(
 
     tracker_url = tracker_url.rstrip("/")
     relay_url = _discover_relay_url(tracker_url)
+    display_fields = _registration_display_fields(node_name)
     if max_loaded_shards < 1:
         raise ValueError("--max-shards must be at least 1")
 
@@ -597,6 +694,8 @@ def run_startup(
 
     print("Detecting hardware...", flush=True)
     hw = detect_hardware()
+    if force_cpu:
+        hw = with_forced_cpu(hw)
     torch_thread_config = _configure_torch_threads(torch_threads, torch_interop_threads)
     if torch_thread_config:
         hw.update(torch_thread_config)
@@ -613,6 +712,16 @@ def run_startup(
         vram_mb = vram_mb_override
         shared_vram_mb = 0
         print(f"  Memory budget overridden to {vram_mb / 1024:.1f} GB via --memory", flush=True)
+    elif force_cpu:
+        if gpu_name and vram_mb > 0:
+            print(
+                f"  --cpu: ignoring {gpu_name} "
+                f"({vram_mb / 1024:.1f} GB VRAM); running in CPU mode "
+                f"({ram_mb / 1024:.1f} GB RAM)",
+                flush=True,
+            )
+        else:
+            print(f"  --cpu: running in CPU mode ({ram_mb / 1024:.1f} GB RAM)", flush=True)
     elif device == "cpu":
         gpu_suffix = ""
         if gpu_name and vram_mb > 0:
@@ -634,7 +743,7 @@ def run_startup(
     print(f"  Memory budget: {memory_budget_mb / 1024:.1f} GB {memory_budget_source}", flush=True)
 
     print("Benchmarking compute...", flush=True)
-    if device != "cuda" and gpu_name:
+    if device != "cuda" and gpu_name and not force_cpu:
         _cuda_score, cuda_ok, cuda_error = benchmark_throughput_checked("cuda")
         hw["cuda_benchmark_ok"] = cuda_ok
         if cuda_error:
@@ -693,6 +802,25 @@ def run_startup(
                     if net_asgn.get("hf_repo") == model_id and net_asgn.get("gap_found"):
                         shard_start = net_asgn["shard_start"]
                         shard_end = net_asgn["shard_end"]
+                        asgn_total_layers = int(net_asgn.get("num_layers") or detected)
+                        asgn_bytes_per_layer = _assignment_bytes_per_layer(net_asgn, quantization)
+                        capped_shard_end, was_capped, max_runtime_layers = _cap_auto_assigned_shard(
+                            shard_start,
+                            shard_end,
+                            asgn_total_layers,
+                            memory_budget_mb,
+                            asgn_bytes_per_layer,
+                            device,
+                        )
+                        if was_capped:
+                            original_end = shard_end
+                            shard_end = capped_shard_end
+                            print(
+                                f"  WARNING: tracker assigned layers {shard_start}-{original_end}, "
+                                f"but CPU-safe runtime budget fits {max_runtime_layers}/{asgn_total_layers} layers; "
+                                f"loading layers {shard_start}-{shard_end} instead.",
+                                flush=True,
+                            )
                         full_sources = (
                             [] if tracker_source_disabled
                             else _full_model_sources(net_asgn.get("model_sources", []))
@@ -708,7 +836,7 @@ def run_startup(
                             )
                         print(
                             f"  Tracker found uncovered shard: "
-                            f"layers {shard_start}–{shard_end} (of {detected})",
+                            f"layers {shard_start}-{shard_end} (of {detected})",
                             flush=True,
                         )
                 except Exception:
@@ -730,17 +858,16 @@ def run_startup(
             cache_dir=cache_dir,
             debug=debug,
             max_loaded_shards=max_loaded_shards,
+            force_cpu=force_cpu,
         )
         _node_start_time = time.monotonic()
         actual_port = node.start()
         total_layers = getattr(getattr(node, "backend", None), "total_layers", None)
-        if isinstance(total_layers, int) and total_layers > 0:
-            layer_count = shard_end - shard_start + 1
-            shard_label = f"layers {shard_start}–{shard_end}; {layer_count} of {total_layers}"
-        else:
-            shard_label = f"layers {shard_start}–{shard_end}"
+        shard_label = _format_shard_label(shard_start, shard_end, total_layers)
         public_host = advertise_host or (socket.getfqdn() if host == "0.0.0.0" else host)
         endpoint = f"http://{public_host}:{actual_port}"
+        if hasattr(node, "set_advertised_endpoint"):
+            node.set_advertised_endpoint(endpoint)
         local_base_url = f"http://127.0.0.1:{actual_port}"
         relay_bridge, relay_fields = _start_relay_bridge_if_available(
             tracker_url,
@@ -748,6 +875,7 @@ def run_startup(
             local_base_url,
             endpoint,
             relay_url=relay_url,
+            node_name=node_name,
         )
         _attach_relay_bridge(node, relay_bridge)
         # Register with tracker so other nodes can auto-join this model.
@@ -781,6 +909,7 @@ def run_startup(
             ),
             **registration_capabilities,
             **relay_fields,
+            **display_fields,
         }
         tracker_node_id = _register_with_tracker(
             tracker_url, reg_payload, node, _node_start_time,
@@ -795,6 +924,7 @@ def run_startup(
             f"  {_shard_budget_line(memory_budget_mb, memory_budget_source, total_layers, quantization)}\n"
             f"  Quantization: {quantization}\n"
             f"  Endpoint:     {endpoint}\n"
+            f"{_relay_ready_line(relay_fields)}"
             f"  Node ID:      {tracker_node_id or 'unregistered'}\n"
             f"  Hardware:     {_hardware_label(device, gpu_name)}\n"
             f"  Benchmark:    {bench_tps:,.0f} (throughput index)\n"
@@ -827,18 +957,36 @@ def run_startup(
         assigned_shard_start: int = net_assignment["shard_start"]
         assigned_shard_end: int = net_assignment["shard_end"]
         assigned_num_layers: int = net_assignment["num_layers"]
+        assigned_bytes_per_layer = _assignment_bytes_per_layer(net_assignment, quantization)
+        capped_shard_end, was_capped, max_runtime_layers = _cap_auto_assigned_shard(
+            assigned_shard_start,
+            assigned_shard_end,
+            assigned_num_layers,
+            memory_budget_mb,
+            assigned_bytes_per_layer,
+            device,
+        )
+        if was_capped:
+            original_end = assigned_shard_end
+            assigned_shard_end = capped_shard_end
+            print(
+                f"  WARNING: tracker assigned layers {assigned_shard_start}-{original_end}, "
+                f"but CPU-safe runtime budget fits {max_runtime_layers}/{assigned_num_layers} layers; "
+                f"loading layers {assigned_shard_start}-{assigned_shard_end} instead.",
+                flush=True,
+            )
         assigned_model_sources: list[dict] = net_assignment.get("model_sources", [])
         if _gap_found:
             print(
                 f"  Assigned gap: {assigned_hf_repo} "
-                f"layers {assigned_shard_start}–{assigned_shard_end} "
+                f"layers {assigned_shard_start}-{assigned_shard_end} "
                 f"(of {assigned_num_layers})",
                 flush=True,
             )
         else:
             print(
                 f"  Assigned redundant copy: {assigned_hf_repo} "
-                f"layers {assigned_shard_start}–{assigned_shard_end} "
+                f"layers {assigned_shard_start}-{assigned_shard_end} "
                 f"(of {assigned_num_layers})",
                 flush=True,
             )
@@ -866,11 +1014,14 @@ def run_startup(
             cache_dir=cache_dir,
             debug=debug,
             max_loaded_shards=max_loaded_shards,
+            force_cpu=force_cpu,
         )
         _node_start_time = time.monotonic()
         actual_port = node.start()
         public_host = advertise_host or (socket.getfqdn() if host == "0.0.0.0" else host)
         endpoint = f"http://{public_host}:{actual_port}"
+        if hasattr(node, "set_advertised_endpoint"):
+            node.set_advertised_endpoint(endpoint)
         local_base_url = f"http://127.0.0.1:{actual_port}"
         relay_bridge, relay_fields = _start_relay_bridge_if_available(
             tracker_url,
@@ -878,6 +1029,7 @@ def run_startup(
             local_base_url,
             endpoint,
             relay_url=relay_url,
+            node_name=node_name,
         )
         _attach_relay_bridge(node, relay_bridge)
         model_cache_path = _model_cache_path(assigned_hf_repo, cache_dir)
@@ -909,21 +1061,26 @@ def run_startup(
             ),
             **registration_capabilities,
             **relay_fields,
+            **display_fields,
         }
         tracker_node_id = _register_with_tracker(
             tracker_url, auto_reg_payload, node, _node_start_time,
         )
-        shard_count = assigned_shard_end - assigned_shard_start + 1
+        shard_label = _format_shard_label(
+            assigned_shard_start,
+            assigned_shard_end,
+            assigned_num_layers,
+        )
         print(
             f"\n{'=' * 32}\n"
             f"meshnet-node ready (auto-joined)\n"
             f"  Wallet:       {address}\n"
             f"  Model ID:     {assigned_hf_repo}\n"
-            f"  Shard:        layers {assigned_shard_start}–{assigned_shard_end} "
-            f"({shard_count} of {assigned_num_layers})\n"
-            f"  {_shard_budget_line(memory_budget_mb, memory_budget_source, assigned_num_layers, quantization)}\n"
+            f"  Shard:        {shard_label}\n"
+            f"  {_shard_budget_line(memory_budget_mb, memory_budget_source, assigned_num_layers, quantization, bytes_per_layer=assigned_bytes_per_layer, safety_fraction=_runtime_shard_safety_fraction(device))}\n"
             f"  Quantization: {quantization}\n"
             f"  Endpoint:     {endpoint}\n"
+            f"{_relay_ready_line(relay_fields)}"
             f"  Node ID:      {tracker_node_id or 'unregistered'}\n"
             f"  Hardware:     {_hardware_label(device, gpu_name)}\n"
             f"  Benchmark:    {bench_tps:,.0f} (throughput index)\n"
@@ -967,13 +1124,36 @@ def run_startup(
     peers: list[dict] = assignment.get("peers", [])
     model_sources: list[dict] = [] if tracker_source_disabled else assignment.get("model_sources", [])
     assignment_bytes_per_layer = _assignment_bytes_per_layer(assignment, quantization)
+    model_layers_end = assignment.get("model_layers_end")
+    assigned_total_layers = (
+        int(model_layers_end) + 1
+        if model_layers_end is not None
+        else None
+    )
+    shard_label = _format_shard_label(
+        shard_start,
+        shard_end,
+        assigned_total_layers,
+        model_name=assigned_model,
+    )
     if user_pinned_shard:
-        print(
-            f"  Shard: layers {shard_start}-{shard_end} of {assigned_model} (pinned)",
-            flush=True,
+        shard_label = f"{shard_label} (pinned)"
+    if user_pinned_shard and assigned_total_layers and assignment_bytes_per_layer:
+        pinned_layers = shard_end - shard_start + 1
+        max_layers = _max_assignable_layers(
+            memory_budget_mb,
+            assigned_total_layers,
+            assignment_bytes_per_layer,
+            safety_fraction=_runtime_shard_safety_fraction(device),
         )
-    else:
-        print(f"  Shard: layers {shard_start}-{shard_end} of {assigned_model}", flush=True)
+        if pinned_layers > max_layers:
+            raise ValueError(
+                f"Pinned shard layers {shard_start}–{shard_end} ({pinned_layers} layers) exceed "
+                f"the {memory_budget_mb / 1024:.1f} GB {memory_budget_source} budget "
+                f"(fits up to {max_layers}/{assigned_total_layers} layers at bfloat16). "
+                "Drop --shard-start/--shard-end to let the tracker auto-assign, or pin a smaller range."
+            )
+    print(f"  Shard: {shard_label}", flush=True)
 
     # 4. Download shard
     print("Downloading shard...", flush=True)
@@ -998,7 +1178,83 @@ def run_startup(
     )
     print(f"  Cached at: {shard_path}", flush=True)
 
-    # 5. Start HTTP server
+    # 5. Start HTTP server — real HF weights use TorchNodeServer; stub-model stays stub.
+    _node_start_time = time.monotonic()
+    if hf_repo and assigned_model != "stub-model":
+        print("Loading real PyTorch model shard...", flush=True)
+        node = TorchNodeServer(
+            host=host,
+            port=port,
+            model_id=hf_repo,
+            shard_start=shard_start,
+            shard_end=shard_end,
+            quantization=quantization,
+            tracker_url=tracker_url,
+            route_timeout=route_timeout,
+            cache_dir=shard_path,
+            debug=debug,
+            max_loaded_shards=max_loaded_shards,
+            force_cpu=force_cpu,
+        )
+        actual_port = node.start()
+        total_layers = getattr(getattr(node, "backend", None), "total_layers", None) or assigned_total_layers
+        shard_label = _format_shard_label(shard_start, shard_end, total_layers, model_name=assigned_model)
+        if user_pinned_shard:
+            shard_label = f"{shard_label} (pinned)"
+        public_host = advertise_host or (socket.getfqdn() if host == "0.0.0.0" else host)
+        endpoint = f"http://{public_host}:{actual_port}"
+        if hasattr(node, "set_advertised_endpoint"):
+            node.set_advertised_endpoint(endpoint)
+        local_base_url = f"http://127.0.0.1:{actual_port}"
+        relay_bridge, relay_fields = _start_relay_bridge_if_available(
+            tracker_url,
+            address,
+            local_base_url,
+            endpoint,
+            relay_url=relay_url,
+            node_name=node_name,
+        )
+        _attach_relay_bridge(node, relay_bridge)
+        reg_payload = {
+            "endpoint": endpoint,
+            "model": assigned_model,
+            "hf_repo": hf_repo,
+            "num_layers": total_layers,
+            "shard_start": shard_start,
+            "shard_end": shard_end,
+            "downloaded_models": downloaded_models,
+            "hardware_profile": hw,
+            "wallet_address": address,
+            "quantization": quantization,
+            "score": 1.0,
+            "tracker_mode": (shard_start == 0),
+            "managed_assignment": not user_pinned_shard,
+            "model_metadata": model_metadata_for(hf_repo, total_layers, cache_dir=shard_path),
+            **registration_capabilities,
+            **relay_fields,
+            **display_fields,
+        }
+        tracker_node_id = _register_with_tracker(
+            tracker_url, reg_payload, node, _node_start_time,
+        )
+        print(
+            f"\n{'=' * 32}\n"
+            f"meshnet-node ready\n"
+            f"  Wallet:       {address}\n"
+            f"  Model ID:     {hf_repo}\n"
+            f"  Shard:        {shard_label}\n"
+            f"  {_shard_budget_line(memory_budget_mb, memory_budget_source, total_layers, quantization, bytes_per_layer=assignment_bytes_per_layer, safety_fraction=_runtime_shard_safety_fraction(device))}\n"
+            f"  Quantization: {quantization}\n"
+            f"  Endpoint:     {endpoint}\n"
+            f"{_relay_ready_line(relay_fields)}"
+            f"  Node ID:      {tracker_node_id or 'unregistered'}\n"
+            f"  Hardware:     {_hardware_label(device, gpu_name)}\n"
+            f"  Benchmark:    {bench_tps:,.0f} (throughput index)\n"
+            f"{'=' * 32}",
+            flush=True,
+        )
+        return node
+
     is_last = shard_end >= assignment.get("model_layers_end", shard_end)
     node = StubNodeServer(
         host=host,
@@ -1009,7 +1265,6 @@ def run_startup(
         model=assigned_model,
         shard_path=shard_path,
     )
-    _node_start_time = time.monotonic()
     actual_port = node.start()
     public_host = advertise_host or (socket.getfqdn() if host == "0.0.0.0" else host)
     endpoint = f"http://{public_host}:{actual_port}"
@@ -1020,6 +1275,7 @@ def run_startup(
         local_base_url,
         endpoint,
         relay_url=relay_url,
+        node_name=node_name,
     )
     _attach_relay_bridge(node, relay_bridge)
 
@@ -1038,6 +1294,7 @@ def run_startup(
         "managed_assignment": not user_pinned_shard,
         **registration_capabilities,
         **relay_fields,
+        **display_fields,
     }
     try:
         reg_resp = _post_json(
@@ -1055,13 +1312,20 @@ def run_startup(
     hw_str = device.upper()
     if gpu_name:
         hw_str += f" ({gpu_name}, {vram_mb / 1024:.1f} GB)"
+    shard_label = _format_shard_label(
+        shard_start,
+        shard_end,
+        assigned_total_layers,
+        model_name=assigned_model,
+    )
     print(
         f"\n{'=' * 32}\n"
         f"meshnet-node ready\n"
         f"  Wallet:   {address}\n"
-        f"  Shard:    layers {shard_start}-{shard_end} ({assigned_model})\n"
-        f"  {_shard_budget_line(memory_budget_mb, memory_budget_source, assignment.get('model_layers_end', shard_end) + 1, quantization, bytes_per_layer=assignment_bytes_per_layer)}\n"
+        f"  Shard:    {shard_label}\n"
+        f"  {_shard_budget_line(memory_budget_mb, memory_budget_source, assigned_total_layers, quantization, bytes_per_layer=assignment_bytes_per_layer, safety_fraction=_runtime_shard_safety_fraction(device))}\n"
         f"  Endpoint: {endpoint}\n"
+        f"{_relay_ready_line(relay_fields)}"
         f"  Node ID:  {node_id}\n"
         f"  Hardware: {hw_str}\n"
         f"  Benchmark: {bench_tps:,.0f} (throughput index)\n"

packages/node/meshnet_node/torch_server.py

@@ -17,11 +17,17 @@ from typing import Any
 
 from .model_backend import (
     InsufficientVRAMError,
+    KVCacheMiss,
     MissingModelDependencyError,
     Quantization,
+    TailTokenResult,
     TorchModelShard,
     validate_quantization,
 )
+
+
+class _PipelineCacheMiss(Exception):
+    """A downstream hop reported 409 cache_miss — head must re-prefill."""
 from .server import (
     _WIRE_VERSION,
     _compress_body,
@@ -31,6 +37,69 @@ from .server import (
 )
 
 
+def _endpoint_key(url: str) -> str:
+    """Normalize http(s) endpoints for host:port comparison."""
+    parsed = urllib.parse.urlparse(url.rstrip("/"))
+    host = (parsed.hostname or "").lower()
+    if not host:
+        return url.rstrip("/").lower()
+    port = parsed.port
+    if port is None:
+        port = 443 if parsed.scheme == "https" else 80
+    return f"{host}:{port}"
+
+
+def _own_endpoint_key(server: _TorchHTTPServer) -> str:
+    advertised = getattr(server, "advertised_endpoint", None)
+    if advertised:
+        return _endpoint_key(advertised)
+    host, port = server.server_address
+    return _endpoint_key(f"http://{host}:{port}")
+
+
+def _clamp_downstream_hops(
+    hops: list[dict],
+    backend: TorchModelShard | None,
+) -> list[dict]:
+    """Ensure downstream start_layer continues after this shard's layers."""
+    if not hops or backend is None:
+        return hops
+    shard_end = getattr(backend, "shard_end", None)
+    if shard_end is None:
+        return hops
+    min_start = int(shard_end) + 1
+    clamped: list[dict] = []
+    for hop in hops:
+        adjusted = dict(hop)
+        if int(adjusted.get("start_layer", 0)) < min_start:
+            adjusted["start_layer"] = min_start
+        clamped.append(adjusted)
+    return clamped
+
+
+def _format_downstream_route(hops: list[dict]) -> str:
+    return ", ".join(
+        f"{h['endpoint']}@{h.get('start_layer', 0)}" for h in hops
+    )
+
+
+def _write_progress_line(state: list[bool], message: str, *, final: bool = False) -> None:
+    """Rewrite one in-place progress line (\\r) or finish with a newline."""
+    if final:
+        if state[0]:
+            sys.stdout.write("\r" + message + "\n")
+            state[0] = False
+        else:
+            print(message, flush=True)
+        return
+    if state[0]:
+        sys.stdout.write("\r" + message)
+    else:
+        sys.stdout.write(message)
+        state[0] = True
+    sys.stdout.flush()
+
+
 def _relay_hop(
     relay_addr: str,
     path: str,
@@ -40,23 +109,33 @@ def _relay_hop(
 ) -> tuple[int, dict[str, str], bytes]:
     """Send a single HTTP-shaped request through a relay RPC WebSocket.
 
-    relay_addr is the wss://relay.../rpc/{peer_id} URL.
+    relay_addr is the wss://relay.../rpc/{peer_id} URL. The request and any
+    binary response travel as binary frames (JSON header + raw body); relay
+    error responses and legacy peers still answer with JSON text frames.
     Returns (status, response_headers_lower, response_body).
     Raises on connection failure so callers can fall back to direct.
     """
     import websockets.sync.client as wsc  # type: ignore[import]
 
+    from .relay_bridge import decode_binary_frame, encode_binary_frame, ws_max_size
+
     request_id = f"{time.time_ns():x}"
-    payload = json.dumps({
+    frame = encode_binary_frame({
         "request_id": request_id,
         "method": "POST",
         "path": path,
         "headers": headers,
-        "body_base64": base64.b64encode(body).decode(),
-    })
-    with wsc.connect(relay_addr, open_timeout=timeout) as ws:
-        ws.send(payload)
+    }, body)
+    with wsc.connect(
+        relay_addr, open_timeout=timeout, max_size=ws_max_size(), compression=None,
+    ) as ws:
+        ws.send(frame)
         raw = ws.recv(timeout=timeout)
+    if isinstance(raw, (bytes, bytearray)):
+        resp_header, resp_body = decode_binary_frame(bytes(raw))
+        status = int(resp_header.get("status", 503))
+        resp_headers = {k.lower(): v for k, v in (resp_header.get("headers") or {}).items()}
+        return status, resp_headers, resp_body
     resp = json.loads(raw)
     status = int(resp.get("status", 503))
     resp_headers = {k.lower(): v for k, v in (resp.get("headers") or {}).items()}
@@ -65,6 +144,39 @@ def _relay_hop(
     return status, resp_headers, resp_body
 
 
+# Below this, zstd overhead outweighs the win (per-token decode bodies are ~KBs).
+_COMPRESS_MIN_BYTES = 64 * 1024
+
+
+def _maybe_compress_activation(body: bytes) -> tuple[bytes, str | None]:
+    """zstd-compress large activation bodies; returns (wire_body, encoding)."""
+    if len(body) < _COMPRESS_MIN_BYTES:
+        return body, None
+    try:
+        return _compress_body(body, "zstd"), "zstd"
+    except Exception:
+        return body, None
+
+
+def _is_cache_miss_body(body: bytes) -> bool:
+    try:
+        return json.loads(body).get("error") == "cache_miss"
+    except (json.JSONDecodeError, AttributeError, UnicodeDecodeError):
+        return False
+
+
+def _response_error_snippet(body: bytes, limit: int = 500) -> str:
+    """Return a compact error string from a downstream JSON/text response body."""
+    try:
+        payload = json.loads(body)
+        if isinstance(payload, dict):
+            message = payload.get("error") or payload.get("detail") or payload
+            return str(message)[:limit]
+    except (json.JSONDecodeError, TypeError, UnicodeDecodeError):
+        pass
+    return body.decode("utf-8", errors="replace")[:limit]
+
+
 class _TorchHTTPServer(http.server.HTTPServer):
     def __init__(
         self,
@@ -87,10 +199,60 @@ class _TorchHTTPServer(http.server.HTTPServer):
         self.route_timeout = route_timeout
         self.debug = debug
         self.max_loaded_shards = max(1, max_loaded_shards)
+        self.advertised_endpoint: str | None = None
         self.total_requests: int = 0
         self.failed_requests: int = 0
         self.queue_depth: int = 0
         self._stats_lock = threading.Lock()
+        self._active_requests: dict[str, dict[str, Any]] = {}
+        self._decode_log: dict[str, dict[str, float]] = {}
+
+    def note_decode_step(
+        self, session: str, now: float | None = None,
+    ) -> int | None:
+        """Count one decode forward; return the cumulative step count when a
+        log line is due (first step of a session, then every 5s), else None."""
+        if now is None:
+            now = time.monotonic()
+        with self._stats_lock:
+            rec = self._decode_log.get(session)
+            if rec is None:
+                if len(self._decode_log) >= 64:
+                    stale = [
+                        sid for sid, r in self._decode_log.items()
+                        if now - r["seen"] > 600.0
+                    ]
+                    for sid in stale:
+                        del self._decode_log[sid]
+                    while len(self._decode_log) >= 64:
+                        self._decode_log.pop(next(iter(self._decode_log)))
+                self._decode_log[session] = {"steps": 1.0, "logged": now, "seen": now}
+                return 1
+            rec["steps"] += 1
+            rec["seen"] = now
+            if now - rec["logged"] >= 5.0:
+                rec["logged"] = now
+                return int(rec["steps"])
+            return None
+
+    def snapshot_current_requests(self) -> list[dict[str, Any]]:
+        """In-flight request snapshots for tracker heartbeats."""
+        now = time.monotonic()
+        with self._stats_lock:
+            out: list[dict[str, Any]] = []
+            for rec in self._active_requests.values():
+                elapsed = max(now - float(rec["started"]), 1e-6)
+                tokens = int(rec.get("tokens") or 0)
+                out.append({
+                    "request_id": str(rec["request_id"]),
+                    "model": str(rec.get("model") or ""),
+                    "kind": str(rec.get("kind") or "chat"),
+                    "tokens": tokens,
+                    "elapsed_seconds": round(elapsed, 1),
+                    "tokens_per_sec": round(tokens / elapsed, 2) if tokens > 0 else 0.0,
+                    "routing_complete": bool(rec.get("routing_complete")),
+                })
+            return out
 
     def resolve_backend(self, model_name: str | None) -> TorchModelShard | None:
         if not model_name:
@@ -113,6 +275,53 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
     def log_message(self, fmt, *args):  # noqa: suppress request logs in tests
         pass
 
+    def _request_id(self) -> str:
+        return (
+            self.headers.get("X-Meshnet-Request-Id")
+            or self.headers.get("X-Request-Id")
+            or f"local-{time.time_ns():x}"
+        )
+
+    def _request_log_suffix(self) -> str:
+        req_id = self.headers.get("X-Meshnet-Request-Id") or self.headers.get("X-Request-Id")
+        return f" request_id={req_id}" if req_id else ""
+
+    def _track_request_begin(
+        self,
+        server: "_TorchHTTPServer",
+        request_id: str,
+        model: str,
+    ) -> None:
+        with server._stats_lock:
+            server._active_requests[request_id] = {
+                "request_id": request_id,
+                "model": model,
+                "kind": "chat",
+                "started": time.monotonic(),
+                "tokens": 0,
+                "routing_complete": False,
+            }
+
+    def _track_request_progress(
+        self,
+        server: "_TorchHTTPServer",
+        request_id: str,
+        *,
+        tokens: int,
+        routing_complete: bool = False,
+    ) -> None:
+        with server._stats_lock:
+            rec = server._active_requests.get(request_id)
+            if rec is None:
+                return
+            rec["tokens"] = tokens
+            if routing_complete:
+                rec["routing_complete"] = True
+
+    def _track_request_end(self, server: "_TorchHTTPServer", request_id: str) -> None:
+        with server._stats_lock:
+            server._active_requests.pop(request_id, None)
+
     def do_POST(self):
         server: _TorchHTTPServer = self.server  # type: ignore[assignment]
         if self.path == "/forward":
@@ -199,11 +408,45 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
             return
 
         server.forward_chunk_count += 1
-        if int(self.headers.get("X-Meshnet-Hop-Index", "0")) > 0:
+        hop_index = int(self.headers.get("X-Meshnet-Hop-Index", "0"))
+        if hop_index > 0:
             server.received_activations = True
+        # Session KV-cache protocol: prefill establishes per-session state on
+        # this node's layer range; decode reuses it. Absent header = legacy
+        # stateless call (also the signature fake backends implement).
+        cache_mode = self.headers.get("X-Meshnet-Cache")
+        if chunk_index_value == 0:
+            shard_start = getattr(server.backend, "shard_start", "?")
+            shard_end = getattr(server.backend, "shard_end", "?")
+            if cache_mode == "decode":
+                # One decode forward arrives per generated token — log a
+                # periodic per-session summary instead of one line per token.
+                steps = server.note_decode_step(session)
+                if steps is not None:
+                    print(
+                        f"  [node] decoding layers={shard_start}-{shard_end} "
+                        f"session={session[:8]} steps={steps}"
+                        f"{self._request_log_suffix()}",
+                        flush=True,
+                    )
+            else:
+                print(
+                    f"  [node] forward hop={hop_index} "
+                    f"layers={shard_start}-{shard_end} "
+                    f"session={session[:8]}{self._request_log_suffix()}",
+                    flush=True,
+                )
 
         start_layer_header = self.headers.get("X-Meshnet-Start-Layer")
         start_layer = int(start_layer_header) if start_layer_header else None
+        forward_kwargs: dict[str, object] = {}
+        if cache_mode in ("prefill", "decode"):
+            past_len_header = self.headers.get("X-Meshnet-Past-Len")
+            forward_kwargs = {
+                "session_id": session,
+                "cache_mode": cache_mode,
+                "past_len": int(past_len_header) if past_len_header else None,
+            }
 
         try:
             result = server.backend.forward_bytes(
@@ -212,11 +455,24 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
                 self.headers.get("X-Meshnet-Attn-Mask"),
                 self.headers.get("X-Meshnet-Position-Ids"),
                 start_layer=start_layer,
+                **forward_kwargs,
             )
+        except KVCacheMiss as exc:
+            self._send_json(409, {"error": "cache_miss", "detail": str(exc)})
+            return
         except Exception as exc:
+            print(
+                f"  [node] forward failed layers={getattr(server.backend, 'shard_start', '?')}-"
+                f"{getattr(server.backend, 'shard_end', '?')} session={session[:8]}: {exc}"
+                f"{self._request_log_suffix()}",
+                flush=True,
+            )
             self._send_json(500, {"error": str(exc)})
             return
 
+        if isinstance(result, TailTokenResult):
+            self._send_json(200, {"text": result.text, "token_id": result.token_id})
+            return
         if isinstance(result, str):
             self._send_json(200, {"text": result})
             return
@@ -280,12 +536,14 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
 
     def _handle_chat_completions(self) -> None:
         server: _TorchHTTPServer = self.server  # type: ignore[assignment]
+        request_id = self._request_id()
         with server._stats_lock:
             server.total_requests += 1
             server.queue_depth += 1
         try:
-            self._do_chat_completions(server)
+            self._do_chat_completions(server, request_id)
         finally:
+            self._track_request_end(server, request_id)
             with server._stats_lock:
                 server.queue_depth -= 1
 
@@ -294,7 +552,7 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
         with server._stats_lock:
             server.failed_requests += 1
 
-    def _do_chat_completions(self, server: "_TorchHTTPServer") -> None:
+    def _do_chat_completions(self, server: "_TorchHTTPServer", request_id: str) -> None:
         body = self._read_json_body()
         if body is None:
             return
@@ -307,31 +565,74 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
         if backend is None or not backend.is_head:
             self._send_json(400, {"error": "model not loaded on this node"})
             return
-        max_tokens = int(body.get("max_tokens") or body.get("max_new_tokens") or 256)
+        max_tokens = int(body.get("max_tokens") or body.get("max_new_tokens") or 5120)
         temperature = float(body.get("temperature") or 1.0)
         top_p = float(body.get("top_p") or 1.0)
 
+        self._track_request_begin(server, request_id, model_name)
+        print(
+            f"  [node] processing chat model={model_name!r} stream={stream} "
+            f"max_tokens={max_tokens}{self._request_log_suffix()}",
+            flush=True,
+        )
+
         # Fast path: this node owns the complete model — use HF generate() with KV cache.
         # Avoids the single-token-per-forward-pass limitation of the distributed path.
         if backend.is_head and backend.is_tail:
+            gen_started = time.monotonic()
+            progress_line = [False]
             try:
                 if stream:
-                    self._stream_openai_response(
-                        backend.generate_text_streaming(messages, max_tokens, temperature, top_p),
-                        model_name,
+                    token_count = 0
+
+                    def _counting_stream():
+                        nonlocal token_count
+                        for token_text in backend.generate_text_streaming(
+                            messages, max_tokens, temperature, top_p,
+                        ):
+                            if token_text:
+                                token_count += 1
+                                self._track_request_progress(
+                                    server, request_id, tokens=token_count, routing_complete=True,
+                                )
+                            yield token_text
+
+                    self._stream_openai_response(_counting_stream(), model_name)
+                    elapsed = time.monotonic() - gen_started
+                    tps = token_count / max(elapsed, 1e-6)
+                    _write_progress_line(
+                        progress_line,
+                        f"  [node] chat complete (stream) tokens={token_count} "
+                        f"elapsed_s={elapsed:.1f} tps={tps:.2f}{self._request_log_suffix()}",
+                        final=True,
                     )
                 else:
                     text = backend.generate_text(messages, max_tokens, temperature, top_p)
+                    completion_tokens = _backend_token_count(
+                        backend, "count_text_tokens", text, fallback=len(text.split()) or 1,
+                    )
+                    print(
+                        f"  [node] chat complete tokens={completion_tokens} "
+                        f"elapsed_s={time.monotonic() - gen_started:.1f}{self._request_log_suffix()}",
+                        flush=True,
+                    )
                     self._send_openai_response(text, model_name, False, messages, backend=backend)
             except Exception as exc:
                 self._record_failed_request()
+                print(
+                    f"  [node] chat failed after {time.monotonic() - gen_started:.1f}s: {exc}"
+                    f"{self._request_log_suffix()}",
+                    flush=True,
+                )
                 self._send_json(500, {"error": f"generation failed: {exc}"})
             return
 
-        # Distributed path: autoregressive generation across shards.
-        # We do N single-step forward passes (no cross-node KV cache), which is slow
-        # but correct. Each step: head encodes current sequence → forwards through route
-        # → tail returns the next token string → append → repeat.
+        # Distributed path: autoregressive generation across shards with a
+        # sharded per-node KV cache. Step 0 prefills the full prompt through the
+        # route (each node caches state for its own layer range, keyed by a
+        # per-generation session id); steps 1+ send only the newest token's
+        # hidden state. A 409 cache_miss from any hop (eviction/restart/route
+        # change) falls back to a full re-prefill — the old stateless behavior.
         remaining_route = self._get_remaining_route(model_name, backend=backend)
         print(
             f"  [node] chat route model={model_name!r} max_tokens={max_tokens} "
@@ -364,30 +665,133 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
         generated: list[str] = []
         current_text = prompt_text
 
+        session_id = str(uuid.uuid4())
+        use_kv = bool(getattr(backend, "supports_kv_cache", False))
+        # EOS detection by id must work on the stateless path too: the tail
+        # returns token_id regardless of caching, and EOS usually decodes to
+        # "" (skip_special_tokens), so the text comparison never fires.
+        eos_ids: set[int] = set()
+        try:
+            eos_ids = set(backend.eos_token_ids())
+        except Exception:
+            eos_ids = set()
+
         stream_emit = None
         if stream:
             stream_emit = self._start_openai_stream(model_name)
+            self._track_request_progress(server, request_id, tokens=0, routing_complete=True)
 
-        for _ in range(max_tokens):
+        _GENERATION_LOG_INTERVAL = 5.0
+        gen_started = time.monotonic()
+        last_gen_log = gen_started
+        progress_line = [False]
+        last_token_id: int | None = None
+        failure_reason: str | None = None
+
+        def _prefill_step() -> tuple[str, int | None]:
+            """Full-sequence prefill: initial step and cache-miss recovery."""
+            payload = (
+                backend.encode_prompt(current_text, session_id=session_id)
+                if use_kv
+                else backend.encode_prompt(current_text)
+            )
+            return self._run_downstream_pipeline(
+                payload, remaining_route, backend=backend,
+                session=session_id, cache_mode="prefill" if use_kv else None,
+            )
+
+        for step in range(max_tokens):
             try:
-                payload = backend.encode_prompt(current_text)
+                if use_kv and step > 0 and last_token_id is not None:
+                    try:
+                        payload = backend.encode_next_token(last_token_id, session_id)
+                        token_str, token_id = self._run_downstream_pipeline(
+                            payload, remaining_route, backend=backend,
+                            session=session_id, cache_mode="decode",
+                        )
+                    except (KVCacheMiss, _PipelineCacheMiss) as miss:
+                        # Evicted/restarted node or head lost its own session:
+                        # re-prefill the whole sequence once and continue cached.
+                        print(
+                            f"  [node] kv cache miss at step {step} ({miss}); "
+                            f"re-prefilling {len(current_text)} chars",
+                            flush=True,
+                        )
+                        token_str, token_id = _prefill_step()
+                else:
+                    token_str, token_id = _prefill_step()
+            except _PipelineCacheMiss as exc:
+                print(f"  [node] unexpected cache miss on prefill: {exc}", flush=True)
+                failure_reason = f"cache miss on prefill: {exc}"
+                break
             except Exception as exc:
                 print(f"  [node] distributed encode error: {exc}", flush=True)
-                break
-            token_str = self._run_downstream_pipeline(payload, remaining_route, backend=backend)
-            if not token_str:
+                failure_reason = f"distributed encode error: {exc}"
                 break
             # Stop on error responses or EOS.
             if token_str.startswith(("pipeline error", "decode error", "no downstream", "error:")):
+                failure_reason = token_str
+                break
+            if token_id is not None and token_id in eos_ids:
                 break
             if eos_token and token_str == eos_token:
                 break
-            generated.append(token_str)
-            if stream_emit is not None:
-                stream_emit(token_str)
-            current_text = current_text + token_str
+            if not token_str and token_id is None:
+                break
+            last_token_id = token_id
+            # token_str can be empty for a skipped special token that is not
+            # EOS — keep generating from its token_id without emitting text.
+            if token_str:
+                generated.append(token_str)
+                if stream_emit is not None:
+                    stream_emit(token_str)
+                current_text = current_text + token_str
+            self._track_request_progress(
+                server,
+                request_id,
+                tokens=len(generated),
+                routing_complete=True,
+            )
+            now = time.monotonic()
+            if step == 0 or now - last_gen_log >= _GENERATION_LOG_INTERVAL:
+                elapsed = now - gen_started
+                token_count = len(generated)
+                tps = token_count / max(elapsed, 1e-6)
+                _write_progress_line(
+                    progress_line,
+                    f"  [node] generating step={step + 1}/{max_tokens} "
+                    f"tokens={token_count} elapsed_s={elapsed:.1f} tps={tps:.2f}",
+                )
+                last_gen_log = now
+
+        if use_kv:
+            try:
+                backend.release_session(session_id)
+            except Exception:
+                pass
+
+        if generated:
+            elapsed = time.monotonic() - gen_started
+            token_count = len(generated)
+            tps = token_count / max(elapsed, 1e-6)
+            _write_progress_line(
+                progress_line,
+                f"  [node] generation complete tokens={token_count} "
+                f"elapsed_s={elapsed:.1f} tps={tps:.2f}",
+                final=True,
+            )
 
         result_text = "".join(generated)
+        # A failure before the first token is an upstream error, not an empty
+        # completion — tell the client instead of returning a blank 200.
+        if failure_reason and not generated:
+            if stream_emit is not None:
+                stream_emit(None, error=failure_reason)
+                return
+            self._send_json(502, {
+                "error": {"message": failure_reason, "type": "upstream_error"},
+            })
+            return
         if stream_emit is not None:
             stream_emit(None)
             return
@@ -401,6 +805,9 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
         start_layer tells each downstream node which layer to begin from,
         enabling correct execution when shard ranges overlap.
         """
+        server: _TorchHTTPServer = self.server  # type: ignore[assignment]
+        active_backend = backend or server.backend
+
         # Fast path: tracker pre-resolved the downstream route and injected it as a header.
         injected = self.headers.get("X-Meshnet-Route")
         if injected:
@@ -419,14 +826,16 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
                             hops.append(hop)
                         elif isinstance(item, str):
                             hops.append({"endpoint": item, "start_layer": 0})
-                    print(f"  [node] using injected downstream route: {[h['endpoint'] for h in hops]}", flush=True)
+                    hops = _clamp_downstream_hops(hops, active_backend)
+                    print(
+                        f"  [node] using injected downstream route: {_format_downstream_route(hops)}",
+                        flush=True,
+                    )
                     return hops
             except (json.JSONDecodeError, TypeError, KeyError):
                 pass
 
         # Slow path: query the tracker (direct node-to-node calls, or tracker didn't inject).
-        server: _TorchHTTPServer = self.server  # type: ignore[assignment]
-        active_backend = backend or server.backend
         if server.tracker_url is None:
             return []
         route_model = getattr(active_backend, "model_id", None) or model
@@ -434,29 +843,51 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
             url = f"{server.tracker_url}/v1/route?model={urllib.parse.quote(route_model)}"
             with urllib.request.urlopen(url, timeout=server.route_timeout) as r:
                 route_resp = json.loads(r.read())
-            own_port = server.server_address[1]
+            own_key = _own_endpoint_key(server)
             nodes_info = route_resp.get("nodes", [])
-            hops = []
-            covered_up_to: int | None = None
+            hops: list[dict] = []
+            passed_self = False
             for node_info in nodes_info:
                 ep = node_info.get("endpoint", "")
-                if ep.rstrip("/").endswith(f":{own_port}"):
-                    covered_up_to = node_info.get("shard_end")
+                if not ep:
                     continue
-                if covered_up_to is None:
-                    covered_up_to = (node_info.get("shard_start") or 1) - 1
-                hop = {"endpoint": ep, "start_layer": covered_up_to + 1}
+                if _endpoint_key(ep) == own_key:
+                    passed_self = True
+                    continue
+                if not passed_self:
+                    continue
+                hop = {
+                    "endpoint": ep,
+                    "start_layer": int(node_info.get("start_layer", 0)),
+                }
                 if node_info.get("relay_addr"):
                     hop["relay_addr"] = str(node_info["relay_addr"])
                 hops.append(hop)
-                covered_up_to = node_info.get("shard_end", covered_up_to)
-            print(f"  [node] tracker downstream route: {[h['endpoint'] for h in hops]}", flush=True)
+            hops = _clamp_downstream_hops(hops, active_backend)
+            print(
+                f"  [node] tracker downstream route: {_format_downstream_route(hops)}",
+                flush=True,
+            )
             return hops
         except Exception as exc:
             print(f"  [node] WARNING: route lookup failed for {route_model!r}: {exc}", flush=True)
             return []
 
-    def _run_downstream_pipeline(self, payload: object, route: list[dict], *, backend: TorchModelShard | None = None) -> str:
+    def _run_downstream_pipeline(
+        self,
+        payload: object,
+        route: list[dict],
+        *,
+        backend: TorchModelShard | None = None,
+        session: str | None = None,
+        cache_mode: str | None = None,
+    ) -> tuple[str, int | None]:
+        """Forward an activation through the downstream route.
+
+        Returns (token_text, token_id) — token_id is None when a hop predates
+        the KV-cache protocol. Raises _PipelineCacheMiss when a hop responds
+        409 cache_miss (evicted/restarted node) so the caller can re-prefill.
+        """
         server: _TorchHTTPServer = self.server  # type: ignore[assignment]
         active_backend = backend or server.backend
         if not route:
@@ -468,12 +899,17 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
                         bytearray(payload.body),  # type: ignore[union-attr]
                         dtype=active_backend.torch.bfloat16,
                     ).reshape(payload.shape).to(active_backend.device)  # type: ignore[union-attr]
-                    return active_backend.decode_tail(tensor)
+                    if hasattr(active_backend, "decode_tail_token"):
+                        tail = active_backend.decode_tail_token(tensor)
+                        return tail.text, tail.token_id
+                    return active_backend.decode_tail(tensor), None
                 except Exception as exc:
-                    return f"decode error: {exc}"
-            return "no downstream route available for non-tail shard"
+                    return f"decode error: {exc}", None
+            return "no downstream route available for non-tail shard", None
 
-        session = str(uuid.uuid4())
+        # Session is stable across all steps of one generation when the caller
+        # provides it (KV-cache protocol); fresh per call otherwise (legacy).
+        session = session or str(uuid.uuid4())
         shape = payload.shape  # type: ignore[union-attr]
         attn_mask = payload.attention_mask_header  # type: ignore[union-attr]
         pos_ids = payload.position_ids_header  # type: ignore[union-attr]
@@ -492,6 +928,7 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
                     + (f" relay={relay_addr}" if relay_addr else ""),
                     flush=True,
                 )
+            wire_body, wire_encoding = _maybe_compress_activation(current_body)
             headers: dict[str, str] = {
                 "Content-Type": "application/octet-stream",
                 "X-Meshnet-Wire": _WIRE_VERSION,
@@ -503,6 +940,13 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
                 "X-Meshnet-Hop-Index": str(hop_index),
                 "X-Meshnet-Start-Layer": str(start_layer),
             }
+            if wire_encoding:
+                headers["X-Meshnet-Encoding"] = wire_encoding
+            if cache_mode:
+                headers["X-Meshnet-Cache"] = cache_mode
+                past_len = getattr(payload, "past_len", None)
+                if cache_mode == "decode" and past_len is not None:
+                    headers["X-Meshnet-Past-Len"] = str(past_len)
             if current_attn:
                 headers["X-Meshnet-Attn-Mask"] = current_attn
             if current_pos:
@@ -510,14 +954,19 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
             if relay_addr:
                 try:
                     status, resp_headers, resp_body = _relay_hop(
-                        relay_addr, "/forward", current_body, headers, timeout=120.0,
+                        relay_addr, "/forward", wire_body, headers, timeout=120.0,
                     )
+                    if status == 409 and _is_cache_miss_body(resp_body):
+                        raise _PipelineCacheMiss(node_url)
                     if status >= 400:
+                        detail = _response_error_snippet(resp_body)
                         print(
-                            f"  [node] relay hop {hop_index} returned {status} from {relay_addr}",
+                            f"  [node] relay hop {hop_index} returned {status} from {relay_addr}: {detail}",
                             flush=True,
                         )
-                        return f"pipeline error at {node_url} via relay: status {status}"
+                        return f"pipeline error at {node_url} via relay: status {status}: {detail}", None
+                except _PipelineCacheMiss:
+                    raise
                 except Exception as exc:
                     print(
                         f"  [node] relay hop {hop_index} failed at {relay_addr}: {exc}; "
@@ -528,7 +977,7 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
             if not relay_addr:
                 req = urllib.request.Request(
                     f"{node_url}/forward",
-                    data=current_body,
+                    data=wire_body,
                     headers=headers,
                     method="POST",
                 )
@@ -536,26 +985,40 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
                     with urllib.request.urlopen(req, timeout=120.0) as r:
                         resp_body = r.read()
                         resp_headers = {k.lower(): v for k, v in r.headers.items()}
+                except urllib.error.HTTPError as exc:
+                    body = exc.read()
+                    if exc.code == 409 and _is_cache_miss_body(body):
+                        raise _PipelineCacheMiss(node_url) from exc
+                    detail = _response_error_snippet(body)
+                    print(f"  [node] pipeline hop {hop_index} failed at {node_url}: {exc}: {detail}", flush=True)
+                    return f"pipeline error at {node_url}: {exc}: {detail}", None
                 except Exception as exc:
                     print(f"  [node] pipeline hop {hop_index} failed at {node_url}: {exc}", flush=True)
-                    return f"pipeline error at {node_url}: {exc}"
+                    return f"pipeline error at {node_url}: {exc}", None
             content_type = resp_headers.get("content-type", "")
             if "application/json" in content_type:
                 try:
                     data = json.loads(resp_body)
                     text = str(data.get("text", ""))
+                    token_id = data.get("token_id")
                     if server.debug:
                         print(f"  [node] pipeline hop {hop_index} returned text={text!r}", flush=True)
-                    return text
+                    return text, int(token_id) if token_id is not None else None
                 except json.JSONDecodeError:
-                    return resp_body.decode("utf-8", errors="replace")
+                    return resp_body.decode("utf-8", errors="replace"), None
             # Binary activation — update and forward to next node
             shape_header = resp_headers.get("x-meshnet-shape", ",".join(str(d) for d in current_shape))
             current_shape = _parse_shape(shape_header)
-            current_body = resp_body
+            try:
+                current_body = _decompress_body(
+                    resp_body, resp_headers.get("x-meshnet-encoding")
+                )
+            except ValueError as exc:
+                print(f"  [node] pipeline hop {hop_index} bad response encoding: {exc}", flush=True)
+                return f"pipeline error at {node_url}: {exc}", None
             current_attn = resp_headers.get("x-meshnet-attn-mask")
             current_pos = resp_headers.get("x-meshnet-position-ids")
-        return ""
+        return "", None
 
     def _stream_openai_response(self, token_iter, model: str) -> None:
         """Stream tokens from an iterator as SSE chunks."""
@@ -588,7 +1051,17 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
             "choices": [{"index": 0, "delta": {"role": "assistant", "content": ""}, "finish_reason": None}],
         }))
 
-        def emit_token(token_text: str | None) -> None:
+        def emit_token(token_text: str | None, *, error: str | None = None) -> None:
+            if error is not None:
+                # OpenAI-style mid-stream error frame; clients surface it
+                # instead of showing an empty completion.
+                _emit(json.dumps({"error": {"message": error, "type": "upstream_error"}}))
+                try:
+                    self.wfile.write(b"data: [DONE]\n\n")
+                    self.wfile.flush()
+                except (BrokenPipeError, ConnectionResetError):
+                    pass
+                return
             if token_text is None:
                 _emit(json.dumps({
                     "id": chunk_id, "object": "chat.completion.chunk", "created": created,
@@ -733,6 +1206,7 @@ class TorchNodeServer:
         cache_dir: Path | None = None,
         debug: bool = False,
         max_loaded_shards: int = 1,
+        force_cpu: bool = False,
     ) -> None:
         self._host = host
         self._requested_port = port
@@ -743,6 +1217,7 @@ class TorchNodeServer:
             shard_end,
             quantization,
             cache_dir,
+            force_cpu=force_cpu,
         )
         self._backends: dict[str, TorchModelShard] = {self._backend.model_id: self._backend}
         # Auto-detect tracker mode: enabled when shard_start == 0 or explicitly set
@@ -783,6 +1258,12 @@ class TorchNodeServer:
     def queue_depth(self) -> int:
         return self._server.queue_depth if self._server is not None else 0
 
+    @property
+    def current_requests(self) -> list[dict[str, Any]]:
+        if self._server is None:
+            return []
+        return self._server.snapshot_current_requests()
+
     @property
     def loaded_model_ids(self) -> list[str]:
         return list(self._backends.keys())
@@ -862,6 +1343,11 @@ class TorchNodeServer:
         self._thread.start()
         return self.port
 
+    def set_advertised_endpoint(self, endpoint: str) -> None:
+        """Set the LAN-facing endpoint used for route self-detection."""
+        if self._server is not None:
+            self._server.advertised_endpoint = endpoint
+
     def stop(self) -> None:
         if self._server is None:
             return
@@ -880,12 +1366,15 @@ def _load_backend(
     shard_end: int,
     quantization: str,
     cache_dir: Path | None = None,
+    force_cpu: bool = False,
 ) -> TorchModelShard:
     from .model_backend import load_torch_shard
 
     quant = validate_quantization(quantization)
     try:
-        return load_torch_shard(model_id, shard_start, shard_end, quant, cache_dir)
+        return load_torch_shard(
+            model_id, shard_start, shard_end, quant, cache_dir, force_cpu=force_cpu
+        )
     except MissingModelDependencyError:
         raise
     except InsufficientVRAMError as exc:

packages/node/pyproject.toml

@@ -16,7 +16,8 @@ dependencies = [
     "rich>=13",
     "safetensors>=0.4",
     "torch>=2.1",
-    "transformers>=4.39",
+    "transformers>=5.12",
+    "triton-windows>=3.7; platform_system == 'Windows'",
     "websockets>=13",
     "zstandard>=0.22",
     "kernels>=0.11.1,<0.16",

packages/relay/meshnet_relay/server.py

@@ -15,6 +15,7 @@ from __future__ import annotations
 import asyncio
 import json
 import logging
+import os
 import threading
 import uuid
 from pathlib import Path
@@ -23,6 +24,42 @@ from .peer_registry import PeerRegistry
 
 log = logging.getLogger(__name__)
 
+# Activation tensors ride the relay as base64 inside one JSON frame, so the
+# websockets default of 1 MiB rejects any real prefill (close code 1009).
+DEFAULT_WS_MAX_BYTES = 256 * 1024 * 1024
+
+
+def ws_max_size() -> int | None:
+    """Max inbound WebSocket frame size; MESHNET_WS_MAX_BYTES<=0 means unlimited."""
+    raw = os.environ.get("MESHNET_WS_MAX_BYTES", "").strip()
+    if not raw:
+        return DEFAULT_WS_MAX_BYTES
+    try:
+        value = int(raw)
+    except ValueError:
+        return DEFAULT_WS_MAX_BYTES
+    return None if value <= 0 else value
+
+
+# Binary relay frame: JSON header (request/response metadata) + raw body in one
+# WebSocket binary message. Activation bodies stay bytes end to end — no base64
+# inflation, no JSON re-encode of megabytes per hop. Text JSON frames remain the
+# control plane (gossip, peer-register, streamed SSE responses).
+BINARY_FRAME_MAGIC = b"MRF1"
+
+
+def encode_binary_frame(header: dict, body: bytes) -> bytes:
+    header_bytes = json.dumps(header, separators=(",", ":")).encode()
+    return BINARY_FRAME_MAGIC + len(header_bytes).to_bytes(4, "big") + header_bytes + body
+
+
+def decode_binary_frame(frame: bytes) -> tuple[dict, bytes]:
+    if len(frame) < 8 or frame[:4] != BINARY_FRAME_MAGIC:
+        raise ValueError("not a meshnet binary relay frame")
+    header_len = int.from_bytes(frame[4:8], "big")
+    header = json.loads(frame[8:8 + header_len].decode())
+    return header, bytes(frame[8 + header_len:])
+
 
 class RelayServer:
     """Async WebSocket relay server that runs in a background thread.
@@ -100,6 +137,10 @@ class RelayServer:
             self.host,
             self.port,
             ssl=ssl_ctx,
+            max_size=ws_max_size(),
+            # Bulk payloads are zstd-compressed at the pipeline layer already;
+            # per-message deflate would recompress them on every hop for nothing.
+            compression=None,
         )
         # Record actual port after bind
         for sock in server.sockets or []:
@@ -144,6 +185,17 @@ class RelayServer:
 
         try:
             async for raw in ws:
+                # Binary frames are relay-http-response bodies from a bridge —
+                # route them to the waiting rpc requester as-is, never fan out.
+                if isinstance(raw, (bytes, bytearray)):
+                    try:
+                        header, _ = decode_binary_frame(bytes(raw))
+                    except (ValueError, json.JSONDecodeError):
+                        continue
+                    queue = self._pending_rpc.get(header.get("request_id"))
+                    if queue is not None:
+                        queue.put_nowait(bytes(raw))
+                    continue
                 try:
                     envelope = json.loads(raw)
                 except (json.JSONDecodeError, TypeError):
@@ -234,14 +286,27 @@ class RelayServer:
 
         try:
             raw = await asyncio.wait_for(ws_requester.recv(), timeout=30.0)
-            payload = json.loads(raw)
+            if isinstance(raw, (bytes, bytearray)):
+                header, body = decode_binary_frame(bytes(raw))
+                request_id = str(header.get("request_id") or uuid.uuid4())
+                header["request_id"] = request_id
+                header["target_peer"] = target_peer_id
+                outbound: str | bytes = encode_binary_frame(header, body)
+            else:
+                payload = json.loads(raw)
+                request_id = str(payload.get("request_id") or uuid.uuid4())
+                payload["request_id"] = request_id
+                payload["target_peer"] = target_peer_id
+                outbound = json.dumps({
+                    "topic": "relay-http-request",
+                    "version": 1,
+                    "from_peer": "relay",
+                    "payload": payload,
+                })
         except Exception:
             await ws_requester.close(1003, "invalid relay rpc request")
             return
 
-        request_id = str(payload.get("request_id") or uuid.uuid4())
-        payload["request_id"] = request_id
-        payload["target_peer"] = target_peer_id
         queue: asyncio.Queue = asyncio.Queue()
         self._pending_rpc[request_id] = queue
         overall_timeout = 310.0
@@ -249,15 +314,11 @@ class RelayServer:
         loop = asyncio.get_running_loop()
         deadline = loop.time() + overall_timeout
         try:
-            await target.ws.send(json.dumps({
-                "topic": "relay-http-request",
-                "version": 1,
-                "from_peer": "relay",
-                "payload": payload,
-            }))
+            await target.ws.send(outbound)
             # Forward frames until a terminal one: streamed responses (US-036)
             # end with {"stream": true, "done": true}; a frame without "stream"
-            # is a complete legacy single response.
+            # is a complete legacy single response. A binary frame is always a
+            # complete single response.
             while True:
                 remaining = deadline - loop.time()
                 if remaining <= 0:
@@ -265,6 +326,9 @@ class RelayServer:
                 frame = await asyncio.wait_for(
                     queue.get(), timeout=min(idle_timeout, remaining)
                 )
+                if isinstance(frame, (bytes, bytearray)):
+                    await ws_requester.send(frame)
+                    break
                 await ws_requester.send(json.dumps(frame))
                 if not frame.get("stream") or frame.get("done"):
                     break

packages/tracker/meshnet_tracker/accounts.py

@@ -8,7 +8,8 @@ regular user.
 Mutations are append-only events with unique ids — the same replication
 model as ``BillingLedger`` — so accounts and API keys converge across the
 tracker hive via gossip, and every dashboard can serve registration/login.
-Sessions are deliberately local to each tracker (bearer tokens in memory).
+Sessions are local to each tracker and persisted so dashboard cookies survive
+tracker restarts.
 """
 
 from __future__ import annotations
@@ -26,9 +27,24 @@ DEFAULT_ACCOUNTS_DB_PATH = "accounts.sqlite"
 SESSION_TTL = 7 * 86400.0  # seconds
 PBKDF2_ITERATIONS = 200_000
 MIN_PASSWORD_LENGTH = 8
+_MAX_NICKNAME_LENGTH = 64
 API_KEY_PREFIX = "sk-mesh-"
 
 _EMAIL_RE = re.compile(r"^[^@\s]+@[^@\s]+\.[^@\s]+$")
+_UNSET = object()
+
+
+def _normalize_nickname(value: object) -> str | None:
+    if value is None:
+        return None
+    if not isinstance(value, str):
+        raise ValueError("nickname must be a string")
+    nickname = value.strip()
+    if not nickname:
+        return None
+    if len(nickname) > _MAX_NICKNAME_LENGTH:
+        raise ValueError(f"nickname must be at most {_MAX_NICKNAME_LENGTH} characters")
+    return nickname
 
 
 def _hash_password(password: str, salt: str) -> str:
@@ -63,6 +79,7 @@ class AccountStore:
         email: str | None = None,
         wallet: str | None = None,
         password: str,
+        nickname: str | None = None,
     ) -> dict:
         """Create an account. The first account becomes the admin.
 
@@ -76,6 +93,7 @@ class AccountStore:
             raise ValueError("invalid email address")
         if len(password or "") < MIN_PASSWORD_LENGTH:
             raise ValueError(f"password must be at least {MIN_PASSWORD_LENGTH} characters")
+        nickname = _normalize_nickname(nickname)
         with self._lock:
             for identifier in filter(None, (email, wallet)):
                 if identifier.lower() in self._by_identifier:
@@ -90,11 +108,30 @@ class AccountStore:
                 "role": "admin" if not self._accounts else "user",
                 "password_hash": _hash_password(password, salt),
                 "salt": salt,
+                "nickname": nickname,
                 "ts": time.time(),
             }
             self._apply_locked(event)
             return self._public_view(self._accounts[event["account_id"]])
 
+    def update_profile(self, account_id: str, *, nickname: str | None = _UNSET) -> dict:
+        """Update display fields for an account. Pass nickname=None to clear."""
+        if nickname is not _UNSET:
+            nickname = _normalize_nickname(nickname)
+        with self._lock:
+            if account_id not in self._accounts:
+                raise ValueError("unknown account")
+            event = {
+                "id": f"profile-{uuid.uuid4().hex}",
+                "type": "update_profile",
+                "account_id": account_id,
+                "ts": time.time(),
+            }
+            if nickname is not _UNSET:
+                event["nickname"] = nickname
+            self._apply_locked(event)
+            return self._public_view(self._accounts[account_id])
+
     def verify_login(self, identifier: str, password: str) -> dict | None:
         """Return the public account view when credentials match, else None."""
         with self._lock:
@@ -115,6 +152,8 @@ class AccountStore:
                 "account_id": account_id,
                 "expires": time.time() + SESSION_TTL,
             }
+            self._dirty = True
+        self.save_to_db()
         return token
 
     def session_account(self, token: str | None) -> dict | None:
@@ -134,7 +173,9 @@ class AccountStore:
         if not token:
             return
         with self._lock:
-            self._sessions.pop(token, None)
+            if self._sessions.pop(token, None) is not None:
+                self._dirty = True
+        self.save_to_db()
 
     # ---- API keys ----
 
@@ -191,6 +232,7 @@ class AccountStore:
             "account_id": record["account_id"],
             "email": record.get("email"),
             "wallet": record.get("wallet"),
+            "nickname": record.get("nickname"),
             "role": record["role"],
             "created_ts": record.get("ts", 0.0),
         }
@@ -244,11 +286,19 @@ class AccountStore:
                 "role": event.get("role", "user"),
                 "password_hash": event["password_hash"],
                 "salt": event["salt"],
+                "nickname": event.get("nickname"),
                 "ts": float(event.get("ts", 0.0)),
             }
             self._accounts[account_id] = record
             for identifier in filter(None, (record["email"], record["wallet"])):
                 self._by_identifier.setdefault(identifier.lower(), account_id)
+        elif etype == "update_profile":
+            account_id = event["account_id"]
+            record = self._accounts.get(account_id)
+            if record is None:
+                return
+            if "nickname" in event:
+                record["nickname"] = event.get("nickname")
         elif etype == "create_key":
             api_key = event["api_key"]
             if api_key not in self._revoked_keys:
@@ -271,6 +321,10 @@ class AccountStore:
             "CREATE TABLE IF NOT EXISTS account_events "
             "(event_id TEXT PRIMARY KEY, payload TEXT NOT NULL, ts REAL NOT NULL)"
         )
+        con.execute(
+            "CREATE TABLE IF NOT EXISTS account_sessions "
+            "(token TEXT PRIMARY KEY, account_id TEXT NOT NULL, expires REAL NOT NULL)"
+        )
         con.commit()
         con.close()
 
@@ -279,6 +333,10 @@ class AccountStore:
         rows = con.execute(
             "SELECT payload FROM account_events ORDER BY ts, event_id"
         ).fetchall()
+        session_rows = con.execute(
+            "SELECT token, account_id, expires FROM account_sessions WHERE expires >= ?",
+            (time.time(),),
+        ).fetchall()
         con.close()
         with self._lock:
             for (payload,) in rows:
@@ -288,6 +346,11 @@ class AccountStore:
                     continue
                 if event.get("id") not in self._seen_event_ids:
                     self._apply_locked(event)
+            self._sessions = {
+                token: {"account_id": account_id, "expires": float(expires)}
+                for token, account_id, expires in session_rows
+                if account_id in self._accounts
+            }
             self._dirty = False
 
     def save_to_db(self) -> None:
@@ -297,11 +360,21 @@ class AccountStore:
             if not self._dirty:
                 return
             events = list(self._event_log)
+            sessions = [
+                (token, session["account_id"], float(session["expires"]))
+                for token, session in self._sessions.items()
+                if session["expires"] >= time.time()
+            ]
             self._dirty = False
         con = sqlite3.connect(self._db_path)  # type: ignore[arg-type]
         con.executemany(
             "INSERT OR IGNORE INTO account_events (event_id, payload, ts) VALUES (?, ?, ?)",
             [(e["id"], json.dumps(e), float(e.get("ts", 0.0))) for e in events],
         )
+        con.execute("DELETE FROM account_sessions")
+        con.executemany(
+            "INSERT INTO account_sessions (token, account_id, expires) VALUES (?, ?, ?)",
+            sessions,
+        )
         con.commit()
         con.close()

packages/tracker/meshnet_tracker/billing.py

@@ -453,13 +453,12 @@ class BillingLedger:
         with self._lock:
             return self._node_pending.get(wallet, 0.0)
 
-    def usage_for(self, api_keys: list[str], *, recent_limit: int | None = None) -> dict:
-        """Aggregate charge history for a set of API keys (dashboard view)."""
+    def usage_totals_for(self, api_keys: list[str]) -> dict:
+        """Aggregate charge totals without per-request records (dashboard summary)."""
         keys = set(api_keys)
         requests = 0
         total_tokens = 0
         total_cost = 0.0
-        records: list[dict] = []
         with self._lock:
             for event in self._event_log:
                 if event.get("type") != "charge" or event.get("api_key") not in keys:
@@ -467,6 +466,20 @@ class BillingLedger:
                 requests += 1
                 total_tokens += int(event.get("total_tokens", 0))
                 total_cost += float(event.get("cost", 0.0))
+        return {
+            "requests": requests,
+            "total_tokens": total_tokens,
+            "total_cost": total_cost,
+        }
+
+    def usage_for(self, api_keys: list[str], *, recent_limit: int | None = None) -> dict:
+        """Aggregate charge history for a set of API keys (dashboard view)."""
+        keys = set(api_keys)
+        records: list[dict] = []
+        with self._lock:
+            for event in self._event_log:
+                if event.get("type") != "charge" or event.get("api_key") not in keys:
+                    continue
                 records.append({
                     "api_key": event["api_key"],
                     "model": event.get("model"),
@@ -476,9 +489,9 @@ class BillingLedger:
                 })
         recent = records[-recent_limit:] if recent_limit is not None else records
         return {
-            "requests": requests,
-            "total_tokens": total_tokens,
-            "total_cost": total_cost,
+            "requests": len(records),
+            "total_tokens": sum(int(r.get("total_tokens", 0)) for r in records),
+            "total_cost": sum(float(r.get("cost", 0.0)) for r in records),
             "records": records,
             "recent": recent,
         }

packages/tracker/meshnet_tracker/cli.py

@@ -1,336 +1,445 @@
-"""meshnet-tracker CLI entry point."""
-
-import argparse
-import os
-import sys
-import time
-from pathlib import Path
-
-from .accounts import DEFAULT_ACCOUNTS_DB_PATH
-from .billing import DEFAULT_BILLING_DB_PATH
-from .hf_pricing import DEFAULT_HF_PRICING_LOG_DB_PATH
-from .server import (
-    DEFAULT_CALLER_CREDIT_USDT,
-    DEFAULT_DEVNET_TOPUP_USDT,
-    TrackerServer,
-    derive_relay_url_from_public_tracker_url,
-)
-
-DEFAULT_REGISTRY_DB_PATH = "meshnet_registry.sqlite3"
-
-
-def _load_env_file(path: Path) -> None:
-    """Load simple KEY=VALUE pairs from an env file without overriding env vars."""
-    if not path.exists():
-        return
-    try:
-        lines = path.read_text().splitlines()
-    except OSError:
-        return
-    for line in lines:
-        text = line.strip()
-        if not text or text.startswith("#"):
-            continue
-        if text.startswith("export "):
-            text = text[len("export "):].strip()
-        if "=" not in text:
-            continue
-        key, value = text.split("=", 1)
-        key = key.strip()
-        if not key or key in os.environ:
-            continue
-        value = value.strip()
-        if len(value) >= 2 and value[0] == value[-1] and value[0] in {"'", '"'}:
-            value = value[1:-1]
-        os.environ[key] = value
-
-
-def _load_env_defaults() -> None:
-    """Load local and user-level tracker env defaults before parsing arguments."""
-    _load_env_file(Path.cwd() / ".env")
-    _load_env_file(Path.home() / ".config" / "meshnet" / "secrets.env")
-
-
-def main() -> None:
-    _load_env_defaults()
-    common = argparse.ArgumentParser(add_help=False)
-    common.add_argument("--host", default="0.0.0.0", help="Host interface to listen on")
-    common.add_argument("--port", type=int, default=8080, help="Port to listen on")
-    common.add_argument(
-        "--heartbeat-timeout",
-        type=float,
-        default=30.0,
-        help="Seconds before a node is removed from the registry after missed heartbeat",
-    )
-    common.add_argument(
-        "--cluster-peers",
-        default="",
-        help="Comma-separated URLs of peer tracker nodes (enables Raft cluster mode)",
-    )
-    common.add_argument(
-        "--self-url",
-        default=None,
-        help="This tracker's own URL as seen by peers (auto-derived from --host/--port if omitted)",
-    )
-    common.add_argument(
-        "--stats-db",
-        default=None,
-        metavar="PATH",
-        help="SQLite database path for persistent model usage statistics",
-    )
-    common.add_argument(
-        "--relay-url",
-        default=None,
-        help="Public ws(s):// relay URL advertised to nodes, for example wss://ai.neuron.d-popov.com/ws",
-    )
-    common.add_argument(
-        "--billing-db",
-        default=DEFAULT_BILLING_DB_PATH,
-        metavar="PATH",
-        help=(
-            "SQLite database path for the USDT billing ledger "
-            f"(default: {DEFAULT_BILLING_DB_PATH}; ADR-0015)"
-        ),
-    )
-    common.add_argument(
-        "--no-billing",
-        action="store_true",
-        help="Disable the USDT billing ledger",
-    )
-    common.add_argument(
-        "--max-charge-per-request",
-        type=float,
-        default=None,
-        help=(
-            "Reject chat completion requests whose prompt plus requested completion "
-            "token bound would cost more than this many USDT"
-        ),
-    )
-    common.add_argument(
-        "--starting-credit",
-        type=float,
-        default=DEFAULT_CALLER_CREDIT_USDT,
-        metavar="USDT",
-        help=(
-            "One-time Caller Credit granted when an account creates its first "
-            f"API key (default: {DEFAULT_CALLER_CREDIT_USDT}; set 0 to require "
-            "deposits before inference)"
-        ),
-    )
-    common.add_argument(
-        "--devnet-topup",
-        type=float,
-        default=DEFAULT_DEVNET_TOPUP_USDT,
-        metavar="USDT",
-        help=(
-            "Dashboard devnet top-up faucet: each click credits this many USDT "
-            f"to one of the account's keys (default: {DEFAULT_DEVNET_TOPUP_USDT}; "
-            "MUST be 0 on mainnet deployments)"
-        ),
-    )
-    common.add_argument(
-        "--registry-db",
-        default=DEFAULT_REGISTRY_DB_PATH,
-        metavar="PATH",
-        help=(
-            "SQLite database path for persisted strike/ban/reputation registry "
-            f"state (default: {DEFAULT_REGISTRY_DB_PATH})"
-        ),
-    )
-    common.add_argument(
-        "--no-registry-contracts",
-        action="store_true",
-        help="Disable the local contract registry used for strike/ban/reputation enforcement",
-    )
-    common.add_argument(
-        "--accounts-db",
-        default=DEFAULT_ACCOUNTS_DB_PATH,
-        metavar="PATH",
-        help=(
-            "SQLite database path for dashboard user accounts "
-            f"(default: {DEFAULT_ACCOUNTS_DB_PATH})"
-        ),
-    )
-    common.add_argument(
-        "--no-accounts",
-        action="store_true",
-        help="Disable dashboard user accounts (registration/login)",
-    )
-    common.add_argument(
-        "--solana-rpc-url",
-        default=None,
-        help="Solana RPC URL (e.g. https://api.devnet.solana.com); enables the on-chain treasury",
-    )
-    common.add_argument(
-        "--usdt-mint",
-        default=None,
-        help="SPL mint address of (mock) USDT — see scripts/devnet_setup.py",
-    )
-    common.add_argument(
-        "--treasury-keypair",
-        default=None,
-        metavar="PATH",
-        help="Treasury keypair JSON path (only on settlement-capable trackers)",
-    )
-    common.add_argument(
-        "--settle-period",
-        type=float,
-        default=86400.0,
-        help="Max seconds between payouts to a node (dev: 60, prod: 86400)",
-    )
-    common.add_argument(
-        "--payout-threshold",
-        type=float,
-        default=5.0,
-        help="Pending USDT that triggers an immediate payout (dev: 0)",
-    )
-    common.add_argument(
-        "--payout-dust-floor",
-        type=float,
-        default=0.01,
-        help="Never pay out less than this many USDT",
-    )
-    common.add_argument(
-        "--validator-service-token",
-        default=None,
-        help=(
-            "Service token the validator uses on POST /v1/billing/forfeit "
-            "(default: MESHNET_VALIDATOR_SERVICE_TOKEN env; ADR-0017)"
-        ),
-    )
-    common.add_argument(
-        "--hive-secret",
-        default=None,
-        help=(
-            "Shared secret authenticating gossip between tracker peers "
-            "(default: MESHNET_HIVE_SECRET env; required for multi-tracker replication)"
-        ),
-    )
-    common.add_argument(
-        "--toploc-calibration-db",
-        default=None,
-        metavar="PATH",
-        help=(
-            "SQLite path for the AH-021 honest-noise TOPLOC calibration corpus "
-            "(enables POST /v1/calibration/toploc/run + GET /v1/calibration/toploc/results)"
-        ),
-    )
-    common.add_argument(
-        "--toploc-reference-node-url",
-        default=None,
-        help="Reference node the calibration job teacher-forces claimed tokens against (see validator README)",
-    )
-    common.add_argument(
-        "--toploc-calibration-gate-min-hardware-profiles",
-        type=int,
-        default=1,
-        help=(
-            "Distinct (GPU model, dtype) profiles the corpus must cover before "
-            "gate_status.ready is true (alpha exception: fleet size is acceptable)"
-        ),
-    )
-    common.add_argument(
-        "--enable-hf-pricing",
-        action="store_true",
-        help=(
-            "Enable the daily dynamic pricing refresh (issue 23): for presets with a "
-            "curated hf_aliases list, sets the client price to 80%% of the cheapest "
-            "matching HuggingFace inference-marketplace rate. Presets without "
-            "hf_aliases are unaffected and keep their static price."
-        ),
-    )
-    common.add_argument(
-        "--hf-pricing-log-db",
-        default=None,
-        metavar="PATH",
-        help=(
-            "SQLite database path for the dynamic pricing change log "
-            f"(default when --enable-hf-pricing is set: {DEFAULT_HF_PRICING_LOG_DB_PATH}; "
-            "enables GET /v1/pricing/hf/history)"
-        ),
-    )
-    common.add_argument(
-        "--hf-pricing-refresh-interval",
-        type=float,
-        default=86400.0,
-        help="Seconds between dynamic pricing refresh passes (default: daily)",
-    )
-    common.add_argument(
-        "--models-dir",
-        default=None,
-        metavar="PATH",
-        help="Local HuggingFace snapshot root advertised as tracker model-file source (default: MESHNET_MODELS_DIR)",
-    )
-
-    parser = argparse.ArgumentParser(
-        prog="meshnet-tracker",
-        description="Distributed Inference Network node registry and route selection",
-        parents=[common],
-    )
-    subparsers = parser.add_subparsers(dest="command")
-
-    subparsers.add_parser("start", help="Start the tracker server", parents=[common])
-
-    args = parser.parse_args()
-
-    if args.command in {None, "start"}:
-        cluster_peers = [u.strip() for u in args.cluster_peers.split(",") if u.strip()]
-        relay_url = args.relay_url or derive_relay_url_from_public_tracker_url(args.self_url)
-        treasury = None
-        if args.solana_rpc_url and args.usdt_mint and args.treasury_keypair:
-            from meshnet_contracts.solana_adapter import SolanaCustodialTreasury
-
-            treasury = SolanaCustodialTreasury(
-                args.solana_rpc_url, args.usdt_mint, args.treasury_keypair,
-            )
-        contracts = None
-        if not args.no_registry_contracts:
-            from meshnet_contracts import LocalSolanaContracts  # type: ignore[import-not-found]
-
-            contracts = LocalSolanaContracts(registry_db=args.registry_db)
-        server = TrackerServer(
-            host=args.host,
-            port=args.port,
-            heartbeat_timeout=args.heartbeat_timeout,
-            cluster_peers=cluster_peers or None,
-            cluster_self_url=args.self_url,
-            stats_db=getattr(args, "stats_db", None),
-            relay_url=relay_url,
-            enable_billing=not args.no_billing,
-            billing_db=None if args.no_billing else args.billing_db,
-            max_charge_per_request=args.max_charge_per_request,
-            starting_credit=args.starting_credit,
-            devnet_topup_amount=args.devnet_topup,
-            contracts=contracts,
-            accounts_db=None if args.no_accounts else args.accounts_db,
-            treasury=treasury,
-            settle_period=args.settle_period,
-            payout_threshold=args.payout_threshold,
-            payout_dust_floor=args.payout_dust_floor,
-            validator_service_token=args.validator_service_token,
-            hive_secret=args.hive_secret,
-            toploc_calibration_db=args.toploc_calibration_db,
-            toploc_reference_node_url=args.toploc_reference_node_url,
-            toploc_calibration_gate_min_hardware_profiles=args.toploc_calibration_gate_min_hardware_profiles,
-            enable_hf_pricing=args.enable_hf_pricing,
-            hf_pricing_log_db=(
-                args.hf_pricing_log_db
-                or (DEFAULT_HF_PRICING_LOG_DB_PATH if args.enable_hf_pricing else None)
-            ),
-            hf_pricing_refresh_interval=args.hf_pricing_refresh_interval,
-            models_dir=args.models_dir,
-        )
-        port = server.start()
-        print(f"meshnet-tracker listening on http://{args.host}:{port}", flush=True)
-        try:
-            while True:
-                time.sleep(1)
-        except KeyboardInterrupt:
-            server.stop()
-            sys.exit(0)
-    else:
-        parser.print_help()
-
-
-if __name__ == "__main__":
-    main()
+"""meshnet-tracker CLI entry point."""
+
+import argparse
+import os
+import sys
+import time
+from pathlib import Path
+
+from .accounts import DEFAULT_ACCOUNTS_DB_PATH
+from .billing import DEFAULT_BILLING_DB_PATH
+from .hf_pricing import DEFAULT_HF_PRICING_LOG_DB_PATH
+from .logging_setup import (
+    DEFAULT_LOG_BACKUP_COUNT,
+    DEFAULT_LOG_DIR,
+    DEFAULT_LOG_MAX_BYTES,
+    configure_tracker_file_logging,
+)
+from .routing_stats import RoutingConfig
+from .server import (
+    DEFAULT_CALLER_CREDIT_USDT,
+    DEFAULT_DEVNET_TOPUP_USDT,
+    TrackerServer,
+    derive_relay_url_from_public_tracker_url,
+)
+
+DEFAULT_REGISTRY_DB_PATH = "meshnet_registry.sqlite3"
+
+
+def _load_env_file(path: Path) -> None:
+    """Load simple KEY=VALUE pairs from an env file without overriding env vars."""
+    if not path.exists():
+        return
+    try:
+        lines = path.read_text().splitlines()
+    except OSError:
+        return
+    for line in lines:
+        text = line.strip()
+        if not text or text.startswith("#"):
+            continue
+        if text.startswith("export "):
+            text = text[len("export "):].strip()
+        if "=" not in text:
+            continue
+        key, value = text.split("=", 1)
+        key = key.strip()
+        if not key or key in os.environ:
+            continue
+        value = value.strip()
+        if len(value) >= 2 and value[0] == value[-1] and value[0] in {"'", '"'}:
+            value = value[1:-1]
+        os.environ[key] = value
+
+
+def _load_env_defaults() -> None:
+    """Load local and user-level tracker env defaults before parsing arguments."""
+    _load_env_file(Path.cwd() / ".env")
+    _load_env_file(Path.home() / ".config" / "meshnet" / "secrets.env")
+
+
+def _routing_config_from_args(args: argparse.Namespace) -> RoutingConfig | None:
+    """Build a RoutingConfig from CLI flags; None keeps env-var/server defaults."""
+    overrides = {
+        "explore_share": args.route_explore_share,
+        "weight_alpha": args.route_weight_alpha,
+        "stats_half_life_seconds": args.route_stats_half_life,
+    }
+    set_values = {key: value for key, value in overrides.items() if value is not None}
+    if not set_values:
+        return None
+    return RoutingConfig(**set_values)
+
+
+def main() -> None:
+    _load_env_defaults()
+    common = argparse.ArgumentParser(add_help=False)
+    common.add_argument("--host", default="0.0.0.0", help="Host interface to listen on")
+    common.add_argument("--port", type=int, default=8080, help="Port to listen on")
+    common.add_argument(
+        "--heartbeat-timeout",
+        type=float,
+        default=30.0,
+        help="Seconds before a node is removed from the registry after missed heartbeat",
+    )
+    common.add_argument(
+        "--cluster-peers",
+        default="",
+        help="Comma-separated URLs of peer tracker nodes (enables Raft cluster mode)",
+    )
+    common.add_argument(
+        "--self-url",
+        default=None,
+        help="This tracker's own URL as seen by peers (auto-derived from --host/--port if omitted)",
+    )
+    common.add_argument(
+        "--stats-db",
+        default=None,
+        metavar="PATH",
+        help="SQLite database path for persistent model usage statistics",
+    )
+    common.add_argument(
+        "--relay-url",
+        default=None,
+        help="Public ws(s):// relay URL advertised to nodes, for example wss://ai.neuron.d-popov.com/ws",
+    )
+    common.add_argument(
+        "--embedded-relay",
+        action="store_true",
+        help="Run the relay WebSocket server in this tracker process (still uses meshnet_relay.RelayServer)",
+    )
+    common.add_argument(
+        "--relay-host",
+        default="127.0.0.1",
+        help="Bind address for --embedded-relay (default: 127.0.0.1; use 0.0.0.0 only when exposing the relay port directly)",
+    )
+    common.add_argument(
+        "--relay-port",
+        type=int,
+        default=8765,
+        help="Bind port for --embedded-relay (default: 8765)",
+    )
+    common.add_argument(
+        "--relay-max-peers",
+        type=int,
+        default=500,
+        help="Maximum WebSocket peers accepted by --embedded-relay",
+    )
+    common.add_argument(
+        "--billing-db",
+        default=DEFAULT_BILLING_DB_PATH,
+        metavar="PATH",
+        help=(
+            "SQLite database path for the USDT billing ledger "
+            f"(default: {DEFAULT_BILLING_DB_PATH}; ADR-0015)"
+        ),
+    )
+    common.add_argument(
+        "--no-billing",
+        action="store_true",
+        help="Disable the USDT billing ledger",
+    )
+    common.add_argument(
+        "--max-charge-per-request",
+        type=float,
+        default=None,
+        help=(
+            "Reject chat completion requests whose prompt plus requested completion "
+            "token bound would cost more than this many USDT"
+        ),
+    )
+    common.add_argument(
+        "--starting-credit",
+        type=float,
+        default=DEFAULT_CALLER_CREDIT_USDT,
+        metavar="USDT",
+        help=(
+            "One-time Caller Credit granted when an account creates its first "
+            f"API key (default: {DEFAULT_CALLER_CREDIT_USDT}; set 0 to require "
+            "deposits before inference)"
+        ),
+    )
+    common.add_argument(
+        "--devnet-topup",
+        type=float,
+        default=DEFAULT_DEVNET_TOPUP_USDT,
+        metavar="USDT",
+        help=(
+            "Dashboard devnet top-up faucet: each click credits this many USDT "
+            f"to one of the account's keys (default: {DEFAULT_DEVNET_TOPUP_USDT}; "
+            "MUST be 0 on mainnet deployments)"
+        ),
+    )
+    common.add_argument(
+        "--registry-db",
+        default=DEFAULT_REGISTRY_DB_PATH,
+        metavar="PATH",
+        help=(
+            "SQLite database path for persisted strike/ban/reputation registry "
+            f"state (default: {DEFAULT_REGISTRY_DB_PATH})"
+        ),
+    )
+    common.add_argument(
+        "--no-registry-contracts",
+        action="store_true",
+        help="Disable the local contract registry used for strike/ban/reputation enforcement",
+    )
+    common.add_argument(
+        "--accounts-db",
+        default=DEFAULT_ACCOUNTS_DB_PATH,
+        metavar="PATH",
+        help=(
+            "SQLite database path for dashboard user accounts "
+            f"(default: {DEFAULT_ACCOUNTS_DB_PATH})"
+        ),
+    )
+    common.add_argument(
+        "--no-accounts",
+        action="store_true",
+        help="Disable dashboard user accounts (registration/login)",
+    )
+    common.add_argument(
+        "--solana-rpc-url",
+        default=None,
+        help="Solana RPC URL (e.g. https://api.devnet.solana.com); enables the on-chain treasury",
+    )
+    common.add_argument(
+        "--usdt-mint",
+        default=None,
+        help="SPL mint address of (mock) USDT — see scripts/devnet_setup.py",
+    )
+    common.add_argument(
+        "--treasury-keypair",
+        default=None,
+        metavar="PATH",
+        help="Treasury keypair JSON path (only on settlement-capable trackers)",
+    )
+    common.add_argument(
+        "--settle-period",
+        type=float,
+        default=86400.0,
+        help="Max seconds between payouts to a node (dev: 60, prod: 86400)",
+    )
+    common.add_argument(
+        "--payout-threshold",
+        type=float,
+        default=5.0,
+        help="Pending USDT that triggers an immediate payout (dev: 0)",
+    )
+    common.add_argument(
+        "--payout-dust-floor",
+        type=float,
+        default=0.01,
+        help="Never pay out less than this many USDT",
+    )
+    common.add_argument(
+        "--validator-service-token",
+        default=None,
+        help=(
+            "Service token the validator uses on POST /v1/billing/forfeit "
+            "(default: MESHNET_VALIDATOR_SERVICE_TOKEN env; ADR-0017)"
+        ),
+    )
+    common.add_argument(
+        "--hive-secret",
+        default=None,
+        help=(
+            "Shared secret authenticating gossip between tracker peers "
+            "(default: MESHNET_HIVE_SECRET env; required for multi-tracker replication)"
+        ),
+    )
+    common.add_argument(
+        "--toploc-calibration-db",
+        default=None,
+        metavar="PATH",
+        help=(
+            "SQLite path for the AH-021 honest-noise TOPLOC calibration corpus "
+            "(enables POST /v1/calibration/toploc/run + GET /v1/calibration/toploc/results)"
+        ),
+    )
+    common.add_argument(
+        "--toploc-reference-node-url",
+        default=None,
+        help="Reference node the calibration job teacher-forces claimed tokens against (see validator README)",
+    )
+    common.add_argument(
+        "--toploc-calibration-gate-min-hardware-profiles",
+        type=int,
+        default=1,
+        help=(
+            "Distinct (GPU model, dtype) profiles the corpus must cover before "
+            "gate_status.ready is true (alpha exception: fleet size is acceptable)"
+        ),
+    )
+    common.add_argument(
+        "--enable-hf-pricing",
+        action="store_true",
+        help=(
+            "Enable the daily dynamic pricing refresh (issue 23): for presets with a "
+            "curated hf_aliases list, sets the client price to 80%% of the cheapest "
+            "matching HuggingFace inference-marketplace rate. Presets without "
+            "hf_aliases are unaffected and keep their static price."
+        ),
+    )
+    common.add_argument(
+        "--hf-pricing-log-db",
+        default=None,
+        metavar="PATH",
+        help=(
+            "SQLite database path for the dynamic pricing change log "
+            f"(default when --enable-hf-pricing is set: {DEFAULT_HF_PRICING_LOG_DB_PATH}; "
+            "enables GET /v1/pricing/hf/history)"
+        ),
+    )
+    common.add_argument(
+        "--hf-pricing-refresh-interval",
+        type=float,
+        default=86400.0,
+        help="Seconds between dynamic pricing refresh passes (default: daily)",
+    )
+    common.add_argument(
+        "--models-dir",
+        default=None,
+        metavar="PATH",
+        help="Local HuggingFace snapshot root advertised as tracker model-file source (default: MESHNET_MODELS_DIR)",
+    )
+    common.add_argument(
+        "--route-explore-share",
+        type=float,
+        default=None,
+        metavar="FRACTION",
+        help=(
+            "Fraction of requests routed down unproven/stale routes to gather "
+            "throughput statistics (ADR-0021; default 0.3, lower once traffic grows)"
+        ),
+    )
+    common.add_argument(
+        "--route-weight-alpha",
+        type=float,
+        default=None,
+        metavar="ALPHA",
+        help=(
+            "Traffic weight exponent among proven routes: share ∝ tps^alpha "
+            "(default 1.0 — a 1.5x-faster route gets 1.5x the traffic)"
+        ),
+    )
+    common.add_argument(
+        "--route-stats-half-life",
+        type=float,
+        default=None,
+        metavar="SECONDS",
+        help="Half-life for decaying route throughput observations (default 600)",
+    )
+    common.add_argument(
+        "--log-dir",
+        default=DEFAULT_LOG_DIR,
+        metavar="PATH",
+        help=(
+            "Directory for rotating tracker logs "
+            f"(default: {DEFAULT_LOG_DIR}; files: info.log, warning.log, error.log)"
+        ),
+    )
+    common.add_argument(
+        "--log-max-bytes",
+        type=int,
+        default=DEFAULT_LOG_MAX_BYTES,
+        metavar="BYTES",
+        help=f"Rotate each tracker log file after this many bytes (default: {DEFAULT_LOG_MAX_BYTES})",
+    )
+    common.add_argument(
+        "--log-backup-count",
+        type=int,
+        default=DEFAULT_LOG_BACKUP_COUNT,
+        metavar="N",
+        help=f"Number of rotated tracker log files to keep per level (default: {DEFAULT_LOG_BACKUP_COUNT})",
+    )
+    common.add_argument(
+        "--no-file-logs",
+        action="store_true",
+        help="Disable rotating tracker log files and only write to the terminal",
+    )
+
+    parser = argparse.ArgumentParser(
+        prog="meshnet-tracker",
+        description="Distributed Inference Network node registry and route selection",
+        parents=[common],
+    )
+    subparsers = parser.add_subparsers(dest="command")
+
+    subparsers.add_parser("start", help="Start the tracker server", parents=[common])
+
+    args = parser.parse_args()
+
+    if args.command in {None, "start"}:
+        if not args.no_file_logs:
+            log_dir = configure_tracker_file_logging(
+                args.log_dir,
+                max_bytes=args.log_max_bytes,
+                backup_count=args.log_backup_count,
+            )
+            print(f"meshnet-tracker logs: {log_dir}", flush=True)
+        cluster_peers = [u.strip() for u in args.cluster_peers.split(",") if u.strip()]
+        relay_url = args.relay_url or derive_relay_url_from_public_tracker_url(args.self_url)
+        treasury = None
+        if args.solana_rpc_url and args.usdt_mint and args.treasury_keypair:
+            from meshnet_contracts.solana_adapter import SolanaCustodialTreasury
+
+            treasury = SolanaCustodialTreasury(
+                args.solana_rpc_url, args.usdt_mint, args.treasury_keypair,
+            )
+        contracts = None
+        if not args.no_registry_contracts:
+            from meshnet_contracts import LocalSolanaContracts  # type: ignore[import-not-found]
+
+            contracts = LocalSolanaContracts(registry_db=args.registry_db)
+        server = TrackerServer(
+            host=args.host,
+            port=args.port,
+            heartbeat_timeout=args.heartbeat_timeout,
+            cluster_peers=cluster_peers or None,
+            cluster_self_url=args.self_url,
+            stats_db=getattr(args, "stats_db", None),
+            relay_url=relay_url,
+            embedded_relay=args.embedded_relay,
+            embedded_relay_host=args.relay_host,
+            embedded_relay_port=args.relay_port,
+            embedded_relay_max_peers=args.relay_max_peers,
+            enable_billing=not args.no_billing,
+            billing_db=None if args.no_billing else args.billing_db,
+            max_charge_per_request=args.max_charge_per_request,
+            starting_credit=args.starting_credit,
+            devnet_topup_amount=args.devnet_topup,
+            contracts=contracts,
+            accounts_db=None if args.no_accounts else args.accounts_db,
+            treasury=treasury,
+            settle_period=args.settle_period,
+            payout_threshold=args.payout_threshold,
+            payout_dust_floor=args.payout_dust_floor,
+            validator_service_token=args.validator_service_token,
+            hive_secret=args.hive_secret,
+            toploc_calibration_db=args.toploc_calibration_db,
+            toploc_reference_node_url=args.toploc_reference_node_url,
+            toploc_calibration_gate_min_hardware_profiles=args.toploc_calibration_gate_min_hardware_profiles,
+            enable_hf_pricing=args.enable_hf_pricing,
+            hf_pricing_log_db=(
+                args.hf_pricing_log_db
+                or (DEFAULT_HF_PRICING_LOG_DB_PATH if args.enable_hf_pricing else None)
+            ),
+            hf_pricing_refresh_interval=args.hf_pricing_refresh_interval,
+            models_dir=args.models_dir,
+            routing_config=_routing_config_from_args(args),
+        )
+        port = server.start()
+        print(f"meshnet-tracker listening on http://{args.host}:{port}", flush=True)
+        try:
+            while True:
+                time.sleep(1)
+        except KeyboardInterrupt:
+            server.stop()
+            sys.exit(0)
+    else:
+        parser.print_help()
+
+
+if __name__ == "__main__":
+    main()

packages/tracker/meshnet_tracker/favicon.svg

@@ -0,0 +1,18 @@
+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1254 1254" width="1254" height="1254">
+  <defs>
+    <radialGradient id="bg" cx="50%" cy="50%" r="70%">
+      <stop offset="0%" stop-color="#071229"/>
+      <stop offset="100%" stop-color="#000615"/>
+    </radialGradient>
+    <linearGradient id="fg" x1="145" y1="180" x2="1110" y2="1080" gradientUnits="userSpaceOnUse">
+      <stop offset="0%" stop-color="#1cc8ff"/>
+      <stop offset="50%" stop-color="#2d80ff"/>
+      <stop offset="100%" stop-color="#b548f2"/>
+    </linearGradient>
+    <filter id="softShadow" x="-10%" y="-10%" width="120%" height="120%">
+      <feDropShadow dx="0" dy="8" stdDeviation="10" flood-color="#000" flood-opacity="0.25"/>
+    </filter>
+  </defs>
+  <rect x="0" y="0" width="1254" height="1254" rx="285" ry="285" fill="url(#bg)"/>
+  <path d="M 710.0 843.0 L 663.0 857.0 L 640.0 870.0 L 629.0 881.0 L 620.0 896.0 L 615.0 918.0 L 615.0 1008.0 L 601.0 1015.0 L 593.0 1023.0 L 585.0 1041.0 L 585.0 1056.0 L 589.0 1068.0 L 601.0 1082.0 L 612.0 1088.0 L 628.0 1090.0 L 638.0 1088.0 L 654.0 1078.0 L 660.0 1070.0 L 666.0 1052.0 L 664.0 1036.0 L 658.0 1024.0 L 648.0 1014.0 L 635.0 1008.0 L 635.0 920.0 L 639.0 904.0 L 649.0 888.0 L 666.0 874.0 L 678.0 868.0 L 717.0 856.0 L 717.0 853.0 Z M 409.0 833.0 L 353.0 891.0 L 337.0 886.0 L 321.0 886.0 L 310.0 890.0 L 296.0 902.0 L 291.0 910.0 L 288.0 920.0 L 288.0 936.0 L 294.0 951.0 L 305.0 963.0 L 321.0 970.0 L 337.0 970.0 L 352.0 964.0 L 361.0 956.0 L 368.0 944.0 L 371.0 932.0 L 370.0 917.0 L 366.0 906.0 L 417.0 852.0 Z M 843.0 831.0 L 835.0 850.0 L 892.0 905.0 L 887.0 920.0 L 887.0 934.0 L 894.0 951.0 L 904.0 961.0 L 919.0 967.0 L 937.0 966.0 L 952.0 958.0 L 961.0 947.0 L 966.0 931.0 L 964.0 914.0 L 957.0 901.0 L 945.0 891.0 L 934.0 887.0 L 919.0 887.0 L 905.0 891.0 Z M 516.0 474.0 L 487.0 500.0 L 472.0 520.0 L 456.0 551.0 L 446.0 587.0 L 443.0 612.0 L 443.0 813.0 L 446.0 824.0 L 451.0 833.0 L 459.0 841.0 L 466.0 845.0 L 477.0 848.0 L 491.0 847.0 L 502.0 842.0 L 511.0 834.0 L 519.0 818.0 L 520.0 615.0 L 525.0 592.0 L 535.0 570.0 L 555.0 546.0 L 573.0 533.0 L 588.0 526.0 L 605.0 521.0 L 637.0 519.0 L 652.0 521.0 L 669.0 526.0 L 690.0 537.0 L 712.0 557.0 L 726.0 580.0 L 733.0 606.0 L 734.0 818.0 L 741.0 833.0 L 749.0 841.0 L 756.0 845.0 L 776.0 848.0 L 784.0 846.0 L 796.0 839.0 L 806.0 826.0 L 810.0 810.0 L 809.0 597.0 L 806.0 578.0 L 797.0 549.0 L 785.0 525.0 L 771.0 505.0 L 746.0 480.0 L 719.0 462.0 L 684.0 448.0 L 655.0 442.0 L 609.0 441.0 L 583.0 445.0 L 548.0 456.0 Z M 920.0 261.0 L 902.0 263.0 L 891.0 269.0 L 882.0 278.0 L 874.0 296.0 L 874.0 312.0 L 881.0 331.0 L 803.0 419.0 L 793.0 437.0 L 789.0 455.0 L 790.0 469.0 L 815.0 496.0 L 835.0 524.0 L 850.0 552.0 L 858.0 574.0 L 863.0 599.0 L 864.0 624.0 L 860.0 656.0 L 852.0 690.0 L 870.0 654.0 L 877.0 644.0 L 888.0 634.0 L 904.0 626.0 L 914.0 624.0 L 962.0 623.0 L 1019.0 624.0 L 1028.0 641.0 L 1041.0 652.0 L 1055.0 657.0 L 1070.0 657.0 L 1084.0 652.0 L 1097.0 641.0 L 1104.0 629.0 L 1107.0 617.0 L 1106.0 601.0 L 1097.0 583.0 L 1084.0 572.0 L 1071.0 567.0 L 1055.0 567.0 L 1038.0 574.0 L 1026.0 586.0 L 1019.0 603.0 L 916.0 603.0 L 899.0 599.0 L 883.0 589.0 L 873.0 578.0 L 844.0 517.0 L 820.0 482.0 L 813.0 464.0 L 813.0 451.0 L 816.0 440.0 L 829.0 419.0 L 895.0 344.0 L 908.0 347.0 L 922.0 347.0 L 932.0 344.0 L 940.0 339.0 L 949.0 330.0 L 955.0 319.0 L 957.0 312.0 L 957.0 296.0 L 951.0 281.0 L 937.0 267.0 Z M 658.0 154.0 L 647.0 145.0 L 637.0 141.0 L 617.0 141.0 L 601.0 149.0 L 594.0 156.0 L 587.0 170.0 L 585.0 181.0 L 586.0 191.0 L 592.0 206.0 L 602.0 217.0 L 616.0 224.0 L 616.0 326.0 L 608.0 349.0 L 596.0 364.0 L 575.0 379.0 L 528.0 396.0 L 501.0 410.0 L 479.0 426.0 L 476.0 426.0 L 449.0 406.0 L 375.0 329.0 L 381.0 312.0 L 381.0 295.0 L 375.0 280.0 L 364.0 268.0 L 358.0 264.0 L 346.0 260.0 L 331.0 260.0 L 324.0 262.0 L 311.0 270.0 L 302.0 281.0 L 296.0 299.0 L 297.0 314.0 L 303.0 328.0 L 312.0 338.0 L 331.0 347.0 L 346.0 347.0 L 360.0 343.0 L 417.0 401.0 L 434.0 421.0 L 441.0 436.0 L 441.0 453.0 L 438.0 462.0 L 416.0 496.0 L 402.0 523.0 L 384.0 574.0 L 380.0 580.0 L 366.0 593.0 L 351.0 600.0 L 336.0 603.0 L 234.0 603.0 L 225.0 584.0 L 213.0 573.0 L 198.0 567.0 L 182.0 567.0 L 175.0 569.0 L 160.0 578.0 L 148.0 596.0 L 145.0 616.0 L 147.0 626.0 L 155.0 641.0 L 172.0 654.0 L 182.0 657.0 L 197.0 657.0 L 205.0 655.0 L 216.0 649.0 L 225.0 640.0 L 233.0 624.0 L 346.0 625.0 L 366.0 632.0 L 379.0 644.0 L 384.0 652.0 L 389.0 669.0 L 400.0 692.0 L 398.0 616.0 L 404.0 574.0 L 416.0 537.0 L 426.0 516.0 L 440.0 493.0 L 459.0 469.0 L 492.0 439.0 L 533.0 415.0 L 553.0 407.0 L 578.0 400.0 L 598.0 390.0 L 617.0 374.0 L 630.0 354.0 L 635.0 339.0 L 637.0 324.0 L 637.0 224.0 L 650.0 218.0 L 660.0 208.0 L 668.0 190.0 L 667.0 171.0 Z" fill="url(#fg)" fill-rule="evenodd" filter="url(#softShadow)"/>
+</svg>

packages/tracker/meshnet_tracker/logging_setup.py

@@ -0,0 +1,99 @@
+"""Rotating file logging for the tracker CLI."""
+
+from __future__ import annotations
+
+import logging
+import sys
+from logging.handlers import RotatingFileHandler
+from pathlib import Path
+from typing import TextIO
+
+
+DEFAULT_LOG_DIR = "logs/tracker"
+DEFAULT_LOG_MAX_BYTES = 10 * 1024 * 1024
+DEFAULT_LOG_BACKUP_COUNT = 5
+TRACKER_LOGGER_NAME = "meshnet.tracker"
+
+
+class _ExactLevelFilter(logging.Filter):
+    def __init__(self, level: int) -> None:
+        super().__init__()
+        self._level = level
+
+    def filter(self, record: logging.LogRecord) -> bool:
+        return record.levelno == self._level
+
+
+class _TeeStream:
+    def __init__(self, stream: TextIO, logger: logging.Logger, level: int) -> None:
+        self._stream = stream
+        self._logger = logger
+        self._level = level
+        self._buffer = ""
+
+    def write(self, text: str) -> int:
+        self._stream.write(text)
+        self._stream.flush()
+        self._buffer += text
+        while "\n" in self._buffer:
+            line, self._buffer = self._buffer.split("\n", 1)
+            line = line.rstrip()
+            if line:
+                self._logger.log(self._level, line)
+        return len(text)
+
+    def flush(self) -> None:
+        self._stream.flush()
+        line = self._buffer.rstrip()
+        if line:
+            self._logger.log(self._level, line)
+        self._buffer = ""
+
+    def isatty(self) -> bool:
+        return self._stream.isatty()
+
+
+def _make_handler(path: Path, level: int, *, max_bytes: int, backup_count: int) -> RotatingFileHandler:
+    handler = RotatingFileHandler(
+        path,
+        maxBytes=max_bytes,
+        backupCount=backup_count,
+        encoding="utf-8",
+    )
+    handler.setLevel(level)
+    handler.addFilter(_ExactLevelFilter(level))
+    handler.setFormatter(logging.Formatter("%(asctime)s %(levelname)s %(message)s"))
+    return handler
+
+
+def configure_tracker_file_logging(
+    log_dir: str | Path = DEFAULT_LOG_DIR,
+    *,
+    max_bytes: int = DEFAULT_LOG_MAX_BYTES,
+    backup_count: int = DEFAULT_LOG_BACKUP_COUNT,
+    tee_stdio: bool = True,
+) -> Path:
+    """Configure rotatable info/warning/error log files and return the directory."""
+
+    path = Path(log_dir).expanduser()
+    path.mkdir(parents=True, exist_ok=True)
+
+    logger = logging.getLogger(TRACKER_LOGGER_NAME)
+    logger.setLevel(logging.INFO)
+    logger.propagate = False
+    logger.handlers.clear()
+    logger.addHandler(_make_handler(path / "info.log", logging.INFO, max_bytes=max_bytes, backup_count=backup_count))
+    logger.addHandler(_make_handler(path / "warning.log", logging.WARNING, max_bytes=max_bytes, backup_count=backup_count))
+    logger.addHandler(_make_handler(path / "error.log", logging.ERROR, max_bytes=max_bytes, backup_count=backup_count))
+
+    if tee_stdio:
+        if not isinstance(sys.stdout, _TeeStream):
+            sys.stdout = _TeeStream(sys.stdout, logger, logging.INFO)  # type: ignore[assignment]
+        if not isinstance(sys.stderr, _TeeStream):
+            sys.stderr = _TeeStream(sys.stderr, logger, logging.ERROR)  # type: ignore[assignment]
+
+    return path
+
+
+def tracker_logger() -> logging.Logger:
+    return logging.getLogger(TRACKER_LOGGER_NAME)

packages/tracker/meshnet_tracker/model_presets.json

@@ -39,6 +39,38 @@
         ]
       }
     },
+    "qwen2.5-0.5b-instruct": {
+      "layers_start": 0,
+      "layers_end": 23,
+      "hf_repo": "Qwen/Qwen2.5-0.5B-Instruct",
+      "aliases": [
+        "qwen2.5-0.5b",
+        "Qwen2.5-0.5B-Instruct",
+        "Qwen/Qwen2.5-0.5B-Instruct"
+      ],
+      "deployment_status": "supported",
+      "price_per_1k_tokens": 0.002,
+      "input_price_per_1k_tokens": 0.002,
+      "output_price_per_1k_tokens": 0.002,
+      "hf_aliases": [],
+      "hf_verified_match_note": "Static 10× dev-funding markup over ~$0.20/1M commercial API reference (Qwen-class hosted rates). $0.002/1k = $2/1M blended input+output.",
+      "required_model_bytes": 1056964608,
+      "download_size_bytes": 1056964608,
+      "native_quantization": "bfloat16",
+      "canonical_audit_dtype": "bfloat16",
+      "canonical_audit_quantization": "bfloat16",
+      "bytes_per_layer": {
+        "bfloat16": 44040192
+      },
+      "metadata": {
+        "architecture": "Dense transformer (GQA)",
+        "total_parameters": "0.5B",
+        "num_layers": 24,
+        "context_length": 32768,
+        "native_quantization": "bfloat16",
+        "download_size_gb": 1
+      }
+    },
     "qwen3.6-35b-a3b": {
       "layers_start": 0,
       "layers_end": 39,

packages/tracker/meshnet_tracker/routing_stats.py

@@ -0,0 +1,257 @@
+"""Learned route statistics for dynamic bandit-style route selection (ADR-0021).
+
+The tracker treats each viable route (ordered chain of node shards covering a
+model) as a bandit arm. Observed end-to-end tokens/sec per route is kept as a
+time-decayed EWMA. Selection splits traffic between:
+
+- **exploit**: weighted-random among *proven* routes, weight ∝ tps ** alpha
+  (alpha=1.0 → a 1.5x-faster route gets 1.5x the traffic);
+- **scout**: with probability `explore_share`, the least-measured unproven or
+  stale route is chosen so the tracker keeps learning as the network morphs.
+
+Staleness has two mechanisms:
+- continuous: sample mass decays with `stats_half_life_seconds`, so old
+  observations fade;
+- abrupt: every node join/leave bumps the model's *topology epoch*; stats from
+  an older epoch keep their EWMA as a prior but drop back into the scout pool
+  until re-measured.
+
+Route signatures embed node ids and shard ranges, so a node re-registering
+with a different shard produces a new arm automatically.
+"""
+
+from __future__ import annotations
+
+import random
+import threading
+import time
+from dataclasses import dataclass, field
+from typing import Any, Iterable
+
+
+@dataclass(frozen=True)
+class RoutingConfig:
+    explore_share: float = 0.3
+    weight_alpha: float = 1.0
+    stats_half_life_seconds: float = 600.0
+    min_sample_tokens: int = 8
+    # One fresh sample has mass 1.0 and decays from there; 0.5 keeps a single
+    # observation "proven" for one half-life before demoting it to the scout pool.
+    min_proven_weight: float = 0.5
+    max_candidate_routes: int = 8
+    prune_after_seconds: float = 86400.0
+
+
+@dataclass
+class RouteStat:
+    ewma_tps: float = 0.0
+    weight: float = 0.0  # decayed effective sample mass
+    last_sample_ts: float = 0.0
+    epoch: int = 0
+    samples: int = 0  # lifetime raw sample count (display only)
+
+    def decayed_weight(self, now: float, half_life: float) -> float:
+        if self.weight <= 0.0:
+            return 0.0
+        age = max(0.0, now - self.last_sample_ts)
+        return self.weight * 0.5 ** (age / half_life)
+
+
+@dataclass
+class RouteCandidate:
+    nodes: list[Any]
+    signature: str
+    prior_tps: float = 0.0
+
+
+def route_signature(model_key: str, nodes: Iterable[Any]) -> str:
+    hops = "->".join(
+        f"{getattr(n, 'node_id', '?')}[{getattr(n, 'shard_start', '?')}-{getattr(n, 'shard_end', '?')}]"
+        for n in nodes
+    )
+    return f"{model_key}|{hops}"
+
+
+class RouteStatsStore:
+    """Thread-safe per-route decayed throughput statistics."""
+
+    def __init__(self, config: RoutingConfig | None = None) -> None:
+        self.config = config or RoutingConfig()
+        self._lock = threading.Lock()
+        self._stats: dict[str, RouteStat] = {}
+        self._epochs: dict[str, int] = {}
+
+    def epoch(self, model_key: str) -> int:
+        with self._lock:
+            return self._epochs.get(model_key, 0)
+
+    def bump_epoch(self, model_keys: Iterable[str | None]) -> None:
+        """Mark the topology changed for the given model keys (node join/leave)."""
+        with self._lock:
+            for key in model_keys:
+                if key:
+                    self._epochs[key] = self._epochs.get(key, 0) + 1
+
+    def record_sample(
+        self,
+        model_key: str,
+        signature: str,
+        tokens: int,
+        elapsed_seconds: float,
+        now: float | None = None,
+    ) -> bool:
+        """Fold one completed request into the route's EWMA.
+
+        Returns False (and records nothing) for samples below
+        `min_sample_tokens` — near-empty completions come from broken routes
+        and would poison the arm with meaningless throughput values.
+        """
+        cfg = self.config
+        if tokens < cfg.min_sample_tokens or elapsed_seconds <= 0.0:
+            return False
+        tps = tokens / elapsed_seconds
+        ts = time.time() if now is None else now
+        with self._lock:
+            stat = self._stats.get(signature)
+            if stat is None:
+                stat = RouteStat()
+                self._stats[signature] = stat
+            carried = stat.decayed_weight(ts, cfg.stats_half_life_seconds)
+            total = carried + 1.0
+            stat.ewma_tps = (stat.ewma_tps * carried + tps) / total
+            stat.weight = total
+            stat.last_sample_ts = ts
+            stat.epoch = self._epochs.get(model_key, 0)
+            stat.samples += 1
+        return True
+
+    def snapshot(self, signature: str, model_key: str, now: float | None = None) -> dict:
+        """Point-in-time view of one route's learned state."""
+        ts = time.time() if now is None else now
+        cfg = self.config
+        with self._lock:
+            stat = self._stats.get(signature)
+            current_epoch = self._epochs.get(model_key, 0)
+        if stat is None:
+            return {"tps": None, "weight": 0.0, "samples": 0, "status": "unsampled"}
+        weight = stat.decayed_weight(ts, cfg.stats_half_life_seconds)
+        if stat.epoch != current_epoch:
+            status = "stale"
+        elif weight < cfg.min_proven_weight:
+            status = "decayed" if stat.samples else "unsampled"
+        else:
+            status = "proven"
+        return {
+            "tps": round(stat.ewma_tps, 4) if stat.samples else None,
+            "weight": round(weight, 4),
+            "samples": stat.samples,
+            "status": status,
+        }
+
+    def prune(self, now: float | None = None) -> int:
+        """Drop routes with no samples for `prune_after_seconds`."""
+        ts = time.time() if now is None else now
+        cutoff = ts - self.config.prune_after_seconds
+        with self._lock:
+            dead = [sig for sig, stat in self._stats.items() if stat.last_sample_ts < cutoff]
+            for sig in dead:
+                del self._stats[sig]
+        return len(dead)
+
+
+def choose_route(
+    candidates: list[RouteCandidate],
+    store: RouteStatsStore,
+    model_key: str,
+    rng: random.Random | None = None,
+    now: float | None = None,
+) -> tuple[RouteCandidate | None, dict]:
+    """Pick a route: ε-scout among unproven arms, else weighted ∝ tps**alpha.
+
+    Returns (candidate, decision) where decision explains the pick for logs
+    and diagnostics: {"mode": "scout"|"exploit"|"prior", ...}.
+    """
+    if not candidates:
+        return None, {"mode": "none"}
+    rng = rng or random
+    cfg = store.config
+    proven: list[tuple[RouteCandidate, float]] = []
+    scouts: list[tuple[RouteCandidate, float]] = []
+    for cand in candidates:
+        snap = store.snapshot(cand.signature, model_key, now=now)
+        if snap["status"] == "proven":
+            proven.append((cand, max(float(snap["tps"] or 0.0), 1e-6)))
+        else:
+            scouts.append((cand, float(snap["weight"])))
+    if scouts and (not proven or rng.random() < cfg.explore_share):
+        # Least-measured first so new/stale arms accumulate samples fastest;
+        # tiebreak on prior estimate so plausible routes get scouted first.
+        scouts.sort(key=lambda item: (item[1], -item[0].prior_tps))
+        pick = scouts[0][0]
+        return pick, {"mode": "scout", "signature": pick.signature}
+    if proven:
+        weights = [tps ** cfg.weight_alpha for _, tps in proven]
+        pick = rng.choices([cand for cand, _ in proven], weights=weights, k=1)[0]
+        return pick, {
+            "mode": "exploit",
+            "signature": pick.signature,
+            "candidates": len(proven),
+        }
+    # No stats anywhere yet — fall back to the prior (benchmark-derived) estimate.
+    weights = [max(cand.prior_tps, 1e-6) ** cfg.weight_alpha for cand in candidates]
+    pick = rng.choices(candidates, weights=weights, k=1)[0]
+    return pick, {"mode": "prior", "signature": pick.signature}
+
+
+def route_table(
+    candidates: list[RouteCandidate],
+    store: RouteStatsStore,
+    model_key: str,
+    now: float | None = None,
+) -> list[dict]:
+    """Diagnostics rows: learned tps, coefficient vs best, expected traffic share."""
+    cfg = store.config
+    rows = []
+    for cand in candidates:
+        snap = store.snapshot(cand.signature, model_key, now=now)
+        rows.append({"candidate": cand, **snap})
+    proven = [r for r in rows if r["status"] == "proven"]
+    scouts = [r for r in rows if r["status"] != "proven"]
+    best_tps = max((float(r["tps"]) for r in proven), default=0.0)
+    exploit_budget = 1.0 - (cfg.explore_share if scouts and proven else 0.0)
+    if not proven:
+        exploit_budget = 0.0
+    weight_sum = sum(float(r["tps"]) ** cfg.weight_alpha for r in proven) or 1.0
+    out = []
+    for r in rows:
+        cand: RouteCandidate = r["candidate"]
+        if r["status"] == "proven":
+            share = exploit_budget * (float(r["tps"]) ** cfg.weight_alpha) / weight_sum
+            coefficient = round(float(r["tps"]) / best_tps, 3) if best_tps else None
+        else:
+            share = (
+                (cfg.explore_share if proven else 1.0) / len(scouts)
+                if scouts
+                else 0.0
+            )
+            coefficient = None
+        out.append({
+            "signature": cand.signature,
+            "hops": [
+                {
+                    "node_id": getattr(n, "node_id", "?"),
+                    "shard": f"{getattr(n, 'shard_start', '?')}-{getattr(n, 'shard_end', '?')}",
+                    "endpoint": getattr(n, "endpoint", "?"),
+                }
+                for n in cand.nodes
+            ],
+            "tps": r["tps"],
+            "coefficient": coefficient,
+            "expected_share": round(share, 4),
+            "samples": r["samples"],
+            "weight": r["weight"],
+            "status": r["status"],
+            "prior_tps": round(cand.prior_tps, 4),
+        })
+    out.sort(key=lambda r: (-(r["tps"] or 0.0), -r["prior_tps"]))
+    return out

packages/tracker/pyproject.toml

@@ -21,4 +21,4 @@ where = ["."]
 include = ["meshnet_tracker*"]
 
 [tool.setuptools.package-data]
-meshnet_tracker = ["*.json", "*.html"]
+meshnet_tracker = ["*.json", "*.html", "*.svg"]

packages/tracker/uv.lock

@@ -0,0 +1,270 @@
+version = 1
+requires-python = ">=3.10"
+
+[[package]]
+name = "cffi"
+version = "2.1.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+    { name = "pycparser", marker = "implementation_name != 'PyPy'" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/57/5f/ff100cae70ebe9d8df1c01a00e510e45d9adb5c1fdda84791b199141de97/cffi-2.1.0.tar.gz", hash = "sha256:efc1cdd798b1aaf39b4610bba7aad28c9bea9b910f25c784ccf9ec1fa719d1f9", size = 531036 }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/c0/e9/6d7724983b3d5a0908dbf74f64038ade77c18646ff6636ec7894fd392ce1/cffi-2.1.0-cp310-cp310-macosx_10_15_x86_64.whl", hash = "sha256:b65f590ef2a44640f9a05dbb548a429b4ade77913ce683ac8b1480777658a6c0", size = 183837 },
+    { url = "https://files.pythonhosted.org/packages/69/aa/24580a278de21fd7322635556334d9b535f1cbc00b0a3919447cdf464c65/cffi-2.1.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:164bff1657b2a74f0b6d54e11c9b375bc97b931f2ca9c43fcf875838da1570dd", size = 184226 },
+    { url = "https://files.pythonhosted.org/packages/88/a9/02cae418ec4beb282ace11958d9d4737793439d561fadc7e6d56f2e2b354/cffi-2.1.0-cp310-cp310-manylinux1_i686.manylinux2014_i686.manylinux_2_17_i686.manylinux_2_5_i686.whl", hash = "sha256:c941bb58d5a6e1c3892d86e42927ed6c180302f07e6d395d08c416e594b98b46", size = 211107 },
+    { url = "https://files.pythonhosted.org/packages/3b/30/c806937ed5e4c2c7ac30d9d6b76b5dc57ff8b75d83800d9bb11a8253cf2a/cffi-2.1.0-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:a016194dbe13d14ee9556e734b772d8d67b947092b268d757fd4290e3ba2dfc2", size = 218733 },
+    { url = "https://files.pythonhosted.org/packages/f9/cf/398272b8bbfd58aa314fda5a7f1cdbb26d1d78ae324a11211521315dd1f0/cffi-2.1.0-cp310-cp310-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:03e9810d18c646077e501f661b682fbf5dee4676048527ca3cffe66faa9960dd", size = 205543 },
+    { url = "https://files.pythonhosted.org/packages/45/ca/f91641185cdd90c36d317a9dc7f85e88ef8682d8b300977baff5e23c35d8/cffi-2.1.0-cp310-cp310-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:19c54ac121cad98450b4896fa9a43ee0180d57bc4bc911a33db6cab1efab6cd3", size = 205460 },
+    { url = "https://files.pythonhosted.org/packages/38/66/04781a77b411f0bb5b234d62c1814754ab75ebe455ccff1b08e8d7aae98f/cffi-2.1.0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:4d433a51f1870e43a13b6732f92aaf540ff77c2015097c78556f75a2d6c030e0", size = 218760 },
+    { url = "https://files.pythonhosted.org/packages/d0/9a/bb1d5ed9c3fcae158e9f6391bf309c95d98c2ac37ed56573228471d0af5e/cffi-2.1.0-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:3d7f118b5adbfdfead90c25822690b02bc8074fba949bb7858bec4ebd55adb43", size = 221230 },
+    { url = "https://files.pythonhosted.org/packages/41/aa/3c1409cdd26094efacd1c36c66e0a6eb9d4296e4fd4f9901b8b2042f4323/cffi-2.1.0-cp310-cp310-musllinux_1_2_i686.whl", hash = "sha256:c5f5df567f6eb216de69be06ce55c8b714090fae02b18a3b40da8163b8c5fa9c", size = 213524 },
+    { url = "https://files.pythonhosted.org/packages/fa/75/74dfb7c3fc6ebbd408038476bd4c1d7e925c62614e7b9c534ecc34218288/cffi-2.1.0-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:11b3fb55f4f8ad92274ed26705f65d8f91457de71f5380061eb6d125a768fecd", size = 220341 },
+    { url = "https://files.pythonhosted.org/packages/70/b6/9003c33a3e7d2c1306f5962e646457dcfe5a8cd8fce6bbe02d7af25db783/cffi-2.1.0-cp310-cp310-win32.whl", hash = "sha256:9d72af0cf10a76a600a9690078fe31c63b9588c8e86bf9fd353f713c84b5db0f", size = 174578 },
+    { url = "https://files.pythonhosted.org/packages/8a/26/710688310447531c7a22f857c7f79d9855ec18b03e04494ced723fb37e2f/cffi-2.1.0-cp310-cp310-win_amd64.whl", hash = "sha256:fb62edb5bb52cca65fab91a63afa7561607120d26090a7e8fda6fb9f064726da", size = 185071 },
+    { url = "https://files.pythonhosted.org/packages/d3/67/85c89a59ba36a671e79638f44d466749f08179266a57e4f2ffdf92174072/cffi-2.1.0-cp311-cp311-macosx_10_15_x86_64.whl", hash = "sha256:02cb7ff33ded4f1532476731f89ede53e2e488a8e6205515a82144246ffa7dcc", size = 183845 },
+    { url = "https://files.pythonhosted.org/packages/ea/dd/e3b0baa2d3d6a857ac72b7efbf18e32e487c9cdafcc13049ad765495b15e/cffi-2.1.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:f5bce581e6b8c235e566a14768a943b172ada3ed73537bb0c0be1edee312d4e7", size = 184186 },
+    { url = "https://files.pythonhosted.org/packages/65/68/9f3ef890cf3c6ab97bd531c5677f67613d302165d16f8142b2811782a614/cffi-2.1.0-cp311-cp311-manylinux1_i686.manylinux2014_i686.manylinux_2_17_i686.manylinux_2_5_i686.whl", hash = "sha256:30b65779d598c370374fefabf138d456fd6f3216bfa7bedfab1ba82025b0cd93", size = 211892 },
+    { url = "https://files.pythonhosted.org/packages/22/d7/1a74539db16d8bfd839ff1515948948efbb162e574650fd3d846896eea95/cffi-2.1.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:88023dfe18799507b73f1dbb0d14326a17465de1bc9c9c7655c22845e9ddc3a2", size = 218793 },
+    { url = "https://files.pythonhosted.org/packages/ec/d1/9a5b7169499e8e8d8e636de70b97ac7c9447104d2ff1a2cd94790cea5162/cffi-2.1.0-cp311-cp311-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:0a96b74cda968eebbad56d973efe5098974f0a9fb323865bf99ea1fd24e3e64c", size = 205737 },
+    { url = "https://files.pythonhosted.org/packages/ba/b0/e131a9c41f10607926278453d9596163594fe1c4ebc46efe3b5e5b34eb84/cffi-2.1.0-cp311-cp311-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:a5781494d4d400a3f47f8f1da94b324f6e6b440a53387774002890a2a2f4b50f", size = 204909 },
+    { url = "https://files.pythonhosted.org/packages/fb/d2/4398416cd699b35167947c6e22aca52c47e69ad5695073c9f1f2c52e04aa/cffi-2.1.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:aa7a1b53a2a4452ada2d1b5dade9960b2522f1e61293a811a077439e39029565", size = 217883 },
+    { url = "https://files.pythonhosted.org/packages/a2/a5/d4fe77b589e5e82d43ebc809bf2e6474afe8e48e32ea050b9357645b6471/cffi-2.1.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:9d8272c0e483b024e1b9ad029821470ed8ec65631dbd90217469da0e7cd89f1c", size = 221251 },
+    { url = "https://files.pythonhosted.org/packages/22/f0/a2fc43084c0433caf7f461bccc013e28f848d04ee1c5ed7fce71423cf4d9/cffi-2.1.0-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:7762faa47e8ff7eb80bd261d9a7d8eea2d8baa69de5e95b70c1f338bbe712f02", size = 214250 },
+    { url = "https://files.pythonhosted.org/packages/04/8c/b925975448cf20634a9fbd5efceb807219db452653648d2897c0989cab2d/cffi-2.1.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:89095c1968b4ba8285840e131bf2891b09ae137fe2146905acae0354fbce1b5e", size = 219441 },
+    { url = "https://files.pythonhosted.org/packages/eb/da/5c4918a2d61d86fa927d716cb3d8e4626ef8dc8f605a599d32f33897f59a/cffi-2.1.0-cp311-cp311-win32.whl", hash = "sha256:64c753a0f87a256020004f37a1c8c02c480e725f910f0b2a0f3f07debd1b2479", size = 174496 },
+    { url = "https://files.pythonhosted.org/packages/f9/c8/6c2de1d55cf35ef8b92885d5ef280790f0fb9634d87ea1cc315176aecd61/cffi-2.1.0-cp311-cp311-win_amd64.whl", hash = "sha256:4f26194e3d95e06501b942642855aed4f953d55e95d7d01b7c4483db3ecff458", size = 185113 },
+    { url = "https://files.pythonhosted.org/packages/9e/4e/e8d7cb5783f1841a3c8fb3a7735838d7484d08ec08c9f984b14cac1ac0e9/cffi-2.1.0-cp311-cp311-win_arm64.whl", hash = "sha256:35aaea0c7ee0e58a5cd8c2fd1a48fdf7ece0d2699b7ecdda08194e9ce5dd9b3d", size = 179927 },
+    { url = "https://files.pythonhosted.org/packages/1e/85/990925db5df586ec90beb97529c853497e7f85ba0234830447faf41c3057/cffi-2.1.0-cp312-cp312-macosx_10_15_x86_64.whl", hash = "sha256:df2b82571a1b30f58a87bf4e5a9e78d2b1eff6c6ce8fd3aa3757221f93f0863f", size = 184829 },
+    { url = "https://files.pythonhosted.org/packages/4b/92/e7bb136ad6b5352603732cf907ef862ca103f20f2031c1735a46300c20c9/cffi-2.1.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:78474632761faa0fb96f30b1c928c84ebcf68713cbb80d15bab09dfe61640fde", size = 184728 },
+    { url = "https://files.pythonhosted.org/packages/c3/c0/d1ec30ffb370f748f2fb54425972bfef9871e0132e82fb589c46b6676049/cffi-2.1.0-cp312-cp312-manylinux1_i686.manylinux2014_i686.manylinux_2_17_i686.manylinux_2_5_i686.whl", hash = "sha256:5972433ad71a9e46516584ef60a0fda12d9dc459938d1539c3ddecf9bdc1368d", size = 214815 },
+    { url = "https://files.pythonhosted.org/packages/1b/dc/5620cf930688be01f2d673804291de757a934c90b946dbdc3d84130c2ea4/cffi-2.1.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:b6422532152adf4e59b110cb2808cee7a033800952f5c036b4af047ee43199e7", size = 222429 },
+    { url = "https://files.pythonhosted.org/packages/4b/a4/77b53abbf7a1e0beb9637edbef2a94d15f9c822f591e85d439ffd91519a6/cffi-2.1.0-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:46b1c8db8f6122420f32d02fffb924c2fe9bc772d228c7c711748fff56aabb2b", size = 210315 },
+    { url = "https://files.pythonhosted.org/packages/58/0c/f528df19cc94b675087324d4760d9e6d5bfae97d6217aa4fac43de4f5fcc/cffi-2.1.0-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:d9fafc5aa2e2a39aaf7f8cc0c1f044a9b07fca12e558dca53a3cc5c654ad67a7", size = 208859 },
+    { url = "https://files.pythonhosted.org/packages/62/f2/c9522a81c32132799a1972c39f5c5f8b4c8b9f00488a23feaa6c06f07741/cffi-2.1.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:1e9f50d192a3e525b15a75ab5114e442d83d657b7ec29182a991bc9a88fd3a66", size = 221844 },
+    { url = "https://files.pythonhosted.org/packages/6e/28/bd53988b9833e8f8ad539d26f4c07a6b3f6bcb1e9e02e7ca038250b3428d/cffi-2.1.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:98fff996e983a36d3aa2eca83af40c5821202e7e6f32d13ae94e3d2286f10cfe", size = 225287 },
+    { url = "https://files.pythonhosted.org/packages/79/99/0d0fd37f055224085f42bbb2c022d002e17dde4a97972822327b07d84101/cffi-2.1.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:379de10ce1ba048b1448599d1b37b24caee16309d1ac98d3982fc997f768700b", size = 223681 },
+    { url = "https://files.pythonhosted.org/packages/b0/80/c138990aa2a70b1a269f6e06348729836d733d6f970867943f61d367f8cc/cffi-2.1.0-cp312-cp312-win32.whl", hash = "sha256:9b8f0f26ca4e7513c534d351eca551947d053fac438f2a04ac96d882909b0d3a", size = 175269 },
+    { url = "https://files.pythonhosted.org/packages/a8/eb/f636456ff21a83fc13c032b58cc5dde061691546ac79efa284b2989b7982/cffi-2.1.0-cp312-cp312-win_amd64.whl", hash = "sha256:c97f080ea627e2863524c5af3836e2270b5f5dfff1f104392b959f8df0c5d384", size = 185881 },
+    { url = "https://files.pythonhosted.org/packages/dd/2c/400ea43e721727dca8a65c4521390e9196757caba4a45643acb2b63271b8/cffi-2.1.0-cp312-cp312-win_arm64.whl", hash = "sha256:6d194185eabd279f1c05ebe3504265ddfc5ad2b58d0714f7db9f01da592e9eb6", size = 180088 },
+    { url = "https://files.pythonhosted.org/packages/96/88/a996879e2eeccb815f6e3a5967b12a308257412acec882039d386bd2aa7b/cffi-2.1.0-cp313-cp313-ios_13_0_arm64_iphoneos.whl", hash = "sha256:10537b1df4967ca26d21e5072d7d54188354483b91dc75058968d3f0cf13fbda", size = 194331 },
+    { url = "https://files.pythonhosted.org/packages/58/85/7ae00d5c8dd6266f4e944c3db630f3c5c9a98b61d469c714d848b1d8138a/cffi-2.1.0-cp313-cp313-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:a95b05f9baf29b91171b3a8bd2020b028835243e7b0ff6bb23e2a3c228518b1b", size = 196966 },
+    { url = "https://files.pythonhosted.org/packages/8c/e9/45c3a76ad8d43ad9261f4c95436da61128d3ca545d72b9612c0ab5be0b1c/cffi-2.1.0-cp313-cp313-macosx_10_15_x86_64.whl", hash = "sha256:15faec4adfff450819f3aee0e2e02c812de6edb88203aa58807955db2003472a", size = 184795 },
+    { url = "https://files.pythonhosted.org/packages/84/4c/82f132cb4418ee6d953d982b19191e87e2a6372c8a4ce36e50b69d6ade4a/cffi-2.1.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:716ff8ec22f20b4d988b12884086bcef0fc99737043e503f7a3935a6be99b1ea", size = 184746 },
+    { url = "https://files.pythonhosted.org/packages/a0/1c/4ed5a0e5bdca6cbc275556de3328dd1b76fd0c11cc13c88fe66d1d8715f2/cffi-2.1.0-cp313-cp313-manylinux1_i686.manylinux2014_i686.manylinux_2_17_i686.manylinux_2_5_i686.whl", hash = "sha256:63960549e4f8dc41e31accb97b975abaecfc44c03e396c093a6436763c2ea7db", size = 214747 },
+    { url = "https://files.pythonhosted.org/packages/3a/a6/e879bb68cc23a2bc9ba8f4b7d8019f0c2694bad2ab6c4a3701d429439f58/cffi-2.1.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:ff067a8d8d880e7809e4ac88eb009bb848870115317b306666502ccad30b147f", size = 222392 },
+    { url = "https://files.pythonhosted.org/packages/88/f6/01890cfd63c08f8eb96a8319b0443690197d240a8bd6346048cf7bde9190/cffi-2.1.0-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:3b926723c13eba9f81d2ef3820d63aeceec3b2d4639906047bf675cb8a7a500d", size = 210285 },
+    { url = "https://files.pythonhosted.org/packages/a6/cf/2b684132056f438567b61e19d690dd31cd0921ace051e0a458be6074369e/cffi-2.1.0-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:47ff3a8bfd8cb9da1af7524b965127095055654c177fcfc7578debcb015eecd0", size = 208801 },
+    { url = "https://files.pythonhosted.org/packages/6f/08/f2e7d62c460faae0926f2d6e423694aa409ced3bc1fe2927a0a6e5f05416/cffi-2.1.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:799416bae98336e400981ff6e532d67d5c709cfb30afb79865a1315f94b0e224", size = 221808 },
+    { url = "https://files.pythonhosted.org/packages/38/37/04f54b8e63a02f3d908332c9effbf8c366167c6f733ed8a3d4f79b7e2a1e/cffi-2.1.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:961be50688f7fba2fa65f63712d3b9b341a22311f5253460ce933f52f0de1c8c", size = 225241 },
+    { url = "https://files.pythonhosted.org/packages/a9/d6/c72eecca433cd3e681c65ed313ab4835d9d4a379704d0f628a6a05f51c2e/cffi-2.1.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:bf5c6cf48238b0eb4c086978c492ad1cbc22373fc5b2d7353b3a598ce6db887a", size = 223588 },
+    { url = "https://files.pythonhosted.org/packages/c6/4b/e706f67279140f92939da3475ad610df18bfd52d50f14953a8e5fede71d5/cffi-2.1.0-cp313-cp313-win32.whl", hash = "sha256:db3eb7d46527159a878ec3460e9d40615bc25ba337d477db681aea6e4f05c5d2", size = 175248 },
+    { url = "https://files.pythonhosted.org/packages/5a/47/59eb7975cb0e4ef0afa764ea945b29a5bb4537a9f771cb7d6c8a5dd74c95/cffi-2.1.0-cp313-cp313-win_amd64.whl", hash = "sha256:8e74a6135550c4748af665b1b1118b6aab33b1fc6a16f9aff630af107c3b4512", size = 185717 },
+    { url = "https://files.pythonhosted.org/packages/5a/af/34fee85c48f8d94efc8597bc09470c9dd274c145f1c12e0fbc6ab6d38d74/cffi-2.1.0-cp313-cp313-win_arm64.whl", hash = "sha256:2282cd5e38aa8accd03e99d1256af8411c84cdbee6a89d841b563fdbd1f3e50f", size = 180114 },
+    { url = "https://files.pythonhosted.org/packages/d8/f0/81478e482afa03f6d18dc8f2afb5edc45b3080853b634b5ed91961be0998/cffi-2.1.0-cp314-cp314-ios_13_0_arm64_iphoneos.whl", hash = "sha256:d2117334c3af3bdcb9a88522b844a2bdb5efdc4f71c6c822df55486ae1c3347a", size = 194142 },
+    { url = "https://files.pythonhosted.org/packages/7d/95/8de304305cd9204974b0ca051b86d307cafca13aa575a0ef1b44d92c0d8c/cffi-2.1.0-cp314-cp314-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:702c436735fbe99d59ada02a1f65cfc0d31c0ee8b7290912f8fbc5cd1e4b16c3", size = 196819 },
+    { url = "https://files.pythonhosted.org/packages/20/71/7c8372d30e42415602ed9f268f7cfd66f1b855fed881ecd168bcb45dbc0b/cffi-2.1.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:1ff3456eab0d889592d1936d6125bbfbc7ae4d3354a700f8bd80450a66445d4d", size = 184965 },
+    { url = "https://files.pythonhosted.org/packages/d6/5c/584e626835f0375c928176c04137c96927165cb8733cdb3150ec04e5ee5e/cffi-2.1.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:c4165821e131d6d4ca444347c2b694e2311bcfa3fe5a861cc72968f28867beac", size = 184952 },
+    { url = "https://files.pythonhosted.org/packages/2e/d2/065fcae1c73979fac8e054462478d0ff8a29c40cdc2ed7ea5676a061df53/cffi-2.1.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:276f20fffd7b396e12516ba8edf9509210ac248cbbc5acbc39cd512f9f59ebe6", size = 222353 },
+    { url = "https://files.pythonhosted.org/packages/ed/a5/e8bbb1ce5b3ac2f53ad6a10bde44318a5a8d99d4f4a000d44a6e39aeb3e4/cffi-2.1.0-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:7d5980a3433d4b71a5e120f9dd551403d7824e31e2e67124fe2769c404c06913", size = 210051 },
+    { url = "https://files.pythonhosted.org/packages/28/ed/c127d3ac36e899c965e3361357c3befacd6578c03f40125183e41c3b219e/cffi-2.1.0-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:6ca4919c6e4f89aa99c42510b42cf54596892c00b3f9077f6bdd1505e24b9c8d", size = 208630 },
+    { url = "https://files.pythonhosted.org/packages/cc/d7/97d3136f81db489ec8d1d67748c110d6c994268fd7528014aa9f2b085e4e/cffi-2.1.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:d53d10f7da99ae46f7373b9150393e9c5eab9b224909982b43832668de4779f5", size = 221593 },
+    { url = "https://files.pythonhosted.org/packages/d3/27/93195977168ee63aed233a1a0993a2178798654d1f4bddcdd321d6fd3b21/cffi-2.1.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:c351efb95e832a853a29361675f33a7ce53de1a109cd73fd47af0712213aa4ce", size = 225146 },
+    { url = "https://files.pythonhosted.org/packages/b3/c1/6dbd291ee2ae5a50a034aa057207081f545923bbf15dad4511e985aafff5/cffi-2.1.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:dbf7c7a88e2bac086f06d14577332760bdeecc42bdec8ac4077f6260557d9326", size = 223240 },
+    { url = "https://files.pythonhosted.org/packages/0f/6f/ade5ce9863a57992a6ea3d0d10d7e29b8749fc127204b3d493d667b2815f/cffi-2.1.0-cp314-cp314-win32.whl", hash = "sha256:1854b724d00f6654c742097d5387569021be12d3a0f770eae1df8f8acfcc6acd", size = 177723 },
+    { url = "https://files.pythonhosted.org/packages/41/de/92b9eeed4ae4a21d6fd9b2a2c8505cbed573299902ea73981cc13f7ff62c/cffi-2.1.0-cp314-cp314-win_amd64.whl", hash = "sha256:1b96bfe2c4bd825681b7d311ad6d9b7280a091f43e8f63da5729638083cd3bfb", size = 187937 },
+    { url = "https://files.pythonhosted.org/packages/2e/1a/cc6ae6c2913a03aab8898eee57963cf1035b8df5872ed8b9115fcc7e2be8/cffi-2.1.0-cp314-cp314-win_arm64.whl", hash = "sha256:7d28dff1db6764108bc30788d85d61c876beff416d9a49cb9dd7c5a9f34f5804", size = 183001 },
+    { url = "https://files.pythonhosted.org/packages/14/f0/134c00ce0779ec86dea2aa1aac69339c2741a8045072676763512363a2ea/cffi-2.1.0-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:7ea6b3e2c4250ff1de21c630fe72d0f63eb95c2c32ffbf64a358cf4a8836d714", size = 188538 },
+    { url = "https://files.pythonhosted.org/packages/50/d8/3b86aba791cb610d24e8a3e1b2cd529e71fa15096b04e4d4e360049d4a4c/cffi-2.1.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:6af371f3767faeffc6ac1ef57cdfd25844403e9d3f476c5537caee499de96376", size = 188230 },
+    { url = "https://files.pythonhosted.org/packages/14/d0/117dcd9209255ad8571fbc8c92ef32593a1d294dcec91ddc4e4db50606f2/cffi-2.1.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:eb4e8997a49aa2c08a3e43c9045d224448b8941d88e7ac163c7d383e560cbf98", size = 223899 },
+    { url = "https://files.pythonhosted.org/packages/b6/3d/f20f8b886b254e3ad10e15cd4186d3aed49f3e6a35ab37aab9f8f25f7c03/cffi-2.1.0-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:bf01d8c84cbea96b944c73b22182e6c7c432b3475632b8111dbfdc95ddad6e13", size = 211652 },
+    { url = "https://files.pythonhosted.org/packages/28/3b/fad54de07260b93ddeef4b96d0131d57ea900675df1d410ae1deee52d7a6/cffi-2.1.0-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:33eb1ad83ebe8f313e0df035c406227d55a79456704a863fad9842136af5ad7d", size = 210755 },
+    { url = "https://files.pythonhosted.org/packages/cc/82/3d5c705acb7abbba9bbd7d79b8e62e0f25b6120eb7ae6ac49f1b721722fe/cffi-2.1.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:ac0f1a2d0cfa7eea3f2aaf006ab6e70e8feeb16b75d65b7e5939982ca2f11056", size = 223933 },
+    { url = "https://files.pythonhosted.org/packages/6c/d0/47e338384ab6b1004241002fa616301020cea4fc95f283506565d252f276/cffi-2.1.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:c16914df9fb7f500e440e6875fa23ff5e0b31db01fa9c06af98d59a91f0dc2e4", size = 226749 },
+    { url = "https://files.pythonhosted.org/packages/70/25/65bd5b58ea4bfdfc15cde02cb5365f89ef8ab8b2adfb8fe5c4bd4233382f/cffi-2.1.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:5ecbd0499275d57506d397eebe1981cee87b47fcd9ef5c22cab7ed7644a39a94", size = 225703 },
+    { url = "https://files.pythonhosted.org/packages/dc/78/aa01ac599a8a4322533d45a1f9bc93b338276d2d59dabbe7c6d92a775c81/cffi-2.1.0-cp314-cp314t-win32.whl", hash = "sha256:7d034dcffa09e9a46c93fa3a3be402096cb5354ac6e41ab8e5cc9cd8b642ad76", size = 182857 },
+    { url = "https://files.pythonhosted.org/packages/b9/26/d00496b22de4d4228f32dde94ad996f350c8aad676d63bcca0743c8dea4d/cffi-2.1.0-cp314-cp314t-win_amd64.whl", hash = "sha256:0582a58f3051372229ca8e7f5f589f9e5632678208d8636fea3676711fdf7fe5", size = 194065 },
+    { url = "https://files.pythonhosted.org/packages/d5/dd/0c7dbf815a579ff005008a2d815a55d6bb047c349eef536d9dc53d3f0a8d/cffi-2.1.0-cp314-cp314t-win_arm64.whl", hash = "sha256:510aeeeac94811b138077451da1fb18b308a5feab47dd2b603af55804155e1c8", size = 186404 },
+    { url = "https://files.pythonhosted.org/packages/55/c7/8c8c50cb11c6750051daf12164098a9a6f027ac4356967fd4d800a07f242/cffi-2.1.0-cp315-cp315-ios_13_0_arm64_iphoneos.whl", hash = "sha256:2e9dabb9abcb7ad15938c7196ad5c1718a4e6d33cc79b4c0209bdb64c4a54a5c", size = 194121 },
+    { url = "https://files.pythonhosted.org/packages/99/e2/67680bf19a6b60d2bb7ff83baefa2a4c3d2d7dc0f3277034b802e1fc504c/cffi-2.1.0-cp315-cp315-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:37f525a7e7e50c017fdebe58b787be310ad59357ae43a053943a6e1a6c526001", size = 196820 },
+    { url = "https://files.pythonhosted.org/packages/ed/da/4bbe583a3b3a5c8c60892124fe17f3fa3656523faf0d3484eae90f091853/cffi-2.1.0-cp315-cp315-macosx_10_15_x86_64.whl", hash = "sha256:95f2954c2c9473d892eca6e0409f3568b37ab62a8eedb122461f73cc273476e3", size = 184936 },
+    { url = "https://files.pythonhosted.org/packages/e5/4b/1f4c36ab273980d7aa75bb126ea4f8971f24a96108acad3a0a084028c57b/cffi-2.1.0-cp315-cp315-macosx_11_0_arm64.whl", hash = "sha256:cdf2448aab5f661c9315308ec8b93f4e8a1a67a3c733f8631067a2b67d5913dc", size = 185045 },
+    { url = "https://files.pythonhosted.org/packages/ef/c3/ad299dc38f3583f8d916b299f028af418a9ec98bc695fcbebeae7420691c/cffi-2.1.0-cp315-cp315-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:90bec57cf82089383bd06a605b3eb8daebf7e5a668520beaf6e327a83a947699", size = 222342 },
+    { url = "https://files.pythonhosted.org/packages/eb/d8/df4543cc087245044ed02ef3ad8e0a26619d0075ac7a77a12dc81177851b/cffi-2.1.0-cp315-cp315-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:6274dcb2d15cef48daa73ed1be5a40d501d74dccd0cd6db364776d12cb6ba022", size = 210073 },
+    { url = "https://files.pythonhosted.org/packages/2c/0e/fac738d73728c6cea2a88a2883dca54892496cbba88a1dc1f2909cb8a6f5/cffi-2.1.0-cp315-cp315-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:2b71d409cccee78310ab5dec549aed052aaea483346e282c7b02362596e01bb0", size = 208551 },
+    { url = "https://files.pythonhosted.org/packages/e6/3f/0b04a700dd64f465c93020253a793a82c9b4dff9961f48facd0df945d9b8/cffi-2.1.0-cp315-cp315-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:7d3538f9c0e50670f4deb93dbb696576e60590369cae2faf7de681e597a8a1f1", size = 221649 },
+    { url = "https://files.pythonhosted.org/packages/5d/7c/b7379a5704c79eda57ce075869ba70a0368d1c850f803b3c0d078d39dcaf/cffi-2.1.0-cp315-cp315-musllinux_1_2_aarch64.whl", hash = "sha256:8f9ec95b8a043d3dfbc74d9abc6f7baf524dd27a8dc160b0a32ff9cdab650c28", size = 225203 },
+    { url = "https://files.pythonhosted.org/packages/5a/02/d5e6c43ea85c41bda2a184a3418f195fe7cf602967a8d2b94e085b83deef/cffi-2.1.0-cp315-cp315-musllinux_1_2_x86_64.whl", hash = "sha256:af5e2915d41fe6c961694d7bfdc8562942638200f3ce2765dfb8b745cf997629", size = 223263 },
+    { url = "https://files.pythonhosted.org/packages/2c/d8/772b8259bf75749adffb1c546828978381fb516f60cf701f6c83daf60c85/cffi-2.1.0-cp315-cp315-win32.whl", hash = "sha256:0a42c688d19fca6e095a53c6a6e2295a5b050a8b289f109adab02a9e61a25de6", size = 177696 },
+    { url = "https://files.pythonhosted.org/packages/2f/dd/afa2191fc6d57fedd26e5844a2fe2fcc0bbfa00961bbaa5a41e4921e7cca/cffi-2.1.0-cp315-cp315-win_amd64.whl", hash = "sha256:bccbbb5ee76a61f9d99b5bf3846a51d7fca4b6a732fe46f89295610edaf41853", size = 187914 },
+    { url = "https://files.pythonhosted.org/packages/05/ef/6cd4f8c671517162379dc79cfae5aea9106bc38abb89628d5c16adf6a838/cffi-2.1.0-cp315-cp315-win_arm64.whl", hash = "sha256:8d35c139744adb3e727cd51b1a18324bbe44b8bd41bf8322bca4d41289f48eda", size = 183004 },
+    { url = "https://files.pythonhosted.org/packages/11/b6/12fc55092817a5faa26fb8c40c7f9d662e11a46ee248c137aafc42517d92/cffi-2.1.0-cp315-cp315t-macosx_10_15_x86_64.whl", hash = "sha256:f9912624a0c0b834b7520d7769b3644453aabc0a7e1c839da7359f050750e9bc", size = 188378 },
+    { url = "https://files.pythonhosted.org/packages/8d/2e/cdac88979f295fde5daa69622c7d2111e56e7ceb94f211357fbe452339e4/cffi-2.1.0-cp315-cp315t-macosx_11_0_arm64.whl", hash = "sha256:df92f2aba50eb4d96718b68ef76f2e57a57b54f2fa62333496d16c6d585a85ca", size = 188319 },
+    { url = "https://files.pythonhosted.org/packages/e0/27/1d0b408497e41a74795af122d7b603c418c5fed0171450f899afd04e594f/cffi-2.1.0-cp315-cp315t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:0520e1f4c35f44e209cbbb421b67eec42e6a157f59444dfb6058874ff3610e5d", size = 223904 },
+    { url = "https://files.pythonhosted.org/packages/8b/31/e115c985105dd7ffb32444505f18ceb874bb42d992af05d5dced7ecf1980/cffi-2.1.0-cp315-cp315t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:3681e031db29958a7502f5c0c9d6bbc4c36cb20f7b104086fa642d1799631ff8", size = 211554 },
+    { url = "https://files.pythonhosted.org/packages/5a/67/9e6e09409336d9e515c58367e7cfcf4f89df06ad25252675595a58eb59d5/cffi-2.1.0-cp315-cp315t-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:762f99479dcb369f60ab9017ad4ab97a36a1dd7c1ee5a3b15db0f4b8659120cd", size = 210795 },
+    { url = "https://files.pythonhosted.org/packages/19/e5/d3cc82a4a0be7902af279c04181ad038449c096734464a5ae1de3e1401bd/cffi-2.1.0-cp315-cp315t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:0611e7ebf90573a535ebdc33ae9da222d037853983e13359f580fab781ca017f", size = 223843 },
+    { url = "https://files.pythonhosted.org/packages/b9/65/b434abc97ce7cecc2c640fde160507c0ecc7e21544b483ba3325d2e2ea17/cffi-2.1.0-cp315-cp315t-musllinux_1_2_aarch64.whl", hash = "sha256:86cf8755a791f72c85dc287128cc62d4f24d392e3f1e15837245623f4a33cccc", size = 226773 },
+    { url = "https://files.pythonhosted.org/packages/b5/9f/d4dc66ca651eb1145a133314cda721abf13cfac3d28c4a0402263ae6ad75/cffi-2.1.0-cp315-cp315t-musllinux_1_2_x86_64.whl", hash = "sha256:ba00f661f8ba35d075c937174e27c2c421cec3942fd2e0ea3e66996757c0fdd9", size = 225719 },
+    { url = "https://files.pythonhosted.org/packages/68/5a/e536c528bc8057496c360c0978559a2dc45653f89dd6151078aa7d8fca1a/cffi-2.1.0-cp315-cp315t-win32.whl", hash = "sha256:cb96698e3c7413d906ce83f8ffd245ec1bd94707541f299d0ce4d6b0193e982b", size = 182760 },
+    { url = "https://files.pythonhosted.org/packages/d3/0b/0ffe8b82d3875bced5fa1e7986a7a46b748262a40ab7f60b475eb9fb1bb3/cffi-2.1.0-cp315-cp315t-win_amd64.whl", hash = "sha256:f146d154428a2523f9cc7936c02353c2459b8f6cf07d3cd1ee1c0a611109c5d5", size = 193769 },
+    { url = "https://files.pythonhosted.org/packages/a0/17/1073b53b68c9b5ca6914adf5f8bf55aacc2d3be102418c90700160ea8605/cffi-2.1.0-cp315-cp315t-win_arm64.whl", hash = "sha256:cbb7640ce37159548d2147b5b8c241f962143d4c71231431820783f4dc78f210", size = 186405 },
+]
+
+[[package]]
+name = "cryptography"
+version = "49.0.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+    { name = "cffi", marker = "platform_python_implementation != 'PyPy'" },
+    { name = "typing-extensions", marker = "python_full_version < '3.11'" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/1f/99/d1c90d6041656cc6ee229dc99cd67fd0cd5aec3c5f7d72fffc27cc750054/cryptography-49.0.0.tar.gz", hash = "sha256:f89660a348f4f78a92366240a61404e337586ef7f5909a2fef59ca88ef505493", size = 854345 }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/9b/22/adf66990e63584a68dfb50c24f48a125c07b1699899381c8151e63ed458c/cryptography-49.0.0-cp311-abi3-macosx_11_0_arm64.whl", hash = "sha256:966fe0e9c67490071f14c0d2b1cb2dfb3023c5ce39457343931415f08382f2db", size = 4032100 },
+    { url = "https://files.pythonhosted.org/packages/09/41/3797cfaf69cae04a13ee78ebd83f0678d9c02b4779d21ce24445326f1a69/cryptography-49.0.0-cp311-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:36d1709f992593689b45bda411498d62c6e365f2ca00b84657d4dadd24de16db", size = 4692978 },
+    { url = "https://files.pythonhosted.org/packages/e6/8b/43011f7ebe515a8aa20d61f290a326cd890c2e738e16e59eaff8d9c3a412/cryptography-49.0.0-cp311-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:0e959b578856a3924bc0cbb710fc12c387b9412a951389f3ca61704a9e25f325", size = 4716422 },
+    { url = "https://files.pythonhosted.org/packages/4a/91/01ce7303a4579e6d3a6abef01bd322848e9ea7a219adcabc5048b9033571/cryptography-49.0.0-cp311-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:53ecee2e23f7169b6117e99fc8a944e5e50f79e69758a83b52a00cb98ab2b2d2", size = 4700503 },
+    { url = "https://files.pythonhosted.org/packages/62/99/a2c95cf8293f07491e9e27c20cc4dcd18176d944e674679adeb1d0173fd6/cryptography-49.0.0-cp311-abi3-manylinux_2_28_ppc64le.whl", hash = "sha256:2eda353d8a27bcbcaa4cbed18994a74ab4d19a2ca897db188ea269ab9b71419b", size = 5309779 },
+    { url = "https://files.pythonhosted.org/packages/20/2c/0622f20ff02b2ef32558733443805dc82fd4c275be01b2d19d14676f3a1b/cryptography-49.0.0-cp311-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:2afe9051da7ae7bd5905da5a949280c7d2bb75682e188f650a9d0f2756b834c6", size = 4749683 },
+    { url = "https://files.pythonhosted.org/packages/a3/5b/c5246635d5fd3b64e0d45ae10e99fd32fe9676a79915ccfe5a61ba9af1a5/cryptography-49.0.0-cp311-abi3-manylinux_2_31_armv7l.whl", hash = "sha256:0b82e28ee398a386f0807bba7884d30f25218855690f45115831bcce5d90822c", size = 4337874 },
+    { url = "https://files.pythonhosted.org/packages/6d/88/05563c7fe2e914e87d1a536d06fe83e66b4e1d95cb593e05aea375531da8/cryptography-49.0.0-cp311-abi3-manylinux_2_34_aarch64.whl", hash = "sha256:ccac2bfebc306b862133e3bb71f3f6ee8bb525240089b2d952e4144b3a6d5da7", size = 4700283 },
+    { url = "https://files.pythonhosted.org/packages/c4/b6/d7696e4e890d6ae1469935164c9e5215c557671cb78d6e3f458ccceaa632/cryptography-49.0.0-cp311-abi3-manylinux_2_34_ppc64le.whl", hash = "sha256:d0527ce944105f257f605a827d6ebead966c752038b6e8656abb9c5edee6fc68", size = 5265844 },
+    { url = "https://files.pythonhosted.org/packages/a9/3c/f3ad17eecc1a57b0ba236dc01f90e783c51f4a2f35f64777cc4f47a184b2/cryptography-49.0.0-cp311-abi3-manylinux_2_34_x86_64.whl", hash = "sha256:cbc77da8c523d5abd028635ba850a6966fcee2c82e2bf65a41d1d8afe0f98be9", size = 4749290 },
+    { url = "https://files.pythonhosted.org/packages/4f/01/339573cf1023163a400b0b5d16f6d507de413b9f60be6fd1b77feeaf6737/cryptography-49.0.0-cp311-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:b87e65d263b3e5d3bb92a57e2a6638e2f31110fa7aa890c7b2dbba42248d0a3f", size = 4834612 },
+    { url = "https://files.pythonhosted.org/packages/71/fd/577302e213a1be9468f92d1afef66fcf1ef83d516819d9992ca547f592bd/cryptography-49.0.0-cp311-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:66ec79c3904820572d7e987abdf304281f141d37ad9a489b8e97066e7b9b6459", size = 4980804 },
+    { url = "https://files.pythonhosted.org/packages/1f/09/f42b1d190c5ba75f72062a387f8030d1d75f6ab035788f1d9c4b01de6525/cryptography-49.0.0-cp311-abi3-win_amd64.whl", hash = "sha256:e5dfc1e64de5677cec922ffa8da89c546d0415bf6efdf081842e5d44c84e1f0e", size = 3810026 },
+    { url = "https://files.pythonhosted.org/packages/ec/9e/db72b3ae7fc9cfad53e630e56c6ae83b9b6ff0bf3718ffb8012d20b3aabf/cryptography-49.0.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:73a205dce83953d131a4aa1e0fd917a2fd1c5b1eef251e9d7152efefcbf5caf7", size = 4013892 },
+    { url = "https://files.pythonhosted.org/packages/86/12/c48a424f38db03027be9f7ed5c7dc5de9933dbee992865f98b13727a009d/cryptography-49.0.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:196ecd6a36e4e9aa10270393bb98d8df88fccee0bf1e5128b91ae4eb4375896d", size = 4678835 },
+    { url = "https://files.pythonhosted.org/packages/68/28/8a3ad4653662c93fc44dc4e5d8fd374c25c42e07b34bbfbadf49cf57a5a8/cryptography-49.0.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:7abcee80084cda3f7691f3eb1ce480d8df49cec637b429aa35986c1de71738aa", size = 4697239 },
+    { url = "https://files.pythonhosted.org/packages/a8/b2/2193fc74f81aee4f9b62733133b73b5176718932ed8f2e4b03fa040480a6/cryptography-49.0.0-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:4ae387c9cb68ea569ca17e490d66d8142b81c3cc814bf179974b7d146e490bbb", size = 4685593 },
+    { url = "https://files.pythonhosted.org/packages/47/f1/1d3eaa243bfc5de4a187b22aa8c048b3e4980bfbe830ac46e6bac2e66947/cryptography-49.0.0-cp314-cp314t-manylinux_2_28_ppc64le.whl", hash = "sha256:f37d847238971164fdbc68ade6f6574aecc9c0af714190e2083429ff68f4ce9d", size = 5289961 },
+    { url = "https://files.pythonhosted.org/packages/58/39/2d51306721330c486495853eda1c567880ff036de15a14c4b74f399934af/cryptography-49.0.0-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:c2bc30226390d60ea19d9f82b19db005fe0452154a23c1c410c12ea801e43561", size = 4731145 },
+    { url = "https://files.pythonhosted.org/packages/17/50/983e838c7fd0d87fd8c969bcdd328edaf5f756e38df5281637424c155873/cryptography-49.0.0-cp314-cp314t-manylinux_2_31_armv7l.whl", hash = "sha256:07cab27cc7b7e0fd28e5e26bb9eeedde5c135c868b46de4a27845abe94af6122", size = 4321719 },
+    { url = "https://files.pythonhosted.org/packages/a7/f5/8f571d7e27c55bce9f76f026143bcb1e040a4233149ecca0bea5fa5dd5f7/cryptography-49.0.0-cp314-cp314t-manylinux_2_34_aarch64.whl", hash = "sha256:b20133d204d2bb56ba047642199603876c872026ca53e79c35b83772ab2cc505", size = 4685209 },
+    { url = "https://files.pythonhosted.org/packages/e7/84/0e27016a6fc5a0886f797018b26aa42f40c09a82332bff77822a451deaaa/cryptography-49.0.0-cp314-cp314t-manylinux_2_34_ppc64le.whl", hash = "sha256:b970c6da94d5bb18629db453d14f2a1300f6bf59b61e9b82377931ef95504866", size = 5246285 },
+    { url = "https://files.pythonhosted.org/packages/11/2d/5e1fb307cb5931881516b464c98774b3f2c36b5d4bb9a2830253cf553cad/cryptography-49.0.0-cp314-cp314t-manylinux_2_34_x86_64.whl", hash = "sha256:d8ecde755e2e91bf773fc94e8c9d730cd7f2007004cb492263a794ec3899a1c8", size = 4730441 },
+    { url = "https://files.pythonhosted.org/packages/e4/c0/bff5a02ee731d207d6a1ed51732549d8c53d2bc8da1d10ec6f2844201d68/cryptography-49.0.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:e3fb64c420688e5319ae25113a354015abbd8dffbfbc41781a1ea66fc7622ac3", size = 4815869 },
+    { url = "https://files.pythonhosted.org/packages/b9/26/814681d14248d95d73d5c3eea0c39a94eb8302df966f670a2c60de90974b/cryptography-49.0.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:32703d93296f5c1f4b53349ad3a250c2cae0fdecd3a3dd5d47e616d8d616af27", size = 4960948 },
+    { url = "https://files.pythonhosted.org/packages/4c/fe/93ecac273d3738939d023612ad12cca9a3740a5345d69fda04134c43fd96/cryptography-49.0.0-cp314-cp314t-win_amd64.whl", hash = "sha256:33cd0565932807baddb67b96dbee92f2c374b5c89dee09fd74079aeb8c8dba61", size = 3799153 },
+    { url = "https://files.pythonhosted.org/packages/19/2a/5bb823f5bedcf80718cea7fbc95ec5515cca3769633c4b01a32be7f30e7c/cryptography-49.0.0-cp39-abi3-macosx_11_0_arm64.whl", hash = "sha256:ec5e529fb80935c94fe7b729f9972b50e351a0e6b50aa294fd5cabb109fcc29a", size = 4025947 },
+    { url = "https://files.pythonhosted.org/packages/3d/df/40577043ca124e17012f408ddddaeb213b856336ac82ddb3bc915f39e29f/cryptography-49.0.0-cp39-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:f78ff2c9ed8dc2d036b0f4d640e22522213d047c1b14e61205a7e55c80a494d4", size = 4692429 },
+    { url = "https://files.pythonhosted.org/packages/2c/99/2d13299eb3dd27b02dcfaafcc91d6b5cb3329f7cbd6d8f51921acd566c1a/cryptography-49.0.0-cp39-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:35b151772baff2c74cba7fa290ceaff4c3b11c0c881eb93eb5dbc05a7cfbba18", size = 4700968 },
+    { url = "https://files.pythonhosted.org/packages/a5/4d/9c0cd02f95e2602dd5e563da149ee0830abef3537be8b34dc56281ebe27a/cryptography-49.0.0-cp39-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:0f21641cf4b30fca7aee061ced0ec7ad7b073518088b7c9969a297c0ae796c69", size = 4697758 },
+    { url = "https://files.pythonhosted.org/packages/24/01/186c825898477d77e2324d5360fefe622ff1d8d1963ec0554e2cada8ec77/cryptography-49.0.0-cp39-abi3-manylinux_2_28_ppc64le.whl", hash = "sha256:9e82dcc8e56052715fb18b2429e3bca4823b1629136a2084fc45a9a5cecb9b64", size = 5298863 },
+    { url = "https://files.pythonhosted.org/packages/b8/7b/62cbbab75d0659865bf0273790031544a0b16c8072d258f9428dcd8190dc/cryptography-49.0.0-cp39-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:6f2debedf9ca60cf1d5bd466475638af5130f89965605cd818484d19987d3a21", size = 4735983 },
+    { url = "https://files.pythonhosted.org/packages/6c/72/3e798c064bc39e471008075d0f9bc9daf77a80879c092e4a8e170c585ed4/cryptography-49.0.0-cp39-abi3-manylinux_2_31_armv7l.whl", hash = "sha256:8c25ceb16df5b9435f3f6a9829204985b0e0cbee3b48aacd432c7d2c850b44d9", size = 4334173 },
+    { url = "https://files.pythonhosted.org/packages/f0/ee/6fca21d1ac73e06f8bef71940abfd4d2f6472b4bca284d770f32bd4086f6/cryptography-49.0.0-cp39-abi3-manylinux_2_34_aarch64.whl", hash = "sha256:28d8b15e6275f12c8a207dc309dfa957903c927d08d0cc937ee3f63f200693cc", size = 4697298 },
+    { url = "https://files.pythonhosted.org/packages/67/d0/a5fcd3515f0bae49a7b6d0413cc1bdccdcc1fc0047037a0d480642cdc5d6/cryptography-49.0.0-cp39-abi3-manylinux_2_34_ppc64le.whl", hash = "sha256:6fc361c34fb6aac015ce19435876635e5c6d21db31998b0920f675f131e043b8", size = 5254338 },
+    { url = "https://files.pythonhosted.org/packages/a0/84/84fe36f19caf857d61cb7fc9c63035a47ffabd84ea12d1d393148efa3615/cryptography-49.0.0-cp39-abi3-manylinux_2_34_x86_64.whl", hash = "sha256:2400ef9c9e2299a25614eb1dea3db54a69b1349efd043bfac9c67630d136df36", size = 4735650 },
+    { url = "https://files.pythonhosted.org/packages/6c/a0/db537264e234f7273a73ec020873d6d6b39dfd8a53db78b550ca8320440e/cryptography-49.0.0-cp39-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:67e1d20ad9ef3a563c59ef22e7a8a0b8210bd26604369ea4a30a7c66aefe504e", size = 4834820 },
+    { url = "https://files.pythonhosted.org/packages/93/77/8df9eb486495979bccecd1062e2eaf435250e84437040295b57d09048b0b/cryptography-49.0.0-cp39-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:42b0684e0e40cf26122427802486f6d93aea593612603a94fbf260c7eb1e9c1b", size = 4967968 },
+    { url = "https://files.pythonhosted.org/packages/c2/e6/f60198ea8d9dfa15fff9ed4ca02ce362f6eadd9ba757dcc50634c4257b63/cryptography-49.0.0-cp39-abi3-win_amd64.whl", hash = "sha256:026ac7423e6fa66872d3bf889be5974507da3944f866f704fa200eadacd00001", size = 3785547 },
+    { url = "https://files.pythonhosted.org/packages/63/d3/4a83af35d65e3fad632c926fad684c193ea4398569ccb0bbbc7fe8f5dc9a/cryptography-49.0.0-pp311-pypy311_pp73-macosx_11_0_arm64.whl", hash = "sha256:fc1e275c2f1d97b1a6450b8b0ea3ebfa6e087a611c2b26cb2404d48588abab7b", size = 3993685 },
+    { url = "https://files.pythonhosted.org/packages/d6/a7/f9dac0ab7f80368c56993a7bf638ef9935f825c91902798481fac0898138/cryptography-49.0.0-pp311-pypy311_pp73-manylinux_2_28_aarch64.whl", hash = "sha256:c83782480a4a9da4d0feb51950131ba32e12e70813848b3343f6e18c28a66838", size = 4676239 },
+    { url = "https://files.pythonhosted.org/packages/d7/70/2ba3769dd0ae167e2f33dfa9592d45db6ff9a61d62ca1a5b3d1bdd09068f/cryptography-49.0.0-pp311-pypy311_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:b39efa323140595abd3ecca8529d321ae50f55f3aa3ba9cc81ea56a6011953d5", size = 4715584 },
+    { url = "https://files.pythonhosted.org/packages/94/64/2923570ac1c0bd3a737aa366ac3abbbbde273042308b8cde95e2364a6e6a/cryptography-49.0.0-pp311-pypy311_pp73-manylinux_2_34_aarch64.whl", hash = "sha256:b47db11c2c3525083296069b98ac5221907455e989ae0c2e3008bde851921615", size = 4675885 },
+    { url = "https://files.pythonhosted.org/packages/ab/f8/614dc7e051418cfe53d55173c1e24c6b0085e89996fe90508c2fdf769aef/cryptography-49.0.0-pp311-pypy311_pp73-manylinux_2_34_x86_64.whl", hash = "sha256:084ef1af862eb07ec46d25f68689f2102a9fc0e05ce7b80f14f5fe51e4eef0f6", size = 4715449 },
+    { url = "https://files.pythonhosted.org/packages/aa/50/a9caea39ad19c431c1a3f8a31114df65b260cdfe67786b6c7e7c040c4c44/cryptography-49.0.0-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:be9fcb48a55f023493482827d4f459bd263cc20efde64f204b97c123201850c6", size = 3783731 },
+]
+
+[[package]]
+name = "meshnet-tracker"
+version = "0.1.0"
+source = { editable = "." }
+dependencies = [
+    { name = "cryptography" },
+    { name = "websockets" },
+]
+
+[package.metadata]
+requires-dist = [
+    { name = "cryptography", specifier = ">=41" },
+    { name = "websockets", specifier = ">=13" },
+]
+
+[[package]]
+name = "pycparser"
+version = "3.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/1b/7d/92392ff7815c21062bea51aa7b87d45576f649f16458d78b7cf94b9ab2e6/pycparser-3.0.tar.gz", hash = "sha256:600f49d217304a5902ac3c37e1281c9fe94e4d0489de643a9504c5cdfdfc6b29", size = 103492 }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/0c/c3/44f3fbbfa403ea2a7c779186dc20772604442dde72947e7d01069cbe98e3/pycparser-3.0-py3-none-any.whl", hash = "sha256:b727414169a36b7d524c1c3e31839a521725078d7b2ff038656844266160a992", size = 48172 },
+]
+
+[[package]]
+name = "typing-extensions"
+version = "4.16.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/f6/cc/6253133b5bb138fc3306cebfbda2c520f545d36b5be2c7255cc528bb45d6/typing_extensions-4.16.0.tar.gz", hash = "sha256:dc983d19a509c94dba722ee6abd33940f7c05a89e243c47e907eb4db6f1a43e5", size = 113555 }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/49/d3/b8441a820a491ddfc024b0b0cf0393375b75ea13866d9c66727e54c2fc80/typing_extensions-4.16.0-py3-none-any.whl", hash = "sha256:481caa481374e813c1b176ada14e97f1f67a4539ce9cfeb3f350d78d6370c2e8", size = 45571 },
+]
+
+[[package]]
+name = "websockets"
+version = "16.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/04/24/4b2031d72e840ce4c1ccb255f693b15c334757fc50023e4db9537080b8c4/websockets-16.0.tar.gz", hash = "sha256:5f6261a5e56e8d5c42a4497b364ea24d94d9563e8fbd44e78ac40879c60179b5", size = 179346 }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/20/74/221f58decd852f4b59cc3354cccaf87e8ef695fede361d03dc9a7396573b/websockets-16.0-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:04cdd5d2d1dacbad0a7bf36ccbcd3ccd5a30ee188f2560b7a62a30d14107b31a", size = 177343 },
+    { url = "https://files.pythonhosted.org/packages/19/0f/22ef6107ee52ab7f0b710d55d36f5a5d3ef19e8a205541a6d7ffa7994e5a/websockets-16.0-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:8ff32bb86522a9e5e31439a58addbb0166f0204d64066fb955265c4e214160f0", size = 175021 },
+    { url = "https://files.pythonhosted.org/packages/10/40/904a4cb30d9b61c0e278899bf36342e9b0208eb3c470324a9ecbaac2a30f/websockets-16.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:583b7c42688636f930688d712885cf1531326ee05effd982028212ccc13e5957", size = 175320 },
+    { url = "https://files.pythonhosted.org/packages/9d/2f/4b3ca7e106bc608744b1cdae041e005e446124bebb037b18799c2d356864/websockets-16.0-cp310-cp310-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:7d837379b647c0c4c2355c2499723f82f1635fd2c26510e1f587d89bc2199e72", size = 183815 },
+    { url = "https://files.pythonhosted.org/packages/86/26/d40eaa2a46d4302becec8d15b0fc5e45bdde05191e7628405a19cf491ccd/websockets-16.0-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:df57afc692e517a85e65b72e165356ed1df12386ecb879ad5693be08fac65dde", size = 185054 },
+    { url = "https://files.pythonhosted.org/packages/b0/ba/6500a0efc94f7373ee8fefa8c271acdfd4dca8bd49a90d4be7ccabfc397e/websockets-16.0-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:2b9f1e0d69bc60a4a87349d50c09a037a2607918746f07de04df9e43252c77a3", size = 184565 },
+    { url = "https://files.pythonhosted.org/packages/04/b4/96bf2cee7c8d8102389374a2616200574f5f01128d1082f44102140344cc/websockets-16.0-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:335c23addf3d5e6a8633f9f8eda77efad001671e80b95c491dd0924587ece0b3", size = 183848 },
+    { url = "https://files.pythonhosted.org/packages/02/8e/81f40fb00fd125357814e8c3025738fc4ffc3da4b6b4a4472a82ba304b41/websockets-16.0-cp310-cp310-win32.whl", hash = "sha256:37b31c1623c6605e4c00d466c9d633f9b812ea430c11c8a278774a1fde1acfa9", size = 178249 },
+    { url = "https://files.pythonhosted.org/packages/b4/5f/7e40efe8df57db9b91c88a43690ac66f7b7aa73a11aa6a66b927e44f26fa/websockets-16.0-cp310-cp310-win_amd64.whl", hash = "sha256:8e1dab317b6e77424356e11e99a432b7cb2f3ec8c5ab4dabbcee6add48f72b35", size = 178685 },
+    { url = "https://files.pythonhosted.org/packages/f2/db/de907251b4ff46ae804ad0409809504153b3f30984daf82a1d84a9875830/websockets-16.0-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:31a52addea25187bde0797a97d6fc3d2f92b6f72a9370792d65a6e84615ac8a8", size = 177340 },
+    { url = "https://files.pythonhosted.org/packages/f3/fa/abe89019d8d8815c8781e90d697dec52523fb8ebe308bf11664e8de1877e/websockets-16.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:417b28978cdccab24f46400586d128366313e8a96312e4b9362a4af504f3bbad", size = 175022 },
+    { url = "https://files.pythonhosted.org/packages/58/5d/88ea17ed1ded2079358b40d31d48abe90a73c9e5819dbcde1606e991e2ad/websockets-16.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:af80d74d4edfa3cb9ed973a0a5ba2b2a549371f8a741e0800cb07becdd20f23d", size = 175319 },
+    { url = "https://files.pythonhosted.org/packages/d2/ae/0ee92b33087a33632f37a635e11e1d99d429d3d323329675a6022312aac2/websockets-16.0-cp311-cp311-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:08d7af67b64d29823fed316505a89b86705f2b7981c07848fb5e3ea3020c1abe", size = 184631 },
+    { url = "https://files.pythonhosted.org/packages/c8/c5/27178df583b6c5b31b29f526ba2da5e2f864ecc79c99dae630a85d68c304/websockets-16.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:7be95cfb0a4dae143eaed2bcba8ac23f4892d8971311f1b06f3c6b78952ee70b", size = 185870 },
+    { url = "https://files.pythonhosted.org/packages/87/05/536652aa84ddc1c018dbb7e2c4cbcd0db884580bf8e95aece7593fde526f/websockets-16.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:d6297ce39ce5c2e6feb13c1a996a2ded3b6832155fcfc920265c76f24c7cceb5", size = 185361 },
+    { url = "https://files.pythonhosted.org/packages/6d/e2/d5332c90da12b1e01f06fb1b85c50cfc489783076547415bf9f0a659ec19/websockets-16.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:1c1b30e4f497b0b354057f3467f56244c603a79c0d1dafce1d16c283c25f6e64", size = 184615 },
+    { url = "https://files.pythonhosted.org/packages/77/fb/d3f9576691cae9253b51555f841bc6600bf0a983a461c79500ace5a5b364/websockets-16.0-cp311-cp311-win32.whl", hash = "sha256:5f451484aeb5cafee1ccf789b1b66f535409d038c56966d6101740c1614b86c6", size = 178246 },
+    { url = "https://files.pythonhosted.org/packages/54/67/eaff76b3dbaf18dcddabc3b8c1dba50b483761cccff67793897945b37408/websockets-16.0-cp311-cp311-win_amd64.whl", hash = "sha256:8d7f0659570eefb578dacde98e24fb60af35350193e4f56e11190787bee77dac", size = 178684 },
+    { url = "https://files.pythonhosted.org/packages/84/7b/bac442e6b96c9d25092695578dda82403c77936104b5682307bd4deb1ad4/websockets-16.0-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:71c989cbf3254fbd5e84d3bff31e4da39c43f884e64f2551d14bb3c186230f00", size = 177365 },
+    { url = "https://files.pythonhosted.org/packages/b0/fe/136ccece61bd690d9c1f715baaeefd953bb2360134de73519d5df19d29ca/websockets-16.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:8b6e209ffee39ff1b6d0fa7bfef6de950c60dfb91b8fcead17da4ee539121a79", size = 175038 },
+    { url = "https://files.pythonhosted.org/packages/40/1e/9771421ac2286eaab95b8575b0cb701ae3663abf8b5e1f64f1fd90d0a673/websockets-16.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:86890e837d61574c92a97496d590968b23c2ef0aeb8a9bc9421d174cd378ae39", size = 175328 },
+    { url = "https://files.pythonhosted.org/packages/18/29/71729b4671f21e1eaa5d6573031ab810ad2936c8175f03f97f3ff164c802/websockets-16.0-cp312-cp312-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:9b5aca38b67492ef518a8ab76851862488a478602229112c4b0d58d63a7a4d5c", size = 184915 },
+    { url = "https://files.pythonhosted.org/packages/97/bb/21c36b7dbbafc85d2d480cd65df02a1dc93bf76d97147605a8e27ff9409d/websockets-16.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e0334872c0a37b606418ac52f6ab9cfd17317ac26365f7f65e203e2d0d0d359f", size = 186152 },
+    { url = "https://files.pythonhosted.org/packages/4a/34/9bf8df0c0cf88fa7bfe36678dc7b02970c9a7d5e065a3099292db87b1be2/websockets-16.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:a0b31e0b424cc6b5a04b8838bbaec1688834b2383256688cf47eb97412531da1", size = 185583 },
+    { url = "https://files.pythonhosted.org/packages/47/88/4dd516068e1a3d6ab3c7c183288404cd424a9a02d585efbac226cb61ff2d/websockets-16.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:485c49116d0af10ac698623c513c1cc01c9446c058a4e61e3bf6c19dff7335a2", size = 184880 },
+    { url = "https://files.pythonhosted.org/packages/91/d6/7d4553ad4bf1c0421e1ebd4b18de5d9098383b5caa1d937b63df8d04b565/websockets-16.0-cp312-cp312-win32.whl", hash = "sha256:eaded469f5e5b7294e2bdca0ab06becb6756ea86894a47806456089298813c89", size = 178261 },
+    { url = "https://files.pythonhosted.org/packages/c3/f0/f3a17365441ed1c27f850a80b2bc680a0fa9505d733fe152fdf5e98c1c0b/websockets-16.0-cp312-cp312-win_amd64.whl", hash = "sha256:5569417dc80977fc8c2d43a86f78e0a5a22fee17565d78621b6bb264a115d4ea", size = 178693 },
+    { url = "https://files.pythonhosted.org/packages/cc/9c/baa8456050d1c1b08dd0ec7346026668cbc6f145ab4e314d707bb845bf0d/websockets-16.0-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:878b336ac47938b474c8f982ac2f7266a540adc3fa4ad74ae96fea9823a02cc9", size = 177364 },
+    { url = "https://files.pythonhosted.org/packages/7e/0c/8811fc53e9bcff68fe7de2bcbe75116a8d959ac699a3200f4847a8925210/websockets-16.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:52a0fec0e6c8d9a784c2c78276a48a2bdf099e4ccc2a4cad53b27718dbfd0230", size = 175039 },
+    { url = "https://files.pythonhosted.org/packages/aa/82/39a5f910cb99ec0b59e482971238c845af9220d3ab9fa76dd9162cda9d62/websockets-16.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:e6578ed5b6981005df1860a56e3617f14a6c307e6a71b4fff8c48fdc50f3ed2c", size = 175323 },
+    { url = "https://files.pythonhosted.org/packages/bd/28/0a25ee5342eb5d5f297d992a77e56892ecb65e7854c7898fb7d35e9b33bd/websockets-16.0-cp313-cp313-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:95724e638f0f9c350bb1c2b0a7ad0e83d9cc0c9259f3ea94e40d7b02a2179ae5", size = 184975 },
+    { url = "https://files.pythonhosted.org/packages/f9/66/27ea52741752f5107c2e41fda05e8395a682a1e11c4e592a809a90c6a506/websockets-16.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c0204dc62a89dc9d50d682412c10b3542d748260d743500a85c13cd1ee4bde82", size = 186203 },
+    { url = "https://files.pythonhosted.org/packages/37/e5/8e32857371406a757816a2b471939d51c463509be73fa538216ea52b792a/websockets-16.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:52ac480f44d32970d66763115edea932f1c5b1312de36df06d6b219f6741eed8", size = 185653 },
+    { url = "https://files.pythonhosted.org/packages/9b/67/f926bac29882894669368dc73f4da900fcdf47955d0a0185d60103df5737/websockets-16.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:6e5a82b677f8f6f59e8dfc34ec06ca6b5b48bc4fcda346acd093694cc2c24d8f", size = 184920 },
+    { url = "https://files.pythonhosted.org/packages/3c/a1/3d6ccdcd125b0a42a311bcd15a7f705d688f73b2a22d8cf1c0875d35d34a/websockets-16.0-cp313-cp313-win32.whl", hash = "sha256:abf050a199613f64c886ea10f38b47770a65154dc37181bfaff70c160f45315a", size = 178255 },
+    { url = "https://files.pythonhosted.org/packages/6b/ae/90366304d7c2ce80f9b826096a9e9048b4bb760e44d3b873bb272cba696b/websockets-16.0-cp313-cp313-win_amd64.whl", hash = "sha256:3425ac5cf448801335d6fdc7ae1eb22072055417a96cc6b31b3861f455fbc156", size = 178689 },
+    { url = "https://files.pythonhosted.org/packages/f3/1d/e88022630271f5bd349ed82417136281931e558d628dd52c4d8621b4a0b2/websockets-16.0-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:8cc451a50f2aee53042ac52d2d053d08bf89bcb31ae799cb4487587661c038a0", size = 177406 },
+    { url = "https://files.pythonhosted.org/packages/f2/78/e63be1bf0724eeb4616efb1ae1c9044f7c3953b7957799abb5915bffd38e/websockets-16.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:daa3b6ff70a9241cf6c7fc9e949d41232d9d7d26fd3522b1ad2b4d62487e9904", size = 175085 },
+    { url = "https://files.pythonhosted.org/packages/bb/f4/d3c9220d818ee955ae390cf319a7c7a467beceb24f05ee7aaaa2414345ba/websockets-16.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:fd3cb4adb94a2a6e2b7c0d8d05cb94e6f1c81a0cf9dc2694fb65c7e8d94c42e4", size = 175328 },
+    { url = "https://files.pythonhosted.org/packages/63/bc/d3e208028de777087e6fb2b122051a6ff7bbcca0d6df9d9c2bf1dd869ae9/websockets-16.0-cp314-cp314-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:781caf5e8eee67f663126490c2f96f40906594cb86b408a703630f95550a8c3e", size = 185044 },
+    { url = "https://files.pythonhosted.org/packages/ad/6e/9a0927ac24bd33a0a9af834d89e0abc7cfd8e13bed17a86407a66773cc0e/websockets-16.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:caab51a72c51973ca21fa8a18bd8165e1a0183f1ac7066a182ff27107b71e1a4", size = 186279 },
+    { url = "https://files.pythonhosted.org/packages/b9/ca/bf1c68440d7a868180e11be653c85959502efd3a709323230314fda6e0b3/websockets-16.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:19c4dc84098e523fd63711e563077d39e90ec6702aff4b5d9e344a60cb3c0cb1", size = 185711 },
+    { url = "https://files.pythonhosted.org/packages/c4/f8/fdc34643a989561f217bb477cbc47a3a07212cbda91c0e4389c43c296ebf/websockets-16.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:a5e18a238a2b2249c9a9235466b90e96ae4795672598a58772dd806edc7ac6d3", size = 184982 },
+    { url = "https://files.pythonhosted.org/packages/dd/d1/574fa27e233764dbac9c52730d63fcf2823b16f0856b3329fc6268d6ae4f/websockets-16.0-cp314-cp314-win32.whl", hash = "sha256:a069d734c4a043182729edd3e9f247c3b2a4035415a9172fd0f1b71658a320a8", size = 177915 },
+    { url = "https://files.pythonhosted.org/packages/8a/f1/ae6b937bf3126b5134ce1f482365fde31a357c784ac51852978768b5eff4/websockets-16.0-cp314-cp314-win_amd64.whl", hash = "sha256:c0ee0e63f23914732c6d7e0cce24915c48f3f1512ec1d079ed01fc629dab269d", size = 178381 },
+    { url = "https://files.pythonhosted.org/packages/06/9b/f791d1db48403e1f0a27577a6beb37afae94254a8c6f08be4a23e4930bc0/websockets-16.0-cp314-cp314t-macosx_10_15_universal2.whl", hash = "sha256:a35539cacc3febb22b8f4d4a99cc79b104226a756aa7400adc722e83b0d03244", size = 177737 },
+    { url = "https://files.pythonhosted.org/packages/bd/40/53ad02341fa33b3ce489023f635367a4ac98b73570102ad2cdd770dacc9a/websockets-16.0-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:b784ca5de850f4ce93ec85d3269d24d4c82f22b7212023c974c401d4980ebc5e", size = 175268 },
+    { url = "https://files.pythonhosted.org/packages/74/9b/6158d4e459b984f949dcbbb0c5d270154c7618e11c01029b9bbd1bb4c4f9/websockets-16.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:569d01a4e7fba956c5ae4fc988f0d4e187900f5497ce46339c996dbf24f17641", size = 175486 },
+    { url = "https://files.pythonhosted.org/packages/e5/2d/7583b30208b639c8090206f95073646c2c9ffd66f44df967981a64f849ad/websockets-16.0-cp314-cp314t-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:50f23cdd8343b984957e4077839841146f67a3d31ab0d00e6b824e74c5b2f6e8", size = 185331 },
+    { url = "https://files.pythonhosted.org/packages/45/b0/cce3784eb519b7b5ad680d14b9673a31ab8dcb7aad8b64d81709d2430aa8/websockets-16.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:152284a83a00c59b759697b7f9e9cddf4e3c7861dd0d964b472b70f78f89e80e", size = 186501 },
+    { url = "https://files.pythonhosted.org/packages/19/60/b8ebe4c7e89fb5f6cdf080623c9d92789a53636950f7abacfc33fe2b3135/websockets-16.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:bc59589ab64b0022385f429b94697348a6a234e8ce22544e3681b2e9331b5944", size = 186062 },
+    { url = "https://files.pythonhosted.org/packages/88/a8/a080593f89b0138b6cba1b28f8df5673b5506f72879322288b031337c0b8/websockets-16.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:32da954ffa2814258030e5a57bc73a3635463238e797c7375dc8091327434206", size = 185356 },
+    { url = "https://files.pythonhosted.org/packages/c2/b6/b9afed2afadddaf5ebb2afa801abf4b0868f42f8539bfe4b071b5266c9fe/websockets-16.0-cp314-cp314t-win32.whl", hash = "sha256:5a4b4cc550cb665dd8a47f868c8d04c8230f857363ad3c9caf7a0c3bf8c61ca6", size = 178085 },
+    { url = "https://files.pythonhosted.org/packages/9f/3e/28135a24e384493fa804216b79a6a6759a38cc4ff59118787b9fb693df93/websockets-16.0-cp314-cp314t-win_amd64.whl", hash = "sha256:b14dc141ed6d2dde437cddb216004bcac6a1df0935d79656387bd41632ba0bbd", size = 178531 },
+    { url = "https://files.pythonhosted.org/packages/72/07/c98a68571dcf256e74f1f816b8cc5eae6eb2d3d5cfa44d37f801619d9166/websockets-16.0-pp311-pypy311_pp73-macosx_10_15_x86_64.whl", hash = "sha256:349f83cd6c9a415428ee1005cadb5c2c56f4389bc06a9af16103c3bc3dcc8b7d", size = 174947 },
+    { url = "https://files.pythonhosted.org/packages/7e/52/93e166a81e0305b33fe416338be92ae863563fe7bce446b0f687b9df5aea/websockets-16.0-pp311-pypy311_pp73-macosx_11_0_arm64.whl", hash = "sha256:4a1aba3340a8dca8db6eb5a7986157f52eb9e436b74813764241981ca4888f03", size = 175260 },
+    { url = "https://files.pythonhosted.org/packages/56/0c/2dbf513bafd24889d33de2ff0368190a0e69f37bcfa19009ef819fe4d507/websockets-16.0-pp311-pypy311_pp73-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:f4a32d1bd841d4bcbffdcb3d2ce50c09c3909fbead375ab28d0181af89fd04da", size = 176071 },
+    { url = "https://files.pythonhosted.org/packages/a5/8f/aea9c71cc92bf9b6cc0f7f70df8f0b420636b6c96ef4feee1e16f80f75dd/websockets-16.0-pp311-pypy311_pp73-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0298d07ee155e2e9fda5be8a9042200dd2e3bb0b8a38482156576f863a9d457c", size = 176968 },
+    { url = "https://files.pythonhosted.org/packages/9a/3f/f70e03f40ffc9a30d817eef7da1be72ee4956ba8d7255c399a01b135902a/websockets-16.0-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:a653aea902e0324b52f1613332ddf50b00c06fdaf7e92624fbf8c77c78fa5767", size = 178735 },
+    { url = "https://files.pythonhosted.org/packages/6f/28/258ebab549c2bf3e64d2b0217b973467394a9cea8c42f70418ca2c5d0d2e/websockets-16.0-py3-none-any.whl", hash = "sha256:1637db62fad1dc833276dded54215f2c7fa46912301a24bd94d45d46a011ceec", size = 171598 },
+]

tests/test_accounts.py

@@ -1,319 +1,449 @@
-"""Dashboard user accounts: registration, login, roles, API keys, usage.
-
-Unit tests for AccountStore plus HTTP integration on the tracker:
-register/login/logout, per-account balance and usage, API-key lifecycle
-(revoked keys rejected by the OpenAI proxy), and the admin listing.
-"""
-
-import json
-import urllib.error
-import urllib.request
-
-import pytest
-
-from meshnet_tracker.accounts import AccountStore
-from meshnet_tracker.auth import sign_hive_request
-from meshnet_tracker.billing import BillingLedger
-from meshnet_tracker.server import TrackerServer
-
-HIVE_SECRET = "test-hive-secret"
-
-
-# ---------------------------------------------------------------- unit tests
-
-
-def test_first_account_is_admin_then_users():
-    store = AccountStore()
-    first = store.register(email="admin@example.com", password="secret-123")
-    second = store.register(email="user@example.com", password="secret-123")
-    assert first["role"] == "admin"
-    assert second["role"] == "user"
-
-
-def test_register_requires_email_or_wallet_and_password_length():
-    store = AccountStore()
-    with pytest.raises(ValueError, match="email or a wallet"):
-        store.register(password="secret-123")
-    with pytest.raises(ValueError, match="invalid email"):
-        store.register(email="not-an-email", password="secret-123")
-    with pytest.raises(ValueError, match="at least 8"):
-        store.register(email="a@b.co", password="short")
-
-
-def test_register_rejects_duplicate_identifiers():
-    store = AccountStore()
-    store.register(email="dup@example.com", password="secret-123")
-    with pytest.raises(ValueError, match="already exists"):
-        store.register(email="DUP@example.com", password="other-secret")
-
-
-def test_login_by_email_or_wallet():
-    store = AccountStore()
-    account = store.register(
-        email="both@example.com", wallet="WalletXYZ", password="secret-123"
-    )
-    assert store.verify_login("both@example.com", "secret-123")["account_id"] == account["account_id"]
-    assert store.verify_login("WalletXYZ", "secret-123")["account_id"] == account["account_id"]
-    assert store.verify_login("both@example.com", "wrong-password") is None
-    assert store.verify_login("nobody@example.com", "secret-123") is None
-
-
-def test_sessions_resolve_and_destroy():
-    store = AccountStore()
-    account = store.register(email="s@example.com", password="secret-123")
-    token = store.create_session(account["account_id"])
-    assert store.session_account(token)["account_id"] == account["account_id"]
-    store.destroy_session(token)
-    assert store.session_account(token) is None
-    assert store.session_account("bogus") is None
-
-
-def test_api_key_lifecycle():
-    store = AccountStore()
-    account = store.register(email="k@example.com", password="secret-123")
-    other = store.register(email="other@example.com", password="secret-123")
-    key = store.create_api_key(account["account_id"])
-    assert key.startswith("sk-mesh-")
-    assert store.keys_for(account["account_id"]) == [key]
-    # someone else's account cannot revoke it
-    assert store.revoke_api_key(other["account_id"], key) is False
-    assert store.revoke_api_key(account["account_id"], key) is True
-    assert store.keys_for(account["account_id"]) == []
-    assert store.is_key_revoked(key)
-
-
-def test_accounts_persist_across_restart(tmp_path):
-    db = str(tmp_path / "accounts.db")
-    store = AccountStore(db_path=db)
-    account = store.register(email="p@example.com", password="secret-123")
-    key = store.create_api_key(account["account_id"])
-    store.save_to_db()
-
-    reloaded = AccountStore(db_path=db)
-    assert reloaded.verify_login("p@example.com", "secret-123") is not None
-    assert reloaded.keys_for(account["account_id"]) == [key]
-
-
-def test_account_events_replicate_and_dedupe():
-    leader = AccountStore()
-    follower = AccountStore()
-    account = leader.register(email="r@example.com", password="secret-123")
-    key = leader.create_api_key(account["account_id"])
-    leader.revoke_api_key(account["account_id"], key)
-
-    events, cursor = leader.events_since(0)
-    assert follower.apply_events(events) == len(events)
-    assert follower.apply_events(events) == 0  # replay is a no-op
-    assert follower.verify_login("r@example.com", "secret-123") is not None
-    assert follower.is_key_revoked(key)
-    more, _ = leader.events_since(cursor)
-    assert more == []
-
-
-# ---------------------------------------------------------- HTTP integration
-
-
-def _call(url, method="GET", body=None, token=None):
-    headers = {"Content-Type": "application/json"}
-    if token:
-        headers["Authorization"] = f"Bearer {token}"
-    data = json.dumps(body).encode() if body is not None else None
-    req = urllib.request.Request(url, data=data, headers=headers, method=method)
-    with urllib.request.urlopen(req) as r:
-        return json.loads(r.read())
-
-
-@pytest.fixture
-def account_tracker():
-    """Tracker with credit features pinned OFF (defaults are devnet-friendly 1.0)."""
-    ledger = BillingLedger(starting_credit=0.0, default_price_per_1k=0.02)
-    tracker = TrackerServer(
-        billing=ledger,
-        accounts=AccountStore(),
-        hive_secret=HIVE_SECRET,
-        starting_credit=0.0,
-        devnet_topup_amount=0.0,
-    )
-    port = tracker.start()
-    yield f"http://127.0.0.1:{port}", ledger
-    tracker.stop()
-
-
-def test_register_login_and_account_view(account_tracker):
-    url, _ = account_tracker
-    reg = _call(f"{url}/v1/auth/register", "POST",
-                {"email": "admin@example.com", "password": "secret-123"})
-    assert reg["account"]["role"] == "admin"
-    assert reg["api_key"].startswith("sk-mesh-")
-    assert reg["session_token"]
-
-    login = _call(f"{url}/v1/auth/login", "POST",
-                  {"identifier": "admin@example.com", "password": "secret-123"})
-    me = _call(f"{url}/v1/account", token=login["session_token"])
-    assert me["account"]["email"] == "admin@example.com"
-    assert me["api_keys"] == [reg["api_key"]]
-    assert me["total_balance"] == pytest.approx(0.0)
-    assert me["usage"]["requests"] == 0
-
-
-def test_bad_credentials_and_missing_session_are_401(account_tracker):
-    url, _ = account_tracker
-    _call(f"{url}/v1/auth/register", "POST",
-          {"email": "a@example.com", "password": "secret-123"})
-    with pytest.raises(urllib.error.HTTPError) as exc_info:
-        _call(f"{url}/v1/auth/login", "POST",
-              {"identifier": "a@example.com", "password": "wrong-pass"})
-    assert exc_info.value.code == 401
-    with pytest.raises(urllib.error.HTTPError) as exc_info:
-        _call(f"{url}/v1/account")
-    assert exc_info.value.code == 401
-
-
-def test_key_create_revoke_and_revoked_key_rejected_by_proxy(account_tracker):
-    url, _ = account_tracker
-    reg = _call(f"{url}/v1/auth/register", "POST",
-                {"email": "k@example.com", "password": "secret-123"})
-    token = reg["session_token"]
-
-    new_key = _call(f"{url}/v1/account/keys", "POST", {}, token=token)["api_key"]
-    me = _call(f"{url}/v1/account", token=token)
-    assert sorted(me["api_keys"]) == sorted([reg["api_key"], new_key])
-
-    _call(f"{url}/v1/account/keys/revoke", "POST", {"api_key": new_key}, token=token)
-    with pytest.raises(urllib.error.HTTPError) as exc_info:
-        _call(f"{url}/v1/chat/completions", "POST",
-              {"model": "any", "messages": []}, token=new_key)
-    assert exc_info.value.code == 401
-    assert "revoked" in exc_info.value.read().decode()
-
-
-def test_admin_listing_requires_admin_role(account_tracker):
-    url, _ = account_tracker
-    admin = _call(f"{url}/v1/auth/register", "POST",
-                  {"email": "admin@example.com", "password": "secret-123"})
-    user = _call(f"{url}/v1/auth/register", "POST",
-                 {"wallet": "WalletUser1", "password": "secret-123"})
-
-    with pytest.raises(urllib.error.HTTPError) as exc_info:
-        _call(f"{url}/v1/admin/accounts", token=user["session_token"])
-    assert exc_info.value.code == 403
-
-    listing = _call(f"{url}/v1/admin/accounts", token=admin["session_token"])
-    accounts = listing["accounts"]
-    assert len(accounts) == 2
-    assert accounts[0]["role"] == "admin"
-    assert accounts[1]["wallet"] == "WalletUser1"
-    assert "balances" in accounts[0]
-
-
-def test_accounts_gossip_endpoint_applies_events(account_tracker):
-    url, _ = account_tracker
-    peer = AccountStore()
-    peer.register(email="remote@example.com", password="secret-123")
-    events, _ = peer.events_since(0)
-    body = json.dumps({"events": events}).encode()
-    req = urllib.request.Request(
-        f"{url}/v1/accounts/gossip", data=body,
-        headers={"Content-Type": "application/json", **sign_hive_request(HIVE_SECRET, body)},
-        method="POST",
-    )
-    with urllib.request.urlopen(req) as r:
-        result = json.loads(r.read())
-    assert result["applied"] == len(events)
-    login = _call(f"{url}/v1/auth/login", "POST",
-                  {"identifier": "remote@example.com", "password": "secret-123"})
-    assert login["account"]["email"] == "remote@example.com"
-
-
-def test_accounts_endpoints_404_when_disabled():
-    tracker = TrackerServer()  # no accounts, no billing
-    port = tracker.start()
-    try:
-        with pytest.raises(urllib.error.HTTPError) as exc_info:
-            _call(f"http://127.0.0.1:{port}/v1/auth/register", "POST",
-                  {"email": "x@example.com", "password": "secret-123"})
-        assert exc_info.value.code == 404
-    finally:
-        tracker.stop()
-
-
-# ------------------------------------------- US-039/US-040: credit and top-up
-
-
-@pytest.fixture
-def funded_tracker():
-    """Tracker with Caller Credit and the devnet top-up faucet enabled."""
-    ledger = BillingLedger(starting_credit=0.0, default_price_per_1k=0.02)
-    tracker = TrackerServer(
-        billing=ledger,
-        accounts=AccountStore(),
-        hive_secret=HIVE_SECRET,
-        starting_credit=1.0,
-        devnet_topup_amount=10.0,
-    )
-    port = tracker.start()
-    yield f"http://127.0.0.1:{port}", ledger
-    tracker.stop()
-
-
-def test_caller_credit_granted_once_per_account(funded_tracker):
-    url, ledger = funded_tracker
-    reg = _call(f"{url}/v1/auth/register", "POST",
-                {"email": "c@example.com", "password": "secret-123"})
-    token = reg["session_token"]
-    first_key = reg["api_key"]
-    assert ledger.get_client_balance(first_key) == pytest.approx(1.0)
-
-    # A second key never re-grants — not even after revoking the first.
-    second = _call(f"{url}/v1/account/keys", "POST", {}, token=token)
-    assert second["caller_credit_granted"] is False
-    assert ledger.get_client_balance(second["api_key"]) == pytest.approx(0.0)
-    _call(f"{url}/v1/account/keys/revoke", "POST", {"api_key": first_key}, token=token)
-    third = _call(f"{url}/v1/account/keys", "POST", {}, token=token)
-    assert third["caller_credit_granted"] is False
-    assert ledger.get_client_balance(third["api_key"]) == pytest.approx(0.0)
-
-
-def test_unknown_bearer_key_rejected_by_proxy(funded_tracker):
-    url, ledger = funded_tracker
-    with pytest.raises(urllib.error.HTTPError) as exc_info:
-        _call(f"{url}/v1/chat/completions", "POST",
-              {"model": "any", "messages": []}, token="sk-mesh-made-up-key")
-    assert exc_info.value.code == 401
-    assert "unknown API key" in exc_info.value.read().decode()
-    # The invented key must not have become a billable client.
-    assert ledger.get_client_balance("sk-mesh-made-up-key") == pytest.approx(0.0)
-
-
-def test_devnet_topup_credits_own_key_only(funded_tracker):
-    url, ledger = funded_tracker
-    owner = _call(f"{url}/v1/auth/register", "POST",
-                  {"email": "own@example.com", "password": "secret-123"})
-    other = _call(f"{url}/v1/auth/register", "POST",
-                  {"email": "oth@example.com", "password": "secret-123"})
-
-    me = _call(f"{url}/v1/account", token=owner["session_token"])
-    assert me["topup_amount"] == pytest.approx(10.0)
-
-    result = _call(f"{url}/v1/account/topup", "POST",
-                   {"api_key": owner["api_key"]}, token=owner["session_token"])
-    assert result["credited"] == pytest.approx(10.0)
-    assert result["balance"] == pytest.approx(11.0)  # 1.0 caller credit + 10.0
-
-    with pytest.raises(urllib.error.HTTPError) as exc_info:
-        _call(f"{url}/v1/account/topup", "POST",
-              {"api_key": owner["api_key"]}, token=other["session_token"])
-    assert exc_info.value.code == 403
-    assert ledger.get_client_balance(owner["api_key"]) == pytest.approx(11.0)
-
-
-def test_topup_404_when_disabled(account_tracker):
-    url, _ = account_tracker
-    reg = _call(f"{url}/v1/auth/register", "POST",
-                {"email": "t@example.com", "password": "secret-123"})
-    me = _call(f"{url}/v1/account", token=reg["session_token"])
-    assert me["topup_amount"] == pytest.approx(0.0)
-    with pytest.raises(urllib.error.HTTPError) as exc_info:
-        _call(f"{url}/v1/account/topup", "POST",
-              {"api_key": reg["api_key"]}, token=reg["session_token"])
-    assert exc_info.value.code == 404
+"""Dashboard user accounts: registration, login, roles, API keys, usage.
+
+Unit tests for AccountStore plus HTTP integration on the tracker:
+register/login/logout, per-account balance and usage, API-key lifecycle
+(revoked keys rejected by the OpenAI proxy), and the admin listing.
+"""
+
+import http.cookies
+import json
+import urllib.error
+import urllib.request
+
+import pytest
+
+from meshnet_tracker.accounts import AccountStore
+from meshnet_tracker.auth import sign_hive_request
+from meshnet_tracker.billing import BillingLedger
+from meshnet_tracker.server import TrackerServer
+
+HIVE_SECRET = "test-hive-secret"
+
+
+# ---------------------------------------------------------------- unit tests
+
+
+def test_first_account_is_admin_then_users():
+    store = AccountStore()
+    first = store.register(email="admin@example.com", password="secret-123")
+    second = store.register(email="user@example.com", password="secret-123")
+    assert first["role"] == "admin"
+    assert second["role"] == "user"
+
+
+def test_register_requires_email_or_wallet_and_password_length():
+    store = AccountStore()
+    with pytest.raises(ValueError, match="email or a wallet"):
+        store.register(password="secret-123")
+    with pytest.raises(ValueError, match="invalid email"):
+        store.register(email="not-an-email", password="secret-123")
+    with pytest.raises(ValueError, match="at least 8"):
+        store.register(email="a@b.co", password="short")
+
+
+def test_register_rejects_duplicate_identifiers():
+    store = AccountStore()
+    store.register(email="dup@example.com", password="secret-123")
+    with pytest.raises(ValueError, match="already exists"):
+        store.register(email="DUP@example.com", password="other-secret")
+
+
+def test_register_and_update_nickname():
+    store = AccountStore()
+    account = store.register(
+        email="nick@example.com",
+        password="secret-123",
+        nickname="  Alpha  ",
+    )
+    assert account["nickname"] == "Alpha"
+    updated = store.update_profile(account["account_id"], nickname="Beta")
+    assert updated["nickname"] == "Beta"
+    cleared = store.update_profile(account["account_id"], nickname=None)
+    assert cleared["nickname"] is None
+
+
+def test_nickname_replicates_across_stores():
+    leader = AccountStore()
+    follower = AccountStore()
+    account = leader.register(
+        email="nick@example.com",
+        password="secret-123",
+        nickname="HiveNick",
+    )
+    leader.update_profile(account["account_id"], nickname="Renamed")
+    events, _ = leader.events_since(0)
+    follower.apply_events(events)
+    view = follower.get_account(account["account_id"])
+    assert view is not None
+    assert view["nickname"] == "Renamed"
+
+
+def test_login_by_email_or_wallet():
+    store = AccountStore()
+    account = store.register(
+        email="both@example.com", wallet="WalletXYZ", password="secret-123"
+    )
+    assert store.verify_login("both@example.com", "secret-123")["account_id"] == account["account_id"]
+    assert store.verify_login("WalletXYZ", "secret-123")["account_id"] == account["account_id"]
+    assert store.verify_login("both@example.com", "wrong-password") is None
+    assert store.verify_login("nobody@example.com", "secret-123") is None
+
+
+def test_sessions_resolve_and_destroy():
+    store = AccountStore()
+    account = store.register(email="s@example.com", password="secret-123")
+    token = store.create_session(account["account_id"])
+    assert store.session_account(token)["account_id"] == account["account_id"]
+    store.destroy_session(token)
+    assert store.session_account(token) is None
+    assert store.session_account("bogus") is None
+
+
+def test_sessions_persist_across_restart(tmp_path):
+    db = str(tmp_path / "accounts.db")
+    store = AccountStore(db_path=db)
+    account = store.register(email="cookie@example.com", password="secret-123")
+    token = store.create_session(account["account_id"])
+    store.save_to_db()
+
+    reloaded = AccountStore(db_path=db)
+    assert reloaded.session_account(token)["account_id"] == account["account_id"]
+
+
+def test_api_key_lifecycle():
+    store = AccountStore()
+    account = store.register(email="k@example.com", password="secret-123")
+    other = store.register(email="other@example.com", password="secret-123")
+    key = store.create_api_key(account["account_id"])
+    assert key.startswith("sk-mesh-")
+    assert store.keys_for(account["account_id"]) == [key]
+    # someone else's account cannot revoke it
+    assert store.revoke_api_key(other["account_id"], key) is False
+    assert store.revoke_api_key(account["account_id"], key) is True
+    assert store.keys_for(account["account_id"]) == []
+    assert store.is_key_revoked(key)
+
+
+def test_accounts_persist_across_restart(tmp_path):
+    db = str(tmp_path / "accounts.db")
+    store = AccountStore(db_path=db)
+    account = store.register(email="p@example.com", password="secret-123")
+    key = store.create_api_key(account["account_id"])
+    store.save_to_db()
+
+    reloaded = AccountStore(db_path=db)
+    assert reloaded.verify_login("p@example.com", "secret-123") is not None
+    assert reloaded.keys_for(account["account_id"]) == [key]
+
+
+def test_account_events_replicate_and_dedupe():
+    leader = AccountStore()
+    follower = AccountStore()
+    account = leader.register(email="r@example.com", password="secret-123")
+    key = leader.create_api_key(account["account_id"])
+    leader.revoke_api_key(account["account_id"], key)
+
+    events, cursor = leader.events_since(0)
+    assert follower.apply_events(events) == len(events)
+    assert follower.apply_events(events) == 0  # replay is a no-op
+    assert follower.verify_login("r@example.com", "secret-123") is not None
+    assert follower.is_key_revoked(key)
+    more, _ = leader.events_since(cursor)
+    assert more == []
+
+
+# ---------------------------------------------------------- HTTP integration
+
+
+def _call(url, method="GET", body=None, token=None):
+    headers = {"Content-Type": "application/json"}
+    if token:
+        headers["Authorization"] = f"Bearer {token}"
+    data = json.dumps(body).encode() if body is not None else None
+    req = urllib.request.Request(url, data=data, headers=headers, method=method)
+    with urllib.request.urlopen(req) as r:
+        return json.loads(r.read())
+
+
+@pytest.fixture
+def account_tracker():
+    """Tracker with credit features pinned OFF (defaults are devnet-friendly 1.0)."""
+    ledger = BillingLedger(starting_credit=0.0, default_price_per_1k=0.02)
+    tracker = TrackerServer(
+        billing=ledger,
+        accounts=AccountStore(),
+        hive_secret=HIVE_SECRET,
+        starting_credit=0.0,
+        devnet_topup_amount=0.0,
+    )
+    port = tracker.start()
+    yield f"http://127.0.0.1:{port}", ledger
+    tracker.stop()
+
+
+def test_register_login_and_account_view(account_tracker):
+    url, _ = account_tracker
+    reg = _call(f"{url}/v1/auth/register", "POST",
+                {"email": "admin@example.com", "password": "secret-123"})
+    assert reg["account"]["role"] == "admin"
+    assert reg["api_key"].startswith("sk-mesh-")
+    assert reg["session_token"]
+
+    login = _call(f"{url}/v1/auth/login", "POST",
+                  {"identifier": "admin@example.com", "password": "secret-123"})
+    me = _call(f"{url}/v1/account", token=login["session_token"])
+    assert me["account"]["email"] == "admin@example.com"
+    assert me["api_keys"] == [reg["api_key"]]
+    assert me["total_balance"] == pytest.approx(0.0)
+    assert me["usage"]["requests"] == 0
+    assert "records" not in me["usage"]
+    assert "recent" not in me["usage"]
+
+
+def test_account_usage_endpoint_returns_records(account_tracker):
+    url, ledger = account_tracker
+    reg = _call(f"{url}/v1/auth/register", "POST",
+                {"email": "usage@example.com", "password": "secret-123"})
+    ledger.charge_request(reg["api_key"], "test-model", total_tokens=42, node_work=[("wallet-1", 1)])
+    usage = _call(f"{url}/v1/account/usage", token=reg["session_token"])
+    assert usage["requests"] == 1
+    assert usage["total_tokens"] == 42
+    assert len(usage["records"]) == 1
+    assert usage["records"][0]["model"] == "test-model"
+
+
+def test_account_nickname_register_and_profile_update(account_tracker):
+    url, _ = account_tracker
+    reg = _call(f"{url}/v1/auth/register", "POST", {
+        "email": "nick@example.com",
+        "password": "secret-123",
+        "nickname": "Operator",
+    })
+    assert reg["account"]["nickname"] == "Operator"
+
+    updated = _call(
+        f"{url}/v1/account/profile",
+        "POST",
+        {"nickname": "Renamed"},
+        token=reg["session_token"],
+    )
+    assert updated["account"]["nickname"] == "Renamed"
+
+    me = _call(f"{url}/v1/account", token=reg["session_token"])
+    assert me["account"]["nickname"] == "Renamed"
+
+
+def test_login_sets_cookie_and_cookie_auth_survives_tracker_restart(tmp_path):
+    accounts_db = str(tmp_path / "accounts.db")
+    tracker = TrackerServer(
+        billing=BillingLedger(starting_credit=0.0, default_price_per_1k=0.02),
+        accounts_db=accounts_db,
+        starting_credit=0.0,
+        devnet_topup_amount=0.0,
+    )
+    port = tracker.start()
+    url = f"http://127.0.0.1:{port}"
+    try:
+        _call(f"{url}/v1/auth/register", "POST",
+              {"email": "cookie-http@example.com", "password": "secret-123"})
+        req = urllib.request.Request(
+            f"{url}/v1/auth/login",
+            data=json.dumps({
+                "identifier": "cookie-http@example.com",
+                "password": "secret-123",
+            }).encode(),
+            headers={"Content-Type": "application/json"},
+            method="POST",
+        )
+        with urllib.request.urlopen(req) as r:
+            assert json.loads(r.read())["session_token"]
+            cookie_header = r.headers["Set-Cookie"]
+    finally:
+        tracker.stop()
+
+    cookie = http.cookies.SimpleCookie(cookie_header)
+    session_cookie = cookie["meshnet_session"].OutputString()
+
+    restarted = TrackerServer(
+        billing=BillingLedger(starting_credit=0.0, default_price_per_1k=0.02),
+        accounts_db=accounts_db,
+        starting_credit=0.0,
+        devnet_topup_amount=0.0,
+    )
+    restarted_port = restarted.start()
+    restarted_url = f"http://127.0.0.1:{restarted_port}"
+    try:
+        req = urllib.request.Request(
+            f"{restarted_url}/v1/account",
+            headers={"Cookie": session_cookie},
+            method="GET",
+        )
+        with urllib.request.urlopen(req) as r:
+            me = json.loads(r.read())
+    finally:
+        restarted.stop()
+
+    assert me["account"]["email"] == "cookie-http@example.com"
+
+
+def test_bad_credentials_and_missing_session_are_401(account_tracker):
+    url, _ = account_tracker
+    _call(f"{url}/v1/auth/register", "POST",
+          {"email": "a@example.com", "password": "secret-123"})
+    with pytest.raises(urllib.error.HTTPError) as exc_info:
+        _call(f"{url}/v1/auth/login", "POST",
+              {"identifier": "a@example.com", "password": "wrong-pass"})
+    assert exc_info.value.code == 401
+    with pytest.raises(urllib.error.HTTPError) as exc_info:
+        _call(f"{url}/v1/account")
+    assert exc_info.value.code == 401
+
+
+def test_key_create_revoke_and_revoked_key_rejected_by_proxy(account_tracker):
+    url, _ = account_tracker
+    reg = _call(f"{url}/v1/auth/register", "POST",
+                {"email": "k@example.com", "password": "secret-123"})
+    token = reg["session_token"]
+
+    new_key = _call(f"{url}/v1/account/keys", "POST", {}, token=token)["api_key"]
+    me = _call(f"{url}/v1/account", token=token)
+    assert sorted(me["api_keys"]) == sorted([reg["api_key"], new_key])
+
+    _call(f"{url}/v1/account/keys/revoke", "POST", {"api_key": new_key}, token=token)
+    with pytest.raises(urllib.error.HTTPError) as exc_info:
+        _call(f"{url}/v1/chat/completions", "POST",
+              {"model": "any", "messages": []}, token=new_key)
+    assert exc_info.value.code == 401
+    assert "revoked" in exc_info.value.read().decode()
+
+
+def test_admin_listing_requires_admin_role(account_tracker):
+    url, _ = account_tracker
+    admin = _call(f"{url}/v1/auth/register", "POST",
+                  {"email": "admin@example.com", "password": "secret-123"})
+    user = _call(f"{url}/v1/auth/register", "POST",
+                 {"wallet": "WalletUser1", "password": "secret-123"})
+
+    with pytest.raises(urllib.error.HTTPError) as exc_info:
+        _call(f"{url}/v1/admin/accounts", token=user["session_token"])
+    assert exc_info.value.code == 403
+
+    listing = _call(f"{url}/v1/admin/accounts", token=admin["session_token"])
+    accounts = listing["accounts"]
+    assert len(accounts) == 2
+    assert accounts[0]["role"] == "admin"
+    assert accounts[1]["wallet"] == "WalletUser1"
+    assert "balances" in accounts[0]
+
+
+def test_accounts_gossip_endpoint_applies_events(account_tracker):
+    url, _ = account_tracker
+    peer = AccountStore()
+    peer.register(email="remote@example.com", password="secret-123")
+    events, _ = peer.events_since(0)
+    body = json.dumps({"events": events}).encode()
+    req = urllib.request.Request(
+        f"{url}/v1/accounts/gossip", data=body,
+        headers={"Content-Type": "application/json", **sign_hive_request(HIVE_SECRET, body)},
+        method="POST",
+    )
+    with urllib.request.urlopen(req) as r:
+        result = json.loads(r.read())
+    assert result["applied"] == len(events)
+    login = _call(f"{url}/v1/auth/login", "POST",
+                  {"identifier": "remote@example.com", "password": "secret-123"})
+    assert login["account"]["email"] == "remote@example.com"
+
+
+def test_accounts_endpoints_404_when_disabled():
+    tracker = TrackerServer()  # no accounts, no billing
+    port = tracker.start()
+    try:
+        with pytest.raises(urllib.error.HTTPError) as exc_info:
+            _call(f"http://127.0.0.1:{port}/v1/auth/register", "POST",
+                  {"email": "x@example.com", "password": "secret-123"})
+        assert exc_info.value.code == 404
+    finally:
+        tracker.stop()
+
+
+# ------------------------------------------- US-039/US-040: credit and top-up
+
+
+@pytest.fixture
+def funded_tracker():
+    """Tracker with Caller Credit and the devnet top-up faucet enabled."""
+    ledger = BillingLedger(starting_credit=0.0, default_price_per_1k=0.02)
+    tracker = TrackerServer(
+        billing=ledger,
+        accounts=AccountStore(),
+        hive_secret=HIVE_SECRET,
+        starting_credit=1.0,
+        devnet_topup_amount=10.0,
+    )
+    port = tracker.start()
+    yield f"http://127.0.0.1:{port}", ledger
+    tracker.stop()
+
+
+def test_caller_credit_granted_once_per_account(funded_tracker):
+    url, ledger = funded_tracker
+    reg = _call(f"{url}/v1/auth/register", "POST",
+                {"email": "c@example.com", "password": "secret-123"})
+    token = reg["session_token"]
+    first_key = reg["api_key"]
+    assert ledger.get_client_balance(first_key) == pytest.approx(1.0)
+
+    # A second key never re-grants — not even after revoking the first.
+    second = _call(f"{url}/v1/account/keys", "POST", {}, token=token)
+    assert second["caller_credit_granted"] is False
+    assert ledger.get_client_balance(second["api_key"]) == pytest.approx(0.0)
+    _call(f"{url}/v1/account/keys/revoke", "POST", {"api_key": first_key}, token=token)
+    third = _call(f"{url}/v1/account/keys", "POST", {}, token=token)
+    assert third["caller_credit_granted"] is False
+    assert ledger.get_client_balance(third["api_key"]) == pytest.approx(0.0)
+
+
+def test_unknown_bearer_key_rejected_by_proxy(funded_tracker):
+    url, ledger = funded_tracker
+    with pytest.raises(urllib.error.HTTPError) as exc_info:
+        _call(f"{url}/v1/chat/completions", "POST",
+              {"model": "any", "messages": []}, token="sk-mesh-made-up-key")
+    assert exc_info.value.code == 401
+    assert "unknown API key" in exc_info.value.read().decode()
+    # The invented key must not have become a billable client.
+    assert ledger.get_client_balance("sk-mesh-made-up-key") == pytest.approx(0.0)
+
+
+def test_devnet_topup_credits_own_key_only(funded_tracker):
+    url, ledger = funded_tracker
+    owner = _call(f"{url}/v1/auth/register", "POST",
+                  {"email": "own@example.com", "password": "secret-123"})
+    other = _call(f"{url}/v1/auth/register", "POST",
+                  {"email": "oth@example.com", "password": "secret-123"})
+
+    me = _call(f"{url}/v1/account", token=owner["session_token"])
+    assert me["topup_amount"] == pytest.approx(10.0)
+
+    result = _call(f"{url}/v1/account/topup", "POST",
+                   {"api_key": owner["api_key"]}, token=owner["session_token"])
+    assert result["credited"] == pytest.approx(10.0)
+    assert result["balance"] == pytest.approx(11.0)  # 1.0 caller credit + 10.0
+
+    with pytest.raises(urllib.error.HTTPError) as exc_info:
+        _call(f"{url}/v1/account/topup", "POST",
+              {"api_key": owner["api_key"]}, token=other["session_token"])
+    assert exc_info.value.code == 403
+    assert ledger.get_client_balance(owner["api_key"]) == pytest.approx(11.0)
+
+
+def test_topup_404_when_disabled(account_tracker):
+    url, _ = account_tracker
+    reg = _call(f"{url}/v1/auth/register", "POST",
+                {"email": "t@example.com", "password": "secret-123"})
+    me = _call(f"{url}/v1/account", token=reg["session_token"])
+    assert me["topup_amount"] == pytest.approx(0.0)
+    with pytest.raises(urllib.error.HTTPError) as exc_info:
+        _call(f"{url}/v1/account/topup", "POST",
+              {"api_key": reg["api_key"]}, token=reg["session_token"])
+    assert exc_info.value.code == 404

tests/test_dashboard.py

@@ -28,11 +28,133 @@ def test_dashboard_served_with_all_panels():
         ).read().decode()
         for panel in PANELS:
             assert panel in html
+        assert '<link rel="icon" type="image/svg+xml" href="/favicon.svg">' in html
+        favicon = urllib.request.urlopen(f"http://127.0.0.1:{port}/favicon.svg").read()
+        assert favicon.startswith(b"<svg")
+        assert b"meshnet" in favicon
         assert "<script>" in html  # polling client embedded, no build step
+        assert "resolveModelGroup" in html
+        assert "buildModelAliasMap" in html
+        assert "modelAliasKey(raw)" in html
     finally:
         tracker.stop()
 
 
+def test_dashboard_chat_uses_streaming_fetch():
+    tracker = TrackerServer(billing=BillingLedger())
+    port = tracker.start()
+    try:
+        html = urllib.request.urlopen(
+            f"http://127.0.0.1:{port}/dashboard"
+        ).read().decode()
+    finally:
+        tracker.stop()
+
+    assert "stream: true" in html
+    assert ".body.getReader()" in html
+    assert '=== "[DONE]"' in html
+    assert 'console.error("chat stream failed", err)' in html
+    assert "preloadChatModels" in html
+    assert "renderChatModels(true)" in html
+
+
+def test_network_map_includes_node_friendly_name():
+    tracker = TrackerServer()
+    port = tracker.start()
+    try:
+        body = json.dumps({
+            "endpoint": "http://127.0.0.1:9010",
+            "model": "stub-model",
+            "shard_start": 0,
+            "shard_end": 3,
+            "hardware_profile": {},
+            "friendly_name": "Kitchen GPU",
+        }).encode()
+        req = urllib.request.Request(
+            f"http://127.0.0.1:{port}/v1/nodes/register",
+            data=body,
+            headers={"Content-Type": "application/json"},
+            method="POST",
+        )
+        urllib.request.urlopen(req).read()
+        network = json.loads(
+            urllib.request.urlopen(f"http://127.0.0.1:{port}/v1/network/map").read()
+        )
+    finally:
+        tracker.stop()
+
+    assert network["nodes"][0]["friendly_name"] == "Kitchen GPU"
+
+
+def test_dashboard_chat_model_selector_shows_health_and_speed():
+    tracker = TrackerServer()
+    port = tracker.start()
+    try:
+        html = urllib.request.urlopen(
+            f"http://127.0.0.1:{port}/dashboard"
+        ).read().decode()
+    finally:
+        tracker.stop()
+
+    assert "chatModelHealthHp" in html
+    assert "modelServedCopiesFromMap" in html
+    assert "nodeDisplayName" in html
+    assert "accountDisplayName" in html
+    assert "saveNickname" in html
+    assert "chatModelTypicalTps" in html
+    assert "chatModelOptionLabel" in html
+    assert "findRoutingForModel" in html
+    assert "tok/s" in html
+    assert "toFixed(2)}HP" in html or '${copies(v)}HP' in html
+
+
+def test_dashboard_chat_sessions_use_delegated_handlers():
+    tracker = TrackerServer()
+    port = tracker.start()
+    try:
+        html = urllib.request.urlopen(
+            f"http://127.0.0.1:{port}/dashboard"
+        ).read().decode()
+    finally:
+        tracker.stop()
+
+    assert "bindChatSessionList" in html
+    assert "dataset.sessionId" in html
+    assert "dataset.deleteSession" in html
+    assert '[data-session-id]' in html
+    assert 'onclick="selectChatSession(' not in html
+    assert 'onclick="deleteChatSession(' not in html
+
+
+def test_dashboard_incremental_refresh_helpers():
+    tracker = TrackerServer()
+    port = tracker.start()
+    try:
+        html = urllib.request.urlopen(
+            f"http://127.0.0.1:{port}/dashboard"
+        ).read().decode()
+    finally:
+        tracker.stop()
+
+    assert "renderIfChanged" in html
+    assert "syncKeyedList" in html
+    assert "refreshBlocked" in html
+    assert "patchAccountPanelView" in html
+    assert "buildAccountPanelShell" in html
+    assert "refreshActiveTab" in html
+    assert "TAB_FETCHERS" in html
+    assert "loadAccountSummary" in html
+    assert "loadAccountUsage" in html
+    assert "fetchOverviewTab" in html
+    assert "pollCallWallIfIdle" in html
+    assert "pendingChatModelRefresh" in html
+    assert 'renderChatHistory(true)' in html
+    assert "renderChatHistory();" not in html
+    assert "refreshIfIdle" not in html
+    assert "refreshAccountIfIdle" not in html
+    assert "setInterval(refreshIfIdle" not in html
+
+
 def test_dashboard_served_by_follower():
     """A tracker that is not the leader (unreachable peers → never elected)
     still serves the dashboard from its own replicated state."""

tests/test_dynamic_routing.py

@@ -0,0 +1,306 @@
+"""ADR-0021: dynamic bandit-style route selection with learned statistics."""
+
+import http.server
+import json
+import random
+import threading
+import types
+import urllib.request
+
+from meshnet_tracker.routing_stats import (
+    RouteCandidate,
+    RouteStatsStore,
+    RoutingConfig,
+    choose_route,
+    route_signature,
+    route_table,
+)
+from meshnet_tracker.server import TrackerServer, _enumerate_routes
+
+
+def _post_json(url: str, payload: dict) -> dict:
+    req = urllib.request.Request(
+        url,
+        data=json.dumps(payload).encode(),
+        headers={"Content-Type": "application/json"},
+        method="POST",
+    )
+    with urllib.request.urlopen(req, timeout=10.0) as resp:
+        return json.loads(resp.read())
+
+
+def _get_json(url: str) -> dict:
+    with urllib.request.urlopen(url, timeout=10.0) as resp:
+        return json.loads(resp.read())
+
+
+def _fake_node(node_id, shard_start, shard_end, benchmark=100.0, endpoint=None):
+    return types.SimpleNamespace(
+        node_id=node_id,
+        endpoint=endpoint or f"http://{node_id}:7000",
+        model="qwen3.6-35b-a3b",
+        hf_repo="unsloth/Qwen3.6-35B-A3B",
+        shard_start=shard_start,
+        shard_end=shard_end,
+        num_layers=40,
+        benchmark_tokens_per_sec=benchmark,
+        model_tokens_per_sec={},
+        queue_depth=0,
+        proxy_inflight=0,
+        wallet_address=None,
+        relay_addr=None,
+    )
+
+
+# ---- RouteStatsStore ----------------------------------------------------
+
+
+def test_route_stats_sample_becomes_proven_and_decays():
+    store = RouteStatsStore(RoutingConfig(stats_half_life_seconds=100.0))
+    sig = "m|a[0-39]"
+    assert store.snapshot(sig, "m", now=0.0)["status"] == "unsampled"
+    assert store.record_sample("m", sig, tokens=100, elapsed_seconds=10.0, now=0.0)
+    snap = store.snapshot(sig, "m", now=1.0)
+    assert snap["status"] == "proven"
+    assert snap["tps"] == 10.0
+    # After many half-lives the sample mass decays below the proven threshold.
+    assert store.snapshot(sig, "m", now=1000.0)["status"] == "decayed"
+
+
+def test_route_stats_rejects_near_empty_samples():
+    store = RouteStatsStore(RoutingConfig(min_sample_tokens=8))
+    assert not store.record_sample("m", "sig", tokens=3, elapsed_seconds=1.0)
+    assert store.snapshot("sig", "m")["samples"] == 0
+
+
+def test_route_stats_epoch_bump_marks_stale():
+    store = RouteStatsStore()
+    sig = "m|a[0-39]"
+    store.record_sample("m", sig, tokens=100, elapsed_seconds=10.0, now=0.0)
+    assert store.snapshot(sig, "m", now=1.0)["status"] == "proven"
+    store.bump_epoch(["m"])
+    snap = store.snapshot(sig, "m", now=1.0)
+    assert snap["status"] == "stale"
+    assert snap["tps"] == 10.0  # EWMA kept as a prior for display
+    # A fresh sample under the new epoch re-proves the route.
+    store.record_sample("m", sig, tokens=100, elapsed_seconds=10.0, now=2.0)
+    assert store.snapshot(sig, "m", now=3.0)["status"] == "proven"
+
+
+def test_route_stats_ewma_averages_samples():
+    store = RouteStatsStore(RoutingConfig(stats_half_life_seconds=1e9))
+    sig = "m|a"
+    store.record_sample("m", sig, tokens=100, elapsed_seconds=10.0, now=0.0)  # 10 tps
+    store.record_sample("m", sig, tokens=200, elapsed_seconds=10.0, now=1.0)  # 20 tps
+    snap = store.snapshot(sig, "m", now=2.0)
+    assert 14.9 < snap["tps"] < 15.1
+
+
+# ---- choose_route --------------------------------------------------------
+
+
+def _candidates_two_routes():
+    fast = RouteCandidate(nodes=[], signature="m|fast", prior_tps=100.0)
+    slow = RouteCandidate(nodes=[], signature="m|slow", prior_tps=50.0)
+    return fast, slow
+
+
+def test_choose_route_without_samples_is_deterministic_best_prior():
+    store = RouteStatsStore()
+    fast, slow = _candidates_two_routes()
+    for _ in range(20):
+        picked, decision = choose_route([slow, fast], store, "m", rng=random.Random(7))
+        assert picked is fast
+        assert decision["mode"] == "scout"
+
+
+def test_choose_route_traffic_proportional_to_tps():
+    store = RouteStatsStore(RoutingConfig(stats_half_life_seconds=1e9))
+    fast, slow = _candidates_two_routes()
+    now = 0.0
+    for _ in range(5):
+        now += 1.0
+        store.record_sample("m", fast.signature, tokens=150, elapsed_seconds=10.0, now=now)
+        store.record_sample("m", slow.signature, tokens=100, elapsed_seconds=10.0, now=now)
+    rng = random.Random(42)
+    picks = {"m|fast": 0, "m|slow": 0}
+    for _ in range(4000):
+        picked, decision = choose_route([fast, slow], store, "m", rng=rng, now=now)
+        assert decision["mode"] == "exploit"
+        picks[picked.signature] += 1
+    share = picks["m|fast"] / 4000
+    # 15 tps vs 10 tps at alpha=1 → expected fast share 0.6
+    assert 0.55 < share < 0.65
+
+
+def test_choose_route_scouts_unproven_routes_at_explore_share():
+    store = RouteStatsStore(RoutingConfig(explore_share=0.25, stats_half_life_seconds=1e9))
+    fast, slow = _candidates_two_routes()
+    now = 1.0
+    store.record_sample("m", fast.signature, tokens=150, elapsed_seconds=10.0, now=now)
+    rng = random.Random(11)
+    scouted = 0
+    for _ in range(4000):
+        picked, decision = choose_route([fast, slow], store, "m", rng=rng, now=now)
+        if decision["mode"] == "scout":
+            scouted += 1
+            assert picked is slow
+    assert 0.20 < scouted / 4000 < 0.30
+
+
+# ---- _enumerate_routes ---------------------------------------------------
+
+
+def test_enumerate_routes_mixed_topology_yields_both_routes():
+    gpu = _fake_node("gpu", 0, 21, benchmark=11000.0)
+    cpu = _fake_node("cpu", 0, 39, benchmark=425.0)
+    candidates = _enumerate_routes([gpu, cpu], 0, 39, model="qwen3.6-35b-a3b")
+    signatures = {c.signature for c in candidates}
+    assert signatures == {
+        route_signature("qwen3.6-35b-a3b", [gpu, cpu]),
+        route_signature("qwen3.6-35b-a3b", [cpu]),
+    }
+    hybrid = next(c for c in candidates if len(c.nodes) == 2)
+    assert [n.node_id for n in hybrid.nodes] == ["gpu", "cpu"]
+    # Hybrid route's prior is its bottleneck hop, not the fast head.
+    assert hybrid.prior_tps == 425.0
+
+
+def test_enumerate_routes_requires_head_at_first_layer():
+    tail_only = _fake_node("tail", 22, 39)
+    assert _enumerate_routes([tail_only], 0, 39, model="m") == []
+
+
+def test_route_table_reports_coefficient_and_share():
+    store = RouteStatsStore(RoutingConfig(explore_share=0.3, stats_half_life_seconds=1e9))
+    fast, slow = _candidates_two_routes()
+    now = 1.0
+    for _ in range(3):
+        store.record_sample("m", fast.signature, tokens=150, elapsed_seconds=10.0, now=now)
+        store.record_sample("m", slow.signature, tokens=100, elapsed_seconds=10.0, now=now)
+        now += 1.0
+    rows = route_table([fast, slow], store, "m", now=now)
+    by_sig = {r["signature"]: r for r in rows}
+    assert by_sig["m|fast"]["coefficient"] == 1.0
+    assert abs(by_sig["m|slow"]["coefficient"] - (10.0 / 15.0)) < 0.01
+    # No scouts → full exploit budget split 0.6 / 0.4.
+    assert abs(by_sig["m|fast"]["expected_share"] - 0.6) < 0.01
+    assert abs(by_sig["m|slow"]["expected_share"] - 0.4) < 0.01
+
+
+# ---- integration: proxy uses route head + /v1/routing --------------------
+
+
+def test_proxy_head_is_route_head_and_routing_endpoint_lists_routes():
+    """Mixed topology (partial head 0-21 + full node 0-39): the proxy target
+    must be the selected route's own head, downstream hops must continue at
+    head.shard_end + 1 (the ADR-0020 flaw), and /v1/routing must list both
+    candidate routes."""
+
+    class ChatHandler(http.server.BaseHTTPRequestHandler):
+        def log_message(self, *args):  # noqa: ARG002
+            pass
+
+        def do_POST(self):
+            length = int(self.headers.get("Content-Length", 0))
+            self.rfile.read(length)
+            route_header = self.headers.get("X-Meshnet-Route") or "[]"
+            body = json.dumps({
+                "choices": [{"message": {"role": "assistant", "content": route_header}}],
+                "usage": {"prompt_tokens": 10, "completion_tokens": 40},
+            }).encode()
+            self.send_response(200)
+            self.send_header("Content-Type", "application/json")
+            self.send_header("Content-Length", str(len(body)))
+            self.end_headers()
+            self.wfile.write(body)
+
+    stubs = []
+    threads = []
+    for _ in range(2):
+        stub = http.server.HTTPServer(("127.0.0.1", 0), ChatHandler)
+        thread = threading.Thread(target=stub.serve_forever, daemon=True)
+        thread.start()
+        stubs.append(stub)
+        threads.append(thread)
+    gpu_stub, cpu_stub = stubs
+
+    tracker = TrackerServer(model_presets={
+        "qwen3.6-35b-a3b": {
+            "layers_start": 0,
+            "layers_end": 39,
+            "hf_repo": "unsloth/Qwen3.6-35B-A3B",
+            "aliases": ["Qwen3.6-35B-A3B"],
+        }
+    })
+    tracker_port = tracker.start()
+    try:
+        tracker._server.route_rng = random.Random(3)
+        for stub, shard_end, bench in ((gpu_stub, 21, 11000.0), (cpu_stub, 39, 425.0)):
+            _post_json(
+                f"http://127.0.0.1:{tracker_port}/v1/nodes/register",
+                {"endpoint": f"http://127.0.0.1:{stub.server_address[1]}",
+                 "model": "qwen3.6-35b-a3b",
+                 "hf_repo": "unsloth/Qwen3.6-35B-A3B",
+                 "num_layers": 40,
+                 "shard_start": 0,
+                 "shard_end": shard_end,
+                 "tracker_mode": True,
+                 "benchmark_tokens_per_sec": bench,
+                 "hardware_profile": {},
+                 "score": 1.0},
+            )
+
+        for _ in range(8):
+            _post_json(
+                f"http://127.0.0.1:{tracker_port}/v1/chat/completions",
+                {"model": "Qwen3.6-35B-A3B",
+                 "messages": [{"role": "user", "content": "hi"}]},
+            )
+
+        console = _get_json(f"http://127.0.0.1:{tracker_port}/v1/console")
+        routing = _get_json(f"http://127.0.0.1:{tracker_port}/v1/routing")
+    finally:
+        tracker.stop()
+        for stub, thread in zip(stubs, threads):
+            stub.shutdown()
+            stub.server_close()
+            thread.join(timeout=1.0)
+
+    gpu_endpoint = f"http://127.0.0.1:{gpu_stub.server_address[1]}"
+    cpu_endpoint = f"http://127.0.0.1:{cpu_stub.server_address[1]}"
+    selected = [e for e in console["events"] if e["message"] == "proxy route selected"]
+    assert selected
+    for event in selected:
+        fields = event["fields"]
+        nodes = fields["nodes"]
+        # The proxy head must be the route's first hop (ADR-0020 regression).
+        assert fields["head_endpoint"] == nodes[0]["endpoint"]
+        downstream = json.loads(fields["downstream"])
+        if fields["head_endpoint"] == gpu_endpoint:
+            # Partial head: downstream continues at layer 22, never 0.
+            assert downstream == [{"endpoint": cpu_endpoint, "start_layer": 22}]
+        else:
+            assert fields["head_endpoint"] == cpu_endpoint
+            assert downstream == []
+
+    table = routing["models"]["qwen3.6-35b-a3b"]
+    assert len(table["routes"]) == 2
+    sampled = [r for r in table["routes"] if r["samples"] > 0]
+    assert sampled, "completed requests must produce route samples"
+
+
+def test_endpoint_key_distinguishes_same_port_different_hosts():
+    from meshnet_node.torch_server import _clamp_downstream_hops, _endpoint_key
+
+    assert _endpoint_key("http://192.168.0.20:7000") == "192.168.0.20:7000"
+    assert _endpoint_key("http://192.168.0.179:7000") == "192.168.0.179:7000"
+    assert _endpoint_key("http://192.168.0.20:7000") != _endpoint_key("http://192.168.0.179:7000")
+
+    class Backend:
+        shard_end = 21
+
+    hops = [{"endpoint": "http://192.168.0.179:7000", "start_layer": 0}]
+    assert _clamp_downstream_hops(hops, Backend()) == [
+        {"endpoint": "http://192.168.0.179:7000", "start_layer": 22},
+    ]

tests/test_gossip_and_relay.py

@@ -41,6 +41,28 @@ def test_identity_different_for_different_paths(tmp_path):
     assert a["peer_id"] != b["peer_id"]
 
 
+def test_relay_peer_id_includes_node_name_for_shared_wallet():
+    from meshnet_node.relay_bridge import peer_id_from_wallet
+
+    wallet = "5gMLrmyBYTpkFjmyc4eGwcaWhYquyWgCBFFEqHzR5Qur"
+
+    assert peer_id_from_wallet(wallet, node_name="gpu head") == "5gMLrmyBYTpkFjmy-gpu-head"
+    assert peer_id_from_wallet(wallet, node_name="cpu-tail") == "5gMLrmyBYTpkFjmy-cpu-tail"
+
+
+def test_relay_peer_id_falls_back_to_endpoint_port_integer():
+    from meshnet_node.relay_bridge import peer_id_from_wallet
+
+    wallet = "5gMLrmyBYTpkFjmyc4eGwcaWhYquyWgCBFFEqHzR5Qur"
+
+    first = peer_id_from_wallet(wallet, advertised_addr="http://192.168.0.20:7001")
+    second = peer_id_from_wallet(wallet, advertised_addr="http://192.168.0.20:7002")
+
+    assert first == "5gMLrmyBYTpkFjmy-7001"
+    assert second == "5gMLrmyBYTpkFjmy-7002"
+    assert first != second
+
+
 # ---------------------------------------------------------------------------
 # TLS / certificate tests
 # ---------------------------------------------------------------------------
@@ -350,6 +372,109 @@ def test_relay_rpc_round_trips_http_request_to_peer():
     assert json.loads(response["body"]) == {"ok": True, "path": "/v1/chat/completions"}
 
 
+def test_binary_relay_frame_codecs_interoperate():
+    """Node and relay ship the same binary frame format as separate copies."""
+    import os
+
+    from meshnet_node import relay_bridge
+    from meshnet_relay import server as relay_server
+
+    header = {"request_id": "r-1", "status": 200, "headers": {"X": "y"}}
+    body = os.urandom(4096)
+    for encoder, decoder in (
+        (relay_bridge.encode_binary_frame, relay_server.decode_binary_frame),
+        (relay_server.encode_binary_frame, relay_bridge.decode_binary_frame),
+    ):
+        assert decoder(encoder(header, body)) == (header, body)
+
+    try:
+        relay_server.decode_binary_frame(b"not a frame")
+    except ValueError:
+        pass
+    else:
+        raise AssertionError("garbage bytes must not decode as a binary frame")
+
+
+def test_activation_compression_round_trips_and_skips_small_bodies():
+    """Pipeline hops zstd-compress large activations; tiny decode bodies pass raw."""
+    import os
+
+    from meshnet_node.server import _decompress_body
+    from meshnet_node.torch_server import _maybe_compress_activation
+
+    small = os.urandom(1024)
+    assert _maybe_compress_activation(small) == (small, None)
+
+    large = os.urandom(96 * 1024) * 2  # repetition so zstd actually shrinks it
+    wire, encoding = _maybe_compress_activation(large)
+    assert encoding == "zstd"
+    assert len(wire) < len(large)
+    assert _decompress_body(wire, encoding) == large
+
+
+def test_relay_rpc_carries_activation_sized_frames():
+    """A >1 MiB activation body must survive the full relay round trip.
+
+    Regression: the websockets library caps frames at 1 MiB by default, so
+    prefill activations forwarded via /rpc/<peer> died with close code 1009
+    at every hop (requester → relay, relay → bridge, bridge → relay → requester).
+    The body now travels as binary frames — raw bytes, no base64.
+    """
+    import http.server
+    import os
+
+    from meshnet_node.relay_bridge import RelayHttpBridge
+    from meshnet_node.torch_server import _relay_hop
+    from meshnet_relay.server import RelayServer
+
+    body = os.urandom(2 * 1024 * 1024)  # 2 MiB raw, ~2.7 MiB once base64-wrapped
+
+    class EchoHandler(http.server.BaseHTTPRequestHandler):
+        def do_POST(self):
+            data = self.rfile.read(int(self.headers.get("Content-Length", 0)))
+            self.send_response(200)
+            self.send_header("Content-Type", "application/octet-stream")
+            self.send_header("Content-Length", str(len(data)))
+            self.end_headers()
+            self.wfile.write(data)
+
+        def log_message(self, *args):
+            pass
+
+    local = http.server.HTTPServer(("127.0.0.1", 0), EchoHandler)
+    local_thread = threading.Thread(target=local.serve_forever, daemon=True)
+    local_thread.start()
+
+    relay = RelayServer(host="127.0.0.1", port=0)
+    port = relay.start()
+
+    bridge = RelayHttpBridge(
+        relay_url=f"ws://127.0.0.1:{port}/ws",
+        peer_id="big_peer",
+        local_base_url=f"http://127.0.0.1:{local.server_address[1]}",
+        advertised_addr="",
+    )
+    try:
+        bridge.start()
+        assert bridge.wait_connected(timeout=5.0)
+        time.sleep(0.2)  # let peer-register land in the relay registry
+        status, headers, resp_body = _relay_hop(
+            f"ws://127.0.0.1:{port}/rpc/big_peer",
+            "/forward",
+            body,
+            {"Content-Type": "application/octet-stream"},
+            timeout=30.0,
+        )
+    finally:
+        bridge.stop()
+        relay.stop()
+        local.shutdown()
+        local_thread.join(timeout=3)
+
+    assert status == 200
+    assert resp_body == body
+
+
 def test_node_relay_bridge_reconnects_after_failed_connection(monkeypatch):
     """Node-side relay bridge keeps retrying its outbound WebSocket connection."""
     import websockets.sync.client as wsc  # type: ignore[import]
@@ -370,7 +495,7 @@ def test_node_relay_bridge_reconnects_after_failed_connection(monkeypatch):
         def recv(self, timeout=1):
             raise TimeoutError
 
-    def fake_connect(url, open_timeout=5):
+    def fake_connect(url, open_timeout=5, **kwargs):
         attempts.append((url, open_timeout))
         if len(attempts) == 1:
             raise OSError("temporary relay outage")
@@ -451,6 +576,8 @@ def test_tracker_derives_relay_url_from_public_self_url():
         with urllib.request.urlopen(f"http://127.0.0.1:{port}/v1/network/map", timeout=5) as resp:
             body = _json.loads(resp.read())
         assert body["relay_url"] == "wss://ai.neuron.d-popov.com/ws"
+        assert body["relay"]["mode"] == "external"
+        assert body["relay"]["url"] == "wss://ai.neuron.d-popov.com/ws"
     finally:
         tracker.stop()
 
@@ -501,10 +628,46 @@ def test_tracker_network_map_exposes_relay_and_registered_peer():
         tracker.stop()
 
     assert body["relay_url"] == "wss://ai.neuron.d-popov.com/ws"
+    assert body["relay"]["mode"] == "external"
     assert body["nodes"][0]["relay_addr"] == "wss://ai.neuron.d-popov.com/rpc/peer123"
     assert body["nodes"][0]["peer_id"] == "peer123"
 
 
+def test_tracker_can_embed_relay_server_and_advertise_it():
+    import json as _json
+    import urllib.request
+
+    import websockets.sync.client as wsc  # type: ignore[import]
+    from meshnet_tracker.server import TrackerServer
+
+    tracker = TrackerServer(
+        host="127.0.0.1",
+        port=0,
+        embedded_relay=True,
+        embedded_relay_host="127.0.0.1",
+        embedded_relay_port=0,
+    )
+    port = tracker.start()
+    try:
+        with urllib.request.urlopen(f"http://127.0.0.1:{port}/v1/network/map", timeout=5) as resp:
+            body = _json.loads(resp.read())
+        assert body["relay"]["mode"] == "embedded"
+        assert body["relay_url"].startswith("ws://127.0.0.1:")
+        assert body["relay_url"].endswith("/ws")
+        with wsc.connect(body["relay_url"], open_timeout=5) as ws:
+            ws.send(_json.dumps({
+                "topic": "peer-register",
+                "version": 1,
+                "from_peer": "embedded-peer",
+                "msg_id": "embedded-reg-1",
+                "payload": {"peer_id": "embedded-peer", "addr": ""},
+            }))
+            peer_list = _json.loads(ws.recv(timeout=5))
+        assert peer_list["topic"] == "peer-list"
+    finally:
+        tracker.stop()
+
+
 # ---------------------------------------------------------------------------
 # mDNS (no-op without zeroconf installed)
 # ---------------------------------------------------------------------------

tests/test_hf_pricing.py

@@ -186,3 +186,24 @@ def test_qwen_preset_prices_apply_to_all_aliases(tmp_path):
         assert billing.price_for("some/other-model") == pytest.approx(0.02)
     finally:
         pass
+
+
+def test_qwen25_preset_price_is_ten_x_commercial_reference(tmp_path):
+    """Qwen2.5-0.5B bills at 10× ~$0.20/1M reference ($0.002/1k), not the 0.02 default."""
+    import pytest
+    from meshnet_tracker.server import TrackerServer, _resolve_model_preset, DEFAULT_MODEL_PRESETS
+
+    name, preset = _resolve_model_preset(DEFAULT_MODEL_PRESETS, "Qwen/Qwen2.5-0.5B-Instruct")
+    assert name == "qwen2.5-0.5b-instruct"
+    assert preset["price_per_1k_tokens"] == pytest.approx(0.002)
+
+    tracker = TrackerServer(billing_db=str(tmp_path / "billing.sqlite"))
+    billing = tracker._billing
+    assert billing is not None
+    for key in (
+        "qwen2.5-0.5b",
+        "Qwen2.5-0.5B-Instruct",
+        "Qwen/Qwen2.5-0.5B-Instruct",
+    ):
+        assert billing.price_for(key) == pytest.approx(0.002), key
+    assert billing.price_for("unrelated-model") == pytest.approx(0.02)

tests/test_kv_cache_distributed.py

@@ -0,0 +1,565 @@
+"""AH-25: sharded per-node KV cache for distributed generation.
+
+Covers the SessionCacheStore (TTL + LRU + mismatch handling), the HTTP
+session protocol (stable session id, O(1) decode payloads, 409 cache-miss
+fallback, legacy stateless compatibility), and an env-gated golden test that
+proves cached and stateless distributed generation produce identical tokens
+on a real two-shard Qwen2.5-0.5B split.
+"""
+
+import json
+import os
+import urllib.error
+import urllib.request
+
+import pytest
+
+from meshnet_node.model_backend import (
+    KVCacheMiss,
+    SessionCacheStore,
+    TailTokenResult,
+    TensorPayload,
+    TorchModelShard,
+)
+from meshnet_node.torch_server import TorchNodeServer
+
+
+# ---------------------------------------------------------------------------
+# SessionCacheStore units
+# ---------------------------------------------------------------------------
+
+class _Clock:
+    def __init__(self) -> None:
+        self.now = 0.0
+
+    def __call__(self) -> float:
+        return self.now
+
+
+def test_store_lookup_roundtrip_advances_lru():
+    store = SessionCacheStore(max_sessions=4, ttl_seconds=100.0, clock=_Clock())
+    store.store("s1", cache=object(), seq_len=6, effective_start=12)
+    entry = store.lookup("s1", expected_seq_len=6, effective_start=12)
+    assert entry.seq_len == 6
+    entry.seq_len += 1
+    assert store.lookup("s1", expected_seq_len=7).seq_len == 7
+
+
+def test_lookup_unknown_session_raises_cache_miss():
+    store = SessionCacheStore(max_sessions=4, ttl_seconds=100.0)
+    with pytest.raises(KVCacheMiss):
+        store.lookup("nope")
+
+
+def test_seq_len_mismatch_drops_entry_and_raises():
+    store = SessionCacheStore(max_sessions=4, ttl_seconds=100.0)
+    store.store("s1", cache=object(), seq_len=6, effective_start=0)
+    with pytest.raises(KVCacheMiss):
+        store.lookup("s1", expected_seq_len=9)
+    # Entry must be gone — a poisoned cache is never reused.
+    with pytest.raises(KVCacheMiss):
+        store.lookup("s1")
+
+
+def test_effective_start_mismatch_raises():
+    store = SessionCacheStore(max_sessions=4, ttl_seconds=100.0)
+    store.store("s1", cache=object(), seq_len=6, effective_start=12)
+    with pytest.raises(KVCacheMiss):
+        store.lookup("s1", effective_start=21)
+
+
+def test_ttl_expiry_evicts_stale_sessions():
+    clock = _Clock()
+    store = SessionCacheStore(max_sessions=4, ttl_seconds=60.0, clock=clock)
+    store.store("s1", cache=object(), seq_len=6, effective_start=0)
+    clock.now = 61.0
+    with pytest.raises(KVCacheMiss):
+        store.lookup("s1")
+    assert len(store) == 0
+
+
+def test_lru_eviction_bounds_session_count():
+    clock = _Clock()
+    store = SessionCacheStore(max_sessions=2, ttl_seconds=1000.0, clock=clock)
+    store.store("s1", cache=object(), seq_len=1, effective_start=0)
+    store.store("s2", cache=object(), seq_len=1, effective_start=0)
+    store.lookup("s1")  # s1 becomes most recent → s2 is LRU
+    store.store("s3", cache=object(), seq_len=1, effective_start=0)
+    assert len(store) == 2
+    with pytest.raises(KVCacheMiss):
+        store.lookup("s2")
+    store.lookup("s1")
+    store.lookup("s3")
+
+
+def test_drop_removes_session():
+    store = SessionCacheStore(max_sessions=4, ttl_seconds=100.0)
+    store.store("s1", cache=object(), seq_len=1, effective_start=0)
+    store.drop("s1")
+    with pytest.raises(KVCacheMiss):
+        store.lookup("s1")
+
+
+def test_prefill_cache_triton_cpu_failure_disables_cache_and_retries_stateless():
+    """CPU shards must recover when hybrid model cache path dispatches Triton."""
+    shard = object.__new__(TorchModelShard)
+    shard.model_id = "fake-hybrid"
+    shard.supports_kv_cache = True
+    shard._effective_start = lambda start_layer=None: 22
+    shard._new_session_cache = lambda: object()
+
+    calls = []
+
+    def fake_run_layers(hidden_states, attention_mask, position_ids, *, start_layer=None, cache=None, past_len=0):
+        calls.append({"cache": cache, "past_len": past_len})
+        if cache is not None:
+            raise RuntimeError("Pointer argument cannot be accessed from Triton (cpu tensor?)")
+        return "stateless-ok"
+
+    shard._run_layers = fake_run_layers
+
+    result = TorchModelShard._run_layers_session(
+        shard,
+        hidden_states=object(),
+        attention_mask=None,
+        position_ids=None,
+        session_id="session-1",
+        cache_mode="prefill",
+    )
+
+    assert result == "stateless-ok"
+    assert shard.supports_kv_cache is False
+    assert len(calls) == 2
+    assert calls[0]["cache"] is not None
+    assert calls[1]["cache"] is None
+
+
+# ---------------------------------------------------------------------------
+# HTTP session protocol with fake cached backends
+# ---------------------------------------------------------------------------
+
+class _ChatTokenizer:
+    eos_token = ""
+
+    def apply_chat_template(self, messages, add_generation_prompt=True, tokenize=False):
+        return "debug prompt"
+
+
+class _CachedHeadBackend:
+    model_id = "fake-model"
+    total_layers = 12
+    is_head = True
+    is_tail = False
+    supports_kv_cache = True
+    tokenizer = _ChatTokenizer()
+
+    def __init__(self) -> None:
+        self.prefills: list[str | None] = []
+        self.decode_calls: list[tuple[int, str]] = []
+        self.released: list[str] = []
+        self._seq: dict[str, int] = {}
+
+    def eos_token_ids(self) -> list[int]:
+        return [99]
+
+    def release_session(self, session_id: str) -> None:
+        self.released.append(session_id)
+
+    def encode_prompt(self, prompt: str, session_id: str | None = None) -> TensorPayload:
+        self.prefills.append(session_id)
+        if session_id:
+            self._seq[session_id] = 6
+        return TensorPayload(
+            body=b"\x00" * (1 * 6 * 8 * 2),
+            shape=[1, 6, 8],
+            attention_mask_header=None,
+            position_ids_header=None,
+        )
+
+    def encode_next_token(self, token_id: int, session_id: str) -> TensorPayload:
+        self.decode_calls.append((token_id, session_id))
+        past = self._seq[session_id]
+        self._seq[session_id] = past + 1
+        return TensorPayload(
+            body=b"\x00" * (1 * 1 * 8 * 2),
+            shape=[1, 1, 8],
+            attention_mask_header=None,
+            position_ids_header=None,
+            past_len=past,
+        )
+
+
+class _CachedTailBackend:
+    model_id = "fake-model"
+    total_layers = 12
+    is_head = False
+    is_tail = True
+    supports_kv_cache = True
+
+    def __init__(self, tokens, miss_on_call: int | None = None) -> None:
+        self._tokens = list(tokens)
+        self.miss_on_call = miss_on_call
+        self.calls: list[dict] = []
+
+    def forward_bytes(
+        self,
+        body,
+        shape,
+        attention_mask_header,
+        position_ids_header,
+        start_layer=None,
+        session_id=None,
+        cache_mode=None,
+        past_len=None,
+    ):
+        call_index = len(self.calls)
+        self.calls.append({
+            "session": session_id,
+            "mode": cache_mode,
+            "past_len": past_len,
+            "shape": list(shape),
+        })
+        if self.miss_on_call is not None and call_index == self.miss_on_call:
+            raise KVCacheMiss("session evicted (test)")
+        text, token_id = self._tokens.pop(0)
+        return TailTokenResult(text=text, token_id=token_id)
+
+
+def _chat_once(head_port: int, tail_port: int, max_tokens: int) -> str:
+    payload = json.dumps({
+        "model": "fake-model",
+        "messages": [{"role": "user", "content": "hello"}],
+        "max_tokens": max_tokens,
+    }).encode()
+    req = urllib.request.Request(
+        f"http://127.0.0.1:{head_port}/v1/chat/completions",
+        data=payload,
+        headers={
+            "Content-Type": "application/json",
+            "X-Meshnet-Route": json.dumps([
+                {"endpoint": f"http://127.0.0.1:{tail_port}", "start_layer": 6},
+            ]),
+        },
+        method="POST",
+    )
+    with urllib.request.urlopen(req, timeout=10) as resp:
+        body = json.loads(resp.read())
+    return body["choices"][0]["message"]["content"]
+
+
+def test_session_is_stable_and_decode_payloads_are_single_token():
+    head_backend = _CachedHeadBackend()
+    tail_backend = _CachedTailBackend([(" a", 1), (" b", 2), (" c", 3)])
+    head = TorchNodeServer(backend=head_backend, tracker_mode=True)
+    tail = TorchNodeServer(backend=tail_backend)
+    head_port = head.start()
+    tail_port = tail.start()
+    try:
+        content = _chat_once(head_port, tail_port, max_tokens=3)
+    finally:
+        head.stop()
+        tail.stop()
+
+    assert content == " a b c"
+    assert len(tail_backend.calls) == 3
+    # Step 0 is a full-prompt prefill; steps 1+ carry only the new token.
+    assert tail_backend.calls[0]["mode"] == "prefill"
+    assert tail_backend.calls[0]["shape"] == [1, 6, 8]
+    for step, call in enumerate(tail_backend.calls[1:], start=1):
+        assert call["mode"] == "decode"
+        assert call["shape"] == [1, 1, 8]
+        assert call["past_len"] == 6 + (step - 1)
+    # One session id across every step of the generation.
+    sessions = {call["session"] for call in tail_backend.calls}
+    assert len(sessions) == 1
+    session_id = sessions.pop()
+    assert head_backend.prefills == [session_id]
+    assert head_backend.decode_calls == [(1, session_id), (2, session_id)]
+    # Head releases its own session state when the generation ends.
+    assert head_backend.released == [session_id]
+
+
+class _BrokenTailBackend(_CachedTailBackend):
+    """Tail whose forward always fails (e.g. missing compiler, OOM)."""
+
+    def forward_bytes(self, *args, **kwargs):
+        raise RuntimeError("Failed to find C compiler (test)")
+
+
+def test_pipeline_failure_before_first_token_returns_502():
+    """A dead hop must surface as an error, not an empty 200 completion."""
+    head = TorchNodeServer(backend=_CachedHeadBackend(), tracker_mode=True)
+    tail = TorchNodeServer(backend=_BrokenTailBackend([]))
+    head_port = head.start()
+    tail_port = tail.start()
+    try:
+        try:
+            _chat_once(head_port, tail_port, max_tokens=3)
+        except urllib.error.HTTPError as exc:
+            assert exc.code == 502
+            body = json.loads(exc.read())
+            assert "pipeline error" in body["error"]["message"]
+            assert "C compiler" in body["error"]["message"]
+        else:
+            raise AssertionError("expected HTTP 502 from a failed pipeline")
+    finally:
+        head.stop()
+        tail.stop()
+
+
+def test_pipeline_failure_in_stream_emits_error_frame():
+    """Streaming requests get an OpenAI-style error frame before [DONE]."""
+    head = TorchNodeServer(backend=_CachedHeadBackend(), tracker_mode=True)
+    tail = TorchNodeServer(backend=_BrokenTailBackend([]))
+    head_port = head.start()
+    tail_port = tail.start()
+    try:
+        payload = json.dumps({
+            "model": "fake-model",
+            "messages": [{"role": "user", "content": "hello"}],
+            "max_tokens": 3,
+            "stream": True,
+        }).encode()
+        req = urllib.request.Request(
+            f"http://127.0.0.1:{head_port}/v1/chat/completions",
+            data=payload,
+            headers={
+                "Content-Type": "application/json",
+                "X-Meshnet-Route": json.dumps([
+                    {"endpoint": f"http://127.0.0.1:{tail_port}", "start_layer": 6},
+                ]),
+            },
+            method="POST",
+        )
+        with urllib.request.urlopen(req, timeout=10) as resp:
+            assert resp.status == 200
+            events = resp.read().decode().strip().split("\n\n")
+    finally:
+        head.stop()
+        tail.stop()
+
+    assert events[-1] == "data: [DONE]"
+    error_frame = json.loads(events[-2][len("data: "):])
+    assert error_frame["error"]["type"] == "upstream_error"
+    assert "C compiler" in error_frame["error"]["message"]
+
+
+def test_large_prefill_activation_survives_zstd_compressed_hop():
+    """A prefill body above _COMPRESS_MIN_BYTES travels the hop zstd-compressed.
+
+    The head compresses and sets X-Meshnet-Encoding; the tail's /forward must
+    decompress before shape validation, so a passing generation proves the
+    compressed round trip (a mishandled encoding fails validation with 400).
+    """
+
+    class _BigHeadBackend(_CachedHeadBackend):
+        def encode_prompt(self, prompt, session_id=None):
+            self.prefills.append(session_id)
+            if session_id:
+                self._seq[session_id] = 2048
+            return TensorPayload(
+                body=b"\x00" * (1 * 2048 * 32 * 2),  # 128 KiB, above the zstd threshold
+                shape=[1, 2048, 32],
+                attention_mask_header=None,
+                position_ids_header=None,
+            )
+
+    head_backend = _BigHeadBackend()
+    tail_backend = _CachedTailBackend([(" a", 1), (" b", 2)])
+    head = TorchNodeServer(backend=head_backend, tracker_mode=True)
+    tail = TorchNodeServer(backend=tail_backend)
+    head_port = head.start()
+    tail_port = tail.start()
+    try:
+        content = _chat_once(head_port, tail_port, max_tokens=2)
+    finally:
+        head.stop()
+        tail.stop()
+
+    assert content == " a b"
+    assert tail_backend.calls[0]["mode"] == "prefill"
+    assert tail_backend.calls[0]["shape"] == [1, 2048, 32]
+
+
+def test_eos_token_id_stops_generation():
+    head_backend = _CachedHeadBackend()
+    tail_backend = _CachedTailBackend([(" a", 1), ("", 99)])
+    head = TorchNodeServer(backend=head_backend, tracker_mode=True)
+    tail = TorchNodeServer(backend=tail_backend)
+    head_port = head.start()
+    tail_port = tail.start()
+    try:
+        content = _chat_once(head_port, tail_port, max_tokens=8)
+    finally:
+        head.stop()
+        tail.stop()
+
+    assert content == " a"
+    assert len(tail_backend.calls) == 2
+
+
+def test_stateless_fallback_stops_at_eos_token_id():
+    """When kv caching is off, EOS must still stop generation by token id —
+    EOS decodes to "" (skip_special_tokens) so the text check never fires."""
+
+    class _StatelessHead(_CachedHeadBackend):
+        supports_kv_cache = False
+
+    head_backend = _StatelessHead()
+    tail_backend = _CachedTailBackend([(" a", 1), ("", 99), (" never", 3)])
+    head = TorchNodeServer(backend=head_backend, tracker_mode=True)
+    tail = TorchNodeServer(backend=tail_backend)
+    head_port = head.start()
+    tail_port = tail.start()
+    try:
+        content = _chat_once(head_port, tail_port, max_tokens=8)
+    finally:
+        head.stop()
+        tail.stop()
+
+    assert content == " a"
+    # Stops after the EOS step instead of burning steps until max_tokens.
+    assert len(tail_backend.calls) == 2
+    assert head_backend.decode_calls == []
+
+
+def test_decode_forward_logging_is_rate_limited():
+    """Shard nodes log a per-session decode summary, not one line per token."""
+    tail_backend = _CachedTailBackend([])
+    tail = TorchNodeServer(backend=tail_backend)
+    tail.start()
+    try:
+        srv = tail._server
+        assert srv.note_decode_step("s1", now=0.0) == 1
+        assert srv.note_decode_step("s1", now=1.0) is None
+        assert srv.note_decode_step("s1", now=4.9) is None
+        assert srv.note_decode_step("s1", now=5.5) == 4
+        assert srv.note_decode_step("s1", now=6.0) is None
+        # Sessions are throttled independently.
+        assert srv.note_decode_step("s2", now=6.0) == 1
+    finally:
+        tail.stop()
+
+
+def test_downstream_cache_miss_falls_back_to_full_reprefill():
+    head_backend = _CachedHeadBackend()
+    # Call 1 (the first decode) raises KVCacheMiss → node answers 409 →
+    # head re-prefills the full sequence and keeps generating.
+    tail_backend = _CachedTailBackend(
+        [(" a", 1), (" b", 2), (" c", 3)], miss_on_call=1,
+    )
+    head = TorchNodeServer(backend=head_backend, tracker_mode=True)
+    tail = TorchNodeServer(backend=tail_backend)
+    head_port = head.start()
+    tail_port = tail.start()
+    try:
+        content = _chat_once(head_port, tail_port, max_tokens=3)
+    finally:
+        head.stop()
+        tail.stop()
+
+    assert content == " a b c"
+    modes = [call["mode"] for call in tail_backend.calls]
+    assert modes == ["prefill", "decode", "prefill", "decode"]
+    # Head re-prefilled once, with the same stable session id.
+    assert len(head_backend.prefills) == 2
+    assert len(set(head_backend.prefills)) == 1
+
+
+def test_kv_head_with_legacy_tail_reprefills_every_step():
+    """Mixed fleet: tail predates the protocol and returns no token_id."""
+
+    class _LegacyTailBackend:
+        model_id = "fake-model"
+        total_layers = 12
+        is_head = False
+        is_tail = True
+
+        def __init__(self) -> None:
+            self.calls = 0
+
+        def forward_bytes(self, body, shape, attention_mask_header,
+                          position_ids_header, start_layer=None, **kwargs):
+            self.calls += 1
+            return " x" if self.calls < 3 else ""
+
+    head_backend = _CachedHeadBackend()
+    tail_backend = _LegacyTailBackend()
+    head = TorchNodeServer(backend=head_backend, tracker_mode=True)
+    tail = TorchNodeServer(backend=tail_backend)
+    head_port = head.start()
+    tail_port = tail.start()
+    try:
+        content = _chat_once(head_port, tail_port, max_tokens=5)
+    finally:
+        head.stop()
+        tail.stop()
+
+    assert content == " x x"
+    # No token_id from the tail → every step is a full prefill (legacy cost),
+    # never a decode against a cache the tail doesn't keep.
+    assert head_backend.decode_calls == []
+    assert len(head_backend.prefills) == 3
+
+
+# ---------------------------------------------------------------------------
+# Golden test on a real two-shard split (env-gated: loads Qwen2.5-0.5B twice)
+# ---------------------------------------------------------------------------
+
+_GOLDEN_MODEL = "Qwen/Qwen2.5-0.5B-Instruct"
+
+requires_real_model = pytest.mark.skipif(
+    os.environ.get("MESHNET_REAL_MODEL_TESTS") != "1",
+    reason="set MESHNET_REAL_MODEL_TESTS=1 to run the real-model golden test",
+)
+
+
+@requires_real_model
+def test_cached_distributed_generation_matches_stateless_golden():
+    pytest.importorskip("torch")
+    from meshnet_node.model_backend import TorchModelShard
+
+    head = TorchModelShard(_GOLDEN_MODEL, 0, 11)
+    tail = TorchModelShard(_GOLDEN_MODEL, 12, 23)
+    steps = 12
+    prompt = head.tokenizer.apply_chat_template(
+        [{"role": "user", "content": "Count from 1 to 5."}],
+        add_generation_prompt=True,
+        tokenize=False,
+    )
+
+    # Reference: today's stateless path — re-encode the full sequence each step.
+    stateless_ids: list[int] = []
+    text = prompt
+    for _ in range(steps):
+        payload = head.encode_prompt(text)
+        result = tail.forward_bytes(
+            payload.body, payload.shape,
+            payload.attention_mask_header, payload.position_ids_header,
+            start_layer=12,
+        )
+        stateless_ids.append(result.token_id)
+        text += result.text
+
+    # Cached path: one prefill, then single-token decode steps.
+    session = "golden-session"
+    cached_ids: list[int] = []
+    payload = head.encode_prompt(prompt, session_id=session)
+    result = tail.forward_bytes(
+        payload.body, payload.shape,
+        payload.attention_mask_header, payload.position_ids_header,
+        start_layer=12, session_id=session, cache_mode="prefill",
+    )
+    cached_ids.append(result.token_id)
+    for _ in range(steps - 1):
+        payload = head.encode_next_token(cached_ids[-1], session)
+        assert payload.shape[1] == 1, "decode payload must be a single token"
+        result = tail.forward_bytes(
+            payload.body, payload.shape,
+            None, payload.position_ids_header,
+            start_layer=12, session_id=session, cache_mode="decode",
+            past_len=payload.past_len,
+        )
+        cached_ids.append(result.token_id)
+
+    assert cached_ids == stateless_ids

tests/test_mining_cli.py

@@ -610,3 +610,44 @@ def test_default_cli_passes_advertise_host(monkeypatch):
     assert captured["tracker_url"] == "http://192.168.0.179:8081"
     assert captured["advertise_host"] == "192.168.0.42"
     assert captured["debug"] is True
+
+
+def test_default_cli_passes_force_cpu(monkeypatch):
+    """`meshnet-node --cpu` forwards force_cpu into run_startup."""
+    from meshnet_node.cli import main
+
+    captured = {}
+
+    def fake_run_startup(*args, **kwargs):
+        captured.update(kwargs)
+
+        class _FakeNode:
+            chat_completion_count = 0
+
+            def stop(self):
+                pass
+
+        return _FakeNode()
+
+    saved = {
+        "tracker_url": "http://localhost:8080",
+        "model_name": "stub-model",
+        "model_hf_repo": "",
+        "quantization": "auto",
+        "download_dir": "/tmp/models",
+        "wallet_path": "/tmp/wallet.json",
+        "port": 7000,
+        "host": "0.0.0.0",
+    }
+
+    monkeypatch.setattr(sys, "argv", ["meshnet-node", "--cpu"])
+
+    with patch("meshnet_node.config.load_config", return_value=saved):
+        with patch("meshnet_node.startup.run_startup", side_effect=fake_run_startup):
+            with patch("meshnet_node.dashboard.run_dashboard", side_effect=KeyboardInterrupt):
+                try:
+                    main()
+                except SystemExit as exc:
+                    assert exc.code == 0
+
+    assert captured["force_cpu"] is True

tests/test_node_startup.py

@@ -34,6 +34,27 @@ from meshnet_tracker.server import TrackerServer
 # ---------------------------------------------------------------------------
 
 
+def test_with_forced_cpu_overrides_device_but_keeps_gpu_inventory():
+    """--cpu should register and run on CPU while preserving detected GPU metadata."""
+    import meshnet_node.hardware as hardware_mod
+
+    hw = hardware_mod.with_forced_cpu(
+        {
+            "device": "cuda",
+            "gpu_name": "NVIDIA GeForce RTX 4060",
+            "vram_mb": 8192,
+            "dedicated_vram_mb": 8192,
+            "shared_vram_mb": 0,
+            "ram_mb": 32768,
+            "cuda_available": True,
+        }
+    )
+    assert hw["device"] == "cpu"
+    assert hw["cuda_available"] is False
+    assert hw["gpu_name"] == "NVIDIA GeForce RTX 4060"
+    assert hw["vram_mb"] == 8192
+
+
 def test_detect_hardware_returns_valid_profile():
     """Hardware detection always returns a dict with required keys."""
     hw = detect_hardware()
@@ -128,6 +149,59 @@ def test_nvidia_smi_without_torch_cuda_keeps_cpu_execution(monkeypatch):
     assert hw["ram_mb"] == 80 * 1024
 
 
+def test_torch_rocm_inventory_is_reported_when_kernels_are_not_executable(monkeypatch):
+    """ROCm can expose GPU metadata even when this torch wheel cannot run kernels."""
+    import meshnet_node.hardware as hardware_mod
+
+    class FakeProps:
+        total_memory = 64 * 1024 * 1024 * 1024
+        gcnArchName = "gfx1151"
+
+    class FakeCuda:
+        @staticmethod
+        def is_available():
+            return True
+
+        @staticmethod
+        def device_count():
+            return 1
+
+        @staticmethod
+        def current_device():
+            return 0
+
+        @staticmethod
+        def get_device_name(_idx):
+            return "AMD Radeon 8060S"
+
+        @staticmethod
+        def get_device_properties(_idx):
+            return FakeProps()
+
+        @staticmethod
+        def synchronize():
+            raise AssertionError("synchronize should not run after empty() fails")
+
+    fake_torch = types.SimpleNamespace(
+        cuda=FakeCuda(),
+        empty=lambda *args, **kwargs: (_ for _ in ()).throw(
+            RuntimeError("HIP error: invalid device function")
+        ),
+    )
+
+    monkeypatch.setattr(hardware_mod, "_detect_ram_mb", lambda: 125 * 1024)
+    monkeypatch.setitem(sys.modules, "torch", fake_torch)
+
+    hw = hardware_mod.detect_hardware()
+
+    assert hw["device"] == "cpu"
+    assert hw["cuda_available"] is False
+    assert hw["gpu_name"] == "AMD Radeon 8060S"
+    assert hw["vram_mb"] == 64 * 1024
+    assert hw["shared_vram_mb"] == 64_000
+    assert hw["gcn_arch"] == "gfx1151"
+
+
 def test_memory_budget_uses_ram_for_cpu_and_shared_memory_for_cuda():
     assert _memory_budget("cpu", vram_mb=8192, ram_mb=80 * 1024, shared_vram_mb=40 * 1024) == (
         80 * 1024,
@@ -1118,7 +1192,7 @@ def test_real_model_startup_summary_shows_total_layers(tmp_path, monkeypatch, ca
     assert captured_registration["vram_bytes"] == 6144 * 1024 * 1024
     assert captured_registration["max_loaded_shards"] == 2
     output = capsys.readouterr().out
-    assert "Shard:        layers 0–23; 24 of 24" in output
+    assert "Shard:        layers 0–23 (24 of 24)" in output
     assert "Node ID:      node-test-123" in output
 
 
@@ -1273,6 +1347,7 @@ def test_public_tracker_model_node_registers_relay_metadata_from_tracker_url_onl
     output = capsys.readouterr().out
     assert "Relay advertised by tracker" in output
     assert "Cross-host pipeline hops WILL time out" not in output
+    assert f"  Relay:        {registered['relay_addr']}" in output
 
 
 def test_public_tracker_relay_suppresses_virtual_ip_warning(
@@ -1646,6 +1721,166 @@ def test_preset_model_startup_honors_pinned_shard_range(tmp_path, monkeypatch):
         tracker.stop()
 
 
+def test_preset_startup_rejects_pinned_shard_above_memory_budget(tmp_path, monkeypatch):
+    """Pinned layer ranges that exceed the node memory budget fail before model load."""
+    import meshnet_node.startup as startup_mod
+
+    monkeypatch.setattr(
+        startup_mod,
+        "detect_hardware",
+        lambda: {"device": "cpu", "gpu_name": None, "vram_mb": 0, "ram_mb": 8 * 1024},
+    )
+
+    tracker = TrackerServer(model_presets={
+        "big-model": {
+            "layers_start": 0,
+            "layers_end": 39,
+            "hf_repo": "org/big-model",
+            "bytes_per_layer": {"bfloat16": 2 * 1024 * 1024 * 1024},
+        },
+    })
+    tracker_port = tracker.start()
+    tracker_url = f"http://127.0.0.1:{tracker_port}"
+    try:
+        with pytest.raises(ValueError, match="Pinned shard layers 0–39"):
+            run_startup(
+                tracker_url=tracker_url,
+                model="big-model",
+                shard_start=0,
+                shard_end=39,
+                wallet_path=tmp_path / "wallet.json",
+                cache_dir=tmp_path / "shards",
+            )
+    finally:
+        tracker.stop()
+
+
+def test_network_auto_join_clips_oversized_cpu_assignment(tmp_path, monkeypatch, capsys):
+    """Old trackers may assign too many CPU layers; node clips before model load."""
+    import meshnet_node.startup as startup_mod
+
+    torch_calls: list[dict] = []
+    registrations: list[dict] = []
+
+    class FakeBackend:
+        total_layers = 40
+
+    class FakeTorchNodeServer:
+        def __init__(self, **kwargs):
+            torch_calls.append(kwargs)
+            self.backend = FakeBackend()
+            self.tracker_node_id = None
+
+        def start(self):
+            return 7000
+
+        def stop(self):
+            pass
+
+    oversized_assignment = {
+        "hf_repo": "unsloth/Qwen3.6-35B-A3B",
+        "model": "qwen3.6-35b-a3b",
+        "shard_start": 0,
+        "shard_end": 36,
+        "num_layers": 40,
+        "gap_found": False,
+        "bytes_per_layer": {"bfloat16": 1_797_594_419},
+        "model_sources": [],
+    }
+
+    monkeypatch.setattr(
+        startup_mod,
+        "detect_hardware",
+        lambda: {"device": "cpu", "gpu_name": None, "vram_mb": 0, "ram_mb": 79 * 1024},
+    )
+    monkeypatch.setattr(startup_mod, "TorchNodeServer", FakeTorchNodeServer)
+    monkeypatch.setattr(startup_mod, "_get_json", lambda *_args, **_kwargs: oversized_assignment)
+    monkeypatch.setattr(startup_mod, "_post_json", lambda _url, payload: registrations.append(payload) or {"node_id": "n1"})
+    monkeypatch.setattr(startup_mod, "_start_heartbeat", lambda *_args, **_kwargs: None)
+    monkeypatch.setattr(startup_mod, "model_metadata_for", lambda *_args, **_kwargs: {"num_layers": 40})
+
+    node = run_startup(
+        tracker_url="http://127.0.0.1:8080",
+        wallet_path=tmp_path / "wallet.json",
+        tracker_source_disabled=True,
+    )
+    try:
+        assert torch_calls[0]["shard_start"] == 0
+        assert torch_calls[0]["shard_end"] == 24
+        assert registrations[0]["shard_end"] == 24
+        output = capsys.readouterr().out
+        assert "CPU-safe runtime budget fits 25/40 layers" in output
+        assert "layers 0-24" in output
+    finally:
+        node.stop()
+
+
+def test_preset_model_with_hf_repo_loads_torch_backend(tmp_path, monkeypatch, capsys):
+    """Named presets that advertise hf_repo must load TorchNodeServer, not the stub server."""
+    import meshnet_node.startup as startup_mod
+
+    class FakeBackend:
+        total_layers = 16
+
+    torch_calls: list[dict] = []
+
+    class FakeTorchNodeServer:
+        def __init__(self, **kwargs):
+            torch_calls.append(kwargs)
+            self.backend = FakeBackend()
+            self.port = None
+            self.chat_completion_count = 0
+            self.tracker_node_id = None
+
+        def start(self):
+            self.port = 7002
+            return self.port
+
+        def stop(self):
+            pass
+
+    monkeypatch.setattr(
+        startup_mod,
+        "detect_hardware",
+        lambda: {"device": "cpu", "gpu_name": None, "vram_mb": 0, "ram_mb": 16 * 1024},
+    )
+    monkeypatch.setattr(startup_mod, "TorchNodeServer", FakeTorchNodeServer)
+    monkeypatch.setattr(startup_mod, "StubNodeServer", lambda **_kw: (_ for _ in ()).throw(AssertionError("preset with hf_repo must not use StubNodeServer")))
+
+    model_dir = tmp_path / "node-shards" / "tiny-llama"
+    model_dir.mkdir(parents=True)
+    (model_dir / "config.json").write_text('{"num_hidden_layers": 16}')
+    monkeypatch.setattr(startup_mod, "download_shard", lambda *_a, **_kw: model_dir)
+
+    tracker = TrackerServer(model_presets={
+        "tiny-llama": {"layers_start": 0, "layers_end": 15, "hf_repo": "org/tiny-llama-shards"}
+    })
+    tracker_port = tracker.start()
+    tracker_url = f"http://127.0.0.1:{tracker_port}"
+    try:
+        node = run_startup(
+            tracker_url=tracker_url,
+            model="tiny-llama",
+            wallet_path=tmp_path / "wallet.json",
+            cache_dir=tmp_path / "node-shards",
+        )
+        try:
+            assert len(torch_calls) == 1
+            assert torch_calls[0]["model_id"] == "org/tiny-llama-shards"
+            assert torch_calls[0]["cache_dir"] == model_dir
+            output = capsys.readouterr().out
+            assert "Loading real PyTorch model shard..." in output
+            assert "Model ID:     org/tiny-llama-shards" in output
+            network_map = _get_json(f"{tracker_url}/v1/network/map")
+            registered = network_map["nodes"][0]
+            assert registered["hf_repo"] == "org/tiny-llama-shards"
+            assert registered["num_layers"] == 16
+        finally:
+            node.stop()
+    finally:
+        tracker.stop()
+
+
 def test_torch_startup_retries_registration_when_tracker_unreachable(
     tmp_path,
     monkeypatch,
@@ -1896,6 +2131,55 @@ def test_network_assign_gap_found_field():
         tracker.stop()
 
 
+def test_network_assign_uses_conservative_cpu_runtime_budget():
+    """CPU assignments leave headroom for partial-load overhead, not just raw weights."""
+    import json as _json
+    import urllib.request as _ur
+
+    tracker = TrackerServer(model_presets={
+        "qwen3.6-35b-a3b": {
+            "hf_repo": "unsloth/Qwen3.6-35B-A3B",
+            "aliases": ["unsloth/Qwen3.6-35B-A3B"],
+            "layers_start": 0,
+            "layers_end": 39,
+            "recommended": True,
+            "bytes_per_layer": {"bfloat16": 1_797_594_419},
+        },
+    })
+    port = tracker.start()
+    try:
+        data = _json.dumps({
+            "endpoint": "http://127.0.0.1:9200",
+            "model": "qwen3.6-35b-a3b",
+            "hf_repo": "unsloth/Qwen3.6-35B-A3B",
+            "num_layers": 40,
+            "shard_start": 0,
+            "shard_end": 39,
+            "hardware_profile": {},
+            "score": 1.0,
+        }).encode()
+        req = _ur.Request(
+            f"http://127.0.0.1:{port}/v1/nodes/register",
+            data=data,
+            headers={"Content-Type": "application/json"},
+            method="POST",
+        )
+        with _ur.urlopen(req) as r:
+            r.read()
+
+        resp = _get_json(
+            f"http://127.0.0.1:{port}/v1/network/assign"
+            "?device=cpu&vram_mb=0&ram_mb=80896"
+            "&hf_repo=unsloth/Qwen3.6-35B-A3B"
+        )
+
+        assert resp["gap_found"] is False
+        assert resp["shard_start"] == 0
+        assert resp["shard_end"] == 24
+    finally:
+        tracker.stop()
+
+
 def test_route_finds_hf_model_across_two_nodes():
     """Tracker /v1/route returns ordered route for HF model even without a preset."""
     import json as _json

tests/test_real_model_backend.py

@@ -1,5 +1,6 @@
 """US-012 tests for the real PyTorch node backend."""
 
+from collections import OrderedDict
 import json
 import os
 from pathlib import Path
@@ -14,13 +15,16 @@ import pytest
 from meshnet_node.model_backend import (
     InsufficientVRAMError,
     PartialModelLoadUnsupported,
+    ShardCacheMiss,
     TensorPayload,
     TorchModelShard,
     _call_layer,
+    _checkpoint_tensor_name_for_model,
     _load_partial_model_from_snapshot,
     _should_partial_materialize_shard,
     _decoder_attention_mask,
     _int_tensor_header,
+    _torch_cuda_is_executable,
     build_quantization_config,
     validate_quantization,
 )
@@ -42,7 +46,15 @@ class _FakeBackend:
             position_ids_header=None,
         )
 
-    def forward_bytes(self, body, shape, attention_mask_header, position_ids_header, start_layer=None):
+    def forward_bytes(
+        self,
+        body,
+        shape,
+        attention_mask_header,
+        position_ids_header,
+        start_layer=None,
+        **kwargs,  # noqa: ARG002
+    ):
         assert shape == [1, 6, 8]
         return TensorPayload(
             body=body,
@@ -56,7 +68,15 @@ class _FakeTailBackend(_FakeBackend):
     is_head = False
     is_tail = True
 
-    def forward_bytes(self, body, shape, attention_mask_header, position_ids_header, start_layer=None):
+    def forward_bytes(
+        self,
+        body,
+        shape,
+        attention_mask_header,
+        position_ids_header,
+        start_layer=None,
+        **kwargs,  # noqa: ARG002
+    ):
         assert len(body) == 1 * 6 * 8 * 2
         return " Paris"
 
@@ -113,7 +133,15 @@ class _FakePipelineTailBackend(_FakeTailBackend):
     def __init__(self) -> None:
         self.start_layers: list[int | None] = []
 
-    def forward_bytes(self, body, shape, attention_mask_header, position_ids_header, start_layer=None):
+    def forward_bytes(
+        self,
+        body,
+        shape,
+        attention_mask_header,
+        position_ids_header,
+        start_layer=None,
+        **kwargs,  # noqa: ARG002
+    ):
         self.start_layers.append(start_layer)
         assert len(body) == 1 * 6 * 8 * 2
         return " token"
@@ -124,7 +152,15 @@ class _BlockingStreamingTailBackend(_FakeTailBackend):
         self._release = second_token_release
         self.calls = 0
 
-    def forward_bytes(self, body, shape, attention_mask_header, position_ids_header, start_layer=None):
+    def forward_bytes(
+        self,
+        body,
+        shape,
+        attention_mask_header,
+        position_ids_header,
+        start_layer=None,
+        **kwargs,  # noqa: ARG002
+    ):
         self.calls += 1
         if self.calls == 1:
             return " first"
@@ -174,6 +210,26 @@ def test_bitsandbytes_configs_are_created_lazily(monkeypatch):
     ]
 
 
+def test_rocm_inventory_without_executable_kernels_is_not_used_as_cuda():
+    class FakeCuda:
+        @staticmethod
+        def is_available():
+            return True
+
+        @staticmethod
+        def synchronize():
+            raise AssertionError("synchronize should not run after empty() fails")
+
+    fake_torch = types.SimpleNamespace(
+        cuda=FakeCuda(),
+        empty=lambda *args, **kwargs: (_ for _ in ()).throw(
+            RuntimeError("HIP error: invalid device function")
+        ),
+    )
+
+    assert _torch_cuda_is_executable(fake_torch) is False
+
+
 def test_head_forward_accepts_text_prompt_and_returns_bfloat16_activations():
     node = TorchNodeServer(backend=_FakeBackend())
     port = node.start()
@@ -225,7 +281,7 @@ def test_tail_forward_returns_text_completion_from_binary_activations():
         node.stop()
 
 
-def test_full_model_chat_completion_uses_generation_not_single_token_decode():
+def test_full_model_chat_completion_uses_generation_not_single_token_decode(capsys):
     node = TorchNodeServer(backend=_FakeFullBackend())
     port = node.start()
     try:
@@ -237,7 +293,10 @@ def test_full_model_chat_completion_uses_generation_not_single_token_decode():
         req = urllib.request.Request(
             f"http://127.0.0.1:{port}/v1/chat/completions",
             data=payload,
-            headers={"Content-Type": "application/json"},
+            headers={
+                "Content-Type": "application/json",
+                "X-Meshnet-Request-Id": "req-test-123",
+            },
             method="POST",
         )
         with urllib.request.urlopen(req, timeout=5) as resp:
@@ -248,6 +307,10 @@ def test_full_model_chat_completion_uses_generation_not_single_token_decode():
     finally:
         node.stop()
 
+    out = capsys.readouterr().out
+    assert "  [node] processing chat model='fake-model' stream=False max_tokens=7 request_id=req-test-123" in out
+    assert "  [node] chat complete tokens=1 elapsed_s=" in out
+
 
 def test_pipeline_hop_logs_are_suppressed_without_debug(capsys):
     tail_backend = _FakePipelineTailBackend()
@@ -368,6 +431,92 @@ def test_split_shard_chat_streams_each_generated_token_incrementally():
     assert "data: [DONE]" in rest
 
 
+def test_current_requests_snapshot_while_generating():
+    release_second = threading.Event()
+    head = TorchNodeServer(backend=_FakePipelineHeadBackend(), tracker_mode=True)
+    tail = TorchNodeServer(backend=_BlockingStreamingTailBackend(release_second))
+    head_port = head.start()
+    tail_port = tail.start()
+    response = None
+    try:
+        payload = json.dumps({
+            "model": "fake-model",
+            "messages": [{"role": "user", "content": "hello"}],
+            "stream": True,
+            "max_tokens": 2,
+        }).encode()
+        req = urllib.request.Request(
+            f"http://127.0.0.1:{head_port}/v1/chat/completions",
+            data=payload,
+            headers={
+                "Content-Type": "application/json",
+                "X-Meshnet-Request-Id": "req-live-1",
+                "X-Meshnet-Route": json.dumps([
+                    {"endpoint": f"http://127.0.0.1:{tail_port}", "start_layer": 22},
+                ]),
+            },
+            method="POST",
+        )
+        response = urllib.request.urlopen(req, timeout=5)
+        deadline = time.time() + 2.0
+        while time.time() < deadline:
+            live = head.current_requests
+            if live and live[0]["request_id"] == "req-live-1" and live[0]["tokens"] >= 1:
+                break
+            time.sleep(0.02)
+        assert head.current_requests
+        snap = head.current_requests[0]
+        assert snap["request_id"] == "req-live-1"
+        assert snap["tokens"] >= 1
+        assert snap["tokens_per_sec"] >= 0
+        assert snap["routing_complete"] is True
+        release_second.set()
+        response.read()
+    finally:
+        release_second.set()
+        if response is not None:
+            response.close()
+        head.stop()
+        tail.stop()
+
+    assert head.current_requests == []
+
+
+def test_distributed_generating_log_includes_tps(capsys):
+    head = TorchNodeServer(backend=_FakePipelineHeadBackend(), tracker_mode=True)
+    tail = TorchNodeServer(backend=_FakePipelineTailBackend())
+    head_port = head.start()
+    tail_port = tail.start()
+    try:
+        payload = json.dumps({
+            "model": "fake-model",
+            "messages": [{"role": "user", "content": "hello"}],
+            "max_tokens": 1,
+        }).encode()
+        req = urllib.request.Request(
+            f"http://127.0.0.1:{head_port}/v1/chat/completions",
+            data=payload,
+            headers={
+                "Content-Type": "application/json",
+                "X-Meshnet-Route": json.dumps([
+                    {"endpoint": f"http://127.0.0.1:{tail_port}", "start_layer": 22},
+                ]),
+            },
+            method="POST",
+        )
+        with urllib.request.urlopen(req, timeout=5) as resp:
+            json.loads(resp.read())
+    finally:
+        head.stop()
+        tail.stop()
+
+    out = capsys.readouterr().out
+    assert "generating step=1/1" in out
+    assert " tps=" in out
+    assert "generation complete tokens=1" in out
+    assert out.count("generating step=1/1") == 1
+
+
 def test_int_tensor_header_serializes_torch_tensors():
     torch = pytest.importorskip("torch")
 
@@ -394,13 +543,118 @@ def test_call_layer_passes_rotary_position_embeddings():
             assert kwargs["position_embeddings"] == "rotary"
             return hidden_states
 
-    assert _call_layer(
+    hidden, cache_state = _call_layer(
         NeedsPositionEmbeddings(),
         "hidden",
         attention_mask=None,
         position_ids="positions",
         position_embeddings="rotary",
-    ) == "hidden"
+    )
+
+    assert hidden == "hidden"
+    assert cache_state is None
+
+
+def _fake_cache_shard(torch, *, max_sessions=16, ttl=600.0):
+    class RecordingLayer:
+        def __init__(self, index):
+            self.index = index
+            self.calls = []
+
+        def __call__(self, hidden_states, **kwargs):
+            self.calls.append({
+                "shape": tuple(hidden_states.shape),
+                "use_cache": kwargs.get("use_cache"),
+                "past_key_value": kwargs.get("past_key_value"),
+            })
+            present = {
+                "layer": self.index,
+                "shape": tuple(hidden_states.shape),
+                "opaque": object(),
+            }
+            return hidden_states + (self.index + 1), present
+
+    shard = object.__new__(TorchModelShard)
+    shard.shard_start = 0
+    shard.shard_end = 1
+    shard.torch = torch
+    shard.model = types.SimpleNamespace(model=types.SimpleNamespace(layers=[]))
+    shard.layers = [RecordingLayer(0), RecordingLayer(1)]
+    shard._session_cache = OrderedDict()
+    shard._cache_max_sessions = max_sessions
+    shard._cache_ttl_seconds = ttl
+    return shard
+
+
+def test_shard_cache_prefill_then_decode_reuses_opaque_layer_state():
+    torch = pytest.importorskip("torch")
+    shard = _fake_cache_shard(torch)
+
+    prefill_hidden = torch.zeros((1, 4, 2), dtype=torch.bfloat16)
+    prefill_mask = torch.ones((1, 4), dtype=torch.long)
+    prefill_positions = torch.arange(4, dtype=torch.long).reshape(1, 4)
+    shard._run_layers(
+        prefill_hidden,
+        prefill_mask,
+        prefill_positions,
+        session_id="session-1",
+        cache_mode="prefill",
+        seq_len=4,
+    )
+
+    assert len(shard._session_cache) == 1
+    cached_states = next(iter(shard._session_cache.values())).layer_states
+    assert len(cached_states) == 2
+    assert cached_states[0]["shape"] == (1, 4, 2)
+
+    decode_hidden = torch.zeros((1, 1, 2), dtype=torch.bfloat16)
+    decode_mask = torch.ones((1, 5), dtype=torch.long)
+    decode_positions = torch.tensor([[4]], dtype=torch.long)
+    shard._run_layers(
+        decode_hidden,
+        decode_mask,
+        decode_positions,
+        session_id="session-1",
+        cache_mode="decode",
+        seq_len=5,
+    )
+
+    assert shard.layers[0].calls[-1]["shape"] == (1, 1, 2)
+    assert shard.layers[0].calls[-1]["past_key_value"] is cached_states[0]
+    assert shard.layers[1].calls[-1]["past_key_value"] is cached_states[1]
+    assert next(iter(shard._session_cache.values())).seq_len == 5
+
+
+def test_shard_cache_decode_miss_is_explicit():
+    torch = pytest.importorskip("torch")
+    shard = _fake_cache_shard(torch)
+
+    with pytest.raises(ShardCacheMiss):
+        shard._run_layers(
+            torch.zeros((1, 1, 2), dtype=torch.bfloat16),
+            torch.ones((1, 5), dtype=torch.long),
+            torch.tensor([[4]], dtype=torch.long),
+            session_id="missing",
+            cache_mode="decode",
+            seq_len=5,
+        )
+
+
+def test_shard_cache_lru_bounds_sessions():
+    torch = pytest.importorskip("torch")
+    shard = _fake_cache_shard(torch, max_sessions=1)
+
+    for session in ("old", "new"):
+        shard._run_layers(
+            torch.zeros((1, 2, 2), dtype=torch.bfloat16),
+            torch.ones((1, 2), dtype=torch.long),
+            torch.arange(2, dtype=torch.long).reshape(1, 2),
+            session_id=session,
+            cache_mode="prefill",
+            seq_len=2,
+        )
+
+    assert list(shard._session_cache.keys()) == [("new", 0, 1)]
 
 
 def test_partial_materialize_guard_requires_local_non_full_non_quantized_snapshot(tmp_path):
@@ -422,7 +676,7 @@ def test_partial_materialize_guard_requires_local_non_full_non_quantized_snapsho
         39,
         total_layers_hint=40,
         uses_quantized_weights=False,
-    ) is False
+    ) is True
     assert _should_partial_materialize_shard(
         str(snapshot_dir),
         4,
@@ -439,6 +693,208 @@ def test_partial_materialize_guard_requires_local_non_full_non_quantized_snapsho
     ) is False
 
 
+def test_checkpoint_tensor_name_remapped_for_text_only_causal_lm():
+    class TextOnlyModel:
+        def __init__(self):
+            self.model = types.SimpleNamespace(layers=[])
+
+    model = TextOnlyModel()
+    assert _checkpoint_tensor_name_for_model(
+        model,
+        "model.language_model.layers.0.mlp.gate.weight",
+    ) == "model.layers.0.mlp.gate.weight"
+    assert _checkpoint_tensor_name_for_model(
+        model,
+        "model.language_model.embed_tokens.weight",
+    ) == "model.embed_tokens.weight"
+
+
+def test_checkpoint_tensor_name_kept_for_multimodal_backbone():
+    class MultimodalModel:
+        def __init__(self):
+            self.model = types.SimpleNamespace(language_model=types.SimpleNamespace())
+
+    model = MultimodalModel()
+    name = "model.language_model.layers.0.mlp.gate.weight"
+    assert _checkpoint_tensor_name_for_model(model, name) == name
+
+
+def test_partial_snapshot_loader_remaps_language_model_checkpoint_keys(tmp_path):
+    snapshot_dir = tmp_path / "snapshot"
+    snapshot_dir.mkdir()
+    (snapshot_dir / "config.json").write_text(json.dumps({
+        "text_config": {"num_hidden_layers": 3},
+    }))
+    (snapshot_dir / "model.safetensors.index.json").write_text(json.dumps({
+        "weight_map": {
+            "model.language_model.layers.1.self_attn.q_proj.weight": "shard-2.safetensors",
+        }
+    }))
+    (snapshot_dir / "shard-2.safetensors").write_bytes(b"stub")
+
+    class FakeModule:
+        def __init__(self):
+            self.to_calls = []
+
+        def to(self, device):
+            self.to_calls.append(device)
+            return self
+
+    class FakeModel:
+        def __init__(self):
+            self.model = types.SimpleNamespace(
+                layers=[FakeModule(), FakeModule(), FakeModule()],
+                rotary_emb=FakeModule(),
+            )
+
+        def tie_weights(self):
+            pass
+
+    class AutoConfigStub:
+        @staticmethod
+        def from_pretrained(model_id):
+            return types.SimpleNamespace(
+                text_config=types.SimpleNamespace(num_hidden_layers=3),
+                get_text_config=lambda: types.SimpleNamespace(num_hidden_layers=3),
+            )
+
+    class AutoModelStub:
+        @staticmethod
+        def from_config(cfg, torch_dtype=None):
+            return FakeModel()
+
+    set_calls = []
+
+    def fake_set_tensor(module, tensor_name, device, value=None, dtype=None):
+        set_calls.append(tensor_name)
+
+    class FakeSafeOpen:
+        def __init__(self, filename, framework, device):
+            self.filename = Path(filename).name
+
+        def __enter__(self):
+            return self
+
+        def __exit__(self, exc_type, exc, tb):
+            return False
+
+        def get_tensor(self, tensor_name):
+            return tensor_name
+
+    class UnusedContext:
+        def __enter__(self):
+            return None
+
+        def __exit__(self, exc_type, exc, tb):
+            return False
+
+    _load_partial_model_from_snapshot(
+        AutoConfigStub,
+        AutoModelStub,
+        types.SimpleNamespace(),
+        str(snapshot_dir),
+        1,
+        1,
+        "bf16",
+        "cpu:0",
+        init_empty_weights_fn=UnusedContext,
+        set_tensor_fn=fake_set_tensor,
+        safe_open_fn=FakeSafeOpen,
+    )
+
+    assert set_calls == ["model.layers.1.self_attn.q_proj.weight"]
+
+
+def test_partial_snapshot_loader_skips_tensors_absent_from_causal_lm(tmp_path):
+    # Multimodal/MTP checkpoints (Qwen3.5/3.6-MoE) carry mtp.* and model.visual.*
+    # tensors that the text-only CausalLM never builds — they must be skipped,
+    # not assigned (assignment raises AttributeError: 'mtp' / 'visual').
+    snapshot_dir = tmp_path / "snapshot"
+    snapshot_dir.mkdir()
+    (snapshot_dir / "config.json").write_text(json.dumps({
+        "text_config": {"num_hidden_layers": 3},
+    }))
+    (snapshot_dir / "model.safetensors.index.json").write_text(json.dumps({
+        "weight_map": {
+            "model.language_model.layers.1.self_attn.q_proj.weight": "shard-2.safetensors",
+            "mtp.layers.1.input_layernorm.weight": "shard-2.safetensors",
+            "model.visual.blocks.1.attn.qkv.weight": "shard-2.safetensors",
+        }
+    }))
+    (snapshot_dir / "shard-2.safetensors").write_bytes(b"stub")
+
+    class FakeModule:
+        def to(self, device):
+            return self
+
+    class FakeModel:
+        def __init__(self):
+            self.model = types.SimpleNamespace(
+                layers=[FakeModule(), FakeModule(), FakeModule()],
+                rotary_emb=FakeModule(),
+            )
+
+        def tie_weights(self):
+            pass
+
+        def state_dict(self):
+            return {"model.layers.1.self_attn.q_proj.weight": None}
+
+    class AutoConfigStub:
+        @staticmethod
+        def from_pretrained(model_id):
+            return types.SimpleNamespace(
+                text_config=types.SimpleNamespace(num_hidden_layers=3),
+                get_text_config=lambda: types.SimpleNamespace(num_hidden_layers=3),
+            )
+
+    class AutoModelStub:
+        @staticmethod
+        def from_config(cfg, torch_dtype=None):
+            return FakeModel()
+
+    set_calls = []
+
+    def fake_set_tensor(module, tensor_name, device, value=None, dtype=None):
+        set_calls.append(tensor_name)
+
+    class FakeSafeOpen:
+        def __init__(self, filename, framework, device):
+            pass
+
+        def __enter__(self):
+            return self
+
+        def __exit__(self, exc_type, exc, tb):
+            return False
+
+        def get_tensor(self, tensor_name):
+            return tensor_name
+
+    class UnusedContext:
+        def __enter__(self):
+            return None
+
+        def __exit__(self, exc_type, exc, tb):
+            return False
+
+    _load_partial_model_from_snapshot(
+        AutoConfigStub,
+        AutoModelStub,
+        types.SimpleNamespace(),
+        str(snapshot_dir),
+        1,
+        1,
+        "bf16",
+        "cpu:0",
+        init_empty_weights_fn=UnusedContext,
+        set_tensor_fn=fake_set_tensor,
+        safe_open_fn=FakeSafeOpen,
+    )
+
+    assert set_calls == ["model.layers.1.self_attn.q_proj.weight"]
+
+
 def test_partial_snapshot_loader_materializes_only_assigned_tensors(tmp_path):
     snapshot_dir = tmp_path / "snapshot"
     snapshot_dir.mkdir()

tests/test_tracker_as_node.py

@@ -142,6 +142,21 @@ def _send_chat_request(gateway_url: str, prompt: str) -> dict:
         return json.loads(r.read())
 
 
+def _send_streaming_chat_request(gateway_url: str, prompt: str):
+    data = json.dumps({
+        "model": GPT2_MODEL,
+        "messages": [{"role": "user", "content": prompt}],
+        "stream": True,
+    }).encode()
+    req = urllib.request.Request(
+        f"{gateway_url}/v1/chat/completions",
+        data=data,
+        headers={"Content-Type": "application/json"},
+        method="POST",
+    )
+    return urllib.request.urlopen(req)
+
+
 def test_all_responses_valid_openai_format(tracker_node_setup):
     """Ten requests via gateway all return valid OpenAI chat completion format."""
     gateway_url, _, _ = tracker_node_setup
@@ -155,6 +170,30 @@ def test_all_responses_valid_openai_format(tracker_node_setup):
         assert isinstance(message.get("content"), str), f"request {i}: content must be a string"
 
 
+def test_streaming_head_worker_response_is_not_buffered_with_content_length(tracker_node_setup):
+    """Gateway must relay head-worker SSE as a live stream, not a buffered JSON-sized body."""
+    gateway_url, _, _ = tracker_node_setup
+
+    with _send_streaming_chat_request(gateway_url, "stream through head worker") as resp:
+        assert resp.status == 200
+        assert "text/event-stream" in resp.headers["Content-Type"]
+        assert "Content-Length" not in resp.headers
+        data_lines = []
+        while len(data_lines) < 4:
+            line = resp.readline().decode().strip()
+            if line.startswith("data: "):
+                data_lines.append(line)
+            if line == "data: [DONE]":
+                break
+
+    assert data_lines[-1] == "data: [DONE]"
+    content = "".join(
+        json.loads(line[6:])["choices"][0].get("delta", {}).get("content", "")
+        for line in data_lines[:-1]
+    )
+    assert "head-worker" in content
+
+
 def test_both_tracker_nodes_receive_load(tracker_node_setup):
     """Both head workers handle at least one request each out of ten."""
     gateway_url, tracker_node_a, tracker_node_b = tracker_node_setup

tests/test_tracker_logging.py

@@ -0,0 +1,67 @@
+import logging
+import sys
+
+from meshnet_tracker.logging_setup import configure_tracker_file_logging, tracker_logger
+
+
+def test_tracker_file_logging_writes_separate_level_files(tmp_path):
+    original_stdout = sys.stdout
+    original_stderr = sys.stderr
+    try:
+        log_dir = configure_tracker_file_logging(tmp_path, tee_stdio=False)
+        logger = tracker_logger()
+
+        logger.info("info-event")
+        logger.warning("warning-event")
+        logger.error("error-event")
+        for handler in logger.handlers:
+            handler.flush()
+
+        assert (log_dir / "info.log").read_text().count("info-event") == 1
+        assert "warning-event" not in (log_dir / "info.log").read_text()
+        assert "error-event" not in (log_dir / "info.log").read_text()
+
+        assert "warning-event" in (log_dir / "warning.log").read_text()
+        assert "info-event" not in (log_dir / "warning.log").read_text()
+        assert "error-event" not in (log_dir / "warning.log").read_text()
+
+        assert "error-event" in (log_dir / "error.log").read_text()
+        assert "info-event" not in (log_dir / "error.log").read_text()
+        assert "warning-event" not in (log_dir / "error.log").read_text()
+    finally:
+        sys.stdout = original_stdout
+        sys.stderr = original_stderr
+
+
+def test_tracker_file_logging_tees_stdio_and_rotates(tmp_path):
+    original_stdout = sys.stdout
+    original_stderr = sys.stderr
+    try:
+        log_dir = configure_tracker_file_logging(
+            tmp_path,
+            max_bytes=120,
+            backup_count=1,
+        )
+
+        print("stdout goes to info", flush=True)
+        print("stderr goes to error", file=sys.stderr, flush=True)
+        for handler in tracker_logger().handlers:
+            handler.flush()
+
+        assert "stdout goes to info" in (log_dir / "info.log").read_text()
+        assert "stderr goes to error" in (log_dir / "error.log").read_text()
+
+        for index in range(12):
+            tracker_logger().info("rotating-info-line-%02d", index)
+        for handler in tracker_logger().handlers:
+            handler.flush()
+
+        assert (log_dir / "info.log.1").exists()
+    finally:
+        sys.stdout = original_stdout
+        sys.stderr = original_stderr
+        logger = tracker_logger()
+        for handler in logger.handlers:
+            handler.close()
+        logger.handlers.clear()
+        logger.setLevel(logging.NOTSET)

tests/test_tracker_routing.py

@@ -5,6 +5,7 @@ import json
 import threading
 import time
 import urllib.error
+import urllib.parse
 import urllib.request
 
 import pytest
@@ -466,6 +467,7 @@ def test_tracker_logs_stream_progress_before_request_completes():
             method="POST",
         )
         response = urllib.request.urlopen(req, timeout=3.0)
+        assert response.headers["X-Accel-Buffering"] == "no"
         first_line = response.readline()
         assert first_line.startswith(b"data:")
         assert chunk_sent.wait(timeout=1.0)
@@ -501,6 +503,169 @@ def test_tracker_logs_stream_progress_before_request_completes():
         node_thread.join(timeout=1.0)
 
 
+def test_tracker_stream_survives_idle_gap_between_sse_chunks():
+    first_chunk_sent = threading.Event()
+
+    class IdleStreamingChatHandler(http.server.BaseHTTPRequestHandler):
+        def log_message(self, format, *args):
+            pass
+
+        def do_POST(self):
+            if self.path != "/v1/chat/completions":
+                self.send_response(404)
+                self.end_headers()
+                return
+            self.rfile.read(int(self.headers.get("Content-Length", 0)))
+            self.send_response(200)
+            self.send_header("Content-Type", "text/event-stream; charset=utf-8")
+            self.end_headers()
+            first = json.dumps({
+                "choices": [{"delta": {"content": "hello"}}],
+            }).encode()
+            second = json.dumps({
+                "choices": [{"delta": {"content": " world"}}],
+            }).encode()
+            self.wfile.write(b"data: " + first + b"\n\n")
+            self.wfile.flush()
+            first_chunk_sent.set()
+            time.sleep(1.0)
+            self.wfile.write(b"data: " + second + b"\n\n")
+            self.wfile.write(b"data: [DONE]\n\n")
+            self.wfile.flush()
+
+    node = http.server.HTTPServer(("127.0.0.1", 0), IdleStreamingChatHandler)
+    node_thread = threading.Thread(target=node.serve_forever, daemon=True)
+    node_thread.start()
+    tracker = TrackerServer(heartbeat_timeout=60.0)
+    tracker_port = tracker.start()
+    response = None
+    try:
+        _post_json(
+            f"http://127.0.0.1:{tracker_port}/v1/nodes/register",
+            {"endpoint": f"http://127.0.0.1:{node.server_address[1]}",
+             "model": "idle-stream-model", "num_layers": 1,
+             "shard_start": 0, "shard_end": 0,
+             "hardware_profile": {}, "score": 1.0},
+        )
+        req = urllib.request.Request(
+            f"http://127.0.0.1:{tracker_port}/v1/chat/completions",
+            data=json.dumps({
+                "model": "idle-stream-model",
+                "stream": True,
+                "messages": [{"role": "user", "content": "hi"}],
+            }).encode(),
+            headers={"Content-Type": "application/json"},
+            method="POST",
+        )
+        response = urllib.request.urlopen(req, timeout=3.0)
+        assert response.readline().startswith(b"data:")
+        assert first_chunk_sent.wait(timeout=1.0)
+
+        remaining = response.read().splitlines()
+        assert b"data: [DONE]" in remaining
+    finally:
+        if response is not None:
+            response.close()
+        tracker.stop()
+        node.shutdown()
+        node.server_close()
+        node_thread.join(timeout=1.0)
+
+
+def test_tracker_dashboard_can_cancel_inflight_proxy():
+    chunk_sent = threading.Event()
+    release = threading.Event()
+
+    class StreamingChatHandler(http.server.BaseHTTPRequestHandler):
+        def log_message(self, fmt, *args):
+            pass
+
+        def do_POST(self):
+            if self.path != "/v1/chat/completions":
+                self.send_response(404)
+                self.end_headers()
+                return
+            self.rfile.read(int(self.headers.get("Content-Length", 0)))
+            self.send_response(200)
+            self.send_header("Content-Type", "text/event-stream; charset=utf-8")
+            self.end_headers()
+            payload = json.dumps({
+                "choices": [{"delta": {"content": "hello world"}}],
+            }).encode()
+            self.wfile.write(b"data: " + payload + b"\n\n")
+            self.wfile.flush()
+            chunk_sent.set()
+            release.wait(timeout=3.0)
+            self.wfile.write(b"data: [DONE]\n\n")
+            self.wfile.flush()
+
+    node = http.server.HTTPServer(("127.0.0.1", 0), StreamingChatHandler)
+    node_thread = threading.Thread(target=node.serve_forever, daemon=True)
+    node_thread.start()
+    tracker = TrackerServer(heartbeat_timeout=60.0)
+    tracker_port = tracker.start()
+    response = None
+    request_id = None
+    try:
+        _post_json(
+            f"http://127.0.0.1:{tracker_port}/v1/nodes/register",
+            {"endpoint": f"http://127.0.0.1:{node.server_address[1]}",
+             "model": "cancel-proxy-model", "num_layers": 1,
+             "shard_start": 0, "shard_end": 0,
+             "hardware_profile": {}, "score": 1.0},
+        )
+        req = urllib.request.Request(
+            f"http://127.0.0.1:{tracker_port}/v1/chat/completions",
+            data=json.dumps({
+                "model": "cancel-proxy-model",
+                "stream": True,
+                "messages": [{"role": "user", "content": "hi"}],
+            }).encode(),
+            headers={"Content-Type": "application/json"},
+            method="POST",
+        )
+        response = urllib.request.urlopen(req, timeout=3.0)
+        first_line = response.readline()
+        assert first_line.startswith(b"data:")
+        assert chunk_sent.wait(timeout=1.0)
+
+        console = _get_json(f"http://127.0.0.1:{tracker_port}/v1/console")
+        selected = [
+            event for event in console["events"]
+            if event["message"] == "proxy route selected"
+        ]
+        assert selected
+        request_id = selected[-1]["fields"]["request_id"]
+
+        cancel = _post_json(
+            f"http://127.0.0.1:{tracker_port}/v1/proxy/requests/{urllib.parse.quote(request_id, safe='')}/cancel",
+            {},
+        )
+        assert cancel["status"] == "canceled"
+
+        deadline = time.time() + 5.0
+        canceled_events = []
+        while time.time() < deadline:
+            console = _get_json(f"http://127.0.0.1:{tracker_port}/v1/console")
+            canceled_events = [
+                event for event in console["events"]
+                if event["message"] == "proxy canceled"
+                and event["fields"].get("request_id") == request_id
+            ]
+            if canceled_events:
+                break
+            time.sleep(0.05)
+        assert canceled_events
+    finally:
+        release.set()
+        if response is not None:
+            response.close()
+        tracker.stop()
+        node.shutdown()
+        node.server_close()
+        node_thread.join(timeout=1.0)
+
+
 def test_tracker_routes_hf_model_alias_from_quickstart():
     """The documented qwen2.5-0.5b alias resolves a full HF repo registration."""
     tracker = TrackerServer()
@@ -746,6 +911,57 @@ def test_tracker_route_endpoint_ignores_model_case_and_outer_whitespace():
     assert response["route"] == ["http://127.0.0.1:9101"]
 
 
+def test_tracker_route_prefers_distributed_over_single_full_shard():
+    """When a full 0-39 node and a partial 0-21 head coexist, /v1/route
+    should return both hops — not the full shard alone."""
+    tracker = TrackerServer(model_presets={
+        "qwen3.6-35b-a3b": {
+            "layers_start": 0,
+            "layers_end": 39,
+            "hf_repo": "unsloth/Qwen3.6-35B-A3B",
+            "aliases": ["Qwen3.6-35B-A3B"],
+        }
+    })
+    tracker_port = tracker.start()
+    try:
+        _post_json(
+            f"http://127.0.0.1:{tracker_port}/v1/nodes/register",
+            {"endpoint": "http://192.168.0.179:7000",
+             "model": "qwen3.6-35b-a3b",
+             "hf_repo": "unsloth/Qwen3.6-35B-A3B",
+             "num_layers": 40,
+             "shard_start": 0,
+             "shard_end": 39,
+             "tracker_mode": True,
+             "hardware_profile": {},
+             "score": 1.0},
+        )
+        _post_json(
+            f"http://127.0.0.1:{tracker_port}/v1/nodes/register",
+            {"endpoint": "http://192.168.0.20:7000",
+             "model": "Qwen3.6-35B-A3B",
+             "hf_repo": "unsloth/Qwen3.6-35B-A3B",
+             "num_layers": 40,
+             "shard_start": 0,
+             "shard_end": 21,
+             "tracker_mode": True,
+             "hardware_profile": {},
+             "score": 1.0},
+        )
+
+        response = _get_json(
+            f"http://127.0.0.1:{tracker_port}/v1/route?model=qwen3.6-35b-a3b"
+        )
+    finally:
+        tracker.stop()
+
+    assert response["route"] == [
+        "http://192.168.0.20:7000",
+        "http://192.168.0.179:7000",
+    ]
+    assert [node["start_layer"] for node in response["nodes"]] == [0, 22]
+
+
 def test_tracker_proxy_ignores_model_case_and_outer_whitespace():
     class ChatHandler(http.server.BaseHTTPRequestHandler):
         def log_message(self, fmt, *args):
@@ -1403,6 +1619,70 @@ def test_tracker_heartbeat_updates_node():
         tracker.stop()
 
 
+def test_tracker_heartbeat_stores_current_requests():
+    """Node-reported in-flight request snapshots appear on the network map."""
+    tracker = TrackerServer()
+    tracker_port = tracker.start()
+    try:
+        reg = _post_json(
+            f"http://127.0.0.1:{tracker_port}/v1/nodes/register",
+            {
+                "endpoint": "http://127.0.0.1:9001",
+                "model": "progress-model",
+                "shard_start": 0,
+                "shard_end": 31,
+                "hardware_profile": {},
+                "score": 1.0,
+            },
+        )
+        node_id = reg["node_id"]
+        _post_json(
+            f"http://127.0.0.1:{tracker_port}/v1/nodes/{node_id}/heartbeat",
+            {
+                "queue_depth": 1,
+                "current_requests": [{
+                    "request_id": "req-abc123",
+                    "model": "progress-model",
+                    "kind": "chat",
+                    "tokens": 17,
+                    "elapsed_seconds": 42.5,
+                    "tokens_per_sec": 0.4,
+                    "routing_complete": True,
+                }],
+            },
+        )
+        network = _get_json(f"http://127.0.0.1:{tracker_port}/v1/network/map")
+        node = next(item for item in network["nodes"] if item["node_id"] == node_id)
+        assert node["stats"]["queue_depth"] == 1
+        assert node["stats"]["current_requests"] == [{
+            "request_id": "req-abc123",
+            "model": "progress-model",
+            "kind": "chat",
+            "tokens": 17,
+            "elapsed_seconds": 42.5,
+            "tokens_per_sec": 0.4,
+            "routing_complete": True,
+        }]
+    finally:
+        tracker.stop()
+
+
+def test_normalize_current_requests_sanitizes_payload():
+    from meshnet_tracker.server import _normalize_current_requests
+
+    assert _normalize_current_requests(None) == []
+    assert _normalize_current_requests([
+        {"request_id": "req-1", "model": "m", "tokens": "9", "tokens_per_sec": "1.5"},
+        {"model": "missing-id"},
+        "bad",
+    ]) == [{
+        "request_id": "req-1",
+        "model": "m",
+        "tokens": 9,
+        "tokens_per_sec": 1.5,
+    }]
+
+
 def test_tracker_heartbeat_expiry():
     """Nodes that miss their heartbeat window are excluded from routes."""
     tracker = TrackerServer(heartbeat_timeout=0.05)  # 50 ms