From e2b20883cadc6fa92c6c3d2c986a949e61a34cff Mon Sep 17 00:00:00 2001 From: Dobromir Popov Date: Tue, 7 Jul 2026 19:48:43 +0200 Subject: [PATCH 1/9] Stream chat responses in the dashboard with live progress and unified styles Chat now sends stream=true and renders SSE tokens incrementally with live tok/s status, a stop button (AbortController), and a blinking cursor; because streamed requests emit tracker 'proxy progress' events, the Call wall now shows in-flight requests with live TPS too. Chat colors route through :root tokens instead of hardcoded hex values. ADR-0020 documents the changes and the mixed-topology routing flaw: a partial GPU head (0-21) + full CPU node (0-39) gets downstream start_layer=0 instead of 22, corrupting activations into 1-token generations that were billed and polluted throughput stats. Fix steps recorded, not yet implemented. Co-Authored-By: Claude Fable 5 --- ...ive-progress-and-mixed-topology-routing.md | 127 ++++++++ .../tracker/meshnet_tracker/dashboard.html | 192 +++++++++--- tests/test_dashboard.py | 276 +++++++++--------- 3 files changed, 420 insertions(+), 175 deletions(-) create mode 100644 docs/adr/0020-chat-streaming-live-progress-and-mixed-topology-routing.md diff --git a/docs/adr/0020-chat-streaming-live-progress-and-mixed-topology-routing.md b/docs/adr/0020-chat-streaming-live-progress-and-mixed-topology-routing.md new file mode 100644 index 0000000..62fe4da --- /dev/null +++ b/docs/adr/0020-chat-streaming-live-progress-and-mixed-topology-routing.md @@ -0,0 +1,127 @@ +# ADR-0020: Dashboard chat streaming, live request progress, and the mixed-topology routing flaw + +## Status: Accepted (chat/streaming/styles implemented); routing flaw documented, fix pending + +## Context + +Live alpha testing (2026-07-07) with `Qwen3.6-35B-A3B` split across two LAN nodes surfaced +three UX gaps and one routing correctness flaw: + +1. **No visibility while a request is processing.** The Call wall showed + "no in-flight requests" during a 52-second generation. Cause: the dashboard chat sent + `stream: false`, and the tracker only emits `proxy progress` console events (the Call + wall's live-status source, `_tracker_log_proxy_progress`, `server.py` ~2199) for + **streamed** requests. Non-streamed proxying produces only + `route selected → connected → complete`, and short requests complete inside the + dashboard's 4-second poll window. +2. **Chat did not stream.** The nodes support SSE token-by-token generation + (`generate_text_streaming`, hardened earlier for split shards), and the tracker proxy + passes `text/event-stream` through (`server.py` ~3256), but the chat panel blocked on + full JSON and showed nothing until completion. +3. **Chat panel styles drifted.** The "new chat layout" redesign left hardcoded one-off + colors (`#1f4788`, `#2563b8`, `#10151d`, `#1a1012`, `#5c2020`, `#ffb4b4`) mixed with + the CSS custom-property palette. + +## Decisions + +### 1. Chat streams by default (SSE) + +`dashboard.html` `sendChat()` now sends `stream: true` and consumes the SSE body with a +`ReadableStream` reader: + +- Assistant tokens render incrementally into the last bubble (direct DOM update, full + re-render only at boundaries), with a blinking `▍` cursor while streaming. +- Chat status shows live progress: `generating… N tokens · X tok/s`. +- The send button becomes a stop button (`■`) during generation, backed by an + `AbortController`; a stopped generation keeps the partial text. +- Non-SSE responses (JSON fallback, errors) are still handled; `data: {"error": ...}` + stream events surface as error bubbles. +- `streaming` flags are stripped when loading persisted sessions so an interrupted + generation never leaves a stuck cursor. + +### 2. Live in-flight visibility rides on streaming + +No tracker change was needed: because chat now streams, the tracker emits `proxy progress` +events (throttled to stdout, updated in place in the console ring via +`update_console_key`), and the existing Call wall state machine +(`buildCallWallStates`) renders processing rows with live tokens/TPS/queue. + +**Known limitation (accepted):** non-streamed API requests still show no progress between +`proxy connected` and `proxy complete` — there is nothing to report until the node +returns. Callers wanting live visibility should use `stream: true`. + +### 3. Chat style tokens + +All chat colors route through `:root` custom properties (`--hover-bg`, `--chat-user-bg` +`#1f6feb`, `--chat-user-border`, `--chat-error-bg/border/fg`). No hardcoded hex values +remain in chat rules, so future palette changes are single-line edits. + +## Documented flaw: mixed-topology routing (partial GPU head + full CPU node) + +### Observed (2026-07-07, tracker 192.168.0.179:8080) + +Two nodes registered for `qwen3.6-35b-a3b`: + +| node | hardware | shard | benchmark | +|---|---|---|---| +| `5gMLrmyB-ec3afe6f1a03` (192.168.0.20) | RTX 4060, CUDA | 0–21 (partial, fast) | 11,164 | +| `7j77FsPY-55249b0583e5` (192.168.0.179) | CPU | 0–39 (full, slow) | 425 | + +When the tracker selected the GPU node as head, it injected: + +``` +downstream=[{"endpoint": "http://192.168.0.179:7000", "start_layer": 0}] +``` + +`start_layer: 0` — not 22. The downstream full node re-ran **all 40 layers from layer 0 +on hidden states that had already passed through the head's layers 0–21**, producing +garbage logits. Evidence from the logs: + +- GPU-headed requests: `generation complete tokens=1` and billed `out=0`/`out=1`/`out=3` + — near-instant EOS from corrupt activations. +- The same prompt routed directly to the CPU full node: 209 tokens over 52 s (healthy). +- Observed TPS for GPU-headed requests was meaningless (2.5–19.0 "tok/s" on 0–3 token + outputs), and those samples now pollute the rolling per-`(node, model)` throughput + stats used for routing preference. +- Clients were **billed** for these broken 1-token responses. + +### Root cause + +The route planner treats the full-coverage node as a standalone complete route +(`route=7j77FsPY…[0-39]`) but still injects it as the head's downstream with the +downstream node's own `shard_start` (0) instead of `head.shard_end + 1` (22). A partial +head + full-model downstream is a topology the planner never had to handle before — +prior split tests used disjoint shards (0–11 + 12–23) where `shard_start` happened to +equal the correct continuation layer. + +### Required fix (not yet implemented) + +1. **Correct continuation layer:** when hop N ends at layer `e`, hop N+1 must execute + from `start_layer = e + 1` regardless of the downstream node's own `shard_start` + (the `X-Meshnet-Start-Layer` overlapping-shard mechanism from ADR-0012 exists for + exactly this; the planner must set it for full-model downstream nodes too). +2. **Route preference sanity:** with a healthy single-node full route available, prefer + it over a multi-hop route unless the pipeline is estimated faster; a fast head that + forces a slow full-model tail wins nothing (every token still crosses the CPU node). +3. **Stat hygiene:** exclude or flag throughput samples from responses with ≤ a few + output tokens, so broken routes don't skew routing preference. +4. **Billing guard (consider):** suspiciously short completions from multi-hop routes + during this window were billed; a minimum-viability check (or refund path) may be + warranted once audits land. + +### Verification for the fix + +Reproduce with a partial GPU head (0–21) + full CPU node (0–39): a chat request routed +through the GPU head must produce output equivalent to the direct CPU route, with +`downstream start_layer=22` visible in `proxy route selected`, and multi-token streamed +output on the Call wall. + +## Verification of this ADR's implemented changes + +- `pytest tests/test_dashboard.py` — 5 passed (stale "Chat / inference" panel assertion + updated to the tabbed layout). +- Embedded dashboard JS parses (`new Function(script)` under Node 22). +- Live check: open `/dashboard` → Chat, send a prompt to `qwen3.6-35b-a3b` — tokens + must appear incrementally with live tok/s in the status line, the Call wall must show + the request as `processing` with live TPS, and the send button must stop generation + mid-stream keeping partial text. diff --git a/packages/tracker/meshnet_tracker/dashboard.html b/packages/tracker/meshnet_tracker/dashboard.html index faa07fb..72e16a2 100644 --- a/packages/tracker/meshnet_tracker/dashboard.html +++ b/packages/tracker/meshnet_tracker/dashboard.html @@ -7,7 +7,10 @@ - - -
-

meshnet tracker

- - -
- -
-

Account

loading…
-

Tracker hive

loading…
-

Nodes & coverage

loading…
-

Model usage (RPM)

loading…
-

Routing (learned)

loading…
-

Call wall

loading...
-
-

Chat / inference

-
- -
-
- -
select a model to start
-
-
-
Send a message to start this conversation.
-
-
-
- - -
-
-
-
-
-

Usage summary

login required
-

Node throughput

login required
-

Request history

login required
-

Node pending payouts

admin login required
-

Settlement history

admin login required
-

All accounts (admin)

-

Strikes / bans / forfeitures

admin login required
-

Client balances

admin login required
-

Console output

admin login required
-
- - - + + + + + +meshnet tracker + + + +
+

meshnet tracker

+ + +
+ +
+

Account

loading…
+

Tracker hive

loading…
+

Nodes & coverage

loading…
+

Model usage (RPM)

loading…
+

Routing (learned)

loading…
+

Call wall

loading...
+
+

Chat / inference

+
+ +
+
+ +
select a model to start
+
+
+
Send a message to start this conversation.
+
+
+
+ + +
+
+
+
+
+

Usage summary

login required
+

Node throughput

login required
+

Request history

login required
+

Node pending payouts

admin login required
+

Settlement history

admin login required
+

All accounts (admin)

+

Strikes / bans / forfeitures

admin login required
+

Client balances

admin login required
+

Console output

admin login required
+
+ + + diff --git a/packages/validator/meshnet_validator/__init__.py b/packages/validator/meshnet_validator/__init__.py index bf6c96b..b55b104 100644 --- a/packages/validator/meshnet_validator/__init__.py +++ b/packages/validator/meshnet_validator/__init__.py @@ -1,552 +1,552 @@ -"""Optimistic fraud validator for completed inference requests.""" - -import json -import math -import random -import threading -import time -import urllib.request -from typing import Any - -from .audit import ( - ToplocAuditConfig, - ToplocProofClaim, - ToplocVerificationResult, - verify_activation_proofs, - verify_activation_proofs_detailed, -) -from .sampling import AdaptiveAuditSampler, AuditRateConfig -from .tripwire import detect_output_tripwire - -__version__ = "0.1.0" - - -class ValidatorProcess: - """Separate validator loop that samples completed requests and submits slashes.""" - - def __init__( - self, - *, - contracts: Any, - reference_node_url: str, - sample_rate: float = 0.05, - tolerance: float = 1e-6, - slash_amount: int = 100, - strike_threshold: int = 3, - random_seed: int | None = None, - webhook_url: str | None = None, - interval_seconds: float = 1.0, - billing: Any | None = None, - toploc_config: ToplocAuditConfig | None = None, - toploc_backend: Any | None = None, - audit_sampler: AdaptiveAuditSampler | None = None, - ) -> None: - if not 0.0 <= sample_rate <= 1.0: - raise ValueError("sample_rate must be between 0 and 1") - if tolerance < 0: - raise ValueError("tolerance must be non-negative") - if slash_amount <= 0: - raise ValueError("slash_amount must be positive") - if strike_threshold <= 0: - raise ValueError("strike_threshold must be positive") - if interval_seconds <= 0: - raise ValueError("interval_seconds must be positive") - - self._contracts = contracts - self._billing = billing - self._reference_node_url = reference_node_url.rstrip("/") - self._sample_rate = sample_rate - self._tolerance = tolerance - self._slash_amount = slash_amount - self._strike_threshold = strike_threshold - self._webhook_url = webhook_url - self._interval_seconds = interval_seconds - self._toploc_config = toploc_config or ToplocAuditConfig() - self._toploc_backend = toploc_backend - self._audit_sampler = audit_sampler - self._random = random.Random(random_seed) - self._last_event_index = -1 - self._running = False - self._thread: threading.Thread | None = None - self.sampled_count = 0 - - def validate_once(self) -> list[Any]: - """Run one validation cycle and return slash receipts submitted this cycle.""" - receipts: list[Any] = [] - events = self._contracts.validation.list_completed_inferences( - after_index=self._last_event_index, - ) - for event in events: - self._last_event_index = max(self._last_event_index, event.index) - if not self._should_sample(event): - continue - self.sampled_count += 1 - audit_result = self._validate_event(event) - if audit_result.ok: - self._record_clean_audit(event) - continue - receipts.extend(self._slash_node( - audit_result.culprit_node, - event.observed_output, - audit_result.reference_output, - reason=audit_result.reason, - )) - return receipts - - def _should_sample(self, event: Any) -> bool: - """ADR-0018 §1/§6-7: flat sample_rate stays the default; when an - AdaptiveAuditSampler is configured, the decision is reputation- and - tenure-weighted and budget-balanced against the fleet-wide target - instead of a uniform coin flip.""" - if self._audit_sampler is None: - return self._random.random() < self._sample_rate - - tripwire = detect_output_tripwire(_event_value(event, "observed_output") or "") - wallets = _route_wallets(event) - if not wallets: - return self._audit_sampler.should_audit( - completed_job_count=0, reputation=1.0, tripwire=tripwire, - ) - # A route is only as trustworthy as its least-trusted hop -- audit - # against whichever wallet on the route looks riskiest. - riskiest = min( - (self._contracts.registry.get_wallet(wallet) for wallet in wallets), - key=lambda wallet: wallet.reputation, - ) - return self._audit_sampler.should_audit( - completed_job_count=riskiest.completed_job_count, - reputation=riskiest.reputation, - tripwire=tripwire, - ) - - def start(self) -> None: - if self._running: - raise RuntimeError("ValidatorProcess is already running") - self._running = True - self._thread = threading.Thread(target=self._run_loop, daemon=True) - self._thread.start() - - def stop(self) -> None: - self._running = False - if self._thread is not None: - self._thread.join(timeout=2) - self._thread = None - - def _run_loop(self) -> None: - while self._running: - self.validate_once() - time.sleep(self._interval_seconds) - - def _run_reference(self, messages: list[dict]) -> str: - response = _post_json( - f"{self._reference_node_url}/v1/infer", - {"messages": messages}, - ) - text = response.get("text") - if not isinstance(text, str): - raise ValueError("reference node response did not contain text") - return text - - def _validate_event(self, event: Any) -> "_AuditResult": - event = self._event_with_on_demand_commitments(event) - hop_commitments = _hop_commitments_from_event(event) - if hop_commitments is not None and self._commitment_expired(event): - # ADR-0018 §3: the on-demand retention window has passed — nodes - # are no longer expected to hold the boundary activations needed - # to verify this commitment, so fall back to the text-only path. - hop_commitments = None - - if hop_commitments is None: - reference_output = self._run_reference(event.messages) - ok = _outputs_match(event.observed_output, reference_output, self._tolerance) - return _AuditResult( - ok=ok, - reference_output=reference_output, - reason="reference output diverged", - # Text comparison has no per-hop signal; the last hop is the - # best-effort guess (text-only fallback), never used when - # hop-boundary commitments make real bisection possible. - culprit_node=None if ok else _final_text_node(event.route_nodes), - ) - - if len(hop_commitments) == 1: - # Single-commitment route (AH-006 whole-route format, or a - # genuinely one-hop pipeline): reuse the original teacher-forced - # call so existing single-hop reference integrations keep working. - only = hop_commitments[0] - reference_activations_by_hop = [self._run_teacher_forced_prefill( - model=_event_value(event, "model"), - messages=_event_value(event, "messages"), - claimed_token_ids=only.token_ids, - claim=only.claim, - )] - else: - reference_activations_by_hop = self._run_teacher_forced_prefill_hops( - model=_event_value(event, "model"), - messages=_event_value(event, "messages"), - hop_commitments=hop_commitments, - ) - culprit_index = _first_divergent_hop( - hop_commitments, - reference_activations_by_hop, - config=self._toploc_config, - backend=self._toploc_backend, - ) - ok = culprit_index is None - return _AuditResult( - ok=ok, - reference_output=( - "TOPLOC activation proof accepted" - if ok - else f"TOPLOC activation proof mismatch at hop {culprit_index}" - ), - reason="TOPLOC activation proof mismatch", - culprit_node=None if ok else hop_commitments[culprit_index].node, - ) - - def _event_with_on_demand_commitments(self, event: Any) -> Any: - """Fetch missing per-hop TOPLOC commitments only after audit sampling. - - Tracker validation events deliberately carry ordinary route metadata, - not a pre-announced audit flag. When this validator samples an event, it - asks each hop for its short-lived boundary commitment and splices the - returned proof into a local event copy for the bisection verifier. - """ - route_nodes = _event_value(event, "route_nodes") or [] - if not isinstance(route_nodes, list) or not route_nodes: - return event - updated_nodes: list[dict] = [] - changed = False - for node in route_nodes: - if not isinstance(node, dict): - updated_nodes.append(node) - continue - updated = dict(node) - if _mapping_value(updated, "toploc_proof") is None: - commitment = self._fetch_hop_commitment(event, updated) - if commitment is not None: - updated.update(commitment) - changed = True - updated_nodes.append(updated) - if not changed: - return event - if isinstance(event, dict): - copied = dict(event) - else: - copied = dict(vars(event)) - copied["route_nodes"] = updated_nodes - return copied - - def _fetch_hop_commitment(self, event: Any, node: dict) -> dict[str, Any] | None: - endpoint = node.get("endpoint") - if not isinstance(endpoint, str) or not endpoint: - return None - try: - response = _post_json( - f"{endpoint.rstrip('/')}/v1/audit/toploc/commitment", - { - "session_id": _event_value(event, "session_id"), - "model": _event_value(event, "model"), - "messages": _event_value(event, "messages") or [], - "shard_start": node.get("shard_start"), - "shard_end": node.get("shard_end"), - }, - timeout=2.0, - ) - except (OSError, ValueError, json.JSONDecodeError): - return None - proof = response.get("toploc_proof") or response.get("activation_proof") - token_ids = response.get("claimed_token_ids") or response.get("output_token_ids") - if not isinstance(proof, dict): - return None - if not isinstance(token_ids, list) or not all(isinstance(token, int) for token in token_ids): - return None - return {"toploc_proof": proof, "claimed_token_ids": token_ids} - - def _commitment_expired(self, event: Any) -> bool: - ts = _event_value(event, "ts") - if ts is None: - return False - return (time.time() - float(ts)) > self._toploc_config.commitment_ttl_seconds - - def _run_teacher_forced_prefill( - self, - *, - model: str, - messages: list[dict], - claimed_token_ids: list[int], - claim: ToplocProofClaim, - ) -> list[Any]: - response = _post_json( - f"{self._reference_node_url}/v1/audit/toploc", - { - "model": model, - "messages": messages, - "claimed_token_ids": claimed_token_ids, - "dtype": claim.dtype, - "quantization": claim.quantization, - "decode_batching_size": claim.decode_batching_size, - "topk": claim.topk, - "skip_prefill": claim.skip_prefill, - }, - ) - activations = response.get("activations") - if not isinstance(activations, list): - raise ValueError("reference node audit response did not contain activations") - return activations - - def _run_teacher_forced_prefill_hops( - self, - *, - model: str, - messages: list[dict], - hop_commitments: list["_HopCommitment"], - ) -> list[list[Any]]: - """Teacher-force the claimed tokens once and collect reference - activations at every hop's boundary layer (ADR-0018 §4 / research §1.2: - one referee forward pass, compared at each cut-point).""" - reference_claim = hop_commitments[0].claim - response = _post_json( - f"{self._reference_node_url}/v1/audit/toploc", - { - "model": model, - "messages": messages, - "claimed_token_ids": hop_commitments[-1].token_ids, - "hop_boundaries": [hop.shard_end for hop in hop_commitments], - "dtype": reference_claim.dtype, - "quantization": reference_claim.quantization, - "decode_batching_size": reference_claim.decode_batching_size, - "topk": reference_claim.topk, - "skip_prefill": reference_claim.skip_prefill, - }, - ) - activations_by_hop = response.get("activations_by_hop") - if not isinstance(activations_by_hop, list) or len(activations_by_hop) != len(hop_commitments): - raise ValueError("reference node audit response did not contain per-hop activations") - return activations_by_hop - - def _record_clean_audit(self, event: Any) -> None: - """ADR-0018 §6: reputation derives only from tracker-verified audit - outcomes — a clean audit credits every node on the verified route.""" - for wallet_address in _route_wallets(event): - if self._contracts.registry.get_wallet(wallet_address).banned: - continue - self._contracts.registry.record_audit_outcome(wallet_address, passed=True) - - def _slash_node( - self, - node: dict | None, - observed_output: str, - reference_output: str, - *, - reason: str = "reference output diverged", - ) -> list[Any]: - receipts: list[Any] = [] - wallet_address = node.get("wallet_address") if node else None - if not wallet_address: - return receipts - if self._contracts.registry.get_wallet(wallet_address).banned: - return receipts - receipts.append(self._contracts.registry.submit_slash_proof( - wallet_address=wallet_address, - slash_amount=self._slash_amount, - strike_threshold=self._strike_threshold, - reason=( - f"{reason} " - f"(observed={observed_output!r}, reference={reference_output!r})" - ), - webhook_url=self._webhook_url, - )) - # ADR-0018 §6: reputation loss is separate from the strike/ban that - # submit_slash_proof already recorded above — never double-strike. - self._contracts.registry.record_audit_outcome(wallet_address, passed=False) - # ADR-0015: the pending balance is the collateral — forfeit it in the - # same validation cycle as the strike. - if self._billing is not None: - forfeit = self._billing.forfeit_pending(wallet_address, reason="fraud-divergence") - print( - f"[validator] forfeited pending balance of {wallet_address}: " - f"{forfeit['amount']:.6f} USDT (fraud-divergence)", - flush=True, - ) - return receipts - - -def _route_wallets(event: Any) -> list[str]: - """Unique wallet addresses across a route, in hop order.""" - route_nodes = _event_value(event, "route_nodes") or [] - seen: set[str] = set() - wallets: list[str] = [] - for node in route_nodes: - wallet_address = node.get("wallet_address") if isinstance(node, dict) else None - if wallet_address and wallet_address not in seen: - seen.add(wallet_address) - wallets.append(wallet_address) - return wallets - - -def _final_text_node(route_nodes: list[dict]) -> dict | None: - """Text-only fallback blame: when the audit has no per-hop fingerprints - to bisect (free-running text comparison only), guess the last hop. - Never used once hop-boundary commitments make real bisection possible.""" - if not route_nodes: - return None - return max(route_nodes, key=lambda node: int(node.get("shard_end", 0))) - - -class _AuditResult: - def __init__( - self, - *, - ok: bool, - reference_output: str, - reason: str, - culprit_node: dict | None = None, - ) -> None: - self.ok = ok - self.reference_output = reference_output - self.reason = reason - self.culprit_node = culprit_node - - -class _HopCommitment: - """One hop's on-demand TOPLOC commitment plus the route node it blames.""" - - def __init__(self, node: dict | None, claim: ToplocProofClaim, token_ids: list[int]) -> None: - self.node = node - self.claim = claim - self.token_ids = token_ids - self.shard_end = int(node.get("shard_end", 0)) if node else None - - -def _hop_commitments_from_event(event: Any) -> list[_HopCommitment] | None: - """Per-hop bisection commitments (ADR-0018 §3/§4): each route node reports - its own output-boundary fingerprint. Falls back to the AH-006 whole-route - commitment format (one fingerprint, no bisection) when hops don't carry - individual commitments.""" - route_nodes = _event_value(event, "route_nodes") or [] - per_hop_nodes = [ - node for node in route_nodes - if isinstance(node, dict) and _mapping_value(node, "toploc_proof") is not None - ] - if per_hop_nodes and len(per_hop_nodes) == len(route_nodes): - ordered = sorted(route_nodes, key=lambda node: int(node.get("shard_start", 0))) - default_token_ids = _event_value(event, "claimed_token_ids") - commitments = [] - for node in ordered: - token_ids = node.get("claimed_token_ids", default_token_ids) - if not isinstance(token_ids, list) or not all(isinstance(t, int) for t in token_ids): - raise ValueError("TOPLOC hop commitments must include claimed_token_ids") - commitments.append(_HopCommitment( - node, - ToplocProofClaim.from_mapping(node["toploc_proof"]), - token_ids, - )) - return commitments - - whole_route = _toploc_audit_from_event(event) - if whole_route is None: - return None - token_ids, claim = whole_route - return [_HopCommitment(route_nodes[-1] if route_nodes else None, claim, token_ids)] - - -def _first_divergent_hop( - hop_commitments: list[_HopCommitment], - reference_activations_by_hop: list[Any], - *, - config: ToplocAuditConfig, - backend: Any | None, -) -> int | None: - """First hop whose committed output fingerprint diverges from the - referee's independently-computed reference activations at that same - cut-point (research §1.2: no interactive game needed at hop granularity — - the referee checks every cut-point in one replay).""" - for index, commitment in enumerate(hop_commitments): - ok = verify_activation_proofs( - reference_activations_by_hop[index], - commitment.claim, - config=config, - backend=backend, - ) - if not ok: - return index - return None - - -def _toploc_audit_from_event(event: Any) -> tuple[list[int], ToplocProofClaim] | None: - audit = _event_mapping(event, "audit") - claim_data = ( - _event_mapping(event, "toploc_proof") - or _event_mapping(event, "activation_proof") - or _mapping_value(audit, "toploc_proof") - or _mapping_value(audit, "activation_proof") - or _mapping_value(audit, "toploc") - ) - if claim_data is None: - return None - token_ids = ( - _event_value(event, "claimed_token_ids") - or _event_value(event, "output_token_ids") - or _mapping_value(audit, "claimed_token_ids") - or _mapping_value(audit, "output_token_ids") - ) - if not isinstance(token_ids, list) or not all(isinstance(token, int) for token in token_ids): - raise ValueError("TOPLOC audit events must include claimed_token_ids") - return token_ids, ToplocProofClaim.from_mapping(claim_data) - - -def _event_value(event: Any, name: str) -> Any: - if hasattr(event, name): - return getattr(event, name) - if isinstance(event, dict): - return event.get(name) - return None - - -def _event_mapping(event: Any, name: str) -> dict[str, Any] | None: - value = _event_value(event, name) - return value if isinstance(value, dict) else None - - -def _mapping_value(mapping: dict[str, Any] | None, name: str) -> Any: - if mapping is None: - return None - return mapping.get(name) - - -def _outputs_match(observed: str, reference: str, tolerance: float) -> bool: - observed_float = _parse_float(observed) - reference_float = _parse_float(reference) - if observed_float is not None and reference_float is not None: - return math.isclose(observed_float, reference_float, rel_tol=tolerance, abs_tol=tolerance) - return observed == reference - - -def _parse_float(value: str) -> float | None: - try: - return float(value) - except ValueError: - return None - - -def _post_json(url: str, payload: dict, timeout: float = 5.0) -> dict: - data = json.dumps(payload).encode() - req = urllib.request.Request( - url, - data=data, - headers={"Content-Type": "application/json"}, - method="POST", - ) - with urllib.request.urlopen(req, timeout=timeout) as response: - return json.loads(response.read()) - - -__all__ = [ - "ToplocAuditConfig", - "ToplocProofClaim", - "ValidatorProcess", - "AdaptiveAuditSampler", - "AuditRateConfig", - "detect_output_tripwire", -] +"""Optimistic fraud validator for completed inference requests.""" + +import json +import math +import random +import threading +import time +import urllib.request +from typing import Any + +from .audit import ( + ToplocAuditConfig, + ToplocProofClaim, + ToplocVerificationResult, + verify_activation_proofs, + verify_activation_proofs_detailed, +) +from .sampling import AdaptiveAuditSampler, AuditRateConfig +from .tripwire import detect_output_tripwire + +__version__ = "0.1.0" + + +class ValidatorProcess: + """Separate validator loop that samples completed requests and submits slashes.""" + + def __init__( + self, + *, + contracts: Any, + reference_node_url: str, + sample_rate: float = 0.05, + tolerance: float = 1e-6, + slash_amount: int = 100, + strike_threshold: int = 3, + random_seed: int | None = None, + webhook_url: str | None = None, + interval_seconds: float = 1.0, + billing: Any | None = None, + toploc_config: ToplocAuditConfig | None = None, + toploc_backend: Any | None = None, + audit_sampler: AdaptiveAuditSampler | None = None, + ) -> None: + if not 0.0 <= sample_rate <= 1.0: + raise ValueError("sample_rate must be between 0 and 1") + if tolerance < 0: + raise ValueError("tolerance must be non-negative") + if slash_amount <= 0: + raise ValueError("slash_amount must be positive") + if strike_threshold <= 0: + raise ValueError("strike_threshold must be positive") + if interval_seconds <= 0: + raise ValueError("interval_seconds must be positive") + + self._contracts = contracts + self._billing = billing + self._reference_node_url = reference_node_url.rstrip("/") + self._sample_rate = sample_rate + self._tolerance = tolerance + self._slash_amount = slash_amount + self._strike_threshold = strike_threshold + self._webhook_url = webhook_url + self._interval_seconds = interval_seconds + self._toploc_config = toploc_config or ToplocAuditConfig() + self._toploc_backend = toploc_backend + self._audit_sampler = audit_sampler + self._random = random.Random(random_seed) + self._last_event_index = -1 + self._running = False + self._thread: threading.Thread | None = None + self.sampled_count = 0 + + def validate_once(self) -> list[Any]: + """Run one validation cycle and return slash receipts submitted this cycle.""" + receipts: list[Any] = [] + events = self._contracts.validation.list_completed_inferences( + after_index=self._last_event_index, + ) + for event in events: + self._last_event_index = max(self._last_event_index, event.index) + if not self._should_sample(event): + continue + self.sampled_count += 1 + audit_result = self._validate_event(event) + if audit_result.ok: + self._record_clean_audit(event) + continue + receipts.extend(self._slash_node( + audit_result.culprit_node, + event.observed_output, + audit_result.reference_output, + reason=audit_result.reason, + )) + return receipts + + def _should_sample(self, event: Any) -> bool: + """ADR-0018 §1/§6-7: flat sample_rate stays the default; when an + AdaptiveAuditSampler is configured, the decision is reputation- and + tenure-weighted and budget-balanced against the fleet-wide target + instead of a uniform coin flip.""" + if self._audit_sampler is None: + return self._random.random() < self._sample_rate + + tripwire = detect_output_tripwire(_event_value(event, "observed_output") or "") + wallets = _route_wallets(event) + if not wallets: + return self._audit_sampler.should_audit( + completed_job_count=0, reputation=1.0, tripwire=tripwire, + ) + # A route is only as trustworthy as its least-trusted hop -- audit + # against whichever wallet on the route looks riskiest. + riskiest = min( + (self._contracts.registry.get_wallet(wallet) for wallet in wallets), + key=lambda wallet: wallet.reputation, + ) + return self._audit_sampler.should_audit( + completed_job_count=riskiest.completed_job_count, + reputation=riskiest.reputation, + tripwire=tripwire, + ) + + def start(self) -> None: + if self._running: + raise RuntimeError("ValidatorProcess is already running") + self._running = True + self._thread = threading.Thread(target=self._run_loop, daemon=True) + self._thread.start() + + def stop(self) -> None: + self._running = False + if self._thread is not None: + self._thread.join(timeout=2) + self._thread = None + + def _run_loop(self) -> None: + while self._running: + self.validate_once() + time.sleep(self._interval_seconds) + + def _run_reference(self, messages: list[dict]) -> str: + response = _post_json( + f"{self._reference_node_url}/v1/infer", + {"messages": messages}, + ) + text = response.get("text") + if not isinstance(text, str): + raise ValueError("reference node response did not contain text") + return text + + def _validate_event(self, event: Any) -> "_AuditResult": + event = self._event_with_on_demand_commitments(event) + hop_commitments = _hop_commitments_from_event(event) + if hop_commitments is not None and self._commitment_expired(event): + # ADR-0018 §3: the on-demand retention window has passed — nodes + # are no longer expected to hold the boundary activations needed + # to verify this commitment, so fall back to the text-only path. + hop_commitments = None + + if hop_commitments is None: + reference_output = self._run_reference(event.messages) + ok = _outputs_match(event.observed_output, reference_output, self._tolerance) + return _AuditResult( + ok=ok, + reference_output=reference_output, + reason="reference output diverged", + # Text comparison has no per-hop signal; the last hop is the + # best-effort guess (text-only fallback), never used when + # hop-boundary commitments make real bisection possible. + culprit_node=None if ok else _final_text_node(event.route_nodes), + ) + + if len(hop_commitments) == 1: + # Single-commitment route (AH-006 whole-route format, or a + # genuinely one-hop pipeline): reuse the original teacher-forced + # call so existing single-hop reference integrations keep working. + only = hop_commitments[0] + reference_activations_by_hop = [self._run_teacher_forced_prefill( + model=_event_value(event, "model"), + messages=_event_value(event, "messages"), + claimed_token_ids=only.token_ids, + claim=only.claim, + )] + else: + reference_activations_by_hop = self._run_teacher_forced_prefill_hops( + model=_event_value(event, "model"), + messages=_event_value(event, "messages"), + hop_commitments=hop_commitments, + ) + culprit_index = _first_divergent_hop( + hop_commitments, + reference_activations_by_hop, + config=self._toploc_config, + backend=self._toploc_backend, + ) + ok = culprit_index is None + return _AuditResult( + ok=ok, + reference_output=( + "TOPLOC activation proof accepted" + if ok + else f"TOPLOC activation proof mismatch at hop {culprit_index}" + ), + reason="TOPLOC activation proof mismatch", + culprit_node=None if ok else hop_commitments[culprit_index].node, + ) + + def _event_with_on_demand_commitments(self, event: Any) -> Any: + """Fetch missing per-hop TOPLOC commitments only after audit sampling. + + Tracker validation events deliberately carry ordinary route metadata, + not a pre-announced audit flag. When this validator samples an event, it + asks each hop for its short-lived boundary commitment and splices the + returned proof into a local event copy for the bisection verifier. + """ + route_nodes = _event_value(event, "route_nodes") or [] + if not isinstance(route_nodes, list) or not route_nodes: + return event + updated_nodes: list[dict] = [] + changed = False + for node in route_nodes: + if not isinstance(node, dict): + updated_nodes.append(node) + continue + updated = dict(node) + if _mapping_value(updated, "toploc_proof") is None: + commitment = self._fetch_hop_commitment(event, updated) + if commitment is not None: + updated.update(commitment) + changed = True + updated_nodes.append(updated) + if not changed: + return event + if isinstance(event, dict): + copied = dict(event) + else: + copied = dict(vars(event)) + copied["route_nodes"] = updated_nodes + return copied + + def _fetch_hop_commitment(self, event: Any, node: dict) -> dict[str, Any] | None: + endpoint = node.get("endpoint") + if not isinstance(endpoint, str) or not endpoint: + return None + try: + response = _post_json( + f"{endpoint.rstrip('/')}/v1/audit/toploc/commitment", + { + "session_id": _event_value(event, "session_id"), + "model": _event_value(event, "model"), + "messages": _event_value(event, "messages") or [], + "shard_start": node.get("shard_start"), + "shard_end": node.get("shard_end"), + }, + timeout=2.0, + ) + except (OSError, ValueError, json.JSONDecodeError): + return None + proof = response.get("toploc_proof") or response.get("activation_proof") + token_ids = response.get("claimed_token_ids") or response.get("output_token_ids") + if not isinstance(proof, dict): + return None + if not isinstance(token_ids, list) or not all(isinstance(token, int) for token in token_ids): + return None + return {"toploc_proof": proof, "claimed_token_ids": token_ids} + + def _commitment_expired(self, event: Any) -> bool: + ts = _event_value(event, "ts") + if ts is None: + return False + return (time.time() - float(ts)) > self._toploc_config.commitment_ttl_seconds + + def _run_teacher_forced_prefill( + self, + *, + model: str, + messages: list[dict], + claimed_token_ids: list[int], + claim: ToplocProofClaim, + ) -> list[Any]: + response = _post_json( + f"{self._reference_node_url}/v1/audit/toploc", + { + "model": model, + "messages": messages, + "claimed_token_ids": claimed_token_ids, + "dtype": claim.dtype, + "quantization": claim.quantization, + "decode_batching_size": claim.decode_batching_size, + "topk": claim.topk, + "skip_prefill": claim.skip_prefill, + }, + ) + activations = response.get("activations") + if not isinstance(activations, list): + raise ValueError("reference node audit response did not contain activations") + return activations + + def _run_teacher_forced_prefill_hops( + self, + *, + model: str, + messages: list[dict], + hop_commitments: list["_HopCommitment"], + ) -> list[list[Any]]: + """Teacher-force the claimed tokens once and collect reference + activations at every hop's boundary layer (ADR-0018 §4 / research §1.2: + one referee forward pass, compared at each cut-point).""" + reference_claim = hop_commitments[0].claim + response = _post_json( + f"{self._reference_node_url}/v1/audit/toploc", + { + "model": model, + "messages": messages, + "claimed_token_ids": hop_commitments[-1].token_ids, + "hop_boundaries": [hop.shard_end for hop in hop_commitments], + "dtype": reference_claim.dtype, + "quantization": reference_claim.quantization, + "decode_batching_size": reference_claim.decode_batching_size, + "topk": reference_claim.topk, + "skip_prefill": reference_claim.skip_prefill, + }, + ) + activations_by_hop = response.get("activations_by_hop") + if not isinstance(activations_by_hop, list) or len(activations_by_hop) != len(hop_commitments): + raise ValueError("reference node audit response did not contain per-hop activations") + return activations_by_hop + + def _record_clean_audit(self, event: Any) -> None: + """ADR-0018 §6: reputation derives only from tracker-verified audit + outcomes — a clean audit credits every node on the verified route.""" + for wallet_address in _route_wallets(event): + if self._contracts.registry.get_wallet(wallet_address).banned: + continue + self._contracts.registry.record_audit_outcome(wallet_address, passed=True) + + def _slash_node( + self, + node: dict | None, + observed_output: str, + reference_output: str, + *, + reason: str = "reference output diverged", + ) -> list[Any]: + receipts: list[Any] = [] + wallet_address = node.get("wallet_address") if node else None + if not wallet_address: + return receipts + if self._contracts.registry.get_wallet(wallet_address).banned: + return receipts + receipts.append(self._contracts.registry.submit_slash_proof( + wallet_address=wallet_address, + slash_amount=self._slash_amount, + strike_threshold=self._strike_threshold, + reason=( + f"{reason} " + f"(observed={observed_output!r}, reference={reference_output!r})" + ), + webhook_url=self._webhook_url, + )) + # ADR-0018 §6: reputation loss is separate from the strike/ban that + # submit_slash_proof already recorded above — never double-strike. + self._contracts.registry.record_audit_outcome(wallet_address, passed=False) + # ADR-0015: the pending balance is the collateral — forfeit it in the + # same validation cycle as the strike. + if self._billing is not None: + forfeit = self._billing.forfeit_pending(wallet_address, reason="fraud-divergence") + print( + f"[validator] forfeited pending balance of {wallet_address}: " + f"{forfeit['amount']:.6f} USDT (fraud-divergence)", + flush=True, + ) + return receipts + + +def _route_wallets(event: Any) -> list[str]: + """Unique wallet addresses across a route, in hop order.""" + route_nodes = _event_value(event, "route_nodes") or [] + seen: set[str] = set() + wallets: list[str] = [] + for node in route_nodes: + wallet_address = node.get("wallet_address") if isinstance(node, dict) else None + if wallet_address and wallet_address not in seen: + seen.add(wallet_address) + wallets.append(wallet_address) + return wallets + + +def _final_text_node(route_nodes: list[dict]) -> dict | None: + """Text-only fallback blame: when the audit has no per-hop fingerprints + to bisect (free-running text comparison only), guess the last hop. + Never used once hop-boundary commitments make real bisection possible.""" + if not route_nodes: + return None + return max(route_nodes, key=lambda node: int(node.get("shard_end", 0))) + + +class _AuditResult: + def __init__( + self, + *, + ok: bool, + reference_output: str, + reason: str, + culprit_node: dict | None = None, + ) -> None: + self.ok = ok + self.reference_output = reference_output + self.reason = reason + self.culprit_node = culprit_node + + +class _HopCommitment: + """One hop's on-demand TOPLOC commitment plus the route node it blames.""" + + def __init__(self, node: dict | None, claim: ToplocProofClaim, token_ids: list[int]) -> None: + self.node = node + self.claim = claim + self.token_ids = token_ids + self.shard_end = int(node.get("shard_end", 0)) if node else None + + +def _hop_commitments_from_event(event: Any) -> list[_HopCommitment] | None: + """Per-hop bisection commitments (ADR-0018 §3/§4): each route node reports + its own output-boundary fingerprint. Falls back to the AH-006 whole-route + commitment format (one fingerprint, no bisection) when hops don't carry + individual commitments.""" + route_nodes = _event_value(event, "route_nodes") or [] + per_hop_nodes = [ + node for node in route_nodes + if isinstance(node, dict) and _mapping_value(node, "toploc_proof") is not None + ] + if per_hop_nodes and len(per_hop_nodes) == len(route_nodes): + ordered = sorted(route_nodes, key=lambda node: int(node.get("shard_start", 0))) + default_token_ids = _event_value(event, "claimed_token_ids") + commitments = [] + for node in ordered: + token_ids = node.get("claimed_token_ids", default_token_ids) + if not isinstance(token_ids, list) or not all(isinstance(t, int) for t in token_ids): + raise ValueError("TOPLOC hop commitments must include claimed_token_ids") + commitments.append(_HopCommitment( + node, + ToplocProofClaim.from_mapping(node["toploc_proof"]), + token_ids, + )) + return commitments + + whole_route = _toploc_audit_from_event(event) + if whole_route is None: + return None + token_ids, claim = whole_route + return [_HopCommitment(route_nodes[-1] if route_nodes else None, claim, token_ids)] + + +def _first_divergent_hop( + hop_commitments: list[_HopCommitment], + reference_activations_by_hop: list[Any], + *, + config: ToplocAuditConfig, + backend: Any | None, +) -> int | None: + """First hop whose committed output fingerprint diverges from the + referee's independently-computed reference activations at that same + cut-point (research §1.2: no interactive game needed at hop granularity — + the referee checks every cut-point in one replay).""" + for index, commitment in enumerate(hop_commitments): + ok = verify_activation_proofs( + reference_activations_by_hop[index], + commitment.claim, + config=config, + backend=backend, + ) + if not ok: + return index + return None + + +def _toploc_audit_from_event(event: Any) -> tuple[list[int], ToplocProofClaim] | None: + audit = _event_mapping(event, "audit") + claim_data = ( + _event_mapping(event, "toploc_proof") + or _event_mapping(event, "activation_proof") + or _mapping_value(audit, "toploc_proof") + or _mapping_value(audit, "activation_proof") + or _mapping_value(audit, "toploc") + ) + if claim_data is None: + return None + token_ids = ( + _event_value(event, "claimed_token_ids") + or _event_value(event, "output_token_ids") + or _mapping_value(audit, "claimed_token_ids") + or _mapping_value(audit, "output_token_ids") + ) + if not isinstance(token_ids, list) or not all(isinstance(token, int) for token in token_ids): + raise ValueError("TOPLOC audit events must include claimed_token_ids") + return token_ids, ToplocProofClaim.from_mapping(claim_data) + + +def _event_value(event: Any, name: str) -> Any: + if hasattr(event, name): + return getattr(event, name) + if isinstance(event, dict): + return event.get(name) + return None + + +def _event_mapping(event: Any, name: str) -> dict[str, Any] | None: + value = _event_value(event, name) + return value if isinstance(value, dict) else None + + +def _mapping_value(mapping: dict[str, Any] | None, name: str) -> Any: + if mapping is None: + return None + return mapping.get(name) + + +def _outputs_match(observed: str, reference: str, tolerance: float) -> bool: + observed_float = _parse_float(observed) + reference_float = _parse_float(reference) + if observed_float is not None and reference_float is not None: + return math.isclose(observed_float, reference_float, rel_tol=tolerance, abs_tol=tolerance) + return observed == reference + + +def _parse_float(value: str) -> float | None: + try: + return float(value) + except ValueError: + return None + + +def _post_json(url: str, payload: dict, timeout: float = 5.0) -> dict: + data = json.dumps(payload).encode() + req = urllib.request.Request( + url, + data=data, + headers={"Content-Type": "application/json"}, + method="POST", + ) + with urllib.request.urlopen(req, timeout=timeout) as response: + return json.loads(response.read()) + + +__all__ = [ + "ToplocAuditConfig", + "ToplocProofClaim", + "ValidatorProcess", + "AdaptiveAuditSampler", + "AuditRateConfig", + "detect_output_tripwire", +] diff --git a/tests/test_accounts.py b/tests/test_accounts.py index c90d375..62388ef 100644 --- a/tests/test_accounts.py +++ b/tests/test_accounts.py @@ -1,384 +1,384 @@ -"""Dashboard user accounts: registration, login, roles, API keys, usage. - -Unit tests for AccountStore plus HTTP integration on the tracker: -register/login/logout, per-account balance and usage, API-key lifecycle -(revoked keys rejected by the OpenAI proxy), and the admin listing. -""" - -import http.cookies -import json -import urllib.error -import urllib.request - -import pytest - -from meshnet_tracker.accounts import AccountStore -from meshnet_tracker.auth import sign_hive_request -from meshnet_tracker.billing import BillingLedger -from meshnet_tracker.server import TrackerServer - -HIVE_SECRET = "test-hive-secret" - - -# ---------------------------------------------------------------- unit tests - - -def test_first_account_is_admin_then_users(): - store = AccountStore() - first = store.register(email="admin@example.com", password="secret-123") - second = store.register(email="user@example.com", password="secret-123") - assert first["role"] == "admin" - assert second["role"] == "user" - - -def test_register_requires_email_or_wallet_and_password_length(): - store = AccountStore() - with pytest.raises(ValueError, match="email or a wallet"): - store.register(password="secret-123") - with pytest.raises(ValueError, match="invalid email"): - store.register(email="not-an-email", password="secret-123") - with pytest.raises(ValueError, match="at least 8"): - store.register(email="a@b.co", password="short") - - -def test_register_rejects_duplicate_identifiers(): - store = AccountStore() - store.register(email="dup@example.com", password="secret-123") - with pytest.raises(ValueError, match="already exists"): - store.register(email="DUP@example.com", password="other-secret") - - -def test_login_by_email_or_wallet(): - store = AccountStore() - account = store.register( - email="both@example.com", wallet="WalletXYZ", password="secret-123" - ) - assert store.verify_login("both@example.com", "secret-123")["account_id"] == account["account_id"] - assert store.verify_login("WalletXYZ", "secret-123")["account_id"] == account["account_id"] - assert store.verify_login("both@example.com", "wrong-password") is None - assert store.verify_login("nobody@example.com", "secret-123") is None - - -def test_sessions_resolve_and_destroy(): - store = AccountStore() - account = store.register(email="s@example.com", password="secret-123") - token = store.create_session(account["account_id"]) - assert store.session_account(token)["account_id"] == account["account_id"] - store.destroy_session(token) - assert store.session_account(token) is None - assert store.session_account("bogus") is None - - -def test_sessions_persist_across_restart(tmp_path): - db = str(tmp_path / "accounts.db") - store = AccountStore(db_path=db) - account = store.register(email="cookie@example.com", password="secret-123") - token = store.create_session(account["account_id"]) - store.save_to_db() - - reloaded = AccountStore(db_path=db) - assert reloaded.session_account(token)["account_id"] == account["account_id"] - - -def test_api_key_lifecycle(): - store = AccountStore() - account = store.register(email="k@example.com", password="secret-123") - other = store.register(email="other@example.com", password="secret-123") - key = store.create_api_key(account["account_id"]) - assert key.startswith("sk-mesh-") - assert store.keys_for(account["account_id"]) == [key] - # someone else's account cannot revoke it - assert store.revoke_api_key(other["account_id"], key) is False - assert store.revoke_api_key(account["account_id"], key) is True - assert store.keys_for(account["account_id"]) == [] - assert store.is_key_revoked(key) - - -def test_accounts_persist_across_restart(tmp_path): - db = str(tmp_path / "accounts.db") - store = AccountStore(db_path=db) - account = store.register(email="p@example.com", password="secret-123") - key = store.create_api_key(account["account_id"]) - store.save_to_db() - - reloaded = AccountStore(db_path=db) - assert reloaded.verify_login("p@example.com", "secret-123") is not None - assert reloaded.keys_for(account["account_id"]) == [key] - - -def test_account_events_replicate_and_dedupe(): - leader = AccountStore() - follower = AccountStore() - account = leader.register(email="r@example.com", password="secret-123") - key = leader.create_api_key(account["account_id"]) - leader.revoke_api_key(account["account_id"], key) - - events, cursor = leader.events_since(0) - assert follower.apply_events(events) == len(events) - assert follower.apply_events(events) == 0 # replay is a no-op - assert follower.verify_login("r@example.com", "secret-123") is not None - assert follower.is_key_revoked(key) - more, _ = leader.events_since(cursor) - assert more == [] - - -# ---------------------------------------------------------- HTTP integration - - -def _call(url, method="GET", body=None, token=None): - headers = {"Content-Type": "application/json"} - if token: - headers["Authorization"] = f"Bearer {token}" - data = json.dumps(body).encode() if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req) as r: - return json.loads(r.read()) - - -@pytest.fixture -def account_tracker(): - """Tracker with credit features pinned OFF (defaults are devnet-friendly 1.0).""" - ledger = BillingLedger(starting_credit=0.0, default_price_per_1k=0.02) - tracker = TrackerServer( - billing=ledger, - accounts=AccountStore(), - hive_secret=HIVE_SECRET, - starting_credit=0.0, - devnet_topup_amount=0.0, - ) - port = tracker.start() - yield f"http://127.0.0.1:{port}", ledger - tracker.stop() - - -def test_register_login_and_account_view(account_tracker): - url, _ = account_tracker - reg = _call(f"{url}/v1/auth/register", "POST", - {"email": "admin@example.com", "password": "secret-123"}) - assert reg["account"]["role"] == "admin" - assert reg["api_key"].startswith("sk-mesh-") - assert reg["session_token"] - - login = _call(f"{url}/v1/auth/login", "POST", - {"identifier": "admin@example.com", "password": "secret-123"}) - me = _call(f"{url}/v1/account", token=login["session_token"]) - assert me["account"]["email"] == "admin@example.com" - assert me["api_keys"] == [reg["api_key"]] - assert me["total_balance"] == pytest.approx(0.0) - assert me["usage"]["requests"] == 0 - - -def test_login_sets_cookie_and_cookie_auth_survives_tracker_restart(tmp_path): - accounts_db = str(tmp_path / "accounts.db") - tracker = TrackerServer( - billing=BillingLedger(starting_credit=0.0, default_price_per_1k=0.02), - accounts_db=accounts_db, - starting_credit=0.0, - devnet_topup_amount=0.0, - ) - port = tracker.start() - url = f"http://127.0.0.1:{port}" - try: - _call(f"{url}/v1/auth/register", "POST", - {"email": "cookie-http@example.com", "password": "secret-123"}) - req = urllib.request.Request( - f"{url}/v1/auth/login", - data=json.dumps({ - "identifier": "cookie-http@example.com", - "password": "secret-123", - }).encode(), - headers={"Content-Type": "application/json"}, - method="POST", - ) - with urllib.request.urlopen(req) as r: - assert json.loads(r.read())["session_token"] - cookie_header = r.headers["Set-Cookie"] - finally: - tracker.stop() - - cookie = http.cookies.SimpleCookie(cookie_header) - session_cookie = cookie["meshnet_session"].OutputString() - - restarted = TrackerServer( - billing=BillingLedger(starting_credit=0.0, default_price_per_1k=0.02), - accounts_db=accounts_db, - starting_credit=0.0, - devnet_topup_amount=0.0, - ) - restarted_port = restarted.start() - restarted_url = f"http://127.0.0.1:{restarted_port}" - try: - req = urllib.request.Request( - f"{restarted_url}/v1/account", - headers={"Cookie": session_cookie}, - method="GET", - ) - with urllib.request.urlopen(req) as r: - me = json.loads(r.read()) - finally: - restarted.stop() - - assert me["account"]["email"] == "cookie-http@example.com" - - -def test_bad_credentials_and_missing_session_are_401(account_tracker): - url, _ = account_tracker - _call(f"{url}/v1/auth/register", "POST", - {"email": "a@example.com", "password": "secret-123"}) - with pytest.raises(urllib.error.HTTPError) as exc_info: - _call(f"{url}/v1/auth/login", "POST", - {"identifier": "a@example.com", "password": "wrong-pass"}) - assert exc_info.value.code == 401 - with pytest.raises(urllib.error.HTTPError) as exc_info: - _call(f"{url}/v1/account") - assert exc_info.value.code == 401 - - -def test_key_create_revoke_and_revoked_key_rejected_by_proxy(account_tracker): - url, _ = account_tracker - reg = _call(f"{url}/v1/auth/register", "POST", - {"email": "k@example.com", "password": "secret-123"}) - token = reg["session_token"] - - new_key = _call(f"{url}/v1/account/keys", "POST", {}, token=token)["api_key"] - me = _call(f"{url}/v1/account", token=token) - assert sorted(me["api_keys"]) == sorted([reg["api_key"], new_key]) - - _call(f"{url}/v1/account/keys/revoke", "POST", {"api_key": new_key}, token=token) - with pytest.raises(urllib.error.HTTPError) as exc_info: - _call(f"{url}/v1/chat/completions", "POST", - {"model": "any", "messages": []}, token=new_key) - assert exc_info.value.code == 401 - assert "revoked" in exc_info.value.read().decode() - - -def test_admin_listing_requires_admin_role(account_tracker): - url, _ = account_tracker - admin = _call(f"{url}/v1/auth/register", "POST", - {"email": "admin@example.com", "password": "secret-123"}) - user = _call(f"{url}/v1/auth/register", "POST", - {"wallet": "WalletUser1", "password": "secret-123"}) - - with pytest.raises(urllib.error.HTTPError) as exc_info: - _call(f"{url}/v1/admin/accounts", token=user["session_token"]) - assert exc_info.value.code == 403 - - listing = _call(f"{url}/v1/admin/accounts", token=admin["session_token"]) - accounts = listing["accounts"] - assert len(accounts) == 2 - assert accounts[0]["role"] == "admin" - assert accounts[1]["wallet"] == "WalletUser1" - assert "balances" in accounts[0] - - -def test_accounts_gossip_endpoint_applies_events(account_tracker): - url, _ = account_tracker - peer = AccountStore() - peer.register(email="remote@example.com", password="secret-123") - events, _ = peer.events_since(0) - body = json.dumps({"events": events}).encode() - req = urllib.request.Request( - f"{url}/v1/accounts/gossip", data=body, - headers={"Content-Type": "application/json", **sign_hive_request(HIVE_SECRET, body)}, - method="POST", - ) - with urllib.request.urlopen(req) as r: - result = json.loads(r.read()) - assert result["applied"] == len(events) - login = _call(f"{url}/v1/auth/login", "POST", - {"identifier": "remote@example.com", "password": "secret-123"}) - assert login["account"]["email"] == "remote@example.com" - - -def test_accounts_endpoints_404_when_disabled(): - tracker = TrackerServer() # no accounts, no billing - port = tracker.start() - try: - with pytest.raises(urllib.error.HTTPError) as exc_info: - _call(f"http://127.0.0.1:{port}/v1/auth/register", "POST", - {"email": "x@example.com", "password": "secret-123"}) - assert exc_info.value.code == 404 - finally: - tracker.stop() - - -# ------------------------------------------- US-039/US-040: credit and top-up - - -@pytest.fixture -def funded_tracker(): - """Tracker with Caller Credit and the devnet top-up faucet enabled.""" - ledger = BillingLedger(starting_credit=0.0, default_price_per_1k=0.02) - tracker = TrackerServer( - billing=ledger, - accounts=AccountStore(), - hive_secret=HIVE_SECRET, - starting_credit=1.0, - devnet_topup_amount=10.0, - ) - port = tracker.start() - yield f"http://127.0.0.1:{port}", ledger - tracker.stop() - - -def test_caller_credit_granted_once_per_account(funded_tracker): - url, ledger = funded_tracker - reg = _call(f"{url}/v1/auth/register", "POST", - {"email": "c@example.com", "password": "secret-123"}) - token = reg["session_token"] - first_key = reg["api_key"] - assert ledger.get_client_balance(first_key) == pytest.approx(1.0) - - # A second key never re-grants — not even after revoking the first. - second = _call(f"{url}/v1/account/keys", "POST", {}, token=token) - assert second["caller_credit_granted"] is False - assert ledger.get_client_balance(second["api_key"]) == pytest.approx(0.0) - _call(f"{url}/v1/account/keys/revoke", "POST", {"api_key": first_key}, token=token) - third = _call(f"{url}/v1/account/keys", "POST", {}, token=token) - assert third["caller_credit_granted"] is False - assert ledger.get_client_balance(third["api_key"]) == pytest.approx(0.0) - - -def test_unknown_bearer_key_rejected_by_proxy(funded_tracker): - url, ledger = funded_tracker - with pytest.raises(urllib.error.HTTPError) as exc_info: - _call(f"{url}/v1/chat/completions", "POST", - {"model": "any", "messages": []}, token="sk-mesh-made-up-key") - assert exc_info.value.code == 401 - assert "unknown API key" in exc_info.value.read().decode() - # The invented key must not have become a billable client. - assert ledger.get_client_balance("sk-mesh-made-up-key") == pytest.approx(0.0) - - -def test_devnet_topup_credits_own_key_only(funded_tracker): - url, ledger = funded_tracker - owner = _call(f"{url}/v1/auth/register", "POST", - {"email": "own@example.com", "password": "secret-123"}) - other = _call(f"{url}/v1/auth/register", "POST", - {"email": "oth@example.com", "password": "secret-123"}) - - me = _call(f"{url}/v1/account", token=owner["session_token"]) - assert me["topup_amount"] == pytest.approx(10.0) - - result = _call(f"{url}/v1/account/topup", "POST", - {"api_key": owner["api_key"]}, token=owner["session_token"]) - assert result["credited"] == pytest.approx(10.0) - assert result["balance"] == pytest.approx(11.0) # 1.0 caller credit + 10.0 - - with pytest.raises(urllib.error.HTTPError) as exc_info: - _call(f"{url}/v1/account/topup", "POST", - {"api_key": owner["api_key"]}, token=other["session_token"]) - assert exc_info.value.code == 403 - assert ledger.get_client_balance(owner["api_key"]) == pytest.approx(11.0) - - -def test_topup_404_when_disabled(account_tracker): - url, _ = account_tracker - reg = _call(f"{url}/v1/auth/register", "POST", - {"email": "t@example.com", "password": "secret-123"}) - me = _call(f"{url}/v1/account", token=reg["session_token"]) - assert me["topup_amount"] == pytest.approx(0.0) - with pytest.raises(urllib.error.HTTPError) as exc_info: - _call(f"{url}/v1/account/topup", "POST", - {"api_key": reg["api_key"]}, token=reg["session_token"]) - assert exc_info.value.code == 404 +"""Dashboard user accounts: registration, login, roles, API keys, usage. + +Unit tests for AccountStore plus HTTP integration on the tracker: +register/login/logout, per-account balance and usage, API-key lifecycle +(revoked keys rejected by the OpenAI proxy), and the admin listing. +""" + +import http.cookies +import json +import urllib.error +import urllib.request + +import pytest + +from meshnet_tracker.accounts import AccountStore +from meshnet_tracker.auth import sign_hive_request +from meshnet_tracker.billing import BillingLedger +from meshnet_tracker.server import TrackerServer + +HIVE_SECRET = "test-hive-secret" + + +# ---------------------------------------------------------------- unit tests + + +def test_first_account_is_admin_then_users(): + store = AccountStore() + first = store.register(email="admin@example.com", password="secret-123") + second = store.register(email="user@example.com", password="secret-123") + assert first["role"] == "admin" + assert second["role"] == "user" + + +def test_register_requires_email_or_wallet_and_password_length(): + store = AccountStore() + with pytest.raises(ValueError, match="email or a wallet"): + store.register(password="secret-123") + with pytest.raises(ValueError, match="invalid email"): + store.register(email="not-an-email", password="secret-123") + with pytest.raises(ValueError, match="at least 8"): + store.register(email="a@b.co", password="short") + + +def test_register_rejects_duplicate_identifiers(): + store = AccountStore() + store.register(email="dup@example.com", password="secret-123") + with pytest.raises(ValueError, match="already exists"): + store.register(email="DUP@example.com", password="other-secret") + + +def test_login_by_email_or_wallet(): + store = AccountStore() + account = store.register( + email="both@example.com", wallet="WalletXYZ", password="secret-123" + ) + assert store.verify_login("both@example.com", "secret-123")["account_id"] == account["account_id"] + assert store.verify_login("WalletXYZ", "secret-123")["account_id"] == account["account_id"] + assert store.verify_login("both@example.com", "wrong-password") is None + assert store.verify_login("nobody@example.com", "secret-123") is None + + +def test_sessions_resolve_and_destroy(): + store = AccountStore() + account = store.register(email="s@example.com", password="secret-123") + token = store.create_session(account["account_id"]) + assert store.session_account(token)["account_id"] == account["account_id"] + store.destroy_session(token) + assert store.session_account(token) is None + assert store.session_account("bogus") is None + + +def test_sessions_persist_across_restart(tmp_path): + db = str(tmp_path / "accounts.db") + store = AccountStore(db_path=db) + account = store.register(email="cookie@example.com", password="secret-123") + token = store.create_session(account["account_id"]) + store.save_to_db() + + reloaded = AccountStore(db_path=db) + assert reloaded.session_account(token)["account_id"] == account["account_id"] + + +def test_api_key_lifecycle(): + store = AccountStore() + account = store.register(email="k@example.com", password="secret-123") + other = store.register(email="other@example.com", password="secret-123") + key = store.create_api_key(account["account_id"]) + assert key.startswith("sk-mesh-") + assert store.keys_for(account["account_id"]) == [key] + # someone else's account cannot revoke it + assert store.revoke_api_key(other["account_id"], key) is False + assert store.revoke_api_key(account["account_id"], key) is True + assert store.keys_for(account["account_id"]) == [] + assert store.is_key_revoked(key) + + +def test_accounts_persist_across_restart(tmp_path): + db = str(tmp_path / "accounts.db") + store = AccountStore(db_path=db) + account = store.register(email="p@example.com", password="secret-123") + key = store.create_api_key(account["account_id"]) + store.save_to_db() + + reloaded = AccountStore(db_path=db) + assert reloaded.verify_login("p@example.com", "secret-123") is not None + assert reloaded.keys_for(account["account_id"]) == [key] + + +def test_account_events_replicate_and_dedupe(): + leader = AccountStore() + follower = AccountStore() + account = leader.register(email="r@example.com", password="secret-123") + key = leader.create_api_key(account["account_id"]) + leader.revoke_api_key(account["account_id"], key) + + events, cursor = leader.events_since(0) + assert follower.apply_events(events) == len(events) + assert follower.apply_events(events) == 0 # replay is a no-op + assert follower.verify_login("r@example.com", "secret-123") is not None + assert follower.is_key_revoked(key) + more, _ = leader.events_since(cursor) + assert more == [] + + +# ---------------------------------------------------------- HTTP integration + + +def _call(url, method="GET", body=None, token=None): + headers = {"Content-Type": "application/json"} + if token: + headers["Authorization"] = f"Bearer {token}" + data = json.dumps(body).encode() if body is not None else None + req = urllib.request.Request(url, data=data, headers=headers, method=method) + with urllib.request.urlopen(req) as r: + return json.loads(r.read()) + + +@pytest.fixture +def account_tracker(): + """Tracker with credit features pinned OFF (defaults are devnet-friendly 1.0).""" + ledger = BillingLedger(starting_credit=0.0, default_price_per_1k=0.02) + tracker = TrackerServer( + billing=ledger, + accounts=AccountStore(), + hive_secret=HIVE_SECRET, + starting_credit=0.0, + devnet_topup_amount=0.0, + ) + port = tracker.start() + yield f"http://127.0.0.1:{port}", ledger + tracker.stop() + + +def test_register_login_and_account_view(account_tracker): + url, _ = account_tracker + reg = _call(f"{url}/v1/auth/register", "POST", + {"email": "admin@example.com", "password": "secret-123"}) + assert reg["account"]["role"] == "admin" + assert reg["api_key"].startswith("sk-mesh-") + assert reg["session_token"] + + login = _call(f"{url}/v1/auth/login", "POST", + {"identifier": "admin@example.com", "password": "secret-123"}) + me = _call(f"{url}/v1/account", token=login["session_token"]) + assert me["account"]["email"] == "admin@example.com" + assert me["api_keys"] == [reg["api_key"]] + assert me["total_balance"] == pytest.approx(0.0) + assert me["usage"]["requests"] == 0 + + +def test_login_sets_cookie_and_cookie_auth_survives_tracker_restart(tmp_path): + accounts_db = str(tmp_path / "accounts.db") + tracker = TrackerServer( + billing=BillingLedger(starting_credit=0.0, default_price_per_1k=0.02), + accounts_db=accounts_db, + starting_credit=0.0, + devnet_topup_amount=0.0, + ) + port = tracker.start() + url = f"http://127.0.0.1:{port}" + try: + _call(f"{url}/v1/auth/register", "POST", + {"email": "cookie-http@example.com", "password": "secret-123"}) + req = urllib.request.Request( + f"{url}/v1/auth/login", + data=json.dumps({ + "identifier": "cookie-http@example.com", + "password": "secret-123", + }).encode(), + headers={"Content-Type": "application/json"}, + method="POST", + ) + with urllib.request.urlopen(req) as r: + assert json.loads(r.read())["session_token"] + cookie_header = r.headers["Set-Cookie"] + finally: + tracker.stop() + + cookie = http.cookies.SimpleCookie(cookie_header) + session_cookie = cookie["meshnet_session"].OutputString() + + restarted = TrackerServer( + billing=BillingLedger(starting_credit=0.0, default_price_per_1k=0.02), + accounts_db=accounts_db, + starting_credit=0.0, + devnet_topup_amount=0.0, + ) + restarted_port = restarted.start() + restarted_url = f"http://127.0.0.1:{restarted_port}" + try: + req = urllib.request.Request( + f"{restarted_url}/v1/account", + headers={"Cookie": session_cookie}, + method="GET", + ) + with urllib.request.urlopen(req) as r: + me = json.loads(r.read()) + finally: + restarted.stop() + + assert me["account"]["email"] == "cookie-http@example.com" + + +def test_bad_credentials_and_missing_session_are_401(account_tracker): + url, _ = account_tracker + _call(f"{url}/v1/auth/register", "POST", + {"email": "a@example.com", "password": "secret-123"}) + with pytest.raises(urllib.error.HTTPError) as exc_info: + _call(f"{url}/v1/auth/login", "POST", + {"identifier": "a@example.com", "password": "wrong-pass"}) + assert exc_info.value.code == 401 + with pytest.raises(urllib.error.HTTPError) as exc_info: + _call(f"{url}/v1/account") + assert exc_info.value.code == 401 + + +def test_key_create_revoke_and_revoked_key_rejected_by_proxy(account_tracker): + url, _ = account_tracker + reg = _call(f"{url}/v1/auth/register", "POST", + {"email": "k@example.com", "password": "secret-123"}) + token = reg["session_token"] + + new_key = _call(f"{url}/v1/account/keys", "POST", {}, token=token)["api_key"] + me = _call(f"{url}/v1/account", token=token) + assert sorted(me["api_keys"]) == sorted([reg["api_key"], new_key]) + + _call(f"{url}/v1/account/keys/revoke", "POST", {"api_key": new_key}, token=token) + with pytest.raises(urllib.error.HTTPError) as exc_info: + _call(f"{url}/v1/chat/completions", "POST", + {"model": "any", "messages": []}, token=new_key) + assert exc_info.value.code == 401 + assert "revoked" in exc_info.value.read().decode() + + +def test_admin_listing_requires_admin_role(account_tracker): + url, _ = account_tracker + admin = _call(f"{url}/v1/auth/register", "POST", + {"email": "admin@example.com", "password": "secret-123"}) + user = _call(f"{url}/v1/auth/register", "POST", + {"wallet": "WalletUser1", "password": "secret-123"}) + + with pytest.raises(urllib.error.HTTPError) as exc_info: + _call(f"{url}/v1/admin/accounts", token=user["session_token"]) + assert exc_info.value.code == 403 + + listing = _call(f"{url}/v1/admin/accounts", token=admin["session_token"]) + accounts = listing["accounts"] + assert len(accounts) == 2 + assert accounts[0]["role"] == "admin" + assert accounts[1]["wallet"] == "WalletUser1" + assert "balances" in accounts[0] + + +def test_accounts_gossip_endpoint_applies_events(account_tracker): + url, _ = account_tracker + peer = AccountStore() + peer.register(email="remote@example.com", password="secret-123") + events, _ = peer.events_since(0) + body = json.dumps({"events": events}).encode() + req = urllib.request.Request( + f"{url}/v1/accounts/gossip", data=body, + headers={"Content-Type": "application/json", **sign_hive_request(HIVE_SECRET, body)}, + method="POST", + ) + with urllib.request.urlopen(req) as r: + result = json.loads(r.read()) + assert result["applied"] == len(events) + login = _call(f"{url}/v1/auth/login", "POST", + {"identifier": "remote@example.com", "password": "secret-123"}) + assert login["account"]["email"] == "remote@example.com" + + +def test_accounts_endpoints_404_when_disabled(): + tracker = TrackerServer() # no accounts, no billing + port = tracker.start() + try: + with pytest.raises(urllib.error.HTTPError) as exc_info: + _call(f"http://127.0.0.1:{port}/v1/auth/register", "POST", + {"email": "x@example.com", "password": "secret-123"}) + assert exc_info.value.code == 404 + finally: + tracker.stop() + + +# ------------------------------------------- US-039/US-040: credit and top-up + + +@pytest.fixture +def funded_tracker(): + """Tracker with Caller Credit and the devnet top-up faucet enabled.""" + ledger = BillingLedger(starting_credit=0.0, default_price_per_1k=0.02) + tracker = TrackerServer( + billing=ledger, + accounts=AccountStore(), + hive_secret=HIVE_SECRET, + starting_credit=1.0, + devnet_topup_amount=10.0, + ) + port = tracker.start() + yield f"http://127.0.0.1:{port}", ledger + tracker.stop() + + +def test_caller_credit_granted_once_per_account(funded_tracker): + url, ledger = funded_tracker + reg = _call(f"{url}/v1/auth/register", "POST", + {"email": "c@example.com", "password": "secret-123"}) + token = reg["session_token"] + first_key = reg["api_key"] + assert ledger.get_client_balance(first_key) == pytest.approx(1.0) + + # A second key never re-grants — not even after revoking the first. + second = _call(f"{url}/v1/account/keys", "POST", {}, token=token) + assert second["caller_credit_granted"] is False + assert ledger.get_client_balance(second["api_key"]) == pytest.approx(0.0) + _call(f"{url}/v1/account/keys/revoke", "POST", {"api_key": first_key}, token=token) + third = _call(f"{url}/v1/account/keys", "POST", {}, token=token) + assert third["caller_credit_granted"] is False + assert ledger.get_client_balance(third["api_key"]) == pytest.approx(0.0) + + +def test_unknown_bearer_key_rejected_by_proxy(funded_tracker): + url, ledger = funded_tracker + with pytest.raises(urllib.error.HTTPError) as exc_info: + _call(f"{url}/v1/chat/completions", "POST", + {"model": "any", "messages": []}, token="sk-mesh-made-up-key") + assert exc_info.value.code == 401 + assert "unknown API key" in exc_info.value.read().decode() + # The invented key must not have become a billable client. + assert ledger.get_client_balance("sk-mesh-made-up-key") == pytest.approx(0.0) + + +def test_devnet_topup_credits_own_key_only(funded_tracker): + url, ledger = funded_tracker + owner = _call(f"{url}/v1/auth/register", "POST", + {"email": "own@example.com", "password": "secret-123"}) + other = _call(f"{url}/v1/auth/register", "POST", + {"email": "oth@example.com", "password": "secret-123"}) + + me = _call(f"{url}/v1/account", token=owner["session_token"]) + assert me["topup_amount"] == pytest.approx(10.0) + + result = _call(f"{url}/v1/account/topup", "POST", + {"api_key": owner["api_key"]}, token=owner["session_token"]) + assert result["credited"] == pytest.approx(10.0) + assert result["balance"] == pytest.approx(11.0) # 1.0 caller credit + 10.0 + + with pytest.raises(urllib.error.HTTPError) as exc_info: + _call(f"{url}/v1/account/topup", "POST", + {"api_key": owner["api_key"]}, token=other["session_token"]) + assert exc_info.value.code == 403 + assert ledger.get_client_balance(owner["api_key"]) == pytest.approx(11.0) + + +def test_topup_404_when_disabled(account_tracker): + url, _ = account_tracker + reg = _call(f"{url}/v1/auth/register", "POST", + {"email": "t@example.com", "password": "secret-123"}) + me = _call(f"{url}/v1/account", token=reg["session_token"]) + assert me["topup_amount"] == pytest.approx(0.0) + with pytest.raises(urllib.error.HTTPError) as exc_info: + _call(f"{url}/v1/account/topup", "POST", + {"api_key": reg["api_key"]}, token=reg["session_token"]) + assert exc_info.value.code == 404 diff --git a/tests/test_billing_ledger.py b/tests/test_billing_ledger.py index 362a8d8..40e0dff 100644 --- a/tests/test_billing_ledger.py +++ b/tests/test_billing_ledger.py @@ -1,674 +1,674 @@ -"""US-031: billing ledger — per-token pricing, 90/10 split, pending balances. - -Unit tests for BillingLedger math/persistence/replication, plus HTTP -integration: 401 without an API key, 402 for unfunded keys, billed 200 after -explicit credit, and 402 once the balance is exhausted. -""" - -import http.server -import json -import socketserver -import threading -import urllib.error -import urllib.request - -import pytest - -from meshnet_tracker.auth import sign_hive_request -from meshnet_tracker.billing import DEFAULT_STARTING_CREDIT, BillingLedger -from meshnet_tracker.server import ( - TrackerServer, - _billable_non_stream_tokens, - _billable_stream_tokens, - _estimate_prompt_tokens, - _observed_non_stream_completion_tokens, - _observed_stream_tokens, -) - -MODEL = "openai-community/gpt2" -HIVE_SECRET = "test-hive-secret" - - -# ---------------------------------------------------------------- unit tests - - -def test_charge_single_node_gets_90_percent(): - ledger = BillingLedger(starting_credit=1.0, default_price_per_1k=0.02) - ledger.ensure_client("key-a") - event = ledger.charge_request("key-a", MODEL, total_tokens=1000, node_work=[("wallet-1", 12)]) - assert event["cost"] == pytest.approx(0.02) - assert ledger.get_client_balance("key-a") == pytest.approx(1.0 - 0.02) - assert ledger.get_node_pending("wallet-1") == pytest.approx(0.02 * 0.90) - assert ledger.snapshot()["protocol_cut"] == pytest.approx(0.02 * 0.10) - - -def test_default_starting_credit_is_zero_and_fresh_key_is_unfunded(): - ledger = BillingLedger(default_price_per_1k=0.02) - - assert DEFAULT_STARTING_CREDIT == 0.0 - assert ledger.ensure_client("fresh-key") == pytest.approx(0.0) - assert ledger.get_client_balance("fresh-key") == pytest.approx(0.0) - assert ledger.has_funds("fresh-key") is False - assert ledger.events_since(0)[0] == [] - - -def test_charge_three_node_split_by_work_units(): - ledger = BillingLedger(starting_credit=1.0, default_price_per_1k=0.02) - ledger.ensure_client("key-a") - ledger.charge_request( - "key-a", MODEL, total_tokens=1000, - node_work=[("wallet-1", 6), ("wallet-2", 3), ("wallet-3", 3)], - ) - pool = 0.02 * 0.90 - assert ledger.get_node_pending("wallet-1") == pytest.approx(pool * 6 / 12) - assert ledger.get_node_pending("wallet-2") == pytest.approx(pool * 3 / 12) - assert ledger.get_node_pending("wallet-3") == pytest.approx(pool * 3 / 12) - assert ledger.snapshot()["protocol_cut"] == pytest.approx(0.02 * 0.10) - - -def test_walletless_node_share_accrues_to_protocol_cut(): - ledger = BillingLedger(starting_credit=1.0, default_price_per_1k=0.02) - ledger.ensure_client("key-a") - ledger.charge_request( - "key-a", MODEL, total_tokens=1000, - node_work=[("wallet-1", 6), (None, 6)], - ) - pool = 0.02 * 0.90 - assert ledger.get_node_pending("wallet-1") == pytest.approx(pool / 2) - # walletless half of the pool + the 10% cut both land in protocol_cut - assert ledger.snapshot()["protocol_cut"] == pytest.approx(pool / 2 + 0.02 * 0.10) - - -def test_per_model_price_override(): - ledger = BillingLedger( - starting_credit=1.0, - default_price_per_1k=0.02, - prices={MODEL: 0.10}, - ) - ledger.ensure_client("key-a") - event = ledger.charge_request("key-a", MODEL, 500, [("wallet-1", 1)]) - assert event["cost"] == pytest.approx(0.10 * 500 / 1000) - event = ledger.charge_request("key-a", "other-model", 500, [("wallet-1", 1)]) - assert event["cost"] == pytest.approx(0.02 * 500 / 1000) - - -def test_non_stream_billable_tokens_cap_usage_by_request_bound(): - payload = {"usage": {"total_tokens": 1_000_000}} - request = {"messages": [{"role": "user", "content": "hello tracker"}], "max_tokens": 3} - - assert _estimate_prompt_tokens(request) == 2 - assert _billable_non_stream_tokens(payload, request) == 5 - - -def test_non_stream_billable_tokens_fallback_when_usage_missing(): - payload = {"choices": [{"message": {"role": "assistant", "content": "ok now"}}]} - request = {"messages": [{"role": "user", "content": "hello tracker"}], "max_tokens": 10} - - assert _observed_non_stream_completion_tokens(payload) == 2 - assert _billable_non_stream_tokens(payload, request) == 4 - - -def test_stream_billable_tokens_allow_usage_to_lower_observed_count_only(): - observed_payload = { - "choices": [{ - "index": 0, - "delta": {"content": "hello tracker"}, - "finish_reason": None, - }] - } - - assert _observed_stream_tokens(observed_payload) == 2 - assert _billable_stream_tokens(observed_tokens=2, reported_tokens=1_000_000) == 2 - assert _billable_stream_tokens(observed_tokens=2, reported_tokens=1) == 1 - - -def test_payout_and_forfeit_hooks(): - ledger = BillingLedger(starting_credit=1.0, default_price_per_1k=0.02) - ledger.charge_request("key-a", MODEL, 1000, [("wallet-1", 1)]) - pending = ledger.get_node_pending("wallet-1") - ledger.settle_node_payout("wallet-1", pending, reference="tx-sig-1") - assert ledger.get_node_pending("wallet-1") == pytest.approx(0.0) - - ledger.charge_request("key-a", MODEL, 1000, [("wallet-1", 1)]) - cut_before = ledger.snapshot()["protocol_cut"] - forfeited = ledger.forfeit_pending("wallet-1")["amount"] - assert forfeited == pytest.approx(0.02 * 0.90) - assert ledger.get_node_pending("wallet-1") == pytest.approx(0.0) - assert ledger.snapshot()["protocol_cut"] == pytest.approx(cut_before + forfeited) - - -def test_restart_persistence(tmp_path): - db = str(tmp_path / "billing.db") - ledger = BillingLedger(db_path=db, starting_credit=1.0, default_price_per_1k=0.02) - ledger.credit_client("key-a", 5.0) - ledger.charge_request("key-a", MODEL, 1000, [("wallet-1", 12)]) - ledger.save_to_db() - - reloaded = BillingLedger(db_path=db, starting_credit=1.0, default_price_per_1k=0.02) - assert reloaded.get_client_balance("key-a") == pytest.approx( - ledger.get_client_balance("key-a") - ) - assert reloaded.get_node_pending("wallet-1") == pytest.approx(0.02 * 0.90) - assert reloaded.snapshot()["protocol_cut"] == pytest.approx(0.02 * 0.10) - - -def test_tracker_enables_billing_with_default_db_when_requested(tmp_path, monkeypatch): - from meshnet_tracker.billing import DEFAULT_BILLING_DB_PATH - - monkeypatch.chdir(tmp_path) - tracker = TrackerServer(enable_billing=True) - port = tracker.start() - try: - # /v1/billing/summary is admin-gated now; just confirm the server is up. - health = json.loads( - urllib.request.urlopen(f"http://127.0.0.1:{port}/v1/health").read() - ) - assert health["status"] == "ok" - finally: - tracker.stop() - # enabling billing creates the ledger DB in the working directory - assert (tmp_path / DEFAULT_BILLING_DB_PATH).exists() - - -def test_event_replication_converges_and_dedupes(): - leader = BillingLedger(starting_credit=1.0, default_price_per_1k=0.02) - follower = BillingLedger(starting_credit=1.0, default_price_per_1k=0.02) - - leader.credit_client("key-a", 10.0) - leader.charge_request("key-a", MODEL, 2000, [("wallet-1", 8), ("wallet-2", 4)]) - - events, cursor = leader.events_since(0) - assert follower.apply_events(events) == len(events) - # replaying the same batch must be a no-op - assert follower.apply_events(events) == 0 - - assert follower.get_client_balance("key-a") == pytest.approx( - leader.get_client_balance("key-a") - ) - assert follower.get_node_pending("wallet-1") == pytest.approx( - leader.get_node_pending("wallet-1") - ) - assert follower.snapshot()["protocol_cut"] == pytest.approx( - leader.snapshot()["protocol_cut"] - ) - # incremental cursor: nothing new after full sync - more, _ = leader.events_since(cursor) - assert more == [] - - -# ---------------------------------------------------------- HTTP integration - - -class _UsageStubNode: - """Minimal head node returning a chat completion with real usage numbers.""" - - def __init__( - self, - total_tokens: int = 1000, - *, - stream_chunks: list[str] | None = None, - stream_usage_total: int | None = None, - ): - self.total_tokens = total_tokens - self.stream_chunks = stream_chunks or [] - self.stream_usage_total = stream_usage_total - self.request_count = 0 - outer = self - - class Handler(http.server.BaseHTTPRequestHandler): - def log_message(self, fmt, *args): - pass - - def do_POST(self): - outer.request_count += 1 - length = int(self.headers.get("Content-Length", 0)) - raw_body = self.rfile.read(length) - try: - request_body = json.loads(raw_body) - except json.JSONDecodeError: - request_body = {} - if request_body.get("stream"): - self.send_response(200) - self.send_header("Content-Type", "text/event-stream; charset=utf-8") - self.end_headers() - for chunk in outer.stream_chunks: - payload = json.dumps({ - "id": "chatcmpl-stub", - "object": "chat.completion.chunk", - "model": MODEL, - "choices": [{ - "index": 0, - "delta": {"content": chunk}, - "finish_reason": None, - }], - }).encode() - self.wfile.write(b"data: " + payload + b"\n\n") - if outer.stream_usage_total is not None: - usage_payload = json.dumps({ - "id": "chatcmpl-stub", - "object": "chat.completion.chunk", - "model": MODEL, - "choices": [], - "usage": { - "prompt_tokens": 0, - "completion_tokens": outer.stream_usage_total, - "total_tokens": outer.stream_usage_total, - }, - }).encode() - self.wfile.write(b"data: " + usage_payload + b"\n\n") - self.wfile.write(b"data: [DONE]\n\n") - return - body = json.dumps({ - "id": "chatcmpl-stub", - "object": "chat.completion", - "model": MODEL, - "choices": [{ - "index": 0, - "message": {"role": "assistant", "content": "ok"}, - "finish_reason": "stop", - }], - "usage": { - "prompt_tokens": 0, - "completion_tokens": outer.total_tokens, - "total_tokens": outer.total_tokens, - }, - }).encode() - self.send_response(200) - self.send_header("Content-Type", "application/json") - self.send_header("Content-Length", str(len(body))) - self.end_headers() - self.wfile.write(body) - - self._server = socketserver.ThreadingTCPServer(("127.0.0.1", 0), Handler) - self._server.daemon_threads = True - self._thread: threading.Thread | None = None - - def start(self) -> int: - self._thread = threading.Thread(target=self._server.serve_forever, daemon=True) - self._thread.start() - return self._server.server_address[1] - - def stop(self): - self._server.shutdown() - self._server.server_close() - - -@pytest.fixture -def billed_tracker(): - ledger = BillingLedger(starting_credit=0.0, default_price_per_1k=0.02) - tracker = TrackerServer( - model_presets={ - MODEL: { - "layers_start": 0, - "layers_end": 11, - "bytes_per_layer": {"bfloat16": 30 * 1024 * 1024}, - } - }, - billing=ledger, - hive_secret=HIVE_SECRET, - ) - port = tracker.start() - tracker_url = f"http://127.0.0.1:{port}" - - stub = _UsageStubNode(total_tokens=1000) - stub_port = stub.start() - reg = json.dumps({ - "endpoint": f"http://127.0.0.1:{stub_port}", - "shard_start": 0, - "shard_end": 11, - "model": MODEL, - "hardware_profile": {}, - "score": 1.0, - "tracker_mode": True, - "wallet_address": "wallet-head", - }).encode() - req = urllib.request.Request( - f"{tracker_url}/v1/nodes/register", - data=reg, - headers={"Content-Type": "application/json"}, - method="POST", - ) - with urllib.request.urlopen(req) as r: - r.read() - - yield tracker_url, ledger, stub - - stub.stop() - tracker.stop() - - -def _chat(tracker_url: str, api_key: str | None, **body_overrides): - body = { - "model": MODEL, - "messages": [{"role": "user", "content": "hi"}], - } - body.update(body_overrides) - data = json.dumps(body).encode() - headers = {"Content-Type": "application/json"} - if api_key: - headers["Authorization"] = f"Bearer {api_key}" - req = urllib.request.Request( - f"{tracker_url}/v1/chat/completions", - data=data, - headers=headers, - method="POST", - ) - with urllib.request.urlopen(req) as r: - if body.get("stream"): - return r.read().decode() - return json.loads(r.read()) - - -def test_proxy_chat_requires_api_key_when_billing_enabled(billed_tracker): - tracker_url, _, _ = billed_tracker - with pytest.raises(urllib.error.HTTPError) as exc_info: - _chat(tracker_url, api_key=None) - assert exc_info.value.code == 401 - - -def test_proxy_chat_402_for_fresh_key_before_routing(billed_tracker): - tracker_url, ledger, stub = billed_tracker - with pytest.raises(urllib.error.HTTPError) as exc_info: - _chat(tracker_url, api_key="fresh-client") - assert exc_info.value.code == 402 - assert ledger.get_client_balance("fresh-client") == pytest.approx(0.0) - assert stub.request_count == 0 - - -def test_proxy_chat_bills_credited_client_and_credits_node(billed_tracker): - tracker_url, ledger, _ = billed_tracker - ledger.credit_client("client-1", 0.03, note="admin-credit") - _chat(tracker_url, api_key="client-1") - # 1000 tokens at 0.02/1K = 0.02 USDT; explicit credit 0.03 - assert ledger.get_client_balance("client-1") == pytest.approx(0.03 - 0.02) - assert ledger.get_node_pending("wallet-head") == pytest.approx(0.02 * 0.90) - - # /v1/billing/summary is admin-gated now (ADR-0017); auth is covered in - # test_auth_boundary.py, so verify the numbers via the ledger snapshot. - summary = ledger.snapshot() - assert summary["node_pending"]["wallet-head"] == pytest.approx(0.02 * 0.90) - assert summary["protocol_cut"] == pytest.approx(0.02 * 0.10) - - stats = json.loads(urllib.request.urlopen(f"{tracker_url}/v1/stats").read()) - node_stats = next(iter(stats["nodes"].values()))["models"][MODEL] - assert node_stats["tokens_per_sec_last_hour"] is not None - assert node_stats["sample_count_last_hour"] == 1 - - -def test_proxy_chat_caps_inflated_non_streaming_usage_by_request_bounds(billed_tracker): - tracker_url, ledger, stub = billed_tracker - stub.total_tokens = 1_000_000 - ledger.credit_client("bounded-client", 100.0, note="admin-credit") - - _chat(tracker_url, api_key="bounded-client", max_tokens=2) - - # messages=[{"content": "hi"}] has a local prompt estimate of one token, so - # billable total is capped at max_tokens + prompt estimate, not node usage. - expected_tokens = 3 - assert ledger.get_client_balance("bounded-client") == pytest.approx(100.0 - 0.02 * expected_tokens / 1000) - events, _ = ledger.events_since(0) - charge = next(event for event in events if event["type"] == "charge") - assert charge["total_tokens"] == expected_tokens - - -def test_proxy_chat_caps_inflated_streaming_usage_by_observed_chunks(): - ledger = BillingLedger(starting_credit=0.0, default_price_per_1k=0.02) - ledger.credit_client("stream-client", 1.0, note="admin-credit") - tracker = TrackerServer( - model_presets={ - MODEL: { - "layers_start": 0, - "layers_end": 11, - "bytes_per_layer": {"bfloat16": 30 * 1024 * 1024}, - } - }, - billing=ledger, - ) - port = tracker.start() - tracker_url = f"http://127.0.0.1:{port}" - stub = _UsageStubNode(stream_chunks=["one", "two"], stream_usage_total=1_000_000) - stub_port = stub.start() - try: - reg = json.dumps({ - "endpoint": f"http://127.0.0.1:{stub_port}", - "shard_start": 0, - "shard_end": 11, - "model": MODEL, - "hardware_profile": {}, - "score": 1.0, - "tracker_mode": True, - "wallet_address": "wallet-head", - }).encode() - req = urllib.request.Request( - f"{tracker_url}/v1/nodes/register", - data=reg, - headers={"Content-Type": "application/json"}, - method="POST", - ) - with urllib.request.urlopen(req) as r: - r.read() - - _chat(tracker_url, api_key="stream-client", stream=True, max_tokens=2) - - events, _ = ledger.events_since(0) - charge = next(event for event in events if event["type"] == "charge") - assert charge["total_tokens"] == 2 - assert ledger.get_client_balance("stream-client") == pytest.approx(1.0 - 0.02 * 2 / 1000) - finally: - stub.stop() - tracker.stop() - - -def test_proxy_chat_splits_payout_by_tracker_assigned_route_span(): - ledger = BillingLedger(starting_credit=0.0, default_price_per_1k=0.02) - ledger.credit_client("route-client", 1.0, note="admin-credit") - tracker = TrackerServer( - model_presets={ - MODEL: { - "total_layers": 12, - "bytes_per_layer": {"bfloat16": 1_000}, - } - }, - billing=ledger, - ) - port = tracker.start() - tracker_url = f"http://127.0.0.1:{port}" - head = _UsageStubNode(total_tokens=1000) - head_port = head.start() - try: - for endpoint, wallet, vram, bench in ( - (f"http://127.0.0.1:{head_port}", "wallet-head", 5_000, 2.0), - ("http://127.0.0.1:1", "wallet-tail", 10_000, 1.0), - ): - reg = json.dumps({ - "endpoint": endpoint, - "model": MODEL, - "vram_bytes": vram, - "ram_bytes": 10_000, - "quantizations": ["bfloat16"], - "benchmark_tokens_per_sec": bench, - "hardware_profile": {}, - "score": 1.0, - "tracker_mode": endpoint.endswith(str(head_port)), - "wallet_address": wallet, - "managed_assignment": True, - "shard_start": 0, - "shard_end": 999, - }).encode() - req = urllib.request.Request( - f"{tracker_url}/v1/nodes/register", - data=reg, - headers={"Content-Type": "application/json"}, - method="POST", - ) - with urllib.request.urlopen(req) as r: - r.read() - - _chat(tracker_url, api_key="route-client", max_tokens=1000) - - pool = 0.02 * 0.90 - assert ledger.get_node_pending("wallet-head") == pytest.approx(pool * 4 / 12) - assert ledger.get_node_pending("wallet-tail") == pytest.approx(pool * 8 / 12) - finally: - head.stop() - tracker.stop() - - -def test_proxy_chat_402_when_balance_exhausted(billed_tracker): - tracker_url, ledger, _ = billed_tracker - ledger.credit_client("client-2", 0.03, note="admin-credit") - _chat(tracker_url, api_key="client-2") # 0.03 -> 0.01 - _chat(tracker_url, api_key="client-2") # 0.01 -> -0.01 (post-pay drift) - assert ledger.get_client_balance("client-2") == pytest.approx(-0.01) - with pytest.raises(urllib.error.HTTPError) as exc_info: - _chat(tracker_url, api_key="client-2") - assert exc_info.value.code == 402 - # rejected before routing: nothing further was billed - assert ledger.get_client_balance("client-2") == pytest.approx(-0.01) - - -def test_proxy_chat_rejects_request_above_spend_cap_before_routing(): - ledger = BillingLedger(starting_credit=0.0, default_price_per_1k=0.02) - ledger.credit_client("capped-client", 10.0, note="admin-credit") - tracker = TrackerServer( - model_presets={ - MODEL: { - "layers_start": 0, - "layers_end": 11, - "bytes_per_layer": {"bfloat16": 30 * 1024 * 1024}, - } - }, - billing=ledger, - max_charge_per_request=0.01, - ) - port = tracker.start() - tracker_url = f"http://127.0.0.1:{port}" - stub = _UsageStubNode(total_tokens=1000) - stub_port = stub.start() - try: - reg = json.dumps({ - "endpoint": f"http://127.0.0.1:{stub_port}", - "shard_start": 0, - "shard_end": 11, - "model": MODEL, - "hardware_profile": {}, - "score": 1.0, - "tracker_mode": True, - "wallet_address": "wallet-head", - }).encode() - req = urllib.request.Request( - f"{tracker_url}/v1/nodes/register", - data=reg, - headers={"Content-Type": "application/json"}, - method="POST", - ) - with urllib.request.urlopen(req) as r: - r.read() - - with pytest.raises(urllib.error.HTTPError) as exc_info: - _chat( - tracker_url, - api_key="capped-client", - messages=[{"role": "user", "content": " ".join(["prompt"] * 1000)}], - max_tokens=1, - ) - body = json.loads(exc_info.value.read()) - assert exc_info.value.code == 402 - assert body["error"]["code"] == "spend_cap_exceeded" - assert "max_charge_per_request" in body["error"]["message"] - assert ledger.get_client_balance("capped-client") == pytest.approx(10.0) - assert stub.request_count == 0 - finally: - stub.stop() - tracker.stop() - - -def test_proxy_chat_records_validation_event_with_plain_route_metadata(): - class FakeRegistry: - def get_wallet(self, wallet_address): - return type("Wallet", (), {"banned": False})() - - class FakeValidation: - def __init__(self): - self.events = [] - - def record_completed_inference(self, **kwargs): - self.events.append(kwargs) - return kwargs - - class FakeContracts: - def __init__(self): - self.registry = FakeRegistry() - self.validation = FakeValidation() - - contracts = FakeContracts() - tracker = TrackerServer( - model_presets={ - MODEL: { - "layers_start": 0, - "layers_end": 11, - "bytes_per_layer": {"bfloat16": 30 * 1024 * 1024}, - } - }, - contracts=contracts, - ) - port = tracker.start() - tracker_url = f"http://127.0.0.1:{port}" - stub = _UsageStubNode(total_tokens=1000) - stub_port = stub.start() - try: - reg = json.dumps({ - "endpoint": f"http://127.0.0.1:{stub_port}", - "shard_start": 0, - "shard_end": 11, - "model": MODEL, - "hardware_profile": {}, - "score": 1.0, - "tracker_mode": True, - "wallet_address": "wallet-head", - }).encode() - req = urllib.request.Request( - f"{tracker_url}/v1/nodes/register", - data=reg, - headers={"Content-Type": "application/json"}, - method="POST", - ) - with urllib.request.urlopen(req) as r: - r.read() - - _chat(tracker_url, api_key=None) - - assert len(contracts.validation.events) == 1 - event = contracts.validation.events[0] - assert event["model"] == MODEL - assert event["messages"] == [{"role": "user", "content": "hi"}] - assert event["observed_output"] == "ok" - assert event["route_nodes"] == [{ - "node_id": next(iter(tracker._registry)), - "endpoint": f"http://127.0.0.1:{stub_port}", - "wallet_address": "wallet-head", - "shard_start": 0, - "shard_end": 11, - }] - assert "toploc_proof" not in event["route_nodes"][0] - finally: - stub.stop() - tracker.stop() - - -def test_billing_gossip_endpoint_applies_events(billed_tracker): - tracker_url, ledger, _ = billed_tracker - peer = BillingLedger(starting_credit=0.03, default_price_per_1k=0.02) - peer.credit_client("remote-client", 7.0) - events, _ = peer.events_since(0) - body = json.dumps({"events": events}).encode() - req = urllib.request.Request( - f"{tracker_url}/v1/billing/gossip", - data=body, - headers={"Content-Type": "application/json", **sign_hive_request(HIVE_SECRET, body)}, - method="POST", - ) - with urllib.request.urlopen(req) as r: - assert json.loads(r.read())["applied"] == len(events) - # only the replicated credit event lands here. - assert ledger.get_client_balance("remote-client") == pytest.approx(7.0) +"""US-031: billing ledger — per-token pricing, 90/10 split, pending balances. + +Unit tests for BillingLedger math/persistence/replication, plus HTTP +integration: 401 without an API key, 402 for unfunded keys, billed 200 after +explicit credit, and 402 once the balance is exhausted. +""" + +import http.server +import json +import socketserver +import threading +import urllib.error +import urllib.request + +import pytest + +from meshnet_tracker.auth import sign_hive_request +from meshnet_tracker.billing import DEFAULT_STARTING_CREDIT, BillingLedger +from meshnet_tracker.server import ( + TrackerServer, + _billable_non_stream_tokens, + _billable_stream_tokens, + _estimate_prompt_tokens, + _observed_non_stream_completion_tokens, + _observed_stream_tokens, +) + +MODEL = "openai-community/gpt2" +HIVE_SECRET = "test-hive-secret" + + +# ---------------------------------------------------------------- unit tests + + +def test_charge_single_node_gets_90_percent(): + ledger = BillingLedger(starting_credit=1.0, default_price_per_1k=0.02) + ledger.ensure_client("key-a") + event = ledger.charge_request("key-a", MODEL, total_tokens=1000, node_work=[("wallet-1", 12)]) + assert event["cost"] == pytest.approx(0.02) + assert ledger.get_client_balance("key-a") == pytest.approx(1.0 - 0.02) + assert ledger.get_node_pending("wallet-1") == pytest.approx(0.02 * 0.90) + assert ledger.snapshot()["protocol_cut"] == pytest.approx(0.02 * 0.10) + + +def test_default_starting_credit_is_zero_and_fresh_key_is_unfunded(): + ledger = BillingLedger(default_price_per_1k=0.02) + + assert DEFAULT_STARTING_CREDIT == 0.0 + assert ledger.ensure_client("fresh-key") == pytest.approx(0.0) + assert ledger.get_client_balance("fresh-key") == pytest.approx(0.0) + assert ledger.has_funds("fresh-key") is False + assert ledger.events_since(0)[0] == [] + + +def test_charge_three_node_split_by_work_units(): + ledger = BillingLedger(starting_credit=1.0, default_price_per_1k=0.02) + ledger.ensure_client("key-a") + ledger.charge_request( + "key-a", MODEL, total_tokens=1000, + node_work=[("wallet-1", 6), ("wallet-2", 3), ("wallet-3", 3)], + ) + pool = 0.02 * 0.90 + assert ledger.get_node_pending("wallet-1") == pytest.approx(pool * 6 / 12) + assert ledger.get_node_pending("wallet-2") == pytest.approx(pool * 3 / 12) + assert ledger.get_node_pending("wallet-3") == pytest.approx(pool * 3 / 12) + assert ledger.snapshot()["protocol_cut"] == pytest.approx(0.02 * 0.10) + + +def test_walletless_node_share_accrues_to_protocol_cut(): + ledger = BillingLedger(starting_credit=1.0, default_price_per_1k=0.02) + ledger.ensure_client("key-a") + ledger.charge_request( + "key-a", MODEL, total_tokens=1000, + node_work=[("wallet-1", 6), (None, 6)], + ) + pool = 0.02 * 0.90 + assert ledger.get_node_pending("wallet-1") == pytest.approx(pool / 2) + # walletless half of the pool + the 10% cut both land in protocol_cut + assert ledger.snapshot()["protocol_cut"] == pytest.approx(pool / 2 + 0.02 * 0.10) + + +def test_per_model_price_override(): + ledger = BillingLedger( + starting_credit=1.0, + default_price_per_1k=0.02, + prices={MODEL: 0.10}, + ) + ledger.ensure_client("key-a") + event = ledger.charge_request("key-a", MODEL, 500, [("wallet-1", 1)]) + assert event["cost"] == pytest.approx(0.10 * 500 / 1000) + event = ledger.charge_request("key-a", "other-model", 500, [("wallet-1", 1)]) + assert event["cost"] == pytest.approx(0.02 * 500 / 1000) + + +def test_non_stream_billable_tokens_cap_usage_by_request_bound(): + payload = {"usage": {"total_tokens": 1_000_000}} + request = {"messages": [{"role": "user", "content": "hello tracker"}], "max_tokens": 3} + + assert _estimate_prompt_tokens(request) == 2 + assert _billable_non_stream_tokens(payload, request) == 5 + + +def test_non_stream_billable_tokens_fallback_when_usage_missing(): + payload = {"choices": [{"message": {"role": "assistant", "content": "ok now"}}]} + request = {"messages": [{"role": "user", "content": "hello tracker"}], "max_tokens": 10} + + assert _observed_non_stream_completion_tokens(payload) == 2 + assert _billable_non_stream_tokens(payload, request) == 4 + + +def test_stream_billable_tokens_allow_usage_to_lower_observed_count_only(): + observed_payload = { + "choices": [{ + "index": 0, + "delta": {"content": "hello tracker"}, + "finish_reason": None, + }] + } + + assert _observed_stream_tokens(observed_payload) == 2 + assert _billable_stream_tokens(observed_tokens=2, reported_tokens=1_000_000) == 2 + assert _billable_stream_tokens(observed_tokens=2, reported_tokens=1) == 1 + + +def test_payout_and_forfeit_hooks(): + ledger = BillingLedger(starting_credit=1.0, default_price_per_1k=0.02) + ledger.charge_request("key-a", MODEL, 1000, [("wallet-1", 1)]) + pending = ledger.get_node_pending("wallet-1") + ledger.settle_node_payout("wallet-1", pending, reference="tx-sig-1") + assert ledger.get_node_pending("wallet-1") == pytest.approx(0.0) + + ledger.charge_request("key-a", MODEL, 1000, [("wallet-1", 1)]) + cut_before = ledger.snapshot()["protocol_cut"] + forfeited = ledger.forfeit_pending("wallet-1")["amount"] + assert forfeited == pytest.approx(0.02 * 0.90) + assert ledger.get_node_pending("wallet-1") == pytest.approx(0.0) + assert ledger.snapshot()["protocol_cut"] == pytest.approx(cut_before + forfeited) + + +def test_restart_persistence(tmp_path): + db = str(tmp_path / "billing.db") + ledger = BillingLedger(db_path=db, starting_credit=1.0, default_price_per_1k=0.02) + ledger.credit_client("key-a", 5.0) + ledger.charge_request("key-a", MODEL, 1000, [("wallet-1", 12)]) + ledger.save_to_db() + + reloaded = BillingLedger(db_path=db, starting_credit=1.0, default_price_per_1k=0.02) + assert reloaded.get_client_balance("key-a") == pytest.approx( + ledger.get_client_balance("key-a") + ) + assert reloaded.get_node_pending("wallet-1") == pytest.approx(0.02 * 0.90) + assert reloaded.snapshot()["protocol_cut"] == pytest.approx(0.02 * 0.10) + + +def test_tracker_enables_billing_with_default_db_when_requested(tmp_path, monkeypatch): + from meshnet_tracker.billing import DEFAULT_BILLING_DB_PATH + + monkeypatch.chdir(tmp_path) + tracker = TrackerServer(enable_billing=True) + port = tracker.start() + try: + # /v1/billing/summary is admin-gated now; just confirm the server is up. + health = json.loads( + urllib.request.urlopen(f"http://127.0.0.1:{port}/v1/health").read() + ) + assert health["status"] == "ok" + finally: + tracker.stop() + # enabling billing creates the ledger DB in the working directory + assert (tmp_path / DEFAULT_BILLING_DB_PATH).exists() + + +def test_event_replication_converges_and_dedupes(): + leader = BillingLedger(starting_credit=1.0, default_price_per_1k=0.02) + follower = BillingLedger(starting_credit=1.0, default_price_per_1k=0.02) + + leader.credit_client("key-a", 10.0) + leader.charge_request("key-a", MODEL, 2000, [("wallet-1", 8), ("wallet-2", 4)]) + + events, cursor = leader.events_since(0) + assert follower.apply_events(events) == len(events) + # replaying the same batch must be a no-op + assert follower.apply_events(events) == 0 + + assert follower.get_client_balance("key-a") == pytest.approx( + leader.get_client_balance("key-a") + ) + assert follower.get_node_pending("wallet-1") == pytest.approx( + leader.get_node_pending("wallet-1") + ) + assert follower.snapshot()["protocol_cut"] == pytest.approx( + leader.snapshot()["protocol_cut"] + ) + # incremental cursor: nothing new after full sync + more, _ = leader.events_since(cursor) + assert more == [] + + +# ---------------------------------------------------------- HTTP integration + + +class _UsageStubNode: + """Minimal head node returning a chat completion with real usage numbers.""" + + def __init__( + self, + total_tokens: int = 1000, + *, + stream_chunks: list[str] | None = None, + stream_usage_total: int | None = None, + ): + self.total_tokens = total_tokens + self.stream_chunks = stream_chunks or [] + self.stream_usage_total = stream_usage_total + self.request_count = 0 + outer = self + + class Handler(http.server.BaseHTTPRequestHandler): + def log_message(self, fmt, *args): + pass + + def do_POST(self): + outer.request_count += 1 + length = int(self.headers.get("Content-Length", 0)) + raw_body = self.rfile.read(length) + try: + request_body = json.loads(raw_body) + except json.JSONDecodeError: + request_body = {} + if request_body.get("stream"): + self.send_response(200) + self.send_header("Content-Type", "text/event-stream; charset=utf-8") + self.end_headers() + for chunk in outer.stream_chunks: + payload = json.dumps({ + "id": "chatcmpl-stub", + "object": "chat.completion.chunk", + "model": MODEL, + "choices": [{ + "index": 0, + "delta": {"content": chunk}, + "finish_reason": None, + }], + }).encode() + self.wfile.write(b"data: " + payload + b"\n\n") + if outer.stream_usage_total is not None: + usage_payload = json.dumps({ + "id": "chatcmpl-stub", + "object": "chat.completion.chunk", + "model": MODEL, + "choices": [], + "usage": { + "prompt_tokens": 0, + "completion_tokens": outer.stream_usage_total, + "total_tokens": outer.stream_usage_total, + }, + }).encode() + self.wfile.write(b"data: " + usage_payload + b"\n\n") + self.wfile.write(b"data: [DONE]\n\n") + return + body = json.dumps({ + "id": "chatcmpl-stub", + "object": "chat.completion", + "model": MODEL, + "choices": [{ + "index": 0, + "message": {"role": "assistant", "content": "ok"}, + "finish_reason": "stop", + }], + "usage": { + "prompt_tokens": 0, + "completion_tokens": outer.total_tokens, + "total_tokens": outer.total_tokens, + }, + }).encode() + self.send_response(200) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(body))) + self.end_headers() + self.wfile.write(body) + + self._server = socketserver.ThreadingTCPServer(("127.0.0.1", 0), Handler) + self._server.daemon_threads = True + self._thread: threading.Thread | None = None + + def start(self) -> int: + self._thread = threading.Thread(target=self._server.serve_forever, daemon=True) + self._thread.start() + return self._server.server_address[1] + + def stop(self): + self._server.shutdown() + self._server.server_close() + + +@pytest.fixture +def billed_tracker(): + ledger = BillingLedger(starting_credit=0.0, default_price_per_1k=0.02) + tracker = TrackerServer( + model_presets={ + MODEL: { + "layers_start": 0, + "layers_end": 11, + "bytes_per_layer": {"bfloat16": 30 * 1024 * 1024}, + } + }, + billing=ledger, + hive_secret=HIVE_SECRET, + ) + port = tracker.start() + tracker_url = f"http://127.0.0.1:{port}" + + stub = _UsageStubNode(total_tokens=1000) + stub_port = stub.start() + reg = json.dumps({ + "endpoint": f"http://127.0.0.1:{stub_port}", + "shard_start": 0, + "shard_end": 11, + "model": MODEL, + "hardware_profile": {}, + "score": 1.0, + "tracker_mode": True, + "wallet_address": "wallet-head", + }).encode() + req = urllib.request.Request( + f"{tracker_url}/v1/nodes/register", + data=reg, + headers={"Content-Type": "application/json"}, + method="POST", + ) + with urllib.request.urlopen(req) as r: + r.read() + + yield tracker_url, ledger, stub + + stub.stop() + tracker.stop() + + +def _chat(tracker_url: str, api_key: str | None, **body_overrides): + body = { + "model": MODEL, + "messages": [{"role": "user", "content": "hi"}], + } + body.update(body_overrides) + data = json.dumps(body).encode() + headers = {"Content-Type": "application/json"} + if api_key: + headers["Authorization"] = f"Bearer {api_key}" + req = urllib.request.Request( + f"{tracker_url}/v1/chat/completions", + data=data, + headers=headers, + method="POST", + ) + with urllib.request.urlopen(req) as r: + if body.get("stream"): + return r.read().decode() + return json.loads(r.read()) + + +def test_proxy_chat_requires_api_key_when_billing_enabled(billed_tracker): + tracker_url, _, _ = billed_tracker + with pytest.raises(urllib.error.HTTPError) as exc_info: + _chat(tracker_url, api_key=None) + assert exc_info.value.code == 401 + + +def test_proxy_chat_402_for_fresh_key_before_routing(billed_tracker): + tracker_url, ledger, stub = billed_tracker + with pytest.raises(urllib.error.HTTPError) as exc_info: + _chat(tracker_url, api_key="fresh-client") + assert exc_info.value.code == 402 + assert ledger.get_client_balance("fresh-client") == pytest.approx(0.0) + assert stub.request_count == 0 + + +def test_proxy_chat_bills_credited_client_and_credits_node(billed_tracker): + tracker_url, ledger, _ = billed_tracker + ledger.credit_client("client-1", 0.03, note="admin-credit") + _chat(tracker_url, api_key="client-1") + # 1000 tokens at 0.02/1K = 0.02 USDT; explicit credit 0.03 + assert ledger.get_client_balance("client-1") == pytest.approx(0.03 - 0.02) + assert ledger.get_node_pending("wallet-head") == pytest.approx(0.02 * 0.90) + + # /v1/billing/summary is admin-gated now (ADR-0017); auth is covered in + # test_auth_boundary.py, so verify the numbers via the ledger snapshot. + summary = ledger.snapshot() + assert summary["node_pending"]["wallet-head"] == pytest.approx(0.02 * 0.90) + assert summary["protocol_cut"] == pytest.approx(0.02 * 0.10) + + stats = json.loads(urllib.request.urlopen(f"{tracker_url}/v1/stats").read()) + node_stats = next(iter(stats["nodes"].values()))["models"][MODEL] + assert node_stats["tokens_per_sec_last_hour"] is not None + assert node_stats["sample_count_last_hour"] == 1 + + +def test_proxy_chat_caps_inflated_non_streaming_usage_by_request_bounds(billed_tracker): + tracker_url, ledger, stub = billed_tracker + stub.total_tokens = 1_000_000 + ledger.credit_client("bounded-client", 100.0, note="admin-credit") + + _chat(tracker_url, api_key="bounded-client", max_tokens=2) + + # messages=[{"content": "hi"}] has a local prompt estimate of one token, so + # billable total is capped at max_tokens + prompt estimate, not node usage. + expected_tokens = 3 + assert ledger.get_client_balance("bounded-client") == pytest.approx(100.0 - 0.02 * expected_tokens / 1000) + events, _ = ledger.events_since(0) + charge = next(event for event in events if event["type"] == "charge") + assert charge["total_tokens"] == expected_tokens + + +def test_proxy_chat_caps_inflated_streaming_usage_by_observed_chunks(): + ledger = BillingLedger(starting_credit=0.0, default_price_per_1k=0.02) + ledger.credit_client("stream-client", 1.0, note="admin-credit") + tracker = TrackerServer( + model_presets={ + MODEL: { + "layers_start": 0, + "layers_end": 11, + "bytes_per_layer": {"bfloat16": 30 * 1024 * 1024}, + } + }, + billing=ledger, + ) + port = tracker.start() + tracker_url = f"http://127.0.0.1:{port}" + stub = _UsageStubNode(stream_chunks=["one", "two"], stream_usage_total=1_000_000) + stub_port = stub.start() + try: + reg = json.dumps({ + "endpoint": f"http://127.0.0.1:{stub_port}", + "shard_start": 0, + "shard_end": 11, + "model": MODEL, + "hardware_profile": {}, + "score": 1.0, + "tracker_mode": True, + "wallet_address": "wallet-head", + }).encode() + req = urllib.request.Request( + f"{tracker_url}/v1/nodes/register", + data=reg, + headers={"Content-Type": "application/json"}, + method="POST", + ) + with urllib.request.urlopen(req) as r: + r.read() + + _chat(tracker_url, api_key="stream-client", stream=True, max_tokens=2) + + events, _ = ledger.events_since(0) + charge = next(event for event in events if event["type"] == "charge") + assert charge["total_tokens"] == 2 + assert ledger.get_client_balance("stream-client") == pytest.approx(1.0 - 0.02 * 2 / 1000) + finally: + stub.stop() + tracker.stop() + + +def test_proxy_chat_splits_payout_by_tracker_assigned_route_span(): + ledger = BillingLedger(starting_credit=0.0, default_price_per_1k=0.02) + ledger.credit_client("route-client", 1.0, note="admin-credit") + tracker = TrackerServer( + model_presets={ + MODEL: { + "total_layers": 12, + "bytes_per_layer": {"bfloat16": 1_000}, + } + }, + billing=ledger, + ) + port = tracker.start() + tracker_url = f"http://127.0.0.1:{port}" + head = _UsageStubNode(total_tokens=1000) + head_port = head.start() + try: + for endpoint, wallet, vram, bench in ( + (f"http://127.0.0.1:{head_port}", "wallet-head", 5_000, 2.0), + ("http://127.0.0.1:1", "wallet-tail", 10_000, 1.0), + ): + reg = json.dumps({ + "endpoint": endpoint, + "model": MODEL, + "vram_bytes": vram, + "ram_bytes": 10_000, + "quantizations": ["bfloat16"], + "benchmark_tokens_per_sec": bench, + "hardware_profile": {}, + "score": 1.0, + "tracker_mode": endpoint.endswith(str(head_port)), + "wallet_address": wallet, + "managed_assignment": True, + "shard_start": 0, + "shard_end": 999, + }).encode() + req = urllib.request.Request( + f"{tracker_url}/v1/nodes/register", + data=reg, + headers={"Content-Type": "application/json"}, + method="POST", + ) + with urllib.request.urlopen(req) as r: + r.read() + + _chat(tracker_url, api_key="route-client", max_tokens=1000) + + pool = 0.02 * 0.90 + assert ledger.get_node_pending("wallet-head") == pytest.approx(pool * 4 / 12) + assert ledger.get_node_pending("wallet-tail") == pytest.approx(pool * 8 / 12) + finally: + head.stop() + tracker.stop() + + +def test_proxy_chat_402_when_balance_exhausted(billed_tracker): + tracker_url, ledger, _ = billed_tracker + ledger.credit_client("client-2", 0.03, note="admin-credit") + _chat(tracker_url, api_key="client-2") # 0.03 -> 0.01 + _chat(tracker_url, api_key="client-2") # 0.01 -> -0.01 (post-pay drift) + assert ledger.get_client_balance("client-2") == pytest.approx(-0.01) + with pytest.raises(urllib.error.HTTPError) as exc_info: + _chat(tracker_url, api_key="client-2") + assert exc_info.value.code == 402 + # rejected before routing: nothing further was billed + assert ledger.get_client_balance("client-2") == pytest.approx(-0.01) + + +def test_proxy_chat_rejects_request_above_spend_cap_before_routing(): + ledger = BillingLedger(starting_credit=0.0, default_price_per_1k=0.02) + ledger.credit_client("capped-client", 10.0, note="admin-credit") + tracker = TrackerServer( + model_presets={ + MODEL: { + "layers_start": 0, + "layers_end": 11, + "bytes_per_layer": {"bfloat16": 30 * 1024 * 1024}, + } + }, + billing=ledger, + max_charge_per_request=0.01, + ) + port = tracker.start() + tracker_url = f"http://127.0.0.1:{port}" + stub = _UsageStubNode(total_tokens=1000) + stub_port = stub.start() + try: + reg = json.dumps({ + "endpoint": f"http://127.0.0.1:{stub_port}", + "shard_start": 0, + "shard_end": 11, + "model": MODEL, + "hardware_profile": {}, + "score": 1.0, + "tracker_mode": True, + "wallet_address": "wallet-head", + }).encode() + req = urllib.request.Request( + f"{tracker_url}/v1/nodes/register", + data=reg, + headers={"Content-Type": "application/json"}, + method="POST", + ) + with urllib.request.urlopen(req) as r: + r.read() + + with pytest.raises(urllib.error.HTTPError) as exc_info: + _chat( + tracker_url, + api_key="capped-client", + messages=[{"role": "user", "content": " ".join(["prompt"] * 1000)}], + max_tokens=1, + ) + body = json.loads(exc_info.value.read()) + assert exc_info.value.code == 402 + assert body["error"]["code"] == "spend_cap_exceeded" + assert "max_charge_per_request" in body["error"]["message"] + assert ledger.get_client_balance("capped-client") == pytest.approx(10.0) + assert stub.request_count == 0 + finally: + stub.stop() + tracker.stop() + + +def test_proxy_chat_records_validation_event_with_plain_route_metadata(): + class FakeRegistry: + def get_wallet(self, wallet_address): + return type("Wallet", (), {"banned": False})() + + class FakeValidation: + def __init__(self): + self.events = [] + + def record_completed_inference(self, **kwargs): + self.events.append(kwargs) + return kwargs + + class FakeContracts: + def __init__(self): + self.registry = FakeRegistry() + self.validation = FakeValidation() + + contracts = FakeContracts() + tracker = TrackerServer( + model_presets={ + MODEL: { + "layers_start": 0, + "layers_end": 11, + "bytes_per_layer": {"bfloat16": 30 * 1024 * 1024}, + } + }, + contracts=contracts, + ) + port = tracker.start() + tracker_url = f"http://127.0.0.1:{port}" + stub = _UsageStubNode(total_tokens=1000) + stub_port = stub.start() + try: + reg = json.dumps({ + "endpoint": f"http://127.0.0.1:{stub_port}", + "shard_start": 0, + "shard_end": 11, + "model": MODEL, + "hardware_profile": {}, + "score": 1.0, + "tracker_mode": True, + "wallet_address": "wallet-head", + }).encode() + req = urllib.request.Request( + f"{tracker_url}/v1/nodes/register", + data=reg, + headers={"Content-Type": "application/json"}, + method="POST", + ) + with urllib.request.urlopen(req) as r: + r.read() + + _chat(tracker_url, api_key=None) + + assert len(contracts.validation.events) == 1 + event = contracts.validation.events[0] + assert event["model"] == MODEL + assert event["messages"] == [{"role": "user", "content": "hi"}] + assert event["observed_output"] == "ok" + assert event["route_nodes"] == [{ + "node_id": next(iter(tracker._registry)), + "endpoint": f"http://127.0.0.1:{stub_port}", + "wallet_address": "wallet-head", + "shard_start": 0, + "shard_end": 11, + }] + assert "toploc_proof" not in event["route_nodes"][0] + finally: + stub.stop() + tracker.stop() + + +def test_billing_gossip_endpoint_applies_events(billed_tracker): + tracker_url, ledger, _ = billed_tracker + peer = BillingLedger(starting_credit=0.03, default_price_per_1k=0.02) + peer.credit_client("remote-client", 7.0) + events, _ = peer.events_since(0) + body = json.dumps({"events": events}).encode() + req = urllib.request.Request( + f"{tracker_url}/v1/billing/gossip", + data=body, + headers={"Content-Type": "application/json", **sign_hive_request(HIVE_SECRET, body)}, + method="POST", + ) + with urllib.request.urlopen(req) as r: + assert json.loads(r.read())["applied"] == len(events) + # only the replicated credit event lands here. + assert ledger.get_client_balance("remote-client") == pytest.approx(7.0) diff --git a/tests/test_real_model_backend.py b/tests/test_real_model_backend.py index 87caa67..38815fe 100644 --- a/tests/test_real_model_backend.py +++ b/tests/test_real_model_backend.py @@ -1,1051 +1,1051 @@ -"""US-012 tests for the real PyTorch node backend.""" - -import json -import os -from pathlib import Path -import sys -import threading -import time -import types -import urllib.request - -import pytest - -from meshnet_node.model_backend import ( - InsufficientVRAMError, - PartialModelLoadUnsupported, - TensorPayload, - TorchModelShard, - _call_layer, - _checkpoint_tensor_name_for_model, - _load_partial_model_from_snapshot, - _should_partial_materialize_shard, - _decoder_attention_mask, - _int_tensor_header, - build_quantization_config, - validate_quantization, -) -from meshnet_node.torch_server import TorchNodeServer - - -class _FakeBackend: - model_id = "fake-model" - total_layers = 12 - is_head = True - is_tail = False - - def encode_prompt(self, prompt: str) -> TensorPayload: - assert prompt == "The capital of France is" - return TensorPayload( - body=b"\x00" * (1 * 6 * 8 * 2), - shape=[1, 6, 8], - attention_mask_header=None, - position_ids_header=None, - ) - - def forward_bytes(self, body, shape, attention_mask_header, position_ids_header, start_layer=None): - assert shape == [1, 6, 8] - return TensorPayload( - body=body, - shape=shape, - attention_mask_header=attention_mask_header, - position_ids_header=position_ids_header, - ) - - -class _FakeTailBackend(_FakeBackend): - is_head = False - is_tail = True - - def forward_bytes(self, body, shape, attention_mask_header, position_ids_header, start_layer=None): - assert len(body) == 1 * 6 * 8 * 2 - return " Paris" - - -class _FakeFullBackend(_FakeBackend): - is_head = True - is_tail = True - - def generate_text( - self, - messages: list[dict], - max_new_tokens: int = 16, - temperature: float = 1.0, - top_p: float = 1.0, - ) -> str: - assert messages == [{"role": "user", "content": "What is 7 times 8?"}] - assert max_new_tokens == 7 - assert temperature == 1.0 - assert top_p == 1.0 - return "56" - - def count_prompt_tokens(self, messages: list[dict]) -> int: - assert messages == [{"role": "user", "content": "What is 7 times 8?"}] - return 8 - - def count_text_tokens(self, text: str) -> int: - assert text == "56" - return 1 - - -class _FakeChatTokenizer: - eos_token = "" - - def apply_chat_template(self, messages, add_generation_prompt=True, tokenize=False): - assert add_generation_prompt is True - assert tokenize is False - return "debug prompt" - - -class _FakePipelineHeadBackend(_FakeBackend): - tokenizer = _FakeChatTokenizer() - - def encode_prompt(self, prompt: str) -> TensorPayload: - assert prompt.startswith("debug prompt") - return TensorPayload( - body=b"\x00" * (1 * 6 * 8 * 2), - shape=[1, 6, 8], - attention_mask_header=None, - position_ids_header=None, - ) - - -class _FakePipelineTailBackend(_FakeTailBackend): - def __init__(self) -> None: - self.start_layers: list[int | None] = [] - - def forward_bytes(self, body, shape, attention_mask_header, position_ids_header, start_layer=None): - self.start_layers.append(start_layer) - assert len(body) == 1 * 6 * 8 * 2 - return " token" - - -class _BlockingStreamingTailBackend(_FakeTailBackend): - def __init__(self, second_token_release: threading.Event) -> None: - self._release = second_token_release - self.calls = 0 - - def forward_bytes(self, body, shape, attention_mask_header, position_ids_header, start_layer=None): - self.calls += 1 - if self.calls == 1: - return " first" - self._release.wait(timeout=3.0) - return " second" - - -def test_quantization_flag_validation(): - assert validate_quantization("bfloat16") == "bfloat16" - assert validate_quantization("int8") == "int8" - assert validate_quantization("nf4") == "nf4" - with pytest.raises(ValueError, match="quantization"): - validate_quantization("float32") - - -def test_node_package_declares_torch_dependency(): - pyproject = Path("packages/node/pyproject.toml").read_text(encoding="utf-8") - - assert '"torch>=' in pyproject - - -def test_bitsandbytes_configs_are_created_lazily(monkeypatch): - calls = [] - - class FakeBitsAndBytesConfig: - def __init__(self, **kwargs): - calls.append(kwargs) - - monkeypatch.setitem(sys.modules, "torch", types.SimpleNamespace(bfloat16="bf16")) - monkeypatch.setitem( - sys.modules, - "transformers", - types.SimpleNamespace(BitsAndBytesConfig=FakeBitsAndBytesConfig), - ) - - assert build_quantization_config("bfloat16") is None - build_quantization_config("int8") - build_quantization_config("nf4") - - assert calls == [ - {"load_in_8bit": True}, - { - "load_in_4bit": True, - "bnb_4bit_quant_type": "nf4", - "bnb_4bit_compute_dtype": "bf16", - }, - ] - - -def test_head_forward_accepts_text_prompt_and_returns_bfloat16_activations(): - node = TorchNodeServer(backend=_FakeBackend()) - port = node.start() - try: - payload = json.dumps({"prompt": "The capital of France is"}).encode() - req = urllib.request.Request( - f"http://127.0.0.1:{port}/forward", - data=payload, - headers={"Content-Type": "application/json"}, - method="POST", - ) - with urllib.request.urlopen(req, timeout=5) as resp: - body = resp.read() - headers = {key.lower(): value for key, value in resp.headers.items()} - - assert len(body) == 1 * 6 * 8 * 2 - assert headers["x-meshnet-shape"] == "1,6,8" - assert headers["x-meshnet-dtype"] == "bfloat16" - assert headers["x-meshnet-wire"] == "2" - finally: - node.stop() - - -def test_tail_forward_returns_text_completion_from_binary_activations(): - node = TorchNodeServer(backend=_FakeTailBackend()) - port = node.start() - try: - req = urllib.request.Request( - f"http://127.0.0.1:{port}/forward", - data=b"\x00" * (1 * 6 * 8 * 2), - headers={ - "Content-Type": "application/octet-stream", - "X-Meshnet-Shape": "1,6,8", - "X-Meshnet-Dtype": "bfloat16", - "X-Meshnet-Session": "session-1", - "X-Meshnet-Chunk-Index": "0", - "X-Meshnet-Chunk-Total": "1", - "X-Meshnet-Hop-Index": "1", - }, - method="POST", - ) - with urllib.request.urlopen(req, timeout=5) as resp: - body = json.loads(resp.read()) - - assert body == {"text": " Paris"} - assert node.received_activations - assert node.forward_chunk_count == 1 - finally: - node.stop() - - -def test_full_model_chat_completion_uses_generation_not_single_token_decode(capsys): - node = TorchNodeServer(backend=_FakeFullBackend()) - port = node.start() - try: - payload = json.dumps({ - "model": "fake-model", - "messages": [{"role": "user", "content": "What is 7 times 8?"}], - "max_tokens": 7, - }).encode() - req = urllib.request.Request( - f"http://127.0.0.1:{port}/v1/chat/completions", - data=payload, - headers={ - "Content-Type": "application/json", - "X-Meshnet-Request-Id": "req-test-123", - }, - method="POST", - ) - with urllib.request.urlopen(req, timeout=5) as resp: - body = json.loads(resp.read()) - - assert body["choices"][0]["message"]["content"] == "56" - assert body["usage"] == {"prompt_tokens": 8, "completion_tokens": 1, "total_tokens": 9} - finally: - node.stop() - - out = capsys.readouterr().out - assert " [node] processing chat model='fake-model' stream=False max_tokens=7 request_id=req-test-123" in out - assert " [node] chat complete tokens=1 elapsed_s=" in out - - -def test_pipeline_hop_logs_are_suppressed_without_debug(capsys): - tail_backend = _FakePipelineTailBackend() - head = TorchNodeServer(backend=_FakePipelineHeadBackend(), tracker_mode=True) - tail = TorchNodeServer(backend=tail_backend) - head_port = head.start() - tail_port = tail.start() - try: - payload = json.dumps({ - "model": "fake-model", - "messages": [{"role": "user", "content": "hello"}], - "max_tokens": 1, - }).encode() - req = urllib.request.Request( - f"http://127.0.0.1:{head_port}/v1/chat/completions", - data=payload, - headers={ - "Content-Type": "application/json", - "X-Meshnet-Route": json.dumps([ - {"endpoint": f"http://127.0.0.1:{tail_port}", "start_layer": 22}, - ]), - }, - method="POST", - ) - with urllib.request.urlopen(req, timeout=5) as resp: - body = json.loads(resp.read()) - finally: - head.stop() - tail.stop() - - out = capsys.readouterr().out - assert body["choices"][0]["message"]["content"] == " token" - assert tail_backend.start_layers == [22] - assert "pipeline hop 0:" not in out - assert "pipeline hop 0 returned text" not in out - - -def test_pipeline_hop_logs_are_enabled_with_debug(capsys): - head = TorchNodeServer(backend=_FakePipelineHeadBackend(), tracker_mode=True, debug=True) - tail = TorchNodeServer(backend=_FakePipelineTailBackend()) - head_port = head.start() - tail_port = tail.start() - try: - payload = json.dumps({ - "model": "fake-model", - "messages": [{"role": "user", "content": "hello"}], - "max_tokens": 1, - }).encode() - req = urllib.request.Request( - f"http://127.0.0.1:{head_port}/v1/chat/completions", - data=payload, - headers={ - "Content-Type": "application/json", - "X-Meshnet-Route": json.dumps([ - {"endpoint": f"http://127.0.0.1:{tail_port}", "start_layer": 22}, - ]), - }, - method="POST", - ) - with urllib.request.urlopen(req, timeout=5) as resp: - json.loads(resp.read()) - finally: - head.stop() - tail.stop() - - out = capsys.readouterr().out - assert f" [node] pipeline hop 0: http://127.0.0.1:{tail_port} start_layer=22" in out - assert " [node] pipeline hop 0 returned text=' token'" in out - - -def test_split_shard_chat_streams_each_generated_token_incrementally(): - release_second = threading.Event() - head = TorchNodeServer(backend=_FakePipelineHeadBackend(), tracker_mode=True) - tail = TorchNodeServer(backend=_BlockingStreamingTailBackend(release_second)) - head_port = head.start() - tail_port = tail.start() - response = None - try: - payload = json.dumps({ - "model": "fake-model", - "messages": [{"role": "user", "content": "hello"}], - "stream": True, - "max_tokens": 2, - }).encode() - req = urllib.request.Request( - f"http://127.0.0.1:{head_port}/v1/chat/completions", - data=payload, - headers={ - "Content-Type": "application/json", - "X-Meshnet-Route": json.dumps([ - {"endpoint": f"http://127.0.0.1:{tail_port}", "start_layer": 22}, - ]), - }, - method="POST", - ) - response = urllib.request.urlopen(req, timeout=5) - - first_token_line = "" - deadline = time.time() + 2.0 - while time.time() < deadline: - line = response.readline().decode() - if '"content": " first"' in line: - first_token_line = line - break - - assert first_token_line - assert not release_second.is_set() - release_second.set() - rest = response.read().decode() - finally: - release_second.set() - if response is not None: - response.close() - head.stop() - tail.stop() - - assert '"content": " second"' in rest - assert "data: [DONE]" in rest - - -def test_current_requests_snapshot_while_generating(): - release_second = threading.Event() - head = TorchNodeServer(backend=_FakePipelineHeadBackend(), tracker_mode=True) - tail = TorchNodeServer(backend=_BlockingStreamingTailBackend(release_second)) - head_port = head.start() - tail_port = tail.start() - response = None - try: - payload = json.dumps({ - "model": "fake-model", - "messages": [{"role": "user", "content": "hello"}], - "stream": True, - "max_tokens": 2, - }).encode() - req = urllib.request.Request( - f"http://127.0.0.1:{head_port}/v1/chat/completions", - data=payload, - headers={ - "Content-Type": "application/json", - "X-Meshnet-Request-Id": "req-live-1", - "X-Meshnet-Route": json.dumps([ - {"endpoint": f"http://127.0.0.1:{tail_port}", "start_layer": 22}, - ]), - }, - method="POST", - ) - response = urllib.request.urlopen(req, timeout=5) - deadline = time.time() + 2.0 - while time.time() < deadline: - live = head.current_requests - if live and live[0]["request_id"] == "req-live-1" and live[0]["tokens"] >= 1: - break - time.sleep(0.02) - assert head.current_requests - snap = head.current_requests[0] - assert snap["request_id"] == "req-live-1" - assert snap["tokens"] >= 1 - assert snap["tokens_per_sec"] >= 0 - assert snap["routing_complete"] is True - release_second.set() - response.read() - finally: - release_second.set() - if response is not None: - response.close() - head.stop() - tail.stop() - - assert head.current_requests == [] - - -def test_distributed_generating_log_includes_tps(capsys): - head = TorchNodeServer(backend=_FakePipelineHeadBackend(), tracker_mode=True) - tail = TorchNodeServer(backend=_FakePipelineTailBackend()) - head_port = head.start() - tail_port = tail.start() - try: - payload = json.dumps({ - "model": "fake-model", - "messages": [{"role": "user", "content": "hello"}], - "max_tokens": 1, - }).encode() - req = urllib.request.Request( - f"http://127.0.0.1:{head_port}/v1/chat/completions", - data=payload, - headers={ - "Content-Type": "application/json", - "X-Meshnet-Route": json.dumps([ - {"endpoint": f"http://127.0.0.1:{tail_port}", "start_layer": 22}, - ]), - }, - method="POST", - ) - with urllib.request.urlopen(req, timeout=5) as resp: - json.loads(resp.read()) - finally: - head.stop() - tail.stop() - - out = capsys.readouterr().out - assert "generating step=1/1" in out - assert " tps=" in out - assert "generation complete tokens=1" in out - assert out.count("generating step=1/1") == 1 - - -def test_int_tensor_header_serializes_torch_tensors(): - torch = pytest.importorskip("torch") - - header = _int_tensor_header(torch.tensor([[1, 2, 3]], dtype=torch.long)) - - assert header.startswith("1,3:") - - -def test_decoder_attention_mask_is_causal_float_mask(): - torch = pytest.importorskip("torch") - - hidden_states = torch.zeros((1, 3, 8), dtype=torch.bfloat16) - mask = _decoder_attention_mask(torch.ones((1, 3), dtype=torch.long), hidden_states, torch) - - assert mask.shape == (1, 1, 3, 3) - assert mask.dtype == torch.bfloat16 - assert mask[0, 0, 0, 1] < 0 - assert mask[0, 0, 2, 0] == 0 - - -def test_call_layer_passes_rotary_position_embeddings(): - class NeedsPositionEmbeddings: - def __call__(self, hidden_states, **kwargs): - assert kwargs["position_embeddings"] == "rotary" - return hidden_states - - assert _call_layer( - NeedsPositionEmbeddings(), - "hidden", - attention_mask=None, - position_ids="positions", - position_embeddings="rotary", - ) == "hidden" - - -def test_partial_materialize_guard_requires_local_non_full_non_quantized_snapshot(tmp_path): - snapshot_dir = tmp_path / "snapshot" - snapshot_dir.mkdir() - (snapshot_dir / "config.json").write_text("{}") - (snapshot_dir / "model.safetensors.index.json").write_text('{"weight_map": {}}') - - assert _should_partial_materialize_shard( - str(snapshot_dir), - 4, - 7, - total_layers_hint=40, - uses_quantized_weights=False, - ) is True - assert _should_partial_materialize_shard( - str(snapshot_dir), - 0, - 39, - total_layers_hint=40, - uses_quantized_weights=False, - ) is True - assert _should_partial_materialize_shard( - str(snapshot_dir), - 4, - 7, - total_layers_hint=40, - uses_quantized_weights=True, - ) is False - assert _should_partial_materialize_shard( - "repo/model", - 4, - 7, - total_layers_hint=40, - uses_quantized_weights=False, - ) is False - - -def test_checkpoint_tensor_name_remapped_for_text_only_causal_lm(): - class TextOnlyModel: - def __init__(self): - self.model = types.SimpleNamespace(layers=[]) - - model = TextOnlyModel() - assert _checkpoint_tensor_name_for_model( - model, - "model.language_model.layers.0.mlp.gate.weight", - ) == "model.layers.0.mlp.gate.weight" - assert _checkpoint_tensor_name_for_model( - model, - "model.language_model.embed_tokens.weight", - ) == "model.embed_tokens.weight" - - -def test_checkpoint_tensor_name_kept_for_multimodal_backbone(): - class MultimodalModel: - def __init__(self): - self.model = types.SimpleNamespace(language_model=types.SimpleNamespace()) - - model = MultimodalModel() - name = "model.language_model.layers.0.mlp.gate.weight" - assert _checkpoint_tensor_name_for_model(model, name) == name - - -def test_partial_snapshot_loader_remaps_language_model_checkpoint_keys(tmp_path): - snapshot_dir = tmp_path / "snapshot" - snapshot_dir.mkdir() - (snapshot_dir / "config.json").write_text(json.dumps({ - "text_config": {"num_hidden_layers": 3}, - })) - (snapshot_dir / "model.safetensors.index.json").write_text(json.dumps({ - "weight_map": { - "model.language_model.layers.1.self_attn.q_proj.weight": "shard-2.safetensors", - } - })) - (snapshot_dir / "shard-2.safetensors").write_bytes(b"stub") - - class FakeModule: - def __init__(self): - self.to_calls = [] - - def to(self, device): - self.to_calls.append(device) - return self - - class FakeModel: - def __init__(self): - self.model = types.SimpleNamespace( - layers=[FakeModule(), FakeModule(), FakeModule()], - rotary_emb=FakeModule(), - ) - - def tie_weights(self): - pass - - class AutoConfigStub: - @staticmethod - def from_pretrained(model_id): - return types.SimpleNamespace( - text_config=types.SimpleNamespace(num_hidden_layers=3), - get_text_config=lambda: types.SimpleNamespace(num_hidden_layers=3), - ) - - class AutoModelStub: - @staticmethod - def from_config(cfg, torch_dtype=None): - return FakeModel() - - set_calls = [] - - def fake_set_tensor(module, tensor_name, device, value=None, dtype=None): - set_calls.append(tensor_name) - - class FakeSafeOpen: - def __init__(self, filename, framework, device): - self.filename = Path(filename).name - - def __enter__(self): - return self - - def __exit__(self, exc_type, exc, tb): - return False - - def get_tensor(self, tensor_name): - return tensor_name - - class UnusedContext: - def __enter__(self): - return None - - def __exit__(self, exc_type, exc, tb): - return False - - _load_partial_model_from_snapshot( - AutoConfigStub, - AutoModelStub, - types.SimpleNamespace(), - str(snapshot_dir), - 1, - 1, - "bf16", - "cpu:0", - init_empty_weights_fn=UnusedContext, - set_tensor_fn=fake_set_tensor, - safe_open_fn=FakeSafeOpen, - ) - - assert set_calls == ["model.layers.1.self_attn.q_proj.weight"] - - -def test_partial_snapshot_loader_skips_tensors_absent_from_causal_lm(tmp_path): - # Multimodal/MTP checkpoints (Qwen3.5/3.6-MoE) carry mtp.* and model.visual.* - # tensors that the text-only CausalLM never builds — they must be skipped, - # not assigned (assignment raises AttributeError: 'mtp' / 'visual'). - snapshot_dir = tmp_path / "snapshot" - snapshot_dir.mkdir() - (snapshot_dir / "config.json").write_text(json.dumps({ - "text_config": {"num_hidden_layers": 3}, - })) - (snapshot_dir / "model.safetensors.index.json").write_text(json.dumps({ - "weight_map": { - "model.language_model.layers.1.self_attn.q_proj.weight": "shard-2.safetensors", - "mtp.layers.1.input_layernorm.weight": "shard-2.safetensors", - "model.visual.blocks.1.attn.qkv.weight": "shard-2.safetensors", - } - })) - (snapshot_dir / "shard-2.safetensors").write_bytes(b"stub") - - class FakeModule: - def to(self, device): - return self - - class FakeModel: - def __init__(self): - self.model = types.SimpleNamespace( - layers=[FakeModule(), FakeModule(), FakeModule()], - rotary_emb=FakeModule(), - ) - - def tie_weights(self): - pass - - def state_dict(self): - return {"model.layers.1.self_attn.q_proj.weight": None} - - class AutoConfigStub: - @staticmethod - def from_pretrained(model_id): - return types.SimpleNamespace( - text_config=types.SimpleNamespace(num_hidden_layers=3), - get_text_config=lambda: types.SimpleNamespace(num_hidden_layers=3), - ) - - class AutoModelStub: - @staticmethod - def from_config(cfg, torch_dtype=None): - return FakeModel() - - set_calls = [] - - def fake_set_tensor(module, tensor_name, device, value=None, dtype=None): - set_calls.append(tensor_name) - - class FakeSafeOpen: - def __init__(self, filename, framework, device): - pass - - def __enter__(self): - return self - - def __exit__(self, exc_type, exc, tb): - return False - - def get_tensor(self, tensor_name): - return tensor_name - - class UnusedContext: - def __enter__(self): - return None - - def __exit__(self, exc_type, exc, tb): - return False - - _load_partial_model_from_snapshot( - AutoConfigStub, - AutoModelStub, - types.SimpleNamespace(), - str(snapshot_dir), - 1, - 1, - "bf16", - "cpu:0", - init_empty_weights_fn=UnusedContext, - set_tensor_fn=fake_set_tensor, - safe_open_fn=FakeSafeOpen, - ) - - assert set_calls == ["model.layers.1.self_attn.q_proj.weight"] - - -def test_partial_snapshot_loader_materializes_only_assigned_tensors(tmp_path): - snapshot_dir = tmp_path / "snapshot" - snapshot_dir.mkdir() - (snapshot_dir / "config.json").write_text("{}") - (snapshot_dir / "model.safetensors.index.json").write_text(json.dumps({ - "weight_map": { - "model.embed_tokens.weight": "shard-1.safetensors", - "model.layers.0.self_attn.q_proj.weight": "shard-1.safetensors", - "model.layers.1.self_attn.q_proj.weight": "shard-2.safetensors", - "model.layers.2.self_attn.q_proj.weight": "shard-3.safetensors", - "model.norm.weight": "shard-3.safetensors", - "lm_head.weight": "shard-3.safetensors", - } - })) - for rel in ("shard-1.safetensors", "shard-2.safetensors", "shard-3.safetensors"): - (snapshot_dir / rel).write_bytes(b"stub") - - class FakeModule: - def __init__(self, name): - self.name = name - self.to_calls = [] - - def to(self, device): - self.to_calls.append(device) - return self - - class FakeModel: - def __init__(self): - self.model = types.SimpleNamespace( - embed_tokens=FakeModule("embed"), - layers=[FakeModule("layer0"), FakeModule("layer1"), FakeModule("layer2")], - rotary_emb=FakeModule("rotary"), - norm=FakeModule("norm"), - ) - self.lm_head = FakeModule("lm_head") - self.tie_weights_called = 0 - - def tie_weights(self): - self.tie_weights_called += 1 - - class AutoConfigStub: - @staticmethod - def from_pretrained(model_id): - assert model_id == str(snapshot_dir) - return types.SimpleNamespace(num_hidden_layers=3) - - class AutoModelStub: - @staticmethod - def from_config(cfg, torch_dtype=None): - assert cfg.num_hidden_layers == 3 - assert torch_dtype == "bf16" - return FakeModel() - - class EmptyWeights: - def __init__(self): - self.entered = 0 - self.exited = 0 - - def __call__(self): - return self - - def __enter__(self): - self.entered += 1 - return None - - def __exit__(self, exc_type, exc, tb): - self.exited += 1 - return False - - init_empty_weights = EmptyWeights() - set_calls = [] - - def fake_set_tensor(module, tensor_name, device, value=None, dtype=None): - set_calls.append((tensor_name, device, value, dtype)) - - tensors = { - "shard-1.safetensors": { - "model.embed_tokens.weight": "embed", - "model.layers.0.self_attn.q_proj.weight": "layer0", - }, - "shard-2.safetensors": { - "model.layers.1.self_attn.q_proj.weight": "layer1", - }, - "shard-3.safetensors": { - "model.layers.2.self_attn.q_proj.weight": "layer2", - "model.norm.weight": "norm", - "lm_head.weight": "lm_head", - }, - } - - class FakeSafeOpen: - def __init__(self, filename, framework, device): - assert framework == "pt" - assert device == "cpu" - self.filename = Path(filename).name - - def __enter__(self): - return self - - def __exit__(self, exc_type, exc, tb): - return False - - def get_tensor(self, tensor_name): - return tensors[self.filename][tensor_name] - - model = _load_partial_model_from_snapshot( - AutoConfigStub, - AutoModelStub, - types.SimpleNamespace(), - str(snapshot_dir), - 1, - 1, - "bf16", - "cpu:0", - init_empty_weights_fn=init_empty_weights, - set_tensor_fn=fake_set_tensor, - safe_open_fn=FakeSafeOpen, - ) - - assert init_empty_weights.entered == 1 - assert init_empty_weights.exited == 1 - assert model.tie_weights_called == 1 - assert [call[0] for call in set_calls] == ["model.layers.1.self_attn.q_proj.weight"] - assert model.model.layers[1].to_calls == ["cpu:0"] - assert model.model.layers[0].to_calls == [] - assert model.model.layers[2].to_calls == [] - assert model.model.embed_tokens.to_calls == [] - assert model.model.norm.to_calls == [] - assert model.lm_head.to_calls == [] - assert model.model.rotary_emb.to_calls == ["cpu:0"] - - -def test_partial_snapshot_loader_requires_known_layer_count(tmp_path): - snapshot_dir = tmp_path / "snapshot" - snapshot_dir.mkdir() - (snapshot_dir / "config.json").write_text("{}") - (snapshot_dir / "model.safetensors.index.json").write_text(json.dumps({ - "weight_map": {"model.layers.0.self_attn.q_proj.weight": "shard.safetensors"} - })) - (snapshot_dir / "shard.safetensors").write_bytes(b"stub") - - class AutoConfigStub: - @staticmethod - def from_pretrained(model_id): - return types.SimpleNamespace() - - class AutoModelStub: - @staticmethod - def from_config(cfg, torch_dtype=None): - raise AssertionError("from_config should not run without a known layer count") - - class UnusedContext: - def __enter__(self): - return None - - def __exit__(self, exc_type, exc, tb): - return False - - with pytest.raises(PartialModelLoadUnsupported, match="num_hidden_layers"): - _load_partial_model_from_snapshot( - AutoConfigStub, - AutoModelStub, - types.SimpleNamespace(), - str(snapshot_dir), - 0, - 0, - "bf16", - "cpu:0", - init_empty_weights_fn=lambda: UnusedContext(), - set_tensor_fn=lambda *args, **kwargs: None, - safe_open_fn=lambda *args, **kwargs: None, - ) - - -def test_torch_model_shard_prefers_partial_loader_for_local_snapshot(tmp_path, monkeypatch): - import meshnet_node.model_backend as backend - - snapshot_dir = tmp_path / "snapshot" - snapshot_dir.mkdir() - (snapshot_dir / "config.json").write_text("{}") - (snapshot_dir / "model.safetensors.index.json").write_text('{"weight_map": {}}') - - class FakeModel: - def __init__(self): - self.model = types.SimpleNamespace( - layers=[object(), object(), object()], - embed_tokens=object(), - ) - self.config = types.SimpleNamespace(hidden_size=8) - self.eval_called = 0 - - def eval(self): - self.eval_called += 1 - - fake_model = FakeModel() - partial_calls = [] - - class AutoConfigStub: - @staticmethod - def from_pretrained(model_id, cache_dir=None): - return types.SimpleNamespace(num_hidden_layers=3, text_config=types.SimpleNamespace(dtype="torch.bfloat16")) - - class AutoModelStub: - @staticmethod - def from_pretrained(*args, **kwargs): - raise AssertionError("full model load should not run for partial local shards") - - class AutoTokenizerStub: - @staticmethod - def from_pretrained(model_id, cache_dir=None): - assert model_id == str(snapshot_dir) - return types.SimpleNamespace() - - monkeypatch.setitem( - sys.modules, - "torch", - types.SimpleNamespace( - cuda=types.SimpleNamespace(is_available=lambda: False), - device=lambda value: value, - bfloat16="bf16", - ), - ) - monkeypatch.setitem( - sys.modules, - "transformers", - types.SimpleNamespace( - AutoConfig=AutoConfigStub, - AutoModelForCausalLM=AutoModelStub, - AutoTokenizer=AutoTokenizerStub, - ), - ) - monkeypatch.setattr( - backend, - "_load_partial_model_from_snapshot", - lambda *args, **kwargs: partial_calls.append((args, kwargs)) or fake_model, - ) - - shard = TorchModelShard( - "repo/model", - 1, - 1, - quantization="auto", - cache_dir=snapshot_dir, - ) - - assert len(partial_calls) == 1 - assert shard.model is fake_model - assert fake_model.eval_called == 1 - assert shard.total_layers == 3 - assert shard.is_head is False - assert shard.is_tail is False - - -@pytest.mark.integration -def test_two_node_gpt2_completion_is_deterministic(): - if os.environ.get("CI"): - pytest.skip("GPT-2 integration test is skipped in CI") - torch = pytest.importorskip("torch") - pytest.importorskip("transformers") - pytest.importorskip("safetensors") - pytest.importorskip("accelerate") - pytest.importorskip("bitsandbytes") - if not torch.cuda.is_available(): - pytest.skip("GPT-2 integration test requires a CUDA GPU") - - head = TorchNodeServer( - model_id="openai-community/gpt2", - shard_start=0, - shard_end=6, - quantization="bfloat16", - ) - tail = TorchNodeServer( - model_id="openai-community/gpt2", - shard_start=6, - shard_end=12, - quantization="bfloat16", - ) - head_port = head.start() - tail_port = tail.start() - try: - prompt_req = urllib.request.Request( - f"http://127.0.0.1:{head_port}/forward", - data=json.dumps({"prompt": "The capital of France is"}).encode(), - headers={"Content-Type": "application/json"}, - method="POST", - ) - with urllib.request.urlopen(prompt_req, timeout=60) as resp: - activation = resp.read() - head_headers = resp.headers - - tail_req = urllib.request.Request( - f"http://127.0.0.1:{tail_port}/forward", - data=activation, - headers={ - "Content-Type": "application/octet-stream", - "X-Meshnet-Shape": head_headers["X-Meshnet-Shape"], - "X-Meshnet-Dtype": head_headers["X-Meshnet-Dtype"], - "X-Meshnet-Session": "gpt2-session", - "X-Meshnet-Chunk-Index": "0", - "X-Meshnet-Chunk-Total": "1", - "X-Meshnet-Hop-Index": "1", - "X-Meshnet-Attn-Mask": head_headers["X-Meshnet-Attn-Mask"], - "X-Meshnet-Position-Ids": head_headers["X-Meshnet-Position-Ids"], - }, - method="POST", - ) - with urllib.request.urlopen(tail_req, timeout=60) as resp: - body = json.loads(resp.read()) - - assert body["text"].strip() - assert body["text"] == " Paris" - finally: - head.stop() - tail.stop() +"""US-012 tests for the real PyTorch node backend.""" + +import json +import os +from pathlib import Path +import sys +import threading +import time +import types +import urllib.request + +import pytest + +from meshnet_node.model_backend import ( + InsufficientVRAMError, + PartialModelLoadUnsupported, + TensorPayload, + TorchModelShard, + _call_layer, + _checkpoint_tensor_name_for_model, + _load_partial_model_from_snapshot, + _should_partial_materialize_shard, + _decoder_attention_mask, + _int_tensor_header, + build_quantization_config, + validate_quantization, +) +from meshnet_node.torch_server import TorchNodeServer + + +class _FakeBackend: + model_id = "fake-model" + total_layers = 12 + is_head = True + is_tail = False + + def encode_prompt(self, prompt: str) -> TensorPayload: + assert prompt == "The capital of France is" + return TensorPayload( + body=b"\x00" * (1 * 6 * 8 * 2), + shape=[1, 6, 8], + attention_mask_header=None, + position_ids_header=None, + ) + + def forward_bytes(self, body, shape, attention_mask_header, position_ids_header, start_layer=None): + assert shape == [1, 6, 8] + return TensorPayload( + body=body, + shape=shape, + attention_mask_header=attention_mask_header, + position_ids_header=position_ids_header, + ) + + +class _FakeTailBackend(_FakeBackend): + is_head = False + is_tail = True + + def forward_bytes(self, body, shape, attention_mask_header, position_ids_header, start_layer=None): + assert len(body) == 1 * 6 * 8 * 2 + return " Paris" + + +class _FakeFullBackend(_FakeBackend): + is_head = True + is_tail = True + + def generate_text( + self, + messages: list[dict], + max_new_tokens: int = 16, + temperature: float = 1.0, + top_p: float = 1.0, + ) -> str: + assert messages == [{"role": "user", "content": "What is 7 times 8?"}] + assert max_new_tokens == 7 + assert temperature == 1.0 + assert top_p == 1.0 + return "56" + + def count_prompt_tokens(self, messages: list[dict]) -> int: + assert messages == [{"role": "user", "content": "What is 7 times 8?"}] + return 8 + + def count_text_tokens(self, text: str) -> int: + assert text == "56" + return 1 + + +class _FakeChatTokenizer: + eos_token = "" + + def apply_chat_template(self, messages, add_generation_prompt=True, tokenize=False): + assert add_generation_prompt is True + assert tokenize is False + return "debug prompt" + + +class _FakePipelineHeadBackend(_FakeBackend): + tokenizer = _FakeChatTokenizer() + + def encode_prompt(self, prompt: str) -> TensorPayload: + assert prompt.startswith("debug prompt") + return TensorPayload( + body=b"\x00" * (1 * 6 * 8 * 2), + shape=[1, 6, 8], + attention_mask_header=None, + position_ids_header=None, + ) + + +class _FakePipelineTailBackend(_FakeTailBackend): + def __init__(self) -> None: + self.start_layers: list[int | None] = [] + + def forward_bytes(self, body, shape, attention_mask_header, position_ids_header, start_layer=None): + self.start_layers.append(start_layer) + assert len(body) == 1 * 6 * 8 * 2 + return " token" + + +class _BlockingStreamingTailBackend(_FakeTailBackend): + def __init__(self, second_token_release: threading.Event) -> None: + self._release = second_token_release + self.calls = 0 + + def forward_bytes(self, body, shape, attention_mask_header, position_ids_header, start_layer=None): + self.calls += 1 + if self.calls == 1: + return " first" + self._release.wait(timeout=3.0) + return " second" + + +def test_quantization_flag_validation(): + assert validate_quantization("bfloat16") == "bfloat16" + assert validate_quantization("int8") == "int8" + assert validate_quantization("nf4") == "nf4" + with pytest.raises(ValueError, match="quantization"): + validate_quantization("float32") + + +def test_node_package_declares_torch_dependency(): + pyproject = Path("packages/node/pyproject.toml").read_text(encoding="utf-8") + + assert '"torch>=' in pyproject + + +def test_bitsandbytes_configs_are_created_lazily(monkeypatch): + calls = [] + + class FakeBitsAndBytesConfig: + def __init__(self, **kwargs): + calls.append(kwargs) + + monkeypatch.setitem(sys.modules, "torch", types.SimpleNamespace(bfloat16="bf16")) + monkeypatch.setitem( + sys.modules, + "transformers", + types.SimpleNamespace(BitsAndBytesConfig=FakeBitsAndBytesConfig), + ) + + assert build_quantization_config("bfloat16") is None + build_quantization_config("int8") + build_quantization_config("nf4") + + assert calls == [ + {"load_in_8bit": True}, + { + "load_in_4bit": True, + "bnb_4bit_quant_type": "nf4", + "bnb_4bit_compute_dtype": "bf16", + }, + ] + + +def test_head_forward_accepts_text_prompt_and_returns_bfloat16_activations(): + node = TorchNodeServer(backend=_FakeBackend()) + port = node.start() + try: + payload = json.dumps({"prompt": "The capital of France is"}).encode() + req = urllib.request.Request( + f"http://127.0.0.1:{port}/forward", + data=payload, + headers={"Content-Type": "application/json"}, + method="POST", + ) + with urllib.request.urlopen(req, timeout=5) as resp: + body = resp.read() + headers = {key.lower(): value for key, value in resp.headers.items()} + + assert len(body) == 1 * 6 * 8 * 2 + assert headers["x-meshnet-shape"] == "1,6,8" + assert headers["x-meshnet-dtype"] == "bfloat16" + assert headers["x-meshnet-wire"] == "2" + finally: + node.stop() + + +def test_tail_forward_returns_text_completion_from_binary_activations(): + node = TorchNodeServer(backend=_FakeTailBackend()) + port = node.start() + try: + req = urllib.request.Request( + f"http://127.0.0.1:{port}/forward", + data=b"\x00" * (1 * 6 * 8 * 2), + headers={ + "Content-Type": "application/octet-stream", + "X-Meshnet-Shape": "1,6,8", + "X-Meshnet-Dtype": "bfloat16", + "X-Meshnet-Session": "session-1", + "X-Meshnet-Chunk-Index": "0", + "X-Meshnet-Chunk-Total": "1", + "X-Meshnet-Hop-Index": "1", + }, + method="POST", + ) + with urllib.request.urlopen(req, timeout=5) as resp: + body = json.loads(resp.read()) + + assert body == {"text": " Paris"} + assert node.received_activations + assert node.forward_chunk_count == 1 + finally: + node.stop() + + +def test_full_model_chat_completion_uses_generation_not_single_token_decode(capsys): + node = TorchNodeServer(backend=_FakeFullBackend()) + port = node.start() + try: + payload = json.dumps({ + "model": "fake-model", + "messages": [{"role": "user", "content": "What is 7 times 8?"}], + "max_tokens": 7, + }).encode() + req = urllib.request.Request( + f"http://127.0.0.1:{port}/v1/chat/completions", + data=payload, + headers={ + "Content-Type": "application/json", + "X-Meshnet-Request-Id": "req-test-123", + }, + method="POST", + ) + with urllib.request.urlopen(req, timeout=5) as resp: + body = json.loads(resp.read()) + + assert body["choices"][0]["message"]["content"] == "56" + assert body["usage"] == {"prompt_tokens": 8, "completion_tokens": 1, "total_tokens": 9} + finally: + node.stop() + + out = capsys.readouterr().out + assert " [node] processing chat model='fake-model' stream=False max_tokens=7 request_id=req-test-123" in out + assert " [node] chat complete tokens=1 elapsed_s=" in out + + +def test_pipeline_hop_logs_are_suppressed_without_debug(capsys): + tail_backend = _FakePipelineTailBackend() + head = TorchNodeServer(backend=_FakePipelineHeadBackend(), tracker_mode=True) + tail = TorchNodeServer(backend=tail_backend) + head_port = head.start() + tail_port = tail.start() + try: + payload = json.dumps({ + "model": "fake-model", + "messages": [{"role": "user", "content": "hello"}], + "max_tokens": 1, + }).encode() + req = urllib.request.Request( + f"http://127.0.0.1:{head_port}/v1/chat/completions", + data=payload, + headers={ + "Content-Type": "application/json", + "X-Meshnet-Route": json.dumps([ + {"endpoint": f"http://127.0.0.1:{tail_port}", "start_layer": 22}, + ]), + }, + method="POST", + ) + with urllib.request.urlopen(req, timeout=5) as resp: + body = json.loads(resp.read()) + finally: + head.stop() + tail.stop() + + out = capsys.readouterr().out + assert body["choices"][0]["message"]["content"] == " token" + assert tail_backend.start_layers == [22] + assert "pipeline hop 0:" not in out + assert "pipeline hop 0 returned text" not in out + + +def test_pipeline_hop_logs_are_enabled_with_debug(capsys): + head = TorchNodeServer(backend=_FakePipelineHeadBackend(), tracker_mode=True, debug=True) + tail = TorchNodeServer(backend=_FakePipelineTailBackend()) + head_port = head.start() + tail_port = tail.start() + try: + payload = json.dumps({ + "model": "fake-model", + "messages": [{"role": "user", "content": "hello"}], + "max_tokens": 1, + }).encode() + req = urllib.request.Request( + f"http://127.0.0.1:{head_port}/v1/chat/completions", + data=payload, + headers={ + "Content-Type": "application/json", + "X-Meshnet-Route": json.dumps([ + {"endpoint": f"http://127.0.0.1:{tail_port}", "start_layer": 22}, + ]), + }, + method="POST", + ) + with urllib.request.urlopen(req, timeout=5) as resp: + json.loads(resp.read()) + finally: + head.stop() + tail.stop() + + out = capsys.readouterr().out + assert f" [node] pipeline hop 0: http://127.0.0.1:{tail_port} start_layer=22" in out + assert " [node] pipeline hop 0 returned text=' token'" in out + + +def test_split_shard_chat_streams_each_generated_token_incrementally(): + release_second = threading.Event() + head = TorchNodeServer(backend=_FakePipelineHeadBackend(), tracker_mode=True) + tail = TorchNodeServer(backend=_BlockingStreamingTailBackend(release_second)) + head_port = head.start() + tail_port = tail.start() + response = None + try: + payload = json.dumps({ + "model": "fake-model", + "messages": [{"role": "user", "content": "hello"}], + "stream": True, + "max_tokens": 2, + }).encode() + req = urllib.request.Request( + f"http://127.0.0.1:{head_port}/v1/chat/completions", + data=payload, + headers={ + "Content-Type": "application/json", + "X-Meshnet-Route": json.dumps([ + {"endpoint": f"http://127.0.0.1:{tail_port}", "start_layer": 22}, + ]), + }, + method="POST", + ) + response = urllib.request.urlopen(req, timeout=5) + + first_token_line = "" + deadline = time.time() + 2.0 + while time.time() < deadline: + line = response.readline().decode() + if '"content": " first"' in line: + first_token_line = line + break + + assert first_token_line + assert not release_second.is_set() + release_second.set() + rest = response.read().decode() + finally: + release_second.set() + if response is not None: + response.close() + head.stop() + tail.stop() + + assert '"content": " second"' in rest + assert "data: [DONE]" in rest + + +def test_current_requests_snapshot_while_generating(): + release_second = threading.Event() + head = TorchNodeServer(backend=_FakePipelineHeadBackend(), tracker_mode=True) + tail = TorchNodeServer(backend=_BlockingStreamingTailBackend(release_second)) + head_port = head.start() + tail_port = tail.start() + response = None + try: + payload = json.dumps({ + "model": "fake-model", + "messages": [{"role": "user", "content": "hello"}], + "stream": True, + "max_tokens": 2, + }).encode() + req = urllib.request.Request( + f"http://127.0.0.1:{head_port}/v1/chat/completions", + data=payload, + headers={ + "Content-Type": "application/json", + "X-Meshnet-Request-Id": "req-live-1", + "X-Meshnet-Route": json.dumps([ + {"endpoint": f"http://127.0.0.1:{tail_port}", "start_layer": 22}, + ]), + }, + method="POST", + ) + response = urllib.request.urlopen(req, timeout=5) + deadline = time.time() + 2.0 + while time.time() < deadline: + live = head.current_requests + if live and live[0]["request_id"] == "req-live-1" and live[0]["tokens"] >= 1: + break + time.sleep(0.02) + assert head.current_requests + snap = head.current_requests[0] + assert snap["request_id"] == "req-live-1" + assert snap["tokens"] >= 1 + assert snap["tokens_per_sec"] >= 0 + assert snap["routing_complete"] is True + release_second.set() + response.read() + finally: + release_second.set() + if response is not None: + response.close() + head.stop() + tail.stop() + + assert head.current_requests == [] + + +def test_distributed_generating_log_includes_tps(capsys): + head = TorchNodeServer(backend=_FakePipelineHeadBackend(), tracker_mode=True) + tail = TorchNodeServer(backend=_FakePipelineTailBackend()) + head_port = head.start() + tail_port = tail.start() + try: + payload = json.dumps({ + "model": "fake-model", + "messages": [{"role": "user", "content": "hello"}], + "max_tokens": 1, + }).encode() + req = urllib.request.Request( + f"http://127.0.0.1:{head_port}/v1/chat/completions", + data=payload, + headers={ + "Content-Type": "application/json", + "X-Meshnet-Route": json.dumps([ + {"endpoint": f"http://127.0.0.1:{tail_port}", "start_layer": 22}, + ]), + }, + method="POST", + ) + with urllib.request.urlopen(req, timeout=5) as resp: + json.loads(resp.read()) + finally: + head.stop() + tail.stop() + + out = capsys.readouterr().out + assert "generating step=1/1" in out + assert " tps=" in out + assert "generation complete tokens=1" in out + assert out.count("generating step=1/1") == 1 + + +def test_int_tensor_header_serializes_torch_tensors(): + torch = pytest.importorskip("torch") + + header = _int_tensor_header(torch.tensor([[1, 2, 3]], dtype=torch.long)) + + assert header.startswith("1,3:") + + +def test_decoder_attention_mask_is_causal_float_mask(): + torch = pytest.importorskip("torch") + + hidden_states = torch.zeros((1, 3, 8), dtype=torch.bfloat16) + mask = _decoder_attention_mask(torch.ones((1, 3), dtype=torch.long), hidden_states, torch) + + assert mask.shape == (1, 1, 3, 3) + assert mask.dtype == torch.bfloat16 + assert mask[0, 0, 0, 1] < 0 + assert mask[0, 0, 2, 0] == 0 + + +def test_call_layer_passes_rotary_position_embeddings(): + class NeedsPositionEmbeddings: + def __call__(self, hidden_states, **kwargs): + assert kwargs["position_embeddings"] == "rotary" + return hidden_states + + assert _call_layer( + NeedsPositionEmbeddings(), + "hidden", + attention_mask=None, + position_ids="positions", + position_embeddings="rotary", + ) == "hidden" + + +def test_partial_materialize_guard_requires_local_non_full_non_quantized_snapshot(tmp_path): + snapshot_dir = tmp_path / "snapshot" + snapshot_dir.mkdir() + (snapshot_dir / "config.json").write_text("{}") + (snapshot_dir / "model.safetensors.index.json").write_text('{"weight_map": {}}') + + assert _should_partial_materialize_shard( + str(snapshot_dir), + 4, + 7, + total_layers_hint=40, + uses_quantized_weights=False, + ) is True + assert _should_partial_materialize_shard( + str(snapshot_dir), + 0, + 39, + total_layers_hint=40, + uses_quantized_weights=False, + ) is True + assert _should_partial_materialize_shard( + str(snapshot_dir), + 4, + 7, + total_layers_hint=40, + uses_quantized_weights=True, + ) is False + assert _should_partial_materialize_shard( + "repo/model", + 4, + 7, + total_layers_hint=40, + uses_quantized_weights=False, + ) is False + + +def test_checkpoint_tensor_name_remapped_for_text_only_causal_lm(): + class TextOnlyModel: + def __init__(self): + self.model = types.SimpleNamespace(layers=[]) + + model = TextOnlyModel() + assert _checkpoint_tensor_name_for_model( + model, + "model.language_model.layers.0.mlp.gate.weight", + ) == "model.layers.0.mlp.gate.weight" + assert _checkpoint_tensor_name_for_model( + model, + "model.language_model.embed_tokens.weight", + ) == "model.embed_tokens.weight" + + +def test_checkpoint_tensor_name_kept_for_multimodal_backbone(): + class MultimodalModel: + def __init__(self): + self.model = types.SimpleNamespace(language_model=types.SimpleNamespace()) + + model = MultimodalModel() + name = "model.language_model.layers.0.mlp.gate.weight" + assert _checkpoint_tensor_name_for_model(model, name) == name + + +def test_partial_snapshot_loader_remaps_language_model_checkpoint_keys(tmp_path): + snapshot_dir = tmp_path / "snapshot" + snapshot_dir.mkdir() + (snapshot_dir / "config.json").write_text(json.dumps({ + "text_config": {"num_hidden_layers": 3}, + })) + (snapshot_dir / "model.safetensors.index.json").write_text(json.dumps({ + "weight_map": { + "model.language_model.layers.1.self_attn.q_proj.weight": "shard-2.safetensors", + } + })) + (snapshot_dir / "shard-2.safetensors").write_bytes(b"stub") + + class FakeModule: + def __init__(self): + self.to_calls = [] + + def to(self, device): + self.to_calls.append(device) + return self + + class FakeModel: + def __init__(self): + self.model = types.SimpleNamespace( + layers=[FakeModule(), FakeModule(), FakeModule()], + rotary_emb=FakeModule(), + ) + + def tie_weights(self): + pass + + class AutoConfigStub: + @staticmethod + def from_pretrained(model_id): + return types.SimpleNamespace( + text_config=types.SimpleNamespace(num_hidden_layers=3), + get_text_config=lambda: types.SimpleNamespace(num_hidden_layers=3), + ) + + class AutoModelStub: + @staticmethod + def from_config(cfg, torch_dtype=None): + return FakeModel() + + set_calls = [] + + def fake_set_tensor(module, tensor_name, device, value=None, dtype=None): + set_calls.append(tensor_name) + + class FakeSafeOpen: + def __init__(self, filename, framework, device): + self.filename = Path(filename).name + + def __enter__(self): + return self + + def __exit__(self, exc_type, exc, tb): + return False + + def get_tensor(self, tensor_name): + return tensor_name + + class UnusedContext: + def __enter__(self): + return None + + def __exit__(self, exc_type, exc, tb): + return False + + _load_partial_model_from_snapshot( + AutoConfigStub, + AutoModelStub, + types.SimpleNamespace(), + str(snapshot_dir), + 1, + 1, + "bf16", + "cpu:0", + init_empty_weights_fn=UnusedContext, + set_tensor_fn=fake_set_tensor, + safe_open_fn=FakeSafeOpen, + ) + + assert set_calls == ["model.layers.1.self_attn.q_proj.weight"] + + +def test_partial_snapshot_loader_skips_tensors_absent_from_causal_lm(tmp_path): + # Multimodal/MTP checkpoints (Qwen3.5/3.6-MoE) carry mtp.* and model.visual.* + # tensors that the text-only CausalLM never builds — they must be skipped, + # not assigned (assignment raises AttributeError: 'mtp' / 'visual'). + snapshot_dir = tmp_path / "snapshot" + snapshot_dir.mkdir() + (snapshot_dir / "config.json").write_text(json.dumps({ + "text_config": {"num_hidden_layers": 3}, + })) + (snapshot_dir / "model.safetensors.index.json").write_text(json.dumps({ + "weight_map": { + "model.language_model.layers.1.self_attn.q_proj.weight": "shard-2.safetensors", + "mtp.layers.1.input_layernorm.weight": "shard-2.safetensors", + "model.visual.blocks.1.attn.qkv.weight": "shard-2.safetensors", + } + })) + (snapshot_dir / "shard-2.safetensors").write_bytes(b"stub") + + class FakeModule: + def to(self, device): + return self + + class FakeModel: + def __init__(self): + self.model = types.SimpleNamespace( + layers=[FakeModule(), FakeModule(), FakeModule()], + rotary_emb=FakeModule(), + ) + + def tie_weights(self): + pass + + def state_dict(self): + return {"model.layers.1.self_attn.q_proj.weight": None} + + class AutoConfigStub: + @staticmethod + def from_pretrained(model_id): + return types.SimpleNamespace( + text_config=types.SimpleNamespace(num_hidden_layers=3), + get_text_config=lambda: types.SimpleNamespace(num_hidden_layers=3), + ) + + class AutoModelStub: + @staticmethod + def from_config(cfg, torch_dtype=None): + return FakeModel() + + set_calls = [] + + def fake_set_tensor(module, tensor_name, device, value=None, dtype=None): + set_calls.append(tensor_name) + + class FakeSafeOpen: + def __init__(self, filename, framework, device): + pass + + def __enter__(self): + return self + + def __exit__(self, exc_type, exc, tb): + return False + + def get_tensor(self, tensor_name): + return tensor_name + + class UnusedContext: + def __enter__(self): + return None + + def __exit__(self, exc_type, exc, tb): + return False + + _load_partial_model_from_snapshot( + AutoConfigStub, + AutoModelStub, + types.SimpleNamespace(), + str(snapshot_dir), + 1, + 1, + "bf16", + "cpu:0", + init_empty_weights_fn=UnusedContext, + set_tensor_fn=fake_set_tensor, + safe_open_fn=FakeSafeOpen, + ) + + assert set_calls == ["model.layers.1.self_attn.q_proj.weight"] + + +def test_partial_snapshot_loader_materializes_only_assigned_tensors(tmp_path): + snapshot_dir = tmp_path / "snapshot" + snapshot_dir.mkdir() + (snapshot_dir / "config.json").write_text("{}") + (snapshot_dir / "model.safetensors.index.json").write_text(json.dumps({ + "weight_map": { + "model.embed_tokens.weight": "shard-1.safetensors", + "model.layers.0.self_attn.q_proj.weight": "shard-1.safetensors", + "model.layers.1.self_attn.q_proj.weight": "shard-2.safetensors", + "model.layers.2.self_attn.q_proj.weight": "shard-3.safetensors", + "model.norm.weight": "shard-3.safetensors", + "lm_head.weight": "shard-3.safetensors", + } + })) + for rel in ("shard-1.safetensors", "shard-2.safetensors", "shard-3.safetensors"): + (snapshot_dir / rel).write_bytes(b"stub") + + class FakeModule: + def __init__(self, name): + self.name = name + self.to_calls = [] + + def to(self, device): + self.to_calls.append(device) + return self + + class FakeModel: + def __init__(self): + self.model = types.SimpleNamespace( + embed_tokens=FakeModule("embed"), + layers=[FakeModule("layer0"), FakeModule("layer1"), FakeModule("layer2")], + rotary_emb=FakeModule("rotary"), + norm=FakeModule("norm"), + ) + self.lm_head = FakeModule("lm_head") + self.tie_weights_called = 0 + + def tie_weights(self): + self.tie_weights_called += 1 + + class AutoConfigStub: + @staticmethod + def from_pretrained(model_id): + assert model_id == str(snapshot_dir) + return types.SimpleNamespace(num_hidden_layers=3) + + class AutoModelStub: + @staticmethod + def from_config(cfg, torch_dtype=None): + assert cfg.num_hidden_layers == 3 + assert torch_dtype == "bf16" + return FakeModel() + + class EmptyWeights: + def __init__(self): + self.entered = 0 + self.exited = 0 + + def __call__(self): + return self + + def __enter__(self): + self.entered += 1 + return None + + def __exit__(self, exc_type, exc, tb): + self.exited += 1 + return False + + init_empty_weights = EmptyWeights() + set_calls = [] + + def fake_set_tensor(module, tensor_name, device, value=None, dtype=None): + set_calls.append((tensor_name, device, value, dtype)) + + tensors = { + "shard-1.safetensors": { + "model.embed_tokens.weight": "embed", + "model.layers.0.self_attn.q_proj.weight": "layer0", + }, + "shard-2.safetensors": { + "model.layers.1.self_attn.q_proj.weight": "layer1", + }, + "shard-3.safetensors": { + "model.layers.2.self_attn.q_proj.weight": "layer2", + "model.norm.weight": "norm", + "lm_head.weight": "lm_head", + }, + } + + class FakeSafeOpen: + def __init__(self, filename, framework, device): + assert framework == "pt" + assert device == "cpu" + self.filename = Path(filename).name + + def __enter__(self): + return self + + def __exit__(self, exc_type, exc, tb): + return False + + def get_tensor(self, tensor_name): + return tensors[self.filename][tensor_name] + + model = _load_partial_model_from_snapshot( + AutoConfigStub, + AutoModelStub, + types.SimpleNamespace(), + str(snapshot_dir), + 1, + 1, + "bf16", + "cpu:0", + init_empty_weights_fn=init_empty_weights, + set_tensor_fn=fake_set_tensor, + safe_open_fn=FakeSafeOpen, + ) + + assert init_empty_weights.entered == 1 + assert init_empty_weights.exited == 1 + assert model.tie_weights_called == 1 + assert [call[0] for call in set_calls] == ["model.layers.1.self_attn.q_proj.weight"] + assert model.model.layers[1].to_calls == ["cpu:0"] + assert model.model.layers[0].to_calls == [] + assert model.model.layers[2].to_calls == [] + assert model.model.embed_tokens.to_calls == [] + assert model.model.norm.to_calls == [] + assert model.lm_head.to_calls == [] + assert model.model.rotary_emb.to_calls == ["cpu:0"] + + +def test_partial_snapshot_loader_requires_known_layer_count(tmp_path): + snapshot_dir = tmp_path / "snapshot" + snapshot_dir.mkdir() + (snapshot_dir / "config.json").write_text("{}") + (snapshot_dir / "model.safetensors.index.json").write_text(json.dumps({ + "weight_map": {"model.layers.0.self_attn.q_proj.weight": "shard.safetensors"} + })) + (snapshot_dir / "shard.safetensors").write_bytes(b"stub") + + class AutoConfigStub: + @staticmethod + def from_pretrained(model_id): + return types.SimpleNamespace() + + class AutoModelStub: + @staticmethod + def from_config(cfg, torch_dtype=None): + raise AssertionError("from_config should not run without a known layer count") + + class UnusedContext: + def __enter__(self): + return None + + def __exit__(self, exc_type, exc, tb): + return False + + with pytest.raises(PartialModelLoadUnsupported, match="num_hidden_layers"): + _load_partial_model_from_snapshot( + AutoConfigStub, + AutoModelStub, + types.SimpleNamespace(), + str(snapshot_dir), + 0, + 0, + "bf16", + "cpu:0", + init_empty_weights_fn=lambda: UnusedContext(), + set_tensor_fn=lambda *args, **kwargs: None, + safe_open_fn=lambda *args, **kwargs: None, + ) + + +def test_torch_model_shard_prefers_partial_loader_for_local_snapshot(tmp_path, monkeypatch): + import meshnet_node.model_backend as backend + + snapshot_dir = tmp_path / "snapshot" + snapshot_dir.mkdir() + (snapshot_dir / "config.json").write_text("{}") + (snapshot_dir / "model.safetensors.index.json").write_text('{"weight_map": {}}') + + class FakeModel: + def __init__(self): + self.model = types.SimpleNamespace( + layers=[object(), object(), object()], + embed_tokens=object(), + ) + self.config = types.SimpleNamespace(hidden_size=8) + self.eval_called = 0 + + def eval(self): + self.eval_called += 1 + + fake_model = FakeModel() + partial_calls = [] + + class AutoConfigStub: + @staticmethod + def from_pretrained(model_id, cache_dir=None): + return types.SimpleNamespace(num_hidden_layers=3, text_config=types.SimpleNamespace(dtype="torch.bfloat16")) + + class AutoModelStub: + @staticmethod + def from_pretrained(*args, **kwargs): + raise AssertionError("full model load should not run for partial local shards") + + class AutoTokenizerStub: + @staticmethod + def from_pretrained(model_id, cache_dir=None): + assert model_id == str(snapshot_dir) + return types.SimpleNamespace() + + monkeypatch.setitem( + sys.modules, + "torch", + types.SimpleNamespace( + cuda=types.SimpleNamespace(is_available=lambda: False), + device=lambda value: value, + bfloat16="bf16", + ), + ) + monkeypatch.setitem( + sys.modules, + "transformers", + types.SimpleNamespace( + AutoConfig=AutoConfigStub, + AutoModelForCausalLM=AutoModelStub, + AutoTokenizer=AutoTokenizerStub, + ), + ) + monkeypatch.setattr( + backend, + "_load_partial_model_from_snapshot", + lambda *args, **kwargs: partial_calls.append((args, kwargs)) or fake_model, + ) + + shard = TorchModelShard( + "repo/model", + 1, + 1, + quantization="auto", + cache_dir=snapshot_dir, + ) + + assert len(partial_calls) == 1 + assert shard.model is fake_model + assert fake_model.eval_called == 1 + assert shard.total_layers == 3 + assert shard.is_head is False + assert shard.is_tail is False + + +@pytest.mark.integration +def test_two_node_gpt2_completion_is_deterministic(): + if os.environ.get("CI"): + pytest.skip("GPT-2 integration test is skipped in CI") + torch = pytest.importorskip("torch") + pytest.importorskip("transformers") + pytest.importorskip("safetensors") + pytest.importorskip("accelerate") + pytest.importorskip("bitsandbytes") + if not torch.cuda.is_available(): + pytest.skip("GPT-2 integration test requires a CUDA GPU") + + head = TorchNodeServer( + model_id="openai-community/gpt2", + shard_start=0, + shard_end=6, + quantization="bfloat16", + ) + tail = TorchNodeServer( + model_id="openai-community/gpt2", + shard_start=6, + shard_end=12, + quantization="bfloat16", + ) + head_port = head.start() + tail_port = tail.start() + try: + prompt_req = urllib.request.Request( + f"http://127.0.0.1:{head_port}/forward", + data=json.dumps({"prompt": "The capital of France is"}).encode(), + headers={"Content-Type": "application/json"}, + method="POST", + ) + with urllib.request.urlopen(prompt_req, timeout=60) as resp: + activation = resp.read() + head_headers = resp.headers + + tail_req = urllib.request.Request( + f"http://127.0.0.1:{tail_port}/forward", + data=activation, + headers={ + "Content-Type": "application/octet-stream", + "X-Meshnet-Shape": head_headers["X-Meshnet-Shape"], + "X-Meshnet-Dtype": head_headers["X-Meshnet-Dtype"], + "X-Meshnet-Session": "gpt2-session", + "X-Meshnet-Chunk-Index": "0", + "X-Meshnet-Chunk-Total": "1", + "X-Meshnet-Hop-Index": "1", + "X-Meshnet-Attn-Mask": head_headers["X-Meshnet-Attn-Mask"], + "X-Meshnet-Position-Ids": head_headers["X-Meshnet-Position-Ids"], + }, + method="POST", + ) + with urllib.request.urlopen(tail_req, timeout=60) as resp: + body = json.loads(resp.read()) + + assert body["text"].strip() + assert body["text"] == " Paris" + finally: + head.stop() + tail.stop() From 7419ace9262a9d336a54317b503eaa563e5066d0 Mon Sep 17 00:00:00 2001 From: Dobromir Popov Date: Wed, 8 Jul 2026 17:29:23 +0200 Subject: [PATCH 5/9] md --- QUICKSTART.md | 39 +++++++++++++++++++++++++++++---------- 1 file changed, 29 insertions(+), 10 deletions(-) diff --git a/QUICKSTART.md b/QUICKSTART.md index 5ee3989..43040e0 100644 --- a/QUICKSTART.md +++ b/QUICKSTART.md @@ -7,28 +7,47 @@ Tested on: AMD Ryzen AI Max (Strix Halo APU), 124 GB RAM, Linux, CPU inference. ## Prerequisites +Choose the shell you will actually run `meshnet-*` from. Editable installs point +the command wrappers at this source tree, so normal code edits are picked up +without reinstalling. + +**Linux / WSL** + ```bash # Clone and enter repo cd /run/media/popov/d/DEV/repos/d-popov.com/AI # Create the virtualenv if it does not exist yet python3 -m venv .venv - +source .venv/bin/activate # Keep packaging tools current enough for editable installs .venv/bin/python -m pip install --upgrade pip setuptools wheel - # Install Python packages (editable — picks up code changes immediately) -.venv/bin/pip install -e packages/tracker -e packages/node -e packages/p2p -e packages/gateway -e packages/relay - -# CPU-only PyTorch (skip if you have CUDA/ROCm already) -.venv/bin/pip install torch --index-url https://download.pytorch.org/whl/cpu - +.venv/bin/python -m pip install -e packages/tracker -e packages/node -e packages/p2p -e packages/gateway -e packages/relay +# PyTorch: choose one +.venv/bin/python -m pip install torch +# .venv/bin/python -m pip install torch --index-url https://download.pytorch.org/whl/cpu +# .venv/bin/python -m pip install torch --index-url https://download.pytorch.org/whl/rocm6.2 # HuggingFace model libraries -.venv/bin/pip install "transformers>=5.12" accelerate +.venv/bin/python -m pip install "transformers>=5.12" accelerate ``` -> **NVIDIA GPU (CUDA):** replace the torch line with `pip install torch` (default index). -> **AMD GPU (ROCm):** `pip install torch --index-url https://download.pytorch.org/whl/rocm6.2` +**Windows PowerShell** + +```powershell +cd D:\DEV\workspace\REPOS\git.d-popov.com\neuron-tai +python -m venv .venv +.\.venv\Scripts\Activate.ps1 +.\.venv\Scripts\python.exe -m pip install --upgrade pip setuptools wheel +.\.venv\Scripts\python.exe -m pip install -e .\packages\tracker -e .\packages\node -e .\packages\p2p -e .\packages\gateway -e .\packages\relay +# PyTorch: choose one +.\.venv\Scripts\python.exe -m pip install torch +# .\.venv\Scripts\python.exe -m pip install torch --index-url https://download.pytorch.org/whl/cpu +# .\.venv\Scripts\python.exe -m pip install torch --index-url https://download.pytorch.org/whl/rocm6.2 +.\.venv\Scripts\python.exe -m pip install "transformers>=5.12" accelerate +``` + +> Torch choices: default index for NVIDIA/CUDA, CPU index for CPU-only, ROCm index for AMD GPU. ### Version and library notes for Qwen3.5/3.6-MoE models From 194fa1d926497bec908ce54fbf1f51eec18b00bd Mon Sep 17 00:00:00 2001 From: Dobromir Popov Date: Wed, 8 Jul 2026 17:38:00 +0200 Subject: [PATCH 6/9] Flatten QUICKSTART commands to single lines for easier copy-paste. Co-authored-by: Cursor --- QUICKSTART.md | 111 +++++++++----------------------------------------- 1 file changed, 19 insertions(+), 92 deletions(-) diff --git a/QUICKSTART.md b/QUICKSTART.md index 43040e0..fac8cd0 100644 --- a/QUICKSTART.md +++ b/QUICKSTART.md @@ -131,10 +131,7 @@ URL that nodes should use for outbound relay connections: .venv/bin/meshnet-relay --host 0.0.0.0 --port 8765 # Terminal 2 — tracker -.venv/bin/meshnet-tracker start \ - --host 0.0.0.0 \ - --port 8081 \ - --relay-url wss://ai.neuron.d-popov.com/ws +.venv/bin/meshnet-tracker start --host 0.0.0.0 --port 8081 --relay-url wss://ai.neuron.d-popov.com/ws ``` If this host sits behind Nginx Proxy Manager, point `/` and `/v1/*` at tracker @@ -249,9 +246,7 @@ meshnet-node start --tracker https://ai.neuron.d-popov.com --model Qwen/Qwen2.5- If the wrong entry point is shadowing, invoke via the full conda path: ```powershell -C:\Users\popov\miniforge3\Scripts\meshnet-node.exe start ` - --tracker https://ai.neuron.d-popov.com ` - --model Qwen/Qwen2.5-0.5B-Instruct +C:\Users\popov\miniforge3\Scripts\meshnet-node.exe start --tracker https://ai.neuron.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct ``` #### Option B — isolated virtualenv (fresh machine, no existing torch) @@ -297,12 +292,7 @@ Use the IPv4 address on the active Ethernet/Wi-Fi adapter, for example Administrator once: ```powershell -New-NetFirewallRule ` - -DisplayName "Meshnet node 8005" ` - -Direction Inbound ` - -Action Allow ` - -Protocol TCP ` - -LocalPort 8005 +New-NetFirewallRule -DisplayName "Meshnet node 8005" -Direction Inbound -Action Allow -Protocol TCP -LocalPort 8005 ``` 5. Start the Windows node from normal PowerShell. Replace the tracker and @@ -310,18 +300,10 @@ advertised host values with your actual LAN addresses: ```powershell $env:HF_HOME = "D:\DEV\models" - -.\.venv\Scripts\meshnet-node.exe start ` - --tracker http://192.168.0.179:8081 ` - --model Qwen/Qwen2.5-0.5B-Instruct ` - --shard-start 12 --shard-end 23 ` - --quantization bfloat16 ` - --host 0.0.0.0 ` - --advertise-host 192.168.0.42 ` - --port 8005 +.\.venv\Scripts\meshnet-node.exe start --tracker http://192.168.0.179:8081 --model Qwen/Qwen2.5-0.5B-Instruct --shard-start 12 --shard-end 23 --quantization bfloat16 --host 0.0.0.0 --advertise-host 192.168.0.42 --port 8005 ``` -One-line variants (direct LAN — node must be reachable by IP from other machines): +Other start examples (direct LAN — node must be reachable by IP from other machines): ```powershell .\.venv\Scripts\meshnet-node.exe start --tracker http://192.168.0.179:8081 --model Qwen/Qwen2.5-0.5B-Instruct --advertise-host 192.168.0.20 @@ -466,10 +448,7 @@ After NPM is correct, start relay and tracker on the LAN machine: .venv/bin/meshnet-relay --host 0.0.0.0 --port 8765 # Terminal 2 — tracker (advertises relay to nodes) -.venv/bin/meshnet-tracker start \ - --host 0.0.0.0 \ - --port 8081 \ - --relay-url wss://ai.neuron.d-popov.com/ws +.venv/bin/meshnet-tracker start --host 0.0.0.0 --port 8081 --relay-url wss://ai.neuron.d-popov.com/ws ``` Nodes using `https://ai.neuron.d-popov.com` should then log: @@ -491,9 +470,7 @@ opens a persistent outbound WebSocket. If the relay connection drops, the node keeps retrying it. ```bash -.venv/bin/meshnet-node start \ - --tracker https://ai.neuron.d-popov.com \ - --model Qwen/Qwen2.5-0.5B-Instruct +.venv/bin/meshnet-node start --tracker https://ai.neuron.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct ``` No authentication is required in the prototype. The first public node for a model @@ -555,16 +532,10 @@ In WSL2 (which gets a `172.x.x.x` virtual IP — unreachable from other machines ```bash # WSL2 Terminal 1 — head node (layers 0–11, handles chat requests) -.venv/bin/meshnet-node start \ - --tracker https://ai.neuron.d-popov.com \ - --model Qwen/Qwen2.5-0.5B-Instruct \ - --shard-start 0 --shard-end 11 +.venv/bin/meshnet-node start --tracker https://ai.neuron.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct --shard-start 0 --shard-end 11 # WSL2 Terminal 2 — tail node (layers 12–23) -.venv/bin/meshnet-node start \ - --tracker https://ai.neuron.d-popov.com \ - --model Qwen/Qwen2.5-0.5B-Instruct \ - --shard-start 12 --shard-end 23 +.venv/bin/meshnet-node start --tracker https://ai.neuron.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct --shard-start 12 --shard-end 23 ``` Both nodes connect to the relay automatically. When a chat request arrives at Node A, @@ -573,14 +544,7 @@ it forwards activations to Node B via `wss://ai.neuron.d-popov.com/rpc/{peer_id_ Send inference through the tracker (which picks the head node and injects the route): ```bash -curl -s https://ai.neuron.d-popov.com/v1/chat/completions \ - -H "Content-Type: application/json" \ - -H "Authorization: Bearer sk-mesh-" \ - -d '{ - "model": "Qwen/Qwen2.5-0.5B-Instruct", - "messages": [{"role": "user", "content": "What is 7 times 8?"}], - "stream": false - }' | python3 -m json.tool +curl -s https://ai.neuron.d-popov.com/v1/chat/completions -H "Content-Type: application/json" -H "Authorization: Bearer sk-mesh-" -d '{"model": "Qwen/Qwen2.5-0.5B-Instruct", "messages": [{"role": "user", "content": "What is 7 times 8?"}], "stream": false}' | python3 -m json.tool ``` Or send directly to Node A's local port (within WSL): @@ -608,24 +572,18 @@ refill during testing. ```bash # 1. Register (once) -curl -s https:///v1/auth/register \ - -H "Content-Type: application/json" \ - -d '{"email": "you@example.com", "password": "hunter22-or-better"}' +curl -s https:///v1/auth/register -H "Content-Type: application/json" -d '{"email": "you@example.com", "password": "hunter22-or-better"}' # → {"session_token": "...", ...} # 2. Create an API key (session token from step 1) -curl -s https:///v1/account/keys -X POST \ - -H "Authorization: Bearer " +curl -s https:///v1/account/keys -X POST -H "Authorization: Bearer " # → {"api_key": "sk-mesh-...", "caller_credit_granted": true} # 3. Check balance / usage curl -s https:///v1/account -H "Authorization: Bearer " # 4. (devnet trackers only) top up a key -curl -s https:///v1/account/topup -X POST \ - -H "Authorization: Bearer " \ - -H "Content-Type: application/json" \ - -d '{"api_key": "sk-mesh-..."}' +curl -s https:///v1/account/topup -X POST -H "Authorization: Bearer " -H "Content-Type: application/json" -d '{"api_key": "sk-mesh-..."}' ``` Operator side: both features default to 1 USDT (`--starting-credit` / @@ -661,12 +619,7 @@ Keep this terminal open. ```bash cd /run/media/popov/d/DEV/repos/d-popov.com/AI -HF_HOME=/run/media/popov/d/DEV/models \ -.venv/bin/meshnet-node start \ - --model Qwen/Qwen2.5-0.5B-Instruct \ - --quantization bfloat16 \ - --tracker http://localhost:8080 \ - --port 8001 +HF_HOME=/run/media/popov/d/DEV/models .venv/bin/meshnet-node start --model Qwen/Qwen2.5-0.5B-Instruct --quantization bfloat16 --tracker http://localhost:8080 --port 8001 ``` Shard range is **auto-detected** from the curated catalog (no network call for known @@ -708,13 +661,7 @@ If you started the node with `--port 8001`, send the request directly to that head node: ```bash Qwen2.5-0.5B-Instruct -curl -s http://localhost:8001/v1/chat/completions \ - -H "Content-Type: application/json" \ - -d '{ - "model": "qwen2.5-0.5b", - "messages": [{"role": "user", "content": "What is 7 times 8? Answer in one word."}], - "stream": false - }' | python3 -m json.tool +curl -s http://localhost:8001/v1/chat/completions -H "Content-Type: application/json" -d '{"model": "qwen2.5-0.5b", "messages": [{"role": "user", "content": "What is 7 times 8? Answer in one word."}], "stream": false}' | python3 -m json.tool ``` If you did not pass `--port`, `meshnet-node start` uses the first free port at @@ -724,21 +671,13 @@ To test tracker routing/proxying, send the same OpenAI-compatible request to the tracker, using either the full HuggingFace repo or the quick alias: ```bash -curl -s http://localhost:8080/v1/chat/completions \ - -H "Content-Type: application/json" \ - -d '{ - "model": "qwen2.5-0.5b", - "messages": [{"role": "user", "content": "What is 7 times 8? Answer in one word."}], - "stream": false - }' | python3 -m json.tool +curl -s http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{"model": "qwen2.5-0.5b", "messages": [{"role": "user", "content": "What is 7 times 8? Answer in one word."}], "stream": false}' | python3 -m json.tool ``` Or use the test script: ```bash -.venv/bin/python scripts/test_lan_inference.py \ - --tracker http://localhost:8080 \ - --gateway http://localhost:8001 +.venv/bin/python scripts/test_lan_inference.py --tracker http://localhost:8080 --gateway http://localhost:8001 ``` --- @@ -749,24 +688,12 @@ Split Qwen2.5-0.5B's 24 layers across two node processes to test the sharded pip **Node A — layers 0–11 (tracker mode, serves chat completions):** ```bash -HF_HOME=/run/media/popov/d/DEV/models \ -.venv/bin/meshnet-node start \ - --model Qwen/Qwen2.5-0.5B-Instruct \ - --shard-start 0 --shard-end 11 \ - --quantization bfloat16 \ - --tracker http://localhost:8080 \ - --port 8001 +HF_HOME=/run/media/popov/d/DEV/models .venv/bin/meshnet-node start --model Qwen/Qwen2.5-0.5B-Instruct --shard-start 0 --shard-end 11 --quantization bfloat16 --tracker http://localhost:8080 --port 8001 ``` **Node B — layers 12–23:** ```bash -HF_HOME=/run/media/popov/d/DEV/models \ -.venv/bin/meshnet-node start \ - --model Qwen/Qwen2.5-0.5B-Instruct \ - --shard-start 12 --shard-end 23 \ - --quantization bfloat16 \ - --tracker http://localhost:8080 \ - --port 8002 +HF_HOME=/run/media/popov/d/DEV/models .venv/bin/meshnet-node start --model Qwen/Qwen2.5-0.5B-Instruct --shard-start 12 --shard-end 23 --quantization bfloat16 --tracker http://localhost:8080 --port 8002 ``` Send the request to Node A — it tokenizes, runs layers 0–11, passes binary From e06969fcb5169fedcd01f2e0c853f348506788f0 Mon Sep 17 00:00:00 2001 From: Dobromir Popov Date: Wed, 8 Jul 2026 17:59:08 +0200 Subject: [PATCH 7/9] md rework. new code --- .claude/memory/project-status.md | 2 +- QUICKSTART.md | 926 ++++++++------------- packages/node/meshnet_node/startup.py | 91 +- packages/tracker/meshnet_tracker/server.py | 17 +- tests/test_node_startup.py | 109 +++ 5 files changed, 563 insertions(+), 582 deletions(-) diff --git a/.claude/memory/project-status.md b/.claude/memory/project-status.md index cb015bc..4334a9b 100644 --- a/.claude/memory/project-status.md +++ b/.claude/memory/project-status.md @@ -42,7 +42,7 @@ Historical handoff note: `/mnt/c/Users/popov/Downloads/neuron-tai-alpha-handoff- - Verification: downloader/startup targeted subset passes (`pytest tests/test_node_startup.py -k "download_shard or same_shard"`). Full `tests/test_node_startup.py` has 46 passed and 4 unrelated Windows chmod/path separator failures. - Live Windows confirmation: `meshnet-node start --tracker http://192.168.0.179:8080 --model Qwen3.6-35B-A3B` reuses `F:\_STORAGE\models\qwen3.6-35b-a3b`, prints `Cached at`, registers, and reaches ready as node `5gMLrmyB-26b1f8a4204a`. - Follow-up fix: preset-model startup now starts the heartbeat thread after registration; without this, the node appeared briefly on the dashboard and was purged on first inference/route after heartbeat expiry. Tracker dashboard now has a "Console output" panel backed by `/v1/console` for node register/expiry, routing failures, and proxy events. -- Qwen3.6-35B-A3B reserve-based split is expected: an 79 GB CPU node may be assigned layers 0-36, and a second node fills 37-39. Do not "fix" this by bypassing the 20% assignment reserve unless the shard-planning policy changes. +- Qwen3.6-35B-A3B CPU runtime cap (2026-07-08): the old reserve-based split could assign an 79 GB CPU node layers 0-36, but real partial loading can exceed that budget and die without a Python traceback. Node startup now clips oversized CPU auto-assignments before loading, and tracker CPU assignment uses a stricter runtime headroom factor; do not revert this to the old 20% reserve-only policy. - Route hardening: tracker chat proxy and `/v1/route` diagnostics now use alias-aware preset node matching for split Qwen3.6 routes; dashboard derives grouped inference history from proxy route/complete console events and shows observed TPS after completion. - Live proxy hardening: model lookup trims outer whitespace before alias matching (`qwen3.6-35b-a3b ` resolves), and tracker route logs/dashboard queue depth combine heartbeat queue with tracker-local proxy in-flight counts so Postman-style bursts no longer show every selected route as queue `0`. - Split-shard streaming hardening: Qwen3.6-style distributed generation now emits SSE chunks token-by-token from the head node instead of buffering all generated text until completion. Tracker direct/relay stream proxy logs `proxy progress` with live tokens/TPS, dashboard Inference history shows currently processing requests with live TPS/tokens/queue, and relay stream completion no longer references an undefined `session_id`. diff --git a/QUICKSTART.md b/QUICKSTART.md index fac8cd0..cfc7360 100644 --- a/QUICKSTART.md +++ b/QUICKSTART.md @@ -1,38 +1,74 @@ # Quickstart — Running a node and testing inference -This guide gets you from zero to a live inference request in three terminals. +Get from zero to a live inference request in **three terminals**: install once, start +the tracker, start a node, send a request. + Tested on: AMD Ryzen AI Max (Strix Halo APU), 124 GB RAM, Linux, CPU inference. +**Active development models** (what we run day-to-day): + +| Role | `--model` / alias | HF repo | Notes | +|------|-------------------|---------|-------| +| Smoke tests, small splits | `Qwen/Qwen2.5-0.5B-Instruct` | same | 24 layers, ~1 GB BF16, no gating — default for new setups | +| Alpha / production target | `qwen3.6-35b-a3b` | `unsloth/Qwen3.6-35B-A3B` | 40 layers, ~72 GB BF16, hybrid linear-attention MoE; aliases include `Qwen3.6-35B-A3B`, `Qwen/Qwen3.6-35B-A3B` | + --- -## Prerequisites +## At a glance -Choose the shell you will actually run `meshnet-*` from. Editable installs point -the command wrappers at this source tree, so normal code edits are picked up -without reinstalling. +| Step | What | Terminals | +|------|------|-----------| +| **0** | Install Python packages | once per machine | +| **1** | Start tracker (and relay if needed) | 1–2 | +| **2** | Start node(s) | 1+ | +| **3** | Send inference request | 1 | -**Linux / WSL** +**Pick your connectivity mode** — this determines which flags you need on the node: + +| Mode | When to use | Tracker URL | Node extras | +|------|-------------|-------------|-------------| +| **Local dev** | Everything on one machine | `http://localhost:8080` | none | +| **Direct LAN** | Node has a real LAN IP other machines can reach | `http://:8080` | `--host 0.0.0.0 --advertise-host ` + firewall | +| **Relay / public** | WSL2, NAT, 5G, or any unreachable inbound port | `https://ai.neuron.d-popov.com` (or your public URL) | none — relay handles routing | + +> **WSL2:** not reachable from other LAN machines by default. Use the **relay / public** +> tracker URL, or run the node in native Windows PowerShell with direct LAN mode. + +**Command prefix by shell** (used in examples below): + +| Shell | Prefix | Model cache env | +|-------|--------|-----------------| +| Linux / WSL | `.venv/bin/` | `HF_HOME=/path/to/models` | +| Windows PowerShell | `.\.venv\Scripts\` | `$env:HF_HOME = "D:\DEV\models"` | + +--- + +## 0. Install prerequisites (once per machine) + +Editable installs point wrappers at this source tree — code edits apply without +reinstalling. + +### Node machine — full install + +
+Linux / WSL ```bash -# Clone and enter repo -cd /run/media/popov/d/DEV/repos/d-popov.com/AI - -# Create the virtualenv if it does not exist yet +cd /path/to/neuron-tai python3 -m venv .venv source .venv/bin/activate -# Keep packaging tools current enough for editable installs .venv/bin/python -m pip install --upgrade pip setuptools wheel -# Install Python packages (editable — picks up code changes immediately) .venv/bin/python -m pip install -e packages/tracker -e packages/node -e packages/p2p -e packages/gateway -e packages/relay -# PyTorch: choose one -.venv/bin/python -m pip install torch -# .venv/bin/python -m pip install torch --index-url https://download.pytorch.org/whl/cpu -# .venv/bin/python -m pip install torch --index-url https://download.pytorch.org/whl/rocm6.2 -# HuggingFace model libraries .venv/bin/python -m pip install "transformers>=5.12" accelerate +.venv/bin/meshnet-node --help ``` -**Windows PowerShell** +
+ +
+Windows PowerShell (.venv) + +Requires Python 3.11+ and Git for Windows. ```powershell cd D:\DEV\workspace\REPOS\git.d-popov.com\neuron-tai @@ -40,321 +76,163 @@ python -m venv .venv .\.venv\Scripts\Activate.ps1 .\.venv\Scripts\python.exe -m pip install --upgrade pip setuptools wheel .\.venv\Scripts\python.exe -m pip install -e .\packages\tracker -e .\packages\node -e .\packages\p2p -e .\packages\gateway -e .\packages\relay -# PyTorch: choose one -.\.venv\Scripts\python.exe -m pip install torch -# .\.venv\Scripts\python.exe -m pip install torch --index-url https://download.pytorch.org/whl/cpu -# .\.venv\Scripts\python.exe -m pip install torch --index-url https://download.pytorch.org/whl/rocm6.2 .\.venv\Scripts\python.exe -m pip install "transformers>=5.12" accelerate +.\.venv\Scripts\meshnet-node.exe --help ``` -> Torch choices: default index for NVIDIA/CUDA, CPU index for CPU-only, ROCm index for AMD GPU. +
-### Version and library notes for Qwen3.5/3.6-MoE models - -- **transformers ≥ 5.12 is required** for Qwen3.5/3.6-MoE (e.g. `Qwen3.6-35B-A3B`). - Older versions fail at load time with - `'Qwen3_5MoeConfig' object has no attribute 'vocab_size'`. Check with - `python -c "import transformers; print(transformers.__version__)"` and upgrade - with `pip install -U transformers` in the environment that runs `meshnet-node` - (conda/miniforge users: upgrade inside that env, not a layered `.venv`). -- **Linear-attention fast path (GPU only).** Qwen3.5/3.6 use hybrid linear-attention - layers; without optional CUDA kernels, Transformers falls back to slower pure-PyTorch - code and prints `The fast path is not available…` at startup. That warning is - harmless — inference still works. On native Windows, install `triton-windows` in - the same env as `meshnet-node`; otherwise `flash-linear-attention` can fail during - import with `Could not import module 'Qwen3_5MoeForCausalLM'`. Install the - acceleration packages into the same env as `meshnet-node` for GPU speed; skip on - CPU-only nodes: - - ```bash - # Native Windows - pip install triton-windows - - # NVIDIA (CUDA) - pip install flash-linear-attention[cuda] causal-conv1d - - # AMD (ROCm) — match your torch index, then: - pip install flash-linear-attention[rocm] causal-conv1d - ``` - - Restart the node after install; the warning should disappear. Expect the largest - gain on GPU nodes serving linear-attention layers (roughly three quarters of - Qwen3.6 layers); end-to-end chat speed still depends on the slowest hop in a - split route. -- `pip install nvidia-ml-py` silences the pynvml deprecation warning on NVIDIA hosts. - -## Bootstrap a tracker on a new machine - -Use this when provisioning a fresh LAN/public tracker host. The tracker itself is -lightweight; install the relay too if nodes will connect from NAT, WSL2, mobile, -or other networks where inbound node ports are not reachable. - -```bash -# 1. Get the repo onto the tracker host -git clone https://git.d-popov.com/popov/neuron-tai.git AI -cd AI - -# 2. Create an isolated Python environment -python3 -m venv .venv -.venv/bin/python -m pip install --upgrade pip setuptools wheel - -# 3. Install only the services needed by the tracker host -.venv/bin/pip install -e packages/tracker -e packages/relay -e packages/gateway -``` - -For a private LAN tracker, start only the tracker and open the selected TCP port -on the host firewall if other machines will join: - -```bash -.venv/bin/meshnet-tracker start --host 0.0.0.0 --port 8080 -# --starting-credit 1 --devnet-topup 10 -``` - -Verify from the tracker host: - -```bash -curl -s http://localhost:8080/v1/network/map | python3 -m json.tool -``` - -Verify from another LAN machine, replacing the IP with the tracker host's LAN IP: - -```bash -curl -s http://192.168.0.179:8080/v1/network/map | python3 -m json.tool -``` - -For a public tracker with relay support, run both services. The relay listens on -`8765`; the tracker below listens on `8081` and advertises the public WebSocket -URL that nodes should use for outbound relay connections: - -```bash -# Terminal 1 — relay -.venv/bin/meshnet-relay --host 0.0.0.0 --port 8765 - -# Terminal 2 — tracker -.venv/bin/meshnet-tracker start --host 0.0.0.0 --port 8081 --relay-url wss://ai.neuron.d-popov.com/ws -``` - -If this host sits behind Nginx Proxy Manager, point `/` and `/v1/*` at tracker -port `8081`, and point `/ws` plus `/rpc` at relay port `8765` as shown in the -public tracker section below. After the proxy is configured, verify the public -bootstrap endpoint: - -```bash -curl -s https://ai.neuron.d-popov.com/v1/network/map | python3 -m json.tool -``` - -Nodes can then join with either the LAN tracker URL or the public URL: - -```bash -.venv/bin/meshnet-node start --tracker http://192.168.0.179:8080 --model Qwen/Qwen2.5-0.5B-Instruct -.venv/bin/meshnet-node start --tracker https://ai.neuron.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct -.venv/bin/meshnet-node start --tracker https://ai.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct -``` - -### Windows / WSL2 - -Run the Linux commands from WSL, not Git Bash. From the repo opened in Git Bash: - -```bash -wsl -cd /mnt/d/DEV/workspace/REPOS/git.d-popov.com/neuron-tai -python3 -m venv .venv -.venv/bin/python -m pip install --upgrade pip setuptools wheel -.venv/bin/pip install -e packages/tracker -e packages/node -e packages/p2p -e packages/gateway -e packages/relay -.venv/bin/pip install torch --index-url https://download.pytorch.org/whl/cpu -.venv/bin/pip install "transformers>=5.12" accelerate -.venv/bin/meshnet-node --help -``` - -If `.venv/bin/meshnet-node` is missing, the editable install step did not finish -successfully. Re-run the `.venv/bin/pip install -e ...` command above inside WSL. - -WSL2 sits behind Windows NAT and is **not directly reachable** from other LAN machines. -Direct cross-host hops time out. The relay path (see below) solves this: the WSL2 node -opens an outbound WebSocket to the relay server and all traffic flows through that tunnel. -No firewall rules, no `--advertise-host` needed — just point at the public tracker URL. - -### Native Windows PowerShell node (not WSL) - -Use this when the tracker is on another machine and you want Windows to host a -reachable node on the LAN. - -#### Option A — existing conda/miniforge environment with CUDA torch (recommended if you already have it) - -First, make sure the conda base environment is active so that `python` and `pip` both -resolve to the same miniforge installation: +
+Windows — conda/miniforge with CUDA (skip if using .venv above) ```powershell conda activate base -deactivate # drop any .venv that may be layered on top; safe no-op if none active -``` - -Install project packages into the active conda/miniforge env: - -```powershell +deactivate # drop any layered .venv; safe no-op if none active cd D:\DEV\workspace\REPOS\git.d-popov.com\neuron-tai - pip install -e packages\tracker -e packages\node -e packages\p2p -e packages\gateway -e packages\relay -pip install "transformers>=5.12" accelerate safetensors # torch is already present -``` - -> Conda/miniforge envs often carry an older `transformers` pinned by other tools -> (aider, etc.). Qwen3.5/3.6-MoE models need **transformers ≥ 5.12** — verify with -> `python -c "import transformers; print(transformers.__version__)"`. The pip -> resolver may print dependency-conflict warnings for those other tools; they don't -> affect `meshnet-node`. - -Verify torch is importable and CUDA is live **before** starting the node: - -```powershell +pip install "transformers>=5.12" accelerate safetensors python -c "import torch; print(torch.__version__, torch.cuda.is_available())" # Expected: 2.x.x+cuXXX True ``` -If you get `ModuleNotFoundError: No module named 'torch'` even though `pip install torch` -says "already satisfied", the `torch/` package directory is missing while the metadata -stub remains (can happen after a conda-managed install). Force-reinstall all three -PyTorch packages together so their versions stay in sync: +If `torch` import fails despite pip saying "already satisfied", force-reinstall all +three together (never upgrade `torch` alone — breaks `torchvision`): ```powershell pip install --force-reinstall torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118 ``` -> **Important:** always reinstall `torch`, `torchvision`, and `torchaudio` as a group. -> Upgrading only `torch` leaves `torchvision` on an incompatible version, which causes -> `RuntimeError: operator torchvision::nms does not exist` and makes transformers fail -> to import any model class (the error surfaces as `Could not import module 'Qwen2ForCausalLM'`). +If `.venv\Scripts\meshnet-node.exe` shadows the conda binary, use the full path: +`C:\Users\\miniforge3\Scripts\meshnet-node.exe`. -Then re-run the verify step above. +
-If that prints `True` but `meshnet-node` still can't find torch, the venv entry point -is shadowing the conda one. Check which binary wins: +> Run Linux/WSL commands from **WSL**, not Git Bash. From Git Bash: `wsl`, then `cd` +> to the repo under `/mnt/d/...`. -```powershell -(Get-Command meshnet-node).Source -# Should show: C:\Users\\miniforge3\Scripts\meshnet-node.exe -# If it shows .venv\Scripts\meshnet-node.exe, use the full path below instead -``` +### Tracker host — lightweight install -To start a node: - -```powershell -$env:HF_HOME = "D:\DEV\models" -meshnet-node start --tracker https://ai.neuron.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct -``` - -If the wrong entry point is shadowing, invoke via the full conda path: - -```powershell -C:\Users\popov\miniforge3\Scripts\meshnet-node.exe start --tracker https://ai.neuron.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct -``` - -#### Option B — isolated virtualenv (fresh machine, no existing torch) - -1. Install prerequisites on Windows: - - Python 3.11 or 3.12 from - - Git for Windows from - -2. Open **PowerShell** in the cloned repo and install the node packages: - -```powershell -# Example repo path; adjust to wherever you cloned it -cd D:\DEV\workspace\REPOS\git.d-popov.com\neuron-tai - -python -m venv .venv -.\.venv\Scripts\python.exe -m pip install --upgrade pip setuptools wheel -.\.venv\Scripts\pip.exe install -e packages\tracker -e packages\node -e packages\p2p -e packages\gateway -e packages\relay - -# CPU-only PyTorch. For NVIDIA CUDA, use `pip install torch` instead. -.\.venv\Scripts\pip.exe install torch --index-url https://download.pytorch.org/whl/cpu -.\.venv\Scripts\pip.exe install "transformers>=5.12" accelerate - -.\.venv\Scripts\meshnet-node.exe --help -``` - -For `start`-specific flags, run: - -```powershell -.\.venv\Scripts\meshnet-node.exe start --help -``` - -3. Find the Windows LAN IP address: - -```powershell -ipconfig -``` - -Use the IPv4 address on the active Ethernet/Wi-Fi adapter, for example -`192.168.0.42`. Avoid WSL/Docker/Hyper-V adapter addresses like `172.16.x.x`, -`172.17.x.x`, or other virtual adapter IPs. - -4. Allow inbound traffic for the node port in Windows Firewall. Run PowerShell as -Administrator once: - -```powershell -New-NetFirewallRule -DisplayName "Meshnet node 8005" -Direction Inbound -Action Allow -Protocol TCP -LocalPort 8005 -``` - -5. Start the Windows node from normal PowerShell. Replace the tracker and -advertised host values with your actual LAN addresses: - -```powershell -$env:HF_HOME = "D:\DEV\models" -.\.venv\Scripts\meshnet-node.exe start --tracker http://192.168.0.179:8081 --model Qwen/Qwen2.5-0.5B-Instruct --shard-start 12 --shard-end 23 --quantization bfloat16 --host 0.0.0.0 --advertise-host 192.168.0.42 --port 8005 -``` - -Other start examples (direct LAN — node must be reachable by IP from other machines): - -```powershell -.\.venv\Scripts\meshnet-node.exe start --tracker http://192.168.0.179:8081 --model Qwen/Qwen2.5-0.5B-Instruct --advertise-host 192.168.0.20 -``` - -Via public hostname with relay (works from behind NAT, WSL2, 5G — no `--advertise-host` needed): - -```powershell -.\.venv\Scripts\meshnet-node.exe start --tracker https://ai.neuron.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct -``` - -WSL (same relay path — no `--advertise-host`): +Tracker + relay only; skip node packages unless this machine also runs nodes. ```bash -.venv/bin/meshnet-node start --tracker https://ai.neuron.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct -.venv/bin/meshnet-node start --tracker http://192.168.0.179:8081 --model Qwen/Qwen2.5-0.5B-Instruct +git clone https://git.d-popov.com/popov/neuron-tai.git AI && cd AI +python3 -m venv .venv +.venv/bin/python -m pip install --upgrade pip setuptools wheel +.venv/bin/pip install -e packages/tracker -e packages/relay -e packages/gateway ``` -`--host 0.0.0.0` binds the node to all Windows interfaces. `--advertise-host` -is what the tracker gives to other nodes for direct connections; omit it when using -the relay path since all traffic flows through the relay tunnel instead. +### PyTorch variant -If you want verbose per-hop pipeline logs while debugging a split model, add -`--debug`. Leave it off for normal runs; otherwise every generated token logs -lines like: +Install **one** torch line into the same env as `meshnet-node`: -```text - [node] pipeline hop 0: http://127.0.0.1:8005 start_layer=22 - [node] pipeline hop 0 returned text=' token' - [node] pipeline hop 1: wss://ai.neuron.d-popov.com/rpc/abc123 relay start_layer=12 -``` +| Hardware | Install | +|----------|---------| +| NVIDIA CUDA | `pip install torch` (default index) | +| CPU only | `pip install torch --index-url https://download.pytorch.org/whl/cpu` | +| AMD ROCm | `pip install torch --index-url https://download.pytorch.org/whl/rocm6.2` | -6. From the tracker machine, verify Windows is reachable: +On Windows `.venv`, prefix with `.\.venv\Scripts\pip.exe`. Conda users with CUDA +torch already installed can skip this step. -```bash -curl http://192.168.0.42:8005/v1/health -``` +### Qwen3.5/3.6-MoE notes -If that endpoint returns 404 or 501, that is okay: it still proves the TCP -connection reached the node process. If it times out or connection-refuses, check -the Windows Firewall rule, `--host 0.0.0.0`, the selected LAN IP, and that the -node is still running. +Applies to **`qwen3.6-35b-a3b`** and other hybrid linear-attention models. **`Qwen2.5-0.5B`** +does not need any of this — it is a standard transformer with no FLA fast path. + +- **transformers ≥ 5.12 required** — older versions fail with + `'Qwen3_5MoeConfig' object has no attribute 'vocab_size'`. Check: + `python -c "import transformers; print(transformers.__version__)"`. +- **Optional GPU kernels** — without them inference still works; startup may print + `The fast path is not available…` (harmless on Windows; slower on linear-attention + layers). Install **only for your platform**: + + | Platform | Install | Fast path? | + |----------|---------|------------| + | **Native Windows** | `pip install triton-windows` (also pulled by `meshnet-node` on Windows) | **Load only** — model imports and runs on PyTorch fallback. Do **not** `pip install flash-linear-attention[cuda] causal-conv1d` (see below). | + | **Linux + NVIDIA CUDA** | `pip install flash-linear-attention[cuda]` | **Yes** — `causal-conv1d` optional since FLA ≥0.3.2 ships Triton conv1d; add it only if transformers still asks for it. Needs CUDA toolkit (`nvcc`) or a matching prebuilt wheel. | + | **Linux + AMD ROCm** | `pip install flash-linear-attention[rocm]` | **Yes** — same optional `causal-conv1d` note. | + + On native Windows, `triton-windows` is enough for Qwen3.6-MoE to **load**. Without it, + startup fails with misleading `Could not import module 'Qwen3_5MoeForCausalLM'`. + +
+Why there is no supported Windows fast path (research notes) + +We checked upstream docs and real install attempts (2026-07): + +1. **`causal-conv1d`** — no official Windows wheels on PyPI. Source builds need `nvcc` + plus a torch/CUDA stack that matches; without `nvcc` pip fails with + `bare_metal_version is not defined`. Open upstream issues confirm Windows is + unsupported / broken for most setups. +2. **`flash-linear-attention[cuda]`** — FLA's install guide targets Linux (CUDA / ROCm / + XPU / NPU). The `[cuda]` extra pulls PyPI `triton`, which conflicts with + `triton-windows` on native Windows. FLA does not document or CI-test Windows. +3. **What works today on native Windows GPU** — CUDA torch + `triton-windows` → Qwen3.6 + loads and infers via Transformers' pure-PyTorch fallback (~¾ of layers are + linear-attention; those hops are slower). This is what we use in alpha. +4. **If you need the fast path on a Windows PC** — run the GPU node in **WSL2 with Linux + CUDA** (or a Linux box): `pip install flash-linear-attention[cuda]` there. Keep + Windows native for reachability / relay if needed. +5. **Experimental / unsupported** — community `causal-conv1d` wheels and + `triton-windows` + pinned `flash-linear-attention --no-deps` hacks exist for narrow + Python/torch/CUDA combos; we do not support or test these for meshnet-node. + +
+ +- `pip install nvidia-ml-py` silences the pynvml deprecation warning on NVIDIA hosts. --- -## Public tracker + relay (internet / NAT nodes) +## 1. Start the tracker -This setup lets nodes connect from anywhere — behind home NAT, 5G, WSL2, or -on a different continent — without opening firewall ports. +### LAN tracker (private network, direct node reachability) -### Architecture +**Terminal 1:** + +```bash +.venv/bin/meshnet-tracker start --host 0.0.0.0 --port 8080 +# Optional devnet billing: --starting-credit 1 --devnet-topup 10 +``` + +Expected: `Tracker listening on 0.0.0.0:8080`. Open the port on the host firewall +if other machines will join. + +**Verify:** + +```bash +curl -s http://localhost:8080/v1/network/map | python3 -m json.tool +curl -s http://192.168.0.179:8080/v1/network/map | python3 -m json.tool # from another LAN machine +``` + +### Public tracker + relay (NAT / WSL2 / internet nodes) + +Nodes behind NAT cannot receive inbound connections. Run **both** services on the +tracker host; the tracker advertises the relay URL in `/v1/network/map`. + +**Terminal 1 — relay:** + +```bash +.venv/bin/meshnet-relay --host 0.0.0.0 --port 8765 +``` + +**Terminal 2 — tracker:** + +```bash +.venv/bin/meshnet-tracker start --host 0.0.0.0 --port 8081 --relay-url wss://ai.neuron.d-popov.com/ws +``` + +**Verify:** + +```bash +curl -s https://ai.neuron.d-popov.com/v1/network/map | python3 -m json.tool +``` + +Nodes should log `Relay connected — wss://…/rpc/` on startup. + +
+Nginx Proxy Manager setup (public hostname) + +Architecture: ``` Client → HTTPS → ai.neuron.d-popov.com (nginx) @@ -363,64 +241,27 @@ Client → HTTPS → ai.neuron.d-popov.com (nginx) └─ /rpc/* → meshnet-relay :8765 (caller opens WS per hop) ``` -### Nginx Proxy Manager (Docker) +Use **one** proxy host. Route sub-paths via **Custom locations** — do not create +a second host for the same domain. -Use **one** proxy host for the domain. Do not create a second host for the same -domain to reach another port — path routing is done with **Custom locations** on -that same host. - -**1. Details tab** (default `/` → tracker) +**Details tab** (default `/` → tracker): | Field | Value | |-------|--------| | Domain Names | `ai.neuron.d-popov.com` | | Scheme | `http` | -| Forward Hostname / IP | LAN IP of the tracker machine (e.g. `192.168.0.179`) | +| Forward Hostname / IP | LAN IP of tracker machine (e.g. `192.168.0.179`) | | Forward Port | `8081` | | Websockets Support | ON | -This serves `/v1/network/map`, `/v1/chat/completions`, and the rest of the tracker API. +**Custom locations** (both → relay port `8765`, sub-folder path empty): -**2. Custom locations tab** (sub-paths → relay) +| Location | Forward to | +|----------|--------------| +| `/ws` | `192.168.0.179:8765` | +| `/rpc` | `192.168.0.179:8765` | -The Custom locations form has **no separate Websockets toggle** — only location, -scheme, forward host, optional sub-folder path, and port. Add **two** locations -(both pointing at the relay process on port `8765`). Leave **“Add a path for -sub-folder forwarding”** empty so the full URI reaches the relay -(`/ws`, `/rpc/`). - -Location A — persistent node connections: - -| Field | Value | -|-------|--------| -| Define location | `/ws` | -| Scheme | `http` | -| Forward Hostname / IP | `192.168.0.179` | -| Forward Port | `8765` | -| Sub-folder path | *(leave empty)* | - -Location B — per-hop RPC: - -| Field | Value | -|-------|--------| -| Define location | `/rpc` | -| Scheme | `http` | -| Forward Hostname / IP | `192.168.0.179` | -| Forward Port | `8765` | -| Sub-folder path | *(leave empty)* | - -Nginx matches the longer prefixes first: `/ws` and `/rpc/…` go to relay; everything -else stays on `8081`. - -**3. SSL tab** - -Use your existing Let’s Encrypt certificate (unchanged). - -**4. Advanced tab** (only if WebSocket upgrade fails on `/ws` or `/rpc`) - -Custom locations do not expose a Websockets checkbox. If nodes show -`Relay configured but not connected yet` while `/v1/network/map` works, add this -snippet on the **proxy host** Advanced tab: +**Advanced tab** (only if WebSocket upgrade fails): ```nginx proxy_http_version 1.1; @@ -430,317 +271,266 @@ proxy_read_timeout 3600s; proxy_send_timeout 3600s; ``` -**5. Verify routing** +**Verify routing:** ```bash -# Tracker (8081 via default location) curl -s https://ai.neuron.d-popov.com/v1/network/map | python3 -m json.tool - -# Relay paths should not 502/404 from the tracker — check response headers/status curl -sI https://ai.neuron.d-popov.com/ws curl -sI https://ai.neuron.d-popov.com/rpc/test-peer ``` -After NPM is correct, start relay and tracker on the LAN machine: +
+ +--- + +## 2. Start a node + +**Starter model:** `Qwen/Qwen2.5-0.5B-Instruct` — 0.5B params, ~1 GB BF16, 24 layers, +no HuggingFace gating. Best for first-time setup. + +**Alpha model:** `qwen3.6-35b-a3b` — 40 layers, ~72 GB BF16 download, MoE with hybrid +linear attention. Use on machines with enough RAM/VRAM; see Qwen3.5/3.6 notes above for +`triton-windows` (Windows) or `flash-linear-attention` (Linux GPU). Tracker accepts the +alias or full repo id (`unsloth/Qwen3.6-35B-A3B`). + +Downloads cache under `~/.meshnet/models/` (or `$HF_HOME` / `$env:HF_HOME`). + +Shard range is auto-detected from the curated catalog. For unknown repos the node +fetches only `config.json`. Override with `--shard-start` / `--shard-end` for partial +shards or multi-node splits. + +### Core command + +Replace `` and adjust the prefix for your shell (see table above). + +**Linux / WSL:** ```bash -# Terminal 1 — relay -.venv/bin/meshnet-relay --host 0.0.0.0 --port 8765 - -# Terminal 2 — tracker (advertises relay to nodes) -.venv/bin/meshnet-tracker start --host 0.0.0.0 --port 8081 --relay-url wss://ai.neuron.d-popov.com/ws +HF_HOME=/path/to/models .venv/bin/meshnet-node start --tracker --model Qwen/Qwen2.5-0.5B-Instruct --quantization bfloat16 ``` -Nodes using `https://ai.neuron.d-popov.com` should then log: +**Windows PowerShell:** -```text -Relay advertised by tracker — using outbound tunnel wss://ai.neuron.d-popov.com/ws - Relay connected — wss://ai.neuron.d-popov.com/rpc/ +```powershell +$env:HF_HOME = "D:\DEV\models" +.\.venv\Scripts\meshnet-node.exe start --tracker --model Qwen/Qwen2.5-0.5B-Instruct --quantization bfloat16 ``` -The `--relay-url` flag embeds the relay address in `/v1/network/map`. Every node -queries that endpoint on startup and auto-connects if a relay URL is present. +### Ready-to-run examples -### Start a node (any machine, any network) +**Local dev (same machine as tracker):** -No `--advertise-host`, firewall rule, port forwarding, relay URL, or peer URL is -needed on the node. The public tracker is the only bootstrap URL the user types. -The node queries the tracker for `/v1/network/map`, discovers the relay URL, and -opens a persistent outbound WebSocket. If the relay connection drops, the node -keeps retrying it. +```bash +HF_HOME=/path/to/models .venv/bin/meshnet-node start --tracker http://localhost:8080 --model Qwen/Qwen2.5-0.5B-Instruct --quantization bfloat16 --port 8001 +``` + +**Public / relay (works from WSL2, NAT, 5G — no extra flags):** ```bash .venv/bin/meshnet-node start --tracker https://ai.neuron.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct ``` -No authentication is required in the prototype. The first public node for a model -must still choose that model with `--model` or a saved wizard config. After at -least one HF model node is registered, later nodes can join the public network -with only the tracker URL; the tracker assigns an uncovered shard if one exists: +**Alpha model (Qwen3.6, Windows GPU — PyTorch fallback, no FLA fast path):** + +```powershell +$env:HF_HOME = "D:\DEV\models" +meshnet-node start --tracker http://192.168.0.179:8080 --model qwen3.6-35b-a3b --quantization bfloat16 +``` + +**Alpha model (Qwen3.6, Linux GPU — with fast path):** + +```bash +HF_HOME=/path/to/models .venv/bin/meshnet-node start --tracker --model qwen3.6-35b-a3b --quantization bfloat16 +# Install once on that machine: pip install flash-linear-attention[cuda] +``` + +After the first node registers a model, later nodes can join with only the tracker +URL (shard auto-assigned): ```bash .venv/bin/meshnet-node start --tracker https://ai.neuron.d-popov.com ``` -Use the public tracker to verify registration and routing: +**Direct LAN (Windows node reachable by IP):** -```bash -curl -s "https://ai.neuron.d-popov.com/v1/network/map" | python3 -m json.tool -curl -s "https://ai.neuron.d-popov.com/v1/route?model=qwen2.5-0.5b" | python3 -m json.tool +```powershell +$env:HF_HOME = "D:\DEV\models" +.\.venv\Scripts\meshnet-node.exe start --tracker http://192.168.0.179:8081 --model Qwen/Qwen2.5-0.5B-Instruct --shard-start 12 --shard-end 23 --quantization bfloat16 --host 0.0.0.0 --advertise-host 192.168.0.42 --port 8005 ``` -Expected startup output (relay path): +
+Windows direct LAN — firewall and IP checklist + +1. Find LAN IP: `ipconfig` — use active Ethernet/Wi-Fi IPv4 (e.g. `192.168.0.42`). + Avoid WSL/Docker/Hyper-V addresses (`172.x.x.x`). +2. Allow inbound port (Administrator PowerShell, once): + +```powershell +New-NetFirewallRule -DisplayName "Meshnet node 8005" -Direction Inbound -Action Allow -Protocol TCP -LocalPort 8005 +``` + +3. Verify from tracker machine: + +```bash +curl http://192.168.0.42:8005/v1/health +``` + +404/501 is fine — it proves TCP reached the node. Timeout = check firewall, +`--host 0.0.0.0`, and `--advertise-host`. + +
+ +
+Two-node split (same or different machines) + +**Node A — layers 0–11 (head, serves chat):** + +```bash +HF_HOME=/path/to/models .venv/bin/meshnet-node start --tracker --model Qwen/Qwen2.5-0.5B-Instruct --shard-start 0 --shard-end 11 --quantization bfloat16 --port 8001 +``` + +**Node B — layers 12–23:** + +```bash +HF_HOME=/path/to/models .venv/bin/meshnet-node start --tracker --model Qwen/Qwen2.5-0.5B-Instruct --shard-start 12 --shard-end 23 --quantization bfloat16 --port 8002 +``` + +Send inference to Node A. For cross-machine LAN tests see `docs/TWO_MACHINE_TEST.md`. + +
+ +### Useful flags + +| Flag | Purpose | +|------|---------| +| `--port 8001` | Fixed listen port (default: first free ≥ 7000) | +| `--host 0.0.0.0` | Bind all interfaces (needed for direct LAN) | +| `--advertise-host ` | LAN IP the tracker tells other nodes (direct LAN only) | +| `--shard-start N --shard-end M` | Partial layer range | +| `--debug` | Verbose per-hop pipeline logs (noisy; off by default) | + +`--host 0.0.0.0` binds locally; `--advertise-host` is what peers use for direct +hops. Omit both when using the relay path. + +### Expected output ``` Auto-detected 24 layers → shard 0–23 - Relay connected — wss://ai.neuron.d-popov.com/rpc/abc1def2ef3f4567 + Relay connected — wss://ai.neuron.d-popov.com/rpc/abc1def2ef3f4567 # relay mode only ================================ meshnet-node ready Wallet:
Model ID: Qwen/Qwen2.5-0.5B-Instruct Shard: layers 0–23; 24 of 24 Quantization: bfloat16 - Endpoint: http://172.29.104.23:7001 + Endpoint: http://:8001 Node ID: Hardware: CPU ================================ ``` -The `Endpoint` shown is the local IP (unreachable from outside). Other nodes reach -this one via `wss://ai.neuron.d-popov.com/rpc/` instead. +The `Endpoint` is the local address. In relay mode, peers reach this node via +`wss:///rpc/` instead. -### How relay hops work +### Other CPU-friendly models -When node A needs to forward activations to node B (behind NAT): - -1. Tracker injects `X-Meshnet-Route` with `relay_addr` for each behind-NAT hop. -2. Node A opens a WebSocket to `wss://relay/rpc/{peer_id_B}`. -3. Relay forwards the `relay-http-request` envelope to Node B's persistent connection. -4. Node B processes `/forward` locally, returns `relay-http-response`. -5. Relay sends the response back to Node A over the same WebSocket. -6. Node A closes the WebSocket and continues the pipeline. - -Binary activation tensors (bfloat16) are Base64-encoded through the relay JSON -protocol and decoded on both sides — no precision loss. - -If the relay hop fails (relay down, peer disconnected), the node logs a warning and -falls back to a direct HTTP attempt before returning an error. - -### Test from WSL2 using the public tracker - -In WSL2 (which gets a `172.x.x.x` virtual IP — unreachable from other machines): - -```bash -# WSL2 Terminal 1 — head node (layers 0–11, handles chat requests) -.venv/bin/meshnet-node start --tracker https://ai.neuron.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct --shard-start 0 --shard-end 11 - -# WSL2 Terminal 2 — tail node (layers 12–23) -.venv/bin/meshnet-node start --tracker https://ai.neuron.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct --shard-start 12 --shard-end 23 -``` - -Both nodes connect to the relay automatically. When a chat request arrives at Node A, -it forwards activations to Node B via `wss://ai.neuron.d-popov.com/rpc/{peer_id_B}`. - -Send inference through the tracker (which picks the head node and injects the route): - -```bash -curl -s https://ai.neuron.d-popov.com/v1/chat/completions -H "Content-Type: application/json" -H "Authorization: Bearer sk-mesh-" -d '{"model": "Qwen/Qwen2.5-0.5B-Instruct", "messages": [{"role": "user", "content": "What is 7 times 8?"}], "stream": false}' | python3 -m json.tool -``` - -Or send directly to Node A's local port (within WSL): - -```bash -curl -s http://localhost:7001/v1/chat/completions \ - -H "Content-Type: application/json" \ - -d '{"model": "Qwen/Qwen2.5-0.5B-Instruct", "messages": [{"role": "user", "content": "Hi"}]}' -``` - -## Accounts, API keys, and credit (billing-enabled trackers) - -Public trackers run with billing on: `/v1/chat/completions` requires a real -API key from a registered account. Unknown bearer strings get `401`; a key -with no balance gets `402 insufficient balance`. - -**Dashboard flow (easiest):** open `https:///dashboard`, register with -an email + password, then click **+ new key**. The key (`sk-mesh-…`) shows its -balance next to it. If the tracker was started with `--starting-credit`, your -first key arrives pre-funded (Caller Credit, once per account). If it was -started with `--devnet-topup`, every key row has a **+$N (devnet)** button to -refill during testing. - -**Curl flow:** - -```bash -# 1. Register (once) -curl -s https:///v1/auth/register -H "Content-Type: application/json" -d '{"email": "you@example.com", "password": "hunter22-or-better"}' -# → {"session_token": "...", ...} - -# 2. Create an API key (session token from step 1) -curl -s https:///v1/account/keys -X POST -H "Authorization: Bearer " -# → {"api_key": "sk-mesh-...", "caller_credit_granted": true} - -# 3. Check balance / usage -curl -s https:///v1/account -H "Authorization: Bearer " - -# 4. (devnet trackers only) top up a key -curl -s https:///v1/account/topup -X POST -H "Authorization: Bearer " -H "Content-Type: application/json" -d '{"api_key": "sk-mesh-..."}' -``` - -Operator side: both features default to 1 USDT (`--starting-credit` / -`--devnet-topup`). Set both to 0 on mainnet deployments — real deposits flow -through the on-chain USDT treasury watcher instead. - ---- - -## Step 1 — Start the tracker (Terminal 1) - -```bash -cd /run/media/popov/d/DEV/repos/d-popov.com/AI -.venv/bin/meshnet-tracker start --port 8080 -``` - -Expected output: -``` -Tracker listening on 0.0.0.0:8080 -``` - -Keep this terminal open. - ---- - -## Step 2 — Start a node (Terminal 2) - -### Recommended model: Qwen2.5-0.5B-Instruct - -- 0.5B parameters, ~1 GB in BF16 -- No HuggingFace account or license required -- Downloads once to `~/.meshnet/models/`, cached for future runs -- 24 transformer layers (auto-detected — no need to specify) - -```bash -cd /run/media/popov/d/DEV/repos/d-popov.com/AI -HF_HOME=/run/media/popov/d/DEV/models .venv/bin/meshnet-node start --model Qwen/Qwen2.5-0.5B-Instruct --quantization bfloat16 --tracker http://localhost:8080 --port 8001 -``` - -Shard range is **auto-detected** from the curated catalog (no network call for known -models). For unknown repos, the node fetches only `config.json` (~1 KB) to read -`num_hidden_layers`. You can still pass `--shard-start` / `--shard-end` explicitly -to run a partial shard on one machine. - -Expected output (after model loads): -``` - Auto-detected 24 layers → shard 0–23 -================================ -meshnet-node ready - Wallet:
- Model ID: Qwen/Qwen2.5-0.5B-Instruct - Shard: layers 0–23 - Quantization: bfloat16 - Endpoint: http://:8001 - Hardware: CPU -================================ -``` - -### Other model options (all CPU-friendly) - -| Model | HF repo | Layers | BF16 size | Notes | -|-------|---------|--------|-----------|-------| -| Qwen2.5-0.5B | `Qwen/Qwen2.5-0.5B-Instruct` | 24 | ~1 GB | Fastest, no gating | +| Model | `--model` / alias | Layers | BF16 size | Notes | +|-------|-------------------|--------|-----------|-------| +| **Qwen2.5-0.5B** (dev default) | `Qwen/Qwen2.5-0.5B-Instruct` | 24 | ~1 GB | Fastest, no gating | +| **Qwen3.6-35B-A3B** (alpha) | `qwen3.6-35b-a3b` | 40 | ~72 GB | MoE; needs transformers ≥5.12; see Qwen3.5/3.6 notes | | Qwen2.5-1.5B | `Qwen/Qwen2.5-1.5B-Instruct` | 28 | ~3 GB | Better quality | | Phi-3-mini | `microsoft/Phi-3-mini-4k-instruct` | 32 | ~7.5 GB | Best CPU quality | | Llama-3.2-1B | `meta-llama/Llama-3.2-1B-Instruct` | 16 | ~2 GB | Requires HF login | | Llama-3.2-3B | `meta-llama/Llama-3.2-3B-Instruct` | 28 | ~6 GB | Requires HF login | -For gated models (Llama), run `huggingface-cli login` first. +For gated models (Llama): `huggingface-cli login` first. + +Browse more: `.venv/bin/meshnet-node models` or `.venv/bin/meshnet-node models --browse`. --- -## Step 3 — Send an inference request (Terminal 3) +## 3. Send an inference request -If you started the node with `--port 8001`, send the request directly to that -head node: +**Terminal 3** — direct to the head node (replace port if you omitted `--port`): -```bash Qwen2.5-0.5B-Instruct +```bash curl -s http://localhost:8001/v1/chat/completions -H "Content-Type: application/json" -d '{"model": "qwen2.5-0.5b", "messages": [{"role": "user", "content": "What is 7 times 8? Answer in one word."}], "stream": false}' | python3 -m json.tool ``` -If you did not pass `--port`, `meshnet-node start` uses the first free port at -or above `7000`. Use the `Endpoint:` printed by the node instead of `8001`. - -To test tracker routing/proxying, send the same OpenAI-compatible request to the -tracker, using either the full HuggingFace repo or the quick alias: +**Via tracker** (tests routing / proxying): ```bash curl -s http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{"model": "qwen2.5-0.5b", "messages": [{"role": "user", "content": "What is 7 times 8? Answer in one word."}], "stream": false}' | python3 -m json.tool ``` -Or use the test script: +**Public tracker:** + +```bash +curl -s https://ai.neuron.d-popov.com/v1/chat/completions -H "Content-Type: application/json" -H "Authorization: Bearer sk-mesh-" -d '{"model": "Qwen/Qwen2.5-0.5B-Instruct", "messages": [{"role": "user", "content": "What is 7 times 8?"}], "stream": false}' | python3 -m json.tool +``` + +**Test script:** ```bash .venv/bin/python scripts/test_lan_inference.py --tracker http://localhost:8080 --gateway http://localhost:8001 ``` ---- +**Verify registration on public tracker:** -## Two-node split (same machine, two terminals) - -Split Qwen2.5-0.5B's 24 layers across two node processes to test the sharded pipeline: - -**Node A — layers 0–11 (tracker mode, serves chat completions):** ```bash -HF_HOME=/run/media/popov/d/DEV/models .venv/bin/meshnet-node start --model Qwen/Qwen2.5-0.5B-Instruct --shard-start 0 --shard-end 11 --quantization bfloat16 --tracker http://localhost:8080 --port 8001 +curl -s "https://ai.neuron.d-popov.com/v1/network/map" | python3 -m json.tool +curl -s "https://ai.neuron.d-popov.com/v1/route?model=qwen2.5-0.5b" | python3 -m json.tool ``` -**Node B — layers 12–23:** +
+Accounts, API keys, and credit (billing-enabled trackers) + +Public trackers require a real API key for `/v1/chat/completions`. Unknown bearer → +`401`; zero balance → `402 insufficient balance`. + +**Dashboard:** open `https:///dashboard`, register, click **+ new key**. +With `--starting-credit`, the first key is pre-funded. With `--devnet-topup`, use +**+$N (devnet)** to refill during testing. + +**Curl flow:** + ```bash -HF_HOME=/run/media/popov/d/DEV/models .venv/bin/meshnet-node start --model Qwen/Qwen2.5-0.5B-Instruct --shard-start 12 --shard-end 23 --quantization bfloat16 --tracker http://localhost:8080 --port 8002 +curl -s https:///v1/auth/register -H "Content-Type: application/json" -d '{"email": "you@example.com", "password": "hunter22-or-better"}' +curl -s https:///v1/account/keys -X POST -H "Authorization: Bearer " +curl -s https:///v1/account -H "Authorization: Bearer " +curl -s https:///v1/account/topup -X POST -H "Authorization: Bearer " -H "Content-Type: application/json" -d '{"api_key": "sk-mesh-..."}' ``` -Send the request to Node A — it tokenizes, runs layers 0–11, passes binary -activations to Node B, and streams the final response back. +Operator defaults: `--starting-credit` and `--devnet-topup` both default to 1 USDT. +Set both to 0 on mainnet. + +
--- -## Two-machine LAN test (Linux + Windows/WSL2) +## Reference -See `docs/TWO_MACHINE_TEST.md` (created by US-018). +### How relay hops work -For WSL2 nodes, registration only proves the node can reach the tracker -outbound. Tracker-routed inference also requires the tracker to reach the node's -advertised endpoint inbound. Either run the node in native Windows PowerShell, -configure Windows port forwarding into WSL for the node port, or start the -tracker with a relay URL so the node registers a `relay_addr`. +When node A forwards activations to node B (behind NAT): ---- +1. Tracker injects `X-Meshnet-Route` with `relay_addr` for behind-NAT hops. +2. Node A opens WebSocket to `wss://relay/rpc/{peer_id_B}`. +3. Relay forwards `relay-http-request` to Node B's persistent connection. +4. Node B processes `/forward`, returns `relay-http-response`. +5. Relay sends response back to Node A; Node A continues the pipeline. -## Browse available models +Activations (bfloat16) are Base64-encoded in JSON — no precision loss. On relay +failure the node logs a warning and falls back to direct HTTP before erroring. + +### Interactive wizard ```bash -# Show curated list with VRAM requirements -.venv/bin/meshnet-node models - -# Browse HuggingFace Hub top-20 text-generation models -.venv/bin/meshnet-node models --browse -``` - ---- - -## Start with the interactive wizard - -```bash -# First run: wizard detects GPU, shows model list, saves config -.venv/bin/meshnet-node - -# Subsequent runs: starts directly from saved config -.venv/bin/meshnet-node - -# Re-run wizard even with saved config +.venv/bin/meshnet-node # first run: wizard; later: saved config .venv/bin/meshnet-node --reset-config ``` ---- - -## Run all tests +### Run all tests ```bash .venv/bin/python -m pytest -q diff --git a/packages/node/meshnet_node/startup.py b/packages/node/meshnet_node/startup.py index f3fd24d..14ebac6 100644 --- a/packages/node/meshnet_node/startup.py +++ b/packages/node/meshnet_node/startup.py @@ -197,12 +197,43 @@ def _max_assignable_layers( memory_mb: int, total_layers: int | None, bytes_per_layer: int | None = None, + *, + safety_fraction: float = 0.8, ) -> int: if total_layers is None or total_layers <= 0 or memory_mb <= 0: return 0 budget_bytes = memory_mb * 1024 * 1024 layer_bytes = bytes_per_layer or _DEFAULT_BYTES_PER_LAYER - return min(total_layers, int((budget_bytes * 0.8) // layer_bytes)) + return min(total_layers, int((budget_bytes * safety_fraction) // layer_bytes)) + + +def _runtime_shard_safety_fraction(device: str) -> float: + """CPU partial loads need room for model skeletons, tokenizer, and allocator peaks.""" + return 0.55 if device != "cuda" else 0.8 + + +def _cap_auto_assigned_shard( + shard_start: int, + shard_end: int, + total_layers: int | None, + memory_mb: int, + bytes_per_layer: int | None, + device: str, +) -> tuple[int, bool, int]: + if bytes_per_layer is None or total_layers is None: + return shard_end, False, 0 + max_layers = _max_assignable_layers( + memory_mb, + total_layers, + bytes_per_layer=bytes_per_layer, + safety_fraction=_runtime_shard_safety_fraction(device), + ) + if max_layers <= 0: + return shard_end, False, max_layers + assigned_layers = shard_end - shard_start + 1 + if assigned_layers <= max_layers: + return shard_end, False, max_layers + return min(total_layers - 1, shard_start + max_layers - 1), True, max_layers def _format_shard_label( @@ -226,13 +257,19 @@ def _shard_budget_line( total_layers: int | None, quantization: str, bytes_per_layer: int | None = None, + safety_fraction: float = 0.8, ) -> str: memory_gb = memory_mb / 1024 gb_str = f"{memory_gb:.1f} GB" budget_quantization = "bfloat16" if quantization == "auto" else quantization if total_layers is None or total_layers <= 0: return f"Memory budget: {gb_str} {memory_source}; shard budget: unknown model layer count" - max_layers = _max_assignable_layers(memory_mb, total_layers, bytes_per_layer=bytes_per_layer) + max_layers = _max_assignable_layers( + memory_mb, + total_layers, + bytes_per_layer=bytes_per_layer, + safety_fraction=safety_fraction, + ) # Remaining capacity after one full model load (rough estimate) shard_bytes = max_layers * (bytes_per_layer or _DEFAULT_BYTES_PER_LAYER) remaining_gb = (memory_mb * 1024 * 1024 - shard_bytes) / (1024 ** 3) @@ -729,6 +766,25 @@ def run_startup( if net_asgn.get("hf_repo") == model_id and net_asgn.get("gap_found"): shard_start = net_asgn["shard_start"] shard_end = net_asgn["shard_end"] + asgn_total_layers = int(net_asgn.get("num_layers") or detected) + asgn_bytes_per_layer = _assignment_bytes_per_layer(net_asgn, quantization) + capped_shard_end, was_capped, max_runtime_layers = _cap_auto_assigned_shard( + shard_start, + shard_end, + asgn_total_layers, + memory_budget_mb, + asgn_bytes_per_layer, + device, + ) + if was_capped: + original_end = shard_end + shard_end = capped_shard_end + print( + f" WARNING: tracker assigned layers {shard_start}-{original_end}, " + f"but CPU-safe runtime budget fits {max_runtime_layers}/{asgn_total_layers} layers; " + f"loading layers {shard_start}-{shard_end} instead.", + flush=True, + ) full_sources = ( [] if tracker_source_disabled else _full_model_sources(net_asgn.get("model_sources", [])) @@ -744,7 +800,7 @@ def run_startup( ) print( f" Tracker found uncovered shard: " - f"layers {shard_start}–{shard_end} (of {detected})", + f"layers {shard_start}-{shard_end} (of {detected})", flush=True, ) except Exception: @@ -859,18 +915,36 @@ def run_startup( assigned_shard_start: int = net_assignment["shard_start"] assigned_shard_end: int = net_assignment["shard_end"] assigned_num_layers: int = net_assignment["num_layers"] + assigned_bytes_per_layer = _assignment_bytes_per_layer(net_assignment, quantization) + capped_shard_end, was_capped, max_runtime_layers = _cap_auto_assigned_shard( + assigned_shard_start, + assigned_shard_end, + assigned_num_layers, + memory_budget_mb, + assigned_bytes_per_layer, + device, + ) + if was_capped: + original_end = assigned_shard_end + assigned_shard_end = capped_shard_end + print( + f" WARNING: tracker assigned layers {assigned_shard_start}-{original_end}, " + f"but CPU-safe runtime budget fits {max_runtime_layers}/{assigned_num_layers} layers; " + f"loading layers {assigned_shard_start}-{assigned_shard_end} instead.", + flush=True, + ) assigned_model_sources: list[dict] = net_assignment.get("model_sources", []) if _gap_found: print( f" Assigned gap: {assigned_hf_repo} " - f"layers {assigned_shard_start}–{assigned_shard_end} " + f"layers {assigned_shard_start}-{assigned_shard_end} " f"(of {assigned_num_layers})", flush=True, ) else: print( f" Assigned redundant copy: {assigned_hf_repo} " - f"layers {assigned_shard_start}–{assigned_shard_end} " + f"layers {assigned_shard_start}-{assigned_shard_end} " f"(of {assigned_num_layers})", flush=True, ) @@ -956,7 +1030,7 @@ def run_startup( f" Wallet: {address}\n" f" Model ID: {assigned_hf_repo}\n" f" Shard: {shard_label}\n" - f" {_shard_budget_line(memory_budget_mb, memory_budget_source, assigned_num_layers, quantization)}\n" + f" {_shard_budget_line(memory_budget_mb, memory_budget_source, assigned_num_layers, quantization, bytes_per_layer=assigned_bytes_per_layer, safety_fraction=_runtime_shard_safety_fraction(device))}\n" f" Quantization: {quantization}\n" f" Endpoint: {endpoint}\n" f" Node ID: {tracker_node_id or 'unregistered'}\n" @@ -1022,6 +1096,7 @@ def run_startup( memory_budget_mb, assigned_total_layers, assignment_bytes_per_layer, + safety_fraction=_runtime_shard_safety_fraction(device), ) if pinned_layers > max_layers: raise ValueError( @@ -1115,7 +1190,7 @@ def run_startup( f" Wallet: {address}\n" f" Model ID: {hf_repo}\n" f" Shard: {shard_label}\n" - f" {_shard_budget_line(memory_budget_mb, memory_budget_source, total_layers, quantization, bytes_per_layer=assignment_bytes_per_layer)}\n" + f" {_shard_budget_line(memory_budget_mb, memory_budget_source, total_layers, quantization, bytes_per_layer=assignment_bytes_per_layer, safety_fraction=_runtime_shard_safety_fraction(device))}\n" f" Quantization: {quantization}\n" f" Endpoint: {endpoint}\n" f" Node ID: {tracker_node_id or 'unregistered'}\n" @@ -1192,7 +1267,7 @@ def run_startup( f"meshnet-node ready\n" f" Wallet: {address}\n" f" Shard: {shard_label}\n" - f" {_shard_budget_line(memory_budget_mb, memory_budget_source, assigned_total_layers, quantization, bytes_per_layer=assignment_bytes_per_layer)}\n" + f" {_shard_budget_line(memory_budget_mb, memory_budget_source, assigned_total_layers, quantization, bytes_per_layer=assignment_bytes_per_layer, safety_fraction=_runtime_shard_safety_fraction(device))}\n" f" Endpoint: {endpoint}\n" f" Node ID: {node_id}\n" f" Hardware: {hw_str}\n" diff --git a/packages/tracker/meshnet_tracker/server.py b/packages/tracker/meshnet_tracker/server.py index e165ce0..48f0216 100644 --- a/packages/tracker/meshnet_tracker/server.py +++ b/packages/tracker/meshnet_tracker/server.py @@ -1275,16 +1275,23 @@ def _deployment_summary(nodes: list[_NodeEntry], preset: dict | None) -> dict: } -def _max_layers_for_memory(memory_mb: int, total_layers: int, preset: dict | None = None) -> int: +def _max_layers_for_memory( + memory_mb: int, + total_layers: int, + preset: dict | None = None, + *, + device: str | None = None, +) -> int: if total_layers <= 0: return 0 if memory_mb <= 0: return max(1, total_layers // 2) memory_bytes = memory_mb * 1024 * 1024 bytes_per_layer = next(iter(_preset_bytes_per_layer(preset).values())) if preset is not None else 30 * 1024 * 1024 + safety_fraction = 0.55 if device == "cpu" else 0.8 return min( total_layers, - max(1, int((memory_bytes * 0.8) // bytes_per_layer)), + max(1, int((memory_bytes * safety_fraction) // bytes_per_layer)), ) @@ -5273,7 +5280,7 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler): required_start, required_end = _preset_layer_bounds(preset) total_l = required_end - required_start + 1 memory_mb = vram_mb if (device == "cuda" and vram_mb > 0) else ram_mb - max_layers = _max_layers_for_memory(memory_mb, total_l, preset) + max_layers = _max_layers_for_memory(memory_mb, total_l, preset, device=device) shard_start = required_start shard_end = min(required_end, shard_start + max_layers - 1) self._send_json(200, { @@ -5373,11 +5380,11 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler): memory_mb = vram_mb if (device == "cuda" and vram_mb > 0) else ram_mb resolved_name, best_preset = _resolve_model_preset(server.model_presets, str(best_repo)) if memory_mb > 0: - max_layers = _max_layers_for_memory(memory_mb, total_l, best_preset) + max_layers = _max_layers_for_memory(memory_mb, total_l, best_preset, device=device) elif device == "cuda" and vram_mb >= 8192: max_layers = total_l else: - max_layers = _max_layers_for_memory(memory_mb, total_l, best_preset) + max_layers = _max_layers_for_memory(memory_mb, total_l, best_preset, device=device) shard_start = best_gap_start shard_end = min(total_l - 1, shard_start + max_layers - 1) diff --git a/tests/test_node_startup.py b/tests/test_node_startup.py index fdf09a9..0385f46 100644 --- a/tests/test_node_startup.py +++ b/tests/test_node_startup.py @@ -1680,6 +1680,66 @@ def test_preset_startup_rejects_pinned_shard_above_memory_budget(tmp_path, monke tracker.stop() +def test_network_auto_join_clips_oversized_cpu_assignment(tmp_path, monkeypatch, capsys): + """Old trackers may assign too many CPU layers; node clips before model load.""" + import meshnet_node.startup as startup_mod + + torch_calls: list[dict] = [] + registrations: list[dict] = [] + + class FakeBackend: + total_layers = 40 + + class FakeTorchNodeServer: + def __init__(self, **kwargs): + torch_calls.append(kwargs) + self.backend = FakeBackend() + self.tracker_node_id = None + + def start(self): + return 7000 + + def stop(self): + pass + + oversized_assignment = { + "hf_repo": "unsloth/Qwen3.6-35B-A3B", + "model": "qwen3.6-35b-a3b", + "shard_start": 0, + "shard_end": 36, + "num_layers": 40, + "gap_found": False, + "bytes_per_layer": {"bfloat16": 1_797_594_419}, + "model_sources": [], + } + + monkeypatch.setattr( + startup_mod, + "detect_hardware", + lambda: {"device": "cpu", "gpu_name": None, "vram_mb": 0, "ram_mb": 79 * 1024}, + ) + monkeypatch.setattr(startup_mod, "TorchNodeServer", FakeTorchNodeServer) + monkeypatch.setattr(startup_mod, "_get_json", lambda *_args, **_kwargs: oversized_assignment) + monkeypatch.setattr(startup_mod, "_post_json", lambda _url, payload: registrations.append(payload) or {"node_id": "n1"}) + monkeypatch.setattr(startup_mod, "_start_heartbeat", lambda *_args, **_kwargs: None) + monkeypatch.setattr(startup_mod, "model_metadata_for", lambda *_args, **_kwargs: {"num_layers": 40}) + + node = run_startup( + tracker_url="http://127.0.0.1:8080", + wallet_path=tmp_path / "wallet.json", + tracker_source_disabled=True, + ) + try: + assert torch_calls[0]["shard_start"] == 0 + assert torch_calls[0]["shard_end"] == 24 + assert registrations[0]["shard_end"] == 24 + output = capsys.readouterr().out + assert "CPU-safe runtime budget fits 25/40 layers" in output + assert "layers 0-24" in output + finally: + node.stop() + + def test_preset_model_with_hf_repo_loads_torch_backend(tmp_path, monkeypatch, capsys): """Named presets that advertise hf_repo must load TorchNodeServer, not the stub server.""" import meshnet_node.startup as startup_mod @@ -1996,6 +2056,55 @@ def test_network_assign_gap_found_field(): tracker.stop() +def test_network_assign_uses_conservative_cpu_runtime_budget(): + """CPU assignments leave headroom for partial-load overhead, not just raw weights.""" + import json as _json + import urllib.request as _ur + + tracker = TrackerServer(model_presets={ + "qwen3.6-35b-a3b": { + "hf_repo": "unsloth/Qwen3.6-35B-A3B", + "aliases": ["unsloth/Qwen3.6-35B-A3B"], + "layers_start": 0, + "layers_end": 39, + "recommended": True, + "bytes_per_layer": {"bfloat16": 1_797_594_419}, + }, + }) + port = tracker.start() + try: + data = _json.dumps({ + "endpoint": "http://127.0.0.1:9200", + "model": "qwen3.6-35b-a3b", + "hf_repo": "unsloth/Qwen3.6-35B-A3B", + "num_layers": 40, + "shard_start": 0, + "shard_end": 39, + "hardware_profile": {}, + "score": 1.0, + }).encode() + req = _ur.Request( + f"http://127.0.0.1:{port}/v1/nodes/register", + data=data, + headers={"Content-Type": "application/json"}, + method="POST", + ) + with _ur.urlopen(req) as r: + r.read() + + resp = _get_json( + f"http://127.0.0.1:{port}/v1/network/assign" + "?device=cpu&vram_mb=0&ram_mb=80896" + "&hf_repo=unsloth/Qwen3.6-35B-A3B" + ) + + assert resp["gap_found"] is False + assert resp["shard_start"] == 0 + assert resp["shard_end"] == 24 + finally: + tracker.stop() + + def test_route_finds_hf_model_across_two_nodes(): """Tracker /v1/route returns ordered route for HF model even without a preset.""" import json as _json From 29db25108f5a4689ca5f7ab6a48076b00d0f4a8f Mon Sep 17 00:00:00 2001 From: Dobromir Popov Date: Wed, 8 Jul 2026 18:24:45 +0200 Subject: [PATCH 8/9] dash --- QUICKSTART.md | 77 +++++++++++-------- .../tracker/meshnet_tracker/dashboard.html | 69 +++++++++++++++-- tests/test_dashboard.py | 4 +- 3 files changed, 112 insertions(+), 38 deletions(-) diff --git a/QUICKSTART.md b/QUICKSTART.md index cfc7360..eabb88f 100644 --- a/QUICKSTART.md +++ b/QUICKSTART.md @@ -142,40 +142,52 @@ does not need any of this — it is a standard transformer with no FLA fast path - **transformers ≥ 5.12 required** — older versions fail with `'Qwen3_5MoeConfig' object has no attribute 'vocab_size'`. Check: `python -c "import transformers; print(transformers.__version__)"`. -- **Optional GPU kernels** — without them inference still works; startup may print - `The fast path is not available…` (harmless on Windows; slower on linear-attention - layers). Install **only for your platform**: +- **GPU fast path (optional)** — without it inference still works; startup prints + `The fast path is not available…` and linear-attention layers use a slower PyTorch + fallback. Install **only for your platform**: - | Platform | Install | Fast path? | - |----------|---------|------------| - | **Native Windows** | `pip install triton-windows` (also pulled by `meshnet-node` on Windows) | **Load only** — model imports and runs on PyTorch fallback. Do **not** `pip install flash-linear-attention[cuda] causal-conv1d` (see below). | - | **Linux + NVIDIA CUDA** | `pip install flash-linear-attention[cuda]` | **Yes** — `causal-conv1d` optional since FLA ≥0.3.2 ships Triton conv1d; add it only if transformers still asks for it. Needs CUDA toolkit (`nvcc`) or a matching prebuilt wheel. | - | **Linux + AMD ROCm** | `pip install flash-linear-attention[rocm]` | **Yes** — same optional `causal-conv1d` note. | + | Platform | Install | Notes | + |----------|---------|-------| + | **Native Windows + NVIDIA** | `pip install triton-windows` then `pip install flash-linear-attention` | **Fast path works.** FLA [officially supports `triton-windows`](https://github.com/fla-org/flash-linear-attention/pull/757) (tested Win11, PyTorch 2.10, triton-windows 3.6). Do **not** use the `[cuda]` extra on Windows — pip looks for Linux `triton` and fails. Do **not** install `causal-conv1d` — FLA ≥0.3.2 ships Triton conv1d; the separate package is Linux-only and breaks on Windows (`bare_metal_version` / nvcc errors). | + | **Linux + NVIDIA CUDA** | `pip install flash-linear-attention[cuda]` | `causal-conv1d` optional (same FLA built-in conv1d note). Needs CUDA toolkit (`nvcc`) matching torch, or a prebuilt wheel. | + | **Linux + AMD ROCm** | `pip install flash-linear-attention[rocm]` | Same optional `causal-conv1d` note. | - On native Windows, `triton-windows` is enough for Qwen3.6-MoE to **load**. Without it, + **Windows verify** (after install): + + ```powershell + python -c "import triton; import fla; print('triton', triton.__version__, 'fla ok')" + ``` + + `triton-windows` is also pulled by `meshnet-node` on Windows. Without it, Qwen3.6-MoE startup fails with misleading `Could not import module 'Qwen3_5MoeForCausalLM'`.
-Why there is no supported Windows fast path (research notes) +Windows fast path — what failed and what actually works -We checked upstream docs and real install attempts (2026-07): +The command that failed — `pip install flash-linear-attention[cuda] causal-conv1d` — mixes +two different things: -1. **`causal-conv1d`** — no official Windows wheels on PyPI. Source builds need `nvcc` - plus a torch/CUDA stack that matches; without `nvcc` pip fails with - `bare_metal_version is not defined`. Open upstream issues confirm Windows is - unsupported / broken for most setups. -2. **`flash-linear-attention[cuda]`** — FLA's install guide targets Linux (CUDA / ROCm / - XPU / NPU). The `[cuda]` extra pulls PyPI `triton`, which conflicts with - `triton-windows` on native Windows. FLA does not document or CI-test Windows. -3. **What works today on native Windows GPU** — CUDA torch + `triton-windows` → Qwen3.6 - loads and infers via Transformers' pure-PyTorch fallback (~¾ of layers are - linear-attention; those hops are slower). This is what we use in alpha. -4. **If you need the fast path on a Windows PC** — run the GPU node in **WSL2 with Linux - CUDA** (or a Linux box): `pip install flash-linear-attention[cuda]` there. Keep - Windows native for reachability / relay if needed. -5. **Experimental / unsupported** — community `causal-conv1d` wheels and - `triton-windows` + pinned `flash-linear-attention --no-deps` hacks exist for narrow - Python/torch/CUDA combos; we do not support or test these for meshnet-node. +1. **`flash-linear-attention[cuda]` on Windows** — wrong extra. `[cuda]` pulls PyPI + `triton>=3.3`, which does not exist for Windows (`No matching distribution found`). + Use plain `pip install flash-linear-attention` **after** `triton-windows` is already + installed; FLA detects `triton-windows` and uses it. +2. **`causal-conv1d`** — separate Dao-AILab CUDA extension, **not required** for FLA or + Qwen3.6 when FLA is installed. No official Windows wheels. Source builds need `nvcc` + whose major version matches torch's CUDA (e.g. torch `+cu118` needs CUDA 11.8 toolkit, + not 12.5). Community wheels exist for narrow Python/torch combos + ([PR #46](https://github.com/Dao-AILab/causal-conv1d/pull/46)) but we skip them. + +**Working Windows stack** (confirmed on this repo's dev machine: Python 3.12, torch +2.7.1+cu118, triton-windows 3.7.1, flash-linear-attention 0.5.0): + +```powershell +pip install triton-windows +pip install -U flash-linear-attention +python -c "import triton; import fla; print('ok')" +``` + +If the fast-path warning persists after that, upgrade FLA to ≥0.5.1 (includes the +`triton-windows` detection from PR #757) and restart the node.
@@ -289,9 +301,8 @@ curl -sI https://ai.neuron.d-popov.com/rpc/test-peer no HuggingFace gating. Best for first-time setup. **Alpha model:** `qwen3.6-35b-a3b` — 40 layers, ~72 GB BF16 download, MoE with hybrid -linear attention. Use on machines with enough RAM/VRAM; see Qwen3.5/3.6 notes above for -`triton-windows` (Windows) or `flash-linear-attention` (Linux GPU). Tracker accepts the -alias or full repo id (`unsloth/Qwen3.6-35B-A3B`). +linear attention. On Windows install `triton-windows` + `flash-linear-attention`; on Linux +GPU use `flash-linear-attention[cuda]`. Tracker accepts the alias or full repo id (`unsloth/Qwen3.6-35B-A3B`). Downloads cache under `~/.meshnet/models/` (or `$HF_HOME` / `$env:HF_HOME`). @@ -330,13 +341,17 @@ HF_HOME=/path/to/models .venv/bin/meshnet-node start --tracker http://localhost: .venv/bin/meshnet-node start --tracker https://ai.neuron.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct ``` -**Alpha model (Qwen3.6, Windows GPU — PyTorch fallback, no FLA fast path):** +**Alpha model (Qwen3.6, Windows GPU — enable fast path):** ```powershell $env:HF_HOME = "D:\DEV\models" +pip install triton-windows +pip install -U flash-linear-attention meshnet-node start --tracker http://192.168.0.179:8080 --model qwen3.6-35b-a3b --quantization bfloat16 ``` +Do not add `causal-conv1d` or `flash-linear-attention[cuda]` on Windows (see Qwen3.5/3.6 notes). + **Alpha model (Qwen3.6, Linux GPU — with fast path):** ```bash diff --git a/packages/tracker/meshnet_tracker/dashboard.html b/packages/tracker/meshnet_tracker/dashboard.html index 40c6145..9fec492 100644 --- a/packages/tracker/meshnet_tracker/dashboard.html +++ b/packages/tracker/meshnet_tracker/dashboard.html @@ -279,6 +279,41 @@ const tps = v => (v === null || v === undefined) ? "?" : (Math.round(v * 10) / 1 const copies = v => (v === null || v === undefined) ? "?" : Number(v).toFixed(2); const short = (s, n=14) => { s = String(s); return s.length > n ? s.slice(0, 6) + "…" + s.slice(-5) : s; }; +function modelAliasKey(value) { + if (!value) return ""; + const text = String(value).trim(); + if (!text) return ""; + const shortName = text.includes("/") ? text.split("/").pop() : text; + return shortName.toLowerCase(); +} + +function buildModelAliasMap(map) { + const byAlias = new Map(); + const register = (display, ...names) => { + if (!display) return; + for (const name of names) { + const key = modelAliasKey(name); + if (key) byAlias.set(key, display); + } + }; + for (const entry of (map && map.recommended_models) || []) { + register(entry.id, entry.id, entry.hf_repo, ...(entry.aliases || [])); + } + for (const entry of availableModels || []) { + register(entry.name || entry.id, entry.id, entry.name, ...(entry.aliases || [])); + } + return byAlias; +} + +function resolveModelGroup(node, aliasMap) { + for (const candidate of [node.hf_repo, node.model]) { + if (!candidate) continue; + const hit = aliasMap.get(modelAliasKey(candidate)); + if (hit) return hit; + } + return node.hf_repo || node.model || "?"; +} + async function fetchJson(path) { try { const headers = {}; @@ -315,15 +350,20 @@ function renderNodes(map) { if (!nodes.length) { $("nodes").innerHTML = '
no nodes registered
'; return; } + const aliasMap = buildModelAliasMap(map); const byModel = {}; for (const n of nodes) { - const key = n.model || n.hf_repo || "?"; + const key = resolveModelGroup(n, aliasMap); (byModel[key] = byModel[key] || []).push(n); } + const modelNames = Object.keys(byModel).sort((a, b) => a.localeCompare(b)); let html = ""; - for (const [model, group] of Object.entries(byModel)) { - const supply = group.find(n => n.model_supply && n.model_supply.served_model_copies !== undefined); - const served = supply && supply.model_supply && supply.model_supply.served_model_copies; + for (const model of modelNames) { + const group = byModel[model]; + const servedValues = group + .map(n => n.model_supply && n.model_supply.served_model_copies) + .filter(v => v !== null && v !== undefined); + const served = servedValues.length ? Math.max(...servedValues) : undefined; html += `
${esc(model)} (${group.length} node${group.length===1?"":"s"} · ${esc(copies(served))} served)
`; html += table(["node", "shard", "tps (1h)", "queue", "served"], group.map(n => { const modelStats = (n.throughput && (n.throughput[n.hf_repo] || n.throughput[n.model])) || {}; @@ -1552,6 +1592,19 @@ async function refresh() { renderChatHistory(); $("refreshed").textContent = "refreshed " + new Date().toLocaleTimeString(); } + +const REFRESH_MS = 10000; + +function selectionActive() { + const sel = window.getSelection(); + return sel && !sel.isCollapsed && sel.toString().length > 0; +} + +async function refreshIfIdle() { + if (selectionActive()) return; + await refresh(); +} + refresh(); initChatSessions(); bindChatPromptShortcuts(); @@ -1559,8 +1612,12 @@ renderAccountPanel(); renderChatModels(); renderChatHistory(); renderChatAuthHint(); -setInterval(refresh, 4000); -setInterval(() => { if (sessionToken || isLoggedIn) renderAccountPanel(); }, 8000); +setInterval(refreshIfIdle, REFRESH_MS); +setInterval(() => { + if (!sessionToken && !isLoggedIn) return; + if (selectionActive()) return; + renderAccountPanel(); +}, REFRESH_MS); diff --git a/tests/test_dashboard.py b/tests/test_dashboard.py index 421cfac..632f019 100644 --- a/tests/test_dashboard.py +++ b/tests/test_dashboard.py @@ -28,7 +28,9 @@ def test_dashboard_served_with_all_panels(): ).read().decode() for panel in PANELS: assert panel in html - assert "