new tasks, model pricing, auto quantisation, etc...

This commit is contained in:
Dobromir Popov
2026-07-06 17:11:53 +03:00
parent 7f67e29d76
commit ccb69c41e3
14 changed files with 466 additions and 34 deletions

Binary file not shown.

View File

@@ -0,0 +1,55 @@
# US-042 — GGUF/llama.cpp node backend
Status: planned
Priority: High (unlocks big MoE models on volunteer hardware — the pool's core value)
Stage: Draft design
## Context
The node backend is transformers-only (`model_backend.py`
`AutoModelForCausalLM`). For DeepSeek-V4-Flash (158B MoE, official weights FP8
160 GB) the only quantizations that run on consumer hardware are GGUF
(IQ2 87 GB → Q4_K_M-XL 175 GB) — llama.cpp format. The transformers-compatible
quants (FP8, NVFP4, GPTQ W4A16) all need datacenter GPUs. Volunteer machines —
including our own Strix Halo boxes (128 GB and 80 GB unified memory, GPU via
Vulkan/ROCm, no FP8 support on RDNA3.5) — run these models today only under
llama.cpp.
## Design directions to evaluate (design-it-twice)
**A. llama.cpp as a per-node shard executor.** Node loads a *layer range* of a
GGUF via llama-cpp-python; our existing hop protocol (X-Meshnet-Route,
activations over HTTP/relay) moves hidden states between nodes. Requires
llama.cpp partial-layer loading and activation import/export — investigate
feasibility first; this is the riskiest unknown.
**B. llama.cpp RPC mode under tracker orchestration.** llama.cpp ships a
native RPC backend that splits one model across machines. The tracker would
provision/route to an llama.cpp RPC cluster rather than our own hop pipeline.
Less code, but bypasses our billing/telemetry hop instrumentation and relay
NAT path — needs a story for both.
**C. Whole-model GGUF nodes (no sharding).** A node with enough memory serves
a full GGUF (e.g. IQ2/IQ3 on a 128 GB box); the tracker routes whole requests
to it (single-hop route). Smallest step, no cross-node activation work, and
already useful: Strix Halo 128 GB serves DeepSeek-V4-Flash IQ3_XXS (114 GB)
via llama.cpp Vulkan today.
Recommended sequencing: C first (small, real value), then A/B investigation.
## Also in scope
- Model catalog: allow GGUF entries with quant selection; feature
`DeepSeek-V4-Flash` IQ4_XS/UD-Q4_K_XL as a curated/featured entry once at
least direction C works (a featured model nobody can load is an anti-feature)
- Hardware detection: recognize Strix Halo/unified-memory APUs and Vulkan
(`hardware.py` currently reports "CPU mode" on these boxes)
- `MESHNET_DOWNLOAD_DIR`/`--download-dir` applies to GGUF files as well
## Acceptance criteria (phase C)
- A node with `--gguf <repo-or-path> --quant IQ3_XXS` serves
`/v1/chat/completions` via llama.cpp with GPU offload where available
- Tracker treats it as a full-coverage node (single-hop routes, billing works)
- Streamed responses work through the tracker proxy and the relay (US-036)
- `python -m pytest` passes from repo root (llama.cpp behind an optional extra)

View File

@@ -0,0 +1,38 @@
# US-043 — Dashboard model search and model cards
Status: planned
Priority: Medium (post-deploy polish)
Stage: Idea
## Context
The dashboard shows nodes/routes/billing but nothing model-centric. Operators
and testers should be able to search for a model and see, per model, a card
with what the network knows about it.
## Scope
- **Search**: query box hitting a new tracker endpoint that proxies the HF Hub
search API (server-side, so the dashboard stays CSP-clean and unauthenticated
browsers aren't rate-limited) merged with the tracker's own model presets and
currently-served models.
- **Model card** per result:
- name, architecture, params, layer count (reuse `model_metadata_for`,
which now handles nested `text_config` — US layer-detection fix)
- coverage on the network: which layer ranges are served, by how many nodes,
coverage gaps (the Coverage Map already exists on the tracker)
- price per 1K tokens, availability (routable now? single-hop or pipeline?)
- memory footprint per quantization where known (bf16 / GGUF sizes)
- action: "request this model" — flags demand so node operators (or
auto-shard assignment) know what to load next
- Featured models section driven by the curated catalog (`CURATED_MODELS`),
including GGUF entries once US-042 lands.
## Acceptance criteria
- Searching a HF repo id or free text returns results without the browser
calling HF directly
- A served model's card shows live coverage and a working "chat now" state
- An unserved model's card shows the "request" action and estimated memory
per quant
- `python -m pytest` passes from repo root

View File

@@ -0,0 +1,65 @@
# US-044 — Tracker as model-file source; nodes download only their shard
Status: planned
Priority: High (blocks multi-machine big-model serving; pairs with US-042)
Stage: Designed (grill remaining decisions before build)
## Context
Common deployment: the tracker and the first node share a machine that already
holds the model files (e.g. `/run/media/popov/DATA/llm/safetensor`). When a
second node joins with no model selected, the tracker assigns it the uncovered
layer range — and today that node then downloads the **entire snapshot from
HuggingFace**, even for a 20-layer shard of a 160 GB model.
What exists already (build on it, don't duplicate):
- Nodes serve their shard dir as a tar at `GET /v1/shards/download` with
checksum verification; `download_shard` tries assignment-provided `peers`
before HF (`downloader.py`). But it only matches **identical layer ranges**,
and the HF fallback runs `snapshot_download` of the whole repo.
- The torch path (`--model-id`) bypasses `download_shard` entirely:
`TorchModelShard``from_pretrained` downloads **and loads into RAM** the
full model, then executes only the assigned layers. Sharding currently saves
compute, not memory or bandwidth.
## Design
1. **Tracker `--models-dir PATH`** (env `MESHNET_MODELS_DIR`). When set, the
tracker indexes HF-layout snapshots under it and advertises itself as a
model-file source in `/v1/nodes/assign` responses.
2. **Layer-aware file selection.** For safetensors models, read
`model.safetensors.index.json` and map the assigned layer range → the
subset of weight files containing those layers, plus the always-needed
files (config, tokenizer, index, embeddings/head files for head/tail
shards). Serve exactly that subset (tar stream, per-file checksums).
GGUF (US-042): single file or naive byte-range — phase 2.
3. **Node download order**: exact-shard peer (existing) → tracker/peer file
subset (new) → HF `snapshot_download` with `allow_patterns` for the same
subset (new — stop downloading the whole repo even from HF) → full snapshot
(last resort).
4. **Partial LOAD (the hard half).** Downloading a subset is wasted unless the
node stops instantiating the full model: build the model skeleton on the
`meta` device, materialize only assigned layers (+embeddings/norm/head as
role requires) from the local files, leave the rest on meta. Without this,
an 80 GB machine can never hold a shard of a 160 GB model regardless of
how the bytes arrive. This is the acceptance bar for the issue.
## Open questions (grill before building)
- Trust: joining nodes fetch weights from the tracker/peers — checksum against
what root of trust? (HF etag/sha vs tracker-signed manifest.)
- Disk layout: partial snapshots must not corrupt the HF cache dir; probably
a meshnet-owned layout keyed by repo+revision.
- Serving cost: a 100 GB tar stream per joining node on the tracker box —
rate-limit/queue? LAN-only heuristic?
## Acceptance criteria
- Two-machine test: machine A (tracker + node, holds full snapshot) serves
layers 0k; machine B joins with no model and receives **only** the files
for its assigned range from A — nothing fetched from HF
- Machine B's resident memory scales with its shard size, not model size
- Checksums verified end-to-end; corrupted transfer falls back cleanly
- Single-node/full-model flows unchanged
- `python -m pytest` passes from repo root

Binary file not shown.

View File

@@ -56,7 +56,7 @@ def _run_node(cfg: dict) -> None:
model_id=cfg.get("model_hf_repo") or None, model_id=cfg.get("model_hf_repo") or None,
shard_start=cfg.get("shard_start"), shard_start=cfg.get("shard_start"),
shard_end=cfg.get("shard_end"), shard_end=cfg.get("shard_end"),
quantization=cfg.get("quantization", "int8").replace("bf16", "bfloat16"), quantization=cfg.get("quantization", "auto").replace("bf16", "bfloat16"),
wallet_path=Path(cfg["wallet_path"]) if cfg.get("wallet_path") else None, wallet_path=Path(cfg["wallet_path"]) if cfg.get("wallet_path") else None,
cache_dir=Path(cfg["download_dir"]) if cfg.get("download_dir") else None, cache_dir=Path(cfg["download_dir"]) if cfg.get("download_dir") else None,
host=cfg.get("host", "0.0.0.0"), host=cfg.get("host", "0.0.0.0"),
@@ -199,17 +199,19 @@ def _cmd_config(args) -> int:
def _cmd_start(args) -> int: def _cmd_start(args) -> int:
"""Legacy `start` subcommand — preserves backward compatibility with existing tests.""" """Legacy `start` subcommand — preserves backward compatibility with existing tests."""
from .config import load_config, DEFAULTS from .config import DEFAULTS
# Build a transient config from flags (don't write to disk) # Build a transient config from flags (don't write to disk)
cfg = dict(DEFAULTS) cfg = dict(DEFAULTS)
cfg["tracker_url"] = args.tracker if args.tracker:
cfg["tracker_url"] = args.tracker
cfg["port"] = args.port if args.port is not None else _first_available_port(args.host) cfg["port"] = args.port if args.port is not None else _first_available_port(args.host)
if args.model_id is None and "/" in args.model: model = args.model or cfg.get("model_hf_repo") or cfg.get("model_name") or "stub-model"
cfg["model_hf_repo"] = args.model if args.model_id is None and "/" in model:
cfg["model_name"] = args.model.split("/")[-1] cfg["model_hf_repo"] = model
cfg["model_name"] = model.split("/")[-1]
else: else:
cfg["model_name"] = args.model cfg["model_name"] = model
cfg["quantization"] = args.quantization cfg["quantization"] = args.quantization
cfg["host"] = args.host cfg["host"] = args.host
if args.model_id: if args.model_id:
@@ -307,13 +309,13 @@ def main() -> None:
# start subcommand (legacy / backward-compat) # start subcommand (legacy / backward-compat)
start_cmd = subparsers.add_parser("start", help="Start node (legacy flags)") start_cmd = subparsers.add_parser("start", help="Start node (legacy flags)")
start_cmd.add_argument("--tracker", default="http://localhost:8080") start_cmd.add_argument("--tracker")
start_cmd.add_argument("--port", type=int) start_cmd.add_argument("--port", type=int)
start_cmd.add_argument("--model", default="stub-model") start_cmd.add_argument("--model")
start_cmd.add_argument("--model-id", help="HuggingFace repo ID") start_cmd.add_argument("--model-id", help="HuggingFace repo ID")
start_cmd.add_argument("--shard-start", type=int) start_cmd.add_argument("--shard-start", type=int)
start_cmd.add_argument("--shard-end", type=int) start_cmd.add_argument("--shard-end", type=int)
start_cmd.add_argument("--quantization", choices=["bfloat16", "int8", "nf4", "bf16"], default="int8") start_cmd.add_argument("--quantization", choices=["auto", "bfloat16", "int8", "nf4", "bf16"], default="auto")
start_cmd.add_argument("--host", default="0.0.0.0") start_cmd.add_argument("--host", default="0.0.0.0")
start_cmd.add_argument("--advertise-host") start_cmd.add_argument("--advertise-host")
start_cmd.add_argument("--tracker-mode", action="store_true") start_cmd.add_argument("--tracker-mode", action="store_true")

View File

@@ -14,13 +14,16 @@ _DEFAULT_CONFIG_FILE = _DEFAULT_CONFIG_DIR / "config.json"
_DEFAULT_DOWNLOAD_DIR = Path( _DEFAULT_DOWNLOAD_DIR = Path(
os.environ.get("MESHNET_DOWNLOAD_DIR", str(Path.home() / ".meshnet" / "models")) os.environ.get("MESHNET_DOWNLOAD_DIR", str(Path.home() / ".meshnet" / "models"))
) )
_DEFAULT_TRACKER_URL = "http://localhost:8080" _DEFAULT_TRACKER_URL = os.environ.get("MESHNET_TRACKER_URL", "http://localhost:8080")
_DEFAULT_WALLET_PATH = str(Path.home() / ".config" / "meshnet" / "wallet.json") _DEFAULT_WALLET_PATH = str(Path.home() / ".config" / "meshnet" / "wallet.json")
_DEFAULT_QUANTIZATION = "nf4" _DEFAULT_QUANTIZATION = "auto"
_DEFAULT_MODEL = os.environ.get("MESHNET_MODEL_ID") or os.environ.get("MESHNET_MODEL", "")
_DEFAULT_MODEL_HF_REPO = _DEFAULT_MODEL if "/" in _DEFAULT_MODEL else ""
_DEFAULT_MODEL_NAME = _DEFAULT_MODEL.split("/")[-1] if "/" in _DEFAULT_MODEL else _DEFAULT_MODEL
DEFAULTS = { DEFAULTS = {
"model_hf_repo": "", "model_hf_repo": _DEFAULT_MODEL_HF_REPO,
"model_name": "", "model_name": _DEFAULT_MODEL_NAME,
"quantization": _DEFAULT_QUANTIZATION, "quantization": _DEFAULT_QUANTIZATION,
"download_dir": str(_DEFAULT_DOWNLOAD_DIR), "download_dir": str(_DEFAULT_DOWNLOAD_DIR),
"tracker_url": _DEFAULT_TRACKER_URL, "tracker_url": _DEFAULT_TRACKER_URL,

View File

@@ -7,7 +7,7 @@ from dataclasses import dataclass
from pathlib import Path from pathlib import Path
from typing import Any, Literal from typing import Any, Literal
Quantization = Literal["bfloat16", "int8", "nf4"] Quantization = Literal["auto", "bfloat16", "int8", "nf4"]
class ModelBackendError(RuntimeError): class ModelBackendError(RuntimeError):
@@ -31,14 +31,14 @@ class TensorPayload:
def validate_quantization(value: str) -> Quantization: def validate_quantization(value: str) -> Quantization:
if value not in {"bfloat16", "int8", "nf4"}: if value not in {"auto", "bfloat16", "int8", "nf4"}:
raise ValueError("quantization must be one of: bfloat16, int8, nf4") raise ValueError("quantization must be one of: auto, bfloat16, int8, nf4")
return value # type: ignore[return-value] return value # type: ignore[return-value]
def build_quantization_config(quantization: Quantization) -> Any | None: def build_quantization_config(quantization: Quantization) -> Any | None:
"""Return a transformers BitsAndBytesConfig for quantized weights.""" """Return a transformers BitsAndBytesConfig for quantized weights."""
if quantization == "bfloat16": if quantization in {"auto", "bfloat16"}:
return None return None
try: try:
import torch import torch
@@ -65,7 +65,7 @@ class TorchModelShard:
model_id: str, model_id: str,
shard_start: int, shard_start: int,
shard_end: int, shard_end: int,
quantization: Quantization = "bfloat16", quantization: Quantization = "auto",
cache_dir: Path | None = None, cache_dir: Path | None = None,
) -> None: ) -> None:
if shard_start < 0 or shard_end < 0 or shard_start > shard_end: if shard_start < 0 or shard_end < 0 or shard_start > shard_end:
@@ -77,7 +77,7 @@ class TorchModelShard:
try: try:
import torch import torch
from transformers import AutoModelForCausalLM, AutoTokenizer from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer
except ModuleNotFoundError as exc: except ModuleNotFoundError as exc:
raise MissingModelDependencyError( raise MissingModelDependencyError(
"real model backend requires torch, transformers, safetensors, accelerate, and bitsandbytes" "real model backend requires torch, transformers, safetensors, accelerate, and bitsandbytes"
@@ -85,17 +85,27 @@ class TorchModelShard:
self.torch = torch self.torch = torch
self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu") self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
quant_config = build_quantization_config(quantization) quant_config, dtype, uses_quantized_weights = _model_load_plan(
AutoConfig,
model_id,
quantization,
torch,
cache_dir,
)
try: try:
load_kwargs = {
"device_map": "auto" if uses_quantized_weights else None,
"dtype": dtype,
"low_cpu_mem_usage": True,
"cache_dir": str(cache_dir) if cache_dir is not None else None,
}
if quant_config is not None:
load_kwargs["quantization_config"] = quant_config
self.model = AutoModelForCausalLM.from_pretrained( self.model = AutoModelForCausalLM.from_pretrained(
model_id, model_id,
quantization_config=quant_config, **load_kwargs,
device_map="auto" if quant_config is not None else None,
dtype=torch.bfloat16,
low_cpu_mem_usage=True,
cache_dir=str(cache_dir) if cache_dir is not None else None,
) )
if quant_config is None: if not uses_quantized_weights:
self.model.to(self.device) self.model.to(self.device)
except Exception as exc: except Exception as exc:
if _looks_like_oom(exc): if _looks_like_oom(exc):
@@ -340,12 +350,71 @@ def load_torch_shard(
model_id: str, model_id: str,
shard_start: int, shard_start: int,
shard_end: int, shard_end: int,
quantization: Quantization = "bfloat16", quantization: Quantization = "auto",
cache_dir: Path | None = None, cache_dir: Path | None = None,
) -> TorchModelShard: ) -> TorchModelShard:
return TorchModelShard(model_id, shard_start, shard_end, quantization, cache_dir) return TorchModelShard(model_id, shard_start, shard_end, quantization, cache_dir)
def _model_load_plan(
auto_config: Any,
model_id: str,
quantization: Quantization,
torch: Any,
cache_dir: Path | None = None,
) -> tuple[Any | None, Any, bool]:
"""Return (explicit quant config, dtype, uses quantized weights)."""
if quantization != "auto":
quant_config = build_quantization_config(quantization)
return quant_config, torch.bfloat16, quant_config is not None
cfg = auto_config.from_pretrained(
model_id,
cache_dir=str(cache_dir) if cache_dir is not None else None,
)
if _native_quantization_config(cfg) is not None:
return None, _native_torch_dtype(cfg, torch), True
return None, _native_torch_dtype(cfg, torch), False
def _config_candidates(cfg: Any) -> list[Any]:
candidates = [cfg]
get_text_config = getattr(cfg, "get_text_config", None)
if callable(get_text_config):
try:
candidates.append(get_text_config())
except Exception:
pass
text_config = getattr(cfg, "text_config", None)
if text_config is not None:
candidates.append(text_config)
return candidates
def _native_quantization_config(cfg: Any) -> Any | None:
for candidate in _config_candidates(cfg):
quant_config = getattr(candidate, "quantization_config", None)
if quant_config:
return quant_config
return None
def _native_torch_dtype(cfg: Any, torch: Any) -> Any:
for candidate in _config_candidates(cfg):
for attr in ("dtype", "torch_dtype"):
dtype = getattr(candidate, attr, None)
if dtype is None:
continue
if isinstance(dtype, str):
dtype_name = dtype.removeprefix("torch.")
dtype_value = getattr(torch, dtype_name, None)
if dtype_value is not None:
return dtype_value
else:
return dtype
return torch.bfloat16
def _model_layers(model: Any) -> Any: def _model_layers(model: Any) -> Any:
if hasattr(model, "model") and hasattr(model.model, "layers"): if hasattr(model, "model") and hasattr(model.model, "layers"):
return model.model.layers return model.model.layers

View File

@@ -52,6 +52,7 @@ def _max_assignable_layers(memory_mb: int, total_layers: int | None) -> int:
def _shard_budget_line(memory_mb: int, memory_source: str, total_layers: int | None, quantization: str) -> str: def _shard_budget_line(memory_mb: int, memory_source: str, total_layers: int | None, quantization: str) -> str:
memory_gb = memory_mb / 1024 memory_gb = memory_mb / 1024
gb_str = f"{memory_gb:.1f} GB" gb_str = f"{memory_gb:.1f} GB"
budget_quantization = "bfloat16" if quantization == "auto" else quantization
if total_layers is None or total_layers <= 0: if total_layers is None or total_layers <= 0:
return f"Memory budget: {gb_str} {memory_source}; shard budget: unknown model layer count" return f"Memory budget: {gb_str} {memory_source}; shard budget: unknown model layer count"
max_layers = _max_assignable_layers(memory_mb, total_layers) max_layers = _max_assignable_layers(memory_mb, total_layers)
@@ -61,7 +62,7 @@ def _shard_budget_line(memory_mb: int, memory_source: str, total_layers: int | N
remaining_str = f"; {remaining_gb:.1f} GB remaining after full load" if remaining_gb > 1 else "" remaining_str = f"; {remaining_gb:.1f} GB remaining after full load" if remaining_gb > 1 else ""
return ( return (
f"Memory budget: {gb_str} {memory_source}; " f"Memory budget: {gb_str} {memory_source}; "
f"Shard budget: up to {max_layers}/{total_layers} layers at {quantization}" f"Shard budget: up to {max_layers}/{total_layers} layers at {budget_quantization}"
f"{remaining_str}" f"{remaining_str}"
) )
@@ -306,7 +307,7 @@ def run_startup(
model_id: str | None = None, model_id: str | None = None,
shard_start: int | None = None, shard_start: int | None = None,
shard_end: int | None = None, shard_end: int | None = None,
quantization: str = "bfloat16", quantization: str = "auto",
wallet_path: Path | None = None, wallet_path: Path | None = None,
cache_dir: Path | None = None, cache_dir: Path | None = None,
host: str = "127.0.0.1", host: str = "127.0.0.1",

View File

@@ -12,7 +12,7 @@
"recommended": true, "recommended": true,
"deployment_status": "recommended", "deployment_status": "recommended",
"hf_aliases": [], "hf_aliases": [],
"hf_verified_match_note": "Pending human curation (issue 23) no HF inference-marketplace listing has been confirmed as a comparable params/quantization match for this preset yet. Leave empty until a human signs off; an empty hf_aliases list keeps this model on the static default price.", "hf_verified_match_note": "Pending human curation (issue 23) \u2014 no HF inference-marketplace listing has been confirmed as a comparable params/quantization match for this preset yet. Leave empty until a human signs off; an empty hf_aliases list keeps this model on the static default price.",
"required_model_bytes": 638876385280, "required_model_bytes": 638876385280,
"download_size_bytes": 638876385280, "download_size_bytes": 638876385280,
"native_quantization": "int4", "native_quantization": "int4",
@@ -38,6 +38,41 @@
"KTransformers" "KTransformers"
] ]
} }
},
"qwen3.6-35b-a3b": {
"layers_start": 0,
"layers_end": 39,
"hf_repo": "unsloth/Qwen3.6-35B-A3B",
"aliases": [
"qwen3.6-35b-a3b",
"Qwen3.6-35B-A3B",
"unsloth/Qwen3.6-35B-A3B",
"Qwen/Qwen3.6-35B-A3B"
],
"recommended": true,
"deployment_status": "recommended",
"price_per_1k_tokens": 0.00044,
"hf_aliases": [
"qwen/qwen3.6-35b-a3b"
],
"hf_verified_match_note": "Verified 2026-07-06: unsloth/Qwen3.6-35B-A3B is a bf16 mirror of Qwen/Qwen3.6-35B-A3B; deepinfra and featherless-ai serve the official weights on the HF inference marketplace, so their rates are a fair comparable. Static price 0.00044 = 80% of deepinfra's blended $0.55/1M ($0.15 in / $0.95 out); the nightly refresher keeps it tracking.",
"required_model_bytes": 71903776776,
"download_size_bytes": 71903776776,
"native_quantization": "bfloat16",
"canonical_audit_dtype": "bfloat16",
"canonical_audit_quantization": "bfloat16",
"bytes_per_layer": {
"bfloat16": 1797594419
},
"metadata": {
"architecture": "Mixture-of-Experts (MoE, hybrid linear attention)",
"total_parameters": "35B",
"activated_parameters": "3B",
"num_layers": 40,
"context_length": 262144,
"native_quantization": "bfloat16",
"download_size_gb": 72
}
} }
} }
} }

View File

@@ -46,6 +46,24 @@ from .gossip import NodeGossip
from .raft import RaftNode from .raft import RaftNode
def _preset_price_keys(name: str, preset: dict) -> set[str]:
"""All model strings a client may bill under for one preset.
``BillingLedger.price_for`` is keyed by the raw ``model`` string in the
request, so the preset price must be registered under the preset name,
its ``hf_repo``, and every alias — otherwise ``unsloth/Qwen…`` style
requests silently fall back to the default rate.
"""
keys = {name}
hf_repo = preset.get("hf_repo")
if isinstance(hf_repo, str) and hf_repo:
keys.add(hf_repo)
for alias in preset.get("aliases") or []:
if isinstance(alias, str) and alias:
keys.add(alias)
return keys
def derive_relay_url_from_public_tracker_url(url: str | None) -> str | None: def derive_relay_url_from_public_tracker_url(url: str | None) -> str | None:
"""Return wss://host/ws when url is a public HTTPS tracker origin.""" """Return wss://host/ws when url is a public HTTPS tracker origin."""
if not url: if not url:
@@ -4065,9 +4083,10 @@ class TrackerServer:
db_path = DEFAULT_BILLING_DB_PATH db_path = DEFAULT_BILLING_DB_PATH
if db_path: if db_path:
preset_prices = { preset_prices = {
name: float(preset["price_per_1k_tokens"]) key: float(preset["price_per_1k_tokens"])
for name, preset in self._model_presets.items() for name, preset in self._model_presets.items()
if isinstance(preset, dict) and "price_per_1k_tokens" in preset if isinstance(preset, dict) and "price_per_1k_tokens" in preset
for key in _preset_price_keys(name, preset)
} }
billing = BillingLedger(db_path=db_path, prices=preset_prices) billing = BillingLedger(db_path=db_path, prices=preset_prices)
self._billing: BillingLedger | None = billing self._billing: BillingLedger | None = billing
@@ -4332,7 +4351,8 @@ class TrackerServer:
continue continue
if result is None: if result is None:
continue continue
billing.set_price(name, result["new_price_per_1k"]) for key in _preset_price_keys(name, preset):
billing.set_price(key, result["new_price_per_1k"])
preset["hf_last_price_per_1k"] = result["new_price_per_1k"] preset["hf_last_price_per_1k"] = result["new_price_per_1k"]
preset["hf_last_updated"] = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()) preset["hf_last_updated"] = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
if self._hf_pricing_log is not None: if self._hf_pricing_log is not None:

View File

@@ -148,3 +148,41 @@ def test_hf_pricing_log_persists_and_is_queryable(tmp_path):
# Reopening against the same db path recovers the log (billing.py pattern). # Reopening against the same db path recovers the log (billing.py pattern).
reopened = HfPricingLog(db_path=db_path) reopened = HfPricingLog(db_path=db_path)
assert len(reopened.history()) == 1 assert len(reopened.history()) == 1
def test_preset_price_keys_cover_name_repo_and_aliases():
from meshnet_tracker.server import _preset_price_keys
preset = {
"hf_repo": "unsloth/Qwen3.6-35B-A3B",
"aliases": ["qwen3.6-35b-a3b", "Qwen/Qwen3.6-35B-A3B"],
}
keys = _preset_price_keys("qwen3.6-35b-a3b", preset)
assert keys == {
"qwen3.6-35b-a3b",
"unsloth/Qwen3.6-35B-A3B",
"Qwen/Qwen3.6-35B-A3B",
}
assert _preset_price_keys("bare", {}) == {"bare"}
def test_qwen_preset_prices_apply_to_all_aliases(tmp_path):
"""Requests naming the repo id (what nodes register) bill at the preset price."""
from meshnet_tracker.server import TrackerServer
import pytest
tracker = TrackerServer(billing_db=str(tmp_path / "billing.sqlite"))
try:
billing = tracker._billing
assert billing is not None
for key in (
"qwen3.6-35b-a3b",
"unsloth/Qwen3.6-35B-A3B",
"Qwen/Qwen3.6-35B-A3B",
):
assert billing.price_for(key) == pytest.approx(0.00044), key
# Unknown models keep the default rate
assert billing.price_for("some/other-model") == pytest.approx(0.02)
finally:
pass

View File

@@ -388,6 +388,48 @@ def test_legacy_start_treats_repo_model_as_model_id(monkeypatch):
assert captured["model_id"] == "Qwen/Qwen2.5-0.5B-Instruct" assert captured["model_id"] == "Qwen/Qwen2.5-0.5B-Instruct"
def test_legacy_start_falls_back_to_env_tracker_and_model(monkeypatch):
"""`meshnet-node start` uses env defaults when tracker/model flags are omitted."""
import importlib
from meshnet_node import config as config_mod
from meshnet_node.cli import main
monkeypatch.setenv("MESHNET_TRACKER_URL", "http://env-tracker:8081")
monkeypatch.setenv("MESHNET_MODEL", "Qwen/Qwen2.5-0.5B-Instruct")
importlib.reload(config_mod)
captured = {}
def fake_run_startup(*args, **kwargs):
captured.update(kwargs)
class _FakeNode:
chat_completion_count = 0
def stop(self): pass
return _FakeNode()
monkeypatch.setattr(sys, "argv", [
"meshnet-node", "start",
"--port", "0",
])
try:
with patch("meshnet_node.startup.run_startup", side_effect=fake_run_startup):
with patch("time.sleep", side_effect=KeyboardInterrupt):
try:
main()
except SystemExit as exc:
assert exc.code == 0
finally:
monkeypatch.delenv("MESHNET_TRACKER_URL", raising=False)
monkeypatch.delenv("MESHNET_MODEL", raising=False)
importlib.reload(config_mod)
assert captured["tracker_url"] == "http://env-tracker:8081"
assert captured["model"] == "Qwen2.5-0.5B-Instruct"
assert captured["model_id"] == "Qwen/Qwen2.5-0.5B-Instruct"
def test_legacy_start_without_port_uses_next_available_port(monkeypatch): def test_legacy_start_without_port_uses_next_available_port(monkeypatch):
"""Omitting --port skips an occupied default port before startup loads the model.""" """Omitting --port skips an occupied default port before startup loads the model."""
from meshnet_node.cli import main from meshnet_node.cli import main

View File

@@ -1449,3 +1449,67 @@ def test_cli_loads_local_env_before_config_defaults(tmp_path, monkeypatch):
assert config_mod.DEFAULTS["download_dir"] == "/run/media/popov/DATA/llm/safetensor/models" assert config_mod.DEFAULTS["download_dir"] == "/run/media/popov/DATA/llm/safetensor/models"
assert os.environ["HF_TOKEN"] == "hf_test_token" assert os.environ["HF_TOKEN"] == "hf_test_token"
def test_default_quantization_is_auto(monkeypatch):
import importlib
from meshnet_node import config as config_mod
from meshnet_node.model_backend import validate_quantization
monkeypatch.delenv("MESHNET_DOWNLOAD_DIR", raising=False)
importlib.reload(config_mod)
assert config_mod.DEFAULTS["quantization"] == "auto"
assert validate_quantization("auto") == "auto"
def test_auto_quantization_uses_native_model_dtype_for_unquantized_config():
from meshnet_node.model_backend import _model_load_plan
class AutoConfigStub:
@staticmethod
def from_pretrained(model_id, cache_dir=None):
assert model_id == "repo/model"
assert cache_dir is None
return types.SimpleNamespace(
text_config=types.SimpleNamespace(dtype="torch.bfloat16"),
)
torch_stub = types.SimpleNamespace(bfloat16="bf16", float16="fp16")
quant_config, dtype, uses_quantized_weights = _model_load_plan(
AutoConfigStub,
"repo/model",
"auto",
torch_stub,
)
assert quant_config is None
assert dtype == "bf16"
assert uses_quantized_weights is False
def test_auto_quantization_preserves_native_quantized_config():
from meshnet_node.model_backend import _model_load_plan
class AutoConfigStub:
@staticmethod
def from_pretrained(model_id, cache_dir=None):
return types.SimpleNamespace(
quantization_config={"quant_method": "gptq"},
torch_dtype="float16",
)
torch_stub = types.SimpleNamespace(bfloat16="bf16", float16="fp16")
quant_config, dtype, uses_quantized_weights = _model_load_plan(
AutoConfigStub,
"repo/model",
"auto",
torch_stub,
)
assert quant_config is None
assert dtype == "fp16"
assert uses_quantized_weights is True