new tasks, model pricing, auto quantisation, etc...
This commit is contained in:
BIN
billing.sqlite
BIN
billing.sqlite
Binary file not shown.
55
docs/issues/42-gguf-llamacpp-node-backend.md
Normal file
55
docs/issues/42-gguf-llamacpp-node-backend.md
Normal file
@@ -0,0 +1,55 @@
|
||||
# US-042 — GGUF/llama.cpp node backend
|
||||
|
||||
Status: planned
|
||||
Priority: High (unlocks big MoE models on volunteer hardware — the pool's core value)
|
||||
Stage: Draft design
|
||||
|
||||
## Context
|
||||
|
||||
The node backend is transformers-only (`model_backend.py` →
|
||||
`AutoModelForCausalLM`). For DeepSeek-V4-Flash (158B MoE, official weights FP8
|
||||
160 GB) the only quantizations that run on consumer hardware are GGUF
|
||||
(IQ2 87 GB → Q4_K_M-XL 175 GB) — llama.cpp format. The transformers-compatible
|
||||
quants (FP8, NVFP4, GPTQ W4A16) all need datacenter GPUs. Volunteer machines —
|
||||
including our own Strix Halo boxes (128 GB and 80 GB unified memory, GPU via
|
||||
Vulkan/ROCm, no FP8 support on RDNA3.5) — run these models today only under
|
||||
llama.cpp.
|
||||
|
||||
## Design directions to evaluate (design-it-twice)
|
||||
|
||||
**A. llama.cpp as a per-node shard executor.** Node loads a *layer range* of a
|
||||
GGUF via llama-cpp-python; our existing hop protocol (X-Meshnet-Route,
|
||||
activations over HTTP/relay) moves hidden states between nodes. Requires
|
||||
llama.cpp partial-layer loading and activation import/export — investigate
|
||||
feasibility first; this is the riskiest unknown.
|
||||
|
||||
**B. llama.cpp RPC mode under tracker orchestration.** llama.cpp ships a
|
||||
native RPC backend that splits one model across machines. The tracker would
|
||||
provision/route to an llama.cpp RPC cluster rather than our own hop pipeline.
|
||||
Less code, but bypasses our billing/telemetry hop instrumentation and relay
|
||||
NAT path — needs a story for both.
|
||||
|
||||
**C. Whole-model GGUF nodes (no sharding).** A node with enough memory serves
|
||||
a full GGUF (e.g. IQ2/IQ3 on a 128 GB box); the tracker routes whole requests
|
||||
to it (single-hop route). Smallest step, no cross-node activation work, and
|
||||
already useful: Strix Halo 128 GB serves DeepSeek-V4-Flash IQ3_XXS (114 GB)
|
||||
via llama.cpp Vulkan today.
|
||||
|
||||
Recommended sequencing: C first (small, real value), then A/B investigation.
|
||||
|
||||
## Also in scope
|
||||
|
||||
- Model catalog: allow GGUF entries with quant selection; feature
|
||||
`DeepSeek-V4-Flash` IQ4_XS/UD-Q4_K_XL as a curated/featured entry once at
|
||||
least direction C works (a featured model nobody can load is an anti-feature)
|
||||
- Hardware detection: recognize Strix Halo/unified-memory APUs and Vulkan
|
||||
(`hardware.py` currently reports "CPU mode" on these boxes)
|
||||
- `MESHNET_DOWNLOAD_DIR`/`--download-dir` applies to GGUF files as well
|
||||
|
||||
## Acceptance criteria (phase C)
|
||||
|
||||
- A node with `--gguf <repo-or-path> --quant IQ3_XXS` serves
|
||||
`/v1/chat/completions` via llama.cpp with GPU offload where available
|
||||
- Tracker treats it as a full-coverage node (single-hop routes, billing works)
|
||||
- Streamed responses work through the tracker proxy and the relay (US-036)
|
||||
- `python -m pytest` passes from repo root (llama.cpp behind an optional extra)
|
||||
38
docs/issues/43-dashboard-model-search-cards.md
Normal file
38
docs/issues/43-dashboard-model-search-cards.md
Normal file
@@ -0,0 +1,38 @@
|
||||
# US-043 — Dashboard model search and model cards
|
||||
|
||||
Status: planned
|
||||
Priority: Medium (post-deploy polish)
|
||||
Stage: Idea
|
||||
|
||||
## Context
|
||||
|
||||
The dashboard shows nodes/routes/billing but nothing model-centric. Operators
|
||||
and testers should be able to search for a model and see, per model, a card
|
||||
with what the network knows about it.
|
||||
|
||||
## Scope
|
||||
|
||||
- **Search**: query box hitting a new tracker endpoint that proxies the HF Hub
|
||||
search API (server-side, so the dashboard stays CSP-clean and unauthenticated
|
||||
browsers aren't rate-limited) merged with the tracker's own model presets and
|
||||
currently-served models.
|
||||
- **Model card** per result:
|
||||
- name, architecture, params, layer count (reuse `model_metadata_for`,
|
||||
which now handles nested `text_config` — US layer-detection fix)
|
||||
- coverage on the network: which layer ranges are served, by how many nodes,
|
||||
coverage gaps (the Coverage Map already exists on the tracker)
|
||||
- price per 1K tokens, availability (routable now? single-hop or pipeline?)
|
||||
- memory footprint per quantization where known (bf16 / GGUF sizes)
|
||||
- action: "request this model" — flags demand so node operators (or
|
||||
auto-shard assignment) know what to load next
|
||||
- Featured models section driven by the curated catalog (`CURATED_MODELS`),
|
||||
including GGUF entries once US-042 lands.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- Searching a HF repo id or free text returns results without the browser
|
||||
calling HF directly
|
||||
- A served model's card shows live coverage and a working "chat now" state
|
||||
- An unserved model's card shows the "request" action and estimated memory
|
||||
per quant
|
||||
- `python -m pytest` passes from repo root
|
||||
65
docs/issues/44-tracker-shard-source-partial-download.md
Normal file
65
docs/issues/44-tracker-shard-source-partial-download.md
Normal file
@@ -0,0 +1,65 @@
|
||||
# US-044 — Tracker as model-file source; nodes download only their shard
|
||||
|
||||
Status: planned
|
||||
Priority: High (blocks multi-machine big-model serving; pairs with US-042)
|
||||
Stage: Designed (grill remaining decisions before build)
|
||||
|
||||
## Context
|
||||
|
||||
Common deployment: the tracker and the first node share a machine that already
|
||||
holds the model files (e.g. `/run/media/popov/DATA/llm/safetensor`). When a
|
||||
second node joins with no model selected, the tracker assigns it the uncovered
|
||||
layer range — and today that node then downloads the **entire snapshot from
|
||||
HuggingFace**, even for a 20-layer shard of a 160 GB model.
|
||||
|
||||
What exists already (build on it, don't duplicate):
|
||||
|
||||
- Nodes serve their shard dir as a tar at `GET /v1/shards/download` with
|
||||
checksum verification; `download_shard` tries assignment-provided `peers`
|
||||
before HF (`downloader.py`). But it only matches **identical layer ranges**,
|
||||
and the HF fallback runs `snapshot_download` of the whole repo.
|
||||
- The torch path (`--model-id`) bypasses `download_shard` entirely:
|
||||
`TorchModelShard` → `from_pretrained` downloads **and loads into RAM** the
|
||||
full model, then executes only the assigned layers. Sharding currently saves
|
||||
compute, not memory or bandwidth.
|
||||
|
||||
## Design
|
||||
|
||||
1. **Tracker `--models-dir PATH`** (env `MESHNET_MODELS_DIR`). When set, the
|
||||
tracker indexes HF-layout snapshots under it and advertises itself as a
|
||||
model-file source in `/v1/nodes/assign` responses.
|
||||
2. **Layer-aware file selection.** For safetensors models, read
|
||||
`model.safetensors.index.json` and map the assigned layer range → the
|
||||
subset of weight files containing those layers, plus the always-needed
|
||||
files (config, tokenizer, index, embeddings/head files for head/tail
|
||||
shards). Serve exactly that subset (tar stream, per-file checksums).
|
||||
GGUF (US-042): single file or naive byte-range — phase 2.
|
||||
3. **Node download order**: exact-shard peer (existing) → tracker/peer file
|
||||
subset (new) → HF `snapshot_download` with `allow_patterns` for the same
|
||||
subset (new — stop downloading the whole repo even from HF) → full snapshot
|
||||
(last resort).
|
||||
4. **Partial LOAD (the hard half).** Downloading a subset is wasted unless the
|
||||
node stops instantiating the full model: build the model skeleton on the
|
||||
`meta` device, materialize only assigned layers (+embeddings/norm/head as
|
||||
role requires) from the local files, leave the rest on meta. Without this,
|
||||
an 80 GB machine can never hold a shard of a 160 GB model regardless of
|
||||
how the bytes arrive. This is the acceptance bar for the issue.
|
||||
|
||||
## Open questions (grill before building)
|
||||
|
||||
- Trust: joining nodes fetch weights from the tracker/peers — checksum against
|
||||
what root of trust? (HF etag/sha vs tracker-signed manifest.)
|
||||
- Disk layout: partial snapshots must not corrupt the HF cache dir; probably
|
||||
a meshnet-owned layout keyed by repo+revision.
|
||||
- Serving cost: a 100 GB tar stream per joining node on the tracker box —
|
||||
rate-limit/queue? LAN-only heuristic?
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- Two-machine test: machine A (tracker + node, holds full snapshot) serves
|
||||
layers 0–k; machine B joins with no model and receives **only** the files
|
||||
for its assigned range from A — nothing fetched from HF
|
||||
- Machine B's resident memory scales with its shard size, not model size
|
||||
- Checksums verified end-to-end; corrupted transfer falls back cleanly
|
||||
- Single-node/full-model flows unchanged
|
||||
- `python -m pytest` passes from repo root
|
||||
Binary file not shown.
@@ -56,7 +56,7 @@ def _run_node(cfg: dict) -> None:
|
||||
model_id=cfg.get("model_hf_repo") or None,
|
||||
shard_start=cfg.get("shard_start"),
|
||||
shard_end=cfg.get("shard_end"),
|
||||
quantization=cfg.get("quantization", "int8").replace("bf16", "bfloat16"),
|
||||
quantization=cfg.get("quantization", "auto").replace("bf16", "bfloat16"),
|
||||
wallet_path=Path(cfg["wallet_path"]) if cfg.get("wallet_path") else None,
|
||||
cache_dir=Path(cfg["download_dir"]) if cfg.get("download_dir") else None,
|
||||
host=cfg.get("host", "0.0.0.0"),
|
||||
@@ -199,17 +199,19 @@ def _cmd_config(args) -> int:
|
||||
|
||||
def _cmd_start(args) -> int:
|
||||
"""Legacy `start` subcommand — preserves backward compatibility with existing tests."""
|
||||
from .config import load_config, DEFAULTS
|
||||
from .config import DEFAULTS
|
||||
|
||||
# Build a transient config from flags (don't write to disk)
|
||||
cfg = dict(DEFAULTS)
|
||||
if args.tracker:
|
||||
cfg["tracker_url"] = args.tracker
|
||||
cfg["port"] = args.port if args.port is not None else _first_available_port(args.host)
|
||||
if args.model_id is None and "/" in args.model:
|
||||
cfg["model_hf_repo"] = args.model
|
||||
cfg["model_name"] = args.model.split("/")[-1]
|
||||
model = args.model or cfg.get("model_hf_repo") or cfg.get("model_name") or "stub-model"
|
||||
if args.model_id is None and "/" in model:
|
||||
cfg["model_hf_repo"] = model
|
||||
cfg["model_name"] = model.split("/")[-1]
|
||||
else:
|
||||
cfg["model_name"] = args.model
|
||||
cfg["model_name"] = model
|
||||
cfg["quantization"] = args.quantization
|
||||
cfg["host"] = args.host
|
||||
if args.model_id:
|
||||
@@ -307,13 +309,13 @@ def main() -> None:
|
||||
|
||||
# start subcommand (legacy / backward-compat)
|
||||
start_cmd = subparsers.add_parser("start", help="Start node (legacy flags)")
|
||||
start_cmd.add_argument("--tracker", default="http://localhost:8080")
|
||||
start_cmd.add_argument("--tracker")
|
||||
start_cmd.add_argument("--port", type=int)
|
||||
start_cmd.add_argument("--model", default="stub-model")
|
||||
start_cmd.add_argument("--model")
|
||||
start_cmd.add_argument("--model-id", help="HuggingFace repo ID")
|
||||
start_cmd.add_argument("--shard-start", type=int)
|
||||
start_cmd.add_argument("--shard-end", type=int)
|
||||
start_cmd.add_argument("--quantization", choices=["bfloat16", "int8", "nf4", "bf16"], default="int8")
|
||||
start_cmd.add_argument("--quantization", choices=["auto", "bfloat16", "int8", "nf4", "bf16"], default="auto")
|
||||
start_cmd.add_argument("--host", default="0.0.0.0")
|
||||
start_cmd.add_argument("--advertise-host")
|
||||
start_cmd.add_argument("--tracker-mode", action="store_true")
|
||||
|
||||
@@ -14,13 +14,16 @@ _DEFAULT_CONFIG_FILE = _DEFAULT_CONFIG_DIR / "config.json"
|
||||
_DEFAULT_DOWNLOAD_DIR = Path(
|
||||
os.environ.get("MESHNET_DOWNLOAD_DIR", str(Path.home() / ".meshnet" / "models"))
|
||||
)
|
||||
_DEFAULT_TRACKER_URL = "http://localhost:8080"
|
||||
_DEFAULT_TRACKER_URL = os.environ.get("MESHNET_TRACKER_URL", "http://localhost:8080")
|
||||
_DEFAULT_WALLET_PATH = str(Path.home() / ".config" / "meshnet" / "wallet.json")
|
||||
_DEFAULT_QUANTIZATION = "nf4"
|
||||
_DEFAULT_QUANTIZATION = "auto"
|
||||
_DEFAULT_MODEL = os.environ.get("MESHNET_MODEL_ID") or os.environ.get("MESHNET_MODEL", "")
|
||||
_DEFAULT_MODEL_HF_REPO = _DEFAULT_MODEL if "/" in _DEFAULT_MODEL else ""
|
||||
_DEFAULT_MODEL_NAME = _DEFAULT_MODEL.split("/")[-1] if "/" in _DEFAULT_MODEL else _DEFAULT_MODEL
|
||||
|
||||
DEFAULTS = {
|
||||
"model_hf_repo": "",
|
||||
"model_name": "",
|
||||
"model_hf_repo": _DEFAULT_MODEL_HF_REPO,
|
||||
"model_name": _DEFAULT_MODEL_NAME,
|
||||
"quantization": _DEFAULT_QUANTIZATION,
|
||||
"download_dir": str(_DEFAULT_DOWNLOAD_DIR),
|
||||
"tracker_url": _DEFAULT_TRACKER_URL,
|
||||
|
||||
@@ -7,7 +7,7 @@ from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any, Literal
|
||||
|
||||
Quantization = Literal["bfloat16", "int8", "nf4"]
|
||||
Quantization = Literal["auto", "bfloat16", "int8", "nf4"]
|
||||
|
||||
|
||||
class ModelBackendError(RuntimeError):
|
||||
@@ -31,14 +31,14 @@ class TensorPayload:
|
||||
|
||||
|
||||
def validate_quantization(value: str) -> Quantization:
|
||||
if value not in {"bfloat16", "int8", "nf4"}:
|
||||
raise ValueError("quantization must be one of: bfloat16, int8, nf4")
|
||||
if value not in {"auto", "bfloat16", "int8", "nf4"}:
|
||||
raise ValueError("quantization must be one of: auto, bfloat16, int8, nf4")
|
||||
return value # type: ignore[return-value]
|
||||
|
||||
|
||||
def build_quantization_config(quantization: Quantization) -> Any | None:
|
||||
"""Return a transformers BitsAndBytesConfig for quantized weights."""
|
||||
if quantization == "bfloat16":
|
||||
if quantization in {"auto", "bfloat16"}:
|
||||
return None
|
||||
try:
|
||||
import torch
|
||||
@@ -65,7 +65,7 @@ class TorchModelShard:
|
||||
model_id: str,
|
||||
shard_start: int,
|
||||
shard_end: int,
|
||||
quantization: Quantization = "bfloat16",
|
||||
quantization: Quantization = "auto",
|
||||
cache_dir: Path | None = None,
|
||||
) -> None:
|
||||
if shard_start < 0 or shard_end < 0 or shard_start > shard_end:
|
||||
@@ -77,7 +77,7 @@ class TorchModelShard:
|
||||
|
||||
try:
|
||||
import torch
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer
|
||||
except ModuleNotFoundError as exc:
|
||||
raise MissingModelDependencyError(
|
||||
"real model backend requires torch, transformers, safetensors, accelerate, and bitsandbytes"
|
||||
@@ -85,17 +85,27 @@ class TorchModelShard:
|
||||
|
||||
self.torch = torch
|
||||
self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
|
||||
quant_config = build_quantization_config(quantization)
|
||||
quant_config, dtype, uses_quantized_weights = _model_load_plan(
|
||||
AutoConfig,
|
||||
model_id,
|
||||
quantization,
|
||||
torch,
|
||||
cache_dir,
|
||||
)
|
||||
try:
|
||||
load_kwargs = {
|
||||
"device_map": "auto" if uses_quantized_weights else None,
|
||||
"dtype": dtype,
|
||||
"low_cpu_mem_usage": True,
|
||||
"cache_dir": str(cache_dir) if cache_dir is not None else None,
|
||||
}
|
||||
if quant_config is not None:
|
||||
load_kwargs["quantization_config"] = quant_config
|
||||
self.model = AutoModelForCausalLM.from_pretrained(
|
||||
model_id,
|
||||
quantization_config=quant_config,
|
||||
device_map="auto" if quant_config is not None else None,
|
||||
dtype=torch.bfloat16,
|
||||
low_cpu_mem_usage=True,
|
||||
cache_dir=str(cache_dir) if cache_dir is not None else None,
|
||||
**load_kwargs,
|
||||
)
|
||||
if quant_config is None:
|
||||
if not uses_quantized_weights:
|
||||
self.model.to(self.device)
|
||||
except Exception as exc:
|
||||
if _looks_like_oom(exc):
|
||||
@@ -340,12 +350,71 @@ def load_torch_shard(
|
||||
model_id: str,
|
||||
shard_start: int,
|
||||
shard_end: int,
|
||||
quantization: Quantization = "bfloat16",
|
||||
quantization: Quantization = "auto",
|
||||
cache_dir: Path | None = None,
|
||||
) -> TorchModelShard:
|
||||
return TorchModelShard(model_id, shard_start, shard_end, quantization, cache_dir)
|
||||
|
||||
|
||||
def _model_load_plan(
|
||||
auto_config: Any,
|
||||
model_id: str,
|
||||
quantization: Quantization,
|
||||
torch: Any,
|
||||
cache_dir: Path | None = None,
|
||||
) -> tuple[Any | None, Any, bool]:
|
||||
"""Return (explicit quant config, dtype, uses quantized weights)."""
|
||||
if quantization != "auto":
|
||||
quant_config = build_quantization_config(quantization)
|
||||
return quant_config, torch.bfloat16, quant_config is not None
|
||||
|
||||
cfg = auto_config.from_pretrained(
|
||||
model_id,
|
||||
cache_dir=str(cache_dir) if cache_dir is not None else None,
|
||||
)
|
||||
if _native_quantization_config(cfg) is not None:
|
||||
return None, _native_torch_dtype(cfg, torch), True
|
||||
return None, _native_torch_dtype(cfg, torch), False
|
||||
|
||||
|
||||
def _config_candidates(cfg: Any) -> list[Any]:
|
||||
candidates = [cfg]
|
||||
get_text_config = getattr(cfg, "get_text_config", None)
|
||||
if callable(get_text_config):
|
||||
try:
|
||||
candidates.append(get_text_config())
|
||||
except Exception:
|
||||
pass
|
||||
text_config = getattr(cfg, "text_config", None)
|
||||
if text_config is not None:
|
||||
candidates.append(text_config)
|
||||
return candidates
|
||||
|
||||
|
||||
def _native_quantization_config(cfg: Any) -> Any | None:
|
||||
for candidate in _config_candidates(cfg):
|
||||
quant_config = getattr(candidate, "quantization_config", None)
|
||||
if quant_config:
|
||||
return quant_config
|
||||
return None
|
||||
|
||||
|
||||
def _native_torch_dtype(cfg: Any, torch: Any) -> Any:
|
||||
for candidate in _config_candidates(cfg):
|
||||
for attr in ("dtype", "torch_dtype"):
|
||||
dtype = getattr(candidate, attr, None)
|
||||
if dtype is None:
|
||||
continue
|
||||
if isinstance(dtype, str):
|
||||
dtype_name = dtype.removeprefix("torch.")
|
||||
dtype_value = getattr(torch, dtype_name, None)
|
||||
if dtype_value is not None:
|
||||
return dtype_value
|
||||
else:
|
||||
return dtype
|
||||
return torch.bfloat16
|
||||
|
||||
|
||||
def _model_layers(model: Any) -> Any:
|
||||
if hasattr(model, "model") and hasattr(model.model, "layers"):
|
||||
return model.model.layers
|
||||
|
||||
@@ -52,6 +52,7 @@ def _max_assignable_layers(memory_mb: int, total_layers: int | None) -> int:
|
||||
def _shard_budget_line(memory_mb: int, memory_source: str, total_layers: int | None, quantization: str) -> str:
|
||||
memory_gb = memory_mb / 1024
|
||||
gb_str = f"{memory_gb:.1f} GB"
|
||||
budget_quantization = "bfloat16" if quantization == "auto" else quantization
|
||||
if total_layers is None or total_layers <= 0:
|
||||
return f"Memory budget: {gb_str} {memory_source}; shard budget: unknown model layer count"
|
||||
max_layers = _max_assignable_layers(memory_mb, total_layers)
|
||||
@@ -61,7 +62,7 @@ def _shard_budget_line(memory_mb: int, memory_source: str, total_layers: int | N
|
||||
remaining_str = f"; {remaining_gb:.1f} GB remaining after full load" if remaining_gb > 1 else ""
|
||||
return (
|
||||
f"Memory budget: {gb_str} {memory_source}; "
|
||||
f"Shard budget: up to {max_layers}/{total_layers} layers at {quantization}"
|
||||
f"Shard budget: up to {max_layers}/{total_layers} layers at {budget_quantization}"
|
||||
f"{remaining_str}"
|
||||
)
|
||||
|
||||
@@ -306,7 +307,7 @@ def run_startup(
|
||||
model_id: str | None = None,
|
||||
shard_start: int | None = None,
|
||||
shard_end: int | None = None,
|
||||
quantization: str = "bfloat16",
|
||||
quantization: str = "auto",
|
||||
wallet_path: Path | None = None,
|
||||
cache_dir: Path | None = None,
|
||||
host: str = "127.0.0.1",
|
||||
|
||||
@@ -12,7 +12,7 @@
|
||||
"recommended": true,
|
||||
"deployment_status": "recommended",
|
||||
"hf_aliases": [],
|
||||
"hf_verified_match_note": "Pending human curation (issue 23) — no HF inference-marketplace listing has been confirmed as a comparable params/quantization match for this preset yet. Leave empty until a human signs off; an empty hf_aliases list keeps this model on the static default price.",
|
||||
"hf_verified_match_note": "Pending human curation (issue 23) \u2014 no HF inference-marketplace listing has been confirmed as a comparable params/quantization match for this preset yet. Leave empty until a human signs off; an empty hf_aliases list keeps this model on the static default price.",
|
||||
"required_model_bytes": 638876385280,
|
||||
"download_size_bytes": 638876385280,
|
||||
"native_quantization": "int4",
|
||||
@@ -38,6 +38,41 @@
|
||||
"KTransformers"
|
||||
]
|
||||
}
|
||||
},
|
||||
"qwen3.6-35b-a3b": {
|
||||
"layers_start": 0,
|
||||
"layers_end": 39,
|
||||
"hf_repo": "unsloth/Qwen3.6-35B-A3B",
|
||||
"aliases": [
|
||||
"qwen3.6-35b-a3b",
|
||||
"Qwen3.6-35B-A3B",
|
||||
"unsloth/Qwen3.6-35B-A3B",
|
||||
"Qwen/Qwen3.6-35B-A3B"
|
||||
],
|
||||
"recommended": true,
|
||||
"deployment_status": "recommended",
|
||||
"price_per_1k_tokens": 0.00044,
|
||||
"hf_aliases": [
|
||||
"qwen/qwen3.6-35b-a3b"
|
||||
],
|
||||
"hf_verified_match_note": "Verified 2026-07-06: unsloth/Qwen3.6-35B-A3B is a bf16 mirror of Qwen/Qwen3.6-35B-A3B; deepinfra and featherless-ai serve the official weights on the HF inference marketplace, so their rates are a fair comparable. Static price 0.00044 = 80% of deepinfra's blended $0.55/1M ($0.15 in / $0.95 out); the nightly refresher keeps it tracking.",
|
||||
"required_model_bytes": 71903776776,
|
||||
"download_size_bytes": 71903776776,
|
||||
"native_quantization": "bfloat16",
|
||||
"canonical_audit_dtype": "bfloat16",
|
||||
"canonical_audit_quantization": "bfloat16",
|
||||
"bytes_per_layer": {
|
||||
"bfloat16": 1797594419
|
||||
},
|
||||
"metadata": {
|
||||
"architecture": "Mixture-of-Experts (MoE, hybrid linear attention)",
|
||||
"total_parameters": "35B",
|
||||
"activated_parameters": "3B",
|
||||
"num_layers": 40,
|
||||
"context_length": 262144,
|
||||
"native_quantization": "bfloat16",
|
||||
"download_size_gb": 72
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -46,6 +46,24 @@ from .gossip import NodeGossip
|
||||
from .raft import RaftNode
|
||||
|
||||
|
||||
def _preset_price_keys(name: str, preset: dict) -> set[str]:
|
||||
"""All model strings a client may bill under for one preset.
|
||||
|
||||
``BillingLedger.price_for`` is keyed by the raw ``model`` string in the
|
||||
request, so the preset price must be registered under the preset name,
|
||||
its ``hf_repo``, and every alias — otherwise ``unsloth/Qwen…`` style
|
||||
requests silently fall back to the default rate.
|
||||
"""
|
||||
keys = {name}
|
||||
hf_repo = preset.get("hf_repo")
|
||||
if isinstance(hf_repo, str) and hf_repo:
|
||||
keys.add(hf_repo)
|
||||
for alias in preset.get("aliases") or []:
|
||||
if isinstance(alias, str) and alias:
|
||||
keys.add(alias)
|
||||
return keys
|
||||
|
||||
|
||||
def derive_relay_url_from_public_tracker_url(url: str | None) -> str | None:
|
||||
"""Return wss://host/ws when url is a public HTTPS tracker origin."""
|
||||
if not url:
|
||||
@@ -4065,9 +4083,10 @@ class TrackerServer:
|
||||
db_path = DEFAULT_BILLING_DB_PATH
|
||||
if db_path:
|
||||
preset_prices = {
|
||||
name: float(preset["price_per_1k_tokens"])
|
||||
key: float(preset["price_per_1k_tokens"])
|
||||
for name, preset in self._model_presets.items()
|
||||
if isinstance(preset, dict) and "price_per_1k_tokens" in preset
|
||||
for key in _preset_price_keys(name, preset)
|
||||
}
|
||||
billing = BillingLedger(db_path=db_path, prices=preset_prices)
|
||||
self._billing: BillingLedger | None = billing
|
||||
@@ -4332,7 +4351,8 @@ class TrackerServer:
|
||||
continue
|
||||
if result is None:
|
||||
continue
|
||||
billing.set_price(name, result["new_price_per_1k"])
|
||||
for key in _preset_price_keys(name, preset):
|
||||
billing.set_price(key, result["new_price_per_1k"])
|
||||
preset["hf_last_price_per_1k"] = result["new_price_per_1k"]
|
||||
preset["hf_last_updated"] = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
|
||||
if self._hf_pricing_log is not None:
|
||||
|
||||
@@ -148,3 +148,41 @@ def test_hf_pricing_log_persists_and_is_queryable(tmp_path):
|
||||
# Reopening against the same db path recovers the log (billing.py pattern).
|
||||
reopened = HfPricingLog(db_path=db_path)
|
||||
assert len(reopened.history()) == 1
|
||||
|
||||
|
||||
def test_preset_price_keys_cover_name_repo_and_aliases():
|
||||
from meshnet_tracker.server import _preset_price_keys
|
||||
|
||||
preset = {
|
||||
"hf_repo": "unsloth/Qwen3.6-35B-A3B",
|
||||
"aliases": ["qwen3.6-35b-a3b", "Qwen/Qwen3.6-35B-A3B"],
|
||||
}
|
||||
keys = _preset_price_keys("qwen3.6-35b-a3b", preset)
|
||||
assert keys == {
|
||||
"qwen3.6-35b-a3b",
|
||||
"unsloth/Qwen3.6-35B-A3B",
|
||||
"Qwen/Qwen3.6-35B-A3B",
|
||||
}
|
||||
assert _preset_price_keys("bare", {}) == {"bare"}
|
||||
|
||||
|
||||
def test_qwen_preset_prices_apply_to_all_aliases(tmp_path):
|
||||
"""Requests naming the repo id (what nodes register) bill at the preset price."""
|
||||
from meshnet_tracker.server import TrackerServer
|
||||
|
||||
import pytest
|
||||
|
||||
tracker = TrackerServer(billing_db=str(tmp_path / "billing.sqlite"))
|
||||
try:
|
||||
billing = tracker._billing
|
||||
assert billing is not None
|
||||
for key in (
|
||||
"qwen3.6-35b-a3b",
|
||||
"unsloth/Qwen3.6-35B-A3B",
|
||||
"Qwen/Qwen3.6-35B-A3B",
|
||||
):
|
||||
assert billing.price_for(key) == pytest.approx(0.00044), key
|
||||
# Unknown models keep the default rate
|
||||
assert billing.price_for("some/other-model") == pytest.approx(0.02)
|
||||
finally:
|
||||
pass
|
||||
|
||||
@@ -388,6 +388,48 @@ def test_legacy_start_treats_repo_model_as_model_id(monkeypatch):
|
||||
assert captured["model_id"] == "Qwen/Qwen2.5-0.5B-Instruct"
|
||||
|
||||
|
||||
def test_legacy_start_falls_back_to_env_tracker_and_model(monkeypatch):
|
||||
"""`meshnet-node start` uses env defaults when tracker/model flags are omitted."""
|
||||
import importlib
|
||||
|
||||
from meshnet_node import config as config_mod
|
||||
from meshnet_node.cli import main
|
||||
|
||||
monkeypatch.setenv("MESHNET_TRACKER_URL", "http://env-tracker:8081")
|
||||
monkeypatch.setenv("MESHNET_MODEL", "Qwen/Qwen2.5-0.5B-Instruct")
|
||||
importlib.reload(config_mod)
|
||||
|
||||
captured = {}
|
||||
|
||||
def fake_run_startup(*args, **kwargs):
|
||||
captured.update(kwargs)
|
||||
class _FakeNode:
|
||||
chat_completion_count = 0
|
||||
def stop(self): pass
|
||||
return _FakeNode()
|
||||
|
||||
monkeypatch.setattr(sys, "argv", [
|
||||
"meshnet-node", "start",
|
||||
"--port", "0",
|
||||
])
|
||||
|
||||
try:
|
||||
with patch("meshnet_node.startup.run_startup", side_effect=fake_run_startup):
|
||||
with patch("time.sleep", side_effect=KeyboardInterrupt):
|
||||
try:
|
||||
main()
|
||||
except SystemExit as exc:
|
||||
assert exc.code == 0
|
||||
finally:
|
||||
monkeypatch.delenv("MESHNET_TRACKER_URL", raising=False)
|
||||
monkeypatch.delenv("MESHNET_MODEL", raising=False)
|
||||
importlib.reload(config_mod)
|
||||
|
||||
assert captured["tracker_url"] == "http://env-tracker:8081"
|
||||
assert captured["model"] == "Qwen2.5-0.5B-Instruct"
|
||||
assert captured["model_id"] == "Qwen/Qwen2.5-0.5B-Instruct"
|
||||
|
||||
|
||||
def test_legacy_start_without_port_uses_next_available_port(monkeypatch):
|
||||
"""Omitting --port skips an occupied default port before startup loads the model."""
|
||||
from meshnet_node.cli import main
|
||||
|
||||
@@ -1449,3 +1449,67 @@ def test_cli_loads_local_env_before_config_defaults(tmp_path, monkeypatch):
|
||||
|
||||
assert config_mod.DEFAULTS["download_dir"] == "/run/media/popov/DATA/llm/safetensor/models"
|
||||
assert os.environ["HF_TOKEN"] == "hf_test_token"
|
||||
|
||||
|
||||
def test_default_quantization_is_auto(monkeypatch):
|
||||
import importlib
|
||||
|
||||
from meshnet_node import config as config_mod
|
||||
from meshnet_node.model_backend import validate_quantization
|
||||
|
||||
monkeypatch.delenv("MESHNET_DOWNLOAD_DIR", raising=False)
|
||||
importlib.reload(config_mod)
|
||||
|
||||
assert config_mod.DEFAULTS["quantization"] == "auto"
|
||||
assert validate_quantization("auto") == "auto"
|
||||
|
||||
|
||||
def test_auto_quantization_uses_native_model_dtype_for_unquantized_config():
|
||||
from meshnet_node.model_backend import _model_load_plan
|
||||
|
||||
class AutoConfigStub:
|
||||
@staticmethod
|
||||
def from_pretrained(model_id, cache_dir=None):
|
||||
assert model_id == "repo/model"
|
||||
assert cache_dir is None
|
||||
return types.SimpleNamespace(
|
||||
text_config=types.SimpleNamespace(dtype="torch.bfloat16"),
|
||||
)
|
||||
|
||||
torch_stub = types.SimpleNamespace(bfloat16="bf16", float16="fp16")
|
||||
|
||||
quant_config, dtype, uses_quantized_weights = _model_load_plan(
|
||||
AutoConfigStub,
|
||||
"repo/model",
|
||||
"auto",
|
||||
torch_stub,
|
||||
)
|
||||
|
||||
assert quant_config is None
|
||||
assert dtype == "bf16"
|
||||
assert uses_quantized_weights is False
|
||||
|
||||
|
||||
def test_auto_quantization_preserves_native_quantized_config():
|
||||
from meshnet_node.model_backend import _model_load_plan
|
||||
|
||||
class AutoConfigStub:
|
||||
@staticmethod
|
||||
def from_pretrained(model_id, cache_dir=None):
|
||||
return types.SimpleNamespace(
|
||||
quantization_config={"quant_method": "gptq"},
|
||||
torch_dtype="float16",
|
||||
)
|
||||
|
||||
torch_stub = types.SimpleNamespace(bfloat16="bf16", float16="fp16")
|
||||
|
||||
quant_config, dtype, uses_quantized_weights = _model_load_plan(
|
||||
AutoConfigStub,
|
||||
"repo/model",
|
||||
"auto",
|
||||
torch_stub,
|
||||
)
|
||||
|
||||
assert quant_config is None
|
||||
assert dtype == "fp16"
|
||||
assert uses_quantized_weights is True
|
||||
|
||||
Reference in New Issue
Block a user