new tasks, model pricing, auto quantisation, etc...
This commit is contained in:
BIN
billing.sqlite
BIN
billing.sqlite
Binary file not shown.
55
docs/issues/42-gguf-llamacpp-node-backend.md
Normal file
55
docs/issues/42-gguf-llamacpp-node-backend.md
Normal file
@@ -0,0 +1,55 @@
|
|||||||
|
# US-042 — GGUF/llama.cpp node backend
|
||||||
|
|
||||||
|
Status: planned
|
||||||
|
Priority: High (unlocks big MoE models on volunteer hardware — the pool's core value)
|
||||||
|
Stage: Draft design
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
The node backend is transformers-only (`model_backend.py` →
|
||||||
|
`AutoModelForCausalLM`). For DeepSeek-V4-Flash (158B MoE, official weights FP8
|
||||||
|
160 GB) the only quantizations that run on consumer hardware are GGUF
|
||||||
|
(IQ2 87 GB → Q4_K_M-XL 175 GB) — llama.cpp format. The transformers-compatible
|
||||||
|
quants (FP8, NVFP4, GPTQ W4A16) all need datacenter GPUs. Volunteer machines —
|
||||||
|
including our own Strix Halo boxes (128 GB and 80 GB unified memory, GPU via
|
||||||
|
Vulkan/ROCm, no FP8 support on RDNA3.5) — run these models today only under
|
||||||
|
llama.cpp.
|
||||||
|
|
||||||
|
## Design directions to evaluate (design-it-twice)
|
||||||
|
|
||||||
|
**A. llama.cpp as a per-node shard executor.** Node loads a *layer range* of a
|
||||||
|
GGUF via llama-cpp-python; our existing hop protocol (X-Meshnet-Route,
|
||||||
|
activations over HTTP/relay) moves hidden states between nodes. Requires
|
||||||
|
llama.cpp partial-layer loading and activation import/export — investigate
|
||||||
|
feasibility first; this is the riskiest unknown.
|
||||||
|
|
||||||
|
**B. llama.cpp RPC mode under tracker orchestration.** llama.cpp ships a
|
||||||
|
native RPC backend that splits one model across machines. The tracker would
|
||||||
|
provision/route to an llama.cpp RPC cluster rather than our own hop pipeline.
|
||||||
|
Less code, but bypasses our billing/telemetry hop instrumentation and relay
|
||||||
|
NAT path — needs a story for both.
|
||||||
|
|
||||||
|
**C. Whole-model GGUF nodes (no sharding).** A node with enough memory serves
|
||||||
|
a full GGUF (e.g. IQ2/IQ3 on a 128 GB box); the tracker routes whole requests
|
||||||
|
to it (single-hop route). Smallest step, no cross-node activation work, and
|
||||||
|
already useful: Strix Halo 128 GB serves DeepSeek-V4-Flash IQ3_XXS (114 GB)
|
||||||
|
via llama.cpp Vulkan today.
|
||||||
|
|
||||||
|
Recommended sequencing: C first (small, real value), then A/B investigation.
|
||||||
|
|
||||||
|
## Also in scope
|
||||||
|
|
||||||
|
- Model catalog: allow GGUF entries with quant selection; feature
|
||||||
|
`DeepSeek-V4-Flash` IQ4_XS/UD-Q4_K_XL as a curated/featured entry once at
|
||||||
|
least direction C works (a featured model nobody can load is an anti-feature)
|
||||||
|
- Hardware detection: recognize Strix Halo/unified-memory APUs and Vulkan
|
||||||
|
(`hardware.py` currently reports "CPU mode" on these boxes)
|
||||||
|
- `MESHNET_DOWNLOAD_DIR`/`--download-dir` applies to GGUF files as well
|
||||||
|
|
||||||
|
## Acceptance criteria (phase C)
|
||||||
|
|
||||||
|
- A node with `--gguf <repo-or-path> --quant IQ3_XXS` serves
|
||||||
|
`/v1/chat/completions` via llama.cpp with GPU offload where available
|
||||||
|
- Tracker treats it as a full-coverage node (single-hop routes, billing works)
|
||||||
|
- Streamed responses work through the tracker proxy and the relay (US-036)
|
||||||
|
- `python -m pytest` passes from repo root (llama.cpp behind an optional extra)
|
||||||
38
docs/issues/43-dashboard-model-search-cards.md
Normal file
38
docs/issues/43-dashboard-model-search-cards.md
Normal file
@@ -0,0 +1,38 @@
|
|||||||
|
# US-043 — Dashboard model search and model cards
|
||||||
|
|
||||||
|
Status: planned
|
||||||
|
Priority: Medium (post-deploy polish)
|
||||||
|
Stage: Idea
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
The dashboard shows nodes/routes/billing but nothing model-centric. Operators
|
||||||
|
and testers should be able to search for a model and see, per model, a card
|
||||||
|
with what the network knows about it.
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
- **Search**: query box hitting a new tracker endpoint that proxies the HF Hub
|
||||||
|
search API (server-side, so the dashboard stays CSP-clean and unauthenticated
|
||||||
|
browsers aren't rate-limited) merged with the tracker's own model presets and
|
||||||
|
currently-served models.
|
||||||
|
- **Model card** per result:
|
||||||
|
- name, architecture, params, layer count (reuse `model_metadata_for`,
|
||||||
|
which now handles nested `text_config` — US layer-detection fix)
|
||||||
|
- coverage on the network: which layer ranges are served, by how many nodes,
|
||||||
|
coverage gaps (the Coverage Map already exists on the tracker)
|
||||||
|
- price per 1K tokens, availability (routable now? single-hop or pipeline?)
|
||||||
|
- memory footprint per quantization where known (bf16 / GGUF sizes)
|
||||||
|
- action: "request this model" — flags demand so node operators (or
|
||||||
|
auto-shard assignment) know what to load next
|
||||||
|
- Featured models section driven by the curated catalog (`CURATED_MODELS`),
|
||||||
|
including GGUF entries once US-042 lands.
|
||||||
|
|
||||||
|
## Acceptance criteria
|
||||||
|
|
||||||
|
- Searching a HF repo id or free text returns results without the browser
|
||||||
|
calling HF directly
|
||||||
|
- A served model's card shows live coverage and a working "chat now" state
|
||||||
|
- An unserved model's card shows the "request" action and estimated memory
|
||||||
|
per quant
|
||||||
|
- `python -m pytest` passes from repo root
|
||||||
65
docs/issues/44-tracker-shard-source-partial-download.md
Normal file
65
docs/issues/44-tracker-shard-source-partial-download.md
Normal file
@@ -0,0 +1,65 @@
|
|||||||
|
# US-044 — Tracker as model-file source; nodes download only their shard
|
||||||
|
|
||||||
|
Status: planned
|
||||||
|
Priority: High (blocks multi-machine big-model serving; pairs with US-042)
|
||||||
|
Stage: Designed (grill remaining decisions before build)
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
Common deployment: the tracker and the first node share a machine that already
|
||||||
|
holds the model files (e.g. `/run/media/popov/DATA/llm/safetensor`). When a
|
||||||
|
second node joins with no model selected, the tracker assigns it the uncovered
|
||||||
|
layer range — and today that node then downloads the **entire snapshot from
|
||||||
|
HuggingFace**, even for a 20-layer shard of a 160 GB model.
|
||||||
|
|
||||||
|
What exists already (build on it, don't duplicate):
|
||||||
|
|
||||||
|
- Nodes serve their shard dir as a tar at `GET /v1/shards/download` with
|
||||||
|
checksum verification; `download_shard` tries assignment-provided `peers`
|
||||||
|
before HF (`downloader.py`). But it only matches **identical layer ranges**,
|
||||||
|
and the HF fallback runs `snapshot_download` of the whole repo.
|
||||||
|
- The torch path (`--model-id`) bypasses `download_shard` entirely:
|
||||||
|
`TorchModelShard` → `from_pretrained` downloads **and loads into RAM** the
|
||||||
|
full model, then executes only the assigned layers. Sharding currently saves
|
||||||
|
compute, not memory or bandwidth.
|
||||||
|
|
||||||
|
## Design
|
||||||
|
|
||||||
|
1. **Tracker `--models-dir PATH`** (env `MESHNET_MODELS_DIR`). When set, the
|
||||||
|
tracker indexes HF-layout snapshots under it and advertises itself as a
|
||||||
|
model-file source in `/v1/nodes/assign` responses.
|
||||||
|
2. **Layer-aware file selection.** For safetensors models, read
|
||||||
|
`model.safetensors.index.json` and map the assigned layer range → the
|
||||||
|
subset of weight files containing those layers, plus the always-needed
|
||||||
|
files (config, tokenizer, index, embeddings/head files for head/tail
|
||||||
|
shards). Serve exactly that subset (tar stream, per-file checksums).
|
||||||
|
GGUF (US-042): single file or naive byte-range — phase 2.
|
||||||
|
3. **Node download order**: exact-shard peer (existing) → tracker/peer file
|
||||||
|
subset (new) → HF `snapshot_download` with `allow_patterns` for the same
|
||||||
|
subset (new — stop downloading the whole repo even from HF) → full snapshot
|
||||||
|
(last resort).
|
||||||
|
4. **Partial LOAD (the hard half).** Downloading a subset is wasted unless the
|
||||||
|
node stops instantiating the full model: build the model skeleton on the
|
||||||
|
`meta` device, materialize only assigned layers (+embeddings/norm/head as
|
||||||
|
role requires) from the local files, leave the rest on meta. Without this,
|
||||||
|
an 80 GB machine can never hold a shard of a 160 GB model regardless of
|
||||||
|
how the bytes arrive. This is the acceptance bar for the issue.
|
||||||
|
|
||||||
|
## Open questions (grill before building)
|
||||||
|
|
||||||
|
- Trust: joining nodes fetch weights from the tracker/peers — checksum against
|
||||||
|
what root of trust? (HF etag/sha vs tracker-signed manifest.)
|
||||||
|
- Disk layout: partial snapshots must not corrupt the HF cache dir; probably
|
||||||
|
a meshnet-owned layout keyed by repo+revision.
|
||||||
|
- Serving cost: a 100 GB tar stream per joining node on the tracker box —
|
||||||
|
rate-limit/queue? LAN-only heuristic?
|
||||||
|
|
||||||
|
## Acceptance criteria
|
||||||
|
|
||||||
|
- Two-machine test: machine A (tracker + node, holds full snapshot) serves
|
||||||
|
layers 0–k; machine B joins with no model and receives **only** the files
|
||||||
|
for its assigned range from A — nothing fetched from HF
|
||||||
|
- Machine B's resident memory scales with its shard size, not model size
|
||||||
|
- Checksums verified end-to-end; corrupted transfer falls back cleanly
|
||||||
|
- Single-node/full-model flows unchanged
|
||||||
|
- `python -m pytest` passes from repo root
|
||||||
Binary file not shown.
@@ -56,7 +56,7 @@ def _run_node(cfg: dict) -> None:
|
|||||||
model_id=cfg.get("model_hf_repo") or None,
|
model_id=cfg.get("model_hf_repo") or None,
|
||||||
shard_start=cfg.get("shard_start"),
|
shard_start=cfg.get("shard_start"),
|
||||||
shard_end=cfg.get("shard_end"),
|
shard_end=cfg.get("shard_end"),
|
||||||
quantization=cfg.get("quantization", "int8").replace("bf16", "bfloat16"),
|
quantization=cfg.get("quantization", "auto").replace("bf16", "bfloat16"),
|
||||||
wallet_path=Path(cfg["wallet_path"]) if cfg.get("wallet_path") else None,
|
wallet_path=Path(cfg["wallet_path"]) if cfg.get("wallet_path") else None,
|
||||||
cache_dir=Path(cfg["download_dir"]) if cfg.get("download_dir") else None,
|
cache_dir=Path(cfg["download_dir"]) if cfg.get("download_dir") else None,
|
||||||
host=cfg.get("host", "0.0.0.0"),
|
host=cfg.get("host", "0.0.0.0"),
|
||||||
@@ -199,17 +199,19 @@ def _cmd_config(args) -> int:
|
|||||||
|
|
||||||
def _cmd_start(args) -> int:
|
def _cmd_start(args) -> int:
|
||||||
"""Legacy `start` subcommand — preserves backward compatibility with existing tests."""
|
"""Legacy `start` subcommand — preserves backward compatibility with existing tests."""
|
||||||
from .config import load_config, DEFAULTS
|
from .config import DEFAULTS
|
||||||
|
|
||||||
# Build a transient config from flags (don't write to disk)
|
# Build a transient config from flags (don't write to disk)
|
||||||
cfg = dict(DEFAULTS)
|
cfg = dict(DEFAULTS)
|
||||||
cfg["tracker_url"] = args.tracker
|
if args.tracker:
|
||||||
|
cfg["tracker_url"] = args.tracker
|
||||||
cfg["port"] = args.port if args.port is not None else _first_available_port(args.host)
|
cfg["port"] = args.port if args.port is not None else _first_available_port(args.host)
|
||||||
if args.model_id is None and "/" in args.model:
|
model = args.model or cfg.get("model_hf_repo") or cfg.get("model_name") or "stub-model"
|
||||||
cfg["model_hf_repo"] = args.model
|
if args.model_id is None and "/" in model:
|
||||||
cfg["model_name"] = args.model.split("/")[-1]
|
cfg["model_hf_repo"] = model
|
||||||
|
cfg["model_name"] = model.split("/")[-1]
|
||||||
else:
|
else:
|
||||||
cfg["model_name"] = args.model
|
cfg["model_name"] = model
|
||||||
cfg["quantization"] = args.quantization
|
cfg["quantization"] = args.quantization
|
||||||
cfg["host"] = args.host
|
cfg["host"] = args.host
|
||||||
if args.model_id:
|
if args.model_id:
|
||||||
@@ -307,13 +309,13 @@ def main() -> None:
|
|||||||
|
|
||||||
# start subcommand (legacy / backward-compat)
|
# start subcommand (legacy / backward-compat)
|
||||||
start_cmd = subparsers.add_parser("start", help="Start node (legacy flags)")
|
start_cmd = subparsers.add_parser("start", help="Start node (legacy flags)")
|
||||||
start_cmd.add_argument("--tracker", default="http://localhost:8080")
|
start_cmd.add_argument("--tracker")
|
||||||
start_cmd.add_argument("--port", type=int)
|
start_cmd.add_argument("--port", type=int)
|
||||||
start_cmd.add_argument("--model", default="stub-model")
|
start_cmd.add_argument("--model")
|
||||||
start_cmd.add_argument("--model-id", help="HuggingFace repo ID")
|
start_cmd.add_argument("--model-id", help="HuggingFace repo ID")
|
||||||
start_cmd.add_argument("--shard-start", type=int)
|
start_cmd.add_argument("--shard-start", type=int)
|
||||||
start_cmd.add_argument("--shard-end", type=int)
|
start_cmd.add_argument("--shard-end", type=int)
|
||||||
start_cmd.add_argument("--quantization", choices=["bfloat16", "int8", "nf4", "bf16"], default="int8")
|
start_cmd.add_argument("--quantization", choices=["auto", "bfloat16", "int8", "nf4", "bf16"], default="auto")
|
||||||
start_cmd.add_argument("--host", default="0.0.0.0")
|
start_cmd.add_argument("--host", default="0.0.0.0")
|
||||||
start_cmd.add_argument("--advertise-host")
|
start_cmd.add_argument("--advertise-host")
|
||||||
start_cmd.add_argument("--tracker-mode", action="store_true")
|
start_cmd.add_argument("--tracker-mode", action="store_true")
|
||||||
|
|||||||
@@ -14,13 +14,16 @@ _DEFAULT_CONFIG_FILE = _DEFAULT_CONFIG_DIR / "config.json"
|
|||||||
_DEFAULT_DOWNLOAD_DIR = Path(
|
_DEFAULT_DOWNLOAD_DIR = Path(
|
||||||
os.environ.get("MESHNET_DOWNLOAD_DIR", str(Path.home() / ".meshnet" / "models"))
|
os.environ.get("MESHNET_DOWNLOAD_DIR", str(Path.home() / ".meshnet" / "models"))
|
||||||
)
|
)
|
||||||
_DEFAULT_TRACKER_URL = "http://localhost:8080"
|
_DEFAULT_TRACKER_URL = os.environ.get("MESHNET_TRACKER_URL", "http://localhost:8080")
|
||||||
_DEFAULT_WALLET_PATH = str(Path.home() / ".config" / "meshnet" / "wallet.json")
|
_DEFAULT_WALLET_PATH = str(Path.home() / ".config" / "meshnet" / "wallet.json")
|
||||||
_DEFAULT_QUANTIZATION = "nf4"
|
_DEFAULT_QUANTIZATION = "auto"
|
||||||
|
_DEFAULT_MODEL = os.environ.get("MESHNET_MODEL_ID") or os.environ.get("MESHNET_MODEL", "")
|
||||||
|
_DEFAULT_MODEL_HF_REPO = _DEFAULT_MODEL if "/" in _DEFAULT_MODEL else ""
|
||||||
|
_DEFAULT_MODEL_NAME = _DEFAULT_MODEL.split("/")[-1] if "/" in _DEFAULT_MODEL else _DEFAULT_MODEL
|
||||||
|
|
||||||
DEFAULTS = {
|
DEFAULTS = {
|
||||||
"model_hf_repo": "",
|
"model_hf_repo": _DEFAULT_MODEL_HF_REPO,
|
||||||
"model_name": "",
|
"model_name": _DEFAULT_MODEL_NAME,
|
||||||
"quantization": _DEFAULT_QUANTIZATION,
|
"quantization": _DEFAULT_QUANTIZATION,
|
||||||
"download_dir": str(_DEFAULT_DOWNLOAD_DIR),
|
"download_dir": str(_DEFAULT_DOWNLOAD_DIR),
|
||||||
"tracker_url": _DEFAULT_TRACKER_URL,
|
"tracker_url": _DEFAULT_TRACKER_URL,
|
||||||
|
|||||||
@@ -7,7 +7,7 @@ from dataclasses import dataclass
|
|||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Any, Literal
|
from typing import Any, Literal
|
||||||
|
|
||||||
Quantization = Literal["bfloat16", "int8", "nf4"]
|
Quantization = Literal["auto", "bfloat16", "int8", "nf4"]
|
||||||
|
|
||||||
|
|
||||||
class ModelBackendError(RuntimeError):
|
class ModelBackendError(RuntimeError):
|
||||||
@@ -31,14 +31,14 @@ class TensorPayload:
|
|||||||
|
|
||||||
|
|
||||||
def validate_quantization(value: str) -> Quantization:
|
def validate_quantization(value: str) -> Quantization:
|
||||||
if value not in {"bfloat16", "int8", "nf4"}:
|
if value not in {"auto", "bfloat16", "int8", "nf4"}:
|
||||||
raise ValueError("quantization must be one of: bfloat16, int8, nf4")
|
raise ValueError("quantization must be one of: auto, bfloat16, int8, nf4")
|
||||||
return value # type: ignore[return-value]
|
return value # type: ignore[return-value]
|
||||||
|
|
||||||
|
|
||||||
def build_quantization_config(quantization: Quantization) -> Any | None:
|
def build_quantization_config(quantization: Quantization) -> Any | None:
|
||||||
"""Return a transformers BitsAndBytesConfig for quantized weights."""
|
"""Return a transformers BitsAndBytesConfig for quantized weights."""
|
||||||
if quantization == "bfloat16":
|
if quantization in {"auto", "bfloat16"}:
|
||||||
return None
|
return None
|
||||||
try:
|
try:
|
||||||
import torch
|
import torch
|
||||||
@@ -65,7 +65,7 @@ class TorchModelShard:
|
|||||||
model_id: str,
|
model_id: str,
|
||||||
shard_start: int,
|
shard_start: int,
|
||||||
shard_end: int,
|
shard_end: int,
|
||||||
quantization: Quantization = "bfloat16",
|
quantization: Quantization = "auto",
|
||||||
cache_dir: Path | None = None,
|
cache_dir: Path | None = None,
|
||||||
) -> None:
|
) -> None:
|
||||||
if shard_start < 0 or shard_end < 0 or shard_start > shard_end:
|
if shard_start < 0 or shard_end < 0 or shard_start > shard_end:
|
||||||
@@ -77,7 +77,7 @@ class TorchModelShard:
|
|||||||
|
|
||||||
try:
|
try:
|
||||||
import torch
|
import torch
|
||||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer
|
||||||
except ModuleNotFoundError as exc:
|
except ModuleNotFoundError as exc:
|
||||||
raise MissingModelDependencyError(
|
raise MissingModelDependencyError(
|
||||||
"real model backend requires torch, transformers, safetensors, accelerate, and bitsandbytes"
|
"real model backend requires torch, transformers, safetensors, accelerate, and bitsandbytes"
|
||||||
@@ -85,17 +85,27 @@ class TorchModelShard:
|
|||||||
|
|
||||||
self.torch = torch
|
self.torch = torch
|
||||||
self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
|
self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
|
||||||
quant_config = build_quantization_config(quantization)
|
quant_config, dtype, uses_quantized_weights = _model_load_plan(
|
||||||
|
AutoConfig,
|
||||||
|
model_id,
|
||||||
|
quantization,
|
||||||
|
torch,
|
||||||
|
cache_dir,
|
||||||
|
)
|
||||||
try:
|
try:
|
||||||
|
load_kwargs = {
|
||||||
|
"device_map": "auto" if uses_quantized_weights else None,
|
||||||
|
"dtype": dtype,
|
||||||
|
"low_cpu_mem_usage": True,
|
||||||
|
"cache_dir": str(cache_dir) if cache_dir is not None else None,
|
||||||
|
}
|
||||||
|
if quant_config is not None:
|
||||||
|
load_kwargs["quantization_config"] = quant_config
|
||||||
self.model = AutoModelForCausalLM.from_pretrained(
|
self.model = AutoModelForCausalLM.from_pretrained(
|
||||||
model_id,
|
model_id,
|
||||||
quantization_config=quant_config,
|
**load_kwargs,
|
||||||
device_map="auto" if quant_config is not None else None,
|
|
||||||
dtype=torch.bfloat16,
|
|
||||||
low_cpu_mem_usage=True,
|
|
||||||
cache_dir=str(cache_dir) if cache_dir is not None else None,
|
|
||||||
)
|
)
|
||||||
if quant_config is None:
|
if not uses_quantized_weights:
|
||||||
self.model.to(self.device)
|
self.model.to(self.device)
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
if _looks_like_oom(exc):
|
if _looks_like_oom(exc):
|
||||||
@@ -340,12 +350,71 @@ def load_torch_shard(
|
|||||||
model_id: str,
|
model_id: str,
|
||||||
shard_start: int,
|
shard_start: int,
|
||||||
shard_end: int,
|
shard_end: int,
|
||||||
quantization: Quantization = "bfloat16",
|
quantization: Quantization = "auto",
|
||||||
cache_dir: Path | None = None,
|
cache_dir: Path | None = None,
|
||||||
) -> TorchModelShard:
|
) -> TorchModelShard:
|
||||||
return TorchModelShard(model_id, shard_start, shard_end, quantization, cache_dir)
|
return TorchModelShard(model_id, shard_start, shard_end, quantization, cache_dir)
|
||||||
|
|
||||||
|
|
||||||
|
def _model_load_plan(
|
||||||
|
auto_config: Any,
|
||||||
|
model_id: str,
|
||||||
|
quantization: Quantization,
|
||||||
|
torch: Any,
|
||||||
|
cache_dir: Path | None = None,
|
||||||
|
) -> tuple[Any | None, Any, bool]:
|
||||||
|
"""Return (explicit quant config, dtype, uses quantized weights)."""
|
||||||
|
if quantization != "auto":
|
||||||
|
quant_config = build_quantization_config(quantization)
|
||||||
|
return quant_config, torch.bfloat16, quant_config is not None
|
||||||
|
|
||||||
|
cfg = auto_config.from_pretrained(
|
||||||
|
model_id,
|
||||||
|
cache_dir=str(cache_dir) if cache_dir is not None else None,
|
||||||
|
)
|
||||||
|
if _native_quantization_config(cfg) is not None:
|
||||||
|
return None, _native_torch_dtype(cfg, torch), True
|
||||||
|
return None, _native_torch_dtype(cfg, torch), False
|
||||||
|
|
||||||
|
|
||||||
|
def _config_candidates(cfg: Any) -> list[Any]:
|
||||||
|
candidates = [cfg]
|
||||||
|
get_text_config = getattr(cfg, "get_text_config", None)
|
||||||
|
if callable(get_text_config):
|
||||||
|
try:
|
||||||
|
candidates.append(get_text_config())
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
text_config = getattr(cfg, "text_config", None)
|
||||||
|
if text_config is not None:
|
||||||
|
candidates.append(text_config)
|
||||||
|
return candidates
|
||||||
|
|
||||||
|
|
||||||
|
def _native_quantization_config(cfg: Any) -> Any | None:
|
||||||
|
for candidate in _config_candidates(cfg):
|
||||||
|
quant_config = getattr(candidate, "quantization_config", None)
|
||||||
|
if quant_config:
|
||||||
|
return quant_config
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _native_torch_dtype(cfg: Any, torch: Any) -> Any:
|
||||||
|
for candidate in _config_candidates(cfg):
|
||||||
|
for attr in ("dtype", "torch_dtype"):
|
||||||
|
dtype = getattr(candidate, attr, None)
|
||||||
|
if dtype is None:
|
||||||
|
continue
|
||||||
|
if isinstance(dtype, str):
|
||||||
|
dtype_name = dtype.removeprefix("torch.")
|
||||||
|
dtype_value = getattr(torch, dtype_name, None)
|
||||||
|
if dtype_value is not None:
|
||||||
|
return dtype_value
|
||||||
|
else:
|
||||||
|
return dtype
|
||||||
|
return torch.bfloat16
|
||||||
|
|
||||||
|
|
||||||
def _model_layers(model: Any) -> Any:
|
def _model_layers(model: Any) -> Any:
|
||||||
if hasattr(model, "model") and hasattr(model.model, "layers"):
|
if hasattr(model, "model") and hasattr(model.model, "layers"):
|
||||||
return model.model.layers
|
return model.model.layers
|
||||||
|
|||||||
@@ -52,6 +52,7 @@ def _max_assignable_layers(memory_mb: int, total_layers: int | None) -> int:
|
|||||||
def _shard_budget_line(memory_mb: int, memory_source: str, total_layers: int | None, quantization: str) -> str:
|
def _shard_budget_line(memory_mb: int, memory_source: str, total_layers: int | None, quantization: str) -> str:
|
||||||
memory_gb = memory_mb / 1024
|
memory_gb = memory_mb / 1024
|
||||||
gb_str = f"{memory_gb:.1f} GB"
|
gb_str = f"{memory_gb:.1f} GB"
|
||||||
|
budget_quantization = "bfloat16" if quantization == "auto" else quantization
|
||||||
if total_layers is None or total_layers <= 0:
|
if total_layers is None or total_layers <= 0:
|
||||||
return f"Memory budget: {gb_str} {memory_source}; shard budget: unknown model layer count"
|
return f"Memory budget: {gb_str} {memory_source}; shard budget: unknown model layer count"
|
||||||
max_layers = _max_assignable_layers(memory_mb, total_layers)
|
max_layers = _max_assignable_layers(memory_mb, total_layers)
|
||||||
@@ -61,7 +62,7 @@ def _shard_budget_line(memory_mb: int, memory_source: str, total_layers: int | N
|
|||||||
remaining_str = f"; {remaining_gb:.1f} GB remaining after full load" if remaining_gb > 1 else ""
|
remaining_str = f"; {remaining_gb:.1f} GB remaining after full load" if remaining_gb > 1 else ""
|
||||||
return (
|
return (
|
||||||
f"Memory budget: {gb_str} {memory_source}; "
|
f"Memory budget: {gb_str} {memory_source}; "
|
||||||
f"Shard budget: up to {max_layers}/{total_layers} layers at {quantization}"
|
f"Shard budget: up to {max_layers}/{total_layers} layers at {budget_quantization}"
|
||||||
f"{remaining_str}"
|
f"{remaining_str}"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -306,7 +307,7 @@ def run_startup(
|
|||||||
model_id: str | None = None,
|
model_id: str | None = None,
|
||||||
shard_start: int | None = None,
|
shard_start: int | None = None,
|
||||||
shard_end: int | None = None,
|
shard_end: int | None = None,
|
||||||
quantization: str = "bfloat16",
|
quantization: str = "auto",
|
||||||
wallet_path: Path | None = None,
|
wallet_path: Path | None = None,
|
||||||
cache_dir: Path | None = None,
|
cache_dir: Path | None = None,
|
||||||
host: str = "127.0.0.1",
|
host: str = "127.0.0.1",
|
||||||
|
|||||||
@@ -12,7 +12,7 @@
|
|||||||
"recommended": true,
|
"recommended": true,
|
||||||
"deployment_status": "recommended",
|
"deployment_status": "recommended",
|
||||||
"hf_aliases": [],
|
"hf_aliases": [],
|
||||||
"hf_verified_match_note": "Pending human curation (issue 23) — no HF inference-marketplace listing has been confirmed as a comparable params/quantization match for this preset yet. Leave empty until a human signs off; an empty hf_aliases list keeps this model on the static default price.",
|
"hf_verified_match_note": "Pending human curation (issue 23) \u2014 no HF inference-marketplace listing has been confirmed as a comparable params/quantization match for this preset yet. Leave empty until a human signs off; an empty hf_aliases list keeps this model on the static default price.",
|
||||||
"required_model_bytes": 638876385280,
|
"required_model_bytes": 638876385280,
|
||||||
"download_size_bytes": 638876385280,
|
"download_size_bytes": 638876385280,
|
||||||
"native_quantization": "int4",
|
"native_quantization": "int4",
|
||||||
@@ -38,6 +38,41 @@
|
|||||||
"KTransformers"
|
"KTransformers"
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
|
},
|
||||||
|
"qwen3.6-35b-a3b": {
|
||||||
|
"layers_start": 0,
|
||||||
|
"layers_end": 39,
|
||||||
|
"hf_repo": "unsloth/Qwen3.6-35B-A3B",
|
||||||
|
"aliases": [
|
||||||
|
"qwen3.6-35b-a3b",
|
||||||
|
"Qwen3.6-35B-A3B",
|
||||||
|
"unsloth/Qwen3.6-35B-A3B",
|
||||||
|
"Qwen/Qwen3.6-35B-A3B"
|
||||||
|
],
|
||||||
|
"recommended": true,
|
||||||
|
"deployment_status": "recommended",
|
||||||
|
"price_per_1k_tokens": 0.00044,
|
||||||
|
"hf_aliases": [
|
||||||
|
"qwen/qwen3.6-35b-a3b"
|
||||||
|
],
|
||||||
|
"hf_verified_match_note": "Verified 2026-07-06: unsloth/Qwen3.6-35B-A3B is a bf16 mirror of Qwen/Qwen3.6-35B-A3B; deepinfra and featherless-ai serve the official weights on the HF inference marketplace, so their rates are a fair comparable. Static price 0.00044 = 80% of deepinfra's blended $0.55/1M ($0.15 in / $0.95 out); the nightly refresher keeps it tracking.",
|
||||||
|
"required_model_bytes": 71903776776,
|
||||||
|
"download_size_bytes": 71903776776,
|
||||||
|
"native_quantization": "bfloat16",
|
||||||
|
"canonical_audit_dtype": "bfloat16",
|
||||||
|
"canonical_audit_quantization": "bfloat16",
|
||||||
|
"bytes_per_layer": {
|
||||||
|
"bfloat16": 1797594419
|
||||||
|
},
|
||||||
|
"metadata": {
|
||||||
|
"architecture": "Mixture-of-Experts (MoE, hybrid linear attention)",
|
||||||
|
"total_parameters": "35B",
|
||||||
|
"activated_parameters": "3B",
|
||||||
|
"num_layers": 40,
|
||||||
|
"context_length": 262144,
|
||||||
|
"native_quantization": "bfloat16",
|
||||||
|
"download_size_gb": 72
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -46,6 +46,24 @@ from .gossip import NodeGossip
|
|||||||
from .raft import RaftNode
|
from .raft import RaftNode
|
||||||
|
|
||||||
|
|
||||||
|
def _preset_price_keys(name: str, preset: dict) -> set[str]:
|
||||||
|
"""All model strings a client may bill under for one preset.
|
||||||
|
|
||||||
|
``BillingLedger.price_for`` is keyed by the raw ``model`` string in the
|
||||||
|
request, so the preset price must be registered under the preset name,
|
||||||
|
its ``hf_repo``, and every alias — otherwise ``unsloth/Qwen…`` style
|
||||||
|
requests silently fall back to the default rate.
|
||||||
|
"""
|
||||||
|
keys = {name}
|
||||||
|
hf_repo = preset.get("hf_repo")
|
||||||
|
if isinstance(hf_repo, str) and hf_repo:
|
||||||
|
keys.add(hf_repo)
|
||||||
|
for alias in preset.get("aliases") or []:
|
||||||
|
if isinstance(alias, str) and alias:
|
||||||
|
keys.add(alias)
|
||||||
|
return keys
|
||||||
|
|
||||||
|
|
||||||
def derive_relay_url_from_public_tracker_url(url: str | None) -> str | None:
|
def derive_relay_url_from_public_tracker_url(url: str | None) -> str | None:
|
||||||
"""Return wss://host/ws when url is a public HTTPS tracker origin."""
|
"""Return wss://host/ws when url is a public HTTPS tracker origin."""
|
||||||
if not url:
|
if not url:
|
||||||
@@ -4065,9 +4083,10 @@ class TrackerServer:
|
|||||||
db_path = DEFAULT_BILLING_DB_PATH
|
db_path = DEFAULT_BILLING_DB_PATH
|
||||||
if db_path:
|
if db_path:
|
||||||
preset_prices = {
|
preset_prices = {
|
||||||
name: float(preset["price_per_1k_tokens"])
|
key: float(preset["price_per_1k_tokens"])
|
||||||
for name, preset in self._model_presets.items()
|
for name, preset in self._model_presets.items()
|
||||||
if isinstance(preset, dict) and "price_per_1k_tokens" in preset
|
if isinstance(preset, dict) and "price_per_1k_tokens" in preset
|
||||||
|
for key in _preset_price_keys(name, preset)
|
||||||
}
|
}
|
||||||
billing = BillingLedger(db_path=db_path, prices=preset_prices)
|
billing = BillingLedger(db_path=db_path, prices=preset_prices)
|
||||||
self._billing: BillingLedger | None = billing
|
self._billing: BillingLedger | None = billing
|
||||||
@@ -4332,7 +4351,8 @@ class TrackerServer:
|
|||||||
continue
|
continue
|
||||||
if result is None:
|
if result is None:
|
||||||
continue
|
continue
|
||||||
billing.set_price(name, result["new_price_per_1k"])
|
for key in _preset_price_keys(name, preset):
|
||||||
|
billing.set_price(key, result["new_price_per_1k"])
|
||||||
preset["hf_last_price_per_1k"] = result["new_price_per_1k"]
|
preset["hf_last_price_per_1k"] = result["new_price_per_1k"]
|
||||||
preset["hf_last_updated"] = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
|
preset["hf_last_updated"] = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
|
||||||
if self._hf_pricing_log is not None:
|
if self._hf_pricing_log is not None:
|
||||||
|
|||||||
@@ -148,3 +148,41 @@ def test_hf_pricing_log_persists_and_is_queryable(tmp_path):
|
|||||||
# Reopening against the same db path recovers the log (billing.py pattern).
|
# Reopening against the same db path recovers the log (billing.py pattern).
|
||||||
reopened = HfPricingLog(db_path=db_path)
|
reopened = HfPricingLog(db_path=db_path)
|
||||||
assert len(reopened.history()) == 1
|
assert len(reopened.history()) == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_preset_price_keys_cover_name_repo_and_aliases():
|
||||||
|
from meshnet_tracker.server import _preset_price_keys
|
||||||
|
|
||||||
|
preset = {
|
||||||
|
"hf_repo": "unsloth/Qwen3.6-35B-A3B",
|
||||||
|
"aliases": ["qwen3.6-35b-a3b", "Qwen/Qwen3.6-35B-A3B"],
|
||||||
|
}
|
||||||
|
keys = _preset_price_keys("qwen3.6-35b-a3b", preset)
|
||||||
|
assert keys == {
|
||||||
|
"qwen3.6-35b-a3b",
|
||||||
|
"unsloth/Qwen3.6-35B-A3B",
|
||||||
|
"Qwen/Qwen3.6-35B-A3B",
|
||||||
|
}
|
||||||
|
assert _preset_price_keys("bare", {}) == {"bare"}
|
||||||
|
|
||||||
|
|
||||||
|
def test_qwen_preset_prices_apply_to_all_aliases(tmp_path):
|
||||||
|
"""Requests naming the repo id (what nodes register) bill at the preset price."""
|
||||||
|
from meshnet_tracker.server import TrackerServer
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
tracker = TrackerServer(billing_db=str(tmp_path / "billing.sqlite"))
|
||||||
|
try:
|
||||||
|
billing = tracker._billing
|
||||||
|
assert billing is not None
|
||||||
|
for key in (
|
||||||
|
"qwen3.6-35b-a3b",
|
||||||
|
"unsloth/Qwen3.6-35B-A3B",
|
||||||
|
"Qwen/Qwen3.6-35B-A3B",
|
||||||
|
):
|
||||||
|
assert billing.price_for(key) == pytest.approx(0.00044), key
|
||||||
|
# Unknown models keep the default rate
|
||||||
|
assert billing.price_for("some/other-model") == pytest.approx(0.02)
|
||||||
|
finally:
|
||||||
|
pass
|
||||||
|
|||||||
@@ -388,6 +388,48 @@ def test_legacy_start_treats_repo_model_as_model_id(monkeypatch):
|
|||||||
assert captured["model_id"] == "Qwen/Qwen2.5-0.5B-Instruct"
|
assert captured["model_id"] == "Qwen/Qwen2.5-0.5B-Instruct"
|
||||||
|
|
||||||
|
|
||||||
|
def test_legacy_start_falls_back_to_env_tracker_and_model(monkeypatch):
|
||||||
|
"""`meshnet-node start` uses env defaults when tracker/model flags are omitted."""
|
||||||
|
import importlib
|
||||||
|
|
||||||
|
from meshnet_node import config as config_mod
|
||||||
|
from meshnet_node.cli import main
|
||||||
|
|
||||||
|
monkeypatch.setenv("MESHNET_TRACKER_URL", "http://env-tracker:8081")
|
||||||
|
monkeypatch.setenv("MESHNET_MODEL", "Qwen/Qwen2.5-0.5B-Instruct")
|
||||||
|
importlib.reload(config_mod)
|
||||||
|
|
||||||
|
captured = {}
|
||||||
|
|
||||||
|
def fake_run_startup(*args, **kwargs):
|
||||||
|
captured.update(kwargs)
|
||||||
|
class _FakeNode:
|
||||||
|
chat_completion_count = 0
|
||||||
|
def stop(self): pass
|
||||||
|
return _FakeNode()
|
||||||
|
|
||||||
|
monkeypatch.setattr(sys, "argv", [
|
||||||
|
"meshnet-node", "start",
|
||||||
|
"--port", "0",
|
||||||
|
])
|
||||||
|
|
||||||
|
try:
|
||||||
|
with patch("meshnet_node.startup.run_startup", side_effect=fake_run_startup):
|
||||||
|
with patch("time.sleep", side_effect=KeyboardInterrupt):
|
||||||
|
try:
|
||||||
|
main()
|
||||||
|
except SystemExit as exc:
|
||||||
|
assert exc.code == 0
|
||||||
|
finally:
|
||||||
|
monkeypatch.delenv("MESHNET_TRACKER_URL", raising=False)
|
||||||
|
monkeypatch.delenv("MESHNET_MODEL", raising=False)
|
||||||
|
importlib.reload(config_mod)
|
||||||
|
|
||||||
|
assert captured["tracker_url"] == "http://env-tracker:8081"
|
||||||
|
assert captured["model"] == "Qwen2.5-0.5B-Instruct"
|
||||||
|
assert captured["model_id"] == "Qwen/Qwen2.5-0.5B-Instruct"
|
||||||
|
|
||||||
|
|
||||||
def test_legacy_start_without_port_uses_next_available_port(monkeypatch):
|
def test_legacy_start_without_port_uses_next_available_port(monkeypatch):
|
||||||
"""Omitting --port skips an occupied default port before startup loads the model."""
|
"""Omitting --port skips an occupied default port before startup loads the model."""
|
||||||
from meshnet_node.cli import main
|
from meshnet_node.cli import main
|
||||||
|
|||||||
@@ -1449,3 +1449,67 @@ def test_cli_loads_local_env_before_config_defaults(tmp_path, monkeypatch):
|
|||||||
|
|
||||||
assert config_mod.DEFAULTS["download_dir"] == "/run/media/popov/DATA/llm/safetensor/models"
|
assert config_mod.DEFAULTS["download_dir"] == "/run/media/popov/DATA/llm/safetensor/models"
|
||||||
assert os.environ["HF_TOKEN"] == "hf_test_token"
|
assert os.environ["HF_TOKEN"] == "hf_test_token"
|
||||||
|
|
||||||
|
|
||||||
|
def test_default_quantization_is_auto(monkeypatch):
|
||||||
|
import importlib
|
||||||
|
|
||||||
|
from meshnet_node import config as config_mod
|
||||||
|
from meshnet_node.model_backend import validate_quantization
|
||||||
|
|
||||||
|
monkeypatch.delenv("MESHNET_DOWNLOAD_DIR", raising=False)
|
||||||
|
importlib.reload(config_mod)
|
||||||
|
|
||||||
|
assert config_mod.DEFAULTS["quantization"] == "auto"
|
||||||
|
assert validate_quantization("auto") == "auto"
|
||||||
|
|
||||||
|
|
||||||
|
def test_auto_quantization_uses_native_model_dtype_for_unquantized_config():
|
||||||
|
from meshnet_node.model_backend import _model_load_plan
|
||||||
|
|
||||||
|
class AutoConfigStub:
|
||||||
|
@staticmethod
|
||||||
|
def from_pretrained(model_id, cache_dir=None):
|
||||||
|
assert model_id == "repo/model"
|
||||||
|
assert cache_dir is None
|
||||||
|
return types.SimpleNamespace(
|
||||||
|
text_config=types.SimpleNamespace(dtype="torch.bfloat16"),
|
||||||
|
)
|
||||||
|
|
||||||
|
torch_stub = types.SimpleNamespace(bfloat16="bf16", float16="fp16")
|
||||||
|
|
||||||
|
quant_config, dtype, uses_quantized_weights = _model_load_plan(
|
||||||
|
AutoConfigStub,
|
||||||
|
"repo/model",
|
||||||
|
"auto",
|
||||||
|
torch_stub,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert quant_config is None
|
||||||
|
assert dtype == "bf16"
|
||||||
|
assert uses_quantized_weights is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_auto_quantization_preserves_native_quantized_config():
|
||||||
|
from meshnet_node.model_backend import _model_load_plan
|
||||||
|
|
||||||
|
class AutoConfigStub:
|
||||||
|
@staticmethod
|
||||||
|
def from_pretrained(model_id, cache_dir=None):
|
||||||
|
return types.SimpleNamespace(
|
||||||
|
quantization_config={"quant_method": "gptq"},
|
||||||
|
torch_dtype="float16",
|
||||||
|
)
|
||||||
|
|
||||||
|
torch_stub = types.SimpleNamespace(bfloat16="bf16", float16="fp16")
|
||||||
|
|
||||||
|
quant_config, dtype, uses_quantized_weights = _model_load_plan(
|
||||||
|
AutoConfigStub,
|
||||||
|
"repo/model",
|
||||||
|
"auto",
|
||||||
|
torch_stub,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert quant_config is None
|
||||||
|
assert dtype == "fp16"
|
||||||
|
assert uses_quantized_weights is True
|
||||||
|
|||||||
Reference in New Issue
Block a user