diff --git a/.claude/memory/project-status.md b/.claude/memory/project-status.md index cb015bc..4334a9b 100644 --- a/.claude/memory/project-status.md +++ b/.claude/memory/project-status.md @@ -42,7 +42,7 @@ Historical handoff note: `/mnt/c/Users/popov/Downloads/neuron-tai-alpha-handoff- - Verification: downloader/startup targeted subset passes (`pytest tests/test_node_startup.py -k "download_shard or same_shard"`). Full `tests/test_node_startup.py` has 46 passed and 4 unrelated Windows chmod/path separator failures. - Live Windows confirmation: `meshnet-node start --tracker http://192.168.0.179:8080 --model Qwen3.6-35B-A3B` reuses `F:\_STORAGE\models\qwen3.6-35b-a3b`, prints `Cached at`, registers, and reaches ready as node `5gMLrmyB-26b1f8a4204a`. - Follow-up fix: preset-model startup now starts the heartbeat thread after registration; without this, the node appeared briefly on the dashboard and was purged on first inference/route after heartbeat expiry. Tracker dashboard now has a "Console output" panel backed by `/v1/console` for node register/expiry, routing failures, and proxy events. -- Qwen3.6-35B-A3B reserve-based split is expected: an 79 GB CPU node may be assigned layers 0-36, and a second node fills 37-39. Do not "fix" this by bypassing the 20% assignment reserve unless the shard-planning policy changes. +- Qwen3.6-35B-A3B CPU runtime cap (2026-07-08): the old reserve-based split could assign an 79 GB CPU node layers 0-36, but real partial loading can exceed that budget and die without a Python traceback. Node startup now clips oversized CPU auto-assignments before loading, and tracker CPU assignment uses a stricter runtime headroom factor; do not revert this to the old 20% reserve-only policy. - Route hardening: tracker chat proxy and `/v1/route` diagnostics now use alias-aware preset node matching for split Qwen3.6 routes; dashboard derives grouped inference history from proxy route/complete console events and shows observed TPS after completion. - Live proxy hardening: model lookup trims outer whitespace before alias matching (`qwen3.6-35b-a3b ` resolves), and tracker route logs/dashboard queue depth combine heartbeat queue with tracker-local proxy in-flight counts so Postman-style bursts no longer show every selected route as queue `0`. - Split-shard streaming hardening: Qwen3.6-style distributed generation now emits SSE chunks token-by-token from the head node instead of buffering all generated text until completion. Tracker direct/relay stream proxy logs `proxy progress` with live tokens/TPS, dashboard Inference history shows currently processing requests with live TPS/tokens/queue, and relay stream completion no longer references an undefined `session_id`. diff --git a/QUICKSTART.md b/QUICKSTART.md index fac8cd0..cfc7360 100644 --- a/QUICKSTART.md +++ b/QUICKSTART.md @@ -1,38 +1,74 @@ # Quickstart — Running a node and testing inference -This guide gets you from zero to a live inference request in three terminals. +Get from zero to a live inference request in **three terminals**: install once, start +the tracker, start a node, send a request. + Tested on: AMD Ryzen AI Max (Strix Halo APU), 124 GB RAM, Linux, CPU inference. +**Active development models** (what we run day-to-day): + +| Role | `--model` / alias | HF repo | Notes | +|------|-------------------|---------|-------| +| Smoke tests, small splits | `Qwen/Qwen2.5-0.5B-Instruct` | same | 24 layers, ~1 GB BF16, no gating — default for new setups | +| Alpha / production target | `qwen3.6-35b-a3b` | `unsloth/Qwen3.6-35B-A3B` | 40 layers, ~72 GB BF16, hybrid linear-attention MoE; aliases include `Qwen3.6-35B-A3B`, `Qwen/Qwen3.6-35B-A3B` | + --- -## Prerequisites +## At a glance -Choose the shell you will actually run `meshnet-*` from. Editable installs point -the command wrappers at this source tree, so normal code edits are picked up -without reinstalling. +| Step | What | Terminals | +|------|------|-----------| +| **0** | Install Python packages | once per machine | +| **1** | Start tracker (and relay if needed) | 1–2 | +| **2** | Start node(s) | 1+ | +| **3** | Send inference request | 1 | -**Linux / WSL** +**Pick your connectivity mode** — this determines which flags you need on the node: + +| Mode | When to use | Tracker URL | Node extras | +|------|-------------|-------------|-------------| +| **Local dev** | Everything on one machine | `http://localhost:8080` | none | +| **Direct LAN** | Node has a real LAN IP other machines can reach | `http://:8080` | `--host 0.0.0.0 --advertise-host ` + firewall | +| **Relay / public** | WSL2, NAT, 5G, or any unreachable inbound port | `https://ai.neuron.d-popov.com` (or your public URL) | none — relay handles routing | + +> **WSL2:** not reachable from other LAN machines by default. Use the **relay / public** +> tracker URL, or run the node in native Windows PowerShell with direct LAN mode. + +**Command prefix by shell** (used in examples below): + +| Shell | Prefix | Model cache env | +|-------|--------|-----------------| +| Linux / WSL | `.venv/bin/` | `HF_HOME=/path/to/models` | +| Windows PowerShell | `.\.venv\Scripts\` | `$env:HF_HOME = "D:\DEV\models"` | + +--- + +## 0. Install prerequisites (once per machine) + +Editable installs point wrappers at this source tree — code edits apply without +reinstalling. + +### Node machine — full install + +
+Linux / WSL ```bash -# Clone and enter repo -cd /run/media/popov/d/DEV/repos/d-popov.com/AI - -# Create the virtualenv if it does not exist yet +cd /path/to/neuron-tai python3 -m venv .venv source .venv/bin/activate -# Keep packaging tools current enough for editable installs .venv/bin/python -m pip install --upgrade pip setuptools wheel -# Install Python packages (editable — picks up code changes immediately) .venv/bin/python -m pip install -e packages/tracker -e packages/node -e packages/p2p -e packages/gateway -e packages/relay -# PyTorch: choose one -.venv/bin/python -m pip install torch -# .venv/bin/python -m pip install torch --index-url https://download.pytorch.org/whl/cpu -# .venv/bin/python -m pip install torch --index-url https://download.pytorch.org/whl/rocm6.2 -# HuggingFace model libraries .venv/bin/python -m pip install "transformers>=5.12" accelerate +.venv/bin/meshnet-node --help ``` -**Windows PowerShell** +
+ +
+Windows PowerShell (.venv) + +Requires Python 3.11+ and Git for Windows. ```powershell cd D:\DEV\workspace\REPOS\git.d-popov.com\neuron-tai @@ -40,321 +76,163 @@ python -m venv .venv .\.venv\Scripts\Activate.ps1 .\.venv\Scripts\python.exe -m pip install --upgrade pip setuptools wheel .\.venv\Scripts\python.exe -m pip install -e .\packages\tracker -e .\packages\node -e .\packages\p2p -e .\packages\gateway -e .\packages\relay -# PyTorch: choose one -.\.venv\Scripts\python.exe -m pip install torch -# .\.venv\Scripts\python.exe -m pip install torch --index-url https://download.pytorch.org/whl/cpu -# .\.venv\Scripts\python.exe -m pip install torch --index-url https://download.pytorch.org/whl/rocm6.2 .\.venv\Scripts\python.exe -m pip install "transformers>=5.12" accelerate +.\.venv\Scripts\meshnet-node.exe --help ``` -> Torch choices: default index for NVIDIA/CUDA, CPU index for CPU-only, ROCm index for AMD GPU. +
-### Version and library notes for Qwen3.5/3.6-MoE models - -- **transformers ≥ 5.12 is required** for Qwen3.5/3.6-MoE (e.g. `Qwen3.6-35B-A3B`). - Older versions fail at load time with - `'Qwen3_5MoeConfig' object has no attribute 'vocab_size'`. Check with - `python -c "import transformers; print(transformers.__version__)"` and upgrade - with `pip install -U transformers` in the environment that runs `meshnet-node` - (conda/miniforge users: upgrade inside that env, not a layered `.venv`). -- **Linear-attention fast path (GPU only).** Qwen3.5/3.6 use hybrid linear-attention - layers; without optional CUDA kernels, Transformers falls back to slower pure-PyTorch - code and prints `The fast path is not available…` at startup. That warning is - harmless — inference still works. On native Windows, install `triton-windows` in - the same env as `meshnet-node`; otherwise `flash-linear-attention` can fail during - import with `Could not import module 'Qwen3_5MoeForCausalLM'`. Install the - acceleration packages into the same env as `meshnet-node` for GPU speed; skip on - CPU-only nodes: - - ```bash - # Native Windows - pip install triton-windows - - # NVIDIA (CUDA) - pip install flash-linear-attention[cuda] causal-conv1d - - # AMD (ROCm) — match your torch index, then: - pip install flash-linear-attention[rocm] causal-conv1d - ``` - - Restart the node after install; the warning should disappear. Expect the largest - gain on GPU nodes serving linear-attention layers (roughly three quarters of - Qwen3.6 layers); end-to-end chat speed still depends on the slowest hop in a - split route. -- `pip install nvidia-ml-py` silences the pynvml deprecation warning on NVIDIA hosts. - -## Bootstrap a tracker on a new machine - -Use this when provisioning a fresh LAN/public tracker host. The tracker itself is -lightweight; install the relay too if nodes will connect from NAT, WSL2, mobile, -or other networks where inbound node ports are not reachable. - -```bash -# 1. Get the repo onto the tracker host -git clone https://git.d-popov.com/popov/neuron-tai.git AI -cd AI - -# 2. Create an isolated Python environment -python3 -m venv .venv -.venv/bin/python -m pip install --upgrade pip setuptools wheel - -# 3. Install only the services needed by the tracker host -.venv/bin/pip install -e packages/tracker -e packages/relay -e packages/gateway -``` - -For a private LAN tracker, start only the tracker and open the selected TCP port -on the host firewall if other machines will join: - -```bash -.venv/bin/meshnet-tracker start --host 0.0.0.0 --port 8080 -# --starting-credit 1 --devnet-topup 10 -``` - -Verify from the tracker host: - -```bash -curl -s http://localhost:8080/v1/network/map | python3 -m json.tool -``` - -Verify from another LAN machine, replacing the IP with the tracker host's LAN IP: - -```bash -curl -s http://192.168.0.179:8080/v1/network/map | python3 -m json.tool -``` - -For a public tracker with relay support, run both services. The relay listens on -`8765`; the tracker below listens on `8081` and advertises the public WebSocket -URL that nodes should use for outbound relay connections: - -```bash -# Terminal 1 — relay -.venv/bin/meshnet-relay --host 0.0.0.0 --port 8765 - -# Terminal 2 — tracker -.venv/bin/meshnet-tracker start --host 0.0.0.0 --port 8081 --relay-url wss://ai.neuron.d-popov.com/ws -``` - -If this host sits behind Nginx Proxy Manager, point `/` and `/v1/*` at tracker -port `8081`, and point `/ws` plus `/rpc` at relay port `8765` as shown in the -public tracker section below. After the proxy is configured, verify the public -bootstrap endpoint: - -```bash -curl -s https://ai.neuron.d-popov.com/v1/network/map | python3 -m json.tool -``` - -Nodes can then join with either the LAN tracker URL or the public URL: - -```bash -.venv/bin/meshnet-node start --tracker http://192.168.0.179:8080 --model Qwen/Qwen2.5-0.5B-Instruct -.venv/bin/meshnet-node start --tracker https://ai.neuron.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct -.venv/bin/meshnet-node start --tracker https://ai.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct -``` - -### Windows / WSL2 - -Run the Linux commands from WSL, not Git Bash. From the repo opened in Git Bash: - -```bash -wsl -cd /mnt/d/DEV/workspace/REPOS/git.d-popov.com/neuron-tai -python3 -m venv .venv -.venv/bin/python -m pip install --upgrade pip setuptools wheel -.venv/bin/pip install -e packages/tracker -e packages/node -e packages/p2p -e packages/gateway -e packages/relay -.venv/bin/pip install torch --index-url https://download.pytorch.org/whl/cpu -.venv/bin/pip install "transformers>=5.12" accelerate -.venv/bin/meshnet-node --help -``` - -If `.venv/bin/meshnet-node` is missing, the editable install step did not finish -successfully. Re-run the `.venv/bin/pip install -e ...` command above inside WSL. - -WSL2 sits behind Windows NAT and is **not directly reachable** from other LAN machines. -Direct cross-host hops time out. The relay path (see below) solves this: the WSL2 node -opens an outbound WebSocket to the relay server and all traffic flows through that tunnel. -No firewall rules, no `--advertise-host` needed — just point at the public tracker URL. - -### Native Windows PowerShell node (not WSL) - -Use this when the tracker is on another machine and you want Windows to host a -reachable node on the LAN. - -#### Option A — existing conda/miniforge environment with CUDA torch (recommended if you already have it) - -First, make sure the conda base environment is active so that `python` and `pip` both -resolve to the same miniforge installation: +
+Windows — conda/miniforge with CUDA (skip if using .venv above) ```powershell conda activate base -deactivate # drop any .venv that may be layered on top; safe no-op if none active -``` - -Install project packages into the active conda/miniforge env: - -```powershell +deactivate # drop any layered .venv; safe no-op if none active cd D:\DEV\workspace\REPOS\git.d-popov.com\neuron-tai - pip install -e packages\tracker -e packages\node -e packages\p2p -e packages\gateway -e packages\relay -pip install "transformers>=5.12" accelerate safetensors # torch is already present -``` - -> Conda/miniforge envs often carry an older `transformers` pinned by other tools -> (aider, etc.). Qwen3.5/3.6-MoE models need **transformers ≥ 5.12** — verify with -> `python -c "import transformers; print(transformers.__version__)"`. The pip -> resolver may print dependency-conflict warnings for those other tools; they don't -> affect `meshnet-node`. - -Verify torch is importable and CUDA is live **before** starting the node: - -```powershell +pip install "transformers>=5.12" accelerate safetensors python -c "import torch; print(torch.__version__, torch.cuda.is_available())" # Expected: 2.x.x+cuXXX True ``` -If you get `ModuleNotFoundError: No module named 'torch'` even though `pip install torch` -says "already satisfied", the `torch/` package directory is missing while the metadata -stub remains (can happen after a conda-managed install). Force-reinstall all three -PyTorch packages together so their versions stay in sync: +If `torch` import fails despite pip saying "already satisfied", force-reinstall all +three together (never upgrade `torch` alone — breaks `torchvision`): ```powershell pip install --force-reinstall torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118 ``` -> **Important:** always reinstall `torch`, `torchvision`, and `torchaudio` as a group. -> Upgrading only `torch` leaves `torchvision` on an incompatible version, which causes -> `RuntimeError: operator torchvision::nms does not exist` and makes transformers fail -> to import any model class (the error surfaces as `Could not import module 'Qwen2ForCausalLM'`). +If `.venv\Scripts\meshnet-node.exe` shadows the conda binary, use the full path: +`C:\Users\\miniforge3\Scripts\meshnet-node.exe`. -Then re-run the verify step above. +
-If that prints `True` but `meshnet-node` still can't find torch, the venv entry point -is shadowing the conda one. Check which binary wins: +> Run Linux/WSL commands from **WSL**, not Git Bash. From Git Bash: `wsl`, then `cd` +> to the repo under `/mnt/d/...`. -```powershell -(Get-Command meshnet-node).Source -# Should show: C:\Users\\miniforge3\Scripts\meshnet-node.exe -# If it shows .venv\Scripts\meshnet-node.exe, use the full path below instead -``` +### Tracker host — lightweight install -To start a node: - -```powershell -$env:HF_HOME = "D:\DEV\models" -meshnet-node start --tracker https://ai.neuron.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct -``` - -If the wrong entry point is shadowing, invoke via the full conda path: - -```powershell -C:\Users\popov\miniforge3\Scripts\meshnet-node.exe start --tracker https://ai.neuron.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct -``` - -#### Option B — isolated virtualenv (fresh machine, no existing torch) - -1. Install prerequisites on Windows: - - Python 3.11 or 3.12 from - - Git for Windows from - -2. Open **PowerShell** in the cloned repo and install the node packages: - -```powershell -# Example repo path; adjust to wherever you cloned it -cd D:\DEV\workspace\REPOS\git.d-popov.com\neuron-tai - -python -m venv .venv -.\.venv\Scripts\python.exe -m pip install --upgrade pip setuptools wheel -.\.venv\Scripts\pip.exe install -e packages\tracker -e packages\node -e packages\p2p -e packages\gateway -e packages\relay - -# CPU-only PyTorch. For NVIDIA CUDA, use `pip install torch` instead. -.\.venv\Scripts\pip.exe install torch --index-url https://download.pytorch.org/whl/cpu -.\.venv\Scripts\pip.exe install "transformers>=5.12" accelerate - -.\.venv\Scripts\meshnet-node.exe --help -``` - -For `start`-specific flags, run: - -```powershell -.\.venv\Scripts\meshnet-node.exe start --help -``` - -3. Find the Windows LAN IP address: - -```powershell -ipconfig -``` - -Use the IPv4 address on the active Ethernet/Wi-Fi adapter, for example -`192.168.0.42`. Avoid WSL/Docker/Hyper-V adapter addresses like `172.16.x.x`, -`172.17.x.x`, or other virtual adapter IPs. - -4. Allow inbound traffic for the node port in Windows Firewall. Run PowerShell as -Administrator once: - -```powershell -New-NetFirewallRule -DisplayName "Meshnet node 8005" -Direction Inbound -Action Allow -Protocol TCP -LocalPort 8005 -``` - -5. Start the Windows node from normal PowerShell. Replace the tracker and -advertised host values with your actual LAN addresses: - -```powershell -$env:HF_HOME = "D:\DEV\models" -.\.venv\Scripts\meshnet-node.exe start --tracker http://192.168.0.179:8081 --model Qwen/Qwen2.5-0.5B-Instruct --shard-start 12 --shard-end 23 --quantization bfloat16 --host 0.0.0.0 --advertise-host 192.168.0.42 --port 8005 -``` - -Other start examples (direct LAN — node must be reachable by IP from other machines): - -```powershell -.\.venv\Scripts\meshnet-node.exe start --tracker http://192.168.0.179:8081 --model Qwen/Qwen2.5-0.5B-Instruct --advertise-host 192.168.0.20 -``` - -Via public hostname with relay (works from behind NAT, WSL2, 5G — no `--advertise-host` needed): - -```powershell -.\.venv\Scripts\meshnet-node.exe start --tracker https://ai.neuron.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct -``` - -WSL (same relay path — no `--advertise-host`): +Tracker + relay only; skip node packages unless this machine also runs nodes. ```bash -.venv/bin/meshnet-node start --tracker https://ai.neuron.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct -.venv/bin/meshnet-node start --tracker http://192.168.0.179:8081 --model Qwen/Qwen2.5-0.5B-Instruct +git clone https://git.d-popov.com/popov/neuron-tai.git AI && cd AI +python3 -m venv .venv +.venv/bin/python -m pip install --upgrade pip setuptools wheel +.venv/bin/pip install -e packages/tracker -e packages/relay -e packages/gateway ``` -`--host 0.0.0.0` binds the node to all Windows interfaces. `--advertise-host` -is what the tracker gives to other nodes for direct connections; omit it when using -the relay path since all traffic flows through the relay tunnel instead. +### PyTorch variant -If you want verbose per-hop pipeline logs while debugging a split model, add -`--debug`. Leave it off for normal runs; otherwise every generated token logs -lines like: +Install **one** torch line into the same env as `meshnet-node`: -```text - [node] pipeline hop 0: http://127.0.0.1:8005 start_layer=22 - [node] pipeline hop 0 returned text=' token' - [node] pipeline hop 1: wss://ai.neuron.d-popov.com/rpc/abc123 relay start_layer=12 -``` +| Hardware | Install | +|----------|---------| +| NVIDIA CUDA | `pip install torch` (default index) | +| CPU only | `pip install torch --index-url https://download.pytorch.org/whl/cpu` | +| AMD ROCm | `pip install torch --index-url https://download.pytorch.org/whl/rocm6.2` | -6. From the tracker machine, verify Windows is reachable: +On Windows `.venv`, prefix with `.\.venv\Scripts\pip.exe`. Conda users with CUDA +torch already installed can skip this step. -```bash -curl http://192.168.0.42:8005/v1/health -``` +### Qwen3.5/3.6-MoE notes -If that endpoint returns 404 or 501, that is okay: it still proves the TCP -connection reached the node process. If it times out or connection-refuses, check -the Windows Firewall rule, `--host 0.0.0.0`, the selected LAN IP, and that the -node is still running. +Applies to **`qwen3.6-35b-a3b`** and other hybrid linear-attention models. **`Qwen2.5-0.5B`** +does not need any of this — it is a standard transformer with no FLA fast path. + +- **transformers ≥ 5.12 required** — older versions fail with + `'Qwen3_5MoeConfig' object has no attribute 'vocab_size'`. Check: + `python -c "import transformers; print(transformers.__version__)"`. +- **Optional GPU kernels** — without them inference still works; startup may print + `The fast path is not available…` (harmless on Windows; slower on linear-attention + layers). Install **only for your platform**: + + | Platform | Install | Fast path? | + |----------|---------|------------| + | **Native Windows** | `pip install triton-windows` (also pulled by `meshnet-node` on Windows) | **Load only** — model imports and runs on PyTorch fallback. Do **not** `pip install flash-linear-attention[cuda] causal-conv1d` (see below). | + | **Linux + NVIDIA CUDA** | `pip install flash-linear-attention[cuda]` | **Yes** — `causal-conv1d` optional since FLA ≥0.3.2 ships Triton conv1d; add it only if transformers still asks for it. Needs CUDA toolkit (`nvcc`) or a matching prebuilt wheel. | + | **Linux + AMD ROCm** | `pip install flash-linear-attention[rocm]` | **Yes** — same optional `causal-conv1d` note. | + + On native Windows, `triton-windows` is enough for Qwen3.6-MoE to **load**. Without it, + startup fails with misleading `Could not import module 'Qwen3_5MoeForCausalLM'`. + +
+Why there is no supported Windows fast path (research notes) + +We checked upstream docs and real install attempts (2026-07): + +1. **`causal-conv1d`** — no official Windows wheels on PyPI. Source builds need `nvcc` + plus a torch/CUDA stack that matches; without `nvcc` pip fails with + `bare_metal_version is not defined`. Open upstream issues confirm Windows is + unsupported / broken for most setups. +2. **`flash-linear-attention[cuda]`** — FLA's install guide targets Linux (CUDA / ROCm / + XPU / NPU). The `[cuda]` extra pulls PyPI `triton`, which conflicts with + `triton-windows` on native Windows. FLA does not document or CI-test Windows. +3. **What works today on native Windows GPU** — CUDA torch + `triton-windows` → Qwen3.6 + loads and infers via Transformers' pure-PyTorch fallback (~¾ of layers are + linear-attention; those hops are slower). This is what we use in alpha. +4. **If you need the fast path on a Windows PC** — run the GPU node in **WSL2 with Linux + CUDA** (or a Linux box): `pip install flash-linear-attention[cuda]` there. Keep + Windows native for reachability / relay if needed. +5. **Experimental / unsupported** — community `causal-conv1d` wheels and + `triton-windows` + pinned `flash-linear-attention --no-deps` hacks exist for narrow + Python/torch/CUDA combos; we do not support or test these for meshnet-node. + +
+ +- `pip install nvidia-ml-py` silences the pynvml deprecation warning on NVIDIA hosts. --- -## Public tracker + relay (internet / NAT nodes) +## 1. Start the tracker -This setup lets nodes connect from anywhere — behind home NAT, 5G, WSL2, or -on a different continent — without opening firewall ports. +### LAN tracker (private network, direct node reachability) -### Architecture +**Terminal 1:** + +```bash +.venv/bin/meshnet-tracker start --host 0.0.0.0 --port 8080 +# Optional devnet billing: --starting-credit 1 --devnet-topup 10 +``` + +Expected: `Tracker listening on 0.0.0.0:8080`. Open the port on the host firewall +if other machines will join. + +**Verify:** + +```bash +curl -s http://localhost:8080/v1/network/map | python3 -m json.tool +curl -s http://192.168.0.179:8080/v1/network/map | python3 -m json.tool # from another LAN machine +``` + +### Public tracker + relay (NAT / WSL2 / internet nodes) + +Nodes behind NAT cannot receive inbound connections. Run **both** services on the +tracker host; the tracker advertises the relay URL in `/v1/network/map`. + +**Terminal 1 — relay:** + +```bash +.venv/bin/meshnet-relay --host 0.0.0.0 --port 8765 +``` + +**Terminal 2 — tracker:** + +```bash +.venv/bin/meshnet-tracker start --host 0.0.0.0 --port 8081 --relay-url wss://ai.neuron.d-popov.com/ws +``` + +**Verify:** + +```bash +curl -s https://ai.neuron.d-popov.com/v1/network/map | python3 -m json.tool +``` + +Nodes should log `Relay connected — wss://…/rpc/` on startup. + +
+Nginx Proxy Manager setup (public hostname) + +Architecture: ``` Client → HTTPS → ai.neuron.d-popov.com (nginx) @@ -363,64 +241,27 @@ Client → HTTPS → ai.neuron.d-popov.com (nginx) └─ /rpc/* → meshnet-relay :8765 (caller opens WS per hop) ``` -### Nginx Proxy Manager (Docker) +Use **one** proxy host. Route sub-paths via **Custom locations** — do not create +a second host for the same domain. -Use **one** proxy host for the domain. Do not create a second host for the same -domain to reach another port — path routing is done with **Custom locations** on -that same host. - -**1. Details tab** (default `/` → tracker) +**Details tab** (default `/` → tracker): | Field | Value | |-------|--------| | Domain Names | `ai.neuron.d-popov.com` | | Scheme | `http` | -| Forward Hostname / IP | LAN IP of the tracker machine (e.g. `192.168.0.179`) | +| Forward Hostname / IP | LAN IP of tracker machine (e.g. `192.168.0.179`) | | Forward Port | `8081` | | Websockets Support | ON | -This serves `/v1/network/map`, `/v1/chat/completions`, and the rest of the tracker API. +**Custom locations** (both → relay port `8765`, sub-folder path empty): -**2. Custom locations tab** (sub-paths → relay) +| Location | Forward to | +|----------|--------------| +| `/ws` | `192.168.0.179:8765` | +| `/rpc` | `192.168.0.179:8765` | -The Custom locations form has **no separate Websockets toggle** — only location, -scheme, forward host, optional sub-folder path, and port. Add **two** locations -(both pointing at the relay process on port `8765`). Leave **“Add a path for -sub-folder forwarding”** empty so the full URI reaches the relay -(`/ws`, `/rpc/`). - -Location A — persistent node connections: - -| Field | Value | -|-------|--------| -| Define location | `/ws` | -| Scheme | `http` | -| Forward Hostname / IP | `192.168.0.179` | -| Forward Port | `8765` | -| Sub-folder path | *(leave empty)* | - -Location B — per-hop RPC: - -| Field | Value | -|-------|--------| -| Define location | `/rpc` | -| Scheme | `http` | -| Forward Hostname / IP | `192.168.0.179` | -| Forward Port | `8765` | -| Sub-folder path | *(leave empty)* | - -Nginx matches the longer prefixes first: `/ws` and `/rpc/…` go to relay; everything -else stays on `8081`. - -**3. SSL tab** - -Use your existing Let’s Encrypt certificate (unchanged). - -**4. Advanced tab** (only if WebSocket upgrade fails on `/ws` or `/rpc`) - -Custom locations do not expose a Websockets checkbox. If nodes show -`Relay configured but not connected yet` while `/v1/network/map` works, add this -snippet on the **proxy host** Advanced tab: +**Advanced tab** (only if WebSocket upgrade fails): ```nginx proxy_http_version 1.1; @@ -430,317 +271,266 @@ proxy_read_timeout 3600s; proxy_send_timeout 3600s; ``` -**5. Verify routing** +**Verify routing:** ```bash -# Tracker (8081 via default location) curl -s https://ai.neuron.d-popov.com/v1/network/map | python3 -m json.tool - -# Relay paths should not 502/404 from the tracker — check response headers/status curl -sI https://ai.neuron.d-popov.com/ws curl -sI https://ai.neuron.d-popov.com/rpc/test-peer ``` -After NPM is correct, start relay and tracker on the LAN machine: +
+ +--- + +## 2. Start a node + +**Starter model:** `Qwen/Qwen2.5-0.5B-Instruct` — 0.5B params, ~1 GB BF16, 24 layers, +no HuggingFace gating. Best for first-time setup. + +**Alpha model:** `qwen3.6-35b-a3b` — 40 layers, ~72 GB BF16 download, MoE with hybrid +linear attention. Use on machines with enough RAM/VRAM; see Qwen3.5/3.6 notes above for +`triton-windows` (Windows) or `flash-linear-attention` (Linux GPU). Tracker accepts the +alias or full repo id (`unsloth/Qwen3.6-35B-A3B`). + +Downloads cache under `~/.meshnet/models/` (or `$HF_HOME` / `$env:HF_HOME`). + +Shard range is auto-detected from the curated catalog. For unknown repos the node +fetches only `config.json`. Override with `--shard-start` / `--shard-end` for partial +shards or multi-node splits. + +### Core command + +Replace `` and adjust the prefix for your shell (see table above). + +**Linux / WSL:** ```bash -# Terminal 1 — relay -.venv/bin/meshnet-relay --host 0.0.0.0 --port 8765 - -# Terminal 2 — tracker (advertises relay to nodes) -.venv/bin/meshnet-tracker start --host 0.0.0.0 --port 8081 --relay-url wss://ai.neuron.d-popov.com/ws +HF_HOME=/path/to/models .venv/bin/meshnet-node start --tracker --model Qwen/Qwen2.5-0.5B-Instruct --quantization bfloat16 ``` -Nodes using `https://ai.neuron.d-popov.com` should then log: +**Windows PowerShell:** -```text -Relay advertised by tracker — using outbound tunnel wss://ai.neuron.d-popov.com/ws - Relay connected — wss://ai.neuron.d-popov.com/rpc/ +```powershell +$env:HF_HOME = "D:\DEV\models" +.\.venv\Scripts\meshnet-node.exe start --tracker --model Qwen/Qwen2.5-0.5B-Instruct --quantization bfloat16 ``` -The `--relay-url` flag embeds the relay address in `/v1/network/map`. Every node -queries that endpoint on startup and auto-connects if a relay URL is present. +### Ready-to-run examples -### Start a node (any machine, any network) +**Local dev (same machine as tracker):** -No `--advertise-host`, firewall rule, port forwarding, relay URL, or peer URL is -needed on the node. The public tracker is the only bootstrap URL the user types. -The node queries the tracker for `/v1/network/map`, discovers the relay URL, and -opens a persistent outbound WebSocket. If the relay connection drops, the node -keeps retrying it. +```bash +HF_HOME=/path/to/models .venv/bin/meshnet-node start --tracker http://localhost:8080 --model Qwen/Qwen2.5-0.5B-Instruct --quantization bfloat16 --port 8001 +``` + +**Public / relay (works from WSL2, NAT, 5G — no extra flags):** ```bash .venv/bin/meshnet-node start --tracker https://ai.neuron.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct ``` -No authentication is required in the prototype. The first public node for a model -must still choose that model with `--model` or a saved wizard config. After at -least one HF model node is registered, later nodes can join the public network -with only the tracker URL; the tracker assigns an uncovered shard if one exists: +**Alpha model (Qwen3.6, Windows GPU — PyTorch fallback, no FLA fast path):** + +```powershell +$env:HF_HOME = "D:\DEV\models" +meshnet-node start --tracker http://192.168.0.179:8080 --model qwen3.6-35b-a3b --quantization bfloat16 +``` + +**Alpha model (Qwen3.6, Linux GPU — with fast path):** + +```bash +HF_HOME=/path/to/models .venv/bin/meshnet-node start --tracker --model qwen3.6-35b-a3b --quantization bfloat16 +# Install once on that machine: pip install flash-linear-attention[cuda] +``` + +After the first node registers a model, later nodes can join with only the tracker +URL (shard auto-assigned): ```bash .venv/bin/meshnet-node start --tracker https://ai.neuron.d-popov.com ``` -Use the public tracker to verify registration and routing: +**Direct LAN (Windows node reachable by IP):** -```bash -curl -s "https://ai.neuron.d-popov.com/v1/network/map" | python3 -m json.tool -curl -s "https://ai.neuron.d-popov.com/v1/route?model=qwen2.5-0.5b" | python3 -m json.tool +```powershell +$env:HF_HOME = "D:\DEV\models" +.\.venv\Scripts\meshnet-node.exe start --tracker http://192.168.0.179:8081 --model Qwen/Qwen2.5-0.5B-Instruct --shard-start 12 --shard-end 23 --quantization bfloat16 --host 0.0.0.0 --advertise-host 192.168.0.42 --port 8005 ``` -Expected startup output (relay path): +
+Windows direct LAN — firewall and IP checklist + +1. Find LAN IP: `ipconfig` — use active Ethernet/Wi-Fi IPv4 (e.g. `192.168.0.42`). + Avoid WSL/Docker/Hyper-V addresses (`172.x.x.x`). +2. Allow inbound port (Administrator PowerShell, once): + +```powershell +New-NetFirewallRule -DisplayName "Meshnet node 8005" -Direction Inbound -Action Allow -Protocol TCP -LocalPort 8005 +``` + +3. Verify from tracker machine: + +```bash +curl http://192.168.0.42:8005/v1/health +``` + +404/501 is fine — it proves TCP reached the node. Timeout = check firewall, +`--host 0.0.0.0`, and `--advertise-host`. + +
+ +
+Two-node split (same or different machines) + +**Node A — layers 0–11 (head, serves chat):** + +```bash +HF_HOME=/path/to/models .venv/bin/meshnet-node start --tracker --model Qwen/Qwen2.5-0.5B-Instruct --shard-start 0 --shard-end 11 --quantization bfloat16 --port 8001 +``` + +**Node B — layers 12–23:** + +```bash +HF_HOME=/path/to/models .venv/bin/meshnet-node start --tracker --model Qwen/Qwen2.5-0.5B-Instruct --shard-start 12 --shard-end 23 --quantization bfloat16 --port 8002 +``` + +Send inference to Node A. For cross-machine LAN tests see `docs/TWO_MACHINE_TEST.md`. + +
+ +### Useful flags + +| Flag | Purpose | +|------|---------| +| `--port 8001` | Fixed listen port (default: first free ≥ 7000) | +| `--host 0.0.0.0` | Bind all interfaces (needed for direct LAN) | +| `--advertise-host ` | LAN IP the tracker tells other nodes (direct LAN only) | +| `--shard-start N --shard-end M` | Partial layer range | +| `--debug` | Verbose per-hop pipeline logs (noisy; off by default) | + +`--host 0.0.0.0` binds locally; `--advertise-host` is what peers use for direct +hops. Omit both when using the relay path. + +### Expected output ``` Auto-detected 24 layers → shard 0–23 - Relay connected — wss://ai.neuron.d-popov.com/rpc/abc1def2ef3f4567 + Relay connected — wss://ai.neuron.d-popov.com/rpc/abc1def2ef3f4567 # relay mode only ================================ meshnet-node ready Wallet:
Model ID: Qwen/Qwen2.5-0.5B-Instruct Shard: layers 0–23; 24 of 24 Quantization: bfloat16 - Endpoint: http://172.29.104.23:7001 + Endpoint: http://:8001 Node ID: Hardware: CPU ================================ ``` -The `Endpoint` shown is the local IP (unreachable from outside). Other nodes reach -this one via `wss://ai.neuron.d-popov.com/rpc/` instead. +The `Endpoint` is the local address. In relay mode, peers reach this node via +`wss:///rpc/` instead. -### How relay hops work +### Other CPU-friendly models -When node A needs to forward activations to node B (behind NAT): - -1. Tracker injects `X-Meshnet-Route` with `relay_addr` for each behind-NAT hop. -2. Node A opens a WebSocket to `wss://relay/rpc/{peer_id_B}`. -3. Relay forwards the `relay-http-request` envelope to Node B's persistent connection. -4. Node B processes `/forward` locally, returns `relay-http-response`. -5. Relay sends the response back to Node A over the same WebSocket. -6. Node A closes the WebSocket and continues the pipeline. - -Binary activation tensors (bfloat16) are Base64-encoded through the relay JSON -protocol and decoded on both sides — no precision loss. - -If the relay hop fails (relay down, peer disconnected), the node logs a warning and -falls back to a direct HTTP attempt before returning an error. - -### Test from WSL2 using the public tracker - -In WSL2 (which gets a `172.x.x.x` virtual IP — unreachable from other machines): - -```bash -# WSL2 Terminal 1 — head node (layers 0–11, handles chat requests) -.venv/bin/meshnet-node start --tracker https://ai.neuron.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct --shard-start 0 --shard-end 11 - -# WSL2 Terminal 2 — tail node (layers 12–23) -.venv/bin/meshnet-node start --tracker https://ai.neuron.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct --shard-start 12 --shard-end 23 -``` - -Both nodes connect to the relay automatically. When a chat request arrives at Node A, -it forwards activations to Node B via `wss://ai.neuron.d-popov.com/rpc/{peer_id_B}`. - -Send inference through the tracker (which picks the head node and injects the route): - -```bash -curl -s https://ai.neuron.d-popov.com/v1/chat/completions -H "Content-Type: application/json" -H "Authorization: Bearer sk-mesh-" -d '{"model": "Qwen/Qwen2.5-0.5B-Instruct", "messages": [{"role": "user", "content": "What is 7 times 8?"}], "stream": false}' | python3 -m json.tool -``` - -Or send directly to Node A's local port (within WSL): - -```bash -curl -s http://localhost:7001/v1/chat/completions \ - -H "Content-Type: application/json" \ - -d '{"model": "Qwen/Qwen2.5-0.5B-Instruct", "messages": [{"role": "user", "content": "Hi"}]}' -``` - -## Accounts, API keys, and credit (billing-enabled trackers) - -Public trackers run with billing on: `/v1/chat/completions` requires a real -API key from a registered account. Unknown bearer strings get `401`; a key -with no balance gets `402 insufficient balance`. - -**Dashboard flow (easiest):** open `https:///dashboard`, register with -an email + password, then click **+ new key**. The key (`sk-mesh-…`) shows its -balance next to it. If the tracker was started with `--starting-credit`, your -first key arrives pre-funded (Caller Credit, once per account). If it was -started with `--devnet-topup`, every key row has a **+$N (devnet)** button to -refill during testing. - -**Curl flow:** - -```bash -# 1. Register (once) -curl -s https:///v1/auth/register -H "Content-Type: application/json" -d '{"email": "you@example.com", "password": "hunter22-or-better"}' -# → {"session_token": "...", ...} - -# 2. Create an API key (session token from step 1) -curl -s https:///v1/account/keys -X POST -H "Authorization: Bearer " -# → {"api_key": "sk-mesh-...", "caller_credit_granted": true} - -# 3. Check balance / usage -curl -s https:///v1/account -H "Authorization: Bearer " - -# 4. (devnet trackers only) top up a key -curl -s https:///v1/account/topup -X POST -H "Authorization: Bearer " -H "Content-Type: application/json" -d '{"api_key": "sk-mesh-..."}' -``` - -Operator side: both features default to 1 USDT (`--starting-credit` / -`--devnet-topup`). Set both to 0 on mainnet deployments — real deposits flow -through the on-chain USDT treasury watcher instead. - ---- - -## Step 1 — Start the tracker (Terminal 1) - -```bash -cd /run/media/popov/d/DEV/repos/d-popov.com/AI -.venv/bin/meshnet-tracker start --port 8080 -``` - -Expected output: -``` -Tracker listening on 0.0.0.0:8080 -``` - -Keep this terminal open. - ---- - -## Step 2 — Start a node (Terminal 2) - -### Recommended model: Qwen2.5-0.5B-Instruct - -- 0.5B parameters, ~1 GB in BF16 -- No HuggingFace account or license required -- Downloads once to `~/.meshnet/models/`, cached for future runs -- 24 transformer layers (auto-detected — no need to specify) - -```bash -cd /run/media/popov/d/DEV/repos/d-popov.com/AI -HF_HOME=/run/media/popov/d/DEV/models .venv/bin/meshnet-node start --model Qwen/Qwen2.5-0.5B-Instruct --quantization bfloat16 --tracker http://localhost:8080 --port 8001 -``` - -Shard range is **auto-detected** from the curated catalog (no network call for known -models). For unknown repos, the node fetches only `config.json` (~1 KB) to read -`num_hidden_layers`. You can still pass `--shard-start` / `--shard-end` explicitly -to run a partial shard on one machine. - -Expected output (after model loads): -``` - Auto-detected 24 layers → shard 0–23 -================================ -meshnet-node ready - Wallet:
- Model ID: Qwen/Qwen2.5-0.5B-Instruct - Shard: layers 0–23 - Quantization: bfloat16 - Endpoint: http://:8001 - Hardware: CPU -================================ -``` - -### Other model options (all CPU-friendly) - -| Model | HF repo | Layers | BF16 size | Notes | -|-------|---------|--------|-----------|-------| -| Qwen2.5-0.5B | `Qwen/Qwen2.5-0.5B-Instruct` | 24 | ~1 GB | Fastest, no gating | +| Model | `--model` / alias | Layers | BF16 size | Notes | +|-------|-------------------|--------|-----------|-------| +| **Qwen2.5-0.5B** (dev default) | `Qwen/Qwen2.5-0.5B-Instruct` | 24 | ~1 GB | Fastest, no gating | +| **Qwen3.6-35B-A3B** (alpha) | `qwen3.6-35b-a3b` | 40 | ~72 GB | MoE; needs transformers ≥5.12; see Qwen3.5/3.6 notes | | Qwen2.5-1.5B | `Qwen/Qwen2.5-1.5B-Instruct` | 28 | ~3 GB | Better quality | | Phi-3-mini | `microsoft/Phi-3-mini-4k-instruct` | 32 | ~7.5 GB | Best CPU quality | | Llama-3.2-1B | `meta-llama/Llama-3.2-1B-Instruct` | 16 | ~2 GB | Requires HF login | | Llama-3.2-3B | `meta-llama/Llama-3.2-3B-Instruct` | 28 | ~6 GB | Requires HF login | -For gated models (Llama), run `huggingface-cli login` first. +For gated models (Llama): `huggingface-cli login` first. + +Browse more: `.venv/bin/meshnet-node models` or `.venv/bin/meshnet-node models --browse`. --- -## Step 3 — Send an inference request (Terminal 3) +## 3. Send an inference request -If you started the node with `--port 8001`, send the request directly to that -head node: +**Terminal 3** — direct to the head node (replace port if you omitted `--port`): -```bash Qwen2.5-0.5B-Instruct +```bash curl -s http://localhost:8001/v1/chat/completions -H "Content-Type: application/json" -d '{"model": "qwen2.5-0.5b", "messages": [{"role": "user", "content": "What is 7 times 8? Answer in one word."}], "stream": false}' | python3 -m json.tool ``` -If you did not pass `--port`, `meshnet-node start` uses the first free port at -or above `7000`. Use the `Endpoint:` printed by the node instead of `8001`. - -To test tracker routing/proxying, send the same OpenAI-compatible request to the -tracker, using either the full HuggingFace repo or the quick alias: +**Via tracker** (tests routing / proxying): ```bash curl -s http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{"model": "qwen2.5-0.5b", "messages": [{"role": "user", "content": "What is 7 times 8? Answer in one word."}], "stream": false}' | python3 -m json.tool ``` -Or use the test script: +**Public tracker:** + +```bash +curl -s https://ai.neuron.d-popov.com/v1/chat/completions -H "Content-Type: application/json" -H "Authorization: Bearer sk-mesh-" -d '{"model": "Qwen/Qwen2.5-0.5B-Instruct", "messages": [{"role": "user", "content": "What is 7 times 8?"}], "stream": false}' | python3 -m json.tool +``` + +**Test script:** ```bash .venv/bin/python scripts/test_lan_inference.py --tracker http://localhost:8080 --gateway http://localhost:8001 ``` ---- +**Verify registration on public tracker:** -## Two-node split (same machine, two terminals) - -Split Qwen2.5-0.5B's 24 layers across two node processes to test the sharded pipeline: - -**Node A — layers 0–11 (tracker mode, serves chat completions):** ```bash -HF_HOME=/run/media/popov/d/DEV/models .venv/bin/meshnet-node start --model Qwen/Qwen2.5-0.5B-Instruct --shard-start 0 --shard-end 11 --quantization bfloat16 --tracker http://localhost:8080 --port 8001 +curl -s "https://ai.neuron.d-popov.com/v1/network/map" | python3 -m json.tool +curl -s "https://ai.neuron.d-popov.com/v1/route?model=qwen2.5-0.5b" | python3 -m json.tool ``` -**Node B — layers 12–23:** +
+Accounts, API keys, and credit (billing-enabled trackers) + +Public trackers require a real API key for `/v1/chat/completions`. Unknown bearer → +`401`; zero balance → `402 insufficient balance`. + +**Dashboard:** open `https:///dashboard`, register, click **+ new key**. +With `--starting-credit`, the first key is pre-funded. With `--devnet-topup`, use +**+$N (devnet)** to refill during testing. + +**Curl flow:** + ```bash -HF_HOME=/run/media/popov/d/DEV/models .venv/bin/meshnet-node start --model Qwen/Qwen2.5-0.5B-Instruct --shard-start 12 --shard-end 23 --quantization bfloat16 --tracker http://localhost:8080 --port 8002 +curl -s https:///v1/auth/register -H "Content-Type: application/json" -d '{"email": "you@example.com", "password": "hunter22-or-better"}' +curl -s https:///v1/account/keys -X POST -H "Authorization: Bearer " +curl -s https:///v1/account -H "Authorization: Bearer " +curl -s https:///v1/account/topup -X POST -H "Authorization: Bearer " -H "Content-Type: application/json" -d '{"api_key": "sk-mesh-..."}' ``` -Send the request to Node A — it tokenizes, runs layers 0–11, passes binary -activations to Node B, and streams the final response back. +Operator defaults: `--starting-credit` and `--devnet-topup` both default to 1 USDT. +Set both to 0 on mainnet. + +
--- -## Two-machine LAN test (Linux + Windows/WSL2) +## Reference -See `docs/TWO_MACHINE_TEST.md` (created by US-018). +### How relay hops work -For WSL2 nodes, registration only proves the node can reach the tracker -outbound. Tracker-routed inference also requires the tracker to reach the node's -advertised endpoint inbound. Either run the node in native Windows PowerShell, -configure Windows port forwarding into WSL for the node port, or start the -tracker with a relay URL so the node registers a `relay_addr`. +When node A forwards activations to node B (behind NAT): ---- +1. Tracker injects `X-Meshnet-Route` with `relay_addr` for behind-NAT hops. +2. Node A opens WebSocket to `wss://relay/rpc/{peer_id_B}`. +3. Relay forwards `relay-http-request` to Node B's persistent connection. +4. Node B processes `/forward`, returns `relay-http-response`. +5. Relay sends response back to Node A; Node A continues the pipeline. -## Browse available models +Activations (bfloat16) are Base64-encoded in JSON — no precision loss. On relay +failure the node logs a warning and falls back to direct HTTP before erroring. + +### Interactive wizard ```bash -# Show curated list with VRAM requirements -.venv/bin/meshnet-node models - -# Browse HuggingFace Hub top-20 text-generation models -.venv/bin/meshnet-node models --browse -``` - ---- - -## Start with the interactive wizard - -```bash -# First run: wizard detects GPU, shows model list, saves config -.venv/bin/meshnet-node - -# Subsequent runs: starts directly from saved config -.venv/bin/meshnet-node - -# Re-run wizard even with saved config +.venv/bin/meshnet-node # first run: wizard; later: saved config .venv/bin/meshnet-node --reset-config ``` ---- - -## Run all tests +### Run all tests ```bash .venv/bin/python -m pytest -q diff --git a/packages/node/meshnet_node/startup.py b/packages/node/meshnet_node/startup.py index f3fd24d..14ebac6 100644 --- a/packages/node/meshnet_node/startup.py +++ b/packages/node/meshnet_node/startup.py @@ -197,12 +197,43 @@ def _max_assignable_layers( memory_mb: int, total_layers: int | None, bytes_per_layer: int | None = None, + *, + safety_fraction: float = 0.8, ) -> int: if total_layers is None or total_layers <= 0 or memory_mb <= 0: return 0 budget_bytes = memory_mb * 1024 * 1024 layer_bytes = bytes_per_layer or _DEFAULT_BYTES_PER_LAYER - return min(total_layers, int((budget_bytes * 0.8) // layer_bytes)) + return min(total_layers, int((budget_bytes * safety_fraction) // layer_bytes)) + + +def _runtime_shard_safety_fraction(device: str) -> float: + """CPU partial loads need room for model skeletons, tokenizer, and allocator peaks.""" + return 0.55 if device != "cuda" else 0.8 + + +def _cap_auto_assigned_shard( + shard_start: int, + shard_end: int, + total_layers: int | None, + memory_mb: int, + bytes_per_layer: int | None, + device: str, +) -> tuple[int, bool, int]: + if bytes_per_layer is None or total_layers is None: + return shard_end, False, 0 + max_layers = _max_assignable_layers( + memory_mb, + total_layers, + bytes_per_layer=bytes_per_layer, + safety_fraction=_runtime_shard_safety_fraction(device), + ) + if max_layers <= 0: + return shard_end, False, max_layers + assigned_layers = shard_end - shard_start + 1 + if assigned_layers <= max_layers: + return shard_end, False, max_layers + return min(total_layers - 1, shard_start + max_layers - 1), True, max_layers def _format_shard_label( @@ -226,13 +257,19 @@ def _shard_budget_line( total_layers: int | None, quantization: str, bytes_per_layer: int | None = None, + safety_fraction: float = 0.8, ) -> str: memory_gb = memory_mb / 1024 gb_str = f"{memory_gb:.1f} GB" budget_quantization = "bfloat16" if quantization == "auto" else quantization if total_layers is None or total_layers <= 0: return f"Memory budget: {gb_str} {memory_source}; shard budget: unknown model layer count" - max_layers = _max_assignable_layers(memory_mb, total_layers, bytes_per_layer=bytes_per_layer) + max_layers = _max_assignable_layers( + memory_mb, + total_layers, + bytes_per_layer=bytes_per_layer, + safety_fraction=safety_fraction, + ) # Remaining capacity after one full model load (rough estimate) shard_bytes = max_layers * (bytes_per_layer or _DEFAULT_BYTES_PER_LAYER) remaining_gb = (memory_mb * 1024 * 1024 - shard_bytes) / (1024 ** 3) @@ -729,6 +766,25 @@ def run_startup( if net_asgn.get("hf_repo") == model_id and net_asgn.get("gap_found"): shard_start = net_asgn["shard_start"] shard_end = net_asgn["shard_end"] + asgn_total_layers = int(net_asgn.get("num_layers") or detected) + asgn_bytes_per_layer = _assignment_bytes_per_layer(net_asgn, quantization) + capped_shard_end, was_capped, max_runtime_layers = _cap_auto_assigned_shard( + shard_start, + shard_end, + asgn_total_layers, + memory_budget_mb, + asgn_bytes_per_layer, + device, + ) + if was_capped: + original_end = shard_end + shard_end = capped_shard_end + print( + f" WARNING: tracker assigned layers {shard_start}-{original_end}, " + f"but CPU-safe runtime budget fits {max_runtime_layers}/{asgn_total_layers} layers; " + f"loading layers {shard_start}-{shard_end} instead.", + flush=True, + ) full_sources = ( [] if tracker_source_disabled else _full_model_sources(net_asgn.get("model_sources", [])) @@ -744,7 +800,7 @@ def run_startup( ) print( f" Tracker found uncovered shard: " - f"layers {shard_start}–{shard_end} (of {detected})", + f"layers {shard_start}-{shard_end} (of {detected})", flush=True, ) except Exception: @@ -859,18 +915,36 @@ def run_startup( assigned_shard_start: int = net_assignment["shard_start"] assigned_shard_end: int = net_assignment["shard_end"] assigned_num_layers: int = net_assignment["num_layers"] + assigned_bytes_per_layer = _assignment_bytes_per_layer(net_assignment, quantization) + capped_shard_end, was_capped, max_runtime_layers = _cap_auto_assigned_shard( + assigned_shard_start, + assigned_shard_end, + assigned_num_layers, + memory_budget_mb, + assigned_bytes_per_layer, + device, + ) + if was_capped: + original_end = assigned_shard_end + assigned_shard_end = capped_shard_end + print( + f" WARNING: tracker assigned layers {assigned_shard_start}-{original_end}, " + f"but CPU-safe runtime budget fits {max_runtime_layers}/{assigned_num_layers} layers; " + f"loading layers {assigned_shard_start}-{assigned_shard_end} instead.", + flush=True, + ) assigned_model_sources: list[dict] = net_assignment.get("model_sources", []) if _gap_found: print( f" Assigned gap: {assigned_hf_repo} " - f"layers {assigned_shard_start}–{assigned_shard_end} " + f"layers {assigned_shard_start}-{assigned_shard_end} " f"(of {assigned_num_layers})", flush=True, ) else: print( f" Assigned redundant copy: {assigned_hf_repo} " - f"layers {assigned_shard_start}–{assigned_shard_end} " + f"layers {assigned_shard_start}-{assigned_shard_end} " f"(of {assigned_num_layers})", flush=True, ) @@ -956,7 +1030,7 @@ def run_startup( f" Wallet: {address}\n" f" Model ID: {assigned_hf_repo}\n" f" Shard: {shard_label}\n" - f" {_shard_budget_line(memory_budget_mb, memory_budget_source, assigned_num_layers, quantization)}\n" + f" {_shard_budget_line(memory_budget_mb, memory_budget_source, assigned_num_layers, quantization, bytes_per_layer=assigned_bytes_per_layer, safety_fraction=_runtime_shard_safety_fraction(device))}\n" f" Quantization: {quantization}\n" f" Endpoint: {endpoint}\n" f" Node ID: {tracker_node_id or 'unregistered'}\n" @@ -1022,6 +1096,7 @@ def run_startup( memory_budget_mb, assigned_total_layers, assignment_bytes_per_layer, + safety_fraction=_runtime_shard_safety_fraction(device), ) if pinned_layers > max_layers: raise ValueError( @@ -1115,7 +1190,7 @@ def run_startup( f" Wallet: {address}\n" f" Model ID: {hf_repo}\n" f" Shard: {shard_label}\n" - f" {_shard_budget_line(memory_budget_mb, memory_budget_source, total_layers, quantization, bytes_per_layer=assignment_bytes_per_layer)}\n" + f" {_shard_budget_line(memory_budget_mb, memory_budget_source, total_layers, quantization, bytes_per_layer=assignment_bytes_per_layer, safety_fraction=_runtime_shard_safety_fraction(device))}\n" f" Quantization: {quantization}\n" f" Endpoint: {endpoint}\n" f" Node ID: {tracker_node_id or 'unregistered'}\n" @@ -1192,7 +1267,7 @@ def run_startup( f"meshnet-node ready\n" f" Wallet: {address}\n" f" Shard: {shard_label}\n" - f" {_shard_budget_line(memory_budget_mb, memory_budget_source, assigned_total_layers, quantization, bytes_per_layer=assignment_bytes_per_layer)}\n" + f" {_shard_budget_line(memory_budget_mb, memory_budget_source, assigned_total_layers, quantization, bytes_per_layer=assignment_bytes_per_layer, safety_fraction=_runtime_shard_safety_fraction(device))}\n" f" Endpoint: {endpoint}\n" f" Node ID: {node_id}\n" f" Hardware: {hw_str}\n" diff --git a/packages/tracker/meshnet_tracker/server.py b/packages/tracker/meshnet_tracker/server.py index e165ce0..48f0216 100644 --- a/packages/tracker/meshnet_tracker/server.py +++ b/packages/tracker/meshnet_tracker/server.py @@ -1275,16 +1275,23 @@ def _deployment_summary(nodes: list[_NodeEntry], preset: dict | None) -> dict: } -def _max_layers_for_memory(memory_mb: int, total_layers: int, preset: dict | None = None) -> int: +def _max_layers_for_memory( + memory_mb: int, + total_layers: int, + preset: dict | None = None, + *, + device: str | None = None, +) -> int: if total_layers <= 0: return 0 if memory_mb <= 0: return max(1, total_layers // 2) memory_bytes = memory_mb * 1024 * 1024 bytes_per_layer = next(iter(_preset_bytes_per_layer(preset).values())) if preset is not None else 30 * 1024 * 1024 + safety_fraction = 0.55 if device == "cpu" else 0.8 return min( total_layers, - max(1, int((memory_bytes * 0.8) // bytes_per_layer)), + max(1, int((memory_bytes * safety_fraction) // bytes_per_layer)), ) @@ -5273,7 +5280,7 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler): required_start, required_end = _preset_layer_bounds(preset) total_l = required_end - required_start + 1 memory_mb = vram_mb if (device == "cuda" and vram_mb > 0) else ram_mb - max_layers = _max_layers_for_memory(memory_mb, total_l, preset) + max_layers = _max_layers_for_memory(memory_mb, total_l, preset, device=device) shard_start = required_start shard_end = min(required_end, shard_start + max_layers - 1) self._send_json(200, { @@ -5373,11 +5380,11 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler): memory_mb = vram_mb if (device == "cuda" and vram_mb > 0) else ram_mb resolved_name, best_preset = _resolve_model_preset(server.model_presets, str(best_repo)) if memory_mb > 0: - max_layers = _max_layers_for_memory(memory_mb, total_l, best_preset) + max_layers = _max_layers_for_memory(memory_mb, total_l, best_preset, device=device) elif device == "cuda" and vram_mb >= 8192: max_layers = total_l else: - max_layers = _max_layers_for_memory(memory_mb, total_l, best_preset) + max_layers = _max_layers_for_memory(memory_mb, total_l, best_preset, device=device) shard_start = best_gap_start shard_end = min(total_l - 1, shard_start + max_layers - 1) diff --git a/tests/test_node_startup.py b/tests/test_node_startup.py index fdf09a9..0385f46 100644 --- a/tests/test_node_startup.py +++ b/tests/test_node_startup.py @@ -1680,6 +1680,66 @@ def test_preset_startup_rejects_pinned_shard_above_memory_budget(tmp_path, monke tracker.stop() +def test_network_auto_join_clips_oversized_cpu_assignment(tmp_path, monkeypatch, capsys): + """Old trackers may assign too many CPU layers; node clips before model load.""" + import meshnet_node.startup as startup_mod + + torch_calls: list[dict] = [] + registrations: list[dict] = [] + + class FakeBackend: + total_layers = 40 + + class FakeTorchNodeServer: + def __init__(self, **kwargs): + torch_calls.append(kwargs) + self.backend = FakeBackend() + self.tracker_node_id = None + + def start(self): + return 7000 + + def stop(self): + pass + + oversized_assignment = { + "hf_repo": "unsloth/Qwen3.6-35B-A3B", + "model": "qwen3.6-35b-a3b", + "shard_start": 0, + "shard_end": 36, + "num_layers": 40, + "gap_found": False, + "bytes_per_layer": {"bfloat16": 1_797_594_419}, + "model_sources": [], + } + + monkeypatch.setattr( + startup_mod, + "detect_hardware", + lambda: {"device": "cpu", "gpu_name": None, "vram_mb": 0, "ram_mb": 79 * 1024}, + ) + monkeypatch.setattr(startup_mod, "TorchNodeServer", FakeTorchNodeServer) + monkeypatch.setattr(startup_mod, "_get_json", lambda *_args, **_kwargs: oversized_assignment) + monkeypatch.setattr(startup_mod, "_post_json", lambda _url, payload: registrations.append(payload) or {"node_id": "n1"}) + monkeypatch.setattr(startup_mod, "_start_heartbeat", lambda *_args, **_kwargs: None) + monkeypatch.setattr(startup_mod, "model_metadata_for", lambda *_args, **_kwargs: {"num_layers": 40}) + + node = run_startup( + tracker_url="http://127.0.0.1:8080", + wallet_path=tmp_path / "wallet.json", + tracker_source_disabled=True, + ) + try: + assert torch_calls[0]["shard_start"] == 0 + assert torch_calls[0]["shard_end"] == 24 + assert registrations[0]["shard_end"] == 24 + output = capsys.readouterr().out + assert "CPU-safe runtime budget fits 25/40 layers" in output + assert "layers 0-24" in output + finally: + node.stop() + + def test_preset_model_with_hf_repo_loads_torch_backend(tmp_path, monkeypatch, capsys): """Named presets that advertise hf_repo must load TorchNodeServer, not the stub server.""" import meshnet_node.startup as startup_mod @@ -1996,6 +2056,55 @@ def test_network_assign_gap_found_field(): tracker.stop() +def test_network_assign_uses_conservative_cpu_runtime_budget(): + """CPU assignments leave headroom for partial-load overhead, not just raw weights.""" + import json as _json + import urllib.request as _ur + + tracker = TrackerServer(model_presets={ + "qwen3.6-35b-a3b": { + "hf_repo": "unsloth/Qwen3.6-35B-A3B", + "aliases": ["unsloth/Qwen3.6-35B-A3B"], + "layers_start": 0, + "layers_end": 39, + "recommended": True, + "bytes_per_layer": {"bfloat16": 1_797_594_419}, + }, + }) + port = tracker.start() + try: + data = _json.dumps({ + "endpoint": "http://127.0.0.1:9200", + "model": "qwen3.6-35b-a3b", + "hf_repo": "unsloth/Qwen3.6-35B-A3B", + "num_layers": 40, + "shard_start": 0, + "shard_end": 39, + "hardware_profile": {}, + "score": 1.0, + }).encode() + req = _ur.Request( + f"http://127.0.0.1:{port}/v1/nodes/register", + data=data, + headers={"Content-Type": "application/json"}, + method="POST", + ) + with _ur.urlopen(req) as r: + r.read() + + resp = _get_json( + f"http://127.0.0.1:{port}/v1/network/assign" + "?device=cpu&vram_mb=0&ram_mb=80896" + "&hf_repo=unsloth/Qwen3.6-35B-A3B" + ) + + assert resp["gap_found"] is False + assert resp["shard_start"] == 0 + assert resp["shard_end"] == 24 + finally: + tracker.stop() + + def test_route_finds_hf_model_across_two_nodes(): """Tracker /v1/route returns ordered route for HF model even without a preset.""" import json as _json