# Quickstart — Running a node and testing inference Get from zero to a live inference request in **three terminals**: install once, start the tracker, start a node, send a request. Tested on: AMD Ryzen AI Max (Strix Halo APU), 124 GB RAM, Linux CPU inference. ROCm GPU setup is covered below, but must be verified on the host because ROCm support depends on the exact AMD GPU/APU, kernel, driver, and ROCm runtime. **Support here is evidence, not a promise.** No combination of GPU, torch build, optional accelerator package and model is "supported" until `meshnet-node doctor` has pushed a real forward through the exact shard you intend to serve, on that machine. Detecting a GPU proves nothing; installing an optional package proves nothing. See [Validate before you serve](#validate-before-you-serve-meshnet-node-doctor). **Active development models** — what we run day-to-day, so their setup notes are the most exercised. This is *not* a support matrix: any model the node can load is fair game, and any model on this list can still fail `doctor` on your hardware. | Role | `--model` / alias | HF repo | Notes | |------|-------------------|---------|-------| | Smoke tests, small splits | `Qwen/Qwen2.5-0.5B-Instruct` | same | 24 layers, ~1 GB BF16, no gating — default for new setups | | Alpha / production target | `qwen3.6-35b-a3b` | `unsloth/Qwen3.6-35B-A3B` | 40 layers, ~72 GB BF16, hybrid linear-attention MoE; aliases include `Qwen3.6-35B-A3B`, `Qwen/Qwen3.6-35B-A3B` | | Recommended dense model | `qwen3.6-27b` | `Qwen/Qwen3.6-27B` | 64 layers, ~56 GB BF16, text-only; canonical revision is pinned by the tracker | --- ## Models The tracker advertises recommended models through `GET /v1/models`; the chat dashboard shows the same catalog. `Qwen/Qwen3.6-27B` is recommended even before it has complete coverage, so users and operators can see that the network intends to serve it. ```bash curl -s http://localhost:8080/v1/models | python -m json.tool ``` Each model reports its shard coverage and its selectable quantizations. A quantization is selectable only when the tracker can build a complete route from shards at that precision or higher: - `bfloat16` requires BF16 for every shard. - `int8` may combine INT8 and BF16 shards. - `nf4` may combine NF4, INT8, and BF16 shards. The chat UI chooses the highest complete precision by default. Uncovered variants remain visible as coverage requests: use the small **Vote for coverage** control to register demand without sending an unusable chat request. Clients can request a minimum precision explicitly. Omit `quantization` to request BF16: ```bash curl http://localhost:8080/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{ "model": "qwen3.6-27b", "quantization": "int8", "messages": [{"role": "user", "content": "Explain sharded inference."}] }' ``` The first valid request for a recommended model is demand proof. When the shared tracker pool has enough spare eligible capacity, it queues a tracker-managed initial deployment. Until a complete route exists, the request returns retryable `503 model_loading` rather than silently lowering precision. Nodes may serve BF16, INT8, or NF4 according to their declared capability. The official BF16 snapshot remains the canonical Qwen source; tracker/peer sources are preferred and Hugging Face is the fallback. A node's startup-selected `(model, shard range, quantization)` is pinned, while spare capacity can be used for tracker-managed work. --- ## At a glance | Step | What | Terminals | |------|------|-----------| | **0** | Install Python packages | once per machine | | **1** | Start tracker (and relay if needed) | 1–2 | | **2** | Start node(s) | 1+ | | **3** | Send inference request | 1 | **Pick your connectivity mode** — this determines which flags you need on the node: | Mode | When to use | Tracker URL | Node extras | |------|-------------|-------------|-------------| | **Local dev** | Everything on one machine | `http://localhost:8080` | none | | **Direct LAN** | Node has a real LAN IP other machines can reach | `http://:8080` | `--host 0.0.0.0 --advertise-host ` + firewall | | **Relay / public** | WSL2, NAT, 5G, or any unreachable inbound port | `https://ai.neuron.d-popov.com` (or your public URL) | none — relay handles routing | > **WSL2:** not reachable from other LAN machines by default. Use the **relay / public** > tracker URL, or run the node in native Windows PowerShell with direct LAN mode. **Command prefix by shell** (used in examples below): | Shell | Prefix | Model cache env | |-------|--------|-----------------| | Linux / WSL CPU | `.venv/bin/` | `HF_HOME=/path/to/models` | | Linux AMD ROCm / Radeon | `.venv-rocm/bin/` | `HF_HOME=/path/to/models` | | Windows PowerShell | `.\.venv\Scripts\` | `$env:HF_HOME = "D:\DEV\models"` | > **Ryzen AI Max / Radeon 8060S developers:** use `.venv-rocm/bin/` for every > node command and test that needs real GPU inference. The repository's default > `.venv` currently uses Python 3.14 and is not the ROCm node runtime. --- ## 0. Install prerequisites (once per machine) Editable installs point wrappers at this source tree — code edits apply without reinstalling. ### Node machine — full install
Linux / WSL ```bash cd /path/to/neuron-tai python3 -m venv .venv source .venv/bin/activate .venv/bin/python -m pip install --upgrade pip setuptools wheel .venv/bin/python -m pip install -e packages/tracker -e packages/node -e packages/p2p -e packages/gateway -e packages/relay .venv/bin/python -m pip install "transformers>=5.12" accelerate .venv/bin/meshnet-node --help ```
Windows PowerShell (.venv) Requires Python 3.11+ and Git for Windows. ```powershell cd D:\DEV\workspace\REPOS\git.d-popov.com\neuron-tai python -m venv .venv .\.venv\Scripts\Activate.ps1 .\.venv\Scripts\python.exe -m pip install --upgrade pip setuptools wheel .\.venv\Scripts\python.exe -m pip install -e .\packages\tracker -e .\packages\node -e .\packages\p2p -e .\packages\gateway -e .\packages\relay .\.venv\Scripts\python.exe -m pip install "transformers>=5.12" accelerate .\.venv\Scripts\meshnet-node.exe --help ```
Windows — conda/miniforge with CUDA (skip if using .venv above) ```powershell conda activate base deactivate # drop any layered .venv; safe no-op if none active cd D:\DEV\workspace\REPOS\git.d-popov.com\neuron-tai pip install -e packages\tracker -e packages\node -e packages\p2p -e packages\gateway -e packages\relay pip install "transformers>=5.12" accelerate safetensors python -c "import torch; print(torch.__version__, torch.cuda.is_available())" # Expected: 2.x.x+cuXXX True ``` If `torch` import fails despite pip saying "already satisfied", force-reinstall all three together (never upgrade `torch` alone — breaks `torchvision`): ```powershell pip install --force-reinstall torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118 ``` If `.venv\Scripts\meshnet-node.exe` shadows the conda binary, use the full path: `C:\Users\\miniforge3\Scripts\meshnet-node.exe`.
> Run Linux/WSL commands from **WSL**, not Git Bash. From Git Bash: `wsl`, then `cd` > to the repo under `/mnt/d/...`. ### Tracker host — lightweight install Tracker + relay only; skip node packages unless this machine also runs nodes. ```bash git clone https://git.d-popov.com/popov/neuron-tai.git AI && cd AI python3 -m venv .venv .venv/bin/python -m pip install --upgrade pip setuptools wheel .venv/bin/pip install -e packages/tracker -e packages/relay -e packages/gateway ``` ### PyTorch variant Install **one** torch line into the same env as `meshnet-node`: | Hardware | Install | |----------|---------| | NVIDIA CUDA | `pip install torch` (default index) | | CPU only | `pip install torch --index-url https://download.pytorch.org/whl/cpu` | | AMD ROCm (discrete, arch in official wheels) | `pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm6.3` | | AMD Strix Halo / Ryzen AI Max (gfx1151) | `pip install torch torchvision torchaudio --index-url https://rocm.nightlies.amd.com/v2/gfx1151/` | On Windows `.venv`, prefix with `.\.venv\Scripts\pip.exe`. Conda users with CUDA torch already installed can skip this step. ### Linux AMD ROCm GPU install Use this when the node machine has an AMD GPU/APU and you want PyTorch to run on ROCm instead of CPU. The Python wheel is not enough by itself: the host must have working AMD GPU device access and a compatible ROCm runtime. **Host prerequisites:** 1. Confirm the AMD GPU is visible: ```bash lspci | grep -Ei 'vga|3d|display|amd|ati' ls -l /dev/kfd /dev/dri/renderD* 2>/dev/null ``` 2. Make sure the node user can access GPU devices. AMD ROCm documents the normal Linux permission path as membership in both `video` and `render`: ```bash groups sudo usermod -a -G video,render "$LOGNAME" # log out and back in before continuing ``` 3. Confirm the ROCm runtime tools work if they are installed: ```bash rocminfo | head ``` If `rocminfo` is missing or cannot see the GPU, fix the host ROCm install first. Do not debug `meshnet-node` until this works. **Install ROCm PyTorch into the node env:** ```bash cd /path/to/neuron-tai python3.12 -m venv .venv-rocm source .venv-rocm/bin/activate python -m pip install --upgrade pip setuptools wheel python -m pip install -e packages/tracker -e packages/node -e packages/p2p -e packages/gateway -e packages/relay python -m pip install "transformers>=5.12" accelerate safetensors python -m pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm6.3 ``` **Strix Halo / Ryzen AI Max APUs (gfx1151, e.g. Radeon 8060S):** the official `download.pytorch.org` ROCm wheels do NOT ship gfx1151 kernels — every GPU op fails with `HIP error: invalid device function` and `HSA_OVERRIDE_GFX_VERSION` does not help. Install AMD's gfx1151-native builds instead (TheRock nightlies, self-contained, no system ROCm required): ```bash python -m pip install torch torchvision torchaudio --index-url https://rocm.nightlies.amd.com/v2/gfx1151/ ``` Check that your arch is actually in the wheel: `python -c "import torch; print(torch.cuda.get_arch_list())"` must list your GPU's `gcnArchName` (from `torch.cuda.get_device_properties(0)`). Keep this separate from a known-good CPU `.venv` until ROCm is verified on that machine. ROCm wheels are large and host-runtime-sensitive; a failed ROCm install should not break the CPU fallback environment. Use Python 3.12 for this env. Python 3.14 is currently a bad fit for the Qwen3.6/FLA path because `torch.compile` is not supported there. **Verify PyTorch sees ROCm:** ```bash python - <<'PY' import torch print("torch", torch.__version__) print("hip", torch.version.hip) print("cuda api available", torch.cuda.is_available()) if torch.cuda.is_available(): print("device", torch.cuda.get_device_name(0)) x = torch.ones((1,), device="cuda") torch.cuda.synchronize() print("tensor", x) PY ``` Expected: `torch.version.hip` is not `None`, `torch.cuda.is_available()` is `True`, and the tensor allocation succeeds. PyTorch intentionally exposes ROCm through the `torch.cuda` API. **Start an AMD ROCm node:** ```bash HF_HOME=/path/to/models .venv-rocm/bin/meshnet-node start \ --tracker \ --model Qwen/Qwen2.5-0.5B-Instruct \ --quantization bfloat16 ``` A model whose recipe uses an optional accelerator needs that package in the *same* env. For example, `qwen3.6-35b-a3b` on Linux ROCm can use FLA: ```bash .venv-rocm/bin/pip install 'flash-linear-attention[rocm]' HF_HOME=/path/to/models .venv-rocm/bin/meshnet-node start \ --tracker \ --model qwen3.6-35b-a3b \ --quantization bfloat16 ``` This is a per-model, per-platform example. A successful `pip install` is not evidence that the kernel runs on your GPU — optional-kernel support varies by architecture, and we make no universal claim here. Prove it with `.venv-rocm/bin/meshnet-node doctor --model ` before starting the node; startup will refuse to register anyway if the shard cannot execute. ### Linux ROCm: Triton JIT compiler prerequisite Some model/runtime paths invoke Triton at the first real forward. Triton builds a local HIP support module before that kernel can run, so a host C compiler must be discoverable on `PATH`. A successful PyTorch allocation or Node startup does not prove this prerequisite; without it, the first `/forward` can fail with `Failed to find C compiler`. On Fedora, install Clang once: ```bash sudo dnf install -y clang ``` Restart the Node from a shell where `clang` is on `PATH`. If a custom shell or service does not inherit the system path, set the compiler explicitly for that launch: ```bash CC=/usr/bin/clang CXX=/usr/bin/clang++ \ HF_HOME=/path/to/models .venv-rocm/bin/meshnet-node start \ --tracker --model ``` `CC`/`CXX` are normally unnecessary after Clang is installed; they are a diagnostic override, not a Python dependency. Other Linux distributions should install their system `clang` package through the OS package manager. **Windows NVIDIA/Triton:** use native PowerShell and install `triton-windows`, then install or upgrade `flash-linear-attention` when the selected model uses it. `triton-windows` supplies the supported Windows compiler path; do not apply Linux `dnf`/`CC` instructions to Windows. If a Windows Node still reports a compiler error, capture `python -c "import triton; print(triton.__version__)"` and the exact error before installing arbitrary CUDA toolkits or `causal-conv1d`. **Troubleshooting notes:** - `torch.version.hip is None` means you installed a CPU/CUDA torch build, not ROCm. - `torch.cuda.is_available() == False` with a ROCm build usually means host driver, permissions, unsupported hardware, or missing runtime libraries. - `which meshnet-node` should not point at `~/.local/bin/meshnet-node` for ROCm testing. Run `.venv-rocm/bin/meshnet-node ...` so the node uses the same ROCm PyTorch, `transformers`, and FLA packages you verified. - Missing libraries such as `libamdhip64.so`, `libMIOpen.so`, `librocsolver.so`, or `libroctx64.so` are host ROCm runtime problems, not meshnet-node problems. - Some AMD APUs and consumer GPUs require newer ROCm/Radeon support than server Instinct cards. Check AMD's ROCm Radeon/Ryzen support matrix for the exact model. - `HIP error: invalid device function` (or `no kernel image is available`) on a working driver means the installed torch wheel has no kernels compiled for your GPU arch. Compare `torch.cuda.get_device_properties(0).gcnArchName` against `torch.cuda.get_arch_list()`; if your arch is missing, install a wheel built for it (see the Strix Halo/gfx1151 note above). - `Failed to find C compiler` during a real forward is a Triton JIT host-toolchain failure. On Fedora install `clang` as shown above, confirm `command -v clang`, and restart the Node; it is separate from ROCm driver and device-access troubleshooting. ### Qwen3.5/3.6-MoE notes Applies to **`qwen3.6-35b-a3b`** and other hybrid linear-attention models. **`Qwen2.5-0.5B`** does not need any of this — it is a standard transformer with no FLA fast path. - **transformers ≥ 5.12 required** — older versions fail with `'Qwen3_5MoeConfig' object has no attribute 'vocab_size'`. Check: `python -c "import transformers; print(transformers.__version__)"`. - **GPU fast path (optional)** — without it inference still works; startup prints `The fast path is not available…` and linear-attention layers use a slower PyTorch fallback. **The fast path runs on NVIDIA CUDA GPUs on both Linux and native Windows** — the FLA kernels are Triton-compiled, and `triton-windows` compiles them for CUDA on Windows just like Linux Triton does. Only the pip command differs per platform. Install **only for your platform**: | Platform | Install | Notes | |----------|---------|-------| | **Native Windows + NVIDIA CUDA** | `pip install triton-windows` then `pip install flash-linear-attention` | **Fast path works on the CUDA GPU** — no CUDA toolkit / `nvcc` needed; `triton-windows` bundles its own compiler. FLA [officially supports `triton-windows`](https://github.com/fla-org/flash-linear-attention/pull/757) (tested Win11, PyTorch 2.10, triton-windows 3.6). Do **not** use the `[cuda]` extra on Windows — that extra only pins Linux PyPI `triton` and fails; it is a packaging name, not a GPU requirement. Do **not** install `causal-conv1d` — FLA ≥0.3.2 ships Triton conv1d; the separate package is Linux-only and breaks on Windows (`bare_metal_version` / nvcc errors). | | **Linux + NVIDIA CUDA** | `pip install flash-linear-attention[cuda]` | `causal-conv1d` optional (same FLA built-in conv1d note). Needs CUDA toolkit (`nvcc`) matching torch, or a prebuilt wheel. | | **Linux + AMD ROCm** | `pip install flash-linear-attention[rocm]` | Same optional `causal-conv1d` note. | **Windows verify** (after install): ```powershell python -c "import triton; import fla; print('triton', triton.__version__, 'fla ok')" ``` `triton-windows` is also pulled by `meshnet-node` on Windows. Without it, Qwen3.6-MoE startup fails with misleading `Could not import module 'Qwen3_5MoeForCausalLM'`.
Windows fast path — what failed and what actually works The command that failed — `pip install flash-linear-attention[cuda] causal-conv1d` — mixes two different things: 1. **`flash-linear-attention[cuda]` on Windows** — wrong extra. `[cuda]` pulls PyPI `triton>=3.3`, which does not exist for Windows (`No matching distribution found`). Use plain `pip install flash-linear-attention` **after** `triton-windows` is already installed; FLA detects `triton-windows` and uses it. 2. **`causal-conv1d`** — separate Dao-AILab CUDA extension, **not required** for FLA or Qwen3.6 when FLA is installed. No official Windows wheels. Source builds need `nvcc` whose major version matches torch's CUDA (e.g. torch `+cu118` needs CUDA 11.8 toolkit, not 12.5). Community wheels exist for narrow Python/torch combos ([PR #46](https://github.com/Dao-AILab/causal-conv1d/pull/46)) but we skip them. **Working Windows stack** (confirmed on this repo's dev machine: Python 3.12, torch 2.7.1+cu118, triton-windows 3.7.1, flash-linear-attention 0.5.0): ```powershell pip install triton-windows pip install -U flash-linear-attention python -c "import triton; import fla; print('ok')" ``` If the fast-path warning persists after that, upgrade FLA to ≥0.5.1 (includes the `triton-windows` detection from PR #757) and restart the node.
- `pip install nvidia-ml-py` silences the pynvml deprecation warning on NVIDIA hosts. --- ## 1. Start the tracker ### LAN tracker (private network, direct node reachability) **Terminal 1:** ```bash .venv/bin/meshnet-tracker start --host 0.0.0.0 --port 8080 # Optional devnet billing: --starting-credit 1 --devnet-topup 10 ``` Expected: `Tracker listening on 0.0.0.0:8080`. Open the port on the host firewall if other machines will join. **Verify:** ```bash curl -s http://localhost:8080/v1/network/map | python3 -m json.tool curl -s http://192.168.0.179:8080/v1/network/map | python3 -m json.tool # from another LAN machine ``` ### Public tracker + relay (NAT / WSL2 / internet nodes) Nodes behind NAT cannot receive inbound connections. Keep the relay as its own component, but for alpha deployments you can run it **embedded in the tracker process**. This still uses the same `meshnet_relay.RelayServer` class as a relay-only node, so relay-only hosts remain a clean scaling path later. **Recommended alpha: one process on the tracker host, relay bound locally and published through the same public hostname:** ```bash .venv/bin/meshnet-tracker start \ --host 0.0.0.0 --port 8081 \ --self-url https://ai.neuron.d-popov.com \ --embedded-relay --relay-host 127.0.0.1 --relay-port 8765 ``` The tracker advertises `wss://ai.neuron.d-popov.com/ws` in `/v1/network/map`. If you want a relay-only process instead, keep running: ```bash .venv/bin/meshnet-relay --host 0.0.0.0 --port 8765 .venv/bin/meshnet-tracker start --host 0.0.0.0 --port 8081 --relay-url wss://ai.neuron.d-popov.com/ws ``` **Verify:** ```bash curl -s https://ai.neuron.d-popov.com/v1/network/map | python3 -m json.tool ``` Nodes should log `Relay connected — wss://…/rpc/` on startup.
Nginx Proxy Manager setup (public hostname) Architecture: ``` Client → HTTPS → ai.neuron.d-popov.com (nginx) ├─ /v1/* → meshnet-tracker :8081 ├─ /ws → meshnet-relay :8765 (node persistent outbound WS) └─ /rpc/* → meshnet-relay :8765 (caller opens WS per hop) ``` Use **one** proxy host. Route sub-paths via **Custom locations** — do not create a second host for the same domain. **Details tab** (default `/` → tracker): | Field | Value | |-------|--------| | Domain Names | `ai.neuron.d-popov.com` | | Scheme | `http` | | Forward Hostname / IP | LAN IP of tracker machine (e.g. `192.168.0.179`) | | Forward Port | `8081` | | Websockets Support | ON | **Custom locations** (both → relay port `8765`, sub-folder path empty): | Location | Forward to | |----------|--------------| | `/ws` | `192.168.0.179:8765` | | `/rpc` | `192.168.0.179:8765` | **Advanced tab** (only if WebSocket upgrade fails): ```nginx proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection $http_connection; proxy_read_timeout 3600s; proxy_send_timeout 3600s; ``` **Verify routing:** ```bash curl -s https://ai.neuron.d-popov.com/v1/network/map | python3 -m json.tool curl -sI https://ai.neuron.d-popov.com/ws curl -sI https://ai.neuron.d-popov.com/rpc/test-peer ```
--- ## 2. Start a node **Starter model:** `Qwen/Qwen2.5-0.5B-Instruct` — 0.5B params, ~1 GB BF16, 24 layers, no HuggingFace gating. Best for first-time setup. **Alpha model:** `qwen3.6-35b-a3b` — 40 layers, ~72 GB BF16 download, MoE with hybrid linear attention. Tracker accepts the alias or full repo id (`unsloth/Qwen3.6-35B-A3B`). Some models have an *optional* accelerator path (for this one, Triton + `flash-linear-attention`); those installs are per-model, per-platform examples, not a requirement the node imposes and not a guarantee the path will work — see [Optional accelerator packages](#optional-accelerator-packages). Downloads cache under `~/.meshnet/models/` (or `$HF_HOME` / `$env:HF_HOME`). Shard range is auto-detected from the curated catalog. For unknown repos the node fetches only `config.json`. Override with `--shard-start` / `--shard-end` for partial shards or multi-node splits. ### Validate before you serve: `meshnet-node doctor` `doctor` resolves exactly the model, shard, quantization and recipe that `start` would load, loads it, and pushes one bounded **real forward** through it. It is offline — it never contacts the tracker — and it is the only thing that turns a guess into evidence. ```bash HF_HOME=/path/to/models .venv/bin/meshnet-node doctor --model --quantization bfloat16 ``` ``` meshnet-node doctor Model: Shard: layers 0–23; 24 of 24 Quantization: bfloat16 [PASS] recipe default (v1) on cuda — 412 ms OK — the selected shard ran a real forward for 1 recipe. Capability report: ~/.meshnet/capability.json ``` Exit code is 0 on pass, 1 on failure. It writes a **capability report** either way — a failure is evidence too. Useful flags: `--recipe ` to validate one named recipe, `--all-recipes` to try every recipe for the selection (CI and diagnosis), `--report PATH` to choose where the report lands, `--json` for machine-readable output, `--debug` for the full traceback behind a failure. The selection flags (`--model`, `--quantization`, `--shard-start`/`--shard-end`, `--cpu`) work the same as on `start`. **Three states — do not confuse them:** | State | What it means | How it is established | |-------|---------------|-----------------------| | **Detected hardware** | A GPU, a torch build, and an optional package are *present*. | Inventory/`torch.cuda.is_available()`. Proves nothing about whether your shard runs. | | **Validated recipe** | This machine executed a real forward for *this* model + shard + recipe + device. | A passing `doctor` run → a capability report. | | **Routable Node** | The tracker will send paid work here. | Startup re-proves the loaded shard and registers the report; the tracker admits it. | Each state is strictly stronger than the one above it. A machine can have a working GPU (detected) and still fail to execute a shard (no validated recipe), and a node can hold a valid report and still not route (the tracker moved it to a shard range it never proved). **Startup is fail-closed.** `meshnet-node start` runs the same validation against the backend it just loaded, *before* it opens a port or registers. If the shard cannot execute, the node exits non-zero — it never binds a socket and never registers. There is no production bypass; a node that cannot do the work never advertises that it can. **The tracker admits, it does not re-run.** The node ships its report with registration. Registration always succeeds, so a broken node is visible rather than invisible — but only a report that *covers what the node advertises* makes it routable. The verdict comes back in the registration response, is logged, and is exposed per node on `GET /v1/network/map` under `capability` (`admitted`, `absent`, `invalid`, `failed`, `stale`, `model-mismatch`, `shard-mismatch`, `recipe-mismatch`, `catalogue-incompatible`). A node that is registered but dark is showing you one of those. Full semantics, and the transitional `compat` vs `enforce` policy, are in [ADR-0023](docs/adr/0023-model-agnostic-node-capability-admission.md). **When `doctor` fails** it prints a category and an actionable hint, not a traceback: | Category | Do this | |----------|---------| | `no-model-selected` | Pass `--model`, or run `meshnet-node` once to save a config. | | `missing-dependency` | Install the node's model extras (torch, transformers, safetensors, accelerate). | | `model-unavailable` | Check the model id, `--download-dir`, and that the artifact is downloaded/reachable. | | `insufficient-memory` | Serve fewer layers (`--shard-start`/`--shard-end`) or a smaller quantization (`-q int8`, `-q nf4`). | | `invalid-shard` | The layer range does not exist in this model — check it against the layer count. | | `unsupported-recipe` | This backend cannot apply that execution setting; pick another `--recipe`. | | `load-failed` | The shard would not load; re-run with `--debug`. | | `forward-failed` | It loaded but cannot execute. This machine cannot serve this shard; re-run with `--debug`. | **Scope:** the node validates recipes it already ships. It does **not** download executable recipes, install Python/OS packages, or update drivers for you — every install on this page is something you run deliberately. Signed node updates are a deliberate follow-up, not part of this release. ### Core command Replace `` and adjust the prefix for your shell (see table above). **Linux / WSL:** ```bash HF_HOME=/path/to/models .venv/bin/meshnet-node start --tracker --model Qwen/Qwen2.5-0.5B-Instruct --quantization bfloat16 ``` **Windows PowerShell:** ```powershell $env:HF_HOME = "D:\DEV\models" .\.venv\Scripts\meshnet-node.exe start --tracker --model Qwen/Qwen2.5-0.5B-Instruct --quantization bfloat16 ``` ### Ready-to-run examples **Local dev (same machine as tracker):** ```bash HF_HOME=/path/to/models .venv/bin/meshnet-node start --tracker http://localhost:8080 --model Qwen/Qwen2.5-0.5B-Instruct --quantization bfloat16 --port 8001 ``` **Public / relay (works from WSL2, NAT, 5G — no extra flags):** ```bash .venv/bin/meshnet-node start --tracker https://ai.neuron.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct ``` #### Optional accelerator packages Some models have an optional fast path that a *specific* model's recipe can use if the package is importable. The examples below are exactly that — **worked examples for one development model on one platform**, not a supported-configuration list. Installing the package does not mean the fast path works on your GPU: a package can import cleanly and still fail to execute a kernel on your architecture. `doctor` is what tells you, and the node serves the recipe that actually validated. If none of this is installed, the model still runs on the standard path. **Example — `qwen3.6-35b-a3b` (Triton + FLA), Windows GPU:** ```powershell $env:HF_HOME = "D:\DEV\models" pip install triton-windows pip install -U flash-linear-attention meshnet-node doctor --model qwen3.6-35b-a3b --quantization bfloat16 meshnet-node start --tracker http://192.168.0.179:8080 --model qwen3.6-35b-a3b --quantization bfloat16 ``` Do not add `causal-conv1d` or `flash-linear-attention[cuda]` on Windows (see Qwen3.5/3.6 notes). **Example — `qwen3.6-35b-a3b`, Linux NVIDIA GPU:** ```bash # Install once on that machine: pip install flash-linear-attention[cuda] HF_HOME=/path/to/models .venv/bin/meshnet-node start --tracker --model qwen3.6-35b-a3b --quantization bfloat16 ``` **Example — `qwen3.6-35b-a3b`, Linux AMD ROCm GPU:** ```bash # Install once on that machine: .venv-rocm/bin/pip install 'flash-linear-attention[rocm]' HF_HOME=/path/to/models .venv-rocm/bin/meshnet-node start --tracker --model qwen3.6-35b-a3b --quantization bfloat16 ``` Run `doctor` after any of these installs: it is the only way to learn whether the optional path executes here, and it costs one bounded forward. After the first node registers a model, later nodes can join with only the tracker URL (shard auto-assigned): ```bash .venv/bin/meshnet-node start --tracker https://ai.neuron.d-popov.com ``` **Direct LAN (Windows node reachable by IP):** ```powershell $env:HF_HOME = "D:\DEV\models" .\.venv\Scripts\meshnet-node.exe start --tracker http://192.168.0.179:8081 --model Qwen/Qwen2.5-0.5B-Instruct --shard-start 12 --shard-end 23 --quantization bfloat16 --host 0.0.0.0 --advertise-host 192.168.0.42 --port 8005 ```
Windows direct LAN — firewall and IP checklist 1. Find LAN IP: `ipconfig` — use active Ethernet/Wi-Fi IPv4 (e.g. `192.168.0.42`). Avoid WSL/Docker/Hyper-V addresses (`172.x.x.x`). 2. Allow inbound port (Administrator PowerShell, once): ```powershell New-NetFirewallRule -DisplayName "Meshnet node 8005" -Direction Inbound -Action Allow -Protocol TCP -LocalPort 8005 ``` 3. Verify from tracker machine: ```bash curl http://192.168.0.42:8005/v1/health ``` 404/501 is fine — it proves TCP reached the node. Timeout = check firewall, `--host 0.0.0.0`, and `--advertise-host`.
Two-node split (same or different machines) **Node A — layers 0–11 (head, serves chat):** ```bash HF_HOME=/path/to/models .venv/bin/meshnet-node start --tracker --model Qwen/Qwen2.5-0.5B-Instruct --shard-start 0 --shard-end 11 --quantization bfloat16 --port 8001 ``` **Node B — layers 12–23:** ```bash HF_HOME=/path/to/models .venv/bin/meshnet-node start --tracker --model Qwen/Qwen2.5-0.5B-Instruct --shard-start 12 --shard-end 23 --quantization bfloat16 --port 8002 ``` Send inference to Node A. For cross-machine LAN tests see `docs/TWO_MACHINE_TEST.md`.
### Useful flags | Flag | Purpose | |------|---------| | `--port 8001` | Fixed listen port (default: first free ≥ 7000) | | `--host 0.0.0.0` | Bind all interfaces (needed for direct LAN) | | `--advertise-host ` | LAN IP the tracker tells other nodes (direct LAN only) | | `--shard-start N --shard-end M` | Partial layer range | | `--debug` | Verbose per-hop pipeline logs (noisy; off by default) | `--host 0.0.0.0` binds locally; `--advertise-host` is what peers use for direct hops. Omit both when using the relay path. ### Expected output ``` Auto-detected 24 layers → shard 0–23 Relay connected — wss://ai.neuron.d-popov.com/rpc/abc1def2ef3f4567 # relay mode only ================================ meshnet-node ready Wallet:
Model ID: Qwen/Qwen2.5-0.5B-Instruct Shard: layers 0–23; 24 of 24 Quantization: bfloat16 Endpoint: http://:8001 Node ID: Hardware: CPU ================================ ``` The `Endpoint` is the local address. In relay mode, peers reach this node via `wss:///rpc/` instead. ### Other CPU-friendly models | Model | `--model` / alias | Layers | BF16 size | Notes | |-------|-------------------|--------|-----------|-------| | **Qwen2.5-0.5B** (dev default) | `Qwen/Qwen2.5-0.5B-Instruct` | 24 | ~1 GB | Fastest, no gating | | **Qwen3.6-35B-A3B** (alpha) | `qwen3.6-35b-a3b` | 40 | ~72 GB | MoE; needs transformers ≥5.12; see Qwen3.5/3.6 notes | | Qwen2.5-1.5B | `Qwen/Qwen2.5-1.5B-Instruct` | 28 | ~3 GB | Better quality | | Phi-3-mini | `microsoft/Phi-3-mini-4k-instruct` | 32 | ~7.5 GB | Best CPU quality | | Llama-3.2-1B | `meta-llama/Llama-3.2-1B-Instruct` | 16 | ~2 GB | Requires HF login | | Llama-3.2-3B | `meta-llama/Llama-3.2-3B-Instruct` | 28 | ~6 GB | Requires HF login | For gated models (Llama): `huggingface-cli login` first. Browse more: `.venv/bin/meshnet-node models` or `.venv/bin/meshnet-node models --browse`. --- ## 3. Send an inference request **Terminal 3** — direct to the head node (replace port if you omitted `--port`): ```bash curl -s http://localhost:8001/v1/chat/completions -H "Content-Type: application/json" -d '{"model": "qwen2.5-0.5b", "messages": [{"role": "user", "content": "What is 7 times 8? Answer in one word."}], "stream": false}' | python3 -m json.tool ``` **Via tracker** (tests routing / proxying): ```bash curl -s http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{"model": "qwen2.5-0.5b", "messages": [{"role": "user", "content": "What is 7 times 8? Answer in one word."}], "stream": false}' | python3 -m json.tool ``` **Public tracker:** ```bash curl -s https://ai.neuron.d-popov.com/v1/chat/completions -H "Content-Type: application/json" -H "Authorization: Bearer sk-mesh-" -d '{"model": "Qwen/Qwen2.5-0.5B-Instruct", "messages": [{"role": "user", "content": "What is 7 times 8?"}], "stream": false}' | python3 -m json.tool ``` **Test script:** ```bash .venv/bin/python scripts/test_lan_inference.py --tracker http://localhost:8080 --gateway http://localhost:8001 ``` **Verify registration on public tracker:** ```bash curl -s "https://ai.neuron.d-popov.com/v1/network/map" | python3 -m json.tool curl -s "https://ai.neuron.d-popov.com/v1/route?model=qwen2.5-0.5b" | python3 -m json.tool ```
Accounts, API keys, and credit (billing-enabled trackers) Public trackers require a real API key for `/v1/chat/completions`. Unknown bearer → `401`; zero balance → `402 insufficient balance`. **Dashboard:** open `https:///dashboard`, register, click **+ new key**. With `--starting-credit`, the first key is pre-funded. With `--devnet-topup`, use **+$N (devnet)** to refill during testing. **Curl flow:** ```bash curl -s https:///v1/auth/register -H "Content-Type: application/json" -d '{"email": "you@example.com", "password": "hunter22-or-better"}' curl -s https:///v1/account/keys -X POST -H "Authorization: Bearer " curl -s https:///v1/account -H "Authorization: Bearer " curl -s https:///v1/account/topup -X POST -H "Authorization: Bearer " -H "Content-Type: application/json" -d '{"api_key": "sk-mesh-..."}' ``` Operator defaults: `--starting-credit` and `--devnet-topup` both default to 1 USDT. Set both to 0 on mainnet.
--- ## Reference ### How relay hops work When node A forwards activations to node B (behind NAT): 1. Tracker injects `X-Meshnet-Route` with `relay_addr` for behind-NAT hops. 2. Node A opens WebSocket to `wss://relay/rpc/{peer_id_B}`. 3. Relay forwards `relay-http-request` to Node B's persistent connection. 4. Node B processes `/forward`, returns `relay-http-response`. 5. Relay sends response back to Node A; Node A continues the pipeline. Activations (bfloat16) are Base64-encoded in JSON — no precision loss. On relay failure the node logs a warning and falls back to direct HTTP before erroring. ### Interactive wizard ```bash .venv/bin/meshnet-node # first run: wizard; later: saved config .venv/bin/meshnet-node --reset-config ``` ### Run all tests ```bash .venv/bin/python -m pytest -q ``` Tests marked `integration` download models or need a GPU; the default lane is `pytest -m "not integration"`. The opt-in real-model doctor check takes its model from the environment and skips without it — see [docs/dev/certified-hardware-lanes.md](docs/dev/certified-hardware-lanes.md) for the release contract that certifies a hardware lane.