neuron-tai/QUICKSTART.md

Quickstart — Running a node and testing inference

Get from zero to a live inference request in three terminals: install once, start the tracker, start a node, send a request.

Tested on: AMD Ryzen AI Max (Strix Halo APU), 124 GB RAM, Linux CPU inference. ROCm GPU setup is covered below, but must be verified on the host because ROCm support depends on the exact AMD GPU/APU, kernel, driver, and ROCm runtime.

Active development models (what we run day-to-day):

Role	--model / alias	HF repo	Notes
Smoke tests, small splits	Qwen/Qwen2.5-0.5B-Instruct	same	24 layers, ~1 GB BF16, no gating — default for new setups
Alpha / production target	qwen3.6-35b-a3b	unsloth/Qwen3.6-35B-A3B	40 layers, ~72 GB BF16, hybrid linear-attention MoE; aliases include Qwen3.6-35B-A3B, Qwen/Qwen3.6-35B-A3B
Recommended dense model	qwen3.6-27b	Qwen/Qwen3.6-27B	64 layers, ~56 GB BF16, text-only; canonical revision is pinned by the tracker

---

Models

The tracker advertises recommended models through GET /v1/models; the chat dashboard shows the same catalog. Qwen/Qwen3.6-27B is recommended even before it has complete coverage, so users and operators can see that the network intends to serve it.

curl -s http://localhost:8080/v1/models | python -m json.tool

Each model reports its shard coverage and its selectable quantizations. A quantization is selectable only when the tracker can build a complete route from shards at that precision or higher:

• bfloat16 requires BF16 for every shard.
• int8 may combine INT8 and BF16 shards.
• nf4 may combine NF4, INT8, and BF16 shards.

The chat UI chooses the highest complete precision by default. Uncovered variants remain visible as coverage requests: use the small Vote for coverage control to register demand without sending an unusable chat request.

Clients can request a minimum precision explicitly. Omit quantization to request BF16:

curl http://localhost:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen3.6-27b",
    "quantization": "int8",
    "messages": [{"role": "user", "content": "Explain sharded inference."}]
  }'

The first valid request for a recommended model is demand proof. When the shared tracker pool has enough spare eligible capacity, it queues a tracker-managed initial deployment. Until a complete route exists, the request returns retryable 503 model_loading rather than silently lowering precision.

Nodes may serve BF16, INT8, or NF4 according to their declared capability. The official BF16 snapshot remains the canonical Qwen source; tracker/peer sources are preferred and Hugging Face is the fallback. A node's startup-selected (model, shard range, quantization) is pinned, while spare capacity can be used for tracker-managed work.

---

At a glance

Step	What	Terminals
0	Install Python packages	once per machine
1	Start tracker (and relay if needed)	1–2
2	Start node(s)	1+
3	Send inference request	1

Pick your connectivity mode — this determines which flags you need on the node:

Mode	When to use	Tracker URL	Node extras
Local dev	Everything on one machine	http://localhost:8080	none
Direct LAN	Node has a real LAN IP other machines can reach	http://<tracker-ip>:8080	--host 0.0.0.0 --advertise-host <your-lan-ip> + firewall
Relay / public	WSL2, NAT, 5G, or any unreachable inbound port	https://ai.neuron.d-popov.com (or your public URL)	none — relay handles routing

WSL2: not reachable from other LAN machines by default. Use the relay / public tracker URL, or run the node in native Windows PowerShell with direct LAN mode.

Command prefix by shell (used in examples below):

Shell	Prefix	Model cache env
Linux / WSL CPU	.venv/bin/	HF_HOME=/path/to/models
Linux AMD ROCm / Radeon	.venv-rocm/bin/	HF_HOME=/path/to/models
Windows PowerShell	.\.venv\Scripts\	$env:HF_HOME = "D:\DEV\models"

Ryzen AI Max / Radeon 8060S developers: use .venv-rocm/bin/ for every node command and test that needs real GPU inference. The repository's default .venv currently uses Python 3.14 and is not the ROCm node runtime.

---

0. Install prerequisites (once per machine)

Editable installs point wrappers at this source tree — code edits apply without reinstalling.

Node machine — full install

Linux / WSL

cd /path/to/neuron-tai
python3 -m venv .venv
source .venv/bin/activate
.venv/bin/python -m pip install --upgrade pip setuptools wheel
.venv/bin/python -m pip install -e packages/tracker -e packages/node -e packages/p2p -e packages/gateway -e packages/relay
.venv/bin/python -m pip install "transformers>=5.12" accelerate
.venv/bin/meshnet-node --help

Windows PowerShell (.venv)

Requires Python 3.11+ and Git for Windows.

cd D:\DEV\workspace\REPOS\git.d-popov.com\neuron-tai
python -m venv .venv
.\.venv\Scripts\Activate.ps1
.\.venv\Scripts\python.exe -m pip install --upgrade pip setuptools wheel
.\.venv\Scripts\python.exe -m pip install -e .\packages\tracker -e .\packages\node -e .\packages\p2p -e .\packages\gateway -e .\packages\relay
.\.venv\Scripts\python.exe -m pip install "transformers>=5.12" accelerate
.\.venv\Scripts\meshnet-node.exe --help

Windows — conda/miniforge with CUDA (skip if using .venv above)

conda activate base
deactivate   # drop any layered .venv; safe no-op if none active
cd D:\DEV\workspace\REPOS\git.d-popov.com\neuron-tai
pip install -e packages\tracker -e packages\node -e packages\p2p -e packages\gateway -e packages\relay
pip install "transformers>=5.12" accelerate safetensors
python -c "import torch; print(torch.__version__, torch.cuda.is_available())"
# Expected: 2.x.x+cuXXX  True

If torch import fails despite pip saying "already satisfied", force-reinstall all three together (never upgrade torch alone — breaks torchvision):

pip install --force-reinstall torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

If .venv\Scripts\meshnet-node.exe shadows the conda binary, use the full path: C:\Users\<you>\miniforge3\Scripts\meshnet-node.exe.

Run Linux/WSL commands from WSL, not Git Bash. From Git Bash: wsl, then cd to the repo under /mnt/d/....

Tracker host — lightweight install

Tracker + relay only; skip node packages unless this machine also runs nodes.

git clone https://git.d-popov.com/popov/neuron-tai.git AI && cd AI
python3 -m venv .venv
.venv/bin/python -m pip install --upgrade pip setuptools wheel
.venv/bin/pip install -e packages/tracker -e packages/relay -e packages/gateway

PyTorch variant

Install one torch line into the same env as meshnet-node:

Hardware	Install
NVIDIA CUDA	pip install torch (default index)
CPU only	pip install torch --index-url https://download.pytorch.org/whl/cpu
AMD ROCm (discrete, arch in official wheels)	pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm6.3
AMD Strix Halo / Ryzen AI Max (gfx1151)	pip install torch torchvision torchaudio --index-url https://rocm.nightlies.amd.com/v2/gfx1151/

On Windows .venv, prefix with .\.venv\Scripts\pip.exe. Conda users with CUDA torch already installed can skip this step.

Linux AMD ROCm GPU install

Use this when the node machine has an AMD GPU/APU and you want PyTorch to run on ROCm instead of CPU. The Python wheel is not enough by itself: the host must have working AMD GPU device access and a compatible ROCm runtime.

Host prerequisites:

1. Confirm the AMD GPU is visible:

lspci | grep -Ei 'vga|3d|display|amd|ati'
ls -l /dev/kfd /dev/dri/renderD* 2>/dev/null

2. Make sure the node user can access GPU devices. AMD ROCm documents the normal Linux permission path as membership in both video and render:

groups
sudo usermod -a -G video,render "$LOGNAME"
# log out and back in before continuing

3. Confirm the ROCm runtime tools work if they are installed:

rocminfo | head

If rocminfo is missing or cannot see the GPU, fix the host ROCm install first. Do not debug meshnet-node until this works.

Install ROCm PyTorch into the node env:

cd /path/to/neuron-tai
python3.12 -m venv .venv-rocm
source .venv-rocm/bin/activate
python -m pip install --upgrade pip setuptools wheel
python -m pip install -e packages/tracker -e packages/node -e packages/p2p -e packages/gateway -e packages/relay
python -m pip install "transformers>=5.12" accelerate safetensors
python -m pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm6.3

Strix Halo / Ryzen AI Max APUs (gfx1151, e.g. Radeon 8060S): the official download.pytorch.org ROCm wheels do NOT ship gfx1151 kernels — every GPU op fails with HIP error: invalid device function and HSA_OVERRIDE_GFX_VERSION does not help. Install AMD's gfx1151-native builds instead (TheRock nightlies, self-contained, no system ROCm required):

python -m pip install torch torchvision torchaudio --index-url https://rocm.nightlies.amd.com/v2/gfx1151/

Check that your arch is actually in the wheel: python -c "import torch; print(torch.cuda.get_arch_list())" must list your GPU's gcnArchName (from torch.cuda.get_device_properties(0)).

Keep this separate from a known-good CPU .venv until ROCm is verified on that machine. ROCm wheels are large and host-runtime-sensitive; a failed ROCm install should not break the CPU fallback environment.

Use Python 3.12 for this env. Python 3.14 is currently a bad fit for the Qwen3.6/FLA path because torch.compile is not supported there.

Verify PyTorch sees ROCm:

python - <<'PY'
import torch
print("torch", torch.__version__)
print("hip", torch.version.hip)
print("cuda api available", torch.cuda.is_available())
if torch.cuda.is_available():
    print("device", torch.cuda.get_device_name(0))
    x = torch.ones((1,), device="cuda")
    torch.cuda.synchronize()
    print("tensor", x)
PY

Expected: torch.version.hip is not None, torch.cuda.is_available() is True, and the tensor allocation succeeds. PyTorch intentionally exposes ROCm through the torch.cuda API.

Start an AMD ROCm node:

HF_HOME=/path/to/models .venv-rocm/bin/meshnet-node start \
  --tracker <tracker-url> \
  --model Qwen/Qwen2.5-0.5B-Instruct \
  --quantization bfloat16

For the Qwen3.6 alpha model on Linux ROCm, install the optional FLA ROCm fast path in the same env:

.venv-rocm/bin/pip install 'flash-linear-attention[rocm]'
HF_HOME=/path/to/models .venv-rocm/bin/meshnet-node start \
  --tracker <tracker-url> \
  --model qwen3.6-35b-a3b \
  --quantization bfloat16

Linux ROCm: Triton JIT compiler prerequisite

Some model/runtime paths invoke Triton at the first real forward. Triton builds a local HIP support module before that kernel can run, so a host C compiler must be discoverable on PATH. A successful PyTorch allocation or Node startup does not prove this prerequisite; without it, the first /forward can fail with Failed to find C compiler.

On Fedora, install Clang once:

sudo dnf install -y clang

Restart the Node from a shell where clang is on PATH. If a custom shell or service does not inherit the system path, set the compiler explicitly for that launch:

CC=/usr/bin/clang CXX=/usr/bin/clang++ \
  HF_HOME=/path/to/models .venv-rocm/bin/meshnet-node start \
  --tracker <tracker-url> --model <selected-model>

CC/CXX are normally unnecessary after Clang is installed; they are a diagnostic override, not a Python dependency. Other Linux distributions should install their system clang package through the OS package manager.

Windows NVIDIA/Triton: use native PowerShell and install triton-windows, then install or upgrade flash-linear-attention when the selected model uses it. triton-windows supplies the supported Windows compiler path; do not apply Linux dnf/CC instructions to Windows. If a Windows Node still reports a compiler error, capture python -c "import triton; print(triton.__version__)" and the exact error before installing arbitrary CUDA toolkits or causal-conv1d.

Troubleshooting notes:

• torch.version.hip is None means you installed a CPU/CUDA torch build, not ROCm.
• torch.cuda.is_available() == False with a ROCm build usually means host driver, permissions, unsupported hardware, or missing runtime libraries.
• which meshnet-node should not point at ~/.local/bin/meshnet-node for ROCm testing. Run .venv-rocm/bin/meshnet-node ... so the node uses the same ROCm PyTorch, transformers, and FLA packages you verified.
• Missing libraries such as libamdhip64.so, libMIOpen.so, librocsolver.so, or libroctx64.so are host ROCm runtime problems, not meshnet-node problems.
• Some AMD APUs and consumer GPUs require newer ROCm/Radeon support than server Instinct cards. Check AMD's ROCm Radeon/Ryzen support matrix for the exact model.
• HIP error: invalid device function (or no kernel image is available) on a working driver means the installed torch wheel has no kernels compiled for your GPU arch. Compare torch.cuda.get_device_properties(0).gcnArchName against torch.cuda.get_arch_list(); if your arch is missing, install a wheel built for it (see the Strix Halo/gfx1151 note above).
• Failed to find C compiler during a real forward is a Triton JIT host-toolchain failure. On Fedora install clang as shown above, confirm command -v clang, and restart the Node; it is separate from ROCm driver and device-access troubleshooting.

Qwen3.5/3.6-MoE notes

Applies to qwen3.6-35b-a3b and other hybrid linear-attention models. Qwen2.5-0.5B does not need any of this — it is a standard transformer with no FLA fast path.

• transformers ≥ 5.12 required — older versions fail with 'Qwen3_5MoeConfig' object has no attribute 'vocab_size'. Check: python -c "import transformers; print(transformers.__version__)".

• GPU fast path (optional) — without it inference still works; startup prints The fast path is not available… and linear-attention layers use a slower PyTorch fallback. The fast path runs on NVIDIA CUDA GPUs on both Linux and native Windows — the FLA kernels are Triton-compiled, and triton-windows compiles them for CUDA on Windows just like Linux Triton does. Only the pip command differs per platform. Install only for your platform:

  Platform	Install	Notes
  Native Windows + NVIDIA CUDA	pip install triton-windows then pip install flash-linear-attention	Fast path works on the CUDA GPU — no CUDA toolkit / nvcc needed; triton-windows bundles its own compiler. FLA officially supports triton-windows (tested Win11, PyTorch 2.10, triton-windows 3.6). Do not use the [cuda] extra on Windows — that extra only pins Linux PyPI triton and fails; it is a packaging name, not a GPU requirement. Do not install causal-conv1d — FLA ≥0.3.2 ships Triton conv1d; the separate package is Linux-only and breaks on Windows (bare_metal_version / nvcc errors).
  Linux + NVIDIA CUDA	pip install flash-linear-attention[cuda]	causal-conv1d optional (same FLA built-in conv1d note). Needs CUDA toolkit (nvcc) matching torch, or a prebuilt wheel.
  Linux + AMD ROCm	pip install flash-linear-attention[rocm]	Same optional causal-conv1d note.

  Windows verify (after install):

  python -c "import triton; import fla; print('triton', triton.__version__, 'fla ok')"
  

  triton-windows is also pulled by meshnet-node on Windows. Without it, Qwen3.6-MoE startup fails with misleading Could not import module 'Qwen3_5MoeForCausalLM'.

Windows fast path — what failed and what actually works

The command that failed — pip install flash-linear-attention[cuda] causal-conv1d — mixes two different things:

1. flash-linear-attention[cuda] on Windows — wrong extra. [cuda] pulls PyPI triton>=3.3, which does not exist for Windows (No matching distribution found). Use plain pip install flash-linear-attention after triton-windows is already installed; FLA detects triton-windows and uses it.
2. causal-conv1d — separate Dao-AILab CUDA extension, not required for FLA or Qwen3.6 when FLA is installed. No official Windows wheels. Source builds need nvcc whose major version matches torch's CUDA (e.g. torch +cu118 needs CUDA 11.8 toolkit, not 12.5). Community wheels exist for narrow Python/torch combos (PR #46) but we skip them.

Working Windows stack (confirmed on this repo's dev machine: Python 3.12, torch 2.7.1+cu118, triton-windows 3.7.1, flash-linear-attention 0.5.0):

pip install triton-windows
pip install -U flash-linear-attention
python -c "import triton; import fla; print('ok')"

If the fast-path warning persists after that, upgrade FLA to ≥0.5.1 (includes the triton-windows detection from PR #757) and restart the node.

• pip install nvidia-ml-py silences the pynvml deprecation warning on NVIDIA hosts.

---

1. Start the tracker

LAN tracker (private network, direct node reachability)

Terminal 1:

.venv/bin/meshnet-tracker start --host 0.0.0.0 --port 8080
# Optional devnet billing: --starting-credit 1 --devnet-topup 10

Expected: Tracker listening on 0.0.0.0:8080. Open the port on the host firewall if other machines will join.

Verify:

curl -s http://localhost:8080/v1/network/map | python3 -m json.tool
curl -s http://192.168.0.179:8080/v1/network/map | python3 -m json.tool   # from another LAN machine

Public tracker + relay (NAT / WSL2 / internet nodes)

Nodes behind NAT cannot receive inbound connections. Keep the relay as its own component, but for alpha deployments you can run it embedded in the tracker process. This still uses the same meshnet_relay.RelayServer class as a relay-only node, so relay-only hosts remain a clean scaling path later.

Recommended alpha: one process on the tracker host, relay bound locally and published through the same public hostname:

.venv/bin/meshnet-tracker start \
  --host 0.0.0.0 --port 8081 \
  --self-url https://ai.neuron.d-popov.com \
  --embedded-relay --relay-host 127.0.0.1 --relay-port 8765

The tracker advertises wss://ai.neuron.d-popov.com/ws in /v1/network/map. If you want a relay-only process instead, keep running:

.venv/bin/meshnet-relay --host 0.0.0.0 --port 8765
.venv/bin/meshnet-tracker start --host 0.0.0.0 --port 8081 --relay-url wss://ai.neuron.d-popov.com/ws

Verify:

curl -s https://ai.neuron.d-popov.com/v1/network/map | python3 -m json.tool

Nodes should log Relay connected — wss://…/rpc/<peer_id> on startup.

Nginx Proxy Manager setup (public hostname)

Architecture:

Client → HTTPS → ai.neuron.d-popov.com (nginx)
                  ├─ /v1/*   → meshnet-tracker :8081
                  ├─ /ws     → meshnet-relay :8765  (node persistent outbound WS)
                  └─ /rpc/*  → meshnet-relay :8765  (caller opens WS per hop)

Use one proxy host. Route sub-paths via Custom locations — do not create a second host for the same domain.

Details tab (default / → tracker):

Field	Value
Domain Names	ai.neuron.d-popov.com
Scheme	http
Forward Hostname / IP	LAN IP of tracker machine (e.g. 192.168.0.179)
Forward Port	8081
Websockets Support	ON

Custom locations (both → relay port 8765, sub-folder path empty):

Location	Forward to
/ws	192.168.0.179:8765
/rpc	192.168.0.179:8765

Advanced tab (only if WebSocket upgrade fails):

proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $http_connection;
proxy_read_timeout 3600s;
proxy_send_timeout 3600s;

Verify routing:

curl -s https://ai.neuron.d-popov.com/v1/network/map | python3 -m json.tool
curl -sI https://ai.neuron.d-popov.com/ws
curl -sI https://ai.neuron.d-popov.com/rpc/test-peer

---

2. Start a node

Starter model: Qwen/Qwen2.5-0.5B-Instruct — 0.5B params, ~1 GB BF16, 24 layers, no HuggingFace gating. Best for first-time setup.

Alpha model: qwen3.6-35b-a3b — 40 layers, ~72 GB BF16 download, MoE with hybrid linear attention. On Windows install triton-windows + flash-linear-attention; on Linux GPU use flash-linear-attention[cuda]. Tracker accepts the alias or full repo id (unsloth/Qwen3.6-35B-A3B).

Downloads cache under ~/.meshnet/models/ (or $HF_HOME / $env:HF_HOME).

Shard range is auto-detected from the curated catalog. For unknown repos the node fetches only config.json. Override with --shard-start / --shard-end for partial shards or multi-node splits.

Core command

Replace <tracker-url> and adjust the prefix for your shell (see table above).

Linux / WSL:

HF_HOME=/path/to/models .venv/bin/meshnet-node start --tracker <tracker-url> --model Qwen/Qwen2.5-0.5B-Instruct --quantization bfloat16

Windows PowerShell:

$env:HF_HOME = "D:\DEV\models"
.\.venv\Scripts\meshnet-node.exe start --tracker <tracker-url> --model Qwen/Qwen2.5-0.5B-Instruct --quantization bfloat16

Ready-to-run examples

Local dev (same machine as tracker):

HF_HOME=/path/to/models .venv/bin/meshnet-node start --tracker http://localhost:8080 --model Qwen/Qwen2.5-0.5B-Instruct --quantization bfloat16 --port 8001

Public / relay (works from WSL2, NAT, 5G — no extra flags):

.venv/bin/meshnet-node start --tracker https://ai.neuron.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct

Alpha model (Qwen3.6, Windows GPU — enable fast path):

$env:HF_HOME = "D:\DEV\models"
pip install triton-windows
pip install -U flash-linear-attention
meshnet-node start --tracker http://192.168.0.179:8080 --model qwen3.6-35b-a3b --quantization bfloat16

Do not add causal-conv1d or flash-linear-attention[cuda] on Windows (see Qwen3.5/3.6 notes).

Alpha model (Qwen3.6, Linux NVIDIA GPU — with fast path):

HF_HOME=/path/to/models .venv/bin/meshnet-node start --tracker <tracker-url> --model qwen3.6-35b-a3b --quantization bfloat16
# Install once on that machine: pip install flash-linear-attention[cuda]

Alpha model (Qwen3.6, Linux AMD ROCm GPU — with fast path):

HF_HOME=/path/to/models .venv-rocm/bin/meshnet-node start --tracker <tracker-url> --model qwen3.6-35b-a3b --quantization bfloat16
# Install once on that machine: .venv-rocm/bin/pip install 'flash-linear-attention[rocm]'

After the first node registers a model, later nodes can join with only the tracker URL (shard auto-assigned):

.venv/bin/meshnet-node start --tracker https://ai.neuron.d-popov.com

Direct LAN (Windows node reachable by IP):

$env:HF_HOME = "D:\DEV\models"
.\.venv\Scripts\meshnet-node.exe start --tracker http://192.168.0.179:8081 --model Qwen/Qwen2.5-0.5B-Instruct --shard-start 12 --shard-end 23 --quantization bfloat16 --host 0.0.0.0 --advertise-host 192.168.0.42 --port 8005

Windows direct LAN — firewall and IP checklist

1. Find LAN IP: ipconfig — use active Ethernet/Wi-Fi IPv4 (e.g. 192.168.0.42). Avoid WSL/Docker/Hyper-V addresses (172.x.x.x).
2. Allow inbound port (Administrator PowerShell, once):

New-NetFirewallRule -DisplayName "Meshnet node 8005" -Direction Inbound -Action Allow -Protocol TCP -LocalPort 8005

3. Verify from tracker machine:

curl http://192.168.0.42:8005/v1/health

404/501 is fine — it proves TCP reached the node. Timeout = check firewall, --host 0.0.0.0, and --advertise-host.

Two-node split (same or different machines)

Node A — layers 0–11 (head, serves chat):

HF_HOME=/path/to/models .venv/bin/meshnet-node start --tracker <tracker-url> --model Qwen/Qwen2.5-0.5B-Instruct --shard-start 0 --shard-end 11 --quantization bfloat16 --port 8001

Node B — layers 12–23:

HF_HOME=/path/to/models .venv/bin/meshnet-node start --tracker <tracker-url> --model Qwen/Qwen2.5-0.5B-Instruct --shard-start 12 --shard-end 23 --quantization bfloat16 --port 8002

Send inference to Node A. For cross-machine LAN tests see docs/TWO_MACHINE_TEST.md.

Useful flags

Flag	Purpose
--port 8001	Fixed listen port (default: first free ≥ 7000)
--host 0.0.0.0	Bind all interfaces (needed for direct LAN)
--advertise-host <ip>	LAN IP the tracker tells other nodes (direct LAN only)
--shard-start N --shard-end M	Partial layer range
--debug	Verbose per-hop pipeline logs (noisy; off by default)

--host 0.0.0.0 binds locally; --advertise-host is what peers use for direct hops. Omit both when using the relay path.

Expected output

  Auto-detected 24 layers → shard 0–23
  Relay connected — wss://ai.neuron.d-popov.com/rpc/abc1def2ef3f4567   # relay mode only
================================
meshnet-node ready
  Wallet:       <address>
  Model ID:     Qwen/Qwen2.5-0.5B-Instruct
  Shard:        layers 0–23; 24 of 24
  Quantization: bfloat16
  Endpoint:     http://<host>:8001
  Node ID:      <id>
  Hardware:     CPU
================================

The Endpoint is the local address. In relay mode, peers reach this node via wss://<relay>/rpc/<peer_id> instead.

Other CPU-friendly models

Model	--model / alias	Layers	BF16 size	Notes
Qwen2.5-0.5B (dev default)	Qwen/Qwen2.5-0.5B-Instruct	24	~1 GB	Fastest, no gating
Qwen3.6-35B-A3B (alpha)	qwen3.6-35b-a3b	40	~72 GB	MoE; needs transformers ≥5.12; see Qwen3.5/3.6 notes
Qwen2.5-1.5B	Qwen/Qwen2.5-1.5B-Instruct	28	~3 GB	Better quality
Phi-3-mini	microsoft/Phi-3-mini-4k-instruct	32	~7.5 GB	Best CPU quality
Llama-3.2-1B	meta-llama/Llama-3.2-1B-Instruct	16	~2 GB	Requires HF login
Llama-3.2-3B	meta-llama/Llama-3.2-3B-Instruct	28	~6 GB	Requires HF login

For gated models (Llama): huggingface-cli login first.

Browse more: .venv/bin/meshnet-node models or .venv/bin/meshnet-node models --browse.

---

3. Send an inference request

Terminal 3 — direct to the head node (replace port if you omitted --port):

curl -s http://localhost:8001/v1/chat/completions -H "Content-Type: application/json" -d '{"model": "qwen2.5-0.5b", "messages": [{"role": "user", "content": "What is 7 times 8? Answer in one word."}], "stream": false}' | python3 -m json.tool

Via tracker (tests routing / proxying):

curl -s http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{"model": "qwen2.5-0.5b", "messages": [{"role": "user", "content": "What is 7 times 8? Answer in one word."}], "stream": false}' | python3 -m json.tool

Public tracker:

curl -s https://ai.neuron.d-popov.com/v1/chat/completions -H "Content-Type: application/json" -H "Authorization: Bearer sk-mesh-<your-key>" -d '{"model": "Qwen/Qwen2.5-0.5B-Instruct", "messages": [{"role": "user", "content": "What is 7 times 8?"}], "stream": false}' | python3 -m json.tool

Test script:

.venv/bin/python scripts/test_lan_inference.py --tracker http://localhost:8080 --gateway http://localhost:8001

Verify registration on public tracker:

curl -s "https://ai.neuron.d-popov.com/v1/network/map" | python3 -m json.tool
curl -s "https://ai.neuron.d-popov.com/v1/route?model=qwen2.5-0.5b" | python3 -m json.tool

Accounts, API keys, and credit (billing-enabled trackers)

Public trackers require a real API key for /v1/chat/completions. Unknown bearer → 401; zero balance → 402 insufficient balance.

Dashboard: open https://<tracker>/dashboard, register, click + new key. With --starting-credit, the first key is pre-funded. With --devnet-topup, use +$N (devnet) to refill during testing.

Curl flow:

curl -s https://<tracker>/v1/auth/register -H "Content-Type: application/json" -d '{"email": "you@example.com", "password": "hunter22-or-better"}'
curl -s https://<tracker>/v1/account/keys -X POST -H "Authorization: Bearer <session_token>"
curl -s https://<tracker>/v1/account -H "Authorization: Bearer <session_token>"
curl -s https://<tracker>/v1/account/topup -X POST -H "Authorization: Bearer <session_token>" -H "Content-Type: application/json" -d '{"api_key": "sk-mesh-..."}'

Operator defaults: --starting-credit and --devnet-topup both default to 1 USDT. Set both to 0 on mainnet.

---

Reference

How relay hops work

When node A forwards activations to node B (behind NAT):

1. Tracker injects X-Meshnet-Route with relay_addr for behind-NAT hops.
2. Node A opens WebSocket to wss://relay/rpc/{peer_id_B}.
3. Relay forwards relay-http-request to Node B's persistent connection.
4. Node B processes /forward, returns relay-http-response.
5. Relay sends response back to Node A; Node A continues the pipeline.

Activations (bfloat16) are Base64-encoded in JSON — no precision loss. On relay failure the node logs a warning and falls back to direct HTTP before erroring.

Interactive wizard

.venv/bin/meshnet-node              # first run: wizard; later: saved config
.venv/bin/meshnet-node --reset-config

Run all tests

.venv/bin/python -m pytest -q