Files

Dobromir Popov 0b39d80375 md

2026-07-08 20:01:31 +02:00

20 KiB

Raw Blame History

Quickstart — Running a node and testing inference

Get from zero to a live inference request in three terminals: install once, start the tracker, start a node, send a request.

Tested on: AMD Ryzen AI Max (Strix Halo APU), 124 GB RAM, Linux, CPU inference.

Active development models (what we run day-to-day):

Role	`--model` / alias	HF repo	Notes
Smoke tests, small splits	`Qwen/Qwen2.5-0.5B-Instruct`	same	24 layers, ~1 GB BF16, no gating — default for new setups
Alpha / production target	`qwen3.6-35b-a3b`	`unsloth/Qwen3.6-35B-A3B`	40 layers, ~72 GB BF16, hybrid linear-attention MoE; aliases include `Qwen3.6-35B-A3B`, `Qwen/Qwen3.6-35B-A3B`

At a glance

Step	What	Terminals
0	Install Python packages	once per machine
1	Start tracker (and relay if needed)	1–2
2	Start node(s)	1+
3	Send inference request	1

Pick your connectivity mode — this determines which flags you need on the node:

Mode	When to use	Tracker URL	Node extras
Local dev	Everything on one machine	`http://localhost:8080`	none
Direct LAN	Node has a real LAN IP other machines can reach	`http://<tracker-ip>:8080`	`--host 0.0.0.0 --advertise-host <your-lan-ip>` + firewall
Relay / public	WSL2, NAT, 5G, or any unreachable inbound port	`https://ai.neuron.d-popov.com` (or your public URL)	none — relay handles routing

WSL2: not reachable from other LAN machines by default. Use the relay / public tracker URL, or run the node in native Windows PowerShell with direct LAN mode.

Command prefix by shell (used in examples below):

Shell	Prefix	Model cache env
Linux / WSL	`.venv/bin/`	`HF_HOME=/path/to/models`
Windows PowerShell	`.\.venv\Scripts\`	`$env:HF_HOME = "D:\DEV\models"`

0. Install prerequisites (once per machine)

Editable installs point wrappers at this source tree — code edits apply without reinstalling.

Node machine — full install

Linux / WSL

cd /path/to/neuron-tai
python3 -m venv .venv
source .venv/bin/activate
.venv/bin/python -m pip install --upgrade pip setuptools wheel
.venv/bin/python -m pip install -e packages/tracker -e packages/node -e packages/p2p -e packages/gateway -e packages/relay
.venv/bin/python -m pip install "transformers>=5.12" accelerate
.venv/bin/meshnet-node --help

Windows PowerShell (.venv)

Requires Python 3.11+ and Git for Windows.

cd D:\DEV\workspace\REPOS\git.d-popov.com\neuron-tai
python -m venv .venv
.\.venv\Scripts\Activate.ps1
.\.venv\Scripts\python.exe -m pip install --upgrade pip setuptools wheel
.\.venv\Scripts\python.exe -m pip install -e .\packages\tracker -e .\packages\node -e .\packages\p2p -e .\packages\gateway -e .\packages\relay
.\.venv\Scripts\python.exe -m pip install "transformers>=5.12" accelerate
.\.venv\Scripts\meshnet-node.exe --help

Windows — conda/miniforge with CUDA (skip if using .venv above)

conda activate base
deactivate   # drop any layered .venv; safe no-op if none active
cd D:\DEV\workspace\REPOS\git.d-popov.com\neuron-tai
pip install -e packages\tracker -e packages\node -e packages\p2p -e packages\gateway -e packages\relay
pip install "transformers>=5.12" accelerate safetensors
python -c "import torch; print(torch.__version__, torch.cuda.is_available())"
# Expected: 2.x.x+cuXXX  True

If torch import fails despite pip saying "already satisfied", force-reinstall all three together (never upgrade torch alone — breaks torchvision):

pip install --force-reinstall torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

If .venv\Scripts\meshnet-node.exe shadows the conda binary, use the full path: C:\Users\<you>\miniforge3\Scripts\meshnet-node.exe.

Run Linux/WSL commands from WSL, not Git Bash. From Git Bash: wsl, then cd to the repo under /mnt/d/....

Tracker host — lightweight install

Tracker + relay only; skip node packages unless this machine also runs nodes.

git clone https://git.d-popov.com/popov/neuron-tai.git AI && cd AI
python3 -m venv .venv
.venv/bin/python -m pip install --upgrade pip setuptools wheel
.venv/bin/pip install -e packages/tracker -e packages/relay -e packages/gateway

PyTorch variant

Install one torch line into the same env as meshnet-node:

Hardware	Install
NVIDIA CUDA	`pip install torch` (default index)
CPU only	`pip install torch --index-url https://download.pytorch.org/whl/cpu`
AMD ROCm	`pip install torch --index-url https://download.pytorch.org/whl/rocm6.2`

On Windows .venv, prefix with .\.venv\Scripts\pip.exe. Conda users with CUDA torch already installed can skip this step.

Qwen3.5/3.6-MoE notes

Applies to qwen3.6-35b-a3b and other hybrid linear-attention models. Qwen2.5-0.5B does not need any of this — it is a standard transformer with no FLA fast path.

transformers ≥ 5.12 required — older versions fail with 'Qwen3_5MoeConfig' object has no attribute 'vocab_size'. Check: python -c "import transformers; print(transformers.__version__)".

GPU fast path (optional) — without it inference still works; startup prints The fast path is not available… and linear-attention layers use a slower PyTorch fallback. The fast path runs on NVIDIA CUDA GPUs on both Linux and native Windows — the FLA kernels are Triton-compiled, and triton-windows compiles them for CUDA on Windows just like Linux Triton does. Only the pip command differs per platform. Install only for your platform:

Platform	Install	Notes
Native Windows + NVIDIA CUDA	`pip install triton-windows` then `pip install flash-linear-attention`	Fast path works on the CUDA GPU — no CUDA toolkit / `nvcc` needed; `triton-windows` bundles its own compiler. FLA officially supports `triton-windows` (tested Win11, PyTorch 2.10, triton-windows 3.6). Do not use the `[cuda]` extra on Windows — that extra only pins Linux PyPI `triton` and fails; it is a packaging name, not a GPU requirement. Do not install `causal-conv1d` — FLA ≥0.3.2 ships Triton conv1d; the separate package is Linux-only and breaks on Windows (`bare_metal_version` / nvcc errors).
Linux + NVIDIA CUDA	`pip install flash-linear-attention[cuda]`	`causal-conv1d` optional (same FLA built-in conv1d note). Needs CUDA toolkit (`nvcc`) matching torch, or a prebuilt wheel.
Linux + AMD ROCm	`pip install flash-linear-attention[rocm]`	Same optional `causal-conv1d` note.

Windows verify (after install):

python -c "import triton; import fla; print('triton', triton.__version__, 'fla ok')"

triton-windows is also pulled by meshnet-node on Windows. Without it, Qwen3.6-MoE startup fails with misleading Could not import module 'Qwen3_5MoeForCausalLM'.

Windows fast path — what failed and what actually works

The command that failed — pip install flash-linear-attention[cuda] causal-conv1d — mixes two different things:

flash-linear-attention[cuda] on Windows — wrong extra. [cuda] pulls PyPI triton>=3.3, which does not exist for Windows (No matching distribution found). Use plain pip install flash-linear-attention after triton-windows is already installed; FLA detects triton-windows and uses it.
causal-conv1d — separate Dao-AILab CUDA extension, not required for FLA or Qwen3.6 when FLA is installed. No official Windows wheels. Source builds need nvcc whose major version matches torch's CUDA (e.g. torch +cu118 needs CUDA 11.8 toolkit, not 12.5). Community wheels exist for narrow Python/torch combos (PR #46) but we skip them.

Working Windows stack (confirmed on this repo's dev machine: Python 3.12, torch 2.7.1+cu118, triton-windows 3.7.1, flash-linear-attention 0.5.0):

pip install triton-windows
pip install -U flash-linear-attention
python -c "import triton; import fla; print('ok')"

If the fast-path warning persists after that, upgrade FLA to ≥0.5.1 (includes the triton-windows detection from PR #757) and restart the node.

pip install nvidia-ml-py silences the pynvml deprecation warning on NVIDIA hosts.

1. Start the tracker

LAN tracker (private network, direct node reachability)

Terminal 1:

.venv/bin/meshnet-tracker start --host 0.0.0.0 --port 8080
# Optional devnet billing: --starting-credit 1 --devnet-topup 10

Expected: Tracker listening on 0.0.0.0:8080. Open the port on the host firewall if other machines will join.

Verify:

curl -s http://localhost:8080/v1/network/map | python3 -m json.tool
curl -s http://192.168.0.179:8080/v1/network/map | python3 -m json.tool   # from another LAN machine

Public tracker + relay (NAT / WSL2 / internet nodes)

Nodes behind NAT cannot receive inbound connections. Run both services on the tracker host; the tracker advertises the relay URL in /v1/network/map.

Terminal 1 — relay:

.venv/bin/meshnet-relay --host 0.0.0.0 --port 8765

Terminal 2 — tracker:

.venv/bin/meshnet-tracker start --host 0.0.0.0 --port 8081 --relay-url wss://ai.neuron.d-popov.com/ws

Verify:

curl -s https://ai.neuron.d-popov.com/v1/network/map | python3 -m json.tool

Nodes should log Relay connected — wss://…/rpc/<peer_id> on startup.

Nginx Proxy Manager setup (public hostname)

Architecture:

Client → HTTPS → ai.neuron.d-popov.com (nginx)
                  ├─ /v1/*   → meshnet-tracker :8081
                  ├─ /ws     → meshnet-relay :8765  (node persistent outbound WS)
                  └─ /rpc/*  → meshnet-relay :8765  (caller opens WS per hop)

Use one proxy host. Route sub-paths via Custom locations — do not create a second host for the same domain.

Details tab (default / → tracker):

Field	Value
Domain Names	`ai.neuron.d-popov.com`
Scheme	`http`
Forward Hostname / IP	LAN IP of tracker machine (e.g. `192.168.0.179`)
Forward Port	`8081`
Websockets Support	ON

Custom locations (both → relay port 8765, sub-folder path empty):

Location	Forward to
`/ws`	`192.168.0.179:8765`
`/rpc`	`192.168.0.179:8765`

Advanced tab (only if WebSocket upgrade fails):

proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $http_connection;
proxy_read_timeout 3600s;
proxy_send_timeout 3600s;

Verify routing:

curl -s https://ai.neuron.d-popov.com/v1/network/map | python3 -m json.tool
curl -sI https://ai.neuron.d-popov.com/ws
curl -sI https://ai.neuron.d-popov.com/rpc/test-peer

2. Start a node

Starter model: Qwen/Qwen2.5-0.5B-Instruct — 0.5B params, ~1 GB BF16, 24 layers, no HuggingFace gating. Best for first-time setup.

Alpha model: qwen3.6-35b-a3b — 40 layers, ~72 GB BF16 download, MoE with hybrid linear attention. On Windows install triton-windows + flash-linear-attention; on Linux GPU use flash-linear-attention[cuda]. Tracker accepts the alias or full repo id (unsloth/Qwen3.6-35B-A3B).

Downloads cache under ~/.meshnet/models/ (or $HF_HOME / $env:HF_HOME).

Shard range is auto-detected from the curated catalog. For unknown repos the node fetches only config.json. Override with --shard-start / --shard-end for partial shards or multi-node splits.

Core command

Replace <tracker-url> and adjust the prefix for your shell (see table above).

Linux / WSL:

HF_HOME=/path/to/models .venv/bin/meshnet-node start --tracker <tracker-url> --model Qwen/Qwen2.5-0.5B-Instruct --quantization bfloat16

Windows PowerShell:

$env:HF_HOME = "D:\DEV\models"
.\.venv\Scripts\meshnet-node.exe start --tracker <tracker-url> --model Qwen/Qwen2.5-0.5B-Instruct --quantization bfloat16

Ready-to-run examples

Local dev (same machine as tracker):

HF_HOME=/path/to/models .venv/bin/meshnet-node start --tracker http://localhost:8080 --model Qwen/Qwen2.5-0.5B-Instruct --quantization bfloat16 --port 8001

Public / relay (works from WSL2, NAT, 5G — no extra flags):

.venv/bin/meshnet-node start --tracker https://ai.neuron.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct

Alpha model (Qwen3.6, Windows GPU — enable fast path):

$env:HF_HOME = "D:\DEV\models"
pip install triton-windows
pip install -U flash-linear-attention
meshnet-node start --tracker http://192.168.0.179:8080 --model qwen3.6-35b-a3b --quantization bfloat16

Do not add causal-conv1d or flash-linear-attention[cuda] on Windows (see Qwen3.5/3.6 notes).

Alpha model (Qwen3.6, Linux GPU — with fast path):

HF_HOME=/path/to/models .venv/bin/meshnet-node start --tracker <tracker-url> --model qwen3.6-35b-a3b --quantization bfloat16
# Install once on that machine: pip install flash-linear-attention[cuda]

After the first node registers a model, later nodes can join with only the tracker URL (shard auto-assigned):

.venv/bin/meshnet-node start --tracker https://ai.neuron.d-popov.com

Direct LAN (Windows node reachable by IP):

$env:HF_HOME = "D:\DEV\models"
.\.venv\Scripts\meshnet-node.exe start --tracker http://192.168.0.179:8081 --model Qwen/Qwen2.5-0.5B-Instruct --shard-start 12 --shard-end 23 --quantization bfloat16 --host 0.0.0.0 --advertise-host 192.168.0.42 --port 8005

Windows direct LAN — firewall and IP checklist

Find LAN IP: ipconfig — use active Ethernet/Wi-Fi IPv4 (e.g. 192.168.0.42). Avoid WSL/Docker/Hyper-V addresses (172.x.x.x).
Allow inbound port (Administrator PowerShell, once):

New-NetFirewallRule -DisplayName "Meshnet node 8005" -Direction Inbound -Action Allow -Protocol TCP -LocalPort 8005

Verify from tracker machine:

curl http://192.168.0.42:8005/v1/health

404/501 is fine — it proves TCP reached the node. Timeout = check firewall, --host 0.0.0.0, and --advertise-host.

Two-node split (same or different machines)

Node A — layers 0–11 (head, serves chat):

HF_HOME=/path/to/models .venv/bin/meshnet-node start --tracker <tracker-url> --model Qwen/Qwen2.5-0.5B-Instruct --shard-start 0 --shard-end 11 --quantization bfloat16 --port 8001

Node B — layers 12–23:

HF_HOME=/path/to/models .venv/bin/meshnet-node start --tracker <tracker-url> --model Qwen/Qwen2.5-0.5B-Instruct --shard-start 12 --shard-end 23 --quantization bfloat16 --port 8002

Send inference to Node A. For cross-machine LAN tests see docs/TWO_MACHINE_TEST.md.

Useful flags

Flag	Purpose
`--port 8001`	Fixed listen port (default: first free ≥ 7000)
`--host 0.0.0.0`	Bind all interfaces (needed for direct LAN)
`--advertise-host <ip>`	LAN IP the tracker tells other nodes (direct LAN only)
`--shard-start N --shard-end M`	Partial layer range
`--debug`	Verbose per-hop pipeline logs (noisy; off by default)

--host 0.0.0.0 binds locally; --advertise-host is what peers use for direct hops. Omit both when using the relay path.

Expected output

  Auto-detected 24 layers → shard 0–23
  Relay connected — wss://ai.neuron.d-popov.com/rpc/abc1def2ef3f4567   # relay mode only
================================
meshnet-node ready
  Wallet:       <address>
  Model ID:     Qwen/Qwen2.5-0.5B-Instruct
  Shard:        layers 0–23; 24 of 24
  Quantization: bfloat16
  Endpoint:     http://<host>:8001
  Node ID:      <id>
  Hardware:     CPU
================================

The Endpoint is the local address. In relay mode, peers reach this node via wss://<relay>/rpc/<peer_id> instead.

Other CPU-friendly models

Model	`--model` / alias	Layers	BF16 size	Notes
Qwen2.5-0.5B (dev default)	`Qwen/Qwen2.5-0.5B-Instruct`	24	~1 GB	Fastest, no gating
Qwen3.6-35B-A3B (alpha)	`qwen3.6-35b-a3b`	40	~72 GB	MoE; needs transformers ≥5.12; see Qwen3.5/3.6 notes
Qwen2.5-1.5B	`Qwen/Qwen2.5-1.5B-Instruct`	28	~3 GB	Better quality
Phi-3-mini	`microsoft/Phi-3-mini-4k-instruct`	32	~7.5 GB	Best CPU quality
Llama-3.2-1B	`meta-llama/Llama-3.2-1B-Instruct`	16	~2 GB	Requires HF login
Llama-3.2-3B	`meta-llama/Llama-3.2-3B-Instruct`	28	~6 GB	Requires HF login

For gated models (Llama): huggingface-cli login first.

Browse more: .venv/bin/meshnet-node models or .venv/bin/meshnet-node models --browse.

3. Send an inference request

Terminal 3 — direct to the head node (replace port if you omitted --port):

curl -s http://localhost:8001/v1/chat/completions -H "Content-Type: application/json" -d '{"model": "qwen2.5-0.5b", "messages": [{"role": "user", "content": "What is 7 times 8? Answer in one word."}], "stream": false}' | python3 -m json.tool

Via tracker (tests routing / proxying):

curl -s http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{"model": "qwen2.5-0.5b", "messages": [{"role": "user", "content": "What is 7 times 8? Answer in one word."}], "stream": false}' | python3 -m json.tool

Public tracker:

curl -s https://ai.neuron.d-popov.com/v1/chat/completions -H "Content-Type: application/json" -H "Authorization: Bearer sk-mesh-<your-key>" -d '{"model": "Qwen/Qwen2.5-0.5B-Instruct", "messages": [{"role": "user", "content": "What is 7 times 8?"}], "stream": false}' | python3 -m json.tool

Test script:

.venv/bin/python scripts/test_lan_inference.py --tracker http://localhost:8080 --gateway http://localhost:8001

Verify registration on public tracker:

curl -s "https://ai.neuron.d-popov.com/v1/network/map" | python3 -m json.tool
curl -s "https://ai.neuron.d-popov.com/v1/route?model=qwen2.5-0.5b" | python3 -m json.tool

Accounts, API keys, and credit (billing-enabled trackers)

Public trackers require a real API key for /v1/chat/completions. Unknown bearer → 401; zero balance → 402 insufficient balance.

Dashboard: open https://<tracker>/dashboard, register, click + new key. With --starting-credit, the first key is pre-funded. With --devnet-topup, use +$N (devnet) to refill during testing.

Curl flow:

curl -s https://<tracker>/v1/auth/register -H "Content-Type: application/json" -d '{"email": "you@example.com", "password": "hunter22-or-better"}'
curl -s https://<tracker>/v1/account/keys -X POST -H "Authorization: Bearer <session_token>"
curl -s https://<tracker>/v1/account -H "Authorization: Bearer <session_token>"
curl -s https://<tracker>/v1/account/topup -X POST -H "Authorization: Bearer <session_token>" -H "Content-Type: application/json" -d '{"api_key": "sk-mesh-..."}'

Operator defaults: --starting-credit and --devnet-topup both default to 1 USDT. Set both to 0 on mainnet.

Reference

How relay hops work

When node A forwards activations to node B (behind NAT):

Tracker injects X-Meshnet-Route with relay_addr for behind-NAT hops.
Node A opens WebSocket to wss://relay/rpc/{peer_id_B}.
Relay forwards relay-http-request to Node B's persistent connection.
Node B processes /forward, returns relay-http-response.
Relay sends response back to Node A; Node A continues the pipeline.

Activations (bfloat16) are Base64-encoded in JSON — no precision loss. On relay failure the node logs a warning and falls back to direct HTTP before erroring.

Interactive wizard

.venv/bin/meshnet-node              # first run: wizard; later: saved config
.venv/bin/meshnet-node --reset-config

Run all tests

.venv/bin/python -m pytest -q

20 KiB Raw Blame History Unescape Escape