popov/neuron-tai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
1b9f62f82f73ee681dde6ecd9a784e8a80f8c548
neuron-tai/QUICKSTART.md
Dobromir Popov 29db25108f dash
2026-07-08 18:24:45 +02:00

20 KiB
Raw Blame History

Quickstart — Running a node and testing inference

Get from zero to a live inference request in three terminals: install once, start the tracker, start a node, send a request.

Tested on: AMD Ryzen AI Max (Strix Halo APU), 124 GB RAM, Linux, CPU inference.

Active development models (what we run day-to-day):

Role --model / alias HF repo Notes
Smoke tests, small splits Qwen/Qwen2.5-0.5B-Instruct same 24 layers, ~1 GB BF16, no gating — default for new setups
Alpha / production target qwen3.6-35b-a3b unsloth/Qwen3.6-35B-A3B 40 layers, ~72 GB BF16, hybrid linear-attention MoE; aliases include Qwen3.6-35B-A3B, Qwen/Qwen3.6-35B-A3B

At a glance

Step What Terminals
0 Install Python packages once per machine
1 Start tracker (and relay if needed) 1–2
2 Start node(s) 1+
3 Send inference request 1

Pick your connectivity mode — this determines which flags you need on the node:

Mode When to use Tracker URL Node extras
Local dev Everything on one machine http://localhost:8080 none
Direct LAN Node has a real LAN IP other machines can reach http://<tracker-ip>:8080 --host 0.0.0.0 --advertise-host <your-lan-ip> + firewall
Relay / public WSL2, NAT, 5G, or any unreachable inbound port https://ai.neuron.d-popov.com (or your public URL) none — relay handles routing

WSL2: not reachable from other LAN machines by default. Use the relay / public tracker URL, or run the node in native Windows PowerShell with direct LAN mode.

Command prefix by shell (used in examples below):

Shell Prefix Model cache env
Linux / WSL .venv/bin/ HF_HOME=/path/to/models
Windows PowerShell .\.venv\Scripts\ $env:HF_HOME = "D:\DEV\models"

0. Install prerequisites (once per machine)

Editable installs point wrappers at this source tree — code edits apply without reinstalling.

Node machine — full install

Linux / WSL 
cd /path/to/neuron-tai
python3 -m venv .venv
source .venv/bin/activate
.venv/bin/python -m pip install --upgrade pip setuptools wheel
.venv/bin/python -m pip install -e packages/tracker -e packages/node -e packages/p2p -e packages/gateway -e packages/relay
.venv/bin/python -m pip install "transformers>=5.12" accelerate
.venv/bin/meshnet-node --help
Windows PowerShell (.venv)

Requires Python 3.11+ and Git for Windows.

cd D:\DEV\workspace\REPOS\git.d-popov.com\neuron-tai
python -m venv .venv
.\.venv\Scripts\Activate.ps1
.\.venv\Scripts\python.exe -m pip install --upgrade pip setuptools wheel
.\.venv\Scripts\python.exe -m pip install -e .\packages\tracker -e .\packages\node -e .\packages\p2p -e .\packages\gateway -e .\packages\relay
.\.venv\Scripts\python.exe -m pip install "transformers>=5.12" accelerate
.\.venv\Scripts\meshnet-node.exe --help
Windows — conda/miniforge with CUDA (skip if using .venv above) 
conda activate base
deactivate   # drop any layered .venv; safe no-op if none active
cd D:\DEV\workspace\REPOS\git.d-popov.com\neuron-tai
pip install -e packages\tracker -e packages\node -e packages\p2p -e packages\gateway -e packages\relay
pip install "transformers>=5.12" accelerate safetensors
python -c "import torch; print(torch.__version__, torch.cuda.is_available())"
# Expected: 2.x.x+cuXXX  True

If torch import fails despite pip saying "already satisfied", force-reinstall all three together (never upgrade torch alone — breaks torchvision):

pip install --force-reinstall torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

If .venv\Scripts\meshnet-node.exe shadows the conda binary, use the full path: C:\Users\<you>\miniforge3\Scripts\meshnet-node.exe.

Run Linux/WSL commands from WSL, not Git Bash. From Git Bash: wsl, then cd to the repo under /mnt/d/....

Tracker host — lightweight install

Tracker + relay only; skip node packages unless this machine also runs nodes.

git clone https://git.d-popov.com/popov/neuron-tai.git AI && cd AI
python3 -m venv .venv
.venv/bin/python -m pip install --upgrade pip setuptools wheel
.venv/bin/pip install -e packages/tracker -e packages/relay -e packages/gateway

PyTorch variant

Install one torch line into the same env as meshnet-node:

Hardware Install
NVIDIA CUDA pip install torch (default index)
CPU only pip install torch --index-url https://download.pytorch.org/whl/cpu
AMD ROCm pip install torch --index-url https://download.pytorch.org/whl/rocm6.2

On Windows .venv, prefix with .\.venv\Scripts\pip.exe. Conda users with CUDA torch already installed can skip this step.

Qwen3.5/3.6-MoE notes

Applies to qwen3.6-35b-a3b and other hybrid linear-attention models. Qwen2.5-0.5B does not need any of this — it is a standard transformer with no FLA fast path.

  • transformers ≥ 5.12 required — older versions fail with 'Qwen3_5MoeConfig' object has no attribute 'vocab_size'. Check: python -c "import transformers; print(transformers.__version__)".

  • GPU fast path (optional) — without it inference still works; startup prints The fast path is not available… and linear-attention layers use a slower PyTorch fallback. Install only for your platform:

    Platform Install Notes
    Native Windows + NVIDIA pip install triton-windows then pip install flash-linear-attention Fast path works. FLA officially supports triton-windows (tested Win11, PyTorch 2.10, triton-windows 3.6). Do not use the [cuda] extra on Windows — pip looks for Linux triton and fails. Do not install causal-conv1d — FLA ≥0.3.2 ships Triton conv1d; the separate package is Linux-only and breaks on Windows (bare_metal_version / nvcc errors).
    Linux + NVIDIA CUDA pip install flash-linear-attention[cuda] causal-conv1d optional (same FLA built-in conv1d note). Needs CUDA toolkit (nvcc) matching torch, or a prebuilt wheel.
    Linux + AMD ROCm pip install flash-linear-attention[rocm] Same optional causal-conv1d note.

    Windows verify (after install):

    python -c "import triton; import fla; print('triton', triton.__version__, 'fla ok')"

    triton-windows is also pulled by meshnet-node on Windows. Without it, Qwen3.6-MoE startup fails with misleading Could not import module 'Qwen3_5MoeForCausalLM'.

Windows fast path — what failed and what actually works

The command that failed — pip install flash-linear-attention[cuda] causal-conv1d — mixes two different things:

  1. flash-linear-attention[cuda] on Windows — wrong extra. [cuda] pulls PyPI triton>=3.3, which does not exist for Windows (No matching distribution found). Use plain pip install flash-linear-attention after triton-windows is already installed; FLA detects triton-windows and uses it.
  2. causal-conv1d — separate Dao-AILab CUDA extension, not required for FLA or Qwen3.6 when FLA is installed. No official Windows wheels. Source builds need nvcc whose major version matches torch's CUDA (e.g. torch +cu118 needs CUDA 11.8 toolkit, not 12.5). Community wheels exist for narrow Python/torch combos (PR #46) but we skip them.

Working Windows stack (confirmed on this repo's dev machine: Python 3.12, torch 2.7.1+cu118, triton-windows 3.7.1, flash-linear-attention 0.5.0):

pip install triton-windows
pip install -U flash-linear-attention
python -c "import triton; import fla; print('ok')"

If the fast-path warning persists after that, upgrade FLA to ≥0.5.1 (includes the triton-windows detection from PR #757) and restart the node.

  • pip install nvidia-ml-py silences the pynvml deprecation warning on NVIDIA hosts.

1. Start the tracker

LAN tracker (private network, direct node reachability)

Terminal 1:

.venv/bin/meshnet-tracker start --host 0.0.0.0 --port 8080
# Optional devnet billing: --starting-credit 1 --devnet-topup 10

Expected: Tracker listening on 0.0.0.0:8080. Open the port on the host firewall if other machines will join.

Verify:

curl -s http://localhost:8080/v1/network/map | python3 -m json.tool
curl -s http://192.168.0.179:8080/v1/network/map | python3 -m json.tool   # from another LAN machine

Public tracker + relay (NAT / WSL2 / internet nodes)

Nodes behind NAT cannot receive inbound connections. Run both services on the tracker host; the tracker advertises the relay URL in /v1/network/map.

Terminal 1 — relay:

.venv/bin/meshnet-relay --host 0.0.0.0 --port 8765

Terminal 2 — tracker:

.venv/bin/meshnet-tracker start --host 0.0.0.0 --port 8081 --relay-url wss://ai.neuron.d-popov.com/ws

Verify:

curl -s https://ai.neuron.d-popov.com/v1/network/map | python3 -m json.tool

Nodes should log Relay connected — wss://…/rpc/<peer_id> on startup.

Nginx Proxy Manager setup (public hostname)

Architecture:

Client → HTTPS → ai.neuron.d-popov.com (nginx)
                  ├─ /v1/*   → meshnet-tracker :8081
                  ├─ /ws     → meshnet-relay :8765  (node persistent outbound WS)
                  └─ /rpc/*  → meshnet-relay :8765  (caller opens WS per hop)

Use one proxy host. Route sub-paths via Custom locations — do not create a second host for the same domain.

Details tab (default / → tracker):

Field Value
Domain Names ai.neuron.d-popov.com
Scheme http
Forward Hostname / IP LAN IP of tracker machine (e.g. 192.168.0.179)
Forward Port 8081
Websockets Support ON

Custom locations (both → relay port 8765, sub-folder path empty):

Location Forward to
/ws 192.168.0.179:8765
/rpc 192.168.0.179:8765

Advanced tab (only if WebSocket upgrade fails):

proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $http_connection;
proxy_read_timeout 3600s;
proxy_send_timeout 3600s;

Verify routing:

curl -s https://ai.neuron.d-popov.com/v1/network/map | python3 -m json.tool
curl -sI https://ai.neuron.d-popov.com/ws
curl -sI https://ai.neuron.d-popov.com/rpc/test-peer

2. Start a node

Starter model: Qwen/Qwen2.5-0.5B-Instruct — 0.5B params, ~1 GB BF16, 24 layers, no HuggingFace gating. Best for first-time setup.

Alpha model: qwen3.6-35b-a3b — 40 layers, ~72 GB BF16 download, MoE with hybrid linear attention. On Windows install triton-windows + flash-linear-attention; on Linux GPU use flash-linear-attention[cuda]. Tracker accepts the alias or full repo id (unsloth/Qwen3.6-35B-A3B).

Downloads cache under ~/.meshnet/models/ (or $HF_HOME / $env:HF_HOME).

Shard range is auto-detected from the curated catalog. For unknown repos the node fetches only config.json. Override with --shard-start / --shard-end for partial shards or multi-node splits.

Core command

Replace <tracker-url> and adjust the prefix for your shell (see table above).

Linux / WSL:

HF_HOME=/path/to/models .venv/bin/meshnet-node start --tracker <tracker-url> --model Qwen/Qwen2.5-0.5B-Instruct --quantization bfloat16

Windows PowerShell:

$env:HF_HOME = "D:\DEV\models"
.\.venv\Scripts\meshnet-node.exe start --tracker <tracker-url> --model Qwen/Qwen2.5-0.5B-Instruct --quantization bfloat16

Ready-to-run examples

Local dev (same machine as tracker):

HF_HOME=/path/to/models .venv/bin/meshnet-node start --tracker http://localhost:8080 --model Qwen/Qwen2.5-0.5B-Instruct --quantization bfloat16 --port 8001

Public / relay (works from WSL2, NAT, 5G — no extra flags):

.venv/bin/meshnet-node start --tracker https://ai.neuron.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct

Alpha model (Qwen3.6, Windows GPU — enable fast path):

$env:HF_HOME = "D:\DEV\models"
pip install triton-windows
pip install -U flash-linear-attention
meshnet-node start --tracker http://192.168.0.179:8080 --model qwen3.6-35b-a3b --quantization bfloat16

Do not add causal-conv1d or flash-linear-attention[cuda] on Windows (see Qwen3.5/3.6 notes).

Alpha model (Qwen3.6, Linux GPU — with fast path):

HF_HOME=/path/to/models .venv/bin/meshnet-node start --tracker <tracker-url> --model qwen3.6-35b-a3b --quantization bfloat16
# Install once on that machine: pip install flash-linear-attention[cuda]

After the first node registers a model, later nodes can join with only the tracker URL (shard auto-assigned):

.venv/bin/meshnet-node start --tracker https://ai.neuron.d-popov.com

Direct LAN (Windows node reachable by IP):

$env:HF_HOME = "D:\DEV\models"
.\.venv\Scripts\meshnet-node.exe start --tracker http://192.168.0.179:8081 --model Qwen/Qwen2.5-0.5B-Instruct --shard-start 12 --shard-end 23 --quantization bfloat16 --host 0.0.0.0 --advertise-host 192.168.0.42 --port 8005
Windows direct LAN — firewall and IP checklist
  1. Find LAN IP: ipconfig — use active Ethernet/Wi-Fi IPv4 (e.g. 192.168.0.42). Avoid WSL/Docker/Hyper-V addresses (172.x.x.x).
  2. Allow inbound port (Administrator PowerShell, once):
New-NetFirewallRule -DisplayName "Meshnet node 8005" -Direction Inbound -Action Allow -Protocol TCP -LocalPort 8005
  1. Verify from tracker machine:
curl http://192.168.0.42:8005/v1/health

404/501 is fine — it proves TCP reached the node. Timeout = check firewall, --host 0.0.0.0, and --advertise-host.

Two-node split (same or different machines)

Node A — layers 0–11 (head, serves chat):

HF_HOME=/path/to/models .venv/bin/meshnet-node start --tracker <tracker-url> --model Qwen/Qwen2.5-0.5B-Instruct --shard-start 0 --shard-end 11 --quantization bfloat16 --port 8001

Node B — layers 12–23:

HF_HOME=/path/to/models .venv/bin/meshnet-node start --tracker <tracker-url> --model Qwen/Qwen2.5-0.5B-Instruct --shard-start 12 --shard-end 23 --quantization bfloat16 --port 8002

Send inference to Node A. For cross-machine LAN tests see docs/TWO_MACHINE_TEST.md.

Useful flags

Flag Purpose
--port 8001 Fixed listen port (default: first free ≥ 7000)
--host 0.0.0.0 Bind all interfaces (needed for direct LAN)
--advertise-host <ip> LAN IP the tracker tells other nodes (direct LAN only)
--shard-start N --shard-end M Partial layer range
--debug Verbose per-hop pipeline logs (noisy; off by default)

--host 0.0.0.0 binds locally; --advertise-host is what peers use for direct hops. Omit both when using the relay path.

Expected output

  Auto-detected 24 layers → shard 0–23
  Relay connected — wss://ai.neuron.d-popov.com/rpc/abc1def2ef3f4567   # relay mode only
================================
meshnet-node ready
  Wallet:       <address>
  Model ID:     Qwen/Qwen2.5-0.5B-Instruct
  Shard:        layers 0–23; 24 of 24
  Quantization: bfloat16
  Endpoint:     http://<host>:8001
  Node ID:      <id>
  Hardware:     CPU
================================

The Endpoint is the local address. In relay mode, peers reach this node via wss://<relay>/rpc/<peer_id> instead.

Other CPU-friendly models

Model --model / alias Layers BF16 size Notes
Qwen2.5-0.5B (dev default) Qwen/Qwen2.5-0.5B-Instruct 24 ~1 GB Fastest, no gating
Qwen3.6-35B-A3B (alpha) qwen3.6-35b-a3b 40 ~72 GB MoE; needs transformers ≥5.12; see Qwen3.5/3.6 notes
Qwen2.5-1.5B Qwen/Qwen2.5-1.5B-Instruct 28 ~3 GB Better quality
Phi-3-mini microsoft/Phi-3-mini-4k-instruct 32 ~7.5 GB Best CPU quality
Llama-3.2-1B meta-llama/Llama-3.2-1B-Instruct 16 ~2 GB Requires HF login
Llama-3.2-3B meta-llama/Llama-3.2-3B-Instruct 28 ~6 GB Requires HF login

For gated models (Llama): huggingface-cli login first.

Browse more: .venv/bin/meshnet-node models or .venv/bin/meshnet-node models --browse.

3. Send an inference request

Terminal 3 — direct to the head node (replace port if you omitted --port):

curl -s http://localhost:8001/v1/chat/completions -H "Content-Type: application/json" -d '{"model": "qwen2.5-0.5b", "messages": [{"role": "user", "content": "What is 7 times 8? Answer in one word."}], "stream": false}' | python3 -m json.tool

Via tracker (tests routing / proxying):

curl -s http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{"model": "qwen2.5-0.5b", "messages": [{"role": "user", "content": "What is 7 times 8? Answer in one word."}], "stream": false}' | python3 -m json.tool

Public tracker:

curl -s https://ai.neuron.d-popov.com/v1/chat/completions -H "Content-Type: application/json" -H "Authorization: Bearer sk-mesh-<your-key>" -d '{"model": "Qwen/Qwen2.5-0.5B-Instruct", "messages": [{"role": "user", "content": "What is 7 times 8?"}], "stream": false}' | python3 -m json.tool

Test script:

.venv/bin/python scripts/test_lan_inference.py --tracker http://localhost:8080 --gateway http://localhost:8001

Verify registration on public tracker:

curl -s "https://ai.neuron.d-popov.com/v1/network/map" | python3 -m json.tool
curl -s "https://ai.neuron.d-popov.com/v1/route?model=qwen2.5-0.5b" | python3 -m json.tool
Accounts, API keys, and credit (billing-enabled trackers)

Public trackers require a real API key for /v1/chat/completions. Unknown bearer → 401; zero balance → 402 insufficient balance.

Dashboard: open https://<tracker>/dashboard, register, click + new key. With --starting-credit, the first key is pre-funded. With --devnet-topup, use +$N (devnet) to refill during testing.

Curl flow:

curl -s https://<tracker>/v1/auth/register -H "Content-Type: application/json" -d '{"email": "you@example.com", "password": "hunter22-or-better"}'
curl -s https://<tracker>/v1/account/keys -X POST -H "Authorization: Bearer <session_token>"
curl -s https://<tracker>/v1/account -H "Authorization: Bearer <session_token>"
curl -s https://<tracker>/v1/account/topup -X POST -H "Authorization: Bearer <session_token>" -H "Content-Type: application/json" -d '{"api_key": "sk-mesh-..."}'

Operator defaults: --starting-credit and --devnet-topup both default to 1 USDT. Set both to 0 on mainnet.

Reference

How relay hops work

When node A forwards activations to node B (behind NAT):

  1. Tracker injects X-Meshnet-Route with relay_addr for behind-NAT hops.
  2. Node A opens WebSocket to wss://relay/rpc/{peer_id_B}.
  3. Relay forwards relay-http-request to Node B's persistent connection.
  4. Node B processes /forward, returns relay-http-response.
  5. Relay sends response back to Node A; Node A continues the pipeline.

Activations (bfloat16) are Base64-encoded in JSON — no precision loss. On relay failure the node logs a warning and falls back to direct HTTP before erroring.

Interactive wizard

.venv/bin/meshnet-node              # first run: wizard; later: saved config
.venv/bin/meshnet-node --reset-config

Run all tests

.venv/bin/python -m pytest -q
Reference in New Issue View Git Blame Copy Permalink