neuron-tai/QUICKSTART.md

Quickstart — Running a node and testing inference

This guide gets you from zero to a live inference request in three terminals. Tested on: AMD Ryzen AI Max (Strix Halo APU), 124 GB RAM, Linux, CPU inference.

---

Prerequisites

# Clone and enter repo
cd /run/media/popov/d/DEV/repos/d-popov.com/AI

# Create the virtualenv if it does not exist yet
python3 -m venv .venv

# Keep packaging tools current enough for editable installs
.venv/bin/python -m pip install --upgrade pip setuptools wheel

# Install Python packages (editable — picks up code changes immediately)
.venv/bin/pip install -e packages/tracker -e packages/node -e packages/p2p -e packages/gateway -e packages/relay

# CPU-only PyTorch (skip if you have CUDA/ROCm already)
.venv/bin/pip install torch --index-url https://download.pytorch.org/whl/cpu

# HuggingFace model libraries
.venv/bin/pip install transformers accelerate

NVIDIA GPU (CUDA): replace the torch line with pip install torch (default index).
AMD GPU (ROCm): pip install torch --index-url https://download.pytorch.org/whl/rocm6.2

Windows / WSL2

Run the Linux commands from WSL, not Git Bash. From the repo opened in Git Bash:

wsl
cd /mnt/d/DEV/workspace/REPOS/git.d-popov.com/neuron-tai
python3 -m venv .venv
.venv/bin/python -m pip install --upgrade pip setuptools wheel
.venv/bin/pip install -e packages/tracker -e packages/node -e packages/p2p -e packages/gateway -e packages/relay
.venv/bin/pip install torch --index-url https://download.pytorch.org/whl/cpu
.venv/bin/pip install transformers accelerate
.venv/bin/meshnet-node --help

If .venv/bin/meshnet-node is missing, the editable install step did not finish successfully. Re-run the .venv/bin/pip install -e ... command above inside WSL.

WSL2 sits behind Windows NAT and is not directly reachable from other LAN machines. Direct cross-host hops time out. The relay path (see below) solves this: the WSL2 node opens an outbound WebSocket to the relay server and all traffic flows through that tunnel. No firewall rules, no --advertise-host needed — just point at the public tracker URL.

Native Windows PowerShell node (not WSL)

Use this when the tracker is on another machine and you want Windows to host a reachable node on the LAN.

1. Install prerequisites on Windows:

   • Python 3.11 or 3.12 from https://www.python.org/downloads/windows/
   • Git for Windows from https://git-scm.com/download/win

2. Open PowerShell in the cloned repo and install the node packages:

# Example repo path; adjust to wherever you cloned it
cd D:\DEV\workspace\REPOS\git.d-popov.com\neuron-tai

python -m venv .venv
.\.venv\Scripts\python.exe -m pip install --upgrade pip setuptools wheel
.\.venv\Scripts\pip.exe install -e packages\tracker -e packages\node -e packages\p2p -e packages\gateway -e packages\relay

# CPU-only PyTorch. For NVIDIA CUDA, use `pip install torch` instead.
.\.venv\Scripts\pip.exe install torch --index-url https://download.pytorch.org/whl/cpu
.\.venv\Scripts\pip.exe install transformers accelerate

.\.venv\Scripts\meshnet-node.exe --help

For start-specific flags, run:

.\.venv\Scripts\meshnet-node.exe start --help

3. Find the Windows LAN IP address:

ipconfig

Use the IPv4 address on the active Ethernet/Wi-Fi adapter, for example 192.168.0.42. Avoid WSL/Docker/Hyper-V adapter addresses like 172.16.x.x, 172.17.x.x, or other virtual adapter IPs.

4. Allow inbound traffic for the node port in Windows Firewall. Run PowerShell as Administrator once:

New-NetFirewallRule `
  -DisplayName "Meshnet node 8005" `
  -Direction Inbound `
  -Action Allow `
  -Protocol TCP `
  -LocalPort 8005

5. Start the Windows node from normal PowerShell. Replace the tracker and advertised host values with your actual LAN addresses:

$env:HF_HOME = "D:\DEV\models"

.\.venv\Scripts\meshnet-node.exe start `
  --tracker http://192.168.0.179:8081 `
  --model Qwen/Qwen2.5-0.5B-Instruct `
  --shard-start 12 --shard-end 23 `
  --quantization bfloat16 `
  --host 0.0.0.0 `
  --advertise-host 192.168.0.42 `
  --port 8005

One-line variants (direct LAN — node must be reachable by IP from other machines):

.\.venv\Scripts\meshnet-node.exe start --tracker http://192.168.0.179:8081 --model Qwen/Qwen2.5-0.5B-Instruct --advertise-host 192.168.0.20

Via public hostname with relay (works from behind NAT, WSL2, 5G — no --advertise-host needed):

.\.venv\Scripts\meshnet-node.exe start --tracker https://ai.neuron.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct

--host 0.0.0.0 binds the node to all Windows interfaces. --advertise-host is what the tracker gives to other nodes for direct connections; omit it when using the relay path since all traffic flows through the relay tunnel instead.

If you want verbose per-hop pipeline logs while debugging a split model, add --debug. Leave it off for normal runs; otherwise every generated token logs lines like:

  [node] pipeline hop 0: http://127.0.0.1:8005 start_layer=22
  [node] pipeline hop 0 returned text=' token'
  [node] pipeline hop 1: wss://ai.neuron.d-popov.com/rpc/abc123 relay start_layer=12

6. From the tracker machine, verify Windows is reachable:

curl http://192.168.0.42:8005/v1/health

If that endpoint returns 404 or 501, that is okay: it still proves the TCP connection reached the node process. If it times out or connection-refuses, check the Windows Firewall rule, --host 0.0.0.0, the selected LAN IP, and that the node is still running.

---

Public tracker + relay (internet / NAT nodes)

This setup lets nodes connect from anywhere — behind home NAT, 5G, WSL2, or on a different continent — without opening firewall ports.

Architecture

Client → HTTPS → ai.neuron.d-popov.com (nginx)
                  ├─ /v1/*   → meshnet-tracker :8081
                  ├─ /ws     → meshnet-relay :8765  (node persistent outbound WS)
                  └─ /rpc/*  → meshnet-relay :8765  (caller opens WS per hop)

Start the relay and tracker (server side)

# Terminal 1 — relay (WebSocket hub)
.venv/bin/meshnet-relay --host 0.0.0.0 --port 8765

# Terminal 2 — tracker (advertises relay URL to nodes)
.venv/bin/meshnet-tracker start \
  --host 0.0.0.0 \
  --port 8081 \
  --relay-url wss://ai.neuron.d-popov.com/ws

The --relay-url flag embeds the relay address in /v1/network/map. Every node queries that endpoint on startup and auto-connects if a relay URL is present.

Start a node (any machine, any network)

No --advertise-host needed. The node discovers the relay URL from the tracker and opens a persistent outbound WebSocket:

.venv/bin/meshnet-node start \
  --tracker https://ai.neuron.d-popov.com \
  --model Qwen/Qwen2.5-0.5B-Instruct

Expected startup output (relay path):

  Auto-detected 24 layers → shard 0–23
  Relay connected — wss://ai.neuron.d-popov.com/rpc/abc1def2ef3f4567
================================
meshnet-node ready
  Wallet:       <address>
  Model ID:     Qwen/Qwen2.5-0.5B-Instruct
  Shard:        layers 0–23; 24 of 24
  Quantization: bfloat16
  Endpoint:     http://172.29.104.23:7001
  Node ID:      <id>
  Hardware:     CPU
================================

The Endpoint shown is the local IP (unreachable from outside). Other nodes reach this one via wss://ai.neuron.d-popov.com/rpc/<peer_id> instead.

How relay hops work

When node A needs to forward activations to node B (behind NAT):

1. Tracker injects X-Meshnet-Route with relay_addr for each behind-NAT hop.
2. Node A opens a WebSocket to wss://relay/rpc/{peer_id_B}.
3. Relay forwards the relay-http-request envelope to Node B's persistent connection.
4. Node B processes /forward locally, returns relay-http-response.
5. Relay sends the response back to Node A over the same WebSocket.
6. Node A closes the WebSocket and continues the pipeline.

Binary activation tensors (bfloat16) are Base64-encoded through the relay JSON protocol and decoded on both sides — no precision loss.

If the relay hop fails (relay down, peer disconnected), the node logs a warning and falls back to a direct HTTP attempt before returning an error.

Test from WSL2 using the public tracker

In WSL2 (which gets a 172.x.x.x virtual IP — unreachable from other machines):

# WSL2 Terminal 1 — head node (layers 0–11, handles chat requests)
.venv/bin/meshnet-node start \
  --tracker https://ai.neuron.d-popov.com \
  --model Qwen/Qwen2.5-0.5B-Instruct \
  --shard-start 0 --shard-end 11

# WSL2 Terminal 2 — tail node (layers 12–23)
.venv/bin/meshnet-node start \
  --tracker https://ai.neuron.d-popov.com \
  --model Qwen/Qwen2.5-0.5B-Instruct \
  --shard-start 12 --shard-end 23

Both nodes connect to the relay automatically. When a chat request arrives at Node A, it forwards activations to Node B via wss://ai.neuron.d-popov.com/rpc/{peer_id_B}.

Send inference through the tracker (which picks the head node and injects the route):

curl -s https://ai.neuron.d-popov.com/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-0.5B-Instruct",
    "messages": [{"role": "user", "content": "What is 7 times 8?"}],
    "stream": false
  }' | python3 -m json.tool

Or send directly to Node A's local port (within WSL):

curl -s http://localhost:7001/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "Qwen/Qwen2.5-0.5B-Instruct", "messages": [{"role": "user", "content": "Hi"}]}'

---

Step 1 — Start the tracker (Terminal 1)

cd /run/media/popov/d/DEV/repos/d-popov.com/AI
.venv/bin/meshnet-tracker start --port 8080

Expected output:

Tracker listening on 0.0.0.0:8080

Keep this terminal open.

---

Step 2 — Start a node (Terminal 2)

Recommended model: Qwen2.5-0.5B-Instruct

• 0.5B parameters, ~1 GB in BF16
• No HuggingFace account or license required
• Downloads once to ~/.meshnet/models/, cached for future runs
• 24 transformer layers (auto-detected — no need to specify)

cd /run/media/popov/d/DEV/repos/d-popov.com/AI
HF_HOME=/run/media/popov/d/DEV/models \
.venv/bin/meshnet-node start \
  --model Qwen/Qwen2.5-0.5B-Instruct \
  --quantization bfloat16 \
  --tracker http://localhost:8080 \
  --port 8001

Shard range is auto-detected from the curated catalog (no network call for known models). For unknown repos, the node fetches only config.json (~1 KB) to read num_hidden_layers. You can still pass --shard-start / --shard-end explicitly to run a partial shard on one machine.

Expected output (after model loads):

  Auto-detected 24 layers → shard 0–23
================================
meshnet-node ready
  Wallet:       <address>
  Model ID:     Qwen/Qwen2.5-0.5B-Instruct
  Shard:        layers 0–23
  Quantization: bfloat16
  Endpoint:     http://<host>:8001
  Hardware:     CPU
================================

Other model options (all CPU-friendly)

Model	HF repo	Layers	BF16 size	Notes
Qwen2.5-0.5B	Qwen/Qwen2.5-0.5B-Instruct	24	~1 GB	Fastest, no gating
Qwen2.5-1.5B	Qwen/Qwen2.5-1.5B-Instruct	28	~3 GB	Better quality
Phi-3-mini	microsoft/Phi-3-mini-4k-instruct	32	~7.5 GB	Best CPU quality
Llama-3.2-1B	meta-llama/Llama-3.2-1B-Instruct	16	~2 GB	Requires HF login
Llama-3.2-3B	meta-llama/Llama-3.2-3B-Instruct	28	~6 GB	Requires HF login

For gated models (Llama), run huggingface-cli login first.

---

Step 3 — Send an inference request (Terminal 3)

curl -s http://localhost:8001/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2.5-0.5b",
    "messages": [{"role": "user", "content": "What is 7 times 8? Answer in one word."}],
    "stream": false
  }' | python3 -m json.tool

Or use the test script:

.venv/bin/python scripts/test_lan_inference.py \
  --tracker http://localhost:8080 \
  --gateway http://localhost:8001

---

Two-node split (same machine, two terminals)

Split Qwen2.5-0.5B's 24 layers across two node processes to test the sharded pipeline:

Node A — layers 0–11 (tracker mode, serves chat completions):

HF_HOME=/run/media/popov/d/DEV/models \
.venv/bin/meshnet-node start \
  --model Qwen/Qwen2.5-0.5B-Instruct \
  --shard-start 0 --shard-end 11 \
  --quantization bfloat16 \
  --tracker http://localhost:8080 \
  --port 8001

Node B — layers 12–23:

HF_HOME=/run/media/popov/d/DEV/models \
.venv/bin/meshnet-node start \
  --model Qwen/Qwen2.5-0.5B-Instruct \
  --shard-start 12 --shard-end 23 \
  --quantization bfloat16 \
  --tracker http://localhost:8080 \
  --port 8002

Send the request to Node A — it tokenizes, runs layers 0–11, passes binary activations to Node B, and streams the final response back.

---

Two-machine LAN test (Linux + Windows/WSL2)

See docs/TWO_MACHINE_TEST.md (created by US-018).

---

Browse available models

# Show curated list with VRAM requirements
.venv/bin/meshnet-node models

# Browse HuggingFace Hub top-20 text-generation models
.venv/bin/meshnet-node models --browse

---

Start with the interactive wizard

# First run: wizard detects GPU, shows model list, saves config
.venv/bin/meshnet-node

# Subsequent runs: starts directly from saved config
.venv/bin/meshnet-node

# Re-run wizard even with saved config
.venv/bin/meshnet-node --reset-config

---

Run all tests

.venv/bin/python -m pytest -q