Files

Dobromir Popov 5b142b9976 tracher md

2026-07-02 22:03:37 +03:00

23 KiB

Raw Blame History

Quickstart — Running a node and testing inference

This guide gets you from zero to a live inference request in three terminals. Tested on: AMD Ryzen AI Max (Strix Halo APU), 124 GB RAM, Linux, CPU inference.

Prerequisites

# Clone and enter repo
cd /run/media/popov/d/DEV/repos/d-popov.com/AI

# Create the virtualenv if it does not exist yet
python3 -m venv .venv

# Keep packaging tools current enough for editable installs
.venv/bin/python -m pip install --upgrade pip setuptools wheel

# Install Python packages (editable — picks up code changes immediately)
.venv/bin/pip install -e packages/tracker -e packages/node -e packages/p2p -e packages/gateway -e packages/relay

# CPU-only PyTorch (skip if you have CUDA/ROCm already)
.venv/bin/pip install torch --index-url https://download.pytorch.org/whl/cpu

# HuggingFace model libraries
.venv/bin/pip install transformers accelerate

NVIDIA GPU (CUDA): replace the torch line with pip install torch (default index).
AMD GPU (ROCm): pip install torch --index-url https://download.pytorch.org/whl/rocm6.2

Bootstrap a tracker on a new machine

Use this when provisioning a fresh LAN/public tracker host. The tracker itself is lightweight; install the relay too if nodes will connect from NAT, WSL2, mobile, or other networks where inbound node ports are not reachable.

# 1. Get the repo onto the tracker host
git clone https://git.d-popov.com/popov/neuron-tai.git AI
cd AI

# 2. Create an isolated Python environment
python3 -m venv .venv
.venv/bin/python -m pip install --upgrade pip setuptools wheel

# 3. Install only the services needed by the tracker host
.venv/bin/pip install -e packages/tracker -e packages/relay -e packages/gateway

For a private LAN tracker, start only the tracker and open the selected TCP port on the host firewall if other machines will join:

.venv/bin/meshnet-tracker start --host 0.0.0.0 --port 8080

Verify from the tracker host:

curl -s http://localhost:8080/v1/network/map | python3 -m json.tool

Verify from another LAN machine, replacing the IP with the tracker host's LAN IP:

curl -s http://192.168.0.179:8080/v1/network/map | python3 -m json.tool

For a public tracker with relay support, run both services. The relay listens on 8765; the tracker below listens on 8081 and advertises the public WebSocket URL that nodes should use for outbound relay connections:

# Terminal 1 — relay
.venv/bin/meshnet-relay --host 0.0.0.0 --port 8765

# Terminal 2 — tracker
.venv/bin/meshnet-tracker start \
  --host 0.0.0.0 \
  --port 8081 \
  --relay-url wss://ai.neuron.d-popov.com/ws

If this host sits behind Nginx Proxy Manager, point / and /v1/* at tracker port 8081, and point /ws plus /rpc at relay port 8765 as shown in the public tracker section below. After the proxy is configured, verify the public bootstrap endpoint:

curl -s https://ai.neuron.d-popov.com/v1/network/map | python3 -m json.tool

Nodes can then join with either the LAN tracker URL or the public URL:

.venv/bin/meshnet-node start --tracker http://192.168.0.179:8080 --model Qwen/Qwen2.5-0.5B-Instruct
.venv/bin/meshnet-node start --tracker https://ai.neuron.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct

Windows / WSL2

Run the Linux commands from WSL, not Git Bash. From the repo opened in Git Bash:

wsl
cd /mnt/d/DEV/workspace/REPOS/git.d-popov.com/neuron-tai
python3 -m venv .venv
.venv/bin/python -m pip install --upgrade pip setuptools wheel
.venv/bin/pip install -e packages/tracker -e packages/node -e packages/p2p -e packages/gateway -e packages/relay
.venv/bin/pip install torch --index-url https://download.pytorch.org/whl/cpu
.venv/bin/pip install transformers accelerate
.venv/bin/meshnet-node --help

If .venv/bin/meshnet-node is missing, the editable install step did not finish successfully. Re-run the .venv/bin/pip install -e ... command above inside WSL.

WSL2 sits behind Windows NAT and is not directly reachable from other LAN machines. Direct cross-host hops time out. The relay path (see below) solves this: the WSL2 node opens an outbound WebSocket to the relay server and all traffic flows through that tunnel. No firewall rules, no --advertise-host needed — just point at the public tracker URL.

Native Windows PowerShell node (not WSL)

Use this when the tracker is on another machine and you want Windows to host a reachable node on the LAN.

Option A — existing conda/miniforge environment with CUDA torch (recommended if you already have it)

First, make sure the conda base environment is active so that python and pip both resolve to the same miniforge installation:

conda activate base
deactivate   # drop any .venv that may be layered on top; safe no-op if none active

Install project packages into the active conda/miniforge env:

cd D:\DEV\workspace\REPOS\git.d-popov.com\neuron-tai

pip install -e packages\tracker -e packages\node -e packages\p2p -e packages\gateway -e packages\relay
pip install transformers accelerate safetensors   # torch is already present

Verify torch is importable and CUDA is live before starting the node:

python -c "import torch; print(torch.__version__, torch.cuda.is_available())"
# Expected: 2.x.x+cuXXX  True

If you get ModuleNotFoundError: No module named 'torch' even though pip install torch says "already satisfied", the torch/ package directory is missing while the metadata stub remains (can happen after a conda-managed install). Force-reinstall all three PyTorch packages together so their versions stay in sync:

pip install --force-reinstall torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

Important: always reinstall torch, torchvision, and torchaudio as a group. Upgrading only torch leaves torchvision on an incompatible version, which causes RuntimeError: operator torchvision::nms does not exist and makes transformers fail to import any model class (the error surfaces as Could not import module 'Qwen2ForCausalLM').

Then re-run the verify step above.

If that prints True but meshnet-node still can't find torch, the venv entry point is shadowing the conda one. Check which binary wins:

(Get-Command meshnet-node).Source
# Should show: C:\Users\<you>\miniforge3\Scripts\meshnet-node.exe
# If it shows .venv\Scripts\meshnet-node.exe, use the full path below instead

To start a node:

$env:HF_HOME = "D:\DEV\models"
meshnet-node start --tracker https://ai.neuron.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct

If the wrong entry point is shadowing, invoke via the full conda path:

C:\Users\popov\miniforge3\Scripts\meshnet-node.exe start `
  --tracker https://ai.neuron.d-popov.com `
  --model Qwen/Qwen2.5-0.5B-Instruct

Option B — isolated virtualenv (fresh machine, no existing torch)

Install prerequisites on Windows:
- Python 3.11 or 3.12 from https://www.python.org/downloads/windows/
- Git for Windows from https://git-scm.com/download/win
Open PowerShell in the cloned repo and install the node packages:

# Example repo path; adjust to wherever you cloned it
cd D:\DEV\workspace\REPOS\git.d-popov.com\neuron-tai

python -m venv .venv
.\.venv\Scripts\python.exe -m pip install --upgrade pip setuptools wheel
.\.venv\Scripts\pip.exe install -e packages\tracker -e packages\node -e packages\p2p -e packages\gateway -e packages\relay

# CPU-only PyTorch. For NVIDIA CUDA, use `pip install torch` instead.
.\.venv\Scripts\pip.exe install torch --index-url https://download.pytorch.org/whl/cpu
.\.venv\Scripts\pip.exe install transformers accelerate

.\.venv\Scripts\meshnet-node.exe --help

For start-specific flags, run:

.\.venv\Scripts\meshnet-node.exe start --help

Find the Windows LAN IP address:

ipconfig

Use the IPv4 address on the active Ethernet/Wi-Fi adapter, for example 192.168.0.42. Avoid WSL/Docker/Hyper-V adapter addresses like 172.16.x.x, 172.17.x.x, or other virtual adapter IPs.

Allow inbound traffic for the node port in Windows Firewall. Run PowerShell as Administrator once:

New-NetFirewallRule `
  -DisplayName "Meshnet node 8005" `
  -Direction Inbound `
  -Action Allow `
  -Protocol TCP `
  -LocalPort 8005

Start the Windows node from normal PowerShell. Replace the tracker and advertised host values with your actual LAN addresses:

$env:HF_HOME = "D:\DEV\models"

.\.venv\Scripts\meshnet-node.exe start `
  --tracker http://192.168.0.179:8081 `
  --model Qwen/Qwen2.5-0.5B-Instruct `
  --shard-start 12 --shard-end 23 `
  --quantization bfloat16 `
  --host 0.0.0.0 `
  --advertise-host 192.168.0.42 `
  --port 8005

One-line variants (direct LAN — node must be reachable by IP from other machines):

.\.venv\Scripts\meshnet-node.exe start --tracker http://192.168.0.179:8081 --model Qwen/Qwen2.5-0.5B-Instruct --advertise-host 192.168.0.20

Via public hostname with relay (works from behind NAT, WSL2, 5G — no --advertise-host needed):

.\.venv\Scripts\meshnet-node.exe start --tracker https://ai.neuron.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct

WSL (same relay path — no --advertise-host):

.venv/bin/meshnet-node start --tracker https://ai.neuron.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct
.venv/bin/meshnet-node start --tracker http://192.168.0.179:8081 --model Qwen/Qwen2.5-0.5B-Instruct

--host 0.0.0.0 binds the node to all Windows interfaces. --advertise-host is what the tracker gives to other nodes for direct connections; omit it when using the relay path since all traffic flows through the relay tunnel instead.

If you want verbose per-hop pipeline logs while debugging a split model, add --debug. Leave it off for normal runs; otherwise every generated token logs lines like:

  [node] pipeline hop 0: http://127.0.0.1:8005 start_layer=22
  [node] pipeline hop 0 returned text=' token'
  [node] pipeline hop 1: wss://ai.neuron.d-popov.com/rpc/abc123 relay start_layer=12

From the tracker machine, verify Windows is reachable:

curl http://192.168.0.42:8005/v1/health

If that endpoint returns 404 or 501, that is okay: it still proves the TCP connection reached the node process. If it times out or connection-refuses, check the Windows Firewall rule, --host 0.0.0.0, the selected LAN IP, and that the node is still running.

Public tracker + relay (internet / NAT nodes)

This setup lets nodes connect from anywhere — behind home NAT, 5G, WSL2, or on a different continent — without opening firewall ports.

Architecture

Client → HTTPS → ai.neuron.d-popov.com (nginx)
                  ├─ /v1/*   → meshnet-tracker :8081
                  ├─ /ws     → meshnet-relay :8765  (node persistent outbound WS)
                  └─ /rpc/*  → meshnet-relay :8765  (caller opens WS per hop)

Nginx Proxy Manager (Docker)

Use one proxy host for the domain. Do not create a second host for the same domain to reach another port — path routing is done with Custom locations on that same host.

1. Details tab (default / → tracker)

Field	Value
Domain Names	`ai.neuron.d-popov.com`
Scheme	`http`
Forward Hostname / IP	LAN IP of the tracker machine (e.g. `192.168.0.179`)
Forward Port	`8081`
Websockets Support	ON

This serves /v1/network/map, /v1/chat/completions, and the rest of the tracker API.

2. Custom locations tab (sub-paths → relay)

The Custom locations form has no separate Websockets toggle — only location, scheme, forward host, optional sub-folder path, and port. Add two locations (both pointing at the relay process on port 8765). Leave “Add a path for sub-folder forwarding” empty so the full URI reaches the relay (/ws, /rpc/<peer_id>).

Location A — persistent node connections:

Field	Value
Define location	`/ws`
Scheme	`http`
Forward Hostname / IP	`192.168.0.179`
Forward Port	`8765`
Sub-folder path	(leave empty)

Location B — per-hop RPC:

Field	Value
Define location	`/rpc`
Scheme	`http`
Forward Hostname / IP	`192.168.0.179`
Forward Port	`8765`
Sub-folder path	(leave empty)

Nginx matches the longer prefixes first: /ws and /rpc/… go to relay; everything else stays on 8081.

3. SSL tab

Use your existing Let’s Encrypt certificate (unchanged).

4. Advanced tab (only if WebSocket upgrade fails on /ws or /rpc)

Custom locations do not expose a Websockets checkbox. If nodes show Relay configured but not connected yet while /v1/network/map works, add this snippet on the proxy host Advanced tab:

proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $http_connection;
proxy_read_timeout 3600s;
proxy_send_timeout 3600s;

5. Verify routing

# Tracker (8081 via default location)
curl -s https://ai.neuron.d-popov.com/v1/network/map | python3 -m json.tool

# Relay paths should not 502/404 from the tracker — check response headers/status
curl -sI https://ai.neuron.d-popov.com/ws
curl -sI https://ai.neuron.d-popov.com/rpc/test-peer

After NPM is correct, start relay and tracker on the LAN machine:

# Terminal 1 — relay
.venv/bin/meshnet-relay --host 0.0.0.0 --port 8765

# Terminal 2 — tracker (advertises relay to nodes)
.venv/bin/meshnet-tracker start \
  --host 0.0.0.0 \
  --port 8081 \
  --relay-url wss://ai.neuron.d-popov.com/ws

Nodes using https://ai.neuron.d-popov.com should then log:

Relay advertised by tracker — using outbound tunnel wss://ai.neuron.d-popov.com/ws
  Relay connected — wss://ai.neuron.d-popov.com/rpc/<peer_id>

The --relay-url flag embeds the relay address in /v1/network/map. Every node queries that endpoint on startup and auto-connects if a relay URL is present.

Start a node (any machine, any network)

No --advertise-host, firewall rule, port forwarding, relay URL, or peer URL is needed on the node. The public tracker is the only bootstrap URL the user types. The node queries the tracker for /v1/network/map, discovers the relay URL, and opens a persistent outbound WebSocket. If the relay connection drops, the node keeps retrying it.

.venv/bin/meshnet-node start \
  --tracker https://ai.neuron.d-popov.com \
  --model Qwen/Qwen2.5-0.5B-Instruct

No authentication is required in the prototype. The first public node for a model must still choose that model with --model or a saved wizard config. After at least one HF model node is registered, later nodes can join the public network with only the tracker URL; the tracker assigns an uncovered shard if one exists:

.venv/bin/meshnet-node start --tracker https://ai.neuron.d-popov.com

Use the public tracker to verify registration and routing:

curl -s "https://ai.neuron.d-popov.com/v1/network/map" | python3 -m json.tool
curl -s "https://ai.neuron.d-popov.com/v1/route?model=qwen2.5-0.5b" | python3 -m json.tool

Expected startup output (relay path):

  Auto-detected 24 layers → shard 0–23
  Relay connected — wss://ai.neuron.d-popov.com/rpc/abc1def2ef3f4567
================================
meshnet-node ready
  Wallet:       <address>
  Model ID:     Qwen/Qwen2.5-0.5B-Instruct
  Shard:        layers 0–23; 24 of 24
  Quantization: bfloat16
  Endpoint:     http://172.29.104.23:7001
  Node ID:      <id>
  Hardware:     CPU
================================

The Endpoint shown is the local IP (unreachable from outside). Other nodes reach this one via wss://ai.neuron.d-popov.com/rpc/<peer_id> instead.

How relay hops work

When node A needs to forward activations to node B (behind NAT):

Tracker injects X-Meshnet-Route with relay_addr for each behind-NAT hop.
Node A opens a WebSocket to wss://relay/rpc/{peer_id_B}.
Relay forwards the relay-http-request envelope to Node B's persistent connection.
Node B processes /forward locally, returns relay-http-response.
Relay sends the response back to Node A over the same WebSocket.
Node A closes the WebSocket and continues the pipeline.

Binary activation tensors (bfloat16) are Base64-encoded through the relay JSON protocol and decoded on both sides — no precision loss.

If the relay hop fails (relay down, peer disconnected), the node logs a warning and falls back to a direct HTTP attempt before returning an error.

Test from WSL2 using the public tracker

In WSL2 (which gets a 172.x.x.x virtual IP — unreachable from other machines):

# WSL2 Terminal 1 — head node (layers 0–11, handles chat requests)
.venv/bin/meshnet-node start \
  --tracker https://ai.neuron.d-popov.com \
  --model Qwen/Qwen2.5-0.5B-Instruct \
  --shard-start 0 --shard-end 11

# WSL2 Terminal 2 — tail node (layers 12–23)
.venv/bin/meshnet-node start \
  --tracker https://ai.neuron.d-popov.com \
  --model Qwen/Qwen2.5-0.5B-Instruct \
  --shard-start 12 --shard-end 23

Both nodes connect to the relay automatically. When a chat request arrives at Node A, it forwards activations to Node B via wss://ai.neuron.d-popov.com/rpc/{peer_id_B}.

Send inference through the tracker (which picks the head node and injects the route):

curl -s https://ai.neuron.d-popov.com/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-0.5B-Instruct",
    "messages": [{"role": "user", "content": "What is 7 times 8?"}],
    "stream": false
  }' | python3 -m json.tool

Or send directly to Node A's local port (within WSL):

curl -s http://localhost:7001/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "Qwen/Qwen2.5-0.5B-Instruct", "messages": [{"role": "user", "content": "Hi"}]}'

Step 1 — Start the tracker (Terminal 1)

cd /run/media/popov/d/DEV/repos/d-popov.com/AI
.venv/bin/meshnet-tracker start --port 8080

Expected output:

Tracker listening on 0.0.0.0:8080

Keep this terminal open.

Step 2 — Start a node (Terminal 2)

Recommended model: Qwen2.5-0.5B-Instruct

0.5B parameters, ~1 GB in BF16
No HuggingFace account or license required
Downloads once to ~/.meshnet/models/, cached for future runs
24 transformer layers (auto-detected — no need to specify)

cd /run/media/popov/d/DEV/repos/d-popov.com/AI
HF_HOME=/run/media/popov/d/DEV/models \
.venv/bin/meshnet-node start \
  --model Qwen/Qwen2.5-0.5B-Instruct \
  --quantization bfloat16 \
  --tracker http://localhost:8080 \
  --port 8001

Shard range is auto-detected from the curated catalog (no network call for known models). For unknown repos, the node fetches only config.json (~1 KB) to read num_hidden_layers. You can still pass --shard-start / --shard-end explicitly to run a partial shard on one machine.

Expected output (after model loads):

  Auto-detected 24 layers → shard 0–23
================================
meshnet-node ready
  Wallet:       <address>
  Model ID:     Qwen/Qwen2.5-0.5B-Instruct
  Shard:        layers 0–23
  Quantization: bfloat16
  Endpoint:     http://<host>:8001
  Hardware:     CPU
================================

Other model options (all CPU-friendly)

Model	HF repo	Layers	BF16 size	Notes
Qwen2.5-0.5B	`Qwen/Qwen2.5-0.5B-Instruct`	24	~1 GB	Fastest, no gating
Qwen2.5-1.5B	`Qwen/Qwen2.5-1.5B-Instruct`	28	~3 GB	Better quality
Phi-3-mini	`microsoft/Phi-3-mini-4k-instruct`	32	~7.5 GB	Best CPU quality
Llama-3.2-1B	`meta-llama/Llama-3.2-1B-Instruct`	16	~2 GB	Requires HF login
Llama-3.2-3B	`meta-llama/Llama-3.2-3B-Instruct`	28	~6 GB	Requires HF login

For gated models (Llama), run huggingface-cli login first.

Step 3 — Send an inference request (Terminal 3)

If you started the node with --port 8001, send the request directly to that head node:

curl -s http://localhost:8001/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2.5-0.5b",
    "messages": [{"role": "user", "content": "What is 7 times 8? Answer in one word."}],
    "stream": false
  }' | python3 -m json.tool

If you did not pass --port, meshnet-node start uses the first free port at or above 7000. Use the Endpoint: printed by the node instead of 8001.

To test tracker routing/proxying, send the same OpenAI-compatible request to the tracker, using either the full HuggingFace repo or the quick alias:

curl -s http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2.5-0.5b",
    "messages": [{"role": "user", "content": "What is 7 times 8? Answer in one word."}],
    "stream": false
  }' | python3 -m json.tool

Or use the test script:

.venv/bin/python scripts/test_lan_inference.py \
  --tracker http://localhost:8080 \
  --gateway http://localhost:8001

Two-node split (same machine, two terminals)

Split Qwen2.5-0.5B's 24 layers across two node processes to test the sharded pipeline:

Node A — layers 0–11 (tracker mode, serves chat completions):

HF_HOME=/run/media/popov/d/DEV/models \
.venv/bin/meshnet-node start \
  --model Qwen/Qwen2.5-0.5B-Instruct \
  --shard-start 0 --shard-end 11 \
  --quantization bfloat16 \
  --tracker http://localhost:8080 \
  --port 8001

Node B — layers 12–23:

HF_HOME=/run/media/popov/d/DEV/models \
.venv/bin/meshnet-node start \
  --model Qwen/Qwen2.5-0.5B-Instruct \
  --shard-start 12 --shard-end 23 \
  --quantization bfloat16 \
  --tracker http://localhost:8080 \
  --port 8002

Send the request to Node A — it tokenizes, runs layers 0–11, passes binary activations to Node B, and streams the final response back.

Two-machine LAN test (Linux + Windows/WSL2)

See docs/TWO_MACHINE_TEST.md (created by US-018).

For WSL2 nodes, registration only proves the node can reach the tracker outbound. Tracker-routed inference also requires the tracker to reach the node's advertised endpoint inbound. Either run the node in native Windows PowerShell, configure Windows port forwarding into WSL for the node port, or start the tracker with a relay URL so the node registers a relay_addr.

Browse available models

# Show curated list with VRAM requirements
.venv/bin/meshnet-node models

# Browse HuggingFace Hub top-20 text-generation models
.venv/bin/meshnet-node models --browse

Start with the interactive wizard

# First run: wizard detects GPU, shows model list, saves config
.venv/bin/meshnet-node

# Subsequent runs: starts directly from saved config
.venv/bin/meshnet-node

# Re-run wizard even with saved config
.venv/bin/meshnet-node --reset-config

Run all tests

.venv/bin/python -m pytest -q

23 KiB Raw Blame History Unescape Escape