Files

Dobromir Popov 6c46f96aaf relaying/ RPC

2026-06-30 18:30:54 +03:00

10 KiB

Raw Blame History

Quickstart — Running a node and testing inference

This guide gets you from zero to a live inference request in three terminals. Tested on: AMD Ryzen AI Max (Strix Halo APU), 124 GB RAM, Linux, CPU inference.

Prerequisites

# Clone and enter repo
cd /run/media/popov/d/DEV/repos/d-popov.com/AI

# Create the virtualenv if it does not exist yet
python3 -m venv .venv

# Keep packaging tools current enough for editable installs
.venv/bin/python -m pip install --upgrade pip setuptools wheel

# Install Python packages (editable — picks up code changes immediately)
.venv/bin/pip install -e packages/tracker -e packages/node -e packages/p2p -e packages/gateway -e packages/relay

# CPU-only PyTorch (skip if you have CUDA/ROCm already)
.venv/bin/pip install torch --index-url https://download.pytorch.org/whl/cpu

# HuggingFace model libraries
.venv/bin/pip install transformers accelerate

NVIDIA GPU (CUDA): replace the torch line with pip install torch (default index).
AMD GPU (ROCm): pip install torch --index-url https://download.pytorch.org/whl/rocm6.2

Windows / WSL2

Run the Linux commands from WSL, not Git Bash. From the repo opened in Git Bash:

wsl
cd /mnt/d/DEV/workspace/REPOS/git.d-popov.com/neuron-tai
python3 -m venv .venv
.venv/bin/python -m pip install --upgrade pip setuptools wheel
.venv/bin/pip install -e packages/tracker -e packages/node -e packages/p2p -e packages/gateway -e packages/relay
.venv/bin/pip install torch --index-url https://download.pytorch.org/whl/cpu
.venv/bin/pip install transformers accelerate
.venv/bin/meshnet-node --help

If .venv/bin/meshnet-node is missing, the editable install step did not finish successfully. Re-run the .venv/bin/pip install -e ... command above inside WSL.

WSL2 is still useful for local development, but do not rely on it for the "another machine connects back to this node" LAN case. WSL2 commonly sits behind Windows NAT/port-proxy behavior and may not accept inbound traffic from other LAN machines without extra host networking setup. We intentionally leave that unfixed because it is useful for testing NAT/relay scenarios. If you just want to bring up a Windows node that other machines can reach directly, run the node in native Windows PowerShell instead.

Native Windows PowerShell node (not WSL)

Use this when the tracker is on another machine and you want Windows to host a reachable node on the LAN.

Install prerequisites on Windows:
- Python 3.11 or 3.12 from https://www.python.org/downloads/windows/
- Git for Windows from https://git-scm.com/download/win
Open PowerShell in the cloned repo and install the node packages:

# Example repo path; adjust to wherever you cloned it
cd D:\DEV\workspace\REPOS\git.d-popov.com\neuron-tai

py -3 -m venv .venv
.\.venv\Scripts\python.exe -m pip install --upgrade pip setuptools wheel
.\.venv\Scripts\pip.exe install -e packages\tracker -e packages\node -e packages\p2p -e packages\gateway -e packages\relay

# CPU-only PyTorch. For NVIDIA CUDA, use `pip install torch` instead.
.\.venv\Scripts\pip.exe install torch --index-url https://download.pytorch.org/whl/cpu
.\.venv\Scripts\pip.exe install transformers accelerate

.\.venv\Scripts\meshnet-node.exe --help

Find the Windows LAN IP address:

ipconfig

Use the IPv4 address on the active Ethernet/Wi-Fi adapter, for example 192.168.0.42. Avoid WSL/Docker/Hyper-V adapter addresses like 172.16.x.x, 172.17.x.x, or other virtual adapter IPs.

Allow inbound traffic for the node port in Windows Firewall. Run PowerShell as Administrator once:

New-NetFirewallRule `
  -DisplayName "Meshnet node 8005" `
  -Direction Inbound `
  -Action Allow `
  -Protocol TCP `
  -LocalPort 8005

Start the Windows node from normal PowerShell. Replace the tracker and advertised host values with your actual LAN addresses:

$env:HF_HOME = "D:\DEV\models"

.\.venv\Scripts\meshnet-node.exe start `
  --tracker http://192.168.0.179:8081 `
  --model-id Qwen/Qwen2.5-0.5B-Instruct `
  --shard-start 12 --shard-end 23 `
  --quantization bfloat16 `
  --host 0.0.0.0 `
  --advertise-host 192.168.0.42 `
  --port 8005

The example above starts the Windows node as the second half of a split Qwen model. To run the full model on Windows instead, omit --shard-start and --shard-end.

--host 0.0.0.0 binds the node to all Windows interfaces. --advertise-host is what the tracker gives to other nodes, so it must be the Windows LAN IP that the tracker and peer nodes can actually reach.

If you want verbose per-hop pipeline logs while debugging a split model, add --debug. Leave it off for normal runs; otherwise every generated token logs lines like:

  [node] pipeline hop 0: http://127.0.0.1:8005 start_layer=22
  [node] pipeline hop 0 returned text=' token'

From the tracker machine, verify Windows is reachable:

curl http://192.168.0.42:8005/v1/health

If that endpoint returns 404 or 501, that is okay: it still proves the TCP connection reached the node process. If it times out or connection-refuses, check the Windows Firewall rule, --host 0.0.0.0, the selected LAN IP, and that the node is still running.

Public tracker + WSS relay

For internet nodes, expose one public HTTPS host and proxy these paths:

/v1/*   -> meshnet-tracker, for registration, heartbeats, routing, and OpenAI requests
/ws     -> meshnet-relay, for outbound node gossip/bridge connections
/rpc/*  -> meshnet-relay, for tracker-to-node relay requests

Start the tracker with the public relay URL it should advertise:

.venv/bin/meshnet-relay --host 0.0.0.0 --port 8765
.venv/bin/meshnet-tracker start \
  --host 0.0.0.0 \
  --port 8081 \
  --relay-url wss://ai.neuron.d-popov.com/ws

Then a node only needs the public tracker address:

.venv/bin/meshnet-node start \
  --tracker https://ai.neuron.d-popov.com \
  --model-id Qwen/Qwen2.5-0.5B-Instruct

Step 1 — Start the tracker (Terminal 1)

cd /run/media/popov/d/DEV/repos/d-popov.com/AI
.venv/bin/meshnet-tracker start --port 8080

Expected output:

Tracker listening on 0.0.0.0:8080

Keep this terminal open.

Step 2 — Start a node (Terminal 2)

Recommended model: Qwen2.5-0.5B-Instruct

0.5B parameters, ~1 GB in BF16
No HuggingFace account or license required
Downloads once to ~/.meshnet/models/, cached for future runs
24 transformer layers (auto-detected — no need to specify)

cd /run/media/popov/d/DEV/repos/d-popov.com/AI
HF_HOME=/run/media/popov/d/DEV/models \
.venv/bin/meshnet-node start \
  --model-id Qwen/Qwen2.5-0.5B-Instruct \
  --quantization bfloat16 \
  --tracker http://localhost:8080 \
  --port 8001

Shard range is auto-detected from the curated catalog (no network call for known models). For unknown repos, the node fetches only config.json (~1 KB) to read num_hidden_layers. You can still pass --shard-start / --shard-end explicitly to run a partial shard on one machine.

Expected output (after model loads):

  Auto-detected 24 layers → shard 0–23
================================
meshnet-node ready
  Wallet:       <address>
  Model ID:     Qwen/Qwen2.5-0.5B-Instruct
  Shard:        layers 0–23
  Quantization: bfloat16
  Endpoint:     http://<host>:8001
  Hardware:     CPU
================================

Other model options (all CPU-friendly)

Model	HF repo	Layers	BF16 size	Notes
Qwen2.5-0.5B	`Qwen/Qwen2.5-0.5B-Instruct`	24	~1 GB	Fastest, no gating
Qwen2.5-1.5B	`Qwen/Qwen2.5-1.5B-Instruct`	28	~3 GB	Better quality
Phi-3-mini	`microsoft/Phi-3-mini-4k-instruct`	32	~7.5 GB	Best CPU quality
Llama-3.2-1B	`meta-llama/Llama-3.2-1B-Instruct`	16	~2 GB	Requires HF login
Llama-3.2-3B	`meta-llama/Llama-3.2-3B-Instruct`	28	~6 GB	Requires HF login

For gated models (Llama), run huggingface-cli login first.

Step 3 — Send an inference request (Terminal 3)

curl -s http://localhost:8001/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2.5-0.5b",
    "messages": [{"role": "user", "content": "What is 7 times 8? Answer in one word."}],
    "stream": false
  }' | python3 -m json.tool

Or use the test script:

.venv/bin/python scripts/test_lan_inference.py \
  --tracker http://localhost:8080 \
  --gateway http://localhost:8001

Two-node split (same machine, two terminals)

Split Qwen2.5-0.5B's 24 layers across two node processes to test the sharded pipeline:

Node A — layers 0–11 (tracker mode, serves chat completions):

HF_HOME=/run/media/popov/d/DEV/models \
.venv/bin/meshnet-node start \
  --model-id Qwen/Qwen2.5-0.5B-Instruct \
  --shard-start 0 --shard-end 11 \
  --quantization bfloat16 \
  --tracker http://localhost:8080 \
  --port 8001

Node B — layers 12–23:

HF_HOME=/run/media/popov/d/DEV/models \
.venv/bin/meshnet-node start \
  --model-id Qwen/Qwen2.5-0.5B-Instruct \
  --shard-start 12 --shard-end 23 \
  --quantization bfloat16 \
  --tracker http://localhost:8080 \
  --port 8002

Send the request to Node A — it tokenizes, runs layers 0–13, passes binary activations to Node B, and streams the final response back.

Two-machine LAN test (Linux + Windows/WSL2)

See docs/TWO_MACHINE_TEST.md (created by US-018).

Browse available models

# Show curated list with VRAM requirements
.venv/bin/meshnet-node models

# Browse HuggingFace Hub top-20 text-generation models
.venv/bin/meshnet-node models --browse

Start with the interactive wizard

# First run: wizard detects GPU, shows model list, saves config
.venv/bin/meshnet-node

# Subsequent runs: starts directly from saved config
.venv/bin/meshnet-node

# Re-run wizard even with saved config
.venv/bin/meshnet-node --reset-config

Start the relay node (for NAT traversal)

.venv/bin/pip install -e packages/relay
.venv/bin/meshnet-relay --port 8765

Nodes behind NAT connect to the relay and advertise their relay address to the tracker. See docs/adr/0010-p2p-gossip-and-nat-relay.md.

Run all tests

.venv/bin/python -m pytest -q

10 KiB Raw Blame History Unescape Escape