Files
neuron-tai/QUICKSTART.md
2026-06-30 21:06:39 +02:00

581 lines
18 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Quickstart — Running a node and testing inference
This guide gets you from zero to a live inference request in three terminals.
Tested on: AMD Ryzen AI Max (Strix Halo APU), 124 GB RAM, Linux, CPU inference.
---
## Prerequisites
```bash
# Clone and enter repo
cd /run/media/popov/d/DEV/repos/d-popov.com/AI
# Create the virtualenv if it does not exist yet
python3 -m venv .venv
# Keep packaging tools current enough for editable installs
.venv/bin/python -m pip install --upgrade pip setuptools wheel
# Install Python packages (editable — picks up code changes immediately)
.venv/bin/pip install -e packages/tracker -e packages/node -e packages/p2p -e packages/gateway -e packages/relay
# CPU-only PyTorch (skip if you have CUDA/ROCm already)
.venv/bin/pip install torch --index-url https://download.pytorch.org/whl/cpu
# HuggingFace model libraries
.venv/bin/pip install transformers accelerate
```
> **NVIDIA GPU (CUDA):** replace the torch line with `pip install torch` (default index).
> **AMD GPU (ROCm):** `pip install torch --index-url https://download.pytorch.org/whl/rocm6.2`
### Windows / WSL2
Run the Linux commands from WSL, not Git Bash. From the repo opened in Git Bash:
```bash
wsl
cd /mnt/d/DEV/workspace/REPOS/git.d-popov.com/neuron-tai
python3 -m venv .venv
.venv/bin/python -m pip install --upgrade pip setuptools wheel
.venv/bin/pip install -e packages/tracker -e packages/node -e packages/p2p -e packages/gateway -e packages/relay
.venv/bin/pip install torch --index-url https://download.pytorch.org/whl/cpu
.venv/bin/pip install transformers accelerate
.venv/bin/meshnet-node --help
```
If `.venv/bin/meshnet-node` is missing, the editable install step did not finish
successfully. Re-run the `.venv/bin/pip install -e ...` command above inside WSL.
WSL2 sits behind Windows NAT and is **not directly reachable** from other LAN machines.
Direct cross-host hops time out. The relay path (see below) solves this: the WSL2 node
opens an outbound WebSocket to the relay server and all traffic flows through that tunnel.
No firewall rules, no `--advertise-host` needed — just point at the public tracker URL.
### Native Windows PowerShell node (not WSL)
Use this when the tracker is on another machine and you want Windows to host a
reachable node on the LAN.
1. Install prerequisites on Windows:
- Python 3.11 or 3.12 from <https://www.python.org/downloads/windows/>
- Git for Windows from <https://git-scm.com/download/win>
2. Open **PowerShell** in the cloned repo and install the node packages:
```powershell
# Example repo path; adjust to wherever you cloned it
cd D:\DEV\workspace\REPOS\git.d-popov.com\neuron-tai
python -m venv .venv
.\.venv\Scripts\python.exe -m pip install --upgrade pip setuptools wheel
.\.venv\Scripts\pip.exe install -e packages\tracker -e packages\node -e packages\p2p -e packages\gateway -e packages\relay
# CPU-only PyTorch. For NVIDIA CUDA, use `pip install torch` instead.
.\.venv\Scripts\pip.exe install torch --index-url https://download.pytorch.org/whl/cpu
.\.venv\Scripts\pip.exe install transformers accelerate
.\.venv\Scripts\meshnet-node.exe --help
```
For `start`-specific flags, run:
```powershell
.\.venv\Scripts\meshnet-node.exe start --help
```
3. Find the Windows LAN IP address:
```powershell
ipconfig
```
Use the IPv4 address on the active Ethernet/Wi-Fi adapter, for example
`192.168.0.42`. Avoid WSL/Docker/Hyper-V adapter addresses like `172.16.x.x`,
`172.17.x.x`, or other virtual adapter IPs.
4. Allow inbound traffic for the node port in Windows Firewall. Run PowerShell as
Administrator once:
```powershell
New-NetFirewallRule `
-DisplayName "Meshnet node 8005" `
-Direction Inbound `
-Action Allow `
-Protocol TCP `
-LocalPort 8005
```
5. Start the Windows node from normal PowerShell. Replace the tracker and
advertised host values with your actual LAN addresses:
```powershell
$env:HF_HOME = "D:\DEV\models"
.\.venv\Scripts\meshnet-node.exe start `
--tracker http://192.168.0.179:8081 `
--model Qwen/Qwen2.5-0.5B-Instruct `
--shard-start 12 --shard-end 23 `
--quantization bfloat16 `
--host 0.0.0.0 `
--advertise-host 192.168.0.42 `
--port 8005
```
One-line variants (direct LAN — node must be reachable by IP from other machines):
```powershell
.\.venv\Scripts\meshnet-node.exe start --tracker http://192.168.0.179:8081 --model Qwen/Qwen2.5-0.5B-Instruct --advertise-host 192.168.0.20
```
Via public hostname with relay (works from behind NAT, WSL2, 5G — no `--advertise-host` needed):
```powershell
.\.venv\Scripts\meshnet-node.exe start --tracker https://ai.neuron.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct
```
WSL (same relay path — no `--advertise-host`):
```bash
.venv/bin/meshnet-node start --tracker https://ai.neuron.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct
.venv/bin/meshnet-node start --tracker http://192.168.0.179:8081 --model Qwen/Qwen2.5-0.5B-Instruct
```
`--host 0.0.0.0` binds the node to all Windows interfaces. `--advertise-host`
is what the tracker gives to other nodes for direct connections; omit it when using
the relay path since all traffic flows through the relay tunnel instead.
If you want verbose per-hop pipeline logs while debugging a split model, add
`--debug`. Leave it off for normal runs; otherwise every generated token logs
lines like:
```text
[node] pipeline hop 0: http://127.0.0.1:8005 start_layer=22
[node] pipeline hop 0 returned text=' token'
[node] pipeline hop 1: wss://ai.neuron.d-popov.com/rpc/abc123 relay start_layer=12
```
6. From the tracker machine, verify Windows is reachable:
```bash
curl http://192.168.0.42:8005/v1/health
```
If that endpoint returns 404 or 501, that is okay: it still proves the TCP
connection reached the node process. If it times out or connection-refuses, check
the Windows Firewall rule, `--host 0.0.0.0`, the selected LAN IP, and that the
node is still running.
---
## Public tracker + relay (internet / NAT nodes)
This setup lets nodes connect from anywhere — behind home NAT, 5G, WSL2, or
on a different continent — without opening firewall ports.
### Architecture
```
Client → HTTPS → ai.neuron.d-popov.com (nginx)
├─ /v1/* → meshnet-tracker :8081
├─ /ws → meshnet-relay :8765 (node persistent outbound WS)
└─ /rpc/* → meshnet-relay :8765 (caller opens WS per hop)
```
### Nginx Proxy Manager (Docker)
Use **one** proxy host for the domain. Do not create a second host for the same
domain to reach another port — path routing is done with **Custom locations** on
that same host.
**1. Details tab** (default `/` → tracker)
| Field | Value |
|-------|--------|
| Domain Names | `ai.neuron.d-popov.com` |
| Scheme | `http` |
| Forward Hostname / IP | LAN IP of the tracker machine (e.g. `192.168.0.179`) |
| Forward Port | `8081` |
| Websockets Support | ON |
This serves `/v1/network/map`, `/v1/chat/completions`, and the rest of the tracker API.
**2. Custom locations tab** (sub-paths → relay)
The Custom locations form has **no separate Websockets toggle** — only location,
scheme, forward host, optional sub-folder path, and port. Add **two** locations
(both pointing at the relay process on port `8765`). Leave **“Add a path for
sub-folder forwarding”** empty so the full URI reaches the relay
(`/ws`, `/rpc/<peer_id>`).
Location A — persistent node connections:
| Field | Value |
|-------|--------|
| Define location | `/ws` |
| Scheme | `http` |
| Forward Hostname / IP | `192.168.0.179` |
| Forward Port | `8765` |
| Sub-folder path | *(leave empty)* |
Location B — per-hop RPC:
| Field | Value |
|-------|--------|
| Define location | `/rpc` |
| Scheme | `http` |
| Forward Hostname / IP | `192.168.0.179` |
| Forward Port | `8765` |
| Sub-folder path | *(leave empty)* |
Nginx matches the longer prefixes first: `/ws` and `/rpc/…` go to relay; everything
else stays on `8081`.
**3. SSL tab**
Use your existing Lets Encrypt certificate (unchanged).
**4. Advanced tab** (only if WebSocket upgrade fails on `/ws` or `/rpc`)
Custom locations do not expose a Websockets checkbox. If nodes show
`Relay configured but not connected yet` while `/v1/network/map` works, add this
snippet on the **proxy host** Advanced tab:
```nginx
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $http_connection;
proxy_read_timeout 3600s;
proxy_send_timeout 3600s;
```
**5. Verify routing**
```bash
# Tracker (8081 via default location)
curl -s https://ai.neuron.d-popov.com/v1/network/map | python3 -m json.tool
# Relay paths should not 502/404 from the tracker — check response headers/status
curl -sI https://ai.neuron.d-popov.com/ws
curl -sI https://ai.neuron.d-popov.com/rpc/test-peer
```
After NPM is correct, start relay and tracker on the LAN machine:
```bash
# Terminal 1 — relay
.venv/bin/meshnet-relay --host 0.0.0.0 --port 8765
# Terminal 2 — tracker (advertises relay to nodes)
.venv/bin/meshnet-tracker start \
--host 0.0.0.0 \
--port 8081 \
--relay-url wss://ai.neuron.d-popov.com/ws
```
Nodes using `https://ai.neuron.d-popov.com` should then log:
```text
Relay advertised by tracker — using outbound tunnel wss://ai.neuron.d-popov.com/ws
Relay connected — wss://ai.neuron.d-popov.com/rpc/<peer_id>
```
The `--relay-url` flag embeds the relay address in `/v1/network/map`. Every node
queries that endpoint on startup and auto-connects if a relay URL is present.
### Start a node (any machine, any network)
No `--advertise-host`, firewall rule, port forwarding, relay URL, or peer URL is
needed on the node. The public tracker is the only bootstrap URL the user types.
The node queries the tracker for `/v1/network/map`, discovers the relay URL, and
opens a persistent outbound WebSocket. If the relay connection drops, the node
keeps retrying it.
```bash
.venv/bin/meshnet-node start \
--tracker https://ai.neuron.d-popov.com \
--model Qwen/Qwen2.5-0.5B-Instruct
```
No authentication is required in the prototype. The first public node for a model
must still choose that model with `--model` or a saved wizard config. After at
least one HF model node is registered, later nodes can join the public network
with only the tracker URL; the tracker assigns an uncovered shard if one exists:
```bash
.venv/bin/meshnet-node start --tracker https://ai.neuron.d-popov.com
```
Use the public tracker to verify registration and routing:
```bash
curl -s "https://ai.neuron.d-popov.com/v1/network/map" | python3 -m json.tool
curl -s "https://ai.neuron.d-popov.com/v1/route?model=qwen2.5-0.5b" | python3 -m json.tool
```
Expected startup output (relay path):
```
Auto-detected 24 layers → shard 023
Relay connected — wss://ai.neuron.d-popov.com/rpc/abc1def2ef3f4567
================================
meshnet-node ready
Wallet: <address>
Model ID: Qwen/Qwen2.5-0.5B-Instruct
Shard: layers 023; 24 of 24
Quantization: bfloat16
Endpoint: http://172.29.104.23:7001
Node ID: <id>
Hardware: CPU
================================
```
The `Endpoint` shown is the local IP (unreachable from outside). Other nodes reach
this one via `wss://ai.neuron.d-popov.com/rpc/<peer_id>` instead.
### How relay hops work
When node A needs to forward activations to node B (behind NAT):
1. Tracker injects `X-Meshnet-Route` with `relay_addr` for each behind-NAT hop.
2. Node A opens a WebSocket to `wss://relay/rpc/{peer_id_B}`.
3. Relay forwards the `relay-http-request` envelope to Node B's persistent connection.
4. Node B processes `/forward` locally, returns `relay-http-response`.
5. Relay sends the response back to Node A over the same WebSocket.
6. Node A closes the WebSocket and continues the pipeline.
Binary activation tensors (bfloat16) are Base64-encoded through the relay JSON
protocol and decoded on both sides — no precision loss.
If the relay hop fails (relay down, peer disconnected), the node logs a warning and
falls back to a direct HTTP attempt before returning an error.
### Test from WSL2 using the public tracker
In WSL2 (which gets a `172.x.x.x` virtual IP — unreachable from other machines):
```bash
# WSL2 Terminal 1 — head node (layers 011, handles chat requests)
.venv/bin/meshnet-node start \
--tracker https://ai.neuron.d-popov.com \
--model Qwen/Qwen2.5-0.5B-Instruct \
--shard-start 0 --shard-end 11
# WSL2 Terminal 2 — tail node (layers 1223)
.venv/bin/meshnet-node start \
--tracker https://ai.neuron.d-popov.com \
--model Qwen/Qwen2.5-0.5B-Instruct \
--shard-start 12 --shard-end 23
```
Both nodes connect to the relay automatically. When a chat request arrives at Node A,
it forwards activations to Node B via `wss://ai.neuron.d-popov.com/rpc/{peer_id_B}`.
Send inference through the tracker (which picks the head node and injects the route):
```bash
curl -s https://ai.neuron.d-popov.com/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen2.5-0.5B-Instruct",
"messages": [{"role": "user", "content": "What is 7 times 8?"}],
"stream": false
}' | python3 -m json.tool
```
Or send directly to Node A's local port (within WSL):
```bash
curl -s http://localhost:7001/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model": "Qwen/Qwen2.5-0.5B-Instruct", "messages": [{"role": "user", "content": "Hi"}]}'
```
---
## Step 1 — Start the tracker (Terminal 1)
```bash
cd /run/media/popov/d/DEV/repos/d-popov.com/AI
.venv/bin/meshnet-tracker start --port 8080
```
Expected output:
```
Tracker listening on 0.0.0.0:8080
```
Keep this terminal open.
---
## Step 2 — Start a node (Terminal 2)
### Recommended model: Qwen2.5-0.5B-Instruct
- 0.5B parameters, ~1 GB in BF16
- No HuggingFace account or license required
- Downloads once to `~/.meshnet/models/`, cached for future runs
- 24 transformer layers (auto-detected — no need to specify)
```bash
cd /run/media/popov/d/DEV/repos/d-popov.com/AI
HF_HOME=/run/media/popov/d/DEV/models \
.venv/bin/meshnet-node start \
--model Qwen/Qwen2.5-0.5B-Instruct \
--quantization bfloat16 \
--tracker http://localhost:8080 \
--port 8001
```
Shard range is **auto-detected** from the curated catalog (no network call for known
models). For unknown repos, the node fetches only `config.json` (~1 KB) to read
`num_hidden_layers`. You can still pass `--shard-start` / `--shard-end` explicitly
to run a partial shard on one machine.
Expected output (after model loads):
```
Auto-detected 24 layers → shard 023
================================
meshnet-node ready
Wallet: <address>
Model ID: Qwen/Qwen2.5-0.5B-Instruct
Shard: layers 023
Quantization: bfloat16
Endpoint: http://<host>:8001
Hardware: CPU
================================
```
### Other model options (all CPU-friendly)
| Model | HF repo | Layers | BF16 size | Notes |
|-------|---------|--------|-----------|-------|
| Qwen2.5-0.5B | `Qwen/Qwen2.5-0.5B-Instruct` | 24 | ~1 GB | Fastest, no gating |
| Qwen2.5-1.5B | `Qwen/Qwen2.5-1.5B-Instruct` | 28 | ~3 GB | Better quality |
| Phi-3-mini | `microsoft/Phi-3-mini-4k-instruct` | 32 | ~7.5 GB | Best CPU quality |
| Llama-3.2-1B | `meta-llama/Llama-3.2-1B-Instruct` | 16 | ~2 GB | Requires HF login |
| Llama-3.2-3B | `meta-llama/Llama-3.2-3B-Instruct` | 28 | ~6 GB | Requires HF login |
For gated models (Llama), run `huggingface-cli login` first.
---
## Step 3 — Send an inference request (Terminal 3)
If you started the node with `--port 8001`, send the request directly to that
head node:
```bash Qwen2.5-0.5B-Instruct
curl -s http://localhost:8001/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "qwen2.5-0.5b",
"messages": [{"role": "user", "content": "What is 7 times 8? Answer in one word."}],
"stream": false
}' | python3 -m json.tool
```
If you did not pass `--port`, `meshnet-node start` uses the first free port at
or above `7000`. Use the `Endpoint:` printed by the node instead of `8001`.
To test tracker routing/proxying, send the same OpenAI-compatible request to the
tracker, using either the full HuggingFace repo or the quick alias:
```bash
curl -s http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "qwen2.5-0.5b",
"messages": [{"role": "user", "content": "What is 7 times 8? Answer in one word."}],
"stream": false
}' | python3 -m json.tool
```
Or use the test script:
```bash
.venv/bin/python scripts/test_lan_inference.py \
--tracker http://localhost:8080 \
--gateway http://localhost:8001
```
---
## Two-node split (same machine, two terminals)
Split Qwen2.5-0.5B's 24 layers across two node processes to test the sharded pipeline:
**Node A — layers 011 (tracker mode, serves chat completions):**
```bash
HF_HOME=/run/media/popov/d/DEV/models \
.venv/bin/meshnet-node start \
--model Qwen/Qwen2.5-0.5B-Instruct \
--shard-start 0 --shard-end 11 \
--quantization bfloat16 \
--tracker http://localhost:8080 \
--port 8001
```
**Node B — layers 1223:**
```bash
HF_HOME=/run/media/popov/d/DEV/models \
.venv/bin/meshnet-node start \
--model Qwen/Qwen2.5-0.5B-Instruct \
--shard-start 12 --shard-end 23 \
--quantization bfloat16 \
--tracker http://localhost:8080 \
--port 8002
```
Send the request to Node A — it tokenizes, runs layers 011, passes binary
activations to Node B, and streams the final response back.
---
## Two-machine LAN test (Linux + Windows/WSL2)
See `docs/TWO_MACHINE_TEST.md` (created by US-018).
For WSL2 nodes, registration only proves the node can reach the tracker
outbound. Tracker-routed inference also requires the tracker to reach the node's
advertised endpoint inbound. Either run the node in native Windows PowerShell,
configure Windows port forwarding into WSL for the node port, or start the
tracker with a relay URL so the node registers a `relay_addr`.
---
## Browse available models
```bash
# Show curated list with VRAM requirements
.venv/bin/meshnet-node models
# Browse HuggingFace Hub top-20 text-generation models
.venv/bin/meshnet-node models --browse
```
---
## Start with the interactive wizard
```bash
# First run: wizard detects GPU, shows model list, saves config
.venv/bin/meshnet-node
# Subsequent runs: starts directly from saved config
.venv/bin/meshnet-node
# Re-run wizard even with saved config
.venv/bin/meshnet-node --reset-config
```
---
## Run all tests
```bash
.venv/bin/python -m pytest -q
```