neuron-tai/QUICKSTART.md

# Quickstart — Running a node and testing inference

This guide gets you from zero to a live inference request in three terminals.
Tested on: AMD Ryzen AI Max (Strix Halo APU), 124 GB RAM, Linux, CPU inference.

---

## Prerequisites

```bash
# Clone and enter repo
cd /run/media/popov/d/DEV/repos/d-popov.com/AI

# Create the virtualenv if it does not exist yet
python3 -m venv .venv

# Keep packaging tools current enough for editable installs
.venv/bin/python -m pip install --upgrade pip setuptools wheel

# Install Python packages (editable — picks up code changes immediately)
.venv/bin/pip install -e packages/tracker -e packages/node -e packages/p2p -e packages/gateway -e packages/relay

# CPU-only PyTorch (skip if you have CUDA/ROCm already)
.venv/bin/pip install torch --index-url https://download.pytorch.org/whl/cpu

# HuggingFace model libraries
.venv/bin/pip install transformers accelerate
```

> **NVIDIA GPU (CUDA):** replace the torch line with `pip install torch` (default index).  
> **AMD GPU (ROCm):** `pip install torch --index-url https://download.pytorch.org/whl/rocm6.2`

### Windows / WSL2

Run the Linux commands from WSL, not Git Bash. From the repo opened in Git Bash:

```bash
wsl
cd /mnt/d/DEV/workspace/REPOS/git.d-popov.com/neuron-tai
python3 -m venv .venv
.venv/bin/python -m pip install --upgrade pip setuptools wheel
.venv/bin/pip install -e packages/tracker -e packages/node -e packages/p2p -e packages/gateway -e packages/relay
.venv/bin/pip install torch --index-url https://download.pytorch.org/whl/cpu
.venv/bin/pip install transformers accelerate
.venv/bin/meshnet-node --help
```

If `.venv/bin/meshnet-node` is missing, the editable install step did not finish
successfully. Re-run the `.venv/bin/pip install -e ...` command above inside WSL.

WSL2 sits behind Windows NAT and is **not directly reachable** from other LAN machines.
Direct cross-host hops time out. The relay path (see below) solves this: the WSL2 node
opens an outbound WebSocket to the relay server and all traffic flows through that tunnel.
No firewall rules, no `--advertise-host` needed — just point at the public tracker URL.

### Native Windows PowerShell node (not WSL)

Use this when the tracker is on another machine and you want Windows to host a
reachable node on the LAN.

1. Install prerequisites on Windows:
   - Python 3.11 or 3.12 from <https://www.python.org/downloads/windows/>
   - Git for Windows from <https://git-scm.com/download/win>

2. Open **PowerShell** in the cloned repo and install the node packages:

```powershell
# Example repo path; adjust to wherever you cloned it
cd D:\DEV\workspace\REPOS\git.d-popov.com\neuron-tai

python -m venv .venv
.\.venv\Scripts\python.exe -m pip install --upgrade pip setuptools wheel
.\.venv\Scripts\pip.exe install -e packages\tracker -e packages\node -e packages\p2p -e packages\gateway -e packages\relay

# CPU-only PyTorch. For NVIDIA CUDA, use `pip install torch` instead.
.\.venv\Scripts\pip.exe install torch --index-url https://download.pytorch.org/whl/cpu
.\.venv\Scripts\pip.exe install transformers accelerate

.\.venv\Scripts\meshnet-node.exe --help
```

For `start`-specific flags, run:

```powershell
.\.venv\Scripts\meshnet-node.exe start --help
```

3. Find the Windows LAN IP address:

```powershell
ipconfig
```

Use the IPv4 address on the active Ethernet/Wi-Fi adapter, for example
`192.168.0.42`. Avoid WSL/Docker/Hyper-V adapter addresses like `172.16.x.x`,
`172.17.x.x`, or other virtual adapter IPs.

4. Allow inbound traffic for the node port in Windows Firewall. Run PowerShell as
Administrator once:

```powershell
New-NetFirewallRule `
  -DisplayName "Meshnet node 8005" `
  -Direction Inbound `
  -Action Allow `
  -Protocol TCP `
  -LocalPort 8005
```

5. Start the Windows node from normal PowerShell. Replace the tracker and
advertised host values with your actual LAN addresses:

```powershell
$env:HF_HOME = "D:\DEV\models"

.\.venv\Scripts\meshnet-node.exe start `
  --tracker http://192.168.0.179:8081 `
  --model Qwen/Qwen2.5-0.5B-Instruct `
  --shard-start 12 --shard-end 23 `
  --quantization bfloat16 `
  --host 0.0.0.0 `
  --advertise-host 192.168.0.42 `
  --port 8005
```

One-line variants (direct LAN — node must be reachable by IP from other machines):

```powershell
.\.venv\Scripts\meshnet-node.exe start --tracker http://192.168.0.179:8081 --model Qwen/Qwen2.5-0.5B-Instruct --advertise-host 192.168.0.20
```

Via public hostname with relay (works from behind NAT, WSL2, 5G — no `--advertise-host` needed):

```powershell
.\.venv\Scripts\meshnet-node.exe start --tracker https://ai.neuron.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct
```

`--host 0.0.0.0` binds the node to all Windows interfaces. `--advertise-host`
is what the tracker gives to other nodes for direct connections; omit it when using
the relay path since all traffic flows through the relay tunnel instead.

If you want verbose per-hop pipeline logs while debugging a split model, add
`--debug`. Leave it off for normal runs; otherwise every generated token logs
lines like:

```text
  [node] pipeline hop 0: http://127.0.0.1:8005 start_layer=22
  [node] pipeline hop 0 returned text=' token'
  [node] pipeline hop 1: wss://ai.neuron.d-popov.com/rpc/abc123 relay start_layer=12
```

6. From the tracker machine, verify Windows is reachable:

```bash
curl http://192.168.0.42:8005/v1/health
```

If that endpoint returns 404 or 501, that is okay: it still proves the TCP
connection reached the node process. If it times out or connection-refuses, check
the Windows Firewall rule, `--host 0.0.0.0`, the selected LAN IP, and that the
node is still running.

---

## Public tracker + relay (internet / NAT nodes)

This setup lets nodes connect from anywhere — behind home NAT, 5G, WSL2, or
on a different continent — without opening firewall ports.

### Architecture

```
Client → HTTPS → ai.neuron.d-popov.com (nginx)
                  ├─ /v1/*   → meshnet-tracker :8081
                  ├─ /ws     → meshnet-relay :8765  (node persistent outbound WS)
                  └─ /rpc/*  → meshnet-relay :8765  (caller opens WS per hop)
```

### Start the relay and tracker (server side)

```bash
# Terminal 1 — relay (WebSocket hub)
.venv/bin/meshnet-relay --host 0.0.0.0 --port 8765

# Terminal 2 — tracker (advertises relay URL to nodes)
.venv/bin/meshnet-tracker start \
  --host 0.0.0.0 \
  --port 8081 \
  --relay-url wss://ai.neuron.d-popov.com/ws
```

The `--relay-url` flag embeds the relay address in `/v1/network/map`. Every node
queries that endpoint on startup and auto-connects if a relay URL is present.

### Start a node (any machine, any network)

No `--advertise-host` needed. The node discovers the relay URL from the tracker
and opens a persistent outbound WebSocket:

```bash
.venv/bin/meshnet-node start \
  --tracker https://ai.neuron.d-popov.com \
  --model Qwen/Qwen2.5-0.5B-Instruct
```

Expected startup output (relay path):

```
  Auto-detected 24 layers → shard 0–23
  Relay connected — wss://ai.neuron.d-popov.com/rpc/abc1def2ef3f4567
================================
meshnet-node ready
  Wallet:       <address>
  Model ID:     Qwen/Qwen2.5-0.5B-Instruct
  Shard:        layers 0–23; 24 of 24
  Quantization: bfloat16
  Endpoint:     http://172.29.104.23:7001
  Node ID:      <id>
  Hardware:     CPU
================================
```

The `Endpoint` shown is the local IP (unreachable from outside). Other nodes reach
this one via `wss://ai.neuron.d-popov.com/rpc/<peer_id>` instead.

### How relay hops work

When node A needs to forward activations to node B (behind NAT):

1. Tracker injects `X-Meshnet-Route` with `relay_addr` for each behind-NAT hop.
2. Node A opens a WebSocket to `wss://relay/rpc/{peer_id_B}`.
3. Relay forwards the `relay-http-request` envelope to Node B's persistent connection.
4. Node B processes `/forward` locally, returns `relay-http-response`.
5. Relay sends the response back to Node A over the same WebSocket.
6. Node A closes the WebSocket and continues the pipeline.

Binary activation tensors (bfloat16) are Base64-encoded through the relay JSON
protocol and decoded on both sides — no precision loss.

If the relay hop fails (relay down, peer disconnected), the node logs a warning and
falls back to a direct HTTP attempt before returning an error.

### Test from WSL2 using the public tracker

In WSL2 (which gets a `172.x.x.x` virtual IP — unreachable from other machines):

```bash
# WSL2 Terminal 1 — head node (layers 0–11, handles chat requests)
.venv/bin/meshnet-node start \
  --tracker https://ai.neuron.d-popov.com \
  --model Qwen/Qwen2.5-0.5B-Instruct \
  --shard-start 0 --shard-end 11

# WSL2 Terminal 2 — tail node (layers 12–23)
.venv/bin/meshnet-node start \
  --tracker https://ai.neuron.d-popov.com \
  --model Qwen/Qwen2.5-0.5B-Instruct \
  --shard-start 12 --shard-end 23
```

Both nodes connect to the relay automatically. When a chat request arrives at Node A,
it forwards activations to Node B via `wss://ai.neuron.d-popov.com/rpc/{peer_id_B}`.

Send inference through the tracker (which picks the head node and injects the route):

```bash
curl -s https://ai.neuron.d-popov.com/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-0.5B-Instruct",
    "messages": [{"role": "user", "content": "What is 7 times 8?"}],
    "stream": false
  }' | python3 -m json.tool
```

Or send directly to Node A's local port (within WSL):

```bash
curl -s http://localhost:7001/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "Qwen/Qwen2.5-0.5B-Instruct", "messages": [{"role": "user", "content": "Hi"}]}'
```

---

## Step 1 — Start the tracker (Terminal 1)

```bash
cd /run/media/popov/d/DEV/repos/d-popov.com/AI
.venv/bin/meshnet-tracker start --port 8080
```

Expected output:
```
Tracker listening on 0.0.0.0:8080
```

Keep this terminal open.

---

## Step 2 — Start a node (Terminal 2)

### Recommended model: Qwen2.5-0.5B-Instruct

- 0.5B parameters, ~1 GB in BF16
- No HuggingFace account or license required
- Downloads once to `~/.meshnet/models/`, cached for future runs
- 24 transformer layers (auto-detected — no need to specify)

```bash
cd /run/media/popov/d/DEV/repos/d-popov.com/AI
HF_HOME=/run/media/popov/d/DEV/models \
.venv/bin/meshnet-node start \
  --model Qwen/Qwen2.5-0.5B-Instruct \
  --quantization bfloat16 \
  --tracker http://localhost:8080 \
  --port 8001
```

Shard range is **auto-detected** from the curated catalog (no network call for known
models). For unknown repos, the node fetches only `config.json` (~1 KB) to read
`num_hidden_layers`. You can still pass `--shard-start` / `--shard-end` explicitly
to run a partial shard on one machine.

Expected output (after model loads):
```
  Auto-detected 24 layers → shard 0–23
================================
meshnet-node ready
  Wallet:       <address>
  Model ID:     Qwen/Qwen2.5-0.5B-Instruct
  Shard:        layers 0–23
  Quantization: bfloat16
  Endpoint:     http://<host>:8001
  Hardware:     CPU
================================
```

### Other model options (all CPU-friendly)

| Model | HF repo | Layers | BF16 size | Notes |
|-------|---------|--------|-----------|-------|
| Qwen2.5-0.5B | `Qwen/Qwen2.5-0.5B-Instruct` | 24 | ~1 GB | Fastest, no gating |
| Qwen2.5-1.5B | `Qwen/Qwen2.5-1.5B-Instruct` | 28 | ~3 GB | Better quality |
| Phi-3-mini | `microsoft/Phi-3-mini-4k-instruct` | 32 | ~7.5 GB | Best CPU quality |
| Llama-3.2-1B | `meta-llama/Llama-3.2-1B-Instruct` | 16 | ~2 GB | Requires HF login |
| Llama-3.2-3B | `meta-llama/Llama-3.2-3B-Instruct` | 28 | ~6 GB | Requires HF login |

For gated models (Llama), run `huggingface-cli login` first.

---

## Step 3 — Send an inference request (Terminal 3)

```bash Qwen2.5-0.5B-Instruct
curl -s http://localhost:8001/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2.5-0.5b",
    "messages": [{"role": "user", "content": "What is 7 times 8? Answer in one word."}],
    "stream": false
  }' | python3 -m json.tool
```

Or use the test script:

```bash
.venv/bin/python scripts/test_lan_inference.py \
  --tracker http://localhost:8080 \
  --gateway http://localhost:8001
```

---

## Two-node split (same machine, two terminals)

Split Qwen2.5-0.5B's 24 layers across two node processes to test the sharded pipeline:

**Node A — layers 0–11 (tracker mode, serves chat completions):**
```bash
HF_HOME=/run/media/popov/d/DEV/models \
.venv/bin/meshnet-node start \
  --model Qwen/Qwen2.5-0.5B-Instruct \
  --shard-start 0 --shard-end 11 \
  --quantization bfloat16 \
  --tracker http://localhost:8080 \
  --port 8001
```

**Node B — layers 12–23:**
```bash
HF_HOME=/run/media/popov/d/DEV/models \
.venv/bin/meshnet-node start \
  --model Qwen/Qwen2.5-0.5B-Instruct \
  --shard-start 12 --shard-end 23 \
  --quantization bfloat16 \
  --tracker http://localhost:8080 \
  --port 8002
```

Send the request to Node A — it tokenizes, runs layers 0–11, passes binary
activations to Node B, and streams the final response back.

---

## Two-machine LAN test (Linux + Windows/WSL2)

See `docs/TWO_MACHINE_TEST.md` (created by US-018).

---

## Browse available models

```bash
# Show curated list with VRAM requirements
.venv/bin/meshnet-node models

# Browse HuggingFace Hub top-20 text-generation models
.venv/bin/meshnet-node models --browse
```

---

## Start with the interactive wizard

```bash
# First run: wizard detects GPU, shows model list, saves config
.venv/bin/meshnet-node

# Subsequent runs: starts directly from saved config
.venv/bin/meshnet-node

# Re-run wizard even with saved config
.venv/bin/meshnet-node --reset-config
```

---

## Run all tests

```bash
.venv/bin/python -m pytest -q
```