neuron-tai/QUICKSTART.md

# Quickstart — Running a node and testing inference

This guide gets you from zero to a live inference request in three terminals.
Tested on: AMD Ryzen AI Max (Strix Halo APU), 124 GB RAM, Linux, CPU inference.

---

## Prerequisites

```bash
# Clone and enter repo
cd /run/media/popov/d/DEV/repos/d-popov.com/AI

# Create the virtualenv if it does not exist yet
python3 -m venv .venv

# Keep packaging tools current enough for editable installs
.venv/bin/python -m pip install --upgrade pip setuptools wheel

# Install Python packages (editable — picks up code changes immediately)
.venv/bin/pip install -e packages/tracker -e packages/node -e packages/p2p -e packages/gateway -e packages/relay

# CPU-only PyTorch (skip if you have CUDA/ROCm already)
.venv/bin/pip install torch --index-url https://download.pytorch.org/whl/cpu

# HuggingFace model libraries
.venv/bin/pip install transformers accelerate
```

> **NVIDIA GPU (CUDA):** replace the torch line with `pip install torch` (default index).  
> **AMD GPU (ROCm):** `pip install torch --index-url https://download.pytorch.org/whl/rocm6.2`

### Windows / WSL2

Run the Linux commands from WSL, not Git Bash. From the repo opened in Git Bash:

```bash
wsl
cd /mnt/d/DEV/workspace/REPOS/git.d-popov.com/neuron-tai
python3 -m venv .venv
.venv/bin/python -m pip install --upgrade pip setuptools wheel
.venv/bin/pip install -e packages/tracker -e packages/node -e packages/p2p -e packages/gateway -e packages/relay
.venv/bin/pip install torch --index-url https://download.pytorch.org/whl/cpu
.venv/bin/pip install transformers accelerate
.venv/bin/meshnet-node --help
```

If `.venv/bin/meshnet-node` is missing, the editable install step did not finish
successfully. Re-run the `.venv/bin/pip install -e ...` command above inside WSL.

### Public tracker + WSS relay

For internet nodes, expose one public HTTPS host and proxy these paths:

```text
/v1/*   -> meshnet-tracker, for registration, heartbeats, routing, and OpenAI requests
/ws     -> meshnet-relay, for outbound node gossip/bridge connections
/rpc/*  -> meshnet-relay, for tracker-to-node relay requests
```

Start the tracker with the public relay URL it should advertise:

```bash
.venv/bin/meshnet-relay --host 0.0.0.0 --port 8765
.venv/bin/meshnet-tracker start \
  --host 0.0.0.0 \
  --port 8081 \
  --relay-url wss://ai.neuron.d-popov.com/ws
```

Then a node only needs the public tracker address:

```bash
.venv/bin/meshnet-node start \
  --tracker https://ai.neuron.d-popov.com \
  --model-id Qwen/Qwen2.5-0.5B-Instruct
```

---

## Step 1 — Start the tracker (Terminal 1)

```bash
cd /run/media/popov/d/DEV/repos/d-popov.com/AI
.venv/bin/meshnet-tracker start --port 8080
```

Expected output:
```
Tracker listening on 0.0.0.0:8080
```

Keep this terminal open.

---

## Step 2 — Start a node (Terminal 2)

### Recommended model: Qwen2.5-0.5B-Instruct

- 0.5B parameters, ~1 GB in BF16
- No HuggingFace account or license required
- Downloads once to `~/.meshnet/models/`, cached for future runs
- 24 transformer layers (auto-detected — no need to specify)

```bash
cd /run/media/popov/d/DEV/repos/d-popov.com/AI
HF_HOME=/run/media/popov/d/DEV/models \
.venv/bin/meshnet-node start \
  --model-id Qwen/Qwen2.5-0.5B-Instruct \
  --quantization bfloat16 \
  --tracker http://localhost:8080 \
  --port 8001
```

Shard range is **auto-detected** from the curated catalog (no network call for known
models). For unknown repos, the node fetches only `config.json` (~1 KB) to read
`num_hidden_layers`. You can still pass `--shard-start` / `--shard-end` explicitly
to run a partial shard on one machine.

Expected output (after model loads):
```
  Auto-detected 24 layers → shard 0–23
================================
meshnet-node ready
  Wallet:       <address>
  Model ID:     Qwen/Qwen2.5-0.5B-Instruct
  Shard:        layers 0–23
  Quantization: bfloat16
  Endpoint:     http://<host>:8001
  Hardware:     CPU
================================
```

### Other model options (all CPU-friendly)

| Model | HF repo | Layers | BF16 size | Notes |
|-------|---------|--------|-----------|-------|
| Qwen2.5-0.5B | `Qwen/Qwen2.5-0.5B-Instruct` | 24 | ~1 GB | Fastest, no gating |
| Qwen2.5-1.5B | `Qwen/Qwen2.5-1.5B-Instruct` | 28 | ~3 GB | Better quality |
| Phi-3-mini | `microsoft/Phi-3-mini-4k-instruct` | 32 | ~7.5 GB | Best CPU quality |
| Llama-3.2-1B | `meta-llama/Llama-3.2-1B-Instruct` | 16 | ~2 GB | Requires HF login |
| Llama-3.2-3B | `meta-llama/Llama-3.2-3B-Instruct` | 28 | ~6 GB | Requires HF login |

For gated models (Llama), run `huggingface-cli login` first.

---

## Step 3 — Send an inference request (Terminal 3)

```bash
curl -s http://localhost:8001/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2.5-0.5b",
    "messages": [{"role": "user", "content": "What is 7 times 8? Answer in one word."}],
    "stream": false
  }' | python3 -m json.tool
```

Or use the test script:

```bash
.venv/bin/python scripts/test_lan_inference.py \
  --tracker http://localhost:8080 \
  --gateway http://localhost:8001
```

---

## Two-node split (same machine, two terminals)

Split Qwen2.5-0.5B's 24 layers across two node processes to test the sharded pipeline:

**Node A — layers 0–11 (tracker mode, serves chat completions):**
```bash
HF_HOME=/run/media/popov/d/DEV/models \
.venv/bin/meshnet-node start \
  --model-id Qwen/Qwen2.5-0.5B-Instruct \
  --shard-start 0 --shard-end 11 \
  --quantization bfloat16 \
  --tracker http://localhost:8080 \
  --port 8001
```

**Node B — layers 12–23:**
```bash
HF_HOME=/run/media/popov/d/DEV/models \
.venv/bin/meshnet-node start \
  --model-id Qwen/Qwen2.5-0.5B-Instruct \
  --shard-start 12 --shard-end 23 \
  --quantization bfloat16 \
  --tracker http://localhost:8080 \
  --port 8002
```

Send the request to Node A — it tokenizes, runs layers 0–13, passes binary
activations to Node B, and streams the final response back.

---

## Two-machine LAN test (Linux + Windows/WSL2)

See `docs/TWO_MACHINE_TEST.md` (created by US-018).

---

## Browse available models

```bash
# Show curated list with VRAM requirements
.venv/bin/meshnet-node models

# Browse HuggingFace Hub top-20 text-generation models
.venv/bin/meshnet-node models --browse
```

---

## Start with the interactive wizard

```bash
# First run: wizard detects GPU, shows model list, saves config
.venv/bin/meshnet-node

# Subsequent runs: starts directly from saved config
.venv/bin/meshnet-node

# Re-run wizard even with saved config
.venv/bin/meshnet-node --reset-config
```

---

## Start the relay node (for NAT traversal)

```bash
.venv/bin/pip install -e packages/relay
.venv/bin/meshnet-relay --port 8765
```

Nodes behind NAT connect to the relay and advertise their relay address to the
tracker. See `docs/adr/0010-p2p-gossip-and-nat-relay.md`.

---

## Run all tests

```bash
.venv/bin/python -m pytest -q
```