# Quickstart — Running a node and testing inference

Get from zero to a live inference request in **three terminals**: install once, start
the tracker, start a node, send a request.

Tested on: AMD Ryzen AI Max (Strix Halo APU), 124 GB RAM, Linux, CPU inference.

**Active development models** (what we run day-to-day):

| Role | `--model` / alias | HF repo | Notes |
|------|-------------------|---------|-------|
| Smoke tests, small splits | `Qwen/Qwen2.5-0.5B-Instruct` | same | 24 layers, ~1 GB BF16, no gating — default for new setups |
| Alpha / production target | `qwen3.6-35b-a3b` | `unsloth/Qwen3.6-35B-A3B` | 40 layers, ~72 GB BF16, hybrid linear-attention MoE; aliases include `Qwen3.6-35B-A3B`, `Qwen/Qwen3.6-35B-A3B` |

---

## At a glance

| Step | What | Terminals |
|------|------|-----------|
| **0** | Install Python packages | once per machine |
| **1** | Start tracker (and relay if needed) | 1–2 |
| **2** | Start node(s) | 1+ |
| **3** | Send inference request | 1 |

**Pick your connectivity mode** — this determines which flags you need on the node:

| Mode | When to use | Tracker URL | Node extras |
|------|-------------|-------------|-------------|
| **Local dev** | Everything on one machine | `http://localhost:8080` | none |
| **Direct LAN** | Node has a real LAN IP other machines can reach | `http://<tracker-ip>:8080` | `--host 0.0.0.0 --advertise-host <your-lan-ip>` + firewall |
| **Relay / public** | WSL2, NAT, 5G, or any unreachable inbound port | `https://ai.neuron.d-popov.com` (or your public URL) | none — relay handles routing |

> **WSL2:** not reachable from other LAN machines by default. Use the **relay / public**
> tracker URL, or run the node in native Windows PowerShell with direct LAN mode.

**Command prefix by shell** (used in examples below):

| Shell | Prefix | Model cache env |
|-------|--------|-----------------|
| Linux / WSL | `.venv/bin/` | `HF_HOME=/path/to/models` |
| Windows PowerShell | `.\.venv\Scripts\` | `$env:HF_HOME = "D:\DEV\models"` |

---

## 0. Install prerequisites (once per machine)

Editable installs point wrappers at this source tree — code edits apply without
reinstalling.

### Node machine — full install

<details>
<summary><strong>Linux / WSL</strong></summary>

```bash
cd /path/to/neuron-tai
python3 -m venv .venv
source .venv/bin/activate
.venv/bin/python -m pip install --upgrade pip setuptools wheel
.venv/bin/python -m pip install -e packages/tracker -e packages/node -e packages/p2p -e packages/gateway -e packages/relay
.venv/bin/python -m pip install "transformers>=5.12" accelerate
.venv/bin/meshnet-node --help
```

</details>

<details>
<summary><strong>Windows PowerShell (.venv)</strong></summary>

Requires Python 3.11+ and Git for Windows.

```powershell
cd D:\DEV\workspace\REPOS\git.d-popov.com\neuron-tai
python -m venv .venv
.\.venv\Scripts\Activate.ps1
.\.venv\Scripts\python.exe -m pip install --upgrade pip setuptools wheel
.\.venv\Scripts\python.exe -m pip install -e .\packages\tracker -e .\packages\node -e .\packages\p2p -e .\packages\gateway -e .\packages\relay
.\.venv\Scripts\python.exe -m pip install "transformers>=5.12" accelerate
.\.venv\Scripts\meshnet-node.exe --help
```

</details>

<details>
<summary><strong>Windows — conda/miniforge with CUDA (skip if using .venv above)</strong></summary>

```powershell
conda activate base
deactivate   # drop any layered .venv; safe no-op if none active
cd D:\DEV\workspace\REPOS\git.d-popov.com\neuron-tai
pip install -e packages\tracker -e packages\node -e packages\p2p -e packages\gateway -e packages\relay
pip install "transformers>=5.12" accelerate safetensors
python -c "import torch; print(torch.__version__, torch.cuda.is_available())"
# Expected: 2.x.x+cuXXX  True
```

If `torch` import fails despite pip saying "already satisfied", force-reinstall all
three together (never upgrade `torch` alone — breaks `torchvision`):

```powershell
pip install --force-reinstall torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
```

If `.venv\Scripts\meshnet-node.exe` shadows the conda binary, use the full path:
`C:\Users\<you>\miniforge3\Scripts\meshnet-node.exe`.

</details>

> Run Linux/WSL commands from **WSL**, not Git Bash. From Git Bash: `wsl`, then `cd`
> to the repo under `/mnt/d/...`.

### Tracker host — lightweight install

Tracker + relay only; skip node packages unless this machine also runs nodes.

```bash
git clone https://git.d-popov.com/popov/neuron-tai.git AI && cd AI
python3 -m venv .venv
.venv/bin/python -m pip install --upgrade pip setuptools wheel
.venv/bin/pip install -e packages/tracker -e packages/relay -e packages/gateway
```

### PyTorch variant

Install **one** torch line into the same env as `meshnet-node`:

| Hardware | Install |
|----------|---------|
| NVIDIA CUDA | `pip install torch` (default index) |
| CPU only | `pip install torch --index-url https://download.pytorch.org/whl/cpu` |
| AMD ROCm | `pip install torch --index-url https://download.pytorch.org/whl/rocm6.2` |

On Windows `.venv`, prefix with `.\.venv\Scripts\pip.exe`. Conda users with CUDA
torch already installed can skip this step.

### Qwen3.5/3.6-MoE notes

Applies to **`qwen3.6-35b-a3b`** and other hybrid linear-attention models. **`Qwen2.5-0.5B`**
does not need any of this — it is a standard transformer with no FLA fast path.

- **transformers ≥ 5.12 required** — older versions fail with
  `'Qwen3_5MoeConfig' object has no attribute 'vocab_size'`. Check:
  `python -c "import transformers; print(transformers.__version__)"`.
- **GPU fast path (optional)** — without it inference still works; startup prints
  `The fast path is not available…` and linear-attention layers use a slower PyTorch
  fallback. Install **only for your platform**:

  | Platform | Install | Notes |
  |----------|---------|-------|
  | **Native Windows + NVIDIA** | `pip install triton-windows` then `pip install flash-linear-attention` | **Fast path works.** FLA [officially supports `triton-windows`](https://github.com/fla-org/flash-linear-attention/pull/757) (tested Win11, PyTorch 2.10, triton-windows 3.6). Do **not** use the `[cuda]` extra on Windows — pip looks for Linux `triton` and fails. Do **not** install `causal-conv1d` — FLA ≥0.3.2 ships Triton conv1d; the separate package is Linux-only and breaks on Windows (`bare_metal_version` / nvcc errors). |
  | **Linux + NVIDIA CUDA** | `pip install flash-linear-attention[cuda]` | `causal-conv1d` optional (same FLA built-in conv1d note). Needs CUDA toolkit (`nvcc`) matching torch, or a prebuilt wheel. |
  | **Linux + AMD ROCm** | `pip install flash-linear-attention[rocm]` | Same optional `causal-conv1d` note. |

  **Windows verify** (after install):

  ```powershell
  python -c "import triton; import fla; print('triton', triton.__version__, 'fla ok')"
  ```

  `triton-windows` is also pulled by `meshnet-node` on Windows. Without it, Qwen3.6-MoE
  startup fails with misleading `Could not import module 'Qwen3_5MoeForCausalLM'`.

<details>
<summary><strong>Windows fast path — what failed and what actually works</strong></summary>

The command that failed — `pip install flash-linear-attention[cuda] causal-conv1d` — mixes
two different things:

1. **`flash-linear-attention[cuda]` on Windows** — wrong extra. `[cuda]` pulls PyPI
   `triton>=3.3`, which does not exist for Windows (`No matching distribution found`).
   Use plain `pip install flash-linear-attention` **after** `triton-windows` is already
   installed; FLA detects `triton-windows` and uses it.
2. **`causal-conv1d`** — separate Dao-AILab CUDA extension, **not required** for FLA or
   Qwen3.6 when FLA is installed. No official Windows wheels. Source builds need `nvcc`
   whose major version matches torch's CUDA (e.g. torch `+cu118` needs CUDA 11.8 toolkit,
   not 12.5). Community wheels exist for narrow Python/torch combos
   ([PR #46](https://github.com/Dao-AILab/causal-conv1d/pull/46)) but we skip them.

**Working Windows stack** (confirmed on this repo's dev machine: Python 3.12, torch
2.7.1+cu118, triton-windows 3.7.1, flash-linear-attention 0.5.0):

```powershell
pip install triton-windows
pip install -U flash-linear-attention
python -c "import triton; import fla; print('ok')"
```

If the fast-path warning persists after that, upgrade FLA to ≥0.5.1 (includes the
`triton-windows` detection from PR #757) and restart the node.

</details>

- `pip install nvidia-ml-py` silences the pynvml deprecation warning on NVIDIA hosts.

---

## 1. Start the tracker

### LAN tracker (private network, direct node reachability)

**Terminal 1:**

```bash
.venv/bin/meshnet-tracker start --host 0.0.0.0 --port 8080
# Optional devnet billing: --starting-credit 1 --devnet-topup 10
```

Expected: `Tracker listening on 0.0.0.0:8080`. Open the port on the host firewall
if other machines will join.

**Verify:**

```bash
curl -s http://localhost:8080/v1/network/map | python3 -m json.tool
curl -s http://192.168.0.179:8080/v1/network/map | python3 -m json.tool   # from another LAN machine
```

### Public tracker + relay (NAT / WSL2 / internet nodes)

Nodes behind NAT cannot receive inbound connections. Run **both** services on the
tracker host; the tracker advertises the relay URL in `/v1/network/map`.

**Terminal 1 — relay:**

```bash
.venv/bin/meshnet-relay --host 0.0.0.0 --port 8765
```

**Terminal 2 — tracker:**

```bash
.venv/bin/meshnet-tracker start --host 0.0.0.0 --port 8081 --relay-url wss://ai.neuron.d-popov.com/ws
```

**Verify:**

```bash
curl -s https://ai.neuron.d-popov.com/v1/network/map | python3 -m json.tool
```

Nodes should log `Relay connected — wss://…/rpc/<peer_id>` on startup.

<details>
<summary><strong>Nginx Proxy Manager setup (public hostname)</strong></summary>

Architecture:

```
Client → HTTPS → ai.neuron.d-popov.com (nginx)
                  ├─ /v1/*   → meshnet-tracker :8081
                  ├─ /ws     → meshnet-relay :8765  (node persistent outbound WS)
                  └─ /rpc/*  → meshnet-relay :8765  (caller opens WS per hop)
```

Use **one** proxy host. Route sub-paths via **Custom locations** — do not create
a second host for the same domain.

**Details tab** (default `/` → tracker):

| Field | Value |
|-------|--------|
| Domain Names | `ai.neuron.d-popov.com` |
| Scheme | `http` |
| Forward Hostname / IP | LAN IP of tracker machine (e.g. `192.168.0.179`) |
| Forward Port | `8081` |
| Websockets Support | ON |

**Custom locations** (both → relay port `8765`, sub-folder path empty):

| Location | Forward to |
|----------|--------------|
| `/ws` | `192.168.0.179:8765` |
| `/rpc` | `192.168.0.179:8765` |

**Advanced tab** (only if WebSocket upgrade fails):

```nginx
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $http_connection;
proxy_read_timeout 3600s;
proxy_send_timeout 3600s;
```

**Verify routing:**

```bash
curl -s https://ai.neuron.d-popov.com/v1/network/map | python3 -m json.tool
curl -sI https://ai.neuron.d-popov.com/ws
curl -sI https://ai.neuron.d-popov.com/rpc/test-peer
```

</details>

---

## 2. Start a node

**Starter model:** `Qwen/Qwen2.5-0.5B-Instruct` — 0.5B params, ~1 GB BF16, 24 layers,
no HuggingFace gating. Best for first-time setup.

**Alpha model:** `qwen3.6-35b-a3b` — 40 layers, ~72 GB BF16 download, MoE with hybrid
linear attention. On Windows install `triton-windows` + `flash-linear-attention`; on Linux
GPU use `flash-linear-attention[cuda]`. Tracker accepts the alias or full repo id (`unsloth/Qwen3.6-35B-A3B`).

Downloads cache under `~/.meshnet/models/` (or `$HF_HOME` / `$env:HF_HOME`).

Shard range is auto-detected from the curated catalog. For unknown repos the node
fetches only `config.json`. Override with `--shard-start` / `--shard-end` for partial
shards or multi-node splits.

### Core command

Replace `<tracker-url>` and adjust the prefix for your shell (see table above).

**Linux / WSL:**

```bash
HF_HOME=/path/to/models .venv/bin/meshnet-node start --tracker <tracker-url> --model Qwen/Qwen2.5-0.5B-Instruct --quantization bfloat16
```

**Windows PowerShell:**

```powershell
$env:HF_HOME = "D:\DEV\models"
.\.venv\Scripts\meshnet-node.exe start --tracker <tracker-url> --model Qwen/Qwen2.5-0.5B-Instruct --quantization bfloat16
```

### Ready-to-run examples

**Local dev (same machine as tracker):**

```bash
HF_HOME=/path/to/models .venv/bin/meshnet-node start --tracker http://localhost:8080 --model Qwen/Qwen2.5-0.5B-Instruct --quantization bfloat16 --port 8001
```

**Public / relay (works from WSL2, NAT, 5G — no extra flags):**

```bash
.venv/bin/meshnet-node start --tracker https://ai.neuron.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct
```

**Alpha model (Qwen3.6, Windows GPU — enable fast path):**

```powershell
$env:HF_HOME = "D:\DEV\models"
pip install triton-windows
pip install -U flash-linear-attention
meshnet-node start --tracker http://192.168.0.179:8080 --model qwen3.6-35b-a3b --quantization bfloat16
```

Do not add `causal-conv1d` or `flash-linear-attention[cuda]` on Windows (see Qwen3.5/3.6 notes).

**Alpha model (Qwen3.6, Linux GPU — with fast path):**

```bash
HF_HOME=/path/to/models .venv/bin/meshnet-node start --tracker <tracker-url> --model qwen3.6-35b-a3b --quantization bfloat16
# Install once on that machine: pip install flash-linear-attention[cuda]
```

After the first node registers a model, later nodes can join with only the tracker
URL (shard auto-assigned):

```bash
.venv/bin/meshnet-node start --tracker https://ai.neuron.d-popov.com
```

**Direct LAN (Windows node reachable by IP):**

```powershell
$env:HF_HOME = "D:\DEV\models"
.\.venv\Scripts\meshnet-node.exe start --tracker http://192.168.0.179:8081 --model Qwen/Qwen2.5-0.5B-Instruct --shard-start 12 --shard-end 23 --quantization bfloat16 --host 0.0.0.0 --advertise-host 192.168.0.42 --port 8005
```

<details>
<summary><strong>Windows direct LAN — firewall and IP checklist</strong></summary>

1. Find LAN IP: `ipconfig` — use active Ethernet/Wi-Fi IPv4 (e.g. `192.168.0.42`).
   Avoid WSL/Docker/Hyper-V addresses (`172.x.x.x`).
2. Allow inbound port (Administrator PowerShell, once):

```powershell
New-NetFirewallRule -DisplayName "Meshnet node 8005" -Direction Inbound -Action Allow -Protocol TCP -LocalPort 8005
```

3. Verify from tracker machine:

```bash
curl http://192.168.0.42:8005/v1/health
```

404/501 is fine — it proves TCP reached the node. Timeout = check firewall,
`--host 0.0.0.0`, and `--advertise-host`.

</details>

<details>
<summary><strong>Two-node split (same or different machines)</strong></summary>

**Node A — layers 0–11 (head, serves chat):**

```bash
HF_HOME=/path/to/models .venv/bin/meshnet-node start --tracker <tracker-url> --model Qwen/Qwen2.5-0.5B-Instruct --shard-start 0 --shard-end 11 --quantization bfloat16 --port 8001
```

**Node B — layers 12–23:**

```bash
HF_HOME=/path/to/models .venv/bin/meshnet-node start --tracker <tracker-url> --model Qwen/Qwen2.5-0.5B-Instruct --shard-start 12 --shard-end 23 --quantization bfloat16 --port 8002
```

Send inference to Node A. For cross-machine LAN tests see `docs/TWO_MACHINE_TEST.md`.

</details>

### Useful flags

| Flag | Purpose |
|------|---------|
| `--port 8001` | Fixed listen port (default: first free ≥ 7000) |
| `--host 0.0.0.0` | Bind all interfaces (needed for direct LAN) |
| `--advertise-host <ip>` | LAN IP the tracker tells other nodes (direct LAN only) |
| `--shard-start N --shard-end M` | Partial layer range |
| `--debug` | Verbose per-hop pipeline logs (noisy; off by default) |

`--host 0.0.0.0` binds locally; `--advertise-host` is what peers use for direct
hops. Omit both when using the relay path.

### Expected output

```
  Auto-detected 24 layers → shard 0–23
  Relay connected — wss://ai.neuron.d-popov.com/rpc/abc1def2ef3f4567   # relay mode only
================================
meshnet-node ready
  Wallet:       <address>
  Model ID:     Qwen/Qwen2.5-0.5B-Instruct
  Shard:        layers 0–23; 24 of 24
  Quantization: bfloat16
  Endpoint:     http://<host>:8001
  Node ID:      <id>
  Hardware:     CPU
================================
```

The `Endpoint` is the local address. In relay mode, peers reach this node via
`wss://<relay>/rpc/<peer_id>` instead.

### Other CPU-friendly models

| Model | `--model` / alias | Layers | BF16 size | Notes |
|-------|-------------------|--------|-----------|-------|
| **Qwen2.5-0.5B** (dev default) | `Qwen/Qwen2.5-0.5B-Instruct` | 24 | ~1 GB | Fastest, no gating |
| **Qwen3.6-35B-A3B** (alpha) | `qwen3.6-35b-a3b` | 40 | ~72 GB | MoE; needs transformers ≥5.12; see Qwen3.5/3.6 notes |
| Qwen2.5-1.5B | `Qwen/Qwen2.5-1.5B-Instruct` | 28 | ~3 GB | Better quality |
| Phi-3-mini | `microsoft/Phi-3-mini-4k-instruct` | 32 | ~7.5 GB | Best CPU quality |
| Llama-3.2-1B | `meta-llama/Llama-3.2-1B-Instruct` | 16 | ~2 GB | Requires HF login |
| Llama-3.2-3B | `meta-llama/Llama-3.2-3B-Instruct` | 28 | ~6 GB | Requires HF login |

For gated models (Llama): `huggingface-cli login` first.

Browse more: `.venv/bin/meshnet-node models` or `.venv/bin/meshnet-node models --browse`.

---

## 3. Send an inference request

**Terminal 3** — direct to the head node (replace port if you omitted `--port`):

```bash
curl -s http://localhost:8001/v1/chat/completions -H "Content-Type: application/json" -d '{"model": "qwen2.5-0.5b", "messages": [{"role": "user", "content": "What is 7 times 8? Answer in one word."}], "stream": false}' | python3 -m json.tool
```

**Via tracker** (tests routing / proxying):

```bash
curl -s http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{"model": "qwen2.5-0.5b", "messages": [{"role": "user", "content": "What is 7 times 8? Answer in one word."}], "stream": false}' | python3 -m json.tool
```

**Public tracker:**

```bash
curl -s https://ai.neuron.d-popov.com/v1/chat/completions -H "Content-Type: application/json" -H "Authorization: Bearer sk-mesh-<your-key>" -d '{"model": "Qwen/Qwen2.5-0.5B-Instruct", "messages": [{"role": "user", "content": "What is 7 times 8?"}], "stream": false}' | python3 -m json.tool
```

**Test script:**

```bash
.venv/bin/python scripts/test_lan_inference.py --tracker http://localhost:8080 --gateway http://localhost:8001
```

**Verify registration on public tracker:**

```bash
curl -s "https://ai.neuron.d-popov.com/v1/network/map" | python3 -m json.tool
curl -s "https://ai.neuron.d-popov.com/v1/route?model=qwen2.5-0.5b" | python3 -m json.tool
```

<details>
<summary><strong>Accounts, API keys, and credit (billing-enabled trackers)</strong></summary>

Public trackers require a real API key for `/v1/chat/completions`. Unknown bearer →
`401`; zero balance → `402 insufficient balance`.

**Dashboard:** open `https://<tracker>/dashboard`, register, click **+ new key**.
With `--starting-credit`, the first key is pre-funded. With `--devnet-topup`, use
**+$N (devnet)** to refill during testing.

**Curl flow:**

```bash
curl -s https://<tracker>/v1/auth/register -H "Content-Type: application/json" -d '{"email": "you@example.com", "password": "hunter22-or-better"}'
curl -s https://<tracker>/v1/account/keys -X POST -H "Authorization: Bearer <session_token>"
curl -s https://<tracker>/v1/account -H "Authorization: Bearer <session_token>"
curl -s https://<tracker>/v1/account/topup -X POST -H "Authorization: Bearer <session_token>" -H "Content-Type: application/json" -d '{"api_key": "sk-mesh-..."}'
```

Operator defaults: `--starting-credit` and `--devnet-topup` both default to 1 USDT.
Set both to 0 on mainnet.

</details>

---

## Reference

### How relay hops work

When node A forwards activations to node B (behind NAT):

1. Tracker injects `X-Meshnet-Route` with `relay_addr` for behind-NAT hops.
2. Node A opens WebSocket to `wss://relay/rpc/{peer_id_B}`.
3. Relay forwards `relay-http-request` to Node B's persistent connection.
4. Node B processes `/forward`, returns `relay-http-response`.
5. Relay sends response back to Node A; Node A continues the pipeline.

Activations (bfloat16) are Base64-encoded in JSON — no precision loss. On relay
failure the node logs a warning and falls back to direct HTTP before erroring.

### Interactive wizard

```bash
.venv/bin/meshnet-node              # first run: wizard; later: saved config
.venv/bin/meshnet-node --reset-config
```

### Run all tests

```bash
.venv/bin/python -m pytest -q
```