# Quickstart — Running a node and testing inference

Get from zero to a live inference request in **three terminals**: install once, start
the tracker, start a node, send a request.

Tested on: AMD Ryzen AI Max (Strix Halo APU), 124 GB RAM, Linux CPU inference.
ROCm GPU setup is covered below, but must be verified on the host because ROCm
support depends on the exact AMD GPU/APU, kernel, driver, and ROCm runtime.

**Active development models** (what we run day-to-day):

| Role | `--model` / alias | HF repo | Notes |
|------|-------------------|---------|-------|
| Smoke tests, small splits | `Qwen/Qwen2.5-0.5B-Instruct` | same | 24 layers, ~1 GB BF16, no gating — default for new setups |
| Alpha / production target | `qwen3.6-35b-a3b` | `unsloth/Qwen3.6-35B-A3B` | 40 layers, ~72 GB BF16, hybrid linear-attention MoE; aliases include `Qwen3.6-35B-A3B`, `Qwen/Qwen3.6-35B-A3B` |

---

## At a glance

| Step | What | Terminals |
|------|------|-----------|
| **0** | Install Python packages | once per machine |
| **1** | Start tracker (and relay if needed) | 1–2 |
| **2** | Start node(s) | 1+ |
| **3** | Send inference request | 1 |

**Pick your connectivity mode** — this determines which flags you need on the node:

| Mode | When to use | Tracker URL | Node extras |
|------|-------------|-------------|-------------|
| **Local dev** | Everything on one machine | `http://localhost:8080` | none |
| **Direct LAN** | Node has a real LAN IP other machines can reach | `http://<tracker-ip>:8080` | `--host 0.0.0.0 --advertise-host <your-lan-ip>` + firewall |
| **Relay / public** | WSL2, NAT, 5G, or any unreachable inbound port | `https://ai.neuron.d-popov.com` (or your public URL) | none — relay handles routing |

> **WSL2:** not reachable from other LAN machines by default. Use the **relay / public**
> tracker URL, or run the node in native Windows PowerShell with direct LAN mode.

**Command prefix by shell** (used in examples below):

| Shell | Prefix | Model cache env |
|-------|--------|-----------------|
| Linux / WSL | `.venv/bin/` | `HF_HOME=/path/to/models` |
| Windows PowerShell | `.\.venv\Scripts\` | `$env:HF_HOME = "D:\DEV\models"` |

---

## 0. Install prerequisites (once per machine)

Editable installs point wrappers at this source tree — code edits apply without
reinstalling.

### Node machine — full install

<details>
<summary><strong>Linux / WSL</strong></summary>

```bash
cd /path/to/neuron-tai
python3 -m venv .venv
source .venv/bin/activate
.venv/bin/python -m pip install --upgrade pip setuptools wheel
.venv/bin/python -m pip install -e packages/tracker -e packages/node -e packages/p2p -e packages/gateway -e packages/relay
.venv/bin/python -m pip install "transformers>=5.12" accelerate
.venv/bin/meshnet-node --help
```

</details>

<details>
<summary><strong>Windows PowerShell (.venv)</strong></summary>

Requires Python 3.11+ and Git for Windows.

```powershell
cd D:\DEV\workspace\REPOS\git.d-popov.com\neuron-tai
python -m venv .venv
.\.venv\Scripts\Activate.ps1
.\.venv\Scripts\python.exe -m pip install --upgrade pip setuptools wheel
.\.venv\Scripts\python.exe -m pip install -e .\packages\tracker -e .\packages\node -e .\packages\p2p -e .\packages\gateway -e .\packages\relay
.\.venv\Scripts\python.exe -m pip install "transformers>=5.12" accelerate
.\.venv\Scripts\meshnet-node.exe --help
```

</details>

<details>
<summary><strong>Windows — conda/miniforge with CUDA (skip if using .venv above)</strong></summary>

```powershell
conda activate base
deactivate   # drop any layered .venv; safe no-op if none active
cd D:\DEV\workspace\REPOS\git.d-popov.com\neuron-tai
pip install -e packages\tracker -e packages\node -e packages\p2p -e packages\gateway -e packages\relay
pip install "transformers>=5.12" accelerate safetensors
python -c "import torch; print(torch.__version__, torch.cuda.is_available())"
# Expected: 2.x.x+cuXXX  True
```

If `torch` import fails despite pip saying "already satisfied", force-reinstall all
three together (never upgrade `torch` alone — breaks `torchvision`):

```powershell
pip install --force-reinstall torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
```

If `.venv\Scripts\meshnet-node.exe` shadows the conda binary, use the full path:
`C:\Users\<you>\miniforge3\Scripts\meshnet-node.exe`.

</details>

> Run Linux/WSL commands from **WSL**, not Git Bash. From Git Bash: `wsl`, then `cd`
> to the repo under `/mnt/d/...`.

### Tracker host — lightweight install

Tracker + relay only; skip node packages unless this machine also runs nodes.

```bash
git clone https://git.d-popov.com/popov/neuron-tai.git AI && cd AI
python3 -m venv .venv
.venv/bin/python -m pip install --upgrade pip setuptools wheel
.venv/bin/pip install -e packages/tracker -e packages/relay -e packages/gateway
```

### PyTorch variant

Install **one** torch line into the same env as `meshnet-node`:

| Hardware | Install |
|----------|---------|
| NVIDIA CUDA | `pip install torch` (default index) |
| CPU only | `pip install torch --index-url https://download.pytorch.org/whl/cpu` |
| AMD ROCm | `pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm6.3` |

On Windows `.venv`, prefix with `.\.venv\Scripts\pip.exe`. Conda users with CUDA
torch already installed can skip this step.

### Linux AMD ROCm GPU install

Use this when the node machine has an AMD GPU/APU and you want PyTorch to run on
ROCm instead of CPU. The Python wheel is not enough by itself: the host must have
working AMD GPU device access and a compatible ROCm runtime.

**Host prerequisites:**

1. Confirm the AMD GPU is visible:

```bash
lspci | grep -Ei 'vga|3d|display|amd|ati'
ls -l /dev/kfd /dev/dri/renderD* 2>/dev/null
```

2. Make sure the node user can access GPU devices. AMD ROCm documents the normal
   Linux permission path as membership in both `video` and `render`:

```bash
groups
sudo usermod -a -G video,render "$LOGNAME"
# log out and back in before continuing
```

3. Confirm the ROCm runtime tools work if they are installed:

```bash
rocminfo | head
```

If `rocminfo` is missing or cannot see the GPU, fix the host ROCm install first.
Do not debug `meshnet-node` until this works.

**Install ROCm PyTorch into the node env:**

```bash
cd /path/to/neuron-tai
python3 -m venv .venv-rocm
source .venv-rocm/bin/activate
python -m pip install --upgrade pip setuptools wheel
python -m pip install -e packages/tracker -e packages/node -e packages/p2p -e packages/gateway -e packages/relay
python -m pip install "transformers>=5.12" accelerate safetensors
python -m pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm6.3
```

Keep this separate from a known-good CPU `.venv` until ROCm is verified on that
machine. ROCm wheels are large and host-runtime-sensitive; a failed ROCm install
should not break the CPU fallback environment.

**Verify PyTorch sees ROCm:**

```bash
python - <<'PY'
import torch
print("torch", torch.__version__)
print("hip", torch.version.hip)
print("cuda api available", torch.cuda.is_available())
if torch.cuda.is_available():
    print("device", torch.cuda.get_device_name(0))
    x = torch.ones((1,), device="cuda")
    torch.cuda.synchronize()
    print("tensor", x)
PY
```

Expected: `torch.version.hip` is not `None`, `torch.cuda.is_available()` is
`True`, and the tensor allocation succeeds. PyTorch intentionally exposes ROCm
through the `torch.cuda` API.

**Start an AMD ROCm node:**

```bash
HF_HOME=/path/to/models .venv-rocm/bin/meshnet-node start \
  --tracker <tracker-url> \
  --model Qwen/Qwen2.5-0.5B-Instruct \
  --quantization bfloat16
```

For the Qwen3.6 alpha model on Linux ROCm, install the optional FLA ROCm fast
path in the same env:

```bash
.venv-rocm/bin/pip install 'flash-linear-attention[rocm]'
HF_HOME=/path/to/models .venv-rocm/bin/meshnet-node start \
  --tracker <tracker-url> \
  --model qwen3.6-35b-a3b \
  --quantization bfloat16
```

**Troubleshooting notes:**

- `torch.version.hip is None` means you installed a CPU/CUDA torch build, not ROCm.
- `torch.cuda.is_available() == False` with a ROCm build usually means host driver,
  permissions, unsupported hardware, or missing runtime libraries.
- Missing libraries such as `libamdhip64.so`, `libMIOpen.so`, `librocsolver.so`,
  or `libroctx64.so` are host ROCm runtime problems, not meshnet-node problems.
- Some AMD APUs and consumer GPUs require newer ROCm/Radeon support than server
  Instinct cards. Check AMD's ROCm Radeon/Ryzen support matrix for the exact model.

### Qwen3.5/3.6-MoE notes

Applies to **`qwen3.6-35b-a3b`** and other hybrid linear-attention models. **`Qwen2.5-0.5B`**
does not need any of this — it is a standard transformer with no FLA fast path.

- **transformers ≥ 5.12 required** — older versions fail with
  `'Qwen3_5MoeConfig' object has no attribute 'vocab_size'`. Check:
  `python -c "import transformers; print(transformers.__version__)"`.
- **GPU fast path (optional)** — without it inference still works; startup prints
  `The fast path is not available…` and linear-attention layers use a slower PyTorch
  fallback. **The fast path runs on NVIDIA CUDA GPUs on both Linux and native
  Windows** — the FLA kernels are Triton-compiled, and `triton-windows` compiles them
  for CUDA on Windows just like Linux Triton does. Only the pip command differs per
  platform. Install **only for your platform**:

  | Platform | Install | Notes |
  |----------|---------|-------|
  | **Native Windows + NVIDIA CUDA** | `pip install triton-windows` then `pip install flash-linear-attention` | **Fast path works on the CUDA GPU** — no CUDA toolkit / `nvcc` needed; `triton-windows` bundles its own compiler. FLA [officially supports `triton-windows`](https://github.com/fla-org/flash-linear-attention/pull/757) (tested Win11, PyTorch 2.10, triton-windows 3.6). Do **not** use the `[cuda]` extra on Windows — that extra only pins Linux PyPI `triton` and fails; it is a packaging name, not a GPU requirement. Do **not** install `causal-conv1d` — FLA ≥0.3.2 ships Triton conv1d; the separate package is Linux-only and breaks on Windows (`bare_metal_version` / nvcc errors). |
  | **Linux + NVIDIA CUDA** | `pip install flash-linear-attention[cuda]` | `causal-conv1d` optional (same FLA built-in conv1d note). Needs CUDA toolkit (`nvcc`) matching torch, or a prebuilt wheel. |
  | **Linux + AMD ROCm** | `pip install flash-linear-attention[rocm]` | Same optional `causal-conv1d` note. |

  **Windows verify** (after install):

  ```powershell
  python -c "import triton; import fla; print('triton', triton.__version__, 'fla ok')"
  ```

  `triton-windows` is also pulled by `meshnet-node` on Windows. Without it, Qwen3.6-MoE
  startup fails with misleading `Could not import module 'Qwen3_5MoeForCausalLM'`.

<details>
<summary><strong>Windows fast path — what failed and what actually works</strong></summary>

The command that failed — `pip install flash-linear-attention[cuda] causal-conv1d` — mixes
two different things:

1. **`flash-linear-attention[cuda]` on Windows** — wrong extra. `[cuda]` pulls PyPI
   `triton>=3.3`, which does not exist for Windows (`No matching distribution found`).
   Use plain `pip install flash-linear-attention` **after** `triton-windows` is already
   installed; FLA detects `triton-windows` and uses it.
2. **`causal-conv1d`** — separate Dao-AILab CUDA extension, **not required** for FLA or
   Qwen3.6 when FLA is installed. No official Windows wheels. Source builds need `nvcc`
   whose major version matches torch's CUDA (e.g. torch `+cu118` needs CUDA 11.8 toolkit,
   not 12.5). Community wheels exist for narrow Python/torch combos
   ([PR #46](https://github.com/Dao-AILab/causal-conv1d/pull/46)) but we skip them.

**Working Windows stack** (confirmed on this repo's dev machine: Python 3.12, torch
2.7.1+cu118, triton-windows 3.7.1, flash-linear-attention 0.5.0):

```powershell
pip install triton-windows
pip install -U flash-linear-attention
python -c "import triton; import fla; print('ok')"
```

If the fast-path warning persists after that, upgrade FLA to ≥0.5.1 (includes the
`triton-windows` detection from PR #757) and restart the node.

</details>

- `pip install nvidia-ml-py` silences the pynvml deprecation warning on NVIDIA hosts.

---

## 1. Start the tracker

### LAN tracker (private network, direct node reachability)

**Terminal 1:**

```bash
.venv/bin/meshnet-tracker start --host 0.0.0.0 --port 8080
# Optional devnet billing: --starting-credit 1 --devnet-topup 10
```

Expected: `Tracker listening on 0.0.0.0:8080`. Open the port on the host firewall
if other machines will join.

**Verify:**

```bash
curl -s http://localhost:8080/v1/network/map | python3 -m json.tool
curl -s http://192.168.0.179:8080/v1/network/map | python3 -m json.tool   # from another LAN machine
```

### Public tracker + relay (NAT / WSL2 / internet nodes)

Nodes behind NAT cannot receive inbound connections. Run **both** services on the
tracker host; the tracker advertises the relay URL in `/v1/network/map`.

**Terminal 1 — relay:**

```bash
.venv/bin/meshnet-relay --host 0.0.0.0 --port 8765
```

**Terminal 2 — tracker:**

```bash
.venv/bin/meshnet-tracker start --host 0.0.0.0 --port 8081 --relay-url wss://ai.neuron.d-popov.com/ws
```

**Verify:**

```bash
curl -s https://ai.neuron.d-popov.com/v1/network/map | python3 -m json.tool
```

Nodes should log `Relay connected — wss://…/rpc/<peer_id>` on startup.

<details>
<summary><strong>Nginx Proxy Manager setup (public hostname)</strong></summary>

Architecture:

```
Client → HTTPS → ai.neuron.d-popov.com (nginx)
                  ├─ /v1/*   → meshnet-tracker :8081
                  ├─ /ws     → meshnet-relay :8765  (node persistent outbound WS)
                  └─ /rpc/*  → meshnet-relay :8765  (caller opens WS per hop)
```

Use **one** proxy host. Route sub-paths via **Custom locations** — do not create
a second host for the same domain.

**Details tab** (default `/` → tracker):

| Field | Value |
|-------|--------|
| Domain Names | `ai.neuron.d-popov.com` |
| Scheme | `http` |
| Forward Hostname / IP | LAN IP of tracker machine (e.g. `192.168.0.179`) |
| Forward Port | `8081` |
| Websockets Support | ON |

**Custom locations** (both → relay port `8765`, sub-folder path empty):

| Location | Forward to |
|----------|--------------|
| `/ws` | `192.168.0.179:8765` |
| `/rpc` | `192.168.0.179:8765` |

**Advanced tab** (only if WebSocket upgrade fails):

```nginx
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $http_connection;
proxy_read_timeout 3600s;
proxy_send_timeout 3600s;
```

**Verify routing:**

```bash
curl -s https://ai.neuron.d-popov.com/v1/network/map | python3 -m json.tool
curl -sI https://ai.neuron.d-popov.com/ws
curl -sI https://ai.neuron.d-popov.com/rpc/test-peer
```

</details>

---

## 2. Start a node

**Starter model:** `Qwen/Qwen2.5-0.5B-Instruct` — 0.5B params, ~1 GB BF16, 24 layers,
no HuggingFace gating. Best for first-time setup.

**Alpha model:** `qwen3.6-35b-a3b` — 40 layers, ~72 GB BF16 download, MoE with hybrid
linear attention. On Windows install `triton-windows` + `flash-linear-attention`; on Linux
GPU use `flash-linear-attention[cuda]`. Tracker accepts the alias or full repo id (`unsloth/Qwen3.6-35B-A3B`).

Downloads cache under `~/.meshnet/models/` (or `$HF_HOME` / `$env:HF_HOME`).

Shard range is auto-detected from the curated catalog. For unknown repos the node
fetches only `config.json`. Override with `--shard-start` / `--shard-end` for partial
shards or multi-node splits.

### Core command

Replace `<tracker-url>` and adjust the prefix for your shell (see table above).

**Linux / WSL:**

```bash
HF_HOME=/path/to/models .venv/bin/meshnet-node start --tracker <tracker-url> --model Qwen/Qwen2.5-0.5B-Instruct --quantization bfloat16
```

**Windows PowerShell:**

```powershell
$env:HF_HOME = "D:\DEV\models"
.\.venv\Scripts\meshnet-node.exe start --tracker <tracker-url> --model Qwen/Qwen2.5-0.5B-Instruct --quantization bfloat16
```

### Ready-to-run examples

**Local dev (same machine as tracker):**

```bash
HF_HOME=/path/to/models .venv/bin/meshnet-node start --tracker http://localhost:8080 --model Qwen/Qwen2.5-0.5B-Instruct --quantization bfloat16 --port 8001
```

**Public / relay (works from WSL2, NAT, 5G — no extra flags):**

```bash
.venv/bin/meshnet-node start --tracker https://ai.neuron.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct
```

**Alpha model (Qwen3.6, Windows GPU — enable fast path):**

```powershell
$env:HF_HOME = "D:\DEV\models"
pip install triton-windows
pip install -U flash-linear-attention
meshnet-node start --tracker http://192.168.0.179:8080 --model qwen3.6-35b-a3b --quantization bfloat16
```

Do not add `causal-conv1d` or `flash-linear-attention[cuda]` on Windows (see Qwen3.5/3.6 notes).

**Alpha model (Qwen3.6, Linux NVIDIA GPU — with fast path):**

```bash
HF_HOME=/path/to/models .venv/bin/meshnet-node start --tracker <tracker-url> --model qwen3.6-35b-a3b --quantization bfloat16
# Install once on that machine: pip install flash-linear-attention[cuda]
```

**Alpha model (Qwen3.6, Linux AMD ROCm GPU — with fast path):**

```bash
HF_HOME=/path/to/models .venv-rocm/bin/meshnet-node start --tracker <tracker-url> --model qwen3.6-35b-a3b --quantization bfloat16
# Install once on that machine: .venv-rocm/bin/pip install 'flash-linear-attention[rocm]'
```

After the first node registers a model, later nodes can join with only the tracker
URL (shard auto-assigned):

```bash
.venv/bin/meshnet-node start --tracker https://ai.neuron.d-popov.com
```

**Direct LAN (Windows node reachable by IP):**

```powershell
$env:HF_HOME = "D:\DEV\models"
.\.venv\Scripts\meshnet-node.exe start --tracker http://192.168.0.179:8081 --model Qwen/Qwen2.5-0.5B-Instruct --shard-start 12 --shard-end 23 --quantization bfloat16 --host 0.0.0.0 --advertise-host 192.168.0.42 --port 8005
```

<details>
<summary><strong>Windows direct LAN — firewall and IP checklist</strong></summary>

1. Find LAN IP: `ipconfig` — use active Ethernet/Wi-Fi IPv4 (e.g. `192.168.0.42`).
   Avoid WSL/Docker/Hyper-V addresses (`172.x.x.x`).
2. Allow inbound port (Administrator PowerShell, once):

```powershell
New-NetFirewallRule -DisplayName "Meshnet node 8005" -Direction Inbound -Action Allow -Protocol TCP -LocalPort 8005
```

3. Verify from tracker machine:

```bash
curl http://192.168.0.42:8005/v1/health
```

404/501 is fine — it proves TCP reached the node. Timeout = check firewall,
`--host 0.0.0.0`, and `--advertise-host`.

</details>

<details>
<summary><strong>Two-node split (same or different machines)</strong></summary>

**Node A — layers 0–11 (head, serves chat):**

```bash
HF_HOME=/path/to/models .venv/bin/meshnet-node start --tracker <tracker-url> --model Qwen/Qwen2.5-0.5B-Instruct --shard-start 0 --shard-end 11 --quantization bfloat16 --port 8001
```

**Node B — layers 12–23:**

```bash
HF_HOME=/path/to/models .venv/bin/meshnet-node start --tracker <tracker-url> --model Qwen/Qwen2.5-0.5B-Instruct --shard-start 12 --shard-end 23 --quantization bfloat16 --port 8002
```

Send inference to Node A. For cross-machine LAN tests see `docs/TWO_MACHINE_TEST.md`.

</details>

### Useful flags

| Flag | Purpose |
|------|---------|
| `--port 8001` | Fixed listen port (default: first free ≥ 7000) |
| `--host 0.0.0.0` | Bind all interfaces (needed for direct LAN) |
| `--advertise-host <ip>` | LAN IP the tracker tells other nodes (direct LAN only) |
| `--shard-start N --shard-end M` | Partial layer range |
| `--debug` | Verbose per-hop pipeline logs (noisy; off by default) |

`--host 0.0.0.0` binds locally; `--advertise-host` is what peers use for direct
hops. Omit both when using the relay path.

### Expected output

```
  Auto-detected 24 layers → shard 0–23
  Relay connected — wss://ai.neuron.d-popov.com/rpc/abc1def2ef3f4567   # relay mode only
================================
meshnet-node ready
  Wallet:       <address>
  Model ID:     Qwen/Qwen2.5-0.5B-Instruct
  Shard:        layers 0–23; 24 of 24
  Quantization: bfloat16
  Endpoint:     http://<host>:8001
  Node ID:      <id>
  Hardware:     CPU
================================
```

The `Endpoint` is the local address. In relay mode, peers reach this node via
`wss://<relay>/rpc/<peer_id>` instead.

### Other CPU-friendly models

| Model | `--model` / alias | Layers | BF16 size | Notes |
|-------|-------------------|--------|-----------|-------|
| **Qwen2.5-0.5B** (dev default) | `Qwen/Qwen2.5-0.5B-Instruct` | 24 | ~1 GB | Fastest, no gating |
| **Qwen3.6-35B-A3B** (alpha) | `qwen3.6-35b-a3b` | 40 | ~72 GB | MoE; needs transformers ≥5.12; see Qwen3.5/3.6 notes |
| Qwen2.5-1.5B | `Qwen/Qwen2.5-1.5B-Instruct` | 28 | ~3 GB | Better quality |
| Phi-3-mini | `microsoft/Phi-3-mini-4k-instruct` | 32 | ~7.5 GB | Best CPU quality |
| Llama-3.2-1B | `meta-llama/Llama-3.2-1B-Instruct` | 16 | ~2 GB | Requires HF login |
| Llama-3.2-3B | `meta-llama/Llama-3.2-3B-Instruct` | 28 | ~6 GB | Requires HF login |

For gated models (Llama): `huggingface-cli login` first.

Browse more: `.venv/bin/meshnet-node models` or `.venv/bin/meshnet-node models --browse`.

---

## 3. Send an inference request

**Terminal 3** — direct to the head node (replace port if you omitted `--port`):

```bash
curl -s http://localhost:8001/v1/chat/completions -H "Content-Type: application/json" -d '{"model": "qwen2.5-0.5b", "messages": [{"role": "user", "content": "What is 7 times 8? Answer in one word."}], "stream": false}' | python3 -m json.tool
```

**Via tracker** (tests routing / proxying):

```bash
curl -s http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{"model": "qwen2.5-0.5b", "messages": [{"role": "user", "content": "What is 7 times 8? Answer in one word."}], "stream": false}' | python3 -m json.tool
```

**Public tracker:**

```bash
curl -s https://ai.neuron.d-popov.com/v1/chat/completions -H "Content-Type: application/json" -H "Authorization: Bearer sk-mesh-<your-key>" -d '{"model": "Qwen/Qwen2.5-0.5B-Instruct", "messages": [{"role": "user", "content": "What is 7 times 8?"}], "stream": false}' | python3 -m json.tool
```

**Test script:**

```bash
.venv/bin/python scripts/test_lan_inference.py --tracker http://localhost:8080 --gateway http://localhost:8001
```

**Verify registration on public tracker:**

```bash
curl -s "https://ai.neuron.d-popov.com/v1/network/map" | python3 -m json.tool
curl -s "https://ai.neuron.d-popov.com/v1/route?model=qwen2.5-0.5b" | python3 -m json.tool
```

<details>
<summary><strong>Accounts, API keys, and credit (billing-enabled trackers)</strong></summary>

Public trackers require a real API key for `/v1/chat/completions`. Unknown bearer →
`401`; zero balance → `402 insufficient balance`.

**Dashboard:** open `https://<tracker>/dashboard`, register, click **+ new key**.
With `--starting-credit`, the first key is pre-funded. With `--devnet-topup`, use
**+$N (devnet)** to refill during testing.

**Curl flow:**

```bash
curl -s https://<tracker>/v1/auth/register -H "Content-Type: application/json" -d '{"email": "you@example.com", "password": "hunter22-or-better"}'
curl -s https://<tracker>/v1/account/keys -X POST -H "Authorization: Bearer <session_token>"
curl -s https://<tracker>/v1/account -H "Authorization: Bearer <session_token>"
curl -s https://<tracker>/v1/account/topup -X POST -H "Authorization: Bearer <session_token>" -H "Content-Type: application/json" -d '{"api_key": "sk-mesh-..."}'
```

Operator defaults: `--starting-credit` and `--devnet-topup` both default to 1 USDT.
Set both to 0 on mainnet.

</details>

---

## Reference

### How relay hops work

When node A forwards activations to node B (behind NAT):

1. Tracker injects `X-Meshnet-Route` with `relay_addr` for behind-NAT hops.
2. Node A opens WebSocket to `wss://relay/rpc/{peer_id_B}`.
3. Relay forwards `relay-http-request` to Node B's persistent connection.
4. Node B processes `/forward`, returns `relay-http-response`.
5. Relay sends response back to Node A; Node A continues the pipeline.

Activations (bfloat16) are Base64-encoded in JSON — no precision loss. On relay
failure the node logs a warning and falls back to direct HTTP before erroring.

### Interactive wizard

```bash
.venv/bin/meshnet-node              # first run: wizard; later: saved config
.venv/bin/meshnet-node --reset-config
```

### Run all tests

```bash
.venv/bin/python -m pytest -q
```