neuron-tai/QUICKSTART.md

Quickstart — Running a node and testing inference

This guide gets you from zero to a live inference request in three terminals. Tested on: AMD Ryzen AI Max (Strix Halo APU), 124 GB RAM, Linux, CPU inference.

---

Prerequisites

# Clone and enter repo
cd /run/media/popov/d/DEV/repos/d-popov.com/AI

# Install Python packages (editable — picks up code changes immediately)
.venv/bin/pip install -e packages/tracker packages/node packages/p2p packages/relay

# CPU-only PyTorch (skip if you have CUDA/ROCm already)
.venv/bin/pip install torch --index-url https://download.pytorch.org/whl/cpu

# HuggingFace model libraries
.venv/bin/pip install transformers accelerate

NVIDIA GPU (CUDA): replace the torch line with pip install torch (default index).
AMD GPU (ROCm): pip install torch --index-url https://download.pytorch.org/whl/rocm6.2

---

Step 1 — Start the tracker (Terminal 1)

cd /run/media/popov/d/DEV/repos/d-popov.com/AI
.venv/bin/meshnet-tracker start --port 8080

Expected output:

Tracker listening on 0.0.0.0:8080

Keep this terminal open.

---

Step 2 — Start a node (Terminal 2)

Recommended model: Qwen2.5-0.5B-Instruct

• 0.5B parameters, ~1 GB in BF16
• No HuggingFace account or license required
• Downloads once to ~/.meshnet/models/, cached for future runs
• 24 transformer layers (auto-detected — no need to specify)

cd /run/media/popov/d/DEV/repos/d-popov.com/AI
HF_HOME=/run/media/popov/d/DEV/models \
.venv/bin/meshnet-node start \
  --model-id Qwen/Qwen2.5-0.5B-Instruct \
  --quantization bfloat16 \
  --tracker http://localhost:8080 \
  --port 8001

Shard range is auto-detected from the curated catalog (no network call for known models). For unknown repos, the node fetches only config.json (~1 KB) to read num_hidden_layers. You can still pass --shard-start / --shard-end explicitly to run a partial shard on one machine.

Expected output (after model loads):

  Auto-detected 24 layers → shard 0–23
================================
meshnet-node ready
  Wallet:       <address>
  Model ID:     Qwen/Qwen2.5-0.5B-Instruct
  Shard:        layers 0–23
  Quantization: bfloat16
  Endpoint:     http://<host>:8001
  Hardware:     CPU
================================

Other model options (all CPU-friendly)

Model	HF repo	Layers	BF16 size	Notes
Qwen2.5-0.5B	Qwen/Qwen2.5-0.5B-Instruct	24	~1 GB	Fastest, no gating
Qwen2.5-1.5B	Qwen/Qwen2.5-1.5B-Instruct	28	~3 GB	Better quality
Phi-3-mini	microsoft/Phi-3-mini-4k-instruct	32	~7.5 GB	Best CPU quality
Llama-3.2-1B	meta-llama/Llama-3.2-1B-Instruct	16	~2 GB	Requires HF login
Llama-3.2-3B	meta-llama/Llama-3.2-3B-Instruct	28	~6 GB	Requires HF login

For gated models (Llama), run huggingface-cli login first.

---

Step 3 — Send an inference request (Terminal 3)

curl -s http://localhost:8001/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2.5-0.5b",
    "messages": [{"role": "user", "content": "What is 7 times 8? Answer in one word."}],
    "stream": false
  }' | python3 -m json.tool

Or use the test script:

.venv/bin/python scripts/test_lan_inference.py \
  --tracker http://localhost:8080 \
  --gateway http://localhost:8001

---

Two-node split (same machine, two terminals)

Split Qwen2.5-0.5B's 24 layers across two node processes to test the sharded pipeline:

Node A — layers 0–11 (tracker mode, serves chat completions):

HF_HOME=/run/media/popov/d/DEV/models \
.venv/bin/meshnet-node start \
  --model-id Qwen/Qwen2.5-0.5B-Instruct \
  --shard-start 0 --shard-end 11 \
  --quantization bfloat16 \
  --tracker http://localhost:8080 \
  --port 8001

Node B — layers 12–23:

HF_HOME=/run/media/popov/d/DEV/models \
.venv/bin/meshnet-node start \
  --model-id Qwen/Qwen2.5-0.5B-Instruct \
  --shard-start 12 --shard-end 23 \
  --quantization bfloat16 \
  --tracker http://localhost:8080 \
  --port 8002

Send the request to Node A — it tokenizes, runs layers 0–13, passes binary activations to Node B, and streams the final response back.

---

Two-machine LAN test (Linux + Windows/WSL2)

See docs/TWO_MACHINE_TEST.md (created by US-018).

---

Browse available models

# Show curated list with VRAM requirements
.venv/bin/meshnet-node models

# Browse HuggingFace Hub top-20 text-generation models
.venv/bin/meshnet-node models --browse

---

Start with the interactive wizard

# First run: wizard detects GPU, shows model list, saves config
.venv/bin/meshnet-node

# Subsequent runs: starts directly from saved config
.venv/bin/meshnet-node

# Re-run wizard even with saved config
.venv/bin/meshnet-node --reset-config

---

Start the relay node (for NAT traversal)

.venv/bin/pip install -e packages/relay
.venv/bin/meshnet-relay --port 8765

Nodes behind NAT connect to the relay and advertise their relay address to the tracker. See docs/adr/0010-p2p-gossip-and-nat-relay.md.

---

Run all tests

.venv/bin/python -m pytest -q