From ded8c06e7733df1b2cc8a1d1c73b800f4c5e8e8a Mon Sep 17 00:00:00 2001 From: Dobromir Popov Date: Mon, 29 Jun 2026 18:28:23 +0300 Subject: [PATCH] docs: update QUICKSTART to reflect auto-shard detection No need for --shard-start/--shard-end in the basic start command; fix layer count for Qwen2.5-0.5B from 28 to 24 (verified via AutoConfig). Co-Authored-By: Claude Sonnet 4.6 --- QUICKSTART.md | 199 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 199 insertions(+) create mode 100644 QUICKSTART.md diff --git a/QUICKSTART.md b/QUICKSTART.md new file mode 100644 index 0000000..e1d2c9b --- /dev/null +++ b/QUICKSTART.md @@ -0,0 +1,199 @@ +# Quickstart — Running a node and testing inference + +This guide gets you from zero to a live inference request in three terminals. +Tested on: AMD Ryzen AI Max (Strix Halo APU), 124 GB RAM, Linux, CPU inference. + +--- + +## Prerequisites + +```bash +# Clone and enter repo +cd /run/media/popov/d/DEV/repos/d-popov.com/AI + +# Install Python packages (editable — picks up code changes immediately) +.venv/bin/pip install -e packages/tracker packages/node packages/p2p packages/relay + +# CPU-only PyTorch (skip if you have CUDA/ROCm already) +.venv/bin/pip install torch --index-url https://download.pytorch.org/whl/cpu + +# HuggingFace model libraries +.venv/bin/pip install transformers accelerate +``` + +> **NVIDIA GPU (CUDA):** replace the torch line with `pip install torch` (default index). +> **AMD GPU (ROCm):** `pip install torch --index-url https://download.pytorch.org/whl/rocm6.2` + +--- + +## Step 1 — Start the tracker (Terminal 1) + +```bash +cd /run/media/popov/d/DEV/repos/d-popov.com/AI +.venv/bin/meshnet-tracker start --port 8080 +``` + +Expected output: +``` +Tracker listening on 0.0.0.0:8080 +``` + +Keep this terminal open. + +--- + +## Step 2 — Start a node (Terminal 2) + +### Recommended model: Qwen2.5-0.5B-Instruct + +- 0.5B parameters, ~1 GB in BF16 +- No HuggingFace account or license required +- Downloads once to `~/.meshnet/models/`, cached for future runs +- 24 transformer layers (auto-detected — no need to specify) + +```bash +cd /run/media/popov/d/DEV/repos/d-popov.com/AI +HF_HOME=/run/media/popov/d/DEV/models \ +.venv/bin/meshnet-node start \ + --model-id Qwen/Qwen2.5-0.5B-Instruct \ + --quantization bfloat16 \ + --tracker http://localhost:8080 \ + --port 8001 +``` + +Shard range is **auto-detected** from the curated catalog (no network call for known +models). For unknown repos, the node fetches only `config.json` (~1 KB) to read +`num_hidden_layers`. You can still pass `--shard-start` / `--shard-end` explicitly +to run a partial shard on one machine. + +Expected output (after model loads): +``` + Auto-detected 24 layers → shard 0–23 +================================ +meshnet-node ready + Wallet:
+ Model ID: Qwen/Qwen2.5-0.5B-Instruct + Shard: layers 0–23 + Quantization: bfloat16 + Endpoint: http://:8001 + Hardware: CPU +================================ +``` + +### Other model options (all CPU-friendly) + +| Model | HF repo | Layers | BF16 size | Notes | +|-------|---------|--------|-----------|-------| +| Qwen2.5-0.5B | `Qwen/Qwen2.5-0.5B-Instruct` | 24 | ~1 GB | Fastest, no gating | +| Qwen2.5-1.5B | `Qwen/Qwen2.5-1.5B-Instruct` | 28 | ~3 GB | Better quality | +| Phi-3-mini | `microsoft/Phi-3-mini-4k-instruct` | 32 | ~7.5 GB | Best CPU quality | +| Llama-3.2-1B | `meta-llama/Llama-3.2-1B-Instruct` | 16 | ~2 GB | Requires HF login | +| Llama-3.2-3B | `meta-llama/Llama-3.2-3B-Instruct` | 28 | ~6 GB | Requires HF login | + +For gated models (Llama), run `huggingface-cli login` first. + +--- + +## Step 3 — Send an inference request (Terminal 3) + +```bash +curl -s http://localhost:8001/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "qwen2.5-0.5b", + "messages": [{"role": "user", "content": "What is 7 times 8? Answer in one word."}], + "stream": false + }' | python3 -m json.tool +``` + +Or use the test script: + +```bash +.venv/bin/python scripts/test_lan_inference.py \ + --tracker http://localhost:8080 \ + --gateway http://localhost:8001 +``` + +--- + +## Two-node split (same machine, two terminals) + +Split Qwen2.5-0.5B's 24 layers across two node processes to test the sharded pipeline: + +**Node A — layers 0–11 (tracker mode, serves chat completions):** +```bash +HF_HOME=/run/media/popov/d/DEV/models \ +.venv/bin/meshnet-node start \ + --model-id Qwen/Qwen2.5-0.5B-Instruct \ + --shard-start 0 --shard-end 11 \ + --quantization bfloat16 \ + --tracker http://localhost:8080 \ + --port 8001 +``` + +**Node B — layers 12–23:** +```bash +HF_HOME=/run/media/popov/d/DEV/models \ +.venv/bin/meshnet-node start \ + --model-id Qwen/Qwen2.5-0.5B-Instruct \ + --shard-start 12 --shard-end 23 \ + --quantization bfloat16 \ + --tracker http://localhost:8080 \ + --port 8002 +``` + +Send the request to Node A — it tokenizes, runs layers 0–13, passes binary +activations to Node B, and streams the final response back. + +--- + +## Two-machine LAN test (Linux + Windows/WSL2) + +See `docs/TWO_MACHINE_TEST.md` (created by US-018). + +--- + +## Browse available models + +```bash +# Show curated list with VRAM requirements +.venv/bin/meshnet-node models + +# Browse HuggingFace Hub top-20 text-generation models +.venv/bin/meshnet-node models --browse +``` + +--- + +## Start with the interactive wizard + +```bash +# First run: wizard detects GPU, shows model list, saves config +.venv/bin/meshnet-node + +# Subsequent runs: starts directly from saved config +.venv/bin/meshnet-node + +# Re-run wizard even with saved config +.venv/bin/meshnet-node --reset-config +``` + +--- + +## Start the relay node (for NAT traversal) + +```bash +.venv/bin/pip install -e packages/relay +.venv/bin/meshnet-relay --port 8765 +``` + +Nodes behind NAT connect to the relay and advertise their relay address to the +tracker. See `docs/adr/0010-p2p-gossip-and-nat-relay.md`. + +--- + +## Run all tests + +```bash +.venv/bin/python -m pytest -q +```