docs: update QUICKSTART to reflect auto-shard detection
No need for --shard-start/--shard-end in the basic start command; fix layer count for Qwen2.5-0.5B from 28 to 24 (verified via AutoConfig). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
199
QUICKSTART.md
Normal file
199
QUICKSTART.md
Normal file
@@ -0,0 +1,199 @@
|
|||||||
|
# Quickstart — Running a node and testing inference
|
||||||
|
|
||||||
|
This guide gets you from zero to a live inference request in three terminals.
|
||||||
|
Tested on: AMD Ryzen AI Max (Strix Halo APU), 124 GB RAM, Linux, CPU inference.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Clone and enter repo
|
||||||
|
cd /run/media/popov/d/DEV/repos/d-popov.com/AI
|
||||||
|
|
||||||
|
# Install Python packages (editable — picks up code changes immediately)
|
||||||
|
.venv/bin/pip install -e packages/tracker packages/node packages/p2p packages/relay
|
||||||
|
|
||||||
|
# CPU-only PyTorch (skip if you have CUDA/ROCm already)
|
||||||
|
.venv/bin/pip install torch --index-url https://download.pytorch.org/whl/cpu
|
||||||
|
|
||||||
|
# HuggingFace model libraries
|
||||||
|
.venv/bin/pip install transformers accelerate
|
||||||
|
```
|
||||||
|
|
||||||
|
> **NVIDIA GPU (CUDA):** replace the torch line with `pip install torch` (default index).
|
||||||
|
> **AMD GPU (ROCm):** `pip install torch --index-url https://download.pytorch.org/whl/rocm6.2`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Step 1 — Start the tracker (Terminal 1)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd /run/media/popov/d/DEV/repos/d-popov.com/AI
|
||||||
|
.venv/bin/meshnet-tracker start --port 8080
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected output:
|
||||||
|
```
|
||||||
|
Tracker listening on 0.0.0.0:8080
|
||||||
|
```
|
||||||
|
|
||||||
|
Keep this terminal open.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Step 2 — Start a node (Terminal 2)
|
||||||
|
|
||||||
|
### Recommended model: Qwen2.5-0.5B-Instruct
|
||||||
|
|
||||||
|
- 0.5B parameters, ~1 GB in BF16
|
||||||
|
- No HuggingFace account or license required
|
||||||
|
- Downloads once to `~/.meshnet/models/`, cached for future runs
|
||||||
|
- 24 transformer layers (auto-detected — no need to specify)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd /run/media/popov/d/DEV/repos/d-popov.com/AI
|
||||||
|
HF_HOME=/run/media/popov/d/DEV/models \
|
||||||
|
.venv/bin/meshnet-node start \
|
||||||
|
--model-id Qwen/Qwen2.5-0.5B-Instruct \
|
||||||
|
--quantization bfloat16 \
|
||||||
|
--tracker http://localhost:8080 \
|
||||||
|
--port 8001
|
||||||
|
```
|
||||||
|
|
||||||
|
Shard range is **auto-detected** from the curated catalog (no network call for known
|
||||||
|
models). For unknown repos, the node fetches only `config.json` (~1 KB) to read
|
||||||
|
`num_hidden_layers`. You can still pass `--shard-start` / `--shard-end` explicitly
|
||||||
|
to run a partial shard on one machine.
|
||||||
|
|
||||||
|
Expected output (after model loads):
|
||||||
|
```
|
||||||
|
Auto-detected 24 layers → shard 0–23
|
||||||
|
================================
|
||||||
|
meshnet-node ready
|
||||||
|
Wallet: <address>
|
||||||
|
Model ID: Qwen/Qwen2.5-0.5B-Instruct
|
||||||
|
Shard: layers 0–23
|
||||||
|
Quantization: bfloat16
|
||||||
|
Endpoint: http://<host>:8001
|
||||||
|
Hardware: CPU
|
||||||
|
================================
|
||||||
|
```
|
||||||
|
|
||||||
|
### Other model options (all CPU-friendly)
|
||||||
|
|
||||||
|
| Model | HF repo | Layers | BF16 size | Notes |
|
||||||
|
|-------|---------|--------|-----------|-------|
|
||||||
|
| Qwen2.5-0.5B | `Qwen/Qwen2.5-0.5B-Instruct` | 24 | ~1 GB | Fastest, no gating |
|
||||||
|
| Qwen2.5-1.5B | `Qwen/Qwen2.5-1.5B-Instruct` | 28 | ~3 GB | Better quality |
|
||||||
|
| Phi-3-mini | `microsoft/Phi-3-mini-4k-instruct` | 32 | ~7.5 GB | Best CPU quality |
|
||||||
|
| Llama-3.2-1B | `meta-llama/Llama-3.2-1B-Instruct` | 16 | ~2 GB | Requires HF login |
|
||||||
|
| Llama-3.2-3B | `meta-llama/Llama-3.2-3B-Instruct` | 28 | ~6 GB | Requires HF login |
|
||||||
|
|
||||||
|
For gated models (Llama), run `huggingface-cli login` first.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Step 3 — Send an inference request (Terminal 3)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s http://localhost:8001/v1/chat/completions \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{
|
||||||
|
"model": "qwen2.5-0.5b",
|
||||||
|
"messages": [{"role": "user", "content": "What is 7 times 8? Answer in one word."}],
|
||||||
|
"stream": false
|
||||||
|
}' | python3 -m json.tool
|
||||||
|
```
|
||||||
|
|
||||||
|
Or use the test script:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
.venv/bin/python scripts/test_lan_inference.py \
|
||||||
|
--tracker http://localhost:8080 \
|
||||||
|
--gateway http://localhost:8001
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Two-node split (same machine, two terminals)
|
||||||
|
|
||||||
|
Split Qwen2.5-0.5B's 24 layers across two node processes to test the sharded pipeline:
|
||||||
|
|
||||||
|
**Node A — layers 0–11 (tracker mode, serves chat completions):**
|
||||||
|
```bash
|
||||||
|
HF_HOME=/run/media/popov/d/DEV/models \
|
||||||
|
.venv/bin/meshnet-node start \
|
||||||
|
--model-id Qwen/Qwen2.5-0.5B-Instruct \
|
||||||
|
--shard-start 0 --shard-end 11 \
|
||||||
|
--quantization bfloat16 \
|
||||||
|
--tracker http://localhost:8080 \
|
||||||
|
--port 8001
|
||||||
|
```
|
||||||
|
|
||||||
|
**Node B — layers 12–23:**
|
||||||
|
```bash
|
||||||
|
HF_HOME=/run/media/popov/d/DEV/models \
|
||||||
|
.venv/bin/meshnet-node start \
|
||||||
|
--model-id Qwen/Qwen2.5-0.5B-Instruct \
|
||||||
|
--shard-start 12 --shard-end 23 \
|
||||||
|
--quantization bfloat16 \
|
||||||
|
--tracker http://localhost:8080 \
|
||||||
|
--port 8002
|
||||||
|
```
|
||||||
|
|
||||||
|
Send the request to Node A — it tokenizes, runs layers 0–13, passes binary
|
||||||
|
activations to Node B, and streams the final response back.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Two-machine LAN test (Linux + Windows/WSL2)
|
||||||
|
|
||||||
|
See `docs/TWO_MACHINE_TEST.md` (created by US-018).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Browse available models
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Show curated list with VRAM requirements
|
||||||
|
.venv/bin/meshnet-node models
|
||||||
|
|
||||||
|
# Browse HuggingFace Hub top-20 text-generation models
|
||||||
|
.venv/bin/meshnet-node models --browse
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Start with the interactive wizard
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# First run: wizard detects GPU, shows model list, saves config
|
||||||
|
.venv/bin/meshnet-node
|
||||||
|
|
||||||
|
# Subsequent runs: starts directly from saved config
|
||||||
|
.venv/bin/meshnet-node
|
||||||
|
|
||||||
|
# Re-run wizard even with saved config
|
||||||
|
.venv/bin/meshnet-node --reset-config
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Start the relay node (for NAT traversal)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
.venv/bin/pip install -e packages/relay
|
||||||
|
.venv/bin/meshnet-relay --port 8765
|
||||||
|
```
|
||||||
|
|
||||||
|
Nodes behind NAT connect to the relay and advertise their relay address to the
|
||||||
|
tracker. See `docs/adr/0010-p2p-gossip-and-nat-relay.md`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Run all tests
|
||||||
|
|
||||||
|
```bash
|
||||||
|
.venv/bin/python -m pytest -q
|
||||||
|
```
|
||||||
Reference in New Issue
Block a user