Compare commits
2 Commits
ac0ca20b56
...
481ce6c6f5
| Author | SHA1 | Date | |
|---|---|---|---|
|
481ce6c6f5 | ||
|
|
7ba87051f5 |
1544
QUICKSTART.md
1544
QUICKSTART.md
File diff suppressed because it is too large
Load Diff
@@ -1,212 +1,220 @@
|
||||
# Installing meshnet-node on Windows 11 with WSL2
|
||||
|
||||
This guide covers setting up a meshnet-node on a Windows 11 machine using WSL2 with CUDA passthrough so it can join an existing inference network over LAN.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Windows 11 with WSL2 support (most systems with Windows 10 version 2004+ qualify)
|
||||
- NVIDIA GPU with CUDA support (driver ≥ 525.x recommended for WSL2 CUDA)
|
||||
- At least 8 GB RAM + enough VRAM for the model shard you intend to serve
|
||||
- The Linux machine (other node) is reachable on your LAN
|
||||
|
||||
---
|
||||
|
||||
## Step 1 — Enable WSL2 and install Ubuntu
|
||||
|
||||
Open **PowerShell as Administrator** and run:
|
||||
|
||||
```powershell
|
||||
wsl --install -d Ubuntu-24.04
|
||||
```
|
||||
|
||||
This installs WSL2 with Ubuntu 24.04. Reboot when prompted.
|
||||
|
||||
After reboot, Ubuntu starts and asks you to create a UNIX username/password. Choose anything convenient.
|
||||
|
||||
Verify WSL version:
|
||||
|
||||
```powershell
|
||||
wsl -l -v
|
||||
```
|
||||
|
||||
Output should show `VERSION 2`.
|
||||
|
||||
---
|
||||
|
||||
## Step 2 — Install NVIDIA GPU driver on Windows (NOT inside WSL)
|
||||
|
||||
WSL2 CUDA passthrough works through the Windows host driver. **Do not install CUDA inside WSL2.**
|
||||
|
||||
1. Download the latest Game Ready or Studio driver for your GPU from https://www.nvidia.com/drivers
|
||||
2. Install on Windows normally (standard installer).
|
||||
3. Inside WSL2 (Ubuntu terminal), verify:
|
||||
|
||||
```bash
|
||||
nvidia-smi
|
||||
```
|
||||
|
||||
Expected output: your GPU name, driver version, CUDA version. If this command fails, the Windows driver is too old — update it.
|
||||
|
||||
> **Note:** The `cuda-toolkit` package inside WSL2 is optional and only needed if you compile CUDA kernels. For inference with `torch`, the Windows host driver is sufficient.
|
||||
|
||||
---
|
||||
|
||||
## Step 3 — Install Python 3.11+ inside WSL2
|
||||
|
||||
Ubuntu 24.04 ships Python 3.12. Confirm:
|
||||
|
||||
```bash
|
||||
python3 --version
|
||||
```
|
||||
|
||||
If it shows 3.10 or older:
|
||||
|
||||
```bash
|
||||
sudo add-apt-repository ppa:deadsnakes/ppa
|
||||
sudo apt update
|
||||
sudo apt install python3.12 python3.12-venv python3.12-dev
|
||||
```
|
||||
|
||||
Install pip:
|
||||
|
||||
```bash
|
||||
curl -sS https://bootstrap.pypa.io/get-pip.py | python3
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 4 — Clone the repository
|
||||
|
||||
Inside WSL2:
|
||||
|
||||
```bash
|
||||
# Store the repo in the Linux filesystem (faster I/O than /mnt/c)
|
||||
cd ~
|
||||
git clone https://github.com/YOUR_ORG/d-popov.com.git
|
||||
cd d-popov.com/AI
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 5 — Create a virtualenv and install meshnet-node
|
||||
|
||||
```bash
|
||||
python3 -m venv .venv
|
||||
source .venv/bin/activate
|
||||
|
||||
# Install node + PyTorch (CUDA build)
|
||||
pip install torch --index-url https://download.pytorch.org/whl/cu124
|
||||
pip install -e "packages/node[torch]"
|
||||
```
|
||||
|
||||
Verify the install:
|
||||
|
||||
```bash
|
||||
meshnet-node --help
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 6 — Pre-download the model shard
|
||||
|
||||
Download the model before starting the node so the startup process doesn't time out on the tracker side:
|
||||
|
||||
```bash
|
||||
python3 - <<'EOF'
|
||||
from transformers import AutoConfig
|
||||
AutoConfig.from_pretrained("microsoft/Phi-3-medium-128k-instruct")
|
||||
EOF
|
||||
```
|
||||
|
||||
For the full model weights (needed at runtime), `transformers` downloads them automatically on first `meshnet-node` start. If you want to pre-fetch:
|
||||
|
||||
```bash
|
||||
python3 -c "
|
||||
from transformers import AutoModelForCausalLM
|
||||
AutoModelForCausalLM.from_pretrained('microsoft/Phi-3-medium-128k-instruct', device_map='cpu')
|
||||
"
|
||||
```
|
||||
|
||||
This can take 10–30 minutes on first run.
|
||||
|
||||
---
|
||||
|
||||
## Step 7 — Expose the node port to your LAN
|
||||
|
||||
WSL2 runs behind a NAT with a virtual IP (typically `172.x.x.x`). Your LAN sees the Windows host IP. You need to forward the node port.
|
||||
|
||||
**Option A — Windows port proxy (recommended for simple setups):**
|
||||
|
||||
In **PowerShell as Administrator**:
|
||||
|
||||
```powershell
|
||||
# Get the current WSL2 IP (changes on each WSL restart)
|
||||
$wslIp = (wsl hostname -I).Trim()
|
||||
|
||||
# Forward Windows host port 8001 → WSL2 port 8001
|
||||
netsh interface portproxy add v4tov4 `
|
||||
listenport=8001 listenaddress=0.0.0.0 `
|
||||
connectport=8001 connectaddress=$wslIp
|
||||
|
||||
# Allow inbound on Windows Firewall
|
||||
New-NetFirewallRule -DisplayName "meshnet-node" `
|
||||
-Direction Inbound -Protocol TCP -LocalPort 8001 -Action Allow
|
||||
```
|
||||
|
||||
Verify: from the Linux machine, `curl http://WINDOWS_LAN_IP:8001/v1/health` should return a response once the node is running.
|
||||
|
||||
**Redo this after every WSL2 restart** — the WSL2 IP changes.
|
||||
|
||||
**Option B — P2P relay (US-017, no port forwarding needed):**
|
||||
|
||||
Start a relay node on the Linux machine. The WSL2 node connects outbound through the relay. No firewall rules needed. See `docs/TWO_MACHINE_TEST.md` for details.
|
||||
|
||||
---
|
||||
|
||||
## Step 8 — Start the node
|
||||
|
||||
Replace `192.168.1.10` with the actual LAN IP of the Linux machine running the tracker.
|
||||
Replace shard range with the complementary range to what the Linux node is serving.
|
||||
|
||||
```bash
|
||||
source .venv/bin/activate
|
||||
|
||||
meshnet-node \
|
||||
--model microsoft/Phi-3-medium-128k-instruct \
|
||||
--quantization bf16 \
|
||||
--shard-start 20 --shard-end 39 \
|
||||
--tracker http://192.168.1.10:8080 \
|
||||
--port 8001 \
|
||||
--host 0.0.0.0 \
|
||||
--advertise-host WINDOWS_LAN_IP
|
||||
```
|
||||
|
||||
The `--advertise-host` flag tells the tracker what IP the Linux machine should use to reach this node. Use your Windows machine's LAN IP (e.g. `192.168.1.20`), **not** the WSL2 internal IP.
|
||||
|
||||
Expected startup output:
|
||||
|
||||
```
|
||||
Detecting hardware...
|
||||
GPU: NVIDIA GeForce RTX 3080 (10240 MB VRAM)
|
||||
Loading wallet...
|
||||
Wallet: 5K7r...
|
||||
Loading real PyTorch model shard...
|
||||
Auto-detected 40 layers → shard 20–39
|
||||
================================
|
||||
meshnet-node ready
|
||||
Model ID: microsoft/Phi-3-medium-128k-instruct
|
||||
Shard: layers 20–39; 20 of 40
|
||||
Endpoint: http://192.168.1.20:8001
|
||||
Hardware: CUDA
|
||||
================================
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Known issues
|
||||
|
||||
- **WSL2 IP changes on restart.** Always re-run the `netsh` port-proxy command after restarting WSL2 or Windows.
|
||||
- **CUDA not visible in WSL2.** If `nvidia-smi` fails inside WSL2, update the Windows host GPU driver to ≥ 525.x. Installing CUDA inside WSL2 will not fix it.
|
||||
- **Model download is slow.** HuggingFace downloads happen over HTTPS. Pre-fetch the model before a timed test (see Step 6).
|
||||
- **Port 8001 already in use.** Change `--port` to another value and update the firewall/portproxy rules accordingly.
|
||||
- **`bf16` not supported on older GPUs.** Use `--quantization int8` on Turing (RTX 20xx) cards or earlier if bfloat16 ops fail.
|
||||
# Installing meshnet-node on Windows 11 with WSL2
|
||||
|
||||
This guide covers setting up a meshnet-node on a Windows 11 machine using WSL2 with CUDA passthrough so it can join an existing inference network over LAN.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Windows 11 with WSL2 support (most systems with Windows 10 version 2004+ qualify)
|
||||
- NVIDIA GPU with CUDA support (driver ≥ 525.x recommended for WSL2 CUDA)
|
||||
- At least 8 GB RAM + enough VRAM for the model shard you intend to serve
|
||||
- The Linux machine (other node) is reachable on your LAN
|
||||
|
||||
---
|
||||
|
||||
## Step 1 — Enable WSL2 and install Ubuntu
|
||||
|
||||
Open **PowerShell as Administrator** and run:
|
||||
|
||||
```powershell
|
||||
wsl --install -d Ubuntu-24.04
|
||||
```
|
||||
|
||||
This installs WSL2 with Ubuntu 24.04. Reboot when prompted.
|
||||
|
||||
After reboot, Ubuntu starts and asks you to create a UNIX username/password. Choose anything convenient.
|
||||
|
||||
Verify WSL version:
|
||||
|
||||
```powershell
|
||||
wsl -l -v
|
||||
```
|
||||
|
||||
Output should show `VERSION 2`.
|
||||
|
||||
---
|
||||
|
||||
## Step 2 — Install NVIDIA GPU driver on Windows (NOT inside WSL)
|
||||
|
||||
WSL2 CUDA passthrough works through the Windows host driver. **Do not install CUDA inside WSL2.**
|
||||
|
||||
1. Download the latest Game Ready or Studio driver for your GPU from https://www.nvidia.com/drivers
|
||||
2. Install on Windows normally (standard installer).
|
||||
3. Inside WSL2 (Ubuntu terminal), verify:
|
||||
|
||||
```bash
|
||||
nvidia-smi
|
||||
```
|
||||
|
||||
Expected output: your GPU name, driver version, CUDA version. If this command fails, the Windows driver is too old — update it.
|
||||
|
||||
> **Note:** The `cuda-toolkit` package inside WSL2 is optional and only needed if you compile CUDA kernels. For inference with `torch`, the Windows host driver is sufficient.
|
||||
|
||||
---
|
||||
|
||||
## Step 3 — Install Python 3.11+ inside WSL2
|
||||
|
||||
Ubuntu 24.04 ships Python 3.12. Confirm:
|
||||
|
||||
```bash
|
||||
python3 --version
|
||||
```
|
||||
|
||||
If it shows 3.10 or older:
|
||||
|
||||
```bash
|
||||
sudo add-apt-repository ppa:deadsnakes/ppa
|
||||
sudo apt update
|
||||
sudo apt install python3.12 python3.12-venv python3.12-dev
|
||||
```
|
||||
|
||||
Install pip:
|
||||
|
||||
```bash
|
||||
curl -sS https://bootstrap.pypa.io/get-pip.py | python3
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 4 — Clone the repository
|
||||
|
||||
Inside WSL2:
|
||||
|
||||
```bash
|
||||
# Store the repo in the Linux filesystem (faster I/O than /mnt/c)
|
||||
cd ~
|
||||
git clone https://github.com/YOUR_ORG/d-popov.com.git
|
||||
cd d-popov.com/AI
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 5 — Create a virtualenv and install meshnet-node
|
||||
|
||||
```bash
|
||||
python3 -m venv .venv
|
||||
source .venv/bin/activate
|
||||
|
||||
# Install node + PyTorch (CUDA build)
|
||||
pip install torch --index-url https://download.pytorch.org/whl/cu124
|
||||
pip install -e "packages/node[torch]"
|
||||
```
|
||||
|
||||
Verify the install:
|
||||
|
||||
```bash
|
||||
meshnet-node --help
|
||||
python -c "import transformers; print(transformers.__version__)"
|
||||
```
|
||||
|
||||
`transformers` must be **≥ 5.12** for Qwen3.5/3.6-MoE models (older versions fail
|
||||
with `'Qwen3_5MoeConfig' object has no attribute 'vocab_size'`). If you install
|
||||
into an existing conda/miniforge env instead of a fresh venv, run
|
||||
`pip install -U transformers` there. The startup warning about
|
||||
`flash-linear-attention` / `causal-conv1d` ("fast path is not available") is
|
||||
harmless on CPU — those are optional CUDA-only kernels.
|
||||
|
||||
---
|
||||
|
||||
## Step 6 — Pre-download the model shard
|
||||
|
||||
Download the model before starting the node so the startup process doesn't time out on the tracker side:
|
||||
|
||||
```bash
|
||||
python3 - <<'EOF'
|
||||
from transformers import AutoConfig
|
||||
AutoConfig.from_pretrained("microsoft/Phi-3-medium-128k-instruct")
|
||||
EOF
|
||||
```
|
||||
|
||||
For the full model weights (needed at runtime), `transformers` downloads them automatically on first `meshnet-node` start. If you want to pre-fetch:
|
||||
|
||||
```bash
|
||||
python3 -c "
|
||||
from transformers import AutoModelForCausalLM
|
||||
AutoModelForCausalLM.from_pretrained('microsoft/Phi-3-medium-128k-instruct', device_map='cpu')
|
||||
"
|
||||
```
|
||||
|
||||
This can take 10–30 minutes on first run.
|
||||
|
||||
---
|
||||
|
||||
## Step 7 — Expose the node port to your LAN
|
||||
|
||||
WSL2 runs behind a NAT with a virtual IP (typically `172.x.x.x`). Your LAN sees the Windows host IP. You need to forward the node port.
|
||||
|
||||
**Option A — Windows port proxy (recommended for simple setups):**
|
||||
|
||||
In **PowerShell as Administrator**:
|
||||
|
||||
```powershell
|
||||
# Get the current WSL2 IP (changes on each WSL restart)
|
||||
$wslIp = (wsl hostname -I).Trim()
|
||||
|
||||
# Forward Windows host port 8001 → WSL2 port 8001
|
||||
netsh interface portproxy add v4tov4 `
|
||||
listenport=8001 listenaddress=0.0.0.0 `
|
||||
connectport=8001 connectaddress=$wslIp
|
||||
|
||||
# Allow inbound on Windows Firewall
|
||||
New-NetFirewallRule -DisplayName "meshnet-node" `
|
||||
-Direction Inbound -Protocol TCP -LocalPort 8001 -Action Allow
|
||||
```
|
||||
|
||||
Verify: from the Linux machine, `curl http://WINDOWS_LAN_IP:8001/v1/health` should return a response once the node is running.
|
||||
|
||||
**Redo this after every WSL2 restart** — the WSL2 IP changes.
|
||||
|
||||
**Option B — P2P relay (US-017, no port forwarding needed):**
|
||||
|
||||
Start a relay node on the Linux machine. The WSL2 node connects outbound through the relay. No firewall rules needed. See `docs/TWO_MACHINE_TEST.md` for details.
|
||||
|
||||
---
|
||||
|
||||
## Step 8 — Start the node
|
||||
|
||||
Replace `192.168.1.10` with the actual LAN IP of the Linux machine running the tracker.
|
||||
Replace shard range with the complementary range to what the Linux node is serving.
|
||||
|
||||
```bash
|
||||
source .venv/bin/activate
|
||||
|
||||
meshnet-node \
|
||||
--model microsoft/Phi-3-medium-128k-instruct \
|
||||
--quantization bf16 \
|
||||
--shard-start 20 --shard-end 39 \
|
||||
--tracker http://192.168.1.10:8080 \
|
||||
--port 8001 \
|
||||
--host 0.0.0.0 \
|
||||
--advertise-host WINDOWS_LAN_IP
|
||||
```
|
||||
|
||||
The `--advertise-host` flag tells the tracker what IP the Linux machine should use to reach this node. Use your Windows machine's LAN IP (e.g. `192.168.1.20`), **not** the WSL2 internal IP.
|
||||
|
||||
Expected startup output:
|
||||
|
||||
```
|
||||
Detecting hardware...
|
||||
GPU: NVIDIA GeForce RTX 3080 (10240 MB VRAM)
|
||||
Loading wallet...
|
||||
Wallet: 5K7r...
|
||||
Loading real PyTorch model shard...
|
||||
Auto-detected 40 layers → shard 20–39
|
||||
================================
|
||||
meshnet-node ready
|
||||
Model ID: microsoft/Phi-3-medium-128k-instruct
|
||||
Shard: layers 20–39; 20 of 40
|
||||
Endpoint: http://192.168.1.20:8001
|
||||
Hardware: CUDA
|
||||
================================
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Known issues
|
||||
|
||||
- **WSL2 IP changes on restart.** Always re-run the `netsh` port-proxy command after restarting WSL2 or Windows.
|
||||
- **CUDA not visible in WSL2.** If `nvidia-smi` fails inside WSL2, update the Windows host GPU driver to ≥ 525.x. Installing CUDA inside WSL2 will not fix it.
|
||||
- **Model download is slow.** HuggingFace downloads happen over HTTPS. Pre-fetch the model before a timed test (see Step 6).
|
||||
- **Port 8001 already in use.** Change `--port` to another value and update the firewall/portproxy rules accordingly.
|
||||
- **`bf16` not supported on older GPUs.** Use `--quantization int8` on Turing (RTX 20xx) cards or earlier if bfloat16 ops fail.
|
||||
|
||||
@@ -1,33 +1,33 @@
|
||||
[build-system]
|
||||
requires = ["setuptools>=64"]
|
||||
build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "meshnet-node"
|
||||
version = "0.1.0"
|
||||
description = "Distributed Inference Network node client"
|
||||
requires-python = ">=3.10"
|
||||
|
||||
dependencies = [
|
||||
"cryptography>=41",
|
||||
"huggingface-hub>=0.20",
|
||||
"accelerate>=0.28",
|
||||
"bitsandbytes>=0.43",
|
||||
"rich>=13",
|
||||
"safetensors>=0.4",
|
||||
"torch>=2.1",
|
||||
"transformers>=4.39",
|
||||
"websockets>=13",
|
||||
"zstandard>=0.22",
|
||||
"kernels>=0.11.1,<0.16",
|
||||
]
|
||||
|
||||
[project.scripts]
|
||||
meshnet-node = "meshnet_node.cli:main"
|
||||
|
||||
[tool.setuptools.packages.find]
|
||||
where = ["."]
|
||||
include = ["meshnet_node*"]
|
||||
|
||||
[tool.setuptools.package-data]
|
||||
meshnet_node = ["*.json"]
|
||||
[build-system]
|
||||
requires = ["setuptools>=64"]
|
||||
build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "meshnet-node"
|
||||
version = "0.1.0"
|
||||
description = "Distributed Inference Network node client"
|
||||
requires-python = ">=3.10"
|
||||
|
||||
dependencies = [
|
||||
"cryptography>=41",
|
||||
"huggingface-hub>=0.20",
|
||||
"accelerate>=0.28",
|
||||
"bitsandbytes>=0.43",
|
||||
"rich>=13",
|
||||
"safetensors>=0.4",
|
||||
"torch>=2.1",
|
||||
"transformers>=5.12",
|
||||
"websockets>=13",
|
||||
"zstandard>=0.22",
|
||||
"kernels>=0.11.1,<0.16",
|
||||
]
|
||||
|
||||
[project.scripts]
|
||||
meshnet-node = "meshnet_node.cli:main"
|
||||
|
||||
[tool.setuptools.packages.find]
|
||||
where = ["."]
|
||||
include = ["meshnet_node*"]
|
||||
|
||||
[tool.setuptools.package-data]
|
||||
meshnet_node = ["*.json"]
|
||||
|
||||
Reference in New Issue
Block a user