popov/neuron-tai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Document transformers>=5.12 requirement and Qwen3.5/3.6-MoE fast-path notes

Browse Source 
Bump the node package's transformers floor to 5.12 (older versions lack
composite Qwen3_5MoeConfig handling and fail with missing vocab_size), and
explain in QUICKSTART/INSTALL_WINDOWS that the flash-linear-attention /
causal-conv1d fast-path warning is a harmless CPU fallback.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Dobromir Popov
2026-07-07 19:18:51 +02:00
parent 471893c9d5
commit 7ba87051f5
3 changed files with 1036 additions and 1006 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1544
QUICKSTART.md
View File

File diff suppressed because it is too large Load Diff

432
docs/INSTALL_WINDOWS.md
View File

@@ -1,212 +1,220 @@
# Installing meshnet-node on Windows 11 with WSL2 # Installing meshnet-node on Windows 11 with WSL2
This guide covers setting up a meshnet-node on a Windows 11 machine using WSL2 with CUDA passthrough so it can join an existing inference network over LAN. This guide covers setting up a meshnet-node on a Windows 11 machine using WSL2 with CUDA passthrough so it can join an existing inference network over LAN.
## Prerequisites ## Prerequisites
- Windows 11 with WSL2 support (most systems with Windows 10 version 2004+ qualify) - Windows 11 with WSL2 support (most systems with Windows 10 version 2004+ qualify)
- NVIDIA GPU with CUDA support (driver ≥ 525.x recommended for WSL2 CUDA) - NVIDIA GPU with CUDA support (driver ≥ 525.x recommended for WSL2 CUDA)
- At least 8 GB RAM + enough VRAM for the model shard you intend to serve - At least 8 GB RAM + enough VRAM for the model shard you intend to serve
- The Linux machine (other node) is reachable on your LAN - The Linux machine (other node) is reachable on your LAN
--- ---
## Step 1 — Enable WSL2 and install Ubuntu ## Step 1 — Enable WSL2 and install Ubuntu
Open **PowerShell as Administrator** and run: Open **PowerShell as Administrator** and run:
```powershell ```powershell
wsl --install -d Ubuntu-24.04 wsl --install -d Ubuntu-24.04
``` ```
This installs WSL2 with Ubuntu 24.04. Reboot when prompted. This installs WSL2 with Ubuntu 24.04. Reboot when prompted.
After reboot, Ubuntu starts and asks you to create a UNIX username/password. Choose anything convenient. After reboot, Ubuntu starts and asks you to create a UNIX username/password. Choose anything convenient.
Verify WSL version: Verify WSL version:
```powershell ```powershell
wsl -l -v wsl -l -v
``` ```
Output should show `VERSION 2`. Output should show `VERSION 2`.
--- ---
## Step 2 — Install NVIDIA GPU driver on Windows (NOT inside WSL) ## Step 2 — Install NVIDIA GPU driver on Windows (NOT inside WSL)
WSL2 CUDA passthrough works through the Windows host driver. **Do not install CUDA inside WSL2.** WSL2 CUDA passthrough works through the Windows host driver. **Do not install CUDA inside WSL2.**
1. Download the latest Game Ready or Studio driver for your GPU from https://www.nvidia.com/drivers 1. Download the latest Game Ready or Studio driver for your GPU from https://www.nvidia.com/drivers
2. Install on Windows normally (standard installer). 2. Install on Windows normally (standard installer).
3. Inside WSL2 (Ubuntu terminal), verify: 3. Inside WSL2 (Ubuntu terminal), verify:
```bash ```bash
nvidia-smi nvidia-smi
``` ```
Expected output: your GPU name, driver version, CUDA version. If this command fails, the Windows driver is too old — update it. Expected output: your GPU name, driver version, CUDA version. If this command fails, the Windows driver is too old — update it.
> **Note:** The `cuda-toolkit` package inside WSL2 is optional and only needed if you compile CUDA kernels. For inference with `torch`, the Windows host driver is sufficient. > **Note:** The `cuda-toolkit` package inside WSL2 is optional and only needed if you compile CUDA kernels. For inference with `torch`, the Windows host driver is sufficient.
--- ---
## Step 3 — Install Python 3.11+ inside WSL2 ## Step 3 — Install Python 3.11+ inside WSL2
Ubuntu 24.04 ships Python 3.12. Confirm: Ubuntu 24.04 ships Python 3.12. Confirm:
```bash ```bash
python3 --version python3 --version
``` ```
If it shows 3.10 or older: If it shows 3.10 or older:
```bash ```bash
sudo add-apt-repository ppa:deadsnakes/ppa sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt update sudo apt update
sudo apt install python3.12 python3.12-venv python3.12-dev sudo apt install python3.12 python3.12-venv python3.12-dev
``` ```
Install pip: Install pip:
```bash ```bash
curl -sS https://bootstrap.pypa.io/get-pip.py | python3 curl -sS https://bootstrap.pypa.io/get-pip.py | python3
``` ```
--- ---
## Step 4 — Clone the repository ## Step 4 — Clone the repository
Inside WSL2: Inside WSL2:
```bash ```bash
# Store the repo in the Linux filesystem (faster I/O than /mnt/c) # Store the repo in the Linux filesystem (faster I/O than /mnt/c)
cd ~ cd ~
git clone https://github.com/YOUR_ORG/d-popov.com.git git clone https://github.com/YOUR_ORG/d-popov.com.git
cd d-popov.com/AI cd d-popov.com/AI
``` ```
--- ---
## Step 5 — Create a virtualenv and install meshnet-node ## Step 5 — Create a virtualenv and install meshnet-node
```bash ```bash
python3 -m venv .venv python3 -m venv .venv
source .venv/bin/activate source .venv/bin/activate
# Install node + PyTorch (CUDA build) # Install node + PyTorch (CUDA build)
pip install torch --index-url https://download.pytorch.org/whl/cu124 pip install torch --index-url https://download.pytorch.org/whl/cu124
pip install -e "packages/node[torch]" pip install -e "packages/node[torch]"
``` ```
Verify the install: Verify the install:
```bash ```bash
meshnet-node --help meshnet-node --help
``` python -c "import transformers; print(transformers.__version__)"
```
---
`transformers` must be **≥ 5.12** for Qwen3.5/3.6-MoE models (older versions fail
## Step 6 — Pre-download the model shard with `'Qwen3_5MoeConfig' object has no attribute 'vocab_size'`). If you install
into an existing conda/miniforge env instead of a fresh venv, run
Download the model before starting the node so the startup process doesn't time out on the tracker side: `pip install -U transformers` there. The startup warning about
`flash-linear-attention` / `causal-conv1d` ("fast path is not available") is
```bash harmless on CPU — those are optional CUDA-only kernels.
python3 - <<'EOF'
from transformers import AutoConfig ---
AutoConfig.from_pretrained("microsoft/Phi-3-medium-128k-instruct")
EOF ## Step 6 — Pre-download the model shard
```
Download the model before starting the node so the startup process doesn't time out on the tracker side:
For the full model weights (needed at runtime), `transformers` downloads them automatically on first `meshnet-node` start. If you want to pre-fetch:
```bash
```bash python3 - <<'EOF'
python3 -c " from transformers import AutoConfig
from transformers import AutoModelForCausalLM AutoConfig.from_pretrained("microsoft/Phi-3-medium-128k-instruct")
AutoModelForCausalLM.from_pretrained('microsoft/Phi-3-medium-128k-instruct', device_map='cpu') EOF
" ```
```
For the full model weights (needed at runtime), `transformers` downloads them automatically on first `meshnet-node` start. If you want to pre-fetch:
This can take 10–30 minutes on first run.
```bash
--- python3 -c "
from transformers import AutoModelForCausalLM
## Step 7 — Expose the node port to your LAN AutoModelForCausalLM.from_pretrained('microsoft/Phi-3-medium-128k-instruct', device_map='cpu')
"
WSL2 runs behind a NAT with a virtual IP (typically `172.x.x.x`). Your LAN sees the Windows host IP. You need to forward the node port. ```
**Option A — Windows port proxy (recommended for simple setups):** This can take 10–30 minutes on first run.
In **PowerShell as Administrator**: ---
```powershell ## Step 7 — Expose the node port to your LAN
# Get the current WSL2 IP (changes on each WSL restart)
$wslIp = (wsl hostname -I).Trim() WSL2 runs behind a NAT with a virtual IP (typically `172.x.x.x`). Your LAN sees the Windows host IP. You need to forward the node port.
# Forward Windows host port 8001 → WSL2 port 8001 **Option A — Windows port proxy (recommended for simple setups):**
netsh interface portproxy add v4tov4 `
listenport=8001 listenaddress=0.0.0.0 ` In **PowerShell as Administrator**:
connectport=8001 connectaddress=$wslIp
```powershell
# Allow inbound on Windows Firewall # Get the current WSL2 IP (changes on each WSL restart)
New-NetFirewallRule -DisplayName "meshnet-node" ` $wslIp = (wsl hostname -I).Trim()
-Direction Inbound -Protocol TCP -LocalPort 8001 -Action Allow
``` # Forward Windows host port 8001 → WSL2 port 8001
netsh interface portproxy add v4tov4 `
Verify: from the Linux machine, `curl http://WINDOWS_LAN_IP:8001/v1/health` should return a response once the node is running. listenport=8001 listenaddress=0.0.0.0 `
connectport=8001 connectaddress=$wslIp
**Redo this after every WSL2 restart** — the WSL2 IP changes.
# Allow inbound on Windows Firewall
**Option B — P2P relay (US-017, no port forwarding needed):** New-NetFirewallRule -DisplayName "meshnet-node" `
-Direction Inbound -Protocol TCP -LocalPort 8001 -Action Allow
Start a relay node on the Linux machine. The WSL2 node connects outbound through the relay. No firewall rules needed. See `docs/TWO_MACHINE_TEST.md` for details. ```
--- Verify: from the Linux machine, `curl http://WINDOWS_LAN_IP:8001/v1/health` should return a response once the node is running.
## Step 8 — Start the node **Redo this after every WSL2 restart** — the WSL2 IP changes.
Replace `192.168.1.10` with the actual LAN IP of the Linux machine running the tracker. **Option B — P2P relay (US-017, no port forwarding needed):**
Replace shard range with the complementary range to what the Linux node is serving.
Start a relay node on the Linux machine. The WSL2 node connects outbound through the relay. No firewall rules needed. See `docs/TWO_MACHINE_TEST.md` for details.
```bash
source .venv/bin/activate ---
meshnet-node \ ## Step 8 — Start the node
--model microsoft/Phi-3-medium-128k-instruct \
--quantization bf16 \ Replace `192.168.1.10` with the actual LAN IP of the Linux machine running the tracker.
--shard-start 20 --shard-end 39 \ Replace shard range with the complementary range to what the Linux node is serving.
--tracker http://192.168.1.10:8080 \
--port 8001 \ ```bash
--host 0.0.0.0 \ source .venv/bin/activate
--advertise-host WINDOWS_LAN_IP
``` meshnet-node \
--model microsoft/Phi-3-medium-128k-instruct \
The `--advertise-host` flag tells the tracker what IP the Linux machine should use to reach this node. Use your Windows machine's LAN IP (e.g. `192.168.1.20`), **not** the WSL2 internal IP. --quantization bf16 \
--shard-start 20 --shard-end 39 \
Expected startup output: --tracker http://192.168.1.10:8080 \
--port 8001 \
``` --host 0.0.0.0 \
Detecting hardware... --advertise-host WINDOWS_LAN_IP
GPU: NVIDIA GeForce RTX 3080 (10240 MB VRAM) ```
Loading wallet...
Wallet: 5K7r... The `--advertise-host` flag tells the tracker what IP the Linux machine should use to reach this node. Use your Windows machine's LAN IP (e.g. `192.168.1.20`), **not** the WSL2 internal IP.
Loading real PyTorch model shard...
Auto-detected 40 layers → shard 20–39 Expected startup output:
================================
meshnet-node ready ```
Model ID: microsoft/Phi-3-medium-128k-instruct Detecting hardware...
Shard: layers 20–39; 20 of 40 GPU: NVIDIA GeForce RTX 3080 (10240 MB VRAM)
Endpoint: http://192.168.1.20:8001 Loading wallet...
Hardware: CUDA Wallet: 5K7r...
================================ Loading real PyTorch model shard...
``` Auto-detected 40 layers → shard 20–39
================================
--- meshnet-node ready
Model ID: microsoft/Phi-3-medium-128k-instruct
## Known issues Shard: layers 20–39; 20 of 40
Endpoint: http://192.168.1.20:8001
- **WSL2 IP changes on restart.** Always re-run the `netsh` port-proxy command after restarting WSL2 or Windows. Hardware: CUDA
- **CUDA not visible in WSL2.** If `nvidia-smi` fails inside WSL2, update the Windows host GPU driver to ≥ 525.x. Installing CUDA inside WSL2 will not fix it. ================================
- **Model download is slow.** HuggingFace downloads happen over HTTPS. Pre-fetch the model before a timed test (see Step 6). ```
- **Port 8001 already in use.** Change `--port` to another value and update the firewall/portproxy rules accordingly.
- **`bf16` not supported on older GPUs.** Use `--quantization int8` on Turing (RTX 20xx) cards or earlier if bfloat16 ops fail. ---
## Known issues
- **WSL2 IP changes on restart.** Always re-run the `netsh` port-proxy command after restarting WSL2 or Windows.
- **CUDA not visible in WSL2.** If `nvidia-smi` fails inside WSL2, update the Windows host GPU driver to ≥ 525.x. Installing CUDA inside WSL2 will not fix it.
- **Model download is slow.** HuggingFace downloads happen over HTTPS. Pre-fetch the model before a timed test (see Step 6).
- **Port 8001 already in use.** Change `--port` to another value and update the firewall/portproxy rules accordingly.
- **`bf16` not supported on older GPUs.** Use `--quantization int8` on Turing (RTX 20xx) cards or earlier if bfloat16 ops fail.

66
packages/node/pyproject.toml
View File

@@ -1,33 +1,33 @@
[build-system] [build-system]
requires = ["setuptools>=64"] requires = ["setuptools>=64"]
build-backend = "setuptools.build_meta" build-backend = "setuptools.build_meta"
[project] [project]
name = "meshnet-node" name = "meshnet-node"
version = "0.1.0" version = "0.1.0"
description = "Distributed Inference Network node client" description = "Distributed Inference Network node client"
requires-python = ">=3.10" requires-python = ">=3.10"
dependencies = [ dependencies = [
"cryptography>=41", "cryptography>=41",
"huggingface-hub>=0.20", "huggingface-hub>=0.20",
"accelerate>=0.28", "accelerate>=0.28",
"bitsandbytes>=0.43", "bitsandbytes>=0.43",
"rich>=13", "rich>=13",
"safetensors>=0.4", "safetensors>=0.4",
"torch>=2.1", "torch>=2.1",
"transformers>=4.39", "transformers>=5.12",
"websockets>=13", "websockets>=13",
"zstandard>=0.22", "zstandard>=0.22",
"kernels>=0.11.1,<0.16", "kernels>=0.11.1,<0.16",
] ]
[project.scripts] [project.scripts]
meshnet-node = "meshnet_node.cli:main" meshnet-node = "meshnet_node.cli:main"
[tool.setuptools.packages.find] [tool.setuptools.packages.find]
where = ["."] where = ["."]
include = ["meshnet_node*"] include = ["meshnet_node*"]
[tool.setuptools.package-data] [tool.setuptools.package-data]
meshnet_node = ["*.json"] meshnet_node = ["*.json"]