popov/neuron-tai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Normalize line endings to LF via .gitattributes

Browse Source 
Adds a committed .gitattributes so Windows and Linux checkouts converge
on LF for all text files, overriding each developer's local core.autocrlf.
Renormalizes existing blobs (server.py, dashboard.html, etc.) that had
CRLF baked in, clearing the repo-wide phantom "modified" churn.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Dobromir Popov
2026-07-08 16:15:32 +02:00
parent 9c73db0ef2
commit 560de08edd
13 changed files with 6705 additions and 6677 deletions
Download Patch File Download Diff File Expand all files Collapse all files

424
docs/INSTALL_WINDOWS.md
View File

@@ -1,111 +1,111 @@
# Installing meshnet-node on Windows 11 with WSL2
This guide covers setting up a meshnet-node on a Windows 11 machine using WSL2 with CUDA passthrough so it can join an existing inference network over LAN.
## Prerequisites
- Windows 11 with WSL2 support (most systems with Windows 10 version 2004+ qualify)
- NVIDIA GPU with CUDA support (driver ≥ 525.x recommended for WSL2 CUDA)
- At least 8 GB RAM + enough VRAM for the model shard you intend to serve
- The Linux machine (other node) is reachable on your LAN
---
## Step 1 — Enable WSL2 and install Ubuntu
Open **PowerShell as Administrator** and run:
```powershell
wsl --install -d Ubuntu-24.04
```
This installs WSL2 with Ubuntu 24.04. Reboot when prompted.
After reboot, Ubuntu starts and asks you to create a UNIX username/password. Choose anything convenient.
Verify WSL version:
```powershell
wsl -l -v
```
Output should show `VERSION 2`.
---
## Step 2 — Install NVIDIA GPU driver on Windows (NOT inside WSL)
WSL2 CUDA passthrough works through the Windows host driver. **Do not install CUDA inside WSL2.**
1. Download the latest Game Ready or Studio driver for your GPU from https://www.nvidia.com/drivers
2. Install on Windows normally (standard installer).
3. Inside WSL2 (Ubuntu terminal), verify:
```bash
nvidia-smi
```
Expected output: your GPU name, driver version, CUDA version. If this command fails, the Windows driver is too old — update it.
> **Note:** The `cuda-toolkit` package inside WSL2 is optional and only needed if you compile CUDA kernels. For inference with `torch`, the Windows host driver is sufficient.
---
## Step 3 — Install Python 3.11+ inside WSL2
Ubuntu 24.04 ships Python 3.12. Confirm:
```bash
python3 --version
```
If it shows 3.10 or older:
```bash
sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt update
sudo apt install python3.12 python3.12-venv python3.12-dev
```
Install pip:
```bash
curl -sS https://bootstrap.pypa.io/get-pip.py | python3
```
---
## Step 4 — Clone the repository
Inside WSL2:
```bash
# Store the repo in the Linux filesystem (faster I/O than /mnt/c)
cd ~
git clone https://github.com/YOUR_ORG/d-popov.com.git
cd d-popov.com/AI
```
---
## Step 5 — Create a virtualenv and install meshnet-node
```bash
python3 -m venv .venv
source .venv/bin/activate
# Install node + PyTorch (CUDA build)
pip install torch --index-url https://download.pytorch.org/whl/cu124
pip install -e "packages/node[torch]"
```
Verify the install:
```bash
meshnet-node --help
python -c "import transformers; print(transformers.__version__)"
```
# Installing meshnet-node on Windows 11 with WSL2
This guide covers setting up a meshnet-node on a Windows 11 machine using WSL2 with CUDA passthrough so it can join an existing inference network over LAN.
## Prerequisites
- Windows 11 with WSL2 support (most systems with Windows 10 version 2004+ qualify)
- NVIDIA GPU with CUDA support (driver ≥ 525.x recommended for WSL2 CUDA)
- At least 8 GB RAM + enough VRAM for the model shard you intend to serve
- The Linux machine (other node) is reachable on your LAN
---
## Step 1 — Enable WSL2 and install Ubuntu
Open **PowerShell as Administrator** and run:
```powershell
wsl --install -d Ubuntu-24.04
```
This installs WSL2 with Ubuntu 24.04. Reboot when prompted.
After reboot, Ubuntu starts and asks you to create a UNIX username/password. Choose anything convenient.
Verify WSL version:
```powershell
wsl -l -v
```
Output should show `VERSION 2`.
---
## Step 2 — Install NVIDIA GPU driver on Windows (NOT inside WSL)
WSL2 CUDA passthrough works through the Windows host driver. **Do not install CUDA inside WSL2.**
1. Download the latest Game Ready or Studio driver for your GPU from https://www.nvidia.com/drivers
2. Install on Windows normally (standard installer).
3. Inside WSL2 (Ubuntu terminal), verify:
```bash
nvidia-smi
```
Expected output: your GPU name, driver version, CUDA version. If this command fails, the Windows driver is too old — update it.
> **Note:** The `cuda-toolkit` package inside WSL2 is optional and only needed if you compile CUDA kernels. For inference with `torch`, the Windows host driver is sufficient.
---
## Step 3 — Install Python 3.11+ inside WSL2
Ubuntu 24.04 ships Python 3.12. Confirm:
```bash
python3 --version
```
If it shows 3.10 or older:
```bash
sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt update
sudo apt install python3.12 python3.12-venv python3.12-dev
```
Install pip:
```bash
curl -sS https://bootstrap.pypa.io/get-pip.py | python3
```
---
## Step 4 — Clone the repository
Inside WSL2:
```bash
# Store the repo in the Linux filesystem (faster I/O than /mnt/c)
cd ~
git clone https://github.com/YOUR_ORG/d-popov.com.git
cd d-popov.com/AI
```
---
## Step 5 — Create a virtualenv and install meshnet-node
```bash
python3 -m venv .venv
source .venv/bin/activate
# Install node + PyTorch (CUDA build)
pip install torch --index-url https://download.pytorch.org/whl/cu124
pip install -e "packages/node[torch]"
```
Verify the install:
```bash
meshnet-node --help
python -c "import transformers; print(transformers.__version__)"
```
`transformers` must be **≥ 5.12** for Qwen3.5/3.6-MoE models (older versions fail
with `'Qwen3_5MoeConfig' object has no attribute 'vocab_size'`). If you install
into an existing conda/miniforge env instead of a fresh venv, run
@@ -124,107 +124,107 @@ Without it, Qwen3.5/3.6-MoE startup can fail with the misleading message
`Could not import module 'Qwen3_5MoeForCausalLM'`.
---
## Step 6 — Pre-download the model shard
Download the model before starting the node so the startup process doesn't time out on the tracker side:
```bash
python3 - <<'EOF'
from transformers import AutoConfig
AutoConfig.from_pretrained("microsoft/Phi-3-medium-128k-instruct")
EOF
```
For the full model weights (needed at runtime), `transformers` downloads them automatically on first `meshnet-node` start. If you want to pre-fetch:
```bash
python3 -c "
from transformers import AutoModelForCausalLM
AutoModelForCausalLM.from_pretrained('microsoft/Phi-3-medium-128k-instruct', device_map='cpu')
"
```
This can take 10–30 minutes on first run.
---
## Step 7 — Expose the node port to your LAN
WSL2 runs behind a NAT with a virtual IP (typically `172.x.x.x`). Your LAN sees the Windows host IP. You need to forward the node port.
**Option A — Windows port proxy (recommended for simple setups):**
In **PowerShell as Administrator**:
```powershell
# Get the current WSL2 IP (changes on each WSL restart)
$wslIp = (wsl hostname -I).Trim()
# Forward Windows host port 8001 → WSL2 port 8001
netsh interface portproxy add v4tov4 `
listenport=8001 listenaddress=0.0.0.0 `
connectport=8001 connectaddress=$wslIp
# Allow inbound on Windows Firewall
New-NetFirewallRule -DisplayName "meshnet-node" `
-Direction Inbound -Protocol TCP -LocalPort 8001 -Action Allow
```
Verify: from the Linux machine, `curl http://WINDOWS_LAN_IP:8001/v1/health` should return a response once the node is running.
**Redo this after every WSL2 restart** — the WSL2 IP changes.
**Option B — P2P relay (US-017, no port forwarding needed):**
Start a relay node on the Linux machine. The WSL2 node connects outbound through the relay. No firewall rules needed. See `docs/TWO_MACHINE_TEST.md` for details.
---
## Step 8 — Start the node
Replace `192.168.1.10` with the actual LAN IP of the Linux machine running the tracker.
Replace shard range with the complementary range to what the Linux node is serving.
```bash
source .venv/bin/activate
meshnet-node \
--model microsoft/Phi-3-medium-128k-instruct \
--quantization bf16 \
--shard-start 20 --shard-end 39 \
--tracker http://192.168.1.10:8080 \
--port 8001 \
--host 0.0.0.0 \
--advertise-host WINDOWS_LAN_IP
```
The `--advertise-host` flag tells the tracker what IP the Linux machine should use to reach this node. Use your Windows machine's LAN IP (e.g. `192.168.1.20`), **not** the WSL2 internal IP.
Expected startup output:
```
Detecting hardware...
GPU: NVIDIA GeForce RTX 3080 (10240 MB VRAM)
Loading wallet...
Wallet: 5K7r...
Loading real PyTorch model shard...
Auto-detected 40 layers → shard 20–39
================================
meshnet-node ready
Model ID: microsoft/Phi-3-medium-128k-instruct
Shard: layers 20–39; 20 of 40
Endpoint: http://192.168.1.20:8001
Hardware: CUDA
================================
```
---
## Known issues
- **WSL2 IP changes on restart.** Always re-run the `netsh` port-proxy command after restarting WSL2 or Windows.
- **CUDA not visible in WSL2.** If `nvidia-smi` fails inside WSL2, update the Windows host GPU driver to ≥ 525.x. Installing CUDA inside WSL2 will not fix it.
- **Model download is slow.** HuggingFace downloads happen over HTTPS. Pre-fetch the model before a timed test (see Step 6).
- **Port 8001 already in use.** Change `--port` to another value and update the firewall/portproxy rules accordingly.
- **`bf16` not supported on older GPUs.** Use `--quantization int8` on Turing (RTX 20xx) cards or earlier if bfloat16 ops fail.
## Step 6 — Pre-download the model shard
Download the model before starting the node so the startup process doesn't time out on the tracker side:
```bash
python3 - <<'EOF'
from transformers import AutoConfig
AutoConfig.from_pretrained("microsoft/Phi-3-medium-128k-instruct")
EOF
```
For the full model weights (needed at runtime), `transformers` downloads them automatically on first `meshnet-node` start. If you want to pre-fetch:
```bash
python3 -c "
from transformers import AutoModelForCausalLM
AutoModelForCausalLM.from_pretrained('microsoft/Phi-3-medium-128k-instruct', device_map='cpu')
"
```
This can take 10–30 minutes on first run.
---
## Step 7 — Expose the node port to your LAN
WSL2 runs behind a NAT with a virtual IP (typically `172.x.x.x`). Your LAN sees the Windows host IP. You need to forward the node port.
**Option A — Windows port proxy (recommended for simple setups):**
In **PowerShell as Administrator**:
```powershell
# Get the current WSL2 IP (changes on each WSL restart)
$wslIp = (wsl hostname -I).Trim()
# Forward Windows host port 8001 → WSL2 port 8001
netsh interface portproxy add v4tov4 `
listenport=8001 listenaddress=0.0.0.0 `
connectport=8001 connectaddress=$wslIp
# Allow inbound on Windows Firewall
New-NetFirewallRule -DisplayName "meshnet-node" `
-Direction Inbound -Protocol TCP -LocalPort 8001 -Action Allow
```
Verify: from the Linux machine, `curl http://WINDOWS_LAN_IP:8001/v1/health` should return a response once the node is running.
**Redo this after every WSL2 restart** — the WSL2 IP changes.
**Option B — P2P relay (US-017, no port forwarding needed):**
Start a relay node on the Linux machine. The WSL2 node connects outbound through the relay. No firewall rules needed. See `docs/TWO_MACHINE_TEST.md` for details.
---
## Step 8 — Start the node
Replace `192.168.1.10` with the actual LAN IP of the Linux machine running the tracker.
Replace shard range with the complementary range to what the Linux node is serving.
```bash
source .venv/bin/activate
meshnet-node \
--model microsoft/Phi-3-medium-128k-instruct \
--quantization bf16 \
--shard-start 20 --shard-end 39 \
--tracker http://192.168.1.10:8080 \
--port 8001 \
--host 0.0.0.0 \
--advertise-host WINDOWS_LAN_IP
```
The `--advertise-host` flag tells the tracker what IP the Linux machine should use to reach this node. Use your Windows machine's LAN IP (e.g. `192.168.1.20`), **not** the WSL2 internal IP.
Expected startup output:
```
Detecting hardware...
GPU: NVIDIA GeForce RTX 3080 (10240 MB VRAM)
Loading wallet...
Wallet: 5K7r...
Loading real PyTorch model shard...
Auto-detected 40 layers → shard 20–39
================================
meshnet-node ready
Model ID: microsoft/Phi-3-medium-128k-instruct
Shard: layers 20–39; 20 of 40
Endpoint: http://192.168.1.20:8001
Hardware: CUDA
================================
```
---
## Known issues
- **WSL2 IP changes on restart.** Always re-run the `netsh` port-proxy command after restarting WSL2 or Windows.
- **CUDA not visible in WSL2.** If `nvidia-smi` fails inside WSL2, update the Windows host GPU driver to ≥ 525.x. Installing CUDA inside WSL2 will not fix it.
- **Model download is slow.** HuggingFace downloads happen over HTTPS. Pre-fetch the model before a timed test (see Step 6).
- **Port 8001 already in use.** Change `--port` to another value and update the firewall/portproxy rules accordingly.
- **`bf16` not supported on older GPUs.** Use `--quantization int8` on Turing (RTX 20xx) cards or earlier if bfloat16 ops fail.

96
docs/issues/30-manual-route-and-hop-benchmark.md
View File

@@ -1,48 +1,48 @@
# US-020 — Manual route selection + hop-penalty benchmarking
## Context
The tracker auto-selects inference routes based on synthetic benchmark scores. To measure
the real cost of adding hops (latency per node boundary), we need:
1. A way to pin a request to a specific route so we control the variable.
2. A benchmark endpoint that runs the same prompt through 1-node, 2-node, and 3-node
routes and records per-hop latency.
Results are stored to disk. Routing algorithm is **not** changed in this story — this is
data collection only. The data will inform a future routing optimisation story.
## Design decisions (grilled 2026-07-01)
| Decision | Choice |
|---|---|
| Route spec | Optional `route` field in JSON request body (list of node IDs) |
| Trigger | Explicit only — `POST /v1/benchmark/hop-penalty` endpoint |
| Auth | Header-presence stub (`Authorization` must be non-empty); real auth in future story |
| Routing integration | Store data only; routing algorithm unchanged |
| Persistence | Append to `benchmark_results.json` in tracker working dir; in-memory queryable |
## Acceptance criteria
- `POST /v1/chat/completions` accepts optional `"route": ["<node_id>", ...]` in the
request body. If present, the tracker uses those nodes in order instead of auto-selecting.
If absent, existing routing is unchanged (no breaking change for unaware clients).
- Missing or invalid node IDs in `route` return HTTP 400 with a descriptive error.
- `POST /v1/benchmark/hop-penalty` is auth-gated: requests without a non-empty
`Authorization` header return HTTP 401. Body: `{"model": "...", "prompt": "...",
"max_new_tokens": 64}`.
- Benchmark fans out to up to three routes: 1-node (single node covering all layers),
2-node (two consecutive shard nodes), 3-node (three nodes) — using whatever is
currently registered. Routes with insufficient coverage are skipped, not errored.
- Response includes per-route breakdown: `total_ms`, `per_hop_ms: [...]`,
`tokens_generated`, `route: [node_id, ...]`.
- Results are appended to `<tracker_working_dir>/benchmark_results.json` (created if
absent) as a JSON array. Each entry includes timestamp, model, prompt hash, and the
per-route breakdown.
- `GET /v1/benchmark/results` returns the stored results array. Also auth-gated.
- Clients that never send `route` or call `/v1/benchmark/*` are completely unaffected.
- Integration test: send the same prompt via a pinned 1-node route and a pinned 2-node
route; assert 2-node result has 2 entries in `per_hop_ms`; assert both records appear
in `benchmark_results.json`.
- `python -m pytest` passes from repo root.
- Commit only this story's changes.
# US-020 — Manual route selection + hop-penalty benchmarking
## Context
The tracker auto-selects inference routes based on synthetic benchmark scores. To measure
the real cost of adding hops (latency per node boundary), we need:
1. A way to pin a request to a specific route so we control the variable.
2. A benchmark endpoint that runs the same prompt through 1-node, 2-node, and 3-node
routes and records per-hop latency.
Results are stored to disk. Routing algorithm is **not** changed in this story — this is
data collection only. The data will inform a future routing optimisation story.
## Design decisions (grilled 2026-07-01)
| Decision | Choice |
|---|---|
| Route spec | Optional `route` field in JSON request body (list of node IDs) |
| Trigger | Explicit only — `POST /v1/benchmark/hop-penalty` endpoint |
| Auth | Header-presence stub (`Authorization` must be non-empty); real auth in future story |
| Routing integration | Store data only; routing algorithm unchanged |
| Persistence | Append to `benchmark_results.json` in tracker working dir; in-memory queryable |
## Acceptance criteria
- `POST /v1/chat/completions` accepts optional `"route": ["<node_id>", ...]` in the
request body. If present, the tracker uses those nodes in order instead of auto-selecting.
If absent, existing routing is unchanged (no breaking change for unaware clients).
- Missing or invalid node IDs in `route` return HTTP 400 with a descriptive error.
- `POST /v1/benchmark/hop-penalty` is auth-gated: requests without a non-empty
`Authorization` header return HTTP 401. Body: `{"model": "...", "prompt": "...",
"max_new_tokens": 64}`.
- Benchmark fans out to up to three routes: 1-node (single node covering all layers),
2-node (two consecutive shard nodes), 3-node (three nodes) — using whatever is
currently registered. Routes with insufficient coverage are skipped, not errored.
- Response includes per-route breakdown: `total_ms`, `per_hop_ms: [...]`,
`tokens_generated`, `route: [node_id, ...]`.
- Results are appended to `<tracker_working_dir>/benchmark_results.json` (created if
absent) as a JSON array. Each entry includes timestamp, model, prompt hash, and the
per-route breakdown.
- `GET /v1/benchmark/results` returns the stored results array. Also auth-gated.
- Clients that never send `route` or call `/v1/benchmark/*` are completely unaffected.
- Integration test: send the same prompt via a pinned 1-node route and a pinned 2-node
route; assert 2-node result has 2 entries in `per_hop_ms`; assert both records appear
in `benchmark_results.json`.
- `python -m pytest` passes from repo root.
- Commit only this story's changes.