158 lines
6.1 KiB
Markdown
158 lines
6.1 KiB
Markdown
# US-018 — End-to-end two-machine LAN inference test
|
||
|
||
## Goal
|
||
|
||
Run real distributed inference across two physical machines: the Linux rig and a Windows 11 rig running WSL2. Document every setup step, firewall rule, and gotcha so this is repeatable. The test script exits 0 with token output and timing, proving the network works.
|
||
|
||
## Network topology for LAN test
|
||
|
||
```
|
||
[Linux machine] [Windows 11 / WSL2]
|
||
meshnet-tracker :8080 meshnet-node (shard B)
|
||
meshnet-node :8001 (shard A, tracker-mode)
|
||
meshnet-gateway :8000 (optional, for OpenAI-compat)
|
||
|
||
Client (either machine):
|
||
scripts/test_lan_inference.py --tracker http://192.168.1.10:8080
|
||
```
|
||
|
||
The Linux machine runs the tracker + the first-shard node (tracker-mode). The Windows/WSL2 machine runs the second-shard node. A small model (e.g. Phi-3-medium at BF16, fits on one GPU each) is split across both.
|
||
|
||
## WSL2 setup (Windows side)
|
||
|
||
`docs/INSTALL_WINDOWS.md` covers:
|
||
|
||
1. Enable WSL2: `wsl --install -d Ubuntu-24.04`
|
||
2. CUDA in WSL2: install NVIDIA driver on Windows (NOT inside WSL); WSL2 gets CUDA automatically
|
||
- Verify: `nvidia-smi` inside WSL2 should show GPU
|
||
3. Install Python 3.11+ and pip inside WSL2
|
||
4. `pip install -e packages/node packages/p2p` (clone repo first)
|
||
5. Firewall: Windows Defender must allow inbound WSL2 → LAN on node port
|
||
- PowerShell: `New-NetFirewallRule -DisplayName "meshnet-node" -Direction Inbound -Protocol TCP -LocalPort 8001 -Action Allow`
|
||
6. WSL2 IP: WSL2 has its own NAT'd IP (172.x.x.x); to expose to LAN, either:
|
||
- Option A: `netsh interface portproxy add v4tov4 listenport=8001 listenaddress=0.0.0.0 connectport=8001 connectaddress=$(wsl hostname -I)`
|
||
- Option B: use the relay node (US-017) — no port forwarding needed
|
||
|
||
## Linux setup
|
||
|
||
Standard install (already done after US-016). Firewall:
|
||
|
||
```bash
|
||
# If using ufw
|
||
sudo ufw allow 8080/tcp # tracker
|
||
sudo ufw allow 8001/tcp # node
|
||
sudo ufw allow 8000/tcp # gateway (optional)
|
||
```
|
||
|
||
## Model split
|
||
|
||
For the test, use a model that has enough layers to split meaningfully but fits comfortably in memory. Phi-3-medium-128k-instruct (40 layers, BF16 15GB) works on a single 24GB GPU on each machine:
|
||
|
||
- Linux node: layers 0–19 (tracker-mode, owns tokenizer + embed_tokens)
|
||
- Windows/WSL2 node: layers 20–39
|
||
|
||
Start sequence:
|
||
```bash
|
||
# Terminal 1 (Linux) — tracker
|
||
meshnet-tracker --port 8080
|
||
|
||
# Terminal 2 (Linux) — first-shard node (tracker-mode auto-detected because shard_start=0)
|
||
meshnet-node --model microsoft/Phi-3-medium-128k-instruct \
|
||
--quantization bf16 \
|
||
--shard-start 0 --shard-end 19 \
|
||
--tracker http://localhost:8080 \
|
||
--port 8001
|
||
|
||
# Terminal 3 (Windows WSL2) — second-shard node
|
||
meshnet-node --model microsoft/Phi-3-medium-128k-instruct \
|
||
--quantization bf16 \
|
||
--shard-start 20 --shard-end 39 \
|
||
--tracker http://192.168.1.10:8080 \
|
||
--port 8001
|
||
```
|
||
|
||
## Test script
|
||
|
||
`scripts/test_lan_inference.py`:
|
||
|
||
```python
|
||
#!/usr/bin/env python3
|
||
"""
|
||
End-to-end LAN inference test.
|
||
Usage: python scripts/test_lan_inference.py --tracker http://192.168.1.10:8080
|
||
"""
|
||
import argparse, time, httpx, json
|
||
|
||
MESSAGES = [
|
||
{"role": "user", "content": "What is 7 × 8? Answer in one word."},
|
||
{"role": "user", "content": "Name the capital of France in one word."},
|
||
{"role": "user", "content": "Complete the sequence: 1, 1, 2, 3, 5, ___"},
|
||
]
|
||
|
||
def run_test(tracker_url: str, gateway_url: str | None):
|
||
# Discover inference entry point via tracker if gateway not given
|
||
if not gateway_url:
|
||
r = httpx.get(f"{tracker_url}/v1/tracker-nodes/phi-3-medium", timeout=5)
|
||
r.raise_for_status()
|
||
nodes = r.json()
|
||
assert nodes, "No tracker-mode nodes registered — is the first-shard node running?"
|
||
gateway_url = nodes[0]["url"]
|
||
|
||
print(f"Inference endpoint: {gateway_url}")
|
||
print(f"Tracker: {tracker_url}")
|
||
print()
|
||
|
||
for i, msg in enumerate(MESSAGES):
|
||
t0 = time.monotonic()
|
||
r = httpx.post(
|
||
f"{gateway_url}/v1/chat/completions",
|
||
json={"model": "phi-3-medium", "messages": [msg], "stream": False},
|
||
timeout=60,
|
||
)
|
||
r.raise_for_status()
|
||
data = r.json()
|
||
elapsed = time.monotonic() - t0
|
||
|
||
content = data["choices"][0]["message"]["content"]
|
||
tokens = data["usage"]["completion_tokens"]
|
||
tps = tokens / elapsed if elapsed > 0 else 0
|
||
|
||
print(f"[{i+1}] Q: {msg['content']}")
|
||
print(f" A: {content}")
|
||
print(f" {tokens} tokens {elapsed:.2f}s {tps:.1f} t/s")
|
||
print()
|
||
|
||
print("✓ All 3 requests completed successfully")
|
||
|
||
if __name__ == "__main__":
|
||
p = argparse.ArgumentParser()
|
||
p.add_argument("--tracker", required=True)
|
||
p.add_argument("--gateway", default=None)
|
||
args = p.parse_args()
|
||
run_test(args.tracker, args.gateway)
|
||
```
|
||
|
||
## Docs: TWO_MACHINE_TEST.md
|
||
|
||
`docs/TWO_MACHINE_TEST.md` must cover:
|
||
|
||
1. Prerequisites (models downloaded on both machines, same model ID, complementary shard ranges)
|
||
2. Start order: tracker first, then nodes, then test script
|
||
3. How to verify nodes are registered: `GET /v1/nodes` on tracker
|
||
4. How to verify coverage: `GET /v1/coverage/phi-3-medium` — all 40 layers must show node_count ≥ 1
|
||
5. How to run the test script
|
||
6. Expected output
|
||
7. Latency breakdown: how to read per-hop latency from node logs
|
||
8. **Known Issues** section — updated during actual test run with real gotchas
|
||
|
||
## Acceptance criteria
|
||
|
||
- `docs/INSTALL_WINDOWS.md` covers WSL2 + CUDA + meshnet-node install end-to-end
|
||
- `docs/TWO_MACHINE_TEST.md` covers the full two-machine setup and test procedure
|
||
- `scripts/test_lan_inference.py` exists and is executable
|
||
- When run against a real two-machine LAN setup: script exits 0, prints 3 valid answers with timing
|
||
- Coverage map shows 100% coverage (no gap) after both nodes register
|
||
- Known Issues section in TWO_MACHINE_TEST.md contains at least the issues encountered during this test run
|
||
- No new pytest failures from repo root (this story adds docs + a script, not new Python packages)
|
||
- Commit only this story's changes
|