inference working
This commit is contained in:
Dobromir Popov
@@ -0,0 +1,157 @@
|
||||
# US-018 — End-to-end two-machine LAN inference test
|
||||
|
||||
## Goal
|
||||
|
||||
Run real distributed inference across two physical machines: the Linux rig and a Windows 11 rig running WSL2. Document every setup step, firewall rule, and gotcha so this is repeatable. The test script exits 0 with token output and timing, proving the network works.
|
||||
|
||||
## Network topology for LAN test
|
||||
|
||||
```
|
||||
[Linux machine] [Windows 11 / WSL2]
|
||||
meshnet-tracker :8080 meshnet-node (shard B)
|
||||
meshnet-node :8001 (shard A, tracker-mode)
|
||||
meshnet-gateway :8000 (optional, for OpenAI-compat)
|
||||
|
||||
Client (either machine):
|
||||
scripts/test_lan_inference.py --tracker http://192.168.1.10:8080
|
||||
```
|
||||
|
||||
The Linux machine runs the tracker + the first-shard node (tracker-mode). The Windows/WSL2 machine runs the second-shard node. A small model (e.g. Phi-3-medium at BF16, fits on one GPU each) is split across both.
|
||||
|
||||
## WSL2 setup (Windows side)
|
||||
|
||||
`docs/INSTALL_WINDOWS.md` covers:
|
||||
|
||||
1. Enable WSL2: `wsl --install -d Ubuntu-24.04`
|
||||
2. CUDA in WSL2: install NVIDIA driver on Windows (NOT inside WSL); WSL2 gets CUDA automatically
|
||||
- Verify: `nvidia-smi` inside WSL2 should show GPU
|
||||
3. Install Python 3.11+ and pip inside WSL2
|
||||
4. `pip install -e packages/node packages/p2p` (clone repo first)
|
||||
5. Firewall: Windows Defender must allow inbound WSL2 → LAN on node port
|
||||
- PowerShell: `New-NetFirewallRule -DisplayName "meshnet-node" -Direction Inbound -Protocol TCP -LocalPort 8001 -Action Allow`
|
||||
6. WSL2 IP: WSL2 has its own NAT'd IP (172.x.x.x); to expose to LAN, either:
|
||||
- Option A: `netsh interface portproxy add v4tov4 listenport=8001 listenaddress=0.0.0.0 connectport=8001 connectaddress=$(wsl hostname -I)`
|
||||
- Option B: use the relay node (US-017) — no port forwarding needed
|
||||
|
||||
## Linux setup
|
||||
|
||||
Standard install (already done after US-016). Firewall:
|
||||
|
||||
```bash
|
||||
# If using ufw
|
||||
sudo ufw allow 8080/tcp # tracker
|
||||
sudo ufw allow 8001/tcp # node
|
||||
sudo ufw allow 8000/tcp # gateway (optional)
|
||||
```
|
||||
|
||||
## Model split
|
||||
|
||||
For the test, use a model that has enough layers to split meaningfully but fits comfortably in memory. Phi-3-medium-128k-instruct (40 layers, BF16 15GB) works on a single 24GB GPU on each machine:
|
||||
|
||||
- Linux node: layers 0–19 (tracker-mode, owns tokenizer + embed_tokens)
|
||||
- Windows/WSL2 node: layers 20–39
|
||||
|
||||
Start sequence:
|
||||
```bash
|
||||
# Terminal 1 (Linux) — tracker
|
||||
meshnet-tracker --port 8080
|
||||
|
||||
# Terminal 2 (Linux) — first-shard node (tracker-mode auto-detected because shard_start=0)
|
||||
meshnet-node --model microsoft/Phi-3-medium-128k-instruct \
|
||||
--quantization bf16 \
|
||||
--shard-start 0 --shard-end 19 \
|
||||
--tracker http://localhost:8080 \
|
||||
--port 8001
|
||||
|
||||
# Terminal 3 (Windows WSL2) — second-shard node
|
||||
meshnet-node --model microsoft/Phi-3-medium-128k-instruct \
|
||||
--quantization bf16 \
|
||||
--shard-start 20 --shard-end 39 \
|
||||
--tracker http://192.168.1.10:8080 \
|
||||
--port 8001
|
||||
```
|
||||
|
||||
## Test script
|
||||
|
||||
`scripts/test_lan_inference.py`:
|
||||
|
||||
```python
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
End-to-end LAN inference test.
|
||||
Usage: python scripts/test_lan_inference.py --tracker http://192.168.1.10:8080
|
||||
"""
|
||||
import argparse, time, httpx, json
|
||||
|
||||
MESSAGES = [
|
||||
{"role": "user", "content": "What is 7 × 8? Answer in one word."},
|
||||
{"role": "user", "content": "Name the capital of France in one word."},
|
||||
{"role": "user", "content": "Complete the sequence: 1, 1, 2, 3, 5, ___"},
|
||||
]
|
||||
|
||||
def run_test(tracker_url: str, gateway_url: str | None):
|
||||
# Discover inference entry point via tracker if gateway not given
|
||||
if not gateway_url:
|
||||
r = httpx.get(f"{tracker_url}/v1/tracker-nodes/phi-3-medium", timeout=5)
|
||||
r.raise_for_status()
|
||||
nodes = r.json()
|
||||
assert nodes, "No tracker-mode nodes registered — is the first-shard node running?"
|
||||
gateway_url = nodes[0]["url"]
|
||||
|
||||
print(f"Inference endpoint: {gateway_url}")
|
||||
print(f"Tracker: {tracker_url}")
|
||||
print()
|
||||
|
||||
for i, msg in enumerate(MESSAGES):
|
||||
t0 = time.monotonic()
|
||||
r = httpx.post(
|
||||
f"{gateway_url}/v1/chat/completions",
|
||||
json={"model": "phi-3-medium", "messages": [msg], "stream": False},
|
||||
timeout=60,
|
||||
)
|
||||
r.raise_for_status()
|
||||
data = r.json()
|
||||
elapsed = time.monotonic() - t0
|
||||
|
||||
content = data["choices"][0]["message"]["content"]
|
||||
tokens = data["usage"]["completion_tokens"]
|
||||
tps = tokens / elapsed if elapsed > 0 else 0
|
||||
|
||||
print(f"[{i+1}] Q: {msg['content']}")
|
||||
print(f" A: {content}")
|
||||
print(f" {tokens} tokens {elapsed:.2f}s {tps:.1f} t/s")
|
||||
print()
|
||||
|
||||
print("✓ All 3 requests completed successfully")
|
||||
|
||||
if __name__ == "__main__":
|
||||
p = argparse.ArgumentParser()
|
||||
p.add_argument("--tracker", required=True)
|
||||
p.add_argument("--gateway", default=None)
|
||||
args = p.parse_args()
|
||||
run_test(args.tracker, args.gateway)
|
||||
```
|
||||
|
||||
## Docs: TWO_MACHINE_TEST.md
|
||||
|
||||
`docs/TWO_MACHINE_TEST.md` must cover:
|
||||
|
||||
1. Prerequisites (models downloaded on both machines, same model ID, complementary shard ranges)
|
||||
2. Start order: tracker first, then nodes, then test script
|
||||
3. How to verify nodes are registered: `GET /v1/nodes` on tracker
|
||||
4. How to verify coverage: `GET /v1/coverage/phi-3-medium` — all 40 layers must show node_count ≥ 1
|
||||
5. How to run the test script
|
||||
6. Expected output
|
||||
7. Latency breakdown: how to read per-hop latency from node logs
|
||||
8. **Known Issues** section — updated during actual test run with real gotchas
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- `docs/INSTALL_WINDOWS.md` covers WSL2 + CUDA + meshnet-node install end-to-end
|
||||
- `docs/TWO_MACHINE_TEST.md` covers the full two-machine setup and test procedure
|
||||
- `scripts/test_lan_inference.py` exists and is executable
|
||||
- When run against a real two-machine LAN setup: script exits 0, prints 3 valid answers with timing
|
||||
- Coverage map shows 100% coverage (no gap) after both nodes register
|
||||
- Known Issues section in TWO_MACHINE_TEST.md contains at least the issues encountered during this test run
|
||||
- No new pytest failures from repo root (this story adds docs + a script, not new Python packages)
|
||||
- Commit only this story's changes
|
||||
Reference in New Issue
Block a user