neuron-tai/docs/issues/18-two-machine-lan-test.md

US-018 — End-to-end two-machine LAN inference test

Goal

Run real distributed inference across two physical machines: the Linux rig and a Windows 11 rig running WSL2. Document every setup step, firewall rule, and gotcha so this is repeatable. The test script exits 0 with token output and timing, proving the network works.

Network topology for LAN test

[Linux machine]                           [Windows 11 / WSL2]
  meshnet-tracker  :8080                    meshnet-node (shard B)
  meshnet-node     :8001  (shard A, tracker-mode)
  meshnet-gateway  :8000  (optional, for OpenAI-compat)

Client (either machine):
  scripts/test_lan_inference.py --tracker http://192.168.1.10:8080

The Linux machine runs the tracker + the first-shard node (tracker-mode). The Windows/WSL2 machine runs the second-shard node. A small model (e.g. Phi-3-medium at BF16, fits on one GPU each) is split across both.

WSL2 setup (Windows side)

docs/INSTALL_WINDOWS.md covers:

1. Enable WSL2: wsl --install -d Ubuntu-24.04
2. CUDA in WSL2: install NVIDIA driver on Windows (NOT inside WSL); WSL2 gets CUDA automatically
   • Verify: nvidia-smi inside WSL2 should show GPU
3. Install Python 3.11+ and pip inside WSL2
4. pip install -e packages/node packages/p2p (clone repo first)
5. Firewall: Windows Defender must allow inbound WSL2 → LAN on node port
   • PowerShell: New-NetFirewallRule -DisplayName "meshnet-node" -Direction Inbound -Protocol TCP -LocalPort 8001 -Action Allow
6. WSL2 IP: WSL2 has its own NAT'd IP (172.x.x.x); to expose to LAN, either:
   • Option A: netsh interface portproxy add v4tov4 listenport=8001 listenaddress=0.0.0.0 connectport=8001 connectaddress=$(wsl hostname -I)
   • Option B: use the relay node (US-017) — no port forwarding needed

Linux setup

Standard install (already done after US-016). Firewall:

# If using ufw
sudo ufw allow 8080/tcp  # tracker
sudo ufw allow 8001/tcp  # node
sudo ufw allow 8000/tcp  # gateway (optional)

Model split

For the test, use a model that has enough layers to split meaningfully but fits comfortably in memory. Phi-3-medium-128k-instruct (40 layers, BF16 15GB) works on a single 24GB GPU on each machine:

• Linux node: layers 0–19 (tracker-mode, owns tokenizer + embed_tokens)
• Windows/WSL2 node: layers 20–39

Start sequence:

# Terminal 1 (Linux) — tracker
meshnet-tracker --port 8080

# Terminal 2 (Linux) — first-shard node (tracker-mode auto-detected because shard_start=0)
meshnet-node --model microsoft/Phi-3-medium-128k-instruct \
             --quantization bf16 \
             --shard-start 0 --shard-end 19 \
             --tracker http://localhost:8080 \
             --port 8001

# Terminal 3 (Windows WSL2) — second-shard node
meshnet-node --model microsoft/Phi-3-medium-128k-instruct \
             --quantization bf16 \
             --shard-start 20 --shard-end 39 \
             --tracker http://192.168.1.10:8080 \
             --port 8001

Test script

scripts/test_lan_inference.py:

#!/usr/bin/env python3
"""
End-to-end LAN inference test.
Usage: python scripts/test_lan_inference.py --tracker http://192.168.1.10:8080
"""
import argparse, time, httpx, json

MESSAGES = [
    {"role": "user", "content": "What is 7 × 8? Answer in one word."},
    {"role": "user", "content": "Name the capital of France in one word."},
    {"role": "user", "content": "Complete the sequence: 1, 1, 2, 3, 5, ___"},
]

def run_test(tracker_url: str, gateway_url: str | None):
    # Discover inference entry point via tracker if gateway not given
    if not gateway_url:
        r = httpx.get(f"{tracker_url}/v1/tracker-nodes/phi-3-medium", timeout=5)
        r.raise_for_status()
        nodes = r.json()
        assert nodes, "No tracker-mode nodes registered — is the first-shard node running?"
        gateway_url = nodes[0]["url"]

    print(f"Inference endpoint: {gateway_url}")
    print(f"Tracker: {tracker_url}")
    print()

    for i, msg in enumerate(MESSAGES):
        t0 = time.monotonic()
        r = httpx.post(
            f"{gateway_url}/v1/chat/completions",
            json={"model": "phi-3-medium", "messages": [msg], "stream": False},
            timeout=60,
        )
        r.raise_for_status()
        data = r.json()
        elapsed = time.monotonic() - t0

        content = data["choices"][0]["message"]["content"]
        tokens = data["usage"]["completion_tokens"]
        tps = tokens / elapsed if elapsed > 0 else 0

        print(f"[{i+1}] Q: {msg['content']}")
        print(f"     A: {content}")
        print(f"     {tokens} tokens  {elapsed:.2f}s  {tps:.1f} t/s")
        print()

    print("✓ All 3 requests completed successfully")

if __name__ == "__main__":
    p = argparse.ArgumentParser()
    p.add_argument("--tracker", required=True)
    p.add_argument("--gateway", default=None)
    args = p.parse_args()
    run_test(args.tracker, args.gateway)

Docs: TWO_MACHINE_TEST.md

docs/TWO_MACHINE_TEST.md must cover:

1. Prerequisites (models downloaded on both machines, same model ID, complementary shard ranges)
2. Start order: tracker first, then nodes, then test script
3. How to verify nodes are registered: GET /v1/nodes on tracker
4. How to verify coverage: GET /v1/coverage/phi-3-medium — all 40 layers must show node_count ≥ 1
5. How to run the test script
6. Expected output
7. Latency breakdown: how to read per-hop latency from node logs
8. Known Issues section — updated during actual test run with real gotchas

Acceptance criteria

• docs/INSTALL_WINDOWS.md covers WSL2 + CUDA + meshnet-node install end-to-end
• docs/TWO_MACHINE_TEST.md covers the full two-machine setup and test procedure
• scripts/test_lan_inference.py exists and is executable
• When run against a real two-machine LAN setup: script exits 0, prints 3 valid answers with timing
• Coverage map shows 100% coverage (no gap) after both nodes register
• Known Issues section in TWO_MACHINE_TEST.md contains at least the issues encountered during this test run
• No new pytest failures from repo root (this story adds docs + a script, not new Python packages)
• Commit only this story's changes