neuron-tai/docs/TWO_MACHINE_TEST.md

Two-machine LAN inference test

This guide proves that distributed inference works across two physical machines: a Linux rig (tracker + first shard) and a Windows 11 / WSL2 rig (second shard). A test script sends real inference requests and validates the output.

Network topology

[Linux machine — 192.168.1.10]
  meshnet-tracker  :8080
  meshnet-node A   :8001   shard 0–19  (tracker-mode, entry point)

[Windows 11 / WSL2 — 192.168.1.20]
  meshnet-node B   :8001   shard 20–39

[Client — either machine]
  scripts/test_lan_inference.py --tracker http://192.168.1.10:8080

Adjust the IPs and shard ranges to match your hardware. Use a model that fits (sharded) in both GPUs combined. The example uses microsoft/Phi-3-medium-128k-instruct (40 layers, BF16 ~15 GB each shard ~7.5 GB).

---

Prerequisites

Both machines:

• Python 3.11+ with meshnet-node installed (see docs/INSTALL_WINDOWS.md for Windows)
• Model weights already downloaded (pre-fetch prevents timeout on first startup)
• LAN connectivity verified: ping 192.168.1.10 from Windows, ping 192.168.1.20 from Linux

Linux machine ports open:

# ufw (skip if firewall is off)
sudo ufw allow 8080/tcp   # tracker
sudo ufw allow 8001/tcp   # node A

Windows machine port forwarded (WSL2 only):

# Run in PowerShell as Administrator — redo after every WSL restart
$wsl = (wsl hostname -I).Trim()
netsh interface portproxy add v4tov4 listenport=8001 listenaddress=0.0.0.0 connectport=8001 connectaddress=$wsl
New-NetFirewallRule -DisplayName "meshnet-node" -Direction Inbound -Protocol TCP -LocalPort 8001 -Action Allow

---

Start sequence

Always start in this order: tracker → node A → node B → test.

Terminal 1 — Linux: tracker

meshnet-tracker --port 8080

Expected:

[tracker] listening on 0.0.0.0:8080

Terminal 2 — Linux: node A (shard 0–19, tracker-mode)

meshnet-node \
  --model microsoft/Phi-3-medium-128k-instruct \
  --quantization bf16 \
  --shard-start 0 --shard-end 19 \
  --tracker http://localhost:8080 \
  --port 8001 \
  --host 0.0.0.0

shard_start=0 auto-sets tracker_mode=True — this node accepts inference requests.

Wait until you see meshnet-node ready before continuing.

Terminal 3 — Windows WSL2: node B (shard 20–39)

meshnet-node \
  --model microsoft/Phi-3-medium-128k-instruct \
  --quantization bf16 \
  --shard-start 20 --shard-end 39 \
  --tracker http://192.168.1.10:8080 \
  --port 8001 \
  --host 0.0.0.0 \
  --advertise-host 192.168.1.20

--advertise-host must be the Windows LAN IP (not the WSL2 internal 172.x.x.x IP) so the Linux node can reach it.

---

Verify nodes are registered

From any machine with curl:

# List all registered nodes
curl http://192.168.1.10:8080/v1/nodes

# Check route for the model — should list both node endpoints in order
curl "http://192.168.1.10:8080/v1/route?model=microsoft/Phi-3-medium-128k-instruct"

Expected route response:

{
  "route": [
    "http://192.168.1.10:8001",
    "http://192.168.1.20:8001"
  ]
}

If only one endpoint appears, node B hasn't registered yet — wait a few seconds and retry.

---

Run the test script

# From any machine that can reach the tracker
python3 scripts/test_lan_inference.py \
  --tracker http://192.168.1.10:8080 \
  --gateway http://192.168.1.10:8001

Expected output:

Inference endpoint: http://192.168.1.10:8001
Tracker: http://192.168.1.10:8080

Route: ['http://192.168.1.10:8001', 'http://192.168.1.20:8001']

[1] Q: What is 7 × 8? Answer in one word.
     A: 56
     3 tokens  2.41s  1.2 t/s

[2] Q: Name the capital of France in one word.
     A: Paris
     2 tokens  1.87s  1.1 t/s

[3] Q: Complete the sequence: 1, 1, 2, 3, 5, ___
     A: 8
     2 tokens  1.93s  1.0 t/s

All 3 requests completed successfully.
Exit code: 0

The script exits 0 if all 3 requests complete with valid OpenAI-format responses.

---

Reading latency from node logs

The node logs show per-hop timing. On node A terminal look for:

[node] forwarding to downstream: http://192.168.1.20:8001  (took 1.23s)

Approximate breakdown:

• client → node A (encode + first shard): full request latency minus the downstream time
• node A → node B (pipeline): the forwarding to downstream duration
• node B → node A (tail decode + token): included in downstream duration

Full end-to-end latency = prompt encode + shard A forward + network transfer + shard B forward + decode.

With LAN latency < 1 ms, the network transfer is negligible. Bottleneck is GPU compute.

---

Known Issues

WSL2 IP changes after restart. The netsh portproxy forwarding rule uses a fixed WSL2 IP. If Windows or WSL2 restarts, the IP changes and the rule breaks. Redo the netsh and New-NetFirewallRule commands. To automate this, add a Task Scheduler job on WSL start.

Node B registers with internal WSL2 IP (172.x.x.x) instead of LAN IP. Symptom: route response lists 172.x.x.x and node A cannot reach it. Fix: always pass --advertise-host 192.168.1.20 (your Windows LAN IP) when starting node B.

Model download times out node registration. If the model hasn't been pre-fetched, transformers downloads it during node startup, which can take 20+ minutes. The tracker heartbeat timeout (90s) will expire, and node A will deregister node B. Pre-download the model weights before starting the node (see docs/INSTALL_WINDOWS.md Step 6). Node B re-registers automatically via the heartbeat re-registration loop once it's up.

bf16 unsupported on older NVIDIA GPUs. GPUs before Ampere (RTX 30xx) have limited bfloat16 support. Use --quantization int8 on RTX 20xx and earlier.

Windows Defender blocks inbound connection on WSL2. Even with the firewall rule added, Windows Defender SmartScreen or a corporate security policy can block the connection. Verify by checking Windows Event Viewer → Security → Filtering Platform Connection for blocked connections on port 8001.

Route returns only one node. If node B registers but the route only returns one endpoint, check that both nodes use the same --model string (full HuggingFace repo path). Route lookup matches on hf_repo — a short name vs. full path mismatch causes the node to be excluded.