neuron-tai/docs/TWO_MACHINE_TEST.md

# Two-machine LAN inference test

This guide proves that distributed inference works across two physical machines: a Linux rig (tracker + first shard) and a Windows 11 / WSL2 rig (second shard). A test script sends real inference requests and validates the output.

## Network topology

```
[Linux machine — 192.168.1.10]
  meshnet-tracker  :8080
  meshnet-node A   :8001   shard 0–19  (tracker-mode, entry point)

[Windows 11 / WSL2 — 192.168.1.20]
  meshnet-node B   :8001   shard 20–39

[Client — either machine]
  scripts/test_lan_inference.py --tracker http://192.168.1.10:8080
```

Adjust the IPs and shard ranges to match your hardware. Use a model that fits (sharded) in both GPUs combined. The example uses `microsoft/Phi-3-medium-128k-instruct` (40 layers, BF16 ~15 GB each shard ~7.5 GB).

---

## Prerequisites

**Both machines:**
- Python 3.11+ with `meshnet-node` installed (see `docs/INSTALL_WINDOWS.md` for Windows)
- Model weights already downloaded (pre-fetch prevents timeout on first startup)
- LAN connectivity verified: `ping 192.168.1.10` from Windows, `ping 192.168.1.20` from Linux

**Linux machine ports open:**

```bash
# ufw (skip if firewall is off)
sudo ufw allow 8080/tcp   # tracker
sudo ufw allow 8001/tcp   # node A
```

**Windows machine port forwarded (WSL2 only):**

```powershell
# Run in PowerShell as Administrator — redo after every WSL restart
$wsl = (wsl hostname -I).Trim()
netsh interface portproxy add v4tov4 listenport=8001 listenaddress=0.0.0.0 connectport=8001 connectaddress=$wsl
New-NetFirewallRule -DisplayName "meshnet-node" -Direction Inbound -Protocol TCP -LocalPort 8001 -Action Allow
```

---

## Start sequence

**Always start in this order: tracker → node A → node B → test.**

### Terminal 1 — Linux: tracker

```bash
meshnet-tracker --port 8080
```

Expected:

```
[tracker] listening on 0.0.0.0:8080
```

### Terminal 2 — Linux: node A (shard 0–19, tracker-mode)

```bash
meshnet-node \
  --model microsoft/Phi-3-medium-128k-instruct \
  --quantization bf16 \
  --shard-start 0 --shard-end 19 \
  --tracker http://localhost:8080 \
  --port 8001 \
  --host 0.0.0.0
```

`shard_start=0` auto-sets `tracker_mode=True` — this node accepts inference requests.

Wait until you see `meshnet-node ready` before continuing.

### Terminal 3 — Windows WSL2: node B (shard 20–39)

```bash
meshnet-node \
  --model microsoft/Phi-3-medium-128k-instruct \
  --quantization bf16 \
  --shard-start 20 --shard-end 39 \
  --tracker http://192.168.1.10:8080 \
  --port 8001 \
  --host 0.0.0.0 \
  --advertise-host 192.168.1.20
```

`--advertise-host` must be the Windows **LAN IP** (not the WSL2 internal 172.x.x.x IP) so the Linux node can reach it.

---

## Verify nodes are registered

From any machine with `curl`:

```bash
# List all registered nodes
curl http://192.168.1.10:8080/v1/nodes

# Check route for the model — should list both node endpoints in order
curl "http://192.168.1.10:8080/v1/route?model=microsoft/Phi-3-medium-128k-instruct"
```

Expected route response:

```json
{
  "route": [
    "http://192.168.1.10:8001",
    "http://192.168.1.20:8001"
  ]
}
```

If only one endpoint appears, node B hasn't registered yet — wait a few seconds and retry.

---

## Run the test script

```bash
# From any machine that can reach the tracker
python3 scripts/test_lan_inference.py \
  --tracker http://192.168.1.10:8080 \
  --gateway http://192.168.1.10:8001
```

Expected output:

```
Inference endpoint: http://192.168.1.10:8001
Tracker: http://192.168.1.10:8080

Route: ['http://192.168.1.10:8001', 'http://192.168.1.20:8001']

[1] Q: What is 7 × 8? Answer in one word.
     A: 56
     3 tokens  2.41s  1.2 t/s

[2] Q: Name the capital of France in one word.
     A: Paris
     2 tokens  1.87s  1.1 t/s

[3] Q: Complete the sequence: 1, 1, 2, 3, 5, ___
     A: 8
     2 tokens  1.93s  1.0 t/s

All 3 requests completed successfully.
Exit code: 0
```

The script exits 0 if all 3 requests complete with valid OpenAI-format responses.

---

## Reading latency from node logs

The node logs show per-hop timing. On node A terminal look for:

```
[node] forwarding to downstream: http://192.168.1.20:8001  (took 1.23s)
```

Approximate breakdown:
- **client → node A (encode + first shard):** full request latency minus the downstream time
- **node A → node B (pipeline):** the `forwarding to downstream` duration
- **node B → node A (tail decode + token):** included in downstream duration

Full end-to-end latency = prompt encode + shard A forward + network transfer + shard B forward + decode.

With LAN latency < 1 ms, the network transfer is negligible. Bottleneck is GPU compute.

---

## Known Issues

**WSL2 IP changes after restart.**
The `netsh portproxy` forwarding rule uses a fixed WSL2 IP. If Windows or WSL2 restarts, the IP changes and the rule breaks. Redo the `netsh` and `New-NetFirewallRule` commands. To automate this, add a Task Scheduler job on WSL start.

**Node B registers with internal WSL2 IP (172.x.x.x) instead of LAN IP.**
Symptom: route response lists `172.x.x.x` and node A cannot reach it.
Fix: always pass `--advertise-host 192.168.1.20` (your Windows LAN IP) when starting node B.

**Model download times out node registration.**
If the model hasn't been pre-fetched, `transformers` downloads it during node startup, which can take 20+ minutes. The tracker heartbeat timeout (90s) will expire, and node A will deregister node B. Pre-download the model weights before starting the node (see `docs/INSTALL_WINDOWS.md` Step 6). Node B re-registers automatically via the heartbeat re-registration loop once it's up.

**`bf16` unsupported on older NVIDIA GPUs.**
GPUs before Ampere (RTX 30xx) have limited bfloat16 support. Use `--quantization int8` on RTX 20xx and earlier.

**Windows Defender blocks inbound connection on WSL2.**
Even with the firewall rule added, Windows Defender SmartScreen or a corporate security policy can block the connection. Verify by checking Windows Event Viewer → Security → Filtering Platform Connection for blocked connections on port 8001.

**Route returns only one node.**
If node B registers but the route only returns one endpoint, check that both nodes use the same `--model` string (full HuggingFace repo path). Route lookup matches on `hf_repo` — a short name vs. full path mismatch causes the node to be excluded.