popov/neuron-tai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge branch 'master' of https://git.d-popov.com/popov/neuron-tai

Browse Source
This commit is contained in:
Dobromir Popov
2026-06-30 19:28:43 +02:00
parent f1e4870124 7866723c82
commit d0307fcc84
8 changed files with 590 additions and 72 deletions
Download Patch File Download Diff File Expand all files Collapse all files

156
QUICKSTART.md
View File

@@ -48,13 +48,10 @@ python3 -m venv .venv
If `.venv/bin/meshnet-node` is missing, the editable install step did not finish
successfully. Re-run the `.venv/bin/pip install -e ...` command above inside WSL.
WSL2 is still useful for local development, but do not rely on it for the
"another machine connects back to this node" LAN case. WSL2 commonly sits behind
Windows NAT/port-proxy behavior and may not accept inbound traffic from other LAN
machines without extra host networking setup. We intentionally leave that unfixed
because it is useful for testing NAT/relay scenarios. If you just want to bring up
a Windows node that other machines can reach directly, run the node in native
Windows PowerShell instead.
WSL2 sits behind Windows NAT and is **not directly reachable** from other LAN machines.
Direct cross-host hops time out. The relay path (see below) solves this: the WSL2 node
opens an outbound WebSocket to the relay server and all traffic flows through that tunnel.
No firewall rules, no `--advertise-host` needed — just point at the public tracker URL.
### Native Windows PowerShell node (not WSL)
@@ -126,16 +123,21 @@ $env:HF_HOME = "D:\DEV\models"
--port 8005
```
One-line variants:
One-line variants (direct LAN — node must be reachable by IP from other machines):
```powershell
.\.venv\Scripts\meshnet-node.exe start --tracker http://192.168.0.179:8081 --model Qwen/Qwen2.5-0.5B-Instruct --advertise-host 192.168.0.20
.\.venv\Scripts\meshnet-node.exe start --tracker http://ai.neuron.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct --advertise-host 192.168.0.20
```
Via public hostname with relay (works from behind NAT, WSL2, 5G — no `--advertise-host` needed):
```powershell
.\.venv\Scripts\meshnet-node.exe start --tracker https://ai.neuron.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct
```
`--host 0.0.0.0` binds the node to all Windows interfaces. `--advertise-host`
is what the tracker gives to other nodes, so it must be the Windows LAN IP that
the tracker and peer nodes can actually reach.
is what the tracker gives to other nodes for direct connections; omit it when using
the relay path since all traffic flows through the relay tunnel instead.
If you want verbose per-hop pipeline logs while debugging a split model, add
`--debug`. Leave it off for normal runs; otherwise every generated token logs
@@ -144,6 +146,7 @@ lines like:
```text
[node] pipeline hop 0: http://127.0.0.1:8005 start_layer=22
[node] pipeline hop 0 returned text=' token'
[node] pipeline hop 1: wss://ai.neuron.d-popov.com/rpc/abc123 relay start_layer=12
```
6. From the tracker machine, verify Windows is reachable:
@@ -152,32 +155,47 @@ lines like:
curl http://192.168.0.42:8005/v1/health
```
If that endpoint returns 404, that is okay: it still proves the TCP connection
reached the node process. If it times out or connection-refuses, check the
Windows Firewall rule, `--host 0.0.0.0`, the selected LAN IP, and that the node is
still running.
If that endpoint returns 404 or 501, that is okay: it still proves the TCP
connection reached the node process. If it times out or connection-refuses, check
the Windows Firewall rule, `--host 0.0.0.0`, the selected LAN IP, and that the
node is still running.
### Public tracker + WSS relay
---
For internet nodes, expose one public HTTPS host and proxy these paths:
## Public tracker + relay (internet / NAT nodes)
```text
/v1/* -> meshnet-tracker, for registration, heartbeats, routing, and OpenAI requests
/ws -> meshnet-relay, for outbound node gossip/bridge connections
/rpc/* -> meshnet-relay, for tracker-to-node relay requests
This setup lets nodes connect from anywhere — behind home NAT, 5G, WSL2, or
on a different continent — without opening firewall ports.
### Architecture
```
Client → HTTPS → ai.neuron.d-popov.com (nginx)
├─ /v1/* → meshnet-tracker :8081
├─ /ws → meshnet-relay :8765 (node persistent outbound WS)
└─ /rpc/* → meshnet-relay :8765 (caller opens WS per hop)
```
Start the tracker with the public relay URL it should advertise:
### Start the relay and tracker (server side)
```bash
# Terminal 1 — relay (WebSocket hub)
.venv/bin/meshnet-relay --host 0.0.0.0 --port 8765
# Terminal 2 — tracker (advertises relay URL to nodes)
.venv/bin/meshnet-tracker start \
--host 0.0.0.0 \
--port 8081 \
--relay-url wss://ai.neuron.d-popov.com/ws
```
Then a node only needs the public tracker address:
The `--relay-url` flag embeds the relay address in `/v1/network/map`. Every node
queries that endpoint on startup and auto-connects if a relay URL is present.
### Start a node (any machine, any network)
No `--advertise-host` needed. The node discovers the relay URL from the tracker
and opens a persistent outbound WebSocket:
```bash
.venv/bin/meshnet-node start \
@@ -201,6 +219,84 @@ curl -s "https://ai.neuron.d-popov.com/v1/network/map" | python3 -m json.tool
curl -s "https://ai.neuron.d-popov.com/v1/route?model=qwen2.5-0.5b" | python3 -m json.tool
```
Expected startup output (relay path):
```
Auto-detected 24 layers → shard 0–23
Relay connected — wss://ai.neuron.d-popov.com/rpc/abc1def2ef3f4567
================================
meshnet-node ready
Wallet: <address>
Model ID: Qwen/Qwen2.5-0.5B-Instruct
Shard: layers 0–23; 24 of 24
Quantization: bfloat16
Endpoint: http://172.29.104.23:7001
Node ID: <id>
Hardware: CPU
================================
```
The `Endpoint` shown is the local IP (unreachable from outside). Other nodes reach
this one via `wss://ai.neuron.d-popov.com/rpc/<peer_id>` instead.
### How relay hops work
When node A needs to forward activations to node B (behind NAT):
1. Tracker injects `X-Meshnet-Route` with `relay_addr` for each behind-NAT hop.
2. Node A opens a WebSocket to `wss://relay/rpc/{peer_id_B}`.
3. Relay forwards the `relay-http-request` envelope to Node B's persistent connection.
4. Node B processes `/forward` locally, returns `relay-http-response`.
5. Relay sends the response back to Node A over the same WebSocket.
6. Node A closes the WebSocket and continues the pipeline.
Binary activation tensors (bfloat16) are Base64-encoded through the relay JSON
protocol and decoded on both sides — no precision loss.
If the relay hop fails (relay down, peer disconnected), the node logs a warning and
falls back to a direct HTTP attempt before returning an error.
### Test from WSL2 using the public tracker
In WSL2 (which gets a `172.x.x.x` virtual IP — unreachable from other machines):
```bash
# WSL2 Terminal 1 — head node (layers 0–11, handles chat requests)
.venv/bin/meshnet-node start \
--tracker https://ai.neuron.d-popov.com \
--model Qwen/Qwen2.5-0.5B-Instruct \
--shard-start 0 --shard-end 11
# WSL2 Terminal 2 — tail node (layers 12–23)
.venv/bin/meshnet-node start \
--tracker https://ai.neuron.d-popov.com \
--model Qwen/Qwen2.5-0.5B-Instruct \
--shard-start 12 --shard-end 23
```
Both nodes connect to the relay automatically. When a chat request arrives at Node A,
it forwards activations to Node B via `wss://ai.neuron.d-popov.com/rpc/{peer_id_B}`.
Send inference through the tracker (which picks the head node and injects the route):
```bash
curl -s https://ai.neuron.d-popov.com/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen2.5-0.5B-Instruct",
"messages": [{"role": "user", "content": "What is 7 times 8?"}],
"stream": false
}' | python3 -m json.tool
```
Or send directly to Node A's local port (within WSL):
```bash
curl -s http://localhost:7001/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model": "Qwen/Qwen2.5-0.5B-Instruct", "messages": [{"role": "user", "content": "Hi"}]}'
```
---
## Step 1 — Start the tracker (Terminal 1)
@@ -338,7 +434,7 @@ HF_HOME=/run/media/popov/d/DEV/models \
--port 8002
```
Send the request to Node A — it tokenizes, runs layers 0–13, passes binary
Send the request to Node A — it tokenizes, runs layers 0–11, passes binary
activations to Node B, and streams the final response back.
---
@@ -382,18 +478,6 @@ tracker with a relay URL so the node registers a `relay_addr`.
---
## Start the relay node (for NAT traversal)
```bash
.venv/bin/pip install -e packages/relay
.venv/bin/meshnet-relay --port 8765
```
Nodes behind NAT connect to the relay and advertise their relay address to the
tracker. See `docs/adr/0010-p2p-gossip-and-nat-relay.md`.
---
## Run all tests
```bash