popov/neuron-tai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

inference working

Browse Source
This commit is contained in:
Dobromir Popov
2026-06-29 23:54:35 +03:00
parent 607d49f5b0
commit 1bdfce657d
16 changed files with 1899 additions and 73 deletions
Download Patch File Download Diff File Expand all files Collapse all files

206
.scratch/distributed-inference-network/issues/16-mining-cli-ux.md Normal file
View File

@@ -0,0 +1,206 @@
# US-016 — Mining-style node startup CLI + live dashboard
## Goal
Replace the bare flag-driven `meshnet-node start` with a wizard-guided first-run experience modelled on GPU mining clients (like PhoenixMiner, lolMiner, etc.). After the wizard, the terminal switches to a live status dashboard showing real-time node health and earnings.
## Wizard flow (first run only)
```
╔══════════════════════════════════════════════════════════╗
║ meshnet-node v0.1.0 ║
║ Distributed AI Inference — Node Setup ║
╚══════════════════════════════════════════════════════════╝
Detecting hardware...
GPU 0: NVIDIA RTX 4090 24 GB VRAM ✓
GPU 1: NVIDIA RTX 3090 24 GB VRAM ✓
Select a model to serve:
# Model Layers NF4 INT8 BF16
1 Llama-3-70B-Instruct 80 ✓18GB ✓40GB ✗80GB
2 Qwen-2.5-72B-Instruct 80 ✓19GB ✗41GB ✗81GB
3 Mixtral-8x7B-Instruct-v0.1 32 ✓ 7GB ✓14GB ✓27GB
4 Phi-3-medium-128k-instruct 40 ✓ 4GB ✓ 8GB ✓15GB
5 [Browse HuggingFace…]
Enter number [1]: _
Quantization [nf4/int8/bf16] (nf4 recommended for 24GB): _
Download directory [~/.meshnet/models]: _
Tracker URL [http://localhost:8080]: _
Wallet path [~/.config/meshnet/wallet.json] (new wallet will be created): _
Config saved to ~/.config/meshnet/config.json
Starting node…
```
Second run with existing config:
```
meshnet-node
Reading config from ~/.config/meshnet/config.json
Model: Llama-3-70B-Instruct Quant: nf4 Shard: layers 0–15
Tracker: http://192.168.1.10:8080
Starting…
```
## Live dashboard (once running)
Renders every 2 seconds using `rich.live`. Fallback: plain-text status line if `rich` is unavailable or terminal is not a TTY (important for WSL2 / SSH).
```
meshnet-node Llama-3-70B-Instruct [nf4] shard 0–15/80 up 00:03:22
GPU 0 RTX 4090 GPU ████████░░ 73% VRAM 18.2/24.0 GB 45°C
GPU 1 RTX 3090 GPU ███░░░░░░░ 28% VRAM 8.7/24.0 GB 38°C
Tokens/sec ▁▂▃▄▅▆▇█ 42.3 t/s (EMA 30s)
Requests 1,247 served 3 active
Peers 8 connected (tracker: ✓ relay: ✓)
TAI earned 0.00 TAI (payments active after US-006)
Uptime 00:03:22
[q] quit [r] reset stats [c] compact view
```
Compact mode (`--compact` or pressing `c`) shows a single status line:
```
[43t/s VRAM18.2GB req1247 peers8 up3m22s]
```
## Implementation notes
### Hardware detection
```python
import torch
def detect_gpus() -> list[dict]:
gpus = []
if torch.cuda.is_available():
for i in range(torch.cuda.device_count()):
props = torch.cuda.get_device_properties(i)
gpus.append({
"index": i,
"name": props.name,
"vram_gb": props.total_memory / 1e9,
"backend": "cuda"
})
# ROCm / Apple Silicon stubs for later
return gpus
```
### Curated model list
`packages/node/meshnet_node/model_catalog.py` — a hardcoded list of `ModelPreset` dataclasses:
```python
@dataclass
class ModelPreset:
name: str # display name
hf_repo: str # HuggingFace repo ID
num_layers: int
vram_gb: dict # {"nf4": 18, "int8": 40, "bf16": 80}
description: str # one-line description
```
Initial list (expand over time):
- `meta-llama/Meta-Llama-3-70B-Instruct` — 80L, NF4 18GB, INT8 40GB, BF16 80GB
- `Qwen/Qwen2.5-72B-Instruct` — 80L, NF4 19GB, INT8 41GB, BF16 81GB
- `mistralai/Mixtral-8x7B-Instruct-v0.1` — 32L, NF4 7GB, INT8 14GB, BF16 27GB
- `microsoft/Phi-3-medium-128k-instruct` — 40L, NF4 4GB, INT8 8GB, BF16 15GB
- `google/gemma-2-27b-it` — 46L, NF4 10GB, INT8 20GB, BF16 40GB
### HuggingFace Browse
```python
from huggingface_hub import list_models
def browse_hf(top_n=20) -> list[dict]:
models = list_models(
pipeline_tag="text-generation",
library="transformers",
sort="downloads",
direction=-1,
limit=top_n,
cardData=True,
)
return [{"repo": m.modelId, "downloads": m.downloads} for m in models]
```
### Persistent config
`~/.config/meshnet/config.json`:
```json
{
"model_hf_repo": "meta-llama/Meta-Llama-3-70B-Instruct",
"quantization": "nf4",
"download_dir": "~/.meshnet/models",
"tracker_url": "http://192.168.1.10:8080",
"wallet_path": "~/.config/meshnet/wallet.json",
"shard_start": null,
"shard_end": null,
"updatedAt": "2026-06-29T..."
}
```
`shard_start`/`shard_end`: null means tracker auto-assigns. User can pin a range for dedicated partial-model nodes.
### CLI flags
All wizard answers are overridable without re-running the wizard:
```
meshnet-node [start]
--model <hf-repo-id> # e.g. meta-llama/Meta-Llama-3-70B-Instruct
--quantization [bf16|int8|nf4]
--download-dir <path>
--tracker <url>
--wallet <path>
--shard-start <int> # pin shard range (optional)
--shard-end <int>
--reset-config # ignore saved config, re-run wizard
--no-tui # plain-text output (for CI / headless)
--compact # single-line status instead of full dashboard
meshnet-node models # list curated models and exit
meshnet-node models --browse # list HF Hub top-20 and exit
meshnet-node config # print current config and exit
```
### WSL2 / non-TTY fallback
```python
import sys, os
def is_interactive_tty() -> bool:
return sys.stdout.isatty() and os.environ.get("TERM") not in ("dumb", "")
if not is_interactive_tty():
# fall back to plain-text periodic status
run_plain_status_loop(node)
else:
run_rich_dashboard(node)
```
Do NOT use `termios`, `fcntl`, or `/dev/tty` — these break in Windows cmd.exe and some WSL2 terminal emulators.
## Acceptance criteria
- `meshnet-node` with no args and no config → wizard starts
- Wizard detects GPU and marks `[too large]` for models that exceed available VRAM
- `meshnet-node models` prints curated list and exits
- `meshnet-node models --browse` calls HF Hub API, prints top-20, exits
- Second run (config exists) → skips wizard, starts immediately
- `--reset-config` re-runs wizard even with config present
- All wizard inputs override-able via CLI flags
- Live rich dashboard renders and updates every 2s when running in a TTY
- Falls back to plain-text when not a TTY (CI / WSL2 without TERM set)
- Ctrl-C prints a clean summary line and exits 0
- `python -m pytest` passes from repo root
- Commit only this story's changes

205
.scratch/distributed-inference-network/issues/17-p2p-gossip-relay-ssl.md Normal file
View File

@@ -0,0 +1,205 @@
# US-017 — P2P gossip, NAT-traversal relay node, and SSL/TLS
## Goal
Nodes must work behind NAT (home routers, cloud VMs without public IPs) and must communicate securely. Implement:
1. **SSL/TLS everywhere** — all HTTP between nodes/tracker is HTTPS; all WebSocket gossip is WSS
2. **mDNS peer discovery** — nodes on the same LAN find each other automatically (no config)
3. **WebSocket gossip PubSub** — nodes propagate join/leave/coverage-update events in near-real-time
4. **Circuit relay node** — team-run public relay (`packages/relay`) that enables NAT traversal and bootstraps new nodes joining from the internet
Architecture is designed to migrate to libp2p GossipSub + Kademlia DHT without breaking the message schema (topic names and payload formats are stable contracts).
## Gossip protocol
### Transport
WebSocket (`wss://`) using the `websockets` Python library. Each node maintains persistent WSS connections to:
- The relay node (always, bootstraps peer list)
- Up to 8 direct peers (Kademlia-style target fanout; peers discovered via mDNS + relay peer list)
### Topics
All messages are JSON with an envelope:
```json
{
"topic": "node-join",
"version": 1,
"from_peer": "<peer_id>",
"timestamp": "<iso8601>",
"payload": { ... }
}
```
| Topic | Direction | Payload |
|-------|-----------|---------|
| `node-join` | broadcast | `{peer_id, addr, models: [{model_preset, shard_start, shard_end}], vram_gb, quant}` |
| `node-leave` | broadcast | `{peer_id, reason}` |
| `coverage-update` | broadcast | `{model_preset, coverage: [{start, end, count}]}` |
| `heartbeat` | peer→relay | `{peer_id, addr, uptime_s, tokens_per_sec}` |
| `peer-list` | relay→peer | `{peers: [{peer_id, addr}]}` |
| `relay-announce` | relay→all | `{relay_id, relay_url, capacity}` |
Gossip fanout: each node re-broadcasts received messages to all its peers (simple flooding with `seen_ids` dedup, TTL=3 hops). Migration to GossipSub mesh routing is a later ADR.
### Peer ID
`peer_id = sha256(public_key)[:16].hex()` — generated on first run, stored in `~/.config/meshnet/identity.json`. The same keypair is used for TLS client certificates (mTLS) in future work.
## mDNS LAN discovery
Use Python `zeroconf` library. Service type: `_meshnet._tcp.local.`
```python
from zeroconf import ServiceInfo, Zeroconf
info = ServiceInfo(
"_meshnet._tcp.local.",
f"{peer_id}._meshnet._tcp.local.",
addresses=[socket.inet_aton(local_ip)],
port=node_port,
properties={"peer_id": peer_id, "version": "1"},
)
zc = Zeroconf()
zc.register_service(info)
```
On startup, nodes also browse for `_meshnet._tcp.local.` to discover existing nodes. mDNS is LAN-only (does not traverse routers), which is correct for LAN discovery.
## NAT traversal: circuit relay
### How it works
1. Node A (behind NAT) cannot accept inbound TCP connections
2. Node A connects outbound to the public relay via WSS
3. Node A tells the tracker: `"effective_addr": "wss://relay.meshnet.ai/relay/{peer_id_A}"`
4. Node B (wants to call A) connects to the relay at the above URL
5. Relay proxies the TCP stream between A and B
Hole-punching (direct connection via STUN) is attempted first (future work). Relay is the fallback.
### meshnet-relay
`packages/relay/meshnet_relay/server.py` — a standalone aiohttp server:
```
GET /health → {status: ok}
GET /v1/peers → [{peer_id, addr, last_seen}]
POST /v1/gossip → receive a gossip message, fan out to connected peers
WSS /ws → persistent gossip connection (subscribe to all topics)
WSS /relay/{peer_id} → circuit relay proxy to that peer_id
GET /v1/relay/capacity → {connected_peers: N, max_peers: 500}
```
CLI:
```
meshnet-relay [--port 8443] [--cert path/to/cert.pem] [--key path/to/key.pem]
[--tracker-url http://...] [--max-peers 500]
```
The relay can optionally proxy to the tracker (so `relay.meshnet.ai` is the single internet-visible endpoint).
## SSL/TLS setup
### Node certificate (self-signed, auto-generated)
On first run, `meshnet-node` generates a self-signed RSA-2048 cert valid for 10 years:
```python
from cryptography import x509
from cryptography.hazmat.primitives import hashes, serialization
from cryptography.hazmat.primitives.asymmetric import rsa
```
Cert saved to `~/.config/meshnet/node_cert.pem` + `node_key.pem`. Fingerprint stored in config and shared with tracker via heartbeat. Nodes connecting to each other validate the fingerprint (TOFU — trust on first use), not the CA chain.
### Relay certificate
The relay uses a real Let's Encrypt cert (cert-bot or acme.sh). The relay cert is pinned in `packages/p2p/relay_bootstrap.json`:
```json
{
"relays": [
{
"url": "wss://relay.meshnet.ai:8443",
"cert_fingerprint": "sha256:<hex>",
"operator": "meshnet-team"
}
]
}
```
### All HTTP switched to HTTPS
`meshnet-node` starts an HTTPS server using `ssl.SSLContext`. `meshnet-tracker` similarly. All outbound `httpx` / `aiohttp` calls use TLS verification against pinned fingerprints (not the system CA store — too many corporate proxies break this).
## Tracker changes
Heartbeat payload gains new fields:
```json
{
"peer_id": "a1b2c3d4e5f6a1b2",
"effective_addr": "https://192.168.1.42:8001",
"relay_addr": "wss://relay.meshnet.ai:8443/relay/a1b2c3d4e5f6a1b2",
"cert_fingerprint": "sha256:...",
"gossip_peers": ["peer_id_1", "peer_id_2"]
}
```
Tracker uses `effective_addr` (direct) or `relay_addr` (fallback) when building inference routes.
## Integration test
```
tests/test_gossip_and_relay.py
scenario:
1. Start a local relay (localhost:18443)
2. Start node A (no inbound port — simulate NAT by binding to 127.0.0.1 only)
3. Start node B (public-reachable on localhost)
4. Both register with relay; relay peer-list includes both
5. Node B sends a gossip node-join message
6. Assert node A receives it within 500ms
7. Start tracker; confirm tracker's node registry includes node A via relay_addr
8. Send inference request; assert it routes through relay to node A
```
## Package layout
```
packages/relay/
pyproject.toml
meshnet_relay/
__init__.py
server.py # aiohttp relay + gossip hub + circuit relay proxy
cli.py # meshnet-relay entrypoint
peer_registry.py # in-memory {peer_id: {addr, last_seen, ...}}
circuit_relay.py # WSS proxy between two peers
packages/p2p/
meshnet_p2p/
gossip.py # GossipClient — connect to relay + peers, pub/sub
mdns.py # ZeroconfDiscovery — mDNS announce + browse
identity.py # PeerIdentity — generate/load peer_id + keypair
tls.py # cert generation, fingerprint, SSLContext helpers
packages/node/meshnet_node/
gossip_integration.py # wires GossipClient into node lifecycle
```
## Acceptance criteria
- All node↔node and node↔tracker HTTP uses HTTPS; self-signed cert auto-generated on first run
- `cert_fingerprint` included in heartbeat; tracker stores and logs it
- mDNS: two nodes on the same LAN discover each other without manual tracker URL (test with two localhost processes using different mDNS names)
- Relay: `meshnet-relay` starts, accepts WSS connections, fans out gossip messages to all connected peers
- Circuit relay: node A (127.0.0.1-only) can receive a gossip message via the relay from node B
- Tracker routes inference to node A using `relay_addr` when direct addr not reachable
- `relay_bootstrap.json` exists in `packages/p2p/` with at least one entry (localhost for tests)
- ADR-0010 documents the gossip architecture and libp2p migration path
- `python -m pytest` passes from repo root
- Commit only this story's changes

157
.scratch/distributed-inference-network/issues/18-two-machine-lan-test.md Normal file
View File

@@ -0,0 +1,157 @@
# US-018 — End-to-end two-machine LAN inference test
## Goal
Run real distributed inference across two physical machines: the Linux rig and a Windows 11 rig running WSL2. Document every setup step, firewall rule, and gotcha so this is repeatable. The test script exits 0 with token output and timing, proving the network works.
## Network topology for LAN test
```
[Linux machine] [Windows 11 / WSL2]
meshnet-tracker :8080 meshnet-node (shard B)
meshnet-node :8001 (shard A, tracker-mode)
meshnet-gateway :8000 (optional, for OpenAI-compat)
Client (either machine):
scripts/test_lan_inference.py --tracker http://192.168.1.10:8080
```
The Linux machine runs the tracker + the first-shard node (tracker-mode). The Windows/WSL2 machine runs the second-shard node. A small model (e.g. Phi-3-medium at BF16, fits on one GPU each) is split across both.
## WSL2 setup (Windows side)
`docs/INSTALL_WINDOWS.md` covers:
1. Enable WSL2: `wsl --install -d Ubuntu-24.04`
2. CUDA in WSL2: install NVIDIA driver on Windows (NOT inside WSL); WSL2 gets CUDA automatically
- Verify: `nvidia-smi` inside WSL2 should show GPU
3. Install Python 3.11+ and pip inside WSL2
4. `pip install -e packages/node packages/p2p` (clone repo first)
5. Firewall: Windows Defender must allow inbound WSL2 → LAN on node port
- PowerShell: `New-NetFirewallRule -DisplayName "meshnet-node" -Direction Inbound -Protocol TCP -LocalPort 8001 -Action Allow`
6. WSL2 IP: WSL2 has its own NAT'd IP (172.x.x.x); to expose to LAN, either:
- Option A: `netsh interface portproxy add v4tov4 listenport=8001 listenaddress=0.0.0.0 connectport=8001 connectaddress=$(wsl hostname -I)`
- Option B: use the relay node (US-017) — no port forwarding needed
## Linux setup
Standard install (already done after US-016). Firewall:
```bash
# If using ufw
sudo ufw allow 8080/tcp # tracker
sudo ufw allow 8001/tcp # node
sudo ufw allow 8000/tcp # gateway (optional)
```
## Model split
For the test, use a model that has enough layers to split meaningfully but fits comfortably in memory. Phi-3-medium-128k-instruct (40 layers, BF16 15GB) works on a single 24GB GPU on each machine:
- Linux node: layers 0–19 (tracker-mode, owns tokenizer + embed_tokens)
- Windows/WSL2 node: layers 20–39
Start sequence:
```bash
# Terminal 1 (Linux) — tracker
meshnet-tracker --port 8080
# Terminal 2 (Linux) — first-shard node (tracker-mode auto-detected because shard_start=0)
meshnet-node --model microsoft/Phi-3-medium-128k-instruct \
--quantization bf16 \
--shard-start 0 --shard-end 19 \
--tracker http://localhost:8080 \
--port 8001
# Terminal 3 (Windows WSL2) — second-shard node
meshnet-node --model microsoft/Phi-3-medium-128k-instruct \
--quantization bf16 \
--shard-start 20 --shard-end 39 \
--tracker http://192.168.1.10:8080 \
--port 8001
```
## Test script
`scripts/test_lan_inference.py`:
```python
#!/usr/bin/env python3
"""
End-to-end LAN inference test.
Usage: python scripts/test_lan_inference.py --tracker http://192.168.1.10:8080
"""
import argparse, time, httpx, json
MESSAGES = [
{"role": "user", "content": "What is 7 × 8? Answer in one word."},
{"role": "user", "content": "Name the capital of France in one word."},
{"role": "user", "content": "Complete the sequence: 1, 1, 2, 3, 5, ___"},
]
def run_test(tracker_url: str, gateway_url: str | None):
# Discover inference entry point via tracker if gateway not given
if not gateway_url:
r = httpx.get(f"{tracker_url}/v1/tracker-nodes/phi-3-medium", timeout=5)
r.raise_for_status()
nodes = r.json()
assert nodes, "No tracker-mode nodes registered — is the first-shard node running?"
gateway_url = nodes[0]["url"]
print(f"Inference endpoint: {gateway_url}")
print(f"Tracker: {tracker_url}")
print()
for i, msg in enumerate(MESSAGES):
t0 = time.monotonic()
r = httpx.post(
f"{gateway_url}/v1/chat/completions",
json={"model": "phi-3-medium", "messages": [msg], "stream": False},
timeout=60,
)
r.raise_for_status()
data = r.json()
elapsed = time.monotonic() - t0
content = data["choices"][0]["message"]["content"]
tokens = data["usage"]["completion_tokens"]
tps = tokens / elapsed if elapsed > 0 else 0
print(f"[{i+1}] Q: {msg['content']}")
print(f" A: {content}")
print(f" {tokens} tokens {elapsed:.2f}s {tps:.1f} t/s")
print()
print("✓ All 3 requests completed successfully")
if __name__ == "__main__":
p = argparse.ArgumentParser()
p.add_argument("--tracker", required=True)
p.add_argument("--gateway", default=None)
args = p.parse_args()
run_test(args.tracker, args.gateway)
```
## Docs: TWO_MACHINE_TEST.md
`docs/TWO_MACHINE_TEST.md` must cover:
1. Prerequisites (models downloaded on both machines, same model ID, complementary shard ranges)
2. Start order: tracker first, then nodes, then test script
3. How to verify nodes are registered: `GET /v1/nodes` on tracker
4. How to verify coverage: `GET /v1/coverage/phi-3-medium` — all 40 layers must show node_count ≥ 1
5. How to run the test script
6. Expected output
7. Latency breakdown: how to read per-hop latency from node logs
8. **Known Issues** section — updated during actual test run with real gotchas
## Acceptance criteria
- `docs/INSTALL_WINDOWS.md` covers WSL2 + CUDA + meshnet-node install end-to-end
- `docs/TWO_MACHINE_TEST.md` covers the full two-machine setup and test procedure
- `scripts/test_lan_inference.py` exists and is executable
- When run against a real two-machine LAN setup: script exits 0, prints 3 valid answers with timing
- Coverage map shows 100% coverage (no gap) after both nodes register
- Known Issues section in TWO_MACHINE_TEST.md contains at least the issues encountered during this test run
- No new pytest failures from repo root (this story adds docs + a script, not new Python packages)
- Commit only this story's changes