docs: consolidate all docs under docs/ — single source of truth
Move issues (01–29) and PRD from .scratch/distributed-inference-network/ into docs/issues/ and docs/. Update ralph_progress.py DEFAULT_PRD path and rewrite docs/agents/issue-tracker.md to reflect the new layout. The distributed_inference_network.egg-info/docs/ mirror is a build artifact already covered by *.egg-info/ in .gitignore — not committed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Dobromir Popov
205
docs/issues/17-p2p-gossip-relay-ssl.md
Normal file
205
docs/issues/17-p2p-gossip-relay-ssl.md
Normal file
@@ -0,0 +1,205 @@
|
||||
# US-017 — P2P gossip, NAT-traversal relay node, and SSL/TLS
|
||||
|
||||
## Goal
|
||||
|
||||
Nodes must work behind NAT (home routers, cloud VMs without public IPs) and must communicate securely. Implement:
|
||||
|
||||
1. **SSL/TLS everywhere** — all HTTP between nodes/tracker is HTTPS; all WebSocket gossip is WSS
|
||||
2. **mDNS peer discovery** — nodes on the same LAN find each other automatically (no config)
|
||||
3. **WebSocket gossip PubSub** — nodes propagate join/leave/coverage-update events in near-real-time
|
||||
4. **Circuit relay node** — team-run public relay (`packages/relay`) that enables NAT traversal and bootstraps new nodes joining from the internet
|
||||
|
||||
Architecture is designed to migrate to libp2p GossipSub + Kademlia DHT without breaking the message schema (topic names and payload formats are stable contracts).
|
||||
|
||||
## Gossip protocol
|
||||
|
||||
### Transport
|
||||
|
||||
WebSocket (`wss://`) using the `websockets` Python library. Each node maintains persistent WSS connections to:
|
||||
- The relay node (always, bootstraps peer list)
|
||||
- Up to 8 direct peers (Kademlia-style target fanout; peers discovered via mDNS + relay peer list)
|
||||
|
||||
### Topics
|
||||
|
||||
All messages are JSON with an envelope:
|
||||
|
||||
```json
|
||||
{
|
||||
"topic": "node-join",
|
||||
"version": 1,
|
||||
"from_peer": "<peer_id>",
|
||||
"timestamp": "<iso8601>",
|
||||
"payload": { ... }
|
||||
}
|
||||
```
|
||||
|
||||
| Topic | Direction | Payload |
|
||||
|-------|-----------|---------|
|
||||
| `node-join` | broadcast | `{peer_id, addr, models: [{model_preset, shard_start, shard_end}], vram_gb, quant}` |
|
||||
| `node-leave` | broadcast | `{peer_id, reason}` |
|
||||
| `coverage-update` | broadcast | `{model_preset, coverage: [{start, end, count}]}` |
|
||||
| `heartbeat` | peer→relay | `{peer_id, addr, uptime_s, tokens_per_sec}` |
|
||||
| `peer-list` | relay→peer | `{peers: [{peer_id, addr}]}` |
|
||||
| `relay-announce` | relay→all | `{relay_id, relay_url, capacity}` |
|
||||
|
||||
Gossip fanout: each node re-broadcasts received messages to all its peers (simple flooding with `seen_ids` dedup, TTL=3 hops). Migration to GossipSub mesh routing is a later ADR.
|
||||
|
||||
### Peer ID
|
||||
|
||||
`peer_id = sha256(public_key)[:16].hex()` — generated on first run, stored in `~/.config/meshnet/identity.json`. The same keypair is used for TLS client certificates (mTLS) in future work.
|
||||
|
||||
## mDNS LAN discovery
|
||||
|
||||
Use Python `zeroconf` library. Service type: `_meshnet._tcp.local.`
|
||||
|
||||
```python
|
||||
from zeroconf import ServiceInfo, Zeroconf
|
||||
|
||||
info = ServiceInfo(
|
||||
"_meshnet._tcp.local.",
|
||||
f"{peer_id}._meshnet._tcp.local.",
|
||||
addresses=[socket.inet_aton(local_ip)],
|
||||
port=node_port,
|
||||
properties={"peer_id": peer_id, "version": "1"},
|
||||
)
|
||||
zc = Zeroconf()
|
||||
zc.register_service(info)
|
||||
```
|
||||
|
||||
On startup, nodes also browse for `_meshnet._tcp.local.` to discover existing nodes. mDNS is LAN-only (does not traverse routers), which is correct for LAN discovery.
|
||||
|
||||
## NAT traversal: circuit relay
|
||||
|
||||
### How it works
|
||||
|
||||
1. Node A (behind NAT) cannot accept inbound TCP connections
|
||||
2. Node A connects outbound to the public relay via WSS
|
||||
3. Node A tells the tracker: `"effective_addr": "wss://relay.meshnet.ai/relay/{peer_id_A}"`
|
||||
4. Node B (wants to call A) connects to the relay at the above URL
|
||||
5. Relay proxies the TCP stream between A and B
|
||||
|
||||
Hole-punching (direct connection via STUN) is attempted first (future work). Relay is the fallback.
|
||||
|
||||
### meshnet-relay
|
||||
|
||||
`packages/relay/meshnet_relay/server.py` — a standalone aiohttp server:
|
||||
|
||||
```
|
||||
GET /health → {status: ok}
|
||||
GET /v1/peers → [{peer_id, addr, last_seen}]
|
||||
POST /v1/gossip → receive a gossip message, fan out to connected peers
|
||||
WSS /ws → persistent gossip connection (subscribe to all topics)
|
||||
WSS /relay/{peer_id} → circuit relay proxy to that peer_id
|
||||
GET /v1/relay/capacity → {connected_peers: N, max_peers: 500}
|
||||
```
|
||||
|
||||
CLI:
|
||||
|
||||
```
|
||||
meshnet-relay [--port 8443] [--cert path/to/cert.pem] [--key path/to/key.pem]
|
||||
[--tracker-url http://...] [--max-peers 500]
|
||||
```
|
||||
|
||||
The relay can optionally proxy to the tracker (so `relay.meshnet.ai` is the single internet-visible endpoint).
|
||||
|
||||
## SSL/TLS setup
|
||||
|
||||
### Node certificate (self-signed, auto-generated)
|
||||
|
||||
On first run, `meshnet-node` generates a self-signed RSA-2048 cert valid for 10 years:
|
||||
|
||||
```python
|
||||
from cryptography import x509
|
||||
from cryptography.hazmat.primitives import hashes, serialization
|
||||
from cryptography.hazmat.primitives.asymmetric import rsa
|
||||
```
|
||||
|
||||
Cert saved to `~/.config/meshnet/node_cert.pem` + `node_key.pem`. Fingerprint stored in config and shared with tracker via heartbeat. Nodes connecting to each other validate the fingerprint (TOFU — trust on first use), not the CA chain.
|
||||
|
||||
### Relay certificate
|
||||
|
||||
The relay uses a real Let's Encrypt cert (cert-bot or acme.sh). The relay cert is pinned in `packages/p2p/relay_bootstrap.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"relays": [
|
||||
{
|
||||
"url": "wss://relay.meshnet.ai:8443",
|
||||
"cert_fingerprint": "sha256:<hex>",
|
||||
"operator": "meshnet-team"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### All HTTP switched to HTTPS
|
||||
|
||||
`meshnet-node` starts an HTTPS server using `ssl.SSLContext`. `meshnet-tracker` similarly. All outbound `httpx` / `aiohttp` calls use TLS verification against pinned fingerprints (not the system CA store — too many corporate proxies break this).
|
||||
|
||||
## Tracker changes
|
||||
|
||||
Heartbeat payload gains new fields:
|
||||
|
||||
```json
|
||||
{
|
||||
"peer_id": "a1b2c3d4e5f6a1b2",
|
||||
"effective_addr": "https://192.168.1.42:8001",
|
||||
"relay_addr": "wss://relay.meshnet.ai:8443/relay/a1b2c3d4e5f6a1b2",
|
||||
"cert_fingerprint": "sha256:...",
|
||||
"gossip_peers": ["peer_id_1", "peer_id_2"]
|
||||
}
|
||||
```
|
||||
|
||||
Tracker uses `effective_addr` (direct) or `relay_addr` (fallback) when building inference routes.
|
||||
|
||||
## Integration test
|
||||
|
||||
```
|
||||
tests/test_gossip_and_relay.py
|
||||
|
||||
scenario:
|
||||
1. Start a local relay (localhost:18443)
|
||||
2. Start node A (no inbound port — simulate NAT by binding to 127.0.0.1 only)
|
||||
3. Start node B (public-reachable on localhost)
|
||||
4. Both register with relay; relay peer-list includes both
|
||||
5. Node B sends a gossip node-join message
|
||||
6. Assert node A receives it within 500ms
|
||||
7. Start tracker; confirm tracker's node registry includes node A via relay_addr
|
||||
8. Send inference request; assert it routes through relay to node A
|
||||
```
|
||||
|
||||
## Package layout
|
||||
|
||||
```
|
||||
packages/relay/
|
||||
pyproject.toml
|
||||
meshnet_relay/
|
||||
__init__.py
|
||||
server.py # aiohttp relay + gossip hub + circuit relay proxy
|
||||
cli.py # meshnet-relay entrypoint
|
||||
peer_registry.py # in-memory {peer_id: {addr, last_seen, ...}}
|
||||
circuit_relay.py # WSS proxy between two peers
|
||||
|
||||
packages/p2p/
|
||||
meshnet_p2p/
|
||||
gossip.py # GossipClient — connect to relay + peers, pub/sub
|
||||
mdns.py # ZeroconfDiscovery — mDNS announce + browse
|
||||
identity.py # PeerIdentity — generate/load peer_id + keypair
|
||||
tls.py # cert generation, fingerprint, SSLContext helpers
|
||||
|
||||
packages/node/meshnet_node/
|
||||
gossip_integration.py # wires GossipClient into node lifecycle
|
||||
```
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- All node↔node and node↔tracker HTTP uses HTTPS; self-signed cert auto-generated on first run
|
||||
- `cert_fingerprint` included in heartbeat; tracker stores and logs it
|
||||
- mDNS: two nodes on the same LAN discover each other without manual tracker URL (test with two localhost processes using different mDNS names)
|
||||
- Relay: `meshnet-relay` starts, accepts WSS connections, fans out gossip messages to all connected peers
|
||||
- Circuit relay: node A (127.0.0.1-only) can receive a gossip message via the relay from node B
|
||||
- Tracker routes inference to node A using `relay_addr` when direct addr not reachable
|
||||
- `relay_bootstrap.json` exists in `packages/p2p/` with at least one entry (localhost for tests)
|
||||
- ADR-0010 documents the gossip architecture and libp2p migration path
|
||||
- `python -m pytest` passes from repo root
|
||||
- Commit only this story's changes
|
||||
Reference in New Issue
Block a user