neuron-tai/.scratch/distributed-inference-network/issues/17-p2p-gossip-relay-ssl.md

US-017 — P2P gossip, NAT-traversal relay node, and SSL/TLS

Goal

Nodes must work behind NAT (home routers, cloud VMs without public IPs) and must communicate securely. Implement:

1. SSL/TLS everywhere — all HTTP between nodes/tracker is HTTPS; all WebSocket gossip is WSS
2. mDNS peer discovery — nodes on the same LAN find each other automatically (no config)
3. WebSocket gossip PubSub — nodes propagate join/leave/coverage-update events in near-real-time
4. Circuit relay node — team-run public relay (packages/relay) that enables NAT traversal and bootstraps new nodes joining from the internet

Architecture is designed to migrate to libp2p GossipSub + Kademlia DHT without breaking the message schema (topic names and payload formats are stable contracts).

Gossip protocol

Transport

WebSocket (wss://) using the websockets Python library. Each node maintains persistent WSS connections to:

• The relay node (always, bootstraps peer list)
• Up to 8 direct peers (Kademlia-style target fanout; peers discovered via mDNS + relay peer list)

Topics

All messages are JSON with an envelope:

{
  "topic": "node-join",
  "version": 1,
  "from_peer": "<peer_id>",
  "timestamp": "<iso8601>",
  "payload": { ... }
}

Topic	Direction	Payload
node-join	broadcast	{peer_id, addr, models: [{model_preset, shard_start, shard_end}], vram_gb, quant}
node-leave	broadcast	{peer_id, reason}
coverage-update	broadcast	{model_preset, coverage: [{start, end, count}]}
heartbeat	peer→relay	{peer_id, addr, uptime_s, tokens_per_sec}
peer-list	relay→peer	{peers: [{peer_id, addr}]}
relay-announce	relay→all	{relay_id, relay_url, capacity}

Gossip fanout: each node re-broadcasts received messages to all its peers (simple flooding with seen_ids dedup, TTL=3 hops). Migration to GossipSub mesh routing is a later ADR.

Peer ID

peer_id = sha256(public_key)[:16].hex() — generated on first run, stored in ~/.config/meshnet/identity.json. The same keypair is used for TLS client certificates (mTLS) in future work.

mDNS LAN discovery

Use Python zeroconf library. Service type: _meshnet._tcp.local.

from zeroconf import ServiceInfo, Zeroconf

info = ServiceInfo(
    "_meshnet._tcp.local.",
    f"{peer_id}._meshnet._tcp.local.",
    addresses=[socket.inet_aton(local_ip)],
    port=node_port,
    properties={"peer_id": peer_id, "version": "1"},
)
zc = Zeroconf()
zc.register_service(info)

On startup, nodes also browse for _meshnet._tcp.local. to discover existing nodes. mDNS is LAN-only (does not traverse routers), which is correct for LAN discovery.

NAT traversal: circuit relay

How it works

1. Node A (behind NAT) cannot accept inbound TCP connections
2. Node A connects outbound to the public relay via WSS
3. Node A tells the tracker: "effective_addr": "wss://relay.meshnet.ai/relay/{peer_id_A}"
4. Node B (wants to call A) connects to the relay at the above URL
5. Relay proxies the TCP stream between A and B

Hole-punching (direct connection via STUN) is attempted first (future work). Relay is the fallback.

meshnet-relay

packages/relay/meshnet_relay/server.py — a standalone aiohttp server:

GET  /health                        → {status: ok}
GET  /v1/peers                      → [{peer_id, addr, last_seen}]
POST /v1/gossip                     → receive a gossip message, fan out to connected peers
WSS  /ws                            → persistent gossip connection (subscribe to all topics)
WSS  /relay/{peer_id}               → circuit relay proxy to that peer_id
GET  /v1/relay/capacity             → {connected_peers: N, max_peers: 500}

CLI:

meshnet-relay [--port 8443] [--cert path/to/cert.pem] [--key path/to/key.pem]
              [--tracker-url http://...] [--max-peers 500]

The relay can optionally proxy to the tracker (so relay.meshnet.ai is the single internet-visible endpoint).

SSL/TLS setup

Node certificate (self-signed, auto-generated)

On first run, meshnet-node generates a self-signed RSA-2048 cert valid for 10 years:

from cryptography import x509
from cryptography.hazmat.primitives import hashes, serialization
from cryptography.hazmat.primitives.asymmetric import rsa

Cert saved to ~/.config/meshnet/node_cert.pem + node_key.pem. Fingerprint stored in config and shared with tracker via heartbeat. Nodes connecting to each other validate the fingerprint (TOFU — trust on first use), not the CA chain.

Relay certificate

The relay uses a real Let's Encrypt cert (cert-bot or acme.sh). The relay cert is pinned in packages/p2p/relay_bootstrap.json:

{
  "relays": [
    {
      "url": "wss://relay.meshnet.ai:8443",
      "cert_fingerprint": "sha256:<hex>",
      "operator": "meshnet-team"
    }
  ]
}

All HTTP switched to HTTPS

meshnet-node starts an HTTPS server using ssl.SSLContext. meshnet-tracker similarly. All outbound httpx / aiohttp calls use TLS verification against pinned fingerprints (not the system CA store — too many corporate proxies break this).

Tracker changes

Heartbeat payload gains new fields:

{
  "peer_id": "a1b2c3d4e5f6a1b2",
  "effective_addr": "https://192.168.1.42:8001",
  "relay_addr": "wss://relay.meshnet.ai:8443/relay/a1b2c3d4e5f6a1b2",
  "cert_fingerprint": "sha256:...",
  "gossip_peers": ["peer_id_1", "peer_id_2"]
}

Tracker uses effective_addr (direct) or relay_addr (fallback) when building inference routes.

Integration test

tests/test_gossip_and_relay.py

scenario:
  1. Start a local relay (localhost:18443)
  2. Start node A (no inbound port — simulate NAT by binding to 127.0.0.1 only)
  3. Start node B (public-reachable on localhost)
  4. Both register with relay; relay peer-list includes both
  5. Node B sends a gossip node-join message
  6. Assert node A receives it within 500ms
  7. Start tracker; confirm tracker's node registry includes node A via relay_addr
  8. Send inference request; assert it routes through relay to node A

Package layout

packages/relay/
  pyproject.toml
  meshnet_relay/
    __init__.py
    server.py          # aiohttp relay + gossip hub + circuit relay proxy
    cli.py             # meshnet-relay entrypoint
    peer_registry.py   # in-memory {peer_id: {addr, last_seen, ...}}
    circuit_relay.py   # WSS proxy between two peers

packages/p2p/
  meshnet_p2p/
    gossip.py          # GossipClient — connect to relay + peers, pub/sub
    mdns.py            # ZeroconfDiscovery — mDNS announce + browse
    identity.py        # PeerIdentity — generate/load peer_id + keypair
    tls.py             # cert generation, fingerprint, SSLContext helpers

packages/node/meshnet_node/
  gossip_integration.py  # wires GossipClient into node lifecycle

Acceptance criteria

• All node↔node and node↔tracker HTTP uses HTTPS; self-signed cert auto-generated on first run
• cert_fingerprint included in heartbeat; tracker stores and logs it
• mDNS: two nodes on the same LAN discover each other without manual tracker URL (test with two localhost processes using different mDNS names)
• Relay: meshnet-relay starts, accepts WSS connections, fans out gossip messages to all connected peers
• Circuit relay: node A (127.0.0.1-only) can receive a gossip message via the relay from node B
• Tracker routes inference to node A using relay_addr when direct addr not reachable
• relay_bootstrap.json exists in packages/p2p/ with at least one entry (localhost for tests)
• ADR-0010 documents the gossip architecture and libp2p migration path
• python -m pytest passes from repo root
• Commit only this story's changes