neuron-tai/docs/adr/0014-relay-outbound-client.md

ADR-0014: Relay outbound client for NAT/internet pipeline hops

Status: Accepted

Context

ADR-0010 describes the relay server: a public WebSocket hub where nodes behind NAT connect outbound and register as reachable peers. That ADR focused on the inbound side: how the tracker reaches a behind-NAT node for the initial chat request.

The pipeline hop problem is different: when node A has the head shard and node B (behind NAT) has the tail shard, node A must forward binary activations to node B for every generated token. Direct HTTP from A to B is blocked. The relay must carry this per-hop activation traffic.

Why this is harder than tracker → node

The tracker-to-node relay (ADR-0010) proxies a single JSON request. The activation hop carries raw bfloat16 tensors — binary data that must survive round-tripping through the relay's JSON message envelope without precision loss.

Also, the relay /rpc/{peer_id} endpoint (one WebSocket connection per request) must be opened and closed for every token in the autoregressive loop. Latency of connection setup matters.

Options considered

A. Relay hop (WebSocket per hop, chosen)
Node A opens a WebSocket to wss://relay/rpc/{peer_id_B}, sends the activation, receives the response, closes. The relay's _handle_rpc forwards it to B's persistent connection via the existing relay-http-request envelope mechanism.

Pros: reuses the existing relay server unchanged. Each hop is independent; failures don't affect other requests.

Cons: WebSocket connection setup adds ~50–150 ms per hop on a fast relay. For autoregressive inference (N tokens × M hops), this adds up.

B. Persistent per-session tunnel
Node A opens a persistent WebSocket to the relay for the duration of an inference session and multiplexes all token hops over it.

Pros: amortises connection setup across tokens.

Cons: requires session-level state on the relay; complicates relay shutdown/failover; the current relay is stateless by design. Deferred for a future optimization.

C. Tracker-proxied activations
Route all activation traffic through the tracker's HTTP proxy.

Cons: the tracker is the control plane, not the data plane. High-volume binary tensor traffic through the tracker would saturate it. Rejected.

Decision

Option A — per-hop WebSocket relay. Simple, reuses existing infrastructure, correct. Option B is noted as a future optimization when activation-path latency becomes the bottleneck.

Protocol

Node A opens WS → wss://relay/rpc/{peer_id_B}
Node A sends:
  {
    "request_id": "<hex>",
    "method": "POST",
    "path": "/forward",
    "headers": { "X-Meshnet-Shape": "...", "X-Meshnet-Start-Layer": "12", ... },
    "body_base64": "<base64(bfloat16 tensor)>"
  }

Relay forwards to Node B as relay-http-request envelope.
Node B's RelayHttpBridge decodes body_base64, calls POST /forward locally.
Response:
  {
    "request_id": "<hex>",
    "status": 200,
    "headers": { "x-meshnet-shape": "...", "content-type": "application/octet-stream" },
    "body_base64": "<base64(output tensor)>"   ← for binary responses
    # OR
    "body": "<json string>"                    ← for text (last-hop decode)
  }
Relay sends response JSON back to Node A.
Node A decodes body_base64, continues pipeline.

Binary data through JSON: base64

Raw bfloat16 bytes cannot safely transit JSON (no UTF-8 guarantee, lossy decode). body_base64 carries the tensor as base64; the bridge decodes it before calling the local HTTP endpoint, and re-encodes the response. No precision loss.

Text responses (final hop, application/json) use body (plain string) for efficiency.

Fallback

If _relay_hop raises (relay unreachable, peer disconnected), _run_downstream_pipeline logs a warning and retries via direct HTTP. If both fail, the hop returns a pipeline error string and the token is skipped.

Tracker injection

The tracker's _handle_proxy_chat includes relay_addr in each downstream hop dict when the node has one registered:

{"endpoint": "http://172.29.x.x:7002", "start_layer": 12, "relay_addr": "wss://relay/rpc/abc123"}

The head node reads relay_addr from the injected X-Meshnet-Route header and calls _relay_hop instead of direct HTTP.

Consequences

• Nodes behind NAT (WSL2, 5G, home routers) can now participate in distributed pipeline inference without opening firewall ports
• relay_addr is a stable registration field; nodes without a relay omit it and receive direct HTTP hops
• Per-hop WebSocket setup adds latency proportional to relay RTT; acceptable for prototype, optimize later with persistent tunnels
• Base64 encoding increases payload size by ~33%; acceptable for prototype
• The relay server remains stateless and horizontally scalable; only the persistent per-peer /ws connections are stateful