ADR-0014: Relay outbound client for NAT/internet pipeline hops

Status: Accepted, amended 2026-07-10

Context

ADR-0010 describes the relay server: a public WebSocket hub where nodes behind NAT connect outbound and register as reachable peers. That ADR focused on the inbound side: how the tracker reaches a behind-NAT node for the initial chat request.

The pipeline hop problem is different: when node A has the head shard and node B (behind NAT) has the tail shard, node A must forward binary activations to node B for every generated token. Direct HTTP from A to B is blocked. The relay must carry this per-hop activation traffic.

Why this is harder than tracker → node

The tracker-to-node relay (ADR-0010) proxies a single JSON request. The activation hop carries raw bfloat16 tensors — binary data that must survive round-tripping through the relay's JSON message envelope without precision loss.

Also, the relay /rpc/{peer_id} endpoint (one WebSocket connection per request) must be opened and closed for every token in the autoregressive loop. Latency of connection setup matters.

Options considered

A. Relay hop (persistent per Route Session, chosen) Node A opens a WebSocket to wss://relay/rpc/{peer_id_B}, sends activation requests sequentially for the Route Session, then closes it when generation ends. The relay's _handle_rpc forwards each request to B's persistent connection via the existing relay-http-request envelope mechanism.

Pros: reuses the existing relay server unchanged. Each hop is independent; failures don't affect other requests.

The original implementation opened and closed this socket per token. It was amended to retain one requester socket per downstream relay address for the generation, so connection setup is amortized across all tokens.

B. Multiplexed persistent tunnel Node A sends multiple concurrent Route Sessions over a shared WebSocket and demultiplexes responses by request id.

Pros: amortises connection setup across tokens.

Cons: requires session-level state on the relay; complicates relay shutdown/failover; the current relay is stateless by design. Deferred for a future optimization.

C. Tracker-proxied activations
Route all activation traffic through the tracker's HTTP proxy.

Cons: the tracker is the control plane, not the data plane. High-volume binary tensor traffic through the tracker would saturate it. Rejected.

Decision

Option A — one sequential WebSocket per relayed Activation Seam and Route Session. Each activation still has a unique request id for response correlation, while X-Meshnet-Session remains stable for KV state. Option B remains a possible connection-count optimization for high-concurrency workloads.

Protocol

Node A opens WS once → wss://relay/rpc/{peer_id_B}
Node A sends repeatedly:
  {
    "request_id": "<hex>",
    "method": "POST",
    "path": "/forward",
    "headers": { "X-Meshnet-Shape": "...", "X-Meshnet-Start-Layer": "12", ... },
    "body_base64": "<base64(bfloat16 tensor)>"
  }

Relay forwards to Node B as relay-http-request envelope.
Node B's RelayHttpBridge decodes body_base64, calls POST /forward locally.
Response:
  {
    "request_id": "<hex>",
    "status": 200,
    "headers": { "x-meshnet-shape": "...", "content-type": "application/octet-stream" },
    "body_base64": "<base64(output tensor)>"   ← for binary responses
    # OR
    "body": "<json string>"                    ← for text (last-hop decode)
  }
Relay sends each response back to Node A without closing the requester socket.
Node A continues the pipeline and closes the socket when generation ends.

Binary data through JSON: base64

Raw bfloat16 bytes cannot safely transit JSON (no UTF-8 guarantee, lossy decode). body_base64 carries the tensor as base64; the bridge decodes it before calling the local HTTP endpoint, and re-encodes the response. No precision loss.

Text responses (final hop, application/json) use body (plain string) for efficiency.

Fallback

If _relay_hop raises (relay unreachable, peer disconnected), _run_downstream_pipeline logs a warning and retries via direct HTTP. If both fail, the hop returns a pipeline error string and the token is skipped.

Tracker injection

The tracker's _handle_proxy_chat includes relay_addr in each downstream hop dict when the node has one registered:

{"endpoint": "http://172.29.x.x:7002", "start_layer": 12, "relay_addr": "wss://relay/rpc/abc123"}

The head node reads relay_addr from the injected X-Meshnet-Route header and calls _relay_hop instead of direct HTTP.

Consequences

Nodes behind NAT (WSL2, 5G, home routers) can now participate in distributed pipeline inference without opening firewall ports
relay_addr is a stable registration field; nodes without a relay omit it and receive direct HTTP hops
WebSocket/TCP/TLS setup occurs once per relayed Activation Seam per Route Session, not once per generated token
Base64 encoding increases payload size by ~33%; acceptable for prototype
The relay server remains stateless and horizontally scalable; only the persistent per-peer /ws connections are stateful

5.1 KiB Raw Blame History