# ADR-0014: Relay outbound client for NAT/internet pipeline hops

## Status: Accepted, amended 2026-07-10

## Context

ADR-0010 describes the relay server: a public WebSocket hub where nodes behind NAT
connect outbound and register as reachable peers. That ADR focused on the *inbound*
side: how the tracker reaches a behind-NAT node for the initial chat request.

The *pipeline hop* problem is different: when node A has the head shard and node B
(behind NAT) has the tail shard, node A must forward binary activations to node B
for *every generated token*. Direct HTTP from A to B is blocked. The relay must
carry this per-hop activation traffic.

### Why this is harder than tracker → node

The tracker-to-node relay (ADR-0010) proxies a single JSON request. The activation
hop carries raw bfloat16 tensors — binary data that must survive round-tripping
through the relay's JSON message envelope without precision loss.

Also, the relay `/rpc/{peer_id}` endpoint (one WebSocket connection per request)
must be opened and closed for every token in the autoregressive loop. Latency
of connection setup matters.

## Options considered

**A. Relay hop (persistent per Route Session, chosen)**
Node A opens a WebSocket to `wss://relay/rpc/{peer_id_B}`, sends activation requests
sequentially for the Route Session, then closes it when generation ends. The relay's
`_handle_rpc` forwards each request to B's persistent connection via the existing
`relay-http-request` envelope mechanism.

Pros: reuses the existing relay server unchanged. Each hop is independent; failures don't
affect other requests.

The original implementation opened and closed this socket per token. It was amended
to retain one requester socket per downstream relay address for the generation, so
connection setup is amortized across all tokens.

**B. Multiplexed persistent tunnel**
Node A sends multiple concurrent Route Sessions over a shared WebSocket and demultiplexes
responses by request id.

Pros: amortises connection setup across tokens.

Cons: requires session-level state on the relay; complicates relay shutdown/failover;
the current relay is stateless by design. Deferred for a future optimization.

**C. Tracker-proxied activations**  
Route all activation traffic through the tracker's HTTP proxy.

Cons: the tracker is the control plane, not the data plane. High-volume binary tensor
traffic through the tracker would saturate it. Rejected.

## Decision

Option A — one sequential WebSocket per relayed Activation Seam and Route Session.
Each activation still has a unique request id for response correlation, while
`X-Meshnet-Session` remains stable for KV state. Option B remains a possible
connection-count optimization for high-concurrency workloads.

## Protocol

```
Node A opens WS once → wss://relay/rpc/{peer_id_B}
Node A sends repeatedly:
  {
    "request_id": "<hex>",
    "method": "POST",
    "path": "/forward",
    "headers": { "X-Meshnet-Shape": "...", "X-Meshnet-Start-Layer": "12", ... },
    "body_base64": "<base64(bfloat16 tensor)>"
  }

Relay forwards to Node B as relay-http-request envelope.
Node B's RelayHttpBridge decodes body_base64, calls POST /forward locally.
Response:
  {
    "request_id": "<hex>",
    "status": 200,
    "headers": { "x-meshnet-shape": "...", "content-type": "application/octet-stream" },
    "body_base64": "<base64(output tensor)>"   ← for binary responses
    # OR
    "body": "<json string>"                    ← for text (last-hop decode)
  }
Relay sends each response back to Node A without closing the requester socket.
Node A continues the pipeline and closes the socket when generation ends.
```

### Binary data through JSON: base64

Raw bfloat16 bytes cannot safely transit JSON (no UTF-8 guarantee, lossy decode).
`body_base64` carries the tensor as base64; the bridge decodes it before calling
the local HTTP endpoint, and re-encodes the response. No precision loss.

Text responses (final hop, `application/json`) use `body` (plain string) for efficiency.

### Fallback

If `_relay_hop` raises (relay unreachable, peer disconnected), `_run_downstream_pipeline`
logs a warning and retries via direct HTTP. If both fail, the hop returns a pipeline error
string and the token is skipped.

### Tracker injection

The tracker's `_handle_proxy_chat` includes `relay_addr` in each downstream hop dict
when the node has one registered:

```json
{"endpoint": "http://172.29.x.x:7002", "start_layer": 12, "relay_addr": "wss://relay/rpc/abc123"}
```

The head node reads `relay_addr` from the injected `X-Meshnet-Route` header and calls
`_relay_hop` instead of direct HTTP.

## Consequences

- Nodes behind NAT (WSL2, 5G, home routers) can now participate in distributed pipeline inference without opening firewall ports
- `relay_addr` is a stable registration field; nodes without a relay omit it and receive direct HTTP hops
- WebSocket/TCP/TLS setup occurs once per relayed Activation Seam per Route Session, not once per generated token
- Base64 encoding increases payload size by ~33%; acceptable for prototype
- The relay server remains stateless and horizontally scalable; only the persistent per-peer `/ws` connections are stateful
