# ADR-0014: Relay outbound client for NAT/internet pipeline hops

## Status: Accepted

## Context

ADR-0010 describes the relay server: a public WebSocket hub where nodes behind NAT
connect outbound and register as reachable peers. That ADR focused on the *inbound*
side: how the tracker reaches a behind-NAT node for the initial chat request.

The *pipeline hop* problem is different: when node A has the head shard and node B
(behind NAT) has the tail shard, node A must forward binary activations to node B
for *every generated token*. Direct HTTP from A to B is blocked. The relay must
carry this per-hop activation traffic.

### Why this is harder than tracker → node

The tracker-to-node relay (ADR-0010) proxies a single JSON request. The activation
hop carries raw bfloat16 tensors — binary data that must survive round-tripping
through the relay's JSON message envelope without precision loss.

Also, the relay `/rpc/{peer_id}` endpoint (one WebSocket connection per request)
must be opened and closed for every token in the autoregressive loop. Latency
of connection setup matters.

## Options considered

**A. Relay hop (WebSocket per hop, chosen)**  
Node A opens a WebSocket to `wss://relay/rpc/{peer_id_B}`, sends the activation,
receives the response, closes. The relay's `_handle_rpc` forwards it to B's persistent
connection via the existing `relay-http-request` envelope mechanism.

Pros: reuses the existing relay server unchanged. Each hop is independent; failures don't
affect other requests.

Cons: WebSocket connection setup adds ~50–150 ms per hop on a fast relay. For
autoregressive inference (N tokens × M hops), this adds up.

**B. Persistent per-session tunnel**  
Node A opens a persistent WebSocket to the relay for the duration of an inference session
and multiplexes all token hops over it.

Pros: amortises connection setup across tokens.

Cons: requires session-level state on the relay; complicates relay shutdown/failover;
the current relay is stateless by design. Deferred for a future optimization.

**C. Tracker-proxied activations**  
Route all activation traffic through the tracker's HTTP proxy.

Cons: the tracker is the control plane, not the data plane. High-volume binary tensor
traffic through the tracker would saturate it. Rejected.

## Decision

Option A — per-hop WebSocket relay. Simple, reuses existing infrastructure, correct.
Option B is noted as a future optimization when activation-path latency becomes the
bottleneck.

## Protocol

```
Node A opens WS → wss://relay/rpc/{peer_id_B}
Node A sends:
  {
    "request_id": "<hex>",
    "method": "POST",
    "path": "/forward",
    "headers": { "X-Meshnet-Shape": "...", "X-Meshnet-Start-Layer": "12", ... },
    "body_base64": "<base64(bfloat16 tensor)>"
  }

Relay forwards to Node B as relay-http-request envelope.
Node B's RelayHttpBridge decodes body_base64, calls POST /forward locally.
Response:
  {
    "request_id": "<hex>",
    "status": 200,
    "headers": { "x-meshnet-shape": "...", "content-type": "application/octet-stream" },
    "body_base64": "<base64(output tensor)>"   ← for binary responses
    # OR
    "body": "<json string>"                    ← for text (last-hop decode)
  }
Relay sends response JSON back to Node A.
Node A decodes body_base64, continues pipeline.
```

### Binary data through JSON: base64

Raw bfloat16 bytes cannot safely transit JSON (no UTF-8 guarantee, lossy decode).
`body_base64` carries the tensor as base64; the bridge decodes it before calling
the local HTTP endpoint, and re-encodes the response. No precision loss.

Text responses (final hop, `application/json`) use `body` (plain string) for efficiency.

### Fallback

If `_relay_hop` raises (relay unreachable, peer disconnected), `_run_downstream_pipeline`
logs a warning and retries via direct HTTP. If both fail, the hop returns a pipeline error
string and the token is skipped.

### Tracker injection

The tracker's `_handle_proxy_chat` includes `relay_addr` in each downstream hop dict
when the node has one registered:

```json
{"endpoint": "http://172.29.x.x:7002", "start_layer": 12, "relay_addr": "wss://relay/rpc/abc123"}
```

The head node reads `relay_addr` from the injected `X-Meshnet-Route` header and calls
`_relay_hop` instead of direct HTTP.

## Consequences

- Nodes behind NAT (WSL2, 5G, home routers) can now participate in distributed pipeline inference without opening firewall ports
- `relay_addr` is a stable registration field; nodes without a relay omit it and receive direct HTTP hops
- Per-hop WebSocket setup adds latency proportional to relay RTT; acceptable for prototype, optimize later with persistent tunnels
- Base64 encoding increases payload size by ~33%; acceptable for prototype
- The relay server remains stateless and horizontally scalable; only the persistent per-peer `/ws` connections are stateful