docs: add US-020–029 issue files, ADR 0011–0014, update prd.json to 29/29
Issue files (.scratch/issues/20-29): retrospective specs for all work done in the current sprint — hardening, route-timeout, start-layer protocol, heartbeat stats, availability map, rolling RPM, smart assignment, throughput routing, routing tests, relay outbound client. ADRs (docs/adr/0011-0014): 0011 — Auto-shard from memory budget and tracker network assignment 0012 — X-Meshnet-Start-Layer overlapping shard execution protocol 0013 — Rolling RPM statistics, smart assignment scoring, throughput routing 0014 — Relay outbound client for NAT/internet pipeline hops prd.json: US-020 through US-029 added, all marked done. ralph_progress.py now shows 29/29 complete (100%). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
120
docs/adr/0014-relay-outbound-client.md
Normal file
120
docs/adr/0014-relay-outbound-client.md
Normal file
@@ -0,0 +1,120 @@
|
||||
# ADR-0014: Relay outbound client for NAT/internet pipeline hops
|
||||
|
||||
## Status: Accepted
|
||||
|
||||
## Context
|
||||
|
||||
ADR-0010 describes the relay server: a public WebSocket hub where nodes behind NAT
|
||||
connect outbound and register as reachable peers. That ADR focused on the *inbound*
|
||||
side: how the tracker reaches a behind-NAT node for the initial chat request.
|
||||
|
||||
The *pipeline hop* problem is different: when node A has the head shard and node B
|
||||
(behind NAT) has the tail shard, node A must forward binary activations to node B
|
||||
for *every generated token*. Direct HTTP from A to B is blocked. The relay must
|
||||
carry this per-hop activation traffic.
|
||||
|
||||
### Why this is harder than tracker → node
|
||||
|
||||
The tracker-to-node relay (ADR-0010) proxies a single JSON request. The activation
|
||||
hop carries raw bfloat16 tensors — binary data that must survive round-tripping
|
||||
through the relay's JSON message envelope without precision loss.
|
||||
|
||||
Also, the relay `/rpc/{peer_id}` endpoint (one WebSocket connection per request)
|
||||
must be opened and closed for every token in the autoregressive loop. Latency
|
||||
of connection setup matters.
|
||||
|
||||
## Options considered
|
||||
|
||||
**A. Relay hop (WebSocket per hop, chosen)**
|
||||
Node A opens a WebSocket to `wss://relay/rpc/{peer_id_B}`, sends the activation,
|
||||
receives the response, closes. The relay's `_handle_rpc` forwards it to B's persistent
|
||||
connection via the existing `relay-http-request` envelope mechanism.
|
||||
|
||||
Pros: reuses the existing relay server unchanged. Each hop is independent; failures don't
|
||||
affect other requests.
|
||||
|
||||
Cons: WebSocket connection setup adds ~50–150 ms per hop on a fast relay. For
|
||||
autoregressive inference (N tokens × M hops), this adds up.
|
||||
|
||||
**B. Persistent per-session tunnel**
|
||||
Node A opens a persistent WebSocket to the relay for the duration of an inference session
|
||||
and multiplexes all token hops over it.
|
||||
|
||||
Pros: amortises connection setup across tokens.
|
||||
|
||||
Cons: requires session-level state on the relay; complicates relay shutdown/failover;
|
||||
the current relay is stateless by design. Deferred for a future optimization.
|
||||
|
||||
**C. Tracker-proxied activations**
|
||||
Route all activation traffic through the tracker's HTTP proxy.
|
||||
|
||||
Cons: the tracker is the control plane, not the data plane. High-volume binary tensor
|
||||
traffic through the tracker would saturate it. Rejected.
|
||||
|
||||
## Decision
|
||||
|
||||
Option A — per-hop WebSocket relay. Simple, reuses existing infrastructure, correct.
|
||||
Option B is noted as a future optimization when activation-path latency becomes the
|
||||
bottleneck.
|
||||
|
||||
## Protocol
|
||||
|
||||
```
|
||||
Node A opens WS → wss://relay/rpc/{peer_id_B}
|
||||
Node A sends:
|
||||
{
|
||||
"request_id": "<hex>",
|
||||
"method": "POST",
|
||||
"path": "/forward",
|
||||
"headers": { "X-Meshnet-Shape": "...", "X-Meshnet-Start-Layer": "12", ... },
|
||||
"body_base64": "<base64(bfloat16 tensor)>"
|
||||
}
|
||||
|
||||
Relay forwards to Node B as relay-http-request envelope.
|
||||
Node B's RelayHttpBridge decodes body_base64, calls POST /forward locally.
|
||||
Response:
|
||||
{
|
||||
"request_id": "<hex>",
|
||||
"status": 200,
|
||||
"headers": { "x-meshnet-shape": "...", "content-type": "application/octet-stream" },
|
||||
"body_base64": "<base64(output tensor)>" ← for binary responses
|
||||
# OR
|
||||
"body": "<json string>" ← for text (last-hop decode)
|
||||
}
|
||||
Relay sends response JSON back to Node A.
|
||||
Node A decodes body_base64, continues pipeline.
|
||||
```
|
||||
|
||||
### Binary data through JSON: base64
|
||||
|
||||
Raw bfloat16 bytes cannot safely transit JSON (no UTF-8 guarantee, lossy decode).
|
||||
`body_base64` carries the tensor as base64; the bridge decodes it before calling
|
||||
the local HTTP endpoint, and re-encodes the response. No precision loss.
|
||||
|
||||
Text responses (final hop, `application/json`) use `body` (plain string) for efficiency.
|
||||
|
||||
### Fallback
|
||||
|
||||
If `_relay_hop` raises (relay unreachable, peer disconnected), `_run_downstream_pipeline`
|
||||
logs a warning and retries via direct HTTP. If both fail, the hop returns a pipeline error
|
||||
string and the token is skipped.
|
||||
|
||||
### Tracker injection
|
||||
|
||||
The tracker's `_handle_proxy_chat` includes `relay_addr` in each downstream hop dict
|
||||
when the node has one registered:
|
||||
|
||||
```json
|
||||
{"endpoint": "http://172.29.x.x:7002", "start_layer": 12, "relay_addr": "wss://relay/rpc/abc123"}
|
||||
```
|
||||
|
||||
The head node reads `relay_addr` from the injected `X-Meshnet-Route` header and calls
|
||||
`_relay_hop` instead of direct HTTP.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Nodes behind NAT (WSL2, 5G, home routers) can now participate in distributed pipeline inference without opening firewall ports
|
||||
- `relay_addr` is a stable registration field; nodes without a relay omit it and receive direct HTTP hops
|
||||
- Per-hop WebSocket setup adds latency proportional to relay RTT; acceptable for prototype, optimize later with persistent tunnels
|
||||
- Base64 encoding increases payload size by ~33%; acceptable for prototype
|
||||
- The relay server remains stateless and horizontally scalable; only the persistent per-peer `/ws` connections are stateful
|
||||
Reference in New Issue
Block a user