docs: add US-020–029 issue files, ADR 0011–0014, update prd.json to 29/29

Issue files (.scratch/issues/20-29): retrospective specs for all work
done in the current sprint — hardening, route-timeout, start-layer
protocol, heartbeat stats, availability map, rolling RPM, smart
assignment, throughput routing, routing tests, relay outbound client.

ADRs (docs/adr/0011-0014):
  0011 — Auto-shard from memory budget and tracker network assignment
  0012 — X-Meshnet-Start-Layer overlapping shard execution protocol
  0013 — Rolling RPM statistics, smart assignment scoring, throughput routing
  0014 — Relay outbound client for NAT/internet pipeline hops

prd.json: US-020 through US-029 added, all marked done. ralph_progress.py
now shows 29/29 complete (100%).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Dobromir Popov
2026-06-30 22:15:41 +03:00
parent d0307fcc84
commit 2b439e8a5f
15 changed files with 870 additions and 40 deletions

View File

@@ -0,0 +1,120 @@
# ADR-0014: Relay outbound client for NAT/internet pipeline hops
## Status: Accepted
## Context
ADR-0010 describes the relay server: a public WebSocket hub where nodes behind NAT
connect outbound and register as reachable peers. That ADR focused on the *inbound*
side: how the tracker reaches a behind-NAT node for the initial chat request.
The *pipeline hop* problem is different: when node A has the head shard and node B
(behind NAT) has the tail shard, node A must forward binary activations to node B
for *every generated token*. Direct HTTP from A to B is blocked. The relay must
carry this per-hop activation traffic.
### Why this is harder than tracker → node
The tracker-to-node relay (ADR-0010) proxies a single JSON request. The activation
hop carries raw bfloat16 tensors — binary data that must survive round-tripping
through the relay's JSON message envelope without precision loss.
Also, the relay `/rpc/{peer_id}` endpoint (one WebSocket connection per request)
must be opened and closed for every token in the autoregressive loop. Latency
of connection setup matters.
## Options considered
**A. Relay hop (WebSocket per hop, chosen)**
Node A opens a WebSocket to `wss://relay/rpc/{peer_id_B}`, sends the activation,
receives the response, closes. The relay's `_handle_rpc` forwards it to B's persistent
connection via the existing `relay-http-request` envelope mechanism.
Pros: reuses the existing relay server unchanged. Each hop is independent; failures don't
affect other requests.
Cons: WebSocket connection setup adds ~50150 ms per hop on a fast relay. For
autoregressive inference (N tokens × M hops), this adds up.
**B. Persistent per-session tunnel**
Node A opens a persistent WebSocket to the relay for the duration of an inference session
and multiplexes all token hops over it.
Pros: amortises connection setup across tokens.
Cons: requires session-level state on the relay; complicates relay shutdown/failover;
the current relay is stateless by design. Deferred for a future optimization.
**C. Tracker-proxied activations**
Route all activation traffic through the tracker's HTTP proxy.
Cons: the tracker is the control plane, not the data plane. High-volume binary tensor
traffic through the tracker would saturate it. Rejected.
## Decision
Option A — per-hop WebSocket relay. Simple, reuses existing infrastructure, correct.
Option B is noted as a future optimization when activation-path latency becomes the
bottleneck.
## Protocol
```
Node A opens WS → wss://relay/rpc/{peer_id_B}
Node A sends:
{
"request_id": "<hex>",
"method": "POST",
"path": "/forward",
"headers": { "X-Meshnet-Shape": "...", "X-Meshnet-Start-Layer": "12", ... },
"body_base64": "<base64(bfloat16 tensor)>"
}
Relay forwards to Node B as relay-http-request envelope.
Node B's RelayHttpBridge decodes body_base64, calls POST /forward locally.
Response:
{
"request_id": "<hex>",
"status": 200,
"headers": { "x-meshnet-shape": "...", "content-type": "application/octet-stream" },
"body_base64": "<base64(output tensor)>" ← for binary responses
# OR
"body": "<json string>" ← for text (last-hop decode)
}
Relay sends response JSON back to Node A.
Node A decodes body_base64, continues pipeline.
```
### Binary data through JSON: base64
Raw bfloat16 bytes cannot safely transit JSON (no UTF-8 guarantee, lossy decode).
`body_base64` carries the tensor as base64; the bridge decodes it before calling
the local HTTP endpoint, and re-encodes the response. No precision loss.
Text responses (final hop, `application/json`) use `body` (plain string) for efficiency.
### Fallback
If `_relay_hop` raises (relay unreachable, peer disconnected), `_run_downstream_pipeline`
logs a warning and retries via direct HTTP. If both fail, the hop returns a pipeline error
string and the token is skipped.
### Tracker injection
The tracker's `_handle_proxy_chat` includes `relay_addr` in each downstream hop dict
when the node has one registered:
```json
{"endpoint": "http://172.29.x.x:7002", "start_layer": 12, "relay_addr": "wss://relay/rpc/abc123"}
```
The head node reads `relay_addr` from the injected `X-Meshnet-Route` header and calls
`_relay_hop` instead of direct HTTP.
## Consequences
- Nodes behind NAT (WSL2, 5G, home routers) can now participate in distributed pipeline inference without opening firewall ports
- `relay_addr` is a stable registration field; nodes without a relay omit it and receive direct HTTP hops
- Per-hop WebSocket setup adds latency proportional to relay RTT; acceptable for prototype, optimize later with persistent tunnels
- Base64 encoding increases payload size by ~33%; acceptable for prototype
- The relay server remains stateless and horizontally scalable; only the persistent per-peer `/ws` connections are stateful