US-036 — Streamed chat completions over the relay RPC path

Status: planned Priority: Critical (blocks public friends-test deployment) Stage: Designed

Context

With the tracker deployed on a public VPS (cloud.neuron.d-popov.com), every node is behind NAT, so every chat request is proxied tracker → relay → head node via _relay_http_request (packages/tracker/meshnet_tracker/server.py). That function does a single blocking ws.recv() and the node's RelayHttpBridge._handle_request does a single resp.read(). Two consequences for stream: true requests:

No live streaming — the client sees nothing until generation completes, then receives the entire SSE body at once with a Content-Length header.
Zero billing — the tracker runs _billable_non_stream_tokens(json.loads(body)) on the buffered body; SSE text is not JSON, the parse fails, and the request is billed/credited as 0 tokens. Off-LAN this silently zeroes out all streamed-request accounting.

Decision (grilled 2026-07-06): implement true multi-frame streaming over the relay RPC WebSocket, scoped to the tracker → head-node leg. Rejected alternatives: billing-only SSE parse (fragile heuristic, blank-screen UX stays) and forcing stream:false over the relay (exact billing but still no live tokens for testers). Streaming through the existing SSE accounting loop fixes both symptoms with one mechanism. Inter-node /forward activation hops stay single-frame (ADR-0014) — they are one-tensor-in/one-tensor-out and gain nothing from chunking.

Protocol

A relayed response becomes a sequence of relay-http-response envelopes sharing one request_id:

// first frame — status + headers, opens the stream
{"request_id": "<id>", "status": 200, "headers": {"Content-Type": "text/event-stream"},
 "stream": true, "chunk": "data: {...}\n\n", "done": false}
// zero or more continuation frames
{"request_id": "<id>", "stream": true, "chunk": "data: {...}\n\n", "done": false}
// terminal frame
{"request_id": "<id>", "stream": true, "done": true}

Backward compatibility: a frame with no stream key is a complete single response (today's format, still used for /forward hops, non-SSE responses, and older nodes). The relay and tracker treat it as terminal.

Per component

Node bridge (packages/node/meshnet_node/relay_bridge.py): when the local response Content-Type is text/event-stream, read line-by-line and emit chunk frames as lines arrive; otherwise keep the existing single-frame path (including body_base64 for binary). Frame sends go through the bridge's WS send lock (US-037) so frames from concurrent requests interleave whole, never torn.
Relay server (packages/relay/meshnet_relay/server.py): _handle_rpc replaces the single asyncio.Future in _pending_rpc with a per-request asyncio.Queue. Frames arriving on the target peer's gossip connection are routed by request_id and forwarded to the requester WS until done (or a terminal legacy frame). Timeouts: keep the 310 s overall cap; add a 120 s per-frame idle timeout so a dead node doesn't pin the queue.
Tracker (packages/tracker/meshnet_tracker/server.py): _relay_http_request grows a streaming mode — loop ws.recv(); on the first frame send status/headers to the client; write each chunk to the client immediately and feed it through the same SSE token-accounting used by the direct-proxy stream loop (reported_stream_tokens / _record_observed_throughput / _bill_completed), so relayed streams bill identically to direct streams. Non-stream frames keep the current buffered handling.

Known limitation (accepted for alpha)

If the client disconnects mid-stream, the relay drops undeliverable frames but the node keeps generating until completion — wasted compute bounded by one generation. Cancellation propagation is future work.

Acceptance criteria

stream: true chat request via relay delivers SSE chunks to the client incrementally (test observes ≥2 distinct frame arrivals before [DONE])
Relayed streamed request records nonzero billed tokens and node work credit
Non-streamed relayed requests and /forward binary hops behave exactly as before (single frame, body_base64 round-trip intact)
Legacy single-frame response from an old node is accepted as terminal
Idle stream (no frame for 120 s) returns 504 to the client and cleans up the relay-side queue
Extend tests/test_gossip_and_relay.py alongside test_relay_rpc_round_trips_http_request_to_peer
python -m pytest passes from repo root

4.5 KiB Raw Permalink Blame History