# US-036 — Streamed chat completions over the relay RPC path

Status: done (implemented — `_stream_relayed_frames` in `server.py`; verify on public NAT relay before friends-test)

## Context

With the tracker deployed on a public VPS (`cloud.neuron.d-popov.com`), every node is
behind NAT, so **every** chat request is proxied tracker → relay → head node via
`_relay_http_request` (`packages/tracker/meshnet_tracker/server.py`). That function does a
single blocking `ws.recv()` and the node's `RelayHttpBridge._handle_request` does a single
`resp.read()`. Two consequences for `stream: true` requests:

1. **No live streaming** — the client sees nothing until generation completes, then
   receives the entire SSE body at once with a `Content-Length` header.
2. **Zero billing** — the tracker runs `_billable_non_stream_tokens(json.loads(body))`
   on the buffered body; SSE text is not JSON, the parse fails, and the request is
   billed/credited as 0 tokens. Off-LAN this silently zeroes out *all* streamed-request
   accounting.

Decision (grilled 2026-07-06): implement **true multi-frame streaming** over the relay
RPC WebSocket, scoped to the tracker → head-node leg. Rejected alternatives:
billing-only SSE parse (fragile heuristic, blank-screen UX stays) and forcing
`stream:false` over the relay (exact billing but still no live tokens for testers).
Streaming through the existing SSE accounting loop fixes both symptoms with one
mechanism. Inter-node `/forward` activation hops stay single-frame (ADR-0014) — they
are one-tensor-in/one-tensor-out and gain nothing from chunking.

## Protocol

A relayed response becomes a sequence of `relay-http-response` envelopes sharing one
`request_id`:

```json
// first frame — status + headers, opens the stream
{"request_id": "<id>", "status": 200, "headers": {"Content-Type": "text/event-stream"},
 "stream": true, "chunk": "data: {...}\n\n", "done": false}
// zero or more continuation frames
{"request_id": "<id>", "stream": true, "chunk": "data: {...}\n\n", "done": false}
// terminal frame
{"request_id": "<id>", "stream": true, "done": true}
```

**Backward compatibility:** a frame with no `stream` key is a complete single response
(today's format, still used for `/forward` hops, non-SSE responses, and older nodes).
The relay and tracker treat it as terminal.

### Per component

- **Node bridge** (`packages/node/meshnet_node/relay_bridge.py`): when the local
  response `Content-Type` is `text/event-stream`, read line-by-line and emit chunk
  frames as lines arrive; otherwise keep the existing single-frame path (including
  `body_base64` for binary). Frame sends go through the bridge's WS send lock
  (US-037) so frames from concurrent requests interleave whole, never torn.
- **Relay server** (`packages/relay/meshnet_relay/server.py`): `_handle_rpc` replaces
  the single `asyncio.Future` in `_pending_rpc` with a per-request `asyncio.Queue`.
  Frames arriving on the target peer's gossip connection are routed by `request_id`
  and forwarded to the requester WS until `done` (or a terminal legacy frame).
  Timeouts: keep the 310 s overall cap; add a 120 s per-frame idle timeout so a dead
  node doesn't pin the queue.
- **Tracker** (`packages/tracker/meshnet_tracker/server.py`): `_relay_http_request`
  grows a streaming mode — loop `ws.recv()`; on the first frame send status/headers to
  the client; write each `chunk` to the client immediately and feed it through the
  same SSE token-accounting used by the direct-proxy stream loop
  (`reported_stream_tokens` / `_record_observed_throughput` / `_bill_completed`), so
  relayed streams bill identically to direct streams. Non-stream frames keep the
  current buffered handling.

### Known limitation (accepted for alpha)

If the client disconnects mid-stream, the relay drops undeliverable frames but the
node keeps generating until completion — wasted compute bounded by one generation.
Cancellation propagation is future work.

## Acceptance criteria

- `stream: true` chat request via relay delivers SSE chunks to the client
  incrementally (test observes ≥2 distinct frame arrivals before `[DONE]`)
- Relayed streamed request records nonzero billed tokens and node work credit
- Non-streamed relayed requests and `/forward` binary hops behave exactly as before
  (single frame, `body_base64` round-trip intact)
- Legacy single-frame response from an old node is accepted as terminal
- Idle stream (no frame for 120 s) returns 504 to the client and cleans up the
  relay-side queue
- Extend `tests/test_gossip_and_relay.py` alongside
  `test_relay_rpc_round_trips_http_request_to_peer`
- `python -m pytest` passes from repo root