89 lines
4.5 KiB
Markdown
89 lines
4.5 KiB
Markdown
# US-036 — Streamed chat completions over the relay RPC path
|
|
|
|
Status: planned
|
|
Priority: Critical (blocks public friends-test deployment)
|
|
Stage: Designed
|
|
|
|
## Context
|
|
|
|
With the tracker deployed on a public VPS (`cloud.neuron.d-popov.com`), every node is
|
|
behind NAT, so **every** chat request is proxied tracker → relay → head node via
|
|
`_relay_http_request` (`packages/tracker/meshnet_tracker/server.py`). That function does a
|
|
single blocking `ws.recv()` and the node's `RelayHttpBridge._handle_request` does a single
|
|
`resp.read()`. Two consequences for `stream: true` requests:
|
|
|
|
1. **No live streaming** — the client sees nothing until generation completes, then
|
|
receives the entire SSE body at once with a `Content-Length` header.
|
|
2. **Zero billing** — the tracker runs `_billable_non_stream_tokens(json.loads(body))`
|
|
on the buffered body; SSE text is not JSON, the parse fails, and the request is
|
|
billed/credited as 0 tokens. Off-LAN this silently zeroes out *all* streamed-request
|
|
accounting.
|
|
|
|
Decision (grilled 2026-07-06): implement **true multi-frame streaming** over the relay
|
|
RPC WebSocket, scoped to the tracker → head-node leg. Rejected alternatives:
|
|
billing-only SSE parse (fragile heuristic, blank-screen UX stays) and forcing
|
|
`stream:false` over the relay (exact billing but still no live tokens for testers).
|
|
Streaming through the existing SSE accounting loop fixes both symptoms with one
|
|
mechanism. Inter-node `/forward` activation hops stay single-frame (ADR-0014) — they
|
|
are one-tensor-in/one-tensor-out and gain nothing from chunking.
|
|
|
|
## Protocol
|
|
|
|
A relayed response becomes a sequence of `relay-http-response` envelopes sharing one
|
|
`request_id`:
|
|
|
|
```json
|
|
// first frame — status + headers, opens the stream
|
|
{"request_id": "<id>", "status": 200, "headers": {"Content-Type": "text/event-stream"},
|
|
"stream": true, "chunk": "data: {...}\n\n", "done": false}
|
|
// zero or more continuation frames
|
|
{"request_id": "<id>", "stream": true, "chunk": "data: {...}\n\n", "done": false}
|
|
// terminal frame
|
|
{"request_id": "<id>", "stream": true, "done": true}
|
|
```
|
|
|
|
**Backward compatibility:** a frame with no `stream` key is a complete single response
|
|
(today's format, still used for `/forward` hops, non-SSE responses, and older nodes).
|
|
The relay and tracker treat it as terminal.
|
|
|
|
### Per component
|
|
|
|
- **Node bridge** (`packages/node/meshnet_node/relay_bridge.py`): when the local
|
|
response `Content-Type` is `text/event-stream`, read line-by-line and emit chunk
|
|
frames as lines arrive; otherwise keep the existing single-frame path (including
|
|
`body_base64` for binary). Frame sends go through the bridge's WS send lock
|
|
(US-037) so frames from concurrent requests interleave whole, never torn.
|
|
- **Relay server** (`packages/relay/meshnet_relay/server.py`): `_handle_rpc` replaces
|
|
the single `asyncio.Future` in `_pending_rpc` with a per-request `asyncio.Queue`.
|
|
Frames arriving on the target peer's gossip connection are routed by `request_id`
|
|
and forwarded to the requester WS until `done` (or a terminal legacy frame).
|
|
Timeouts: keep the 310 s overall cap; add a 120 s per-frame idle timeout so a dead
|
|
node doesn't pin the queue.
|
|
- **Tracker** (`packages/tracker/meshnet_tracker/server.py`): `_relay_http_request`
|
|
grows a streaming mode — loop `ws.recv()`; on the first frame send status/headers to
|
|
the client; write each `chunk` to the client immediately and feed it through the
|
|
same SSE token-accounting used by the direct-proxy stream loop
|
|
(`reported_stream_tokens` / `_record_observed_throughput` / `_bill_completed`), so
|
|
relayed streams bill identically to direct streams. Non-stream frames keep the
|
|
current buffered handling.
|
|
|
|
### Known limitation (accepted for alpha)
|
|
|
|
If the client disconnects mid-stream, the relay drops undeliverable frames but the
|
|
node keeps generating until completion — wasted compute bounded by one generation.
|
|
Cancellation propagation is future work.
|
|
|
|
## Acceptance criteria
|
|
|
|
- `stream: true` chat request via relay delivers SSE chunks to the client
|
|
incrementally (test observes ≥2 distinct frame arrivals before `[DONE]`)
|
|
- Relayed streamed request records nonzero billed tokens and node work credit
|
|
- Non-streamed relayed requests and `/forward` binary hops behave exactly as before
|
|
(single frame, `body_base64` round-trip intact)
|
|
- Legacy single-frame response from an old node is accepted as terminal
|
|
- Idle stream (no frame for 120 s) returns 504 to the client and cleans up the
|
|
relay-side queue
|
|
- Extend `tests/test_gossip_and_relay.py` alongside
|
|
`test_relay_rpc_round_trips_http_request_to_peer`
|
|
- `python -m pytest` passes from repo root
|