52 lines
2.7 KiB
Markdown
52 lines
2.7 KiB
Markdown
# US-037 — Concurrent request handling in the node relay bridge
|
|
|
|
Status: planned
|
|
Priority: Critical (blocks public friends-test deployment)
|
|
Stage: Designed
|
|
|
|
## Context
|
|
|
|
`RelayHttpBridge._run` (`packages/node/meshnet_node/relay_bridge.py`) handles
|
|
`relay-http-request` envelopes serially inside its recv loop: `_handle_request` blocks
|
|
on `urllib.request.urlopen(..., timeout=300)` before the next envelope is read.
|
|
|
|
Off-LAN this is a correctness bug, not just a throughput limit: a node can be the
|
|
**head** of one inference route and a **downstream hop** of another at the same time.
|
|
While the head request occupies the bridge (up to 300 s of generation), the other
|
|
route's per-token `/forward` calls sit unread in the WebSocket buffer — overlapping
|
|
routes through a shared node are effectively broken.
|
|
|
|
Decision (grilled 2026-07-06): dispatch requests on a **bounded worker pool, default
|
|
8, configurable**. Rejected alternatives: unbounded thread-per-request (a public
|
|
deployment exposes volunteer machines to request stampedes) and reject-when-full
|
|
(bouncing `/forward` hops kills other routes' in-flight sessions; queueing beyond the
|
|
cap is today's behavior, just 8-wide).
|
|
|
|
## Design
|
|
|
|
- Recv loop only parses envelopes and submits to a
|
|
`ThreadPoolExecutor(max_workers=N)`; workers run `_handle_request` and send the
|
|
response frame(s).
|
|
- All sends on the single relay WebSocket go through a `threading.Lock`. With US-036
|
|
a response may be multiple frames — the lock is held **per frame**, so streams from
|
|
concurrent requests interleave frame-atomically (receivers demux by `request_id`).
|
|
- `N` defaults to 8; configurable via `meshnet-node start --relay-concurrency N`
|
|
(env `MESHNET_RELAY_CONCURRENCY`). Requests beyond `N` queue in the executor.
|
|
- On reconnect (`_run`'s outer loop), in-flight workers from the dead connection may
|
|
still try to send; the send helper swallows failures on a closed socket, and the
|
|
relay side times the orphaned request out (US-036 idle timeout / existing 310 s cap).
|
|
- `stop()` shuts the executor down without waiting for stragglers (daemon threads,
|
|
same as today's bridge thread).
|
|
|
|
## Acceptance criteria
|
|
|
|
- While one relayed request is in flight (slow local handler), a second
|
|
`relay-http-request` to the same node completes without waiting for the first
|
|
- Responses are correctly matched by `request_id` when they complete out of order
|
|
- More than `N` simultaneous requests queue and all eventually complete; thread count
|
|
never exceeds `N` workers
|
|
- Bridge survives a relay reconnect with workers still in flight (no crash, no
|
|
deadlock; orphaned responses dropped)
|
|
- Extend `tests/test_gossip_and_relay.py`
|
|
- `python -m pytest` passes from repo root
|