neuron-tai/docs/issues/37-relay-bridge-concurrency.md

US-037 — Concurrent request handling in the node relay bridge

Status: planned Priority: Critical (blocks public friends-test deployment) Stage: Designed

Context

RelayHttpBridge._run (packages/node/meshnet_node/relay_bridge.py) handles relay-http-request envelopes serially inside its recv loop: _handle_request blocks on urllib.request.urlopen(..., timeout=300) before the next envelope is read.

Off-LAN this is a correctness bug, not just a throughput limit: a node can be the head of one inference route and a downstream hop of another at the same time. While the head request occupies the bridge (up to 300 s of generation), the other route's per-token /forward calls sit unread in the WebSocket buffer — overlapping routes through a shared node are effectively broken.

Decision (grilled 2026-07-06): dispatch requests on a bounded worker pool, default 8, configurable. Rejected alternatives: unbounded thread-per-request (a public deployment exposes volunteer machines to request stampedes) and reject-when-full (bouncing /forward hops kills other routes' in-flight sessions; queueing beyond the cap is today's behavior, just 8-wide).

Design

• Recv loop only parses envelopes and submits to a ThreadPoolExecutor(max_workers=N); workers run _handle_request and send the response frame(s).
• All sends on the single relay WebSocket go through a threading.Lock. With US-036 a response may be multiple frames — the lock is held per frame, so streams from concurrent requests interleave frame-atomically (receivers demux by request_id).
• N defaults to 8; configurable via meshnet-node start --relay-concurrency N (env MESHNET_RELAY_CONCURRENCY). Requests beyond N queue in the executor.
• On reconnect (_run's outer loop), in-flight workers from the dead connection may still try to send; the send helper swallows failures on a closed socket, and the relay side times the orphaned request out (US-036 idle timeout / existing 310 s cap).
• stop() shuts the executor down without waiting for stragglers (daemon threads, same as today's bridge thread).

Acceptance criteria

• While one relayed request is in flight (slow local handler), a second relay-http-request to the same node completes without waiting for the first
• Responses are correctly matched by request_id when they complete out of order
• More than N simultaneous requests queue and all eventually complete; thread count never exceeds N workers
• Bridge survives a relay reconnect with workers still in flight (no crash, no deadlock; orphaned responses dropped)
• Extend tests/test_gossip_and_relay.py
• python -m pytest passes from repo root