perf: tracker injects pre-resolved route; node skips redundant tracker query

When the tracker proxies /v1/chat/completions to a first-shard node it
already holds the full registry picture. It now resolves the downstream
route inline via _select_route, strips the proxied node, and sends the
result as X-Meshnet-Route header alongside the request body.

The first-shard node reads this header in _get_remaining_route and
returns it directly, skipping the second tracker HTTP call entirely.
Falls back to the tracker query transparently when the header is absent
(direct node-to-node calls or older tracker versions).

Reduces per-inference tracker round-trips from 2 to 1.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Dobromir Popov
2026-06-30 10:44:18 +03:00
parent ae5ff6a805
commit 4a803377dc
2 changed files with 35 additions and 4 deletions

View File

@@ -294,18 +294,26 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
self._send_openai_response(result_text, model_name, stream, messages)
def _get_remaining_route(self, model: str) -> list[str]:
# Fast path: tracker pre-resolved the downstream route and injected it as a header.
injected = self.headers.get("X-Meshnet-Route")
if injected:
try:
route = json.loads(injected)
if isinstance(route, list):
return [str(ep) for ep in route]
except (json.JSONDecodeError, TypeError):
pass
# Slow path: query the tracker (direct node-to-node calls, or tracker didn't inject).
server: _TorchHTTPServer = self.server # type: ignore[assignment]
if server.tracker_url is None:
return []
# Use the backend's actual hf_repo, not the client-provided model name (which may be
# a lowercased or abbreviated alias that doesn't match what the tracker registered).
route_model = getattr(server.backend, "model_id", None) or model
try:
url = f"{server.tracker_url}/v1/route?model={urllib.parse.quote(route_model)}"
with urllib.request.urlopen(url, timeout=5.0) as r:
route_resp = json.loads(r.read())
route = route_resp.get("route", [])
# Skip our own endpoint from the route (match by port so host aliases don't matter).
own_port = server.server_address[1]
return [ep for ep in route if not ep.rstrip("/").endswith(f":{own_port}")]
except Exception as exc: