perf: tracker injects pre-resolved route; node skips redundant tracker query
When the tracker proxies /v1/chat/completions to a first-shard node it already holds the full registry picture. It now resolves the downstream route inline via _select_route, strips the proxied node, and sends the result as X-Meshnet-Route header alongside the request body. The first-shard node reads this header in _get_remaining_route and returns it directly, skipping the second tracker HTTP call entirely. Falls back to the tracker query transparently when the header is absent (direct node-to-node calls or older tracker versions). Reduces per-inference tracker round-trips from 2 to 1. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -294,18 +294,26 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
|
||||
self._send_openai_response(result_text, model_name, stream, messages)
|
||||
|
||||
def _get_remaining_route(self, model: str) -> list[str]:
|
||||
# Fast path: tracker pre-resolved the downstream route and injected it as a header.
|
||||
injected = self.headers.get("X-Meshnet-Route")
|
||||
if injected:
|
||||
try:
|
||||
route = json.loads(injected)
|
||||
if isinstance(route, list):
|
||||
return [str(ep) for ep in route]
|
||||
except (json.JSONDecodeError, TypeError):
|
||||
pass
|
||||
|
||||
# Slow path: query the tracker (direct node-to-node calls, or tracker didn't inject).
|
||||
server: _TorchHTTPServer = self.server # type: ignore[assignment]
|
||||
if server.tracker_url is None:
|
||||
return []
|
||||
# Use the backend's actual hf_repo, not the client-provided model name (which may be
|
||||
# a lowercased or abbreviated alias that doesn't match what the tracker registered).
|
||||
route_model = getattr(server.backend, "model_id", None) or model
|
||||
try:
|
||||
url = f"{server.tracker_url}/v1/route?model={urllib.parse.quote(route_model)}"
|
||||
with urllib.request.urlopen(url, timeout=5.0) as r:
|
||||
route_resp = json.loads(r.read())
|
||||
route = route_resp.get("route", [])
|
||||
# Skip our own endpoint from the route (match by port so host aliases don't matter).
|
||||
own_port = server.server_address[1]
|
||||
return [ep for ep in route if not ep.rstrip("/").endswith(f":{own_port}")]
|
||||
except Exception as exc:
|
||||
|
||||
Reference in New Issue
Block a user