perf: tracker injects pre-resolved route; node skips redundant tracker query
When the tracker proxies /v1/chat/completions to a first-shard node it already holds the full registry picture. It now resolves the downstream route inline via _select_route, strips the proxied node, and sends the result as X-Meshnet-Route header alongside the request body. The first-shard node reads this header in _get_remaining_route and returns it directly, skipping the second tracker HTTP call entirely. Falls back to the tracker query transparently when the header is absent (direct node-to-node calls or older tracker versions). Reduces per-inference tracker round-trips from 2 to 1. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -606,10 +606,33 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
|
||||
node = candidates[int(time.time() * 1000) % len(candidates)]
|
||||
target_url = f"{node.endpoint}/v1/chat/completions"
|
||||
|
||||
# Pre-resolve the downstream route so the first-shard node skips its own
|
||||
# tracker query. We already hold the full registry picture — no need for
|
||||
# a second round-trip.
|
||||
route_model = node.hf_repo or node.model or model
|
||||
with server.lock:
|
||||
if route_model in server.model_presets:
|
||||
preset = server.model_presets[route_model]
|
||||
rs, re = _preset_layer_bounds(preset)
|
||||
all_nodes: list = [n for n in server.registry.values() if n.model == route_model]
|
||||
else:
|
||||
all_nodes = [
|
||||
n for n in server.registry.values()
|
||||
if (n.hf_repo == route_model or n.model == route_model)
|
||||
and n.shard_start is not None and n.num_layers is not None
|
||||
]
|
||||
rs, re = 0, (max((n.num_layers for n in all_nodes), default=1) - 1)
|
||||
route_nodes, _ = _select_route(all_nodes, rs, re)
|
||||
# Strip the first-shard node we're about to proxy to — it's already handling the request.
|
||||
downstream_urls = json.dumps([n.endpoint for n in route_nodes if n.endpoint != node.endpoint])
|
||||
|
||||
req = urllib.request.Request(
|
||||
target_url,
|
||||
data=raw_body,
|
||||
headers={"Content-Type": "application/json"},
|
||||
headers={
|
||||
"Content-Type": "application/json",
|
||||
"X-Meshnet-Route": downstream_urls,
|
||||
},
|
||||
method="POST",
|
||||
)
|
||||
# Copy Authorization header from client if present
|
||||
|
||||
Reference in New Issue
Block a user