perf: tracker injects pre-resolved route; node skips redundant tracker query

When the tracker proxies /v1/chat/completions to a first-shard node it
already holds the full registry picture. It now resolves the downstream
route inline via _select_route, strips the proxied node, and sends the
result as X-Meshnet-Route header alongside the request body.

The first-shard node reads this header in _get_remaining_route and
returns it directly, skipping the second tracker HTTP call entirely.
Falls back to the tracker query transparently when the header is absent
(direct node-to-node calls or older tracker versions).

Reduces per-inference tracker round-trips from 2 to 1.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Dobromir Popov
2026-06-30 10:44:18 +03:00
parent ae5ff6a805
commit 4a803377dc
2 changed files with 35 additions and 4 deletions

View File

@@ -606,10 +606,33 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
node = candidates[int(time.time() * 1000) % len(candidates)]
target_url = f"{node.endpoint}/v1/chat/completions"
# Pre-resolve the downstream route so the first-shard node skips its own
# tracker query. We already hold the full registry picture — no need for
# a second round-trip.
route_model = node.hf_repo or node.model or model
with server.lock:
if route_model in server.model_presets:
preset = server.model_presets[route_model]
rs, re = _preset_layer_bounds(preset)
all_nodes: list = [n for n in server.registry.values() if n.model == route_model]
else:
all_nodes = [
n for n in server.registry.values()
if (n.hf_repo == route_model or n.model == route_model)
and n.shard_start is not None and n.num_layers is not None
]
rs, re = 0, (max((n.num_layers for n in all_nodes), default=1) - 1)
route_nodes, _ = _select_route(all_nodes, rs, re)
# Strip the first-shard node we're about to proxy to — it's already handling the request.
downstream_urls = json.dumps([n.endpoint for n in route_nodes if n.endpoint != node.endpoint])
req = urllib.request.Request(
target_url,
data=raw_body,
headers={"Content-Type": "application/json"},
headers={
"Content-Type": "application/json",
"X-Meshnet-Route": downstream_urls,
},
method="POST",
)
# Copy Authorization header from client if present