neuron-tai/docs/issues/30-manual-route-and-hop-benchmark.md

US-020 — Manual route selection + hop-penalty benchmarking

Context

The tracker auto-selects inference routes based on synthetic benchmark scores. To measure the real cost of adding hops (latency per node boundary), we need:

1. A way to pin a request to a specific route so we control the variable.
2. A benchmark endpoint that runs the same prompt through 1-node, 2-node, and 3-node routes and records per-hop latency.

Results are stored to disk. Routing algorithm is not changed in this story — this is data collection only. The data will inform a future routing optimisation story.

Design decisions (grilled 2026-07-01)

Decision	Choice
Route spec	Optional route field in JSON request body (list of node IDs)
Trigger	Explicit only — POST /v1/benchmark/hop-penalty endpoint
Auth	Header-presence stub (Authorization must be non-empty); real auth in future story
Routing integration	Store data only; routing algorithm unchanged
Persistence	Append to benchmark_results.json in tracker working dir; in-memory queryable

Acceptance criteria

• POST /v1/chat/completions accepts optional "route": ["<node_id>", ...] in the request body. If present, the tracker uses those nodes in order instead of auto-selecting. If absent, existing routing is unchanged (no breaking change for unaware clients).
• Missing or invalid node IDs in route return HTTP 400 with a descriptive error.
• POST /v1/benchmark/hop-penalty is auth-gated: requests without a non-empty Authorization header return HTTP 401. Body: {"model": "...", "prompt": "...", "max_new_tokens": 64}.
• Benchmark fans out to up to three routes: 1-node (single node covering all layers), 2-node (two consecutive shard nodes), 3-node (three nodes) — using whatever is currently registered. Routes with insufficient coverage are skipped, not errored.
• Response includes per-route breakdown: total_ms, per_hop_ms: [...], tokens_generated, route: [node_id, ...].
• Results are appended to <tracker_working_dir>/benchmark_results.json (created if absent) as a JSON array. Each entry includes timestamp, model, prompt hash, and the per-route breakdown.
• GET /v1/benchmark/results returns the stored results array. Also auth-gated.
• Clients that never send route or call /v1/benchmark/* are completely unaffected.
• Integration test: send the same prompt via a pinned 1-node route and a pinned 2-node route; assert 2-node result has 2 entries in per_hop_ms; assert both records appear in benchmark_results.json.
• python -m pytest passes from repo root.
• Commit only this story's changes.