49 lines
2.7 KiB
Markdown
49 lines
2.7 KiB
Markdown
# US-020 — Manual route selection + hop-penalty benchmarking
|
|
|
|
## Context
|
|
|
|
The tracker auto-selects inference routes based on synthetic benchmark scores. To measure
|
|
the real cost of adding hops (latency per node boundary), we need:
|
|
|
|
1. A way to pin a request to a specific route so we control the variable.
|
|
2. A benchmark endpoint that runs the same prompt through 1-node, 2-node, and 3-node
|
|
routes and records per-hop latency.
|
|
|
|
Results are stored to disk. Routing algorithm is **not** changed in this story — this is
|
|
data collection only. The data will inform a future routing optimisation story.
|
|
|
|
## Design decisions (grilled 2026-07-01)
|
|
|
|
| Decision | Choice |
|
|
|---|---|
|
|
| Route spec | Optional `route` field in JSON request body (list of node IDs) |
|
|
| Trigger | Explicit only — `POST /v1/benchmark/hop-penalty` endpoint |
|
|
| Auth | Header-presence stub (`Authorization` must be non-empty); real auth in future story |
|
|
| Routing integration | Store data only; routing algorithm unchanged |
|
|
| Persistence | Append to `benchmark_results.json` in tracker working dir; in-memory queryable |
|
|
|
|
## Acceptance criteria
|
|
|
|
- `POST /v1/chat/completions` accepts optional `"route": ["<node_id>", ...]` in the
|
|
request body. If present, the tracker uses those nodes in order instead of auto-selecting.
|
|
If absent, existing routing is unchanged (no breaking change for unaware clients).
|
|
- Missing or invalid node IDs in `route` return HTTP 400 with a descriptive error.
|
|
- `POST /v1/benchmark/hop-penalty` is auth-gated: requests without a non-empty
|
|
`Authorization` header return HTTP 401. Body: `{"model": "...", "prompt": "...",
|
|
"max_new_tokens": 64}`.
|
|
- Benchmark fans out to up to three routes: 1-node (single node covering all layers),
|
|
2-node (two consecutive shard nodes), 3-node (three nodes) — using whatever is
|
|
currently registered. Routes with insufficient coverage are skipped, not errored.
|
|
- Response includes per-route breakdown: `total_ms`, `per_hop_ms: [...]`,
|
|
`tokens_generated`, `route: [node_id, ...]`.
|
|
- Results are appended to `<tracker_working_dir>/benchmark_results.json` (created if
|
|
absent) as a JSON array. Each entry includes timestamp, model, prompt hash, and the
|
|
per-route breakdown.
|
|
- `GET /v1/benchmark/results` returns the stored results array. Also auth-gated.
|
|
- Clients that never send `route` or call `/v1/benchmark/*` are completely unaffected.
|
|
- Integration test: send the same prompt via a pinned 1-node route and a pinned 2-node
|
|
route; assert 2-node result has 2 entries in `per_hop_ms`; assert both records appear
|
|
in `benchmark_results.json`.
|
|
- `python -m pytest` passes from repo root.
|
|
- Commit only this story's changes.
|