560de08edd
Adds a committed .gitattributes so Windows and Linux checkouts converge on LF for all text files, overriding each developer's local core.autocrlf. Renormalizes existing blobs (server.py, dashboard.html, etc.) that had CRLF baked in, clearing the repo-wide phantom "modified" churn. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
49 lines
2.6 KiB
Markdown
49 lines
2.6 KiB
Markdown
# US-020 — Manual route selection + hop-penalty benchmarking
|
|
|
|
## Context
|
|
|
|
The tracker auto-selects inference routes based on synthetic benchmark scores. To measure
|
|
the real cost of adding hops (latency per node boundary), we need:
|
|
|
|
1. A way to pin a request to a specific route so we control the variable.
|
|
2. A benchmark endpoint that runs the same prompt through 1-node, 2-node, and 3-node
|
|
routes and records per-hop latency.
|
|
|
|
Results are stored to disk. Routing algorithm is **not** changed in this story — this is
|
|
data collection only. The data will inform a future routing optimisation story.
|
|
|
|
## Design decisions (grilled 2026-07-01)
|
|
|
|
| Decision | Choice |
|
|
|---|---|
|
|
| Route spec | Optional `route` field in JSON request body (list of node IDs) |
|
|
| Trigger | Explicit only — `POST /v1/benchmark/hop-penalty` endpoint |
|
|
| Auth | Header-presence stub (`Authorization` must be non-empty); real auth in future story |
|
|
| Routing integration | Store data only; routing algorithm unchanged |
|
|
| Persistence | Append to `benchmark_results.json` in tracker working dir; in-memory queryable |
|
|
|
|
## Acceptance criteria
|
|
|
|
- `POST /v1/chat/completions` accepts optional `"route": ["<node_id>", ...]` in the
|
|
request body. If present, the tracker uses those nodes in order instead of auto-selecting.
|
|
If absent, existing routing is unchanged (no breaking change for unaware clients).
|
|
- Missing or invalid node IDs in `route` return HTTP 400 with a descriptive error.
|
|
- `POST /v1/benchmark/hop-penalty` is auth-gated: requests without a non-empty
|
|
`Authorization` header return HTTP 401. Body: `{"model": "...", "prompt": "...",
|
|
"max_new_tokens": 64}`.
|
|
- Benchmark fans out to up to three routes: 1-node (single node covering all layers),
|
|
2-node (two consecutive shard nodes), 3-node (three nodes) — using whatever is
|
|
currently registered. Routes with insufficient coverage are skipped, not errored.
|
|
- Response includes per-route breakdown: `total_ms`, `per_hop_ms: [...]`,
|
|
`tokens_generated`, `route: [node_id, ...]`.
|
|
- Results are appended to `<tracker_working_dir>/benchmark_results.json` (created if
|
|
absent) as a JSON array. Each entry includes timestamp, model, prompt hash, and the
|
|
per-route breakdown.
|
|
- `GET /v1/benchmark/results` returns the stored results array. Also auth-gated.
|
|
- Clients that never send `route` or call `/v1/benchmark/*` are completely unaffected.
|
|
- Integration test: send the same prompt via a pinned 1-node route and a pinned 2-node
|
|
route; assert 2-node result has 2 entries in `per_hop_ms`; assert both records appear
|
|
in `benchmark_results.json`.
|
|
- `python -m pytest` passes from repo root.
|
|
- Commit only this story's changes.
|