1.1 KiB
1.1 KiB
Status: done (2026-07-14)
01 — Baseline and profiling harness
What to build
Create a deterministic stub-backed benchmark for a Route Session that measures prefill and cached decode across direct and relay paths. Attribute time to model execution, activation encoding/decoding, compression, connection setup, relay queueing, local HTTP forwarding, and end-to-end seam latency. Record payload sizes and connection counts without requiring a real model or external host.
Acceptance criteria
- The harness runs a fixed prompt and fixed generated-token count through a two-node route in direct and relay modes.
- It reports p50/p95 per-token latency, per-hop latency, payload bytes, compression ratio, connection attempts, and queue wait.
- It distinguishes prefill from decode and cached from stateless mode.
- It emits machine-readable JSON suitable for CI artifacts and a concise human-readable summary.
- A test fixture can assert connection attempts and output token identity.
Blocked by
None - completed. Verified with PYTHONPATH=packages/node pytest -q tests/test_route_session_benchmark.py (7 passed).