34 lines
1.4 KiB
Markdown
34 lines
1.4 KiB
Markdown
Status: ready-for-agent
|
|
|
|
# 08 — End-to-end distributed performance gate
|
|
|
|
## What to build
|
|
|
|
Turn the benchmark and optimizations into a repeatable performance gate for a
|
|
small two-node route and a relay route. Compare stateless legacy mode, cached
|
|
decode, direct HTTP, and persistent relay. Fail only on stable regressions and
|
|
publish the measurements needed to decide whether further work belongs in
|
|
transport, serialization, queueing, or model execution.
|
|
|
|
## Acceptance criteria
|
|
|
|
- [ ] CI/local benchmark runs a deterministic fixed-token scenario without a
|
|
real model or external network.
|
|
- [ ] The report compares tokens/sec, p50/p95 token latency, seam latency,
|
|
bytes/token, connection count, compression CPU, and peak buffered bytes.
|
|
- [ ] Thresholds are documented and tolerant of normal host variance while
|
|
catching a meaningful regression.
|
|
- [ ] A real-model opt-in command records the same metrics for LAN validation.
|
|
- [ ] The gate verifies output token identity, Route Session stability, and
|
|
cleanup of sessions, sockets, queues, and telemetry state.
|
|
|
|
## Blocked by
|
|
|
|
- 02 — Persistent relay compatibility hardening.
|
|
- 03 — Direct and bridge HTTP keep-alive.
|
|
- 04 — Activation Seam telemetry and bounded progress reporting.
|
|
- 05 — Trace-driven activation compression.
|
|
- 06 — Activation framing and copy reduction.
|
|
- 07 — Bounded prefill chunk backpressure.
|
|
|