neuron-tai/.scratch/distributed-inference-performance/issues/08-end-to-end-performance-gate.md

Status: ready-for-agent

08 — End-to-end distributed performance gate

What to build

Turn the benchmark and optimizations into a repeatable performance gate for a small two-node route and a relay route. Compare stateless legacy mode, cached decode, direct HTTP, and persistent relay. Fail only on stable regressions and publish the measurements needed to decide whether further work belongs in transport, serialization, queueing, or model execution.

Acceptance criteria

• CI/local benchmark runs a deterministic fixed-token scenario without a real model or external network.
• The report compares tokens/sec, p50/p95 token latency, seam latency, bytes/token, connection count, compression CPU, and peak buffered bytes.
• Thresholds are documented and tolerant of normal host variance while catching a meaningful regression.
• A real-model opt-in command records the same metrics for LAN validation.
• The gate verifies output token identity, Route Session stability, and cleanup of sessions, sockets, queues, and telemetry state.

Blocked by

• 02 — Persistent relay compatibility hardening.
• 03 — Direct and bridge HTTP keep-alive.
• 04 — Activation Seam telemetry and bounded progress reporting.
• 05 — Trace-driven activation compression.
• 06 — Activation framing and copy reduction.
• 07 — Bounded prefill chunk backpressure.