neuron-tai/.scratch/distributed-inference-performance/issues/01-baseline-profiling-harness.md

Status: ready-for-agent

01 — Baseline and profiling harness

What to build

Create a deterministic stub-backed benchmark for a Route Session that measures prefill and cached decode across direct and relay paths. Attribute time to model execution, activation encoding/decoding, compression, connection setup, relay queueing, local HTTP forwarding, and end-to-end seam latency. Record payload sizes and connection counts without requiring a real model or external host.

Acceptance criteria

• The harness runs a fixed prompt and fixed generated-token count through a two-node route in direct and relay modes.
• It reports p50/p95 per-token latency, per-hop latency, payload bytes, compression ratio, connection attempts, and queue wait.
• It distinguishes prefill from decode and cached from stateless mode.
• It emits machine-readable JSON suitable for CI artifacts and a concise human-readable summary.
• A test fixture can assert connection attempts and output token identity.

Blocked by

None - can start immediately.