# PRD: Distributed inference performance

## Problem

Distributed decode already avoids full-prompt recomputation when the local KV
path is active, but each Activation Seam can still pay transport and data-plane
overhead for every generated token. Relay logs show a new `request_id` per
token; that is correct correlation, but the old relay implementation also
opened a new WebSocket per token. Direct hops and relay bridge forwarding use
fresh HTTP requests as well. Without timing and byte measurements, compression,
copy, and buffering choices cannot be ranked safely.

## Outcome

For a cached Route Session, connection setup is amortized across the session,
decode payloads remain one-step activations, progress reporting is bounded, and
the benchmark can attribute latency to model execution, serialization, relay,
HTTP, queueing, and backpressure. Optimizations must preserve output tokens,
KV semantics, failure behavior, and compatibility with legacy one-shot peers.

## Non-goals

- No speculative decoding or multi-token model execution in this feature.
- No QUIC/WebRTC/custom transport rewrite.
- No centralized Hot KV State.
- No silent reuse of a `request_id`; each activation remains independently
  traceable.

## Acceptance criteria

- A reproducible local two-node and relay benchmark reports per-token and
  per-seam timing plus bytes.
- Cached decode does not perform a new TCP/WebSocket connection per token.
- Direct and relay-to-local HTTP paths reuse connections safely or document why
  a path cannot do so.
- Compression and copy decisions are based on recorded traces, not guesses.
- Slow prefill consumers apply bounded backpressure rather than unbounded body
  buffering.
- A benchmark regression threshold catches a meaningful transport slowdown.