41 lines
1.7 KiB
Markdown
41 lines
1.7 KiB
Markdown
# PRD: Distributed inference performance
|
|
|
|
## Problem
|
|
|
|
Distributed decode already avoids full-prompt recomputation when the local KV
|
|
path is active, but each Activation Seam can still pay transport and data-plane
|
|
overhead for every generated token. Relay logs show a new `request_id` per
|
|
token; that is correct correlation, but the old relay implementation also
|
|
opened a new WebSocket per token. Direct hops and relay bridge forwarding use
|
|
fresh HTTP requests as well. Without timing and byte measurements, compression,
|
|
copy, and buffering choices cannot be ranked safely.
|
|
|
|
## Outcome
|
|
|
|
For a cached Route Session, connection setup is amortized across the session,
|
|
decode payloads remain one-step activations, progress reporting is bounded, and
|
|
the benchmark can attribute latency to model execution, serialization, relay,
|
|
HTTP, queueing, and backpressure. Optimizations must preserve output tokens,
|
|
KV semantics, failure behavior, and compatibility with legacy one-shot peers.
|
|
|
|
## Non-goals
|
|
|
|
- No speculative decoding or multi-token model execution in this feature.
|
|
- No QUIC/WebRTC/custom transport rewrite.
|
|
- No centralized Hot KV State.
|
|
- No silent reuse of a `request_id`; each activation remains independently
|
|
traceable.
|
|
|
|
## Acceptance criteria
|
|
|
|
- A reproducible local two-node and relay benchmark reports per-token and
|
|
per-seam timing plus bytes.
|
|
- Cached decode does not perform a new TCP/WebSocket connection per token.
|
|
- Direct and relay-to-local HTTP paths reuse connections safely or document why
|
|
a path cannot do so.
|
|
- Compression and copy decisions are based on recorded traces, not guesses.
|
|
- Slow prefill consumers apply bounded backpressure rather than unbounded body
|
|
buffering.
|
|
- A benchmark regression threshold catches a meaningful transport slowdown.
|
|
|