# Distributed inference performance

Status: draft scratch package.

This feature measures and reduces avoidable overhead around the existing
Route Session, prefill/decode, local Hot KV State, and binary activation path.
It does not replace the distributed GGUF runtime plan. The goal is to make
transport and data movement cheap enough that model execution, rather than
connection setup, logging, or serialization, dominates token latency.

## Scope

- Baseline per-token compute, seam, connection, serialization, and queue time.
- Keep one connection alive for a Route Session wherever protocol semantics allow.
- Add bounded, actionable Generation Telemetry for each Activation Seam.
- Tune compression and buffer conversion from measured activation traces.
- Add bounded prefill backpressure and an end-to-end benchmark gate.

## Existing decisions preserved

- `X-Meshnet-Session` is stable for one Route Session.
- `request_id` remains unique per activation request for correlation.
- Hot KV State remains local to each Shard node.
- v1 activation transfer remains binary HTTP-shaped traffic.
- Streaming output remains preferred and telemetry remains mandatory.

## Task order

1. 01 — baseline and profiling harness
2. 02 — persistent relay compatibility hardening
3. 03 — direct and bridge HTTP keep-alive
4. 04 — seam telemetry and bounded progress reporting
5. 05 — adaptive activation compression
6. 06 — activation framing and copy reduction
7. 07 — prefill chunk backpressure
8. 08 — end-to-end performance gate

