38 lines
1.5 KiB
Markdown
38 lines
1.5 KiB
Markdown
# Distributed inference performance
|
|
|
|
Status: draft scratch package.
|
|
|
|
This feature measures and reduces avoidable overhead around the existing
|
|
Route Session, prefill/decode, local Hot KV State, and binary activation path.
|
|
It does not replace the distributed GGUF runtime plan. The goal is to make
|
|
transport and data movement cheap enough that model execution, rather than
|
|
connection setup, logging, or serialization, dominates token latency.
|
|
|
|
## Scope
|
|
|
|
- Baseline per-token compute, seam, connection, serialization, and queue time.
|
|
- Keep one connection alive for a Route Session wherever protocol semantics allow.
|
|
- Add bounded, actionable Generation Telemetry for each Activation Seam.
|
|
- Tune compression and buffer conversion from measured activation traces.
|
|
- Add bounded prefill backpressure and an end-to-end benchmark gate.
|
|
|
|
## Existing decisions preserved
|
|
|
|
- `X-Meshnet-Session` is stable for one Route Session.
|
|
- `request_id` remains unique per activation request for correlation.
|
|
- Hot KV State remains local to each Shard node.
|
|
- v1 activation transfer remains binary HTTP-shaped traffic.
|
|
- Streaming output remains preferred and telemetry remains mandatory.
|
|
|
|
## Task order
|
|
|
|
1. 01 — baseline and profiling harness
|
|
2. 02 — persistent relay compatibility hardening
|
|
3. 03 — direct and bridge HTTP keep-alive
|
|
4. 04 — seam telemetry and bounded progress reporting
|
|
5. 05 — adaptive activation compression
|
|
6. 06 — activation framing and copy reduction
|
|
7. 07 — prefill chunk backpressure
|
|
8. 08 — end-to-end performance gate
|
|
|