issues, chat FPS; optimisations
This commit is contained in:
@@ -0,0 +1,27 @@
|
||||
Status: ready-for-agent
|
||||
|
||||
# 07 — Bounded prefill chunk backpressure
|
||||
|
||||
## What to build
|
||||
|
||||
Make large prefill transfer bounded across every Activation Seam. Chunk prompt
|
||||
activations, limit in-flight chunks, and propagate downstream congestion so a
|
||||
slow Shard node cannot cause the head or relay bridge to buffer the entire
|
||||
context in memory. Keep decode-step traffic sequential and low-latency.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- [ ] Configurable prefill chunk size and in-flight limit exist with safe
|
||||
defaults.
|
||||
- [ ] Peak per-hop buffered bytes are bounded by the configured limits.
|
||||
- [ ] A slow downstream stub applies backpressure and does not lose or reorder
|
||||
chunks within a Route Session.
|
||||
- [ ] Cancellation and route failure release queued chunks and local buffers.
|
||||
- [ ] Tests cover small prompts, multi-chunk prompts, slow consumers, retry/fail
|
||||
closeout, and legacy single-chunk peers.
|
||||
|
||||
## Blocked by
|
||||
|
||||
- 01 — Baseline and profiling harness.
|
||||
- 04 — Activation Seam telemetry and bounded progress reporting.
|
||||
|
||||
Reference in New Issue
Block a user