28 lines
1014 B
Markdown
28 lines
1014 B
Markdown
Status: ready-for-agent
|
|
|
|
# 07 — Bounded prefill chunk backpressure
|
|
|
|
## What to build
|
|
|
|
Make large prefill transfer bounded across every Activation Seam. Chunk prompt
|
|
activations, limit in-flight chunks, and propagate downstream congestion so a
|
|
slow Shard node cannot cause the head or relay bridge to buffer the entire
|
|
context in memory. Keep decode-step traffic sequential and low-latency.
|
|
|
|
## Acceptance criteria
|
|
|
|
- [ ] Configurable prefill chunk size and in-flight limit exist with safe
|
|
defaults.
|
|
- [ ] Peak per-hop buffered bytes are bounded by the configured limits.
|
|
- [ ] A slow downstream stub applies backpressure and does not lose or reorder
|
|
chunks within a Route Session.
|
|
- [ ] Cancellation and route failure release queued chunks and local buffers.
|
|
- [ ] Tests cover small prompts, multi-chunk prompts, slow consumers, retry/fail
|
|
closeout, and legacy single-chunk peers.
|
|
|
|
## Blocked by
|
|
|
|
- 01 — Baseline and profiling harness.
|
|
- 04 — Activation Seam telemetry and bounded progress reporting.
|
|
|