neuron-tai/.scratch/distributed-inference-performance/issues/07-prefill-backpressure.md

Status: ready-for-agent

07 — Bounded prefill chunk backpressure

What to build

Make large prefill transfer bounded across every Activation Seam. Chunk prompt activations, limit in-flight chunks, and propagate downstream congestion so a slow Shard node cannot cause the head or relay bridge to buffer the entire context in memory. Keep decode-step traffic sequential and low-latency.

Acceptance criteria

• Configurable prefill chunk size and in-flight limit exist with safe defaults.
• Peak per-hop buffered bytes are bounded by the configured limits.
• A slow downstream stub applies backpressure and does not lose or reorder chunks within a Route Session.
• Cancellation and route failure release queued chunks and local buffers.
• Tests cover small prompts, multi-chunk prompts, slow consumers, retry/fail closeout, and legacy single-chunk peers.

Blocked by

• 01 — Baseline and profiling harness.
• 04 — Activation Seam telemetry and bounded progress reporting.