issues, chat FPS; optimisations
This commit is contained in:
@@ -0,0 +1,27 @@
|
||||
Status: ready-for-agent
|
||||
|
||||
# 06 — Activation framing and copy reduction
|
||||
|
||||
## What to build
|
||||
|
||||
Profile and reduce avoidable allocations while activation data crosses a seam:
|
||||
binary frame assembly, header JSON, base64 metadata, CPU/GPU conversion, and
|
||||
response decompression. Preserve the current binary wire contract and use
|
||||
zero-copy or pooled buffers only where ownership and lifetime are explicit.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- [ ] The benchmark identifies copy/allocation cost separately from model and
|
||||
network time.
|
||||
- [ ] Decode hidden-state conversion has no unnecessary float32 round trip.
|
||||
- [ ] Binary framing avoids base64 for activation bodies and does not retain
|
||||
buffers after a request completes.
|
||||
- [ ] Position/attention metadata is validated and encoded efficiently without
|
||||
changing semantic headers or cache positions.
|
||||
- [ ] A focused test proves byte-for-byte wire compatibility and stable output
|
||||
tokens before and after the optimization.
|
||||
|
||||
## Blocked by
|
||||
|
||||
- 01 — Baseline and profiling harness.
|
||||
|
||||
Reference in New Issue
Block a user