34 lines
1.1 KiB
Markdown
34 lines
1.1 KiB
Markdown
# 02 — Stable session and distributed KV in PyTorch path
|
|
|
|
Status: ready-for-agent
|
|
|
|
## Goal
|
|
|
|
Fix the existing distributed PyTorch path so it does not recompute the full growing prompt for every output token.
|
|
|
|
## Scope
|
|
|
|
- Introduce stable `session_id` for one request/session.
|
|
- Add per-node session cache keyed by `session_id`.
|
|
- Split `/forward` semantics into prefill and decode-step.
|
|
- Use model cache objects / `past_key_values` where supported.
|
|
- Keep hot KV local to each shard node.
|
|
- Add cleanup/TTL for abandoned sessions.
|
|
|
|
## Current Problem
|
|
|
|
The current distributed path:
|
|
|
|
- calls `encode_prompt(current_text)` for every generated token
|
|
- sends full-sequence activations through the route
|
|
- calls layers with `use_cache=False`
|
|
- creates a fresh UUID inside `_run_downstream_pipeline()`
|
|
|
|
## Acceptance
|
|
|
|
- Decode seam payload is one token / one hidden state after prefill.
|
|
- Per-shard cache grows locally with generated tokens.
|
|
- Regression test proves layer calls use cache after prefill.
|
|
- Fallback error is explicit for models whose manual cache API is unsupported.
|
|
|