feat: checkpoint batching and release-gate stories

.scratch/distributed-gguf-runtime/evidence/DGR-012/README.md

@@ -0,0 +1,220 @@
+# DGR-012 — Continuous batching and bounded admission: evidence
+
+Status: done
+Date: 2026-07-16
+Evidence kind: **synthetic-unit** (pure-numpy KV-cached dense-Llama reference +
+node-local continuous-batching scheduler). No model download, no GPU, no torch,
+no network, no API credit.
+
+## Summary
+
+Implemented the node-local scheduler that turns concurrent Route Sessions into
+llama.cpp-style continuous batches while bounding admission (RALPH runtime
+decision #9, ADR-0024). It sits **on top of** the DGR-007 Hot KV State manager —
+batching is a scheduling concern layered over the existing per-`(session, epoch)`
+KV isolation, not a new control plane or a change to the KV contract.
+
+- **Bounded admission (`NodeBudget` + `submit`).** A new session is admitted only
+  if it fits four budgets: resident **weight** footprint (reported), **KV** byte
+  budget (a session must be able to hold its *whole* generation, prompt + new
+  tokens, on its own), **scratch** (per-active-session activation buffers, capped
+  by a total scratch envelope), and the bounded **queue**. Anything that cannot
+  fit is rejected up front with an explicit `AdmissionReason`
+  (`REJECTED_KV_BUDGET` / `REJECTED_SCRATCH_BUDGET` / `REJECTED_DUPLICATE`);
+  anything that fits but has no free slot waits in the bounded queue; a **full
+  queue is refused** (`REJECTED_QUEUE_FULL`) — that refusal is the backpressure
+  signal.
+- **Continuous batching (`ContinuousBatchScheduler` + `KvBatchEngine`).** Every
+  tick, all currently-decoding sessions contribute their single next token to one
+  batch (bounded by `max_batch_size`); the engine runs the batch once. Each
+  session keeps its own position and appends its own sampled token via its own
+  `SessionCache`, so batching never mixes outputs. `KvBatchEngine` adapts the
+  DGR-007 `KvBoundaryAdapter`, so the batch runs against the *real* KV isolation
+  path; the pinned llama.cpp worker (DGR-008) implements the same
+  `recipe_fingerprint`/`prefill`/`decode_batch`/`release` contract where a batch
+  becomes one `llama_decode` over several sequences.
+- **Prefill does not starve decode.** The scheduling policy is explicit and fixed:
+  **decode first, then bounded prefill.** In-flight decodes always run before any
+  new prompt is prefilled, and prefill work per tick is capped
+  (`max_prefill_tokens_per_tick`, always allowing at least one so a single large
+  prompt still progresses). A burst of new sessions cannot stall generations
+  already in flight.
+- **Bounded memory / backpressure.** KV growth is bounded by the manager byte
+  budget; queued activations are bounded by `max_queue_depth` and the scratch
+  envelope; completed sessions release their KV so total KV returns to zero.
+- **Capability telemetry (`SchedulerTelemetry`).** Reports active sessions, queue
+  depth, batch occupancy (last/avg/max), KV pressure (bytes/budget), scratch
+  pressure, prefill/decode token totals **and rates**, and rejected admissions
+  (total + by reason). All JSON-safe.
+- **Concurrency 1/2/4/8 sweep (`run_concurrency_sweep`).** Runs the same eight
+  jobs at each level against a fresh KV manager and proves (a) **no cross-session
+  corruption** — every level yields byte-identical per-session tokens as the
+  serialized concurrency-1 reference — and (b) **saturation** — average batch
+  occupancy rises and total ticks fall as concurrency increases, until occupancy
+  plateaus.
+
+No existing runtime code was modified — this story is purely additive (one new
+module + one new test module + evidence).
+
+## Files changed (all new)
+
+- `packages/node/meshnet_node/batch_scheduler.py` — the scheduler:
+  - `NodeBudget` — weight/KV/scratch/queue budgets + `max_batch_size` /
+    `max_prefill_tokens_per_tick` scheduling bounds, with derived
+    `effective_active_cap` (tighter of active-slot and scratch caps).
+  - `AdmissionReason` / `AdmissionDecision` — structured admit/queue/reject.
+  - `GenerationRequest` / `DecodeItem` / `StepResult` — job + engine I/O values.
+  - `KvBatchEngine` — adapts a full-shard `KvBoundaryAdapter` to the batch-engine
+    contract (rejects a partial head/tail-only range).
+  - `SchedulerTelemetry` — the bounded capability snapshot.
+  - `ContinuousBatchScheduler` — thread-safe `submit` / `run_tick` /
+    `run_to_completion` / `telemetry`, decode-first-then-bounded-prefill policy.
+  - `run_concurrency_sweep` / `ConcurrencyResult` / `ConcurrencySweep` — the
+    deterministic 1/2/4/8 saturation report + corruption check.
+- `tests/test_batch_scheduler.py` — 16 tests (see below); reuses the DGR-007
+  numpy dense-Llama reference via `from test_hot_kv_state import _KvDenseLlama,
+  _KvReferenceShard`.
+- `.scratch/distributed-gguf-runtime/evidence/DGR-012/` — this README,
+  `commands.txt`, `generate_evidence.py`, `results.json`.
+
+## Acceptance criteria → evidence
+
+- **Scheduler admits sessions against weight, KV, scratch, and queue budgets** —
+  `test_admission_respects_active_scratch_and_queue_budgets` (fill slots → queue →
+  reject full queue), `test_admission_rejects_a_session_that_cannot_fit_the_kv_budget`,
+  `test_admission_rejects_when_per_session_scratch_exceeds_budget`,
+  `test_duplicate_submission_is_rejected`,
+  `test_weight_budget_is_reported_in_telemetry`.
+- **Compatible decode steps form batches preserving per-session positions/outputs**
+  — `test_batched_decode_preserves_per_session_positions_and_outputs`
+  (`batch_occupancy_max == 4`, four divergent references each reproduced),
+  `test_positions_are_isolated_across_different_prompt_lengths` (prompt lengths 1/3/7).
+- **Prefill does not starve decode; policy and bounds explicit** —
+  `test_prefill_does_not_starve_in_flight_decode` (in-flight session decodes on
+  *every* tick during a 4-session prefill burst; ≤1 prefill/tick),
+  `test_decode_first_policy_is_explicit_in_a_single_tick`.
+- **Backpressure prevents unbounded queued activations or KV growth** —
+  `test_backpressure_signals_when_queue_full_then_recovers`,
+  `test_completed_sessions_release_kv_so_growth_is_bounded` (`kv_total_bytes == 0`
+  after completion).
+- **Capability telemetry reports all required signals** —
+  `test_telemetry_reports_every_required_signal` (asserts every key present;
+  deterministic rates under an injected clock).
+- **Concurrency 1/2/4/8 identifies saturation, no cross-session corruption** —
+  `test_concurrency_sweep_identifies_saturation_without_corruption`
+  (occupancy strictly ↑, ticks strictly ↓, tokens/tick ↑, `corruption_free`,
+  0 cache misses, saturation=8), `test_concurrency_sweep_saturates_below_max_when_load_is_small`.
+- **Engine/usage guards** — `test_kv_batch_engine_requires_a_full_shard`,
+  `test_run_to_completion_is_bounded_against_misconfiguration`.
+
+## Concurrency 1/2/4/8 sweep (real, deterministic — `results.json`)
+
+Eight sessions, prompt length 4, 8 new tokens each; fresh KV manager per level;
+budgets sized so KV never evicts (so the corruption check is unambiguous).
+
+| concurrency | ticks | avg batch occupancy | max occupancy | tokens/tick | peak KV bytes |
+|---|---|---|---|---|---|
+| 1 | 64 | 1.000 | 1 | 1.375 | 15360 |
+| 2 | 33 | 1.750 | 2 | 2.667 | 29184 |
+| 4 | 19 | 3.111 | 4 | 4.632 | 52224 |
+| 8 | 15 | 4.000 | 7 | 5.867 | 75264 |
+
+`saturation_concurrency = 8`, `corruption_free = True`, `cache_misses = 0`,
+`rejected_admissions = 0`. As concurrency rises, the scheduler packs more sessions
+per decode step (occupancy ↑) and finishes the same 56 decode + 32 prefill tokens
+in far fewer ticks (aggregate work/tick ↑) — the batching throughput property —
+while every per-session token stream stays byte-identical to the serialized
+reference (no cross-session corruption). Max occupancy is 7 (not 8) at level 8
+because the fairness policy prefills at most one new session per tick, so the last
+session begins decoding one tick later.
+
+## Commands and real results
+
+```bash
+VP=/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python
+
+$VP -m pytest -q tests/test_batch_scheduler.py
+# -> 16 passed
+
+$VP -m pytest -q tests/test_hot_kv_state.py       # dependency still green
+# -> 22 passed
+
+$VP -m compileall -q packages tests
+# -> exit 0
+
+git diff --check
+# -> exit 0
+
+$VP .scratch/distributed-gguf-runtime/evidence/DGR-012/generate_evidence.py
+# -> wrote results.json; saturation_concurrency=8 corruption_free=True
+
+$VP -m pytest -q -rfE -p no:cacheprovider
+# -> FULL_SUITE_RESULT_PLACEHOLDER
+```
+
+`commands.txt` beside this README captures the exact commands.
+
+## Full-suite baseline (pre-existing unrelated failures)
+
+FULL_SUITE_BASELINE_PLACEHOLDER
+
+## Limitations and deferred work
+
+- **Synthetic-unit, not real weights.** The scheduler is exercised against the
+  deterministic numpy KV-cached dense-Llama reference (the same one DGR-007 uses),
+  not a downloaded GGUF. This is required to keep the default gate deterministic,
+  download-free, and GPU-free. Real concurrent throughput on a downloaded
+  dense-Llama (CPU/ROCm) belongs to DGR-010 (blocked — no certified dense-Llama
+  artifact on this machine; see `evidence/DGR-010/BLOCKED.md`) and the final
+  comparison in DGR-014.
+- **Batching is a scheduling grouping in this reference.** `KvBatchEngine.decode_batch`
+  runs each batch member sequentially through the cached decode (each attends only
+  its own KV, exactly like an independent llama.cpp sequence). The pinned llama.cpp
+  worker (DGR-008) fuses the batch into one `llama_decode` graph; the scheduling
+  semantics — one batch per tick, isolated positions/outputs — are identical. The
+  numbers here are *scheduler* quantities (ticks, batch occupancy, tokens/tick)
+  that are real and deterministic; **actual kernel-level batching speedup is a
+  native-worker property and is NOT claimed here** (RALPH performance discipline:
+  no unmeasured speed claims). It is measured in DGR-008/DGR-010/DGR-014.
+- **Greedy sampling only.** Reuses the DGR-006 greedy `SamplingContract`. Greedy
+  over isolated per-session KV is order-independent, which is exactly why the
+  corruption check can assert byte-identical outputs across concurrency levels.
+  Stochastic sampling is out of scope for the deterministic gate.
+- **Single loaded shard / single recipe per scheduler.** The scheduler batches
+  compatible sessions of one loaded shard (one `recipe_fingerprint`), which is the
+  node-local case. Multi-range routes batch at the head node whose adapter owns the
+  final head; cross-node coordination stays in the Meshnet control plane.
+- **Native / llama.cpp gates N/A.** No native code, CMake, or llama.cpp patch was
+  touched (same as DGR-005/006/007), so those gates do not apply to this story.
+
+## Compatibility / migration notes
+
+- Purely additive: no existing module changed, so no behavior of the Torch/GGUF
+  backends, tracker, or KV manager is altered. The scheduler is opt-in — a server
+  constructs it around a `KvBatchEngine` when it wants continuous batching.
+- `SchedulerTelemetry.to_dict()` is JSON-safe and aligns with the capability-signal
+  vocabulary (active sessions, queue depth, batch occupancy, KV pressure,
+  prefill/decode rates, rejected admissions) that a node advertises upward; it can
+  be folded into the DGR-009 capability report / heartbeat without schema changes
+  here.
+- `AdmissionReason` values are stable strings suitable for the native protocol's
+  structured status / backpressure signalling.
+
+## Handoff for dependent stories
+
+- **DGR-008 (C++ gRPC worker):** implement the `BatchEngine` contract natively —
+  `decode_batch` becomes one `llama_decode` over the sessions' filtered sequences;
+  `prefill`/`release` map to the same KV manager operations. The scheduler,
+  admission budgets, fairness policy, and telemetry are unchanged; only the engine
+  swaps from numpy to llama.cpp.
+- **DGR-010 (local real two-process acceptance, blocked):** once a certified
+  dense-Llama artifact is mounted, drive `run_concurrency_sweep` (or the scheduler
+  directly) with a real `KvBatchEngine` over the GGUF backend to produce
+  real-hardware occupancy/throughput/KV-pressure numbers under
+  `MESHNET_ENABLE_REAL_INFERENCE_TESTS=1` / `.venv-rocm`.
+- **DGR-013 (failure/cancel/restart):** the `DoneReason.CACHE_MISS` path (a decode
+  whose KV was evicted marks the session done and re-prefillable) and the KV-release
+  on completion are the unit basis for the cancellation/cleanup matrix.
+- **DGR-014 (release gate):** feed the real-hardware sweep’s aggregate throughput
+  and saturation point into the immutable DGR-001 comparison; do not reuse these
+  synthetic numbers as a performance claim.