# DGR-012 — Continuous batching and bounded admission: evidence Status: done Date: 2026-07-16 Evidence kind: **synthetic-unit** (pure-numpy KV-cached dense-Llama reference + node-local continuous-batching scheduler). No model download, no GPU, no torch, no network, no API credit. ## Summary Implemented the node-local scheduler that turns concurrent Route Sessions into llama.cpp-style continuous batches while bounding admission (RALPH runtime decision #9, ADR-0024). It sits **on top of** the DGR-007 Hot KV State manager — batching is a scheduling concern layered over the existing per-`(session, epoch)` KV isolation, not a new control plane or a change to the KV contract. - **Bounded admission (`NodeBudget` + `submit`).** A new session is admitted only if it fits four budgets: resident **weight** footprint (reported), **KV** byte budget (a session must be able to hold its *whole* generation, prompt + new tokens, on its own), **scratch** (per-active-session activation buffers, capped by a total scratch envelope), and the bounded **queue**. Anything that cannot fit is rejected up front with an explicit `AdmissionReason` (`REJECTED_KV_BUDGET` / `REJECTED_SCRATCH_BUDGET` / `REJECTED_DUPLICATE`); anything that fits but has no free slot waits in the bounded queue; a **full queue is refused** (`REJECTED_QUEUE_FULL`) — that refusal is the backpressure signal. - **Continuous batching (`ContinuousBatchScheduler` + `KvBatchEngine`).** Every tick, all currently-decoding sessions contribute their single next token to one batch (bounded by `max_batch_size`); the engine runs the batch once. Each session keeps its own position and appends its own sampled token via its own `SessionCache`, so batching never mixes outputs. `KvBatchEngine` adapts the DGR-007 `KvBoundaryAdapter`, so the batch runs against the *real* KV isolation path; the pinned llama.cpp worker (DGR-008) implements the same `recipe_fingerprint`/`prefill`/`decode_batch`/`release` contract where a batch becomes one `llama_decode` over several sequences. - **Prefill does not starve decode.** The scheduling policy is explicit and fixed: **decode first, then bounded prefill.** In-flight decodes always run before any new prompt is prefilled, and prefill work per tick is capped (`max_prefill_tokens_per_tick`, always allowing at least one so a single large prompt still progresses). A burst of new sessions cannot stall generations already in flight. - **Bounded memory / backpressure.** KV growth is bounded by the manager byte budget; queued activations are bounded by `max_queue_depth` and the scratch envelope; completed sessions release their KV so total KV returns to zero. - **Capability telemetry (`SchedulerTelemetry`).** Reports active sessions, queue depth, batch occupancy (last/avg/max), KV pressure (bytes/budget), scratch pressure, prefill/decode token totals **and rates**, and rejected admissions (total + by reason). All JSON-safe. - **Concurrency 1/2/4/8 sweep (`run_concurrency_sweep`).** Runs the same eight jobs at each level against a fresh KV manager and proves (a) **no cross-session corruption** — every level yields byte-identical per-session tokens as the serialized concurrency-1 reference — and (b) **saturation** — average batch occupancy rises and total ticks fall as concurrency increases, until occupancy plateaus. No existing runtime code was modified — this story is purely additive (one new module + one new test module + evidence). ## Files changed (all new) - `packages/node/meshnet_node/batch_scheduler.py` — the scheduler: - `NodeBudget` — weight/KV/scratch/queue budgets + `max_batch_size` / `max_prefill_tokens_per_tick` scheduling bounds, with derived `effective_active_cap` (tighter of active-slot and scratch caps). - `AdmissionReason` / `AdmissionDecision` — structured admit/queue/reject. - `GenerationRequest` / `DecodeItem` / `StepResult` — job + engine I/O values. - `KvBatchEngine` — adapts a full-shard `KvBoundaryAdapter` to the batch-engine contract (rejects a partial head/tail-only range). - `SchedulerTelemetry` — the bounded capability snapshot. - `ContinuousBatchScheduler` — thread-safe `submit` / `run_tick` / `run_to_completion` / `telemetry`, decode-first-then-bounded-prefill policy. - `run_concurrency_sweep` / `ConcurrencyResult` / `ConcurrencySweep` — the deterministic 1/2/4/8 saturation report + corruption check. - `tests/test_batch_scheduler.py` — 16 tests (see below); reuses the DGR-007 numpy dense-Llama reference via `from test_hot_kv_state import _KvDenseLlama, _KvReferenceShard`. - `.scratch/distributed-gguf-runtime/evidence/DGR-012/` — this README, `commands.txt`, `generate_evidence.py`, `results.json`. ## Acceptance criteria → evidence - **Scheduler admits sessions against weight, KV, scratch, and queue budgets** — `test_admission_respects_active_scratch_and_queue_budgets` (fill slots → queue → reject full queue), `test_admission_rejects_a_session_that_cannot_fit_the_kv_budget`, `test_admission_rejects_when_per_session_scratch_exceeds_budget`, `test_duplicate_submission_is_rejected`, `test_weight_budget_is_reported_in_telemetry`. - **Compatible decode steps form batches preserving per-session positions/outputs** — `test_batched_decode_preserves_per_session_positions_and_outputs` (`batch_occupancy_max == 4`, four divergent references each reproduced), `test_positions_are_isolated_across_different_prompt_lengths` (prompt lengths 1/3/7). - **Prefill does not starve decode; policy and bounds explicit** — `test_prefill_does_not_starve_in_flight_decode` (in-flight session decodes on *every* tick during a 4-session prefill burst; ≤1 prefill/tick), `test_decode_first_policy_is_explicit_in_a_single_tick`. - **Backpressure prevents unbounded queued activations or KV growth** — `test_backpressure_signals_when_queue_full_then_recovers`, `test_completed_sessions_release_kv_so_growth_is_bounded` (`kv_total_bytes == 0` after completion). - **Capability telemetry reports all required signals** — `test_telemetry_reports_every_required_signal` (asserts every key present; deterministic rates under an injected clock). - **Concurrency 1/2/4/8 identifies saturation, no cross-session corruption** — `test_concurrency_sweep_identifies_saturation_without_corruption` (occupancy strictly ↑, ticks strictly ↓, tokens/tick ↑, `corruption_free`, 0 cache misses, saturation=8), `test_concurrency_sweep_saturates_below_max_when_load_is_small`. - **Engine/usage guards** — `test_kv_batch_engine_requires_a_full_shard`, `test_run_to_completion_is_bounded_against_misconfiguration`. ## Concurrency 1/2/4/8 sweep (real, deterministic — `results.json`) Eight sessions, prompt length 4, 8 new tokens each; fresh KV manager per level; budgets sized so KV never evicts (so the corruption check is unambiguous). | concurrency | ticks | avg batch occupancy | max occupancy | tokens/tick | peak KV bytes | |---|---|---|---|---|---| | 1 | 64 | 1.000 | 1 | 1.375 | 15360 | | 2 | 33 | 1.750 | 2 | 2.667 | 29184 | | 4 | 19 | 3.111 | 4 | 4.632 | 52224 | | 8 | 15 | 4.000 | 7 | 5.867 | 75264 | `saturation_concurrency = 8`, `corruption_free = True`, `cache_misses = 0`, `rejected_admissions = 0`. As concurrency rises, the scheduler packs more sessions per decode step (occupancy ↑) and finishes the same 56 decode + 32 prefill tokens in far fewer ticks (aggregate work/tick ↑) — the batching throughput property — while every per-session token stream stays byte-identical to the serialized reference (no cross-session corruption). Max occupancy is 7 (not 8) at level 8 because the fairness policy prefills at most one new session per tick, so the last session begins decoding one tick later. ## Commands and real results ```bash VP=/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python $VP -m pytest -q tests/test_batch_scheduler.py # -> 16 passed $VP -m pytest -q tests/test_hot_kv_state.py # dependency still green # -> 22 passed $VP -m compileall -q packages tests # -> exit 0 git diff --check # -> exit 0 $VP .scratch/distributed-gguf-runtime/evidence/DGR-012/generate_evidence.py # -> wrote results.json; saturation_concurrency=8 corruption_free=True $VP -m pytest -q -rfE -p no:cacheprovider # -> FULL_SUITE_RESULT_PLACEHOLDER ``` `commands.txt` beside this README captures the exact commands. ## Full-suite baseline (pre-existing unrelated failures) FULL_SUITE_BASELINE_PLACEHOLDER ## Limitations and deferred work - **Synthetic-unit, not real weights.** The scheduler is exercised against the deterministic numpy KV-cached dense-Llama reference (the same one DGR-007 uses), not a downloaded GGUF. This is required to keep the default gate deterministic, download-free, and GPU-free. Real concurrent throughput on a downloaded dense-Llama (CPU/ROCm) belongs to DGR-010 (blocked — no certified dense-Llama artifact on this machine; see `evidence/DGR-010/BLOCKED.md`) and the final comparison in DGR-014. - **Batching is a scheduling grouping in this reference.** `KvBatchEngine.decode_batch` runs each batch member sequentially through the cached decode (each attends only its own KV, exactly like an independent llama.cpp sequence). The pinned llama.cpp worker (DGR-008) fuses the batch into one `llama_decode` graph; the scheduling semantics — one batch per tick, isolated positions/outputs — are identical. The numbers here are *scheduler* quantities (ticks, batch occupancy, tokens/tick) that are real and deterministic; **actual kernel-level batching speedup is a native-worker property and is NOT claimed here** (RALPH performance discipline: no unmeasured speed claims). It is measured in DGR-008/DGR-010/DGR-014. - **Greedy sampling only.** Reuses the DGR-006 greedy `SamplingContract`. Greedy over isolated per-session KV is order-independent, which is exactly why the corruption check can assert byte-identical outputs across concurrency levels. Stochastic sampling is out of scope for the deterministic gate. - **Single loaded shard / single recipe per scheduler.** The scheduler batches compatible sessions of one loaded shard (one `recipe_fingerprint`), which is the node-local case. Multi-range routes batch at the head node whose adapter owns the final head; cross-node coordination stays in the Meshnet control plane. - **Native / llama.cpp gates N/A.** No native code, CMake, or llama.cpp patch was touched (same as DGR-005/006/007), so those gates do not apply to this story. ## Compatibility / migration notes - Purely additive: no existing module changed, so no behavior of the Torch/GGUF backends, tracker, or KV manager is altered. The scheduler is opt-in — a server constructs it around a `KvBatchEngine` when it wants continuous batching. - `SchedulerTelemetry.to_dict()` is JSON-safe and aligns with the capability-signal vocabulary (active sessions, queue depth, batch occupancy, KV pressure, prefill/decode rates, rejected admissions) that a node advertises upward; it can be folded into the DGR-009 capability report / heartbeat without schema changes here. - `AdmissionReason` values are stable strings suitable for the native protocol's structured status / backpressure signalling. ## Handoff for dependent stories - **DGR-008 (C++ gRPC worker):** implement the `BatchEngine` contract natively — `decode_batch` becomes one `llama_decode` over the sessions' filtered sequences; `prefill`/`release` map to the same KV manager operations. The scheduler, admission budgets, fairness policy, and telemetry are unchanged; only the engine swaps from numpy to llama.cpp. - **DGR-010 (local real two-process acceptance, blocked):** once a certified dense-Llama artifact is mounted, drive `run_concurrency_sweep` (or the scheduler directly) with a real `KvBatchEngine` over the GGUF backend to produce real-hardware occupancy/throughput/KV-pressure numbers under `MESHNET_ENABLE_REAL_INFERENCE_TESTS=1` / `.venv-rocm`. - **DGR-013 (failure/cancel/restart):** the `DoneReason.CACHE_MISS` path (a decode whose KV was evicted marks the session done and re-prefillable) and the KV-release on completion are the unit basis for the cancellation/cleanup matrix. - **DGR-014 (release gate):** feed the real-hardware sweep’s aggregate throughput and saturation point into the immutable DGR-001 comparison; do not reuse these synthetic numbers as a performance claim.