DGR-012 — Continuous batching and bounded admission: evidence

Status: done Date: 2026-07-16 Evidence kind: synthetic-unit (pure-numpy KV-cached dense-Llama reference + node-local continuous-batching scheduler). No model download, no GPU, no torch, no network, no API credit.

Summary

Implemented the node-local scheduler that turns concurrent Route Sessions into llama.cpp-style continuous batches while bounding admission (RALPH runtime decision #9, ADR-0024). It sits on top of the DGR-007 Hot KV State manager — batching is a scheduling concern layered over the existing per-(session, epoch) KV isolation, not a new control plane or a change to the KV contract.

Bounded admission (NodeBudget + submit). A new session is admitted only if it fits four budgets: resident weight footprint (reported), KV byte budget (a session must be able to hold its whole generation, prompt + new tokens, on its own), scratch (per-active-session activation buffers, capped by a total scratch envelope), and the bounded queue. Anything that cannot fit is rejected up front with an explicit AdmissionReason (REJECTED_KV_BUDGET / REJECTED_SCRATCH_BUDGET / REJECTED_DUPLICATE); anything that fits but has no free slot waits in the bounded queue; a full queue is refused (REJECTED_QUEUE_FULL) — that refusal is the backpressure signal.
Continuous batching (ContinuousBatchScheduler + KvBatchEngine). Every tick, all currently-decoding sessions contribute their single next token to one batch (bounded by max_batch_size); the engine runs the batch once. Each session keeps its own position and appends its own sampled token via its own SessionCache, so batching never mixes outputs. KvBatchEngine adapts the DGR-007 KvBoundaryAdapter, so the batch runs against the real KV isolation path; the pinned llama.cpp worker (DGR-008) implements the same recipe_fingerprint/prefill/decode_batch/release contract where a batch becomes one llama_decode over several sequences.
Prefill does not starve decode. The scheduling policy is explicit and fixed: decode first, then bounded prefill. In-flight decodes always run before any new prompt is prefilled, and prefill work per tick is capped (max_prefill_tokens_per_tick, always allowing at least one so a single large prompt still progresses). A burst of new sessions cannot stall generations already in flight.
Bounded memory / backpressure. KV growth is bounded by the manager byte budget; queued activations are bounded by max_queue_depth and the scratch envelope; completed sessions release their KV so total KV returns to zero.
Capability telemetry (SchedulerTelemetry). Reports active sessions, queue depth, batch occupancy (last/avg/max), KV pressure (bytes/budget), scratch pressure, prefill/decode token totals and rates, and rejected admissions (total + by reason). All JSON-safe.
Concurrency 1/2/4/8 sweep (run_concurrency_sweep). Runs the same eight jobs at each level against a fresh KV manager and proves (a) no cross-session corruption — every level yields byte-identical per-session tokens as the serialized concurrency-1 reference — and (b) saturation — average batch occupancy rises and total ticks fall as concurrency increases, until occupancy plateaus.

No existing runtime code was modified — this story is purely additive (one new module + one new test module + evidence).

Files changed (all new)

packages/node/meshnet_node/batch_scheduler.py — the scheduler:
- NodeBudget — weight/KV/scratch/queue budgets + max_batch_size / max_prefill_tokens_per_tick scheduling bounds, with derived effective_active_cap (tighter of active-slot and scratch caps).
- AdmissionReason / AdmissionDecision — structured admit/queue/reject.
- GenerationRequest / DecodeItem / StepResult — job + engine I/O values.
- KvBatchEngine — adapts a full-shard KvBoundaryAdapter to the batch-engine contract (rejects a partial head/tail-only range).
- SchedulerTelemetry — the bounded capability snapshot.
- ContinuousBatchScheduler — thread-safe submit / run_tick / run_to_completion / telemetry, decode-first-then-bounded-prefill policy.
- run_concurrency_sweep / ConcurrencyResult / ConcurrencySweep — the deterministic 1/2/4/8 saturation report + corruption check.
tests/test_batch_scheduler.py — 16 tests (see below); reuses the DGR-007 numpy dense-Llama reference via from test_hot_kv_state import _KvDenseLlama, _KvReferenceShard.
.scratch/distributed-gguf-runtime/evidence/DGR-012/ — this README, commands.txt, generate_evidence.py, results.json.

Acceptance criteria → evidence

Scheduler admits sessions against weight, KV, scratch, and queue budgets — test_admission_respects_active_scratch_and_queue_budgets (fill slots → queue → reject full queue), test_admission_rejects_a_session_that_cannot_fit_the_kv_budget, test_admission_rejects_when_per_session_scratch_exceeds_budget, test_duplicate_submission_is_rejected, test_weight_budget_is_reported_in_telemetry.
Compatible decode steps form batches preserving per-session positions/outputs — test_batched_decode_preserves_per_session_positions_and_outputs (batch_occupancy_max == 4, four divergent references each reproduced), test_positions_are_isolated_across_different_prompt_lengths (prompt lengths 1/3/7).
Prefill does not starve decode; policy and bounds explicit — test_prefill_does_not_starve_in_flight_decode (in-flight session decodes on every tick during a 4-session prefill burst; ≤1 prefill/tick), test_decode_first_policy_is_explicit_in_a_single_tick.
Backpressure prevents unbounded queued activations or KV growth — test_backpressure_signals_when_queue_full_then_recovers, test_completed_sessions_release_kv_so_growth_is_bounded (kv_total_bytes == 0 after completion).
Capability telemetry reports all required signals — test_telemetry_reports_every_required_signal (asserts every key present; deterministic rates under an injected clock).
Concurrency 1/2/4/8 identifies saturation, no cross-session corruption — test_concurrency_sweep_identifies_saturation_without_corruption (occupancy strictly ↑, ticks strictly ↓, tokens/tick ↑, corruption_free, 0 cache misses, saturation=8), test_concurrency_sweep_saturates_below_max_when_load_is_small.
Engine/usage guards — test_kv_batch_engine_requires_a_full_shard, test_run_to_completion_is_bounded_against_misconfiguration.

Concurrency 1/2/4/8 sweep (real, deterministic — `results.json`)

Eight sessions, prompt length 4, 8 new tokens each; fresh KV manager per level; budgets sized so KV never evicts (so the corruption check is unambiguous).

concurrency	ticks	avg batch occupancy	max occupancy	tokens/tick	peak KV bytes
1	64	1.000	1	1.375	15360
2	33	1.750	2	2.667	29184
4	19	3.111	4	4.632	52224
8	15	4.000	7	5.867	75264

saturation_concurrency = 8, corruption_free = True, cache_misses = 0, rejected_admissions = 0. As concurrency rises, the scheduler packs more sessions per decode step (occupancy ↑) and finishes the same 56 decode + 32 prefill tokens in far fewer ticks (aggregate work/tick ↑) — the batching throughput property — while every per-session token stream stays byte-identical to the serialized reference (no cross-session corruption). Max occupancy is 7 (not 8) at level 8 because the fairness policy prefills at most one new session per tick, so the last session begins decoding one tick later.

Commands and real results

VP=/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python

$VP -m pytest -q tests/test_batch_scheduler.py
# -> 16 passed

$VP -m pytest -q tests/test_hot_kv_state.py       # dependency still green
# -> 22 passed

$VP -m compileall -q packages tests
# -> exit 0

git diff --check
# -> exit 0

$VP .scratch/distributed-gguf-runtime/evidence/DGR-012/generate_evidence.py
# -> wrote results.json; saturation_concurrency=8 corruption_free=True

$VP -m pytest -q -rfE -p no:cacheprovider
# -> FULL_SUITE_RESULT_PLACEHOLDER

commands.txt beside this README captures the exact commands.

Full-suite baseline (pre-existing unrelated failures)

FULL_SUITE_BASELINE_PLACEHOLDER

Limitations and deferred work

Synthetic-unit, not real weights. The scheduler is exercised against the deterministic numpy KV-cached dense-Llama reference (the same one DGR-007 uses), not a downloaded GGUF. This is required to keep the default gate deterministic, download-free, and GPU-free. Real concurrent throughput on a downloaded dense-Llama (CPU/ROCm) belongs to DGR-010 (blocked — no certified dense-Llama artifact on this machine; see evidence/DGR-010/BLOCKED.md) and the final comparison in DGR-014.
Batching is a scheduling grouping in this reference. KvBatchEngine.decode_batch runs each batch member sequentially through the cached decode (each attends only its own KV, exactly like an independent llama.cpp sequence). The pinned llama.cpp worker (DGR-008) fuses the batch into one llama_decode graph; the scheduling semantics — one batch per tick, isolated positions/outputs — are identical. The numbers here are scheduler quantities (ticks, batch occupancy, tokens/tick) that are real and deterministic; actual kernel-level batching speedup is a native-worker property and is NOT claimed here (RALPH performance discipline: no unmeasured speed claims). It is measured in DGR-008/DGR-010/DGR-014.
Greedy sampling only. Reuses the DGR-006 greedy SamplingContract. Greedy over isolated per-session KV is order-independent, which is exactly why the corruption check can assert byte-identical outputs across concurrency levels. Stochastic sampling is out of scope for the deterministic gate.
Single loaded shard / single recipe per scheduler. The scheduler batches compatible sessions of one loaded shard (one recipe_fingerprint), which is the node-local case. Multi-range routes batch at the head node whose adapter owns the final head; cross-node coordination stays in the Meshnet control plane.
Native / llama.cpp gates N/A. No native code, CMake, or llama.cpp patch was touched (same as DGR-005/006/007), so those gates do not apply to this story.

Compatibility / migration notes

Purely additive: no existing module changed, so no behavior of the Torch/GGUF backends, tracker, or KV manager is altered. The scheduler is opt-in — a server constructs it around a KvBatchEngine when it wants continuous batching.
SchedulerTelemetry.to_dict() is JSON-safe and aligns with the capability-signal vocabulary (active sessions, queue depth, batch occupancy, KV pressure, prefill/decode rates, rejected admissions) that a node advertises upward; it can be folded into the DGR-009 capability report / heartbeat without schema changes here.
AdmissionReason values are stable strings suitable for the native protocol's structured status / backpressure signalling.

Handoff for dependent stories

DGR-008 (C++ gRPC worker): implement the BatchEngine contract natively — decode_batch becomes one llama_decode over the sessions' filtered sequences; prefill/release map to the same KV manager operations. The scheduler, admission budgets, fairness policy, and telemetry are unchanged; only the engine swaps from numpy to llama.cpp.
DGR-010 (local real two-process acceptance, blocked): once a certified dense-Llama artifact is mounted, drive run_concurrency_sweep (or the scheduler directly) with a real KvBatchEngine over the GGUF backend to produce real-hardware occupancy/throughput/KV-pressure numbers under MESHNET_ENABLE_REAL_INFERENCE_TESTS=1 / .venv-rocm.
DGR-013 (failure/cancel/restart): the DoneReason.CACHE_MISS path (a decode whose KV was evicted marks the session done and re-prefillable) and the KV-release on completion are the unit basis for the cancellation/cleanup matrix.
DGR-014 (release gate): feed the real-hardware sweep’s aggregate throughput and saturation point into the immutable DGR-001 comparison; do not reuse these synthetic numbers as a performance claim.

12 KiB Raw Blame History Unescape Escape