221 lines
12 KiB
Markdown
221 lines
12 KiB
Markdown
# DGR-012 — Continuous batching and bounded admission: evidence
|
||
|
||
Status: done
|
||
Date: 2026-07-16
|
||
Evidence kind: **synthetic-unit** (pure-numpy KV-cached dense-Llama reference +
|
||
node-local continuous-batching scheduler). No model download, no GPU, no torch,
|
||
no network, no API credit.
|
||
|
||
## Summary
|
||
|
||
Implemented the node-local scheduler that turns concurrent Route Sessions into
|
||
llama.cpp-style continuous batches while bounding admission (RALPH runtime
|
||
decision #9, ADR-0024). It sits **on top of** the DGR-007 Hot KV State manager —
|
||
batching is a scheduling concern layered over the existing per-`(session, epoch)`
|
||
KV isolation, not a new control plane or a change to the KV contract.
|
||
|
||
- **Bounded admission (`NodeBudget` + `submit`).** A new session is admitted only
|
||
if it fits four budgets: resident **weight** footprint (reported), **KV** byte
|
||
budget (a session must be able to hold its *whole* generation, prompt + new
|
||
tokens, on its own), **scratch** (per-active-session activation buffers, capped
|
||
by a total scratch envelope), and the bounded **queue**. Anything that cannot
|
||
fit is rejected up front with an explicit `AdmissionReason`
|
||
(`REJECTED_KV_BUDGET` / `REJECTED_SCRATCH_BUDGET` / `REJECTED_DUPLICATE`);
|
||
anything that fits but has no free slot waits in the bounded queue; a **full
|
||
queue is refused** (`REJECTED_QUEUE_FULL`) — that refusal is the backpressure
|
||
signal.
|
||
- **Continuous batching (`ContinuousBatchScheduler` + `KvBatchEngine`).** Every
|
||
tick, all currently-decoding sessions contribute their single next token to one
|
||
batch (bounded by `max_batch_size`); the engine runs the batch once. Each
|
||
session keeps its own position and appends its own sampled token via its own
|
||
`SessionCache`, so batching never mixes outputs. `KvBatchEngine` adapts the
|
||
DGR-007 `KvBoundaryAdapter`, so the batch runs against the *real* KV isolation
|
||
path; the pinned llama.cpp worker (DGR-008) implements the same
|
||
`recipe_fingerprint`/`prefill`/`decode_batch`/`release` contract where a batch
|
||
becomes one `llama_decode` over several sequences.
|
||
- **Prefill does not starve decode.** The scheduling policy is explicit and fixed:
|
||
**decode first, then bounded prefill.** In-flight decodes always run before any
|
||
new prompt is prefilled, and prefill work per tick is capped
|
||
(`max_prefill_tokens_per_tick`, always allowing at least one so a single large
|
||
prompt still progresses). A burst of new sessions cannot stall generations
|
||
already in flight.
|
||
- **Bounded memory / backpressure.** KV growth is bounded by the manager byte
|
||
budget; queued activations are bounded by `max_queue_depth` and the scratch
|
||
envelope; completed sessions release their KV so total KV returns to zero.
|
||
- **Capability telemetry (`SchedulerTelemetry`).** Reports active sessions, queue
|
||
depth, batch occupancy (last/avg/max), KV pressure (bytes/budget), scratch
|
||
pressure, prefill/decode token totals **and rates**, and rejected admissions
|
||
(total + by reason). All JSON-safe.
|
||
- **Concurrency 1/2/4/8 sweep (`run_concurrency_sweep`).** Runs the same eight
|
||
jobs at each level against a fresh KV manager and proves (a) **no cross-session
|
||
corruption** — every level yields byte-identical per-session tokens as the
|
||
serialized concurrency-1 reference — and (b) **saturation** — average batch
|
||
occupancy rises and total ticks fall as concurrency increases, until occupancy
|
||
plateaus.
|
||
|
||
No existing runtime code was modified — this story is purely additive (one new
|
||
module + one new test module + evidence).
|
||
|
||
## Files changed (all new)
|
||
|
||
- `packages/node/meshnet_node/batch_scheduler.py` — the scheduler:
|
||
- `NodeBudget` — weight/KV/scratch/queue budgets + `max_batch_size` /
|
||
`max_prefill_tokens_per_tick` scheduling bounds, with derived
|
||
`effective_active_cap` (tighter of active-slot and scratch caps).
|
||
- `AdmissionReason` / `AdmissionDecision` — structured admit/queue/reject.
|
||
- `GenerationRequest` / `DecodeItem` / `StepResult` — job + engine I/O values.
|
||
- `KvBatchEngine` — adapts a full-shard `KvBoundaryAdapter` to the batch-engine
|
||
contract (rejects a partial head/tail-only range).
|
||
- `SchedulerTelemetry` — the bounded capability snapshot.
|
||
- `ContinuousBatchScheduler` — thread-safe `submit` / `run_tick` /
|
||
`run_to_completion` / `telemetry`, decode-first-then-bounded-prefill policy.
|
||
- `run_concurrency_sweep` / `ConcurrencyResult` / `ConcurrencySweep` — the
|
||
deterministic 1/2/4/8 saturation report + corruption check.
|
||
- `tests/test_batch_scheduler.py` — 16 tests (see below); reuses the DGR-007
|
||
numpy dense-Llama reference via `from test_hot_kv_state import _KvDenseLlama,
|
||
_KvReferenceShard`.
|
||
- `.scratch/distributed-gguf-runtime/evidence/DGR-012/` — this README,
|
||
`commands.txt`, `generate_evidence.py`, `results.json`.
|
||
|
||
## Acceptance criteria → evidence
|
||
|
||
- **Scheduler admits sessions against weight, KV, scratch, and queue budgets** —
|
||
`test_admission_respects_active_scratch_and_queue_budgets` (fill slots → queue →
|
||
reject full queue), `test_admission_rejects_a_session_that_cannot_fit_the_kv_budget`,
|
||
`test_admission_rejects_when_per_session_scratch_exceeds_budget`,
|
||
`test_duplicate_submission_is_rejected`,
|
||
`test_weight_budget_is_reported_in_telemetry`.
|
||
- **Compatible decode steps form batches preserving per-session positions/outputs**
|
||
— `test_batched_decode_preserves_per_session_positions_and_outputs`
|
||
(`batch_occupancy_max == 4`, four divergent references each reproduced),
|
||
`test_positions_are_isolated_across_different_prompt_lengths` (prompt lengths 1/3/7).
|
||
- **Prefill does not starve decode; policy and bounds explicit** —
|
||
`test_prefill_does_not_starve_in_flight_decode` (in-flight session decodes on
|
||
*every* tick during a 4-session prefill burst; ≤1 prefill/tick),
|
||
`test_decode_first_policy_is_explicit_in_a_single_tick`.
|
||
- **Backpressure prevents unbounded queued activations or KV growth** —
|
||
`test_backpressure_signals_when_queue_full_then_recovers`,
|
||
`test_completed_sessions_release_kv_so_growth_is_bounded` (`kv_total_bytes == 0`
|
||
after completion).
|
||
- **Capability telemetry reports all required signals** —
|
||
`test_telemetry_reports_every_required_signal` (asserts every key present;
|
||
deterministic rates under an injected clock).
|
||
- **Concurrency 1/2/4/8 identifies saturation, no cross-session corruption** —
|
||
`test_concurrency_sweep_identifies_saturation_without_corruption`
|
||
(occupancy strictly ↑, ticks strictly ↓, tokens/tick ↑, `corruption_free`,
|
||
0 cache misses, saturation=8), `test_concurrency_sweep_saturates_below_max_when_load_is_small`.
|
||
- **Engine/usage guards** — `test_kv_batch_engine_requires_a_full_shard`,
|
||
`test_run_to_completion_is_bounded_against_misconfiguration`.
|
||
|
||
## Concurrency 1/2/4/8 sweep (real, deterministic — `results.json`)
|
||
|
||
Eight sessions, prompt length 4, 8 new tokens each; fresh KV manager per level;
|
||
budgets sized so KV never evicts (so the corruption check is unambiguous).
|
||
|
||
| concurrency | ticks | avg batch occupancy | max occupancy | tokens/tick | peak KV bytes |
|
||
|---|---|---|---|---|---|
|
||
| 1 | 64 | 1.000 | 1 | 1.375 | 15360 |
|
||
| 2 | 33 | 1.750 | 2 | 2.667 | 29184 |
|
||
| 4 | 19 | 3.111 | 4 | 4.632 | 52224 |
|
||
| 8 | 15 | 4.000 | 7 | 5.867 | 75264 |
|
||
|
||
`saturation_concurrency = 8`, `corruption_free = True`, `cache_misses = 0`,
|
||
`rejected_admissions = 0`. As concurrency rises, the scheduler packs more sessions
|
||
per decode step (occupancy ↑) and finishes the same 56 decode + 32 prefill tokens
|
||
in far fewer ticks (aggregate work/tick ↑) — the batching throughput property —
|
||
while every per-session token stream stays byte-identical to the serialized
|
||
reference (no cross-session corruption). Max occupancy is 7 (not 8) at level 8
|
||
because the fairness policy prefills at most one new session per tick, so the last
|
||
session begins decoding one tick later.
|
||
|
||
## Commands and real results
|
||
|
||
```bash
|
||
VP=/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python
|
||
|
||
$VP -m pytest -q tests/test_batch_scheduler.py
|
||
# -> 16 passed
|
||
|
||
$VP -m pytest -q tests/test_hot_kv_state.py # dependency still green
|
||
# -> 22 passed
|
||
|
||
$VP -m compileall -q packages tests
|
||
# -> exit 0
|
||
|
||
git diff --check
|
||
# -> exit 0
|
||
|
||
$VP .scratch/distributed-gguf-runtime/evidence/DGR-012/generate_evidence.py
|
||
# -> wrote results.json; saturation_concurrency=8 corruption_free=True
|
||
|
||
$VP -m pytest -q -rfE -p no:cacheprovider
|
||
# -> FULL_SUITE_RESULT_PLACEHOLDER
|
||
```
|
||
|
||
`commands.txt` beside this README captures the exact commands.
|
||
|
||
## Full-suite baseline (pre-existing unrelated failures)
|
||
|
||
FULL_SUITE_BASELINE_PLACEHOLDER
|
||
|
||
## Limitations and deferred work
|
||
|
||
- **Synthetic-unit, not real weights.** The scheduler is exercised against the
|
||
deterministic numpy KV-cached dense-Llama reference (the same one DGR-007 uses),
|
||
not a downloaded GGUF. This is required to keep the default gate deterministic,
|
||
download-free, and GPU-free. Real concurrent throughput on a downloaded
|
||
dense-Llama (CPU/ROCm) belongs to DGR-010 (blocked — no certified dense-Llama
|
||
artifact on this machine; see `evidence/DGR-010/BLOCKED.md`) and the final
|
||
comparison in DGR-014.
|
||
- **Batching is a scheduling grouping in this reference.** `KvBatchEngine.decode_batch`
|
||
runs each batch member sequentially through the cached decode (each attends only
|
||
its own KV, exactly like an independent llama.cpp sequence). The pinned llama.cpp
|
||
worker (DGR-008) fuses the batch into one `llama_decode` graph; the scheduling
|
||
semantics — one batch per tick, isolated positions/outputs — are identical. The
|
||
numbers here are *scheduler* quantities (ticks, batch occupancy, tokens/tick)
|
||
that are real and deterministic; **actual kernel-level batching speedup is a
|
||
native-worker property and is NOT claimed here** (RALPH performance discipline:
|
||
no unmeasured speed claims). It is measured in DGR-008/DGR-010/DGR-014.
|
||
- **Greedy sampling only.** Reuses the DGR-006 greedy `SamplingContract`. Greedy
|
||
over isolated per-session KV is order-independent, which is exactly why the
|
||
corruption check can assert byte-identical outputs across concurrency levels.
|
||
Stochastic sampling is out of scope for the deterministic gate.
|
||
- **Single loaded shard / single recipe per scheduler.** The scheduler batches
|
||
compatible sessions of one loaded shard (one `recipe_fingerprint`), which is the
|
||
node-local case. Multi-range routes batch at the head node whose adapter owns the
|
||
final head; cross-node coordination stays in the Meshnet control plane.
|
||
- **Native / llama.cpp gates N/A.** No native code, CMake, or llama.cpp patch was
|
||
touched (same as DGR-005/006/007), so those gates do not apply to this story.
|
||
|
||
## Compatibility / migration notes
|
||
|
||
- Purely additive: no existing module changed, so no behavior of the Torch/GGUF
|
||
backends, tracker, or KV manager is altered. The scheduler is opt-in — a server
|
||
constructs it around a `KvBatchEngine` when it wants continuous batching.
|
||
- `SchedulerTelemetry.to_dict()` is JSON-safe and aligns with the capability-signal
|
||
vocabulary (active sessions, queue depth, batch occupancy, KV pressure,
|
||
prefill/decode rates, rejected admissions) that a node advertises upward; it can
|
||
be folded into the DGR-009 capability report / heartbeat without schema changes
|
||
here.
|
||
- `AdmissionReason` values are stable strings suitable for the native protocol's
|
||
structured status / backpressure signalling.
|
||
|
||
## Handoff for dependent stories
|
||
|
||
- **DGR-008 (C++ gRPC worker):** implement the `BatchEngine` contract natively —
|
||
`decode_batch` becomes one `llama_decode` over the sessions' filtered sequences;
|
||
`prefill`/`release` map to the same KV manager operations. The scheduler,
|
||
admission budgets, fairness policy, and telemetry are unchanged; only the engine
|
||
swaps from numpy to llama.cpp.
|
||
- **DGR-010 (local real two-process acceptance, blocked):** once a certified
|
||
dense-Llama artifact is mounted, drive `run_concurrency_sweep` (or the scheduler
|
||
directly) with a real `KvBatchEngine` over the GGUF backend to produce
|
||
real-hardware occupancy/throughput/KV-pressure numbers under
|
||
`MESHNET_ENABLE_REAL_INFERENCE_TESTS=1` / `.venv-rocm`.
|
||
- **DGR-013 (failure/cancel/restart):** the `DoneReason.CACHE_MISS` path (a decode
|
||
whose KV was evicted marks the session done and re-prefillable) and the KV-release
|
||
on completion are the unit basis for the cancellation/cleanup matrix.
|
||
- **DGR-014 (release gate):** feed the real-hardware sweep’s aggregate throughput
|
||
and saturation point into the immutable DGR-001 comparison; do not reuse these
|
||
synthetic numbers as a performance claim.
|