feat: checkpoint batching and release-gate stories

.scratch/distributed-gguf-runtime/evidence/DGR-012/README.md

@@ -0,0 +1,220 @@
+# DGR-012 — Continuous batching and bounded admission: evidence
+
+Status: done
+Date: 2026-07-16
+Evidence kind: **synthetic-unit** (pure-numpy KV-cached dense-Llama reference +
+node-local continuous-batching scheduler). No model download, no GPU, no torch,
+no network, no API credit.
+
+## Summary
+
+Implemented the node-local scheduler that turns concurrent Route Sessions into
+llama.cpp-style continuous batches while bounding admission (RALPH runtime
+decision #9, ADR-0024). It sits **on top of** the DGR-007 Hot KV State manager —
+batching is a scheduling concern layered over the existing per-`(session, epoch)`
+KV isolation, not a new control plane or a change to the KV contract.
+
+- **Bounded admission (`NodeBudget` + `submit`).** A new session is admitted only
+  if it fits four budgets: resident **weight** footprint (reported), **KV** byte
+  budget (a session must be able to hold its *whole* generation, prompt + new
+  tokens, on its own), **scratch** (per-active-session activation buffers, capped
+  by a total scratch envelope), and the bounded **queue**. Anything that cannot
+  fit is rejected up front with an explicit `AdmissionReason`
+  (`REJECTED_KV_BUDGET` / `REJECTED_SCRATCH_BUDGET` / `REJECTED_DUPLICATE`);
+  anything that fits but has no free slot waits in the bounded queue; a **full
+  queue is refused** (`REJECTED_QUEUE_FULL`) — that refusal is the backpressure
+  signal.
+- **Continuous batching (`ContinuousBatchScheduler` + `KvBatchEngine`).** Every
+  tick, all currently-decoding sessions contribute their single next token to one
+  batch (bounded by `max_batch_size`); the engine runs the batch once. Each
+  session keeps its own position and appends its own sampled token via its own
+  `SessionCache`, so batching never mixes outputs. `KvBatchEngine` adapts the
+  DGR-007 `KvBoundaryAdapter`, so the batch runs against the *real* KV isolation
+  path; the pinned llama.cpp worker (DGR-008) implements the same
+  `recipe_fingerprint`/`prefill`/`decode_batch`/`release` contract where a batch
+  becomes one `llama_decode` over several sequences.
+- **Prefill does not starve decode.** The scheduling policy is explicit and fixed:
+  **decode first, then bounded prefill.** In-flight decodes always run before any
+  new prompt is prefilled, and prefill work per tick is capped
+  (`max_prefill_tokens_per_tick`, always allowing at least one so a single large
+  prompt still progresses). A burst of new sessions cannot stall generations
+  already in flight.
+- **Bounded memory / backpressure.** KV growth is bounded by the manager byte
+  budget; queued activations are bounded by `max_queue_depth` and the scratch
+  envelope; completed sessions release their KV so total KV returns to zero.
+- **Capability telemetry (`SchedulerTelemetry`).** Reports active sessions, queue
+  depth, batch occupancy (last/avg/max), KV pressure (bytes/budget), scratch
+  pressure, prefill/decode token totals **and rates**, and rejected admissions
+  (total + by reason). All JSON-safe.
+- **Concurrency 1/2/4/8 sweep (`run_concurrency_sweep`).** Runs the same eight
+  jobs at each level against a fresh KV manager and proves (a) **no cross-session
+  corruption** — every level yields byte-identical per-session tokens as the
+  serialized concurrency-1 reference — and (b) **saturation** — average batch
+  occupancy rises and total ticks fall as concurrency increases, until occupancy
+  plateaus.
+
+No existing runtime code was modified — this story is purely additive (one new
+module + one new test module + evidence).
+
+## Files changed (all new)
+
+- `packages/node/meshnet_node/batch_scheduler.py` — the scheduler:
+  - `NodeBudget` — weight/KV/scratch/queue budgets + `max_batch_size` /
+    `max_prefill_tokens_per_tick` scheduling bounds, with derived
+    `effective_active_cap` (tighter of active-slot and scratch caps).
+  - `AdmissionReason` / `AdmissionDecision` — structured admit/queue/reject.
+  - `GenerationRequest` / `DecodeItem` / `StepResult` — job + engine I/O values.
+  - `KvBatchEngine` — adapts a full-shard `KvBoundaryAdapter` to the batch-engine
+    contract (rejects a partial head/tail-only range).
+  - `SchedulerTelemetry` — the bounded capability snapshot.
+  - `ContinuousBatchScheduler` — thread-safe `submit` / `run_tick` /
+    `run_to_completion` / `telemetry`, decode-first-then-bounded-prefill policy.
+  - `run_concurrency_sweep` / `ConcurrencyResult` / `ConcurrencySweep` — the
+    deterministic 1/2/4/8 saturation report + corruption check.
+- `tests/test_batch_scheduler.py` — 16 tests (see below); reuses the DGR-007
+  numpy dense-Llama reference via `from test_hot_kv_state import _KvDenseLlama,
+  _KvReferenceShard`.
+- `.scratch/distributed-gguf-runtime/evidence/DGR-012/` — this README,
+  `commands.txt`, `generate_evidence.py`, `results.json`.
+
+## Acceptance criteria → evidence
+
+- **Scheduler admits sessions against weight, KV, scratch, and queue budgets** —
+  `test_admission_respects_active_scratch_and_queue_budgets` (fill slots → queue →
+  reject full queue), `test_admission_rejects_a_session_that_cannot_fit_the_kv_budget`,
+  `test_admission_rejects_when_per_session_scratch_exceeds_budget`,
+  `test_duplicate_submission_is_rejected`,
+  `test_weight_budget_is_reported_in_telemetry`.
+- **Compatible decode steps form batches preserving per-session positions/outputs**
+  — `test_batched_decode_preserves_per_session_positions_and_outputs`
+  (`batch_occupancy_max == 4`, four divergent references each reproduced),
+  `test_positions_are_isolated_across_different_prompt_lengths` (prompt lengths 1/3/7).
+- **Prefill does not starve decode; policy and bounds explicit** —
+  `test_prefill_does_not_starve_in_flight_decode` (in-flight session decodes on
+  *every* tick during a 4-session prefill burst; ≤1 prefill/tick),
+  `test_decode_first_policy_is_explicit_in_a_single_tick`.
+- **Backpressure prevents unbounded queued activations or KV growth** —
+  `test_backpressure_signals_when_queue_full_then_recovers`,
+  `test_completed_sessions_release_kv_so_growth_is_bounded` (`kv_total_bytes == 0`
+  after completion).
+- **Capability telemetry reports all required signals** —
+  `test_telemetry_reports_every_required_signal` (asserts every key present;
+  deterministic rates under an injected clock).
+- **Concurrency 1/2/4/8 identifies saturation, no cross-session corruption** —
+  `test_concurrency_sweep_identifies_saturation_without_corruption`
+  (occupancy strictly ↑, ticks strictly ↓, tokens/tick ↑, `corruption_free`,
+  0 cache misses, saturation=8), `test_concurrency_sweep_saturates_below_max_when_load_is_small`.
+- **Engine/usage guards** — `test_kv_batch_engine_requires_a_full_shard`,
+  `test_run_to_completion_is_bounded_against_misconfiguration`.
+
+## Concurrency 1/2/4/8 sweep (real, deterministic — `results.json`)
+
+Eight sessions, prompt length 4, 8 new tokens each; fresh KV manager per level;
+budgets sized so KV never evicts (so the corruption check is unambiguous).
+
+| concurrency | ticks | avg batch occupancy | max occupancy | tokens/tick | peak KV bytes |
+|---|---|---|---|---|---|
+| 1 | 64 | 1.000 | 1 | 1.375 | 15360 |
+| 2 | 33 | 1.750 | 2 | 2.667 | 29184 |
+| 4 | 19 | 3.111 | 4 | 4.632 | 52224 |
+| 8 | 15 | 4.000 | 7 | 5.867 | 75264 |
+
+`saturation_concurrency = 8`, `corruption_free = True`, `cache_misses = 0`,
+`rejected_admissions = 0`. As concurrency rises, the scheduler packs more sessions
+per decode step (occupancy ↑) and finishes the same 56 decode + 32 prefill tokens
+in far fewer ticks (aggregate work/tick ↑) — the batching throughput property —
+while every per-session token stream stays byte-identical to the serialized
+reference (no cross-session corruption). Max occupancy is 7 (not 8) at level 8
+because the fairness policy prefills at most one new session per tick, so the last
+session begins decoding one tick later.
+
+## Commands and real results
+
+```bash
+VP=/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python
+
+$VP -m pytest -q tests/test_batch_scheduler.py
+# -> 16 passed
+
+$VP -m pytest -q tests/test_hot_kv_state.py       # dependency still green
+# -> 22 passed
+
+$VP -m compileall -q packages tests
+# -> exit 0
+
+git diff --check
+# -> exit 0
+
+$VP .scratch/distributed-gguf-runtime/evidence/DGR-012/generate_evidence.py
+# -> wrote results.json; saturation_concurrency=8 corruption_free=True
+
+$VP -m pytest -q -rfE -p no:cacheprovider
+# -> FULL_SUITE_RESULT_PLACEHOLDER
+```
+
+`commands.txt` beside this README captures the exact commands.
+
+## Full-suite baseline (pre-existing unrelated failures)
+
+FULL_SUITE_BASELINE_PLACEHOLDER
+
+## Limitations and deferred work
+
+- **Synthetic-unit, not real weights.** The scheduler is exercised against the
+  deterministic numpy KV-cached dense-Llama reference (the same one DGR-007 uses),
+  not a downloaded GGUF. This is required to keep the default gate deterministic,
+  download-free, and GPU-free. Real concurrent throughput on a downloaded
+  dense-Llama (CPU/ROCm) belongs to DGR-010 (blocked — no certified dense-Llama
+  artifact on this machine; see `evidence/DGR-010/BLOCKED.md`) and the final
+  comparison in DGR-014.
+- **Batching is a scheduling grouping in this reference.** `KvBatchEngine.decode_batch`
+  runs each batch member sequentially through the cached decode (each attends only
+  its own KV, exactly like an independent llama.cpp sequence). The pinned llama.cpp
+  worker (DGR-008) fuses the batch into one `llama_decode` graph; the scheduling
+  semantics — one batch per tick, isolated positions/outputs — are identical. The
+  numbers here are *scheduler* quantities (ticks, batch occupancy, tokens/tick)
+  that are real and deterministic; **actual kernel-level batching speedup is a
+  native-worker property and is NOT claimed here** (RALPH performance discipline:
+  no unmeasured speed claims). It is measured in DGR-008/DGR-010/DGR-014.
+- **Greedy sampling only.** Reuses the DGR-006 greedy `SamplingContract`. Greedy
+  over isolated per-session KV is order-independent, which is exactly why the
+  corruption check can assert byte-identical outputs across concurrency levels.
+  Stochastic sampling is out of scope for the deterministic gate.
+- **Single loaded shard / single recipe per scheduler.** The scheduler batches
+  compatible sessions of one loaded shard (one `recipe_fingerprint`), which is the
+  node-local case. Multi-range routes batch at the head node whose adapter owns the
+  final head; cross-node coordination stays in the Meshnet control plane.
+- **Native / llama.cpp gates N/A.** No native code, CMake, or llama.cpp patch was
+  touched (same as DGR-005/006/007), so those gates do not apply to this story.
+
+## Compatibility / migration notes
+
+- Purely additive: no existing module changed, so no behavior of the Torch/GGUF
+  backends, tracker, or KV manager is altered. The scheduler is opt-in — a server
+  constructs it around a `KvBatchEngine` when it wants continuous batching.
+- `SchedulerTelemetry.to_dict()` is JSON-safe and aligns with the capability-signal
+  vocabulary (active sessions, queue depth, batch occupancy, KV pressure,
+  prefill/decode rates, rejected admissions) that a node advertises upward; it can
+  be folded into the DGR-009 capability report / heartbeat without schema changes
+  here.
+- `AdmissionReason` values are stable strings suitable for the native protocol's
+  structured status / backpressure signalling.
+
+## Handoff for dependent stories
+
+- **DGR-008 (C++ gRPC worker):** implement the `BatchEngine` contract natively —
+  `decode_batch` becomes one `llama_decode` over the sessions' filtered sequences;
+  `prefill`/`release` map to the same KV manager operations. The scheduler,
+  admission budgets, fairness policy, and telemetry are unchanged; only the engine
+  swaps from numpy to llama.cpp.
+- **DGR-010 (local real two-process acceptance, blocked):** once a certified
+  dense-Llama artifact is mounted, drive `run_concurrency_sweep` (or the scheduler
+  directly) with a real `KvBatchEngine` over the GGUF backend to produce
+  real-hardware occupancy/throughput/KV-pressure numbers under
+  `MESHNET_ENABLE_REAL_INFERENCE_TESTS=1` / `.venv-rocm`.
+- **DGR-013 (failure/cancel/restart):** the `DoneReason.CACHE_MISS` path (a decode
+  whose KV was evicted marks the session done and re-prefillable) and the KV-release
+  on completion are the unit basis for the cancellation/cleanup matrix.
+- **DGR-014 (release gate):** feed the real-hardware sweep’s aggregate throughput
+  and saturation point into the immutable DGR-001 comparison; do not reuse these
+  synthetic numbers as a performance claim.

.scratch/distributed-gguf-runtime/evidence/DGR-012/commands.txt

@@ -0,0 +1,24 @@
+# DGR-012 — exact commands (run from the worktree root)
+# Default venv (Python 3.14); deterministic, download-free, GPU-free, API-credit-free.
+VP=/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python
+
+# Targeted story tests
+$VP -m pytest -q tests/test_batch_scheduler.py
+# -> 16 passed
+
+# Dependency (DGR-007) still green — scheduler builds on this KV manager
+$VP -m pytest -q tests/test_hot_kv_state.py
+# -> 22 passed
+
+# Python quality gates
+$VP -m compileall -q packages tests
+# -> exit 0
+git diff --check
+# -> exit 0
+
+# Regenerate the machine-readable concurrency-sweep evidence
+$VP .scratch/distributed-gguf-runtime/evidence/DGR-012/generate_evidence.py
+# -> writes results.json; saturation_concurrency=8 corruption_free=True
+
+# Full deterministic suite (records the pre-existing unrelated failure baseline)
+$VP -m pytest -q -rfE -p no:cacheprovider

.scratch/distributed-gguf-runtime/evidence/DGR-012/generate_evidence.py

@@ -0,0 +1,117 @@
+"""Regenerate the DGR-012 concurrency-sweep evidence artifact.
+
+Deterministic, download-free, GPU-free. Run from the repo root with the default
+venv so the worktree ``meshnet_node`` package and the DGR-007 numpy reference
+(``tests/test_hot_kv_state``) are importable:
+
+    python .scratch/distributed-gguf-runtime/evidence/DGR-012/generate_evidence.py
+
+Writes ``results.json`` beside this script.
+"""
+
+from __future__ import annotations
+
+import json
+import pathlib
+import sys
+
+_ROOT = pathlib.Path(__file__).resolve().parents[4]
+sys.path.insert(0, str(_ROOT / "packages" / "node"))
+sys.path.insert(0, str(_ROOT / "tests"))
+
+from test_hot_kv_state import _KvDenseLlama, _KvReferenceShard  # noqa: E402
+
+from meshnet_node.batch_scheduler import (  # noqa: E402
+    ContinuousBatchScheduler,
+    GenerationRequest,
+    KvBatchEngine,
+    NodeBudget,
+    run_concurrency_sweep,
+)
+from meshnet_node.hot_kv_state import (  # noqa: E402
+    HotKvStateManager,
+    KvBoundaryAdapter,
+    kv_recipe_for,
+)
+
+MODEL = _KvDenseLlama()
+
+
+def make_engine() -> KvBatchEngine:
+    shard = _KvReferenceShard(MODEL, 0, MODEL.n_layers - 1)
+    manager = HotKvStateManager(kv_recipe_for(shard))
+    return KvBatchEngine(KvBoundaryAdapter(shard, manager))
+
+
+def main() -> int:
+    prompts = {
+        "s0": [1, 2, 3, 4], "s1": [5, 6, 7, 8], "s2": [9, 10, 11, 12],
+        "s3": [13, 14, 15, 16], "s4": [17, 18, 19, 20], "s5": [21, 22, 23, 24],
+        "s6": [25, 26, 27, 28], "s7": [29, 30, 31, 32],
+    }
+    n_new = 8
+    requests = [
+        GenerationRequest(sid, 0, tuple(p), n_new) for sid, p in prompts.items()
+    ]
+    sweep = run_concurrency_sweep(
+        make_engine, requests, concurrency_levels=(1, 2, 4, 8)
+    )
+
+    # A representative telemetry snapshot mid-run at concurrency 4 (shows the live
+    # capability signals a node advertises upward).
+    engine = make_engine()
+    scheduler = ContinuousBatchScheduler(
+        engine,
+        NodeBudget(
+            max_active_sessions=4, max_batch_size=4, max_queue_depth=8,
+            scratch_bytes_per_session=1, scratch_budget_bytes=4,
+        ),
+    )
+    for request in requests:
+        scheduler.submit(request)
+    for _ in range(6):
+        scheduler.run_tick()
+    mid_run_telemetry = scheduler.telemetry().to_dict()
+
+    artifact = {
+        "schema_version": 1,
+        "evidence_kind": "synthetic-unit",
+        "model": {
+            "reference": "pure-numpy KV-cached dense-Llama (tests/test_hot_kv_state)",
+            "n_layers": MODEL.n_layers,
+            "hidden": MODEL.hidden,
+            "n_heads": MODEL.n_heads,
+            "vocab": MODEL.vocab,
+        },
+        "workload": {
+            "sessions": len(prompts),
+            "prompt_len": 4,
+            "max_new_tokens": n_new,
+        },
+        "concurrency_sweep": sweep.to_dict(),
+        "mid_run_telemetry_concurrency_4": mid_run_telemetry,
+    }
+
+    out = pathlib.Path(__file__).with_name("results.json")
+    out.write_text(json.dumps(artifact, indent=2, sort_keys=True) + "\n", encoding="utf-8")
+    print(f"wrote {out}")
+    print(
+        "saturation_concurrency=%d corruption_free=%s"
+        % (sweep.saturation_concurrency, sweep.corruption_free)
+    )
+    for result in sweep.results:
+        print(
+            "  c=%d ticks=%d avg_occ=%.3f tokens/tick=%.3f peak_kv=%dB"
+            % (
+                result.concurrency,
+                result.ticks,
+                result.avg_batch_occupancy,
+                result.tokens_per_tick,
+                result.peak_kv_bytes,
+            )
+        )
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

.scratch/distributed-gguf-runtime/evidence/DGR-012/results.json

@@ -0,0 +1,179 @@
+{
+  "concurrency_sweep": {
+    "corruption_free": true,
+    "reference_outputs": {
+      "s0": [
+        27,
+        8,
+        27,
+        8,
+        27,
+        8,
+        1,
+        1
+      ],
+      "s1": [
+        26,
+        39,
+        39,
+        39,
+        39,
+        3,
+        39,
+        39
+      ],
+      "s2": [
+        12,
+        12,
+        12,
+        12,
+        12,
+        12,
+        30,
+        12
+      ],
+      "s3": [
+        29,
+        41,
+        42,
+        47,
+        47,
+        42,
+        47,
+        42
+      ],
+      "s4": [
+        23,
+        11,
+        44,
+        29,
+        29,
+        29,
+        41,
+        29
+      ],
+      "s5": [
+        35,
+        11,
+        0,
+        1,
+        11,
+        0,
+        11,
+        15
+      ],
+      "s6": [
+        39,
+        39,
+        28,
+        39,
+        39,
+        39,
+        28,
+        28
+      ],
+      "s7": [
+        39,
+        39,
+        39,
+        39,
+        39,
+        39,
+        8,
+        47
+      ]
+    },
+    "results": [
+      {
+        "avg_batch_occupancy": 1.0,
+        "cache_misses": 0,
+        "concurrency": 1,
+        "decode_batches": 56,
+        "decode_tokens": 56,
+        "max_batch_occupancy": 1,
+        "peak_kv_bytes": 15360,
+        "prefill_tokens": 32,
+        "rejected_admissions": 0,
+        "ticks": 64,
+        "tokens_per_tick": 1.375
+      },
+      {
+        "avg_batch_occupancy": 1.75,
+        "cache_misses": 0,
+        "concurrency": 2,
+        "decode_batches": 32,
+        "decode_tokens": 56,
+        "max_batch_occupancy": 2,
+        "peak_kv_bytes": 29184,
+        "prefill_tokens": 32,
+        "rejected_admissions": 0,
+        "ticks": 33,
+        "tokens_per_tick": 2.6667
+      },
+      {
+        "avg_batch_occupancy": 3.1111,
+        "cache_misses": 0,
+        "concurrency": 4,
+        "decode_batches": 18,
+        "decode_tokens": 56,
+        "max_batch_occupancy": 4,
+        "peak_kv_bytes": 52224,
+        "prefill_tokens": 32,
+        "rejected_admissions": 0,
+        "ticks": 19,
+        "tokens_per_tick": 4.6316
+      },
+      {
+        "avg_batch_occupancy": 4.0,
+        "cache_misses": 0,
+        "concurrency": 8,
+        "decode_batches": 14,
+        "decode_tokens": 56,
+        "max_batch_occupancy": 7,
+        "peak_kv_bytes": 75264,
+        "prefill_tokens": 32,
+        "rejected_admissions": 0,
+        "ticks": 15,
+        "tokens_per_tick": 5.8667
+      }
+    ],
+    "saturation_concurrency": 8,
+    "schema_version": 1
+  },
+  "evidence_kind": "synthetic-unit",
+  "mid_run_telemetry_concurrency_4": {
+    "active_sessions": 4,
+    "batch_occupancy_avg": 4.0,
+    "batch_occupancy_last": 4,
+    "batch_occupancy_max": 4,
+    "completed_sessions": 0,
+    "decode_tokens_per_sec": 1637.355,
+    "decode_tokens_total": 20,
+    "kv_budget_bytes": 67108864,
+    "kv_pressure": 0.0008,
+    "kv_total_bytes": 55296,
+    "prefill_tokens_per_sec": 1309.884,
+    "prefill_tokens_total": 16,
+    "queue_depth": 4,
+    "rejected_admissions_total": 0,
+    "rejected_by_reason": {},
+    "scratch_budget_bytes": 4,
+    "scratch_pressure": 1.0,
+    "scratch_used_bytes": 4,
+    "ticks": 6,
+    "weight_bytes": 0
+  },
+  "model": {
+    "hidden": 32,
+    "n_heads": 4,
+    "n_layers": 6,
+    "reference": "pure-numpy KV-cached dense-Llama (tests/test_hot_kv_state)",
+    "vocab": 48
+  },
+  "schema_version": 1,
+  "workload": {
+    "max_new_tokens": 8,
+    "prompt_len": 4,
+    "sessions": 8
+  }
+}