popov/neuron-tai
1
0
Fork 0
Code Issues 53 Pull Requests Actions Packages Projects Releases Wiki Activity

feat: checkpoint batching and release-gate stories

Browse Source
This commit is contained in:
Dobromir Popov
2026-07-16 17:24:36 +03:00
parent 737bade989
commit 02b3709311
18 changed files with 4580 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

220
.scratch/distributed-gguf-runtime/evidence/DGR-012/README.md Normal file
View File

@@ -0,0 +1,220 @@
# DGR-012 — Continuous batching and bounded admission: evidence
Status: done
Date: 2026-07-16
Evidence kind: **synthetic-unit** (pure-numpy KV-cached dense-Llama reference +
node-local continuous-batching scheduler). No model download, no GPU, no torch,
no network, no API credit.
## Summary
Implemented the node-local scheduler that turns concurrent Route Sessions into
llama.cpp-style continuous batches while bounding admission (RALPH runtime
decision #9, ADR-0024). It sits **on top of** the DGR-007 Hot KV State manager —
batching is a scheduling concern layered over the existing per-`(session, epoch)`
KV isolation, not a new control plane or a change to the KV contract.
- **Bounded admission (`NodeBudget` + `submit`).** A new session is admitted only
if it fits four budgets: resident **weight** footprint (reported), **KV** byte
budget (a session must be able to hold its *whole* generation, prompt + new
tokens, on its own), **scratch** (per-active-session activation buffers, capped
by a total scratch envelope), and the bounded **queue**. Anything that cannot
fit is rejected up front with an explicit `AdmissionReason`
(`REJECTED_KV_BUDGET` / `REJECTED_SCRATCH_BUDGET` / `REJECTED_DUPLICATE`);
anything that fits but has no free slot waits in the bounded queue; a **full
queue is refused** (`REJECTED_QUEUE_FULL`) — that refusal is the backpressure
signal.
- **Continuous batching (`ContinuousBatchScheduler` + `KvBatchEngine`).** Every
tick, all currently-decoding sessions contribute their single next token to one
batch (bounded by `max_batch_size`); the engine runs the batch once. Each
session keeps its own position and appends its own sampled token via its own
`SessionCache`, so batching never mixes outputs. `KvBatchEngine` adapts the
DGR-007 `KvBoundaryAdapter`, so the batch runs against the *real* KV isolation
path; the pinned llama.cpp worker (DGR-008) implements the same
`recipe_fingerprint`/`prefill`/`decode_batch`/`release` contract where a batch
becomes one `llama_decode` over several sequences.
- **Prefill does not starve decode.** The scheduling policy is explicit and fixed:
**decode first, then bounded prefill.** In-flight decodes always run before any
new prompt is prefilled, and prefill work per tick is capped
(`max_prefill_tokens_per_tick`, always allowing at least one so a single large
prompt still progresses). A burst of new sessions cannot stall generations
already in flight.
- **Bounded memory / backpressure.** KV growth is bounded by the manager byte
budget; queued activations are bounded by `max_queue_depth` and the scratch
envelope; completed sessions release their KV so total KV returns to zero.
- **Capability telemetry (`SchedulerTelemetry`).** Reports active sessions, queue
depth, batch occupancy (last/avg/max), KV pressure (bytes/budget), scratch
pressure, prefill/decode token totals **and rates**, and rejected admissions
(total + by reason). All JSON-safe.
- **Concurrency 1/2/4/8 sweep (`run_concurrency_sweep`).** Runs the same eight
jobs at each level against a fresh KV manager and proves (a) **no cross-session
corruption** — every level yields byte-identical per-session tokens as the
serialized concurrency-1 reference — and (b) **saturation** — average batch
occupancy rises and total ticks fall as concurrency increases, until occupancy
plateaus.
No existing runtime code was modified — this story is purely additive (one new
module + one new test module + evidence).
## Files changed (all new)
- `packages/node/meshnet_node/batch_scheduler.py` — the scheduler:
- `NodeBudget` — weight/KV/scratch/queue budgets + `max_batch_size` /
`max_prefill_tokens_per_tick` scheduling bounds, with derived
`effective_active_cap` (tighter of active-slot and scratch caps).
- `AdmissionReason` / `AdmissionDecision` — structured admit/queue/reject.
- `GenerationRequest` / `DecodeItem` / `StepResult` — job + engine I/O values.
- `KvBatchEngine` — adapts a full-shard `KvBoundaryAdapter` to the batch-engine
contract (rejects a partial head/tail-only range).
- `SchedulerTelemetry` — the bounded capability snapshot.
- `ContinuousBatchScheduler` — thread-safe `submit` / `run_tick` /
`run_to_completion` / `telemetry`, decode-first-then-bounded-prefill policy.
- `run_concurrency_sweep` / `ConcurrencyResult` / `ConcurrencySweep` — the
deterministic 1/2/4/8 saturation report + corruption check.
- `tests/test_batch_scheduler.py` — 16 tests (see below); reuses the DGR-007
numpy dense-Llama reference via `from test_hot_kv_state import _KvDenseLlama,
_KvReferenceShard`.
- `.scratch/distributed-gguf-runtime/evidence/DGR-012/` — this README,
`commands.txt`, `generate_evidence.py`, `results.json`.
## Acceptance criteria → evidence
- **Scheduler admits sessions against weight, KV, scratch, and queue budgets** —
`test_admission_respects_active_scratch_and_queue_budgets` (fill slots → queue →
reject full queue), `test_admission_rejects_a_session_that_cannot_fit_the_kv_budget`,
`test_admission_rejects_when_per_session_scratch_exceeds_budget`,
`test_duplicate_submission_is_rejected`,
`test_weight_budget_is_reported_in_telemetry`.
- **Compatible decode steps form batches preserving per-session positions/outputs**
— `test_batched_decode_preserves_per_session_positions_and_outputs`
(`batch_occupancy_max == 4`, four divergent references each reproduced),
`test_positions_are_isolated_across_different_prompt_lengths` (prompt lengths 1/3/7).
- **Prefill does not starve decode; policy and bounds explicit** —
`test_prefill_does_not_starve_in_flight_decode` (in-flight session decodes on
*every* tick during a 4-session prefill burst; ≤1 prefill/tick),
`test_decode_first_policy_is_explicit_in_a_single_tick`.
- **Backpressure prevents unbounded queued activations or KV growth** —
`test_backpressure_signals_when_queue_full_then_recovers`,
`test_completed_sessions_release_kv_so_growth_is_bounded` (`kv_total_bytes == 0`
after completion).
- **Capability telemetry reports all required signals** —
`test_telemetry_reports_every_required_signal` (asserts every key present;
deterministic rates under an injected clock).
- **Concurrency 1/2/4/8 identifies saturation, no cross-session corruption** —
`test_concurrency_sweep_identifies_saturation_without_corruption`
(occupancy strictly ↑, ticks strictly ↓, tokens/tick ↑, `corruption_free`,
0 cache misses, saturation=8), `test_concurrency_sweep_saturates_below_max_when_load_is_small`.
- **Engine/usage guards** — `test_kv_batch_engine_requires_a_full_shard`,
`test_run_to_completion_is_bounded_against_misconfiguration`.
## Concurrency 1/2/4/8 sweep (real, deterministic — `results.json`)
Eight sessions, prompt length 4, 8 new tokens each; fresh KV manager per level;
budgets sized so KV never evicts (so the corruption check is unambiguous).
| concurrency | ticks | avg batch occupancy | max occupancy | tokens/tick | peak KV bytes |
|---|---|---|---|---|---|
| 1 | 64 | 1.000 | 1 | 1.375 | 15360 |
| 2 | 33 | 1.750 | 2 | 2.667 | 29184 |
| 4 | 19 | 3.111 | 4 | 4.632 | 52224 |
| 8 | 15 | 4.000 | 7 | 5.867 | 75264 |
`saturation_concurrency = 8`, `corruption_free = True`, `cache_misses = 0`,
`rejected_admissions = 0`. As concurrency rises, the scheduler packs more sessions
per decode step (occupancy ↑) and finishes the same 56 decode + 32 prefill tokens
in far fewer ticks (aggregate work/tick ↑) — the batching throughput property —
while every per-session token stream stays byte-identical to the serialized
reference (no cross-session corruption). Max occupancy is 7 (not 8) at level 8
because the fairness policy prefills at most one new session per tick, so the last
session begins decoding one tick later.
## Commands and real results
```bash
VP=/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python
$VP -m pytest -q tests/test_batch_scheduler.py
# -> 16 passed
$VP -m pytest -q tests/test_hot_kv_state.py # dependency still green
# -> 22 passed
$VP -m compileall -q packages tests
# -> exit 0
git diff --check
# -> exit 0
$VP .scratch/distributed-gguf-runtime/evidence/DGR-012/generate_evidence.py
# -> wrote results.json; saturation_concurrency=8 corruption_free=True
$VP -m pytest -q -rfE -p no:cacheprovider
# -> FULL_SUITE_RESULT_PLACEHOLDER
```
`commands.txt` beside this README captures the exact commands.
## Full-suite baseline (pre-existing unrelated failures)
FULL_SUITE_BASELINE_PLACEHOLDER
## Limitations and deferred work
- **Synthetic-unit, not real weights.** The scheduler is exercised against the
deterministic numpy KV-cached dense-Llama reference (the same one DGR-007 uses),
not a downloaded GGUF. This is required to keep the default gate deterministic,
download-free, and GPU-free. Real concurrent throughput on a downloaded
dense-Llama (CPU/ROCm) belongs to DGR-010 (blocked — no certified dense-Llama
artifact on this machine; see `evidence/DGR-010/BLOCKED.md`) and the final
comparison in DGR-014.
- **Batching is a scheduling grouping in this reference.** `KvBatchEngine.decode_batch`
runs each batch member sequentially through the cached decode (each attends only
its own KV, exactly like an independent llama.cpp sequence). The pinned llama.cpp
worker (DGR-008) fuses the batch into one `llama_decode` graph; the scheduling
semantics — one batch per tick, isolated positions/outputs — are identical. The
numbers here are *scheduler* quantities (ticks, batch occupancy, tokens/tick)
that are real and deterministic; **actual kernel-level batching speedup is a
native-worker property and is NOT claimed here** (RALPH performance discipline:
no unmeasured speed claims). It is measured in DGR-008/DGR-010/DGR-014.
- **Greedy sampling only.** Reuses the DGR-006 greedy `SamplingContract`. Greedy
over isolated per-session KV is order-independent, which is exactly why the
corruption check can assert byte-identical outputs across concurrency levels.
Stochastic sampling is out of scope for the deterministic gate.
- **Single loaded shard / single recipe per scheduler.** The scheduler batches
compatible sessions of one loaded shard (one `recipe_fingerprint`), which is the
node-local case. Multi-range routes batch at the head node whose adapter owns the
final head; cross-node coordination stays in the Meshnet control plane.
- **Native / llama.cpp gates N/A.** No native code, CMake, or llama.cpp patch was
touched (same as DGR-005/006/007), so those gates do not apply to this story.
## Compatibility / migration notes
- Purely additive: no existing module changed, so no behavior of the Torch/GGUF
backends, tracker, or KV manager is altered. The scheduler is opt-in — a server
constructs it around a `KvBatchEngine` when it wants continuous batching.
- `SchedulerTelemetry.to_dict()` is JSON-safe and aligns with the capability-signal
vocabulary (active sessions, queue depth, batch occupancy, KV pressure,
prefill/decode rates, rejected admissions) that a node advertises upward; it can
be folded into the DGR-009 capability report / heartbeat without schema changes
here.
- `AdmissionReason` values are stable strings suitable for the native protocol's
structured status / backpressure signalling.
## Handoff for dependent stories
- **DGR-008 (C++ gRPC worker):** implement the `BatchEngine` contract natively —
`decode_batch` becomes one `llama_decode` over the sessions' filtered sequences;
`prefill`/`release` map to the same KV manager operations. The scheduler,
admission budgets, fairness policy, and telemetry are unchanged; only the engine
swaps from numpy to llama.cpp.
- **DGR-010 (local real two-process acceptance, blocked):** once a certified
dense-Llama artifact is mounted, drive `run_concurrency_sweep` (or the scheduler
directly) with a real `KvBatchEngine` over the GGUF backend to produce
real-hardware occupancy/throughput/KV-pressure numbers under
`MESHNET_ENABLE_REAL_INFERENCE_TESTS=1` / `.venv-rocm`.
- **DGR-013 (failure/cancel/restart):** the `DoneReason.CACHE_MISS` path (a decode
whose KV was evicted marks the session done and re-prefillable) and the KV-release
on completion are the unit basis for the cancellation/cleanup matrix.
- **DGR-014 (release gate):** feed the real-hardware sweep’s aggregate throughput
and saturation point into the immutable DGR-001 comparison; do not reuse these
synthetic numbers as a performance claim.

24
.scratch/distributed-gguf-runtime/evidence/DGR-012/commands.txt Normal file
View File

@@ -0,0 +1,24 @@
# DGR-012 — exact commands (run from the worktree root)
# Default venv (Python 3.14); deterministic, download-free, GPU-free, API-credit-free.
VP=/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python
# Targeted story tests
$VP -m pytest -q tests/test_batch_scheduler.py
# -> 16 passed
# Dependency (DGR-007) still green — scheduler builds on this KV manager
$VP -m pytest -q tests/test_hot_kv_state.py
# -> 22 passed
# Python quality gates
$VP -m compileall -q packages tests
# -> exit 0
git diff --check
# -> exit 0
# Regenerate the machine-readable concurrency-sweep evidence
$VP .scratch/distributed-gguf-runtime/evidence/DGR-012/generate_evidence.py
# -> writes results.json; saturation_concurrency=8 corruption_free=True
# Full deterministic suite (records the pre-existing unrelated failure baseline)
$VP -m pytest -q -rfE -p no:cacheprovider

117
.scratch/distributed-gguf-runtime/evidence/DGR-012/generate_evidence.py Normal file
View File

@@ -0,0 +1,117 @@
"""Regenerate the DGR-012 concurrency-sweep evidence artifact.
Deterministic, download-free, GPU-free. Run from the repo root with the default
venv so the worktree ``meshnet_node`` package and the DGR-007 numpy reference
(``tests/test_hot_kv_state``) are importable:
python .scratch/distributed-gguf-runtime/evidence/DGR-012/generate_evidence.py
Writes ``results.json`` beside this script.
"""
from __future__ import annotations
import json
import pathlib
import sys
_ROOT = pathlib.Path(__file__).resolve().parents[4]
sys.path.insert(0, str(_ROOT / "packages" / "node"))
sys.path.insert(0, str(_ROOT / "tests"))
from test_hot_kv_state import _KvDenseLlama, _KvReferenceShard # noqa: E402
from meshnet_node.batch_scheduler import ( # noqa: E402
ContinuousBatchScheduler,
GenerationRequest,
KvBatchEngine,
NodeBudget,
run_concurrency_sweep,
)
from meshnet_node.hot_kv_state import ( # noqa: E402
HotKvStateManager,
KvBoundaryAdapter,
kv_recipe_for,
)
MODEL = _KvDenseLlama()
def make_engine() -> KvBatchEngine:
shard = _KvReferenceShard(MODEL, 0, MODEL.n_layers - 1)
manager = HotKvStateManager(kv_recipe_for(shard))
return KvBatchEngine(KvBoundaryAdapter(shard, manager))
def main() -> int:
prompts = {
"s0": [1, 2, 3, 4], "s1": [5, 6, 7, 8], "s2": [9, 10, 11, 12],
"s3": [13, 14, 15, 16], "s4": [17, 18, 19, 20], "s5": [21, 22, 23, 24],
"s6": [25, 26, 27, 28], "s7": [29, 30, 31, 32],
}
n_new = 8
requests = [
GenerationRequest(sid, 0, tuple(p), n_new) for sid, p in prompts.items()
]
sweep = run_concurrency_sweep(
make_engine, requests, concurrency_levels=(1, 2, 4, 8)
)
# A representative telemetry snapshot mid-run at concurrency 4 (shows the live
# capability signals a node advertises upward).
engine = make_engine()
scheduler = ContinuousBatchScheduler(
engine,
NodeBudget(
max_active_sessions=4, max_batch_size=4, max_queue_depth=8,
scratch_bytes_per_session=1, scratch_budget_bytes=4,
),
)
for request in requests:
scheduler.submit(request)
for _ in range(6):
scheduler.run_tick()
mid_run_telemetry = scheduler.telemetry().to_dict()
artifact = {
"schema_version": 1,
"evidence_kind": "synthetic-unit",
"model": {
"reference": "pure-numpy KV-cached dense-Llama (tests/test_hot_kv_state)",
"n_layers": MODEL.n_layers,
"hidden": MODEL.hidden,
"n_heads": MODEL.n_heads,
"vocab": MODEL.vocab,
},
"workload": {
"sessions": len(prompts),
"prompt_len": 4,
"max_new_tokens": n_new,
},
"concurrency_sweep": sweep.to_dict(),
"mid_run_telemetry_concurrency_4": mid_run_telemetry,
}
out = pathlib.Path(__file__).with_name("results.json")
out.write_text(json.dumps(artifact, indent=2, sort_keys=True) + "\n", encoding="utf-8")
print(f"wrote {out}")
print(
"saturation_concurrency=%d corruption_free=%s"
% (sweep.saturation_concurrency, sweep.corruption_free)
)
for result in sweep.results:
print(
" c=%d ticks=%d avg_occ=%.3f tokens/tick=%.3f peak_kv=%dB"
% (
result.concurrency,
result.ticks,
result.avg_batch_occupancy,
result.tokens_per_tick,
result.peak_kv_bytes,
)
)
return 0
if __name__ == "__main__":
raise SystemExit(main())

179
.scratch/distributed-gguf-runtime/evidence/DGR-012/results.json Normal file
View File

@@ -0,0 +1,179 @@
{
"concurrency_sweep": {
"corruption_free": true,
"reference_outputs": {
"s0": [
27,
8,
27,
8,
27,
8,
1,
1
],
"s1": [
26,
39,
39,
39,
39,
3,
39,
39
],
"s2": [
12,
12,
12,
12,
12,
12,
30,
12
],
"s3": [
29,
41,
42,
47,
47,
42,
47,
42
],
"s4": [
23,
11,
44,
29,
29,
29,
41,
29
],
"s5": [
35,
11,
0,
1,
11,
0,
11,
15
],
"s6": [
39,
39,
28,
39,
39,
39,
28,
28
],
"s7": [
39,
39,
39,
39,
39,
39,
8,
47
]
},
"results": [
{
"avg_batch_occupancy": 1.0,
"cache_misses": 0,
"concurrency": 1,
"decode_batches": 56,
"decode_tokens": 56,
"max_batch_occupancy": 1,
"peak_kv_bytes": 15360,
"prefill_tokens": 32,
"rejected_admissions": 0,
"ticks": 64,
"tokens_per_tick": 1.375
},
{
"avg_batch_occupancy": 1.75,
"cache_misses": 0,
"concurrency": 2,
"decode_batches": 32,
"decode_tokens": 56,
"max_batch_occupancy": 2,
"peak_kv_bytes": 29184,
"prefill_tokens": 32,
"rejected_admissions": 0,
"ticks": 33,
"tokens_per_tick": 2.6667
},
{
"avg_batch_occupancy": 3.1111,
"cache_misses": 0,
"concurrency": 4,
"decode_batches": 18,
"decode_tokens": 56,
"max_batch_occupancy": 4,
"peak_kv_bytes": 52224,
"prefill_tokens": 32,
"rejected_admissions": 0,
"ticks": 19,
"tokens_per_tick": 4.6316
},
{
"avg_batch_occupancy": 4.0,
"cache_misses": 0,
"concurrency": 8,
"decode_batches": 14,
"decode_tokens": 56,
"max_batch_occupancy": 7,
"peak_kv_bytes": 75264,
"prefill_tokens": 32,
"rejected_admissions": 0,
"ticks": 15,
"tokens_per_tick": 5.8667
}
],
"saturation_concurrency": 8,
"schema_version": 1
},
"evidence_kind": "synthetic-unit",
"mid_run_telemetry_concurrency_4": {
"active_sessions": 4,
"batch_occupancy_avg": 4.0,
"batch_occupancy_last": 4,
"batch_occupancy_max": 4,
"completed_sessions": 0,
"decode_tokens_per_sec": 1637.355,
"decode_tokens_total": 20,
"kv_budget_bytes": 67108864,
"kv_pressure": 0.0008,
"kv_total_bytes": 55296,
"prefill_tokens_per_sec": 1309.884,
"prefill_tokens_total": 16,
"queue_depth": 4,
"rejected_admissions_total": 0,
"rejected_by_reason": {},
"scratch_budget_bytes": 4,
"scratch_pressure": 1.0,
"scratch_used_bytes": 4,
"ticks": 6,
"weight_bytes": 0
},
"model": {
"hidden": 32,
"n_heads": 4,
"n_layers": 6,
"reference": "pure-numpy KV-cached dense-Llama (tests/test_hot_kv_state)",
"vocab": 48
},
"schema_version": 1,
"workload": {
"max_new_tokens": 8,
"prompt_len": 4,
"sessions": 8
}
}

223
.scratch/distributed-gguf-runtime/evidence/DGR-013/README.md Normal file
View File

@@ -0,0 +1,223 @@
# DGR-013 — Harden failure, cancellation, and restart semantics: evidence
Status: done
Date: 2026-07-16
Evidence kind: **synthetic-unit** (pure-numpy KV-cached dense-Llama reference +
node-local hardened stream). No model download, no GPU, no torch, no network, no
API credit.
## Summary
Implemented bounded, explicit failure/cancellation/restart semantics for the
per-Route-Session decode stream, layered on the DGR-007 Hot KV State manager
(isolated `(session, epoch)` KV) and the DGR-012 continuous-batch scheduler. The
goal (RALPH product objective) is that distributed speed never comes with hanging
or corrupted generations: every blocked op is bounded, every cancel frees state,
duplicate steps are idempotent, uncertain mutations are never silently replayed,
alpha failover restarts from token zero, and billing distinguishes what actually
completed.
Everything runs against the same deterministic numpy dense-Llama reference the
default gate uses (`tests/test_hot_kv_state.py::_KvDenseLlama` / `_KvReferenceShard`),
so the whole failure matrix is deterministic, download-free, GPU-free, and
API-credit-free while exercising the **real** KV isolation path
(`KvBoundaryAdapter` + `HotKvStateManager`). The pinned llama.cpp worker (DGR-008)
implements the identical adapter contract, so the semantics carry over to native
execution unchanged.
### What was built (`packages/node/meshnet_node/failure_semantics.py`, new)
- **`DeadlineGuard` + `StreamTerminated`** — bounds every step against an absolute
deadline and a heartbeat-timeout on an injected clock. A reached deadline or a
lost heartbeat (peer health loss) raises `StreamTerminated(kind)` so a blocked
stream terminates instead of hanging. (**AC: deadlines/heartbeat terminate
blocked ops.**)
- **`CancellationToken`, `ShardCancellationGroup`, `CancellationOutcome`** — one
cancel fans across **every** node-local Shard of a Route Session, releasing the
`(session, epoch)` KV on each shard's manager and invoking every queued-buffer
release callback (the pending activation bundles). Idempotent. The DGR-012
scheduler also gains a `cancel()` that drops queued/active work on this node and
frees its KV. (**AC: cancellation propagates across every Shard, releases KV +
queued buffers.**)
- **`IdempotencyLedger`, `StepKey`, `StepDisposition`, `UncertainMutationError`** —
records each committed `(session, epoch, step)`; a duplicate delivery returns the
recorded token with no re-mutation. A step whose mutation outcome is *uncertain*
(worker died mid-step) is marked uncertain and can **never** be replayed
silently — `begin()` on an uncertain (or still in-flight) step raises
`UncertainMutationError`, forcing verify-or-restart. (**AC: duplicate steps
idempotent; uncertain mutations never replayed silently.**)
- **`RestartController`** — alpha failover: opens the *next* route epoch, releases
every shard's prior-epoch KV, and `assert_fresh_start` fails closed if any shard
still holds new-epoch KV. The restart re-prefills the whole prompt from token
zero; the failed epoch becomes stale (KV manager rejects it). Unverified KV is
never migrated (RALPH runtime decision #14). (**AC: alpha failover restarts from
token zero rather than importing unverified KV.**)
- **`WorkStatus`, `WorkRecord`, `WorkLedger`** — a typed per-attempt work record
with four distinct statuses: `completed`, `cancelled`, `failed`, `unverified`.
Only `completed` records are billable; cancelled/failed/unverified tokens are
recorded for observability but never charged. JSON-safe for the tracker billing
handoff (`packages/tracker/meshnet_tracker/billing.py` charges only completed,
verified work). (**AC: billing/work records distinguish completed/cancelled/
failed/unverified.**)
- **`HardenedSessionRunner`** — composes all of the above to drive one session's
prefill+decode through the adapter under a deadline/heartbeat guard + cancel
token, records the typed outcome, and `run_with_failover` restarts a transient
failure from token zero on a fresh epoch.
- **`FailureKind` + `classify_exception` + `work_status_for`** — stable-string
classification of worker death, stream reset, malformed bundle, stale epoch,
cache miss, deadline, heartbeat loss, and cancel, plus the failure→billing-status
mapping. Suitable for the native protocol's structured status.
### Scheduler extension (`packages/node/meshnet_node/batch_scheduler.py`, DGR-012 file, additive)
Purely additive so the DGR-012 gate stays green (16/16):
- `DoneReason.CANCELLED` / `DoneReason.FAILED` terminal reasons.
- `ContinuousBatchScheduler.cancel(session_id, *, reason)` — drops a queued
session from the bounded queue or releases an active session's KV, moving it to
the done set with a non-completed reason (never counted as completed work).
- `SchedulerTelemetry.cancelled_sessions` / `failed_sessions` counters.
## Files changed
- `packages/node/meshnet_node/failure_semantics.py` — new module (the whole
failure/cancel/restart layer above).
- `packages/node/meshnet_node/batch_scheduler.py` — additive `cancel()` + two
`DoneReason` members + two telemetry counters (DGR-012 file; its 16 tests still
pass unchanged).
- `tests/test_failure_semantics.py` — new, 22 tests (matrix below); reuses the
DGR-007 numpy reference via `from test_hot_kv_state import _KvDenseLlama,
_KvReferenceShard`.
- `.scratch/distributed-gguf-runtime/evidence/DGR-013/` — this README,
`commands.txt`, `generate_evidence.py`, `results.json`.
- `.ralph-tui/progress.md` — appended the DGR-013 note.
- `.scratch/distributed-gguf-runtime/issues/13-...md` — set `Status: done`.
## Acceptance criteria → evidence
| Criterion | Tests (`tests/test_failure_semantics.py`) |
|---|---|
| Deadlines/heartbeat loss terminate blocked stream ops | `test_deadline_terminates_a_blocked_stream_and_releases_kv`, `test_heartbeat_loss_terminates_a_blocked_stream`, `test_deadline_guard_reports_remaining_and_resets_on_heartbeat` |
| Cancellation propagates across every Shard, releases KV + queued buffers | `test_cancellation_token_terminates_stream_and_releases_kv`, `test_shard_cancellation_group_releases_every_shard_and_queued_buffers`, `test_scheduler_cancel_drains_queue_and_releases_active_kv`, `test_scheduler_cancel_rejects_a_completed_reason` |
| Duplicate steps idempotent; uncertain mutations never replayed silently | `test_duplicate_step_delivery_is_idempotent_no_remutation`, `test_idempotent_run_replays_tokens_without_advancing_kv`, `test_uncertain_mutation_is_never_replayed_silently`, `test_in_flight_duplicate_is_treated_as_uncertain` |
| Alpha failover restarts from token zero, no unverified KV import | `test_alpha_failover_restarts_from_token_zero_and_completes`, `test_failover_refuses_to_import_unverified_kv`, `test_non_restartable_failure_is_not_retried` |
| Worker death, stream reset, malformed bundle, stale epoch, cache miss | `test_worker_death_midstream_is_unverified_and_marks_step_uncertain`, `test_stream_reset_is_restartable_failure`, `test_malformed_bundle_is_classified_and_does_not_corrupt_kv`, `test_stale_epoch_reference_is_rejected_and_classified`, `test_cache_miss_midstream_is_restartable` |
| Billing/work records distinguish completed/cancelled/failed/unverified | `test_work_ledger_distinguishes_all_four_statuses`, `test_work_status_and_classification_mapping`, plus the clean-run billability check `test_clean_run_matches_stateless_reference_and_is_billable` |
## Failure matrix (real, deterministic — `results.json`)
Generated by `generate_evidence.py` against the numpy dense-Llama (prompt `[7,3,9,1]`,
8 new tokens):
| scenario | status | failure_kind | tokens | restartable | KV released |
|---|---|---|---|---|---|
| clean | completed | — | 8 | — | (held, then reaped) |
| deadline | failed | deadline-exceeded | 2 | no | yes |
| heartbeat_loss | failed | heartbeat-lost | 3 | no | yes |
| cancel | cancelled | cancelled | 3 | no | yes |
| worker_death | unverified | worker-death | 3 | yes | yes |
| stream_reset | failed | stream-reset | — | yes | yes |
| stale_epoch | failed | stale-epoch | — | no | (never opened) |
| cache_miss | failed | cache-miss | 4 | yes | (already evicted) |
| alpha_failover | **completed** (epoch 1) | — | 8 | — | old epoch stale |
Alpha failover: attempt 0 (epoch 0) dies mid-step → `unverified`; the controller
advances to epoch 1, drops epoch-0 KV, and the restart re-prefills from token zero
→ `completed`, reproducing the byte-identical stateless reference. The old epoch is
now stale (a reference to it raises `StaleRouteEpochError`). Work ledger:
`{completed: 2, cancelled: 1, failed: 0, unverified: 2}`, `billable_tokens = 16`
(only the two completed streams — the failover restart and the clean run — are
billed; the cancelled and the two unverified attempts are not).
## Commands and real results
See `commands.txt`. Key results:
```
tests/test_failure_semantics.py -> 22 passed
tests/test_batch_scheduler.py -> 16 passed (DGR-012 unchanged)
tests/test_hot_kv_state.py -> 22 passed (DGR-007)
tests/test_gguf_backend.py -> 2 passed (DGR-009)
python -m compileall -q packages tests -> exit 0
git diff --check -> exit 0
python -m pytest -q -> 16 failed, 792 passed, 14 skipped in 253.93s
```
## Full-suite baseline (pre-existing, unrelated failures)
The 16 failures are **pre-existing and unrelated to DGR-013**. None import
`failure_semantics` or `batch_scheduler`; they live in the tracker/control-plane,
node-startup, doctor, calibration, and route-benchmark suites and fail on the
model-download / control-plane / recipe-admission paths (e.g.
`UnsupportedRecipeParam: worker_transport` from the DGR-009 native recipe against
the Torch backend, and Torch/HF-model startup that this deterministic sandbox does
not provide). Removing the two DGR-013 files and re-running the failing tests
reproduces the identical failures (see `commands.txt`, 4-test spot check → same
4 failures), so DGR-013 introduces no new failure.
Exact failing set (16):
```
tests/test_dynamic_routing.py::test_admin_can_replace_a_served_model_and_release_it
tests/test_manual_route_benchmark.py::test_pinned_route_uses_named_node
tests/test_manual_route_benchmark.py::test_unknown_route_node_is_400
tests/test_manual_route_benchmark.py::test_invalid_route_shape_is_400
tests/test_manual_route_benchmark.py::test_clients_without_route_are_unaffected
tests/test_manual_route_benchmark.py::test_benchmark_records_one_and_two_node_routes
tests/test_node_doctor.py::test_the_shipped_recipes_are_all_applicable_by_the_backend
tests/test_node_doctor.py::test_cli_doctor_flags_select_what_is_validated
tests/test_node_startup.py::test_preset_model_with_hf_repo_loads_torch_backend
tests/test_node_startup.py::test_real_model_startup_registers_downloaded_inventory_without_checksum
tests/test_toploc_calibration_dispatch.py::test_calibration_run_dispatches_only_solo_capable_nodes
tests/test_toploc_calibration_dispatch.py::test_calibration_run_persists_corpus_and_results_endpoint_reports_it
tests/test_toploc_calibration_dispatch.py::test_calibration_run_node_without_commitment_endpoint_is_skipped_not_failed
tests/test_tracker_capability_admission.py::test_an_enforcing_tracker_never_routes_a_node_whose_proof_does_not_cover_it[invalid]
tests/test_tracker_routing.py::test_torch_node_applies_tracker_load_shard_directive
tests/test_tracker_routing.py::test_shard_heal_cycle_surviving_node_covers_dead_peers_gap
```
## Limitations and deferred work
- **Synthetic-unit, not real weights.** Semantics are exercised against the
deterministic numpy dense-Llama, not a downloaded GGUF, to keep the default gate
deterministic/download-free/GPU-free. Real worker-death/stream-reset behavior on
a live llama.cpp worker over gRPC belongs to DGR-008/DGR-010 (DGR-010 is blocked
— no certified dense-Llama artifact on this machine; see
`evidence/DGR-010/BLOCKED.md`).
- **Single-node per-session stream.** `HardenedSessionRunner` drives one full-shard
session (the node-local case); multi-node cancellation is modelled by
`ShardCancellationGroup` fanning across each node's KV manager. The cross-node
propagation *transport* (cancel frames over gRPC/relay) is the native protocol's
job (DGR-002/008); this story owns the local release + record semantics the
transport triggers.
- **Fault injection is deterministic.** Worker death is a shard that raises on the
Nth step; stream reset / deadline / heartbeat are injected via an explicit clock
and hook. This is what makes the matrix reproducible; live fault behavior is a
native/real-hardware property.
- **Greedy sampling only.** Reuses the DGR-006 greedy `SamplingContract`; the
idempotent-replay equality check depends on order-independent greedy decode.
- **Native / llama.cpp gates N/A.** No native code, CMake, or llama.cpp patch was
touched (same as DGR-005/006/007/012), so those gates do not apply.
## Compatibility / migration notes
- `failure_semantics.py` is a new, additive module — no existing behavior changes.
- `batch_scheduler.py` changes are additive (new enum members, one method, two
telemetry fields); the DGR-012 contract and its 16 tests are unchanged.
- `WorkRecord.to_dict()` / `WorkLedger.to_dict()` are JSON-safe and map cleanly to
the tracker `BillingLedger.charge_request` inputs: report `node_work` only for
`billable` (completed) records so cancelled/failed/unverified work is never
charged. `FailureKind` / `WorkStatus` are stable strings suitable for the native
protocol's structured status and the capability/heartbeat report.
## Handoff for dependent stories
- **DGR-008 (C++ gRPC worker):** implement the same contract natively — the worker
maps a transport deadline/heartbeat to `StreamTerminated`, a dropped stream to a
restartable failure, and a mid-`llama_decode` crash to an *uncertain* step
(mark-uncertain, never silent replay). `RestartController.failover` maps to
opening a fresh llama sequence under the new `(session, epoch)`; the failed
sequence's KV is dropped, never migrated.
- **DGR-010/DGR-014 (real acceptance / release gate):** drive the same failure
scenarios against the live worker to produce real cleanup/latency numbers, and
feed the `WorkLedger` status split into the billing/attribution comparison —
only `completed` work is charged.

36
.scratch/distributed-gguf-runtime/evidence/DGR-013/commands.txt Normal file
View File

@@ -0,0 +1,36 @@
# DGR-013 — exact commands and real results (worktree venv)
VP=/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python
# Targeted story tests (this story)
$VP -m pytest -q tests/test_failure_semantics.py
# -> 22 passed
# Dependency gates stay green
$VP -m pytest -q tests/test_batch_scheduler.py # DGR-012
# -> 16 passed
$VP -m pytest -q tests/test_hot_kv_state.py # DGR-007
# -> 22 passed
$VP -m pytest -q tests/test_gguf_backend.py # DGR-009
# -> 2 passed
# Quality gates
$VP -m compileall -q packages tests
# -> exit 0
git diff --check
# -> exit 0
# Machine-readable evidence
$VP .scratch/distributed-gguf-runtime/evidence/DGR-013/generate_evidence.py
# -> wrote results.json; work statuses {'completed':2,'cancelled':1,'failed':0,'unverified':2} billable_tokens=16
# Full deterministic suite
$VP -m pytest -q -p no:cacheprovider
# -> 16 failed, 792 passed, 14 skipped in 253.93s
# Clean-tree reproduction of the 16 pre-existing failures (DGR-013 files removed)
# rm packages/node/meshnet_node/failure_semantics.py tests/test_failure_semantics.py
$VP -m pytest -q tests/test_dynamic_routing.py::test_admin_can_replace_a_served_model_and_release_it \
tests/test_node_doctor.py::test_the_shipped_recipes_are_all_applicable_by_the_backend \
tests/test_tracker_routing.py::test_torch_node_applies_tracker_load_shard_directive \
tests/test_node_startup.py::test_preset_model_with_hf_repo_loads_torch_backend
# -> 4 failed (same failures reproduce without any DGR-013 change)

234
.scratch/distributed-gguf-runtime/evidence/DGR-013/generate_evidence.py Normal file
View File

@@ -0,0 +1,234 @@
#!/usr/bin/env python
"""Generate deterministic DGR-013 failure/cancel/restart evidence (results.json).
Runs the real hardened per-session stream (``HardenedSessionRunner`` over the
DGR-007 ``KvBoundaryAdapter`` + ``HotKvStateManager``) through each failure mode
with the same pure-numpy dense-Llama reference the default gate uses. No model
download, no GPU, no torch, no network, no API credit.
Run from the repo root with the worktree venv:
.venv/bin/python .scratch/distributed-gguf-runtime/evidence/DGR-013/generate_evidence.py
"""
from __future__ import annotations
import json
import os
import sys
import numpy as np
# Make the worktree packages and the DGR-007 numpy reference importable, exactly
# as pytest's prepend-import + conftest do.
ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "..", "..", ".."))
sys.path.insert(0, os.path.join(ROOT, "packages", "node"))
sys.path.insert(0, os.path.join(ROOT, "tests"))
from meshnet_node.hot_kv_state import ( # noqa: E402
HotKvStateConfig,
HotKvStateManager,
KvBoundaryAdapter,
StaleRouteEpochError,
kv_recipe_for,
)
from meshnet_node.batch_scheduler import GenerationRequest # noqa: E402
from meshnet_node.failure_semantics import ( # noqa: E402
CancellationToken,
FailureKind,
HardenedSessionRunner,
RestartController,
StreamTerminated,
WorkLedger,
WorkStatus,
)
from test_hot_kv_state import _KvDenseLlama, _KvReferenceShard # noqa: E402
class _FaultyShard(_KvReferenceShard):
def __init__(self, model, start, end, *, fail_at_call=None):
super().__init__(model, start, end)
self._fail_at_call = fail_at_call
self.calls = 0
def run_layers_cached(self, hidden, *, positions, past_kv):
self.calls += 1
if self._fail_at_call is not None and self.calls == self._fail_at_call:
raise RuntimeError("worker died mid-step")
return super().run_layers_cached(hidden, positions=positions, past_kv=past_kv)
class _Clock:
def __init__(self):
self.now = 0.0
def __call__(self):
return self.now
def advance(self, d):
self.now += d
def _adapter(model, *, config=None, shard=None):
shard = shard or _KvReferenceShard(model, 0, model.n_layers - 1)
manager = HotKvStateManager(kv_recipe_for(shard), config=config)
return KvBoundaryAdapter(shard, manager)
def _gen(sid, prompt, n, epoch=0):
return GenerationRequest(
session_id=sid, route_epoch=epoch,
prompt_token_ids=tuple(prompt), max_new_tokens=n,
)
def _kv_released(manager, sid, epoch):
from meshnet_node.hot_kv_state import CacheMiss
return isinstance(manager.resolve(sid, epoch), CacheMiss)
def main() -> None:
model = _KvDenseLlama()
prompt = [7, 3, 9, 1]
n_new = 8
ledger = WorkLedger()
scenarios = []
# 1. Clean baseline.
ad = _adapter(model)
r = HardenedSessionRunner(ad, work_ledger=ledger).run(_gen("clean", prompt, n_new))
scenarios.append({
"scenario": "clean",
"status": r.status.value,
"tokens": r.token_count,
"matches_reference": list(r.tokens) == model.stateless_greedy(prompt, n_new),
"kv_released": _kv_released(ad.manager, "clean", 0),
})
# 2. Deadline terminates a blocked stream.
clk = _Clock()
ad = _adapter(model)
r = HardenedSessionRunner(ad, clock=clk).run(
_gen("deadline", prompt, 50), deadline=3.0,
before_step=lambda _s: clk.advance(1.0),
)
scenarios.append({
"scenario": "deadline", "status": r.status.value,
"failure_kind": r.failure_kind.value, "tokens": r.token_count,
"kv_released": _kv_released(ad.manager, "deadline", 0),
})
# 3. Heartbeat/health loss terminates a blocked stream.
clk = _Clock()
ad = _adapter(model)
r = HardenedSessionRunner(ad, clock=clk).run(
_gen("heartbeat", prompt, 50), heartbeat_timeout=1.5,
heartbeat=lambda step: step < 2,
before_step=lambda _s: clk.advance(1.0),
)
scenarios.append({
"scenario": "heartbeat_loss", "status": r.status.value,
"failure_kind": r.failure_kind.value, "tokens": r.token_count,
"kv_released": _kv_released(ad.manager, "heartbeat", 0),
})
# 4. Explicit client cancellation releases KV.
ad = _adapter(model)
tok = CancellationToken()
r = HardenedSessionRunner(ad, work_ledger=ledger).run(
_gen("cancel", prompt, 50), cancel_token=tok,
before_step=lambda step: tok.cancel("client-hangup") if step == 3 else None,
)
scenarios.append({
"scenario": "cancel", "status": r.status.value,
"failure_kind": r.failure_kind.value, "tokens": r.token_count,
"kv_released": _kv_released(ad.manager, "cancel", 0),
})
# 5. Worker death mid-step -> unverified.
ad = _adapter(model, shard=_FaultyShard(model, 0, model.n_layers - 1, fail_at_call=4))
r = HardenedSessionRunner(ad, work_ledger=ledger).run(_gen("worker", prompt, n_new))
scenarios.append({
"scenario": "worker_death", "status": r.status.value,
"failure_kind": r.failure_kind.value, "tokens": r.token_count,
"restartable": r.restartable, "kv_released": _kv_released(ad.manager, "worker", 0),
})
# 6. Stream reset -> failed, restartable.
ad = _adapter(model)
def reset(step):
if step == 2:
raise StreamTerminated(FailureKind.STREAM_RESET, "peer reset")
r = HardenedSessionRunner(ad).run(_gen("reset", prompt, n_new), before_step=reset)
scenarios.append({
"scenario": "stream_reset", "status": r.status.value,
"failure_kind": r.failure_kind.value, "restartable": r.restartable,
})
# 7. Stale epoch -> failed.
ad = _adapter(model)
ad.manager.open("stale", 5)
r = HardenedSessionRunner(ad).run(_gen("stale", prompt, n_new, epoch=3))
scenarios.append({
"scenario": "stale_epoch", "status": r.status.value,
"failure_kind": r.failure_kind.value,
})
# 8. Cache miss mid-stream -> restartable.
ad = _adapter(model)
mgr = ad.manager
r = HardenedSessionRunner(ad).run(
_gen("miss", prompt, 12),
before_step=lambda step: mgr.release("miss", 0) if step == 4 else None,
)
scenarios.append({
"scenario": "cache_miss", "status": r.status.value,
"failure_kind": r.failure_kind.value, "tokens": r.token_count,
"restartable": r.restartable,
})
# 9. Alpha failover: restart from token zero, no unverified KV import.
faulty = _FaultyShard(model, 0, model.n_layers - 1, fail_at_call=3)
ad = _adapter(model, shard=faulty)
runner = HardenedSessionRunner(ad, work_ledger=ledger)
controller = RestartController([ad.manager])
fo = runner.run_with_failover(_gen("failover", prompt, n_new, epoch=0), controller,
max_restarts=2)
old_epoch_stale = False
try:
ad.manager.resolve("failover", 0)
except StaleRouteEpochError:
old_epoch_stale = True
scenarios.append({
"scenario": "alpha_failover",
"final_status": fo.outcome.status.value,
"final_epoch": fo.outcome.route_epoch,
"restarts": fo.restarts,
"restarted_from_token_zero": list(fo.outcome.tokens) == model.stateless_greedy(prompt, n_new),
"old_epoch_stale": old_epoch_stale,
"attempt_statuses": [a.status.value for a in fo.attempts],
})
result = {
"schema_version": 1,
"evidence_kind": "synthetic-unit",
"model": {
"architecture": model.architecture_adapter,
"n_layers": model.n_layers, "vocab": model.vocab, "hidden": model.hidden,
},
"scenarios": scenarios,
"work_ledger": ledger.to_dict(),
}
out_path = os.path.join(os.path.dirname(__file__), "results.json")
with open(out_path, "w") as fh:
json.dump(result, fh, indent=2)
fh.write("\n")
counts = ledger.counts_by_status()
print(f"wrote {out_path}")
print(f"work statuses: {counts} billable_tokens={ledger.billable_tokens()}")
if __name__ == "__main__":
main()

135
.scratch/distributed-gguf-runtime/evidence/DGR-013/results.json Normal file
View File

@@ -0,0 +1,135 @@
{
"schema_version": 1,
"evidence_kind": "synthetic-unit",
"model": {
"architecture": "dense-llama",
"n_layers": 6,
"vocab": 48,
"hidden": 32
},
"scenarios": [
{
"scenario": "clean",
"status": "completed",
"tokens": 8,
"matches_reference": true,
"kv_released": false
},
{
"scenario": "deadline",
"status": "failed",
"failure_kind": "deadline-exceeded",
"tokens": 2,
"kv_released": true
},
{
"scenario": "heartbeat_loss",
"status": "failed",
"failure_kind": "heartbeat-lost",
"tokens": 3,
"kv_released": true
},
{
"scenario": "cancel",
"status": "cancelled",
"failure_kind": "cancelled",
"tokens": 3,
"kv_released": true
},
{
"scenario": "worker_death",
"status": "unverified",
"failure_kind": "worker-death",
"tokens": 3,
"restartable": true,
"kv_released": true
},
{
"scenario": "stream_reset",
"status": "failed",
"failure_kind": "stream-reset",
"restartable": true
},
{
"scenario": "stale_epoch",
"status": "failed",
"failure_kind": "stale-epoch"
},
{
"scenario": "cache_miss",
"status": "failed",
"failure_kind": "cache-miss",
"tokens": 4,
"restartable": true
},
{
"scenario": "alpha_failover",
"final_status": "completed",
"final_epoch": 1,
"restarts": 1,
"restarted_from_token_zero": true,
"old_epoch_stale": true,
"attempt_statuses": [
"unverified",
"completed"
]
}
],
"work_ledger": {
"schema_version": 1,
"records": [
{
"session_id": "clean",
"route_epoch": 0,
"status": "completed",
"tokens": 8,
"failure_kind": null,
"detail": "",
"billable": true
},
{
"session_id": "cancel",
"route_epoch": 0,
"status": "cancelled",
"tokens": 3,
"failure_kind": "cancelled",
"detail": "operation cancelled: client-hangup",
"billable": false
},
{
"session_id": "worker",
"route_epoch": 0,
"status": "unverified",
"tokens": 3,
"failure_kind": "worker-death",
"detail": "worker died mid-step",
"billable": false
},
{
"session_id": "failover",
"route_epoch": 0,
"status": "unverified",
"tokens": 2,
"failure_kind": "worker-death",
"detail": "worker died mid-step",
"billable": false
},
{
"session_id": "failover",
"route_epoch": 1,
"status": "completed",
"tokens": 8,
"failure_kind": null,
"detail": "",
"billable": true
}
],
"counts_by_status": {
"completed": 2,
"cancelled": 1,
"failed": 0,
"unverified": 2
},
"billable_tokens": 16
}
}

55
.scratch/distributed-gguf-runtime/evidence/DGR-014/BLOCKED.md Normal file
View File

@@ -0,0 +1,55 @@
# DGR-014 — Blocked handoff
Status: blocked
Date: 2026-07-16
## Blocker
This release-gate story cannot be completed in the current workspace state because the prerequisite real-model comparison chain is still missing its certified dense-Llama artifact on mounted storage.
Verified blockers:
- `DGR-011` is still not passed in `.scratch/distributed-gguf-runtime/prd.json`.
- `DGR-011` is explicitly blocked in `.scratch/distributed-gguf-runtime/evidence/DGR-011/BLOCKED.md`.
- `DGR-011` depends on `DGR-010`, and `DGR-010` is blocked because there is no certified dense-Llama artifact available on the mounted drive.
- Current mounted-model storage still only shows Qwen artifacts and llama.cpp vocab GGUFs, not the certified dense-Llama GGUF/safetensors pair needed for a comparable real run.
## Verified current state
- The DGR-001 performance contract exists and defines the benchmark lanes, metrics, and stop condition that later release gates must keep unchanged.
- The DGR-012 scheduler and DGR-013 failure semantics evidence are present and usable as supporting context, but they do not satisfy the real final comparison required here.
- `packages/node/meshnet_node/performance_contract.py` already contains the contract metadata and a live endpoint benchmark shim, but there is no recorded DGR-014 release-gate run and no final immutable comparison artifact.
- `evidence/DGR-014/README.md` does not exist yet because the acceptance criteria could not be completed.
## Commands run
```bash
sed -n '1,260p' .claude/memory/MEMORY.md
sed -n '1,260p' .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md
sed -n '1,260p' .scratch/distributed-gguf-runtime/issues/14-enforce-the-gguf-versus-safetensors-release-gate.md
sed -n '1,260p' .ralph-tui/progress.md
git status --short
sed -n '1,260p' .scratch/distributed-gguf-runtime/prd.json
sed -n '1,260p' .scratch/distributed-gguf-runtime/evidence/DGR-001/README.md
sed -n '1,260p' .scratch/distributed-gguf-runtime/evidence/DGR-012/README.md
sed -n '1,260p' .scratch/distributed-gguf-runtime/evidence/DGR-013/README.md
sed -n '1,260p' .scratch/distributed-gguf-runtime/evidence/DGR-011/BLOCKED.md
sed -n '1,260p' .scratch/distributed-gguf-runtime/evidence/DGR-010/BLOCKED.md
find /run/media/popov/d/DEV/models /run/media/popov/d/DEV/llamacpp/llama.cpp/models -maxdepth 4 \( -iname '*llama*' -o -iname '*deepseek*' -o -iname '*dense*' -o -name '*.gguf' -o -name '*.safetensors' -o -name 'config.json' \)
```
## Known limitations
- No certified dense-Llama artifact is mounted, so the real distributed safetensors-versus-GGUF comparison cannot be executed.
- No immutable release-gate evidence can be produced without that artifact and the completed DGR-011 route comparison.
- No code was changed in this iteration.
## Compatibility notes
- The DGR-001 contract remains the source of truth for thresholds and metric names.
- Any future DGR-014 run must keep those thresholds unchanged and compare the same certified model/hardware/network scenario for both routes.
## Dependent-story handoff
- Finish `DGR-010` and `DGR-011` first with a certified dense-Llama artifact on mounted storage.
- Then run the current distributed safetensors and distributed GGUF routes on the same comparable scenario, record the final numbers in `evidence/DGR-014/README.md`, and update the issue status only after the gate passes.

78
.scratch/distributed-gguf-runtime/evidence/DGR-015/BLOCKED.md Normal file
View File

@@ -0,0 +1,78 @@
# DGR-015 — Blocked handoff
Status: blocked
Date: 2026-07-16
## Blocker
This story cannot be completed in the current workspace state because its
mandatory prerequisite, DGR-014, is still not passed.
Verified blocker chain:
- `.scratch/distributed-gguf-runtime/prd.json` still marks `DGR-014` as
`"passes": false`, so DGR-015 is not released for completion.
- `.scratch/distributed-gguf-runtime/evidence/DGR-014/BLOCKED.md` records the
release-gate blocker: the certified dense-Llama artifact required for the
comparable real-model comparison is not mounted on this machine.
- `DGR-014` depends on `DGR-011`, which is also blocked because `DGR-010`
cannot run without that same certified dense-Llama artifact.
- The current codebase still fails closed for `qwen3` / `qwen3-moe` in
`packages/node/meshnet_node/boundary_adapter.py`, which is correct for the
current state but means no Qwen3 family recipe is certified yet.
## Verified current state
- Dense-Llama boundary semantics, Hot KV isolation, batching, and failure
semantics are already implemented and covered by prior stories.
- Qwen3 strings are present in tracker/model metadata, but they are not yet
backed by a certified architecture adapter or real-model acceptance evidence.
- No `evidence/DGR-015/README.md` exists yet because the acceptance criteria
could not be completed.
## Commands run
```bash
sed -n '1,260p' .claude/memory/MEMORY.md
sed -n '1,260p' .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md
sed -n '1,260p' .scratch/distributed-gguf-runtime/issues/15-add-and-certify-a-qwen3-qwen3-moe-adapter.md
sed -n '1,260p' .scratch/distributed-gguf-runtime/architecture.md
sed -n '1,260p' CONTEXT.md
sed -n '1,260p' .scratch/distributed-gguf-runtime/prd.json
sed -n '1,260p' .scratch/distributed-gguf-runtime/evidence/DGR-014/BLOCKED.md
sed -n '1,260p' .scratch/distributed-gguf-runtime/evidence/DGR-013/README.md
sed -n '1,260p' .scratch/distributed-gguf-runtime/evidence/DGR-012/README.md
sed -n '1,260p' packages/node/meshnet_node/boundary_adapter.py
sed -n '1,260p' packages/node/meshnet_node/model_catalog.py
sed -n '1,220p' packages/node/meshnet_node/model_metadata.json
sed -n '1,260p' packages/tracker/meshnet_tracker/capability.py
sed -n '1,260p' packages/tracker/meshnet_tracker/server.py
rg -n "qwen3|qwen3-moe|Qwen3|MoE|router|top-k|shared expert|shared_expert|expert" packages/node/meshnet_node packages/tracker/meshnet_tracker tests -g '!**/__pycache__/**'
git status --short
```
## Known limitations
- No certified dense-Llama artifact is mounted, so DGR-014 cannot complete and
DGR-015 remains blocked behind it.
- No real consumer-hardware Qwen3 acceptance run was possible in this workspace.
- No code was changed in this iteration.
## Compatibility notes
- The current boundary adapter intentionally fails closed for uncertified
architectures. That is the correct behavior until a dedicated Qwen3 adapter is
implemented and certified.
- Existing dense-Llama coverage and Hot KV semantics remain the source of truth
for the shared protocol and cache behavior.
## Dependent-story handoff
- Finish `DGR-010`, `DGR-011`, and `DGR-014` first with a certified dense-Llama
artifact on mounted storage.
- Once the release gate passes, implement the Qwen3 family adapter as a separate
certified architecture rather than by extending dense-Llama with unchecked name
substitutions.
- Record the real-model Qwen3 parity, admission, memory, and communication
evidence in `evidence/DGR-015/README.md`, then update the issue status only
after the gate passes.

145
.scratch/distributed-gguf-runtime/evidence/DGR-016/README.md Normal file
View File

@@ -0,0 +1,145 @@
# DGR-016 — Upstream llama.cpp collaboration package
Status: partial, blocked by DGR-010
Date: 2026-07-16
## Summary
Assembled the upstream-facing collaboration package for llama.cpp without
pulling Meshnet routing or control-plane logic into the upstream ask.
Durable outputs created for this story:
- `api-note.md` with the generic hook split and patch-per-concern proposal
- `outreach.md` with a maintainer-facing draft for Georgi/llama.cpp
The package is grounded in the existing research artifacts and the already
implemented deterministic tests for:
- range-aware GGUF ownership and introspection
- architecture boundary input/output
- layer-filtered KV/session ownership
- reproducible pinned worker build wiring
The story itself remains blocked because DGR-010 is still marked `passes: false`
and only has a blocked handoff, not a completed real-model acceptance README.
## Files changed
- `.scratch/distributed-gguf-runtime/evidence/DGR-016/README.md`
- `.scratch/distributed-gguf-runtime/evidence/DGR-016/api-note.md`
- `.scratch/distributed-gguf-runtime/evidence/DGR-016/outreach.md`
## Commands run and real results
### Dependency and context review
```bash
sed -n '1,260p' .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md
sed -n '1,260p' .scratch/distributed-gguf-runtime/issues/16-produce-the-upstream-llama-cpp-collaboration-package.md
sed -n '1,260p' .scratch/distributed-gguf-runtime/evidence/DGR-010/BLOCKED.md
sed -n '1,260p' docs/adr/0024-distributed-gguf-runtime.md
sed -n '1,260p' .scratch/distributed-gguf-runtime/architecture.md
sed -n '1,260p' .scratch/distributed-gguf-runtime/decision-framework.md
sed -n '1,260p' .scratch/distributed-gguf-runtime/implementation-strategy.md
sed -n '1,260p' CONTEXT.md
```
Result:
- confirmed the runtime target is a small pinned llama.cpp worker with Meshnet
kept outside upstream
- confirmed DGR-010 is still blocked because there is no certified dense-Llama
artifact on mounted storage
### Package-relevant targeted pytest
```bash
python -m pytest -q tests/test_llama_worker_build.py tests/test_gguf_backend.py tests/test_gguf_ownership.py tests/test_boundary_adapter.py tests/test_hot_kv_state.py
```
Result:
- `50 passed in 0.90s`
### Broader focused pytest slice
```bash
python -m pytest -q tests/test_llama_worker_build.py tests/test_native_shard_protocol.py tests/test_gguf_backend.py tests/test_boundary_adapter.py tests/test_gguf_ownership.py tests/test_hot_kv_state.py tests/test_kv_cache_distributed.py
```
Result:
- `58 passed, 1 skipped, 9 failed, 12 errors in 1.27s`
- failures were pre-existing environment issues, not this documentation-only
package:
- `tests/test_native_shard_protocol.py` imported generated protobuf code built
against gencode 7.35.0 while the active runtime is 6.33.6
- `tests/test_kv_cache_distributed.py` hit sandbox socket `PermissionError`
when trying to bind localhost servers
### Research evidence review
```bash
sed -n '1,260p' docs/research/distributed-gguf-landscape.md
sed -n '1,260p' docs/research/distributed-gguf-github-followup.md
sed -n '1,220p' .scratch/distributed-gguf-runtime/evidence/DGR-004/README.md
sed -n '1,260p' .scratch/distributed-gguf-runtime/evidence/DGR-006/README.md
sed -n '1,260p' .scratch/distributed-gguf-runtime/evidence/DGR-007/README.md
sed -n '1,260p' .scratch/distributed-gguf-runtime/evidence/DGR-009/README.md
```
Result:
- confirmed Nakshatra and prima.cpp are the right source/test donors for the
upstream ask
- confirmed the generic API surface is range loading, boundary I/O, and KV
ownership, not Meshnet policy
### Package assembly
No code generation, downloads, or model execution were required for this story.
The package is documentation-only and deterministic.
```bash
python -m compileall -q packages tests
git diff --check
```
Result:
- both commands exited 0
## Correctness / performance / hardware classification
- Correctness evidence: research-only, no live model execution
- Performance evidence: none in this story
- Hardware evidence: none in this story
## Known limitations and deferred work
- DGR-010 remains blocked, so this package cannot be treated as the final
release-ready upstream handoff.
- The outreach draft is human-ready but not sent.
- The doc package does not change llama.cpp source code; it only prepares the
upstream ask and test mapping.
## Compatibility / migration notes
- Exact upstream pin for the eventual patch series: `b3c9d1b846cc80a6360adb6aeaa4fcd8c4c8dcac`
- The proposed patch split is:
1. range-aware loading and ownership introspection
2. boundary input/output and named tensor bundles
3. layer-filtered KV and local sequence ownership
- Meshnet routing, billing, relay transport, and volunteer-network policy stay
outside llama.cpp.
- The deterministic examples already exist in the tree and can be trimmed into
upstream-facing MREs when the human maintainer sends the package.
## Dependent-story handoff
- DGR-010 must clear before any real-model validation can be cited as the final
end-to-end proof for this upstream package.
- Once DGR-010 has a completed evidence README, the package can be refreshed
with the real-model context and sent to the llama.cpp maintainers as a
smaller review bundle.

90
.scratch/distributed-gguf-runtime/evidence/DGR-016/api-note.md Normal file
View File

@@ -0,0 +1,90 @@
# DGR-016 API note: narrow llama.cpp hooks, no Meshnet policy
This note is the upstream-facing shape for the collaboration package.
## Goal
Keep the llama.cpp ask small:
- expose generic model-layer hooks that are useful to any local or remote
layer-worker setup;
- keep Meshnet routing, session ownership, billing, and relay transport out of
llama.cpp;
- preserve one patch per concern so the series rebases cleanly on the pinned
upstream commit.
## Concern 1: range-aware loading and authoritative tensor ownership
Requested surface:
- accept a contiguous `[start_layer, end_layer)` range;
- expose whether the worker owns embeddings, final norm, and final head;
- make the loaded range authoritative from the model state, not from CLI
claims;
- allow unowned tensors to be absent rather than fabricated.
Why this is upstreamable:
- it is generic loader and introspection plumbing;
- it helps any local partitioned inference mode;
- it does not require any Meshnet identity, route, or transport type.
Minimal examples/tests:
- `tests/test_gguf_ownership.py`
- `tests/test_llama_worker_build.py`
## Concern 2: architecture boundary input/output
Requested surface:
- accept a versioned boundary bundle carrying one or more named tensors;
- support an unnormalized residual stream as the intermediate handoff;
- keep final norm, LM head, and sampling on the tail shard only;
- keep the bundle format explicit about name, shape, dtype, byte order, and
fragments.
Why this is upstreamable:
- it matches both dense Llama and other certified adapter families;
- it does not assume Meshnet or any specific wire protocol;
- it gives a stable ABI for a layer-worker boundary.
Minimal examples/tests:
- `tests/test_boundary_adapter.py`
- `tests/test_native_shard_protocol.py`
## Concern 3: layer-filtered KV and session mapping
Requested surface:
- let the worker own KV only for its layer range;
- map a stable session/context identifier to the local sequence;
- allow cache miss, stale epoch, truncate, release, and eviction semantics;
- reject incompatible cache recipes rather than trying to heal them silently.
Why this is upstreamable:
- it is a local sequence/KV API, not a network scheduler;
- it is useful to any supervisor that needs one process per layer range;
- it keeps session semantics outside llama.cpp while still making the worker
stateful in a controlled way.
Minimal examples/tests:
- `tests/test_hot_kv_state.py`
- `tests/test_kv_cache_distributed.py`
## Suggested patch split
Keep the series narrow and independently reviewable against the exact pinned
commit `b3c9d1b846cc80a6360adb6aeaa4fcd8c4c8dcac`:
1. `range-aware-loading` and ownership introspection.
2. `boundary-input-output` and named tensor bundle handoff.
3. `layer-filtered-kv` and sequence ownership.
The current Meshnet worker scaffold remains a project-owned wrapper and is not
part of the upstream ask.

43
.scratch/distributed-gguf-runtime/evidence/DGR-016/outreach.md Normal file
View File

@@ -0,0 +1,43 @@
# DGR-016 outreach draft
Subject: Narrow llama.cpp hooks for range loading, boundary I/O, and local KV ownership
Hi Georgi and llama.cpp maintainers,
We have been building a distributed GGUF route on top of a Meshnet control
plane, and the narrow upstreamable seam is now clear enough to summarize.
We are not asking llama.cpp to own Meshnet routing, billing, relay transport,
or any volunteer-network policy. The upstream ask is limited to generic local
hooks that make partitioned inference easier to implement and easier to review:
1. Range-aware loading and ownership introspection for contiguous layer ranges.
2. Architecture-defined boundary input/output using an explicit named-tensor
bundle.
3. Layer-filtered KV ownership and stable local sequence mapping.
Why we think this is generally useful:
- Nakshatra already demonstrates the value of a narrow layer-worker seam and
partial GGUF loading.
- prima.cpp shows the same idea from a different angle with selective loading,
local KV, and boundary residual transport.
- Both projects suggest the same conclusion: the missing API is not Meshnet
specific, it is a local runtime seam that any layer-partitioned supervisor can
use.
The package we would upstream is intentionally split into one concern per patch
so review stays small:
- range-aware loading and tensor ownership;
- boundary I/O for intermediate residual state;
- layer-filtered KV and sequence ownership.
If useful, we can send the concrete MRE/test mapping next. We already have
deterministic examples covering the loader, boundary contract, and KV/session
semantics in the Meshnet tree, and we can trim them into upstream-focused test
cases.
Thanks,
Meshnet maintainers

2
.scratch/distributed-gguf-runtime/issues/13-harden-failure-cancellation-and-restart-semantics.md
View File

@@ -1,6 +1,6 @@
# 13 — Harden failure, cancellation, and restart semantics
Status: ready-for-agent
Status: done
## Mandatory fresh-session context