# DGR-020 evidence — run the controlled whole-model GGUF baseline

**Completed:** 2026-07-22
**Branch:** `ralph/distributed-gguf-runtime`
**Authority:** `.scratch/distributed-gguf-runtime/prd.json`
**Dependency:** DGR-019 (`evidence/DGR-019/README.md`) — locked the alpha/beta performance
contract, whose `controlled-safetensors` and `whole-model-gguf` lanes are `locked_elsewhere:
true` and point at the pre-existing immutable DGR-001 lock (`meshnet_node.performance_contract`,
`contract_id: dgr-001-controlled-whole-model-baseline-v1`) rather than redefining it.

## Objective

Per `.scratch/distributed-gguf-runtime/issues/020-run-the-controlled-whole-model-gguf-baseline.md`:
execute the exact locked safetensors and whole-model llama.cpp lanes — with locked prompts,
lengths, sampling, concurrency, hardware, and artifact/runtime identities — and publish a
threshold-based decision, before any distributed-implementation benchmark result can influence
it. Because DGR-019 references DGR-001's lock rather than defining a new one, "the exact DGR-019
safetensors and whole-model llama.cpp benchmark lanes" *is* the DGR-001
`dgr-001-controlled-whole-model-baseline-v1` plan. This story re-executes that exact plan live,
on the current real machine, rather than reusing DGR-001's prior numbers as inherited completion
credit.

## Pre-existing state found (not caused by this story)

Before any change, `git status` showed `.scratch/distributed-gguf-runtime/prd.json` already
modified relative to `HEAD` (`47bad0b`). Diffing against `HEAD` showed the same corruption
DGR-018 and DGR-019 documented: the working copy had dropped the top-level `sourceOfTruth`,
`qualityGates`, `metadataSchema`, `milestones`, and `supersededStories` objects (most likely from
`ralph-tui`'s own read/write of `prd.json`, which round-trips only the fields it models). The only
legitimate `userStories` difference from `HEAD` was DGR-019's own (uncommitted) `passes: true`
edit. Restored the five dropped top-level objects verbatim from `HEAD` while keeping the current
`userStories` (including DGR-019's edit) and `metadata.updatedAt`. `tests/test_ralph_prd_schema.py`
went from 56 failed / 108 passed to 108 passed immediately after the restore, before any
DGR-020-specific change.

## Reproducibility verification before running

Every identity DGR-001/DGR-019 pinned was independently re-checked against the current real
machine before the benchmark ran — nothing was assumed from prior evidence:

| Identity | Pinned (DGR-001) | Measured now | Match |
|---|---|---|---|
| llama.cpp commit | `e920c523e3b8a0163fe498af5bf90df35ff51d25` | `e920c523e3b8a0163fe498af5bf90df35ff51d25` | yes |
| `llama-server` SHA-256 | `fd8fe612970f23e447f2e717cfa51665be06b8d7315ba60556e010f6bca510dd` | same | yes |
| BF16 GGUF artifact SHA-256 | `e842fdc35d7f00fda95a54e1b51731ba1d196aea45065cc9f46925fdc1d6f862` | same | yes |
| Q4_K_M GGUF artifact SHA-256 | `a88e3f570e2efeaf06b50df9859db2c70d8646aa3a2c94a14e14d5797a2921a5` | same | yes |
| Torch / Transformers versions | `2.10.0+rocm7.13.0a20260513` / `5.13.0` | same | yes |

The safetensors snapshot, both GGUF artifacts, the pinned `llama-server` binary, and the pinned
Python runtime were all still present unmodified on `/run/media/popov/DATA/llm/`, so this session
reused them exactly rather than reconverting or requantizing (which would itself have been a
silent redefinition of an immutable artifact identity).

## Real results — fresh run on real hardware

`.scratch/distributed-gguf-runtime/evidence/DGR-020/benchmark-config.json` and
`performance-contract.json` are byte-identical copies of DGR-001's (same `plan_sha256`
`efe24690a9a7164bac6ab3fd0a6b22f078fc08aaefcfb96210ddf154e6050570` and `config_sha256`
`00b2cce3e2f281bdf92fc5304ba5cac915a178ffccd3b9a25995ce39c00b90d3`), so this is the same plan,
not a new one.

```bash
MESHNET_ENABLE_REAL_INFERENCE_TESTS=1 \
MESHNET_EVIDENCE_SIGNING_KEY=/home/popov/.config/neuron-tai/keys/dgr-001-evidence-ed25519.pem \
PYTHONPATH=packages/node .venv-rocm/bin/python -m meshnet_node.recipe_benchmark \
  --config .scratch/distributed-gguf-runtime/evidence/DGR-020/benchmark-config.json \
  --json-out .scratch/distributed-gguf-runtime/evidence/DGR-020/results.json \
  --summary-out .scratch/distributed-gguf-runtime/evidence/DGR-020/results.txt
```

All three recipes completed every request with zero failures, on CPU, `fedora`
`7.0.14-101.fc43.x86_64`, 32 logical CPUs:

| Metric | Transformers BF16 (ref) | llama.cpp BF16 | llama.cpp Q4_K_M | DGR-001 (prior run, same plan) |
|---|---:|---:|---:|---|
| Decode tok/s, c=1 | 50.8 | 102.5 | 213.1 | 40.8 / 98.5 / 207.7 |
| Aggregate decode tok/s, c=4 | 48.8 | 218.1 | 235.7 | 46.5 / 222.8 / 195.7 |
| TTFT p50, c=1 | 32.9 ms | 15.1 ms | 17.3 ms | 40.0 / 15.1 / 21.6 ms |
| Peak resident memory, c=1 | 1.93 GB | 1.11 GB | 0.54 GB | 1.94 / 1.11 / 0.54 GB |
| Artifact size | 1.00 GB | 0.99 GB | 0.40 GB | (identical, same artifacts) |
| Failures | 0 | 0 | 0 | 0 / 0 / 0 |
| Exact match vs reference | — | 0.3333 | 0.00 (advisory) | 0.3333 |
| Mean similarity vs reference | — | 0.9471 | 0.456 (advisory) | 0.9471 |

Per-recipe measurements against the reference (`baseline.json`, `contract-evaluation.json`):

- `llama-cpp-near-lossless-quality` (BF16, quality lane): decode speedup **2.02x**, aggregate
  throughput speedup (c=4) **4.47x**, resident-memory ratio **0.574x**, TTFT ratio **0.459x** —
  but `quality_pass: false` (exact match 0.33 < required 0.90).
- `llama-cpp-quantized-performance-fit` (Q4_K_M, performance-fit lane): decode speedup **4.19x**,
  aggregate throughput speedup (c=4) **4.83x**, resident-memory ratio **0.280x**, artifact-size
  ratio **0.398x**, TTFT ratio **0.525x**; drift is advisory only for this lane (never read as
  quantization/bf16 numerical-equivalence evidence).

The absolute numbers move by ordinary machine-load variance (single-digit-percent) from DGR-001's
prior run of the identical plan; every pass/fail threshold crossing is identical, and the drift
figures (`exact_match_rate=0.3333`, `mean_similarity=0.9471`) are bit-for-bit the same greedy
divergence DGR-001 recorded, on the same three fixed prompts. This is a genuine independent
reproduction, not a copy: `results.json`'s `provenance.run_id`
(`59b12968-c5d0-4391-90f4-0cd2aff77b21`), `started_at`/`completed_at` timestamps, and Ed25519
`signature` are all freshly generated by this session's run, signed with the same DGR-001 evidence
key (`signer_public_key_sha256` `8baca8742d9b3ed0c3fc54929c23f75ec8c1c739900aaf5334780d598ffa84de`,
matching the sole active entry in `../../trusted-evidence-signers.json`).

## Gain attribution — quantization/model-fit versus runtime/transport/kernel

Per DGR-019's `dgr_performance` contract `gain_attribution` rule ("a speed or fit claim must cite
which axis moved it"):

- **Quantization/model-fit metrics** (`resident_memory_ratio`, `artifact_size_ratio`,
  `exact_match_rate`, `mean_similarity`): the Q4_K_M recipe's memory win (0.280x) and size win
  (0.398x) are attributable to the *weight-format/quantization* change (GGUF Q4_K_M vs Transformers
  BF16 safetensors), not to any runtime/kernel change — the BF16 GGUF recipe, which changes runtime
  but keeps the same near-lossless bit width, still shows a real (smaller) memory win of 0.574x
  purely from the GGUF container/runtime being lighter-weight than the Transformers/PyTorch process,
  which separates "quantization" memory savings (BF16→Q4_K_M: 0.574x→0.280x) from "runtime/format"
  memory savings (safetensors→BF16 GGUF: 1.0x→0.574x). The quality-lane failure
  (`exact_match_rate=0.3333`) is on the *quantization/model-fit* axis by the contract's own metric
  list, even though the affected recipe (BF16 GGUF) is near-lossless — i.e. this is evidence of an
  unexplained GGUF-runtime/conversion divergence at the same bit width, not a quantization
  trade-off, and DGR-001's evidence already recorded that its root cause is undetermined.
- **Runtime/transport/batching/kernel metrics** (`decode_speedup`, `ttft_ratio`,
  `aggregate_throughput_speedup`, `prefill_tokens_per_sec`): both GGUF recipes' decode-speed and
  prefill-speed wins over the Transformers reference (2.02x/4.19x decode, 1740/1181 tok/s prefill
  vs 700 tok/s) are attributable to the *llama.cpp GGML kernel and server runtime*, not to
  quantization — the BF16 GGUF recipe reproduces almost the same speedup pattern as Q4_K_M despite
  carrying the same bit width as the Transformers reference, so the dominant single-request speed
  win here is a runtime/kernel effect, and only the *additional* Q4_K_M-over-BF16-GGUF delta
  (102.5→213.1 tok/s decode, ~2.08x) is attributable to quantization on top of that runtime effect.
  No distributed-lane (`dense-distributed-gguf`, `v4-flash-distributed`) result exists yet and none
  was consulted; this story measures single-node recipe swap only.

## Failed / unavailable lanes

None. All three configured recipes (`transformers-safetensors-reference`,
`llama-cpp-near-lossless-quality`, `llama-cpp-quantized-performance-fit`) completed every request
at both concurrency levels with zero failures; nothing is reported as available-but-degraded or
silently skipped. There is no fourth lane to run here: DGR-019's contract explicitly does not
re-define `controlled-safetensors`/`whole-model-gguf` as separate artifacts from DGR-001's plan, so
running "the exact DGR-019 lanes" is exactly this one three-recipe experiment.

## Decision

`contract-evaluation.json` (evaluated with the unmodified, immutable
`meshnet_node.performance_contract` v1 thresholds — `min_decode_speedup=1.25`,
`max_ttft_ratio=1.25`, `min_aggregate_throughput_speedup=1.25`, `max_resident_memory_ratio=0.75`,
`min_quality_exact_match_rate=0.90`, `min_quality_mean_similarity=0.97`, `max_failure_rate=0.0`)
records:

```text
speed_benefit: true
fit_benefit: true
quality_lane_pass: false
stop_condition_met: true
verdict: stop
```

Mapped to this story's `go` / `optimize baseline` / `stop` vocabulary: **stop**. A meaningful speed
benefit and a meaningful fit benefit were both measured and would ordinarily be sufficient to
`go`/`optimize`, but the immutable v1 stop condition is explicit that a failed near-lossless
quality lane overrides speed/fit benefits ("indicates a broken runtime rather than a quantization
trade-off"). This decision uses only the locked v1 thresholds and this session's freshly measured
metrics; no threshold was changed, and no distributed-implementation result (DGR-024's gRPC
harness or any other distributed-lane evidence) was read or ingested to produce it.

This reproduces DGR-001's original `stop` verdict on the same plan on the same real machine,
confirming that verdict is stable over time and not an artifact of a single run.

## Limitations

- This is a **0.5B CPU baseline** (`Qwen/Qwen2.5-0.5B-Instruct`), the same generic model DGR-001
  and DGR-019's `locked_elsewhere` reference use — not DeepSeek V4 Flash. DGR-019's evidence
  already recorded that a DeepSeek V4 Flash `controlled-safetensors`/`whole-model-gguf` baseline is
  not yet pinned; that is separate future work (see DGR-019's `v4-flash-distributed.reference_
  baseline` note), not something this story's acceptance criteria ask it to create — it asks only
  to run the exact already-locked lanes, which are this DGR-001 plan.
  - The `whole-model-gguf` quality-lane exact-match divergence (0.33 vs 0.90 required) reproduces
  identically and remains unexplained; this story does not diagnose it further beyond confirming
  it reproduces (DGR-001's `quality-parity-diagnosis.md` documents the CPU-vs-ROCm split already
  known).
- Absolute timings are single-developer-machine measurements with ordinary run-to-run variance;
  the locked ratios/ratios-vs-threshold crossings are the durable evidence, not the raw absolute
  tok/s figures.
- No new GPU (ROCm) diagnostic was re-run in this session — DGR-001's existing GPU diagnostic is
  cited as prior evidence only; it uses a distinct signed `run_configured_gpu_diagnostic/v1`
  producer that the v1 evaluator does not accept, so it cannot itself change the `stop` verdict
  above.

## Files changed

- `.scratch/distributed-gguf-runtime/evidence/DGR-020/benchmark-config.json` (new) — byte-identical
  copy of DGR-001's locked plan.
- `.scratch/distributed-gguf-runtime/evidence/DGR-020/performance-contract.json` (new) —
  byte-identical copy of DGR-001's immutable v1 thresholds.
- `.scratch/distributed-gguf-runtime/evidence/DGR-020/results.json` / `results.txt` (new) — raw
  signed real evidence from this session's fresh run.
- `.scratch/distributed-gguf-runtime/evidence/DGR-020/baseline.json` / `contract-evaluation.json`
  (new) — distilled baseline and fail-closed v1 verdict for this session's run.
- `.scratch/distributed-gguf-runtime/evidence/DGR-020/README.md` (new, this file).
- `.scratch/distributed-gguf-runtime/prd.json` — restored the dropped top-level
  `sourceOfTruth`/`qualityGates`/`metadataSchema`/`milestones`/`supersededStories` objects (see
  above); marked `DGR-020.passes = true` with `completionNotes`.
- `.scratch/distributed-gguf-runtime/issues/020-run-the-controlled-whole-model-gguf-baseline.md` —
  regenerated via `python scripts/ralph_prd_schema.py render` to reflect `passes: true`.

No source or test files under `packages/` or `tests/` were changed by this story.

## Commands and results

```bash
python3 scripts/ralph_prd_schema.py validate .scratch/distributed-gguf-runtime/prd.json
```
```text
OK: 55 stories validated.
```

```bash
.venv-rocm/bin/python -m pytest -q tests/test_recipe_benchmark.py tests/test_dgr_performance_contract.py tests/test_ralph_prd_schema.py
```
```text
164 passed in 0.69s
```

```bash
.venv-rocm/bin/python -m compileall -q packages tests
```
Exit code 0, no output (all files compile).

```bash
git diff --check
```
Exit code 0 (no whitespace errors).

## Dependency handoff

DGR-054 (enforce the alpha gate) may cite this evidence when it fills in
`alpha.useful_speed.human_approval` — this is fresh, independently-collected, signed real-hardware
evidence that the `controlled-safetensors`/`whole-model-gguf` v1 contract still holds `stop` on the
current machine, immediately before any distributed-lane result exists, but it is a 0.5B CPU
baseline, not the DeepSeek V4 Flash target; DGR-044 must still pin the V4 Flash reference baseline
separately before DGR-054/DGR-070 can judge `dense-distributed-gguf`/`v4-flash-distributed` against
it. No threshold in either `meshnet_node.performance_contract` or `meshnet_node.dgr_performance`
was changed by this story.
