neuron-tai/.scratch/distributed-gguf-runtime/evidence/DGR-020/README.md

DGR-020 evidence — run the controlled whole-model GGUF baseline

Completed: 2026-07-22 Branch: ralph/distributed-gguf-runtime Authority: .scratch/distributed-gguf-runtime/prd.json Dependency: DGR-019 (evidence/DGR-019/README.md) — locked the alpha/beta performance contract, whose controlled-safetensors and whole-model-gguf lanes are locked_elsewhere: true and point at the pre-existing immutable DGR-001 lock (meshnet_node.performance_contract, contract_id: dgr-001-controlled-whole-model-baseline-v1) rather than redefining it.

Objective

Per .scratch/distributed-gguf-runtime/issues/020-run-the-controlled-whole-model-gguf-baseline.md: execute the exact locked safetensors and whole-model llama.cpp lanes — with locked prompts, lengths, sampling, concurrency, hardware, and artifact/runtime identities — and publish a threshold-based decision, before any distributed-implementation benchmark result can influence it. Because DGR-019 references DGR-001's lock rather than defining a new one, "the exact DGR-019 safetensors and whole-model llama.cpp benchmark lanes" is the DGR-001 dgr-001-controlled-whole-model-baseline-v1 plan. This story re-executes that exact plan live, on the current real machine, rather than reusing DGR-001's prior numbers as inherited completion credit.

Pre-existing state found (not caused by this story)

Before any change, git status showed .scratch/distributed-gguf-runtime/prd.json already modified relative to HEAD (47bad0b). Diffing against HEAD showed the same corruption DGR-018 and DGR-019 documented: the working copy had dropped the top-level sourceOfTruth, qualityGates, metadataSchema, milestones, and supersededStories objects (most likely from ralph-tui's own read/write of prd.json, which round-trips only the fields it models). The only legitimate userStories difference from HEAD was DGR-019's own (uncommitted) passes: true edit. Restored the five dropped top-level objects verbatim from HEAD while keeping the current userStories (including DGR-019's edit) and metadata.updatedAt. tests/test_ralph_prd_schema.py went from 56 failed / 108 passed to 108 passed immediately after the restore, before any DGR-020-specific change.

Reproducibility verification before running

Every identity DGR-001/DGR-019 pinned was independently re-checked against the current real machine before the benchmark ran — nothing was assumed from prior evidence:

Identity	Pinned (DGR-001)	Measured now	Match
llama.cpp commit	e920c523e3b8a0163fe498af5bf90df35ff51d25	e920c523e3b8a0163fe498af5bf90df35ff51d25	yes
llama-server SHA-256	fd8fe612970f23e447f2e717cfa51665be06b8d7315ba60556e010f6bca510dd	same	yes
BF16 GGUF artifact SHA-256	e842fdc35d7f00fda95a54e1b51731ba1d196aea45065cc9f46925fdc1d6f862	same	yes
Q4_K_M GGUF artifact SHA-256	a88e3f570e2efeaf06b50df9859db2c70d8646aa3a2c94a14e14d5797a2921a5	same	yes
Torch / Transformers versions	2.10.0+rocm7.13.0a20260513 / 5.13.0	same	yes

The safetensors snapshot, both GGUF artifacts, the pinned llama-server binary, and the pinned Python runtime were all still present unmodified on /run/media/popov/DATA/llm/, so this session reused them exactly rather than reconverting or requantizing (which would itself have been a silent redefinition of an immutable artifact identity).

Real results — fresh run on real hardware

.scratch/distributed-gguf-runtime/evidence/DGR-020/benchmark-config.json and performance-contract.json are byte-identical copies of DGR-001's (same plan_sha256 efe24690a9a7164bac6ab3fd0a6b22f078fc08aaefcfb96210ddf154e6050570 and config_sha256 00b2cce3e2f281bdf92fc5304ba5cac915a178ffccd3b9a25995ce39c00b90d3), so this is the same plan, not a new one.

MESHNET_ENABLE_REAL_INFERENCE_TESTS=1 \
MESHNET_EVIDENCE_SIGNING_KEY=/home/popov/.config/neuron-tai/keys/dgr-001-evidence-ed25519.pem \
PYTHONPATH=packages/node .venv-rocm/bin/python -m meshnet_node.recipe_benchmark \
  --config .scratch/distributed-gguf-runtime/evidence/DGR-020/benchmark-config.json \
  --json-out .scratch/distributed-gguf-runtime/evidence/DGR-020/results.json \
  --summary-out .scratch/distributed-gguf-runtime/evidence/DGR-020/results.txt

All three recipes completed every request with zero failures, on CPU, fedora 7.0.14-101.fc43.x86_64, 32 logical CPUs:

Metric	Transformers BF16 (ref)	llama.cpp BF16	llama.cpp Q4_K_M	DGR-001 (prior run, same plan)
Decode tok/s, c=1	50.8	102.5	213.1	40.8 / 98.5 / 207.7
Aggregate decode tok/s, c=4	48.8	218.1	235.7	46.5 / 222.8 / 195.7
TTFT p50, c=1	32.9 ms	15.1 ms	17.3 ms	40.0 / 15.1 / 21.6 ms
Peak resident memory, c=1	1.93 GB	1.11 GB	0.54 GB	1.94 / 1.11 / 0.54 GB
Artifact size	1.00 GB	0.99 GB	0.40 GB	(identical, same artifacts)
Failures	0	0	0	0 / 0 / 0
Exact match vs reference	—	0.3333	0.00 (advisory)	0.3333
Mean similarity vs reference	—	0.9471	0.456 (advisory)	0.9471

Per-recipe measurements against the reference (baseline.json, contract-evaluation.json):

• llama-cpp-near-lossless-quality (BF16, quality lane): decode speedup 2.02x, aggregate throughput speedup (c=4) 4.47x, resident-memory ratio 0.574x, TTFT ratio 0.459x — but quality_pass: false (exact match 0.33 < required 0.90).
• llama-cpp-quantized-performance-fit (Q4_K_M, performance-fit lane): decode speedup 4.19x, aggregate throughput speedup (c=4) 4.83x, resident-memory ratio 0.280x, artifact-size ratio 0.398x, TTFT ratio 0.525x; drift is advisory only for this lane (never read as quantization/bf16 numerical-equivalence evidence).

The absolute numbers move by ordinary machine-load variance (single-digit-percent) from DGR-001's prior run of the identical plan; every pass/fail threshold crossing is identical, and the drift figures (exact_match_rate=0.3333, mean_similarity=0.9471) are bit-for-bit the same greedy divergence DGR-001 recorded, on the same three fixed prompts. This is a genuine independent reproduction, not a copy: results.json's provenance.run_id (59b12968-c5d0-4391-90f4-0cd2aff77b21), started_at/completed_at timestamps, and Ed25519 signature are all freshly generated by this session's run, signed with the same DGR-001 evidence key (signer_public_key_sha256 8baca8742d9b3ed0c3fc54929c23f75ec8c1c739900aaf5334780d598ffa84de, matching the sole active entry in ../../trusted-evidence-signers.json).

Gain attribution — quantization/model-fit versus runtime/transport/kernel

Per DGR-019's dgr_performance contract gain_attribution rule ("a speed or fit claim must cite which axis moved it"):

• Quantization/model-fit metrics (resident_memory_ratio, artifact_size_ratio, exact_match_rate, mean_similarity): the Q4_K_M recipe's memory win (0.280x) and size win (0.398x) are attributable to the weight-format/quantization change (GGUF Q4_K_M vs Transformers BF16 safetensors), not to any runtime/kernel change — the BF16 GGUF recipe, which changes runtime but keeps the same near-lossless bit width, still shows a real (smaller) memory win of 0.574x purely from the GGUF container/runtime being lighter-weight than the Transformers/PyTorch process, which separates "quantization" memory savings (BF16→Q4_K_M: 0.574x→0.280x) from "runtime/format" memory savings (safetensors→BF16 GGUF: 1.0x→0.574x). The quality-lane failure (exact_match_rate=0.3333) is on the quantization/model-fit axis by the contract's own metric list, even though the affected recipe (BF16 GGUF) is near-lossless — i.e. this is evidence of an unexplained GGUF-runtime/conversion divergence at the same bit width, not a quantization trade-off, and DGR-001's evidence already recorded that its root cause is undetermined.
• Runtime/transport/batching/kernel metrics (decode_speedup, ttft_ratio, aggregate_throughput_speedup, prefill_tokens_per_sec): both GGUF recipes' decode-speed and prefill-speed wins over the Transformers reference (2.02x/4.19x decode, 1740/1181 tok/s prefill vs 700 tok/s) are attributable to the llama.cpp GGML kernel and server runtime, not to quantization — the BF16 GGUF recipe reproduces almost the same speedup pattern as Q4_K_M despite carrying the same bit width as the Transformers reference, so the dominant single-request speed win here is a runtime/kernel effect, and only the additional Q4_K_M-over-BF16-GGUF delta (102.5→213.1 tok/s decode, ~2.08x) is attributable to quantization on top of that runtime effect. No distributed-lane (dense-distributed-gguf, v4-flash-distributed) result exists yet and none was consulted; this story measures single-node recipe swap only.

Failed / unavailable lanes

None. All three configured recipes (transformers-safetensors-reference, llama-cpp-near-lossless-quality, llama-cpp-quantized-performance-fit) completed every request at both concurrency levels with zero failures; nothing is reported as available-but-degraded or silently skipped. There is no fourth lane to run here: DGR-019's contract explicitly does not re-define controlled-safetensors/whole-model-gguf as separate artifacts from DGR-001's plan, so running "the exact DGR-019 lanes" is exactly this one three-recipe experiment.

Decision

contract-evaluation.json (evaluated with the unmodified, immutable meshnet_node.performance_contract v1 thresholds — min_decode_speedup=1.25, max_ttft_ratio=1.25, min_aggregate_throughput_speedup=1.25, max_resident_memory_ratio=0.75, min_quality_exact_match_rate=0.90, min_quality_mean_similarity=0.97, max_failure_rate=0.0) records:

speed_benefit: true
fit_benefit: true
quality_lane_pass: false
stop_condition_met: true
verdict: stop

Mapped to this story's go / optimize baseline / stop vocabulary: stop. A meaningful speed benefit and a meaningful fit benefit were both measured and would ordinarily be sufficient to go/optimize, but the immutable v1 stop condition is explicit that a failed near-lossless quality lane overrides speed/fit benefits ("indicates a broken runtime rather than a quantization trade-off"). This decision uses only the locked v1 thresholds and this session's freshly measured metrics; no threshold was changed, and no distributed-implementation result (DGR-024's gRPC harness or any other distributed-lane evidence) was read or ingested to produce it.

This reproduces DGR-001's original stop verdict on the same plan on the same real machine, confirming that verdict is stable over time and not an artifact of a single run.

Limitations

• This is a 0.5B CPU baseline (Qwen/Qwen2.5-0.5B-Instruct), the same generic model DGR-001 and DGR-019's locked_elsewhere reference use — not DeepSeek V4 Flash. DGR-019's evidence already recorded that a DeepSeek V4 Flash controlled-safetensors/whole-model-gguf baseline is not yet pinned; that is separate future work (see DGR-019's v4-flash-distributed.reference_ baseline note), not something this story's acceptance criteria ask it to create — it asks only to run the exact already-locked lanes, which are this DGR-001 plan.
  • The whole-model-gguf quality-lane exact-match divergence (0.33 vs 0.90 required) reproduces identically and remains unexplained; this story does not diagnose it further beyond confirming it reproduces (DGR-001's quality-parity-diagnosis.md documents the CPU-vs-ROCm split already known).
• Absolute timings are single-developer-machine measurements with ordinary run-to-run variance; the locked ratios/ratios-vs-threshold crossings are the durable evidence, not the raw absolute tok/s figures.
• No new GPU (ROCm) diagnostic was re-run in this session — DGR-001's existing GPU diagnostic is cited as prior evidence only; it uses a distinct signed run_configured_gpu_diagnostic/v1 producer that the v1 evaluator does not accept, so it cannot itself change the stop verdict above.

Files changed

• .scratch/distributed-gguf-runtime/evidence/DGR-020/benchmark-config.json (new) — byte-identical copy of DGR-001's locked plan.
• .scratch/distributed-gguf-runtime/evidence/DGR-020/performance-contract.json (new) — byte-identical copy of DGR-001's immutable v1 thresholds.
• .scratch/distributed-gguf-runtime/evidence/DGR-020/results.json / results.txt (new) — raw signed real evidence from this session's fresh run.
• .scratch/distributed-gguf-runtime/evidence/DGR-020/baseline.json / contract-evaluation.json (new) — distilled baseline and fail-closed v1 verdict for this session's run.
• .scratch/distributed-gguf-runtime/evidence/DGR-020/README.md (new, this file).
• .scratch/distributed-gguf-runtime/prd.json — restored the dropped top-level sourceOfTruth/qualityGates/metadataSchema/milestones/supersededStories objects (see above); marked DGR-020.passes = true with completionNotes.
• .scratch/distributed-gguf-runtime/issues/020-run-the-controlled-whole-model-gguf-baseline.md — regenerated via python scripts/ralph_prd_schema.py render to reflect passes: true.

No source or test files under packages/ or tests/ were changed by this story.

Commands and results

python3 scripts/ralph_prd_schema.py validate .scratch/distributed-gguf-runtime/prd.json

OK: 55 stories validated.

.venv-rocm/bin/python -m pytest -q tests/test_recipe_benchmark.py tests/test_dgr_performance_contract.py tests/test_ralph_prd_schema.py

164 passed in 0.69s

.venv-rocm/bin/python -m compileall -q packages tests

Exit code 0, no output (all files compile).

git diff --check

Exit code 0 (no whitespace errors).

Dependency handoff

DGR-054 (enforce the alpha gate) may cite this evidence when it fills in alpha.useful_speed.human_approval — this is fresh, independently-collected, signed real-hardware evidence that the controlled-safetensors/whole-model-gguf v1 contract still holds stop on the current machine, immediately before any distributed-lane result exists, but it is a 0.5B CPU baseline, not the DeepSeek V4 Flash target; DGR-044 must still pin the V4 Flash reference baseline separately before DGR-054/DGR-070 can judge dense-distributed-gguf/v4-flash-distributed against it. No threshold in either meshnet_node.performance_contract or meshnet_node.dgr_performance was changed by this story.