DGR-001 — Safetensors versus GGUF performance contract

Status: complete; immutable v1 verdict is stop.

DGR-001 successfully produced a controlled local-real CPU baseline. Completion means the experiment and decision contract are durable and verified; it does not mean the native GGUF track is approved to continue. The locked quality gate failed, so dependent runtime work requires a human decision or a new, explicitly versioned experiment/contract rather than silently weakening v1.

Controlled workload

Model: Qwen/Qwen2.5-0.5B-Instruct
Exact source revision: 7ae557604adf67be50417f59c2c2f167def9a775
Machine: fedora, Linux 7.0.14-101.fc43.x86_64, 32 logical CPUs
Device: CPU for every recipe; VRAM is therefore correctly reported as zero
Runtime reference: Transformers 5.13.0, PyTorch 2.10.0+rocm7.13.0a20260513, BF16 safetensors
GGUF runtime: llama.cpp version 9991, commit e920c523e3b8a0163fe498af5bf90df35ff51d25
Workload: three fixed short/medium/long prompts, greedy sampling, 32 output tokens, three repeats, two warmups, concurrency 1 and 4, 16 CPU threads
Evidence class: local-real

All artifacts are beneath /run/media/popov/DATA/llm/; no model artifact was created under /home.

Recipes and exact artifacts

Recipe	Artifact	SHA-256
Transformers BF16 reference	complete mounted Hugging Face snapshot	`e596e9d6205fdc9177569cccd7f8b471b058f66e3630c8e4326d5aad52bd18b6`
llama.cpp BF16 quality lane	`Qwen2.5-0.5B-Instruct-7ae5576-BF16.gguf`	`e842fdc35d7f00fda95a54e1b51731ba1d196aea45065cc9f46925fdc1d6f862`
llama.cpp Q4_K_M performance/fit lane	`Qwen2.5-0.5B-Instruct-7ae5576-Q4_K_M.gguf`	`a88e3f570e2efeaf06b50df9859db2c70d8646aa3a2c94a14e14d5797a2921a5`

The snapshot digest covers every sorted relative path, resolved size, and file byte, so tokenizer/config drift is included. The BF16 GGUF was converted directly from the exact snapshot while preserving BF16 weights. Q4_K_M was quantized from an exact-revision F16 conversion with the pinned quantizer. Runtime validation recomputes every declared digest before model loading.

Real results

All recipes completed every request with zero failures.

Metric	Transformers BF16	llama.cpp BF16	llama.cpp Q4_K_M
Decode tok/s, c=1	40.8	98.5	207.7
Aggregate decode tok/s, c=4	46.5	222.8	195.7
TTFT p50, c=1	40.0 ms	15.1 ms	21.6 ms
Peak resident memory, c=1	1.94 GB	1.11 GB	0.54 GB
Artifact size	1.00 GB	0.99 GB	0.40 GB
Failures	0	0	0

Against the reference, the eligible Q4_K_M lane measured:

single-request decode speedup: 5.10×;
concurrency-4 aggregate throughput speedup: 4.20×;
resident-memory ratio: 0.279×;
artifact-size ratio: 0.398×.

The near-lossless BF16 quality lane compared all three prompts but measured:

exact match: 0.3333 (v1 requires at least 0.90);
mean text similarity: 0.9471 (v1 requires at least 0.97).

Tokenization and stopping were controlled: every runtime saw the same prompt token counts and reported 31 post-TTFT decode tokens. The v1 mismatch is a real greedy-output divergence on two prompts, not missing coverage or a text-length artifact. Its root cause remains undetermined; no post-contract logit-tie claim is acceptance evidence. Therefore contract-evaluation.json records:

verdict: stop
quality_lane_pass: false
speed_benefit: true
fit_benefit: true
stop_condition_met: true

Thresholds were not changed after observing these results.

Post-contract parity and ROCm diagnostics

summarize-quality-parity.py verifies and separates two signed sources. The CPU v1 row uses CPU kernels and a Transformers BF16 oracle; it remains at 0.3333 exact match with an unexplained divergence. The ROCm row uses a different plan, GPU kernels, and a Transformers float32 oracle. In that narrower diagnostic, the same BF16 GGUF artifact matches all three 32-token sequences exactly (1.0 exact match and 1.0 similarity). No conversion corruption was observed in that three-sequence ROCm sample; this does not prove global conversion correctness or explain the CPU result.

A separate HIP build at commit e920c523 was compiled for gfx1151 and measured ROCm0: Radeon 8060S Graphics; its llama-server SHA-256 is b6bb4da687dbde86e243ba006cef05919b7b97255cd7e2371e1d451220aca139. A signed gpu-diagnostic profile measured zero failures:

GPU metric	Transformers BF16 ROCm	llama.cpp Q4 ROCm	Q4 ratio
Decode tok/s, c=1	81.12	251.25	3.10×
Aggregate decode tok/s, c=4	91.24	511.33	5.60×
TTFT p50, c=1	13.77 ms	11.80 ms	0.857×

The GPU report is signed under the distinct run_configured_gpu_diagnostic/v1 producer. The v1 evaluator rejects that producer even when its signature is valid. llama-server process VRAM remains unmeasured, so this diagnostic cannot replace or satisfy the immutable v1 contract. Its signed backend detail records the measured ROCm0: Radeon 8060S Graphics device and 25/25 offloaded layers.

Implementation

recipe_benchmark.py provides the runtime-neutral measurement core, true concurrency, continuous in-flight peak-memory sampling, percentile/throughput aggregation, failures, and output drift.
recipe_drivers.py provides opt-in Transformers and llama-server drivers, mounted-drive confinement, exact artifact/runtime verification, equal device/thread budgets, greedy-only validation, measured host provenance, a CPU-only v1 guard until process VRAM can be measured honestly, and a distinct signed GPU diagnostic profile that the v1 evaluator cannot accept.
Peak RSS is runtime-scoped: Transformers reports growth above its pre-runtime Python baseline, while llama.cpp reports its isolated server process tree. Both are sampled continuously during in-flight requests.
TTFT uses each runtime's prompt/first-token compute boundary; end-to-end HTTP, scheduling, and queue overhead remains in latency and queue_wait_ms.
The exact canonical plan SHA-256 locks prompts, model/revision, sampling, output length, repeats, warmups, and concurrency. The evaluator also requires equal prompt/decode token counts across recipes.
llama.cpp's predicted_n includes the first token while predicted_ms begins after it; the driver subtracts that token so decode throughput matches the Transformers inter-token convention.
performance_contract.py rejects wrong plans, unsigned or incorrectly signed real evidence, wrong config/artifact/runtime/backend/host bindings, missing recipes/concurrency, mixed model revisions, incomplete quality coverage, and failed references.
Every non-synthetic report is Ed25519-signed over the complete canonical JSON, including raw outcomes and metrics. The contract pins the public key and exact config SHA-256; the private key remains outside Git at mode 0600.
The signer fingerprint is independently anchored outside this evidence directory in ../../trusted-evidence-signers.json and checked by tests.
Quantized drift remains advisory. Only the near-lossless lane can satisfy the quality gate, and only performance-fit recipes can earn speed/fit benefits.

Evidence files

performance-contract.json — immutable v1 thresholds and stop condition
benchmark-config.json — exact real-run plan, drivers, artifacts, and hashes
results.json — raw machine-readable per-request and aggregate evidence
results.txt — human-readable benchmark summary
baseline.json — distilled measurements for later comparison
contract-evaluation.json — fail-closed v1 verdict
quality-parity-diagnosis.json / .md — run/device-scoped signed-evidence summary
summarize-quality-parity.py — verifies both evidence chains and regenerates it
gpu-diagnostic-config.json — exact ROCm diagnostic artifacts and runtimes
gpu-diagnostic-results.json / .txt — signed GPU outcomes and summary
commands.txt — reproducible conversion, benchmark, evaluation, and test commands
BLOCKED.md — downstream stop-condition handoff
known-unrelated-failure.md — clean-base reproduction of the tracker race
../../trusted-evidence-signers.json — repository-reviewed signer fingerprint

Verification

Targeted: 28 passed (5/5 consecutive focused runs)
Latest full suite: 755 passed, 13 skipped
Earlier full suite: 751 passed, 13 skipped
Current cancellation retry matrix, DGR-001: 4/5 passed
Earlier cancellation retry matrix, clean d904c40: 4/5 passed
compileall: passed
git diff --check: passed
Evidence JSON parse/integrity checks: passed

The intermittent tracker cancellation race reproduced at the same rate on the clean base and is retained in known-unrelated-failure.md; the final full suite completed green. DGR-001 changes no tracker/proxy files.

The earlier Ralph claim that the full suite was blocked by Protobuf 6.33.6 was invalid: it used Hermes Agent's internal venv. Verification above used the project .venv, which has the DGR-002-compatible runtime. Real inference used .venv-rocm Python 3.12.

Limitations and dependent-story handoff

The immutable contract result is a 0.5B CPU baseline. The separate Radeon diagnostic is real local GPU evidence, but neither result covers a large model, distributed execution, network transport, or a native shard worker.
A separate GGML_HIP=ON llama.cpp build exists and produced GPU timings, but llama-server process VRAM is not measurable by the current driver; GPU memory/fit claims therefore remain ineligible for v1.
Absolute timings are developer-machine measurements; locked ratios and raw artifacts are provided for reproducibility.
DGR-014 may consume v1 only with the exact plan/evidence requirements enforced by performance_contract.py.
DGR-004 and later native-runtime work must not treat DGR-001 completion as a promotion. V1 says stop; proceeding requires a human decision backed by a separately versioned GPU/large-model contract or a diagnosed quality fix.

9.9 KiB Raw Blame History Unescape Escape