DGR-001 — Safetensors versus GGUF performance contract

Status: complete; immutable v1 verdict is stop.

DGR-001 successfully produced a controlled local-real CPU baseline. Completion means the experiment and decision contract are durable and verified; it does not mean the native GGUF track is approved to continue. The locked quality gate failed, so dependent runtime work requires a human decision or a new, explicitly versioned experiment/contract rather than silently weakening v1.

Controlled workload

Model: Qwen/Qwen2.5-0.5B-Instruct
Exact source revision: 7ae557604adf67be50417f59c2c2f167def9a775
Machine: fedora, Linux 7.0.14-101.fc43.x86_64, 32 logical CPUs
Device: CPU for every recipe; VRAM is therefore correctly reported as zero
Runtime reference: Transformers 5.13.0, PyTorch 2.10.0+rocm7.13.0a20260513, BF16 safetensors
GGUF runtime: llama.cpp version 9991, commit e920c523e3b8a0163fe498af5bf90df35ff51d25
Workload: three fixed short/medium/long prompts, greedy sampling, 32 output tokens, three repeats, two warmups, concurrency 1 and 4, 16 CPU threads
Evidence class: local-real

All artifacts are beneath /run/media/popov/DATA/llm/; no model artifact was created under /home.

Recipes and exact artifacts

Recipe	Artifact	SHA-256
Transformers BF16 reference	complete mounted Hugging Face snapshot	`e596e9d6205fdc9177569cccd7f8b471b058f66e3630c8e4326d5aad52bd18b6`
llama.cpp BF16 quality lane	`Qwen2.5-0.5B-Instruct-7ae5576-BF16.gguf`	`e842fdc35d7f00fda95a54e1b51731ba1d196aea45065cc9f46925fdc1d6f862`
llama.cpp Q4_K_M performance/fit lane	`Qwen2.5-0.5B-Instruct-7ae5576-Q4_K_M.gguf`	`a88e3f570e2efeaf06b50df9859db2c70d8646aa3a2c94a14e14d5797a2921a5`

The snapshot digest covers every sorted relative path, resolved size, and file byte, so tokenizer/config drift is included. The BF16 GGUF was converted directly from the exact snapshot while preserving BF16 weights. Q4_K_M was quantized from an exact-revision F16 conversion with the pinned quantizer. Runtime validation recomputes every declared digest before model loading.

Real results

All recipes completed every request with zero failures.

Metric	Transformers BF16	llama.cpp BF16	llama.cpp Q4_K_M
Decode tok/s, c=1	40.8	98.5	207.7
Aggregate decode tok/s, c=4	46.5	222.8	195.7
TTFT p50, c=1	40.0 ms	15.1 ms	21.6 ms
Peak resident memory, c=1	1.94 GB	1.11 GB	0.54 GB
Artifact size	1.00 GB	0.99 GB	0.40 GB
Failures	0	0	0

Against the reference, the eligible Q4_K_M lane measured:

single-request decode speedup: 5.10×;
concurrency-4 aggregate throughput speedup: 4.20×;
resident-memory ratio: 0.279×;
artifact-size ratio: 0.398×.

The near-lossless BF16 quality lane compared all three prompts but measured:

exact match: 0.3333 (v1 requires at least 0.90);
mean text similarity: 0.9471 (v1 requires at least 0.97).

Tokenization and stopping were controlled: every runtime saw the same prompt token counts and reported 31 post-TTFT decode tokens. The mismatch is genuine greedy runtime divergence on two prompts, not missing coverage or a text-length artifact. Therefore contract-evaluation.json records:

verdict: stop
quality_lane_pass: false
speed_benefit: true
fit_benefit: true
stop_condition_met: true

Thresholds were not changed after observing these results.

Implementation

recipe_benchmark.py provides the runtime-neutral measurement core, true concurrency, continuous in-flight peak-memory sampling, percentile/throughput aggregation, failures, and output drift.
recipe_drivers.py provides opt-in Transformers and llama-server drivers, mounted-drive confinement, exact artifact/runtime verification, equal device/thread budgets, greedy-only validation, measured host provenance, and a CPU-only v1 guard until process VRAM can be measured honestly.
Peak RSS is runtime-scoped: Transformers reports growth above its pre-runtime Python baseline, while llama.cpp reports its isolated server process tree. Both are sampled continuously during in-flight requests.
TTFT uses each runtime's prompt/first-token compute boundary; end-to-end HTTP, scheduling, and queue overhead remains in latency and queue_wait_ms.
The exact canonical plan SHA-256 locks prompts, model/revision, sampling, output length, repeats, warmups, and concurrency. The evaluator also requires equal prompt/decode token counts across recipes.
llama.cpp's predicted_n includes the first token while predicted_ms begins after it; the driver subtracts that token so decode throughput matches the Transformers inter-token convention.
performance_contract.py rejects wrong plans, unsigned or incorrectly signed real evidence, wrong config/artifact/runtime/backend/host bindings, missing recipes/concurrency, mixed model revisions, incomplete quality coverage, and failed references.
Every non-synthetic report is Ed25519-signed over the complete canonical JSON, including raw outcomes and metrics. The contract pins the public key and exact config SHA-256; the private key remains outside Git at mode 0600.
Quantized drift remains advisory. Only the near-lossless lane can satisfy the quality gate, and only performance-fit recipes can earn speed/fit benefits.

Evidence files

performance-contract.json — immutable v1 thresholds and stop condition
benchmark-config.json — exact real-run plan, drivers, artifacts, and hashes
results.json — raw machine-readable per-request and aggregate evidence
results.txt — human-readable benchmark summary
baseline.json — distilled measurements for later comparison
contract-evaluation.json — fail-closed v1 verdict
commands.txt — reproducible conversion, benchmark, evaluation, and test commands
BLOCKED.md — downstream stop-condition handoff
known-unrelated-failure.md — clean-base reproduction of the tracker race

Verification

Targeted: 24 passed
Full suite: 751 passed, 13 skipped
Earlier cancellation retry matrix, DGR-001: 4/5 passed
Earlier cancellation retry matrix, clean d904c40: 4/5 passed
compileall: passed
git diff --check: passed
Evidence JSON parse/integrity checks: passed

An earlier intermittent tracker cancellation race reproduced at the same rate on the clean base and is retained in known-unrelated-failure.md; the final suite completed green. DGR-001 changes no tracker/proxy files.

The earlier Ralph claim that the full suite was blocked by Protobuf 6.33.6 was invalid: it used Hermes Agent's internal venv. Verification above used the project .venv, which has the DGR-002-compatible runtime. Real inference used .venv-rocm Python 3.12.

Limitations and dependent-story handoff

This is a 0.5B CPU baseline, not evidence for a large model, Radeon GPU, distributed execution, network transport, or native shard worker.
The installed llama.cpp build is CPU-only (GGML_HIP=OFF). No GPU comparison is claimed.
Absolute timings are developer-machine measurements; locked ratios and raw artifacts are provided for reproducibility.
DGR-014 may consume v1 only with the exact plan/evidence requirements enforced by performance_contract.py.
DGR-004 and later native-runtime work must not treat DGR-001 completion as a promotion. V1 says stop; proceeding requires a human decision backed by a separately versioned GPU/large-model contract or a diagnosed quality fix.

7.4 KiB Raw Blame History Unescape Escape