neuron-tai/.scratch/distributed-gguf-runtime/evidence/DGR-001/README.md

DGR-001 — Safetensors versus GGUF performance contract

Status: complete; immutable v1 verdict is stop.

DGR-001 successfully produced a controlled local-real CPU baseline. Completion means the experiment and decision contract are durable and verified; it does not mean the native GGUF track is approved to continue. The locked quality gate failed, so dependent runtime work requires a human decision or a new, explicitly versioned experiment/contract rather than silently weakening v1.

Controlled workload

• Model: Qwen/Qwen2.5-0.5B-Instruct
• Exact source revision: 7ae557604adf67be50417f59c2c2f167def9a775
• Machine: fedora, Linux 7.0.14-101.fc43.x86_64, 32 logical CPUs
• Device: CPU for every recipe; VRAM is therefore correctly reported as zero
• Runtime reference: Transformers 5.13.0, PyTorch 2.10.0+rocm7.13.0a20260513, BF16 safetensors
• GGUF runtime: llama.cpp version 9991, commit e920c523e3b8a0163fe498af5bf90df35ff51d25
• Workload: three fixed short/medium/long prompts, greedy sampling, 32 output tokens, three repeats, two warmups, concurrency 1 and 4, 16 CPU threads
• Evidence class: local-real

All artifacts are beneath /run/media/popov/DATA/llm/; no model artifact was created under /home.

Recipes and exact artifacts

Recipe	Artifact	SHA-256
Transformers BF16 reference	complete mounted Hugging Face snapshot	e596e9d6205fdc9177569cccd7f8b471b058f66e3630c8e4326d5aad52bd18b6
llama.cpp BF16 quality lane	Qwen2.5-0.5B-Instruct-7ae5576-BF16.gguf	e842fdc35d7f00fda95a54e1b51731ba1d196aea45065cc9f46925fdc1d6f862
llama.cpp Q4_K_M performance/fit lane	Qwen2.5-0.5B-Instruct-7ae5576-Q4_K_M.gguf	a88e3f570e2efeaf06b50df9859db2c70d8646aa3a2c94a14e14d5797a2921a5

The snapshot digest covers every sorted relative path, resolved size, and file byte, so tokenizer/config drift is included. The BF16 GGUF was converted directly from the exact snapshot while preserving BF16 weights. Q4_K_M was quantized from an exact-revision F16 conversion with the pinned quantizer. Runtime validation recomputes every declared digest before model loading.

Real results

All recipes completed every request with zero failures.

Metric	Transformers BF16	llama.cpp BF16	llama.cpp Q4_K_M
Decode tok/s, c=1	40.8	98.5	207.7
Aggregate decode tok/s, c=4	46.5	222.8	195.7
TTFT p50, c=1	40.0 ms	15.1 ms	21.6 ms
Peak resident memory, c=1	1.94 GB	1.11 GB	0.54 GB
Artifact size	1.00 GB	0.99 GB	0.40 GB
Failures	0	0	0

Against the reference, the eligible Q4_K_M lane measured:

• single-request decode speedup: 5.10×;
• concurrency-4 aggregate throughput speedup: 4.20×;
• resident-memory ratio: 0.279×;
• artifact-size ratio: 0.398×.

The near-lossless BF16 quality lane compared all three prompts but measured:

• exact match: 0.3333 (v1 requires at least 0.90);
• mean text similarity: 0.9471 (v1 requires at least 0.97).

Tokenization and stopping were controlled: every runtime saw the same prompt token counts and reported 31 post-TTFT decode tokens. The mismatch is genuine greedy runtime divergence on two prompts, not missing coverage or a text-length artifact. Therefore contract-evaluation.json records:

verdict: stop
quality_lane_pass: false
speed_benefit: true
fit_benefit: true
stop_condition_met: true

Thresholds were not changed after observing these results.

Implementation

• recipe_benchmark.py provides the runtime-neutral measurement core, true concurrency, continuous in-flight peak-memory sampling, percentile/throughput aggregation, failures, and output drift.
• recipe_drivers.py provides opt-in Transformers and llama-server drivers, mounted-drive confinement, exact artifact/runtime verification, equal device/thread budgets, greedy-only validation, measured host provenance, and a CPU-only v1 guard until process VRAM can be measured honestly.
• Peak RSS is runtime-scoped: Transformers reports growth above its pre-runtime Python baseline, while llama.cpp reports its isolated server process tree. Both are sampled continuously during in-flight requests.
• TTFT uses each runtime's prompt/first-token compute boundary; end-to-end HTTP, scheduling, and queue overhead remains in latency and queue_wait_ms.
• The exact canonical plan SHA-256 locks prompts, model/revision, sampling, output length, repeats, warmups, and concurrency. The evaluator also requires equal prompt/decode token counts across recipes.
• llama.cpp's predicted_n includes the first token while predicted_ms begins after it; the driver subtracts that token so decode throughput matches the Transformers inter-token convention.
• performance_contract.py rejects wrong plans, unsigned or incorrectly signed real evidence, wrong config/artifact/runtime/backend/host bindings, missing recipes/concurrency, mixed model revisions, incomplete quality coverage, and failed references.
• Every non-synthetic report is Ed25519-signed over the complete canonical JSON, including raw outcomes and metrics. The contract pins the public key and exact config SHA-256; the private key remains outside Git at mode 0600.
• Quantized drift remains advisory. Only the near-lossless lane can satisfy the quality gate, and only performance-fit recipes can earn speed/fit benefits.

Evidence files

• performance-contract.json — immutable v1 thresholds and stop condition
• benchmark-config.json — exact real-run plan, drivers, artifacts, and hashes
• results.json — raw machine-readable per-request and aggregate evidence
• results.txt — human-readable benchmark summary
• baseline.json — distilled measurements for later comparison
• contract-evaluation.json — fail-closed v1 verdict
• commands.txt — reproducible conversion, benchmark, evaluation, and test commands
• BLOCKED.md — downstream stop-condition handoff
• known-unrelated-failure.md — clean-base reproduction of the tracker race

Verification

Targeted: 24 passed
Full suite: 751 passed, 13 skipped
Earlier cancellation retry matrix, DGR-001: 4/5 passed
Earlier cancellation retry matrix, clean d904c40: 4/5 passed
compileall: passed
git diff --check: passed
Evidence JSON parse/integrity checks: passed

An earlier intermittent tracker cancellation race reproduced at the same rate on the clean base and is retained in known-unrelated-failure.md; the final suite completed green. DGR-001 changes no tracker/proxy files.

The earlier Ralph claim that the full suite was blocked by Protobuf 6.33.6 was invalid: it used Hermes Agent's internal venv. Verification above used the project .venv, which has the DGR-002-compatible runtime. Real inference used .venv-rocm Python 3.12.

Limitations and dependent-story handoff

• This is a 0.5B CPU baseline, not evidence for a large model, Radeon GPU, distributed execution, network transport, or native shard worker.
• The installed llama.cpp build is CPU-only (GGML_HIP=OFF). No GPU comparison is claimed.
• Absolute timings are developer-machine measurements; locked ratios and raw artifacts are provided for reproducibility.
• DGR-014 may consume v1 only with the exact plan/evidence requirements enforced by performance_contract.py.
• DGR-004 and later native-runtime work must not treat DGR-001 completion as a promotion. V1 says stop; proceeding requires a human decision backed by a separately versioned GPU/large-model contract or a diagnosed quality fix.