neuron-tai/.scratch/distributed-gguf-runtime/evidence/DGR-001/README.md

# DGR-001 — Safetensors versus GGUF performance contract

Status: **complete; immutable v1 verdict is `stop`.**

DGR-001 successfully produced a controlled local-real CPU baseline. Completion
means the experiment and decision contract are durable and verified; it does
**not** mean the native GGUF track is approved to continue. The locked quality
gate failed, so dependent runtime work requires a human decision or a new,
explicitly versioned experiment/contract rather than silently weakening v1.

## Controlled workload

- Model: `Qwen/Qwen2.5-0.5B-Instruct`
- Exact source revision: `7ae557604adf67be50417f59c2c2f167def9a775`
- Machine: `fedora`, Linux `7.0.14-101.fc43.x86_64`, 32 logical CPUs
- Device: CPU for every recipe; VRAM is therefore correctly reported as zero
- Runtime reference: Transformers `5.13.0`, PyTorch
  `2.10.0+rocm7.13.0a20260513`, BF16 safetensors
- GGUF runtime: llama.cpp version 9991, commit
  `e920c523e3b8a0163fe498af5bf90df35ff51d25`
- Workload: three fixed short/medium/long prompts, greedy sampling, 32 output
  tokens, three repeats, two warmups, concurrency 1 and 4, 16 CPU threads
- Evidence class: `local-real`

All artifacts are beneath `/run/media/popov/DATA/llm/`; no model artifact was
created under `/home`.

## Recipes and exact artifacts

| Recipe | Artifact | SHA-256 |
|---|---|---|
| Transformers BF16 reference | complete mounted Hugging Face snapshot | `e596e9d6205fdc9177569cccd7f8b471b058f66e3630c8e4326d5aad52bd18b6` |
| llama.cpp BF16 quality lane | `Qwen2.5-0.5B-Instruct-7ae5576-BF16.gguf` | `e842fdc35d7f00fda95a54e1b51731ba1d196aea45065cc9f46925fdc1d6f862` |
| llama.cpp Q4_K_M performance/fit lane | `Qwen2.5-0.5B-Instruct-7ae5576-Q4_K_M.gguf` | `a88e3f570e2efeaf06b50df9859db2c70d8646aa3a2c94a14e14d5797a2921a5` |

The snapshot digest covers every sorted relative path, resolved size, and file
byte, so tokenizer/config drift is included. The BF16 GGUF was converted
directly from the exact snapshot while preserving BF16 weights. Q4_K_M was
quantized from an exact-revision F16 conversion with the pinned quantizer.
Runtime validation recomputes every declared digest before model loading.

## Real results

All recipes completed every request with zero failures.

| Metric | Transformers BF16 | llama.cpp BF16 | llama.cpp Q4_K_M |
|---|---:|---:|---:|
| Decode tok/s, c=1 | 40.8 | 98.5 | 207.7 |
| Aggregate decode tok/s, c=4 | 46.5 | 222.8 | 195.7 |
| TTFT p50, c=1 | 40.0 ms | 15.1 ms | 21.6 ms |
| Peak resident memory, c=1 | 1.94 GB | 1.11 GB | 0.54 GB |
| Artifact size | 1.00 GB | 0.99 GB | 0.40 GB |
| Failures | 0 | 0 | 0 |

Against the reference, the eligible Q4_K_M lane measured:

- single-request decode speedup: **5.10×**;
- concurrency-4 aggregate throughput speedup: **4.20×**;
- resident-memory ratio: **0.279×**;
- artifact-size ratio: **0.398×**.

The near-lossless BF16 quality lane compared all three prompts but measured:

- exact match: **0.3333** (v1 requires at least `0.90`);
- mean text similarity: **0.9471** (v1 requires at least `0.97`).

Tokenization and stopping were controlled: every runtime saw the same prompt
token counts and reported 31 post-TTFT decode tokens. The v1 mismatch is a
real greedy-output divergence on two prompts, not missing coverage or a
text-length artifact. Its root cause remains undetermined; no post-contract
logit-tie claim is acceptance evidence. Therefore `contract-evaluation.json`
records:

```text
verdict: stop
quality_lane_pass: false
speed_benefit: true
fit_benefit: true
stop_condition_met: true
```

Thresholds were not changed after observing these results.

## Post-contract parity and ROCm diagnostics

`summarize-quality-parity.py` verifies and separates two signed sources. The CPU
v1 row uses CPU kernels and a Transformers BF16 oracle; it remains at `0.3333`
exact match with an unexplained divergence. The ROCm row uses a different plan,
GPU kernels, and a Transformers float32 oracle. In that narrower diagnostic,
the same BF16 GGUF artifact matches all three 32-token sequences exactly (`1.0`
exact match and `1.0` similarity). No conversion corruption was observed in
that three-sequence ROCm sample; this does not prove global conversion
correctness or explain the CPU result.

A separate HIP build at commit `e920c523` was compiled for `gfx1151` and
measured `ROCm0: Radeon 8060S Graphics`; its `llama-server` SHA-256 is
`b6bb4da687dbde86e243ba006cef05919b7b97255cd7e2371e1d451220aca139`.
A signed `gpu-diagnostic` profile measured zero failures:

| GPU metric | Transformers BF16 ROCm | llama.cpp Q4 ROCm | Q4 ratio |
|---|---:|---:|---:|
| Decode tok/s, c=1 | 81.12 | 251.25 | **3.10×** |
| Aggregate decode tok/s, c=4 | 91.24 | 511.33 | **5.60×** |
| TTFT p50, c=1 | 13.77 ms | 11.80 ms | **0.857×** |

The GPU report is signed under the distinct
`run_configured_gpu_diagnostic/v1` producer. The v1 evaluator rejects that
producer even when its signature is valid. llama-server process VRAM remains
unmeasured, so this diagnostic cannot replace or satisfy the immutable v1
contract. Its signed backend detail records the measured `ROCm0: Radeon 8060S
Graphics` device and `25/25` offloaded layers.

## Implementation

- `recipe_benchmark.py` provides the runtime-neutral measurement core, true
  concurrency, continuous in-flight peak-memory sampling, percentile/throughput
  aggregation, failures, and output drift.
- `recipe_drivers.py` provides opt-in Transformers and llama-server drivers,
  mounted-drive confinement, exact artifact/runtime verification, equal
  device/thread budgets, greedy-only validation, measured host provenance, a
  CPU-only v1 guard until process VRAM can be measured honestly, and a distinct
  signed GPU diagnostic profile that the v1 evaluator cannot accept.
- Peak RSS is runtime-scoped: Transformers reports growth above its pre-runtime
  Python baseline, while llama.cpp reports its isolated server process tree.
  Both are sampled continuously during in-flight requests.
- TTFT uses each runtime's prompt/first-token compute boundary; end-to-end HTTP,
  scheduling, and queue overhead remains in latency and `queue_wait_ms`.
- The exact canonical plan SHA-256 locks prompts, model/revision, sampling,
  output length, repeats, warmups, and concurrency. The evaluator also requires
  equal prompt/decode token counts across recipes.
- llama.cpp's `predicted_n` includes the first token while `predicted_ms` begins
  after it; the driver subtracts that token so decode throughput matches the
  Transformers inter-token convention.
- `performance_contract.py` rejects wrong plans, unsigned or incorrectly signed
  real evidence, wrong config/artifact/runtime/backend/host bindings, missing
  recipes/concurrency, mixed model revisions, incomplete quality coverage, and
  failed references.
- Every non-synthetic report is Ed25519-signed over the complete canonical JSON,
  including raw outcomes and metrics. The contract pins the public key and exact
  config SHA-256; the private key remains outside Git at mode `0600`.
- The signer fingerprint is independently anchored outside this evidence
  directory in `../../trusted-evidence-signers.json` and checked by tests.
- Quantized drift remains advisory. Only the near-lossless lane can satisfy the
  quality gate, and only performance-fit recipes can earn speed/fit benefits.

## Evidence files

- `performance-contract.json` — immutable v1 thresholds and stop condition
- `benchmark-config.json` — exact real-run plan, drivers, artifacts, and hashes
- `results.json` — raw machine-readable per-request and aggregate evidence
- `results.txt` — human-readable benchmark summary
- `baseline.json` — distilled measurements for later comparison
- `contract-evaluation.json` — fail-closed v1 verdict
- `quality-parity-diagnosis.json` / `.md` — run/device-scoped signed-evidence summary
- `summarize-quality-parity.py` — verifies both evidence chains and regenerates it
- `gpu-diagnostic-config.json` — exact ROCm diagnostic artifacts and runtimes
- `gpu-diagnostic-results.json` / `.txt` — signed GPU outcomes and summary
- `commands.txt` — reproducible conversion, benchmark, evaluation, and test commands
- `BLOCKED.md` — downstream stop-condition handoff
- `known-unrelated-failure.md` — clean-base reproduction of the tracker race
- `../../trusted-evidence-signers.json` — repository-reviewed signer fingerprint

## Verification

```text
Targeted: 28 passed (5/5 consecutive focused runs)
Latest full suite: 755 passed, 13 skipped
Earlier full suite: 751 passed, 13 skipped
Current cancellation retry matrix, DGR-001: 4/5 passed
Earlier cancellation retry matrix, clean d904c40: 4/5 passed
compileall: passed
git diff --check: passed
Evidence JSON parse/integrity checks: passed
```

The intermittent tracker cancellation race reproduced at the same rate on the
clean base and is retained in `known-unrelated-failure.md`; the final full suite
completed green. DGR-001 changes no tracker/proxy files.

The earlier Ralph claim that the full suite was blocked by Protobuf 6.33.6 was
invalid: it used Hermes Agent's internal venv. Verification above used the
project `.venv`, which has the DGR-002-compatible runtime. Real inference used
`.venv-rocm` Python 3.12.

## Limitations and dependent-story handoff

- The immutable contract result is a **0.5B CPU baseline**. The separate Radeon
  diagnostic is real local GPU evidence, but neither result covers a large
  model, distributed execution, network transport, or a native shard worker.
- A separate `GGML_HIP=ON` llama.cpp build exists and produced GPU timings, but
  llama-server process VRAM is not measurable by the current driver; GPU
  memory/fit claims therefore remain ineligible for v1.
- Absolute timings are developer-machine measurements; locked ratios and raw
  artifacts are provided for reproducibility.
- DGR-014 may consume v1 only with the exact plan/evidence requirements enforced
  by `performance_contract.py`.
- DGR-004 and later native-runtime work must not treat DGR-001 completion as a
  promotion. V1 says `stop`; proceeding requires a human decision backed by a
  separately versioned GPU/large-model contract or a diagnosed quality fix.