fix: harden DGR-001 performance contract evidence

This commit is contained in:
Dobromir Popov
2026-07-13 19:10:24 +03:00
parent e24db7854f
commit 9e67b829e3
16 changed files with 3674 additions and 151 deletions

View File

@@ -1,30 +1,43 @@
# DGR-001 real-benchmark blocker
# DGR-001 downstream stop-condition handoff
Status: blocked only for the required real-model measurement. The deterministic
harness, report schema, and immutable contract are implemented and tested; this
file deliberately does not turn an unrun benchmark into a passing result.
Status: **DGR-001 is complete; native-track promotion is blocked by the immutable v1 verdict.**
## Verified environment state (2026-07-13)
This is no longer an execution-prerequisite blocker. The required real benchmark
ran successfully, every recipe completed at concurrency 1 and 4, artifacts were
verified, and deterministic/full test gates passed.
- Mounted GGUF artifacts exist under `/run/media/popov/DATA/llm/`.
- `llama-server` is not on `PATH`.
- The available Python test environment has neither `torch` nor `transformers`.
- No matching local safetensors snapshot was found for an installed GGUF recipe.
## Locked result
Therefore this session cannot run the controlled same-model, same-revision,
same-machine comparison without downloading/installing new runtime/model assets.
That is intentionally not inferred from the story request.
`contract-evaluation.json` records:
## Continuation
```text
verdict: stop
quality_lane_pass: false
speed_benefit: true
fit_benefit: true
stop_condition_met: true
```
1. Put a matching safetensors snapshot and near-lossless plus quantized GGUF
artifacts below one mounted-drive root, never `/home`.
2. Install or build the pinned `llama-server`, and use `.venv-rocm` when testing
the Radeon backend.
3. Compute each artifact SHA-256 and create a config declaring the same
`source_model_id` and `source_model_revision` for every recipe.
4. Run the command in `commands.txt` with
`MESHNET_ENABLE_REAL_INFERENCE_TESTS=1`; save its JSON report and summary in
this directory, then evaluate it against `performance-contract.json`.
5. Only after those results and all quality gates pass may DGR-001 be marked
done and DGR-004 consume the baseline.
The exact-revision BF16 GGUF quality lane compared every prompt but achieved
`0.3333` exact match and `0.9471` mean similarity against the Transformers BF16
reference. V1 requires `0.90` and `0.97`. Quantized Q4_K_M had substantial speed
and fit benefits, but the contract explicitly forbids speed from redeeming a
failed near-lossless quality lane.
## Scope of this stop
The measured baseline is Qwen2.5-0.5B on CPU using a CPU-only llama.cpp build.
It is not a Radeon, large-model, distributed, or native-shard result. Therefore:
1. Do not silently mark v1 promoted or weaken its thresholds after observing the
data.
2. Do not let DGR-004 or later runtime stories treat DGR-001 completion as a
positive promotion signal.
3. A human may choose one of these explicit paths:
- stop the native GGUF track as v1 directs;
- diagnose and fix the BF16 runtime divergence, then rerun the exact v1 plan;
- authorize a separately versioned GPU/large-model contract whose scope and
workload are locked before its measurements.
All raw evidence, configuration, artifacts, hashes, and reproduction commands
are in this directory and `README.md`.