fix: harden DGR-001 performance contract evidence
This commit is contained in:
@@ -1,30 +1,43 @@
|
||||
# DGR-001 real-benchmark blocker
|
||||
# DGR-001 downstream stop-condition handoff
|
||||
|
||||
Status: blocked only for the required real-model measurement. The deterministic
|
||||
harness, report schema, and immutable contract are implemented and tested; this
|
||||
file deliberately does not turn an unrun benchmark into a passing result.
|
||||
Status: **DGR-001 is complete; native-track promotion is blocked by the immutable v1 verdict.**
|
||||
|
||||
## Verified environment state (2026-07-13)
|
||||
This is no longer an execution-prerequisite blocker. The required real benchmark
|
||||
ran successfully, every recipe completed at concurrency 1 and 4, artifacts were
|
||||
verified, and deterministic/full test gates passed.
|
||||
|
||||
- Mounted GGUF artifacts exist under `/run/media/popov/DATA/llm/`.
|
||||
- `llama-server` is not on `PATH`.
|
||||
- The available Python test environment has neither `torch` nor `transformers`.
|
||||
- No matching local safetensors snapshot was found for an installed GGUF recipe.
|
||||
## Locked result
|
||||
|
||||
Therefore this session cannot run the controlled same-model, same-revision,
|
||||
same-machine comparison without downloading/installing new runtime/model assets.
|
||||
That is intentionally not inferred from the story request.
|
||||
`contract-evaluation.json` records:
|
||||
|
||||
## Continuation
|
||||
```text
|
||||
verdict: stop
|
||||
quality_lane_pass: false
|
||||
speed_benefit: true
|
||||
fit_benefit: true
|
||||
stop_condition_met: true
|
||||
```
|
||||
|
||||
1. Put a matching safetensors snapshot and near-lossless plus quantized GGUF
|
||||
artifacts below one mounted-drive root, never `/home`.
|
||||
2. Install or build the pinned `llama-server`, and use `.venv-rocm` when testing
|
||||
the Radeon backend.
|
||||
3. Compute each artifact SHA-256 and create a config declaring the same
|
||||
`source_model_id` and `source_model_revision` for every recipe.
|
||||
4. Run the command in `commands.txt` with
|
||||
`MESHNET_ENABLE_REAL_INFERENCE_TESTS=1`; save its JSON report and summary in
|
||||
this directory, then evaluate it against `performance-contract.json`.
|
||||
5. Only after those results and all quality gates pass may DGR-001 be marked
|
||||
done and DGR-004 consume the baseline.
|
||||
The exact-revision BF16 GGUF quality lane compared every prompt but achieved
|
||||
`0.3333` exact match and `0.9471` mean similarity against the Transformers BF16
|
||||
reference. V1 requires `0.90` and `0.97`. Quantized Q4_K_M had substantial speed
|
||||
and fit benefits, but the contract explicitly forbids speed from redeeming a
|
||||
failed near-lossless quality lane.
|
||||
|
||||
## Scope of this stop
|
||||
|
||||
The measured baseline is Qwen2.5-0.5B on CPU using a CPU-only llama.cpp build.
|
||||
It is not a Radeon, large-model, distributed, or native-shard result. Therefore:
|
||||
|
||||
1. Do not silently mark v1 promoted or weaken its thresholds after observing the
|
||||
data.
|
||||
2. Do not let DGR-004 or later runtime stories treat DGR-001 completion as a
|
||||
positive promotion signal.
|
||||
3. A human may choose one of these explicit paths:
|
||||
- stop the native GGUF track as v1 directs;
|
||||
- diagnose and fix the BF16 runtime divergence, then rerun the exact v1 plan;
|
||||
- authorize a separately versioned GPU/large-model contract whose scope and
|
||||
workload are locked before its measurements.
|
||||
|
||||
All raw evidence, configuration, artifacts, hashes, and reproduction commands
|
||||
are in this directory and `README.md`.
|
||||
|
||||
Reference in New Issue
Block a user