31 lines
1.5 KiB
Markdown
31 lines
1.5 KiB
Markdown
# DGR-001 real-benchmark blocker
|
|
|
|
Status: blocked only for the required real-model measurement. The deterministic
|
|
harness, report schema, and immutable contract are implemented and tested; this
|
|
file deliberately does not turn an unrun benchmark into a passing result.
|
|
|
|
## Verified environment state (2026-07-13)
|
|
|
|
- Mounted GGUF artifacts exist under `/run/media/popov/DATA/llm/`.
|
|
- `llama-server` is not on `PATH`.
|
|
- The available Python test environment has neither `torch` nor `transformers`.
|
|
- No matching local safetensors snapshot was found for an installed GGUF recipe.
|
|
|
|
Therefore this session cannot run the controlled same-model, same-revision,
|
|
same-machine comparison without downloading/installing new runtime/model assets.
|
|
That is intentionally not inferred from the story request.
|
|
|
|
## Continuation
|
|
|
|
1. Put a matching safetensors snapshot and near-lossless plus quantized GGUF
|
|
artifacts below one mounted-drive root, never `/home`.
|
|
2. Install or build the pinned `llama-server`, and use `.venv-rocm` when testing
|
|
the Radeon backend.
|
|
3. Compute each artifact SHA-256 and create a config declaring the same
|
|
`source_model_id` and `source_model_revision` for every recipe.
|
|
4. Run the command in `commands.txt` with
|
|
`MESHNET_ENABLE_REAL_INFERENCE_TESTS=1`; save its JSON report and summary in
|
|
this directory, then evaluate it against `performance-contract.json`.
|
|
5. Only after those results and all quality gates pass may DGR-001 be marked
|
|
done and DGR-004 consume the baseline.
|