feat: add deterministic CPU/GPU benchmark runner slice

This commit is contained in:
Dobromir Popov
2026-07-14 21:39:13 +03:00
parent 5b33bf8b99
commit e6f6782995
4 changed files with 517 additions and 5 deletions

View File

@@ -19,6 +19,21 @@
- concurrency levels `1` and `4`
- the required metrics list
- an explicit stop condition for “no meaningful speed or fit benefit”
- Adds a deterministic stub benchmark report so the contract now has an executable report shape end to end.
## Recent benchmark runner slice
The runner currently uses a deterministic stub backend to exercise the comparison matrix without downloading a model. It emits:
- `.scratch/distributed-gguf-runtime/evidence/DGR-001/performance-contract.json`
- `.scratch/distributed-gguf-runtime/evidence/DGR-001/stub-benchmark-report.json`
The report includes per-device comparisons for:
- `transformers-safetensors-cpu` vs `llama-cpp-gguf-cpu`
- `transformers-safetensors-gpu` vs `llama-cpp-gguf-gpu`
and records the memory metric (`rss_bytes` on CPU, `vram_bytes` on GPU), decode speedup, artifact ratio, and output drift.
## Exact commands and real results