feat: add deterministic CPU/GPU benchmark runner slice
This commit is contained in:
@@ -19,6 +19,21 @@
|
||||
- concurrency levels `1` and `4`
|
||||
- the required metrics list
|
||||
- an explicit stop condition for “no meaningful speed or fit benefit”
|
||||
- Adds a deterministic stub benchmark report so the contract now has an executable report shape end to end.
|
||||
|
||||
## Recent benchmark runner slice
|
||||
|
||||
The runner currently uses a deterministic stub backend to exercise the comparison matrix without downloading a model. It emits:
|
||||
|
||||
- `.scratch/distributed-gguf-runtime/evidence/DGR-001/performance-contract.json`
|
||||
- `.scratch/distributed-gguf-runtime/evidence/DGR-001/stub-benchmark-report.json`
|
||||
|
||||
The report includes per-device comparisons for:
|
||||
|
||||
- `transformers-safetensors-cpu` vs `llama-cpp-gguf-cpu`
|
||||
- `transformers-safetensors-gpu` vs `llama-cpp-gguf-gpu`
|
||||
|
||||
and records the memory metric (`rss_bytes` on CPU, `vram_bytes` on GPU), decode speedup, artifact ratio, and output drift.
|
||||
|
||||
## Exact commands and real results
|
||||
|
||||
|
||||
Reference in New Issue
Block a user