63 lines
2.5 KiB
Markdown
63 lines
2.5 KiB
Markdown
# DGR-001 — Safetensors versus GGUF performance contract
|
|
|
|
Status: **blocked for real evidence; deterministic implementation complete.**
|
|
No model benchmark is claimed. See `BLOCKED.md` and the explicitly `not-run`
|
|
`results.json`.
|
|
|
|
## What is implemented
|
|
|
|
- `recipe_benchmark.py` is a deterministic measurement core that runs the exact
|
|
same plan for every recipe and reports TTFT, prefill/decode rates, p50/p95
|
|
latency, aggregate throughput, RSS, VRAM, artifact bytes, request failures,
|
|
and per-prompt output drift in JSON.
|
|
- `recipe_drivers.py` supplies opt-in Transformers/safetensors and whole-model
|
|
llama.cpp-server drivers. Real execution requires
|
|
`MESHNET_ENABLE_REAL_INFERENCE_TESTS=1`, refuses model paths outside the
|
|
declared mounted-drive root, requires a SHA-256 per artifact, records host
|
|
facts, and requires the same declared source model and revision for every
|
|
recipe.
|
|
- `performance_contract.py` separates a near-lossless quality lane from the
|
|
quantized performance/fit lane. Quantized drift is advisory; only the quality
|
|
lane can establish parity. `performance-contract.json` locks v1 thresholds
|
|
and the stop condition before any result exists.
|
|
|
|
## Files changed
|
|
|
|
- `packages/node/meshnet_node/recipe_benchmark.py`
|
|
- `packages/node/meshnet_node/recipe_drivers.py`
|
|
- `packages/node/meshnet_node/performance_contract.py`
|
|
- `tests/test_recipe_benchmark.py`
|
|
- This evidence directory.
|
|
|
|
## Commands and results
|
|
|
|
`commands.txt` contains exact commands. Final targeted result:
|
|
|
|
```text
|
|
pytest -q tests/test_recipe_benchmark.py -> 15 passed
|
|
python -m compileall -q packages tests -> exit 0
|
|
git diff --check -> exit 0
|
|
```
|
|
|
|
The full suite was attempted and is blocked during collection by the unrelated,
|
|
pre-existing DGR-002 runtime dependency mismatch:
|
|
|
|
```text
|
|
google.protobuf.runtime_version.VersionError:
|
|
gencode 7.35.0 runtime 6.33.6
|
|
```
|
|
|
|
This was reproduced from a clean `git archive HEAD` extracted to
|
|
`/tmp/dgr-001-clean`, with the same command and same failure before any
|
|
uncommitted DGR-001 changes were present. No real benchmark command was run
|
|
because the prerequisites in `BLOCKED.md` are absent.
|
|
|
|
## Compatibility and handoff
|
|
|
|
This is additive: it does not alter the current Transformers route, Tracker,
|
|
relay, or native protocol. DGR-014 must load `performance-contract.json`, run
|
|
the same controlled plan at concurrency 1 and 4, and make only its
|
|
promote/optimize/stop recommendation from a `local-real` or
|
|
`multi-machine-real` report. DGR-004 remains blocked on this story's real
|
|
baseline decision.
|