feat: DGR-001 - Lock the safetensors-versus-GGUF performance contract
This commit is contained in:
62
.scratch/distributed-gguf-runtime/evidence/DGR-001/README.md
Normal file
62
.scratch/distributed-gguf-runtime/evidence/DGR-001/README.md
Normal file
@@ -0,0 +1,62 @@
|
||||
# DGR-001 — Safetensors versus GGUF performance contract
|
||||
|
||||
Status: **blocked for real evidence; deterministic implementation complete.**
|
||||
No model benchmark is claimed. See `BLOCKED.md` and the explicitly `not-run`
|
||||
`results.json`.
|
||||
|
||||
## What is implemented
|
||||
|
||||
- `recipe_benchmark.py` is a deterministic measurement core that runs the exact
|
||||
same plan for every recipe and reports TTFT, prefill/decode rates, p50/p95
|
||||
latency, aggregate throughput, RSS, VRAM, artifact bytes, request failures,
|
||||
and per-prompt output drift in JSON.
|
||||
- `recipe_drivers.py` supplies opt-in Transformers/safetensors and whole-model
|
||||
llama.cpp-server drivers. Real execution requires
|
||||
`MESHNET_ENABLE_REAL_INFERENCE_TESTS=1`, refuses model paths outside the
|
||||
declared mounted-drive root, requires a SHA-256 per artifact, records host
|
||||
facts, and requires the same declared source model and revision for every
|
||||
recipe.
|
||||
- `performance_contract.py` separates a near-lossless quality lane from the
|
||||
quantized performance/fit lane. Quantized drift is advisory; only the quality
|
||||
lane can establish parity. `performance-contract.json` locks v1 thresholds
|
||||
and the stop condition before any result exists.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `packages/node/meshnet_node/recipe_benchmark.py`
|
||||
- `packages/node/meshnet_node/recipe_drivers.py`
|
||||
- `packages/node/meshnet_node/performance_contract.py`
|
||||
- `tests/test_recipe_benchmark.py`
|
||||
- This evidence directory.
|
||||
|
||||
## Commands and results
|
||||
|
||||
`commands.txt` contains exact commands. Final targeted result:
|
||||
|
||||
```text
|
||||
pytest -q tests/test_recipe_benchmark.py -> 15 passed
|
||||
python -m compileall -q packages tests -> exit 0
|
||||
git diff --check -> exit 0
|
||||
```
|
||||
|
||||
The full suite was attempted and is blocked during collection by the unrelated,
|
||||
pre-existing DGR-002 runtime dependency mismatch:
|
||||
|
||||
```text
|
||||
google.protobuf.runtime_version.VersionError:
|
||||
gencode 7.35.0 runtime 6.33.6
|
||||
```
|
||||
|
||||
This was reproduced from a clean `git archive HEAD` extracted to
|
||||
`/tmp/dgr-001-clean`, with the same command and same failure before any
|
||||
uncommitted DGR-001 changes were present. No real benchmark command was run
|
||||
because the prerequisites in `BLOCKED.md` are absent.
|
||||
|
||||
## Compatibility and handoff
|
||||
|
||||
This is additive: it does not alter the current Transformers route, Tracker,
|
||||
relay, or native protocol. DGR-014 must load `performance-contract.json`, run
|
||||
the same controlled plan at concurrency 1 and 4, and make only its
|
||||
promote/optimize/stop recommendation from a `local-real` or
|
||||
`multi-machine-real` report. DGR-004 remains blocked on this story's real
|
||||
baseline decision.
|
||||
Reference in New Issue
Block a user