fix: harden DGR-001 performance contract evidence
This commit is contained in:
Dobromir Popov
@@ -1,62 +1,153 @@
|
||||
# DGR-001 — Safetensors versus GGUF performance contract
|
||||
|
||||
Status: **blocked for real evidence; deterministic implementation complete.**
|
||||
No model benchmark is claimed. See `BLOCKED.md` and the explicitly `not-run`
|
||||
`results.json`.
|
||||
Status: **complete; immutable v1 verdict is `stop`.**
|
||||
|
||||
## What is implemented
|
||||
DGR-001 successfully produced a controlled local-real CPU baseline. Completion
|
||||
means the experiment and decision contract are durable and verified; it does
|
||||
**not** mean the native GGUF track is approved to continue. The locked quality
|
||||
gate failed, so dependent runtime work requires a human decision or a new,
|
||||
explicitly versioned experiment/contract rather than silently weakening v1.
|
||||
|
||||
- `recipe_benchmark.py` is a deterministic measurement core that runs the exact
|
||||
same plan for every recipe and reports TTFT, prefill/decode rates, p50/p95
|
||||
latency, aggregate throughput, RSS, VRAM, artifact bytes, request failures,
|
||||
and per-prompt output drift in JSON.
|
||||
- `recipe_drivers.py` supplies opt-in Transformers/safetensors and whole-model
|
||||
llama.cpp-server drivers. Real execution requires
|
||||
`MESHNET_ENABLE_REAL_INFERENCE_TESTS=1`, refuses model paths outside the
|
||||
declared mounted-drive root, requires a SHA-256 per artifact, records host
|
||||
facts, and requires the same declared source model and revision for every
|
||||
recipe.
|
||||
- `performance_contract.py` separates a near-lossless quality lane from the
|
||||
quantized performance/fit lane. Quantized drift is advisory; only the quality
|
||||
lane can establish parity. `performance-contract.json` locks v1 thresholds
|
||||
and the stop condition before any result exists.
|
||||
## Controlled workload
|
||||
|
||||
## Files changed
|
||||
- Model: `Qwen/Qwen2.5-0.5B-Instruct`
|
||||
- Exact source revision: `7ae557604adf67be50417f59c2c2f167def9a775`
|
||||
- Machine: `fedora`, Linux `7.0.14-101.fc43.x86_64`, 32 logical CPUs
|
||||
- Device: CPU for every recipe; VRAM is therefore correctly reported as zero
|
||||
- Runtime reference: Transformers `5.13.0`, PyTorch
|
||||
`2.10.0+rocm7.13.0a20260513`, BF16 safetensors
|
||||
- GGUF runtime: llama.cpp version 9991, commit
|
||||
`e920c523e3b8a0163fe498af5bf90df35ff51d25`
|
||||
- Workload: three fixed short/medium/long prompts, greedy sampling, 32 output
|
||||
tokens, three repeats, two warmups, concurrency 1 and 4, 16 CPU threads
|
||||
- Evidence class: `local-real`
|
||||
|
||||
- `packages/node/meshnet_node/recipe_benchmark.py`
|
||||
- `packages/node/meshnet_node/recipe_drivers.py`
|
||||
- `packages/node/meshnet_node/performance_contract.py`
|
||||
- `tests/test_recipe_benchmark.py`
|
||||
- This evidence directory.
|
||||
All artifacts are beneath `/run/media/popov/DATA/llm/`; no model artifact was
|
||||
created under `/home`.
|
||||
|
||||
## Commands and results
|
||||
## Recipes and exact artifacts
|
||||
|
||||
`commands.txt` contains exact commands. Final targeted result:
|
||||
| Recipe | Artifact | SHA-256 |
|
||||
|---|---|---|
|
||||
| Transformers BF16 reference | complete mounted Hugging Face snapshot | `e596e9d6205fdc9177569cccd7f8b471b058f66e3630c8e4326d5aad52bd18b6` |
|
||||
| llama.cpp BF16 quality lane | `Qwen2.5-0.5B-Instruct-7ae5576-BF16.gguf` | `e842fdc35d7f00fda95a54e1b51731ba1d196aea45065cc9f46925fdc1d6f862` |
|
||||
| llama.cpp Q4_K_M performance/fit lane | `Qwen2.5-0.5B-Instruct-7ae5576-Q4_K_M.gguf` | `a88e3f570e2efeaf06b50df9859db2c70d8646aa3a2c94a14e14d5797a2921a5` |
|
||||
|
||||
The snapshot digest covers every sorted relative path, resolved size, and file
|
||||
byte, so tokenizer/config drift is included. The BF16 GGUF was converted
|
||||
directly from the exact snapshot while preserving BF16 weights. Q4_K_M was
|
||||
quantized from an exact-revision F16 conversion with the pinned quantizer.
|
||||
Runtime validation recomputes every declared digest before model loading.
|
||||
|
||||
## Real results
|
||||
|
||||
All recipes completed every request with zero failures.
|
||||
|
||||
| Metric | Transformers BF16 | llama.cpp BF16 | llama.cpp Q4_K_M |
|
||||
|---|---:|---:|---:|
|
||||
| Decode tok/s, c=1 | 46.1 | 88.0 | 170.1 |
|
||||
| Aggregate decode tok/s, c=4 | 47.1 | 211.4 | 206.4 |
|
||||
| TTFT p50, c=1 | 37.5 ms | 43.9 ms | 23.8 ms |
|
||||
| Peak resident memory, c=1 | 1.94 GB | 1.11 GB | 0.54 GB |
|
||||
| Artifact size | 1.00 GB | 0.99 GB | 0.40 GB |
|
||||
| Failures | 0 | 0 | 0 |
|
||||
|
||||
Against the reference, the eligible Q4_K_M lane measured:
|
||||
|
||||
- single-request decode speedup: **3.69×**;
|
||||
- concurrency-4 aggregate throughput speedup: **4.38×**;
|
||||
- resident-memory ratio: **0.279×**;
|
||||
- artifact-size ratio: **0.398×**.
|
||||
|
||||
The near-lossless BF16 quality lane compared all three prompts but measured:
|
||||
|
||||
- exact match: **0.3333** (v1 requires at least `0.90`);
|
||||
- mean text similarity: **0.9471** (v1 requires at least `0.97`).
|
||||
|
||||
Tokenization and stopping were controlled: every runtime saw the same prompt
|
||||
token counts and reported 31 post-TTFT decode tokens. The mismatch is genuine
|
||||
greedy runtime divergence on two prompts, not missing coverage or a text-length
|
||||
artifact. Therefore `contract-evaluation.json` records:
|
||||
|
||||
```text
|
||||
pytest -q tests/test_recipe_benchmark.py -> 15 passed
|
||||
python -m compileall -q packages tests -> exit 0
|
||||
git diff --check -> exit 0
|
||||
verdict: stop
|
||||
quality_lane_pass: false
|
||||
speed_benefit: true
|
||||
fit_benefit: true
|
||||
stop_condition_met: true
|
||||
```
|
||||
|
||||
The full suite was attempted and is blocked during collection by the unrelated,
|
||||
pre-existing DGR-002 runtime dependency mismatch:
|
||||
Thresholds were not changed after observing these results.
|
||||
|
||||
## Implementation
|
||||
|
||||
- `recipe_benchmark.py` provides the runtime-neutral measurement core, true
|
||||
concurrency, continuous in-flight peak-memory sampling, percentile/throughput
|
||||
aggregation, failures, and output drift.
|
||||
- `recipe_drivers.py` provides opt-in Transformers and llama-server drivers,
|
||||
mounted-drive confinement, exact artifact/runtime verification, equal
|
||||
device/thread budgets, greedy-only validation, measured host provenance, and
|
||||
a CPU-only v1 guard until process VRAM can be measured honestly.
|
||||
- Peak RSS is runtime-scoped: Transformers reports growth above its pre-runtime
|
||||
Python baseline, while llama.cpp reports its isolated server process tree.
|
||||
Both are sampled continuously during in-flight requests.
|
||||
- TTFT uses each runtime's prompt/first-token compute boundary; end-to-end HTTP,
|
||||
scheduling, and queue overhead remains in latency and `queue_wait_ms`.
|
||||
- The exact canonical plan SHA-256 locks prompts, model/revision, sampling,
|
||||
output length, repeats, warmups, and concurrency. The evaluator also requires
|
||||
equal prompt/decode token counts across recipes.
|
||||
- llama.cpp's `predicted_n` includes the first token while `predicted_ms` begins
|
||||
after it; the driver subtracts that token so decode throughput matches the
|
||||
Transformers inter-token convention.
|
||||
- `performance_contract.py` rejects wrong plans, synthetic evidence, missing
|
||||
recipes/concurrency, mixed model revisions, incomplete quality coverage,
|
||||
failed references, and missing artifact/host provenance.
|
||||
- Quantized drift remains advisory. Only the near-lossless lane can satisfy the
|
||||
quality gate, and only performance-fit recipes can earn speed/fit benefits.
|
||||
|
||||
## Evidence files
|
||||
|
||||
- `performance-contract.json` — immutable v1 thresholds and stop condition
|
||||
- `benchmark-config.json` — exact real-run plan, drivers, artifacts, and hashes
|
||||
- `results.json` — raw machine-readable per-request and aggregate evidence
|
||||
- `results.txt` — human-readable benchmark summary
|
||||
- `baseline.json` — distilled measurements for later comparison
|
||||
- `contract-evaluation.json` — fail-closed v1 verdict
|
||||
- `commands.txt` — reproducible conversion, benchmark, evaluation, and test commands
|
||||
- `BLOCKED.md` — downstream stop-condition handoff
|
||||
- `known-unrelated-failure.md` — clean-base reproduction of the tracker race
|
||||
|
||||
## Verification
|
||||
|
||||
```text
|
||||
google.protobuf.runtime_version.VersionError:
|
||||
gencode 7.35.0 runtime 6.33.6
|
||||
Targeted: 22 passed
|
||||
Full suite: 749 passed, 13 skipped
|
||||
Earlier cancellation retry matrix, DGR-001: 4/5 passed
|
||||
Earlier cancellation retry matrix, clean d904c40: 4/5 passed
|
||||
compileall: passed
|
||||
git diff --check: passed
|
||||
Evidence JSON parse/integrity checks: passed
|
||||
```
|
||||
|
||||
This was reproduced from a clean `git archive HEAD` extracted to
|
||||
`/tmp/dgr-001-clean`, with the same command and same failure before any
|
||||
uncommitted DGR-001 changes were present. No real benchmark command was run
|
||||
because the prerequisites in `BLOCKED.md` are absent.
|
||||
The full-suite exception is documented in `known-unrelated-failure.md` and
|
||||
satisfies the issue's explicit clean-tree reproduction clause. DGR-001 changes
|
||||
no tracker/proxy files.
|
||||
|
||||
## Compatibility and handoff
|
||||
The earlier Ralph claim that the full suite was blocked by Protobuf 6.33.6 was
|
||||
invalid: it used Hermes Agent's internal venv. Verification above used the
|
||||
project `.venv`, which has the DGR-002-compatible runtime. Real inference used
|
||||
`.venv-rocm` Python 3.12.
|
||||
|
||||
This is additive: it does not alter the current Transformers route, Tracker,
|
||||
relay, or native protocol. DGR-014 must load `performance-contract.json`, run
|
||||
the same controlled plan at concurrency 1 and 4, and make only its
|
||||
promote/optimize/stop recommendation from a `local-real` or
|
||||
`multi-machine-real` report. DGR-004 remains blocked on this story's real
|
||||
baseline decision.
|
||||
## Limitations and dependent-story handoff
|
||||
|
||||
- This is a **0.5B CPU baseline**, not evidence for a large model, Radeon GPU,
|
||||
distributed execution, network transport, or native shard worker.
|
||||
- The installed llama.cpp build is CPU-only (`GGML_HIP=OFF`). No GPU comparison
|
||||
is claimed.
|
||||
- Absolute timings are developer-machine measurements; locked ratios and raw
|
||||
artifacts are provided for reproducibility.
|
||||
- DGR-014 may consume v1 only with the exact plan/evidence requirements enforced
|
||||
by `performance_contract.py`.
|
||||
- DGR-004 and later native-runtime work must not treat DGR-001 completion as a
|
||||
promotion. V1 says `stop`; proceeding requires a human decision backed by a
|
||||
separately versioned GPU/large-model contract or a diagnosed quality fix.
|
||||
|
||||
Reference in New Issue
Block a user