fix: cryptographically bind DGR-001 evidence
This commit is contained in:
Dobromir Popov
@@ -45,17 +45,17 @@ All recipes completed every request with zero failures.
|
||||
|
||||
| Metric | Transformers BF16 | llama.cpp BF16 | llama.cpp Q4_K_M |
|
||||
|---|---:|---:|---:|
|
||||
| Decode tok/s, c=1 | 46.1 | 88.0 | 170.1 |
|
||||
| Aggregate decode tok/s, c=4 | 47.1 | 211.4 | 206.4 |
|
||||
| TTFT p50, c=1 | 37.5 ms | 43.9 ms | 23.8 ms |
|
||||
| Decode tok/s, c=1 | 40.8 | 98.5 | 207.7 |
|
||||
| Aggregate decode tok/s, c=4 | 46.5 | 222.8 | 195.7 |
|
||||
| TTFT p50, c=1 | 40.0 ms | 15.1 ms | 21.6 ms |
|
||||
| Peak resident memory, c=1 | 1.94 GB | 1.11 GB | 0.54 GB |
|
||||
| Artifact size | 1.00 GB | 0.99 GB | 0.40 GB |
|
||||
| Failures | 0 | 0 | 0 |
|
||||
|
||||
Against the reference, the eligible Q4_K_M lane measured:
|
||||
|
||||
- single-request decode speedup: **3.69×**;
|
||||
- concurrency-4 aggregate throughput speedup: **4.38×**;
|
||||
- single-request decode speedup: **5.10×**;
|
||||
- concurrency-4 aggregate throughput speedup: **4.20×**;
|
||||
- resident-memory ratio: **0.279×**;
|
||||
- artifact-size ratio: **0.398×**.
|
||||
|
||||
@@ -99,9 +99,13 @@ Thresholds were not changed after observing these results.
|
||||
- llama.cpp's `predicted_n` includes the first token while `predicted_ms` begins
|
||||
after it; the driver subtracts that token so decode throughput matches the
|
||||
Transformers inter-token convention.
|
||||
- `performance_contract.py` rejects wrong plans, synthetic evidence, missing
|
||||
recipes/concurrency, mixed model revisions, incomplete quality coverage,
|
||||
failed references, and missing artifact/host provenance.
|
||||
- `performance_contract.py` rejects wrong plans, unsigned or incorrectly signed
|
||||
real evidence, wrong config/artifact/runtime/backend/host bindings, missing
|
||||
recipes/concurrency, mixed model revisions, incomplete quality coverage, and
|
||||
failed references.
|
||||
- Every non-synthetic report is Ed25519-signed over the complete canonical JSON,
|
||||
including raw outcomes and metrics. The contract pins the public key and exact
|
||||
config SHA-256; the private key remains outside Git at mode `0600`.
|
||||
- Quantized drift remains advisory. Only the near-lossless lane can satisfy the
|
||||
quality gate, and only performance-fit recipes can earn speed/fit benefits.
|
||||
|
||||
@@ -120,8 +124,8 @@ Thresholds were not changed after observing these results.
|
||||
## Verification
|
||||
|
||||
```text
|
||||
Targeted: 22 passed
|
||||
Full suite: 749 passed, 13 skipped
|
||||
Targeted: 24 passed
|
||||
Full suite: 751 passed, 13 skipped
|
||||
Earlier cancellation retry matrix, DGR-001: 4/5 passed
|
||||
Earlier cancellation retry matrix, clean d904c40: 4/5 passed
|
||||
compileall: passed
|
||||
@@ -129,9 +133,9 @@ git diff --check: passed
|
||||
Evidence JSON parse/integrity checks: passed
|
||||
```
|
||||
|
||||
The full-suite exception is documented in `known-unrelated-failure.md` and
|
||||
satisfies the issue's explicit clean-tree reproduction clause. DGR-001 changes
|
||||
no tracker/proxy files.
|
||||
An earlier intermittent tracker cancellation race reproduced at the same rate on
|
||||
the clean base and is retained in `known-unrelated-failure.md`; the final suite
|
||||
completed green. DGR-001 changes no tracker/proxy files.
|
||||
|
||||
The earlier Ralph claim that the full suite was blocked by Protobuf 6.33.6 was
|
||||
invalid: it used Hermes Agent's internal venv. Verification above used the
|
||||
|
||||
@@ -1,4 +1,14 @@
|
||||
{
|
||||
"artifact_sha256": {
|
||||
"llama-cpp-near-lossless-quality": "e842fdc35d7f00fda95a54e1b51731ba1d196aea45065cc9f46925fdc1d6f862",
|
||||
"llama-cpp-quantized-performance-fit": "a88e3f570e2efeaf06b50df9859db2c70d8646aa3a2c94a14e14d5797a2921a5",
|
||||
"transformers-safetensors-reference": "e596e9d6205fdc9177569cccd7f8b471b058f66e3630c8e4326d5aad52bd18b6"
|
||||
},
|
||||
"backend_detail": {
|
||||
"llama-cpp-near-lossless-quality": "version: 9991 (e920c523) | built with GNU 15.2.1 for Linux x86_64; binary sha256 fd8fe612970f23e447f2e717cfa51665be06b8d7315ba60556e010f6bca510dd; threads 16; parallel slots 4; ctx/slot 512; gpu layers 0",
|
||||
"llama-cpp-quantized-performance-fit": "version: 9991 (e920c523) | built with GNU 15.2.1 for Linux x86_64; binary sha256 fd8fe612970f23e447f2e717cfa51665be06b8d7315ba60556e010f6bca510dd; threads 16; parallel slots 4; ctx/slot 512; gpu layers 0",
|
||||
"transformers-safetensors-reference": "torch 2.10.0+rocm7.13.0a20260513; dtype bfloat16; device cpu; intra-op threads 16"
|
||||
},
|
||||
"evidence_class": "local-real",
|
||||
"host": {
|
||||
"accelerator_name": "Radeon 8060S Graphics",
|
||||
@@ -10,6 +20,12 @@
|
||||
"hostname": "fedora",
|
||||
"llama_cpp_commit": "e920c523e3b8a0163fe498af5bf90df35ff51d25",
|
||||
"llama_cpp_version": "9991",
|
||||
"llama_server_identities": {
|
||||
"/run/media/popov/d/DEV/llamacpp/llama.cpp/build/bin/llama-server": {
|
||||
"sha256": "fd8fe612970f23e447f2e717cfa51665be06b8d7315ba60556e010f6bca510dd",
|
||||
"version": "version: 9991 (e920c523) | built with GNU 15.2.1 for Linux x86_64"
|
||||
}
|
||||
},
|
||||
"llama_server_sha256": "fd8fe612970f23e447f2e717cfa51665be06b8d7315ba60556e010f6bca510dd",
|
||||
"platform": "Linux-7.0.14-101.fc43.x86_64-x86_64-with-glibc2.42",
|
||||
"python": "3.12.13",
|
||||
@@ -20,34 +36,65 @@
|
||||
"model_id": "Qwen/Qwen2.5-0.5B-Instruct",
|
||||
"model_revision": "7ae557604adf67be50417f59c2c2f167def9a775",
|
||||
"plan_sha256": "efe24690a9a7164bac6ab3fd0a6b22f078fc08aaefcfb96210ddf154e6050570",
|
||||
"provenance": {
|
||||
"completed_at": "2026-07-13T16:27:19.647692Z",
|
||||
"config_sha256": "00b2cce3e2f281bdf92fc5304ba5cac915a178ffccd3b9a25995ce39c00b90d3",
|
||||
"producer": "meshnet_node.recipe_drivers.run_configured_benchmark/v1",
|
||||
"run_id": "e4eedadf-22f6-4907-8990-985456961099",
|
||||
"schema_version": 1,
|
||||
"signature": "owev+/ToswP20C923G6E+srOCUBV5vrjmndVatr9CbTXakiFGqlHrTiEo+aymA4BcSwmG6KJTxlxO6WpLnpcAg==",
|
||||
"signature_algorithm": "ed25519",
|
||||
"signer_public_key_sha256": "8baca8742d9b3ed0c3fc54929c23f75ec8c1c739900aaf5334780d598ffa84de",
|
||||
"started_at": "2026-07-13T16:26:22.361501Z"
|
||||
},
|
||||
"recipe_runtime": {
|
||||
"llama-cpp-near-lossless-quality": {
|
||||
"device": "cpu",
|
||||
"runtime": "llama.cpp-9991-e920c523",
|
||||
"weight_format": "gguf",
|
||||
"weight_quantization": "bfloat16"
|
||||
},
|
||||
"llama-cpp-quantized-performance-fit": {
|
||||
"device": "cpu",
|
||||
"runtime": "llama.cpp-9991-e920c523",
|
||||
"weight_format": "gguf",
|
||||
"weight_quantization": "Q4_K_M"
|
||||
},
|
||||
"transformers-safetensors-reference": {
|
||||
"device": "cpu",
|
||||
"runtime": "transformers-5.13.0",
|
||||
"weight_format": "safetensors",
|
||||
"weight_quantization": "bfloat16"
|
||||
}
|
||||
},
|
||||
"recipes": {
|
||||
"llama-cpp-near-lossless-quality": {
|
||||
"artifact_bytes": 994156448,
|
||||
"available": true,
|
||||
"concurrency": {
|
||||
"1": {
|
||||
"aggregate_decode_tokens_per_sec": 73.7861,
|
||||
"decode_tokens_per_sec": 87.9728,
|
||||
"aggregate_decode_tokens_per_sec": 86.7339,
|
||||
"decode_tokens_per_sec": 98.5178,
|
||||
"failures": 0,
|
||||
"latency_p50_ms": 385.2049,
|
||||
"latency_p95_ms": 560.2939,
|
||||
"peak_rss_bytes": 1110708224,
|
||||
"latency_p50_ms": 333.023,
|
||||
"latency_p95_ms": 383.0597,
|
||||
"peak_rss_bytes": 1110728704,
|
||||
"peak_vram_bytes": 0,
|
||||
"prefill_tokens_per_sec": 1427.2072,
|
||||
"ttft_p50_ms": 43.929,
|
||||
"ttft_p95_ms": 107.003
|
||||
"prefill_tokens_per_sec": 1717.9451,
|
||||
"ttft_p50_ms": 15.069,
|
||||
"ttft_p95_ms": 63.766
|
||||
},
|
||||
"4": {
|
||||
"aggregate_decode_tokens_per_sec": 211.3515,
|
||||
"decode_tokens_per_sec": 73.8932,
|
||||
"aggregate_decode_tokens_per_sec": 222.788,
|
||||
"decode_tokens_per_sec": 76.6297,
|
||||
"failures": 0,
|
||||
"latency_p50_ms": 467.5094,
|
||||
"latency_p95_ms": 790.862,
|
||||
"peak_rss_bytes": 1129578496,
|
||||
"latency_p50_ms": 490.8738,
|
||||
"latency_p95_ms": 646.26,
|
||||
"peak_rss_bytes": 1139466240,
|
||||
"peak_vram_bytes": 0,
|
||||
"prefill_tokens_per_sec": 1077.8162,
|
||||
"ttft_p50_ms": 33.612,
|
||||
"ttft_p95_ms": 128.501
|
||||
"prefill_tokens_per_sec": 859.8985,
|
||||
"ttft_p50_ms": 32.445,
|
||||
"ttft_p95_ms": 218.387
|
||||
}
|
||||
},
|
||||
"device": "cpu",
|
||||
@@ -58,28 +105,28 @@
|
||||
"available": true,
|
||||
"concurrency": {
|
||||
"1": {
|
||||
"aggregate_decode_tokens_per_sec": 110.0458,
|
||||
"decode_tokens_per_sec": 170.131,
|
||||
"aggregate_decode_tokens_per_sec": 139.2693,
|
||||
"decode_tokens_per_sec": 207.712,
|
||||
"failures": 0,
|
||||
"latency_p50_ms": 258.0681,
|
||||
"latency_p95_ms": 465.8523,
|
||||
"peak_rss_bytes": 542167040,
|
||||
"latency_p50_ms": 168.3307,
|
||||
"latency_p95_ms": 305.1338,
|
||||
"peak_rss_bytes": 542081024,
|
||||
"peak_vram_bytes": 0,
|
||||
"prefill_tokens_per_sec": 783.3775,
|
||||
"ttft_p50_ms": 23.847,
|
||||
"ttft_p95_ms": 237.696
|
||||
"prefill_tokens_per_sec": 967.0195,
|
||||
"ttft_p50_ms": 21.582,
|
||||
"ttft_p95_ms": 147.859
|
||||
},
|
||||
"4": {
|
||||
"aggregate_decode_tokens_per_sec": 206.377,
|
||||
"decode_tokens_per_sec": 83.543,
|
||||
"aggregate_decode_tokens_per_sec": 195.6789,
|
||||
"decode_tokens_per_sec": 76.9497,
|
||||
"failures": 0,
|
||||
"latency_p50_ms": 413.3897,
|
||||
"latency_p95_ms": 910.253,
|
||||
"peak_rss_bytes": 572788736,
|
||||
"latency_p50_ms": 437.9196,
|
||||
"latency_p95_ms": 885.5355,
|
||||
"peak_rss_bytes": 573259776,
|
||||
"peak_vram_bytes": 0,
|
||||
"prefill_tokens_per_sec": 474.3116,
|
||||
"ttft_p50_ms": 67.945,
|
||||
"ttft_p95_ms": 431.804
|
||||
"prefill_tokens_per_sec": 572.4424,
|
||||
"ttft_p50_ms": 48.127,
|
||||
"ttft_p95_ms": 416.531
|
||||
}
|
||||
},
|
||||
"device": "cpu",
|
||||
@@ -90,28 +137,28 @@
|
||||
"available": true,
|
||||
"concurrency": {
|
||||
"1": {
|
||||
"aggregate_decode_tokens_per_sec": 40.3425,
|
||||
"decode_tokens_per_sec": 46.1451,
|
||||
"aggregate_decode_tokens_per_sec": 35.4722,
|
||||
"decode_tokens_per_sec": 40.7545,
|
||||
"failures": 0,
|
||||
"latency_p50_ms": 795.4807,
|
||||
"latency_p95_ms": 930.9725,
|
||||
"peak_rss_bytes": 1941213184,
|
||||
"latency_p50_ms": 818.3864,
|
||||
"latency_p95_ms": 1258.0673,
|
||||
"peak_rss_bytes": 1941458944,
|
||||
"peak_vram_bytes": 0,
|
||||
"prefill_tokens_per_sec": 671.8016,
|
||||
"ttft_p50_ms": 37.4548,
|
||||
"ttft_p95_ms": 193.4633
|
||||
"prefill_tokens_per_sec": 625.6467,
|
||||
"ttft_p50_ms": 40.0018,
|
||||
"ttft_p95_ms": 195.2551
|
||||
},
|
||||
"4": {
|
||||
"aggregate_decode_tokens_per_sec": 47.0903,
|
||||
"decode_tokens_per_sec": 13.1337,
|
||||
"aggregate_decode_tokens_per_sec": 46.5375,
|
||||
"decode_tokens_per_sec": 12.9506,
|
||||
"failures": 0,
|
||||
"latency_p50_ms": 2631.0031,
|
||||
"latency_p95_ms": 3073.7389,
|
||||
"peak_rss_bytes": 2177265664,
|
||||
"latency_p50_ms": 2481.8662,
|
||||
"latency_p95_ms": 3365.8395,
|
||||
"peak_rss_bytes": 2104832000,
|
||||
"peak_vram_bytes": 0,
|
||||
"prefill_tokens_per_sec": 247.5617,
|
||||
"ttft_p50_ms": 94.3995,
|
||||
"ttft_p95_ms": 444.6749
|
||||
"prefill_tokens_per_sec": 264.0101,
|
||||
"ttft_p50_ms": 97.0403,
|
||||
"ttft_p95_ms": 429.0665
|
||||
}
|
||||
},
|
||||
"device": "cpu",
|
||||
|
||||
@@ -4,6 +4,10 @@ LLAMA=/run/media/popov/d/DEV/llamacpp/llama.cpp
|
||||
ROCM_PY=/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv-rocm/bin/python
|
||||
PROJECT_PY=/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python
|
||||
OUT=/run/media/popov/DATA/llm/dgr-001
|
||||
SIGNING_KEY=/home/popov/.config/neuron-tai/keys/dgr-001-evidence-ed25519.pem
|
||||
|
||||
# Private signing key is outside Git and must remain owner-only
|
||||
stat -c '%a %n' "$SIGNING_KEY" # expected: 600
|
||||
|
||||
# Converter support check (no writes)
|
||||
$ROCM_PY $LLAMA/convert_hf_to_gguf.py "$SOURCE" --outtype f16 --outfile "$OUT/Qwen2.5-0.5B-Instruct-7ae5576-F16.gguf" --dry-run
|
||||
@@ -27,7 +31,8 @@ print(_artifact_sha256(Path('/run/media/popov/DATA/llm/safetensor/models/models-
|
||||
PY
|
||||
|
||||
# Canonical opt-in local-real benchmark
|
||||
MESHNET_ENABLE_REAL_INFERENCE_TESTS=1 PYTHONPATH=packages/node $ROCM_PY -m meshnet_node.recipe_benchmark \
|
||||
MESHNET_ENABLE_REAL_INFERENCE_TESTS=1 MESHNET_EVIDENCE_SIGNING_KEY="$SIGNING_KEY" \
|
||||
PYTHONPATH=packages/node $ROCM_PY -m meshnet_node.recipe_benchmark \
|
||||
--config .scratch/distributed-gguf-runtime/evidence/DGR-001/benchmark-config.json \
|
||||
--json-out .scratch/distributed-gguf-runtime/evidence/DGR-001/results.json \
|
||||
--summary-out .scratch/distributed-gguf-runtime/evidence/DGR-001/results.txt
|
||||
|
||||
@@ -17,22 +17,22 @@
|
||||
"lane": "quality",
|
||||
"measurements": {
|
||||
"aggregate_concurrency": 4,
|
||||
"aggregate_throughput_speedup": 4.4882,
|
||||
"aggregate_throughput_speedup": 4.7873,
|
||||
"artifact_size_ratio": 0.9946,
|
||||
"artifact_size_win": false,
|
||||
"compared_prompts": 3,
|
||||
"decode_speedup": 1.9064,
|
||||
"decode_speedup": 2.4173,
|
||||
"exact_match_rate": 0.3333,
|
||||
"expected_prompts": 3,
|
||||
"failure_rate": 0.0,
|
||||
"mean_similarity": 0.9471,
|
||||
"resident_memory_ratio": 0.5722,
|
||||
"ttft_ratio": 1.1729
|
||||
"resident_memory_ratio": 0.5721,
|
||||
"ttft_ratio": 0.3767
|
||||
},
|
||||
"quality_pass": false,
|
||||
"reasons": [
|
||||
"single-request decode 1.91x reference (>= 1.25x) at TTFT ratio 1.17",
|
||||
"aggregate throughput at concurrency 4 is 4.49x reference (>= 1.25x)",
|
||||
"single-request decode 2.42x reference (>= 1.25x) at TTFT ratio 0.38",
|
||||
"aggregate throughput at concurrency 4 is 4.79x reference (>= 1.25x)",
|
||||
"peak resident memory is 0.57x reference (<= 0.75x)",
|
||||
"quality lane exact-match 0.33 / similarity 0.947 versus the reference (fail)"
|
||||
],
|
||||
@@ -47,18 +47,18 @@
|
||||
"lane": "performance-fit",
|
||||
"measurements": {
|
||||
"aggregate_concurrency": 4,
|
||||
"aggregate_throughput_speedup": 4.3826,
|
||||
"aggregate_throughput_speedup": 4.2048,
|
||||
"artifact_size_ratio": 0.398,
|
||||
"artifact_size_win": true,
|
||||
"decode_speedup": 3.6869,
|
||||
"decode_speedup": 5.0967,
|
||||
"failure_rate": 0.0,
|
||||
"resident_memory_ratio": 0.2793,
|
||||
"ttft_ratio": 0.6367
|
||||
"resident_memory_ratio": 0.2792,
|
||||
"ttft_ratio": 0.5395
|
||||
},
|
||||
"quality_pass": null,
|
||||
"reasons": [
|
||||
"single-request decode 3.69x reference (>= 1.25x) at TTFT ratio 0.64",
|
||||
"aggregate throughput at concurrency 4 is 4.38x reference (>= 1.25x)",
|
||||
"single-request decode 5.10x reference (>= 1.25x) at TTFT ratio 0.54",
|
||||
"aggregate throughput at concurrency 4 is 4.20x reference (>= 1.25x)",
|
||||
"peak resident memory is 0.28x reference (<= 0.75x)"
|
||||
],
|
||||
"recipe_id": "llama-cpp-quantized-performance-fit",
|
||||
|
||||
@@ -37,8 +37,51 @@
|
||||
"minimum_prompt_count": 3,
|
||||
"minimum_repeats": 3,
|
||||
"minimum_output_tokens": 32,
|
||||
"required_device": "cpu"
|
||||
"required_device": "cpu",
|
||||
"required_config_sha256": "00b2cce3e2f281bdf92fc5304ba5cac915a178ffccd3b9a25995ce39c00b90d3",
|
||||
"required_signer_public_key": "zQ/qRMwF/ydazzaxEI24Xvnrl5bZxzw16JYpP0bfRuI=",
|
||||
"required_artifact_sha256": {
|
||||
"transformers-safetensors-reference": "e596e9d6205fdc9177569cccd7f8b471b058f66e3630c8e4326d5aad52bd18b6",
|
||||
"llama-cpp-near-lossless-quality": "e842fdc35d7f00fda95a54e1b51731ba1d196aea45065cc9f46925fdc1d6f862",
|
||||
"llama-cpp-quantized-performance-fit": "a88e3f570e2efeaf06b50df9859db2c70d8646aa3a2c94a14e14d5797a2921a5"
|
||||
},
|
||||
"required_recipe_runtime": {
|
||||
"transformers-safetensors-reference": {
|
||||
"runtime": "transformers-5.13.0",
|
||||
"weight_format": "safetensors",
|
||||
"weight_quantization": "bfloat16",
|
||||
"device": "cpu"
|
||||
},
|
||||
"llama-cpp-near-lossless-quality": {
|
||||
"runtime": "llama.cpp-9991-e920c523",
|
||||
"weight_format": "gguf",
|
||||
"weight_quantization": "bfloat16",
|
||||
"device": "cpu"
|
||||
},
|
||||
"llama-cpp-quantized-performance-fit": {
|
||||
"runtime": "llama.cpp-9991-e920c523",
|
||||
"weight_format": "gguf",
|
||||
"weight_quantization": "Q4_K_M",
|
||||
"device": "cpu"
|
||||
}
|
||||
},
|
||||
"required_backend_detail": {
|
||||
"transformers-safetensors-reference": "torch 2.10.0+rocm7.13.0a20260513; dtype bfloat16; device cpu; intra-op threads 16",
|
||||
"llama-cpp-near-lossless-quality": "version: 9991 (e920c523) | built with GNU 15.2.1 for Linux x86_64; binary sha256 fd8fe612970f23e447f2e717cfa51665be06b8d7315ba60556e010f6bca510dd; threads 16; parallel slots 4; ctx/slot 512; gpu layers 0",
|
||||
"llama-cpp-quantized-performance-fit": "version: 9991 (e920c523) | built with GNU 15.2.1 for Linux x86_64; binary sha256 fd8fe612970f23e447f2e717cfa51665be06b8d7315ba60556e010f6bca510dd; threads 16; parallel slots 4; ctx/slot 512; gpu layers 0"
|
||||
},
|
||||
"required_host_identity": {
|
||||
"python": "3.12.13",
|
||||
"torch_version": "2.10.0+rocm7.13.0a20260513",
|
||||
"transformers_version": "5.13.0",
|
||||
"llama_server_identities": {
|
||||
"/run/media/popov/d/DEV/llamacpp/llama.cpp/build/bin/llama-server": {
|
||||
"sha256": "fd8fe612970f23e447f2e717cfa51665be06b8d7315ba60556e010f6bca510dd",
|
||||
"version": "version: 9991 (e920c523) | built with GNU 15.2.1 for Linux x86_64"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"stop_condition": "Stop the native llama.cpp/GGUF track when, on the same machine and device as the Transformers/safetensors reference and under this plan, no performance-fit GGUF recipe delivers either a meaningful speed benefit (>=25% higher single-request decode tokens/sec without a >25% worse TTFT, or >=25% higher aggregate throughput under concurrency) or a meaningful fit benefit (>=25% lower peak resident memory), or when the near-lossless quality lane fails, which indicates a broken runtime rather than a quantization trade-off.",
|
||||
"notes": "Quantized performance-fit output drift is reported as advisory only. It is not numerical-equivalence evidence. DGR-014 consumes this immutable v1 contract."
|
||||
"notes": "Quantized performance-fit output drift is reported as advisory only. It is not numerical-equivalence evidence. DGR-014 consumes this immutable v1 contract. Non-synthetic evidence must be Ed25519-signed by the pinned key and match the exact locked config, artifacts, runtimes, backends, and host runtime identity."
|
||||
}
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,10 +1,10 @@
|
||||
Recipe benchmark dgr-001-controlled-whole-model-baseline-v1 (local-real)
|
||||
model Qwen/Qwen2.5-0.5B-Instruct@7ae557604adf67be50417f59c2c2f167def9a775
|
||||
transformers-safetensors-reference [quality ] c= 1 ttft p50/p95 37.5/ 193.5 ms; prefill 671.8 tok/s; decode 46.1 tok/s; aggregate 40.3 tok/s; rss 1.94 GB; vram 0.00 GB; artifact 1.00 GB; failures 0
|
||||
transformers-safetensors-reference [quality ] c= 4 ttft p50/p95 94.4/ 444.7 ms; prefill 247.6 tok/s; decode 13.1 tok/s; aggregate 47.1 tok/s; rss 2.18 GB; vram 0.00 GB; artifact 1.00 GB; failures 0
|
||||
llama-cpp-near-lossless-quality [quality ] c= 1 ttft p50/p95 43.9/ 107.0 ms; prefill 1427.2 tok/s; decode 88.0 tok/s; aggregate 73.8 tok/s; rss 1.11 GB; vram 0.00 GB; artifact 0.99 GB; failures 0
|
||||
llama-cpp-near-lossless-quality [quality ] c= 4 ttft p50/p95 33.6/ 128.5 ms; prefill 1077.8 tok/s; decode 73.9 tok/s; aggregate 211.4 tok/s; rss 1.13 GB; vram 0.00 GB; artifact 0.99 GB; failures 0
|
||||
llama-cpp-quantized-performance-fit [performance-fit ] c= 1 ttft p50/p95 23.8/ 237.7 ms; prefill 783.4 tok/s; decode 170.1 tok/s; aggregate 110.0 tok/s; rss 0.54 GB; vram 0.00 GB; artifact 0.40 GB; failures 0
|
||||
llama-cpp-quantized-performance-fit [performance-fit ] c= 4 ttft p50/p95 67.9/ 431.8 ms; prefill 474.3 tok/s; decode 83.5 tok/s; aggregate 206.4 tok/s; rss 0.57 GB; vram 0.00 GB; artifact 0.40 GB; failures 0
|
||||
transformers-safetensors-reference [quality ] c= 1 ttft p50/p95 40.0/ 195.3 ms; prefill 625.6 tok/s; decode 40.8 tok/s; aggregate 35.5 tok/s; rss 1.94 GB; vram 0.00 GB; artifact 1.00 GB; failures 0
|
||||
transformers-safetensors-reference [quality ] c= 4 ttft p50/p95 97.0/ 429.1 ms; prefill 264.0 tok/s; decode 13.0 tok/s; aggregate 46.5 tok/s; rss 2.10 GB; vram 0.00 GB; artifact 1.00 GB; failures 0
|
||||
llama-cpp-near-lossless-quality [quality ] c= 1 ttft p50/p95 15.1/ 63.8 ms; prefill 1717.9 tok/s; decode 98.5 tok/s; aggregate 86.7 tok/s; rss 1.11 GB; vram 0.00 GB; artifact 0.99 GB; failures 0
|
||||
llama-cpp-near-lossless-quality [quality ] c= 4 ttft p50/p95 32.4/ 218.4 ms; prefill 859.9 tok/s; decode 76.6 tok/s; aggregate 222.8 tok/s; rss 1.14 GB; vram 0.00 GB; artifact 0.99 GB; failures 0
|
||||
llama-cpp-quantized-performance-fit [performance-fit ] c= 1 ttft p50/p95 21.6/ 147.9 ms; prefill 967.0 tok/s; decode 207.7 tok/s; aggregate 139.3 tok/s; rss 0.54 GB; vram 0.00 GB; artifact 0.40 GB; failures 0
|
||||
llama-cpp-quantized-performance-fit [performance-fit ] c= 4 ttft p50/p95 48.1/ 416.5 ms; prefill 572.4 tok/s; decode 76.9 tok/s; aggregate 195.7 tok/s; rss 0.57 GB; vram 0.00 GB; artifact 0.40 GB; failures 0
|
||||
drift llama-cpp-near-lossless-quality vs transformers-safetensors-reference exact 0.33; similarity 0.947 (gated)
|
||||
drift llama-cpp-quantized-performance-fit vs transformers-safetensors-reference exact 0.00; similarity 0.456 (advisory)
|
||||
|
||||
Reference in New Issue
Block a user