feat: add signed ROCm diagnostic lane
This commit is contained in:
Dobromir Popov
@@ -8,3 +8,4 @@
|
||||
- **Node capability admission** — `.scratch/node-capability-admission/` (P0 plan: generic doctor/real-forward validation, fail-closed readiness, tracker admission gate; [PRD](../../.scratch/node-capability-admission/PRD.md), [README](../../.scratch/node-capability-admission/README.md), ADR-0023)
|
||||
- **Distributed relay performance** — relay `/rpc` requester sockets are persistent per Route Session and Activation Seam as of 2026-07-10; `request_id` remains unique per activation while `X-Meshnet-Session` remains stable for KV state. Next low-risk priorities: persistent direct/loopback HTTP, seam byte/latency telemetry, then trace-driven zstd tuning.
|
||||
- **Distributed GGUF direction** — benchmark-gated native runtime: compare controlled Transformers/safetensors and whole-model llama.cpp lanes before expensive work; ship only for measured speed or model-fit advantage. Public parallelism is contiguous Shards in an Inference Route; concurrency comes from per-node continuous batching across isolated Route Sessions, while tensor/expert collectives stay inside optional trusted composite providers. Native data plane uses versioned Protobuf over long-lived gRPC/HTTP2 seam streams, with existing relay carrying the same opaque frames when needed. llama.cpp/GGML remains the substrate behind a project-owned standalone worker and small pinned fork; vLLM is an optional complete managed provider and concept donor, not a fork. Nakshatra, `prima.cpp`, `llama-gguf`, LiGGUF and historical GPUStack are source/test donors only. Active plan: [README](../../.scratch/distributed-gguf-runtime/README.md), [architecture](../../.scratch/distributed-gguf-runtime/architecture.md), [PRD](../../.scratch/distributed-gguf-runtime/PRD.md), [Ralph backlog](../../.scratch/distributed-gguf-runtime/prd.json). Research: [landscape](../../docs/research/distributed-gguf-landscape.md), [GitHub follow-up](../../docs/research/distributed-gguf-github-followup.md), [vLLM](../../docs/research/vllm-distributed-gguf-assessment.md).
|
||||
- [DGR ROCm setup](dgr-rocm-setup.md) — version-matched TheRock SDK layout, relocated devel payload, verified `gfx1151` HIP llama.cpp build, and GPU-diagnostic boundary.
|
||||
|
||||
29
.claude/memory/dgr-rocm-setup.md
Normal file
29
.claude/memory/dgr-rocm-setup.md
Normal file
@@ -0,0 +1,29 @@
|
||||
# DGR ROCm and llama.cpp setup
|
||||
|
||||
As of 2026-07-13:
|
||||
|
||||
- Project ROCm runtime: `/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv-rocm`
|
||||
- ROCm/TheRock build: `7.13.0a20260513`, target `gfx1151`
|
||||
- `rocm-sdk-devel` is installed. Its expanded SDK lives under the venv at
|
||||
`site-packages/_rocm_sdk_devel`.
|
||||
- The wheel's redundant packaged payload was relocated to
|
||||
`/home/popov/.local/share/rocm-sdk/7.13.0a20260513/rocm_sdk_devel` and symlinked
|
||||
back into the venv because installing both packaged and expanded forms filled
|
||||
the mounted drive. Do not reinstall it blindly; the wheel expands beyond
|
||||
20 GB.
|
||||
- HIP llama.cpp source: `/run/media/popov/d/DEV/llamacpp/llama.cpp`, commit
|
||||
`e920c523e3b8a0163fe498af5bf90df35ff51d25` (version 9991).
|
||||
- HIP build: `/run/media/popov/d/DEV/llamacpp/llama.cpp/build-hip`
|
||||
- HIP `llama-server` SHA-256:
|
||||
`b6bb4da687dbde86e243ba006cef05919b7b97255cd7e2371e1d451220aca139`
|
||||
- Verified device: `ROCm0: Radeon 8060S Graphics`, `gfx1151`.
|
||||
- Model artifacts remain under `/run/media/popov/DATA/llm`; none were put under
|
||||
`/home`.
|
||||
|
||||
DGR-001's immutable contract remains CPU-only. GPU evidence uses the distinct
|
||||
signed `gpu-diagnostic` profile because llama-server process VRAM is not yet
|
||||
measurable by the benchmark driver. The profile must capture measured
|
||||
llama-server startup evidence for `ROCm0` and the actual offloaded/total layer
|
||||
count; configured `device` and `n_gpu_layers` values alone are not evidence.
|
||||
The accepted signer fingerprint is anchored in
|
||||
`.scratch/distributed-gguf-runtime/trusted-evidence-signers.json`.
|
||||
@@ -65,9 +65,11 @@ The near-lossless BF16 quality lane compared all three prompts but measured:
|
||||
- mean text similarity: **0.9471** (v1 requires at least `0.97`).
|
||||
|
||||
Tokenization and stopping were controlled: every runtime saw the same prompt
|
||||
token counts and reported 31 post-TTFT decode tokens. The mismatch is genuine
|
||||
greedy runtime divergence on two prompts, not missing coverage or a text-length
|
||||
artifact. Therefore `contract-evaluation.json` records:
|
||||
token counts and reported 31 post-TTFT decode tokens. The v1 mismatch is a
|
||||
real greedy-output divergence on two prompts, not missing coverage or a
|
||||
text-length artifact. Its root cause remains undetermined; no post-contract
|
||||
logit-tie claim is acceptance evidence. Therefore `contract-evaluation.json`
|
||||
records:
|
||||
|
||||
```text
|
||||
verdict: stop
|
||||
@@ -79,6 +81,35 @@ stop_condition_met: true
|
||||
|
||||
Thresholds were not changed after observing these results.
|
||||
|
||||
## Post-contract parity and ROCm diagnostics
|
||||
|
||||
`summarize-quality-parity.py` verifies and separates two signed sources. The CPU
|
||||
v1 row uses CPU kernels and a Transformers BF16 oracle; it remains at `0.3333`
|
||||
exact match with an unexplained divergence. The ROCm row uses a different plan,
|
||||
GPU kernels, and a Transformers float32 oracle. In that narrower diagnostic,
|
||||
the same BF16 GGUF artifact matches all three 32-token sequences exactly (`1.0`
|
||||
exact match and `1.0` similarity). No conversion corruption was observed in
|
||||
that three-sequence ROCm sample; this does not prove global conversion
|
||||
correctness or explain the CPU result.
|
||||
|
||||
A separate HIP build at commit `e920c523` was compiled for `gfx1151` and
|
||||
measured `ROCm0: Radeon 8060S Graphics`; its `llama-server` SHA-256 is
|
||||
`b6bb4da687dbde86e243ba006cef05919b7b97255cd7e2371e1d451220aca139`.
|
||||
A signed `gpu-diagnostic` profile measured zero failures:
|
||||
|
||||
| GPU metric | Transformers BF16 ROCm | llama.cpp Q4 ROCm | Q4 ratio |
|
||||
|---|---:|---:|---:|
|
||||
| Decode tok/s, c=1 | 81.12 | 251.25 | **3.10×** |
|
||||
| Aggregate decode tok/s, c=4 | 91.24 | 511.33 | **5.60×** |
|
||||
| TTFT p50, c=1 | 13.77 ms | 11.80 ms | **0.857×** |
|
||||
|
||||
The GPU report is signed under the distinct
|
||||
`run_configured_gpu_diagnostic/v1` producer. The v1 evaluator rejects that
|
||||
producer even when its signature is valid. llama-server process VRAM remains
|
||||
unmeasured, so this diagnostic cannot replace or satisfy the immutable v1
|
||||
contract. Its signed backend detail records the measured `ROCm0: Radeon 8060S
|
||||
Graphics` device and `25/25` offloaded layers.
|
||||
|
||||
## Implementation
|
||||
|
||||
- `recipe_benchmark.py` provides the runtime-neutral measurement core, true
|
||||
@@ -86,8 +117,9 @@ Thresholds were not changed after observing these results.
|
||||
aggregation, failures, and output drift.
|
||||
- `recipe_drivers.py` provides opt-in Transformers and llama-server drivers,
|
||||
mounted-drive confinement, exact artifact/runtime verification, equal
|
||||
device/thread budgets, greedy-only validation, measured host provenance, and
|
||||
a CPU-only v1 guard until process VRAM can be measured honestly.
|
||||
device/thread budgets, greedy-only validation, measured host provenance, a
|
||||
CPU-only v1 guard until process VRAM can be measured honestly, and a distinct
|
||||
signed GPU diagnostic profile that the v1 evaluator cannot accept.
|
||||
- Peak RSS is runtime-scoped: Transformers reports growth above its pre-runtime
|
||||
Python baseline, while llama.cpp reports its isolated server process tree.
|
||||
Both are sampled continuously during in-flight requests.
|
||||
@@ -106,6 +138,8 @@ Thresholds were not changed after observing these results.
|
||||
- Every non-synthetic report is Ed25519-signed over the complete canonical JSON,
|
||||
including raw outcomes and metrics. The contract pins the public key and exact
|
||||
config SHA-256; the private key remains outside Git at mode `0600`.
|
||||
- The signer fingerprint is independently anchored outside this evidence
|
||||
directory in `../../trusted-evidence-signers.json` and checked by tests.
|
||||
- Quantized drift remains advisory. Only the near-lossless lane can satisfy the
|
||||
quality gate, and only performance-fit recipes can earn speed/fit benefits.
|
||||
|
||||
@@ -117,24 +151,30 @@ Thresholds were not changed after observing these results.
|
||||
- `results.txt` — human-readable benchmark summary
|
||||
- `baseline.json` — distilled measurements for later comparison
|
||||
- `contract-evaluation.json` — fail-closed v1 verdict
|
||||
- `quality-parity-diagnosis.json` / `.md` — run/device-scoped signed-evidence summary
|
||||
- `summarize-quality-parity.py` — verifies both evidence chains and regenerates it
|
||||
- `gpu-diagnostic-config.json` — exact ROCm diagnostic artifacts and runtimes
|
||||
- `gpu-diagnostic-results.json` / `.txt` — signed GPU outcomes and summary
|
||||
- `commands.txt` — reproducible conversion, benchmark, evaluation, and test commands
|
||||
- `BLOCKED.md` — downstream stop-condition handoff
|
||||
- `known-unrelated-failure.md` — clean-base reproduction of the tracker race
|
||||
- `../../trusted-evidence-signers.json` — repository-reviewed signer fingerprint
|
||||
|
||||
## Verification
|
||||
|
||||
```text
|
||||
Targeted: 24 passed
|
||||
Full suite: 751 passed, 13 skipped
|
||||
Earlier cancellation retry matrix, DGR-001: 4/5 passed
|
||||
Targeted: 28 passed (5/5 consecutive focused runs)
|
||||
Latest full suite: 755 passed, 13 skipped
|
||||
Earlier full suite: 751 passed, 13 skipped
|
||||
Current cancellation retry matrix, DGR-001: 4/5 passed
|
||||
Earlier cancellation retry matrix, clean d904c40: 4/5 passed
|
||||
compileall: passed
|
||||
git diff --check: passed
|
||||
Evidence JSON parse/integrity checks: passed
|
||||
```
|
||||
|
||||
An earlier intermittent tracker cancellation race reproduced at the same rate on
|
||||
the clean base and is retained in `known-unrelated-failure.md`; the final suite
|
||||
The intermittent tracker cancellation race reproduced at the same rate on the
|
||||
clean base and is retained in `known-unrelated-failure.md`; the final full suite
|
||||
completed green. DGR-001 changes no tracker/proxy files.
|
||||
|
||||
The earlier Ralph claim that the full suite was blocked by Protobuf 6.33.6 was
|
||||
@@ -144,10 +184,12 @@ project `.venv`, which has the DGR-002-compatible runtime. Real inference used
|
||||
|
||||
## Limitations and dependent-story handoff
|
||||
|
||||
- This is a **0.5B CPU baseline**, not evidence for a large model, Radeon GPU,
|
||||
distributed execution, network transport, or native shard worker.
|
||||
- The installed llama.cpp build is CPU-only (`GGML_HIP=OFF`). No GPU comparison
|
||||
is claimed.
|
||||
- The immutable contract result is a **0.5B CPU baseline**. The separate Radeon
|
||||
diagnostic is real local GPU evidence, but neither result covers a large
|
||||
model, distributed execution, network transport, or a native shard worker.
|
||||
- A separate `GGML_HIP=ON` llama.cpp build exists and produced GPU timings, but
|
||||
llama-server process VRAM is not measurable by the current driver; GPU
|
||||
memory/fit claims therefore remain ineligible for v1.
|
||||
- Absolute timings are developer-machine measurements; locked ratios and raw
|
||||
artifacts are provided for reproducibility.
|
||||
- DGR-014 may consume v1 only with the exact plan/evidence requirements enforced
|
||||
|
||||
@@ -49,6 +49,37 @@ contract = load_contract(root / 'performance-contract.json')
|
||||
(root / 'contract-evaluation.json').write_text(json.dumps(evaluate_contract(contract, report).to_dict(), indent=2, sort_keys=True) + '\n')
|
||||
PY
|
||||
|
||||
# Optional ROCm GPU diagnostic (not eligible for immutable v1)
|
||||
# The version-matched rocm[devel] wheel expands beyond 20 GB; ensure sufficient
|
||||
# space or relocate its packaged payload before installation.
|
||||
uv pip install --python "$ROCM_PY" --prerelease=allow \
|
||||
--index-url https://rocm.nightlies.amd.com/v2/gfx1151/ \
|
||||
'rocm[devel]==7.13.0a20260513'
|
||||
ROCM_VENV=/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv-rocm
|
||||
ROCM_SDK="$ROCM_VENV/bin/rocm-sdk"
|
||||
ROCM_ROOT="$($ROCM_SDK path --root)"
|
||||
ROCM_BIN="$($ROCM_SDK path --bin)"
|
||||
export PATH="$ROCM_VENV/bin:$ROCM_BIN:$PATH"
|
||||
export ROCM_PATH="$ROCM_ROOT" HIP_PATH="$ROCM_ROOT"
|
||||
export CMAKE_PREFIX_PATH="$($ROCM_SDK path --cmake):$ROCM_ROOT"
|
||||
export LD_LIBRARY_PATH="$ROCM_ROOT/lib:$ROCM_ROOT/lib64:${LD_LIBRARY_PATH:-}"
|
||||
$ROCM_VENV/bin/cmake -S /run/media/popov/d/DEV/llamacpp/llama.cpp \
|
||||
-B /run/media/popov/d/DEV/llamacpp/llama.cpp/build-hip -G Ninja \
|
||||
-DGGML_HIP=ON -DGPU_TARGETS=gfx1151 \
|
||||
-DCMAKE_HIP_COMPILER="$ROCM_VENV/bin/amdclang++" \
|
||||
-DCMAKE_BUILD_TYPE=Release -DLLAMA_BUILD_TESTS=OFF \
|
||||
-DLLAMA_BUILD_EXAMPLES=ON -DLLAMA_BUILD_SERVER=ON
|
||||
$ROCM_VENV/bin/cmake --build /run/media/popov/d/DEV/llamacpp/llama.cpp/build-hip \
|
||||
--target llama-server llama-cli llama-bench -j 16
|
||||
MESHNET_ENABLE_REAL_INFERENCE_TESTS=1 MESHNET_EVIDENCE_SIGNING_KEY="$SIGNING_KEY" \
|
||||
PYTHONPATH=packages/node $ROCM_PY -m meshnet_node.recipe_benchmark \
|
||||
--profile gpu-diagnostic \
|
||||
--config .scratch/distributed-gguf-runtime/evidence/DGR-001/gpu-diagnostic-config.json \
|
||||
--json-out .scratch/distributed-gguf-runtime/evidence/DGR-001/gpu-diagnostic-results.json \
|
||||
--summary-out .scratch/distributed-gguf-runtime/evidence/DGR-001/gpu-diagnostic-results.txt
|
||||
PYTHONPATH=packages/node $PROJECT_PY \
|
||||
.scratch/distributed-gguf-runtime/evidence/DGR-001/summarize-quality-parity.py
|
||||
|
||||
# Deterministic verification
|
||||
PYTHONPATH=packages/node $PROJECT_PY -m pytest -q tests/test_recipe_benchmark.py
|
||||
PYTHONPATH=packages/node $PROJECT_PY -m pytest -q
|
||||
|
||||
@@ -0,0 +1,143 @@
|
||||
{
|
||||
"artifact_storage_root": "/run/media/popov/DATA/llm",
|
||||
"evidence_class": "local-real",
|
||||
"host": {
|
||||
"benchmark_lane": "rocm-gpu-diagnostic",
|
||||
"llama_cpp_commit": "e920c523e3b8a0163fe498af5bf90df35ff51d25",
|
||||
"llama_cpp_version": "9991",
|
||||
"llama_server_sha256": "b6bb4da687dbde86e243ba006cef05919b7b97255cd7e2371e1d451220aca139",
|
||||
"converter_sha256": "c819f18fb22927b49fabc3b35d1c9e21ee638b3817eccd1bd4efbcc7116eeb4d",
|
||||
"quantizer_sha256": "bd0cc8c7be6d48aad4755b31062e0e59a887cbadd43dbb8771853d5858bb198f",
|
||||
"transformers_version": "5.13.0",
|
||||
"rocm_target": "gfx1151"
|
||||
},
|
||||
"plan": {
|
||||
"plan_id": "dgr-001-rocm-gpu-diagnostic-v1",
|
||||
"model_id": "Qwen/Qwen2.5-0.5B-Instruct",
|
||||
"model_revision": "7ae557604adf67be50417f59c2c2f167def9a775",
|
||||
"prompts": [
|
||||
{
|
||||
"id": "short-fact",
|
||||
"text": "The capital of France is",
|
||||
"context_class": "short"
|
||||
},
|
||||
{
|
||||
"id": "medium-code",
|
||||
"text": "Complete this Python function without commentary:\n\ndef fibonacci(n):\n \"\"\"Return the nth Fibonacci number for n >= 0.\"\"\"\n",
|
||||
"context_class": "medium"
|
||||
},
|
||||
{
|
||||
"id": "long-summary",
|
||||
"text": "A distributed inference service divides a transformer across consumer machines. The tracker owns admission, routing, cancellation, accounting, and telemetry, while workers own only model execution. Every request carries an immutable model identity and revision. Workers must reject incompatible protocol versions and resource demands before allocating large buffers. Activation tensors are chunked, checksummed, bounded by negotiated limits, and propagated with explicit flow-control credits. A caller may disconnect at any time, so cancellation must release queued work, in-flight transfers, and cache reservations without double billing. Retries can occur after network failures, requiring idempotent request identifiers and deterministic completion accounting. The system keeps the existing safetensors path as a correctness reference while a native GGUF path is measured. Benchmarks compare the same prompts, output lengths, sampling policy, device, and concurrency, and they separate near-lossless quality checks from quantized speed and fit claims. Summarize the design priorities in three concise bullet points.",
|
||||
"context_class": "long"
|
||||
}
|
||||
],
|
||||
"sampling": {
|
||||
"temperature": 0.0,
|
||||
"top_p": 1.0,
|
||||
"top_k": 1,
|
||||
"seed": 1234,
|
||||
"max_output_tokens": 32
|
||||
},
|
||||
"concurrency_levels": [
|
||||
1,
|
||||
4
|
||||
],
|
||||
"repeats": 3,
|
||||
"warmup_requests": 2
|
||||
},
|
||||
"recipes": [
|
||||
{
|
||||
"id": "transformers-fp32-rocm-quality-oracle",
|
||||
"runtime": "transformers-5.13.0-rocm-float32",
|
||||
"weight_format": "safetensors",
|
||||
"weight_quantization": "bfloat16-weights-float32-accumulation",
|
||||
"lane": "quality",
|
||||
"device": "cuda",
|
||||
"artifact_path": "/run/media/popov/DATA/llm/safetensor/models/models--Qwen--Qwen2.5-0.5B-Instruct/snapshots/7ae557604adf67be50417f59c2c2f167def9a775",
|
||||
"artifact_sha256": "e596e9d6205fdc9177569cccd7f8b471b058f66e3630c8e4326d5aad52bd18b6",
|
||||
"source_model_id": "Qwen/Qwen2.5-0.5B-Instruct",
|
||||
"source_model_revision": "7ae557604adf67be50417f59c2c2f167def9a775",
|
||||
"is_reference": true,
|
||||
"notes": "artifact_sha256 is the deterministic digest of every snapshot path and file byte",
|
||||
"driver": {
|
||||
"type": "transformers",
|
||||
"model_path": "/run/media/popov/DATA/llm/safetensor/models/models--Qwen--Qwen2.5-0.5B-Instruct/snapshots/7ae557604adf67be50417f59c2c2f167def9a775",
|
||||
"device": "cuda",
|
||||
"dtype": "float32",
|
||||
"threads": 16
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "llama-cpp-bf16-rocm-quality",
|
||||
"runtime": "llama.cpp-9991-e920c523-rocm-gfx1151",
|
||||
"weight_format": "gguf",
|
||||
"weight_quantization": "bfloat16",
|
||||
"lane": "quality",
|
||||
"device": "cuda",
|
||||
"artifact_path": "/run/media/popov/DATA/llm/dgr-001/Qwen2.5-0.5B-Instruct-7ae5576-BF16.gguf",
|
||||
"artifact_sha256": "e842fdc35d7f00fda95a54e1b51731ba1d196aea45065cc9f46925fdc1d6f862",
|
||||
"source_model_id": "Qwen/Qwen2.5-0.5B-Instruct",
|
||||
"source_model_revision": "7ae557604adf67be50417f59c2c2f167def9a775",
|
||||
"is_reference": false,
|
||||
"notes": "Converted directly from the exact mounted safetensors revision while preserving BF16 weights with pinned llama.cpp",
|
||||
"driver": {
|
||||
"type": "llama-cpp-server",
|
||||
"binary": "/run/media/popov/d/DEV/llamacpp/llama.cpp/build-hip/bin/llama-server",
|
||||
"binary_sha256": "b6bb4da687dbde86e243ba006cef05919b7b97255cd7e2371e1d451220aca139",
|
||||
"gguf_path": "/run/media/popov/DATA/llm/dgr-001/Qwen2.5-0.5B-Instruct-7ae5576-BF16.gguf",
|
||||
"device": "cuda",
|
||||
"threads": 16,
|
||||
"n_parallel": 4,
|
||||
"context_per_slot": 512,
|
||||
"n_gpu_layers": 99
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "transformers-bf16-rocm-throughput",
|
||||
"runtime": "transformers-5.13.0-rocm-bfloat16",
|
||||
"weight_format": "safetensors",
|
||||
"weight_quantization": "bfloat16",
|
||||
"lane": "performance-fit",
|
||||
"device": "cuda",
|
||||
"artifact_path": "/run/media/popov/DATA/llm/safetensor/models/models--Qwen--Qwen2.5-0.5B-Instruct/snapshots/7ae557604adf67be50417f59c2c2f167def9a775",
|
||||
"artifact_sha256": "e596e9d6205fdc9177569cccd7f8b471b058f66e3630c8e4326d5aad52bd18b6",
|
||||
"source_model_id": "Qwen/Qwen2.5-0.5B-Instruct",
|
||||
"source_model_revision": "7ae557604adf67be50417f59c2c2f167def9a775",
|
||||
"is_reference": false,
|
||||
"notes": "artifact_sha256 is the deterministic digest of every snapshot path and file byte",
|
||||
"driver": {
|
||||
"type": "transformers",
|
||||
"model_path": "/run/media/popov/DATA/llm/safetensor/models/models--Qwen--Qwen2.5-0.5B-Instruct/snapshots/7ae557604adf67be50417f59c2c2f167def9a775",
|
||||
"device": "cuda",
|
||||
"dtype": "bfloat16",
|
||||
"threads": 16
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "llama-cpp-q4-rocm-throughput",
|
||||
"runtime": "llama.cpp-9991-e920c523-rocm-gfx1151",
|
||||
"weight_format": "gguf",
|
||||
"weight_quantization": "Q4_K_M",
|
||||
"lane": "performance-fit",
|
||||
"device": "cuda",
|
||||
"artifact_path": "/run/media/popov/DATA/llm/dgr-001/Qwen2.5-0.5B-Instruct-7ae5576-Q4_K_M.gguf",
|
||||
"artifact_sha256": "a88e3f570e2efeaf06b50df9859db2c70d8646aa3a2c94a14e14d5797a2921a5",
|
||||
"source_model_id": "Qwen/Qwen2.5-0.5B-Instruct",
|
||||
"source_model_revision": "7ae557604adf67be50417f59c2c2f167def9a775",
|
||||
"is_reference": false,
|
||||
"notes": "Quantized from the exact-revision F16 GGUF with pinned llama-quantize",
|
||||
"driver": {
|
||||
"type": "llama-cpp-server",
|
||||
"binary": "/run/media/popov/d/DEV/llamacpp/llama.cpp/build-hip/bin/llama-server",
|
||||
"binary_sha256": "b6bb4da687dbde86e243ba006cef05919b7b97255cd7e2371e1d451220aca139",
|
||||
"gguf_path": "/run/media/popov/DATA/llm/dgr-001/Qwen2.5-0.5B-Instruct-7ae5576-Q4_K_M.gguf",
|
||||
"device": "cuda",
|
||||
"threads": 16,
|
||||
"n_parallel": 4,
|
||||
"context_per_slot": 512,
|
||||
"n_gpu_layers": 99
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,13 @@
|
||||
Recipe benchmark dgr-001-rocm-gpu-diagnostic-v1 (local-real)
|
||||
model Qwen/Qwen2.5-0.5B-Instruct@7ae557604adf67be50417f59c2c2f167def9a775
|
||||
transformers-fp32-rocm-quality-oracle [quality ] c= 1 ttft p50/p95 11.0/ 35.5 ms; prefill 5746.7 tok/s; decode 35.7 tok/s; aggregate 29.6 tok/s; rss 1.39 GB; vram 2.26 GB; artifact 1.00 GB; failures 0
|
||||
transformers-fp32-rocm-quality-oracle [quality ] c= 4 ttft p50/p95 27.5/ 80.4 ms; prefill 1985.4 tok/s; decode 9.4 tok/s; aggregate 35.4 tok/s; rss 1.39 GB; vram 2.74 GB; artifact 1.00 GB; failures 0
|
||||
llama-cpp-bf16-rocm-quality [quality ] c= 1 ttft p50/p95 13.2/ 83.4 ms; prefill 4154.4 tok/s; decode 148.0 tok/s; aggregate 127.4 tok/s; rss 0.84 GB; vram 0.00 GB; artifact 0.99 GB; failures 0
|
||||
llama-cpp-bf16-rocm-quality [quality ] c= 4 ttft p50/p95 25.1/ 52.1 ms; prefill 2205.4 tok/s; decode 115.1 tok/s; aggregate 337.1 tok/s; rss 0.86 GB; vram 0.00 GB; artifact 0.99 GB; failures 0
|
||||
transformers-bf16-rocm-throughput [performance-fit ] c= 1 ttft p50/p95 13.8/ 22.2 ms; prefill 4787.3 tok/s; decode 81.1 tok/s; aggregate 73.5 tok/s; rss 0.07 GB; vram 2.74 GB; artifact 1.00 GB; failures 0
|
||||
transformers-bf16-rocm-throughput [performance-fit ] c= 4 ttft p50/p95 29.7/ 58.5 ms; prefill 2666.5 tok/s; decode 24.4 tok/s; aggregate 91.2 tok/s; rss 0.07 GB; vram 2.74 GB; artifact 1.00 GB; failures 0
|
||||
llama-cpp-q4-rocm-throughput [performance-fit ] c= 1 ttft p50/p95 11.8/ 37.1 ms; prefill 4219.3 tok/s; decode 251.2 tok/s; aggregate 200.1 tok/s; rss 0.69 GB; vram 0.00 GB; artifact 0.40 GB; failures 0
|
||||
llama-cpp-q4-rocm-throughput [performance-fit ] c= 4 ttft p50/p95 21.4/ 101.0 ms; prefill 2126.9 tok/s; decode 189.7 tok/s; aggregate 511.3 tok/s; rss 0.72 GB; vram 0.00 GB; artifact 0.40 GB; failures 0
|
||||
drift llama-cpp-bf16-rocm-quality vs transformers-fp32-rocm-quality-oracle exact 1.00; similarity 1.000 (gated)
|
||||
drift transformers-bf16-rocm-throughput vs transformers-fp32-rocm-quality-oracle exact 0.33; similarity 0.946 (advisory)
|
||||
drift llama-cpp-q4-rocm-throughput vs transformers-fp32-rocm-quality-oracle exact 0.00; similarity 0.628 (advisory)
|
||||
@@ -1,7 +1,7 @@
|
||||
# Observed pre-existing intermittent tracker race
|
||||
|
||||
This file records an earlier unrelated timing observation; it is **not** the
|
||||
final DGR-001 verification result.
|
||||
This file records an unrelated timing observation and its repeated reproduction;
|
||||
it is **not** a DGR-001 benchmark/contract failure.
|
||||
|
||||
Test:
|
||||
|
||||
@@ -15,18 +15,41 @@ One earlier full-suite run produced:
|
||||
1 failed, 745 passed, 13 skipped
|
||||
```
|
||||
|
||||
A five-run isolated retry matrix reproduced the same rate on both branches:
|
||||
A five-run isolated retry matrix reproduced the same rate repeatedly:
|
||||
|
||||
```text
|
||||
DGR-001 branch: 4/5 passed, 1/5 failed
|
||||
clean d904c40: 4/5 passed, 1/5 failed
|
||||
current DGR-001 branch: 4/5 passed, 1/5 failed
|
||||
clean d904c40: 4/5 passed, 1/5 failed
|
||||
```
|
||||
|
||||
The final full-suite run on the exact hardened DGR-001 state completed green:
|
||||
An earlier full-suite run on the signed-provenance DGR-001 state completed
|
||||
green:
|
||||
|
||||
```text
|
||||
749 passed, 13 skipped in 251.42s
|
||||
751 passed, 13 skipped
|
||||
```
|
||||
|
||||
The earlier race was therefore timing-sensitive, pre-existing, and unrelated
|
||||
to the DGR-001 benchmark/contract files.
|
||||
Two full-suite runs after adding the isolated GPU diagnostic profile each hit
|
||||
the same race and otherwise passed:
|
||||
|
||||
```text
|
||||
1 failed, 750 passed, 13 skipped
|
||||
```
|
||||
|
||||
The latest expanded hardening suite hit the same race and otherwise passed:
|
||||
|
||||
```text
|
||||
1 failed, 754 passed, 13 skipped
|
||||
```
|
||||
|
||||
The final hardened state subsequently completed a full green run:
|
||||
|
||||
```text
|
||||
755 passed, 13 skipped
|
||||
```
|
||||
|
||||
In each failure, the mock upstream's three-second release timeout completed the
|
||||
stream before the cancel POST, so the request was already absent and the cancel
|
||||
endpoint returned 404. No tracker/proxy file changed in DGR-001. The race is
|
||||
therefore timing-sensitive, pre-existing, and unrelated to the benchmark,
|
||||
provenance, or GPU-diagnostic code.
|
||||
|
||||
@@ -0,0 +1,47 @@
|
||||
{
|
||||
"conclusion": {
|
||||
"conversion_corruption_observed_in_rocm_sample": false,
|
||||
"cpu_bf16_divergence_explained": false,
|
||||
"recommended_v2_design": "Predeclare a float32 quality oracle separately from the BF16 performance reference, with a larger prompt corpus and immutable thresholds.",
|
||||
"scope": "The ROCm diagnostic establishes only that the same BF16 GGUF artifact matched the float32 oracle for three GPU sequences; it does not explain the CPU BF16 divergence or prove global conversion correctness.",
|
||||
"v1_verdict_changed": false
|
||||
},
|
||||
"cpu_v1": {
|
||||
"candidate": "llama.cpp BF16 GGUF",
|
||||
"candidate_artifact_sha256": "e842fdc35d7f00fda95a54e1b51731ba1d196aea45065cc9f46925fdc1d6f862",
|
||||
"config_sha256": "00b2cce3e2f281bdf92fc5304ba5cac915a178ffccd3b9a25995ce39c00b90d3",
|
||||
"contract_verdict": "stop",
|
||||
"device": "cpu",
|
||||
"exact_match_rate": 0.3333,
|
||||
"mean_similarity": 0.9471,
|
||||
"plan_id": "dgr-001-controlled-whole-model-baseline-v1",
|
||||
"plan_sha256": "efe24690a9a7164bac6ab3fd0a6b22f078fc08aaefcfb96210ddf154e6050570",
|
||||
"quality_oracle": "Transformers BF16 safetensors",
|
||||
"report": "results.json",
|
||||
"report_sha256": "5d99a58806f39821c9206728047b8c5d605027d8a41b88639089b2418da890b5",
|
||||
"root_cause": "undetermined; no logit-tie claim is acceptance evidence",
|
||||
"run_id": "e4eedadf-22f6-4907-8990-985456961099"
|
||||
},
|
||||
"model_id": "Qwen/Qwen2.5-0.5B-Instruct",
|
||||
"model_revision": "7ae557604adf67be50417f59c2c2f167def9a775",
|
||||
"rocm_diagnostic": {
|
||||
"candidate": "llama.cpp BF16 GGUF",
|
||||
"candidate_artifact_sha256": "e842fdc35d7f00fda95a54e1b51731ba1d196aea45065cc9f46925fdc1d6f862",
|
||||
"config_sha256": "b0f0c846c818f1307d034cee1f81daa311efc20985c32a4cdbbbd8ffe4153892",
|
||||
"device": "cuda (ROCm)",
|
||||
"exact_match_rate": 1.0,
|
||||
"failures": 0,
|
||||
"mean_similarity": 1.0,
|
||||
"measured_backend_detail": "version: 9991 (e920c523) | built with GNU 15.2.1 for Linux x86_64; binary sha256 b6bb4da687dbde86e243ba006cef05919b7b97255cd7e2371e1d451220aca139; threads 16; parallel slots 4; ctx/slot 512; requested gpu layers 99; measured accelerator ROCm0: Radeon 8060S Graphics; measured offload 25/25 layers",
|
||||
"plan_id": "dgr-001-rocm-gpu-diagnostic-v1",
|
||||
"plan_sha256": "dae8e40963588f71f5d201fd163d39bd762e392544b5603d483e90d21abee2e8",
|
||||
"producer": "meshnet_node.recipe_drivers.run_configured_gpu_diagnostic/v1",
|
||||
"quality_oracle": "Transformers float32 safetensors",
|
||||
"report": "gpu-diagnostic-results.json",
|
||||
"report_sha256": "527b33d03627d57d60b30331e6b9119f579a828d6f6acb5c74ca25bab0af5f3d",
|
||||
"run_id": "31bf44e7-ccd4-4277-84ac-c775dee65411",
|
||||
"signer_fingerprint": "8baca8742d9b3ed0c3fc54929c23f75ec8c1c739900aaf5334780d598ffa84de",
|
||||
"v1_eligible": false
|
||||
},
|
||||
"schema_version": 2
|
||||
}
|
||||
@@ -0,0 +1,31 @@
|
||||
# DGR-001 quality-parity evidence summary
|
||||
|
||||
This summary is generated by `summarize-quality-parity.py` from signed reports.
|
||||
It contains no independent logit measurements or self-asserted verification flag.
|
||||
|
||||
| Source | Device | Quality oracle | BF16 GGUF candidate | Exact | Similarity | Status |
|
||||
|---|---|---|---|---:|---:|---|
|
||||
| CPU v1 (`e4eedadf-22f6-4907-8990-985456961099`) | CPU | Transformers BF16 | llama.cpp BF16 | 0.3333 | 0.9471 | immutable `stop` |
|
||||
| ROCm diagnostic (`31bf44e7-ccd4-4277-84ac-c775dee65411`) | ROCm0 / Radeon 8060S | Transformers float32 | llama.cpp BF16 | 1.0000 | 1.0000 | diagnostic only |
|
||||
|
||||
## Interpretation
|
||||
|
||||
The CPU and ROCm rows use different plans, devices, kernels, and quality oracles.
|
||||
The CPU BF16 divergence remains unexplained and v1 remains `stop`. The signed
|
||||
ROCm report establishes the narrower fact that the same BF16 GGUF artifact
|
||||
matched the float32 oracle for all three GPU sequences with zero failures.
|
||||
Its signed backend detail records `ROCm0: Radeon 8060S Graphics` and measured
|
||||
`25/25` layer offload.
|
||||
|
||||
No conversion corruption was observed in that three-sequence ROCm sample. This
|
||||
does not prove global conversion correctness and does not retroactively change
|
||||
or explain the CPU result. A future v2 should predeclare a float32 quality oracle
|
||||
separately from its BF16 performance reference and use a larger corpus.
|
||||
|
||||
## Reproduction and bindings
|
||||
|
||||
- CPU report SHA-256: `5d99a58806f39821c9206728047b8c5d605027d8a41b88639089b2418da890b5`
|
||||
- GPU report SHA-256: `527b33d03627d57d60b30331e6b9119f579a828d6f6acb5c74ca25bab0af5f3d`
|
||||
- BF16 GGUF SHA-256: `e842fdc35d7f00fda95a54e1b51731ba1d196aea45065cc9f46925fdc1d6f862`
|
||||
- Signer fingerprint: `8baca8742d9b3ed0c3fc54929c23f75ec8c1c739900aaf5334780d598ffa84de`
|
||||
- Exact verification command: see `commands.txt`.
|
||||
@@ -0,0 +1,261 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Build the DGR-001 parity summary from cryptographically verified reports."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PublicKey
|
||||
|
||||
from meshnet_node.performance_contract import (
|
||||
_canonical_sha256,
|
||||
evaluate_contract,
|
||||
load_contract,
|
||||
report_signing_payload,
|
||||
)
|
||||
|
||||
ROOT = Path(__file__).resolve().parent
|
||||
|
||||
|
||||
def _read(name: str) -> dict:
|
||||
return json.loads((ROOT / name).read_text(encoding="utf-8"))
|
||||
|
||||
|
||||
def _file_sha256(name: str) -> str:
|
||||
return hashlib.sha256((ROOT / name).read_bytes()).hexdigest()
|
||||
|
||||
|
||||
def _drift(report: dict, recipe_id: str) -> dict:
|
||||
return next(item for item in report["drift"] if item["recipe_id"] == recipe_id)
|
||||
|
||||
|
||||
def _recipe(report: dict, recipe_id: str) -> dict:
|
||||
return next(item for item in report["recipes"] if item["recipe"]["id"] == recipe_id)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
contract = load_contract(ROOT / "performance-contract.json")
|
||||
cpu_report = _read("results.json")
|
||||
gpu_config = _read("gpu-diagnostic-config.json")
|
||||
gpu_report = _read("gpu-diagnostic-results.json")
|
||||
|
||||
cpu_evaluation = evaluate_contract(contract, cpu_report)
|
||||
if cpu_evaluation.verdict != "stop":
|
||||
raise RuntimeError("immutable CPU v1 evidence no longer evaluates to stop")
|
||||
|
||||
public_key_bytes = base64.b64decode(contract.baseline["required_signer_public_key"])
|
||||
public_key = Ed25519PublicKey.from_public_bytes(public_key_bytes)
|
||||
public_key.verify(
|
||||
base64.b64decode(gpu_report["provenance"]["signature"]),
|
||||
report_signing_payload(gpu_report),
|
||||
)
|
||||
signer_fingerprint = hashlib.sha256(public_key_bytes).hexdigest()
|
||||
if gpu_report["provenance"]["signer_public_key_sha256"] != signer_fingerprint:
|
||||
raise RuntimeError("GPU report signer fingerprint does not match the contract trust key")
|
||||
if gpu_report["provenance"]["config_sha256"] != _canonical_sha256(gpu_config):
|
||||
raise RuntimeError("GPU report is not bound to gpu-diagnostic-config.json")
|
||||
|
||||
if gpu_report.get("schema_version") != 1 or gpu_report.get("evidence_class") != "local-real":
|
||||
raise RuntimeError("GPU report must be schema-v1 local-real evidence")
|
||||
expected_producer = "meshnet_node.recipe_drivers.run_configured_gpu_diagnostic/v1"
|
||||
if gpu_report["provenance"].get("producer") != expected_producer:
|
||||
raise RuntimeError("GPU report was not emitted by the canonical diagnostic producer")
|
||||
if gpu_report.get("reference_recipe_id") != "transformers-fp32-rocm-quality-oracle":
|
||||
raise RuntimeError("GPU report uses the wrong quality reference")
|
||||
if gpu_report.get("host", {}).get("benchmark_lane") != "rocm-gpu-diagnostic":
|
||||
raise RuntimeError("GPU report lacks the diagnostic host marker")
|
||||
|
||||
trusted = json.loads(
|
||||
(ROOT.parents[1] / "trusted-evidence-signers.json").read_text(encoding="utf-8")
|
||||
)
|
||||
if not any(
|
||||
signer.get("algorithm") == "ed25519"
|
||||
and signer.get("fingerprint_sha256") == signer_fingerprint
|
||||
and signer.get("status") == "active"
|
||||
for signer in trusted.get("signers", ())
|
||||
):
|
||||
raise RuntimeError("GPU signer is not active in the trusted-signers registry")
|
||||
|
||||
for field in ("model_id", "model_revision"):
|
||||
if gpu_report["plan"].get(field) != cpu_report["plan"].get(field):
|
||||
raise RuntimeError(f"CPU and GPU reports do not share {field}")
|
||||
if gpu_config["plan"].get(field) != gpu_report["plan"].get(field):
|
||||
raise RuntimeError(f"GPU config and report do not share {field}")
|
||||
|
||||
expected_recipes = {
|
||||
"transformers-fp32-rocm-quality-oracle": ("quality", "cuda"),
|
||||
"llama-cpp-bf16-rocm-quality": ("quality", "cuda"),
|
||||
"transformers-bf16-rocm-throughput": ("performance-fit", "cuda"),
|
||||
"llama-cpp-q4-rocm-throughput": ("performance-fit", "cuda"),
|
||||
}
|
||||
actual_recipes = {
|
||||
entry["recipe"]["id"]: (entry["recipe"]["lane"], entry["recipe"]["device"])
|
||||
for entry in gpu_report["recipes"]
|
||||
}
|
||||
if actual_recipes != expected_recipes:
|
||||
raise RuntimeError("GPU report recipe identities, lanes, or devices changed")
|
||||
|
||||
gpu_prompt_ids = {prompt["id"] for prompt in gpu_report["plan"]["prompts"]}
|
||||
levels = {int(level) for level in gpu_report["plan"]["concurrency_levels"]}
|
||||
repeats = int(gpu_report["plan"]["repeats"])
|
||||
expected_outcomes = len(gpu_prompt_ids) * repeats * sum(levels)
|
||||
for entry in gpu_report["recipes"]:
|
||||
recipe_id = entry["recipe"]["id"]
|
||||
if not entry.get("available") or len(entry.get("outcomes", ())) != expected_outcomes:
|
||||
raise RuntimeError(f"GPU recipe {recipe_id!r} lacks complete outcomes")
|
||||
if any(
|
||||
not outcome.get("ok")
|
||||
or outcome.get("recipe_id") != recipe_id
|
||||
or outcome.get("prompt_id") not in gpu_prompt_ids
|
||||
or int(outcome.get("concurrency", 0)) not in levels
|
||||
or not 0 <= int(outcome.get("repeat", -1)) < repeats
|
||||
for outcome in entry["outcomes"]
|
||||
):
|
||||
raise RuntimeError(f"GPU recipe {recipe_id!r} contains failed or invalid outcomes")
|
||||
if {int(level) for level in entry["concurrency"]} != levels:
|
||||
raise RuntimeError(f"GPU recipe {recipe_id!r} has wrong concurrency cells")
|
||||
for prompt_id in gpu_prompt_ids:
|
||||
for level in levels:
|
||||
for repeat in range(repeats):
|
||||
count = sum(
|
||||
outcome["prompt_id"] == prompt_id
|
||||
and int(outcome["concurrency"]) == level
|
||||
and int(outcome["repeat"]) == repeat
|
||||
for outcome in entry["outcomes"]
|
||||
)
|
||||
if count != level:
|
||||
raise RuntimeError(
|
||||
f"GPU recipe {recipe_id!r} lacks complete request coverage"
|
||||
)
|
||||
if any(
|
||||
int(cell.get("failures", -1)) != 0
|
||||
or int(cell.get("requests", -1))
|
||||
!= len(
|
||||
[
|
||||
outcome
|
||||
for outcome in entry["outcomes"]
|
||||
if int(outcome["concurrency"]) == int(level)
|
||||
]
|
||||
)
|
||||
for level, cell in entry["concurrency"].items()
|
||||
):
|
||||
raise RuntimeError(f"GPU recipe {recipe_id!r} aggregates do not match outcomes")
|
||||
|
||||
cpu_quality = _drift(cpu_report, "llama-cpp-near-lossless-quality")
|
||||
gpu_quality = _drift(gpu_report, "llama-cpp-bf16-rocm-quality")
|
||||
cpu_recipe = _recipe(cpu_report, "llama-cpp-near-lossless-quality")
|
||||
gpu_recipe = _recipe(gpu_report, "llama-cpp-bf16-rocm-quality")
|
||||
gpu_backend = gpu_recipe["load"]["backend_detail"]
|
||||
if "measured accelerator ROCm0: Radeon 8060S Graphics" not in gpu_backend:
|
||||
raise RuntimeError("GPU report lacks measured ROCm device evidence")
|
||||
if "measured offload 25/25 layers" not in gpu_backend:
|
||||
raise RuntimeError("GPU report lacks measured layer-offload evidence")
|
||||
if cpu_recipe["recipe"]["artifact_sha256"] != gpu_recipe["recipe"]["artifact_sha256"]:
|
||||
raise RuntimeError("CPU and GPU diagnostics use different BF16 GGUF artifacts")
|
||||
if gpu_quality.get("compared_prompts") != len(gpu_prompt_ids):
|
||||
raise RuntimeError("GPU quality drift lacks complete prompt coverage")
|
||||
if {item["prompt_id"] for item in gpu_quality.get("per_prompt", ())} != gpu_prompt_ids:
|
||||
raise RuntimeError("GPU quality drift prompt identities do not match the plan")
|
||||
|
||||
summary = {
|
||||
"schema_version": 2,
|
||||
"model_id": cpu_report["plan"]["model_id"],
|
||||
"model_revision": cpu_report["plan"]["model_revision"],
|
||||
"cpu_v1": {
|
||||
"report": "results.json",
|
||||
"report_sha256": _file_sha256("results.json"),
|
||||
"run_id": cpu_report["provenance"]["run_id"],
|
||||
"plan_id": cpu_report["plan"]["plan_id"],
|
||||
"plan_sha256": _canonical_sha256(cpu_report["plan"]),
|
||||
"config_sha256": cpu_report["provenance"]["config_sha256"],
|
||||
"device": "cpu",
|
||||
"quality_oracle": "Transformers BF16 safetensors",
|
||||
"candidate": "llama.cpp BF16 GGUF",
|
||||
"candidate_artifact_sha256": cpu_recipe["recipe"]["artifact_sha256"],
|
||||
"exact_match_rate": cpu_quality["exact_match_rate"],
|
||||
"mean_similarity": cpu_quality["mean_similarity"],
|
||||
"contract_verdict": cpu_evaluation.verdict,
|
||||
"root_cause": "undetermined; no logit-tie claim is acceptance evidence",
|
||||
},
|
||||
"rocm_diagnostic": {
|
||||
"report": "gpu-diagnostic-results.json",
|
||||
"report_sha256": _file_sha256("gpu-diagnostic-results.json"),
|
||||
"run_id": gpu_report["provenance"]["run_id"],
|
||||
"producer": gpu_report["provenance"]["producer"],
|
||||
"signer_fingerprint": signer_fingerprint,
|
||||
"plan_id": gpu_report["plan"]["plan_id"],
|
||||
"plan_sha256": _canonical_sha256(gpu_report["plan"]),
|
||||
"config_sha256": gpu_report["provenance"]["config_sha256"],
|
||||
"device": "cuda (ROCm)",
|
||||
"quality_oracle": "Transformers float32 safetensors",
|
||||
"candidate": "llama.cpp BF16 GGUF",
|
||||
"candidate_artifact_sha256": gpu_recipe["recipe"]["artifact_sha256"],
|
||||
"measured_backend_detail": gpu_backend,
|
||||
"exact_match_rate": gpu_quality["exact_match_rate"],
|
||||
"mean_similarity": gpu_quality["mean_similarity"],
|
||||
"failures": sum(
|
||||
metrics["failures"]
|
||||
for entry in gpu_report["recipes"]
|
||||
for metrics in entry["concurrency"].values()
|
||||
),
|
||||
"v1_eligible": False,
|
||||
},
|
||||
"conclusion": {
|
||||
"v1_verdict_changed": False,
|
||||
"cpu_bf16_divergence_explained": False,
|
||||
"conversion_corruption_observed_in_rocm_sample": False,
|
||||
"scope": (
|
||||
"The ROCm diagnostic establishes only that the same BF16 GGUF artifact "
|
||||
"matched the float32 oracle for three GPU sequences; it does not explain "
|
||||
"the CPU BF16 divergence or prove global conversion correctness."
|
||||
),
|
||||
"recommended_v2_design": (
|
||||
"Predeclare a float32 quality oracle separately from the BF16 performance "
|
||||
"reference, with a larger prompt corpus and immutable thresholds."
|
||||
),
|
||||
},
|
||||
}
|
||||
|
||||
(ROOT / "quality-parity-diagnosis.json").write_text(
|
||||
json.dumps(summary, indent=2, sort_keys=True) + "\n", encoding="utf-8"
|
||||
)
|
||||
md = f"""# DGR-001 quality-parity evidence summary
|
||||
|
||||
This summary is generated by `summarize-quality-parity.py` from signed reports.
|
||||
It contains no independent logit measurements or self-asserted verification flag.
|
||||
|
||||
| Source | Device | Quality oracle | BF16 GGUF candidate | Exact | Similarity | Status |
|
||||
|---|---|---|---|---:|---:|---|
|
||||
| CPU v1 (`{summary['cpu_v1']['run_id']}`) | CPU | Transformers BF16 | llama.cpp BF16 | {summary['cpu_v1']['exact_match_rate']:.4f} | {summary['cpu_v1']['mean_similarity']:.4f} | immutable `stop` |
|
||||
| ROCm diagnostic (`{summary['rocm_diagnostic']['run_id']}`) | ROCm0 / Radeon 8060S | Transformers float32 | llama.cpp BF16 | {summary['rocm_diagnostic']['exact_match_rate']:.4f} | {summary['rocm_diagnostic']['mean_similarity']:.4f} | diagnostic only |
|
||||
|
||||
## Interpretation
|
||||
|
||||
The CPU and ROCm rows use different plans, devices, kernels, and quality oracles.
|
||||
The CPU BF16 divergence remains unexplained and v1 remains `stop`. The signed
|
||||
ROCm report establishes the narrower fact that the same BF16 GGUF artifact
|
||||
matched the float32 oracle for all three GPU sequences with zero failures.
|
||||
Its signed backend detail records `ROCm0: Radeon 8060S Graphics` and measured
|
||||
`25/25` layer offload.
|
||||
|
||||
No conversion corruption was observed in that three-sequence ROCm sample. This
|
||||
does not prove global conversion correctness and does not retroactively change
|
||||
or explain the CPU result. A future v2 should predeclare a float32 quality oracle
|
||||
separately from its BF16 performance reference and use a larger corpus.
|
||||
|
||||
## Reproduction and bindings
|
||||
|
||||
- CPU report SHA-256: `{summary['cpu_v1']['report_sha256']}`
|
||||
- GPU report SHA-256: `{summary['rocm_diagnostic']['report_sha256']}`
|
||||
- BF16 GGUF SHA-256: `{summary['rocm_diagnostic']['candidate_artifact_sha256']}`
|
||||
- Signer fingerprint: `{signer_fingerprint}`
|
||||
- Exact verification command: see `commands.txt`.
|
||||
"""
|
||||
(ROOT / "quality-parity-diagnosis.md").write_text(md, encoding="utf-8")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"schema_version": 1,
|
||||
"signers": [
|
||||
{
|
||||
"algorithm": "ed25519",
|
||||
"fingerprint_sha256": "8baca8742d9b3ed0c3fc54929c23f75ec8c1c739900aaf5334780d598ffa84de",
|
||||
"scope": "DGR-001 local-real and gpu-diagnostic benchmark evidence",
|
||||
"status": "active"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -29,6 +29,7 @@ import base64
|
||||
import binascii
|
||||
import hashlib
|
||||
import json
|
||||
from collections import Counter
|
||||
from dataclasses import asdict, dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any, Mapping
|
||||
@@ -370,22 +371,101 @@ def _validate_report(contract: PerformanceContract, report: Mapping[str, Any]) -
|
||||
)
|
||||
return {key: sorted(values) for key, values in counts.items()}
|
||||
|
||||
reference_counts = token_counts(reference)
|
||||
def outcome_counts(entry: Mapping[str, Any]) -> Counter[tuple[str, int, int]]:
|
||||
return Counter(
|
||||
(
|
||||
str(outcome.get("prompt_id")),
|
||||
int(outcome.get("concurrency", 0)),
|
||||
int(outcome.get("repeat", -1)),
|
||||
)
|
||||
for outcome in entry.get("outcomes", ())
|
||||
)
|
||||
|
||||
prompt_ids = {str(prompt["id"]) for prompt in plan["prompts"]}
|
||||
repeats = int(plan["repeats"])
|
||||
for entry in recipes:
|
||||
if not entry.get("available"):
|
||||
continue
|
||||
recipe_id = str(entry["recipe"]["id"])
|
||||
cells = entry.get("concurrency", {})
|
||||
actual_levels = {int(level) for level in cells}
|
||||
if actual_levels != levels:
|
||||
raise PerformanceContractError(
|
||||
f"recipe {recipe_id!r} has unexpected concurrency cells"
|
||||
)
|
||||
for outcome in entry.get("outcomes", ()):
|
||||
try:
|
||||
outcome_recipe_id = str(outcome["recipe_id"])
|
||||
prompt_id = str(outcome["prompt_id"])
|
||||
level = int(outcome["concurrency"])
|
||||
repeat = int(outcome["repeat"])
|
||||
except (KeyError, TypeError, ValueError) as exc:
|
||||
raise PerformanceContractError(
|
||||
f"recipe {recipe_id!r} contains a malformed raw outcome"
|
||||
) from exc
|
||||
if outcome_recipe_id != recipe_id:
|
||||
raise PerformanceContractError(
|
||||
f"recipe {recipe_id!r} contains an outcome for {outcome_recipe_id!r}"
|
||||
)
|
||||
if prompt_id not in prompt_ids or level not in levels or not 0 <= repeat < repeats:
|
||||
raise PerformanceContractError(
|
||||
f"recipe {recipe_id!r} contains an out-of-domain raw outcome"
|
||||
)
|
||||
if not isinstance(outcome.get("ok"), bool):
|
||||
raise PerformanceContractError(
|
||||
f"recipe {recipe_id!r} contains an outcome without boolean ok status"
|
||||
)
|
||||
|
||||
coverage = outcome_counts(entry)
|
||||
for prompt_id in prompt_ids:
|
||||
for level in levels:
|
||||
for repeat in range(repeats):
|
||||
if coverage[(prompt_id, level, repeat)] != level:
|
||||
raise PerformanceContractError(
|
||||
f"recipe {recipe_id!r} lacks complete request coverage"
|
||||
)
|
||||
for level in levels:
|
||||
cell = cells.get(str(level), cells.get(level))
|
||||
raw = [
|
||||
outcome
|
||||
for outcome in entry.get("outcomes", ())
|
||||
if int(outcome["concurrency"]) == level
|
||||
]
|
||||
if int(cell.get("concurrency", -1)) != level:
|
||||
raise PerformanceContractError(
|
||||
f"recipe {recipe_id!r} cell identity does not match concurrency {level}"
|
||||
)
|
||||
if int(cell.get("requests", -1)) != len(raw):
|
||||
raise PerformanceContractError(
|
||||
f"recipe {recipe_id!r} aggregate requests do not match raw outcomes"
|
||||
)
|
||||
raw_failures = sum(not outcome["ok"] for outcome in raw)
|
||||
if int(cell.get("failures", -1)) != raw_failures:
|
||||
raise PerformanceContractError(
|
||||
f"recipe {recipe_id!r} aggregate failures do not match raw outcomes"
|
||||
)
|
||||
|
||||
reference_counts = token_counts(reference)
|
||||
for prompt_id in prompt_ids:
|
||||
for level in required_levels:
|
||||
for level in levels:
|
||||
for repeat in range(repeats):
|
||||
values = reference_counts.get((prompt_id, level, repeat), ())
|
||||
if len(values) != level:
|
||||
raise PerformanceContractError(
|
||||
"reference recipe lacks complete prompt/repeat/concurrency coverage"
|
||||
)
|
||||
if contract.thresholds.max_failure_rate != 0:
|
||||
raise PerformanceContractError(
|
||||
"nonzero failure tolerance requires an explicit failed-request token policy"
|
||||
)
|
||||
for entry in recipes:
|
||||
if entry.get("available") and token_counts(entry) != reference_counts:
|
||||
raise PerformanceContractError(
|
||||
f"recipe {entry['recipe']['id']!r} used different prompt/decode token counts"
|
||||
)
|
||||
if not entry.get("available"):
|
||||
continue
|
||||
for key, values in token_counts(entry).items():
|
||||
if Counter(values) - Counter(reference_counts.get(key, ())):
|
||||
raise PerformanceContractError(
|
||||
f"recipe {entry['recipe']['id']!r} used different prompt/decode token counts"
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
@@ -543,7 +623,12 @@ def _evaluate_recipe(
|
||||
requests = sum(cell["requests"] for cell in entry["concurrency"].values())
|
||||
failure_rate = _ratio(failures, requests) if requests else 0.0
|
||||
measurements["failure_rate"] = failure_rate
|
||||
if failure_rate > thresholds.max_failure_rate:
|
||||
failure_limit_exceeded = requests == 0 or (
|
||||
failures > 0
|
||||
if thresholds.max_failure_rate == 0
|
||||
else failures / requests > thresholds.max_failure_rate
|
||||
)
|
||||
if failure_limit_exceeded:
|
||||
reasons.append(f"failure rate {failure_rate:.2%} exceeds the contract limit")
|
||||
speed_benefit = False
|
||||
fit_benefit = False
|
||||
|
||||
@@ -660,13 +660,27 @@ def main(argv: list[str] | None = None) -> int:
|
||||
description="Run the controlled safetensors-versus-GGUF recipe benchmark"
|
||||
)
|
||||
parser.add_argument("--config", type=Path, required=True, help="benchmark configuration JSON")
|
||||
parser.add_argument(
|
||||
"--profile",
|
||||
choices=("contract-v1", "gpu-diagnostic"),
|
||||
default="contract-v1",
|
||||
help="validation and provenance profile (GPU diagnostics are not v1-eligible)",
|
||||
)
|
||||
parser.add_argument("--json-out", type=Path, help="write the JSON report to this path")
|
||||
parser.add_argument("--summary-out", type=Path, help="write the text summary to this path")
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
from .recipe_drivers import run_configured_benchmark # heavy runtimes: import on demand
|
||||
from .recipe_drivers import ( # heavy runtimes: import on demand
|
||||
run_configured_benchmark,
|
||||
run_configured_gpu_diagnostic,
|
||||
)
|
||||
|
||||
report = run_configured_benchmark(json.loads(args.config.read_text(encoding="utf-8")))
|
||||
runner = (
|
||||
run_configured_gpu_diagnostic
|
||||
if args.profile == "gpu-diagnostic"
|
||||
else run_configured_benchmark
|
||||
)
|
||||
report = runner(json.loads(args.config.read_text(encoding="utf-8")))
|
||||
summary = format_summary(report)
|
||||
if args.json_out:
|
||||
args.json_out.write_text(json.dumps(report, indent=2, sort_keys=True) + "\n", encoding="utf-8")
|
||||
|
||||
@@ -64,6 +64,9 @@ from .recipe_benchmark import (
|
||||
|
||||
REAL_INFERENCE_ENV = "MESHNET_ENABLE_REAL_INFERENCE_TESTS"
|
||||
EVIDENCE_SIGNING_KEY_ENV = "MESHNET_EVIDENCE_SIGNING_KEY"
|
||||
CONTRACT_V1_PROFILE = "contract-v1"
|
||||
GPU_DIAGNOSTIC_PROFILE = "gpu-diagnostic"
|
||||
GPU_DIAGNOSTIC_REPORT_PRODUCER = "meshnet_node.recipe_drivers.run_configured_gpu_diagnostic/v1"
|
||||
|
||||
|
||||
def real_inference_enabled() -> bool:
|
||||
@@ -214,8 +217,12 @@ def _host_manifest(config: Mapping[str, Any] | None = None) -> dict[str, Any]:
|
||||
return manifest
|
||||
|
||||
|
||||
def _validate_config(config: Mapping[str, Any]) -> None:
|
||||
def _validate_config(
|
||||
config: Mapping[str, Any], *, profile: str = CONTRACT_V1_PROFILE
|
||||
) -> None:
|
||||
"""Reject comparisons that mix models, artifacts, devices, or budgets."""
|
||||
if profile not in {CONTRACT_V1_PROFILE, GPU_DIAGNOSTIC_PROFILE}:
|
||||
raise BenchmarkError(f"unknown benchmark validation profile {profile!r}")
|
||||
try:
|
||||
plan = config["plan"]
|
||||
root = Path(config["artifact_storage_root"]).resolve(strict=True)
|
||||
@@ -267,7 +274,11 @@ def _validate_config(config: Mapping[str, Any]) -> None:
|
||||
|
||||
driver = spec.get("driver")
|
||||
if not isinstance(driver, Mapping):
|
||||
raise BenchmarkError("every recipe needs a driver object")
|
||||
raise BenchmarkError("every recipe must declare a driver block")
|
||||
if "artifact_sha256" in driver:
|
||||
raise BenchmarkError(
|
||||
"driver artifact_sha256 is forbidden; the validated recipe digest is authoritative"
|
||||
)
|
||||
kind = driver.get("type")
|
||||
if kind == "transformers":
|
||||
driver_artifact = Path(driver.get("model_path", "")).resolve(strict=True)
|
||||
@@ -283,18 +294,40 @@ def _validate_config(config: Mapping[str, Any]) -> None:
|
||||
raise BenchmarkError("llama.cpp binary SHA-256 mismatch")
|
||||
if int(driver.get("n_parallel", max_concurrency)) < max_concurrency:
|
||||
raise BenchmarkError("llama.cpp parallel slots must cover maximum concurrency")
|
||||
if driver.get("device", "cpu") != "cpu" or int(driver.get("n_gpu_layers", 0)) != 0:
|
||||
driver_device = driver.get("device", "cpu")
|
||||
gpu_layers = int(driver.get("n_gpu_layers", 0))
|
||||
if profile == CONTRACT_V1_PROFILE and (
|
||||
driver_device != "cpu" or gpu_layers != 0
|
||||
):
|
||||
raise BenchmarkError(
|
||||
"v1 benchmark supports CPU-only llama.cpp until process VRAM is measurable"
|
||||
)
|
||||
if profile == GPU_DIAGNOSTIC_PROFILE and (
|
||||
driver_device != "cuda" or gpu_layers <= 0
|
||||
):
|
||||
raise BenchmarkError(
|
||||
"GPU diagnostic requires CUDA/ROCm llama.cpp with positive n_gpu_layers"
|
||||
)
|
||||
else:
|
||||
raise BenchmarkError(f"unknown driver type {kind!r}")
|
||||
if driver_artifact != artifact:
|
||||
raise BenchmarkError("driver artifact path must match the hashed recipe artifact")
|
||||
if driver.get("device", "cpu") != spec.get("device"):
|
||||
raise BenchmarkError("recipe and driver must declare the same device")
|
||||
if profile == CONTRACT_V1_PROFILE and spec.get("device") != "cpu":
|
||||
raise BenchmarkError("contract-v1 requires every recipe to run on CPU")
|
||||
if profile == GPU_DIAGNOSTIC_PROFILE and spec.get("device") != "cuda":
|
||||
raise BenchmarkError("GPU diagnostic requires every recipe to declare device 'cuda'")
|
||||
thread_budgets.add(int(driver.get("threads", 8)))
|
||||
|
||||
host = config.get("host", {})
|
||||
if not isinstance(host, Mapping):
|
||||
raise BenchmarkError("benchmark host metadata must be an object")
|
||||
if profile == GPU_DIAGNOSTIC_PROFILE and host.get(
|
||||
"benchmark_lane"
|
||||
) != "rocm-gpu-diagnostic":
|
||||
raise BenchmarkError("GPU diagnostic requires the rocm-gpu-diagnostic host marker")
|
||||
|
||||
if len(thread_budgets) != 1:
|
||||
raise BenchmarkError("every recipe must use the same CPU thread budget")
|
||||
|
||||
@@ -313,11 +346,13 @@ class TransformersDriver:
|
||||
self,
|
||||
model_path: str,
|
||||
*,
|
||||
artifact_sha256: str | None = None,
|
||||
device: str = "cpu",
|
||||
dtype: str = "bfloat16",
|
||||
threads: int = 8,
|
||||
) -> None:
|
||||
self.model_path = Path(model_path)
|
||||
self.artifact_sha256 = artifact_sha256
|
||||
self.device = device
|
||||
self.dtype = dtype
|
||||
self.threads = threads
|
||||
@@ -327,6 +362,10 @@ class TransformersDriver:
|
||||
self._rss_baseline = 0
|
||||
|
||||
def load(self) -> LoadStats:
|
||||
if self.artifact_sha256 is not None:
|
||||
measured_artifact_sha256 = _artifact_sha256(self.model_path)
|
||||
if not hmac.compare_digest(self.artifact_sha256, measured_artifact_sha256):
|
||||
raise BenchmarkError("Transformers artifact changed after config validation")
|
||||
self._rss_baseline = _process_rss()
|
||||
import torch
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
@@ -445,6 +484,39 @@ def _free_port() -> int:
|
||||
return int(probe.getsockname()[1])
|
||||
|
||||
|
||||
def _gpu_offload_evidence(log_text: str, requested_layers: int) -> str:
|
||||
"""Extract measured ROCm device and layer placement from llama-server logs."""
|
||||
device_matches = re.findall(
|
||||
r"-\s+(ROCm\d+)\s+:\s+(.+?)\s+\(\d+\s+MiB", log_text
|
||||
)
|
||||
offload_matches = re.findall(
|
||||
r"offloaded\s+(\d+)/(\d+)\s+layers\s+to\s+GPU", log_text
|
||||
)
|
||||
if not device_matches:
|
||||
raise BenchmarkError("GPU diagnostic found no measured ROCm device in llama-server logs")
|
||||
if not offload_matches:
|
||||
raise BenchmarkError("GPU diagnostic found no measured layer offload in llama-server logs")
|
||||
|
||||
backend, device_name = device_matches[-1]
|
||||
offloaded, total = (int(value) for value in offload_matches[-1])
|
||||
required = min(requested_layers, total)
|
||||
if requested_layers <= 0 or offloaded < required:
|
||||
raise BenchmarkError(
|
||||
f"GPU diagnostic requested {requested_layers} layers but measured "
|
||||
f"only {offloaded}/{total} offloaded"
|
||||
)
|
||||
return (
|
||||
f"measured accelerator {backend}: {device_name}; "
|
||||
f"measured offload {offloaded}/{total} layers"
|
||||
)
|
||||
|
||||
|
||||
def _gpu_layer_config_detail(device: str, layers: int) -> str:
|
||||
# Immutable v1 pins the historical CPU wording exactly. GPU diagnostics use
|
||||
# the clearer requested/measured distinction without changing v1 identity.
|
||||
return f"gpu layers {layers}" if device == "cpu" else f"requested gpu layers {layers}"
|
||||
|
||||
|
||||
class LlamaCppServerDriver:
|
||||
"""The whole-model llama.cpp/GGUF recipe, driven through ``llama-server``.
|
||||
|
||||
@@ -462,6 +534,7 @@ class LlamaCppServerDriver:
|
||||
gguf_path: str,
|
||||
*,
|
||||
binary_sha256: str,
|
||||
artifact_sha256: str | None = None,
|
||||
device: str = "cpu",
|
||||
threads: int = 8,
|
||||
n_parallel: int = 4,
|
||||
@@ -472,6 +545,7 @@ class LlamaCppServerDriver:
|
||||
self.binary = Path(binary)
|
||||
self.binary_sha256 = binary_sha256
|
||||
self.gguf_path = Path(gguf_path)
|
||||
self.artifact_sha256 = artifact_sha256
|
||||
self.device = device
|
||||
self.threads = threads
|
||||
self.n_parallel = n_parallel
|
||||
@@ -486,21 +560,27 @@ class LlamaCppServerDriver:
|
||||
def _url(self) -> str:
|
||||
return f"http://127.0.0.1:{self._port}"
|
||||
|
||||
def _log_excerpt(self) -> str:
|
||||
def _log_text(self) -> str:
|
||||
if self._log is None:
|
||||
return ""
|
||||
try:
|
||||
self._log.flush()
|
||||
self._log.seek(0)
|
||||
return self._log.read()[-4096:].decode("utf-8", errors="replace").strip()
|
||||
size = min(os.fstat(self._log.fileno()).st_size, 8 * 1024 * 1024)
|
||||
return os.pread(self._log.fileno(), size, 0).decode("utf-8", errors="replace")
|
||||
except Exception:
|
||||
return ""
|
||||
|
||||
def _log_excerpt(self) -> str:
|
||||
return self._log_text()[-4096:].strip()
|
||||
|
||||
def load(self) -> LoadStats:
|
||||
if not self.binary.exists():
|
||||
raise BenchmarkError(f"llama-server binary not found at {self.binary}")
|
||||
if not self.gguf_path.exists():
|
||||
raise BenchmarkError(f"GGUF artifact not found at {self.gguf_path}")
|
||||
if self.artifact_sha256 is not None:
|
||||
measured_artifact_sha256 = _artifact_sha256(self.gguf_path)
|
||||
if not hmac.compare_digest(self.artifact_sha256, measured_artifact_sha256):
|
||||
raise BenchmarkError("GGUF artifact changed after config validation")
|
||||
measured_binary_sha256 = _artifact_sha256(self.binary)
|
||||
if not hmac.compare_digest(self.binary_sha256, measured_binary_sha256):
|
||||
raise BenchmarkError("llama-server binary changed after config validation")
|
||||
@@ -516,8 +596,12 @@ class LlamaCppServerDriver:
|
||||
)
|
||||
|
||||
self._port = _free_port()
|
||||
command = [
|
||||
str(self.binary),
|
||||
command = [str(self.binary)]
|
||||
if self.device == "cuda":
|
||||
# Debug verbosity is required to capture measured device placement
|
||||
# and the offloaded/total layer count in signed diagnostics.
|
||||
command.extend(["-lv", "5"])
|
||||
command.extend([
|
||||
"--model", str(self.gguf_path),
|
||||
"--host", "127.0.0.1",
|
||||
"--port", str(self._port),
|
||||
@@ -528,7 +612,7 @@ class LlamaCppServerDriver:
|
||||
"--ctx-size", str(self.context_per_slot * self.n_parallel),
|
||||
"--n-gpu-layers", str(self.n_gpu_layers),
|
||||
"--no-webui",
|
||||
]
|
||||
])
|
||||
started = time.monotonic()
|
||||
self._log = tempfile.TemporaryFile(mode="w+b")
|
||||
self._process = subprocess.Popen(
|
||||
@@ -536,6 +620,9 @@ class LlamaCppServerDriver:
|
||||
)
|
||||
self._await_health(started)
|
||||
load_ms = (time.monotonic() - started) * 1000
|
||||
gpu_evidence = ""
|
||||
if self.device == "cuda":
|
||||
gpu_evidence = _gpu_offload_evidence(self._log_text(), self.n_gpu_layers)
|
||||
|
||||
return LoadStats(
|
||||
artifact_bytes=self.gguf_path.stat().st_size,
|
||||
@@ -545,7 +632,9 @@ class LlamaCppServerDriver:
|
||||
backend_detail=(
|
||||
f"{version}; binary sha256 {measured_binary_sha256}; "
|
||||
f"threads {self.threads}; parallel slots {self.n_parallel}; "
|
||||
f"ctx/slot {self.context_per_slot}; gpu layers {self.n_gpu_layers}"
|
||||
f"ctx/slot {self.context_per_slot}; "
|
||||
f"{_gpu_layer_config_detail(self.device, self.n_gpu_layers)}"
|
||||
+ (f"; {gpu_evidence}" if gpu_evidence else "")
|
||||
),
|
||||
)
|
||||
|
||||
@@ -653,6 +742,7 @@ def build_driver(spec: Mapping[str, Any], plan: BenchmarkPlan) -> RecipeDriverBu
|
||||
"""Construct the driver named by a recipe's ``driver`` block."""
|
||||
driver_spec = dict(spec["driver"])
|
||||
kind = driver_spec.pop("type")
|
||||
driver_spec["artifact_sha256"] = spec.get("artifact_sha256")
|
||||
if kind == "transformers":
|
||||
return TransformersDriver(**driver_spec)
|
||||
if kind == "llama-cpp-server":
|
||||
@@ -696,14 +786,28 @@ def _recipe_from_config(spec: Mapping[str, Any]) -> RecipeSpec:
|
||||
|
||||
|
||||
def run_configured_benchmark(config: Mapping[str, Any]) -> dict:
|
||||
"""Run every recipe in ``config`` against one shared plan and return the report.
|
||||
"""Run contract-v1 evidence through the CPU-only, fail-closed profile."""
|
||||
return _run_profiled_benchmark(config, profile=CONTRACT_V1_PROFILE)
|
||||
|
||||
A recipe whose runtime cannot start is recorded as unavailable with the real
|
||||
reason rather than dropped: a report that silently omits the recipe that
|
||||
crashed would read as a clean result.
|
||||
"""
|
||||
|
||||
def run_configured_gpu_diagnostic(config: Mapping[str, Any]) -> dict:
|
||||
"""Run signed ROCm diagnostics that are intentionally ineligible for v1."""
|
||||
return _run_profiled_benchmark(config, profile=GPU_DIAGNOSTIC_PROFILE)
|
||||
|
||||
|
||||
def _producer_for_profile(profile: str) -> str:
|
||||
if profile == CONTRACT_V1_PROFILE:
|
||||
return REAL_REPORT_PRODUCER
|
||||
if profile == GPU_DIAGNOSTIC_PROFILE:
|
||||
return GPU_DIAGNOSTIC_REPORT_PRODUCER
|
||||
raise BenchmarkError(f"unknown benchmark validation profile {profile!r}")
|
||||
|
||||
|
||||
def _run_profiled_benchmark(config: Mapping[str, Any], *, profile: str) -> dict:
|
||||
"""Validate a closed profile and derive its producer before measurement."""
|
||||
require_real_inference()
|
||||
_validate_config(config)
|
||||
_validate_config(config, profile=profile)
|
||||
producer = _producer_for_profile(profile)
|
||||
evidence_class = config.get("evidence_class", "local-real")
|
||||
if evidence_class not in {"local-real", "multi-machine-real"}:
|
||||
raise BenchmarkError("canonical real runner cannot emit synthetic evidence")
|
||||
@@ -739,7 +843,7 @@ def run_configured_benchmark(config: Mapping[str, Any]) -> dict:
|
||||
evidence_class=evidence_class,
|
||||
provenance={
|
||||
"schema_version": PROVENANCE_SCHEMA_VERSION,
|
||||
"producer": REAL_REPORT_PRODUCER,
|
||||
"producer": producer,
|
||||
"run_id": run_id,
|
||||
"started_at": started_at,
|
||||
"completed_at": _utc_now(),
|
||||
|
||||
@@ -11,12 +11,16 @@ from __future__ import annotations
|
||||
import base64
|
||||
import copy
|
||||
import hashlib
|
||||
import json
|
||||
import time
|
||||
from dataclasses import replace
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from cryptography.hazmat.primitives import serialization
|
||||
from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey
|
||||
from meshnet_node import recipe_benchmark as recipe_benchmark_module
|
||||
from meshnet_node import recipe_drivers as recipe_drivers_module
|
||||
from meshnet_node.performance_contract import (
|
||||
PROVENANCE_SCHEMA_VERSION,
|
||||
REAL_REPORT_PRODUCER,
|
||||
@@ -30,8 +34,15 @@ from meshnet_node.performance_contract import (
|
||||
report_signing_payload,
|
||||
)
|
||||
from meshnet_node.recipe_drivers import (
|
||||
CONTRACT_V1_PROFILE,
|
||||
GPU_DIAGNOSTIC_PROFILE,
|
||||
GPU_DIAGNOSTIC_REPORT_PRODUCER,
|
||||
_artifact_sha256,
|
||||
_gpu_offload_evidence,
|
||||
_gpu_layer_config_detail,
|
||||
_producer_for_profile,
|
||||
_validate_config,
|
||||
build_driver,
|
||||
require_real_inference,
|
||||
)
|
||||
from meshnet_node.recipe_benchmark import (
|
||||
@@ -353,6 +364,19 @@ def test_contract_requires_a_quality_lane_then_allows_quantized_fit_benefit():
|
||||
recipe("gguf-q4", Lane.PERFORMANCE_FIT), plan()
|
||||
)
|
||||
report = build_report(plan(), [reference, quality, q4], host={}, evidence_class="synthetic")
|
||||
reference_entry = next(
|
||||
entry for entry in report["recipes"] if entry["recipe"]["id"] == "safetensors"
|
||||
)
|
||||
q4_entry = next(
|
||||
entry for entry in report["recipes"] if entry["recipe"]["id"] == "gguf-q4"
|
||||
)
|
||||
for level, q4_cell in q4_entry["concurrency"].items():
|
||||
reference_cell = reference_entry["concurrency"][level]
|
||||
q4_cell["decode_tokens_per_sec"] = reference_cell["decode_tokens_per_sec"] / 2
|
||||
q4_cell["aggregate_decode_tokens_per_sec"] = (
|
||||
reference_cell["aggregate_decode_tokens_per_sec"] / 2
|
||||
)
|
||||
q4_cell["ttft_p50_ms"] = reference_cell["ttft_p50_ms"] * 2
|
||||
contract = PerformanceContract(
|
||||
contract_version=1, locked_at="2026-07-13T00:00:00Z", locked_by="test",
|
||||
plan_id="test-plan", thresholds=ContractThresholds(), baseline={}, stop_condition="test",
|
||||
@@ -429,6 +453,24 @@ def test_config_rejects_an_artifact_digest_mismatch(tmp_path: Path):
|
||||
}
|
||||
_validate_config(config)
|
||||
|
||||
nested_digest = copy.deepcopy(config)
|
||||
nested_digest["recipes"][0]["driver"]["artifact_sha256"] = "f" * 64
|
||||
with pytest.raises(BenchmarkError, match="artifact_sha256 is forbidden"):
|
||||
_validate_config(nested_digest)
|
||||
driver = build_driver(nested_digest["recipes"][0], plan())
|
||||
assert driver.artifact_sha256 == config["recipes"][0]["artifact_sha256"]
|
||||
|
||||
cuda_transformers = copy.deepcopy(config)
|
||||
cuda_transformers["recipes"][0]["device"] = "cuda"
|
||||
cuda_transformers["recipes"][0]["driver"] = {
|
||||
"type": "transformers",
|
||||
"model_path": str(artifact),
|
||||
"device": "cuda",
|
||||
"threads": 8,
|
||||
}
|
||||
with pytest.raises(BenchmarkError, match="every recipe to run on CPU"):
|
||||
_validate_config(cuda_transformers)
|
||||
|
||||
config["recipes"][0]["artifact_sha256"] = "0" * 64
|
||||
with pytest.raises(BenchmarkError, match="digest mismatch"):
|
||||
_validate_config(config)
|
||||
@@ -443,6 +485,131 @@ def test_config_rejects_an_artifact_digest_mismatch(tmp_path: Path):
|
||||
with pytest.raises(BenchmarkError, match="CPU-only"):
|
||||
_validate_config(config)
|
||||
|
||||
config["recipes"][0]["device"] = "cuda"
|
||||
config["recipes"][0]["driver"]["device"] = "cuda"
|
||||
with pytest.raises(BenchmarkError, match="host marker"):
|
||||
_validate_config(config, profile=GPU_DIAGNOSTIC_PROFILE)
|
||||
|
||||
config["host"] = []
|
||||
with pytest.raises(BenchmarkError, match="host metadata must be an object"):
|
||||
_validate_config(config, profile=GPU_DIAGNOSTIC_PROFILE)
|
||||
|
||||
config["host"] = {"benchmark_lane": "rocm-gpu-diagnostic"}
|
||||
_validate_config(config, profile=GPU_DIAGNOSTIC_PROFILE)
|
||||
|
||||
config["recipes"][0]["driver"]["n_gpu_layers"] = 0
|
||||
with pytest.raises(BenchmarkError, match="positive n_gpu_layers"):
|
||||
_validate_config(config, profile=GPU_DIAGNOSTIC_PROFILE)
|
||||
|
||||
|
||||
def test_gpu_offload_evidence_requires_measured_rocm_placement():
|
||||
assert _gpu_layer_config_detail("cpu", 0) == "gpu layers 0"
|
||||
assert _gpu_layer_config_detail("cuda", 99) == "requested gpu layers 99"
|
||||
measured = """
|
||||
common_param: - ROCm0 : Radeon 8060S Graphics (63963 MiB, 33058 MiB free)
|
||||
load_tensors: offloaded 25/25 layers to GPU
|
||||
"""
|
||||
assert _gpu_offload_evidence(measured, 99) == (
|
||||
"measured accelerator ROCm0: Radeon 8060S Graphics; "
|
||||
"measured offload 25/25 layers"
|
||||
)
|
||||
|
||||
with pytest.raises(BenchmarkError, match="no measured ROCm device"):
|
||||
_gpu_offload_evidence("offloaded 25/25 layers to GPU", 99)
|
||||
with pytest.raises(BenchmarkError, match="no measured layer offload"):
|
||||
_gpu_offload_evidence(
|
||||
"- ROCm0 : Radeon 8060S Graphics (63963 MiB, 33058 MiB free)", 99
|
||||
)
|
||||
with pytest.raises(BenchmarkError, match="measured only 4/25"):
|
||||
_gpu_offload_evidence(measured.replace("25/25", "4/25"), 99)
|
||||
|
||||
|
||||
def test_profiles_derive_fixed_producers_and_cli_dispatch(monkeypatch, tmp_path: Path):
|
||||
assert _producer_for_profile(CONTRACT_V1_PROFILE) == REAL_REPORT_PRODUCER
|
||||
assert _producer_for_profile(GPU_DIAGNOSTIC_PROFILE) == GPU_DIAGNOSTIC_REPORT_PRODUCER
|
||||
with pytest.raises(BenchmarkError, match="unknown benchmark validation profile"):
|
||||
_producer_for_profile("caller-selected-producer")
|
||||
|
||||
profiled_calls = []
|
||||
|
||||
def profiled_runner(config, *, profile):
|
||||
profiled_calls.append((config, profile))
|
||||
return {"profile": profile}
|
||||
|
||||
monkeypatch.setattr(recipe_drivers_module, "_run_profiled_benchmark", profiled_runner)
|
||||
assert recipe_drivers_module.run_configured_benchmark({"source": "v1"}) == {
|
||||
"profile": CONTRACT_V1_PROFILE
|
||||
}
|
||||
assert recipe_drivers_module.run_configured_gpu_diagnostic({"source": "gpu"}) == {
|
||||
"profile": GPU_DIAGNOSTIC_PROFILE
|
||||
}
|
||||
assert profiled_calls == [
|
||||
({"source": "v1"}, CONTRACT_V1_PROFILE),
|
||||
({"source": "gpu"}, GPU_DIAGNOSTIC_PROFILE),
|
||||
]
|
||||
|
||||
config_path = tmp_path / "config.json"
|
||||
config_path.write_text(json.dumps({"test": "config"}), encoding="utf-8")
|
||||
calls = []
|
||||
|
||||
def contract_runner(config):
|
||||
calls.append((CONTRACT_V1_PROFILE, config))
|
||||
return {"profile": CONTRACT_V1_PROFILE}
|
||||
|
||||
def gpu_runner(config):
|
||||
calls.append((GPU_DIAGNOSTIC_PROFILE, config))
|
||||
return {"profile": GPU_DIAGNOSTIC_PROFILE}
|
||||
|
||||
monkeypatch.setattr(recipe_drivers_module, "run_configured_benchmark", contract_runner)
|
||||
monkeypatch.setattr(recipe_drivers_module, "run_configured_gpu_diagnostic", gpu_runner)
|
||||
monkeypatch.setattr(
|
||||
recipe_benchmark_module, "format_summary", lambda report: report["profile"]
|
||||
)
|
||||
|
||||
recipe_benchmark_module.main(["--config", str(config_path)])
|
||||
recipe_benchmark_module.main(
|
||||
["--profile", GPU_DIAGNOSTIC_PROFILE, "--config", str(config_path)]
|
||||
)
|
||||
assert calls == [
|
||||
(CONTRACT_V1_PROFILE, {"test": "config"}),
|
||||
(GPU_DIAGNOSTIC_PROFILE, {"test": "config"}),
|
||||
]
|
||||
|
||||
|
||||
def test_committed_config_digest_matches_immutable_contract():
|
||||
evidence = (
|
||||
Path(__file__).parents[1]
|
||||
/ ".scratch"
|
||||
/ "distributed-gguf-runtime"
|
||||
/ "evidence"
|
||||
/ "DGR-001"
|
||||
)
|
||||
contract = parse_contract(
|
||||
json.loads((evidence / "performance-contract.json").read_text())
|
||||
)
|
||||
config = json.loads((evidence / "benchmark-config.json").read_text())
|
||||
assert _canonical_sha256(config) == contract.baseline["required_config_sha256"]
|
||||
trusted = json.loads(
|
||||
(evidence.parents[1] / "trusted-evidence-signers.json").read_text()
|
||||
)
|
||||
public_key = base64.b64decode(contract.baseline["required_signer_public_key"])
|
||||
fingerprint = hashlib.sha256(public_key).hexdigest()
|
||||
assert any(
|
||||
signer["algorithm"] == "ed25519"
|
||||
and signer["fingerprint_sha256"] == fingerprint
|
||||
and signer["status"] == "active"
|
||||
for signer in trusted["signers"]
|
||||
)
|
||||
expected_cpu_gpu_detail = _gpu_layer_config_detail("cpu", 0)
|
||||
llama_backends = [
|
||||
detail
|
||||
for recipe_id, detail in contract.baseline["required_backend_detail"].items()
|
||||
if recipe_id.startswith("llama-cpp-")
|
||||
]
|
||||
assert llama_backends
|
||||
assert all(expected_cpu_gpu_detail in detail for detail in llama_backends)
|
||||
assert all("requested gpu layers" not in detail for detail in llama_backends)
|
||||
|
||||
|
||||
_TEST_SIGNING_KEY = Ed25519PrivateKey.generate()
|
||||
|
||||
@@ -559,6 +726,12 @@ def test_signed_real_report_rejects_tampering_and_rebinding():
|
||||
with pytest.raises(PerformanceContractError, match="signature"):
|
||||
evaluate_contract(contract, unsigned)
|
||||
|
||||
gpu_diagnostic = copy.deepcopy(report)
|
||||
gpu_diagnostic["provenance"]["producer"] = GPU_DIAGNOSTIC_REPORT_PRODUCER
|
||||
_resign_test_report(gpu_diagnostic)
|
||||
with pytest.raises(PerformanceContractError, match="canonical real runner"):
|
||||
evaluate_contract(contract, gpu_diagnostic)
|
||||
|
||||
tampered = copy.deepcopy(report)
|
||||
tampered["recipes"][0]["recipe"]["artifact_sha256"] = "b" * 64
|
||||
with pytest.raises(PerformanceContractError, match="signature verification failed"):
|
||||
@@ -625,6 +798,83 @@ def test_quality_lane_requires_every_prompt_to_be_compared():
|
||||
assert evaluation.verdict == "stop"
|
||||
|
||||
|
||||
def test_failed_request_reaches_zero_tolerance_gate_instead_of_validation_error():
|
||||
texts = {prompt.text: "same greedy answer" for prompt in PROMPTS}
|
||||
measurements = [
|
||||
measure_recipe(
|
||||
FakeDriver(texts=texts), recipe("ref", Lane.QUALITY, reference=True), plan()
|
||||
),
|
||||
measure_recipe(FakeDriver(texts=texts), recipe("quality", Lane.QUALITY), plan()),
|
||||
measure_recipe(FakeDriver(texts=texts), recipe("q4", Lane.PERFORMANCE_FIT), plan()),
|
||||
]
|
||||
report = build_report(plan(), measurements, host={}, evidence_class="synthetic")
|
||||
contract = _lock_real_report(report)
|
||||
failed = copy.deepcopy(report)
|
||||
quality_entry = next(
|
||||
entry for entry in failed["recipes"] if entry["recipe"]["id"] == "quality"
|
||||
)
|
||||
outcome = quality_entry["outcomes"][0]
|
||||
outcome["ok"] = False
|
||||
outcome["error"] = "simulated runtime failure"
|
||||
cell_key = next(
|
||||
key for key in quality_entry["concurrency"] if int(key) == outcome["concurrency"]
|
||||
)
|
||||
quality_entry["concurrency"][cell_key]["failures"] = 1
|
||||
_resign_test_report(failed)
|
||||
|
||||
evaluation = evaluate_contract(contract, failed)
|
||||
|
||||
quality_evaluation = next(
|
||||
item for item in evaluation.recipes if item.recipe_id == "quality"
|
||||
)
|
||||
assert quality_evaluation.failures == 1
|
||||
assert quality_evaluation.quality_pass is False
|
||||
assert evaluation.verdict == "stop"
|
||||
|
||||
permissive_contract = replace(
|
||||
contract,
|
||||
thresholds=replace(contract.thresholds, max_failure_rate=0.01),
|
||||
)
|
||||
with pytest.raises(PerformanceContractError, match="explicit failed-request token policy"):
|
||||
evaluate_contract(permissive_contract, failed)
|
||||
|
||||
inconsistent = copy.deepcopy(failed)
|
||||
quality_entry = next(
|
||||
entry for entry in inconsistent["recipes"] if entry["recipe"]["id"] == "quality"
|
||||
)
|
||||
quality_entry["concurrency"][cell_key]["failures"] = 0
|
||||
_resign_test_report(inconsistent)
|
||||
with pytest.raises(PerformanceContractError, match="failures do not match raw outcomes"):
|
||||
evaluate_contract(contract, inconsistent)
|
||||
|
||||
wrong_requests = copy.deepcopy(report)
|
||||
quality_entry = next(
|
||||
entry for entry in wrong_requests["recipes"] if entry["recipe"]["id"] == "quality"
|
||||
)
|
||||
quality_entry["concurrency"][cell_key]["requests"] += 1
|
||||
_resign_test_report(wrong_requests)
|
||||
with pytest.raises(PerformanceContractError, match="requests do not match raw outcomes"):
|
||||
evaluate_contract(contract, wrong_requests)
|
||||
|
||||
misidentified = copy.deepcopy(report)
|
||||
quality_entry = next(
|
||||
entry for entry in misidentified["recipes"] if entry["recipe"]["id"] == "quality"
|
||||
)
|
||||
quality_entry["outcomes"][0]["recipe_id"] = "other-recipe"
|
||||
_resign_test_report(misidentified)
|
||||
with pytest.raises(PerformanceContractError, match="contains an outcome for"):
|
||||
evaluate_contract(contract, misidentified)
|
||||
|
||||
missing = copy.deepcopy(failed)
|
||||
quality_entry = next(
|
||||
entry for entry in missing["recipes"] if entry["recipe"]["id"] == "quality"
|
||||
)
|
||||
quality_entry["outcomes"].pop()
|
||||
_resign_test_report(missing)
|
||||
with pytest.raises(PerformanceContractError, match="complete request coverage"):
|
||||
evaluate_contract(contract, missing)
|
||||
|
||||
|
||||
def test_locked_contract_rejects_changed_plan_and_token_counts():
|
||||
texts = {prompt.text: "same greedy answer" for prompt in PROMPTS}
|
||||
reference = measure_recipe(
|
||||
|
||||
Reference in New Issue
Block a user