popov/neuron-tai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: add signed ROCm diagnostic lane

Browse Source
This commit is contained in:
Dobromir Popov
2026-07-13 21:24:43 +03:00
parent b1c9deeb01
commit ef2a9e67e8
16 changed files with 4443 additions and 50 deletions
Download Patch File Download Diff File Expand all files Collapse all files

70
.scratch/distributed-gguf-runtime/evidence/DGR-001/README.md
View File

@@ -65,9 +65,11 @@ The near-lossless BF16 quality lane compared all three prompts but measured:
- mean text similarity: **0.9471** (v1 requires at least `0.97`).
Tokenization and stopping were controlled: every runtime saw the same prompt
token counts and reported 31 post-TTFT decode tokens. The mismatch is genuine
greedy runtime divergence on two prompts, not missing coverage or a text-length
artifact. Therefore `contract-evaluation.json` records:
token counts and reported 31 post-TTFT decode tokens. The v1 mismatch is a
real greedy-output divergence on two prompts, not missing coverage or a
text-length artifact. Its root cause remains undetermined; no post-contract
logit-tie claim is acceptance evidence. Therefore `contract-evaluation.json`
records:
```text
verdict: stop
@@ -79,6 +81,35 @@ stop_condition_met: true
Thresholds were not changed after observing these results.
## Post-contract parity and ROCm diagnostics
`summarize-quality-parity.py` verifies and separates two signed sources. The CPU
v1 row uses CPU kernels and a Transformers BF16 oracle; it remains at `0.3333`
exact match with an unexplained divergence. The ROCm row uses a different plan,
GPU kernels, and a Transformers float32 oracle. In that narrower diagnostic,
the same BF16 GGUF artifact matches all three 32-token sequences exactly (`1.0`
exact match and `1.0` similarity). No conversion corruption was observed in
that three-sequence ROCm sample; this does not prove global conversion
correctness or explain the CPU result.
A separate HIP build at commit `e920c523` was compiled for `gfx1151` and
measured `ROCm0: Radeon 8060S Graphics`; its `llama-server` SHA-256 is
`b6bb4da687dbde86e243ba006cef05919b7b97255cd7e2371e1d451220aca139`.
A signed `gpu-diagnostic` profile measured zero failures:
| GPU metric | Transformers BF16 ROCm | llama.cpp Q4 ROCm | Q4 ratio |
|---|---:|---:|---:|
| Decode tok/s, c=1 | 81.12 | 251.25 | **3.10×** |
| Aggregate decode tok/s, c=4 | 91.24 | 511.33 | **5.60×** |
| TTFT p50, c=1 | 13.77 ms | 11.80 ms | **0.857×** |
The GPU report is signed under the distinct
`run_configured_gpu_diagnostic/v1` producer. The v1 evaluator rejects that
producer even when its signature is valid. llama-server process VRAM remains
unmeasured, so this diagnostic cannot replace or satisfy the immutable v1
contract. Its signed backend detail records the measured `ROCm0: Radeon 8060S
Graphics` device and `25/25` offloaded layers.
## Implementation
- `recipe_benchmark.py` provides the runtime-neutral measurement core, true
@@ -86,8 +117,9 @@ Thresholds were not changed after observing these results.
aggregation, failures, and output drift.
- `recipe_drivers.py` provides opt-in Transformers and llama-server drivers,
mounted-drive confinement, exact artifact/runtime verification, equal
device/thread budgets, greedy-only validation, measured host provenance, and
a CPU-only v1 guard until process VRAM can be measured honestly.
device/thread budgets, greedy-only validation, measured host provenance, a
CPU-only v1 guard until process VRAM can be measured honestly, and a distinct
signed GPU diagnostic profile that the v1 evaluator cannot accept.
- Peak RSS is runtime-scoped: Transformers reports growth above its pre-runtime
Python baseline, while llama.cpp reports its isolated server process tree.
Both are sampled continuously during in-flight requests.
@@ -106,6 +138,8 @@ Thresholds were not changed after observing these results.
- Every non-synthetic report is Ed25519-signed over the complete canonical JSON,
including raw outcomes and metrics. The contract pins the public key and exact
config SHA-256; the private key remains outside Git at mode `0600`.
- The signer fingerprint is independently anchored outside this evidence
directory in `../../trusted-evidence-signers.json` and checked by tests.
- Quantized drift remains advisory. Only the near-lossless lane can satisfy the
quality gate, and only performance-fit recipes can earn speed/fit benefits.
@@ -117,24 +151,30 @@ Thresholds were not changed after observing these results.
- `results.txt` — human-readable benchmark summary
- `baseline.json` — distilled measurements for later comparison
- `contract-evaluation.json` — fail-closed v1 verdict
- `quality-parity-diagnosis.json` / `.md` — run/device-scoped signed-evidence summary
- `summarize-quality-parity.py` — verifies both evidence chains and regenerates it
- `gpu-diagnostic-config.json` — exact ROCm diagnostic artifacts and runtimes
- `gpu-diagnostic-results.json` / `.txt` — signed GPU outcomes and summary
- `commands.txt` — reproducible conversion, benchmark, evaluation, and test commands
- `BLOCKED.md` — downstream stop-condition handoff
- `known-unrelated-failure.md` — clean-base reproduction of the tracker race
- `../../trusted-evidence-signers.json` — repository-reviewed signer fingerprint
## Verification
```text
Targeted: 24 passed
Full suite: 751 passed, 13 skipped
Earlier cancellation retry matrix, DGR-001: 4/5 passed
Targeted: 28 passed (5/5 consecutive focused runs)
Latest full suite: 755 passed, 13 skipped
Earlier full suite: 751 passed, 13 skipped
Current cancellation retry matrix, DGR-001: 4/5 passed
Earlier cancellation retry matrix, clean d904c40: 4/5 passed
compileall: passed
git diff --check: passed
Evidence JSON parse/integrity checks: passed
```
An earlier intermittent tracker cancellation race reproduced at the same rate on
the clean base and is retained in `known-unrelated-failure.md`; the final suite
The intermittent tracker cancellation race reproduced at the same rate on the
clean base and is retained in `known-unrelated-failure.md`; the final full suite
completed green. DGR-001 changes no tracker/proxy files.
The earlier Ralph claim that the full suite was blocked by Protobuf 6.33.6 was
@@ -144,10 +184,12 @@ project `.venv`, which has the DGR-002-compatible runtime. Real inference used
## Limitations and dependent-story handoff
- This is a **0.5B CPU baseline**, not evidence for a large model, Radeon GPU,
distributed execution, network transport, or native shard worker.
- The installed llama.cpp build is CPU-only (`GGML_HIP=OFF`). No GPU comparison
is claimed.
- The immutable contract result is a **0.5B CPU baseline**. The separate Radeon
diagnostic is real local GPU evidence, but neither result covers a large
model, distributed execution, network transport, or a native shard worker.
- A separate `GGML_HIP=ON` llama.cpp build exists and produced GPU timings, but
llama-server process VRAM is not measurable by the current driver; GPU
memory/fit claims therefore remain ineligible for v1.
- Absolute timings are developer-machine measurements; locked ratios and raw
artifacts are provided for reproducibility.
- DGR-014 may consume v1 only with the exact plan/evidence requirements enforced

31
.scratch/distributed-gguf-runtime/evidence/DGR-001/commands.txt
View File

@@ -49,6 +49,37 @@ contract = load_contract(root / 'performance-contract.json')
(root / 'contract-evaluation.json').write_text(json.dumps(evaluate_contract(contract, report).to_dict(), indent=2, sort_keys=True) + '\n')
PY
# Optional ROCm GPU diagnostic (not eligible for immutable v1)
# The version-matched rocm[devel] wheel expands beyond 20 GB; ensure sufficient
# space or relocate its packaged payload before installation.
uv pip install --python "$ROCM_PY" --prerelease=allow \
--index-url https://rocm.nightlies.amd.com/v2/gfx1151/ \
'rocm[devel]==7.13.0a20260513'
ROCM_VENV=/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv-rocm
ROCM_SDK="$ROCM_VENV/bin/rocm-sdk"
ROCM_ROOT="$($ROCM_SDK path --root)"
ROCM_BIN="$($ROCM_SDK path --bin)"
export PATH="$ROCM_VENV/bin:$ROCM_BIN:$PATH"
export ROCM_PATH="$ROCM_ROOT" HIP_PATH="$ROCM_ROOT"
export CMAKE_PREFIX_PATH="$($ROCM_SDK path --cmake):$ROCM_ROOT"
export LD_LIBRARY_PATH="$ROCM_ROOT/lib:$ROCM_ROOT/lib64:${LD_LIBRARY_PATH:-}"
$ROCM_VENV/bin/cmake -S /run/media/popov/d/DEV/llamacpp/llama.cpp \
-B /run/media/popov/d/DEV/llamacpp/llama.cpp/build-hip -G Ninja \
-DGGML_HIP=ON -DGPU_TARGETS=gfx1151 \
-DCMAKE_HIP_COMPILER="$ROCM_VENV/bin/amdclang++" \
-DCMAKE_BUILD_TYPE=Release -DLLAMA_BUILD_TESTS=OFF \
-DLLAMA_BUILD_EXAMPLES=ON -DLLAMA_BUILD_SERVER=ON
$ROCM_VENV/bin/cmake --build /run/media/popov/d/DEV/llamacpp/llama.cpp/build-hip \
--target llama-server llama-cli llama-bench -j 16
MESHNET_ENABLE_REAL_INFERENCE_TESTS=1 MESHNET_EVIDENCE_SIGNING_KEY="$SIGNING_KEY" \
PYTHONPATH=packages/node $ROCM_PY -m meshnet_node.recipe_benchmark \
--profile gpu-diagnostic \
--config .scratch/distributed-gguf-runtime/evidence/DGR-001/gpu-diagnostic-config.json \
--json-out .scratch/distributed-gguf-runtime/evidence/DGR-001/gpu-diagnostic-results.json \
--summary-out .scratch/distributed-gguf-runtime/evidence/DGR-001/gpu-diagnostic-results.txt
PYTHONPATH=packages/node $PROJECT_PY \
.scratch/distributed-gguf-runtime/evidence/DGR-001/summarize-quality-parity.py
# Deterministic verification
PYTHONPATH=packages/node $PROJECT_PY -m pytest -q tests/test_recipe_benchmark.py
PYTHONPATH=packages/node $PROJECT_PY -m pytest -q

143
.scratch/distributed-gguf-runtime/evidence/DGR-001/gpu-diagnostic-config.json Normal file
View File

@@ -0,0 +1,143 @@
{
"artifact_storage_root": "/run/media/popov/DATA/llm",
"evidence_class": "local-real",
"host": {
"benchmark_lane": "rocm-gpu-diagnostic",
"llama_cpp_commit": "e920c523e3b8a0163fe498af5bf90df35ff51d25",
"llama_cpp_version": "9991",
"llama_server_sha256": "b6bb4da687dbde86e243ba006cef05919b7b97255cd7e2371e1d451220aca139",
"converter_sha256": "c819f18fb22927b49fabc3b35d1c9e21ee638b3817eccd1bd4efbcc7116eeb4d",
"quantizer_sha256": "bd0cc8c7be6d48aad4755b31062e0e59a887cbadd43dbb8771853d5858bb198f",
"transformers_version": "5.13.0",
"rocm_target": "gfx1151"
},
"plan": {
"plan_id": "dgr-001-rocm-gpu-diagnostic-v1",
"model_id": "Qwen/Qwen2.5-0.5B-Instruct",
"model_revision": "7ae557604adf67be50417f59c2c2f167def9a775",
"prompts": [
{
"id": "short-fact",
"text": "The capital of France is",
"context_class": "short"
},
{
"id": "medium-code",
"text": "Complete this Python function without commentary:\n\ndef fibonacci(n):\n \"\"\"Return the nth Fibonacci number for n >= 0.\"\"\"\n",
"context_class": "medium"
},
{
"id": "long-summary",
"text": "A distributed inference service divides a transformer across consumer machines. The tracker owns admission, routing, cancellation, accounting, and telemetry, while workers own only model execution. Every request carries an immutable model identity and revision. Workers must reject incompatible protocol versions and resource demands before allocating large buffers. Activation tensors are chunked, checksummed, bounded by negotiated limits, and propagated with explicit flow-control credits. A caller may disconnect at any time, so cancellation must release queued work, in-flight transfers, and cache reservations without double billing. Retries can occur after network failures, requiring idempotent request identifiers and deterministic completion accounting. The system keeps the existing safetensors path as a correctness reference while a native GGUF path is measured. Benchmarks compare the same prompts, output lengths, sampling policy, device, and concurrency, and they separate near-lossless quality checks from quantized speed and fit claims. Summarize the design priorities in three concise bullet points.",
"context_class": "long"
}
],
"sampling": {
"temperature": 0.0,
"top_p": 1.0,
"top_k": 1,
"seed": 1234,
"max_output_tokens": 32
},
"concurrency_levels": [
1,
4
],
"repeats": 3,
"warmup_requests": 2
},
"recipes": [
{
"id": "transformers-fp32-rocm-quality-oracle",
"runtime": "transformers-5.13.0-rocm-float32",
"weight_format": "safetensors",
"weight_quantization": "bfloat16-weights-float32-accumulation",
"lane": "quality",
"device": "cuda",
"artifact_path": "/run/media/popov/DATA/llm/safetensor/models/models--Qwen--Qwen2.5-0.5B-Instruct/snapshots/7ae557604adf67be50417f59c2c2f167def9a775",
"artifact_sha256": "e596e9d6205fdc9177569cccd7f8b471b058f66e3630c8e4326d5aad52bd18b6",
"source_model_id": "Qwen/Qwen2.5-0.5B-Instruct",
"source_model_revision": "7ae557604adf67be50417f59c2c2f167def9a775",
"is_reference": true,
"notes": "artifact_sha256 is the deterministic digest of every snapshot path and file byte",
"driver": {
"type": "transformers",
"model_path": "/run/media/popov/DATA/llm/safetensor/models/models--Qwen--Qwen2.5-0.5B-Instruct/snapshots/7ae557604adf67be50417f59c2c2f167def9a775",
"device": "cuda",
"dtype": "float32",
"threads": 16
}
},
{
"id": "llama-cpp-bf16-rocm-quality",
"runtime": "llama.cpp-9991-e920c523-rocm-gfx1151",
"weight_format": "gguf",
"weight_quantization": "bfloat16",
"lane": "quality",
"device": "cuda",
"artifact_path": "/run/media/popov/DATA/llm/dgr-001/Qwen2.5-0.5B-Instruct-7ae5576-BF16.gguf",
"artifact_sha256": "e842fdc35d7f00fda95a54e1b51731ba1d196aea45065cc9f46925fdc1d6f862",
"source_model_id": "Qwen/Qwen2.5-0.5B-Instruct",
"source_model_revision": "7ae557604adf67be50417f59c2c2f167def9a775",
"is_reference": false,
"notes": "Converted directly from the exact mounted safetensors revision while preserving BF16 weights with pinned llama.cpp",
"driver": {
"type": "llama-cpp-server",
"binary": "/run/media/popov/d/DEV/llamacpp/llama.cpp/build-hip/bin/llama-server",
"binary_sha256": "b6bb4da687dbde86e243ba006cef05919b7b97255cd7e2371e1d451220aca139",
"gguf_path": "/run/media/popov/DATA/llm/dgr-001/Qwen2.5-0.5B-Instruct-7ae5576-BF16.gguf",
"device": "cuda",
"threads": 16,
"n_parallel": 4,
"context_per_slot": 512,
"n_gpu_layers": 99
}
},
{
"id": "transformers-bf16-rocm-throughput",
"runtime": "transformers-5.13.0-rocm-bfloat16",
"weight_format": "safetensors",
"weight_quantization": "bfloat16",
"lane": "performance-fit",
"device": "cuda",
"artifact_path": "/run/media/popov/DATA/llm/safetensor/models/models--Qwen--Qwen2.5-0.5B-Instruct/snapshots/7ae557604adf67be50417f59c2c2f167def9a775",
"artifact_sha256": "e596e9d6205fdc9177569cccd7f8b471b058f66e3630c8e4326d5aad52bd18b6",
"source_model_id": "Qwen/Qwen2.5-0.5B-Instruct",
"source_model_revision": "7ae557604adf67be50417f59c2c2f167def9a775",
"is_reference": false,
"notes": "artifact_sha256 is the deterministic digest of every snapshot path and file byte",
"driver": {
"type": "transformers",
"model_path": "/run/media/popov/DATA/llm/safetensor/models/models--Qwen--Qwen2.5-0.5B-Instruct/snapshots/7ae557604adf67be50417f59c2c2f167def9a775",
"device": "cuda",
"dtype": "bfloat16",
"threads": 16
}
},
{
"id": "llama-cpp-q4-rocm-throughput",
"runtime": "llama.cpp-9991-e920c523-rocm-gfx1151",
"weight_format": "gguf",
"weight_quantization": "Q4_K_M",
"lane": "performance-fit",
"device": "cuda",
"artifact_path": "/run/media/popov/DATA/llm/dgr-001/Qwen2.5-0.5B-Instruct-7ae5576-Q4_K_M.gguf",
"artifact_sha256": "a88e3f570e2efeaf06b50df9859db2c70d8646aa3a2c94a14e14d5797a2921a5",
"source_model_id": "Qwen/Qwen2.5-0.5B-Instruct",
"source_model_revision": "7ae557604adf67be50417f59c2c2f167def9a775",
"is_reference": false,
"notes": "Quantized from the exact-revision F16 GGUF with pinned llama-quantize",
"driver": {
"type": "llama-cpp-server",
"binary": "/run/media/popov/d/DEV/llamacpp/llama.cpp/build-hip/bin/llama-server",
"binary_sha256": "b6bb4da687dbde86e243ba006cef05919b7b97255cd7e2371e1d451220aca139",
"gguf_path": "/run/media/popov/DATA/llm/dgr-001/Qwen2.5-0.5B-Instruct-7ae5576-Q4_K_M.gguf",
"device": "cuda",
"threads": 16,
"n_parallel": 4,
"context_per_slot": 512,
"n_gpu_layers": 99
}
}
]
}

3308
.scratch/distributed-gguf-runtime/evidence/DGR-001/gpu-diagnostic-results.json Normal file
View File

File diff suppressed because it is too large Load Diff

13
.scratch/distributed-gguf-runtime/evidence/DGR-001/gpu-diagnostic-results.txt Normal file
View File

@@ -0,0 +1,13 @@
Recipe benchmark dgr-001-rocm-gpu-diagnostic-v1 (local-real)
model Qwen/Qwen2.5-0.5B-Instruct@7ae557604adf67be50417f59c2c2f167def9a775
transformers-fp32-rocm-quality-oracle [quality ] c= 1 ttft p50/p95 11.0/ 35.5 ms; prefill 5746.7 tok/s; decode 35.7 tok/s; aggregate 29.6 tok/s; rss 1.39 GB; vram 2.26 GB; artifact 1.00 GB; failures 0
transformers-fp32-rocm-quality-oracle [quality ] c= 4 ttft p50/p95 27.5/ 80.4 ms; prefill 1985.4 tok/s; decode 9.4 tok/s; aggregate 35.4 tok/s; rss 1.39 GB; vram 2.74 GB; artifact 1.00 GB; failures 0
llama-cpp-bf16-rocm-quality [quality ] c= 1 ttft p50/p95 13.2/ 83.4 ms; prefill 4154.4 tok/s; decode 148.0 tok/s; aggregate 127.4 tok/s; rss 0.84 GB; vram 0.00 GB; artifact 0.99 GB; failures 0
llama-cpp-bf16-rocm-quality [quality ] c= 4 ttft p50/p95 25.1/ 52.1 ms; prefill 2205.4 tok/s; decode 115.1 tok/s; aggregate 337.1 tok/s; rss 0.86 GB; vram 0.00 GB; artifact 0.99 GB; failures 0
transformers-bf16-rocm-throughput [performance-fit ] c= 1 ttft p50/p95 13.8/ 22.2 ms; prefill 4787.3 tok/s; decode 81.1 tok/s; aggregate 73.5 tok/s; rss 0.07 GB; vram 2.74 GB; artifact 1.00 GB; failures 0
transformers-bf16-rocm-throughput [performance-fit ] c= 4 ttft p50/p95 29.7/ 58.5 ms; prefill 2666.5 tok/s; decode 24.4 tok/s; aggregate 91.2 tok/s; rss 0.07 GB; vram 2.74 GB; artifact 1.00 GB; failures 0
llama-cpp-q4-rocm-throughput [performance-fit ] c= 1 ttft p50/p95 11.8/ 37.1 ms; prefill 4219.3 tok/s; decode 251.2 tok/s; aggregate 200.1 tok/s; rss 0.69 GB; vram 0.00 GB; artifact 0.40 GB; failures 0
llama-cpp-q4-rocm-throughput [performance-fit ] c= 4 ttft p50/p95 21.4/ 101.0 ms; prefill 2126.9 tok/s; decode 189.7 tok/s; aggregate 511.3 tok/s; rss 0.72 GB; vram 0.00 GB; artifact 0.40 GB; failures 0
drift llama-cpp-bf16-rocm-quality vs transformers-fp32-rocm-quality-oracle exact 1.00; similarity 1.000 (gated)
drift transformers-bf16-rocm-throughput vs transformers-fp32-rocm-quality-oracle exact 0.33; similarity 0.946 (advisory)
drift llama-cpp-q4-rocm-throughput vs transformers-fp32-rocm-quality-oracle exact 0.00; similarity 0.628 (advisory)

41
.scratch/distributed-gguf-runtime/evidence/DGR-001/known-unrelated-failure.md
View File

@@ -1,7 +1,7 @@
# Observed pre-existing intermittent tracker race
This file records an earlier unrelated timing observation; it is **not** the
final DGR-001 verification result.
This file records an unrelated timing observation and its repeated reproduction;
it is **not** a DGR-001 benchmark/contract failure.
Test:
@@ -15,18 +15,41 @@ One earlier full-suite run produced:
1 failed, 745 passed, 13 skipped
```
A five-run isolated retry matrix reproduced the same rate on both branches:
A five-run isolated retry matrix reproduced the same rate repeatedly:
```text
DGR-001 branch: 4/5 passed, 1/5 failed
clean d904c40: 4/5 passed, 1/5 failed
current DGR-001 branch: 4/5 passed, 1/5 failed
clean d904c40: 4/5 passed, 1/5 failed
```
The final full-suite run on the exact hardened DGR-001 state completed green:
An earlier full-suite run on the signed-provenance DGR-001 state completed
green:
```text
749 passed, 13 skipped in 251.42s
751 passed, 13 skipped
```
The earlier race was therefore timing-sensitive, pre-existing, and unrelated
to the DGR-001 benchmark/contract files.
Two full-suite runs after adding the isolated GPU diagnostic profile each hit
the same race and otherwise passed:
```text
1 failed, 750 passed, 13 skipped
```
The latest expanded hardening suite hit the same race and otherwise passed:
```text
1 failed, 754 passed, 13 skipped
```
The final hardened state subsequently completed a full green run:
```text
755 passed, 13 skipped
```
In each failure, the mock upstream's three-second release timeout completed the
stream before the cancel POST, so the request was already absent and the cancel
endpoint returned 404. No tracker/proxy file changed in DGR-001. The race is
therefore timing-sensitive, pre-existing, and unrelated to the benchmark,
provenance, or GPU-diagnostic code.

47
.scratch/distributed-gguf-runtime/evidence/DGR-001/quality-parity-diagnosis.json Normal file
View File

@@ -0,0 +1,47 @@
{
"conclusion": {
"conversion_corruption_observed_in_rocm_sample": false,
"cpu_bf16_divergence_explained": false,
"recommended_v2_design": "Predeclare a float32 quality oracle separately from the BF16 performance reference, with a larger prompt corpus and immutable thresholds.",
"scope": "The ROCm diagnostic establishes only that the same BF16 GGUF artifact matched the float32 oracle for three GPU sequences; it does not explain the CPU BF16 divergence or prove global conversion correctness.",
"v1_verdict_changed": false
},
"cpu_v1": {
"candidate": "llama.cpp BF16 GGUF",
"candidate_artifact_sha256": "e842fdc35d7f00fda95a54e1b51731ba1d196aea45065cc9f46925fdc1d6f862",
"config_sha256": "00b2cce3e2f281bdf92fc5304ba5cac915a178ffccd3b9a25995ce39c00b90d3",
"contract_verdict": "stop",
"device": "cpu",
"exact_match_rate": 0.3333,
"mean_similarity": 0.9471,
"plan_id": "dgr-001-controlled-whole-model-baseline-v1",
"plan_sha256": "efe24690a9a7164bac6ab3fd0a6b22f078fc08aaefcfb96210ddf154e6050570",
"quality_oracle": "Transformers BF16 safetensors",
"report": "results.json",
"report_sha256": "5d99a58806f39821c9206728047b8c5d605027d8a41b88639089b2418da890b5",
"root_cause": "undetermined; no logit-tie claim is acceptance evidence",
"run_id": "e4eedadf-22f6-4907-8990-985456961099"
},
"model_id": "Qwen/Qwen2.5-0.5B-Instruct",
"model_revision": "7ae557604adf67be50417f59c2c2f167def9a775",
"rocm_diagnostic": {
"candidate": "llama.cpp BF16 GGUF",
"candidate_artifact_sha256": "e842fdc35d7f00fda95a54e1b51731ba1d196aea45065cc9f46925fdc1d6f862",
"config_sha256": "b0f0c846c818f1307d034cee1f81daa311efc20985c32a4cdbbbd8ffe4153892",
"device": "cuda (ROCm)",
"exact_match_rate": 1.0,
"failures": 0,
"mean_similarity": 1.0,
"measured_backend_detail": "version: 9991 (e920c523) | built with GNU 15.2.1 for Linux x86_64; binary sha256 b6bb4da687dbde86e243ba006cef05919b7b97255cd7e2371e1d451220aca139; threads 16; parallel slots 4; ctx/slot 512; requested gpu layers 99; measured accelerator ROCm0: Radeon 8060S Graphics; measured offload 25/25 layers",
"plan_id": "dgr-001-rocm-gpu-diagnostic-v1",
"plan_sha256": "dae8e40963588f71f5d201fd163d39bd762e392544b5603d483e90d21abee2e8",
"producer": "meshnet_node.recipe_drivers.run_configured_gpu_diagnostic/v1",
"quality_oracle": "Transformers float32 safetensors",
"report": "gpu-diagnostic-results.json",
"report_sha256": "527b33d03627d57d60b30331e6b9119f579a828d6f6acb5c74ca25bab0af5f3d",
"run_id": "31bf44e7-ccd4-4277-84ac-c775dee65411",
"signer_fingerprint": "8baca8742d9b3ed0c3fc54929c23f75ec8c1c739900aaf5334780d598ffa84de",
"v1_eligible": false
},
"schema_version": 2
}

31
.scratch/distributed-gguf-runtime/evidence/DGR-001/quality-parity-diagnosis.md Normal file
View File

@@ -0,0 +1,31 @@
# DGR-001 quality-parity evidence summary
This summary is generated by `summarize-quality-parity.py` from signed reports.
It contains no independent logit measurements or self-asserted verification flag.
| Source | Device | Quality oracle | BF16 GGUF candidate | Exact | Similarity | Status |
|---|---|---|---|---:|---:|---|
| CPU v1 (`e4eedadf-22f6-4907-8990-985456961099`) | CPU | Transformers BF16 | llama.cpp BF16 | 0.3333 | 0.9471 | immutable `stop` |
| ROCm diagnostic (`31bf44e7-ccd4-4277-84ac-c775dee65411`) | ROCm0 / Radeon 8060S | Transformers float32 | llama.cpp BF16 | 1.0000 | 1.0000 | diagnostic only |
## Interpretation
The CPU and ROCm rows use different plans, devices, kernels, and quality oracles.
The CPU BF16 divergence remains unexplained and v1 remains `stop`. The signed
ROCm report establishes the narrower fact that the same BF16 GGUF artifact
matched the float32 oracle for all three GPU sequences with zero failures.
Its signed backend detail records `ROCm0: Radeon 8060S Graphics` and measured
`25/25` layer offload.
No conversion corruption was observed in that three-sequence ROCm sample. This
does not prove global conversion correctness and does not retroactively change
or explain the CPU result. A future v2 should predeclare a float32 quality oracle
separately from its BF16 performance reference and use a larger corpus.
## Reproduction and bindings
- CPU report SHA-256: `5d99a58806f39821c9206728047b8c5d605027d8a41b88639089b2418da890b5`
- GPU report SHA-256: `527b33d03627d57d60b30331e6b9119f579a828d6f6acb5c74ca25bab0af5f3d`
- BF16 GGUF SHA-256: `e842fdc35d7f00fda95a54e1b51731ba1d196aea45065cc9f46925fdc1d6f862`
- Signer fingerprint: `8baca8742d9b3ed0c3fc54929c23f75ec8c1c739900aaf5334780d598ffa84de`
- Exact verification command: see `commands.txt`.

261
.scratch/distributed-gguf-runtime/evidence/DGR-001/summarize-quality-parity.py Normal file
View File

@@ -0,0 +1,261 @@
#!/usr/bin/env python3
"""Build the DGR-001 parity summary from cryptographically verified reports."""
from __future__ import annotations
import base64
import hashlib
import json
from pathlib import Path
from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PublicKey
from meshnet_node.performance_contract import (
_canonical_sha256,
evaluate_contract,
load_contract,
report_signing_payload,
)
ROOT = Path(__file__).resolve().parent
def _read(name: str) -> dict:
return json.loads((ROOT / name).read_text(encoding="utf-8"))
def _file_sha256(name: str) -> str:
return hashlib.sha256((ROOT / name).read_bytes()).hexdigest()
def _drift(report: dict, recipe_id: str) -> dict:
return next(item for item in report["drift"] if item["recipe_id"] == recipe_id)
def _recipe(report: dict, recipe_id: str) -> dict:
return next(item for item in report["recipes"] if item["recipe"]["id"] == recipe_id)
def main() -> None:
contract = load_contract(ROOT / "performance-contract.json")
cpu_report = _read("results.json")
gpu_config = _read("gpu-diagnostic-config.json")
gpu_report = _read("gpu-diagnostic-results.json")
cpu_evaluation = evaluate_contract(contract, cpu_report)
if cpu_evaluation.verdict != "stop":
raise RuntimeError("immutable CPU v1 evidence no longer evaluates to stop")
public_key_bytes = base64.b64decode(contract.baseline["required_signer_public_key"])
public_key = Ed25519PublicKey.from_public_bytes(public_key_bytes)
public_key.verify(
base64.b64decode(gpu_report["provenance"]["signature"]),
report_signing_payload(gpu_report),
)
signer_fingerprint = hashlib.sha256(public_key_bytes).hexdigest()
if gpu_report["provenance"]["signer_public_key_sha256"] != signer_fingerprint:
raise RuntimeError("GPU report signer fingerprint does not match the contract trust key")
if gpu_report["provenance"]["config_sha256"] != _canonical_sha256(gpu_config):
raise RuntimeError("GPU report is not bound to gpu-diagnostic-config.json")
if gpu_report.get("schema_version") != 1 or gpu_report.get("evidence_class") != "local-real":
raise RuntimeError("GPU report must be schema-v1 local-real evidence")
expected_producer = "meshnet_node.recipe_drivers.run_configured_gpu_diagnostic/v1"
if gpu_report["provenance"].get("producer") != expected_producer:
raise RuntimeError("GPU report was not emitted by the canonical diagnostic producer")
if gpu_report.get("reference_recipe_id") != "transformers-fp32-rocm-quality-oracle":
raise RuntimeError("GPU report uses the wrong quality reference")
if gpu_report.get("host", {}).get("benchmark_lane") != "rocm-gpu-diagnostic":
raise RuntimeError("GPU report lacks the diagnostic host marker")
trusted = json.loads(
(ROOT.parents[1] / "trusted-evidence-signers.json").read_text(encoding="utf-8")
)
if not any(
signer.get("algorithm") == "ed25519"
and signer.get("fingerprint_sha256") == signer_fingerprint
and signer.get("status") == "active"
for signer in trusted.get("signers", ())
):
raise RuntimeError("GPU signer is not active in the trusted-signers registry")
for field in ("model_id", "model_revision"):
if gpu_report["plan"].get(field) != cpu_report["plan"].get(field):
raise RuntimeError(f"CPU and GPU reports do not share {field}")
if gpu_config["plan"].get(field) != gpu_report["plan"].get(field):
raise RuntimeError(f"GPU config and report do not share {field}")
expected_recipes = {
"transformers-fp32-rocm-quality-oracle": ("quality", "cuda"),
"llama-cpp-bf16-rocm-quality": ("quality", "cuda"),
"transformers-bf16-rocm-throughput": ("performance-fit", "cuda"),
"llama-cpp-q4-rocm-throughput": ("performance-fit", "cuda"),
}
actual_recipes = {
entry["recipe"]["id"]: (entry["recipe"]["lane"], entry["recipe"]["device"])
for entry in gpu_report["recipes"]
}
if actual_recipes != expected_recipes:
raise RuntimeError("GPU report recipe identities, lanes, or devices changed")
gpu_prompt_ids = {prompt["id"] for prompt in gpu_report["plan"]["prompts"]}
levels = {int(level) for level in gpu_report["plan"]["concurrency_levels"]}
repeats = int(gpu_report["plan"]["repeats"])
expected_outcomes = len(gpu_prompt_ids) * repeats * sum(levels)
for entry in gpu_report["recipes"]:
recipe_id = entry["recipe"]["id"]
if not entry.get("available") or len(entry.get("outcomes", ())) != expected_outcomes:
raise RuntimeError(f"GPU recipe {recipe_id!r} lacks complete outcomes")
if any(
not outcome.get("ok")
or outcome.get("recipe_id") != recipe_id
or outcome.get("prompt_id") not in gpu_prompt_ids
or int(outcome.get("concurrency", 0)) not in levels
or not 0 <= int(outcome.get("repeat", -1)) < repeats
for outcome in entry["outcomes"]
):
raise RuntimeError(f"GPU recipe {recipe_id!r} contains failed or invalid outcomes")
if {int(level) for level in entry["concurrency"]} != levels:
raise RuntimeError(f"GPU recipe {recipe_id!r} has wrong concurrency cells")
for prompt_id in gpu_prompt_ids:
for level in levels:
for repeat in range(repeats):
count = sum(
outcome["prompt_id"] == prompt_id
and int(outcome["concurrency"]) == level
and int(outcome["repeat"]) == repeat
for outcome in entry["outcomes"]
)
if count != level:
raise RuntimeError(
f"GPU recipe {recipe_id!r} lacks complete request coverage"
)
if any(
int(cell.get("failures", -1)) != 0
or int(cell.get("requests", -1))
!= len(
[
outcome
for outcome in entry["outcomes"]
if int(outcome["concurrency"]) == int(level)
]
)
for level, cell in entry["concurrency"].items()
):
raise RuntimeError(f"GPU recipe {recipe_id!r} aggregates do not match outcomes")
cpu_quality = _drift(cpu_report, "llama-cpp-near-lossless-quality")
gpu_quality = _drift(gpu_report, "llama-cpp-bf16-rocm-quality")
cpu_recipe = _recipe(cpu_report, "llama-cpp-near-lossless-quality")
gpu_recipe = _recipe(gpu_report, "llama-cpp-bf16-rocm-quality")
gpu_backend = gpu_recipe["load"]["backend_detail"]
if "measured accelerator ROCm0: Radeon 8060S Graphics" not in gpu_backend:
raise RuntimeError("GPU report lacks measured ROCm device evidence")
if "measured offload 25/25 layers" not in gpu_backend:
raise RuntimeError("GPU report lacks measured layer-offload evidence")
if cpu_recipe["recipe"]["artifact_sha256"] != gpu_recipe["recipe"]["artifact_sha256"]:
raise RuntimeError("CPU and GPU diagnostics use different BF16 GGUF artifacts")
if gpu_quality.get("compared_prompts") != len(gpu_prompt_ids):
raise RuntimeError("GPU quality drift lacks complete prompt coverage")
if {item["prompt_id"] for item in gpu_quality.get("per_prompt", ())} != gpu_prompt_ids:
raise RuntimeError("GPU quality drift prompt identities do not match the plan")
summary = {
"schema_version": 2,
"model_id": cpu_report["plan"]["model_id"],
"model_revision": cpu_report["plan"]["model_revision"],
"cpu_v1": {
"report": "results.json",
"report_sha256": _file_sha256("results.json"),
"run_id": cpu_report["provenance"]["run_id"],
"plan_id": cpu_report["plan"]["plan_id"],
"plan_sha256": _canonical_sha256(cpu_report["plan"]),
"config_sha256": cpu_report["provenance"]["config_sha256"],
"device": "cpu",
"quality_oracle": "Transformers BF16 safetensors",
"candidate": "llama.cpp BF16 GGUF",
"candidate_artifact_sha256": cpu_recipe["recipe"]["artifact_sha256"],
"exact_match_rate": cpu_quality["exact_match_rate"],
"mean_similarity": cpu_quality["mean_similarity"],
"contract_verdict": cpu_evaluation.verdict,
"root_cause": "undetermined; no logit-tie claim is acceptance evidence",
},
"rocm_diagnostic": {
"report": "gpu-diagnostic-results.json",
"report_sha256": _file_sha256("gpu-diagnostic-results.json"),
"run_id": gpu_report["provenance"]["run_id"],
"producer": gpu_report["provenance"]["producer"],
"signer_fingerprint": signer_fingerprint,
"plan_id": gpu_report["plan"]["plan_id"],
"plan_sha256": _canonical_sha256(gpu_report["plan"]),
"config_sha256": gpu_report["provenance"]["config_sha256"],
"device": "cuda (ROCm)",
"quality_oracle": "Transformers float32 safetensors",
"candidate": "llama.cpp BF16 GGUF",
"candidate_artifact_sha256": gpu_recipe["recipe"]["artifact_sha256"],
"measured_backend_detail": gpu_backend,
"exact_match_rate": gpu_quality["exact_match_rate"],
"mean_similarity": gpu_quality["mean_similarity"],
"failures": sum(
metrics["failures"]
for entry in gpu_report["recipes"]
for metrics in entry["concurrency"].values()
),
"v1_eligible": False,
},
"conclusion": {
"v1_verdict_changed": False,
"cpu_bf16_divergence_explained": False,
"conversion_corruption_observed_in_rocm_sample": False,
"scope": (
"The ROCm diagnostic establishes only that the same BF16 GGUF artifact "
"matched the float32 oracle for three GPU sequences; it does not explain "
"the CPU BF16 divergence or prove global conversion correctness."
),
"recommended_v2_design": (
"Predeclare a float32 quality oracle separately from the BF16 performance "
"reference, with a larger prompt corpus and immutable thresholds."
),
},
}
(ROOT / "quality-parity-diagnosis.json").write_text(
json.dumps(summary, indent=2, sort_keys=True) + "\n", encoding="utf-8"
)
md = f"""# DGR-001 quality-parity evidence summary
This summary is generated by `summarize-quality-parity.py` from signed reports.
It contains no independent logit measurements or self-asserted verification flag.
| Source | Device | Quality oracle | BF16 GGUF candidate | Exact | Similarity | Status |
|---|---|---|---|---:|---:|---|
| CPU v1 (`{summary['cpu_v1']['run_id']}`) | CPU | Transformers BF16 | llama.cpp BF16 | {summary['cpu_v1']['exact_match_rate']:.4f} | {summary['cpu_v1']['mean_similarity']:.4f} | immutable `stop` |
| ROCm diagnostic (`{summary['rocm_diagnostic']['run_id']}`) | ROCm0 / Radeon 8060S | Transformers float32 | llama.cpp BF16 | {summary['rocm_diagnostic']['exact_match_rate']:.4f} | {summary['rocm_diagnostic']['mean_similarity']:.4f} | diagnostic only |
## Interpretation
The CPU and ROCm rows use different plans, devices, kernels, and quality oracles.
The CPU BF16 divergence remains unexplained and v1 remains `stop`. The signed
ROCm report establishes the narrower fact that the same BF16 GGUF artifact
matched the float32 oracle for all three GPU sequences with zero failures.
Its signed backend detail records `ROCm0: Radeon 8060S Graphics` and measured
`25/25` layer offload.
No conversion corruption was observed in that three-sequence ROCm sample. This
does not prove global conversion correctness and does not retroactively change
or explain the CPU result. A future v2 should predeclare a float32 quality oracle
separately from its BF16 performance reference and use a larger corpus.
## Reproduction and bindings
- CPU report SHA-256: `{summary['cpu_v1']['report_sha256']}`
- GPU report SHA-256: `{summary['rocm_diagnostic']['report_sha256']}`
- BF16 GGUF SHA-256: `{summary['rocm_diagnostic']['candidate_artifact_sha256']}`
- Signer fingerprint: `{signer_fingerprint}`
- Exact verification command: see `commands.txt`.
"""
(ROOT / "quality-parity-diagnosis.md").write_text(md, encoding="utf-8")
if __name__ == "__main__":
main()

11
.scratch/distributed-gguf-runtime/trusted-evidence-signers.json Normal file
View File

@@ -0,0 +1,11 @@
{
"schema_version": 1,
"signers": [
{
"algorithm": "ed25519",
"fingerprint_sha256": "8baca8742d9b3ed0c3fc54929c23f75ec8c1c739900aaf5334780d598ffa84de",
"scope": "DGR-001 local-real and gpu-diagnostic benchmark evidence",
"status": "active"
}
]
}