feat: add signed ROCm diagnostic lane
This commit is contained in:
@@ -65,9 +65,11 @@ The near-lossless BF16 quality lane compared all three prompts but measured:
|
||||
- mean text similarity: **0.9471** (v1 requires at least `0.97`).
|
||||
|
||||
Tokenization and stopping were controlled: every runtime saw the same prompt
|
||||
token counts and reported 31 post-TTFT decode tokens. The mismatch is genuine
|
||||
greedy runtime divergence on two prompts, not missing coverage or a text-length
|
||||
artifact. Therefore `contract-evaluation.json` records:
|
||||
token counts and reported 31 post-TTFT decode tokens. The v1 mismatch is a
|
||||
real greedy-output divergence on two prompts, not missing coverage or a
|
||||
text-length artifact. Its root cause remains undetermined; no post-contract
|
||||
logit-tie claim is acceptance evidence. Therefore `contract-evaluation.json`
|
||||
records:
|
||||
|
||||
```text
|
||||
verdict: stop
|
||||
@@ -79,6 +81,35 @@ stop_condition_met: true
|
||||
|
||||
Thresholds were not changed after observing these results.
|
||||
|
||||
## Post-contract parity and ROCm diagnostics
|
||||
|
||||
`summarize-quality-parity.py` verifies and separates two signed sources. The CPU
|
||||
v1 row uses CPU kernels and a Transformers BF16 oracle; it remains at `0.3333`
|
||||
exact match with an unexplained divergence. The ROCm row uses a different plan,
|
||||
GPU kernels, and a Transformers float32 oracle. In that narrower diagnostic,
|
||||
the same BF16 GGUF artifact matches all three 32-token sequences exactly (`1.0`
|
||||
exact match and `1.0` similarity). No conversion corruption was observed in
|
||||
that three-sequence ROCm sample; this does not prove global conversion
|
||||
correctness or explain the CPU result.
|
||||
|
||||
A separate HIP build at commit `e920c523` was compiled for `gfx1151` and
|
||||
measured `ROCm0: Radeon 8060S Graphics`; its `llama-server` SHA-256 is
|
||||
`b6bb4da687dbde86e243ba006cef05919b7b97255cd7e2371e1d451220aca139`.
|
||||
A signed `gpu-diagnostic` profile measured zero failures:
|
||||
|
||||
| GPU metric | Transformers BF16 ROCm | llama.cpp Q4 ROCm | Q4 ratio |
|
||||
|---|---:|---:|---:|
|
||||
| Decode tok/s, c=1 | 81.12 | 251.25 | **3.10×** |
|
||||
| Aggregate decode tok/s, c=4 | 91.24 | 511.33 | **5.60×** |
|
||||
| TTFT p50, c=1 | 13.77 ms | 11.80 ms | **0.857×** |
|
||||
|
||||
The GPU report is signed under the distinct
|
||||
`run_configured_gpu_diagnostic/v1` producer. The v1 evaluator rejects that
|
||||
producer even when its signature is valid. llama-server process VRAM remains
|
||||
unmeasured, so this diagnostic cannot replace or satisfy the immutable v1
|
||||
contract. Its signed backend detail records the measured `ROCm0: Radeon 8060S
|
||||
Graphics` device and `25/25` offloaded layers.
|
||||
|
||||
## Implementation
|
||||
|
||||
- `recipe_benchmark.py` provides the runtime-neutral measurement core, true
|
||||
@@ -86,8 +117,9 @@ Thresholds were not changed after observing these results.
|
||||
aggregation, failures, and output drift.
|
||||
- `recipe_drivers.py` provides opt-in Transformers and llama-server drivers,
|
||||
mounted-drive confinement, exact artifact/runtime verification, equal
|
||||
device/thread budgets, greedy-only validation, measured host provenance, and
|
||||
a CPU-only v1 guard until process VRAM can be measured honestly.
|
||||
device/thread budgets, greedy-only validation, measured host provenance, a
|
||||
CPU-only v1 guard until process VRAM can be measured honestly, and a distinct
|
||||
signed GPU diagnostic profile that the v1 evaluator cannot accept.
|
||||
- Peak RSS is runtime-scoped: Transformers reports growth above its pre-runtime
|
||||
Python baseline, while llama.cpp reports its isolated server process tree.
|
||||
Both are sampled continuously during in-flight requests.
|
||||
@@ -106,6 +138,8 @@ Thresholds were not changed after observing these results.
|
||||
- Every non-synthetic report is Ed25519-signed over the complete canonical JSON,
|
||||
including raw outcomes and metrics. The contract pins the public key and exact
|
||||
config SHA-256; the private key remains outside Git at mode `0600`.
|
||||
- The signer fingerprint is independently anchored outside this evidence
|
||||
directory in `../../trusted-evidence-signers.json` and checked by tests.
|
||||
- Quantized drift remains advisory. Only the near-lossless lane can satisfy the
|
||||
quality gate, and only performance-fit recipes can earn speed/fit benefits.
|
||||
|
||||
@@ -117,24 +151,30 @@ Thresholds were not changed after observing these results.
|
||||
- `results.txt` — human-readable benchmark summary
|
||||
- `baseline.json` — distilled measurements for later comparison
|
||||
- `contract-evaluation.json` — fail-closed v1 verdict
|
||||
- `quality-parity-diagnosis.json` / `.md` — run/device-scoped signed-evidence summary
|
||||
- `summarize-quality-parity.py` — verifies both evidence chains and regenerates it
|
||||
- `gpu-diagnostic-config.json` — exact ROCm diagnostic artifacts and runtimes
|
||||
- `gpu-diagnostic-results.json` / `.txt` — signed GPU outcomes and summary
|
||||
- `commands.txt` — reproducible conversion, benchmark, evaluation, and test commands
|
||||
- `BLOCKED.md` — downstream stop-condition handoff
|
||||
- `known-unrelated-failure.md` — clean-base reproduction of the tracker race
|
||||
- `../../trusted-evidence-signers.json` — repository-reviewed signer fingerprint
|
||||
|
||||
## Verification
|
||||
|
||||
```text
|
||||
Targeted: 24 passed
|
||||
Full suite: 751 passed, 13 skipped
|
||||
Earlier cancellation retry matrix, DGR-001: 4/5 passed
|
||||
Targeted: 28 passed (5/5 consecutive focused runs)
|
||||
Latest full suite: 755 passed, 13 skipped
|
||||
Earlier full suite: 751 passed, 13 skipped
|
||||
Current cancellation retry matrix, DGR-001: 4/5 passed
|
||||
Earlier cancellation retry matrix, clean d904c40: 4/5 passed
|
||||
compileall: passed
|
||||
git diff --check: passed
|
||||
Evidence JSON parse/integrity checks: passed
|
||||
```
|
||||
|
||||
An earlier intermittent tracker cancellation race reproduced at the same rate on
|
||||
the clean base and is retained in `known-unrelated-failure.md`; the final suite
|
||||
The intermittent tracker cancellation race reproduced at the same rate on the
|
||||
clean base and is retained in `known-unrelated-failure.md`; the final full suite
|
||||
completed green. DGR-001 changes no tracker/proxy files.
|
||||
|
||||
The earlier Ralph claim that the full suite was blocked by Protobuf 6.33.6 was
|
||||
@@ -144,10 +184,12 @@ project `.venv`, which has the DGR-002-compatible runtime. Real inference used
|
||||
|
||||
## Limitations and dependent-story handoff
|
||||
|
||||
- This is a **0.5B CPU baseline**, not evidence for a large model, Radeon GPU,
|
||||
distributed execution, network transport, or native shard worker.
|
||||
- The installed llama.cpp build is CPU-only (`GGML_HIP=OFF`). No GPU comparison
|
||||
is claimed.
|
||||
- The immutable contract result is a **0.5B CPU baseline**. The separate Radeon
|
||||
diagnostic is real local GPU evidence, but neither result covers a large
|
||||
model, distributed execution, network transport, or a native shard worker.
|
||||
- A separate `GGML_HIP=ON` llama.cpp build exists and produced GPU timings, but
|
||||
llama-server process VRAM is not measurable by the current driver; GPU
|
||||
memory/fit claims therefore remain ineligible for v1.
|
||||
- Absolute timings are developer-machine measurements; locked ratios and raw
|
||||
artifacts are provided for reproducibility.
|
||||
- DGR-014 may consume v1 only with the exact plan/evidence requirements enforced
|
||||
|
||||
Reference in New Issue
Block a user