# DGR-030 evidence — accelerator build presets and native CI/build matrix **Status:** implementation complete, live-verified in this session (2026-07-23). **Authority:** local `prd.json` is authoritative; Gitea is a projection. **Upstream pin:** `e920c523e3b8a0163fe498af5bf90df35ff51d25` (`llama.cpp`, unchanged from DGR-027..029). ## What existed before this session DGR-029 locked exactly one build lane — the deterministic CPU-only lane — in `UPSTREAM_LOCK.json`'s `build` section, plus `scripts/llama_cpp_dependency.py`'s `build()`/`smoke()`/`ctest_lane()`/`reproduce()`. There was no accelerator preset, no SDK-availability probing, and no matrix runner: only the one CPU lane existed, and there was no mechanism that could ever advertise a GPU backend as compiled or capable. ## What changed in this session - `packages/node/native/llama/UPSTREAM_LOCK.json`: added a new top-level `accelerator_presets` object with one entry each for `cuda` (`GGML_CUDA`), `rocm` (`GGML_HIP`), `vulkan` (`GGML_VULKAN`), and `metal` (`GGML_METAL`). Each entry names only the one backend flag it flips and an `sdk_probe` (a binary to resolve on `PATH`, an optional env-var override, and — for Metal — a `platform_only: "darwin"` gate). **The existing `build` section — the deterministic CPU default DGR-029 locked — is untouched.** - `scripts/llama_cpp_dependency.py`: - `_load_lock()` now calls a new `_verify_accelerator_presets()`, which fail-closed-rejects any preset whose named backend flag is not `OFF` in the CPU default's `configure_flags` — structurally guaranteeing a preset can only ever *add* one backend on top of the untouched CPU baseline, never redefine it. - `accelerator_configure_flags(lock, name)` returns a **new** flag list — the CPU default's own `configure_flags` list is never mutated — with exactly the named preset's backend flag flipped `ON` and every other flag (including `GGML_CPU=ON`, the fallback ops backend GPU builds still need) left exactly as the CPU default declares it. - `_sdk_probe(probe)` / `accelerator_status(name, lock)` resolve a lane's SDK without ever raising: an absent SDK is returned as `{"available": false, "reason": " is unavailable on PATH"}` (or a platform-mismatch reason for Metal), so "unavailable" is data a caller reports, never an exception a caller has to remember to catch. - `accelerator_build(source, name, build_dir)` compiles one lane into its own out-of-tree `build_dir` (an isolated directory, never DGR-029's CPU `build_dir`), using the same patched-source verification and `native_targets` as the CPU lane, then writes a `meshnet-build-metadata.json` recording the exact `commit`/`commit_tree`, per-patch SHA-256 digests, the lane's overridden `configure_flags`, the resolved `cmake`/`cxx`/SDK-binary versions/paths, and explicit `model_downloads: false`, `hardware_execution: false`, `hardware_certified: false`, `semantic_certification: false` fields plus a `note` stating the lane is registered-dark until a real-hardware certification record exists. It **never** calls `smoke()`/`ctest_lane()` — running a binary linked against a real accelerator backend would touch real hardware, which this story deliberately keeps out of scope. - Added `accelerator-status --name ` and `accelerator-build --name --source-dir --build-dir` CLI subcommands, mirroring the existing `ctest`/`build` subcommand pattern. - `scripts/native_accelerator_matrix.py` (new): the native CI/build matrix. `run_matrix(workspace)` fetches and applies the locked pin/patch stack once, runs the unchanged CPU lane (build → smoke → ctest, exactly DGR-029's contract), then for each `accelerator_presets` entry either reports `{"status": "skipped", "reason": ...}` (SDK absent) or compiles it via `accelerator_build` and reports `{"status": "built", ...}` — never silently treating a skip as a pass. Any `DependencyError` from a lane (CPU or accelerator) is caught per-lane and reported as `{"status": "failed", ...}` without aborting the remaining lanes or skipping cleanup. `reverse()` always runs in a `finally`, restoring the exact pristine pin/tree regardless of lane outcomes. The CLI prints a JSON report and exits non-zero only if any lane actually `failed` (a `skipped` lane never fails the run). - `tests/test_llama_cpp_dependency.py`: added 7 new tests — `test_accelerator_presets_isolate_one_backend_without_touching_the_cpu_default` (every preset flips exactly its own flag and the CPU default list is never mutated), `test_accelerator_configure_flags_rejects_an_unknown_lane`, `test_accelerator_status_reports_unavailable_sdks_without_raising` (asserts the exact reason string for cuda/rocm/vulkan/metal absence), `test_accelerator_status_honors_an_explicit_sdk_override`, `test_accelerator_status_rejects_an_unknown_lane`, `test_accelerator_build_refuses_to_compile_an_unavailable_lane` (asserts no build directory is created), and a `requires_cmake`-gated `test_accelerator_build_compiles_the_available_lane_with_isolated_evidence`, which builds a tiny synthetic CMake project (not the full llama.cpp tree) to prove `accelerator_build`'s "SDK present" path really configures with the overridden flag, compiles, and writes the registered-dark metadata — in about a second, without a real GPU SDK. - `tests/test_native_accelerator_matrix.py` (new): 3 offline tests exercising `run_matrix`'s orchestration with `llama_cpp_dependency`'s fetch/apply/reverse/build/smoke/ctest_lane/accelerator_status/ accelerator_build stubbed out — proving unavailable SDKs are reported `skipped` (never a false pass), an available accelerator lane is compiled without ever calling `smoke`/`ctest_lane`, and a lane failure is reported per-lane without aborting sibling lanes or skipping the `reverse()` cleanup. ## Toolchain note As in DGR-029, neither the ambient system Python nor `.venv-rocm` has `cmake`; this session's `.venv` also had no `cmake` (a prior session's install did not persist). This session ran `.venv/bin/python3 -m ensurepip --upgrade` (no `pip` was present in `.venv` either) and then `.venv/bin/python3 -m pip install cmake`, landing the same PyPI wheel (`cmake==4.4.0`) DGR-029 used, at `.venv/bin/cmake` / `.venv/bin/ctest`. All commands below were run with that `.venv/bin` prepended to `PATH`. No CUDA, ROCm, or Vulkan SDK (`nvcc`, `hipcc`, `glslc`) is installed in this environment, and the host platform is Linux, not `darwin` — so all four accelerator lanes are genuinely `skipped` in this environment's own live run below, which is real evidence for AC2 ("unavailable SDKs ... explicit unavailable/skipped lanes"), not a simulated one. ## Verification — live native CI/build matrix run ```text $ rm -rf build/llama.cpp/build build/llama.cpp/build-cuda build/llama.cpp/build-rocm build/llama.cpp/build-vulkan build/llama.cpp/build-metal $ python3 scripts/native_accelerator_matrix.py reused verified offline cache: .../build/llama.cpp/source usage: .../build/llama.cpp/build/bin/llama-gguf-hash [options] GGUF_IN ... Test project .../build/llama.cpp/build Start 27: test-meshnet-range-ownership 1/1 Test #27: test-meshnet-range-ownership ..... Passed 0.01 sec 100% tests passed out of 1 { "failed_lanes": [], "hardware_certified": false, "lanes": [ { "build_dir": ".../build/llama.cpp/build", "lane": "cpu", "metadata": { "cmake": "cmake version 4.4.0", "commit": "e920c523e3b8a0163fe498af5bf90df35ff51d25", "commit_tree": "6c91a11407a3a3fb160f5dac705f9c59718f54f1", "configure_flags": [ "-DCMAKE_BUILD_TYPE=Release", "-DLLAMA_BUILD_TESTS=ON", "-DLLAMA_BUILD_EXAMPLES=ON", "-DLLAMA_BUILD_SERVER=OFF", "-DLLAMA_BUILD_TOOLS=OFF", "-DLLAMA_BUILD_APP=OFF", "-DLLAMA_CURL=OFF", "-DGGML_CPU=ON", "-DGGML_BLAS=OFF", "-DGGML_CUDA=OFF", "-DGGML_HIP=OFF", "-DGGML_VULKAN=OFF", "-DGGML_METAL=OFF" ], "cxx": "c++ (GCC) 15.2.1 20260123 (Red Hat 15.2.1-7)", "model_downloads": false, "patches": { "...": "... (5 entries, unchanged sha256 digests from DGR-029)" }, "semantic_certification": false }, "status": "built" }, {"lane": "cuda", "reason": "nvcc is unavailable on PATH", "status": "skipped"}, {"lane": "rocm", "reason": "hipcc is unavailable on PATH", "status": "skipped"}, {"lane": "vulkan", "reason": "glslc is unavailable on PATH", "status": "skipped"}, {"lane": "metal", "reason": "platform 'linux' is not 'darwin'", "status": "skipped"} ], "note": "A `built` lane means it compiled with the exact recorded compiler/SDK/upstream-pin/patch-stack/build-option evidence — it never means an accelerator device was exercised. Every backend/model/recipe lane stays registered-dark until a separate real-hardware certification record exists." } $ echo $? 0 ``` Wall-clock: `real 2m19.797s` — matches DGR-029's ~2m16s CPU-lane compile; no accelerator lane actually compiled in this environment (all four SDKs are genuinely absent), so this run's added cost over DGR-029's own CPU-only `reproduce()` is just the four fast SDK probes. Post-run checks (source checkout left pristine by the matrix's `reverse()`): ```text $ git -C build/llama.cpp/source status --short --branch --untracked-files=all ## HEAD (no branch) $ git -C build/llama.cpp/source rev-parse HEAD HEAD^{tree} e920c523e3b8a0163fe498af5bf90df35ff51d25 6c91a11407a3a3fb160f5dac705f9c59718f54f1 $ ls build/llama.cpp/ | grep build build ``` Only the CPU lane's `build/` directory was created — no `build-cuda`, `build-rocm`, `build-vulkan`, or `build-metal` directory exists, because every accelerator lane was genuinely skipped rather than attempted. ## Verification — targeted test suites and shared gates | Command | Result | | --- | --- | | `python3 -m pytest -q tests/test_llama_cpp_dependency.py tests/test_native_accelerator_matrix.py` | `19 passed` (9 pre-existing + 7 new accelerator-lane tests in `test_llama_cpp_dependency.py`, 3 new in `test_native_accelerator_matrix.py`; the `requires_cmake`-gated compile test ran for real, not skipped) | | `python3 -m compileall -q packages tests` | exit 0 | | `git diff --check -- packages/node/native/llama/UPSTREAM_LOCK.json scripts/llama_cpp_dependency.py tests/test_llama_cpp_dependency.py scripts/native_accelerator_matrix.py tests/test_native_accelerator_matrix.py` | exit 0 | | `python3 scripts/ralph_prd_schema.py validate .scratch/distributed-gguf-runtime/prd.json` | `OK: 55 stories validated.` | `git diff --check` against the full working tree separately reports one pre-existing trailing-whitespace line in `.ralph-tui-run.log`, which was already modified before this session started (see the session's initial `git status`) and is unrelated to this story's scope; it is excluded above by naming this story's own changed files explicitly. `python3 -m pytest -q tests/test_ralph_prd_schema.py` reports `55 failed, 53 passed` in this session (all `test_render_issue_markdown_matches_committed_file` drift between `prd.json` and committed issue Markdown for other stories, e.g. `DGR-053`..`DGR-071`). `git stash`-ing this session's changes and rerunning reproduces `56 failed, 52 passed` identically — the same 56 failures minus the one this session's own `DGR-030` regeneration fixed, confirming the remaining 55 predate this story and are out of scope to fix here. This session did regenerate `.scratch/distributed-gguf-runtime/issues/030-add-accelerator- build-presets-and-native-ci-matrix.md` via `python3 scripts/ralph_prd_schema.py render ... DGR-030` so DGR-030's own generated issue Markdown matches `prd.json` byte-for-byte (confirmed by the `test_render_issue_markdown_matches_committed_file[DGR-030]` case no longer appearing in the failure list). ## Ensuring build success does not advertise capability - Every accelerator lane's `meshnet-build-metadata.json` explicitly records `hardware_execution: false`, `hardware_certified: false`, and `semantic_certification: false`, plus a `note` stating the lane is registered-dark until a separate real-hardware certification record exists — the same "artifact states this, not just prose" pattern DGR-029 used for the CPU lane's `model_downloads`/`semantic_certification` fields. - `accelerator_build` never runs `smoke()` or `ctest_lane()`: it only configures and compiles the exact `native_targets` DGR-029 already locked (`llama-gguf-hash`, `test-meshnet-range-ownership`) — no binary linked against a real accelerator backend is ever executed by this story's code. - `_verify_accelerator_presets()` structurally refuses any preset whose backend flag is not `OFF` in the locked CPU default, so a preset can never be defined in a way that redefines (rather than adds one backend on top of) DGR-029's deterministic CPU lane. - The matrix's top-level report always carries `"hardware_certified": false` regardless of how many lanes built, and its `note` field states this explicitly for any consumer reading only the report, not the per-lane metadata. ## Limitations - This story proves accelerator lanes *compile* with correct, isolated flags and preserves exact evidence when a lane's SDK is present. It proves nothing about numerical correctness, performance, or any backend/model/ recipe capability on real accelerator hardware — that is explicitly deferred to DGR-041 (capability registration), DGR-053 (real 2-4 stage certification), and DGR-067 (capability matrix certification), all of which remain unimplemented. - No CUDA, ROCm, or Vulkan SDK, and no macOS/Metal toolchain, is available in this session's environment, so the "compile an available accelerator lane" path is proven end-to-end only via the `requires_cmake`-gated synthetic- project unit test and the offline matrix-orchestration tests, not via a live compile of the real llama.cpp tree under `GGML_CUDA=ON` (etc.). A future session with a real SDK installed will exercise `accelerator_build`'s real-lane path against the genuine llama.cpp source for the first time; nothing in this story's design assumes that hasn't happened yet. - The accelerator lanes reuse the CPU lane's exact `native_targets` (`llama-gguf-hash`, `test-meshnet-range-ownership`), so a passing accelerator compile also proves the DGR-027/DGR-028 patch stack's range-ownership code compiles under that backend flag combination — but, per the point above, only structurally; it says nothing about GPU execution correctness. - `cmake`/`ctest` remain absent system-wide in this environment; this session reinstalled them into `.venv` exactly as DGR-029 did, and that install does not appear to persist across sessions (this session found `.venv` without `cmake` despite DGR-029's evidence recording its earlier install). A future session without a `cmake`-equipped `.venv` will see the same actionable "cmake is unavailable" failure DGR-029 demonstrated, not a silent pass, and the new `requires_cmake`-gated tests will be skipped rather than failing. - `git diff --check` and `tests/test_ralph_prd_schema.py` both carry pre-existing, out-of-scope failures unrelated to this story (see the gates table above); this story's own changed files pass both checks cleanly. ## Dependency handoff DGR-053 (real 2-4 stage certification), DGR-067 (capability matrix certification), and DGR-068 (packaged releases) may rely on: four isolated, out-of-tree accelerator build presets (`cuda`/`rocm`/`vulkan`/`metal`) in `UPSTREAM_LOCK.json`'s `accelerator_presets`, each toggling exactly one backend flag on top of DGR-029's unchanged CPU default; a native CI/build matrix (`scripts/native_accelerator_matrix.py`) that compiles every SDK-available lane with full compiler/SDK/upstream-pin/patch-stack/build- option evidence and reports SDK-unavailable lanes as explicit `skipped` lanes, never a false pass; and a compile-only contract (no lane here ever runs a binary against real accelerator hardware). Real-hardware execution, numerical correctness, performance measurement, and backend/model/recipe certification for any accelerator remain entirely unimplemented and must not be assumed from any lane's green compile.