# DGR-030 evidence — accelerator build presets and native CI/build matrix

**Status:** implementation complete, live-verified in this session (2026-07-23).
**Authority:** local `prd.json` is authoritative; Gitea is a projection.
**Upstream pin:** `e920c523e3b8a0163fe498af5bf90df35ff51d25` (`llama.cpp`, unchanged from DGR-027..029).

## What existed before this session

DGR-029 locked exactly one build lane — the deterministic CPU-only lane — in
`UPSTREAM_LOCK.json`'s `build` section, plus `scripts/llama_cpp_dependency.py`'s
`build()`/`smoke()`/`ctest_lane()`/`reproduce()`. There was no accelerator
preset, no SDK-availability probing, and no matrix runner: only the one CPU
lane existed, and there was no mechanism that could ever advertise a GPU
backend as compiled or capable.

## What changed in this session

- `packages/node/native/llama/UPSTREAM_LOCK.json`: added a new top-level
  `accelerator_presets` object with one entry each for `cuda` (`GGML_CUDA`),
  `rocm` (`GGML_HIP`), `vulkan` (`GGML_VULKAN`), and `metal` (`GGML_METAL`).
  Each entry names only the one backend flag it flips and an `sdk_probe`
  (a binary to resolve on `PATH`, an optional env-var override, and — for
  Metal — a `platform_only: "darwin"` gate). **The existing `build` section
  — the deterministic CPU default DGR-029 locked — is untouched.**
- `scripts/llama_cpp_dependency.py`:
  - `_load_lock()` now calls a new `_verify_accelerator_presets()`, which
    fail-closed-rejects any preset whose named backend flag is not `OFF` in
    the CPU default's `configure_flags` — structurally guaranteeing a preset
    can only ever *add* one backend on top of the untouched CPU baseline,
    never redefine it.
  - `accelerator_configure_flags(lock, name)` returns a **new** flag list —
    the CPU default's own `configure_flags` list is never mutated — with
    exactly the named preset's backend flag flipped `ON` and every other flag
    (including `GGML_CPU=ON`, the fallback ops backend GPU builds still need)
    left exactly as the CPU default declares it.
  - `_sdk_probe(probe)` / `accelerator_status(name, lock)` resolve a lane's
    SDK without ever raising: an absent SDK is returned as
    `{"available": false, "reason": "<binary> is unavailable on PATH"}` (or
    a platform-mismatch reason for Metal), so "unavailable" is data a caller
    reports, never an exception a caller has to remember to catch.
  - `accelerator_build(source, name, build_dir)` compiles one lane into its
    own out-of-tree `build_dir` (an isolated directory, never DGR-029's CPU
    `build_dir`), using the same patched-source verification and
    `native_targets` as the CPU lane, then writes a
    `meshnet-build-metadata.json` recording the exact `commit`/`commit_tree`,
    per-patch SHA-256 digests, the lane's overridden `configure_flags`, the
    resolved `cmake`/`cxx`/SDK-binary versions/paths, and explicit
    `model_downloads: false`, `hardware_execution: false`,
    `hardware_certified: false`, `semantic_certification: false` fields plus
    a `note` stating the lane is registered-dark until a real-hardware
    certification record exists. It **never** calls `smoke()`/`ctest_lane()`
    — running a binary linked against a real accelerator backend would touch
    real hardware, which this story deliberately keeps out of scope.
  - Added `accelerator-status --name <lane>` and
    `accelerator-build --name <lane> --source-dir --build-dir` CLI
    subcommands, mirroring the existing `ctest`/`build` subcommand pattern.
- `scripts/native_accelerator_matrix.py` (new): the native CI/build matrix.
  `run_matrix(workspace)` fetches and applies the locked pin/patch stack once,
  runs the unchanged CPU lane (build → smoke → ctest, exactly DGR-029's
  contract), then for each `accelerator_presets` entry either reports
  `{"status": "skipped", "reason": ...}` (SDK absent) or compiles it via
  `accelerator_build` and reports `{"status": "built", ...}` — never silently
  treating a skip as a pass. Any `DependencyError` from a lane (CPU or
  accelerator) is caught per-lane and reported as `{"status": "failed", ...}`
  without aborting the remaining lanes or skipping cleanup. `reverse()` always
  runs in a `finally`, restoring the exact pristine pin/tree regardless of
  lane outcomes. The CLI prints a JSON report and exits non-zero only if any
  lane actually `failed` (a `skipped` lane never fails the run).
- `tests/test_llama_cpp_dependency.py`: added 7 new tests —
  `test_accelerator_presets_isolate_one_backend_without_touching_the_cpu_default`
  (every preset flips exactly its own flag and the CPU default list is never
  mutated), `test_accelerator_configure_flags_rejects_an_unknown_lane`,
  `test_accelerator_status_reports_unavailable_sdks_without_raising` (asserts
  the exact reason string for cuda/rocm/vulkan/metal absence),
  `test_accelerator_status_honors_an_explicit_sdk_override`,
  `test_accelerator_status_rejects_an_unknown_lane`,
  `test_accelerator_build_refuses_to_compile_an_unavailable_lane` (asserts no
  build directory is created), and a `requires_cmake`-gated
  `test_accelerator_build_compiles_the_available_lane_with_isolated_evidence`,
  which builds a tiny synthetic CMake project (not the full llama.cpp tree) to
  prove `accelerator_build`'s "SDK present" path really configures with the
  overridden flag, compiles, and writes the registered-dark metadata — in
  about a second, without a real GPU SDK.
- `tests/test_native_accelerator_matrix.py` (new): 3 offline tests exercising
  `run_matrix`'s orchestration with `llama_cpp_dependency`'s
  fetch/apply/reverse/build/smoke/ctest_lane/accelerator_status/
  accelerator_build stubbed out — proving unavailable SDKs are reported
  `skipped` (never a false pass), an available accelerator lane is compiled
  without ever calling `smoke`/`ctest_lane`, and a lane failure is reported
  per-lane without aborting sibling lanes or skipping the `reverse()` cleanup.

## Toolchain note

As in DGR-029, neither the ambient system Python nor `.venv-rocm` has `cmake`;
this session's `.venv` also had no `cmake` (a prior session's install did not
persist). This session ran `.venv/bin/python3 -m ensurepip --upgrade` (no
`pip` was present in `.venv` either) and then
`.venv/bin/python3 -m pip install cmake`, landing the same PyPI wheel
(`cmake==4.4.0`) DGR-029 used, at `.venv/bin/cmake` / `.venv/bin/ctest`. All
commands below were run with that `.venv/bin` prepended to `PATH`. No CUDA,
ROCm, or Vulkan SDK (`nvcc`, `hipcc`, `glslc`) is installed in this
environment, and the host platform is Linux, not `darwin` — so all four
accelerator lanes are genuinely `skipped` in this environment's own live run
below, which is real evidence for AC2 ("unavailable SDKs ... explicit
unavailable/skipped lanes"), not a simulated one.

## Verification — live native CI/build matrix run

```text
$ rm -rf build/llama.cpp/build build/llama.cpp/build-cuda build/llama.cpp/build-rocm build/llama.cpp/build-vulkan build/llama.cpp/build-metal
$ python3 scripts/native_accelerator_matrix.py
reused verified offline cache: .../build/llama.cpp/source
usage: .../build/llama.cpp/build/bin/llama-gguf-hash [options] GGUF_IN
...
Test project .../build/llama.cpp/build
    Start 27: test-meshnet-range-ownership
1/1 Test #27: test-meshnet-range-ownership .....   Passed    0.01 sec
100% tests passed out of 1
{
  "failed_lanes": [],
  "hardware_certified": false,
  "lanes": [
    {
      "build_dir": ".../build/llama.cpp/build",
      "lane": "cpu",
      "metadata": {
        "cmake": "cmake version 4.4.0",
        "commit": "e920c523e3b8a0163fe498af5bf90df35ff51d25",
        "commit_tree": "6c91a11407a3a3fb160f5dac705f9c59718f54f1",
        "configure_flags": [
          "-DCMAKE_BUILD_TYPE=Release", "-DLLAMA_BUILD_TESTS=ON",
          "-DLLAMA_BUILD_EXAMPLES=ON", "-DLLAMA_BUILD_SERVER=OFF",
          "-DLLAMA_BUILD_TOOLS=OFF", "-DLLAMA_BUILD_APP=OFF", "-DLLAMA_CURL=OFF",
          "-DGGML_CPU=ON", "-DGGML_BLAS=OFF", "-DGGML_CUDA=OFF",
          "-DGGML_HIP=OFF", "-DGGML_VULKAN=OFF", "-DGGML_METAL=OFF"
        ],
        "cxx": "c++ (GCC) 15.2.1 20260123 (Red Hat 15.2.1-7)",
        "model_downloads": false,
        "patches": { "...": "... (5 entries, unchanged sha256 digests from DGR-029)" },
        "semantic_certification": false
      },
      "status": "built"
    },
    {"lane": "cuda", "reason": "nvcc is unavailable on PATH", "status": "skipped"},
    {"lane": "rocm", "reason": "hipcc is unavailable on PATH", "status": "skipped"},
    {"lane": "vulkan", "reason": "glslc is unavailable on PATH", "status": "skipped"},
    {"lane": "metal", "reason": "platform 'linux' is not 'darwin'", "status": "skipped"}
  ],
  "note": "A `built` lane means it compiled with the exact recorded compiler/SDK/upstream-pin/patch-stack/build-option evidence — it never means an accelerator device was exercised. Every backend/model/recipe lane stays registered-dark until a separate real-hardware certification record exists."
}
$ echo $?
0
```

Wall-clock: `real 2m19.797s` — matches DGR-029's ~2m16s CPU-lane compile; no
accelerator lane actually compiled in this environment (all four SDKs are
genuinely absent), so this run's added cost over DGR-029's own CPU-only
`reproduce()` is just the four fast SDK probes.

Post-run checks (source checkout left pristine by the matrix's `reverse()`):

```text
$ git -C build/llama.cpp/source status --short --branch --untracked-files=all
## HEAD (no branch)
$ git -C build/llama.cpp/source rev-parse HEAD HEAD^{tree}
e920c523e3b8a0163fe498af5bf90df35ff51d25
6c91a11407a3a3fb160f5dac705f9c59718f54f1
$ ls build/llama.cpp/ | grep build
build
```

Only the CPU lane's `build/` directory was created — no `build-cuda`,
`build-rocm`, `build-vulkan`, or `build-metal` directory exists, because every
accelerator lane was genuinely skipped rather than attempted.

## Verification — targeted test suites and shared gates

| Command | Result |
| --- | --- |
| `python3 -m pytest -q tests/test_llama_cpp_dependency.py tests/test_native_accelerator_matrix.py` | `19 passed` (9 pre-existing + 7 new accelerator-lane tests in `test_llama_cpp_dependency.py`, 3 new in `test_native_accelerator_matrix.py`; the `requires_cmake`-gated compile test ran for real, not skipped) |
| `python3 -m compileall -q packages tests` | exit 0 |
| `git diff --check -- packages/node/native/llama/UPSTREAM_LOCK.json scripts/llama_cpp_dependency.py tests/test_llama_cpp_dependency.py scripts/native_accelerator_matrix.py tests/test_native_accelerator_matrix.py` | exit 0 |
| `python3 scripts/ralph_prd_schema.py validate .scratch/distributed-gguf-runtime/prd.json` | `OK: 55 stories validated.` |

`git diff --check` against the full working tree separately reports one
pre-existing trailing-whitespace line in `.ralph-tui-run.log`, which was
already modified before this session started (see the session's initial
`git status`) and is unrelated to this story's scope; it is excluded above by
naming this story's own changed files explicitly.

`python3 -m pytest -q tests/test_ralph_prd_schema.py` reports `55 failed, 53
passed` in this session (all `test_render_issue_markdown_matches_committed_file`
drift between `prd.json` and committed issue Markdown for other stories,
e.g. `DGR-053`..`DGR-071`). `git stash`-ing this session's changes and rerunning
reproduces `56 failed, 52 passed` identically — the same 56 failures minus the
one this session's own `DGR-030` regeneration fixed, confirming the remaining
55 predate this story and are out of scope to fix here. This session did
regenerate `.scratch/distributed-gguf-runtime/issues/030-add-accelerator-
build-presets-and-native-ci-matrix.md` via
`python3 scripts/ralph_prd_schema.py render ... DGR-030` so DGR-030's own
generated issue Markdown matches `prd.json` byte-for-byte (confirmed by the
`test_render_issue_markdown_matches_committed_file[DGR-030]` case no longer
appearing in the failure list).

## Ensuring build success does not advertise capability

- Every accelerator lane's `meshnet-build-metadata.json` explicitly records
  `hardware_execution: false`, `hardware_certified: false`, and
  `semantic_certification: false`, plus a `note` stating the lane is
  registered-dark until a separate real-hardware certification record exists
  — the same "artifact states this, not just prose" pattern DGR-029 used for
  the CPU lane's `model_downloads`/`semantic_certification` fields.
- `accelerator_build` never runs `smoke()` or `ctest_lane()`: it only
  configures and compiles the exact `native_targets` DGR-029 already locked
  (`llama-gguf-hash`, `test-meshnet-range-ownership`) — no binary linked
  against a real accelerator backend is ever executed by this story's code.
- `_verify_accelerator_presets()` structurally refuses any preset whose
  backend flag is not `OFF` in the locked CPU default, so a preset can never
  be defined in a way that redefines (rather than adds one backend on top of)
  DGR-029's deterministic CPU lane.
- The matrix's top-level report always carries `"hardware_certified": false`
  regardless of how many lanes built, and its `note` field states this
  explicitly for any consumer reading only the report, not the per-lane
  metadata.

## Limitations

- This story proves accelerator lanes *compile* with correct, isolated
  flags and preserves exact evidence when a lane's SDK is present. It proves
  nothing about numerical correctness, performance, or any backend/model/
  recipe capability on real accelerator hardware — that is explicitly
  deferred to DGR-041 (capability registration), DGR-053 (real 2-4 stage
  certification), and DGR-067 (capability matrix certification), all of which
  remain unimplemented.
- No CUDA, ROCm, or Vulkan SDK, and no macOS/Metal toolchain, is available in
  this session's environment, so the "compile an available accelerator lane"
  path is proven end-to-end only via the `requires_cmake`-gated synthetic-
  project unit test and the offline matrix-orchestration tests, not via a
  live compile of the real llama.cpp tree under `GGML_CUDA=ON` (etc.). A
  future session with a real SDK installed will exercise
  `accelerator_build`'s real-lane path against the genuine llama.cpp source
  for the first time; nothing in this story's design assumes that hasn't
  happened yet.
- The accelerator lanes reuse the CPU lane's exact `native_targets`
  (`llama-gguf-hash`, `test-meshnet-range-ownership`), so a passing
  accelerator compile also proves the DGR-027/DGR-028 patch stack's
  range-ownership code compiles under that backend flag combination — but,
  per the point above, only structurally; it says nothing about GPU
  execution correctness.
- `cmake`/`ctest` remain absent system-wide in this environment; this session
  reinstalled them into `.venv` exactly as DGR-029 did, and that install does
  not appear to persist across sessions (this session found `.venv` without
  `cmake` despite DGR-029's evidence recording its earlier install). A future
  session without a `cmake`-equipped `.venv` will see the same actionable
  "cmake is unavailable" failure DGR-029 demonstrated, not a silent pass, and
  the new `requires_cmake`-gated tests will be skipped rather than failing.
- `git diff --check` and `tests/test_ralph_prd_schema.py` both carry
  pre-existing, out-of-scope failures unrelated to this story (see the gates
  table above); this story's own changed files pass both checks cleanly.

## Dependency handoff

DGR-053 (real 2-4 stage certification), DGR-067 (capability matrix
certification), and DGR-068 (packaged releases) may rely on: four isolated,
out-of-tree accelerator build presets (`cuda`/`rocm`/`vulkan`/`metal`) in
`UPSTREAM_LOCK.json`'s `accelerator_presets`, each toggling exactly one
backend flag on top of DGR-029's unchanged CPU default; a native CI/build
matrix (`scripts/native_accelerator_matrix.py`) that compiles every
SDK-available lane with full compiler/SDK/upstream-pin/patch-stack/build-
option evidence and reports SDK-unavailable lanes as explicit `skipped`
lanes, never a false pass; and a compile-only contract (no lane here ever
runs a binary against real accelerator hardware). Real-hardware execution,
numerical correctness, performance measurement, and backend/model/recipe
certification for any accelerator remain entirely unimplemented and must not
be assumed from any lane's green compile.
