story: DGR-030 Add accelerator build presets and native CI matrix

2026-07-23 10:51:08 +03:00
parent 254297660a
commit fd742d35c0
13 changed files with 2576 additions and 14 deletions
--- a/.scratch/distributed-gguf-runtime/evidence/DGR-030/README.md
+++ b/.scratch/distributed-gguf-runtime/evidence/DGR-030/README.md
@@ -0,0 +1,275 @@
+# DGR-030 evidence — accelerator build presets and native CI/build matrix
+
+**Status:** implementation complete, live-verified in this session (2026-07-23).
+**Authority:** local `prd.json` is authoritative; Gitea is a projection.
+**Upstream pin:** `e920c523e3b8a0163fe498af5bf90df35ff51d25` (`llama.cpp`, unchanged from DGR-027..029).
+
+## What existed before this session
+
+DGR-029 locked exactly one build lane — the deterministic CPU-only lane — in
+`UPSTREAM_LOCK.json`'s `build` section, plus `scripts/llama_cpp_dependency.py`'s
+`build()`/`smoke()`/`ctest_lane()`/`reproduce()`. There was no accelerator
+preset, no SDK-availability probing, and no matrix runner: only the one CPU
+lane existed, and there was no mechanism that could ever advertise a GPU
+backend as compiled or capable.
+
+## What changed in this session
+
+- `packages/node/native/llama/UPSTREAM_LOCK.json`: added a new top-level
+  `accelerator_presets` object with one entry each for `cuda` (`GGML_CUDA`),
+  `rocm` (`GGML_HIP`), `vulkan` (`GGML_VULKAN`), and `metal` (`GGML_METAL`).
+  Each entry names only the one backend flag it flips and an `sdk_probe`
+  (a binary to resolve on `PATH`, an optional env-var override, and — for
+  Metal — a `platform_only: "darwin"` gate). **The existing `build` section
+  — the deterministic CPU default DGR-029 locked — is untouched.**
+- `scripts/llama_cpp_dependency.py`:
+  - `_load_lock()` now calls a new `_verify_accelerator_presets()`, which
+    fail-closed-rejects any preset whose named backend flag is not `OFF` in
+    the CPU default's `configure_flags` — structurally guaranteeing a preset
+    can only ever *add* one backend on top of the untouched CPU baseline,
+    never redefine it.
+  - `accelerator_configure_flags(lock, name)` returns a **new** flag list —
+    the CPU default's own `configure_flags` list is never mutated — with
+    exactly the named preset's backend flag flipped `ON` and every other flag
+    (including `GGML_CPU=ON`, the fallback ops backend GPU builds still need)
+    left exactly as the CPU default declares it.
+  - `_sdk_probe(probe)` / `accelerator_status(name, lock)` resolve a lane's
+    SDK without ever raising: an absent SDK is returned as
+    `{"available": false, "reason": "<binary> is unavailable on PATH"}` (or
+    a platform-mismatch reason for Metal), so "unavailable" is data a caller
+    reports, never an exception a caller has to remember to catch.
+  - `accelerator_build(source, name, build_dir)` compiles one lane into its
+    own out-of-tree `build_dir` (an isolated directory, never DGR-029's CPU
+    `build_dir`), using the same patched-source verification and
+    `native_targets` as the CPU lane, then writes a
+    `meshnet-build-metadata.json` recording the exact `commit`/`commit_tree`,
+    per-patch SHA-256 digests, the lane's overridden `configure_flags`, the
+    resolved `cmake`/`cxx`/SDK-binary versions/paths, and explicit
+    `model_downloads: false`, `hardware_execution: false`,
+    `hardware_certified: false`, `semantic_certification: false` fields plus
+    a `note` stating the lane is registered-dark until a real-hardware
+    certification record exists. It **never** calls `smoke()`/`ctest_lane()`
+    — running a binary linked against a real accelerator backend would touch
+    real hardware, which this story deliberately keeps out of scope.
+  - Added `accelerator-status --name <lane>` and
+    `accelerator-build --name <lane> --source-dir --build-dir` CLI
+    subcommands, mirroring the existing `ctest`/`build` subcommand pattern.
+- `scripts/native_accelerator_matrix.py` (new): the native CI/build matrix.
+  `run_matrix(workspace)` fetches and applies the locked pin/patch stack once,
+  runs the unchanged CPU lane (build → smoke → ctest, exactly DGR-029's
+  contract), then for each `accelerator_presets` entry either reports
+  `{"status": "skipped", "reason": ...}` (SDK absent) or compiles it via
+  `accelerator_build` and reports `{"status": "built", ...}` — never silently
+  treating a skip as a pass. Any `DependencyError` from a lane (CPU or
+  accelerator) is caught per-lane and reported as `{"status": "failed", ...}`
+  without aborting the remaining lanes or skipping cleanup. `reverse()` always
+  runs in a `finally`, restoring the exact pristine pin/tree regardless of
+  lane outcomes. The CLI prints a JSON report and exits non-zero only if any
+  lane actually `failed` (a `skipped` lane never fails the run).
+- `tests/test_llama_cpp_dependency.py`: added 7 new tests —
+  `test_accelerator_presets_isolate_one_backend_without_touching_the_cpu_default`
+  (every preset flips exactly its own flag and the CPU default list is never
+  mutated), `test_accelerator_configure_flags_rejects_an_unknown_lane`,
+  `test_accelerator_status_reports_unavailable_sdks_without_raising` (asserts
+  the exact reason string for cuda/rocm/vulkan/metal absence),
+  `test_accelerator_status_honors_an_explicit_sdk_override`,
+  `test_accelerator_status_rejects_an_unknown_lane`,
+  `test_accelerator_build_refuses_to_compile_an_unavailable_lane` (asserts no
+  build directory is created), and a `requires_cmake`-gated
+  `test_accelerator_build_compiles_the_available_lane_with_isolated_evidence`,
+  which builds a tiny synthetic CMake project (not the full llama.cpp tree) to
+  prove `accelerator_build`'s "SDK present" path really configures with the
+  overridden flag, compiles, and writes the registered-dark metadata — in
+  about a second, without a real GPU SDK.
+- `tests/test_native_accelerator_matrix.py` (new): 3 offline tests exercising
+  `run_matrix`'s orchestration with `llama_cpp_dependency`'s
+  fetch/apply/reverse/build/smoke/ctest_lane/accelerator_status/
+  accelerator_build stubbed out — proving unavailable SDKs are reported
+  `skipped` (never a false pass), an available accelerator lane is compiled
+  without ever calling `smoke`/`ctest_lane`, and a lane failure is reported
+  per-lane without aborting sibling lanes or skipping the `reverse()` cleanup.
+
+## Toolchain note
+
+As in DGR-029, neither the ambient system Python nor `.venv-rocm` has `cmake`;
+this session's `.venv` also had no `cmake` (a prior session's install did not
+persist). This session ran `.venv/bin/python3 -m ensurepip --upgrade` (no
+`pip` was present in `.venv` either) and then
+`.venv/bin/python3 -m pip install cmake`, landing the same PyPI wheel
+(`cmake==4.4.0`) DGR-029 used, at `.venv/bin/cmake` / `.venv/bin/ctest`. All
+commands below were run with that `.venv/bin` prepended to `PATH`. No CUDA,
+ROCm, or Vulkan SDK (`nvcc`, `hipcc`, `glslc`) is installed in this
+environment, and the host platform is Linux, not `darwin` — so all four
+accelerator lanes are genuinely `skipped` in this environment's own live run
+below, which is real evidence for AC2 ("unavailable SDKs ... explicit
+unavailable/skipped lanes"), not a simulated one.
+
+## Verification — live native CI/build matrix run
+
+```text
+$ rm -rf build/llama.cpp/build build/llama.cpp/build-cuda build/llama.cpp/build-rocm build/llama.cpp/build-vulkan build/llama.cpp/build-metal
+$ python3 scripts/native_accelerator_matrix.py
+reused verified offline cache: .../build/llama.cpp/source
+usage: .../build/llama.cpp/build/bin/llama-gguf-hash [options] GGUF_IN
+...
+Test project .../build/llama.cpp/build
+    Start 27: test-meshnet-range-ownership
+1/1 Test #27: test-meshnet-range-ownership .....   Passed    0.01 sec
+100% tests passed out of 1
+{
+  "failed_lanes": [],
+  "hardware_certified": false,
+  "lanes": [
+    {
+      "build_dir": ".../build/llama.cpp/build",
+      "lane": "cpu",
+      "metadata": {
+        "cmake": "cmake version 4.4.0",
+        "commit": "e920c523e3b8a0163fe498af5bf90df35ff51d25",
+        "commit_tree": "6c91a11407a3a3fb160f5dac705f9c59718f54f1",
+        "configure_flags": [
+          "-DCMAKE_BUILD_TYPE=Release", "-DLLAMA_BUILD_TESTS=ON",
+          "-DLLAMA_BUILD_EXAMPLES=ON", "-DLLAMA_BUILD_SERVER=OFF",
+          "-DLLAMA_BUILD_TOOLS=OFF", "-DLLAMA_BUILD_APP=OFF", "-DLLAMA_CURL=OFF",
+          "-DGGML_CPU=ON", "-DGGML_BLAS=OFF", "-DGGML_CUDA=OFF",
+          "-DGGML_HIP=OFF", "-DGGML_VULKAN=OFF", "-DGGML_METAL=OFF"
+        ],
+        "cxx": "c++ (GCC) 15.2.1 20260123 (Red Hat 15.2.1-7)",
+        "model_downloads": false,
+        "patches": { "...": "... (5 entries, unchanged sha256 digests from DGR-029)" },
+        "semantic_certification": false
+      },
+      "status": "built"
+    },
+    {"lane": "cuda", "reason": "nvcc is unavailable on PATH", "status": "skipped"},
+    {"lane": "rocm", "reason": "hipcc is unavailable on PATH", "status": "skipped"},
+    {"lane": "vulkan", "reason": "glslc is unavailable on PATH", "status": "skipped"},
+    {"lane": "metal", "reason": "platform 'linux' is not 'darwin'", "status": "skipped"}
+  ],
+  "note": "A `built` lane means it compiled with the exact recorded compiler/SDK/upstream-pin/patch-stack/build-option evidence — it never means an accelerator device was exercised. Every backend/model/recipe lane stays registered-dark until a separate real-hardware certification record exists."
+}
+$ echo $?
+0
+```
+
+Wall-clock: `real 2m19.797s` — matches DGR-029's ~2m16s CPU-lane compile; no
+accelerator lane actually compiled in this environment (all four SDKs are
+genuinely absent), so this run's added cost over DGR-029's own CPU-only
+`reproduce()` is just the four fast SDK probes.
+
+Post-run checks (source checkout left pristine by the matrix's `reverse()`):
+
+```text
+$ git -C build/llama.cpp/source status --short --branch --untracked-files=all
+## HEAD (no branch)
+$ git -C build/llama.cpp/source rev-parse HEAD HEAD^{tree}
+e920c523e3b8a0163fe498af5bf90df35ff51d25
+6c91a11407a3a3fb160f5dac705f9c59718f54f1
+$ ls build/llama.cpp/ | grep build
+build
+```
+
+Only the CPU lane's `build/` directory was created — no `build-cuda`,
+`build-rocm`, `build-vulkan`, or `build-metal` directory exists, because every
+accelerator lane was genuinely skipped rather than attempted.
+
+## Verification — targeted test suites and shared gates
+
+| Command | Result |
+| --- | --- |
+| `python3 -m pytest -q tests/test_llama_cpp_dependency.py tests/test_native_accelerator_matrix.py` | `19 passed` (9 pre-existing + 7 new accelerator-lane tests in `test_llama_cpp_dependency.py`, 3 new in `test_native_accelerator_matrix.py`; the `requires_cmake`-gated compile test ran for real, not skipped) |
+| `python3 -m compileall -q packages tests` | exit 0 |
+| `git diff --check -- packages/node/native/llama/UPSTREAM_LOCK.json scripts/llama_cpp_dependency.py tests/test_llama_cpp_dependency.py scripts/native_accelerator_matrix.py tests/test_native_accelerator_matrix.py` | exit 0 |
+| `python3 scripts/ralph_prd_schema.py validate .scratch/distributed-gguf-runtime/prd.json` | `OK: 55 stories validated.` |
+
+`git diff --check` against the full working tree separately reports one
+pre-existing trailing-whitespace line in `.ralph-tui-run.log`, which was
+already modified before this session started (see the session's initial
+`git status`) and is unrelated to this story's scope; it is excluded above by
+naming this story's own changed files explicitly.
+
+`python3 -m pytest -q tests/test_ralph_prd_schema.py` reports `55 failed, 53
+passed` in this session (all `test_render_issue_markdown_matches_committed_file`
+drift between `prd.json` and committed issue Markdown for other stories,
+e.g. `DGR-053`..`DGR-071`). `git stash`-ing this session's changes and rerunning
+reproduces `56 failed, 52 passed` identically — the same 56 failures minus the
+one this session's own `DGR-030` regeneration fixed, confirming the remaining
+55 predate this story and are out of scope to fix here. This session did
+regenerate `.scratch/distributed-gguf-runtime/issues/030-add-accelerator-
+build-presets-and-native-ci-matrix.md` via
+`python3 scripts/ralph_prd_schema.py render ... DGR-030` so DGR-030's own
+generated issue Markdown matches `prd.json` byte-for-byte (confirmed by the
+`test_render_issue_markdown_matches_committed_file[DGR-030]` case no longer
+appearing in the failure list).
+
+## Ensuring build success does not advertise capability
+
+- Every accelerator lane's `meshnet-build-metadata.json` explicitly records
+  `hardware_execution: false`, `hardware_certified: false`, and
+  `semantic_certification: false`, plus a `note` stating the lane is
+  registered-dark until a separate real-hardware certification record exists
+  — the same "artifact states this, not just prose" pattern DGR-029 used for
+  the CPU lane's `model_downloads`/`semantic_certification` fields.
+- `accelerator_build` never runs `smoke()` or `ctest_lane()`: it only
+  configures and compiles the exact `native_targets` DGR-029 already locked
+  (`llama-gguf-hash`, `test-meshnet-range-ownership`) — no binary linked
+  against a real accelerator backend is ever executed by this story's code.
+- `_verify_accelerator_presets()` structurally refuses any preset whose
+  backend flag is not `OFF` in the locked CPU default, so a preset can never
+  be defined in a way that redefines (rather than adds one backend on top of)
+  DGR-029's deterministic CPU lane.
+- The matrix's top-level report always carries `"hardware_certified": false`
+  regardless of how many lanes built, and its `note` field states this
+  explicitly for any consumer reading only the report, not the per-lane
+  metadata.
+
+## Limitations
+
+- This story proves accelerator lanes *compile* with correct, isolated
+  flags and preserves exact evidence when a lane's SDK is present. It proves
+  nothing about numerical correctness, performance, or any backend/model/
+  recipe capability on real accelerator hardware — that is explicitly
+  deferred to DGR-041 (capability registration), DGR-053 (real 2-4 stage
+  certification), and DGR-067 (capability matrix certification), all of which
+  remain unimplemented.
+- No CUDA, ROCm, or Vulkan SDK, and no macOS/Metal toolchain, is available in
+  this session's environment, so the "compile an available accelerator lane"
+  path is proven end-to-end only via the `requires_cmake`-gated synthetic-
+  project unit test and the offline matrix-orchestration tests, not via a
+  live compile of the real llama.cpp tree under `GGML_CUDA=ON` (etc.). A
+  future session with a real SDK installed will exercise
+  `accelerator_build`'s real-lane path against the genuine llama.cpp source
+  for the first time; nothing in this story's design assumes that hasn't
+  happened yet.
+- The accelerator lanes reuse the CPU lane's exact `native_targets`
+  (`llama-gguf-hash`, `test-meshnet-range-ownership`), so a passing
+  accelerator compile also proves the DGR-027/DGR-028 patch stack's
+  range-ownership code compiles under that backend flag combination — but,
+  per the point above, only structurally; it says nothing about GPU
+  execution correctness.
+- `cmake`/`ctest` remain absent system-wide in this environment; this session
+  reinstalled them into `.venv` exactly as DGR-029 did, and that install does
+  not appear to persist across sessions (this session found `.venv` without
+  `cmake` despite DGR-029's evidence recording its earlier install). A future
+  session without a `cmake`-equipped `.venv` will see the same actionable
+  "cmake is unavailable" failure DGR-029 demonstrated, not a silent pass, and
+  the new `requires_cmake`-gated tests will be skipped rather than failing.
+- `git diff --check` and `tests/test_ralph_prd_schema.py` both carry
+  pre-existing, out-of-scope failures unrelated to this story (see the gates
+  table above); this story's own changed files pass both checks cleanly.
+
+## Dependency handoff
+
+DGR-053 (real 2-4 stage certification), DGR-067 (capability matrix
+certification), and DGR-068 (packaged releases) may rely on: four isolated,
+out-of-tree accelerator build presets (`cuda`/`rocm`/`vulkan`/`metal`) in
+`UPSTREAM_LOCK.json`'s `accelerator_presets`, each toggling exactly one
+backend flag on top of DGR-029's unchanged CPU default; a native CI/build
+matrix (`scripts/native_accelerator_matrix.py`) that compiles every
+SDK-available lane with full compiler/SDK/upstream-pin/patch-stack/build-
+option evidence and reports SDK-unavailable lanes as explicit `skipped`
+lanes, never a false pass; and a compile-only contract (no lane here ever
+runs a binary against real accelerator hardware). Real-hardware execution,
+numerical correctness, performance measurement, and backend/model/recipe
+certification for any accelerator remain entirely unimplemented and must not
+be assumed from any lane's green compile.
--- a/.scratch/distributed-gguf-runtime/issues/030-add-accelerator-build-presets-and-native-ci-matrix.md
+++ b/.scratch/distributed-gguf-runtime/issues/030-add-accelerator-build-presets-and-native-ci-matrix.md
@@ -1,7 +1,7 @@
 <!-- GENERATED FROM prd.json — DO NOT EDIT AS AN INDEPENDENT SOURCE. prd.json IS AUTHORITATIVE. -->
 # DGR-030: Add accelerator build presets and native CI matrix

- **Status / triage:** specification only; `ready-for-agent`; `passes: false`
+- **Status / triage:** completed; `passes: true`
 - **Execution mode:** `AFK`
 - **Milestone:** `M1`
 - **Dependencies:** `DGR-029`
@@ -18,11 +18,11 @@ Fresh Ralph session: read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md`,

 ## Acceptance criteria

- [ ] Add isolated out-of-tree presets for CUDA, ROCm, Vulkan, and Metal without changing the deterministic CPU default.
- [ ] Add a native CI/build matrix that reports unavailable SDKs as explicit unavailable/skipped lanes rather than false success.
- [ ] Compile each available lane and preserve exact compiler, SDK, upstream pin, patch-stack, and build-option evidence.
- [ ] Keep every backend/model/recipe lane registered-dark until a separate real-hardware certification record exists.
- [ ] Applicable shared quality gates in `prd.json` pass, and the evidence handoff records exact commands/results, changed files, limitations, and dependency handoff.
+- [x] Add isolated out-of-tree presets for CUDA, ROCm, Vulkan, and Metal without changing the deterministic CPU default.
+- [x] Add a native CI/build matrix that reports unavailable SDKs as explicit unavailable/skipped lanes rather than false success.
+- [x] Compile each available lane and preserve exact compiler, SDK, upstream pin, patch-stack, and build-option evidence.
+- [x] Keep every backend/model/recipe lane registered-dark until a separate real-hardware certification record exists.
+- [x] Applicable shared quality gates in `prd.json` pass, and the evidence handoff records exact commands/results, changed files, limitations, and dependency handoff.

 ## Shared quality gates

@@ -30,10 +30,7 @@ Fresh Ralph session: read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md`,
 - `git diff --check` passes.
 - Default tests are model-download-free, API-credit-free, and GPU-free.
 - Evidence README records exact changed files, commands/results, limitations, and dependency handoff; no fabricated evidence or inherited completion credit.
- Native changes pass focused out-of-tree CMake build and CTest; patch changes verify clean apply/check/reverse against the exact llama.cpp pin.
- Runs are opt-in and record exact artifact/split hashes, runtime/upstream pin, backend/driver, hardware, network, commands, and raw metrics. Model artifacts use configured mounted-drive storage and never `/home`.
- Preserve existing Transformers behavior and backend-agnostic Tracker routing/load balancing/billing/relay semantics unless an explicit versioned contract says otherwise. One scoped story commit is expected during execution, but this specification-materialization change is not committed.

 ## Evidence handoff

-Write and verify `.scratch/distributed-gguf-runtime/evidence/DGR-030/README.md`. Until every criterion and applicable gate has real evidence, this story remains `passes: false`. Legacy evidence is provenance only, not completion credit.
+Verified evidence: `.scratch/distributed-gguf-runtime/evidence/DGR-030/README.md`. Legacy evidence remains provenance only and grants no implementation completion credit.
--- a/.scratch/distributed-gguf-runtime/prd.json
+++ b/.scratch/distributed-gguf-runtime/prd.json
@@ -538,13 +538,14 @@
        "Keep every backend/model/recipe lane registered-dark until a separate real-hardware certification record exists.",
        "Applicable shared quality gates in `prd.json` pass, and the evidence handoff records exact commands/results, changed files, limitations, and dependency handoff."
      ],
-      "passes": false,
+      "passes": true,
      "notes": "Generated source issue: .scratch/distributed-gguf-runtime/issues/030-add-accelerator-build-presets-and-native-ci-matrix.md; prd.json is authoritative.",
      "blocks": [
        "DGR-053",
        "DGR-067",
        "DGR-068"
-      ]
+      ],
+      "completionNotes": "Completed by agent"
    },
    {
      "id": "DGR-031",
@@ -2161,6 +2162,6 @@
    }
  ],
  "metadata": {
-    "updatedAt": "2026-07-22T06:44:18.107Z"
+    "updatedAt": "2026-07-23T07:51:08.112Z"
  }
 }