neuron-tai/.scratch/distributed-gguf-runtime/evidence/DGR-029

DGR-029 evidence — native CMake skeleton and deterministic CPU lane

Status: implementation complete, live-verified in this session (2026-07-22). Authority: local prd.json is authoritative; Gitea is a projection. Upstream pin: e920c523e3b8a0163fe498af5bf90df35ff51d25 (llama.cpp, from UPSTREAM_LOCK.json).

What existed before this session

scripts/llama_cpp_dependency.py already had build(), smoke(), and reproduce() functions and UPSTREAM_LOCK.json already had a build section (both landed as part of DGR-028's commit 7da90ef), but:

• No test in tests/test_llama_cpp_dependency.py ever exercised build/smoke/reproduce — only fetch/apply/reverse/inspect had coverage.
• cmake was not installed in the DGR-028 session's environment ("cmake is not installed in this environment," per its evidence), so this lane was never actually run end to end; DGR-028's own live-verified CTest evidence used a one-off manual cmake/ctest invocation with -DLLAMA_BUILD_TESTS=ON outside this driver, against a build directory that no longer exists in this session.
• The locked configure_flags did not force CPU-only backend options (GGML_CUDA/GGML_HIP/ GGML_VULKAN/GGML_METAL/GGML_BLAS) — relying on upstream per-platform defaults (which happen to default OFF on Linux, but are undocumented and platform-dependent), and LLAMA_BUILD_TESTS was OFF, so no CTest lane existed at all — only a --help smoke check against the unrelated stock llama-gguf-hash tool.

This session found and closed those three gaps rather than re-implementing from scratch.

What changed in this session

• packages/node/native/llama/UPSTREAM_LOCK.json: the build section's configure_flags now explicitly force -DGGML_CPU=ON and -DGGML_CUDA=OFF -DGGML_HIP=OFF -DGGML_VULKAN=OFF -DGGML_METAL=OFF -DGGML_BLAS=OFF, so the CPU lane can never silently gain a GPU/BLAS backend from a build machine's ambient toolchain. -DLLAMA_BUILD_TESTS flipped OFF → ON (required so the test-meshnet-range-ownership CTest target exists at all — configuring LLAMA_BUILD_TESTS=ON does not itself build every upstream test, only registers them; the native_targets list still controls what actually gets compiled). Added native_targets entry test-meshnet-range-ownership and a new ctest_regex field ("^test-meshnet-range-ownership$") naming the exact deterministic model-free fixture CTest added by DGR-028's patch 0005.
• scripts/llama_cpp_dependency.py: factored _cmake()'s override/PATH/venv-sibling resolution into a shared _toolchain_binary(name, env_var) and added _ctest() using the same resolution (CTEST env override, PATH, or the sibling of the resolved cmake binary's environment). Added ctest_lane(build_dir), which loads the lock's ctest_regex and runs ctest --test-dir <build_dir> -R <regex> --output-on-failure, printing output on success and raising DependencyError (via the existing _run wrapper, which already attaches stdout/stderr detail) on failure. Added a ctest CLI subcommand (--build-dir). Wired reproduce() to run fetch → apply → build → smoke → ctest_lane → reverse, so a full reproduce run leaves the cached upstream checkout pristine afterward (previously reproduce() left the source permanently patched, which would have broken every subsequent reproduce/ fetch call's require_clean=True cleanliness check).
• tests/test_llama_cpp_dependency.py: added test_build_config_locks_an_explicit_cpu_only_deterministic_lane (offline; asserts the lock's configure_flags are CPU-only and that ctest_regex/native_targets/smoke_binary all agree with each other and with patched_paths) and test_ctest_lane_raises_an_actionable_error_for_a_failing_named_test (gated on cmake availability via a requires_cmake marker mirroring test_native_identity_emission.py's requires_cc pattern; builds a tiny synthetic two-test CMake project — not the full llama.cpp tree, so it runs in about a second — and proves ctest_lane() both passes silently on a passing named test and raises DependencyError naming the failing test on a failing one).

Toolchain note

Neither the ambient system Python nor .venv-rocm has cmake. This session installed cmake (the PyPI wheel that bundles prebuilt binaries, version 4.4.0) into the pre-existing repo-root .venv used by earlier DGR-024/DGR-026 sessions (.venv/bin/cmake, .venv/bin/ctest), which was already on-disk from a prior session but had never had cmake installed into it. All commands below were run with that .venv/bin prepended to PATH. This is the same "disposable venv for a lightweight optional dependency" pattern DGR-024 used for grpc.

Verification — full live reproduce run (fresh out-of-tree build)

$ rm -rf build/llama.cpp/build
$ python3 scripts/llama_cpp_dependency.py reproduce
reused verified offline cache: .../build/llama.cpp/source
usage: .../build/llama.cpp/build/bin/llama-gguf-hash [options] GGUF_IN
Hash a GGUF file
options: ...
Test project .../build/llama.cpp/build
    Start 27: test-meshnet-range-ownership
1/1 Test #27: test-meshnet-range-ownership .....   Passed    0.01 sec
100% tests passed out of 1
$ echo $?
0

Wall-clock: real 2m16.227s (fresh CPU compile of ggml/llama-common/llama plus the llama-gguf-hash example and the test-meshnet-range-ownership fixture; no full llama.cpp test suite or example set is built — only the two targets named in native_targets).

Post-run checks:

$ ls build/llama.cpp/build/bin/*.so*
libggml-base.so libggml-base.so.0 libggml-base.so.0.16.0
libggml-cpu.so   libggml-cpu.so.0   libggml-cpu.so.0.16.0
libggml.so       libggml.so.0       libggml.so.0.16.0
libllama-common.so ... libllama.so ...
# no libggml-cuda*, libggml-hip*, libggml-vulkan*, or libggml-metal* — CPU-only backend built

$ grep -E '^GGML_(CPU|CUDA|HIP|VULKAN|METAL|BLAS):' build/llama.cpp/build/CMakeCache.txt
GGML_BLAS:BOOL=OFF
GGML_CPU:BOOL=ON
GGML_CUDA:BOOL=OFF
GGML_HIP:BOOL=OFF
GGML_METAL:BOOL=OFF
GGML_VULKAN:BOOL=OFF

$ cat build/llama.cpp/build/meshnet-build-metadata.json
{
  "model_downloads": false,
  "semantic_certification": false,
  ...
}

$ git -C build/llama.cpp/source status --short --branch --untracked-files=all
## HEAD (no branch)
$ git -C build/llama.cpp/source rev-parse HEAD HEAD^{tree}
e920c523e3b8a0163fe498af5bf90df35ff51d25
6c91a11407a3a3fb160f5dac705f9c59718f54f1

reproduce()'s final reverse(source) call restored the exact locked pin/tree — the cached workspace is reusable for a subsequent fetch/reproduce without re-cloning.

Verification — actionable toolchain failure (missing cmake)

$ python3 scripts/llama_cpp_dependency.py apply --source-dir build/llama.cpp/source
$ env -i HOME="$HOME" PATH=/usr/bin:/bin python3 scripts/llama_cpp_dependency.py build \
    --source-dir build/llama.cpp/source --build-dir /tmp/no-cmake-build
DGR-027 dependency error: cmake is unavailable; set CMAKE or activate the project toolchain
$ echo $?
2
$ python3 scripts/llama_cpp_dependency.py reverse --source-dir build/llama.cpp/source   # restore pristine

Verification — targeted test suites and shared gates

Command	Result
python3 -m pytest -q tests/test_llama_cpp_dependency.py	9 passed in 1.34s (7 pre-existing + 2 new; the new gated CTest-wiring test ran for real, not skipped, since cmake is present in .venv)
python3 -m pytest -q tests/test_llama_cpp_dependency.py tests/test_ralph_prd_schema.py	117 passed
python3 -m compileall -q packages tests	exit 0
git diff --check	exit 0 (no output)

Ensuring build success does not advertise capability

• The locked configure_flags disable every accelerator backend explicitly (GGML_CUDA/HIP/VULKAN/METAL/BLAS=OFF) rather than relying on per-platform defaults, so a successful configure/build can only ever mean "the CPU reference backend compiled" — never an accelerator claim, and never dependent on whether the build host happens to have a GPU SDK installed.
• meshnet-build-metadata.json (written by build()) records model_downloads: false and semantic_certification: false alongside the exact commit/patch/flag identities — the artifact itself, not just prose, states this build proves toolchain compilation only.
• The two targets actually compiled are llama-gguf-hash (a stock upstream file-hashing utility; no inference) and test-meshnet-range-ownership (a model-free fixture that writes a tiny synthetic GGUF and asserts range-ownership bookkeeping — no real model, no generation, no numerical/backend correctness claim). Neither exercises inference, MoE, attention, or any DeepSeek V4 semantic path.

Changed files

• packages/node/native/llama/UPSTREAM_LOCK.json
• scripts/llama_cpp_dependency.py
• tests/test_llama_cpp_dependency.py
• .scratch/distributed-gguf-runtime/prd.json
• .scratch/distributed-gguf-runtime/issues/029-create-the-native-cmake-skeleton-and-deterministic-cpu-lane.md
• .scratch/distributed-gguf-runtime/evidence/DGR-029/README.md (new)

Limitations

• This is a toolchain/compile-and-link gate plus one model-free ownership-bookkeeping fixture — it proves the CPU lane builds and the DGR-027/DGR-028 patch stack functions structurally on CPU. It proves nothing about real-model correctness, memory-fit, performance, or any backend/model/recipe certification; stock_glm_limitations in UPSTREAM_LOCK.json and DGR-028's own limitations continue to apply unchanged.
• cmake/ctest are not installed system-wide or in .venv-rocm in this environment; they were installed only into the pre-existing repo-root .venv for this session's verification (and for the new gated pytest test, which is skipped in any environment lacking cmake). A future session without that .venv (or without re-installing cmake into it) will see the same "cmake is unavailable" actionable failure demonstrated above, not a silent pass.
• Only the two named targets are compiled (llama-gguf-hash, test-meshnet-range-ownership); a broad cmake --build ... --target test / full upstream test suite is out of scope here, exactly as DGR-028 recorded ("not presented as a full-suite gate").
• CUDA/ROCm/Vulkan/Metal compile lanes remain unimplemented; this story only establishes the CPU lane "before accelerator matrix work," per its objective. Those lanes are separate future work.

Dependency handoff

DGR-030 and DGR-034 (this story's declared blockers) may rely on: an out-of-tree, CPU-only, explicit-backend-flag native build (scripts/llama_cpp_dependency.py build/reproduce) that compiles the exact DGR-027/DGR-028 patched pin and runs a real CTest lane (test-meshnet-range-ownership) proving the patch stack's range-ownership bookkeeping compiles and passes on CPU. Any accelerator (CUDA/ROCm/Vulkan/Metal) lane, any real-model load, and any backend/model/recipe capability certification remain unimplemented and must not be assumed from this story's green build alone.