[verified] feat: harden llama.cpp provenance workspace
This commit is contained in:
72
.scratch/distributed-gguf-runtime/evidence/DGR-027/README.md
Normal file
72
.scratch/distributed-gguf-runtime/evidence/DGR-027/README.md
Normal file
@@ -0,0 +1,72 @@
|
|||||||
|
# DGR-027 evidence — exact llama.cpp provenance manifest and fetch workspace
|
||||||
|
|
||||||
|
**Completed implementation:** 2026-07-17
|
||||||
|
**Branch:** `ralph/dgr-small-terra`
|
||||||
|
**Authority:** live Gitea issue #11. The controller fetched and claimed the issue
|
||||||
|
through the Gitea API before launch; the isolated agent received that exact body.
|
||||||
|
|
||||||
|
## Changed files
|
||||||
|
|
||||||
|
- `packages/node/native/llama/UPSTREAM_LOCK.json`
|
||||||
|
- `packages/node/native/llama/PATCH-STACK.md`
|
||||||
|
- `scripts/llama_cpp_dependency.py`
|
||||||
|
- `tests/test_llama_cpp_dependency.py`
|
||||||
|
- `.scratch/distributed-gguf-runtime/evidence/DGR-027/README.md`
|
||||||
|
|
||||||
|
## Provenance and retrieval contract
|
||||||
|
|
||||||
|
`UPSTREAM_LOCK.json` records the upstream Git URL, immutable 40-character
|
||||||
|
commit `e920c523e3b8a0163fe498af5bf90df35ff51d25`, expected Git tree
|
||||||
|
`6c91a11407a3a3fb160f5dac705f9c59718f54f1`, MIT license, and the sole
|
||||||
|
retrieval method: `git-clone-detached-commit` into `build/llama.cpp/source`.
|
||||||
|
|
||||||
|
`python3 scripts/llama_cpp_dependency.py fetch` has no branch, tag, ref, or
|
||||||
|
repository override. On a first fetch it clones the manifest URL, checks out
|
||||||
|
the detached commit, and verifies commit, tree, required upstream blobs,
|
||||||
|
license, and cleanliness. If the workspace already exists, it makes no network
|
||||||
|
request and accepts it only after the same verification. Dirty or mismatched
|
||||||
|
caches fail closed. The build directory is already ignored by `.gitignore`.
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
|
||||||
|
| Command | Result |
|
||||||
|
| --- | --- |
|
||||||
|
| `python3 -m pytest -q tests/test_llama_cpp_dependency.py` | `7 passed in 0.22s` |
|
||||||
|
| `python3 -m compileall packages tests` | passed |
|
||||||
|
| `git diff --check` | passed (no output) |
|
||||||
|
| `python3 scripts/llama_cpp_dependency.py inspect` | passed; reports exact commit/tree, retrieval workspace, MIT license, and two-patch stack |
|
||||||
|
| `python3 scripts/llama_cpp_dependency.py fetch --workspace /tmp/not-llama-workspace` | failed closed with status 2: workspace outside the locked ignored build root |
|
||||||
|
| symlinked workspace regression | passed; both a `build/` ancestor symlink and a final `source` symlink escaping the repository are refused |
|
||||||
|
| attached-branch cache regression | passed; an exact commit on a local branch is refused until checked out as detached HEAD |
|
||||||
|
| ignored/excluded injection regression | passed; a file hidden by `.git/info/exclude` is detected and refused |
|
||||||
|
| tracked injection regression | passed; modified tracked content hidden by both `assume-unchanged` and `skip-worktree` is content-hashed and refused |
|
||||||
|
| executable-mode regression | passed on the POSIX fixture for both index flags; the mounted project workspace has `core.filemode=false`, so its exact index tree is the canonical mode record and physical mode bits are not treated as meaningful |
|
||||||
|
| `git check-ignore -v build/llama.cpp/source` | passed; `.gitignore:6:build/` |
|
||||||
|
| `git diff --summary` and `git ls-files build packages/node/native/llama` | no source checkout or new submodule introduced; only manifest/docs/patches/native wrapper are tracked |
|
||||||
|
| `python3 scripts/llama_cpp_dependency.py fetch` (controller network lane) | passed; fetched the exact detached commit and verified HEAD `e920c523e3b8a0163fe498af5bf90df35ff51d25` and tree `6c91a11407a3a3fb160f5dac705f9c59718f54f1` in the ignored workspace |
|
||||||
|
| `python3 scripts/llama_cpp_dependency.py apply --source-dir build/llama.cpp/source` | failed on the pre-existing `0002-dense-llama-owned-range-loader.patch` as a corrupt patch at line 26; this is an explicit DGR-028 blocker and no native-build claim is made |
|
||||||
|
|
||||||
|
The targeted test suite creates a local Git fixture to prove offline cache reuse
|
||||||
|
after full identity verification, then proves a dirty cache is rejected. It
|
||||||
|
also proves the CLI rejects a repository/branch override and an arbitrary
|
||||||
|
workspace.
|
||||||
|
|
||||||
|
## Limitations
|
||||||
|
|
||||||
|
- The controller successfully materialized and verified the exact upstream
|
||||||
|
commit/tree, so the DGR-027 fetch and offline-cache boundary has real upstream
|
||||||
|
evidence rather than fixture-only evidence.
|
||||||
|
- The existing `0002-dense-llama-owned-range-loader.patch` is malformed and
|
||||||
|
cannot pass `git apply --check` against the exact pin. DGR-027 changes no patch
|
||||||
|
file; repairing and certifying the numbered patch stack belongs to DGR-028.
|
||||||
|
Until that story closes, the repository must not claim patched-tree, native
|
||||||
|
CMake/CTest, or reverse-apply certification.
|
||||||
|
- No model, API credits, GPU, or model artifact storage was used.
|
||||||
|
|
||||||
|
## Dependency handoff
|
||||||
|
|
||||||
|
DGR-028, DGR-029, and DGR-044 must invoke the manifest-owned `fetch` command
|
||||||
|
before touching llama.cpp source. They may use only the verified
|
||||||
|
`build/llama.cpp/source` checkout and must record any native build, CTest, and
|
||||||
|
patch apply/check/reverse evidence against the exact manifest pin. DGR-017's
|
||||||
|
cleanup remains provenance only and grants no inherited completion credit.
|
||||||
@@ -673,8 +673,8 @@
|
|||||||
"Offline reuse is supported only after the cached tree’s exact identity is verified.",
|
"Offline reuse is supported only after the cached tree’s exact identity is verified.",
|
||||||
"Applicable shared quality gates in `prd.json` pass, and the evidence handoff records exact commands/results, changed files, limitations, and dependency handoff."
|
"Applicable shared quality gates in `prd.json` pass, and the evidence handoff records exact commands/results, changed files, limitations, and dependency handoff."
|
||||||
],
|
],
|
||||||
"passes": false,
|
"passes": true,
|
||||||
"notes": "Generated source issue: .scratch/distributed-gguf-runtime/issues/027-add-exact-llama-cpp-provenance-manifest-and-fetch-workspace.md; prd.json is authoritative.",
|
"notes": "Completed from Gitea #11 via the Terra Ralph lane; independently reviewed after fail-closed cache identity hardening. Verified 7 focused tests, exact real-cache reuse, compileall, and diff-check. The pre-existing malformed 0002 patch is handed to DGR-028.",
|
||||||
"blocks": [
|
"blocks": [
|
||||||
"DGR-028",
|
"DGR-028",
|
||||||
"DGR-029",
|
"DGR-029",
|
||||||
|
|||||||
@@ -15,11 +15,19 @@ loading, endpoint ownership, architecture-defined intermediate boundaries, and
|
|||||||
layer-filtered KV/session mapping. Meshnet routing, Tracker, gRPC, relay,
|
layer-filtered KV/session mapping. Meshnet routing, Tracker, gRPC, relay,
|
||||||
billing, authentication, and telemetry must remain outside this directory.
|
billing, authentication, and telemetry must remain outside this directory.
|
||||||
|
|
||||||
`scripts/llama_cpp_dependency.py` verifies the exact commit/tree and baseline
|
`scripts/llama_cpp_dependency.py fetch` reads the in-repo manifest and checks out
|
||||||
blobs, validates every patch digest and context with `git apply --check`, then
|
only its exact commit as detached HEAD in `build/llama.cpp/source`, an ignored
|
||||||
applies the series in `patches/series` order. It refuses a dirty source tree,
|
build workspace. It verifies the exact commit/tree and baseline blobs before use. A
|
||||||
wrong commit/tree/blob, changed patch digest, reordered series, or an existing
|
later offline `fetch` may reuse that cache only after the same clean identity
|
||||||
destination/work directory.
|
verification; an attached branch, tag/repository override, arbitrary destination,
|
||||||
|
symlinked workspace, dirty checkout, ignored injected file, or tracked-file
|
||||||
|
modification hidden by Git index flags is refused. The tool
|
||||||
|
validates every patch digest and context
|
||||||
|
with `git apply --check`, then applies the series in `patches/series` order.
|
||||||
|
Tracked executable modes are checked physically when Git reports
|
||||||
|
`core.filemode=true`; on mounted filesystems without POSIX mode fidelity, the
|
||||||
|
locked index tree remains the canonical mode record while every working-file
|
||||||
|
blob is independently re-hashed.
|
||||||
|
|
||||||
## Current semantic boundary
|
## Current semantic boundary
|
||||||
|
|
||||||
|
|||||||
@@ -3,6 +3,13 @@
|
|||||||
"upstream": "https://github.com/ggml-org/llama.cpp.git",
|
"upstream": "https://github.com/ggml-org/llama.cpp.git",
|
||||||
"commit": "e920c523e3b8a0163fe498af5bf90df35ff51d25",
|
"commit": "e920c523e3b8a0163fe498af5bf90df35ff51d25",
|
||||||
"commit_tree": "6c91a11407a3a3fb160f5dac705f9c59718f54f1",
|
"commit_tree": "6c91a11407a3a3fb160f5dac705f9c59718f54f1",
|
||||||
|
"expected_source": {
|
||||||
|
"git_tree": "6c91a11407a3a3fb160f5dac705f9c59718f54f1"
|
||||||
|
},
|
||||||
|
"retrieval": {
|
||||||
|
"method": "git-clone-detached-commit",
|
||||||
|
"workspace": "build/llama.cpp"
|
||||||
|
},
|
||||||
"patched_tree": "322d8b463df74a2226f0b513176643d815f54452",
|
"patched_tree": "322d8b463df74a2226f0b513176643d815f54452",
|
||||||
"upstream_license": "MIT",
|
"upstream_license": "MIT",
|
||||||
"patch_series": [
|
"patch_series": [
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
#!/usr/bin/env python3
|
#!/usr/bin/env python3
|
||||||
"""Materialize, verify, build, and smoke-test DGR-004's llama.cpp pin.
|
"""Fetch, verify, build, and smoke-test DGR-027's exact llama.cpp pin.
|
||||||
|
|
||||||
This tool deliberately owns only a source dependency boundary. It never
|
This tool deliberately owns only a source dependency boundary. It never
|
||||||
downloads a model, invokes inference, or interprets generated text.
|
downloads a model, invokes inference, or interprets generated text.
|
||||||
@@ -12,6 +12,7 @@ import hashlib
|
|||||||
import json
|
import json
|
||||||
import os
|
import os
|
||||||
import pathlib
|
import pathlib
|
||||||
|
import re
|
||||||
import shutil
|
import shutil
|
||||||
import subprocess
|
import subprocess
|
||||||
import sys
|
import sys
|
||||||
@@ -67,14 +68,27 @@ def _load_lock() -> dict[str, Any]:
|
|||||||
raise DependencyError(f"invalid upstream lock: {LOCK_PATH}: {error}") from error
|
raise DependencyError(f"invalid upstream lock: {LOCK_PATH}: {error}") from error
|
||||||
required = {
|
required = {
|
||||||
"upstream", "commit", "commit_tree", "patched_tree", "patch_series",
|
"upstream", "commit", "commit_tree", "patched_tree", "patch_series",
|
||||||
"required_upstream_blobs", "patched_paths", "build",
|
"required_upstream_blobs", "patched_paths", "build", "upstream_license",
|
||||||
|
"expected_source", "retrieval",
|
||||||
}
|
}
|
||||||
missing = sorted(required - lock.keys())
|
missing = sorted(required - lock.keys())
|
||||||
if missing:
|
if missing:
|
||||||
raise DependencyError(f"upstream lock is missing fields: {', '.join(missing)}")
|
raise DependencyError(f"upstream lock is missing fields: {', '.join(missing)}")
|
||||||
commit_file = (LLAMA_DIR / "UPSTREAM_COMMIT").read_text().strip()
|
commit_file = (LLAMA_DIR / "UPSTREAM_COMMIT").read_text().strip()
|
||||||
if lock["commit"] != commit_file or len(commit_file) != 40:
|
object_ids = [commit_file, lock["commit"], lock["commit_tree"], lock["patched_tree"]]
|
||||||
raise DependencyError("UPSTREAM_COMMIT and UPSTREAM_LOCK.json do not agree on a full commit")
|
if lock["commit"] != commit_file or not all(
|
||||||
|
isinstance(value, str) and re.fullmatch(r"[0-9a-f]{40}", value)
|
||||||
|
for value in object_ids
|
||||||
|
):
|
||||||
|
raise DependencyError("UPSTREAM_COMMIT and UPSTREAM_LOCK.json do not agree on full hexadecimal object IDs")
|
||||||
|
if lock["expected_source"] != {"git_tree": lock["commit_tree"]}:
|
||||||
|
raise DependencyError("expected_source must record the locked git tree")
|
||||||
|
retrieval = lock["retrieval"]
|
||||||
|
if retrieval != {
|
||||||
|
"method": "git-clone-detached-commit",
|
||||||
|
"workspace": "build/llama.cpp",
|
||||||
|
}:
|
||||||
|
raise DependencyError("retrieval must use the locked detached-commit build workspace")
|
||||||
return lock
|
return lock
|
||||||
|
|
||||||
|
|
||||||
@@ -103,6 +117,56 @@ def _git(source: pathlib.Path, *args: str) -> str:
|
|||||||
return _run("git", "-C", str(source), *args)
|
return _run("git", "-C", str(source), *args)
|
||||||
|
|
||||||
|
|
||||||
|
def _verify_tracked_content(source: pathlib.Path, lock: dict[str, Any]) -> None:
|
||||||
|
if _git(source, "write-tree") != lock["commit_tree"]:
|
||||||
|
raise DependencyError("materialized checkout index differs from the locked tree")
|
||||||
|
filemode_trusted = _git(source, "config", "--bool", "core.filemode") == "true"
|
||||||
|
records = _git(source, "ls-files", "-s", "-z").split("\0")
|
||||||
|
paths: list[str] = []
|
||||||
|
expected: list[str] = []
|
||||||
|
for record in records:
|
||||||
|
if not record:
|
||||||
|
continue
|
||||||
|
metadata, path = record.split("\t", 1)
|
||||||
|
mode, blob, stage = metadata.split()
|
||||||
|
if stage != "0" or mode not in {"100644", "100755", "120000"}:
|
||||||
|
raise DependencyError(f"unsupported tracked entry in materialized checkout: {record!r}")
|
||||||
|
if "\n" in path:
|
||||||
|
raise DependencyError(f"newline-bearing tracked path is unsupported: {path!r}")
|
||||||
|
candidate = source / path
|
||||||
|
cursor = source
|
||||||
|
for part in pathlib.Path(path).parts[:-1]:
|
||||||
|
cursor = cursor / part
|
||||||
|
if cursor.is_symlink():
|
||||||
|
raise DependencyError(f"tracked path traverses a symlink: {path!r}")
|
||||||
|
if mode == "120000":
|
||||||
|
if not candidate.is_symlink():
|
||||||
|
raise DependencyError(f"tracked symlink type differs from the locked tree: {path!r}")
|
||||||
|
else:
|
||||||
|
if candidate.is_symlink() or not candidate.is_file():
|
||||||
|
raise DependencyError(f"tracked file type differs from the locked tree: {path!r}")
|
||||||
|
executable = bool(candidate.stat().st_mode & 0o111)
|
||||||
|
if filemode_trusted and executable != (mode == "100755"):
|
||||||
|
raise DependencyError(f"tracked executable mode differs from the locked tree: {path!r}")
|
||||||
|
paths.append(path)
|
||||||
|
expected.append(blob)
|
||||||
|
try:
|
||||||
|
completed = subprocess.run(
|
||||||
|
["git", "hash-object", "--stdin-paths"],
|
||||||
|
cwd=source,
|
||||||
|
input="".join(f"{path}\n" for path in paths),
|
||||||
|
check=True,
|
||||||
|
text=True,
|
||||||
|
stdout=subprocess.PIPE,
|
||||||
|
stderr=subprocess.PIPE,
|
||||||
|
)
|
||||||
|
except subprocess.CalledProcessError as error:
|
||||||
|
raise DependencyError(f"unable to hash tracked source content: {error.stderr.strip()}") from error
|
||||||
|
actual = completed.stdout.splitlines()
|
||||||
|
if len(actual) != len(expected) or actual != expected:
|
||||||
|
raise DependencyError("tracked source content differs from the locked tree")
|
||||||
|
|
||||||
|
|
||||||
def _verify_source(source: pathlib.Path, lock: dict[str, Any], *, require_clean: bool) -> None:
|
def _verify_source(source: pathlib.Path, lock: dict[str, Any], *, require_clean: bool) -> None:
|
||||||
if not (source / ".git").exists():
|
if not (source / ".git").exists():
|
||||||
raise DependencyError(f"not a materialized git checkout: {source}")
|
raise DependencyError(f"not a materialized git checkout: {source}")
|
||||||
@@ -110,8 +174,15 @@ def _verify_source(source: pathlib.Path, lock: dict[str, Any], *, require_clean:
|
|||||||
raise DependencyError("upstream drift: checkout HEAD does not equal the locked commit")
|
raise DependencyError("upstream drift: checkout HEAD does not equal the locked commit")
|
||||||
if _git(source, "rev-parse", "HEAD^{tree}") != lock["commit_tree"]:
|
if _git(source, "rev-parse", "HEAD^{tree}") != lock["commit_tree"]:
|
||||||
raise DependencyError("upstream drift: checkout tree does not equal the locked tree")
|
raise DependencyError("upstream drift: checkout tree does not equal the locked tree")
|
||||||
if require_clean and _git(source, "status", "--porcelain"):
|
if _git(source, "rev-parse", "--abbrev-ref", "HEAD") != "HEAD":
|
||||||
raise DependencyError("local edits detected in materialized llama.cpp checkout")
|
raise DependencyError("materialized llama.cpp checkout must have a detached HEAD")
|
||||||
|
if require_clean:
|
||||||
|
_verify_tracked_content(source, lock)
|
||||||
|
tracked = _git(source, "status", "--porcelain", "--untracked-files=no")
|
||||||
|
untracked = _git(source, "ls-files", "--others", "--exclude-standard")
|
||||||
|
ignored = _git(source, "ls-files", "--others", "--ignored", "--exclude-standard")
|
||||||
|
if tracked or untracked or ignored:
|
||||||
|
raise DependencyError("local edits or unmanifested files detected in materialized llama.cpp checkout")
|
||||||
for relative, expected in lock["required_upstream_blobs"].items():
|
for relative, expected in lock["required_upstream_blobs"].items():
|
||||||
actual = _git(source, "rev-parse", f"HEAD:{relative}")
|
actual = _git(source, "rev-parse", f"HEAD:{relative}")
|
||||||
if actual != expected:
|
if actual != expected:
|
||||||
@@ -122,15 +193,43 @@ def _verify_source(source: pathlib.Path, lock: dict[str, Any], *, require_clean:
|
|||||||
raise DependencyError("upstream LICENSE is missing; refusing to drop required attribution")
|
raise DependencyError("upstream LICENSE is missing; refusing to drop required attribution")
|
||||||
|
|
||||||
|
|
||||||
def materialize(source: pathlib.Path, repository: str) -> None:
|
def _workspace_source(workspace: pathlib.Path, lock: dict[str, Any]) -> pathlib.Path:
|
||||||
|
relative = pathlib.Path(lock["retrieval"]["workspace"])
|
||||||
|
expected = (ROOT / relative).absolute()
|
||||||
|
supplied = workspace.absolute()
|
||||||
|
if supplied != expected:
|
||||||
|
raise DependencyError(f"--workspace must equal the locked ignored build root: {expected}")
|
||||||
|
cursor = ROOT
|
||||||
|
for part in relative.parts:
|
||||||
|
cursor = cursor / part
|
||||||
|
if cursor.is_symlink():
|
||||||
|
raise DependencyError(f"locked build workspace may not traverse a symlink: {cursor}")
|
||||||
|
resolved = workspace.resolve()
|
||||||
|
try:
|
||||||
|
resolved.relative_to(ROOT.resolve())
|
||||||
|
except ValueError as error:
|
||||||
|
raise DependencyError("locked build workspace escapes the repository root") from error
|
||||||
|
source = resolved / "source"
|
||||||
|
if source.is_symlink():
|
||||||
|
raise DependencyError(f"locked source checkout may not be a symlink: {source}")
|
||||||
|
return source
|
||||||
|
|
||||||
|
|
||||||
|
def fetch(workspace: pathlib.Path) -> pathlib.Path:
|
||||||
|
"""Fetch once, or verify an exact clean cached checkout for offline reuse."""
|
||||||
lock = _load_lock()
|
lock = _load_lock()
|
||||||
_patches(lock)
|
_patches(lock)
|
||||||
|
source = _workspace_source(workspace, lock)
|
||||||
if source.exists():
|
if source.exists():
|
||||||
raise DependencyError(f"destination already exists; refusing to reuse possibly edited source: {source}")
|
_verify_source(source, lock, require_clean=True)
|
||||||
|
print(f"reused verified offline cache: {source}")
|
||||||
|
return source
|
||||||
source.parent.mkdir(parents=True, exist_ok=True)
|
source.parent.mkdir(parents=True, exist_ok=True)
|
||||||
_run("git", "clone", "--no-checkout", repository, str(source))
|
_run("git", "clone", "--no-checkout", lock["upstream"], str(source))
|
||||||
_git(source, "checkout", "--detach", lock["commit"])
|
_git(source, "checkout", "--detach", lock["commit"])
|
||||||
_verify_source(source, lock, require_clean=True)
|
_verify_source(source, lock, require_clean=True)
|
||||||
|
print(f"fetched and verified exact source: {source}")
|
||||||
|
return source
|
||||||
|
|
||||||
|
|
||||||
def apply(source: pathlib.Path) -> None:
|
def apply(source: pathlib.Path) -> None:
|
||||||
@@ -198,18 +297,11 @@ def smoke(binary: pathlib.Path) -> None:
|
|||||||
print(output)
|
print(output)
|
||||||
|
|
||||||
|
|
||||||
def reproduce(work_dir: pathlib.Path, repository: str) -> None:
|
def reproduce(workspace: pathlib.Path) -> None:
|
||||||
resolved = work_dir.resolve()
|
source = fetch(workspace)
|
||||||
build_root = (ROOT / "build").resolve()
|
build_dir = workspace.resolve() / "build"
|
||||||
if build_root not in resolved.parents:
|
if build_dir.exists():
|
||||||
raise DependencyError(f"--work-dir must be below {build_root}: {resolved}")
|
raise DependencyError(f"build directory already exists; refusing to erase possible local edits: {build_dir}")
|
||||||
if resolved.exists():
|
|
||||||
raise DependencyError(
|
|
||||||
f"work directory already exists; refusing to erase possible local edits: {resolved}"
|
|
||||||
)
|
|
||||||
source = resolved / "source"
|
|
||||||
build_dir = resolved / "build"
|
|
||||||
materialize(source, repository)
|
|
||||||
apply(source)
|
apply(source)
|
||||||
smoke(build(source, build_dir))
|
smoke(build(source, build_dir))
|
||||||
|
|
||||||
@@ -219,6 +311,9 @@ def inspect() -> None:
|
|||||||
patches = _patches(lock)
|
patches = _patches(lock)
|
||||||
print(json.dumps({
|
print(json.dumps({
|
||||||
"commit": lock["commit"],
|
"commit": lock["commit"],
|
||||||
|
"commit_tree": lock["commit_tree"],
|
||||||
|
"retrieval": lock["retrieval"],
|
||||||
|
"upstream_license": lock["upstream_license"],
|
||||||
"patch_count": len(patches),
|
"patch_count": len(patches),
|
||||||
"patches": [patch.name for patch in patches],
|
"patches": [patch.name for patch in patches],
|
||||||
"model_downloads": False,
|
"model_downloads": False,
|
||||||
@@ -231,9 +326,8 @@ def main() -> int:
|
|||||||
parser = argparse.ArgumentParser(description=__doc__)
|
parser = argparse.ArgumentParser(description=__doc__)
|
||||||
subcommands = parser.add_subparsers(dest="command", required=True)
|
subcommands = parser.add_subparsers(dest="command", required=True)
|
||||||
subcommands.add_parser("inspect")
|
subcommands.add_parser("inspect")
|
||||||
materialize_parser = subcommands.add_parser("materialize")
|
fetch_parser = subcommands.add_parser("fetch")
|
||||||
materialize_parser.add_argument("--source-dir", type=pathlib.Path, required=True)
|
fetch_parser.add_argument("--workspace", type=pathlib.Path, default=ROOT / "build/llama.cpp")
|
||||||
materialize_parser.add_argument("--source-repository", default=_load_lock()["upstream"])
|
|
||||||
apply_parser = subcommands.add_parser("apply")
|
apply_parser = subcommands.add_parser("apply")
|
||||||
apply_parser.add_argument("--source-dir", type=pathlib.Path, required=True)
|
apply_parser.add_argument("--source-dir", type=pathlib.Path, required=True)
|
||||||
build_parser = subcommands.add_parser("build")
|
build_parser = subcommands.add_parser("build")
|
||||||
@@ -242,14 +336,13 @@ def main() -> int:
|
|||||||
smoke_parser = subcommands.add_parser("smoke")
|
smoke_parser = subcommands.add_parser("smoke")
|
||||||
smoke_parser.add_argument("--binary", type=pathlib.Path, required=True)
|
smoke_parser.add_argument("--binary", type=pathlib.Path, required=True)
|
||||||
reproduce_parser = subcommands.add_parser("reproduce")
|
reproduce_parser = subcommands.add_parser("reproduce")
|
||||||
reproduce_parser.add_argument("--work-dir", type=pathlib.Path, default=ROOT / "build/dgr-004-smoke")
|
reproduce_parser.add_argument("--workspace", type=pathlib.Path, default=ROOT / "build/llama.cpp")
|
||||||
reproduce_parser.add_argument("--source-repository", default=_load_lock()["upstream"])
|
|
||||||
args = parser.parse_args()
|
args = parser.parse_args()
|
||||||
try:
|
try:
|
||||||
if args.command == "inspect":
|
if args.command == "inspect":
|
||||||
inspect()
|
inspect()
|
||||||
elif args.command == "materialize":
|
elif args.command == "fetch":
|
||||||
materialize(args.source_dir, args.source_repository)
|
fetch(args.workspace)
|
||||||
elif args.command == "apply":
|
elif args.command == "apply":
|
||||||
apply(args.source_dir)
|
apply(args.source_dir)
|
||||||
elif args.command == "build":
|
elif args.command == "build":
|
||||||
@@ -257,9 +350,9 @@ def main() -> int:
|
|||||||
elif args.command == "smoke":
|
elif args.command == "smoke":
|
||||||
smoke(args.binary)
|
smoke(args.binary)
|
||||||
else:
|
else:
|
||||||
reproduce(args.work_dir, args.source_repository)
|
reproduce(args.workspace)
|
||||||
except DependencyError as error:
|
except DependencyError as error:
|
||||||
print(f"DGR-004 dependency error: {error}", file=sys.stderr)
|
print(f"DGR-027 dependency error: {error}", file=sys.stderr)
|
||||||
return 2
|
return 2
|
||||||
return 0
|
return 0
|
||||||
|
|
||||||
|
|||||||
@@ -1,8 +1,9 @@
|
|||||||
"""Offline guards for DGR-004's pinned llama.cpp dependency boundary."""
|
"""Offline guards for DGR-027's pinned llama.cpp dependency boundary."""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import hashlib
|
import hashlib
|
||||||
|
import importlib.util
|
||||||
import json
|
import json
|
||||||
import pathlib
|
import pathlib
|
||||||
import subprocess
|
import subprocess
|
||||||
@@ -33,6 +34,10 @@ def test_lock_and_patch_manifest_are_self_consistent_and_exact() -> None:
|
|||||||
|
|
||||||
assert commit == lock["commit"]
|
assert commit == lock["commit"]
|
||||||
assert len(commit) == 40
|
assert len(commit) == 40
|
||||||
|
assert lock["retrieval"]["method"] == "git-clone-detached-commit"
|
||||||
|
assert lock["retrieval"]["workspace"] == "build/llama.cpp"
|
||||||
|
assert lock["expected_source"]["git_tree"] == lock["commit_tree"]
|
||||||
|
assert lock["upstream_license"] == "MIT"
|
||||||
assert patches == lock["patch_series"]
|
assert patches == lock["patch_series"]
|
||||||
assert patches == sorted(patches)
|
assert patches == sorted(patches)
|
||||||
assert patches
|
assert patches
|
||||||
@@ -42,6 +47,188 @@ def test_lock_and_patch_manifest_are_self_consistent_and_exact() -> None:
|
|||||||
assert "Subject: [PATCH" in patch.read_text()
|
assert "Subject: [PATCH" in patch.read_text()
|
||||||
|
|
||||||
|
|
||||||
|
def test_fetch_refuses_a_workspace_outside_the_ignored_build_root(tmp_path: pathlib.Path) -> None:
|
||||||
|
completed = subprocess.run(
|
||||||
|
[sys.executable, str(SCRIPT), "fetch", "--workspace", str(tmp_path / "llama.cpp")],
|
||||||
|
cwd=ROOT,
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert completed.returncode == 2
|
||||||
|
assert "--workspace must equal" in completed.stderr
|
||||||
|
|
||||||
|
|
||||||
|
def test_workspace_refuses_a_symlinked_build_ancestor(tmp_path: pathlib.Path, monkeypatch) -> None:
|
||||||
|
spec = importlib.util.spec_from_file_location("llama_cpp_dependency_symlink", SCRIPT)
|
||||||
|
assert spec and spec.loader
|
||||||
|
dependency = importlib.util.module_from_spec(spec)
|
||||||
|
spec.loader.exec_module(dependency)
|
||||||
|
root = tmp_path / "repo"
|
||||||
|
outside = tmp_path / "outside"
|
||||||
|
root.mkdir()
|
||||||
|
outside.mkdir()
|
||||||
|
(root / "build").symlink_to(outside, target_is_directory=True)
|
||||||
|
monkeypatch.setattr(dependency, "ROOT", root)
|
||||||
|
|
||||||
|
try:
|
||||||
|
dependency._workspace_source(
|
||||||
|
root / "build/llama.cpp",
|
||||||
|
{"retrieval": {"workspace": "build/llama.cpp"}},
|
||||||
|
)
|
||||||
|
except dependency.DependencyError as error:
|
||||||
|
assert "may not traverse a symlink" in str(error)
|
||||||
|
else:
|
||||||
|
raise AssertionError("symlinked build ancestor must be refused")
|
||||||
|
|
||||||
|
(root / "build").unlink()
|
||||||
|
workspace = root / "build/llama.cpp"
|
||||||
|
workspace.mkdir(parents=True)
|
||||||
|
(workspace / "source").symlink_to(outside, target_is_directory=True)
|
||||||
|
try:
|
||||||
|
dependency._workspace_source(
|
||||||
|
workspace,
|
||||||
|
{"retrieval": {"workspace": "build/llama.cpp"}},
|
||||||
|
)
|
||||||
|
except dependency.DependencyError as error:
|
||||||
|
assert "source checkout may not be a symlink" in str(error)
|
||||||
|
else:
|
||||||
|
raise AssertionError("symlinked source checkout must be refused")
|
||||||
|
|
||||||
|
|
||||||
|
def test_fetch_refuses_a_branch_or_repository_override() -> None:
|
||||||
|
completed = subprocess.run(
|
||||||
|
[sys.executable, str(SCRIPT), "fetch", "--source-repository", "main"],
|
||||||
|
cwd=ROOT,
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert completed.returncode == 2
|
||||||
|
assert "unrecognized arguments" in completed.stderr
|
||||||
|
|
||||||
|
|
||||||
|
def test_fetch_reuses_only_a_verified_cached_tree_offline(
|
||||||
|
tmp_path: pathlib.Path, monkeypatch
|
||||||
|
) -> None:
|
||||||
|
spec = importlib.util.spec_from_file_location("llama_cpp_dependency", SCRIPT)
|
||||||
|
assert spec and spec.loader
|
||||||
|
dependency = importlib.util.module_from_spec(spec)
|
||||||
|
spec.loader.exec_module(dependency)
|
||||||
|
|
||||||
|
root = tmp_path / "repo"
|
||||||
|
source = root / "build/llama.cpp/source"
|
||||||
|
upstream = tmp_path / "upstream"
|
||||||
|
upstream.mkdir()
|
||||||
|
subprocess.run(["git", "init", "-q", str(upstream)], check=True)
|
||||||
|
subprocess.run(["git", "-C", str(upstream), "config", "user.email", "test@example.invalid"], check=True)
|
||||||
|
subprocess.run(["git", "-C", str(upstream), "config", "user.name", "test"], check=True)
|
||||||
|
(upstream / "CMakeLists.txt").write_text("cmake_minimum_required(VERSION 3.14)\n")
|
||||||
|
(upstream / "LICENSE").write_text("MIT\n")
|
||||||
|
(upstream / "tool.sh").write_text("#!/bin/sh\nexit 0\n")
|
||||||
|
(upstream / "tool.sh").chmod(0o755)
|
||||||
|
subprocess.run(["git", "-C", str(upstream), "add", "."], check=True)
|
||||||
|
subprocess.run(["git", "-C", str(upstream), "commit", "-qm", "fixture"], check=True)
|
||||||
|
commit = subprocess.run(
|
||||||
|
["git", "-C", str(upstream), "rev-parse", "HEAD"], check=True, capture_output=True, text=True
|
||||||
|
).stdout.strip()
|
||||||
|
tree = subprocess.run(
|
||||||
|
["git", "-C", str(upstream), "rev-parse", "HEAD^{tree}"], check=True, capture_output=True, text=True
|
||||||
|
).stdout.strip()
|
||||||
|
blob = subprocess.run(
|
||||||
|
["git", "-C", str(upstream), "rev-parse", "HEAD:CMakeLists.txt"],
|
||||||
|
check=True,
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
|
).stdout.strip()
|
||||||
|
source.parent.mkdir(parents=True)
|
||||||
|
subprocess.run(["git", "clone", "-q", str(upstream), str(source)], check=True)
|
||||||
|
|
||||||
|
llama_dir = root / "packages/node/native/llama"
|
||||||
|
patch_dir = llama_dir / "patches"
|
||||||
|
patch_dir.mkdir(parents=True)
|
||||||
|
(llama_dir / "UPSTREAM_COMMIT").write_text(f"{commit}\n")
|
||||||
|
(patch_dir / "series").write_text("0001-fixture.patch\n")
|
||||||
|
patch = patch_dir / "0001-fixture.patch"
|
||||||
|
patch.write_text("fixture patch\n")
|
||||||
|
(patch_dir / "SHA256SUMS").write_text(f"{_sha256(patch)} {patch.name}\n")
|
||||||
|
(llama_dir / "UPSTREAM_LOCK.json").write_text(json.dumps({
|
||||||
|
"upstream": str(upstream),
|
||||||
|
"commit": commit,
|
||||||
|
"commit_tree": tree,
|
||||||
|
"expected_source": {"git_tree": tree},
|
||||||
|
"retrieval": {"method": "git-clone-detached-commit", "workspace": "build/llama.cpp"},
|
||||||
|
"patched_tree": tree,
|
||||||
|
"patch_series": [patch.name],
|
||||||
|
"required_upstream_blobs": {"CMakeLists.txt": blob},
|
||||||
|
"patched_paths": [],
|
||||||
|
"build": {},
|
||||||
|
"upstream_license": "MIT",
|
||||||
|
}))
|
||||||
|
monkeypatch.setattr(dependency, "ROOT", root)
|
||||||
|
monkeypatch.setattr(dependency, "LLAMA_DIR", llama_dir)
|
||||||
|
monkeypatch.setattr(dependency, "LOCK_PATH", llama_dir / "UPSTREAM_LOCK.json")
|
||||||
|
monkeypatch.setattr(dependency, "PATCH_DIR", patch_dir)
|
||||||
|
|
||||||
|
try:
|
||||||
|
dependency.fetch(root / "build/llama.cpp")
|
||||||
|
except dependency.DependencyError as error:
|
||||||
|
assert "detached HEAD" in str(error)
|
||||||
|
else:
|
||||||
|
raise AssertionError("attached branch cache must be refused")
|
||||||
|
subprocess.run(["git", "-C", str(source), "checkout", "--detach", "-q", commit], check=True)
|
||||||
|
assert dependency.fetch(root / "build/llama.cpp") == source
|
||||||
|
|
||||||
|
tracked_path = source / "CMakeLists.txt"
|
||||||
|
for flag, clear_flag in (
|
||||||
|
("--assume-unchanged", "--no-assume-unchanged"),
|
||||||
|
("--skip-worktree", "--no-skip-worktree"),
|
||||||
|
):
|
||||||
|
subprocess.run(["git", "-C", str(source), "update-index", flag, "CMakeLists.txt"], check=True)
|
||||||
|
tracked_path.write_text("injected tracked build input\n")
|
||||||
|
try:
|
||||||
|
dependency.fetch(root / "build/llama.cpp")
|
||||||
|
except dependency.DependencyError as error:
|
||||||
|
assert "tracked source content differs" in str(error)
|
||||||
|
else:
|
||||||
|
raise AssertionError(f"tracked cache injection hidden by {flag} must be refused")
|
||||||
|
subprocess.run(["git", "-C", str(source), "update-index", clear_flag, "CMakeLists.txt"], check=True)
|
||||||
|
subprocess.run(["git", "-C", str(source), "checkout", "--", "CMakeLists.txt"], check=True)
|
||||||
|
|
||||||
|
executable_path = source / "tool.sh"
|
||||||
|
for flag, clear_flag in (
|
||||||
|
("--assume-unchanged", "--no-assume-unchanged"),
|
||||||
|
("--skip-worktree", "--no-skip-worktree"),
|
||||||
|
):
|
||||||
|
subprocess.run(["git", "-C", str(source), "update-index", flag, "tool.sh"], check=True)
|
||||||
|
executable_path.chmod(0o644)
|
||||||
|
try:
|
||||||
|
dependency.fetch(root / "build/llama.cpp")
|
||||||
|
except dependency.DependencyError as error:
|
||||||
|
assert "executable mode differs" in str(error)
|
||||||
|
else:
|
||||||
|
raise AssertionError(f"tracked mode change hidden by {flag} must be refused")
|
||||||
|
subprocess.run(["git", "-C", str(source), "update-index", clear_flag, "tool.sh"], check=True)
|
||||||
|
subprocess.run(["git", "-C", str(source), "checkout", "--", "tool.sh"], check=True)
|
||||||
|
|
||||||
|
(source / "untracked.txt").write_text("edited\n")
|
||||||
|
try:
|
||||||
|
dependency.fetch(root / "build/llama.cpp")
|
||||||
|
except dependency.DependencyError as error:
|
||||||
|
assert "local edits" in str(error)
|
||||||
|
else:
|
||||||
|
raise AssertionError("dirty cached source must be refused")
|
||||||
|
(source / "untracked.txt").unlink()
|
||||||
|
(source / ".git/info/exclude").write_text("injected.cmake\n")
|
||||||
|
(source / "injected.cmake").write_text("unmanifested input\n")
|
||||||
|
try:
|
||||||
|
dependency.fetch(root / "build/llama.cpp")
|
||||||
|
except dependency.DependencyError as error:
|
||||||
|
assert "unmanifested files" in str(error)
|
||||||
|
else:
|
||||||
|
raise AssertionError("ignored cached source input must be refused")
|
||||||
|
|
||||||
|
|
||||||
def test_dependency_script_reports_the_locked_boundary_without_network() -> None:
|
def test_dependency_script_reports_the_locked_boundary_without_network() -> None:
|
||||||
completed = subprocess.run(
|
completed = subprocess.run(
|
||||||
[sys.executable, str(SCRIPT), "inspect"],
|
[sys.executable, str(SCRIPT), "inspect"],
|
||||||
|
|||||||
Reference in New Issue
Block a user