Compare commits
7 Commits
520bb60f5f
...
ralph/terr
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
902ecde363 | ||
|
|
db59caa8e9 | ||
|
|
ad66f7a4d8 | ||
|
|
f83cf331c3 | ||
|
|
ae51526e85 | ||
|
|
8563d218c9 | ||
|
|
66d9888a11 |
@@ -44,6 +44,10 @@ Historical handoff note: `/mnt/c/Users/popov/Downloads/neuron-tai-alpha-handoff-
|
|||||||
|
|
||||||
Planning is ready at `.scratch/node-capability-admission/` with five sequential Ralph stories and ADR-0023. The design is model-agnostic: a Node must validate its selected Model Artifact/shard with a bounded real forward before Tracker routing; Qwen3.6 is only an optional development fixture. P0 adds a versioned local recipe-manifest/report contract, `meshnet-node doctor`, fail-closed startup admission, and tracker route gating. It intentionally excludes dynamic recipe/dependency installation and the future signed Node updater.
|
Planning is ready at `.scratch/node-capability-admission/` with five sequential Ralph stories and ADR-0023. The design is model-agnostic: a Node must validate its selected Model Artifact/shard with a bounded real forward before Tracker routing; Qwen3.6 is only an optional development fixture. P0 adds a versioned local recipe-manifest/report contract, `meshnet-node doctor`, fail-closed startup admission, and tracker route gating. It intentionally excludes dynamic recipe/dependency installation and the future signed Node updater.
|
||||||
|
|
||||||
|
## Gitea DGR sync (2026-07-17)
|
||||||
|
|
||||||
|
Gitea is ahead of the local Markdown backlog with open DGR-022..DGR-071. The first executable P0 dependency frontier is DGR-022 (Shard lifecycle and structured status RPCs), DGR-023 (reproducible protobuf generation), DGR-025 (artifact/runtime recipe identity), and DGR-027 (llama.cpp provenance manifest). DGR-021, the named-tensor stream envelope prerequisite for DGR-022/023/025, is closed. DGR-022 is the next dependency-ordered issue and blocks DGR-024, DGR-033, and DGR-037.
|
||||||
|
|
||||||
## Windows CUDA node (working as of 2026-07-01)
|
## Windows CUDA node (working as of 2026-07-01)
|
||||||
- miniforge3 base env, torch 2.7.1+cu118, torchvision 0.22.x+cu118
|
- miniforge3 base env, torch 2.7.1+cu118, torchvision 0.22.x+cu118
|
||||||
- RTX 4060 Laptop GPU, 8 GB VRAM, benchmark index ~11,200
|
- RTX 4060 Laptop GPU, 8 GB VRAM, benchmark index ~11,200
|
||||||
|
|||||||
126
.scratch/distributed-gguf-runtime/evidence/DGR-023/README.md
Normal file
126
.scratch/distributed-gguf-runtime/evidence/DGR-023/README.md
Normal file
@@ -0,0 +1,126 @@
|
|||||||
|
# DGR-023 evidence — reproducible Python and C++ protobuf/gRPC generation
|
||||||
|
|
||||||
|
**Status:** complete after controller verification and independent-review repairs on 2026-07-17.
|
||||||
|
|
||||||
|
**Authority:** live Gitea issue #7. The local PRD is a secondary projection.
|
||||||
|
|
||||||
|
## Implemented contract
|
||||||
|
|
||||||
|
- Python generation requires exactly `grpcio-tools==1.82.1`; the generator checks installed distribution metadata and rejects missing or different versions with an actionable exact install command.
|
||||||
|
- The C++ bootstrap builds one ignored toolchain prefix from exact inputs:
|
||||||
|
- Protobuf release `33.1` (`protobuf-config` version `33.1.0`);
|
||||||
|
- Abseil release `20250814.1`;
|
||||||
|
- gRPC C++ `1.82.1` at commit `acccf84c0df20487d64101f528e5d426541ca4e5`;
|
||||||
|
- gRPC's exact-commit submodules for c-ares, RE2, OpenSSL, and zlib.
|
||||||
|
- Protobuf is configured with local dependencies only after the exact Abseil build. gRPC uses the installed Protobuf/Abseil packages and commit-pinned module dependencies, avoiding unpinned system development packages and download fallbacks.
|
||||||
|
- CMake requires exact Protobuf `33.1.0` and gRPC `1.82.1`, requires the exported `gRPC::grpc_cpp_plugin` target, and always generates/builds both message and service stubs in the ignored build tree.
|
||||||
|
- Python bindings remain committed package output; `--check` regenerates into a temporary directory and compares output. C++ bindings are never committed.
|
||||||
|
- The C++ conformance test parses Python-produced vectors, validates fields/CRC32C, and emits `cpp_roundtrip.binpb`; Python compares that artifact byte-for-byte.
|
||||||
|
|
||||||
|
## Defects found and fixed
|
||||||
|
|
||||||
|
1. A relative bootstrap prefix was resolved after entering the temporary source directory, so successful output was deleted by cleanup. The script now canonicalizes the caller-relative destination first. The regression executes `--print-prefix` from a temporary working directory and validates the resulting path behavior.
|
||||||
|
2. The original native path omitted gRPC C++ and accepted any discoverable plugin. The bootstrap now builds exact gRPC/plugin sources, and CMake rejects absent/incompatible versions.
|
||||||
|
3. The Python script named the `grpcio-tools` pin but did not validate the installed distribution. It now refuses mismatched versions.
|
||||||
|
4. Protobuf ignored a stale provider option and attempted to download a different Abseil. The build was stopped; exact Abseil is now built first and Protobuf uses `LOCAL_DEPENDENCIES_ONLY`.
|
||||||
|
5. The host lacked OpenSSL development headers. Rather than add a floating system dependency, gRPC now uses the submodule pinned by its exact commit.
|
||||||
|
6. Documentation uses `bash scripts/bootstrap_native_toolchain.sh ...`, so a normal checkout does not depend on executable-mode preservation.
|
||||||
|
|
||||||
|
## Verified toolchain
|
||||||
|
|
||||||
|
```text
|
||||||
|
cmake version 4.4.0
|
||||||
|
c++ (GCC) 15.2.1 20260123 (Red Hat 15.2.1-7)
|
||||||
|
libprotoc 33.1
|
||||||
|
protobuf CMake package 33.1.0
|
||||||
|
grpcio-tools 1.82.1
|
||||||
|
grpcio 1.82.1
|
||||||
|
protobuf Python runtime 7.35.1
|
||||||
|
gRPC C++ 1.82.1
|
||||||
|
commit acccf84c0df20487d64101f528e5d426541ca4e5
|
||||||
|
grpc_cpp_plugin sha256 995ca8ac620fe83532b649a7c8c0a9341c7003da927fe0e4a8f821bfc579206d
|
||||||
|
```
|
||||||
|
|
||||||
|
The native toolchain and generated/build artifacts live under ignored mounted-drive `build/` paths; model/build artifacts were not stored under `/home`.
|
||||||
|
|
||||||
|
## Commands and results
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bash scripts/bootstrap_native_toolchain.sh build/native-toolchain
|
||||||
|
```
|
||||||
|
|
||||||
|
```text
|
||||||
|
passed from a clean build directory
|
||||||
|
libprotoc 33.1
|
||||||
|
gRPC 1.82.1 commit acccf84c0df20487d64101f528e5d426541ca4e5
|
||||||
|
grpc_cpp_plugin sha256 995ca8ac620fe83532b649a7c8c0a9341c7003da927fe0e4a8f821bfc579206d
|
||||||
|
```
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cmake -S packages/node/native -B build/native \
|
||||||
|
-DCMAKE_PREFIX_PATH="$PWD/build/native-toolchain"
|
||||||
|
cmake --build build/native -j"$(nproc)"
|
||||||
|
test -f build/native/shard_runtime.grpc.pb.cc
|
||||||
|
test -f build/native/shard_runtime.grpc.pb.h
|
||||||
|
test -f build/native/libshard_runtime_grpc.a
|
||||||
|
ctest --test-dir build/native --output-on-failure
|
||||||
|
```
|
||||||
|
|
||||||
|
```text
|
||||||
|
Pinned gRPC 1.82.1: building ShardRuntime service stubs
|
||||||
|
shard_runtime_proto built
|
||||||
|
shard_runtime_grpc built
|
||||||
|
1/1 shard_protocol_conformance passed
|
||||||
|
```
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 -m pytest -q tests/test_native_shard_protocol.py
|
||||||
|
```
|
||||||
|
|
||||||
|
```text
|
||||||
|
50 passed, 2 optional-path skips
|
||||||
|
```
|
||||||
|
|
||||||
|
All DGR-023-required checks were selected explicitly:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 -m pytest -q -rs tests/test_native_shard_protocol.py \
|
||||||
|
-k 'cpp_and_python_agree_byte_for_byte or generated_python_stubs_match_the_proto or native_toolchain_bootstrap or wrong_grpcio'
|
||||||
|
```
|
||||||
|
|
||||||
|
```text
|
||||||
|
4 passed, 48 deselected
|
||||||
|
```
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 scripts/generate_native_protocol.py --check
|
||||||
|
python3 scripts/generate_protocol_goldens.py --check
|
||||||
|
python3 scripts/ralph_prd_schema.py validate .scratch/distributed-gguf-runtime/prd.json
|
||||||
|
python3 -m compileall -q packages tests
|
||||||
|
git diff --check
|
||||||
|
```
|
||||||
|
|
||||||
|
```text
|
||||||
|
generated stubs are up to date
|
||||||
|
conformance vectors are up to date
|
||||||
|
OK: 55 stories validated
|
||||||
|
compileall passed
|
||||||
|
git diff --check passed
|
||||||
|
```
|
||||||
|
|
||||||
|
## Changed files
|
||||||
|
|
||||||
|
- `scripts/bootstrap_native_toolchain.sh`
|
||||||
|
- `scripts/generate_native_protocol.py`
|
||||||
|
- `packages/node/native/CMakeLists.txt`
|
||||||
|
- `packages/node/native/README.md`
|
||||||
|
- `tests/test_native_shard_protocol.py`
|
||||||
|
- `.scratch/distributed-gguf-runtime/evidence/DGR-023/README.md`
|
||||||
|
- `.scratch/distributed-gguf-runtime/prd.json` (secondary completion projection only)
|
||||||
|
|
||||||
|
## Limitations and dependency handoff
|
||||||
|
|
||||||
|
- This story proves exact schema/message/service generation and cross-language conformance. It does not implement or run the standalone worker service itself; DGR-033/DGR-037 own worker behavior.
|
||||||
|
- The plugin SHA is evidence for this verified build. Reproducibility authority is the exact gRPC commit plus its submodule graph, not an assumption that different compilers produce byte-identical executables.
|
||||||
|
- No model, GPU, API credits, or model download was used.
|
||||||
|
- DGR-024 and DGR-037 may consume this completed generation dependency but must provide their own transport/worker evidence.
|
||||||
210
.scratch/distributed-gguf-runtime/evidence/DGR-025/README.md
Normal file
210
.scratch/distributed-gguf-runtime/evidence/DGR-025/README.md
Normal file
@@ -0,0 +1,210 @@
|
|||||||
|
# DGR-025 evidence — exact artifact and runtime recipe identity
|
||||||
|
|
||||||
|
**Completed:** 2026-07-17
|
||||||
|
**Branch:** `ralph/fable-architecture-loop` (Claude Fable architecture lane)
|
||||||
|
**Authority:** `.scratch/distributed-gguf-runtime/prd.json`
|
||||||
|
**Dependencies:** DGR-018 (`evidence/DGR-018/README.md` — canonical backlog schema and
|
||||||
|
issue projection), DGR-021 (`evidence/DGR-021/README.md` — versioned activation
|
||||||
|
envelope). Both read before changing code.
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
Ensure the tracker and worker only combine numerically and operationally
|
||||||
|
compatible shards: fingerprint every axis that moves the numbers, bind shards to
|
||||||
|
exact half-open ranges, fail closed on any mismatch, and keep uncertified
|
||||||
|
recipes registered-but-dark.
|
||||||
|
|
||||||
|
## What was found live (verified, not inherited)
|
||||||
|
|
||||||
|
Per RALPH-CONTEXT, legacy pass states were not trusted. The DGR-003-lineage
|
||||||
|
identity core was inspected and exercised live before any change:
|
||||||
|
|
||||||
|
- `packages/node/meshnet_node/runtime_recipe.py` — node-side identity:
|
||||||
|
domain-separated digests (`meshnet.model-artifact.v1`,
|
||||||
|
`meshnet.runtime-recipe.v1`, `meshnet.shard-binding.v1`) over the source
|
||||||
|
artifact SHA (`source_digest`, with split artifacts bound to their exact
|
||||||
|
source via `DerivativeBinding`), tokenizer revision (pin-enforced),
|
||||||
|
architecture adapter + architecture/config digest, boundary and protocol
|
||||||
|
schema versions, backend, weight quantization, activation/compute dtypes, and
|
||||||
|
KV dtype/layout (`RECIPE_AXES`). Shard ranges are half-open
|
||||||
|
(`shard_start`/`shard_end`, end-exclusive, protocol convention) with no
|
||||||
|
topology or quant constants anywhere; `check_route` accepts any tiling of
|
||||||
|
`[0, layer_count)`. Route, handshake (`check_handshake`), and session-open
|
||||||
|
(`check_session_open`) checks fail closed with structured `RouteMismatch`
|
||||||
|
reasons mapped to specific protocol error codes (`handshake_error`).
|
||||||
|
- `packages/tracker/meshnet_tracker/recipe.py` — deliberately independent
|
||||||
|
tracker re-derivation (no `meshnet_node` import); declared fingerprints are
|
||||||
|
recomputed, never trusted (`parse_identity`, `FingerprintMismatch`). The
|
||||||
|
`CertificationLedger` keeps every registered recipe dark until a real
|
||||||
|
distributed forward — at least 2 distinct nodes, whole-model coverage,
|
||||||
|
non-synthetic, tokens actually generated — certifies it; dark recipes may
|
||||||
|
route only to certify.
|
||||||
|
- The two implementations are pinned by committed conformance vectors
|
||||||
|
(`tests/data/recipe_fingerprint_vectors.json`).
|
||||||
|
|
||||||
|
Live verification of that pre-existing core before changes:
|
||||||
|
`PYTHONPATH=packages/node:packages/tracker python3 -m pytest -q tests/test_runtime_recipe_identity.py`
|
||||||
|
→ `45 passed`; plus `tests/test_native_identity_emission.py`,
|
||||||
|
`tests/test_tracker_capability_admission.py`, `tests/test_node_admission.py`
|
||||||
|
→ `59 passed`.
|
||||||
|
|
||||||
|
## Gap found and closed (this story's change)
|
||||||
|
|
||||||
|
**The `runtime_version` recipe axis was a label, not a pin.** It was an opaque
|
||||||
|
caller-supplied string: nothing derived it from the DGR-027 lock manifest, and
|
||||||
|
neither identity implementation rejected a moving reference (`"latest"` was
|
||||||
|
accepted), so two workers could run different llama.cpp pins or patch stacks
|
||||||
|
under one label and still agree on the recipe digest. The acceptance criterion
|
||||||
|
explicitly requires fingerprinting the "runtime pin/patch stack".
|
||||||
|
|
||||||
|
### Changed files
|
||||||
|
|
||||||
|
- `packages/node/meshnet_node/runtime_pin.py` (new) — derives the canonical
|
||||||
|
`runtime_version` axis value from the DGR-027 lock workspace
|
||||||
|
(`packages/node/native/llama`):
|
||||||
|
`<runtime>@<40-hex upstream commit>+patchstack.<sha256>` where the stack
|
||||||
|
digest commits, under the `meshnet.runtime-patch-stack.v1` domain, to the
|
||||||
|
*ordered* `(patch name, patch bytes sha256)` stack. Fails closed on: missing
|
||||||
|
or malformed `UPSTREAM_LOCK.json`, unknown schema version, non-40-hex/moving
|
||||||
|
commit, `UPSTREAM_COMMIT` disagreement, any disagreement among the lock's
|
||||||
|
`patch_series`, `patches/series`, and `patches/SHA256SUMS`, a missing patch
|
||||||
|
file, or a patch whose bytes don't match their recorded digest. Reads the
|
||||||
|
committed manifest only; fetching/patching stays with
|
||||||
|
`scripts/llama_cpp_dependency.py` (DGR-027).
|
||||||
|
- `packages/node/meshnet_node/runtime_recipe.py` — `runtime_version` is now
|
||||||
|
pin-enforced (`_require_pin`) exactly like `tokenizer_revision`; for the
|
||||||
|
llama.cpp backend it must also match the canonical
|
||||||
|
`llama.cpp@<40-hex>+patchstack.<64-hex>` grammar.
|
||||||
|
- `packages/node/meshnet_node/native_backend.py` — the production native
|
||||||
|
identity seam no longer accepts a caller-supplied runtime string. It derives
|
||||||
|
`runtime_version` directly through `load_runtime_pin()` from the committed
|
||||||
|
lock and rejects a non-llama backend at this llama.cpp-specific boundary.
|
||||||
|
- `packages/tracker/meshnet_tracker/recipe.py` — the independent tracker
|
||||||
|
implementation applies the same backend-specific grammar before re-deriving
|
||||||
|
the recipe digest, so forged operator labels cannot register or certify.
|
||||||
|
- `tests/test_runtime_pin_identity.py` and
|
||||||
|
`tests/test_native_identity_emission.py` — deterministic tests cover lock
|
||||||
|
derivation, production native emission, and node/tracker rejection of the
|
||||||
|
forged values from independent review. Conformance vectors were regenerated
|
||||||
|
through `scripts/gen_recipe_fingerprint_vectors.py` for the tightened wire
|
||||||
|
contract.
|
||||||
|
|
||||||
|
### Backlog-consistency repair (pre-existing damage, honestly recorded)
|
||||||
|
|
||||||
|
`tests/test_ralph_prd_schema.py` had 4 pre-existing failures before this story
|
||||||
|
touched anything, left by prior sessions and the alternate-history merge:
|
||||||
|
|
||||||
|
- DGR-022 and DGR-027 were marked `passes: true` without `completionNotes` and
|
||||||
|
without regenerated issue projections. Added their `completionNotes`
|
||||||
|
(explicitly labeled as added during this repair, content drawn from their own
|
||||||
|
evidence READMEs) and regenerated
|
||||||
|
`issues/022-…` / `issues/027-…` via `scripts/ralph_prd_schema.py render`.
|
||||||
|
- Three pre-DGR legacy GLM alpha issue files (`18-…`, `19-…`, `20-…`,
|
||||||
|
committed 2026-07-14, before DGR-018 established the generated-only
|
||||||
|
convention; they carry no authority disclaimer because they are *not*
|
||||||
|
generated from prd.json) were relocated via `git mv` to
|
||||||
|
`issues/legacy/` — preserved as provenance, out of the generated namespace.
|
||||||
|
|
||||||
|
### prd.json
|
||||||
|
|
||||||
|
Marked `DGR-025.passes = true` with `completionNotes`; regenerated
|
||||||
|
`issues/025-define-exact-artifact-and-runtime-recipe-identity.md`.
|
||||||
|
|
||||||
|
## Acceptance criteria → evidence
|
||||||
|
|
||||||
|
1. **Fingerprint all axes** — `RECIPE_AXES` + `ArtifactIdentity` cover source
|
||||||
|
artifact SHA, tokenizer revision, architecture adapter/version (adapter axis
|
||||||
|
+ architecture/config digest), boundary schema (boundary + protocol schema
|
||||||
|
versions), backend, quant, activation/compute dtype, KV/state layout; the
|
||||||
|
runtime pin/patch stack is now committed via the derived `runtime_version`
|
||||||
|
axis (`runtime_pin.py`). Verified by `test_runtime_recipe_identity.py` and
|
||||||
|
`test_runtime_pin_identity.py`.
|
||||||
|
2. **Exact half-open range, no hardcoded topology/quant** — `ShardIdentity`
|
||||||
|
end-exclusive ranges, `DerivativeBinding` coverage checks, `check_route`
|
||||||
|
tiling over arbitrary layouts; quant/dtype values are open strings
|
||||||
|
(dynamic recipe inputs). Verified by `test_runtime_recipe_identity.py`
|
||||||
|
(routes of 1, 2, and 5 shards; no product constants).
|
||||||
|
3. **Fail closed on any mismatch** — artifact, adapter, boundary/schema, cache
|
||||||
|
layout, backend, and runtime mismatches each produce structured
|
||||||
|
`RouteMismatch` reasons and protocol error codes; the tracker recomputes
|
||||||
|
digests and rejects inconsistent claims; moving runtime references are now
|
||||||
|
rejected on both sides.
|
||||||
|
4. **Registered-but-dark** — `CertificationLedger`: unknown recipes cannot be
|
||||||
|
certified, registered recipes are dark, only a real ≥2-distinct-node
|
||||||
|
whole-model non-synthetic forward promotes; verified by
|
||||||
|
`test_runtime_recipe_identity.py` / `test_tracker_capability_admission.py`.
|
||||||
|
5. **Gates + this handoff** — below.
|
||||||
|
|
||||||
|
## Commands and results
|
||||||
|
|
||||||
|
```bash
|
||||||
|
PYTHONPATH=packages/node:packages/tracker python3 -m pytest -q tests/test_runtime_pin_identity.py
|
||||||
|
```
|
||||||
|
```text
|
||||||
|
23 passed in 0.15s
|
||||||
|
```
|
||||||
|
|
||||||
|
```bash
|
||||||
|
PYTHONPATH=packages/node:packages/tracker python3 -m pytest -q \
|
||||||
|
tests/test_runtime_pin_identity.py tests/test_runtime_recipe_identity.py \
|
||||||
|
tests/test_native_identity_emission.py tests/test_tracker_capability_admission.py \
|
||||||
|
tests/test_node_admission.py tests/test_node_capability.py tests/test_recipe_benchmark.py
|
||||||
|
```
|
||||||
|
```text
|
||||||
|
202 passed, 1 warning in 5.38s
|
||||||
|
```
|
||||||
|
|
||||||
|
```bash
|
||||||
|
PYTHONPATH=packages/node:packages/tracker python3 -m pytest -q tests/test_ralph_prd_schema.py
|
||||||
|
```
|
||||||
|
```text
|
||||||
|
108 passed
|
||||||
|
```
|
||||||
|
(4 failed before this story's backlog repair; 0 after.)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 -m compileall -q packages tests # exit 0
|
||||||
|
git diff --check # exit 0
|
||||||
|
python3 scripts/ralph_prd_schema.py validate .scratch/distributed-gguf-runtime/prd.json
|
||||||
|
# OK: 55 stories validated.
|
||||||
|
```
|
||||||
|
|
||||||
|
Default tests are model-download-free, API-credit-free, and GPU-free; no model
|
||||||
|
artifact was touched and nothing was written under `/home`.
|
||||||
|
|
||||||
|
## Limitations
|
||||||
|
|
||||||
|
- The production native identity seam now derives the manifest pin and cannot
|
||||||
|
accept an operator-supplied runtime label. It still cannot attest that the
|
||||||
|
running binary was built from those locked bytes. Embedding the patched-tree
|
||||||
|
hash at build time and echoing it through the DGR-022 status contract belongs
|
||||||
|
with DGR-028+/DGR-031; real distributed certification remains the final trust
|
||||||
|
boundary.
|
||||||
|
- The DGR-027-recorded blocker stands: `0002-dense-llama-owned-range-loader.patch`
|
||||||
|
does not apply cleanly against the pin (DGR-028). That does not affect this
|
||||||
|
story: the identity commits to the patch *bytes as committed*, which is
|
||||||
|
precisely what makes a later repaired patch a *different* runtime identity.
|
||||||
|
- No native/CMake change was made, so the native build/CTest gate is not
|
||||||
|
applicable; no llama.cpp patch content was changed, so apply/check/reverse
|
||||||
|
verification is not applicable (and is blocked by the DGR-028 defect anyway).
|
||||||
|
- Tracker routing, load balancing, billing, telemetry, and relay semantics are
|
||||||
|
untouched; the only behavior change outside the new module is the stricter
|
||||||
|
(fail-closed) rejection of moving `runtime_version` values.
|
||||||
|
|
||||||
|
## Dependency handoff
|
||||||
|
|
||||||
|
- **DGR-026** (split-GGUF provisioning): bind each provisioned split via
|
||||||
|
`DerivativeBinding` to the exact source digest recorded in its hashed
|
||||||
|
manifest; the per-split `shard_binding_digest` is what certification pins.
|
||||||
|
- **DGR-031** (`ShardEngine`): construct worker identity through
|
||||||
|
`shard_identity_from_native_report` and populate `runtime_version` from
|
||||||
|
`meshnet_node.runtime_pin.load_runtime_pin().runtime_version` — never from an
|
||||||
|
operator string. A build-time echo of the patched-tree hash through the
|
||||||
|
status contract would close the manifest-vs-binary gap noted above.
|
||||||
|
- **DGR-041** (capability registration): the tracker already re-derives and
|
||||||
|
fail-closes on presented identities (`parse_identity`); register recipes
|
||||||
|
through the `CertificationLedger` so they arrive dark.
|
||||||
|
- **DGR-044** (DeepSeek V4 Flash target): pin the target's artifact identity
|
||||||
|
the same way `glm_alpha_artifact` does — read locked manifests, never restate
|
||||||
|
digests — and note `layer_count` must count the routed transformer stack the
|
||||||
|
route tiles, excluding MTP (reserved for beta).
|
||||||
72
.scratch/distributed-gguf-runtime/evidence/DGR-027/README.md
Normal file
72
.scratch/distributed-gguf-runtime/evidence/DGR-027/README.md
Normal file
@@ -0,0 +1,72 @@
|
|||||||
|
# DGR-027 evidence — exact llama.cpp provenance manifest and fetch workspace
|
||||||
|
|
||||||
|
**Completed implementation:** 2026-07-17
|
||||||
|
**Branch:** `ralph/dgr-small-terra`
|
||||||
|
**Authority:** live Gitea issue #11. The controller fetched and claimed the issue
|
||||||
|
through the Gitea API before launch; the isolated agent received that exact body.
|
||||||
|
|
||||||
|
## Changed files
|
||||||
|
|
||||||
|
- `packages/node/native/llama/UPSTREAM_LOCK.json`
|
||||||
|
- `packages/node/native/llama/PATCH-STACK.md`
|
||||||
|
- `scripts/llama_cpp_dependency.py`
|
||||||
|
- `tests/test_llama_cpp_dependency.py`
|
||||||
|
- `.scratch/distributed-gguf-runtime/evidence/DGR-027/README.md`
|
||||||
|
|
||||||
|
## Provenance and retrieval contract
|
||||||
|
|
||||||
|
`UPSTREAM_LOCK.json` records the upstream Git URL, immutable 40-character
|
||||||
|
commit `e920c523e3b8a0163fe498af5bf90df35ff51d25`, expected Git tree
|
||||||
|
`6c91a11407a3a3fb160f5dac705f9c59718f54f1`, MIT license, and the sole
|
||||||
|
retrieval method: `git-clone-detached-commit` into `build/llama.cpp/source`.
|
||||||
|
|
||||||
|
`python3 scripts/llama_cpp_dependency.py fetch` has no branch, tag, ref, or
|
||||||
|
repository override. On a first fetch it clones the manifest URL, checks out
|
||||||
|
the detached commit, and verifies commit, tree, required upstream blobs,
|
||||||
|
license, and cleanliness. If the workspace already exists, it makes no network
|
||||||
|
request and accepts it only after the same verification. Dirty or mismatched
|
||||||
|
caches fail closed. The build directory is already ignored by `.gitignore`.
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
|
||||||
|
| Command | Result |
|
||||||
|
| --- | --- |
|
||||||
|
| `python3 -m pytest -q tests/test_llama_cpp_dependency.py` | `7 passed in 0.22s` |
|
||||||
|
| `python3 -m compileall packages tests` | passed |
|
||||||
|
| `git diff --check` | passed (no output) |
|
||||||
|
| `python3 scripts/llama_cpp_dependency.py inspect` | passed; reports exact commit/tree, retrieval workspace, MIT license, and two-patch stack |
|
||||||
|
| `python3 scripts/llama_cpp_dependency.py fetch --workspace /tmp/not-llama-workspace` | failed closed with status 2: workspace outside the locked ignored build root |
|
||||||
|
| symlinked workspace regression | passed; both a `build/` ancestor symlink and a final `source` symlink escaping the repository are refused |
|
||||||
|
| attached-branch cache regression | passed; an exact commit on a local branch is refused until checked out as detached HEAD |
|
||||||
|
| ignored/excluded injection regression | passed; a file hidden by `.git/info/exclude` is detected and refused |
|
||||||
|
| tracked injection regression | passed; modified tracked content hidden by both `assume-unchanged` and `skip-worktree` is content-hashed and refused |
|
||||||
|
| executable-mode regression | passed on the POSIX fixture for both index flags; the mounted project workspace has `core.filemode=false`, so its exact index tree is the canonical mode record and physical mode bits are not treated as meaningful |
|
||||||
|
| `git check-ignore -v build/llama.cpp/source` | passed; `.gitignore:6:build/` |
|
||||||
|
| `git diff --summary` and `git ls-files build packages/node/native/llama` | no source checkout or new submodule introduced; only manifest/docs/patches/native wrapper are tracked |
|
||||||
|
| `python3 scripts/llama_cpp_dependency.py fetch` (controller network lane) | passed; fetched the exact detached commit and verified HEAD `e920c523e3b8a0163fe498af5bf90df35ff51d25` and tree `6c91a11407a3a3fb160f5dac705f9c59718f54f1` in the ignored workspace |
|
||||||
|
| `python3 scripts/llama_cpp_dependency.py apply --source-dir build/llama.cpp/source` | failed on the pre-existing `0002-dense-llama-owned-range-loader.patch` as a corrupt patch at line 26; this is an explicit DGR-028 blocker and no native-build claim is made |
|
||||||
|
|
||||||
|
The targeted test suite creates a local Git fixture to prove offline cache reuse
|
||||||
|
after full identity verification, then proves a dirty cache is rejected. It
|
||||||
|
also proves the CLI rejects a repository/branch override and an arbitrary
|
||||||
|
workspace.
|
||||||
|
|
||||||
|
## Limitations
|
||||||
|
|
||||||
|
- The controller successfully materialized and verified the exact upstream
|
||||||
|
commit/tree, so the DGR-027 fetch and offline-cache boundary has real upstream
|
||||||
|
evidence rather than fixture-only evidence.
|
||||||
|
- The existing `0002-dense-llama-owned-range-loader.patch` is malformed and
|
||||||
|
cannot pass `git apply --check` against the exact pin. DGR-027 changes no patch
|
||||||
|
file; repairing and certifying the numbered patch stack belongs to DGR-028.
|
||||||
|
Until that story closes, the repository must not claim patched-tree, native
|
||||||
|
CMake/CTest, or reverse-apply certification.
|
||||||
|
- No model, API credits, GPU, or model artifact storage was used.
|
||||||
|
|
||||||
|
## Dependency handoff
|
||||||
|
|
||||||
|
DGR-028, DGR-029, and DGR-044 must invoke the manifest-owned `fetch` command
|
||||||
|
before touching llama.cpp source. They may use only the verified
|
||||||
|
`build/llama.cpp/source` checkout and must record any native build, CTest, and
|
||||||
|
patch apply/check/reverse evidence against the exact manifest pin. DGR-017's
|
||||||
|
cleanup remains provenance only and grants no inherited completion credit.
|
||||||
@@ -1,7 +1,7 @@
|
|||||||
<!-- GENERATED FROM prd.json — DO NOT EDIT AS AN INDEPENDENT SOURCE. prd.json IS AUTHORITATIVE. -->
|
<!-- GENERATED FROM prd.json — DO NOT EDIT AS AN INDEPENDENT SOURCE. prd.json IS AUTHORITATIVE. -->
|
||||||
# DGR-022: Define Shard lifecycle and structured status RPCs
|
# DGR-022: Define Shard lifecycle and structured status RPCs
|
||||||
|
|
||||||
- **Status / triage:** specification only; `ready-for-agent`; `passes: false`
|
- **Status / triage:** completed; `passes: true`
|
||||||
- **Execution mode:** `AFK`
|
- **Execution mode:** `AFK`
|
||||||
- **Milestone:** `M1`
|
- **Milestone:** `M1`
|
||||||
- **Dependencies:** `DGR-021`
|
- **Dependencies:** `DGR-021`
|
||||||
@@ -18,11 +18,11 @@ Fresh Ralph session: read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md`,
|
|||||||
|
|
||||||
## Acceptance criteria
|
## Acceptance criteria
|
||||||
|
|
||||||
- [ ] Define capability, health, bidirectional session stream, cancellation, release, and metrics RPCs.
|
- [x] Define capability, health, bidirectional session stream, cancellation, release, and metrics RPCs.
|
||||||
- [ ] Specify deadlines, cancellation propagation, bounded flow control, cache expectations/results, and structured error taxonomy.
|
- [x] Specify deadlines, cancellation propagation, bounded flow control, cache expectations/results, and structured error taxonomy.
|
||||||
- [ ] Specify TLS/auth hooks without moving Meshnet authentication or billing into the worker.
|
- [x] Specify TLS/auth hooks without moving Meshnet authentication or billing into the worker.
|
||||||
- [ ] Add compatibility tests for supported versions and fail-closed tests for unsupported versions and malformed lifecycle transitions.
|
- [x] Add compatibility tests for supported versions and fail-closed tests for unsupported versions and malformed lifecycle transitions.
|
||||||
- [ ] Applicable shared quality gates in `prd.json` pass, and the evidence handoff records exact commands/results, changed files, limitations, and dependency handoff.
|
- [x] Applicable shared quality gates in `prd.json` pass, and the evidence handoff records exact commands/results, changed files, limitations, and dependency handoff.
|
||||||
|
|
||||||
## Shared quality gates
|
## Shared quality gates
|
||||||
|
|
||||||
@@ -36,4 +36,4 @@ Fresh Ralph session: read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md`,
|
|||||||
|
|
||||||
## Evidence handoff
|
## Evidence handoff
|
||||||
|
|
||||||
Write and verify `.scratch/distributed-gguf-runtime/evidence/DGR-022/README.md`. Until every criterion and applicable gate has real evidence, this story remains `passes: false`. Legacy evidence is provenance only, not completion credit.
|
Verified evidence: `.scratch/distributed-gguf-runtime/evidence/DGR-022/README.md`. Legacy evidence remains provenance only and grants no implementation completion credit.
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
<!-- GENERATED FROM prd.json — DO NOT EDIT AS AN INDEPENDENT SOURCE. prd.json IS AUTHORITATIVE. -->
|
<!-- GENERATED FROM prd.json — DO NOT EDIT AS AN INDEPENDENT SOURCE. prd.json IS AUTHORITATIVE. -->
|
||||||
# DGR-025: Define exact artifact and runtime recipe identity
|
# DGR-025: Define exact artifact and runtime recipe identity
|
||||||
|
|
||||||
- **Status / triage:** specification only; `ready-for-agent`; `passes: false`
|
- **Status / triage:** completed; `passes: true`
|
||||||
- **Execution mode:** `AFK`
|
- **Execution mode:** `AFK`
|
||||||
- **Milestone:** `M1`
|
- **Milestone:** `M1`
|
||||||
- **Dependencies:** `DGR-018`, `DGR-021`
|
- **Dependencies:** `DGR-018`, `DGR-021`
|
||||||
@@ -18,11 +18,11 @@ Fresh Ralph session: read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md`,
|
|||||||
|
|
||||||
## Acceptance criteria
|
## Acceptance criteria
|
||||||
|
|
||||||
- [ ] Fingerprint source artifact SHA, tokenizer revision, architecture adapter/version, boundary schema, runtime pin/patch stack, backend, quant, activation/compute dtype, and KV/state layout.
|
- [x] Fingerprint source artifact SHA, tokenizer revision, architecture adapter/version, boundary schema, runtime pin/patch stack, backend, quant, activation/compute dtype, and KV/state layout.
|
||||||
- [ ] Bind each shard to an exact half-open range without hardcoding a topology or quant.
|
- [x] Bind each shard to an exact half-open range without hardcoding a topology or quant.
|
||||||
- [ ] Fail closed on any artifact, adapter, boundary, cache, backend, or runtime mismatch.
|
- [x] Fail closed on any artifact, adapter, boundary, cache, backend, or runtime mismatch.
|
||||||
- [ ] Unsupported recipes remain registered-but-dark until real-hardware evidence certifies them.
|
- [x] Unsupported recipes remain registered-but-dark until real-hardware evidence certifies them.
|
||||||
- [ ] Applicable shared quality gates in `prd.json` pass, and the evidence handoff records exact commands/results, changed files, limitations, and dependency handoff.
|
- [x] Applicable shared quality gates in `prd.json` pass, and the evidence handoff records exact commands/results, changed files, limitations, and dependency handoff.
|
||||||
|
|
||||||
## Shared quality gates
|
## Shared quality gates
|
||||||
|
|
||||||
@@ -36,4 +36,4 @@ Fresh Ralph session: read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md`,
|
|||||||
|
|
||||||
## Evidence handoff
|
## Evidence handoff
|
||||||
|
|
||||||
Write and verify `.scratch/distributed-gguf-runtime/evidence/DGR-025/README.md`. Until every criterion and applicable gate has real evidence, this story remains `passes: false`. Legacy evidence is provenance only, not completion credit.
|
Verified evidence: `.scratch/distributed-gguf-runtime/evidence/DGR-025/README.md`. Legacy evidence remains provenance only and grants no implementation completion credit.
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
<!-- GENERATED FROM prd.json — DO NOT EDIT AS AN INDEPENDENT SOURCE. prd.json IS AUTHORITATIVE. -->
|
<!-- GENERATED FROM prd.json — DO NOT EDIT AS AN INDEPENDENT SOURCE. prd.json IS AUTHORITATIVE. -->
|
||||||
# DGR-027: Add exact llama.cpp provenance manifest and fetch workspace
|
# DGR-027: Add exact llama.cpp provenance manifest and fetch workspace
|
||||||
|
|
||||||
- **Status / triage:** specification only; `ready-for-agent`; `passes: false`
|
- **Status / triage:** completed; `passes: true`
|
||||||
- **Execution mode:** `AFK`
|
- **Execution mode:** `AFK`
|
||||||
- **Milestone:** `M1`
|
- **Milestone:** `M1`
|
||||||
- **Dependencies:** `DGR-017`
|
- **Dependencies:** `DGR-017`
|
||||||
@@ -18,11 +18,11 @@ Fresh Ralph session: read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md`,
|
|||||||
|
|
||||||
## Acceptance criteria
|
## Acceptance criteria
|
||||||
|
|
||||||
- [ ] Manifest records upstream URL, exact commit, expected source archive/tree hash, license, and retrieval method.
|
- [x] Manifest records upstream URL, exact commit, expected source archive/tree hash, license, and retrieval method.
|
||||||
- [ ] Fetch tooling verifies identity before use and refuses an unpinned branch/tag.
|
- [x] Fetch tooling verifies identity before use and refuses an unpinned branch/tag.
|
||||||
- [ ] Source is fetched into an ignored build workspace; no submodule, vendored source tree, or permanent fork is introduced.
|
- [x] Source is fetched into an ignored build workspace; no submodule, vendored source tree, or permanent fork is introduced.
|
||||||
- [ ] Offline reuse is supported only after the cached tree’s exact identity is verified.
|
- [x] Offline reuse is supported only after the cached tree’s exact identity is verified.
|
||||||
- [ ] Applicable shared quality gates in `prd.json` pass, and the evidence handoff records exact commands/results, changed files, limitations, and dependency handoff.
|
- [x] Applicable shared quality gates in `prd.json` pass, and the evidence handoff records exact commands/results, changed files, limitations, and dependency handoff.
|
||||||
|
|
||||||
## Shared quality gates
|
## Shared quality gates
|
||||||
|
|
||||||
@@ -36,4 +36,4 @@ Fresh Ralph session: read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md`,
|
|||||||
|
|
||||||
## Evidence handoff
|
## Evidence handoff
|
||||||
|
|
||||||
Write and verify `.scratch/distributed-gguf-runtime/evidence/DGR-027/README.md`. Until every criterion and applicable gate has real evidence, this story remains `passes: false`. Legacy evidence is provenance only, not completion credit.
|
Verified evidence: `.scratch/distributed-gguf-runtime/evidence/DGR-027/README.md`. Legacy evidence remains provenance only and grants no implementation completion credit.
|
||||||
|
|||||||
@@ -489,7 +489,8 @@
|
|||||||
"DGR-024",
|
"DGR-024",
|
||||||
"DGR-033",
|
"DGR-033",
|
||||||
"DGR-037"
|
"DGR-037"
|
||||||
]
|
],
|
||||||
|
"completionNotes": "Completed 2026-07-17. Implemented the versioned backend-neutral Shard lifecycle/status contract in packages/node/meshnet_node/shard_lifecycle.py (RPC names, schema-version negotiation with fail-closed unsupported versions, structured status/error taxonomy, lifecycle state machine, monotonic idempotency enforcement, bounded flow control, cache expectation/result types, deadline policy and TLS/auth hooks) with deterministic tests in tests/test_shard_lifecycle.py (17 passed alongside the DGR-021 envelope tests). Generated protobuf bindings remain DGR-023. Evidence: evidence/DGR-022/README.md. These completionNotes were added during the DGR-025 backlog-consistency repair; the DGR-022 session omitted them."
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"id": "DGR-023",
|
"id": "DGR-023",
|
||||||
@@ -521,8 +522,8 @@
|
|||||||
"A clean checkout regenerates bindings deterministically or fails with an actionable toolchain error.",
|
"A clean checkout regenerates bindings deterministically or fails with an actionable toolchain error.",
|
||||||
"Applicable shared quality gates in `prd.json` pass, and the evidence handoff records exact commands/results, changed files, limitations, and dependency handoff."
|
"Applicable shared quality gates in `prd.json` pass, and the evidence handoff records exact commands/results, changed files, limitations, and dependency handoff."
|
||||||
],
|
],
|
||||||
"passes": false,
|
"passes": true,
|
||||||
"notes": "Generated source issue: .scratch/distributed-gguf-runtime/issues/023-make-python-and-c-protobuf-generation-reproducible.md; prd.json is authoritative.",
|
"notes": "Completed from Gitea #7 after controller provisioned and exercised the exact Python/C++ toolchains. Verified deterministic generation, native CMake/CTest, Python↔C++ byte parity, compileall, and diff checks; fixed relative bootstrap prefix resolution.",
|
||||||
"blocks": [
|
"blocks": [
|
||||||
"DGR-024",
|
"DGR-024",
|
||||||
"DGR-037"
|
"DGR-037"
|
||||||
@@ -597,14 +598,15 @@
|
|||||||
"Unsupported recipes remain registered-but-dark until real-hardware evidence certifies them.",
|
"Unsupported recipes remain registered-but-dark until real-hardware evidence certifies them.",
|
||||||
"Applicable shared quality gates in `prd.json` pass, and the evidence handoff records exact commands/results, changed files, limitations, and dependency handoff."
|
"Applicable shared quality gates in `prd.json` pass, and the evidence handoff records exact commands/results, changed files, limitations, and dependency handoff."
|
||||||
],
|
],
|
||||||
"passes": false,
|
"passes": true,
|
||||||
"notes": "Generated source issue: .scratch/distributed-gguf-runtime/issues/025-define-exact-artifact-and-runtime-recipe-identity.md; prd.json is authoritative.",
|
"notes": "Generated source issue: .scratch/distributed-gguf-runtime/issues/025-define-exact-artifact-and-runtime-recipe-identity.md; prd.json is authoritative.",
|
||||||
"blocks": [
|
"blocks": [
|
||||||
"DGR-026",
|
"DGR-026",
|
||||||
"DGR-031",
|
"DGR-031",
|
||||||
"DGR-041",
|
"DGR-041",
|
||||||
"DGR-044"
|
"DGR-044"
|
||||||
]
|
],
|
||||||
|
"completionNotes": "Completed 2026-07-17. Verified the live DGR-003-lineage identity core against every criterion: node packages/node/meshnet_node/runtime_recipe.py and the independent tracker packages/tracker/meshnet_tracker/recipe.py (pinned together by tests/data/recipe_fingerprint_vectors.json) fingerprint the source artifact SHA, tokenizer pin, architecture adapter and config digest, boundary/protocol schema versions, backend, weight quant, activation/compute dtypes, and KV dtype/layout under domain-separated digests; shards bind to exact half-open ranges with no topology or quant constants; route/handshake/session checks fail closed with structured mismatch reasons; recipes stay registered-but-dark in the tracker CertificationLedger until a real >=2-distinct-node whole-model distributed forward certifies them. Closed the one open criterion gap (runtime pin/patch stack): new packages/node/meshnet_node/runtime_pin.py derives the runtime_version axis from the DGR-027 lock manifest — exact upstream commit plus a digest over the ordered patch-stack bytes — failing closed on any UPSTREAM_LOCK.json/UPSTREAM_COMMIT/series/SHA256SUMS/patch-byte disagreement, and both identity implementations now reject a moving runtime_version reference. Tests: tests/test_runtime_pin_identity.py (17 passed) plus 196 passing impacted identity/admission/native-emission tests; python -m compileall and git diff --check clean. Also repaired backlog consistency left by prior sessions: added the missing DGR-022/DGR-027 completionNotes, regenerated the DGR-022/025/027 issue projections, and relocated three pre-DGR legacy GLM alpha issue files to issues/legacy/."
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"id": "DGR-026",
|
"id": "DGR-026",
|
||||||
@@ -673,13 +675,14 @@
|
|||||||
"Offline reuse is supported only after the cached tree’s exact identity is verified.",
|
"Offline reuse is supported only after the cached tree’s exact identity is verified.",
|
||||||
"Applicable shared quality gates in `prd.json` pass, and the evidence handoff records exact commands/results, changed files, limitations, and dependency handoff."
|
"Applicable shared quality gates in `prd.json` pass, and the evidence handoff records exact commands/results, changed files, limitations, and dependency handoff."
|
||||||
],
|
],
|
||||||
"passes": false,
|
"passes": true,
|
||||||
"notes": "Generated source issue: .scratch/distributed-gguf-runtime/issues/027-add-exact-llama-cpp-provenance-manifest-and-fetch-workspace.md; prd.json is authoritative.",
|
"notes": "Completed from Gitea #11 via the Terra Ralph lane; independently reviewed after fail-closed cache identity hardening. Verified 7 focused tests, exact real-cache reuse, compileall, and diff-check. The pre-existing malformed 0002 patch is handed to DGR-028.",
|
||||||
"blocks": [
|
"blocks": [
|
||||||
"DGR-028",
|
"DGR-028",
|
||||||
"DGR-029",
|
"DGR-029",
|
||||||
"DGR-044"
|
"DGR-044"
|
||||||
]
|
],
|
||||||
|
"completionNotes": "Completed 2026-07-17. Added the exact llama.cpp provenance manifest packages/node/native/llama/UPSTREAM_LOCK.json pinning commit e920c523e3b8a0163fe498af5bf90df35ff51d25 (tree 6c91a11407a3a3fb160f5dac705f9c59718f54f1) with the sole git-clone-detached-commit retrieval into the ignored build/llama.cpp workspace, fail-closed dirty/mismatched-cache verification, PATCH-STACK.md, scripts/llama_cpp_dependency.py, and tests/test_llama_cpp_dependency.py (7 passed). Known blocker recorded for DGR-028: 0002-dense-llama-owned-range-loader.patch fails git apply --check against the pin, so no patched-tree/native-build claim is made. Evidence: evidence/DGR-027/README.md. These completionNotes were added during the DGR-025 backlog-consistency repair; the DGR-027 session omitted them."
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"id": "DGR-028",
|
"id": "DGR-028",
|
||||||
|
|||||||
@@ -12,6 +12,7 @@ from __future__ import annotations
|
|||||||
from dataclasses import dataclass
|
from dataclasses import dataclass
|
||||||
|
|
||||||
from .native_protocol import BUNDLE_VERSION, SCHEMA_VERSION, pb
|
from .native_protocol import BUNDLE_VERSION, SCHEMA_VERSION, pb
|
||||||
|
from .runtime_pin import load_runtime_pin
|
||||||
from .runtime_recipe import (
|
from .runtime_recipe import (
|
||||||
ArtifactIdentity,
|
ArtifactIdentity,
|
||||||
DerivativeBinding,
|
DerivativeBinding,
|
||||||
@@ -72,7 +73,6 @@ class NativeNumericalRecipe:
|
|||||||
kv_layout: str
|
kv_layout: str
|
||||||
architecture_adapter: str
|
architecture_adapter: str
|
||||||
backend_id: str
|
backend_id: str
|
||||||
runtime_version: str
|
|
||||||
recipe_id: str
|
recipe_id: str
|
||||||
recipe_version: str
|
recipe_version: str
|
||||||
catalogue_version: str
|
catalogue_version: str
|
||||||
@@ -95,6 +95,11 @@ def shard_identity_from_native_report(inputs: NativeIdentityInputs) -> ShardIden
|
|||||||
report = inputs.loaded_artifact
|
report = inputs.loaded_artifact
|
||||||
pin = inputs.artifact_pin
|
pin = inputs.artifact_pin
|
||||||
recipe = inputs.numerical_recipe
|
recipe = inputs.numerical_recipe
|
||||||
|
if recipe.backend_id.strip().lower() not in {"llama.cpp", "llama-cpp"}:
|
||||||
|
raise RecipeIdentityError(
|
||||||
|
"native llama.cpp identity requires backend_id 'llama.cpp' or 'llama-cpp'"
|
||||||
|
)
|
||||||
|
runtime_version = load_runtime_pin().runtime_version
|
||||||
artifact = ArtifactIdentity(
|
artifact = ArtifactIdentity(
|
||||||
artifact_id=pin.artifact_id,
|
artifact_id=pin.artifact_id,
|
||||||
revision=pin.revision,
|
revision=pin.revision,
|
||||||
@@ -115,7 +120,7 @@ def shard_identity_from_native_report(inputs: NativeIdentityInputs) -> ShardIden
|
|||||||
tokenizer_revision=inputs.tokenizer_revision,
|
tokenizer_revision=inputs.tokenizer_revision,
|
||||||
architecture_adapter=recipe.architecture_adapter,
|
architecture_adapter=recipe.architecture_adapter,
|
||||||
backend_id=recipe.backend_id,
|
backend_id=recipe.backend_id,
|
||||||
runtime_version=recipe.runtime_version,
|
runtime_version=runtime_version,
|
||||||
boundary_schema_version=recipe.boundary_schema_version,
|
boundary_schema_version=recipe.boundary_schema_version,
|
||||||
protocol_schema_version=recipe.protocol_schema_version,
|
protocol_schema_version=recipe.protocol_schema_version,
|
||||||
recipe_id=recipe.recipe_id,
|
recipe_id=recipe.recipe_id,
|
||||||
|
|||||||
221
packages/node/meshnet_node/runtime_pin.py
Normal file
221
packages/node/meshnet_node/runtime_pin.py
Normal file
@@ -0,0 +1,221 @@
|
|||||||
|
"""Canonical runtime pin identity for the recipe fingerprint (DGR-025).
|
||||||
|
|
||||||
|
The recipe digest (:mod:`meshnet_node.runtime_recipe`) commits to a
|
||||||
|
``runtime_version`` axis, but a string the operator typed is a label, not a
|
||||||
|
pin: two workers could run different patch stacks under the same label and
|
||||||
|
still hash to the same recipe. The DGR-027 lock manifest
|
||||||
|
(``packages/node/native/llama``) already records the one exact upstream commit
|
||||||
|
and the ordered patch stack the native runtime is built from, so the axis
|
||||||
|
value is *derived* from that manifest, never asserted.
|
||||||
|
|
||||||
|
The derived value has three load-bearing parts, and each is separately fatal
|
||||||
|
to compatibility: the runtime name (from the upstream URL), the exact
|
||||||
|
40-character upstream commit, and a digest over the ordered patch-stack bytes.
|
||||||
|
A different upstream pin, a reordered stack, or a single changed patch byte
|
||||||
|
each produce a different axis value, which produces a different recipe digest,
|
||||||
|
which partitions the route — exactly the fail-closed behavior DGR-025 asks
|
||||||
|
for.
|
||||||
|
|
||||||
|
Every consistency check here fails closed. The manifest keeps three records of
|
||||||
|
the stack — ``UPSTREAM_LOCK.json``'s ``patch_series``, ``patches/series``, and
|
||||||
|
``patches/SHA256SUMS`` — plus the ``UPSTREAM_COMMIT`` convenience file, and a
|
||||||
|
disagreement between any two of them means the workspace's identity is
|
||||||
|
unknowable, not "probably fine". This module reads the committed manifest
|
||||||
|
only; fetching and patching the actual source tree stays with
|
||||||
|
``scripts/llama_cpp_dependency.py`` (DGR-027).
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
import re
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
# Domain separation, matching the runtime_recipe digest convention: a patch
|
||||||
|
# stack digest can never be confused with an artifact or recipe digest.
|
||||||
|
PATCH_STACK_DIGEST_DOMAIN = "meshnet.runtime-patch-stack.v1"
|
||||||
|
|
||||||
|
# The UPSTREAM_LOCK.json layout this reader understands (DGR-027 schema).
|
||||||
|
RUNTIME_PIN_SCHEMA_VERSION = 1
|
||||||
|
|
||||||
|
# The committed DGR-027 manifest for the llama.cpp runtime.
|
||||||
|
DEFAULT_LOCK_DIR = Path(__file__).resolve().parent.parent / "native" / "llama"
|
||||||
|
|
||||||
|
_HEX40 = re.compile(r"^[0-9a-f]{40}$")
|
||||||
|
_HEX64 = re.compile(r"^[0-9a-f]{64}$")
|
||||||
|
|
||||||
|
|
||||||
|
class RuntimePinError(ValueError):
|
||||||
|
"""The lock workspace is missing, malformed, or internally inconsistent."""
|
||||||
|
|
||||||
|
|
||||||
|
def _canonical_sha256(value: object) -> str:
|
||||||
|
payload = json.dumps(
|
||||||
|
value, sort_keys=True, separators=(",", ":"), ensure_ascii=False
|
||||||
|
)
|
||||||
|
return hashlib.sha256(payload.encode("utf-8")).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class RuntimePin:
|
||||||
|
"""One exact runtime: a name, an upstream commit, and an ordered patch stack."""
|
||||||
|
|
||||||
|
runtime_name: str
|
||||||
|
upstream_commit: str
|
||||||
|
patch_series: tuple[str, ...]
|
||||||
|
patch_digests: tuple[str, ...]
|
||||||
|
|
||||||
|
@property
|
||||||
|
def patch_stack_digest(self) -> str:
|
||||||
|
"""A digest over the ordered (name, bytes-digest) stack.
|
||||||
|
|
||||||
|
Order is digested deliberately: applying the same patches in a
|
||||||
|
different order can produce a different tree, so a reordered stack is
|
||||||
|
a different runtime.
|
||||||
|
"""
|
||||||
|
return _canonical_sha256(
|
||||||
|
{
|
||||||
|
"domain": PATCH_STACK_DIGEST_DOMAIN,
|
||||||
|
"body": {
|
||||||
|
"patches": [
|
||||||
|
[name, digest]
|
||||||
|
for name, digest in zip(self.patch_series, self.patch_digests)
|
||||||
|
]
|
||||||
|
},
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def runtime_version(self) -> str:
|
||||||
|
"""The exact ``runtime_version`` recipe axis value for this pin."""
|
||||||
|
return (
|
||||||
|
f"{self.runtime_name}@{self.upstream_commit}"
|
||||||
|
f"+patchstack.{self.patch_stack_digest}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _read_text(path: Path, what: str) -> str:
|
||||||
|
try:
|
||||||
|
return path.read_text(encoding="utf-8")
|
||||||
|
except FileNotFoundError:
|
||||||
|
raise RuntimePinError(f"{what} not found at {path}") from None
|
||||||
|
except OSError as exc:
|
||||||
|
raise RuntimePinError(f"{what} at {path} is unreadable: {exc}") from exc
|
||||||
|
|
||||||
|
|
||||||
|
def _read_series_file(path: Path) -> list[str]:
|
||||||
|
lines = _read_text(path, "patches/series").splitlines()
|
||||||
|
return [line.strip() for line in lines if line.strip() and not line.startswith("#")]
|
||||||
|
|
||||||
|
|
||||||
|
def _read_sums_file(path: Path) -> list[tuple[str, str]]:
|
||||||
|
entries: list[tuple[str, str]] = []
|
||||||
|
for line in _read_text(path, "patches/SHA256SUMS").splitlines():
|
||||||
|
line = line.strip()
|
||||||
|
if not line or line.startswith("#"):
|
||||||
|
continue
|
||||||
|
parts = line.split(None, 1)
|
||||||
|
if len(parts) != 2 or not _HEX64.match(parts[0]):
|
||||||
|
raise RuntimePinError(
|
||||||
|
"patches/SHA256SUMS contains a line that is not "
|
||||||
|
"'<sha256> <patch name>'"
|
||||||
|
)
|
||||||
|
entries.append((parts[0], parts[1].strip()))
|
||||||
|
return entries
|
||||||
|
|
||||||
|
|
||||||
|
def load_runtime_pin(lock_dir: Path = DEFAULT_LOCK_DIR) -> RuntimePin:
|
||||||
|
"""Derive the exact runtime pin from a DGR-027 lock workspace, or refuse.
|
||||||
|
|
||||||
|
Refuses — rather than guessing — on a missing or malformed lock, a moving
|
||||||
|
commit reference, a disagreement between the lock's ``patch_series``, the
|
||||||
|
``patches/series`` file, ``patches/SHA256SUMS``, or the actual patch
|
||||||
|
bytes, and on an ``UPSTREAM_COMMIT`` file that names a different commit.
|
||||||
|
"""
|
||||||
|
lock_path = lock_dir / "UPSTREAM_LOCK.json"
|
||||||
|
raw = _read_text(lock_path, "UPSTREAM_LOCK.json")
|
||||||
|
try:
|
||||||
|
lock = json.loads(raw)
|
||||||
|
except json.JSONDecodeError as exc:
|
||||||
|
raise RuntimePinError(f"UPSTREAM_LOCK.json is not valid JSON: {exc}") from exc
|
||||||
|
if not isinstance(lock, dict):
|
||||||
|
raise RuntimePinError("UPSTREAM_LOCK.json must be a JSON object")
|
||||||
|
|
||||||
|
schema = lock.get("schema_version")
|
||||||
|
if schema != RUNTIME_PIN_SCHEMA_VERSION:
|
||||||
|
raise RuntimePinError(
|
||||||
|
f"UPSTREAM_LOCK.json declares schema version {schema!r}; this reader "
|
||||||
|
f"understands version {RUNTIME_PIN_SCHEMA_VERSION}"
|
||||||
|
)
|
||||||
|
|
||||||
|
upstream = lock.get("upstream")
|
||||||
|
if not isinstance(upstream, str) or not upstream.strip():
|
||||||
|
raise RuntimePinError("UPSTREAM_LOCK.json is missing the upstream URL")
|
||||||
|
runtime_name = upstream.rstrip("/").rsplit("/", 1)[-1]
|
||||||
|
if runtime_name.endswith(".git"):
|
||||||
|
runtime_name = runtime_name[: -len(".git")]
|
||||||
|
if not runtime_name:
|
||||||
|
raise RuntimePinError("the upstream URL does not name a runtime")
|
||||||
|
|
||||||
|
commit = lock.get("commit")
|
||||||
|
if not isinstance(commit, str) or not _HEX40.match(commit):
|
||||||
|
raise RuntimePinError(
|
||||||
|
f"UPSTREAM_LOCK.json commit {commit!r} is not an exact 40-character "
|
||||||
|
"hexadecimal object id; a moving reference is not a pin"
|
||||||
|
)
|
||||||
|
|
||||||
|
commit_file = _read_text(lock_dir / "UPSTREAM_COMMIT", "UPSTREAM_COMMIT")
|
||||||
|
recorded = commit_file.strip().splitlines()[0].strip() if commit_file.strip() else ""
|
||||||
|
if recorded != commit:
|
||||||
|
raise RuntimePinError(
|
||||||
|
"UPSTREAM_COMMIT and UPSTREAM_LOCK.json disagree on the pinned commit"
|
||||||
|
)
|
||||||
|
|
||||||
|
lock_series = lock.get("patch_series")
|
||||||
|
if not isinstance(lock_series, list) or not all(
|
||||||
|
isinstance(name, str) and name.strip() for name in lock_series
|
||||||
|
):
|
||||||
|
raise RuntimePinError(
|
||||||
|
"UPSTREAM_LOCK.json patch_series must be a list of patch file names"
|
||||||
|
)
|
||||||
|
if len(set(lock_series)) != len(lock_series):
|
||||||
|
raise RuntimePinError("UPSTREAM_LOCK.json patch_series contains a duplicate")
|
||||||
|
|
||||||
|
series = _read_series_file(lock_dir / "patches" / "series")
|
||||||
|
if series != lock_series:
|
||||||
|
raise RuntimePinError(
|
||||||
|
"patches/series and UPSTREAM_LOCK.json patch_series disagree on the "
|
||||||
|
"ordered patch stack"
|
||||||
|
)
|
||||||
|
|
||||||
|
sums = _read_sums_file(lock_dir / "patches" / "SHA256SUMS")
|
||||||
|
if [name for _, name in sums] != lock_series:
|
||||||
|
raise RuntimePinError(
|
||||||
|
"patches/SHA256SUMS does not record exactly the ordered patch stack "
|
||||||
|
"named by UPSTREAM_LOCK.json"
|
||||||
|
)
|
||||||
|
|
||||||
|
digests: list[str] = []
|
||||||
|
for (expected_digest, name) in sums:
|
||||||
|
patch_path = lock_dir / "patches" / name
|
||||||
|
try:
|
||||||
|
body = patch_path.read_bytes()
|
||||||
|
except FileNotFoundError:
|
||||||
|
raise RuntimePinError(f"patch file {name} is named but missing") from None
|
||||||
|
except OSError as exc:
|
||||||
|
raise RuntimePinError(f"patch file {name} is unreadable: {exc}") from exc
|
||||||
|
actual = hashlib.sha256(body).hexdigest()
|
||||||
|
if actual != expected_digest:
|
||||||
|
raise RuntimePinError(
|
||||||
|
f"patch file {name} does not match its patches/SHA256SUMS digest"
|
||||||
|
)
|
||||||
|
digests.append(actual)
|
||||||
|
|
||||||
|
return RuntimePin(
|
||||||
|
runtime_name=runtime_name,
|
||||||
|
upstream_commit=commit,
|
||||||
|
patch_series=tuple(lock_series),
|
||||||
|
patch_digests=tuple(digests),
|
||||||
|
)
|
||||||
@@ -125,6 +125,10 @@ _AXIS_MISMATCH: Mapping[str, str] = {
|
|||||||
}
|
}
|
||||||
|
|
||||||
_HEX64 = re.compile(r"^[0-9a-f]{64}$")
|
_HEX64 = re.compile(r"^[0-9a-f]{64}$")
|
||||||
|
_LLAMA_CPP_RUNTIME_PIN = re.compile(
|
||||||
|
r"^llama\.cpp@[0-9a-f]{40}\+patchstack\.[0-9a-f]{64}$"
|
||||||
|
)
|
||||||
|
_LLAMA_CPP_BACKEND_IDS = frozenset({"llama.cpp", "llama-cpp"})
|
||||||
|
|
||||||
# A revision that can move is not a pin. DGR-017 learned this on the artifact;
|
# A revision that can move is not a pin. DGR-017 learned this on the artifact;
|
||||||
# it is just as true of a tokenizer.
|
# it is just as true of a tokenizer.
|
||||||
@@ -194,6 +198,17 @@ def _require_pin(value: Any, what: str) -> str:
|
|||||||
return text
|
return text
|
||||||
|
|
||||||
|
|
||||||
|
def _require_runtime_pin(value: Any, backend_id: Any) -> str:
|
||||||
|
text = _require_pin(value, "recipe.runtime_version")
|
||||||
|
backend = _require_text(backend_id, "recipe.backend_id").strip().lower()
|
||||||
|
if backend in _LLAMA_CPP_BACKEND_IDS and not _LLAMA_CPP_RUNTIME_PIN.fullmatch(text):
|
||||||
|
raise RecipeIdentityError(
|
||||||
|
"'recipe.runtime_version' for llama.cpp must be "
|
||||||
|
"'llama.cpp@<40-hex commit>+patchstack.<64-hex digest>'"
|
||||||
|
)
|
||||||
|
return text
|
||||||
|
|
||||||
|
|
||||||
def _as_mapping(value: Any, what: str) -> Mapping[str, Any]:
|
def _as_mapping(value: Any, what: str) -> Mapping[str, Any]:
|
||||||
if not isinstance(value, Mapping):
|
if not isinstance(value, Mapping):
|
||||||
raise RecipeIdentityError(
|
raise RecipeIdentityError(
|
||||||
@@ -374,6 +389,11 @@ class RuntimeRecipe:
|
|||||||
one in fp16, produce different logits from the same bytes. Keeping the axes
|
one in fp16, produce different logits from the same bytes. Keeping the axes
|
||||||
apart is the entire safety property; see :data:`RECIPE_AXES`.
|
apart is the entire safety property; see :data:`RECIPE_AXES`.
|
||||||
|
|
||||||
|
`tokenizer_revision` and `runtime_version` must be exact pins, never moving
|
||||||
|
references. For the native runtime the canonical `runtime_version` value —
|
||||||
|
committing to the exact upstream commit *and* the ordered patch stack — is
|
||||||
|
derived from the DGR-027 lock manifest by :mod:`meshnet_node.runtime_pin`.
|
||||||
|
|
||||||
The three label fields are diagnosis only and are not digested.
|
The three label fields are diagnosis only and are not digested.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
@@ -400,6 +420,7 @@ class RuntimeRecipe:
|
|||||||
else:
|
else:
|
||||||
_require_text(value, f"recipe.{axis}")
|
_require_text(value, f"recipe.{axis}")
|
||||||
_require_pin(self.tokenizer_revision, "recipe.tokenizer_revision")
|
_require_pin(self.tokenizer_revision, "recipe.tokenizer_revision")
|
||||||
|
_require_runtime_pin(self.runtime_version, self.backend_id)
|
||||||
_require_text(self.recipe_id, "recipe.recipe_id")
|
_require_text(self.recipe_id, "recipe.recipe_id")
|
||||||
_require_text(self.recipe_version, "recipe.recipe_version")
|
_require_text(self.recipe_version, "recipe.recipe_version")
|
||||||
_require_text(self.catalogue_version, "recipe.catalogue_version")
|
_require_text(self.catalogue_version, "recipe.catalogue_version")
|
||||||
|
|||||||
@@ -4,10 +4,8 @@
|
|||||||
# never committed. A C++ consumer already needs a toolchain, so committing
|
# never committed. A C++ consumer already needs a toolchain, so committing
|
||||||
# generated C++ would only create a second copy of the schema that can rot.
|
# generated C++ would only create a second copy of the schema that can rot.
|
||||||
#
|
#
|
||||||
# gRPC C++ is optional here on purpose. The conformance test only needs message
|
# Protobuf and gRPC C++ are required together so message and service bindings are
|
||||||
# types, so the schema can be verified on a machine that has protobuf but not
|
# generated by one exact toolchain. The ignored bootstrap prefix supplies both.
|
||||||
# the gRPC C++ stack. When gRPC *is* found, the service stubs are generated too
|
|
||||||
# and exported as `shard_runtime_grpc` for the worker (DGR-008) to link.
|
|
||||||
#
|
#
|
||||||
# Build:
|
# Build:
|
||||||
# cmake -S packages/node/native -B build/native -DCMAKE_PREFIX_PATH=<protobuf-install>
|
# cmake -S packages/node/native -B build/native -DCMAKE_PREFIX_PATH=<protobuf-install>
|
||||||
@@ -23,8 +21,17 @@ project(meshnet_shard_protocol CXX)
|
|||||||
set(CMAKE_CXX_STANDARD 17)
|
set(CMAKE_CXX_STANDARD 17)
|
||||||
set(CMAKE_CXX_STANDARD_REQUIRED ON)
|
set(CMAKE_CXX_STANDARD_REQUIRED ON)
|
||||||
|
|
||||||
find_package(protobuf CONFIG REQUIRED)
|
# Protobuf and gRPC are one pinned generation toolchain. Configure only against
|
||||||
find_package(gRPC CONFIG QUIET)
|
# the ignored prefix produced by scripts/bootstrap_native_toolchain.sh; accepting
|
||||||
|
# an arbitrary system plugin would make generated service bindings host-dependent.
|
||||||
|
set(MESHNET_PROTOBUF_VERSION "33.1.0")
|
||||||
|
set(MESHNET_GRPC_VERSION "1.82.1")
|
||||||
|
find_package(protobuf ${MESHNET_PROTOBUF_VERSION} EXACT CONFIG REQUIRED)
|
||||||
|
find_package(gRPC ${MESHNET_GRPC_VERSION} EXACT CONFIG REQUIRED)
|
||||||
|
|
||||||
|
if(NOT TARGET gRPC::grpc_cpp_plugin)
|
||||||
|
message(FATAL_ERROR "pinned gRPC package does not export grpc_cpp_plugin")
|
||||||
|
endif()
|
||||||
|
|
||||||
set(SHARD_PROTO "${CMAKE_CURRENT_SOURCE_DIR}/proto/shard_runtime.proto")
|
set(SHARD_PROTO "${CMAKE_CURRENT_SOURCE_DIR}/proto/shard_runtime.proto")
|
||||||
|
|
||||||
@@ -39,24 +46,19 @@ protobuf_generate(
|
|||||||
PROTOC_OUT_DIR "${CMAKE_CURRENT_BINARY_DIR}"
|
PROTOC_OUT_DIR "${CMAKE_CURRENT_BINARY_DIR}"
|
||||||
)
|
)
|
||||||
|
|
||||||
# Service stubs: only when the gRPC C++ stack is present.
|
# Service stubs are part of the reproducibility contract, not an optional branch.
|
||||||
if(gRPC_FOUND)
|
add_library(shard_runtime_grpc STATIC "${SHARD_PROTO}")
|
||||||
add_library(shard_runtime_grpc STATIC "${SHARD_PROTO}")
|
target_link_libraries(shard_runtime_grpc PUBLIC shard_runtime_proto gRPC::grpc++)
|
||||||
target_link_libraries(shard_runtime_grpc PUBLIC shard_runtime_proto gRPC::grpc++)
|
target_include_directories(shard_runtime_grpc PUBLIC "${CMAKE_CURRENT_BINARY_DIR}")
|
||||||
target_include_directories(shard_runtime_grpc PUBLIC "${CMAKE_CURRENT_BINARY_DIR}")
|
protobuf_generate(
|
||||||
protobuf_generate(
|
TARGET shard_runtime_grpc
|
||||||
TARGET shard_runtime_grpc
|
LANGUAGE grpc
|
||||||
LANGUAGE grpc
|
GENERATE_EXTENSIONS .grpc.pb.h .grpc.pb.cc
|
||||||
GENERATE_EXTENSIONS .grpc.pb.h .grpc.pb.cc
|
PLUGIN "protoc-gen-grpc=$<TARGET_FILE:gRPC::grpc_cpp_plugin>"
|
||||||
PLUGIN "protoc-gen-grpc=$<TARGET_FILE:gRPC::grpc_cpp_plugin>"
|
IMPORT_DIRS "${CMAKE_CURRENT_SOURCE_DIR}/proto"
|
||||||
IMPORT_DIRS "${CMAKE_CURRENT_SOURCE_DIR}/proto"
|
PROTOC_OUT_DIR "${CMAKE_CURRENT_BINARY_DIR}"
|
||||||
PROTOC_OUT_DIR "${CMAKE_CURRENT_BINARY_DIR}"
|
)
|
||||||
)
|
message(STATUS "Pinned gRPC ${gRPC_VERSION}: building ShardRuntime service stubs")
|
||||||
message(STATUS "gRPC C++ found: building ShardRuntime service stubs")
|
|
||||||
else()
|
|
||||||
message(STATUS "gRPC C++ not found: building message types only "
|
|
||||||
"(sufficient for the conformance test)")
|
|
||||||
endif()
|
|
||||||
|
|
||||||
enable_testing()
|
enable_testing()
|
||||||
|
|
||||||
|
|||||||
@@ -46,7 +46,7 @@ sampled token to request/recipe identity and sampling/template/reasoning inputs.
|
|||||||
If the machine has no protobuf C++ toolchain:
|
If the machine has no protobuf C++ toolchain:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
scripts/bootstrap_native_toolchain.sh build/native-toolchain
|
bash scripts/bootstrap_native_toolchain.sh build/native-toolchain
|
||||||
```
|
```
|
||||||
|
|
||||||
Then:
|
Then:
|
||||||
@@ -58,10 +58,10 @@ cmake --build build/native -j
|
|||||||
ctest --test-dir build/native --output-on-failure
|
ctest --test-dir build/native --output-on-failure
|
||||||
```
|
```
|
||||||
|
|
||||||
gRPC C++ is optional: without it, CMake builds the message types only, which is
|
The bootstrap pins and builds Protobuf `33.1`, gRPC C++ `1.82.1`, and the
|
||||||
all the conformance test needs. When gRPC C++ *is* found, the `ShardRuntime`
|
matching `grpc_cpp_plugin` into one ignored prefix. CMake requires those exact
|
||||||
service stubs are built too and exported as `shard_runtime_grpc` for the worker
|
package versions and always generates both message and service stubs; it does
|
||||||
(DGR-008) to link.
|
not fall back to an arbitrary system plugin.
|
||||||
|
|
||||||
## How the cross-language check actually proves something
|
## How the cross-language check actually proves something
|
||||||
|
|
||||||
|
|||||||
@@ -15,11 +15,19 @@ loading, endpoint ownership, architecture-defined intermediate boundaries, and
|
|||||||
layer-filtered KV/session mapping. Meshnet routing, Tracker, gRPC, relay,
|
layer-filtered KV/session mapping. Meshnet routing, Tracker, gRPC, relay,
|
||||||
billing, authentication, and telemetry must remain outside this directory.
|
billing, authentication, and telemetry must remain outside this directory.
|
||||||
|
|
||||||
`scripts/llama_cpp_dependency.py` verifies the exact commit/tree and baseline
|
`scripts/llama_cpp_dependency.py fetch` reads the in-repo manifest and checks out
|
||||||
blobs, validates every patch digest and context with `git apply --check`, then
|
only its exact commit as detached HEAD in `build/llama.cpp/source`, an ignored
|
||||||
applies the series in `patches/series` order. It refuses a dirty source tree,
|
build workspace. It verifies the exact commit/tree and baseline blobs before use. A
|
||||||
wrong commit/tree/blob, changed patch digest, reordered series, or an existing
|
later offline `fetch` may reuse that cache only after the same clean identity
|
||||||
destination/work directory.
|
verification; an attached branch, tag/repository override, arbitrary destination,
|
||||||
|
symlinked workspace, dirty checkout, ignored injected file, or tracked-file
|
||||||
|
modification hidden by Git index flags is refused. The tool
|
||||||
|
validates every patch digest and context
|
||||||
|
with `git apply --check`, then applies the series in `patches/series` order.
|
||||||
|
Tracked executable modes are checked physically when Git reports
|
||||||
|
`core.filemode=true`; on mounted filesystems without POSIX mode fidelity, the
|
||||||
|
locked index tree remains the canonical mode record while every working-file
|
||||||
|
blob is independently re-hashed.
|
||||||
|
|
||||||
## Current semantic boundary
|
## Current semantic boundary
|
||||||
|
|
||||||
|
|||||||
@@ -3,6 +3,13 @@
|
|||||||
"upstream": "https://github.com/ggml-org/llama.cpp.git",
|
"upstream": "https://github.com/ggml-org/llama.cpp.git",
|
||||||
"commit": "e920c523e3b8a0163fe498af5bf90df35ff51d25",
|
"commit": "e920c523e3b8a0163fe498af5bf90df35ff51d25",
|
||||||
"commit_tree": "6c91a11407a3a3fb160f5dac705f9c59718f54f1",
|
"commit_tree": "6c91a11407a3a3fb160f5dac705f9c59718f54f1",
|
||||||
|
"expected_source": {
|
||||||
|
"git_tree": "6c91a11407a3a3fb160f5dac705f9c59718f54f1"
|
||||||
|
},
|
||||||
|
"retrieval": {
|
||||||
|
"method": "git-clone-detached-commit",
|
||||||
|
"workspace": "build/llama.cpp"
|
||||||
|
},
|
||||||
"patched_tree": "322d8b463df74a2226f0b513176643d815f54452",
|
"patched_tree": "322d8b463df74a2226f0b513176643d815f54452",
|
||||||
"upstream_license": "MIT",
|
"upstream_license": "MIT",
|
||||||
"patch_series": [
|
"patch_series": [
|
||||||
|
|||||||
@@ -70,6 +70,10 @@ STATUS_CERTIFIED = "certified"
|
|||||||
MIN_CERTIFYING_NODES = 2
|
MIN_CERTIFYING_NODES = 2
|
||||||
|
|
||||||
_HEX64 = re.compile(r"^[0-9a-f]{64}$")
|
_HEX64 = re.compile(r"^[0-9a-f]{64}$")
|
||||||
|
_LLAMA_CPP_RUNTIME_PIN = re.compile(
|
||||||
|
r"^llama\.cpp@[0-9a-f]{40}\+patchstack\.[0-9a-f]{64}$"
|
||||||
|
)
|
||||||
|
_LLAMA_CPP_BACKEND_IDS = frozenset({"llama.cpp", "llama-cpp"})
|
||||||
_MOVING_REFS = frozenset({"main", "master", "head", "latest", "dev", "trunk"})
|
_MOVING_REFS = frozenset({"main", "master", "head", "latest", "dev", "trunk"})
|
||||||
|
|
||||||
|
|
||||||
@@ -128,6 +132,17 @@ def _pin(value: Any, what: str) -> str:
|
|||||||
return text
|
return text
|
||||||
|
|
||||||
|
|
||||||
|
def _runtime_pin(value: Any, backend_id: Any) -> str:
|
||||||
|
text = _pin(value, "recipe.runtime_version")
|
||||||
|
backend = _text(backend_id, "recipe.backend_id").strip().lower()
|
||||||
|
if backend in _LLAMA_CPP_BACKEND_IDS and not _LLAMA_CPP_RUNTIME_PIN.fullmatch(text):
|
||||||
|
raise RecipeIdentityError(
|
||||||
|
"'recipe.runtime_version' for llama.cpp must bind a 40-hex commit "
|
||||||
|
"and a 64-hex ordered patch-stack digest"
|
||||||
|
)
|
||||||
|
return text
|
||||||
|
|
||||||
|
|
||||||
def _mapping(value: Any, what: str) -> Mapping[str, Any]:
|
def _mapping(value: Any, what: str) -> Mapping[str, Any]:
|
||||||
if not isinstance(value, Mapping):
|
if not isinstance(value, Mapping):
|
||||||
raise RecipeIdentityError(f"{what!r} must be a JSON object")
|
raise RecipeIdentityError(f"{what!r} must be a JSON object")
|
||||||
@@ -333,6 +348,7 @@ def parse_identity(data: Any) -> PresentedIdentity:
|
|||||||
else:
|
else:
|
||||||
axes[axis] = _text(value, f"recipe.{axis}")
|
axes[axis] = _text(value, f"recipe.{axis}")
|
||||||
_pin(axes["tokenizer_revision"], "recipe.tokenizer_revision")
|
_pin(axes["tokenizer_revision"], "recipe.tokenizer_revision")
|
||||||
|
_runtime_pin(axes["runtime_version"], axes["backend_id"])
|
||||||
|
|
||||||
identity = PresentedIdentity(
|
identity = PresentedIdentity(
|
||||||
artifact_id=_text(artifact.get("artifact_id"), "artifact.artifact_id"),
|
artifact_id=_text(artifact.get("artifact_id"), "artifact.artifact_id"),
|
||||||
|
|||||||
@@ -1,18 +1,13 @@
|
|||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
# Build a protobuf C++ toolchain for the native Shard protocol.
|
# Build a protobuf C++ toolchain for the native Shard protocol.
|
||||||
#
|
#
|
||||||
# The Python side needs nothing beyond `pip install grpcio-tools` — it bundles
|
# The Python side uses the exact grpcio-tools pin declared below. The C++ side
|
||||||
# protoc. The C++ side needs libprotobuf headers and a protoc binary, and a
|
# builds exact Protobuf, Abseil, and gRPC source revisions so `protoc`,
|
||||||
# machine that has neither (no protobuf-devel, no cmake, no system protoc) can
|
# `grpc_cpp_plugin`, headers, and libraries all come from one ignored prefix.
|
||||||
# still get a working one from source with this script. It is the exact recipe
|
# No system Protobuf/gRPC installation is accepted by the documented build.
|
||||||
# DGR-002 used to build and run the C++ conformance test.
|
|
||||||
#
|
|
||||||
# gRPC C++ is deliberately NOT built here. The conformance test only needs
|
|
||||||
# message types, so verifying the schema does not require the whole gRPC stack.
|
|
||||||
# The worker (DGR-008) will need gRPC C++ and should extend this script then.
|
|
||||||
#
|
#
|
||||||
# Usage:
|
# Usage:
|
||||||
# scripts/bootstrap_native_toolchain.sh [install-prefix]
|
# bash scripts/bootstrap_native_toolchain.sh [install-prefix]
|
||||||
#
|
#
|
||||||
# Then:
|
# Then:
|
||||||
# cmake -S packages/node/native -B build/native -DCMAKE_PREFIX_PATH=<prefix>
|
# cmake -S packages/node/native -B build/native -DCMAKE_PREFIX_PATH=<prefix>
|
||||||
@@ -21,7 +16,17 @@
|
|||||||
|
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
PREFIX="${1:-${PWD}/build/native-toolchain}"
|
resolve_prefix() {
|
||||||
|
local candidate="${1:-${PWD}/build/native-toolchain}"
|
||||||
|
realpath -m -- "${candidate}"
|
||||||
|
}
|
||||||
|
|
||||||
|
if [[ "${1:-}" == "--print-prefix" ]]; then
|
||||||
|
resolve_prefix "${2:-}"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
PREFIX="$(resolve_prefix "${1:-}")"
|
||||||
WORK="$(mktemp -d)"
|
WORK="$(mktemp -d)"
|
||||||
trap 'rm -rf "${WORK}"' EXIT
|
trap 'rm -rf "${WORK}"' EXIT
|
||||||
|
|
||||||
@@ -29,11 +34,15 @@ trap 'rm -rf "${WORK}"' EXIT
|
|||||||
# that stub is allowed to use, so these are exact, not floating.
|
# that stub is allowed to use, so these are exact, not floating.
|
||||||
PROTOBUF_VERSION="33.1"
|
PROTOBUF_VERSION="33.1"
|
||||||
ABSEIL_VERSION="20250814.1"
|
ABSEIL_VERSION="20250814.1"
|
||||||
|
GRPC_VERSION="1.82.1"
|
||||||
|
GRPC_COMMIT="acccf84c0df20487d64101f528e5d426541ca4e5"
|
||||||
|
|
||||||
command -v cmake >/dev/null || {
|
for tool in cmake curl git realpath sha256sum tar; do
|
||||||
echo "cmake is required (pip install cmake==4.4.0)" >&2
|
command -v "${tool}" >/dev/null || {
|
||||||
exit 1
|
echo "${tool} is required" >&2
|
||||||
}
|
exit 1
|
||||||
|
}
|
||||||
|
done
|
||||||
|
|
||||||
echo "--- fetching protobuf ${PROTOBUF_VERSION} and abseil ${ABSEIL_VERSION}"
|
echo "--- fetching protobuf ${PROTOBUF_VERSION} and abseil ${ABSEIL_VERSION}"
|
||||||
cd "${WORK}"
|
cd "${WORK}"
|
||||||
@@ -41,26 +50,77 @@ curl -sfL -o protobuf.tar.gz \
|
|||||||
"https://github.com/protocolbuffers/protobuf/releases/download/v${PROTOBUF_VERSION}/protobuf-${PROTOBUF_VERSION}.tar.gz"
|
"https://github.com/protocolbuffers/protobuf/releases/download/v${PROTOBUF_VERSION}/protobuf-${PROTOBUF_VERSION}.tar.gz"
|
||||||
tar xzf protobuf.tar.gz
|
tar xzf protobuf.tar.gz
|
||||||
|
|
||||||
# The protobuf release tarball ships utf8_range but not abseil, and its default
|
|
||||||
# CMake provider expects abseil as a submodule, so vendor it into place.
|
|
||||||
curl -sfL -o abseil.tar.gz \
|
curl -sfL -o abseil.tar.gz \
|
||||||
"https://github.com/abseil/abseil-cpp/releases/download/${ABSEIL_VERSION}/abseil-cpp-${ABSEIL_VERSION}.tar.gz"
|
"https://github.com/abseil/abseil-cpp/releases/download/${ABSEIL_VERSION}/abseil-cpp-${ABSEIL_VERSION}.tar.gz"
|
||||||
tar xzf abseil.tar.gz
|
tar xzf abseil.tar.gz
|
||||||
rm -rf "protobuf-${PROTOBUF_VERSION}/third_party/abseil-cpp"
|
|
||||||
mv "abseil-cpp-${ABSEIL_VERSION}" "protobuf-${PROTOBUF_VERSION}/third_party/abseil-cpp"
|
|
||||||
|
|
||||||
echo "--- building protobuf into ${PREFIX}"
|
echo "--- building abseil ${ABSEIL_VERSION} into ${PREFIX}"
|
||||||
cmake -S "protobuf-${PROTOBUF_VERSION}" -B build \
|
cmake -S "abseil-cpp-${ABSEIL_VERSION}" -B abseil-build \
|
||||||
-DCMAKE_BUILD_TYPE=Release \
|
-DCMAKE_BUILD_TYPE=Release \
|
||||||
-DCMAKE_INSTALL_PREFIX="${PREFIX}" \
|
-DCMAKE_INSTALL_PREFIX="${PREFIX}" \
|
||||||
-DCMAKE_POSITION_INDEPENDENT_CODE=ON \
|
-DCMAKE_POSITION_INDEPENDENT_CODE=ON \
|
||||||
-Dprotobuf_ABSL_PROVIDER=module \
|
-DABSL_ENABLE_INSTALL=ON \
|
||||||
-Dprotobuf_BUILD_TESTS=OFF \
|
-DABSL_BUILD_TESTING=OFF \
|
||||||
-Dprotobuf_BUILD_SHARED_LIBS=OFF \
|
|
||||||
-DABSL_PROPAGATE_CXX_STD=ON
|
-DABSL_PROPAGATE_CXX_STD=ON
|
||||||
cmake --build build -j"$(nproc)"
|
cmake --build abseil-build -j"$(nproc)"
|
||||||
cmake --install build
|
cmake --install abseil-build
|
||||||
|
|
||||||
|
echo "--- building protobuf ${PROTOBUF_VERSION} into ${PREFIX}"
|
||||||
|
cmake -S "protobuf-${PROTOBUF_VERSION}" -B protobuf-build \
|
||||||
|
-DCMAKE_BUILD_TYPE=Release \
|
||||||
|
-DCMAKE_INSTALL_PREFIX="${PREFIX}" \
|
||||||
|
-DCMAKE_PREFIX_PATH="${PREFIX}" \
|
||||||
|
-Dabsl_DIR="${PREFIX}/lib64/cmake/absl" \
|
||||||
|
-DCMAKE_POSITION_INDEPENDENT_CODE=ON \
|
||||||
|
-Dprotobuf_LOCAL_DEPENDENCIES_ONLY=ON \
|
||||||
|
-Dprotobuf_BUILD_TESTS=OFF \
|
||||||
|
-Dprotobuf_BUILD_SHARED_LIBS=OFF
|
||||||
|
cmake --build protobuf-build -j"$(nproc)"
|
||||||
|
cmake --install protobuf-build
|
||||||
|
|
||||||
|
echo "--- fetching gRPC ${GRPC_VERSION} at ${GRPC_COMMIT}"
|
||||||
|
git init -q grpc-source
|
||||||
|
git -C grpc-source remote add origin https://github.com/grpc/grpc.git
|
||||||
|
git -C grpc-source fetch --depth 1 origin "${GRPC_COMMIT}"
|
||||||
|
git -C grpc-source checkout --detach FETCH_HEAD
|
||||||
|
git -C grpc-source submodule update --init --recursive --depth 1
|
||||||
|
[[ "$(git -C grpc-source rev-parse HEAD)" == "${GRPC_COMMIT}" ]] || {
|
||||||
|
echo "gRPC checkout identity mismatch" >&2
|
||||||
|
exit 1
|
||||||
|
}
|
||||||
|
|
||||||
|
echo "--- building gRPC ${GRPC_VERSION} and grpc_cpp_plugin into ${PREFIX}"
|
||||||
|
cmake -S grpc-source -B grpc-build \
|
||||||
|
-DCMAKE_BUILD_TYPE=Release \
|
||||||
|
-DCMAKE_INSTALL_PREFIX="${PREFIX}" \
|
||||||
|
-DCMAKE_PREFIX_PATH="${PREFIX}" \
|
||||||
|
-DProtobuf_DIR="${PREFIX}/lib64/cmake/protobuf" \
|
||||||
|
-Dabsl_DIR="${PREFIX}/lib64/cmake/absl" \
|
||||||
|
-DgRPC_INSTALL=ON \
|
||||||
|
-DgRPC_BUILD_TESTS=OFF \
|
||||||
|
-DgRPC_PROTOBUF_PROVIDER=package \
|
||||||
|
-DgRPC_ABSL_PROVIDER=package \
|
||||||
|
-DgRPC_CARES_PROVIDER=module \
|
||||||
|
-DgRPC_RE2_PROVIDER=module \
|
||||||
|
-DgRPC_SSL_PROVIDER=module \
|
||||||
|
-DgRPC_ZLIB_PROVIDER=module \
|
||||||
|
-DgRPC_BUILD_GRPC_CPP_PLUGIN=ON \
|
||||||
|
-DgRPC_BUILD_GRPC_CSHARP_PLUGIN=OFF \
|
||||||
|
-DgRPC_BUILD_GRPC_NODE_PLUGIN=OFF \
|
||||||
|
-DgRPC_BUILD_GRPC_OBJECTIVE_C_PLUGIN=OFF \
|
||||||
|
-DgRPC_BUILD_GRPC_PHP_PLUGIN=OFF \
|
||||||
|
-DgRPC_BUILD_GRPC_PYTHON_PLUGIN=OFF \
|
||||||
|
-DgRPC_BUILD_GRPC_RUBY_PLUGIN=OFF
|
||||||
|
cmake --build grpc-build -j"$(nproc)"
|
||||||
|
cmake --install grpc-build
|
||||||
|
|
||||||
echo "--- done"
|
echo "--- done"
|
||||||
"${PREFIX}/bin/protoc" --version
|
"${PREFIX}/bin/protoc" --version
|
||||||
|
[[ -x "${PREFIX}/bin/grpc_cpp_plugin" ]] || {
|
||||||
|
echo "grpc_cpp_plugin was not installed" >&2
|
||||||
|
exit 1
|
||||||
|
}
|
||||||
|
printf 'gRPC %s commit %s\n' "${GRPC_VERSION}" "${GRPC_COMMIT}"
|
||||||
|
printf 'grpc_cpp_plugin sha256 '
|
||||||
|
sha256sum "${PREFIX}/bin/grpc_cpp_plugin" | cut -d' ' -f1
|
||||||
echo "configure the protocol build with: -DCMAKE_PREFIX_PATH=${PREFIX}"
|
echo "configure the protocol build with: -DCMAKE_PREFIX_PATH=${PREFIX}"
|
||||||
|
|||||||
@@ -49,7 +49,7 @@ _RECIPE = RuntimeRecipe(
|
|||||||
tokenizer_revision="0123456789abcdef",
|
tokenizer_revision="0123456789abcdef",
|
||||||
architecture_adapter="llama/range-v1",
|
architecture_adapter="llama/range-v1",
|
||||||
backend_id="llama.cpp",
|
backend_id="llama.cpp",
|
||||||
runtime_version="llama.cpp@deadbeef+meshnet.1",
|
runtime_version="llama.cpp@" + "d" * 40 + "+patchstack." + "e" * 64,
|
||||||
recipe_id="example-gguf",
|
recipe_id="example-gguf",
|
||||||
recipe_version="1",
|
recipe_version="1",
|
||||||
catalogue_version="2026.07.1",
|
catalogue_version="2026.07.1",
|
||||||
|
|||||||
@@ -20,6 +20,7 @@ already requires a toolchain and nothing is gained by committing them.
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import argparse
|
import argparse
|
||||||
|
from importlib import metadata
|
||||||
import pathlib
|
import pathlib
|
||||||
import shutil
|
import shutil
|
||||||
import subprocess
|
import subprocess
|
||||||
@@ -38,15 +39,32 @@ REQUIRED_GRPCIO_TOOLS = "1.82.1"
|
|||||||
_HEADER = "# Generated by scripts/generate_native_protocol.py. Do not edit.\n"
|
_HEADER = "# Generated by scripts/generate_native_protocol.py. Do not edit.\n"
|
||||||
|
|
||||||
|
|
||||||
def _generate(into: pathlib.Path) -> None:
|
def _require_grpcio_tools_version() -> None:
|
||||||
"""Run protoc, writing generated modules into `into`."""
|
|
||||||
try:
|
try:
|
||||||
from grpc_tools import protoc
|
actual = metadata.version("grpcio-tools")
|
||||||
except ImportError: # pragma: no cover - exercised only without the toolchain
|
except metadata.PackageNotFoundError:
|
||||||
sys.exit(
|
sys.exit(
|
||||||
"grpc_tools is required to generate stubs:\n"
|
"grpc_tools is required to generate stubs:\n"
|
||||||
f" pip install grpcio-tools=={REQUIRED_GRPCIO_TOOLS}"
|
f" pip install grpcio-tools=={REQUIRED_GRPCIO_TOOLS}"
|
||||||
)
|
)
|
||||||
|
if actual != REQUIRED_GRPCIO_TOOLS:
|
||||||
|
sys.exit(
|
||||||
|
"wrong grpcio-tools version for deterministic generation: "
|
||||||
|
f"found {actual}, require {REQUIRED_GRPCIO_TOOLS}\n"
|
||||||
|
f" pip install --upgrade grpcio-tools=={REQUIRED_GRPCIO_TOOLS}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _generate(into: pathlib.Path) -> None:
|
||||||
|
"""Run the exactly pinned protoc, writing generated modules into `into`."""
|
||||||
|
_require_grpcio_tools_version()
|
||||||
|
try:
|
||||||
|
from grpc_tools import protoc
|
||||||
|
except ImportError: # pragma: no cover - inconsistent/broken installation
|
||||||
|
sys.exit(
|
||||||
|
"grpcio-tools metadata exists but grpc_tools cannot be imported; reinstall it:\n"
|
||||||
|
f" pip install --force-reinstall grpcio-tools=={REQUIRED_GRPCIO_TOOLS}"
|
||||||
|
)
|
||||||
|
|
||||||
into.mkdir(parents=True, exist_ok=True)
|
into.mkdir(parents=True, exist_ok=True)
|
||||||
# grpc_tools bundles protoc and the well-known types, so generation needs no
|
# grpc_tools bundles protoc and the well-known types, so generation needs no
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
#!/usr/bin/env python3
|
#!/usr/bin/env python3
|
||||||
"""Materialize, verify, build, and smoke-test DGR-004's llama.cpp pin.
|
"""Fetch, verify, build, and smoke-test DGR-027's exact llama.cpp pin.
|
||||||
|
|
||||||
This tool deliberately owns only a source dependency boundary. It never
|
This tool deliberately owns only a source dependency boundary. It never
|
||||||
downloads a model, invokes inference, or interprets generated text.
|
downloads a model, invokes inference, or interprets generated text.
|
||||||
@@ -12,6 +12,7 @@ import hashlib
|
|||||||
import json
|
import json
|
||||||
import os
|
import os
|
||||||
import pathlib
|
import pathlib
|
||||||
|
import re
|
||||||
import shutil
|
import shutil
|
||||||
import subprocess
|
import subprocess
|
||||||
import sys
|
import sys
|
||||||
@@ -67,14 +68,27 @@ def _load_lock() -> dict[str, Any]:
|
|||||||
raise DependencyError(f"invalid upstream lock: {LOCK_PATH}: {error}") from error
|
raise DependencyError(f"invalid upstream lock: {LOCK_PATH}: {error}") from error
|
||||||
required = {
|
required = {
|
||||||
"upstream", "commit", "commit_tree", "patched_tree", "patch_series",
|
"upstream", "commit", "commit_tree", "patched_tree", "patch_series",
|
||||||
"required_upstream_blobs", "patched_paths", "build",
|
"required_upstream_blobs", "patched_paths", "build", "upstream_license",
|
||||||
|
"expected_source", "retrieval",
|
||||||
}
|
}
|
||||||
missing = sorted(required - lock.keys())
|
missing = sorted(required - lock.keys())
|
||||||
if missing:
|
if missing:
|
||||||
raise DependencyError(f"upstream lock is missing fields: {', '.join(missing)}")
|
raise DependencyError(f"upstream lock is missing fields: {', '.join(missing)}")
|
||||||
commit_file = (LLAMA_DIR / "UPSTREAM_COMMIT").read_text().strip()
|
commit_file = (LLAMA_DIR / "UPSTREAM_COMMIT").read_text().strip()
|
||||||
if lock["commit"] != commit_file or len(commit_file) != 40:
|
object_ids = [commit_file, lock["commit"], lock["commit_tree"], lock["patched_tree"]]
|
||||||
raise DependencyError("UPSTREAM_COMMIT and UPSTREAM_LOCK.json do not agree on a full commit")
|
if lock["commit"] != commit_file or not all(
|
||||||
|
isinstance(value, str) and re.fullmatch(r"[0-9a-f]{40}", value)
|
||||||
|
for value in object_ids
|
||||||
|
):
|
||||||
|
raise DependencyError("UPSTREAM_COMMIT and UPSTREAM_LOCK.json do not agree on full hexadecimal object IDs")
|
||||||
|
if lock["expected_source"] != {"git_tree": lock["commit_tree"]}:
|
||||||
|
raise DependencyError("expected_source must record the locked git tree")
|
||||||
|
retrieval = lock["retrieval"]
|
||||||
|
if retrieval != {
|
||||||
|
"method": "git-clone-detached-commit",
|
||||||
|
"workspace": "build/llama.cpp",
|
||||||
|
}:
|
||||||
|
raise DependencyError("retrieval must use the locked detached-commit build workspace")
|
||||||
return lock
|
return lock
|
||||||
|
|
||||||
|
|
||||||
@@ -103,6 +117,56 @@ def _git(source: pathlib.Path, *args: str) -> str:
|
|||||||
return _run("git", "-C", str(source), *args)
|
return _run("git", "-C", str(source), *args)
|
||||||
|
|
||||||
|
|
||||||
|
def _verify_tracked_content(source: pathlib.Path, lock: dict[str, Any]) -> None:
|
||||||
|
if _git(source, "write-tree") != lock["commit_tree"]:
|
||||||
|
raise DependencyError("materialized checkout index differs from the locked tree")
|
||||||
|
filemode_trusted = _git(source, "config", "--bool", "core.filemode") == "true"
|
||||||
|
records = _git(source, "ls-files", "-s", "-z").split("\0")
|
||||||
|
paths: list[str] = []
|
||||||
|
expected: list[str] = []
|
||||||
|
for record in records:
|
||||||
|
if not record:
|
||||||
|
continue
|
||||||
|
metadata, path = record.split("\t", 1)
|
||||||
|
mode, blob, stage = metadata.split()
|
||||||
|
if stage != "0" or mode not in {"100644", "100755", "120000"}:
|
||||||
|
raise DependencyError(f"unsupported tracked entry in materialized checkout: {record!r}")
|
||||||
|
if "\n" in path:
|
||||||
|
raise DependencyError(f"newline-bearing tracked path is unsupported: {path!r}")
|
||||||
|
candidate = source / path
|
||||||
|
cursor = source
|
||||||
|
for part in pathlib.Path(path).parts[:-1]:
|
||||||
|
cursor = cursor / part
|
||||||
|
if cursor.is_symlink():
|
||||||
|
raise DependencyError(f"tracked path traverses a symlink: {path!r}")
|
||||||
|
if mode == "120000":
|
||||||
|
if not candidate.is_symlink():
|
||||||
|
raise DependencyError(f"tracked symlink type differs from the locked tree: {path!r}")
|
||||||
|
else:
|
||||||
|
if candidate.is_symlink() or not candidate.is_file():
|
||||||
|
raise DependencyError(f"tracked file type differs from the locked tree: {path!r}")
|
||||||
|
executable = bool(candidate.stat().st_mode & 0o111)
|
||||||
|
if filemode_trusted and executable != (mode == "100755"):
|
||||||
|
raise DependencyError(f"tracked executable mode differs from the locked tree: {path!r}")
|
||||||
|
paths.append(path)
|
||||||
|
expected.append(blob)
|
||||||
|
try:
|
||||||
|
completed = subprocess.run(
|
||||||
|
["git", "hash-object", "--stdin-paths"],
|
||||||
|
cwd=source,
|
||||||
|
input="".join(f"{path}\n" for path in paths),
|
||||||
|
check=True,
|
||||||
|
text=True,
|
||||||
|
stdout=subprocess.PIPE,
|
||||||
|
stderr=subprocess.PIPE,
|
||||||
|
)
|
||||||
|
except subprocess.CalledProcessError as error:
|
||||||
|
raise DependencyError(f"unable to hash tracked source content: {error.stderr.strip()}") from error
|
||||||
|
actual = completed.stdout.splitlines()
|
||||||
|
if len(actual) != len(expected) or actual != expected:
|
||||||
|
raise DependencyError("tracked source content differs from the locked tree")
|
||||||
|
|
||||||
|
|
||||||
def _verify_source(source: pathlib.Path, lock: dict[str, Any], *, require_clean: bool) -> None:
|
def _verify_source(source: pathlib.Path, lock: dict[str, Any], *, require_clean: bool) -> None:
|
||||||
if not (source / ".git").exists():
|
if not (source / ".git").exists():
|
||||||
raise DependencyError(f"not a materialized git checkout: {source}")
|
raise DependencyError(f"not a materialized git checkout: {source}")
|
||||||
@@ -110,8 +174,15 @@ def _verify_source(source: pathlib.Path, lock: dict[str, Any], *, require_clean:
|
|||||||
raise DependencyError("upstream drift: checkout HEAD does not equal the locked commit")
|
raise DependencyError("upstream drift: checkout HEAD does not equal the locked commit")
|
||||||
if _git(source, "rev-parse", "HEAD^{tree}") != lock["commit_tree"]:
|
if _git(source, "rev-parse", "HEAD^{tree}") != lock["commit_tree"]:
|
||||||
raise DependencyError("upstream drift: checkout tree does not equal the locked tree")
|
raise DependencyError("upstream drift: checkout tree does not equal the locked tree")
|
||||||
if require_clean and _git(source, "status", "--porcelain"):
|
if _git(source, "rev-parse", "--abbrev-ref", "HEAD") != "HEAD":
|
||||||
raise DependencyError("local edits detected in materialized llama.cpp checkout")
|
raise DependencyError("materialized llama.cpp checkout must have a detached HEAD")
|
||||||
|
if require_clean:
|
||||||
|
_verify_tracked_content(source, lock)
|
||||||
|
tracked = _git(source, "status", "--porcelain", "--untracked-files=no")
|
||||||
|
untracked = _git(source, "ls-files", "--others", "--exclude-standard")
|
||||||
|
ignored = _git(source, "ls-files", "--others", "--ignored", "--exclude-standard")
|
||||||
|
if tracked or untracked or ignored:
|
||||||
|
raise DependencyError("local edits or unmanifested files detected in materialized llama.cpp checkout")
|
||||||
for relative, expected in lock["required_upstream_blobs"].items():
|
for relative, expected in lock["required_upstream_blobs"].items():
|
||||||
actual = _git(source, "rev-parse", f"HEAD:{relative}")
|
actual = _git(source, "rev-parse", f"HEAD:{relative}")
|
||||||
if actual != expected:
|
if actual != expected:
|
||||||
@@ -122,15 +193,43 @@ def _verify_source(source: pathlib.Path, lock: dict[str, Any], *, require_clean:
|
|||||||
raise DependencyError("upstream LICENSE is missing; refusing to drop required attribution")
|
raise DependencyError("upstream LICENSE is missing; refusing to drop required attribution")
|
||||||
|
|
||||||
|
|
||||||
def materialize(source: pathlib.Path, repository: str) -> None:
|
def _workspace_source(workspace: pathlib.Path, lock: dict[str, Any]) -> pathlib.Path:
|
||||||
|
relative = pathlib.Path(lock["retrieval"]["workspace"])
|
||||||
|
expected = (ROOT / relative).absolute()
|
||||||
|
supplied = workspace.absolute()
|
||||||
|
if supplied != expected:
|
||||||
|
raise DependencyError(f"--workspace must equal the locked ignored build root: {expected}")
|
||||||
|
cursor = ROOT
|
||||||
|
for part in relative.parts:
|
||||||
|
cursor = cursor / part
|
||||||
|
if cursor.is_symlink():
|
||||||
|
raise DependencyError(f"locked build workspace may not traverse a symlink: {cursor}")
|
||||||
|
resolved = workspace.resolve()
|
||||||
|
try:
|
||||||
|
resolved.relative_to(ROOT.resolve())
|
||||||
|
except ValueError as error:
|
||||||
|
raise DependencyError("locked build workspace escapes the repository root") from error
|
||||||
|
source = resolved / "source"
|
||||||
|
if source.is_symlink():
|
||||||
|
raise DependencyError(f"locked source checkout may not be a symlink: {source}")
|
||||||
|
return source
|
||||||
|
|
||||||
|
|
||||||
|
def fetch(workspace: pathlib.Path) -> pathlib.Path:
|
||||||
|
"""Fetch once, or verify an exact clean cached checkout for offline reuse."""
|
||||||
lock = _load_lock()
|
lock = _load_lock()
|
||||||
_patches(lock)
|
_patches(lock)
|
||||||
|
source = _workspace_source(workspace, lock)
|
||||||
if source.exists():
|
if source.exists():
|
||||||
raise DependencyError(f"destination already exists; refusing to reuse possibly edited source: {source}")
|
_verify_source(source, lock, require_clean=True)
|
||||||
|
print(f"reused verified offline cache: {source}")
|
||||||
|
return source
|
||||||
source.parent.mkdir(parents=True, exist_ok=True)
|
source.parent.mkdir(parents=True, exist_ok=True)
|
||||||
_run("git", "clone", "--no-checkout", repository, str(source))
|
_run("git", "clone", "--no-checkout", lock["upstream"], str(source))
|
||||||
_git(source, "checkout", "--detach", lock["commit"])
|
_git(source, "checkout", "--detach", lock["commit"])
|
||||||
_verify_source(source, lock, require_clean=True)
|
_verify_source(source, lock, require_clean=True)
|
||||||
|
print(f"fetched and verified exact source: {source}")
|
||||||
|
return source
|
||||||
|
|
||||||
|
|
||||||
def apply(source: pathlib.Path) -> None:
|
def apply(source: pathlib.Path) -> None:
|
||||||
@@ -198,18 +297,11 @@ def smoke(binary: pathlib.Path) -> None:
|
|||||||
print(output)
|
print(output)
|
||||||
|
|
||||||
|
|
||||||
def reproduce(work_dir: pathlib.Path, repository: str) -> None:
|
def reproduce(workspace: pathlib.Path) -> None:
|
||||||
resolved = work_dir.resolve()
|
source = fetch(workspace)
|
||||||
build_root = (ROOT / "build").resolve()
|
build_dir = workspace.resolve() / "build"
|
||||||
if build_root not in resolved.parents:
|
if build_dir.exists():
|
||||||
raise DependencyError(f"--work-dir must be below {build_root}: {resolved}")
|
raise DependencyError(f"build directory already exists; refusing to erase possible local edits: {build_dir}")
|
||||||
if resolved.exists():
|
|
||||||
raise DependencyError(
|
|
||||||
f"work directory already exists; refusing to erase possible local edits: {resolved}"
|
|
||||||
)
|
|
||||||
source = resolved / "source"
|
|
||||||
build_dir = resolved / "build"
|
|
||||||
materialize(source, repository)
|
|
||||||
apply(source)
|
apply(source)
|
||||||
smoke(build(source, build_dir))
|
smoke(build(source, build_dir))
|
||||||
|
|
||||||
@@ -219,6 +311,9 @@ def inspect() -> None:
|
|||||||
patches = _patches(lock)
|
patches = _patches(lock)
|
||||||
print(json.dumps({
|
print(json.dumps({
|
||||||
"commit": lock["commit"],
|
"commit": lock["commit"],
|
||||||
|
"commit_tree": lock["commit_tree"],
|
||||||
|
"retrieval": lock["retrieval"],
|
||||||
|
"upstream_license": lock["upstream_license"],
|
||||||
"patch_count": len(patches),
|
"patch_count": len(patches),
|
||||||
"patches": [patch.name for patch in patches],
|
"patches": [patch.name for patch in patches],
|
||||||
"model_downloads": False,
|
"model_downloads": False,
|
||||||
@@ -231,9 +326,8 @@ def main() -> int:
|
|||||||
parser = argparse.ArgumentParser(description=__doc__)
|
parser = argparse.ArgumentParser(description=__doc__)
|
||||||
subcommands = parser.add_subparsers(dest="command", required=True)
|
subcommands = parser.add_subparsers(dest="command", required=True)
|
||||||
subcommands.add_parser("inspect")
|
subcommands.add_parser("inspect")
|
||||||
materialize_parser = subcommands.add_parser("materialize")
|
fetch_parser = subcommands.add_parser("fetch")
|
||||||
materialize_parser.add_argument("--source-dir", type=pathlib.Path, required=True)
|
fetch_parser.add_argument("--workspace", type=pathlib.Path, default=ROOT / "build/llama.cpp")
|
||||||
materialize_parser.add_argument("--source-repository", default=_load_lock()["upstream"])
|
|
||||||
apply_parser = subcommands.add_parser("apply")
|
apply_parser = subcommands.add_parser("apply")
|
||||||
apply_parser.add_argument("--source-dir", type=pathlib.Path, required=True)
|
apply_parser.add_argument("--source-dir", type=pathlib.Path, required=True)
|
||||||
build_parser = subcommands.add_parser("build")
|
build_parser = subcommands.add_parser("build")
|
||||||
@@ -242,14 +336,13 @@ def main() -> int:
|
|||||||
smoke_parser = subcommands.add_parser("smoke")
|
smoke_parser = subcommands.add_parser("smoke")
|
||||||
smoke_parser.add_argument("--binary", type=pathlib.Path, required=True)
|
smoke_parser.add_argument("--binary", type=pathlib.Path, required=True)
|
||||||
reproduce_parser = subcommands.add_parser("reproduce")
|
reproduce_parser = subcommands.add_parser("reproduce")
|
||||||
reproduce_parser.add_argument("--work-dir", type=pathlib.Path, default=ROOT / "build/dgr-004-smoke")
|
reproduce_parser.add_argument("--workspace", type=pathlib.Path, default=ROOT / "build/llama.cpp")
|
||||||
reproduce_parser.add_argument("--source-repository", default=_load_lock()["upstream"])
|
|
||||||
args = parser.parse_args()
|
args = parser.parse_args()
|
||||||
try:
|
try:
|
||||||
if args.command == "inspect":
|
if args.command == "inspect":
|
||||||
inspect()
|
inspect()
|
||||||
elif args.command == "materialize":
|
elif args.command == "fetch":
|
||||||
materialize(args.source_dir, args.source_repository)
|
fetch(args.workspace)
|
||||||
elif args.command == "apply":
|
elif args.command == "apply":
|
||||||
apply(args.source_dir)
|
apply(args.source_dir)
|
||||||
elif args.command == "build":
|
elif args.command == "build":
|
||||||
@@ -257,9 +350,9 @@ def main() -> int:
|
|||||||
elif args.command == "smoke":
|
elif args.command == "smoke":
|
||||||
smoke(args.binary)
|
smoke(args.binary)
|
||||||
else:
|
else:
|
||||||
reproduce(args.work_dir, args.source_repository)
|
reproduce(args.workspace)
|
||||||
except DependencyError as error:
|
except DependencyError as error:
|
||||||
print(f"DGR-004 dependency error: {error}", file=sys.stderr)
|
print(f"DGR-027 dependency error: {error}", file=sys.stderr)
|
||||||
return 2
|
return 2
|
||||||
return 0
|
return 0
|
||||||
|
|
||||||
|
|||||||
@@ -8,9 +8,9 @@
|
|||||||
"model_artifact_digest": "8a0f43d6aa49d77834bdb47bcae9f42c886b7ccfe0ac014932b2a2b38697a47b",
|
"model_artifact_digest": "8a0f43d6aa49d77834bdb47bcae9f42c886b7ccfe0ac014932b2a2b38697a47b",
|
||||||
"recipe_id": "example-gguf",
|
"recipe_id": "example-gguf",
|
||||||
"recipe_version": "1",
|
"recipe_version": "1",
|
||||||
"runtime_recipe_digest": "9b14d70b0835a6428457e4888d453649dd0d2e41fc8ac9d84d232c8c237e68fa"
|
"runtime_recipe_digest": "63001e0efeada5b97f2f3562562dc0fd2d9bd8904dc6b7b99a71d98f3e938bd0"
|
||||||
},
|
},
|
||||||
"fingerprint_proto_hex": "0a40386130663433643661613439643737383334626462343762636165396634326338383662376363666530616330313439333262326132623338363937613437621240396231346437306230383335613634323834353765343838386434353336343964643064326534316663386163396438346432333263386332333765363866611a0c6578616d706c652d676775662201312a09323032362e30372e31",
|
"fingerprint_proto_hex": "0a40386130663433643661613439643737383334626462343762636165396634326338383662376363666530616330313439333262326132623338363937613437621240363330303165306566656164613562393766326633353632353632646330666432643962643839303464633662376239396137316439386633653933386264301a0c6578616d706c652d676775662201312a09323032362e30372e31",
|
||||||
"identity": {
|
"identity": {
|
||||||
"artifact": {
|
"artifact": {
|
||||||
"architecture": "dense-llama",
|
"architecture": "dense-llama",
|
||||||
@@ -26,7 +26,7 @@
|
|||||||
"model_artifact_digest": "8a0f43d6aa49d77834bdb47bcae9f42c886b7ccfe0ac014932b2a2b38697a47b",
|
"model_artifact_digest": "8a0f43d6aa49d77834bdb47bcae9f42c886b7ccfe0ac014932b2a2b38697a47b",
|
||||||
"recipe_id": "example-gguf",
|
"recipe_id": "example-gguf",
|
||||||
"recipe_version": "1",
|
"recipe_version": "1",
|
||||||
"runtime_recipe_digest": "9b14d70b0835a6428457e4888d453649dd0d2e41fc8ac9d84d232c8c237e68fa"
|
"runtime_recipe_digest": "63001e0efeada5b97f2f3562562dc0fd2d9bd8904dc6b7b99a71d98f3e938bd0"
|
||||||
},
|
},
|
||||||
"recipe": {
|
"recipe": {
|
||||||
"activation_dtype": "bfloat16",
|
"activation_dtype": "bfloat16",
|
||||||
@@ -40,7 +40,7 @@
|
|||||||
"protocol_schema_version": 1,
|
"protocol_schema_version": 1,
|
||||||
"recipe_id": "example-gguf",
|
"recipe_id": "example-gguf",
|
||||||
"recipe_version": "1",
|
"recipe_version": "1",
|
||||||
"runtime_version": "llama.cpp@deadbeef+meshnet.1",
|
"runtime_version": "llama.cpp@dddddddddddddddddddddddddddddddddddddddd+patchstack.eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee",
|
||||||
"tokenizer_revision": "0123456789abcdef",
|
"tokenizer_revision": "0123456789abcdef",
|
||||||
"weight_quantization": "Q4_K_M"
|
"weight_quantization": "Q4_K_M"
|
||||||
},
|
},
|
||||||
@@ -58,9 +58,9 @@
|
|||||||
"model_artifact_digest": "8a0f43d6aa49d77834bdb47bcae9f42c886b7ccfe0ac014932b2a2b38697a47b",
|
"model_artifact_digest": "8a0f43d6aa49d77834bdb47bcae9f42c886b7ccfe0ac014932b2a2b38697a47b",
|
||||||
"recipe_id": "example-gguf",
|
"recipe_id": "example-gguf",
|
||||||
"recipe_version": "1",
|
"recipe_version": "1",
|
||||||
"runtime_recipe_digest": "9b14d70b0835a6428457e4888d453649dd0d2e41fc8ac9d84d232c8c237e68fa"
|
"runtime_recipe_digest": "63001e0efeada5b97f2f3562562dc0fd2d9bd8904dc6b7b99a71d98f3e938bd0"
|
||||||
},
|
},
|
||||||
"fingerprint_proto_hex": "0a40386130663433643661613439643737383334626462343762636165396634326338383662376363666530616330313439333262326132623338363937613437621240396231346437306230383335613634323834353765343838386434353336343964643064326534316663386163396438346432333263386332333765363866611a0c6578616d706c652d676775662201312a09323032362e30372e31",
|
"fingerprint_proto_hex": "0a40386130663433643661613439643737383334626462343762636165396634326338383662376363666530616330313439333262326132623338363937613437621240363330303165306566656164613562393766326633353632353632646330666432643962643839303464633662376239396137316439386633653933386264301a0c6578616d706c652d676775662201312a09323032362e30372e31",
|
||||||
"identity": {
|
"identity": {
|
||||||
"artifact": {
|
"artifact": {
|
||||||
"architecture": "dense-llama",
|
"architecture": "dense-llama",
|
||||||
@@ -80,7 +80,7 @@
|
|||||||
"model_artifact_digest": "8a0f43d6aa49d77834bdb47bcae9f42c886b7ccfe0ac014932b2a2b38697a47b",
|
"model_artifact_digest": "8a0f43d6aa49d77834bdb47bcae9f42c886b7ccfe0ac014932b2a2b38697a47b",
|
||||||
"recipe_id": "example-gguf",
|
"recipe_id": "example-gguf",
|
||||||
"recipe_version": "1",
|
"recipe_version": "1",
|
||||||
"runtime_recipe_digest": "9b14d70b0835a6428457e4888d453649dd0d2e41fc8ac9d84d232c8c237e68fa"
|
"runtime_recipe_digest": "63001e0efeada5b97f2f3562562dc0fd2d9bd8904dc6b7b99a71d98f3e938bd0"
|
||||||
},
|
},
|
||||||
"recipe": {
|
"recipe": {
|
||||||
"activation_dtype": "bfloat16",
|
"activation_dtype": "bfloat16",
|
||||||
@@ -94,7 +94,7 @@
|
|||||||
"protocol_schema_version": 1,
|
"protocol_schema_version": 1,
|
||||||
"recipe_id": "example-gguf",
|
"recipe_id": "example-gguf",
|
||||||
"recipe_version": "1",
|
"recipe_version": "1",
|
||||||
"runtime_version": "llama.cpp@deadbeef+meshnet.1",
|
"runtime_version": "llama.cpp@dddddddddddddddddddddddddddddddddddddddd+patchstack.eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee",
|
||||||
"tokenizer_revision": "0123456789abcdef",
|
"tokenizer_revision": "0123456789abcdef",
|
||||||
"weight_quantization": "Q4_K_M"
|
"weight_quantization": "Q4_K_M"
|
||||||
},
|
},
|
||||||
|
|||||||
@@ -1,8 +1,9 @@
|
|||||||
"""Offline guards for DGR-004's pinned llama.cpp dependency boundary."""
|
"""Offline guards for DGR-027's pinned llama.cpp dependency boundary."""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import hashlib
|
import hashlib
|
||||||
|
import importlib.util
|
||||||
import json
|
import json
|
||||||
import pathlib
|
import pathlib
|
||||||
import subprocess
|
import subprocess
|
||||||
@@ -33,6 +34,10 @@ def test_lock_and_patch_manifest_are_self_consistent_and_exact() -> None:
|
|||||||
|
|
||||||
assert commit == lock["commit"]
|
assert commit == lock["commit"]
|
||||||
assert len(commit) == 40
|
assert len(commit) == 40
|
||||||
|
assert lock["retrieval"]["method"] == "git-clone-detached-commit"
|
||||||
|
assert lock["retrieval"]["workspace"] == "build/llama.cpp"
|
||||||
|
assert lock["expected_source"]["git_tree"] == lock["commit_tree"]
|
||||||
|
assert lock["upstream_license"] == "MIT"
|
||||||
assert patches == lock["patch_series"]
|
assert patches == lock["patch_series"]
|
||||||
assert patches == sorted(patches)
|
assert patches == sorted(patches)
|
||||||
assert patches
|
assert patches
|
||||||
@@ -42,6 +47,188 @@ def test_lock_and_patch_manifest_are_self_consistent_and_exact() -> None:
|
|||||||
assert "Subject: [PATCH" in patch.read_text()
|
assert "Subject: [PATCH" in patch.read_text()
|
||||||
|
|
||||||
|
|
||||||
|
def test_fetch_refuses_a_workspace_outside_the_ignored_build_root(tmp_path: pathlib.Path) -> None:
|
||||||
|
completed = subprocess.run(
|
||||||
|
[sys.executable, str(SCRIPT), "fetch", "--workspace", str(tmp_path / "llama.cpp")],
|
||||||
|
cwd=ROOT,
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert completed.returncode == 2
|
||||||
|
assert "--workspace must equal" in completed.stderr
|
||||||
|
|
||||||
|
|
||||||
|
def test_workspace_refuses_a_symlinked_build_ancestor(tmp_path: pathlib.Path, monkeypatch) -> None:
|
||||||
|
spec = importlib.util.spec_from_file_location("llama_cpp_dependency_symlink", SCRIPT)
|
||||||
|
assert spec and spec.loader
|
||||||
|
dependency = importlib.util.module_from_spec(spec)
|
||||||
|
spec.loader.exec_module(dependency)
|
||||||
|
root = tmp_path / "repo"
|
||||||
|
outside = tmp_path / "outside"
|
||||||
|
root.mkdir()
|
||||||
|
outside.mkdir()
|
||||||
|
(root / "build").symlink_to(outside, target_is_directory=True)
|
||||||
|
monkeypatch.setattr(dependency, "ROOT", root)
|
||||||
|
|
||||||
|
try:
|
||||||
|
dependency._workspace_source(
|
||||||
|
root / "build/llama.cpp",
|
||||||
|
{"retrieval": {"workspace": "build/llama.cpp"}},
|
||||||
|
)
|
||||||
|
except dependency.DependencyError as error:
|
||||||
|
assert "may not traverse a symlink" in str(error)
|
||||||
|
else:
|
||||||
|
raise AssertionError("symlinked build ancestor must be refused")
|
||||||
|
|
||||||
|
(root / "build").unlink()
|
||||||
|
workspace = root / "build/llama.cpp"
|
||||||
|
workspace.mkdir(parents=True)
|
||||||
|
(workspace / "source").symlink_to(outside, target_is_directory=True)
|
||||||
|
try:
|
||||||
|
dependency._workspace_source(
|
||||||
|
workspace,
|
||||||
|
{"retrieval": {"workspace": "build/llama.cpp"}},
|
||||||
|
)
|
||||||
|
except dependency.DependencyError as error:
|
||||||
|
assert "source checkout may not be a symlink" in str(error)
|
||||||
|
else:
|
||||||
|
raise AssertionError("symlinked source checkout must be refused")
|
||||||
|
|
||||||
|
|
||||||
|
def test_fetch_refuses_a_branch_or_repository_override() -> None:
|
||||||
|
completed = subprocess.run(
|
||||||
|
[sys.executable, str(SCRIPT), "fetch", "--source-repository", "main"],
|
||||||
|
cwd=ROOT,
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert completed.returncode == 2
|
||||||
|
assert "unrecognized arguments" in completed.stderr
|
||||||
|
|
||||||
|
|
||||||
|
def test_fetch_reuses_only_a_verified_cached_tree_offline(
|
||||||
|
tmp_path: pathlib.Path, monkeypatch
|
||||||
|
) -> None:
|
||||||
|
spec = importlib.util.spec_from_file_location("llama_cpp_dependency", SCRIPT)
|
||||||
|
assert spec and spec.loader
|
||||||
|
dependency = importlib.util.module_from_spec(spec)
|
||||||
|
spec.loader.exec_module(dependency)
|
||||||
|
|
||||||
|
root = tmp_path / "repo"
|
||||||
|
source = root / "build/llama.cpp/source"
|
||||||
|
upstream = tmp_path / "upstream"
|
||||||
|
upstream.mkdir()
|
||||||
|
subprocess.run(["git", "init", "-q", str(upstream)], check=True)
|
||||||
|
subprocess.run(["git", "-C", str(upstream), "config", "user.email", "test@example.invalid"], check=True)
|
||||||
|
subprocess.run(["git", "-C", str(upstream), "config", "user.name", "test"], check=True)
|
||||||
|
(upstream / "CMakeLists.txt").write_text("cmake_minimum_required(VERSION 3.14)\n")
|
||||||
|
(upstream / "LICENSE").write_text("MIT\n")
|
||||||
|
(upstream / "tool.sh").write_text("#!/bin/sh\nexit 0\n")
|
||||||
|
(upstream / "tool.sh").chmod(0o755)
|
||||||
|
subprocess.run(["git", "-C", str(upstream), "add", "."], check=True)
|
||||||
|
subprocess.run(["git", "-C", str(upstream), "commit", "-qm", "fixture"], check=True)
|
||||||
|
commit = subprocess.run(
|
||||||
|
["git", "-C", str(upstream), "rev-parse", "HEAD"], check=True, capture_output=True, text=True
|
||||||
|
).stdout.strip()
|
||||||
|
tree = subprocess.run(
|
||||||
|
["git", "-C", str(upstream), "rev-parse", "HEAD^{tree}"], check=True, capture_output=True, text=True
|
||||||
|
).stdout.strip()
|
||||||
|
blob = subprocess.run(
|
||||||
|
["git", "-C", str(upstream), "rev-parse", "HEAD:CMakeLists.txt"],
|
||||||
|
check=True,
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
|
).stdout.strip()
|
||||||
|
source.parent.mkdir(parents=True)
|
||||||
|
subprocess.run(["git", "clone", "-q", str(upstream), str(source)], check=True)
|
||||||
|
|
||||||
|
llama_dir = root / "packages/node/native/llama"
|
||||||
|
patch_dir = llama_dir / "patches"
|
||||||
|
patch_dir.mkdir(parents=True)
|
||||||
|
(llama_dir / "UPSTREAM_COMMIT").write_text(f"{commit}\n")
|
||||||
|
(patch_dir / "series").write_text("0001-fixture.patch\n")
|
||||||
|
patch = patch_dir / "0001-fixture.patch"
|
||||||
|
patch.write_text("fixture patch\n")
|
||||||
|
(patch_dir / "SHA256SUMS").write_text(f"{_sha256(patch)} {patch.name}\n")
|
||||||
|
(llama_dir / "UPSTREAM_LOCK.json").write_text(json.dumps({
|
||||||
|
"upstream": str(upstream),
|
||||||
|
"commit": commit,
|
||||||
|
"commit_tree": tree,
|
||||||
|
"expected_source": {"git_tree": tree},
|
||||||
|
"retrieval": {"method": "git-clone-detached-commit", "workspace": "build/llama.cpp"},
|
||||||
|
"patched_tree": tree,
|
||||||
|
"patch_series": [patch.name],
|
||||||
|
"required_upstream_blobs": {"CMakeLists.txt": blob},
|
||||||
|
"patched_paths": [],
|
||||||
|
"build": {},
|
||||||
|
"upstream_license": "MIT",
|
||||||
|
}))
|
||||||
|
monkeypatch.setattr(dependency, "ROOT", root)
|
||||||
|
monkeypatch.setattr(dependency, "LLAMA_DIR", llama_dir)
|
||||||
|
monkeypatch.setattr(dependency, "LOCK_PATH", llama_dir / "UPSTREAM_LOCK.json")
|
||||||
|
monkeypatch.setattr(dependency, "PATCH_DIR", patch_dir)
|
||||||
|
|
||||||
|
try:
|
||||||
|
dependency.fetch(root / "build/llama.cpp")
|
||||||
|
except dependency.DependencyError as error:
|
||||||
|
assert "detached HEAD" in str(error)
|
||||||
|
else:
|
||||||
|
raise AssertionError("attached branch cache must be refused")
|
||||||
|
subprocess.run(["git", "-C", str(source), "checkout", "--detach", "-q", commit], check=True)
|
||||||
|
assert dependency.fetch(root / "build/llama.cpp") == source
|
||||||
|
|
||||||
|
tracked_path = source / "CMakeLists.txt"
|
||||||
|
for flag, clear_flag in (
|
||||||
|
("--assume-unchanged", "--no-assume-unchanged"),
|
||||||
|
("--skip-worktree", "--no-skip-worktree"),
|
||||||
|
):
|
||||||
|
subprocess.run(["git", "-C", str(source), "update-index", flag, "CMakeLists.txt"], check=True)
|
||||||
|
tracked_path.write_text("injected tracked build input\n")
|
||||||
|
try:
|
||||||
|
dependency.fetch(root / "build/llama.cpp")
|
||||||
|
except dependency.DependencyError as error:
|
||||||
|
assert "tracked source content differs" in str(error)
|
||||||
|
else:
|
||||||
|
raise AssertionError(f"tracked cache injection hidden by {flag} must be refused")
|
||||||
|
subprocess.run(["git", "-C", str(source), "update-index", clear_flag, "CMakeLists.txt"], check=True)
|
||||||
|
subprocess.run(["git", "-C", str(source), "checkout", "--", "CMakeLists.txt"], check=True)
|
||||||
|
|
||||||
|
executable_path = source / "tool.sh"
|
||||||
|
for flag, clear_flag in (
|
||||||
|
("--assume-unchanged", "--no-assume-unchanged"),
|
||||||
|
("--skip-worktree", "--no-skip-worktree"),
|
||||||
|
):
|
||||||
|
subprocess.run(["git", "-C", str(source), "update-index", flag, "tool.sh"], check=True)
|
||||||
|
executable_path.chmod(0o644)
|
||||||
|
try:
|
||||||
|
dependency.fetch(root / "build/llama.cpp")
|
||||||
|
except dependency.DependencyError as error:
|
||||||
|
assert "executable mode differs" in str(error)
|
||||||
|
else:
|
||||||
|
raise AssertionError(f"tracked mode change hidden by {flag} must be refused")
|
||||||
|
subprocess.run(["git", "-C", str(source), "update-index", clear_flag, "tool.sh"], check=True)
|
||||||
|
subprocess.run(["git", "-C", str(source), "checkout", "--", "tool.sh"], check=True)
|
||||||
|
|
||||||
|
(source / "untracked.txt").write_text("edited\n")
|
||||||
|
try:
|
||||||
|
dependency.fetch(root / "build/llama.cpp")
|
||||||
|
except dependency.DependencyError as error:
|
||||||
|
assert "local edits" in str(error)
|
||||||
|
else:
|
||||||
|
raise AssertionError("dirty cached source must be refused")
|
||||||
|
(source / "untracked.txt").unlink()
|
||||||
|
(source / ".git/info/exclude").write_text("injected.cmake\n")
|
||||||
|
(source / "injected.cmake").write_text("unmanifested input\n")
|
||||||
|
try:
|
||||||
|
dependency.fetch(root / "build/llama.cpp")
|
||||||
|
except dependency.DependencyError as error:
|
||||||
|
assert "unmanifested files" in str(error)
|
||||||
|
else:
|
||||||
|
raise AssertionError("ignored cached source input must be refused")
|
||||||
|
|
||||||
|
|
||||||
def test_dependency_script_reports_the_locked_boundary_without_network() -> None:
|
def test_dependency_script_reports_the_locked_boundary_without_network() -> None:
|
||||||
completed = subprocess.run(
|
completed = subprocess.run(
|
||||||
[sys.executable, str(SCRIPT), "inspect"],
|
[sys.executable, str(SCRIPT), "inspect"],
|
||||||
|
|||||||
@@ -15,6 +15,7 @@ from meshnet_node.native_backend import (
|
|||||||
shard_identity_from_native_report,
|
shard_identity_from_native_report,
|
||||||
)
|
)
|
||||||
from meshnet_node.native_protocol import SCHEMA_VERSION, pb
|
from meshnet_node.native_protocol import SCHEMA_VERSION, pb
|
||||||
|
from meshnet_node.runtime_pin import load_runtime_pin
|
||||||
from meshnet_node.recipe_manifest import parse_recipe_manifest
|
from meshnet_node.recipe_manifest import parse_recipe_manifest
|
||||||
from meshnet_tracker.capability import STATE_UNCERTIFIED, evaluate_report
|
from meshnet_tracker.capability import STATE_UNCERTIFIED, evaluate_report
|
||||||
|
|
||||||
@@ -42,7 +43,6 @@ def _inputs(**changes: object) -> NativeIdentityInputs:
|
|||||||
kv_layout="llama-kv-v1",
|
kv_layout="llama-kv-v1",
|
||||||
architecture_adapter="dense-llama-v1",
|
architecture_adapter="dense-llama-v1",
|
||||||
backend_id="llama-cpp",
|
backend_id="llama-cpp",
|
||||||
runtime_version="llama.cpp:e920c523",
|
|
||||||
recipe_id="native",
|
recipe_id="native",
|
||||||
recipe_version="1",
|
recipe_version="1",
|
||||||
catalogue_version="2026.07.1",
|
catalogue_version="2026.07.1",
|
||||||
@@ -84,6 +84,7 @@ def test_native_identity_uses_loaded_report_not_a_caller_range():
|
|||||||
assert (identity.shard_start, identity.shard_end) == (2, 6)
|
assert (identity.shard_start, identity.shard_end) == (2, 6)
|
||||||
assert identity.artifact.architecture == "llama"
|
assert identity.artifact.architecture == "llama"
|
||||||
assert identity.artifact.layer_count == 8
|
assert identity.artifact.layer_count == 8
|
||||||
|
assert identity.recipe.runtime_version == load_runtime_pin().runtime_version
|
||||||
|
|
||||||
|
|
||||||
def test_native_identity_requires_an_immutable_pin_and_gguf_range():
|
def test_native_identity_requires_an_immutable_pin_and_gguf_range():
|
||||||
|
|||||||
@@ -612,6 +612,27 @@ def test_a_peer_still_sending_the_retired_field_does_not_corrupt_the_tensor():
|
|||||||
assert decode_tensor(pb.NamedTensor.FromString(wire)) == b"\xaa" * 32
|
assert decode_tensor(pb.NamedTensor.FromString(wire)) == b"\xaa" * 32
|
||||||
|
|
||||||
|
|
||||||
|
def test_native_toolchain_bootstrap_resolves_relative_prefix_before_temp_chdir(tmp_path):
|
||||||
|
script = REPO_ROOT / "scripts/bootstrap_native_toolchain.sh"
|
||||||
|
result = subprocess.run(
|
||||||
|
["bash", str(script), "--print-prefix", "relative/toolchain"],
|
||||||
|
cwd=tmp_path,
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert result.returncode == 0, result.stdout + result.stderr
|
||||||
|
assert pathlib.Path(result.stdout.strip()) == tmp_path / "relative/toolchain"
|
||||||
|
|
||||||
|
|
||||||
|
def test_python_generator_rejects_the_wrong_grpcio_tools_version(monkeypatch):
|
||||||
|
from scripts import generate_native_protocol
|
||||||
|
|
||||||
|
monkeypatch.setattr(generate_native_protocol.metadata, "version", lambda _: "0.0.0")
|
||||||
|
with pytest.raises(SystemExit, match="found 0.0.0, require 1.82.1"):
|
||||||
|
generate_native_protocol._require_grpcio_tools_version()
|
||||||
|
|
||||||
|
|
||||||
def test_generated_python_stubs_match_the_proto():
|
def test_generated_python_stubs_match_the_proto():
|
||||||
pytest.importorskip("grpc_tools", reason="protoc toolchain is required to verify")
|
pytest.importorskip("grpc_tools", reason="protoc toolchain is required to verify")
|
||||||
result = subprocess.run(
|
result = subprocess.run(
|
||||||
|
|||||||
312
tests/test_runtime_pin_identity.py
Normal file
312
tests/test_runtime_pin_identity.py
Normal file
@@ -0,0 +1,312 @@
|
|||||||
|
"""DGR-025: the recipe's runtime axis commits to the exact pin and patch stack.
|
||||||
|
|
||||||
|
The fingerprint (DGR-003 lineage) digests a ``runtime_version`` string, but a
|
||||||
|
string an operator typed is a label, not a pin: two workers could run different
|
||||||
|
patch stacks under the same label and still agree on the digest. These tests
|
||||||
|
pin the axis to the DGR-027 lock manifest — the exact upstream commit plus a
|
||||||
|
digest over the ordered patch-stack bytes — and prove both identity
|
||||||
|
implementations reject a moving runtime reference.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from meshnet_node.runtime_pin import (
|
||||||
|
DEFAULT_LOCK_DIR,
|
||||||
|
RuntimePinError,
|
||||||
|
load_runtime_pin,
|
||||||
|
)
|
||||||
|
from meshnet_node.runtime_recipe import RecipeIdentityError, RuntimeRecipe
|
||||||
|
from meshnet_tracker.recipe import (
|
||||||
|
RecipeIdentityError as TrackerRecipeIdentityError,
|
||||||
|
parse_identity,
|
||||||
|
)
|
||||||
|
|
||||||
|
REPO_LOCK_DIR = (
|
||||||
|
Path(__file__).resolve().parent.parent / "packages" / "node" / "native" / "llama"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _recipe(**changes: object) -> RuntimeRecipe:
|
||||||
|
fields: dict[str, object] = {
|
||||||
|
"weight_quantization": "Q4_K_M",
|
||||||
|
"activation_dtype": "bfloat16",
|
||||||
|
"compute_dtype": "float32",
|
||||||
|
"kv_dtype": "q8_0",
|
||||||
|
"kv_layout": "paged-v1",
|
||||||
|
"tokenizer_revision": "0123456789abcdef",
|
||||||
|
"architecture_adapter": "llama/range-v1",
|
||||||
|
"backend_id": "llama.cpp",
|
||||||
|
"runtime_version": "llama.cpp@" + "d" * 40 + "+patchstack." + "e" * 64,
|
||||||
|
"recipe_id": "example-gguf",
|
||||||
|
"recipe_version": "1",
|
||||||
|
"catalogue_version": "2026.07.1",
|
||||||
|
}
|
||||||
|
fields.update(changes)
|
||||||
|
return RuntimeRecipe(**fields) # type: ignore[arg-type]
|
||||||
|
|
||||||
|
|
||||||
|
def _write_fixture_workspace(
|
||||||
|
root: Path,
|
||||||
|
*,
|
||||||
|
commit: str = "e" * 40,
|
||||||
|
patches: dict[str, bytes] | None = None,
|
||||||
|
lock_series: list[str] | None = None,
|
||||||
|
series_lines: list[str] | None = None,
|
||||||
|
sums_lines: list[str] | None = None,
|
||||||
|
schema_version: int = 1,
|
||||||
|
upstream: str = "https://github.com/ggml-org/llama.cpp.git",
|
||||||
|
upstream_commit_file: str | None = None,
|
||||||
|
) -> Path:
|
||||||
|
"""A minimal DGR-027-shaped lock workspace; overrides create disagreement."""
|
||||||
|
if patches is None:
|
||||||
|
patches = {
|
||||||
|
"0001-first.patch": b"--- a\n+++ b\n",
|
||||||
|
"0002-second.patch": b"--- c\n+++ d\n",
|
||||||
|
}
|
||||||
|
names = list(patches)
|
||||||
|
digests = {
|
||||||
|
name: hashlib.sha256(body).hexdigest() for name, body in patches.items()
|
||||||
|
}
|
||||||
|
|
||||||
|
patch_dir = root / "patches"
|
||||||
|
patch_dir.mkdir(parents=True)
|
||||||
|
for name, body in patches.items():
|
||||||
|
(patch_dir / name).write_bytes(body)
|
||||||
|
|
||||||
|
(root / "UPSTREAM_LOCK.json").write_text(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"schema_version": schema_version,
|
||||||
|
"upstream": upstream,
|
||||||
|
"commit": commit,
|
||||||
|
"patch_series": names if lock_series is None else lock_series,
|
||||||
|
}
|
||||||
|
),
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
(root / "UPSTREAM_COMMIT").write_text(
|
||||||
|
(commit if upstream_commit_file is None else upstream_commit_file) + "\n",
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
(patch_dir / "series").write_text(
|
||||||
|
"\n".join(names if series_lines is None else series_lines) + "\n",
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
if sums_lines is None:
|
||||||
|
sums_lines = ["# ordered digests"] + [
|
||||||
|
f"{digests[name]} {name}" for name in names
|
||||||
|
]
|
||||||
|
(patch_dir / "SHA256SUMS").write_text(
|
||||||
|
"\n".join(sums_lines) + "\n", encoding="utf-8"
|
||||||
|
)
|
||||||
|
return root
|
||||||
|
|
||||||
|
|
||||||
|
# --- the committed manifest is the identity source -------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_default_lock_dir_is_the_committed_manifest():
|
||||||
|
assert DEFAULT_LOCK_DIR == REPO_LOCK_DIR
|
||||||
|
|
||||||
|
|
||||||
|
def test_committed_manifest_derives_a_deterministic_runtime_pin():
|
||||||
|
pin = load_runtime_pin(REPO_LOCK_DIR)
|
||||||
|
again = load_runtime_pin(REPO_LOCK_DIR)
|
||||||
|
assert pin == again
|
||||||
|
lock = json.loads(
|
||||||
|
(REPO_LOCK_DIR / "UPSTREAM_LOCK.json").read_text(encoding="utf-8")
|
||||||
|
)
|
||||||
|
assert pin.upstream_commit == lock["commit"]
|
||||||
|
assert list(pin.patch_series) == lock["patch_series"]
|
||||||
|
# The axis value names the runtime, the exact commit, and the stack digest,
|
||||||
|
# so changing any of the three changes every downstream recipe digest.
|
||||||
|
assert pin.runtime_version == (
|
||||||
|
f"llama.cpp@{lock['commit']}+patchstack.{pin.patch_stack_digest}"
|
||||||
|
)
|
||||||
|
assert len(pin.patch_stack_digest) == 64
|
||||||
|
|
||||||
|
|
||||||
|
def test_derived_axis_value_is_a_valid_recipe_pin():
|
||||||
|
pin = load_runtime_pin(REPO_LOCK_DIR)
|
||||||
|
recipe = _recipe(runtime_version=pin.runtime_version)
|
||||||
|
assert recipe.runtime_version == pin.runtime_version
|
||||||
|
assert len(recipe.runtime_recipe_digest) == 64
|
||||||
|
|
||||||
|
|
||||||
|
def test_patch_byte_change_changes_the_runtime_identity(tmp_path):
|
||||||
|
baseline = load_runtime_pin(_write_fixture_workspace(tmp_path / "a"))
|
||||||
|
changed = load_runtime_pin(
|
||||||
|
_write_fixture_workspace(
|
||||||
|
tmp_path / "b",
|
||||||
|
patches={
|
||||||
|
"0001-first.patch": b"--- a\n+++ b\n",
|
||||||
|
"0002-second.patch": b"--- c\n+++ DIFFERENT\n",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
)
|
||||||
|
assert baseline.patch_stack_digest != changed.patch_stack_digest
|
||||||
|
assert baseline.runtime_version != changed.runtime_version
|
||||||
|
|
||||||
|
|
||||||
|
def test_patch_order_is_part_of_the_stack_identity(tmp_path):
|
||||||
|
patches = {
|
||||||
|
"0001-first.patch": b"--- a\n+++ b\n",
|
||||||
|
"0002-second.patch": b"--- c\n+++ d\n",
|
||||||
|
}
|
||||||
|
forward = load_runtime_pin(
|
||||||
|
_write_fixture_workspace(tmp_path / "a", patches=patches)
|
||||||
|
)
|
||||||
|
names = list(patches)
|
||||||
|
reversed_names = list(reversed(names))
|
||||||
|
digests = {
|
||||||
|
name: hashlib.sha256(body).hexdigest() for name, body in patches.items()
|
||||||
|
}
|
||||||
|
swapped = load_runtime_pin(
|
||||||
|
_write_fixture_workspace(
|
||||||
|
tmp_path / "b",
|
||||||
|
patches=patches,
|
||||||
|
lock_series=reversed_names,
|
||||||
|
series_lines=reversed_names,
|
||||||
|
sums_lines=[f"{digests[name]} {name}" for name in reversed_names],
|
||||||
|
)
|
||||||
|
)
|
||||||
|
assert forward.patch_stack_digest != swapped.patch_stack_digest
|
||||||
|
|
||||||
|
|
||||||
|
# --- every manifest disagreement fails closed ------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_missing_lock_file_fails_closed(tmp_path):
|
||||||
|
with pytest.raises(RuntimePinError, match="UPSTREAM_LOCK.json"):
|
||||||
|
load_runtime_pin(tmp_path)
|
||||||
|
|
||||||
|
|
||||||
|
def test_malformed_lock_json_fails_closed(tmp_path):
|
||||||
|
_write_fixture_workspace(tmp_path)
|
||||||
|
(tmp_path / "UPSTREAM_LOCK.json").write_text("{not json", encoding="utf-8")
|
||||||
|
with pytest.raises(RuntimePinError, match="not valid JSON"):
|
||||||
|
load_runtime_pin(tmp_path)
|
||||||
|
|
||||||
|
|
||||||
|
def test_unknown_lock_schema_fails_closed(tmp_path):
|
||||||
|
_write_fixture_workspace(tmp_path, schema_version=2)
|
||||||
|
with pytest.raises(RuntimePinError, match="schema"):
|
||||||
|
load_runtime_pin(tmp_path)
|
||||||
|
|
||||||
|
|
||||||
|
def test_moving_commit_reference_fails_closed(tmp_path):
|
||||||
|
_write_fixture_workspace(tmp_path, commit="master")
|
||||||
|
with pytest.raises(RuntimePinError, match="exact 40"):
|
||||||
|
load_runtime_pin(tmp_path)
|
||||||
|
|
||||||
|
|
||||||
|
def test_upstream_commit_file_disagreement_fails_closed(tmp_path):
|
||||||
|
_write_fixture_workspace(tmp_path, upstream_commit_file="f" * 40)
|
||||||
|
with pytest.raises(RuntimePinError, match="UPSTREAM_COMMIT"):
|
||||||
|
load_runtime_pin(tmp_path)
|
||||||
|
|
||||||
|
|
||||||
|
def test_series_file_disagreement_fails_closed(tmp_path):
|
||||||
|
_write_fixture_workspace(
|
||||||
|
tmp_path, series_lines=["0002-second.patch", "0001-first.patch"]
|
||||||
|
)
|
||||||
|
with pytest.raises(RuntimePinError, match="series"):
|
||||||
|
load_runtime_pin(tmp_path)
|
||||||
|
|
||||||
|
|
||||||
|
def test_missing_patch_file_fails_closed(tmp_path):
|
||||||
|
_write_fixture_workspace(tmp_path)
|
||||||
|
(tmp_path / "patches" / "0002-second.patch").unlink()
|
||||||
|
with pytest.raises(RuntimePinError, match="0002-second.patch"):
|
||||||
|
load_runtime_pin(tmp_path)
|
||||||
|
|
||||||
|
|
||||||
|
def test_checksum_disagreement_fails_closed(tmp_path):
|
||||||
|
_write_fixture_workspace(tmp_path)
|
||||||
|
patch = tmp_path / "patches" / "0002-second.patch"
|
||||||
|
patch.write_bytes(patch.read_bytes() + b"tampered\n")
|
||||||
|
with pytest.raises(RuntimePinError, match="SHA256SUMS"):
|
||||||
|
load_runtime_pin(tmp_path)
|
||||||
|
|
||||||
|
|
||||||
|
def test_sums_entry_missing_fails_closed(tmp_path):
|
||||||
|
patches = {
|
||||||
|
"0001-first.patch": b"--- a\n+++ b\n",
|
||||||
|
"0002-second.patch": b"--- c\n+++ d\n",
|
||||||
|
}
|
||||||
|
digest = hashlib.sha256(patches["0001-first.patch"]).hexdigest()
|
||||||
|
_write_fixture_workspace(
|
||||||
|
tmp_path,
|
||||||
|
patches=patches,
|
||||||
|
sums_lines=[f"{digest} 0001-first.patch"],
|
||||||
|
)
|
||||||
|
with pytest.raises(RuntimePinError, match="SHA256SUMS"):
|
||||||
|
load_runtime_pin(tmp_path)
|
||||||
|
|
||||||
|
|
||||||
|
def test_empty_patch_series_requires_empty_series_files(tmp_path):
|
||||||
|
# An unpatched runtime is a legal pin; a lock that *hides* patches is not.
|
||||||
|
_write_fixture_workspace(tmp_path, lock_series=[])
|
||||||
|
with pytest.raises(RuntimePinError, match="series"):
|
||||||
|
load_runtime_pin(tmp_path)
|
||||||
|
|
||||||
|
|
||||||
|
# --- both identity implementations reject a moving runtime -----------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_node_recipe_rejects_a_moving_runtime_version():
|
||||||
|
with pytest.raises(RecipeIdentityError, match="moving reference"):
|
||||||
|
_recipe(runtime_version="latest")
|
||||||
|
|
||||||
|
|
||||||
|
def test_tracker_rejects_a_moving_runtime_version():
|
||||||
|
vectors = json.loads(
|
||||||
|
(Path(__file__).parent / "data" / "recipe_fingerprint_vectors.json").read_text(
|
||||||
|
encoding="utf-8"
|
||||||
|
)
|
||||||
|
)
|
||||||
|
doc = json.loads(json.dumps(vectors["vectors"][0]["identity"]))
|
||||||
|
doc["recipe"]["runtime_version"] = "latest"
|
||||||
|
doc.pop("fingerprint", None)
|
||||||
|
with pytest.raises(TrackerRecipeIdentityError, match="moving reference"):
|
||||||
|
parse_identity(doc)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
"forged",
|
||||||
|
[
|
||||||
|
"llama.cpp@master+patchstack.not-a-digest",
|
||||||
|
"llama.cpp@e920c523+patchstack.forged",
|
||||||
|
"release-that-operator-typed",
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_node_recipe_rejects_noncanonical_llama_runtime_pins(forged):
|
||||||
|
with pytest.raises(RecipeIdentityError, match="40-hex commit"):
|
||||||
|
_recipe(runtime_version=forged)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
"forged",
|
||||||
|
[
|
||||||
|
"llama.cpp@master+patchstack.not-a-digest",
|
||||||
|
"llama.cpp@e920c523+patchstack.forged",
|
||||||
|
"release-that-operator-typed",
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_tracker_rejects_noncanonical_llama_runtime_pins(forged):
|
||||||
|
vectors = json.loads(
|
||||||
|
(Path(__file__).parent / "data" / "recipe_fingerprint_vectors.json").read_text(
|
||||||
|
encoding="utf-8"
|
||||||
|
)
|
||||||
|
)
|
||||||
|
doc = json.loads(json.dumps(vectors["vectors"][0]["identity"]))
|
||||||
|
doc["recipe"]["runtime_version"] = forged
|
||||||
|
doc.pop("fingerprint", None)
|
||||||
|
with pytest.raises(TrackerRecipeIdentityError, match="40-hex commit"):
|
||||||
|
parse_identity(doc)
|
||||||
@@ -65,7 +65,7 @@ def _recipe(**changes: object) -> RuntimeRecipe:
|
|||||||
"tokenizer_revision": "0123456789abcdef",
|
"tokenizer_revision": "0123456789abcdef",
|
||||||
"architecture_adapter": "llama/range-v1",
|
"architecture_adapter": "llama/range-v1",
|
||||||
"backend_id": "llama.cpp",
|
"backend_id": "llama.cpp",
|
||||||
"runtime_version": "llama.cpp@deadbeef+meshnet.1",
|
"runtime_version": "llama.cpp@" + "d" * 40 + "+patchstack." + "e" * 64,
|
||||||
"recipe_id": "example-gguf",
|
"recipe_id": "example-gguf",
|
||||||
"recipe_version": "1",
|
"recipe_version": "1",
|
||||||
"catalogue_version": "2026.07.1",
|
"catalogue_version": "2026.07.1",
|
||||||
@@ -241,7 +241,7 @@ def test_committed_vectors_cover_a_whole_model_and_a_derivative_shard():
|
|||||||
("tokenizer_revision", "fedcba9876543210"),
|
("tokenizer_revision", "fedcba9876543210"),
|
||||||
("architecture_adapter", "llama/range-v2"),
|
("architecture_adapter", "llama/range-v2"),
|
||||||
("backend_id", "other-backend"),
|
("backend_id", "other-backend"),
|
||||||
("runtime_version", "llama.cpp@other+meshnet.1"),
|
("runtime_version", "llama.cpp@" + "c" * 40 + "+patchstack." + "b" * 64),
|
||||||
("boundary_schema_version", 2),
|
("boundary_schema_version", 2),
|
||||||
("protocol_schema_version", 2),
|
("protocol_schema_version", 2),
|
||||||
],
|
],
|
||||||
|
|||||||
Reference in New Issue
Block a user