Compare commits
7 Commits
master
...
ralph/terr
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
902ecde363 | ||
|
|
db59caa8e9 | ||
|
|
ad66f7a4d8 | ||
|
|
f83cf331c3 | ||
|
|
ae51526e85 | ||
|
|
8563d218c9 | ||
|
|
66d9888a11 |
@@ -8,4 +8,3 @@
|
||||
- **Node capability admission** — `.scratch/node-capability-admission/` (P0 plan; [ADR-0023](../../docs/adr/0023-model-agnostic-node-capability-admission.md), [ADR-0026](../../docs/adr/0026-node-assignment-ownership-and-managed-placement.md))
|
||||
- **Distributed relay performance** — relay `/rpc` requester sockets are persistent per Route Session and Activation Seam as of 2026-07-10; `request_id` remains unique per activation while `X-Meshnet-Session` remains stable for KV state. Next low-risk priorities: persistent direct/loopback HTTP, seam byte/latency telemetry, then trace-driven zstd tuning.
|
||||
- **Distributed GGUF direction** — benchmark-gated native runtime: compare controlled Transformers/safetensors and whole-model llama.cpp lanes before expensive work; ship only for measured speed or model-fit advantage. Public parallelism is contiguous Shards in an Inference Route; concurrency comes from per-node continuous batching across isolated Route Sessions, while tensor/expert collectives stay inside optional trusted composite providers. Native data plane uses versioned Protobuf over long-lived gRPC/HTTP2 seam streams, with existing relay carrying the same opaque frames when needed. llama.cpp/GGML remains the substrate behind a project-owned standalone worker and small pinned fork; vLLM is an optional complete managed provider and concept donor, not a fork. Nakshatra, `prima.cpp`, `llama-gguf`, LiGGUF and historical GPUStack are source/test donors only. Active plan: [README](../../.scratch/distributed-gguf-runtime/README.md), [architecture](../../.scratch/distributed-gguf-runtime/architecture.md), [PRD](../../.scratch/distributed-gguf-runtime/PRD.md), [Ralph backlog](../../.scratch/distributed-gguf-runtime/prd.json). ADR: [0024](../../docs/adr/0024-distributed-gguf-runtime.md). Research: [landscape](../../docs/research/distributed-gguf-landscape.md), [GitHub follow-up](../../docs/research/distributed-gguf-github-followup.md), [vLLM](../../docs/research/vllm-distributed-gguf-assessment.md).
|
||||
- **Multi-subscription orchestration policy** — keep one fixed worktree per provider/agent and one user-selected integration branch. Do not switch the integration branch or create per-task branches/worktrees without explicit user confirmation. Parallel mode is default; task assignments should be disjoint and integration/push is serialized after each independently verified task. Serial mode uses an explicit provider priority, consumes the preferred subscription until its authoritative limit, then falls back in order and returns to higher priority after its official reset. Because Git cannot check out one named branch in multiple worktrees, provider worktrees should normally remain detached at the integration HEAD; the controller cherry-picks each verified task into the unchanged integration branch, tests, and pushes, then resynchronizes every fixed worktree.
|
||||
|
||||
@@ -8,23 +8,6 @@ metadata:
|
||||
|
||||
# Project Status (2026-07-13)
|
||||
|
||||
## Distributed GGUF controller checkpoint (2026-07-21)
|
||||
|
||||
- All three fixed detached lanes from the 2026-07-18 checkpoint (Fable/DGR-025, Kimi/DGR-028, Terra/DGR-024) were reviewed, committed, and merged into `ralph/distributed-gguf-runtime`, now at `cd6b4d9`. The `.claude/worktrees/ralph-fable-loop`, `ralph-kimi-loop`, `ralph-terra-loop`, `ralph-cursor-loop`, and `ralph-next-task` worktrees were removed after merge (cursor-loop/next-task were idle with no work in progress). Only `.claude/worktrees/distributed-gguf-runtime` (the integration checkout) remains, plus the unrelated `fix-tracker-incomplete-snapshot` worktree (locked, not part of this arc).
|
||||
- DGR-025 and DGR-028 merged cleanly with no conflicts (105 + 7 tests passing, 112 together). DGR-024 required real completion work first: the worktree's `prd.json` note was stale (written for the old, policy-rejected in-memory fake-seam story). The actual code already present (`shard_runtime_server.py`, `test_shard_runtime_harness.py`) had correctly pivoted to a real subprocess/socket gRPC harness with direct-vs-opaque-relay byte-identity proof, but was missing the required fail-closed negative paths (stale epoch, expired deadline, malformed/checksum-corrupt fragments, exhausted flow-control credit, duplicate idempotency steps, in-band/out-of-band cancel). Implemented those (`SessionState` per `route_session_id`, `_validate_bundle`), added 9 new tests (11 total, all passing), and rewrote `evidence/DGR-024/README.md` to describe the actual implementation instead of the nonexistent `FakeShardSeam`.
|
||||
- `prd.json`'s `passes` field was left `false` for all three stories — that flag is only flipped by the project's own independent controller review process, not by whoever lands the merge. DGR-028's evidence README explicitly still awaits that P0/P1 review.
|
||||
- Post-merge full suite (before the DGR-024 merge): 3 failed / 1116 passed / 20 skipped — down from the prior 9-failure baseline, and the 3 remaining failures (billing default-db, dynamic-routing ADD_SHARD/LOAD_SHARD, preset dedup) are pre-existing and unrelated to native/runtime-identity code. Did not get a chance to rerun the full suite after the DGR-024 merge landed (interrupted); only the focused `tests/test_shard_runtime_harness.py` (11 passed) was reconfirmed post-merge — a fresh full-suite run is worth doing before treating the whole arc as done.
|
||||
- The unrelated dirty edits on the integration worktree (`model_catalog.py` / `tests/test_mining_cli.py`, adding a `Qwen2.5-Coder-1.5B-Instruct-Q2_K-GGUF` preset) were stashed during the merge sequence and popped back afterward — still uncommitted, as before.
|
||||
|
||||
**Why:** user directed "distributed-gguf-runtime is where we need to merge all ralph-* branches" and to prune worktrees once their task's work is merged, so completed lanes don't linger and new worktrees signal new tasks unambiguously.
|
||||
**How to apply:** next Ralph session picking up this project should start from `ralph/distributed-gguf-runtime` at `cd6b4d9`, rerun the full suite once to get a clean current baseline, and check whether independent review has flipped DGR-024/025/028's `passes` flags before selecting new work.
|
||||
|
||||
## Distributed GGUF controller checkpoint (2026-07-18)
|
||||
|
||||
- Integration branch `ralph/distributed-gguf-runtime` is at `377bc3475c41b762ebbcab038c05adf15d7749d0`, matching its remote; preserve the unrelated dirty integration edits in `model_catalog.py` and `test_mining_cli.py`.
|
||||
- Fixed detached lanes hold three claimed, uncommitted tasks: Fable DGR-025/Gitea #9 executing-artifact identity repair; Terra DGR-024/#8 fake gRPC seam repair; Kimi DGR-028/#12 numbered llama.cpp patch stack. No provider worker was running at the latest reconciliation, so continue these dirty trees before selecting new work.
|
||||
- DGR-028 controller repair added working apply/reverse/verify enforcement, exact assumptions, and a passing exact-pin native fixture; final independent P0/P1 review is still required before commit/integration. DGR-024 and DGR-025 focused gates pass but likewise remain provisional pending the current independent reviews. Controller reruns on 2026-07-18 passed DGR-025 (235 impacted tests), DGR-024 (78 shared/focused tests), and DGR-028 (7 Python tests plus exact-pin native CTest 1/1 and clean apply/reverse). Integrate DGR-028 before DGR-025, then replay DGR-025 on the new integration HEAD and regenerate/retest its runtime fingerprint vectors because DGR-028 changes the pinned patch-stack/tree identity. The integration full-suite baseline is 9 failures/1076 passes/22 skips: missing optional zstd/langchain dependencies plus pre-existing billing and dynamic-routing expectations; lane-only extra failures map to stale DGR-023 projection ancestry and a timing-sensitive cancel test, not the focused story paths.
|
||||
|
||||
## Selected-node model placement (2026-07-14)
|
||||
|
||||
- Admin Model placement now opens a node selector for load and release; the control-plane accepts optional `node_id` and targets only that registry assignment. Multi-model serving remains supported through `ADD_SHARD` and `max_loaded_shards`.
|
||||
@@ -61,6 +44,10 @@ Historical handoff note: `/mnt/c/Users/popov/Downloads/neuron-tai-alpha-handoff-
|
||||
|
||||
Planning is ready at `.scratch/node-capability-admission/` with five sequential Ralph stories and ADR-0023. The design is model-agnostic: a Node must validate its selected Model Artifact/shard with a bounded real forward before Tracker routing; Qwen3.6 is only an optional development fixture. P0 adds a versioned local recipe-manifest/report contract, `meshnet-node doctor`, fail-closed startup admission, and tracker route gating. It intentionally excludes dynamic recipe/dependency installation and the future signed Node updater.
|
||||
|
||||
## Gitea DGR sync (2026-07-17)
|
||||
|
||||
Gitea is ahead of the local Markdown backlog with open DGR-022..DGR-071. The first executable P0 dependency frontier is DGR-022 (Shard lifecycle and structured status RPCs), DGR-023 (reproducible protobuf generation), DGR-025 (artifact/runtime recipe identity), and DGR-027 (llama.cpp provenance manifest). DGR-021, the named-tensor stream envelope prerequisite for DGR-022/023/025, is closed. DGR-022 is the next dependency-ordered issue and blocks DGR-024, DGR-033, and DGR-037.
|
||||
|
||||
## Windows CUDA node (working as of 2026-07-01)
|
||||
- miniforge3 base env, torch 2.7.1+cu118, torchvision 0.22.x+cu118
|
||||
- RTX 4060 Laptop GPU, 8 GB VRAM, benchmark index ~11,200
|
||||
|
||||
126
.scratch/distributed-gguf-runtime/evidence/DGR-023/README.md
Normal file
126
.scratch/distributed-gguf-runtime/evidence/DGR-023/README.md
Normal file
@@ -0,0 +1,126 @@
|
||||
# DGR-023 evidence — reproducible Python and C++ protobuf/gRPC generation
|
||||
|
||||
**Status:** complete after controller verification and independent-review repairs on 2026-07-17.
|
||||
|
||||
**Authority:** live Gitea issue #7. The local PRD is a secondary projection.
|
||||
|
||||
## Implemented contract
|
||||
|
||||
- Python generation requires exactly `grpcio-tools==1.82.1`; the generator checks installed distribution metadata and rejects missing or different versions with an actionable exact install command.
|
||||
- The C++ bootstrap builds one ignored toolchain prefix from exact inputs:
|
||||
- Protobuf release `33.1` (`protobuf-config` version `33.1.0`);
|
||||
- Abseil release `20250814.1`;
|
||||
- gRPC C++ `1.82.1` at commit `acccf84c0df20487d64101f528e5d426541ca4e5`;
|
||||
- gRPC's exact-commit submodules for c-ares, RE2, OpenSSL, and zlib.
|
||||
- Protobuf is configured with local dependencies only after the exact Abseil build. gRPC uses the installed Protobuf/Abseil packages and commit-pinned module dependencies, avoiding unpinned system development packages and download fallbacks.
|
||||
- CMake requires exact Protobuf `33.1.0` and gRPC `1.82.1`, requires the exported `gRPC::grpc_cpp_plugin` target, and always generates/builds both message and service stubs in the ignored build tree.
|
||||
- Python bindings remain committed package output; `--check` regenerates into a temporary directory and compares output. C++ bindings are never committed.
|
||||
- The C++ conformance test parses Python-produced vectors, validates fields/CRC32C, and emits `cpp_roundtrip.binpb`; Python compares that artifact byte-for-byte.
|
||||
|
||||
## Defects found and fixed
|
||||
|
||||
1. A relative bootstrap prefix was resolved after entering the temporary source directory, so successful output was deleted by cleanup. The script now canonicalizes the caller-relative destination first. The regression executes `--print-prefix` from a temporary working directory and validates the resulting path behavior.
|
||||
2. The original native path omitted gRPC C++ and accepted any discoverable plugin. The bootstrap now builds exact gRPC/plugin sources, and CMake rejects absent/incompatible versions.
|
||||
3. The Python script named the `grpcio-tools` pin but did not validate the installed distribution. It now refuses mismatched versions.
|
||||
4. Protobuf ignored a stale provider option and attempted to download a different Abseil. The build was stopped; exact Abseil is now built first and Protobuf uses `LOCAL_DEPENDENCIES_ONLY`.
|
||||
5. The host lacked OpenSSL development headers. Rather than add a floating system dependency, gRPC now uses the submodule pinned by its exact commit.
|
||||
6. Documentation uses `bash scripts/bootstrap_native_toolchain.sh ...`, so a normal checkout does not depend on executable-mode preservation.
|
||||
|
||||
## Verified toolchain
|
||||
|
||||
```text
|
||||
cmake version 4.4.0
|
||||
c++ (GCC) 15.2.1 20260123 (Red Hat 15.2.1-7)
|
||||
libprotoc 33.1
|
||||
protobuf CMake package 33.1.0
|
||||
grpcio-tools 1.82.1
|
||||
grpcio 1.82.1
|
||||
protobuf Python runtime 7.35.1
|
||||
gRPC C++ 1.82.1
|
||||
commit acccf84c0df20487d64101f528e5d426541ca4e5
|
||||
grpc_cpp_plugin sha256 995ca8ac620fe83532b649a7c8c0a9341c7003da927fe0e4a8f821bfc579206d
|
||||
```
|
||||
|
||||
The native toolchain and generated/build artifacts live under ignored mounted-drive `build/` paths; model/build artifacts were not stored under `/home`.
|
||||
|
||||
## Commands and results
|
||||
|
||||
```bash
|
||||
bash scripts/bootstrap_native_toolchain.sh build/native-toolchain
|
||||
```
|
||||
|
||||
```text
|
||||
passed from a clean build directory
|
||||
libprotoc 33.1
|
||||
gRPC 1.82.1 commit acccf84c0df20487d64101f528e5d426541ca4e5
|
||||
grpc_cpp_plugin sha256 995ca8ac620fe83532b649a7c8c0a9341c7003da927fe0e4a8f821bfc579206d
|
||||
```
|
||||
|
||||
```bash
|
||||
cmake -S packages/node/native -B build/native \
|
||||
-DCMAKE_PREFIX_PATH="$PWD/build/native-toolchain"
|
||||
cmake --build build/native -j"$(nproc)"
|
||||
test -f build/native/shard_runtime.grpc.pb.cc
|
||||
test -f build/native/shard_runtime.grpc.pb.h
|
||||
test -f build/native/libshard_runtime_grpc.a
|
||||
ctest --test-dir build/native --output-on-failure
|
||||
```
|
||||
|
||||
```text
|
||||
Pinned gRPC 1.82.1: building ShardRuntime service stubs
|
||||
shard_runtime_proto built
|
||||
shard_runtime_grpc built
|
||||
1/1 shard_protocol_conformance passed
|
||||
```
|
||||
|
||||
```bash
|
||||
python3 -m pytest -q tests/test_native_shard_protocol.py
|
||||
```
|
||||
|
||||
```text
|
||||
50 passed, 2 optional-path skips
|
||||
```
|
||||
|
||||
All DGR-023-required checks were selected explicitly:
|
||||
|
||||
```bash
|
||||
python3 -m pytest -q -rs tests/test_native_shard_protocol.py \
|
||||
-k 'cpp_and_python_agree_byte_for_byte or generated_python_stubs_match_the_proto or native_toolchain_bootstrap or wrong_grpcio'
|
||||
```
|
||||
|
||||
```text
|
||||
4 passed, 48 deselected
|
||||
```
|
||||
|
||||
```bash
|
||||
python3 scripts/generate_native_protocol.py --check
|
||||
python3 scripts/generate_protocol_goldens.py --check
|
||||
python3 scripts/ralph_prd_schema.py validate .scratch/distributed-gguf-runtime/prd.json
|
||||
python3 -m compileall -q packages tests
|
||||
git diff --check
|
||||
```
|
||||
|
||||
```text
|
||||
generated stubs are up to date
|
||||
conformance vectors are up to date
|
||||
OK: 55 stories validated
|
||||
compileall passed
|
||||
git diff --check passed
|
||||
```
|
||||
|
||||
## Changed files
|
||||
|
||||
- `scripts/bootstrap_native_toolchain.sh`
|
||||
- `scripts/generate_native_protocol.py`
|
||||
- `packages/node/native/CMakeLists.txt`
|
||||
- `packages/node/native/README.md`
|
||||
- `tests/test_native_shard_protocol.py`
|
||||
- `.scratch/distributed-gguf-runtime/evidence/DGR-023/README.md`
|
||||
- `.scratch/distributed-gguf-runtime/prd.json` (secondary completion projection only)
|
||||
|
||||
## Limitations and dependency handoff
|
||||
|
||||
- This story proves exact schema/message/service generation and cross-language conformance. It does not implement or run the standalone worker service itself; DGR-033/DGR-037 own worker behavior.
|
||||
- The plugin SHA is evidence for this verified build. Reproducibility authority is the exact gRPC commit plus its submodule graph, not an assumption that different compilers produce byte-identical executables.
|
||||
- No model, GPU, API credits, or model download was used.
|
||||
- DGR-024 and DGR-037 may consume this completed generation dependency but must provide their own transport/worker evidence.
|
||||
210
.scratch/distributed-gguf-runtime/evidence/DGR-025/README.md
Normal file
210
.scratch/distributed-gguf-runtime/evidence/DGR-025/README.md
Normal file
@@ -0,0 +1,210 @@
|
||||
# DGR-025 evidence — exact artifact and runtime recipe identity
|
||||
|
||||
**Completed:** 2026-07-17
|
||||
**Branch:** `ralph/fable-architecture-loop` (Claude Fable architecture lane)
|
||||
**Authority:** `.scratch/distributed-gguf-runtime/prd.json`
|
||||
**Dependencies:** DGR-018 (`evidence/DGR-018/README.md` — canonical backlog schema and
|
||||
issue projection), DGR-021 (`evidence/DGR-021/README.md` — versioned activation
|
||||
envelope). Both read before changing code.
|
||||
|
||||
## Objective
|
||||
|
||||
Ensure the tracker and worker only combine numerically and operationally
|
||||
compatible shards: fingerprint every axis that moves the numbers, bind shards to
|
||||
exact half-open ranges, fail closed on any mismatch, and keep uncertified
|
||||
recipes registered-but-dark.
|
||||
|
||||
## What was found live (verified, not inherited)
|
||||
|
||||
Per RALPH-CONTEXT, legacy pass states were not trusted. The DGR-003-lineage
|
||||
identity core was inspected and exercised live before any change:
|
||||
|
||||
- `packages/node/meshnet_node/runtime_recipe.py` — node-side identity:
|
||||
domain-separated digests (`meshnet.model-artifact.v1`,
|
||||
`meshnet.runtime-recipe.v1`, `meshnet.shard-binding.v1`) over the source
|
||||
artifact SHA (`source_digest`, with split artifacts bound to their exact
|
||||
source via `DerivativeBinding`), tokenizer revision (pin-enforced),
|
||||
architecture adapter + architecture/config digest, boundary and protocol
|
||||
schema versions, backend, weight quantization, activation/compute dtypes, and
|
||||
KV dtype/layout (`RECIPE_AXES`). Shard ranges are half-open
|
||||
(`shard_start`/`shard_end`, end-exclusive, protocol convention) with no
|
||||
topology or quant constants anywhere; `check_route` accepts any tiling of
|
||||
`[0, layer_count)`. Route, handshake (`check_handshake`), and session-open
|
||||
(`check_session_open`) checks fail closed with structured `RouteMismatch`
|
||||
reasons mapped to specific protocol error codes (`handshake_error`).
|
||||
- `packages/tracker/meshnet_tracker/recipe.py` — deliberately independent
|
||||
tracker re-derivation (no `meshnet_node` import); declared fingerprints are
|
||||
recomputed, never trusted (`parse_identity`, `FingerprintMismatch`). The
|
||||
`CertificationLedger` keeps every registered recipe dark until a real
|
||||
distributed forward — at least 2 distinct nodes, whole-model coverage,
|
||||
non-synthetic, tokens actually generated — certifies it; dark recipes may
|
||||
route only to certify.
|
||||
- The two implementations are pinned by committed conformance vectors
|
||||
(`tests/data/recipe_fingerprint_vectors.json`).
|
||||
|
||||
Live verification of that pre-existing core before changes:
|
||||
`PYTHONPATH=packages/node:packages/tracker python3 -m pytest -q tests/test_runtime_recipe_identity.py`
|
||||
→ `45 passed`; plus `tests/test_native_identity_emission.py`,
|
||||
`tests/test_tracker_capability_admission.py`, `tests/test_node_admission.py`
|
||||
→ `59 passed`.
|
||||
|
||||
## Gap found and closed (this story's change)
|
||||
|
||||
**The `runtime_version` recipe axis was a label, not a pin.** It was an opaque
|
||||
caller-supplied string: nothing derived it from the DGR-027 lock manifest, and
|
||||
neither identity implementation rejected a moving reference (`"latest"` was
|
||||
accepted), so two workers could run different llama.cpp pins or patch stacks
|
||||
under one label and still agree on the recipe digest. The acceptance criterion
|
||||
explicitly requires fingerprinting the "runtime pin/patch stack".
|
||||
|
||||
### Changed files
|
||||
|
||||
- `packages/node/meshnet_node/runtime_pin.py` (new) — derives the canonical
|
||||
`runtime_version` axis value from the DGR-027 lock workspace
|
||||
(`packages/node/native/llama`):
|
||||
`<runtime>@<40-hex upstream commit>+patchstack.<sha256>` where the stack
|
||||
digest commits, under the `meshnet.runtime-patch-stack.v1` domain, to the
|
||||
*ordered* `(patch name, patch bytes sha256)` stack. Fails closed on: missing
|
||||
or malformed `UPSTREAM_LOCK.json`, unknown schema version, non-40-hex/moving
|
||||
commit, `UPSTREAM_COMMIT` disagreement, any disagreement among the lock's
|
||||
`patch_series`, `patches/series`, and `patches/SHA256SUMS`, a missing patch
|
||||
file, or a patch whose bytes don't match their recorded digest. Reads the
|
||||
committed manifest only; fetching/patching stays with
|
||||
`scripts/llama_cpp_dependency.py` (DGR-027).
|
||||
- `packages/node/meshnet_node/runtime_recipe.py` — `runtime_version` is now
|
||||
pin-enforced (`_require_pin`) exactly like `tokenizer_revision`; for the
|
||||
llama.cpp backend it must also match the canonical
|
||||
`llama.cpp@<40-hex>+patchstack.<64-hex>` grammar.
|
||||
- `packages/node/meshnet_node/native_backend.py` — the production native
|
||||
identity seam no longer accepts a caller-supplied runtime string. It derives
|
||||
`runtime_version` directly through `load_runtime_pin()` from the committed
|
||||
lock and rejects a non-llama backend at this llama.cpp-specific boundary.
|
||||
- `packages/tracker/meshnet_tracker/recipe.py` — the independent tracker
|
||||
implementation applies the same backend-specific grammar before re-deriving
|
||||
the recipe digest, so forged operator labels cannot register or certify.
|
||||
- `tests/test_runtime_pin_identity.py` and
|
||||
`tests/test_native_identity_emission.py` — deterministic tests cover lock
|
||||
derivation, production native emission, and node/tracker rejection of the
|
||||
forged values from independent review. Conformance vectors were regenerated
|
||||
through `scripts/gen_recipe_fingerprint_vectors.py` for the tightened wire
|
||||
contract.
|
||||
|
||||
### Backlog-consistency repair (pre-existing damage, honestly recorded)
|
||||
|
||||
`tests/test_ralph_prd_schema.py` had 4 pre-existing failures before this story
|
||||
touched anything, left by prior sessions and the alternate-history merge:
|
||||
|
||||
- DGR-022 and DGR-027 were marked `passes: true` without `completionNotes` and
|
||||
without regenerated issue projections. Added their `completionNotes`
|
||||
(explicitly labeled as added during this repair, content drawn from their own
|
||||
evidence READMEs) and regenerated
|
||||
`issues/022-…` / `issues/027-…` via `scripts/ralph_prd_schema.py render`.
|
||||
- Three pre-DGR legacy GLM alpha issue files (`18-…`, `19-…`, `20-…`,
|
||||
committed 2026-07-14, before DGR-018 established the generated-only
|
||||
convention; they carry no authority disclaimer because they are *not*
|
||||
generated from prd.json) were relocated via `git mv` to
|
||||
`issues/legacy/` — preserved as provenance, out of the generated namespace.
|
||||
|
||||
### prd.json
|
||||
|
||||
Marked `DGR-025.passes = true` with `completionNotes`; regenerated
|
||||
`issues/025-define-exact-artifact-and-runtime-recipe-identity.md`.
|
||||
|
||||
## Acceptance criteria → evidence
|
||||
|
||||
1. **Fingerprint all axes** — `RECIPE_AXES` + `ArtifactIdentity` cover source
|
||||
artifact SHA, tokenizer revision, architecture adapter/version (adapter axis
|
||||
+ architecture/config digest), boundary schema (boundary + protocol schema
|
||||
versions), backend, quant, activation/compute dtype, KV/state layout; the
|
||||
runtime pin/patch stack is now committed via the derived `runtime_version`
|
||||
axis (`runtime_pin.py`). Verified by `test_runtime_recipe_identity.py` and
|
||||
`test_runtime_pin_identity.py`.
|
||||
2. **Exact half-open range, no hardcoded topology/quant** — `ShardIdentity`
|
||||
end-exclusive ranges, `DerivativeBinding` coverage checks, `check_route`
|
||||
tiling over arbitrary layouts; quant/dtype values are open strings
|
||||
(dynamic recipe inputs). Verified by `test_runtime_recipe_identity.py`
|
||||
(routes of 1, 2, and 5 shards; no product constants).
|
||||
3. **Fail closed on any mismatch** — artifact, adapter, boundary/schema, cache
|
||||
layout, backend, and runtime mismatches each produce structured
|
||||
`RouteMismatch` reasons and protocol error codes; the tracker recomputes
|
||||
digests and rejects inconsistent claims; moving runtime references are now
|
||||
rejected on both sides.
|
||||
4. **Registered-but-dark** — `CertificationLedger`: unknown recipes cannot be
|
||||
certified, registered recipes are dark, only a real ≥2-distinct-node
|
||||
whole-model non-synthetic forward promotes; verified by
|
||||
`test_runtime_recipe_identity.py` / `test_tracker_capability_admission.py`.
|
||||
5. **Gates + this handoff** — below.
|
||||
|
||||
## Commands and results
|
||||
|
||||
```bash
|
||||
PYTHONPATH=packages/node:packages/tracker python3 -m pytest -q tests/test_runtime_pin_identity.py
|
||||
```
|
||||
```text
|
||||
23 passed in 0.15s
|
||||
```
|
||||
|
||||
```bash
|
||||
PYTHONPATH=packages/node:packages/tracker python3 -m pytest -q \
|
||||
tests/test_runtime_pin_identity.py tests/test_runtime_recipe_identity.py \
|
||||
tests/test_native_identity_emission.py tests/test_tracker_capability_admission.py \
|
||||
tests/test_node_admission.py tests/test_node_capability.py tests/test_recipe_benchmark.py
|
||||
```
|
||||
```text
|
||||
202 passed, 1 warning in 5.38s
|
||||
```
|
||||
|
||||
```bash
|
||||
PYTHONPATH=packages/node:packages/tracker python3 -m pytest -q tests/test_ralph_prd_schema.py
|
||||
```
|
||||
```text
|
||||
108 passed
|
||||
```
|
||||
(4 failed before this story's backlog repair; 0 after.)
|
||||
|
||||
```bash
|
||||
python3 -m compileall -q packages tests # exit 0
|
||||
git diff --check # exit 0
|
||||
python3 scripts/ralph_prd_schema.py validate .scratch/distributed-gguf-runtime/prd.json
|
||||
# OK: 55 stories validated.
|
||||
```
|
||||
|
||||
Default tests are model-download-free, API-credit-free, and GPU-free; no model
|
||||
artifact was touched and nothing was written under `/home`.
|
||||
|
||||
## Limitations
|
||||
|
||||
- The production native identity seam now derives the manifest pin and cannot
|
||||
accept an operator-supplied runtime label. It still cannot attest that the
|
||||
running binary was built from those locked bytes. Embedding the patched-tree
|
||||
hash at build time and echoing it through the DGR-022 status contract belongs
|
||||
with DGR-028+/DGR-031; real distributed certification remains the final trust
|
||||
boundary.
|
||||
- The DGR-027-recorded blocker stands: `0002-dense-llama-owned-range-loader.patch`
|
||||
does not apply cleanly against the pin (DGR-028). That does not affect this
|
||||
story: the identity commits to the patch *bytes as committed*, which is
|
||||
precisely what makes a later repaired patch a *different* runtime identity.
|
||||
- No native/CMake change was made, so the native build/CTest gate is not
|
||||
applicable; no llama.cpp patch content was changed, so apply/check/reverse
|
||||
verification is not applicable (and is blocked by the DGR-028 defect anyway).
|
||||
- Tracker routing, load balancing, billing, telemetry, and relay semantics are
|
||||
untouched; the only behavior change outside the new module is the stricter
|
||||
(fail-closed) rejection of moving `runtime_version` values.
|
||||
|
||||
## Dependency handoff
|
||||
|
||||
- **DGR-026** (split-GGUF provisioning): bind each provisioned split via
|
||||
`DerivativeBinding` to the exact source digest recorded in its hashed
|
||||
manifest; the per-split `shard_binding_digest` is what certification pins.
|
||||
- **DGR-031** (`ShardEngine`): construct worker identity through
|
||||
`shard_identity_from_native_report` and populate `runtime_version` from
|
||||
`meshnet_node.runtime_pin.load_runtime_pin().runtime_version` — never from an
|
||||
operator string. A build-time echo of the patched-tree hash through the
|
||||
status contract would close the manifest-vs-binary gap noted above.
|
||||
- **DGR-041** (capability registration): the tracker already re-derives and
|
||||
fail-closes on presented identities (`parse_identity`); register recipes
|
||||
through the `CertificationLedger` so they arrive dark.
|
||||
- **DGR-044** (DeepSeek V4 Flash target): pin the target's artifact identity
|
||||
the same way `glm_alpha_artifact` does — read locked manifests, never restate
|
||||
digests — and note `layer_count` must count the routed transformer stack the
|
||||
route tiles, excluding MTP (reserved for beta).
|
||||
72
.scratch/distributed-gguf-runtime/evidence/DGR-027/README.md
Normal file
72
.scratch/distributed-gguf-runtime/evidence/DGR-027/README.md
Normal file
@@ -0,0 +1,72 @@
|
||||
# DGR-027 evidence — exact llama.cpp provenance manifest and fetch workspace
|
||||
|
||||
**Completed implementation:** 2026-07-17
|
||||
**Branch:** `ralph/dgr-small-terra`
|
||||
**Authority:** live Gitea issue #11. The controller fetched and claimed the issue
|
||||
through the Gitea API before launch; the isolated agent received that exact body.
|
||||
|
||||
## Changed files
|
||||
|
||||
- `packages/node/native/llama/UPSTREAM_LOCK.json`
|
||||
- `packages/node/native/llama/PATCH-STACK.md`
|
||||
- `scripts/llama_cpp_dependency.py`
|
||||
- `tests/test_llama_cpp_dependency.py`
|
||||
- `.scratch/distributed-gguf-runtime/evidence/DGR-027/README.md`
|
||||
|
||||
## Provenance and retrieval contract
|
||||
|
||||
`UPSTREAM_LOCK.json` records the upstream Git URL, immutable 40-character
|
||||
commit `e920c523e3b8a0163fe498af5bf90df35ff51d25`, expected Git tree
|
||||
`6c91a11407a3a3fb160f5dac705f9c59718f54f1`, MIT license, and the sole
|
||||
retrieval method: `git-clone-detached-commit` into `build/llama.cpp/source`.
|
||||
|
||||
`python3 scripts/llama_cpp_dependency.py fetch` has no branch, tag, ref, or
|
||||
repository override. On a first fetch it clones the manifest URL, checks out
|
||||
the detached commit, and verifies commit, tree, required upstream blobs,
|
||||
license, and cleanliness. If the workspace already exists, it makes no network
|
||||
request and accepts it only after the same verification. Dirty or mismatched
|
||||
caches fail closed. The build directory is already ignored by `.gitignore`.
|
||||
|
||||
## Verification
|
||||
|
||||
| Command | Result |
|
||||
| --- | --- |
|
||||
| `python3 -m pytest -q tests/test_llama_cpp_dependency.py` | `7 passed in 0.22s` |
|
||||
| `python3 -m compileall packages tests` | passed |
|
||||
| `git diff --check` | passed (no output) |
|
||||
| `python3 scripts/llama_cpp_dependency.py inspect` | passed; reports exact commit/tree, retrieval workspace, MIT license, and two-patch stack |
|
||||
| `python3 scripts/llama_cpp_dependency.py fetch --workspace /tmp/not-llama-workspace` | failed closed with status 2: workspace outside the locked ignored build root |
|
||||
| symlinked workspace regression | passed; both a `build/` ancestor symlink and a final `source` symlink escaping the repository are refused |
|
||||
| attached-branch cache regression | passed; an exact commit on a local branch is refused until checked out as detached HEAD |
|
||||
| ignored/excluded injection regression | passed; a file hidden by `.git/info/exclude` is detected and refused |
|
||||
| tracked injection regression | passed; modified tracked content hidden by both `assume-unchanged` and `skip-worktree` is content-hashed and refused |
|
||||
| executable-mode regression | passed on the POSIX fixture for both index flags; the mounted project workspace has `core.filemode=false`, so its exact index tree is the canonical mode record and physical mode bits are not treated as meaningful |
|
||||
| `git check-ignore -v build/llama.cpp/source` | passed; `.gitignore:6:build/` |
|
||||
| `git diff --summary` and `git ls-files build packages/node/native/llama` | no source checkout or new submodule introduced; only manifest/docs/patches/native wrapper are tracked |
|
||||
| `python3 scripts/llama_cpp_dependency.py fetch` (controller network lane) | passed; fetched the exact detached commit and verified HEAD `e920c523e3b8a0163fe498af5bf90df35ff51d25` and tree `6c91a11407a3a3fb160f5dac705f9c59718f54f1` in the ignored workspace |
|
||||
| `python3 scripts/llama_cpp_dependency.py apply --source-dir build/llama.cpp/source` | failed on the pre-existing `0002-dense-llama-owned-range-loader.patch` as a corrupt patch at line 26; this is an explicit DGR-028 blocker and no native-build claim is made |
|
||||
|
||||
The targeted test suite creates a local Git fixture to prove offline cache reuse
|
||||
after full identity verification, then proves a dirty cache is rejected. It
|
||||
also proves the CLI rejects a repository/branch override and an arbitrary
|
||||
workspace.
|
||||
|
||||
## Limitations
|
||||
|
||||
- The controller successfully materialized and verified the exact upstream
|
||||
commit/tree, so the DGR-027 fetch and offline-cache boundary has real upstream
|
||||
evidence rather than fixture-only evidence.
|
||||
- The existing `0002-dense-llama-owned-range-loader.patch` is malformed and
|
||||
cannot pass `git apply --check` against the exact pin. DGR-027 changes no patch
|
||||
file; repairing and certifying the numbered patch stack belongs to DGR-028.
|
||||
Until that story closes, the repository must not claim patched-tree, native
|
||||
CMake/CTest, or reverse-apply certification.
|
||||
- No model, API credits, GPU, or model artifact storage was used.
|
||||
|
||||
## Dependency handoff
|
||||
|
||||
DGR-028, DGR-029, and DGR-044 must invoke the manifest-owned `fetch` command
|
||||
before touching llama.cpp source. They may use only the verified
|
||||
`build/llama.cpp/source` checkout and must record any native build, CTest, and
|
||||
patch apply/check/reverse evidence against the exact manifest pin. DGR-017's
|
||||
cleanup remains provenance only and grants no inherited completion credit.
|
||||
@@ -1,7 +1,7 @@
|
||||
<!-- GENERATED FROM prd.json — DO NOT EDIT AS AN INDEPENDENT SOURCE. prd.json IS AUTHORITATIVE. -->
|
||||
# DGR-022: Define Shard lifecycle and structured status RPCs
|
||||
|
||||
- **Status / triage:** specification only; `ready-for-agent`; `passes: false`
|
||||
- **Status / triage:** completed; `passes: true`
|
||||
- **Execution mode:** `AFK`
|
||||
- **Milestone:** `M1`
|
||||
- **Dependencies:** `DGR-021`
|
||||
@@ -18,11 +18,11 @@ Fresh Ralph session: read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md`,
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- [ ] Define capability, health, bidirectional session stream, cancellation, release, and metrics RPCs.
|
||||
- [ ] Specify deadlines, cancellation propagation, bounded flow control, cache expectations/results, and structured error taxonomy.
|
||||
- [ ] Specify TLS/auth hooks without moving Meshnet authentication or billing into the worker.
|
||||
- [ ] Add compatibility tests for supported versions and fail-closed tests for unsupported versions and malformed lifecycle transitions.
|
||||
- [ ] Applicable shared quality gates in `prd.json` pass, and the evidence handoff records exact commands/results, changed files, limitations, and dependency handoff.
|
||||
- [x] Define capability, health, bidirectional session stream, cancellation, release, and metrics RPCs.
|
||||
- [x] Specify deadlines, cancellation propagation, bounded flow control, cache expectations/results, and structured error taxonomy.
|
||||
- [x] Specify TLS/auth hooks without moving Meshnet authentication or billing into the worker.
|
||||
- [x] Add compatibility tests for supported versions and fail-closed tests for unsupported versions and malformed lifecycle transitions.
|
||||
- [x] Applicable shared quality gates in `prd.json` pass, and the evidence handoff records exact commands/results, changed files, limitations, and dependency handoff.
|
||||
|
||||
## Shared quality gates
|
||||
|
||||
@@ -36,4 +36,4 @@ Fresh Ralph session: read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md`,
|
||||
|
||||
## Evidence handoff
|
||||
|
||||
Write and verify `.scratch/distributed-gguf-runtime/evidence/DGR-022/README.md`. Until every criterion and applicable gate has real evidence, this story remains `passes: false`. Legacy evidence is provenance only, not completion credit.
|
||||
Verified evidence: `.scratch/distributed-gguf-runtime/evidence/DGR-022/README.md`. Legacy evidence remains provenance only and grants no implementation completion credit.
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
<!-- GENERATED FROM prd.json — DO NOT EDIT AS AN INDEPENDENT SOURCE. prd.json IS AUTHORITATIVE. -->
|
||||
# DGR-025: Define exact artifact and runtime recipe identity
|
||||
|
||||
- **Status / triage:** specification only; `ready-for-agent`; `passes: false`
|
||||
- **Status / triage:** completed; `passes: true`
|
||||
- **Execution mode:** `AFK`
|
||||
- **Milestone:** `M1`
|
||||
- **Dependencies:** `DGR-018`, `DGR-021`
|
||||
@@ -18,11 +18,11 @@ Fresh Ralph session: read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md`,
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- [ ] Fingerprint source artifact SHA, tokenizer revision, architecture adapter/version, boundary schema, runtime pin/patch stack, backend, quant, activation/compute dtype, and KV/state layout.
|
||||
- [ ] Bind each shard to an exact half-open range without hardcoding a topology or quant.
|
||||
- [ ] Fail closed on any artifact, adapter, boundary, cache, backend, or runtime mismatch.
|
||||
- [ ] Unsupported recipes remain registered-but-dark until real-hardware evidence certifies them.
|
||||
- [ ] Applicable shared quality gates in `prd.json` pass, and the evidence handoff records exact commands/results, changed files, limitations, and dependency handoff.
|
||||
- [x] Fingerprint source artifact SHA, tokenizer revision, architecture adapter/version, boundary schema, runtime pin/patch stack, backend, quant, activation/compute dtype, and KV/state layout.
|
||||
- [x] Bind each shard to an exact half-open range without hardcoding a topology or quant.
|
||||
- [x] Fail closed on any artifact, adapter, boundary, cache, backend, or runtime mismatch.
|
||||
- [x] Unsupported recipes remain registered-but-dark until real-hardware evidence certifies them.
|
||||
- [x] Applicable shared quality gates in `prd.json` pass, and the evidence handoff records exact commands/results, changed files, limitations, and dependency handoff.
|
||||
|
||||
## Shared quality gates
|
||||
|
||||
@@ -36,4 +36,4 @@ Fresh Ralph session: read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md`,
|
||||
|
||||
## Evidence handoff
|
||||
|
||||
Write and verify `.scratch/distributed-gguf-runtime/evidence/DGR-025/README.md`. Until every criterion and applicable gate has real evidence, this story remains `passes: false`. Legacy evidence is provenance only, not completion credit.
|
||||
Verified evidence: `.scratch/distributed-gguf-runtime/evidence/DGR-025/README.md`. Legacy evidence remains provenance only and grants no implementation completion credit.
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
<!-- GENERATED FROM prd.json — DO NOT EDIT AS AN INDEPENDENT SOURCE. prd.json IS AUTHORITATIVE. -->
|
||||
# DGR-027: Add exact llama.cpp provenance manifest and fetch workspace
|
||||
|
||||
- **Status / triage:** specification only; `ready-for-agent`; `passes: false`
|
||||
- **Status / triage:** completed; `passes: true`
|
||||
- **Execution mode:** `AFK`
|
||||
- **Milestone:** `M1`
|
||||
- **Dependencies:** `DGR-017`
|
||||
@@ -18,11 +18,11 @@ Fresh Ralph session: read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md`,
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- [ ] Manifest records upstream URL, exact commit, expected source archive/tree hash, license, and retrieval method.
|
||||
- [ ] Fetch tooling verifies identity before use and refuses an unpinned branch/tag.
|
||||
- [ ] Source is fetched into an ignored build workspace; no submodule, vendored source tree, or permanent fork is introduced.
|
||||
- [ ] Offline reuse is supported only after the cached tree’s exact identity is verified.
|
||||
- [ ] Applicable shared quality gates in `prd.json` pass, and the evidence handoff records exact commands/results, changed files, limitations, and dependency handoff.
|
||||
- [x] Manifest records upstream URL, exact commit, expected source archive/tree hash, license, and retrieval method.
|
||||
- [x] Fetch tooling verifies identity before use and refuses an unpinned branch/tag.
|
||||
- [x] Source is fetched into an ignored build workspace; no submodule, vendored source tree, or permanent fork is introduced.
|
||||
- [x] Offline reuse is supported only after the cached tree’s exact identity is verified.
|
||||
- [x] Applicable shared quality gates in `prd.json` pass, and the evidence handoff records exact commands/results, changed files, limitations, and dependency handoff.
|
||||
|
||||
## Shared quality gates
|
||||
|
||||
@@ -36,4 +36,4 @@ Fresh Ralph session: read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md`,
|
||||
|
||||
## Evidence handoff
|
||||
|
||||
Write and verify `.scratch/distributed-gguf-runtime/evidence/DGR-027/README.md`. Until every criterion and applicable gate has real evidence, this story remains `passes: false`. Legacy evidence is provenance only, not completion credit.
|
||||
Verified evidence: `.scratch/distributed-gguf-runtime/evidence/DGR-027/README.md`. Legacy evidence remains provenance only and grants no implementation completion credit.
|
||||
|
||||
@@ -489,7 +489,8 @@
|
||||
"DGR-024",
|
||||
"DGR-033",
|
||||
"DGR-037"
|
||||
]
|
||||
],
|
||||
"completionNotes": "Completed 2026-07-17. Implemented the versioned backend-neutral Shard lifecycle/status contract in packages/node/meshnet_node/shard_lifecycle.py (RPC names, schema-version negotiation with fail-closed unsupported versions, structured status/error taxonomy, lifecycle state machine, monotonic idempotency enforcement, bounded flow control, cache expectation/result types, deadline policy and TLS/auth hooks) with deterministic tests in tests/test_shard_lifecycle.py (17 passed alongside the DGR-021 envelope tests). Generated protobuf bindings remain DGR-023. Evidence: evidence/DGR-022/README.md. These completionNotes were added during the DGR-025 backlog-consistency repair; the DGR-022 session omitted them."
|
||||
},
|
||||
{
|
||||
"id": "DGR-023",
|
||||
@@ -521,8 +522,8 @@
|
||||
"A clean checkout regenerates bindings deterministically or fails with an actionable toolchain error.",
|
||||
"Applicable shared quality gates in `prd.json` pass, and the evidence handoff records exact commands/results, changed files, limitations, and dependency handoff."
|
||||
],
|
||||
"passes": false,
|
||||
"notes": "Generated source issue: .scratch/distributed-gguf-runtime/issues/023-make-python-and-c-protobuf-generation-reproducible.md; prd.json is authoritative.",
|
||||
"passes": true,
|
||||
"notes": "Completed from Gitea #7 after controller provisioned and exercised the exact Python/C++ toolchains. Verified deterministic generation, native CMake/CTest, Python↔C++ byte parity, compileall, and diff checks; fixed relative bootstrap prefix resolution.",
|
||||
"blocks": [
|
||||
"DGR-024",
|
||||
"DGR-037"
|
||||
@@ -597,14 +598,15 @@
|
||||
"Unsupported recipes remain registered-but-dark until real-hardware evidence certifies them.",
|
||||
"Applicable shared quality gates in `prd.json` pass, and the evidence handoff records exact commands/results, changed files, limitations, and dependency handoff."
|
||||
],
|
||||
"passes": false,
|
||||
"passes": true,
|
||||
"notes": "Generated source issue: .scratch/distributed-gguf-runtime/issues/025-define-exact-artifact-and-runtime-recipe-identity.md; prd.json is authoritative.",
|
||||
"blocks": [
|
||||
"DGR-026",
|
||||
"DGR-031",
|
||||
"DGR-041",
|
||||
"DGR-044"
|
||||
]
|
||||
],
|
||||
"completionNotes": "Completed 2026-07-17. Verified the live DGR-003-lineage identity core against every criterion: node packages/node/meshnet_node/runtime_recipe.py and the independent tracker packages/tracker/meshnet_tracker/recipe.py (pinned together by tests/data/recipe_fingerprint_vectors.json) fingerprint the source artifact SHA, tokenizer pin, architecture adapter and config digest, boundary/protocol schema versions, backend, weight quant, activation/compute dtypes, and KV dtype/layout under domain-separated digests; shards bind to exact half-open ranges with no topology or quant constants; route/handshake/session checks fail closed with structured mismatch reasons; recipes stay registered-but-dark in the tracker CertificationLedger until a real >=2-distinct-node whole-model distributed forward certifies them. Closed the one open criterion gap (runtime pin/patch stack): new packages/node/meshnet_node/runtime_pin.py derives the runtime_version axis from the DGR-027 lock manifest — exact upstream commit plus a digest over the ordered patch-stack bytes — failing closed on any UPSTREAM_LOCK.json/UPSTREAM_COMMIT/series/SHA256SUMS/patch-byte disagreement, and both identity implementations now reject a moving runtime_version reference. Tests: tests/test_runtime_pin_identity.py (17 passed) plus 196 passing impacted identity/admission/native-emission tests; python -m compileall and git diff --check clean. Also repaired backlog consistency left by prior sessions: added the missing DGR-022/DGR-027 completionNotes, regenerated the DGR-022/025/027 issue projections, and relocated three pre-DGR legacy GLM alpha issue files to issues/legacy/."
|
||||
},
|
||||
{
|
||||
"id": "DGR-026",
|
||||
@@ -673,13 +675,14 @@
|
||||
"Offline reuse is supported only after the cached tree’s exact identity is verified.",
|
||||
"Applicable shared quality gates in `prd.json` pass, and the evidence handoff records exact commands/results, changed files, limitations, and dependency handoff."
|
||||
],
|
||||
"passes": false,
|
||||
"notes": "Generated source issue: .scratch/distributed-gguf-runtime/issues/027-add-exact-llama-cpp-provenance-manifest-and-fetch-workspace.md; prd.json is authoritative.",
|
||||
"passes": true,
|
||||
"notes": "Completed from Gitea #11 via the Terra Ralph lane; independently reviewed after fail-closed cache identity hardening. Verified 7 focused tests, exact real-cache reuse, compileall, and diff-check. The pre-existing malformed 0002 patch is handed to DGR-028.",
|
||||
"blocks": [
|
||||
"DGR-028",
|
||||
"DGR-029",
|
||||
"DGR-044"
|
||||
]
|
||||
],
|
||||
"completionNotes": "Completed 2026-07-17. Added the exact llama.cpp provenance manifest packages/node/native/llama/UPSTREAM_LOCK.json pinning commit e920c523e3b8a0163fe498af5bf90df35ff51d25 (tree 6c91a11407a3a3fb160f5dac705f9c59718f54f1) with the sole git-clone-detached-commit retrieval into the ignored build/llama.cpp workspace, fail-closed dirty/mismatched-cache verification, PATCH-STACK.md, scripts/llama_cpp_dependency.py, and tests/test_llama_cpp_dependency.py (7 passed). Known blocker recorded for DGR-028: 0002-dense-llama-owned-range-loader.patch fails git apply --check against the pin, so no patched-tree/native-build claim is made. Evidence: evidence/DGR-027/README.md. These completionNotes were added during the DGR-025 backlog-consistency repair; the DGR-027 session omitted them."
|
||||
},
|
||||
{
|
||||
"id": "DGR-028",
|
||||
|
||||
@@ -12,6 +12,7 @@ from __future__ import annotations
|
||||
from dataclasses import dataclass
|
||||
|
||||
from .native_protocol import BUNDLE_VERSION, SCHEMA_VERSION, pb
|
||||
from .runtime_pin import load_runtime_pin
|
||||
from .runtime_recipe import (
|
||||
ArtifactIdentity,
|
||||
DerivativeBinding,
|
||||
@@ -72,7 +73,6 @@ class NativeNumericalRecipe:
|
||||
kv_layout: str
|
||||
architecture_adapter: str
|
||||
backend_id: str
|
||||
runtime_version: str
|
||||
recipe_id: str
|
||||
recipe_version: str
|
||||
catalogue_version: str
|
||||
@@ -95,6 +95,11 @@ def shard_identity_from_native_report(inputs: NativeIdentityInputs) -> ShardIden
|
||||
report = inputs.loaded_artifact
|
||||
pin = inputs.artifact_pin
|
||||
recipe = inputs.numerical_recipe
|
||||
if recipe.backend_id.strip().lower() not in {"llama.cpp", "llama-cpp"}:
|
||||
raise RecipeIdentityError(
|
||||
"native llama.cpp identity requires backend_id 'llama.cpp' or 'llama-cpp'"
|
||||
)
|
||||
runtime_version = load_runtime_pin().runtime_version
|
||||
artifact = ArtifactIdentity(
|
||||
artifact_id=pin.artifact_id,
|
||||
revision=pin.revision,
|
||||
@@ -115,7 +120,7 @@ def shard_identity_from_native_report(inputs: NativeIdentityInputs) -> ShardIden
|
||||
tokenizer_revision=inputs.tokenizer_revision,
|
||||
architecture_adapter=recipe.architecture_adapter,
|
||||
backend_id=recipe.backend_id,
|
||||
runtime_version=recipe.runtime_version,
|
||||
runtime_version=runtime_version,
|
||||
boundary_schema_version=recipe.boundary_schema_version,
|
||||
protocol_schema_version=recipe.protocol_schema_version,
|
||||
recipe_id=recipe.recipe_id,
|
||||
|
||||
221
packages/node/meshnet_node/runtime_pin.py
Normal file
221
packages/node/meshnet_node/runtime_pin.py
Normal file
@@ -0,0 +1,221 @@
|
||||
"""Canonical runtime pin identity for the recipe fingerprint (DGR-025).
|
||||
|
||||
The recipe digest (:mod:`meshnet_node.runtime_recipe`) commits to a
|
||||
``runtime_version`` axis, but a string the operator typed is a label, not a
|
||||
pin: two workers could run different patch stacks under the same label and
|
||||
still hash to the same recipe. The DGR-027 lock manifest
|
||||
(``packages/node/native/llama``) already records the one exact upstream commit
|
||||
and the ordered patch stack the native runtime is built from, so the axis
|
||||
value is *derived* from that manifest, never asserted.
|
||||
|
||||
The derived value has three load-bearing parts, and each is separately fatal
|
||||
to compatibility: the runtime name (from the upstream URL), the exact
|
||||
40-character upstream commit, and a digest over the ordered patch-stack bytes.
|
||||
A different upstream pin, a reordered stack, or a single changed patch byte
|
||||
each produce a different axis value, which produces a different recipe digest,
|
||||
which partitions the route — exactly the fail-closed behavior DGR-025 asks
|
||||
for.
|
||||
|
||||
Every consistency check here fails closed. The manifest keeps three records of
|
||||
the stack — ``UPSTREAM_LOCK.json``'s ``patch_series``, ``patches/series``, and
|
||||
``patches/SHA256SUMS`` — plus the ``UPSTREAM_COMMIT`` convenience file, and a
|
||||
disagreement between any two of them means the workspace's identity is
|
||||
unknowable, not "probably fine". This module reads the committed manifest
|
||||
only; fetching and patching the actual source tree stays with
|
||||
``scripts/llama_cpp_dependency.py`` (DGR-027).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
|
||||
# Domain separation, matching the runtime_recipe digest convention: a patch
|
||||
# stack digest can never be confused with an artifact or recipe digest.
|
||||
PATCH_STACK_DIGEST_DOMAIN = "meshnet.runtime-patch-stack.v1"
|
||||
|
||||
# The UPSTREAM_LOCK.json layout this reader understands (DGR-027 schema).
|
||||
RUNTIME_PIN_SCHEMA_VERSION = 1
|
||||
|
||||
# The committed DGR-027 manifest for the llama.cpp runtime.
|
||||
DEFAULT_LOCK_DIR = Path(__file__).resolve().parent.parent / "native" / "llama"
|
||||
|
||||
_HEX40 = re.compile(r"^[0-9a-f]{40}$")
|
||||
_HEX64 = re.compile(r"^[0-9a-f]{64}$")
|
||||
|
||||
|
||||
class RuntimePinError(ValueError):
|
||||
"""The lock workspace is missing, malformed, or internally inconsistent."""
|
||||
|
||||
|
||||
def _canonical_sha256(value: object) -> str:
|
||||
payload = json.dumps(
|
||||
value, sort_keys=True, separators=(",", ":"), ensure_ascii=False
|
||||
)
|
||||
return hashlib.sha256(payload.encode("utf-8")).hexdigest()
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class RuntimePin:
|
||||
"""One exact runtime: a name, an upstream commit, and an ordered patch stack."""
|
||||
|
||||
runtime_name: str
|
||||
upstream_commit: str
|
||||
patch_series: tuple[str, ...]
|
||||
patch_digests: tuple[str, ...]
|
||||
|
||||
@property
|
||||
def patch_stack_digest(self) -> str:
|
||||
"""A digest over the ordered (name, bytes-digest) stack.
|
||||
|
||||
Order is digested deliberately: applying the same patches in a
|
||||
different order can produce a different tree, so a reordered stack is
|
||||
a different runtime.
|
||||
"""
|
||||
return _canonical_sha256(
|
||||
{
|
||||
"domain": PATCH_STACK_DIGEST_DOMAIN,
|
||||
"body": {
|
||||
"patches": [
|
||||
[name, digest]
|
||||
for name, digest in zip(self.patch_series, self.patch_digests)
|
||||
]
|
||||
},
|
||||
}
|
||||
)
|
||||
|
||||
@property
|
||||
def runtime_version(self) -> str:
|
||||
"""The exact ``runtime_version`` recipe axis value for this pin."""
|
||||
return (
|
||||
f"{self.runtime_name}@{self.upstream_commit}"
|
||||
f"+patchstack.{self.patch_stack_digest}"
|
||||
)
|
||||
|
||||
|
||||
def _read_text(path: Path, what: str) -> str:
|
||||
try:
|
||||
return path.read_text(encoding="utf-8")
|
||||
except FileNotFoundError:
|
||||
raise RuntimePinError(f"{what} not found at {path}") from None
|
||||
except OSError as exc:
|
||||
raise RuntimePinError(f"{what} at {path} is unreadable: {exc}") from exc
|
||||
|
||||
|
||||
def _read_series_file(path: Path) -> list[str]:
|
||||
lines = _read_text(path, "patches/series").splitlines()
|
||||
return [line.strip() for line in lines if line.strip() and not line.startswith("#")]
|
||||
|
||||
|
||||
def _read_sums_file(path: Path) -> list[tuple[str, str]]:
|
||||
entries: list[tuple[str, str]] = []
|
||||
for line in _read_text(path, "patches/SHA256SUMS").splitlines():
|
||||
line = line.strip()
|
||||
if not line or line.startswith("#"):
|
||||
continue
|
||||
parts = line.split(None, 1)
|
||||
if len(parts) != 2 or not _HEX64.match(parts[0]):
|
||||
raise RuntimePinError(
|
||||
"patches/SHA256SUMS contains a line that is not "
|
||||
"'<sha256> <patch name>'"
|
||||
)
|
||||
entries.append((parts[0], parts[1].strip()))
|
||||
return entries
|
||||
|
||||
|
||||
def load_runtime_pin(lock_dir: Path = DEFAULT_LOCK_DIR) -> RuntimePin:
|
||||
"""Derive the exact runtime pin from a DGR-027 lock workspace, or refuse.
|
||||
|
||||
Refuses — rather than guessing — on a missing or malformed lock, a moving
|
||||
commit reference, a disagreement between the lock's ``patch_series``, the
|
||||
``patches/series`` file, ``patches/SHA256SUMS``, or the actual patch
|
||||
bytes, and on an ``UPSTREAM_COMMIT`` file that names a different commit.
|
||||
"""
|
||||
lock_path = lock_dir / "UPSTREAM_LOCK.json"
|
||||
raw = _read_text(lock_path, "UPSTREAM_LOCK.json")
|
||||
try:
|
||||
lock = json.loads(raw)
|
||||
except json.JSONDecodeError as exc:
|
||||
raise RuntimePinError(f"UPSTREAM_LOCK.json is not valid JSON: {exc}") from exc
|
||||
if not isinstance(lock, dict):
|
||||
raise RuntimePinError("UPSTREAM_LOCK.json must be a JSON object")
|
||||
|
||||
schema = lock.get("schema_version")
|
||||
if schema != RUNTIME_PIN_SCHEMA_VERSION:
|
||||
raise RuntimePinError(
|
||||
f"UPSTREAM_LOCK.json declares schema version {schema!r}; this reader "
|
||||
f"understands version {RUNTIME_PIN_SCHEMA_VERSION}"
|
||||
)
|
||||
|
||||
upstream = lock.get("upstream")
|
||||
if not isinstance(upstream, str) or not upstream.strip():
|
||||
raise RuntimePinError("UPSTREAM_LOCK.json is missing the upstream URL")
|
||||
runtime_name = upstream.rstrip("/").rsplit("/", 1)[-1]
|
||||
if runtime_name.endswith(".git"):
|
||||
runtime_name = runtime_name[: -len(".git")]
|
||||
if not runtime_name:
|
||||
raise RuntimePinError("the upstream URL does not name a runtime")
|
||||
|
||||
commit = lock.get("commit")
|
||||
if not isinstance(commit, str) or not _HEX40.match(commit):
|
||||
raise RuntimePinError(
|
||||
f"UPSTREAM_LOCK.json commit {commit!r} is not an exact 40-character "
|
||||
"hexadecimal object id; a moving reference is not a pin"
|
||||
)
|
||||
|
||||
commit_file = _read_text(lock_dir / "UPSTREAM_COMMIT", "UPSTREAM_COMMIT")
|
||||
recorded = commit_file.strip().splitlines()[0].strip() if commit_file.strip() else ""
|
||||
if recorded != commit:
|
||||
raise RuntimePinError(
|
||||
"UPSTREAM_COMMIT and UPSTREAM_LOCK.json disagree on the pinned commit"
|
||||
)
|
||||
|
||||
lock_series = lock.get("patch_series")
|
||||
if not isinstance(lock_series, list) or not all(
|
||||
isinstance(name, str) and name.strip() for name in lock_series
|
||||
):
|
||||
raise RuntimePinError(
|
||||
"UPSTREAM_LOCK.json patch_series must be a list of patch file names"
|
||||
)
|
||||
if len(set(lock_series)) != len(lock_series):
|
||||
raise RuntimePinError("UPSTREAM_LOCK.json patch_series contains a duplicate")
|
||||
|
||||
series = _read_series_file(lock_dir / "patches" / "series")
|
||||
if series != lock_series:
|
||||
raise RuntimePinError(
|
||||
"patches/series and UPSTREAM_LOCK.json patch_series disagree on the "
|
||||
"ordered patch stack"
|
||||
)
|
||||
|
||||
sums = _read_sums_file(lock_dir / "patches" / "SHA256SUMS")
|
||||
if [name for _, name in sums] != lock_series:
|
||||
raise RuntimePinError(
|
||||
"patches/SHA256SUMS does not record exactly the ordered patch stack "
|
||||
"named by UPSTREAM_LOCK.json"
|
||||
)
|
||||
|
||||
digests: list[str] = []
|
||||
for (expected_digest, name) in sums:
|
||||
patch_path = lock_dir / "patches" / name
|
||||
try:
|
||||
body = patch_path.read_bytes()
|
||||
except FileNotFoundError:
|
||||
raise RuntimePinError(f"patch file {name} is named but missing") from None
|
||||
except OSError as exc:
|
||||
raise RuntimePinError(f"patch file {name} is unreadable: {exc}") from exc
|
||||
actual = hashlib.sha256(body).hexdigest()
|
||||
if actual != expected_digest:
|
||||
raise RuntimePinError(
|
||||
f"patch file {name} does not match its patches/SHA256SUMS digest"
|
||||
)
|
||||
digests.append(actual)
|
||||
|
||||
return RuntimePin(
|
||||
runtime_name=runtime_name,
|
||||
upstream_commit=commit,
|
||||
patch_series=tuple(lock_series),
|
||||
patch_digests=tuple(digests),
|
||||
)
|
||||
@@ -125,6 +125,10 @@ _AXIS_MISMATCH: Mapping[str, str] = {
|
||||
}
|
||||
|
||||
_HEX64 = re.compile(r"^[0-9a-f]{64}$")
|
||||
_LLAMA_CPP_RUNTIME_PIN = re.compile(
|
||||
r"^llama\.cpp@[0-9a-f]{40}\+patchstack\.[0-9a-f]{64}$"
|
||||
)
|
||||
_LLAMA_CPP_BACKEND_IDS = frozenset({"llama.cpp", "llama-cpp"})
|
||||
|
||||
# A revision that can move is not a pin. DGR-017 learned this on the artifact;
|
||||
# it is just as true of a tokenizer.
|
||||
@@ -194,6 +198,17 @@ def _require_pin(value: Any, what: str) -> str:
|
||||
return text
|
||||
|
||||
|
||||
def _require_runtime_pin(value: Any, backend_id: Any) -> str:
|
||||
text = _require_pin(value, "recipe.runtime_version")
|
||||
backend = _require_text(backend_id, "recipe.backend_id").strip().lower()
|
||||
if backend in _LLAMA_CPP_BACKEND_IDS and not _LLAMA_CPP_RUNTIME_PIN.fullmatch(text):
|
||||
raise RecipeIdentityError(
|
||||
"'recipe.runtime_version' for llama.cpp must be "
|
||||
"'llama.cpp@<40-hex commit>+patchstack.<64-hex digest>'"
|
||||
)
|
||||
return text
|
||||
|
||||
|
||||
def _as_mapping(value: Any, what: str) -> Mapping[str, Any]:
|
||||
if not isinstance(value, Mapping):
|
||||
raise RecipeIdentityError(
|
||||
@@ -374,6 +389,11 @@ class RuntimeRecipe:
|
||||
one in fp16, produce different logits from the same bytes. Keeping the axes
|
||||
apart is the entire safety property; see :data:`RECIPE_AXES`.
|
||||
|
||||
`tokenizer_revision` and `runtime_version` must be exact pins, never moving
|
||||
references. For the native runtime the canonical `runtime_version` value —
|
||||
committing to the exact upstream commit *and* the ordered patch stack — is
|
||||
derived from the DGR-027 lock manifest by :mod:`meshnet_node.runtime_pin`.
|
||||
|
||||
The three label fields are diagnosis only and are not digested.
|
||||
"""
|
||||
|
||||
@@ -400,6 +420,7 @@ class RuntimeRecipe:
|
||||
else:
|
||||
_require_text(value, f"recipe.{axis}")
|
||||
_require_pin(self.tokenizer_revision, "recipe.tokenizer_revision")
|
||||
_require_runtime_pin(self.runtime_version, self.backend_id)
|
||||
_require_text(self.recipe_id, "recipe.recipe_id")
|
||||
_require_text(self.recipe_version, "recipe.recipe_version")
|
||||
_require_text(self.catalogue_version, "recipe.catalogue_version")
|
||||
|
||||
@@ -4,10 +4,8 @@
|
||||
# never committed. A C++ consumer already needs a toolchain, so committing
|
||||
# generated C++ would only create a second copy of the schema that can rot.
|
||||
#
|
||||
# gRPC C++ is optional here on purpose. The conformance test only needs message
|
||||
# types, so the schema can be verified on a machine that has protobuf but not
|
||||
# the gRPC C++ stack. When gRPC *is* found, the service stubs are generated too
|
||||
# and exported as `shard_runtime_grpc` for the worker (DGR-008) to link.
|
||||
# Protobuf and gRPC C++ are required together so message and service bindings are
|
||||
# generated by one exact toolchain. The ignored bootstrap prefix supplies both.
|
||||
#
|
||||
# Build:
|
||||
# cmake -S packages/node/native -B build/native -DCMAKE_PREFIX_PATH=<protobuf-install>
|
||||
@@ -23,8 +21,17 @@ project(meshnet_shard_protocol CXX)
|
||||
set(CMAKE_CXX_STANDARD 17)
|
||||
set(CMAKE_CXX_STANDARD_REQUIRED ON)
|
||||
|
||||
find_package(protobuf CONFIG REQUIRED)
|
||||
find_package(gRPC CONFIG QUIET)
|
||||
# Protobuf and gRPC are one pinned generation toolchain. Configure only against
|
||||
# the ignored prefix produced by scripts/bootstrap_native_toolchain.sh; accepting
|
||||
# an arbitrary system plugin would make generated service bindings host-dependent.
|
||||
set(MESHNET_PROTOBUF_VERSION "33.1.0")
|
||||
set(MESHNET_GRPC_VERSION "1.82.1")
|
||||
find_package(protobuf ${MESHNET_PROTOBUF_VERSION} EXACT CONFIG REQUIRED)
|
||||
find_package(gRPC ${MESHNET_GRPC_VERSION} EXACT CONFIG REQUIRED)
|
||||
|
||||
if(NOT TARGET gRPC::grpc_cpp_plugin)
|
||||
message(FATAL_ERROR "pinned gRPC package does not export grpc_cpp_plugin")
|
||||
endif()
|
||||
|
||||
set(SHARD_PROTO "${CMAKE_CURRENT_SOURCE_DIR}/proto/shard_runtime.proto")
|
||||
|
||||
@@ -39,24 +46,19 @@ protobuf_generate(
|
||||
PROTOC_OUT_DIR "${CMAKE_CURRENT_BINARY_DIR}"
|
||||
)
|
||||
|
||||
# Service stubs: only when the gRPC C++ stack is present.
|
||||
if(gRPC_FOUND)
|
||||
add_library(shard_runtime_grpc STATIC "${SHARD_PROTO}")
|
||||
target_link_libraries(shard_runtime_grpc PUBLIC shard_runtime_proto gRPC::grpc++)
|
||||
target_include_directories(shard_runtime_grpc PUBLIC "${CMAKE_CURRENT_BINARY_DIR}")
|
||||
protobuf_generate(
|
||||
# Service stubs are part of the reproducibility contract, not an optional branch.
|
||||
add_library(shard_runtime_grpc STATIC "${SHARD_PROTO}")
|
||||
target_link_libraries(shard_runtime_grpc PUBLIC shard_runtime_proto gRPC::grpc++)
|
||||
target_include_directories(shard_runtime_grpc PUBLIC "${CMAKE_CURRENT_BINARY_DIR}")
|
||||
protobuf_generate(
|
||||
TARGET shard_runtime_grpc
|
||||
LANGUAGE grpc
|
||||
GENERATE_EXTENSIONS .grpc.pb.h .grpc.pb.cc
|
||||
PLUGIN "protoc-gen-grpc=$<TARGET_FILE:gRPC::grpc_cpp_plugin>"
|
||||
IMPORT_DIRS "${CMAKE_CURRENT_SOURCE_DIR}/proto"
|
||||
PROTOC_OUT_DIR "${CMAKE_CURRENT_BINARY_DIR}"
|
||||
)
|
||||
message(STATUS "gRPC C++ found: building ShardRuntime service stubs")
|
||||
else()
|
||||
message(STATUS "gRPC C++ not found: building message types only "
|
||||
"(sufficient for the conformance test)")
|
||||
endif()
|
||||
)
|
||||
message(STATUS "Pinned gRPC ${gRPC_VERSION}: building ShardRuntime service stubs")
|
||||
|
||||
enable_testing()
|
||||
|
||||
|
||||
@@ -46,7 +46,7 @@ sampled token to request/recipe identity and sampling/template/reasoning inputs.
|
||||
If the machine has no protobuf C++ toolchain:
|
||||
|
||||
```bash
|
||||
scripts/bootstrap_native_toolchain.sh build/native-toolchain
|
||||
bash scripts/bootstrap_native_toolchain.sh build/native-toolchain
|
||||
```
|
||||
|
||||
Then:
|
||||
@@ -58,10 +58,10 @@ cmake --build build/native -j
|
||||
ctest --test-dir build/native --output-on-failure
|
||||
```
|
||||
|
||||
gRPC C++ is optional: without it, CMake builds the message types only, which is
|
||||
all the conformance test needs. When gRPC C++ *is* found, the `ShardRuntime`
|
||||
service stubs are built too and exported as `shard_runtime_grpc` for the worker
|
||||
(DGR-008) to link.
|
||||
The bootstrap pins and builds Protobuf `33.1`, gRPC C++ `1.82.1`, and the
|
||||
matching `grpc_cpp_plugin` into one ignored prefix. CMake requires those exact
|
||||
package versions and always generates both message and service stubs; it does
|
||||
not fall back to an arbitrary system plugin.
|
||||
|
||||
## How the cross-language check actually proves something
|
||||
|
||||
|
||||
@@ -15,11 +15,19 @@ loading, endpoint ownership, architecture-defined intermediate boundaries, and
|
||||
layer-filtered KV/session mapping. Meshnet routing, Tracker, gRPC, relay,
|
||||
billing, authentication, and telemetry must remain outside this directory.
|
||||
|
||||
`scripts/llama_cpp_dependency.py` verifies the exact commit/tree and baseline
|
||||
blobs, validates every patch digest and context with `git apply --check`, then
|
||||
applies the series in `patches/series` order. It refuses a dirty source tree,
|
||||
wrong commit/tree/blob, changed patch digest, reordered series, or an existing
|
||||
destination/work directory.
|
||||
`scripts/llama_cpp_dependency.py fetch` reads the in-repo manifest and checks out
|
||||
only its exact commit as detached HEAD in `build/llama.cpp/source`, an ignored
|
||||
build workspace. It verifies the exact commit/tree and baseline blobs before use. A
|
||||
later offline `fetch` may reuse that cache only after the same clean identity
|
||||
verification; an attached branch, tag/repository override, arbitrary destination,
|
||||
symlinked workspace, dirty checkout, ignored injected file, or tracked-file
|
||||
modification hidden by Git index flags is refused. The tool
|
||||
validates every patch digest and context
|
||||
with `git apply --check`, then applies the series in `patches/series` order.
|
||||
Tracked executable modes are checked physically when Git reports
|
||||
`core.filemode=true`; on mounted filesystems without POSIX mode fidelity, the
|
||||
locked index tree remains the canonical mode record while every working-file
|
||||
blob is independently re-hashed.
|
||||
|
||||
## Current semantic boundary
|
||||
|
||||
|
||||
@@ -3,6 +3,13 @@
|
||||
"upstream": "https://github.com/ggml-org/llama.cpp.git",
|
||||
"commit": "e920c523e3b8a0163fe498af5bf90df35ff51d25",
|
||||
"commit_tree": "6c91a11407a3a3fb160f5dac705f9c59718f54f1",
|
||||
"expected_source": {
|
||||
"git_tree": "6c91a11407a3a3fb160f5dac705f9c59718f54f1"
|
||||
},
|
||||
"retrieval": {
|
||||
"method": "git-clone-detached-commit",
|
||||
"workspace": "build/llama.cpp"
|
||||
},
|
||||
"patched_tree": "322d8b463df74a2226f0b513176643d815f54452",
|
||||
"upstream_license": "MIT",
|
||||
"patch_series": [
|
||||
|
||||
@@ -99,18 +99,9 @@ def select_safetensors_files_for_layers(
|
||||
if not isinstance(tensor_name, str) or not isinstance(rel_file, str):
|
||||
continue
|
||||
if _tensor_belongs_to_range(tensor_name, start_layer, end_layer, inferred_total_layers):
|
||||
rel = _normalise_relative_file(rel_file)
|
||||
if not (root / rel).is_file():
|
||||
# A required weight file is missing from this local snapshot: the
|
||||
# snapshot is incomplete for this layer range, so it must not be
|
||||
# advertised as a source at all. Silently dropping just this file
|
||||
# (as before) made a partial snapshot look "fully cached" to
|
||||
# downstream clients, which then skipped the real download and
|
||||
# only discovered the missing weights at model-load time.
|
||||
return []
|
||||
selected.add(rel)
|
||||
selected.add(_normalise_relative_file(rel_file))
|
||||
|
||||
return sorted(selected)
|
||||
return sorted(rel for rel in selected if (root / rel).is_file())
|
||||
|
||||
|
||||
def _tensor_belongs_to_range(
|
||||
|
||||
@@ -70,6 +70,10 @@ STATUS_CERTIFIED = "certified"
|
||||
MIN_CERTIFYING_NODES = 2
|
||||
|
||||
_HEX64 = re.compile(r"^[0-9a-f]{64}$")
|
||||
_LLAMA_CPP_RUNTIME_PIN = re.compile(
|
||||
r"^llama\.cpp@[0-9a-f]{40}\+patchstack\.[0-9a-f]{64}$"
|
||||
)
|
||||
_LLAMA_CPP_BACKEND_IDS = frozenset({"llama.cpp", "llama-cpp"})
|
||||
_MOVING_REFS = frozenset({"main", "master", "head", "latest", "dev", "trunk"})
|
||||
|
||||
|
||||
@@ -128,6 +132,17 @@ def _pin(value: Any, what: str) -> str:
|
||||
return text
|
||||
|
||||
|
||||
def _runtime_pin(value: Any, backend_id: Any) -> str:
|
||||
text = _pin(value, "recipe.runtime_version")
|
||||
backend = _text(backend_id, "recipe.backend_id").strip().lower()
|
||||
if backend in _LLAMA_CPP_BACKEND_IDS and not _LLAMA_CPP_RUNTIME_PIN.fullmatch(text):
|
||||
raise RecipeIdentityError(
|
||||
"'recipe.runtime_version' for llama.cpp must bind a 40-hex commit "
|
||||
"and a 64-hex ordered patch-stack digest"
|
||||
)
|
||||
return text
|
||||
|
||||
|
||||
def _mapping(value: Any, what: str) -> Mapping[str, Any]:
|
||||
if not isinstance(value, Mapping):
|
||||
raise RecipeIdentityError(f"{what!r} must be a JSON object")
|
||||
@@ -333,6 +348,7 @@ def parse_identity(data: Any) -> PresentedIdentity:
|
||||
else:
|
||||
axes[axis] = _text(value, f"recipe.{axis}")
|
||||
_pin(axes["tokenizer_revision"], "recipe.tokenizer_revision")
|
||||
_runtime_pin(axes["runtime_version"], axes["backend_id"])
|
||||
|
||||
identity = PresentedIdentity(
|
||||
artifact_id=_text(artifact.get("artifact_id"), "artifact.artifact_id"),
|
||||
|
||||
@@ -1,18 +1,13 @@
|
||||
#!/usr/bin/env bash
|
||||
# Build a protobuf C++ toolchain for the native Shard protocol.
|
||||
#
|
||||
# The Python side needs nothing beyond `pip install grpcio-tools` — it bundles
|
||||
# protoc. The C++ side needs libprotobuf headers and a protoc binary, and a
|
||||
# machine that has neither (no protobuf-devel, no cmake, no system protoc) can
|
||||
# still get a working one from source with this script. It is the exact recipe
|
||||
# DGR-002 used to build and run the C++ conformance test.
|
||||
#
|
||||
# gRPC C++ is deliberately NOT built here. The conformance test only needs
|
||||
# message types, so verifying the schema does not require the whole gRPC stack.
|
||||
# The worker (DGR-008) will need gRPC C++ and should extend this script then.
|
||||
# The Python side uses the exact grpcio-tools pin declared below. The C++ side
|
||||
# builds exact Protobuf, Abseil, and gRPC source revisions so `protoc`,
|
||||
# `grpc_cpp_plugin`, headers, and libraries all come from one ignored prefix.
|
||||
# No system Protobuf/gRPC installation is accepted by the documented build.
|
||||
#
|
||||
# Usage:
|
||||
# scripts/bootstrap_native_toolchain.sh [install-prefix]
|
||||
# bash scripts/bootstrap_native_toolchain.sh [install-prefix]
|
||||
#
|
||||
# Then:
|
||||
# cmake -S packages/node/native -B build/native -DCMAKE_PREFIX_PATH=<prefix>
|
||||
@@ -21,7 +16,17 @@
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
PREFIX="${1:-${PWD}/build/native-toolchain}"
|
||||
resolve_prefix() {
|
||||
local candidate="${1:-${PWD}/build/native-toolchain}"
|
||||
realpath -m -- "${candidate}"
|
||||
}
|
||||
|
||||
if [[ "${1:-}" == "--print-prefix" ]]; then
|
||||
resolve_prefix "${2:-}"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
PREFIX="$(resolve_prefix "${1:-}")"
|
||||
WORK="$(mktemp -d)"
|
||||
trap 'rm -rf "${WORK}"' EXIT
|
||||
|
||||
@@ -29,11 +34,15 @@ trap 'rm -rf "${WORK}"' EXIT
|
||||
# that stub is allowed to use, so these are exact, not floating.
|
||||
PROTOBUF_VERSION="33.1"
|
||||
ABSEIL_VERSION="20250814.1"
|
||||
GRPC_VERSION="1.82.1"
|
||||
GRPC_COMMIT="acccf84c0df20487d64101f528e5d426541ca4e5"
|
||||
|
||||
command -v cmake >/dev/null || {
|
||||
echo "cmake is required (pip install cmake==4.4.0)" >&2
|
||||
for tool in cmake curl git realpath sha256sum tar; do
|
||||
command -v "${tool}" >/dev/null || {
|
||||
echo "${tool} is required" >&2
|
||||
exit 1
|
||||
}
|
||||
}
|
||||
done
|
||||
|
||||
echo "--- fetching protobuf ${PROTOBUF_VERSION} and abseil ${ABSEIL_VERSION}"
|
||||
cd "${WORK}"
|
||||
@@ -41,26 +50,77 @@ curl -sfL -o protobuf.tar.gz \
|
||||
"https://github.com/protocolbuffers/protobuf/releases/download/v${PROTOBUF_VERSION}/protobuf-${PROTOBUF_VERSION}.tar.gz"
|
||||
tar xzf protobuf.tar.gz
|
||||
|
||||
# The protobuf release tarball ships utf8_range but not abseil, and its default
|
||||
# CMake provider expects abseil as a submodule, so vendor it into place.
|
||||
curl -sfL -o abseil.tar.gz \
|
||||
"https://github.com/abseil/abseil-cpp/releases/download/${ABSEIL_VERSION}/abseil-cpp-${ABSEIL_VERSION}.tar.gz"
|
||||
tar xzf abseil.tar.gz
|
||||
rm -rf "protobuf-${PROTOBUF_VERSION}/third_party/abseil-cpp"
|
||||
mv "abseil-cpp-${ABSEIL_VERSION}" "protobuf-${PROTOBUF_VERSION}/third_party/abseil-cpp"
|
||||
|
||||
echo "--- building protobuf into ${PREFIX}"
|
||||
cmake -S "protobuf-${PROTOBUF_VERSION}" -B build \
|
||||
echo "--- building abseil ${ABSEIL_VERSION} into ${PREFIX}"
|
||||
cmake -S "abseil-cpp-${ABSEIL_VERSION}" -B abseil-build \
|
||||
-DCMAKE_BUILD_TYPE=Release \
|
||||
-DCMAKE_INSTALL_PREFIX="${PREFIX}" \
|
||||
-DCMAKE_POSITION_INDEPENDENT_CODE=ON \
|
||||
-Dprotobuf_ABSL_PROVIDER=module \
|
||||
-Dprotobuf_BUILD_TESTS=OFF \
|
||||
-Dprotobuf_BUILD_SHARED_LIBS=OFF \
|
||||
-DABSL_ENABLE_INSTALL=ON \
|
||||
-DABSL_BUILD_TESTING=OFF \
|
||||
-DABSL_PROPAGATE_CXX_STD=ON
|
||||
cmake --build build -j"$(nproc)"
|
||||
cmake --install build
|
||||
cmake --build abseil-build -j"$(nproc)"
|
||||
cmake --install abseil-build
|
||||
|
||||
echo "--- building protobuf ${PROTOBUF_VERSION} into ${PREFIX}"
|
||||
cmake -S "protobuf-${PROTOBUF_VERSION}" -B protobuf-build \
|
||||
-DCMAKE_BUILD_TYPE=Release \
|
||||
-DCMAKE_INSTALL_PREFIX="${PREFIX}" \
|
||||
-DCMAKE_PREFIX_PATH="${PREFIX}" \
|
||||
-Dabsl_DIR="${PREFIX}/lib64/cmake/absl" \
|
||||
-DCMAKE_POSITION_INDEPENDENT_CODE=ON \
|
||||
-Dprotobuf_LOCAL_DEPENDENCIES_ONLY=ON \
|
||||
-Dprotobuf_BUILD_TESTS=OFF \
|
||||
-Dprotobuf_BUILD_SHARED_LIBS=OFF
|
||||
cmake --build protobuf-build -j"$(nproc)"
|
||||
cmake --install protobuf-build
|
||||
|
||||
echo "--- fetching gRPC ${GRPC_VERSION} at ${GRPC_COMMIT}"
|
||||
git init -q grpc-source
|
||||
git -C grpc-source remote add origin https://github.com/grpc/grpc.git
|
||||
git -C grpc-source fetch --depth 1 origin "${GRPC_COMMIT}"
|
||||
git -C grpc-source checkout --detach FETCH_HEAD
|
||||
git -C grpc-source submodule update --init --recursive --depth 1
|
||||
[[ "$(git -C grpc-source rev-parse HEAD)" == "${GRPC_COMMIT}" ]] || {
|
||||
echo "gRPC checkout identity mismatch" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
echo "--- building gRPC ${GRPC_VERSION} and grpc_cpp_plugin into ${PREFIX}"
|
||||
cmake -S grpc-source -B grpc-build \
|
||||
-DCMAKE_BUILD_TYPE=Release \
|
||||
-DCMAKE_INSTALL_PREFIX="${PREFIX}" \
|
||||
-DCMAKE_PREFIX_PATH="${PREFIX}" \
|
||||
-DProtobuf_DIR="${PREFIX}/lib64/cmake/protobuf" \
|
||||
-Dabsl_DIR="${PREFIX}/lib64/cmake/absl" \
|
||||
-DgRPC_INSTALL=ON \
|
||||
-DgRPC_BUILD_TESTS=OFF \
|
||||
-DgRPC_PROTOBUF_PROVIDER=package \
|
||||
-DgRPC_ABSL_PROVIDER=package \
|
||||
-DgRPC_CARES_PROVIDER=module \
|
||||
-DgRPC_RE2_PROVIDER=module \
|
||||
-DgRPC_SSL_PROVIDER=module \
|
||||
-DgRPC_ZLIB_PROVIDER=module \
|
||||
-DgRPC_BUILD_GRPC_CPP_PLUGIN=ON \
|
||||
-DgRPC_BUILD_GRPC_CSHARP_PLUGIN=OFF \
|
||||
-DgRPC_BUILD_GRPC_NODE_PLUGIN=OFF \
|
||||
-DgRPC_BUILD_GRPC_OBJECTIVE_C_PLUGIN=OFF \
|
||||
-DgRPC_BUILD_GRPC_PHP_PLUGIN=OFF \
|
||||
-DgRPC_BUILD_GRPC_PYTHON_PLUGIN=OFF \
|
||||
-DgRPC_BUILD_GRPC_RUBY_PLUGIN=OFF
|
||||
cmake --build grpc-build -j"$(nproc)"
|
||||
cmake --install grpc-build
|
||||
|
||||
echo "--- done"
|
||||
"${PREFIX}/bin/protoc" --version
|
||||
[[ -x "${PREFIX}/bin/grpc_cpp_plugin" ]] || {
|
||||
echo "grpc_cpp_plugin was not installed" >&2
|
||||
exit 1
|
||||
}
|
||||
printf 'gRPC %s commit %s\n' "${GRPC_VERSION}" "${GRPC_COMMIT}"
|
||||
printf 'grpc_cpp_plugin sha256 '
|
||||
sha256sum "${PREFIX}/bin/grpc_cpp_plugin" | cut -d' ' -f1
|
||||
echo "configure the protocol build with: -DCMAKE_PREFIX_PATH=${PREFIX}"
|
||||
|
||||
@@ -49,7 +49,7 @@ _RECIPE = RuntimeRecipe(
|
||||
tokenizer_revision="0123456789abcdef",
|
||||
architecture_adapter="llama/range-v1",
|
||||
backend_id="llama.cpp",
|
||||
runtime_version="llama.cpp@deadbeef+meshnet.1",
|
||||
runtime_version="llama.cpp@" + "d" * 40 + "+patchstack." + "e" * 64,
|
||||
recipe_id="example-gguf",
|
||||
recipe_version="1",
|
||||
catalogue_version="2026.07.1",
|
||||
|
||||
@@ -20,6 +20,7 @@ already requires a toolchain and nothing is gained by committing them.
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
from importlib import metadata
|
||||
import pathlib
|
||||
import shutil
|
||||
import subprocess
|
||||
@@ -38,15 +39,32 @@ REQUIRED_GRPCIO_TOOLS = "1.82.1"
|
||||
_HEADER = "# Generated by scripts/generate_native_protocol.py. Do not edit.\n"
|
||||
|
||||
|
||||
def _generate(into: pathlib.Path) -> None:
|
||||
"""Run protoc, writing generated modules into `into`."""
|
||||
def _require_grpcio_tools_version() -> None:
|
||||
try:
|
||||
from grpc_tools import protoc
|
||||
except ImportError: # pragma: no cover - exercised only without the toolchain
|
||||
actual = metadata.version("grpcio-tools")
|
||||
except metadata.PackageNotFoundError:
|
||||
sys.exit(
|
||||
"grpc_tools is required to generate stubs:\n"
|
||||
f" pip install grpcio-tools=={REQUIRED_GRPCIO_TOOLS}"
|
||||
)
|
||||
if actual != REQUIRED_GRPCIO_TOOLS:
|
||||
sys.exit(
|
||||
"wrong grpcio-tools version for deterministic generation: "
|
||||
f"found {actual}, require {REQUIRED_GRPCIO_TOOLS}\n"
|
||||
f" pip install --upgrade grpcio-tools=={REQUIRED_GRPCIO_TOOLS}"
|
||||
)
|
||||
|
||||
|
||||
def _generate(into: pathlib.Path) -> None:
|
||||
"""Run the exactly pinned protoc, writing generated modules into `into`."""
|
||||
_require_grpcio_tools_version()
|
||||
try:
|
||||
from grpc_tools import protoc
|
||||
except ImportError: # pragma: no cover - inconsistent/broken installation
|
||||
sys.exit(
|
||||
"grpcio-tools metadata exists but grpc_tools cannot be imported; reinstall it:\n"
|
||||
f" pip install --force-reinstall grpcio-tools=={REQUIRED_GRPCIO_TOOLS}"
|
||||
)
|
||||
|
||||
into.mkdir(parents=True, exist_ok=True)
|
||||
# grpc_tools bundles protoc and the well-known types, so generation needs no
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Materialize, verify, build, and smoke-test DGR-004's llama.cpp pin.
|
||||
"""Fetch, verify, build, and smoke-test DGR-027's exact llama.cpp pin.
|
||||
|
||||
This tool deliberately owns only a source dependency boundary. It never
|
||||
downloads a model, invokes inference, or interprets generated text.
|
||||
@@ -12,6 +12,7 @@ import hashlib
|
||||
import json
|
||||
import os
|
||||
import pathlib
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
@@ -67,14 +68,27 @@ def _load_lock() -> dict[str, Any]:
|
||||
raise DependencyError(f"invalid upstream lock: {LOCK_PATH}: {error}") from error
|
||||
required = {
|
||||
"upstream", "commit", "commit_tree", "patched_tree", "patch_series",
|
||||
"required_upstream_blobs", "patched_paths", "build",
|
||||
"required_upstream_blobs", "patched_paths", "build", "upstream_license",
|
||||
"expected_source", "retrieval",
|
||||
}
|
||||
missing = sorted(required - lock.keys())
|
||||
if missing:
|
||||
raise DependencyError(f"upstream lock is missing fields: {', '.join(missing)}")
|
||||
commit_file = (LLAMA_DIR / "UPSTREAM_COMMIT").read_text().strip()
|
||||
if lock["commit"] != commit_file or len(commit_file) != 40:
|
||||
raise DependencyError("UPSTREAM_COMMIT and UPSTREAM_LOCK.json do not agree on a full commit")
|
||||
object_ids = [commit_file, lock["commit"], lock["commit_tree"], lock["patched_tree"]]
|
||||
if lock["commit"] != commit_file or not all(
|
||||
isinstance(value, str) and re.fullmatch(r"[0-9a-f]{40}", value)
|
||||
for value in object_ids
|
||||
):
|
||||
raise DependencyError("UPSTREAM_COMMIT and UPSTREAM_LOCK.json do not agree on full hexadecimal object IDs")
|
||||
if lock["expected_source"] != {"git_tree": lock["commit_tree"]}:
|
||||
raise DependencyError("expected_source must record the locked git tree")
|
||||
retrieval = lock["retrieval"]
|
||||
if retrieval != {
|
||||
"method": "git-clone-detached-commit",
|
||||
"workspace": "build/llama.cpp",
|
||||
}:
|
||||
raise DependencyError("retrieval must use the locked detached-commit build workspace")
|
||||
return lock
|
||||
|
||||
|
||||
@@ -103,6 +117,56 @@ def _git(source: pathlib.Path, *args: str) -> str:
|
||||
return _run("git", "-C", str(source), *args)
|
||||
|
||||
|
||||
def _verify_tracked_content(source: pathlib.Path, lock: dict[str, Any]) -> None:
|
||||
if _git(source, "write-tree") != lock["commit_tree"]:
|
||||
raise DependencyError("materialized checkout index differs from the locked tree")
|
||||
filemode_trusted = _git(source, "config", "--bool", "core.filemode") == "true"
|
||||
records = _git(source, "ls-files", "-s", "-z").split("\0")
|
||||
paths: list[str] = []
|
||||
expected: list[str] = []
|
||||
for record in records:
|
||||
if not record:
|
||||
continue
|
||||
metadata, path = record.split("\t", 1)
|
||||
mode, blob, stage = metadata.split()
|
||||
if stage != "0" or mode not in {"100644", "100755", "120000"}:
|
||||
raise DependencyError(f"unsupported tracked entry in materialized checkout: {record!r}")
|
||||
if "\n" in path:
|
||||
raise DependencyError(f"newline-bearing tracked path is unsupported: {path!r}")
|
||||
candidate = source / path
|
||||
cursor = source
|
||||
for part in pathlib.Path(path).parts[:-1]:
|
||||
cursor = cursor / part
|
||||
if cursor.is_symlink():
|
||||
raise DependencyError(f"tracked path traverses a symlink: {path!r}")
|
||||
if mode == "120000":
|
||||
if not candidate.is_symlink():
|
||||
raise DependencyError(f"tracked symlink type differs from the locked tree: {path!r}")
|
||||
else:
|
||||
if candidate.is_symlink() or not candidate.is_file():
|
||||
raise DependencyError(f"tracked file type differs from the locked tree: {path!r}")
|
||||
executable = bool(candidate.stat().st_mode & 0o111)
|
||||
if filemode_trusted and executable != (mode == "100755"):
|
||||
raise DependencyError(f"tracked executable mode differs from the locked tree: {path!r}")
|
||||
paths.append(path)
|
||||
expected.append(blob)
|
||||
try:
|
||||
completed = subprocess.run(
|
||||
["git", "hash-object", "--stdin-paths"],
|
||||
cwd=source,
|
||||
input="".join(f"{path}\n" for path in paths),
|
||||
check=True,
|
||||
text=True,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
)
|
||||
except subprocess.CalledProcessError as error:
|
||||
raise DependencyError(f"unable to hash tracked source content: {error.stderr.strip()}") from error
|
||||
actual = completed.stdout.splitlines()
|
||||
if len(actual) != len(expected) or actual != expected:
|
||||
raise DependencyError("tracked source content differs from the locked tree")
|
||||
|
||||
|
||||
def _verify_source(source: pathlib.Path, lock: dict[str, Any], *, require_clean: bool) -> None:
|
||||
if not (source / ".git").exists():
|
||||
raise DependencyError(f"not a materialized git checkout: {source}")
|
||||
@@ -110,8 +174,15 @@ def _verify_source(source: pathlib.Path, lock: dict[str, Any], *, require_clean:
|
||||
raise DependencyError("upstream drift: checkout HEAD does not equal the locked commit")
|
||||
if _git(source, "rev-parse", "HEAD^{tree}") != lock["commit_tree"]:
|
||||
raise DependencyError("upstream drift: checkout tree does not equal the locked tree")
|
||||
if require_clean and _git(source, "status", "--porcelain"):
|
||||
raise DependencyError("local edits detected in materialized llama.cpp checkout")
|
||||
if _git(source, "rev-parse", "--abbrev-ref", "HEAD") != "HEAD":
|
||||
raise DependencyError("materialized llama.cpp checkout must have a detached HEAD")
|
||||
if require_clean:
|
||||
_verify_tracked_content(source, lock)
|
||||
tracked = _git(source, "status", "--porcelain", "--untracked-files=no")
|
||||
untracked = _git(source, "ls-files", "--others", "--exclude-standard")
|
||||
ignored = _git(source, "ls-files", "--others", "--ignored", "--exclude-standard")
|
||||
if tracked or untracked or ignored:
|
||||
raise DependencyError("local edits or unmanifested files detected in materialized llama.cpp checkout")
|
||||
for relative, expected in lock["required_upstream_blobs"].items():
|
||||
actual = _git(source, "rev-parse", f"HEAD:{relative}")
|
||||
if actual != expected:
|
||||
@@ -122,15 +193,43 @@ def _verify_source(source: pathlib.Path, lock: dict[str, Any], *, require_clean:
|
||||
raise DependencyError("upstream LICENSE is missing; refusing to drop required attribution")
|
||||
|
||||
|
||||
def materialize(source: pathlib.Path, repository: str) -> None:
|
||||
def _workspace_source(workspace: pathlib.Path, lock: dict[str, Any]) -> pathlib.Path:
|
||||
relative = pathlib.Path(lock["retrieval"]["workspace"])
|
||||
expected = (ROOT / relative).absolute()
|
||||
supplied = workspace.absolute()
|
||||
if supplied != expected:
|
||||
raise DependencyError(f"--workspace must equal the locked ignored build root: {expected}")
|
||||
cursor = ROOT
|
||||
for part in relative.parts:
|
||||
cursor = cursor / part
|
||||
if cursor.is_symlink():
|
||||
raise DependencyError(f"locked build workspace may not traverse a symlink: {cursor}")
|
||||
resolved = workspace.resolve()
|
||||
try:
|
||||
resolved.relative_to(ROOT.resolve())
|
||||
except ValueError as error:
|
||||
raise DependencyError("locked build workspace escapes the repository root") from error
|
||||
source = resolved / "source"
|
||||
if source.is_symlink():
|
||||
raise DependencyError(f"locked source checkout may not be a symlink: {source}")
|
||||
return source
|
||||
|
||||
|
||||
def fetch(workspace: pathlib.Path) -> pathlib.Path:
|
||||
"""Fetch once, or verify an exact clean cached checkout for offline reuse."""
|
||||
lock = _load_lock()
|
||||
_patches(lock)
|
||||
source = _workspace_source(workspace, lock)
|
||||
if source.exists():
|
||||
raise DependencyError(f"destination already exists; refusing to reuse possibly edited source: {source}")
|
||||
_verify_source(source, lock, require_clean=True)
|
||||
print(f"reused verified offline cache: {source}")
|
||||
return source
|
||||
source.parent.mkdir(parents=True, exist_ok=True)
|
||||
_run("git", "clone", "--no-checkout", repository, str(source))
|
||||
_run("git", "clone", "--no-checkout", lock["upstream"], str(source))
|
||||
_git(source, "checkout", "--detach", lock["commit"])
|
||||
_verify_source(source, lock, require_clean=True)
|
||||
print(f"fetched and verified exact source: {source}")
|
||||
return source
|
||||
|
||||
|
||||
def apply(source: pathlib.Path) -> None:
|
||||
@@ -198,18 +297,11 @@ def smoke(binary: pathlib.Path) -> None:
|
||||
print(output)
|
||||
|
||||
|
||||
def reproduce(work_dir: pathlib.Path, repository: str) -> None:
|
||||
resolved = work_dir.resolve()
|
||||
build_root = (ROOT / "build").resolve()
|
||||
if build_root not in resolved.parents:
|
||||
raise DependencyError(f"--work-dir must be below {build_root}: {resolved}")
|
||||
if resolved.exists():
|
||||
raise DependencyError(
|
||||
f"work directory already exists; refusing to erase possible local edits: {resolved}"
|
||||
)
|
||||
source = resolved / "source"
|
||||
build_dir = resolved / "build"
|
||||
materialize(source, repository)
|
||||
def reproduce(workspace: pathlib.Path) -> None:
|
||||
source = fetch(workspace)
|
||||
build_dir = workspace.resolve() / "build"
|
||||
if build_dir.exists():
|
||||
raise DependencyError(f"build directory already exists; refusing to erase possible local edits: {build_dir}")
|
||||
apply(source)
|
||||
smoke(build(source, build_dir))
|
||||
|
||||
@@ -219,6 +311,9 @@ def inspect() -> None:
|
||||
patches = _patches(lock)
|
||||
print(json.dumps({
|
||||
"commit": lock["commit"],
|
||||
"commit_tree": lock["commit_tree"],
|
||||
"retrieval": lock["retrieval"],
|
||||
"upstream_license": lock["upstream_license"],
|
||||
"patch_count": len(patches),
|
||||
"patches": [patch.name for patch in patches],
|
||||
"model_downloads": False,
|
||||
@@ -231,9 +326,8 @@ def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
subcommands = parser.add_subparsers(dest="command", required=True)
|
||||
subcommands.add_parser("inspect")
|
||||
materialize_parser = subcommands.add_parser("materialize")
|
||||
materialize_parser.add_argument("--source-dir", type=pathlib.Path, required=True)
|
||||
materialize_parser.add_argument("--source-repository", default=_load_lock()["upstream"])
|
||||
fetch_parser = subcommands.add_parser("fetch")
|
||||
fetch_parser.add_argument("--workspace", type=pathlib.Path, default=ROOT / "build/llama.cpp")
|
||||
apply_parser = subcommands.add_parser("apply")
|
||||
apply_parser.add_argument("--source-dir", type=pathlib.Path, required=True)
|
||||
build_parser = subcommands.add_parser("build")
|
||||
@@ -242,14 +336,13 @@ def main() -> int:
|
||||
smoke_parser = subcommands.add_parser("smoke")
|
||||
smoke_parser.add_argument("--binary", type=pathlib.Path, required=True)
|
||||
reproduce_parser = subcommands.add_parser("reproduce")
|
||||
reproduce_parser.add_argument("--work-dir", type=pathlib.Path, default=ROOT / "build/dgr-004-smoke")
|
||||
reproduce_parser.add_argument("--source-repository", default=_load_lock()["upstream"])
|
||||
reproduce_parser.add_argument("--workspace", type=pathlib.Path, default=ROOT / "build/llama.cpp")
|
||||
args = parser.parse_args()
|
||||
try:
|
||||
if args.command == "inspect":
|
||||
inspect()
|
||||
elif args.command == "materialize":
|
||||
materialize(args.source_dir, args.source_repository)
|
||||
elif args.command == "fetch":
|
||||
fetch(args.workspace)
|
||||
elif args.command == "apply":
|
||||
apply(args.source_dir)
|
||||
elif args.command == "build":
|
||||
@@ -257,9 +350,9 @@ def main() -> int:
|
||||
elif args.command == "smoke":
|
||||
smoke(args.binary)
|
||||
else:
|
||||
reproduce(args.work_dir, args.source_repository)
|
||||
reproduce(args.workspace)
|
||||
except DependencyError as error:
|
||||
print(f"DGR-004 dependency error: {error}", file=sys.stderr)
|
||||
print(f"DGR-027 dependency error: {error}", file=sys.stderr)
|
||||
return 2
|
||||
return 0
|
||||
|
||||
|
||||
@@ -8,9 +8,9 @@
|
||||
"model_artifact_digest": "8a0f43d6aa49d77834bdb47bcae9f42c886b7ccfe0ac014932b2a2b38697a47b",
|
||||
"recipe_id": "example-gguf",
|
||||
"recipe_version": "1",
|
||||
"runtime_recipe_digest": "9b14d70b0835a6428457e4888d453649dd0d2e41fc8ac9d84d232c8c237e68fa"
|
||||
"runtime_recipe_digest": "63001e0efeada5b97f2f3562562dc0fd2d9bd8904dc6b7b99a71d98f3e938bd0"
|
||||
},
|
||||
"fingerprint_proto_hex": "0a40386130663433643661613439643737383334626462343762636165396634326338383662376363666530616330313439333262326132623338363937613437621240396231346437306230383335613634323834353765343838386434353336343964643064326534316663386163396438346432333263386332333765363866611a0c6578616d706c652d676775662201312a09323032362e30372e31",
|
||||
"fingerprint_proto_hex": "0a40386130663433643661613439643737383334626462343762636165396634326338383662376363666530616330313439333262326132623338363937613437621240363330303165306566656164613562393766326633353632353632646330666432643962643839303464633662376239396137316439386633653933386264301a0c6578616d706c652d676775662201312a09323032362e30372e31",
|
||||
"identity": {
|
||||
"artifact": {
|
||||
"architecture": "dense-llama",
|
||||
@@ -26,7 +26,7 @@
|
||||
"model_artifact_digest": "8a0f43d6aa49d77834bdb47bcae9f42c886b7ccfe0ac014932b2a2b38697a47b",
|
||||
"recipe_id": "example-gguf",
|
||||
"recipe_version": "1",
|
||||
"runtime_recipe_digest": "9b14d70b0835a6428457e4888d453649dd0d2e41fc8ac9d84d232c8c237e68fa"
|
||||
"runtime_recipe_digest": "63001e0efeada5b97f2f3562562dc0fd2d9bd8904dc6b7b99a71d98f3e938bd0"
|
||||
},
|
||||
"recipe": {
|
||||
"activation_dtype": "bfloat16",
|
||||
@@ -40,7 +40,7 @@
|
||||
"protocol_schema_version": 1,
|
||||
"recipe_id": "example-gguf",
|
||||
"recipe_version": "1",
|
||||
"runtime_version": "llama.cpp@deadbeef+meshnet.1",
|
||||
"runtime_version": "llama.cpp@dddddddddddddddddddddddddddddddddddddddd+patchstack.eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee",
|
||||
"tokenizer_revision": "0123456789abcdef",
|
||||
"weight_quantization": "Q4_K_M"
|
||||
},
|
||||
@@ -58,9 +58,9 @@
|
||||
"model_artifact_digest": "8a0f43d6aa49d77834bdb47bcae9f42c886b7ccfe0ac014932b2a2b38697a47b",
|
||||
"recipe_id": "example-gguf",
|
||||
"recipe_version": "1",
|
||||
"runtime_recipe_digest": "9b14d70b0835a6428457e4888d453649dd0d2e41fc8ac9d84d232c8c237e68fa"
|
||||
"runtime_recipe_digest": "63001e0efeada5b97f2f3562562dc0fd2d9bd8904dc6b7b99a71d98f3e938bd0"
|
||||
},
|
||||
"fingerprint_proto_hex": "0a40386130663433643661613439643737383334626462343762636165396634326338383662376363666530616330313439333262326132623338363937613437621240396231346437306230383335613634323834353765343838386434353336343964643064326534316663386163396438346432333263386332333765363866611a0c6578616d706c652d676775662201312a09323032362e30372e31",
|
||||
"fingerprint_proto_hex": "0a40386130663433643661613439643737383334626462343762636165396634326338383662376363666530616330313439333262326132623338363937613437621240363330303165306566656164613562393766326633353632353632646330666432643962643839303464633662376239396137316439386633653933386264301a0c6578616d706c652d676775662201312a09323032362e30372e31",
|
||||
"identity": {
|
||||
"artifact": {
|
||||
"architecture": "dense-llama",
|
||||
@@ -80,7 +80,7 @@
|
||||
"model_artifact_digest": "8a0f43d6aa49d77834bdb47bcae9f42c886b7ccfe0ac014932b2a2b38697a47b",
|
||||
"recipe_id": "example-gguf",
|
||||
"recipe_version": "1",
|
||||
"runtime_recipe_digest": "9b14d70b0835a6428457e4888d453649dd0d2e41fc8ac9d84d232c8c237e68fa"
|
||||
"runtime_recipe_digest": "63001e0efeada5b97f2f3562562dc0fd2d9bd8904dc6b7b99a71d98f3e938bd0"
|
||||
},
|
||||
"recipe": {
|
||||
"activation_dtype": "bfloat16",
|
||||
@@ -94,7 +94,7 @@
|
||||
"protocol_schema_version": 1,
|
||||
"recipe_id": "example-gguf",
|
||||
"recipe_version": "1",
|
||||
"runtime_version": "llama.cpp@deadbeef+meshnet.1",
|
||||
"runtime_version": "llama.cpp@dddddddddddddddddddddddddddddddddddddddd+patchstack.eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee",
|
||||
"tokenizer_revision": "0123456789abcdef",
|
||||
"weight_quantization": "Q4_K_M"
|
||||
},
|
||||
|
||||
@@ -1,8 +1,9 @@
|
||||
"""Offline guards for DGR-004's pinned llama.cpp dependency boundary."""
|
||||
"""Offline guards for DGR-027's pinned llama.cpp dependency boundary."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import importlib.util
|
||||
import json
|
||||
import pathlib
|
||||
import subprocess
|
||||
@@ -33,6 +34,10 @@ def test_lock_and_patch_manifest_are_self_consistent_and_exact() -> None:
|
||||
|
||||
assert commit == lock["commit"]
|
||||
assert len(commit) == 40
|
||||
assert lock["retrieval"]["method"] == "git-clone-detached-commit"
|
||||
assert lock["retrieval"]["workspace"] == "build/llama.cpp"
|
||||
assert lock["expected_source"]["git_tree"] == lock["commit_tree"]
|
||||
assert lock["upstream_license"] == "MIT"
|
||||
assert patches == lock["patch_series"]
|
||||
assert patches == sorted(patches)
|
||||
assert patches
|
||||
@@ -42,6 +47,188 @@ def test_lock_and_patch_manifest_are_self_consistent_and_exact() -> None:
|
||||
assert "Subject: [PATCH" in patch.read_text()
|
||||
|
||||
|
||||
def test_fetch_refuses_a_workspace_outside_the_ignored_build_root(tmp_path: pathlib.Path) -> None:
|
||||
completed = subprocess.run(
|
||||
[sys.executable, str(SCRIPT), "fetch", "--workspace", str(tmp_path / "llama.cpp")],
|
||||
cwd=ROOT,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
|
||||
assert completed.returncode == 2
|
||||
assert "--workspace must equal" in completed.stderr
|
||||
|
||||
|
||||
def test_workspace_refuses_a_symlinked_build_ancestor(tmp_path: pathlib.Path, monkeypatch) -> None:
|
||||
spec = importlib.util.spec_from_file_location("llama_cpp_dependency_symlink", SCRIPT)
|
||||
assert spec and spec.loader
|
||||
dependency = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(dependency)
|
||||
root = tmp_path / "repo"
|
||||
outside = tmp_path / "outside"
|
||||
root.mkdir()
|
||||
outside.mkdir()
|
||||
(root / "build").symlink_to(outside, target_is_directory=True)
|
||||
monkeypatch.setattr(dependency, "ROOT", root)
|
||||
|
||||
try:
|
||||
dependency._workspace_source(
|
||||
root / "build/llama.cpp",
|
||||
{"retrieval": {"workspace": "build/llama.cpp"}},
|
||||
)
|
||||
except dependency.DependencyError as error:
|
||||
assert "may not traverse a symlink" in str(error)
|
||||
else:
|
||||
raise AssertionError("symlinked build ancestor must be refused")
|
||||
|
||||
(root / "build").unlink()
|
||||
workspace = root / "build/llama.cpp"
|
||||
workspace.mkdir(parents=True)
|
||||
(workspace / "source").symlink_to(outside, target_is_directory=True)
|
||||
try:
|
||||
dependency._workspace_source(
|
||||
workspace,
|
||||
{"retrieval": {"workspace": "build/llama.cpp"}},
|
||||
)
|
||||
except dependency.DependencyError as error:
|
||||
assert "source checkout may not be a symlink" in str(error)
|
||||
else:
|
||||
raise AssertionError("symlinked source checkout must be refused")
|
||||
|
||||
|
||||
def test_fetch_refuses_a_branch_or_repository_override() -> None:
|
||||
completed = subprocess.run(
|
||||
[sys.executable, str(SCRIPT), "fetch", "--source-repository", "main"],
|
||||
cwd=ROOT,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
|
||||
assert completed.returncode == 2
|
||||
assert "unrecognized arguments" in completed.stderr
|
||||
|
||||
|
||||
def test_fetch_reuses_only_a_verified_cached_tree_offline(
|
||||
tmp_path: pathlib.Path, monkeypatch
|
||||
) -> None:
|
||||
spec = importlib.util.spec_from_file_location("llama_cpp_dependency", SCRIPT)
|
||||
assert spec and spec.loader
|
||||
dependency = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(dependency)
|
||||
|
||||
root = tmp_path / "repo"
|
||||
source = root / "build/llama.cpp/source"
|
||||
upstream = tmp_path / "upstream"
|
||||
upstream.mkdir()
|
||||
subprocess.run(["git", "init", "-q", str(upstream)], check=True)
|
||||
subprocess.run(["git", "-C", str(upstream), "config", "user.email", "test@example.invalid"], check=True)
|
||||
subprocess.run(["git", "-C", str(upstream), "config", "user.name", "test"], check=True)
|
||||
(upstream / "CMakeLists.txt").write_text("cmake_minimum_required(VERSION 3.14)\n")
|
||||
(upstream / "LICENSE").write_text("MIT\n")
|
||||
(upstream / "tool.sh").write_text("#!/bin/sh\nexit 0\n")
|
||||
(upstream / "tool.sh").chmod(0o755)
|
||||
subprocess.run(["git", "-C", str(upstream), "add", "."], check=True)
|
||||
subprocess.run(["git", "-C", str(upstream), "commit", "-qm", "fixture"], check=True)
|
||||
commit = subprocess.run(
|
||||
["git", "-C", str(upstream), "rev-parse", "HEAD"], check=True, capture_output=True, text=True
|
||||
).stdout.strip()
|
||||
tree = subprocess.run(
|
||||
["git", "-C", str(upstream), "rev-parse", "HEAD^{tree}"], check=True, capture_output=True, text=True
|
||||
).stdout.strip()
|
||||
blob = subprocess.run(
|
||||
["git", "-C", str(upstream), "rev-parse", "HEAD:CMakeLists.txt"],
|
||||
check=True,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
).stdout.strip()
|
||||
source.parent.mkdir(parents=True)
|
||||
subprocess.run(["git", "clone", "-q", str(upstream), str(source)], check=True)
|
||||
|
||||
llama_dir = root / "packages/node/native/llama"
|
||||
patch_dir = llama_dir / "patches"
|
||||
patch_dir.mkdir(parents=True)
|
||||
(llama_dir / "UPSTREAM_COMMIT").write_text(f"{commit}\n")
|
||||
(patch_dir / "series").write_text("0001-fixture.patch\n")
|
||||
patch = patch_dir / "0001-fixture.patch"
|
||||
patch.write_text("fixture patch\n")
|
||||
(patch_dir / "SHA256SUMS").write_text(f"{_sha256(patch)} {patch.name}\n")
|
||||
(llama_dir / "UPSTREAM_LOCK.json").write_text(json.dumps({
|
||||
"upstream": str(upstream),
|
||||
"commit": commit,
|
||||
"commit_tree": tree,
|
||||
"expected_source": {"git_tree": tree},
|
||||
"retrieval": {"method": "git-clone-detached-commit", "workspace": "build/llama.cpp"},
|
||||
"patched_tree": tree,
|
||||
"patch_series": [patch.name],
|
||||
"required_upstream_blobs": {"CMakeLists.txt": blob},
|
||||
"patched_paths": [],
|
||||
"build": {},
|
||||
"upstream_license": "MIT",
|
||||
}))
|
||||
monkeypatch.setattr(dependency, "ROOT", root)
|
||||
monkeypatch.setattr(dependency, "LLAMA_DIR", llama_dir)
|
||||
monkeypatch.setattr(dependency, "LOCK_PATH", llama_dir / "UPSTREAM_LOCK.json")
|
||||
monkeypatch.setattr(dependency, "PATCH_DIR", patch_dir)
|
||||
|
||||
try:
|
||||
dependency.fetch(root / "build/llama.cpp")
|
||||
except dependency.DependencyError as error:
|
||||
assert "detached HEAD" in str(error)
|
||||
else:
|
||||
raise AssertionError("attached branch cache must be refused")
|
||||
subprocess.run(["git", "-C", str(source), "checkout", "--detach", "-q", commit], check=True)
|
||||
assert dependency.fetch(root / "build/llama.cpp") == source
|
||||
|
||||
tracked_path = source / "CMakeLists.txt"
|
||||
for flag, clear_flag in (
|
||||
("--assume-unchanged", "--no-assume-unchanged"),
|
||||
("--skip-worktree", "--no-skip-worktree"),
|
||||
):
|
||||
subprocess.run(["git", "-C", str(source), "update-index", flag, "CMakeLists.txt"], check=True)
|
||||
tracked_path.write_text("injected tracked build input\n")
|
||||
try:
|
||||
dependency.fetch(root / "build/llama.cpp")
|
||||
except dependency.DependencyError as error:
|
||||
assert "tracked source content differs" in str(error)
|
||||
else:
|
||||
raise AssertionError(f"tracked cache injection hidden by {flag} must be refused")
|
||||
subprocess.run(["git", "-C", str(source), "update-index", clear_flag, "CMakeLists.txt"], check=True)
|
||||
subprocess.run(["git", "-C", str(source), "checkout", "--", "CMakeLists.txt"], check=True)
|
||||
|
||||
executable_path = source / "tool.sh"
|
||||
for flag, clear_flag in (
|
||||
("--assume-unchanged", "--no-assume-unchanged"),
|
||||
("--skip-worktree", "--no-skip-worktree"),
|
||||
):
|
||||
subprocess.run(["git", "-C", str(source), "update-index", flag, "tool.sh"], check=True)
|
||||
executable_path.chmod(0o644)
|
||||
try:
|
||||
dependency.fetch(root / "build/llama.cpp")
|
||||
except dependency.DependencyError as error:
|
||||
assert "executable mode differs" in str(error)
|
||||
else:
|
||||
raise AssertionError(f"tracked mode change hidden by {flag} must be refused")
|
||||
subprocess.run(["git", "-C", str(source), "update-index", clear_flag, "tool.sh"], check=True)
|
||||
subprocess.run(["git", "-C", str(source), "checkout", "--", "tool.sh"], check=True)
|
||||
|
||||
(source / "untracked.txt").write_text("edited\n")
|
||||
try:
|
||||
dependency.fetch(root / "build/llama.cpp")
|
||||
except dependency.DependencyError as error:
|
||||
assert "local edits" in str(error)
|
||||
else:
|
||||
raise AssertionError("dirty cached source must be refused")
|
||||
(source / "untracked.txt").unlink()
|
||||
(source / ".git/info/exclude").write_text("injected.cmake\n")
|
||||
(source / "injected.cmake").write_text("unmanifested input\n")
|
||||
try:
|
||||
dependency.fetch(root / "build/llama.cpp")
|
||||
except dependency.DependencyError as error:
|
||||
assert "unmanifested files" in str(error)
|
||||
else:
|
||||
raise AssertionError("ignored cached source input must be refused")
|
||||
|
||||
|
||||
def test_dependency_script_reports_the_locked_boundary_without_network() -> None:
|
||||
completed = subprocess.run(
|
||||
[sys.executable, str(SCRIPT), "inspect"],
|
||||
|
||||
@@ -15,6 +15,7 @@ from meshnet_node.native_backend import (
|
||||
shard_identity_from_native_report,
|
||||
)
|
||||
from meshnet_node.native_protocol import SCHEMA_VERSION, pb
|
||||
from meshnet_node.runtime_pin import load_runtime_pin
|
||||
from meshnet_node.recipe_manifest import parse_recipe_manifest
|
||||
from meshnet_tracker.capability import STATE_UNCERTIFIED, evaluate_report
|
||||
|
||||
@@ -42,7 +43,6 @@ def _inputs(**changes: object) -> NativeIdentityInputs:
|
||||
kv_layout="llama-kv-v1",
|
||||
architecture_adapter="dense-llama-v1",
|
||||
backend_id="llama-cpp",
|
||||
runtime_version="llama.cpp:e920c523",
|
||||
recipe_id="native",
|
||||
recipe_version="1",
|
||||
catalogue_version="2026.07.1",
|
||||
@@ -84,6 +84,7 @@ def test_native_identity_uses_loaded_report_not_a_caller_range():
|
||||
assert (identity.shard_start, identity.shard_end) == (2, 6)
|
||||
assert identity.artifact.architecture == "llama"
|
||||
assert identity.artifact.layer_count == 8
|
||||
assert identity.recipe.runtime_version == load_runtime_pin().runtime_version
|
||||
|
||||
|
||||
def test_native_identity_requires_an_immutable_pin_and_gguf_range():
|
||||
|
||||
@@ -612,6 +612,27 @@ def test_a_peer_still_sending_the_retired_field_does_not_corrupt_the_tensor():
|
||||
assert decode_tensor(pb.NamedTensor.FromString(wire)) == b"\xaa" * 32
|
||||
|
||||
|
||||
def test_native_toolchain_bootstrap_resolves_relative_prefix_before_temp_chdir(tmp_path):
|
||||
script = REPO_ROOT / "scripts/bootstrap_native_toolchain.sh"
|
||||
result = subprocess.run(
|
||||
["bash", str(script), "--print-prefix", "relative/toolchain"],
|
||||
cwd=tmp_path,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
|
||||
assert result.returncode == 0, result.stdout + result.stderr
|
||||
assert pathlib.Path(result.stdout.strip()) == tmp_path / "relative/toolchain"
|
||||
|
||||
|
||||
def test_python_generator_rejects_the_wrong_grpcio_tools_version(monkeypatch):
|
||||
from scripts import generate_native_protocol
|
||||
|
||||
monkeypatch.setattr(generate_native_protocol.metadata, "version", lambda _: "0.0.0")
|
||||
with pytest.raises(SystemExit, match="found 0.0.0, require 1.82.1"):
|
||||
generate_native_protocol._require_grpcio_tools_version()
|
||||
|
||||
|
||||
def test_generated_python_stubs_match_the_proto():
|
||||
pytest.importorskip("grpc_tools", reason="protoc toolchain is required to verify")
|
||||
result = subprocess.run(
|
||||
|
||||
312
tests/test_runtime_pin_identity.py
Normal file
312
tests/test_runtime_pin_identity.py
Normal file
@@ -0,0 +1,312 @@
|
||||
"""DGR-025: the recipe's runtime axis commits to the exact pin and patch stack.
|
||||
|
||||
The fingerprint (DGR-003 lineage) digests a ``runtime_version`` string, but a
|
||||
string an operator typed is a label, not a pin: two workers could run different
|
||||
patch stacks under the same label and still agree on the digest. These tests
|
||||
pin the axis to the DGR-027 lock manifest — the exact upstream commit plus a
|
||||
digest over the ordered patch-stack bytes — and prove both identity
|
||||
implementations reject a moving runtime reference.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from meshnet_node.runtime_pin import (
|
||||
DEFAULT_LOCK_DIR,
|
||||
RuntimePinError,
|
||||
load_runtime_pin,
|
||||
)
|
||||
from meshnet_node.runtime_recipe import RecipeIdentityError, RuntimeRecipe
|
||||
from meshnet_tracker.recipe import (
|
||||
RecipeIdentityError as TrackerRecipeIdentityError,
|
||||
parse_identity,
|
||||
)
|
||||
|
||||
REPO_LOCK_DIR = (
|
||||
Path(__file__).resolve().parent.parent / "packages" / "node" / "native" / "llama"
|
||||
)
|
||||
|
||||
|
||||
def _recipe(**changes: object) -> RuntimeRecipe:
|
||||
fields: dict[str, object] = {
|
||||
"weight_quantization": "Q4_K_M",
|
||||
"activation_dtype": "bfloat16",
|
||||
"compute_dtype": "float32",
|
||||
"kv_dtype": "q8_0",
|
||||
"kv_layout": "paged-v1",
|
||||
"tokenizer_revision": "0123456789abcdef",
|
||||
"architecture_adapter": "llama/range-v1",
|
||||
"backend_id": "llama.cpp",
|
||||
"runtime_version": "llama.cpp@" + "d" * 40 + "+patchstack." + "e" * 64,
|
||||
"recipe_id": "example-gguf",
|
||||
"recipe_version": "1",
|
||||
"catalogue_version": "2026.07.1",
|
||||
}
|
||||
fields.update(changes)
|
||||
return RuntimeRecipe(**fields) # type: ignore[arg-type]
|
||||
|
||||
|
||||
def _write_fixture_workspace(
|
||||
root: Path,
|
||||
*,
|
||||
commit: str = "e" * 40,
|
||||
patches: dict[str, bytes] | None = None,
|
||||
lock_series: list[str] | None = None,
|
||||
series_lines: list[str] | None = None,
|
||||
sums_lines: list[str] | None = None,
|
||||
schema_version: int = 1,
|
||||
upstream: str = "https://github.com/ggml-org/llama.cpp.git",
|
||||
upstream_commit_file: str | None = None,
|
||||
) -> Path:
|
||||
"""A minimal DGR-027-shaped lock workspace; overrides create disagreement."""
|
||||
if patches is None:
|
||||
patches = {
|
||||
"0001-first.patch": b"--- a\n+++ b\n",
|
||||
"0002-second.patch": b"--- c\n+++ d\n",
|
||||
}
|
||||
names = list(patches)
|
||||
digests = {
|
||||
name: hashlib.sha256(body).hexdigest() for name, body in patches.items()
|
||||
}
|
||||
|
||||
patch_dir = root / "patches"
|
||||
patch_dir.mkdir(parents=True)
|
||||
for name, body in patches.items():
|
||||
(patch_dir / name).write_bytes(body)
|
||||
|
||||
(root / "UPSTREAM_LOCK.json").write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"schema_version": schema_version,
|
||||
"upstream": upstream,
|
||||
"commit": commit,
|
||||
"patch_series": names if lock_series is None else lock_series,
|
||||
}
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
(root / "UPSTREAM_COMMIT").write_text(
|
||||
(commit if upstream_commit_file is None else upstream_commit_file) + "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
(patch_dir / "series").write_text(
|
||||
"\n".join(names if series_lines is None else series_lines) + "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
if sums_lines is None:
|
||||
sums_lines = ["# ordered digests"] + [
|
||||
f"{digests[name]} {name}" for name in names
|
||||
]
|
||||
(patch_dir / "SHA256SUMS").write_text(
|
||||
"\n".join(sums_lines) + "\n", encoding="utf-8"
|
||||
)
|
||||
return root
|
||||
|
||||
|
||||
# --- the committed manifest is the identity source -------------------------
|
||||
|
||||
|
||||
def test_default_lock_dir_is_the_committed_manifest():
|
||||
assert DEFAULT_LOCK_DIR == REPO_LOCK_DIR
|
||||
|
||||
|
||||
def test_committed_manifest_derives_a_deterministic_runtime_pin():
|
||||
pin = load_runtime_pin(REPO_LOCK_DIR)
|
||||
again = load_runtime_pin(REPO_LOCK_DIR)
|
||||
assert pin == again
|
||||
lock = json.loads(
|
||||
(REPO_LOCK_DIR / "UPSTREAM_LOCK.json").read_text(encoding="utf-8")
|
||||
)
|
||||
assert pin.upstream_commit == lock["commit"]
|
||||
assert list(pin.patch_series) == lock["patch_series"]
|
||||
# The axis value names the runtime, the exact commit, and the stack digest,
|
||||
# so changing any of the three changes every downstream recipe digest.
|
||||
assert pin.runtime_version == (
|
||||
f"llama.cpp@{lock['commit']}+patchstack.{pin.patch_stack_digest}"
|
||||
)
|
||||
assert len(pin.patch_stack_digest) == 64
|
||||
|
||||
|
||||
def test_derived_axis_value_is_a_valid_recipe_pin():
|
||||
pin = load_runtime_pin(REPO_LOCK_DIR)
|
||||
recipe = _recipe(runtime_version=pin.runtime_version)
|
||||
assert recipe.runtime_version == pin.runtime_version
|
||||
assert len(recipe.runtime_recipe_digest) == 64
|
||||
|
||||
|
||||
def test_patch_byte_change_changes_the_runtime_identity(tmp_path):
|
||||
baseline = load_runtime_pin(_write_fixture_workspace(tmp_path / "a"))
|
||||
changed = load_runtime_pin(
|
||||
_write_fixture_workspace(
|
||||
tmp_path / "b",
|
||||
patches={
|
||||
"0001-first.patch": b"--- a\n+++ b\n",
|
||||
"0002-second.patch": b"--- c\n+++ DIFFERENT\n",
|
||||
},
|
||||
)
|
||||
)
|
||||
assert baseline.patch_stack_digest != changed.patch_stack_digest
|
||||
assert baseline.runtime_version != changed.runtime_version
|
||||
|
||||
|
||||
def test_patch_order_is_part_of_the_stack_identity(tmp_path):
|
||||
patches = {
|
||||
"0001-first.patch": b"--- a\n+++ b\n",
|
||||
"0002-second.patch": b"--- c\n+++ d\n",
|
||||
}
|
||||
forward = load_runtime_pin(
|
||||
_write_fixture_workspace(tmp_path / "a", patches=patches)
|
||||
)
|
||||
names = list(patches)
|
||||
reversed_names = list(reversed(names))
|
||||
digests = {
|
||||
name: hashlib.sha256(body).hexdigest() for name, body in patches.items()
|
||||
}
|
||||
swapped = load_runtime_pin(
|
||||
_write_fixture_workspace(
|
||||
tmp_path / "b",
|
||||
patches=patches,
|
||||
lock_series=reversed_names,
|
||||
series_lines=reversed_names,
|
||||
sums_lines=[f"{digests[name]} {name}" for name in reversed_names],
|
||||
)
|
||||
)
|
||||
assert forward.patch_stack_digest != swapped.patch_stack_digest
|
||||
|
||||
|
||||
# --- every manifest disagreement fails closed ------------------------------
|
||||
|
||||
|
||||
def test_missing_lock_file_fails_closed(tmp_path):
|
||||
with pytest.raises(RuntimePinError, match="UPSTREAM_LOCK.json"):
|
||||
load_runtime_pin(tmp_path)
|
||||
|
||||
|
||||
def test_malformed_lock_json_fails_closed(tmp_path):
|
||||
_write_fixture_workspace(tmp_path)
|
||||
(tmp_path / "UPSTREAM_LOCK.json").write_text("{not json", encoding="utf-8")
|
||||
with pytest.raises(RuntimePinError, match="not valid JSON"):
|
||||
load_runtime_pin(tmp_path)
|
||||
|
||||
|
||||
def test_unknown_lock_schema_fails_closed(tmp_path):
|
||||
_write_fixture_workspace(tmp_path, schema_version=2)
|
||||
with pytest.raises(RuntimePinError, match="schema"):
|
||||
load_runtime_pin(tmp_path)
|
||||
|
||||
|
||||
def test_moving_commit_reference_fails_closed(tmp_path):
|
||||
_write_fixture_workspace(tmp_path, commit="master")
|
||||
with pytest.raises(RuntimePinError, match="exact 40"):
|
||||
load_runtime_pin(tmp_path)
|
||||
|
||||
|
||||
def test_upstream_commit_file_disagreement_fails_closed(tmp_path):
|
||||
_write_fixture_workspace(tmp_path, upstream_commit_file="f" * 40)
|
||||
with pytest.raises(RuntimePinError, match="UPSTREAM_COMMIT"):
|
||||
load_runtime_pin(tmp_path)
|
||||
|
||||
|
||||
def test_series_file_disagreement_fails_closed(tmp_path):
|
||||
_write_fixture_workspace(
|
||||
tmp_path, series_lines=["0002-second.patch", "0001-first.patch"]
|
||||
)
|
||||
with pytest.raises(RuntimePinError, match="series"):
|
||||
load_runtime_pin(tmp_path)
|
||||
|
||||
|
||||
def test_missing_patch_file_fails_closed(tmp_path):
|
||||
_write_fixture_workspace(tmp_path)
|
||||
(tmp_path / "patches" / "0002-second.patch").unlink()
|
||||
with pytest.raises(RuntimePinError, match="0002-second.patch"):
|
||||
load_runtime_pin(tmp_path)
|
||||
|
||||
|
||||
def test_checksum_disagreement_fails_closed(tmp_path):
|
||||
_write_fixture_workspace(tmp_path)
|
||||
patch = tmp_path / "patches" / "0002-second.patch"
|
||||
patch.write_bytes(patch.read_bytes() + b"tampered\n")
|
||||
with pytest.raises(RuntimePinError, match="SHA256SUMS"):
|
||||
load_runtime_pin(tmp_path)
|
||||
|
||||
|
||||
def test_sums_entry_missing_fails_closed(tmp_path):
|
||||
patches = {
|
||||
"0001-first.patch": b"--- a\n+++ b\n",
|
||||
"0002-second.patch": b"--- c\n+++ d\n",
|
||||
}
|
||||
digest = hashlib.sha256(patches["0001-first.patch"]).hexdigest()
|
||||
_write_fixture_workspace(
|
||||
tmp_path,
|
||||
patches=patches,
|
||||
sums_lines=[f"{digest} 0001-first.patch"],
|
||||
)
|
||||
with pytest.raises(RuntimePinError, match="SHA256SUMS"):
|
||||
load_runtime_pin(tmp_path)
|
||||
|
||||
|
||||
def test_empty_patch_series_requires_empty_series_files(tmp_path):
|
||||
# An unpatched runtime is a legal pin; a lock that *hides* patches is not.
|
||||
_write_fixture_workspace(tmp_path, lock_series=[])
|
||||
with pytest.raises(RuntimePinError, match="series"):
|
||||
load_runtime_pin(tmp_path)
|
||||
|
||||
|
||||
# --- both identity implementations reject a moving runtime -----------------
|
||||
|
||||
|
||||
def test_node_recipe_rejects_a_moving_runtime_version():
|
||||
with pytest.raises(RecipeIdentityError, match="moving reference"):
|
||||
_recipe(runtime_version="latest")
|
||||
|
||||
|
||||
def test_tracker_rejects_a_moving_runtime_version():
|
||||
vectors = json.loads(
|
||||
(Path(__file__).parent / "data" / "recipe_fingerprint_vectors.json").read_text(
|
||||
encoding="utf-8"
|
||||
)
|
||||
)
|
||||
doc = json.loads(json.dumps(vectors["vectors"][0]["identity"]))
|
||||
doc["recipe"]["runtime_version"] = "latest"
|
||||
doc.pop("fingerprint", None)
|
||||
with pytest.raises(TrackerRecipeIdentityError, match="moving reference"):
|
||||
parse_identity(doc)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"forged",
|
||||
[
|
||||
"llama.cpp@master+patchstack.not-a-digest",
|
||||
"llama.cpp@e920c523+patchstack.forged",
|
||||
"release-that-operator-typed",
|
||||
],
|
||||
)
|
||||
def test_node_recipe_rejects_noncanonical_llama_runtime_pins(forged):
|
||||
with pytest.raises(RecipeIdentityError, match="40-hex commit"):
|
||||
_recipe(runtime_version=forged)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"forged",
|
||||
[
|
||||
"llama.cpp@master+patchstack.not-a-digest",
|
||||
"llama.cpp@e920c523+patchstack.forged",
|
||||
"release-that-operator-typed",
|
||||
],
|
||||
)
|
||||
def test_tracker_rejects_noncanonical_llama_runtime_pins(forged):
|
||||
vectors = json.loads(
|
||||
(Path(__file__).parent / "data" / "recipe_fingerprint_vectors.json").read_text(
|
||||
encoding="utf-8"
|
||||
)
|
||||
)
|
||||
doc = json.loads(json.dumps(vectors["vectors"][0]["identity"]))
|
||||
doc["recipe"]["runtime_version"] = forged
|
||||
doc.pop("fingerprint", None)
|
||||
with pytest.raises(TrackerRecipeIdentityError, match="40-hex commit"):
|
||||
parse_identity(doc)
|
||||
@@ -65,7 +65,7 @@ def _recipe(**changes: object) -> RuntimeRecipe:
|
||||
"tokenizer_revision": "0123456789abcdef",
|
||||
"architecture_adapter": "llama/range-v1",
|
||||
"backend_id": "llama.cpp",
|
||||
"runtime_version": "llama.cpp@deadbeef+meshnet.1",
|
||||
"runtime_version": "llama.cpp@" + "d" * 40 + "+patchstack." + "e" * 64,
|
||||
"recipe_id": "example-gguf",
|
||||
"recipe_version": "1",
|
||||
"catalogue_version": "2026.07.1",
|
||||
@@ -241,7 +241,7 @@ def test_committed_vectors_cover_a_whole_model_and_a_derivative_shard():
|
||||
("tokenizer_revision", "fedcba9876543210"),
|
||||
("architecture_adapter", "llama/range-v2"),
|
||||
("backend_id", "other-backend"),
|
||||
("runtime_version", "llama.cpp@other+meshnet.1"),
|
||||
("runtime_version", "llama.cpp@" + "c" * 40 + "+patchstack." + "b" * 64),
|
||||
("boundary_schema_version", 2),
|
||||
("protocol_schema_version", 2),
|
||||
],
|
||||
|
||||
@@ -1,71 +0,0 @@
|
||||
"""Tests for tracker-side layer-aware SafeTensors file selection.
|
||||
|
||||
The tracker advertises its local snapshot as a downloadable "model source" for
|
||||
whatever layer range a node needs. If a required weight file for that range is
|
||||
missing from the tracker's own disk, the tracker must refuse to advertise
|
||||
itself as a source for that range — not quietly report the subset of files it
|
||||
does have as if that were the complete, correct set. A partial-but-"complete"
|
||||
answer makes the requesting node believe the download is already satisfied,
|
||||
so it never fetches the real weights and only discovers the gap much later
|
||||
at model-load time.
|
||||
"""
|
||||
|
||||
import json
|
||||
|
||||
from meshnet_tracker.model_files import select_safetensors_files_for_layers
|
||||
|
||||
|
||||
def _write_index(tmp_path, *, config=None):
|
||||
(tmp_path / "config.json").write_text(
|
||||
json.dumps(config or {"num_hidden_layers": 5}),
|
||||
encoding="utf-8",
|
||||
)
|
||||
(tmp_path / "model.safetensors.index.json").write_text(
|
||||
json.dumps({
|
||||
"weight_map": {
|
||||
"model.embed_tokens.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.self_attn.q_proj.weight": "model-00004-of-00004.safetensors",
|
||||
"model.norm.weight": "model-00004-of-00004.safetensors",
|
||||
"lm_head.weight": "model-00004-of-00004.safetensors",
|
||||
},
|
||||
}),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
|
||||
def _touch(path, size=1):
|
||||
path.write_bytes(b"0" * size)
|
||||
|
||||
|
||||
def test_selects_files_when_snapshot_is_complete(tmp_path):
|
||||
"Selects files when snapshot is complete\n\nTags: general"
|
||||
_write_index(tmp_path)
|
||||
_touch(tmp_path / "model-00001-of-00004.safetensors")
|
||||
|
||||
files = select_safetensors_files_for_layers(tmp_path, 0, 0)
|
||||
|
||||
assert files == ["config.json", "model-00001-of-00004.safetensors", "model.safetensors.index.json"]
|
||||
|
||||
|
||||
def test_returns_empty_when_a_required_weight_file_is_missing_on_disk(tmp_path):
|
||||
"Returns empty when a required weight file is missing on disk\n\nTags: general"
|
||||
_write_index(tmp_path)
|
||||
# model-00001-of-00004.safetensors is required for layer 0 but was never
|
||||
# downloaded onto this tracker host — the snapshot is incomplete.
|
||||
|
||||
files = select_safetensors_files_for_layers(tmp_path, 0, 0)
|
||||
|
||||
assert files == []
|
||||
|
||||
|
||||
def test_tail_range_returns_empty_when_only_head_shard_present(tmp_path):
|
||||
"Tail range returns empty when only head shard present\n\nTags: general"
|
||||
_write_index(tmp_path)
|
||||
_touch(tmp_path / "model-00001-of-00004.safetensors")
|
||||
# model-00004-of-00004.safetensors (norm/lm_head) is required for the tail
|
||||
# range but missing — must not be silently dropped from the result.
|
||||
|
||||
files = select_safetensors_files_for_layers(tmp_path, 4, 4)
|
||||
|
||||
assert files == []
|
||||
Reference in New Issue
Block a user