neuron-tai/.scratch/distributed-gguf-runtime/evidence/DGR-025

DGR-025 evidence — exact artifact and runtime recipe identity

Status: in progress — controller gates pass; final independent P0/P1 re-review is pending. Branch: fixed detached Claude Fable provider lane Authority: live Gitea #9; the local PRD is a secondary projection. Dependencies: DGR-018 (evidence/DGR-018/README.md — canonical backlog schema and issue projection), DGR-021 (evidence/DGR-021/README.md — versioned activation envelope). Both read before changing code.

Objective

Ensure the tracker and worker only combine numerically and operationally compatible shards: fingerprint every axis that moves the numbers, bind shards to exact half-open ranges, fail closed on any mismatch, and keep uncertified recipes registered-but-dark.

What was found live (verified, not inherited)

Per RALPH-CONTEXT, legacy pass states were not trusted. The DGR-003-lineage identity core was inspected and exercised live before any change:

• packages/node/meshnet_node/runtime_recipe.py — node-side identity: domain-separated digests (meshnet.model-artifact.v1, meshnet.runtime-recipe.v1, meshnet.shard-binding.v1) over the source artifact SHA (source_digest, with split artifacts bound to their exact source via DerivativeBinding), tokenizer revision (pin-enforced), architecture adapter + architecture/config digest, boundary and protocol schema versions, backend, weight quantization, activation/compute dtypes, and KV dtype/layout (RECIPE_AXES). Shard ranges are half-open (shard_start/shard_end, end-exclusive, protocol convention) with no topology or quant constants anywhere; check_route accepts any tiling of [0, layer_count). Route, handshake (check_handshake), and session-open (check_session_open) checks fail closed with structured RouteMismatch reasons mapped to specific protocol error codes (handshake_error).
• packages/tracker/meshnet_tracker/recipe.py — deliberately independent tracker re-derivation (no meshnet_node import); declared fingerprints are recomputed, never trusted (parse_identity, FingerprintMismatch). The CertificationLedger keeps every registered recipe dark until a real distributed forward — at least 2 distinct nodes, whole-model coverage, non-synthetic, tokens actually generated — certifies it; dark recipes may route only to certify.
• The two implementations are pinned by committed conformance vectors (tests/data/recipe_fingerprint_vectors.json).

Live verification of that pre-existing core before changes: PYTHONPATH=packages/node:packages/tracker python3 -m pytest -q tests/test_runtime_recipe_identity.py → 45 passed; plus tests/test_native_identity_emission.py, tests/test_tracker_capability_admission.py, tests/test_node_admission.py → 59 passed.

Gap found and closed (this story's change)

The runtime_version recipe axis was a label, not a pin. It was an opaque caller-supplied string: nothing derived it from the DGR-027 lock manifest, and neither identity implementation rejected a moving reference ("latest" was accepted), so two workers could run different llama.cpp pins or patch stacks under one label and still agree on the recipe digest. The acceptance criterion explicitly requires fingerprinting the "runtime pin/patch stack".

Changed files

• packages/node/meshnet_node/runtime_pin.py (new) — derives the canonical runtime_version axis value from the DGR-027 lock workspace (packages/node/native/llama): <runtime>@<40-hex upstream commit>+patchstack.<sha256> where the stack digest commits, under the meshnet.runtime-patch-stack.v1 domain, to the ordered (patch name, patch bytes sha256) stack. Fails closed on: missing or malformed UPSTREAM_LOCK.json, unknown schema version, non-40-hex/moving commit, UPSTREAM_COMMIT disagreement, any disagreement among the lock's patch_series, patches/series, and patches/SHA256SUMS, a missing patch file, or a patch whose bytes don't match their recorded digest. Reads the committed manifest only; fetching/patching stays with scripts/llama_cpp_dependency.py (DGR-027).
• packages/node/meshnet_node/runtime_recipe.py — runtime_version is now pin-enforced (_require_pin) exactly like tokenizer_revision; for the llama.cpp backend it must also match the canonical llama.cpp@<40-hex>+patchstack.<64-hex> grammar.
• packages/node/meshnet_node/native_backend.py — the production native identity seam no longer accepts a caller-supplied runtime string. It derives runtime_version directly through load_runtime_pin() from the committed lock and rejects a non-llama backend at this llama.cpp-specific boundary.
• packages/tracker/meshnet_tracker/recipe.py — the independent tracker implementation applies the same backend-specific grammar before re-deriving the recipe digest, so forged operator labels cannot register or certify.
• tests/test_runtime_pin_identity.py and tests/test_native_identity_emission.py — deterministic tests cover lock derivation, production native emission, and node/tracker rejection of the forged values from independent review. Conformance vectors were regenerated through scripts/gen_recipe_fingerprint_vectors.py for the tightened wire contract.

Backlog-consistency repair (pre-existing damage, honestly recorded)

tests/test_ralph_prd_schema.py had 4 pre-existing failures before this story touched anything, left by prior sessions and the alternate-history merge:

• DGR-022 and DGR-027 were marked passes: true without completionNotes and without regenerated issue projections. Added their completionNotes (explicitly labeled as added during this repair, content drawn from their own evidence READMEs) and regenerated issues/022-… / issues/027-… via scripts/ralph_prd_schema.py render.
• Three pre-DGR legacy GLM alpha issue files (18-…, 19-…, 20-…, committed 2026-07-14, before DGR-018 established the generated-only convention; they carry no authority disclaimer because they are not generated from prd.json) were relocated via git mv to issues/legacy/ — preserved as provenance, out of the generated namespace.

prd.json

Marked DGR-025.passes = true with completionNotes; regenerated issues/025-define-exact-artifact-and-runtime-recipe-identity.md.

Acceptance criteria → evidence

1. Fingerprint all axes — RECIPE_AXES + ArtifactIdentity cover source artifact SHA, tokenizer revision, architecture adapter/version (adapter axis
   • architecture/config digest), boundary schema (boundary + protocol schema versions), backend, quant, activation/compute dtype, KV/state layout; the runtime pin/patch stack is now committed via the derived runtime_version axis (runtime_pin.py). Verified by test_runtime_recipe_identity.py and test_runtime_pin_identity.py.
2. Exact half-open range, no hardcoded topology/quant — ShardIdentity end-exclusive ranges, DerivativeBinding coverage checks, check_route tiling over arbitrary layouts; quant/dtype values are open strings (dynamic recipe inputs). Verified by test_runtime_recipe_identity.py (routes of 1, 2, and 5 shards; no product constants).
3. Fail closed on any mismatch — artifact, adapter, boundary/schema, cache layout, backend, and runtime mismatches each produce structured RouteMismatch reasons and protocol error codes; the tracker recomputes digests and rejects inconsistent claims; moving runtime references are now rejected on both sides.
4. Registered-but-dark — CertificationLedger: unknown recipes cannot be certified, registered recipes are dark, only a real ≥2-distinct-node whole-model non-synthetic forward promotes; verified by test_runtime_recipe_identity.py / test_tracker_capability_admission.py.
5. Gates + this handoff — below.

Commands and results

PYTHONPATH=packages/node:packages/tracker python3 -m pytest -q tests/test_runtime_pin_identity.py

23 passed in 0.15s

PYTHONPATH=packages/node:packages/tracker python3 -m pytest -q \
  tests/test_runtime_pin_identity.py tests/test_runtime_recipe_identity.py \
  tests/test_native_identity_emission.py tests/test_tracker_capability_admission.py \
  tests/test_node_admission.py tests/test_node_capability.py tests/test_recipe_benchmark.py

202 passed, 1 warning in 5.38s

PYTHONPATH=packages/node:packages/tracker python3 -m pytest -q tests/test_ralph_prd_schema.py

108 passed

(4 failed before this story's backlog repair; 0 after.)

python3 -m compileall -q packages tests   # exit 0
git diff --check                          # exit 0
python3 scripts/ralph_prd_schema.py validate .scratch/distributed-gguf-runtime/prd.json
# OK: 55 stories validated.

Default tests are model-download-free, API-credit-free, and GPU-free; no model artifact was touched and nothing was written under /home.

Limitations

• The production native identity seam now derives the manifest pin and cannot accept an operator-supplied runtime label. It still cannot attest that the running binary was built from those locked bytes. Embedding the patched-tree hash at build time and echoing it through the DGR-022 status contract belongs with DGR-028+/DGR-031; real distributed certification remains the final trust boundary.
• The DGR-027-recorded blocker stands: 0002-dense-llama-owned-range-loader.patch does not apply cleanly against the pin (DGR-028). That does not affect this story: the identity commits to the patch bytes as committed, which is precisely what makes a later repaired patch a different runtime identity.
• No native/CMake change was made, so the native build/CTest gate is not applicable; no llama.cpp patch content was changed, so apply/check/reverse verification is not applicable (and is blocked by the DGR-028 defect anyway).
• Tracker routing, load balancing, billing, telemetry, and relay semantics are untouched; the only behavior change outside the new module is the stricter (fail-closed) rejection of moving runtime_version values.

Dependency handoff

• DGR-026 (split-GGUF provisioning): bind each provisioned split via DerivativeBinding to the exact source digest recorded in its hashed manifest; the per-split shard_binding_digest is what certification pins.
• DGR-031 (ShardEngine): construct worker identity through shard_identity_from_native_report and populate runtime_version from meshnet_node.runtime_pin.load_runtime_pin().runtime_version — never from an operator string. A build-time echo of the patched-tree hash through the status contract would close the manifest-vs-binary gap noted above.
• DGR-041 (capability registration): the tracker already re-derives and fail-closes on presented identities (parse_identity); register recipes through the CertificationLedger so they arrive dark.
• DGR-044 (DeepSeek V4 Flash target): pin the target's artifact identity the same way glm_alpha_artifact does — read locked manifests, never restate digests — and note layer_count must count the routed transformer stack the route tiles, excluding MTP (reserved for beta).

Reopened P1 repair — 2026-07-18

The earlier evidence above is provenance only. Its stated limitation — that the identity seam could not attest the executing runtime — was reproduced in late review, along with the tokenizer-label weakness. This repair replaces both claims at the production identity boundary.

Changed files

• packages/node/meshnet_node/runtime_recipe.py — replaces the moving-ref denylist with the sole valid tokenizer.v1:<sha256> form, derived from an ordered map of named tokenizer/config byte digests. A label, tag, branch, or symbolic ref cannot be a valid identity.
• packages/tracker/meshnet_tracker/recipe.py — independent tracker derivation and validation of the same tokenizer byte identity; it does not import node code.
• packages/node/meshnet_node/runtime_pin.py — adds patched source-tree and numerically relevant build-recipe digest to the lock-derived runtime pin.
• packages/node/meshnet_node/native_backend.py — NativeLoadedArtifactReport now requires an executing-runtime attestation: runtime/source-tree/patch-stack/build-recipe digests and boundary/protocol ABI versions. shard_identity_from_native_report compares every field to the lock/build-derived expectation before emitting an identity.
• scripts/gen_recipe_fingerprint_vectors.py and tests/data/recipe_fingerprint_vectors.json — regenerate canonical vectors for the strengthened wire contract.
• tests/test_runtime_pin_identity.py, tests/test_runtime_recipe_identity.py, and tests/test_native_identity_emission.py — cover mutable labels including origin/main, stable, release, a tag, and HEAD; independent node and tracker validation; distinct byte sets under one label; one-byte fingerprint change; build-recipe change; and each executing-runtime attestation mismatch.

Verification

PYTHONPATH=packages/node:packages/tracker python3 scripts/gen_recipe_fingerprint_vectors.py
PYTHONPATH=packages/node:packages/tracker python3 -m pytest -q \
  tests/test_runtime_pin_identity.py tests/test_native_identity_emission.py \
  tests/test_runtime_recipe_identity.py

92 passed

PYTHONPATH=packages/node:packages/tracker python3 -m pytest -q \
  tests/test_runtime_pin_identity.py tests/test_runtime_recipe_identity.py \
  tests/test_native_identity_emission.py tests/test_node_admission.py \
  tests/test_node_capability.py tests/test_recipe_benchmark.py

188 passed, 1 pre-existing pytest thread warning

python3 scripts/ralph_prd_schema.py validate .scratch/distributed-gguf-runtime/prd.json, python3 -m compileall -q packages tests, and git diff --check each exit 0. The broader PRD pytest projection suite has two unrelated existing DGR-023 failures: its passes: true entry has no completion notes and its generated issue file is stale. The socket-backed subset of test_tracker_capability_admission.py is additionally un-runnable in this sandbox (PermissionError: [Errno 1] Operation not permitted creating an AF_INET socket); its deterministic non-socket identity coverage is included in the passing runs above.

Remaining boundary (superseded 2026-07-18, same day — see below)

The attestation at this point was a native runtime report contract: a Python dataclass the worker was trusted to populate. Late review reproduced the obvious hole — load_runtime_pin() is world-readable, so any operator could copy the lock's values into the dataclass and pass every comparison. The section below closes that hole.

Executing-artifact evidence binding — 2026-07-18 (this repair)

The executing native runtime's identity must not be forgeable by copying repository lock values into a Python self-report. Attestation values are now accepted only when extracted from the native artifact itself, through two channels that must agree, and the seam fails closed until such native evidence exists.

The boundary

meshnet_node.native_backend now defines the attestation extraction contract:

• Static channel — the artifact's bytes must embed exactly one NUL-terminated MESHNET-RUNTIME-ATTESTATION.v1:<canonical json> marker. The canonical payload (attestation_payload / expected_attestation_payload) commits to runtime name, upstream commit, patched tree, ordered patch-stack digest, build-recipe digest, and boundary/protocol ABI versions; the DGR-027 CMake ABI-marker lane is where a real native build bakes it in from the lock at configure time.
• Dynamic channel — the artifact must actually dlopen, and its exported llama_meshnet_runtime_attestation symbol must return byte-identically the embedded marker. A marker pasted into a plain file is not an executing runtime.
• Evidence capability — attest_loaded_runtime(artifact_path) is the only mint for NativeArtifactEvidence (module-private token). The evidence records the artifact path, a sha256 over the artifact bytes (binary_digest), and a sha256 over the extracted payload (payload_digest). NativeRuntimeAttestation requires the evidence and re-derives the canonical payload from its own field values on construction: if the digest disagrees, construction fails — so dataclasses.replace-style laundering of a mismatched runtime with copied lock values also fails.
• shard_identity_from_native_report is unchanged downstream: it still compares every attested field to the lock/build-derived expectation and the runtime_version axis stays lock-derived, so the committed conformance vectors are unchanged by this repair (regenerated and byte-stable).

Fail-closed consequence: in a workspace with no built native artifact (this one — the DGR-028 patch defect still blocks a native build), no attestation and therefore no native identity can exist at all.

Changed files

• packages/node/meshnet_node/native_backend.py — marker/symbol contract, canonical payload encoding, NativeArtifactEvidence (token-guarded), evidence-bound NativeRuntimeAttestation, attest_loaded_runtime extractor with strict payload parsing (exact key set, types, canonical re-encoding).
• tests/test_native_identity_emission.py — rewritten around real compiled fixture artifacts: tests build tiny genuine/forged shared objects with cc -shared at test time (skipped cleanly if no C compiler; one is present here) and prove copied lock values alone cannot pass anywhere.

Behavior tests proving copied lock values cannot pass

• Bare NativeRuntimeAttestation(**lock_values) (the pre-repair forgery) is unconstructible; evidence=None and hand-authored/object()-token NativeArtifactEvidence each raise.
• The true marker bytes written into a plain file fail (not a loadable).
• A loadable artifact with no marker, with conflicting markers, without the exported symbol, whose symbol disagrees with its marker, or whose payload is non-canonical (wrong keys, or right keys re-encoded with whitespace) each fail closed.
• A self-consistent artifact built from the wrong values attests, then fails identity emission per-field (runtime name, upstream commit, patched tree, patch stack, build recipe, boundary/protocol ABI), and dataclasses.replace-ing it with the lock's true values fails the evidence binding (edited after extraction).
• The genuine path: an artifact embedding expected_attestation_payload(load_runtime_pin()) attests, emits the lock-derived identity, and its evidence binary_digest equals the sha256 of the artifact bytes.

Verification (all in this worktree, 2026-07-18)

PYTHONPATH=packages/node:packages/tracker python3 -m pytest -q \
  tests/test_native_identity_emission.py tests/test_runtime_pin_identity.py \
  tests/test_runtime_recipe_identity.py

104 passed

PYTHONPATH=packages/node:packages/tracker python3 -m pytest -q \
  tests/test_node_admission.py tests/test_node_capability.py \
  tests/test_recipe_benchmark.py

96 passed, 1 pre-existing pytest thread warning

PYTHONPATH=packages/node:packages/tracker python3 -m pytest -q \
  tests/test_tracker_capability_admission.py

34 passed   # socket-backed subset ran in this session's sandbox

PYTHONPATH=packages/node:packages/tracker python3 scripts/gen_recipe_fingerprint_vectors.py reproduces the committed vectors byte-for-byte; python3 -m compileall -q packages tests and git diff --check each exit 0.

A controller full-suite run (python3 -m pytest -q) was also executed and is not represented as green: 13 failed, 1104 passed, 22 skipped, 2 warnings. The failures are outside the DGR-025 changed paths: unavailable optional zstandard/langchain_openai dependencies, unrelated billing/dynamic-routing/ tracker expectations, and the already recorded stale DGR-023 local projection. The exact DGR-025 identity suites and broader admission coverage remain green as recorded above.

Remaining boundary

What is now proven: no identity can be constructed, registered, admitted, or certified without evidence extracted from an actual loadable native artifact that both embeds and reports the attestation, and the extracted values cannot be edited afterward. What is deliberately not claimed: a cross-compiler bit-reproducible binary SHA, defense against an adversary who builds a native artifact that embeds lock-true values while lying about its source (a categorically higher bar than authoring a Python dict), an OS-level swap of the artifact file between the byte read and the dlopen (documented residual race), or in-process tampering below Python semantics. Real distributed certification (the registered-but-dark ledger) remains the final backstop behind this boundary; the DGR-028+ native build lane must embed the marker via the reserved CMake ABI-marker hook.

Executing-byte identity repair — 2026-07-18 controller follow-up

A later controller review rejected the preceding remaining-boundary claim as insufficient for DGR-025: a separately built loadable shared object could copy all public lock values into both marker channels and receive the same runtime_version as a certified artifact. The repair now appends +artifact.<sha256> to the llama.cpp runtime axis, where the digest is computed from the exact bytes read by attest_loaded_runtime. Node and tracker parsers independently require this suffix. Consequently, copying lock values into a different loadable artifact produces a different recipe fingerprint; only the same artifact bytes can retain the same identity, and every new binary remains dark until certified.

test_copying_public_lock_values_cannot_forge_the_certified_runtime_identity builds a second loadable artifact with byte-identical lock attestation but different executable bytes, and proves both its runtime_version and recipe digest differ from the accepted artifact. Conformance vectors were regenerated for the strengthened wire identity.

Controller verification:

python3 scripts/gen_recipe_fingerprint_vectors.py
python3 -m pytest -q tests/test_native_identity_emission.py \
  tests/test_runtime_pin_identity.py tests/test_runtime_recipe_identity.py
# 105 passed in 0.52s
python3 -m compileall -q packages/node/meshnet_node \
  packages/tracker/meshnet_tracker tests scripts/gen_recipe_fingerprint_vectors.py
# exit 0
git diff --check
# exit 0