neuron-tai/.scratch/distributed-gguf-runtime/evidence/DGR-025/README.md

# DGR-025 evidence — exact artifact and runtime recipe identity

**Status:** in progress — controller gates pass; final independent P0/P1 re-review is pending.
**Branch:** fixed detached Claude Fable provider lane
**Authority:** live Gitea #9; the local PRD is a secondary projection.
**Dependencies:** DGR-018 (`evidence/DGR-018/README.md` — canonical backlog schema and
issue projection), DGR-021 (`evidence/DGR-021/README.md` — versioned activation
envelope). Both read before changing code.

## Objective

Ensure the tracker and worker only combine numerically and operationally
compatible shards: fingerprint every axis that moves the numbers, bind shards to
exact half-open ranges, fail closed on any mismatch, and keep uncertified
recipes registered-but-dark.

## What was found live (verified, not inherited)

Per RALPH-CONTEXT, legacy pass states were not trusted. The DGR-003-lineage
identity core was inspected and exercised live before any change:

- `packages/node/meshnet_node/runtime_recipe.py` — node-side identity:
  domain-separated digests (`meshnet.model-artifact.v1`,
  `meshnet.runtime-recipe.v1`, `meshnet.shard-binding.v1`) over the source
  artifact SHA (`source_digest`, with split artifacts bound to their exact
  source via `DerivativeBinding`), tokenizer revision (pin-enforced),
  architecture adapter + architecture/config digest, boundary and protocol
  schema versions, backend, weight quantization, activation/compute dtypes, and
  KV dtype/layout (`RECIPE_AXES`). Shard ranges are half-open
  (`shard_start`/`shard_end`, end-exclusive, protocol convention) with no
  topology or quant constants anywhere; `check_route` accepts any tiling of
  `[0, layer_count)`. Route, handshake (`check_handshake`), and session-open
  (`check_session_open`) checks fail closed with structured `RouteMismatch`
  reasons mapped to specific protocol error codes (`handshake_error`).
- `packages/tracker/meshnet_tracker/recipe.py` — deliberately independent
  tracker re-derivation (no `meshnet_node` import); declared fingerprints are
  recomputed, never trusted (`parse_identity`, `FingerprintMismatch`). The
  `CertificationLedger` keeps every registered recipe dark until a real
  distributed forward — at least 2 distinct nodes, whole-model coverage,
  non-synthetic, tokens actually generated — certifies it; dark recipes may
  route only to certify.
- The two implementations are pinned by committed conformance vectors
  (`tests/data/recipe_fingerprint_vectors.json`).

Live verification of that pre-existing core before changes:
`PYTHONPATH=packages/node:packages/tracker python3 -m pytest -q tests/test_runtime_recipe_identity.py`
→ `45 passed`; plus `tests/test_native_identity_emission.py`,
`tests/test_tracker_capability_admission.py`, `tests/test_node_admission.py`
→ `59 passed`.

## Gap found and closed (this story's change)

**The `runtime_version` recipe axis was a label, not a pin.** It was an opaque
caller-supplied string: nothing derived it from the DGR-027 lock manifest, and
neither identity implementation rejected a moving reference (`"latest"` was
accepted), so two workers could run different llama.cpp pins or patch stacks
under one label and still agree on the recipe digest. The acceptance criterion
explicitly requires fingerprinting the "runtime pin/patch stack".

### Changed files

- `packages/node/meshnet_node/runtime_pin.py` (new) — derives the canonical
  `runtime_version` axis value from the DGR-027 lock workspace
  (`packages/node/native/llama`):
  `<runtime>@<40-hex upstream commit>+patchstack.<sha256>` where the stack
  digest commits, under the `meshnet.runtime-patch-stack.v1` domain, to the
  *ordered* `(patch name, patch bytes sha256)` stack. Fails closed on: missing
  or malformed `UPSTREAM_LOCK.json`, unknown schema version, non-40-hex/moving
  commit, `UPSTREAM_COMMIT` disagreement, any disagreement among the lock's
  `patch_series`, `patches/series`, and `patches/SHA256SUMS`, a missing patch
  file, or a patch whose bytes don't match their recorded digest. Reads the
  committed manifest only; fetching/patching stays with
  `scripts/llama_cpp_dependency.py` (DGR-027).
- `packages/node/meshnet_node/runtime_recipe.py` — `runtime_version` is now
  pin-enforced (`_require_pin`) exactly like `tokenizer_revision`; for the
  llama.cpp backend it must also match the canonical
  `llama.cpp@<40-hex>+patchstack.<64-hex>` grammar.
- `packages/node/meshnet_node/native_backend.py` — the production native
  identity seam no longer accepts a caller-supplied runtime string. It derives
  `runtime_version` directly through `load_runtime_pin()` from the committed
  lock and rejects a non-llama backend at this llama.cpp-specific boundary.
- `packages/tracker/meshnet_tracker/recipe.py` — the independent tracker
  implementation applies the same backend-specific grammar before re-deriving
  the recipe digest, so forged operator labels cannot register or certify.
- `tests/test_runtime_pin_identity.py` and
  `tests/test_native_identity_emission.py` — deterministic tests cover lock
  derivation, production native emission, and node/tracker rejection of the
  forged values from independent review. Conformance vectors were regenerated
  through `scripts/gen_recipe_fingerprint_vectors.py` for the tightened wire
  contract.

### Backlog-consistency repair (pre-existing damage, honestly recorded)

`tests/test_ralph_prd_schema.py` had 4 pre-existing failures before this story
touched anything, left by prior sessions and the alternate-history merge:

- DGR-022 and DGR-027 were marked `passes: true` without `completionNotes` and
  without regenerated issue projections. Added their `completionNotes`
  (explicitly labeled as added during this repair, content drawn from their own
  evidence READMEs) and regenerated
  `issues/022-…` / `issues/027-…` via `scripts/ralph_prd_schema.py render`.
- Three pre-DGR legacy GLM alpha issue files (`18-…`, `19-…`, `20-…`,
  committed 2026-07-14, before DGR-018 established the generated-only
  convention; they carry no authority disclaimer because they are *not*
  generated from prd.json) were relocated via `git mv` to
  `issues/legacy/` — preserved as provenance, out of the generated namespace.

### prd.json

Marked `DGR-025.passes = true` with `completionNotes`; regenerated
`issues/025-define-exact-artifact-and-runtime-recipe-identity.md`.

## Acceptance criteria → evidence

1. **Fingerprint all axes** — `RECIPE_AXES` + `ArtifactIdentity` cover source
   artifact SHA, tokenizer revision, architecture adapter/version (adapter axis
   + architecture/config digest), boundary schema (boundary + protocol schema
   versions), backend, quant, activation/compute dtype, KV/state layout; the
   runtime pin/patch stack is now committed via the derived `runtime_version`
   axis (`runtime_pin.py`). Verified by `test_runtime_recipe_identity.py` and
   `test_runtime_pin_identity.py`.
2. **Exact half-open range, no hardcoded topology/quant** — `ShardIdentity`
   end-exclusive ranges, `DerivativeBinding` coverage checks, `check_route`
   tiling over arbitrary layouts; quant/dtype values are open strings
   (dynamic recipe inputs). Verified by `test_runtime_recipe_identity.py`
   (routes of 1, 2, and 5 shards; no product constants).
3. **Fail closed on any mismatch** — artifact, adapter, boundary/schema, cache
   layout, backend, and runtime mismatches each produce structured
   `RouteMismatch` reasons and protocol error codes; the tracker recomputes
   digests and rejects inconsistent claims; moving runtime references are now
   rejected on both sides.
4. **Registered-but-dark** — `CertificationLedger`: unknown recipes cannot be
   certified, registered recipes are dark, only a real ≥2-distinct-node
   whole-model non-synthetic forward promotes; verified by
   `test_runtime_recipe_identity.py` / `test_tracker_capability_admission.py`.
5. **Gates + this handoff** — below.

## Commands and results

```bash
PYTHONPATH=packages/node:packages/tracker python3 -m pytest -q tests/test_runtime_pin_identity.py
```
```text
23 passed in 0.15s
```

```bash
PYTHONPATH=packages/node:packages/tracker python3 -m pytest -q \
  tests/test_runtime_pin_identity.py tests/test_runtime_recipe_identity.py \
  tests/test_native_identity_emission.py tests/test_tracker_capability_admission.py \
  tests/test_node_admission.py tests/test_node_capability.py tests/test_recipe_benchmark.py
```
```text
202 passed, 1 warning in 5.38s
```

```bash
PYTHONPATH=packages/node:packages/tracker python3 -m pytest -q tests/test_ralph_prd_schema.py
```
```text
108 passed
```
(4 failed before this story's backlog repair; 0 after.)

```bash
python3 -m compileall -q packages tests   # exit 0
git diff --check                          # exit 0
python3 scripts/ralph_prd_schema.py validate .scratch/distributed-gguf-runtime/prd.json
# OK: 55 stories validated.
```

Default tests are model-download-free, API-credit-free, and GPU-free; no model
artifact was touched and nothing was written under `/home`.

## Limitations

- The production native identity seam now derives the manifest pin and cannot
  accept an operator-supplied runtime label. It still cannot attest that the
  running binary was built from those locked bytes. Embedding the patched-tree
  hash at build time and echoing it through the DGR-022 status contract belongs
  with DGR-028+/DGR-031; real distributed certification remains the final trust
  boundary.
- The DGR-027-recorded blocker stands: `0002-dense-llama-owned-range-loader.patch`
  does not apply cleanly against the pin (DGR-028). That does not affect this
  story: the identity commits to the patch *bytes as committed*, which is
  precisely what makes a later repaired patch a *different* runtime identity.
- No native/CMake change was made, so the native build/CTest gate is not
  applicable; no llama.cpp patch content was changed, so apply/check/reverse
  verification is not applicable (and is blocked by the DGR-028 defect anyway).
- Tracker routing, load balancing, billing, telemetry, and relay semantics are
  untouched; the only behavior change outside the new module is the stricter
  (fail-closed) rejection of moving `runtime_version` values.

## Dependency handoff

- **DGR-026** (split-GGUF provisioning): bind each provisioned split via
  `DerivativeBinding` to the exact source digest recorded in its hashed
  manifest; the per-split `shard_binding_digest` is what certification pins.
- **DGR-031** (`ShardEngine`): construct worker identity through
  `shard_identity_from_native_report` and populate `runtime_version` from
  `meshnet_node.runtime_pin.load_runtime_pin().runtime_version` — never from an
  operator string. A build-time echo of the patched-tree hash through the
  status contract would close the manifest-vs-binary gap noted above.
- **DGR-041** (capability registration): the tracker already re-derives and
  fail-closes on presented identities (`parse_identity`); register recipes
  through the `CertificationLedger` so they arrive dark.
- **DGR-044** (DeepSeek V4 Flash target): pin the target's artifact identity
  the same way `glm_alpha_artifact` does — read locked manifests, never restate
  digests — and note `layer_count` must count the routed transformer stack the
  route tiles, excluding MTP (reserved for beta).

## Reopened P1 repair — 2026-07-18

The earlier evidence above is provenance only. Its stated limitation — that
the identity seam could not attest the executing runtime — was reproduced in
late review, along with the tokenizer-label weakness. This repair replaces
both claims at the production identity boundary.

### Changed files

- `packages/node/meshnet_node/runtime_recipe.py` — replaces the moving-ref
  denylist with the sole valid `tokenizer.v1:<sha256>` form, derived from an
  ordered map of named tokenizer/config byte digests. A label, tag, branch, or
  symbolic ref cannot be a valid identity.
- `packages/tracker/meshnet_tracker/recipe.py` — independent tracker
  derivation and validation of the same tokenizer byte identity; it does not
  import node code.
- `packages/node/meshnet_node/runtime_pin.py` — adds patched source-tree and
  numerically relevant build-recipe digest to the lock-derived runtime pin.
- `packages/node/meshnet_node/native_backend.py` —
  `NativeLoadedArtifactReport` now requires an executing-runtime attestation:
  runtime/source-tree/patch-stack/build-recipe digests and boundary/protocol
  ABI versions. `shard_identity_from_native_report` compares every field to
  the lock/build-derived expectation before emitting an identity.
- `scripts/gen_recipe_fingerprint_vectors.py` and
  `tests/data/recipe_fingerprint_vectors.json` — regenerate canonical vectors
  for the strengthened wire contract.
- `tests/test_runtime_pin_identity.py`,
  `tests/test_runtime_recipe_identity.py`, and
  `tests/test_native_identity_emission.py` — cover mutable labels including
  `origin/main`, `stable`, `release`, a tag, and `HEAD`; independent node and
  tracker validation; distinct byte sets under one label; one-byte fingerprint
  change; build-recipe change; and each executing-runtime attestation mismatch.

### Verification

```bash
PYTHONPATH=packages/node:packages/tracker python3 scripts/gen_recipe_fingerprint_vectors.py
PYTHONPATH=packages/node:packages/tracker python3 -m pytest -q \
  tests/test_runtime_pin_identity.py tests/test_native_identity_emission.py \
  tests/test_runtime_recipe_identity.py
```
```text
92 passed
```

```bash
PYTHONPATH=packages/node:packages/tracker python3 -m pytest -q \
  tests/test_runtime_pin_identity.py tests/test_runtime_recipe_identity.py \
  tests/test_native_identity_emission.py tests/test_node_admission.py \
  tests/test_node_capability.py tests/test_recipe_benchmark.py
```
```text
188 passed, 1 pre-existing pytest thread warning
```

`python3 scripts/ralph_prd_schema.py validate
.scratch/distributed-gguf-runtime/prd.json`, `python3 -m compileall -q packages
tests`, and `git diff --check` each exit 0. The broader PRD pytest projection
suite has two unrelated existing DGR-023 failures: its `passes: true` entry has
no completion notes and its generated issue file is stale. The socket-backed
subset of `test_tracker_capability_admission.py` is additionally un-runnable in
this sandbox (`PermissionError: [Errno 1] Operation not permitted` creating an
AF_INET socket); its deterministic non-socket identity coverage is included in
the passing runs above.

### Remaining boundary (superseded 2026-07-18, same day — see below)

The attestation at this point was a native runtime report *contract*: a
Python dataclass the worker was trusted to populate. Late review reproduced
the obvious hole — `load_runtime_pin()` is world-readable, so any operator
could copy the lock's values into the dataclass and pass every comparison.
The section below closes that hole.

## Executing-artifact evidence binding — 2026-07-18 (this repair)

The executing native runtime's identity must not be forgeable by copying
repository lock values into a Python self-report. Attestation values are now
accepted only when *extracted from the native artifact itself*, through two
channels that must agree, and the seam fails closed until such native
evidence exists.

### The boundary

`meshnet_node.native_backend` now defines the attestation extraction
contract:

- **Static channel** — the artifact's bytes must embed exactly one
  NUL-terminated `MESHNET-RUNTIME-ATTESTATION.v1:<canonical json>` marker.
  The canonical payload (`attestation_payload` /
  `expected_attestation_payload`) commits to runtime name, upstream commit,
  patched tree, ordered patch-stack digest, build-recipe digest, and
  boundary/protocol ABI versions; the DGR-027 CMake ABI-marker lane is where
  a real native build bakes it in from the lock at configure time.
- **Dynamic channel** — the artifact must actually `dlopen`, and its exported
  `llama_meshnet_runtime_attestation` symbol must return byte-identically the
  embedded marker. A marker pasted into a plain file is not an executing
  runtime.
- **Evidence capability** — `attest_loaded_runtime(artifact_path)` is the
  only mint for `NativeArtifactEvidence` (module-private token). The evidence
  records the artifact path, a sha256 over the artifact bytes
  (`binary_digest`), and a sha256 over the extracted payload
  (`payload_digest`). `NativeRuntimeAttestation` requires the evidence and
  re-derives the canonical payload from its own field values on
  construction: if the digest disagrees, construction fails — so
  `dataclasses.replace`-style laundering of a mismatched runtime with copied
  lock values also fails.
- `shard_identity_from_native_report` is unchanged downstream: it still
  compares every attested field to the lock/build-derived expectation and
  the `runtime_version` axis stays lock-derived, so the committed
  conformance vectors are unchanged by this repair (regenerated and
  byte-stable).

Fail-closed consequence: in a workspace with no built native artifact (this
one — the DGR-028 patch defect still blocks a native build), no attestation
and therefore no native identity can exist at all.

### Changed files

- `packages/node/meshnet_node/native_backend.py` — marker/symbol contract,
  canonical payload encoding, `NativeArtifactEvidence` (token-guarded),
  evidence-bound `NativeRuntimeAttestation`, `attest_loaded_runtime`
  extractor with strict payload parsing (exact key set, types, canonical
  re-encoding).
- `tests/test_native_identity_emission.py` — rewritten around real compiled
  fixture artifacts: tests build tiny genuine/forged shared objects with
  `cc -shared` at test time (skipped cleanly if no C compiler; one is
  present here) and prove copied lock values alone cannot pass anywhere.

### Behavior tests proving copied lock values cannot pass

- Bare `NativeRuntimeAttestation(**lock_values)` (the pre-repair forgery) is
  unconstructible; `evidence=None` and hand-authored/`object()`-token
  `NativeArtifactEvidence` each raise.
- The true marker bytes written into a plain file fail (`not a loadable`).
- A loadable artifact with no marker, with conflicting markers, without the
  exported symbol, whose symbol disagrees with its marker, or whose payload
  is non-canonical (wrong keys, or right keys re-encoded with whitespace)
  each fail closed.
- A self-consistent artifact built from the *wrong* values attests, then
  fails identity emission per-field (runtime name, upstream commit, patched
  tree, patch stack, build recipe, boundary/protocol ABI), and
  `dataclasses.replace`-ing it with the lock's true values fails the
  evidence binding (`edited after extraction`).
- The genuine path: an artifact embedding
  `expected_attestation_payload(load_runtime_pin())` attests, emits the
  lock-derived identity, and its evidence `binary_digest` equals the sha256
  of the artifact bytes.

### Verification (all in this worktree, 2026-07-18)

```bash
PYTHONPATH=packages/node:packages/tracker python3 -m pytest -q \
  tests/test_native_identity_emission.py tests/test_runtime_pin_identity.py \
  tests/test_runtime_recipe_identity.py
```
```text
104 passed
```

```bash
PYTHONPATH=packages/node:packages/tracker python3 -m pytest -q \
  tests/test_node_admission.py tests/test_node_capability.py \
  tests/test_recipe_benchmark.py
```
```text
96 passed, 1 pre-existing pytest thread warning
```

```bash
PYTHONPATH=packages/node:packages/tracker python3 -m pytest -q \
  tests/test_tracker_capability_admission.py
```
```text
34 passed   # socket-backed subset ran in this session's sandbox
```

`PYTHONPATH=packages/node:packages/tracker python3
scripts/gen_recipe_fingerprint_vectors.py` reproduces the committed vectors
byte-for-byte; `python3 -m compileall -q packages tests` and
`git diff --check` each exit 0.

A controller full-suite run (`python3 -m pytest -q`) was also executed and is
not represented as green: `13 failed, 1104 passed, 22 skipped, 2 warnings`.
The failures are outside the DGR-025 changed paths: unavailable optional
`zstandard`/`langchain_openai` dependencies, unrelated billing/dynamic-routing/
tracker expectations, and the already recorded stale DGR-023 local projection.
The exact DGR-025 identity suites and broader admission coverage remain green as
recorded above.

### Remaining boundary

What is now proven: no identity can be constructed, registered, admitted, or
certified without evidence extracted from an actual loadable native artifact
that both embeds and reports the attestation, and the extracted values cannot
be edited afterward. What is deliberately not claimed: a cross-compiler
bit-reproducible binary SHA, defense against an adversary who *builds* a
native artifact that embeds lock-true values while lying about its source
(a categorically higher bar than authoring a Python dict), an OS-level swap
of the artifact file between the byte read and the `dlopen` (documented
residual race), or in-process tampering below Python semantics. Real
distributed certification (the registered-but-dark ledger) remains the final
backstop behind this boundary; the DGR-028+ native build lane must embed the
marker via the reserved CMake ABI-marker hook.

## Executing-byte identity repair — 2026-07-18 controller follow-up

A later controller review rejected the preceding remaining-boundary claim as
insufficient for DGR-025: a separately built loadable shared object could copy
all public lock values into both marker channels and receive the same
`runtime_version` as a certified artifact. The repair now appends
`+artifact.<sha256>` to the llama.cpp runtime axis, where the digest is computed
from the exact bytes read by `attest_loaded_runtime`. Node and tracker parsers
independently require this suffix. Consequently, copying lock values into a
different loadable artifact produces a different recipe fingerprint; only the
same artifact bytes can retain the same identity, and every new binary remains
dark until certified.

`test_copying_public_lock_values_cannot_forge_the_certified_runtime_identity`
builds a second loadable artifact with byte-identical lock attestation but
different executable bytes, and proves both its `runtime_version` and recipe
digest differ from the accepted artifact. Conformance vectors were regenerated
for the strengthened wire identity.

Controller verification:

```text
python3 scripts/gen_recipe_fingerprint_vectors.py
python3 -m pytest -q tests/test_native_identity_emission.py \
  tests/test_runtime_pin_identity.py tests/test_runtime_recipe_identity.py
# 105 passed in 0.52s
python3 -m compileall -q packages/node/meshnet_node \
  packages/tracker/meshnet_tracker tests scripts/gen_recipe_fingerprint_vectors.py
# exit 0
git diff --check
# exit 0
```