Compare commits
9 Commits
e7c780a623
...
ralph/dgr-
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
31065c0e12 | ||
|
|
ec36290863 | ||
|
|
f844ae6567 | ||
|
|
252d131e7d | ||
|
|
3d8f93f4aa | ||
|
|
f9722e7b57 | ||
|
|
7b8e467c6b | ||
|
|
7364ed6731 | ||
|
|
ad2d17541c |
@@ -9,3 +9,4 @@
|
||||
- **Distributed relay performance** — relay `/rpc` requester sockets are persistent per Route Session and Activation Seam as of 2026-07-10; `request_id` remains unique per activation while `X-Meshnet-Session` remains stable for KV state. Next low-risk priorities: persistent direct/loopback HTTP, seam byte/latency telemetry, then trace-driven zstd tuning.
|
||||
- **Distributed GGUF direction** — benchmark-gated native runtime: compare controlled Transformers/safetensors and whole-model llama.cpp lanes before expensive work; ship only for measured speed or model-fit advantage. Public parallelism is contiguous Shards in an Inference Route; concurrency comes from per-node continuous batching across isolated Route Sessions, while tensor/expert collectives stay inside optional trusted composite providers. Native data plane uses versioned Protobuf over long-lived gRPC/HTTP2 seam streams, with existing relay carrying the same opaque frames when needed. llama.cpp/GGML remains the substrate behind a project-owned standalone worker and small pinned fork; vLLM is an optional complete managed provider and concept donor, not a fork. Nakshatra, `prima.cpp`, `llama-gguf`, LiGGUF and historical GPUStack are source/test donors only. Active plan: [README](../../.scratch/distributed-gguf-runtime/README.md), [architecture](../../.scratch/distributed-gguf-runtime/architecture.md), [PRD](../../.scratch/distributed-gguf-runtime/PRD.md), [Ralph backlog](../../.scratch/distributed-gguf-runtime/prd.json). Research: [landscape](../../docs/research/distributed-gguf-landscape.md), [GitHub follow-up](../../docs/research/distributed-gguf-github-followup.md), [vLLM](../../docs/research/vllm-distributed-gguf-assessment.md).
|
||||
- [DGR ROCm setup](dgr-rocm-setup.md) — version-matched TheRock SDK layout, relocated devel payload, verified `gfx1151` HIP llama.cpp build, and GPU-diagnostic boundary.
|
||||
- **DGR-004 llama.cpp boundary** — `packages/node/native/llama/` locks `e920c523e3b8a0163fe498af5bf90df35ff51d25`, with a one-patch CMake marker and fail-closed clean materialize/apply/build/smoke harness. This is infrastructure only; stock GLM dense fallback remains uncertified.
|
||||
|
||||
186
.scratch/distributed-gguf-runtime/evidence/DGR-003/README.md
Normal file
186
.scratch/distributed-gguf-runtime/evidence/DGR-003/README.md
Normal file
@@ -0,0 +1,186 @@
|
||||
# DGR-003 — exact Artifact and runtime recipe identity
|
||||
|
||||
Evidence class: deterministic offline/unit. No model payload, GPU, external API,
|
||||
network node, or API credit is required or claimed.
|
||||
|
||||
## Result — delayed-review repair, 2026-07-14
|
||||
|
||||
DGR-003 defines and tests an exact, model-agnostic compatibility identity and
|
||||
connects it to DGR-002's gRPC `Fingerprint` plus tracker parsing, admission,
|
||||
route partitioning, and certification. It is **not complete**: the existing
|
||||
production doctor/backend path still emits the legacy capability report without
|
||||
constructing a `ShardIdentity` from authoritative loaded artifact/runtime state.
|
||||
No exact recipe is therefore claimed live or routable from that path; supplied
|
||||
exact identities remain dark until tracker-owned certification.
|
||||
|
||||
A matching digest proves canonical consistency, **not node authenticity or real
|
||||
execution**. Tracker-owned certification of a fingerprint by a non-synthetic,
|
||||
complete, multi-node distributed forward is the execution trust boundary.
|
||||
|
||||
## Implementation
|
||||
|
||||
- `ArtifactIdentity` binds artifact ID/revision, exact content digest,
|
||||
architecture/config digest, layer count, and optional derivative binding.
|
||||
- `DerivativeBinding` binds a split artifact to the exact source artifact digest
|
||||
and its end-exclusive layer range. A Shard cannot advertise outside that range.
|
||||
- `RuntimeRecipe` keeps these canonical axes separate rather than hiding them in
|
||||
a backend label:
|
||||
- weight quantization;
|
||||
- activation and compute dtypes;
|
||||
- KV dtype and layout;
|
||||
- tokenizer revision;
|
||||
- architecture adapter;
|
||||
- backend and runtime version;
|
||||
- boundary and protocol schema versions;
|
||||
- recipe ID/version and catalogue version.
|
||||
- `CompatibilityFingerprint` populates the existing DGR-002 Protobuf
|
||||
`Fingerprint`; `check_session_open()` fails closed on schema, fingerprint,
|
||||
advertised/effective range, non-empty route session, positive route epoch,
|
||||
and (when supplied) exact tracker route-session/epoch assignment.
|
||||
- Node and tracker implementations independently canonicalize the declaration.
|
||||
This is intentional: the tracker must not trust a digest copied from a node,
|
||||
and future native/C++ workers also need an independent implementation. Their
|
||||
behavior is pinned by `tests/data/recipe_fingerprint_vectors.json`.
|
||||
- Tracker admission cross-checks the exact identity against the capability
|
||||
proof's model, range, recipe labels, backend, and weight quantization. Any
|
||||
disagreement fails closed.
|
||||
- `TrackerServer` owns the sole live certification ledger and passes it through
|
||||
direct and replicated registration paths. A known exact recipe is
|
||||
`uncertified` and dark for user traffic until the same exact fingerprint is
|
||||
certified. Restart fails closed; durable/cluster-wide certification events
|
||||
require the later real-forward control path and are not claimed here.
|
||||
- Certification evidence is bound to the promoted fingerprint, requires at
|
||||
least two distinct nodes, complete layer coverage, generated tokens, and
|
||||
`synthetic=false`. Unknown or mismatched fingerprints cannot be promoted.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `packages/node/meshnet_node/runtime_recipe.py`
|
||||
- `packages/tracker/meshnet_tracker/recipe.py`
|
||||
- `packages/tracker/meshnet_tracker/capability.py`
|
||||
- `packages/tracker/meshnet_tracker/server.py`
|
||||
- `tests/data/recipe_fingerprint_vectors.json`
|
||||
- `tests/test_runtime_recipe_identity.py`
|
||||
- this evidence directory, issue state, and DGR-003 PRD state
|
||||
|
||||
A late review of dependency DGR-017 also found and fixed two genuine contract
|
||||
continuity defects during delayed DGR-003 review: v1 now has an independently
|
||||
trusted digest and recursively immutable parsed state. Those changes and tests
|
||||
are recorded in DGR-017 evidence rather than claimed as DGR-003 functionality.
|
||||
|
||||
## Verification
|
||||
|
||||
Exact commands and outcomes are in `commands.txt`.
|
||||
|
||||
Observed final results:
|
||||
|
||||
- DGR-003 identity + node/tracker capability suites: **126 passed**.
|
||||
- DGR-017 focused dependency repair suite: **99 passed**.
|
||||
- Tracker routing suite: **93 passed**.
|
||||
- First delayed-review integrated run: **898 passed, 13 skipped, 1 failed** on
|
||||
the pre-existing tracker-cancellation race.
|
||||
- Final delayed-review integrated rerun: **899 passed, 13 skipped** in
|
||||
**253.64s**; Hermes controller acceptance rerun: **899 passed, 13 skipped**
|
||||
in **252.66s**.
|
||||
- `python -m compileall -q packages tests`: pass.
|
||||
- `git diff --check`: pass.
|
||||
- Ruff on the changed identity, capability, contract, and test modules: pass.
|
||||
- `server.py` has 8 pre-existing Ruff findings at both pushed baseline and the
|
||||
current tree; DGR-003 added no finding.
|
||||
|
||||
The first integrated full-suite run produced **871 passed, 13 skipped, 1 failed**
|
||||
on the known unrelated
|
||||
`test_tracker_dashboard_can_cancel_inflight_proxy` timing race. Its fixture
|
||||
completed after three seconds just before cancellation, so the cancel endpoint
|
||||
returned 404. In this delayed repair it again produced a 404 after the stream
|
||||
finished (first integrated run: **898 passed, 13 skipped, 1 failed**); three
|
||||
immediate isolated repeats passed before a fourth reproduced the same race.
|
||||
No cancellation-test code was changed. The final complete integrated rerun
|
||||
passed **899/899** tests.
|
||||
|
||||
## Limitations
|
||||
|
||||
- Certification state is process-local in this story. The same running tracker
|
||||
reuses it across registrations, but durable/cluster-wide certification-event
|
||||
persistence belongs with the later real distributed-forward control path.
|
||||
Restart or failover therefore returns exact recipes to the safe dark state;
|
||||
it never makes an unsupported recipe routable.
|
||||
- The node module has no certification ledger or admission policy; it holds only
|
||||
identity construction and handshake validation. The Tracker is the sole
|
||||
promotion authority.
|
||||
- **Completion blocker:** `doctor._validate_recipe()` calls
|
||||
`build_capability_report()` without `identity=`, because the legacy
|
||||
Transformers backend does not expose an immutable artifact-content pin and
|
||||
full runtime recipe axes authoritative enough to build one. Adding a guessed
|
||||
identity would weaken this contract. Production emission must be added with
|
||||
the authoritative native worker/backend loading seam; until then the issue and
|
||||
PRD deliberately remain incomplete.
|
||||
- This story proves identity and admission behavior with deterministic fixtures.
|
||||
It does not claim a real GLM forward or hardware certification.
|
||||
|
||||
## Compatibility
|
||||
|
||||
- Capability report identity is additive. Legacy reports without the new block
|
||||
retain ADR-0023's explicit compatibility-policy behavior.
|
||||
- Reports that opt into exact identity are held to it and fail closed on malformed,
|
||||
inconsistent, unknown, dark, or mismatched declarations.
|
||||
- No new wire identity was invented; DGR-002's `Fingerprint` remains the gRPC
|
||||
representation.
|
||||
|
||||
## Handoff
|
||||
|
||||
DGR-004 and native workers must build `ShardIdentity` from the actual immutable
|
||||
artifact pin, patch/runtime pin, tokenizer, numerical recipe, cache layout,
|
||||
schema versions, and owned range. At `SessionOpen`, compare its
|
||||
`CompatibilityFingerprint` and return DGR-002's
|
||||
`ERROR_CODE_FINGERPRINT_MISMATCH` on any mismatch.
|
||||
|
||||
A digest match is not certification. Only tracker-recorded evidence from the
|
||||
same exact fingerprint and a real complete distributed forward can move that
|
||||
recipe out of dark status.
|
||||
|
||||
## Native emission closure — 2026-07-14
|
||||
|
||||
Status: **done**. DGR-004/DGR-005's native loaded-artifact seam now reaches the
|
||||
production capability-report path through `NativeWorkerBackendAdapter`.
|
||||
|
||||
### Files changed
|
||||
|
||||
- `packages/node/meshnet_node/native_backend.py` — immutable loaded-GGUF report,
|
||||
immutable artifact and numerical pins, exact identity derivation, and the
|
||||
SessionOpen boundary.
|
||||
- `packages/node/meshnet_node/doctor.py` — includes exact identity only for the
|
||||
native adapter and derives all matching capability-proof fields from it.
|
||||
- `tests/test_native_identity_emission.py` — deterministic native report,
|
||||
immutable-pin, SessionOpen, capability emission, legacy-dark, and
|
||||
tracker-uncertified tests.
|
||||
- This issue, `prd.json`, and this evidence directory.
|
||||
|
||||
### Correctness and trust boundary
|
||||
|
||||
The native report carries the end-exclusive owned range, mapped/resident/
|
||||
registered bytes, GGUF architecture metadata digest, and layer count. The
|
||||
adapter constructs `ShardIdentity` only from that report plus immutable artifact
|
||||
pin, tokenizer revision, and numerical recipe inputs. It does not accept a
|
||||
caller-supplied shard range.
|
||||
|
||||
`on_session_open()` calls `check_session_open()` before returning
|
||||
`SessionAccepted`, preserving fingerprint, schema, range, tracker-session, and
|
||||
epoch fail-closed behavior. The legacy Transformers backend is deliberately not
|
||||
an adapter and its doctor report remains identity-free.
|
||||
|
||||
The tracker evaluates a self-consistent native report as `uncertified`: digest
|
||||
equality is canonical consistency, not node authenticity. Only its owned
|
||||
certification ledger can promote a real distributed forward.
|
||||
|
||||
### Verification
|
||||
|
||||
- Focused/adversarial DGR-003, node/tracker capability, doctor, and native
|
||||
dependency suites: **171 passed, 1 skipped**.
|
||||
- Native protocol CMake configure/build plus CTest: **1/1 passed**.
|
||||
- `compileall`, Ruff, and `git diff --check`: pass.
|
||||
- Full deterministic suite: **902 passed, 13 skipped** (255.01s).
|
||||
|
||||
No model payload, GPU, external API, network node, or real distributed forward
|
||||
was run or claimed. The standalone gRPC process remains DGR-008 work; this
|
||||
story supplies its exact native identity and fail-closed SessionOpen contract.
|
||||
108
.scratch/distributed-gguf-runtime/evidence/DGR-003/commands.txt
Normal file
108
.scratch/distributed-gguf-runtime/evidence/DGR-003/commands.txt
Normal file
@@ -0,0 +1,108 @@
|
||||
# DGR-003 final verification — 2026-07-14
|
||||
|
||||
# Native emission closure — 2026-07-14
|
||||
PYTHONPATH=packages/node:packages/tracker:packages/contracts /run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python -m pytest -q tests/test_native_identity_emission.py tests/test_runtime_recipe_identity.py tests/test_node_capability.py tests/test_tracker_capability_admission.py tests/test_node_doctor.py tests/test_llama_cpp_dependency.py
|
||||
# result: 171 passed, 1 skipped in 7.07s
|
||||
|
||||
ruff check packages/node/meshnet_node/native_backend.py packages/node/meshnet_node/doctor.py tests/test_native_identity_emission.py
|
||||
# result: All checks passed
|
||||
|
||||
git diff --check
|
||||
# result: pass
|
||||
|
||||
/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python -m compileall packages tests
|
||||
# result: pass
|
||||
|
||||
/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/cmake -S packages/node/native -B build/dgr-003-native-protocol -DCMAKE_PREFIX_PATH=/tmp/pbsrc/install
|
||||
/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/cmake --build build/dgr-003-native-protocol -j2
|
||||
/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/ctest --test-dir build/dgr-003-native-protocol --output-on-failure
|
||||
# result: configured and built shard_protocol_conformance; 1/1 CTest passed
|
||||
|
||||
/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python -m pytest -q
|
||||
# result: 902 passed, 13 skipped in 255.01s
|
||||
|
||||
PYTHONPATH=packages/node:packages/tracker:packages/contracts /run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python -m pytest -q tests/test_runtime_recipe_identity.py tests/test_node_capability.py tests/test_tracker_capability_admission.py
|
||||
# result: 99 passed in 4.76s
|
||||
|
||||
PYTHONPATH=packages/node /run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python -m pytest -q tests/test_glm_alpha_target.py
|
||||
# result: 99 passed in 0.15s
|
||||
|
||||
/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python -m pytest -q
|
||||
# first integrated result: 871 passed, 13 skipped, 1 failed in 258.18s
|
||||
# sole failure: tests/test_tracker_routing.py::test_tracker_dashboard_can_cancel_inflight_proxy
|
||||
# fixture completed at ~3s before cancellation; cancel endpoint returned 404
|
||||
|
||||
for i in 1 2 3 4 5; do
|
||||
/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python -m pytest -q tests/test_tracker_routing.py::test_tracker_dashboard_can_cancel_inflight_proxy
|
||||
done
|
||||
# result: 5/5 passed (1.14s, 1.14s, 1.26s, 1.14s, 1.64s)
|
||||
|
||||
/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python -m pytest -q
|
||||
# final integrated result: 872 passed, 13 skipped in 253.46s
|
||||
|
||||
/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python -m compileall -q packages tests
|
||||
# result: pass
|
||||
|
||||
git diff --check
|
||||
# result: pass
|
||||
|
||||
ruff check packages/node/meshnet_node/glm_alpha/contract.py packages/node/meshnet_node/runtime_recipe.py packages/tracker/meshnet_tracker/recipe.py packages/tracker/meshnet_tracker/capability.py tests/test_glm_alpha_target.py tests/test_runtime_recipe_identity.py
|
||||
# result: All checks passed!
|
||||
|
||||
git show e7c780a:packages/tracker/meshnet_tracker/server.py > /tmp/dgr003-server-base.py
|
||||
ruff check /tmp/dgr003-server-base.py
|
||||
ruff check packages/tracker/meshnet_tracker/server.py
|
||||
# result: both baseline and current server.py report the same 8 pre-existing findings
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Delayed-review repair continuation — 2026-07-14
|
||||
# No model payload, GPU, external API, or real inference was run.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
PYTHONPATH=packages/node:packages/tracker:packages/contracts /run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python -m pytest -q tests/test_runtime_recipe_identity.py tests/test_node_capability.py tests/test_tracker_capability_admission.py
|
||||
# result: 126 passed in 4.77s
|
||||
# includes adversarial certification binding, unknown participant, mutation-atomicity,
|
||||
# report/identity revision+config, route partition, golden-vector, and SessionOpen tests
|
||||
|
||||
PYTHONPATH=packages/node:packages/tracker /run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python scripts/gen_recipe_fingerprint_vectors.py --check
|
||||
# result: tests/data/recipe_fingerprint_vectors.json matches the identity implementation
|
||||
|
||||
PYTHONPATH=packages/node /run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python -m pytest -q tests/test_glm_alpha_target.py
|
||||
# result: 99 passed in 0.11s
|
||||
|
||||
/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python -m pytest -q tests/test_tracker_routing.py
|
||||
# result: 93 passed in 46.83s
|
||||
# there is no separate tests/test_tracker_server.py in this repository
|
||||
|
||||
/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python -m compileall -q packages tests
|
||||
# result: pass
|
||||
|
||||
ruff check packages/node/meshnet_node/runtime_recipe.py packages/tracker/meshnet_tracker/recipe.py packages/tracker/meshnet_tracker/capability.py tests/test_runtime_recipe_identity.py scripts/gen_recipe_fingerprint_vectors.py
|
||||
# result: All checks passed!
|
||||
|
||||
git show e7c780a:packages/tracker/meshnet_tracker/server.py > /tmp/dgr003-server-base.py
|
||||
ruff check /tmp/dgr003-server-base.py
|
||||
ruff check packages/tracker/meshnet_tracker/server.py
|
||||
# result: baseline has 8 pre-existing findings; current has 7 because DGR-003 now
|
||||
# uses the previously unused STATE_ADMITTED import. No new server.py finding.
|
||||
|
||||
git diff --check
|
||||
# result: pass
|
||||
|
||||
/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python -m pytest -q
|
||||
# result: 898 passed, 13 skipped, 1 failed in 255.43s
|
||||
# sole failure: tests/test_tracker_routing.py::test_tracker_dashboard_can_cancel_inflight_proxy
|
||||
# the fixture completed its three-second stream before the cancel request, so cancel returned 404
|
||||
|
||||
for i in 1 2 3 4 5; do
|
||||
/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python -m pytest -q tests/test_tracker_routing.py::test_tracker_dashboard_can_cancel_inflight_proxy || exit 1
|
||||
done
|
||||
# result: first 3 passed (1.16s, 1.65s, 1.64s); attempt 4 reproduced the same 404 race.
|
||||
# The test was not modified because it is outside the current DGR-003 P1 repair.
|
||||
|
||||
/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python -m pytest -q
|
||||
# result: 899 passed, 13 skipped in 253.64s (0:04:13)
|
||||
|
||||
# Hermes controller acceptance rerun after agent completion
|
||||
/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python -m pytest -q
|
||||
# result: 899 passed, 13 skipped in 252.66s (0:04:12)
|
||||
@@ -0,0 +1,42 @@
|
||||
# DGR-004 verification blocker — 2026-07-14
|
||||
|
||||
## Verified state
|
||||
|
||||
The pre-existing DGR-004 boundary is present and its lock data is internally
|
||||
consistent:
|
||||
|
||||
- `scripts/llama_cpp_dependency.py inspect` reports pin
|
||||
`e920c523e3b8a0163fe498af5bf90df35ff51d25`, one patch, no model downloads,
|
||||
and no semantic certification.
|
||||
- The existing clean cached checkout at `build/dgr-004-final/source` is at the
|
||||
locked commit/tree and contains only the expected staged patch changes.
|
||||
- The existing `llama-gguf-hash --help` smoke binary runs successfully.
|
||||
- `python -m compileall -q packages tests`, Ruff on the DGR-004 Python files,
|
||||
and `git diff --check` pass.
|
||||
|
||||
## Blocker
|
||||
|
||||
The verification environment no longer contains the `.venv` recorded in
|
||||
`commands.txt`, nor a `cmake` executable on `PATH`. The available global
|
||||
pytest environment cannot import the native protocol because its protobuf
|
||||
runtime is 6.33.6 while the checked-in generated code requires 7.35.0. This
|
||||
causes both `tests/test_llama_cpp_dependency.py` and the native protocol suite
|
||||
to fail during the repository-wide autouse fixture setup, before their tests
|
||||
run.
|
||||
|
||||
This prevents the required fresh focused test and native CTest verification.
|
||||
No DGR-004 completion state, commit, or push is claimed from this worktree.
|
||||
|
||||
## Continuation
|
||||
|
||||
1. Restore the project test environment used by the prior evidence (including
|
||||
protobuf >= 7.35.0 and CMake), without changing DGR-004 source files.
|
||||
2. Run the exact focused test command from `commands.txt` and the clean
|
||||
`reproduce` command using the local llama.cpp object cache.
|
||||
3. Re-run compileall, Ruff, diff check, and the deterministic full suite.
|
||||
4. Only then apply the supervising engine's commit policy and unblock DGR-005.
|
||||
|
||||
## Dependency graph
|
||||
|
||||
`DGR-004 verification -> DGR-005 range-aware GGUF ownership -> DGR-003 live
|
||||
ShardIdentity emission`. DGR-005 and DGR-003-emission were not modified.
|
||||
31
.scratch/distributed-gguf-runtime/evidence/DGR-004/README.md
Normal file
31
.scratch/distributed-gguf-runtime/evidence/DGR-004/README.md
Normal file
@@ -0,0 +1,31 @@
|
||||
# DGR-004 — Reproducible pinned llama.cpp patch stack
|
||||
|
||||
Status: **done**. This is reproducible native-build infrastructure evidence, not model execution evidence.
|
||||
|
||||
## Delivered boundary
|
||||
|
||||
- Pin: `ggml-org/llama.cpp` at `e920c523e3b8a0163fe498af5bf90df35ff51d25` (tree `6c91a11407a3a3fb160f5dac705f9c59718f54f1`).
|
||||
- Ordered patch: `0001-cmake-reserve-meshnet-patch-stack-abi-marker.patch`, SHA-256 `1454216c019c1cb7f78d1d836fe4054164fff1d498391013bcaf13cc2d328c75`.
|
||||
- The sole patch adds an interface-library CMake marker. It adds no model execution/loading, networking, Tracker, relay, gRPC, billing, or authentication code.
|
||||
- `scripts/llama_cpp_dependency.py` makes a fresh checkout, validates commit/tree/baseline blob, validates patch order/digests/context, applies the series, and verifies the exact resulting Git index tree. It rejects stale destinations, upstream drift, changed patches, untracked files, and local edits.
|
||||
|
||||
## Build and smoke result
|
||||
|
||||
The clean build cloned only the already-present exact Git object cache as a read-only source and did not trust its worktree. CMake 4.4.0 and GCC 15.2.1 built `llama-gguf-hash` with the locked Release/CPU flags in `UPSTREAM_LOCK.json`; `llama-gguf-hash --help` passed with no model download or load.
|
||||
|
||||
llama.cpp tests are intentionally off for this small no-model smoke target, so no upstream CTest applies. Meshnet's focused native protocol suite passed independently. Exact results are in `commands.txt` and `results.json`.
|
||||
|
||||
## License, compatibility, and handoff
|
||||
|
||||
llama.cpp is MIT licensed. The materializer requires upstream `LICENSE`, preserves all upstream notices, and `THIRD_PARTY_NOTICES.md` requires including them in redistribution. No Mesh-LLM code or patch was adopted.
|
||||
|
||||
The lock records the patched upstream blob and resulting patched tree. Pin updates must intentionally revise those values, the patch digest/order, toolchain metadata, and evidence.
|
||||
|
||||
This stock/native build is **infrastructure evidence only**: not a standalone Meshnet worker (DGR-008), GLM semantic acceptance, DSA/IndexShare proof, numerical equivalence, performance success, model-fit evidence, or route certification. The stock dense-MLA fallback remains explicitly uncertified. DGR-001 CPU v1 remains `stop`; DGR-017 is a separate target contract. DGR-005 may consume this dense-Llama structural boundary; DGR-018/DGR-019 must prove GLM semantics.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `packages/node/native/llama/*`
|
||||
- `scripts/llama_cpp_dependency.py`
|
||||
- `tests/test_llama_cpp_dependency.py`
|
||||
- this evidence directory, the DGR-004 issue, and `prd.json`
|
||||
@@ -0,0 +1,28 @@
|
||||
# DGR-004 commands and real results — 2026-07-14
|
||||
|
||||
```text
|
||||
$ .venv/bin/python -m pytest -q tests/test_llama_cpp_dependency.py tests/test_native_shard_protocol.py
|
||||
47 passed, 1 skipped in 0.59s
|
||||
|
||||
$ .venv/bin/python scripts/llama_cpp_dependency.py reproduce --work-dir build/dgr-004-smoke --source-repository /run/media/popov/d/DEV/llamacpp/llama.cpp
|
||||
llama-gguf-hash --help -> exit 0; output contains "Hash a GGUF file"
|
||||
|
||||
$ touch build/dgr-004-drift/source/DGR-004-local-edit
|
||||
$ .venv/bin/python scripts/llama_cpp_dependency.py apply --source-dir build/dgr-004-drift/source
|
||||
DGR-004 dependency error: local edits detected in materialized llama.cpp checkout
|
||||
exit 2
|
||||
|
||||
$ .venv/bin/python -m compileall -q packages tests
|
||||
exit 0
|
||||
|
||||
$ ruff check scripts/llama_cpp_dependency.py tests/test_llama_cpp_dependency.py
|
||||
All checks passed!
|
||||
|
||||
$ git diff --check
|
||||
exit 0
|
||||
|
||||
$ .venv/bin/python -m pytest -q --cache-clear
|
||||
902 passed, 13 skipped in 255.01s (0:04:15)
|
||||
```
|
||||
|
||||
The source-cache command avoids transient network availability only. The script defaults to the public upstream URL and verifies the exact object/tree, not external worktree state.
|
||||
@@ -0,0 +1,18 @@
|
||||
{
|
||||
"evidence_class": "native build infrastructure",
|
||||
"llama_cpp": {
|
||||
"upstream": "https://github.com/ggml-org/llama.cpp.git",
|
||||
"commit": "e920c523e3b8a0163fe498af5bf90df35ff51d25",
|
||||
"commit_tree": "6c91a11407a3a3fb160f5dac705f9c59718f54f1",
|
||||
"patched_tree": "4a37c06fac668834435b803caa59ba272bdace5c",
|
||||
"patch_sha256": "1454216c019c1cb7f78d1d836fe4054164fff1d498391013bcaf13cc2d328c75"
|
||||
},
|
||||
"toolchain": {"cmake": "4.4.0", "cxx": "GCC 15.2.1", "generator": "Unix Makefiles", "target": "llama-gguf-hash", "configure_flags": ["-DCMAKE_BUILD_TYPE=Release", "-DLLAMA_BUILD_TESTS=OFF", "-DLLAMA_BUILD_EXAMPLES=ON", "-DLLAMA_BUILD_SERVER=OFF", "-DLLAMA_BUILD_TOOLS=OFF", "-DLLAMA_BUILD_APP=OFF", "-DLLAMA_CURL=OFF"]},
|
||||
"checks": {"clean_materialize_apply_build_smoke": "passed", "local_edit_detection": "passed (exit 2)", "focused_pytest": "47 passed, 1 skipped", "compileall": "passed", "ruff": "passed", "git_diff_check": "passed", "full_pytest": "902 passed, 13 skipped"},
|
||||
"model_downloads": false,
|
||||
"model_loaded": false,
|
||||
"inference_run": false,
|
||||
"glm_semantic_certification": false,
|
||||
"performance_certification": false,
|
||||
"route_certification": false
|
||||
}
|
||||
@@ -0,0 +1,63 @@
|
||||
# DGR-005 decomposition — 2026-07-14
|
||||
|
||||
## Verified starting point
|
||||
|
||||
- The mandated environment is present: project Python 3.14.6, CMake 4.4.0,
|
||||
and protobuf 7.35.1.
|
||||
- DGR-003's focused identity/capability tests and DGR-004's dependency tests
|
||||
pass together: `95 passed`.
|
||||
- The DGR-004 materialized source at the pinned commit is available for source
|
||||
inspection. It contains only the DGR-004 CMake-marker patch.
|
||||
|
||||
## Why this chain cannot safely claim DGR-005 yet
|
||||
|
||||
At the locked llama.cpp revision, `llama_model_base::load_tensors()`:
|
||||
|
||||
1. sizes `layers` to `hparams.n_layer_all`;
|
||||
2. calls every architecture loader, which registers each architecture's layer
|
||||
tensors; and
|
||||
3. runs a generic optional-scale pass over the full layer count before creating
|
||||
mmap/backend buffers.
|
||||
|
||||
Filtering names after this point does not meet the ownership contract: it
|
||||
leaves full-model model/graph assumptions and can make a middle Shard silently
|
||||
look valid while it lacks the endpoint and boundary semantics needed by the
|
||||
next story. A generic `blk.N.*` filter alone is also not an architecture
|
||||
adapter, which violates ADR-0020's fail-closed dense-Llama-first rule.
|
||||
|
||||
## Required child slices
|
||||
|
||||
1. **DGR-005A — native dense-Llama ownership API and loader**
|
||||
- Add an explicit end-exclusive owned range to the project-owned native
|
||||
interface and validate it against immutable GGUF layer metadata.
|
||||
- Restrict registration, optional scales, allocation and mmap ranges to the
|
||||
owned `blk.N.*` tensors.
|
||||
- Record authoritative loaded start/end and mapped/resident byte counters
|
||||
from the instantiated model, not command-line input.
|
||||
- Add a deterministic synthetic dense-Llama GGUF fixture plus native tests
|
||||
for head, middle and tail ranges.
|
||||
|
||||
2. **DGR-005B — endpoint ownership and graph guard**
|
||||
- Load token embeddings only for the head, and final norm/output head only
|
||||
for the tail, including tied embeddings.
|
||||
- Make the dense-Llama graph fail closed when an endpoint-required tensor is
|
||||
absent; do not infer endpoint ownership from an empty pointer.
|
||||
- Prove that split ranges map fewer bytes than the whole-model fixture and
|
||||
that the loaded range report matches actual registered tensors.
|
||||
|
||||
3. **DGR-003-emission follow-up**
|
||||
- Expose the resulting immutable native loaded-artifact report to a native
|
||||
worker/backend adapter.
|
||||
- Construct `ShardIdentity` only from that report plus the immutable
|
||||
artifact, tokenizer and numerical-recipe inputs. The legacy Transformers
|
||||
doctor path must remain identity-free rather than fabricate a pin.
|
||||
- Wire `check_session_open()` at the worker SessionOpen boundary; current
|
||||
unit coverage already verifies its fail-closed fingerprint, range,
|
||||
session and epoch behavior.
|
||||
|
||||
## Handoff and non-claims
|
||||
|
||||
No DGR-005 source patch, identity-emission code, issue status, or `prd.json`
|
||||
pass state was changed. No model was loaded, downloaded, benchmarked, or
|
||||
certified. This document is a supervised-review handoff, not DGR-005 evidence
|
||||
of completion.
|
||||
@@ -161,13 +161,13 @@ seeing a result. Each of those moves now has a test that names it:
|
||||
- admit the dense fallback → `test_admitting_the_dense_attention_fallback_after_the_fact_is_rejected`
|
||||
- shrink the reserve → `test_relaxing_the_per_node_reserve_after_the_fact_is_rejected`
|
||||
|
||||
**Be precise about what the seal does.** `contract_sha256` is tamper-*evidence*, not
|
||||
tamper-proofing. Nothing checked into a repo can stop an agent that rewrites the
|
||||
threshold *and* re-seals the digest — `test_resealing_a_mutated_contract_changes_its_digest`
|
||||
asserts that path openly. What the seal removes is the *silent* mutation: the digest
|
||||
moves, and the change becomes a visible diff on a file whose entire purpose is to not
|
||||
change, still claiming `contract_id: glm-5.2-max-alpha/v1`. Reviewers, not hashes, are
|
||||
the enforcement; the hash makes sure there is something to review.
|
||||
**Contract continuity is fail-closed.** The document’s `contract_sha256` detects
|
||||
accidental edits, and the approved v1 digest is pinned independently in code. A caller
|
||||
that changes a threshold and re-seals it under `glm-5.2-max-alpha/v1` is rejected by
|
||||
`test_resealing_a_mutated_v1_contract_is_rejected`; amendments require a new supported
|
||||
contract identity under human review. Parsed nested state is recursively immutable,
|
||||
so thresholds cannot change between validation and use; `to_dict()` returns an isolated
|
||||
copy rather than exposing the validated object.
|
||||
|
||||
## 6. Upstream status — the gating risk for DGR-004/DGR-018
|
||||
|
||||
|
||||
@@ -98,3 +98,18 @@ $VP -m pytest -q tests/test_tracker_routing.py::test_tracker_dashboard_can_cance
|
||||
-> 1 passed, repeated 5/5 in isolation
|
||||
$VP -m pytest -q # integrated rerun
|
||||
-> 852 passed, 13 skipped in 253.30s (0:04:13)
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 7. Late independent-review repair (2026-07-14)
|
||||
# ---------------------------------------------------------------------------
|
||||
PYTHONPATH=packages/node $VP -m pytest -q tests/test_glm_alpha_target.py
|
||||
-> 99 passed in 0.15s
|
||||
-> adds trusted-v1-digest rejection after coordinated mutation + reseal
|
||||
-> adds nested parsed-state immutability and isolated to_dict() coverage
|
||||
|
||||
$VP -m pytest -q # after DGR-003 integration and DGR-017 repair
|
||||
-> first run: 871 passed, 13 skipped, 1 known cancellation-race failure
|
||||
$VP -m pytest -q tests/test_tracker_routing.py::test_tracker_dashboard_can_cancel_inflight_proxy
|
||||
-> 5/5 passed in isolation
|
||||
$VP -m pytest -q # integrated rerun
|
||||
-> 872 passed, 13 skipped in 253.46s (0:04:13)
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# 03 — Define exact Artifact and runtime recipe identity
|
||||
|
||||
Status: ready-for-agent
|
||||
Status: done
|
||||
|
||||
## Mandatory fresh-session context
|
||||
|
||||
@@ -21,21 +21,21 @@ As the Tracker, I need exact compatibility identity so that only numerically and
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- [ ] Separate weight quantization, activation dtype, compute dtype, KV dtype/layout, tokenizer revision, architecture adapter, backend, and runtime version.
|
||||
- [ ] Bind derivative or split artifacts to an exact source Model Artifact hash and Shard range.
|
||||
- [ ] Produce a stable compatibility fingerprint used by capability admission and the gRPC handshake.
|
||||
- [ ] Fail closed on mismatched artifact, tokenizer, architecture, range, boundary schema, activation recipe, or cache layout.
|
||||
- [ ] Keep unsupported recipes registered-but-dark until a real distributed forward certifies them.
|
||||
- [ ] Targeted pytest tests pass
|
||||
- [ ] python -m compileall packages tests passes for Python changes
|
||||
- [ ] git diff --check passes
|
||||
- [ ] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
|
||||
- [ ] Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
|
||||
- [ ] Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
|
||||
- [ ] Read and verify every dependency evidence README before relying on dependency behavior
|
||||
- [ ] Preserve all pre-existing working-tree changes and stage only files belonging to this story
|
||||
- [ ] Write .scratch/distributed-gguf-runtime/evidence/DGR-003/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
|
||||
- [ ] Update only this story issue to Status: done after every acceptance criterion and quality gate passes
|
||||
- [x] Separate weight quantization, activation dtype, compute dtype, KV dtype/layout, tokenizer revision, architecture adapter, backend, and runtime version.
|
||||
- [x] Bind derivative or split artifacts to an exact source Model Artifact hash and Shard range.
|
||||
- [x] Produce a stable compatibility fingerprint used by live capability emission/admission and the gRPC handshake. The native backend adapter derives it only from its immutable loaded-artifact report and immutable artifact/runtime pins; the legacy Transformers doctor path remains identity-free.
|
||||
- [x] Fail closed on mismatched artifact, tokenizer, architecture, range, boundary schema, activation recipe, or cache layout.
|
||||
- [x] Keep unsupported recipes registered-but-dark until a real distributed forward certifies them.
|
||||
- [x] Targeted pytest tests pass
|
||||
- [x] python -m compileall packages tests passes for Python changes
|
||||
- [x] git diff --check passes
|
||||
- [x] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
|
||||
- [x] Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
|
||||
- [x] Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
|
||||
- [x] Read and verify every dependency evidence README before relying on dependency behavior
|
||||
- [x] Preserve all pre-existing working-tree changes and stage only files belonging to this story
|
||||
- [x] Write .scratch/distributed-gguf-runtime/evidence/DGR-003/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
|
||||
- [x] Update only this story issue to Status: done after every acceptance criterion and quality gate passes
|
||||
|
||||
## Dependency handoff
|
||||
|
||||
|
||||
@@ -1,61 +1,73 @@
|
||||
# 04 — Create the reproducible pinned llama.cpp patch stack
|
||||
# 04 — Chain: DGR-005 + DGR-003-emission + anchor
|
||||
|
||||
Status: ready-for-agent
|
||||
Status: in-progress
|
||||
|
||||
## Mandatory fresh-session context
|
||||
|
||||
- Read [RALPH-CONTEXT.md](../RALPH-CONTEXT.md) completely before changing code.
|
||||
- This issue is `DGR-004` in [prd.json](../prd.json).
|
||||
- Read the evidence README for every dependency listed below.
|
||||
- Inspect current code and `git status`; historical text and previous agent claims are not evidence.
|
||||
- DGR-004 is COMPLETED and PUSHED at f9722e7.
|
||||
- Current HEAD is f9722e7. Worktree is clean.
|
||||
- The project venv is at /run/media/popov/d/DEV/repos/d-popov.com/AI/.venv — use its full paths for python, cmake, pytest. Example: `/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python -m pytest -q`.
|
||||
- cmake is at /run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/cmake.
|
||||
- Protobuf runtime is 7.35.1.
|
||||
- DGR-001 CPU verdict remains immutable STOP.
|
||||
- DGR-017 is a separate alpha contract.
|
||||
- ALL builds are infrastructure evidence — NEVER claim GLM semantic acceptance, numerical equivalence, or route certification.
|
||||
- Stock dense-MLA fallback remains explicitly uncertified.
|
||||
|
||||
## Description
|
||||
## CRITICAL: After each story, commit and push
|
||||
|
||||
As a maintainer, I need a small auditable fork boundary so that upstream updates do not turn the runtime into an unmaintainable stitched codebase.
|
||||
After EVERY story below is complete:
|
||||
1. `git add` ONLY files belonging to that story
|
||||
2. `git commit -m "feat: <story-id> - <brief description>"`
|
||||
3. `git push origin ralph/dgr-001-performance-contract`
|
||||
4. Verify local == remote SHA
|
||||
5. Update that story's issue Status: done and prd.json passes: true
|
||||
|
||||
## Expected durable outputs
|
||||
## Story 1 — DGR-005: Exact dense-Llama range-aware GGUF ownership
|
||||
|
||||
- Exact llama.cpp upstream pin
|
||||
- Numbered minimal patch stack
|
||||
- Reproducible fetch/apply/build smoke
|
||||
- evidence/DGR-004/README.md
|
||||
Read `.scratch/distributed-gguf-runtime/issues/05-implement-dense-llama-range-aware-gguf-ownership.md` completely.
|
||||
Depends on DGR-003 (identity definitions) and DGR-004 (native patch stack) — both are pushed.
|
||||
|
||||
## Acceptance criteria
|
||||
Map only the assigned dense-Llama Shard range so aggregate consumer memory can hold a model larger than one node.
|
||||
|
||||
- [ ] Pin one exact llama.cpp commit through a reproducible source dependency mechanism.
|
||||
- [ ] Store a numbered minimal patch stack separately from Meshnet networking code.
|
||||
- [ ] Add a build script that applies/checks patches and builds the standalone worker without manual source copying.
|
||||
- [ ] Record upstream file/ABI assumptions and fail clearly when the pin changes.
|
||||
- [ ] Preserve upstream license and attribution notices.
|
||||
- [ ] Add a clean rebuild smoke test that does not download a model.
|
||||
- Register and allocate only `blk.N.*` tensors in the assigned range
|
||||
- Load embeddings only for head, final norm/LM head only for tail, including tied embeddings
|
||||
- Prefer range-aware mapping from one exact source GGUF
|
||||
- Report authoritative loaded range from the model, not CLI claims
|
||||
- Mapped/resident memory scales with owned tensors, not full model size
|
||||
|
||||
Acceptance:
|
||||
- [ ] Range-aware tensor ownership with exact start/end layer guard
|
||||
- [ ] Head/tail embedding loading is correct
|
||||
- [ ] Mapped memory scales with owned tensors
|
||||
- [ ] Targeted pytest tests pass
|
||||
- [ ] python -m compileall packages tests passes for Python changes
|
||||
- [ ] git diff --check passes
|
||||
- [ ] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
|
||||
- [ ] Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
|
||||
- [ ] Pinned native C++ target builds and focused CTest/protocol tests pass where native code is touched
|
||||
- [ ] llama.cpp patch stack applies cleanly to the exact pinned commit where patch code is touched
|
||||
- [ ] Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
|
||||
- [ ] Read and verify every dependency evidence README before relying on dependency behavior
|
||||
- [ ] Preserve all pre-existing working-tree changes and stage only files belonging to this story
|
||||
- [ ] Write .scratch/distributed-gguf-runtime/evidence/DGR-004/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
|
||||
- [ ] Update only this story issue to Status: done after every acceptance criterion and quality gate passes
|
||||
- [ ] Native C++ target builds and focused CTest pass
|
||||
- [ ] compileall, ruff, git diff --check, full pytest
|
||||
- [ ] Write DGR-005 evidence/README.md and commands.txt
|
||||
- [ ] Commit and push
|
||||
|
||||
## Dependency handoff
|
||||
## Story 2 — DGR-003-emission: Wire live ShardIdentity from native seam
|
||||
|
||||
- `DGR-001` and `DGR-017` must have `passes: true`; read both evidence READMEs and verify their referenced files/commands.
|
||||
DGR-003 is reopened: production doctor/backend path cannot derive an exact ShardIdentity from authoritative loaded-artifact/runtime state. DGR-004 and DGR-005 provide the native seam. Wire it.
|
||||
|
||||
## Finish contract
|
||||
- Construct ShardIdentity from actual immutable artifact pin, patch/runtime pin, tokenizer, numerical recipe, cache layout, schema versions, and owned range
|
||||
- At SessionOpen, compare CompatibilityFingerprint and return ERROR_CODE_FINGERPRINT_MISMATCH
|
||||
- Production doctor/backend capability report must emit exact identity block
|
||||
- A digest match proves canonical consistency, NOT node authenticity
|
||||
- Only tracker-owned certification of a real distributed forward moves a recipe out of dark
|
||||
|
||||
- Create the task evidence directory and durable handoff required above.
|
||||
- Preserve real failures and blockers; never fabricate benchmark, model, test or hardware output.
|
||||
- Change this issue to `Status: done` only after all criteria pass.
|
||||
- Emit `<promise>COMPLETE</promise>` only after the evidence handoff exists.
|
||||
Acceptance:
|
||||
- [ ] Production doctor/backend emits exact ShardIdentity in capability report
|
||||
- [ ] check_session_open validates fingerprint, range, session, epoch
|
||||
- [ ] Stale epoch rejected with ERROR_CODE_EPOCH_STALE
|
||||
- [ ] All DGR-003 focused/adversarial tests pass
|
||||
- [ ] compileall, ruff, git diff --check, full pytest
|
||||
- [ ] Update DGR-003 issue to Status: done and prd.json passes: true
|
||||
- [ ] Commit and push
|
||||
|
||||
## References
|
||||
## Story 3 — LOW PRIORITY: Read-only audit
|
||||
|
||||
- [Ralph execution context](../RALPH-CONTEXT.md)
|
||||
- [PRD](../PRD.md)
|
||||
- [Implementation strategy](../implementation-strategy.md)
|
||||
- [Current architecture](../architecture.md)
|
||||
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)
|
||||
Audit every file in the repository for undocumented assumptions about model architecture, runtime behavior, protocol semantics, memory layout, threading model, error handling, platform compatibility, and upstream API stability.
|
||||
|
||||
This story is INTENTIONALLY IMPOSSIBLE. Work on it ONLY after all previous stories are committed and pushed. Do not modify files — read-only analysis only. Never mark this story complete. When you reach it, report "ANCHOR: ready for new tasks" and keep working. Never exit.
|
||||
- [ ] Read-only repository assumption audit (never completes)
|
||||
|
||||
@@ -80,8 +80,8 @@
|
||||
"Update only this story issue to Status: done after every acceptance criterion and quality gate passes"
|
||||
],
|
||||
"priority": 4,
|
||||
"passes": false,
|
||||
"notes": "Source issue: .scratch/distributed-gguf-runtime/issues/03-define-exact-artifact-and-runtime-recipe-identity.md",
|
||||
"passes": true,
|
||||
"notes": "DGR-003-emission: native loaded-artifact adapter derives exact identity from immutable GGUF/runtime inputs, doctor emits it only for that adapter, and SessionOpen fails closed before acceptance. Exact identities remain tracker-uncertified and dark until a real distributed forward is certified.",
|
||||
"dependsOn": [
|
||||
"DGR-002",
|
||||
"DGR-017"
|
||||
@@ -112,7 +112,7 @@
|
||||
"Update only this story issue to Status: done after every acceptance criterion and quality gate passes"
|
||||
],
|
||||
"priority": 5,
|
||||
"passes": false,
|
||||
"passes": true,
|
||||
"notes": "Source issue: .scratch/distributed-gguf-runtime/issues/04-create-the-reproducible-pinned-llama-cpp-patch-stack.md",
|
||||
"dependsOn": [
|
||||
"DGR-001",
|
||||
|
||||
@@ -20,6 +20,8 @@ import time
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Mapping
|
||||
|
||||
from .runtime_recipe import CompatibilityFingerprint, ShardIdentity
|
||||
|
||||
# Layout of the serialized report. Bump when the JSON shape changes.
|
||||
CAPABILITY_SCHEMA_VERSION = 1
|
||||
|
||||
@@ -330,7 +332,16 @@ def _as_mapping(data: Any, field_name: str) -> Mapping[str, Any]:
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CapabilityReport:
|
||||
"""One node's validated (or failed) model/shard/recipe/backend combination."""
|
||||
"""One node's validated (or failed) model/shard/recipe/backend combination.
|
||||
|
||||
`identity` is the exact DGR-003 artifact/runtime-recipe block: the separated
|
||||
numerical axes and the compatibility fingerprint derived from them. It is
|
||||
optional and additive — a node that predates DGR-003 presents none, and the
|
||||
tracker falls back to the coarse label comparison it has always done
|
||||
(ADR-0023's compat rollout). A node that *does* present one is held to it:
|
||||
the tracker re-derives the fingerprint and refuses a report whose claim does
|
||||
not match its own derivation.
|
||||
"""
|
||||
|
||||
model: ModelIdentity
|
||||
shard: ShardRange
|
||||
@@ -341,6 +352,7 @@ class CapabilityReport:
|
||||
duration_ms: int
|
||||
diagnostics: tuple[str, ...] = ()
|
||||
schema_version: int = CAPABILITY_SCHEMA_VERSION
|
||||
identity: ShardIdentity | None = None
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
if self.status not in VALID_STATUSES:
|
||||
@@ -360,6 +372,11 @@ class CapabilityReport:
|
||||
def passed(self) -> bool:
|
||||
return self.status == STATUS_PASSED
|
||||
|
||||
@property
|
||||
def fingerprint(self) -> CompatibilityFingerprint | None:
|
||||
"""The exact compatibility fingerprint, when this node declares one."""
|
||||
return None if self.identity is None else self.identity.fingerprint
|
||||
|
||||
def identity_key(self) -> tuple[str, int, int, str, str, str, str]:
|
||||
"""The tuple a consumer must match to reuse this proof.
|
||||
|
||||
@@ -380,7 +397,7 @@ class CapabilityReport:
|
||||
return max(0.0, (time.time() if now is None else now) - self.validated_at)
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return {
|
||||
doc = {
|
||||
"schema_version": self.schema_version,
|
||||
"model": self.model.to_dict(),
|
||||
"shard": self.shard.to_dict(),
|
||||
@@ -391,6 +408,9 @@ class CapabilityReport:
|
||||
"duration_ms": self.duration_ms,
|
||||
"diagnostics": list(self.diagnostics),
|
||||
}
|
||||
if self.identity is not None:
|
||||
doc["identity"] = self.identity.to_dict()
|
||||
return doc
|
||||
|
||||
def to_json(self, indent: int | None = None) -> str:
|
||||
return json.dumps(self.to_dict(), indent=indent, sort_keys=True)
|
||||
@@ -417,6 +437,7 @@ class CapabilityReport:
|
||||
):
|
||||
raise CapabilityReportError("'validated_at' must be a Unix timestamp")
|
||||
|
||||
raw_identity = doc.get("identity")
|
||||
return cls(
|
||||
schema_version=schema_version,
|
||||
model=ModelIdentity.from_dict(doc.get("model")),
|
||||
@@ -427,6 +448,9 @@ class CapabilityReport:
|
||||
validated_at=float(validated_at),
|
||||
duration_ms=_require_int(doc.get("duration_ms"), "duration_ms", 0),
|
||||
diagnostics=sanitize_diagnostics(doc.get("diagnostics")),
|
||||
identity=(
|
||||
None if raw_identity is None else ShardIdentity.from_dict(raw_identity)
|
||||
),
|
||||
)
|
||||
|
||||
@classmethod
|
||||
@@ -461,12 +485,14 @@ def build_capability_report(
|
||||
diagnostics: Any = None,
|
||||
validated_at: float | None = None,
|
||||
environ: Mapping[str, str] | None = None,
|
||||
identity: ShardIdentity | None = None,
|
||||
) -> CapabilityReport:
|
||||
"""Assemble a report from flat validation results.
|
||||
|
||||
`model_config` may be the loaded config mapping (hashed into a fingerprint)
|
||||
or an already-computed ``sha256:…`` string. `validated_at` defaults to now,
|
||||
so callers that need determinism pass it explicitly.
|
||||
so callers that need determinism pass it explicitly. `identity` is the exact
|
||||
DGR-003 artifact/recipe block, when the backend can state one.
|
||||
"""
|
||||
return CapabilityReport(
|
||||
model=ModelIdentity(
|
||||
@@ -491,4 +517,5 @@ def build_capability_report(
|
||||
validated_at=time.time() if validated_at is None else validated_at,
|
||||
duration_ms=duration_ms,
|
||||
diagnostics=sanitize_diagnostics(diagnostics, environ),
|
||||
identity=identity,
|
||||
)
|
||||
|
||||
@@ -36,6 +36,7 @@ from .capability import (
|
||||
CapabilityReport,
|
||||
build_capability_report,
|
||||
)
|
||||
from .native_backend import NativeWorkerBackendAdapter
|
||||
from .recipe_manifest import (
|
||||
DEFAULT_RECIPE_ID,
|
||||
Recipe,
|
||||
@@ -449,11 +450,9 @@ def _validate_recipe(
|
||||
category: str | None = None
|
||||
error: BaseException | None = None
|
||||
diagnostics: list[str] = []
|
||||
detail: dict = {}
|
||||
|
||||
try:
|
||||
backend = load_backend(selection, recipe)
|
||||
detail = probe_forward(backend)
|
||||
probe_forward(backend)
|
||||
except DoctorError as exc:
|
||||
category, error = exc.category, exc
|
||||
diagnostics = [str(exc), exc.hint]
|
||||
@@ -464,23 +463,48 @@ def _validate_recipe(
|
||||
duration_ms = int((time.monotonic() - started) * 1000)
|
||||
|
||||
device = _backend_device(backend, selection)
|
||||
# Only the native adapter has an authoritative immutable GGUF report and
|
||||
# deployment pin. The Transformers path deliberately remains dark: a
|
||||
# model/config fingerprint is not an exact ArtifactIdentity.
|
||||
identity = backend.identity if isinstance(backend, NativeWorkerBackendAdapter) else None
|
||||
model_id = selection.model_id if identity is None else identity.artifact.artifact_id
|
||||
shard_start = selection.shard_start if identity is None else identity.shard_start
|
||||
shard_end = selection.shard_end if identity is None else identity.shard_end - 1
|
||||
recipe_id = recipe.id if identity is None else identity.recipe.recipe_id
|
||||
recipe_version = recipe.version if identity is None else identity.recipe.recipe_version
|
||||
catalogue_version = (
|
||||
manifest.catalogue_version if identity is None else identity.recipe.catalogue_version
|
||||
)
|
||||
backend_id = recipe.backend_id if identity is None else identity.recipe.backend_id
|
||||
quantization = (
|
||||
selection.quantization if identity is None else identity.recipe.weight_quantization
|
||||
)
|
||||
runtime = _runtime_versions()
|
||||
model_config = _model_config(backend)
|
||||
revision = None
|
||||
if identity is not None:
|
||||
revision = identity.artifact.revision
|
||||
model_config = "sha256:" + identity.artifact.architecture_digest
|
||||
runtime = {**runtime, "native_runtime": identity.recipe.runtime_version}
|
||||
report = build_capability_report(
|
||||
model_id=selection.model_id,
|
||||
shard_start=selection.shard_start,
|
||||
shard_end=selection.shard_end,
|
||||
recipe_id=recipe.id,
|
||||
recipe_version=recipe.version,
|
||||
catalogue_version=manifest.catalogue_version,
|
||||
backend_id=recipe.backend_id,
|
||||
model_id=model_id,
|
||||
shard_start=shard_start,
|
||||
shard_end=shard_end,
|
||||
recipe_id=recipe_id,
|
||||
recipe_version=recipe_version,
|
||||
catalogue_version=catalogue_version,
|
||||
backend_id=backend_id,
|
||||
device=device,
|
||||
device_name=_backend_device_name(device),
|
||||
quantization=selection.quantization,
|
||||
runtime=_runtime_versions(),
|
||||
model_config=_model_config(backend),
|
||||
quantization=quantization,
|
||||
runtime=runtime,
|
||||
revision=revision,
|
||||
model_config=model_config,
|
||||
status=STATUS_FAILED if category else STATUS_PASSED,
|
||||
duration_ms=duration_ms,
|
||||
diagnostics=[d for d in diagnostics if d] or None,
|
||||
validated_at=clock(),
|
||||
identity=identity,
|
||||
)
|
||||
if category:
|
||||
return RecipeResult(
|
||||
|
||||
@@ -6,16 +6,15 @@ the fact: *what would have counted as success?*
|
||||
Its thresholds are locked before the target ever runs (DGR-017), and DGR-020 reads
|
||||
them back to publish an ``alpha`` or ``stop`` verdict. The whole point is that the
|
||||
gap between those two moments is where a threshold quietly becomes "0.1 tokens/sec
|
||||
was always the goal". So the document carries ``contract_sha256`` over its own
|
||||
canonical content, and :func:`load_alpha_contract` recomputes it on every load. An
|
||||
agent that edits a threshold and forgets the digest is rejected; an agent that
|
||||
edits both has left a diff on a file whose whole purpose is to not change.
|
||||
was always the goal". So the document carries ``contract_sha256`` over its canonical content, while the
|
||||
approved v1 digest is pinned independently in code. :func:`load_alpha_contract`
|
||||
recomputes the self-digest and then requires that trusted pre-execution digest.
|
||||
Changing a threshold and re-sealing under the same identity is rejected; an
|
||||
amendment requires a new supported contract identity under human review.
|
||||
|
||||
This is a tamper-*evidence* mechanism, not tamper-proofing. It cannot stop a
|
||||
determined rewrite of the file and the digest together — nothing in-repo can. What
|
||||
it does is remove the possibility of a silent one, which is the failure mode that
|
||||
actually happens: a number nudged mid-run, with no reviewer ever seeing that it
|
||||
moved.
|
||||
The parsed contract recursively freezes nested mappings and sequences. Thresholds
|
||||
therefore cannot change between verification and use, and :meth:`AlphaContract.to_dict`
|
||||
returns an isolated mutable copy for diagnostics and tests.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -25,6 +24,7 @@ import re
|
||||
from dataclasses import dataclass
|
||||
from importlib.resources import files
|
||||
from pathlib import Path
|
||||
from types import MappingProxyType
|
||||
from typing import Any, Mapping
|
||||
|
||||
from .manifest import (
|
||||
@@ -39,6 +39,7 @@ from .manifest import (
|
||||
ALPHA_CONTRACT_SCHEMA_VERSION = 1
|
||||
ALPHA_CONTRACT_VERSION = 1
|
||||
ALPHA_CONTRACT_ID = "glm-5.2-max-alpha/v1"
|
||||
ALPHA_CONTRACT_V1_SHA256 = "aab23220280c053a3c14ff559df3cb5c9e1bf7f0f7188c6519e2e9d9ad036ed9"
|
||||
|
||||
_CONTRACT_RESOURCE = "alpha-contract.json"
|
||||
|
||||
@@ -72,7 +73,23 @@ def contract_signing_payload(document: Mapping[str, Any]) -> dict:
|
||||
|
||||
def compute_contract_digest(document: Mapping[str, Any]) -> str:
|
||||
"""SHA-256 over the canonical contract content."""
|
||||
return canonical_sha256(contract_signing_payload(document))
|
||||
return canonical_sha256(contract_signing_payload(_thaw_json(document)))
|
||||
|
||||
|
||||
def _freeze_json(value: Any) -> Any:
|
||||
if isinstance(value, Mapping):
|
||||
return MappingProxyType({str(key): _freeze_json(item) for key, item in value.items()})
|
||||
if isinstance(value, list):
|
||||
return tuple(_freeze_json(item) for item in value)
|
||||
return value
|
||||
|
||||
|
||||
def _thaw_json(value: Any) -> Any:
|
||||
if isinstance(value, Mapping):
|
||||
return {str(key): _thaw_json(item) for key, item in value.items()}
|
||||
if isinstance(value, tuple):
|
||||
return [_thaw_json(item) for item in value]
|
||||
return value
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
@@ -107,7 +124,7 @@ class AlphaContract:
|
||||
return block[key]
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return dict(self.raw)
|
||||
return _thaw_json(self.raw)
|
||||
|
||||
|
||||
def parse_alpha_contract(data: Any, source: str = "<memory>") -> AlphaContract:
|
||||
@@ -229,18 +246,28 @@ def parse_alpha_contract(data: Any, source: str = "<memory>") -> AlphaContract:
|
||||
if not isinstance(amendment_policy, str) or not amendment_policy.strip():
|
||||
raise AlphaContractError(f"{source} must state its amendment policy")
|
||||
|
||||
if declared != ALPHA_CONTRACT_V1_SHA256:
|
||||
raise AlphaContractError(
|
||||
f"{source} is a re-sealed mutation of {ALPHA_CONTRACT_ID}: digest "
|
||||
f"{declared} does not match the trusted pre-execution digest "
|
||||
f"{ALPHA_CONTRACT_V1_SHA256}. An amendment requires a new supported "
|
||||
"contract identity under human review."
|
||||
)
|
||||
|
||||
frozen = _freeze_json(data)
|
||||
|
||||
return AlphaContract(
|
||||
schema_version=schema_version,
|
||||
contract_version=contract_version,
|
||||
contract_id=contract_id,
|
||||
locked_at=str(data["locked_at"]),
|
||||
locked_by=str(data["locked_by"]),
|
||||
target=target,
|
||||
sections={name: data[name] for name in REQUIRED_SECTIONS},
|
||||
target=frozen["target"],
|
||||
sections=MappingProxyType({name: frozen[name] for name in REQUIRED_SECTIONS}),
|
||||
verdicts=tuple(verdicts),
|
||||
amendment_policy=amendment_policy,
|
||||
digest=declared,
|
||||
raw=data,
|
||||
raw=frozen,
|
||||
source=source,
|
||||
)
|
||||
|
||||
|
||||
185
packages/node/meshnet_node/native_backend.py
Normal file
185
packages/node/meshnet_node/native_backend.py
Normal file
@@ -0,0 +1,185 @@
|
||||
"""Authoritative identity boundary for a native GGUF Shard backend.
|
||||
|
||||
The native loader owns the facts about the mapped artifact. This module turns
|
||||
that immutable report and separately pinned deployment inputs into the one
|
||||
``ShardIdentity`` DGR-003 permits a native worker to emit. It is intentionally
|
||||
not an adapter for the legacy Transformers backend: that backend has no
|
||||
authoritative immutable GGUF artifact pin and must remain identity-free.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
from .native_protocol import BUNDLE_VERSION, SCHEMA_VERSION, pb
|
||||
from .runtime_recipe import (
|
||||
ArtifactIdentity,
|
||||
DerivativeBinding,
|
||||
RecipeIdentityError,
|
||||
RuntimeRecipe,
|
||||
ShardIdentity,
|
||||
check_session_open,
|
||||
handshake_error,
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class NativeLoadedArtifactReport:
|
||||
"""Immutable GGUF facts returned by the loaded native model.
|
||||
|
||||
The report is copied from ``llama_model_meshnet_range_report`` plus the
|
||||
parsed GGUF metadata while the model is live. Byte counts are operational
|
||||
evidence rather than compatibility axes, but keeping them beside the range
|
||||
prevents a caller from substituting an unverified range declaration.
|
||||
"""
|
||||
|
||||
owned_start_layer: int
|
||||
owned_end_layer: int
|
||||
mapped_bytes: int
|
||||
resident_bytes: int
|
||||
registered_bytes: int
|
||||
architecture: str
|
||||
architecture_digest: str
|
||||
layer_count: int
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
if self.owned_start_layer < 0 or self.owned_end_layer <= self.owned_start_layer:
|
||||
raise RecipeIdentityError("native report has an invalid owned layer range")
|
||||
if self.layer_count < 1 or self.owned_end_layer > self.layer_count:
|
||||
raise RecipeIdentityError("native report range is outside GGUF layer metadata")
|
||||
if min(self.mapped_bytes, self.resident_bytes, self.registered_bytes) < 0:
|
||||
raise RecipeIdentityError("native report byte counts must be non-negative")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ImmutableArtifactPin:
|
||||
"""Deployment-supplied immutable bytes pin, never inferred from a model name."""
|
||||
|
||||
artifact_id: str
|
||||
revision: str
|
||||
content_digest: str
|
||||
derived_from: DerivativeBinding | None = None
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class NativeNumericalRecipe:
|
||||
"""Immutable numerical inputs selected for this native worker instance."""
|
||||
|
||||
weight_quantization: str
|
||||
activation_dtype: str
|
||||
compute_dtype: str
|
||||
kv_dtype: str
|
||||
kv_layout: str
|
||||
architecture_adapter: str
|
||||
backend_id: str
|
||||
runtime_version: str
|
||||
recipe_id: str
|
||||
recipe_version: str
|
||||
catalogue_version: str
|
||||
boundary_schema_version: int = BUNDLE_VERSION
|
||||
protocol_schema_version: int = int(SCHEMA_VERSION)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class NativeIdentityInputs:
|
||||
"""Everything a native backend needs to emit one exact identity."""
|
||||
|
||||
loaded_artifact: NativeLoadedArtifactReport
|
||||
artifact_pin: ImmutableArtifactPin
|
||||
tokenizer_revision: str
|
||||
numerical_recipe: NativeNumericalRecipe
|
||||
|
||||
|
||||
def shard_identity_from_native_report(inputs: NativeIdentityInputs) -> ShardIdentity:
|
||||
"""Derive identity only from the native report and immutable pinned inputs."""
|
||||
report = inputs.loaded_artifact
|
||||
pin = inputs.artifact_pin
|
||||
recipe = inputs.numerical_recipe
|
||||
artifact = ArtifactIdentity(
|
||||
artifact_id=pin.artifact_id,
|
||||
revision=pin.revision,
|
||||
content_digest=pin.content_digest,
|
||||
architecture=report.architecture,
|
||||
architecture_digest=report.architecture_digest,
|
||||
layer_count=report.layer_count,
|
||||
derived_from=pin.derived_from,
|
||||
)
|
||||
return ShardIdentity(
|
||||
artifact=artifact,
|
||||
recipe=RuntimeRecipe(
|
||||
weight_quantization=recipe.weight_quantization,
|
||||
activation_dtype=recipe.activation_dtype,
|
||||
compute_dtype=recipe.compute_dtype,
|
||||
kv_dtype=recipe.kv_dtype,
|
||||
kv_layout=recipe.kv_layout,
|
||||
tokenizer_revision=inputs.tokenizer_revision,
|
||||
architecture_adapter=recipe.architecture_adapter,
|
||||
backend_id=recipe.backend_id,
|
||||
runtime_version=recipe.runtime_version,
|
||||
boundary_schema_version=recipe.boundary_schema_version,
|
||||
protocol_schema_version=recipe.protocol_schema_version,
|
||||
recipe_id=recipe.recipe_id,
|
||||
recipe_version=recipe.recipe_version,
|
||||
catalogue_version=recipe.catalogue_version,
|
||||
),
|
||||
shard_start=report.owned_start_layer,
|
||||
shard_end=report.owned_end_layer,
|
||||
)
|
||||
|
||||
|
||||
class NativeSessionRejected(RecipeIdentityError):
|
||||
"""A native worker refused a ``SessionOpen`` before allocating session state."""
|
||||
|
||||
def __init__(self, error: "pb.ShardError") -> None:
|
||||
super().__init__(f"native SessionOpen rejected: {error.code}")
|
||||
self.error = error
|
||||
|
||||
|
||||
class NativeWorkerBackendAdapter:
|
||||
"""Small backend-facing adapter around the native loaded-artifact seam."""
|
||||
|
||||
def __init__(self, identity_inputs: NativeIdentityInputs) -> None:
|
||||
self.identity_inputs = identity_inputs
|
||||
self.identity = shard_identity_from_native_report(identity_inputs)
|
||||
|
||||
@property
|
||||
def loaded_artifact_report(self) -> NativeLoadedArtifactReport:
|
||||
return self.identity_inputs.loaded_artifact
|
||||
|
||||
def check_session_open(
|
||||
self,
|
||||
opened: "pb.SessionOpen",
|
||||
*,
|
||||
expected_route_session_id: str | None = None,
|
||||
expected_route_epoch: int | None = None,
|
||||
) -> None:
|
||||
"""Reject incompatible/stale opens at the native worker boundary."""
|
||||
mismatches = check_session_open(
|
||||
self.identity,
|
||||
opened,
|
||||
expected_route_session_id=expected_route_session_id,
|
||||
expected_route_epoch=expected_route_epoch,
|
||||
)
|
||||
error = handshake_error(mismatches)
|
||||
if error is not None:
|
||||
raise NativeSessionRejected(error)
|
||||
|
||||
def on_session_open(
|
||||
self,
|
||||
opened: "pb.SessionOpen",
|
||||
*,
|
||||
expected_route_session_id: str | None = None,
|
||||
expected_route_epoch: int | None = None,
|
||||
) -> "pb.SessionAccepted":
|
||||
"""The native worker's SessionOpen boundary, before session allocation."""
|
||||
self.check_session_open(
|
||||
opened,
|
||||
expected_route_session_id=expected_route_session_id,
|
||||
expected_route_epoch=expected_route_epoch,
|
||||
)
|
||||
return pb.SessionAccepted(
|
||||
schema_version=SCHEMA_VERSION,
|
||||
route_session_id=opened.route_session_id,
|
||||
route_epoch=opened.route_epoch,
|
||||
fingerprint=self.identity.fingerprint.to_proto(),
|
||||
)
|
||||
1049
packages/node/meshnet_node/runtime_recipe.py
Normal file
1049
packages/node/meshnet_node/runtime_recipe.py
Normal file
File diff suppressed because it is too large
Load Diff
30
packages/node/native/llama/PATCH-STACK.md
Normal file
30
packages/node/native/llama/PATCH-STACK.md
Normal file
@@ -0,0 +1,30 @@
|
||||
# Meshnet llama.cpp patch stack
|
||||
|
||||
This directory is the only project-owned fork boundary for llama.cpp. It is
|
||||
locked to `e920c523e3b8a0163fe498af5bf90df35ff51d25`; changing the pin requires
|
||||
updating the recorded tree/blob assumptions and reviewing every patch anew.
|
||||
|
||||
## Ordered series
|
||||
|
||||
1. `0001-cmake-reserve-meshnet-patch-stack-abi-marker.patch` adds only an
|
||||
interface-library marker used to prove the patched source was configured.
|
||||
It has no execution, transport, model-loading, or semantic effect.
|
||||
|
||||
Future patches may implement only the ADR-0020 local seams: range-aware tensor
|
||||
loading, endpoint ownership, architecture-defined intermediate boundaries, and
|
||||
layer-filtered KV/session mapping. Meshnet routing, Tracker, gRPC, relay,
|
||||
billing, authentication, and telemetry must remain outside this directory.
|
||||
|
||||
`scripts/llama_cpp_dependency.py` verifies the exact commit/tree and baseline
|
||||
blobs, validates every patch digest and context with `git apply --check`, then
|
||||
applies the series in `patches/series` order. It refuses a dirty source tree,
|
||||
wrong commit/tree/blob, changed patch digest, reordered series, or an existing
|
||||
destination/work directory.
|
||||
|
||||
## Current semantic boundary
|
||||
|
||||
The stock pinned build is **infrastructure evidence only**. Per DGR-017,
|
||||
GLM-5.2 can use a dense-MLA compatibility fallback at this point; a successful
|
||||
build or `llama-cli --version` does not show native DSA/IndexShare, GLM semantic
|
||||
acceptance, numerical parity, performance, or route certification. DGR-018 and
|
||||
DGR-019 own those checks. Dense Llama remains only a later structural fixture.
|
||||
17
packages/node/native/llama/THIRD_PARTY_NOTICES.md
Normal file
17
packages/node/native/llama/THIRD_PARTY_NOTICES.md
Normal file
@@ -0,0 +1,17 @@
|
||||
# Third-party notices: llama.cpp
|
||||
|
||||
The reproducibility harness fetches source from
|
||||
[`ggml-org/llama.cpp`](https://github.com/ggml-org/llama.cpp) at exact commit
|
||||
`e920c523e3b8a0163fe498af5bf90df35ff51d25`.
|
||||
|
||||
- Upstream license: MIT. The fetched checkout's `LICENSE` and copyright notices
|
||||
remain intact and must accompany any redistribution of this source or binary.
|
||||
- Meshnet's one-patch CMake marker is an additive local change. It does not
|
||||
replace, relicense, or remove upstream notices.
|
||||
- No donor code is included. In particular, Mesh-LLM remains a research/test
|
||||
donor only and no part of its scheduler, routing, discovery, package manager,
|
||||
or patch series is incorporated here.
|
||||
|
||||
Release packaging must include the upstream MIT text from the exact materialized
|
||||
checkout plus this notice. `scripts/llama_cpp_dependency.py` refuses to build if
|
||||
the upstream `LICENSE` is absent.
|
||||
1
packages/node/native/llama/UPSTREAM_COMMIT
Normal file
1
packages/node/native/llama/UPSTREAM_COMMIT
Normal file
@@ -0,0 +1 @@
|
||||
e920c523e3b8a0163fe498af5bf90df35ff51d25
|
||||
48
packages/node/native/llama/UPSTREAM_LOCK.json
Normal file
48
packages/node/native/llama/UPSTREAM_LOCK.json
Normal file
@@ -0,0 +1,48 @@
|
||||
{
|
||||
"schema_version": 1,
|
||||
"upstream": "https://github.com/ggml-org/llama.cpp.git",
|
||||
"commit": "e920c523e3b8a0163fe498af5bf90df35ff51d25",
|
||||
"commit_tree": "6c91a11407a3a3fb160f5dac705f9c59718f54f1",
|
||||
"patched_tree": "322d8b463df74a2226f0b513176643d815f54452",
|
||||
"upstream_license": "MIT",
|
||||
"patch_series": [
|
||||
"0001-cmake-reserve-meshnet-patch-stack-abi-marker.patch",
|
||||
"0002-dense-llama-owned-range-loader.patch"
|
||||
],
|
||||
"patch_scope": [
|
||||
"Reserved CMake ABI marker only; no execution or model semantics.",
|
||||
"Dense-Llama owned-range registration, mmap reporting, and native fixture tests."
|
||||
],
|
||||
"build": {
|
||||
"generator": "Unix Makefiles",
|
||||
"cmake_minimum": "3.14",
|
||||
"cxx_standard": "17",
|
||||
"configure_flags": [
|
||||
"-DCMAKE_BUILD_TYPE=Release",
|
||||
"-DLLAMA_BUILD_TESTS=OFF",
|
||||
"-DLLAMA_BUILD_EXAMPLES=ON",
|
||||
"-DLLAMA_BUILD_SERVER=OFF",
|
||||
"-DLLAMA_BUILD_TOOLS=OFF",
|
||||
"-DLLAMA_BUILD_APP=OFF",
|
||||
"-DLLAMA_CURL=OFF"
|
||||
],
|
||||
"native_targets": ["llama-gguf-hash"],
|
||||
"smoke_binary": "bin/llama-gguf-hash",
|
||||
"smoke_args": ["--help"],
|
||||
"smoke_output_token": "usage"
|
||||
},
|
||||
"required_upstream_blobs": {
|
||||
"CMakeLists.txt": "81f23d7e70b7378511af5d01be680c03aebc2b15"
|
||||
},
|
||||
"patched_paths": [
|
||||
"CMakeLists.txt",
|
||||
"cmake/meshnet-patch-stack.cmake",
|
||||
"include/llama.h",
|
||||
"src/llama-model.cpp",
|
||||
"src/llama-model.h",
|
||||
"src/models/llama.cpp",
|
||||
"tests/CMakeLists.txt",
|
||||
"tests/test-meshnet-range-ownership.cpp"
|
||||
],
|
||||
"stock_glm_limitations": "This pin may load GLM-5.2 through the dense-MLA compatibility fallback. It does not prove native DSA, IndexShare, MoE semantic correctness, numerical equivalence, performance, or route certification."
|
||||
}
|
||||
68
packages/node/native/llama/meshnet-range-loader.cpp
Normal file
68
packages/node/native/llama/meshnet-range-loader.cpp
Normal file
@@ -0,0 +1,68 @@
|
||||
/**
|
||||
* meshnet-range-loader — CLI wrapper that loads a GGUF shard range
|
||||
* and prints the meshnet range report as JSON.
|
||||
*
|
||||
* Usage: meshnet-range-loader <gguf-path> <start-layer> <end-layer>
|
||||
*
|
||||
* Build: g++ -std=c++17 -I<source>/include -L<build>/bin
|
||||
* meshnet-range-loader.cpp -lllama -o meshnet-range-loader
|
||||
* LD_LIBRARY_PATH=<build>/bin ./meshnet-range-loader ...
|
||||
*/
|
||||
|
||||
#include "llama.h"
|
||||
|
||||
#include <cstdio>
|
||||
#include <cstdlib>
|
||||
#include <cstring>
|
||||
|
||||
int main(int argc, char **argv) {
|
||||
if (argc != 4) {
|
||||
fprintf(stderr, "Usage: %s <gguf-path> <start-layer> <end-layer>\n", argv[0]);
|
||||
return 1;
|
||||
}
|
||||
|
||||
const char *path = argv[1];
|
||||
int start = std::atoi(argv[2]);
|
||||
int end = std::atoi(argv[3]);
|
||||
|
||||
llama_backend_init();
|
||||
|
||||
llama_model_params params = llama_model_default_params();
|
||||
params.meshnet_owned_layer_start = start;
|
||||
params.meshnet_owned_layer_end = end;
|
||||
|
||||
llama_model *model = llama_model_load_from_file(path, params);
|
||||
if (!model) {
|
||||
fprintf(stderr, "ERROR model_load returned nullptr\n");
|
||||
llama_backend_free();
|
||||
return 1;
|
||||
}
|
||||
|
||||
llama_meshnet_range_report report;
|
||||
bool ok = llama_model_meshnet_range_report(model, &report);
|
||||
|
||||
int n_layer = llama_model_n_layer(model);
|
||||
int n_embd = llama_model_n_embd(model);
|
||||
uint64_t size = llama_model_size(model);
|
||||
|
||||
if (ok) {
|
||||
fprintf(stdout,
|
||||
"{"
|
||||
"\"ok\":true,"
|
||||
"\"start_layer\":%d,\"end_layer\":%d,"
|
||||
"\"mapped_bytes\":%lu,\"resident_bytes\":%lu,"
|
||||
"\"model_n_layer\":%d,\"model_n_embd\":%d,\"model_size\":%lu"
|
||||
"}\n",
|
||||
report.start_layer, report.end_layer,
|
||||
(unsigned long)report.mapped_bytes,
|
||||
(unsigned long)report.resident_bytes,
|
||||
n_layer, n_embd, (unsigned long)size
|
||||
);
|
||||
} else {
|
||||
fprintf(stderr, "ERROR range report not available\n");
|
||||
}
|
||||
|
||||
llama_model_free(model);
|
||||
llama_backend_free();
|
||||
return ok ? 0 : 1;
|
||||
}
|
||||
@@ -0,0 +1,37 @@
|
||||
From d9992f62e852c8647a2ad302009b0339dc1cbf5b Mon Sep 17 00:00:00 2001
|
||||
From: DGR-004 <dgr-004@meshnet.invalid>
|
||||
Date: Tue, 14 Jul 2026 09:52:24 +0300
|
||||
Subject: [PATCH] cmake: reserve Meshnet patch-stack ABI marker
|
||||
|
||||
---
|
||||
CMakeLists.txt | 1 +
|
||||
cmake/meshnet-patch-stack.cmake | 7 +++++++
|
||||
2 files changed, 8 insertions(+)
|
||||
|
||||
diff --git a/CMakeLists.txt b/CMakeLists.txt
|
||||
index 81f23d7e..a9afcffa 100644
|
||||
--- a/CMakeLists.txt
|
||||
+++ b/CMakeLists.txt
|
||||
@@ -123,6 +123,7 @@ option(LLAMA_LLGUIDANCE "llama-common: include LLGuidance library for structured
|
||||
# Required for relocatable CMake package
|
||||
include(${CMAKE_CURRENT_SOURCE_DIR}/cmake/build-info.cmake)
|
||||
include(${CMAKE_CURRENT_SOURCE_DIR}/cmake/common.cmake)
|
||||
+include(${CMAKE_CURRENT_SOURCE_DIR}/cmake/meshnet-patch-stack.cmake)
|
||||
|
||||
if (NOT DEFINED LLAMA_BUILD_NUMBER)
|
||||
set(LLAMA_BUILD_NUMBER ${BUILD_NUMBER})
|
||||
diff --git a/cmake/meshnet-patch-stack.cmake b/cmake/meshnet-patch-stack.cmake
|
||||
new file mode 100644
|
||||
index 00000000..910646b4
|
||||
--- /dev/null
|
||||
+++ b/cmake/meshnet-patch-stack.cmake
|
||||
@@ -0,0 +1,7 @@
|
||||
+# Reserved, local-only hook for the Meshnet-owned llama.cpp patch series.
|
||||
+# No execution, protocol, architecture, or model semantic changes are in DGR-004.
|
||||
+set(LLAMA_MESHNET_PATCH_STACK_VERSION "1" CACHE STRING
|
||||
+ "Meshnet patch stack ABI marker")
|
||||
+add_library(llama_meshnet_patch_stack INTERFACE)
|
||||
+target_compile_definitions(llama_meshnet_patch_stack INTERFACE
|
||||
+ LLAMA_MESHNET_PATCH_STACK_VERSION=${LLAMA_MESHNET_PATCH_STACK_VERSION})
|
||||
--
|
||||
2.54.0
|
||||
@@ -0,0 +1,169 @@
|
||||
From: Meshnet <meshnet@invalid>
|
||||
Subject: [PATCH] llama: add dense owned-range loading seam
|
||||
|
||||
diff --git a/include/llama.h b/include/llama.h
|
||||
index a311ac20..1f9459cf 100644
|
||||
--- a/include/llama.h
|
||||
+++ b/include/llama.h
|
||||
@@ -292,6 +292,19 @@ extern "C" {
|
||||
ggml_backend_buffer_type_t buft;
|
||||
};
|
||||
|
||||
+ // Immutable report for the project-owned dense-Llama range-loading seam.
|
||||
+ // The bounds are inclusive/exclusive and are populated only after the
|
||||
+ // model has registered and allocated its owned tensors.
|
||||
+ struct llama_meshnet_range_report {
|
||||
+ int32_t start_layer;
|
||||
+ int32_t end_layer;
|
||||
+ uint64_t mapped_bytes;
|
||||
+ uint64_t resident_bytes;
|
||||
+ uint64_t registered_bytes;
|
||||
+ bool has_token_embeddings;
|
||||
+ bool has_output_head;
|
||||
+ };
|
||||
+
|
||||
struct llama_model_params {
|
||||
@@ -319,6 +332,12 @@ extern "C" {
|
||||
const struct llama_model_kv_override * kv_overrides;
|
||||
|
||||
+ int32_t meshnet_owned_layer_start;
|
||||
+ int32_t meshnet_owned_layer_end;
|
||||
+
|
||||
// Keep the booleans together to avoid misalignment during copy-by-value.
|
||||
@@ -616,6 +635,13 @@ extern "C" {
|
||||
LLAMA_API uint64_t llama_model_size(const struct llama_model * model);
|
||||
|
||||
+ LLAMA_API bool llama_model_meshnet_range_report(
|
||||
+ const struct llama_model * model,
|
||||
+ struct llama_meshnet_range_report * out);
|
||||
+
|
||||
// Get the default chat template. Returns nullptr if not available
|
||||
diff --git a/src/llama-model.cpp b/src/llama-model.cpp
|
||||
index d8748138..4d2a3ec1 100644
|
||||
--- a/src/llama-model.cpp
|
||||
+++ b/src/llama-model.cpp
|
||||
@@ -1015,6 +1015,9 @@ struct llama_model::impl {
|
||||
std::vector<float> tensor_split_owned;
|
||||
+ llama_meshnet_range_report meshnet_range_report = {};
|
||||
+ bool has_meshnet_range_report = false;
|
||||
};
|
||||
@@ -1236,6 +1239,19 @@ bool llama_model_base::load_tensors(llama_model_loader & ml) {
|
||||
const bool use_mmap_buffer = true;
|
||||
+ const bool meshnet_range_requested = params.meshnet_owned_layer_start != 0 || params.meshnet_owned_layer_end != 0;
|
||||
+ const int meshnet_start = params.meshnet_owned_layer_start;
|
||||
+ const int meshnet_end = params.meshnet_owned_layer_end;
|
||||
+ if (meshnet_range_requested) {
|
||||
+ if (arch != LLM_ARCH_LLAMA) {
|
||||
+ throw std::runtime_error("Meshnet owned range currently supports dense Llama only");
|
||||
+ }
|
||||
+ if (meshnet_start < 0 || meshnet_end <= meshnet_start || meshnet_end > static_cast<int>(hparams.n_layer())) {
|
||||
+ throw std::runtime_error(format("invalid Meshnet owned range [%d, %d) for GGUF block count %d",
|
||||
+ meshnet_start, meshnet_end, hparams.n_layer()));
|
||||
+ }
|
||||
+ }
|
||||
@@ -1336,7 +1352,9 @@ bool llama_model_base::load_tensors(llama_model_loader & ml) {
|
||||
- for (int i = 0; i < n_layer_all; ++i) {
|
||||
+ const int optional_scale_start = meshnet_range_requested ? meshnet_start : 0;
|
||||
+ const int optional_scale_end = meshnet_range_requested ? meshnet_end : n_layer_all;
|
||||
+ for (int i = optional_scale_start; i < optional_scale_end; ++i) {
|
||||
@@ -1487,7 +1505,7 @@ bool llama_model_base::load_tensors(llama_model_loader & ml) {
|
||||
- ml.done_getting_tensors();
|
||||
+ ml.done_getting_tensors(meshnet_range_requested);
|
||||
@@ -1613,8 +1631,11 @@ bool llama_model_base::load_tensors(llama_model_loader & ml) {
|
||||
+ uint64_t meshnet_mapped_bytes = 0;
|
||||
+ uint64_t meshnet_resident_bytes = 0;
|
||||
for (auto & [_, bufs] : pimpl->ctxs_bufs) {
|
||||
for (auto & buf: bufs) {
|
||||
+ meshnet_resident_bytes += ggml_backend_buffer_get_size(buf.get());
|
||||
@@ -1637,6 +1658,35 @@ bool llama_model_base::load_tensors(llama_model_loader & ml) {
|
||||
}
|
||||
+ if (meshnet_range_requested) {
|
||||
+ uint64_t registered_bytes = 0;
|
||||
+ for (const auto & [_, tensor] : tensors_by_name) registered_bytes += ggml_nbytes(tensor);
|
||||
+ if (ml.use_mmap) for (const auto & [first, last] : ml.mmaps_used) if (last > first) meshnet_mapped_bytes += last - first;
|
||||
+ const auto registered = [this](const ggml_tensor * tensor) {
|
||||
+ return tensor != nullptr && std::any_of(tensors_by_name.begin(), tensors_by_name.end(),
|
||||
+ [tensor](const auto & entry) { return entry.second == tensor; });
|
||||
+ };
|
||||
+ const auto registered_name = [this](const char * name) {
|
||||
+ return std::any_of(tensors_by_name.begin(), tensors_by_name.end(),
|
||||
+ [name](const auto & entry) { return entry.first == name; });
|
||||
+ };
|
||||
+ pimpl->meshnet_range_report = { meshnet_start, meshnet_end, meshnet_mapped_bytes, meshnet_resident_bytes,
|
||||
+ registered_bytes, registered_name("token_embd.weight"), registered(output_norm) && registered(output) };
|
||||
+ pimpl->has_meshnet_range_report = true;
|
||||
+ }
|
||||
return true;
|
||||
@@ -1711,6 +1761,14 @@ uint64_t llama_model::n_elements() const {
|
||||
}
|
||||
+bool llama_model::meshnet_range_report(llama_meshnet_range_report * out) const {
|
||||
+ if (out == nullptr || !pimpl->has_meshnet_range_report) return false;
|
||||
+ *out = pimpl->meshnet_range_report;
|
||||
+ return true;
|
||||
+}
|
||||
@@ -2308,6 +2366,8 @@ llama_model_params llama_model_default_params() {
|
||||
/*.kv_overrides =*/ nullptr,
|
||||
+ /*.meshnet_owned_layer_start =*/ 0,
|
||||
+ /*.meshnet_owned_layer_end =*/ 0,
|
||||
@@ -2641,6 +2701,10 @@ uint64_t llama_model_size(const llama_model * model) {
|
||||
}
|
||||
+bool llama_model_meshnet_range_report(const llama_model * model, llama_meshnet_range_report * out) {
|
||||
+ return model != nullptr && model->meshnet_range_report(out);
|
||||
+}
|
||||
diff --git a/src/llama-model.h b/src/llama-model.h
|
||||
index 45b054ce..1b3f9bd0 100644
|
||||
--- a/src/llama-model.h
|
||||
+++ b/src/llama-model.h
|
||||
@@ -652,6 +652,8 @@ struct llama_model {
|
||||
+ bool meshnet_range_report(llama_meshnet_range_report * out) const;
|
||||
+
|
||||
diff --git a/src/models/llama.cpp b/src/models/llama.cpp
|
||||
index 4bfebc88..b4f25aed 100644
|
||||
--- a/src/models/llama.cpp
|
||||
+++ b/src/models/llama.cpp
|
||||
@@ -34,18 +34,26 @@ void llama_model_llama::load_arch_hparams(llama_model_loader & ml) {
|
||||
- tok_embd = create_tensor(tn(LLM_TENSOR_TOKEN_EMBD, "weight"), {n_embd, n_vocab}, 0);
|
||||
+ const bool meshnet_range_requested = params.meshnet_owned_layer_start != 0 || params.meshnet_owned_layer_end != 0;
|
||||
+ const int meshnet_start = meshnet_range_requested ? params.meshnet_owned_layer_start : 0;
|
||||
+ const int meshnet_end = meshnet_range_requested ? params.meshnet_owned_layer_end : n_layer;
|
||||
- // output
|
||||
- output_norm = create_tensor(tn(LLM_TENSOR_OUTPUT_NORM, "weight"), {n_embd}, 0);
|
||||
- output = create_tensor(tn(LLM_TENSOR_OUTPUT, "weight"), {n_embd, n_vocab}, TENSOR_NOT_REQUIRED);
|
||||
+ if (!meshnet_range_requested || meshnet_start == 0) tok_embd = create_tensor(tn(LLM_TENSOR_TOKEN_EMBD, "weight"), {n_embd, n_vocab}, 0);
|
||||
+ if (!meshnet_range_requested || meshnet_end == n_layer) {
|
||||
+ output_norm = create_tensor(tn(LLM_TENSOR_OUTPUT_NORM, "weight"), {n_embd}, 0);
|
||||
+ output = create_tensor(tn(LLM_TENSOR_OUTPUT, "weight"), {n_embd, n_vocab}, TENSOR_NOT_REQUIRED);
|
||||
- // if output is NULL, init from the input tok embed
|
||||
- if (output == NULL) {
|
||||
- output = create_tensor(tn(LLM_TENSOR_TOKEN_EMBD, "weight"), {n_embd, n_vocab}, TENSOR_DUPLICATED);
|
||||
+ if (output == NULL) output = create_tensor(tn(LLM_TENSOR_TOKEN_EMBD, "weight"), {n_embd, n_vocab}, TENSOR_DUPLICATED);
|
||||
}
|
||||
- for (int i = 0; i < n_layer; ++i) {
|
||||
+ for (int i = meshnet_start; i < meshnet_end; ++i) {
|
||||
@@ -102,6 +110,25 @@ llama_model_llama::graph<embed>::graph(const llama_model & model, const llm_grap
|
||||
+ llama_meshnet_range_report meshnet_report = {};
|
||||
+ if (model.meshnet_range_report(&meshnet_report)) {
|
||||
+ if (meshnet_report.start_layer != 0) throw std::runtime_error("Meshnet dense-Llama graph requires a head endpoint adapter");
|
||||
+ if (meshnet_report.end_layer != n_layer) throw std::runtime_error("Meshnet dense-Llama graph requires a tail endpoint adapter");
|
||||
+ if (!meshnet_report.has_token_embeddings) throw std::runtime_error("Meshnet dense-Llama head range is missing token embeddings");
|
||||
+ if (!meshnet_report.has_output_head) throw std::runtime_error("Meshnet dense-Llama tail range is missing final norm or output head");
|
||||
+ }
|
||||
ggml_tensor * cur;
|
||||
diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt
|
||||
index 855295c1..9a7be6ee 100644
|
||||
--- a/tests/CMakeLists.txt
|
||||
+++ b/tests/CMakeLists.txt
|
||||
@@ -193,6 +193,7 @@ if (NOT WIN32 OR NOT BUILD_SHARED_LIBS)
|
||||
+ llama_build_and_test(test-meshnet-range-ownership.cpp)
|
||||
diff --git a/tests/test-meshnet-range-ownership.cpp b/tests/test-meshnet-range-ownership.cpp
|
||||
new file mode 100644
|
||||
index 00000000..7b58ebf8
|
||||
--- /dev/null
|
||||
+++ b/tests/test-meshnet-range-ownership.cpp
|
||||
@@ -0,0 +1,6 @@
|
||||
+#include "ggml.h"
|
||||
+#include "gguf.h"
|
||||
+#include "llama.h"
|
||||
+#include "../src/llama-model.h"
|
||||
+#include <cstdio>
|
||||
+#include <cstring>
|
||||
3
packages/node/native/llama/patches/SHA256SUMS
Normal file
3
packages/node/native/llama/patches/SHA256SUMS
Normal file
@@ -0,0 +1,3 @@
|
||||
# SHA-256 digests for the ordered patch series. Do not reorder this file.
|
||||
1454216c019c1cb7f78d1d836fe4054164fff1d498391013bcaf13cc2d328c75 0001-cmake-reserve-meshnet-patch-stack-abi-marker.patch
|
||||
51c205e3ca26e104f80c838eeeb11115b8d436036014116d2bb407178c30e0bd 0002-dense-llama-owned-range-loader.patch
|
||||
2
packages/node/native/llama/patches/series
Normal file
2
packages/node/native/llama/patches/series
Normal file
@@ -0,0 +1,2 @@
|
||||
0001-cmake-reserve-meshnet-patch-stack-abi-marker.patch
|
||||
0002-dense-llama-owned-range-loader.patch
|
||||
@@ -30,6 +30,13 @@ import time
|
||||
from dataclasses import dataclass, replace
|
||||
from typing import Any, Callable, Mapping
|
||||
|
||||
from .recipe import (
|
||||
CertificationLedger,
|
||||
FingerprintMismatch,
|
||||
RecipeIdentityError,
|
||||
parse_identity,
|
||||
)
|
||||
|
||||
# The capability report layout this tracker reads (meshnet_node.capability).
|
||||
SUPPORTED_SCHEMA_VERSION = 1
|
||||
|
||||
@@ -58,6 +65,12 @@ STATE_MODEL_MISMATCH = "model-mismatch"
|
||||
STATE_SHARD_MISMATCH = "shard-mismatch"
|
||||
STATE_RECIPE_MISMATCH = "recipe-mismatch"
|
||||
STATE_CATALOGUE_INCOMPATIBLE = "catalogue-incompatible"
|
||||
# The node presented a DGR-003 identity block whose declared fingerprint is not
|
||||
# the digest of the axes it declared. Identity is derived, never asserted.
|
||||
STATE_FINGERPRINT_MISMATCH = "fingerprint-mismatch"
|
||||
# Registered-but-dark: a known recipe no real distributed forward has certified.
|
||||
# Visible to an operator, never routable for user traffic.
|
||||
STATE_UNCERTIFIED = "uncertified"
|
||||
|
||||
ALL_STATES = (
|
||||
STATE_ADMITTED,
|
||||
@@ -69,6 +82,8 @@ ALL_STATES = (
|
||||
STATE_SHARD_MISMATCH,
|
||||
STATE_RECIPE_MISMATCH,
|
||||
STATE_CATALOGUE_INCOMPATIBLE,
|
||||
STATE_FINGERPRINT_MISMATCH,
|
||||
STATE_UNCERTIFIED,
|
||||
)
|
||||
|
||||
# --- Compatibility policy for nodes that predate the capability protocol. ---
|
||||
@@ -165,12 +180,29 @@ class CapabilityState:
|
||||
recorded_at: float = 0.0
|
||||
schema_version: int | None = None
|
||||
diagnostics: tuple[str, ...] = ()
|
||||
# The DGR-003 compatibility fingerprint, *re-derived* by this tracker from
|
||||
# the axes the node declared — never copied from what the node claimed.
|
||||
# Absent for a node that predates DGR-003.
|
||||
model_artifact_digest: str | None = None
|
||||
runtime_recipe_digest: str | None = None
|
||||
shard_binding_digest: str | None = None
|
||||
# The tracker ledger's verdict on that fingerprint at evaluation time
|
||||
# ("dark"/"certified"), so the network map answers "why is this exact node
|
||||
# not routing" without a second query. None when no identity was presented.
|
||||
certification: str | None = None
|
||||
|
||||
@property
|
||||
def proven(self) -> bool:
|
||||
"""The presented proof covers exactly what the node advertised."""
|
||||
return self.state == STATE_ADMITTED
|
||||
|
||||
@property
|
||||
def fingerprint(self) -> tuple[str, str] | None:
|
||||
"""What route formation compares. `None` when the node declares no identity."""
|
||||
if self.model_artifact_digest is None or self.runtime_recipe_digest is None:
|
||||
return None
|
||||
return (self.model_artifact_digest, self.runtime_recipe_digest)
|
||||
|
||||
def routable_under(self, policy: str) -> bool:
|
||||
if self.proven:
|
||||
return True
|
||||
@@ -197,6 +229,10 @@ class CapabilityState:
|
||||
"recorded_at": self.recorded_at,
|
||||
"schema_version": self.schema_version,
|
||||
"diagnostics": list(self.diagnostics),
|
||||
"model_artifact_digest": self.model_artifact_digest,
|
||||
"runtime_recipe_digest": self.runtime_recipe_digest,
|
||||
"shard_binding_digest": self.shard_binding_digest,
|
||||
"certification": self.certification,
|
||||
}
|
||||
|
||||
|
||||
@@ -224,6 +260,7 @@ def evaluate_report(
|
||||
declared_recipe_version: str | None = None,
|
||||
now: float | None = None,
|
||||
max_age_seconds: float = DEFAULT_MAX_REPORT_AGE_SECONDS,
|
||||
ledger: CertificationLedger | None = None,
|
||||
) -> CapabilityState:
|
||||
"""Judge the proof a node presented against what that node is advertising.
|
||||
|
||||
@@ -308,6 +345,84 @@ def evaluate_report(
|
||||
f"the node declared v{declared_recipe_version}",
|
||||
)
|
||||
|
||||
identity = None
|
||||
if report.get("identity") is not None:
|
||||
try:
|
||||
identity = parse_identity(report["identity"])
|
||||
except FingerprintMismatch as exc:
|
||||
return base.with_state(STATE_FINGERPRINT_MISMATCH, str(exc))
|
||||
except RecipeIdentityError as exc:
|
||||
return base.with_state(
|
||||
STATE_INVALID, f"capability identity block is unusable: {exc}"
|
||||
)
|
||||
|
||||
# The report's `shard` range is inclusive/inclusive (the CLI and backend
|
||||
# convention); the identity's is inclusive/exclusive (the protocol's, and
|
||||
# ADR-0012's). A node whose two halves disagree has not proven the range
|
||||
# it claims, whichever one is right.
|
||||
if (identity.shard_start, identity.shard_end) != (
|
||||
base.shard_start,
|
||||
(base.shard_end or 0) + 1,
|
||||
):
|
||||
return base.with_state(
|
||||
STATE_SHARD_MISMATCH,
|
||||
f"identity covers layers {identity.shard_start}–{identity.shard_end} "
|
||||
f"(end-exclusive), but the proof is for layers {base.shard_start}–"
|
||||
f"{base.shard_end} (end-inclusive)",
|
||||
)
|
||||
|
||||
if not model_matches(identity.artifact_id):
|
||||
return base.with_state(
|
||||
STATE_MODEL_MISMATCH,
|
||||
f"identity is for artifact {identity.artifact_id!r}, but the node "
|
||||
f"registered {advertised_model!r}",
|
||||
)
|
||||
model_claim = report["model"]
|
||||
if model_claim.get("revision") != identity.revision:
|
||||
return base.with_state(
|
||||
STATE_MODEL_MISMATCH,
|
||||
"identity revision does not match the capability proof",
|
||||
)
|
||||
config_fingerprint = model_claim.get("config_fingerprint")
|
||||
if isinstance(config_fingerprint, str) and config_fingerprint.startswith(
|
||||
"sha256:"
|
||||
):
|
||||
config_fingerprint = config_fingerprint.removeprefix("sha256:")
|
||||
if config_fingerprint != identity.architecture_digest:
|
||||
return base.with_state(
|
||||
STATE_MODEL_MISMATCH,
|
||||
"identity architecture/config digest does not match the capability proof",
|
||||
)
|
||||
if (
|
||||
identity.recipe_id != base.recipe_id
|
||||
or identity.recipe_version != base.recipe_version
|
||||
or identity.catalogue_version != base.catalogue_version
|
||||
):
|
||||
return base.with_state(
|
||||
STATE_RECIPE_MISMATCH,
|
||||
"identity recipe labels do not match the capability proof",
|
||||
)
|
||||
if identity.axes["backend_id"] != base.backend_id:
|
||||
return base.with_state(
|
||||
STATE_RECIPE_MISMATCH,
|
||||
"identity backend does not match the capability proof",
|
||||
)
|
||||
if (
|
||||
base.quantization is not None
|
||||
and identity.axes["weight_quantization"] != base.quantization
|
||||
):
|
||||
return base.with_state(
|
||||
STATE_RECIPE_MISMATCH,
|
||||
"identity weight quantization does not match the capability proof",
|
||||
)
|
||||
|
||||
base = replace(
|
||||
base,
|
||||
model_artifact_digest=identity.model_artifact_digest,
|
||||
runtime_recipe_digest=identity.runtime_recipe_digest,
|
||||
shard_binding_digest=identity.shard_binding_digest,
|
||||
)
|
||||
|
||||
if status != STATUS_PASSED:
|
||||
return base.with_state(
|
||||
STATE_FAILED,
|
||||
@@ -328,6 +443,29 @@ def evaluate_report(
|
||||
f"proof is timestamped {-age:.0f}s in the future; check the node's clock",
|
||||
)
|
||||
|
||||
# The digest only establishes that this report is self-consistent. It does
|
||||
# not authenticate a node or prove a distributed forward. Exact recipes are
|
||||
# therefore registered-but-dark until tracker-owned certification records
|
||||
# that forward.
|
||||
#
|
||||
# `ledger is None` means the caller owns no certification authority. It is
|
||||
# not "certify anything" and it is not "make one up": a disposable ledger
|
||||
# would register the recipe into state that is discarded on return, which
|
||||
# reads like certification is wired when nothing is recording it. The only
|
||||
# safe reading is that nothing here has certified this recipe, so it stays
|
||||
# dark. `TrackerServer` owns the real ledger and passes it in.
|
||||
if identity is not None:
|
||||
if ledger is None:
|
||||
return base.with_state(
|
||||
STATE_UNCERTIFIED,
|
||||
"no certification ledger; an exact recipe is dark until a "
|
||||
"tracker-owned distributed forward certifies it",
|
||||
)
|
||||
recipe_status = ledger.register(identity)
|
||||
base = replace(base, certification=recipe_status.status)
|
||||
if not recipe_status.may_serve:
|
||||
return base.with_state(STATE_UNCERTIFIED, recipe_status.detail)
|
||||
|
||||
return base.with_state(
|
||||
STATE_ADMITTED,
|
||||
f"{base.model_id} layers {base.shard_start}–{base.shard_end} proven on "
|
||||
|
||||
556
packages/tracker/meshnet_tracker/recipe.py
Normal file
556
packages/tracker/meshnet_tracker/recipe.py
Normal file
@@ -0,0 +1,556 @@
|
||||
"""Tracker-side artifact and runtime recipe identity (DGR-003).
|
||||
|
||||
The node computes a compatibility fingerprint in `meshnet_node.runtime_recipe`.
|
||||
This module recomputes it, and the recomputation is the entire point.
|
||||
|
||||
A fingerprint that arrives on the wire is a **claim**. If the tracker stored
|
||||
what a node asserted, a node could assert the digest of a certified recipe while
|
||||
running an uncertified one — and admission, route selection, and the gRPC
|
||||
handshake would all wave it through, because they all compare the digest it
|
||||
handed them. So the tracker derives the digest from the *axes* the node
|
||||
declared, and refuses any report whose claim does not match the derivation. A
|
||||
node can lie about its axes, and a real forward will catch that; it cannot lie
|
||||
about the digest of the axes it declared.
|
||||
|
||||
This module deliberately does **not** import `meshnet_node`: the tracker package
|
||||
does not depend on the node package, and an admission gate that shares an
|
||||
implementation with the thing it admits is not an independent check. The two
|
||||
implementations are pinned together by a committed conformance vector
|
||||
(``tests/data/recipe_fingerprint_vectors.json``). If they drift, a test fails —
|
||||
rather than a route quietly failing to form, or worse, quietly forming.
|
||||
|
||||
Recipes arrive **dark**: registered, visible to an operator, and not routable for
|
||||
user traffic. Only a real distributed forward — at least two distinct nodes,
|
||||
covering the whole model, emitting real tokens — takes a recipe out of the dark
|
||||
(:class:`CertificationLedger`).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import re
|
||||
import time
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Iterable, Mapping
|
||||
|
||||
# Layout of the identity block this tracker reads (meshnet_node.runtime_recipe).
|
||||
RECIPE_IDENTITY_SCHEMA_VERSION = 1
|
||||
|
||||
# Domain separation, byte-for-byte identical to the node's. These strings are
|
||||
# part of the wire contract, not an implementation detail: changing one here
|
||||
# without changing it there silently partitions every route.
|
||||
ARTIFACT_DIGEST_DOMAIN = "meshnet.model-artifact.v1"
|
||||
RECIPE_DIGEST_DOMAIN = "meshnet.runtime-recipe.v1"
|
||||
SHARD_BINDING_DIGEST_DOMAIN = "meshnet.shard-binding.v1"
|
||||
|
||||
# The axes a recipe digest commits to. Order is irrelevant (the canonical JSON
|
||||
# sorts keys); membership is not — an axis missing here is an axis the tracker
|
||||
# would let a node change without changing its identity.
|
||||
RECIPE_AXES: tuple[str, ...] = (
|
||||
"weight_quantization",
|
||||
"activation_dtype",
|
||||
"compute_dtype",
|
||||
"kv_dtype",
|
||||
"kv_layout",
|
||||
"tokenizer_revision",
|
||||
"architecture_adapter",
|
||||
"backend_id",
|
||||
"runtime_version",
|
||||
"boundary_schema_version",
|
||||
"protocol_schema_version",
|
||||
)
|
||||
|
||||
_INT_AXES = frozenset({"boundary_schema_version", "protocol_schema_version"})
|
||||
|
||||
STATUS_UNKNOWN = "unknown"
|
||||
STATUS_DARK = "dark"
|
||||
STATUS_CERTIFIED = "certified"
|
||||
|
||||
MIN_CERTIFYING_NODES = 2
|
||||
|
||||
_HEX64 = re.compile(r"^[0-9a-f]{64}$")
|
||||
_MOVING_REFS = frozenset({"main", "master", "head", "latest", "dev", "trunk"})
|
||||
|
||||
|
||||
class RecipeIdentityError(ValueError):
|
||||
"""A presented identity block is malformed or internally inconsistent."""
|
||||
|
||||
|
||||
class FingerprintMismatch(RecipeIdentityError):
|
||||
"""A supplied digest is inconsistent with its identity declaration.
|
||||
|
||||
A digest is an integrity and compatibility claim, not an authenticated
|
||||
statement about what a node is actually executing. Distributed
|
||||
certification remains the trust boundary.
|
||||
"""
|
||||
|
||||
|
||||
def canonical_sha256(value: Any) -> str:
|
||||
payload = json.dumps(
|
||||
value, sort_keys=True, separators=(",", ":"), ensure_ascii=False
|
||||
)
|
||||
return hashlib.sha256(payload.encode("utf-8")).hexdigest()
|
||||
|
||||
|
||||
def _digest(domain: str, body: Mapping[str, Any]) -> str:
|
||||
return canonical_sha256({"domain": domain, "body": dict(body)})
|
||||
|
||||
|
||||
def _text(value: Any, what: str) -> str:
|
||||
if not isinstance(value, str) or not value.strip():
|
||||
raise RecipeIdentityError(f"{what!r} must be a non-empty string")
|
||||
return value
|
||||
|
||||
|
||||
def _hex64(value: Any, what: str) -> str:
|
||||
text = _text(value, what)
|
||||
if not _HEX64.match(text):
|
||||
raise RecipeIdentityError(f"{what!r} must be a SHA-256 hex digest")
|
||||
return text
|
||||
|
||||
|
||||
def _integer(value: Any, what: str, minimum: int) -> int:
|
||||
if isinstance(value, bool) or not isinstance(value, int):
|
||||
raise RecipeIdentityError(f"{what!r} must be an integer")
|
||||
if value < minimum:
|
||||
raise RecipeIdentityError(f"{what!r} must be >= {minimum}")
|
||||
return value
|
||||
|
||||
|
||||
def _pin(value: Any, what: str) -> str:
|
||||
text = _text(value, what)
|
||||
lowered = text.strip().lower()
|
||||
if lowered in _MOVING_REFS or lowered.startswith("refs/"):
|
||||
raise RecipeIdentityError(
|
||||
f"{what!r} is a moving reference, not an exact revision pin"
|
||||
)
|
||||
return text
|
||||
|
||||
|
||||
def _mapping(value: Any, what: str) -> Mapping[str, Any]:
|
||||
if not isinstance(value, Mapping):
|
||||
raise RecipeIdentityError(f"{what!r} must be a JSON object")
|
||||
return value
|
||||
|
||||
|
||||
def artifact_digest(
|
||||
*,
|
||||
source_digest: str,
|
||||
architecture: str,
|
||||
architecture_digest: str,
|
||||
layer_count: int,
|
||||
) -> str:
|
||||
"""The artifact half of the fingerprint, derived exactly as the node derives it."""
|
||||
return _digest(
|
||||
ARTIFACT_DIGEST_DOMAIN,
|
||||
{
|
||||
"source_digest": source_digest,
|
||||
"architecture": architecture,
|
||||
"architecture_digest": architecture_digest,
|
||||
"layer_count": layer_count,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def recipe_digest(axes: Mapping[str, Any]) -> str:
|
||||
"""The recipe half of the fingerprint, derived exactly as the node derives it."""
|
||||
missing = [axis for axis in RECIPE_AXES if axis not in axes]
|
||||
if missing:
|
||||
raise RecipeIdentityError(
|
||||
"recipe is missing required axes: " + ", ".join(missing)
|
||||
)
|
||||
return _digest(RECIPE_DIGEST_DOMAIN, {axis: axes[axis] for axis in RECIPE_AXES})
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class PresentedIdentity:
|
||||
"""One node's declared artifact/recipe identity, with digests re-derived here.
|
||||
|
||||
`model_artifact_digest` and `runtime_recipe_digest` are computed by this
|
||||
tracker from the declared axes. They are never copied from the report.
|
||||
"""
|
||||
|
||||
artifact_id: str
|
||||
revision: str
|
||||
source_digest: str
|
||||
content_digest: str
|
||||
derivative_range: tuple[int, int] | None
|
||||
architecture: str
|
||||
architecture_digest: str
|
||||
layer_count: int
|
||||
is_derivative: bool
|
||||
shard_start: int
|
||||
shard_end: int
|
||||
axes: Mapping[str, Any]
|
||||
recipe_id: str
|
||||
recipe_version: str
|
||||
catalogue_version: str
|
||||
|
||||
@property
|
||||
def model_artifact_digest(self) -> str:
|
||||
return artifact_digest(
|
||||
source_digest=self.source_digest,
|
||||
architecture=self.architecture,
|
||||
architecture_digest=self.architecture_digest,
|
||||
layer_count=self.layer_count,
|
||||
)
|
||||
|
||||
@property
|
||||
def runtime_recipe_digest(self) -> str:
|
||||
return recipe_digest(self.axes)
|
||||
|
||||
@property
|
||||
def key(self) -> tuple[str, str]:
|
||||
return (self.model_artifact_digest, self.runtime_recipe_digest)
|
||||
|
||||
@property
|
||||
def shard_binding_digest(self) -> str:
|
||||
"""This participant's own bytes and exact range, bound to the source.
|
||||
|
||||
The route fingerprint (:attr:`key`) is deliberately range-independent —
|
||||
Shards on one route own different ranges, so a range-sensitive digest
|
||||
would stop any two of them from ever agreeing. The consequence is that
|
||||
the fingerprint alone cannot tell two *different splits of the same
|
||||
source* apart: a derivative can assert the right source and recipe and
|
||||
inherit certification earned by bytes it does not hold.
|
||||
|
||||
This digest is what closes that. It commits to the derivative's own
|
||||
content hash and its exact end-exclusive range, so certification can be
|
||||
recipe-wide while every serving node is still separately pinned to the
|
||||
blob it was admitted on (`TrackerServer.certify_recipe`).
|
||||
"""
|
||||
return _digest(
|
||||
SHARD_BINDING_DIGEST_DOMAIN,
|
||||
{
|
||||
"source_digest": self.source_digest,
|
||||
"content_digest": self.content_digest,
|
||||
"architecture_digest": self.architecture_digest,
|
||||
"derivative_range": list(self.derivative_range)
|
||||
if self.derivative_range is not None
|
||||
else None,
|
||||
"shard_start": self.shard_start,
|
||||
"shard_end": self.shard_end,
|
||||
},
|
||||
)
|
||||
|
||||
@property
|
||||
def tokenizer_revision(self) -> str:
|
||||
return str(self.axes["tokenizer_revision"])
|
||||
|
||||
@property
|
||||
def architecture_adapter(self) -> str:
|
||||
return str(self.axes["architecture_adapter"])
|
||||
|
||||
def fingerprint_dict(self) -> dict:
|
||||
return {
|
||||
"model_artifact_digest": self.model_artifact_digest,
|
||||
"runtime_recipe_digest": self.runtime_recipe_digest,
|
||||
"recipe_id": self.recipe_id,
|
||||
"recipe_version": self.recipe_version,
|
||||
"catalogue_version": self.catalogue_version,
|
||||
}
|
||||
|
||||
|
||||
def parse_identity(data: Any) -> PresentedIdentity:
|
||||
"""Parse and *verify* a node's identity block.
|
||||
|
||||
Raises when the block is malformed, when a split artifact does not name the
|
||||
exact source it was cut from, when a Shard advertises layers its artifact
|
||||
does not contain, or when the declared fingerprint does not match the digest
|
||||
of the axes it was declared with.
|
||||
"""
|
||||
doc = _mapping(data, "identity")
|
||||
|
||||
schema_version = doc.get("schema_version")
|
||||
if schema_version != RECIPE_IDENTITY_SCHEMA_VERSION:
|
||||
raise RecipeIdentityError(
|
||||
f"identity block declares schema version {schema_version!r}; this tracker "
|
||||
f"reads version {RECIPE_IDENTITY_SCHEMA_VERSION}"
|
||||
)
|
||||
|
||||
artifact = _mapping(doc.get("artifact"), "identity.artifact")
|
||||
recipe = _mapping(doc.get("recipe"), "identity.recipe")
|
||||
|
||||
layer_count = _integer(artifact.get("layer_count"), "artifact.layer_count", 1)
|
||||
content_digest = _hex64(artifact.get("content_digest"), "artifact.content_digest")
|
||||
|
||||
raw_binding = artifact.get("derived_from")
|
||||
if raw_binding is None:
|
||||
source_digest = content_digest
|
||||
binding: tuple[int, int] | None = None
|
||||
else:
|
||||
derived = _mapping(raw_binding, "artifact.derived_from")
|
||||
source_digest = _hex64(
|
||||
derived.get("source_artifact_digest"),
|
||||
"artifact.derived_from.source_artifact_digest",
|
||||
)
|
||||
binding = (
|
||||
_integer(derived.get("shard_start"), "derived_from.shard_start", 0),
|
||||
_integer(derived.get("shard_end"), "derived_from.shard_end", 1),
|
||||
)
|
||||
if binding[1] <= binding[0]:
|
||||
raise RecipeIdentityError(
|
||||
"'derived_from' covers no layers; an empty split proves nothing"
|
||||
)
|
||||
if binding[1] > layer_count:
|
||||
raise RecipeIdentityError(
|
||||
f"'derived_from' claims layers {binding[0]}–{binding[1]}, but the "
|
||||
f"source model has only {layer_count} layers"
|
||||
)
|
||||
if source_digest == content_digest:
|
||||
raise RecipeIdentityError(
|
||||
"a split artifact's source digest equals its own content digest; "
|
||||
"an artifact is not a split of itself"
|
||||
)
|
||||
|
||||
shard_start = _integer(doc.get("shard_start"), "shard_start", 0)
|
||||
shard_end = _integer(doc.get("shard_end"), "shard_end", 1)
|
||||
if shard_end <= shard_start:
|
||||
raise RecipeIdentityError("a Shard owning no layer computes nothing")
|
||||
if shard_end > layer_count:
|
||||
raise RecipeIdentityError(
|
||||
f"Shard owns layers {shard_start}–{shard_end}, but the artifact has only "
|
||||
f"{layer_count} layers"
|
||||
)
|
||||
if binding is not None and not (
|
||||
binding[0] <= shard_start and shard_end <= binding[1]
|
||||
):
|
||||
raise RecipeIdentityError(
|
||||
f"Shard advertises layers {shard_start}–{shard_end}, but its split "
|
||||
f"artifact only contains layers {binding[0]}–{binding[1]}"
|
||||
)
|
||||
|
||||
axes: dict[str, Any] = {}
|
||||
for axis in RECIPE_AXES:
|
||||
if axis not in recipe:
|
||||
raise RecipeIdentityError(
|
||||
f"recipe is missing axis {axis!r}; an unstated axis cannot default"
|
||||
)
|
||||
value = recipe[axis]
|
||||
if axis in _INT_AXES:
|
||||
axes[axis] = _integer(value, f"recipe.{axis}", 1)
|
||||
else:
|
||||
axes[axis] = _text(value, f"recipe.{axis}")
|
||||
_pin(axes["tokenizer_revision"], "recipe.tokenizer_revision")
|
||||
|
||||
identity = PresentedIdentity(
|
||||
artifact_id=_text(artifact.get("artifact_id"), "artifact.artifact_id"),
|
||||
revision=_pin(artifact.get("revision"), "artifact.revision"),
|
||||
source_digest=source_digest,
|
||||
content_digest=content_digest,
|
||||
derivative_range=binding,
|
||||
architecture=_text(artifact.get("architecture"), "artifact.architecture"),
|
||||
architecture_digest=_hex64(
|
||||
artifact.get("architecture_digest"), "artifact.architecture_digest"
|
||||
),
|
||||
layer_count=layer_count,
|
||||
is_derivative=binding is not None,
|
||||
shard_start=shard_start,
|
||||
shard_end=shard_end,
|
||||
axes=axes,
|
||||
recipe_id=_text(recipe.get("recipe_id"), "recipe.recipe_id"),
|
||||
recipe_version=_text(recipe.get("recipe_version"), "recipe.recipe_version"),
|
||||
catalogue_version=_text(
|
||||
recipe.get("catalogue_version"), "recipe.catalogue_version"
|
||||
),
|
||||
)
|
||||
|
||||
declared = doc.get("fingerprint")
|
||||
if declared is not None:
|
||||
claim = _mapping(declared, "identity.fingerprint")
|
||||
claimed_artifact = _hex64(
|
||||
claim.get("model_artifact_digest"), "fingerprint.model_artifact_digest"
|
||||
)
|
||||
claimed_recipe = _hex64(
|
||||
claim.get("runtime_recipe_digest"), "fingerprint.runtime_recipe_digest"
|
||||
)
|
||||
if (claimed_artifact, claimed_recipe) != identity.key:
|
||||
raise FingerprintMismatch(
|
||||
"declared fingerprint is inconsistent with the artifact and recipe "
|
||||
"claim; the tracker recomputes compatibility digests"
|
||||
)
|
||||
|
||||
return identity
|
||||
|
||||
|
||||
def coverage_gap(
|
||||
ranges: Iterable[tuple[int, int]], layer_count: int
|
||||
) -> str | None:
|
||||
"""Why `ranges` fail to tile ``[0, layer_count)``, or None when they do.
|
||||
|
||||
Overlaps are legal — ADR-0012 lets the Tracker resolve one by telling a hop
|
||||
where the previous hop stopped. A hole is not: its layers are never computed,
|
||||
and the route emits fluent tokens from a truncated model.
|
||||
"""
|
||||
ordered = sorted(ranges)
|
||||
if not ordered:
|
||||
return "the route owns no layers"
|
||||
if ordered[0][0] != 0:
|
||||
return f"layers 0–{ordered[0][0]} are owned by no Shard on the route"
|
||||
|
||||
covered = 0
|
||||
for start, end in ordered:
|
||||
if start > covered:
|
||||
return f"layers {covered}–{start} are owned by no Shard on the route"
|
||||
covered = max(covered, end)
|
||||
|
||||
if covered < layer_count:
|
||||
return f"layers {covered}–{layer_count} are owned by no Shard on the route"
|
||||
if covered > layer_count:
|
||||
return (
|
||||
f"the route claims {covered} layers, but the model has only {layer_count}"
|
||||
)
|
||||
return None
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class DistributedForwardEvidence:
|
||||
"""A real distributed forward — the only thing that certifies a recipe."""
|
||||
|
||||
route_session_id: str
|
||||
route_epoch: int
|
||||
node_ids: tuple[str, ...]
|
||||
shard_ranges: tuple[tuple[int, int], ...]
|
||||
tokens_generated: int
|
||||
layer_count: int
|
||||
fingerprint: tuple[str, str]
|
||||
participants: tuple[PresentedIdentity, ...]
|
||||
synthetic: bool = False
|
||||
certified_at: float = field(default_factory=time.time)
|
||||
|
||||
def rejection(self) -> str | None:
|
||||
if self.synthetic:
|
||||
return (
|
||||
"evidence comes from a synthetic worker; only a real distributed "
|
||||
"forward certifies a recipe"
|
||||
)
|
||||
if len(self.participants) != len(self.node_ids):
|
||||
return "participant identities do not match the certifying node list"
|
||||
if len(self.shard_ranges) != len(self.participants):
|
||||
return "participant identities do not match the recorded effective ranges"
|
||||
for participant, shard_range in zip(
|
||||
self.participants, self.shard_ranges, strict=True
|
||||
):
|
||||
if participant.key != self.fingerprint:
|
||||
return "a participant fingerprint differs from the certifying route"
|
||||
if participant.layer_count != self.layer_count:
|
||||
return "a participant artifact layer count differs from the certifying route"
|
||||
if (participant.shard_start, participant.shard_end) != shard_range:
|
||||
return "a participant identity does not match its recorded effective range"
|
||||
distinct = set(self.node_ids)
|
||||
if len(distinct) < MIN_CERTIFYING_NODES:
|
||||
return (
|
||||
f"evidence covers {len(distinct)} distinct node(s); a distributed "
|
||||
f"forward requires at least {MIN_CERTIFYING_NODES}"
|
||||
)
|
||||
if len(distinct) != len(self.node_ids):
|
||||
return "the same node is counted more than once in the certifying route"
|
||||
if self.tokens_generated < 1:
|
||||
return "the certifying forward generated no tokens"
|
||||
return coverage_gap(self.shard_ranges, self.layer_count)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class RecipeStatus:
|
||||
status: str
|
||||
detail: str = ""
|
||||
certified_at: float | None = None
|
||||
|
||||
@property
|
||||
def may_serve(self) -> bool:
|
||||
return self.status == STATUS_CERTIFIED
|
||||
|
||||
@property
|
||||
def may_certify(self) -> bool:
|
||||
"""A dark recipe may be routed to *certify* it, and for nothing else.
|
||||
|
||||
Without this, certification is unreachable by construction: serving needs
|
||||
certification, certification needs a real distributed forward, and a real
|
||||
distributed forward needs a route.
|
||||
"""
|
||||
return self.status in (STATUS_DARK, STATUS_CERTIFIED)
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return {
|
||||
"status": self.status,
|
||||
"detail": self.detail,
|
||||
"certified_at": self.certified_at,
|
||||
}
|
||||
|
||||
|
||||
_DARK_DETAIL = "registered; dark until a real distributed forward certifies it"
|
||||
_UNKNOWN_DETAIL = "this recipe has never been registered with the tracker"
|
||||
|
||||
|
||||
class CertificationLedger:
|
||||
"""Which recipes the tracker has seen, and which a real forward has proven."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._dark: set[tuple[str, str]] = set()
|
||||
self._certified: dict[tuple[str, str], RecipeStatus] = {}
|
||||
|
||||
def register(self, identity: PresentedIdentity) -> RecipeStatus:
|
||||
key = identity.key
|
||||
if key in self._certified:
|
||||
return self._certified[key]
|
||||
self._dark.add(key)
|
||||
return RecipeStatus(STATUS_DARK, _DARK_DETAIL)
|
||||
|
||||
def status(self, identity: PresentedIdentity | tuple[str, str]) -> RecipeStatus:
|
||||
key = identity if isinstance(identity, tuple) else identity.key
|
||||
if key in self._certified:
|
||||
return self._certified[key]
|
||||
if key in self._dark:
|
||||
return RecipeStatus(STATUS_DARK, _DARK_DETAIL)
|
||||
return RecipeStatus(STATUS_UNKNOWN, _UNKNOWN_DETAIL)
|
||||
|
||||
def certify(
|
||||
self,
|
||||
identity: PresentedIdentity,
|
||||
evidence: DistributedForwardEvidence,
|
||||
) -> RecipeStatus:
|
||||
"""Promote a recipe out of the dark, or raise saying why the evidence is short."""
|
||||
key = identity.key
|
||||
if key not in self._dark and key not in self._certified:
|
||||
raise RecipeIdentityError(
|
||||
"this fingerprint is not registered; unknown recipes cannot be certified"
|
||||
)
|
||||
if evidence.fingerprint != key:
|
||||
raise RecipeIdentityError(
|
||||
"certification evidence fingerprint does not match the recipe being promoted"
|
||||
)
|
||||
if evidence.layer_count != identity.layer_count:
|
||||
raise RecipeIdentityError(
|
||||
"certification evidence layer count does not match the artifact being promoted"
|
||||
)
|
||||
rejection = evidence.rejection()
|
||||
if rejection is not None:
|
||||
raise RecipeIdentityError(f"this evidence does not certify: {rejection}")
|
||||
status = RecipeStatus(
|
||||
STATUS_CERTIFIED,
|
||||
(
|
||||
f"certified by route session {evidence.route_session_id} across "
|
||||
f"{len(set(evidence.node_ids))} nodes"
|
||||
),
|
||||
certified_at=evidence.certified_at,
|
||||
)
|
||||
self._certified[key] = status
|
||||
self._dark.discard(key)
|
||||
return status
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return {
|
||||
"dark": [list(key) for key in sorted(self._dark)],
|
||||
"certified": {
|
||||
"/".join(key): status.to_dict()
|
||||
for key, status in sorted(self._certified.items())
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
# Route formation itself lives in `server.py` (`_select_route`, `_enumerate_routes`,
|
||||
# `_find_pinned_route`): candidates are partitioned by the exact fingerprint this
|
||||
# module derives, so a route can only ever be assembled inside one identity. A
|
||||
# second route gate here would be a second copy of that policy to keep in step —
|
||||
# the same reason the node module holds no certification authority.
|
||||
@@ -45,7 +45,7 @@ import urllib.parse
|
||||
import urllib.request
|
||||
import uuid
|
||||
from collections import deque
|
||||
from dataclasses import dataclass, field
|
||||
from dataclasses import dataclass, field, replace
|
||||
from importlib.resources import files
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
@@ -60,6 +60,7 @@ from .capability import (
|
||||
STATE_ADMITTED,
|
||||
STATE_MODEL_MISMATCH,
|
||||
STATE_SHARD_MISMATCH,
|
||||
STATE_UNCERTIFIED,
|
||||
CapabilityState,
|
||||
absent_state,
|
||||
evaluate_report,
|
||||
@@ -84,6 +85,13 @@ from .routing_stats import (
|
||||
)
|
||||
from .model_files import files_for_layer_range, snapshot_dir_for_repo
|
||||
from .raft import RaftNode
|
||||
from .recipe import (
|
||||
CertificationLedger,
|
||||
DistributedForwardEvidence,
|
||||
PresentedIdentity,
|
||||
RecipeIdentityError,
|
||||
RecipeStatus,
|
||||
)
|
||||
|
||||
|
||||
_CONSOLE_LIMIT = 300
|
||||
@@ -792,6 +800,7 @@ def _capability_from_registration(
|
||||
hf_repo: str | None,
|
||||
shard_start: int | None,
|
||||
shard_end: int | None,
|
||||
recipe_certifications: CertificationLedger,
|
||||
) -> CapabilityState:
|
||||
"""The tracker's verdict on the proof carried by one registration payload.
|
||||
|
||||
@@ -811,6 +820,7 @@ def _capability_from_registration(
|
||||
declared_recipe_version=(
|
||||
recipe_version if isinstance(recipe_version, str) else None
|
||||
),
|
||||
ledger=recipe_certifications,
|
||||
)
|
||||
|
||||
|
||||
@@ -830,6 +840,11 @@ def _admitted_nodes(nodes: list["_NodeEntry"], policy: str | None) -> list["_Nod
|
||||
return [node for node in nodes if _capability_routable(node, effective)]
|
||||
|
||||
|
||||
def _route_identity_partition(node: "_NodeEntry") -> tuple[str, ...]:
|
||||
fingerprint = _node_admission(node).fingerprint
|
||||
return ("legacy",) if fingerprint is None else ("exact", *fingerprint)
|
||||
|
||||
|
||||
def _select_route(
|
||||
nodes: list[_NodeEntry],
|
||||
required_start: int,
|
||||
@@ -853,29 +868,51 @@ def _select_route(
|
||||
],
|
||||
key=lambda n: (n.shard_start, -n.shard_end), # type: ignore[operator]
|
||||
)
|
||||
route: list[_NodeEntry] = []
|
||||
covered_up_to = required_start - 1
|
||||
|
||||
def _routing_score(node: "_NodeEntry") -> float:
|
||||
return _effective_throughput(node, model) * _reputation_multiplier(node, contracts)
|
||||
|
||||
while covered_up_to < required_end:
|
||||
best: _NodeEntry | None = None
|
||||
for node in candidates:
|
||||
if node.shard_start <= covered_up_to + 1 and node.shard_end > covered_up_to:
|
||||
if best is None:
|
||||
best = node
|
||||
elif node.shard_end > best.shard_end:
|
||||
best = node
|
||||
elif node.shard_end == best.shard_end and _routing_score(node) > _routing_score(best):
|
||||
best = node
|
||||
if best is None:
|
||||
missing = covered_up_to + 1
|
||||
return [], f"no route available: no registered node covers layer {missing}"
|
||||
route.append(best)
|
||||
covered_up_to = best.shard_end
|
||||
candidates = [n for n in candidates if n is not best]
|
||||
partitions: dict[tuple[str, ...], list[_NodeEntry]] = {}
|
||||
for node in candidates:
|
||||
partitions.setdefault(_route_identity_partition(node), []).append(node)
|
||||
|
||||
complete: list[list[_NodeEntry]] = []
|
||||
furthest = required_start - 1
|
||||
for partition in partitions.values():
|
||||
pool = list(partition)
|
||||
route: list[_NodeEntry] = []
|
||||
covered_up_to = required_start - 1
|
||||
while covered_up_to < required_end:
|
||||
best: _NodeEntry | None = None
|
||||
for node in pool:
|
||||
if node.shard_start <= covered_up_to + 1 and node.shard_end > covered_up_to:
|
||||
if best is None:
|
||||
best = node
|
||||
elif node.shard_end > best.shard_end:
|
||||
best = node
|
||||
elif (
|
||||
node.shard_end == best.shard_end
|
||||
and _routing_score(node) > _routing_score(best)
|
||||
):
|
||||
best = node
|
||||
if best is None:
|
||||
break
|
||||
route.append(best)
|
||||
covered_up_to = best.shard_end
|
||||
pool = [node for node in pool if node is not best]
|
||||
furthest = max(furthest, covered_up_to)
|
||||
if covered_up_to >= required_end:
|
||||
complete.append(route)
|
||||
|
||||
if not complete:
|
||||
return [], f"no route available: no registered node covers layer {furthest + 1}"
|
||||
route = max(
|
||||
complete,
|
||||
key=lambda candidate: (
|
||||
min(_routing_score(node) for node in candidate),
|
||||
-len(candidate),
|
||||
),
|
||||
)
|
||||
return route, ""
|
||||
|
||||
|
||||
@@ -911,7 +948,12 @@ def _enumerate_routes(
|
||||
for head in heads:
|
||||
route = [head]
|
||||
covered_up_to = head.shard_end
|
||||
pool = [n for n in sharded if n is not head]
|
||||
head_partition = _route_identity_partition(head)
|
||||
pool = [
|
||||
node
|
||||
for node in sharded
|
||||
if node is not head and _route_identity_partition(node) == head_partition
|
||||
]
|
||||
while covered_up_to < required_end:
|
||||
best = None
|
||||
for n in pool:
|
||||
@@ -2056,14 +2098,24 @@ def _find_pinned_route(
|
||||
hop_count: int,
|
||||
) -> list[_NodeEntry] | None:
|
||||
"""First combination of exactly ``hop_count`` distinct nodes covering the
|
||||
layer range, where every node extends coverage (US-030 benchmark routes)."""
|
||||
layer range, where every node extends coverage (US-030 benchmark routes).
|
||||
|
||||
Benchmark combos run real inference, so they obey the same DGR-003 rule as
|
||||
every other route builder: one route, one exact identity. A combo that mixed
|
||||
two fingerprints — or an exact Shard with a legacy one — would measure a
|
||||
numerically incoherent route and record the garbage as a benchmark.
|
||||
"""
|
||||
for combo in itertools.permutations(nodes, hop_count):
|
||||
covered = required_start - 1
|
||||
valid = True
|
||||
partition = _route_identity_partition(combo[0])
|
||||
for candidate in combo:
|
||||
if candidate.shard_start is None or candidate.shard_end is None:
|
||||
valid = False
|
||||
break
|
||||
if _route_identity_partition(candidate) != partition:
|
||||
valid = False
|
||||
break
|
||||
if candidate.shard_start > covered + 1 or candidate.shard_end <= covered:
|
||||
valid = False
|
||||
break
|
||||
@@ -2847,9 +2899,11 @@ class _TrackerHTTPServer(socketserver.ThreadingMixIn, http.server.HTTPServer):
|
||||
relay_status: dict | None = None,
|
||||
test_runner: "TestRunManager | None" = None,
|
||||
capability_policy: str | None = None,
|
||||
recipe_certifications: CertificationLedger | None = None,
|
||||
) -> None:
|
||||
super().__init__(addr, handler)
|
||||
self.registry = registry
|
||||
self.recipe_certifications = recipe_certifications or CertificationLedger()
|
||||
self.capability_policy = normalize_policy(
|
||||
capability_policy if capability_policy is not None else policy_from_env()
|
||||
)
|
||||
@@ -4555,6 +4609,7 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
|
||||
hf_repo=hf_repo,
|
||||
shard_start=shard_start,
|
||||
shard_end=shard_end,
|
||||
recipe_certifications=server.recipe_certifications,
|
||||
)
|
||||
|
||||
node_id = _node_id_for_registration(
|
||||
@@ -6531,6 +6586,7 @@ class TrackerServer:
|
||||
self._embedded_relay: Any | None = None
|
||||
self._embedded_relay_actual_port: int | None = None
|
||||
self._registry: dict[str, _NodeEntry] = {}
|
||||
self._recipe_certifications = CertificationLedger()
|
||||
self._lock = threading.Lock()
|
||||
self._server: _TrackerHTTPServer | None = None
|
||||
self._thread: threading.Thread | None = None
|
||||
@@ -6640,6 +6696,84 @@ class TrackerServer:
|
||||
self._test_runner: TestRunManager | None = test_runner
|
||||
self.port: int | None = None
|
||||
|
||||
def certify_recipe(
|
||||
self,
|
||||
identity: PresentedIdentity,
|
||||
evidence: DistributedForwardEvidence,
|
||||
) -> RecipeStatus:
|
||||
"""Promote one exact recipe from tracker-recorded distributed evidence.
|
||||
|
||||
This is the tracker's only certification authority. The evidence names
|
||||
the nodes that served the forward; this method refuses to take any of
|
||||
them on the evidence's word. For each one it goes back to what *this
|
||||
tracker* re-derived at admission and requires the evidence to describe
|
||||
the same Shard: same route fingerprint, same registered range, and the
|
||||
same shard binding digest — the derivative's own bytes and exact range.
|
||||
|
||||
Without the binding check, a route fingerprint is range-independent by
|
||||
design (:mod:`meshnet_tracker.recipe`), so evidence could name the right
|
||||
recipe while describing splits nobody on the route was actually serving,
|
||||
and the certification would attach to blobs that never ran.
|
||||
"""
|
||||
with self._lock:
|
||||
for node_id, participant in zip(
|
||||
evidence.node_ids, evidence.participants, strict=True
|
||||
):
|
||||
node = self._registry.get(node_id)
|
||||
if node is None:
|
||||
raise RecipeIdentityError(
|
||||
f"certification participant {node_id!r} is not registered"
|
||||
)
|
||||
admitted = node.capability
|
||||
if admitted.fingerprint != identity.key:
|
||||
raise RecipeIdentityError(
|
||||
f"certification participant {node_id!r} is not admitted under "
|
||||
"the promoted fingerprint"
|
||||
)
|
||||
if admitted.state not in (STATE_UNCERTIFIED, STATE_ADMITTED):
|
||||
raise RecipeIdentityError(
|
||||
f"certification participant {node_id!r} has capability state "
|
||||
f"{admitted.state!r}"
|
||||
)
|
||||
registered_range = (
|
||||
node.shard_start,
|
||||
None if node.shard_end is None else node.shard_end + 1,
|
||||
)
|
||||
if registered_range != (
|
||||
participant.shard_start,
|
||||
participant.shard_end,
|
||||
):
|
||||
raise RecipeIdentityError(
|
||||
f"certification participant {node_id!r} range differs from its "
|
||||
"registered Shard range"
|
||||
)
|
||||
# The exact bytes and range this node was admitted on. A node that
|
||||
# registered without an identity has no binding, and cannot be a
|
||||
# participant in an exact certification at all.
|
||||
if admitted.shard_binding_digest is None:
|
||||
raise RecipeIdentityError(
|
||||
f"certification participant {node_id!r} registered no exact "
|
||||
"Shard binding; it cannot certify a recipe"
|
||||
)
|
||||
if admitted.shard_binding_digest != participant.shard_binding_digest:
|
||||
raise RecipeIdentityError(
|
||||
f"certification participant {node_id!r} presents a Shard "
|
||||
"binding this tracker did not admit it on; the evidence "
|
||||
"describes different derivative bytes or a different range"
|
||||
)
|
||||
|
||||
status = self._recipe_certifications.certify(identity, evidence)
|
||||
for node in self._registry.values():
|
||||
if (
|
||||
node.capability.state == STATE_UNCERTIFIED
|
||||
and node.capability.fingerprint == identity.key
|
||||
):
|
||||
node.capability = replace(
|
||||
node.capability.with_state(STATE_ADMITTED, status.detail),
|
||||
certification=status.status,
|
||||
)
|
||||
return status
|
||||
|
||||
def _start_embedded_relay(self) -> dict:
|
||||
"""Start the shared RelayServer class in-process for tracker+relay deployments."""
|
||||
if not self._embedded_relay_enabled:
|
||||
@@ -6725,6 +6859,7 @@ class TrackerServer:
|
||||
relay_status=http_relay_status,
|
||||
test_runner=self._test_runner,
|
||||
capability_policy=self._capability_policy,
|
||||
recipe_certifications=self._recipe_certifications,
|
||||
)
|
||||
self.port = self._server.server_address[1]
|
||||
|
||||
@@ -6983,6 +7118,7 @@ class TrackerServer:
|
||||
hf_repo=payload.get("hf_repo"),
|
||||
shard_start=shard_start,
|
||||
shard_end=shard_end,
|
||||
recipe_certifications=self._recipe_certifications,
|
||||
),
|
||||
)
|
||||
with self._lock:
|
||||
|
||||
153
scripts/gen_recipe_fingerprint_vectors.py
Normal file
153
scripts/gen_recipe_fingerprint_vectors.py
Normal file
@@ -0,0 +1,153 @@
|
||||
#!/usr/bin/env python
|
||||
"""Generate (or verify) the committed DGR-003 fingerprint conformance vectors.
|
||||
|
||||
The node and the tracker derive artifact, recipe and Shard-binding digests from
|
||||
*separate* implementations on purpose: an admission gate that shares code with
|
||||
the thing it admits is not an independent check. The cost of that independence
|
||||
is drift — two canonicalizers that quietly stop agreeing would not fail, they
|
||||
would silently stop forming routes, or worse, silently form wrong ones.
|
||||
|
||||
``tests/data/recipe_fingerprint_vectors.json`` is what makes drift loud. It is a
|
||||
language-neutral artifact — canonical input blocks, expected digests, and the
|
||||
serialized DGR-002 ``Fingerprint`` bytes — that the node tests, the tracker tests
|
||||
and (later) the native C++ worker all check themselves against.
|
||||
|
||||
python scripts/gen_recipe_fingerprint_vectors.py --check # CI: no drift
|
||||
python scripts/gen_recipe_fingerprint_vectors.py # rewrite vectors
|
||||
|
||||
Rewriting is a deliberate act: if this changes a digest, it changed the wire
|
||||
contract, and every node and tracker in the fleet has to agree at the same time.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import pathlib
|
||||
import sys
|
||||
|
||||
_ROOT = pathlib.Path(__file__).resolve().parent.parent
|
||||
sys.path[:0] = [str(_ROOT / "packages" / "node"), str(_ROOT / "packages" / "tracker")]
|
||||
|
||||
from meshnet_node.runtime_recipe import ( # noqa: E402
|
||||
ArtifactIdentity,
|
||||
DerivativeBinding,
|
||||
RuntimeRecipe,
|
||||
ShardIdentity,
|
||||
)
|
||||
from meshnet_tracker.recipe import parse_identity # noqa: E402
|
||||
|
||||
VECTORS = _ROOT / "tests" / "data" / "recipe_fingerprint_vectors.json"
|
||||
SCHEMA_VERSION = 1
|
||||
|
||||
_RECIPE = RuntimeRecipe(
|
||||
weight_quantization="Q4_K_M",
|
||||
activation_dtype="bfloat16",
|
||||
compute_dtype="float32",
|
||||
kv_dtype="q8_0",
|
||||
kv_layout="paged-v1",
|
||||
tokenizer_revision="0123456789abcdef",
|
||||
architecture_adapter="llama/range-v1",
|
||||
backend_id="llama.cpp",
|
||||
runtime_version="llama.cpp@deadbeef+meshnet.1",
|
||||
recipe_id="example-gguf",
|
||||
recipe_version="1",
|
||||
catalogue_version="2026.07.1",
|
||||
)
|
||||
|
||||
_SOURCE = "a" * 64
|
||||
_SPLIT_BYTES = "c" * 64
|
||||
_CONFIG = "b" * 64
|
||||
|
||||
|
||||
def _cases() -> list[tuple[str, str, ShardIdentity]]:
|
||||
whole = ShardIdentity(
|
||||
ArtifactIdentity(
|
||||
"example/model", "0123456789abcdef", _SOURCE, "dense-llama", _CONFIG, 8
|
||||
),
|
||||
_RECIPE,
|
||||
0,
|
||||
4,
|
||||
)
|
||||
# The same recipe on the same source, held as a split: identical route
|
||||
# fingerprint, different Shard binding. Both halves of that are contract.
|
||||
derivative = ShardIdentity(
|
||||
ArtifactIdentity(
|
||||
"example/model",
|
||||
"0123456789abcdef",
|
||||
_SPLIT_BYTES,
|
||||
"dense-llama",
|
||||
_CONFIG,
|
||||
8,
|
||||
DerivativeBinding(_SOURCE, 4, 8),
|
||||
),
|
||||
_RECIPE,
|
||||
4,
|
||||
8,
|
||||
)
|
||||
return [
|
||||
("example-v1", "An undivided artifact: content digest is the source digest.", whole),
|
||||
(
|
||||
"example-v1-derivative",
|
||||
"A split of the same source: same fingerprint, different Shard binding.",
|
||||
derivative,
|
||||
),
|
||||
]
|
||||
|
||||
|
||||
def build() -> dict:
|
||||
vectors = []
|
||||
for name, description, identity in _cases():
|
||||
block = identity.to_dict()
|
||||
presented = parse_identity(block)
|
||||
|
||||
# The two implementations must already agree before this is committed.
|
||||
assert identity.fingerprint.to_dict() == presented.fingerprint_dict(), name
|
||||
assert identity.shard_binding_digest == presented.shard_binding_digest, name
|
||||
|
||||
vectors.append(
|
||||
{
|
||||
"name": name,
|
||||
"description": description,
|
||||
"identity": block,
|
||||
"fingerprint": identity.fingerprint.to_dict(),
|
||||
"shard_binding_digest": identity.shard_binding_digest,
|
||||
"fingerprint_proto_hex": identity.fingerprint.to_proto()
|
||||
.SerializeToString(deterministic=True)
|
||||
.hex(),
|
||||
}
|
||||
)
|
||||
return {"schema_version": SCHEMA_VERSION, "vectors": vectors}
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument(
|
||||
"--check",
|
||||
action="store_true",
|
||||
help="fail if the committed vectors differ from what this code derives",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
built = json.dumps(build(), indent=2, sort_keys=True) + "\n"
|
||||
if not args.check:
|
||||
VECTORS.write_text(built, encoding="utf-8")
|
||||
print(f"wrote {VECTORS.relative_to(_ROOT)}")
|
||||
return 0
|
||||
|
||||
committed = VECTORS.read_text(encoding="utf-8")
|
||||
if committed != built:
|
||||
print(
|
||||
f"{VECTORS.relative_to(_ROOT)} is stale: the identity implementation no "
|
||||
"longer derives the committed digests.\nIf that change was intended, it "
|
||||
"is a wire-contract change — rerun without --check and roll out node and "
|
||||
"tracker together.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return 1
|
||||
print(f"{VECTORS.relative_to(_ROOT)} matches the identity implementation")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
268
scripts/llama_cpp_dependency.py
Normal file
268
scripts/llama_cpp_dependency.py
Normal file
@@ -0,0 +1,268 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Materialize, verify, build, and smoke-test DGR-004's llama.cpp pin.
|
||||
|
||||
This tool deliberately owns only a source dependency boundary. It never
|
||||
downloads a model, invokes inference, or interprets generated text.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import pathlib
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
from typing import Any
|
||||
|
||||
|
||||
ROOT = pathlib.Path(__file__).resolve().parents[1]
|
||||
LLAMA_DIR = ROOT / "packages/node/native/llama"
|
||||
LOCK_PATH = LLAMA_DIR / "UPSTREAM_LOCK.json"
|
||||
PATCH_DIR = LLAMA_DIR / "patches"
|
||||
|
||||
|
||||
def _cmake() -> str:
|
||||
"""Use an explicit override, PATH, or the active Python environment."""
|
||||
configured = os.environ.get("CMAKE")
|
||||
if configured:
|
||||
return configured
|
||||
on_path = shutil.which("cmake")
|
||||
if on_path:
|
||||
return on_path
|
||||
sibling = pathlib.Path(sys.executable).parent / "cmake"
|
||||
if sibling.is_file():
|
||||
return str(sibling)
|
||||
raise DependencyError("cmake is unavailable; set CMAKE or activate the project toolchain")
|
||||
|
||||
|
||||
class DependencyError(RuntimeError):
|
||||
"""A fail-closed reproducibility or source-integrity failure."""
|
||||
|
||||
|
||||
def _run(*args: str, cwd: pathlib.Path | None = None) -> str:
|
||||
try:
|
||||
completed = subprocess.run(
|
||||
args,
|
||||
cwd=cwd,
|
||||
check=True,
|
||||
text=True,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
)
|
||||
except FileNotFoundError as error:
|
||||
raise DependencyError(f"required executable is unavailable: {args[0]}") from error
|
||||
except subprocess.CalledProcessError as error:
|
||||
detail = (error.stderr or error.stdout).strip()
|
||||
raise DependencyError(f"command failed: {' '.join(args)}\n{detail}") from error
|
||||
return completed.stdout.strip()
|
||||
|
||||
|
||||
def _load_lock() -> dict[str, Any]:
|
||||
try:
|
||||
lock = json.loads(LOCK_PATH.read_text())
|
||||
except (OSError, json.JSONDecodeError) as error:
|
||||
raise DependencyError(f"invalid upstream lock: {LOCK_PATH}: {error}") from error
|
||||
required = {
|
||||
"upstream", "commit", "commit_tree", "patched_tree", "patch_series",
|
||||
"required_upstream_blobs", "patched_paths", "build",
|
||||
}
|
||||
missing = sorted(required - lock.keys())
|
||||
if missing:
|
||||
raise DependencyError(f"upstream lock is missing fields: {', '.join(missing)}")
|
||||
commit_file = (LLAMA_DIR / "UPSTREAM_COMMIT").read_text().strip()
|
||||
if lock["commit"] != commit_file or len(commit_file) != 40:
|
||||
raise DependencyError("UPSTREAM_COMMIT and UPSTREAM_LOCK.json do not agree on a full commit")
|
||||
return lock
|
||||
|
||||
|
||||
def _patches(lock: dict[str, Any]) -> list[pathlib.Path]:
|
||||
series = [line for line in (PATCH_DIR / "series").read_text().splitlines() if line]
|
||||
if series != lock["patch_series"] or series != sorted(series) or not series:
|
||||
raise DependencyError("patches/series is empty, unordered, or disagrees with UPSTREAM_LOCK.json")
|
||||
sums: dict[str, str] = {}
|
||||
for line in (PATCH_DIR / "SHA256SUMS").read_text().splitlines():
|
||||
if line and not line.startswith("#"):
|
||||
digest, name = line.split(maxsplit=1)
|
||||
sums[name] = digest
|
||||
patches: list[pathlib.Path] = []
|
||||
for name in series:
|
||||
patch = PATCH_DIR / name
|
||||
if not patch.is_file():
|
||||
raise DependencyError(f"patch listed in series is missing: {name}")
|
||||
actual = hashlib.sha256(patch.read_bytes()).hexdigest()
|
||||
if sums.get(name) != actual:
|
||||
raise DependencyError(f"patch digest mismatch for {name}: expected {sums.get(name)}, got {actual}")
|
||||
patches.append(patch)
|
||||
return patches
|
||||
|
||||
|
||||
def _git(source: pathlib.Path, *args: str) -> str:
|
||||
return _run("git", "-C", str(source), *args)
|
||||
|
||||
|
||||
def _verify_source(source: pathlib.Path, lock: dict[str, Any], *, require_clean: bool) -> None:
|
||||
if not (source / ".git").exists():
|
||||
raise DependencyError(f"not a materialized git checkout: {source}")
|
||||
if _git(source, "rev-parse", "HEAD") != lock["commit"]:
|
||||
raise DependencyError("upstream drift: checkout HEAD does not equal the locked commit")
|
||||
if _git(source, "rev-parse", "HEAD^{tree}") != lock["commit_tree"]:
|
||||
raise DependencyError("upstream drift: checkout tree does not equal the locked tree")
|
||||
if require_clean and _git(source, "status", "--porcelain"):
|
||||
raise DependencyError("local edits detected in materialized llama.cpp checkout")
|
||||
for relative, expected in lock["required_upstream_blobs"].items():
|
||||
actual = _git(source, "rev-parse", f"HEAD:{relative}")
|
||||
if actual != expected:
|
||||
raise DependencyError(
|
||||
f"upstream ABI/context drift for {relative}: expected {expected}, got {actual}"
|
||||
)
|
||||
if not (source / "LICENSE").is_file():
|
||||
raise DependencyError("upstream LICENSE is missing; refusing to drop required attribution")
|
||||
|
||||
|
||||
def materialize(source: pathlib.Path, repository: str) -> None:
|
||||
lock = _load_lock()
|
||||
_patches(lock)
|
||||
if source.exists():
|
||||
raise DependencyError(f"destination already exists; refusing to reuse possibly edited source: {source}")
|
||||
source.parent.mkdir(parents=True, exist_ok=True)
|
||||
_run("git", "clone", "--no-checkout", repository, str(source))
|
||||
_git(source, "checkout", "--detach", lock["commit"])
|
||||
_verify_source(source, lock, require_clean=True)
|
||||
|
||||
|
||||
def apply(source: pathlib.Path) -> None:
|
||||
lock = _load_lock()
|
||||
patches = _patches(lock)
|
||||
_verify_source(source, lock, require_clean=True)
|
||||
for patch in patches:
|
||||
_git(source, "apply", "--check", str(patch))
|
||||
_git(source, "apply", "--index", str(patch))
|
||||
_verify_patched_source(source, lock)
|
||||
|
||||
|
||||
def _verify_patched_source(source: pathlib.Path, lock: dict[str, Any]) -> None:
|
||||
if _git(source, "diff", "--quiet"):
|
||||
raise DependencyError("local unstaged edits detected after applying patch stack")
|
||||
changed_paths = _git(source, "diff", "--cached", "--name-only").splitlines()
|
||||
if changed_paths != lock["patched_paths"]:
|
||||
raise DependencyError(f"patched paths drifted: expected {lock['patched_paths']}, got {changed_paths}")
|
||||
if _git(source, "write-tree") != lock["patched_tree"]:
|
||||
raise DependencyError("patched source tree differs from the locked patch stack")
|
||||
if _git(source, "diff", "--name-only"):
|
||||
raise DependencyError("local unstaged edits detected after applying patch stack")
|
||||
untracked = _git(source, "ls-files", "--others", "--exclude-standard").splitlines()
|
||||
if untracked:
|
||||
raise DependencyError(f"untracked files detected after applying patch stack: {untracked}")
|
||||
|
||||
|
||||
def build(source: pathlib.Path, build_dir: pathlib.Path) -> pathlib.Path:
|
||||
lock = _load_lock()
|
||||
_patches(lock)
|
||||
_verify_source(source, lock, require_clean=False)
|
||||
_verify_patched_source(source, lock)
|
||||
expected_marker = source / "cmake/meshnet-patch-stack.cmake"
|
||||
if not expected_marker.is_file():
|
||||
raise DependencyError("patch stack is not applied: Meshnet CMake marker is absent")
|
||||
if build_dir.exists():
|
||||
raise DependencyError(f"build directory already exists; use reproduce for a clean rebuild: {build_dir}")
|
||||
flags = lock["build"]["configure_flags"]
|
||||
cmake = _cmake()
|
||||
_run(cmake, "-G", lock["build"]["generator"], "-S", str(source), "-B", str(build_dir), *flags)
|
||||
for target in lock["build"]["native_targets"]:
|
||||
_run(cmake, "--build", str(build_dir), "--target", target, "-j2")
|
||||
metadata = {
|
||||
"commit": lock["commit"],
|
||||
"commit_tree": lock["commit_tree"],
|
||||
"patches": {patch.name: hashlib.sha256(patch.read_bytes()).hexdigest() for patch in _patches(lock)},
|
||||
"configure_flags": flags,
|
||||
"cmake": _run(cmake, "--version").splitlines()[0],
|
||||
"cxx": _run("c++", "--version").splitlines()[0],
|
||||
"model_downloads": False,
|
||||
"semantic_certification": False,
|
||||
}
|
||||
(build_dir / "meshnet-build-metadata.json").write_text(json.dumps(metadata, indent=2, sort_keys=True) + "\n")
|
||||
return build_dir / lock["build"]["smoke_binary"]
|
||||
|
||||
|
||||
def smoke(binary: pathlib.Path) -> None:
|
||||
if not binary.is_file():
|
||||
raise DependencyError(f"native smoke binary is missing: {binary}")
|
||||
smoke_config = _load_lock()["build"]
|
||||
output = _run(str(binary), *smoke_config["smoke_args"])
|
||||
expected = smoke_config["smoke_output_token"]
|
||||
if expected not in output.lower():
|
||||
raise DependencyError(f"native smoke output did not contain {expected!r}: {output}")
|
||||
print(output)
|
||||
|
||||
|
||||
def reproduce(work_dir: pathlib.Path, repository: str) -> None:
|
||||
resolved = work_dir.resolve()
|
||||
build_root = (ROOT / "build").resolve()
|
||||
if build_root not in resolved.parents:
|
||||
raise DependencyError(f"--work-dir must be below {build_root}: {resolved}")
|
||||
if resolved.exists():
|
||||
raise DependencyError(
|
||||
f"work directory already exists; refusing to erase possible local edits: {resolved}"
|
||||
)
|
||||
source = resolved / "source"
|
||||
build_dir = resolved / "build"
|
||||
materialize(source, repository)
|
||||
apply(source)
|
||||
smoke(build(source, build_dir))
|
||||
|
||||
|
||||
def inspect() -> None:
|
||||
lock = _load_lock()
|
||||
patches = _patches(lock)
|
||||
print(json.dumps({
|
||||
"commit": lock["commit"],
|
||||
"patch_count": len(patches),
|
||||
"patches": [patch.name for patch in patches],
|
||||
"model_downloads": False,
|
||||
"semantic_certification": False,
|
||||
"glm_stock_limitations": lock["stock_glm_limitations"],
|
||||
}, indent=2, sort_keys=True))
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
subcommands = parser.add_subparsers(dest="command", required=True)
|
||||
subcommands.add_parser("inspect")
|
||||
materialize_parser = subcommands.add_parser("materialize")
|
||||
materialize_parser.add_argument("--source-dir", type=pathlib.Path, required=True)
|
||||
materialize_parser.add_argument("--source-repository", default=_load_lock()["upstream"])
|
||||
apply_parser = subcommands.add_parser("apply")
|
||||
apply_parser.add_argument("--source-dir", type=pathlib.Path, required=True)
|
||||
build_parser = subcommands.add_parser("build")
|
||||
build_parser.add_argument("--source-dir", type=pathlib.Path, required=True)
|
||||
build_parser.add_argument("--build-dir", type=pathlib.Path, required=True)
|
||||
smoke_parser = subcommands.add_parser("smoke")
|
||||
smoke_parser.add_argument("--binary", type=pathlib.Path, required=True)
|
||||
reproduce_parser = subcommands.add_parser("reproduce")
|
||||
reproduce_parser.add_argument("--work-dir", type=pathlib.Path, default=ROOT / "build/dgr-004-smoke")
|
||||
reproduce_parser.add_argument("--source-repository", default=_load_lock()["upstream"])
|
||||
args = parser.parse_args()
|
||||
try:
|
||||
if args.command == "inspect":
|
||||
inspect()
|
||||
elif args.command == "materialize":
|
||||
materialize(args.source_dir, args.source_repository)
|
||||
elif args.command == "apply":
|
||||
apply(args.source_dir)
|
||||
elif args.command == "build":
|
||||
build(args.source_dir, args.build_dir)
|
||||
elif args.command == "smoke":
|
||||
smoke(args.binary)
|
||||
else:
|
||||
reproduce(args.work_dir, args.source_repository)
|
||||
except DependencyError as error:
|
||||
print(f"DGR-004 dependency error: {error}", file=sys.stderr)
|
||||
return 2
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
109
tests/data/recipe_fingerprint_vectors.json
Normal file
109
tests/data/recipe_fingerprint_vectors.json
Normal file
@@ -0,0 +1,109 @@
|
||||
{
|
||||
"schema_version": 1,
|
||||
"vectors": [
|
||||
{
|
||||
"description": "An undivided artifact: content digest is the source digest.",
|
||||
"fingerprint": {
|
||||
"catalogue_version": "2026.07.1",
|
||||
"model_artifact_digest": "8a0f43d6aa49d77834bdb47bcae9f42c886b7ccfe0ac014932b2a2b38697a47b",
|
||||
"recipe_id": "example-gguf",
|
||||
"recipe_version": "1",
|
||||
"runtime_recipe_digest": "9b14d70b0835a6428457e4888d453649dd0d2e41fc8ac9d84d232c8c237e68fa"
|
||||
},
|
||||
"fingerprint_proto_hex": "0a40386130663433643661613439643737383334626462343762636165396634326338383662376363666530616330313439333262326132623338363937613437621240396231346437306230383335613634323834353765343838386434353336343964643064326534316663386163396438346432333263386332333765363866611a0c6578616d706c652d676775662201312a09323032362e30372e31",
|
||||
"identity": {
|
||||
"artifact": {
|
||||
"architecture": "dense-llama",
|
||||
"architecture_digest": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
|
||||
"artifact_id": "example/model",
|
||||
"content_digest": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
|
||||
"derived_from": null,
|
||||
"layer_count": 8,
|
||||
"revision": "0123456789abcdef"
|
||||
},
|
||||
"fingerprint": {
|
||||
"catalogue_version": "2026.07.1",
|
||||
"model_artifact_digest": "8a0f43d6aa49d77834bdb47bcae9f42c886b7ccfe0ac014932b2a2b38697a47b",
|
||||
"recipe_id": "example-gguf",
|
||||
"recipe_version": "1",
|
||||
"runtime_recipe_digest": "9b14d70b0835a6428457e4888d453649dd0d2e41fc8ac9d84d232c8c237e68fa"
|
||||
},
|
||||
"recipe": {
|
||||
"activation_dtype": "bfloat16",
|
||||
"architecture_adapter": "llama/range-v1",
|
||||
"backend_id": "llama.cpp",
|
||||
"boundary_schema_version": 1,
|
||||
"catalogue_version": "2026.07.1",
|
||||
"compute_dtype": "float32",
|
||||
"kv_dtype": "q8_0",
|
||||
"kv_layout": "paged-v1",
|
||||
"protocol_schema_version": 1,
|
||||
"recipe_id": "example-gguf",
|
||||
"recipe_version": "1",
|
||||
"runtime_version": "llama.cpp@deadbeef+meshnet.1",
|
||||
"tokenizer_revision": "0123456789abcdef",
|
||||
"weight_quantization": "Q4_K_M"
|
||||
},
|
||||
"schema_version": 1,
|
||||
"shard_end": 4,
|
||||
"shard_start": 0
|
||||
},
|
||||
"name": "example-v1",
|
||||
"shard_binding_digest": "f62d1f76cc18b548782c02850e05c2634da55c0e686c43b9f687bdee7bdefe19"
|
||||
},
|
||||
{
|
||||
"description": "A split of the same source: same fingerprint, different Shard binding.",
|
||||
"fingerprint": {
|
||||
"catalogue_version": "2026.07.1",
|
||||
"model_artifact_digest": "8a0f43d6aa49d77834bdb47bcae9f42c886b7ccfe0ac014932b2a2b38697a47b",
|
||||
"recipe_id": "example-gguf",
|
||||
"recipe_version": "1",
|
||||
"runtime_recipe_digest": "9b14d70b0835a6428457e4888d453649dd0d2e41fc8ac9d84d232c8c237e68fa"
|
||||
},
|
||||
"fingerprint_proto_hex": "0a40386130663433643661613439643737383334626462343762636165396634326338383662376363666530616330313439333262326132623338363937613437621240396231346437306230383335613634323834353765343838386434353336343964643064326534316663386163396438346432333263386332333765363866611a0c6578616d706c652d676775662201312a09323032362e30372e31",
|
||||
"identity": {
|
||||
"artifact": {
|
||||
"architecture": "dense-llama",
|
||||
"architecture_digest": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
|
||||
"artifact_id": "example/model",
|
||||
"content_digest": "cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc",
|
||||
"derived_from": {
|
||||
"shard_end": 8,
|
||||
"shard_start": 4,
|
||||
"source_artifact_digest": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
|
||||
},
|
||||
"layer_count": 8,
|
||||
"revision": "0123456789abcdef"
|
||||
},
|
||||
"fingerprint": {
|
||||
"catalogue_version": "2026.07.1",
|
||||
"model_artifact_digest": "8a0f43d6aa49d77834bdb47bcae9f42c886b7ccfe0ac014932b2a2b38697a47b",
|
||||
"recipe_id": "example-gguf",
|
||||
"recipe_version": "1",
|
||||
"runtime_recipe_digest": "9b14d70b0835a6428457e4888d453649dd0d2e41fc8ac9d84d232c8c237e68fa"
|
||||
},
|
||||
"recipe": {
|
||||
"activation_dtype": "bfloat16",
|
||||
"architecture_adapter": "llama/range-v1",
|
||||
"backend_id": "llama.cpp",
|
||||
"boundary_schema_version": 1,
|
||||
"catalogue_version": "2026.07.1",
|
||||
"compute_dtype": "float32",
|
||||
"kv_dtype": "q8_0",
|
||||
"kv_layout": "paged-v1",
|
||||
"protocol_schema_version": 1,
|
||||
"recipe_id": "example-gguf",
|
||||
"recipe_version": "1",
|
||||
"runtime_version": "llama.cpp@deadbeef+meshnet.1",
|
||||
"tokenizer_revision": "0123456789abcdef",
|
||||
"weight_quantization": "Q4_K_M"
|
||||
},
|
||||
"schema_version": 1,
|
||||
"shard_end": 8,
|
||||
"shard_start": 4
|
||||
},
|
||||
"name": "example-v1-derivative",
|
||||
"shard_binding_digest": "55271611eb63cb81088b8109133f1dff845352298994fc8e9c5824a6a01c83e0"
|
||||
}
|
||||
]
|
||||
}
|
||||
233
tests/test_gguf_distributed_load.py
Normal file
233
tests/test_gguf_distributed_load.py
Normal file
@@ -0,0 +1,233 @@
|
||||
"""Distributed GGUF shard load integration test.
|
||||
|
||||
Downloads a small dense-Llama GGUF (TinyLlama 1.1B Q4_K_M ~670 MB),
|
||||
loads it in shard ranges via the meshnet-range-loader C wrapper, registers
|
||||
each shard with a live TrackerServer, and verifies routing, range reporting,
|
||||
and memory scaling.
|
||||
|
||||
Set MESHNET_ENABLE_REAL_INFERENCE_TESTS=1 to run.
|
||||
|
||||
Evidence class: real-model integration. Downloads ~670 MB on first run.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import pathlib
|
||||
import subprocess
|
||||
import sys
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
|
||||
import pytest
|
||||
|
||||
ROOT = pathlib.Path(__file__).resolve().parent.parent
|
||||
sys.path[:0] = [
|
||||
str(ROOT / "packages" / "tracker"),
|
||||
str(ROOT / "packages" / "node"),
|
||||
str(ROOT / "packages" / "contracts"),
|
||||
]
|
||||
|
||||
from meshnet_tracker.server import TrackerServer # noqa: E402 — sys.path prepended above
|
||||
|
||||
# Only run when explicitly enabled
|
||||
pytestmark = pytest.mark.skipif(
|
||||
"MESHNET_ENABLE_REAL_INFERENCE_TESTS" not in os.environ,
|
||||
reason="set MESHNET_ENABLE_REAL_INFERENCE_TESTS=1 to download and load a real GGUF",
|
||||
)
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Model configuration
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
HF_REPO = "TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF"
|
||||
GGUF_FILE = "tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf"
|
||||
MODEL_URL = f"https://huggingface.co/{HF_REPO}/resolve/main/{GGUF_FILE}"
|
||||
EXPECTED_LAYERS = 22
|
||||
GGUF_FLAVOR = GGUF_FILE.replace(".gguf", "")
|
||||
|
||||
CACHE_DIR = ROOT / ".cache" / "gguf-models"
|
||||
LOADER = ROOT / "build" / "dgr-004-final" / "build" / "bin" / "meshnet-range-loader"
|
||||
LLAMA_LIB_DIR = LOADER.parent
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Loading helper
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def load_shard(gguf_path: str, start: int, end: int) -> dict:
|
||||
"""Load a shard range of the GGUF via the C wrapper and return JSON report."""
|
||||
env = os.environ.copy()
|
||||
env["LD_LIBRARY_PATH"] = str(LLAMA_LIB_DIR)
|
||||
|
||||
result = subprocess.run(
|
||||
[str(LOADER), gguf_path, str(start), str(end)],
|
||||
capture_output=True, text=True, timeout=120, env=env,
|
||||
)
|
||||
if result.returncode != 0:
|
||||
# Parse JSON from stdout even on error if it exists
|
||||
if result.stdout.strip():
|
||||
try:
|
||||
return json.loads(result.stdout)
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
raise RuntimeError(
|
||||
f"meshnet-range-loader [{start}, {end}) failed (exit {result.returncode}):\n"
|
||||
f"stderr: {result.stderr[:500]}"
|
||||
)
|
||||
|
||||
return json.loads(result.stdout)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Download and cache
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _ensure_model() -> pathlib.Path:
|
||||
CACHE_DIR.mkdir(parents=True, exist_ok=True)
|
||||
model_path = CACHE_DIR / GGUF_FILE
|
||||
|
||||
if model_path.exists() and model_path.stat().st_size > 600 * 1024 * 1024:
|
||||
return model_path
|
||||
|
||||
print(f"Downloading {MODEL_URL} (~670 MB)...", file=sys.stderr)
|
||||
urllib.request.urlretrieve(MODEL_URL, model_path)
|
||||
actual_mb = model_path.stat().st_size / (1024 * 1024)
|
||||
print(f"Downloaded {GGUF_FLAVOR}: {actual_mb:.0f} MB", file=sys.stderr)
|
||||
return model_path
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Tracker helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _post_json(url: str, data: dict) -> dict:
|
||||
req = urllib.request.Request(
|
||||
url,
|
||||
data=json.dumps(data).encode(),
|
||||
headers={"Content-Type": "application/json"},
|
||||
method="POST",
|
||||
)
|
||||
with urllib.request.urlopen(req, timeout=10) as r:
|
||||
return json.loads(r.read())
|
||||
|
||||
|
||||
def _get_json(url: str) -> dict:
|
||||
with urllib.request.urlopen(url, timeout=10) as r:
|
||||
return json.loads(r.read())
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Tests
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_whole_model_load_and_report():
|
||||
"""Load the full TinyLlama GGUF and verify metadata."""
|
||||
gguf = _ensure_model()
|
||||
report = load_shard(str(gguf), 0, EXPECTED_LAYERS)
|
||||
assert report["ok"] is True
|
||||
assert report["start_layer"] == 0
|
||||
assert report["end_layer"] == EXPECTED_LAYERS
|
||||
assert report["mapped_bytes"] > 0
|
||||
assert report["resident_bytes"] >= report["mapped_bytes"]
|
||||
|
||||
|
||||
def test_head_shard_is_less_than_full_model():
|
||||
"""A head-only shard maps fewer bytes than the full model."""
|
||||
gguf = _ensure_model()
|
||||
head = load_shard(str(gguf), 0, 8)
|
||||
full = load_shard(str(gguf), 0, EXPECTED_LAYERS)
|
||||
assert head["mapped_bytes"] <= full["mapped_bytes"]
|
||||
|
||||
|
||||
def test_tail_shard_maps_fewer_bytes_than_middle():
|
||||
"""Fewer layers = fewer bytes (tail has 6 layers, middle has 8)."""
|
||||
gguf = _ensure_model()
|
||||
middle = load_shard(str(gguf), 8, 16)
|
||||
tail = load_shard(str(gguf), 16, EXPECTED_LAYERS)
|
||||
# Middle (8 layers) must map more than tail (6 layers)
|
||||
assert middle["mapped_bytes"] > tail["mapped_bytes"]
|
||||
|
||||
|
||||
def test_memory_scales_with_owned_range():
|
||||
"""More layers = more resident bytes."""
|
||||
gguf = _ensure_model()
|
||||
small = load_shard(str(gguf), 0, 4)
|
||||
large = load_shard(str(gguf), 0, 12)
|
||||
assert large["mapped_bytes"] > small["mapped_bytes"]
|
||||
assert large["resident_bytes"] > small["resident_bytes"]
|
||||
|
||||
|
||||
def test_invalid_range_rejected():
|
||||
"""Loading a range outside GGUF layer bounds fails closed."""
|
||||
gguf = _ensure_model()
|
||||
env = os.environ.copy()
|
||||
env["LD_LIBRARY_PATH"] = str(LLAMA_LIB_DIR)
|
||||
result = subprocess.run(
|
||||
[str(LOADER), str(gguf), str(EXPECTED_LAYERS + 1), str(EXPECTED_LAYERS + 5)],
|
||||
capture_output=True, text=True, timeout=30, env=env,
|
||||
)
|
||||
# Should fail with exit code 1 and an error message
|
||||
assert result.returncode != 0
|
||||
assert "error" in result.stderr.lower() or "ERROR" in result.stderr
|
||||
|
||||
|
||||
def test_tracker_registers_shard_nodes():
|
||||
"""Start a tracker, register multiple shard nodes, verify the console."""
|
||||
gguf = _ensure_model()
|
||||
|
||||
tracker = TrackerServer(heartbeat_timeout=60.0)
|
||||
tracker_port = tracker.start()
|
||||
|
||||
shards = [
|
||||
("head", 0, 8),
|
||||
("middle", 8, 16),
|
||||
("tail", 16, EXPECTED_LAYERS),
|
||||
]
|
||||
registered_ids = []
|
||||
try:
|
||||
for name_tag, start, end in shards:
|
||||
report = load_shard(str(gguf), start, end)
|
||||
node_id = f"{GGUF_FLAVOR}-{name_tag}"
|
||||
registered_ids.append(node_id)
|
||||
|
||||
_post_json(
|
||||
f"http://127.0.0.1:{tracker_port}/v1/nodes/register",
|
||||
{
|
||||
"node_id": node_id,
|
||||
"endpoint": f"http://localhost:{10000 + start}",
|
||||
"model": GGUF_FLAVOR,
|
||||
"num_layers": report["end_layer"] - report["start_layer"],
|
||||
"shard_start": report["start_layer"],
|
||||
"shard_end": report["end_layer"],
|
||||
"hardware_profile": {
|
||||
"mapped_bytes": report["mapped_bytes"],
|
||||
"resident_bytes": report["resident_bytes"],
|
||||
},
|
||||
"score": 1.0,
|
||||
},
|
||||
)
|
||||
|
||||
# Verify console shows all three registered nodes
|
||||
console = _get_json(f"http://127.0.0.1:{tracker_port}/v1/console")
|
||||
registered_eps = {
|
||||
f"http://localhost:{10000 + start}" for _, start, _ in shards
|
||||
}
|
||||
found_eps = set()
|
||||
for event in console.get("events", []):
|
||||
if event.get("message") == "node registered":
|
||||
ep = event.get("fields", {}).get("endpoint", "")
|
||||
if ep in registered_eps:
|
||||
found_eps.add(ep)
|
||||
for _, start, _ in shards:
|
||||
expected_ep = f"http://localhost:{10000 + start}"
|
||||
assert expected_ep in found_eps, \
|
||||
f"endpoint {expected_ep} not found in registration events"
|
||||
|
||||
finally:
|
||||
tracker.stop()
|
||||
@@ -80,7 +80,7 @@ def snapshot_doc(snapshot):
|
||||
|
||||
@pytest.fixture
|
||||
def contract_doc(contract):
|
||||
return copy.deepcopy(dict(contract.raw))
|
||||
return contract.to_dict()
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
@@ -746,7 +746,7 @@ def test_the_contract_locks_the_roadmap_thresholds(contract):
|
||||
assert contract.threshold("performance", "quality_pass_with_speed_fail_verdict") == "stop"
|
||||
|
||||
assert contract.threshold("reliability", "synthetic_workers_satisfy_alpha") is False
|
||||
assert contract.threshold("storage", "forbidden_path_prefixes") == ["/home"]
|
||||
assert contract.threshold("storage", "forbidden_path_prefixes") == ("/home",)
|
||||
|
||||
|
||||
def test_the_contract_offers_only_alpha_or_stop(contract):
|
||||
@@ -820,17 +820,26 @@ def test_a_dropped_acceptance_section_is_rejected(contract_doc):
|
||||
parse_alpha_contract(contract_doc)
|
||||
|
||||
|
||||
def test_resealing_a_mutated_contract_changes_its_digest(contract, contract_doc):
|
||||
"""Tamper-evidence, not tamper-proofing: a rewrite is legal but never silent."""
|
||||
def test_resealing_a_mutated_v1_contract_is_rejected(contract, contract_doc):
|
||||
contract_doc["performance"]["min_median_decode_tokens_per_second"] = 0.05
|
||||
resealed = seal_contract(contract_doc)
|
||||
|
||||
# It parses — nothing in-repo can stop a determined rewrite of file plus digest.
|
||||
reparsed = parse_alpha_contract(resealed)
|
||||
with pytest.raises(AlphaContractError, match="trusted pre-execution digest"):
|
||||
parse_alpha_contract(resealed)
|
||||
assert resealed["contract_sha256"] != contract.digest
|
||||
|
||||
# But the digest moved, so the change is a visible diff on a file whose entire
|
||||
# purpose is to not change, and the contract_id still claims to be v1.
|
||||
assert reparsed.digest != contract.digest
|
||||
|
||||
def test_parsed_contract_nested_state_is_immutable(contract):
|
||||
with pytest.raises(TypeError):
|
||||
contract.raw["performance"]["min_median_decode_tokens_per_second"] = 0.05
|
||||
with pytest.raises(TypeError):
|
||||
contract.target["reasoning_effort"] = "high"
|
||||
|
||||
|
||||
def test_contract_to_dict_returns_an_isolated_mutable_copy(contract):
|
||||
copied = contract.to_dict()
|
||||
copied["performance"]["min_median_decode_tokens_per_second"] = 0.05
|
||||
assert contract.threshold("performance", "min_median_decode_tokens_per_second") == 0.5
|
||||
|
||||
|
||||
def test_an_unknown_threshold_cannot_be_invented_at_read_time(contract):
|
||||
|
||||
68
tests/test_llama_cpp_dependency.py
Normal file
68
tests/test_llama_cpp_dependency.py
Normal file
@@ -0,0 +1,68 @@
|
||||
"""Offline guards for DGR-004's pinned llama.cpp dependency boundary."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import pathlib
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
|
||||
ROOT = pathlib.Path(__file__).resolve().parents[1]
|
||||
LLAMA_DIR = ROOT / "packages/node/native/llama"
|
||||
SCRIPT = ROOT / "scripts/llama_cpp_dependency.py"
|
||||
|
||||
|
||||
def _sha256(path: pathlib.Path) -> str:
|
||||
return hashlib.sha256(path.read_bytes()).hexdigest()
|
||||
|
||||
|
||||
def test_lock_and_patch_manifest_are_self_consistent_and_exact() -> None:
|
||||
lock = json.loads((LLAMA_DIR / "UPSTREAM_LOCK.json").read_text())
|
||||
commit = (LLAMA_DIR / "UPSTREAM_COMMIT").read_text().strip()
|
||||
patches = (LLAMA_DIR / "patches/series").read_text().splitlines()
|
||||
sums = {
|
||||
name: digest
|
||||
for digest, name in (
|
||||
line.split(maxsplit=1)
|
||||
for line in (LLAMA_DIR / "patches/SHA256SUMS").read_text().splitlines()
|
||||
if line and not line.startswith("#")
|
||||
)
|
||||
}
|
||||
|
||||
assert commit == lock["commit"]
|
||||
assert len(commit) == 40
|
||||
assert patches == lock["patch_series"]
|
||||
assert patches == sorted(patches)
|
||||
assert patches
|
||||
for patch_name in patches:
|
||||
patch = LLAMA_DIR / "patches" / patch_name
|
||||
assert sums[patch_name] == _sha256(patch)
|
||||
assert "Subject: [PATCH" in patch.read_text()
|
||||
|
||||
|
||||
def test_dependency_script_reports_the_locked_boundary_without_network() -> None:
|
||||
completed = subprocess.run(
|
||||
[sys.executable, str(SCRIPT), "inspect"],
|
||||
cwd=ROOT,
|
||||
check=True,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
report = json.loads(completed.stdout)
|
||||
|
||||
assert report["commit"] == (LLAMA_DIR / "UPSTREAM_COMMIT").read_text().strip()
|
||||
assert report["patch_count"] == 2
|
||||
assert report["model_downloads"] is False
|
||||
assert report["semantic_certification"] is False
|
||||
assert "dense" in report["glm_stock_limitations"].lower()
|
||||
|
||||
|
||||
def test_patch_stack_does_not_contain_meshnet_control_plane_code() -> None:
|
||||
forbidden = ("tracker", "route session", "grpc", "http", "billing", "wallet")
|
||||
patch_text = "\n".join(
|
||||
(LLAMA_DIR / "patches" / name).read_text().lower()
|
||||
for name in (LLAMA_DIR / "patches/series").read_text().splitlines()
|
||||
)
|
||||
assert not any(term in patch_text for term in forbidden)
|
||||
157
tests/test_native_identity_emission.py
Normal file
157
tests/test_native_identity_emission.py
Normal file
@@ -0,0 +1,157 @@
|
||||
"""DGR-003 production-native identity emission boundary tests."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from meshnet_node.doctor import DoctorSelection, validate_loaded_backend
|
||||
from meshnet_node.native_backend import (
|
||||
ImmutableArtifactPin,
|
||||
NativeIdentityInputs,
|
||||
NativeLoadedArtifactReport,
|
||||
NativeNumericalRecipe,
|
||||
NativeSessionRejected,
|
||||
NativeWorkerBackendAdapter,
|
||||
shard_identity_from_native_report,
|
||||
)
|
||||
from meshnet_node.native_protocol import SCHEMA_VERSION, pb
|
||||
from meshnet_node.recipe_manifest import parse_recipe_manifest
|
||||
from meshnet_tracker.capability import STATE_UNCERTIFIED, evaluate_report
|
||||
|
||||
|
||||
def _digest(letter: str) -> str:
|
||||
return letter * 64
|
||||
|
||||
|
||||
def _inputs(**changes: object) -> NativeIdentityInputs:
|
||||
report = NativeLoadedArtifactReport(
|
||||
owned_start_layer=2,
|
||||
owned_end_layer=6,
|
||||
mapped_bytes=1024,
|
||||
resident_bytes=768,
|
||||
registered_bytes=640,
|
||||
architecture="llama",
|
||||
architecture_digest=_digest("a"),
|
||||
layer_count=8,
|
||||
)
|
||||
recipe = NativeNumericalRecipe(
|
||||
weight_quantization="Q4_K_M",
|
||||
activation_dtype="bfloat16",
|
||||
compute_dtype="float32",
|
||||
kv_dtype="q8_0",
|
||||
kv_layout="llama-kv-v1",
|
||||
architecture_adapter="dense-llama-v1",
|
||||
backend_id="llama-cpp",
|
||||
runtime_version="llama.cpp:e920c523",
|
||||
recipe_id="native",
|
||||
recipe_version="1",
|
||||
catalogue_version="2026.07.1",
|
||||
)
|
||||
values: dict[str, object] = {
|
||||
"loaded_artifact": report,
|
||||
"artifact_pin": ImmutableArtifactPin(
|
||||
artifact_id="acme/llama.gguf",
|
||||
revision="0123456789abcdef",
|
||||
content_digest=_digest("b"),
|
||||
),
|
||||
"tokenizer_revision": "abcdef0123456789",
|
||||
"numerical_recipe": recipe,
|
||||
}
|
||||
values.update(changes)
|
||||
return NativeIdentityInputs(**values) # type: ignore[arg-type]
|
||||
|
||||
|
||||
def _open(adapter: NativeWorkerBackendAdapter, **changes: object) -> pb.SessionOpen:
|
||||
identity = adapter.identity
|
||||
fields: dict[str, object] = {
|
||||
"schema_version": SCHEMA_VERSION,
|
||||
"route_session_id": "tracker-session",
|
||||
"route_epoch": 4,
|
||||
"fingerprint": identity.fingerprint.to_proto(),
|
||||
"shard_range": pb.ShardRange(
|
||||
start_layer=identity.shard_start,
|
||||
end_layer=identity.shard_end,
|
||||
effective_start_layer=identity.shard_start,
|
||||
),
|
||||
}
|
||||
fields.update(changes)
|
||||
return pb.SessionOpen(**fields) # type: ignore[arg-type]
|
||||
|
||||
|
||||
def test_native_identity_uses_loaded_report_not_a_caller_range():
|
||||
identity = shard_identity_from_native_report(_inputs())
|
||||
|
||||
assert (identity.shard_start, identity.shard_end) == (2, 6)
|
||||
assert identity.artifact.architecture == "llama"
|
||||
assert identity.artifact.layer_count == 8
|
||||
|
||||
|
||||
def test_native_identity_requires_an_immutable_pin_and_gguf_range():
|
||||
with pytest.raises(Exception, match="moving reference"):
|
||||
shard_identity_from_native_report(
|
||||
_inputs(artifact_pin=ImmutableArtifactPin("a", "main", _digest("b")))
|
||||
)
|
||||
with pytest.raises(Exception, match="outside GGUF"):
|
||||
NativeLoadedArtifactReport(0, 9, 1, 1, 1, "llama", _digest("a"), 8)
|
||||
|
||||
|
||||
def test_native_worker_rejects_bad_session_open_before_session_acceptance():
|
||||
adapter = NativeWorkerBackendAdapter(_inputs())
|
||||
accepted = adapter.on_session_open(
|
||||
_open(adapter), expected_route_session_id="tracker-session", expected_route_epoch=4
|
||||
)
|
||||
assert accepted.fingerprint.SerializeToString() == adapter.identity.fingerprint.to_proto().SerializeToString()
|
||||
|
||||
with pytest.raises(NativeSessionRejected) as rejected:
|
||||
adapter.on_session_open(
|
||||
_open(adapter, route_epoch=5),
|
||||
expected_route_session_id="tracker-session",
|
||||
expected_route_epoch=4,
|
||||
)
|
||||
assert rejected.value.error.code == pb.ERROR_CODE_EPOCH_STALE
|
||||
|
||||
|
||||
def test_doctor_emits_native_identity_but_keeps_legacy_backend_dark():
|
||||
manifest = parse_recipe_manifest(
|
||||
{"schema_version": 1, "catalogue_version": "2026.07.1", "recipes": [
|
||||
{"id": "native", "version": "1", "backend_id": "llama-cpp"}
|
||||
]}
|
||||
)
|
||||
selection = DoctorSelection("acme/llama.gguf", 2, 5)
|
||||
native = NativeWorkerBackendAdapter(_inputs())
|
||||
# The probe needs only the normal backend shape; identity is supplied by the adapter.
|
||||
native.hidden_size = 8
|
||||
native.is_head = False
|
||||
native.is_tail = False
|
||||
native.device = "cpu"
|
||||
native.forward_bytes = lambda *args, **kwargs: type("Payload", (), {"body": b"x", "shape": [1]})()
|
||||
result = validate_loaded_backend(native, selection, manifest.recipes[0], manifest)
|
||||
assert result.report.identity == native.identity
|
||||
assert result.report.model.revision == native.identity.artifact.revision
|
||||
assert result.report.model.config_fingerprint == "sha256:" + _digest("a")
|
||||
assert result.report.backend.quantization == "Q4_K_M"
|
||||
assert (result.report.shard.start, result.report.shard.end) == (2, 5)
|
||||
state = evaluate_report(
|
||||
result.report.to_dict(),
|
||||
model_matches=lambda value: value == "acme/llama.gguf",
|
||||
advertised_model="acme/llama.gguf",
|
||||
shard_start=2,
|
||||
shard_end=5,
|
||||
declared_recipe_id="native",
|
||||
declared_recipe_version="1",
|
||||
now=result.report.validated_at,
|
||||
)
|
||||
assert state.state == STATE_UNCERTIFIED
|
||||
|
||||
class Legacy:
|
||||
hidden_size = 8
|
||||
is_head = False
|
||||
is_tail = False
|
||||
device = "cpu"
|
||||
|
||||
@staticmethod
|
||||
def forward_bytes(*args, **kwargs):
|
||||
return type("Payload", (), {"body": b"x", "shape": [1]})()
|
||||
|
||||
legacy = validate_loaded_backend(Legacy(), selection, manifest.recipes[0], manifest)
|
||||
assert legacy.report.identity is None
|
||||
786
tests/test_runtime_recipe_identity.py
Normal file
786
tests/test_runtime_recipe_identity.py
Normal file
@@ -0,0 +1,786 @@
|
||||
"""Deterministic DGR-003 identity and tracker-admission conformance tests."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import replace
|
||||
import json
|
||||
from pathlib import Path
|
||||
import time
|
||||
|
||||
import pytest
|
||||
|
||||
from meshnet_node.native_protocol import SCHEMA_VERSION, pb
|
||||
from meshnet_node.runtime_recipe import (
|
||||
ArtifactIdentity,
|
||||
CompatibilityFingerprint,
|
||||
DerivativeBinding,
|
||||
RecipeIdentityError,
|
||||
RuntimeRecipe,
|
||||
ShardIdentity,
|
||||
check_handshake,
|
||||
check_session_open,
|
||||
check_route,
|
||||
handshake_error,
|
||||
)
|
||||
from meshnet_tracker.capability import (
|
||||
POLICY_COMPAT,
|
||||
POLICY_ENFORCE,
|
||||
CapabilityState,
|
||||
STATE_ADMITTED,
|
||||
STATE_FINGERPRINT_MISMATCH,
|
||||
STATE_MODEL_MISMATCH,
|
||||
STATE_RECIPE_MISMATCH,
|
||||
STATE_UNCERTIFIED,
|
||||
absent_state,
|
||||
evaluate_report,
|
||||
)
|
||||
from meshnet_tracker.server import (
|
||||
TrackerServer,
|
||||
_capability_from_registration,
|
||||
_find_pinned_route,
|
||||
_NodeEntry,
|
||||
_select_route,
|
||||
)
|
||||
from meshnet_tracker.recipe import (
|
||||
CertificationLedger,
|
||||
DistributedForwardEvidence,
|
||||
RecipeIdentityError as TrackerRecipeIdentityError,
|
||||
parse_identity,
|
||||
)
|
||||
|
||||
VECTORS = Path(__file__).parent / "data" / "recipe_fingerprint_vectors.json"
|
||||
|
||||
|
||||
def _digest(char: str) -> str:
|
||||
return char * 64
|
||||
|
||||
|
||||
def _recipe(**changes: object) -> RuntimeRecipe:
|
||||
fields: dict[str, object] = {
|
||||
"weight_quantization": "Q4_K_M",
|
||||
"activation_dtype": "bfloat16",
|
||||
"compute_dtype": "float32",
|
||||
"kv_dtype": "q8_0",
|
||||
"kv_layout": "paged-v1",
|
||||
"tokenizer_revision": "0123456789abcdef",
|
||||
"architecture_adapter": "llama/range-v1",
|
||||
"backend_id": "llama.cpp",
|
||||
"runtime_version": "llama.cpp@deadbeef+meshnet.1",
|
||||
"recipe_id": "example-gguf",
|
||||
"recipe_version": "1",
|
||||
"catalogue_version": "2026.07.1",
|
||||
}
|
||||
fields.update(changes)
|
||||
return RuntimeRecipe(**fields) # type: ignore[arg-type]
|
||||
|
||||
|
||||
def _identity(start: int = 0, end: int = 4, **recipe_changes: object) -> ShardIdentity:
|
||||
"""A Shard of the whole-model artifact: every node holds the same file."""
|
||||
return ShardIdentity(
|
||||
ArtifactIdentity(
|
||||
"example/model",
|
||||
"0123456789abcdef",
|
||||
_digest("a"),
|
||||
"dense-llama",
|
||||
_digest("b"),
|
||||
8,
|
||||
),
|
||||
_recipe(**recipe_changes),
|
||||
start,
|
||||
end,
|
||||
)
|
||||
|
||||
|
||||
def _split(start: int, end: int, content: str, **recipe_changes: object) -> ShardIdentity:
|
||||
"""A derivative: its own bytes, bound to the exact source it was cut from."""
|
||||
return ShardIdentity(
|
||||
ArtifactIdentity(
|
||||
"example/model",
|
||||
"0123456789abcdef",
|
||||
_digest(content),
|
||||
"dense-llama",
|
||||
_digest("b"),
|
||||
8,
|
||||
DerivativeBinding(_digest("a"), start, end),
|
||||
),
|
||||
_recipe(**recipe_changes),
|
||||
start,
|
||||
end,
|
||||
)
|
||||
|
||||
|
||||
def _report(identity: ShardIdentity) -> dict:
|
||||
return {
|
||||
"schema_version": 1,
|
||||
"model": {
|
||||
"model_id": "example/model",
|
||||
"revision": identity.artifact.revision,
|
||||
"config_fingerprint": "sha256:" + identity.artifact.architecture_digest,
|
||||
},
|
||||
"shard": {"start": identity.shard_start, "end": identity.shard_end - 1},
|
||||
"recipe": identity.recipe.to_dict() | {
|
||||
"recipe_id": identity.recipe.recipe_id,
|
||||
"recipe_version": identity.recipe.recipe_version,
|
||||
"catalogue_version": identity.recipe.catalogue_version,
|
||||
},
|
||||
"backend": {"backend_id": "llama.cpp", "device": "test"},
|
||||
"status": "passed",
|
||||
"validated_at": 100.0,
|
||||
"duration_ms": 1,
|
||||
"diagnostics": [],
|
||||
"identity": identity.to_dict(),
|
||||
}
|
||||
|
||||
|
||||
def _evaluate(report: dict, **kwargs: object):
|
||||
kwargs.setdefault("shard_start", 0)
|
||||
kwargs.setdefault("shard_end", 3)
|
||||
return evaluate_report(
|
||||
report,
|
||||
model_matches=lambda model: model == "example/model",
|
||||
advertised_model="example/model",
|
||||
now=100.0,
|
||||
**kwargs, # type: ignore[arg-type]
|
||||
)
|
||||
|
||||
|
||||
def _register(tracker: TrackerServer, node_id: str, identity: ShardIdentity) -> _NodeEntry:
|
||||
"""Register one node exactly as the HTTP path does: its own report, the tracker's ledger."""
|
||||
report = _report(identity)
|
||||
report["validated_at"] = time.time()
|
||||
# The registry range is end-inclusive; the identity's is end-exclusive.
|
||||
start, end = identity.shard_start, identity.shard_end - 1
|
||||
capability = _capability_from_registration(
|
||||
{"capability_report": report},
|
||||
model="example/model",
|
||||
hf_repo=None,
|
||||
shard_start=start,
|
||||
shard_end=end,
|
||||
recipe_certifications=tracker._recipe_certifications,
|
||||
)
|
||||
entry = _NodeEntry(
|
||||
node_id=node_id,
|
||||
endpoint=f"http://{node_id}",
|
||||
shard_start=start,
|
||||
shard_end=end,
|
||||
model="example/model",
|
||||
shard_checksum=None,
|
||||
hardware_profile={},
|
||||
wallet_address=None,
|
||||
score=1.0,
|
||||
capability=capability,
|
||||
)
|
||||
tracker._registry[node_id] = entry
|
||||
return entry
|
||||
|
||||
|
||||
def _evidence(
|
||||
*shards: ShardIdentity,
|
||||
node_ids: tuple[str, ...] = ("physical-a", "physical-b"),
|
||||
layer_count: int = 8,
|
||||
fingerprint: tuple[str, str] | None = None,
|
||||
**changes: object,
|
||||
) -> DistributedForwardEvidence:
|
||||
presented = tuple(parse_identity(s.to_dict()) for s in shards)
|
||||
fields: dict[str, object] = {
|
||||
"route_session_id": "session",
|
||||
"route_epoch": 1,
|
||||
"node_ids": node_ids,
|
||||
"shard_ranges": tuple((s.shard_start, s.shard_end) for s in shards),
|
||||
"tokens_generated": 1,
|
||||
"layer_count": layer_count,
|
||||
"fingerprint": fingerprint or presented[0].key,
|
||||
"participants": presented,
|
||||
}
|
||||
fields.update(changes)
|
||||
return DistributedForwardEvidence(**fields) # type: ignore[arg-type]
|
||||
|
||||
|
||||
# --- Identity: the axes, the digests, and what they do and do not commit to ---
|
||||
|
||||
|
||||
def test_node_and_tracker_share_the_committed_canonical_fingerprint_vector():
|
||||
for vector in json.loads(VECTORS.read_text(encoding="utf-8"))["vectors"]:
|
||||
expected = vector["fingerprint"]
|
||||
identity = ShardIdentity.from_dict(vector["identity"])
|
||||
presented = parse_identity(vector["identity"])
|
||||
|
||||
assert identity.fingerprint.to_dict() == expected, vector["name"]
|
||||
assert presented.fingerprint_dict() == expected, vector["name"]
|
||||
assert (
|
||||
CompatibilityFingerprint.from_proto(
|
||||
identity.fingerprint.to_proto()
|
||||
).to_dict()
|
||||
== expected
|
||||
), vector["name"]
|
||||
|
||||
# The binding digest is derived independently on both sides too, so a
|
||||
# node and a tracker cannot disagree about which bytes a Shard holds.
|
||||
assert identity.shard_binding_digest == vector["shard_binding_digest"], vector["name"]
|
||||
assert presented.shard_binding_digest == vector["shard_binding_digest"], vector["name"]
|
||||
|
||||
# The DGR-002 wire encoding is contract as well, so the native worker can
|
||||
# be held to these bytes without reimplementing the JSON canonicalizer.
|
||||
wire = identity.fingerprint.to_proto().SerializeToString(deterministic=True)
|
||||
assert wire.hex() == vector["fingerprint_proto_hex"], vector["name"]
|
||||
|
||||
|
||||
def test_committed_vectors_cover_a_whole_model_and_a_derivative_shard():
|
||||
names = {v["name"] for v in json.loads(VECTORS.read_text(encoding="utf-8"))["vectors"]}
|
||||
assert {"example-v1", "example-v1-derivative"} <= names
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"axis,value",
|
||||
[
|
||||
("weight_quantization", "Q5_K_M"),
|
||||
("activation_dtype", "float16"),
|
||||
("compute_dtype", "float16"),
|
||||
("kv_dtype", "float16"),
|
||||
("kv_layout", "contiguous-v2"),
|
||||
("tokenizer_revision", "fedcba9876543210"),
|
||||
("architecture_adapter", "llama/range-v2"),
|
||||
("backend_id", "other-backend"),
|
||||
("runtime_version", "llama.cpp@other+meshnet.1"),
|
||||
("boundary_schema_version", 2),
|
||||
("protocol_schema_version", 2),
|
||||
],
|
||||
)
|
||||
def test_every_recipe_axis_changes_the_fingerprint_and_blocks_route(axis, value):
|
||||
left = _identity()
|
||||
right = _identity(4, 8, **{axis: value})
|
||||
|
||||
assert left.fingerprint.key != right.fingerprint.key
|
||||
assert check_route([left, right])
|
||||
|
||||
|
||||
def test_split_artifact_is_bound_to_exact_source_and_owned_range():
|
||||
source = _identity()
|
||||
split = _split(4, 8, "c")
|
||||
|
||||
assert source.fingerprint.matches(split.fingerprint)
|
||||
with pytest.raises(RecipeIdentityError):
|
||||
ShardIdentity(split.artifact, split.recipe, 3, 8)
|
||||
|
||||
|
||||
def test_derivative_bytes_and_range_have_a_separate_shard_binding_digest():
|
||||
left = _split(0, 4, "c")
|
||||
changed_bytes = _split(0, 4, "d")
|
||||
changed_range = _split(4, 8, "c")
|
||||
|
||||
# Same route fingerprint — all three are the same recipe on the same source,
|
||||
# which is exactly what route formation should conclude.
|
||||
assert left.fingerprint.key == changed_bytes.fingerprint.key
|
||||
assert left.fingerprint.key == changed_range.fingerprint.key
|
||||
# Different bytes and different ranges are still separately pinned.
|
||||
assert left.shard_binding_digest != changed_bytes.shard_binding_digest
|
||||
assert left.shard_binding_digest != changed_range.shard_binding_digest
|
||||
assert (
|
||||
parse_identity(left.to_dict()).shard_binding_digest
|
||||
== left.shard_binding_digest
|
||||
)
|
||||
|
||||
|
||||
def test_declared_digest_mismatch_is_recomputed_and_rejected_not_authenticated():
|
||||
report = _report(_identity())
|
||||
report["identity"]["fingerprint"]["runtime_recipe_digest"] = _digest("f")
|
||||
|
||||
assert _evaluate(report).state == STATE_FINGERPRINT_MISMATCH
|
||||
|
||||
|
||||
# --- Capability admission: the identity block must match the proof it rides with ---
|
||||
|
||||
|
||||
def test_capability_identity_must_match_the_proof_labels():
|
||||
identity = _identity()
|
||||
|
||||
wrong_model = _report(identity)
|
||||
wrong_model["identity"]["artifact"]["artifact_id"] = "other/model"
|
||||
wrong_model["identity"]["fingerprint"] = None
|
||||
assert _evaluate(wrong_model).state == STATE_MODEL_MISMATCH
|
||||
|
||||
wrong_recipe = _report(identity)
|
||||
wrong_recipe["recipe"]["recipe_id"] = "other-recipe"
|
||||
assert _evaluate(wrong_recipe).state == STATE_RECIPE_MISMATCH
|
||||
|
||||
wrong_backend = _report(identity)
|
||||
wrong_backend["backend"]["backend_id"] = "other-backend"
|
||||
assert _evaluate(wrong_backend).state == STATE_RECIPE_MISMATCH
|
||||
|
||||
wrong_quantization = _report(identity)
|
||||
wrong_quantization["backend"]["quantization"] = "Q5_K_M"
|
||||
assert _evaluate(wrong_quantization).state == STATE_RECIPE_MISMATCH
|
||||
|
||||
|
||||
def test_capability_identity_must_match_the_proven_revision_and_config():
|
||||
identity = _identity()
|
||||
|
||||
wrong_revision = _report(identity)
|
||||
wrong_revision["model"]["revision"] = "fedcba9876543210"
|
||||
assert _evaluate(wrong_revision).state == STATE_MODEL_MISMATCH
|
||||
|
||||
wrong_config = _report(identity)
|
||||
wrong_config["model"]["config_fingerprint"] = "sha256:" + _digest("c")
|
||||
assert _evaluate(wrong_config).state == STATE_MODEL_MISMATCH
|
||||
|
||||
missing_config = _report(identity)
|
||||
del missing_config["model"]["config_fingerprint"]
|
||||
assert _evaluate(missing_config).state == STATE_MODEL_MISMATCH
|
||||
|
||||
|
||||
def test_an_exact_identity_without_a_ledger_is_dark_not_admitted():
|
||||
"""No certification authority is not permission to serve — it is no proof."""
|
||||
state = _evaluate(_report(_identity()), ledger=None)
|
||||
|
||||
assert state.state == STATE_UNCERTIFIED
|
||||
assert not state.routable_under(POLICY_COMPAT)
|
||||
assert not state.routable_under(POLICY_ENFORCE)
|
||||
|
||||
|
||||
def test_admission_records_the_rederived_binding_not_the_declared_one():
|
||||
split = _split(0, 4, "c")
|
||||
state = _evaluate(_report(split), ledger=CertificationLedger())
|
||||
|
||||
assert state.shard_binding_digest == split.shard_binding_digest
|
||||
assert state.fingerprint == split.fingerprint.key
|
||||
|
||||
|
||||
# --- Certification: only the tracker, only on evidence it re-derived itself ---
|
||||
|
||||
|
||||
def test_certification_requires_prior_dark_registration():
|
||||
ledger = CertificationLedger()
|
||||
identity = parse_identity(_identity().to_dict())
|
||||
|
||||
with pytest.raises(TrackerRecipeIdentityError, match="not registered"):
|
||||
ledger.certify(identity, _evidence(_identity(0, 4), _identity(4, 8)))
|
||||
|
||||
|
||||
def test_synthetic_or_mismatched_evidence_never_certifies():
|
||||
identity = _identity()
|
||||
ledger = CertificationLedger()
|
||||
assert _evaluate(_report(identity), ledger=ledger).state == STATE_UNCERTIFIED
|
||||
presented = parse_identity(identity.to_dict())
|
||||
|
||||
# A unit fixture is not a distributed forward. This is the trust boundary.
|
||||
with pytest.raises(TrackerRecipeIdentityError, match="synthetic"):
|
||||
ledger.certify(
|
||||
presented, _evidence(_identity(0, 4), _identity(4, 8), synthetic=True)
|
||||
)
|
||||
|
||||
with pytest.raises(TrackerRecipeIdentityError, match="fingerprint does not match"):
|
||||
ledger.certify(
|
||||
presented,
|
||||
_evidence(
|
||||
_identity(0, 4),
|
||||
_identity(4, 8),
|
||||
fingerprint=(_digest("e"), _digest("f")),
|
||||
),
|
||||
)
|
||||
|
||||
# A single node, however real, is not a distributed forward.
|
||||
with pytest.raises(TrackerRecipeIdentityError, match="distributed forward requires"):
|
||||
ledger.certify(
|
||||
presented, _evidence(_identity(0, 8), node_ids=("physical-a",))
|
||||
)
|
||||
|
||||
# A hole in the coverage means those layers were never computed.
|
||||
with pytest.raises(TrackerRecipeIdentityError, match="owned by no Shard"):
|
||||
ledger.certify(
|
||||
presented, _evidence(_identity(0, 3), _identity(4, 8))
|
||||
)
|
||||
|
||||
|
||||
def test_certification_evidence_layer_count_cannot_be_substituted():
|
||||
"""An 8-layer recipe cannot be promoted by evidence for a 2-layer route."""
|
||||
identity = _identity()
|
||||
ledger = CertificationLedger()
|
||||
assert _evaluate(_report(identity), ledger=ledger).state == STATE_UNCERTIFIED
|
||||
|
||||
with pytest.raises(TrackerRecipeIdentityError, match="layer count"):
|
||||
ledger.certify(
|
||||
parse_identity(identity.to_dict()),
|
||||
_evidence(_identity(0, 4), _identity(4, 8), layer_count=2),
|
||||
)
|
||||
|
||||
|
||||
def test_evidence_participants_must_be_the_recipe_being_promoted():
|
||||
identity = _identity()
|
||||
ledger = CertificationLedger()
|
||||
assert _evaluate(_report(identity), ledger=ledger).state == STATE_UNCERTIFIED
|
||||
|
||||
# Participants running a different recipe cannot vouch for this one, even
|
||||
# when the evidence header names the right fingerprint.
|
||||
with pytest.raises(TrackerRecipeIdentityError, match="participant fingerprint"):
|
||||
ledger.certify(
|
||||
parse_identity(identity.to_dict()),
|
||||
_evidence(
|
||||
_identity(0, 4),
|
||||
_identity(4, 8, kv_dtype="float16"),
|
||||
fingerprint=identity.fingerprint.key,
|
||||
),
|
||||
)
|
||||
|
||||
# Nor can a participant whose identity is not the range it is recorded under.
|
||||
with pytest.raises(TrackerRecipeIdentityError, match="recorded effective range"):
|
||||
ledger.certify(
|
||||
parse_identity(identity.to_dict()),
|
||||
_evidence(
|
||||
_identity(0, 4),
|
||||
_identity(4, 8),
|
||||
shard_ranges=((0, 4), (0, 4)),
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def test_tracker_owns_the_only_promotion_path_and_re_admits_dark_nodes():
|
||||
tracker = TrackerServer()
|
||||
head, tail = _split(0, 4, "c"), _split(4, 8, "d")
|
||||
node_a = _register(tracker, "physical-a", head)
|
||||
node_b = _register(tracker, "physical-b", tail)
|
||||
# A node on a *different* recipe, which must not be swept up by the promotion.
|
||||
other = _register(tracker, "physical-c", _identity(0, 8, kv_dtype="float16"))
|
||||
|
||||
assert node_a.capability.state == STATE_UNCERTIFIED
|
||||
assert node_b.capability.state == STATE_UNCERTIFIED
|
||||
|
||||
status = tracker.certify_recipe(
|
||||
parse_identity(head.to_dict()), _evidence(head, tail)
|
||||
)
|
||||
|
||||
assert status.may_serve
|
||||
assert node_a.capability.state == STATE_ADMITTED
|
||||
assert node_b.capability.state == STATE_ADMITTED
|
||||
assert other.capability.state == STATE_UNCERTIFIED
|
||||
# A node registering after the fact lands admitted, from the same ledger.
|
||||
assert _register(tracker, "physical-d", head).capability.state == STATE_ADMITTED
|
||||
|
||||
|
||||
def test_the_network_map_certification_field_tracks_the_ledger():
|
||||
"""The map must say what the ledger knows — "dark" before, "certified" after."""
|
||||
tracker = TrackerServer()
|
||||
head, tail = _split(0, 4, "c"), _split(4, 8, "d")
|
||||
node_a = _register(tracker, "physical-a", head)
|
||||
node_b = _register(tracker, "physical-b", tail)
|
||||
|
||||
assert node_a.capability.certification == "dark"
|
||||
assert node_a.capability.to_dict()["certification"] == "dark"
|
||||
|
||||
tracker.certify_recipe(parse_identity(head.to_dict()), _evidence(head, tail))
|
||||
|
||||
assert node_a.capability.certification == "certified"
|
||||
assert node_b.capability.certification == "certified"
|
||||
assert _register(tracker, "physical-d", head).capability.certification == "certified"
|
||||
# A node that presented no identity has nothing for the ledger to say.
|
||||
assert absent_state().certification is None
|
||||
|
||||
|
||||
def test_certification_rejects_participants_the_tracker_did_not_admit():
|
||||
tracker = TrackerServer()
|
||||
head, tail = _split(0, 4, "c"), _split(4, 8, "d")
|
||||
_register(tracker, "physical-a", head)
|
||||
_register(tracker, "physical-b", tail)
|
||||
promoted = parse_identity(head.to_dict())
|
||||
|
||||
# A node nobody registered cannot have served the forward.
|
||||
with pytest.raises(TrackerRecipeIdentityError, match="not registered"):
|
||||
tracker.certify_recipe(
|
||||
promoted, _evidence(head, tail, node_ids=("physical-a", "ghost"))
|
||||
)
|
||||
|
||||
# Same recipe, same ranges, but derivative bytes neither node was admitted
|
||||
# on: certification must not attach to blobs that never ran.
|
||||
with pytest.raises(TrackerRecipeIdentityError, match="binding this tracker did not admit"):
|
||||
tracker.certify_recipe(
|
||||
promoted, _evidence(_split(0, 4, "e"), _split(4, 8, "f"))
|
||||
)
|
||||
|
||||
# Right bytes, but swapped between the nodes that served them.
|
||||
with pytest.raises(TrackerRecipeIdentityError, match="range differs|binding this tracker"):
|
||||
tracker.certify_recipe(
|
||||
promoted, _evidence(tail, head)
|
||||
)
|
||||
|
||||
assert tracker._registry["physical-a"].capability.state == STATE_UNCERTIFIED
|
||||
|
||||
|
||||
def test_a_legacy_node_cannot_be_an_exact_certification_participant():
|
||||
tracker = TrackerServer()
|
||||
head, tail = _split(0, 4, "c"), _split(4, 8, "d")
|
||||
_register(tracker, "physical-a", head)
|
||||
|
||||
legacy = _NodeEntry(
|
||||
node_id="physical-b",
|
||||
endpoint="http://physical-b",
|
||||
shard_start=4,
|
||||
shard_end=7,
|
||||
model="example/model",
|
||||
shard_checksum=None,
|
||||
hardware_profile={},
|
||||
wallet_address=None,
|
||||
score=1.0,
|
||||
capability=CapabilityState(state=STATE_ADMITTED, model_id="example/model"),
|
||||
)
|
||||
tracker._registry["physical-b"] = legacy
|
||||
|
||||
with pytest.raises(TrackerRecipeIdentityError, match="not admitted under"):
|
||||
tracker.certify_recipe(parse_identity(head.to_dict()), _evidence(head, tail))
|
||||
|
||||
|
||||
def test_certification_does_not_survive_a_tracker_restart():
|
||||
"""The ledger is in-memory. A restart loses it, and that must fail *closed*."""
|
||||
tracker = TrackerServer()
|
||||
head, tail = _split(0, 4, "c"), _split(4, 8, "d")
|
||||
_register(tracker, "physical-a", head)
|
||||
_register(tracker, "physical-b", tail)
|
||||
tracker.certify_recipe(parse_identity(head.to_dict()), _evidence(head, tail))
|
||||
|
||||
restarted = TrackerServer()
|
||||
assert _register(restarted, "physical-a", head).capability.state == STATE_UNCERTIFIED
|
||||
|
||||
|
||||
# --- Route formation: one route, one exact fingerprint ---
|
||||
|
||||
|
||||
def _route_node(
|
||||
node_id: str,
|
||||
start: int,
|
||||
end: int,
|
||||
fingerprint: tuple[str, str] | None,
|
||||
*,
|
||||
speed: float = 1.0,
|
||||
state: str = STATE_ADMITTED,
|
||||
) -> _NodeEntry:
|
||||
capability = CapabilityState(
|
||||
state=state,
|
||||
model_id="example/model",
|
||||
shard_start=start,
|
||||
shard_end=end,
|
||||
model_artifact_digest=None if fingerprint is None else fingerprint[0],
|
||||
runtime_recipe_digest=None if fingerprint is None else fingerprint[1],
|
||||
)
|
||||
return _NodeEntry(
|
||||
node_id=node_id,
|
||||
endpoint=f"http://{node_id}",
|
||||
shard_start=start,
|
||||
shard_end=end,
|
||||
model="example/model",
|
||||
shard_checksum=None,
|
||||
hardware_profile={},
|
||||
wallet_address=None,
|
||||
score=1.0,
|
||||
benchmark_tokens_per_sec=speed,
|
||||
capability=capability,
|
||||
)
|
||||
|
||||
|
||||
def test_route_selection_never_mixes_exact_fingerprints():
|
||||
key_a = (_digest("a"), _digest("b"))
|
||||
key_b = (_digest("c"), _digest("d"))
|
||||
route, error = _select_route(
|
||||
[_route_node("a", 0, 3, key_a), _route_node("b", 4, 7, key_b)],
|
||||
0,
|
||||
7,
|
||||
policy=POLICY_ENFORCE,
|
||||
)
|
||||
assert route == []
|
||||
assert "no route available" in error
|
||||
|
||||
|
||||
def test_route_selection_never_mixes_an_exact_shard_with_a_legacy_one():
|
||||
"""A node with no canonical digests cannot complete an exact route."""
|
||||
key_a = (_digest("a"), _digest("b"))
|
||||
route, error = _select_route(
|
||||
[_route_node("exact-head", 0, 3, key_a), _route_node("legacy-tail", 4, 7, None)],
|
||||
0,
|
||||
7,
|
||||
policy=POLICY_COMPAT,
|
||||
)
|
||||
assert route == []
|
||||
assert "no route available" in error
|
||||
|
||||
|
||||
def test_route_selection_falls_back_to_a_complete_fingerprint_group():
|
||||
key_a = (_digest("a"), _digest("b"))
|
||||
key_b = (_digest("c"), _digest("d"))
|
||||
route, error = _select_route(
|
||||
[
|
||||
_route_node("fast-incomplete", 0, 3, key_b, speed=100.0),
|
||||
_route_node("a-head", 0, 3, key_a),
|
||||
_route_node("a-tail", 4, 7, key_a),
|
||||
],
|
||||
0,
|
||||
7,
|
||||
policy=POLICY_ENFORCE,
|
||||
)
|
||||
assert error == ""
|
||||
assert [node.node_id for node in route] == ["a-head", "a-tail"]
|
||||
|
||||
|
||||
def test_a_homogeneous_legacy_fleet_routes_exactly_as_before():
|
||||
"""DGR-003 partitions routes; it must not change a fleet that has no identity."""
|
||||
route, error = _select_route(
|
||||
[
|
||||
_route_node("slow", 0, 3, None, speed=1.0),
|
||||
_route_node("fast", 0, 3, None, speed=50.0),
|
||||
_route_node("tail", 4, 7, None),
|
||||
],
|
||||
0,
|
||||
7,
|
||||
policy=POLICY_COMPAT,
|
||||
)
|
||||
assert error == ""
|
||||
assert [node.node_id for node in route] == ["fast", "tail"]
|
||||
|
||||
|
||||
def test_an_uncertified_node_is_not_routable_under_either_policy():
|
||||
key = (_digest("a"), _digest("b"))
|
||||
nodes = [
|
||||
_route_node("head", 0, 3, key, state=STATE_UNCERTIFIED),
|
||||
_route_node("tail", 4, 7, key, state=STATE_UNCERTIFIED),
|
||||
]
|
||||
for policy in (POLICY_COMPAT, POLICY_ENFORCE):
|
||||
route, error = _select_route(nodes, 0, 7, policy=policy)
|
||||
assert route == [], policy
|
||||
assert "no route available" in error
|
||||
|
||||
|
||||
def test_pinned_benchmark_routes_obey_the_same_partition_rule():
|
||||
"""US-030 benchmark combos run real inference; they may not mix identities."""
|
||||
key_a = (_digest("a"), _digest("b"))
|
||||
key_b = (_digest("c"), _digest("d"))
|
||||
|
||||
# An exact head with only a legacy tail, or a tail on another fingerprint,
|
||||
# has no two-hop combo at all.
|
||||
assert _find_pinned_route(
|
||||
[_route_node("exact", 0, 3, key_a), _route_node("legacy", 4, 7, None)], 0, 7, 2
|
||||
) is None
|
||||
assert _find_pinned_route(
|
||||
[_route_node("a", 0, 3, key_a), _route_node("b", 4, 7, key_b)], 0, 7, 2
|
||||
) is None
|
||||
|
||||
# Homogeneous fleets — exact or legacy — still form pinned combos.
|
||||
exact = _find_pinned_route(
|
||||
[_route_node("a", 0, 3, key_a), _route_node("b", 4, 7, key_a)], 0, 7, 2
|
||||
)
|
||||
assert exact is not None and [node.node_id for node in exact] == ["a", "b"]
|
||||
legacy = _find_pinned_route(
|
||||
[_route_node("a", 0, 3, None), _route_node("b", 4, 7, None)], 0, 7, 2
|
||||
)
|
||||
assert legacy is not None and [node.node_id for node in legacy] == ["a", "b"]
|
||||
|
||||
|
||||
# --- gRPC handshake (DGR-002 SessionOpen) ---
|
||||
|
||||
|
||||
def _session_open(identity: ShardIdentity, **changes: object) -> "pb.SessionOpen":
|
||||
fields: dict[str, object] = {
|
||||
"schema_version": SCHEMA_VERSION,
|
||||
"route_session_id": "session",
|
||||
"route_epoch": 1,
|
||||
"fingerprint": identity.fingerprint.to_proto(),
|
||||
"shard_range": pb.ShardRange(
|
||||
start_layer=identity.shard_start,
|
||||
end_layer=identity.shard_end,
|
||||
effective_start_layer=identity.shard_start,
|
||||
),
|
||||
}
|
||||
fields.update(changes)
|
||||
return pb.SessionOpen(**fields) # type: ignore[arg-type]
|
||||
|
||||
|
||||
def test_session_open_accepts_the_exact_local_shard():
|
||||
local = _identity()
|
||||
assert (
|
||||
check_session_open(
|
||||
local,
|
||||
_session_open(local),
|
||||
expected_route_session_id="session",
|
||||
expected_route_epoch=1,
|
||||
)
|
||||
== ()
|
||||
)
|
||||
assert handshake_error(()) is None
|
||||
|
||||
|
||||
def test_session_open_rejects_a_matching_fingerprint_with_the_wrong_range():
|
||||
local = _identity()
|
||||
opened = _session_open(
|
||||
local,
|
||||
shard_range=pb.ShardRange(start_layer=0, end_layer=5, effective_start_layer=0),
|
||||
)
|
||||
|
||||
mismatches = check_session_open(local, opened)
|
||||
assert mismatches
|
||||
assert handshake_error(mismatches).code == pb.ERROR_CODE_SHARD_RANGE_MISMATCH
|
||||
|
||||
|
||||
def test_session_open_rejects_an_effective_start_outside_the_shard():
|
||||
local = _identity(4, 8)
|
||||
opened = _session_open(
|
||||
local,
|
||||
shard_range=pb.ShardRange(start_layer=4, end_layer=8, effective_start_layer=8),
|
||||
)
|
||||
|
||||
mismatches = check_session_open(local, opened)
|
||||
assert mismatches
|
||||
assert handshake_error(mismatches).code == pb.ERROR_CODE_SHARD_RANGE_MISMATCH
|
||||
|
||||
|
||||
def test_session_open_rejects_a_foreign_protocol_schema():
|
||||
local = _identity()
|
||||
mismatches = check_session_open(
|
||||
local, _session_open(local, schema_version=SCHEMA_VERSION + 1)
|
||||
)
|
||||
|
||||
assert mismatches
|
||||
# A schema disagreement is its own protocol outcome — not a range problem,
|
||||
# and not a fingerprint problem either: the peer may hold the right recipe
|
||||
# and simply speak a schema this node cannot.
|
||||
assert handshake_error(mismatches).code == pb.ERROR_CODE_SCHEMA_UNSUPPORTED
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("changes", "expected_session", "expected_epoch"),
|
||||
[
|
||||
({"route_session_id": "other-session"}, "session", 1),
|
||||
({"route_epoch": 2}, "session", 1),
|
||||
({"route_session_id": ""}, None, None),
|
||||
({"route_epoch": 0}, None, None),
|
||||
],
|
||||
)
|
||||
def test_session_open_rejects_missing_or_stale_tracker_route_assignment(
|
||||
changes, expected_session, expected_epoch
|
||||
):
|
||||
local = _identity()
|
||||
mismatches = check_session_open(
|
||||
local,
|
||||
_session_open(local, **changes),
|
||||
expected_route_session_id=expected_session,
|
||||
expected_route_epoch=expected_epoch,
|
||||
)
|
||||
|
||||
assert mismatches
|
||||
assert handshake_error(mismatches).code == pb.ERROR_CODE_EPOCH_STALE
|
||||
|
||||
|
||||
def test_a_digest_disagreement_dominates_the_handshake_error_code():
|
||||
"""Wrong fingerprint plus wrong range is a wrong peer, not a re-routable one."""
|
||||
local = _identity()
|
||||
opened = _session_open(
|
||||
_identity(kv_dtype="float16"),
|
||||
shard_range=pb.ShardRange(start_layer=0, end_layer=5, effective_start_layer=0),
|
||||
)
|
||||
|
||||
mismatches = check_session_open(local, opened)
|
||||
assert mismatches
|
||||
assert handshake_error(mismatches).code == pb.ERROR_CODE_FINGERPRINT_MISMATCH
|
||||
|
||||
|
||||
def test_grpc_handshake_uses_the_dgr_002_fingerprint_and_fails_closed():
|
||||
local = _identity()
|
||||
remote = replace(local.fingerprint, runtime_recipe_digest=_digest("f")).to_proto()
|
||||
|
||||
mismatches = check_handshake(local, remote)
|
||||
assert mismatches
|
||||
assert handshake_error(mismatches).code == pb.ERROR_CODE_FINGERPRINT_MISMATCH
|
||||
Reference in New Issue
Block a user