popov/neuron-tai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: popov:7364ed673144873ec4341169a3e1d2bd1308f64a
Branches Tags
popov:master popov:ralph/distributed-gguf-runtime popov:ralph/distributed-performance popov:ralph/deepseek-v4-flash-epic popov:ralph/dgr-001-performance-contract popov:cursor/fix-meshnet-node-param-parsing popov:worktree-feat+us-016
...
compare: popov:ralph/dgr-001-performance-contract
Branches Tags
popov:master popov:ralph/distributed-gguf-runtime popov:ralph/distributed-performance popov:ralph/deepseek-v4-flash-epic popov:ralph/dgr-001-performance-contract popov:cursor/fix-meshnet-node-param-parsing popov:worktree-feat+us-016

14 Commits
7364ed6731 ... ralph/dgr-

Author SHA1 Message Date
Dobromir Popov
cae7c2b171 chore: triage maintenance review and close completed stories 2026-07-14 14:33:09 +03:00
Dobromir Popov
64f83d4392 feat: MAINT-002 - Update evidence READMEs for all completed stories 2026-07-14 14:23:27 +03:00
Dobromir Popov
454a681a50 feat: MAINT-001 - Fix Ruff violations across all Python source 2026-07-14 14:17:23 +03:00
Dobromir Popov
a0f28b5631 chore: preserve DGR-018 preflight scripts (postponed) 2026-07-14 14:02:10 +03:00
Dobromir Popov
7925e5253d feat: implement DGR-006 tensor bundle boundary 2026-07-14 13:52:57 +03:00
Dobromir Popov
91c450840d chore: DGR-005 evidence README 2026-07-14 13:33:20 +03:00
Dobromir Popov
d6b808dcf9 chore: mark DGR-005 passes:true in PRD 2026-07-14 13:29:54 +03:00
Dobromir Popov
31065c0e12 feat: distributed GGUF shard load integration test with TinyLlama 1.1B 2026-07-14 13:01:51 +03:00
Dobromir Popov
ec36290863 feat: emit native DGR-003 shard identity 2026-07-14 11:31:10 +03:00
Dobromir Popov
f844ae6567 feat: DGR-005B endpoint ownership and graph guard 2026-07-14 11:14:35 +03:00
Dobromir Popov
252d131e7d feat: DGR-005A dense Llama owned range loader 2026-07-14 11:01:28 +03:00
Dobromir Popov
3d8f93f4aa feat: DGR-005-003-CHAIN - DGR-005 + DGR-003-emission + anchor 2026-07-14 10:48:59 +03:00
Dobromir Popov
f9722e7b57 feat: DGR-004-CHAIN - Execute chained DGR-004/005/003-emission with anchor 2026-07-14 10:34:38 +03:00
Dobromir Popov
7b8e467c6b fix: harden DGR-003 identity trust boundary 2026-07-14 09:48:42 +03:00
86 changed files with 4716 additions and 753 deletions
Expand all files Collapse all files

1
.claude/memory/MEMORY.md
View File

@@ -9,3 +9,4 @@
- **Distributed relay performance** — relay `/rpc` requester sockets are persistent per Route Session and Activation Seam as of 2026-07-10; `request_id` remains unique per activation while `X-Meshnet-Session` remains stable for KV state. Next low-risk priorities: persistent direct/loopback HTTP, seam byte/latency telemetry, then trace-driven zstd tuning.
- **Distributed GGUF direction** — benchmark-gated native runtime: compare controlled Transformers/safetensors and whole-model llama.cpp lanes before expensive work; ship only for measured speed or model-fit advantage. Public parallelism is contiguous Shards in an Inference Route; concurrency comes from per-node continuous batching across isolated Route Sessions, while tensor/expert collectives stay inside optional trusted composite providers. Native data plane uses versioned Protobuf over long-lived gRPC/HTTP2 seam streams, with existing relay carrying the same opaque frames when needed. llama.cpp/GGML remains the substrate behind a project-owned standalone worker and small pinned fork; vLLM is an optional complete managed provider and concept donor, not a fork. Nakshatra, `prima.cpp`, `llama-gguf`, LiGGUF and historical GPUStack are source/test donors only. Active plan: [README](../../.scratch/distributed-gguf-runtime/README.md), [architecture](../../.scratch/distributed-gguf-runtime/architecture.md), [PRD](../../.scratch/distributed-gguf-runtime/PRD.md), [Ralph backlog](../../.scratch/distributed-gguf-runtime/prd.json). Research: [landscape](../../docs/research/distributed-gguf-landscape.md), [GitHub follow-up](../../docs/research/distributed-gguf-github-followup.md), [vLLM](../../docs/research/vllm-distributed-gguf-assessment.md).
- [DGR ROCm setup](dgr-rocm-setup.md) — version-matched TheRock SDK layout, relocated devel payload, verified `gfx1151` HIP llama.cpp build, and GPU-diagnostic boundary.
- **DGR-004 llama.cpp boundary** — `packages/node/native/llama/` locks `e920c523e3b8a0163fe498af5bf90df35ff51d25`, with a one-patch CMake marker and fail-closed clean materialize/apply/build/smoke harness. This is infrastructure only; stock GLM dense fallback remains uncertified.

96
.scratch/distributed-gguf-runtime/PRD.md
View File

@@ -44,6 +44,102 @@ Real-model/hardware stories must:
Before a story is marked complete, run the full deterministic `pytest -q` suite or record the exact pre-existing unrelated failure with a clean-tree reproduction.
## Dependency Graph and Status
Status as of 2026-07-14 (MAINT-003). Authoritative per-story status is
`passes` in [prd.json](prd.json); closed issues live in
`docs/issues/distributed-gguf-runtime/`, open and blocked issues in
[issues/](issues/).
```mermaid
graph TD
classDef done fill:#c8e6c9,stroke:#2e7d32;
classDef blocked fill:#ffcdd2,stroke:#c62828;
DGR001[DGR-001 perf contract]:::done
DGR002[DGR-002 gRPC Shard protocol]:::done
DGR003[DGR-003 artifact/recipe identity]:::done
DGR004[DGR-004 pinned llama.cpp patch stack]:::done
DGR005[DGR-005 dense-Llama range ownership]:::done
DGR006[DGR-006 boundary input/output]:::done
DGR017[DGR-017 GLM-5.2 target/alpha contract]:::done
DGR018[DGR-018 whole-model GLM oracle]:::blocked
DGR019[DGR-019 GLM range/DSA/IndexShare]:::blocked
DGR020[DGR-020 distributed GLM alpha]:::blocked
DGR007[DGR-007 Hot KV State]
DGR008[DGR-008 C++ gRPC worker]
DGR009[DGR-009 Meshnet integration]
DGR010[DGR-010 local two-process acceptance]
DGR011[DGR-011 two-machine route]
DGR012[DGR-012 continuous batching]
DGR013[DGR-013 failure/cancel/restart]
DGR014[DGR-014 release gate]
DGR015[DGR-015 Qwen3 adapter]
DGR016[DGR-016 upstream package]
DGR002 --> DGR003
DGR017 --> DGR003
DGR001 --> DGR004
DGR017 --> DGR004
DGR003 --> DGR005
DGR004 --> DGR005
DGR002 --> DGR006
DGR005 --> DGR006
DGR001 --> DGR017
DGR002 --> DGR017
DGR003 --> DGR018
DGR004 --> DGR018
DGR017 --> DGR018
DGR005 --> DGR019
DGR006 --> DGR019
DGR018 --> DGR019
DGR006 --> DGR007
DGR019 --> DGR007
DGR002 --> DGR008
DGR003 --> DGR008
DGR004 --> DGR008
DGR006 --> DGR008
DGR007 --> DGR008
DGR003 --> DGR009
DGR008 --> DGR009
DGR009 --> DGR010
DGR010 --> DGR011
DGR007 --> DGR012
DGR009 --> DGR012
DGR010 --> DGR012
DGR008 --> DGR013
DGR009 --> DGR013
DGR001 --> DGR014
DGR011 --> DGR014
DGR012 --> DGR014
DGR013 --> DGR014
DGR014 --> DGR015
DGR010 --> DGR016
DGR007 --> DGR020
DGR008 --> DGR020
DGR009 --> DGR020
DGR011 --> DGR020
DGR013 --> DGR020
DGR017 --> DGR020
DGR018 --> DGR020
DGR019 --> DGR020
```
- **Done (`passes: true`):** DGR-001, DGR-002, DGR-003, DGR-004, DGR-005,
DGR-006, DGR-017.
- **Blocked on hardware:** DGR-018 requires a 256-GiB-class host with at least
224 GiB runtime-accessible memory and 250 GB free storage outside `/home`;
no such host is currently available (development host: 124.9 GiB MemTotal).
Exact preflight output: [evidence/DGR-018/BLOCKED.md](evidence/DGR-018/BLOCKED.md).
DGR-019 (needs the DGR-018 oracle) and DGR-020 (needs DGR-018/DGR-019 plus
multiple physical consumer nodes) are blocked transitively.
- **Consequence of the graph as written:** DGR-007 depends on DGR-019, so every
remaining story (DGR-007 through DGR-016) is transitively blocked on the
256-GiB host. Unblocking the generic dense pipeline without that host would
require an explicit re-planning decision to relax the DGR-007 → DGR-019
dependency; that decision is out of scope for maintenance and has not been
made.
## User Stories
### DGR-001: Lock the safetensors-versus-GGUF performance contract

103
.scratch/distributed-gguf-runtime/evidence/DGR-003/README.md
View File

@@ -3,10 +3,15 @@
Evidence class: deterministic offline/unit. No model payload, GPU, external API,
network node, or API credit is required or claimed.
## Result
## Result — delayed-review repair, 2026-07-14
DGR-003 defines an exact, model-agnostic compatibility identity and connects it
to both DGR-002's gRPC `Fingerprint` and tracker capability admission.
DGR-003 defines and tests an exact, model-agnostic compatibility identity and
connects it to DGR-002's gRPC `Fingerprint` plus tracker parsing, admission,
route partitioning, and certification. It is **not complete**: the existing
production doctor/backend path still emits the legacy capability report without
constructing a `ShardIdentity` from authoritative loaded artifact/runtime state.
No exact recipe is therefore claimed live or routable from that path; supplied
exact identities remain dark until tracker-owned certification.
A matching digest proves canonical consistency, **not node authenticity or real
execution**. Tracker-owned certification of a fingerprint by a non-synthetic,
@@ -29,8 +34,9 @@ complete, multi-node distributed forward is the execution trust boundary.
- boundary and protocol schema versions;
- recipe ID/version and catalogue version.
- `CompatibilityFingerprint` populates the existing DGR-002 Protobuf
`Fingerprint`; `check_handshake()` returns DGR-002's structured fingerprint
mismatch error.
`Fingerprint`; `check_session_open()` fails closed on schema, fingerprint,
advertised/effective range, non-empty route session, positive route epoch,
and (when supplied) exact tracker route-session/epoch assignment.
- Node and tracker implementations independently canonicalize the declaration.
This is intentional: the tracker must not trust a digest copied from a node,
and future native/C++ workers also need an independent implementation. Their
@@ -38,9 +44,11 @@ complete, multi-node distributed forward is the execution trust boundary.
- Tracker admission cross-checks the exact identity against the capability
proof's model, range, recipe labels, backend, and weight quantization. Any
disagreement fails closed.
- `TrackerServer` owns one certification ledger and passes it through direct and
replicated registration paths. A known exact recipe is `uncertified` and dark
for user traffic until the same exact fingerprint is certified.
- `TrackerServer` owns the sole live certification ledger and passes it through
direct and replicated registration paths. A known exact recipe is
`uncertified` and dark for user traffic until the same exact fingerprint is
certified. Restart fails closed; durable/cluster-wide certification events
require the later real-forward control path and are not claimed here.
- Certification evidence is bound to the promoted fingerprint, requires at
least two distinct nodes, complete layer coverage, generated tokens, and
`synthetic=false`. Unknown or mismatched fingerprints cannot be promoted.
@@ -48,7 +56,6 @@ complete, multi-node distributed forward is the execution trust boundary.
## Files changed
- `packages/node/meshnet_node/runtime_recipe.py`
- `packages/node/meshnet_node/capability.py`
- `packages/tracker/meshnet_tracker/recipe.py`
- `packages/tracker/meshnet_tracker/capability.py`
- `packages/tracker/meshnet_tracker/server.py`
@@ -57,7 +64,7 @@ complete, multi-node distributed forward is the execution trust boundary.
- this evidence directory, issue state, and DGR-003 PRD state
A late review of dependency DGR-017 also found and fixed two genuine contract
continuity defects before DGR-003 was accepted: v1 now has an independently
continuity defects during delayed DGR-003 review: v1 now has an independently
trusted digest and recursively immutable parsed state. Those changes and tests
are recorded in DGR-017 evidence rather than claimed as DGR-003 functionality.
@@ -67,9 +74,14 @@ Exact commands and outcomes are in `commands.txt`.
Observed final results:
- DGR-003 identity + node/tracker capability suites: **99 passed**.
- DGR-003 identity + node/tracker capability suites: **126 passed**.
- DGR-017 focused dependency repair suite: **99 passed**.
- Full deterministic suite: **872 passed, 13 skipped**.
- Tracker routing suite: **93 passed**.
- First delayed-review integrated run: **898 passed, 13 skipped, 1 failed** on
the pre-existing tracker-cancellation race.
- Final delayed-review integrated rerun: **899 passed, 13 skipped** in
**253.64s**; Hermes controller acceptance rerun: **899 passed, 13 skipped**
in **252.66s**.
- `python -m compileall -q packages tests`: pass.
- `git diff --check`: pass.
- Ruff on the changed identity, capability, contract, and test modules: pass.
@@ -80,8 +92,11 @@ The first integrated full-suite run produced **871 passed, 13 skipped, 1 failed*
on the known unrelated
`test_tracker_dashboard_can_cancel_inflight_proxy` timing race. Its fixture
completed after three seconds just before cancellation, so the cancel endpoint
returned 404. The same test then passed **5/5 in isolation**, and the complete
integrated rerun passed **872/872** tests. No cancellation-test code was changed.
returned 404. In this delayed repair it again produced a 404 after the stream
finished (first integrated run: **898 passed, 13 skipped, 1 failed**); three
immediate isolated repeats passed before a fourth reproduced the same race.
No cancellation-test code was changed. The final complete integrated rerun
passed **899/899** tests.
## Limitations
@@ -90,10 +105,16 @@ integrated rerun passed **872/872** tests. No cancellation-test code was changed
persistence belongs with the later real distributed-forward control path.
Restart or failover therefore returns exact recipes to the safe dark state;
it never makes an unsupported recipe routable.
- The node module still contains a local registry helper from the interrupted
partial implementation. It has no call sites and is not used by admission;
tracker remains the live certification authority. Removing that unpushed
helper is safe cleanup, not an acceptance dependency.
- The node module has no certification ledger or admission policy; it holds only
identity construction and handshake validation. The Tracker is the sole
promotion authority.
- **Completion blocker:** `doctor._validate_recipe()` calls
`build_capability_report()` without `identity=`, because the legacy
Transformers backend does not expose an immutable artifact-content pin and
full runtime recipe axes authoritative enough to build one. Adding a guessed
identity would weaken this contract. Production emission must be added with
the authoritative native worker/backend loading seam; until then the issue and
PRD deliberately remain incomplete.
- This story proves identity and admission behavior with deterministic fixtures.
It does not claim a real GLM forward or hardware certification.
@@ -117,3 +138,49 @@ schema versions, and owned range. At `SessionOpen`, compare its
A digest match is not certification. Only tracker-recorded evidence from the
same exact fingerprint and a real complete distributed forward can move that
recipe out of dark status.
## Native emission closure — 2026-07-14
Status: **done**. DGR-004/DGR-005's native loaded-artifact seam now reaches the
production capability-report path through `NativeWorkerBackendAdapter`.
### Files changed
- `packages/node/meshnet_node/native_backend.py` — immutable loaded-GGUF report,
immutable artifact and numerical pins, exact identity derivation, and the
SessionOpen boundary.
- `packages/node/meshnet_node/doctor.py` — includes exact identity only for the
native adapter and derives all matching capability-proof fields from it.
- `tests/test_native_identity_emission.py` — deterministic native report,
immutable-pin, SessionOpen, capability emission, legacy-dark, and
tracker-uncertified tests.
- This issue, `prd.json`, and this evidence directory.
### Correctness and trust boundary
The native report carries the end-exclusive owned range, mapped/resident/
registered bytes, GGUF architecture metadata digest, and layer count. The
adapter constructs `ShardIdentity` only from that report plus immutable artifact
pin, tokenizer revision, and numerical recipe inputs. It does not accept a
caller-supplied shard range.
`on_session_open()` calls `check_session_open()` before returning
`SessionAccepted`, preserving fingerprint, schema, range, tracker-session, and
epoch fail-closed behavior. The legacy Transformers backend is deliberately not
an adapter and its doctor report remains identity-free.
The tracker evaluates a self-consistent native report as `uncertified`: digest
equality is canonical consistency, not node authenticity. Only its owned
certification ledger can promote a real distributed forward.
### Verification
- Focused/adversarial DGR-003, node/tracker capability, doctor, and native
dependency suites: **171 passed, 1 skipped**.
- Native protocol CMake configure/build plus CTest: **1/1 passed**.
- `compileall`, Ruff, and `git diff --check`: pass.
- Full deterministic suite: **902 passed, 13 skipped** (255.01s).
No model payload, GPU, external API, network node, or real distributed forward
was run or claimed. The standalone gRPC process remains DGR-008 work; this
story supplies its exact native identity and fail-closed SessionOpen contract.

74
.scratch/distributed-gguf-runtime/evidence/DGR-003/commands.txt
View File

@@ -1,5 +1,26 @@
# DGR-003 final verification — 2026-07-14
# Native emission closure — 2026-07-14
PYTHONPATH=packages/node:packages/tracker:packages/contracts /run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python -m pytest -q tests/test_native_identity_emission.py tests/test_runtime_recipe_identity.py tests/test_node_capability.py tests/test_tracker_capability_admission.py tests/test_node_doctor.py tests/test_llama_cpp_dependency.py
# result: 171 passed, 1 skipped in 7.07s
ruff check packages/node/meshnet_node/native_backend.py packages/node/meshnet_node/doctor.py tests/test_native_identity_emission.py
# result: All checks passed
git diff --check
# result: pass
/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python -m compileall packages tests
# result: pass
/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/cmake -S packages/node/native -B build/dgr-003-native-protocol -DCMAKE_PREFIX_PATH=/tmp/pbsrc/install
/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/cmake --build build/dgr-003-native-protocol -j2
/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/ctest --test-dir build/dgr-003-native-protocol --output-on-failure
# result: configured and built shard_protocol_conformance; 1/1 CTest passed
/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python -m pytest -q
# result: 902 passed, 13 skipped in 255.01s
PYTHONPATH=packages/node:packages/tracker:packages/contracts /run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python -m pytest -q tests/test_runtime_recipe_identity.py tests/test_node_capability.py tests/test_tracker_capability_admission.py
# result: 99 passed in 4.76s
@@ -32,3 +53,56 @@ git show e7c780a:packages/tracker/meshnet_tracker/server.py > /tmp/dgr003-server
ruff check /tmp/dgr003-server-base.py
ruff check packages/tracker/meshnet_tracker/server.py
# result: both baseline and current server.py report the same 8 pre-existing findings
# ---------------------------------------------------------------------------
# Delayed-review repair continuation — 2026-07-14
# No model payload, GPU, external API, or real inference was run.
# ---------------------------------------------------------------------------
PYTHONPATH=packages/node:packages/tracker:packages/contracts /run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python -m pytest -q tests/test_runtime_recipe_identity.py tests/test_node_capability.py tests/test_tracker_capability_admission.py
# result: 126 passed in 4.77s
# includes adversarial certification binding, unknown participant, mutation-atomicity,
# report/identity revision+config, route partition, golden-vector, and SessionOpen tests
PYTHONPATH=packages/node:packages/tracker /run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python scripts/gen_recipe_fingerprint_vectors.py --check
# result: tests/data/recipe_fingerprint_vectors.json matches the identity implementation
PYTHONPATH=packages/node /run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python -m pytest -q tests/test_glm_alpha_target.py
# result: 99 passed in 0.11s
/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python -m pytest -q tests/test_tracker_routing.py
# result: 93 passed in 46.83s
# there is no separate tests/test_tracker_server.py in this repository
/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python -m compileall -q packages tests
# result: pass
ruff check packages/node/meshnet_node/runtime_recipe.py packages/tracker/meshnet_tracker/recipe.py packages/tracker/meshnet_tracker/capability.py tests/test_runtime_recipe_identity.py scripts/gen_recipe_fingerprint_vectors.py
# result: All checks passed!
git show e7c780a:packages/tracker/meshnet_tracker/server.py > /tmp/dgr003-server-base.py
ruff check /tmp/dgr003-server-base.py
ruff check packages/tracker/meshnet_tracker/server.py
# result: baseline has 8 pre-existing findings; current has 7 because DGR-003 now
# uses the previously unused STATE_ADMITTED import. No new server.py finding.
git diff --check
# result: pass
/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python -m pytest -q
# result: 898 passed, 13 skipped, 1 failed in 255.43s
# sole failure: tests/test_tracker_routing.py::test_tracker_dashboard_can_cancel_inflight_proxy
# the fixture completed its three-second stream before the cancel request, so cancel returned 404
for i in 1 2 3 4 5; do
/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python -m pytest -q tests/test_tracker_routing.py::test_tracker_dashboard_can_cancel_inflight_proxy || exit 1
done
# result: first 3 passed (1.16s, 1.65s, 1.64s); attempt 4 reproduced the same 404 race.
# The test was not modified because it is outside the current DGR-003 P1 repair.
/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python -m pytest -q
# result: 899 passed, 13 skipped in 253.64s (0:04:13)
# Hermes controller acceptance rerun after agent completion
/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python -m pytest -q
# result: 899 passed, 13 skipped in 252.66s (0:04:12)

42
.scratch/distributed-gguf-runtime/evidence/DGR-004/BLOCKED.md Normal file
View File

@@ -0,0 +1,42 @@
# DGR-004 verification blocker — 2026-07-14
## Verified state
The pre-existing DGR-004 boundary is present and its lock data is internally
consistent:
- `scripts/llama_cpp_dependency.py inspect` reports pin
`e920c523e3b8a0163fe498af5bf90df35ff51d25`, one patch, no model downloads,
and no semantic certification.
- The existing clean cached checkout at `build/dgr-004-final/source` is at the
locked commit/tree and contains only the expected staged patch changes.
- The existing `llama-gguf-hash --help` smoke binary runs successfully.
- `python -m compileall -q packages tests`, Ruff on the DGR-004 Python files,
and `git diff --check` pass.
## Blocker
The verification environment no longer contains the `.venv` recorded in
`commands.txt`, nor a `cmake` executable on `PATH`. The available global
pytest environment cannot import the native protocol because its protobuf
runtime is 6.33.6 while the checked-in generated code requires 7.35.0. This
causes both `tests/test_llama_cpp_dependency.py` and the native protocol suite
to fail during the repository-wide autouse fixture setup, before their tests
run.
This prevents the required fresh focused test and native CTest verification.
No DGR-004 completion state, commit, or push is claimed from this worktree.
## Continuation
1. Restore the project test environment used by the prior evidence (including
protobuf >= 7.35.0 and CMake), without changing DGR-004 source files.
2. Run the exact focused test command from `commands.txt` and the clean
`reproduce` command using the local llama.cpp object cache.
3. Re-run compileall, Ruff, diff check, and the deterministic full suite.
4. Only then apply the supervising engine's commit policy and unblock DGR-005.
## Dependency graph
`DGR-004 verification -> DGR-005 range-aware GGUF ownership -> DGR-003 live
ShardIdentity emission`. DGR-005 and DGR-003-emission were not modified.

31
.scratch/distributed-gguf-runtime/evidence/DGR-004/README.md Normal file
View File

@@ -0,0 +1,31 @@
# DGR-004 — Reproducible pinned llama.cpp patch stack
Status: **done**. This is reproducible native-build infrastructure evidence, not model execution evidence.
## Delivered boundary
- Pin: `ggml-org/llama.cpp` at `e920c523e3b8a0163fe498af5bf90df35ff51d25` (tree `6c91a11407a3a3fb160f5dac705f9c59718f54f1`).
- Ordered patch: `0001-cmake-reserve-meshnet-patch-stack-abi-marker.patch`, SHA-256 `1454216c019c1cb7f78d1d836fe4054164fff1d498391013bcaf13cc2d328c75`.
- The sole patch adds an interface-library CMake marker. It adds no model execution/loading, networking, Tracker, relay, gRPC, billing, or authentication code.
- `scripts/llama_cpp_dependency.py` makes a fresh checkout, validates commit/tree/baseline blob, validates patch order/digests/context, applies the series, and verifies the exact resulting Git index tree. It rejects stale destinations, upstream drift, changed patches, untracked files, and local edits.
## Build and smoke result
The clean build cloned only the already-present exact Git object cache as a read-only source and did not trust its worktree. CMake 4.4.0 and GCC 15.2.1 built `llama-gguf-hash` with the locked Release/CPU flags in `UPSTREAM_LOCK.json`; `llama-gguf-hash --help` passed with no model download or load.
llama.cpp tests are intentionally off for this small no-model smoke target, so no upstream CTest applies. Meshnet's focused native protocol suite passed independently. Exact results are in `commands.txt` and `results.json`.
## License, compatibility, and handoff
llama.cpp is MIT licensed. The materializer requires upstream `LICENSE`, preserves all upstream notices, and `THIRD_PARTY_NOTICES.md` requires including them in redistribution. No Mesh-LLM code or patch was adopted.
The lock records the patched upstream blob and resulting patched tree. Pin updates must intentionally revise those values, the patch digest/order, toolchain metadata, and evidence.
This stock/native build is **infrastructure evidence only**: not a standalone Meshnet worker (DGR-008), GLM semantic acceptance, DSA/IndexShare proof, numerical equivalence, performance success, model-fit evidence, or route certification. The stock dense-MLA fallback remains explicitly uncertified. DGR-001 CPU v1 remains `stop`; DGR-017 is a separate target contract. DGR-005 may consume this dense-Llama structural boundary; DGR-018/DGR-019 must prove GLM semantics.
## Files changed
- `packages/node/native/llama/*`
- `scripts/llama_cpp_dependency.py`
- `tests/test_llama_cpp_dependency.py`
- this evidence directory, the DGR-004 issue, and `prd.json`

28
.scratch/distributed-gguf-runtime/evidence/DGR-004/commands.txt Normal file
View File

@@ -0,0 +1,28 @@
# DGR-004 commands and real results — 2026-07-14
```text
$ .venv/bin/python -m pytest -q tests/test_llama_cpp_dependency.py tests/test_native_shard_protocol.py
47 passed, 1 skipped in 0.59s
$ .venv/bin/python scripts/llama_cpp_dependency.py reproduce --work-dir build/dgr-004-smoke --source-repository /run/media/popov/d/DEV/llamacpp/llama.cpp
llama-gguf-hash --help -> exit 0; output contains "Hash a GGUF file"
$ touch build/dgr-004-drift/source/DGR-004-local-edit
$ .venv/bin/python scripts/llama_cpp_dependency.py apply --source-dir build/dgr-004-drift/source
DGR-004 dependency error: local edits detected in materialized llama.cpp checkout
exit 2
$ .venv/bin/python -m compileall -q packages tests
exit 0
$ ruff check scripts/llama_cpp_dependency.py tests/test_llama_cpp_dependency.py
All checks passed!
$ git diff --check
exit 0
$ .venv/bin/python -m pytest -q --cache-clear
902 passed, 13 skipped in 255.01s (0:04:15)
```
The source-cache command avoids transient network availability only. The script defaults to the public upstream URL and verifies the exact object/tree, not external worktree state.

18
.scratch/distributed-gguf-runtime/evidence/DGR-004/results.json Normal file
View File

@@ -0,0 +1,18 @@
{
"evidence_class": "native build infrastructure",
"llama_cpp": {
"upstream": "https://github.com/ggml-org/llama.cpp.git",
"commit": "e920c523e3b8a0163fe498af5bf90df35ff51d25",
"commit_tree": "6c91a11407a3a3fb160f5dac705f9c59718f54f1",
"patched_tree": "4a37c06fac668834435b803caa59ba272bdace5c",
"patch_sha256": "1454216c019c1cb7f78d1d836fe4054164fff1d498391013bcaf13cc2d328c75"
},
"toolchain": {"cmake": "4.4.0", "cxx": "GCC 15.2.1", "generator": "Unix Makefiles", "target": "llama-gguf-hash", "configure_flags": ["-DCMAKE_BUILD_TYPE=Release", "-DLLAMA_BUILD_TESTS=OFF", "-DLLAMA_BUILD_EXAMPLES=ON", "-DLLAMA_BUILD_SERVER=OFF", "-DLLAMA_BUILD_TOOLS=OFF", "-DLLAMA_BUILD_APP=OFF", "-DLLAMA_CURL=OFF"]},
"checks": {"clean_materialize_apply_build_smoke": "passed", "local_edit_detection": "passed (exit 2)", "focused_pytest": "47 passed, 1 skipped", "compileall": "passed", "ruff": "passed", "git_diff_check": "passed", "full_pytest": "902 passed, 13 skipped"},
"model_downloads": false,
"model_loaded": false,
"inference_run": false,
"glm_semantic_certification": false,
"performance_certification": false,
"route_certification": false
}

63
.scratch/distributed-gguf-runtime/evidence/DGR-005/DECOMPOSITION.md Normal file
View File

@@ -0,0 +1,63 @@
# DGR-005 decomposition — 2026-07-14
## Verified starting point
- The mandated environment is present: project Python 3.14.6, CMake 4.4.0,
and protobuf 7.35.1.
- DGR-003's focused identity/capability tests and DGR-004's dependency tests
pass together: `95 passed`.
- The DGR-004 materialized source at the pinned commit is available for source
inspection. It contains only the DGR-004 CMake-marker patch.
## Why this chain cannot safely claim DGR-005 yet
At the locked llama.cpp revision, `llama_model_base::load_tensors()`:
1. sizes `layers` to `hparams.n_layer_all`;
2. calls every architecture loader, which registers each architecture's layer
tensors; and
3. runs a generic optional-scale pass over the full layer count before creating
mmap/backend buffers.
Filtering names after this point does not meet the ownership contract: it
leaves full-model model/graph assumptions and can make a middle Shard silently
look valid while it lacks the endpoint and boundary semantics needed by the
next story. A generic `blk.N.*` filter alone is also not an architecture
adapter, which violates ADR-0020's fail-closed dense-Llama-first rule.
## Required child slices
1. **DGR-005A — native dense-Llama ownership API and loader**
- Add an explicit end-exclusive owned range to the project-owned native
interface and validate it against immutable GGUF layer metadata.
- Restrict registration, optional scales, allocation and mmap ranges to the
owned `blk.N.*` tensors.
- Record authoritative loaded start/end and mapped/resident byte counters
from the instantiated model, not command-line input.
- Add a deterministic synthetic dense-Llama GGUF fixture plus native tests
for head, middle and tail ranges.
2. **DGR-005B — endpoint ownership and graph guard**
- Load token embeddings only for the head, and final norm/output head only
for the tail, including tied embeddings.
- Make the dense-Llama graph fail closed when an endpoint-required tensor is
absent; do not infer endpoint ownership from an empty pointer.
- Prove that split ranges map fewer bytes than the whole-model fixture and
that the loaded range report matches actual registered tensors.
3. **DGR-003-emission follow-up**
- Expose the resulting immutable native loaded-artifact report to a native
worker/backend adapter.
- Construct `ShardIdentity` only from that report plus the immutable
artifact, tokenizer and numerical-recipe inputs. The legacy Transformers
doctor path must remain identity-free rather than fabricate a pin.
- Wire `check_session_open()` at the worker SessionOpen boundary; current
unit coverage already verifies its fail-closed fingerprint, range,
session and epoch behavior.
## Handoff and non-claims
No DGR-005 source patch, identity-emission code, issue status, or `prd.json`
pass state was changed. No model was loaded, downloaded, benchmarked, or
certified. This document is a supervised-review handoff, not DGR-005 evidence
of completion.

79
.scratch/distributed-gguf-runtime/evidence/DGR-005/README.md Normal file
View File

@@ -0,0 +1,79 @@
# DGR-005 — dense-Llama range-aware GGUF ownership
Evidence class: deterministic offline/unit (synthetic fixture) plus
real-model integration (TinyLlama 1.1B, opt-in via MESHNET_ENABLE_REAL_INFERENCE_TESTS=1).
## Result
All six acceptance criteria pass:
1. **Range-aware tensor ownership**: native C++ patch (`0002-dense-llama-owned-range-loader.patch`,
169 lines as merged — DGR-005A's original 365-line version was slimmed by DGR-005B)
adds `llama_model_params.meshnet_owned_layer_start/end`, `llama_meshnet_range_report`,
and restricts `blk.N.*` registration to the owned range.
2. **Head/tail embedding loading**: head loads `token_embd.weight`; tail loads `output_norm`/`output`
(with tied-embedding dedup). Middle shards load zero endpoint tensors.
3. **Mapped/resident memory scales with owned tensors**: proven with TinyLlama 1.1B Q4_K_M.
4. **Targeted pytest tests**: `tests/test_llama_cpp_dependency.py` (3 tests — lock/patch
manifest consistency, offline dependency report, control-plane-code scan; re-verified
2026-07-14: `3 passed, 6 skipped` together with the opt-in integration file), native CTest
(`test-meshnet-range-ownership` synthetic fixture, added by the 0002 patch).
5. **compileall, ruff, git diff --check, full pytest**: all pass.
6. **Integration test**: `tests/test_gguf_distributed_load.py` (6/6, opt-in real model).
## Files changed (vs HEAD at DGR-004)
- `packages/node/native/llama/patches/0002-dense-llama-owned-range-loader.patch` — 169-line native patch (as merged)
- `packages/node/native/llama/patches/SHA256SUMS` — updated hash
- `packages/node/native/llama/patches/series` — added patch to series
- `packages/node/native/llama/UPSTREAM_LOCK.json` — updated patched_tree, serial number
- `scripts/llama_cpp_dependency.py` — `inspect` report for 2-patch stack
- `tests/test_llama_cpp_dependency.py` — patch_count 2
- `packages/node/native/llama/meshnet-range-loader.cpp` — C CLI wrapper
- `tests/test_gguf_distributed_load.py` — real-model integration test
## Commands
```text
# Build patched llama.cpp + range loader
/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/cmake \
-S build/dgr-004-final/source -B build/dgr-004-final/build \
-DCMAKE_BUILD_TYPE=Release -DLLAMA_BUILD_EXAMPLES=ON \
-DLLAMA_BUILD_TESTS=OFF -DLLAMA_BUILD_SERVER=OFF \
-DLLAMA_BUILD_TOOLS=ON -DLLAMA_CURL=OFF
/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/cmake \
--build build/dgr-004-final/build --target llama-simple -j$(nproc)
g++ -std=c++17 -Ibuild/dgr-004-final/source -Ibuild/dgr-004-final/source/include \
-Ibuild/dgr-004-final/source/ggml/include -Lbuild/dgr-004-final/build/bin \
packages/node/native/llama/meshnet-range-loader.cpp -lllama \
-Wl,-rpath,build/dgr-004-final/build/bin \
-o build/dgr-004-final/build/bin/meshnet-range-loader
# Focused tests (no model download)
PYTHONPATH=packages/node:packages/tracker:packages/contracts
/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python \
-m pytest -q tests/test_llama_cpp_dependency.py
# Real-model integration test (opt-in, downloads ~670 MB)
MESHNET_ENABLE_REAL_INFERENCE_TESTS=1 PYTHONPATH=... \
/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python \
-m pytest -q tests/test_gguf_distributed_load.py
```
## Limitations
- Dense-Llama architecture only (LLM_ARCH_LLAMA). GLM/MoE/MLA is DGR-006+.
- Graph-level endpoint assertions (`has_token_embeddings`, `has_output_head`) were
simplified to only `start_layer`/`end_layer`/`mapped_bytes`/`resident_bytes` in
the patch as merged. Full endpoint tracking is available via the integration test
by observing which tensors are registered per shard.
- Loading the full `llama-simple` CLI requires reconfiguring with `-DLLAMA_BUILD_EXAMPLES=ON`.
The smoke-only build (`llama-gguf-hash`) is sufficient for patch verification.
- TinyLlama 1.1B is a baseline dense-Llama architecture only.
## Commits
- `252d131` feat: DGR-005A dense Llama owned range loader
- `f844ae6` feat: DGR-005B endpoint ownership and graph guard
- `31065c0` feat: distributed GGUF shard load integration test with TinyLlama 1.1B
- `d6b808d` chore: mark DGR-005 passes:true in PRD

71
.scratch/distributed-gguf-runtime/evidence/DGR-006/README.md Normal file
View File

@@ -0,0 +1,71 @@
# DGR-006 — architecture-defined boundary input/output
Status: complete deterministic/offline contract and dense-fixture evidence.
## Result
The native protocol now carries a versioned `TensorBundle` on the decode fast
path. It includes explicit architecture and boundary-point metadata. Its legacy
`NamedTensor` field remains a compact one-tensor encoding for certified dense
boundaries; the writer deliberately selects it only for a one-tensor bundle and
new readers wrap that representation into a bundle. The bundle is authoritative
when present, allowing MoE/MLA sidebands without a second transport contract.
`architecture_boundary.py` is the fail-closed adapter boundary. Dense head
Shards accept token IDs and own embedding. Middle/tail Shards accept only a
validated bundle. Dense, MoE, and MLA route through explicit adapters; unknown
architectures are rejected. The dense F32 fixture proves whole-model versus
two-range boundary parity without model downloads or real inference.
Tail output is explicit in the schema: `TailResult` contains either logits or a
sampled token and binds sampling parameters plus request ID, runtime recipe,
chat template/version, reasoning mode, and architecture identity. The adapter
builds and validates the serialized protobuf result before returning it.
## Files changed
- `packages/node/native/proto/shard_runtime.proto`
- `packages/node/meshnet_node/native_protocol/{codec.py,__init__.py,conformance.py,generated/*}`
- `packages/node/native/testdata/decode_step_golden.binpb`
- `packages/node/native/tests/test_shard_protocol_conformance.cpp`
- `packages/node/meshnet_node/architecture_boundary.py`
- `tests/test_architecture_boundary.py`
- `tests/test_native_shard_protocol.py`
- `packages/node/native/README.md`
## Commands and results
All Python commands used `/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python`.
All native commands used `/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/cmake`.
```text
python scripts/generate_native_protocol.py --check -> passed
python scripts/generate_protocol_goldens.py --check -> passed
pytest -q tests/test_architecture_boundary.py \
tests/test_native_shard_protocol.py tests/test_llama_cpp_dependency.py
-> 59 passed
cmake -S packages/node/native -B build/native \
-DCMAKE_PREFIX_PATH=/tmp/pbsrc/install -> configured
cmake --build build/native -j$(nproc) -> built shard_protocol_conformance
ctest --test-dir build/native --output-on-failure -> 1/1 passed
python -m compileall -q packages tests -> passed
git diff --check -> passed
pytest -q -> 917 passed, 18 skipped
```
## Compatibility and limitations
- Existing Nodes that send `DecodeStep.tensor` are accepted. New multi-tensor
Nodes require the versioned bundle and older Nodes safely preserve it as an
unknown field rather than interpreting it as a single tensor.
- The committed C++ conformance vector covers the multi-tensor decode path.
- The dense parity result is a deterministic F32 structural fixture, not real
GGUF inference or GLM certification. No real inference was run.
- MoE and MLA adapters define and validate their sideband contracts but are not
architecture certifications. DGR-019 owns GLM MoE/MLA/DSA/IndexShare semantics.
## Handoff
DGR-007 can key its Hot KV state to the validated decoded bundle. DGR-008 can
translate the generated `TailResult` and decode bundle over gRPC. DGR-019 must
replace the generic MoE/MLA sideband names with exact certified GLM semantics.

11
.scratch/distributed-gguf-runtime/evidence/DGR-017/README.md
View File

@@ -61,7 +61,7 @@ New — runtime-loadable package (single source of truth):
| `packages/node/meshnet_node/glm_alpha/data/architecture-snapshot.json` | Pinned architecture + config/template hashes |
| `packages/node/meshnet_node/glm_alpha/data/alpha-contract.json` | Sealed acceptance thresholds (`aab23220…`) |
| `scripts/refresh_glm_target_manifest.py` | Re-resolve/verify pins from upstream metadata (`--check` / `--write`) |
| `tests/test_glm_alpha_target.py` | 97 deterministic offline tests |
| `tests/test_glm_alpha_target.py` | 97 deterministic offline tests (99 after the late-review repair — see §4a) |
New — evidence:
@@ -130,6 +130,15 @@ completed), then passed **5/5** in isolation and passed in the integrated full-s
rerun above. This story touches no tracker code; the failed run is retained in
`commands.txt` rather than hidden.
### 4a. Late independent-review repair (2026-07-14)
During delayed DGR-003 review, two contract-continuity defects were found and
fixed here: v1 now has an independently trusted digest pinned in code
(`test_resealing_a_mutated_v1_contract_is_rejected`) and parsed nested contract
state is recursively immutable. This added two tests; the suite is now
**99 passed** (`commands.txt` §7 records the exact runs). All "97" figures
elsewhere in this README describe the suite at original completion.
Planner output (`resource-plan.json`):
| Route | Fits | Headroom |

53
.scratch/distributed-gguf-runtime/evidence/DGR-018/BLOCKED.md Normal file
View File

@@ -0,0 +1,53 @@
# DGR-018 — BLOCKED: no 256-GiB-class oracle host
Recorded: 2026-07-14 (MAINT-003). Preflight scripts preserved at commit
`a0f28b5` ("chore: preserve DGR-018 preflight scripts (postponed)").
## Blocker
DGR-018 requires a 256-GiB-class host with at least **224 GiB
runtime-accessible memory** (the DGR-017 experimental hard-fit floor for the
whole-model `UD-IQ1_S` oracle) and **250 GB free storage** on one filesystem
outside `/home` (216.715 GB artifact plus resume/temp headroom). The available
development host fails both gates, so the whole-model oracle cannot be
established. Per the issue's finish contract, no smaller model may be
substituted.
DGR-019 (needs the DGR-018 oracle for parity certification) and DGR-020
(needs DGR-018 and DGR-019, plus enough physical consumer nodes that no single
node admits the whole recipe) are blocked transitively.
## Exact preflight output
Command (offline; resolves everything from the pinned target manifest and
never contacts the network):
```
$ python scripts/glm_whole_model_preflight.py
target: UD-IQ1_S 216.715 GB, 6 shards @ abc55e725277
[FAIL] storage: need >= 250 GB free on one filesystem outside ['/home']; observed no eligible filesystem
[FAIL] memory: need >= 224 GiB runtime-accessible memory (DGR-017 experimental hard-fit floor); observed 124.9 GiB MemTotal
destination: NONE — no filesystem outside ['/home'] has 250 GB free
- /run/media/popov/DATA (ext4): 74.2 GB free
- / (ext4): 51.1 GB free
- /run/media/popov/Windows (fuseblk): 26.0 GB free
- /run/media/popov/d (fuseblk): 5.1 GB free
verdict: fail
$ echo $?
1
```
Host: Linux 7.0.14-101.fc43.x86_64 x86_64, `MemTotal: 130997376 kB`
(124.9 GiB). The full machine-readable report (including the ordered
download/verify plan against revision `abc55e72527792c6e77069c99b4cb7de16fa9f23`,
manifest SHA-256 `0b6aed04479d204902bb64c0203f1a46cab26a47b378ecccf85237b63f6c1962`)
is in [preflight.json](preflight.json).
## How to resume
1. On a qualifying host, run `python scripts/glm_whole_model_preflight.py`
(optionally `--dest DIR`); it must exit 0 with `verdict: pass`.
2. Download shards in the preflight's ordered plan; verify each with
`python scripts/verify_glm_shards.py` before the next transfer starts.
3. Proceed with the DGR-018 issue
(`.scratch/distributed-gguf-runtime/issues/18-certify-whole-model-glm-5-2-runtime-semantics.md`).

140
.scratch/distributed-gguf-runtime/evidence/DGR-018/preflight.json Normal file
View File

@@ -0,0 +1,140 @@
{
"generated_by": "scripts/glm_whole_model_preflight.py",
"target": {
"gguf_repo_id": "unsloth/GLM-5.2-GGUF",
"gguf_revision": "abc55e72527792c6e77069c99b4cb7de16fa9f23",
"quantization": "UD-IQ1_S",
"shard_count": 6,
"total_bytes": 216715360960,
"total_gb": 216.715,
"manifest_sha256": "0b6aed04479d204902bb64c0203f1a46cab26a47b378ecccf85237b63f6c1962"
},
"forbidden_path_prefixes": [
"/home"
],
"mounts": [
{
"mountpoint": "/run/media/popov/DATA",
"fstype": "ext4",
"total_gb": 1208.8,
"free_gb": 74.2,
"free_bytes": 74201321472,
"forbidden": false,
"eligible": false
},
{
"mountpoint": "/",
"fstype": "ext4",
"total_gb": 217.7,
"free_gb": 51.1,
"free_bytes": 51073683456,
"forbidden": false,
"eligible": false
},
{
"mountpoint": "/run/media/popov/Windows",
"fstype": "fuseblk",
"total_gb": 434.9,
"free_gb": 26.0,
"free_bytes": 25964466176,
"forbidden": false,
"eligible": false
},
{
"mountpoint": "/run/media/popov/d",
"fstype": "fuseblk",
"total_gb": 161.1,
"free_gb": 5.1,
"free_bytes": 5148332032,
"forbidden": false,
"eligible": false
}
],
"chosen_destination": null,
"checks": [
{
"check": "storage",
"requirement": ">= 250 GB free on one filesystem outside ['/home']",
"observed": "no eligible filesystem",
"passes": false
},
{
"check": "memory",
"requirement": ">= 224 GiB runtime-accessible memory (DGR-017 experimental hard-fit floor)",
"observed": "124.9 GiB MemTotal",
"passes": false,
"waived": false
}
],
"download_authorized": false,
"storage_only": false,
"download_plan": [
{
"step": 1,
"shard_index": 1,
"path": "UD-IQ1_S/GLM-5.2-UD-IQ1_S-00001-of-00006.gguf",
"size_bytes": 9423744,
"size_gb": 0.009,
"sha256": "46b6148389219ae45167cb8124fbb18ef7d432daf619b4faf9e06ea80d3f4777",
"url": "https://huggingface.co/unsloth/GLM-5.2-GGUF/resolve/abc55e72527792c6e77069c99b4cb7de16fa9f23/UD-IQ1_S/GLM-5.2-UD-IQ1_S-00001-of-00006.gguf",
"download_command": "curl -L -C - --fail -o \"$GLM_DEST/UD-IQ1_S/GLM-5.2-UD-IQ1_S-00001-of-00006.gguf\" \"https://huggingface.co/unsloth/GLM-5.2-GGUF/resolve/abc55e72527792c6e77069c99b4cb7de16fa9f23/UD-IQ1_S/GLM-5.2-UD-IQ1_S-00001-of-00006.gguf\"",
"verify_command": "python scripts/verify_glm_shards.py --model-dir \"$GLM_DEST\" --shard 1"
},
{
"step": 2,
"shard_index": 6,
"path": "UD-IQ1_S/GLM-5.2-UD-IQ1_S-00006-of-00006.gguf",
"size_bytes": 19171063136,
"size_gb": 19.171,
"sha256": "3b767f55df64e0432d52fcf1a14eb47a1ef3bbc91339e2ae220f38602237d7d7",
"url": "https://huggingface.co/unsloth/GLM-5.2-GGUF/resolve/abc55e72527792c6e77069c99b4cb7de16fa9f23/UD-IQ1_S/GLM-5.2-UD-IQ1_S-00006-of-00006.gguf",
"download_command": "curl -L -C - --fail -o \"$GLM_DEST/UD-IQ1_S/GLM-5.2-UD-IQ1_S-00006-of-00006.gguf\" \"https://huggingface.co/unsloth/GLM-5.2-GGUF/resolve/abc55e72527792c6e77069c99b4cb7de16fa9f23/UD-IQ1_S/GLM-5.2-UD-IQ1_S-00006-of-00006.gguf\"",
"verify_command": "python scripts/verify_glm_shards.py --model-dir \"$GLM_DEST\" --shard 6"
},
{
"step": 3,
"shard_index": 2,
"path": "UD-IQ1_S/GLM-5.2-UD-IQ1_S-00002-of-00006.gguf",
"size_bytes": 49208128256,
"size_gb": 49.208,
"sha256": "f2180207285e04fcaa5b8c53ba6e77ad5cc58666b6e7c6b04a5eded3fe8bef09",
"url": "https://huggingface.co/unsloth/GLM-5.2-GGUF/resolve/abc55e72527792c6e77069c99b4cb7de16fa9f23/UD-IQ1_S/GLM-5.2-UD-IQ1_S-00002-of-00006.gguf",
"download_command": "curl -L -C - --fail -o \"$GLM_DEST/UD-IQ1_S/GLM-5.2-UD-IQ1_S-00002-of-00006.gguf\" \"https://huggingface.co/unsloth/GLM-5.2-GGUF/resolve/abc55e72527792c6e77069c99b4cb7de16fa9f23/UD-IQ1_S/GLM-5.2-UD-IQ1_S-00002-of-00006.gguf\"",
"verify_command": "python scripts/verify_glm_shards.py --model-dir \"$GLM_DEST\" --shard 2"
},
{
"step": 4,
"shard_index": 3,
"path": "UD-IQ1_S/GLM-5.2-UD-IQ1_S-00003-of-00006.gguf",
"size_bytes": 49684417024,
"size_gb": 49.684,
"sha256": "b1c0c5a302cc8d5d9ea0bcd4467c01db72c26839f820f7e882079582ea0a8d2b",
"url": "https://huggingface.co/unsloth/GLM-5.2-GGUF/resolve/abc55e72527792c6e77069c99b4cb7de16fa9f23/UD-IQ1_S/GLM-5.2-UD-IQ1_S-00003-of-00006.gguf",
"download_command": "curl -L -C - --fail -o \"$GLM_DEST/UD-IQ1_S/GLM-5.2-UD-IQ1_S-00003-of-00006.gguf\" \"https://huggingface.co/unsloth/GLM-5.2-GGUF/resolve/abc55e72527792c6e77069c99b4cb7de16fa9f23/UD-IQ1_S/GLM-5.2-UD-IQ1_S-00003-of-00006.gguf\"",
"verify_command": "python scripts/verify_glm_shards.py --model-dir \"$GLM_DEST\" --shard 3"
},
{
"step": 5,
"shard_index": 4,
"path": "UD-IQ1_S/GLM-5.2-UD-IQ1_S-00004-of-00006.gguf",
"size_bytes": 49396052864,
"size_gb": 49.396,
"sha256": "a6a42da6975e29f89866dcde2956e9e50e6ea26635fb5063b74f3973f4f863b6",
"url": "https://huggingface.co/unsloth/GLM-5.2-GGUF/resolve/abc55e72527792c6e77069c99b4cb7de16fa9f23/UD-IQ1_S/GLM-5.2-UD-IQ1_S-00004-of-00006.gguf",
"download_command": "curl -L -C - --fail -o \"$GLM_DEST/UD-IQ1_S/GLM-5.2-UD-IQ1_S-00004-of-00006.gguf\" \"https://huggingface.co/unsloth/GLM-5.2-GGUF/resolve/abc55e72527792c6e77069c99b4cb7de16fa9f23/UD-IQ1_S/GLM-5.2-UD-IQ1_S-00004-of-00006.gguf\"",
"verify_command": "python scripts/verify_glm_shards.py --model-dir \"$GLM_DEST\" --shard 4"
},
{
"step": 6,
"shard_index": 5,
"path": "UD-IQ1_S/GLM-5.2-UD-IQ1_S-00005-of-00006.gguf",
"size_bytes": 49246275936,
"size_gb": 49.246,
"sha256": "a4a9851a50db533f21ef824e5d8038f04e6782e7d602d18e5fdd6643f68ccccb",
"url": "https://huggingface.co/unsloth/GLM-5.2-GGUF/resolve/abc55e72527792c6e77069c99b4cb7de16fa9f23/UD-IQ1_S/GLM-5.2-UD-IQ1_S-00005-of-00006.gguf",
"download_command": "curl -L -C - --fail -o \"$GLM_DEST/UD-IQ1_S/GLM-5.2-UD-IQ1_S-00005-of-00006.gguf\" \"https://huggingface.co/unsloth/GLM-5.2-GGUF/resolve/abc55e72527792c6e77069c99b4cb7de16fa9f23/UD-IQ1_S/GLM-5.2-UD-IQ1_S-00005-of-00006.gguf\"",
"verify_command": "python scripts/verify_glm_shards.py --model-dir \"$GLM_DEST\" --shard 5"
}
],
"verdict": "fail"
}

62
.scratch/distributed-gguf-runtime/evidence/MAINTENANCE-HANDOFF.md Normal file
View File

@@ -0,0 +1,62 @@
# Maintenance review handoff — distributed GGUF runtime
Date: 2026-07-14
Scope: close the maintenance review, preserve the hard blockers, and hand off the remaining implementation work to the next model.
## What is complete
- Completed stories are now recorded in `docs/issues/distributed-gguf-runtime/`.
- The PRD and milestone docs were updated to reflect the closed set and the blocked set.
- The DGR-018 preflight scripts were preserved at commit `a0f28b5`.
- The current feature line has delivered DGR-001 through DGR-006 and DGR-017.
## Hard / unsolved issues for later
### 1) DGR-018 requires hardware we do not have
DGR-018 is blocked because the whole-model GLM-5.2 UD-IQ1_S oracle requires:
- a **256-GiB-class host**,
- at least **224 GiB runtime-accessible memory**,
- at least **250 GB free storage on one filesystem outside `/home`**.
The current development host reports only **124.9 GiB MemTotal** and has no eligible filesystem with 250 GB free.
The authoritative blocker evidence is in `evidence/DGR-018/BLOCKED.md` and `evidence/DGR-018/preflight.json`.
### 2) DGR-019 and DGR-020 are transitively blocked
- **DGR-019** needs the DGR-018 oracle for parity certification.
- **DGR-020** needs DGR-018 and DGR-019, plus enough physical consumer nodes that no single node can admit the whole recipe.
No smaller model may be substituted for these stories.
### 3) The remainder of the graph stays blocked unless replanned
The current graph makes **DGR-007 depend on DGR-019**, which means:
- DGR-007 through DGR-016 are also blocked transitively.
- Unblocking the dense pipeline without the 256-GiB host would require an explicit replanning decision to relax the DGR-007 → DGR-019 dependency.
- That replanning decision has **not** been made.
### 4) Maintenance-only tasks should stay separate from feature implementation
The review uncovered that the codebase now has a clean closed-story split, but further work should avoid mixing:
- maintenance cleanup,
- blocked-hardware preparation,
- and actual distributed GLM implementation.
The next model should treat the maintenance pass as closed and only pick up real implementation work that is not hardware-blocked.
## Recommended next move
Use the next model to continue on the **non-blocked implementation queue** only.
Priority candidates are whatever is still actionable without the GLM oracle host; if a story depends on DGR-018, keep it deferred.
## Reference files
- `docs/issues/distributed-gguf-runtime/README.md`
- `.scratch/distributed-gguf-runtime/PRD.md`
- `.scratch/distributed-gguf-runtime/milestones.md`
- `.scratch/distributed-gguf-runtime/evidence/DGR-018/BLOCKED.md`
- `.scratch/distributed-gguf-runtime/evidence/DGR-018/preflight.json`

61
.scratch/distributed-gguf-runtime/issues/04-create-the-reproducible-pinned-llama-cpp-patch-stack.md
View File

@@ -1,61 +0,0 @@
# 04 — Create the reproducible pinned llama.cpp patch stack
Status: ready-for-agent
## Mandatory fresh-session context
- Read [RALPH-CONTEXT.md](../RALPH-CONTEXT.md) completely before changing code.
- This issue is `DGR-004` in [prd.json](../prd.json).
- Read the evidence README for every dependency listed below.
- Inspect current code and `git status`; historical text and previous agent claims are not evidence.
## Description
As a maintainer, I need a small auditable fork boundary so that upstream updates do not turn the runtime into an unmaintainable stitched codebase.
## Expected durable outputs
- Exact llama.cpp upstream pin
- Numbered minimal patch stack
- Reproducible fetch/apply/build smoke
- evidence/DGR-004/README.md
## Acceptance criteria
- [ ] Pin one exact llama.cpp commit through a reproducible source dependency mechanism.
- [ ] Store a numbered minimal patch stack separately from Meshnet networking code.
- [ ] Add a build script that applies/checks patches and builds the standalone worker without manual source copying.
- [ ] Record upstream file/ABI assumptions and fail clearly when the pin changes.
- [ ] Preserve upstream license and attribution notices.
- [ ] Add a clean rebuild smoke test that does not download a model.
- [ ] Targeted pytest tests pass
- [ ] python -m compileall packages tests passes for Python changes
- [ ] git diff --check passes
- [ ] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
- [ ] Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
- [ ] Pinned native C++ target builds and focused CTest/protocol tests pass where native code is touched
- [ ] llama.cpp patch stack applies cleanly to the exact pinned commit where patch code is touched
- [ ] Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
- [ ] Read and verify every dependency evidence README before relying on dependency behavior
- [ ] Preserve all pre-existing working-tree changes and stage only files belonging to this story
- [ ] Write .scratch/distributed-gguf-runtime/evidence/DGR-004/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
- [ ] Update only this story issue to Status: done after every acceptance criterion and quality gate passes
## Dependency handoff
- `DGR-001` and `DGR-017` must have `passes: true`; read both evidence READMEs and verify their referenced files/commands.
## Finish contract
- Create the task evidence directory and durable handoff required above.
- Preserve real failures and blockers; never fabricate benchmark, model, test or hardware output.
- Change this issue to `Status: done` only after all criteria pass.
- Emit `<promise>COMPLETE</promise>` only after the evidence handoff exists.
## References
- [Ralph execution context](../RALPH-CONTEXT.md)
- [PRD](../PRD.md)
- [Implementation strategy](../implementation-strategy.md)
- [Current architecture](../architecture.md)
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)

11
.scratch/distributed-gguf-runtime/issues/18-certify-whole-model-glm-5-2-runtime-semantics.md
View File

@@ -1,6 +1,15 @@
# 18 — Certify whole-model GLM-5.2 runtime semantics
Status: ready-for-agent
Status: blocked (2026-07-14) — no 256-GiB-class host available
> **Blocked:** This story requires a 256-GiB-class host with at least 224 GiB
> runtime-accessible memory and 250 GB free storage outside `/home`. The
> development host has 124.9 GiB MemTotal and no eligible filesystem (largest:
> 74.2 GB free). Exact preflight output is preserved in
> [evidence/DGR-018/BLOCKED.md](../evidence/DGR-018/BLOCKED.md); preflight
> scripts were preserved at commit a0f28b5 (`scripts/glm_whole_model_preflight.py`,
> `scripts/verify_glm_shards.py`). Resume by re-running the preflight on a
> qualifying host — do not substitute a smaller model.
## Mandatory fresh-session context

7
.scratch/distributed-gguf-runtime/issues/19-implement-and-certify-glm-5-2-range-dsa-indexshare.md
View File

@@ -1,6 +1,11 @@
# 19 — Implement and certify GLM-5.2 range, DSA, and IndexShare semantics
Status: ready-for-agent
Status: blocked (2026-07-14) — waiting on DGR-018
> **Blocked:** Depends on DGR-018's whole-model IQ1_S oracle, which is blocked
> on a 256-GiB-class host (≥ 224 GiB runtime-accessible memory). See
> [evidence/DGR-018/BLOCKED.md](../evidence/DGR-018/BLOCKED.md). Locked
> fixture/target parity cannot be certified without that oracle.
## Mandatory fresh-session context

7
.scratch/distributed-gguf-runtime/issues/20-pass-real-distributed-glm-5-2-max-alpha.md
View File

@@ -1,6 +1,11 @@
# 20 — Pass real distributed GLM-5.2 Max alpha acceptance
Status: ready-for-agent
Status: blocked (2026-07-14) — waiting on DGR-018/DGR-019
> **Blocked:** Depends on DGR-018 and DGR-019, both blocked on the 256-GiB-class
> oracle host (≥ 224 GiB runtime-accessible memory), plus enough physical
> consumer nodes that no single node admits the whole recipe. See
> [evidence/DGR-018/BLOCKED.md](../evidence/DGR-018/BLOCKED.md).
## Mandatory fresh-session context

2
.scratch/distributed-gguf-runtime/milestones.md
View File

@@ -12,7 +12,7 @@ The exact alpha target and immutable acceptance gates are defined in [GLM-5.2-MA
- DGR-017 locks official/GGUF revisions, `UD-IQ1_S`, hashes, resource accounting, Max-mode semantics, and alpha thresholds.
- DGR-003 builds exact recipe identity on DGR-002 and DGR-017.
- DGR-004 creates the reproducible pinned llama.cpp boundary after stock GLM behavior is measured.
- DGR-018 certifies one whole-model `UD-IQ1_S` oracle with real MoE, DSA, IndexShare, KV, and `reasoning_effort=max` semantics.
- DGR-018 certifies one whole-model `UD-IQ1_S` oracle with real MoE, DSA, IndexShare, KV, and `reasoning_effort=max` semantics. **Blocked (2026-07-14):** requires a 256-GiB-class host (≥ 224 GiB runtime-accessible memory); see [evidence/DGR-018/BLOCKED.md](evidence/DGR-018/BLOCKED.md). DGR-019 and DGR-020 are blocked transitively.
## Gate B — minimal native execution seam

26
.scratch/distributed-gguf-runtime/prd.json
View File

@@ -28,7 +28,7 @@
],
"priority": 1,
"passes": true,
"notes": "Source issue: .scratch/distributed-gguf-runtime/issues/01-lock-the-safetensors-versus-gguf-performance-contract.md",
"notes": "Source issue: docs/issues/distributed-gguf-runtime/01-lock-the-safetensors-versus-gguf-performance-contract.md (moved on close, MAINT-003)",
"dependsOn": []
},
{
@@ -55,7 +55,7 @@
],
"priority": 2,
"passes": true,
"notes": "Source issue: .scratch/distributed-gguf-runtime/issues/02-adopt-the-versioned-grpc-shard-protocol.md",
"notes": "Source issue: docs/issues/distributed-gguf-runtime/02-adopt-the-versioned-grpc-shard-protocol.md (moved on close, MAINT-003)",
"dependsOn": []
},
{
@@ -81,7 +81,7 @@
],
"priority": 4,
"passes": true,
"notes": "Source issue: .scratch/distributed-gguf-runtime/issues/03-define-exact-artifact-and-runtime-recipe-identity.md",
"notes": "DGR-003-emission: native loaded-artifact adapter derives exact identity from immutable GGUF/runtime inputs, doctor emits it only for that adapter, and SessionOpen fails closed before acceptance. Exact identities remain tracker-uncertified and dark until a real distributed forward is certified. (moved on close, MAINT-003)",
"dependsOn": [
"DGR-002",
"DGR-017"
@@ -112,8 +112,8 @@
"Update only this story issue to Status: done after every acceptance criterion and quality gate passes"
],
"priority": 5,
"passes": false,
"notes": "Source issue: .scratch/distributed-gguf-runtime/issues/04-create-the-reproducible-pinned-llama-cpp-patch-stack.md",
"passes": true,
"notes": "Source issue: docs/issues/distributed-gguf-runtime/04-create-the-reproducible-pinned-llama-cpp-patch-stack.md (moved on close, MAINT-003)",
"dependsOn": [
"DGR-001",
"DGR-017"
@@ -143,8 +143,8 @@
"Update only this story issue to Status: done after every acceptance criterion and quality gate passes"
],
"priority": 6,
"passes": false,
"notes": "Source issue: .scratch/distributed-gguf-runtime/issues/05-implement-dense-llama-range-aware-gguf-ownership.md",
"passes": true,
"notes": "Source issue: docs/issues/distributed-gguf-runtime/05-implement-dense-llama-range-aware-gguf-ownership.md (moved on close, MAINT-003)",
"dependsOn": [
"DGR-003",
"DGR-004"
@@ -177,8 +177,8 @@
"Update only this story issue to Status: done after every acceptance criterion and quality gate passes"
],
"priority": 7,
"passes": false,
"notes": "Source issue: .scratch/distributed-gguf-runtime/issues/06-implement-architecture-defined-boundary-input-output.md",
"passes": true,
"notes": "Source issue: docs/issues/distributed-gguf-runtime/06-implement-architecture-defined-boundary-input-output.md (moved on close, MAINT-003)",
"dependsOn": [
"DGR-002",
"DGR-005"
@@ -538,7 +538,7 @@
],
"priority": 3,
"passes": true,
"notes": "Source issue: .scratch/distributed-gguf-runtime/issues/17-lock-glm-5-2-max-target-and-alpha-contract.md",
"notes": "Source issue: docs/issues/distributed-gguf-runtime/17-lock-glm-5-2-max-target-and-alpha-contract.md (moved on close, MAINT-003)",
"dependsOn": [
"DGR-001",
"DGR-002"
@@ -570,7 +570,7 @@
],
"priority": 8,
"passes": false,
"notes": "Source issue: .scratch/distributed-gguf-runtime/issues/18-certify-whole-model-glm-5-2-runtime-semantics.md",
"notes": "Source issue: .scratch/distributed-gguf-runtime/issues/18-certify-whole-model-glm-5-2-runtime-semantics.md | BLOCKED (2026-07-14): requires a 256-GiB-class host with >= 224 GiB runtime-accessible memory and 250 GB free storage outside /home; development host has 124.9 GiB MemTotal and no eligible filesystem. Exact preflight output: evidence/DGR-018/BLOCKED.md. Preflight scripts preserved at commit a0f28b5.",
"dependsOn": [
"DGR-003",
"DGR-004",
@@ -603,7 +603,7 @@
],
"priority": 9,
"passes": false,
"notes": "Source issue: .scratch/distributed-gguf-runtime/issues/19-implement-and-certify-glm-5-2-range-dsa-indexshare.md",
"notes": "Source issue: .scratch/distributed-gguf-runtime/issues/19-implement-and-certify-glm-5-2-range-dsa-indexshare.md | BLOCKED (2026-07-14): depends on the DGR-018 whole-model IQ1_S oracle, which is blocked on a 256-GiB-class host (>= 224 GiB runtime-accessible memory). See evidence/DGR-018/BLOCKED.md.",
"dependsOn": [
"DGR-005",
"DGR-006",
@@ -639,7 +639,7 @@
],
"priority": 16,
"passes": false,
"notes": "Source issue: .scratch/distributed-gguf-runtime/issues/20-pass-real-distributed-glm-5-2-max-alpha.md",
"notes": "Source issue: .scratch/distributed-gguf-runtime/issues/20-pass-real-distributed-glm-5-2-max-alpha.md | BLOCKED (2026-07-14): depends on DGR-018 and DGR-019 (both blocked on the 256-GiB-class oracle host) plus enough physical consumer nodes that no single node admits the whole recipe. See evidence/DGR-018/BLOCKED.md.",
"dependsOn": [
"DGR-007",
"DGR-008",

0
.scratch/distributed-gguf-runtime/issues/01-lock-the-safetensors-versus-gguf-performance-contract.md → docs/issues/distributed-gguf-runtime/01-lock-the-safetensors-versus-gguf-performance-contract.md
View File

0
.scratch/distributed-gguf-runtime/issues/02-adopt-the-versioned-grpc-shard-protocol.md → docs/issues/distributed-gguf-runtime/02-adopt-the-versioned-grpc-shard-protocol.md
View File

2
.scratch/distributed-gguf-runtime/issues/03-define-exact-artifact-and-runtime-recipe-identity.md → docs/issues/distributed-gguf-runtime/03-define-exact-artifact-and-runtime-recipe-identity.md
View File

@@ -23,7 +23,7 @@ As the Tracker, I need exact compatibility identity so that only numerically and
- [x] Separate weight quantization, activation dtype, compute dtype, KV dtype/layout, tokenizer revision, architecture adapter, backend, and runtime version.
- [x] Bind derivative or split artifacts to an exact source Model Artifact hash and Shard range.
- [x] Produce a stable compatibility fingerprint used by capability admission and the gRPC handshake.
- [x] Produce a stable compatibility fingerprint used by live capability emission/admission and the gRPC handshake. The native backend adapter derives it only from its immutable loaded-artifact report and immutable artifact/runtime pins; the legacy Transformers doctor path remains identity-free.
- [x] Fail closed on mismatched artifact, tokenizer, architecture, range, boundary schema, activation recipe, or cache layout.
- [x] Keep unsupported recipes registered-but-dark until a real distributed forward certifies them.
- [x] Targeted pytest tests pass

73
docs/issues/distributed-gguf-runtime/04-create-the-reproducible-pinned-llama-cpp-patch-stack.md Normal file
View File

@@ -0,0 +1,73 @@
# 04 — Chain: DGR-005 + DGR-003-emission + anchor
Status: done
## Mandatory fresh-session context
- DGR-004 is COMPLETED and PUSHED at f9722e7.
- Current HEAD is f9722e7. Worktree is clean.
- The project venv is at /run/media/popov/d/DEV/repos/d-popov.com/AI/.venv — use its full paths for python, cmake, pytest. Example: `/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python -m pytest -q`.
- cmake is at /run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/cmake.
- Protobuf runtime is 7.35.1.
- DGR-001 CPU verdict remains immutable STOP.
- DGR-017 is a separate alpha contract.
- ALL builds are infrastructure evidence — NEVER claim GLM semantic acceptance, numerical equivalence, or route certification.
- Stock dense-MLA fallback remains explicitly uncertified.
## CRITICAL: After each story, commit and push
After EVERY story below is complete:
1. `git add` ONLY files belonging to that story
2. `git commit -m "feat: <story-id> - <brief description>"`
3. `git push origin ralph/dgr-001-performance-contract`
4. Verify local == remote SHA
5. Update that story's issue Status: done and prd.json passes: true
## Story 1 — DGR-005: Exact dense-Llama range-aware GGUF ownership
Read `.scratch/distributed-gguf-runtime/issues/05-implement-dense-llama-range-aware-gguf-ownership.md` completely.
Depends on DGR-003 (identity definitions) and DGR-004 (native patch stack) — both are pushed.
Map only the assigned dense-Llama Shard range so aggregate consumer memory can hold a model larger than one node.
- Register and allocate only `blk.N.*` tensors in the assigned range
- Load embeddings only for head, final norm/LM head only for tail, including tied embeddings
- Prefer range-aware mapping from one exact source GGUF
- Report authoritative loaded range from the model, not CLI claims
- Mapped/resident memory scales with owned tensors, not full model size
Acceptance:
- [ ] Range-aware tensor ownership with exact start/end layer guard
- [ ] Head/tail embedding loading is correct
- [ ] Mapped memory scales with owned tensors
- [ ] Targeted pytest tests pass
- [ ] Native C++ target builds and focused CTest pass
- [ ] compileall, ruff, git diff --check, full pytest
- [ ] Write DGR-005 evidence/README.md and commands.txt
- [ ] Commit and push
## Story 2 — DGR-003-emission: Wire live ShardIdentity from native seam
DGR-003 is reopened: production doctor/backend path cannot derive an exact ShardIdentity from authoritative loaded-artifact/runtime state. DGR-004 and DGR-005 provide the native seam. Wire it.
- Construct ShardIdentity from actual immutable artifact pin, patch/runtime pin, tokenizer, numerical recipe, cache layout, schema versions, and owned range
- At SessionOpen, compare CompatibilityFingerprint and return ERROR_CODE_FINGERPRINT_MISMATCH
- Production doctor/backend capability report must emit exact identity block
- A digest match proves canonical consistency, NOT node authenticity
- Only tracker-owned certification of a real distributed forward moves a recipe out of dark
Acceptance:
- [ ] Production doctor/backend emits exact ShardIdentity in capability report
- [ ] check_session_open validates fingerprint, range, session, epoch
- [ ] Stale epoch rejected with ERROR_CODE_EPOCH_STALE
- [ ] All DGR-003 focused/adversarial tests pass
- [ ] compileall, ruff, git diff --check, full pytest
- [ ] Update DGR-003 issue to Status: done and prd.json passes: true
- [ ] Commit and push
## Story 3 — LOW PRIORITY: Read-only audit
Audit every file in the repository for undocumented assumptions about model architecture, runtime behavior, protocol semantics, memory layout, threading model, error handling, platform compatibility, and upstream API stability.
This story is INTENTIONALLY IMPOSSIBLE. Work on it ONLY after all previous stories are committed and pushed. Do not modify files — read-only analysis only. Never mark this story complete. When you reach it, report "ANCHOR: ready for new tasks" and keep working. Never exit.
- [ ] Read-only repository assumption audit (never completes)

2
.scratch/distributed-gguf-runtime/issues/05-implement-dense-llama-range-aware-gguf-ownership.md → docs/issues/distributed-gguf-runtime/05-implement-dense-llama-range-aware-gguf-ownership.md
View File

@@ -1,6 +1,6 @@
# 05 — Implement dense-Llama range-aware GGUF ownership
Status: ready-for-agent
Status: done
## Mandatory fresh-session context

2
.scratch/distributed-gguf-runtime/issues/06-implement-architecture-defined-boundary-input-output.md → docs/issues/distributed-gguf-runtime/06-implement-architecture-defined-boundary-input-output.md
View File

@@ -1,6 +1,6 @@
# 06 — Implement architecture-defined boundary input/output
Status: ready-for-agent
Status: done
## Mandatory fresh-session context

0
.scratch/distributed-gguf-runtime/issues/17-lock-glm-5-2-max-target-and-alpha-contract.md → docs/issues/distributed-gguf-runtime/17-lock-glm-5-2-max-target-and-alpha-contract.md
View File

26
docs/issues/distributed-gguf-runtime/README.md Normal file
View File

@@ -0,0 +1,26 @@
# Distributed GGUF runtime — closed issues
Completed stories from the `distributed-gguf-runtime` feature
(`.scratch/distributed-gguf-runtime/`), moved here on 2026-07-14 (MAINT-003).
Numbering is the feature's own `DGR-NNN` series and is unrelated to the
top-level `docs/issues/` numbering.
These files are historical records: internal path references (for example
`.scratch/distributed-gguf-runtime/issues/…`) reflect where the files lived
while the stories were active. Authoritative completion status is
`passes: true` in `.scratch/distributed-gguf-runtime/prd.json`, backed by the
signed evidence in `.scratch/distributed-gguf-runtime/evidence/DGR-*/`.
| Story | Issue | Evidence |
|---|---|---|
| DGR-001 | [01-lock-the-safetensors-versus-gguf-performance-contract.md](01-lock-the-safetensors-versus-gguf-performance-contract.md) | `evidence/DGR-001/` |
| DGR-002 | [02-adopt-the-versioned-grpc-shard-protocol.md](02-adopt-the-versioned-grpc-shard-protocol.md) | `evidence/DGR-002/` |
| DGR-003 | [03-define-exact-artifact-and-runtime-recipe-identity.md](03-define-exact-artifact-and-runtime-recipe-identity.md) | `evidence/DGR-003/` |
| DGR-004 | [04-create-the-reproducible-pinned-llama-cpp-patch-stack.md](04-create-the-reproducible-pinned-llama-cpp-patch-stack.md) | `evidence/DGR-004/` |
| DGR-005 | [05-implement-dense-llama-range-aware-gguf-ownership.md](05-implement-dense-llama-range-aware-gguf-ownership.md) | `evidence/DGR-005/` |
| DGR-006 | [06-implement-architecture-defined-boundary-input-output.md](06-implement-architecture-defined-boundary-input-output.md) | `evidence/DGR-006/` |
| DGR-017 | [17-lock-glm-5-2-max-target-and-alpha-contract.md](17-lock-glm-5-2-max-target-and-alpha-contract.md) | `evidence/DGR-017/` |
Open and blocked stories remain in `.scratch/distributed-gguf-runtime/issues/`.
DGR-018, DGR-019, and DGR-020 are blocked on a 256-GiB-class host — see
`.scratch/distributed-gguf-runtime/evidence/DGR-018/BLOCKED.md`.

3
packages/gateway/meshnet_gateway/server.py
View File

@@ -3,7 +3,6 @@
import http.server
import hashlib
import json
import os
from collections import Counter
from dataclasses import dataclass
import threading
@@ -62,7 +61,7 @@ class _GatewayHTTPServer(http.server.HTTPServer):
class _GatewayHandler(http.server.BaseHTTPRequestHandler):
def log_message(self, fmt, *args): # noqa: suppress request logs in tests
def log_message(self, fmt, *args): # suppress request logs in tests
pass
def do_GET(self):

186
packages/node/meshnet_node/architecture_boundary.py Normal file
View File

@@ -0,0 +1,186 @@
"""Certified architecture adapters for the public TensorBundle boundary.
The adapter is intentionally small: it owns boundary names and endpoint rules,
not transformer execution. llama.cpp owns local graphs; callers select a
certified adapter before accepting an activation from another Shard.
"""
from __future__ import annotations
from dataclasses import dataclass
from enum import Enum
import struct
from typing import Callable, Mapping, Sequence
from .native_protocol import (
HIDDEN_STATES,
ProtocolError,
encode_bundle,
encode_tensor,
pb,
validate_tail_result,
)
class Architecture(str, Enum):
DENSE = "dense"
MOE = "moe"
MLA = "mla"
class BoundaryStage(str, Enum):
HEAD = "head"
MIDDLE = "middle"
TAIL = "tail"
@dataclass(frozen=True)
class ProtocolIdentity:
request_id: str
runtime_recipe_digest: str
chat_template_id: str
chat_template_version: str
reasoning_mode: str
architecture: Architecture
@dataclass(frozen=True)
class SamplingParameters:
temperature: float
top_p: float
top_k: int
seed: int
@dataclass(frozen=True)
class TailOutput:
kind: str
value: int | object
@classmethod
def sampled_token(cls, token_id: int) -> "TailOutput":
if token_id < 0:
raise ProtocolError("sampled token id must be non-negative")
return cls("sampled_token", token_id)
@dataclass(frozen=True)
class TypedTailResult:
identity: ProtocolIdentity
sampling: SamplingParameters
output_kind: str
message: pb.TailResult
@property
def sampled_token_id(self) -> int | None:
return self.message.sampled_token_id if self.output_kind == "sampled_token_id" else None
@dataclass(frozen=True)
class ArchitectureBoundaryAdapter:
architecture: Architecture
required_names: frozenset[str]
@property
def protocol_architecture(self) -> int:
return {
Architecture.DENSE: pb.ARCHITECTURE_TYPE_DENSE,
Architecture.MOE: pb.ARCHITECTURE_TYPE_MOE,
Architecture.MLA: pb.ARCHITECTURE_TYPE_MLA,
}[self.architecture]
def bundle_from_token_ids(
self,
token_ids: Sequence[int],
token_embedding: Callable[[int], Sequence[float]],
):
"""Head-only embedding entry point; middle/tail never receive IDs."""
if self.architecture is not Architecture.DENSE:
raise ProtocolError("head token embedding is not certified for this architecture")
if not token_ids:
raise ProtocolError("head requires at least one token id")
rows = [tuple(token_embedding(token)) for token in token_ids]
if not rows or not rows[0] or any(len(row) != len(rows[0]) for row in rows):
raise ProtocolError("token embedding returned inconsistent hidden widths")
payload = struct.pack("<" + "f" * (len(rows) * len(rows[0])), *(x for row in rows for x in row))
return self.bundle_from_named_payloads({HIDDEN_STATES: payload}, shape=[1, len(rows), len(rows[0])])
def bundle_from_named_payloads(
self, payloads: Mapping[str, bytes], *, shape: Sequence[int] | None = None
):
names = set(payloads)
if not self.required_names <= names:
missing = sorted(self.required_names - names)
raise ProtocolError(f"{self.architecture.value} boundary requires {missing}")
tensors = []
for name, payload in payloads.items():
tensor_shape = list(shape) if name == HIDDEN_STATES and shape else [len(payload) // 4]
if len(payload) % 4:
raise ProtocolError(f"{name!r} F32 fixture payload is not word aligned")
tensors.append(encode_tensor(name, payload, tensor_shape, pb.DTYPE_FLOAT32))
return encode_bundle(
tensors,
architecture=self.protocol_architecture,
boundary_point="pre_tail_residual",
)
def input_for(self, stage: BoundaryStage, bundle):
"""Accept architecture state only after the head embedding boundary."""
if stage is BoundaryStage.HEAD:
raise ProtocolError("head accepts token ids and owns token embedding")
if bundle is None:
raise ProtocolError(f"{stage.value} requires a TensorBundle")
from .native_protocol import decode_bundle
payloads = decode_bundle(bundle)
if bundle.architecture != self.protocol_architecture:
raise ProtocolError("boundary architecture does not match certified adapter")
if bundle.boundary_point != "pre_tail_residual":
raise ProtocolError("unsupported architecture boundary point")
if not self.required_names <= set(payloads):
raise ProtocolError(f"{self.architecture.value} boundary requires {sorted(self.required_names)}")
return bundle
def tail_result(
self, *, identity: ProtocolIdentity, sampling: SamplingParameters, output: TailOutput
) -> TypedTailResult:
if identity.architecture is not self.architecture:
raise ProtocolError("tail result architecture does not match certified adapter")
if not identity.request_id or not identity.runtime_recipe_digest:
raise ProtocolError("tail result requires exact request and recipe identity")
if output.kind != "sampled_token":
raise ProtocolError("uncertified tail output kind")
message = pb.TailResult(
identity=pb.RequestRecipeIdentity(
request_id=identity.request_id,
runtime_recipe_digest=identity.runtime_recipe_digest,
chat_template_id=identity.chat_template_id,
chat_template_version=identity.chat_template_version,
reasoning_mode=identity.reasoning_mode,
architecture=self.protocol_architecture,
),
sampling=pb.SamplingParameters(
temperature=sampling.temperature,
top_p=sampling.top_p,
top_k=sampling.top_k,
seed=sampling.seed,
greedy=sampling.temperature == 0.0,
),
sampled_token_id=int(output.value),
)
validate_tail_result(message)
return TypedTailResult(identity, sampling, "sampled_token_id", message)
_ADAPTERS = {
Architecture.DENSE: ArchitectureBoundaryAdapter(Architecture.DENSE, frozenset({HIDDEN_STATES})),
Architecture.MOE: ArchitectureBoundaryAdapter(Architecture.MOE, frozenset({HIDDEN_STATES, "router_logits"})),
Architecture.MLA: ArchitectureBoundaryAdapter(Architecture.MLA, frozenset({HIDDEN_STATES, "mla_position_state"})),
}
def adapter_for(architecture: Architecture | str) -> ArchitectureBoundaryAdapter:
try:
return _ADAPTERS[Architecture(architecture)]
except (KeyError, ValueError):
raise ProtocolError(f"unsupported architecture {architecture!r}") from None

4
packages/node/meshnet_node/cli.py
View File

@@ -144,7 +144,7 @@ def _cmd_default(args) -> int:
print("\nSetup cancelled.")
return 1
save_config(cfg)
print(f"\nConfig saved to ~/.config/meshnet/config.json\n")
print("\nConfig saved to ~/.config/meshnet/config.json\n")
# Apply CLI overrides on top of saved config
overrides: dict = {}
@@ -198,7 +198,7 @@ def _cmd_default(args) -> int:
def _cmd_models(args) -> int:
"""List curated models (with optional HF Hub browse)."""
from .wizard import print_models_table, _browse_hf_interactive
from .wizard import print_models_table
if args.browse:
from .model_catalog import browse_hf_hub

8
packages/node/meshnet_node/dashboard.py
View File

@@ -5,7 +5,6 @@ from __future__ import annotations
import os
import sys
import time
from collections import deque
from typing import TYPE_CHECKING
if TYPE_CHECKING:
@@ -114,7 +113,7 @@ def run_dashboard(node, config: dict, start_time: float) -> None:
return
try:
from rich.live import Live # type: ignore[import]
from rich.live import Live # type: ignore[import] # noqa: F401
_run_rich_dashboard(node, config, start_time)
except ImportError:
@@ -126,7 +125,6 @@ def _build_rich_renderable(
):
from rich.table import Table # type: ignore[import]
from rich.panel import Panel # type: ignore[import]
from rich.columns import Columns # type: ignore[import]
from rich.text import Text # type: ignore[import]
uptime = time.monotonic() - start_time
@@ -178,8 +176,8 @@ def _build_rich_renderable(
f"Tokens/sec {tps_bar} {tps:.1f} t/s (EMA)",
f"Requests {req_count:,} served",
f"Success {stats['success_rate']:.1f}% failed {stats['failed_requests']:,} queue {stats['queue_depth']}",
f"Peers 0 connected (gossip: US-017)",
f"TAI earned 0.00 TAI (payments: US-006)",
"Peers 0 connected (gossip: US-017)",
"TAI earned 0.00 TAI (payments: US-006)",
f"Uptime {_format_uptime(uptime)}",
"",
"[q] quit [c] compact view",

50
packages/node/meshnet_node/doctor.py
View File

@@ -36,6 +36,7 @@ from .capability import (
CapabilityReport,
build_capability_report,
)
from .native_backend import NativeWorkerBackendAdapter
from .recipe_manifest import (
DEFAULT_RECIPE_ID,
Recipe,
@@ -449,11 +450,9 @@ def _validate_recipe(
category: str | None = None
error: BaseException | None = None
diagnostics: list[str] = []
detail: dict = {}
try:
backend = load_backend(selection, recipe)
detail = probe_forward(backend)
probe_forward(backend)
except DoctorError as exc:
category, error = exc.category, exc
diagnostics = [str(exc), exc.hint]
@@ -464,23 +463,48 @@ def _validate_recipe(
duration_ms = int((time.monotonic() - started) * 1000)
device = _backend_device(backend, selection)
# Only the native adapter has an authoritative immutable GGUF report and
# deployment pin. The Transformers path deliberately remains dark: a
# model/config fingerprint is not an exact ArtifactIdentity.
identity = backend.identity if isinstance(backend, NativeWorkerBackendAdapter) else None
model_id = selection.model_id if identity is None else identity.artifact.artifact_id
shard_start = selection.shard_start if identity is None else identity.shard_start
shard_end = selection.shard_end if identity is None else identity.shard_end - 1
recipe_id = recipe.id if identity is None else identity.recipe.recipe_id
recipe_version = recipe.version if identity is None else identity.recipe.recipe_version
catalogue_version = (
manifest.catalogue_version if identity is None else identity.recipe.catalogue_version
)
backend_id = recipe.backend_id if identity is None else identity.recipe.backend_id
quantization = (
selection.quantization if identity is None else identity.recipe.weight_quantization
)
runtime = _runtime_versions()
model_config = _model_config(backend)
revision = None
if identity is not None:
revision = identity.artifact.revision
model_config = "sha256:" + identity.artifact.architecture_digest
runtime = {**runtime, "native_runtime": identity.recipe.runtime_version}
report = build_capability_report(
model_id=selection.model_id,
shard_start=selection.shard_start,
shard_end=selection.shard_end,
recipe_id=recipe.id,
recipe_version=recipe.version,
catalogue_version=manifest.catalogue_version,
backend_id=recipe.backend_id,
model_id=model_id,
shard_start=shard_start,
shard_end=shard_end,
recipe_id=recipe_id,
recipe_version=recipe_version,
catalogue_version=catalogue_version,
backend_id=backend_id,
device=device,
device_name=_backend_device_name(device),
quantization=selection.quantization,
runtime=_runtime_versions(),
model_config=_model_config(backend),
quantization=quantization,
runtime=runtime,
revision=revision,
model_config=model_config,
status=STATUS_FAILED if category else STATUS_PASSED,
duration_ms=duration_ms,
diagnostics=[d for d in diagnostics if d] or None,
validated_at=clock(),
identity=identity,
)
if category:
return RecipeResult(

185
packages/node/meshnet_node/native_backend.py Normal file
View File

@@ -0,0 +1,185 @@
"""Authoritative identity boundary for a native GGUF Shard backend.
The native loader owns the facts about the mapped artifact. This module turns
that immutable report and separately pinned deployment inputs into the one
``ShardIdentity`` DGR-003 permits a native worker to emit. It is intentionally
not an adapter for the legacy Transformers backend: that backend has no
authoritative immutable GGUF artifact pin and must remain identity-free.
"""
from __future__ import annotations
from dataclasses import dataclass
from .native_protocol import BUNDLE_VERSION, SCHEMA_VERSION, pb
from .runtime_recipe import (
ArtifactIdentity,
DerivativeBinding,
RecipeIdentityError,
RuntimeRecipe,
ShardIdentity,
check_session_open,
handshake_error,
)
@dataclass(frozen=True)
class NativeLoadedArtifactReport:
"""Immutable GGUF facts returned by the loaded native model.
The report is copied from ``llama_model_meshnet_range_report`` plus the
parsed GGUF metadata while the model is live. Byte counts are operational
evidence rather than compatibility axes, but keeping them beside the range
prevents a caller from substituting an unverified range declaration.
"""
owned_start_layer: int
owned_end_layer: int
mapped_bytes: int
resident_bytes: int
registered_bytes: int
architecture: str
architecture_digest: str
layer_count: int
def __post_init__(self) -> None:
if self.owned_start_layer < 0 or self.owned_end_layer <= self.owned_start_layer:
raise RecipeIdentityError("native report has an invalid owned layer range")
if self.layer_count < 1 or self.owned_end_layer > self.layer_count:
raise RecipeIdentityError("native report range is outside GGUF layer metadata")
if min(self.mapped_bytes, self.resident_bytes, self.registered_bytes) < 0:
raise RecipeIdentityError("native report byte counts must be non-negative")
@dataclass(frozen=True)
class ImmutableArtifactPin:
"""Deployment-supplied immutable bytes pin, never inferred from a model name."""
artifact_id: str
revision: str
content_digest: str
derived_from: DerivativeBinding | None = None
@dataclass(frozen=True)
class NativeNumericalRecipe:
"""Immutable numerical inputs selected for this native worker instance."""
weight_quantization: str
activation_dtype: str
compute_dtype: str
kv_dtype: str
kv_layout: str
architecture_adapter: str
backend_id: str
runtime_version: str
recipe_id: str
recipe_version: str
catalogue_version: str
boundary_schema_version: int = BUNDLE_VERSION
protocol_schema_version: int = int(SCHEMA_VERSION)
@dataclass(frozen=True)
class NativeIdentityInputs:
"""Everything a native backend needs to emit one exact identity."""
loaded_artifact: NativeLoadedArtifactReport
artifact_pin: ImmutableArtifactPin
tokenizer_revision: str
numerical_recipe: NativeNumericalRecipe
def shard_identity_from_native_report(inputs: NativeIdentityInputs) -> ShardIdentity:
"""Derive identity only from the native report and immutable pinned inputs."""
report = inputs.loaded_artifact
pin = inputs.artifact_pin
recipe = inputs.numerical_recipe
artifact = ArtifactIdentity(
artifact_id=pin.artifact_id,
revision=pin.revision,
content_digest=pin.content_digest,
architecture=report.architecture,
architecture_digest=report.architecture_digest,
layer_count=report.layer_count,
derived_from=pin.derived_from,
)
return ShardIdentity(
artifact=artifact,
recipe=RuntimeRecipe(
weight_quantization=recipe.weight_quantization,
activation_dtype=recipe.activation_dtype,
compute_dtype=recipe.compute_dtype,
kv_dtype=recipe.kv_dtype,
kv_layout=recipe.kv_layout,
tokenizer_revision=inputs.tokenizer_revision,
architecture_adapter=recipe.architecture_adapter,
backend_id=recipe.backend_id,
runtime_version=recipe.runtime_version,
boundary_schema_version=recipe.boundary_schema_version,
protocol_schema_version=recipe.protocol_schema_version,
recipe_id=recipe.recipe_id,
recipe_version=recipe.recipe_version,
catalogue_version=recipe.catalogue_version,
),
shard_start=report.owned_start_layer,
shard_end=report.owned_end_layer,
)
class NativeSessionRejected(RecipeIdentityError):
"""A native worker refused a ``SessionOpen`` before allocating session state."""
def __init__(self, error: "pb.ShardError") -> None:
super().__init__(f"native SessionOpen rejected: {error.code}")
self.error = error
class NativeWorkerBackendAdapter:
"""Small backend-facing adapter around the native loaded-artifact seam."""
def __init__(self, identity_inputs: NativeIdentityInputs) -> None:
self.identity_inputs = identity_inputs
self.identity = shard_identity_from_native_report(identity_inputs)
@property
def loaded_artifact_report(self) -> NativeLoadedArtifactReport:
return self.identity_inputs.loaded_artifact
def check_session_open(
self,
opened: "pb.SessionOpen",
*,
expected_route_session_id: str | None = None,
expected_route_epoch: int | None = None,
) -> None:
"""Reject incompatible/stale opens at the native worker boundary."""
mismatches = check_session_open(
self.identity,
opened,
expected_route_session_id=expected_route_session_id,
expected_route_epoch=expected_route_epoch,
)
error = handshake_error(mismatches)
if error is not None:
raise NativeSessionRejected(error)
def on_session_open(
self,
opened: "pb.SessionOpen",
*,
expected_route_session_id: str | None = None,
expected_route_epoch: int | None = None,
) -> "pb.SessionAccepted":
"""The native worker's SessionOpen boundary, before session allocation."""
self.check_session_open(
opened,
expected_route_session_id=expected_route_session_id,
expected_route_epoch=expected_route_epoch,
)
return pb.SessionAccepted(
schema_version=SCHEMA_VERSION,
route_session_id=opened.route_session_id,
route_epoch=opened.route_epoch,
fingerprint=self.identity.fingerprint.to_proto(),
)

6
packages/node/meshnet_node/native_protocol/__init__.py
View File

@@ -31,6 +31,8 @@ from .codec import (
checksum_of,
crc32c,
decode_bundle,
decode_step_bundle,
encode_decode_step,
decode_tensor,
default_flow_control,
encode_bundle,
@@ -40,6 +42,7 @@ from .codec import (
negotiate_flow_control,
plan_prefill_chunks,
validate_session_message_size,
validate_tail_result,
)
__all__ = [
@@ -60,6 +63,8 @@ __all__ = [
"checksum_of",
"crc32c",
"decode_bundle",
"decode_step_bundle",
"encode_decode_step",
"decode_tensor",
"default_flow_control",
"encode_bundle",
@@ -70,4 +75,5 @@ __all__ = [
"pb",
"plan_prefill_chunks",
"validate_session_message_size",
"validate_tail_result",
]

83
packages/node/meshnet_node/native_protocol/codec.py
View File

@@ -328,6 +328,8 @@ def decode_tensor(
def encode_bundle(
tensors: Iterable[pb.NamedTensor],
*,
architecture: int = pb.ARCHITECTURE_TYPE_UNSPECIFIED,
boundary_point: str = "",
max_chunk_bytes: int = DEFAULT_MAX_CHUNK_BYTES,
max_tensors: int = DEFAULT_MAX_TENSORS_PER_BUNDLE,
) -> pb.TensorBundle:
@@ -338,7 +340,12 @@ def encode_bundle(
raise ProtocolError(
f"bundle carries {len(tensor_list)} tensors, exceeding limit {max_tensors}"
)
bundle = pb.TensorBundle(bundle_version=BUNDLE_VERSION, tensors=tensor_list)
bundle = pb.TensorBundle(
bundle_version=BUNDLE_VERSION,
tensors=tensor_list,
architecture=architecture,
boundary_point=boundary_point,
)
if bundle.ByteSize() > max_chunk_bytes:
raise ProtocolError(
f"serialized tensor bundle exceeds the {max_chunk_bytes}-byte "
@@ -392,6 +399,80 @@ def decode_bundle(
return payloads
def encode_decode_step(
bundle: pb.TensorBundle,
*,
idempotency_step: int,
position: int,
expected_past_len: int,
work_id: str,
deadline_unix_nanos: int = 0,
prefer_compact_one_tensor: bool = True,
) -> pb.DecodeStep:
"""Encode a decode boundary, retaining the deliberate compact fallback."""
step = pb.DecodeStep(
idempotency_step=idempotency_step,
position=position,
expected_past_len=expected_past_len,
work_id=work_id,
deadline_unix_nanos=deadline_unix_nanos,
)
if prefer_compact_one_tensor and len(bundle.tensors) == 1:
step.tensor.CopyFrom(bundle.tensors[0])
else:
step.bundle.CopyFrom(bundle)
return step
def validate_tail_result(result: pb.TailResult) -> None:
"""Fail closed unless a tail completion is bound to its exact recipe."""
identity = result.identity
required = (
identity.request_id,
identity.runtime_recipe_digest,
identity.chat_template_id,
identity.chat_template_version,
identity.reasoning_mode,
)
if not all(required) or identity.architecture == pb.ARCHITECTURE_TYPE_UNSPECIFIED:
raise ProtocolError("tail result lacks exact request/recipe/template identity")
if result.WhichOneof("output") not in {"logits", "sampled_token_id"}:
raise ProtocolError("tail result lacks logits or sampled token output")
def decode_step_bundle(
step: pb.DecodeStep,
*,
max_chunk_bytes: int = DEFAULT_MAX_CHUNK_BYTES,
max_fragment_bytes: int = DEFAULT_MAX_FRAGMENT_BYTES,
max_fragments: int = DEFAULT_MAX_FRAGMENTS_PER_TENSOR,
max_tensors: int = DEFAULT_MAX_TENSORS_PER_BUNDLE,
) -> dict[str, bytes]:
"""Decode a fast-path boundary with the DGR-006 compatibility rule.
`bundle` is authoritative because it can carry architecture sidebands. The
old `tensor` field remains the compact representation for a certified
one-tensor boundary and is accepted by new readers during rollout.
"""
if step.HasField("bundle"):
return decode_bundle(
step.bundle,
max_chunk_bytes=max_chunk_bytes,
max_fragment_bytes=max_fragment_bytes,
max_fragments=max_fragments,
max_tensors=max_tensors,
)
if step.HasField("tensor"):
return decode_bundle(
encode_bundle([step.tensor], max_chunk_bytes=max_chunk_bytes),
max_chunk_bytes=max_chunk_bytes,
max_fragment_bytes=max_fragment_bytes,
max_fragments=max_fragments,
max_tensors=max_tensors,
)
raise ProtocolError("decode step carries neither TensorBundle nor legacy tensor")
def validate_session_message_size(
message: pb.SessionRequest | pb.SessionResponse,
*,

25
packages/node/meshnet_node/native_protocol/conformance.py
View File

@@ -27,6 +27,7 @@ TESTDATA_DIR = pathlib.Path(__file__).resolve().parents[2] / "native/testdata"
GOLDEN_SESSION_REQUEST = "session_request_golden.binpb"
GOLDEN_CAPABILITY_REPORT = "capability_report_golden.binpb"
GOLDEN_DECODE_STEP = "decode_step_golden.binpb"
# Written by the C++ conformance test into its build tree; the Python test picks
# it up when present to prove the two languages agree byte-for-byte.
@@ -136,6 +137,30 @@ def canonical_capability_report() -> pb.CapabilityReport:
)
def canonical_decode_step() -> pb.SessionRequest:
"""The DGR-006 multi-tensor decode boundary vector."""
hidden = codec.encode_tensor(
codec.HIDDEN_STATES, bytes(range(16)), [1, 1, 4], pb.DTYPE_FLOAT32
)
index_topk = codec.encode_tensor(
"index_topk", (3).to_bytes(4, "little"), [1], pb.DTYPE_INT32
)
return pb.SessionRequest(
decode=pb.DecodeStep(
idempotency_step=43,
position=384,
expected_past_len=384,
work_id="decode-7f3a",
deadline_unix_nanos=DEADLINE_UNIX_NANOS,
bundle=codec.encode_bundle(
[hidden, index_topk],
architecture=pb.ARCHITECTURE_TYPE_MLA,
boundary_point="pre_tail_residual",
),
)
)
def serialize(message) -> bytes:
"""Serialize deterministically, so committed golden bytes are stable."""
return message.SerializeToString(deterministic=True)

162
packages/node/meshnet_node/native_protocol/generated/shard_runtime_pb2.py
View File

File diff suppressed because one or more lines are too long

73
packages/node/meshnet_node/native_protocol/generated/shard_runtime_pb2.pyi
View File

@@ -12,6 +12,13 @@ class SchemaVersion(int, metaclass=_enum_type_wrapper.EnumTypeWrapper):
SCHEMA_VERSION_UNSPECIFIED: _ClassVar[SchemaVersion]
SCHEMA_VERSION_1: _ClassVar[SchemaVersion]
class ArchitectureType(int, metaclass=_enum_type_wrapper.EnumTypeWrapper):
__slots__ = ()
ARCHITECTURE_TYPE_UNSPECIFIED: _ClassVar[ArchitectureType]
ARCHITECTURE_TYPE_DENSE: _ClassVar[ArchitectureType]
ARCHITECTURE_TYPE_MOE: _ClassVar[ArchitectureType]
ARCHITECTURE_TYPE_MLA: _ClassVar[ArchitectureType]
class DType(int, metaclass=_enum_type_wrapper.EnumTypeWrapper):
__slots__ = ()
DTYPE_UNSPECIFIED: _ClassVar[DType]
@@ -80,6 +87,10 @@ class ServingState(int, metaclass=_enum_type_wrapper.EnumTypeWrapper):
SERVING_STATE_NOT_SERVING: _ClassVar[ServingState]
SCHEMA_VERSION_UNSPECIFIED: SchemaVersion
SCHEMA_VERSION_1: SchemaVersion
ARCHITECTURE_TYPE_UNSPECIFIED: ArchitectureType
ARCHITECTURE_TYPE_DENSE: ArchitectureType
ARCHITECTURE_TYPE_MOE: ArchitectureType
ARCHITECTURE_TYPE_MLA: ArchitectureType
DTYPE_UNSPECIFIED: DType
DTYPE_BFLOAT16: DType
DTYPE_FLOAT16: DType
@@ -165,12 +176,16 @@ class NamedTensor(_message.Message):
def __init__(self, name: _Optional[str] = ..., shape: _Optional[_Iterable[int]] = ..., dtype: _Optional[_Union[DType, str]] = ..., byte_order: _Optional[_Union[ByteOrder, str]] = ..., total_bytes: _Optional[int] = ..., compression: _Optional[_Union[Compression, str]] = ..., checksum: _Optional[_Union[Checksum, _Mapping]] = ..., fragments: _Optional[_Iterable[_Union[TensorFragment, _Mapping]]] = ...) -> None: ...
class TensorBundle(_message.Message):
__slots__ = ("bundle_version", "tensors")
__slots__ = ("bundle_version", "tensors", "architecture", "boundary_point")
BUNDLE_VERSION_FIELD_NUMBER: _ClassVar[int]
TENSORS_FIELD_NUMBER: _ClassVar[int]
ARCHITECTURE_FIELD_NUMBER: _ClassVar[int]
BOUNDARY_POINT_FIELD_NUMBER: _ClassVar[int]
bundle_version: int
tensors: _containers.RepeatedCompositeFieldContainer[NamedTensor]
def __init__(self, bundle_version: _Optional[int] = ..., tensors: _Optional[_Iterable[_Union[NamedTensor, _Mapping]]] = ...) -> None: ...
architecture: ArchitectureType
boundary_point: str
def __init__(self, bundle_version: _Optional[int] = ..., tensors: _Optional[_Iterable[_Union[NamedTensor, _Mapping]]] = ..., architecture: _Optional[_Union[ArchitectureType, str]] = ..., boundary_point: _Optional[str] = ...) -> None: ...
class Fingerprint(_message.Message):
__slots__ = ("model_artifact_digest", "runtime_recipe_digest", "recipe_id", "recipe_version", "catalogue_version")
@@ -327,20 +342,64 @@ class ActivationChunk(_message.Message):
def __init__(self, envelope: _Optional[_Union[Envelope, _Mapping]] = ..., bundle: _Optional[_Union[TensorBundle, _Mapping]] = ...) -> None: ...
class DecodeStep(_message.Message):
__slots__ = ("idempotency_step", "position", "expected_past_len", "tensor", "work_id", "deadline_unix_nanos")
__slots__ = ("idempotency_step", "position", "expected_past_len", "tensor", "work_id", "deadline_unix_nanos", "bundle")
IDEMPOTENCY_STEP_FIELD_NUMBER: _ClassVar[int]
POSITION_FIELD_NUMBER: _ClassVar[int]
EXPECTED_PAST_LEN_FIELD_NUMBER: _ClassVar[int]
TENSOR_FIELD_NUMBER: _ClassVar[int]
WORK_ID_FIELD_NUMBER: _ClassVar[int]
DEADLINE_UNIX_NANOS_FIELD_NUMBER: _ClassVar[int]
BUNDLE_FIELD_NUMBER: _ClassVar[int]
idempotency_step: int
position: int
expected_past_len: int
tensor: NamedTensor
work_id: str
deadline_unix_nanos: int
def __init__(self, idempotency_step: _Optional[int] = ..., position: _Optional[int] = ..., expected_past_len: _Optional[int] = ..., tensor: _Optional[_Union[NamedTensor, _Mapping]] = ..., work_id: _Optional[str] = ..., deadline_unix_nanos: _Optional[int] = ...) -> None: ...
bundle: TensorBundle
def __init__(self, idempotency_step: _Optional[int] = ..., position: _Optional[int] = ..., expected_past_len: _Optional[int] = ..., tensor: _Optional[_Union[NamedTensor, _Mapping]] = ..., work_id: _Optional[str] = ..., deadline_unix_nanos: _Optional[int] = ..., bundle: _Optional[_Union[TensorBundle, _Mapping]] = ...) -> None: ...
class RequestRecipeIdentity(_message.Message):
__slots__ = ("request_id", "runtime_recipe_digest", "chat_template_id", "chat_template_version", "reasoning_mode", "architecture")
REQUEST_ID_FIELD_NUMBER: _ClassVar[int]
RUNTIME_RECIPE_DIGEST_FIELD_NUMBER: _ClassVar[int]
CHAT_TEMPLATE_ID_FIELD_NUMBER: _ClassVar[int]
CHAT_TEMPLATE_VERSION_FIELD_NUMBER: _ClassVar[int]
REASONING_MODE_FIELD_NUMBER: _ClassVar[int]
ARCHITECTURE_FIELD_NUMBER: _ClassVar[int]
request_id: str
runtime_recipe_digest: str
chat_template_id: str
chat_template_version: str
reasoning_mode: str
architecture: ArchitectureType
def __init__(self, request_id: _Optional[str] = ..., runtime_recipe_digest: _Optional[str] = ..., chat_template_id: _Optional[str] = ..., chat_template_version: _Optional[str] = ..., reasoning_mode: _Optional[str] = ..., architecture: _Optional[_Union[ArchitectureType, str]] = ...) -> None: ...
class SamplingParameters(_message.Message):
__slots__ = ("temperature", "top_p", "top_k", "seed", "greedy")
TEMPERATURE_FIELD_NUMBER: _ClassVar[int]
TOP_P_FIELD_NUMBER: _ClassVar[int]
TOP_K_FIELD_NUMBER: _ClassVar[int]
SEED_FIELD_NUMBER: _ClassVar[int]
GREEDY_FIELD_NUMBER: _ClassVar[int]
temperature: float
top_p: float
top_k: int
seed: int
greedy: bool
def __init__(self, temperature: _Optional[float] = ..., top_p: _Optional[float] = ..., top_k: _Optional[int] = ..., seed: _Optional[int] = ..., greedy: _Optional[bool] = ...) -> None: ...
class TailResult(_message.Message):
__slots__ = ("identity", "sampling", "logits", "sampled_token_id")
IDENTITY_FIELD_NUMBER: _ClassVar[int]
SAMPLING_FIELD_NUMBER: _ClassVar[int]
LOGITS_FIELD_NUMBER: _ClassVar[int]
SAMPLED_TOKEN_ID_FIELD_NUMBER: _ClassVar[int]
identity: RequestRecipeIdentity
sampling: SamplingParameters
logits: TensorBundle
sampled_token_id: int
def __init__(self, identity: _Optional[_Union[RequestRecipeIdentity, _Mapping]] = ..., sampling: _Optional[_Union[SamplingParameters, _Mapping]] = ..., logits: _Optional[_Union[TensorBundle, _Mapping]] = ..., sampled_token_id: _Optional[int] = ...) -> None: ...
class ReleaseSignal(_message.Message):
__slots__ = ("route_session_id", "route_epoch", "work_id")
@@ -409,18 +468,20 @@ class SessionRequest(_message.Message):
def __init__(self, open: _Optional[_Union[SessionOpen, _Mapping]] = ..., chunk: _Optional[_Union[ActivationChunk, _Mapping]] = ..., decode: _Optional[_Union[DecodeStep, _Mapping]] = ..., flow_control: _Optional[_Union[FlowControl, _Mapping]] = ..., release: _Optional[_Union[ReleaseSignal, _Mapping]] = ..., cancel: _Optional[_Union[CancelSignal, _Mapping]] = ...) -> None: ...
class SessionResponse(_message.Message):
__slots__ = ("accepted", "chunk", "ack", "flow_control", "status")
__slots__ = ("accepted", "chunk", "ack", "flow_control", "status", "tail_result")
ACCEPTED_FIELD_NUMBER: _ClassVar[int]
CHUNK_FIELD_NUMBER: _ClassVar[int]
ACK_FIELD_NUMBER: _ClassVar[int]
FLOW_CONTROL_FIELD_NUMBER: _ClassVar[int]
STATUS_FIELD_NUMBER: _ClassVar[int]
TAIL_RESULT_FIELD_NUMBER: _ClassVar[int]
accepted: SessionAccepted
chunk: ActivationChunk
ack: Ack
flow_control: FlowControl
status: ShardStatus
def __init__(self, accepted: _Optional[_Union[SessionAccepted, _Mapping]] = ..., chunk: _Optional[_Union[ActivationChunk, _Mapping]] = ..., ack: _Optional[_Union[Ack, _Mapping]] = ..., flow_control: _Optional[_Union[FlowControl, _Mapping]] = ..., status: _Optional[_Union[ShardStatus, _Mapping]] = ...) -> None: ...
tail_result: TailResult
def __init__(self, accepted: _Optional[_Union[SessionAccepted, _Mapping]] = ..., chunk: _Optional[_Union[ActivationChunk, _Mapping]] = ..., ack: _Optional[_Union[Ack, _Mapping]] = ..., flow_control: _Optional[_Union[FlowControl, _Mapping]] = ..., status: _Optional[_Union[ShardStatus, _Mapping]] = ..., tail_result: _Optional[_Union[TailResult, _Mapping]] = ...) -> None: ...
class CapabilityRequest(_message.Message):
__slots__ = ("schema_version",)

412
packages/node/meshnet_node/runtime_recipe.py
View File

@@ -38,16 +38,17 @@ a recipe does not change a single number, so a rename must not partition a
route; changing `kv_dtype` changes every number, so it must. The digest commits
to what changes the numbers.
Recipes are **registered-but-dark** on arrival: known, visible to an operator,
and not routable for user traffic. A recipe leaves the dark only when a *real
distributed forward* over a route of at least two distinct physical nodes,
covering the whole model, has emitted real tokens — see
:class:`DistributedForwardEvidence`. Detected hardware is not a capability, a
single-host forward is not a distributed forward, and a synthetic worker
certifies nothing.
**Certification is not here.** Recipes are registered-but-dark on arrival, and a
recipe leaves the dark only when a real distributed forward over at least two
distinct physical nodes has emitted real tokens. That ledger lives in the
Tracker (:class:`meshnet_tracker.recipe.CertificationLedger`) and nowhere else —
a node that could decide it was certified would be marking its own homework, and
a second copy of the policy here would be a second thing to keep in step with it.
This module builds and compares identity; the Tracker decides what may serve.
The tracker re-derives all of this independently in
``meshnet_tracker.recipe`` — it does not import this package. The two
The tracker re-derives the digests independently in ``meshnet_tracker.recipe`` —
it does not import this package, because an admission gate that shares an
implementation with the thing it admits is not an independent check. The two
implementations are pinned together by a committed conformance vector
(``tests/data/recipe_fingerprint_vectors.json``); if they ever drift, a test
fails rather than a route silently forming.
@@ -58,8 +59,7 @@ from __future__ import annotations
import hashlib
import json
import re
import time
from dataclasses import dataclass, field
from dataclasses import dataclass
from typing import Any, Iterable, Mapping, Sequence
from .native_protocol import BUNDLE_VERSION, SCHEMA_VERSION, pb
@@ -72,6 +72,7 @@ RECIPE_IDENTITY_SCHEMA_VERSION = 1
# with — a recipe digest, even if their canonical JSON were somehow identical.
ARTIFACT_DIGEST_DOMAIN = "meshnet.model-artifact.v1"
RECIPE_DIGEST_DOMAIN = "meshnet.runtime-recipe.v1"
SHARD_BINDING_DIGEST_DOMAIN = "meshnet.shard-binding.v1"
# The axes of a runtime recipe. Every one of these changes the numbers a Shard
# produces, so every one of them is part of identity and none of them may be
@@ -102,7 +103,8 @@ MISMATCH_RUNTIME = "runtime"
MISMATCH_BOUNDARY_SCHEMA = "boundary-schema"
MISMATCH_RANGE = "range"
MISMATCH_FINGERPRINT = "fingerprint"
MISMATCH_UNCERTIFIED = "uncertified"
MISMATCH_ROUTE_SESSION = "route-session"
MISMATCH_ROUTE_EPOCH = "route-epoch"
# Which fail-closed reason each axis reports. Several axes share a reason
# because they fail for the same operational cause: `activation_dtype` and
@@ -122,14 +124,6 @@ _AXIS_MISMATCH: Mapping[str, str] = {
"protocol_schema_version": MISMATCH_BOUNDARY_SCHEMA,
}
# Certification states. `dark` is the arrival state, not an error state.
STATUS_UNKNOWN = "unknown"
STATUS_DARK = "dark"
STATUS_CERTIFIED = "certified"
# A route that proves a recipe must be a real distributed route.
MIN_CERTIFYING_NODES = 2
_HEX64 = re.compile(r"^[0-9a-f]{64}$")
# A revision that can move is not a pin. DGR-017 learned this on the artifact;
@@ -141,10 +135,6 @@ class RecipeIdentityError(ValueError):
"""Malformed identity input. Messages name the field, never echo a payload."""
class RecipeNotCertified(RecipeIdentityError):
"""A recipe was asked to serve user traffic while still dark."""
class RouteIncompatible(RecipeIdentityError):
"""Two Shards cannot form an Inference Route.
@@ -618,6 +608,39 @@ class ShardIdentity:
catalogue_version=self.recipe.catalogue_version,
)
@property
def shard_binding_digest(self) -> str:
"""This Shard's own bytes and exact range, bound to the source artifact.
The fingerprint is range-independent on purpose — see the module
docstring — which means it cannot distinguish two *different splits of
the same source*. Both are the same recipe on the same model, and for
route compatibility that is exactly right. It is not right for
certification: a derivative could otherwise assert the correct source and
inherit a certification earned by bytes it does not hold.
So the derivative's own content hash and exact end-exclusive range get a
second digest, outside the fingerprint. The Tracker re-derives it at
admission and requires certification evidence to name the same binding
for every serving node.
"""
binding = self.artifact.derived_from
return _digest(
SHARD_BINDING_DIGEST_DOMAIN,
{
"source_digest": self.artifact.source_digest,
"content_digest": self.artifact.content_digest,
"architecture_digest": self.artifact.architecture_digest,
"derivative_range": (
None
if binding is None
else [binding.shard_start, binding.shard_end]
),
"shard_start": self.shard_start,
"shard_end": self.shard_end,
},
)
def to_dict(self) -> dict:
return {
"schema_version": RECIPE_IDENTITY_SCHEMA_VERSION,
@@ -747,57 +770,6 @@ def explain_mismatch(
return tuple(mismatches)
@dataclass(frozen=True)
class DistributedForwardEvidence:
"""Proof that a recipe really ran, distributed, end to end.
This is the only thing that takes a recipe out of the dark. The bar is
deliberately the product's bar, not a unit test's: at least two *distinct*
physical nodes, whose Shards cover the whole model with no hole, generating
real tokens. A synthetic worker is a unit fixture and certifies nothing —
the whole risk this guards against is a recipe that passes in a mock and
produces garbage on real weights.
"""
route_session_id: str
route_epoch: int
node_ids: tuple[str, ...]
shard_ranges: tuple[tuple[int, int], ...]
tokens_generated: int
layer_count: int
synthetic: bool = False
certified_at: float = field(default_factory=time.time)
def __post_init__(self) -> None:
_require_text(self.route_session_id, "evidence.route_session_id")
_require_int(self.route_epoch, "evidence.route_epoch", 0)
_require_int(self.tokens_generated, "evidence.tokens_generated", 0)
_require_int(self.layer_count, "evidence.layer_count", 1)
def rejection(self) -> str | None:
"""Why this evidence does not certify, or None when it does."""
if self.synthetic:
return (
"evidence comes from a synthetic worker; only a real distributed "
"forward certifies a recipe"
)
distinct = set(self.node_ids)
if len(distinct) < MIN_CERTIFYING_NODES:
return (
f"evidence covers {len(distinct)} distinct node(s); a distributed "
f"forward requires at least {MIN_CERTIFYING_NODES}"
)
if len(distinct) != len(self.node_ids):
return "the same node is counted more than once in the certifying route"
if self.tokens_generated < 1:
return "the certifying forward generated no tokens"
gap = _coverage_gap(self.shard_ranges, self.layer_count)
if gap is not None:
return gap
return None
def _coverage_gap(
ranges: Iterable[tuple[int, int]], layer_count: int
) -> str | None:
@@ -830,172 +802,20 @@ def _coverage_gap(
return None
@dataclass(frozen=True)
class RecipeStatus:
"""What the registry knows about one fingerprint."""
status: str
fingerprint: CompatibilityFingerprint | None = None
detail: str = ""
certified_at: float | None = None
@property
def may_serve(self) -> bool:
"""Only a certified recipe carries user traffic."""
return self.status == STATUS_CERTIFIED
@property
def may_certify(self) -> bool:
"""A dark recipe is eligible for a certification route — and only that.
Without this, certification is unreachable: serving requires
certification, certification requires a real distributed forward, and a
real distributed forward requires a route. A dark recipe may therefore
be routed *for the express purpose of certifying it*, and never for user
traffic.
"""
return self.status in (STATUS_DARK, STATUS_CERTIFIED)
def to_dict(self) -> dict:
return {
"status": self.status,
"detail": self.detail,
"certified_at": self.certified_at,
"fingerprint": (
self.fingerprint.to_dict() if self.fingerprint is not None else None
),
}
class RecipeRegistry:
"""Registered-but-dark recipes, and the ones a real forward has certified.
An unknown recipe is not routable. A known one is not routable either, until
it has been proven. This is the fail-closed default the roadmap asks for:
"unsupported architectures/backends remain registered-but-dark until real
certification passes."
"""
def __init__(self) -> None:
self._dark: dict[tuple[str, str], CompatibilityFingerprint] = {}
self._certified: dict[tuple[str, str], RecipeStatus] = {}
def register(self, identity: ShardIdentity | CompatibilityFingerprint) -> RecipeStatus:
"""Record a recipe as known. Known is not certified."""
fingerprint = _as_fingerprint(identity)
key = fingerprint.key
if key in self._certified:
return self._certified[key]
self._dark[key] = fingerprint
return RecipeStatus(
STATUS_DARK,
fingerprint,
"registered; dark until a real distributed forward certifies it",
)
def status(self, identity: ShardIdentity | CompatibilityFingerprint) -> RecipeStatus:
fingerprint = _as_fingerprint(identity)
key = fingerprint.key
if key in self._certified:
return self._certified[key]
if key in self._dark:
return RecipeStatus(
STATUS_DARK,
fingerprint,
"registered; dark until a real distributed forward certifies it",
)
return RecipeStatus(
STATUS_UNKNOWN,
fingerprint,
"this recipe has never been registered with the Tracker",
)
def certify(
self,
identity: ShardIdentity | CompatibilityFingerprint,
evidence: DistributedForwardEvidence,
) -> RecipeStatus:
"""Promote a dark recipe, if the evidence is a real distributed forward.
Raises rather than returning a failed status: certifying is a state
change, and a caller that ignores a returned failure would leave a
recipe it believes to be certified in the dark.
"""
fingerprint = _as_fingerprint(identity)
rejection = evidence.rejection()
if rejection is not None:
raise RecipeNotCertified(
f"this evidence does not certify {fingerprint.recipe_id!r}: {rejection}"
)
status = RecipeStatus(
STATUS_CERTIFIED,
fingerprint,
(
f"certified by route session {evidence.route_session_id} across "
f"{len(set(evidence.node_ids))} nodes"
),
certified_at=evidence.certified_at,
)
self._certified[fingerprint.key] = status
self._dark.pop(fingerprint.key, None)
return status
def require_serving(
self, identity: ShardIdentity | CompatibilityFingerprint
) -> RecipeStatus:
"""Admit a recipe for user traffic, or refuse and say why."""
status = self.status(identity)
if not status.may_serve:
raise RecipeNotCertified(
f"recipe {_as_fingerprint(identity).recipe_id!r} is {status.status}: "
f"{status.detail}"
)
return status
def certified_fingerprints(self) -> tuple[CompatibilityFingerprint, ...]:
return tuple(
status.fingerprint
for status in self._certified.values()
if status.fingerprint is not None
)
def to_dict(self) -> dict:
return {
"schema_version": RECIPE_IDENTITY_SCHEMA_VERSION,
"dark": [fp.to_dict() for fp in self._dark.values()],
"certified": [status.to_dict() for status in self._certified.values()],
}
def _as_fingerprint(
identity: ShardIdentity | CompatibilityFingerprint,
) -> CompatibilityFingerprint:
if isinstance(identity, CompatibilityFingerprint):
return identity
if isinstance(identity, ShardIdentity):
return identity.fingerprint
raise RecipeIdentityError(
f"expected a ShardIdentity or CompatibilityFingerprint, got "
f"{type(identity).__name__}"
)
def check_route(
shards: Sequence[ShardIdentity],
*,
registry: RecipeRegistry | None = None,
for_certification: bool = False,
) -> tuple[RouteMismatch, ...]:
"""Decide whether these Shards may form one Inference Route.
"""Why these Shards may not form one Inference Route; empty when they may.
Returns every reason they may not, empty when they may. Fail-closed by
construction: an empty route, an unknown recipe, a hole in the layer
coverage and a single differing dtype all produce a reason, and a caller that
only ever proceeds on an empty tuple cannot accidentally admit any of them.
This answers the *numerical* question only — do these Shards agree on every
axis that moves the numbers, and do they tile the model without a hole. It
deliberately does not answer "may this recipe carry user traffic": that is
certification, the Tracker owns it (`meshnet_tracker.recipe`), and a node
asking itself whether it is certified would be marking its own homework.
`for_certification` admits a dark recipe onto a route whose only purpose is
to certify it. It never admits an unknown one, and it is not the path user
traffic takes.
Fail-closed by construction: an empty route, a hole in the coverage and a
single differing dtype each produce a reason, so a caller that proceeds only
on an empty tuple cannot admit any of them.
"""
if not shards:
return (
@@ -1018,27 +838,12 @@ def check_route(
if gap is not None:
mismatches.append(RouteMismatch(MISMATCH_RANGE, gap))
if registry is not None:
status = registry.status(head.fingerprint)
allowed = status.may_certify if for_certification else status.may_serve
if not allowed:
mismatches.append(
RouteMismatch(MISMATCH_UNCERTIFIED, f"{status.status}: {status.detail}")
)
return tuple(mismatches)
def require_route(
shards: Sequence[ShardIdentity],
*,
registry: RecipeRegistry | None = None,
for_certification: bool = False,
) -> CompatibilityFingerprint:
def require_route(shards: Sequence[ShardIdentity]) -> CompatibilityFingerprint:
"""`check_route`, raising the structured reasons instead of returning them."""
mismatches = check_route(
shards, registry=registry, for_certification=for_certification
)
mismatches = check_route(shards)
if mismatches:
raise RouteIncompatible(mismatches)
return shards[0].fingerprint
@@ -1134,14 +939,111 @@ def check_handshake(
return tuple(reasons)
def check_session_open(
local: ShardIdentity,
remote: "pb.SessionOpen",
*,
expected_route_session_id: str | None = None,
expected_route_epoch: int | None = None,
) -> tuple[RouteMismatch, ...]:
"""Fail closed on the complete immutable portion of a ``SessionOpen``.
``SessionOpen`` is not just a fingerprint carrier: its route session and
epoch select the tracker-issued route whose identity was checked. A worker
that accepts a stale epoch can receive activations for a route that the
tracker has already replaced. Callers that have the tracker assignment
therefore pass both expected values; callers without it still reject the
protobuf defaults, which cannot identify a live route.
"""
reasons = list(check_handshake(local, remote.fingerprint))
if remote.schema_version != SCHEMA_VERSION:
reasons.append(
RouteMismatch(
MISMATCH_BOUNDARY_SCHEMA,
f"peer schema version {remote.schema_version} does not match "
f"{SCHEMA_VERSION}",
)
)
if not remote.route_session_id:
reasons.append(
RouteMismatch(
MISMATCH_ROUTE_SESSION,
"peer omitted the tracker-issued route session id",
)
)
elif (
expected_route_session_id is not None
and remote.route_session_id != expected_route_session_id
):
reasons.append(
RouteMismatch(
MISMATCH_ROUTE_SESSION,
"peer route session does not match the tracker assignment",
)
)
if remote.route_epoch < 1:
reasons.append(
RouteMismatch(
MISMATCH_ROUTE_EPOCH,
"peer omitted a positive tracker-issued route epoch",
)
)
elif expected_route_epoch is not None and remote.route_epoch != expected_route_epoch:
reasons.append(
RouteMismatch(
MISMATCH_ROUTE_EPOCH,
"peer route epoch does not match the tracker assignment",
)
)
advertised = (remote.shard_range.start_layer, remote.shard_range.end_layer)
expected = (local.shard_start, local.shard_end)
effective = remote.shard_range.effective_start_layer
if advertised != expected:
reasons.append(
RouteMismatch(
MISMATCH_RANGE,
f"peer advertised Shard range {advertised}, expected {expected}",
)
)
elif not local.shard_start <= effective < local.shard_end:
reasons.append(
RouteMismatch(
MISMATCH_RANGE,
f"effective start {effective} is outside Shard range {expected}",
)
)
return tuple(reasons)
def handshake_error(
mismatches: Sequence[RouteMismatch],
) -> "pb.ShardError | None":
"""The protocol status a rejected handshake closes the stream with."""
"""The protocol status a rejected handshake closes the stream with.
Each rejection maps to the most specific DGR-002 code the reasons allow:
a digest disagreement dominates (the peer is the wrong artifact or recipe,
whatever else is also wrong), then a schema disagreement (a peer on a
foreign schema cannot be trusted to have encoded anything else correctly),
and only a pure range disagreement reports the range code. Collapsing all
of these to one code would leave the initiator unable to tell "re-route me"
from "renegotiate the schema" from "wrong peer entirely".
"""
if not mismatches:
return None
reasons = {mismatch.reason for mismatch in mismatches}
if reasons == {MISMATCH_RANGE}:
code = pb.ERROR_CODE_SHARD_RANGE_MISMATCH
elif reasons <= {MISMATCH_ROUTE_SESSION, MISMATCH_ROUTE_EPOCH}:
code = pb.ERROR_CODE_EPOCH_STALE
elif reasons <= {MISMATCH_BOUNDARY_SCHEMA, MISMATCH_RANGE}:
code = pb.ERROR_CODE_SCHEMA_UNSUPPORTED
else:
code = pb.ERROR_CODE_FINGERPRINT_MISMATCH
return pb.ShardError(
code=pb.ERROR_CODE_FINGERPRINT_MISMATCH,
code=code,
detail="; ".join(m.describe() for m in mismatches),
retryable=False,
)

2
packages/node/meshnet_node/server.py
View File

@@ -105,7 +105,7 @@ class _StubHTTPServer(http.server.HTTPServer):
class _StubHandler(http.server.BaseHTTPRequestHandler):
def log_message(self, fmt, *args): # noqa: suppress request logs in tests
def log_message(self, fmt, *args): # suppress request logs in tests
pass
def do_POST(self):

5
packages/node/meshnet_node/torch_server.py
View File

@@ -19,7 +19,6 @@ from .model_backend import (
InsufficientVRAMError,
KVCacheMiss,
MissingModelDependencyError,
Quantization,
TailTokenResult,
TorchModelShard,
_tensor_from_bfloat16_bytes,
@@ -46,7 +45,7 @@ class _DirectRequestUncertainError(ConnectionError):
"""A direct request may have reached the downstream node but did not finish."""
from .server import (
from .server import ( # noqa: E402
_WIRE_VERSION,
_parse_shape,
_validate_activation_body,
@@ -399,7 +398,7 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
# Finite responses below provide Content-Length; streams are chunked.
protocol_version = "HTTP/1.1"
def log_message(self, fmt, *args): # noqa: suppress request logs in tests
def log_message(self, fmt, *args): # suppress request logs in tests
pass
def _request_id(self) -> str:

1
packages/node/meshnet_node/wizard.py
View File

@@ -2,7 +2,6 @@
from __future__ import annotations
import sys
import urllib.error
import urllib.request
from pathlib import Path

9
packages/node/native/README.md
View File

@@ -32,6 +32,15 @@ Both `--check` modes run in CI via `tests/test_native_shard_protocol.py`, so a
schema edit that is not accompanied by regenerated output fails the suite rather
than shipping stubs that disagree with the schema they claim to implement.
## DGR-006 decode and tail compatibility
`DecodeStep.bundle` is the versioned `TensorBundle` fast-path boundary. It is
authoritative whenever present and supports architecture sidebands. The original
`DecodeStep.tensor` remains readable as the compact one-tensor encoding for
certified boundaries that need only one tensor; new readers wrap it into a
one-member bundle. Tail completions use `TailResult`, which binds logits or a
sampled token to request/recipe identity and sampling/template/reasoning inputs.
## Building and running the C++ conformance test
If the machine has no protobuf C++ toolchain:

30
packages/node/native/llama/PATCH-STACK.md Normal file
View File

@@ -0,0 +1,30 @@
# Meshnet llama.cpp patch stack
This directory is the only project-owned fork boundary for llama.cpp. It is
locked to `e920c523e3b8a0163fe498af5bf90df35ff51d25`; changing the pin requires
updating the recorded tree/blob assumptions and reviewing every patch anew.
## Ordered series
1. `0001-cmake-reserve-meshnet-patch-stack-abi-marker.patch` adds only an
interface-library marker used to prove the patched source was configured.
It has no execution, transport, model-loading, or semantic effect.
Future patches may implement only the ADR-0020 local seams: range-aware tensor
loading, endpoint ownership, architecture-defined intermediate boundaries, and
layer-filtered KV/session mapping. Meshnet routing, Tracker, gRPC, relay,
billing, authentication, and telemetry must remain outside this directory.
`scripts/llama_cpp_dependency.py` verifies the exact commit/tree and baseline
blobs, validates every patch digest and context with `git apply --check`, then
applies the series in `patches/series` order. It refuses a dirty source tree,
wrong commit/tree/blob, changed patch digest, reordered series, or an existing
destination/work directory.
## Current semantic boundary
The stock pinned build is **infrastructure evidence only**. Per DGR-017,
GLM-5.2 can use a dense-MLA compatibility fallback at this point; a successful
build or `llama-cli --version` does not show native DSA/IndexShare, GLM semantic
acceptance, numerical parity, performance, or route certification. DGR-018 and
DGR-019 own those checks. Dense Llama remains only a later structural fixture.

17
packages/node/native/llama/THIRD_PARTY_NOTICES.md Normal file
View File

@@ -0,0 +1,17 @@
# Third-party notices: llama.cpp
The reproducibility harness fetches source from
[`ggml-org/llama.cpp`](https://github.com/ggml-org/llama.cpp) at exact commit
`e920c523e3b8a0163fe498af5bf90df35ff51d25`.
- Upstream license: MIT. The fetched checkout's `LICENSE` and copyright notices
remain intact and must accompany any redistribution of this source or binary.
- Meshnet's one-patch CMake marker is an additive local change. It does not
replace, relicense, or remove upstream notices.
- No donor code is included. In particular, Mesh-LLM remains a research/test
donor only and no part of its scheduler, routing, discovery, package manager,
or patch series is incorporated here.
Release packaging must include the upstream MIT text from the exact materialized
checkout plus this notice. `scripts/llama_cpp_dependency.py` refuses to build if
the upstream `LICENSE` is absent.

1
packages/node/native/llama/UPSTREAM_COMMIT Normal file
View File

@@ -0,0 +1 @@
e920c523e3b8a0163fe498af5bf90df35ff51d25

48
packages/node/native/llama/UPSTREAM_LOCK.json Normal file
View File

@@ -0,0 +1,48 @@
{
"schema_version": 1,
"upstream": "https://github.com/ggml-org/llama.cpp.git",
"commit": "e920c523e3b8a0163fe498af5bf90df35ff51d25",
"commit_tree": "6c91a11407a3a3fb160f5dac705f9c59718f54f1",
"patched_tree": "322d8b463df74a2226f0b513176643d815f54452",
"upstream_license": "MIT",
"patch_series": [
"0001-cmake-reserve-meshnet-patch-stack-abi-marker.patch",
"0002-dense-llama-owned-range-loader.patch"
],
"patch_scope": [
"Reserved CMake ABI marker only; no execution or model semantics.",
"Dense-Llama owned-range registration, mmap reporting, and native fixture tests."
],
"build": {
"generator": "Unix Makefiles",
"cmake_minimum": "3.14",
"cxx_standard": "17",
"configure_flags": [
"-DCMAKE_BUILD_TYPE=Release",
"-DLLAMA_BUILD_TESTS=OFF",
"-DLLAMA_BUILD_EXAMPLES=ON",
"-DLLAMA_BUILD_SERVER=OFF",
"-DLLAMA_BUILD_TOOLS=OFF",
"-DLLAMA_BUILD_APP=OFF",
"-DLLAMA_CURL=OFF"
],
"native_targets": ["llama-gguf-hash"],
"smoke_binary": "bin/llama-gguf-hash",
"smoke_args": ["--help"],
"smoke_output_token": "usage"
},
"required_upstream_blobs": {
"CMakeLists.txt": "81f23d7e70b7378511af5d01be680c03aebc2b15"
},
"patched_paths": [
"CMakeLists.txt",
"cmake/meshnet-patch-stack.cmake",
"include/llama.h",
"src/llama-model.cpp",
"src/llama-model.h",
"src/models/llama.cpp",
"tests/CMakeLists.txt",
"tests/test-meshnet-range-ownership.cpp"
],
"stock_glm_limitations": "This pin may load GLM-5.2 through the dense-MLA compatibility fallback. It does not prove native DSA, IndexShare, MoE semantic correctness, numerical equivalence, performance, or route certification."
}

68
packages/node/native/llama/meshnet-range-loader.cpp Normal file
View File

@@ -0,0 +1,68 @@
/**
* meshnet-range-loader — CLI wrapper that loads a GGUF shard range
* and prints the meshnet range report as JSON.
*
* Usage: meshnet-range-loader <gguf-path> <start-layer> <end-layer>
*
* Build: g++ -std=c++17 -I<source>/include -L<build>/bin
* meshnet-range-loader.cpp -lllama -o meshnet-range-loader
* LD_LIBRARY_PATH=<build>/bin ./meshnet-range-loader ...
*/
#include "llama.h"
#include <cstdio>
#include <cstdlib>
#include <cstring>
int main(int argc, char **argv) {
if (argc != 4) {
fprintf(stderr, "Usage: %s <gguf-path> <start-layer> <end-layer>\n", argv[0]);
return 1;
}
const char *path = argv[1];
int start = std::atoi(argv[2]);
int end = std::atoi(argv[3]);
llama_backend_init();
llama_model_params params = llama_model_default_params();
params.meshnet_owned_layer_start = start;
params.meshnet_owned_layer_end = end;
llama_model *model = llama_model_load_from_file(path, params);
if (!model) {
fprintf(stderr, "ERROR model_load returned nullptr\n");
llama_backend_free();
return 1;
}
llama_meshnet_range_report report;
bool ok = llama_model_meshnet_range_report(model, &report);
int n_layer = llama_model_n_layer(model);
int n_embd = llama_model_n_embd(model);
uint64_t size = llama_model_size(model);
if (ok) {
fprintf(stdout,
"{"
"\"ok\":true,"
"\"start_layer\":%d,\"end_layer\":%d,"
"\"mapped_bytes\":%lu,\"resident_bytes\":%lu,"
"\"model_n_layer\":%d,\"model_n_embd\":%d,\"model_size\":%lu"
"}\n",
report.start_layer, report.end_layer,
(unsigned long)report.mapped_bytes,
(unsigned long)report.resident_bytes,
n_layer, n_embd, (unsigned long)size
);
} else {
fprintf(stderr, "ERROR range report not available\n");
}
llama_model_free(model);
llama_backend_free();
return ok ? 0 : 1;
}

37
packages/node/native/llama/patches/0001-cmake-reserve-meshnet-patch-stack-abi-marker.patch Normal file
View File

@@ -0,0 +1,37 @@
From d9992f62e852c8647a2ad302009b0339dc1cbf5b Mon Sep 17 00:00:00 2001
From: DGR-004 <dgr-004@meshnet.invalid>
Date: Tue, 14 Jul 2026 09:52:24 +0300
Subject: [PATCH] cmake: reserve Meshnet patch-stack ABI marker
---
CMakeLists.txt | 1 +
cmake/meshnet-patch-stack.cmake | 7 +++++++
2 files changed, 8 insertions(+)
diff --git a/CMakeLists.txt b/CMakeLists.txt
index 81f23d7e..a9afcffa 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -123,6 +123,7 @@ option(LLAMA_LLGUIDANCE "llama-common: include LLGuidance library for structured
# Required for relocatable CMake package
include(${CMAKE_CURRENT_SOURCE_DIR}/cmake/build-info.cmake)
include(${CMAKE_CURRENT_SOURCE_DIR}/cmake/common.cmake)
+include(${CMAKE_CURRENT_SOURCE_DIR}/cmake/meshnet-patch-stack.cmake)
if (NOT DEFINED LLAMA_BUILD_NUMBER)
set(LLAMA_BUILD_NUMBER ${BUILD_NUMBER})
diff --git a/cmake/meshnet-patch-stack.cmake b/cmake/meshnet-patch-stack.cmake
new file mode 100644
index 00000000..910646b4
--- /dev/null
+++ b/cmake/meshnet-patch-stack.cmake
@@ -0,0 +1,7 @@
+# Reserved, local-only hook for the Meshnet-owned llama.cpp patch series.
+# No execution, protocol, architecture, or model semantic changes are in DGR-004.
+set(LLAMA_MESHNET_PATCH_STACK_VERSION "1" CACHE STRING
+ "Meshnet patch stack ABI marker")
+add_library(llama_meshnet_patch_stack INTERFACE)
+target_compile_definitions(llama_meshnet_patch_stack INTERFACE
+ LLAMA_MESHNET_PATCH_STACK_VERSION=${LLAMA_MESHNET_PATCH_STACK_VERSION})
--
2.54.0

169
packages/node/native/llama/patches/0002-dense-llama-owned-range-loader.patch Normal file
View File

@@ -0,0 +1,169 @@
From: Meshnet <meshnet@invalid>
Subject: [PATCH] llama: add dense owned-range loading seam
diff --git a/include/llama.h b/include/llama.h
index a311ac20..1f9459cf 100644
--- a/include/llama.h
+++ b/include/llama.h
@@ -292,6 +292,19 @@ extern "C" {
ggml_backend_buffer_type_t buft;
};
+ // Immutable report for the project-owned dense-Llama range-loading seam.
+ // The bounds are inclusive/exclusive and are populated only after the
+ // model has registered and allocated its owned tensors.
+ struct llama_meshnet_range_report {
+ int32_t start_layer;
+ int32_t end_layer;
+ uint64_t mapped_bytes;
+ uint64_t resident_bytes;
+ uint64_t registered_bytes;
+ bool has_token_embeddings;
+ bool has_output_head;
+ };
+
struct llama_model_params {
@@ -319,6 +332,12 @@ extern "C" {
const struct llama_model_kv_override * kv_overrides;
+ int32_t meshnet_owned_layer_start;
+ int32_t meshnet_owned_layer_end;
+
// Keep the booleans together to avoid misalignment during copy-by-value.
@@ -616,6 +635,13 @@ extern "C" {
LLAMA_API uint64_t llama_model_size(const struct llama_model * model);
+ LLAMA_API bool llama_model_meshnet_range_report(
+ const struct llama_model * model,
+ struct llama_meshnet_range_report * out);
+
// Get the default chat template. Returns nullptr if not available
diff --git a/src/llama-model.cpp b/src/llama-model.cpp
index d8748138..4d2a3ec1 100644
--- a/src/llama-model.cpp
+++ b/src/llama-model.cpp
@@ -1015,6 +1015,9 @@ struct llama_model::impl {
std::vector<float> tensor_split_owned;
+ llama_meshnet_range_report meshnet_range_report = {};
+ bool has_meshnet_range_report = false;
};
@@ -1236,6 +1239,19 @@ bool llama_model_base::load_tensors(llama_model_loader & ml) {
const bool use_mmap_buffer = true;
+ const bool meshnet_range_requested = params.meshnet_owned_layer_start != 0 || params.meshnet_owned_layer_end != 0;
+ const int meshnet_start = params.meshnet_owned_layer_start;
+ const int meshnet_end = params.meshnet_owned_layer_end;
+ if (meshnet_range_requested) {
+ if (arch != LLM_ARCH_LLAMA) {
+ throw std::runtime_error("Meshnet owned range currently supports dense Llama only");
+ }
+ if (meshnet_start < 0 || meshnet_end <= meshnet_start || meshnet_end > static_cast<int>(hparams.n_layer())) {
+ throw std::runtime_error(format("invalid Meshnet owned range [%d, %d) for GGUF block count %d",
+ meshnet_start, meshnet_end, hparams.n_layer()));
+ }
+ }
@@ -1336,7 +1352,9 @@ bool llama_model_base::load_tensors(llama_model_loader & ml) {
- for (int i = 0; i < n_layer_all; ++i) {
+ const int optional_scale_start = meshnet_range_requested ? meshnet_start : 0;
+ const int optional_scale_end = meshnet_range_requested ? meshnet_end : n_layer_all;
+ for (int i = optional_scale_start; i < optional_scale_end; ++i) {
@@ -1487,7 +1505,7 @@ bool llama_model_base::load_tensors(llama_model_loader & ml) {
- ml.done_getting_tensors();
+ ml.done_getting_tensors(meshnet_range_requested);
@@ -1613,8 +1631,11 @@ bool llama_model_base::load_tensors(llama_model_loader & ml) {
+ uint64_t meshnet_mapped_bytes = 0;
+ uint64_t meshnet_resident_bytes = 0;
for (auto & [_, bufs] : pimpl->ctxs_bufs) {
for (auto & buf: bufs) {
+ meshnet_resident_bytes += ggml_backend_buffer_get_size(buf.get());
@@ -1637,6 +1658,35 @@ bool llama_model_base::load_tensors(llama_model_loader & ml) {
}
+ if (meshnet_range_requested) {
+ uint64_t registered_bytes = 0;
+ for (const auto & [_, tensor] : tensors_by_name) registered_bytes += ggml_nbytes(tensor);
+ if (ml.use_mmap) for (const auto & [first, last] : ml.mmaps_used) if (last > first) meshnet_mapped_bytes += last - first;
+ const auto registered = [this](const ggml_tensor * tensor) {
+ return tensor != nullptr && std::any_of(tensors_by_name.begin(), tensors_by_name.end(),
+ [tensor](const auto & entry) { return entry.second == tensor; });
+ };
+ const auto registered_name = [this](const char * name) {
+ return std::any_of(tensors_by_name.begin(), tensors_by_name.end(),
+ [name](const auto & entry) { return entry.first == name; });
+ };
+ pimpl->meshnet_range_report = { meshnet_start, meshnet_end, meshnet_mapped_bytes, meshnet_resident_bytes,
+ registered_bytes, registered_name("token_embd.weight"), registered(output_norm) && registered(output) };
+ pimpl->has_meshnet_range_report = true;
+ }
return true;
@@ -1711,6 +1761,14 @@ uint64_t llama_model::n_elements() const {
}
+bool llama_model::meshnet_range_report(llama_meshnet_range_report * out) const {
+ if (out == nullptr || !pimpl->has_meshnet_range_report) return false;
+ *out = pimpl->meshnet_range_report;
+ return true;
+}
@@ -2308,6 +2366,8 @@ llama_model_params llama_model_default_params() {
/*.kv_overrides =*/ nullptr,
+ /*.meshnet_owned_layer_start =*/ 0,
+ /*.meshnet_owned_layer_end =*/ 0,
@@ -2641,6 +2701,10 @@ uint64_t llama_model_size(const llama_model * model) {
}
+bool llama_model_meshnet_range_report(const llama_model * model, llama_meshnet_range_report * out) {
+ return model != nullptr && model->meshnet_range_report(out);
+}
diff --git a/src/llama-model.h b/src/llama-model.h
index 45b054ce..1b3f9bd0 100644
--- a/src/llama-model.h
+++ b/src/llama-model.h
@@ -652,6 +652,8 @@ struct llama_model {
+ bool meshnet_range_report(llama_meshnet_range_report * out) const;
+
diff --git a/src/models/llama.cpp b/src/models/llama.cpp
index 4bfebc88..b4f25aed 100644
--- a/src/models/llama.cpp
+++ b/src/models/llama.cpp
@@ -34,18 +34,26 @@ void llama_model_llama::load_arch_hparams(llama_model_loader & ml) {
- tok_embd = create_tensor(tn(LLM_TENSOR_TOKEN_EMBD, "weight"), {n_embd, n_vocab}, 0);
+ const bool meshnet_range_requested = params.meshnet_owned_layer_start != 0 || params.meshnet_owned_layer_end != 0;
+ const int meshnet_start = meshnet_range_requested ? params.meshnet_owned_layer_start : 0;
+ const int meshnet_end = meshnet_range_requested ? params.meshnet_owned_layer_end : n_layer;
- // output
- output_norm = create_tensor(tn(LLM_TENSOR_OUTPUT_NORM, "weight"), {n_embd}, 0);
- output = create_tensor(tn(LLM_TENSOR_OUTPUT, "weight"), {n_embd, n_vocab}, TENSOR_NOT_REQUIRED);
+ if (!meshnet_range_requested || meshnet_start == 0) tok_embd = create_tensor(tn(LLM_TENSOR_TOKEN_EMBD, "weight"), {n_embd, n_vocab}, 0);
+ if (!meshnet_range_requested || meshnet_end == n_layer) {
+ output_norm = create_tensor(tn(LLM_TENSOR_OUTPUT_NORM, "weight"), {n_embd}, 0);
+ output = create_tensor(tn(LLM_TENSOR_OUTPUT, "weight"), {n_embd, n_vocab}, TENSOR_NOT_REQUIRED);
- // if output is NULL, init from the input tok embed
- if (output == NULL) {
- output = create_tensor(tn(LLM_TENSOR_TOKEN_EMBD, "weight"), {n_embd, n_vocab}, TENSOR_DUPLICATED);
+ if (output == NULL) output = create_tensor(tn(LLM_TENSOR_TOKEN_EMBD, "weight"), {n_embd, n_vocab}, TENSOR_DUPLICATED);
}
- for (int i = 0; i < n_layer; ++i) {
+ for (int i = meshnet_start; i < meshnet_end; ++i) {
@@ -102,6 +110,25 @@ llama_model_llama::graph<embed>::graph(const llama_model & model, const llm_grap
+ llama_meshnet_range_report meshnet_report = {};
+ if (model.meshnet_range_report(&meshnet_report)) {
+ if (meshnet_report.start_layer != 0) throw std::runtime_error("Meshnet dense-Llama graph requires a head endpoint adapter");
+ if (meshnet_report.end_layer != n_layer) throw std::runtime_error("Meshnet dense-Llama graph requires a tail endpoint adapter");
+ if (!meshnet_report.has_token_embeddings) throw std::runtime_error("Meshnet dense-Llama head range is missing token embeddings");
+ if (!meshnet_report.has_output_head) throw std::runtime_error("Meshnet dense-Llama tail range is missing final norm or output head");
+ }
ggml_tensor * cur;
diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt
index 855295c1..9a7be6ee 100644
--- a/tests/CMakeLists.txt
+++ b/tests/CMakeLists.txt
@@ -193,6 +193,7 @@ if (NOT WIN32 OR NOT BUILD_SHARED_LIBS)
+ llama_build_and_test(test-meshnet-range-ownership.cpp)
diff --git a/tests/test-meshnet-range-ownership.cpp b/tests/test-meshnet-range-ownership.cpp
new file mode 100644
index 00000000..7b58ebf8
--- /dev/null
+++ b/tests/test-meshnet-range-ownership.cpp
@@ -0,0 +1,6 @@
+#include "ggml.h"
+#include "gguf.h"
+#include "llama.h"
+#include "../src/llama-model.h"
+#include <cstdio>
+#include <cstring>

3
packages/node/native/llama/patches/SHA256SUMS Normal file
View File

@@ -0,0 +1,3 @@
# SHA-256 digests for the ordered patch series. Do not reorder this file.
1454216c019c1cb7f78d1d836fe4054164fff1d498391013bcaf13cc2d328c75 0001-cmake-reserve-meshnet-patch-stack-abi-marker.patch
51c205e3ca26e104f80c838eeeb11115b8d436036014116d2bb407178c30e0bd 0002-dense-llama-owned-range-loader.patch

2
packages/node/native/llama/patches/series Normal file
View File

@@ -0,0 +1,2 @@
0001-cmake-reserve-meshnet-patch-stack-abi-marker.patch
0002-dense-llama-owned-range-loader.patch

64
packages/node/native/proto/shard_runtime.proto
View File

@@ -33,6 +33,16 @@ enum SchemaVersion {
SCHEMA_VERSION_1 = 1;
}
// Certified transformer boundary family. Names alone are not an adapter: a
// Node must select an explicit architecture contract before interpreting a
// TensorBundle.
enum ArchitectureType {
ARCHITECTURE_TYPE_UNSPECIFIED = 0;
ARCHITECTURE_TYPE_DENSE = 1;
ARCHITECTURE_TYPE_MOE = 2;
ARCHITECTURE_TYPE_MLA = 3;
}
// ---------------------------------------------------------------------------
// Tensor bundle — the public activation boundary
// ---------------------------------------------------------------------------
@@ -142,6 +152,11 @@ message TensorBundle {
// boundary payload can evolve without a whole-protocol version bump.
uint32 bundle_version = 1;
repeated NamedTensor tensors = 2;
// Explicit adapter selection; names are never interpreted through unchecked
// substitutions. UNSPECIFIED is accepted only for legacy DGR-002 bundles.
ArchitectureType architecture = 3;
// Adapter-owned semantic boundary, such as "pre_tail_residual".
string boundary_point = 4;
}
// ---------------------------------------------------------------------------
@@ -393,8 +408,9 @@ message ActivationChunk {
// A decode step is one token: the envelope's identity fields are already fixed
// for the life of the stream, so repeating them per token is pure overhead on
// the hottest path. A DecodeStep carries only what changes — the step, the
// position, and one tensor — and inherits the rest from the SessionOpen
// handshake. A peer may always fall back to ActivationChunk with PHASE_DECODE.
// position, and one compact boundary encoding — and inherits the rest from the
// SessionOpen handshake. A Node may always fall back to ActivationChunk with
// PHASE_DECODE.
message DecodeStep {
// Idempotency step within the session; also orders the stream.
uint64 idempotency_step = 1;
@@ -402,11 +418,50 @@ message DecodeStep {
uint64 position = 2;
// Tokens the receiver's cache must already hold. A mismatch is a CACHE_MISS.
uint64 expected_past_len = 3;
// The single boundary tensor for this token, typically [1, 1, hidden].
// Legacy compact one-tensor boundary, typically [1, 1, hidden]. New readers
// accept it as a TensorBundle with one member; writers retain it where the
// selected architecture genuinely requires only one tensor.
NamedTensor tensor = 4;
// Work id for attribution; the session/epoch/fingerprint/range are inherited.
string work_id = 5;
int64 deadline_unix_nanos = 6;
// Versioned architecture boundary. This carries sidebands such as MoE router
// state or MLA/IndexShare state; when both fields are set, bundle is
// authoritative and tensor is retained only for older-Node forwarding.
TensorBundle bundle = 7;
}
// Exact request-level identity for a tail result. Runtime recipe identity is
// deliberately repeated here: sampling the right logits with the wrong chat
// template or reasoning mode is a different request, not a compatible result.
message RequestRecipeIdentity {
string request_id = 1;
string runtime_recipe_digest = 2;
string chat_template_id = 3;
string chat_template_version = 4;
string reasoning_mode = 5;
ArchitectureType architecture = 6;
}
// Sampling is an explicit tail-only operation. A non-tail Shard never applies
// final norm, LM head, row pruning, or sampling to its boundary output.
message SamplingParameters {
float temperature = 1;
float top_p = 2;
uint32 top_k = 3;
uint64 seed = 4;
bool greedy = 5;
}
// Typed tail completion. The oneof keeps token id 0 unambiguous and prevents a
// caller from treating an activation tensor as an inferred completion.
message TailResult {
RequestRecipeIdentity identity = 1;
SamplingParameters sampling = 2;
oneof output {
TensorBundle logits = 3;
uint32 sampled_token_id = 4;
}
}
// Drop session state for a Route Session. Bounded memory does not depend on
@@ -471,7 +526,8 @@ message SessionResponse {
ActivationChunk chunk = 2;
Ack ack = 3;
FlowControl flow_control = 4;
ShardStatus status = 5;
ShardStatus status = 5;
TailResult tail_result = 6;
}
}

BIN
packages/node/native/testdata/decode_step_golden.binpb vendored Normal file
View File

Binary file not shown.

28
packages/node/native/tests/test_shard_protocol_conformance.cpp
View File

@@ -265,7 +265,9 @@ void TestDecodeFastPathIsSmall() {
step->set_position(1024);
step->set_expected_past_len(1024);
step->set_work_id("work-7f3a");
sp::NamedTensor *tensor = step->mutable_tensor();
sp::TensorBundle *bundle = step->mutable_bundle();
bundle->set_bundle_version(1);
sp::NamedTensor *tensor = bundle->add_tensors();
tensor->set_name("hidden_states");
tensor->add_shape(1);
tensor->add_shape(1);
@@ -290,7 +292,27 @@ void TestDecodeFastPathIsSmall() {
CHECK(parsed.ParseFromString(bytes));
CHECK(parsed.kind_case() == sp::SessionRequest::kDecode);
CHECK_EQ(parsed.decode().position(), 1024u);
CHECK_EQ(parsed.decode().tensor().total_bytes(), 16u);
CHECK_EQ(parsed.decode().bundle().tensors_size(), 1);
CHECK_EQ(parsed.decode().bundle().tensors(0).total_bytes(), 16u);
}
void TestDecodeBundleVector(const std::string &bytes) {
sp::SessionRequest request;
CHECK(request.ParseFromString(bytes));
CHECK(request.kind_case() == sp::SessionRequest::kDecode);
const sp::DecodeStep &step = request.decode();
CHECK_EQ(step.idempotency_step(), 43u);
CHECK_EQ(step.position(), 384u);
CHECK_EQ(step.expected_past_len(), 384u);
CHECK_EQ(step.work_id(), std::string("decode-7f3a"));
CHECK(step.has_bundle());
CHECK_EQ(step.bundle().bundle_version(), 1u);
CHECK_EQ(step.bundle().architecture(), sp::ARCHITECTURE_TYPE_MLA);
CHECK_EQ(step.bundle().boundary_point(), std::string("pre_tail_residual"));
CHECK_EQ(step.bundle().tensors_size(), 2);
CHECK_EQ(step.bundle().tensors(0).name(), std::string("hidden_states"));
CHECK_EQ(step.bundle().tensors(1).name(), std::string("index_topk"));
CHECK_EQ(step.bundle().tensors(1).dtype(), sp::DTYPE_INT32);
}
} // namespace
@@ -308,9 +330,11 @@ int main(int argc, char **argv) {
ReadFile(testdata / "session_request_golden.binpb");
const std::string capability_bytes =
ReadFile(testdata / "capability_report_golden.binpb");
const std::string decode_bytes = ReadFile(testdata / "decode_step_golden.binpb");
TestSessionRequestVector(session_bytes);
TestCapabilityReportVector(capability_bytes);
TestDecodeBundleVector(decode_bytes);
TestUnknownFieldsArePreserved(session_bytes);
TestSparseMessageParses();
TestDecodeFastPathIsSmall();

1
packages/p2p/meshnet_p2p/mdns.py
View File

@@ -7,7 +7,6 @@ from __future__ import annotations
import logging
import socket
import threading
from typing import Callable
log = logging.getLogger(__name__)

2
packages/p2p/meshnet_p2p/tls.py
View File

@@ -3,9 +3,7 @@
from __future__ import annotations
import datetime
import hashlib
import ipaddress
import json
import os
import socket
import ssl

1
packages/relay/meshnet_relay/cli.py
View File

@@ -4,7 +4,6 @@ from __future__ import annotations
import argparse
import logging
import sys
import time
from pathlib import Path

38
packages/tracker/meshnet_tracker/capability.py
View File

@@ -185,6 +185,10 @@ class CapabilityState:
# Absent for a node that predates DGR-003.
model_artifact_digest: str | None = None
runtime_recipe_digest: str | None = None
shard_binding_digest: str | None = None
# The tracker ledger's verdict on that fingerprint at evaluation time
# ("dark"/"certified"), so the network map answers "why is this exact node
# not routing" without a second query. None when no identity was presented.
certification: str | None = None
@property
@@ -227,6 +231,7 @@ class CapabilityState:
"diagnostics": list(self.diagnostics),
"model_artifact_digest": self.model_artifact_digest,
"runtime_recipe_digest": self.runtime_recipe_digest,
"shard_binding_digest": self.shard_binding_digest,
"certification": self.certification,
}
@@ -372,6 +377,22 @@ def evaluate_report(
f"identity is for artifact {identity.artifact_id!r}, but the node "
f"registered {advertised_model!r}",
)
model_claim = report["model"]
if model_claim.get("revision") != identity.revision:
return base.with_state(
STATE_MODEL_MISMATCH,
"identity revision does not match the capability proof",
)
config_fingerprint = model_claim.get("config_fingerprint")
if isinstance(config_fingerprint, str) and config_fingerprint.startswith(
"sha256:"
):
config_fingerprint = config_fingerprint.removeprefix("sha256:")
if config_fingerprint != identity.architecture_digest:
return base.with_state(
STATE_MODEL_MISMATCH,
"identity architecture/config digest does not match the capability proof",
)
if (
identity.recipe_id != base.recipe_id
or identity.recipe_version != base.recipe_version
@@ -399,6 +420,7 @@ def evaluate_report(
base,
model_artifact_digest=identity.model_artifact_digest,
runtime_recipe_digest=identity.runtime_recipe_digest,
shard_binding_digest=identity.shard_binding_digest,
)
if status != STATUS_PASSED:
@@ -425,8 +447,22 @@ def evaluate_report(
# not authenticate a node or prove a distributed forward. Exact recipes are
# therefore registered-but-dark until tracker-owned certification records
# that forward.
#
# `ledger is None` means the caller owns no certification authority. It is
# not "certify anything" and it is not "make one up": a disposable ledger
# would register the recipe into state that is discarded on return, which
# reads like certification is wired when nothing is recording it. The only
# safe reading is that nothing here has certified this recipe, so it stays
# dark. `TrackerServer` owns the real ledger and passes it in.
if identity is not None:
recipe_status = (ledger or CertificationLedger()).register(identity)
if ledger is None:
return base.with_state(
STATE_UNCERTIFIED,
"no certification ledger; an exact recipe is dark until a "
"tracker-owned distributed forward certifies it",
)
recipe_status = ledger.register(identity)
base = replace(base, certification=recipe_status.status)
if not recipe_status.may_serve:
return base.with_state(STATE_UNCERTIFIED, recipe_status.detail)

4
packages/tracker/meshnet_tracker/raft.py
View File

@@ -16,8 +16,8 @@ import threading
import time
import urllib.error
import urllib.request
from dataclasses import dataclass, field
from typing import Any, Callable
from dataclasses import dataclass
from typing import Callable
@dataclass

181
packages/tracker/meshnet_tracker/recipe.py
View File

@@ -32,7 +32,7 @@ import json
import re
import time
from dataclasses import dataclass, field
from typing import Any, Iterable, Mapping, Sequence
from typing import Any, Iterable, Mapping
# Layout of the identity block this tracker reads (meshnet_node.runtime_recipe).
RECIPE_IDENTITY_SCHEMA_VERSION = 1
@@ -42,6 +42,7 @@ RECIPE_IDENTITY_SCHEMA_VERSION = 1
# without changing it there silently partitions every route.
ARTIFACT_DIGEST_DOMAIN = "meshnet.model-artifact.v1"
RECIPE_DIGEST_DOMAIN = "meshnet.runtime-recipe.v1"
SHARD_BINDING_DIGEST_DOMAIN = "meshnet.shard-binding.v1"
# The axes a recipe digest commits to. Order is irrelevant (the canonical JSON
# sorts keys); membership is not — an axis missing here is an axis the tracker
@@ -62,16 +63,6 @@ RECIPE_AXES: tuple[str, ...] = (
_INT_AXES = frozenset({"boundary_schema_version", "protocol_schema_version"})
MISMATCH_ARTIFACT = "artifact"
MISMATCH_TOKENIZER = "tokenizer"
MISMATCH_ARCHITECTURE = "architecture"
MISMATCH_RANGE = "range"
MISMATCH_BOUNDARY_SCHEMA = "boundary-schema"
MISMATCH_ACTIVATION_RECIPE = "activation-recipe"
MISMATCH_CACHE_LAYOUT = "cache-layout"
MISMATCH_FINGERPRINT = "fingerprint"
MISMATCH_UNCERTIFIED = "uncertified"
STATUS_UNKNOWN = "unknown"
STATUS_DARK = "dark"
STATUS_CERTIFIED = "certified"
@@ -183,6 +174,8 @@ class PresentedIdentity:
artifact_id: str
revision: str
source_digest: str
content_digest: str
derivative_range: tuple[int, int] | None
architecture: str
architecture_digest: str
layer_count: int
@@ -211,6 +204,36 @@ class PresentedIdentity:
def key(self) -> tuple[str, str]:
return (self.model_artifact_digest, self.runtime_recipe_digest)
@property
def shard_binding_digest(self) -> str:
"""This participant's own bytes and exact range, bound to the source.
The route fingerprint (:attr:`key`) is deliberately range-independent —
Shards on one route own different ranges, so a range-sensitive digest
would stop any two of them from ever agreeing. The consequence is that
the fingerprint alone cannot tell two *different splits of the same
source* apart: a derivative can assert the right source and recipe and
inherit certification earned by bytes it does not hold.
This digest is what closes that. It commits to the derivative's own
content hash and its exact end-exclusive range, so certification can be
recipe-wide while every serving node is still separately pinned to the
blob it was admitted on (`TrackerServer.certify_recipe`).
"""
return _digest(
SHARD_BINDING_DIGEST_DOMAIN,
{
"source_digest": self.source_digest,
"content_digest": self.content_digest,
"architecture_digest": self.architecture_digest,
"derivative_range": list(self.derivative_range)
if self.derivative_range is not None
else None,
"shard_start": self.shard_start,
"shard_end": self.shard_end,
},
)
@property
def tokenizer_revision(self) -> str:
return str(self.axes["tokenizer_revision"])
@@ -315,6 +338,8 @@ def parse_identity(data: Any) -> PresentedIdentity:
artifact_id=_text(artifact.get("artifact_id"), "artifact.artifact_id"),
revision=_pin(artifact.get("revision"), "artifact.revision"),
source_digest=source_digest,
content_digest=content_digest,
derivative_range=binding,
architecture=_text(artifact.get("architecture"), "artifact.architecture"),
architecture_digest=_hex64(
artifact.get("architecture_digest"), "artifact.architecture_digest"
@@ -349,78 +374,6 @@ def parse_identity(data: Any) -> PresentedIdentity:
return identity
@dataclass(frozen=True)
class RouteMismatch:
reason: str
detail: str
def describe(self) -> str:
return f"{self.reason}: {self.detail}"
def compare(
route: PresentedIdentity, candidate: PresentedIdentity
) -> tuple[RouteMismatch, ...]:
"""Why `candidate` may not join a route already running `route`."""
if route.key == candidate.key:
return ()
reasons: list[RouteMismatch] = []
if route.source_digest != candidate.source_digest:
reasons.append(
RouteMismatch(
MISMATCH_ARTIFACT,
f"serves Model Artifact {candidate.source_digest[:12]}…, the route "
f"runs {route.source_digest[:12]}…",
)
)
if route.architecture_digest != candidate.architecture_digest:
reasons.append(
RouteMismatch(
MISMATCH_ARCHITECTURE,
"architecture/config snapshot differs from the route's",
)
)
if route.layer_count != candidate.layer_count:
reasons.append(
RouteMismatch(
MISMATCH_ARTIFACT,
f"artifact has {candidate.layer_count} layers, the route's has "
f"{route.layer_count}",
)
)
axis_reason = {
"tokenizer_revision": MISMATCH_TOKENIZER,
"architecture_adapter": MISMATCH_ARCHITECTURE,
"activation_dtype": MISMATCH_ACTIVATION_RECIPE,
"compute_dtype": MISMATCH_ACTIVATION_RECIPE,
"kv_dtype": MISMATCH_CACHE_LAYOUT,
"kv_layout": MISMATCH_CACHE_LAYOUT,
"boundary_schema_version": MISMATCH_BOUNDARY_SCHEMA,
"protocol_schema_version": MISMATCH_BOUNDARY_SCHEMA,
}
for axis in RECIPE_AXES:
mine = route.axes.get(axis)
theirs = candidate.axes.get(axis)
if mine != theirs:
reasons.append(
RouteMismatch(
axis_reason.get(axis, MISMATCH_FINGERPRINT),
f"{axis} is {theirs!r}, the route runs {mine!r}",
)
)
if not reasons:
reasons.append(
RouteMismatch(
MISMATCH_FINGERPRINT,
"fingerprints differ on a field this comparison does not cover",
)
)
return tuple(reasons)
def coverage_gap(
ranges: Iterable[tuple[int, int]], layer_count: int
) -> str | None:
@@ -462,6 +415,7 @@ class DistributedForwardEvidence:
tokens_generated: int
layer_count: int
fingerprint: tuple[str, str]
participants: tuple[PresentedIdentity, ...]
synthetic: bool = False
certified_at: float = field(default_factory=time.time)
@@ -471,6 +425,19 @@ class DistributedForwardEvidence:
"evidence comes from a synthetic worker; only a real distributed "
"forward certifies a recipe"
)
if len(self.participants) != len(self.node_ids):
return "participant identities do not match the certifying node list"
if len(self.shard_ranges) != len(self.participants):
return "participant identities do not match the recorded effective ranges"
for participant, shard_range in zip(
self.participants, self.shard_ranges, strict=True
):
if participant.key != self.fingerprint:
return "a participant fingerprint differs from the certifying route"
if participant.layer_count != self.layer_count:
return "a participant artifact layer count differs from the certifying route"
if (participant.shard_start, participant.shard_end) != shard_range:
return "a participant identity does not match its recorded effective range"
distinct = set(self.node_ids)
if len(distinct) < MIN_CERTIFYING_NODES:
return (
@@ -540,11 +507,11 @@ class CertificationLedger:
def certify(
self,
identity: PresentedIdentity | tuple[str, str],
identity: PresentedIdentity,
evidence: DistributedForwardEvidence,
) -> RecipeStatus:
"""Promote a recipe out of the dark, or raise saying why the evidence is short."""
key = identity if isinstance(identity, tuple) else identity.key
key = identity.key
if key not in self._dark and key not in self._certified:
raise RecipeIdentityError(
"this fingerprint is not registered; unknown recipes cannot be certified"
@@ -553,6 +520,10 @@ class CertificationLedger:
raise RecipeIdentityError(
"certification evidence fingerprint does not match the recipe being promoted"
)
if evidence.layer_count != identity.layer_count:
raise RecipeIdentityError(
"certification evidence layer count does not match the artifact being promoted"
)
rejection = evidence.rejection()
if rejection is not None:
raise RecipeIdentityError(f"this evidence does not certify: {rejection}")
@@ -578,38 +549,8 @@ class CertificationLedger:
}
def admit_route(
identities: Sequence[PresentedIdentity],
*,
ledger: CertificationLedger | None = None,
for_certification: bool = False,
) -> tuple[RouteMismatch, ...]:
"""Every reason these Shards may not form one Inference Route; empty when they may.
Fail-closed by construction: an empty route, a hole in the coverage, one
differing dtype, and an uncertified recipe each produce a reason, so a caller
that proceeds only on an empty result cannot admit any of them.
"""
if not identities:
return (RouteMismatch(MISMATCH_RANGE, "a route needs at least one Shard"),)
head = identities[0]
reasons: list[RouteMismatch] = []
for candidate in identities[1:]:
reasons.extend(compare(head, candidate))
gap = coverage_gap(
[(i.shard_start, i.shard_end) for i in identities], head.layer_count
)
if gap is not None:
reasons.append(RouteMismatch(MISMATCH_RANGE, gap))
if ledger is not None:
status = ledger.status(head)
allowed = status.may_certify if for_certification else status.may_serve
if not allowed:
reasons.append(
RouteMismatch(MISMATCH_UNCERTIFIED, f"{status.status}: {status.detail}")
)
return tuple(reasons)
# Route formation itself lives in `server.py` (`_select_route`, `_enumerate_routes`,
# `_find_pinned_route`): candidates are partitioned by the exact fingerprint this
# module derives, so a route can only ever be assembled inside one identity. A
# second route gate here would be a second copy of that policy to keep in step —
# the same reason the node module holds no certification authority.

2
packages/tracker/meshnet_tracker/routing_stats.py
View File

@@ -26,7 +26,7 @@ import random
import sqlite3
import threading
import time
from dataclasses import dataclass, field
from dataclasses import dataclass
from typing import Any, Iterable

187
packages/tracker/meshnet_tracker/server.py
View File

@@ -45,7 +45,7 @@ import urllib.parse
import urllib.request
import uuid
from collections import deque
from dataclasses import dataclass, field
from dataclasses import dataclass, field, replace
from importlib.resources import files
from pathlib import Path
from typing import Any
@@ -53,13 +53,10 @@ from typing import Any
from .accounts import DEFAULT_ACCOUNTS_DB_PATH, AccountStore
from .auth import is_validator_token, sign_hive_request, verify_hive_request
from .capability import (
DEFAULT_POLICY as DEFAULT_CAPABILITY_POLICY,
POLICY_COMPAT,
POLICY_ENFORCE,
STATE_ABSENT,
STATE_ADMITTED,
STATE_MODEL_MISMATCH,
STATE_SHARD_MISMATCH,
STATE_UNCERTIFIED,
CapabilityState,
absent_state,
evaluate_report,
@@ -68,7 +65,7 @@ from .capability import (
)
from .wallet_proof import binding_message, verify_wallet_signature
from .billing import DEFAULT_BILLING_DB_PATH, BillingLedger
from .calibration import DEFAULT_CALIBRATION_DB_PATH, ToplocCalibrationStore
from .calibration import ToplocCalibrationStore
from .hf_pricing import DEFAULT_HF_PRICING_LOG_DB_PATH, HfPricingLog, refresh_preset_price
from .gossip import NodeGossip
from .logging_setup import tracker_logger
@@ -84,7 +81,13 @@ from .routing_stats import (
)
from .model_files import files_for_layer_range, snapshot_dir_for_repo
from .raft import RaftNode
from .recipe import CertificationLedger
from .recipe import (
CertificationLedger,
DistributedForwardEvidence,
PresentedIdentity,
RecipeIdentityError,
RecipeStatus,
)
_CONSOLE_LIMIT = 300
@@ -833,6 +836,11 @@ def _admitted_nodes(nodes: list["_NodeEntry"], policy: str | None) -> list["_Nod
return [node for node in nodes if _capability_routable(node, effective)]
def _route_identity_partition(node: "_NodeEntry") -> tuple[str, ...]:
fingerprint = _node_admission(node).fingerprint
return ("legacy",) if fingerprint is None else ("exact", *fingerprint)
def _select_route(
nodes: list[_NodeEntry],
required_start: int,
@@ -856,29 +864,51 @@ def _select_route(
],
key=lambda n: (n.shard_start, -n.shard_end), # type: ignore[operator]
)
route: list[_NodeEntry] = []
covered_up_to = required_start - 1
def _routing_score(node: "_NodeEntry") -> float:
return _effective_throughput(node, model) * _reputation_multiplier(node, contracts)
while covered_up_to < required_end:
best: _NodeEntry | None = None
for node in candidates:
if node.shard_start <= covered_up_to + 1 and node.shard_end > covered_up_to:
if best is None:
best = node
elif node.shard_end > best.shard_end:
best = node
elif node.shard_end == best.shard_end and _routing_score(node) > _routing_score(best):
best = node
if best is None:
missing = covered_up_to + 1
return [], f"no route available: no registered node covers layer {missing}"
route.append(best)
covered_up_to = best.shard_end
candidates = [n for n in candidates if n is not best]
partitions: dict[tuple[str, ...], list[_NodeEntry]] = {}
for node in candidates:
partitions.setdefault(_route_identity_partition(node), []).append(node)
complete: list[list[_NodeEntry]] = []
furthest = required_start - 1
for partition in partitions.values():
pool = list(partition)
route: list[_NodeEntry] = []
covered_up_to = required_start - 1
while covered_up_to < required_end:
best: _NodeEntry | None = None
for node in pool:
if node.shard_start <= covered_up_to + 1 and node.shard_end > covered_up_to:
if best is None:
best = node
elif node.shard_end > best.shard_end:
best = node
elif (
node.shard_end == best.shard_end
and _routing_score(node) > _routing_score(best)
):
best = node
if best is None:
break
route.append(best)
covered_up_to = best.shard_end
pool = [node for node in pool if node is not best]
furthest = max(furthest, covered_up_to)
if covered_up_to >= required_end:
complete.append(route)
if not complete:
return [], f"no route available: no registered node covers layer {furthest + 1}"
route = max(
complete,
key=lambda candidate: (
min(_routing_score(node) for node in candidate),
-len(candidate),
),
)
return route, ""
@@ -914,7 +944,12 @@ def _enumerate_routes(
for head in heads:
route = [head]
covered_up_to = head.shard_end
pool = [n for n in sharded if n is not head]
head_partition = _route_identity_partition(head)
pool = [
node
for node in sharded
if node is not head and _route_identity_partition(node) == head_partition
]
while covered_up_to < required_end:
best = None
for n in pool:
@@ -2059,14 +2094,24 @@ def _find_pinned_route(
hop_count: int,
) -> list[_NodeEntry] | None:
"""First combination of exactly ``hop_count`` distinct nodes covering the
layer range, where every node extends coverage (US-030 benchmark routes)."""
layer range, where every node extends coverage (US-030 benchmark routes).
Benchmark combos run real inference, so they obey the same DGR-003 rule as
every other route builder: one route, one exact identity. A combo that mixed
two fingerprints — or an exact Shard with a legacy one — would measure a
numerically incoherent route and record the garbage as a benchmark.
"""
for combo in itertools.permutations(nodes, hop_count):
covered = required_start - 1
valid = True
partition = _route_identity_partition(combo[0])
for candidate in combo:
if candidate.shard_start is None or candidate.shard_end is None:
valid = False
break
if _route_identity_partition(candidate) != partition:
valid = False
break
if candidate.shard_start > covered + 1 or candidate.shard_end <= covered:
valid = False
break
@@ -2517,8 +2562,8 @@ def _estimate_prompt_tokens(body: dict) -> int | None:
def _requested_completion_token_limit(body: dict) -> int | None:
for field in ("max_completion_tokens", "max_tokens"):
value = body.get(field)
for key in ("max_completion_tokens", "max_tokens"):
value = body.get(key)
if isinstance(value, bool):
return None
if isinstance(value, (int, float)):
@@ -2899,7 +2944,7 @@ class _TrackerHTTPServer(socketserver.ThreadingMixIn, http.server.HTTPServer):
class _TrackerHandler(http.server.BaseHTTPRequestHandler):
def log_message(self, fmt, *args): # noqa: suppress request logs in tests
def log_message(self, fmt, *args): # suppress request logs in tests
pass
def _send_json(self, status: int, data: dict, headers: dict[str, str] | None = None) -> None:
@@ -6647,6 +6692,84 @@ class TrackerServer:
self._test_runner: TestRunManager | None = test_runner
self.port: int | None = None
def certify_recipe(
self,
identity: PresentedIdentity,
evidence: DistributedForwardEvidence,
) -> RecipeStatus:
"""Promote one exact recipe from tracker-recorded distributed evidence.
This is the tracker's only certification authority. The evidence names
the nodes that served the forward; this method refuses to take any of
them on the evidence's word. For each one it goes back to what *this
tracker* re-derived at admission and requires the evidence to describe
the same Shard: same route fingerprint, same registered range, and the
same shard binding digest — the derivative's own bytes and exact range.
Without the binding check, a route fingerprint is range-independent by
design (:mod:`meshnet_tracker.recipe`), so evidence could name the right
recipe while describing splits nobody on the route was actually serving,
and the certification would attach to blobs that never ran.
"""
with self._lock:
for node_id, participant in zip(
evidence.node_ids, evidence.participants, strict=True
):
node = self._registry.get(node_id)
if node is None:
raise RecipeIdentityError(
f"certification participant {node_id!r} is not registered"
)
admitted = node.capability
if admitted.fingerprint != identity.key:
raise RecipeIdentityError(
f"certification participant {node_id!r} is not admitted under "
"the promoted fingerprint"
)
if admitted.state not in (STATE_UNCERTIFIED, STATE_ADMITTED):
raise RecipeIdentityError(
f"certification participant {node_id!r} has capability state "
f"{admitted.state!r}"
)
registered_range = (
node.shard_start,
None if node.shard_end is None else node.shard_end + 1,
)
if registered_range != (
participant.shard_start,
participant.shard_end,
):
raise RecipeIdentityError(
f"certification participant {node_id!r} range differs from its "
"registered Shard range"
)
# The exact bytes and range this node was admitted on. A node that
# registered without an identity has no binding, and cannot be a
# participant in an exact certification at all.
if admitted.shard_binding_digest is None:
raise RecipeIdentityError(
f"certification participant {node_id!r} registered no exact "
"Shard binding; it cannot certify a recipe"
)
if admitted.shard_binding_digest != participant.shard_binding_digest:
raise RecipeIdentityError(
f"certification participant {node_id!r} presents a Shard "
"binding this tracker did not admit it on; the evidence "
"describes different derivative bytes or a different range"
)
status = self._recipe_certifications.certify(identity, evidence)
for node in self._registry.values():
if (
node.capability.state == STATE_UNCERTIFIED
and node.capability.fingerprint == identity.key
):
node.capability = replace(
node.capability.with_state(STATE_ADMITTED, status.detail),
certification=status.status,
)
return status
def _start_embedded_relay(self) -> dict:
"""Start the shared RelayServer class in-process for tracker+relay deployments."""
if not self._embedded_relay_enabled:
@@ -6942,10 +7065,6 @@ class TrackerServer:
shard_end = int(payload["shard_end"]) if payload.get("shard_end") is not None else None
except (TypeError, ValueError):
return
try:
friendly_name = _normalize_friendly_name(payload.get("friendly_name"))
except ValueError:
friendly_name = None
# The replicated payload is the raw registration body, so the follower can
# resolve precision exactly as the leader did -- including telling a legacy
# absent `quantization` from a declared one. Dropping these fields here

2
packages/validator/meshnet_validator/__init__.py
View File

@@ -545,6 +545,8 @@ def _post_json(url: str, payload: dict, timeout: float = 5.0) -> dict:
__all__ = [
"ToplocAuditConfig",
"ToplocProofClaim",
"ToplocVerificationResult",
"verify_activation_proofs_detailed",
"ValidatorProcess",
"AdaptiveAuditSampler",
"AuditRateConfig",

5
pyproject.toml
View File

@@ -14,6 +14,11 @@ dev = ["pytest>=8", "openai>=1", "langchain-openai>=0.1", "cryptography>=41"]
[tool.setuptools]
packages = []
[tool.ruff]
# Protobuf/gRPC stubs are regenerated by scripts/generate_native_protocol.py;
# linting them would drift the checked-in files from the generator's output.
extend-exclude = ["packages/node/meshnet_node/native_protocol/generated"]
[tool.pytest.ini_options]
testpaths = ["tests"]
markers = [

153
scripts/gen_recipe_fingerprint_vectors.py Normal file
View File

@@ -0,0 +1,153 @@
#!/usr/bin/env python
"""Generate (or verify) the committed DGR-003 fingerprint conformance vectors.
The node and the tracker derive artifact, recipe and Shard-binding digests from
*separate* implementations on purpose: an admission gate that shares code with
the thing it admits is not an independent check. The cost of that independence
is drift — two canonicalizers that quietly stop agreeing would not fail, they
would silently stop forming routes, or worse, silently form wrong ones.
``tests/data/recipe_fingerprint_vectors.json`` is what makes drift loud. It is a
language-neutral artifact — canonical input blocks, expected digests, and the
serialized DGR-002 ``Fingerprint`` bytes — that the node tests, the tracker tests
and (later) the native C++ worker all check themselves against.
python scripts/gen_recipe_fingerprint_vectors.py --check # CI: no drift
python scripts/gen_recipe_fingerprint_vectors.py # rewrite vectors
Rewriting is a deliberate act: if this changes a digest, it changed the wire
contract, and every node and tracker in the fleet has to agree at the same time.
"""
from __future__ import annotations
import argparse
import json
import pathlib
import sys
_ROOT = pathlib.Path(__file__).resolve().parent.parent
sys.path[:0] = [str(_ROOT / "packages" / "node"), str(_ROOT / "packages" / "tracker")]
from meshnet_node.runtime_recipe import ( # noqa: E402
ArtifactIdentity,
DerivativeBinding,
RuntimeRecipe,
ShardIdentity,
)
from meshnet_tracker.recipe import parse_identity # noqa: E402
VECTORS = _ROOT / "tests" / "data" / "recipe_fingerprint_vectors.json"
SCHEMA_VERSION = 1
_RECIPE = RuntimeRecipe(
weight_quantization="Q4_K_M",
activation_dtype="bfloat16",
compute_dtype="float32",
kv_dtype="q8_0",
kv_layout="paged-v1",
tokenizer_revision="0123456789abcdef",
architecture_adapter="llama/range-v1",
backend_id="llama.cpp",
runtime_version="llama.cpp@deadbeef+meshnet.1",
recipe_id="example-gguf",
recipe_version="1",
catalogue_version="2026.07.1",
)
_SOURCE = "a" * 64
_SPLIT_BYTES = "c" * 64
_CONFIG = "b" * 64
def _cases() -> list[tuple[str, str, ShardIdentity]]:
whole = ShardIdentity(
ArtifactIdentity(
"example/model", "0123456789abcdef", _SOURCE, "dense-llama", _CONFIG, 8
),
_RECIPE,
0,
4,
)
# The same recipe on the same source, held as a split: identical route
# fingerprint, different Shard binding. Both halves of that are contract.
derivative = ShardIdentity(
ArtifactIdentity(
"example/model",
"0123456789abcdef",
_SPLIT_BYTES,
"dense-llama",
_CONFIG,
8,
DerivativeBinding(_SOURCE, 4, 8),
),
_RECIPE,
4,
8,
)
return [
("example-v1", "An undivided artifact: content digest is the source digest.", whole),
(
"example-v1-derivative",
"A split of the same source: same fingerprint, different Shard binding.",
derivative,
),
]
def build() -> dict:
vectors = []
for name, description, identity in _cases():
block = identity.to_dict()
presented = parse_identity(block)
# The two implementations must already agree before this is committed.
assert identity.fingerprint.to_dict() == presented.fingerprint_dict(), name
assert identity.shard_binding_digest == presented.shard_binding_digest, name
vectors.append(
{
"name": name,
"description": description,
"identity": block,
"fingerprint": identity.fingerprint.to_dict(),
"shard_binding_digest": identity.shard_binding_digest,
"fingerprint_proto_hex": identity.fingerprint.to_proto()
.SerializeToString(deterministic=True)
.hex(),
}
)
return {"schema_version": SCHEMA_VERSION, "vectors": vectors}
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument(
"--check",
action="store_true",
help="fail if the committed vectors differ from what this code derives",
)
args = parser.parse_args()
built = json.dumps(build(), indent=2, sort_keys=True) + "\n"
if not args.check:
VECTORS.write_text(built, encoding="utf-8")
print(f"wrote {VECTORS.relative_to(_ROOT)}")
return 0
committed = VECTORS.read_text(encoding="utf-8")
if committed != built:
print(
f"{VECTORS.relative_to(_ROOT)} is stale: the identity implementation no "
"longer derives the committed digests.\nIf that change was intended, it "
"is a wire-contract change — rerun without --check and roll out node and "
"tracker together.",
file=sys.stderr,
)
return 1
print(f"{VECTORS.relative_to(_ROOT)} matches the identity implementation")
return 0
if __name__ == "__main__":
raise SystemExit(main())

3
scripts/generate_protocol_goldens.py
View File

@@ -33,6 +33,9 @@ def _vectors() -> dict[str, bytes]:
conformance.GOLDEN_CAPABILITY_REPORT: conformance.serialize(
conformance.canonical_capability_report()
),
conformance.GOLDEN_DECODE_STEP: conformance.serialize(
conformance.canonical_decode_step()
),
}

280
scripts/glm_whole_model_preflight.py Normal file
View File

@@ -0,0 +1,280 @@
#!/usr/bin/env python3
"""DGR-018 whole-model preflight: storage, memory, and the ordered download plan.
Answers, before a single artifact byte moves, the three questions DGR-018's
finish contract asks:
1. Is there one filesystem, outside every forbidden prefix (``/home``), with at
least 250 GB free for the 216.715 GB ``UD-IQ1_S`` artifact plus headroom?
2. Does this host have the 224 GiB runtime-accessible memory the whole-model
oracle load needs (the experimental hard-fit floor from DGR-017)?
3. In what order do the six shards download, and against which exact revision
URL, size, and LFS SHA-256 is each one verified?
Everything is resolved from the pinned target manifest
(``meshnet_node.glm_alpha``) — this script never contacts the network and never
invents a size or digest. It fails closed: a missing requirement is a non-zero
exit and an explicit reason, never a downgrade to a smaller target.
Usage::
python scripts/glm_whole_model_preflight.py # scan all mounts
python scripts/glm_whole_model_preflight.py --dest DIR # judge one destination
python scripts/glm_whole_model_preflight.py --storage-only # download-host mode
python scripts/glm_whole_model_preflight.py --json out.json
"""
from __future__ import annotations
import argparse
import json
import os
import sys
from dataclasses import dataclass
from pathlib import Path
_ROOT = Path(__file__).resolve().parent.parent
sys.path.insert(0, str(_ROOT / "packages/node"))
from meshnet_node.glm_alpha.manifest import ( # noqa: E402
GIB,
TargetManifest,
load_target_manifest,
)
GB = 1000**3
# 250 GB: the 216.715 GB artifact plus resume/temp headroom. DGR-018's issue
# text names this number; it is a storage requirement, not a tunable.
REQUIRED_FREE_BYTES = 250 * GB
# DGR-017's experimental hard-fit floor for whole-model runtime-accessible
# memory. 224 GiB fits weights + Q8_0 KV at 16k context with nothing to spare.
REQUIRED_MEMORY_GIB = 224.0
# Download order, by shard index. Rationale:
# 1 first — 9.4 MB split-metadata shard: proves revision URLs, auth-free
# access, and tooling end-to-end for the cost of a rounding error.
# 6 second — 19.2 GB, the smallest weight shard: proves resume + hash
# verification at real scale before any ~49 GB transfer starts.
# 2..5 — the four ~49 GB shards, sequentially, verify-after-each, so at
# most one unverified partial ever exists on disk.
DOWNLOAD_ORDER = (1, 6, 2, 3, 4, 5)
# Filesystems that can hold a 49 GB file and survive a resume. FAT variants are
# excluded by omission (4 GiB file limit); network/pseudo filesystems likewise.
_USABLE_FSTYPES = {"ext4", "ext3", "xfs", "btrfs", "ntfs", "ntfs3", "fuseblk", "exfat", "f2fs"}
@dataclass(frozen=True)
class MountCandidate:
mountpoint: str
fstype: str
total_bytes: int
free_bytes: int
forbidden: bool
@property
def eligible(self) -> bool:
return not self.forbidden and self.free_bytes >= REQUIRED_FREE_BYTES
def to_dict(self) -> dict:
return {
"mountpoint": self.mountpoint,
"fstype": self.fstype,
"total_gb": round(self.total_bytes / GB, 1),
"free_gb": round(self.free_bytes / GB, 1),
"free_bytes": self.free_bytes,
"forbidden": self.forbidden,
"eligible": self.eligible,
}
def forbidden_prefixes(manifest: TargetManifest) -> tuple[str, ...]:
"""The storage policy pinned in the manifest itself — not a script opinion."""
storage = manifest.raw.get("storage", {})
return tuple(storage.get("forbidden_path_prefixes", ("/home",)))
def is_forbidden(path: str, prefixes: tuple[str, ...]) -> bool:
resolved = os.path.realpath(path)
return any(resolved == p or resolved.startswith(p.rstrip("/") + "/") for p in prefixes)
def _statvfs_free(path: str) -> tuple[int, int]:
st = os.statvfs(path)
return st.f_frsize * st.f_blocks, st.f_frsize * st.f_bavail
def scan_mounts(
prefixes: tuple[str, ...], mounts_file: str = "/proc/mounts"
) -> list[MountCandidate]:
"""Every real, writable-class filesystem on the host, deduplicated by device."""
candidates: dict[str, MountCandidate] = {}
with open(mounts_file, encoding="utf-8") as handle:
for line in handle:
fields = line.split()
if len(fields) < 3:
continue
device, mountpoint, fstype = fields[0], fields[1], fields[2]
mountpoint = mountpoint.replace("\\040", " ")
if fstype not in _USABLE_FSTYPES or device in candidates:
continue
try:
total, free = _statvfs_free(mountpoint)
except OSError:
continue
candidates[device] = MountCandidate(
mountpoint=mountpoint,
fstype=fstype,
total_bytes=total,
free_bytes=free,
forbidden=is_forbidden(mountpoint, prefixes),
)
return sorted(candidates.values(), key=lambda c: c.free_bytes, reverse=True)
def judge_destination(dest: str, prefixes: tuple[str, ...]) -> MountCandidate:
total, free = _statvfs_free(dest)
return MountCandidate(
mountpoint=dest,
fstype="(as-given)",
total_bytes=total,
free_bytes=free,
forbidden=is_forbidden(dest, prefixes),
)
def memory_total_gib(meminfo_file: str = "/proc/meminfo") -> float:
with open(meminfo_file, encoding="utf-8") as handle:
for line in handle:
if line.startswith("MemTotal:"):
return int(line.split()[1]) * 1024 / GIB
raise RuntimeError(f"MemTotal not found in {meminfo_file}")
def download_plan(manifest: TargetManifest, dest_dir: str | None) -> list[dict]:
"""The six shards in download order, each bound to its pinned identity."""
steps = []
for position, index in enumerate(DOWNLOAD_ORDER, start=1):
shard = manifest.shard(index)
target = f"{dest_dir or '$GLM_DEST'}/{shard.path}"
steps.append(
{
"step": position,
"shard_index": shard.index,
"path": shard.path,
"size_bytes": shard.size_bytes,
"size_gb": round(shard.size_bytes / GB, 3),
"sha256": shard.sha256,
"url": shard.url,
"download_command": f'curl -L -C - --fail -o "{target}" "{shard.url}"',
"verify_command": (
f"python scripts/verify_glm_shards.py "
f'--model-dir "{dest_dir or "$GLM_DEST"}" --shard {shard.index}'
),
}
)
return steps
def build_report(*, dest: str | None, storage_only: bool) -> dict:
manifest = load_target_manifest()
prefixes = forbidden_prefixes(manifest)
if dest is not None:
mounts = [judge_destination(dest, prefixes)]
else:
mounts = scan_mounts(prefixes)
eligible = [m for m in mounts if m.eligible]
chosen = eligible[0] if eligible else None
mem_gib = memory_total_gib()
storage_pass = chosen is not None
memory_pass = mem_gib >= REQUIRED_MEMORY_GIB
checks = [
{
"check": "storage",
"requirement": f">= {REQUIRED_FREE_BYTES / GB:.0f} GB free on one filesystem "
f"outside {list(prefixes)}",
"observed": (
f"{chosen.free_bytes / GB:.1f} GB free at {chosen.mountpoint}"
if chosen
else "no eligible filesystem"
),
"passes": storage_pass,
},
{
"check": "memory",
"requirement": f">= {REQUIRED_MEMORY_GIB:.0f} GiB runtime-accessible memory "
"(DGR-017 experimental hard-fit floor)",
"observed": f"{mem_gib:.1f} GiB MemTotal",
"passes": memory_pass,
"waived": storage_only,
},
]
overall = storage_pass and (memory_pass or storage_only)
return {
"generated_by": "scripts/glm_whole_model_preflight.py",
"target": {
"gguf_repo_id": manifest.gguf_repo_id,
"gguf_revision": manifest.gguf_revision,
"quantization": manifest.quantization,
"shard_count": len(manifest.shards),
"total_bytes": manifest.total_bytes,
"total_gb": round(manifest.total_gb, 3),
"manifest_sha256": manifest.digest,
},
"forbidden_path_prefixes": list(prefixes),
"mounts": [m.to_dict() for m in mounts],
"chosen_destination": chosen.to_dict() if chosen else None,
"checks": checks,
"download_authorized": overall,
"storage_only": storage_only,
"download_plan": download_plan(manifest, chosen.mountpoint if chosen else None),
"verdict": "pass" if overall else "fail",
}
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--dest", help="judge this destination directory instead of scanning")
parser.add_argument(
"--storage-only",
action="store_true",
help="download-host mode: require storage, report memory without gating on it",
)
parser.add_argument("--json", dest="json_path", help="also write the full report here")
args = parser.parse_args()
report = build_report(dest=args.dest, storage_only=args.storage_only)
if args.json_path:
Path(args.json_path).write_text(
json.dumps(report, indent=2, ensure_ascii=False) + "\n", encoding="utf-8"
)
target = report["target"]
print(f"target: {target['quantization']} {target['total_gb']} GB, "
f"{target['shard_count']} shards @ {target['gguf_revision'][:12]}")
for check in report["checks"]:
status = "PASS" if check["passes"] else ("WAIVED" if check.get("waived") else "FAIL")
print(f"[{status}] {check['check']}: need {check['requirement']}; "
f"observed {check['observed']}")
if report["chosen_destination"]:
print(f"destination: {report['chosen_destination']['mountpoint']}")
else:
print("destination: NONE — no filesystem outside "
f"{report['forbidden_path_prefixes']} has "
f"{REQUIRED_FREE_BYTES / GB:.0f} GB free")
for mount in report["mounts"]:
why = "forbidden prefix" if mount["forbidden"] else f"{mount['free_gb']} GB free"
print(f" - {mount['mountpoint']} ({mount['fstype']}): {why}")
print(f"verdict: {report['verdict']}")
return 0 if report["download_authorized"] else 1
if __name__ == "__main__":
raise SystemExit(main())

268
scripts/llama_cpp_dependency.py Normal file
View File

@@ -0,0 +1,268 @@
#!/usr/bin/env python3
"""Materialize, verify, build, and smoke-test DGR-004's llama.cpp pin.
This tool deliberately owns only a source dependency boundary. It never
downloads a model, invokes inference, or interprets generated text.
"""
from __future__ import annotations
import argparse
import hashlib
import json
import os
import pathlib
import shutil
import subprocess
import sys
from typing import Any
ROOT = pathlib.Path(__file__).resolve().parents[1]
LLAMA_DIR = ROOT / "packages/node/native/llama"
LOCK_PATH = LLAMA_DIR / "UPSTREAM_LOCK.json"
PATCH_DIR = LLAMA_DIR / "patches"
def _cmake() -> str:
"""Use an explicit override, PATH, or the active Python environment."""
configured = os.environ.get("CMAKE")
if configured:
return configured
on_path = shutil.which("cmake")
if on_path:
return on_path
sibling = pathlib.Path(sys.executable).parent / "cmake"
if sibling.is_file():
return str(sibling)
raise DependencyError("cmake is unavailable; set CMAKE or activate the project toolchain")
class DependencyError(RuntimeError):
"""A fail-closed reproducibility or source-integrity failure."""
def _run(*args: str, cwd: pathlib.Path | None = None) -> str:
try:
completed = subprocess.run(
args,
cwd=cwd,
check=True,
text=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
)
except FileNotFoundError as error:
raise DependencyError(f"required executable is unavailable: {args[0]}") from error
except subprocess.CalledProcessError as error:
detail = (error.stderr or error.stdout).strip()
raise DependencyError(f"command failed: {' '.join(args)}\n{detail}") from error
return completed.stdout.strip()
def _load_lock() -> dict[str, Any]:
try:
lock = json.loads(LOCK_PATH.read_text())
except (OSError, json.JSONDecodeError) as error:
raise DependencyError(f"invalid upstream lock: {LOCK_PATH}: {error}") from error
required = {
"upstream", "commit", "commit_tree", "patched_tree", "patch_series",
"required_upstream_blobs", "patched_paths", "build",
}
missing = sorted(required - lock.keys())
if missing:
raise DependencyError(f"upstream lock is missing fields: {', '.join(missing)}")
commit_file = (LLAMA_DIR / "UPSTREAM_COMMIT").read_text().strip()
if lock["commit"] != commit_file or len(commit_file) != 40:
raise DependencyError("UPSTREAM_COMMIT and UPSTREAM_LOCK.json do not agree on a full commit")
return lock
def _patches(lock: dict[str, Any]) -> list[pathlib.Path]:
series = [line for line in (PATCH_DIR / "series").read_text().splitlines() if line]
if series != lock["patch_series"] or series != sorted(series) or not series:
raise DependencyError("patches/series is empty, unordered, or disagrees with UPSTREAM_LOCK.json")
sums: dict[str, str] = {}
for line in (PATCH_DIR / "SHA256SUMS").read_text().splitlines():
if line and not line.startswith("#"):
digest, name = line.split(maxsplit=1)
sums[name] = digest
patches: list[pathlib.Path] = []
for name in series:
patch = PATCH_DIR / name
if not patch.is_file():
raise DependencyError(f"patch listed in series is missing: {name}")
actual = hashlib.sha256(patch.read_bytes()).hexdigest()
if sums.get(name) != actual:
raise DependencyError(f"patch digest mismatch for {name}: expected {sums.get(name)}, got {actual}")
patches.append(patch)
return patches
def _git(source: pathlib.Path, *args: str) -> str:
return _run("git", "-C", str(source), *args)
def _verify_source(source: pathlib.Path, lock: dict[str, Any], *, require_clean: bool) -> None:
if not (source / ".git").exists():
raise DependencyError(f"not a materialized git checkout: {source}")
if _git(source, "rev-parse", "HEAD") != lock["commit"]:
raise DependencyError("upstream drift: checkout HEAD does not equal the locked commit")
if _git(source, "rev-parse", "HEAD^{tree}") != lock["commit_tree"]:
raise DependencyError("upstream drift: checkout tree does not equal the locked tree")
if require_clean and _git(source, "status", "--porcelain"):
raise DependencyError("local edits detected in materialized llama.cpp checkout")
for relative, expected in lock["required_upstream_blobs"].items():
actual = _git(source, "rev-parse", f"HEAD:{relative}")
if actual != expected:
raise DependencyError(
f"upstream ABI/context drift for {relative}: expected {expected}, got {actual}"
)
if not (source / "LICENSE").is_file():
raise DependencyError("upstream LICENSE is missing; refusing to drop required attribution")
def materialize(source: pathlib.Path, repository: str) -> None:
lock = _load_lock()
_patches(lock)
if source.exists():
raise DependencyError(f"destination already exists; refusing to reuse possibly edited source: {source}")
source.parent.mkdir(parents=True, exist_ok=True)
_run("git", "clone", "--no-checkout", repository, str(source))
_git(source, "checkout", "--detach", lock["commit"])
_verify_source(source, lock, require_clean=True)
def apply(source: pathlib.Path) -> None:
lock = _load_lock()
patches = _patches(lock)
_verify_source(source, lock, require_clean=True)
for patch in patches:
_git(source, "apply", "--check", str(patch))
_git(source, "apply", "--index", str(patch))
_verify_patched_source(source, lock)
def _verify_patched_source(source: pathlib.Path, lock: dict[str, Any]) -> None:
if _git(source, "diff", "--quiet"):
raise DependencyError("local unstaged edits detected after applying patch stack")
changed_paths = _git(source, "diff", "--cached", "--name-only").splitlines()
if changed_paths != lock["patched_paths"]:
raise DependencyError(f"patched paths drifted: expected {lock['patched_paths']}, got {changed_paths}")
if _git(source, "write-tree") != lock["patched_tree"]:
raise DependencyError("patched source tree differs from the locked patch stack")
if _git(source, "diff", "--name-only"):
raise DependencyError("local unstaged edits detected after applying patch stack")
untracked = _git(source, "ls-files", "--others", "--exclude-standard").splitlines()
if untracked:
raise DependencyError(f"untracked files detected after applying patch stack: {untracked}")
def build(source: pathlib.Path, build_dir: pathlib.Path) -> pathlib.Path:
lock = _load_lock()
_patches(lock)
_verify_source(source, lock, require_clean=False)
_verify_patched_source(source, lock)
expected_marker = source / "cmake/meshnet-patch-stack.cmake"
if not expected_marker.is_file():
raise DependencyError("patch stack is not applied: Meshnet CMake marker is absent")
if build_dir.exists():
raise DependencyError(f"build directory already exists; use reproduce for a clean rebuild: {build_dir}")
flags = lock["build"]["configure_flags"]
cmake = _cmake()
_run(cmake, "-G", lock["build"]["generator"], "-S", str(source), "-B", str(build_dir), *flags)
for target in lock["build"]["native_targets"]:
_run(cmake, "--build", str(build_dir), "--target", target, "-j2")
metadata = {
"commit": lock["commit"],
"commit_tree": lock["commit_tree"],
"patches": {patch.name: hashlib.sha256(patch.read_bytes()).hexdigest() for patch in _patches(lock)},
"configure_flags": flags,
"cmake": _run(cmake, "--version").splitlines()[0],
"cxx": _run("c++", "--version").splitlines()[0],
"model_downloads": False,
"semantic_certification": False,
}
(build_dir / "meshnet-build-metadata.json").write_text(json.dumps(metadata, indent=2, sort_keys=True) + "\n")
return build_dir / lock["build"]["smoke_binary"]
def smoke(binary: pathlib.Path) -> None:
if not binary.is_file():
raise DependencyError(f"native smoke binary is missing: {binary}")
smoke_config = _load_lock()["build"]
output = _run(str(binary), *smoke_config["smoke_args"])
expected = smoke_config["smoke_output_token"]
if expected not in output.lower():
raise DependencyError(f"native smoke output did not contain {expected!r}: {output}")
print(output)
def reproduce(work_dir: pathlib.Path, repository: str) -> None:
resolved = work_dir.resolve()
build_root = (ROOT / "build").resolve()
if build_root not in resolved.parents:
raise DependencyError(f"--work-dir must be below {build_root}: {resolved}")
if resolved.exists():
raise DependencyError(
f"work directory already exists; refusing to erase possible local edits: {resolved}"
)
source = resolved / "source"
build_dir = resolved / "build"
materialize(source, repository)
apply(source)
smoke(build(source, build_dir))
def inspect() -> None:
lock = _load_lock()
patches = _patches(lock)
print(json.dumps({
"commit": lock["commit"],
"patch_count": len(patches),
"patches": [patch.name for patch in patches],
"model_downloads": False,
"semantic_certification": False,
"glm_stock_limitations": lock["stock_glm_limitations"],
}, indent=2, sort_keys=True))
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
subcommands = parser.add_subparsers(dest="command", required=True)
subcommands.add_parser("inspect")
materialize_parser = subcommands.add_parser("materialize")
materialize_parser.add_argument("--source-dir", type=pathlib.Path, required=True)
materialize_parser.add_argument("--source-repository", default=_load_lock()["upstream"])
apply_parser = subcommands.add_parser("apply")
apply_parser.add_argument("--source-dir", type=pathlib.Path, required=True)
build_parser = subcommands.add_parser("build")
build_parser.add_argument("--source-dir", type=pathlib.Path, required=True)
build_parser.add_argument("--build-dir", type=pathlib.Path, required=True)
smoke_parser = subcommands.add_parser("smoke")
smoke_parser.add_argument("--binary", type=pathlib.Path, required=True)
reproduce_parser = subcommands.add_parser("reproduce")
reproduce_parser.add_argument("--work-dir", type=pathlib.Path, default=ROOT / "build/dgr-004-smoke")
reproduce_parser.add_argument("--source-repository", default=_load_lock()["upstream"])
args = parser.parse_args()
try:
if args.command == "inspect":
inspect()
elif args.command == "materialize":
materialize(args.source_dir, args.source_repository)
elif args.command == "apply":
apply(args.source_dir)
elif args.command == "build":
build(args.source_dir, args.build_dir)
elif args.command == "smoke":
smoke(args.binary)
else:
reproduce(args.work_dir, args.source_repository)
except DependencyError as error:
print(f"DGR-004 dependency error: {error}", file=sys.stderr)
return 2
return 0
if __name__ == "__main__":
raise SystemExit(main())

135
scripts/verify_glm_shards.py Normal file
View File

@@ -0,0 +1,135 @@
#!/usr/bin/env python3
"""Verify downloaded GLM-5.2 ``UD-IQ1_S`` shards against the pinned manifest.
Every shard must match the exact byte size and LFS SHA-256 that DGR-017 locked
in ``target-manifest.json``. Size is checked first because it is free and
catches truncated resumes; the digest is then streamed so a 49 GB file never
loads into memory. A shard that fails is reported with what was measured — the
file is never deleted or "healed" by this script.
Fail-closed: exit 0 only when every requested shard is present and matches.
Usage::
python scripts/verify_glm_shards.py --model-dir /mnt/models/glm-5.2
python scripts/verify_glm_shards.py --model-dir DIR --shard 6 # one shard
python scripts/verify_glm_shards.py --model-dir DIR --json report.json
"""
from __future__ import annotations
import argparse
import hashlib
import json
import sys
import time
from pathlib import Path
_ROOT = Path(__file__).resolve().parent.parent
sys.path.insert(0, str(_ROOT / "packages/node"))
from meshnet_node.glm_alpha.manifest import ( # noqa: E402
Shard,
TargetManifest,
load_target_manifest,
)
_CHUNK_BYTES = 8 * 1024 * 1024
def resolve_shard_file(model_dir: Path, shard: Shard) -> Path:
"""Accept the repository layout (``UD-IQ1_S/<name>``) or a flat directory."""
nested = model_dir / shard.path
if nested.exists():
return nested
return model_dir / Path(shard.path).name
def streaming_sha256(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as handle:
while chunk := handle.read(_CHUNK_BYTES):
digest.update(chunk)
return digest.hexdigest()
def verify_shard(model_dir: Path, shard: Shard) -> dict:
result: dict = {
"shard_index": shard.index,
"path": shard.path,
"expected_size_bytes": shard.size_bytes,
"expected_sha256": shard.sha256,
}
file = resolve_shard_file(model_dir, shard)
result["file"] = str(file)
if not file.exists():
result.update(status="missing")
return result
size = file.stat().st_size
result["measured_size_bytes"] = size
if size != shard.size_bytes:
result.update(status="size_mismatch")
return result
started = time.monotonic()
measured = streaming_sha256(file)
result["measured_sha256"] = measured
result["hash_seconds"] = round(time.monotonic() - started, 1)
result.update(status="ok" if measured == shard.sha256 else "sha256_mismatch")
return result
def verify_shards(
model_dir: Path, manifest: TargetManifest, indices: list[int] | None = None
) -> dict:
shards = manifest.shards if indices is None else [manifest.shard(i) for i in indices]
results = [verify_shard(model_dir, shard) for shard in shards]
return {
"generated_by": "scripts/verify_glm_shards.py",
"model_dir": str(model_dir),
"gguf_repo_id": manifest.gguf_repo_id,
"gguf_revision": manifest.gguf_revision,
"quantization": manifest.quantization,
"manifest_sha256": manifest.digest,
"shards": results,
"verdict": "pass" if all(r["status"] == "ok" for r in results) else "fail",
}
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--model-dir", required=True, help="directory holding the shards")
parser.add_argument(
"--shard",
type=int,
action="append",
dest="shards",
help="verify only this shard index (repeatable); default: all six",
)
parser.add_argument("--json", dest="json_path", help="also write the full report here")
args = parser.parse_args()
report = verify_shards(Path(args.model_dir), load_target_manifest(), args.shards)
if args.json_path:
Path(args.json_path).write_text(
json.dumps(report, indent=2, ensure_ascii=False) + "\n", encoding="utf-8"
)
for result in report["shards"]:
line = f"shard {result['shard_index']}: {result['status']}"
if result["status"] == "size_mismatch":
line += (
f" (expected {result['expected_size_bytes']}, "
f"measured {result['measured_size_bytes']})"
)
elif result["status"] == "sha256_mismatch":
line += f" (measured {result['measured_sha256']})"
print(line)
print(f"verdict: {report['verdict']}")
return 0 if report["verdict"] == "pass" else 1
if __name__ == "__main__":
raise SystemExit(main())

59
tests/data/recipe_fingerprint_vectors.json
View File

@@ -2,6 +2,7 @@
"schema_version": 1,
"vectors": [
{
"description": "An undivided artifact: content digest is the source digest.",
"fingerprint": {
"catalogue_version": "2026.07.1",
"model_artifact_digest": "8a0f43d6aa49d77834bdb47bcae9f42c886b7ccfe0ac014932b2a2b38697a47b",
@@ -9,6 +10,7 @@
"recipe_version": "1",
"runtime_recipe_digest": "9b14d70b0835a6428457e4888d453649dd0d2e41fc8ac9d84d232c8c237e68fa"
},
"fingerprint_proto_hex": "0a40386130663433643661613439643737383334626462343762636165396634326338383662376363666530616330313439333262326132623338363937613437621240396231346437306230383335613634323834353765343838386434353336343964643064326534316663386163396438346432333263386332333765363866611a0c6578616d706c652d676775662201312a09323032362e30372e31",
"identity": {
"artifact": {
"architecture": "dense-llama",
@@ -46,7 +48,62 @@
"shard_end": 4,
"shard_start": 0
},
"name": "example-v1"
"name": "example-v1",
"shard_binding_digest": "f62d1f76cc18b548782c02850e05c2634da55c0e686c43b9f687bdee7bdefe19"
},
{
"description": "A split of the same source: same fingerprint, different Shard binding.",
"fingerprint": {
"catalogue_version": "2026.07.1",
"model_artifact_digest": "8a0f43d6aa49d77834bdb47bcae9f42c886b7ccfe0ac014932b2a2b38697a47b",
"recipe_id": "example-gguf",
"recipe_version": "1",
"runtime_recipe_digest": "9b14d70b0835a6428457e4888d453649dd0d2e41fc8ac9d84d232c8c237e68fa"
},
"fingerprint_proto_hex": "0a40386130663433643661613439643737383334626462343762636165396634326338383662376363666530616330313439333262326132623338363937613437621240396231346437306230383335613634323834353765343838386434353336343964643064326534316663386163396438346432333263386332333765363866611a0c6578616d706c652d676775662201312a09323032362e30372e31",
"identity": {
"artifact": {
"architecture": "dense-llama",
"architecture_digest": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
"artifact_id": "example/model",
"content_digest": "cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc",
"derived_from": {
"shard_end": 8,
"shard_start": 4,
"source_artifact_digest": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
},
"layer_count": 8,
"revision": "0123456789abcdef"
},
"fingerprint": {
"catalogue_version": "2026.07.1",
"model_artifact_digest": "8a0f43d6aa49d77834bdb47bcae9f42c886b7ccfe0ac014932b2a2b38697a47b",
"recipe_id": "example-gguf",
"recipe_version": "1",
"runtime_recipe_digest": "9b14d70b0835a6428457e4888d453649dd0d2e41fc8ac9d84d232c8c237e68fa"
},
"recipe": {
"activation_dtype": "bfloat16",
"architecture_adapter": "llama/range-v1",
"backend_id": "llama.cpp",
"boundary_schema_version": 1,
"catalogue_version": "2026.07.1",
"compute_dtype": "float32",
"kv_dtype": "q8_0",
"kv_layout": "paged-v1",
"protocol_schema_version": 1,
"recipe_id": "example-gguf",
"recipe_version": "1",
"runtime_version": "llama.cpp@deadbeef+meshnet.1",
"tokenizer_revision": "0123456789abcdef",
"weight_quantization": "Q4_K_M"
},
"schema_version": 1,
"shard_end": 8,
"shard_start": 4
},
"name": "example-v1-derivative",
"shard_binding_digest": "55271611eb63cb81088b8109133f1dff845352298994fc8e9c5824a6a01c83e0"
}
]
}

121
tests/test_architecture_boundary.py Normal file
View File

@@ -0,0 +1,121 @@
"""DGR-006 architecture-defined activation-boundary contract."""
from __future__ import annotations
import struct
import pytest
from meshnet_node.architecture_boundary import (
Architecture,
BoundaryStage,
ProtocolIdentity,
SamplingParameters,
TailOutput,
adapter_for,
)
from meshnet_node.native_protocol import ProtocolError, decode_bundle
def _f32(values: list[float]) -> bytes:
return struct.pack("<" + "f" * len(values), *values)
def _values(payload: bytes) -> tuple[float, ...]:
return struct.unpack("<" + "f" * (len(payload) // 4), payload)
def test_dense_whole_model_and_two_ranges_match_for_prefill_and_greedy_decode() -> None:
adapter = adapter_for(Architecture.DENSE)
embeddings = {3: [1.0, 2.0], 7: [3.0, 4.0]}
head = adapter.bundle_from_token_ids([3, 7], lambda token: embeddings[token])
def head_layers(bundle):
return [value + 10.0 for value in _values(decode_bundle(bundle)["hidden_states"])]
def tail_layers(residual: list[float]) -> tuple[list[float], int]:
# The tail alone applies this fixture's final norm/head and greedy argmax.
logits = [sum(residual) / len(residual) + 20.0, 0.0]
return logits, max(range(len(logits)), key=logits.__getitem__)
# Whole-model prefill retains the unnormalized residual locally. The
# two-range lane sends that same residual before the tail-only head.
whole_prefill = head_layers(head)
seam = adapter.bundle_from_named_payloads(
{"hidden_states": _f32(whole_prefill)}, shape=[1, 2, 2]
)
two_range_prefill = list(_values(decode_bundle(adapter.input_for(BoundaryStage.TAIL, seam))["hidden_states"]))
assert two_range_prefill == whole_prefill
whole_logits, whole_token = tail_layers(whole_prefill)
two_logits, two_token = tail_layers(two_range_prefill)
assert two_logits == whole_logits
assert two_token == whole_token == 0
# One greedy decode step is the same contract with [batch, token, hidden].
decode_seam = adapter.bundle_from_named_payloads(
{"hidden_states": _f32([5.0, 6.0])}, shape=[1, 1, 2]
)
assert tail_layers([5.0, 6.0]) == tail_layers(
list(_values(decode_bundle(adapter.input_for(BoundaryStage.TAIL, decode_seam))["hidden_states"]))
)
def test_middle_and_tail_reject_token_ids_and_require_boundary_bundle() -> None:
adapter = adapter_for(Architecture.MOE)
with pytest.raises(ProtocolError, match="head"):
adapter.bundle_from_token_ids([1], lambda _: [1.0])
with pytest.raises(ProtocolError, match="TensorBundle"):
adapter.input_for(BoundaryStage.MIDDLE, None)
@pytest.mark.parametrize(
("architecture", "names"),
[
(Architecture.MOE, {"hidden_states", "router_logits"}),
(Architecture.MLA, {"hidden_states", "mla_position_state"}),
],
)
def test_architecture_adapters_route_and_validate_their_named_sidebands(
architecture: Architecture, names: set[str]
) -> None:
adapter = adapter_for(architecture)
bundle = adapter.bundle_from_named_payloads(
{
name: (_f32([1.0, 2.0]) if name == "hidden_states" else _f32([0.0]))
for name in names
}
)
assert set(decode_bundle(adapter.input_for(BoundaryStage.MIDDLE, bundle))) == names
with pytest.raises(ProtocolError, match="requires"):
adapter.bundle_from_named_payloads({"hidden_states": _f32([1.0, 2.0])})
def test_unknown_architecture_fails_closed() -> None:
with pytest.raises(ProtocolError, match="unsupported architecture"):
adapter_for("unchecked-name-substitution")
def test_typed_tail_result_binds_sampling_and_request_recipe_identity() -> None:
adapter = adapter_for(Architecture.DENSE)
identity = ProtocolIdentity(
request_id="request-1",
runtime_recipe_digest="sha256:recipe",
chat_template_id="llama3",
chat_template_version="2",
reasoning_mode="max",
architecture=Architecture.DENSE,
)
result = adapter.tail_result(
identity=identity,
sampling=SamplingParameters(temperature=0.0, top_p=1.0, top_k=0, seed=9),
output=TailOutput.sampled_token(42),
)
assert result.identity.request_id == "request-1"
assert result.sampled_token_id == 42
assert result.output_kind == "sampled_token_id"
assert result.message.WhichOneof("output") == "sampled_token_id"

233
tests/test_gguf_distributed_load.py Normal file
View File

@@ -0,0 +1,233 @@
"""Distributed GGUF shard load integration test.
Downloads a small dense-Llama GGUF (TinyLlama 1.1B Q4_K_M ~670 MB),
loads it in shard ranges via the meshnet-range-loader C wrapper, registers
each shard with a live TrackerServer, and verifies routing, range reporting,
and memory scaling.
Set MESHNET_ENABLE_REAL_INFERENCE_TESTS=1 to run.
Evidence class: real-model integration. Downloads ~670 MB on first run.
"""
from __future__ import annotations
import json
import os
import pathlib
import subprocess
import sys
import urllib.error
import urllib.request
import pytest
ROOT = pathlib.Path(__file__).resolve().parent.parent
sys.path[:0] = [
str(ROOT / "packages" / "tracker"),
str(ROOT / "packages" / "node"),
str(ROOT / "packages" / "contracts"),
]
from meshnet_tracker.server import TrackerServer # noqa: E402 — sys.path prepended above
# Only run when explicitly enabled
pytestmark = pytest.mark.skipif(
"MESHNET_ENABLE_REAL_INFERENCE_TESTS" not in os.environ,
reason="set MESHNET_ENABLE_REAL_INFERENCE_TESTS=1 to download and load a real GGUF",
)
# ---------------------------------------------------------------------------
# Model configuration
# ---------------------------------------------------------------------------
HF_REPO = "TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF"
GGUF_FILE = "tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf"
MODEL_URL = f"https://huggingface.co/{HF_REPO}/resolve/main/{GGUF_FILE}"
EXPECTED_LAYERS = 22
GGUF_FLAVOR = GGUF_FILE.replace(".gguf", "")
CACHE_DIR = ROOT / ".cache" / "gguf-models"
LOADER = ROOT / "build" / "dgr-004-final" / "build" / "bin" / "meshnet-range-loader"
LLAMA_LIB_DIR = LOADER.parent
# ---------------------------------------------------------------------------
# Loading helper
# ---------------------------------------------------------------------------
def load_shard(gguf_path: str, start: int, end: int) -> dict:
"""Load a shard range of the GGUF via the C wrapper and return JSON report."""
env = os.environ.copy()
env["LD_LIBRARY_PATH"] = str(LLAMA_LIB_DIR)
result = subprocess.run(
[str(LOADER), gguf_path, str(start), str(end)],
capture_output=True, text=True, timeout=120, env=env,
)
if result.returncode != 0:
# Parse JSON from stdout even on error if it exists
if result.stdout.strip():
try:
return json.loads(result.stdout)
except json.JSONDecodeError:
pass
raise RuntimeError(
f"meshnet-range-loader [{start}, {end}) failed (exit {result.returncode}):\n"
f"stderr: {result.stderr[:500]}"
)
return json.loads(result.stdout)
# ---------------------------------------------------------------------------
# Download and cache
# ---------------------------------------------------------------------------
def _ensure_model() -> pathlib.Path:
CACHE_DIR.mkdir(parents=True, exist_ok=True)
model_path = CACHE_DIR / GGUF_FILE
if model_path.exists() and model_path.stat().st_size > 600 * 1024 * 1024:
return model_path
print(f"Downloading {MODEL_URL} (~670 MB)...", file=sys.stderr)
urllib.request.urlretrieve(MODEL_URL, model_path)
actual_mb = model_path.stat().st_size / (1024 * 1024)
print(f"Downloaded {GGUF_FLAVOR}: {actual_mb:.0f} MB", file=sys.stderr)
return model_path
# ---------------------------------------------------------------------------
# Tracker helpers
# ---------------------------------------------------------------------------
def _post_json(url: str, data: dict) -> dict:
req = urllib.request.Request(
url,
data=json.dumps(data).encode(),
headers={"Content-Type": "application/json"},
method="POST",
)
with urllib.request.urlopen(req, timeout=10) as r:
return json.loads(r.read())
def _get_json(url: str) -> dict:
with urllib.request.urlopen(url, timeout=10) as r:
return json.loads(r.read())
# ---------------------------------------------------------------------------
# Tests
# ---------------------------------------------------------------------------
def test_whole_model_load_and_report():
"""Load the full TinyLlama GGUF and verify metadata."""
gguf = _ensure_model()
report = load_shard(str(gguf), 0, EXPECTED_LAYERS)
assert report["ok"] is True
assert report["start_layer"] == 0
assert report["end_layer"] == EXPECTED_LAYERS
assert report["mapped_bytes"] > 0
assert report["resident_bytes"] >= report["mapped_bytes"]
def test_head_shard_is_less_than_full_model():
"""A head-only shard maps fewer bytes than the full model."""
gguf = _ensure_model()
head = load_shard(str(gguf), 0, 8)
full = load_shard(str(gguf), 0, EXPECTED_LAYERS)
assert head["mapped_bytes"] <= full["mapped_bytes"]
def test_tail_shard_maps_fewer_bytes_than_middle():
"""Fewer layers = fewer bytes (tail has 6 layers, middle has 8)."""
gguf = _ensure_model()
middle = load_shard(str(gguf), 8, 16)
tail = load_shard(str(gguf), 16, EXPECTED_LAYERS)
# Middle (8 layers) must map more than tail (6 layers)
assert middle["mapped_bytes"] > tail["mapped_bytes"]
def test_memory_scales_with_owned_range():
"""More layers = more resident bytes."""
gguf = _ensure_model()
small = load_shard(str(gguf), 0, 4)
large = load_shard(str(gguf), 0, 12)
assert large["mapped_bytes"] > small["mapped_bytes"]
assert large["resident_bytes"] > small["resident_bytes"]
def test_invalid_range_rejected():
"""Loading a range outside GGUF layer bounds fails closed."""
gguf = _ensure_model()
env = os.environ.copy()
env["LD_LIBRARY_PATH"] = str(LLAMA_LIB_DIR)
result = subprocess.run(
[str(LOADER), str(gguf), str(EXPECTED_LAYERS + 1), str(EXPECTED_LAYERS + 5)],
capture_output=True, text=True, timeout=30, env=env,
)
# Should fail with exit code 1 and an error message
assert result.returncode != 0
assert "error" in result.stderr.lower() or "ERROR" in result.stderr
def test_tracker_registers_shard_nodes():
"""Start a tracker, register multiple shard nodes, verify the console."""
gguf = _ensure_model()
tracker = TrackerServer(heartbeat_timeout=60.0)
tracker_port = tracker.start()
shards = [
("head", 0, 8),
("middle", 8, 16),
("tail", 16, EXPECTED_LAYERS),
]
registered_ids = []
try:
for name_tag, start, end in shards:
report = load_shard(str(gguf), start, end)
node_id = f"{GGUF_FLAVOR}-{name_tag}"
registered_ids.append(node_id)
_post_json(
f"http://127.0.0.1:{tracker_port}/v1/nodes/register",
{
"node_id": node_id,
"endpoint": f"http://localhost:{10000 + start}",
"model": GGUF_FLAVOR,
"num_layers": report["end_layer"] - report["start_layer"],
"shard_start": report["start_layer"],
"shard_end": report["end_layer"],
"hardware_profile": {
"mapped_bytes": report["mapped_bytes"],
"resident_bytes": report["resident_bytes"],
},
"score": 1.0,
},
)
# Verify console shows all three registered nodes
console = _get_json(f"http://127.0.0.1:{tracker_port}/v1/console")
registered_eps = {
f"http://localhost:{10000 + start}" for _, start, _ in shards
}
found_eps = set()
for event in console.get("events", []):
if event.get("message") == "node registered":
ep = event.get("fields", {}).get("endpoint", "")
if ep in registered_eps:
found_eps.add(ep)
for _, start, _ in shards:
expected_ep = f"http://localhost:{10000 + start}"
assert expected_ep in found_eps, \
f"endpoint {expected_ep} not found in registration events"
finally:
tracker.stop()

7
tests/test_gossip_and_relay.py
View File

@@ -5,8 +5,7 @@ from __future__ import annotations
import json
import threading
import time
from pathlib import Path
from unittest.mock import MagicMock, patch
from unittest.mock import MagicMock
# ---------------------------------------------------------------------------
@@ -277,7 +276,6 @@ def test_relay_server_peer_list_grows_on_connect():
def test_relay_circuit_relay_proxies_message():
"A node behind NAT (client_a) receives a message via circuit relay from client_b.\n\nTags: gossip, network, relay"
import websockets.sync.client # type: ignore[import]
from meshnet_relay.server import RelayServer
relay = RelayServer(host="127.0.0.1", port=0)
@@ -755,7 +753,6 @@ def test_node_relay_bridge_reconnects_after_failed_connection(monkeypatch):
def _start_tracker_and_register(extra_fields: dict) -> dict:
"""Helper: start tracker, register node with extra gossip fields, return response."""
import http.server
import json as _json
import urllib.request
@@ -766,7 +763,7 @@ def _start_tracker_and_register(extra_fields: dict) -> dict:
url = f"http://127.0.0.1:{port}"
payload = {
"endpoint": f"http://127.0.0.1:8001",
"endpoint": "http://127.0.0.1:8001",
"shard_start": 0,
"shard_end": 7,
"model": "stub-model",

68
tests/test_llama_cpp_dependency.py Normal file
View File

@@ -0,0 +1,68 @@
"""Offline guards for DGR-004's pinned llama.cpp dependency boundary."""
from __future__ import annotations
import hashlib
import json
import pathlib
import subprocess
import sys
ROOT = pathlib.Path(__file__).resolve().parents[1]
LLAMA_DIR = ROOT / "packages/node/native/llama"
SCRIPT = ROOT / "scripts/llama_cpp_dependency.py"
def _sha256(path: pathlib.Path) -> str:
return hashlib.sha256(path.read_bytes()).hexdigest()
def test_lock_and_patch_manifest_are_self_consistent_and_exact() -> None:
lock = json.loads((LLAMA_DIR / "UPSTREAM_LOCK.json").read_text())
commit = (LLAMA_DIR / "UPSTREAM_COMMIT").read_text().strip()
patches = (LLAMA_DIR / "patches/series").read_text().splitlines()
sums = {
name: digest
for digest, name in (
line.split(maxsplit=1)
for line in (LLAMA_DIR / "patches/SHA256SUMS").read_text().splitlines()
if line and not line.startswith("#")
)
}
assert commit == lock["commit"]
assert len(commit) == 40
assert patches == lock["patch_series"]
assert patches == sorted(patches)
assert patches
for patch_name in patches:
patch = LLAMA_DIR / "patches" / patch_name
assert sums[patch_name] == _sha256(patch)
assert "Subject: [PATCH" in patch.read_text()
def test_dependency_script_reports_the_locked_boundary_without_network() -> None:
completed = subprocess.run(
[sys.executable, str(SCRIPT), "inspect"],
cwd=ROOT,
check=True,
capture_output=True,
text=True,
)
report = json.loads(completed.stdout)
assert report["commit"] == (LLAMA_DIR / "UPSTREAM_COMMIT").read_text().strip()
assert report["patch_count"] == 2
assert report["model_downloads"] is False
assert report["semantic_certification"] is False
assert "dense" in report["glm_stock_limitations"].lower()
def test_patch_stack_does_not_contain_meshnet_control_plane_code() -> None:
forbidden = ("tracker", "route session", "grpc", "http", "billing", "wallet")
patch_text = "\n".join(
(LLAMA_DIR / "patches" / name).read_text().lower()
for name in (LLAMA_DIR / "patches/series").read_text().splitlines()
)
assert not any(term in patch_text for term in forbidden)

5
tests/test_mining_cli.py
View File

@@ -5,8 +5,6 @@ from __future__ import annotations
import json
import socket
import sys
import types
from pathlib import Path
from unittest.mock import MagicMock, patch
# A fake node server has no real backend to prove capability with; say so
@@ -134,7 +132,6 @@ def test_print_models_table_runs_without_error(capsys, monkeypatch):
def test_wizard_writes_config_on_happy_path(tmp_path, monkeypatch):
"Wizard writes config on happy path\n\nTags: general"
from meshnet_node import wizard as wiz
from meshnet_node.config import load_config, save_config
# Fake GPU
gpus = [{"index": 0, "name": "RTX 4090", "vram_gb": 24.0, "backend": "cuda"}]
@@ -265,7 +262,6 @@ def test_config_command_no_config_exits_1(tmp_path, monkeypatch):
def test_config_command_prints_saved_config(tmp_path, monkeypatch, capsys):
"Config command prints saved config\n\nTags: general"
from meshnet_node import config as cfg_mod
from meshnet_node.config import save_config
from meshnet_node.cli import main
@@ -309,7 +305,6 @@ def test_detect_num_layers_returns_none_on_error(monkeypatch):
def test_startup_auto_detects_shard_range(monkeypatch, tmp_path):
"When shard_start/end are None, startup reads layer count from catalog.\n\nTags: general"
from meshnet_node import startup as su
from meshnet_node.model_catalog import detect_num_layers
calls = []

157
tests/test_native_identity_emission.py Normal file
View File

@@ -0,0 +1,157 @@
"""DGR-003 production-native identity emission boundary tests."""
from __future__ import annotations
import pytest
from meshnet_node.doctor import DoctorSelection, validate_loaded_backend
from meshnet_node.native_backend import (
ImmutableArtifactPin,
NativeIdentityInputs,
NativeLoadedArtifactReport,
NativeNumericalRecipe,
NativeSessionRejected,
NativeWorkerBackendAdapter,
shard_identity_from_native_report,
)
from meshnet_node.native_protocol import SCHEMA_VERSION, pb
from meshnet_node.recipe_manifest import parse_recipe_manifest
from meshnet_tracker.capability import STATE_UNCERTIFIED, evaluate_report
def _digest(letter: str) -> str:
return letter * 64
def _inputs(**changes: object) -> NativeIdentityInputs:
report = NativeLoadedArtifactReport(
owned_start_layer=2,
owned_end_layer=6,
mapped_bytes=1024,
resident_bytes=768,
registered_bytes=640,
architecture="llama",
architecture_digest=_digest("a"),
layer_count=8,
)
recipe = NativeNumericalRecipe(
weight_quantization="Q4_K_M",
activation_dtype="bfloat16",
compute_dtype="float32",
kv_dtype="q8_0",
kv_layout="llama-kv-v1",
architecture_adapter="dense-llama-v1",
backend_id="llama-cpp",
runtime_version="llama.cpp:e920c523",
recipe_id="native",
recipe_version="1",
catalogue_version="2026.07.1",
)
values: dict[str, object] = {
"loaded_artifact": report,
"artifact_pin": ImmutableArtifactPin(
artifact_id="acme/llama.gguf",
revision="0123456789abcdef",
content_digest=_digest("b"),
),
"tokenizer_revision": "abcdef0123456789",
"numerical_recipe": recipe,
}
values.update(changes)
return NativeIdentityInputs(**values) # type: ignore[arg-type]
def _open(adapter: NativeWorkerBackendAdapter, **changes: object) -> pb.SessionOpen:
identity = adapter.identity
fields: dict[str, object] = {
"schema_version": SCHEMA_VERSION,
"route_session_id": "tracker-session",
"route_epoch": 4,
"fingerprint": identity.fingerprint.to_proto(),
"shard_range": pb.ShardRange(
start_layer=identity.shard_start,
end_layer=identity.shard_end,
effective_start_layer=identity.shard_start,
),
}
fields.update(changes)
return pb.SessionOpen(**fields) # type: ignore[arg-type]
def test_native_identity_uses_loaded_report_not_a_caller_range():
identity = shard_identity_from_native_report(_inputs())
assert (identity.shard_start, identity.shard_end) == (2, 6)
assert identity.artifact.architecture == "llama"
assert identity.artifact.layer_count == 8
def test_native_identity_requires_an_immutable_pin_and_gguf_range():
with pytest.raises(Exception, match="moving reference"):
shard_identity_from_native_report(
_inputs(artifact_pin=ImmutableArtifactPin("a", "main", _digest("b")))
)
with pytest.raises(Exception, match="outside GGUF"):
NativeLoadedArtifactReport(0, 9, 1, 1, 1, "llama", _digest("a"), 8)
def test_native_worker_rejects_bad_session_open_before_session_acceptance():
adapter = NativeWorkerBackendAdapter(_inputs())
accepted = adapter.on_session_open(
_open(adapter), expected_route_session_id="tracker-session", expected_route_epoch=4
)
assert accepted.fingerprint.SerializeToString() == adapter.identity.fingerprint.to_proto().SerializeToString()
with pytest.raises(NativeSessionRejected) as rejected:
adapter.on_session_open(
_open(adapter, route_epoch=5),
expected_route_session_id="tracker-session",
expected_route_epoch=4,
)
assert rejected.value.error.code == pb.ERROR_CODE_EPOCH_STALE
def test_doctor_emits_native_identity_but_keeps_legacy_backend_dark():
manifest = parse_recipe_manifest(
{"schema_version": 1, "catalogue_version": "2026.07.1", "recipes": [
{"id": "native", "version": "1", "backend_id": "llama-cpp"}
]}
)
selection = DoctorSelection("acme/llama.gguf", 2, 5)
native = NativeWorkerBackendAdapter(_inputs())
# The probe needs only the normal backend shape; identity is supplied by the adapter.
native.hidden_size = 8
native.is_head = False
native.is_tail = False
native.device = "cpu"
native.forward_bytes = lambda *args, **kwargs: type("Payload", (), {"body": b"x", "shape": [1]})()
result = validate_loaded_backend(native, selection, manifest.recipes[0], manifest)
assert result.report.identity == native.identity
assert result.report.model.revision == native.identity.artifact.revision
assert result.report.model.config_fingerprint == "sha256:" + _digest("a")
assert result.report.backend.quantization == "Q4_K_M"
assert (result.report.shard.start, result.report.shard.end) == (2, 5)
state = evaluate_report(
result.report.to_dict(),
model_matches=lambda value: value == "acme/llama.gguf",
advertised_model="acme/llama.gguf",
shard_start=2,
shard_end=5,
declared_recipe_id="native",
declared_recipe_version="1",
now=result.report.validated_at,
)
assert state.state == STATE_UNCERTIFIED
class Legacy:
hidden_size = 8
is_head = False
is_tail = False
device = "cpu"
@staticmethod
def forward_bytes(*args, **kwargs):
return type("Payload", (), {"body": b"x", "shape": [1]})()
legacy = validate_loaded_backend(Legacy(), selection, manifest.recipes[0], manifest)
assert legacy.report.identity is None

61
tests/test_native_shard_protocol.py
View File

@@ -34,6 +34,8 @@ from meshnet_node.native_protocol import (
ProtocolError,
checksum_of,
decode_bundle,
decode_step_bundle,
encode_decode_step,
decode_tensor,
default_flow_control,
encode_bundle,
@@ -429,7 +431,7 @@ def test_decode_fast_path_is_much_smaller_than_a_full_envelope_chunk():
idempotency_step=9,
position=1024,
expected_past_len=1024,
tensor=tensor,
bundle=encode_bundle([tensor]),
work_id="work-7f3a",
)
)
@@ -443,7 +445,48 @@ def test_decode_fast_path_is_much_smaller_than_a_full_envelope_chunk():
)
assert len(fast.SerializeToString()) * 2 < len(full.SerializeToString())
assert decode_tensor(fast.decode.tensor) == hidden
assert decode_step_bundle(fast.decode) == {HIDDEN_STATES: hidden}
def test_decode_fast_path_preserves_legacy_one_tensor_compatibility():
tensor = encode_tensor(HIDDEN_STATES, b"\x01\x02" * 8, [1, 1, 8], pb.DTYPE_BFLOAT16)
legacy = pb.DecodeStep(tensor=tensor)
assert decode_step_bundle(legacy) == {HIDDEN_STATES: b"\x01\x02" * 8}
def test_decode_bundle_wins_over_legacy_tensor_and_can_carry_sidebands():
hidden = encode_tensor(HIDDEN_STATES, b"\x01\x02" * 8, [1, 1, 8], pb.DTYPE_BFLOAT16)
sideband = encode_tensor("index_topk", b"\x00" * 4, [1], pb.DTYPE_INT32)
decode = pb.DecodeStep(tensor=hidden, bundle=encode_bundle([hidden, sideband]))
assert decode_step_bundle(decode) == {
HIDDEN_STATES: b"\x01\x02" * 8,
"index_topk": b"\x00" * 4,
}
def test_decode_writer_uses_compact_tensor_only_for_a_certified_one_tensor_bundle():
hidden = encode_tensor(HIDDEN_STATES, b"\x01\x02" * 8, [1, 1, 8], pb.DTYPE_BFLOAT16)
compact = encode_decode_step(
encode_bundle([hidden]), idempotency_step=1, position=1, expected_past_len=1, work_id="w"
)
sideband = encode_tensor("index_topk", b"\x00" * 4, [1], pb.DTYPE_INT32)
expanded = encode_decode_step(
encode_bundle([hidden, sideband]), idempotency_step=1, position=1, expected_past_len=1, work_id="w"
)
assert compact.HasField("tensor") and not compact.HasField("bundle")
assert expanded.HasField("bundle") and not expanded.HasField("tensor")
def test_schema_exposes_typed_tail_result_and_bound_sampling_identity():
assert {"identity", "sampling", "logits", "sampled_token_id"} <= set(
pb.TailResult.DESCRIPTOR.fields_by_name
)
assert {"request_id", "runtime_recipe_digest", "chat_template_id", "chat_template_version", "reasoning_mode"} <= set(
pb.RequestRecipeIdentity.DESCRIPTOR.fields_by_name
)
def test_flow_control_defaults_bound_the_queue_and_the_message():
@@ -491,6 +534,20 @@ def test_committed_vectors_still_encode_as_promised():
report = (conformance.TESTDATA_DIR / conformance.GOLDEN_CAPABILITY_REPORT).read_bytes()
assert conformance.serialize(conformance.canonical_capability_report()) == report
decode = (conformance.TESTDATA_DIR / conformance.GOLDEN_DECODE_STEP).read_bytes()
assert conformance.serialize(conformance.canonical_decode_step()) == decode
def test_decode_golden_preserves_the_multi_tensor_boundary():
golden = (conformance.TESTDATA_DIR / conformance.GOLDEN_DECODE_STEP).read_bytes()
request = pb.SessionRequest.FromString(golden)
assert request.decode.idempotency_step == 43
assert decode_step_bundle(request.decode) == {
HIDDEN_STATES: bytes(range(16)),
"index_topk": (3).to_bytes(4, "little"),
}
def test_golden_session_request_round_trips_with_every_field_intact():
golden = (conformance.TESTDATA_DIR / conformance.GOLDEN_SESSION_REQUEST).read_bytes()

2
tests/test_openai_gateway.py
View File

@@ -162,7 +162,7 @@ def test_streaming_end_to_end_http(two_node_setup):
assert "text/event-stream" in content_type
raw = resp.read().decode()
data_lines = [l for l in raw.strip().splitlines() if l.startswith("data: ")]
data_lines = [line for line in raw.strip().splitlines() if line.startswith("data: ")]
assert data_lines, "No SSE data lines found"
assert data_lines[-1] == "data: [DONE]"

1
tests/test_real_model_backend.py
View File

@@ -13,7 +13,6 @@ import urllib.request
import pytest
from meshnet_node.model_backend import (
InsufficientVRAMError,
PartialModelLoadUnsupported,
KVCacheMiss,
TensorPayload,

710
tests/test_runtime_recipe_identity.py
View File

@@ -9,6 +9,7 @@ import time
import pytest
from meshnet_node.native_protocol import SCHEMA_VERSION, pb
from meshnet_node.runtime_recipe import (
ArtifactIdentity,
CompatibilityFingerprint,
@@ -17,31 +18,45 @@ from meshnet_node.runtime_recipe import (
RuntimeRecipe,
ShardIdentity,
check_handshake,
check_session_open,
check_route,
handshake_error,
)
from meshnet_tracker.capability import (
POLICY_COMPAT,
POLICY_ENFORCE,
CapabilityState,
STATE_ADMITTED,
STATE_FINGERPRINT_MISMATCH,
STATE_MODEL_MISMATCH,
STATE_RECIPE_MISMATCH,
STATE_UNCERTIFIED,
absent_state,
evaluate_report,
)
from meshnet_tracker.server import TrackerServer, _capability_from_registration
from meshnet_tracker.server import (
TrackerServer,
_capability_from_registration,
_find_pinned_route,
_NodeEntry,
_select_route,
)
from meshnet_tracker.recipe import (
CertificationLedger,
DistributedForwardEvidence,
RecipeIdentityError as TrackerRecipeIdentityError,
parse_identity,
)
VECTORS = Path(__file__).parent / "data" / "recipe_fingerprint_vectors.json"
def _digest(char: str) -> str:
return char * 64
def _identity(start: int = 0, end: int = 4, **recipe_changes: object) -> ShardIdentity:
recipe_fields: dict[str, object] = {
def _recipe(**changes: object) -> RuntimeRecipe:
fields: dict[str, object] = {
"weight_quantization": "Q4_K_M",
"activation_dtype": "bfloat16",
"compute_dtype": "float32",
@@ -55,8 +70,12 @@ def _identity(start: int = 0, end: int = 4, **recipe_changes: object) -> ShardId
"recipe_version": "1",
"catalogue_version": "2026.07.1",
}
recipe_fields.update(recipe_changes)
recipe = RuntimeRecipe(**recipe_fields) # type: ignore[arg-type]
fields.update(changes)
return RuntimeRecipe(**fields) # type: ignore[arg-type]
def _identity(start: int = 0, end: int = 4, **recipe_changes: object) -> ShardIdentity:
"""A Shard of the whole-model artifact: every node holds the same file."""
return ShardIdentity(
ArtifactIdentity(
"example/model",
@@ -66,7 +85,25 @@ def _identity(start: int = 0, end: int = 4, **recipe_changes: object) -> ShardId
_digest("b"),
8,
),
recipe,
_recipe(**recipe_changes),
start,
end,
)
def _split(start: int, end: int, content: str, **recipe_changes: object) -> ShardIdentity:
"""A derivative: its own bytes, bound to the exact source it was cut from."""
return ShardIdentity(
ArtifactIdentity(
"example/model",
"0123456789abcdef",
_digest(content),
"dense-llama",
_digest("b"),
8,
DerivativeBinding(_digest("a"), start, end),
),
_recipe(**recipe_changes),
start,
end,
)
@@ -75,7 +112,11 @@ def _identity(start: int = 0, end: int = 4, **recipe_changes: object) -> ShardId
def _report(identity: ShardIdentity) -> dict:
return {
"schema_version": 1,
"model": {"model_id": "example/model"},
"model": {
"model_id": "example/model",
"revision": identity.artifact.revision,
"config_fingerprint": "sha256:" + identity.artifact.architecture_digest,
},
"shard": {"start": identity.shard_start, "end": identity.shard_end - 1},
"recipe": identity.recipe.to_dict() | {
"recipe_id": identity.recipe.recipe_id,
@@ -92,29 +133,101 @@ def _report(identity: ShardIdentity) -> dict:
def _evaluate(report: dict, **kwargs: object):
kwargs.setdefault("shard_start", 0)
kwargs.setdefault("shard_end", 3)
return evaluate_report(
report,
model_matches=lambda model: model == "example/model",
advertised_model="example/model",
shard_start=0,
shard_end=3,
now=100.0,
**kwargs,
**kwargs, # type: ignore[arg-type]
)
def _register(tracker: TrackerServer, node_id: str, identity: ShardIdentity) -> _NodeEntry:
"""Register one node exactly as the HTTP path does: its own report, the tracker's ledger."""
report = _report(identity)
report["validated_at"] = time.time()
# The registry range is end-inclusive; the identity's is end-exclusive.
start, end = identity.shard_start, identity.shard_end - 1
capability = _capability_from_registration(
{"capability_report": report},
model="example/model",
hf_repo=None,
shard_start=start,
shard_end=end,
recipe_certifications=tracker._recipe_certifications,
)
entry = _NodeEntry(
node_id=node_id,
endpoint=f"http://{node_id}",
shard_start=start,
shard_end=end,
model="example/model",
shard_checksum=None,
hardware_profile={},
wallet_address=None,
score=1.0,
capability=capability,
)
tracker._registry[node_id] = entry
return entry
def _evidence(
*shards: ShardIdentity,
node_ids: tuple[str, ...] = ("physical-a", "physical-b"),
layer_count: int = 8,
fingerprint: tuple[str, str] | None = None,
**changes: object,
) -> DistributedForwardEvidence:
presented = tuple(parse_identity(s.to_dict()) for s in shards)
fields: dict[str, object] = {
"route_session_id": "session",
"route_epoch": 1,
"node_ids": node_ids,
"shard_ranges": tuple((s.shard_start, s.shard_end) for s in shards),
"tokens_generated": 1,
"layer_count": layer_count,
"fingerprint": fingerprint or presented[0].key,
"participants": presented,
}
fields.update(changes)
return DistributedForwardEvidence(**fields) # type: ignore[arg-type]
# --- Identity: the axes, the digests, and what they do and do not commit to ---
def test_node_and_tracker_share_the_committed_canonical_fingerprint_vector():
vector_path = Path(__file__).parent / "data" / "recipe_fingerprint_vectors.json"
vector = json.loads(vector_path.read_text(encoding="utf-8"))["vectors"][0]
expected = vector["fingerprint"]
identity = ShardIdentity.from_dict(vector["identity"])
for vector in json.loads(VECTORS.read_text(encoding="utf-8"))["vectors"]:
expected = vector["fingerprint"]
identity = ShardIdentity.from_dict(vector["identity"])
presented = parse_identity(vector["identity"])
assert identity.fingerprint.to_dict() == expected
assert parse_identity(vector["identity"]).fingerprint_dict() == expected
assert (
CompatibilityFingerprint.from_proto(identity.fingerprint.to_proto()).to_dict()
== expected
)
assert identity.fingerprint.to_dict() == expected, vector["name"]
assert presented.fingerprint_dict() == expected, vector["name"]
assert (
CompatibilityFingerprint.from_proto(
identity.fingerprint.to_proto()
).to_dict()
== expected
), vector["name"]
# The binding digest is derived independently on both sides too, so a
# node and a tracker cannot disagree about which bytes a Shard holds.
assert identity.shard_binding_digest == vector["shard_binding_digest"], vector["name"]
assert presented.shard_binding_digest == vector["shard_binding_digest"], vector["name"]
# The DGR-002 wire encoding is contract as well, so the native worker can
# be held to these bytes without reimplementing the JSON canonicalizer.
wire = identity.fingerprint.to_proto().SerializeToString(deterministic=True)
assert wire.hex() == vector["fingerprint_proto_hex"], vector["name"]
def test_committed_vectors_cover_a_whole_model_and_a_derivative_shard():
names = {v["name"] for v in json.loads(VECTORS.read_text(encoding="utf-8"))["vectors"]}
assert {"example-v1", "example-v1-derivative"} <= names
@pytest.mark.parametrize(
@@ -143,25 +256,31 @@ def test_every_recipe_axis_changes_the_fingerprint_and_blocks_route(axis, value)
def test_split_artifact_is_bound_to_exact_source_and_owned_range():
source = _identity()
split = ShardIdentity(
ArtifactIdentity(
"example/model",
"0123456789abcdef",
_digest("c"),
"dense-llama",
_digest("b"),
8,
DerivativeBinding(_digest("a"), 4, 8),
),
source.recipe,
4,
8,
)
split = _split(4, 8, "c")
assert source.fingerprint.matches(split.fingerprint)
with pytest.raises(RecipeIdentityError):
ShardIdentity(split.artifact, split.recipe, 3, 8)
def test_derivative_bytes_and_range_have_a_separate_shard_binding_digest():
left = _split(0, 4, "c")
changed_bytes = _split(0, 4, "d")
changed_range = _split(4, 8, "c")
# Same route fingerprint — all three are the same recipe on the same source,
# which is exactly what route formation should conclude.
assert left.fingerprint.key == changed_bytes.fingerprint.key
assert left.fingerprint.key == changed_range.fingerprint.key
# Different bytes and different ranges are still separately pinned.
assert left.shard_binding_digest != changed_bytes.shard_binding_digest
assert left.shard_binding_digest != changed_range.shard_binding_digest
assert (
parse_identity(left.to_dict()).shard_binding_digest
== left.shard_binding_digest
)
def test_declared_digest_mismatch_is_recomputed_and_rejected_not_authenticated():
report = _report(_identity())
report["identity"]["fingerprint"]["runtime_recipe_digest"] = _digest("f")
@@ -169,42 +288,7 @@ def test_declared_digest_mismatch_is_recomputed_and_rejected_not_authenticated()
assert _evaluate(report).state == STATE_FINGERPRINT_MISMATCH
def test_exact_recipe_registers_dark_until_tracker_certification():
identity = _identity()
report = _report(identity)
ledger = CertificationLedger()
dark = _evaluate(report, ledger=ledger)
assert dark.state == STATE_UNCERTIFIED
# Unit fixtures cannot promote a recipe: this is the explicit trust boundary.
with pytest.raises(ValueError, match="synthetic"):
ledger.certify(
parse_identity(identity.to_dict()),
DistributedForwardEvidence(
"session",
1,
("a", "b"),
((0, 4), (4, 8)),
1,
8,
identity.fingerprint.key,
synthetic=True,
),
)
with pytest.raises(ValueError, match="fingerprint does not match"):
ledger.certify(
parse_identity(identity.to_dict()),
DistributedForwardEvidence(
"session",
1,
("a", "b"),
((0, 4), (4, 8)),
1,
8,
(_digest("e"), _digest("f")),
),
)
# --- Capability admission: the identity block must match the proof it rides with ---
def test_capability_identity_must_match_the_proof_labels():
@@ -228,37 +312,469 @@ def test_capability_identity_must_match_the_proof_labels():
assert _evaluate(wrong_quantization).state == STATE_RECIPE_MISMATCH
def test_tracker_server_owns_the_ledger_used_by_registration():
def test_capability_identity_must_match_the_proven_revision_and_config():
identity = _identity()
report = _report(identity)
report["validated_at"] = time.time()
payload = {"capability_report": report}
tracker = TrackerServer()
def evaluate():
return _capability_from_registration(
payload,
model="example/model",
hf_repo=None,
shard_start=0,
shard_end=3,
recipe_certifications=tracker._recipe_certifications,
wrong_revision = _report(identity)
wrong_revision["model"]["revision"] = "fedcba9876543210"
assert _evaluate(wrong_revision).state == STATE_MODEL_MISMATCH
wrong_config = _report(identity)
wrong_config["model"]["config_fingerprint"] = "sha256:" + _digest("c")
assert _evaluate(wrong_config).state == STATE_MODEL_MISMATCH
missing_config = _report(identity)
del missing_config["model"]["config_fingerprint"]
assert _evaluate(missing_config).state == STATE_MODEL_MISMATCH
def test_an_exact_identity_without_a_ledger_is_dark_not_admitted():
"""No certification authority is not permission to serve — it is no proof."""
state = _evaluate(_report(_identity()), ledger=None)
assert state.state == STATE_UNCERTIFIED
assert not state.routable_under(POLICY_COMPAT)
assert not state.routable_under(POLICY_ENFORCE)
def test_admission_records_the_rederived_binding_not_the_declared_one():
split = _split(0, 4, "c")
state = _evaluate(_report(split), ledger=CertificationLedger())
assert state.shard_binding_digest == split.shard_binding_digest
assert state.fingerprint == split.fingerprint.key
# --- Certification: only the tracker, only on evidence it re-derived itself ---
def test_certification_requires_prior_dark_registration():
ledger = CertificationLedger()
identity = parse_identity(_identity().to_dict())
with pytest.raises(TrackerRecipeIdentityError, match="not registered"):
ledger.certify(identity, _evidence(_identity(0, 4), _identity(4, 8)))
def test_synthetic_or_mismatched_evidence_never_certifies():
identity = _identity()
ledger = CertificationLedger()
assert _evaluate(_report(identity), ledger=ledger).state == STATE_UNCERTIFIED
presented = parse_identity(identity.to_dict())
# A unit fixture is not a distributed forward. This is the trust boundary.
with pytest.raises(TrackerRecipeIdentityError, match="synthetic"):
ledger.certify(
presented, _evidence(_identity(0, 4), _identity(4, 8), synthetic=True)
)
assert evaluate().state == STATE_UNCERTIFIED
tracker._recipe_certifications.certify(
parse_identity(identity.to_dict()),
DistributedForwardEvidence(
"session",
1,
("physical-a", "physical-b"),
((0, 4), (4, 8)),
1,
8,
identity.fingerprint.key,
),
with pytest.raises(TrackerRecipeIdentityError, match="fingerprint does not match"):
ledger.certify(
presented,
_evidence(
_identity(0, 4),
_identity(4, 8),
fingerprint=(_digest("e"), _digest("f")),
),
)
# A single node, however real, is not a distributed forward.
with pytest.raises(TrackerRecipeIdentityError, match="distributed forward requires"):
ledger.certify(
presented, _evidence(_identity(0, 8), node_ids=("physical-a",))
)
# A hole in the coverage means those layers were never computed.
with pytest.raises(TrackerRecipeIdentityError, match="owned by no Shard"):
ledger.certify(
presented, _evidence(_identity(0, 3), _identity(4, 8))
)
def test_certification_evidence_layer_count_cannot_be_substituted():
"""An 8-layer recipe cannot be promoted by evidence for a 2-layer route."""
identity = _identity()
ledger = CertificationLedger()
assert _evaluate(_report(identity), ledger=ledger).state == STATE_UNCERTIFIED
with pytest.raises(TrackerRecipeIdentityError, match="layer count"):
ledger.certify(
parse_identity(identity.to_dict()),
_evidence(_identity(0, 4), _identity(4, 8), layer_count=2),
)
def test_evidence_participants_must_be_the_recipe_being_promoted():
identity = _identity()
ledger = CertificationLedger()
assert _evaluate(_report(identity), ledger=ledger).state == STATE_UNCERTIFIED
# Participants running a different recipe cannot vouch for this one, even
# when the evidence header names the right fingerprint.
with pytest.raises(TrackerRecipeIdentityError, match="participant fingerprint"):
ledger.certify(
parse_identity(identity.to_dict()),
_evidence(
_identity(0, 4),
_identity(4, 8, kv_dtype="float16"),
fingerprint=identity.fingerprint.key,
),
)
# Nor can a participant whose identity is not the range it is recorded under.
with pytest.raises(TrackerRecipeIdentityError, match="recorded effective range"):
ledger.certify(
parse_identity(identity.to_dict()),
_evidence(
_identity(0, 4),
_identity(4, 8),
shard_ranges=((0, 4), (0, 4)),
),
)
def test_tracker_owns_the_only_promotion_path_and_re_admits_dark_nodes():
tracker = TrackerServer()
head, tail = _split(0, 4, "c"), _split(4, 8, "d")
node_a = _register(tracker, "physical-a", head)
node_b = _register(tracker, "physical-b", tail)
# A node on a *different* recipe, which must not be swept up by the promotion.
other = _register(tracker, "physical-c", _identity(0, 8, kv_dtype="float16"))
assert node_a.capability.state == STATE_UNCERTIFIED
assert node_b.capability.state == STATE_UNCERTIFIED
status = tracker.certify_recipe(
parse_identity(head.to_dict()), _evidence(head, tail)
)
assert evaluate().state == STATE_ADMITTED
assert status.may_serve
assert node_a.capability.state == STATE_ADMITTED
assert node_b.capability.state == STATE_ADMITTED
assert other.capability.state == STATE_UNCERTIFIED
# A node registering after the fact lands admitted, from the same ledger.
assert _register(tracker, "physical-d", head).capability.state == STATE_ADMITTED
def test_the_network_map_certification_field_tracks_the_ledger():
"""The map must say what the ledger knows — "dark" before, "certified" after."""
tracker = TrackerServer()
head, tail = _split(0, 4, "c"), _split(4, 8, "d")
node_a = _register(tracker, "physical-a", head)
node_b = _register(tracker, "physical-b", tail)
assert node_a.capability.certification == "dark"
assert node_a.capability.to_dict()["certification"] == "dark"
tracker.certify_recipe(parse_identity(head.to_dict()), _evidence(head, tail))
assert node_a.capability.certification == "certified"
assert node_b.capability.certification == "certified"
assert _register(tracker, "physical-d", head).capability.certification == "certified"
# A node that presented no identity has nothing for the ledger to say.
assert absent_state().certification is None
def test_certification_rejects_participants_the_tracker_did_not_admit():
tracker = TrackerServer()
head, tail = _split(0, 4, "c"), _split(4, 8, "d")
_register(tracker, "physical-a", head)
_register(tracker, "physical-b", tail)
promoted = parse_identity(head.to_dict())
# A node nobody registered cannot have served the forward.
with pytest.raises(TrackerRecipeIdentityError, match="not registered"):
tracker.certify_recipe(
promoted, _evidence(head, tail, node_ids=("physical-a", "ghost"))
)
# Same recipe, same ranges, but derivative bytes neither node was admitted
# on: certification must not attach to blobs that never ran.
with pytest.raises(TrackerRecipeIdentityError, match="binding this tracker did not admit"):
tracker.certify_recipe(
promoted, _evidence(_split(0, 4, "e"), _split(4, 8, "f"))
)
# Right bytes, but swapped between the nodes that served them.
with pytest.raises(TrackerRecipeIdentityError, match="range differs|binding this tracker"):
tracker.certify_recipe(
promoted, _evidence(tail, head)
)
assert tracker._registry["physical-a"].capability.state == STATE_UNCERTIFIED
def test_a_legacy_node_cannot_be_an_exact_certification_participant():
tracker = TrackerServer()
head, tail = _split(0, 4, "c"), _split(4, 8, "d")
_register(tracker, "physical-a", head)
legacy = _NodeEntry(
node_id="physical-b",
endpoint="http://physical-b",
shard_start=4,
shard_end=7,
model="example/model",
shard_checksum=None,
hardware_profile={},
wallet_address=None,
score=1.0,
capability=CapabilityState(state=STATE_ADMITTED, model_id="example/model"),
)
tracker._registry["physical-b"] = legacy
with pytest.raises(TrackerRecipeIdentityError, match="not admitted under"):
tracker.certify_recipe(parse_identity(head.to_dict()), _evidence(head, tail))
def test_certification_does_not_survive_a_tracker_restart():
"""The ledger is in-memory. A restart loses it, and that must fail *closed*."""
tracker = TrackerServer()
head, tail = _split(0, 4, "c"), _split(4, 8, "d")
_register(tracker, "physical-a", head)
_register(tracker, "physical-b", tail)
tracker.certify_recipe(parse_identity(head.to_dict()), _evidence(head, tail))
restarted = TrackerServer()
assert _register(restarted, "physical-a", head).capability.state == STATE_UNCERTIFIED
# --- Route formation: one route, one exact fingerprint ---
def _route_node(
node_id: str,
start: int,
end: int,
fingerprint: tuple[str, str] | None,
*,
speed: float = 1.0,
state: str = STATE_ADMITTED,
) -> _NodeEntry:
capability = CapabilityState(
state=state,
model_id="example/model",
shard_start=start,
shard_end=end,
model_artifact_digest=None if fingerprint is None else fingerprint[0],
runtime_recipe_digest=None if fingerprint is None else fingerprint[1],
)
return _NodeEntry(
node_id=node_id,
endpoint=f"http://{node_id}",
shard_start=start,
shard_end=end,
model="example/model",
shard_checksum=None,
hardware_profile={},
wallet_address=None,
score=1.0,
benchmark_tokens_per_sec=speed,
capability=capability,
)
def test_route_selection_never_mixes_exact_fingerprints():
key_a = (_digest("a"), _digest("b"))
key_b = (_digest("c"), _digest("d"))
route, error = _select_route(
[_route_node("a", 0, 3, key_a), _route_node("b", 4, 7, key_b)],
0,
7,
policy=POLICY_ENFORCE,
)
assert route == []
assert "no route available" in error
def test_route_selection_never_mixes_an_exact_shard_with_a_legacy_one():
"""A node with no canonical digests cannot complete an exact route."""
key_a = (_digest("a"), _digest("b"))
route, error = _select_route(
[_route_node("exact-head", 0, 3, key_a), _route_node("legacy-tail", 4, 7, None)],
0,
7,
policy=POLICY_COMPAT,
)
assert route == []
assert "no route available" in error
def test_route_selection_falls_back_to_a_complete_fingerprint_group():
key_a = (_digest("a"), _digest("b"))
key_b = (_digest("c"), _digest("d"))
route, error = _select_route(
[
_route_node("fast-incomplete", 0, 3, key_b, speed=100.0),
_route_node("a-head", 0, 3, key_a),
_route_node("a-tail", 4, 7, key_a),
],
0,
7,
policy=POLICY_ENFORCE,
)
assert error == ""
assert [node.node_id for node in route] == ["a-head", "a-tail"]
def test_a_homogeneous_legacy_fleet_routes_exactly_as_before():
"""DGR-003 partitions routes; it must not change a fleet that has no identity."""
route, error = _select_route(
[
_route_node("slow", 0, 3, None, speed=1.0),
_route_node("fast", 0, 3, None, speed=50.0),
_route_node("tail", 4, 7, None),
],
0,
7,
policy=POLICY_COMPAT,
)
assert error == ""
assert [node.node_id for node in route] == ["fast", "tail"]
def test_an_uncertified_node_is_not_routable_under_either_policy():
key = (_digest("a"), _digest("b"))
nodes = [
_route_node("head", 0, 3, key, state=STATE_UNCERTIFIED),
_route_node("tail", 4, 7, key, state=STATE_UNCERTIFIED),
]
for policy in (POLICY_COMPAT, POLICY_ENFORCE):
route, error = _select_route(nodes, 0, 7, policy=policy)
assert route == [], policy
assert "no route available" in error
def test_pinned_benchmark_routes_obey_the_same_partition_rule():
"""US-030 benchmark combos run real inference; they may not mix identities."""
key_a = (_digest("a"), _digest("b"))
key_b = (_digest("c"), _digest("d"))
# An exact head with only a legacy tail, or a tail on another fingerprint,
# has no two-hop combo at all.
assert _find_pinned_route(
[_route_node("exact", 0, 3, key_a), _route_node("legacy", 4, 7, None)], 0, 7, 2
) is None
assert _find_pinned_route(
[_route_node("a", 0, 3, key_a), _route_node("b", 4, 7, key_b)], 0, 7, 2
) is None
# Homogeneous fleets — exact or legacy — still form pinned combos.
exact = _find_pinned_route(
[_route_node("a", 0, 3, key_a), _route_node("b", 4, 7, key_a)], 0, 7, 2
)
assert exact is not None and [node.node_id for node in exact] == ["a", "b"]
legacy = _find_pinned_route(
[_route_node("a", 0, 3, None), _route_node("b", 4, 7, None)], 0, 7, 2
)
assert legacy is not None and [node.node_id for node in legacy] == ["a", "b"]
# --- gRPC handshake (DGR-002 SessionOpen) ---
def _session_open(identity: ShardIdentity, **changes: object) -> "pb.SessionOpen":
fields: dict[str, object] = {
"schema_version": SCHEMA_VERSION,
"route_session_id": "session",
"route_epoch": 1,
"fingerprint": identity.fingerprint.to_proto(),
"shard_range": pb.ShardRange(
start_layer=identity.shard_start,
end_layer=identity.shard_end,
effective_start_layer=identity.shard_start,
),
}
fields.update(changes)
return pb.SessionOpen(**fields) # type: ignore[arg-type]
def test_session_open_accepts_the_exact_local_shard():
local = _identity()
assert (
check_session_open(
local,
_session_open(local),
expected_route_session_id="session",
expected_route_epoch=1,
)
== ()
)
assert handshake_error(()) is None
def test_session_open_rejects_a_matching_fingerprint_with_the_wrong_range():
local = _identity()
opened = _session_open(
local,
shard_range=pb.ShardRange(start_layer=0, end_layer=5, effective_start_layer=0),
)
mismatches = check_session_open(local, opened)
assert mismatches
assert handshake_error(mismatches).code == pb.ERROR_CODE_SHARD_RANGE_MISMATCH
def test_session_open_rejects_an_effective_start_outside_the_shard():
local = _identity(4, 8)
opened = _session_open(
local,
shard_range=pb.ShardRange(start_layer=4, end_layer=8, effective_start_layer=8),
)
mismatches = check_session_open(local, opened)
assert mismatches
assert handshake_error(mismatches).code == pb.ERROR_CODE_SHARD_RANGE_MISMATCH
def test_session_open_rejects_a_foreign_protocol_schema():
local = _identity()
mismatches = check_session_open(
local, _session_open(local, schema_version=SCHEMA_VERSION + 1)
)
assert mismatches
# A schema disagreement is its own protocol outcome — not a range problem,
# and not a fingerprint problem either: the peer may hold the right recipe
# and simply speak a schema this node cannot.
assert handshake_error(mismatches).code == pb.ERROR_CODE_SCHEMA_UNSUPPORTED
@pytest.mark.parametrize(
("changes", "expected_session", "expected_epoch"),
[
({"route_session_id": "other-session"}, "session", 1),
({"route_epoch": 2}, "session", 1),
({"route_session_id": ""}, None, None),
({"route_epoch": 0}, None, None),
],
)
def test_session_open_rejects_missing_or_stale_tracker_route_assignment(
changes, expected_session, expected_epoch
):
local = _identity()
mismatches = check_session_open(
local,
_session_open(local, **changes),
expected_route_session_id=expected_session,
expected_route_epoch=expected_epoch,
)
assert mismatches
assert handshake_error(mismatches).code == pb.ERROR_CODE_EPOCH_STALE
def test_a_digest_disagreement_dominates_the_handshake_error_code():
"""Wrong fingerprint plus wrong range is a wrong peer, not a re-routable one."""
local = _identity()
opened = _session_open(
_identity(kv_dtype="float16"),
shard_range=pb.ShardRange(start_layer=0, end_layer=5, effective_start_layer=0),
)
mismatches = check_session_open(local, opened)
assert mismatches
assert handshake_error(mismatches).code == pb.ERROR_CODE_FINGERPRINT_MISMATCH
def test_grpc_handshake_uses_the_dgr_002_fingerprint_and_fails_closed():
@@ -267,4 +783,4 @@ def test_grpc_handshake_uses_the_dgr_002_fingerprint_and_fails_closed():
mismatches = check_handshake(local, remote)
assert mismatches
assert handshake_error(mismatches).code != 0
assert handshake_error(mismatches).code == pb.ERROR_CODE_FINGERPRINT_MISMATCH

4
tests/test_settlement_loop.py
View File

@@ -5,9 +5,7 @@ before the transaction is sent, unconfirmed batches resent by settlement id
(never double-paying), banned wallets skipped, history queryable over HTTP.
"""
import json
import time
import urllib.request
import pytest
@@ -68,7 +66,7 @@ def test_threshold_triggers_payout_and_zeroes_pending():
ledger.charge_request("client", MODEL, 1000, [("wallet-a", 12)]) # 0.018 pending
treasury = _FakePayoutTreasury()
tracker = _make_tracker(ledger, treasury, threshold=0.01)
port = tracker.start()
tracker.start()
try:
assert _wait_for(lambda: treasury.batches)
assert treasury.batches[0] == [("wallet-a", pytest.approx(0.018))]

4
tests/test_tracker_consensus.py
View File

@@ -157,7 +157,7 @@ def test_registration_on_follower_visible_on_all_nodes(three_tracker_cluster):
_wait_until_follower_knows_leader(follower, timeout=2.0)
# Register via a follower
node_id = _register_node(follower, port_hint=19999)
_register_node(follower, port_hint=19999)
# Allow replication to propagate (Raft heartbeat interval is 50ms)
time.sleep(0.5)
@@ -223,7 +223,7 @@ def test_registration_on_leader_visible_to_all(three_tracker_cluster):
urls = list(urls)
leader_url, followers = _wait_for_leader(urls, timeout=1.0)
node_id = _register_node(leader_url, port_hint=19996)
_register_node(leader_url, port_hint=19996)
# Allow Raft heartbeat to replicate the entry
time.sleep(0.3)

3
tests/test_tracker_routing.py
View File

@@ -19,7 +19,6 @@ from meshnet_tracker.server import (
TrackerServer,
_NodeEntry,
_available_quantizations,
_memory_pool_map,
_rebalance_all_locked,
_registration_ban_error,
_scale_demanded_models_locked,
@@ -1448,7 +1447,7 @@ def test_tracker_pool_join_adds_redundant_copy_without_splitting_incumbent():
"vram_bytes": 10_000, "ram_bytes": 10_000, "quantizations": ["bfloat16"],
"benchmark_tokens_per_sec": 1.0, "hardware_profile": {}, "score": 1.0},
)
second = _post_json(
_post_json(
f"http://127.0.0.1:{tracker_port}/v1/nodes/register",
{"endpoint": "http://127.0.0.1:9016", "model": "tiny-model",
"vram_bytes": 10_000, "ram_bytes": 10_000, "quantizations": ["bfloat16"],