feat: checkpoint distributed gguf runtime stories

.scratch/distributed-gguf-runtime/evidence/DGR-006/README.md

@@ -0,0 +1,203 @@
+# DGR-006 — Architecture-defined boundary input/output: evidence
+
+Status: done
+Date: 2026-07-15
+Evidence kind: **synthetic-unit** (pure-numpy dense-Llama reference + boundary
+contract). No model download, no GPU, no torch, no network, no API credit.
+
+## Summary
+
+Implemented the architecture-defined boundary contract that lets disjoint Shard
+processes reproduce whole-model execution (ADR-0024, RALPH runtime decisions #1,
+#6, #13). A public-network Shard is a contiguous inclusive layer range, and this
+story defines exactly what boundary state each range consumes and emits:
+
+- The **head** owns token embedding: it accepts token IDs and produces the
+  residual stream. It refuses an upstream boundary bundle.
+- **Middle and tail** ranges bypass token embedding entirely and accept the
+  named boundary bundle (the residual stream). They refuse token IDs.
+- A **non-tail** range emits the *unnormalized* architecture-defined residual —
+  before the final norm, before the LM head, and before any tail-only row
+  pruning — with every sequence position row intact.
+- The **tail** owns the final norm + LM head, prunes to the final row, and emits
+  a token through an explicit `SamplingContract` (greedy, deterministic).
+- The adapter **fails closed** for uncertified architectures: only certified
+  dense-Llama spellings are accepted; Qwen3/Qwen3-MoE/Mixtral/gpt2/empty all
+  raise `UncertifiedArchitectureError`.
+
+The adapter is backend-agnostic: it drives a duck-typed `ShardComputation`
+(`architecture_adapter`, `start_layer`, `end_layer`, `total_layers`,
+`embed_tokens`, `run_layers(hidden, *, positions)`, `final_norm`, `lm_head`). A
+pure-numpy dense-Llama reference (RMSNorm + RoPE + SwiGLU) implements that
+protocol in the tests and proves whole-model versus two-range **and** three-range
+prefill + greedy-decode parity. torch/transformers are not installed in the
+default `.venv`, so a numpy reference is the only way to keep the parity gate
+deterministic, download-free, and GPU-free — the identical protocol will be
+satisfied by the pinned llama.cpp worker (DGR-008) and the PyTorch backend.
+
+No existing runtime code was modified — this story is purely additive (one new
+module + one new test module). A clean-tree reproduction (files moved aside)
+confirms the full-suite failure set is byte-identical with and without this work.
+
+## Files changed (all new)
+
+- `packages/node/meshnet_node/boundary_adapter.py` — the boundary contract:
+  - `certified_architecture()` / `is_certified_architecture()` and the certified
+    architecture registry (`ArchitectureBoundary`), fail-closed.
+  - `ShardRole` + `role_for_range()` (head/middle/tail/full).
+  - `BoundaryBundle` — the versioned named-tensor bundle carrying the unnormalized
+    residual + positions + seam `next_layer`; `pack()`/`unpack()` for a truly
+    disjoint-process round-trip and `named_tensor_fields()` mapping onto the
+    DGR-002 `NamedTensor` shape (name, shape, dtype, byte order, bytes).
+  - `SamplingContract` — explicit greedy sampling (fails closed on other modes).
+  - `TailOutput` — sampled token + pruned final-row logits + the sampling contract.
+  - `BoundaryAdapter` — enforces the per-role input/output rules and drives the
+    computation.
+- `tests/test_boundary_adapter.py` — pure-numpy dense-Llama reference model
+  (`_ReferenceDenseLlama`) and range shard (`_ReferenceShard`), plus 22 tests:
+  certification/fail-closed, role classification, input-side contract
+  (head-owns-embedding, middle/tail-bypass, seam-layer mismatch, normalized-bundle
+  rejection), output-side contract (unnormalized full-row boundary, tail pruning +
+  sampling), wire round-trip, and the parity gate.
+
+## Acceptance criteria → evidence
+
+- **Head accepts token IDs and owns token embedding** —
+  `test_head_accepts_token_ids_and_owns_embedding`,
+  `BoundaryAdapter._ingest_tokens` (head requires token IDs, refuses a bundle).
+- **Middle/tail bypass token embedding and accept the named boundary bundle** —
+  `test_middle_and_tail_bypass_embedding_and_require_the_bundle`,
+  `_ingest_boundary` (rejects token IDs, requires the bundle).
+- **Non-tail emits the unnormalized boundary before final norm/head and before
+  tail-only row pruning** — `test_non_tail_emits_unnormalized_full_row_boundary`
+  asserts the bundle is `normalized=False`, shape `(1, seq, hidden)` (all rows),
+  and byte-equal to the whole model's residual after the cut layer while *not*
+  equal to its normalized form. `_emit_boundary`.
+- **Tail emits logits/token through an explicit sampling contract** —
+  `test_tail_emits_pruned_logits_through_the_sampling_contract` (logits shape
+  `(1, vocab)` = pruned last row, greedy token = argmax). `_emit_tail`,
+  `SamplingContract`.
+- **Dense-Llama whole-model vs two-range prefill + greedy-decode parity within
+  tolerance** — `test_two_range_prefill_parity_matches_whole_model`,
+  `test_three_range_prefill_parity_exercises_the_middle_role`,
+  `test_two_range_greedy_decode_parity_matches_whole_model`,
+  `test_alias_architecture_still_parity_matches`. Documented tolerance:
+  next-token logits `np.allclose(..., atol=1e-6)` and **identical** greedy token
+  sequences. (The split is bit-exact in practice; the tolerance is a conservative
+  guard.)
+- **Fails closed for uncertified architectures** —
+  `test_uncertified_architectures_fail_closed`,
+  `test_adapter_construction_fails_closed_for_uncertified_backend`.
+- **Targeted pytest** — `22 passed`.
+- **compileall packages tests** — exit 0.
+- **git diff --check** — clean.
+- **Deterministic / download-free / credit-free / GPU-free** — pure numpy; fixed
+  RNG seed; no torch, no network, no model files.
+- **Full deterministic pytest** — `20 failed, 715 passed, 13 skipped, 12 errors`.
+  All 20 failures + 12 errors are pre-existing and unrelated (see below).
+- **Native C++ / CTest / llama.cpp patch stack** — **not touched by this story.**
+  The boundary contract is delivered at the Python adapter level with a numpy
+  parity proof; the equivalent native patches ("architecture-defined intermediate
+  input/output" and "intermediate output before final norm/head") are wired when
+  the standalone C++ worker exists in DGR-008. No native code, CMake, or llama.cpp
+  patch was modified, so those gates are N/A here (same as DGR-005).
+
+## Commands and real results
+
+```bash
+# Targeted tests
+python -m pytest -q tests/test_boundary_adapter.py
+# -> 22 passed in 0.26s
+
+# Python compile check
+python -m compileall -q packages tests
+# -> exit 0
+
+# Diff hygiene
+git diff --check
+# -> exit 0
+
+# Full deterministic suite (with DGR-006 files present)
+python -m pytest -q -rfE
+# -> 20 failed, 715 passed, 13 skipped, 12 errors in 239.77s
+
+# Clean-tree reproduction (DGR-006 files moved aside)
+mv packages/node/meshnet_node/boundary_adapter.py /tmp/ && mv tests/test_boundary_adapter.py /tmp/
+python -m pytest -q -rfE
+# -> 20 failed, 693 passed, 13 skipped, 12 errors in 243.10s
+# (693 = 715 - 22; failure/error SET is byte-identical -> DGR-006 introduced none)
+```
+
+The `commands.txt` and `results.json` beside this README capture the exact
+commands and the machine-readable failure set.
+
+## Pre-existing unrelated failures (full-suite)
+
+`pytest -q` on `ralph/distributed-gguf-runtime` reports 20 failures + 12 errors,
+none of which touch the boundary adapter. Moving the two DGR-006 files aside and
+re-running yields the **identical** failure/error set (only the passed count drops
+by exactly 22). Categories:
+
+- **12 errors — `tests/test_native_shard_protocol.py`:** generated protobuf code
+  expects a newer protobuf runtime than the one installed
+  (`ValidateProtobufRuntimeVersion` mismatch). Pre-existing; documented in the
+  DGR-002 / DGR-005 evidence.
+- **20 failures** across `test_activation_compression.py`,
+  `test_dynamic_routing.py`, `test_gossip_and_relay.py`,
+  `test_manual_route_benchmark.py`, `test_node_doctor.py`,
+  `test_openai_gateway.py` (`langchain` optional dep),
+  `test_toploc_calibration_dispatch.py`, `test_tracker_capability_admission.py`,
+  `test_tracker_control_plane.py`, `test_tracker_routing.py` — tracker/routing/
+  benchmark/socket-bind + optional-dependency failures that exist on the branch
+  independent of this story.
+
+## Limitations and deferred work
+
+- **Numpy reference, not real weights.** The parity gate uses a deterministic
+  numpy dense-Llama, not a downloaded GGUF/safetensors model. Real-model parity on
+  a downloaded dense-Llama (CPU/ROCm) belongs to DGR-010 with
+  `MESHNET_ENABLE_REAL_INFERENCE_TESTS=1` and `.venv-rocm`.
+- **Stateless decode for parity.** Greedy-decode parity recomputes the growing
+  prefix statelessly (no KV reuse). Local Hot KV State + session isolation is
+  DGR-007; the boundary contract here is KV-agnostic.
+- **Native patch wiring deferred.** The C++/llama.cpp expression of this boundary
+  (range-aware intermediate I/O, pre-final-norm output) is implemented in the
+  standalone worker (DGR-008) against this same contract; no native code was
+  touched here.
+- **Greedy-only sampling certified.** `SamplingContract` declares temperature /
+  top-p fields but only certifies `greedy` (deterministic). Stochastic sampling is
+  out of scope for the deterministic parity gate.
+
+## Compatibility / migration notes
+
+- `BOUNDARY_SCHEMA_VERSION = 1` matches `runtime_recipe.RuntimeRecipeIdentity`'s
+  `boundary_schema_version`. A receiver rejects a bundle whose schema, architecture
+  adapter, tensor name, normalization flag, or seam `next_layer` does not match its
+  own range — no silent reinterpretation.
+- `BoundaryBundle.named_tensor_fields()` returns exactly the DGR-002 `NamedTensor`
+  fields (name, shape, dtype, byte order, bytes), so DGR-008 can serialize the seam
+  into the gRPC `TensorBundle` without re-deriving them.
+- Certified architecture ids are canonicalized: `dense-llama` / `dense_llama` /
+  `llama` / `LlamaForCausalLM` / `LlamaModel` all map to the one `dense-llama`
+  adapter. Adding an architecture requires a new certified entry, never a tensor
+  guess (Qwen3 is DGR-015).
+
+## Handoff for dependent stories
+
+- **DGR-007 (Hot KV State):** wrap the same `ShardComputation` so `run_layers`
+  consumes/produces per-session KV; the boundary contract (unnormalized residual,
+  seam `next_layer`, tail pruning) is unchanged. The bundle's `positions` field is
+  the per-token position vector a KV path needs.
+- **DGR-008 (C++ gRPC worker):** implement the `ShardRuntime` servicer against
+  this contract. Map `BoundaryBundle.named_tensor_fields()` → protobuf
+  `NamedTensor`; enforce the same head-embeds / middle-tail-bypass /
+  non-tail-unnormalized / tail-samples rules in native code; expose
+  `certified_architecture` gating so uncertified GGUFs are refused before activation.
+- **DGR-009 (Meshnet integration):** carry `BoundaryBundle.pack()` payloads as
+  opaque relay frames; the seam `next_layer` is the overlap-safe effective start
+  the route must honor.
+- **DGR-010 (real two-process acceptance):** reuse the parity harness shape
+  (whole vs N-range, identical greedy tokens) against a real downloaded dense-Llama
+  under `.venv-rocm`.
+- **DGR-015 (Qwen3 adapter):** add a certified `ArchitectureBoundary` entry only
+  after real certification; today Qwen3 fails closed by design.

.scratch/distributed-gguf-runtime/evidence/DGR-006/commands.txt

@@ -0,0 +1,26 @@
+# DGR-006 exact commands (run from repo worktree root)
+
+# Targeted boundary-adapter tests
+python -m pytest -q tests/test_boundary_adapter.py
+# -> 22 passed in 0.26s
+
+# Python compile check for changed Python
+python -m compileall -q packages tests
+# -> exit 0
+
+# Diff hygiene
+git diff --check
+# -> exit 0
+
+# Full deterministic suite with DGR-006 files present
+python -m pytest -q -rfE
+# -> 20 failed, 715 passed, 13 skipped, 12 errors in 239.77s
+
+# Clean-tree reproduction: move the two new DGR-006 files aside, re-run
+mv packages/node/meshnet_node/boundary_adapter.py /tmp/dgr006_boundary_adapter.py
+mv tests/test_boundary_adapter.py /tmp/dgr006_test_boundary_adapter.py
+python -m pytest -q -rfE
+# -> 20 failed, 693 passed, 13 skipped, 12 errors in 243.10s
+# (693 = 715 - 22; failure/error set byte-identical to the with-files run)
+mv /tmp/dgr006_boundary_adapter.py packages/node/meshnet_node/boundary_adapter.py
+mv /tmp/dgr006_test_boundary_adapter.py tests/test_boundary_adapter.py

.scratch/distributed-gguf-runtime/evidence/DGR-006/results.json

@@ -0,0 +1,161 @@
+{
+  "story": "DGR-006",
+  "date": "2026-07-15",
+  "evidence_kind": "synthetic-unit (pure-numpy dense-Llama parity + boundary contract)",
+  "targeted_tests": {
+    "file": "tests/test_boundary_adapter.py",
+    "result": "22 passed"
+  },
+  "compileall": "exit 0",
+  "git_diff_check": "clean",
+  "parity_tolerance": {
+    "logits_atol": 1e-06,
+    "greedy_tokens": "identical"
+  },
+  "full_suite_with_files": {
+    "failed": 20,
+    "passed": 715,
+    "skipped": 13,
+    "errors": 12,
+    "seconds": 239.77
+  },
+  "full_suite_clean_tree": {
+    "failed": 20,
+    "passed": 693,
+    "skipped": 13,
+    "errors": 12,
+    "seconds": 243.1,
+    "note": "693 = 715 - 22 DGR-006 tests; failure/error set identical"
+  },
+  "failure_set_identical_with_and_without_dgr006": true,
+  "preexisting_unrelated_failures": [
+    {
+      "kind": "ERROR",
+      "nodeid": "tests/test_native_shard_protocol.py::test_capability_and_health_round_trip"
+    },
+    {
+      "kind": "ERROR",
+      "nodeid": "tests/test_native_shard_protocol.py::test_checksum_algorithms_verify"
+    },
+    {
+      "kind": "ERROR",
+      "nodeid": "tests/test_native_shard_protocol.py::test_cross_language_roundtrip_python_and_cpp"
+    },
+    {
+      "kind": "ERROR",
+      "nodeid": "tests/test_native_shard_protocol.py::test_defaults_are_stable_for_backward_compatibility"
+    },
+    {
+      "kind": "ERROR",
+      "nodeid": "tests/test_native_shard_protocol.py::test_fragment_and_reassemble_round_trip_with_checksums"
+    },
+    {
+      "kind": "ERROR",
+      "nodeid": "tests/test_native_shard_protocol.py::test_message_header_carries_every_required_field"
+    },
+    {
+      "kind": "ERROR",
+      "nodeid": "tests/test_native_shard_protocol.py::test_named_tensor_bundle_describes_shape_dtype_byteorder_and_fragments"
+    },
+    {
+      "kind": "ERROR",
+      "nodeid": "tests/test_native_shard_protocol.py::test_reassemble_detects_fragment_corruption"
+    },
+    {
+      "kind": "ERROR",
+      "nodeid": "tests/test_native_shard_protocol.py::test_service_descriptor_exposes_all_operations"
+    },
+    {
+      "kind": "ERROR",
+      "nodeid": "tests/test_native_shard_protocol.py::test_session_response_carries_structured_status_and_results"
+    },
+    {
+      "kind": "ERROR",
+      "nodeid": "tests/test_native_shard_protocol.py::test_session_stream_carries_open_prefill_decode_release_cancel"
+    },
+    {
+      "kind": "ERROR",
+      "nodeid": "tests/test_native_shard_protocol.py::test_unknown_fields_are_preserved_for_forward_compatibility"
+    },
+    {
+      "kind": "FAILED",
+      "nodeid": "tests/test_activation_compression.py::test_compressible_body_uses_zstd_when_it_clears_savings_policy"
+    },
+    {
+      "kind": "FAILED",
+      "nodeid": "tests/test_activation_compression.py::test_incompressible_body_stays_raw_after_measured_trial"
+    },
+    {
+      "kind": "FAILED",
+      "nodeid": "tests/test_activation_compression.py::test_malformed_zstd_and_legacy_raw_bodies_are_handled_explicitly"
+    },
+    {
+      "kind": "FAILED",
+      "nodeid": "tests/test_activation_compression.py::test_threshold_requires_both_byte_and_ratio_savings"
+    },
+    {
+      "kind": "FAILED",
+      "nodeid": "tests/test_dynamic_routing.py::test_admin_can_replace_a_served_model_and_release_it"
+    },
+    {
+      "kind": "FAILED",
+      "nodeid": "tests/test_gossip_and_relay.py::test_activation_compression_round_trips_and_skips_small_bodies"
+    },
+    {
+      "kind": "FAILED",
+      "nodeid": "tests/test_manual_route_benchmark.py::test_benchmark_records_one_and_two_node_routes"
+    },
+    {
+      "kind": "FAILED",
+      "nodeid": "tests/test_manual_route_benchmark.py::test_clients_without_route_are_unaffected"
+    },
+    {
+      "kind": "FAILED",
+      "nodeid": "tests/test_manual_route_benchmark.py::test_invalid_route_shape_is_400"
+    },
+    {
+      "kind": "FAILED",
+      "nodeid": "tests/test_manual_route_benchmark.py::test_pinned_route_uses_named_node"
+    },
+    {
+      "kind": "FAILED",
+      "nodeid": "tests/test_manual_route_benchmark.py::test_unknown_route_node_is_400"
+    },
+    {
+      "kind": "FAILED",
+      "nodeid": "tests/test_node_doctor.py::test_cli_doctor_flags_select_what_is_validated"
+    },
+    {
+      "kind": "FAILED",
+      "nodeid": "tests/test_openai_gateway.py::test_langchain_chat_openai"
+    },
+    {
+      "kind": "FAILED",
+      "nodeid": "tests/test_toploc_calibration_dispatch.py::test_calibration_run_dispatches_only_solo_capable_nodes"
+    },
+    {
+      "kind": "FAILED",
+      "nodeid": "tests/test_toploc_calibration_dispatch.py::test_calibration_run_node_without_commitment_endpoint_is_skipped_not_failed"
+    },
+    {
+      "kind": "FAILED",
+      "nodeid": "tests/test_toploc_calibration_dispatch.py::test_calibration_run_persists_corpus_and_results_endpoint_reports_it"
+    },
+    {
+      "kind": "FAILED",
+      "nodeid": "tests/test_tracker_capability_admission.py::test_an_enforcing_tracker_never_routes_a_node_whose_proof_does_not_cover_it[invalid]"
+    },
+    {
+      "kind": "FAILED",
+      "nodeid": "tests/test_tracker_control_plane.py::test_tracker_startup_does_not_import_or_load_model_backends"
+    },
+    {
+      "kind": "FAILED",
+      "nodeid": "tests/test_tracker_routing.py::test_shard_heal_cycle_surviving_node_covers_dead_peers_gap"
+    },
+    {
+      "kind": "FAILED",
+      "nodeid": "tests/test_tracker_routing.py::test_torch_node_applies_tracker_load_shard_directive"
+    }
+  ]
+}