popov/neuron-tai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: checkpoint distributed gguf runtime stories

Browse Source
This commit is contained in:
Dobromir Popov
2026-07-15 23:42:58 +03:00
parent eaf00f6add
commit 1fe31ef38d
60 changed files with 8478 additions and 105 deletions
Download Patch File Download Diff File Expand all files Collapse all files

176
.scratch/distributed-gguf-runtime/evidence/DGR-002/README.md Normal file
View File

@@ -0,0 +1,176 @@
# DGR-002 — Versioned gRPC Shard protocol: evidence
Status: done
Date: 2026-07-15
Evidence kind: **synthetic-unit** (schema round-trip + cross-language protobuf
compatibility). No model download, no GPU, no network, no API credits.
## Summary
Added the versioned Protocol Buffers schema that is the semantic contract between
Python and C++ Shards (ADR-0024), plus reproducible Python and C++ code
generation/build wiring and generated-schema round-trip + compatibility tests in
**both** languages. The schema defines one long-lived bidirectional gRPC stream
per Route Session Activation Seam, bounded prefill chunking, a small decode fast
path, and a versioned named-tensor bundle carrying every required identifier.
No existing runtime code was modified — this story is purely additive (a new
`.proto`, a `native_protocol` loader package, C++ build wiring, and one new test
module). Generated stubs are produced on demand into gitignored `build/`
directories, so nothing generated is committed.
## Files changed (all new)
- `packages/node/native/proto/shard_runtime.proto` — the schema (package
`meshnet.shard.v1`, proto3). Service `ShardRuntime` with `GetCapability`,
`Health`, `ActivateSession` (bidi stream), `Release`, `Cancel`.
- `packages/node/meshnet_node/native_protocol/__init__.py` — reproducible
on-demand `grpc_tools.protoc` codegen + loader (`load()`, `load_grpc()`) and
shared bundle helpers (`compute_checksum`, `verify_checksum`, `fragment_tensor`,
`reassemble_tensor`).
- `packages/node/native/scripts/generate_python.py` — standalone reproducible
Python generation (self-contained; does not import `meshnet_node`).
- `packages/node/native/scripts/generate_cpp.sh` — reproducible C++ generation
(message stubs always; gRPC service stubs when `grpc_cpp_plugin` is present).
- `packages/node/native/CMakeLists.txt` — C++ build wiring; works with both
CONFIG-mode (`protobuf::libprotobuf`/`protobuf::protoc`) and CMake's
`FindProtobuf` module.
- `packages/node/native/tests/roundtrip_test.cpp` — C++ round-trip / compat test
(`--selftest`, `--read`, `--write`).
- `tests/test_native_shard_protocol.py` — Python round-trip + compatibility tests
and the Python↔C++ cross-language driver.
## Acceptance criteria → evidence
- **Capability/health/session-stream/release/cancellation schema** — the
`ShardRuntime` service's five RPCs; `test_capability_and_health_round_trip`,
`test_session_stream_carries_open_prefill_decode_release_cancel`.
- **One long-lived bidi stream per Activation Seam with deadlines, cancellation,
flow control, structured errors** — `rpc ActivateSession (stream ...) returns
(stream ...)`. Deadlines: gRPC call deadline on direct transport, plus
`SessionOpen.deadline_unix_nanos` for relay-carried frames. Cancellation:
`Cancel` RPC and in-stream `CancelRequest`/`PHASE_CANCEL`. Flow control:
`FlowControl` frames (credits + in-flight byte/message caps). Structured errors:
`Status` (canonical code, message, `RetryClass`, details). Verified by
`test_session_response_carries_structured_status_and_results`.
- **Bounded prefill chunking + small decode fast path** — `PrefillChunk`
(`chunk_index`/`chunk_count`/`final_chunk`, `SessionOpen.max_prefill_tokens_per_chunk`)
and `DecodeStep` (minimal single-bundle path). Bounded fragments via
`SessionOpen.max_fragment_bytes` and `fragment_tensor(...)`.
- **Carries schema version, work ID, Route Session ID, route epoch,
artifact/recipe fingerprint, shard range/effective start, phase, position,
idempotency step, cache expectation, compression, checksum** — all on
`MessageHeader` (+ `ArtifactFingerprint.runtime_recipe_fingerprint`,
`ShardRange.effective_start_layer`). Verified field-by-field by
`test_message_header_carries_every_required_field`.
- **Versioned named-tensor bundle (name, shape, dtype, byte order, fragments)** —
`TensorBundle`/`NamedTensor`/`TensorFragment`;
`test_named_tensor_bundle_describes_shape_dtype_byteorder_and_fragments`,
`test_fragment_and_reassemble_round_trip_with_checksums`.
- **Round-trip + compatibility tests in Python and C++** — Python:
`tests/test_native_shard_protocol.py` (11 tests). C++: `roundtrip_test.cpp`
built via CMake; cross-language driver `test_cross_language_roundtrip_python_and_cpp`
exercises Python→C++ and C++→Python in both directions.
- **Targeted pytest** — `11 passed, 1 skipped` (default env); `12 passed` with the
C++ toolchain on PATH.
- **compileall packages tests** — exit 0.
- **git diff --check** — clean.
- **Deterministic / download-free / credit-free / GPU-free** — all tests are pure
protobuf serialization; the C++ path uses only local compilers.
- **Full deterministic pytest** — `704 passed, 14 skipped, 11 failed`. The 11
failures are pre-existing and unrelated (see below).
## Commands and real results
See `commands.txt` for the exact command list. Key results:
- `python packages/node/native/scripts/generate_python.py` →
`shard_runtime_pb2.py: ok`, `shard_runtime_pb2_grpc.py: ok`.
- `pytest tests/test_native_shard_protocol.py -q` → **11 passed, 1 skipped**
(skip reason: `C++ toolchain unavailable: cmake not found on PATH`).
- With `/tmp/pbsrc/install/bin` (protoc 33.1) and `.venv/bin` (cmake) on PATH and
`CMAKE_PREFIX_PATH=/tmp/pbsrc/install`:
- `generate_cpp.sh` → `shard_runtime.pb.cc`, `shard_runtime.pb.h`
(grpc service stubs skipped: `grpc_cpp_plugin` absent).
- `cmake -S ... -B ...` + `cmake --build ...` → build OK.
- `shard_protocol_roundtrip_test --selftest` → `selftest ok (128 bytes)`, exit 0.
- `ctest` → `1/1 Test #1: shard_protocol_roundtrip ... Passed`.
- `pytest ...::test_cross_language_roundtrip_python_and_cpp -q` → **1 passed**
(Python serializes → C++ parses & verifies → C++ serializes → Python parses
& verifies).
- `compileall -q packages tests` → exit 0.
- `git diff --check` → clean.
## Pre-existing unrelated failures (full-suite)
`pytest -q` on the full tree reports 11 failures, all in tracker routing /
dynamic routing / manual route benchmark / toploc calibration — none import the
Shard protocol. Clean-tree reproduction: with **all DGR-002 files moved aside**
(`git status` shows only the pre-existing `.ralph-tui/config.toml` deletion),
re-running exactly these tests gives `11 failed, 3 passed` — identical failures.
They exist on the `ralph/distributed-gguf-runtime` branch independent of this
story. The full list is in `results.json.preexisting_unrelated_failures`.
Note: the earlier `progress.md` (RCR-001, on master) recorded a different set of
6 optional-dependency failures (zstandard, langchain_openai). Those did **not**
recur here; this environment has those deps. The 11 above are branch-local
routing/benchmark failures, not environmental.
## Limitations and deferred work
- **C++ toolchain is host-provided, not vendored.** The default test env has no
`protoc`/`cmake`/protobuf C++ headers on PATH, so the C++ cross-language test
**skips** by default (explicit skip reason). It was executed for this evidence
using an ephemeral from-source protobuf 33.1 install at `/tmp/pbsrc/install`
plus the `.venv` cmake. DGR-004/DGR-008 should pin the C++ protobuf/gRPC
toolchain (upstream commit + reproducible fetch/build) so this test runs in CI
without relying on an ad-hoc `/tmp` install.
- **gRPC C++ service stubs not built here.** `grpc_cpp_plugin` is absent, so
`generate_cpp.sh` produced message stubs only. The round-trip test needs only
message serialization; the service stubs are DGR-008's concern.
- **No live gRPC transport yet.** This story delivers the schema + serialization
contract and generation/build wiring only. Channel setup, the bidi stream
server/client, deadlines/cancellation propagation over a real HTTP/2 channel,
and relay framing are DGR-008/DGR-009.
- **Protobuf runtime version skew.** Python runtime is pip protobuf 7.35.1; the
C++ side used protoc 33.1. Protobuf wire format is stable across these, and the
cross-language round-trip confirms interop; version pinning is deferred to the
toolchain-pinning stories.
## Compatibility / migration notes
- proto3 with a 0-valued `*_UNSPECIFIED` member on every enum and never-reused
field numbers. Forward compatibility (unknown-field preservation) is verified
behaviourally by `test_unknown_fields_are_preserved_for_forward_compatibility`
— note protobuf 7.x's upb backend does not implement the `UnknownFields()`
introspection accessor, so the test asserts the observable re-serialization
outcome instead. Backward defaults verified by
`test_defaults_are_stable_for_backward_compatibility`.
- Wire schema version is `SchemaVersion.SCHEMA_VERSION_1` (int 1), also exposed as
`meshnet_node.native_protocol.SCHEMA_VERSION`.
## Handoff for dependent stories
- **DGR-003 (recipe/fingerprint):** populate `ArtifactFingerprint`
(`model_id`, `revision`, `artifact_hash`, `quantization`,
`runtime_recipe_fingerprint`). Admission compares these before activation; a
mismatch is a fatal `Status` (`RetryClass.RETRY_CLASS_FATAL`).
- **DGR-004 (llama.cpp pin) / DGR-008 (C++ worker):** pin the C++
protobuf + gRPC toolchain and add `grpc_cpp_plugin`; then `generate_cpp.sh`
emits service stubs and the CMake target can link gRPC. Implement the
`ShardRuntime` servicer; map `(route_session_id, route_epoch)` to an isolated
llama sequence. Use `SessionOpen` for stream-scoped bounds and `FlowControl`
for backpressure.
- **DGR-009 (Meshnet integration/relay):** the relay may carry serialized
`SessionActivation`/`SessionResponse` frames as opaque binary; use the in-message
`deadline_unix_nanos`, `CancelRequest`, and `FlowControl` since gRPC call
metadata is lost over relay.
- **Loader usage:** `from meshnet_node import native_protocol as proto;
pb2 = proto.load()`. Stubs regenerate automatically when the `.proto` changes
(mtime check). `proto.load_grpc()` returns the service stubs (needs the `grpc`
runtime).
- **Gotcha:** the `.venv` installs the meshnet packages editable via a PEP 660
meta-path finder pointing at the **main** checkout. Import the worktree copy by
ensuring the worktree `packages/node` is on `sys.path` first (conftest already
does this for pytest); standalone tooling must derive paths from `__file__` and
not `import meshnet_node` (why `generate_python.py` is self-contained).

40
.scratch/distributed-gguf-runtime/evidence/DGR-002/commands.txt Normal file
View File

@@ -0,0 +1,40 @@
# DGR-002 reproduction commands (run from repo root, project .venv = Python 3.14).
# 1. Generate Python stubs (reproducible; writes to gitignored build/ dir).
.venv/bin/python packages/node/native/scripts/generate_python.py
# 2. Python round-trip + compatibility tests (default env; C++ test skips if
# cmake/protoc absent).
.venv/bin/python -m pytest tests/test_native_shard_protocol.py -q
# => 11 passed, 1 skipped
# 3. Quality gates.
.venv/bin/python -m compileall -q packages tests # exit 0
git diff --check # clean
# 4. Full deterministic suite (records pre-existing unrelated failures).
.venv/bin/python -m pytest -q
# => 704 passed, 14 skipped, 11 failed (all pre-existing, unrelated; see below)
# 5. Clean-tree reproduction of the 11 pre-existing failures (DGR-002 files moved
# aside): same 11 fail => not caused by this story.
# --- C++ / cross-language (requires protoc + protobuf C++ dev + cmake) --------
# On this host a from-source protobuf 33.1 toolchain lives under /tmp/pbsrc/install
# and cmake ships in the .venv. To execute the C++ test instead of skipping it:
export PATH="/tmp/pbsrc/install/bin:$PWD/.venv/bin:$PATH"
export CMAKE_PREFIX_PATH="/tmp/pbsrc/install:$CMAKE_PREFIX_PATH"
# 6. Generate C++ stubs (message stubs always; gRPC service stubs if
# grpc_cpp_plugin present).
packages/node/native/scripts/generate_cpp.sh
# 7. Standalone C++ build + selftest + ctest.
cmake -S packages/node/native -B packages/node/native/build/cpp
cmake --build packages/node/native/build/cpp --target shard_protocol_roundtrip_test
packages/node/native/build/cpp/shard_protocol_roundtrip_test --selftest # "selftest ok (128 bytes)"
(cd packages/node/native/build/cpp && ctest --output-on-failure) # 1/1 passed
# 8. Cross-language Python<->C++ round-trip via the pytest driver (now runs, not skips).
.venv/bin/python -m pytest tests/test_native_shard_protocol.py::test_cross_language_roundtrip_python_and_cpp -q
# => 1 passed

63
.scratch/distributed-gguf-runtime/evidence/DGR-002/results.json Normal file
View File

@@ -0,0 +1,63 @@
{
"task": "DGR-002",
"title": "Adopt the versioned gRPC Shard protocol",
"schema": {
"proto": "packages/node/native/proto/shard_runtime.proto",
"package": "meshnet.shard.v1",
"syntax": "proto3",
"schema_version": 1,
"service": "ShardRuntime",
"rpcs": ["GetCapability", "Health", "ActivateSession", "Release", "Cancel"],
"streaming_seam": "ActivateSession (bidirectional stream)"
},
"toolchain": {
"python": "3.14.6",
"protobuf_runtime_python": "7.35.1",
"grpcio": "1.82.1",
"grpcio_tools": "1.82.1",
"cpp_protoc": "libprotoc 33.1",
"cpp_protobuf_toolchain": "/tmp/pbsrc/install (from-source protobuf 33.1, ephemeral host build)",
"cmake": "4.4.0 (.venv)",
"cxx": "g++ (system)"
},
"generation": {
"python_cmd": "python packages/node/native/scripts/generate_python.py",
"python_out": "packages/node/native/build/python/shard_runtime_pb2{,_grpc}.py (gitignored)",
"cpp_cmd": "packages/node/native/scripts/generate_cpp.sh",
"cpp_out": "packages/node/native/build/cpp-gen/shard_runtime.pb.{h,cc} (gitignored)",
"cpp_build": "cmake -S packages/node/native -B <build> && cmake --build <build>"
},
"tests": {
"python_default_env": {"passed": 11, "skipped": 1, "note": "C++ cross-language test skips when cmake/protoc absent"},
"python_with_cpp_toolchain": {"passed": 12, "skipped": 0},
"cpp_selftest_bytes": 128,
"cpp_ctest": "1/1 passed",
"cross_language": "Python->C++ and C++->Python round-trip verified in both directions"
},
"quality_gates": {
"targeted_pytest": "11 passed, 1 skipped (default); 12 passed with C++ toolchain",
"compileall_packages_tests": "exit 0",
"git_diff_check": "clean",
"full_pytest": {
"passed": 704,
"skipped": 14,
"failed": 11,
"failed_are_preexisting_unrelated": true,
"clean_tree_reproduction": "same 11 fail with all DGR-002 files removed (11 failed, 3 passed)"
}
},
"preexisting_unrelated_failures": [
"tests/test_dynamic_routing.py::test_admin_can_replace_a_served_model_and_release_it",
"tests/test_manual_route_benchmark.py::test_pinned_route_uses_named_node",
"tests/test_manual_route_benchmark.py::test_unknown_route_node_is_400",
"tests/test_manual_route_benchmark.py::test_invalid_route_shape_is_400",
"tests/test_manual_route_benchmark.py::test_clients_without_route_are_unaffected",
"tests/test_manual_route_benchmark.py::test_benchmark_records_one_and_two_node_routes",
"tests/test_toploc_calibration_dispatch.py::test_calibration_run_dispatches_only_solo_capable_nodes",
"tests/test_toploc_calibration_dispatch.py::test_calibration_run_persists_corpus_and_results_endpoint_reports_it",
"tests/test_toploc_calibration_dispatch.py::test_calibration_run_node_without_commitment_endpoint_is_skipped_not_failed",
"tests/test_tracker_routing.py::test_torch_node_applies_tracker_load_shard_directive",
"tests/test_tracker_routing.py::test_shard_heal_cycle_surviving_node_covers_dead_peers_gap"
],
"evidence_kind": "synthetic-unit (schema round-trip + cross-language protobuf; no model, no GPU, no network, no API credits)"
}

86
.scratch/distributed-gguf-runtime/evidence/DGR-003/README.md Normal file
View File

@@ -0,0 +1,86 @@
# DGR-003 — Exact artifact and runtime-recipe identity: evidence
Status: done
Date: 2026-07-15
Evidence kind: **synthetic-unit + repo checks**. No model download, no GPU, no network, no API credits.
## Summary
Implemented exact identity plumbing for shard admission so the node and tracker
compare the same compatibility contract:
- `ArtifactIdentity` binds a shard to an exact source model artifact hash plus
shard range.
- `RuntimeRecipeIdentity` separates weight quantization, activation dtype,
compute dtype, KV dtype/layout, tokenizer revision, architecture adapter,
backend id, runtime version, boundary schema version, and cache layout.
- `compatibility_fingerprint` is stable SHA-256 over the full artifact/runtime
recipe payload.
- Node admission and tracker admission now fail closed on compatibility
mismatches.
- Unsupported recipes remain tracked as dark/unadmitted until a real forward
proves them.
The work also keeps the test helper, doctor path, startup registration payloads,
and tracker storage/admission aligned so the same fingerprint is emitted and
checked across the system.
## Files changed
- `packages/node/meshnet_node/runtime_recipe.py` - new exact artifact/runtime
identity helpers and fingerprint builder.
- `packages/node/meshnet_node/capability.py` - capability report shape now
carries artifact/runtime recipe identity and validates the top-level
compatibility fingerprint.
- `packages/node/meshnet_node/admission.py` - fail-closed admission on
compatibility fingerprint mismatch.
- `packages/node/meshnet_node/doctor.py` - production capability reports now
include the runtime recipe identity.
- `packages/node/meshnet_node/testing.py` - test report builder now mirrors the
production fingerprint fields.
- `packages/node/meshnet_node/startup.py` - registration payload now includes
the compatibility fingerprint.
- `packages/tracker/meshnet_tracker/capability.py` - tracker verdict state now
stores artifact hash and compatibility fingerprints.
- `packages/tracker/meshnet_tracker/server.py` - registration and raft state now
preserve declared compatibility fingerprints.
- `tests/test_node_capability.py` - identity shape and fingerprint regression
tests.
- `tests/test_node_admission.py` - fail-closed admission regression tests.
- `tests/test_tracker_capability_admission.py` - tracker compatibility mismatch
regression tests.
## Commands and real results
- `python -m compileall packages tests` -> exit 0.
- `pytest -q tests/test_node_capability.py` -> `48 passed in 0.09s`.
- `pytest -q tests/test_node_admission.py` -> `20 passed in 0.11s`.
- `pytest -q tests/test_tracker_capability_admission.py -k 'compatibility_mismatch or older_recipe_catalogue or unparseable_catalogue_version or future_dated or unknown_schema_version or malformed_report or recorded_detail_carries_no_credentials or compat_policy_routes_a_legacy_node_but_never_a_broken_proof or policy_is_read_from_the_environment_and_defaults_to_compat or route_selection_drops_every_unadmitted_candidate_under_enforce or node_reassigned_to_a_shard_it_never_proved_stops_routing or admitted_candidates_keep_coverage_first_and_throughput_routing'` -> `18 passed, 17 deselected in 0.11s`.
- `git diff --check` -> exit 0.
- `pytest -q` -> not green in this sandbox. Final result: `210 failed, 423 passed, 13 skipped, 14 warnings, 86 errors in 131.34s`.
## Limitation
The full suite is dominated by tracker and HTTP/socket-backed tests. In this
sandbox, those fail with `PermissionError: [Errno 1] Operation not permitted`
when the tracker attempts to bind a socket. That is an environment restriction,
not a regression from the identity work. The pure unit slices above pass.
## Compatibility notes
- The compatibility fingerprint is now a hash over the exact artifact identity
and runtime recipe payload. It is intended for both node admission and the
gRPC handshake admission path.
- Default fallbacks for fake/test backends are stable and deterministic: cache
layout derives from KV-cache support, architecture adapter falls back to the
backend id, and tokenizer identity prefers model revision/model id rather than
local tokenizer paths.
## Handoff for dependent stories
- DGR-004 / DGR-008 can reuse `runtime_recipe.py` and the compatibility
fingerprint to gate the gRPC handshake before session activation.
- DGR-009 should transmit the same fingerprint over the relay or preserve it in
frame metadata so admission stays aligned end to end.
- Any future recipe expansion should register unsupported recipes as dark until
a real distributed forward certifies them.

130
.scratch/distributed-gguf-runtime/evidence/DGR-004/README.md Normal file
View File

@@ -0,0 +1,130 @@
# DGR-004 — reproducible pinned llama.cpp patch stack evidence
Status: done
Date: 2026-07-15
Evidence kind: **synthetic-build + repo checks**. No model download, no GPU,
no network fetch during validation, no API credits.
## Summary
Implemented the reproducible source-dependency boundary for llama.cpp and kept
the fork seam narrow and auditable:
- exact pinned upstream commit and repository metadata
- numbered patch stack isolated under `packages/node/native/llama/patches/`
- build script that verifies the pin, applies the patch stack, stages notices,
and compiles a standalone worker scaffold without manual source copying
- upstream file assumptions and fail-closed pin checking
- license/attribution preservation by staging upstream `LICENSE` and `AUTHORS`
- clean rebuild smoke test that only uses a fake local checkout and does not
download a model
The native smoke path is intentionally minimal in this story. It proves the
reproducible source dependency and build seam without pulling Meshnet protocol
code into llama.cpp.
## Files changed
- `packages/node/native/llama/UPSTREAM_COMMIT`
- `packages/node/native/llama/UPSTREAM_REPOSITORY`
- `packages/node/native/llama/UPSTREAM_ASSUMPTIONS.md`
- `packages/node/native/llama/README.md`
- `packages/node/native/llama/patches/0001-add-meshnet-worker-scaffold.patch`
- `packages/node/native/llama/templates/meshnet_worker.cpp`
- `packages/node/native/scripts/build_llama_worker.sh`
- `tests/test_llama_worker_build.py`
## Exact commands and real results
### Native smoke build against a fake pinned checkout
```bash
tmpdir=$(mktemp -d)
mkdir -p "$tmpdir/llama.cpp"
printf 'MIT\n' > "$tmpdir/llama.cpp/LICENSE"
printf 'AUTHORS\n' > "$tmpdir/llama.cpp/AUTHORS"
printf '# placeholder\n' > "$tmpdir/llama.cpp/CMakeLists.txt"
printf '%s\n' 'b3c9d1b846cc80a6360adb6aeaa4fcd8c4c8dcac' > "$tmpdir/llama.cpp/.meshnet-upstream-commit"
git init -q "$tmpdir/llama.cpp"
packages/node/native/scripts/build_llama_worker.sh \
--source-dir "$tmpdir/llama.cpp" \
--build-dir "$tmpdir/build"
```
Result:
- `meshnet worker scaffold ok`
- `upstream commit: b3c9d1b846cc80a6360adb6aeaa4fcd8c4c8dcac`
- `patchset version: 0001`
- `build ok: /tmp/.../build/meshnet_worker`
### Targeted pytest
```bash
python -m pytest -q tests/test_llama_worker_build.py
```
Result: `1 passed in 0.53s`
### Python compile check
```bash
python -m compileall -q packages tests
```
Result: exit 0
### Diff hygiene
```bash
git diff --check
```
Result: exit 0
### Full deterministic pytest
```bash
python -m pytest -q
```
Result: `424 passed, 13 skipped, 210 failed, 86 errors in 131.04s`
The failures are pre-existing sandbox socket failures in tracker/HTTP-backed
tests. Representative error:
- `PermissionError: [Errno 1] Operation not permitted` when the tracker tries
to bind a socket.
This matches the previously observed environment limitation in the DGR-002 and
DGR-003 evidence and is unrelated to the llama.cpp pin/build scaffold.
## Limitations
- The sandbox does not provide `cmake`, so the smoke build uses the available
direct C++ compiler path (`g++` here) instead of a CMake-generated target.
- The pinned upstream source was not fetched from GitHub during validation.
The script supports fetching the exact commit when network access is
available, but the validation run used a fake local checkout to keep the test
deterministic and model-free.
- The patch stack in this story is deliberately narrow and additive. It creates
a worker scaffold and build seam, not the final llama.cpp runtime patches.
## Compatibility notes
- The exact upstream pin is `b3c9d1b846cc80a6360adb6aeaa4fcd8c4c8dcac`.
- The build script fails closed if the checkout pin differs from that commit or
if the expected upstream files (`LICENSE`, `AUTHORS`, `CMakeLists.txt`) are
missing.
- The patch stack is isolated from Meshnet networking code and can be applied
to a clean pinned checkout before later worker stories extend the scaffold.
- Upstream attribution notices are preserved in the build output by copying the
staged `LICENSE` and `AUTHORS` files into `build/.../upstream-notices/`.
## Dependent-story handoff
- DGR-008 can replace the scaffold source with the real supervised C++ worker
while keeping the same pin metadata, patch stack, and build script boundary.
- DGR-005 and later native stories should keep using the same exact pin so the
worker seam remains reproducible while range-loading and session logic are
added.

96
.scratch/distributed-gguf-runtime/evidence/DGR-005/README.md Normal file
View File

@@ -0,0 +1,96 @@
# DGR-005 — dense-Llama range-aware GGUF ownership evidence
Status: done
Date: 2026-07-15
Evidence kind: **synthetic-unit + repo checks**. No model download, no GPU, no network, no API credits.
## Summary
Implemented range-aware dense-Llama ownership so the node reports and admits only the tensors it actually loads:
- `blk.N.*` tensors are selected strictly by assigned layer range.
- Embeddings are owned at the head only, while final norm / LM head are owned at the tail only, including tied embeddings.
- Derivative sub-GGUF slices must carry source and slice hashes and cannot claim final artifact semantics.
- The authoritative loaded range and endpoint ownership now come from backend proof state, not CLI shard claims.
- Registration, capability reports, admission fingerprints, and tracker state now carry the backend-derived ownership proof.
The result is a shard model that can reason about memory and admission from owned tensors instead of pretending the full model was loaded.
## Files changed
- `packages/node/meshnet_node/gguf_ownership.py` - dense-Llama tensor selection and authoritative ownership helpers.
- `packages/node/meshnet_node/capability.py` - shard reports now carry endpoint ownership and parse it round-trip.
- `packages/node/meshnet_node/doctor.py` - capability reports now use backend-derived loaded range and endpoint ownership.
- `packages/node/meshnet_node/testing.py` - test capability reports now mirror the authoritative ownership path.
- `packages/node/meshnet_node/admission.py` - admission compatibility fingerprints now include authoritative range/ownership context.
- `packages/node/meshnet_node/model_backend.py` - loaded-range and endpoint-ownership properties on `TorchModelShard`.
- `packages/node/meshnet_node/startup.py` - registration payloads now use the proof-driven shard range.
- `packages/tracker/meshnet_tracker/capability.py` - tracker capability state preserves endpoint ownership.
- `tests/test_gguf_ownership.py` - dense-Llama ownership selection, derivative-slice guard, and memory-scaling tests.
- `tests/test_node_capability.py` - capability report ownership round-trip tests.
- `tests/test_node_admission.py` - backend-loaded range beats CLI claim regression tests.
- `tests/test_tracker_capability_admission.py` - tracker capability proof parsing tests.
## Exact commands and real results
### Targeted pytest slices
```bash
python -m pytest -q tests/test_gguf_ownership.py tests/test_node_capability.py tests/test_node_admission.py
```
Result: `73 passed`
```bash
python -m pytest -q tests/test_tracker_capability_admission.py -k 'test_a_passing_report_that_covers_the_registration_is_admitted or test_a_missing_report_is_absent_not_admitted or test_a_failed_report_is_recorded_as_failed or test_a_report_for_a_different_model_is_a_model_mismatch or test_a_report_for_a_different_shard_is_a_shard_mismatch or test_a_report_for_a_different_recipe_than_the_node_declares_is_a_recipe_mismatch or test_a_report_for_a_different_compatibility_fingerprint_is_a_compatibility_mismatch or test_an_older_recipe_catalogue_is_incompatible or test_an_unparseable_catalogue_version_is_incompatible or test_a_stale_report_is_not_admitted or test_a_future_dated_report_is_not_admitted or test_a_report_from_an_unknown_schema_version_is_invalid or test_a_malformed_report_is_invalid_and_never_admitted or test_recorded_detail_carries_no_credentials_from_node_diagnostics or test_compat_policy_routes_a_legacy_node_but_never_a_broken_proof or test_the_policy_is_read_from_the_environment_and_defaults_to_compat'
```
Result: `22 passed, 13 deselected`
### Python compile check
```bash
python -m compileall -q packages tests
```
Result: exit 0
### Diff hygiene
```bash
git diff --check
```
Result: exit 0
### Full deterministic pytest
```bash
python -m pytest -q
```
Result: `211 failed, 428 passed, 13 skipped, 14 warnings, 86 errors in 135.03s`
The failing set is not caused by this story. The dominant environment issues were:
- tracker and HTTP/socket-backed tests fail with `PermissionError: [Errno 1] Operation not permitted` when the tracker tries to bind sockets in this sandbox
- native protocol tests fail early with a protobuf runtime/gencode mismatch: generated code expects protobuf 7.35.0 while the installed runtime is 6.33.6
## Limitations
- This evidence is intentionally deterministic and model-free.
- The memory-scaling check is synthetic: it validates that owned tensor bytes scale with selected tensors, not a live GGUF download.
- Native C++ code was not changed by this story, so the pinned llama.cpp build validation remains covered by DGR-004 rather than repeated here.
## Compatibility notes
- Dense-Llama ownership is range-first: the shard interior is `blk.N.*`, and endpoint tensors are only attributed to the head or tail owner as appropriate.
- Derivative GGUF slices are explicitly not final artifacts; they must preserve source and slice hashes if used as a temporary compatibility bridge.
- The model proof path is authoritative for reported range and endpoint ownership, so operator CLI claims no longer control what the node advertises.
- Admission and tracker state now consume the same proof-derived ownership shape, keeping capability reports aligned end to end.
## Handoff for dependent stories
- DGR-006 can reuse `gguf_ownership.py` and the new capability fields to wire the shard protocol to proof-derived ownership without re-deriving tensor names.
- DGR-008 and later routing work should continue to treat endpoint ownership as metadata and `blk.N.*` ownership as the core range contract.
- If a future temporary slice path is needed, it should keep source/slice hashes visible and avoid claiming final-artifact semantics until a real proof exists.

203
.scratch/distributed-gguf-runtime/evidence/DGR-006/README.md Normal file
View File

@@ -0,0 +1,203 @@
# DGR-006 — Architecture-defined boundary input/output: evidence
Status: done
Date: 2026-07-15
Evidence kind: **synthetic-unit** (pure-numpy dense-Llama reference + boundary
contract). No model download, no GPU, no torch, no network, no API credit.
## Summary
Implemented the architecture-defined boundary contract that lets disjoint Shard
processes reproduce whole-model execution (ADR-0024, RALPH runtime decisions #1,
#6, #13). A public-network Shard is a contiguous inclusive layer range, and this
story defines exactly what boundary state each range consumes and emits:
- The **head** owns token embedding: it accepts token IDs and produces the
residual stream. It refuses an upstream boundary bundle.
- **Middle and tail** ranges bypass token embedding entirely and accept the
named boundary bundle (the residual stream). They refuse token IDs.
- A **non-tail** range emits the *unnormalized* architecture-defined residual —
before the final norm, before the LM head, and before any tail-only row
pruning — with every sequence position row intact.
- The **tail** owns the final norm + LM head, prunes to the final row, and emits
a token through an explicit `SamplingContract` (greedy, deterministic).
- The adapter **fails closed** for uncertified architectures: only certified
dense-Llama spellings are accepted; Qwen3/Qwen3-MoE/Mixtral/gpt2/empty all
raise `UncertifiedArchitectureError`.
The adapter is backend-agnostic: it drives a duck-typed `ShardComputation`
(`architecture_adapter`, `start_layer`, `end_layer`, `total_layers`,
`embed_tokens`, `run_layers(hidden, *, positions)`, `final_norm`, `lm_head`). A
pure-numpy dense-Llama reference (RMSNorm + RoPE + SwiGLU) implements that
protocol in the tests and proves whole-model versus two-range **and** three-range
prefill + greedy-decode parity. torch/transformers are not installed in the
default `.venv`, so a numpy reference is the only way to keep the parity gate
deterministic, download-free, and GPU-free — the identical protocol will be
satisfied by the pinned llama.cpp worker (DGR-008) and the PyTorch backend.
No existing runtime code was modified — this story is purely additive (one new
module + one new test module). A clean-tree reproduction (files moved aside)
confirms the full-suite failure set is byte-identical with and without this work.
## Files changed (all new)
- `packages/node/meshnet_node/boundary_adapter.py` — the boundary contract:
- `certified_architecture()` / `is_certified_architecture()` and the certified
architecture registry (`ArchitectureBoundary`), fail-closed.
- `ShardRole` + `role_for_range()` (head/middle/tail/full).
- `BoundaryBundle` — the versioned named-tensor bundle carrying the unnormalized
residual + positions + seam `next_layer`; `pack()`/`unpack()` for a truly
disjoint-process round-trip and `named_tensor_fields()` mapping onto the
DGR-002 `NamedTensor` shape (name, shape, dtype, byte order, bytes).
- `SamplingContract` — explicit greedy sampling (fails closed on other modes).
- `TailOutput` — sampled token + pruned final-row logits + the sampling contract.
- `BoundaryAdapter` — enforces the per-role input/output rules and drives the
computation.
- `tests/test_boundary_adapter.py` — pure-numpy dense-Llama reference model
(`_ReferenceDenseLlama`) and range shard (`_ReferenceShard`), plus 22 tests:
certification/fail-closed, role classification, input-side contract
(head-owns-embedding, middle/tail-bypass, seam-layer mismatch, normalized-bundle
rejection), output-side contract (unnormalized full-row boundary, tail pruning +
sampling), wire round-trip, and the parity gate.
## Acceptance criteria → evidence
- **Head accepts token IDs and owns token embedding** —
`test_head_accepts_token_ids_and_owns_embedding`,
`BoundaryAdapter._ingest_tokens` (head requires token IDs, refuses a bundle).
- **Middle/tail bypass token embedding and accept the named boundary bundle** —
`test_middle_and_tail_bypass_embedding_and_require_the_bundle`,
`_ingest_boundary` (rejects token IDs, requires the bundle).
- **Non-tail emits the unnormalized boundary before final norm/head and before
tail-only row pruning** — `test_non_tail_emits_unnormalized_full_row_boundary`
asserts the bundle is `normalized=False`, shape `(1, seq, hidden)` (all rows),
and byte-equal to the whole model's residual after the cut layer while *not*
equal to its normalized form. `_emit_boundary`.
- **Tail emits logits/token through an explicit sampling contract** —
`test_tail_emits_pruned_logits_through_the_sampling_contract` (logits shape
`(1, vocab)` = pruned last row, greedy token = argmax). `_emit_tail`,
`SamplingContract`.
- **Dense-Llama whole-model vs two-range prefill + greedy-decode parity within
tolerance** — `test_two_range_prefill_parity_matches_whole_model`,
`test_three_range_prefill_parity_exercises_the_middle_role`,
`test_two_range_greedy_decode_parity_matches_whole_model`,
`test_alias_architecture_still_parity_matches`. Documented tolerance:
next-token logits `np.allclose(..., atol=1e-6)` and **identical** greedy token
sequences. (The split is bit-exact in practice; the tolerance is a conservative
guard.)
- **Fails closed for uncertified architectures** —
`test_uncertified_architectures_fail_closed`,
`test_adapter_construction_fails_closed_for_uncertified_backend`.
- **Targeted pytest** — `22 passed`.
- **compileall packages tests** — exit 0.
- **git diff --check** — clean.
- **Deterministic / download-free / credit-free / GPU-free** — pure numpy; fixed
RNG seed; no torch, no network, no model files.
- **Full deterministic pytest** — `20 failed, 715 passed, 13 skipped, 12 errors`.
All 20 failures + 12 errors are pre-existing and unrelated (see below).
- **Native C++ / CTest / llama.cpp patch stack** — **not touched by this story.**
The boundary contract is delivered at the Python adapter level with a numpy
parity proof; the equivalent native patches ("architecture-defined intermediate
input/output" and "intermediate output before final norm/head") are wired when
the standalone C++ worker exists in DGR-008. No native code, CMake, or llama.cpp
patch was modified, so those gates are N/A here (same as DGR-005).
## Commands and real results
```bash
# Targeted tests
python -m pytest -q tests/test_boundary_adapter.py
# -> 22 passed in 0.26s
# Python compile check
python -m compileall -q packages tests
# -> exit 0
# Diff hygiene
git diff --check
# -> exit 0
# Full deterministic suite (with DGR-006 files present)
python -m pytest -q -rfE
# -> 20 failed, 715 passed, 13 skipped, 12 errors in 239.77s
# Clean-tree reproduction (DGR-006 files moved aside)
mv packages/node/meshnet_node/boundary_adapter.py /tmp/ && mv tests/test_boundary_adapter.py /tmp/
python -m pytest -q -rfE
# -> 20 failed, 693 passed, 13 skipped, 12 errors in 243.10s
# (693 = 715 - 22; failure/error SET is byte-identical -> DGR-006 introduced none)
```
The `commands.txt` and `results.json` beside this README capture the exact
commands and the machine-readable failure set.
## Pre-existing unrelated failures (full-suite)
`pytest -q` on `ralph/distributed-gguf-runtime` reports 20 failures + 12 errors,
none of which touch the boundary adapter. Moving the two DGR-006 files aside and
re-running yields the **identical** failure/error set (only the passed count drops
by exactly 22). Categories:
- **12 errors — `tests/test_native_shard_protocol.py`:** generated protobuf code
expects a newer protobuf runtime than the one installed
(`ValidateProtobufRuntimeVersion` mismatch). Pre-existing; documented in the
DGR-002 / DGR-005 evidence.
- **20 failures** across `test_activation_compression.py`,
`test_dynamic_routing.py`, `test_gossip_and_relay.py`,
`test_manual_route_benchmark.py`, `test_node_doctor.py`,
`test_openai_gateway.py` (`langchain` optional dep),
`test_toploc_calibration_dispatch.py`, `test_tracker_capability_admission.py`,
`test_tracker_control_plane.py`, `test_tracker_routing.py` — tracker/routing/
benchmark/socket-bind + optional-dependency failures that exist on the branch
independent of this story.
## Limitations and deferred work
- **Numpy reference, not real weights.** The parity gate uses a deterministic
numpy dense-Llama, not a downloaded GGUF/safetensors model. Real-model parity on
a downloaded dense-Llama (CPU/ROCm) belongs to DGR-010 with
`MESHNET_ENABLE_REAL_INFERENCE_TESTS=1` and `.venv-rocm`.
- **Stateless decode for parity.** Greedy-decode parity recomputes the growing
prefix statelessly (no KV reuse). Local Hot KV State + session isolation is
DGR-007; the boundary contract here is KV-agnostic.
- **Native patch wiring deferred.** The C++/llama.cpp expression of this boundary
(range-aware intermediate I/O, pre-final-norm output) is implemented in the
standalone worker (DGR-008) against this same contract; no native code was
touched here.
- **Greedy-only sampling certified.** `SamplingContract` declares temperature /
top-p fields but only certifies `greedy` (deterministic). Stochastic sampling is
out of scope for the deterministic parity gate.
## Compatibility / migration notes
- `BOUNDARY_SCHEMA_VERSION = 1` matches `runtime_recipe.RuntimeRecipeIdentity`'s
`boundary_schema_version`. A receiver rejects a bundle whose schema, architecture
adapter, tensor name, normalization flag, or seam `next_layer` does not match its
own range — no silent reinterpretation.
- `BoundaryBundle.named_tensor_fields()` returns exactly the DGR-002 `NamedTensor`
fields (name, shape, dtype, byte order, bytes), so DGR-008 can serialize the seam
into the gRPC `TensorBundle` without re-deriving them.
- Certified architecture ids are canonicalized: `dense-llama` / `dense_llama` /
`llama` / `LlamaForCausalLM` / `LlamaModel` all map to the one `dense-llama`
adapter. Adding an architecture requires a new certified entry, never a tensor
guess (Qwen3 is DGR-015).
## Handoff for dependent stories
- **DGR-007 (Hot KV State):** wrap the same `ShardComputation` so `run_layers`
consumes/produces per-session KV; the boundary contract (unnormalized residual,
seam `next_layer`, tail pruning) is unchanged. The bundle's `positions` field is
the per-token position vector a KV path needs.
- **DGR-008 (C++ gRPC worker):** implement the `ShardRuntime` servicer against
this contract. Map `BoundaryBundle.named_tensor_fields()` → protobuf
`NamedTensor`; enforce the same head-embeds / middle-tail-bypass /
non-tail-unnormalized / tail-samples rules in native code; expose
`certified_architecture` gating so uncertified GGUFs are refused before activation.
- **DGR-009 (Meshnet integration):** carry `BoundaryBundle.pack()` payloads as
opaque relay frames; the seam `next_layer` is the overlap-safe effective start
the route must honor.
- **DGR-010 (real two-process acceptance):** reuse the parity harness shape
(whole vs N-range, identical greedy tokens) against a real downloaded dense-Llama
under `.venv-rocm`.
- **DGR-015 (Qwen3 adapter):** add a certified `ArchitectureBoundary` entry only
after real certification; today Qwen3 fails closed by design.

26
.scratch/distributed-gguf-runtime/evidence/DGR-006/commands.txt Normal file
View File

@@ -0,0 +1,26 @@
# DGR-006 exact commands (run from repo worktree root)
# Targeted boundary-adapter tests
python -m pytest -q tests/test_boundary_adapter.py
# -> 22 passed in 0.26s
# Python compile check for changed Python
python -m compileall -q packages tests
# -> exit 0
# Diff hygiene
git diff --check
# -> exit 0
# Full deterministic suite with DGR-006 files present
python -m pytest -q -rfE
# -> 20 failed, 715 passed, 13 skipped, 12 errors in 239.77s
# Clean-tree reproduction: move the two new DGR-006 files aside, re-run
mv packages/node/meshnet_node/boundary_adapter.py /tmp/dgr006_boundary_adapter.py
mv tests/test_boundary_adapter.py /tmp/dgr006_test_boundary_adapter.py
python -m pytest -q -rfE
# -> 20 failed, 693 passed, 13 skipped, 12 errors in 243.10s
# (693 = 715 - 22; failure/error set byte-identical to the with-files run)
mv /tmp/dgr006_boundary_adapter.py packages/node/meshnet_node/boundary_adapter.py
mv /tmp/dgr006_test_boundary_adapter.py tests/test_boundary_adapter.py

161
.scratch/distributed-gguf-runtime/evidence/DGR-006/results.json Normal file
View File

@@ -0,0 +1,161 @@
{
"story": "DGR-006",
"date": "2026-07-15",
"evidence_kind": "synthetic-unit (pure-numpy dense-Llama parity + boundary contract)",
"targeted_tests": {
"file": "tests/test_boundary_adapter.py",
"result": "22 passed"
},
"compileall": "exit 0",
"git_diff_check": "clean",
"parity_tolerance": {
"logits_atol": 1e-06,
"greedy_tokens": "identical"
},
"full_suite_with_files": {
"failed": 20,
"passed": 715,
"skipped": 13,
"errors": 12,
"seconds": 239.77
},
"full_suite_clean_tree": {
"failed": 20,
"passed": 693,
"skipped": 13,
"errors": 12,
"seconds": 243.1,
"note": "693 = 715 - 22 DGR-006 tests; failure/error set identical"
},
"failure_set_identical_with_and_without_dgr006": true,
"preexisting_unrelated_failures": [
{
"kind": "ERROR",
"nodeid": "tests/test_native_shard_protocol.py::test_capability_and_health_round_trip"
},
{
"kind": "ERROR",
"nodeid": "tests/test_native_shard_protocol.py::test_checksum_algorithms_verify"
},
{
"kind": "ERROR",
"nodeid": "tests/test_native_shard_protocol.py::test_cross_language_roundtrip_python_and_cpp"
},
{
"kind": "ERROR",
"nodeid": "tests/test_native_shard_protocol.py::test_defaults_are_stable_for_backward_compatibility"
},
{
"kind": "ERROR",
"nodeid": "tests/test_native_shard_protocol.py::test_fragment_and_reassemble_round_trip_with_checksums"
},
{
"kind": "ERROR",
"nodeid": "tests/test_native_shard_protocol.py::test_message_header_carries_every_required_field"
},
{
"kind": "ERROR",
"nodeid": "tests/test_native_shard_protocol.py::test_named_tensor_bundle_describes_shape_dtype_byteorder_and_fragments"
},
{
"kind": "ERROR",
"nodeid": "tests/test_native_shard_protocol.py::test_reassemble_detects_fragment_corruption"
},
{
"kind": "ERROR",
"nodeid": "tests/test_native_shard_protocol.py::test_service_descriptor_exposes_all_operations"
},
{
"kind": "ERROR",
"nodeid": "tests/test_native_shard_protocol.py::test_session_response_carries_structured_status_and_results"
},
{
"kind": "ERROR",
"nodeid": "tests/test_native_shard_protocol.py::test_session_stream_carries_open_prefill_decode_release_cancel"
},
{
"kind": "ERROR",
"nodeid": "tests/test_native_shard_protocol.py::test_unknown_fields_are_preserved_for_forward_compatibility"
},
{
"kind": "FAILED",
"nodeid": "tests/test_activation_compression.py::test_compressible_body_uses_zstd_when_it_clears_savings_policy"
},
{
"kind": "FAILED",
"nodeid": "tests/test_activation_compression.py::test_incompressible_body_stays_raw_after_measured_trial"
},
{
"kind": "FAILED",
"nodeid": "tests/test_activation_compression.py::test_malformed_zstd_and_legacy_raw_bodies_are_handled_explicitly"
},
{
"kind": "FAILED",
"nodeid": "tests/test_activation_compression.py::test_threshold_requires_both_byte_and_ratio_savings"
},
{
"kind": "FAILED",
"nodeid": "tests/test_dynamic_routing.py::test_admin_can_replace_a_served_model_and_release_it"
},
{
"kind": "FAILED",
"nodeid": "tests/test_gossip_and_relay.py::test_activation_compression_round_trips_and_skips_small_bodies"
},
{
"kind": "FAILED",
"nodeid": "tests/test_manual_route_benchmark.py::test_benchmark_records_one_and_two_node_routes"
},
{
"kind": "FAILED",
"nodeid": "tests/test_manual_route_benchmark.py::test_clients_without_route_are_unaffected"
},
{
"kind": "FAILED",
"nodeid": "tests/test_manual_route_benchmark.py::test_invalid_route_shape_is_400"
},
{
"kind": "FAILED",
"nodeid": "tests/test_manual_route_benchmark.py::test_pinned_route_uses_named_node"
},
{
"kind": "FAILED",
"nodeid": "tests/test_manual_route_benchmark.py::test_unknown_route_node_is_400"
},
{
"kind": "FAILED",
"nodeid": "tests/test_node_doctor.py::test_cli_doctor_flags_select_what_is_validated"
},
{
"kind": "FAILED",
"nodeid": "tests/test_openai_gateway.py::test_langchain_chat_openai"
},
{
"kind": "FAILED",
"nodeid": "tests/test_toploc_calibration_dispatch.py::test_calibration_run_dispatches_only_solo_capable_nodes"
},
{
"kind": "FAILED",
"nodeid": "tests/test_toploc_calibration_dispatch.py::test_calibration_run_node_without_commitment_endpoint_is_skipped_not_failed"
},
{
"kind": "FAILED",
"nodeid": "tests/test_toploc_calibration_dispatch.py::test_calibration_run_persists_corpus_and_results_endpoint_reports_it"
},
{
"kind": "FAILED",
"nodeid": "tests/test_tracker_capability_admission.py::test_an_enforcing_tracker_never_routes_a_node_whose_proof_does_not_cover_it[invalid]"
},
{
"kind": "FAILED",
"nodeid": "tests/test_tracker_control_plane.py::test_tracker_startup_does_not_import_or_load_model_backends"
},
{
"kind": "FAILED",
"nodeid": "tests/test_tracker_routing.py::test_shard_heal_cycle_surviving_node_covers_dead_peers_gap"
},
{
"kind": "FAILED",
"nodeid": "tests/test_tracker_routing.py::test_torch_node_applies_tracker_load_shard_directive"
}
]
}

229
.scratch/distributed-gguf-runtime/evidence/DGR-007/README.md Normal file
View File

@@ -0,0 +1,229 @@
# DGR-007 — Isolated concurrent local Hot KV State: evidence
Status: done
Date: 2026-07-15
Evidence kind: **synthetic-unit** (pure-numpy KV-cached dense-Llama reference +
session/KV manager). No model download, no GPU, no torch, no network, no API
credit.
## Summary
Implemented the local Hot KV State manager that maps every
`(Route Session ID, route epoch)` to an isolated, bounded KV context (RALPH
runtime decisions #7 and #8, ADR-0022/0024). The manager owns all cache
mutation, so eviction, byte accounting, and isolation live in one place instead
of being scattered across backends:
- **`(session_id, route_epoch)` → isolated context.** Each key gets its own
`SessionCache` holding independent per-layer K/V; one session can never read or
clear another's state.
- **KV allocated only for owned layers.** A shard constructed for range
`[start, end]` allocates a `LayerKvCache` for exactly those layer indices; a
middle shard `[2,3]` holds `{2,3}` and nothing else.
- **Full lifecycle:** prefill append, decode append, truncate (rollback),
release, TTL eviction, LRU eviction (by session cap and by byte budget), and an
**explicit** `CacheMiss` (unknown-session / evicted-ttl / evicted-lru /
released / superseded-epoch / seq-len-mismatch) so the head degrades to a
from-token-zero re-prefill instead of corrupting output (decision #14).
- **Fails closed on identity.** Stale route epochs raise `StaleRouteEpochError`; a
request carrying an incompatible KV recipe raises `IncompatibleCacheRecipeError`
(fingerprint mismatch of architecture / kv dtype / head geometry / owned range);
a recipe for an uncertified architecture fails closed at construction (reusing
the DGR-006 certified-architecture gate).
- **KV-aware boundary driver.** `KvBoundaryAdapter` wraps the DGR-006
`ShardComputation` (plus `run_layers_cached`) so a shard runs cached
prefill/decode through the manager while honouring the architecture-defined
boundary contract (head embeds tokens, middle/tail bypass embedding and consume
the unnormalized residual bundle, non-tail emits the unnormalized residual, tail
normalizes + heads + prunes + samples). The computation returns the new
position-encoded K/V; the manager commits it under the budget.
A pure-numpy **KV-cached** dense-Llama reference (RMSNorm + RoPE + SwiGLU with an
absolute-position causal mask over cached keys) proves that cached prefill/decode
reproduces the stateless whole-model greedy tokens bit-for-bit, single-range and
across a head/tail seam. torch/transformers are not installed in the default
`.venv`, so a numpy reference is the only way to keep the parity + isolation gate
deterministic, download-free, and GPU-free — the identical manager contract will
be satisfied by the pinned llama.cpp worker (DGR-008), where the KV context maps
onto a llama sequence.
No existing runtime code was modified — this story is purely additive (one new
module + one new test module).
## Files changed (all new)
- `packages/node/meshnet_node/hot_kv_state.py` — the KV/session manager:
- `KvCacheRecipe` — KV layout identity (certified architecture, kv dtype, head
geometry, owned range) with `fingerprint()` / `is_compatible()` /
`bytes_per_token()`; fails closed on uncertified architectures.
- `LayerKvCache` — per-owned-layer `(seq, n_kv_heads, head_dim)` K/V with
`append` / `truncate` / `nbytes`.
- `SessionCache` — the isolated per-`(session, epoch)` context over owned layers.
- `CacheMiss` / `CacheMissReason` — the explicit, serializable miss response.
- `HotKvStateManager` — `open` / `append` / `truncate` / `release` / `resolve` /
`get`, LRU+TTL+byte-budget eviction, stale-epoch + incompatible-recipe
rejection, epoch supersession, thread-safe (RLock), injectable clock.
- `KvBoundaryAdapter` + `kv_recipe_for()` — KV-aware boundary driver.
- `tests/test_hot_kv_state.py` — pure-numpy KV-cached dense-Llama reference and 22
tests (see below).
## Acceptance criteria → evidence
- **Map `(Route Session ID, route epoch)` to an isolated context** —
`test_prefill_then_decode_append_grows_owned_layers`,
`test_four_interleaved_sessions_have_no_kv_cross_talk`,
`HotKvStateManager.open` keys sessions on `(session_id, route_epoch)`.
- **Allocate KV only for owned layers** —
`test_manager_allocates_kv_only_for_owned_layers` (middle `[2,3]` → `{2,3}`),
`test_multi_range_cached_decode_parity_across_a_seam` (head owns `(0,1,2)`, tail
owns `(3,4,5)`), `test_recipe_bytes_per_token_scales_with_owned_layers`.
- **Prefill append / decode append / truncate / release / TTL-LRU eviction /
explicit cache-miss** — `test_prefill_then_decode_append_grows_owned_layers`,
`test_truncate_rolls_back_all_owned_layers`,
`test_release_one_session_leaves_others_intact_and_returns_memory`,
`test_ttl_eviction_yields_an_explicit_cache_miss`,
`test_lru_eviction_by_session_cap_reports_a_miss`,
`test_budget_eviction_keeps_total_within_budget`,
`test_unknown_session_is_an_explicit_cache_miss`,
`test_seq_len_mismatch_is_an_explicit_cache_miss`.
- **Reject stale epochs and incompatible cache recipes** —
`test_stale_route_epoch_is_rejected`,
`test_new_route_epoch_supersedes_and_frees_old_epoch`,
`test_incompatible_cache_recipe_is_rejected`,
`test_uncertified_architecture_recipe_fails_closed`.
- **≥ four concurrent sessions complete without token or KV cross-talk** —
`test_four_interleaved_sessions_have_no_kv_cross_talk` (four interleaved
round-robin sessions, four *distinct* references, each matches its own),
`test_four_sessions_on_real_threads_stay_isolated` (four OS threads).
- **Cancellation/release leaves others intact and memory returns to budget** —
`test_release_one_session_leaves_others_intact_and_returns_memory` (released
session → `CacheMiss(RELEASED)`, `total_bytes` drops, survivors keep matching
their references), `test_single_session_exceeding_budget_raises`.
- **Cached vs stateless correctness core** —
`test_cached_full_shard_decode_matches_stateless_whole_model`,
`test_cached_prefill_next_token_matches_whole_model_logits`,
`test_multi_range_cached_decode_parity_across_a_seam`. Documented tolerance:
**identical** greedy token ids (bit-exact in practice; cached incremental
attention equals stateless full-sequence recompute per query row).
- **Targeted pytest** — `22 passed`.
- **compileall packages tests** — exit 0.
- **git diff --check** — clean.
- **Deterministic / download-free / credit-free / GPU-free** — pure numpy; fixed
RNG seed; injectable clock (no wall-clock in tests); no torch, no network, no
model files.
- **Full deterministic pytest** — `13 failed, 755 passed, 14 skipped in 254.50s`.
All 13 failures are pre-existing and unrelated; the clean-tree reproduction
(DGR-007 files moved aside) gives the **identical** 13-failure set with `733
passed` (exactly −22), so this story introduces no new failures.
- **Native C++ / CTest / llama.cpp patch stack** — **not touched by this story.**
The KV context contract is delivered at the Python manager level with a numpy
parity + isolation proof; the equivalent native layer-filtered KV / session
mapping is wired when the standalone C++ worker exists in DGR-008. No native
code, CMake, or llama.cpp patch was modified, so those gates are N/A here (same
as DGR-005/006).
## Commands and real results
```bash
VP=/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python
$VP -m pytest -q tests/test_hot_kv_state.py
# -> 22 passed in ~0.3s
$VP -m compileall -q packages tests
# -> exit 0
git diff --check
# -> exit 0
$VP -m pytest -q tests/test_boundary_adapter.py tests/test_gguf_ownership.py
# -> 25 passed
$VP -m pytest -q -rfE
# -> 13 failed, 755 passed, 14 skipped in 254.50s
# Clean-tree reproduction (DGR-007 files moved aside)
mv packages/node/meshnet_node/hot_kv_state.py /tmp/ && mv tests/test_hot_kv_state.py /tmp/
$VP -m pytest -q -rfE
# -> 13 failed, 733 passed, 14 skipped in 252.12s (identical FAILED set; passed -22)
```
`commands.txt` beside this README captures the exact commands.
## Pre-existing unrelated failures (full-suite)
`pytest -q -rfE` on `ralph/distributed-gguf-runtime` reports 13 pre-existing
failures (and, in this run, 0 errors — the earlier DGR-005/006-era
`test_native_shard_protocol.py` protobuf errors no longer appear in this
environment). None touch the KV manager. Moving the two DGR-007 files aside and
re-running yields the **byte-identical** 13-`FAILED` set (only the passed count
drops by exactly 22). The exact set (all tracker/routing/benchmark/toploc/doctor,
i.e. socket-bind / control-plane env, not KV):
```
tests/test_dynamic_routing.py::test_admin_can_replace_a_served_model_and_release_it
tests/test_manual_route_benchmark.py::test_benchmark_records_one_and_two_node_routes
tests/test_manual_route_benchmark.py::test_clients_without_route_are_unaffected
tests/test_manual_route_benchmark.py::test_invalid_route_shape_is_400
tests/test_manual_route_benchmark.py::test_pinned_route_uses_named_node
tests/test_manual_route_benchmark.py::test_unknown_route_node_is_400
tests/test_node_doctor.py::test_cli_doctor_flags_select_what_is_validated
tests/test_toploc_calibration_dispatch.py::test_calibration_run_dispatches_only_solo_capable_nodes
tests/test_toploc_calibration_dispatch.py::test_calibration_run_node_without_commitment_endpoint_is_skipped_not_failed
tests/test_toploc_calibration_dispatch.py::test_calibration_run_persists_corpus_and_results_endpoint_reports_it
tests/test_tracker_capability_admission.py::test_an_enforcing_tracker_never_routes_a_node_whose_proof_does_not_cover_it[invalid]
tests/test_tracker_routing.py::test_shard_heal_cycle_surviving_node_covers_dead_peers_gap
tests/test_tracker_routing.py::test_torch_node_applies_tracker_load_shard_directive
```
## Limitations and deferred work
- **Numpy reference, not real weights.** The parity + isolation gate uses a
deterministic numpy KV-cached dense-Llama, not a downloaded GGUF/safetensors
model. Real-model concurrent KV isolation on a downloaded dense-Llama (CPU/ROCm)
belongs to DGR-010/DGR-012 with `MESHNET_ENABLE_REAL_INFERENCE_TESTS=1` and
`.venv-rocm`.
- **Manager-owned storage, native mapping deferred.** The KV bytes are numpy
arrays managed in-process. The llama.cpp expression (a filtered llama sequence
per `(session, epoch)` over owned layers) is implemented in the standalone
worker (DGR-008) against this same manager contract; no native code was touched.
- **Continuous batching is DGR-012.** This story delivers *isolation* and bounded
lifecycle for concurrent sessions; continuous batching of compatible active
sessions inside a node (decision #9) is DGR-012 and builds on this manager.
- **Greedy-only sampling.** Reuses the DGR-006 `SamplingContract` (greedy
certified). Stochastic sampling is out of scope for the deterministic gate.
- **Coexists with legacy `SessionCacheStore`.** The older AH-25
`model_backend.SessionCacheStore` (session-id-only, opaque transformers cache,
HTTP path) is untouched. `HotKvStateManager` is the native-runtime-aligned
successor: it adds route-epoch keying, owned-layer allocation, recipe-fingerprint
rejection, and a byte budget. DGR-008/009 wire the native worker to
`HotKvStateManager`, not `SessionCacheStore`.
## Compatibility / migration notes
- `KvCacheRecipe.fingerprint()` canonicalizes the architecture (via
`certified_architecture`), so `llama` / `LlamaForCausalLM` map to the same
recipe; it aligns field-for-field with the DGR-003 `RuntimeRecipeIdentity`
compatibility discipline and reuses `runtime_recipe.compatibility_fingerprint`.
- `CacheMiss` is a value (not an exception) so it can be serialized into the
DGR-002 native protocol's cache expectation/result field; `resolve()` returns it,
`get()` raises `KvCacheMissError` wrapping it.
- The manager takes an injectable `clock` for deterministic TTL tests; production
defaults to `time.monotonic`.
## Handoff for dependent stories
- **DGR-008 (C++ gRPC worker):** implement the servicer's KV path against
`HotKvStateManager`. Map each `(Route Session ID, route epoch)` to a filtered
llama sequence over owned layers; on decode, read the sequence's cached K/V,
compute the new position-encoded K/V, and commit via `append` (honour the byte
budget and return an explicit `CacheMiss` on eviction). Enforce
`KvCacheRecipe.is_compatible` before activation and reject stale epochs.
- **DGR-009 (Meshnet integration):** the route epoch the tracker assigns is the
`route_epoch` key; carry the `CacheMiss` reason back to the head so it re-prefills
from token zero on eviction/restart.
- **DGR-012 (continuous batching):** batch compatible active sessions whose
`KvCacheRecipe` fingerprints match; each session keeps its own `SessionCache`, so
batching is a scheduling concern layered over this isolation, not a change to it.
- **DGR-013 (failure/cancel matrix):** `release` + the budget-return assertion here
is the unit-level basis for the resource-cleanup matrix.

31
.scratch/distributed-gguf-runtime/evidence/DGR-007/commands.txt Normal file
View File

@@ -0,0 +1,31 @@
# DGR-007 — exact commands (run from the worktree root).
# Python: /run/media/popov/d/DEV/repos/d-popov.com/AI/.venv (Python 3.14.6, numpy 2.4.4).
# Root conftest.py adds packages/* to sys.path, so `meshnet_node` imports work.
VP=/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python
# Targeted tests for this story.
$VP -m pytest -q tests/test_hot_kv_state.py
# -> 22 passed
# Python compile check for the changed packages/tests.
$VP -m compileall -q packages tests
# -> exit 0
# Diff hygiene.
git diff --check
# -> exit 0
# Dependency (DGR-006) + range-ownership (DGR-005) tests still green.
$VP -m pytest -q tests/test_boundary_adapter.py tests/test_gguf_ownership.py
# -> 25 passed
# Full deterministic suite (with DGR-007 files present).
$VP -m pytest -q -rfE
# -> see README (pre-existing unrelated failure set, +22 passed vs baseline)
# Clean-tree reproduction (DGR-007 files moved aside).
mv packages/node/meshnet_node/hot_kv_state.py /tmp/ && mv tests/test_hot_kv_state.py /tmp/
$VP -m pytest -q -rfE
# -> identical failure/error set, passed count drops by exactly 22
mv /tmp/hot_kv_state.py packages/node/meshnet_node/ && mv /tmp/test_hot_kv_state.py tests/

47
.scratch/distributed-gguf-runtime/evidence/DGR-007/results.json Normal file
View File

@@ -0,0 +1,47 @@
{
"task_id": "DGR-007",
"title": "Add isolated concurrent local Hot KV State",
"status": "done",
"date": "2026-07-15",
"evidence_kind": "synthetic-unit",
"python": "/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv (Python 3.14.6, numpy 2.4.4)",
"files_changed": [
"packages/node/meshnet_node/hot_kv_state.py",
"tests/test_hot_kv_state.py"
],
"gates": {
"targeted_pytest": {"command": "pytest -q tests/test_hot_kv_state.py", "result": "22 passed"},
"compileall": {"command": "python -m compileall -q packages tests", "exit": 0},
"git_diff_check": {"command": "git diff --check", "exit": 0},
"dependency_tests": {"command": "pytest -q tests/test_boundary_adapter.py tests/test_gguf_ownership.py", "result": "25 passed"},
"full_suite_with_files": {"command": "pytest -q -rfE", "result": "13 failed, 755 passed, 14 skipped", "seconds": 254.50},
"full_suite_clean_tree": {"command": "pytest -q -rfE (DGR-007 files moved aside)", "result": "13 failed, 733 passed, 14 skipped", "seconds": 252.12}
},
"no_new_failures": true,
"failure_set_identical": true,
"passed_delta": 22,
"preexisting_failures": [
"tests/test_dynamic_routing.py::test_admin_can_replace_a_served_model_and_release_it",
"tests/test_manual_route_benchmark.py::test_benchmark_records_one_and_two_node_routes",
"tests/test_manual_route_benchmark.py::test_clients_without_route_are_unaffected",
"tests/test_manual_route_benchmark.py::test_invalid_route_shape_is_400",
"tests/test_manual_route_benchmark.py::test_pinned_route_uses_named_node",
"tests/test_manual_route_benchmark.py::test_unknown_route_node_is_400",
"tests/test_node_doctor.py::test_cli_doctor_flags_select_what_is_validated",
"tests/test_toploc_calibration_dispatch.py::test_calibration_run_dispatches_only_solo_capable_nodes",
"tests/test_toploc_calibration_dispatch.py::test_calibration_run_node_without_commitment_endpoint_is_skipped_not_failed",
"tests/test_toploc_calibration_dispatch.py::test_calibration_run_persists_corpus_and_results_endpoint_reports_it",
"tests/test_tracker_capability_admission.py::test_an_enforcing_tracker_never_routes_a_node_whose_proof_does_not_cover_it[invalid]",
"tests/test_tracker_routing.py::test_shard_heal_cycle_surviving_node_covers_dead_peers_gap",
"tests/test_tracker_routing.py::test_torch_node_applies_tracker_load_shard_directive"
],
"native_gates_touched": false,
"acceptance": {
"session_epoch_isolated_context": true,
"kv_only_owned_layers": true,
"prefill_decode_truncate_release_ttl_lru_cachemiss": true,
"reject_stale_epoch_and_incompatible_recipe": true,
"four_concurrent_sessions_no_crosstalk": true,
"release_leaves_others_and_returns_memory": true
}
}

83
.scratch/distributed-gguf-runtime/evidence/DGR-009/README.md Normal file
View File

@@ -0,0 +1,83 @@
# DGR-009 — Integrate the native worker with Meshnet: evidence
Status: done
Date: 2026-07-15
Evidence kind: **python-unit + repo-hygiene**. No model download, no GPU, no API
credit.
## Summary
Implemented the Meshnet-facing GGUF backend seam and recipe gating needed for
the native worker path:
- Added `GgufNodeBackend`, a backend-shaped adapter that lets the existing node
HTTP/control-plane code serve GGUF-backed shards without changing the
Transformers/Torch path for the default recipes.
- Added `llama-cpp-native` to the recipe manifest and gated startup so only
recipes with `backend_id == "llama.cpp"` build the GGUF backend.
- Preserved the existing registration/admission flow by carrying the validated
capability report and proof shard through registration.
- Added unit coverage for the GGUF backend seam and for recipe-gated startup.
- Fixed the explicit-shard startup path so the legacy Torch tests that use an
opaque stub model still pass without requiring HuggingFace config discovery.
## Files changed
- `packages/node/meshnet_node/gguf_backend.py` - new GGUF backend adapter and
worker-transport boundary.
- `packages/node/meshnet_node/startup.py` - recipe-gated GGUF backend injection
and explicit-shard startup fix.
- `packages/node/meshnet_node/recipes.json` - added `llama-cpp-native`.
- `tests/test_gguf_backend.py` - backend delegation and recipe-selection tests.
- `.ralph-tui/progress.md` - appended DGR-009 progress note.
- `.scratch/distributed-gguf-runtime/issues/09-integrate-the-native-worker-with-meshnet.md`
- marked `Status: done`.
## Commands and real results
```bash
python -m pytest -q tests/test_gguf_backend.py
# -> 2 passed in 0.05s
python -m pytest -q tests/test_node_admission.py::test_the_served_backend_is_loaded_with_the_recipe_that_was_validated tests/test_node_admission.py::test_backend_validation_failure_registers_nothing
# -> 2 passed in 0.07s
python -m compileall -q packages tests
# -> exit 0
git diff --check
# -> exit 0
python -m pytest -q
# -> 222 failed, 463 passed, 13 skipped, 86 errors in 135.65s
```
## Limitations
- `python -m pytest -q` is still not clean in this sandbox. The dominant
failures are tracker/control-plane socket `PermissionError: [Errno 1]
Operation not permitted` and a native protocol import failure caused by a
protobuf runtime mismatch (`gencode 7.35.0` vs runtime `6.33.6`).
- `tests/test_native_shard_protocol.py` currently fails for the same protobuf
runtime mismatch in this environment.
- `DGR-008` evidence was not present in the tree, so the dependency behavior was
verified by reading the live code and exercising the Python seam instead of
relying on a missing README.
## Compatibility notes
- The default Torch path remains intact; GGUF backend selection is explicit and
recipe-gated.
- `TorchNodeServer` already accepts an injected backend object, so the control
plane stays Meshnet-owned.
- The GGUF adapter currently establishes the seam for the native worker
transport; the compiled worker remains the owner of the gRPC protocol details.
## Dependent-story handoff
- DGR-008 should continue to own the native worker implementation and the
versioned gRPC frame handling behind `MESHNET_NATIVE_WORKER_URL`.
- DGR-010 / DGR-012 can build on this seam without changing the control plane:
the recipe-gated backend and validated capability report are already carried
through startup.

58
.scratch/distributed-gguf-runtime/evidence/DGR-010/BLOCKED.md Normal file
View File

@@ -0,0 +1,58 @@
# DGR-010 — Blocked handoff
Status: blocked
Date: 2026-07-15
## Blocker
I verified the local workspace and mounted-drive model storage, but there is no
certified dense-Llama artifact available on this machine to run the required
real-model two-process acceptance.
What I found:
- `/run/media/popov/d/DEV/models` contains Qwen artifacts and caches, but no
dense-Llama model snapshot or GGUF artifact.
- `/run/media/popov/d/DEV/llamacpp/llama.cpp/models` contains only vocab GGUFs,
not a certified dense-Llama model.
- The existing code paths for real startup, GGUF backend selection, Hot KV
isolation, and benchmark reporting are present and readable, but the actual
DGR-010 acceptance run needs a certified dense-Llama artifact from mounted
storage to satisfy the story contract.
## Verified current state
- DGR-009 evidence was read and verified as the dependency handoff.
- `packages/node/meshnet_node/startup.py` already gates backend selection by
recipe and can load either the Torch path or the explicit GGUF seam.
- `packages/node/meshnet_node/hot_kv_state.py`, `boundary_adapter.py`, and
`gguf_ownership.py` already provide the isolation/parity seams that DGR-010
would exercise.
- The repo has no existing `evidence/DGR-010/README.md` yet, which is expected
because the story has not been completed.
## Commands run
```bash
sed -n '1,260p' .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md
sed -n '1,260p' .scratch/distributed-gguf-runtime/issues/10-pass-local-real-model-two-process-acceptance.md
sed -n '1,260p' .scratch/distributed-gguf-runtime/evidence/DGR-009/README.md
git status --short
find /run/media/popov/d/DEV -type f \( -name '*.gguf' -o -name '*.safetensors' -o -name 'config.json' \) | rg -i 'llama|tinyllama|meta-llama|hf-internal-testing|qwen'
```
## Next step to unblock
Provide or mount a certified dense-Llama artifact on the configured mounted
drive storage, then rerun the DGR-010 acceptance path with
`MESHNET_ENABLE_REAL_INFERENCE_TESTS=1`.
## Continuation note
Once the artifact exists, the next iteration should:
1. Run the two local worker processes against the certified dense-Llama shard
ranges.
2. Capture parity, concurrency, memory, and failure metrics.
3. Write `evidence/DGR-010/README.md` with the real results and then update the
issue status.

70
.scratch/distributed-gguf-runtime/evidence/DGR-011/BLOCKED.md Normal file
View File

@@ -0,0 +1,70 @@
# DGR-011 — Blocked handoff
Status: blocked
Date: 2026-07-15
## Blocker
This story cannot be completed in the current workspace state because its
mandatory dependency, DGR-010, is still not passed.
Verified blockers:
- `.scratch/distributed-gguf-runtime/prd.json` still marks `DGR-010` and
`DGR-011` with `"passes": false`.
- `.scratch/distributed-gguf-runtime/evidence/DGR-010/README.md` does not
exist, and the only DGR-010 evidence artifact present is
`.scratch/distributed-gguf-runtime/evidence/DGR-010/BLOCKED.md`.
- Mounted storage search found Qwen model artifacts and llama.cpp vocab files,
but no certified dense-Llama GGUF artifact suitable for the required real
acceptance run.
## Verified current state
- The repo already contains the Meshnet-facing GGUF backend seam and the
recipe-gated startup path from DGR-009.
- The architecture and Ralph context require real-model execution for this
story, not synthetic workers or unit-only coverage.
- The current environment does not expose the dense-Llama artifact required to
run the prerequisite local real-model acceptance, so the two-machine route
cannot be proven end to end.
## Commands run
```bash
sed -n '1,260p' .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md
sed -n '1,260p' .scratch/distributed-gguf-runtime/issues/11-pass-a-real-heterogeneous-two-machine-route.md
sed -n '1,260p' .ralph-tui/progress.md
sed -n '1,240p' .scratch/distributed-gguf-runtime/evidence/DGR-010/BLOCKED.md
sed -n '1,220p' CONTEXT.md
sed -n '1,260p' docs/adr/0024-distributed-gguf-runtime.md
sed -n '282,350p' .scratch/distributed-gguf-runtime/prd.json
find /run/media/popov/d/DEV/models -maxdepth 3 \( -name '*.gguf' -o -name 'config.json' -o -name '*.safetensors' \)
find /run/media/popov/d/DEV/llamacpp/llama.cpp/models /run/media/popov/d/DEV/models -maxdepth 4 \( -iname '*llama*' -o -iname '*dense*' -o -iname '*qwen*' -o -name 'config.json' -o -name '*.gguf' \)
```
## Known limitations
- No certified dense-Llama artifact is available on mounted storage in this
workspace.
- No real two-machine execution was possible, so there are no real route,
hardware, backend, or drift metrics to record for this story.
- The story remains blocked until DGR-010 is completed with a real-model
evidence README and a confirmed dense-Llama artifact on mounted storage.
## Compatibility notes
- DGR-009's recipe-gated GGUF backend seam is present and can be reused.
- The acceptance path for this story still requires the upstream real-model
evidence from DGR-010 before any heterogeneous two-machine route can be
claimed.
## Dependent-story handoff
- Finish DGR-010 first, including its real-model evidence README and
acceptance run.
- Once DGR-010 passes, rerun the two-machine acceptance against the same
certified dense-Llama artifact, then record the two-host hardware/network
manifest, route, commands, and raw metrics in `evidence/DGR-011/README.md`.
- Do not update the issue to `Status: done` until the real two-machine route
has been executed and recorded.