# DGR-002 — Versioned gRPC Shard protocol: evidence Status: done Date: 2026-07-15 Evidence kind: **synthetic-unit** (schema round-trip + cross-language protobuf compatibility). No model download, no GPU, no network, no API credits. ## Summary Added the versioned Protocol Buffers schema that is the semantic contract between Python and C++ Shards (ADR-0024), plus reproducible Python and C++ code generation/build wiring and generated-schema round-trip + compatibility tests in **both** languages. The schema defines one long-lived bidirectional gRPC stream per Route Session Activation Seam, bounded prefill chunking, a small decode fast path, and a versioned named-tensor bundle carrying every required identifier. No existing runtime code was modified — this story is purely additive (a new `.proto`, a `native_protocol` loader package, C++ build wiring, and one new test module). Generated stubs are produced on demand into gitignored `build/` directories, so nothing generated is committed. ## Files changed (all new) - `packages/node/native/proto/shard_runtime.proto` — the schema (package `meshnet.shard.v1`, proto3). Service `ShardRuntime` with `GetCapability`, `Health`, `ActivateSession` (bidi stream), `Release`, `Cancel`. - `packages/node/meshnet_node/native_protocol/__init__.py` — reproducible on-demand `grpc_tools.protoc` codegen + loader (`load()`, `load_grpc()`) and shared bundle helpers (`compute_checksum`, `verify_checksum`, `fragment_tensor`, `reassemble_tensor`). - `packages/node/native/scripts/generate_python.py` — standalone reproducible Python generation (self-contained; does not import `meshnet_node`). - `packages/node/native/scripts/generate_cpp.sh` — reproducible C++ generation (message stubs always; gRPC service stubs when `grpc_cpp_plugin` is present). - `packages/node/native/CMakeLists.txt` — C++ build wiring; works with both CONFIG-mode (`protobuf::libprotobuf`/`protobuf::protoc`) and CMake's `FindProtobuf` module. - `packages/node/native/tests/roundtrip_test.cpp` — C++ round-trip / compat test (`--selftest`, `--read`, `--write`). - `tests/test_native_shard_protocol.py` — Python round-trip + compatibility tests and the Python↔C++ cross-language driver. ## Acceptance criteria → evidence - **Capability/health/session-stream/release/cancellation schema** — the `ShardRuntime` service's five RPCs; `test_capability_and_health_round_trip`, `test_session_stream_carries_open_prefill_decode_release_cancel`. - **One long-lived bidi stream per Activation Seam with deadlines, cancellation, flow control, structured errors** — `rpc ActivateSession (stream ...) returns (stream ...)`. Deadlines: gRPC call deadline on direct transport, plus `SessionOpen.deadline_unix_nanos` for relay-carried frames. Cancellation: `Cancel` RPC and in-stream `CancelRequest`/`PHASE_CANCEL`. Flow control: `FlowControl` frames (credits + in-flight byte/message caps). Structured errors: `Status` (canonical code, message, `RetryClass`, details). Verified by `test_session_response_carries_structured_status_and_results`. - **Bounded prefill chunking + small decode fast path** — `PrefillChunk` (`chunk_index`/`chunk_count`/`final_chunk`, `SessionOpen.max_prefill_tokens_per_chunk`) and `DecodeStep` (minimal single-bundle path). Bounded fragments via `SessionOpen.max_fragment_bytes` and `fragment_tensor(...)`. - **Carries schema version, work ID, Route Session ID, route epoch, artifact/recipe fingerprint, shard range/effective start, phase, position, idempotency step, cache expectation, compression, checksum** — all on `MessageHeader` (+ `ArtifactFingerprint.runtime_recipe_fingerprint`, `ShardRange.effective_start_layer`). Verified field-by-field by `test_message_header_carries_every_required_field`. - **Versioned named-tensor bundle (name, shape, dtype, byte order, fragments)** — `TensorBundle`/`NamedTensor`/`TensorFragment`; `test_named_tensor_bundle_describes_shape_dtype_byteorder_and_fragments`, `test_fragment_and_reassemble_round_trip_with_checksums`. - **Round-trip + compatibility tests in Python and C++** — Python: `tests/test_native_shard_protocol.py` (11 tests). C++: `roundtrip_test.cpp` built via CMake; cross-language driver `test_cross_language_roundtrip_python_and_cpp` exercises Python→C++ and C++→Python in both directions. - **Targeted pytest** — `11 passed, 1 skipped` (default env); `12 passed` with the C++ toolchain on PATH. - **compileall packages tests** — exit 0. - **git diff --check** — clean. - **Deterministic / download-free / credit-free / GPU-free** — all tests are pure protobuf serialization; the C++ path uses only local compilers. - **Full deterministic pytest** — `704 passed, 14 skipped, 11 failed`. The 11 failures are pre-existing and unrelated (see below). ## Commands and real results See `commands.txt` for the exact command list. Key results: - `python packages/node/native/scripts/generate_python.py` → `shard_runtime_pb2.py: ok`, `shard_runtime_pb2_grpc.py: ok`. - `pytest tests/test_native_shard_protocol.py -q` → **11 passed, 1 skipped** (skip reason: `C++ toolchain unavailable: cmake not found on PATH`). - With `/tmp/pbsrc/install/bin` (protoc 33.1) and `.venv/bin` (cmake) on PATH and `CMAKE_PREFIX_PATH=/tmp/pbsrc/install`: - `generate_cpp.sh` → `shard_runtime.pb.cc`, `shard_runtime.pb.h` (grpc service stubs skipped: `grpc_cpp_plugin` absent). - `cmake -S ... -B ...` + `cmake --build ...` → build OK. - `shard_protocol_roundtrip_test --selftest` → `selftest ok (128 bytes)`, exit 0. - `ctest` → `1/1 Test #1: shard_protocol_roundtrip ... Passed`. - `pytest ...::test_cross_language_roundtrip_python_and_cpp -q` → **1 passed** (Python serializes → C++ parses & verifies → C++ serializes → Python parses & verifies). - `compileall -q packages tests` → exit 0. - `git diff --check` → clean. ## Pre-existing unrelated failures (full-suite) `pytest -q` on the full tree reports 11 failures, all in tracker routing / dynamic routing / manual route benchmark / toploc calibration — none import the Shard protocol. Clean-tree reproduction: with **all DGR-002 files moved aside** (`git status` shows only the pre-existing `.ralph-tui/config.toml` deletion), re-running exactly these tests gives `11 failed, 3 passed` — identical failures. They exist on the `ralph/distributed-gguf-runtime` branch independent of this story. The full list is in `results.json.preexisting_unrelated_failures`. Note: the earlier `progress.md` (RCR-001, on master) recorded a different set of 6 optional-dependency failures (zstandard, langchain_openai). Those did **not** recur here; this environment has those deps. The 11 above are branch-local routing/benchmark failures, not environmental. ## Limitations and deferred work - **C++ toolchain is host-provided, not vendored.** The default test env has no `protoc`/`cmake`/protobuf C++ headers on PATH, so the C++ cross-language test **skips** by default (explicit skip reason). It was executed for this evidence using an ephemeral from-source protobuf 33.1 install at `/tmp/pbsrc/install` plus the `.venv` cmake. DGR-004/DGR-008 should pin the C++ protobuf/gRPC toolchain (upstream commit + reproducible fetch/build) so this test runs in CI without relying on an ad-hoc `/tmp` install. - **gRPC C++ service stubs not built here.** `grpc_cpp_plugin` is absent, so `generate_cpp.sh` produced message stubs only. The round-trip test needs only message serialization; the service stubs are DGR-008's concern. - **No live gRPC transport yet.** This story delivers the schema + serialization contract and generation/build wiring only. Channel setup, the bidi stream server/client, deadlines/cancellation propagation over a real HTTP/2 channel, and relay framing are DGR-008/DGR-009. - **Protobuf runtime version skew.** Python runtime is pip protobuf 7.35.1; the C++ side used protoc 33.1. Protobuf wire format is stable across these, and the cross-language round-trip confirms interop; version pinning is deferred to the toolchain-pinning stories. ## Compatibility / migration notes - proto3 with a 0-valued `*_UNSPECIFIED` member on every enum and never-reused field numbers. Forward compatibility (unknown-field preservation) is verified behaviourally by `test_unknown_fields_are_preserved_for_forward_compatibility` — note protobuf 7.x's upb backend does not implement the `UnknownFields()` introspection accessor, so the test asserts the observable re-serialization outcome instead. Backward defaults verified by `test_defaults_are_stable_for_backward_compatibility`. - Wire schema version is `SchemaVersion.SCHEMA_VERSION_1` (int 1), also exposed as `meshnet_node.native_protocol.SCHEMA_VERSION`. ## Handoff for dependent stories - **DGR-003 (recipe/fingerprint):** populate `ArtifactFingerprint` (`model_id`, `revision`, `artifact_hash`, `quantization`, `runtime_recipe_fingerprint`). Admission compares these before activation; a mismatch is a fatal `Status` (`RetryClass.RETRY_CLASS_FATAL`). - **DGR-004 (llama.cpp pin) / DGR-008 (C++ worker):** pin the C++ protobuf + gRPC toolchain and add `grpc_cpp_plugin`; then `generate_cpp.sh` emits service stubs and the CMake target can link gRPC. Implement the `ShardRuntime` servicer; map `(route_session_id, route_epoch)` to an isolated llama sequence. Use `SessionOpen` for stream-scoped bounds and `FlowControl` for backpressure. - **DGR-009 (Meshnet integration/relay):** the relay may carry serialized `SessionActivation`/`SessionResponse` frames as opaque binary; use the in-message `deadline_unix_nanos`, `CancelRequest`, and `FlowControl` since gRPC call metadata is lost over relay. - **Loader usage:** `from meshnet_node import native_protocol as proto; pb2 = proto.load()`. Stubs regenerate automatically when the `.proto` changes (mtime check). `proto.load_grpc()` returns the service stubs (needs the `grpc` runtime). - **Gotcha:** the `.venv` installs the meshnet packages editable via a PEP 660 meta-path finder pointing at the **main** checkout. Import the worktree copy by ensuring the worktree `packages/node` is on `sys.path` first (conftest already does this for pytest); standalone tooling must derive paths from `__file__` and not `import meshnet_node` (why `generate_python.py` is self-contained).