[verified] feat: pin native protobuf and gRPC generation

.scratch/distributed-gguf-runtime/evidence/DGR-023/README.md

@@ -0,0 +1,126 @@
+# DGR-023 evidence — reproducible Python and C++ protobuf/gRPC generation
+
+**Status:** complete after controller verification and independent-review repairs on 2026-07-17.
+
+**Authority:** live Gitea issue #7. The local PRD is a secondary projection.
+
+## Implemented contract
+
+- Python generation requires exactly `grpcio-tools==1.82.1`; the generator checks installed distribution metadata and rejects missing or different versions with an actionable exact install command.
+- The C++ bootstrap builds one ignored toolchain prefix from exact inputs:
+  - Protobuf release `33.1` (`protobuf-config` version `33.1.0`);
+  - Abseil release `20250814.1`;
+  - gRPC C++ `1.82.1` at commit `acccf84c0df20487d64101f528e5d426541ca4e5`;
+  - gRPC's exact-commit submodules for c-ares, RE2, OpenSSL, and zlib.
+- Protobuf is configured with local dependencies only after the exact Abseil build. gRPC uses the installed Protobuf/Abseil packages and commit-pinned module dependencies, avoiding unpinned system development packages and download fallbacks.
+- CMake requires exact Protobuf `33.1.0` and gRPC `1.82.1`, requires the exported `gRPC::grpc_cpp_plugin` target, and always generates/builds both message and service stubs in the ignored build tree.
+- Python bindings remain committed package output; `--check` regenerates into a temporary directory and compares output. C++ bindings are never committed.
+- The C++ conformance test parses Python-produced vectors, validates fields/CRC32C, and emits `cpp_roundtrip.binpb`; Python compares that artifact byte-for-byte.
+
+## Defects found and fixed
+
+1. A relative bootstrap prefix was resolved after entering the temporary source directory, so successful output was deleted by cleanup. The script now canonicalizes the caller-relative destination first. The regression executes `--print-prefix` from a temporary working directory and validates the resulting path behavior.
+2. The original native path omitted gRPC C++ and accepted any discoverable plugin. The bootstrap now builds exact gRPC/plugin sources, and CMake rejects absent/incompatible versions.
+3. The Python script named the `grpcio-tools` pin but did not validate the installed distribution. It now refuses mismatched versions.
+4. Protobuf ignored a stale provider option and attempted to download a different Abseil. The build was stopped; exact Abseil is now built first and Protobuf uses `LOCAL_DEPENDENCIES_ONLY`.
+5. The host lacked OpenSSL development headers. Rather than add a floating system dependency, gRPC now uses the submodule pinned by its exact commit.
+6. Documentation uses `bash scripts/bootstrap_native_toolchain.sh ...`, so a normal checkout does not depend on executable-mode preservation.
+
+## Verified toolchain
+
+```text
+cmake version 4.4.0
+c++ (GCC) 15.2.1 20260123 (Red Hat 15.2.1-7)
+libprotoc 33.1
+protobuf CMake package 33.1.0
+grpcio-tools 1.82.1
+grpcio 1.82.1
+protobuf Python runtime 7.35.1
+gRPC C++ 1.82.1
+commit acccf84c0df20487d64101f528e5d426541ca4e5
+grpc_cpp_plugin sha256 995ca8ac620fe83532b649a7c8c0a9341c7003da927fe0e4a8f821bfc579206d
+```
+
+The native toolchain and generated/build artifacts live under ignored mounted-drive `build/` paths; model/build artifacts were not stored under `/home`.
+
+## Commands and results
+
+```bash
+bash scripts/bootstrap_native_toolchain.sh build/native-toolchain
+```
+
+```text
+passed from a clean build directory
+libprotoc 33.1
+gRPC 1.82.1 commit acccf84c0df20487d64101f528e5d426541ca4e5
+grpc_cpp_plugin sha256 995ca8ac620fe83532b649a7c8c0a9341c7003da927fe0e4a8f821bfc579206d
+```
+
+```bash
+cmake -S packages/node/native -B build/native \
+  -DCMAKE_PREFIX_PATH="$PWD/build/native-toolchain"
+cmake --build build/native -j"$(nproc)"
+test -f build/native/shard_runtime.grpc.pb.cc
+test -f build/native/shard_runtime.grpc.pb.h
+test -f build/native/libshard_runtime_grpc.a
+ctest --test-dir build/native --output-on-failure
+```
+
+```text
+Pinned gRPC 1.82.1: building ShardRuntime service stubs
+shard_runtime_proto built
+shard_runtime_grpc built
+1/1 shard_protocol_conformance passed
+```
+
+```bash
+python3 -m pytest -q tests/test_native_shard_protocol.py
+```
+
+```text
+50 passed, 2 optional-path skips
+```
+
+All DGR-023-required checks were selected explicitly:
+
+```bash
+python3 -m pytest -q -rs tests/test_native_shard_protocol.py \
+  -k 'cpp_and_python_agree_byte_for_byte or generated_python_stubs_match_the_proto or native_toolchain_bootstrap or wrong_grpcio'
+```
+
+```text
+4 passed, 48 deselected
+```
+
+```bash
+python3 scripts/generate_native_protocol.py --check
+python3 scripts/generate_protocol_goldens.py --check
+python3 scripts/ralph_prd_schema.py validate .scratch/distributed-gguf-runtime/prd.json
+python3 -m compileall -q packages tests
+git diff --check
+```
+
+```text
+generated stubs are up to date
+conformance vectors are up to date
+OK: 55 stories validated
+compileall passed
+git diff --check passed
+```
+
+## Changed files
+
+- `scripts/bootstrap_native_toolchain.sh`
+- `scripts/generate_native_protocol.py`
+- `packages/node/native/CMakeLists.txt`
+- `packages/node/native/README.md`
+- `tests/test_native_shard_protocol.py`
+- `.scratch/distributed-gguf-runtime/evidence/DGR-023/README.md`
+- `.scratch/distributed-gguf-runtime/prd.json` (secondary completion projection only)
+
+## Limitations and dependency handoff
+
+- This story proves exact schema/message/service generation and cross-language conformance. It does not implement or run the standalone worker service itself; DGR-033/DGR-037 own worker behavior.
+- The plugin SHA is evidence for this verified build. Reproducibility authority is the exact gRPC commit plus its submodule graph, not an assumption that different compilers produce byte-identical executables.
+- No model, GPU, API credits, or model download was used.
+- DGR-024 and DGR-037 may consume this completed generation dependency but must provide their own transport/worker evidence.

.scratch/distributed-gguf-runtime/prd.json

@@ -522,8 +522,8 @@
         "A clean checkout regenerates bindings deterministically or fails with an actionable toolchain error.",
         "Applicable shared quality gates in `prd.json` pass, and the evidence handoff records exact commands/results, changed files, limitations, and dependency handoff."
       ],
-      "passes": false,
+      "passes": true,
-      "notes": "Generated source issue: .scratch/distributed-gguf-runtime/issues/023-make-python-and-c-protobuf-generation-reproducible.md; prd.json is authoritative.",
+      "notes": "Completed from Gitea #7 after controller provisioned and exercised the exact Python/C++ toolchains. Verified deterministic generation, native CMake/CTest, Python↔C++ byte parity, compileall, and diff checks; fixed relative bootstrap prefix resolution.",
       "blocks": [
         "DGR-024",
         "DGR-037"

packages/node/native/CMakeLists.txt

@@ -4,10 +4,8 @@
 # never committed. A C++ consumer already needs a toolchain, so committing
 # generated C++ would only create a second copy of the schema that can rot.
 #
-# gRPC C++ is optional here on purpose. The conformance test only needs message
+# Protobuf and gRPC C++ are required together so message and service bindings are
-# types, so the schema can be verified on a machine that has protobuf but not
+# generated by one exact toolchain. The ignored bootstrap prefix supplies both.
-# the gRPC C++ stack. When gRPC *is* found, the service stubs are generated too
-# and exported as `shard_runtime_grpc` for the worker (DGR-008) to link.
 #
 # Build:
 #   cmake -S packages/node/native -B build/native -DCMAKE_PREFIX_PATH=<protobuf-install>
@@ -23,8 +21,17 @@ project(meshnet_shard_protocol CXX)
 set(CMAKE_CXX_STANDARD 17)
 set(CMAKE_CXX_STANDARD_REQUIRED ON)
 
-find_package(protobuf CONFIG REQUIRED)
+# Protobuf and gRPC are one pinned generation toolchain. Configure only against
-find_package(gRPC CONFIG QUIET)
+# the ignored prefix produced by scripts/bootstrap_native_toolchain.sh; accepting
+# an arbitrary system plugin would make generated service bindings host-dependent.
+set(MESHNET_PROTOBUF_VERSION "33.1.0")
+set(MESHNET_GRPC_VERSION "1.82.1")
+find_package(protobuf ${MESHNET_PROTOBUF_VERSION} EXACT CONFIG REQUIRED)
+find_package(gRPC ${MESHNET_GRPC_VERSION} EXACT CONFIG REQUIRED)
+
+if(NOT TARGET gRPC::grpc_cpp_plugin)
+  message(FATAL_ERROR "pinned gRPC package does not export grpc_cpp_plugin")
+endif()
 
 set(SHARD_PROTO "${CMAKE_CURRENT_SOURCE_DIR}/proto/shard_runtime.proto")
 
@@ -39,24 +46,19 @@ protobuf_generate(
   PROTOC_OUT_DIR "${CMAKE_CURRENT_BINARY_DIR}"
 )
 
-# Service stubs: only when the gRPC C++ stack is present.
+# Service stubs are part of the reproducibility contract, not an optional branch.
-if(gRPC_FOUND)
+add_library(shard_runtime_grpc STATIC "${SHARD_PROTO}")
-  add_library(shard_runtime_grpc STATIC "${SHARD_PROTO}")
+target_link_libraries(shard_runtime_grpc PUBLIC shard_runtime_proto gRPC::grpc++)
-  target_link_libraries(shard_runtime_grpc PUBLIC shard_runtime_proto gRPC::grpc++)
+target_include_directories(shard_runtime_grpc PUBLIC "${CMAKE_CURRENT_BINARY_DIR}")
-  target_include_directories(shard_runtime_grpc PUBLIC "${CMAKE_CURRENT_BINARY_DIR}")
+protobuf_generate(
-  protobuf_generate(
+  TARGET shard_runtime_grpc
-    TARGET shard_runtime_grpc
+  LANGUAGE grpc
-    LANGUAGE grpc
+  GENERATE_EXTENSIONS .grpc.pb.h .grpc.pb.cc
-    GENERATE_EXTENSIONS .grpc.pb.h .grpc.pb.cc
+  PLUGIN "protoc-gen-grpc=$<TARGET_FILE:gRPC::grpc_cpp_plugin>"
-    PLUGIN "protoc-gen-grpc=$<TARGET_FILE:gRPC::grpc_cpp_plugin>"
+  IMPORT_DIRS "${CMAKE_CURRENT_SOURCE_DIR}/proto"
-    IMPORT_DIRS "${CMAKE_CURRENT_SOURCE_DIR}/proto"
+  PROTOC_OUT_DIR "${CMAKE_CURRENT_BINARY_DIR}"
-    PROTOC_OUT_DIR "${CMAKE_CURRENT_BINARY_DIR}"
+)
-  )
+message(STATUS "Pinned gRPC ${gRPC_VERSION}: building ShardRuntime service stubs")
-  message(STATUS "gRPC C++ found: building ShardRuntime service stubs")
-else()
-  message(STATUS "gRPC C++ not found: building message types only "
-                 "(sufficient for the conformance test)")
-endif()
 
 enable_testing()
 

packages/node/native/README.md

@@ -46,7 +46,7 @@ sampled token to request/recipe identity and sampling/template/reasoning inputs.
 If the machine has no protobuf C++ toolchain:
 
 ```bash
-scripts/bootstrap_native_toolchain.sh build/native-toolchain
+bash scripts/bootstrap_native_toolchain.sh build/native-toolchain
 ```
 
 Then:
@@ -58,10 +58,10 @@ cmake --build build/native -j
 ctest --test-dir build/native --output-on-failure
 ```
 
-gRPC C++ is optional: without it, CMake builds the message types only, which is
+The bootstrap pins and builds Protobuf `33.1`, gRPC C++ `1.82.1`, and the
-all the conformance test needs. When gRPC C++ *is* found, the `ShardRuntime`
+matching `grpc_cpp_plugin` into one ignored prefix. CMake requires those exact
-service stubs are built too and exported as `shard_runtime_grpc` for the worker
+package versions and always generates both message and service stubs; it does
-(DGR-008) to link.
+not fall back to an arbitrary system plugin.
 
 ## How the cross-language check actually proves something
 

scripts/bootstrap_native_toolchain.sh

@@ -1,18 +1,13 @@
 #!/usr/bin/env bash
 # Build a protobuf C++ toolchain for the native Shard protocol.
 #
-# The Python side needs nothing beyond `pip install grpcio-tools` — it bundles
+# The Python side uses the exact grpcio-tools pin declared below. The C++ side
-# protoc. The C++ side needs libprotobuf headers and a protoc binary, and a
+# builds exact Protobuf, Abseil, and gRPC source revisions so `protoc`,
-# machine that has neither (no protobuf-devel, no cmake, no system protoc) can
+# `grpc_cpp_plugin`, headers, and libraries all come from one ignored prefix.
-# still get a working one from source with this script. It is the exact recipe
+# No system Protobuf/gRPC installation is accepted by the documented build.
-# DGR-002 used to build and run the C++ conformance test.
-#
-# gRPC C++ is deliberately NOT built here. The conformance test only needs
-# message types, so verifying the schema does not require the whole gRPC stack.
-# The worker (DGR-008) will need gRPC C++ and should extend this script then.
 #
 # Usage:
-#   scripts/bootstrap_native_toolchain.sh [install-prefix]
+#   bash scripts/bootstrap_native_toolchain.sh [install-prefix]
 #
 # Then:
 #   cmake -S packages/node/native -B build/native -DCMAKE_PREFIX_PATH=<prefix>
@@ -21,7 +16,17 @@
 
 set -euo pipefail
 
-PREFIX="${1:-${PWD}/build/native-toolchain}"
+resolve_prefix() {
+  local candidate="${1:-${PWD}/build/native-toolchain}"
+  realpath -m -- "${candidate}"
+}
+
+if [[ "${1:-}" == "--print-prefix" ]]; then
+  resolve_prefix "${2:-}"
+  exit 0
+fi
+
+PREFIX="$(resolve_prefix "${1:-}")"
 WORK="$(mktemp -d)"
 trap 'rm -rf "${WORK}"' EXIT
 
@@ -29,11 +34,15 @@ trap 'rm -rf "${WORK}"' EXIT
 # that stub is allowed to use, so these are exact, not floating.
 PROTOBUF_VERSION="33.1"
 ABSEIL_VERSION="20250814.1"
+GRPC_VERSION="1.82.1"
+GRPC_COMMIT="acccf84c0df20487d64101f528e5d426541ca4e5"
 
-command -v cmake >/dev/null || {
+for tool in cmake curl git realpath sha256sum tar; do
-  echo "cmake is required (pip install cmake==4.4.0)" >&2
+  command -v "${tool}" >/dev/null || {
-  exit 1
+    echo "${tool} is required" >&2
-}
+    exit 1
+  }
+done
 
 echo "--- fetching protobuf ${PROTOBUF_VERSION} and abseil ${ABSEIL_VERSION}"
 cd "${WORK}"
@@ -41,26 +50,77 @@ curl -sfL -o protobuf.tar.gz \
   "https://github.com/protocolbuffers/protobuf/releases/download/v${PROTOBUF_VERSION}/protobuf-${PROTOBUF_VERSION}.tar.gz"
 tar xzf protobuf.tar.gz
 
-# The protobuf release tarball ships utf8_range but not abseil, and its default
-# CMake provider expects abseil as a submodule, so vendor it into place.
 curl -sfL -o abseil.tar.gz \
   "https://github.com/abseil/abseil-cpp/releases/download/${ABSEIL_VERSION}/abseil-cpp-${ABSEIL_VERSION}.tar.gz"
 tar xzf abseil.tar.gz
-rm -rf "protobuf-${PROTOBUF_VERSION}/third_party/abseil-cpp"
-mv "abseil-cpp-${ABSEIL_VERSION}" "protobuf-${PROTOBUF_VERSION}/third_party/abseil-cpp"
 
-echo "--- building protobuf into ${PREFIX}"
+echo "--- building abseil ${ABSEIL_VERSION} into ${PREFIX}"
-cmake -S "protobuf-${PROTOBUF_VERSION}" -B build \
+cmake -S "abseil-cpp-${ABSEIL_VERSION}" -B abseil-build \
   -DCMAKE_BUILD_TYPE=Release \
   -DCMAKE_INSTALL_PREFIX="${PREFIX}" \
   -DCMAKE_POSITION_INDEPENDENT_CODE=ON \
-  -Dprotobuf_ABSL_PROVIDER=module \
+  -DABSL_ENABLE_INSTALL=ON \
-  -Dprotobuf_BUILD_TESTS=OFF \
+  -DABSL_BUILD_TESTING=OFF \
-  -Dprotobuf_BUILD_SHARED_LIBS=OFF \
   -DABSL_PROPAGATE_CXX_STD=ON
-cmake --build build -j"$(nproc)"
+cmake --build abseil-build -j"$(nproc)"
-cmake --install build
+cmake --install abseil-build
+
+echo "--- building protobuf ${PROTOBUF_VERSION} into ${PREFIX}"
+cmake -S "protobuf-${PROTOBUF_VERSION}" -B protobuf-build \
+  -DCMAKE_BUILD_TYPE=Release \
+  -DCMAKE_INSTALL_PREFIX="${PREFIX}" \
+  -DCMAKE_PREFIX_PATH="${PREFIX}" \
+  -Dabsl_DIR="${PREFIX}/lib64/cmake/absl" \
+  -DCMAKE_POSITION_INDEPENDENT_CODE=ON \
+  -Dprotobuf_LOCAL_DEPENDENCIES_ONLY=ON \
+  -Dprotobuf_BUILD_TESTS=OFF \
+  -Dprotobuf_BUILD_SHARED_LIBS=OFF
+cmake --build protobuf-build -j"$(nproc)"
+cmake --install protobuf-build
+
+echo "--- fetching gRPC ${GRPC_VERSION} at ${GRPC_COMMIT}"
+git init -q grpc-source
+git -C grpc-source remote add origin https://github.com/grpc/grpc.git
+git -C grpc-source fetch --depth 1 origin "${GRPC_COMMIT}"
+git -C grpc-source checkout --detach FETCH_HEAD
+git -C grpc-source submodule update --init --recursive --depth 1
+[[ "$(git -C grpc-source rev-parse HEAD)" == "${GRPC_COMMIT}" ]] || {
+  echo "gRPC checkout identity mismatch" >&2
+  exit 1
+}
+
+echo "--- building gRPC ${GRPC_VERSION} and grpc_cpp_plugin into ${PREFIX}"
+cmake -S grpc-source -B grpc-build \
+  -DCMAKE_BUILD_TYPE=Release \
+  -DCMAKE_INSTALL_PREFIX="${PREFIX}" \
+  -DCMAKE_PREFIX_PATH="${PREFIX}" \
+  -DProtobuf_DIR="${PREFIX}/lib64/cmake/protobuf" \
+  -Dabsl_DIR="${PREFIX}/lib64/cmake/absl" \
+  -DgRPC_INSTALL=ON \
+  -DgRPC_BUILD_TESTS=OFF \
+  -DgRPC_PROTOBUF_PROVIDER=package \
+  -DgRPC_ABSL_PROVIDER=package \
+  -DgRPC_CARES_PROVIDER=module \
+  -DgRPC_RE2_PROVIDER=module \
+  -DgRPC_SSL_PROVIDER=module \
+  -DgRPC_ZLIB_PROVIDER=module \
+  -DgRPC_BUILD_GRPC_CPP_PLUGIN=ON \
+  -DgRPC_BUILD_GRPC_CSHARP_PLUGIN=OFF \
+  -DgRPC_BUILD_GRPC_NODE_PLUGIN=OFF \
+  -DgRPC_BUILD_GRPC_OBJECTIVE_C_PLUGIN=OFF \
+  -DgRPC_BUILD_GRPC_PHP_PLUGIN=OFF \
+  -DgRPC_BUILD_GRPC_PYTHON_PLUGIN=OFF \
+  -DgRPC_BUILD_GRPC_RUBY_PLUGIN=OFF
+cmake --build grpc-build -j"$(nproc)"
+cmake --install grpc-build
 
 echo "--- done"
 "${PREFIX}/bin/protoc" --version
+[[ -x "${PREFIX}/bin/grpc_cpp_plugin" ]] || {
+  echo "grpc_cpp_plugin was not installed" >&2
+  exit 1
+}
+printf 'gRPC %s commit %s\n' "${GRPC_VERSION}" "${GRPC_COMMIT}"
+printf 'grpc_cpp_plugin sha256 '
+sha256sum "${PREFIX}/bin/grpc_cpp_plugin" | cut -d' ' -f1
 echo "configure the protocol build with: -DCMAKE_PREFIX_PATH=${PREFIX}"

scripts/generate_native_protocol.py

@@ -20,6 +20,7 @@ already requires a toolchain and nothing is gained by committing them.
 from __future__ import annotations
 
 import argparse
+from importlib import metadata
 import pathlib
 import shutil
 import subprocess
@@ -38,15 +39,32 @@ REQUIRED_GRPCIO_TOOLS = "1.82.1"
 _HEADER = "# Generated by scripts/generate_native_protocol.py. Do not edit.\n"
 
 
-def _generate(into: pathlib.Path) -> None:
+def _require_grpcio_tools_version() -> None:
-    """Run protoc, writing generated modules into `into`."""
     try:
-        from grpc_tools import protoc
+        actual = metadata.version("grpcio-tools")
-    except ImportError:  # pragma: no cover - exercised only without the toolchain
+    except metadata.PackageNotFoundError:
         sys.exit(
             "grpc_tools is required to generate stubs:\n"
             f"    pip install grpcio-tools=={REQUIRED_GRPCIO_TOOLS}"
         )
+    if actual != REQUIRED_GRPCIO_TOOLS:
+        sys.exit(
+            "wrong grpcio-tools version for deterministic generation: "
+            f"found {actual}, require {REQUIRED_GRPCIO_TOOLS}\n"
+            f"    pip install --upgrade grpcio-tools=={REQUIRED_GRPCIO_TOOLS}"
+        )
+
+
+def _generate(into: pathlib.Path) -> None:
+    """Run the exactly pinned protoc, writing generated modules into `into`."""
+    _require_grpcio_tools_version()
+    try:
+        from grpc_tools import protoc
+    except ImportError:  # pragma: no cover - inconsistent/broken installation
+        sys.exit(
+            "grpcio-tools metadata exists but grpc_tools cannot be imported; reinstall it:\n"
+            f"    pip install --force-reinstall grpcio-tools=={REQUIRED_GRPCIO_TOOLS}"
+        )
 
     into.mkdir(parents=True, exist_ok=True)
     # grpc_tools bundles protoc and the well-known types, so generation needs no

tests/test_native_shard_protocol.py

@@ -612,6 +612,27 @@ def test_a_peer_still_sending_the_retired_field_does_not_corrupt_the_tensor():
     assert decode_tensor(pb.NamedTensor.FromString(wire)) == b"\xaa" * 32
 
 
+def test_native_toolchain_bootstrap_resolves_relative_prefix_before_temp_chdir(tmp_path):
+    script = REPO_ROOT / "scripts/bootstrap_native_toolchain.sh"
+    result = subprocess.run(
+        ["bash", str(script), "--print-prefix", "relative/toolchain"],
+        cwd=tmp_path,
+        capture_output=True,
+        text=True,
+    )
+
+    assert result.returncode == 0, result.stdout + result.stderr
+    assert pathlib.Path(result.stdout.strip()) == tmp_path / "relative/toolchain"
+
+
+def test_python_generator_rejects_the_wrong_grpcio_tools_version(monkeypatch):
+    from scripts import generate_native_protocol
+
+    monkeypatch.setattr(generate_native_protocol.metadata, "version", lambda _: "0.0.0")
+    with pytest.raises(SystemExit, match="found 0.0.0, require 1.82.1"):
+        generate_native_protocol._require_grpcio_tools_version()
+
+
 def test_generated_python_stubs_match_the_proto():
     pytest.importorskip("grpc_tools", reason="protoc toolchain is required to verify")
     result = subprocess.run(