chore: replace fake protocol story with real harness

.scratch/distributed-gguf-runtime/evidence/DGR-023/README.md

@@ -0,0 +1,126 @@
+# DGR-023 evidence — reproducible Python and C++ protobuf/gRPC generation
+
+**Status:** complete after controller verification and independent-review repairs on 2026-07-17.
+
+**Authority:** live Gitea issue #7. The local PRD is a secondary projection.
+
+## Implemented contract
+
+- Python generation requires exactly `grpcio-tools==1.82.1`; the generator checks installed distribution metadata and rejects missing or different versions with an actionable exact install command.
+- The C++ bootstrap builds one ignored toolchain prefix from exact inputs:
+  - Protobuf release `33.1` (`protobuf-config` version `33.1.0`);
+  - Abseil release `20250814.1`;
+  - gRPC C++ `1.82.1` at commit `acccf84c0df20487d64101f528e5d426541ca4e5`;
+  - gRPC's exact-commit submodules for c-ares, RE2, OpenSSL, and zlib.
+- Protobuf is configured with local dependencies only after the exact Abseil build. gRPC uses the installed Protobuf/Abseil packages and commit-pinned module dependencies, avoiding unpinned system development packages and download fallbacks.
+- CMake requires exact Protobuf `33.1.0` and gRPC `1.82.1`, requires the exported `gRPC::grpc_cpp_plugin` target, and always generates/builds both message and service stubs in the ignored build tree.
+- Python bindings remain committed package output; `--check` regenerates into a temporary directory and compares output. C++ bindings are never committed.
+- The C++ conformance test parses Python-produced vectors, validates fields/CRC32C, and emits `cpp_roundtrip.binpb`; Python compares that artifact byte-for-byte.
+
+## Defects found and fixed
+
+1. A relative bootstrap prefix was resolved after entering the temporary source directory, so successful output was deleted by cleanup. The script now canonicalizes the caller-relative destination first. The regression executes `--print-prefix` from a temporary working directory and validates the resulting path behavior.
+2. The original native path omitted gRPC C++ and accepted any discoverable plugin. The bootstrap now builds exact gRPC/plugin sources, and CMake rejects absent/incompatible versions.
+3. The Python script named the `grpcio-tools` pin but did not validate the installed distribution. It now refuses mismatched versions.
+4. Protobuf ignored a stale provider option and attempted to download a different Abseil. The build was stopped; exact Abseil is now built first and Protobuf uses `LOCAL_DEPENDENCIES_ONLY`.
+5. The host lacked OpenSSL development headers. Rather than add a floating system dependency, gRPC now uses the submodule pinned by its exact commit.
+6. Documentation uses `bash scripts/bootstrap_native_toolchain.sh ...`, so a normal checkout does not depend on executable-mode preservation.
+
+## Verified toolchain
+
+```text
+cmake version 4.4.0
+c++ (GCC) 15.2.1 20260123 (Red Hat 15.2.1-7)
+libprotoc 33.1
+protobuf CMake package 33.1.0
+grpcio-tools 1.82.1
+grpcio 1.82.1
+protobuf Python runtime 7.35.1
+gRPC C++ 1.82.1
+commit acccf84c0df20487d64101f528e5d426541ca4e5
+grpc_cpp_plugin sha256 995ca8ac620fe83532b649a7c8c0a9341c7003da927fe0e4a8f821bfc579206d
+```
+
+The native toolchain and generated/build artifacts live under ignored mounted-drive `build/` paths; model/build artifacts were not stored under `/home`.
+
+## Commands and results
+
+```bash
+bash scripts/bootstrap_native_toolchain.sh build/native-toolchain
+```
+
+```text
+passed from a clean build directory
+libprotoc 33.1
+gRPC 1.82.1 commit acccf84c0df20487d64101f528e5d426541ca4e5
+grpc_cpp_plugin sha256 995ca8ac620fe83532b649a7c8c0a9341c7003da927fe0e4a8f821bfc579206d
+```
+
+```bash
+cmake -S packages/node/native -B build/native \
+  -DCMAKE_PREFIX_PATH="$PWD/build/native-toolchain"
+cmake --build build/native -j"$(nproc)"
+test -f build/native/shard_runtime.grpc.pb.cc
+test -f build/native/shard_runtime.grpc.pb.h
+test -f build/native/libshard_runtime_grpc.a
+ctest --test-dir build/native --output-on-failure
+```
+
+```text
+Pinned gRPC 1.82.1: building ShardRuntime service stubs
+shard_runtime_proto built
+shard_runtime_grpc built
+1/1 shard_protocol_conformance passed
+```
+
+```bash
+python3 -m pytest -q tests/test_native_shard_protocol.py
+```
+
+```text
+50 passed, 2 optional-path skips
+```
+
+All DGR-023-required checks were selected explicitly:
+
+```bash
+python3 -m pytest -q -rs tests/test_native_shard_protocol.py \
+  -k 'cpp_and_python_agree_byte_for_byte or generated_python_stubs_match_the_proto or native_toolchain_bootstrap or wrong_grpcio'
+```
+
+```text
+4 passed, 48 deselected
+```
+
+```bash
+python3 scripts/generate_native_protocol.py --check
+python3 scripts/generate_protocol_goldens.py --check
+python3 scripts/ralph_prd_schema.py validate .scratch/distributed-gguf-runtime/prd.json
+python3 -m compileall -q packages tests
+git diff --check
+```
+
+```text
+generated stubs are up to date
+conformance vectors are up to date
+OK: 55 stories validated
+compileall passed
+git diff --check passed
+```
+
+## Changed files
+
+- `scripts/bootstrap_native_toolchain.sh`
+- `scripts/generate_native_protocol.py`
+- `packages/node/native/CMakeLists.txt`
+- `packages/node/native/README.md`
+- `tests/test_native_shard_protocol.py`
+- `.scratch/distributed-gguf-runtime/evidence/DGR-023/README.md`
+- `.scratch/distributed-gguf-runtime/prd.json` (secondary completion projection only)
+
+## Limitations and dependency handoff
+
+- This story proves exact schema/message/service generation and cross-language conformance. It does not implement or run the standalone worker service itself; DGR-033/DGR-037 own worker behavior.
+- The plugin SHA is evidence for this verified build. Reproducibility authority is the exact gRPC commit plus its submodule graph, not an assumption that different compilers produce byte-identical executables.
+- No model, GPU, API credits, or model download was used.
+- DGR-024 and DGR-037 may consume this completed generation dependency but must provide their own transport/worker evidence.

.scratch/distributed-gguf-runtime/issues/023-make-python-and-c-protobuf-generation-reproducible.md

@@ -1,7 +1,7 @@
 <!-- GENERATED FROM prd.json — DO NOT EDIT AS AN INDEPENDENT SOURCE. prd.json IS AUTHORITATIVE. -->
 # DGR-023: Make Python and C++ protobuf generation reproducible
 
-- **Status / triage:** specification only; `ready-for-agent`; `passes: false`
+- **Status / triage:** completed; `passes: true`
 - **Execution mode:** `AFK`
 - **Milestone:** `M1`
 - **Dependencies:** `DGR-021`
@@ -18,11 +18,11 @@ Fresh Ralph session: read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md`,
 
 ## Acceptance criteria
 
-- [ ] Pin protoc, gRPC, and plugin versions or declare a verified compatible range.
-- [ ] Generate Python and C++ bindings into out-of-tree build/package locations through documented commands.
-- [ ] Add Python↔C++ round-trip and descriptor compatibility tests.
-- [ ] A clean checkout regenerates bindings deterministically or fails with an actionable toolchain error.
-- [ ] Applicable shared quality gates in `prd.json` pass, and the evidence handoff records exact commands/results, changed files, limitations, and dependency handoff.
+- [x] Pin protoc, gRPC, and plugin versions or declare a verified compatible range.
+- [x] Generate Python and C++ bindings into out-of-tree build/package locations through documented commands.
+- [x] Add Python↔C++ round-trip and descriptor compatibility tests.
+- [x] A clean checkout regenerates bindings deterministically or fails with an actionable toolchain error.
+- [x] Applicable shared quality gates in `prd.json` pass, and the evidence handoff records exact commands/results, changed files, limitations, and dependency handoff.
 
 ## Shared quality gates
 
@@ -36,4 +36,4 @@ Fresh Ralph session: read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md`,
 
 ## Evidence handoff
 
-Write and verify `.scratch/distributed-gguf-runtime/evidence/DGR-023/README.md`. Until every criterion and applicable gate has real evidence, this story remains `passes: false`. Legacy evidence is provenance only, not completion credit.
+Verified evidence: `.scratch/distributed-gguf-runtime/evidence/DGR-023/README.md`. Legacy evidence remains provenance only and grants no implementation completion credit.

.scratch/distributed-gguf-runtime/issues/024-implement-real-generated-grpc-protocol-harness.md

@@ -0,0 +1,39 @@
+<!-- GENERATED FROM prd.json — DO NOT EDIT AS AN INDEPENDENT SOURCE. prd.json IS AUTHORITATIVE. -->
+# DGR-024: Implement real generated-gRPC protocol harness
+
+- **Status / triage:** specification only; `ready-for-agent`; `passes: false`
+- **Execution mode:** `AFK`
+- **Milestone:** `M1`
+- **Dependencies:** `DGR-022`, `DGR-023`
+- **Blocks (derived):** `DGR-033`, `DGR-042`
+- **Labels:** `area:protocol`, `area:testing`, `type:vertical-slice`, `priority:p0`, `ready-for-agent`
+- **Evidence class:** `model-free`
+- **Hardware:** `none`
+- **Model:** `none`
+- **Upstream:** `no`
+
+## Objective / description
+
+Build a real generated-gRPC protocol harness around the versioned shard_runtime.proto contract. Use generated Python and C++ stubs over an actual localhost transport and real process lifecycle; exercise captured deterministic protocol vectors and serialized protobuf bytes before a real model worker exists. Do not implement an in-memory fake transport, synthetic model outputs, or a production-looking stub/demo service.
+
+## Acceptance criteria
+
+- [ ] Start a real localhost gRPC server process using generated bindings and connect to it with a generated client; no in-memory fake channel or direct method-only seam.
+- [ ] Exercise prefill fragments, decode frames, release, cancel, flow-control, deadlines, malformed input, checksum failure, duplicates, and stale epochs using serialized protocol messages and captured deterministic vectors.
+- [ ] Prove direct and opaque-relay paths preserve identical protobuf bytes by recording and comparing actual wire frames at both boundaries.
+- [ ] Use real process/socket lifecycle and fail closed on transport, schema, epoch, size, cache, and deadline violations; do not claim model or accelerator behavior that is not exercised.
+- [ ] Applicable shared quality gates pass, and evidence records exact commands, raw outputs, generated artifact identities, wire-frame hashes, changed files, limitations, and dependency handoff.
+
+## Shared quality gates
+
+- Targeted deterministic tests pass; Python changes also pass `python -m compileall packages tests`.
+- `git diff --check` passes.
+- Default tests are model-download-free, API-credit-free, and GPU-free.
+- Evidence README records exact changed files, commands/results, limitations, and dependency handoff; no fabricated evidence or inherited completion credit.
+- Native changes pass focused out-of-tree CMake build and CTest; patch changes verify clean apply/check/reverse against the exact llama.cpp pin.
+- Runs are opt-in and record exact artifact/split hashes, runtime/upstream pin, backend/driver, hardware, network, commands, and raw metrics. Model artifacts use configured mounted-drive storage and never `/home`.
+- Preserve existing Transformers behavior and backend-agnostic Tracker routing/load balancing/billing/relay semantics unless an explicit versioned contract says otherwise. One scoped story commit is expected during execution, but this specification-materialization change is not committed.
+
+## Evidence handoff
+
+Write and verify `.scratch/distributed-gguf-runtime/evidence/DGR-024/README.md`. Until every criterion and applicable gate has real evidence, this story remains `passes: false`. Legacy evidence is provenance only, not completion credit.

.scratch/distributed-gguf-runtime/prd.json

@@ -522,8 +522,9 @@
         "A clean checkout regenerates bindings deterministically or fails with an actionable toolchain error.",
         "Applicable shared quality gates in `prd.json` pass, and the evidence handoff records exact commands/results, changed files, limitations, and dependency handoff."
       ],
-      "passes": false,
-      "notes": "Generated source issue: .scratch/distributed-gguf-runtime/issues/023-make-python-and-c-protobuf-generation-reproducible.md; prd.json is authoritative.",
+      "passes": true,
+      "notes": "Completed from Gitea #7 after controller provisioned and exercised the exact Python/C++ toolchains. Verified deterministic generation, native CMake/CTest, Python↔C++ byte parity, compileall, and diff checks; fixed relative bootstrap prefix resolution.",
+      "completionNotes": "Verified exact grpcio-tools 1.82.1, Protobuf 33.1, Abseil 20250814.1, and gRPC C++ 1.82.1 at commit acccf84c0df20487d64101f528e5d426541ca4e5. Mandatory Python/C++ message and service generation, native CTest, deterministic regeneration, and byte-for-byte Python/C++ parity passed; see evidence/DGR-023/README.md.",
       "blocks": [
         "DGR-024",
         "DGR-037"
@@ -531,7 +532,7 @@
     },
     {
       "id": "DGR-024",
-      "title": "Implement in-memory fake gRPC seam transport",
+      "title": "Implement real generated-gRPC protocol harness",
       "priority": 8,
       "milestone": "M1",
       "executionMode": "AFK",
@@ -542,26 +543,26 @@
         "priority:p0",
         "ready-for-agent"
       ],
-      "evidenceClass": "fixture",
+      "evidenceClass": "model-free",
       "evidencePath": ".scratch/distributed-gguf-runtime/evidence/DGR-024/README.md",
       "hardware": "none",
-      "model": "fake",
+      "model": "none",
       "upstream": "no",
       "dependsOn": [
         "DGR-022",
         "DGR-023"
       ],
       "triage": "ready-for-agent",
-      "description": "Fresh Ralph session: read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md`, source issue `.scratch/distributed-gguf-runtime/issues/024-implement-in-memory-fake-grpc-seam-transport.md`, and evidence READMEs for dependencies (DGR-022, DGR-023) before changing code. Inspect live source/tests rather than trusting legacy pass states. Objective: Exercise the complete streaming protocol deterministically before a real model or worker exists.",
+      "description": "Build a real generated-gRPC protocol harness around the versioned shard_runtime.proto contract. Use generated Python and C++ stubs over an actual localhost transport and real process lifecycle; exercise captured deterministic protocol vectors and serialized protobuf bytes before a real model worker exists. Do not implement an in-memory fake transport, synthetic model outputs, or a production-looking stub/demo service.",
       "acceptanceCriteria": [
-        "Provide a fake bidirectional stream supporting prefill fragments, decode fast-path frames, release, cancel, and structured errors.",
-        "Test flow-control blocking, deadlines, malformed fragments, checksum failure, duplicates, and stale epochs.",
-        "Verify direct and opaque-relay framing preserve identical protobuf bytes.",
-        "Tests require no sockets outside localhost, model downloads, or native accelerator.",
-        "Applicable shared quality gates in `prd.json` pass, and the evidence handoff records exact commands/results, changed files, limitations, and dependency handoff."
+        "Start a real localhost gRPC server process using generated bindings and connect to it with a generated client; no in-memory fake channel or direct method-only seam.",
+        "Exercise prefill fragments, decode frames, release, cancel, flow-control, deadlines, malformed input, checksum failure, duplicates, and stale epochs using serialized protocol messages and captured deterministic vectors.",
+        "Prove direct and opaque-relay paths preserve identical protobuf bytes by recording and comparing actual wire frames at both boundaries.",
+        "Use real process/socket lifecycle and fail closed on transport, schema, epoch, size, cache, and deadline violations; do not claim model or accelerator behavior that is not exercised.",
+        "Applicable shared quality gates pass, and evidence records exact commands, raw outputs, generated artifact identities, wire-frame hashes, changed files, limitations, and dependency handoff."
       ],
       "passes": false,
-      "notes": "Generated source issue: .scratch/distributed-gguf-runtime/issues/024-implement-in-memory-fake-grpc-seam-transport.md; prd.json is authoritative.",
+      "notes": "Revised by policy audit: the former in-memory fake/stub seam task was invalid under the no-fake-data/no-demo-implementation rule. Existing fake-seam work is preserved as unaccepted historical material and must not be integrated.",
       "blocks": [
         "DGR-033",
         "DGR-042"

packages/node/native/CMakeLists.txt

@@ -4,10 +4,8 @@
 # never committed. A C++ consumer already needs a toolchain, so committing
 # generated C++ would only create a second copy of the schema that can rot.
 #
-# gRPC C++ is optional here on purpose. The conformance test only needs message
-# types, so the schema can be verified on a machine that has protobuf but not
-# the gRPC C++ stack. When gRPC *is* found, the service stubs are generated too
-# and exported as `shard_runtime_grpc` for the worker (DGR-008) to link.
+# Protobuf and gRPC C++ are required together so message and service bindings are
+# generated by one exact toolchain. The ignored bootstrap prefix supplies both.
 #
 # Build:
 #   cmake -S packages/node/native -B build/native -DCMAKE_PREFIX_PATH=<protobuf-install>
@@ -23,8 +21,17 @@ project(meshnet_shard_protocol CXX)
 set(CMAKE_CXX_STANDARD 17)
 set(CMAKE_CXX_STANDARD_REQUIRED ON)
 
-find_package(protobuf CONFIG REQUIRED)
-find_package(gRPC CONFIG QUIET)
+# Protobuf and gRPC are one pinned generation toolchain. Configure only against
+# the ignored prefix produced by scripts/bootstrap_native_toolchain.sh; accepting
+# an arbitrary system plugin would make generated service bindings host-dependent.
+set(MESHNET_PROTOBUF_VERSION "33.1.0")
+set(MESHNET_GRPC_VERSION "1.82.1")
+find_package(protobuf ${MESHNET_PROTOBUF_VERSION} EXACT CONFIG REQUIRED)
+find_package(gRPC ${MESHNET_GRPC_VERSION} EXACT CONFIG REQUIRED)
+
+if(NOT TARGET gRPC::grpc_cpp_plugin)
+  message(FATAL_ERROR "pinned gRPC package does not export grpc_cpp_plugin")
+endif()
 
 set(SHARD_PROTO "${CMAKE_CURRENT_SOURCE_DIR}/proto/shard_runtime.proto")
 
@@ -39,8 +46,7 @@ protobuf_generate(
   PROTOC_OUT_DIR "${CMAKE_CURRENT_BINARY_DIR}"
 )
 
-# Service stubs: only when the gRPC C++ stack is present.
-if(gRPC_FOUND)
+# Service stubs are part of the reproducibility contract, not an optional branch.
 add_library(shard_runtime_grpc STATIC "${SHARD_PROTO}")
 target_link_libraries(shard_runtime_grpc PUBLIC shard_runtime_proto gRPC::grpc++)
 target_include_directories(shard_runtime_grpc PUBLIC "${CMAKE_CURRENT_BINARY_DIR}")
@@ -52,11 +58,7 @@ if(gRPC_FOUND)
   IMPORT_DIRS "${CMAKE_CURRENT_SOURCE_DIR}/proto"
   PROTOC_OUT_DIR "${CMAKE_CURRENT_BINARY_DIR}"
 )
-  message(STATUS "gRPC C++ found: building ShardRuntime service stubs")
-else()
-  message(STATUS "gRPC C++ not found: building message types only "
-                 "(sufficient for the conformance test)")
-endif()
+message(STATUS "Pinned gRPC ${gRPC_VERSION}: building ShardRuntime service stubs")
 
 enable_testing()
 

packages/node/native/README.md

@@ -46,7 +46,7 @@ sampled token to request/recipe identity and sampling/template/reasoning inputs.
 If the machine has no protobuf C++ toolchain:
 
 ```bash
-scripts/bootstrap_native_toolchain.sh build/native-toolchain
+bash scripts/bootstrap_native_toolchain.sh build/native-toolchain
 ```
 
 Then:
@@ -58,10 +58,10 @@ cmake --build build/native -j
 ctest --test-dir build/native --output-on-failure
 ```
 
-gRPC C++ is optional: without it, CMake builds the message types only, which is
-all the conformance test needs. When gRPC C++ *is* found, the `ShardRuntime`
-service stubs are built too and exported as `shard_runtime_grpc` for the worker
-(DGR-008) to link.
+The bootstrap pins and builds Protobuf `33.1`, gRPC C++ `1.82.1`, and the
+matching `grpc_cpp_plugin` into one ignored prefix. CMake requires those exact
+package versions and always generates both message and service stubs; it does
+not fall back to an arbitrary system plugin.
 
 ## How the cross-language check actually proves something
 

scripts/bootstrap_native_toolchain.sh

@@ -1,18 +1,13 @@
 #!/usr/bin/env bash
 # Build a protobuf C++ toolchain for the native Shard protocol.
 #
-# The Python side needs nothing beyond `pip install grpcio-tools` — it bundles
-# protoc. The C++ side needs libprotobuf headers and a protoc binary, and a
-# machine that has neither (no protobuf-devel, no cmake, no system protoc) can
-# still get a working one from source with this script. It is the exact recipe
-# DGR-002 used to build and run the C++ conformance test.
-#
-# gRPC C++ is deliberately NOT built here. The conformance test only needs
-# message types, so verifying the schema does not require the whole gRPC stack.
-# The worker (DGR-008) will need gRPC C++ and should extend this script then.
+# The Python side uses the exact grpcio-tools pin declared below. The C++ side
+# builds exact Protobuf, Abseil, and gRPC source revisions so `protoc`,
+# `grpc_cpp_plugin`, headers, and libraries all come from one ignored prefix.
+# No system Protobuf/gRPC installation is accepted by the documented build.
 #
 # Usage:
-#   scripts/bootstrap_native_toolchain.sh [install-prefix]
+#   bash scripts/bootstrap_native_toolchain.sh [install-prefix]
 #
 # Then:
 #   cmake -S packages/node/native -B build/native -DCMAKE_PREFIX_PATH=<prefix>
@@ -21,7 +16,17 @@
 
 set -euo pipefail
 
-PREFIX="${1:-${PWD}/build/native-toolchain}"
+resolve_prefix() {
+  local candidate="${1:-${PWD}/build/native-toolchain}"
+  realpath -m -- "${candidate}"
+}
+
+if [[ "${1:-}" == "--print-prefix" ]]; then
+  resolve_prefix "${2:-}"
+  exit 0
+fi
+
+PREFIX="$(resolve_prefix "${1:-}")"
 WORK="$(mktemp -d)"
 trap 'rm -rf "${WORK}"' EXIT
 
@@ -29,11 +34,15 @@ trap 'rm -rf "${WORK}"' EXIT
 # that stub is allowed to use, so these are exact, not floating.
 PROTOBUF_VERSION="33.1"
 ABSEIL_VERSION="20250814.1"
+GRPC_VERSION="1.82.1"
+GRPC_COMMIT="acccf84c0df20487d64101f528e5d426541ca4e5"
 
-command -v cmake >/dev/null || {
-  echo "cmake is required (pip install cmake==4.4.0)" >&2
+for tool in cmake curl git realpath sha256sum tar; do
+  command -v "${tool}" >/dev/null || {
+    echo "${tool} is required" >&2
     exit 1
   }
+done
 
 echo "--- fetching protobuf ${PROTOBUF_VERSION} and abseil ${ABSEIL_VERSION}"
 cd "${WORK}"
@@ -41,26 +50,77 @@ curl -sfL -o protobuf.tar.gz \
   "https://github.com/protocolbuffers/protobuf/releases/download/v${PROTOBUF_VERSION}/protobuf-${PROTOBUF_VERSION}.tar.gz"
 tar xzf protobuf.tar.gz
 
-# The protobuf release tarball ships utf8_range but not abseil, and its default
-# CMake provider expects abseil as a submodule, so vendor it into place.
 curl -sfL -o abseil.tar.gz \
   "https://github.com/abseil/abseil-cpp/releases/download/${ABSEIL_VERSION}/abseil-cpp-${ABSEIL_VERSION}.tar.gz"
 tar xzf abseil.tar.gz
-rm -rf "protobuf-${PROTOBUF_VERSION}/third_party/abseil-cpp"
-mv "abseil-cpp-${ABSEIL_VERSION}" "protobuf-${PROTOBUF_VERSION}/third_party/abseil-cpp"
 
-echo "--- building protobuf into ${PREFIX}"
-cmake -S "protobuf-${PROTOBUF_VERSION}" -B build \
+echo "--- building abseil ${ABSEIL_VERSION} into ${PREFIX}"
+cmake -S "abseil-cpp-${ABSEIL_VERSION}" -B abseil-build \
   -DCMAKE_BUILD_TYPE=Release \
   -DCMAKE_INSTALL_PREFIX="${PREFIX}" \
   -DCMAKE_POSITION_INDEPENDENT_CODE=ON \
-  -Dprotobuf_ABSL_PROVIDER=module \
-  -Dprotobuf_BUILD_TESTS=OFF \
-  -Dprotobuf_BUILD_SHARED_LIBS=OFF \
+  -DABSL_ENABLE_INSTALL=ON \
+  -DABSL_BUILD_TESTING=OFF \
   -DABSL_PROPAGATE_CXX_STD=ON
-cmake --build build -j"$(nproc)"
-cmake --install build
+cmake --build abseil-build -j"$(nproc)"
+cmake --install abseil-build
+
+echo "--- building protobuf ${PROTOBUF_VERSION} into ${PREFIX}"
+cmake -S "protobuf-${PROTOBUF_VERSION}" -B protobuf-build \
+  -DCMAKE_BUILD_TYPE=Release \
+  -DCMAKE_INSTALL_PREFIX="${PREFIX}" \
+  -DCMAKE_PREFIX_PATH="${PREFIX}" \
+  -Dabsl_DIR="${PREFIX}/lib64/cmake/absl" \
+  -DCMAKE_POSITION_INDEPENDENT_CODE=ON \
+  -Dprotobuf_LOCAL_DEPENDENCIES_ONLY=ON \
+  -Dprotobuf_BUILD_TESTS=OFF \
+  -Dprotobuf_BUILD_SHARED_LIBS=OFF
+cmake --build protobuf-build -j"$(nproc)"
+cmake --install protobuf-build
+
+echo "--- fetching gRPC ${GRPC_VERSION} at ${GRPC_COMMIT}"
+git init -q grpc-source
+git -C grpc-source remote add origin https://github.com/grpc/grpc.git
+git -C grpc-source fetch --depth 1 origin "${GRPC_COMMIT}"
+git -C grpc-source checkout --detach FETCH_HEAD
+git -C grpc-source submodule update --init --recursive --depth 1
+[[ "$(git -C grpc-source rev-parse HEAD)" == "${GRPC_COMMIT}" ]] || {
+  echo "gRPC checkout identity mismatch" >&2
+  exit 1
+}
+
+echo "--- building gRPC ${GRPC_VERSION} and grpc_cpp_plugin into ${PREFIX}"
+cmake -S grpc-source -B grpc-build \
+  -DCMAKE_BUILD_TYPE=Release \
+  -DCMAKE_INSTALL_PREFIX="${PREFIX}" \
+  -DCMAKE_PREFIX_PATH="${PREFIX}" \
+  -DProtobuf_DIR="${PREFIX}/lib64/cmake/protobuf" \
+  -Dabsl_DIR="${PREFIX}/lib64/cmake/absl" \
+  -DgRPC_INSTALL=ON \
+  -DgRPC_BUILD_TESTS=OFF \
+  -DgRPC_PROTOBUF_PROVIDER=package \
+  -DgRPC_ABSL_PROVIDER=package \
+  -DgRPC_CARES_PROVIDER=module \
+  -DgRPC_RE2_PROVIDER=module \
+  -DgRPC_SSL_PROVIDER=module \
+  -DgRPC_ZLIB_PROVIDER=module \
+  -DgRPC_BUILD_GRPC_CPP_PLUGIN=ON \
+  -DgRPC_BUILD_GRPC_CSHARP_PLUGIN=OFF \
+  -DgRPC_BUILD_GRPC_NODE_PLUGIN=OFF \
+  -DgRPC_BUILD_GRPC_OBJECTIVE_C_PLUGIN=OFF \
+  -DgRPC_BUILD_GRPC_PHP_PLUGIN=OFF \
+  -DgRPC_BUILD_GRPC_PYTHON_PLUGIN=OFF \
+  -DgRPC_BUILD_GRPC_RUBY_PLUGIN=OFF
+cmake --build grpc-build -j"$(nproc)"
+cmake --install grpc-build
 
 echo "--- done"
 "${PREFIX}/bin/protoc" --version
+[[ -x "${PREFIX}/bin/grpc_cpp_plugin" ]] || {
+  echo "grpc_cpp_plugin was not installed" >&2
+  exit 1
+}
+printf 'gRPC %s commit %s\n' "${GRPC_VERSION}" "${GRPC_COMMIT}"
+printf 'grpc_cpp_plugin sha256 '
+sha256sum "${PREFIX}/bin/grpc_cpp_plugin" | cut -d' ' -f1
 echo "configure the protocol build with: -DCMAKE_PREFIX_PATH=${PREFIX}"

scripts/generate_native_protocol.py

@@ -20,6 +20,7 @@ already requires a toolchain and nothing is gained by committing them.
 from __future__ import annotations
 
 import argparse
+from importlib import metadata
 import pathlib
 import shutil
 import subprocess
@@ -38,15 +39,32 @@ REQUIRED_GRPCIO_TOOLS = "1.82.1"
 _HEADER = "# Generated by scripts/generate_native_protocol.py. Do not edit.\n"
 
 
-def _generate(into: pathlib.Path) -> None:
-    """Run protoc, writing generated modules into `into`."""
+def _require_grpcio_tools_version() -> None:
     try:
-        from grpc_tools import protoc
-    except ImportError:  # pragma: no cover - exercised only without the toolchain
+        actual = metadata.version("grpcio-tools")
+    except metadata.PackageNotFoundError:
         sys.exit(
             "grpc_tools is required to generate stubs:\n"
             f"    pip install grpcio-tools=={REQUIRED_GRPCIO_TOOLS}"
         )
+    if actual != REQUIRED_GRPCIO_TOOLS:
+        sys.exit(
+            "wrong grpcio-tools version for deterministic generation: "
+            f"found {actual}, require {REQUIRED_GRPCIO_TOOLS}\n"
+            f"    pip install --upgrade grpcio-tools=={REQUIRED_GRPCIO_TOOLS}"
+        )
+
+
+def _generate(into: pathlib.Path) -> None:
+    """Run the exactly pinned protoc, writing generated modules into `into`."""
+    _require_grpcio_tools_version()
+    try:
+        from grpc_tools import protoc
+    except ImportError:  # pragma: no cover - inconsistent/broken installation
+        sys.exit(
+            "grpcio-tools metadata exists but grpc_tools cannot be imported; reinstall it:\n"
+            f"    pip install --force-reinstall grpcio-tools=={REQUIRED_GRPCIO_TOOLS}"
+        )
 
     into.mkdir(parents=True, exist_ok=True)
     # grpc_tools bundles protoc and the well-known types, so generation needs no

tests/test_native_shard_protocol.py

@@ -612,6 +612,27 @@ def test_a_peer_still_sending_the_retired_field_does_not_corrupt_the_tensor():
     assert decode_tensor(pb.NamedTensor.FromString(wire)) == b"\xaa" * 32
 
 
+def test_native_toolchain_bootstrap_resolves_relative_prefix_before_temp_chdir(tmp_path):
+    script = REPO_ROOT / "scripts/bootstrap_native_toolchain.sh"
+    result = subprocess.run(
+        ["bash", str(script), "--print-prefix", "relative/toolchain"],
+        cwd=tmp_path,
+        capture_output=True,
+        text=True,
+    )
+
+    assert result.returncode == 0, result.stdout + result.stderr
+    assert pathlib.Path(result.stdout.strip()) == tmp_path / "relative/toolchain"
+
+
+def test_python_generator_rejects_the_wrong_grpcio_tools_version(monkeypatch):
+    from scripts import generate_native_protocol
+
+    monkeypatch.setattr(generate_native_protocol.metadata, "version", lambda _: "0.0.0")
+    with pytest.raises(SystemExit, match="found 0.0.0, require 1.82.1"):
+        generate_native_protocol._require_grpcio_tools_version()
+
+
 def test_generated_python_stubs_match_the_proto():
     pytest.importorskip("grpc_tools", reason="protoc toolchain is required to verify")
     result = subprocess.run(