diff --git a/.scratch/distributed-gguf-runtime/evidence/DGR-023/README.md b/.scratch/distributed-gguf-runtime/evidence/DGR-023/README.md new file mode 100644 index 0000000..8a29915 --- /dev/null +++ b/.scratch/distributed-gguf-runtime/evidence/DGR-023/README.md @@ -0,0 +1,126 @@ +# DGR-023 evidence — reproducible Python and C++ protobuf/gRPC generation + +**Status:** complete after controller verification and independent-review repairs on 2026-07-17. + +**Authority:** live Gitea issue #7. The local PRD is a secondary projection. + +## Implemented contract + +- Python generation requires exactly `grpcio-tools==1.82.1`; the generator checks installed distribution metadata and rejects missing or different versions with an actionable exact install command. +- The C++ bootstrap builds one ignored toolchain prefix from exact inputs: + - Protobuf release `33.1` (`protobuf-config` version `33.1.0`); + - Abseil release `20250814.1`; + - gRPC C++ `1.82.1` at commit `acccf84c0df20487d64101f528e5d426541ca4e5`; + - gRPC's exact-commit submodules for c-ares, RE2, OpenSSL, and zlib. +- Protobuf is configured with local dependencies only after the exact Abseil build. gRPC uses the installed Protobuf/Abseil packages and commit-pinned module dependencies, avoiding unpinned system development packages and download fallbacks. +- CMake requires exact Protobuf `33.1.0` and gRPC `1.82.1`, requires the exported `gRPC::grpc_cpp_plugin` target, and always generates/builds both message and service stubs in the ignored build tree. +- Python bindings remain committed package output; `--check` regenerates into a temporary directory and compares output. C++ bindings are never committed. +- The C++ conformance test parses Python-produced vectors, validates fields/CRC32C, and emits `cpp_roundtrip.binpb`; Python compares that artifact byte-for-byte. + +## Defects found and fixed + +1. A relative bootstrap prefix was resolved after entering the temporary source directory, so successful output was deleted by cleanup. The script now canonicalizes the caller-relative destination first. The regression executes `--print-prefix` from a temporary working directory and validates the resulting path behavior. +2. The original native path omitted gRPC C++ and accepted any discoverable plugin. The bootstrap now builds exact gRPC/plugin sources, and CMake rejects absent/incompatible versions. +3. The Python script named the `grpcio-tools` pin but did not validate the installed distribution. It now refuses mismatched versions. +4. Protobuf ignored a stale provider option and attempted to download a different Abseil. The build was stopped; exact Abseil is now built first and Protobuf uses `LOCAL_DEPENDENCIES_ONLY`. +5. The host lacked OpenSSL development headers. Rather than add a floating system dependency, gRPC now uses the submodule pinned by its exact commit. +6. Documentation uses `bash scripts/bootstrap_native_toolchain.sh ...`, so a normal checkout does not depend on executable-mode preservation. + +## Verified toolchain + +```text +cmake version 4.4.0 +c++ (GCC) 15.2.1 20260123 (Red Hat 15.2.1-7) +libprotoc 33.1 +protobuf CMake package 33.1.0 +grpcio-tools 1.82.1 +grpcio 1.82.1 +protobuf Python runtime 7.35.1 +gRPC C++ 1.82.1 +commit acccf84c0df20487d64101f528e5d426541ca4e5 +grpc_cpp_plugin sha256 995ca8ac620fe83532b649a7c8c0a9341c7003da927fe0e4a8f821bfc579206d +``` + +The native toolchain and generated/build artifacts live under ignored mounted-drive `build/` paths; model/build artifacts were not stored under `/home`. + +## Commands and results + +```bash +bash scripts/bootstrap_native_toolchain.sh build/native-toolchain +``` + +```text +passed from a clean build directory +libprotoc 33.1 +gRPC 1.82.1 commit acccf84c0df20487d64101f528e5d426541ca4e5 +grpc_cpp_plugin sha256 995ca8ac620fe83532b649a7c8c0a9341c7003da927fe0e4a8f821bfc579206d +``` + +```bash +cmake -S packages/node/native -B build/native \ + -DCMAKE_PREFIX_PATH="$PWD/build/native-toolchain" +cmake --build build/native -j"$(nproc)" +test -f build/native/shard_runtime.grpc.pb.cc +test -f build/native/shard_runtime.grpc.pb.h +test -f build/native/libshard_runtime_grpc.a +ctest --test-dir build/native --output-on-failure +``` + +```text +Pinned gRPC 1.82.1: building ShardRuntime service stubs +shard_runtime_proto built +shard_runtime_grpc built +1/1 shard_protocol_conformance passed +``` + +```bash +python3 -m pytest -q tests/test_native_shard_protocol.py +``` + +```text +50 passed, 2 optional-path skips +``` + +All DGR-023-required checks were selected explicitly: + +```bash +python3 -m pytest -q -rs tests/test_native_shard_protocol.py \ + -k 'cpp_and_python_agree_byte_for_byte or generated_python_stubs_match_the_proto or native_toolchain_bootstrap or wrong_grpcio' +``` + +```text +4 passed, 48 deselected +``` + +```bash +python3 scripts/generate_native_protocol.py --check +python3 scripts/generate_protocol_goldens.py --check +python3 scripts/ralph_prd_schema.py validate .scratch/distributed-gguf-runtime/prd.json +python3 -m compileall -q packages tests +git diff --check +``` + +```text +generated stubs are up to date +conformance vectors are up to date +OK: 55 stories validated +compileall passed +git diff --check passed +``` + +## Changed files + +- `scripts/bootstrap_native_toolchain.sh` +- `scripts/generate_native_protocol.py` +- `packages/node/native/CMakeLists.txt` +- `packages/node/native/README.md` +- `tests/test_native_shard_protocol.py` +- `.scratch/distributed-gguf-runtime/evidence/DGR-023/README.md` +- `.scratch/distributed-gguf-runtime/prd.json` (secondary completion projection only) + +## Limitations and dependency handoff + +- This story proves exact schema/message/service generation and cross-language conformance. It does not implement or run the standalone worker service itself; DGR-033/DGR-037 own worker behavior. +- The plugin SHA is evidence for this verified build. Reproducibility authority is the exact gRPC commit plus its submodule graph, not an assumption that different compilers produce byte-identical executables. +- No model, GPU, API credits, or model download was used. +- DGR-024 and DGR-037 may consume this completed generation dependency but must provide their own transport/worker evidence. diff --git a/.scratch/distributed-gguf-runtime/prd.json b/.scratch/distributed-gguf-runtime/prd.json index 246b066..398d9d0 100644 --- a/.scratch/distributed-gguf-runtime/prd.json +++ b/.scratch/distributed-gguf-runtime/prd.json @@ -522,8 +522,8 @@ "A clean checkout regenerates bindings deterministically or fails with an actionable toolchain error.", "Applicable shared quality gates in `prd.json` pass, and the evidence handoff records exact commands/results, changed files, limitations, and dependency handoff." ], - "passes": false, - "notes": "Generated source issue: .scratch/distributed-gguf-runtime/issues/023-make-python-and-c-protobuf-generation-reproducible.md; prd.json is authoritative.", + "passes": true, + "notes": "Completed from Gitea #7 after controller provisioned and exercised the exact Python/C++ toolchains. Verified deterministic generation, native CMake/CTest, Python↔C++ byte parity, compileall, and diff checks; fixed relative bootstrap prefix resolution.", "blocks": [ "DGR-024", "DGR-037" diff --git a/packages/node/native/CMakeLists.txt b/packages/node/native/CMakeLists.txt index 71aed7a..5622929 100644 --- a/packages/node/native/CMakeLists.txt +++ b/packages/node/native/CMakeLists.txt @@ -4,10 +4,8 @@ # never committed. A C++ consumer already needs a toolchain, so committing # generated C++ would only create a second copy of the schema that can rot. # -# gRPC C++ is optional here on purpose. The conformance test only needs message -# types, so the schema can be verified on a machine that has protobuf but not -# the gRPC C++ stack. When gRPC *is* found, the service stubs are generated too -# and exported as `shard_runtime_grpc` for the worker (DGR-008) to link. +# Protobuf and gRPC C++ are required together so message and service bindings are +# generated by one exact toolchain. The ignored bootstrap prefix supplies both. # # Build: # cmake -S packages/node/native -B build/native -DCMAKE_PREFIX_PATH= @@ -23,8 +21,17 @@ project(meshnet_shard_protocol CXX) set(CMAKE_CXX_STANDARD 17) set(CMAKE_CXX_STANDARD_REQUIRED ON) -find_package(protobuf CONFIG REQUIRED) -find_package(gRPC CONFIG QUIET) +# Protobuf and gRPC are one pinned generation toolchain. Configure only against +# the ignored prefix produced by scripts/bootstrap_native_toolchain.sh; accepting +# an arbitrary system plugin would make generated service bindings host-dependent. +set(MESHNET_PROTOBUF_VERSION "33.1.0") +set(MESHNET_GRPC_VERSION "1.82.1") +find_package(protobuf ${MESHNET_PROTOBUF_VERSION} EXACT CONFIG REQUIRED) +find_package(gRPC ${MESHNET_GRPC_VERSION} EXACT CONFIG REQUIRED) + +if(NOT TARGET gRPC::grpc_cpp_plugin) + message(FATAL_ERROR "pinned gRPC package does not export grpc_cpp_plugin") +endif() set(SHARD_PROTO "${CMAKE_CURRENT_SOURCE_DIR}/proto/shard_runtime.proto") @@ -39,24 +46,19 @@ protobuf_generate( PROTOC_OUT_DIR "${CMAKE_CURRENT_BINARY_DIR}" ) -# Service stubs: only when the gRPC C++ stack is present. -if(gRPC_FOUND) - add_library(shard_runtime_grpc STATIC "${SHARD_PROTO}") - target_link_libraries(shard_runtime_grpc PUBLIC shard_runtime_proto gRPC::grpc++) - target_include_directories(shard_runtime_grpc PUBLIC "${CMAKE_CURRENT_BINARY_DIR}") - protobuf_generate( - TARGET shard_runtime_grpc - LANGUAGE grpc - GENERATE_EXTENSIONS .grpc.pb.h .grpc.pb.cc - PLUGIN "protoc-gen-grpc=$" - IMPORT_DIRS "${CMAKE_CURRENT_SOURCE_DIR}/proto" - PROTOC_OUT_DIR "${CMAKE_CURRENT_BINARY_DIR}" - ) - message(STATUS "gRPC C++ found: building ShardRuntime service stubs") -else() - message(STATUS "gRPC C++ not found: building message types only " - "(sufficient for the conformance test)") -endif() +# Service stubs are part of the reproducibility contract, not an optional branch. +add_library(shard_runtime_grpc STATIC "${SHARD_PROTO}") +target_link_libraries(shard_runtime_grpc PUBLIC shard_runtime_proto gRPC::grpc++) +target_include_directories(shard_runtime_grpc PUBLIC "${CMAKE_CURRENT_BINARY_DIR}") +protobuf_generate( + TARGET shard_runtime_grpc + LANGUAGE grpc + GENERATE_EXTENSIONS .grpc.pb.h .grpc.pb.cc + PLUGIN "protoc-gen-grpc=$" + IMPORT_DIRS "${CMAKE_CURRENT_SOURCE_DIR}/proto" + PROTOC_OUT_DIR "${CMAKE_CURRENT_BINARY_DIR}" +) +message(STATUS "Pinned gRPC ${gRPC_VERSION}: building ShardRuntime service stubs") enable_testing() diff --git a/packages/node/native/README.md b/packages/node/native/README.md index 199b900..942c97c 100644 --- a/packages/node/native/README.md +++ b/packages/node/native/README.md @@ -46,7 +46,7 @@ sampled token to request/recipe identity and sampling/template/reasoning inputs. If the machine has no protobuf C++ toolchain: ```bash -scripts/bootstrap_native_toolchain.sh build/native-toolchain +bash scripts/bootstrap_native_toolchain.sh build/native-toolchain ``` Then: @@ -58,10 +58,10 @@ cmake --build build/native -j ctest --test-dir build/native --output-on-failure ``` -gRPC C++ is optional: without it, CMake builds the message types only, which is -all the conformance test needs. When gRPC C++ *is* found, the `ShardRuntime` -service stubs are built too and exported as `shard_runtime_grpc` for the worker -(DGR-008) to link. +The bootstrap pins and builds Protobuf `33.1`, gRPC C++ `1.82.1`, and the +matching `grpc_cpp_plugin` into one ignored prefix. CMake requires those exact +package versions and always generates both message and service stubs; it does +not fall back to an arbitrary system plugin. ## How the cross-language check actually proves something diff --git a/scripts/bootstrap_native_toolchain.sh b/scripts/bootstrap_native_toolchain.sh index f6dc846..5749ded 100644 --- a/scripts/bootstrap_native_toolchain.sh +++ b/scripts/bootstrap_native_toolchain.sh @@ -1,18 +1,13 @@ #!/usr/bin/env bash # Build a protobuf C++ toolchain for the native Shard protocol. # -# The Python side needs nothing beyond `pip install grpcio-tools` — it bundles -# protoc. The C++ side needs libprotobuf headers and a protoc binary, and a -# machine that has neither (no protobuf-devel, no cmake, no system protoc) can -# still get a working one from source with this script. It is the exact recipe -# DGR-002 used to build and run the C++ conformance test. -# -# gRPC C++ is deliberately NOT built here. The conformance test only needs -# message types, so verifying the schema does not require the whole gRPC stack. -# The worker (DGR-008) will need gRPC C++ and should extend this script then. +# The Python side uses the exact grpcio-tools pin declared below. The C++ side +# builds exact Protobuf, Abseil, and gRPC source revisions so `protoc`, +# `grpc_cpp_plugin`, headers, and libraries all come from one ignored prefix. +# No system Protobuf/gRPC installation is accepted by the documented build. # # Usage: -# scripts/bootstrap_native_toolchain.sh [install-prefix] +# bash scripts/bootstrap_native_toolchain.sh [install-prefix] # # Then: # cmake -S packages/node/native -B build/native -DCMAKE_PREFIX_PATH= @@ -21,7 +16,17 @@ set -euo pipefail -PREFIX="${1:-${PWD}/build/native-toolchain}" +resolve_prefix() { + local candidate="${1:-${PWD}/build/native-toolchain}" + realpath -m -- "${candidate}" +} + +if [[ "${1:-}" == "--print-prefix" ]]; then + resolve_prefix "${2:-}" + exit 0 +fi + +PREFIX="$(resolve_prefix "${1:-}")" WORK="$(mktemp -d)" trap 'rm -rf "${WORK}"' EXIT @@ -29,11 +34,15 @@ trap 'rm -rf "${WORK}"' EXIT # that stub is allowed to use, so these are exact, not floating. PROTOBUF_VERSION="33.1" ABSEIL_VERSION="20250814.1" +GRPC_VERSION="1.82.1" +GRPC_COMMIT="acccf84c0df20487d64101f528e5d426541ca4e5" -command -v cmake >/dev/null || { - echo "cmake is required (pip install cmake==4.4.0)" >&2 - exit 1 -} +for tool in cmake curl git realpath sha256sum tar; do + command -v "${tool}" >/dev/null || { + echo "${tool} is required" >&2 + exit 1 + } +done echo "--- fetching protobuf ${PROTOBUF_VERSION} and abseil ${ABSEIL_VERSION}" cd "${WORK}" @@ -41,26 +50,77 @@ curl -sfL -o protobuf.tar.gz \ "https://github.com/protocolbuffers/protobuf/releases/download/v${PROTOBUF_VERSION}/protobuf-${PROTOBUF_VERSION}.tar.gz" tar xzf protobuf.tar.gz -# The protobuf release tarball ships utf8_range but not abseil, and its default -# CMake provider expects abseil as a submodule, so vendor it into place. curl -sfL -o abseil.tar.gz \ "https://github.com/abseil/abseil-cpp/releases/download/${ABSEIL_VERSION}/abseil-cpp-${ABSEIL_VERSION}.tar.gz" tar xzf abseil.tar.gz -rm -rf "protobuf-${PROTOBUF_VERSION}/third_party/abseil-cpp" -mv "abseil-cpp-${ABSEIL_VERSION}" "protobuf-${PROTOBUF_VERSION}/third_party/abseil-cpp" -echo "--- building protobuf into ${PREFIX}" -cmake -S "protobuf-${PROTOBUF_VERSION}" -B build \ +echo "--- building abseil ${ABSEIL_VERSION} into ${PREFIX}" +cmake -S "abseil-cpp-${ABSEIL_VERSION}" -B abseil-build \ -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_INSTALL_PREFIX="${PREFIX}" \ -DCMAKE_POSITION_INDEPENDENT_CODE=ON \ - -Dprotobuf_ABSL_PROVIDER=module \ - -Dprotobuf_BUILD_TESTS=OFF \ - -Dprotobuf_BUILD_SHARED_LIBS=OFF \ + -DABSL_ENABLE_INSTALL=ON \ + -DABSL_BUILD_TESTING=OFF \ -DABSL_PROPAGATE_CXX_STD=ON -cmake --build build -j"$(nproc)" -cmake --install build +cmake --build abseil-build -j"$(nproc)" +cmake --install abseil-build + +echo "--- building protobuf ${PROTOBUF_VERSION} into ${PREFIX}" +cmake -S "protobuf-${PROTOBUF_VERSION}" -B protobuf-build \ + -DCMAKE_BUILD_TYPE=Release \ + -DCMAKE_INSTALL_PREFIX="${PREFIX}" \ + -DCMAKE_PREFIX_PATH="${PREFIX}" \ + -Dabsl_DIR="${PREFIX}/lib64/cmake/absl" \ + -DCMAKE_POSITION_INDEPENDENT_CODE=ON \ + -Dprotobuf_LOCAL_DEPENDENCIES_ONLY=ON \ + -Dprotobuf_BUILD_TESTS=OFF \ + -Dprotobuf_BUILD_SHARED_LIBS=OFF +cmake --build protobuf-build -j"$(nproc)" +cmake --install protobuf-build + +echo "--- fetching gRPC ${GRPC_VERSION} at ${GRPC_COMMIT}" +git init -q grpc-source +git -C grpc-source remote add origin https://github.com/grpc/grpc.git +git -C grpc-source fetch --depth 1 origin "${GRPC_COMMIT}" +git -C grpc-source checkout --detach FETCH_HEAD +git -C grpc-source submodule update --init --recursive --depth 1 +[[ "$(git -C grpc-source rev-parse HEAD)" == "${GRPC_COMMIT}" ]] || { + echo "gRPC checkout identity mismatch" >&2 + exit 1 +} + +echo "--- building gRPC ${GRPC_VERSION} and grpc_cpp_plugin into ${PREFIX}" +cmake -S grpc-source -B grpc-build \ + -DCMAKE_BUILD_TYPE=Release \ + -DCMAKE_INSTALL_PREFIX="${PREFIX}" \ + -DCMAKE_PREFIX_PATH="${PREFIX}" \ + -DProtobuf_DIR="${PREFIX}/lib64/cmake/protobuf" \ + -Dabsl_DIR="${PREFIX}/lib64/cmake/absl" \ + -DgRPC_INSTALL=ON \ + -DgRPC_BUILD_TESTS=OFF \ + -DgRPC_PROTOBUF_PROVIDER=package \ + -DgRPC_ABSL_PROVIDER=package \ + -DgRPC_CARES_PROVIDER=module \ + -DgRPC_RE2_PROVIDER=module \ + -DgRPC_SSL_PROVIDER=module \ + -DgRPC_ZLIB_PROVIDER=module \ + -DgRPC_BUILD_GRPC_CPP_PLUGIN=ON \ + -DgRPC_BUILD_GRPC_CSHARP_PLUGIN=OFF \ + -DgRPC_BUILD_GRPC_NODE_PLUGIN=OFF \ + -DgRPC_BUILD_GRPC_OBJECTIVE_C_PLUGIN=OFF \ + -DgRPC_BUILD_GRPC_PHP_PLUGIN=OFF \ + -DgRPC_BUILD_GRPC_PYTHON_PLUGIN=OFF \ + -DgRPC_BUILD_GRPC_RUBY_PLUGIN=OFF +cmake --build grpc-build -j"$(nproc)" +cmake --install grpc-build echo "--- done" "${PREFIX}/bin/protoc" --version +[[ -x "${PREFIX}/bin/grpc_cpp_plugin" ]] || { + echo "grpc_cpp_plugin was not installed" >&2 + exit 1 +} +printf 'gRPC %s commit %s\n' "${GRPC_VERSION}" "${GRPC_COMMIT}" +printf 'grpc_cpp_plugin sha256 ' +sha256sum "${PREFIX}/bin/grpc_cpp_plugin" | cut -d' ' -f1 echo "configure the protocol build with: -DCMAKE_PREFIX_PATH=${PREFIX}" diff --git a/scripts/generate_native_protocol.py b/scripts/generate_native_protocol.py index 6b3154f..4e31134 100644 --- a/scripts/generate_native_protocol.py +++ b/scripts/generate_native_protocol.py @@ -20,6 +20,7 @@ already requires a toolchain and nothing is gained by committing them. from __future__ import annotations import argparse +from importlib import metadata import pathlib import shutil import subprocess @@ -38,15 +39,32 @@ REQUIRED_GRPCIO_TOOLS = "1.82.1" _HEADER = "# Generated by scripts/generate_native_protocol.py. Do not edit.\n" -def _generate(into: pathlib.Path) -> None: - """Run protoc, writing generated modules into `into`.""" +def _require_grpcio_tools_version() -> None: try: - from grpc_tools import protoc - except ImportError: # pragma: no cover - exercised only without the toolchain + actual = metadata.version("grpcio-tools") + except metadata.PackageNotFoundError: sys.exit( "grpc_tools is required to generate stubs:\n" f" pip install grpcio-tools=={REQUIRED_GRPCIO_TOOLS}" ) + if actual != REQUIRED_GRPCIO_TOOLS: + sys.exit( + "wrong grpcio-tools version for deterministic generation: " + f"found {actual}, require {REQUIRED_GRPCIO_TOOLS}\n" + f" pip install --upgrade grpcio-tools=={REQUIRED_GRPCIO_TOOLS}" + ) + + +def _generate(into: pathlib.Path) -> None: + """Run the exactly pinned protoc, writing generated modules into `into`.""" + _require_grpcio_tools_version() + try: + from grpc_tools import protoc + except ImportError: # pragma: no cover - inconsistent/broken installation + sys.exit( + "grpcio-tools metadata exists but grpc_tools cannot be imported; reinstall it:\n" + f" pip install --force-reinstall grpcio-tools=={REQUIRED_GRPCIO_TOOLS}" + ) into.mkdir(parents=True, exist_ok=True) # grpc_tools bundles protoc and the well-known types, so generation needs no diff --git a/tests/test_native_shard_protocol.py b/tests/test_native_shard_protocol.py index 0410c62..2c44d55 100644 --- a/tests/test_native_shard_protocol.py +++ b/tests/test_native_shard_protocol.py @@ -612,6 +612,27 @@ def test_a_peer_still_sending_the_retired_field_does_not_corrupt_the_tensor(): assert decode_tensor(pb.NamedTensor.FromString(wire)) == b"\xaa" * 32 +def test_native_toolchain_bootstrap_resolves_relative_prefix_before_temp_chdir(tmp_path): + script = REPO_ROOT / "scripts/bootstrap_native_toolchain.sh" + result = subprocess.run( + ["bash", str(script), "--print-prefix", "relative/toolchain"], + cwd=tmp_path, + capture_output=True, + text=True, + ) + + assert result.returncode == 0, result.stdout + result.stderr + assert pathlib.Path(result.stdout.strip()) == tmp_path / "relative/toolchain" + + +def test_python_generator_rejects_the_wrong_grpcio_tools_version(monkeypatch): + from scripts import generate_native_protocol + + monkeypatch.setattr(generate_native_protocol.metadata, "version", lambda _: "0.0.0") + with pytest.raises(SystemExit, match="found 0.0.0, require 1.82.1"): + generate_native_protocol._require_grpcio_tools_version() + + def test_generated_python_stubs_match_the_proto(): pytest.importorskip("grpc_tools", reason="protoc toolchain is required to verify") result = subprocess.run(