neuron-tai/.scratch/distributed-gguf-runtime/evidence/DGR-023/README.md

# DGR-023 evidence — reproducible Python and C++ protobuf/gRPC generation

**Status:** complete after controller verification and independent-review repairs on 2026-07-17.

**Authority:** live Gitea issue #7. The local PRD is a secondary projection.

## Implemented contract

- Python generation requires exactly `grpcio-tools==1.82.1`; the generator checks installed distribution metadata and rejects missing or different versions with an actionable exact install command.
- The C++ bootstrap builds one ignored toolchain prefix from exact inputs:
  - Protobuf release `33.1` (`protobuf-config` version `33.1.0`);
  - Abseil release `20250814.1`;
  - gRPC C++ `1.82.1` at commit `acccf84c0df20487d64101f528e5d426541ca4e5`;
  - gRPC's exact-commit submodules for c-ares, RE2, OpenSSL, and zlib.
- Protobuf is configured with local dependencies only after the exact Abseil build. gRPC uses the installed Protobuf/Abseil packages and commit-pinned module dependencies, avoiding unpinned system development packages and download fallbacks.
- CMake requires exact Protobuf `33.1.0` and gRPC `1.82.1`, requires the exported `gRPC::grpc_cpp_plugin` target, and always generates/builds both message and service stubs in the ignored build tree.
- Python bindings remain committed package output; `--check` regenerates into a temporary directory and compares output. C++ bindings are never committed.
- The C++ conformance test parses Python-produced vectors, validates fields/CRC32C, and emits `cpp_roundtrip.binpb`; Python compares that artifact byte-for-byte.

## Defects found and fixed

1. A relative bootstrap prefix was resolved after entering the temporary source directory, so successful output was deleted by cleanup. The script now canonicalizes the caller-relative destination first. The regression executes `--print-prefix` from a temporary working directory and validates the resulting path behavior.
2. The original native path omitted gRPC C++ and accepted any discoverable plugin. The bootstrap now builds exact gRPC/plugin sources, and CMake rejects absent/incompatible versions.
3. The Python script named the `grpcio-tools` pin but did not validate the installed distribution. It now refuses mismatched versions.
4. Protobuf ignored a stale provider option and attempted to download a different Abseil. The build was stopped; exact Abseil is now built first and Protobuf uses `LOCAL_DEPENDENCIES_ONLY`.
5. The host lacked OpenSSL development headers. Rather than add a floating system dependency, gRPC now uses the submodule pinned by its exact commit.
6. Documentation uses `bash scripts/bootstrap_native_toolchain.sh ...`, so a normal checkout does not depend on executable-mode preservation.

## Verified toolchain

```text
cmake version 4.4.0
c++ (GCC) 15.2.1 20260123 (Red Hat 15.2.1-7)
libprotoc 33.1
protobuf CMake package 33.1.0
grpcio-tools 1.82.1
grpcio 1.82.1
protobuf Python runtime 7.35.1
gRPC C++ 1.82.1
commit acccf84c0df20487d64101f528e5d426541ca4e5
grpc_cpp_plugin sha256 995ca8ac620fe83532b649a7c8c0a9341c7003da927fe0e4a8f821bfc579206d
```

The native toolchain and generated/build artifacts live under ignored mounted-drive `build/` paths; model/build artifacts were not stored under `/home`.

## Commands and results

```bash
bash scripts/bootstrap_native_toolchain.sh build/native-toolchain
```

```text
passed from a clean build directory
libprotoc 33.1
gRPC 1.82.1 commit acccf84c0df20487d64101f528e5d426541ca4e5
grpc_cpp_plugin sha256 995ca8ac620fe83532b649a7c8c0a9341c7003da927fe0e4a8f821bfc579206d
```

```bash
cmake -S packages/node/native -B build/native \
  -DCMAKE_PREFIX_PATH="$PWD/build/native-toolchain"
cmake --build build/native -j"$(nproc)"
test -f build/native/shard_runtime.grpc.pb.cc
test -f build/native/shard_runtime.grpc.pb.h
test -f build/native/libshard_runtime_grpc.a
ctest --test-dir build/native --output-on-failure
```

```text
Pinned gRPC 1.82.1: building ShardRuntime service stubs
shard_runtime_proto built
shard_runtime_grpc built
1/1 shard_protocol_conformance passed
```

```bash
python3 -m pytest -q tests/test_native_shard_protocol.py
```

```text
50 passed, 2 optional-path skips
```

All DGR-023-required checks were selected explicitly:

```bash
python3 -m pytest -q -rs tests/test_native_shard_protocol.py \
  -k 'cpp_and_python_agree_byte_for_byte or generated_python_stubs_match_the_proto or native_toolchain_bootstrap or wrong_grpcio'
```

```text
4 passed, 48 deselected
```

```bash
python3 scripts/generate_native_protocol.py --check
python3 scripts/generate_protocol_goldens.py --check
python3 scripts/ralph_prd_schema.py validate .scratch/distributed-gguf-runtime/prd.json
python3 -m compileall -q packages tests
git diff --check
```

```text
generated stubs are up to date
conformance vectors are up to date
OK: 55 stories validated
compileall passed
git diff --check passed
```

## Changed files

- `scripts/bootstrap_native_toolchain.sh`
- `scripts/generate_native_protocol.py`
- `packages/node/native/CMakeLists.txt`
- `packages/node/native/README.md`
- `tests/test_native_shard_protocol.py`
- `.scratch/distributed-gguf-runtime/evidence/DGR-023/README.md`
- `.scratch/distributed-gguf-runtime/prd.json` (secondary completion projection only)

## Limitations and dependency handoff

- This story proves exact schema/message/service generation and cross-language conformance. It does not implement or run the standalone worker service itself; DGR-033/DGR-037 own worker behavior.
- The plugin SHA is evidence for this verified build. Reproducibility authority is the exact gRPC commit plus its submodule graph, not an assumption that different compilers produce byte-identical executables.
- No model, GPU, API credits, or model download was used.
- DGR-024 and DGR-037 may consume this completed generation dependency but must provide their own transport/worker evidence.