# DGR-002 — Adopt the versioned gRPC Shard protocol

Status: **done**. Every acceptance criterion is met with real command output.
Evidence class: **synthetic/unit** — this story defines a schema and proves both
languages agree on it. No model, GPU, network peer or benchmark is involved, and
none is claimed.

## 1. Summary

`packages/node/native/proto/shard_runtime.proto` is now the semantic contract for
the native Shard data plane: Protocol Buffers over gRPC/HTTP2 (ADR-0020). Python
and C++ both generate from it, and a shared committed conformance vector proves
they encode it identically — byte for byte.

Design decisions worth carrying forward:

- **Everything gRPC gives you is *also* in the schema.** Deadline, cancellation,
  identity and flow control are carried as fields, not left to HTTP/2 metadata,
  because the existing relay carries these frames as **opaque binary**. A relayed
  frame has no HTTP/2 context to inherit a deadline or a channel identity from.
  If it is not in the schema, it does not survive the relay.
- **Cancellation is both in-band and out-of-band.** `CancelSignal` rides the
  stream; `Cancel` is also a unary RPC. A cancel that can only travel down a
  stream that flow control has wedged is not a cancel.
- **Checksums cover the uncompressed payload.** Compression is a per-hop
  transport decision (reusing the existing `activation_compression` policies), so
  a checksum over the compressed frame would be invalidated by a hop that merely
  chose differently.
- **Application-level flow-control credits, not just HTTP/2 windows.** HTTP/2
  bounds *bytes in flight*; it does not bound how much *work* a worker has queued,
  and a relayed frame gets no window at all. Credits bound queue occupancy and KV
  pressure, and negotiation takes the strictest bound of either peer so a sender
  cannot talk a worker into unbounded queues.

## 2. Files changed

New:

| Path | What |
|---|---|
| `packages/node/native/proto/shard_runtime.proto` | The schema (sha256 `9e211660…`, see `protocol.json`) |
| `packages/node/native/CMakeLists.txt` | C++ generation + build wiring + ctest |
| `packages/node/native/tests/test_shard_protocol_conformance.cpp` | C++ conformance test |
| `packages/node/native/testdata/*.binpb` | Committed cross-language vectors |
| `packages/node/native/README.md` | How to regenerate and build |
| `packages/node/meshnet_node/native_protocol/__init__.py` | Public Python surface |
| `packages/node/meshnet_node/native_protocol/codec.py` | Bundle encode/decode, fragmentation, CRC32C, chunking, FC negotiation |
| `packages/node/meshnet_node/native_protocol/conformance.py` | Canonical vectors shared by both languages |
| `packages/node/meshnet_node/native_protocol/generated/` | Generated Python stubs (committed) |
| `scripts/generate_native_protocol.py` | Python generation, with `--check` |
| `scripts/generate_protocol_goldens.py` | Vector generation, with `--check` |
| `scripts/bootstrap_native_toolchain.sh` | Builds protobuf C++ from source |
| `tests/test_native_shard_protocol.py` | 45 Python tests |

Modified:

- `packages/node/pyproject.toml` — added runtime floors `grpcio>=1.82.1` and
  `protobuf>=7.35.0`, matching the committed generated-code requirements; new
  `proto` extra pinning `grpcio-tools==1.82.1`.
- `packages/node/meshnet_node/activation_compression.py` — optional bounded zstd
  output for untrusted protocol frames; existing callers remain compatible.
- `packages/node/meshnet_node/native_protocol/__init__.py` — exports negotiated
  bound constants and whole-session-message validation.

The canonical PRD marks only DGR-002 passed. `git status` before this story was clean.

## 3. Commands and real results

See `commands.txt` for the exact ordered list. Results:

```
python scripts/generate_native_protocol.py --check    -> generated stubs are up to date
python scripts/generate_protocol_goldens.py --check   -> conformance vectors are up to date

cmake -S packages/node/native -B build/native -DCMAKE_PREFIX_PATH=/tmp/pbsrc/install
  -- gRPC C++ not found: building message types only (sufficient for the conformance test)
cmake --build build/native -j                         -> Built target shard_protocol_conformance
ctest --test-dir build/native --output-on-failure     -> 1/1 Test #1: shard_protocol_conformance ... Passed
                                                          100% tests passed out of 1

cmp build/native/cpp_roundtrip.binpb \
    packages/node/native/testdata/session_request_golden.binpb   -> identical (exit 0)

pytest -q tests/test_native_shard_protocol.py         -> 45 passed
pytest -q tests/test_native_shard_protocol.py \
  tests/test_activation_compression.py                -> 51 passed
pytest -q  (final full suite)                         -> 728 passed, 12 skipped
pytest -q tests/test_tracker_routing.py::test_tracker_dashboard_can_cancel_inflight_proxy
  (after an earlier flaky full-suite failure)          -> 1 passed, 1 passed, 1 passed
clean minimum-runtime import + codec smoke test        -> passed
  grpcio==1.82.1, protobuf==7.35.0
compileall -q packages tests                          -> OK (exit 0)
git diff --check                                      -> clean (exit 0)
```

The C++ lane was rebuilt from scratch by Ralph (`rm -rf build/native`) using only
the documented commands, and reproduced the same result. During controller
review the user explicitly chose not to repeat the destructive build-directory
cleanup, so the independent controller relied on the recorded CMake/CTest run
while reproducing every Python/generation/full-suite gate.

### Controller review corrections

Independent controller review found and fixed two classes of issue before
integration:

1. Generated stubs required gRPC 1.82.1 and Protobuf 7.35.0, while the initial
   package metadata allowed much older runtimes that could fail at import time.
2. Flow-control bounds were described but not enforced by the reference decoder.
   Tensor declarations, shape rank/dimensions, fragment/tensor counts, fragments,
   wire bodies, whole bundles, complete session messages (including envelope
   overhead), and zstd window/output expansion are now fail-closed against the
   negotiated/default bounds. Unspecified bundle versions, compression and
   checksums are rejected rather than interpreted as valid data.
3. Negotiated initial credits could exceed `max_inflight_chunks`; credits are now
   capped by the settled in-flight limit.

Controller results: protocol tests `45 passed`; protocol plus shared compression
tests `51 passed`; final full suite `728 passed, 12 skipped`. A clean environment
at the declared minimum gRPC/Protobuf runtime versions imported both generated
stub modules and round-tripped the codec. Generation checks, `compileall`, static
secret scan, and `git diff --check` all passed.

### Full-suite note — a pre-existing flaky test

`tests/test_tracker_routing.py::test_tracker_dashboard_can_cancel_inflight_proxy`
is **flaky on a clean tree, independent of this story**. Reproduction, run
*before any DGR-002 file existed* (working tree clean, `git status` empty):

```
pytest -q                                       -> 1 failed, 682 passed, 12 skipped
  FAILED tests/test_tracker_routing.py::test_tracker_dashboard_can_cancel_inflight_proxy

# same test, three consecutive isolated runs on the same clean tree:
pytest -q tests/test_tracker_routing.py::test_tracker_dashboard_can_cancel_inflight_proxy
  -> 1 passed in 1.76s
  -> 1 failed in 4.39s
  -> 1 passed in 1.10s
```

It is a timing race in proxy cancellation (a 3-second in-flight generation raced
against the cancel assertion), not a deterministic failure, and it touches no code
this story changes. One controller full-suite run reported exactly that one failure
(`1 failed, 719 passed, 12 skipped`); three immediate isolated retries all passed
in 1.11 seconds, and the final exact-code full suite was green (`728 passed,
12 skipped`). It is flagged for whoever owns the tracker cancel path and is **not**
fixed here, since silently touching another story's code is out of scope.

## 4. Acceptance criteria

| Criterion | Where it is proven |
|---|---|
| Schema for capability, health, session stream, release, cancellation | `shard_runtime.proto` `service ShardRuntime`; `test_service_exposes_capability_health_session_release_and_cancel` |
| One long-lived bidi stream per Activation Seam, with deadlines, cancellation, flow control, structured errors | `rpc Session (stream) returns (stream)`; `test_session_is_one_long_lived_bidirectional_stream`; `Envelope.deadline_unix_nanos`, `CancelSignal` + unary `Cancel`, `FlowControl`, `ShardError` |
| Bounded chunking for prefill; small decode fast path | `ChunkInfo` + `plan_prefill_chunks` (128-token bound, ADR-0008); `DecodeStep`; `test_prefill_is_split_into_bounded_token_aligned_chunks`, `test_decode_fast_path_is_much_smaller_than_a_full_envelope_chunk` |
| Envelope carries schema version, work id, session id, epoch, fingerprint, range/effective start, phase, position, idempotency step, cache expectation, compression, checksum | `Envelope` + `NamedTensor`; `test_envelope_carries_every_field_the_protocol_promises` asserts against the **descriptor**, so deleting a field from the `.proto` fails the test |
| Versioned named-tensor bundle: name, shape, dtype, byte order, fragments | `TensorBundle`/`NamedTensor`/`TensorFragment`; `test_named_tensor_bundle_is_versioned_and_fully_described`, `test_bundle_round_trips_multiple_named_tensors` |
| Round-trip + compatibility tests in Python and C++ | 45 Python tests; C++ `ctest` 1/1; cross-language byte equality |
| Targeted pytest passes | 45 passed |
| `compileall packages tests` | exit 0 |
| `git diff --check` | exit 0 |
| Default tests deterministic, download-free, credit-free, GPU-free | Pure in-memory protobuf; no model, no network, no GPU |
| Full deterministic pytest passes, or pre-existing failure recorded | Final exact-code run: 728 passed, 12 skipped; earlier sole flaky failure documented with clean-tree reproduction and 3/3 passing retries |

## 5. How the cross-language claim is actually earned

Two codecs that each round-trip their own output prove only that each is
self-consistent. Instead:

1. Python builds the canonical `SessionRequest` and commits its bytes.
2. The C++ test parses **those** bytes, asserts every field, recomputes the CRC32C
   **from the polynomial in independent C++ code**, reassembles the multi-fragment
   tensor, and re-serializes to `cpp_roundtrip.binpb`.
3. `test_cpp_and_python_agree_byte_for_byte` asserts that file equals the golden.

Compatibility is tested in both languages: an unknown field from a newer peer
survives a parse/serialize hop (a Shard forwards activations — silently stripping
fields would corrupt a route it is merely a waypoint on), and a sparse message
from an older peer parses to proto3 defaults.

## 6. Limitations and deferred work

- **gRPC C++ was not built or linked.** The C++ lane verifies the *schema* (message
  types), not a running gRPC C++ server, because this machine has no gRPC C++ stack
  and building it is a large dependency the conformance test does not need.
  `CMakeLists.txt` already generates and exports `shard_runtime_grpc` when
  `find_package(gRPC)` succeeds. **DGR-008 must install gRPC C++ and extend
  `scripts/bootstrap_native_toolchain.sh`.**
- **No wire is exercised.** No client, server, or stream lifecycle exists yet — no
  deadline actually fires, no credit is actually consumed. This story defines and
  proves the contract; DGR-008/DGR-009 implement it.
- The protobuf C++ toolchain used here was installed to `/tmp/pbsrc/install` (ephemeral).
  `scripts/bootstrap_native_toolchain.sh` reproduces it; prefer a durable prefix such
  as `build/native-toolchain`.
- `crc32c` has a pure-Python fallback (used here) and picks up `google_crc32c` when
  present. The fallback is byte-exact but slow; a worker on the hot path should install
  the native package. Not a correctness limitation.
- Compression on the wire is zstd-or-none only, matching the existing seam.

## 7. Compatibility and migration notes

- **This does not change the existing HTTP activation wire.** `X-Meshnet-Wire` stays
  at `2` and the legacy `/forward` path is untouched. The native protocol is a
  *separate* contract with its own `SchemaVersion`, starting at 1. Nothing in this
  story is on any live request path — it is additive.
- Semantics are deliberately preserved from the existing ADRs so the two transports
  mean the same thing: `effective_start_layer` (ADR-0012), `CacheMode`/`expected_past_len`
  and `ERROR_CODE_CACHE_MISS` mapping to today's HTTP 409 `cache_miss` (ADR-0022),
  bfloat16 boundary dtype and 128-token prefill chunks (ADR-0008), fingerprint/recipe
  identity mirroring the capability report (ADR-0023).
- `TensorFragment` field 5 (`uncompressed_size`) is **reserved**: it was removed
  because `NamedTensor.total_bytes` is the single source of truth. Never recycle it —
  a recycled field number is the one schema change peers cannot detect, because the
  bytes still parse.
- Committed Python stubs are guarded by `--check` in the test suite, so they cannot
  drift from the schema unnoticed.

## 8. Handoff to dependent stories

- **DGR-003 (runtime recipe/fingerprint):** populate `Fingerprint`
  (`model_artifact_digest`, `runtime_recipe_digest`, `recipe_id`, `recipe_version`,
  `catalogue_version`). The mismatch outcome is already specified:
  `ERROR_CODE_FINGERPRINT_MISMATCH`. Do not invent a second identity struct.
- **DGR-005/006 (range loading, architecture boundary):** the boundary payload is a
  **named bundle**, not a bare tensor — a boundary needing more than one tensor is
  already representable. Execute `[effective_start_layer, end_layer)`, never from
  `start_layer`.
- **DGR-007 (concurrent sessions/KV):** isolate on `(route_session_id, route_epoch)`.
  `CacheExpectation`/`CacheResult` and `ERROR_CODE_CACHE_MISS` are the contract; a
  decode step whose `expected_past_len` does not match **must** miss, never fall back
  to a silent stateless forward. `idempotency_step` means a retried step is
  acknowledged (`Ack.duplicate`), not re-applied — re-applying advances the KV cache
  twice and desynchronises the route.
- **DGR-008 (C++ worker):** link `shard_runtime_grpc` from `CMakeLists.txt`; you must
  first install gRPC C++ (see limitations). Honour `FlowControl` credits and the
  `max_chunk_bytes` bound. Use `packages/node/meshnet_node/native_protocol/codec.py`
  as the reference for fragment reassembly and checksum validation.
- **DGR-009 (Meshnet integration):** the relay may carry these serialized frames as
  opaque binary — that is exactly why deadline/cancel/identity are in-band. Do not add
  a second control plane.
- **Anyone editing the schema:** run both `--check` scripts; if a vector legitimately
  changes, regenerate it and say so, because the C++ test asserts those exact bytes.