DGR-002 — Adopt the versioned gRPC Shard protocol

Status: done. Every acceptance criterion is met with real command output. Evidence class: synthetic/unit — this story defines a schema and proves both languages agree on it. No model, GPU, network peer or benchmark is involved, and none is claimed.

1. Summary

packages/node/native/proto/shard_runtime.proto is now the semantic contract for the native Shard data plane: Protocol Buffers over gRPC/HTTP2 (ADR-0020). Python and C++ both generate from it, and a shared committed conformance vector proves they encode it identically — byte for byte.

Design decisions worth carrying forward:

Everything gRPC gives you is also in the schema. Deadline, cancellation, identity and flow control are carried as fields, not left to HTTP/2 metadata, because the existing relay carries these frames as opaque binary. A relayed frame has no HTTP/2 context to inherit a deadline or a channel identity from. If it is not in the schema, it does not survive the relay.
Cancellation is both in-band and out-of-band. CancelSignal rides the stream; Cancel is also a unary RPC. A cancel that can only travel down a stream that flow control has wedged is not a cancel.
Checksums cover the uncompressed payload. Compression is a per-hop transport decision (reusing the existing activation_compression policies), so a checksum over the compressed frame would be invalidated by a hop that merely chose differently.
Application-level flow-control credits, not just HTTP/2 windows. HTTP/2 bounds bytes in flight; it does not bound how much work a worker has queued, and a relayed frame gets no window at all. Credits bound queue occupancy and KV pressure, and negotiation takes the strictest bound of either peer so a sender cannot talk a worker into unbounded queues.

2. Files changed

New:

Path	What
`packages/node/native/proto/shard_runtime.proto`	The schema (sha256 `077ee349…`, see `protocol.json`)
`packages/node/native/CMakeLists.txt`	C++ generation + build wiring + ctest
`packages/node/native/tests/test_shard_protocol_conformance.cpp`	C++ conformance test
`packages/node/native/testdata/*.binpb`	Committed cross-language vectors
`packages/node/native/README.md`	How to regenerate and build
`packages/node/meshnet_node/native_protocol/__init__.py`	Public Python surface
`packages/node/meshnet_node/native_protocol/codec.py`	Bundle encode/decode, fragmentation, CRC32C, chunking, FC negotiation
`packages/node/meshnet_node/native_protocol/conformance.py`	Canonical vectors shared by both languages
`packages/node/meshnet_node/native_protocol/generated/`	Generated Python stubs (committed)
`scripts/generate_native_protocol.py`	Python generation, with `--check`
`scripts/generate_protocol_goldens.py`	Vector generation, with `--check`
`scripts/bootstrap_native_toolchain.sh`	Builds protobuf C++ from source
`tests/test_native_shard_protocol.py`	29 Python tests

Modified:

packages/node/pyproject.toml — added grpcio>=1.60, protobuf>=5; new proto extra pinning grpcio-tools==1.82.1.

No other working-tree file was touched. git status before this story was clean.

3. Commands and real results

See commands.txt for the exact ordered list. Results:

python scripts/generate_native_protocol.py --check    -> generated stubs are up to date
python scripts/generate_protocol_goldens.py --check   -> conformance vectors are up to date

cmake -S packages/node/native -B build/native -DCMAKE_PREFIX_PATH=/tmp/pbsrc/install
  -- gRPC C++ not found: building message types only (sufficient for the conformance test)
cmake --build build/native -j                         -> Built target shard_protocol_conformance
ctest --test-dir build/native --output-on-failure     -> 1/1 Test #1: shard_protocol_conformance ... Passed
                                                          100% tests passed out of 1

cmp build/native/cpp_roundtrip.binpb \
    packages/node/native/testdata/session_request_golden.binpb   -> identical (exit 0)

pytest -q tests/test_native_shard_protocol.py         -> 29 passed
pytest -q  (full suite)                               -> 712 passed, 12 skipped
compileall -q packages tests                          -> OK (exit 0)
git diff --check                                      -> clean (exit 0)

The C++ lane was rebuilt from scratch (rm -rf build/native) using only the documented commands, and reproduced the same result.

Full-suite note — a pre-existing flaky test

tests/test_tracker_routing.py::test_tracker_dashboard_can_cancel_inflight_proxy is flaky on a clean tree, independent of this story. Reproduction, run before any DGR-002 file existed (working tree clean, git status empty):

pytest -q                                       -> 1 failed, 682 passed, 12 skipped
  FAILED tests/test_tracker_routing.py::test_tracker_dashboard_can_cancel_inflight_proxy

# same test, three consecutive isolated runs on the same clean tree:
pytest -q tests/test_tracker_routing.py::test_tracker_dashboard_can_cancel_inflight_proxy
  -> 1 passed in 1.76s
  -> 1 failed in 4.39s
  -> 1 passed in 1.10s

It is a timing race in proxy cancellation (a 3-second in-flight generation raced against the cancel assertion), not a deterministic failure, and it touches no code this story changes. The final full-suite run with DGR-002 applied was green (712 passed, 12 skipped) — 683 pre-existing tests plus this story's 29. Flagged for whoever owns the tracker cancel path; not fixed here, since silently touching another story's code is out of scope.

4. Acceptance criteria

Criterion	Where it is proven
Schema for capability, health, session stream, release, cancellation	`shard_runtime.proto` `service ShardRuntime`; `test_service_exposes_capability_health_session_release_and_cancel`
One long-lived bidi stream per Activation Seam, with deadlines, cancellation, flow control, structured errors	`rpc Session (stream) returns (stream)`; `test_session_is_one_long_lived_bidirectional_stream`; `Envelope.deadline_unix_nanos`, `CancelSignal` + unary `Cancel`, `FlowControl`, `ShardError`
Bounded chunking for prefill; small decode fast path	`ChunkInfo` + `plan_prefill_chunks` (128-token bound, ADR-0008); `DecodeStep`; `test_prefill_is_split_into_bounded_token_aligned_chunks`, `test_decode_fast_path_is_much_smaller_than_a_full_envelope_chunk`
Envelope carries schema version, work id, session id, epoch, fingerprint, range/effective start, phase, position, idempotency step, cache expectation, compression, checksum	`Envelope` + `NamedTensor`; `test_envelope_carries_every_field_the_protocol_promises` asserts against the descriptor, so deleting a field from the `.proto` fails the test
Versioned named-tensor bundle: name, shape, dtype, byte order, fragments	`TensorBundle`/`NamedTensor`/`TensorFragment`; `test_named_tensor_bundle_is_versioned_and_fully_described`, `test_bundle_round_trips_multiple_named_tensors`
Round-trip + compatibility tests in Python and C++	29 Python tests; C++ `ctest` 1/1; cross-language byte equality
Targeted pytest passes	29 passed
`compileall packages tests`	exit 0
`git diff --check`	exit 0
Default tests deterministic, download-free, credit-free, GPU-free	Pure in-memory protobuf; no model, no network, no GPU
Full deterministic pytest passes, or pre-existing failure recorded	712 passed, 12 skipped; flaky pre-existing test documented above with clean-tree reproduction

5. How the cross-language claim is actually earned

Two codecs that each round-trip their own output prove only that each is self-consistent. Instead:

Python builds the canonical SessionRequest and commits its bytes.
The C++ test parses those bytes, asserts every field, recomputes the CRC32C from the polynomial in independent C++ code, reassembles the multi-fragment tensor, and re-serializes to cpp_roundtrip.binpb.
test_cpp_and_python_agree_byte_for_byte asserts that file equals the golden.

Compatibility is tested in both languages: an unknown field from a newer peer survives a parse/serialize hop (a Shard forwards activations — silently stripping fields would corrupt a route it is merely a waypoint on), and a sparse message from an older peer parses to proto3 defaults.

6. Limitations and deferred work

gRPC C++ was not built or linked. The C++ lane verifies the schema (message types), not a running gRPC C++ server, because this machine has no gRPC C++ stack and building it is a large dependency the conformance test does not need. CMakeLists.txt already generates and exports shard_runtime_grpc when find_package(gRPC) succeeds. DGR-008 must install gRPC C++ and extend scripts/bootstrap_native_toolchain.sh.
No wire is exercised. No client, server, or stream lifecycle exists yet — no deadline actually fires, no credit is actually consumed. This story defines and proves the contract; DGR-008/DGR-009 implement it.
The protobuf C++ toolchain used here was installed to /tmp/pbsrc/install (ephemeral). scripts/bootstrap_native_toolchain.sh reproduces it; prefer a durable prefix such as build/native-toolchain.
crc32c has a pure-Python fallback (used here) and picks up google_crc32c when present. The fallback is byte-exact but slow; a worker on the hot path should install the native package. Not a correctness limitation.
Compression on the wire is zstd-or-none only, matching the existing seam.

7. Compatibility and migration notes

This does not change the existing HTTP activation wire. X-Meshnet-Wire stays at 2 and the legacy /forward path is untouched. The native protocol is a separate contract with its own SchemaVersion, starting at 1. Nothing in this story is on any live request path — it is additive.
Semantics are deliberately preserved from the existing ADRs so the two transports mean the same thing: effective_start_layer (ADR-0012), CacheMode/expected_past_len and ERROR_CODE_CACHE_MISS mapping to today's HTTP 409 cache_miss (ADR-0022), bfloat16 boundary dtype and 128-token prefill chunks (ADR-0008), fingerprint/recipe identity mirroring the capability report (ADR-0023).
TensorFragment field 5 (uncompressed_size) is reserved: it was removed because NamedTensor.total_bytes is the single source of truth. Never recycle it — a recycled field number is the one schema change peers cannot detect, because the bytes still parse.
Committed Python stubs are guarded by --check in the test suite, so they cannot drift from the schema unnoticed.

8. Handoff to dependent stories

DGR-003 (runtime recipe/fingerprint): populate Fingerprint (model_artifact_digest, runtime_recipe_digest, recipe_id, recipe_version, catalogue_version). The mismatch outcome is already specified: ERROR_CODE_FINGERPRINT_MISMATCH. Do not invent a second identity struct.
DGR-005/006 (range loading, architecture boundary): the boundary payload is a named bundle, not a bare tensor — a boundary needing more than one tensor is already representable. Execute [effective_start_layer, end_layer), never from start_layer.
DGR-007 (concurrent sessions/KV): isolate on (route_session_id, route_epoch). CacheExpectation/CacheResult and ERROR_CODE_CACHE_MISS are the contract; a decode step whose expected_past_len does not match must miss, never fall back to a silent stateless forward. idempotency_step means a retried step is acknowledged (Ack.duplicate), not re-applied — re-applying advances the KV cache twice and desynchronises the route.
DGR-008 (C++ worker): link shard_runtime_grpc from CMakeLists.txt; you must first install gRPC C++ (see limitations). Honour FlowControl credits and the max_chunk_bytes bound. Use packages/node/meshnet_node/native_protocol/codec.py as the reference for fragment reassembly and checksum validation.
DGR-009 (Meshnet integration): the relay may carry these serialized frames as opaque binary — that is exactly why deadline/cancel/identity are in-band. Do not add a second control plane.
Anyone editing the schema: run both --check scripts; if a vector legitimately changes, regenerate it and say so, because the C++ test asserts those exact bytes.

12 KiB Raw Blame History