neuron-tai/.scratch/distributed-gguf-runtime/evidence/DGR-002/README.md

DGR-002 — Adopt the versioned gRPC Shard protocol

Status: done. Every acceptance criterion is met with real command output. Evidence class: synthetic/unit — this story defines a schema and proves both languages agree on it. No model, GPU, network peer or benchmark is involved, and none is claimed.

1. Summary

packages/node/native/proto/shard_runtime.proto is now the semantic contract for the native Shard data plane: Protocol Buffers over gRPC/HTTP2 (ADR-0020). Python and C++ both generate from it, and a shared committed conformance vector proves they encode it identically — byte for byte.

Design decisions worth carrying forward:

• Everything gRPC gives you is also in the schema. Deadline, cancellation, identity and flow control are carried as fields, not left to HTTP/2 metadata, because the existing relay carries these frames as opaque binary. A relayed frame has no HTTP/2 context to inherit a deadline or a channel identity from. If it is not in the schema, it does not survive the relay.
• Cancellation is both in-band and out-of-band. CancelSignal rides the stream; Cancel is also a unary RPC. A cancel that can only travel down a stream that flow control has wedged is not a cancel.
• Checksums cover the uncompressed payload. Compression is a per-hop transport decision (reusing the existing activation_compression policies), so a checksum over the compressed frame would be invalidated by a hop that merely chose differently.
• Application-level flow-control credits, not just HTTP/2 windows. HTTP/2 bounds bytes in flight; it does not bound how much work a worker has queued, and a relayed frame gets no window at all. Credits bound queue occupancy and KV pressure, and negotiation takes the strictest bound of either peer so a sender cannot talk a worker into unbounded queues.

2. Files changed

New:

Path	What
packages/node/native/proto/shard_runtime.proto	The schema (sha256 9e211660…, see protocol.json)
packages/node/native/CMakeLists.txt	C++ generation + build wiring + ctest
packages/node/native/tests/test_shard_protocol_conformance.cpp	C++ conformance test
packages/node/native/testdata/*.binpb	Committed cross-language vectors
packages/node/native/README.md	How to regenerate and build
packages/node/meshnet_node/native_protocol/__init__.py	Public Python surface
packages/node/meshnet_node/native_protocol/codec.py	Bundle encode/decode, fragmentation, CRC32C, chunking, FC negotiation
packages/node/meshnet_node/native_protocol/conformance.py	Canonical vectors shared by both languages
packages/node/meshnet_node/native_protocol/generated/	Generated Python stubs (committed)
scripts/generate_native_protocol.py	Python generation, with --check
scripts/generate_protocol_goldens.py	Vector generation, with --check
scripts/bootstrap_native_toolchain.sh	Builds protobuf C++ from source
tests/test_native_shard_protocol.py	45 Python tests

Modified:

• packages/node/pyproject.toml — added runtime floors grpcio>=1.82.1 and protobuf>=7.35.0, matching the committed generated-code requirements; new proto extra pinning grpcio-tools==1.82.1.
• packages/node/meshnet_node/activation_compression.py — optional bounded zstd output for untrusted protocol frames; existing callers remain compatible.
• packages/node/meshnet_node/native_protocol/__init__.py — exports negotiated bound constants and whole-session-message validation.

The canonical PRD marks only DGR-002 passed. git status before this story was clean.

3. Commands and real results

See commands.txt for the exact ordered list. Results:

python scripts/generate_native_protocol.py --check    -> generated stubs are up to date
python scripts/generate_protocol_goldens.py --check   -> conformance vectors are up to date

cmake -S packages/node/native -B build/native -DCMAKE_PREFIX_PATH=/tmp/pbsrc/install
  -- gRPC C++ not found: building message types only (sufficient for the conformance test)
cmake --build build/native -j                         -> Built target shard_protocol_conformance
ctest --test-dir build/native --output-on-failure     -> 1/1 Test #1: shard_protocol_conformance ... Passed
                                                          100% tests passed out of 1

cmp build/native/cpp_roundtrip.binpb \
    packages/node/native/testdata/session_request_golden.binpb   -> identical (exit 0)

pytest -q tests/test_native_shard_protocol.py         -> 45 passed
pytest -q tests/test_native_shard_protocol.py \
  tests/test_activation_compression.py                -> 51 passed
pytest -q  (final full suite)                         -> 728 passed, 12 skipped
pytest -q tests/test_tracker_routing.py::test_tracker_dashboard_can_cancel_inflight_proxy
  (after an earlier flaky full-suite failure)          -> 1 passed, 1 passed, 1 passed
clean minimum-runtime import + codec smoke test        -> passed
  grpcio==1.82.1, protobuf==7.35.0
compileall -q packages tests                          -> OK (exit 0)
git diff --check                                      -> clean (exit 0)

The C++ lane was rebuilt from scratch by Ralph (rm -rf build/native) using only the documented commands, and reproduced the same result. During controller review the user explicitly chose not to repeat the destructive build-directory cleanup, so the independent controller relied on the recorded CMake/CTest run while reproducing every Python/generation/full-suite gate.

Controller review corrections

Independent controller review found and fixed two classes of issue before integration:

1. Generated stubs required gRPC 1.82.1 and Protobuf 7.35.0, while the initial package metadata allowed much older runtimes that could fail at import time.
2. Flow-control bounds were described but not enforced by the reference decoder. Tensor declarations, shape rank/dimensions, fragment/tensor counts, fragments, wire bodies, whole bundles, complete session messages (including envelope overhead), and zstd window/output expansion are now fail-closed against the negotiated/default bounds. Unspecified bundle versions, compression and checksums are rejected rather than interpreted as valid data.
3. Negotiated initial credits could exceed max_inflight_chunks; credits are now capped by the settled in-flight limit.

Controller results: protocol tests 45 passed; protocol plus shared compression tests 51 passed; final full suite 728 passed, 12 skipped. A clean environment at the declared minimum gRPC/Protobuf runtime versions imported both generated stub modules and round-tripped the codec. Generation checks, compileall, static secret scan, and git diff --check all passed.

Full-suite note — a pre-existing flaky test

tests/test_tracker_routing.py::test_tracker_dashboard_can_cancel_inflight_proxy is flaky on a clean tree, independent of this story. Reproduction, run before any DGR-002 file existed (working tree clean, git status empty):

pytest -q                                       -> 1 failed, 682 passed, 12 skipped
  FAILED tests/test_tracker_routing.py::test_tracker_dashboard_can_cancel_inflight_proxy

# same test, three consecutive isolated runs on the same clean tree:
pytest -q tests/test_tracker_routing.py::test_tracker_dashboard_can_cancel_inflight_proxy
  -> 1 passed in 1.76s
  -> 1 failed in 4.39s
  -> 1 passed in 1.10s

It is a timing race in proxy cancellation (a 3-second in-flight generation raced against the cancel assertion), not a deterministic failure, and it touches no code this story changes. One controller full-suite run reported exactly that one failure (1 failed, 719 passed, 12 skipped); three immediate isolated retries all passed in 1.11 seconds, and the final exact-code full suite was green (728 passed, 12 skipped). It is flagged for whoever owns the tracker cancel path and is not fixed here, since silently touching another story's code is out of scope.

4. Acceptance criteria

Criterion	Where it is proven
Schema for capability, health, session stream, release, cancellation	shard_runtime.proto service ShardRuntime; test_service_exposes_capability_health_session_release_and_cancel
One long-lived bidi stream per Activation Seam, with deadlines, cancellation, flow control, structured errors	rpc Session (stream) returns (stream); test_session_is_one_long_lived_bidirectional_stream; Envelope.deadline_unix_nanos, CancelSignal + unary Cancel, FlowControl, ShardError
Bounded chunking for prefill; small decode fast path	ChunkInfo + plan_prefill_chunks (128-token bound, ADR-0008); DecodeStep; test_prefill_is_split_into_bounded_token_aligned_chunks, test_decode_fast_path_is_much_smaller_than_a_full_envelope_chunk
Envelope carries schema version, work id, session id, epoch, fingerprint, range/effective start, phase, position, idempotency step, cache expectation, compression, checksum	Envelope + NamedTensor; test_envelope_carries_every_field_the_protocol_promises asserts against the descriptor, so deleting a field from the .proto fails the test
Versioned named-tensor bundle: name, shape, dtype, byte order, fragments	TensorBundle/NamedTensor/TensorFragment; test_named_tensor_bundle_is_versioned_and_fully_described, test_bundle_round_trips_multiple_named_tensors
Round-trip + compatibility tests in Python and C++	45 Python tests; C++ ctest 1/1; cross-language byte equality
Targeted pytest passes	45 passed
compileall packages tests	exit 0
git diff --check	exit 0
Default tests deterministic, download-free, credit-free, GPU-free	Pure in-memory protobuf; no model, no network, no GPU
Full deterministic pytest passes, or pre-existing failure recorded	Final exact-code run: 728 passed, 12 skipped; earlier sole flaky failure documented with clean-tree reproduction and 3/3 passing retries

5. How the cross-language claim is actually earned

Two codecs that each round-trip their own output prove only that each is self-consistent. Instead:

1. Python builds the canonical SessionRequest and commits its bytes.
2. The C++ test parses those bytes, asserts every field, recomputes the CRC32C from the polynomial in independent C++ code, reassembles the multi-fragment tensor, and re-serializes to cpp_roundtrip.binpb.
3. test_cpp_and_python_agree_byte_for_byte asserts that file equals the golden.

Compatibility is tested in both languages: an unknown field from a newer peer survives a parse/serialize hop (a Shard forwards activations — silently stripping fields would corrupt a route it is merely a waypoint on), and a sparse message from an older peer parses to proto3 defaults.

6. Limitations and deferred work

• gRPC C++ was not built or linked. The C++ lane verifies the schema (message types), not a running gRPC C++ server, because this machine has no gRPC C++ stack and building it is a large dependency the conformance test does not need. CMakeLists.txt already generates and exports shard_runtime_grpc when find_package(gRPC) succeeds. DGR-008 must install gRPC C++ and extend scripts/bootstrap_native_toolchain.sh.
• No wire is exercised. No client, server, or stream lifecycle exists yet — no deadline actually fires, no credit is actually consumed. This story defines and proves the contract; DGR-008/DGR-009 implement it.
• The protobuf C++ toolchain used here was installed to /tmp/pbsrc/install (ephemeral). scripts/bootstrap_native_toolchain.sh reproduces it; prefer a durable prefix such as build/native-toolchain.
• crc32c has a pure-Python fallback (used here) and picks up google_crc32c when present. The fallback is byte-exact but slow; a worker on the hot path should install the native package. Not a correctness limitation.
• Compression on the wire is zstd-or-none only, matching the existing seam.

7. Compatibility and migration notes

• This does not change the existing HTTP activation wire. X-Meshnet-Wire stays at 2 and the legacy /forward path is untouched. The native protocol is a separate contract with its own SchemaVersion, starting at 1. Nothing in this story is on any live request path — it is additive.
• Semantics are deliberately preserved from the existing ADRs so the two transports mean the same thing: effective_start_layer (ADR-0012), CacheMode/expected_past_len and ERROR_CODE_CACHE_MISS mapping to today's HTTP 409 cache_miss (ADR-0022), bfloat16 boundary dtype and 128-token prefill chunks (ADR-0008), fingerprint/recipe identity mirroring the capability report (ADR-0023).
• TensorFragment field 5 (uncompressed_size) is reserved: it was removed because NamedTensor.total_bytes is the single source of truth. Never recycle it — a recycled field number is the one schema change peers cannot detect, because the bytes still parse.
• Committed Python stubs are guarded by --check in the test suite, so they cannot drift from the schema unnoticed.

8. Handoff to dependent stories

• DGR-003 (runtime recipe/fingerprint): populate Fingerprint (model_artifact_digest, runtime_recipe_digest, recipe_id, recipe_version, catalogue_version). The mismatch outcome is already specified: ERROR_CODE_FINGERPRINT_MISMATCH. Do not invent a second identity struct.
• DGR-005/006 (range loading, architecture boundary): the boundary payload is a named bundle, not a bare tensor — a boundary needing more than one tensor is already representable. Execute [effective_start_layer, end_layer), never from start_layer.
• DGR-007 (concurrent sessions/KV): isolate on (route_session_id, route_epoch). CacheExpectation/CacheResult and ERROR_CODE_CACHE_MISS are the contract; a decode step whose expected_past_len does not match must miss, never fall back to a silent stateless forward. idempotency_step means a retried step is acknowledged (Ack.duplicate), not re-applied — re-applying advances the KV cache twice and desynchronises the route.
• DGR-008 (C++ worker): link shard_runtime_grpc from CMakeLists.txt; you must first install gRPC C++ (see limitations). Honour FlowControl credits and the max_chunk_bytes bound. Use packages/node/meshnet_node/native_protocol/codec.py as the reference for fragment reassembly and checksum validation.
• DGR-009 (Meshnet integration): the relay may carry these serialized frames as opaque binary — that is exactly why deadline/cancel/identity are in-band. Do not add a second control plane.
• Anyone editing the schema: run both --check scripts; if a vector legitimately changes, regenerate it and say so, because the C++ test asserts those exact bytes.