243 lines
15 KiB
Markdown
243 lines
15 KiB
Markdown
# DGR-002 — Adopt the versioned gRPC Shard protocol
|
|
|
|
Status: **done**. Every acceptance criterion is met with real command output.
|
|
Evidence class: **synthetic/unit** — this story defines a schema and proves both
|
|
languages agree on it. No model, GPU, network peer or benchmark is involved, and
|
|
none is claimed.
|
|
|
|
## 1. Summary
|
|
|
|
`packages/node/native/proto/shard_runtime.proto` is now the semantic contract for
|
|
the native Shard data plane: Protocol Buffers over gRPC/HTTP2 (ADR-0020). Python
|
|
and C++ both generate from it, and a shared committed conformance vector proves
|
|
they encode it identically — byte for byte.
|
|
|
|
Design decisions worth carrying forward:
|
|
|
|
- **Everything gRPC gives you is *also* in the schema.** Deadline, cancellation,
|
|
identity and flow control are carried as fields, not left to HTTP/2 metadata,
|
|
because the existing relay carries these frames as **opaque binary**. A relayed
|
|
frame has no HTTP/2 context to inherit a deadline or a channel identity from.
|
|
If it is not in the schema, it does not survive the relay.
|
|
- **Cancellation is both in-band and out-of-band.** `CancelSignal` rides the
|
|
stream; `Cancel` is also a unary RPC. A cancel that can only travel down a
|
|
stream that flow control has wedged is not a cancel.
|
|
- **Checksums cover the uncompressed payload.** Compression is a per-hop
|
|
transport decision (reusing the existing `activation_compression` policies), so
|
|
a checksum over the compressed frame would be invalidated by a hop that merely
|
|
chose differently.
|
|
- **Application-level flow-control credits, not just HTTP/2 windows.** HTTP/2
|
|
bounds *bytes in flight*; it does not bound how much *work* a worker has queued,
|
|
and a relayed frame gets no window at all. Credits bound queue occupancy and KV
|
|
pressure, and negotiation takes the strictest bound of either peer so a sender
|
|
cannot talk a worker into unbounded queues.
|
|
|
|
## 2. Files changed
|
|
|
|
New:
|
|
|
|
| Path | What |
|
|
|---|---|
|
|
| `packages/node/native/proto/shard_runtime.proto` | The schema (sha256 `9e211660…`, see `protocol.json`) |
|
|
| `packages/node/native/CMakeLists.txt` | C++ generation + build wiring + ctest |
|
|
| `packages/node/native/tests/test_shard_protocol_conformance.cpp` | C++ conformance test |
|
|
| `packages/node/native/testdata/*.binpb` | Committed cross-language vectors |
|
|
| `packages/node/native/README.md` | How to regenerate and build |
|
|
| `packages/node/meshnet_node/native_protocol/__init__.py` | Public Python surface |
|
|
| `packages/node/meshnet_node/native_protocol/codec.py` | Bundle encode/decode, fragmentation, CRC32C, chunking, FC negotiation |
|
|
| `packages/node/meshnet_node/native_protocol/conformance.py` | Canonical vectors shared by both languages |
|
|
| `packages/node/meshnet_node/native_protocol/generated/` | Generated Python stubs (committed) |
|
|
| `scripts/generate_native_protocol.py` | Python generation, with `--check` |
|
|
| `scripts/generate_protocol_goldens.py` | Vector generation, with `--check` |
|
|
| `scripts/bootstrap_native_toolchain.sh` | Builds protobuf C++ from source |
|
|
| `tests/test_native_shard_protocol.py` | 45 Python tests |
|
|
|
|
Modified:
|
|
|
|
- `packages/node/pyproject.toml` — added runtime floors `grpcio>=1.82.1` and
|
|
`protobuf>=7.35.0`, matching the committed generated-code requirements; new
|
|
`proto` extra pinning `grpcio-tools==1.82.1`.
|
|
- `packages/node/meshnet_node/activation_compression.py` — optional bounded zstd
|
|
output for untrusted protocol frames; existing callers remain compatible.
|
|
- `packages/node/meshnet_node/native_protocol/__init__.py` — exports negotiated
|
|
bound constants and whole-session-message validation.
|
|
|
|
The canonical PRD marks only DGR-002 passed. `git status` before this story was clean.
|
|
|
|
## 3. Commands and real results
|
|
|
|
See `commands.txt` for the exact ordered list. Results:
|
|
|
|
```
|
|
python scripts/generate_native_protocol.py --check -> generated stubs are up to date
|
|
python scripts/generate_protocol_goldens.py --check -> conformance vectors are up to date
|
|
|
|
cmake -S packages/node/native -B build/native -DCMAKE_PREFIX_PATH=/tmp/pbsrc/install
|
|
-- gRPC C++ not found: building message types only (sufficient for the conformance test)
|
|
cmake --build build/native -j -> Built target shard_protocol_conformance
|
|
ctest --test-dir build/native --output-on-failure -> 1/1 Test #1: shard_protocol_conformance ... Passed
|
|
100% tests passed out of 1
|
|
|
|
cmp build/native/cpp_roundtrip.binpb \
|
|
packages/node/native/testdata/session_request_golden.binpb -> identical (exit 0)
|
|
|
|
pytest -q tests/test_native_shard_protocol.py -> 45 passed
|
|
pytest -q tests/test_native_shard_protocol.py \
|
|
tests/test_activation_compression.py -> 51 passed
|
|
pytest -q (final full suite) -> 728 passed, 12 skipped
|
|
pytest -q tests/test_tracker_routing.py::test_tracker_dashboard_can_cancel_inflight_proxy
|
|
(after an earlier flaky full-suite failure) -> 1 passed, 1 passed, 1 passed
|
|
clean minimum-runtime import + codec smoke test -> passed
|
|
grpcio==1.82.1, protobuf==7.35.0
|
|
compileall -q packages tests -> OK (exit 0)
|
|
git diff --check -> clean (exit 0)
|
|
```
|
|
|
|
The C++ lane was rebuilt from scratch by Ralph (`rm -rf build/native`) using only
|
|
the documented commands, and reproduced the same result. During controller
|
|
review the user explicitly chose not to repeat the destructive build-directory
|
|
cleanup, so the independent controller relied on the recorded CMake/CTest run
|
|
while reproducing every Python/generation/full-suite gate.
|
|
|
|
### Controller review corrections
|
|
|
|
Independent controller review found and fixed two classes of issue before
|
|
integration:
|
|
|
|
1. Generated stubs required gRPC 1.82.1 and Protobuf 7.35.0, while the initial
|
|
package metadata allowed much older runtimes that could fail at import time.
|
|
2. Flow-control bounds were described but not enforced by the reference decoder.
|
|
Tensor declarations, shape rank/dimensions, fragment/tensor counts, fragments,
|
|
wire bodies, whole bundles, complete session messages (including envelope
|
|
overhead), and zstd window/output expansion are now fail-closed against the
|
|
negotiated/default bounds. Unspecified bundle versions, compression and
|
|
checksums are rejected rather than interpreted as valid data.
|
|
3. Negotiated initial credits could exceed `max_inflight_chunks`; credits are now
|
|
capped by the settled in-flight limit.
|
|
|
|
Controller results: protocol tests `45 passed`; protocol plus shared compression
|
|
tests `51 passed`; final full suite `728 passed, 12 skipped`. A clean environment
|
|
at the declared minimum gRPC/Protobuf runtime versions imported both generated
|
|
stub modules and round-tripped the codec. Generation checks, `compileall`, static
|
|
secret scan, and `git diff --check` all passed.
|
|
|
|
### Full-suite note — a pre-existing flaky test
|
|
|
|
`tests/test_tracker_routing.py::test_tracker_dashboard_can_cancel_inflight_proxy`
|
|
is **flaky on a clean tree, independent of this story**. Reproduction, run
|
|
*before any DGR-002 file existed* (working tree clean, `git status` empty):
|
|
|
|
```
|
|
pytest -q -> 1 failed, 682 passed, 12 skipped
|
|
FAILED tests/test_tracker_routing.py::test_tracker_dashboard_can_cancel_inflight_proxy
|
|
|
|
# same test, three consecutive isolated runs on the same clean tree:
|
|
pytest -q tests/test_tracker_routing.py::test_tracker_dashboard_can_cancel_inflight_proxy
|
|
-> 1 passed in 1.76s
|
|
-> 1 failed in 4.39s
|
|
-> 1 passed in 1.10s
|
|
```
|
|
|
|
It is a timing race in proxy cancellation (a 3-second in-flight generation raced
|
|
against the cancel assertion), not a deterministic failure, and it touches no code
|
|
this story changes. One controller full-suite run reported exactly that one failure
|
|
(`1 failed, 719 passed, 12 skipped`); three immediate isolated retries all passed
|
|
in 1.11 seconds, and the final exact-code full suite was green (`728 passed,
|
|
12 skipped`). It is flagged for whoever owns the tracker cancel path and is **not**
|
|
fixed here, since silently touching another story's code is out of scope.
|
|
|
|
## 4. Acceptance criteria
|
|
|
|
| Criterion | Where it is proven |
|
|
|---|---|
|
|
| Schema for capability, health, session stream, release, cancellation | `shard_runtime.proto` `service ShardRuntime`; `test_service_exposes_capability_health_session_release_and_cancel` |
|
|
| One long-lived bidi stream per Activation Seam, with deadlines, cancellation, flow control, structured errors | `rpc Session (stream) returns (stream)`; `test_session_is_one_long_lived_bidirectional_stream`; `Envelope.deadline_unix_nanos`, `CancelSignal` + unary `Cancel`, `FlowControl`, `ShardError` |
|
|
| Bounded chunking for prefill; small decode fast path | `ChunkInfo` + `plan_prefill_chunks` (128-token bound, ADR-0008); `DecodeStep`; `test_prefill_is_split_into_bounded_token_aligned_chunks`, `test_decode_fast_path_is_much_smaller_than_a_full_envelope_chunk` |
|
|
| Envelope carries schema version, work id, session id, epoch, fingerprint, range/effective start, phase, position, idempotency step, cache expectation, compression, checksum | `Envelope` + `NamedTensor`; `test_envelope_carries_every_field_the_protocol_promises` asserts against the **descriptor**, so deleting a field from the `.proto` fails the test |
|
|
| Versioned named-tensor bundle: name, shape, dtype, byte order, fragments | `TensorBundle`/`NamedTensor`/`TensorFragment`; `test_named_tensor_bundle_is_versioned_and_fully_described`, `test_bundle_round_trips_multiple_named_tensors` |
|
|
| Round-trip + compatibility tests in Python and C++ | 45 Python tests; C++ `ctest` 1/1; cross-language byte equality |
|
|
| Targeted pytest passes | 45 passed |
|
|
| `compileall packages tests` | exit 0 |
|
|
| `git diff --check` | exit 0 |
|
|
| Default tests deterministic, download-free, credit-free, GPU-free | Pure in-memory protobuf; no model, no network, no GPU |
|
|
| Full deterministic pytest passes, or pre-existing failure recorded | Final exact-code run: 728 passed, 12 skipped; earlier sole flaky failure documented with clean-tree reproduction and 3/3 passing retries |
|
|
|
|
## 5. How the cross-language claim is actually earned
|
|
|
|
Two codecs that each round-trip their own output prove only that each is
|
|
self-consistent. Instead:
|
|
|
|
1. Python builds the canonical `SessionRequest` and commits its bytes.
|
|
2. The C++ test parses **those** bytes, asserts every field, recomputes the CRC32C
|
|
**from the polynomial in independent C++ code**, reassembles the multi-fragment
|
|
tensor, and re-serializes to `cpp_roundtrip.binpb`.
|
|
3. `test_cpp_and_python_agree_byte_for_byte` asserts that file equals the golden.
|
|
|
|
Compatibility is tested in both languages: an unknown field from a newer peer
|
|
survives a parse/serialize hop (a Shard forwards activations — silently stripping
|
|
fields would corrupt a route it is merely a waypoint on), and a sparse message
|
|
from an older peer parses to proto3 defaults.
|
|
|
|
## 6. Limitations and deferred work
|
|
|
|
- **gRPC C++ was not built or linked.** The C++ lane verifies the *schema* (message
|
|
types), not a running gRPC C++ server, because this machine has no gRPC C++ stack
|
|
and building it is a large dependency the conformance test does not need.
|
|
`CMakeLists.txt` already generates and exports `shard_runtime_grpc` when
|
|
`find_package(gRPC)` succeeds. **DGR-008 must install gRPC C++ and extend
|
|
`scripts/bootstrap_native_toolchain.sh`.**
|
|
- **No wire is exercised.** No client, server, or stream lifecycle exists yet — no
|
|
deadline actually fires, no credit is actually consumed. This story defines and
|
|
proves the contract; DGR-008/DGR-009 implement it.
|
|
- The protobuf C++ toolchain used here was installed to `/tmp/pbsrc/install` (ephemeral).
|
|
`scripts/bootstrap_native_toolchain.sh` reproduces it; prefer a durable prefix such
|
|
as `build/native-toolchain`.
|
|
- `crc32c` has a pure-Python fallback (used here) and picks up `google_crc32c` when
|
|
present. The fallback is byte-exact but slow; a worker on the hot path should install
|
|
the native package. Not a correctness limitation.
|
|
- Compression on the wire is zstd-or-none only, matching the existing seam.
|
|
|
|
## 7. Compatibility and migration notes
|
|
|
|
- **This does not change the existing HTTP activation wire.** `X-Meshnet-Wire` stays
|
|
at `2` and the legacy `/forward` path is untouched. The native protocol is a
|
|
*separate* contract with its own `SchemaVersion`, starting at 1. Nothing in this
|
|
story is on any live request path — it is additive.
|
|
- Semantics are deliberately preserved from the existing ADRs so the two transports
|
|
mean the same thing: `effective_start_layer` (ADR-0012), `CacheMode`/`expected_past_len`
|
|
and `ERROR_CODE_CACHE_MISS` mapping to today's HTTP 409 `cache_miss` (ADR-0022),
|
|
bfloat16 boundary dtype and 128-token prefill chunks (ADR-0008), fingerprint/recipe
|
|
identity mirroring the capability report (ADR-0023).
|
|
- `TensorFragment` field 5 (`uncompressed_size`) is **reserved**: it was removed
|
|
because `NamedTensor.total_bytes` is the single source of truth. Never recycle it —
|
|
a recycled field number is the one schema change peers cannot detect, because the
|
|
bytes still parse.
|
|
- Committed Python stubs are guarded by `--check` in the test suite, so they cannot
|
|
drift from the schema unnoticed.
|
|
|
|
## 8. Handoff to dependent stories
|
|
|
|
- **DGR-003 (runtime recipe/fingerprint):** populate `Fingerprint`
|
|
(`model_artifact_digest`, `runtime_recipe_digest`, `recipe_id`, `recipe_version`,
|
|
`catalogue_version`). The mismatch outcome is already specified:
|
|
`ERROR_CODE_FINGERPRINT_MISMATCH`. Do not invent a second identity struct.
|
|
- **DGR-005/006 (range loading, architecture boundary):** the boundary payload is a
|
|
**named bundle**, not a bare tensor — a boundary needing more than one tensor is
|
|
already representable. Execute `[effective_start_layer, end_layer)`, never from
|
|
`start_layer`.
|
|
- **DGR-007 (concurrent sessions/KV):** isolate on `(route_session_id, route_epoch)`.
|
|
`CacheExpectation`/`CacheResult` and `ERROR_CODE_CACHE_MISS` are the contract; a
|
|
decode step whose `expected_past_len` does not match **must** miss, never fall back
|
|
to a silent stateless forward. `idempotency_step` means a retried step is
|
|
acknowledged (`Ack.duplicate`), not re-applied — re-applying advances the KV cache
|
|
twice and desynchronises the route.
|
|
- **DGR-008 (C++ worker):** link `shard_runtime_grpc` from `CMakeLists.txt`; you must
|
|
first install gRPC C++ (see limitations). Honour `FlowControl` credits and the
|
|
`max_chunk_bytes` bound. Use `packages/node/meshnet_node/native_protocol/codec.py`
|
|
as the reference for fragment reassembly and checksum validation.
|
|
- **DGR-009 (Meshnet integration):** the relay may carry these serialized frames as
|
|
opaque binary — that is exactly why deadline/cancel/identity are in-band. Do not add
|
|
a second control plane.
|
|
- **Anyone editing the schema:** run both `--check` scripts; if a vector legitimately
|
|
changes, regenerate it and say so, because the C++ test asserts those exact bytes.
|