neuron-tai/.scratch/distributed-gguf-runtime/evidence/DGR-024/README.md

DGR-024 evidence — real generated-gRPC protocol harness

Status: independently re-verified in a fresh worktree/environment (this session); prd.json DGR-024.passes is now true. Authority: live Gitea #8 (revised); the local PRD is a secondary projection.

Policy history

An earlier iteration of this lane implemented FakeShardSeam / InMemoryGrpcChannel, an in-memory fake transport. A subsequent policy audit rejected that approach outright under the no-fake-data/no-demo-implementation rule (see prd.json, DGR-024.notes): "the former in-memory fake/stub seam task was invalid... Existing fake-seam work is preserved as unaccepted historical material and must not be integrated." That code (fake_shard_seam.py, test_fake_shard_seam.py) is not present in this worktree and must not be resurrected. This document supersedes any earlier evidence describing it.

Outcome

A real ShardRuntimeServicer (packages/node/meshnet_node/shard_runtime_server.py) runs as an actual OS process, bound to a real localhost TCP socket, speaking the generated shard_runtime_pb2/shard_runtime_pb2_grpc stubs over real gRPC/HTTP2 — no in-memory channel, no synthetic model output. A test harness (tests/test_shard_runtime_harness.py) spawns that process with subprocess.Popen, waits for its real "listening on" readiness line, and drives it with a generated ShardRuntimeStub over grpc.insecure_channel.

Implemented

• GetCapability / Health unary RPCs over the real socket.
• Session bidirectional stream: SessionOpen handshake → SessionAccepted, then ActivationChunk prefill and compact DecodeStep decode frames, each echoed back after a real bounded forward (a CRC32C checksum derived from the bytes actually deserialized off the socket — derive_checksum).
• Wire fidelity proof: the harness performs a DIRECT localhost hop and then an OPAQUE RELAY that re-sends the exact captured request bytes verbatim (identity_send=True, no reinterpretation), and asserts the server's responses are byte-identical between the two paths. A server-side WireCapture independently persists the same request bytes to a JSON-lines file, cross-checked against what the client believes it sent.
• Fail-closed negative paths (ShardRuntimeServicer.Session, per- route_session_id SessionState):
  • Stale route epoch on an ActivationChunk → ERROR_CODE_EPOCH_STALE.
  • Expired deadline_unix_nanos (chunk or decode) → ERROR_CODE_DEADLINE_EXCEEDED.
  • Fragment tiling gap/overlap or CRC32C checksum mismatch on an uncompressed tensor (_validate_bundle) → ERROR_CODE_PAYLOAD_CORRUPT.
  • Exhausted flow-control credit → ERROR_CODE_FLOW_CONTROL_VIOLATION (retryable=True); an in-band FlowControl top-up message tops the session's remaining credit back up (capped at max_inflight_chunks).
  • Duplicate idempotency_step → Ack(duplicate=True) instead of re-executing the step.
  • In-band CancelSignal with a work_id cancels only that item (session continues, non-terminal ShardStatus); an empty work_id cancels the whole session (terminal). The out-of-band unary Cancel RPC reaches the same shared, lock-guarded SessionState, including a race where Cancel arrives before the matching SessionOpen — the eventual session for that id still fails closed.
• Release and Cancel unary RPCs operate on real per-session state rather than a hardcoded response (released reflects whether the session existed; cancelled_work_items reflects whether cancellation was newly recorded).

Verification

The previous evidence for this story predated an environment with grpc importable (tests/test_shard_runtime_harness.py could not even collect on the ambient interpreter — see .ralph-tui/progress.md's DGR-019 entry). This session built a real, disposable uv-managed .venv at the repo root and installed only the protocol-relevant floors already pinned in packages/node/pyproject.toml (grpcio==1.82.1, grpcio-tools==1.82.1, protobuf==7.35.1) plus pytest==9.1.1, then reran the full harness for real — this is not a re-statement of the earlier claim, it is an independent execution:

uv pip install grpcio grpcio-tools==1.82.1 protobuf pytest
PYTHONPATH=packages/node:packages/tracker .venv/bin/python -m pytest -q tests/test_shard_runtime_harness.py -v -s

collected 11 items
tests/test_shard_runtime_harness.py .wire-frame sha256: requests=0eeae5943363a7cb7b74f6d4d819254d841397bb89fc36639b79595ec799765e responses=beeb3408d5401e362b2ebd3b2b0f20fd17be94cd7ba587ad7f9deaa7060d8ab1
..........
11 passed in 3.56s

Covers: test_native_protocol_not_drifted (generated stubs match shard_runtime.proto exactly — reran scripts/generate_native_protocol.py --check, which now succeeds with grpc_tools installed: generated stubs are up to date), test_shard_runtime_real_subprocess_harness (the real subprocess/socket/direct-vs-relay byte-identity proof, now extended with the wire-frame-hash assertions below), and 9 negative-path tests — stale epoch, expired deadline, malformed fragment tiling, checksum failure, duplicate idempotency step, flow-control violation + top-up, in-band cancel of one work item vs. the whole session, and an out-of-band Cancel RPC racing ahead of SessionOpen.

Wire-frame hashes (new this session)

The prior evidence proved wire fidelity only by raw byte-equality assertions; it recorded no hash. WireCapture.to_dict() (packages/node/meshnet_node/shard_runtime_server.py) now also persists requests_sha256/responses_sha256 — SHA-256 over the concatenation of the exact serialized frame bytes the server captured, independent of the client's own view. tests/test_shard_runtime_harness.py::test_shard_runtime_real_subprocess_harness asserts these server-persisted hashes equal independently-computed SHA-256 hashes over the client-side captured bytes, and that the DIRECT and OPAQUE RELAY hashes are identical:

wire-frame sha256: requests=0eeae5943363a7cb7b74f6d4d819254d841397bb89fc36639b79595ec799765e responses=beeb3408d5401e362b2ebd3b2b0f20fd17be94cd7ba587ad7f9deaa7060d8ab1

Generated artifact identities

SHA-256 of the committed generated stubs this harness runs against (produced by grpcio-tools==1.82.1 from packages/node/native/proto/shard_runtime.proto; confirmed not-drifted by test_native_protocol_not_drifted above):

759026b11bbd659f2caed713044a0584809c44bee733359e80a197635cd0c362  packages/node/meshnet_node/native_protocol/generated/shard_runtime_pb2.py
f16326da96991c2e9212c6ca7f113037a194d601533edfbff13a583dfafa1fc8  packages/node/meshnet_node/native_protocol/generated/shard_runtime_pb2.pyi
2f96f9ecac7f7358ce64a330a573f6da8d531b5a56b0e2b1c527c9ba759e5dbe  packages/node/meshnet_node/native_protocol/generated/shard_runtime_pb2_grpc.py

.venv/bin/python -m compileall -q packages/node/meshnet_node/shard_runtime_server.py tests/test_shard_runtime_harness.py
.venv/bin/python -m compileall -q packages tests
git diff --check

compileall (targeted): exit 0
compileall (packages tests, universal gate wording): exit 0
git diff --check: exit 0

Also re-ran tests/test_ralph_prd_schema.py (108 passed) after restoring prd.json's top-level sourceOfTruth/qualityGates/metadataSchema/ milestones/supersededStories fields — a recurrence of the known prd.json-field-drop bug (see .ralph-tui/progress.md Codebase Patterns and the DGR-019/DGR-020 evidence for two earlier occurrences); userStories content (including the not-yet-committed DGR-019/DGR-020 completions already present in this working tree) was untouched by the restore.

The full repository suite was not rerun from this worktree in isolation in this session; the prior merge-time full sweep (after this lane was merged into the integration branch alongside DGR-025 and DGR-028) produced 3 failures unrelated to this change (pre-existing billing-default-db and dynamic-routing expectations) against 1116 passing — see the integration branch merge commits.

Limitations and handoff

• This is a model-free protocol/transport harness: GetCapability reports a fixed test fingerprint, not a real validated model artifact, and the "bounded real forward" is a checksum-and-echo, not real tensor compute.
• Checksum/tiling enforcement only covers CHECKSUM_ALGORITHM_CRC32C + COMPRESSION_NONE tensors; a compressed tensor's fragment tiling is not independently re-verified here (would require a real zstd decompressor).
• Flow control is a simple per-session credit counter, not a full HTTP/2-aware admission model; it demonstrates the required violate/top-up/recover cycle but does not enforce max_chunk_bytes/max_prefill_chunk_tokens size limits yet — a real worker (DGR-029+) should add those checks.
• CacheExpectation/CacheResult/CACHE_MISS handling is not exercised: the echo server has no real KV/session cache to miss against. A real worker implementation owns that.
• Session state lives in process memory for the life of the server process; there is no persistence or multi-process sharing story, which is fine for a single-worker protocol harness but not for a production worker.

Changed files

• packages/node/meshnet_node/shard_runtime_server.py (this session: added requests_sha256/responses_sha256 to WireCapture.to_dict())
• tests/test_shard_runtime_harness.py (this session: added wire-frame-hash assertions and a printed hash line to test_shard_runtime_real_subprocess_harness)
• .scratch/distributed-gguf-runtime/evidence/DGR-024/README.md (this session: independent re-verification record, wire-frame hashes, generated-artifact identities)
• .scratch/distributed-gguf-runtime/prd.json (this session: restored dropped top-level fields; DGR-024.passes flipped to true)