neuron-tai/.scratch/distributed-gguf-runtime/evidence/DGR-024/README.md

DGR-024 evidence — real generated-gRPC protocol harness

Status: implementation complete in this detached worktree; independent controller review is still required. This file does not claim Gitea or PRD completion. Authority: live Gitea #8 (revised); the local PRD is a secondary projection.

Policy history

An earlier iteration of this lane implemented FakeShardSeam / InMemoryGrpcChannel, an in-memory fake transport. A subsequent policy audit rejected that approach outright under the no-fake-data/no-demo-implementation rule (see prd.json, DGR-024.notes): "the former in-memory fake/stub seam task was invalid... Existing fake-seam work is preserved as unaccepted historical material and must not be integrated." That code (fake_shard_seam.py, test_fake_shard_seam.py) is not present in this worktree and must not be resurrected. This document supersedes any earlier evidence describing it.

Outcome

A real ShardRuntimeServicer (packages/node/meshnet_node/shard_runtime_server.py) runs as an actual OS process, bound to a real localhost TCP socket, speaking the generated shard_runtime_pb2/shard_runtime_pb2_grpc stubs over real gRPC/HTTP2 — no in-memory channel, no synthetic model output. A test harness (tests/test_shard_runtime_harness.py) spawns that process with subprocess.Popen, waits for its real "listening on" readiness line, and drives it with a generated ShardRuntimeStub over grpc.insecure_channel.

Implemented

• GetCapability / Health unary RPCs over the real socket.
• Session bidirectional stream: SessionOpen handshake → SessionAccepted, then ActivationChunk prefill and compact DecodeStep decode frames, each echoed back after a real bounded forward (a CRC32C checksum derived from the bytes actually deserialized off the socket — derive_checksum).
• Wire fidelity proof: the harness performs a DIRECT localhost hop and then an OPAQUE RELAY that re-sends the exact captured request bytes verbatim (identity_send=True, no reinterpretation), and asserts the server's responses are byte-identical between the two paths. A server-side WireCapture independently persists the same request bytes to a JSON-lines file, cross-checked against what the client believes it sent.
• Fail-closed negative paths (ShardRuntimeServicer.Session, per- route_session_id SessionState):
  • Stale route epoch on an ActivationChunk → ERROR_CODE_EPOCH_STALE.
  • Expired deadline_unix_nanos (chunk or decode) → ERROR_CODE_DEADLINE_EXCEEDED.
  • Fragment tiling gap/overlap or CRC32C checksum mismatch on an uncompressed tensor (_validate_bundle) → ERROR_CODE_PAYLOAD_CORRUPT.
  • Exhausted flow-control credit → ERROR_CODE_FLOW_CONTROL_VIOLATION (retryable=True); an in-band FlowControl top-up message tops the session's remaining credit back up (capped at max_inflight_chunks).
  • Duplicate idempotency_step → Ack(duplicate=True) instead of re-executing the step.
  • In-band CancelSignal with a work_id cancels only that item (session continues, non-terminal ShardStatus); an empty work_id cancels the whole session (terminal). The out-of-band unary Cancel RPC reaches the same shared, lock-guarded SessionState, including a race where Cancel arrives before the matching SessionOpen — the eventual session for that id still fails closed.
• Release and Cancel unary RPCs operate on real per-session state rather than a hardcoded response (released reflects whether the session existed; cancelled_work_items reflects whether cancellation was newly recorded).

Verification

PYTHONPATH=packages/node:packages/tracker python -m pytest -q tests/test_shard_runtime_harness.py -v

11 passed in 3.65s

Covers: test_native_protocol_not_drifted (generated stubs match shard_runtime.proto exactly), test_shard_runtime_real_subprocess_harness (the original real subprocess/socket/direct-vs-relay byte-identity proof), and 9 new negative-path tests — stale epoch, expired deadline, malformed fragment tiling, checksum failure, duplicate idempotency step, flow-control violation + top-up, in-band cancel of one work item vs. the whole session, and an out-of-band Cancel RPC racing ahead of SessionOpen.

python -m compileall -q packages/node/meshnet_node/shard_runtime_server.py tests/test_shard_runtime_harness.py
git diff --check

compileall: exit 0
git diff --check: exit 0

The full repository suite was not rerun from this worktree in isolation; it was rerun after this lane was merged into the integration branch alongside DGR-025 and DGR-028 (see the integration-branch merge commits), where it produced 3 failures unrelated to this change (pre-existing billing-default-db and dynamic-routing expectations) against 1116 passing.

Limitations and handoff

• This is a model-free protocol/transport harness: GetCapability reports a fixed test fingerprint, not a real validated model artifact, and the "bounded real forward" is a checksum-and-echo, not real tensor compute.
• Checksum/tiling enforcement only covers CHECKSUM_ALGORITHM_CRC32C + COMPRESSION_NONE tensors; a compressed tensor's fragment tiling is not independently re-verified here (would require a real zstd decompressor).
• Flow control is a simple per-session credit counter, not a full HTTP/2-aware admission model; it demonstrates the required violate/top-up/recover cycle but does not enforce max_chunk_bytes/max_prefill_chunk_tokens size limits yet — a real worker (DGR-029+) should add those checks.
• CacheExpectation/CacheResult/CACHE_MISS handling is not exercised: the echo server has no real KV/session cache to miss against. A real worker implementation owns that.
• Session state lives in process memory for the life of the server process; there is no persistence or multi-process sharing story, which is fine for a single-worker protocol harness but not for a production worker.

Changed files

• packages/node/meshnet_node/shard_runtime_server.py
• tests/test_shard_runtime_harness.py
• .scratch/distributed-gguf-runtime/evidence/DGR-024/README.md