neuron-tai/.scratch/distributed-gguf-runtime/evidence/DGR-024/README.md

# DGR-024 evidence — real generated-gRPC protocol harness

**Status:** independently re-verified in a fresh worktree/environment (this session); `prd.json` `DGR-024.passes` is now `true`.
**Authority:** live Gitea #8 (revised); the local PRD is a secondary projection.

## Policy history

An earlier iteration of this lane implemented `FakeShardSeam` /
`InMemoryGrpcChannel`, an in-memory fake transport. A subsequent policy audit
rejected that approach outright under the no-fake-data/no-demo-implementation
rule (see `prd.json`, `DGR-024.notes`): "the former in-memory fake/stub seam
task was invalid... Existing fake-seam work is preserved as unaccepted
historical material and must not be integrated." That code
(`fake_shard_seam.py`, `test_fake_shard_seam.py`) is **not present** in this
worktree and must not be resurrected. This document supersedes any earlier
evidence describing it.

## Outcome

A real `ShardRuntimeServicer` (`packages/node/meshnet_node/shard_runtime_server.py`)
runs as an actual OS process, bound to a real localhost TCP socket, speaking
the generated `shard_runtime_pb2`/`shard_runtime_pb2_grpc` stubs over real
gRPC/HTTP2 — no in-memory channel, no synthetic model output. A test harness
(`tests/test_shard_runtime_harness.py`) spawns that process with
`subprocess.Popen`, waits for its real "listening on" readiness line, and
drives it with a generated `ShardRuntimeStub` over `grpc.insecure_channel`.

## Implemented

- `GetCapability` / `Health` unary RPCs over the real socket.
- `Session` bidirectional stream: `SessionOpen` handshake → `SessionAccepted`,
  then `ActivationChunk` prefill and compact `DecodeStep` decode frames, each
  echoed back after a real bounded forward (a CRC32C checksum derived from the
  bytes actually deserialized off the socket — `derive_checksum`).
- **Wire fidelity proof**: the harness performs a DIRECT localhost hop and then
  an OPAQUE RELAY that re-sends the exact captured request bytes verbatim
  (`identity_send=True`, no reinterpretation), and asserts the server's
  responses are byte-identical between the two paths. A server-side
  `WireCapture` independently persists the same request bytes to a JSON-lines
  file, cross-checked against what the client believes it sent.
- **Fail-closed negative paths** (`ShardRuntimeServicer.Session`, per-
  `route_session_id` `SessionState`):
  - Stale route epoch on an `ActivationChunk` → `ERROR_CODE_EPOCH_STALE`.
  - Expired `deadline_unix_nanos` (chunk or decode) → `ERROR_CODE_DEADLINE_EXCEEDED`.
  - Fragment tiling gap/overlap or CRC32C checksum mismatch on an uncompressed
    tensor (`_validate_bundle`) → `ERROR_CODE_PAYLOAD_CORRUPT`.
  - Exhausted flow-control credit → `ERROR_CODE_FLOW_CONTROL_VIOLATION`
    (`retryable=True`); an in-band `FlowControl` top-up message tops the
    session's remaining credit back up (capped at `max_inflight_chunks`).
  - Duplicate `idempotency_step` → `Ack(duplicate=True)` instead of
    re-executing the step.
  - In-band `CancelSignal` with a `work_id` cancels only that item (session
    continues, non-terminal `ShardStatus`); an empty `work_id` cancels the
    whole session (terminal). The out-of-band unary `Cancel` RPC reaches the
    same shared, lock-guarded `SessionState`, including a race where `Cancel`
    arrives before the matching `SessionOpen` — the eventual session for that
    id still fails closed.
- `Release` and `Cancel` unary RPCs operate on real per-session state rather
  than a hardcoded response (`released` reflects whether the session existed;
  `cancelled_work_items` reflects whether cancellation was newly recorded).

## Verification

The previous evidence for this story predated an environment with `grpc`
importable (`tests/test_shard_runtime_harness.py` could not even *collect* on
the ambient interpreter — see `.ralph-tui/progress.md`'s DGR-019 entry). This
session built a real, disposable `uv`-managed `.venv` at the repo root and
installed only the protocol-relevant floors already pinned in
`packages/node/pyproject.toml` (`grpcio==1.82.1`, `grpcio-tools==1.82.1`,
`protobuf==7.35.1`) plus `pytest==9.1.1`, then reran the full harness for
real — this is not a re-statement of the earlier claim, it is an independent
execution:

```bash
uv pip install grpcio grpcio-tools==1.82.1 protobuf pytest
PYTHONPATH=packages/node:packages/tracker .venv/bin/python -m pytest -q tests/test_shard_runtime_harness.py -v -s
```

```text
collected 11 items
tests/test_shard_runtime_harness.py .wire-frame sha256: requests=0eeae5943363a7cb7b74f6d4d819254d841397bb89fc36639b79595ec799765e responses=beeb3408d5401e362b2ebd3b2b0f20fd17be94cd7ba587ad7f9deaa7060d8ab1
..........
11 passed in 3.56s
```

Covers: `test_native_protocol_not_drifted` (generated stubs match
`shard_runtime.proto` exactly — reran `scripts/generate_native_protocol.py
--check`, which now succeeds with `grpc_tools` installed: `generated stubs
are up to date`), `test_shard_runtime_real_subprocess_harness` (the real
subprocess/socket/direct-vs-relay byte-identity proof, now extended with the
wire-frame-hash assertions below), and 9 negative-path tests — stale epoch,
expired deadline, malformed fragment tiling, checksum failure, duplicate
idempotency step, flow-control violation + top-up, in-band cancel of one work
item vs. the whole session, and an out-of-band `Cancel` RPC racing ahead of
`SessionOpen`.

### Wire-frame hashes (new this session)

The prior evidence proved wire fidelity only by raw byte-equality assertions;
it recorded no hash. `WireCapture.to_dict()`
(`packages/node/meshnet_node/shard_runtime_server.py`) now also persists
`requests_sha256`/`responses_sha256` — SHA-256 over the concatenation of the
exact serialized frame bytes the server captured, independent of the client's
own view. `tests/test_shard_runtime_harness.py::test_shard_runtime_real_subprocess_harness`
asserts these server-persisted hashes equal independently-computed SHA-256
hashes over the client-side captured bytes, and that the DIRECT and OPAQUE
RELAY hashes are identical:

```text
wire-frame sha256: requests=0eeae5943363a7cb7b74f6d4d819254d841397bb89fc36639b79595ec799765e responses=beeb3408d5401e362b2ebd3b2b0f20fd17be94cd7ba587ad7f9deaa7060d8ab1
```

### Generated artifact identities

SHA-256 of the committed generated stubs this harness runs against (produced
by `grpcio-tools==1.82.1` from `packages/node/native/proto/shard_runtime.proto`;
confirmed not-drifted by `test_native_protocol_not_drifted` above):

```text
759026b11bbd659f2caed713044a0584809c44bee733359e80a197635cd0c362  packages/node/meshnet_node/native_protocol/generated/shard_runtime_pb2.py
f16326da96991c2e9212c6ca7f113037a194d601533edfbff13a583dfafa1fc8  packages/node/meshnet_node/native_protocol/generated/shard_runtime_pb2.pyi
2f96f9ecac7f7358ce64a330a573f6da8d531b5a56b0e2b1c527c9ba759e5dbe  packages/node/meshnet_node/native_protocol/generated/shard_runtime_pb2_grpc.py
```

```bash
.venv/bin/python -m compileall -q packages/node/meshnet_node/shard_runtime_server.py tests/test_shard_runtime_harness.py
.venv/bin/python -m compileall -q packages tests
git diff --check
```

```text
compileall (targeted): exit 0
compileall (packages tests, universal gate wording): exit 0
git diff --check: exit 0
```

Also re-ran `tests/test_ralph_prd_schema.py` (108 passed) after restoring
`prd.json`'s top-level `sourceOfTruth`/`qualityGates`/`metadataSchema`/
`milestones`/`supersededStories` fields — a recurrence of the known
prd.json-field-drop bug (see `.ralph-tui/progress.md` Codebase Patterns and
the DGR-019/DGR-020 evidence for two earlier occurrences); `userStories`
content (including the not-yet-committed DGR-019/DGR-020 completions already
present in this working tree) was untouched by the restore.

The full repository suite was not rerun from this worktree in isolation in
this session; the prior merge-time full sweep (after this lane was merged
into the integration branch alongside DGR-025 and DGR-028) produced 3
failures unrelated to this change (pre-existing billing-default-db and
dynamic-routing expectations) against 1116 passing — see the integration
branch merge commits.

## Limitations and handoff

- This is a model-free protocol/transport harness: `GetCapability` reports a
  fixed test fingerprint, not a real validated model artifact, and the
  "bounded real forward" is a checksum-and-echo, not real tensor compute.
- Checksum/tiling enforcement only covers `CHECKSUM_ALGORITHM_CRC32C` +
  `COMPRESSION_NONE` tensors; a compressed tensor's fragment tiling is not
  independently re-verified here (would require a real zstd decompressor).
- Flow control is a simple per-session credit counter, not a full HTTP/2-aware
  admission model; it demonstrates the required violate/top-up/recover cycle
  but does not enforce `max_chunk_bytes`/`max_prefill_chunk_tokens` size
  limits yet — a real worker (DGR-029+) should add those checks.
- `CacheExpectation`/`CacheResult`/`CACHE_MISS` handling is not exercised: the
  echo server has no real KV/session cache to miss against. A real worker
  implementation owns that.
- Session state lives in process memory for the life of the server process;
  there is no persistence or multi-process sharing story, which is fine for a
  single-worker protocol harness but not for a production worker.

## Changed files

- `packages/node/meshnet_node/shard_runtime_server.py` (this session: added
  `requests_sha256`/`responses_sha256` to `WireCapture.to_dict()`)
- `tests/test_shard_runtime_harness.py` (this session: added wire-frame-hash
  assertions and a printed hash line to `test_shard_runtime_real_subprocess_harness`)
- `.scratch/distributed-gguf-runtime/evidence/DGR-024/README.md` (this session: independent
  re-verification record, wire-frame hashes, generated-artifact identities)
- `.scratch/distributed-gguf-runtime/prd.json` (this session: restored
  dropped top-level fields; `DGR-024.passes` flipped to `true`)