neuron-tai/.scratch/distributed-gguf-runtime/evidence/DGR-024/README.md

# DGR-024 evidence — real generated-gRPC protocol harness

**Status:** implementation complete in this detached worktree; independent controller review is still required. This file does not claim Gitea or PRD completion.
**Authority:** live Gitea #8 (revised); the local PRD is a secondary projection.

## Policy history

An earlier iteration of this lane implemented `FakeShardSeam` /
`InMemoryGrpcChannel`, an in-memory fake transport. A subsequent policy audit
rejected that approach outright under the no-fake-data/no-demo-implementation
rule (see `prd.json`, `DGR-024.notes`): "the former in-memory fake/stub seam
task was invalid... Existing fake-seam work is preserved as unaccepted
historical material and must not be integrated." That code
(`fake_shard_seam.py`, `test_fake_shard_seam.py`) is **not present** in this
worktree and must not be resurrected. This document supersedes any earlier
evidence describing it.

## Outcome

A real `ShardRuntimeServicer` (`packages/node/meshnet_node/shard_runtime_server.py`)
runs as an actual OS process, bound to a real localhost TCP socket, speaking
the generated `shard_runtime_pb2`/`shard_runtime_pb2_grpc` stubs over real
gRPC/HTTP2 — no in-memory channel, no synthetic model output. A test harness
(`tests/test_shard_runtime_harness.py`) spawns that process with
`subprocess.Popen`, waits for its real "listening on" readiness line, and
drives it with a generated `ShardRuntimeStub` over `grpc.insecure_channel`.

## Implemented

- `GetCapability` / `Health` unary RPCs over the real socket.
- `Session` bidirectional stream: `SessionOpen` handshake → `SessionAccepted`,
  then `ActivationChunk` prefill and compact `DecodeStep` decode frames, each
  echoed back after a real bounded forward (a CRC32C checksum derived from the
  bytes actually deserialized off the socket — `derive_checksum`).
- **Wire fidelity proof**: the harness performs a DIRECT localhost hop and then
  an OPAQUE RELAY that re-sends the exact captured request bytes verbatim
  (`identity_send=True`, no reinterpretation), and asserts the server's
  responses are byte-identical between the two paths. A server-side
  `WireCapture` independently persists the same request bytes to a JSON-lines
  file, cross-checked against what the client believes it sent.
- **Fail-closed negative paths** (`ShardRuntimeServicer.Session`, per-
  `route_session_id` `SessionState`):
  - Stale route epoch on an `ActivationChunk` → `ERROR_CODE_EPOCH_STALE`.
  - Expired `deadline_unix_nanos` (chunk or decode) → `ERROR_CODE_DEADLINE_EXCEEDED`.
  - Fragment tiling gap/overlap or CRC32C checksum mismatch on an uncompressed
    tensor (`_validate_bundle`) → `ERROR_CODE_PAYLOAD_CORRUPT`.
  - Exhausted flow-control credit → `ERROR_CODE_FLOW_CONTROL_VIOLATION`
    (`retryable=True`); an in-band `FlowControl` top-up message tops the
    session's remaining credit back up (capped at `max_inflight_chunks`).
  - Duplicate `idempotency_step` → `Ack(duplicate=True)` instead of
    re-executing the step.
  - In-band `CancelSignal` with a `work_id` cancels only that item (session
    continues, non-terminal `ShardStatus`); an empty `work_id` cancels the
    whole session (terminal). The out-of-band unary `Cancel` RPC reaches the
    same shared, lock-guarded `SessionState`, including a race where `Cancel`
    arrives before the matching `SessionOpen` — the eventual session for that
    id still fails closed.
- `Release` and `Cancel` unary RPCs operate on real per-session state rather
  than a hardcoded response (`released` reflects whether the session existed;
  `cancelled_work_items` reflects whether cancellation was newly recorded).

## Verification

```bash
PYTHONPATH=packages/node:packages/tracker python -m pytest -q tests/test_shard_runtime_harness.py -v
```

```text
11 passed in 3.65s
```

Covers: `test_native_protocol_not_drifted` (generated stubs match
`shard_runtime.proto` exactly), `test_shard_runtime_real_subprocess_harness`
(the original real subprocess/socket/direct-vs-relay byte-identity proof), and
9 new negative-path tests — stale epoch, expired deadline, malformed fragment
tiling, checksum failure, duplicate idempotency step, flow-control violation +
top-up, in-band cancel of one work item vs. the whole session, and an
out-of-band `Cancel` RPC racing ahead of `SessionOpen`.

```bash
python -m compileall -q packages/node/meshnet_node/shard_runtime_server.py tests/test_shard_runtime_harness.py
git diff --check
```

```text
compileall: exit 0
git diff --check: exit 0
```

The full repository suite was not rerun from this worktree in isolation; it
was rerun after this lane was merged into the integration branch alongside
DGR-025 and DGR-028 (see the integration-branch merge commits), where it
produced 3 failures unrelated to this change (pre-existing billing-default-db
and dynamic-routing expectations) against 1116 passing.

## Limitations and handoff

- This is a model-free protocol/transport harness: `GetCapability` reports a
  fixed test fingerprint, not a real validated model artifact, and the
  "bounded real forward" is a checksum-and-echo, not real tensor compute.
- Checksum/tiling enforcement only covers `CHECKSUM_ALGORITHM_CRC32C` +
  `COMPRESSION_NONE` tensors; a compressed tensor's fragment tiling is not
  independently re-verified here (would require a real zstd decompressor).
- Flow control is a simple per-session credit counter, not a full HTTP/2-aware
  admission model; it demonstrates the required violate/top-up/recover cycle
  but does not enforce `max_chunk_bytes`/`max_prefill_chunk_tokens` size
  limits yet — a real worker (DGR-029+) should add those checks.
- `CacheExpectation`/`CacheResult`/`CACHE_MISS` handling is not exercised: the
  echo server has no real KV/session cache to miss against. A real worker
  implementation owns that.
- Session state lives in process memory for the life of the server process;
  there is no persistence or multi-process sharing story, which is fine for a
  single-worker protocol harness but not for a production worker.

## Changed files

- `packages/node/meshnet_node/shard_runtime_server.py`
- `tests/test_shard_runtime_harness.py`
- `.scratch/distributed-gguf-runtime/evidence/DGR-024/README.md`