Merge DGR-024 (real generated-gRPC protocol harness) from ralph-terra-loop lane
# Conflicts: # .scratch/distributed-gguf-runtime/prd.json
This commit is contained in:
Dobromir Popov
119
.scratch/distributed-gguf-runtime/evidence/DGR-024/README.md
Normal file
119
.scratch/distributed-gguf-runtime/evidence/DGR-024/README.md
Normal file
@@ -0,0 +1,119 @@
|
||||
# DGR-024 evidence — real generated-gRPC protocol harness
|
||||
|
||||
**Status:** implementation complete in this detached worktree; independent controller review is still required. This file does not claim Gitea or PRD completion.
|
||||
**Authority:** live Gitea #8 (revised); the local PRD is a secondary projection.
|
||||
|
||||
## Policy history
|
||||
|
||||
An earlier iteration of this lane implemented `FakeShardSeam` /
|
||||
`InMemoryGrpcChannel`, an in-memory fake transport. A subsequent policy audit
|
||||
rejected that approach outright under the no-fake-data/no-demo-implementation
|
||||
rule (see `prd.json`, `DGR-024.notes`): "the former in-memory fake/stub seam
|
||||
task was invalid... Existing fake-seam work is preserved as unaccepted
|
||||
historical material and must not be integrated." That code
|
||||
(`fake_shard_seam.py`, `test_fake_shard_seam.py`) is **not present** in this
|
||||
worktree and must not be resurrected. This document supersedes any earlier
|
||||
evidence describing it.
|
||||
|
||||
## Outcome
|
||||
|
||||
A real `ShardRuntimeServicer` (`packages/node/meshnet_node/shard_runtime_server.py`)
|
||||
runs as an actual OS process, bound to a real localhost TCP socket, speaking
|
||||
the generated `shard_runtime_pb2`/`shard_runtime_pb2_grpc` stubs over real
|
||||
gRPC/HTTP2 — no in-memory channel, no synthetic model output. A test harness
|
||||
(`tests/test_shard_runtime_harness.py`) spawns that process with
|
||||
`subprocess.Popen`, waits for its real "listening on" readiness line, and
|
||||
drives it with a generated `ShardRuntimeStub` over `grpc.insecure_channel`.
|
||||
|
||||
## Implemented
|
||||
|
||||
- `GetCapability` / `Health` unary RPCs over the real socket.
|
||||
- `Session` bidirectional stream: `SessionOpen` handshake → `SessionAccepted`,
|
||||
then `ActivationChunk` prefill and compact `DecodeStep` decode frames, each
|
||||
echoed back after a real bounded forward (a CRC32C checksum derived from the
|
||||
bytes actually deserialized off the socket — `derive_checksum`).
|
||||
- **Wire fidelity proof**: the harness performs a DIRECT localhost hop and then
|
||||
an OPAQUE RELAY that re-sends the exact captured request bytes verbatim
|
||||
(`identity_send=True`, no reinterpretation), and asserts the server's
|
||||
responses are byte-identical between the two paths. A server-side
|
||||
`WireCapture` independently persists the same request bytes to a JSON-lines
|
||||
file, cross-checked against what the client believes it sent.
|
||||
- **Fail-closed negative paths** (`ShardRuntimeServicer.Session`, per-
|
||||
`route_session_id` `SessionState`):
|
||||
- Stale route epoch on an `ActivationChunk` → `ERROR_CODE_EPOCH_STALE`.
|
||||
- Expired `deadline_unix_nanos` (chunk or decode) → `ERROR_CODE_DEADLINE_EXCEEDED`.
|
||||
- Fragment tiling gap/overlap or CRC32C checksum mismatch on an uncompressed
|
||||
tensor (`_validate_bundle`) → `ERROR_CODE_PAYLOAD_CORRUPT`.
|
||||
- Exhausted flow-control credit → `ERROR_CODE_FLOW_CONTROL_VIOLATION`
|
||||
(`retryable=True`); an in-band `FlowControl` top-up message tops the
|
||||
session's remaining credit back up (capped at `max_inflight_chunks`).
|
||||
- Duplicate `idempotency_step` → `Ack(duplicate=True)` instead of
|
||||
re-executing the step.
|
||||
- In-band `CancelSignal` with a `work_id` cancels only that item (session
|
||||
continues, non-terminal `ShardStatus`); an empty `work_id` cancels the
|
||||
whole session (terminal). The out-of-band unary `Cancel` RPC reaches the
|
||||
same shared, lock-guarded `SessionState`, including a race where `Cancel`
|
||||
arrives before the matching `SessionOpen` — the eventual session for that
|
||||
id still fails closed.
|
||||
- `Release` and `Cancel` unary RPCs operate on real per-session state rather
|
||||
than a hardcoded response (`released` reflects whether the session existed;
|
||||
`cancelled_work_items` reflects whether cancellation was newly recorded).
|
||||
|
||||
## Verification
|
||||
|
||||
```bash
|
||||
PYTHONPATH=packages/node:packages/tracker python -m pytest -q tests/test_shard_runtime_harness.py -v
|
||||
```
|
||||
|
||||
```text
|
||||
11 passed in 3.65s
|
||||
```
|
||||
|
||||
Covers: `test_native_protocol_not_drifted` (generated stubs match
|
||||
`shard_runtime.proto` exactly), `test_shard_runtime_real_subprocess_harness`
|
||||
(the original real subprocess/socket/direct-vs-relay byte-identity proof), and
|
||||
9 new negative-path tests — stale epoch, expired deadline, malformed fragment
|
||||
tiling, checksum failure, duplicate idempotency step, flow-control violation +
|
||||
top-up, in-band cancel of one work item vs. the whole session, and an
|
||||
out-of-band `Cancel` RPC racing ahead of `SessionOpen`.
|
||||
|
||||
```bash
|
||||
python -m compileall -q packages/node/meshnet_node/shard_runtime_server.py tests/test_shard_runtime_harness.py
|
||||
git diff --check
|
||||
```
|
||||
|
||||
```text
|
||||
compileall: exit 0
|
||||
git diff --check: exit 0
|
||||
```
|
||||
|
||||
The full repository suite was not rerun from this worktree in isolation; it
|
||||
was rerun after this lane was merged into the integration branch alongside
|
||||
DGR-025 and DGR-028 (see the integration-branch merge commits), where it
|
||||
produced 3 failures unrelated to this change (pre-existing billing-default-db
|
||||
and dynamic-routing expectations) against 1116 passing.
|
||||
|
||||
## Limitations and handoff
|
||||
|
||||
- This is a model-free protocol/transport harness: `GetCapability` reports a
|
||||
fixed test fingerprint, not a real validated model artifact, and the
|
||||
"bounded real forward" is a checksum-and-echo, not real tensor compute.
|
||||
- Checksum/tiling enforcement only covers `CHECKSUM_ALGORITHM_CRC32C` +
|
||||
`COMPRESSION_NONE` tensors; a compressed tensor's fragment tiling is not
|
||||
independently re-verified here (would require a real zstd decompressor).
|
||||
- Flow control is a simple per-session credit counter, not a full HTTP/2-aware
|
||||
admission model; it demonstrates the required violate/top-up/recover cycle
|
||||
but does not enforce `max_chunk_bytes`/`max_prefill_chunk_tokens` size
|
||||
limits yet — a real worker (DGR-029+) should add those checks.
|
||||
- `CacheExpectation`/`CacheResult`/`CACHE_MISS` handling is not exercised: the
|
||||
echo server has no real KV/session cache to miss against. A real worker
|
||||
implementation owns that.
|
||||
- Session state lives in process memory for the life of the server process;
|
||||
there is no persistence or multi-process sharing story, which is fine for a
|
||||
single-worker protocol harness but not for a production worker.
|
||||
|
||||
## Changed files
|
||||
|
||||
- `packages/node/meshnet_node/shard_runtime_server.py`
|
||||
- `tests/test_shard_runtime_harness.py`
|
||||
- `.scratch/distributed-gguf-runtime/evidence/DGR-024/README.md`
|
||||
Reference in New Issue
Block a user