neuron-tai/.scratch/distributed-gguf-runtime/evidence/DGR-033/README.md

# DGR-033 evidence — standalone fake C++ gRPC Shard worker

**Completed:** 2026-07-25 (initial); **repaired:** 2026-07-26 after Codex
GPT-5.5 cross-review BLOCK (see "Cross-review repair" below).
**Branch:** `ralph/distributed-gguf-opus`
**Authority:** `.scratch/distributed-gguf-runtime/prd.json`
**Dependencies:** DGR-022 (lifecycle/status contract), DGR-024 (real generated
gRPC harness + `shard_runtime_server.py` reference semantics), DGR-032
(deterministic fake `ShardEngine` semantics).

## Objective

Prove the standalone worker process, stream, lifecycle, and supervision shape
before any llama.cpp integration: a real C++ executable that serves the whole
ShardRuntime lifecycle/stream contract over gRPC using a model-free fake engine,
driven end-to-end by Python integration tests over a real socket.

## What was found live before changing code

- `packages/node/native/proto/shard_runtime.proto` (DGR-021..023): the single
  semantic contract. Its `ShardRuntime` service has exactly five RPCs —
  `GetCapability`, `Health`, `Session` (bidi stream), `Release`, `Cancel`.
- `packages/node/meshnet_node/shard_runtime_server.py` (DGR-024): the reference
  Python servicer. It performs a *bounded real forward* (a CRC over the received
  bundle bytes) then echoes the chunk, and fails closed on stale epoch, expired
  deadline, corrupt/mis-tiled fragments, exhausted flow-control credit, duplicate
  idempotency step, and in-band/out-of-band cancellation, with per-`route_session_id`
  state kept on the servicer so an out-of-band `Cancel` can reach a live session.
  **Key finding:** despite the schema labelling the checksum `CRC32C`, this
  runtime computes it with `zlib.crc32` (standard CRC-32, *not* Castagnoli). The
  C++ worker mirrors `zlib.crc32` exactly so its checksum acceptance is
  byte-identical to the existing Python surface (the committed C++ *conformance*
  test, by contrast, uses true Castagnoli against separately-generated goldens —
  the two are unrelated code paths).
- `packages/node/native/CMakeLists.txt` (DGR-029/030): configures against the
  ignored `build/native-toolchain` prefix (pinned Protobuf 33.1 + gRPC 1.82.1),
  always generates both message and service stubs, and registers a C++
  conformance CTest. There was **no** worker executable and **no** Python
  worker integration test before this story (confirmed by
  `ls packages/node/native/worker` → absent, and grep for `shard_worker`).
- `packages/node/meshnet_node/fake_shard_engine.py` (DGR-032): the Python fake
  engine, deliberately *not* wired into the gRPC surface. DGR-033's worker is
  its native analogue — a separate executable, not a consumer of that module —
  so both fakes present identical behaviour to a client (deterministic,
  model-free bounded forward; per-session isolation; fail-closed lifecycle).

## What was added (this story's change)

### `packages/node/native/worker/fake_engine.h` (new)

`meshnet::worker::FakeShardEngine` — a header-only, model-free fixture engine.
Its only capability is to validate a `TensorBundle` (fragments tile exactly, the
uncompressed CRC-32 matches the declared checksum, the declared payload stays
within the negotiated `max_chunk_bytes`) and fold the fragment bytes through a
bounded forward. It links, loads, and dispatches to **nothing** — no llama.cpp,
no graph execution. Carries `kEvidenceClass = "fixture"` mirroring the Python
`FakeShardEngine.EVIDENCE_CLASS` for the later DGR-036 parity check.

### `packages/node/native/worker/shard_service.{h,cpp}` (new)

`ShardRuntimeServiceImpl : meshnet::shard::v1::ShardRuntime::Service` — a faithful
C++ port of the DGR-024 Python servicer: the same per-`route_session_id`
identity/credit/dedup state guarded by a mutex, the same fail-closed negative
paths, and the same lifecycle (open → prefill/decode → flow-control top-up →
release/cancel). Each per-request response is computed under the lock and written
*after* releasing it, so a blocking `Write` can never deadlock the out-of-band
`Cancel` RPC that needs the same lock. Bounded messages are enforced two ways: a
per-tensor `RESOURCE_EXHAUSTED` app check against `max_chunk_bytes`, plus a hard
transport receive ceiling.

### `packages/node/native/worker/shard_worker_main.cpp` (new)

The standalone `shard_worker` executable. Binds `MESHNET_SHARD_LISTEN_ADDR`
(or an `argv` address), prints one readiness line (`ShardRuntime worker listening
on <addr>`), and serves until `SIGTERM`/`SIGINT`. **Graceful shutdown** uses a
self-pipe: the async-signal-safe handler writes one byte, a drain thread reads it
and calls `server->Shutdown()`, so in-flight sessions finish and the process
exits `0` printing `ShardRuntime worker shut down cleanly`. A `--selftest` mode
binds an ephemeral port and self-drives capability/health/fragmented-prefill/
decode/release over a real loopback gRPC channel, giving a pure-C++ CTest that
needs no Python.

### `packages/node/native/CMakeLists.txt` (modified)

Adds the `shard_worker` executable (linking only `shard_runtime_grpc` +
`gRPC::grpc++` — no llama.cpp) and registers `shard_worker_selftest` as a CTest.

### `tests/test_native_shard_worker.py` (new)

18 integration tests that spawn the **real compiled binary** as a subprocess and
drive it with the committed generated stubs over a real localhost socket. When
the binary is not built they skip (the DGR-029/030 `requires_cmake` gating
pattern), locating it via `MESHNET_SHARD_WORKER_BIN` or `build/native/shard_worker`.

## Acceptance criteria → evidence

1. **Standalone C++ executable serves the complete lifecycle/stream contract
   using the fake engine** — `shard_worker` builds and serves all five RPCs; the
   `shard_worker_selftest` CTest drives open → fragmented prefill → decode →
   release over real gRPC; the 18 Python tests cover the same against the
   subprocess.
2. **Python integration tests cover startup, health, capability, fragmented
   prefill, decode, release, cancellation, graceful shutdown** —
   `test_worker_startup_and_health`, `test_worker_capability`,
   `test_fragmented_prefill_echoes_reassembled_payload` (3-fragment tiling),
   `test_decode_step_is_served`, `test_release_is_terminal`,
   `test_in_band_cancel_of_single_work_item_does_not_end_stream`,
   `test_in_band_cancel_of_whole_session_is_terminal`,
   `test_out_of_band_cancel_rpc_races_ahead_of_open`,
   `test_graceful_shutdown_on_sigterm` (SIGTERM → exit 0 + clean-shutdown line).
3. **Bounded messages, deadlines, flow control, independent session
   cancellation enforced** — `test_bounded_message_is_rejected`
   (`RESOURCE_EXHAUSTED` on an over-ceiling tensor),
   `test_expired_deadline_is_rejected`, `test_flow_control_violation_and_topup`,
   `test_independent_session_cancellation` (cancelling session A leaves session B
   fully serviceable), plus `test_stale_route_epoch_is_rejected`,
   `test_duplicate_idempotency_step_is_acked`,
   `test_malformed_fragment_tiling_is_rejected`.
4. **Exposes neither llama.cpp RPC nor arbitrary graph execution** —
   `ldd build/native/shard_worker` shows no llama/ggml shared libs;
   `nm -C build/native/shard_worker | grep -icE 'llama_|ggml_'` → `0`; the proto
   exposes exactly one service with five lifecycle RPCs and no graph-exec entry.
5. **Gates + this handoff** — below.

## Commands and results

Toolchain (ignored `build/native-toolchain`, pinned Protobuf 33.1 + gRPC 1.82.1):

```bash
bash scripts/bootstrap_native_toolchain.sh "$PWD/build/native-toolchain"
# ... gRPC 1.82.1 commit acccf84c0df20487d64101f528e5d426541ca4e5
# grpc_cpp_plugin sha256 43705cf26ae9ce98bbcee76b3408f5e171eec746b50bf0dd42dd68d132c6a533
```

Focused out-of-tree CMake build + CTest:

```bash
cmake -S packages/node/native -B build/native -DCMAKE_PREFIX_PATH="$PWD/build/native-toolchain"
cmake --build build/native -j"$(nproc)"
ctest --test-dir build/native --output-on-failure
```
```text
1/2 Test #1: shard_worker_selftest ............   Passed    0.01 sec
2/2 Test #2: shard_protocol_conformance .......   Passed    0.00 sec
100% tests passed out of 2
```

Python integration tests against the real binary:

```bash
PYTHONPATH=packages/node:packages/tracker python -m pytest -q tests/test_native_shard_worker.py
```
```text
18 passed in 3.96s
```

AC4 (no llama.cpp / no graph exec):

```bash
ldd build/native/shard_worker | grep -iE 'llama|ggml'   # -> (no matches)
nm build/native/shard_worker | grep -icE 'llama_|ggml_' # -> 0
```

Shared gates + regression:

```bash
python -m compileall -q packages tests                                  # exit 0
git diff --check -- packages/node/native tests/test_native_shard_worker.py  # exit 0
PYTHONPATH=packages/node:packages/tracker python -m pytest -q \
  tests/test_shard_runtime_harness.py tests/test_native_shard_protocol.py
# -> 61 passed, 2 skipped   (DGR-024 harness + native protocol untouched)
```

Toolchain used: `cmake`/`ctest` from the `distributed-gguf-runtime` worktree's
`.venv` (PyPI `cmake==4.4.0` wheel — no system cmake exists here, same as
DGR-029/030); the Python client uses that venv's `grpcio==1.82.1`,
`grpcio-tools==1.82.1`, `protobuf`, `pytest`. `g++ (GCC) 15.2.1`.

## Limitations

- This is FIXTURE evidence only. The worker's "forward" is a CRC-over-wire-bytes
  echo, not real tensor compute; it proves process/stream/lifecycle/supervision
  shape, nothing about numerical correctness. Real engine binding is DGR-037 and
  numeric parity is DGR-036/052.
- The worker checksum path mirrors the DGR-024 runtime's `zlib.crc32` (standard
  CRC-32 under a `CRC32C` label). Compressed-tensor tiling/checksum is not
  independently verified (no zstd decompressor in the fixture) — identical to the
  DGR-024 limitation.
- Default `pytest` runs skip `tests/test_native_shard_worker.py` unless the
  worker binary is built (or `MESHNET_SHARD_WORKER_BIN` is set); this session
  built it and ran all 18 for real (results above). Building requires the pinned
  gRPC C++ toolchain, which is not present by default and must be bootstrapped.
- No CUDA/ROCm/GPU, no model download, no network at test time — all default
  tests are fixture-only and offline.

## Dependency handoff

- **DGR-036** (fixture vs real-model parity): the worker's `FakeShardEngine`
  carries `kEvidenceClass = "fixture"`; diff it against DGR-037's real engine's
  equivalent marker, and reuse the same lifecycle/stream contract this worker
  serves to prove behavioural parity before numeric parity.
- **DGR-037** (bind llama.cpp): replace `FakeShardEngine`'s bounded forward with
  the real engine behind the *same* `ShardRuntimeServiceImpl` surface; the
  service's session/epoch/credit/dedup/cancel machinery and the graceful-shutdown
  supervision shape are reusable as-is.
- **DGR-040** (worker supervision): `shard_worker` already provides the
  supervision primitives — a readiness line for start detection, `SIGTERM`
  graceful drain with a clean-exit line, and a `--selftest` liveness probe.
  A supervisor can start/monitor/restart the process around these.

## Cross-review repair (2026-07-26)

An independent Codex GPT-5.5 review BLOCKED the initial implementation. Four
root protocol defects in the native worker were fixed in this worktree
(`.claude/worktrees/distributed-gguf-opus`); the fake-engine echo semantics and
supervision shape are unchanged.

### Defects fixed

1. **Activation before SessionOpen bypassed all state.** A chunk/decode whose
   `route_session_id` had no opened session fell through every `if (state && ...)`
   guard and was echoed — bypassing lifecycle, cancellation, epoch and
   flow-control. `SessionState` now carries an `opened` flag set only by a valid
   `SessionOpen`; chunk and decode fail closed with a terminal
   `ERROR_CODE_INTERNAL` and end the stream when it is false. A placeholder state
   created by an out-of-band `Cancel` that races `Open` has `opened == false`, so
   it can never admit work either.
2. **Flow control blindly trusted the peer proposal.** `SessionOpen` copied the
   proposed `credits/max_inflight/max_chunk_bytes` verbatim into session state and
   the accepted reply. New `ShardRuntimeServiceImpl::NegotiateFlow` takes the
   strictest bound of peer-vs-worker for every field (mirroring
   `negotiate_flow_control` in `native_protocol/codec.py`), stores the negotiated
   ceilings on the session, and enforces the negotiated per-session
   `max_chunk_bytes` on every bundle (`FakeShardEngine::Validate` now takes the
   ceiling as an argument instead of a fixed construction-time value).
3. **In-stream `ReleaseSignal` leaked session state.** The stream `release` arm
   wrote a terminal status but never dropped the session. It now erases the
   session under the lock before responding, so KV/credits/dedup are freed
   immediately (the out-of-band `Release` RPC already erased).
4. **`SessionOpen` echoed caller identity instead of validating it.** The handshake
   now rejects an incompatible `schema_version` (`SCHEMA_UNSUPPORTED`), a
   mismatched model/recipe `Fingerprint` (`FINGERPRINT_MISMATCH`), and a
   `ShardRange` outside the worker's served range (`SHARD_RANGE_MISMATCH`), each
   terminal; `SessionAccepted` now reports the worker's own served fingerprint
   rather than a copy of the caller's.

### Changed files (repair)

- `packages/node/native/worker/shard_service.h` — `opened` +
  `max_prefill_chunk_tokens` on `SessionState`; `NegotiateFlow` decl; engine now
  default-constructed.
- `packages/node/native/worker/shard_service.cpp` — worker-identity constants +
  fill helpers; `NegotiateFlow`; `SessionOpen` validation/negotiation; fail-closed
  chunk/decode; per-session `max_chunk_bytes`; in-stream release erase.
- `packages/node/native/worker/fake_engine.h` — `Validate(bundle, max_chunk_bytes)`.
- `tests/test_native_shard_worker.py` — extended `_open` (schema/fingerprint/range/
  flow overrides); fixed `test_release_rpc_is_idempotent` for the new erase
  semantics; added 9 regression tests (chunk/decode before open, flow-control
  clamp, negotiated-ceiling cap, in-stream release erase, schema/fingerprint/range
  rejection, worker-fingerprint-not-caller).

### Re-run gates (real, rebuilt binary)

Build driven through the pinned `cmake` (Unix Makefiles + `gmake`, gRPC 1.82.1):

```text
cmake --build build/native --parallel 8            -> BUILD_EXIT 0
ctest --test-dir build/native --output-on-failure  -> 100% (2/2) passed
  shard_worker_selftest ....... Passed
  shard_protocol_conformance .. Passed
python -m pytest -q tests/test_native_shard_worker.py         -> 27 passed
python -m pytest -q tests/test_shard_runtime_harness.py \
                    tests/test_native_shard_protocol.py       -> 63 passed
python -m compileall -q packages tests                       -> exit 0
git diff --check                                             -> clean
ldd build/native/shard_worker | grep -iE 'llama|ggml'        -> NONE
nm -C build/native/shard_worker | grep -cE 'llama_|ggml_'    -> 0
```

The worker integration suite grew from 18 to 27 tests; all pass against the
freshly compiled binary. No `.ralph-lane` runtime artifacts were touched.