neuron-tai/.scratch/distributed-gguf-runtime/evidence/DGR-033/README.md

DGR-033 evidence — standalone fake C++ gRPC Shard worker

Completed: 2026-07-25 (initial); repaired: 2026-07-26 after Codex GPT-5.5 cross-review BLOCK (see "Cross-review repair" below). Branch: ralph/distributed-gguf-opus Authority: .scratch/distributed-gguf-runtime/prd.json Dependencies: DGR-022 (lifecycle/status contract), DGR-024 (real generated gRPC harness + shard_runtime_server.py reference semantics), DGR-032 (deterministic fake ShardEngine semantics).

Objective

Prove the standalone worker process, stream, lifecycle, and supervision shape before any llama.cpp integration: a real C++ executable that serves the whole ShardRuntime lifecycle/stream contract over gRPC using a model-free fake engine, driven end-to-end by Python integration tests over a real socket.

What was found live before changing code

• packages/node/native/proto/shard_runtime.proto (DGR-021..023): the single semantic contract. Its ShardRuntime service has exactly five RPCs — GetCapability, Health, Session (bidi stream), Release, Cancel.
• packages/node/meshnet_node/shard_runtime_server.py (DGR-024): the reference Python servicer. It performs a bounded real forward (a CRC over the received bundle bytes) then echoes the chunk, and fails closed on stale epoch, expired deadline, corrupt/mis-tiled fragments, exhausted flow-control credit, duplicate idempotency step, and in-band/out-of-band cancellation, with per-route_session_id state kept on the servicer so an out-of-band Cancel can reach a live session. Key finding: despite the schema labelling the checksum CRC32C, this runtime computes it with zlib.crc32 (standard CRC-32, not Castagnoli). The C++ worker mirrors zlib.crc32 exactly so its checksum acceptance is byte-identical to the existing Python surface (the committed C++ conformance test, by contrast, uses true Castagnoli against separately-generated goldens — the two are unrelated code paths).
• packages/node/native/CMakeLists.txt (DGR-029/030): configures against the ignored build/native-toolchain prefix (pinned Protobuf 33.1 + gRPC 1.82.1), always generates both message and service stubs, and registers a C++ conformance CTest. There was no worker executable and no Python worker integration test before this story (confirmed by ls packages/node/native/worker → absent, and grep for shard_worker).
• packages/node/meshnet_node/fake_shard_engine.py (DGR-032): the Python fake engine, deliberately not wired into the gRPC surface. DGR-033's worker is its native analogue — a separate executable, not a consumer of that module — so both fakes present identical behaviour to a client (deterministic, model-free bounded forward; per-session isolation; fail-closed lifecycle).

What was added (this story's change)

packages/node/native/worker/fake_engine.h (new)

meshnet::worker::FakeShardEngine — a header-only, model-free fixture engine. Its only capability is to validate a TensorBundle (fragments tile exactly, the uncompressed CRC-32 matches the declared checksum, the declared payload stays within the negotiated max_chunk_bytes) and fold the fragment bytes through a bounded forward. It links, loads, and dispatches to nothing — no llama.cpp, no graph execution. Carries kEvidenceClass = "fixture" mirroring the Python FakeShardEngine.EVIDENCE_CLASS for the later DGR-036 parity check.

packages/node/native/worker/shard_service.{h,cpp} (new)

ShardRuntimeServiceImpl : meshnet::shard::v1::ShardRuntime::Service — a faithful C++ port of the DGR-024 Python servicer: the same per-route_session_id identity/credit/dedup state guarded by a mutex, the same fail-closed negative paths, and the same lifecycle (open → prefill/decode → flow-control top-up → release/cancel). Each per-request response is computed under the lock and written after releasing it, so a blocking Write can never deadlock the out-of-band Cancel RPC that needs the same lock. Bounded messages are enforced two ways: a per-tensor RESOURCE_EXHAUSTED app check against max_chunk_bytes, plus a hard transport receive ceiling.

packages/node/native/worker/shard_worker_main.cpp (new)

The standalone shard_worker executable. Binds MESHNET_SHARD_LISTEN_ADDR (or an argv address), prints one readiness line (ShardRuntime worker listening on <addr>), and serves until SIGTERM/SIGINT. Graceful shutdown uses a self-pipe: the async-signal-safe handler writes one byte, a drain thread reads it and calls server->Shutdown(), so in-flight sessions finish and the process exits 0 printing ShardRuntime worker shut down cleanly. A --selftest mode binds an ephemeral port and self-drives capability/health/fragmented-prefill/ decode/release over a real loopback gRPC channel, giving a pure-C++ CTest that needs no Python.

packages/node/native/CMakeLists.txt (modified)

Adds the shard_worker executable (linking only shard_runtime_grpc + gRPC::grpc++ — no llama.cpp) and registers shard_worker_selftest as a CTest.

tests/test_native_shard_worker.py (new)

18 integration tests that spawn the real compiled binary as a subprocess and drive it with the committed generated stubs over a real localhost socket. When the binary is not built they skip (the DGR-029/030 requires_cmake gating pattern), locating it via MESHNET_SHARD_WORKER_BIN or build/native/shard_worker.

Acceptance criteria → evidence

1. Standalone C++ executable serves the complete lifecycle/stream contract using the fake engine — shard_worker builds and serves all five RPCs; the shard_worker_selftest CTest drives open → fragmented prefill → decode → release over real gRPC; the 18 Python tests cover the same against the subprocess.
2. Python integration tests cover startup, health, capability, fragmented prefill, decode, release, cancellation, graceful shutdown — test_worker_startup_and_health, test_worker_capability, test_fragmented_prefill_echoes_reassembled_payload (3-fragment tiling), test_decode_step_is_served, test_release_is_terminal, test_in_band_cancel_of_single_work_item_does_not_end_stream, test_in_band_cancel_of_whole_session_is_terminal, test_out_of_band_cancel_rpc_races_ahead_of_open, test_graceful_shutdown_on_sigterm (SIGTERM → exit 0 + clean-shutdown line).
3. Bounded messages, deadlines, flow control, independent session cancellation enforced — test_bounded_message_is_rejected (RESOURCE_EXHAUSTED on an over-ceiling tensor), test_expired_deadline_is_rejected, test_flow_control_violation_and_topup, test_independent_session_cancellation (cancelling session A leaves session B fully serviceable), plus test_stale_route_epoch_is_rejected, test_duplicate_idempotency_step_is_acked, test_malformed_fragment_tiling_is_rejected.
4. Exposes neither llama.cpp RPC nor arbitrary graph execution — ldd build/native/shard_worker shows no llama/ggml shared libs; nm -C build/native/shard_worker | grep -icE 'llama_|ggml_' → 0; the proto exposes exactly one service with five lifecycle RPCs and no graph-exec entry.
5. Gates + this handoff — below.

Commands and results

Toolchain (ignored build/native-toolchain, pinned Protobuf 33.1 + gRPC 1.82.1):

bash scripts/bootstrap_native_toolchain.sh "$PWD/build/native-toolchain"
# ... gRPC 1.82.1 commit acccf84c0df20487d64101f528e5d426541ca4e5
# grpc_cpp_plugin sha256 43705cf26ae9ce98bbcee76b3408f5e171eec746b50bf0dd42dd68d132c6a533

Focused out-of-tree CMake build + CTest:

cmake -S packages/node/native -B build/native -DCMAKE_PREFIX_PATH="$PWD/build/native-toolchain"
cmake --build build/native -j"$(nproc)"
ctest --test-dir build/native --output-on-failure

1/2 Test #1: shard_worker_selftest ............   Passed    0.01 sec
2/2 Test #2: shard_protocol_conformance .......   Passed    0.00 sec
100% tests passed out of 2

Python integration tests against the real binary:

PYTHONPATH=packages/node:packages/tracker python -m pytest -q tests/test_native_shard_worker.py

18 passed in 3.96s

AC4 (no llama.cpp / no graph exec):

ldd build/native/shard_worker | grep -iE 'llama|ggml'   # -> (no matches)
nm build/native/shard_worker | grep -icE 'llama_|ggml_' # -> 0

Shared gates + regression:

python -m compileall -q packages tests                                  # exit 0
git diff --check -- packages/node/native tests/test_native_shard_worker.py  # exit 0
PYTHONPATH=packages/node:packages/tracker python -m pytest -q \
  tests/test_shard_runtime_harness.py tests/test_native_shard_protocol.py
# -> 61 passed, 2 skipped   (DGR-024 harness + native protocol untouched)

Toolchain used: cmake/ctest from the distributed-gguf-runtime worktree's .venv (PyPI cmake==4.4.0 wheel — no system cmake exists here, same as DGR-029/030); the Python client uses that venv's grpcio==1.82.1, grpcio-tools==1.82.1, protobuf, pytest. g++ (GCC) 15.2.1.

Limitations

• This is FIXTURE evidence only. The worker's "forward" is a CRC-over-wire-bytes echo, not real tensor compute; it proves process/stream/lifecycle/supervision shape, nothing about numerical correctness. Real engine binding is DGR-037 and numeric parity is DGR-036/052.
• The worker checksum path mirrors the DGR-024 runtime's zlib.crc32 (standard CRC-32 under a CRC32C label). Compressed-tensor tiling/checksum is not independently verified (no zstd decompressor in the fixture) — identical to the DGR-024 limitation.
• Default pytest runs skip tests/test_native_shard_worker.py unless the worker binary is built (or MESHNET_SHARD_WORKER_BIN is set); this session built it and ran all 18 for real (results above). Building requires the pinned gRPC C++ toolchain, which is not present by default and must be bootstrapped.
• No CUDA/ROCm/GPU, no model download, no network at test time — all default tests are fixture-only and offline.

Dependency handoff

• DGR-036 (fixture vs real-model parity): the worker's FakeShardEngine carries kEvidenceClass = "fixture"; diff it against DGR-037's real engine's equivalent marker, and reuse the same lifecycle/stream contract this worker serves to prove behavioural parity before numeric parity.
• DGR-037 (bind llama.cpp): replace FakeShardEngine's bounded forward with the real engine behind the same ShardRuntimeServiceImpl surface; the service's session/epoch/credit/dedup/cancel machinery and the graceful-shutdown supervision shape are reusable as-is.
• DGR-040 (worker supervision): shard_worker already provides the supervision primitives — a readiness line for start detection, SIGTERM graceful drain with a clean-exit line, and a --selftest liveness probe. A supervisor can start/monitor/restart the process around these.

Cross-review repair (2026-07-26)

An independent Codex GPT-5.5 review BLOCKED the initial implementation. Four root protocol defects in the native worker were fixed in this worktree (.claude/worktrees/distributed-gguf-opus); the fake-engine echo semantics and supervision shape are unchanged.

Defects fixed

1. Activation before SessionOpen bypassed all state. A chunk/decode whose route_session_id had no opened session fell through every if (state && ...) guard and was echoed — bypassing lifecycle, cancellation, epoch and flow-control. SessionState now carries an opened flag set only by a valid SessionOpen; chunk and decode fail closed with a terminal ERROR_CODE_INTERNAL and end the stream when it is false. A placeholder state created by an out-of-band Cancel that races Open has opened == false, so it can never admit work either.
2. Flow control blindly trusted the peer proposal. SessionOpen copied the proposed credits/max_inflight/max_chunk_bytes verbatim into session state and the accepted reply. New ShardRuntimeServiceImpl::NegotiateFlow takes the strictest bound of peer-vs-worker for every field (mirroring negotiate_flow_control in native_protocol/codec.py), stores the negotiated ceilings on the session, and enforces the negotiated per-session max_chunk_bytes on every bundle (FakeShardEngine::Validate now takes the ceiling as an argument instead of a fixed construction-time value).
3. In-stream ReleaseSignal leaked session state. The stream release arm wrote a terminal status but never dropped the session. It now erases the session under the lock before responding, so KV/credits/dedup are freed immediately (the out-of-band Release RPC already erased).
4. SessionOpen echoed caller identity instead of validating it. The handshake now rejects an incompatible schema_version (SCHEMA_UNSUPPORTED), a mismatched model/recipe Fingerprint (FINGERPRINT_MISMATCH), and a ShardRange outside the worker's served range (SHARD_RANGE_MISMATCH), each terminal; SessionAccepted now reports the worker's own served fingerprint rather than a copy of the caller's.

Changed files (repair)

• packages/node/native/worker/shard_service.h — opened + max_prefill_chunk_tokens on SessionState; NegotiateFlow decl; engine now default-constructed.
• packages/node/native/worker/shard_service.cpp — worker-identity constants + fill helpers; NegotiateFlow; SessionOpen validation/negotiation; fail-closed chunk/decode; per-session max_chunk_bytes; in-stream release erase.
• packages/node/native/worker/fake_engine.h — Validate(bundle, max_chunk_bytes).
• tests/test_native_shard_worker.py — extended _open (schema/fingerprint/range/ flow overrides); fixed test_release_rpc_is_idempotent for the new erase semantics; added 9 regression tests (chunk/decode before open, flow-control clamp, negotiated-ceiling cap, in-stream release erase, schema/fingerprint/range rejection, worker-fingerprint-not-caller).

Re-run gates (real, rebuilt binary)

Build driven through the pinned cmake (Unix Makefiles + gmake, gRPC 1.82.1):

cmake --build build/native --parallel 8            -> BUILD_EXIT 0
ctest --test-dir build/native --output-on-failure  -> 100% (2/2) passed
  shard_worker_selftest ....... Passed
  shard_protocol_conformance .. Passed
python -m pytest -q tests/test_native_shard_worker.py         -> 27 passed
python -m pytest -q tests/test_shard_runtime_harness.py \
                    tests/test_native_shard_protocol.py       -> 63 passed
python -m compileall -q packages tests                       -> exit 0
git diff --check                                             -> clean
ldd build/native/shard_worker | grep -iE 'llama|ggml'        -> NONE
nm -C build/native/shard_worker | grep -cE 'llama_|ggml_'    -> 0

The worker integration suite grew from 18 to 27 tests; all pass against the freshly compiled binary. No .ralph-lane runtime artifacts were touched.