story: DGR-033 Build a standalone fake C++ gRPC Shard worker
This commit is contained in:
Dobromir Popov
208
.scratch/distributed-gguf-runtime/evidence/DGR-033/README.md
Normal file
208
.scratch/distributed-gguf-runtime/evidence/DGR-033/README.md
Normal file
@@ -0,0 +1,208 @@
|
||||
# DGR-033 evidence — standalone fake C++ gRPC Shard worker
|
||||
|
||||
**Completed:** 2026-07-25
|
||||
**Branch:** `ralph/distributed-gguf-opus`
|
||||
**Authority:** `.scratch/distributed-gguf-runtime/prd.json`
|
||||
**Dependencies:** DGR-022 (lifecycle/status contract), DGR-024 (real generated
|
||||
gRPC harness + `shard_runtime_server.py` reference semantics), DGR-032
|
||||
(deterministic fake `ShardEngine` semantics).
|
||||
|
||||
## Objective
|
||||
|
||||
Prove the standalone worker process, stream, lifecycle, and supervision shape
|
||||
before any llama.cpp integration: a real C++ executable that serves the whole
|
||||
ShardRuntime lifecycle/stream contract over gRPC using a model-free fake engine,
|
||||
driven end-to-end by Python integration tests over a real socket.
|
||||
|
||||
## What was found live before changing code
|
||||
|
||||
- `packages/node/native/proto/shard_runtime.proto` (DGR-021..023): the single
|
||||
semantic contract. Its `ShardRuntime` service has exactly five RPCs —
|
||||
`GetCapability`, `Health`, `Session` (bidi stream), `Release`, `Cancel`.
|
||||
- `packages/node/meshnet_node/shard_runtime_server.py` (DGR-024): the reference
|
||||
Python servicer. It performs a *bounded real forward* (a CRC over the received
|
||||
bundle bytes) then echoes the chunk, and fails closed on stale epoch, expired
|
||||
deadline, corrupt/mis-tiled fragments, exhausted flow-control credit, duplicate
|
||||
idempotency step, and in-band/out-of-band cancellation, with per-`route_session_id`
|
||||
state kept on the servicer so an out-of-band `Cancel` can reach a live session.
|
||||
**Key finding:** despite the schema labelling the checksum `CRC32C`, this
|
||||
runtime computes it with `zlib.crc32` (standard CRC-32, *not* Castagnoli). The
|
||||
C++ worker mirrors `zlib.crc32` exactly so its checksum acceptance is
|
||||
byte-identical to the existing Python surface (the committed C++ *conformance*
|
||||
test, by contrast, uses true Castagnoli against separately-generated goldens —
|
||||
the two are unrelated code paths).
|
||||
- `packages/node/native/CMakeLists.txt` (DGR-029/030): configures against the
|
||||
ignored `build/native-toolchain` prefix (pinned Protobuf 33.1 + gRPC 1.82.1),
|
||||
always generates both message and service stubs, and registers a C++
|
||||
conformance CTest. There was **no** worker executable and **no** Python
|
||||
worker integration test before this story (confirmed by
|
||||
`ls packages/node/native/worker` → absent, and grep for `shard_worker`).
|
||||
- `packages/node/meshnet_node/fake_shard_engine.py` (DGR-032): the Python fake
|
||||
engine, deliberately *not* wired into the gRPC surface. DGR-033's worker is
|
||||
its native analogue — a separate executable, not a consumer of that module —
|
||||
so both fakes present identical behaviour to a client (deterministic,
|
||||
model-free bounded forward; per-session isolation; fail-closed lifecycle).
|
||||
|
||||
## What was added (this story's change)
|
||||
|
||||
### `packages/node/native/worker/fake_engine.h` (new)
|
||||
|
||||
`meshnet::worker::FakeShardEngine` — a header-only, model-free fixture engine.
|
||||
Its only capability is to validate a `TensorBundle` (fragments tile exactly, the
|
||||
uncompressed CRC-32 matches the declared checksum, the declared payload stays
|
||||
within the negotiated `max_chunk_bytes`) and fold the fragment bytes through a
|
||||
bounded forward. It links, loads, and dispatches to **nothing** — no llama.cpp,
|
||||
no graph execution. Carries `kEvidenceClass = "fixture"` mirroring the Python
|
||||
`FakeShardEngine.EVIDENCE_CLASS` for the later DGR-036 parity check.
|
||||
|
||||
### `packages/node/native/worker/shard_service.{h,cpp}` (new)
|
||||
|
||||
`ShardRuntimeServiceImpl : meshnet::shard::v1::ShardRuntime::Service` — a faithful
|
||||
C++ port of the DGR-024 Python servicer: the same per-`route_session_id`
|
||||
identity/credit/dedup state guarded by a mutex, the same fail-closed negative
|
||||
paths, and the same lifecycle (open → prefill/decode → flow-control top-up →
|
||||
release/cancel). Each per-request response is computed under the lock and written
|
||||
*after* releasing it, so a blocking `Write` can never deadlock the out-of-band
|
||||
`Cancel` RPC that needs the same lock. Bounded messages are enforced two ways: a
|
||||
per-tensor `RESOURCE_EXHAUSTED` app check against `max_chunk_bytes`, plus a hard
|
||||
transport receive ceiling.
|
||||
|
||||
### `packages/node/native/worker/shard_worker_main.cpp` (new)
|
||||
|
||||
The standalone `shard_worker` executable. Binds `MESHNET_SHARD_LISTEN_ADDR`
|
||||
(or an `argv` address), prints one readiness line (`ShardRuntime worker listening
|
||||
on <addr>`), and serves until `SIGTERM`/`SIGINT`. **Graceful shutdown** uses a
|
||||
self-pipe: the async-signal-safe handler writes one byte, a drain thread reads it
|
||||
and calls `server->Shutdown()`, so in-flight sessions finish and the process
|
||||
exits `0` printing `ShardRuntime worker shut down cleanly`. A `--selftest` mode
|
||||
binds an ephemeral port and self-drives capability/health/fragmented-prefill/
|
||||
decode/release over a real loopback gRPC channel, giving a pure-C++ CTest that
|
||||
needs no Python.
|
||||
|
||||
### `packages/node/native/CMakeLists.txt` (modified)
|
||||
|
||||
Adds the `shard_worker` executable (linking only `shard_runtime_grpc` +
|
||||
`gRPC::grpc++` — no llama.cpp) and registers `shard_worker_selftest` as a CTest.
|
||||
|
||||
### `tests/test_native_shard_worker.py` (new)
|
||||
|
||||
18 integration tests that spawn the **real compiled binary** as a subprocess and
|
||||
drive it with the committed generated stubs over a real localhost socket. When
|
||||
the binary is not built they skip (the DGR-029/030 `requires_cmake` gating
|
||||
pattern), locating it via `MESHNET_SHARD_WORKER_BIN` or `build/native/shard_worker`.
|
||||
|
||||
## Acceptance criteria → evidence
|
||||
|
||||
1. **Standalone C++ executable serves the complete lifecycle/stream contract
|
||||
using the fake engine** — `shard_worker` builds and serves all five RPCs; the
|
||||
`shard_worker_selftest` CTest drives open → fragmented prefill → decode →
|
||||
release over real gRPC; the 18 Python tests cover the same against the
|
||||
subprocess.
|
||||
2. **Python integration tests cover startup, health, capability, fragmented
|
||||
prefill, decode, release, cancellation, graceful shutdown** —
|
||||
`test_worker_startup_and_health`, `test_worker_capability`,
|
||||
`test_fragmented_prefill_echoes_reassembled_payload` (3-fragment tiling),
|
||||
`test_decode_step_is_served`, `test_release_is_terminal`,
|
||||
`test_in_band_cancel_of_single_work_item_does_not_end_stream`,
|
||||
`test_in_band_cancel_of_whole_session_is_terminal`,
|
||||
`test_out_of_band_cancel_rpc_races_ahead_of_open`,
|
||||
`test_graceful_shutdown_on_sigterm` (SIGTERM → exit 0 + clean-shutdown line).
|
||||
3. **Bounded messages, deadlines, flow control, independent session
|
||||
cancellation enforced** — `test_bounded_message_is_rejected`
|
||||
(`RESOURCE_EXHAUSTED` on an over-ceiling tensor),
|
||||
`test_expired_deadline_is_rejected`, `test_flow_control_violation_and_topup`,
|
||||
`test_independent_session_cancellation` (cancelling session A leaves session B
|
||||
fully serviceable), plus `test_stale_route_epoch_is_rejected`,
|
||||
`test_duplicate_idempotency_step_is_acked`,
|
||||
`test_malformed_fragment_tiling_is_rejected`.
|
||||
4. **Exposes neither llama.cpp RPC nor arbitrary graph execution** —
|
||||
`ldd build/native/shard_worker` shows no llama/ggml shared libs;
|
||||
`nm -C build/native/shard_worker | grep -icE 'llama_|ggml_'` → `0`; the proto
|
||||
exposes exactly one service with five lifecycle RPCs and no graph-exec entry.
|
||||
5. **Gates + this handoff** — below.
|
||||
|
||||
## Commands and results
|
||||
|
||||
Toolchain (ignored `build/native-toolchain`, pinned Protobuf 33.1 + gRPC 1.82.1):
|
||||
|
||||
```bash
|
||||
bash scripts/bootstrap_native_toolchain.sh "$PWD/build/native-toolchain"
|
||||
# ... gRPC 1.82.1 commit acccf84c0df20487d64101f528e5d426541ca4e5
|
||||
# grpc_cpp_plugin sha256 43705cf26ae9ce98bbcee76b3408f5e171eec746b50bf0dd42dd68d132c6a533
|
||||
```
|
||||
|
||||
Focused out-of-tree CMake build + CTest:
|
||||
|
||||
```bash
|
||||
cmake -S packages/node/native -B build/native -DCMAKE_PREFIX_PATH="$PWD/build/native-toolchain"
|
||||
cmake --build build/native -j"$(nproc)"
|
||||
ctest --test-dir build/native --output-on-failure
|
||||
```
|
||||
```text
|
||||
1/2 Test #1: shard_worker_selftest ............ Passed 0.01 sec
|
||||
2/2 Test #2: shard_protocol_conformance ....... Passed 0.00 sec
|
||||
100% tests passed out of 2
|
||||
```
|
||||
|
||||
Python integration tests against the real binary:
|
||||
|
||||
```bash
|
||||
PYTHONPATH=packages/node:packages/tracker python -m pytest -q tests/test_native_shard_worker.py
|
||||
```
|
||||
```text
|
||||
18 passed in 3.96s
|
||||
```
|
||||
|
||||
AC4 (no llama.cpp / no graph exec):
|
||||
|
||||
```bash
|
||||
ldd build/native/shard_worker | grep -iE 'llama|ggml' # -> (no matches)
|
||||
nm build/native/shard_worker | grep -icE 'llama_|ggml_' # -> 0
|
||||
```
|
||||
|
||||
Shared gates + regression:
|
||||
|
||||
```bash
|
||||
python -m compileall -q packages tests # exit 0
|
||||
git diff --check -- packages/node/native tests/test_native_shard_worker.py # exit 0
|
||||
PYTHONPATH=packages/node:packages/tracker python -m pytest -q \
|
||||
tests/test_shard_runtime_harness.py tests/test_native_shard_protocol.py
|
||||
# -> 61 passed, 2 skipped (DGR-024 harness + native protocol untouched)
|
||||
```
|
||||
|
||||
Toolchain used: `cmake`/`ctest` from the `distributed-gguf-runtime` worktree's
|
||||
`.venv` (PyPI `cmake==4.4.0` wheel — no system cmake exists here, same as
|
||||
DGR-029/030); the Python client uses that venv's `grpcio==1.82.1`,
|
||||
`grpcio-tools==1.82.1`, `protobuf`, `pytest`. `g++ (GCC) 15.2.1`.
|
||||
|
||||
## Limitations
|
||||
|
||||
- This is FIXTURE evidence only. The worker's "forward" is a CRC-over-wire-bytes
|
||||
echo, not real tensor compute; it proves process/stream/lifecycle/supervision
|
||||
shape, nothing about numerical correctness. Real engine binding is DGR-037 and
|
||||
numeric parity is DGR-036/052.
|
||||
- The worker checksum path mirrors the DGR-024 runtime's `zlib.crc32` (standard
|
||||
CRC-32 under a `CRC32C` label). Compressed-tensor tiling/checksum is not
|
||||
independently verified (no zstd decompressor in the fixture) — identical to the
|
||||
DGR-024 limitation.
|
||||
- Default `pytest` runs skip `tests/test_native_shard_worker.py` unless the
|
||||
worker binary is built (or `MESHNET_SHARD_WORKER_BIN` is set); this session
|
||||
built it and ran all 18 for real (results above). Building requires the pinned
|
||||
gRPC C++ toolchain, which is not present by default and must be bootstrapped.
|
||||
- No CUDA/ROCm/GPU, no model download, no network at test time — all default
|
||||
tests are fixture-only and offline.
|
||||
|
||||
## Dependency handoff
|
||||
|
||||
- **DGR-036** (fixture vs real-model parity): the worker's `FakeShardEngine`
|
||||
carries `kEvidenceClass = "fixture"`; diff it against DGR-037's real engine's
|
||||
equivalent marker, and reuse the same lifecycle/stream contract this worker
|
||||
serves to prove behavioural parity before numeric parity.
|
||||
- **DGR-037** (bind llama.cpp): replace `FakeShardEngine`'s bounded forward with
|
||||
the real engine behind the *same* `ShardRuntimeServiceImpl` surface; the
|
||||
service's session/epoch/credit/dedup/cancel machinery and the graceful-shutdown
|
||||
supervision shape are reusable as-is.
|
||||
- **DGR-040** (worker supervision): `shard_worker` already provides the
|
||||
supervision primitives — a readiness line for start detection, `SIGTERM`
|
||||
graceful drain with a clean-exit line, and a `--selftest` liveness probe.
|
||||
A supervisor can start/monitor/restart the process around these.
|
||||
Reference in New Issue
Block a user