# DGR-032 evidence — deterministic fake `ShardEngine` **Completed:** 2026-07-23 **Branch:** `ralph/distributed-gguf-runtime` **Authority:** `.scratch/distributed-gguf-runtime/prd.json` **Dependencies:** DGR-031 (`evidence/DGR-031/README.md` — the project-owned `ShardEngine` abstract contract, `tests/shard_engine_contract.py`'s `assert_shard_engine_contract`, and its own dependency-handoff note that DGR-032 should "subclass `ShardEngine`, add delay/memory-pressure/malformed- output/crash injection, and pass the *same* `assert_shard_engine_contract` ... — no new contract vocabulary should be needed"). ## Objective Provide an engine fixture that deterministically transforms typed boundary bundles and session state: head/middle/tail, prefill/decode, cancellation, release, isolated per-session epoch state, deterministic cache-miss/stale- epoch failures, and configurable delay/memory-pressure/malformed-output/ crash-injection fault surfaces — all without llama.cpp, a GPU, or any I/O. ## What was found live before changing code - `packages/node/meshnet_node/shard_engine.py` (DGR-031): the abstract `ShardEngine` with eight operations (`load`, `capabilities`, `prefill`, `decode`, `cancel`, `release`, `health`, `metrics`) and its project-owned dataclasses (`LoadRequest`, `EngineCapabilities`, `PrefillRequest`/ `DecodeRequest`, `StepResult`, `BoundaryBundle`/`EngineTensor`, `TokenOutput`, `HealthResult`, `MetricsResult`). - `tests/shard_engine_contract.py` (DGR-031): the reusable `assert_shard_engine_contract(make_engine)` helper — nine lifecycle checks any implementation must pass, explicitly designed to be imported by DGR-032 and DGR-037 against their own engines. - `tests/test_shard_engine.py` (DGR-031): its `_ReferenceEngine` is explicitly documented as *not* the DGR-032 fixture ("no delay/memory- pressure/malformed/crash injection... that is a separate, larger story") — confirming this story starts from nothing, not inherited credit. - `grep -rn -i "fakeshardengine\|fake_shard_engine"` across `.py`/`.md` returned no prior matches — no fake engine existed before this story. - No file in `packages/node/meshnet_node/` wires a `ShardEngine` into `shard_runtime_server.py` yet (confirmed by grep for `ShardEngine`/ `shard_engine` in that file — no matches); that wiring is DGR-037's scope, so this fixture is a standalone, importable engine only. Live verification of the pre-existing dependency contract before adding new code: ```bash PYTHONPATH=packages/node:packages/tracker .venv/bin/python3 -m pytest -q tests/test_shard_engine.py ``` ```text 12 passed in 0.13s ``` ## What was added (this story's change) ### `packages/node/meshnet_node/fake_shard_engine.py` (new) `FakeShardEngine(ShardEngine)` — a pure-Python, deterministic fixture: - **Determinism.** Every `prefill`/`decode` output is `SHA-256(seed_bytes + idempotency_step)`, where `seed_bytes` is derived from `token_ids` (head) or the input `BoundaryBundle`'s tensor bytes plus any `token_id_sideband` (middle/tail-in). Replaying identical inputs on a brand-new session produces byte-identical output — proven by `assert_shard_engine_contract`'s own determinism check and reused directly. - **Head/middle/tail.** Tail shards (`shard_end >= total_layers - 1`) return a `TokenOutput` sampled into `[0, TOKEN_ID_VOCAB_SIZE)`; head/middle shards return a `BoundaryBundle` tagged `boundary_point="post_head_residual"` or `"post_middle_residual"` respectively, so the three cases are distinguishable in fixture output, not just in the load request. A middle shard's `token_id_sideband` passes through unchanged from its input bundle to its output bundle (the V4 first-three-hash-routed-layers requirement RALPH-CONTEXT documents), never invented or dropped. - **Isolated session/epoch state.** `_sessions: dict[str, _SessionState]` keyed by `session_id`; each session tracks its own `epoch`/`cancelled` flag. A stale epoch, cancel, or release on one session never touches another's state (`test_session_state_is_isolated_between_two_concurrent_sessions` proves a stale-epoch rejection and a cancel on session `"a"` leave session `"b"` fully serviceable). Decoding an unopened session is a deterministic `NOT_FOUND`/`CacheResult.MISS`, not an exception. - **Configurable delay.** `FakeShardEngineConfig.step_delay_seconds` + injectable `sleep` hook (defaults to `time.sleep`, overridable in tests so they don't block wall-clock time) — invoked once per `prefill`/`decode` call before computing the deterministic output. - **Configurable memory pressure.** `FakeShardEngineConfig.memory_budget_bytes` — the engine accumulates `_bytes_used` across every step's seed bytes; once a step would push cumulative usage past the budget, that step deterministically returns `StatusCode.RESOURCE_EXHAUSTED` (`retryable=True`) with no output, instead of computing one. - **Configurable malformed output.** `FakeShardEngineConfig.malformed_output` — when set, the engine still reports `StatusCode.OK` (the point is a buggy-but-"successful"-looking response, not a status-coded failure) but the payload is structurally valid, semantically wrong: a tail `TokenOutput` is pushed past `MALFORMED_TOKEN_ID_FLOOR` (outside the fixture's own advertised vocab), and a head/middle `BoundaryBundle` gets an `architecture` field prefixed `"malformed:"` and its tensor `data` truncated to one byte — both structurally valid per `EngineTensor`'s and `BoundaryBundle`'s own `__post_init__` validation (which does not cross-check `data` length against `shape`/`dtype`), so a consumer must actually check shape/semantics, not just status codes, to catch it. - **Configurable crash injection.** `FakeShardEngineConfig.crash_after_calls` + `crash_exception_factory` — after the configured number of `prefill`/`decode` calls, the engine raises an arbitrary exception (default `RuntimeError`, injectable) directly out of the call instead of returning a `StepResult`. This is deliberately *not* wrapped in `EngineError`/ `StructuredStatus`: it simulates a whole-process failure (what a worker supervisor — DGR-040 — must catch and restart around), which is a different failure mode from a graceful status-coded rejection. - **Fixture-vs-real marker.** `FakeShardEngine.EVIDENCE_CLASS = "fixture"` — a structural constant (not just docstring prose) so DGR-036's fixture-vs- real-model parity check can assert programmatically that it is comparing a fixture engine against a real one, never two fixtures. - Every fault-injection knob defaults to off (`0`/`None`/`False`), so a bare `FakeShardEngine()` passes `assert_shard_engine_contract` unmodified — fault injection is opt-in, never a baseline behavior change. ### `tests/test_fake_shard_engine.py` (new) - `test_fake_shard_engine_obeys_the_shared_shard_engine_contract` — runs the full DGR-031 contract against a bare `FakeShardEngine`. - `test_fake_shard_engine_declares_fixture_evidence_class` — pins the `EVIDENCE_CLASS` marker DGR-036 will rely on. - Head/middle/tail output-shape tests (`boundary_point`, token-id-sideband pass-through, tail vocab range). - `test_session_state_is_isolated_between_two_concurrent_sessions` — a stale-epoch rejection and a cancel on one session leave a second, concurrently open session fully serviceable. - One test per fault-injection knob (delay hook invocation, memory-budget trip, malformed tail/boundary-bundle output, crash-after-N-calls, configurable crash exception type) plus `FakeShardEngineConfig`'s own `__post_init__` validation (negative delay, negative budget, non-positive `crash_after_calls`). - `test_load_result_and_capabilities_report_recipe_architecture` — the fixture threads `LoadRequest.recipe["architecture"]` through to both `LoadResult.architecture` and `EngineCapabilities.architecture` rather than hardcoding `"dense"`/`"fake"` everywhere, so a future V4 recipe is visible in fixture output too. ### `.scratch/distributed-gguf-runtime/prd.json` / issue markdown Marked `DGR-032.passes = true` with `completionNotes`; regenerated `issues/032-implement-deterministic-fake-shardengine.md` via `scripts/ralph_prd_schema.py render` so it matches `prd.json` byte-for-byte. ## Acceptance criteria → evidence 1. **Head, middle, tail, prefill, decode, cancellation, release with deterministic outputs** — `FakeShardEngine`'s `_transform`, boundary-point tagging, and `assert_shard_engine_contract`'s own determinism/cancel/ release checks. Verified by `test_fake_shard_engine_obeys_the_shared_shard_engine_contract`, `test_head_shard_returns_boundary_bundle_with_post_head_residual_point`, `test_middle_shard_returns_boundary_bundle_and_passes_through_token_sideband`, `test_tail_shard_returns_token_output_within_advertised_vocab`. 2. **Isolated session/epoch state and deterministic cache-miss/stale-epoch failures** — `_sessions` dict keyed per session; `test_session_state_is_isolated_between_two_concurrent_sessions` plus the shared contract's own cache-miss/stale-epoch checks. 3. **Configurable delay, memory pressure, malformed output, crash injection** — `FakeShardEngineConfig`; verified by `test_step_delay_seconds_invokes_the_configured_sleep_hook`, `test_memory_budget_bytes_trips_deterministic_resource_exhausted`, `test_malformed_output_is_structurally_valid_but_semantically_wrong_for_tail`, `test_malformed_output_is_structurally_valid_but_semantically_wrong_for_boundary_bundle`, `test_crash_after_calls_raises_instead_of_returning_a_structured_status`, `test_crash_exception_factory_is_configurable`, `test_config_rejects_invalid_knob_values`. 4. **Contract tests distinguish fixture evidence from real-model certification** — module docstring and this README are explicit that this is FIXTURE evidence only (numeric parity is DGR-036 onward); the `EVIDENCE_CLASS = "fixture"` constant makes that distinction structurally checkable, not just prose, pinned by `test_fake_shard_engine_declares_fixture_evidence_class`. 5. **Gates + this handoff** — below. ## Commands and results ```bash PYTHONPATH=packages/node:packages/tracker .venv/bin/python3 -m pytest -q tests/test_fake_shard_engine.py tests/test_shard_engine.py ``` ```text 26 passed in 0.17s ``` ```bash PYTHONPATH=packages/node:packages/tracker .venv/bin/python3 -m pytest -q \ tests/test_fake_shard_engine.py tests/test_shard_engine.py tests/test_shard_lifecycle.py \ tests/test_architecture_boundary.py tests/test_activation_envelope.py \ tests/test_native_shard_protocol.py tests/test_shard_runtime_harness.py ``` ```text 109 passed, 3 skipped in 3.78s ``` ```bash .venv/bin/python3 -m compileall -q packages tests ``` ```text (no output — clean; exit 0) ``` ```bash git diff --check ``` ```text (no output — clean) ``` ## Limitations - `tests/` as a whole does not collect cleanly in this environment: the same pre-existing collection errors DGR-031's evidence recorded (missing optional dependencies such as `cryptography`) are still present and are unrelated to this story. This story's own gates were run as the targeted, scoped test set above per the shared quality gates' wording ("Targeted deterministic tests pass"). - This is FIXTURE evidence only. `FakeShardEngine` proves lifecycle, session/epoch isolation, and fault-injection semantics; it proves nothing about numerical parity with a real model. That is DGR-036's explicit job once DGR-037's real engine exists, and DGR-053/054 for V4 alpha certification. - `FakeShardEngine` is not wired into `shard_runtime_server.py` or any gRPC surface — it is a standalone, importable engine only. Wiring a `ShardEngine` (fake or real) into the gRPC servicer is DGR-037's scope for the real engine; DGR-033 covers a C++ worker surface, which is a separate native executable, not a consumer of this Python module. - No gRPC/CMake/native-build changes were needed or made — this story is pure Python fixture code (`evidenceClass: fixture`, `hardware: none`), so the native CMake/CTest and patch-stack gates in the shared quality-gate list do not apply here, consistent with DGR-031's own README recording the same non-applicability. ## Dependency handoff - **DGR-033** (standalone fake C++ gRPC Shard worker): its own issue describes a native C++ executable serving the lifecycle/stream RPC contract "using the fake engine" — that is a native analogue, not a consumer of this Python module; DGR-033 should still read this README for the exact deterministic-output/session-isolation/fault-injection semantics its C++ fake engine needs to reproduce so both fakes behave identically from a client's point of view. - **DGR-034/DGR-035** (range-aware GGUF ownership, boundary I/O): `FakeShardEngine` already demonstrates range-driven head/middle/tail behavior purely from `LoadRequest.shard_start`/`shard_end`/`total_layers`; no new range vocabulary was needed. - **DGR-036** (fixture vs real-model parity): compare a `FakeShardEngine` instance's `EVIDENCE_CLASS` (`"fixture"`) against DGR-037's real engine's equivalent marker (expected `"real"`) to assert the parity check is actually comparing two different implementations; reuse `assert_shard_engine_contract` against both to prove lifecycle parity before attempting numeric parity. - **DGR-037** (bind llama.cpp to the worker): `FakeShardEngine` is the reference implementation to diff a real engine's lifecycle behavior against — same request/result types, same session/epoch model, no new contract vocabulary. - **DGR-040** (worker supervision): the crash-injection knob (`crash_after_calls`/`crash_exception_factory`) exists specifically so supervision/restart logic has a deterministic way to trigger and test an unhandled engine failure distinct from a graceful `StructuredStatus` rejection.