story: DGR-032 Implement deterministic fake ShardEngine

.scratch/distributed-gguf-runtime/evidence/DGR-032/README.md

@@ -0,0 +1,259 @@
+# DGR-032 evidence — deterministic fake `ShardEngine`
+
+**Completed:** 2026-07-23
+**Branch:** `ralph/distributed-gguf-runtime`
+**Authority:** `.scratch/distributed-gguf-runtime/prd.json`
+**Dependencies:** DGR-031 (`evidence/DGR-031/README.md` — the project-owned
+`ShardEngine` abstract contract, `tests/shard_engine_contract.py`'s
+`assert_shard_engine_contract`, and its own dependency-handoff note that
+DGR-032 should "subclass `ShardEngine`, add delay/memory-pressure/malformed-
+output/crash injection, and pass the *same* `assert_shard_engine_contract`
+... — no new contract vocabulary should be needed").
+
+## Objective
+
+Provide an engine fixture that deterministically transforms typed boundary
+bundles and session state: head/middle/tail, prefill/decode, cancellation,
+release, isolated per-session epoch state, deterministic cache-miss/stale-
+epoch failures, and configurable delay/memory-pressure/malformed-output/
+crash-injection fault surfaces — all without llama.cpp, a GPU, or any I/O.
+
+## What was found live before changing code
+
+- `packages/node/meshnet_node/shard_engine.py` (DGR-031): the abstract
+  `ShardEngine` with eight operations (`load`, `capabilities`, `prefill`,
+  `decode`, `cancel`, `release`, `health`, `metrics`) and its project-owned
+  dataclasses (`LoadRequest`, `EngineCapabilities`, `PrefillRequest`/
+  `DecodeRequest`, `StepResult`, `BoundaryBundle`/`EngineTensor`,
+  `TokenOutput`, `HealthResult`, `MetricsResult`).
+- `tests/shard_engine_contract.py` (DGR-031): the reusable
+  `assert_shard_engine_contract(make_engine)` helper — nine lifecycle checks
+  any implementation must pass, explicitly designed to be imported by
+  DGR-032 and DGR-037 against their own engines.
+- `tests/test_shard_engine.py` (DGR-031): its `_ReferenceEngine` is
+  explicitly documented as *not* the DGR-032 fixture ("no delay/memory-
+  pressure/malformed/crash injection... that is a separate, larger story") —
+  confirming this story starts from nothing, not inherited credit.
+- `grep -rn -i "fakeshardengine\|fake_shard_engine"` across `.py`/`.md`
+  returned no prior matches — no fake engine existed before this story.
+- No file in `packages/node/meshnet_node/` wires a `ShardEngine` into
+  `shard_runtime_server.py` yet (confirmed by grep for `ShardEngine`/
+  `shard_engine` in that file — no matches); that wiring is DGR-037's scope,
+  so this fixture is a standalone, importable engine only.
+
+Live verification of the pre-existing dependency contract before adding new
+code:
+```bash
+PYTHONPATH=packages/node:packages/tracker .venv/bin/python3 -m pytest -q tests/test_shard_engine.py
+```
+```text
+12 passed in 0.13s
+```
+
+## What was added (this story's change)
+
+### `packages/node/meshnet_node/fake_shard_engine.py` (new)
+
+`FakeShardEngine(ShardEngine)` — a pure-Python, deterministic fixture:
+
+- **Determinism.** Every `prefill`/`decode` output is `SHA-256(seed_bytes +
+  idempotency_step)`, where `seed_bytes` is derived from `token_ids` (head)
+  or the input `BoundaryBundle`'s tensor bytes plus any `token_id_sideband`
+  (middle/tail-in). Replaying identical inputs on a brand-new session
+  produces byte-identical output — proven by
+  `assert_shard_engine_contract`'s own determinism check and reused directly.
+- **Head/middle/tail.** Tail shards (`shard_end >= total_layers - 1`) return
+  a `TokenOutput` sampled into `[0, TOKEN_ID_VOCAB_SIZE)`; head/middle shards
+  return a `BoundaryBundle` tagged `boundary_point="post_head_residual"` or
+  `"post_middle_residual"` respectively, so the three cases are
+  distinguishable in fixture output, not just in the load request. A middle
+  shard's `token_id_sideband` passes through unchanged from its input bundle
+  to its output bundle (the V4 first-three-hash-routed-layers requirement
+  RALPH-CONTEXT documents), never invented or dropped.
+- **Isolated session/epoch state.** `_sessions: dict[str, _SessionState]`
+  keyed by `session_id`; each session tracks its own `epoch`/`cancelled`
+  flag. A stale epoch, cancel, or release on one session never touches
+  another's state (`test_session_state_is_isolated_between_two_concurrent_sessions`
+  proves a stale-epoch rejection and a cancel on session `"a"` leave session
+  `"b"` fully serviceable). Decoding an unopened session is a deterministic
+  `NOT_FOUND`/`CacheResult.MISS`, not an exception.
+- **Configurable delay.** `FakeShardEngineConfig.step_delay_seconds` +
+  injectable `sleep` hook (defaults to `time.sleep`, overridable in tests so
+  they don't block wall-clock time) — invoked once per `prefill`/`decode`
+  call before computing the deterministic output.
+- **Configurable memory pressure.** `FakeShardEngineConfig.memory_budget_bytes`
+  — the engine accumulates `_bytes_used` across every step's seed bytes;
+  once a step would push cumulative usage past the budget, that step
+  deterministically returns `StatusCode.RESOURCE_EXHAUSTED` (`retryable=True`)
+  with no output, instead of computing one.
+- **Configurable malformed output.** `FakeShardEngineConfig.malformed_output`
+  — when set, the engine still reports `StatusCode.OK` (the point is a
+  buggy-but-"successful"-looking response, not a status-coded failure) but
+  the payload is structurally valid, semantically wrong: a tail `TokenOutput`
+  is pushed past `MALFORMED_TOKEN_ID_FLOOR` (outside the fixture's own
+  advertised vocab), and a head/middle `BoundaryBundle` gets an
+  `architecture` field prefixed `"malformed:"` and its tensor `data`
+  truncated to one byte — both structurally valid per `EngineTensor`'s and
+  `BoundaryBundle`'s own `__post_init__` validation (which does not
+  cross-check `data` length against `shape`/`dtype`), so a consumer must
+  actually check shape/semantics, not just status codes, to catch it.
+- **Configurable crash injection.** `FakeShardEngineConfig.crash_after_calls`
+  + `crash_exception_factory` — after the configured number of
+  `prefill`/`decode` calls, the engine raises an arbitrary exception (default
+  `RuntimeError`, injectable) directly out of the call instead of returning a
+  `StepResult`. This is deliberately *not* wrapped in `EngineError`/
+  `StructuredStatus`: it simulates a whole-process failure (what a worker
+  supervisor — DGR-040 — must catch and restart around), which is a
+  different failure mode from a graceful status-coded rejection.
+- **Fixture-vs-real marker.** `FakeShardEngine.EVIDENCE_CLASS = "fixture"` —
+  a structural constant (not just docstring prose) so DGR-036's fixture-vs-
+  real-model parity check can assert programmatically that it is comparing a
+  fixture engine against a real one, never two fixtures.
+- Every fault-injection knob defaults to off (`0`/`None`/`False`), so a bare
+  `FakeShardEngine()` passes `assert_shard_engine_contract` unmodified —
+  fault injection is opt-in, never a baseline behavior change.
+
+### `tests/test_fake_shard_engine.py` (new)
+
+- `test_fake_shard_engine_obeys_the_shared_shard_engine_contract` — runs the
+  full DGR-031 contract against a bare `FakeShardEngine`.
+- `test_fake_shard_engine_declares_fixture_evidence_class` — pins the
+  `EVIDENCE_CLASS` marker DGR-036 will rely on.
+- Head/middle/tail output-shape tests (`boundary_point`, token-id-sideband
+  pass-through, tail vocab range).
+- `test_session_state_is_isolated_between_two_concurrent_sessions` — a
+  stale-epoch rejection and a cancel on one session leave a second,
+  concurrently open session fully serviceable.
+- One test per fault-injection knob (delay hook invocation, memory-budget
+  trip, malformed tail/boundary-bundle output, crash-after-N-calls,
+  configurable crash exception type) plus `FakeShardEngineConfig`'s own
+  `__post_init__` validation (negative delay, negative budget, non-positive
+  `crash_after_calls`).
+- `test_load_result_and_capabilities_report_recipe_architecture` — the
+  fixture threads `LoadRequest.recipe["architecture"]` through to both
+  `LoadResult.architecture` and `EngineCapabilities.architecture` rather than
+  hardcoding `"dense"`/`"fake"` everywhere, so a future V4 recipe is visible
+  in fixture output too.
+
+### `.scratch/distributed-gguf-runtime/prd.json` / issue markdown
+
+Marked `DGR-032.passes = true` with `completionNotes`; regenerated
+`issues/032-implement-deterministic-fake-shardengine.md` via
+`scripts/ralph_prd_schema.py render` so it matches `prd.json` byte-for-byte.
+
+## Acceptance criteria → evidence
+
+1. **Head, middle, tail, prefill, decode, cancellation, release with
+   deterministic outputs** — `FakeShardEngine`'s `_transform`, boundary-point
+   tagging, and `assert_shard_engine_contract`'s own determinism/cancel/
+   release checks. Verified by
+   `test_fake_shard_engine_obeys_the_shared_shard_engine_contract`,
+   `test_head_shard_returns_boundary_bundle_with_post_head_residual_point`,
+   `test_middle_shard_returns_boundary_bundle_and_passes_through_token_sideband`,
+   `test_tail_shard_returns_token_output_within_advertised_vocab`.
+2. **Isolated session/epoch state and deterministic cache-miss/stale-epoch
+   failures** — `_sessions` dict keyed per session;
+   `test_session_state_is_isolated_between_two_concurrent_sessions` plus the
+   shared contract's own cache-miss/stale-epoch checks.
+3. **Configurable delay, memory pressure, malformed output, crash
+   injection** — `FakeShardEngineConfig`; verified by
+   `test_step_delay_seconds_invokes_the_configured_sleep_hook`,
+   `test_memory_budget_bytes_trips_deterministic_resource_exhausted`,
+   `test_malformed_output_is_structurally_valid_but_semantically_wrong_for_tail`,
+   `test_malformed_output_is_structurally_valid_but_semantically_wrong_for_boundary_bundle`,
+   `test_crash_after_calls_raises_instead_of_returning_a_structured_status`,
+   `test_crash_exception_factory_is_configurable`,
+   `test_config_rejects_invalid_knob_values`.
+4. **Contract tests distinguish fixture evidence from real-model
+   certification** — module docstring and this README are explicit that
+   this is FIXTURE evidence only (numeric parity is DGR-036 onward); the
+   `EVIDENCE_CLASS = "fixture"` constant makes that distinction structurally
+   checkable, not just prose, pinned by
+   `test_fake_shard_engine_declares_fixture_evidence_class`.
+5. **Gates + this handoff** — below.
+
+## Commands and results
+
+```bash
+PYTHONPATH=packages/node:packages/tracker .venv/bin/python3 -m pytest -q tests/test_fake_shard_engine.py tests/test_shard_engine.py
+```
+```text
+26 passed in 0.17s
+```
+
+```bash
+PYTHONPATH=packages/node:packages/tracker .venv/bin/python3 -m pytest -q \
+  tests/test_fake_shard_engine.py tests/test_shard_engine.py tests/test_shard_lifecycle.py \
+  tests/test_architecture_boundary.py tests/test_activation_envelope.py \
+  tests/test_native_shard_protocol.py tests/test_shard_runtime_harness.py
+```
+```text
+109 passed, 3 skipped in 3.78s
+```
+
+```bash
+.venv/bin/python3 -m compileall -q packages tests
+```
+```text
+(no output — clean; exit 0)
+```
+
+```bash
+git diff --check
+```
+```text
+(no output — clean)
+```
+
+## Limitations
+
+- `tests/` as a whole does not collect cleanly in this environment: the same
+  pre-existing collection errors DGR-031's evidence recorded (missing
+  optional dependencies such as `cryptography`) are still present and are
+  unrelated to this story. This story's own gates were run as the targeted,
+  scoped test set above per the shared quality gates' wording ("Targeted
+  deterministic tests pass").
+- This is FIXTURE evidence only. `FakeShardEngine` proves lifecycle,
+  session/epoch isolation, and fault-injection semantics; it proves nothing
+  about numerical parity with a real model. That is DGR-036's explicit job
+  once DGR-037's real engine exists, and DGR-053/054 for V4 alpha
+  certification.
+- `FakeShardEngine` is not wired into `shard_runtime_server.py` or any gRPC
+  surface — it is a standalone, importable engine only. Wiring a
+  `ShardEngine` (fake or real) into the gRPC servicer is DGR-037's scope for
+  the real engine; DGR-033 covers a C++ worker surface, which is a separate
+  native executable, not a consumer of this Python module.
+- No gRPC/CMake/native-build changes were needed or made — this story is
+  pure Python fixture code (`evidenceClass: fixture`, `hardware: none`), so
+  the native CMake/CTest and patch-stack gates in the shared quality-gate
+  list do not apply here, consistent with DGR-031's own README recording the
+  same non-applicability.
+
+## Dependency handoff
+
+- **DGR-033** (standalone fake C++ gRPC Shard worker): its own issue
+  describes a native C++ executable serving the lifecycle/stream RPC
+  contract "using the fake engine" — that is a native analogue, not a
+  consumer of this Python module; DGR-033 should still read this README for
+  the exact deterministic-output/session-isolation/fault-injection semantics
+  its C++ fake engine needs to reproduce so both fakes behave identically
+  from a client's point of view.
+- **DGR-034/DGR-035** (range-aware GGUF ownership, boundary I/O):
+  `FakeShardEngine` already demonstrates range-driven head/middle/tail
+  behavior purely from `LoadRequest.shard_start`/`shard_end`/`total_layers`;
+  no new range vocabulary was needed.
+- **DGR-036** (fixture vs real-model parity): compare a `FakeShardEngine`
+  instance's `EVIDENCE_CLASS` (`"fixture"`) against DGR-037's real engine's
+  equivalent marker (expected `"real"`) to assert the parity check is
+  actually comparing two different implementations; reuse
+  `assert_shard_engine_contract` against both to prove lifecycle parity
+  before attempting numeric parity.
+- **DGR-037** (bind llama.cpp to the worker): `FakeShardEngine` is the
+  reference implementation to diff a real engine's lifecycle behavior
+  against — same request/result types, same session/epoch model, no new
+  contract vocabulary.
+- **DGR-040** (worker supervision): the crash-injection knob
+  (`crash_after_calls`/`crash_exception_factory`) exists specifically so
+  supervision/restart logic has a deterministic way to trigger and test an
+  unhandled engine failure distinct from a graceful `StructuredStatus`
+  rejection.

.scratch/distributed-gguf-runtime/issues/032-implement-deterministic-fake-shardengine.md

@@ -1,7 +1,7 @@
 <!-- GENERATED FROM prd.json — DO NOT EDIT AS AN INDEPENDENT SOURCE. prd.json IS AUTHORITATIVE. -->
 # DGR-032: Implement deterministic fake `ShardEngine`
 
-- **Status / triage:** specification only; `ready-for-agent`; `passes: false`
+- **Status / triage:** completed; `passes: true`
 - **Execution mode:** `AFK`
 - **Milestone:** `M1`
 - **Dependencies:** `DGR-031`
@@ -18,11 +18,11 @@ Fresh Ralph session: read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md`,
 
 ## Acceptance criteria
 
-- [ ] Support head, middle, tail, prefill, decode, cancellation, and release with deterministic outputs.
-- [ ] Model isolated session/epoch state and deterministic cache-miss/stale-epoch failures.
-- [ ] Support configurable delay, memory pressure, malformed output, and crash injection.
-- [ ] Contract tests distinguish fixture evidence from real-model certification.
-- [ ] Applicable shared quality gates in `prd.json` pass, and the evidence handoff records exact commands/results, changed files, limitations, and dependency handoff.
+- [x] Support head, middle, tail, prefill, decode, cancellation, and release with deterministic outputs.
+- [x] Model isolated session/epoch state and deterministic cache-miss/stale-epoch failures.
+- [x] Support configurable delay, memory pressure, malformed output, and crash injection.
+- [x] Contract tests distinguish fixture evidence from real-model certification.
+- [x] Applicable shared quality gates in `prd.json` pass, and the evidence handoff records exact commands/results, changed files, limitations, and dependency handoff.
 
 ## Shared quality gates
 
@@ -30,10 +30,7 @@ Fresh Ralph session: read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md`,
 - `git diff --check` passes.
 - Default tests are model-download-free, API-credit-free, and GPU-free.
 - Evidence README records exact changed files, commands/results, limitations, and dependency handoff; no fabricated evidence or inherited completion credit.
-- Native changes pass focused out-of-tree CMake build and CTest; patch changes verify clean apply/check/reverse against the exact llama.cpp pin.
-- Runs are opt-in and record exact artifact/split hashes, runtime/upstream pin, backend/driver, hardware, network, commands, and raw metrics. Model artifacts use configured mounted-drive storage and never `/home`.
-- Preserve existing Transformers behavior and backend-agnostic Tracker routing/load balancing/billing/relay semantics unless an explicit versioned contract says otherwise. One scoped story commit is expected during execution, but this specification-materialization change is not committed.
 
 ## Evidence handoff
 
-Write and verify `.scratch/distributed-gguf-runtime/evidence/DGR-032/README.md`. Until every criterion and applicable gate has real evidence, this story remains `passes: false`. Legacy evidence is provenance only, not completion credit.
+Verified evidence: `.scratch/distributed-gguf-runtime/evidence/DGR-032/README.md`. Legacy evidence remains provenance only and grants no implementation completion credit.

.scratch/distributed-gguf-runtime/prd.json

@@ -617,11 +617,12 @@
         "Contract tests distinguish fixture evidence from real-model certification.",
         "Applicable shared quality gates in `prd.json` pass, and the evidence handoff records exact commands/results, changed files, limitations, and dependency handoff."
       ],
-      "passes": false,
+      "passes": true,
       "notes": "Generated source issue: .scratch/distributed-gguf-runtime/issues/032-implement-deterministic-fake-shardengine.md; prd.json is authoritative.",
       "blocks": [
         "DGR-033"
-      ]
+      ],
+      "completionNotes": "Completed by agent"
     },
     {
       "id": "DGR-033",
@@ -2163,6 +2164,6 @@
     }
   ],
   "metadata": {
-    "updatedAt": "2026-07-23T08:00:32.939Z"
+    "updatedAt": "2026-07-23T08:09:16.081Z"
   }
 }