neuron-tai/.scratch/distributed-gguf-runtime/evidence/DGR-032/README.md

# DGR-032 evidence — deterministic fake `ShardEngine`

**Completed:** 2026-07-23
**Branch:** `ralph/distributed-gguf-runtime`
**Authority:** `.scratch/distributed-gguf-runtime/prd.json`
**Dependencies:** DGR-031 (`evidence/DGR-031/README.md` — the project-owned
`ShardEngine` abstract contract, `tests/shard_engine_contract.py`'s
`assert_shard_engine_contract`, and its own dependency-handoff note that
DGR-032 should "subclass `ShardEngine`, add delay/memory-pressure/malformed-
output/crash injection, and pass the *same* `assert_shard_engine_contract`
... — no new contract vocabulary should be needed").

## Objective

Provide an engine fixture that deterministically transforms typed boundary
bundles and session state: head/middle/tail, prefill/decode, cancellation,
release, isolated per-session epoch state, deterministic cache-miss/stale-
epoch failures, and configurable delay/memory-pressure/malformed-output/
crash-injection fault surfaces — all without llama.cpp, a GPU, or any I/O.

## What was found live before changing code

- `packages/node/meshnet_node/shard_engine.py` (DGR-031): the abstract
  `ShardEngine` with eight operations (`load`, `capabilities`, `prefill`,
  `decode`, `cancel`, `release`, `health`, `metrics`) and its project-owned
  dataclasses (`LoadRequest`, `EngineCapabilities`, `PrefillRequest`/
  `DecodeRequest`, `StepResult`, `BoundaryBundle`/`EngineTensor`,
  `TokenOutput`, `HealthResult`, `MetricsResult`).
- `tests/shard_engine_contract.py` (DGR-031): the reusable
  `assert_shard_engine_contract(make_engine)` helper — nine lifecycle checks
  any implementation must pass, explicitly designed to be imported by
  DGR-032 and DGR-037 against their own engines.
- `tests/test_shard_engine.py` (DGR-031): its `_ReferenceEngine` is
  explicitly documented as *not* the DGR-032 fixture ("no delay/memory-
  pressure/malformed/crash injection... that is a separate, larger story") —
  confirming this story starts from nothing, not inherited credit.
- `grep -rn -i "fakeshardengine\|fake_shard_engine"` across `.py`/`.md`
  returned no prior matches — no fake engine existed before this story.
- No file in `packages/node/meshnet_node/` wires a `ShardEngine` into
  `shard_runtime_server.py` yet (confirmed by grep for `ShardEngine`/
  `shard_engine` in that file — no matches); that wiring is DGR-037's scope,
  so this fixture is a standalone, importable engine only.

Live verification of the pre-existing dependency contract before adding new
code:
```bash
PYTHONPATH=packages/node:packages/tracker .venv/bin/python3 -m pytest -q tests/test_shard_engine.py
```
```text
12 passed in 0.13s
```

## What was added (this story's change)

### `packages/node/meshnet_node/fake_shard_engine.py` (new)

`FakeShardEngine(ShardEngine)` — a pure-Python, deterministic fixture:

- **Determinism.** Every `prefill`/`decode` output is `SHA-256(seed_bytes +
  idempotency_step)`, where `seed_bytes` is derived from `token_ids` (head)
  or the input `BoundaryBundle`'s tensor bytes plus any `token_id_sideband`
  (middle/tail-in). Replaying identical inputs on a brand-new session
  produces byte-identical output — proven by
  `assert_shard_engine_contract`'s own determinism check and reused directly.
- **Head/middle/tail.** Tail shards (`shard_end >= total_layers - 1`) return
  a `TokenOutput` sampled into `[0, TOKEN_ID_VOCAB_SIZE)`; head/middle shards
  return a `BoundaryBundle` tagged `boundary_point="post_head_residual"` or
  `"post_middle_residual"` respectively, so the three cases are
  distinguishable in fixture output, not just in the load request. A middle
  shard's `token_id_sideband` passes through unchanged from its input bundle
  to its output bundle (the V4 first-three-hash-routed-layers requirement
  RALPH-CONTEXT documents), never invented or dropped.
- **Isolated session/epoch state.** `_sessions: dict[str, _SessionState]`
  keyed by `session_id`; each session tracks its own `epoch`/`cancelled`
  flag. A stale epoch, cancel, or release on one session never touches
  another's state (`test_session_state_is_isolated_between_two_concurrent_sessions`
  proves a stale-epoch rejection and a cancel on session `"a"` leave session
  `"b"` fully serviceable). Decoding an unopened session is a deterministic
  `NOT_FOUND`/`CacheResult.MISS`, not an exception.
- **Configurable delay.** `FakeShardEngineConfig.step_delay_seconds` +
  injectable `sleep` hook (defaults to `time.sleep`, overridable in tests so
  they don't block wall-clock time) — invoked once per `prefill`/`decode`
  call before computing the deterministic output.
- **Configurable memory pressure.** `FakeShardEngineConfig.memory_budget_bytes`
  — the engine accumulates `_bytes_used` across every step's seed bytes;
  once a step would push cumulative usage past the budget, that step
  deterministically returns `StatusCode.RESOURCE_EXHAUSTED` (`retryable=True`)
  with no output, instead of computing one.
- **Configurable malformed output.** `FakeShardEngineConfig.malformed_output`
  — when set, the engine still reports `StatusCode.OK` (the point is a
  buggy-but-"successful"-looking response, not a status-coded failure) but
  the payload is structurally valid, semantically wrong: a tail `TokenOutput`
  is pushed past `MALFORMED_TOKEN_ID_FLOOR` (outside the fixture's own
  advertised vocab), and a head/middle `BoundaryBundle` gets an
  `architecture` field prefixed `"malformed:"` and its tensor `data`
  truncated to one byte — both structurally valid per `EngineTensor`'s and
  `BoundaryBundle`'s own `__post_init__` validation (which does not
  cross-check `data` length against `shape`/`dtype`), so a consumer must
  actually check shape/semantics, not just status codes, to catch it.
- **Configurable crash injection.** `FakeShardEngineConfig.crash_after_calls`
  + `crash_exception_factory` — after the configured number of
  `prefill`/`decode` calls, the engine raises an arbitrary exception (default
  `RuntimeError`, injectable) directly out of the call instead of returning a
  `StepResult`. This is deliberately *not* wrapped in `EngineError`/
  `StructuredStatus`: it simulates a whole-process failure (what a worker
  supervisor — DGR-040 — must catch and restart around), which is a
  different failure mode from a graceful status-coded rejection.
- **Fixture-vs-real marker.** `FakeShardEngine.EVIDENCE_CLASS = "fixture"` —
  a structural constant (not just docstring prose) so DGR-036's fixture-vs-
  real-model parity check can assert programmatically that it is comparing a
  fixture engine against a real one, never two fixtures.
- Every fault-injection knob defaults to off (`0`/`None`/`False`), so a bare
  `FakeShardEngine()` passes `assert_shard_engine_contract` unmodified —
  fault injection is opt-in, never a baseline behavior change.

### `tests/test_fake_shard_engine.py` (new)

- `test_fake_shard_engine_obeys_the_shared_shard_engine_contract` — runs the
  full DGR-031 contract against a bare `FakeShardEngine`.
- `test_fake_shard_engine_declares_fixture_evidence_class` — pins the
  `EVIDENCE_CLASS` marker DGR-036 will rely on.
- Head/middle/tail output-shape tests (`boundary_point`, token-id-sideband
  pass-through, tail vocab range).
- `test_session_state_is_isolated_between_two_concurrent_sessions` — a
  stale-epoch rejection and a cancel on one session leave a second,
  concurrently open session fully serviceable.
- One test per fault-injection knob (delay hook invocation, memory-budget
  trip, malformed tail/boundary-bundle output, crash-after-N-calls,
  configurable crash exception type) plus `FakeShardEngineConfig`'s own
  `__post_init__` validation (negative delay, negative budget, non-positive
  `crash_after_calls`).
- `test_load_result_and_capabilities_report_recipe_architecture` — the
  fixture threads `LoadRequest.recipe["architecture"]` through to both
  `LoadResult.architecture` and `EngineCapabilities.architecture` rather than
  hardcoding `"dense"`/`"fake"` everywhere, so a future V4 recipe is visible
  in fixture output too.

### `.scratch/distributed-gguf-runtime/prd.json` / issue markdown

Marked `DGR-032.passes = true` with `completionNotes`; regenerated
`issues/032-implement-deterministic-fake-shardengine.md` via
`scripts/ralph_prd_schema.py render` so it matches `prd.json` byte-for-byte.

## Acceptance criteria → evidence

1. **Head, middle, tail, prefill, decode, cancellation, release with
   deterministic outputs** — `FakeShardEngine`'s `_transform`, boundary-point
   tagging, and `assert_shard_engine_contract`'s own determinism/cancel/
   release checks. Verified by
   `test_fake_shard_engine_obeys_the_shared_shard_engine_contract`,
   `test_head_shard_returns_boundary_bundle_with_post_head_residual_point`,
   `test_middle_shard_returns_boundary_bundle_and_passes_through_token_sideband`,
   `test_tail_shard_returns_token_output_within_advertised_vocab`.
2. **Isolated session/epoch state and deterministic cache-miss/stale-epoch
   failures** — `_sessions` dict keyed per session;
   `test_session_state_is_isolated_between_two_concurrent_sessions` plus the
   shared contract's own cache-miss/stale-epoch checks.
3. **Configurable delay, memory pressure, malformed output, crash
   injection** — `FakeShardEngineConfig`; verified by
   `test_step_delay_seconds_invokes_the_configured_sleep_hook`,
   `test_memory_budget_bytes_trips_deterministic_resource_exhausted`,
   `test_malformed_output_is_structurally_valid_but_semantically_wrong_for_tail`,
   `test_malformed_output_is_structurally_valid_but_semantically_wrong_for_boundary_bundle`,
   `test_crash_after_calls_raises_instead_of_returning_a_structured_status`,
   `test_crash_exception_factory_is_configurable`,
   `test_config_rejects_invalid_knob_values`.
4. **Contract tests distinguish fixture evidence from real-model
   certification** — module docstring and this README are explicit that
   this is FIXTURE evidence only (numeric parity is DGR-036 onward); the
   `EVIDENCE_CLASS = "fixture"` constant makes that distinction structurally
   checkable, not just prose, pinned by
   `test_fake_shard_engine_declares_fixture_evidence_class`.
5. **Gates + this handoff** — below.

## Commands and results

```bash
PYTHONPATH=packages/node:packages/tracker .venv/bin/python3 -m pytest -q tests/test_fake_shard_engine.py tests/test_shard_engine.py
```
```text
26 passed in 0.17s
```

```bash
PYTHONPATH=packages/node:packages/tracker .venv/bin/python3 -m pytest -q \
  tests/test_fake_shard_engine.py tests/test_shard_engine.py tests/test_shard_lifecycle.py \
  tests/test_architecture_boundary.py tests/test_activation_envelope.py \
  tests/test_native_shard_protocol.py tests/test_shard_runtime_harness.py
```
```text
109 passed, 3 skipped in 3.78s
```

```bash
.venv/bin/python3 -m compileall -q packages tests
```
```text
(no output — clean; exit 0)
```

```bash
git diff --check
```
```text
(no output — clean)
```

## Limitations

- `tests/` as a whole does not collect cleanly in this environment: the same
  pre-existing collection errors DGR-031's evidence recorded (missing
  optional dependencies such as `cryptography`) are still present and are
  unrelated to this story. This story's own gates were run as the targeted,
  scoped test set above per the shared quality gates' wording ("Targeted
  deterministic tests pass").
- This is FIXTURE evidence only. `FakeShardEngine` proves lifecycle,
  session/epoch isolation, and fault-injection semantics; it proves nothing
  about numerical parity with a real model. That is DGR-036's explicit job
  once DGR-037's real engine exists, and DGR-053/054 for V4 alpha
  certification.
- `FakeShardEngine` is not wired into `shard_runtime_server.py` or any gRPC
  surface — it is a standalone, importable engine only. Wiring a
  `ShardEngine` (fake or real) into the gRPC servicer is DGR-037's scope for
  the real engine; DGR-033 covers a C++ worker surface, which is a separate
  native executable, not a consumer of this Python module.
- No gRPC/CMake/native-build changes were needed or made — this story is
  pure Python fixture code (`evidenceClass: fixture`, `hardware: none`), so
  the native CMake/CTest and patch-stack gates in the shared quality-gate
  list do not apply here, consistent with DGR-031's own README recording the
  same non-applicability.

## Dependency handoff

- **DGR-033** (standalone fake C++ gRPC Shard worker): its own issue
  describes a native C++ executable serving the lifecycle/stream RPC
  contract "using the fake engine" — that is a native analogue, not a
  consumer of this Python module; DGR-033 should still read this README for
  the exact deterministic-output/session-isolation/fault-injection semantics
  its C++ fake engine needs to reproduce so both fakes behave identically
  from a client's point of view.
- **DGR-034/DGR-035** (range-aware GGUF ownership, boundary I/O):
  `FakeShardEngine` already demonstrates range-driven head/middle/tail
  behavior purely from `LoadRequest.shard_start`/`shard_end`/`total_layers`;
  no new range vocabulary was needed.
- **DGR-036** (fixture vs real-model parity): compare a `FakeShardEngine`
  instance's `EVIDENCE_CLASS` (`"fixture"`) against DGR-037's real engine's
  equivalent marker (expected `"real"`) to assert the parity check is
  actually comparing two different implementations; reuse
  `assert_shard_engine_contract` against both to prove lifecycle parity
  before attempting numeric parity.
- **DGR-037** (bind llama.cpp to the worker): `FakeShardEngine` is the
  reference implementation to diff a real engine's lifecycle behavior
  against — same request/result types, same session/epoch model, no new
  contract vocabulary.
- **DGR-040** (worker supervision): the crash-injection knob
  (`crash_after_calls`/`crash_exception_factory`) exists specifically so
  supervision/restart logic has a deterministic way to trigger and test an
  unhandled engine failure distinct from a graceful `StructuredStatus`
  rejection.