neuron-tai/.scratch/distributed-gguf-runtime/evidence/DGR-032/README.md

DGR-032 evidence — deterministic fake ShardEngine

Completed: 2026-07-23 Branch: ralph/distributed-gguf-runtime Authority: .scratch/distributed-gguf-runtime/prd.json Dependencies: DGR-031 (evidence/DGR-031/README.md — the project-owned ShardEngine abstract contract, tests/shard_engine_contract.py's assert_shard_engine_contract, and its own dependency-handoff note that DGR-032 should "subclass ShardEngine, add delay/memory-pressure/malformed- output/crash injection, and pass the same assert_shard_engine_contract ... — no new contract vocabulary should be needed").

Objective

Provide an engine fixture that deterministically transforms typed boundary bundles and session state: head/middle/tail, prefill/decode, cancellation, release, isolated per-session epoch state, deterministic cache-miss/stale- epoch failures, and configurable delay/memory-pressure/malformed-output/ crash-injection fault surfaces — all without llama.cpp, a GPU, or any I/O.

What was found live before changing code

• packages/node/meshnet_node/shard_engine.py (DGR-031): the abstract ShardEngine with eight operations (load, capabilities, prefill, decode, cancel, release, health, metrics) and its project-owned dataclasses (LoadRequest, EngineCapabilities, PrefillRequest/ DecodeRequest, StepResult, BoundaryBundle/EngineTensor, TokenOutput, HealthResult, MetricsResult).
• tests/shard_engine_contract.py (DGR-031): the reusable assert_shard_engine_contract(make_engine) helper — nine lifecycle checks any implementation must pass, explicitly designed to be imported by DGR-032 and DGR-037 against their own engines.
• tests/test_shard_engine.py (DGR-031): its _ReferenceEngine is explicitly documented as not the DGR-032 fixture ("no delay/memory- pressure/malformed/crash injection... that is a separate, larger story") — confirming this story starts from nothing, not inherited credit.
• grep -rn -i "fakeshardengine\|fake_shard_engine" across .py/.md returned no prior matches — no fake engine existed before this story.
• No file in packages/node/meshnet_node/ wires a ShardEngine into shard_runtime_server.py yet (confirmed by grep for ShardEngine/ shard_engine in that file — no matches); that wiring is DGR-037's scope, so this fixture is a standalone, importable engine only.

Live verification of the pre-existing dependency contract before adding new code:

PYTHONPATH=packages/node:packages/tracker .venv/bin/python3 -m pytest -q tests/test_shard_engine.py

12 passed in 0.13s

What was added (this story's change)

packages/node/meshnet_node/fake_shard_engine.py (new)

FakeShardEngine(ShardEngine) — a pure-Python, deterministic fixture:

• Determinism. Every prefill/decode output is SHA-256(seed_bytes + idempotency_step), where seed_bytes is derived from token_ids (head) or the input BoundaryBundle's tensor bytes plus any token_id_sideband (middle/tail-in). Replaying identical inputs on a brand-new session produces byte-identical output — proven by assert_shard_engine_contract's own determinism check and reused directly.
• Head/middle/tail. Tail shards (shard_end >= total_layers - 1) return a TokenOutput sampled into [0, TOKEN_ID_VOCAB_SIZE); head/middle shards return a BoundaryBundle tagged boundary_point="post_head_residual" or "post_middle_residual" respectively, so the three cases are distinguishable in fixture output, not just in the load request. A middle shard's token_id_sideband passes through unchanged from its input bundle to its output bundle (the V4 first-three-hash-routed-layers requirement RALPH-CONTEXT documents), never invented or dropped.
• Isolated session/epoch state. _sessions: dict[str, _SessionState] keyed by session_id; each session tracks its own epoch/cancelled flag. A stale epoch, cancel, or release on one session never touches another's state (test_session_state_is_isolated_between_two_concurrent_sessions proves a stale-epoch rejection and a cancel on session "a" leave session "b" fully serviceable). Decoding an unopened session is a deterministic NOT_FOUND/CacheResult.MISS, not an exception.
• Configurable delay. FakeShardEngineConfig.step_delay_seconds + injectable sleep hook (defaults to time.sleep, overridable in tests so they don't block wall-clock time) — invoked once per prefill/decode call before computing the deterministic output.
• Configurable memory pressure. FakeShardEngineConfig.memory_budget_bytes — the engine accumulates _bytes_used across every step's seed bytes; once a step would push cumulative usage past the budget, that step deterministically returns StatusCode.RESOURCE_EXHAUSTED (retryable=True) with no output, instead of computing one.
• Configurable malformed output. FakeShardEngineConfig.malformed_output — when set, the engine still reports StatusCode.OK (the point is a buggy-but-"successful"-looking response, not a status-coded failure) but the payload is structurally valid, semantically wrong: a tail TokenOutput is pushed past MALFORMED_TOKEN_ID_FLOOR (outside the fixture's own advertised vocab), and a head/middle BoundaryBundle gets an architecture field prefixed "malformed:" and its tensor data truncated to one byte — both structurally valid per EngineTensor's and BoundaryBundle's own __post_init__ validation (which does not cross-check data length against shape/dtype), so a consumer must actually check shape/semantics, not just status codes, to catch it.
• Configurable crash injection. FakeShardEngineConfig.crash_after_calls
  • crash_exception_factory — after the configured number of prefill/decode calls, the engine raises an arbitrary exception (default RuntimeError, injectable) directly out of the call instead of returning a StepResult. This is deliberately not wrapped in EngineError/ StructuredStatus: it simulates a whole-process failure (what a worker supervisor — DGR-040 — must catch and restart around), which is a different failure mode from a graceful status-coded rejection.
• Fixture-vs-real marker. FakeShardEngine.EVIDENCE_CLASS = "fixture" — a structural constant (not just docstring prose) so DGR-036's fixture-vs- real-model parity check can assert programmatically that it is comparing a fixture engine against a real one, never two fixtures.
• Every fault-injection knob defaults to off (0/None/False), so a bare FakeShardEngine() passes assert_shard_engine_contract unmodified — fault injection is opt-in, never a baseline behavior change.

tests/test_fake_shard_engine.py (new)

• test_fake_shard_engine_obeys_the_shared_shard_engine_contract — runs the full DGR-031 contract against a bare FakeShardEngine.
• test_fake_shard_engine_declares_fixture_evidence_class — pins the EVIDENCE_CLASS marker DGR-036 will rely on.
• Head/middle/tail output-shape tests (boundary_point, token-id-sideband pass-through, tail vocab range).
• test_session_state_is_isolated_between_two_concurrent_sessions — a stale-epoch rejection and a cancel on one session leave a second, concurrently open session fully serviceable.
• One test per fault-injection knob (delay hook invocation, memory-budget trip, malformed tail/boundary-bundle output, crash-after-N-calls, configurable crash exception type) plus FakeShardEngineConfig's own __post_init__ validation (negative delay, negative budget, non-positive crash_after_calls).
• test_load_result_and_capabilities_report_recipe_architecture — the fixture threads LoadRequest.recipe["architecture"] through to both LoadResult.architecture and EngineCapabilities.architecture rather than hardcoding "dense"/"fake" everywhere, so a future V4 recipe is visible in fixture output too.

.scratch/distributed-gguf-runtime/prd.json / issue markdown

Marked DGR-032.passes = true with completionNotes; regenerated issues/032-implement-deterministic-fake-shardengine.md via scripts/ralph_prd_schema.py render so it matches prd.json byte-for-byte.

Acceptance criteria → evidence

1. Head, middle, tail, prefill, decode, cancellation, release with deterministic outputs — FakeShardEngine's _transform, boundary-point tagging, and assert_shard_engine_contract's own determinism/cancel/ release checks. Verified by test_fake_shard_engine_obeys_the_shared_shard_engine_contract, test_head_shard_returns_boundary_bundle_with_post_head_residual_point, test_middle_shard_returns_boundary_bundle_and_passes_through_token_sideband, test_tail_shard_returns_token_output_within_advertised_vocab.
2. Isolated session/epoch state and deterministic cache-miss/stale-epoch failures — _sessions dict keyed per session; test_session_state_is_isolated_between_two_concurrent_sessions plus the shared contract's own cache-miss/stale-epoch checks.
3. Configurable delay, memory pressure, malformed output, crash injection — FakeShardEngineConfig; verified by test_step_delay_seconds_invokes_the_configured_sleep_hook, test_memory_budget_bytes_trips_deterministic_resource_exhausted, test_malformed_output_is_structurally_valid_but_semantically_wrong_for_tail, test_malformed_output_is_structurally_valid_but_semantically_wrong_for_boundary_bundle, test_crash_after_calls_raises_instead_of_returning_a_structured_status, test_crash_exception_factory_is_configurable, test_config_rejects_invalid_knob_values.
4. Contract tests distinguish fixture evidence from real-model certification — module docstring and this README are explicit that this is FIXTURE evidence only (numeric parity is DGR-036 onward); the EVIDENCE_CLASS = "fixture" constant makes that distinction structurally checkable, not just prose, pinned by test_fake_shard_engine_declares_fixture_evidence_class.
5. Gates + this handoff — below.

Commands and results

PYTHONPATH=packages/node:packages/tracker .venv/bin/python3 -m pytest -q tests/test_fake_shard_engine.py tests/test_shard_engine.py

26 passed in 0.17s

PYTHONPATH=packages/node:packages/tracker .venv/bin/python3 -m pytest -q \
  tests/test_fake_shard_engine.py tests/test_shard_engine.py tests/test_shard_lifecycle.py \
  tests/test_architecture_boundary.py tests/test_activation_envelope.py \
  tests/test_native_shard_protocol.py tests/test_shard_runtime_harness.py

109 passed, 3 skipped in 3.78s

.venv/bin/python3 -m compileall -q packages tests

(no output — clean; exit 0)

git diff --check

(no output — clean)

Limitations

• tests/ as a whole does not collect cleanly in this environment: the same pre-existing collection errors DGR-031's evidence recorded (missing optional dependencies such as cryptography) are still present and are unrelated to this story. This story's own gates were run as the targeted, scoped test set above per the shared quality gates' wording ("Targeted deterministic tests pass").
• This is FIXTURE evidence only. FakeShardEngine proves lifecycle, session/epoch isolation, and fault-injection semantics; it proves nothing about numerical parity with a real model. That is DGR-036's explicit job once DGR-037's real engine exists, and DGR-053/054 for V4 alpha certification.
• FakeShardEngine is not wired into shard_runtime_server.py or any gRPC surface — it is a standalone, importable engine only. Wiring a ShardEngine (fake or real) into the gRPC servicer is DGR-037's scope for the real engine; DGR-033 covers a C++ worker surface, which is a separate native executable, not a consumer of this Python module.
• No gRPC/CMake/native-build changes were needed or made — this story is pure Python fixture code (evidenceClass: fixture, hardware: none), so the native CMake/CTest and patch-stack gates in the shared quality-gate list do not apply here, consistent with DGR-031's own README recording the same non-applicability.

Dependency handoff

• DGR-033 (standalone fake C++ gRPC Shard worker): its own issue describes a native C++ executable serving the lifecycle/stream RPC contract "using the fake engine" — that is a native analogue, not a consumer of this Python module; DGR-033 should still read this README for the exact deterministic-output/session-isolation/fault-injection semantics its C++ fake engine needs to reproduce so both fakes behave identically from a client's point of view.
• DGR-034/DGR-035 (range-aware GGUF ownership, boundary I/O): FakeShardEngine already demonstrates range-driven head/middle/tail behavior purely from LoadRequest.shard_start/shard_end/total_layers; no new range vocabulary was needed.
• DGR-036 (fixture vs real-model parity): compare a FakeShardEngine instance's EVIDENCE_CLASS ("fixture") against DGR-037's real engine's equivalent marker (expected "real") to assert the parity check is actually comparing two different implementations; reuse assert_shard_engine_contract against both to prove lifecycle parity before attempting numeric parity.
• DGR-037 (bind llama.cpp to the worker): FakeShardEngine is the reference implementation to diff a real engine's lifecycle behavior against — same request/result types, same session/epoch model, no new contract vocabulary.
• DGR-040 (worker supervision): the crash-injection knob (crash_after_calls/crash_exception_factory) exists specifically so supervision/restart logic has a deterministic way to trigger and test an unhandled engine failure distinct from a graceful StructuredStatus rejection.