diff --git a/.scratch/distributed-gguf-runtime/evidence/DGR-036/BLOCKED.md b/.scratch/distributed-gguf-runtime/evidence/DGR-036/BLOCKED.md new file mode 100644 index 0000000..4d71fad --- /dev/null +++ b/.scratch/distributed-gguf-runtime/evidence/DGR-036/BLOCKED.md @@ -0,0 +1,12 @@ +# DGR-036 real-model lane blocker + +`DGR-036` cannot receive completion credit yet. The live standalone worker is DGR-033's `FakeShardEngine`, a CRC/echo fixture; DGR-037's real llama.cpp `ShardEngine` binding has not been implemented. Consequently no code path can execute a whole or ranged dense GGUF and no real prefill/logit or greedy-token parity result exists. + +The deterministic two-process fake-worker regression is implemented in `tests/test_native_shard_worker.py`, but this sandbox cannot open loopback sockets (`PermissionError: [Errno 1] Operation not permitted`), so that runtime test needs host-side execution as well. + +Unblock in this order: + +1. Complete DGR-037's real ranged llama.cpp worker binding without changing the DGR-035 dense boundary contract. +2. Provision a small exact dense GGUF on mounted-drive storage and record its artifact/split hashes plus runtime/backend/hardware/network identity. +3. Run whole-model and two-range prefill comparison, then record at least 32 greedy token IDs against the locked tolerance and retain raw metrics. +4. Run the deterministic two-process test on a host with loopback sockets, then update `README.md` and only then set `prd.json` completion truth. diff --git a/.scratch/distributed-gguf-runtime/evidence/DGR-036/README.md b/.scratch/distributed-gguf-runtime/evidence/DGR-036/README.md new file mode 100644 index 0000000..5f8d281 --- /dev/null +++ b/.scratch/distributed-gguf-runtime/evidence/DGR-036/README.md @@ -0,0 +1,63 @@ +# DGR-036 evidence — dense fixture and real-model range parity + +**Status:** incomplete; `prd.json` remains authoritative and keeps `DGR-036.passes` as `false`. + +## Deterministic fixture proof implemented + +`tests/test_native_shard_worker.py` now contains `test_two_disjoint_fake_worker_processes_preserve_prefill_and_decode_seam`. It starts two separate DGR-033 `shard_worker` OS processes, opens disjoint requested ranges `[0, 16)` and `[16, 32)`, forwards the first worker's actual protobuf output to the second, and checks one prefill plus 32 sequential decode positions. The test tops up the worker's 16-credit flow-control window before decode positions 16 and 32, so all 32 positions are exercised. + +This is deliberately **fixture evidence only**. The worker's `FakeShardEngine` validates a bundle and echoes its bytes; it has no dense graph, logits, sampler, or GGUF load. The assertions prove the two-process protocol/lifecycle seam and that bytes survive a disjoint-range handoff. They do not claim numerical model or greedy-token parity. + +## Real-model lane: blocked honestly + +DGR-037, which is still `passes: false`, is the story that binds llama.cpp to the standalone worker. The live DGR-033 worker remains the fake CRC/echo fixture, and no `ShardEngine` implementation can load/run a GGUF range. DGR-034 proves tensor ownership and memory reporting, while DGR-035 proves the Python boundary contract; neither supplies a real ranged execution engine. Therefore there is no truthful way to run a small dense GGUF whole-model versus two-range prefill comparison or to compare 32 greedy generated tokens yet. + +The real-model proof must be run after DGR-037 with an exact small dense GGUF, the pinned llama.cpp/runtime identity, two loaded worker ranges, and a raw report containing artifact and split hashes, backend/driver/hardware/network, prefill tolerance, all 32 token IDs, and raw metrics. It must remain opt-in, use mounted-drive artifact storage, and never download an artifact under `/home`. + +## Commands and results + +```bash +TESTPY=/home/popov/.hermes/hermes-agent/venv/bin/python +PYTHONPATH=packages/node:packages/tracker "$TESTPY" -m pytest -q tests/test_dense_range_boundary.py tests/test_architecture_boundary.py tests/test_shard_engine.py tests/test_fake_shard_engine.py +``` + +```text +37 passed in 0.18s +``` + +```bash +"$TESTPY" -m ruff check tests/test_native_shard_worker.py +PYTHONPATH=packages/node "$TESTPY" -m compileall -q packages tests +git diff --check +python3 scripts/ralph_prd_schema.py validate .scratch/distributed-gguf-runtime/prd.json +``` + +```text +All checks passed! +OK: 55 stories validated. +``` + +Attempted two-process fixture command: + +```bash +PYTHONPATH=packages/node:packages/tracker "$TESTPY" -m pytest -q tests/test_native_shard_worker.py -k two_disjoint_fake_worker_processes_preserve_prefill_and_decode_seam +``` + +```text +FAILED: PermissionError: [Errno 1] Operation not permitted at socket.socket(AF_INET, SOCK_STREAM) +``` + +This is the workspace sandbox's known localhost-socket restriction, before any worker is spawned; it is not a test assertion failure. Run that exact command on a host that permits loopback sockets after building `build/native/shard_worker`. + +## Changed files + +- `tests/test_native_shard_worker.py` +- `.scratch/distributed-gguf-runtime/evidence/DGR-036/README.md` +- `.scratch/distributed-gguf-runtime/evidence/DGR-036/BLOCKED.md` +- `.ralph-tui/progress.md` + +## Dependency handoff + +- DGR-033 supplies the process, lifecycle, generated gRPC surface, fake engine, and bounded-flow-control behaviour used by the deterministic test. +- DGR-035 supplies the strict dense residual boundary and tail-only output contract. DGR-037 must preserve that contract when it replaces the echo fake with a real engine. +- Once DGR-037 is complete, return here to run the opt-in numerical lane. Do not turn this fixture test into a claim that a real GGUF can execute ranges. diff --git a/tests/test_native_shard_worker.py b/tests/test_native_shard_worker.py index 6ce60db..18bb3e9 100644 --- a/tests/test_native_shard_worker.py +++ b/tests/test_native_shard_worker.py @@ -316,6 +316,72 @@ def test_decode_step_is_served(worker): assert echoed.chunk.bundle.tensors[0].fragments[0].payload == payload +def test_two_disjoint_fake_worker_processes_preserve_prefill_and_decode_seam(): + """DGR-036 fixture proof across two actual fake-worker processes. + + The DGR-033 worker deliberately has no model graph: its bounded forward + validates and echoes the activation bytes. That makes it suitable for a + deterministic protocol proof only. Keep the two ranges disjoint at the + ``SessionOpen`` boundary, pass each stage-one output through stage two, + and exercise a prefill plus 32 sequential decode positions. Numerical + dense-GGUF parity remains an opt-in DGR-036 real-model lane, not a claim + made by this fixture. + """ + head = _Worker() + tail = _Worker() + try: + head_open = _open( + route_session_id="dgr036-head", + shard_range=pb.ShardRange(start_layer=0, end_layer=16, effective_start_layer=0), + ) + tail_open = _open( + route_session_id="dgr036-tail", + shard_range=pb.ShardRange(start_layer=16, end_layer=32, effective_start_layer=16), + ) + payload = b"dgr036 deterministic dense prefill residual" + + def decode_requests(*, stage: str, payloads: list[bytes]) -> list[pb.SessionRequest]: + requests: list[pb.SessionRequest] = [] + for position, stage_payload in enumerate(payloads, start=1): + # The fixture worker's negotiated default window is 16. Top + # it up before decode 16 and 32 so this exercises all 32 + # sequential positions rather than silently testing only one + # credit window. + if position in {16, 32}: + requests.append(pb.SessionRequest(flow_control=pb.FlowControl(credits_granted=16))) + requests.append(_decode(f"decode-{stage}-{position}", stage_payload, position + 1, position)) + return requests + + head_responses = head.session( + [head_open, _chunk("prefill-head", payload, step=1)] + + decode_requests(stage="head", payloads=[payload] * 32) + ) + assert head_responses[0].WhichOneof("kind") == "accepted" + assert head_responses[1].WhichOneof("kind") == "chunk" + seam_payloads = [ + response.chunk.bundle.tensors[0].fragments[0].payload + for response in head_responses + if response.WhichOneof("kind") == "chunk" + ] + assert len(seam_payloads) == 33 + + tail_responses = tail.session( + [tail_open, _chunk("prefill-tail", seam_payloads[0], step=1, route_session_id="dgr036-tail")] + + decode_requests(stage="tail", payloads=seam_payloads[1:]) + ) + assert tail_responses[0].WhichOneof("kind") == "accepted" + assert tail_responses[1].WhichOneof("kind") == "chunk" + assert tail_responses[1].chunk.bundle.tensors[0].fragments[0].payload == payload + + for tail_response in (response for response in tail_responses if response.WhichOneof("kind") == "chunk"): + assert tail_response.chunk.bundle.tensors[0].fragments[0].payload == payload + finally: + if head.proc.poll() is None: + head.close() + if tail.proc.poll() is None: + tail.close() + + def test_release_is_terminal(worker): responses = worker.session([_open(), _release()]) assert responses[0].WhichOneof("kind") == "accepted"