feat: DGR-001 - Lock the safetensors-versus-GGUF performance contract

.scratch/distributed-gguf-runtime/evidence/DGR-001/BLOCKED.md

@@ -0,0 +1,30 @@
+# DGR-001 real-benchmark blocker
+
+Status: blocked only for the required real-model measurement. The deterministic
+harness, report schema, and immutable contract are implemented and tested; this
+file deliberately does not turn an unrun benchmark into a passing result.
+
+## Verified environment state (2026-07-13)
+
+- Mounted GGUF artifacts exist under `/run/media/popov/DATA/llm/`.
+- `llama-server` is not on `PATH`.
+- The available Python test environment has neither `torch` nor `transformers`.
+- No matching local safetensors snapshot was found for an installed GGUF recipe.
+
+Therefore this session cannot run the controlled same-model, same-revision,
+same-machine comparison without downloading/installing new runtime/model assets.
+That is intentionally not inferred from the story request.
+
+## Continuation
+
+1. Put a matching safetensors snapshot and near-lossless plus quantized GGUF
+   artifacts below one mounted-drive root, never `/home`.
+2. Install or build the pinned `llama-server`, and use `.venv-rocm` when testing
+   the Radeon backend.
+3. Compute each artifact SHA-256 and create a config declaring the same
+   `source_model_id` and `source_model_revision` for every recipe.
+4. Run the command in `commands.txt` with
+   `MESHNET_ENABLE_REAL_INFERENCE_TESTS=1`; save its JSON report and summary in
+   this directory, then evaluate it against `performance-contract.json`.
+5. Only after those results and all quality gates pass may DGR-001 be marked
+   done and DGR-004 consume the baseline.

.scratch/distributed-gguf-runtime/evidence/DGR-001/README.md

@@ -0,0 +1,62 @@
+# DGR-001 — Safetensors versus GGUF performance contract
+
+Status: **blocked for real evidence; deterministic implementation complete.**
+No model benchmark is claimed. See `BLOCKED.md` and the explicitly `not-run`
+`results.json`.
+
+## What is implemented
+
+- `recipe_benchmark.py` is a deterministic measurement core that runs the exact
+  same plan for every recipe and reports TTFT, prefill/decode rates, p50/p95
+  latency, aggregate throughput, RSS, VRAM, artifact bytes, request failures,
+  and per-prompt output drift in JSON.
+- `recipe_drivers.py` supplies opt-in Transformers/safetensors and whole-model
+  llama.cpp-server drivers. Real execution requires
+  `MESHNET_ENABLE_REAL_INFERENCE_TESTS=1`, refuses model paths outside the
+  declared mounted-drive root, requires a SHA-256 per artifact, records host
+  facts, and requires the same declared source model and revision for every
+  recipe.
+- `performance_contract.py` separates a near-lossless quality lane from the
+  quantized performance/fit lane. Quantized drift is advisory; only the quality
+  lane can establish parity. `performance-contract.json` locks v1 thresholds
+  and the stop condition before any result exists.
+
+## Files changed
+
+- `packages/node/meshnet_node/recipe_benchmark.py`
+- `packages/node/meshnet_node/recipe_drivers.py`
+- `packages/node/meshnet_node/performance_contract.py`
+- `tests/test_recipe_benchmark.py`
+- This evidence directory.
+
+## Commands and results
+
+`commands.txt` contains exact commands. Final targeted result:
+
+```text
+pytest -q tests/test_recipe_benchmark.py  -> 15 passed
+python -m compileall -q packages tests    -> exit 0
+git diff --check                          -> exit 0
+```
+
+The full suite was attempted and is blocked during collection by the unrelated,
+pre-existing DGR-002 runtime dependency mismatch:
+
+```text
+google.protobuf.runtime_version.VersionError:
+gencode 7.35.0 runtime 6.33.6
+```
+
+This was reproduced from a clean `git archive HEAD` extracted to
+`/tmp/dgr-001-clean`, with the same command and same failure before any
+uncommitted DGR-001 changes were present. No real benchmark command was run
+because the prerequisites in `BLOCKED.md` are absent.
+
+## Compatibility and handoff
+
+This is additive: it does not alter the current Transformers route, Tracker,
+relay, or native protocol. DGR-014 must load `performance-contract.json`, run
+the same controlled plan at concurrency 1 and 4, and make only its
+promote/optimize/stop recommendation from a `local-real` or
+`multi-machine-real` report. DGR-004 remains blocked on this story's real
+baseline decision.

.scratch/distributed-gguf-runtime/evidence/DGR-001/commands.txt

@@ -0,0 +1,11 @@
+# Deterministic implementation checks completed in this worktree
+PYTHONPATH=packages/node /home/popov/.hermes/hermes-agent/venv/bin/python -m pytest -q tests/test_recipe_benchmark.py
+PYTHONPATH=packages/node /home/popov/.hermes/hermes-agent/venv/bin/python -m compileall -q packages tests
+git diff --check
+
+# Full suite attempted in this worktree and a clean HEAD archive; both stop at
+# protobuf gencode 7.35.0 versus installed runtime 6.33.6 during collection.
+PYTHONPATH=packages/node /home/popov/.hermes/hermes-agent/venv/bin/python -m pytest -q
+
+# Required opt-in real benchmark after the BLOCKED.md prerequisites exist
+MESHNET_ENABLE_REAL_INFERENCE_TESTS=1 PYTHONPATH=packages/node python -m meshnet_node.recipe_benchmark --config /run/media/popov/DATA/meshnet/dgr-001-benchmark.json --json-out .scratch/distributed-gguf-runtime/evidence/DGR-001/results.json --summary-out .scratch/distributed-gguf-runtime/evidence/DGR-001/results.txt

.scratch/distributed-gguf-runtime/evidence/DGR-001/performance-contract.json

@@ -0,0 +1,36 @@
+{
+  "schema_version": 1,
+  "contract_version": 1,
+  "locked_at": "2026-07-13T00:00:00Z",
+  "locked_by": "DGR-001",
+  "plan_id": "dgr-001-controlled-whole-model-baseline-v1",
+  "thresholds": {
+    "min_decode_speedup": 1.25,
+    "max_ttft_ratio": 1.25,
+    "min_aggregate_throughput_speedup": 1.25,
+    "max_resident_memory_ratio": 0.75,
+    "max_artifact_size_ratio": 0.6,
+    "min_quality_exact_match_rate": 0.9,
+    "min_quality_mean_similarity": 0.97,
+    "max_failure_rate": 0.0
+  },
+  "baseline": {
+    "status": "pending-real-evidence",
+    "required_evidence_class": "local-real",
+    "required_recipes": [
+      "transformers-safetensors-reference",
+      "llama-cpp-near-lossless-quality",
+      "llama-cpp-quantized-performance-fit"
+    ],
+    "required_concurrency_levels": [1, 4],
+    "required_controlled_variables": [
+      "model architecture",
+      "model revision",
+      "machine and device",
+      "formatted prompts and context lengths",
+      "output length and greedy sampling policy"
+    ]
+  },
+  "stop_condition": "Stop the native llama.cpp/GGUF track when, on the same machine and device as the Transformers/safetensors reference and under this plan, no performance-fit GGUF recipe delivers either a meaningful speed benefit (at least 25% higher single-request decode tokens/sec without more than 25% worse TTFT, or at least 25% higher aggregate throughput under concurrency) or a meaningful fit benefit (at least 25% lower peak resident memory), or when the near-lossless quality lane fails.",
+  "notes": "Quantized performance-fit output drift is reported as advisory only. It is not numerical-equivalence evidence. DGR-014 consumes this immutable v1 contract."
+}

.scratch/distributed-gguf-runtime/evidence/DGR-001/results.json

@@ -0,0 +1,8 @@
+{
+  "schema_version": 1,
+  "evidence_class": "not-run",
+  "status": "blocked",
+  "measured_at": null,
+  "reason": "No matching local Transformers/safetensors snapshot and whole-model llama-server runtime were available in this execution environment. No performance, memory, latency, failure, or drift values were fabricated.",
+  "required_output": "A local-real recipe benchmark report emitted by python -m meshnet_node.recipe_benchmark with MESHNET_ENABLE_REAL_INFERENCE_TESTS=1."
+}