# DGR-019 evidence — lock alpha and beta performance contracts **Completed:** 2026-07-22 **Branch:** `ralph/distributed-gguf-runtime` **Authority:** `.scratch/distributed-gguf-runtime/prd.json` **Dependency:** DGR-017 (`evidence/DGR-017/README.md`) — cleaned backlog reconciled to `origin/master`; no old pass state transferred. ## Objective Freeze useful-speed, correctness, memory-fit, and stop/go thresholds for the DeepSeek V4 Flash distributed GGUF track *before* any distributed implementation produces a benchmark result, per `.scratch/distributed-gguf-runtime/issues/019-lock-alpha-and-beta-performance-contracts.md`. ## Pre-existing state found (not caused by this story) Before any change in this session, `git status` showed `.scratch/distributed-gguf-runtime/prd.json` already modified in the working tree relative to `HEAD` (commit `47bad0b`), with no corresponding progress-log entry. Diffing against `HEAD` showed the working copy had **dropped** prd.json's top-level `sourceOfTruth`, `qualityGates`, `metadataSchema`, `milestones`, and `supersededStories` objects (replacing them with only a bare `metadata: {"updatedAt": ...}` stamp), while `userStories` itself was byte-identical to `HEAD`. Running `tests/test_ralph_prd_schema.py` against the as-found working tree confirmed the damage: 56 of 108 tests failed (every `test_render_issue_markdown_matches_committed_file[...]` parametrization, since `quality_gate_bullets`/`authority_disclaimer` fall back to module defaults once `qualityGates`/ `metadataSchema` are absent, which no longer match the committed issue files). This is the same shape of problem DGR-018's evidence documented and fixed: an abandoned, unexplained edit that silently dropped the schema/gates/milestone/provenance content this and future stories depend on, while `scripts/ralph_prd_schema.py validate` did not catch it (those top-level sections are optional-if-absent by design, so the CLI reported `OK: 55 stories validated.` even with them missing). The most likely cause is `ralph-tui`'s own read/write of `prd.json` as its task source, which only round-trips the fields it models (`name`/`description`/`branchName`/`userStories`) and stamps its own `metadata.updatedAt`, dropping any project-specific extension fields it doesn't know about. Per `RALPH-CONTEXT.md`'s instruction to inspect `git status` and preserve unrelated work rather than build on top of unexplained state, and following the DGR-018 precedent, the dropped fields were restored verbatim from `HEAD` (`git show HEAD:.scratch/distributed-gguf-runtime/prd.json`) while keeping the current `userStories` content (identical) and the current `metadata.updatedAt` stamp. `tests/test_ralph_prd_schema.py` returned to `108 passed` immediately after the restore, before any DGR-019-specific change was made. ## Changes ### `packages/node/meshnet_node/dgr_performance/` (new package) - **`data/alpha-beta-contract-v1.json`** — the locked, versioned, machine-readable contract. `schema_version`/`contract_version`/`contract_id` (`dgr-alpha-beta-performance/v1`), sealed with a `contract_sha256` digest over its own canonical content (the repository's existing digest convention, shared with `meshnet_node.glm_alpha.contract`). Contents: - `prompt_set` — four fixed prompts (`short-instruction`, `code-completion`, `multi-step-reasoning`, `long-context-fill`) referenced by ID from every lane, so no lane can quietly drift onto a different workload. - `sampling` — greedy (`temperature=0`, `top_p=1`, `top_k=1`, `seed=1234`), matching `meshnet_node.recipe_benchmark.SamplingPolicy` defaults. - `lanes` — all four lanes named in the acceptance criteria. `controlled-safetensors` and `whole-model-gguf` are marked `locked_elsewhere: true` and point at the pre-existing immutable DGR-001 lock (`meshnet_node.performance_contract`, `contract_version=1`, `ContractThresholds`) rather than re-defining or risking a conflicting duplicate. Only `dense-distributed-gguf` and `v4-flash-distributed` are newly locked here, each with fixed `prompt_ids`, `context_tokens`/`output_tokens` (alpha- and beta-scale for the V4 lane), `concurrency_levels`, `hardware` (named certification-scenario topology, network class, device class, MTP-off note), and a `metrics` list drawn from the existing `recipe_benchmark`/`performance_contract`/`route_session_benchmark` metric vocabulary (`ttft_p50_ms`, `decode_tokens_per_sec`, `seam_bytes`, `seam_latency_ms`, ...). - `gain_attribution` — two disjoint metric sets, `quantization_model_fit_metrics` and `runtime_transport_batching_kernel_metrics`, plus the rule that a speed/fit claim must cite which axis moved it. - `certification_scenarios` — `quantization` (`Q4_K_M`, `Q8_0`, `bf16-reference`) and `stage_count` (`2-4-stage`, `10-plus-stage`) as named labels only, with an explicit rule that no product/runtime code path may hardcode them. - `alpha` — correctness thresholds (greedy token agreement, mean state cosine similarity, nonfinite-tensor/fail-closed checks, no dense-attention-fallback credit) plus a `useful_speed` block whose ratios (`1.25`/`0.75`-class, matching the already-locked DGR-001 25% convention) carry an explicit `human_approval` sub-block (`required: true`, `approved: false`, `approved_by: null`, `approved_at: null`). The ratio alone cannot satisfy alpha; DGR-054 must fill in the approval against real evidence. `mtp.reserved=true`/`enabled_for_alpha=false` per `RALPH-CONTEXT.md`. `verdicts: ["alpha", "optimize", "stop"]`. - `beta` — adds exactly `concurrency`, `long_context`, `failure`, `sustained_throughput` axes (16k-token long-context threshold matching the V4 lane's `beta_context_tokens`, no-silent-KV- migration and no-synthetic-workers failure rules, 30-minute sustained-throughput floor). `verdicts: ["beta", "targeted-optimization", "stop-rollback"]`. - `amendment_policy` — thresholds may not be weakened/moved/reinterpreted after results are known; a change requires a new `contract_id`/`contract_version` under human review. - **`contract.py`** — loader/validator mirroring the proven `meshnet_node.glm_alpha.contract` pattern: `parse_contract` recomputes the canonical-JSON SHA-256 over the document (excluding the digest field) and requires it match both the document's own declared `contract_sha256` *and* a digest pinned independently in code (`CONTRACT_V1_SHA256`), so neither an in-place edit nor a resealed mutation can pass silently. Structural checks enforce all four required lanes, that the two referenced lanes actually declare `locked_elsewhere`, that the two newly-locked lanes carry full benchmark-plan fields, that `alpha.verdicts`/`beta.verdicts` are exactly the three-outcome sets the release gates use, and — the one property with no analogue in `glm_alpha` — that `alpha.useful_speed.human_approval.required` is `true`. `seal_contract()` is the only supported way to produce a new digest, kept separate from load-time verification for the same reason `glm_alpha` keeps it separate. - **`__init__.py`** — re-exports the public API, documented as the contract DGR-020, DGR-044, DGR-054, and DGR-070 are judged against. ### `tests/test_dgr_performance_contract.py` (new, 28 tests) Deterministic, offline, GPU-free, model-download-free. Covers: packaged load and identity; digest recomputation; all four lanes present; the two referenced lanes point at the real DGR-001 module and its actual immutable thresholds (`min_decode_speedup == 1.25`, `max_resident_memory_ratio == 0.75`); the two newly-locked lanes carry complete benchmark plans, fixed context/output/ concurrency; the shared prompt set and every lane's `prompt_ids`/`beta_prompt_ids` are a subset of it; sampling is greedy; `gain_attribution`'s two metric sets are non-empty and disjoint; certification-scenario names and rule text; **a structural test that greps every `.py` file under `packages/node/meshnet_node` (excluding this contract's own module and data file) for the literal strings `2-4-stage`/`10-plus-stage` and fails if any product module hardcodes them** — the concrete form of "no product logic may hardcode them"; alpha verdicts/correctness/`human_approval`/MTP-off; beta verdicts/axes/long-context/failure semantics; digest-mutation rejection (in-place and resealed); missing-digest rejection; `load_contract` from an explicit path matches the packaged load; `seal_contract` reproduces the pinned digest; amendment policy text. ### `.scratch/distributed-gguf-runtime/prd.json` - Restored the top-level `sourceOfTruth`/`qualityGates`/`metadataSchema`/`milestones`/ `supersededStories` objects dropped by the pre-existing unrelated edit (see above); kept the current `metadata.updatedAt` tooling stamp. - Marked `DGR-019.passes = true` with `completionNotes` summarizing this outcome. ### `.scratch/distributed-gguf-runtime/issues/019-lock-alpha-and-beta-performance-contracts.md` Regenerated via `python scripts/ralph_prd_schema.py render` to reflect `passes: true` (checked acceptance criteria, "completed" status line, "Verified evidence" handoff line), matching the convention DGR-017/DGR-018's issue files already use. ## Commands and results ```bash .venv-rocm/bin/python -m pytest -q tests/test_dgr_performance_contract.py ``` ```text 28 passed in 0.14s ``` ```bash .venv-rocm/bin/python -m pytest -q tests/test_ralph_prd_schema.py tests/test_dgr_performance_contract.py \ tests/test_glm_alpha_target.py tests/test_recipe_benchmark.py tests/test_route_session_benchmark.py ``` ```text 270 passed in 1.04s ``` ```bash .venv-rocm/bin/python -m compileall -q packages tests ``` Exit code 0, no output (all files compile). ```bash git diff --check ``` Exit code 0 (no whitespace errors). ```bash python3 scripts/ralph_prd_schema.py validate .scratch/distributed-gguf-runtime/prd.json ``` ```text OK: 55 stories validated. ``` ```bash .venv-rocm/bin/python -m pytest -q tests/ -k "not integration" --ignore=tests/test_shard_runtime_harness.py ``` ```text 7 failed, 1146 passed, 11 skipped, 4 deselected, 3 warnings in 261.71s (0:04:21) ``` This full sweep was launched in the background while `prd.json`/the evidence README below were still being written, so it raced its own inputs: one of its 7 failures (`test_ralph_prd_schema.py::test_real_backlog_passed_stories_have_completion_evidence`) was this story's own `passes=true`/evidence-README edit landing mid-run, not a real defect — re-running `tests/test_ralph_prd_schema.py` alone afterward, against the finalized tree, gives `108 passed`. The other 6 failures (`test_billing_ledger.py:: test_tracker_enables_billing_with_default_db`, `test_dynamic_routing.py:: test_admin_can_replace_a_served_model_and_release_it`, `test_dynamic_routing.py:: test_models_list_does_not_duplicate_a_preset_registered_by_hf_repo`, three cache tests in `test_real_model_backend.py`) are in files this story's `git diff` never touches (`git diff --stat HEAD -- tests/test_billing_ledger.py tests/test_dynamic_routing.py tests/test_real_model_backend.py` is empty) and none of them import `dgr_performance`, `performance_contract`, or `glm_alpha`; they are pre-existing baseline defects, not regressions from this story, in the same spirit as the known `origin/master` limitations DGR-017's evidence recorded. ## Known limitations - `tests/test_shard_runtime_harness.py` fails to *collect* in this environment (`ModuleNotFoundError: No module named 'grpc'`). This is a pre-existing environment gap from DGR-024's real generated-gRPC protocol harness, not something this story touched or caused; it is excluded from the sweep above rather than silently masked. - Alpha's `useful_speed` ratios (`1.25`/`0.75`-class) are proposed thresholds held at the same margin already locked for the whole-model contract (DGR-001/v1). They are locked numbers, but `human_approval.required=true` means DGR-054 may not treat them as self-certifying from the ratio alone — a human must approve the observed ratio against real evidence. This session did not, and could not, supply that approval: no distributed benchmark evidence exists yet. - `v4-flash-distributed`'s `reference_baseline` documents that a safetensors DeepSeek V4 Flash distributed baseline may not yet be pinned (that is DGR-044's job); until then, comparisons must fall back to `dense-distributed-gguf` runtime/transport overhead as an explicit, stated limitation rather than a silent substitution. - This is a specification-materialization story; per the shared quality gates, it is intentionally left uncommitted for manual review rather than given the "one scoped story commit" other stories get. ## Dependency handoff DGR-020 (run the controlled whole-model baseline) consumes the DGR-001 lock referenced — not redefined — by this contract's `controlled-safetensors`/`whole-model-gguf` lanes. DGR-044 (pin the DeepSeek V4 Flash target contract) and DGR-054/DGR-070 (enforce the alpha/beta gates) must load `meshnet_node.dgr_performance.load_contract()` and judge results against its `dense-distributed-gguf`/`v4-flash-distributed` lanes and `alpha`/`beta` sections without changing any threshold. DGR-054 specifically must populate `alpha.useful_speed.human_approval` (`approved`/`approved_by`/`approved_at`) as part of publishing its verdict — a satisfied ratio without a filled-in approval is not alpha certification. Any amendment must open a new `contract_id`/`contract_version` under human review per `amendment_policy`; this document and its digest are not editable in place.