neuron-tai/.scratch/distributed-gguf-runtime/evidence/DGR-019

DGR-019 evidence — lock alpha and beta performance contracts

Completed: 2026-07-22 Branch: ralph/distributed-gguf-runtime Authority: .scratch/distributed-gguf-runtime/prd.json Dependency: DGR-017 (evidence/DGR-017/README.md) — cleaned backlog reconciled to origin/master; no old pass state transferred.

Objective

Freeze useful-speed, correctness, memory-fit, and stop/go thresholds for the DeepSeek V4 Flash distributed GGUF track before any distributed implementation produces a benchmark result, per .scratch/distributed-gguf-runtime/issues/019-lock-alpha-and-beta-performance-contracts.md.

Pre-existing state found (not caused by this story)

Before any change in this session, git status showed .scratch/distributed-gguf-runtime/prd.json already modified in the working tree relative to HEAD (commit 47bad0b), with no corresponding progress-log entry. Diffing against HEAD showed the working copy had dropped prd.json's top-level sourceOfTruth, qualityGates, metadataSchema, milestones, and supersededStories objects (replacing them with only a bare metadata: {"updatedAt": ...} stamp), while userStories itself was byte-identical to HEAD. Running tests/test_ralph_prd_schema.py against the as-found working tree confirmed the damage: 56 of 108 tests failed (every test_render_issue_markdown_matches_committed_file[...] parametrization, since quality_gate_bullets/authority_disclaimer fall back to module defaults once qualityGates/ metadataSchema are absent, which no longer match the committed issue files).

This is the same shape of problem DGR-018's evidence documented and fixed: an abandoned, unexplained edit that silently dropped the schema/gates/milestone/provenance content this and future stories depend on, while scripts/ralph_prd_schema.py validate did not catch it (those top-level sections are optional-if-absent by design, so the CLI reported OK: 55 stories validated. even with them missing). The most likely cause is ralph-tui's own read/write of prd.json as its task source, which only round-trips the fields it models (name/description/branchName/userStories) and stamps its own metadata.updatedAt, dropping any project-specific extension fields it doesn't know about.

Per RALPH-CONTEXT.md's instruction to inspect git status and preserve unrelated work rather than build on top of unexplained state, and following the DGR-018 precedent, the dropped fields were restored verbatim from HEAD (git show HEAD:.scratch/distributed-gguf-runtime/prd.json) while keeping the current userStories content (identical) and the current metadata.updatedAt stamp. tests/test_ralph_prd_schema.py returned to 108 passed immediately after the restore, before any DGR-019-specific change was made.

Changes

packages/node/meshnet_node/dgr_performance/ (new package)

• data/alpha-beta-contract-v1.json — the locked, versioned, machine-readable contract. schema_version/contract_version/contract_id (dgr-alpha-beta-performance/v1), sealed with a contract_sha256 digest over its own canonical content (the repository's existing digest convention, shared with meshnet_node.glm_alpha.contract). Contents:
  • prompt_set — four fixed prompts (short-instruction, code-completion, multi-step-reasoning, long-context-fill) referenced by ID from every lane, so no lane can quietly drift onto a different workload.
  • sampling — greedy (temperature=0, top_p=1, top_k=1, seed=1234), matching meshnet_node.recipe_benchmark.SamplingPolicy defaults.
  • lanes — all four lanes named in the acceptance criteria. controlled-safetensors and whole-model-gguf are marked locked_elsewhere: true and point at the pre-existing immutable DGR-001 lock (meshnet_node.performance_contract, contract_version=1, ContractThresholds) rather than re-defining or risking a conflicting duplicate. Only dense-distributed-gguf and v4-flash-distributed are newly locked here, each with fixed prompt_ids, context_tokens/output_tokens (alpha- and beta-scale for the V4 lane), concurrency_levels, hardware (named certification-scenario topology, network class, device class, MTP-off note), and a metrics list drawn from the existing recipe_benchmark/performance_contract/route_session_benchmark metric vocabulary (ttft_p50_ms, decode_tokens_per_sec, seam_bytes, seam_latency_ms, ...).
  • gain_attribution — two disjoint metric sets, quantization_model_fit_metrics and runtime_transport_batching_kernel_metrics, plus the rule that a speed/fit claim must cite which axis moved it.
  • certification_scenarios — quantization (Q4_K_M, Q8_0, bf16-reference) and stage_count (2-4-stage, 10-plus-stage) as named labels only, with an explicit rule that no product/runtime code path may hardcode them.
  • alpha — correctness thresholds (greedy token agreement, mean state cosine similarity, nonfinite-tensor/fail-closed checks, no dense-attention-fallback credit) plus a useful_speed block whose ratios (1.25/0.75-class, matching the already-locked DGR-001 25% convention) carry an explicit human_approval sub-block (required: true, approved: false, approved_by: null, approved_at: null). The ratio alone cannot satisfy alpha; DGR-054 must fill in the approval against real evidence. mtp.reserved=true/enabled_for_alpha=false per RALPH-CONTEXT.md. verdicts: ["alpha", "optimize", "stop"].
  • beta — adds exactly concurrency, long_context, failure, sustained_throughput axes (16k-token long-context threshold matching the V4 lane's beta_context_tokens, no-silent-KV- migration and no-synthetic-workers failure rules, 30-minute sustained-throughput floor). verdicts: ["beta", "targeted-optimization", "stop-rollback"].
  • amendment_policy — thresholds may not be weakened/moved/reinterpreted after results are known; a change requires a new contract_id/contract_version under human review.
• contract.py — loader/validator mirroring the proven meshnet_node.glm_alpha.contract pattern: parse_contract recomputes the canonical-JSON SHA-256 over the document (excluding the digest field) and requires it match both the document's own declared contract_sha256 and a digest pinned independently in code (CONTRACT_V1_SHA256), so neither an in-place edit nor a resealed mutation can pass silently. Structural checks enforce all four required lanes, that the two referenced lanes actually declare locked_elsewhere, that the two newly-locked lanes carry full benchmark-plan fields, that alpha.verdicts/beta.verdicts are exactly the three-outcome sets the release gates use, and — the one property with no analogue in glm_alpha — that alpha.useful_speed.human_approval.required is true. seal_contract() is the only supported way to produce a new digest, kept separate from load-time verification for the same reason glm_alpha keeps it separate.
• __init__.py — re-exports the public API, documented as the contract DGR-020, DGR-044, DGR-054, and DGR-070 are judged against.

tests/test_dgr_performance_contract.py (new, 28 tests)

Deterministic, offline, GPU-free, model-download-free. Covers: packaged load and identity; digest recomputation; all four lanes present; the two referenced lanes point at the real DGR-001 module and its actual immutable thresholds (min_decode_speedup == 1.25, max_resident_memory_ratio == 0.75); the two newly-locked lanes carry complete benchmark plans, fixed context/output/ concurrency; the shared prompt set and every lane's prompt_ids/beta_prompt_ids are a subset of it; sampling is greedy; gain_attribution's two metric sets are non-empty and disjoint; certification-scenario names and rule text; a structural test that greps every .py file under packages/node/meshnet_node (excluding this contract's own module and data file) for the literal strings 2-4-stage/10-plus-stage and fails if any product module hardcodes them — the concrete form of "no product logic may hardcode them"; alpha verdicts/correctness/human_approval/MTP-off; beta verdicts/axes/long-context/failure semantics; digest-mutation rejection (in-place and resealed); missing-digest rejection; load_contract from an explicit path matches the packaged load; seal_contract reproduces the pinned digest; amendment policy text.

.scratch/distributed-gguf-runtime/prd.json

• Restored the top-level sourceOfTruth/qualityGates/metadataSchema/milestones/ supersededStories objects dropped by the pre-existing unrelated edit (see above); kept the current metadata.updatedAt tooling stamp.
• Marked DGR-019.passes = true with completionNotes summarizing this outcome.

.scratch/distributed-gguf-runtime/issues/019-lock-alpha-and-beta-performance-contracts.md

Regenerated via python scripts/ralph_prd_schema.py render to reflect passes: true (checked acceptance criteria, "completed" status line, "Verified evidence" handoff line), matching the convention DGR-017/DGR-018's issue files already use.

Commands and results

.venv-rocm/bin/python -m pytest -q tests/test_dgr_performance_contract.py

28 passed in 0.14s

.venv-rocm/bin/python -m pytest -q tests/test_ralph_prd_schema.py tests/test_dgr_performance_contract.py \
  tests/test_glm_alpha_target.py tests/test_recipe_benchmark.py tests/test_route_session_benchmark.py

270 passed in 1.04s

.venv-rocm/bin/python -m compileall -q packages tests

Exit code 0, no output (all files compile).

git diff --check

Exit code 0 (no whitespace errors).

python3 scripts/ralph_prd_schema.py validate .scratch/distributed-gguf-runtime/prd.json

OK: 55 stories validated.

.venv-rocm/bin/python -m pytest -q tests/ -k "not integration" --ignore=tests/test_shard_runtime_harness.py

7 failed, 1146 passed, 11 skipped, 4 deselected, 3 warnings in 261.71s (0:04:21)

This full sweep was launched in the background while prd.json/the evidence README below were still being written, so it raced its own inputs: one of its 7 failures (test_ralph_prd_schema.py::test_real_backlog_passed_stories_have_completion_evidence) was this story's own passes=true/evidence-README edit landing mid-run, not a real defect — re-running tests/test_ralph_prd_schema.py alone afterward, against the finalized tree, gives 108 passed. The other 6 failures (test_billing_ledger.py:: test_tracker_enables_billing_with_default_db, test_dynamic_routing.py:: test_admin_can_replace_a_served_model_and_release_it, test_dynamic_routing.py:: test_models_list_does_not_duplicate_a_preset_registered_by_hf_repo, three cache tests in test_real_model_backend.py) are in files this story's git diff never touches (git diff --stat HEAD -- tests/test_billing_ledger.py tests/test_dynamic_routing.py tests/test_real_model_backend.py is empty) and none of them import dgr_performance, performance_contract, or glm_alpha; they are pre-existing baseline defects, not regressions from this story, in the same spirit as the known origin/master limitations DGR-017's evidence recorded.

Known limitations

• tests/test_shard_runtime_harness.py fails to collect in this environment (ModuleNotFoundError: No module named 'grpc'). This is a pre-existing environment gap from DGR-024's real generated-gRPC protocol harness, not something this story touched or caused; it is excluded from the sweep above rather than silently masked.
• Alpha's useful_speed ratios (1.25/0.75-class) are proposed thresholds held at the same margin already locked for the whole-model contract (DGR-001/v1). They are locked numbers, but human_approval.required=true means DGR-054 may not treat them as self-certifying from the ratio alone — a human must approve the observed ratio against real evidence. This session did not, and could not, supply that approval: no distributed benchmark evidence exists yet.
• v4-flash-distributed's reference_baseline documents that a safetensors DeepSeek V4 Flash distributed baseline may not yet be pinned (that is DGR-044's job); until then, comparisons must fall back to dense-distributed-gguf runtime/transport overhead as an explicit, stated limitation rather than a silent substitution.
• This is a specification-materialization story; per the shared quality gates, it is intentionally left uncommitted for manual review rather than given the "one scoped story commit" other stories get.

Dependency handoff

DGR-020 (run the controlled whole-model baseline) consumes the DGR-001 lock referenced — not redefined — by this contract's controlled-safetensors/whole-model-gguf lanes.

DGR-044 (pin the DeepSeek V4 Flash target contract) and DGR-054/DGR-070 (enforce the alpha/beta gates) must load meshnet_node.dgr_performance.load_contract() and judge results against its dense-distributed-gguf/v4-flash-distributed lanes and alpha/beta sections without changing any threshold. DGR-054 specifically must populate alpha.useful_speed.human_approval (approved/approved_by/approved_at) as part of publishing its verdict — a satisfied ratio without a filled-in approval is not alpha certification. Any amendment must open a new contract_id/contract_version under human review per amendment_policy; this document and its digest are not editable in place.