distributed-gguf-runtime: add CMake skeleton, gRPC harness, split-GGUF provisioning, performance contracts
DGR-019 Lock alpha/beta performance contracts (evidence + contract framework) DGR-020 Run controlled whole-model GGUF baseline (benchmark results & contracts) DGR-024 Real generated-gRPC protocol harness (shard_runtime_server.py + tests) DGR-026 split-GGUF provisioning outside /home (provision script + manifest + tests) DGR-028 Numbered patch-stack apply & verify (llama_cpp_dependency.py + UPSTREAM_LOCK.json) DGR-029 Native CMake skeleton + deterministic CPU lane (UPSTREAM_LOCK.json + cmake gating) New modules: packages/node/meshnet_node/dgr_performance/ — performance contract framework packages/node/meshnet_node/split_gguf/ — split-GGUF manifest & provisioning scripts/provision_split_gguf.py — artifact provisioning CLI tests/test_dgr_performance_contract.py — contract validation tests tests/test_split_gguf_manifest.py — manifest tests tests/test_split_gguf_provision.py — provisioning tests tests/test_shard_runtime_harness.py — gRPC harness tests
This commit is contained in:
Dobromir Popov
215
.scratch/distributed-gguf-runtime/evidence/DGR-019/README.md
Normal file
215
.scratch/distributed-gguf-runtime/evidence/DGR-019/README.md
Normal file
@@ -0,0 +1,215 @@
|
||||
# DGR-019 evidence — lock alpha and beta performance contracts
|
||||
|
||||
**Completed:** 2026-07-22
|
||||
**Branch:** `ralph/distributed-gguf-runtime`
|
||||
**Authority:** `.scratch/distributed-gguf-runtime/prd.json`
|
||||
**Dependency:** DGR-017 (`evidence/DGR-017/README.md`) — cleaned backlog reconciled to `origin/master`; no old pass state transferred.
|
||||
|
||||
## Objective
|
||||
|
||||
Freeze useful-speed, correctness, memory-fit, and stop/go thresholds for the DeepSeek V4 Flash
|
||||
distributed GGUF track *before* any distributed implementation produces a benchmark result, per
|
||||
`.scratch/distributed-gguf-runtime/issues/019-lock-alpha-and-beta-performance-contracts.md`.
|
||||
|
||||
## Pre-existing state found (not caused by this story)
|
||||
|
||||
Before any change in this session, `git status` showed `.scratch/distributed-gguf-runtime/prd.json`
|
||||
already modified in the working tree relative to `HEAD` (commit `47bad0b`), with no corresponding
|
||||
progress-log entry. Diffing against `HEAD` showed the working copy had **dropped** prd.json's
|
||||
top-level `sourceOfTruth`, `qualityGates`, `metadataSchema`, `milestones`, and `supersededStories`
|
||||
objects (replacing them with only a bare `metadata: {"updatedAt": ...}` stamp), while `userStories`
|
||||
itself was byte-identical to `HEAD`. Running `tests/test_ralph_prd_schema.py` against the
|
||||
as-found working tree confirmed the damage: 56 of 108 tests failed (every
|
||||
`test_render_issue_markdown_matches_committed_file[...]` parametrization, since
|
||||
`quality_gate_bullets`/`authority_disclaimer` fall back to module defaults once `qualityGates`/
|
||||
`metadataSchema` are absent, which no longer match the committed issue files).
|
||||
|
||||
This is the same shape of problem DGR-018's evidence documented and fixed: an abandoned,
|
||||
unexplained edit that silently dropped the schema/gates/milestone/provenance content this and
|
||||
future stories depend on, while `scripts/ralph_prd_schema.py validate` did not catch it (those
|
||||
top-level sections are optional-if-absent by design, so the CLI reported `OK: 55 stories
|
||||
validated.` even with them missing). The most likely cause is `ralph-tui`'s own read/write of
|
||||
`prd.json` as its task source, which only round-trips the fields it models
|
||||
(`name`/`description`/`branchName`/`userStories`) and stamps its own `metadata.updatedAt`,
|
||||
dropping any project-specific extension fields it doesn't know about.
|
||||
|
||||
Per `RALPH-CONTEXT.md`'s instruction to inspect `git status` and preserve unrelated work rather
|
||||
than build on top of unexplained state, and following the DGR-018 precedent, the dropped fields
|
||||
were restored verbatim from `HEAD` (`git show HEAD:.scratch/distributed-gguf-runtime/prd.json`)
|
||||
while keeping the current `userStories` content (identical) and the current `metadata.updatedAt`
|
||||
stamp. `tests/test_ralph_prd_schema.py` returned to `108 passed` immediately after the restore,
|
||||
before any DGR-019-specific change was made.
|
||||
|
||||
## Changes
|
||||
|
||||
### `packages/node/meshnet_node/dgr_performance/` (new package)
|
||||
|
||||
- **`data/alpha-beta-contract-v1.json`** — the locked, versioned, machine-readable contract.
|
||||
`schema_version`/`contract_version`/`contract_id` (`dgr-alpha-beta-performance/v1`), sealed with
|
||||
a `contract_sha256` digest over its own canonical content (the repository's existing digest
|
||||
convention, shared with `meshnet_node.glm_alpha.contract`). Contents:
|
||||
- `prompt_set` — four fixed prompts (`short-instruction`, `code-completion`,
|
||||
`multi-step-reasoning`, `long-context-fill`) referenced by ID from every lane, so no lane can
|
||||
quietly drift onto a different workload.
|
||||
- `sampling` — greedy (`temperature=0`, `top_p=1`, `top_k=1`, `seed=1234`), matching
|
||||
`meshnet_node.recipe_benchmark.SamplingPolicy` defaults.
|
||||
- `lanes` — all four lanes named in the acceptance criteria. `controlled-safetensors` and
|
||||
`whole-model-gguf` are marked `locked_elsewhere: true` and point at the pre-existing immutable
|
||||
DGR-001 lock (`meshnet_node.performance_contract`, `contract_version=1`,
|
||||
`ContractThresholds`) rather than re-defining or risking a conflicting duplicate. Only
|
||||
`dense-distributed-gguf` and `v4-flash-distributed` are newly locked here, each with fixed
|
||||
`prompt_ids`, `context_tokens`/`output_tokens` (alpha- and beta-scale for the V4 lane),
|
||||
`concurrency_levels`, `hardware` (named certification-scenario topology, network class, device
|
||||
class, MTP-off note), and a `metrics` list drawn from the existing
|
||||
`recipe_benchmark`/`performance_contract`/`route_session_benchmark` metric vocabulary
|
||||
(`ttft_p50_ms`, `decode_tokens_per_sec`, `seam_bytes`, `seam_latency_ms`, ...).
|
||||
- `gain_attribution` — two disjoint metric sets, `quantization_model_fit_metrics` and
|
||||
`runtime_transport_batching_kernel_metrics`, plus the rule that a speed/fit claim must cite
|
||||
which axis moved it.
|
||||
- `certification_scenarios` — `quantization` (`Q4_K_M`, `Q8_0`, `bf16-reference`) and
|
||||
`stage_count` (`2-4-stage`, `10-plus-stage`) as named labels only, with an explicit rule that
|
||||
no product/runtime code path may hardcode them.
|
||||
- `alpha` — correctness thresholds (greedy token agreement, mean state cosine similarity,
|
||||
nonfinite-tensor/fail-closed checks, no dense-attention-fallback credit) plus a `useful_speed`
|
||||
block whose ratios (`1.25`/`0.75`-class, matching the already-locked DGR-001 25% convention)
|
||||
carry an explicit `human_approval` sub-block (`required: true`, `approved: false`,
|
||||
`approved_by: null`, `approved_at: null`). The ratio alone cannot satisfy alpha; DGR-054 must
|
||||
fill in the approval against real evidence. `mtp.reserved=true`/`enabled_for_alpha=false` per
|
||||
`RALPH-CONTEXT.md`. `verdicts: ["alpha", "optimize", "stop"]`.
|
||||
- `beta` — adds exactly `concurrency`, `long_context`, `failure`, `sustained_throughput` axes
|
||||
(16k-token long-context threshold matching the V4 lane's `beta_context_tokens`, no-silent-KV-
|
||||
migration and no-synthetic-workers failure rules, 30-minute sustained-throughput floor).
|
||||
`verdicts: ["beta", "targeted-optimization", "stop-rollback"]`.
|
||||
- `amendment_policy` — thresholds may not be weakened/moved/reinterpreted after results are
|
||||
known; a change requires a new `contract_id`/`contract_version` under human review.
|
||||
- **`contract.py`** — loader/validator mirroring the proven
|
||||
`meshnet_node.glm_alpha.contract` pattern: `parse_contract` recomputes the canonical-JSON SHA-256
|
||||
over the document (excluding the digest field) and requires it match both the document's own
|
||||
declared `contract_sha256` *and* a digest pinned independently in code
|
||||
(`CONTRACT_V1_SHA256`), so neither an in-place edit nor a resealed mutation can pass silently.
|
||||
Structural checks enforce all four required lanes, that the two referenced lanes actually
|
||||
declare `locked_elsewhere`, that the two newly-locked lanes carry full benchmark-plan fields,
|
||||
that `alpha.verdicts`/`beta.verdicts` are exactly the three-outcome sets the release gates use,
|
||||
and — the one property with no analogue in `glm_alpha` — that
|
||||
`alpha.useful_speed.human_approval.required` is `true`. `seal_contract()` is the only supported
|
||||
way to produce a new digest, kept separate from load-time verification for the same reason
|
||||
`glm_alpha` keeps it separate.
|
||||
- **`__init__.py`** — re-exports the public API, documented as the contract DGR-020, DGR-044,
|
||||
DGR-054, and DGR-070 are judged against.
|
||||
|
||||
### `tests/test_dgr_performance_contract.py` (new, 28 tests)
|
||||
|
||||
Deterministic, offline, GPU-free, model-download-free. Covers: packaged load and identity; digest
|
||||
recomputation; all four lanes present; the two referenced lanes point at the real DGR-001 module
|
||||
and its actual immutable thresholds (`min_decode_speedup == 1.25`, `max_resident_memory_ratio ==
|
||||
0.75`); the two newly-locked lanes carry complete benchmark plans, fixed context/output/
|
||||
concurrency; the shared prompt set and every lane's `prompt_ids`/`beta_prompt_ids` are a subset of
|
||||
it; sampling is greedy; `gain_attribution`'s two metric sets are non-empty and disjoint;
|
||||
certification-scenario names and rule text; **a structural test that greps every `.py` file under
|
||||
`packages/node/meshnet_node` (excluding this contract's own module and data file) for the literal
|
||||
strings `2-4-stage`/`10-plus-stage` and fails if any product module hardcodes them** — the concrete
|
||||
form of "no product logic may hardcode them"; alpha verdicts/correctness/`human_approval`/MTP-off;
|
||||
beta verdicts/axes/long-context/failure semantics; digest-mutation rejection (in-place and
|
||||
resealed); missing-digest rejection; `load_contract` from an explicit path matches the packaged
|
||||
load; `seal_contract` reproduces the pinned digest; amendment policy text.
|
||||
|
||||
### `.scratch/distributed-gguf-runtime/prd.json`
|
||||
|
||||
- Restored the top-level `sourceOfTruth`/`qualityGates`/`metadataSchema`/`milestones`/
|
||||
`supersededStories` objects dropped by the pre-existing unrelated edit (see above); kept the
|
||||
current `metadata.updatedAt` tooling stamp.
|
||||
- Marked `DGR-019.passes = true` with `completionNotes` summarizing this outcome.
|
||||
|
||||
### `.scratch/distributed-gguf-runtime/issues/019-lock-alpha-and-beta-performance-contracts.md`
|
||||
|
||||
Regenerated via `python scripts/ralph_prd_schema.py render` to reflect `passes: true` (checked
|
||||
acceptance criteria, "completed" status line, "Verified evidence" handoff line), matching the
|
||||
convention DGR-017/DGR-018's issue files already use.
|
||||
|
||||
## Commands and results
|
||||
|
||||
```bash
|
||||
.venv-rocm/bin/python -m pytest -q tests/test_dgr_performance_contract.py
|
||||
```
|
||||
```text
|
||||
28 passed in 0.14s
|
||||
```
|
||||
|
||||
```bash
|
||||
.venv-rocm/bin/python -m pytest -q tests/test_ralph_prd_schema.py tests/test_dgr_performance_contract.py \
|
||||
tests/test_glm_alpha_target.py tests/test_recipe_benchmark.py tests/test_route_session_benchmark.py
|
||||
```
|
||||
```text
|
||||
270 passed in 1.04s
|
||||
```
|
||||
|
||||
```bash
|
||||
.venv-rocm/bin/python -m compileall -q packages tests
|
||||
```
|
||||
Exit code 0, no output (all files compile).
|
||||
|
||||
```bash
|
||||
git diff --check
|
||||
```
|
||||
Exit code 0 (no whitespace errors).
|
||||
|
||||
```bash
|
||||
python3 scripts/ralph_prd_schema.py validate .scratch/distributed-gguf-runtime/prd.json
|
||||
```
|
||||
```text
|
||||
OK: 55 stories validated.
|
||||
```
|
||||
|
||||
```bash
|
||||
.venv-rocm/bin/python -m pytest -q tests/ -k "not integration" --ignore=tests/test_shard_runtime_harness.py
|
||||
```
|
||||
```text
|
||||
7 failed, 1146 passed, 11 skipped, 4 deselected, 3 warnings in 261.71s (0:04:21)
|
||||
```
|
||||
This full sweep was launched in the background while `prd.json`/the evidence README below were
|
||||
still being written, so it raced its own inputs: one of its 7 failures
|
||||
(`test_ralph_prd_schema.py::test_real_backlog_passed_stories_have_completion_evidence`) was this
|
||||
story's own `passes=true`/evidence-README edit landing mid-run, not a real defect — re-running
|
||||
`tests/test_ralph_prd_schema.py` alone afterward, against the finalized tree, gives
|
||||
`108 passed`. The other 6 failures (`test_billing_ledger.py::
|
||||
test_tracker_enables_billing_with_default_db`, `test_dynamic_routing.py::
|
||||
test_admin_can_replace_a_served_model_and_release_it`, `test_dynamic_routing.py::
|
||||
test_models_list_does_not_duplicate_a_preset_registered_by_hf_repo`, three cache tests in
|
||||
`test_real_model_backend.py`) are in files this story's `git diff` never touches (`git diff --stat
|
||||
HEAD -- tests/test_billing_ledger.py tests/test_dynamic_routing.py tests/test_real_model_backend.py`
|
||||
is empty) and none of them import `dgr_performance`, `performance_contract`, or `glm_alpha`; they
|
||||
are pre-existing baseline defects, not regressions from this story, in the same spirit as the
|
||||
known `origin/master` limitations DGR-017's evidence recorded.
|
||||
|
||||
## Known limitations
|
||||
|
||||
- `tests/test_shard_runtime_harness.py` fails to *collect* in this environment
|
||||
(`ModuleNotFoundError: No module named 'grpc'`). This is a pre-existing environment gap from
|
||||
DGR-024's real generated-gRPC protocol harness, not something this story touched or caused; it is
|
||||
excluded from the sweep above rather than silently masked.
|
||||
- Alpha's `useful_speed` ratios (`1.25`/`0.75`-class) are proposed thresholds held at the same
|
||||
margin already locked for the whole-model contract (DGR-001/v1). They are locked numbers, but
|
||||
`human_approval.required=true` means DGR-054 may not treat them as self-certifying from the
|
||||
ratio alone — a human must approve the observed ratio against real evidence. This session did
|
||||
not, and could not, supply that approval: no distributed benchmark evidence exists yet.
|
||||
- `v4-flash-distributed`'s `reference_baseline` documents that a safetensors DeepSeek V4 Flash
|
||||
distributed baseline may not yet be pinned (that is DGR-044's job); until then, comparisons must
|
||||
fall back to `dense-distributed-gguf` runtime/transport overhead as an explicit, stated
|
||||
limitation rather than a silent substitution.
|
||||
- This is a specification-materialization story; per the shared quality gates, it is intentionally
|
||||
left uncommitted for manual review rather than given the "one scoped story commit" other stories
|
||||
get.
|
||||
|
||||
## Dependency handoff
|
||||
|
||||
DGR-020 (run the controlled whole-model baseline) consumes the DGR-001 lock referenced — not
|
||||
redefined — by this contract's `controlled-safetensors`/`whole-model-gguf` lanes.
|
||||
|
||||
DGR-044 (pin the DeepSeek V4 Flash target contract) and DGR-054/DGR-070 (enforce the alpha/beta
|
||||
gates) must load `meshnet_node.dgr_performance.load_contract()` and judge results against its
|
||||
`dense-distributed-gguf`/`v4-flash-distributed` lanes and `alpha`/`beta` sections without changing
|
||||
any threshold. DGR-054 specifically must populate `alpha.useful_speed.human_approval`
|
||||
(`approved`/`approved_by`/`approved_at`) as part of publishing its verdict — a satisfied ratio
|
||||
without a filled-in approval is not alpha certification. Any amendment must open a new
|
||||
`contract_id`/`contract_version` under human review per `amendment_policy`; this document and its
|
||||
digest are not editable in place.
|
||||
Reference in New Issue
Block a user