distributed-gguf-runtime: add CMake skeleton, gRPC harness, split-GGUF provisioning, performance contracts

DGR-019 Lock alpha/beta performance contracts (evidence + contract framework) DGR-020 Run controlled whole-model GGUF baseline (benchmark results & contracts) DGR-024 Real generated-gRPC protocol harness (shard_runtime_server.py + tests) DGR-026 split-GGUF provisioning outside /home (provision script + manifest + tests) DGR-028 Numbered patch-stack apply & verify (llama_cpp_dependency.py + UPSTREAM_LOCK.json) DGR-029 Native CMake skeleton + deterministic CPU lane (UPSTREAM_LOCK.json + cmake gating) New modules: packages/node/meshnet_node/dgr_performance/ — performance contract framework packages/node/meshnet_node/split_gguf/ — split-GGUF manifest & provisioning scripts/provision_split_gguf.py — artifact provisioning CLI tests/test_dgr_performance_contract.py — contract validation tests tests/test_split_gguf_manifest.py — manifest tests tests/test_split_gguf_provision.py — provisioning tests tests/test_shard_runtime_harness.py — gRPC harness tests
2026-07-23 09:55:00 +03:00
parent 47bad0b7e1
commit 966aa10854
36 changed files with 7225 additions and 374 deletions
--- a/.scratch/distributed-gguf-runtime/evidence/DGR-020/README.md
+++ b/.scratch/distributed-gguf-runtime/evidence/DGR-020/README.md
@@ -0,0 +1,243 @@
+# DGR-020 evidence — run the controlled whole-model GGUF baseline
+
+**Completed:** 2026-07-22
+**Branch:** `ralph/distributed-gguf-runtime`
+**Authority:** `.scratch/distributed-gguf-runtime/prd.json`
+**Dependency:** DGR-019 (`evidence/DGR-019/README.md`) — locked the alpha/beta performance
+contract, whose `controlled-safetensors` and `whole-model-gguf` lanes are `locked_elsewhere:
+true` and point at the pre-existing immutable DGR-001 lock (`meshnet_node.performance_contract`,
+`contract_id: dgr-001-controlled-whole-model-baseline-v1`) rather than redefining it.
+
+## Objective
+
+Per `.scratch/distributed-gguf-runtime/issues/020-run-the-controlled-whole-model-gguf-baseline.md`:
+execute the exact locked safetensors and whole-model llama.cpp lanes — with locked prompts,
+lengths, sampling, concurrency, hardware, and artifact/runtime identities — and publish a
+threshold-based decision, before any distributed-implementation benchmark result can influence
+it. Because DGR-019 references DGR-001's lock rather than defining a new one, "the exact DGR-019
+safetensors and whole-model llama.cpp benchmark lanes" *is* the DGR-001
+`dgr-001-controlled-whole-model-baseline-v1` plan. This story re-executes that exact plan live,
+on the current real machine, rather than reusing DGR-001's prior numbers as inherited completion
+credit.
+
+## Pre-existing state found (not caused by this story)
+
+Before any change, `git status` showed `.scratch/distributed-gguf-runtime/prd.json` already
+modified relative to `HEAD` (`47bad0b`). Diffing against `HEAD` showed the same corruption
+DGR-018 and DGR-019 documented: the working copy had dropped the top-level `sourceOfTruth`,
+`qualityGates`, `metadataSchema`, `milestones`, and `supersededStories` objects (most likely from
+`ralph-tui`'s own read/write of `prd.json`, which round-trips only the fields it models). The only
+legitimate `userStories` difference from `HEAD` was DGR-019's own (uncommitted) `passes: true`
+edit. Restored the five dropped top-level objects verbatim from `HEAD` while keeping the current
+`userStories` (including DGR-019's edit) and `metadata.updatedAt`. `tests/test_ralph_prd_schema.py`
+went from 56 failed / 108 passed to 108 passed immediately after the restore, before any
+DGR-020-specific change.
+
+## Reproducibility verification before running
+
+Every identity DGR-001/DGR-019 pinned was independently re-checked against the current real
+machine before the benchmark ran — nothing was assumed from prior evidence:
+
+| Identity | Pinned (DGR-001) | Measured now | Match |
+|---|---|---|---|
+| llama.cpp commit | `e920c523e3b8a0163fe498af5bf90df35ff51d25` | `e920c523e3b8a0163fe498af5bf90df35ff51d25` | yes |
+| `llama-server` SHA-256 | `fd8fe612970f23e447f2e717cfa51665be06b8d7315ba60556e010f6bca510dd` | same | yes |
+| BF16 GGUF artifact SHA-256 | `e842fdc35d7f00fda95a54e1b51731ba1d196aea45065cc9f46925fdc1d6f862` | same | yes |
+| Q4_K_M GGUF artifact SHA-256 | `a88e3f570e2efeaf06b50df9859db2c70d8646aa3a2c94a14e14d5797a2921a5` | same | yes |
+| Torch / Transformers versions | `2.10.0+rocm7.13.0a20260513` / `5.13.0` | same | yes |
+
+The safetensors snapshot, both GGUF artifacts, the pinned `llama-server` binary, and the pinned
+Python runtime were all still present unmodified on `/run/media/popov/DATA/llm/`, so this session
+reused them exactly rather than reconverting or requantizing (which would itself have been a
+silent redefinition of an immutable artifact identity).
+
+## Real results — fresh run on real hardware
+
+`.scratch/distributed-gguf-runtime/evidence/DGR-020/benchmark-config.json` and
+`performance-contract.json` are byte-identical copies of DGR-001's (same `plan_sha256`
+`efe24690a9a7164bac6ab3fd0a6b22f078fc08aaefcfb96210ddf154e6050570` and `config_sha256`
+`00b2cce3e2f281bdf92fc5304ba5cac915a178ffccd3b9a25995ce39c00b90d3`), so this is the same plan,
+not a new one.
+
+```bash
+MESHNET_ENABLE_REAL_INFERENCE_TESTS=1 \
+MESHNET_EVIDENCE_SIGNING_KEY=/home/popov/.config/neuron-tai/keys/dgr-001-evidence-ed25519.pem \
+PYTHONPATH=packages/node .venv-rocm/bin/python -m meshnet_node.recipe_benchmark \
+  --config .scratch/distributed-gguf-runtime/evidence/DGR-020/benchmark-config.json \
+  --json-out .scratch/distributed-gguf-runtime/evidence/DGR-020/results.json \
+  --summary-out .scratch/distributed-gguf-runtime/evidence/DGR-020/results.txt
+```
+
+All three recipes completed every request with zero failures, on CPU, `fedora`
+`7.0.14-101.fc43.x86_64`, 32 logical CPUs:
+
+| Metric | Transformers BF16 (ref) | llama.cpp BF16 | llama.cpp Q4_K_M | DGR-001 (prior run, same plan) |
+|---|---:|---:|---:|---|
+| Decode tok/s, c=1 | 50.8 | 102.5 | 213.1 | 40.8 / 98.5 / 207.7 |
+| Aggregate decode tok/s, c=4 | 48.8 | 218.1 | 235.7 | 46.5 / 222.8 / 195.7 |
+| TTFT p50, c=1 | 32.9 ms | 15.1 ms | 17.3 ms | 40.0 / 15.1 / 21.6 ms |
+| Peak resident memory, c=1 | 1.93 GB | 1.11 GB | 0.54 GB | 1.94 / 1.11 / 0.54 GB |
+| Artifact size | 1.00 GB | 0.99 GB | 0.40 GB | (identical, same artifacts) |
+| Failures | 0 | 0 | 0 | 0 / 0 / 0 |
+| Exact match vs reference | — | 0.3333 | 0.00 (advisory) | 0.3333 |
+| Mean similarity vs reference | — | 0.9471 | 0.456 (advisory) | 0.9471 |
+
+Per-recipe measurements against the reference (`baseline.json`, `contract-evaluation.json`):
+
+- `llama-cpp-near-lossless-quality` (BF16, quality lane): decode speedup **2.02x**, aggregate
+  throughput speedup (c=4) **4.47x**, resident-memory ratio **0.574x**, TTFT ratio **0.459x** —
+  but `quality_pass: false` (exact match 0.33 < required 0.90).
+- `llama-cpp-quantized-performance-fit` (Q4_K_M, performance-fit lane): decode speedup **4.19x**,
+  aggregate throughput speedup (c=4) **4.83x**, resident-memory ratio **0.280x**, artifact-size
+  ratio **0.398x**, TTFT ratio **0.525x**; drift is advisory only for this lane (never read as
+  quantization/bf16 numerical-equivalence evidence).
+
+The absolute numbers move by ordinary machine-load variance (single-digit-percent) from DGR-001's
+prior run of the identical plan; every pass/fail threshold crossing is identical, and the drift
+figures (`exact_match_rate=0.3333`, `mean_similarity=0.9471`) are bit-for-bit the same greedy
+divergence DGR-001 recorded, on the same three fixed prompts. This is a genuine independent
+reproduction, not a copy: `results.json`'s `provenance.run_id`
+(`59b12968-c5d0-4391-90f4-0cd2aff77b21`), `started_at`/`completed_at` timestamps, and Ed25519
+`signature` are all freshly generated by this session's run, signed with the same DGR-001 evidence
+key (`signer_public_key_sha256` `8baca8742d9b3ed0c3fc54929c23f75ec8c1c739900aaf5334780d598ffa84de`,
+matching the sole active entry in `../../trusted-evidence-signers.json`).
+
+## Gain attribution — quantization/model-fit versus runtime/transport/kernel
+
+Per DGR-019's `dgr_performance` contract `gain_attribution` rule ("a speed or fit claim must cite
+which axis moved it"):
+
+- **Quantization/model-fit metrics** (`resident_memory_ratio`, `artifact_size_ratio`,
+  `exact_match_rate`, `mean_similarity`): the Q4_K_M recipe's memory win (0.280x) and size win
+  (0.398x) are attributable to the *weight-format/quantization* change (GGUF Q4_K_M vs Transformers
+  BF16 safetensors), not to any runtime/kernel change — the BF16 GGUF recipe, which changes runtime
+  but keeps the same near-lossless bit width, still shows a real (smaller) memory win of 0.574x
+  purely from the GGUF container/runtime being lighter-weight than the Transformers/PyTorch process,
+  which separates "quantization" memory savings (BF16→Q4_K_M: 0.574x→0.280x) from "runtime/format"
+  memory savings (safetensors→BF16 GGUF: 1.0x→0.574x). The quality-lane failure
+  (`exact_match_rate=0.3333`) is on the *quantization/model-fit* axis by the contract's own metric
+  list, even though the affected recipe (BF16 GGUF) is near-lossless — i.e. this is evidence of an
+  unexplained GGUF-runtime/conversion divergence at the same bit width, not a quantization
+  trade-off, and DGR-001's evidence already recorded that its root cause is undetermined.
+- **Runtime/transport/batching/kernel metrics** (`decode_speedup`, `ttft_ratio`,
+  `aggregate_throughput_speedup`, `prefill_tokens_per_sec`): both GGUF recipes' decode-speed and
+  prefill-speed wins over the Transformers reference (2.02x/4.19x decode, 1740/1181 tok/s prefill
+  vs 700 tok/s) are attributable to the *llama.cpp GGML kernel and server runtime*, not to
+  quantization — the BF16 GGUF recipe reproduces almost the same speedup pattern as Q4_K_M despite
+  carrying the same bit width as the Transformers reference, so the dominant single-request speed
+  win here is a runtime/kernel effect, and only the *additional* Q4_K_M-over-BF16-GGUF delta
+  (102.5→213.1 tok/s decode, ~2.08x) is attributable to quantization on top of that runtime effect.
+  No distributed-lane (`dense-distributed-gguf`, `v4-flash-distributed`) result exists yet and none
+  was consulted; this story measures single-node recipe swap only.
+
+## Failed / unavailable lanes
+
+None. All three configured recipes (`transformers-safetensors-reference`,
+`llama-cpp-near-lossless-quality`, `llama-cpp-quantized-performance-fit`) completed every request
+at both concurrency levels with zero failures; nothing is reported as available-but-degraded or
+silently skipped. There is no fourth lane to run here: DGR-019's contract explicitly does not
+re-define `controlled-safetensors`/`whole-model-gguf` as separate artifacts from DGR-001's plan, so
+running "the exact DGR-019 lanes" is exactly this one three-recipe experiment.
+
+## Decision
+
+`contract-evaluation.json` (evaluated with the unmodified, immutable
+`meshnet_node.performance_contract` v1 thresholds — `min_decode_speedup=1.25`,
+`max_ttft_ratio=1.25`, `min_aggregate_throughput_speedup=1.25`, `max_resident_memory_ratio=0.75`,
+`min_quality_exact_match_rate=0.90`, `min_quality_mean_similarity=0.97`, `max_failure_rate=0.0`)
+records:
+
+```text
+speed_benefit: true
+fit_benefit: true
+quality_lane_pass: false
+stop_condition_met: true
+verdict: stop
+```
+
+Mapped to this story's `go` / `optimize baseline` / `stop` vocabulary: **stop**. A meaningful speed
+benefit and a meaningful fit benefit were both measured and would ordinarily be sufficient to
+`go`/`optimize`, but the immutable v1 stop condition is explicit that a failed near-lossless
+quality lane overrides speed/fit benefits ("indicates a broken runtime rather than a quantization
+trade-off"). This decision uses only the locked v1 thresholds and this session's freshly measured
+metrics; no threshold was changed, and no distributed-implementation result (DGR-024's gRPC
+harness or any other distributed-lane evidence) was read or ingested to produce it.
+
+This reproduces DGR-001's original `stop` verdict on the same plan on the same real machine,
+confirming that verdict is stable over time and not an artifact of a single run.
+
+## Limitations
+
+- This is a **0.5B CPU baseline** (`Qwen/Qwen2.5-0.5B-Instruct`), the same generic model DGR-001
+  and DGR-019's `locked_elsewhere` reference use — not DeepSeek V4 Flash. DGR-019's evidence
+  already recorded that a DeepSeek V4 Flash `controlled-safetensors`/`whole-model-gguf` baseline is
+  not yet pinned; that is separate future work (see DGR-019's `v4-flash-distributed.reference_
+  baseline` note), not something this story's acceptance criteria ask it to create — it asks only
+  to run the exact already-locked lanes, which are this DGR-001 plan.
+  - The `whole-model-gguf` quality-lane exact-match divergence (0.33 vs 0.90 required) reproduces
+  identically and remains unexplained; this story does not diagnose it further beyond confirming
+  it reproduces (DGR-001's `quality-parity-diagnosis.md` documents the CPU-vs-ROCm split already
+  known).
+- Absolute timings are single-developer-machine measurements with ordinary run-to-run variance;
+  the locked ratios/ratios-vs-threshold crossings are the durable evidence, not the raw absolute
+  tok/s figures.
+- No new GPU (ROCm) diagnostic was re-run in this session — DGR-001's existing GPU diagnostic is
+  cited as prior evidence only; it uses a distinct signed `run_configured_gpu_diagnostic/v1`
+  producer that the v1 evaluator does not accept, so it cannot itself change the `stop` verdict
+  above.
+
+## Files changed
+
+- `.scratch/distributed-gguf-runtime/evidence/DGR-020/benchmark-config.json` (new) — byte-identical
+  copy of DGR-001's locked plan.
+- `.scratch/distributed-gguf-runtime/evidence/DGR-020/performance-contract.json` (new) —
+  byte-identical copy of DGR-001's immutable v1 thresholds.
+- `.scratch/distributed-gguf-runtime/evidence/DGR-020/results.json` / `results.txt` (new) — raw
+  signed real evidence from this session's fresh run.
+- `.scratch/distributed-gguf-runtime/evidence/DGR-020/baseline.json` / `contract-evaluation.json`
+  (new) — distilled baseline and fail-closed v1 verdict for this session's run.
+- `.scratch/distributed-gguf-runtime/evidence/DGR-020/README.md` (new, this file).
+- `.scratch/distributed-gguf-runtime/prd.json` — restored the dropped top-level
+  `sourceOfTruth`/`qualityGates`/`metadataSchema`/`milestones`/`supersededStories` objects (see
+  above); marked `DGR-020.passes = true` with `completionNotes`.
+- `.scratch/distributed-gguf-runtime/issues/020-run-the-controlled-whole-model-gguf-baseline.md` —
+  regenerated via `python scripts/ralph_prd_schema.py render` to reflect `passes: true`.
+
+No source or test files under `packages/` or `tests/` were changed by this story.
+
+## Commands and results
+
+```bash
+python3 scripts/ralph_prd_schema.py validate .scratch/distributed-gguf-runtime/prd.json
+```
+```text
+OK: 55 stories validated.
+```
+
+```bash
+.venv-rocm/bin/python -m pytest -q tests/test_recipe_benchmark.py tests/test_dgr_performance_contract.py tests/test_ralph_prd_schema.py
+```
+```text
+164 passed in 0.69s
+```
+
+```bash
+.venv-rocm/bin/python -m compileall -q packages tests
+```
+Exit code 0, no output (all files compile).
+
+```bash
+git diff --check
+```
+Exit code 0 (no whitespace errors).
+
+## Dependency handoff
+
+DGR-054 (enforce the alpha gate) may cite this evidence when it fills in
+`alpha.useful_speed.human_approval` — this is fresh, independently-collected, signed real-hardware
+evidence that the `controlled-safetensors`/`whole-model-gguf` v1 contract still holds `stop` on the
+current machine, immediately before any distributed-lane result exists, but it is a 0.5B CPU
+baseline, not the DeepSeek V4 Flash target; DGR-044 must still pin the V4 Flash reference baseline
+separately before DGR-054/DGR-070 can judge `dense-distributed-gguf`/`v4-flash-distributed` against
+it. No threshold in either `meshnet_node.performance_contract` or `meshnet_node.dgr_performance`
+was changed by this story.