popov/neuron-tai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: harden DGR-001 performance contract evidence

Browse Source
This commit is contained in:
Dobromir Popov
2026-07-13 19:10:24 +03:00
parent e24db7854f
commit 9e67b829e3
16 changed files with 3674 additions and 151 deletions
Download Patch File Download Diff File Expand all files Collapse all files

61
.scratch/distributed-gguf-runtime/evidence/DGR-001/BLOCKED.md
View File

@@ -1,30 +1,43 @@
# DGR-001 real-benchmark blocker # DGR-001 downstream stop-condition handoff
Status: blocked only for the required real-model measurement. The deterministic Status: **DGR-001 is complete; native-track promotion is blocked by the immutable v1 verdict.**
harness, report schema, and immutable contract are implemented and tested; this
file deliberately does not turn an unrun benchmark into a passing result.
## Verified environment state (2026-07-13) This is no longer an execution-prerequisite blocker. The required real benchmark
ran successfully, every recipe completed at concurrency 1 and 4, artifacts were
verified, and deterministic/full test gates passed.
- Mounted GGUF artifacts exist under `/run/media/popov/DATA/llm/`. ## Locked result
- `llama-server` is not on `PATH`.
- The available Python test environment has neither `torch` nor `transformers`.
- No matching local safetensors snapshot was found for an installed GGUF recipe.
Therefore this session cannot run the controlled same-model, same-revision, `contract-evaluation.json` records:
same-machine comparison without downloading/installing new runtime/model assets.
That is intentionally not inferred from the story request.
## Continuation ```text
verdict: stop
quality_lane_pass: false
speed_benefit: true
fit_benefit: true
stop_condition_met: true
```
1. Put a matching safetensors snapshot and near-lossless plus quantized GGUF The exact-revision BF16 GGUF quality lane compared every prompt but achieved
artifacts below one mounted-drive root, never `/home`. `0.3333` exact match and `0.9471` mean similarity against the Transformers BF16
2. Install or build the pinned `llama-server`, and use `.venv-rocm` when testing reference. V1 requires `0.90` and `0.97`. Quantized Q4_K_M had substantial speed
the Radeon backend. and fit benefits, but the contract explicitly forbids speed from redeeming a
3. Compute each artifact SHA-256 and create a config declaring the same failed near-lossless quality lane.
`source_model_id` and `source_model_revision` for every recipe.
4. Run the command in `commands.txt` with ## Scope of this stop
`MESHNET_ENABLE_REAL_INFERENCE_TESTS=1`; save its JSON report and summary in
this directory, then evaluate it against `performance-contract.json`. The measured baseline is Qwen2.5-0.5B on CPU using a CPU-only llama.cpp build.
5. Only after those results and all quality gates pass may DGR-001 be marked It is not a Radeon, large-model, distributed, or native-shard result. Therefore:
done and DGR-004 consume the baseline.
1. Do not silently mark v1 promoted or weaken its thresholds after observing the
data.
2. Do not let DGR-004 or later runtime stories treat DGR-001 completion as a
positive promotion signal.
3. A human may choose one of these explicit paths:
- stop the native GGUF track as v1 directs;
- diagnose and fix the BF16 runtime divergence, then rerun the exact v1 plan;
- authorize a separately versioned GPU/large-model contract whose scope and
workload are locked before its measurements.
All raw evidence, configuration, artifacts, hashes, and reproduction commands
are in this directory and `README.md`.

179
.scratch/distributed-gguf-runtime/evidence/DGR-001/README.md
View File

@@ -1,62 +1,153 @@
# DGR-001 — Safetensors versus GGUF performance contract # DGR-001 — Safetensors versus GGUF performance contract
Status: **blocked for real evidence; deterministic implementation complete.** Status: **complete; immutable v1 verdict is `stop`.**
No model benchmark is claimed. See `BLOCKED.md` and the explicitly `not-run`
`results.json`.
## What is implemented DGR-001 successfully produced a controlled local-real CPU baseline. Completion
means the experiment and decision contract are durable and verified; it does
**not** mean the native GGUF track is approved to continue. The locked quality
gate failed, so dependent runtime work requires a human decision or a new,
explicitly versioned experiment/contract rather than silently weakening v1.
- `recipe_benchmark.py` is a deterministic measurement core that runs the exact ## Controlled workload
same plan for every recipe and reports TTFT, prefill/decode rates, p50/p95
latency, aggregate throughput, RSS, VRAM, artifact bytes, request failures,
and per-prompt output drift in JSON.
- `recipe_drivers.py` supplies opt-in Transformers/safetensors and whole-model
llama.cpp-server drivers. Real execution requires
`MESHNET_ENABLE_REAL_INFERENCE_TESTS=1`, refuses model paths outside the
declared mounted-drive root, requires a SHA-256 per artifact, records host
facts, and requires the same declared source model and revision for every
recipe.
- `performance_contract.py` separates a near-lossless quality lane from the
quantized performance/fit lane. Quantized drift is advisory; only the quality
lane can establish parity. `performance-contract.json` locks v1 thresholds
and the stop condition before any result exists.
## Files changed - Model: `Qwen/Qwen2.5-0.5B-Instruct`
- Exact source revision: `7ae557604adf67be50417f59c2c2f167def9a775`
- Machine: `fedora`, Linux `7.0.14-101.fc43.x86_64`, 32 logical CPUs
- Device: CPU for every recipe; VRAM is therefore correctly reported as zero
- Runtime reference: Transformers `5.13.0`, PyTorch
`2.10.0+rocm7.13.0a20260513`, BF16 safetensors
- GGUF runtime: llama.cpp version 9991, commit
`e920c523e3b8a0163fe498af5bf90df35ff51d25`
- Workload: three fixed short/medium/long prompts, greedy sampling, 32 output
tokens, three repeats, two warmups, concurrency 1 and 4, 16 CPU threads
- Evidence class: `local-real`
- `packages/node/meshnet_node/recipe_benchmark.py` All artifacts are beneath `/run/media/popov/DATA/llm/`; no model artifact was
- `packages/node/meshnet_node/recipe_drivers.py` created under `/home`.
- `packages/node/meshnet_node/performance_contract.py`
- `tests/test_recipe_benchmark.py`
- This evidence directory.
## Commands and results ## Recipes and exact artifacts
`commands.txt` contains exact commands. Final targeted result: | Recipe | Artifact | SHA-256 |
|---|---|---|
| Transformers BF16 reference | complete mounted Hugging Face snapshot | `e596e9d6205fdc9177569cccd7f8b471b058f66e3630c8e4326d5aad52bd18b6` |
| llama.cpp BF16 quality lane | `Qwen2.5-0.5B-Instruct-7ae5576-BF16.gguf` | `e842fdc35d7f00fda95a54e1b51731ba1d196aea45065cc9f46925fdc1d6f862` |
| llama.cpp Q4_K_M performance/fit lane | `Qwen2.5-0.5B-Instruct-7ae5576-Q4_K_M.gguf` | `a88e3f570e2efeaf06b50df9859db2c70d8646aa3a2c94a14e14d5797a2921a5` |
The snapshot digest covers every sorted relative path, resolved size, and file
byte, so tokenizer/config drift is included. The BF16 GGUF was converted
directly from the exact snapshot while preserving BF16 weights. Q4_K_M was
quantized from an exact-revision F16 conversion with the pinned quantizer.
Runtime validation recomputes every declared digest before model loading.
## Real results
All recipes completed every request with zero failures.
| Metric | Transformers BF16 | llama.cpp BF16 | llama.cpp Q4_K_M |
|---|---:|---:|---:|
| Decode tok/s, c=1 | 46.1 | 88.0 | 170.1 |
| Aggregate decode tok/s, c=4 | 47.1 | 211.4 | 206.4 |
| TTFT p50, c=1 | 37.5 ms | 43.9 ms | 23.8 ms |
| Peak resident memory, c=1 | 1.94 GB | 1.11 GB | 0.54 GB |
| Artifact size | 1.00 GB | 0.99 GB | 0.40 GB |
| Failures | 0 | 0 | 0 |
Against the reference, the eligible Q4_K_M lane measured:
- single-request decode speedup: **3.69×**;
- concurrency-4 aggregate throughput speedup: **4.38×**;
- resident-memory ratio: **0.279×**;
- artifact-size ratio: **0.398×**.
The near-lossless BF16 quality lane compared all three prompts but measured:
- exact match: **0.3333** (v1 requires at least `0.90`);
- mean text similarity: **0.9471** (v1 requires at least `0.97`).
Tokenization and stopping were controlled: every runtime saw the same prompt
token counts and reported 31 post-TTFT decode tokens. The mismatch is genuine
greedy runtime divergence on two prompts, not missing coverage or a text-length
artifact. Therefore `contract-evaluation.json` records:
```text ```text
pytest -q tests/test_recipe_benchmark.py -> 15 passed verdict: stop
python -m compileall -q packages tests -> exit 0 quality_lane_pass: false
git diff --check -> exit 0 speed_benefit: true
fit_benefit: true
stop_condition_met: true
``` ```
The full suite was attempted and is blocked during collection by the unrelated, Thresholds were not changed after observing these results.
pre-existing DGR-002 runtime dependency mismatch:
## Implementation
- `recipe_benchmark.py` provides the runtime-neutral measurement core, true
concurrency, continuous in-flight peak-memory sampling, percentile/throughput
aggregation, failures, and output drift.
- `recipe_drivers.py` provides opt-in Transformers and llama-server drivers,
mounted-drive confinement, exact artifact/runtime verification, equal
device/thread budgets, greedy-only validation, measured host provenance, and
a CPU-only v1 guard until process VRAM can be measured honestly.
- Peak RSS is runtime-scoped: Transformers reports growth above its pre-runtime
Python baseline, while llama.cpp reports its isolated server process tree.
Both are sampled continuously during in-flight requests.
- TTFT uses each runtime's prompt/first-token compute boundary; end-to-end HTTP,
scheduling, and queue overhead remains in latency and `queue_wait_ms`.
- The exact canonical plan SHA-256 locks prompts, model/revision, sampling,
output length, repeats, warmups, and concurrency. The evaluator also requires
equal prompt/decode token counts across recipes.
- llama.cpp's `predicted_n` includes the first token while `predicted_ms` begins
after it; the driver subtracts that token so decode throughput matches the
Transformers inter-token convention.
- `performance_contract.py` rejects wrong plans, synthetic evidence, missing
recipes/concurrency, mixed model revisions, incomplete quality coverage,
failed references, and missing artifact/host provenance.
- Quantized drift remains advisory. Only the near-lossless lane can satisfy the
quality gate, and only performance-fit recipes can earn speed/fit benefits.
## Evidence files
- `performance-contract.json` — immutable v1 thresholds and stop condition
- `benchmark-config.json` — exact real-run plan, drivers, artifacts, and hashes
- `results.json` — raw machine-readable per-request and aggregate evidence
- `results.txt` — human-readable benchmark summary
- `baseline.json` — distilled measurements for later comparison
- `contract-evaluation.json` — fail-closed v1 verdict
- `commands.txt` — reproducible conversion, benchmark, evaluation, and test commands
- `BLOCKED.md` — downstream stop-condition handoff
- `known-unrelated-failure.md` — clean-base reproduction of the tracker race
## Verification
```text ```text
google.protobuf.runtime_version.VersionError: Targeted: 22 passed
gencode 7.35.0 runtime 6.33.6 Full suite: 749 passed, 13 skipped
Earlier cancellation retry matrix, DGR-001: 4/5 passed
Earlier cancellation retry matrix, clean d904c40: 4/5 passed
compileall: passed
git diff --check: passed
Evidence JSON parse/integrity checks: passed
``` ```
This was reproduced from a clean `git archive HEAD` extracted to The full-suite exception is documented in `known-unrelated-failure.md` and
`/tmp/dgr-001-clean`, with the same command and same failure before any satisfies the issue's explicit clean-tree reproduction clause. DGR-001 changes
uncommitted DGR-001 changes were present. No real benchmark command was run no tracker/proxy files.
because the prerequisites in `BLOCKED.md` are absent.
## Compatibility and handoff The earlier Ralph claim that the full suite was blocked by Protobuf 6.33.6 was
invalid: it used Hermes Agent's internal venv. Verification above used the
project `.venv`, which has the DGR-002-compatible runtime. Real inference used
`.venv-rocm` Python 3.12.
This is additive: it does not alter the current Transformers route, Tracker, ## Limitations and dependent-story handoff
relay, or native protocol. DGR-014 must load `performance-contract.json`, run
the same controlled plan at concurrency 1 and 4, and make only its - This is a **0.5B CPU baseline**, not evidence for a large model, Radeon GPU,
promote/optimize/stop recommendation from a `local-real` or distributed execution, network transport, or native shard worker.
`multi-machine-real` report. DGR-004 remains blocked on this story's real - The installed llama.cpp build is CPU-only (`GGML_HIP=OFF`). No GPU comparison
baseline decision. is claimed.
- Absolute timings are developer-machine measurements; locked ratios and raw
artifacts are provided for reproducibility.
- DGR-014 may consume v1 only with the exact plan/evidence requirements enforced
by `performance_contract.py`.
- DGR-004 and later native-runtime work must not treat DGR-001 completion as a
promotion. V1 says `stop`; proceeding requires a human decision backed by a
separately versioned GPU/large-model contract or a diagnosed quality fix.

122
.scratch/distributed-gguf-runtime/evidence/DGR-001/baseline.json Normal file
View File

@@ -0,0 +1,122 @@
{
"evidence_class": "local-real",
"host": {
"accelerator_name": "Radeon 8060S Graphics",
"accelerator_runtime": "7.13.26183",
"benchmark_lane": "cpu-controlled-baseline",
"converter_sha256": "c819f18fb22927b49fabc3b35d1c9e21ee638b3817eccd1bd4efbcc7116eeb4d",
"cpu_count": 32,
"cuda_available": true,
"hostname": "fedora",
"llama_cpp_commit": "e920c523e3b8a0163fe498af5bf90df35ff51d25",
"llama_cpp_version": "9991",
"llama_server_sha256": "fd8fe612970f23e447f2e717cfa51665be06b8d7315ba60556e010f6bca510dd",
"platform": "Linux-7.0.14-101.fc43.x86_64-x86_64-with-glibc2.42",
"python": "3.12.13",
"quantizer_sha256": "bd0cc8c7be6d48aad4755b31062e0e59a887cbadd43dbb8771853d5858bb198f",
"torch_version": "2.10.0+rocm7.13.0a20260513",
"transformers_version": "5.13.0"
},
"model_id": "Qwen/Qwen2.5-0.5B-Instruct",
"model_revision": "7ae557604adf67be50417f59c2c2f167def9a775",
"plan_sha256": "efe24690a9a7164bac6ab3fd0a6b22f078fc08aaefcfb96210ddf154e6050570",
"recipes": {
"llama-cpp-near-lossless-quality": {
"artifact_bytes": 994156448,
"available": true,
"concurrency": {
"1": {
"aggregate_decode_tokens_per_sec": 73.7861,
"decode_tokens_per_sec": 87.9728,
"failures": 0,
"latency_p50_ms": 385.2049,
"latency_p95_ms": 560.2939,
"peak_rss_bytes": 1110708224,
"peak_vram_bytes": 0,
"prefill_tokens_per_sec": 1427.2072,
"ttft_p50_ms": 43.929,
"ttft_p95_ms": 107.003
},
"4": {
"aggregate_decode_tokens_per_sec": 211.3515,
"decode_tokens_per_sec": 73.8932,
"failures": 0,
"latency_p50_ms": 467.5094,
"latency_p95_ms": 790.862,
"peak_rss_bytes": 1129578496,
"peak_vram_bytes": 0,
"prefill_tokens_per_sec": 1077.8162,
"ttft_p50_ms": 33.612,
"ttft_p95_ms": 128.501
}
},
"device": "cpu",
"lane": "quality"
},
"llama-cpp-quantized-performance-fit": {
"artifact_bytes": 397807520,
"available": true,
"concurrency": {
"1": {
"aggregate_decode_tokens_per_sec": 110.0458,
"decode_tokens_per_sec": 170.131,
"failures": 0,
"latency_p50_ms": 258.0681,
"latency_p95_ms": 465.8523,
"peak_rss_bytes": 542167040,
"peak_vram_bytes": 0,
"prefill_tokens_per_sec": 783.3775,
"ttft_p50_ms": 23.847,
"ttft_p95_ms": 237.696
},
"4": {
"aggregate_decode_tokens_per_sec": 206.377,
"decode_tokens_per_sec": 83.543,
"failures": 0,
"latency_p50_ms": 413.3897,
"latency_p95_ms": 910.253,
"peak_rss_bytes": 572788736,
"peak_vram_bytes": 0,
"prefill_tokens_per_sec": 474.3116,
"ttft_p50_ms": 67.945,
"ttft_p95_ms": 431.804
}
},
"device": "cpu",
"lane": "performance-fit"
},
"transformers-safetensors-reference": {
"artifact_bytes": 999586347,
"available": true,
"concurrency": {
"1": {
"aggregate_decode_tokens_per_sec": 40.3425,
"decode_tokens_per_sec": 46.1451,
"failures": 0,
"latency_p50_ms": 795.4807,
"latency_p95_ms": 930.9725,
"peak_rss_bytes": 1941213184,
"peak_vram_bytes": 0,
"prefill_tokens_per_sec": 671.8016,
"ttft_p50_ms": 37.4548,
"ttft_p95_ms": 193.4633
},
"4": {
"aggregate_decode_tokens_per_sec": 47.0903,
"decode_tokens_per_sec": 13.1337,
"failures": 0,
"latency_p50_ms": 2631.0031,
"latency_p95_ms": 3073.7389,
"peak_rss_bytes": 2177265664,
"peak_vram_bytes": 0,
"prefill_tokens_per_sec": 247.5617,
"ttft_p50_ms": 94.3995,
"ttft_p95_ms": 444.6749
}
},
"device": "cpu",
"lane": "quality"
}
},
"reference_recipe_id": "transformers-safetensors-reference"
}

118
.scratch/distributed-gguf-runtime/evidence/DGR-001/benchmark-config.json Normal file
View File

@@ -0,0 +1,118 @@
{
"artifact_storage_root": "/run/media/popov/DATA/llm",
"evidence_class": "local-real",
"host": {
"benchmark_lane": "cpu-controlled-baseline",
"llama_cpp_commit": "e920c523e3b8a0163fe498af5bf90df35ff51d25",
"llama_cpp_version": "9991",
"llama_server_sha256": "fd8fe612970f23e447f2e717cfa51665be06b8d7315ba60556e010f6bca510dd",
"converter_sha256": "c819f18fb22927b49fabc3b35d1c9e21ee638b3817eccd1bd4efbcc7116eeb4d",
"quantizer_sha256": "bd0cc8c7be6d48aad4755b31062e0e59a887cbadd43dbb8771853d5858bb198f",
"transformers_version": "5.13.0"
},
"plan": {
"plan_id": "dgr-001-controlled-whole-model-baseline-v1",
"model_id": "Qwen/Qwen2.5-0.5B-Instruct",
"model_revision": "7ae557604adf67be50417f59c2c2f167def9a775",
"prompts": [
{
"id": "short-fact",
"text": "The capital of France is",
"context_class": "short"
},
{
"id": "medium-code",
"text": "Complete this Python function without commentary:\n\ndef fibonacci(n):\n \"\"\"Return the nth Fibonacci number for n >= 0.\"\"\"\n",
"context_class": "medium"
},
{
"id": "long-summary",
"text": "A distributed inference service divides a transformer across consumer machines. The tracker owns admission, routing, cancellation, accounting, and telemetry, while workers own only model execution. Every request carries an immutable model identity and revision. Workers must reject incompatible protocol versions and resource demands before allocating large buffers. Activation tensors are chunked, checksummed, bounded by negotiated limits, and propagated with explicit flow-control credits. A caller may disconnect at any time, so cancellation must release queued work, in-flight transfers, and cache reservations without double billing. Retries can occur after network failures, requiring idempotent request identifiers and deterministic completion accounting. The system keeps the existing safetensors path as a correctness reference while a native GGUF path is measured. Benchmarks compare the same prompts, output lengths, sampling policy, device, and concurrency, and they separate near-lossless quality checks from quantized speed and fit claims. Summarize the design priorities in three concise bullet points.",
"context_class": "long"
}
],
"sampling": {
"temperature": 0.0,
"top_p": 1.0,
"top_k": 1,
"seed": 1234,
"max_output_tokens": 32
},
"concurrency_levels": [1, 4],
"repeats": 3,
"warmup_requests": 2
},
"recipes": [
{
"id": "transformers-safetensors-reference",
"runtime": "transformers-5.13.0",
"weight_format": "safetensors",
"weight_quantization": "bfloat16",
"lane": "quality",
"device": "cpu",
"artifact_path": "/run/media/popov/DATA/llm/safetensor/models/models--Qwen--Qwen2.5-0.5B-Instruct/snapshots/7ae557604adf67be50417f59c2c2f167def9a775",
"artifact_sha256": "e596e9d6205fdc9177569cccd7f8b471b058f66e3630c8e4326d5aad52bd18b6",
"source_model_id": "Qwen/Qwen2.5-0.5B-Instruct",
"source_model_revision": "7ae557604adf67be50417f59c2c2f167def9a775",
"is_reference": true,
"notes": "artifact_sha256 is the deterministic digest of every snapshot path and file byte",
"driver": {
"type": "transformers",
"model_path": "/run/media/popov/DATA/llm/safetensor/models/models--Qwen--Qwen2.5-0.5B-Instruct/snapshots/7ae557604adf67be50417f59c2c2f167def9a775",
"device": "cpu",
"dtype": "bfloat16",
"threads": 16
}
},
{
"id": "llama-cpp-near-lossless-quality",
"runtime": "llama.cpp-9991-e920c523",
"weight_format": "gguf",
"weight_quantization": "bfloat16",
"lane": "quality",
"device": "cpu",
"artifact_path": "/run/media/popov/DATA/llm/dgr-001/Qwen2.5-0.5B-Instruct-7ae5576-BF16.gguf",
"artifact_sha256": "e842fdc35d7f00fda95a54e1b51731ba1d196aea45065cc9f46925fdc1d6f862",
"source_model_id": "Qwen/Qwen2.5-0.5B-Instruct",
"source_model_revision": "7ae557604adf67be50417f59c2c2f167def9a775",
"is_reference": false,
"notes": "Converted directly from the exact mounted safetensors revision while preserving BF16 weights with pinned llama.cpp",
"driver": {
"type": "llama-cpp-server",
"binary": "/run/media/popov/d/DEV/llamacpp/llama.cpp/build/bin/llama-server",
"binary_sha256": "fd8fe612970f23e447f2e717cfa51665be06b8d7315ba60556e010f6bca510dd",
"gguf_path": "/run/media/popov/DATA/llm/dgr-001/Qwen2.5-0.5B-Instruct-7ae5576-BF16.gguf",
"device": "cpu",
"threads": 16,
"n_parallel": 4,
"context_per_slot": 512,
"n_gpu_layers": 0
}
},
{
"id": "llama-cpp-quantized-performance-fit",
"runtime": "llama.cpp-9991-e920c523",
"weight_format": "gguf",
"weight_quantization": "Q4_K_M",
"lane": "performance-fit",
"device": "cpu",
"artifact_path": "/run/media/popov/DATA/llm/dgr-001/Qwen2.5-0.5B-Instruct-7ae5576-Q4_K_M.gguf",
"artifact_sha256": "a88e3f570e2efeaf06b50df9859db2c70d8646aa3a2c94a14e14d5797a2921a5",
"source_model_id": "Qwen/Qwen2.5-0.5B-Instruct",
"source_model_revision": "7ae557604adf67be50417f59c2c2f167def9a775",
"is_reference": false,
"notes": "Quantized from the exact-revision F16 GGUF with pinned llama-quantize",
"driver": {
"type": "llama-cpp-server",
"binary": "/run/media/popov/d/DEV/llamacpp/llama.cpp/build/bin/llama-server",
"binary_sha256": "fd8fe612970f23e447f2e717cfa51665be06b8d7315ba60556e010f6bca510dd",
"gguf_path": "/run/media/popov/DATA/llm/dgr-001/Qwen2.5-0.5B-Instruct-7ae5576-Q4_K_M.gguf",
"device": "cpu",
"threads": 16,
"n_parallel": 4,
"context_per_slot": 512,
"n_gpu_layers": 0
}
}
]
}

60
.scratch/distributed-gguf-runtime/evidence/DGR-001/commands.txt
View File

@@ -1,11 +1,51 @@
# Deterministic implementation checks completed in this worktree # Exact source snapshot (already present on mounted storage)
PYTHONPATH=packages/node /home/popov/.hermes/hermes-agent/venv/bin/python -m pytest -q tests/test_recipe_benchmark.py SOURCE=/run/media/popov/DATA/llm/safetensor/models/models--Qwen--Qwen2.5-0.5B-Instruct/snapshots/7ae557604adf67be50417f59c2c2f167def9a775
PYTHONPATH=packages/node /home/popov/.hermes/hermes-agent/venv/bin/python -m compileall -q packages tests LLAMA=/run/media/popov/d/DEV/llamacpp/llama.cpp
ROCM_PY=/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv-rocm/bin/python
PROJECT_PY=/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python
OUT=/run/media/popov/DATA/llm/dgr-001
# Converter support check (no writes)
$ROCM_PY $LLAMA/convert_hf_to_gguf.py "$SOURCE" --outtype f16 --outfile "$OUT/Qwen2.5-0.5B-Instruct-7ae5576-F16.gguf" --dry-run
# Exact-revision near-lossless and performance-fit artifacts
$ROCM_PY $LLAMA/convert_hf_to_gguf.py "$SOURCE" --outtype f16 --outfile "$OUT/Qwen2.5-0.5B-Instruct-7ae5576-F16.gguf"
$LLAMA/build/bin/llama-quantize "$OUT/Qwen2.5-0.5B-Instruct-7ae5576-F16.gguf" "$OUT/Qwen2.5-0.5B-Instruct-7ae5576-Q4_K_M.gguf" Q4_K_M
$ROCM_PY $LLAMA/convert_hf_to_gguf.py "$SOURCE" --outtype bf16 --outfile "$OUT/Qwen2.5-0.5B-Instruct-7ae5576-BF16.gguf"
# Runtime and artifact identity
git -C "$LLAMA" rev-parse HEAD
$LLAMA/build/bin/llama-server --version
sha256sum "$LLAMA/build/bin/llama-server" "$LLAMA/convert_hf_to_gguf.py" "$LLAMA/build/bin/llama-quantize"
sha256sum "$SOURCE/model.safetensors" "$OUT/Qwen2.5-0.5B-Instruct-7ae5576-BF16.gguf" "$OUT/Qwen2.5-0.5B-Instruct-7ae5576-Q4_K_M.gguf"
# Deterministic complete-snapshot digest used by benchmark-config.json
PYTHONPATH=packages/node $ROCM_PY - <<'PY'
from pathlib import Path
from meshnet_node.recipe_drivers import _artifact_sha256
print(_artifact_sha256(Path('/run/media/popov/DATA/llm/safetensor/models/models--Qwen--Qwen2.5-0.5B-Instruct/snapshots/7ae557604adf67be50417f59c2c2f167def9a775')))
PY
# Canonical opt-in local-real benchmark
MESHNET_ENABLE_REAL_INFERENCE_TESTS=1 PYTHONPATH=packages/node $ROCM_PY -m meshnet_node.recipe_benchmark \
--config .scratch/distributed-gguf-runtime/evidence/DGR-001/benchmark-config.json \
--json-out .scratch/distributed-gguf-runtime/evidence/DGR-001/results.json \
--summary-out .scratch/distributed-gguf-runtime/evidence/DGR-001/results.txt
# Distil the baseline and evaluate immutable v1
PYTHONPATH=packages/node $PROJECT_PY - <<'PY'
from pathlib import Path
import json
from meshnet_node.performance_contract import baseline_from_report, evaluate_contract, load_contract
root = Path('.scratch/distributed-gguf-runtime/evidence/DGR-001')
report = json.loads((root / 'results.json').read_text())
contract = load_contract(root / 'performance-contract.json')
(root / 'baseline.json').write_text(json.dumps(baseline_from_report(report), indent=2, sort_keys=True) + '\n')
(root / 'contract-evaluation.json').write_text(json.dumps(evaluate_contract(contract, report).to_dict(), indent=2, sort_keys=True) + '\n')
PY
# Deterministic verification
PYTHONPATH=packages/node $PROJECT_PY -m pytest -q tests/test_recipe_benchmark.py
PYTHONPATH=packages/node $PROJECT_PY -m pytest -q
PYTHONPATH=packages/node $PROJECT_PY -m compileall -q packages tests
git diff --check git diff --check
# Full suite attempted in this worktree and a clean HEAD archive; both stop at
# protobuf gencode 7.35.0 versus installed runtime 6.33.6 during collection.
PYTHONPATH=packages/node /home/popov/.hermes/hermes-agent/venv/bin/python -m pytest -q
# Required opt-in real benchmark after the BLOCKED.md prerequisites exist
MESHNET_ENABLE_REAL_INFERENCE_TESTS=1 PYTHONPATH=packages/node python -m meshnet_node.recipe_benchmark --config /run/media/popov/DATA/meshnet/dgr-001-benchmark.json --json-out .scratch/distributed-gguf-runtime/evidence/DGR-001/results.json --summary-out .scratch/distributed-gguf-runtime/evidence/DGR-001/results.txt

71
.scratch/distributed-gguf-runtime/evidence/DGR-001/contract-evaluation.json Normal file
View File

@@ -0,0 +1,71 @@
{
"contract_version": 1,
"fit_benefit": true,
"plan_id": "dgr-001-controlled-whole-model-baseline-v1",
"quality_lane_pass": false,
"rationale": [
"the near-lossless quality lane failed: the GGUF runtime disagrees with the safetensors reference beyond what near-lossless weights can explain",
"a meaningful speed benefit was measured",
"a meaningful fit benefit was measured"
],
"recipes": [
{
"comparable": true,
"failures": 0,
"fit_benefit": false,
"incomparable_reason": "",
"lane": "quality",
"measurements": {
"aggregate_concurrency": 4,
"aggregate_throughput_speedup": 4.4882,
"artifact_size_ratio": 0.9946,
"artifact_size_win": false,
"compared_prompts": 3,
"decode_speedup": 1.9064,
"exact_match_rate": 0.3333,
"expected_prompts": 3,
"failure_rate": 0.0,
"mean_similarity": 0.9471,
"resident_memory_ratio": 0.5722,
"ttft_ratio": 1.1729
},
"quality_pass": false,
"reasons": [
"single-request decode 1.91x reference (>= 1.25x) at TTFT ratio 1.17",
"aggregate throughput at concurrency 4 is 4.49x reference (>= 1.25x)",
"peak resident memory is 0.57x reference (<= 0.75x)",
"quality lane exact-match 0.33 / similarity 0.947 versus the reference (fail)"
],
"recipe_id": "llama-cpp-near-lossless-quality",
"speed_benefit": false
},
{
"comparable": true,
"failures": 0,
"fit_benefit": true,
"incomparable_reason": "",
"lane": "performance-fit",
"measurements": {
"aggregate_concurrency": 4,
"aggregate_throughput_speedup": 4.3826,
"artifact_size_ratio": 0.398,
"artifact_size_win": true,
"decode_speedup": 3.6869,
"failure_rate": 0.0,
"resident_memory_ratio": 0.2793,
"ttft_ratio": 0.6367
},
"quality_pass": null,
"reasons": [
"single-request decode 3.69x reference (>= 1.25x) at TTFT ratio 0.64",
"aggregate throughput at concurrency 4 is 4.38x reference (>= 1.25x)",
"peak resident memory is 0.28x reference (<= 0.75x)"
],
"recipe_id": "llama-cpp-quantized-performance-fit",
"speed_benefit": true
}
],
"speed_benefit": true,
"stop_condition_met": true,
"verdict": "stop"
}

32
.scratch/distributed-gguf-runtime/evidence/DGR-001/known-unrelated-failure.md Normal file
View File

@@ -0,0 +1,32 @@
# Observed pre-existing intermittent tracker race
This file records an earlier unrelated timing observation; it is **not** the
final DGR-001 verification result.
Test:
```text
tests/test_tracker_routing.py::test_tracker_dashboard_can_cancel_inflight_proxy
```
One earlier full-suite run produced:
```text
1 failed, 745 passed, 13 skipped
```
A five-run isolated retry matrix reproduced the same rate on both branches:
```text
DGR-001 branch: 4/5 passed, 1/5 failed
clean d904c40: 4/5 passed, 1/5 failed
```
The final full-suite run on the exact hardened DGR-001 state completed green:
```text
749 passed, 13 skipped in 251.42s
```
The earlier race was therefore timing-sensitive, pre-existing, and unrelated
to the DGR-001 benchmark/contract files.

14
.scratch/distributed-gguf-runtime/evidence/DGR-001/performance-contract.json
View File

@@ -22,15 +22,23 @@
"llama-cpp-near-lossless-quality", "llama-cpp-near-lossless-quality",
"llama-cpp-quantized-performance-fit" "llama-cpp-quantized-performance-fit"
], ],
"required_concurrency_levels": [1, 4], "required_concurrency_levels": [
1,
4
],
"required_controlled_variables": [ "required_controlled_variables": [
"model architecture", "model architecture",
"model revision", "model revision",
"machine and device", "machine and device",
"formatted prompts and context lengths", "formatted prompts and context lengths",
"output length and greedy sampling policy" "output length and greedy sampling policy"
] ],
"required_plan_sha256": "efe24690a9a7164bac6ab3fd0a6b22f078fc08aaefcfb96210ddf154e6050570",
"minimum_prompt_count": 3,
"minimum_repeats": 3,
"minimum_output_tokens": 32,
"required_device": "cpu"
}, },
"stop_condition": "Stop the native llama.cpp/GGUF track when, on the same machine and device as the Transformers/safetensors reference and under this plan, no performance-fit GGUF recipe delivers either a meaningful speed benefit (at least 25% higher single-request decode tokens/sec without more than 25% worse TTFT, or at least 25% higher aggregate throughput under concurrency) or a meaningful fit benefit (at least 25% lower peak resident memory), or when the near-lossless quality lane fails.", "stop_condition": "Stop the native llama.cpp/GGUF track when, on the same machine and device as the Transformers/safetensors reference and under this plan, no performance-fit GGUF recipe delivers either a meaningful speed benefit (>=25% higher single-request decode tokens/sec without a >25% worse TTFT, or >=25% higher aggregate throughput under concurrency) or a meaningful fit benefit (>=25% lower peak resident memory), or when the near-lossless quality lane fails, which indicates a broken runtime rather than a quantization trade-off.",
"notes": "Quantized performance-fit output drift is reported as advisory only. It is not numerical-equivalence evidence. DGR-014 consumes this immutable v1 contract." "notes": "Quantized performance-fit output drift is reported as advisory only. It is not numerical-equivalence evidence. DGR-014 consumes this immutable v1 contract."
} }

2478
.scratch/distributed-gguf-runtime/evidence/DGR-001/results.json
View File

File diff suppressed because it is too large Load Diff

10
.scratch/distributed-gguf-runtime/evidence/DGR-001/results.txt Normal file
View File

@@ -0,0 +1,10 @@
Recipe benchmark dgr-001-controlled-whole-model-baseline-v1 (local-real)
model Qwen/Qwen2.5-0.5B-Instruct@7ae557604adf67be50417f59c2c2f167def9a775
transformers-safetensors-reference [quality ] c= 1 ttft p50/p95 37.5/ 193.5 ms; prefill 671.8 tok/s; decode 46.1 tok/s; aggregate 40.3 tok/s; rss 1.94 GB; vram 0.00 GB; artifact 1.00 GB; failures 0
transformers-safetensors-reference [quality ] c= 4 ttft p50/p95 94.4/ 444.7 ms; prefill 247.6 tok/s; decode 13.1 tok/s; aggregate 47.1 tok/s; rss 2.18 GB; vram 0.00 GB; artifact 1.00 GB; failures 0
llama-cpp-near-lossless-quality [quality ] c= 1 ttft p50/p95 43.9/ 107.0 ms; prefill 1427.2 tok/s; decode 88.0 tok/s; aggregate 73.8 tok/s; rss 1.11 GB; vram 0.00 GB; artifact 0.99 GB; failures 0
llama-cpp-near-lossless-quality [quality ] c= 4 ttft p50/p95 33.6/ 128.5 ms; prefill 1077.8 tok/s; decode 73.9 tok/s; aggregate 211.4 tok/s; rss 1.13 GB; vram 0.00 GB; artifact 0.99 GB; failures 0
llama-cpp-quantized-performance-fit [performance-fit ] c= 1 ttft p50/p95 23.8/ 237.7 ms; prefill 783.4 tok/s; decode 170.1 tok/s; aggregate 110.0 tok/s; rss 0.54 GB; vram 0.00 GB; artifact 0.40 GB; failures 0
llama-cpp-quantized-performance-fit [performance-fit ] c= 4 ttft p50/p95 67.9/ 431.8 ms; prefill 474.3 tok/s; decode 83.5 tok/s; aggregate 206.4 tok/s; rss 0.57 GB; vram 0.00 GB; artifact 0.40 GB; failures 0
drift llama-cpp-near-lossless-quality vs transformers-safetensors-reference exact 0.33; similarity 0.947 (gated)
drift llama-cpp-quantized-performance-fit vs transformers-safetensors-reference exact 0.00; similarity 0.456 (advisory)

2
.scratch/distributed-gguf-runtime/issues/01-lock-the-safetensors-versus-gguf-performance-contract.md
View File

@@ -1,6 +1,6 @@
# 01 — Lock the safetensors-versus-GGUF performance contract # 01 — Lock the safetensors-versus-GGUF performance contract
Status: ready-for-agent Status: done
## Mandatory fresh-session context ## Mandatory fresh-session context

34
.scratch/distributed-gguf-runtime/prd.json
View File

@@ -6,7 +6,7 @@
{ {
"id": "DGR-001", "id": "DGR-001",
"title": "Lock the safetensors-versus-GGUF performance contract", "title": "Lock the safetensors-versus-GGUF performance contract",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/01-lock-the-safetensors-versus-gguf-performance-contract.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs a runtime engineer, I need a controlled baseline so that GGUF work proceeds from measured speed, memory, and quality rather than reputation.", "description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/01-lock-the-safetensors-versus-gguf-performance-contract.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker—not a stitched collection of runtimes.\n\nAs a runtime engineer, I need a controlled baseline so that GGUF work proceeds from measured speed, memory, and quality rather than reputation.",
"acceptanceCriteria": [ "acceptanceCriteria": [
"Benchmark the same model architecture/revision, machine, prompts, context lengths, output lengths, sampling policy, and concurrency across the current Transformers/safetensors recipe and whole-model llama.cpp recipes.", "Benchmark the same model architecture/revision, machine, prompts, context lengths, output lengths, sampling policy, and concurrency across the current Transformers/safetensors recipe and whole-model llama.cpp recipes.",
"Separate correctness/quality lanes from quantized performance/fit lanes instead of claiming BF16 and Q4 are numerically equivalent.", "Separate correctness/quality lanes from quantized performance/fit lanes instead of claiming BF16 and Q4 are numerically equivalent.",
@@ -27,14 +27,14 @@
"Update only this story issue to Status: done after every acceptance criterion and quality gate passes" "Update only this story issue to Status: done after every acceptance criterion and quality gate passes"
], ],
"priority": 2, "priority": 2,
"passes": false, "passes": true,
"notes": "Source issue: .scratch/distributed-gguf-runtime/issues/01-lock-the-safetensors-versus-gguf-performance-contract.md", "notes": "Source issue: .scratch/distributed-gguf-runtime/issues/01-lock-the-safetensors-versus-gguf-performance-contract.md",
"dependsOn": [] "dependsOn": []
}, },
{ {
"id": "DGR-002", "id": "DGR-002",
"title": "Adopt the versioned gRPC Shard protocol", "title": "Adopt the versioned gRPC Shard protocol",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/02-adopt-the-versioned-grpc-shard-protocol.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs a node developer, I need a battle-proven streaming protocol so that Python and C++ Shards communicate without a custom socket protocol.", "description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/02-adopt-the-versioned-grpc-shard-protocol.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker—not a stitched collection of runtimes.\n\nAs a node developer, I need a battle-proven streaming protocol so that Python and C++ Shards communicate without a custom socket protocol.",
"acceptanceCriteria": [ "acceptanceCriteria": [
"Add a Protocol Buffers schema for capability, health, session stream, release, and cancellation operations.", "Add a Protocol Buffers schema for capability, health, session stream, release, and cancellation operations.",
"Define one long-lived bidirectional gRPC stream per Route Session Activation Seam with deadlines, cancellation, flow control, and structured errors.", "Define one long-lived bidirectional gRPC stream per Route Session Activation Seam with deadlines, cancellation, flow control, and structured errors.",
@@ -61,7 +61,7 @@
{ {
"id": "DGR-003", "id": "DGR-003",
"title": "Define exact Artifact and runtime recipe identity", "title": "Define exact Artifact and runtime recipe identity",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/03-define-exact-artifact-and-runtime-recipe-identity.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs the Tracker, I need exact compatibility identity so that only numerically and operationally compatible Shards form an Inference Route.", "description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/03-define-exact-artifact-and-runtime-recipe-identity.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker—not a stitched collection of runtimes.\n\nAs the Tracker, I need exact compatibility identity so that only numerically and operationally compatible Shards form an Inference Route.",
"acceptanceCriteria": [ "acceptanceCriteria": [
"Separate weight quantization, activation dtype, compute dtype, KV dtype/layout, tokenizer revision, architecture adapter, backend, and runtime version.", "Separate weight quantization, activation dtype, compute dtype, KV dtype/layout, tokenizer revision, architecture adapter, backend, and runtime version.",
"Bind derivative or split artifacts to an exact source Model Artifact hash and Shard range.", "Bind derivative or split artifacts to an exact source Model Artifact hash and Shard range.",
@@ -89,7 +89,7 @@
{ {
"id": "DGR-004", "id": "DGR-004",
"title": "Create the reproducible pinned llama.cpp patch stack", "title": "Create the reproducible pinned llama.cpp patch stack",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/04-create-the-reproducible-pinned-llama-cpp-patch-stack.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs a maintainer, I need a small auditable fork boundary so that upstream updates do not turn the runtime into an unmaintainable stitched codebase.", "description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/04-create-the-reproducible-pinned-llama-cpp-patch-stack.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker—not a stitched collection of runtimes.\n\nAs a maintainer, I need a small auditable fork boundary so that upstream updates do not turn the runtime into an unmaintainable stitched codebase.",
"acceptanceCriteria": [ "acceptanceCriteria": [
"Pin one exact llama.cpp commit through a reproducible source dependency mechanism.", "Pin one exact llama.cpp commit through a reproducible source dependency mechanism.",
"Store a numbered minimal patch stack separately from Meshnet networking code.", "Store a numbered minimal patch stack separately from Meshnet networking code.",
@@ -120,7 +120,7 @@
{ {
"id": "DGR-005", "id": "DGR-005",
"title": "Implement dense-Llama range-aware GGUF ownership", "title": "Implement dense-Llama range-aware GGUF ownership",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/05-implement-dense-llama-range-aware-gguf-ownership.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs a node, I need to map only my assigned dense-Llama Shard so that aggregate consumer memory can hold a model larger than one node.", "description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/05-implement-dense-llama-range-aware-gguf-ownership.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker—not a stitched collection of runtimes.\n\nAs a node, I need to map only my assigned dense-Llama Shard so that aggregate consumer memory can hold a model larger than one node.",
"acceptanceCriteria": [ "acceptanceCriteria": [
"Register and allocate only `blk.N.*` tensors in the assigned range.", "Register and allocate only `blk.N.*` tensors in the assigned range.",
"Load embeddings only for the head and final norm/LM head only for the tail, including tied embeddings.", "Load embeddings only for the head and final norm/LM head only for the tail, including tied embeddings.",
@@ -151,7 +151,7 @@
{ {
"id": "DGR-006", "id": "DGR-006",
"title": "Implement architecture-defined boundary input/output", "title": "Implement architecture-defined boundary input/output",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/06-implement-architecture-defined-boundary-input-output.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs a Shard, I need to consume and emit the correct transformer boundary state so that disjoint processes reproduce whole-model execution.", "description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/06-implement-architecture-defined-boundary-input-output.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker—not a stitched collection of runtimes.\n\nAs a Shard, I need to consume and emit the correct transformer boundary state so that disjoint processes reproduce whole-model execution.",
"acceptanceCriteria": [ "acceptanceCriteria": [
"Head accepts token IDs and owns token embedding.", "Head accepts token IDs and owns token embedding.",
"Middle/tail bypass token embedding and accept the named boundary bundle.", "Middle/tail bypass token embedding and accept the named boundary bundle.",
@@ -183,7 +183,7 @@
{ {
"id": "DGR-007", "id": "DGR-007",
"title": "Add isolated concurrent local Hot KV State", "title": "Add isolated concurrent local Hot KV State",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/07-add-isolated-concurrent-local-hot-kv-state.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs a client, I need concurrent Route Sessions to retain independent per-Shard cache so that one request cannot clear or corrupt another.", "description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/07-add-isolated-concurrent-local-hot-kv-state.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker—not a stitched collection of runtimes.\n\nAs a client, I need concurrent Route Sessions to retain independent per-Shard cache so that one request cannot clear or corrupt another.",
"acceptanceCriteria": [ "acceptanceCriteria": [
"Map `(Route Session ID, route epoch)` to an isolated llama sequence or bounded context.", "Map `(Route Session ID, route epoch)` to an isolated llama sequence or bounded context.",
"Allocate KV only for owned layers.", "Allocate KV only for owned layers.",
@@ -214,7 +214,7 @@
{ {
"id": "DGR-008", "id": "DGR-008",
"title": "Build the standalone C++ gRPC Shard worker", "title": "Build the standalone C++ gRPC Shard worker",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/08-build-the-standalone-c-grpc-shard-worker.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs a node runtime, I need one supervised native process so that llama.cpp internals remain behind a stable project-owned protocol.", "description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/08-build-the-standalone-c-grpc-shard-worker.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker—not a stitched collection of runtimes.\n\nAs a node runtime, I need one supervised native process so that llama.cpp internals remain behind a stable project-owned protocol.",
"acceptanceCriteria": [ "acceptanceCriteria": [
"Worker exposes capability, health, session stream, release, cancellation, and metrics services from DGR-002.", "Worker exposes capability, health, session stream, release, cancellation, and metrics services from DGR-002.",
"Worker loads one exact Artifact/recipe/Shard identity and refuses mismatched requests.", "Worker loads one exact Artifact/recipe/Shard identity and refuses mismatched requests.",
@@ -249,7 +249,7 @@
{ {
"id": "DGR-009", "id": "DGR-009",
"title": "Integrate the native worker with Meshnet", "title": "Integrate the native worker with Meshnet",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/09-integrate-the-native-worker-with-meshnet.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs the existing node service, I need a GGUF Shard backend adapter so that the Tracker, relay, billing, telemetry, and capability admission remain the sole control plane.", "description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/09-integrate-the-native-worker-with-meshnet.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker—not a stitched collection of runtimes.\n\nAs the existing node service, I need a GGUF Shard backend adapter so that the Tracker, relay, billing, telemetry, and capability admission remain the sole control plane.",
"acceptanceCriteria": [ "acceptanceCriteria": [
"Implement the existing model-backend surface without changing Transformers behavior.", "Implement the existing model-backend surface without changing Transformers behavior.",
"Registration carries exact validated GGUF recipe, Shard, backend and concurrency/KV capacity.", "Registration carries exact validated GGUF recipe, Shard, backend and concurrency/KV capacity.",
@@ -281,7 +281,7 @@
{ {
"id": "DGR-010", "id": "DGR-010",
"title": "Pass local real-model two-process acceptance", "title": "Pass local real-model two-process acceptance",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/10-pass-local-real-model-two-process-acceptance.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs a release engineer, I need real local distributed parity before involving network variability.", "description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/10-pass-local-real-model-two-process-acceptance.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker—not a stitched collection of runtimes.\n\nAs a release engineer, I need real local distributed parity before involving network variability.",
"acceptanceCriteria": [ "acceptanceCriteria": [
"Two local worker processes open disjoint dense-Llama ranges from the certified Artifact.", "Two local worker processes open disjoint dense-Llama ranges from the certified Artifact.",
"Prefill and at least 32 greedy decode tokens match whole-model llama.cpp within the certified tolerance.", "Prefill and at least 32 greedy decode tokens match whole-model llama.cpp within the certified tolerance.",
@@ -314,7 +314,7 @@
{ {
"id": "DGR-011", "id": "DGR-011",
"title": "Pass a real heterogeneous two-machine route", "title": "Pass a real heterogeneous two-machine route",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/11-pass-a-real-heterogeneous-two-machine-route.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs a consumer-hardware operator, I need two physical machines to execute one GGUF model so that the distributed claim is real.", "description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/11-pass-a-real-heterogeneous-two-machine-route.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker—not a stitched collection of runtimes.\n\nAs a consumer-hardware operator, I need two physical machines to execute one GGUF model so that the distributed claim is real.",
"acceptanceCriteria": [ "acceptanceCriteria": [
"Tracker selects two physical nodes with disjoint Shards and one exact certified recipe/compatibility class.", "Tracker selects two physical nodes with disjoint Shards and one exact certified recipe/compatibility class.",
"Actual CPU/GPU execution occurs on both nodes; synthetic workers do not satisfy acceptance.", "Actual CPU/GPU execution occurs on both nodes; synthetic workers do not satisfy acceptance.",
@@ -347,7 +347,7 @@
{ {
"id": "DGR-012", "id": "DGR-012",
"title": "Implement continuous batching and bounded admission", "title": "Implement continuous batching and bounded admission",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/12-implement-continuous-batching-and-bounded-admission.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs a node operator, I need active sessions batched safely so that concurrency increases aggregate throughput rather than serializing every request.", "description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/12-implement-continuous-batching-and-bounded-admission.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker—not a stitched collection of runtimes.\n\nAs a node operator, I need active sessions batched safely so that concurrency increases aggregate throughput rather than serializing every request.",
"acceptanceCriteria": [ "acceptanceCriteria": [
"Node scheduler admits sessions against weight, KV, scratch, and queue budgets.", "Node scheduler admits sessions against weight, KV, scratch, and queue budgets.",
"Compatible decode steps from multiple sessions form llama.cpp batches while preserving per-session positions and outputs.", "Compatible decode steps from multiple sessions form llama.cpp batches while preserving per-session positions and outputs.",
@@ -380,7 +380,7 @@
{ {
"id": "DGR-013", "id": "DGR-013",
"title": "Harden failure, cancellation, and restart semantics", "title": "Harden failure, cancellation, and restart semantics",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/13-harden-failure-cancellation-and-restart-semantics.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs a client, I need failures to be bounded and explicit so that distributed speed does not come with hanging or corrupted generations.", "description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/13-harden-failure-cancellation-and-restart-semantics.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker—not a stitched collection of runtimes.\n\nAs a client, I need failures to be bounded and explicit so that distributed speed does not come with hanging or corrupted generations.",
"acceptanceCriteria": [ "acceptanceCriteria": [
"Deadlines and heartbeat/health loss terminate blocked stream operations.", "Deadlines and heartbeat/health loss terminate blocked stream operations.",
"Cancellation propagates across every Shard and releases local KV and queued buffers.", "Cancellation propagates across every Shard and releases local KV and queued buffers.",
@@ -413,7 +413,7 @@
{ {
"id": "DGR-014", "id": "DGR-014",
"title": "Enforce the GGUF-versus-safetensors release gate", "title": "Enforce the GGUF-versus-safetensors release gate",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/14-enforce-the-gguf-versus-safetensors-release-gate.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs the product owner, I need an end-to-end comparison so that the native runtime ships only if it advances model access or performance.", "description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/14-enforce-the-gguf-versus-safetensors-release-gate.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker—not a stitched collection of runtimes.\n\nAs the product owner, I need an end-to-end comparison so that the native runtime ships only if it advances model access or performance.",
"acceptanceCriteria": [ "acceptanceCriteria": [
"Run current distributed safetensors and distributed GGUF routes on the same certified model/hardware/network scenario where technically comparable.", "Run current distributed safetensors and distributed GGUF routes on the same certified model/hardware/network scenario where technically comparable.",
"Report quality, TTFT, prefill/decode throughput, aggregate concurrency throughput, p95 latency, seam cost, memory, KV pressure, failures, and cleanup.", "Report quality, TTFT, prefill/decode throughput, aggregate concurrency throughput, p95 latency, seam cost, memory, KV pressure, failures, and cleanup.",
@@ -448,7 +448,7 @@
{ {
"id": "DGR-015", "id": "DGR-015",
"title": "Add and certify a Qwen3/Qwen3-MoE adapter", "title": "Add and certify a Qwen3/Qwen3-MoE adapter",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/15-add-and-certify-a-qwen3-qwen3-moe-adapter.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs a client seeking top models, I need a separately certified MoE-capable architecture after the dense runtime proves stable.", "description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/15-add-and-certify-a-qwen3-qwen3-moe-adapter.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker—not a stitched collection of runtimes.\n\nAs a client seeking top models, I need a separately certified MoE-capable architecture after the dense runtime proves stable.",
"acceptanceCriteria": [ "acceptanceCriteria": [
"Implement explicit tensor ownership, router/top-k, expert/shared-expert, Q/K normalization, boundary bundle, and cache semantics for the selected Qwen3 family recipe.", "Implement explicit tensor ownership, router/top-k, expert/shared-expert, Q/K normalization, boundary bundle, and cache semantics for the selected Qwen3 family recipe.",
"Do not reuse the dense-Llama adapter through unchecked name substitutions.", "Do not reuse the dense-Llama adapter through unchecked name substitutions.",
@@ -480,7 +480,7 @@
{ {
"id": "DGR-016", "id": "DGR-016",
"title": "Produce the upstream llama.cpp collaboration package", "title": "Produce the upstream llama.cpp collaboration package",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/16-produce-the-upstream-llama-cpp-collaboration-package.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs a maintainer, I need narrow upstreamable proposals so that our patch burden can shrink without asking llama.cpp to own Meshnet networking.", "description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/16-produce-the-upstream-llama-cpp-collaboration-package.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker—not a stitched collection of runtimes.\n\nAs a maintainer, I need narrow upstreamable proposals so that our patch burden can shrink without asking llama.cpp to own Meshnet networking.",
"acceptanceCriteria": [ "acceptanceCriteria": [
"Separate generic llama.cpp hooks from Meshnet protocol/control-plane code.", "Separate generic llama.cpp hooks from Meshnet protocol/control-plane code.",
"Prepare minimal reproducible examples and tests for range-aware loading, boundary input/output, and layer-filtered KV.", "Prepare minimal reproducible examples and tests for range-aware loading, boundary input/output, and layer-filtered KV.",

198
packages/node/meshnet_node/performance_contract.py
View File

@@ -25,12 +25,13 @@ the release gate is allowed to reach.
from __future__ import annotations from __future__ import annotations
import hashlib
import json import json
from dataclasses import asdict, dataclass from dataclasses import asdict, dataclass
from pathlib import Path from pathlib import Path
from typing import Any, Mapping from typing import Any, Mapping
from .recipe_benchmark import Lane from .recipe_benchmark import Lane, REPORT_SCHEMA_VERSION
# Layout of the contract document understood by this reader. # Layout of the contract document understood by this reader.
CONTRACT_SCHEMA_VERSION = 1 CONTRACT_SCHEMA_VERSION = 1
@@ -140,6 +141,150 @@ def _ratio(value: float, reference: float) -> float:
return round(value / reference, 4) return round(value / reference, 4)
def _canonical_sha256(value: Any) -> str:
payload = json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False)
return hashlib.sha256(payload.encode("utf-8")).hexdigest()
def _validate_report(contract: PerformanceContract, report: Mapping[str, Any]) -> None:
"""Fail closed when a report is not the experiment the contract locked."""
try:
schema_version = report["schema_version"]
evidence_class = report["evidence_class"]
plan = report["plan"]
recipes = report["recipes"]
reference_id = report["reference_recipe_id"]
host = report["host"]
except (KeyError, TypeError) as exc:
raise PerformanceContractError("benchmark report is missing required structure") from exc
if schema_version != REPORT_SCHEMA_VERSION:
raise PerformanceContractError(
f"report schema {schema_version!r} is not supported schema {REPORT_SCHEMA_VERSION}"
)
if plan.get("plan_id") != contract.plan_id:
raise PerformanceContractError(
f"report plan {plan.get('plan_id')!r} does not match locked plan {contract.plan_id!r}"
)
required_plan_sha256 = contract.baseline.get("required_plan_sha256")
measured_plan_sha256 = _canonical_sha256(plan)
if required_plan_sha256 and measured_plan_sha256 != required_plan_sha256:
raise PerformanceContractError(
f"report plan digest {measured_plan_sha256} does not match locked digest "
f"{required_plan_sha256}"
)
minimum_repeats = int(contract.baseline.get("minimum_repeats", 0))
minimum_prompts = int(contract.baseline.get("minimum_prompt_count", 0))
if int(plan.get("repeats", 0)) < minimum_repeats:
raise PerformanceContractError("report has too few repeats for the locked contract")
if len(plan.get("prompts", ())) < minimum_prompts:
raise PerformanceContractError("report has too few prompts for the locked contract")
minimum_output_tokens = int(contract.baseline.get("minimum_output_tokens", 0))
if int(plan.get("sampling", {}).get("max_output_tokens", 0)) < minimum_output_tokens:
raise PerformanceContractError("report output length is below the locked contract")
required_evidence = contract.baseline.get("required_evidence_class")
if required_evidence and evidence_class != required_evidence:
raise PerformanceContractError(
f"report evidence class {evidence_class!r} does not satisfy {required_evidence!r}"
)
if required_evidence and (
not isinstance(host, Mapping)
or any(key not in host for key in ("hostname", "platform", "python", "cpu_count"))
):
raise PerformanceContractError("report lacks measured host provenance")
if not isinstance(recipes, list) or not recipes:
raise PerformanceContractError("report contains no recipes")
recipe_ids = [entry.get("recipe", {}).get("id") for entry in recipes]
if len(set(recipe_ids)) != len(recipe_ids) or None in recipe_ids:
raise PerformanceContractError("report recipe IDs must be present and unique")
required_recipes = set(contract.baseline.get("required_recipes", ()))
missing_recipes = required_recipes - set(recipe_ids)
if missing_recipes:
raise PerformanceContractError(
f"report is missing required recipes {sorted(missing_recipes)}"
)
if reference_id not in recipe_ids:
raise PerformanceContractError("report reference recipe is absent")
levels = {int(level) for level in plan.get("concurrency_levels", ())}
required_levels = {
int(level) for level in contract.baseline.get("required_concurrency_levels", ())
}
if not required_levels.issubset(levels):
raise PerformanceContractError(
f"report concurrency {sorted(levels)} lacks required levels {sorted(required_levels)}"
)
if not plan.get("prompts"):
raise PerformanceContractError("report plan contains no prompts")
model_id = plan.get("model_id")
model_revision = plan.get("model_revision")
required_device = contract.baseline.get("required_device")
for entry in recipes:
recipe = entry.get("recipe", {})
if required_device and recipe.get("device") != required_device:
raise PerformanceContractError(
f"recipe {recipe.get('id')!r} did not run on locked device {required_device!r}"
)
if required_evidence:
if recipe.get("source_model_id") != model_id:
raise PerformanceContractError("report mixes source model IDs")
if recipe.get("source_model_revision") != model_revision:
raise PerformanceContractError("report mixes source model revisions")
digest = recipe.get("artifact_sha256", "")
if not isinstance(digest, str) or len(digest) != 64:
raise PerformanceContractError("report lacks an artifact SHA-256 digest")
if entry.get("available"):
cells = entry.get("concurrency", {})
missing_cells = required_levels - {int(level) for level in cells}
if missing_cells:
raise PerformanceContractError(
f"recipe {recipe.get('id')!r} lacks concurrency cells {sorted(missing_cells)}"
)
reference = next(entry for entry in recipes if entry["recipe"]["id"] == reference_id)
if not reference.get("available"):
raise PerformanceContractError("reference recipe is unavailable")
reference_failures = sum(
int(cell.get("failures", 0)) for cell in reference.get("concurrency", {}).values()
)
if reference_failures:
raise PerformanceContractError("reference recipe contains failed requests")
def token_counts(entry: Mapping[str, Any]) -> dict[tuple[str, int, int], list[tuple[int, int]]]:
counts: dict[tuple[str, int, int], list[tuple[int, int]]] = {}
for outcome in entry.get("outcomes", ()):
if not outcome.get("ok"):
continue
key = (
str(outcome.get("prompt_id")),
int(outcome.get("concurrency", 0)),
int(outcome.get("repeat", -1)),
)
counts.setdefault(key, []).append(
(int(outcome.get("prompt_tokens", 0)), int(outcome.get("decode_tokens", 0)))
)
return {key: sorted(values) for key, values in counts.items()}
reference_counts = token_counts(reference)
prompt_ids = {str(prompt["id"]) for prompt in plan["prompts"]}
repeats = int(plan["repeats"])
for prompt_id in prompt_ids:
for level in required_levels:
for repeat in range(repeats):
values = reference_counts.get((prompt_id, level, repeat), ())
if len(values) != level:
raise PerformanceContractError(
"reference recipe lacks complete prompt/repeat/concurrency coverage"
)
for entry in recipes:
if entry.get("available") and token_counts(entry) != reference_counts:
raise PerformanceContractError(
f"recipe {entry['recipe']['id']!r} used different prompt/decode token counts"
)
@dataclass(frozen=True) @dataclass(frozen=True)
class RecipeEvaluation: class RecipeEvaluation:
"""How one GGUF recipe fared against the reference under the contract.""" """How one GGUF recipe fared against the reference under the contract."""
@@ -196,6 +341,7 @@ def _evaluate_recipe(
drift_by_recipe: Mapping[str, Mapping[str, Any]], drift_by_recipe: Mapping[str, Mapping[str, Any]],
thresholds: ContractThresholds, thresholds: ContractThresholds,
concurrency_levels: list[int], concurrency_levels: list[int],
expected_prompt_count: int,
) -> RecipeEvaluation: ) -> RecipeEvaluation:
recipe = entry["recipe"] recipe = entry["recipe"]
lane = recipe["lane"] lane = recipe["lane"]
@@ -237,7 +383,7 @@ def _evaluate_recipe(
and 0 < ttft_ratio <= thresholds.max_ttft_ratio and 0 < ttft_ratio <= thresholds.max_ttft_ratio
) )
if single_request_win: if single_request_win:
speed_benefit = True speed_benefit = lane == Lane.PERFORMANCE_FIT.value
reasons.append( reasons.append(
f"single-request decode {decode_speedup:.2f}x reference " f"single-request decode {decode_speedup:.2f}x reference "
f"(>= {thresholds.min_decode_speedup:.2f}x) at TTFT ratio {ttft_ratio:.2f}" f"(>= {thresholds.min_decode_speedup:.2f}x) at TTFT ratio {ttft_ratio:.2f}"
@@ -259,7 +405,7 @@ def _evaluate_recipe(
measurements["aggregate_throughput_speedup"] = aggregate_speedup measurements["aggregate_throughput_speedup"] = aggregate_speedup
measurements["aggregate_concurrency"] = top measurements["aggregate_concurrency"] = top
if aggregate_speedup >= thresholds.min_aggregate_throughput_speedup: if aggregate_speedup >= thresholds.min_aggregate_throughput_speedup:
speed_benefit = True speed_benefit = lane == Lane.PERFORMANCE_FIT.value
reasons.append( reasons.append(
f"aggregate throughput at concurrency {top} is {aggregate_speedup:.2f}x reference " f"aggregate throughput at concurrency {top} is {aggregate_speedup:.2f}x reference "
f"(>= {thresholds.min_aggregate_throughput_speedup:.2f}x)" f"(>= {thresholds.min_aggregate_throughput_speedup:.2f}x)"
@@ -279,7 +425,7 @@ def _evaluate_recipe(
measurements["resident_memory_ratio"] = resident_ratio measurements["resident_memory_ratio"] = resident_ratio
measurements["artifact_size_ratio"] = artifact_ratio measurements["artifact_size_ratio"] = artifact_ratio
if 0 < resident_ratio <= thresholds.max_resident_memory_ratio: if 0 < resident_ratio <= thresholds.max_resident_memory_ratio:
fit_benefit = True fit_benefit = lane == Lane.PERFORMANCE_FIT.value
reasons.append( reasons.append(
f"peak resident memory is {resident_ratio:.2f}x reference " f"peak resident memory is {resident_ratio:.2f}x reference "
f"(<= {thresholds.max_resident_memory_ratio:.2f}x)" f"(<= {thresholds.max_resident_memory_ratio:.2f}x)"
@@ -309,12 +455,24 @@ def _evaluate_recipe(
quality_pass = False quality_pass = False
reasons.append("quality-lane recipe has no drift measurement against the reference") reasons.append("quality-lane recipe has no drift measurement against the reference")
else: else:
complete_coverage = drift.get("compared_prompts") == expected_prompt_count
quality_pass = ( quality_pass = (
drift["exact_match_rate"] >= thresholds.min_quality_exact_match_rate complete_coverage
and failures == 0
and drift["exact_match_rate"] >= thresholds.min_quality_exact_match_rate
and drift["mean_similarity"] >= thresholds.min_quality_mean_similarity and drift["mean_similarity"] >= thresholds.min_quality_mean_similarity
) )
measurements["compared_prompts"] = drift.get("compared_prompts", 0)
measurements["expected_prompts"] = expected_prompt_count
measurements["exact_match_rate"] = drift["exact_match_rate"] measurements["exact_match_rate"] = drift["exact_match_rate"]
measurements["mean_similarity"] = drift["mean_similarity"] measurements["mean_similarity"] = drift["mean_similarity"]
if not complete_coverage:
reasons.append(
f"quality lane compared {drift.get('compared_prompts', 0)} of "
f"{expected_prompt_count} required prompts"
)
if failures:
reasons.append("quality lane contains failed requests")
reasons.append( reasons.append(
f"quality lane exact-match {drift['exact_match_rate']:.2f} / similarity " f"quality lane exact-match {drift['exact_match_rate']:.2f} / similarity "
f"{drift['mean_similarity']:.3f} versus the reference " f"{drift['mean_similarity']:.3f} versus the reference "
@@ -339,6 +497,7 @@ def evaluate_contract(
runtime that fails the quality lane is broken, and no amount of speed runtime that fails the quality lane is broken, and no amount of speed
redeems it, so the verdict is ``stop`` regardless of the other numbers. redeems it, so the verdict is ``stop`` regardless of the other numbers.
""" """
_validate_report(contract, report)
entries = _recipe_entries(report) entries = _recipe_entries(report)
reference_id = report["reference_recipe_id"] reference_id = report["reference_recipe_id"]
reference = entries.get(reference_id) reference = entries.get(reference_id)
@@ -351,7 +510,14 @@ def evaluate_contract(
concurrency_levels = list(report["plan"]["concurrency_levels"]) concurrency_levels = list(report["plan"]["concurrency_levels"])
evaluations = tuple( evaluations = tuple(
_evaluate_recipe(entry, reference, drift_by_recipe, contract.thresholds, concurrency_levels) _evaluate_recipe(
entry,
reference,
drift_by_recipe,
contract.thresholds,
concurrency_levels,
len(report["plan"]["prompts"]),
)
for recipe_id, entry in entries.items() for recipe_id, entry in entries.items()
if recipe_id != reference_id if recipe_id != reference_id
) )
@@ -439,18 +605,29 @@ def parse_contract(data: Any, source: str = "<memory>") -> PerformanceContract:
raw_thresholds = data.get("thresholds") raw_thresholds = data.get("thresholds")
if not isinstance(raw_thresholds, Mapping): if not isinstance(raw_thresholds, Mapping):
raise PerformanceContractError(f"'thresholds' in {source} must be a JSON object") raise PerformanceContractError(f"'thresholds' in {source} must be a JSON object")
known = {field for field in ContractThresholds().to_dict()} known = set(ContractThresholds().to_dict())
unknown = set(raw_thresholds) - known unknown = set(raw_thresholds) - known
if unknown: missing = known - set(raw_thresholds)
if unknown or missing:
raise PerformanceContractError( raise PerformanceContractError(
f"{source} carries unknown thresholds {sorted(unknown)}; this node enforces {sorted(known)}" f"{source} threshold keys differ from v1; unknown={sorted(unknown)}, "
f"missing={sorted(missing)}"
) )
thresholds = ContractThresholds(**{ thresholds = ContractThresholds(**{
key: float(value) for key, value in raw_thresholds.items() key: float(value) for key, value in raw_thresholds.items()
}) })
contract_version = int(data["contract_version"])
if contract_version != 1 or thresholds != ContractThresholds():
raise PerformanceContractError(
f"{source} changes immutable v1 thresholds without a supported contract version"
)
if str(data["stop_condition"]) != STOP_CONDITION:
raise PerformanceContractError(
f"{source} stop condition differs from executable v1 semantics"
)
return PerformanceContract( return PerformanceContract(
contract_version=int(data["contract_version"]), contract_version=contract_version,
locked_at=str(data["locked_at"]), locked_at=str(data["locked_at"]),
locked_by=str(data["locked_by"]), locked_by=str(data["locked_by"]),
plan_id=str(data["plan_id"]), plan_id=str(data["plan_id"]),
@@ -486,6 +663,7 @@ def baseline_from_report(report: Mapping[str, Any]) -> dict[str, Any]:
"evidence_class": report["evidence_class"], "evidence_class": report["evidence_class"],
"model_id": report["plan"]["model_id"], "model_id": report["plan"]["model_id"],
"model_revision": report["plan"]["model_revision"], "model_revision": report["plan"]["model_revision"],
"plan_sha256": _canonical_sha256(report["plan"]),
"reference_recipe_id": report["reference_recipe_id"], "reference_recipe_id": report["reference_recipe_id"],
"host": report["host"], "host": report["host"],
"recipes": {}, "recipes": {},

41
packages/node/meshnet_node/recipe_benchmark.py
View File

@@ -28,6 +28,7 @@ from __future__ import annotations
import argparse import argparse
import json import json
import statistics import statistics
import threading
import time import time
from concurrent.futures import ThreadPoolExecutor from concurrent.futures import ThreadPoolExecutor
from dataclasses import asdict, dataclass, field from dataclasses import asdict, dataclass, field
@@ -443,7 +444,7 @@ def compute_drift(
class _PeakMemory: class _PeakMemory:
"""Sample a driver's memory while requests are in flight.""" """Continuously sample a driver's memory while requests are in flight."""
def __init__(self, driver: RecipeDriver) -> None: def __init__(self, driver: RecipeDriver) -> None:
self._driver = driver self._driver = driver
@@ -458,6 +459,12 @@ class _PeakMemory:
self.peak_rss = max(self.peak_rss, rss) self.peak_rss = max(self.peak_rss, rss)
self.peak_vram = max(self.peak_vram, vram) self.peak_vram = max(self.peak_vram, vram)
def sample_until(self, stop: threading.Event, interval_s: float = 0.01) -> None:
"""Sample until ``stop`` is set, including one final post-request probe."""
while not stop.wait(interval_s):
self.sample()
self.sample()
def _run_request( def _run_request(
driver: RecipeDriver, driver: RecipeDriver,
@@ -512,8 +519,6 @@ def measure_recipe(
concurrency 4 still releases its weights before the next recipe loads. concurrency 4 still releases its weights before the next recipe loads.
""" """
load = driver.load() load = driver.load()
memory = _PeakMemory(driver)
memory.sample()
measurement = RecipeMeasurement(recipe=recipe, load=load) measurement = RecipeMeasurement(recipe=recipe, load=load)
try: try:
@@ -524,6 +529,17 @@ def measure_recipe(
break break
for concurrency in plan.concurrency_levels: for concurrency in plan.concurrency_levels:
# Measure each cell independently and sample while requests are in
# flight; post-request probes alone miss transient KV/workspace peaks.
memory = _PeakMemory(driver)
memory.sample()
stop_sampling = threading.Event()
sampler = threading.Thread(
target=memory.sample_until,
args=(stop_sampling,),
name=f"recipe-memory-{recipe.id}-c{concurrency}",
daemon=True,
)
requests = [ requests = [
(prompt, repeat) (prompt, repeat)
for repeat in range(plan.repeats) for repeat in range(plan.repeats)
@@ -531,13 +547,18 @@ def measure_recipe(
for _ in range(concurrency) for _ in range(concurrency)
] ]
started = time.monotonic() started = time.monotonic()
with ThreadPoolExecutor(max_workers=concurrency) as pool: sampler.start()
outcomes = list(pool.map( try:
lambda item: _run_request( with ThreadPoolExecutor(max_workers=concurrency) as pool:
driver, recipe, item[0], plan.sampling, concurrency, item[1], memory outcomes = list(pool.map(
), lambda item: _run_request(
requests, driver, recipe, item[0], plan.sampling, concurrency, item[1], memory
)) ),
requests,
))
finally:
stop_sampling.set()
sampler.join()
wall_ms = (time.monotonic() - started) * 1000 wall_ms = (time.monotonic() - started) * 1000
measurement.outcomes.extend(outcomes) measurement.outcomes.extend(outcomes)

185
packages/node/meshnet_node/recipe_drivers.py
View File

@@ -20,12 +20,16 @@ rules:
from __future__ import annotations from __future__ import annotations
import hashlib
import hmac
import json import json
import os import os
import platform import platform
import re
import socket import socket
import subprocess import subprocess
import sys import sys
import tempfile
import time import time
import urllib.error import urllib.error
import urllib.request import urllib.request
@@ -85,6 +89,37 @@ def _directory_bytes(path: Path) -> int:
return sum(entry.stat().st_size for entry in path.rglob("*") if entry.is_file()) return sum(entry.stat().st_size for entry in path.rglob("*") if entry.is_file())
def _artifact_sha256(path: Path) -> str:
"""Hash an artifact file or a deterministic directory content manifest.
A file uses the ordinary SHA-256 digest. A directory hashes each sorted
relative path, resolved file size, and file bytes, so tokenizer/config drift
cannot hide behind a weight-only digest.
"""
digest = hashlib.sha256()
if path.is_file():
entries = [(None, path)]
else:
entries = [
(entry.relative_to(path).as_posix(), entry)
for entry in sorted(path.rglob("*"))
if entry.is_file()
]
if not entries:
raise BenchmarkError(f"artifact directory is empty: {path}")
for relative, entry in entries:
if relative is not None:
encoded = relative.encode("utf-8")
digest.update(len(encoded).to_bytes(8, "big"))
digest.update(encoded)
digest.update(entry.stat().st_size.to_bytes(8, "big"))
with entry.open("rb") as stream:
while chunk := stream.read(8 * 1024 * 1024):
digest.update(chunk)
return digest.hexdigest()
def _host_manifest() -> dict[str, Any]: def _host_manifest() -> dict[str, Any]:
"""Capture non-secret host facts with the report rather than trusting prose.""" """Capture non-secret host facts with the report rather than trusting prose."""
manifest: dict[str, Any] = { manifest: dict[str, Any] = {
@@ -95,8 +130,10 @@ def _host_manifest() -> dict[str, Any]:
} }
try: try:
import torch import torch
import transformers
manifest["torch_version"] = torch.__version__ manifest["torch_version"] = torch.__version__
manifest["transformers_version"] = transformers.__version__
manifest["cuda_available"] = bool(torch.cuda.is_available()) manifest["cuda_available"] = bool(torch.cuda.is_available())
if torch.cuda.is_available(): if torch.cuda.is_available():
manifest["accelerator_name"] = torch.cuda.get_device_name(0) manifest["accelerator_name"] = torch.cuda.get_device_name(0)
@@ -109,7 +146,7 @@ def _host_manifest() -> dict[str, Any]:
def _validate_config(config: Mapping[str, Any]) -> None: def _validate_config(config: Mapping[str, Any]) -> None:
"""Reject a comparison that could silently mix models or use home storage.""" """Reject comparisons that mix models, artifacts, devices, or budgets."""
try: try:
plan = config["plan"] plan = config["plan"]
root = Path(config["artifact_storage_root"]).resolve(strict=True) root = Path(config["artifact_storage_root"]).resolve(strict=True)
@@ -122,17 +159,75 @@ def _validate_config(config: Mapping[str, Any]) -> None:
raise BenchmarkError("model artifacts must use configured mounted-drive storage, never /home") raise BenchmarkError("model artifacts must use configured mounted-drive storage, never /home")
if not isinstance(recipes, list) or not recipes: if not isinstance(recipes, list) or not recipes:
raise BenchmarkError("benchmark config needs at least one recipe") raise BenchmarkError("benchmark config needs at least one recipe")
sampling = plan.get("sampling", {})
if (
float(sampling.get("temperature", 0.0)) != 0.0
or int(sampling.get("top_k", 1)) != 1
or float(sampling.get("top_p", 1.0)) != 1.0
):
raise BenchmarkError("the quality comparison requires greedy sampling")
if len(plan.get("prompts", ())) < 3 or int(plan.get("repeats", 0)) < 3:
raise BenchmarkError("contract-grade evidence requires at least 3 prompts and 3 repeats")
if int(plan.get("warmup_requests", 0)) < 1:
raise BenchmarkError("contract-grade evidence requires at least one warmup")
if int(sampling.get("max_output_tokens", 0)) < 32:
raise BenchmarkError("contract-grade evidence requires at least 32 output tokens")
thread_budgets: set[int] = set()
max_concurrency = max(int(level) for level in plan.get("concurrency_levels", (1, 4)))
for spec in recipes: for spec in recipes:
if spec.get("source_model_id") != plan.get("model_id"): if spec.get("source_model_id") != plan.get("model_id"):
raise BenchmarkError("every recipe must declare the plan's exact source_model_id") raise BenchmarkError("every recipe must declare the plan's exact source_model_id")
if spec.get("source_model_revision") != plan.get("model_revision"): if spec.get("source_model_revision") != plan.get("model_revision"):
raise BenchmarkError("every recipe must declare the plan's exact source_model_revision") raise BenchmarkError("every recipe must declare the plan's exact source_model_revision")
digest = spec.get("artifact_sha256", "") digest = spec.get("artifact_sha256", "")
if not isinstance(digest, str) or len(digest) != 64: if not isinstance(digest, str) or re.fullmatch(r"[0-9a-f]{64}", digest) is None:
raise BenchmarkError("every recipe must declare its exact 64-character artifact_sha256") raise BenchmarkError("every recipe must declare a lowercase SHA-256 artifact digest")
artifact = Path(spec.get("artifact_path", "")).resolve(strict=True) artifact = Path(spec.get("artifact_path", "")).resolve(strict=True)
if artifact != root and root not in artifact.parents: if artifact != root and root not in artifact.parents:
raise BenchmarkError("every model artifact must be beneath artifact_storage_root") raise BenchmarkError("every model artifact must be beneath artifact_storage_root")
actual_digest = _artifact_sha256(artifact)
if not hmac.compare_digest(digest, actual_digest):
raise BenchmarkError(
f"artifact digest mismatch for {spec.get('id', '<unknown>')}: "
f"declared {digest}, measured {actual_digest}"
)
driver = spec.get("driver")
if not isinstance(driver, Mapping):
raise BenchmarkError("every recipe needs a driver object")
kind = driver.get("type")
if kind == "transformers":
driver_artifact = Path(driver.get("model_path", "")).resolve(strict=True)
elif kind == "llama-cpp-server":
driver_artifact = Path(driver.get("gguf_path", "")).resolve(strict=True)
binary = Path(driver.get("binary", "")).resolve(strict=True)
binary_digest = driver.get("binary_sha256", "")
if (
not isinstance(binary_digest, str)
or re.fullmatch(r"[0-9a-f]{64}", binary_digest) is None
or not hmac.compare_digest(binary_digest, _artifact_sha256(binary))
):
raise BenchmarkError("llama.cpp binary SHA-256 mismatch")
if int(driver.get("n_parallel", max_concurrency)) < max_concurrency:
raise BenchmarkError("llama.cpp parallel slots must cover maximum concurrency")
if driver.get("device", "cpu") != "cpu" or int(driver.get("n_gpu_layers", 0)) != 0:
raise BenchmarkError(
"v1 benchmark supports CPU-only llama.cpp until process VRAM is measurable"
)
else:
raise BenchmarkError(f"unknown driver type {kind!r}")
if driver_artifact != artifact:
raise BenchmarkError("driver artifact path must match the hashed recipe artifact")
if driver.get("device", "cpu") != spec.get("device"):
raise BenchmarkError("recipe and driver must declare the same device")
thread_budgets.add(int(driver.get("threads", 8)))
if len(thread_budgets) != 1:
raise BenchmarkError("every recipe must use the same CPU thread budget")
class TransformersDriver: class TransformersDriver:
@@ -160,8 +255,10 @@ class TransformersDriver:
self._model: Any = None self._model: Any = None
self._tokenizer: Any = None self._tokenizer: Any = None
self._torch: Any = None self._torch: Any = None
self._rss_baseline = 0
def load(self) -> LoadStats: def load(self) -> LoadStats:
self._rss_baseline = _process_rss()
import torch import torch
from transformers import AutoModelForCausalLM, AutoTokenizer from transformers import AutoModelForCausalLM, AutoTokenizer
@@ -185,7 +282,7 @@ class TransformersDriver:
return LoadStats( return LoadStats(
artifact_bytes=_directory_bytes(self.model_path), artifact_bytes=_directory_bytes(self.model_path),
load_ms=round(load_ms, 4), load_ms=round(load_ms, 4),
rss_bytes=_process_rss(), rss_bytes=max(0, _process_rss() - self._rss_baseline),
vram_bytes=self._vram_bytes(), vram_bytes=self._vram_bytes(),
backend_detail=( backend_detail=(
f"torch {torch.__version__}; dtype {self.dtype}; " f"torch {torch.__version__}; dtype {self.dtype}; "
@@ -262,7 +359,7 @@ class TransformersDriver:
return logits.argmax(dim=-1) return logits.argmax(dim=-1)
def memory_probe(self) -> tuple[int, int]: def memory_probe(self) -> tuple[int, int]:
return _process_rss(), self._vram_bytes() return max(0, _process_rss() - self._rss_baseline), self._vram_bytes()
def close(self) -> None: def close(self) -> None:
self._model = None self._model = None
@@ -295,6 +392,7 @@ class LlamaCppServerDriver:
binary: str, binary: str,
gguf_path: str, gguf_path: str,
*, *,
binary_sha256: str,
device: str = "cpu", device: str = "cpu",
threads: int = 8, threads: int = 8,
n_parallel: int = 4, n_parallel: int = 4,
@@ -303,6 +401,7 @@ class LlamaCppServerDriver:
startup_timeout_s: float = 120.0, startup_timeout_s: float = 120.0,
) -> None: ) -> None:
self.binary = Path(binary) self.binary = Path(binary)
self.binary_sha256 = binary_sha256
self.gguf_path = Path(gguf_path) self.gguf_path = Path(gguf_path)
self.device = device self.device = device
self.threads = threads self.threads = threads
@@ -318,11 +417,34 @@ class LlamaCppServerDriver:
def _url(self) -> str: def _url(self) -> str:
return f"http://127.0.0.1:{self._port}" return f"http://127.0.0.1:{self._port}"
def _log_excerpt(self) -> str:
if self._log is None:
return ""
try:
self._log.flush()
self._log.seek(0)
return self._log.read()[-4096:].decode("utf-8", errors="replace").strip()
except Exception:
return ""
def load(self) -> LoadStats: def load(self) -> LoadStats:
if not self.binary.exists(): if not self.binary.exists():
raise BenchmarkError(f"llama-server binary not found at {self.binary}") raise BenchmarkError(f"llama-server binary not found at {self.binary}")
if not self.gguf_path.exists(): if not self.gguf_path.exists():
raise BenchmarkError(f"GGUF artifact not found at {self.gguf_path}") raise BenchmarkError(f"GGUF artifact not found at {self.gguf_path}")
measured_binary_sha256 = _artifact_sha256(self.binary)
if not hmac.compare_digest(self.binary_sha256, measured_binary_sha256):
raise BenchmarkError("llama-server binary changed after config validation")
version = " | ".join(
subprocess.run(
[str(self.binary), "--version"],
check=True,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
text=True,
timeout=10,
).stdout.strip().splitlines()
)
self._port = _free_port() self._port = _free_port()
command = [ command = [
@@ -339,9 +461,9 @@ class LlamaCppServerDriver:
"--no-webui", "--no-webui",
] ]
started = time.monotonic() started = time.monotonic()
self._log = subprocess.PIPE self._log = tempfile.TemporaryFile(mode="w+b")
self._process = subprocess.Popen( self._process = subprocess.Popen(
command, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL command, stdout=self._log, stderr=subprocess.STDOUT
) )
self._await_health(started) self._await_health(started)
load_ms = (time.monotonic() - started) * 1000 load_ms = (time.monotonic() - started) * 1000
@@ -352,7 +474,8 @@ class LlamaCppServerDriver:
rss_bytes=_process_rss(self._process.pid), rss_bytes=_process_rss(self._process.pid),
vram_bytes=0, vram_bytes=0,
backend_detail=( backend_detail=(
f"llama-server; threads {self.threads}; parallel slots {self.n_parallel}; " f"{version}; binary sha256 {measured_binary_sha256}; "
f"threads {self.threads}; parallel slots {self.n_parallel}; "
f"ctx/slot {self.context_per_slot}; gpu layers {self.n_gpu_layers}" f"ctx/slot {self.context_per_slot}; gpu layers {self.n_gpu_layers}"
), ),
) )
@@ -361,7 +484,8 @@ class LlamaCppServerDriver:
while time.monotonic() - started < self.startup_timeout_s: while time.monotonic() - started < self.startup_timeout_s:
if self._process is not None and self._process.poll() is not None: if self._process is not None and self._process.poll() is not None:
raise BenchmarkError( raise BenchmarkError(
f"llama-server exited with code {self._process.returncode} during startup" f"llama-server exited with code {self._process.returncode} during startup; "
f"log tail: {self._log_excerpt()}"
) )
try: try:
with urllib.request.urlopen(f"{self._url}/health", timeout=2) as response: with urllib.request.urlopen(f"{self._url}/health", timeout=2) as response:
@@ -370,7 +494,8 @@ class LlamaCppServerDriver:
except (urllib.error.URLError, OSError): except (urllib.error.URLError, OSError):
time.sleep(0.25) time.sleep(0.25)
raise BenchmarkError( raise BenchmarkError(
f"llama-server did not become healthy within {self.startup_timeout_s:.0f}s" f"llama-server did not become healthy within {self.startup_timeout_s:.0f}s; "
f"log tail: {self._log_excerpt()}"
) )
def generate(self, prompt: str, sampling: SamplingPolicy) -> GenerationSample: def generate(self, prompt: str, sampling: SamplingPolicy) -> GenerationSample:
@@ -397,7 +522,6 @@ class LlamaCppServerDriver:
) )
started = time.monotonic() started = time.monotonic()
ttft_ms = 0.0
chunks: list[str] = [] chunks: list[str] = []
timings: Mapping[str, Any] = {} timings: Mapping[str, Any] = {}
with urllib.request.urlopen(request, timeout=600) as response: with urllib.request.urlopen(request, timeout=600) as response:
@@ -407,8 +531,6 @@ class LlamaCppServerDriver:
continue continue
payload = json.loads(line[len("data:"):].strip()) payload = json.loads(line[len("data:"):].strip())
content = payload.get("content", "") content = payload.get("content", "")
if content and not ttft_ms:
ttft_ms = (time.monotonic() - started) * 1000
chunks.append(content) chunks.append(content)
if payload.get("stop"): if payload.get("stop"):
timings = payload.get("timings") or {} timings = payload.get("timings") or {}
@@ -422,8 +544,14 @@ class LlamaCppServerDriver:
return GenerationSample( return GenerationSample(
text="".join(chunks), text="".join(chunks),
prompt_tokens=int(timings.get("prompt_n", 0)), prompt_tokens=int(timings.get("prompt_n", 0)),
decode_tokens=int(timings.get("predicted_n", 0)), # llama.cpp starts predicted_ms after sampling the first token while
ttft_ms=ttft_ms or total_ms, # predicted_n includes it. Exclude that token to match the
# Transformers inter-token decode metric.
decode_tokens=max(0, int(timings.get("predicted_n", 0)) - 1),
# Use the runtime's prompt/first-token timing, matching the
# in-process Transformers boundary. HTTP/SSE and slot delay remain
# represented by total latency and queue_wait_ms.
ttft_ms=prefill_ms,
prefill_ms=prefill_ms, prefill_ms=prefill_ms,
decode_ms=decode_ms, decode_ms=decode_ms,
total_ms=total_ms, total_ms=total_ms,
@@ -438,15 +566,18 @@ class LlamaCppServerDriver:
return _process_rss(self._process.pid), 0 return _process_rss(self._process.pid), 0
def close(self) -> None: def close(self) -> None:
if self._process is None: if self._process is not None:
return if self._process.poll() is None:
self._process.terminate() self._process.terminate()
try: try:
self._process.wait(timeout=20) self._process.wait(timeout=20)
except subprocess.TimeoutExpired: except subprocess.TimeoutExpired:
self._process.kill() self._process.kill()
self._process.wait(timeout=10) self._process.wait(timeout=10)
self._process = None self._process = None
if self._log is not None:
self._log.close()
self._log = None
def build_driver(spec: Mapping[str, Any], plan: BenchmarkPlan) -> RecipeDriverBundle: def build_driver(spec: Mapping[str, Any], plan: BenchmarkPlan) -> RecipeDriverBundle:
@@ -511,6 +642,7 @@ def run_configured_benchmark(config: Mapping[str, Any]) -> dict:
measurements = [] measurements = []
for spec in config["recipes"]: for spec in config["recipes"]:
recipe = _recipe_from_config(spec) recipe = _recipe_from_config(spec)
driver = None
try: try:
driver = build_driver(spec, plan) driver = build_driver(spec, plan)
measurements.append(measure_recipe(driver, recipe, plan)) measurements.append(measure_recipe(driver, recipe, plan))
@@ -520,10 +652,13 @@ def run_configured_benchmark(config: Mapping[str, Any]) -> dict:
load=LoadStats(artifact_bytes=0, load_ms=0.0), load=LoadStats(artifact_bytes=0, load_ms=0.0),
unavailable_reason=f"{type(exc).__name__}: {exc}", unavailable_reason=f"{type(exc).__name__}: {exc}",
)) ))
finally:
if driver is not None:
driver.close()
return build_report( return build_report(
plan, plan,
measurements, measurements,
host={**_host_manifest(), **dict(config.get("host", {}))}, host={**dict(config.get("host", {})), **_host_manifest()},
evidence_class=config.get("evidence_class", "local-real"), evidence_class=config.get("evidence_class", "local-real"),
) )

220
tests/test_recipe_benchmark.py
View File

@@ -8,12 +8,24 @@ report.
from __future__ import annotations from __future__ import annotations
import pytest import copy
import time import time
from pathlib import Path
import pytest
from meshnet_node.performance_contract import ( from meshnet_node.performance_contract import (
STOP_CONDITION,
ContractThresholds, ContractThresholds,
PerformanceContract, PerformanceContract,
PerformanceContractError,
_canonical_sha256,
evaluate_contract, evaluate_contract,
parse_contract,
)
from meshnet_node.recipe_drivers import (
_artifact_sha256,
_validate_config,
require_real_inference,
) )
from meshnet_node.recipe_benchmark import ( from meshnet_node.recipe_benchmark import (
BenchmarkError, BenchmarkError,
@@ -344,3 +356,209 @@ def test_contract_requires_a_quality_lane_then_allows_quantized_fit_benefit():
assert evaluation.quality_lane_pass is True assert evaluation.quality_lane_pass is True
assert evaluation.fit_benefit is True assert evaluation.fit_benefit is True
assert evaluation.verdict == "optimize" assert evaluation.verdict == "optimize"
def test_peak_memory_is_sampled_while_requests_are_in_flight():
class TransientMemoryDriver(FakeDriver):
def memory_probe(self) -> tuple[int, int]:
return (99_000_000 if self.in_flight else 1_000_000), 0
measurement = measure_recipe(
TransientMemoryDriver(generation_delay_s=0.05),
recipe("transient", Lane.QUALITY, reference=True),
plan(warmup_requests=0),
)
assert measurement.metrics[1].peak_rss_bytes == 99_000_000
assert measurement.metrics[4].peak_rss_bytes == 99_000_000
def test_real_inference_requires_explicit_opt_in(monkeypatch):
monkeypatch.delenv("MESHNET_ENABLE_REAL_INFERENCE_TESTS", raising=False)
with pytest.raises(BenchmarkError, match="opt-in"):
require_real_inference()
def test_config_rejects_an_artifact_digest_mismatch(tmp_path: Path):
artifact = tmp_path / "model.gguf"
artifact.write_bytes(b"real model bytes")
config = {
"artifact_storage_root": str(tmp_path),
"plan": {
"model_id": "test/model",
"model_revision": "revision-1",
"prompts": [
{"id": "p1", "text": "one"},
{"id": "p2", "text": "two"},
{"id": "p3", "text": "three"},
],
"concurrency_levels": [1, 4],
"repeats": 3,
"warmup_requests": 1,
"sampling": {
"temperature": 0.0,
"top_k": 1,
"top_p": 1.0,
"max_output_tokens": 32,
},
},
"recipes": [{
"id": "recipe",
"source_model_id": "test/model",
"source_model_revision": "revision-1",
"artifact_path": str(artifact),
"artifact_sha256": _artifact_sha256(artifact),
"device": "cpu",
"driver": {
"type": "llama-cpp-server",
"binary": str(artifact),
"binary_sha256": _artifact_sha256(artifact),
"gguf_path": str(artifact),
"device": "cpu",
"threads": 8,
"n_parallel": 4,
},
}],
}
_validate_config(config)
config["recipes"][0]["artifact_sha256"] = "0" * 64
with pytest.raises(BenchmarkError, match="digest mismatch"):
_validate_config(config)
config["recipes"][0]["artifact_sha256"] = _artifact_sha256(artifact)
config["recipes"][0]["driver"]["binary_sha256"] = "0" * 64
with pytest.raises(BenchmarkError, match="binary SHA-256 mismatch"):
_validate_config(config)
config["recipes"][0]["driver"]["binary_sha256"] = _artifact_sha256(artifact)
config["recipes"][0]["driver"]["n_gpu_layers"] = 1
with pytest.raises(BenchmarkError, match="CPU-only"):
_validate_config(config)
def _lock_real_report(report: dict) -> PerformanceContract:
report["evidence_class"] = "local-real"
report["host"] = {
"hostname": "test-host",
"platform": "test-platform",
"python": "3.12",
"cpu_count": 8,
}
for entry in report["recipes"]:
entry["recipe"]["source_model_id"] = report["plan"]["model_id"]
entry["recipe"]["source_model_revision"] = report["plan"]["model_revision"]
entry["recipe"]["artifact_sha256"] = "a" * 64
return PerformanceContract(
contract_version=1,
locked_at="2026-07-13T00:00:00Z",
locked_by="test",
plan_id=report["plan"]["plan_id"],
thresholds=ContractThresholds(),
baseline={
"required_evidence_class": "local-real",
"required_recipes": [entry["recipe"]["id"] for entry in report["recipes"]],
"required_concurrency_levels": [1, 4],
},
stop_condition="test",
)
def test_locked_contract_rejects_synthetic_evidence():
reference = measure_recipe(
FakeDriver(), recipe("ref", Lane.QUALITY, reference=True), plan()
)
quality = measure_recipe(FakeDriver(), recipe("quality", Lane.QUALITY), plan())
q4 = measure_recipe(FakeDriver(), recipe("q4", Lane.PERFORMANCE_FIT), plan())
report = build_report(
plan(), [reference, quality, q4], host={}, evidence_class="synthetic"
)
contract = _lock_real_report(report)
report["evidence_class"] = "synthetic"
with pytest.raises(PerformanceContractError, match="evidence class"):
evaluate_contract(contract, report)
def test_quality_lane_requires_every_prompt_to_be_compared():
texts = {prompt.text: "same greedy answer" for prompt in PROMPTS}
reference = measure_recipe(
FakeDriver(texts=texts), recipe("ref", Lane.QUALITY, reference=True), plan()
)
quality = measure_recipe(
FakeDriver(texts=texts), recipe("quality", Lane.QUALITY), plan()
)
q4 = measure_recipe(
FakeDriver(texts=texts, rss_bytes=1_000_000),
recipe("q4", Lane.PERFORMANCE_FIT),
plan(),
)
report = build_report(
plan(), [reference, quality, q4], host={}, evidence_class="synthetic"
)
contract = _lock_real_report(report)
incomplete = copy.deepcopy(report)
quality_drift = next(
drift for drift in incomplete["drift"] if drift["recipe_id"] == "quality"
)
quality_drift["compared_prompts"] = 1
quality_drift["per_prompt"] = quality_drift["per_prompt"][:1]
evaluation = evaluate_contract(contract, incomplete)
assert evaluation.quality_lane_pass is False
assert evaluation.verdict == "stop"
def test_locked_contract_rejects_changed_plan_and_token_counts():
texts = {prompt.text: "same greedy answer" for prompt in PROMPTS}
reference = measure_recipe(
FakeDriver(texts=texts), recipe("ref", Lane.QUALITY, reference=True), plan()
)
quality = measure_recipe(
FakeDriver(texts=texts), recipe("quality", Lane.QUALITY), plan()
)
q4 = measure_recipe(
FakeDriver(texts=texts), recipe("q4", Lane.PERFORMANCE_FIT), plan()
)
report = build_report(
plan(), [reference, quality, q4], host={}, evidence_class="synthetic"
)
contract = _lock_real_report(report)
contract.baseline["required_plan_sha256"] = _canonical_sha256(report["plan"])
changed_plan = copy.deepcopy(report)
changed_plan["plan"]["prompts"][0]["text"] = "changed after locking"
with pytest.raises(PerformanceContractError, match="plan digest"):
evaluate_contract(contract, changed_plan)
changed_tokens = copy.deepcopy(report)
quality_entry = next(
entry for entry in changed_tokens["recipes"] if entry["recipe"]["id"] == "quality"
)
quality_entry["outcomes"][0]["decode_tokens"] += 1
with pytest.raises(PerformanceContractError, match="different prompt/decode token counts"):
evaluate_contract(contract, changed_tokens)
def test_v1_contract_thresholds_and_stop_condition_are_immutable():
raw = PerformanceContract(
contract_version=1,
locked_at="2026-07-13T00:00:00Z",
locked_by="test",
plan_id="test-plan",
thresholds=ContractThresholds(),
baseline={},
stop_condition=STOP_CONDITION,
).to_dict()
parse_contract(raw)
changed_threshold = copy.deepcopy(raw)
changed_threshold["thresholds"]["min_decode_speedup"] = 1.01
with pytest.raises(PerformanceContractError, match="immutable v1 thresholds"):
parse_contract(changed_threshold)
changed_stop = copy.deepcopy(raw)
changed_stop["stop_condition"] = "promote everything"
with pytest.raises(PerformanceContractError, match="stop condition differs"):
parse_contract(changed_stop)