popov/neuron-tai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: harden DGR-001 performance contract evidence

Browse Source
This commit is contained in:
Dobromir Popov
2026-07-13 19:10:24 +03:00
parent e24db7854f
commit 9e67b829e3
16 changed files with 3674 additions and 151 deletions
Download Patch File Download Diff File Expand all files Collapse all files

61
.scratch/distributed-gguf-runtime/evidence/DGR-001/BLOCKED.md
View File

@@ -1,30 +1,43 @@
# DGR-001 real-benchmark blocker
# DGR-001 downstream stop-condition handoff
Status: blocked only for the required real-model measurement. The deterministic
harness, report schema, and immutable contract are implemented and tested; this
file deliberately does not turn an unrun benchmark into a passing result.
Status: **DGR-001 is complete; native-track promotion is blocked by the immutable v1 verdict.**
## Verified environment state (2026-07-13)
This is no longer an execution-prerequisite blocker. The required real benchmark
ran successfully, every recipe completed at concurrency 1 and 4, artifacts were
verified, and deterministic/full test gates passed.
- Mounted GGUF artifacts exist under `/run/media/popov/DATA/llm/`.
- `llama-server` is not on `PATH`.
- The available Python test environment has neither `torch` nor `transformers`.
- No matching local safetensors snapshot was found for an installed GGUF recipe.
## Locked result
Therefore this session cannot run the controlled same-model, same-revision,
same-machine comparison without downloading/installing new runtime/model assets.
That is intentionally not inferred from the story request.
`contract-evaluation.json` records:
## Continuation
```text
verdict: stop
quality_lane_pass: false
speed_benefit: true
fit_benefit: true
stop_condition_met: true
```
1. Put a matching safetensors snapshot and near-lossless plus quantized GGUF
artifacts below one mounted-drive root, never `/home`.
2. Install or build the pinned `llama-server`, and use `.venv-rocm` when testing
the Radeon backend.
3. Compute each artifact SHA-256 and create a config declaring the same
`source_model_id` and `source_model_revision` for every recipe.
4. Run the command in `commands.txt` with
`MESHNET_ENABLE_REAL_INFERENCE_TESTS=1`; save its JSON report and summary in
this directory, then evaluate it against `performance-contract.json`.
5. Only after those results and all quality gates pass may DGR-001 be marked
done and DGR-004 consume the baseline.
The exact-revision BF16 GGUF quality lane compared every prompt but achieved
`0.3333` exact match and `0.9471` mean similarity against the Transformers BF16
reference. V1 requires `0.90` and `0.97`. Quantized Q4_K_M had substantial speed
and fit benefits, but the contract explicitly forbids speed from redeeming a
failed near-lossless quality lane.
## Scope of this stop
The measured baseline is Qwen2.5-0.5B on CPU using a CPU-only llama.cpp build.
It is not a Radeon, large-model, distributed, or native-shard result. Therefore:
1. Do not silently mark v1 promoted or weaken its thresholds after observing the
data.
2. Do not let DGR-004 or later runtime stories treat DGR-001 completion as a
positive promotion signal.
3. A human may choose one of these explicit paths:
- stop the native GGUF track as v1 directs;
- diagnose and fix the BF16 runtime divergence, then rerun the exact v1 plan;
- authorize a separately versioned GPU/large-model contract whose scope and
workload are locked before its measurements.
All raw evidence, configuration, artifacts, hashes, and reproduction commands
are in this directory and `README.md`.

179
.scratch/distributed-gguf-runtime/evidence/DGR-001/README.md
View File

@@ -1,62 +1,153 @@
# DGR-001 — Safetensors versus GGUF performance contract
Status: **blocked for real evidence; deterministic implementation complete.**
No model benchmark is claimed. See `BLOCKED.md` and the explicitly `not-run`
`results.json`.
Status: **complete; immutable v1 verdict is `stop`.**
## What is implemented
DGR-001 successfully produced a controlled local-real CPU baseline. Completion
means the experiment and decision contract are durable and verified; it does
**not** mean the native GGUF track is approved to continue. The locked quality
gate failed, so dependent runtime work requires a human decision or a new,
explicitly versioned experiment/contract rather than silently weakening v1.
- `recipe_benchmark.py` is a deterministic measurement core that runs the exact
same plan for every recipe and reports TTFT, prefill/decode rates, p50/p95
latency, aggregate throughput, RSS, VRAM, artifact bytes, request failures,
and per-prompt output drift in JSON.
- `recipe_drivers.py` supplies opt-in Transformers/safetensors and whole-model
llama.cpp-server drivers. Real execution requires
`MESHNET_ENABLE_REAL_INFERENCE_TESTS=1`, refuses model paths outside the
declared mounted-drive root, requires a SHA-256 per artifact, records host
facts, and requires the same declared source model and revision for every
recipe.
- `performance_contract.py` separates a near-lossless quality lane from the
quantized performance/fit lane. Quantized drift is advisory; only the quality
lane can establish parity. `performance-contract.json` locks v1 thresholds
and the stop condition before any result exists.
## Controlled workload
## Files changed
- Model: `Qwen/Qwen2.5-0.5B-Instruct`
- Exact source revision: `7ae557604adf67be50417f59c2c2f167def9a775`
- Machine: `fedora`, Linux `7.0.14-101.fc43.x86_64`, 32 logical CPUs
- Device: CPU for every recipe; VRAM is therefore correctly reported as zero
- Runtime reference: Transformers `5.13.0`, PyTorch
`2.10.0+rocm7.13.0a20260513`, BF16 safetensors
- GGUF runtime: llama.cpp version 9991, commit
`e920c523e3b8a0163fe498af5bf90df35ff51d25`
- Workload: three fixed short/medium/long prompts, greedy sampling, 32 output
tokens, three repeats, two warmups, concurrency 1 and 4, 16 CPU threads
- Evidence class: `local-real`
- `packages/node/meshnet_node/recipe_benchmark.py`
- `packages/node/meshnet_node/recipe_drivers.py`
- `packages/node/meshnet_node/performance_contract.py`
- `tests/test_recipe_benchmark.py`
- This evidence directory.
All artifacts are beneath `/run/media/popov/DATA/llm/`; no model artifact was
created under `/home`.
## Commands and results
## Recipes and exact artifacts
`commands.txt` contains exact commands. Final targeted result:
| Recipe | Artifact | SHA-256 |
|---|---|---|
| Transformers BF16 reference | complete mounted Hugging Face snapshot | `e596e9d6205fdc9177569cccd7f8b471b058f66e3630c8e4326d5aad52bd18b6` |
| llama.cpp BF16 quality lane | `Qwen2.5-0.5B-Instruct-7ae5576-BF16.gguf` | `e842fdc35d7f00fda95a54e1b51731ba1d196aea45065cc9f46925fdc1d6f862` |
| llama.cpp Q4_K_M performance/fit lane | `Qwen2.5-0.5B-Instruct-7ae5576-Q4_K_M.gguf` | `a88e3f570e2efeaf06b50df9859db2c70d8646aa3a2c94a14e14d5797a2921a5` |
The snapshot digest covers every sorted relative path, resolved size, and file
byte, so tokenizer/config drift is included. The BF16 GGUF was converted
directly from the exact snapshot while preserving BF16 weights. Q4_K_M was
quantized from an exact-revision F16 conversion with the pinned quantizer.
Runtime validation recomputes every declared digest before model loading.
## Real results
All recipes completed every request with zero failures.
| Metric | Transformers BF16 | llama.cpp BF16 | llama.cpp Q4_K_M |
|---|---:|---:|---:|
| Decode tok/s, c=1 | 46.1 | 88.0 | 170.1 |
| Aggregate decode tok/s, c=4 | 47.1 | 211.4 | 206.4 |
| TTFT p50, c=1 | 37.5 ms | 43.9 ms | 23.8 ms |
| Peak resident memory, c=1 | 1.94 GB | 1.11 GB | 0.54 GB |
| Artifact size | 1.00 GB | 0.99 GB | 0.40 GB |
| Failures | 0 | 0 | 0 |
Against the reference, the eligible Q4_K_M lane measured:
- single-request decode speedup: **3.69×**;
- concurrency-4 aggregate throughput speedup: **4.38×**;
- resident-memory ratio: **0.279×**;
- artifact-size ratio: **0.398×**.
The near-lossless BF16 quality lane compared all three prompts but measured:
- exact match: **0.3333** (v1 requires at least `0.90`);
- mean text similarity: **0.9471** (v1 requires at least `0.97`).
Tokenization and stopping were controlled: every runtime saw the same prompt
token counts and reported 31 post-TTFT decode tokens. The mismatch is genuine
greedy runtime divergence on two prompts, not missing coverage or a text-length
artifact. Therefore `contract-evaluation.json` records:
```text
pytest -q tests/test_recipe_benchmark.py -> 15 passed
python -m compileall -q packages tests -> exit 0
git diff --check -> exit 0
verdict: stop
quality_lane_pass: false
speed_benefit: true
fit_benefit: true
stop_condition_met: true
```
The full suite was attempted and is blocked during collection by the unrelated,
pre-existing DGR-002 runtime dependency mismatch:
Thresholds were not changed after observing these results.
## Implementation
- `recipe_benchmark.py` provides the runtime-neutral measurement core, true
concurrency, continuous in-flight peak-memory sampling, percentile/throughput
aggregation, failures, and output drift.
- `recipe_drivers.py` provides opt-in Transformers and llama-server drivers,
mounted-drive confinement, exact artifact/runtime verification, equal
device/thread budgets, greedy-only validation, measured host provenance, and
a CPU-only v1 guard until process VRAM can be measured honestly.
- Peak RSS is runtime-scoped: Transformers reports growth above its pre-runtime
Python baseline, while llama.cpp reports its isolated server process tree.
Both are sampled continuously during in-flight requests.
- TTFT uses each runtime's prompt/first-token compute boundary; end-to-end HTTP,
scheduling, and queue overhead remains in latency and `queue_wait_ms`.
- The exact canonical plan SHA-256 locks prompts, model/revision, sampling,
output length, repeats, warmups, and concurrency. The evaluator also requires
equal prompt/decode token counts across recipes.
- llama.cpp's `predicted_n` includes the first token while `predicted_ms` begins
after it; the driver subtracts that token so decode throughput matches the
Transformers inter-token convention.
- `performance_contract.py` rejects wrong plans, synthetic evidence, missing
recipes/concurrency, mixed model revisions, incomplete quality coverage,
failed references, and missing artifact/host provenance.
- Quantized drift remains advisory. Only the near-lossless lane can satisfy the
quality gate, and only performance-fit recipes can earn speed/fit benefits.
## Evidence files
- `performance-contract.json` — immutable v1 thresholds and stop condition
- `benchmark-config.json` — exact real-run plan, drivers, artifacts, and hashes
- `results.json` — raw machine-readable per-request and aggregate evidence
- `results.txt` — human-readable benchmark summary
- `baseline.json` — distilled measurements for later comparison
- `contract-evaluation.json` — fail-closed v1 verdict
- `commands.txt` — reproducible conversion, benchmark, evaluation, and test commands
- `BLOCKED.md` — downstream stop-condition handoff
- `known-unrelated-failure.md` — clean-base reproduction of the tracker race
## Verification
```text
google.protobuf.runtime_version.VersionError:
gencode 7.35.0 runtime 6.33.6
Targeted: 22 passed
Full suite: 749 passed, 13 skipped
Earlier cancellation retry matrix, DGR-001: 4/5 passed
Earlier cancellation retry matrix, clean d904c40: 4/5 passed
compileall: passed
git diff --check: passed
Evidence JSON parse/integrity checks: passed
```
This was reproduced from a clean `git archive HEAD` extracted to
`/tmp/dgr-001-clean`, with the same command and same failure before any
uncommitted DGR-001 changes were present. No real benchmark command was run
because the prerequisites in `BLOCKED.md` are absent.
The full-suite exception is documented in `known-unrelated-failure.md` and
satisfies the issue's explicit clean-tree reproduction clause. DGR-001 changes
no tracker/proxy files.
## Compatibility and handoff
The earlier Ralph claim that the full suite was blocked by Protobuf 6.33.6 was
invalid: it used Hermes Agent's internal venv. Verification above used the
project `.venv`, which has the DGR-002-compatible runtime. Real inference used
`.venv-rocm` Python 3.12.
This is additive: it does not alter the current Transformers route, Tracker,
relay, or native protocol. DGR-014 must load `performance-contract.json`, run
the same controlled plan at concurrency 1 and 4, and make only its
promote/optimize/stop recommendation from a `local-real` or
`multi-machine-real` report. DGR-004 remains blocked on this story's real
baseline decision.
## Limitations and dependent-story handoff
- This is a **0.5B CPU baseline**, not evidence for a large model, Radeon GPU,
distributed execution, network transport, or native shard worker.
- The installed llama.cpp build is CPU-only (`GGML_HIP=OFF`). No GPU comparison
is claimed.
- Absolute timings are developer-machine measurements; locked ratios and raw
artifacts are provided for reproducibility.
- DGR-014 may consume v1 only with the exact plan/evidence requirements enforced
by `performance_contract.py`.
- DGR-004 and later native-runtime work must not treat DGR-001 completion as a
promotion. V1 says `stop`; proceeding requires a human decision backed by a
separately versioned GPU/large-model contract or a diagnosed quality fix.

122
.scratch/distributed-gguf-runtime/evidence/DGR-001/baseline.json Normal file
View File

@@ -0,0 +1,122 @@
{
"evidence_class": "local-real",
"host": {
"accelerator_name": "Radeon 8060S Graphics",
"accelerator_runtime": "7.13.26183",
"benchmark_lane": "cpu-controlled-baseline",
"converter_sha256": "c819f18fb22927b49fabc3b35d1c9e21ee638b3817eccd1bd4efbcc7116eeb4d",
"cpu_count": 32,
"cuda_available": true,
"hostname": "fedora",
"llama_cpp_commit": "e920c523e3b8a0163fe498af5bf90df35ff51d25",
"llama_cpp_version": "9991",
"llama_server_sha256": "fd8fe612970f23e447f2e717cfa51665be06b8d7315ba60556e010f6bca510dd",
"platform": "Linux-7.0.14-101.fc43.x86_64-x86_64-with-glibc2.42",
"python": "3.12.13",
"quantizer_sha256": "bd0cc8c7be6d48aad4755b31062e0e59a887cbadd43dbb8771853d5858bb198f",
"torch_version": "2.10.0+rocm7.13.0a20260513",
"transformers_version": "5.13.0"
},
"model_id": "Qwen/Qwen2.5-0.5B-Instruct",
"model_revision": "7ae557604adf67be50417f59c2c2f167def9a775",
"plan_sha256": "efe24690a9a7164bac6ab3fd0a6b22f078fc08aaefcfb96210ddf154e6050570",
"recipes": {
"llama-cpp-near-lossless-quality": {
"artifact_bytes": 994156448,
"available": true,
"concurrency": {
"1": {
"aggregate_decode_tokens_per_sec": 73.7861,
"decode_tokens_per_sec": 87.9728,
"failures": 0,
"latency_p50_ms": 385.2049,
"latency_p95_ms": 560.2939,
"peak_rss_bytes": 1110708224,
"peak_vram_bytes": 0,
"prefill_tokens_per_sec": 1427.2072,
"ttft_p50_ms": 43.929,
"ttft_p95_ms": 107.003
},
"4": {
"aggregate_decode_tokens_per_sec": 211.3515,
"decode_tokens_per_sec": 73.8932,
"failures": 0,
"latency_p50_ms": 467.5094,
"latency_p95_ms": 790.862,
"peak_rss_bytes": 1129578496,
"peak_vram_bytes": 0,
"prefill_tokens_per_sec": 1077.8162,
"ttft_p50_ms": 33.612,
"ttft_p95_ms": 128.501
}
},
"device": "cpu",
"lane": "quality"
},
"llama-cpp-quantized-performance-fit": {
"artifact_bytes": 397807520,
"available": true,
"concurrency": {
"1": {
"aggregate_decode_tokens_per_sec": 110.0458,
"decode_tokens_per_sec": 170.131,
"failures": 0,
"latency_p50_ms": 258.0681,
"latency_p95_ms": 465.8523,
"peak_rss_bytes": 542167040,
"peak_vram_bytes": 0,
"prefill_tokens_per_sec": 783.3775,
"ttft_p50_ms": 23.847,
"ttft_p95_ms": 237.696
},
"4": {
"aggregate_decode_tokens_per_sec": 206.377,
"decode_tokens_per_sec": 83.543,
"failures": 0,
"latency_p50_ms": 413.3897,
"latency_p95_ms": 910.253,
"peak_rss_bytes": 572788736,
"peak_vram_bytes": 0,
"prefill_tokens_per_sec": 474.3116,
"ttft_p50_ms": 67.945,
"ttft_p95_ms": 431.804
}
},
"device": "cpu",
"lane": "performance-fit"
},
"transformers-safetensors-reference": {
"artifact_bytes": 999586347,
"available": true,
"concurrency": {
"1": {
"aggregate_decode_tokens_per_sec": 40.3425,
"decode_tokens_per_sec": 46.1451,
"failures": 0,
"latency_p50_ms": 795.4807,
"latency_p95_ms": 930.9725,
"peak_rss_bytes": 1941213184,
"peak_vram_bytes": 0,
"prefill_tokens_per_sec": 671.8016,
"ttft_p50_ms": 37.4548,
"ttft_p95_ms": 193.4633
},
"4": {
"aggregate_decode_tokens_per_sec": 47.0903,
"decode_tokens_per_sec": 13.1337,
"failures": 0,
"latency_p50_ms": 2631.0031,
"latency_p95_ms": 3073.7389,
"peak_rss_bytes": 2177265664,
"peak_vram_bytes": 0,
"prefill_tokens_per_sec": 247.5617,
"ttft_p50_ms": 94.3995,
"ttft_p95_ms": 444.6749
}
},
"device": "cpu",
"lane": "quality"
}
},
"reference_recipe_id": "transformers-safetensors-reference"
}

118
.scratch/distributed-gguf-runtime/evidence/DGR-001/benchmark-config.json Normal file
View File

@@ -0,0 +1,118 @@
{
"artifact_storage_root": "/run/media/popov/DATA/llm",
"evidence_class": "local-real",
"host": {
"benchmark_lane": "cpu-controlled-baseline",
"llama_cpp_commit": "e920c523e3b8a0163fe498af5bf90df35ff51d25",
"llama_cpp_version": "9991",
"llama_server_sha256": "fd8fe612970f23e447f2e717cfa51665be06b8d7315ba60556e010f6bca510dd",
"converter_sha256": "c819f18fb22927b49fabc3b35d1c9e21ee638b3817eccd1bd4efbcc7116eeb4d",
"quantizer_sha256": "bd0cc8c7be6d48aad4755b31062e0e59a887cbadd43dbb8771853d5858bb198f",
"transformers_version": "5.13.0"
},
"plan": {
"plan_id": "dgr-001-controlled-whole-model-baseline-v1",
"model_id": "Qwen/Qwen2.5-0.5B-Instruct",
"model_revision": "7ae557604adf67be50417f59c2c2f167def9a775",
"prompts": [
{
"id": "short-fact",
"text": "The capital of France is",
"context_class": "short"
},
{
"id": "medium-code",
"text": "Complete this Python function without commentary:\n\ndef fibonacci(n):\n \"\"\"Return the nth Fibonacci number for n >= 0.\"\"\"\n",
"context_class": "medium"
},
{
"id": "long-summary",
"text": "A distributed inference service divides a transformer across consumer machines. The tracker owns admission, routing, cancellation, accounting, and telemetry, while workers own only model execution. Every request carries an immutable model identity and revision. Workers must reject incompatible protocol versions and resource demands before allocating large buffers. Activation tensors are chunked, checksummed, bounded by negotiated limits, and propagated with explicit flow-control credits. A caller may disconnect at any time, so cancellation must release queued work, in-flight transfers, and cache reservations without double billing. Retries can occur after network failures, requiring idempotent request identifiers and deterministic completion accounting. The system keeps the existing safetensors path as a correctness reference while a native GGUF path is measured. Benchmarks compare the same prompts, output lengths, sampling policy, device, and concurrency, and they separate near-lossless quality checks from quantized speed and fit claims. Summarize the design priorities in three concise bullet points.",
"context_class": "long"
}
],
"sampling": {
"temperature": 0.0,
"top_p": 1.0,
"top_k": 1,
"seed": 1234,
"max_output_tokens": 32
},
"concurrency_levels": [1, 4],
"repeats": 3,
"warmup_requests": 2
},
"recipes": [
{
"id": "transformers-safetensors-reference",
"runtime": "transformers-5.13.0",
"weight_format": "safetensors",
"weight_quantization": "bfloat16",
"lane": "quality",
"device": "cpu",
"artifact_path": "/run/media/popov/DATA/llm/safetensor/models/models--Qwen--Qwen2.5-0.5B-Instruct/snapshots/7ae557604adf67be50417f59c2c2f167def9a775",
"artifact_sha256": "e596e9d6205fdc9177569cccd7f8b471b058f66e3630c8e4326d5aad52bd18b6",
"source_model_id": "Qwen/Qwen2.5-0.5B-Instruct",
"source_model_revision": "7ae557604adf67be50417f59c2c2f167def9a775",
"is_reference": true,
"notes": "artifact_sha256 is the deterministic digest of every snapshot path and file byte",
"driver": {
"type": "transformers",
"model_path": "/run/media/popov/DATA/llm/safetensor/models/models--Qwen--Qwen2.5-0.5B-Instruct/snapshots/7ae557604adf67be50417f59c2c2f167def9a775",
"device": "cpu",
"dtype": "bfloat16",
"threads": 16
}
},
{
"id": "llama-cpp-near-lossless-quality",
"runtime": "llama.cpp-9991-e920c523",
"weight_format": "gguf",
"weight_quantization": "bfloat16",
"lane": "quality",
"device": "cpu",
"artifact_path": "/run/media/popov/DATA/llm/dgr-001/Qwen2.5-0.5B-Instruct-7ae5576-BF16.gguf",
"artifact_sha256": "e842fdc35d7f00fda95a54e1b51731ba1d196aea45065cc9f46925fdc1d6f862",
"source_model_id": "Qwen/Qwen2.5-0.5B-Instruct",
"source_model_revision": "7ae557604adf67be50417f59c2c2f167def9a775",
"is_reference": false,
"notes": "Converted directly from the exact mounted safetensors revision while preserving BF16 weights with pinned llama.cpp",
"driver": {
"type": "llama-cpp-server",
"binary": "/run/media/popov/d/DEV/llamacpp/llama.cpp/build/bin/llama-server",
"binary_sha256": "fd8fe612970f23e447f2e717cfa51665be06b8d7315ba60556e010f6bca510dd",
"gguf_path": "/run/media/popov/DATA/llm/dgr-001/Qwen2.5-0.5B-Instruct-7ae5576-BF16.gguf",
"device": "cpu",
"threads": 16,
"n_parallel": 4,
"context_per_slot": 512,
"n_gpu_layers": 0
}
},
{
"id": "llama-cpp-quantized-performance-fit",
"runtime": "llama.cpp-9991-e920c523",
"weight_format": "gguf",
"weight_quantization": "Q4_K_M",
"lane": "performance-fit",
"device": "cpu",
"artifact_path": "/run/media/popov/DATA/llm/dgr-001/Qwen2.5-0.5B-Instruct-7ae5576-Q4_K_M.gguf",
"artifact_sha256": "a88e3f570e2efeaf06b50df9859db2c70d8646aa3a2c94a14e14d5797a2921a5",
"source_model_id": "Qwen/Qwen2.5-0.5B-Instruct",
"source_model_revision": "7ae557604adf67be50417f59c2c2f167def9a775",
"is_reference": false,
"notes": "Quantized from the exact-revision F16 GGUF with pinned llama-quantize",
"driver": {
"type": "llama-cpp-server",
"binary": "/run/media/popov/d/DEV/llamacpp/llama.cpp/build/bin/llama-server",
"binary_sha256": "fd8fe612970f23e447f2e717cfa51665be06b8d7315ba60556e010f6bca510dd",
"gguf_path": "/run/media/popov/DATA/llm/dgr-001/Qwen2.5-0.5B-Instruct-7ae5576-Q4_K_M.gguf",
"device": "cpu",
"threads": 16,
"n_parallel": 4,
"context_per_slot": 512,
"n_gpu_layers": 0
}
}
]
}

60
.scratch/distributed-gguf-runtime/evidence/DGR-001/commands.txt
View File

@@ -1,11 +1,51 @@
# Deterministic implementation checks completed in this worktree
PYTHONPATH=packages/node /home/popov/.hermes/hermes-agent/venv/bin/python -m pytest -q tests/test_recipe_benchmark.py
PYTHONPATH=packages/node /home/popov/.hermes/hermes-agent/venv/bin/python -m compileall -q packages tests
# Exact source snapshot (already present on mounted storage)
SOURCE=/run/media/popov/DATA/llm/safetensor/models/models--Qwen--Qwen2.5-0.5B-Instruct/snapshots/7ae557604adf67be50417f59c2c2f167def9a775
LLAMA=/run/media/popov/d/DEV/llamacpp/llama.cpp
ROCM_PY=/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv-rocm/bin/python
PROJECT_PY=/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python
OUT=/run/media/popov/DATA/llm/dgr-001
# Converter support check (no writes)
$ROCM_PY $LLAMA/convert_hf_to_gguf.py "$SOURCE" --outtype f16 --outfile "$OUT/Qwen2.5-0.5B-Instruct-7ae5576-F16.gguf" --dry-run
# Exact-revision near-lossless and performance-fit artifacts
$ROCM_PY $LLAMA/convert_hf_to_gguf.py "$SOURCE" --outtype f16 --outfile "$OUT/Qwen2.5-0.5B-Instruct-7ae5576-F16.gguf"
$LLAMA/build/bin/llama-quantize "$OUT/Qwen2.5-0.5B-Instruct-7ae5576-F16.gguf" "$OUT/Qwen2.5-0.5B-Instruct-7ae5576-Q4_K_M.gguf" Q4_K_M
$ROCM_PY $LLAMA/convert_hf_to_gguf.py "$SOURCE" --outtype bf16 --outfile "$OUT/Qwen2.5-0.5B-Instruct-7ae5576-BF16.gguf"
# Runtime and artifact identity
git -C "$LLAMA" rev-parse HEAD
$LLAMA/build/bin/llama-server --version
sha256sum "$LLAMA/build/bin/llama-server" "$LLAMA/convert_hf_to_gguf.py" "$LLAMA/build/bin/llama-quantize"
sha256sum "$SOURCE/model.safetensors" "$OUT/Qwen2.5-0.5B-Instruct-7ae5576-BF16.gguf" "$OUT/Qwen2.5-0.5B-Instruct-7ae5576-Q4_K_M.gguf"
# Deterministic complete-snapshot digest used by benchmark-config.json
PYTHONPATH=packages/node $ROCM_PY - <<'PY'
from pathlib import Path
from meshnet_node.recipe_drivers import _artifact_sha256
print(_artifact_sha256(Path('/run/media/popov/DATA/llm/safetensor/models/models--Qwen--Qwen2.5-0.5B-Instruct/snapshots/7ae557604adf67be50417f59c2c2f167def9a775')))
PY
# Canonical opt-in local-real benchmark
MESHNET_ENABLE_REAL_INFERENCE_TESTS=1 PYTHONPATH=packages/node $ROCM_PY -m meshnet_node.recipe_benchmark \
--config .scratch/distributed-gguf-runtime/evidence/DGR-001/benchmark-config.json \
--json-out .scratch/distributed-gguf-runtime/evidence/DGR-001/results.json \
--summary-out .scratch/distributed-gguf-runtime/evidence/DGR-001/results.txt
# Distil the baseline and evaluate immutable v1
PYTHONPATH=packages/node $PROJECT_PY - <<'PY'
from pathlib import Path
import json
from meshnet_node.performance_contract import baseline_from_report, evaluate_contract, load_contract
root = Path('.scratch/distributed-gguf-runtime/evidence/DGR-001')
report = json.loads((root / 'results.json').read_text())
contract = load_contract(root / 'performance-contract.json')
(root / 'baseline.json').write_text(json.dumps(baseline_from_report(report), indent=2, sort_keys=True) + '\n')
(root / 'contract-evaluation.json').write_text(json.dumps(evaluate_contract(contract, report).to_dict(), indent=2, sort_keys=True) + '\n')
PY
# Deterministic verification
PYTHONPATH=packages/node $PROJECT_PY -m pytest -q tests/test_recipe_benchmark.py
PYTHONPATH=packages/node $PROJECT_PY -m pytest -q
PYTHONPATH=packages/node $PROJECT_PY -m compileall -q packages tests
git diff --check
# Full suite attempted in this worktree and a clean HEAD archive; both stop at
# protobuf gencode 7.35.0 versus installed runtime 6.33.6 during collection.
PYTHONPATH=packages/node /home/popov/.hermes/hermes-agent/venv/bin/python -m pytest -q
# Required opt-in real benchmark after the BLOCKED.md prerequisites exist
MESHNET_ENABLE_REAL_INFERENCE_TESTS=1 PYTHONPATH=packages/node python -m meshnet_node.recipe_benchmark --config /run/media/popov/DATA/meshnet/dgr-001-benchmark.json --json-out .scratch/distributed-gguf-runtime/evidence/DGR-001/results.json --summary-out .scratch/distributed-gguf-runtime/evidence/DGR-001/results.txt

71
.scratch/distributed-gguf-runtime/evidence/DGR-001/contract-evaluation.json Normal file
View File

@@ -0,0 +1,71 @@
{
"contract_version": 1,
"fit_benefit": true,
"plan_id": "dgr-001-controlled-whole-model-baseline-v1",
"quality_lane_pass": false,
"rationale": [
"the near-lossless quality lane failed: the GGUF runtime disagrees with the safetensors reference beyond what near-lossless weights can explain",
"a meaningful speed benefit was measured",
"a meaningful fit benefit was measured"
],
"recipes": [
{
"comparable": true,
"failures": 0,
"fit_benefit": false,
"incomparable_reason": "",
"lane": "quality",
"measurements": {
"aggregate_concurrency": 4,
"aggregate_throughput_speedup": 4.4882,
"artifact_size_ratio": 0.9946,
"artifact_size_win": false,
"compared_prompts": 3,
"decode_speedup": 1.9064,
"exact_match_rate": 0.3333,
"expected_prompts": 3,
"failure_rate": 0.0,
"mean_similarity": 0.9471,
"resident_memory_ratio": 0.5722,
"ttft_ratio": 1.1729
},
"quality_pass": false,
"reasons": [
"single-request decode 1.91x reference (>= 1.25x) at TTFT ratio 1.17",
"aggregate throughput at concurrency 4 is 4.49x reference (>= 1.25x)",
"peak resident memory is 0.57x reference (<= 0.75x)",
"quality lane exact-match 0.33 / similarity 0.947 versus the reference (fail)"
],
"recipe_id": "llama-cpp-near-lossless-quality",
"speed_benefit": false
},
{
"comparable": true,
"failures": 0,
"fit_benefit": true,
"incomparable_reason": "",
"lane": "performance-fit",
"measurements": {
"aggregate_concurrency": 4,
"aggregate_throughput_speedup": 4.3826,
"artifact_size_ratio": 0.398,
"artifact_size_win": true,
"decode_speedup": 3.6869,
"failure_rate": 0.0,
"resident_memory_ratio": 0.2793,
"ttft_ratio": 0.6367
},
"quality_pass": null,
"reasons": [
"single-request decode 3.69x reference (>= 1.25x) at TTFT ratio 0.64",
"aggregate throughput at concurrency 4 is 4.38x reference (>= 1.25x)",
"peak resident memory is 0.28x reference (<= 0.75x)"
],
"recipe_id": "llama-cpp-quantized-performance-fit",
"speed_benefit": true
}
],
"speed_benefit": true,
"stop_condition_met": true,
"verdict": "stop"
}

32
.scratch/distributed-gguf-runtime/evidence/DGR-001/known-unrelated-failure.md Normal file
View File

@@ -0,0 +1,32 @@
# Observed pre-existing intermittent tracker race
This file records an earlier unrelated timing observation; it is **not** the
final DGR-001 verification result.
Test:
```text
tests/test_tracker_routing.py::test_tracker_dashboard_can_cancel_inflight_proxy
```
One earlier full-suite run produced:
```text
1 failed, 745 passed, 13 skipped
```
A five-run isolated retry matrix reproduced the same rate on both branches:
```text
DGR-001 branch: 4/5 passed, 1/5 failed
clean d904c40: 4/5 passed, 1/5 failed
```
The final full-suite run on the exact hardened DGR-001 state completed green:
```text
749 passed, 13 skipped in 251.42s
```
The earlier race was therefore timing-sensitive, pre-existing, and unrelated
to the DGR-001 benchmark/contract files.

14
.scratch/distributed-gguf-runtime/evidence/DGR-001/performance-contract.json
View File

@@ -22,15 +22,23 @@
"llama-cpp-near-lossless-quality",
"llama-cpp-quantized-performance-fit"
],
"required_concurrency_levels": [1, 4],
"required_concurrency_levels": [
1,
4
],
"required_controlled_variables": [
"model architecture",
"model revision",
"machine and device",
"formatted prompts and context lengths",
"output length and greedy sampling policy"
]
],
"required_plan_sha256": "efe24690a9a7164bac6ab3fd0a6b22f078fc08aaefcfb96210ddf154e6050570",
"minimum_prompt_count": 3,
"minimum_repeats": 3,
"minimum_output_tokens": 32,
"required_device": "cpu"
},
"stop_condition": "Stop the native llama.cpp/GGUF track when, on the same machine and device as the Transformers/safetensors reference and under this plan, no performance-fit GGUF recipe delivers either a meaningful speed benefit (at least 25% higher single-request decode tokens/sec without more than 25% worse TTFT, or at least 25% higher aggregate throughput under concurrency) or a meaningful fit benefit (at least 25% lower peak resident memory), or when the near-lossless quality lane fails.",
"stop_condition": "Stop the native llama.cpp/GGUF track when, on the same machine and device as the Transformers/safetensors reference and under this plan, no performance-fit GGUF recipe delivers either a meaningful speed benefit (>=25% higher single-request decode tokens/sec without a >25% worse TTFT, or >=25% higher aggregate throughput under concurrency) or a meaningful fit benefit (>=25% lower peak resident memory), or when the near-lossless quality lane fails, which indicates a broken runtime rather than a quantization trade-off.",
"notes": "Quantized performance-fit output drift is reported as advisory only. It is not numerical-equivalence evidence. DGR-014 consumes this immutable v1 contract."
}

2478
.scratch/distributed-gguf-runtime/evidence/DGR-001/results.json
View File

File diff suppressed because it is too large Load Diff

10
.scratch/distributed-gguf-runtime/evidence/DGR-001/results.txt Normal file
View File

@@ -0,0 +1,10 @@
Recipe benchmark dgr-001-controlled-whole-model-baseline-v1 (local-real)
model Qwen/Qwen2.5-0.5B-Instruct@7ae557604adf67be50417f59c2c2f167def9a775
transformers-safetensors-reference [quality ] c= 1 ttft p50/p95 37.5/ 193.5 ms; prefill 671.8 tok/s; decode 46.1 tok/s; aggregate 40.3 tok/s; rss 1.94 GB; vram 0.00 GB; artifact 1.00 GB; failures 0
transformers-safetensors-reference [quality ] c= 4 ttft p50/p95 94.4/ 444.7 ms; prefill 247.6 tok/s; decode 13.1 tok/s; aggregate 47.1 tok/s; rss 2.18 GB; vram 0.00 GB; artifact 1.00 GB; failures 0
llama-cpp-near-lossless-quality [quality ] c= 1 ttft p50/p95 43.9/ 107.0 ms; prefill 1427.2 tok/s; decode 88.0 tok/s; aggregate 73.8 tok/s; rss 1.11 GB; vram 0.00 GB; artifact 0.99 GB; failures 0
llama-cpp-near-lossless-quality [quality ] c= 4 ttft p50/p95 33.6/ 128.5 ms; prefill 1077.8 tok/s; decode 73.9 tok/s; aggregate 211.4 tok/s; rss 1.13 GB; vram 0.00 GB; artifact 0.99 GB; failures 0
llama-cpp-quantized-performance-fit [performance-fit ] c= 1 ttft p50/p95 23.8/ 237.7 ms; prefill 783.4 tok/s; decode 170.1 tok/s; aggregate 110.0 tok/s; rss 0.54 GB; vram 0.00 GB; artifact 0.40 GB; failures 0
llama-cpp-quantized-performance-fit [performance-fit ] c= 4 ttft p50/p95 67.9/ 431.8 ms; prefill 474.3 tok/s; decode 83.5 tok/s; aggregate 206.4 tok/s; rss 0.57 GB; vram 0.00 GB; artifact 0.40 GB; failures 0
drift llama-cpp-near-lossless-quality vs transformers-safetensors-reference exact 0.33; similarity 0.947 (gated)
drift llama-cpp-quantized-performance-fit vs transformers-safetensors-reference exact 0.00; similarity 0.456 (advisory)

2
.scratch/distributed-gguf-runtime/issues/01-lock-the-safetensors-versus-gguf-performance-contract.md
View File

@@ -1,6 +1,6 @@
# 01 — Lock the safetensors-versus-GGUF performance contract
Status: ready-for-agent
Status: done
## Mandatory fresh-session context

34
.scratch/distributed-gguf-runtime/prd.json
View File

@@ -6,7 +6,7 @@
{
"id": "DGR-001",
"title": "Lock the safetensors-versus-GGUF performance contract",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/01-lock-the-safetensors-versus-gguf-performance-contract.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs a runtime engineer, I need a controlled baseline so that GGUF work proceeds from measured speed, memory, and quality rather than reputation.",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/01-lock-the-safetensors-versus-gguf-performance-contract.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker—not a stitched collection of runtimes.\n\nAs a runtime engineer, I need a controlled baseline so that GGUF work proceeds from measured speed, memory, and quality rather than reputation.",
"acceptanceCriteria": [
"Benchmark the same model architecture/revision, machine, prompts, context lengths, output lengths, sampling policy, and concurrency across the current Transformers/safetensors recipe and whole-model llama.cpp recipes.",
"Separate correctness/quality lanes from quantized performance/fit lanes instead of claiming BF16 and Q4 are numerically equivalent.",
@@ -27,14 +27,14 @@
"Update only this story issue to Status: done after every acceptance criterion and quality gate passes"
],
"priority": 2,
"passes": false,
"passes": true,
"notes": "Source issue: .scratch/distributed-gguf-runtime/issues/01-lock-the-safetensors-versus-gguf-performance-contract.md",
"dependsOn": []
},
{
"id": "DGR-002",
"title": "Adopt the versioned gRPC Shard protocol",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/02-adopt-the-versioned-grpc-shard-protocol.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs a node developer, I need a battle-proven streaming protocol so that Python and C++ Shards communicate without a custom socket protocol.",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/02-adopt-the-versioned-grpc-shard-protocol.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker—not a stitched collection of runtimes.\n\nAs a node developer, I need a battle-proven streaming protocol so that Python and C++ Shards communicate without a custom socket protocol.",
"acceptanceCriteria": [
"Add a Protocol Buffers schema for capability, health, session stream, release, and cancellation operations.",
"Define one long-lived bidirectional gRPC stream per Route Session Activation Seam with deadlines, cancellation, flow control, and structured errors.",
@@ -61,7 +61,7 @@
{
"id": "DGR-003",
"title": "Define exact Artifact and runtime recipe identity",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/03-define-exact-artifact-and-runtime-recipe-identity.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs the Tracker, I need exact compatibility identity so that only numerically and operationally compatible Shards form an Inference Route.",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/03-define-exact-artifact-and-runtime-recipe-identity.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker—not a stitched collection of runtimes.\n\nAs the Tracker, I need exact compatibility identity so that only numerically and operationally compatible Shards form an Inference Route.",
"acceptanceCriteria": [
"Separate weight quantization, activation dtype, compute dtype, KV dtype/layout, tokenizer revision, architecture adapter, backend, and runtime version.",
"Bind derivative or split artifacts to an exact source Model Artifact hash and Shard range.",
@@ -89,7 +89,7 @@
{
"id": "DGR-004",
"title": "Create the reproducible pinned llama.cpp patch stack",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/04-create-the-reproducible-pinned-llama-cpp-patch-stack.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs a maintainer, I need a small auditable fork boundary so that upstream updates do not turn the runtime into an unmaintainable stitched codebase.",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/04-create-the-reproducible-pinned-llama-cpp-patch-stack.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker—not a stitched collection of runtimes.\n\nAs a maintainer, I need a small auditable fork boundary so that upstream updates do not turn the runtime into an unmaintainable stitched codebase.",
"acceptanceCriteria": [
"Pin one exact llama.cpp commit through a reproducible source dependency mechanism.",
"Store a numbered minimal patch stack separately from Meshnet networking code.",
@@ -120,7 +120,7 @@
{
"id": "DGR-005",
"title": "Implement dense-Llama range-aware GGUF ownership",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/05-implement-dense-llama-range-aware-gguf-ownership.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs a node, I need to map only my assigned dense-Llama Shard so that aggregate consumer memory can hold a model larger than one node.",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/05-implement-dense-llama-range-aware-gguf-ownership.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker—not a stitched collection of runtimes.\n\nAs a node, I need to map only my assigned dense-Llama Shard so that aggregate consumer memory can hold a model larger than one node.",
"acceptanceCriteria": [
"Register and allocate only `blk.N.*` tensors in the assigned range.",
"Load embeddings only for the head and final norm/LM head only for the tail, including tied embeddings.",
@@ -151,7 +151,7 @@
{
"id": "DGR-006",
"title": "Implement architecture-defined boundary input/output",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/06-implement-architecture-defined-boundary-input-output.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs a Shard, I need to consume and emit the correct transformer boundary state so that disjoint processes reproduce whole-model execution.",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/06-implement-architecture-defined-boundary-input-output.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker—not a stitched collection of runtimes.\n\nAs a Shard, I need to consume and emit the correct transformer boundary state so that disjoint processes reproduce whole-model execution.",
"acceptanceCriteria": [
"Head accepts token IDs and owns token embedding.",
"Middle/tail bypass token embedding and accept the named boundary bundle.",
@@ -183,7 +183,7 @@
{
"id": "DGR-007",
"title": "Add isolated concurrent local Hot KV State",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/07-add-isolated-concurrent-local-hot-kv-state.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs a client, I need concurrent Route Sessions to retain independent per-Shard cache so that one request cannot clear or corrupt another.",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/07-add-isolated-concurrent-local-hot-kv-state.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker—not a stitched collection of runtimes.\n\nAs a client, I need concurrent Route Sessions to retain independent per-Shard cache so that one request cannot clear or corrupt another.",
"acceptanceCriteria": [
"Map `(Route Session ID, route epoch)` to an isolated llama sequence or bounded context.",
"Allocate KV only for owned layers.",
@@ -214,7 +214,7 @@
{
"id": "DGR-008",
"title": "Build the standalone C++ gRPC Shard worker",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/08-build-the-standalone-c-grpc-shard-worker.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs a node runtime, I need one supervised native process so that llama.cpp internals remain behind a stable project-owned protocol.",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/08-build-the-standalone-c-grpc-shard-worker.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker—not a stitched collection of runtimes.\n\nAs a node runtime, I need one supervised native process so that llama.cpp internals remain behind a stable project-owned protocol.",
"acceptanceCriteria": [
"Worker exposes capability, health, session stream, release, cancellation, and metrics services from DGR-002.",
"Worker loads one exact Artifact/recipe/Shard identity and refuses mismatched requests.",
@@ -249,7 +249,7 @@
{
"id": "DGR-009",
"title": "Integrate the native worker with Meshnet",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/09-integrate-the-native-worker-with-meshnet.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs the existing node service, I need a GGUF Shard backend adapter so that the Tracker, relay, billing, telemetry, and capability admission remain the sole control plane.",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/09-integrate-the-native-worker-with-meshnet.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker—not a stitched collection of runtimes.\n\nAs the existing node service, I need a GGUF Shard backend adapter so that the Tracker, relay, billing, telemetry, and capability admission remain the sole control plane.",
"acceptanceCriteria": [
"Implement the existing model-backend surface without changing Transformers behavior.",
"Registration carries exact validated GGUF recipe, Shard, backend and concurrency/KV capacity.",
@@ -281,7 +281,7 @@
{
"id": "DGR-010",
"title": "Pass local real-model two-process acceptance",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/10-pass-local-real-model-two-process-acceptance.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs a release engineer, I need real local distributed parity before involving network variability.",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/10-pass-local-real-model-two-process-acceptance.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker—not a stitched collection of runtimes.\n\nAs a release engineer, I need real local distributed parity before involving network variability.",
"acceptanceCriteria": [
"Two local worker processes open disjoint dense-Llama ranges from the certified Artifact.",
"Prefill and at least 32 greedy decode tokens match whole-model llama.cpp within the certified tolerance.",
@@ -314,7 +314,7 @@
{
"id": "DGR-011",
"title": "Pass a real heterogeneous two-machine route",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/11-pass-a-real-heterogeneous-two-machine-route.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs a consumer-hardware operator, I need two physical machines to execute one GGUF model so that the distributed claim is real.",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/11-pass-a-real-heterogeneous-two-machine-route.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker—not a stitched collection of runtimes.\n\nAs a consumer-hardware operator, I need two physical machines to execute one GGUF model so that the distributed claim is real.",
"acceptanceCriteria": [
"Tracker selects two physical nodes with disjoint Shards and one exact certified recipe/compatibility class.",
"Actual CPU/GPU execution occurs on both nodes; synthetic workers do not satisfy acceptance.",
@@ -347,7 +347,7 @@
{
"id": "DGR-012",
"title": "Implement continuous batching and bounded admission",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/12-implement-continuous-batching-and-bounded-admission.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs a node operator, I need active sessions batched safely so that concurrency increases aggregate throughput rather than serializing every request.",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/12-implement-continuous-batching-and-bounded-admission.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker—not a stitched collection of runtimes.\n\nAs a node operator, I need active sessions batched safely so that concurrency increases aggregate throughput rather than serializing every request.",
"acceptanceCriteria": [
"Node scheduler admits sessions against weight, KV, scratch, and queue budgets.",
"Compatible decode steps from multiple sessions form llama.cpp batches while preserving per-session positions and outputs.",
@@ -380,7 +380,7 @@
{
"id": "DGR-013",
"title": "Harden failure, cancellation, and restart semantics",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/13-harden-failure-cancellation-and-restart-semantics.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs a client, I need failures to be bounded and explicit so that distributed speed does not come with hanging or corrupted generations.",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/13-harden-failure-cancellation-and-restart-semantics.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker—not a stitched collection of runtimes.\n\nAs a client, I need failures to be bounded and explicit so that distributed speed does not come with hanging or corrupted generations.",
"acceptanceCriteria": [
"Deadlines and heartbeat/health loss terminate blocked stream operations.",
"Cancellation propagates across every Shard and releases local KV and queued buffers.",
@@ -413,7 +413,7 @@
{
"id": "DGR-014",
"title": "Enforce the GGUF-versus-safetensors release gate",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/14-enforce-the-gguf-versus-safetensors-release-gate.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs the product owner, I need an end-to-end comparison so that the native runtime ships only if it advances model access or performance.",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/14-enforce-the-gguf-versus-safetensors-release-gate.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker—not a stitched collection of runtimes.\n\nAs the product owner, I need an end-to-end comparison so that the native runtime ships only if it advances model access or performance.",
"acceptanceCriteria": [
"Run current distributed safetensors and distributed GGUF routes on the same certified model/hardware/network scenario where technically comparable.",
"Report quality, TTFT, prefill/decode throughput, aggregate concurrency throughput, p95 latency, seam cost, memory, KV pressure, failures, and cleanup.",
@@ -448,7 +448,7 @@
{
"id": "DGR-015",
"title": "Add and certify a Qwen3/Qwen3-MoE adapter",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/15-add-and-certify-a-qwen3-qwen3-moe-adapter.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs a client seeking top models, I need a separately certified MoE-capable architecture after the dense runtime proves stable.",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/15-add-and-certify-a-qwen3-qwen3-moe-adapter.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker—not a stitched collection of runtimes.\n\nAs a client seeking top models, I need a separately certified MoE-capable architecture after the dense runtime proves stable.",
"acceptanceCriteria": [
"Implement explicit tensor ownership, router/top-k, expert/shared-expert, Q/K normalization, boundary bundle, and cache semantics for the selected Qwen3 family recipe.",
"Do not reuse the dense-Llama adapter through unchecked name substitutions.",
@@ -480,7 +480,7 @@
{
"id": "DGR-016",
"title": "Produce the upstream llama.cpp collaboration package",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/16-produce-the-upstream-llama-cpp-collaboration-package.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs a maintainer, I need narrow upstreamable proposals so that our patch burden can shrink without asking llama.cpp to own Meshnet networking.",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/16-produce-the-upstream-llama-cpp-collaboration-package.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker—not a stitched collection of runtimes.\n\nAs a maintainer, I need narrow upstreamable proposals so that our patch burden can shrink without asking llama.cpp to own Meshnet networking.",
"acceptanceCriteria": [
"Separate generic llama.cpp hooks from Meshnet protocol/control-plane code.",
"Prepare minimal reproducible examples and tests for range-aware loading, boundary input/output, and layer-filtered KV.",