popov/neuron-tai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: popov:master
Branches Tags
popov:master popov:ralph/dgr-001-performance-contract popov:cursor/fix-meshnet-node-param-parsing popov:worktree-feat+us-016
...
compare: popov:9e67b829e3e39dc32de4cc4dae6903b67065354a
Branches Tags
popov:ralph/dgr-001-performance-contract popov:master popov:cursor/fix-meshnet-node-param-parsing popov:worktree-feat+us-016

5 Commits
master ... 9e67b829e3

Author SHA1 Message Date
Dobromir Popov
9e67b829e3 fix: harden DGR-001 performance contract evidence 2026-07-13 19:10:24 +03:00
Dobromir Popov
e24db7854f feat: DGR-001 - Lock the safetensors-versus-GGUF performance contract 2026-07-13 17:55:55 +03:00
Dobromir Popov
59f2486bf2 feat: DGR-001 - Lock the safetensors-versus-GGUF performance contract 2026-07-13 17:49:09 +03:00
Dobromir Popov
d904c40f66 fix: harden DGR-002 protocol bounds 2026-07-13 17:30:54 +03:00
Dobromir Popov
30dcf953fe feat: DGR-002 - Adopt the versioned gRPC Shard protocol 2026-07-13 16:00:49 +03:00
39 changed files with 9760 additions and 38 deletions
Expand all files Collapse all files

43
.scratch/distributed-gguf-runtime/evidence/DGR-001/BLOCKED.md Normal file
View File

@@ -0,0 +1,43 @@
# DGR-001 downstream stop-condition handoff
Status: **DGR-001 is complete; native-track promotion is blocked by the immutable v1 verdict.**
This is no longer an execution-prerequisite blocker. The required real benchmark
ran successfully, every recipe completed at concurrency 1 and 4, artifacts were
verified, and deterministic/full test gates passed.
## Locked result
`contract-evaluation.json` records:
```text
verdict: stop
quality_lane_pass: false
speed_benefit: true
fit_benefit: true
stop_condition_met: true
```
The exact-revision BF16 GGUF quality lane compared every prompt but achieved
`0.3333` exact match and `0.9471` mean similarity against the Transformers BF16
reference. V1 requires `0.90` and `0.97`. Quantized Q4_K_M had substantial speed
and fit benefits, but the contract explicitly forbids speed from redeeming a
failed near-lossless quality lane.
## Scope of this stop
The measured baseline is Qwen2.5-0.5B on CPU using a CPU-only llama.cpp build.
It is not a Radeon, large-model, distributed, or native-shard result. Therefore:
1. Do not silently mark v1 promoted or weaken its thresholds after observing the
data.
2. Do not let DGR-004 or later runtime stories treat DGR-001 completion as a
positive promotion signal.
3. A human may choose one of these explicit paths:
- stop the native GGUF track as v1 directs;
- diagnose and fix the BF16 runtime divergence, then rerun the exact v1 plan;
- authorize a separately versioned GPU/large-model contract whose scope and
workload are locked before its measurements.
All raw evidence, configuration, artifacts, hashes, and reproduction commands
are in this directory and `README.md`.

153
.scratch/distributed-gguf-runtime/evidence/DGR-001/README.md Normal file
View File

@@ -0,0 +1,153 @@
# DGR-001 — Safetensors versus GGUF performance contract
Status: **complete; immutable v1 verdict is `stop`.**
DGR-001 successfully produced a controlled local-real CPU baseline. Completion
means the experiment and decision contract are durable and verified; it does
**not** mean the native GGUF track is approved to continue. The locked quality
gate failed, so dependent runtime work requires a human decision or a new,
explicitly versioned experiment/contract rather than silently weakening v1.
## Controlled workload
- Model: `Qwen/Qwen2.5-0.5B-Instruct`
- Exact source revision: `7ae557604adf67be50417f59c2c2f167def9a775`
- Machine: `fedora`, Linux `7.0.14-101.fc43.x86_64`, 32 logical CPUs
- Device: CPU for every recipe; VRAM is therefore correctly reported as zero
- Runtime reference: Transformers `5.13.0`, PyTorch
`2.10.0+rocm7.13.0a20260513`, BF16 safetensors
- GGUF runtime: llama.cpp version 9991, commit
`e920c523e3b8a0163fe498af5bf90df35ff51d25`
- Workload: three fixed short/medium/long prompts, greedy sampling, 32 output
tokens, three repeats, two warmups, concurrency 1 and 4, 16 CPU threads
- Evidence class: `local-real`
All artifacts are beneath `/run/media/popov/DATA/llm/`; no model artifact was
created under `/home`.
## Recipes and exact artifacts
| Recipe | Artifact | SHA-256 |
|---|---|---|
| Transformers BF16 reference | complete mounted Hugging Face snapshot | `e596e9d6205fdc9177569cccd7f8b471b058f66e3630c8e4326d5aad52bd18b6` |
| llama.cpp BF16 quality lane | `Qwen2.5-0.5B-Instruct-7ae5576-BF16.gguf` | `e842fdc35d7f00fda95a54e1b51731ba1d196aea45065cc9f46925fdc1d6f862` |
| llama.cpp Q4_K_M performance/fit lane | `Qwen2.5-0.5B-Instruct-7ae5576-Q4_K_M.gguf` | `a88e3f570e2efeaf06b50df9859db2c70d8646aa3a2c94a14e14d5797a2921a5` |
The snapshot digest covers every sorted relative path, resolved size, and file
byte, so tokenizer/config drift is included. The BF16 GGUF was converted
directly from the exact snapshot while preserving BF16 weights. Q4_K_M was
quantized from an exact-revision F16 conversion with the pinned quantizer.
Runtime validation recomputes every declared digest before model loading.
## Real results
All recipes completed every request with zero failures.
| Metric | Transformers BF16 | llama.cpp BF16 | llama.cpp Q4_K_M |
|---|---:|---:|---:|
| Decode tok/s, c=1 | 46.1 | 88.0 | 170.1 |
| Aggregate decode tok/s, c=4 | 47.1 | 211.4 | 206.4 |
| TTFT p50, c=1 | 37.5 ms | 43.9 ms | 23.8 ms |
| Peak resident memory, c=1 | 1.94 GB | 1.11 GB | 0.54 GB |
| Artifact size | 1.00 GB | 0.99 GB | 0.40 GB |
| Failures | 0 | 0 | 0 |
Against the reference, the eligible Q4_K_M lane measured:
- single-request decode speedup: **3.69×**;
- concurrency-4 aggregate throughput speedup: **4.38×**;
- resident-memory ratio: **0.279×**;
- artifact-size ratio: **0.398×**.
The near-lossless BF16 quality lane compared all three prompts but measured:
- exact match: **0.3333** (v1 requires at least `0.90`);
- mean text similarity: **0.9471** (v1 requires at least `0.97`).
Tokenization and stopping were controlled: every runtime saw the same prompt
token counts and reported 31 post-TTFT decode tokens. The mismatch is genuine
greedy runtime divergence on two prompts, not missing coverage or a text-length
artifact. Therefore `contract-evaluation.json` records:
```text
verdict: stop
quality_lane_pass: false
speed_benefit: true
fit_benefit: true
stop_condition_met: true
```
Thresholds were not changed after observing these results.
## Implementation
- `recipe_benchmark.py` provides the runtime-neutral measurement core, true
concurrency, continuous in-flight peak-memory sampling, percentile/throughput
aggregation, failures, and output drift.
- `recipe_drivers.py` provides opt-in Transformers and llama-server drivers,
mounted-drive confinement, exact artifact/runtime verification, equal
device/thread budgets, greedy-only validation, measured host provenance, and
a CPU-only v1 guard until process VRAM can be measured honestly.
- Peak RSS is runtime-scoped: Transformers reports growth above its pre-runtime
Python baseline, while llama.cpp reports its isolated server process tree.
Both are sampled continuously during in-flight requests.
- TTFT uses each runtime's prompt/first-token compute boundary; end-to-end HTTP,
scheduling, and queue overhead remains in latency and `queue_wait_ms`.
- The exact canonical plan SHA-256 locks prompts, model/revision, sampling,
output length, repeats, warmups, and concurrency. The evaluator also requires
equal prompt/decode token counts across recipes.
- llama.cpp's `predicted_n` includes the first token while `predicted_ms` begins
after it; the driver subtracts that token so decode throughput matches the
Transformers inter-token convention.
- `performance_contract.py` rejects wrong plans, synthetic evidence, missing
recipes/concurrency, mixed model revisions, incomplete quality coverage,
failed references, and missing artifact/host provenance.
- Quantized drift remains advisory. Only the near-lossless lane can satisfy the
quality gate, and only performance-fit recipes can earn speed/fit benefits.
## Evidence files
- `performance-contract.json` — immutable v1 thresholds and stop condition
- `benchmark-config.json` — exact real-run plan, drivers, artifacts, and hashes
- `results.json` — raw machine-readable per-request and aggregate evidence
- `results.txt` — human-readable benchmark summary
- `baseline.json` — distilled measurements for later comparison
- `contract-evaluation.json` — fail-closed v1 verdict
- `commands.txt` — reproducible conversion, benchmark, evaluation, and test commands
- `BLOCKED.md` — downstream stop-condition handoff
- `known-unrelated-failure.md` — clean-base reproduction of the tracker race
## Verification
```text
Targeted: 22 passed
Full suite: 749 passed, 13 skipped
Earlier cancellation retry matrix, DGR-001: 4/5 passed
Earlier cancellation retry matrix, clean d904c40: 4/5 passed
compileall: passed
git diff --check: passed
Evidence JSON parse/integrity checks: passed
```
The full-suite exception is documented in `known-unrelated-failure.md` and
satisfies the issue's explicit clean-tree reproduction clause. DGR-001 changes
no tracker/proxy files.
The earlier Ralph claim that the full suite was blocked by Protobuf 6.33.6 was
invalid: it used Hermes Agent's internal venv. Verification above used the
project `.venv`, which has the DGR-002-compatible runtime. Real inference used
`.venv-rocm` Python 3.12.
## Limitations and dependent-story handoff
- This is a **0.5B CPU baseline**, not evidence for a large model, Radeon GPU,
distributed execution, network transport, or native shard worker.
- The installed llama.cpp build is CPU-only (`GGML_HIP=OFF`). No GPU comparison
is claimed.
- Absolute timings are developer-machine measurements; locked ratios and raw
artifacts are provided for reproducibility.
- DGR-014 may consume v1 only with the exact plan/evidence requirements enforced
by `performance_contract.py`.
- DGR-004 and later native-runtime work must not treat DGR-001 completion as a
promotion. V1 says `stop`; proceeding requires a human decision backed by a
separately versioned GPU/large-model contract or a diagnosed quality fix.

122
.scratch/distributed-gguf-runtime/evidence/DGR-001/baseline.json Normal file
View File

@@ -0,0 +1,122 @@
{
"evidence_class": "local-real",
"host": {
"accelerator_name": "Radeon 8060S Graphics",
"accelerator_runtime": "7.13.26183",
"benchmark_lane": "cpu-controlled-baseline",
"converter_sha256": "c819f18fb22927b49fabc3b35d1c9e21ee638b3817eccd1bd4efbcc7116eeb4d",
"cpu_count": 32,
"cuda_available": true,
"hostname": "fedora",
"llama_cpp_commit": "e920c523e3b8a0163fe498af5bf90df35ff51d25",
"llama_cpp_version": "9991",
"llama_server_sha256": "fd8fe612970f23e447f2e717cfa51665be06b8d7315ba60556e010f6bca510dd",
"platform": "Linux-7.0.14-101.fc43.x86_64-x86_64-with-glibc2.42",
"python": "3.12.13",
"quantizer_sha256": "bd0cc8c7be6d48aad4755b31062e0e59a887cbadd43dbb8771853d5858bb198f",
"torch_version": "2.10.0+rocm7.13.0a20260513",
"transformers_version": "5.13.0"
},
"model_id": "Qwen/Qwen2.5-0.5B-Instruct",
"model_revision": "7ae557604adf67be50417f59c2c2f167def9a775",
"plan_sha256": "efe24690a9a7164bac6ab3fd0a6b22f078fc08aaefcfb96210ddf154e6050570",
"recipes": {
"llama-cpp-near-lossless-quality": {
"artifact_bytes": 994156448,
"available": true,
"concurrency": {
"1": {
"aggregate_decode_tokens_per_sec": 73.7861,
"decode_tokens_per_sec": 87.9728,
"failures": 0,
"latency_p50_ms": 385.2049,
"latency_p95_ms": 560.2939,
"peak_rss_bytes": 1110708224,
"peak_vram_bytes": 0,
"prefill_tokens_per_sec": 1427.2072,
"ttft_p50_ms": 43.929,
"ttft_p95_ms": 107.003
},
"4": {
"aggregate_decode_tokens_per_sec": 211.3515,
"decode_tokens_per_sec": 73.8932,
"failures": 0,
"latency_p50_ms": 467.5094,
"latency_p95_ms": 790.862,
"peak_rss_bytes": 1129578496,
"peak_vram_bytes": 0,
"prefill_tokens_per_sec": 1077.8162,
"ttft_p50_ms": 33.612,
"ttft_p95_ms": 128.501
}
},
"device": "cpu",
"lane": "quality"
},
"llama-cpp-quantized-performance-fit": {
"artifact_bytes": 397807520,
"available": true,
"concurrency": {
"1": {
"aggregate_decode_tokens_per_sec": 110.0458,
"decode_tokens_per_sec": 170.131,
"failures": 0,
"latency_p50_ms": 258.0681,
"latency_p95_ms": 465.8523,
"peak_rss_bytes": 542167040,
"peak_vram_bytes": 0,
"prefill_tokens_per_sec": 783.3775,
"ttft_p50_ms": 23.847,
"ttft_p95_ms": 237.696
},
"4": {
"aggregate_decode_tokens_per_sec": 206.377,
"decode_tokens_per_sec": 83.543,
"failures": 0,
"latency_p50_ms": 413.3897,
"latency_p95_ms": 910.253,
"peak_rss_bytes": 572788736,
"peak_vram_bytes": 0,
"prefill_tokens_per_sec": 474.3116,
"ttft_p50_ms": 67.945,
"ttft_p95_ms": 431.804
}
},
"device": "cpu",
"lane": "performance-fit"
},
"transformers-safetensors-reference": {
"artifact_bytes": 999586347,
"available": true,
"concurrency": {
"1": {
"aggregate_decode_tokens_per_sec": 40.3425,
"decode_tokens_per_sec": 46.1451,
"failures": 0,
"latency_p50_ms": 795.4807,
"latency_p95_ms": 930.9725,
"peak_rss_bytes": 1941213184,
"peak_vram_bytes": 0,
"prefill_tokens_per_sec": 671.8016,
"ttft_p50_ms": 37.4548,
"ttft_p95_ms": 193.4633
},
"4": {
"aggregate_decode_tokens_per_sec": 47.0903,
"decode_tokens_per_sec": 13.1337,
"failures": 0,
"latency_p50_ms": 2631.0031,
"latency_p95_ms": 3073.7389,
"peak_rss_bytes": 2177265664,
"peak_vram_bytes": 0,
"prefill_tokens_per_sec": 247.5617,
"ttft_p50_ms": 94.3995,
"ttft_p95_ms": 444.6749
}
},
"device": "cpu",
"lane": "quality"
}
},
"reference_recipe_id": "transformers-safetensors-reference"
}

118
.scratch/distributed-gguf-runtime/evidence/DGR-001/benchmark-config.json Normal file
View File

@@ -0,0 +1,118 @@
{
"artifact_storage_root": "/run/media/popov/DATA/llm",
"evidence_class": "local-real",
"host": {
"benchmark_lane": "cpu-controlled-baseline",
"llama_cpp_commit": "e920c523e3b8a0163fe498af5bf90df35ff51d25",
"llama_cpp_version": "9991",
"llama_server_sha256": "fd8fe612970f23e447f2e717cfa51665be06b8d7315ba60556e010f6bca510dd",
"converter_sha256": "c819f18fb22927b49fabc3b35d1c9e21ee638b3817eccd1bd4efbcc7116eeb4d",
"quantizer_sha256": "bd0cc8c7be6d48aad4755b31062e0e59a887cbadd43dbb8771853d5858bb198f",
"transformers_version": "5.13.0"
},
"plan": {
"plan_id": "dgr-001-controlled-whole-model-baseline-v1",
"model_id": "Qwen/Qwen2.5-0.5B-Instruct",
"model_revision": "7ae557604adf67be50417f59c2c2f167def9a775",
"prompts": [
{
"id": "short-fact",
"text": "The capital of France is",
"context_class": "short"
},
{
"id": "medium-code",
"text": "Complete this Python function without commentary:\n\ndef fibonacci(n):\n \"\"\"Return the nth Fibonacci number for n >= 0.\"\"\"\n",
"context_class": "medium"
},
{
"id": "long-summary",
"text": "A distributed inference service divides a transformer across consumer machines. The tracker owns admission, routing, cancellation, accounting, and telemetry, while workers own only model execution. Every request carries an immutable model identity and revision. Workers must reject incompatible protocol versions and resource demands before allocating large buffers. Activation tensors are chunked, checksummed, bounded by negotiated limits, and propagated with explicit flow-control credits. A caller may disconnect at any time, so cancellation must release queued work, in-flight transfers, and cache reservations without double billing. Retries can occur after network failures, requiring idempotent request identifiers and deterministic completion accounting. The system keeps the existing safetensors path as a correctness reference while a native GGUF path is measured. Benchmarks compare the same prompts, output lengths, sampling policy, device, and concurrency, and they separate near-lossless quality checks from quantized speed and fit claims. Summarize the design priorities in three concise bullet points.",
"context_class": "long"
}
],
"sampling": {
"temperature": 0.0,
"top_p": 1.0,
"top_k": 1,
"seed": 1234,
"max_output_tokens": 32
},
"concurrency_levels": [1, 4],
"repeats": 3,
"warmup_requests": 2
},
"recipes": [
{
"id": "transformers-safetensors-reference",
"runtime": "transformers-5.13.0",
"weight_format": "safetensors",
"weight_quantization": "bfloat16",
"lane": "quality",
"device": "cpu",
"artifact_path": "/run/media/popov/DATA/llm/safetensor/models/models--Qwen--Qwen2.5-0.5B-Instruct/snapshots/7ae557604adf67be50417f59c2c2f167def9a775",
"artifact_sha256": "e596e9d6205fdc9177569cccd7f8b471b058f66e3630c8e4326d5aad52bd18b6",
"source_model_id": "Qwen/Qwen2.5-0.5B-Instruct",
"source_model_revision": "7ae557604adf67be50417f59c2c2f167def9a775",
"is_reference": true,
"notes": "artifact_sha256 is the deterministic digest of every snapshot path and file byte",
"driver": {
"type": "transformers",
"model_path": "/run/media/popov/DATA/llm/safetensor/models/models--Qwen--Qwen2.5-0.5B-Instruct/snapshots/7ae557604adf67be50417f59c2c2f167def9a775",
"device": "cpu",
"dtype": "bfloat16",
"threads": 16
}
},
{
"id": "llama-cpp-near-lossless-quality",
"runtime": "llama.cpp-9991-e920c523",
"weight_format": "gguf",
"weight_quantization": "bfloat16",
"lane": "quality",
"device": "cpu",
"artifact_path": "/run/media/popov/DATA/llm/dgr-001/Qwen2.5-0.5B-Instruct-7ae5576-BF16.gguf",
"artifact_sha256": "e842fdc35d7f00fda95a54e1b51731ba1d196aea45065cc9f46925fdc1d6f862",
"source_model_id": "Qwen/Qwen2.5-0.5B-Instruct",
"source_model_revision": "7ae557604adf67be50417f59c2c2f167def9a775",
"is_reference": false,
"notes": "Converted directly from the exact mounted safetensors revision while preserving BF16 weights with pinned llama.cpp",
"driver": {
"type": "llama-cpp-server",
"binary": "/run/media/popov/d/DEV/llamacpp/llama.cpp/build/bin/llama-server",
"binary_sha256": "fd8fe612970f23e447f2e717cfa51665be06b8d7315ba60556e010f6bca510dd",
"gguf_path": "/run/media/popov/DATA/llm/dgr-001/Qwen2.5-0.5B-Instruct-7ae5576-BF16.gguf",
"device": "cpu",
"threads": 16,
"n_parallel": 4,
"context_per_slot": 512,
"n_gpu_layers": 0
}
},
{
"id": "llama-cpp-quantized-performance-fit",
"runtime": "llama.cpp-9991-e920c523",
"weight_format": "gguf",
"weight_quantization": "Q4_K_M",
"lane": "performance-fit",
"device": "cpu",
"artifact_path": "/run/media/popov/DATA/llm/dgr-001/Qwen2.5-0.5B-Instruct-7ae5576-Q4_K_M.gguf",
"artifact_sha256": "a88e3f570e2efeaf06b50df9859db2c70d8646aa3a2c94a14e14d5797a2921a5",
"source_model_id": "Qwen/Qwen2.5-0.5B-Instruct",
"source_model_revision": "7ae557604adf67be50417f59c2c2f167def9a775",
"is_reference": false,
"notes": "Quantized from the exact-revision F16 GGUF with pinned llama-quantize",
"driver": {
"type": "llama-cpp-server",
"binary": "/run/media/popov/d/DEV/llamacpp/llama.cpp/build/bin/llama-server",
"binary_sha256": "fd8fe612970f23e447f2e717cfa51665be06b8d7315ba60556e010f6bca510dd",
"gguf_path": "/run/media/popov/DATA/llm/dgr-001/Qwen2.5-0.5B-Instruct-7ae5576-Q4_K_M.gguf",
"device": "cpu",
"threads": 16,
"n_parallel": 4,
"context_per_slot": 512,
"n_gpu_layers": 0
}
}
]
}

51
.scratch/distributed-gguf-runtime/evidence/DGR-001/commands.txt Normal file
View File

@@ -0,0 +1,51 @@
# Exact source snapshot (already present on mounted storage)
SOURCE=/run/media/popov/DATA/llm/safetensor/models/models--Qwen--Qwen2.5-0.5B-Instruct/snapshots/7ae557604adf67be50417f59c2c2f167def9a775
LLAMA=/run/media/popov/d/DEV/llamacpp/llama.cpp
ROCM_PY=/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv-rocm/bin/python
PROJECT_PY=/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python
OUT=/run/media/popov/DATA/llm/dgr-001
# Converter support check (no writes)
$ROCM_PY $LLAMA/convert_hf_to_gguf.py "$SOURCE" --outtype f16 --outfile "$OUT/Qwen2.5-0.5B-Instruct-7ae5576-F16.gguf" --dry-run
# Exact-revision near-lossless and performance-fit artifacts
$ROCM_PY $LLAMA/convert_hf_to_gguf.py "$SOURCE" --outtype f16 --outfile "$OUT/Qwen2.5-0.5B-Instruct-7ae5576-F16.gguf"
$LLAMA/build/bin/llama-quantize "$OUT/Qwen2.5-0.5B-Instruct-7ae5576-F16.gguf" "$OUT/Qwen2.5-0.5B-Instruct-7ae5576-Q4_K_M.gguf" Q4_K_M
$ROCM_PY $LLAMA/convert_hf_to_gguf.py "$SOURCE" --outtype bf16 --outfile "$OUT/Qwen2.5-0.5B-Instruct-7ae5576-BF16.gguf"
# Runtime and artifact identity
git -C "$LLAMA" rev-parse HEAD
$LLAMA/build/bin/llama-server --version
sha256sum "$LLAMA/build/bin/llama-server" "$LLAMA/convert_hf_to_gguf.py" "$LLAMA/build/bin/llama-quantize"
sha256sum "$SOURCE/model.safetensors" "$OUT/Qwen2.5-0.5B-Instruct-7ae5576-BF16.gguf" "$OUT/Qwen2.5-0.5B-Instruct-7ae5576-Q4_K_M.gguf"
# Deterministic complete-snapshot digest used by benchmark-config.json
PYTHONPATH=packages/node $ROCM_PY - <<'PY'
from pathlib import Path
from meshnet_node.recipe_drivers import _artifact_sha256
print(_artifact_sha256(Path('/run/media/popov/DATA/llm/safetensor/models/models--Qwen--Qwen2.5-0.5B-Instruct/snapshots/7ae557604adf67be50417f59c2c2f167def9a775')))
PY
# Canonical opt-in local-real benchmark
MESHNET_ENABLE_REAL_INFERENCE_TESTS=1 PYTHONPATH=packages/node $ROCM_PY -m meshnet_node.recipe_benchmark \
--config .scratch/distributed-gguf-runtime/evidence/DGR-001/benchmark-config.json \
--json-out .scratch/distributed-gguf-runtime/evidence/DGR-001/results.json \
--summary-out .scratch/distributed-gguf-runtime/evidence/DGR-001/results.txt
# Distil the baseline and evaluate immutable v1
PYTHONPATH=packages/node $PROJECT_PY - <<'PY'
from pathlib import Path
import json
from meshnet_node.performance_contract import baseline_from_report, evaluate_contract, load_contract
root = Path('.scratch/distributed-gguf-runtime/evidence/DGR-001')
report = json.loads((root / 'results.json').read_text())
contract = load_contract(root / 'performance-contract.json')
(root / 'baseline.json').write_text(json.dumps(baseline_from_report(report), indent=2, sort_keys=True) + '\n')
(root / 'contract-evaluation.json').write_text(json.dumps(evaluate_contract(contract, report).to_dict(), indent=2, sort_keys=True) + '\n')
PY
# Deterministic verification
PYTHONPATH=packages/node $PROJECT_PY -m pytest -q tests/test_recipe_benchmark.py
PYTHONPATH=packages/node $PROJECT_PY -m pytest -q
PYTHONPATH=packages/node $PROJECT_PY -m compileall -q packages tests
git diff --check

71
.scratch/distributed-gguf-runtime/evidence/DGR-001/contract-evaluation.json Normal file
View File

@@ -0,0 +1,71 @@
{
"contract_version": 1,
"fit_benefit": true,
"plan_id": "dgr-001-controlled-whole-model-baseline-v1",
"quality_lane_pass": false,
"rationale": [
"the near-lossless quality lane failed: the GGUF runtime disagrees with the safetensors reference beyond what near-lossless weights can explain",
"a meaningful speed benefit was measured",
"a meaningful fit benefit was measured"
],
"recipes": [
{
"comparable": true,
"failures": 0,
"fit_benefit": false,
"incomparable_reason": "",
"lane": "quality",
"measurements": {
"aggregate_concurrency": 4,
"aggregate_throughput_speedup": 4.4882,
"artifact_size_ratio": 0.9946,
"artifact_size_win": false,
"compared_prompts": 3,
"decode_speedup": 1.9064,
"exact_match_rate": 0.3333,
"expected_prompts": 3,
"failure_rate": 0.0,
"mean_similarity": 0.9471,
"resident_memory_ratio": 0.5722,
"ttft_ratio": 1.1729
},
"quality_pass": false,
"reasons": [
"single-request decode 1.91x reference (>= 1.25x) at TTFT ratio 1.17",
"aggregate throughput at concurrency 4 is 4.49x reference (>= 1.25x)",
"peak resident memory is 0.57x reference (<= 0.75x)",
"quality lane exact-match 0.33 / similarity 0.947 versus the reference (fail)"
],
"recipe_id": "llama-cpp-near-lossless-quality",
"speed_benefit": false
},
{
"comparable": true,
"failures": 0,
"fit_benefit": true,
"incomparable_reason": "",
"lane": "performance-fit",
"measurements": {
"aggregate_concurrency": 4,
"aggregate_throughput_speedup": 4.3826,
"artifact_size_ratio": 0.398,
"artifact_size_win": true,
"decode_speedup": 3.6869,
"failure_rate": 0.0,
"resident_memory_ratio": 0.2793,
"ttft_ratio": 0.6367
},
"quality_pass": null,
"reasons": [
"single-request decode 3.69x reference (>= 1.25x) at TTFT ratio 0.64",
"aggregate throughput at concurrency 4 is 4.38x reference (>= 1.25x)",
"peak resident memory is 0.28x reference (<= 0.75x)"
],
"recipe_id": "llama-cpp-quantized-performance-fit",
"speed_benefit": true
}
],
"speed_benefit": true,
"stop_condition_met": true,
"verdict": "stop"
}

32
.scratch/distributed-gguf-runtime/evidence/DGR-001/known-unrelated-failure.md Normal file
View File

@@ -0,0 +1,32 @@
# Observed pre-existing intermittent tracker race
This file records an earlier unrelated timing observation; it is **not** the
final DGR-001 verification result.
Test:
```text
tests/test_tracker_routing.py::test_tracker_dashboard_can_cancel_inflight_proxy
```
One earlier full-suite run produced:
```text
1 failed, 745 passed, 13 skipped
```
A five-run isolated retry matrix reproduced the same rate on both branches:
```text
DGR-001 branch: 4/5 passed, 1/5 failed
clean d904c40: 4/5 passed, 1/5 failed
```
The final full-suite run on the exact hardened DGR-001 state completed green:
```text
749 passed, 13 skipped in 251.42s
```
The earlier race was therefore timing-sensitive, pre-existing, and unrelated
to the DGR-001 benchmark/contract files.

44
.scratch/distributed-gguf-runtime/evidence/DGR-001/performance-contract.json Normal file
View File

@@ -0,0 +1,44 @@
{
"schema_version": 1,
"contract_version": 1,
"locked_at": "2026-07-13T00:00:00Z",
"locked_by": "DGR-001",
"plan_id": "dgr-001-controlled-whole-model-baseline-v1",
"thresholds": {
"min_decode_speedup": 1.25,
"max_ttft_ratio": 1.25,
"min_aggregate_throughput_speedup": 1.25,
"max_resident_memory_ratio": 0.75,
"max_artifact_size_ratio": 0.6,
"min_quality_exact_match_rate": 0.9,
"min_quality_mean_similarity": 0.97,
"max_failure_rate": 0.0
},
"baseline": {
"status": "pending-real-evidence",
"required_evidence_class": "local-real",
"required_recipes": [
"transformers-safetensors-reference",
"llama-cpp-near-lossless-quality",
"llama-cpp-quantized-performance-fit"
],
"required_concurrency_levels": [
1,
4
],
"required_controlled_variables": [
"model architecture",
"model revision",
"machine and device",
"formatted prompts and context lengths",
"output length and greedy sampling policy"
],
"required_plan_sha256": "efe24690a9a7164bac6ab3fd0a6b22f078fc08aaefcfb96210ddf154e6050570",
"minimum_prompt_count": 3,
"minimum_repeats": 3,
"minimum_output_tokens": 32,
"required_device": "cpu"
},
"stop_condition": "Stop the native llama.cpp/GGUF track when, on the same machine and device as the Transformers/safetensors reference and under this plan, no performance-fit GGUF recipe delivers either a meaningful speed benefit (>=25% higher single-request decode tokens/sec without a >25% worse TTFT, or >=25% higher aggregate throughput under concurrency) or a meaningful fit benefit (>=25% lower peak resident memory), or when the near-lossless quality lane fails, which indicates a broken runtime rather than a quantization trade-off.",
"notes": "Quantized performance-fit output drift is reported as advisory only. It is not numerical-equivalence evidence. DGR-014 consumes this immutable v1 contract."
}

2474
.scratch/distributed-gguf-runtime/evidence/DGR-001/results.json Normal file
View File

File diff suppressed because it is too large Load Diff

10
.scratch/distributed-gguf-runtime/evidence/DGR-001/results.txt Normal file
View File

@@ -0,0 +1,10 @@
Recipe benchmark dgr-001-controlled-whole-model-baseline-v1 (local-real)
model Qwen/Qwen2.5-0.5B-Instruct@7ae557604adf67be50417f59c2c2f167def9a775
transformers-safetensors-reference [quality ] c= 1 ttft p50/p95 37.5/ 193.5 ms; prefill 671.8 tok/s; decode 46.1 tok/s; aggregate 40.3 tok/s; rss 1.94 GB; vram 0.00 GB; artifact 1.00 GB; failures 0
transformers-safetensors-reference [quality ] c= 4 ttft p50/p95 94.4/ 444.7 ms; prefill 247.6 tok/s; decode 13.1 tok/s; aggregate 47.1 tok/s; rss 2.18 GB; vram 0.00 GB; artifact 1.00 GB; failures 0
llama-cpp-near-lossless-quality [quality ] c= 1 ttft p50/p95 43.9/ 107.0 ms; prefill 1427.2 tok/s; decode 88.0 tok/s; aggregate 73.8 tok/s; rss 1.11 GB; vram 0.00 GB; artifact 0.99 GB; failures 0
llama-cpp-near-lossless-quality [quality ] c= 4 ttft p50/p95 33.6/ 128.5 ms; prefill 1077.8 tok/s; decode 73.9 tok/s; aggregate 211.4 tok/s; rss 1.13 GB; vram 0.00 GB; artifact 0.99 GB; failures 0
llama-cpp-quantized-performance-fit [performance-fit ] c= 1 ttft p50/p95 23.8/ 237.7 ms; prefill 783.4 tok/s; decode 170.1 tok/s; aggregate 110.0 tok/s; rss 0.54 GB; vram 0.00 GB; artifact 0.40 GB; failures 0
llama-cpp-quantized-performance-fit [performance-fit ] c= 4 ttft p50/p95 67.9/ 431.8 ms; prefill 474.3 tok/s; decode 83.5 tok/s; aggregate 206.4 tok/s; rss 0.57 GB; vram 0.00 GB; artifact 0.40 GB; failures 0
drift llama-cpp-near-lossless-quality vs transformers-safetensors-reference exact 0.33; similarity 0.947 (gated)
drift llama-cpp-quantized-performance-fit vs transformers-safetensors-reference exact 0.00; similarity 0.456 (advisory)

242
.scratch/distributed-gguf-runtime/evidence/DGR-002/README.md Normal file
View File

@@ -0,0 +1,242 @@
# DGR-002 — Adopt the versioned gRPC Shard protocol
Status: **done**. Every acceptance criterion is met with real command output.
Evidence class: **synthetic/unit** — this story defines a schema and proves both
languages agree on it. No model, GPU, network peer or benchmark is involved, and
none is claimed.
## 1. Summary
`packages/node/native/proto/shard_runtime.proto` is now the semantic contract for
the native Shard data plane: Protocol Buffers over gRPC/HTTP2 (ADR-0020). Python
and C++ both generate from it, and a shared committed conformance vector proves
they encode it identically — byte for byte.
Design decisions worth carrying forward:
- **Everything gRPC gives you is *also* in the schema.** Deadline, cancellation,
identity and flow control are carried as fields, not left to HTTP/2 metadata,
because the existing relay carries these frames as **opaque binary**. A relayed
frame has no HTTP/2 context to inherit a deadline or a channel identity from.
If it is not in the schema, it does not survive the relay.
- **Cancellation is both in-band and out-of-band.** `CancelSignal` rides the
stream; `Cancel` is also a unary RPC. A cancel that can only travel down a
stream that flow control has wedged is not a cancel.
- **Checksums cover the uncompressed payload.** Compression is a per-hop
transport decision (reusing the existing `activation_compression` policies), so
a checksum over the compressed frame would be invalidated by a hop that merely
chose differently.
- **Application-level flow-control credits, not just HTTP/2 windows.** HTTP/2
bounds *bytes in flight*; it does not bound how much *work* a worker has queued,
and a relayed frame gets no window at all. Credits bound queue occupancy and KV
pressure, and negotiation takes the strictest bound of either peer so a sender
cannot talk a worker into unbounded queues.
## 2. Files changed
New:
| Path | What |
|---|---|
| `packages/node/native/proto/shard_runtime.proto` | The schema (sha256 `9e211660…`, see `protocol.json`) |
| `packages/node/native/CMakeLists.txt` | C++ generation + build wiring + ctest |
| `packages/node/native/tests/test_shard_protocol_conformance.cpp` | C++ conformance test |
| `packages/node/native/testdata/*.binpb` | Committed cross-language vectors |
| `packages/node/native/README.md` | How to regenerate and build |
| `packages/node/meshnet_node/native_protocol/__init__.py` | Public Python surface |
| `packages/node/meshnet_node/native_protocol/codec.py` | Bundle encode/decode, fragmentation, CRC32C, chunking, FC negotiation |
| `packages/node/meshnet_node/native_protocol/conformance.py` | Canonical vectors shared by both languages |
| `packages/node/meshnet_node/native_protocol/generated/` | Generated Python stubs (committed) |
| `scripts/generate_native_protocol.py` | Python generation, with `--check` |
| `scripts/generate_protocol_goldens.py` | Vector generation, with `--check` |
| `scripts/bootstrap_native_toolchain.sh` | Builds protobuf C++ from source |
| `tests/test_native_shard_protocol.py` | 45 Python tests |
Modified:
- `packages/node/pyproject.toml` — added runtime floors `grpcio>=1.82.1` and
`protobuf>=7.35.0`, matching the committed generated-code requirements; new
`proto` extra pinning `grpcio-tools==1.82.1`.
- `packages/node/meshnet_node/activation_compression.py` — optional bounded zstd
output for untrusted protocol frames; existing callers remain compatible.
- `packages/node/meshnet_node/native_protocol/__init__.py` — exports negotiated
bound constants and whole-session-message validation.
The canonical PRD marks only DGR-002 passed. `git status` before this story was clean.
## 3. Commands and real results
See `commands.txt` for the exact ordered list. Results:
```
python scripts/generate_native_protocol.py --check -> generated stubs are up to date
python scripts/generate_protocol_goldens.py --check -> conformance vectors are up to date
cmake -S packages/node/native -B build/native -DCMAKE_PREFIX_PATH=/tmp/pbsrc/install
-- gRPC C++ not found: building message types only (sufficient for the conformance test)
cmake --build build/native -j -> Built target shard_protocol_conformance
ctest --test-dir build/native --output-on-failure -> 1/1 Test #1: shard_protocol_conformance ... Passed
100% tests passed out of 1
cmp build/native/cpp_roundtrip.binpb \
packages/node/native/testdata/session_request_golden.binpb -> identical (exit 0)
pytest -q tests/test_native_shard_protocol.py -> 45 passed
pytest -q tests/test_native_shard_protocol.py \
tests/test_activation_compression.py -> 51 passed
pytest -q (final full suite) -> 728 passed, 12 skipped
pytest -q tests/test_tracker_routing.py::test_tracker_dashboard_can_cancel_inflight_proxy
(after an earlier flaky full-suite failure) -> 1 passed, 1 passed, 1 passed
clean minimum-runtime import + codec smoke test -> passed
grpcio==1.82.1, protobuf==7.35.0
compileall -q packages tests -> OK (exit 0)
git diff --check -> clean (exit 0)
```
The C++ lane was rebuilt from scratch by Ralph (`rm -rf build/native`) using only
the documented commands, and reproduced the same result. During controller
review the user explicitly chose not to repeat the destructive build-directory
cleanup, so the independent controller relied on the recorded CMake/CTest run
while reproducing every Python/generation/full-suite gate.
### Controller review corrections
Independent controller review found and fixed two classes of issue before
integration:
1. Generated stubs required gRPC 1.82.1 and Protobuf 7.35.0, while the initial
package metadata allowed much older runtimes that could fail at import time.
2. Flow-control bounds were described but not enforced by the reference decoder.
Tensor declarations, shape rank/dimensions, fragment/tensor counts, fragments,
wire bodies, whole bundles, complete session messages (including envelope
overhead), and zstd window/output expansion are now fail-closed against the
negotiated/default bounds. Unspecified bundle versions, compression and
checksums are rejected rather than interpreted as valid data.
3. Negotiated initial credits could exceed `max_inflight_chunks`; credits are now
capped by the settled in-flight limit.
Controller results: protocol tests `45 passed`; protocol plus shared compression
tests `51 passed`; final full suite `728 passed, 12 skipped`. A clean environment
at the declared minimum gRPC/Protobuf runtime versions imported both generated
stub modules and round-tripped the codec. Generation checks, `compileall`, static
secret scan, and `git diff --check` all passed.
### Full-suite note — a pre-existing flaky test
`tests/test_tracker_routing.py::test_tracker_dashboard_can_cancel_inflight_proxy`
is **flaky on a clean tree, independent of this story**. Reproduction, run
*before any DGR-002 file existed* (working tree clean, `git status` empty):
```
pytest -q -> 1 failed, 682 passed, 12 skipped
FAILED tests/test_tracker_routing.py::test_tracker_dashboard_can_cancel_inflight_proxy
# same test, three consecutive isolated runs on the same clean tree:
pytest -q tests/test_tracker_routing.py::test_tracker_dashboard_can_cancel_inflight_proxy
-> 1 passed in 1.76s
-> 1 failed in 4.39s
-> 1 passed in 1.10s
```
It is a timing race in proxy cancellation (a 3-second in-flight generation raced
against the cancel assertion), not a deterministic failure, and it touches no code
this story changes. One controller full-suite run reported exactly that one failure
(`1 failed, 719 passed, 12 skipped`); three immediate isolated retries all passed
in 1.11 seconds, and the final exact-code full suite was green (`728 passed,
12 skipped`). It is flagged for whoever owns the tracker cancel path and is **not**
fixed here, since silently touching another story's code is out of scope.
## 4. Acceptance criteria
| Criterion | Where it is proven |
|---|---|
| Schema for capability, health, session stream, release, cancellation | `shard_runtime.proto` `service ShardRuntime`; `test_service_exposes_capability_health_session_release_and_cancel` |
| One long-lived bidi stream per Activation Seam, with deadlines, cancellation, flow control, structured errors | `rpc Session (stream) returns (stream)`; `test_session_is_one_long_lived_bidirectional_stream`; `Envelope.deadline_unix_nanos`, `CancelSignal` + unary `Cancel`, `FlowControl`, `ShardError` |
| Bounded chunking for prefill; small decode fast path | `ChunkInfo` + `plan_prefill_chunks` (128-token bound, ADR-0008); `DecodeStep`; `test_prefill_is_split_into_bounded_token_aligned_chunks`, `test_decode_fast_path_is_much_smaller_than_a_full_envelope_chunk` |
| Envelope carries schema version, work id, session id, epoch, fingerprint, range/effective start, phase, position, idempotency step, cache expectation, compression, checksum | `Envelope` + `NamedTensor`; `test_envelope_carries_every_field_the_protocol_promises` asserts against the **descriptor**, so deleting a field from the `.proto` fails the test |
| Versioned named-tensor bundle: name, shape, dtype, byte order, fragments | `TensorBundle`/`NamedTensor`/`TensorFragment`; `test_named_tensor_bundle_is_versioned_and_fully_described`, `test_bundle_round_trips_multiple_named_tensors` |
| Round-trip + compatibility tests in Python and C++ | 45 Python tests; C++ `ctest` 1/1; cross-language byte equality |
| Targeted pytest passes | 45 passed |
| `compileall packages tests` | exit 0 |
| `git diff --check` | exit 0 |
| Default tests deterministic, download-free, credit-free, GPU-free | Pure in-memory protobuf; no model, no network, no GPU |
| Full deterministic pytest passes, or pre-existing failure recorded | Final exact-code run: 728 passed, 12 skipped; earlier sole flaky failure documented with clean-tree reproduction and 3/3 passing retries |
## 5. How the cross-language claim is actually earned
Two codecs that each round-trip their own output prove only that each is
self-consistent. Instead:
1. Python builds the canonical `SessionRequest` and commits its bytes.
2. The C++ test parses **those** bytes, asserts every field, recomputes the CRC32C
**from the polynomial in independent C++ code**, reassembles the multi-fragment
tensor, and re-serializes to `cpp_roundtrip.binpb`.
3. `test_cpp_and_python_agree_byte_for_byte` asserts that file equals the golden.
Compatibility is tested in both languages: an unknown field from a newer peer
survives a parse/serialize hop (a Shard forwards activations — silently stripping
fields would corrupt a route it is merely a waypoint on), and a sparse message
from an older peer parses to proto3 defaults.
## 6. Limitations and deferred work
- **gRPC C++ was not built or linked.** The C++ lane verifies the *schema* (message
types), not a running gRPC C++ server, because this machine has no gRPC C++ stack
and building it is a large dependency the conformance test does not need.
`CMakeLists.txt` already generates and exports `shard_runtime_grpc` when
`find_package(gRPC)` succeeds. **DGR-008 must install gRPC C++ and extend
`scripts/bootstrap_native_toolchain.sh`.**
- **No wire is exercised.** No client, server, or stream lifecycle exists yet — no
deadline actually fires, no credit is actually consumed. This story defines and
proves the contract; DGR-008/DGR-009 implement it.
- The protobuf C++ toolchain used here was installed to `/tmp/pbsrc/install` (ephemeral).
`scripts/bootstrap_native_toolchain.sh` reproduces it; prefer a durable prefix such
as `build/native-toolchain`.
- `crc32c` has a pure-Python fallback (used here) and picks up `google_crc32c` when
present. The fallback is byte-exact but slow; a worker on the hot path should install
the native package. Not a correctness limitation.
- Compression on the wire is zstd-or-none only, matching the existing seam.
## 7. Compatibility and migration notes
- **This does not change the existing HTTP activation wire.** `X-Meshnet-Wire` stays
at `2` and the legacy `/forward` path is untouched. The native protocol is a
*separate* contract with its own `SchemaVersion`, starting at 1. Nothing in this
story is on any live request path — it is additive.
- Semantics are deliberately preserved from the existing ADRs so the two transports
mean the same thing: `effective_start_layer` (ADR-0012), `CacheMode`/`expected_past_len`
and `ERROR_CODE_CACHE_MISS` mapping to today's HTTP 409 `cache_miss` (ADR-0022),
bfloat16 boundary dtype and 128-token prefill chunks (ADR-0008), fingerprint/recipe
identity mirroring the capability report (ADR-0023).
- `TensorFragment` field 5 (`uncompressed_size`) is **reserved**: it was removed
because `NamedTensor.total_bytes` is the single source of truth. Never recycle it —
a recycled field number is the one schema change peers cannot detect, because the
bytes still parse.
- Committed Python stubs are guarded by `--check` in the test suite, so they cannot
drift from the schema unnoticed.
## 8. Handoff to dependent stories
- **DGR-003 (runtime recipe/fingerprint):** populate `Fingerprint`
(`model_artifact_digest`, `runtime_recipe_digest`, `recipe_id`, `recipe_version`,
`catalogue_version`). The mismatch outcome is already specified:
`ERROR_CODE_FINGERPRINT_MISMATCH`. Do not invent a second identity struct.
- **DGR-005/006 (range loading, architecture boundary):** the boundary payload is a
**named bundle**, not a bare tensor — a boundary needing more than one tensor is
already representable. Execute `[effective_start_layer, end_layer)`, never from
`start_layer`.
- **DGR-007 (concurrent sessions/KV):** isolate on `(route_session_id, route_epoch)`.
`CacheExpectation`/`CacheResult` and `ERROR_CODE_CACHE_MISS` are the contract; a
decode step whose `expected_past_len` does not match **must** miss, never fall back
to a silent stateless forward. `idempotency_step` means a retried step is
acknowledged (`Ack.duplicate`), not re-applied — re-applying advances the KV cache
twice and desynchronises the route.
- **DGR-008 (C++ worker):** link `shard_runtime_grpc` from `CMakeLists.txt`; you must
first install gRPC C++ (see limitations). Honour `FlowControl` credits and the
`max_chunk_bytes` bound. Use `packages/node/meshnet_node/native_protocol/codec.py`
as the reference for fragment reassembly and checksum validation.
- **DGR-009 (Meshnet integration):** the relay may carry these serialized frames as
opaque binary — that is exactly why deadline/cancel/identity are in-band. Do not add
a second control plane.
- **Anyone editing the schema:** run both `--check` scripts; if a vector legitimately
changes, regenerate it and say so, because the C++ test asserts those exact bytes.

45
.scratch/distributed-gguf-runtime/evidence/DGR-002/commands.txt Normal file
View File

@@ -0,0 +1,45 @@
# DGR-002 — exact commands, in order. Run from the repository root.
# Interpreter: <repo>/.venv/bin/python (CPython 3.14.6). Deterministic, GPU-free,
# no model download, no API credits.
# --- toolchain (this machine had no protoc, no cmake, no protobuf C++ headers)
.venv/bin/python -m pip install grpcio-tools==1.82.1 grpcio==1.82.1 cmake==4.4.0
scripts/bootstrap_native_toolchain.sh /tmp/pbsrc/install # protobuf C++ 33.1 + abseil 20250814.1
# --- schema generation (Python stubs; committed)
.venv/bin/python scripts/generate_native_protocol.py
.venv/bin/python scripts/generate_native_protocol.py --check # -> "generated stubs are up to date"
# --- cross-language conformance vectors (committed)
.venv/bin/python scripts/generate_protocol_goldens.py
.venv/bin/python scripts/generate_protocol_goldens.py --check # -> "conformance vectors are up to date"
# --- C++ generation, build and conformance test
cmake -S packages/node/native -B build/native -DCMAKE_PREFIX_PATH=/tmp/pbsrc/install
cmake --build build/native -j"$(nproc)"
ctest --test-dir build/native --output-on-failure # -> 1/1 Passed
cmp build/native/cpp_roundtrip.binpb packages/node/native/testdata/session_request_golden.binpb
# --- Python tests
.venv/bin/python -m pytest -q tests/test_native_shard_protocol.py # -> 29 passed
.venv/bin/python -m pytest -q # full suite
# --- repository gates
.venv/bin/python -m compileall -q packages tests
git diff --check
# --- independent controller review after Ralph
PYTHONPATH=packages/node .venv/bin/python -m pytest -q tests/test_native_shard_protocol.py
# -> 45 passed
PYTHONPATH=packages/node .venv/bin/python -m pytest -q \
tests/test_native_shard_protocol.py tests/test_activation_compression.py
# -> 51 passed
PYTHONPATH=packages/node .venv/bin/python -m pytest -q
# -> final exact-code run: 728 passed, 12 skipped
for i in 1 2 3; do PYTHONPATH=packages/node .venv/bin/python -m pytest -q \
tests/test_tracker_routing.py::test_tracker_dashboard_can_cancel_inflight_proxy; done
# -> 1 passed, 1 passed, 1 passed
# clean minimum-runtime venv: protobuf==7.35.0 grpcio==1.82.1
# generated pb2 + pb2_grpc imports and one-byte codec round trip -> passed
# The user chose to rely on Ralph's recorded successful C++ CMake/CTest run
# rather than repeat deletion of an isolated generated build directory.

95
.scratch/distributed-gguf-runtime/evidence/DGR-002/protocol.json Normal file
View File

@@ -0,0 +1,95 @@
{
"schema_version": "SCHEMA_VERSION_1",
"bundle_version": 1,
"proto_path": "packages/node/native/proto/shard_runtime.proto",
"proto_sha256": "9e211660b3fcefc88bcdf3851c3571088c00349aacb5adc5ef45083c83d0cce2",
"protoc": "grpc_tools 1.82.1 (python) / protobuf 33.1 (C++)",
"service": {
"GetCapability": {
"client_streaming": false,
"server_streaming": false
},
"Health": {
"client_streaming": false,
"server_streaming": false
},
"Session": {
"client_streaming": true,
"server_streaming": true
},
"Release": {
"client_streaming": false,
"server_streaming": false
},
"Cancel": {
"client_streaming": false,
"server_streaming": false
}
},
"envelope_fields": [
"cache_expectation",
"chunk",
"deadline_unix_nanos",
"fingerprint",
"idempotency_step",
"phase",
"position",
"route_epoch",
"route_session_id",
"schema_version",
"shard_range",
"work_id"
],
"named_tensor_fields": [
"byte_order",
"checksum",
"compression",
"dtype",
"fragments",
"name",
"shape",
"total_bytes"
],
"phases": [
"PHASE_UNSPECIFIED",
"PHASE_PREFILL",
"PHASE_DECODE",
"PHASE_RELEASE",
"PHASE_CANCEL"
],
"error_codes": [
"ERROR_CODE_UNSPECIFIED",
"ERROR_CODE_SCHEMA_UNSUPPORTED",
"ERROR_CODE_FINGERPRINT_MISMATCH",
"ERROR_CODE_EPOCH_STALE",
"ERROR_CODE_SHARD_RANGE_MISMATCH",
"ERROR_CODE_CACHE_MISS",
"ERROR_CODE_RESOURCE_EXHAUSTED",
"ERROR_CODE_PAYLOAD_CORRUPT",
"ERROR_CODE_CANCELLED",
"ERROR_CODE_DEADLINE_EXCEEDED",
"ERROR_CODE_FLOW_CONTROL_VIOLATION",
"ERROR_CODE_INTERNAL"
],
"bounds": {
"max_prefill_chunk_tokens": 128,
"max_chunk_bytes": 4194304,
"max_fragment_bytes": 1048576,
"max_inflight_chunks": 8,
"max_fragments_per_tensor": 64,
"max_tensors_per_bundle": 64,
"max_tensor_rank": 8,
"max_tensor_dimension": 2147483647,
"whole_session_message_enforced": true
},
"golden_vectors": {
"session_request_golden.binpb": "c2c3df8a717ddeae7bd99624d2c7f34c09a518988de990237fe313b75cff0817",
"capability_report_golden.binpb": "71ac5f150775f398515b43a63596a5cbe8d2ad607e7e4de56bd44fbe7987080c"
},
"verification": {
"python_protocol_tests": "45 passed",
"python_protocol_and_compression_tests": "51 passed",
"full_suite": "728 passed, 12 skipped",
"minimum_runtime": "grpcio 1.82.1 / protobuf 7.35.0 passed import and codec smoke"
}
}

2
.scratch/distributed-gguf-runtime/issues/01-lock-the-safetensors-versus-gguf-performance-contract.md
View File

@@ -1,6 +1,6 @@
# 01 — Lock the safetensors-versus-GGUF performance contract
Status: ready-for-agent
Status: done
## Mandatory fresh-session context

34
.scratch/distributed-gguf-runtime/issues/02-adopt-the-versioned-grpc-shard-protocol.md
View File

@@ -1,6 +1,6 @@
# 02 — Adopt the versioned gRPC Shard protocol
Status: ready-for-agent
Status: done
## Mandatory fresh-session context
@@ -22,22 +22,22 @@ As a node developer, I need a battle-proven streaming protocol so that Python an
## Acceptance criteria
- [ ] Add a Protocol Buffers schema for capability, health, session stream, release, and cancellation operations.
- [ ] Define one long-lived bidirectional gRPC stream per Route Session Activation Seam with deadlines, cancellation, flow control, and structured errors.
- [ ] Define bounded chunking for prefill and a small decode fast path.
- [ ] Carry schema version, request/work ID, Route Session ID, route epoch, artifact/recipe fingerprint, Shard range/effective start, phase, position, idempotency step, cache expectation, compression, and checksum.
- [ ] Define a versioned named-tensor bundle with per-tensor name, shape, dtype, byte order, and payload fragments.
- [ ] Add generated-schema round-trip and compatibility tests in Python and C++.
- [ ] Targeted pytest tests pass
- [ ] python -m compileall packages tests passes for Python changes
- [ ] git diff --check passes
- [ ] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
- [ ] Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
- [ ] Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
- [ ] Read and verify every dependency evidence README before relying on dependency behavior
- [ ] Preserve all pre-existing working-tree changes and stage only files belonging to this story
- [ ] Write .scratch/distributed-gguf-runtime/evidence/DGR-002/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
- [ ] Update only this story issue to Status: done after every acceptance criterion and quality gate passes
- [x] Add a Protocol Buffers schema for capability, health, session stream, release, and cancellation operations.
- [x] Define one long-lived bidirectional gRPC stream per Route Session Activation Seam with deadlines, cancellation, flow control, and structured errors.
- [x] Define bounded chunking for prefill and a small decode fast path.
- [x] Carry schema version, request/work ID, Route Session ID, route epoch, artifact/recipe fingerprint, Shard range/effective start, phase, position, idempotency step, cache expectation, compression, and checksum.
- [x] Define a versioned named-tensor bundle with per-tensor name, shape, dtype, byte order, and payload fragments.
- [x] Add generated-schema round-trip and compatibility tests in Python and C++.
- [x] Targeted pytest tests pass
- [x] python -m compileall packages tests passes for Python changes
- [x] git diff --check passes
- [x] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
- [x] Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
- [x] Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
- [x] Read and verify every dependency evidence README before relying on dependency behavior
- [x] Preserve all pre-existing working-tree changes and stage only files belonging to this story
- [x] Write .scratch/distributed-gguf-runtime/evidence/DGR-002/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
- [x] Update only this story issue to Status: done after every acceptance criterion and quality gate passes
## Dependency handoff

36
.scratch/distributed-gguf-runtime/prd.json
View File

@@ -6,7 +6,7 @@
{
"id": "DGR-001",
"title": "Lock the safetensors-versus-GGUF performance contract",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/01-lock-the-safetensors-versus-gguf-performance-contract.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs a runtime engineer, I need a controlled baseline so that GGUF work proceeds from measured speed, memory, and quality rather than reputation.",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/01-lock-the-safetensors-versus-gguf-performance-contract.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker—not a stitched collection of runtimes.\n\nAs a runtime engineer, I need a controlled baseline so that GGUF work proceeds from measured speed, memory, and quality rather than reputation.",
"acceptanceCriteria": [
"Benchmark the same model architecture/revision, machine, prompts, context lengths, output lengths, sampling policy, and concurrency across the current Transformers/safetensors recipe and whole-model llama.cpp recipes.",
"Separate correctness/quality lanes from quantized performance/fit lanes instead of claiming BF16 and Q4 are numerically equivalent.",
@@ -27,14 +27,14 @@
"Update only this story issue to Status: done after every acceptance criterion and quality gate passes"
],
"priority": 2,
"passes": false,
"passes": true,
"notes": "Source issue: .scratch/distributed-gguf-runtime/issues/01-lock-the-safetensors-versus-gguf-performance-contract.md",
"dependsOn": []
},
{
"id": "DGR-002",
"title": "Adopt the versioned gRPC Shard protocol",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/02-adopt-the-versioned-grpc-shard-protocol.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs a node developer, I need a battle-proven streaming protocol so that Python and C++ Shards communicate without a custom socket protocol.",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/02-adopt-the-versioned-grpc-shard-protocol.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker—not a stitched collection of runtimes.\n\nAs a node developer, I need a battle-proven streaming protocol so that Python and C++ Shards communicate without a custom socket protocol.",
"acceptanceCriteria": [
"Add a Protocol Buffers schema for capability, health, session stream, release, and cancellation operations.",
"Define one long-lived bidirectional gRPC stream per Route Session Activation Seam with deadlines, cancellation, flow control, and structured errors.",
@@ -54,14 +54,14 @@
"Update only this story issue to Status: done after every acceptance criterion and quality gate passes"
],
"priority": 1,
"passes": false,
"passes": true,
"notes": "Source issue: .scratch/distributed-gguf-runtime/issues/02-adopt-the-versioned-grpc-shard-protocol.md",
"dependsOn": []
},
{
"id": "DGR-003",
"title": "Define exact Artifact and runtime recipe identity",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/03-define-exact-artifact-and-runtime-recipe-identity.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs the Tracker, I need exact compatibility identity so that only numerically and operationally compatible Shards form an Inference Route.",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/03-define-exact-artifact-and-runtime-recipe-identity.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker—not a stitched collection of runtimes.\n\nAs the Tracker, I need exact compatibility identity so that only numerically and operationally compatible Shards form an Inference Route.",
"acceptanceCriteria": [
"Separate weight quantization, activation dtype, compute dtype, KV dtype/layout, tokenizer revision, architecture adapter, backend, and runtime version.",
"Bind derivative or split artifacts to an exact source Model Artifact hash and Shard range.",
@@ -89,7 +89,7 @@
{
"id": "DGR-004",
"title": "Create the reproducible pinned llama.cpp patch stack",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/04-create-the-reproducible-pinned-llama-cpp-patch-stack.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs a maintainer, I need a small auditable fork boundary so that upstream updates do not turn the runtime into an unmaintainable stitched codebase.",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/04-create-the-reproducible-pinned-llama-cpp-patch-stack.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker—not a stitched collection of runtimes.\n\nAs a maintainer, I need a small auditable fork boundary so that upstream updates do not turn the runtime into an unmaintainable stitched codebase.",
"acceptanceCriteria": [
"Pin one exact llama.cpp commit through a reproducible source dependency mechanism.",
"Store a numbered minimal patch stack separately from Meshnet networking code.",
@@ -120,7 +120,7 @@
{
"id": "DGR-005",
"title": "Implement dense-Llama range-aware GGUF ownership",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/05-implement-dense-llama-range-aware-gguf-ownership.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs a node, I need to map only my assigned dense-Llama Shard so that aggregate consumer memory can hold a model larger than one node.",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/05-implement-dense-llama-range-aware-gguf-ownership.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker—not a stitched collection of runtimes.\n\nAs a node, I need to map only my assigned dense-Llama Shard so that aggregate consumer memory can hold a model larger than one node.",
"acceptanceCriteria": [
"Register and allocate only `blk.N.*` tensors in the assigned range.",
"Load embeddings only for the head and final norm/LM head only for the tail, including tied embeddings.",
@@ -151,7 +151,7 @@
{
"id": "DGR-006",
"title": "Implement architecture-defined boundary input/output",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/06-implement-architecture-defined-boundary-input-output.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs a Shard, I need to consume and emit the correct transformer boundary state so that disjoint processes reproduce whole-model execution.",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/06-implement-architecture-defined-boundary-input-output.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker—not a stitched collection of runtimes.\n\nAs a Shard, I need to consume and emit the correct transformer boundary state so that disjoint processes reproduce whole-model execution.",
"acceptanceCriteria": [
"Head accepts token IDs and owns token embedding.",
"Middle/tail bypass token embedding and accept the named boundary bundle.",
@@ -183,7 +183,7 @@
{
"id": "DGR-007",
"title": "Add isolated concurrent local Hot KV State",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/07-add-isolated-concurrent-local-hot-kv-state.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs a client, I need concurrent Route Sessions to retain independent per-Shard cache so that one request cannot clear or corrupt another.",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/07-add-isolated-concurrent-local-hot-kv-state.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker—not a stitched collection of runtimes.\n\nAs a client, I need concurrent Route Sessions to retain independent per-Shard cache so that one request cannot clear or corrupt another.",
"acceptanceCriteria": [
"Map `(Route Session ID, route epoch)` to an isolated llama sequence or bounded context.",
"Allocate KV only for owned layers.",
@@ -214,7 +214,7 @@
{
"id": "DGR-008",
"title": "Build the standalone C++ gRPC Shard worker",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/08-build-the-standalone-c-grpc-shard-worker.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs a node runtime, I need one supervised native process so that llama.cpp internals remain behind a stable project-owned protocol.",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/08-build-the-standalone-c-grpc-shard-worker.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker—not a stitched collection of runtimes.\n\nAs a node runtime, I need one supervised native process so that llama.cpp internals remain behind a stable project-owned protocol.",
"acceptanceCriteria": [
"Worker exposes capability, health, session stream, release, cancellation, and metrics services from DGR-002.",
"Worker loads one exact Artifact/recipe/Shard identity and refuses mismatched requests.",
@@ -249,7 +249,7 @@
{
"id": "DGR-009",
"title": "Integrate the native worker with Meshnet",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/09-integrate-the-native-worker-with-meshnet.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs the existing node service, I need a GGUF Shard backend adapter so that the Tracker, relay, billing, telemetry, and capability admission remain the sole control plane.",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/09-integrate-the-native-worker-with-meshnet.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker—not a stitched collection of runtimes.\n\nAs the existing node service, I need a GGUF Shard backend adapter so that the Tracker, relay, billing, telemetry, and capability admission remain the sole control plane.",
"acceptanceCriteria": [
"Implement the existing model-backend surface without changing Transformers behavior.",
"Registration carries exact validated GGUF recipe, Shard, backend and concurrency/KV capacity.",
@@ -281,7 +281,7 @@
{
"id": "DGR-010",
"title": "Pass local real-model two-process acceptance",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/10-pass-local-real-model-two-process-acceptance.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs a release engineer, I need real local distributed parity before involving network variability.",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/10-pass-local-real-model-two-process-acceptance.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker—not a stitched collection of runtimes.\n\nAs a release engineer, I need real local distributed parity before involving network variability.",
"acceptanceCriteria": [
"Two local worker processes open disjoint dense-Llama ranges from the certified Artifact.",
"Prefill and at least 32 greedy decode tokens match whole-model llama.cpp within the certified tolerance.",
@@ -314,7 +314,7 @@
{
"id": "DGR-011",
"title": "Pass a real heterogeneous two-machine route",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/11-pass-a-real-heterogeneous-two-machine-route.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs a consumer-hardware operator, I need two physical machines to execute one GGUF model so that the distributed claim is real.",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/11-pass-a-real-heterogeneous-two-machine-route.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker—not a stitched collection of runtimes.\n\nAs a consumer-hardware operator, I need two physical machines to execute one GGUF model so that the distributed claim is real.",
"acceptanceCriteria": [
"Tracker selects two physical nodes with disjoint Shards and one exact certified recipe/compatibility class.",
"Actual CPU/GPU execution occurs on both nodes; synthetic workers do not satisfy acceptance.",
@@ -347,7 +347,7 @@
{
"id": "DGR-012",
"title": "Implement continuous batching and bounded admission",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/12-implement-continuous-batching-and-bounded-admission.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs a node operator, I need active sessions batched safely so that concurrency increases aggregate throughput rather than serializing every request.",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/12-implement-continuous-batching-and-bounded-admission.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker—not a stitched collection of runtimes.\n\nAs a node operator, I need active sessions batched safely so that concurrency increases aggregate throughput rather than serializing every request.",
"acceptanceCriteria": [
"Node scheduler admits sessions against weight, KV, scratch, and queue budgets.",
"Compatible decode steps from multiple sessions form llama.cpp batches while preserving per-session positions and outputs.",
@@ -380,7 +380,7 @@
{
"id": "DGR-013",
"title": "Harden failure, cancellation, and restart semantics",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/13-harden-failure-cancellation-and-restart-semantics.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs a client, I need failures to be bounded and explicit so that distributed speed does not come with hanging or corrupted generations.",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/13-harden-failure-cancellation-and-restart-semantics.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker—not a stitched collection of runtimes.\n\nAs a client, I need failures to be bounded and explicit so that distributed speed does not come with hanging or corrupted generations.",
"acceptanceCriteria": [
"Deadlines and heartbeat/health loss terminate blocked stream operations.",
"Cancellation propagates across every Shard and releases local KV and queued buffers.",
@@ -413,7 +413,7 @@
{
"id": "DGR-014",
"title": "Enforce the GGUF-versus-safetensors release gate",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/14-enforce-the-gguf-versus-safetensors-release-gate.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs the product owner, I need an end-to-end comparison so that the native runtime ships only if it advances model access or performance.",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/14-enforce-the-gguf-versus-safetensors-release-gate.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker—not a stitched collection of runtimes.\n\nAs the product owner, I need an end-to-end comparison so that the native runtime ships only if it advances model access or performance.",
"acceptanceCriteria": [
"Run current distributed safetensors and distributed GGUF routes on the same certified model/hardware/network scenario where technically comparable.",
"Report quality, TTFT, prefill/decode throughput, aggregate concurrency throughput, p95 latency, seam cost, memory, KV pressure, failures, and cleanup.",
@@ -448,7 +448,7 @@
{
"id": "DGR-015",
"title": "Add and certify a Qwen3/Qwen3-MoE adapter",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/15-add-and-certify-a-qwen3-qwen3-moe-adapter.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs a client seeking top models, I need a separately certified MoE-capable architecture after the dense runtime proves stable.",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/15-add-and-certify-a-qwen3-qwen3-moe-adapter.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker—not a stitched collection of runtimes.\n\nAs a client seeking top models, I need a separately certified MoE-capable architecture after the dense runtime proves stable.",
"acceptanceCriteria": [
"Implement explicit tensor ownership, router/top-k, expert/shared-expert, Q/K normalization, boundary bundle, and cache semantics for the selected Qwen3 family recipe.",
"Do not reuse the dense-Llama adapter through unchecked name substitutions.",
@@ -480,7 +480,7 @@
{
"id": "DGR-016",
"title": "Produce the upstream llama.cpp collaboration package",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/16-produce-the-upstream-llama-cpp-collaboration-package.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs a maintainer, I need narrow upstreamable proposals so that our patch burden can shrink without asking llama.cpp to own Meshnet networking.",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/16-produce-the-upstream-llama-cpp-collaboration-package.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker—not a stitched collection of runtimes.\n\nAs a maintainer, I need narrow upstreamable proposals so that our patch burden can shrink without asking llama.cpp to own Meshnet networking.",
"acceptanceCriteria": [
"Separate generic llama.cpp hooks from Meshnet protocol/control-plane code.",
"Prepare minimal reproducible examples and tests for range-aware loading, boundary input/output, and layer-filtered KV.",

24
packages/node/meshnet_node/activation_compression.py
View File

@@ -99,7 +99,12 @@ def compress_activation(body: bytes, policy: CompressionPolicy) -> CompressionRe
return CompressionResult(candidate, "zstd", len(body), len(candidate), time.monotonic() - started, "compressed")
def decompress_activation(body: bytes, encoding: str | None) -> CompressionResult:
def decompress_activation(
body: bytes,
encoding: str | None,
*,
max_output_bytes: int | None = None,
) -> CompressionResult:
"""Decode a modern zstd body or preserve a legacy raw body with metrics."""
started = time.monotonic()
if not encoding:
@@ -110,8 +115,23 @@ def decompress_activation(body: bytes, encoding: str | None) -> CompressionResul
import zstandard as zstd
except ImportError as exc:
raise ValueError("zstd support is unavailable") from exc
if max_output_bytes is not None and max_output_bytes < 0:
raise ValueError("max_output_bytes must be non-negative")
try:
raw = zstd.ZstdDecompressor().decompress(body)
if max_output_bytes is None:
raw = zstd.ZstdDecompressor().decompress(body)
else:
# Cap both decoder window allocation and bytes read. zstandard's
# max_window_size unit is KiB.
max_window_kib = max(1024, (max_output_bytes + 1023) // 1024)
decompressor = zstd.ZstdDecompressor(max_window_size=max_window_kib)
# `decompress(max_output_size=...)` may trust a frame's advertised
# content size. A bounded stream read enforces the limit regardless
# of frame metadata and detects trailing expansion with one byte.
with decompressor.stream_reader(body) as reader:
raw = reader.read(max_output_bytes + 1)
if len(raw) > max_output_bytes:
raise ValueError("zstd activation body exceeds its output limit")
except zstd.ZstdError as exc:
raise ValueError("invalid zstd activation body") from exc
return CompressionResult(raw, "zstd", len(body), len(raw), time.monotonic() - started, "decompressed")

73
packages/node/meshnet_node/native_protocol/__init__.py Normal file
View File

@@ -0,0 +1,73 @@
"""The native Shard protocol: Protobuf over gRPC/HTTP2 (ADR-0020).
`packages/node/native/proto/shard_runtime.proto` is the contract. This package
is how Python speaks it: the generated stubs plus the validation, framing and
chunking rules that the stubs cannot express.
Import the message types from here rather than reaching into `.generated`, so
the location of build output stays an implementation detail::
from meshnet_node.native_protocol import pb, encode_tensor, decode_bundle
"""
from __future__ import annotations
from .generated import shard_runtime_pb2 as pb
from .codec import (
BUNDLE_VERSION,
DEFAULT_MAX_CHUNK_BYTES,
DEFAULT_MAX_FRAGMENT_BYTES,
DEFAULT_MAX_FRAGMENTS_PER_TENSOR,
DEFAULT_MAX_INFLIGHT_CHUNKS,
DEFAULT_MAX_PREFILL_CHUNK_TOKENS,
DEFAULT_MAX_TENSORS_PER_BUNDLE,
DEFAULT_MAX_TENSOR_DIMENSION,
DEFAULT_MAX_TENSOR_RANK,
HIDDEN_STATES,
SCHEMA_VERSION,
PayloadCorrupt,
PrefillChunk,
ProtocolError,
checksum_of,
crc32c,
decode_bundle,
decode_tensor,
default_flow_control,
encode_bundle,
encode_tensor,
expected_bytes,
itemsize,
negotiate_flow_control,
plan_prefill_chunks,
validate_session_message_size,
)
__all__ = [
"BUNDLE_VERSION",
"DEFAULT_MAX_CHUNK_BYTES",
"DEFAULT_MAX_FRAGMENT_BYTES",
"DEFAULT_MAX_FRAGMENTS_PER_TENSOR",
"DEFAULT_MAX_INFLIGHT_CHUNKS",
"DEFAULT_MAX_PREFILL_CHUNK_TOKENS",
"DEFAULT_MAX_TENSORS_PER_BUNDLE",
"DEFAULT_MAX_TENSOR_DIMENSION",
"DEFAULT_MAX_TENSOR_RANK",
"HIDDEN_STATES",
"SCHEMA_VERSION",
"PayloadCorrupt",
"PrefillChunk",
"ProtocolError",
"checksum_of",
"crc32c",
"decode_bundle",
"decode_tensor",
"default_flow_control",
"encode_bundle",
"encode_tensor",
"expected_bytes",
"itemsize",
"negotiate_flow_control",
"pb",
"plan_prefill_chunks",
"validate_session_message_size",
]

527
packages/node/meshnet_node/native_protocol/codec.py Normal file
View File

@@ -0,0 +1,527 @@
"""Encode and decode the native Shard protocol's named-tensor bundles.
The generated stubs give us message *structure*; they cannot enforce the
invariants that keep a distributed forward correct. A bundle whose declared
shape disagrees with its byte count, whose fragments leave a hole, or whose
checksum does not match is not a slightly-wrong activation — it is silently
wrong tokens for the rest of the generation. So decoding is validating: every
path into a tensor's bytes goes through :func:`decode_tensor`, which refuses a
payload it cannot fully account for.
Compression is a transport optimisation and is decided by the same policy layer
the existing HTTP seam already uses (``activation_compression``), so a node's
tuned thresholds apply to both transports.
"""
from __future__ import annotations
from dataclasses import dataclass
import struct
from typing import Iterable, Sequence
from ..activation_compression import (
CompressionPolicy,
compress_activation,
decompress_activation,
)
from .generated import shard_runtime_pb2 as pb
# The schema generation this build speaks. A peer offering something else is
# rejected at the handshake rather than being half-understood.
SCHEMA_VERSION = pb.SCHEMA_VERSION_1
# Generation of the tensor-bundle layout, versioned independently of the
# protocol so a boundary payload can evolve without a protocol bump.
BUNDLE_VERSION = 1
# Token-aligned prefill chunk bound. 128 tokens is the size ADR-0008 already
# uses on the HTTP seam; keeping it identical means seam bytes stay comparable
# across transports.
DEFAULT_MAX_PREFILL_CHUNK_TOKENS = 128
# gRPC's default maximum receive size. Fragmenting below it keeps us inside the
# default limits of any conformant peer instead of requiring every client to
# raise its window.
DEFAULT_MAX_CHUNK_BYTES = 4 * 1024 * 1024
# Leave room for envelope and framing overhead inside one chunk message.
DEFAULT_MAX_FRAGMENT_BYTES = 1024 * 1024
DEFAULT_MAX_INFLIGHT_CHUNKS = 8
DEFAULT_MAX_FRAGMENTS_PER_TENSOR = 64
DEFAULT_MAX_TENSORS_PER_BUNDLE = 64
DEFAULT_MAX_TENSOR_RANK = 8
DEFAULT_MAX_TENSOR_DIMENSION = (1 << 31) - 1
# Canonical boundary tensor name for a dense transformer hidden state.
HIDDEN_STATES = "hidden_states"
_DTYPE_ITEMSIZE: dict[int, int] = {
pb.DTYPE_BFLOAT16: 2,
pb.DTYPE_FLOAT16: 2,
pb.DTYPE_FLOAT32: 4,
pb.DTYPE_INT32: 4,
pb.DTYPE_INT64: 8,
pb.DTYPE_UINT8: 1,
pb.DTYPE_INT8: 1,
pb.DTYPE_BOOL: 1,
}
class ProtocolError(Exception):
"""A peer sent something this build cannot safely interpret."""
class PayloadCorrupt(ProtocolError):
"""A tensor payload failed validation: size, coverage, or checksum."""
def itemsize(dtype: int) -> int:
try:
return _DTYPE_ITEMSIZE[dtype]
except KeyError:
raise ProtocolError(f"unsupported dtype {dtype}") from None
def expected_bytes(
shape: Sequence[int],
dtype: int,
*,
max_rank: int = DEFAULT_MAX_TENSOR_RANK,
max_dimension: int = DEFAULT_MAX_TENSOR_DIMENSION,
max_bytes: int | None = None,
) -> int:
"""Byte count a tensor of `shape` and `dtype` must occupy."""
if len(shape) > max_rank:
raise ProtocolError(f"tensor rank {len(shape)} exceeds limit {max_rank}")
if any(dim < 0 or dim > max_dimension for dim in shape):
raise ProtocolError(
f"dimension outside 0..{max_dimension} in shape {list(shape)}"
)
size = itemsize(dtype)
count = 1
for dim in shape:
count *= dim
if max_bytes is not None and count * size > max_bytes:
raise ProtocolError(f"tensor shape {list(shape)} exceeds byte limit {max_bytes}")
return count * size
# --- CRC32C ----------------------------------------------------------------
#
# CRC32C (Castagnoli), not zlib's CRC32: it is the checksum gRPC, and the
# storage systems these payloads pass through, already use, and hardware
# implements it. `google_crc32c` is used when present; the table fallback keeps
# the default test suite dependency-free.
_CRC32C_POLY = 0x82F63B78
_CRC32C_TABLE: list[int] = []
for _i in range(256):
_c = _i
for _ in range(8):
_c = (_c >> 1) ^ (_CRC32C_POLY if _c & 1 else 0)
_CRC32C_TABLE.append(_c)
try: # pragma: no cover - depends on an optional native package
from google_crc32c import value as _fast_crc32c
except ImportError: # pragma: no cover
_fast_crc32c = None
def crc32c(data: bytes) -> int:
if _fast_crc32c is not None: # pragma: no cover - optional fast path
return _fast_crc32c(data)
crc = 0xFFFFFFFF
for byte in data:
crc = (crc >> 8) ^ _CRC32C_TABLE[(crc ^ byte) & 0xFF]
return crc ^ 0xFFFFFFFF
def checksum_of(data: bytes) -> pb.Checksum:
return pb.Checksum(
algorithm=pb.CHECKSUM_ALGORITHM_CRC32C,
value=struct.pack(">I", crc32c(data)),
)
# --- Tensors ---------------------------------------------------------------
def encode_tensor(
name: str,
data: bytes,
shape: Sequence[int],
dtype: int = pb.DTYPE_BFLOAT16,
*,
policy: CompressionPolicy | None = None,
max_chunk_bytes: int = DEFAULT_MAX_CHUNK_BYTES,
max_fragment_bytes: int = DEFAULT_MAX_FRAGMENT_BYTES,
max_fragments: int = DEFAULT_MAX_FRAGMENTS_PER_TENSOR,
) -> pb.NamedTensor:
"""Build a NamedTensor, compressing and fragmenting as needed.
`data` is the uncompressed little-endian payload. The checksum is taken over
it *before* compression so it stays valid whichever framing a hop chooses.
"""
if max_chunk_bytes <= 0 or max_fragment_bytes <= 0 or max_fragments <= 0:
raise ProtocolError("tensor byte/count bounds must be positive")
declared = expected_bytes(shape, dtype, max_bytes=max_chunk_bytes)
if len(data) != declared:
raise ProtocolError(
f"tensor {name!r} declares shape {list(shape)} ({declared} bytes) "
f"but carries {len(data)} bytes"
)
body = data
compression = pb.COMPRESSION_NONE
if policy is not None:
result = compress_activation(data, policy)
if result.compressed:
body = result.body
compression = pb.COMPRESSION_ZSTD
tensor = pb.NamedTensor(
name=name,
shape=list(shape),
dtype=dtype,
byte_order=pb.BYTE_ORDER_LITTLE_ENDIAN,
total_bytes=len(data),
compression=compression,
checksum=checksum_of(data),
)
# Fragment the wire body (compressed if we compressed). Offsets walk the
# wire body so a receiver can verify coverage without assuming arrival
# order; a zstd frame is not decodable per fragment, so reassembly comes
# first and decompression happens once, in decode_tensor.
slices = [body[i : i + max_fragment_bytes] for i in range(0, len(body), max_fragment_bytes)]
if not slices:
# A zero-element tensor is legal (e.g. an empty mask) and still needs a
# fragment, so coverage checks have something to verify.
slices = [b""]
if len(slices) > max_fragments:
raise ProtocolError(
f"tensor {name!r} needs {len(slices)} fragments, exceeding limit {max_fragments}"
)
offset = 0
for index, piece in enumerate(slices):
tensor.fragments.append(
pb.TensorFragment(
fragment_index=index,
fragment_count=len(slices),
byte_offset=offset,
payload=piece,
)
)
offset += len(piece)
return tensor
def decode_tensor(
tensor: pb.NamedTensor,
*,
max_chunk_bytes: int = DEFAULT_MAX_CHUNK_BYTES,
max_fragment_bytes: int = DEFAULT_MAX_FRAGMENT_BYTES,
max_fragments: int = DEFAULT_MAX_FRAGMENTS_PER_TENSOR,
) -> bytes:
"""Reassemble, decompress and validate a NamedTensor's payload.
Raises PayloadCorrupt rather than returning a payload it cannot fully
account for: a hole in the fragments or a bad checksum means the activation
is not what the sender computed, and continuing would corrupt the route.
"""
if max_chunk_bytes <= 0 or max_fragment_bytes <= 0 or max_fragments <= 0:
raise ProtocolError("negotiated byte/count bounds must be positive")
if tensor.total_bytes > max_chunk_bytes:
raise ProtocolError(
f"tensor {tensor.name!r} declares {tensor.total_bytes} bytes, exceeding "
f"the {max_chunk_bytes}-byte negotiated chunk bound"
)
if tensor.byte_order == pb.BYTE_ORDER_BIG_ENDIAN:
raise ProtocolError(f"tensor {tensor.name!r} is big-endian; wire order is little-endian")
if tensor.byte_order != pb.BYTE_ORDER_LITTLE_ENDIAN:
raise ProtocolError(f"tensor {tensor.name!r} declares no byte order")
declared = expected_bytes(
tensor.shape, tensor.dtype, max_bytes=max_chunk_bytes
)
if declared != tensor.total_bytes:
raise PayloadCorrupt(
f"tensor {tensor.name!r} shape {list(tensor.shape)} implies {declared} bytes "
f"but declares {tensor.total_bytes}"
)
if not tensor.fragments:
raise PayloadCorrupt(f"tensor {tensor.name!r} carries no fragments")
if len(tensor.fragments) > max_fragments:
raise PayloadCorrupt(
f"tensor {tensor.name!r} carries {len(tensor.fragments)} fragments, "
f"exceeding limit {max_fragments}"
)
if any(len(fragment.payload) > max_fragment_bytes for fragment in tensor.fragments):
raise PayloadCorrupt(
f"tensor {tensor.name!r} carries a fragment larger than "
f"{max_fragment_bytes} bytes"
)
wire_bytes = sum(len(fragment.payload) for fragment in tensor.fragments)
if wire_bytes > max_chunk_bytes:
raise PayloadCorrupt(
f"tensor {tensor.name!r} wire body exceeds the "
f"{max_chunk_bytes}-byte negotiated chunk bound"
)
fragments = sorted(tensor.fragments, key=lambda f: f.byte_offset)
count = fragments[0].fragment_count
if any(f.fragment_count != count for f in fragments):
raise PayloadCorrupt(f"tensor {tensor.name!r} has inconsistent fragment_count")
if len(fragments) != count:
raise PayloadCorrupt(
f"tensor {tensor.name!r} expects {count} fragments but carries {len(fragments)}"
)
if {f.fragment_index for f in fragments} != set(range(count)):
raise PayloadCorrupt(f"tensor {tensor.name!r} has duplicate or missing fragment indices")
# Contiguity: offsets must tile the body exactly, with no hole and no overlap.
body = bytearray()
for fragment in fragments:
if fragment.byte_offset != len(body):
raise PayloadCorrupt(
f"tensor {tensor.name!r} fragment {fragment.fragment_index} starts at "
f"{fragment.byte_offset}, expected {len(body)}"
)
body.extend(fragment.payload)
if tensor.compression == pb.COMPRESSION_ZSTD:
try:
data = decompress_activation(
bytes(body), "zstd", max_output_bytes=tensor.total_bytes
).body
except ValueError as exc:
raise PayloadCorrupt(
f"tensor {tensor.name!r} has invalid bounded zstd payload"
) from exc
elif tensor.compression == pb.COMPRESSION_NONE:
data = bytes(body)
else:
raise ProtocolError(
f"tensor {tensor.name!r} uses unspecified or unsupported compression"
)
if len(data) != tensor.total_bytes:
raise PayloadCorrupt(
f"tensor {tensor.name!r} declares {tensor.total_bytes} bytes but "
f"reassembled {len(data)}"
)
algorithm = tensor.checksum.algorithm
if algorithm == pb.CHECKSUM_ALGORITHM_CRC32C:
if tensor.checksum.value != struct.pack(">I", crc32c(data)):
raise PayloadCorrupt(f"tensor {tensor.name!r} failed its CRC32C check")
elif algorithm != pb.CHECKSUM_ALGORITHM_NONE:
raise ProtocolError(
f"tensor {tensor.name!r} uses unspecified or unsupported checksum algorithm"
)
return data
def encode_bundle(
tensors: Iterable[pb.NamedTensor],
*,
max_chunk_bytes: int = DEFAULT_MAX_CHUNK_BYTES,
max_tensors: int = DEFAULT_MAX_TENSORS_PER_BUNDLE,
) -> pb.TensorBundle:
if max_chunk_bytes <= 0 or max_tensors <= 0:
raise ProtocolError("bundle byte/count bounds must be positive")
tensor_list = list(tensors)
if len(tensor_list) > max_tensors:
raise ProtocolError(
f"bundle carries {len(tensor_list)} tensors, exceeding limit {max_tensors}"
)
bundle = pb.TensorBundle(bundle_version=BUNDLE_VERSION, tensors=tensor_list)
if bundle.ByteSize() > max_chunk_bytes:
raise ProtocolError(
f"serialized tensor bundle exceeds the {max_chunk_bytes}-byte "
"negotiated chunk bound"
)
return bundle
def decode_bundle(
bundle: pb.TensorBundle,
*,
max_chunk_bytes: int = DEFAULT_MAX_CHUNK_BYTES,
max_fragment_bytes: int = DEFAULT_MAX_FRAGMENT_BYTES,
max_fragments: int = DEFAULT_MAX_FRAGMENTS_PER_TENSOR,
max_tensors: int = DEFAULT_MAX_TENSORS_PER_BUNDLE,
) -> dict[str, bytes]:
"""Validate every tensor in a bundle and return name -> payload."""
if bundle.bundle_version != BUNDLE_VERSION:
raise ProtocolError(
f"bundle version {bundle.bundle_version} is not supported by this build "
f"({BUNDLE_VERSION})"
)
if (
max_chunk_bytes <= 0
or max_fragment_bytes <= 0
or max_fragments <= 0
or max_tensors <= 0
):
raise ProtocolError("negotiated byte/count bounds must be positive")
if len(bundle.tensors) > max_tensors:
raise ProtocolError(
f"bundle carries {len(bundle.tensors)} tensors, exceeding limit {max_tensors}"
)
if bundle.ByteSize() > max_chunk_bytes:
raise ProtocolError(
f"serialized tensor bundle exceeds the {max_chunk_bytes}-byte "
"negotiated chunk bound"
)
payloads: dict[str, bytes] = {}
for tensor in bundle.tensors:
if not tensor.name:
raise ProtocolError("bundle carries an unnamed tensor")
if tensor.name in payloads:
raise ProtocolError(f"bundle carries duplicate tensor {tensor.name!r}")
payloads[tensor.name] = decode_tensor(
tensor,
max_chunk_bytes=max_chunk_bytes,
max_fragment_bytes=max_fragment_bytes,
max_fragments=max_fragments,
)
return payloads
def validate_session_message_size(
message: pb.SessionRequest | pb.SessionResponse,
*,
max_chunk_bytes: int = DEFAULT_MAX_CHUNK_BYTES,
) -> int:
"""Reject an oversized complete stream frame, including protobuf overhead.
Bundle validation alone is insufficient because the envelope and oneof
framing are part of the same gRPC message. Senders call this immediately
before writing; receivers configure gRPC's receive limit to the same value
and call it again before semantic decoding.
"""
if max_chunk_bytes <= 0:
raise ProtocolError("max_chunk_bytes must be positive")
if not isinstance(message, (pb.SessionRequest, pb.SessionResponse)):
raise ProtocolError("size validation requires a session request or response")
size = message.ByteSize()
if size > max_chunk_bytes:
raise ProtocolError(
f"serialized session message is {size} bytes, exceeding the "
f"{max_chunk_bytes}-byte negotiated chunk bound"
)
return size
# --- Bounded prefill chunking ----------------------------------------------
@dataclass(frozen=True)
class PrefillChunk:
"""One token-aligned slice of a prefill."""
chunk_index: int
chunk_count: int
first_position: int
token_count: int
@property
def final_chunk(self) -> bool:
return self.chunk_index == self.chunk_count - 1
def chunk_info(self) -> pb.ChunkInfo:
return pb.ChunkInfo(
chunk_index=self.chunk_index,
chunk_count=self.chunk_count,
final_chunk=self.final_chunk,
)
def position(self) -> pb.PositionSpan:
return pb.PositionSpan(
first_position=self.first_position, token_count=self.token_count
)
def plan_prefill_chunks(
total_tokens: int,
*,
first_position: int = 0,
max_tokens: int = DEFAULT_MAX_PREFILL_CHUNK_TOKENS,
) -> list[PrefillChunk]:
"""Split a prefill into bounded, token-aligned chunks.
Splits fall on token boundaries only (ADR-0008): a fragment of a token's
hidden state is not a thing a receiver can execute.
"""
if total_tokens <= 0:
raise ProtocolError("a prefill must carry at least one token")
if max_tokens <= 0:
raise ProtocolError("max_tokens must be positive")
count = (total_tokens + max_tokens - 1) // max_tokens
chunks = []
for index in range(count):
offset = index * max_tokens
chunks.append(
PrefillChunk(
chunk_index=index,
chunk_count=count,
first_position=first_position + offset,
token_count=min(max_tokens, total_tokens - offset),
)
)
return chunks
def default_flow_control() -> pb.FlowControl:
return pb.FlowControl(
credits_granted=DEFAULT_MAX_INFLIGHT_CHUNKS,
max_inflight_chunks=DEFAULT_MAX_INFLIGHT_CHUNKS,
max_chunk_bytes=DEFAULT_MAX_CHUNK_BYTES,
max_prefill_chunk_tokens=DEFAULT_MAX_PREFILL_CHUNK_TOKENS,
)
def negotiate_flow_control(
proposed: pb.FlowControl, limits: pb.FlowControl
) -> pb.FlowControl:
"""Settle a stream's limits: the strictest bound of either peer wins.
Taking the minimum means neither peer can raise the other's ceiling, so a
misconfigured — or hostile — sender cannot talk a worker into unbounded
queues by proposing a large window.
"""
def _min(a: int, b: int, fallback: int) -> int:
candidates = [v for v in (a, b) if v > 0]
return min(candidates) if candidates else fallback
max_inflight_chunks = _min(
proposed.max_inflight_chunks,
limits.max_inflight_chunks,
DEFAULT_MAX_INFLIGHT_CHUNKS,
)
credits_granted = min(
_min(
proposed.credits_granted,
limits.credits_granted,
DEFAULT_MAX_INFLIGHT_CHUNKS,
),
max_inflight_chunks,
)
return pb.FlowControl(
credits_granted=credits_granted,
max_inflight_chunks=max_inflight_chunks,
max_chunk_bytes=_min(
proposed.max_chunk_bytes, limits.max_chunk_bytes, DEFAULT_MAX_CHUNK_BYTES
),
max_prefill_chunk_tokens=_min(
proposed.max_prefill_chunk_tokens,
limits.max_prefill_chunk_tokens,
DEFAULT_MAX_PREFILL_CHUNK_TOKENS,
),
)

141
packages/node/meshnet_node/native_protocol/conformance.py Normal file
View File

@@ -0,0 +1,141 @@
"""Canonical conformance vectors for the native Shard protocol.
Two independently-written codecs that each round-trip their own output prove
nothing about each other. These vectors are the shared reference: Python builds
the canonical message, the bytes are committed under
`packages/node/native/testdata/`, and the C++ test parses those exact bytes and
asserts the same field values. A change that alters the wire meaning of a field
breaks the vector in both languages instead of drifting silently in one.
The vector deliberately exercises every field group the protocol promises to
carry — identity, epoch, fingerprint, range, phase, position, idempotency,
cache expectation, deadline, chunking, compression, checksum and a multi-
fragment named tensor — so it doubles as an executable inventory of the
contract.
"""
from __future__ import annotations
import pathlib
from . import codec
from .generated import shard_runtime_pb2 as pb
# Committed vectors live beside the schema, under `packages/node/native/`.
# parents[2] is `packages/node`: native_protocol -> meshnet_node -> node.
TESTDATA_DIR = pathlib.Path(__file__).resolve().parents[2] / "native/testdata"
GOLDEN_SESSION_REQUEST = "session_request_golden.binpb"
GOLDEN_CAPABILITY_REPORT = "capability_report_golden.binpb"
# Written by the C++ conformance test into its build tree; the Python test picks
# it up when present to prove the two languages agree byte-for-byte.
CPP_ROUNDTRIP = "cpp_roundtrip.binpb"
# Fixed, non-default values. Every one is chosen to be distinguishable from a
# proto3 default so an unset field can never masquerade as a correct one.
WORK_ID = "work-7f3a"
ROUTE_SESSION_ID = "rs-2b91"
ROUTE_EPOCH = 7
IDEMPOTENCY_STEP = 42
FIRST_POSITION = 256
TOKEN_COUNT = 128
EXPECTED_PAST_LEN = 256
DEADLINE_UNIX_NANOS = 1_800_000_000_000_000_000
MODEL_ARTIFACT_DIGEST = "sha256:1f0c9d2e"
RUNTIME_RECIPE_DIGEST = "sha256:ab77e410"
RECIPE_ID = "llama-gguf-q4km-rocm"
RECIPE_VERSION = "3"
CATALOGUE_VERSION = "2026.07.1"
START_LAYER = 12
END_LAYER = 24
EFFECTIVE_START_LAYER = 16
HIDDEN_SIZE = 8
# A payload big enough to force more than one fragment at the bound below, so
# the vector actually exercises reassembly rather than the one-fragment path.
FRAGMENT_BYTES = 64
TENSOR_SHAPE = [1, TOKEN_COUNT, HIDDEN_SIZE]
def canonical_payload() -> bytes:
"""Deterministic bfloat16-sized payload for the canonical tensor."""
total = codec.expected_bytes(TENSOR_SHAPE, pb.DTYPE_BFLOAT16)
return bytes((i * 7 + 11) % 256 for i in range(total))
def canonical_session_request() -> pb.SessionRequest:
"""The canonical prefill chunk carried on a session stream."""
tensor = codec.encode_tensor(
codec.HIDDEN_STATES,
canonical_payload(),
TENSOR_SHAPE,
pb.DTYPE_BFLOAT16,
max_fragment_bytes=FRAGMENT_BYTES,
)
envelope = pb.Envelope(
schema_version=pb.SCHEMA_VERSION_1,
work_id=WORK_ID,
route_session_id=ROUTE_SESSION_ID,
route_epoch=ROUTE_EPOCH,
fingerprint=pb.Fingerprint(
model_artifact_digest=MODEL_ARTIFACT_DIGEST,
runtime_recipe_digest=RUNTIME_RECIPE_DIGEST,
recipe_id=RECIPE_ID,
recipe_version=RECIPE_VERSION,
catalogue_version=CATALOGUE_VERSION,
),
shard_range=pb.ShardRange(
start_layer=START_LAYER,
end_layer=END_LAYER,
effective_start_layer=EFFECTIVE_START_LAYER,
),
phase=pb.PHASE_PREFILL,
position=pb.PositionSpan(
first_position=FIRST_POSITION, token_count=TOKEN_COUNT
),
idempotency_step=IDEMPOTENCY_STEP,
cache_expectation=pb.CacheExpectation(
mode=pb.CACHE_MODE_PREFILL, expected_past_len=EXPECTED_PAST_LEN
),
deadline_unix_nanos=DEADLINE_UNIX_NANOS,
chunk=pb.ChunkInfo(chunk_index=1, chunk_count=3, final_chunk=False),
)
return pb.SessionRequest(
chunk=pb.ActivationChunk(
envelope=envelope, bundle=codec.encode_bundle([tensor])
)
)
def canonical_capability_report() -> pb.CapabilityReport:
"""The canonical capability report a worker answers admission with."""
return pb.CapabilityReport(
schema_version=pb.SCHEMA_VERSION_1,
fingerprint=pb.Fingerprint(
model_artifact_digest=MODEL_ARTIFACT_DIGEST,
runtime_recipe_digest=RUNTIME_RECIPE_DIGEST,
recipe_id=RECIPE_ID,
recipe_version=RECIPE_VERSION,
catalogue_version=CATALOGUE_VERSION,
),
shard_range=pb.ShardRange(
start_layer=START_LAYER,
end_layer=END_LAYER,
effective_start_layer=EFFECTIVE_START_LAYER,
),
backend="rocm",
device="gfx1151",
validated=True,
max_concurrent_sessions=4,
max_context_tokens=8192,
flow_control=codec.default_flow_control(),
accepted_compression=[pb.COMPRESSION_NONE, pb.COMPRESSION_ZSTD],
supported_schema_versions=[pb.SCHEMA_VERSION_1],
validated_at_unix_nanos=DEADLINE_UNIX_NANOS,
)
def serialize(message) -> bytes:
"""Serialize deterministically, so committed golden bytes are stable."""
return message.SerializeToString(deterministic=True)

2
packages/node/meshnet_node/native_protocol/generated/__init__.py Normal file
View File

@@ -0,0 +1,2 @@
# Generated by scripts/generate_native_protocol.py. Do not edit.
"""Generated protobuf/gRPC stubs for the native Shard protocol."""

117
packages/node/meshnet_node/native_protocol/generated/shard_runtime_pb2.py Normal file
View File

File diff suppressed because one or more lines are too long

525
packages/node/meshnet_node/native_protocol/generated/shard_runtime_pb2.pyi Normal file
View File

@@ -0,0 +1,525 @@
from google.protobuf.internal import containers as _containers
from google.protobuf.internal import enum_type_wrapper as _enum_type_wrapper
from google.protobuf import descriptor as _descriptor
from google.protobuf import message as _message
from collections.abc import Iterable as _Iterable, Mapping as _Mapping
from typing import ClassVar as _ClassVar, Optional as _Optional, Union as _Union
DESCRIPTOR: _descriptor.FileDescriptor
class SchemaVersion(int, metaclass=_enum_type_wrapper.EnumTypeWrapper):
__slots__ = ()
SCHEMA_VERSION_UNSPECIFIED: _ClassVar[SchemaVersion]
SCHEMA_VERSION_1: _ClassVar[SchemaVersion]
class DType(int, metaclass=_enum_type_wrapper.EnumTypeWrapper):
__slots__ = ()
DTYPE_UNSPECIFIED: _ClassVar[DType]
DTYPE_BFLOAT16: _ClassVar[DType]
DTYPE_FLOAT16: _ClassVar[DType]
DTYPE_FLOAT32: _ClassVar[DType]
DTYPE_INT32: _ClassVar[DType]
DTYPE_INT64: _ClassVar[DType]
DTYPE_UINT8: _ClassVar[DType]
DTYPE_INT8: _ClassVar[DType]
DTYPE_BOOL: _ClassVar[DType]
class ByteOrder(int, metaclass=_enum_type_wrapper.EnumTypeWrapper):
__slots__ = ()
BYTE_ORDER_UNSPECIFIED: _ClassVar[ByteOrder]
BYTE_ORDER_LITTLE_ENDIAN: _ClassVar[ByteOrder]
BYTE_ORDER_BIG_ENDIAN: _ClassVar[ByteOrder]
class Compression(int, metaclass=_enum_type_wrapper.EnumTypeWrapper):
__slots__ = ()
COMPRESSION_UNSPECIFIED: _ClassVar[Compression]
COMPRESSION_NONE: _ClassVar[Compression]
COMPRESSION_ZSTD: _ClassVar[Compression]
class ChecksumAlgorithm(int, metaclass=_enum_type_wrapper.EnumTypeWrapper):
__slots__ = ()
CHECKSUM_ALGORITHM_UNSPECIFIED: _ClassVar[ChecksumAlgorithm]
CHECKSUM_ALGORITHM_NONE: _ClassVar[ChecksumAlgorithm]
CHECKSUM_ALGORITHM_CRC32C: _ClassVar[ChecksumAlgorithm]
class Phase(int, metaclass=_enum_type_wrapper.EnumTypeWrapper):
__slots__ = ()
PHASE_UNSPECIFIED: _ClassVar[Phase]
PHASE_PREFILL: _ClassVar[Phase]
PHASE_DECODE: _ClassVar[Phase]
PHASE_RELEASE: _ClassVar[Phase]
PHASE_CANCEL: _ClassVar[Phase]
class CacheMode(int, metaclass=_enum_type_wrapper.EnumTypeWrapper):
__slots__ = ()
CACHE_MODE_UNSPECIFIED: _ClassVar[CacheMode]
CACHE_MODE_STATELESS: _ClassVar[CacheMode]
CACHE_MODE_PREFILL: _ClassVar[CacheMode]
CACHE_MODE_DECODE: _ClassVar[CacheMode]
class ErrorCode(int, metaclass=_enum_type_wrapper.EnumTypeWrapper):
__slots__ = ()
ERROR_CODE_UNSPECIFIED: _ClassVar[ErrorCode]
ERROR_CODE_SCHEMA_UNSUPPORTED: _ClassVar[ErrorCode]
ERROR_CODE_FINGERPRINT_MISMATCH: _ClassVar[ErrorCode]
ERROR_CODE_EPOCH_STALE: _ClassVar[ErrorCode]
ERROR_CODE_SHARD_RANGE_MISMATCH: _ClassVar[ErrorCode]
ERROR_CODE_CACHE_MISS: _ClassVar[ErrorCode]
ERROR_CODE_RESOURCE_EXHAUSTED: _ClassVar[ErrorCode]
ERROR_CODE_PAYLOAD_CORRUPT: _ClassVar[ErrorCode]
ERROR_CODE_CANCELLED: _ClassVar[ErrorCode]
ERROR_CODE_DEADLINE_EXCEEDED: _ClassVar[ErrorCode]
ERROR_CODE_FLOW_CONTROL_VIOLATION: _ClassVar[ErrorCode]
ERROR_CODE_INTERNAL: _ClassVar[ErrorCode]
class ServingState(int, metaclass=_enum_type_wrapper.EnumTypeWrapper):
__slots__ = ()
SERVING_STATE_UNSPECIFIED: _ClassVar[ServingState]
SERVING_STATE_SERVING: _ClassVar[ServingState]
SERVING_STATE_DRAINING: _ClassVar[ServingState]
SERVING_STATE_NOT_SERVING: _ClassVar[ServingState]
SCHEMA_VERSION_UNSPECIFIED: SchemaVersion
SCHEMA_VERSION_1: SchemaVersion
DTYPE_UNSPECIFIED: DType
DTYPE_BFLOAT16: DType
DTYPE_FLOAT16: DType
DTYPE_FLOAT32: DType
DTYPE_INT32: DType
DTYPE_INT64: DType
DTYPE_UINT8: DType
DTYPE_INT8: DType
DTYPE_BOOL: DType
BYTE_ORDER_UNSPECIFIED: ByteOrder
BYTE_ORDER_LITTLE_ENDIAN: ByteOrder
BYTE_ORDER_BIG_ENDIAN: ByteOrder
COMPRESSION_UNSPECIFIED: Compression
COMPRESSION_NONE: Compression
COMPRESSION_ZSTD: Compression
CHECKSUM_ALGORITHM_UNSPECIFIED: ChecksumAlgorithm
CHECKSUM_ALGORITHM_NONE: ChecksumAlgorithm
CHECKSUM_ALGORITHM_CRC32C: ChecksumAlgorithm
PHASE_UNSPECIFIED: Phase
PHASE_PREFILL: Phase
PHASE_DECODE: Phase
PHASE_RELEASE: Phase
PHASE_CANCEL: Phase
CACHE_MODE_UNSPECIFIED: CacheMode
CACHE_MODE_STATELESS: CacheMode
CACHE_MODE_PREFILL: CacheMode
CACHE_MODE_DECODE: CacheMode
ERROR_CODE_UNSPECIFIED: ErrorCode
ERROR_CODE_SCHEMA_UNSUPPORTED: ErrorCode
ERROR_CODE_FINGERPRINT_MISMATCH: ErrorCode
ERROR_CODE_EPOCH_STALE: ErrorCode
ERROR_CODE_SHARD_RANGE_MISMATCH: ErrorCode
ERROR_CODE_CACHE_MISS: ErrorCode
ERROR_CODE_RESOURCE_EXHAUSTED: ErrorCode
ERROR_CODE_PAYLOAD_CORRUPT: ErrorCode
ERROR_CODE_CANCELLED: ErrorCode
ERROR_CODE_DEADLINE_EXCEEDED: ErrorCode
ERROR_CODE_FLOW_CONTROL_VIOLATION: ErrorCode
ERROR_CODE_INTERNAL: ErrorCode
SERVING_STATE_UNSPECIFIED: ServingState
SERVING_STATE_SERVING: ServingState
SERVING_STATE_DRAINING: ServingState
SERVING_STATE_NOT_SERVING: ServingState
class Checksum(_message.Message):
__slots__ = ("algorithm", "value")
ALGORITHM_FIELD_NUMBER: _ClassVar[int]
VALUE_FIELD_NUMBER: _ClassVar[int]
algorithm: ChecksumAlgorithm
value: bytes
def __init__(self, algorithm: _Optional[_Union[ChecksumAlgorithm, str]] = ..., value: _Optional[bytes] = ...) -> None: ...
class TensorFragment(_message.Message):
__slots__ = ("fragment_index", "fragment_count", "byte_offset", "payload")
FRAGMENT_INDEX_FIELD_NUMBER: _ClassVar[int]
FRAGMENT_COUNT_FIELD_NUMBER: _ClassVar[int]
BYTE_OFFSET_FIELD_NUMBER: _ClassVar[int]
PAYLOAD_FIELD_NUMBER: _ClassVar[int]
fragment_index: int
fragment_count: int
byte_offset: int
payload: bytes
def __init__(self, fragment_index: _Optional[int] = ..., fragment_count: _Optional[int] = ..., byte_offset: _Optional[int] = ..., payload: _Optional[bytes] = ...) -> None: ...
class NamedTensor(_message.Message):
__slots__ = ("name", "shape", "dtype", "byte_order", "total_bytes", "compression", "checksum", "fragments")
NAME_FIELD_NUMBER: _ClassVar[int]
SHAPE_FIELD_NUMBER: _ClassVar[int]
DTYPE_FIELD_NUMBER: _ClassVar[int]
BYTE_ORDER_FIELD_NUMBER: _ClassVar[int]
TOTAL_BYTES_FIELD_NUMBER: _ClassVar[int]
COMPRESSION_FIELD_NUMBER: _ClassVar[int]
CHECKSUM_FIELD_NUMBER: _ClassVar[int]
FRAGMENTS_FIELD_NUMBER: _ClassVar[int]
name: str
shape: _containers.RepeatedScalarFieldContainer[int]
dtype: DType
byte_order: ByteOrder
total_bytes: int
compression: Compression
checksum: Checksum
fragments: _containers.RepeatedCompositeFieldContainer[TensorFragment]
def __init__(self, name: _Optional[str] = ..., shape: _Optional[_Iterable[int]] = ..., dtype: _Optional[_Union[DType, str]] = ..., byte_order: _Optional[_Union[ByteOrder, str]] = ..., total_bytes: _Optional[int] = ..., compression: _Optional[_Union[Compression, str]] = ..., checksum: _Optional[_Union[Checksum, _Mapping]] = ..., fragments: _Optional[_Iterable[_Union[TensorFragment, _Mapping]]] = ...) -> None: ...
class TensorBundle(_message.Message):
__slots__ = ("bundle_version", "tensors")
BUNDLE_VERSION_FIELD_NUMBER: _ClassVar[int]
TENSORS_FIELD_NUMBER: _ClassVar[int]
bundle_version: int
tensors: _containers.RepeatedCompositeFieldContainer[NamedTensor]
def __init__(self, bundle_version: _Optional[int] = ..., tensors: _Optional[_Iterable[_Union[NamedTensor, _Mapping]]] = ...) -> None: ...
class Fingerprint(_message.Message):
__slots__ = ("model_artifact_digest", "runtime_recipe_digest", "recipe_id", "recipe_version", "catalogue_version")
MODEL_ARTIFACT_DIGEST_FIELD_NUMBER: _ClassVar[int]
RUNTIME_RECIPE_DIGEST_FIELD_NUMBER: _ClassVar[int]
RECIPE_ID_FIELD_NUMBER: _ClassVar[int]
RECIPE_VERSION_FIELD_NUMBER: _ClassVar[int]
CATALOGUE_VERSION_FIELD_NUMBER: _ClassVar[int]
model_artifact_digest: str
runtime_recipe_digest: str
recipe_id: str
recipe_version: str
catalogue_version: str
def __init__(self, model_artifact_digest: _Optional[str] = ..., runtime_recipe_digest: _Optional[str] = ..., recipe_id: _Optional[str] = ..., recipe_version: _Optional[str] = ..., catalogue_version: _Optional[str] = ...) -> None: ...
class ShardRange(_message.Message):
__slots__ = ("start_layer", "end_layer", "effective_start_layer")
START_LAYER_FIELD_NUMBER: _ClassVar[int]
END_LAYER_FIELD_NUMBER: _ClassVar[int]
EFFECTIVE_START_LAYER_FIELD_NUMBER: _ClassVar[int]
start_layer: int
end_layer: int
effective_start_layer: int
def __init__(self, start_layer: _Optional[int] = ..., end_layer: _Optional[int] = ..., effective_start_layer: _Optional[int] = ...) -> None: ...
class PositionSpan(_message.Message):
__slots__ = ("first_position", "token_count")
FIRST_POSITION_FIELD_NUMBER: _ClassVar[int]
TOKEN_COUNT_FIELD_NUMBER: _ClassVar[int]
first_position: int
token_count: int
def __init__(self, first_position: _Optional[int] = ..., token_count: _Optional[int] = ...) -> None: ...
class ChunkInfo(_message.Message):
__slots__ = ("chunk_index", "chunk_count", "final_chunk")
CHUNK_INDEX_FIELD_NUMBER: _ClassVar[int]
CHUNK_COUNT_FIELD_NUMBER: _ClassVar[int]
FINAL_CHUNK_FIELD_NUMBER: _ClassVar[int]
chunk_index: int
chunk_count: int
final_chunk: bool
def __init__(self, chunk_index: _Optional[int] = ..., chunk_count: _Optional[int] = ..., final_chunk: _Optional[bool] = ...) -> None: ...
class CacheExpectation(_message.Message):
__slots__ = ("mode", "expected_past_len")
MODE_FIELD_NUMBER: _ClassVar[int]
EXPECTED_PAST_LEN_FIELD_NUMBER: _ClassVar[int]
mode: CacheMode
expected_past_len: int
def __init__(self, mode: _Optional[_Union[CacheMode, str]] = ..., expected_past_len: _Optional[int] = ...) -> None: ...
class CacheResult(_message.Message):
__slots__ = ("mode", "past_len", "cache_hit")
MODE_FIELD_NUMBER: _ClassVar[int]
PAST_LEN_FIELD_NUMBER: _ClassVar[int]
CACHE_HIT_FIELD_NUMBER: _ClassVar[int]
mode: CacheMode
past_len: int
cache_hit: bool
def __init__(self, mode: _Optional[_Union[CacheMode, str]] = ..., past_len: _Optional[int] = ..., cache_hit: _Optional[bool] = ...) -> None: ...
class Envelope(_message.Message):
__slots__ = ("schema_version", "work_id", "route_session_id", "route_epoch", "fingerprint", "shard_range", "phase", "position", "idempotency_step", "cache_expectation", "deadline_unix_nanos", "chunk")
SCHEMA_VERSION_FIELD_NUMBER: _ClassVar[int]
WORK_ID_FIELD_NUMBER: _ClassVar[int]
ROUTE_SESSION_ID_FIELD_NUMBER: _ClassVar[int]
ROUTE_EPOCH_FIELD_NUMBER: _ClassVar[int]
FINGERPRINT_FIELD_NUMBER: _ClassVar[int]
SHARD_RANGE_FIELD_NUMBER: _ClassVar[int]
PHASE_FIELD_NUMBER: _ClassVar[int]
POSITION_FIELD_NUMBER: _ClassVar[int]
IDEMPOTENCY_STEP_FIELD_NUMBER: _ClassVar[int]
CACHE_EXPECTATION_FIELD_NUMBER: _ClassVar[int]
DEADLINE_UNIX_NANOS_FIELD_NUMBER: _ClassVar[int]
CHUNK_FIELD_NUMBER: _ClassVar[int]
schema_version: SchemaVersion
work_id: str
route_session_id: str
route_epoch: int
fingerprint: Fingerprint
shard_range: ShardRange
phase: Phase
position: PositionSpan
idempotency_step: int
cache_expectation: CacheExpectation
deadline_unix_nanos: int
chunk: ChunkInfo
def __init__(self, schema_version: _Optional[_Union[SchemaVersion, str]] = ..., work_id: _Optional[str] = ..., route_session_id: _Optional[str] = ..., route_epoch: _Optional[int] = ..., fingerprint: _Optional[_Union[Fingerprint, _Mapping]] = ..., shard_range: _Optional[_Union[ShardRange, _Mapping]] = ..., phase: _Optional[_Union[Phase, str]] = ..., position: _Optional[_Union[PositionSpan, _Mapping]] = ..., idempotency_step: _Optional[int] = ..., cache_expectation: _Optional[_Union[CacheExpectation, _Mapping]] = ..., deadline_unix_nanos: _Optional[int] = ..., chunk: _Optional[_Union[ChunkInfo, _Mapping]] = ...) -> None: ...
class ShardError(_message.Message):
__slots__ = ("code", "detail", "retryable", "actual_past_len")
CODE_FIELD_NUMBER: _ClassVar[int]
DETAIL_FIELD_NUMBER: _ClassVar[int]
RETRYABLE_FIELD_NUMBER: _ClassVar[int]
ACTUAL_PAST_LEN_FIELD_NUMBER: _ClassVar[int]
code: ErrorCode
detail: str
retryable: bool
actual_past_len: int
def __init__(self, code: _Optional[_Union[ErrorCode, str]] = ..., detail: _Optional[str] = ..., retryable: _Optional[bool] = ..., actual_past_len: _Optional[int] = ...) -> None: ...
class FlowControl(_message.Message):
__slots__ = ("credits_granted", "max_inflight_chunks", "max_chunk_bytes", "max_prefill_chunk_tokens")
CREDITS_GRANTED_FIELD_NUMBER: _ClassVar[int]
MAX_INFLIGHT_CHUNKS_FIELD_NUMBER: _ClassVar[int]
MAX_CHUNK_BYTES_FIELD_NUMBER: _ClassVar[int]
MAX_PREFILL_CHUNK_TOKENS_FIELD_NUMBER: _ClassVar[int]
credits_granted: int
max_inflight_chunks: int
max_chunk_bytes: int
max_prefill_chunk_tokens: int
def __init__(self, credits_granted: _Optional[int] = ..., max_inflight_chunks: _Optional[int] = ..., max_chunk_bytes: _Optional[int] = ..., max_prefill_chunk_tokens: _Optional[int] = ...) -> None: ...
class SessionOpen(_message.Message):
__slots__ = ("schema_version", "route_session_id", "route_epoch", "fingerprint", "shard_range", "proposed_flow_control", "accepted_compression")
SCHEMA_VERSION_FIELD_NUMBER: _ClassVar[int]
ROUTE_SESSION_ID_FIELD_NUMBER: _ClassVar[int]
ROUTE_EPOCH_FIELD_NUMBER: _ClassVar[int]
FINGERPRINT_FIELD_NUMBER: _ClassVar[int]
SHARD_RANGE_FIELD_NUMBER: _ClassVar[int]
PROPOSED_FLOW_CONTROL_FIELD_NUMBER: _ClassVar[int]
ACCEPTED_COMPRESSION_FIELD_NUMBER: _ClassVar[int]
schema_version: SchemaVersion
route_session_id: str
route_epoch: int
fingerprint: Fingerprint
shard_range: ShardRange
proposed_flow_control: FlowControl
accepted_compression: _containers.RepeatedScalarFieldContainer[Compression]
def __init__(self, schema_version: _Optional[_Union[SchemaVersion, str]] = ..., route_session_id: _Optional[str] = ..., route_epoch: _Optional[int] = ..., fingerprint: _Optional[_Union[Fingerprint, _Mapping]] = ..., shard_range: _Optional[_Union[ShardRange, _Mapping]] = ..., proposed_flow_control: _Optional[_Union[FlowControl, _Mapping]] = ..., accepted_compression: _Optional[_Iterable[_Union[Compression, str]]] = ...) -> None: ...
class SessionAccepted(_message.Message):
__slots__ = ("schema_version", "route_session_id", "route_epoch", "flow_control", "accepted_compression", "fingerprint")
SCHEMA_VERSION_FIELD_NUMBER: _ClassVar[int]
ROUTE_SESSION_ID_FIELD_NUMBER: _ClassVar[int]
ROUTE_EPOCH_FIELD_NUMBER: _ClassVar[int]
FLOW_CONTROL_FIELD_NUMBER: _ClassVar[int]
ACCEPTED_COMPRESSION_FIELD_NUMBER: _ClassVar[int]
FINGERPRINT_FIELD_NUMBER: _ClassVar[int]
schema_version: SchemaVersion
route_session_id: str
route_epoch: int
flow_control: FlowControl
accepted_compression: _containers.RepeatedScalarFieldContainer[Compression]
fingerprint: Fingerprint
def __init__(self, schema_version: _Optional[_Union[SchemaVersion, str]] = ..., route_session_id: _Optional[str] = ..., route_epoch: _Optional[int] = ..., flow_control: _Optional[_Union[FlowControl, _Mapping]] = ..., accepted_compression: _Optional[_Iterable[_Union[Compression, str]]] = ..., fingerprint: _Optional[_Union[Fingerprint, _Mapping]] = ...) -> None: ...
class ActivationChunk(_message.Message):
__slots__ = ("envelope", "bundle")
ENVELOPE_FIELD_NUMBER: _ClassVar[int]
BUNDLE_FIELD_NUMBER: _ClassVar[int]
envelope: Envelope
bundle: TensorBundle
def __init__(self, envelope: _Optional[_Union[Envelope, _Mapping]] = ..., bundle: _Optional[_Union[TensorBundle, _Mapping]] = ...) -> None: ...
class DecodeStep(_message.Message):
__slots__ = ("idempotency_step", "position", "expected_past_len", "tensor", "work_id", "deadline_unix_nanos")
IDEMPOTENCY_STEP_FIELD_NUMBER: _ClassVar[int]
POSITION_FIELD_NUMBER: _ClassVar[int]
EXPECTED_PAST_LEN_FIELD_NUMBER: _ClassVar[int]
TENSOR_FIELD_NUMBER: _ClassVar[int]
WORK_ID_FIELD_NUMBER: _ClassVar[int]
DEADLINE_UNIX_NANOS_FIELD_NUMBER: _ClassVar[int]
idempotency_step: int
position: int
expected_past_len: int
tensor: NamedTensor
work_id: str
deadline_unix_nanos: int
def __init__(self, idempotency_step: _Optional[int] = ..., position: _Optional[int] = ..., expected_past_len: _Optional[int] = ..., tensor: _Optional[_Union[NamedTensor, _Mapping]] = ..., work_id: _Optional[str] = ..., deadline_unix_nanos: _Optional[int] = ...) -> None: ...
class ReleaseSignal(_message.Message):
__slots__ = ("route_session_id", "route_epoch", "work_id")
ROUTE_SESSION_ID_FIELD_NUMBER: _ClassVar[int]
ROUTE_EPOCH_FIELD_NUMBER: _ClassVar[int]
WORK_ID_FIELD_NUMBER: _ClassVar[int]
route_session_id: str
route_epoch: int
work_id: str
def __init__(self, route_session_id: _Optional[str] = ..., route_epoch: _Optional[int] = ..., work_id: _Optional[str] = ...) -> None: ...
class CancelSignal(_message.Message):
__slots__ = ("route_session_id", "route_epoch", "work_id", "reason")
ROUTE_SESSION_ID_FIELD_NUMBER: _ClassVar[int]
ROUTE_EPOCH_FIELD_NUMBER: _ClassVar[int]
WORK_ID_FIELD_NUMBER: _ClassVar[int]
REASON_FIELD_NUMBER: _ClassVar[int]
route_session_id: str
route_epoch: int
work_id: str
reason: str
def __init__(self, route_session_id: _Optional[str] = ..., route_epoch: _Optional[int] = ..., work_id: _Optional[str] = ..., reason: _Optional[str] = ...) -> None: ...
class Ack(_message.Message):
__slots__ = ("work_id", "idempotency_step", "cache_result", "duplicate", "execution_nanos")
WORK_ID_FIELD_NUMBER: _ClassVar[int]
IDEMPOTENCY_STEP_FIELD_NUMBER: _ClassVar[int]
CACHE_RESULT_FIELD_NUMBER: _ClassVar[int]
DUPLICATE_FIELD_NUMBER: _ClassVar[int]
EXECUTION_NANOS_FIELD_NUMBER: _ClassVar[int]
work_id: str
idempotency_step: int
cache_result: CacheResult
duplicate: bool
execution_nanos: int
def __init__(self, work_id: _Optional[str] = ..., idempotency_step: _Optional[int] = ..., cache_result: _Optional[_Union[CacheResult, _Mapping]] = ..., duplicate: _Optional[bool] = ..., execution_nanos: _Optional[int] = ...) -> None: ...
class ShardStatus(_message.Message):
__slots__ = ("work_id", "route_session_id", "idempotency_step", "error", "terminal")
WORK_ID_FIELD_NUMBER: _ClassVar[int]
ROUTE_SESSION_ID_FIELD_NUMBER: _ClassVar[int]
IDEMPOTENCY_STEP_FIELD_NUMBER: _ClassVar[int]
ERROR_FIELD_NUMBER: _ClassVar[int]
TERMINAL_FIELD_NUMBER: _ClassVar[int]
work_id: str
route_session_id: str
idempotency_step: int
error: ShardError
terminal: bool
def __init__(self, work_id: _Optional[str] = ..., route_session_id: _Optional[str] = ..., idempotency_step: _Optional[int] = ..., error: _Optional[_Union[ShardError, _Mapping]] = ..., terminal: _Optional[bool] = ...) -> None: ...
class SessionRequest(_message.Message):
__slots__ = ("open", "chunk", "decode", "flow_control", "release", "cancel")
OPEN_FIELD_NUMBER: _ClassVar[int]
CHUNK_FIELD_NUMBER: _ClassVar[int]
DECODE_FIELD_NUMBER: _ClassVar[int]
FLOW_CONTROL_FIELD_NUMBER: _ClassVar[int]
RELEASE_FIELD_NUMBER: _ClassVar[int]
CANCEL_FIELD_NUMBER: _ClassVar[int]
open: SessionOpen
chunk: ActivationChunk
decode: DecodeStep
flow_control: FlowControl
release: ReleaseSignal
cancel: CancelSignal
def __init__(self, open: _Optional[_Union[SessionOpen, _Mapping]] = ..., chunk: _Optional[_Union[ActivationChunk, _Mapping]] = ..., decode: _Optional[_Union[DecodeStep, _Mapping]] = ..., flow_control: _Optional[_Union[FlowControl, _Mapping]] = ..., release: _Optional[_Union[ReleaseSignal, _Mapping]] = ..., cancel: _Optional[_Union[CancelSignal, _Mapping]] = ...) -> None: ...
class SessionResponse(_message.Message):
__slots__ = ("accepted", "chunk", "ack", "flow_control", "status")
ACCEPTED_FIELD_NUMBER: _ClassVar[int]
CHUNK_FIELD_NUMBER: _ClassVar[int]
ACK_FIELD_NUMBER: _ClassVar[int]
FLOW_CONTROL_FIELD_NUMBER: _ClassVar[int]
STATUS_FIELD_NUMBER: _ClassVar[int]
accepted: SessionAccepted
chunk: ActivationChunk
ack: Ack
flow_control: FlowControl
status: ShardStatus
def __init__(self, accepted: _Optional[_Union[SessionAccepted, _Mapping]] = ..., chunk: _Optional[_Union[ActivationChunk, _Mapping]] = ..., ack: _Optional[_Union[Ack, _Mapping]] = ..., flow_control: _Optional[_Union[FlowControl, _Mapping]] = ..., status: _Optional[_Union[ShardStatus, _Mapping]] = ...) -> None: ...
class CapabilityRequest(_message.Message):
__slots__ = ("schema_version",)
SCHEMA_VERSION_FIELD_NUMBER: _ClassVar[int]
schema_version: SchemaVersion
def __init__(self, schema_version: _Optional[_Union[SchemaVersion, str]] = ...) -> None: ...
class CapabilityReport(_message.Message):
__slots__ = ("schema_version", "fingerprint", "shard_range", "backend", "device", "validated", "detail", "max_concurrent_sessions", "max_context_tokens", "flow_control", "accepted_compression", "supported_schema_versions", "validated_at_unix_nanos")
SCHEMA_VERSION_FIELD_NUMBER: _ClassVar[int]
FINGERPRINT_FIELD_NUMBER: _ClassVar[int]
SHARD_RANGE_FIELD_NUMBER: _ClassVar[int]
BACKEND_FIELD_NUMBER: _ClassVar[int]
DEVICE_FIELD_NUMBER: _ClassVar[int]
VALIDATED_FIELD_NUMBER: _ClassVar[int]
DETAIL_FIELD_NUMBER: _ClassVar[int]
MAX_CONCURRENT_SESSIONS_FIELD_NUMBER: _ClassVar[int]
MAX_CONTEXT_TOKENS_FIELD_NUMBER: _ClassVar[int]
FLOW_CONTROL_FIELD_NUMBER: _ClassVar[int]
ACCEPTED_COMPRESSION_FIELD_NUMBER: _ClassVar[int]
SUPPORTED_SCHEMA_VERSIONS_FIELD_NUMBER: _ClassVar[int]
VALIDATED_AT_UNIX_NANOS_FIELD_NUMBER: _ClassVar[int]
schema_version: SchemaVersion
fingerprint: Fingerprint
shard_range: ShardRange
backend: str
device: str
validated: bool
detail: str
max_concurrent_sessions: int
max_context_tokens: int
flow_control: FlowControl
accepted_compression: _containers.RepeatedScalarFieldContainer[Compression]
supported_schema_versions: _containers.RepeatedScalarFieldContainer[SchemaVersion]
validated_at_unix_nanos: int
def __init__(self, schema_version: _Optional[_Union[SchemaVersion, str]] = ..., fingerprint: _Optional[_Union[Fingerprint, _Mapping]] = ..., shard_range: _Optional[_Union[ShardRange, _Mapping]] = ..., backend: _Optional[str] = ..., device: _Optional[str] = ..., validated: _Optional[bool] = ..., detail: _Optional[str] = ..., max_concurrent_sessions: _Optional[int] = ..., max_context_tokens: _Optional[int] = ..., flow_control: _Optional[_Union[FlowControl, _Mapping]] = ..., accepted_compression: _Optional[_Iterable[_Union[Compression, str]]] = ..., supported_schema_versions: _Optional[_Iterable[_Union[SchemaVersion, str]]] = ..., validated_at_unix_nanos: _Optional[int] = ...) -> None: ...
class HealthRequest(_message.Message):
__slots__ = ("schema_version",)
SCHEMA_VERSION_FIELD_NUMBER: _ClassVar[int]
schema_version: SchemaVersion
def __init__(self, schema_version: _Optional[_Union[SchemaVersion, str]] = ...) -> None: ...
class HealthReport(_message.Message):
__slots__ = ("schema_version", "state", "active_sessions", "queued_chunks", "batch_occupancy", "kv_pressure", "resident_bytes", "detail")
SCHEMA_VERSION_FIELD_NUMBER: _ClassVar[int]
STATE_FIELD_NUMBER: _ClassVar[int]
ACTIVE_SESSIONS_FIELD_NUMBER: _ClassVar[int]
QUEUED_CHUNKS_FIELD_NUMBER: _ClassVar[int]
BATCH_OCCUPANCY_FIELD_NUMBER: _ClassVar[int]
KV_PRESSURE_FIELD_NUMBER: _ClassVar[int]
RESIDENT_BYTES_FIELD_NUMBER: _ClassVar[int]
DETAIL_FIELD_NUMBER: _ClassVar[int]
schema_version: SchemaVersion
state: ServingState
active_sessions: int
queued_chunks: int
batch_occupancy: int
kv_pressure: float
resident_bytes: int
detail: str
def __init__(self, schema_version: _Optional[_Union[SchemaVersion, str]] = ..., state: _Optional[_Union[ServingState, str]] = ..., active_sessions: _Optional[int] = ..., queued_chunks: _Optional[int] = ..., batch_occupancy: _Optional[int] = ..., kv_pressure: _Optional[float] = ..., resident_bytes: _Optional[int] = ..., detail: _Optional[str] = ...) -> None: ...
class ReleaseRequest(_message.Message):
__slots__ = ("schema_version", "route_session_id", "route_epoch")
SCHEMA_VERSION_FIELD_NUMBER: _ClassVar[int]
ROUTE_SESSION_ID_FIELD_NUMBER: _ClassVar[int]
ROUTE_EPOCH_FIELD_NUMBER: _ClassVar[int]
schema_version: SchemaVersion
route_session_id: str
route_epoch: int
def __init__(self, schema_version: _Optional[_Union[SchemaVersion, str]] = ..., route_session_id: _Optional[str] = ..., route_epoch: _Optional[int] = ...) -> None: ...
class ReleaseResponse(_message.Message):
__slots__ = ("released", "error")
RELEASED_FIELD_NUMBER: _ClassVar[int]
ERROR_FIELD_NUMBER: _ClassVar[int]
released: bool
error: ShardError
def __init__(self, released: _Optional[bool] = ..., error: _Optional[_Union[ShardError, _Mapping]] = ...) -> None: ...
class CancelRequest(_message.Message):
__slots__ = ("schema_version", "route_session_id", "route_epoch", "work_id", "reason")
SCHEMA_VERSION_FIELD_NUMBER: _ClassVar[int]
ROUTE_SESSION_ID_FIELD_NUMBER: _ClassVar[int]
ROUTE_EPOCH_FIELD_NUMBER: _ClassVar[int]
WORK_ID_FIELD_NUMBER: _ClassVar[int]
REASON_FIELD_NUMBER: _ClassVar[int]
schema_version: SchemaVersion
route_session_id: str
route_epoch: int
work_id: str
reason: str
def __init__(self, schema_version: _Optional[_Union[SchemaVersion, str]] = ..., route_session_id: _Optional[str] = ..., route_epoch: _Optional[int] = ..., work_id: _Optional[str] = ..., reason: _Optional[str] = ...) -> None: ...
class CancelResponse(_message.Message):
__slots__ = ("cancelled_work_items", "error")
CANCELLED_WORK_ITEMS_FIELD_NUMBER: _ClassVar[int]
ERROR_FIELD_NUMBER: _ClassVar[int]
cancelled_work_items: int
error: ShardError
def __init__(self, cancelled_work_items: _Optional[int] = ..., error: _Optional[_Union[ShardError, _Mapping]] = ...) -> None: ...

295
packages/node/meshnet_node/native_protocol/generated/shard_runtime_pb2_grpc.py Normal file
View File

@@ -0,0 +1,295 @@
# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
"""Client and server classes corresponding to protobuf-defined services."""
import grpc
import warnings
from . import shard_runtime_pb2 as shard__runtime__pb2
GRPC_GENERATED_VERSION = '1.82.1'
GRPC_VERSION = grpc.__version__
_version_not_supported = False
try:
from grpc._utilities import first_version_is_lower
_version_not_supported = first_version_is_lower(GRPC_VERSION, GRPC_GENERATED_VERSION)
except ImportError:
_version_not_supported = True
if _version_not_supported:
raise RuntimeError(
f'The grpc package installed is at version {GRPC_VERSION},'
+ ' but the generated code in shard_runtime_pb2_grpc.py depends on'
+ f' grpcio>={GRPC_GENERATED_VERSION}.'
+ f' Please upgrade your grpc module to grpcio>={GRPC_GENERATED_VERSION}'
+ f' or downgrade your generated code using grpcio-tools<={GRPC_VERSION}.'
)
class ShardRuntimeStub:
"""---------------------------------------------------------------------------
Service
---------------------------------------------------------------------------
"""
def __init__(self, channel):
"""Constructor.
Args:
channel: A grpc.Channel.
"""
self.GetCapability = channel.unary_unary(
'/meshnet.shard.v1.ShardRuntime/GetCapability',
request_serializer=shard__runtime__pb2.CapabilityRequest.SerializeToString,
response_deserializer=shard__runtime__pb2.CapabilityReport.FromString,
_registered_method=True)
self.Health = channel.unary_unary(
'/meshnet.shard.v1.ShardRuntime/Health',
request_serializer=shard__runtime__pb2.HealthRequest.SerializeToString,
response_deserializer=shard__runtime__pb2.HealthReport.FromString,
_registered_method=True)
self.Session = channel.stream_stream(
'/meshnet.shard.v1.ShardRuntime/Session',
request_serializer=shard__runtime__pb2.SessionRequest.SerializeToString,
response_deserializer=shard__runtime__pb2.SessionResponse.FromString,
_registered_method=True)
self.Release = channel.unary_unary(
'/meshnet.shard.v1.ShardRuntime/Release',
request_serializer=shard__runtime__pb2.ReleaseRequest.SerializeToString,
response_deserializer=shard__runtime__pb2.ReleaseResponse.FromString,
_registered_method=True)
self.Cancel = channel.unary_unary(
'/meshnet.shard.v1.ShardRuntime/Cancel',
request_serializer=shard__runtime__pb2.CancelRequest.SerializeToString,
response_deserializer=shard__runtime__pb2.CancelResponse.FromString,
_registered_method=True)
class ShardRuntimeServicer:
"""---------------------------------------------------------------------------
Service
---------------------------------------------------------------------------
"""
def GetCapability(self, request, context):
"""What this worker can execute. Read before a route is built.
"""
context.set_code(grpc.StatusCode.UNIMPLEMENTED)
context.set_details('Method not implemented!')
raise NotImplementedError('Method not implemented!')
def Health(self, request, context):
"""Live load and serving state.
"""
context.set_code(grpc.StatusCode.UNIMPLEMENTED)
context.set_details('Method not implemented!')
raise NotImplementedError('Method not implemented!')
def Session(self, request_iterator, context):
"""One long-lived bidirectional stream per Route Session Activation Seam.
The stream opens with SessionOpen/SessionAccepted, then carries bounded
prefill chunks and decode steps in both directions for the life of the
session. Per-token channel creation is a non-goal: the handshake cost is
paid once and the hot path carries only what changes.
"""
context.set_code(grpc.StatusCode.UNIMPLEMENTED)
context.set_details('Method not implemented!')
raise NotImplementedError('Method not implemented!')
def Release(self, request, context):
"""Drop session state out of band. Idempotent.
"""
context.set_code(grpc.StatusCode.UNIMPLEMENTED)
context.set_details('Method not implemented!')
raise NotImplementedError('Method not implemented!')
def Cancel(self, request, context):
"""Cancel out of band, on a fresh call.
In-band CancelSignal is preferred, but a sender that is blocked on flow
control cannot write one — a cancel that can only travel down a wedged
stream is not a cancel. This RPC always has a path to the worker.
"""
context.set_code(grpc.StatusCode.UNIMPLEMENTED)
context.set_details('Method not implemented!')
raise NotImplementedError('Method not implemented!')
def add_ShardRuntimeServicer_to_server(servicer, server):
rpc_method_handlers = {
'GetCapability': grpc.unary_unary_rpc_method_handler(
servicer.GetCapability,
request_deserializer=shard__runtime__pb2.CapabilityRequest.FromString,
response_serializer=shard__runtime__pb2.CapabilityReport.SerializeToString,
),
'Health': grpc.unary_unary_rpc_method_handler(
servicer.Health,
request_deserializer=shard__runtime__pb2.HealthRequest.FromString,
response_serializer=shard__runtime__pb2.HealthReport.SerializeToString,
),
'Session': grpc.stream_stream_rpc_method_handler(
servicer.Session,
request_deserializer=shard__runtime__pb2.SessionRequest.FromString,
response_serializer=shard__runtime__pb2.SessionResponse.SerializeToString,
),
'Release': grpc.unary_unary_rpc_method_handler(
servicer.Release,
request_deserializer=shard__runtime__pb2.ReleaseRequest.FromString,
response_serializer=shard__runtime__pb2.ReleaseResponse.SerializeToString,
),
'Cancel': grpc.unary_unary_rpc_method_handler(
servicer.Cancel,
request_deserializer=shard__runtime__pb2.CancelRequest.FromString,
response_serializer=shard__runtime__pb2.CancelResponse.SerializeToString,
),
}
generic_handler = grpc.method_handlers_generic_handler(
'meshnet.shard.v1.ShardRuntime', rpc_method_handlers)
server.add_generic_rpc_handlers((generic_handler,))
server.add_registered_method_handlers('meshnet.shard.v1.ShardRuntime', rpc_method_handlers)
# This class is part of an EXPERIMENTAL API.
class ShardRuntime:
"""---------------------------------------------------------------------------
Service
---------------------------------------------------------------------------
"""
@staticmethod
def GetCapability(request,
target,
options=(),
channel_credentials=None,
call_credentials=None,
insecure=False,
compression=None,
wait_for_ready=None,
timeout=None,
metadata=None):
return grpc.experimental.unary_unary(
request,
target,
'/meshnet.shard.v1.ShardRuntime/GetCapability',
shard__runtime__pb2.CapabilityRequest.SerializeToString,
shard__runtime__pb2.CapabilityReport.FromString,
options,
channel_credentials,
insecure,
call_credentials,
compression,
wait_for_ready,
timeout,
metadata,
_registered_method=True)
@staticmethod
def Health(request,
target,
options=(),
channel_credentials=None,
call_credentials=None,
insecure=False,
compression=None,
wait_for_ready=None,
timeout=None,
metadata=None):
return grpc.experimental.unary_unary(
request,
target,
'/meshnet.shard.v1.ShardRuntime/Health',
shard__runtime__pb2.HealthRequest.SerializeToString,
shard__runtime__pb2.HealthReport.FromString,
options,
channel_credentials,
insecure,
call_credentials,
compression,
wait_for_ready,
timeout,
metadata,
_registered_method=True)
@staticmethod
def Session(request_iterator,
target,
options=(),
channel_credentials=None,
call_credentials=None,
insecure=False,
compression=None,
wait_for_ready=None,
timeout=None,
metadata=None):
return grpc.experimental.stream_stream(
request_iterator,
target,
'/meshnet.shard.v1.ShardRuntime/Session',
shard__runtime__pb2.SessionRequest.SerializeToString,
shard__runtime__pb2.SessionResponse.FromString,
options,
channel_credentials,
insecure,
call_credentials,
compression,
wait_for_ready,
timeout,
metadata,
_registered_method=True)
@staticmethod
def Release(request,
target,
options=(),
channel_credentials=None,
call_credentials=None,
insecure=False,
compression=None,
wait_for_ready=None,
timeout=None,
metadata=None):
return grpc.experimental.unary_unary(
request,
target,
'/meshnet.shard.v1.ShardRuntime/Release',
shard__runtime__pb2.ReleaseRequest.SerializeToString,
shard__runtime__pb2.ReleaseResponse.FromString,
options,
channel_credentials,
insecure,
call_credentials,
compression,
wait_for_ready,
timeout,
metadata,
_registered_method=True)
@staticmethod
def Cancel(request,
target,
options=(),
channel_credentials=None,
call_credentials=None,
insecure=False,
compression=None,
wait_for_ready=None,
timeout=None,
metadata=None):
return grpc.experimental.unary_unary(
request,
target,
'/meshnet.shard.v1.ShardRuntime/Cancel',
shard__runtime__pb2.CancelRequest.SerializeToString,
shard__runtime__pb2.CancelResponse.FromString,
options,
channel_credentials,
insecure,
call_credentials,
compression,
wait_for_ready,
timeout,
metadata,
_registered_method=True)

697
packages/node/meshnet_node/performance_contract.py Normal file
View File

@@ -0,0 +1,697 @@
"""The versioned safetensors-versus-GGUF performance contract.
The contract is the decision rule the native GGUF track is judged by, written
down *before* the numbers arrive and consumed later by the release gate
(DGR-014). Its thresholds are ratios against the Transformers/safetensors
reference recipe rather than absolute tokens/sec, because the absolute figure is
a property of whichever machine ran the benchmark and would have to be re-argued
on every host; a ratio is a claim about the runtime.
Three rules give the contract its teeth:
* **Thresholds are locked.** ``CONTRACT_SCHEMA_VERSION`` and ``locked_at``
travel with the document. Moving a threshold after seeing results is a new
contract version and a human decision, not a tweak.
* **Only like-for-like comparisons count.** A recipe measured on a different
device than the reference is marked non-comparable and is granted no benefit,
so a GPU-versus-CPU mismatch can never be laundered into a speed win.
* **Quantized recipes never claim numerical equivalence.** Quality is gated on
the near-lossless quality lane; the performance-fit lane is judged on speed,
memory and fit alone.
The verdict is one of ``promote``, ``optimize`` or ``stop`` — the three outcomes
the release gate is allowed to reach.
"""
from __future__ import annotations
import hashlib
import json
from dataclasses import asdict, dataclass
from pathlib import Path
from typing import Any, Mapping
from .recipe_benchmark import Lane, REPORT_SCHEMA_VERSION
# Layout of the contract document understood by this reader.
CONTRACT_SCHEMA_VERSION = 1
VERDICT_PROMOTE = "promote"
VERDICT_OPTIMIZE = "optimize"
VERDICT_STOP = "stop"
class PerformanceContractError(ValueError):
"""Raised when a contract is missing, malformed, or of an unsupported version."""
@dataclass(frozen=True)
class ContractThresholds:
"""The locked decision thresholds.
Every value is a ratio of a GGUF recipe's metric to the reference recipe's
metric on the same machine, same device, same plan.
A *meaningful speed benefit* means the GGUF recipe decodes at least 25%
faster for a single request without making time-to-first-token materially
worse, or sustains at least 25% more aggregate throughput under concurrency.
Either route is a real win for the product: one helps a single user, the
other helps a loaded node.
A *meaningful fit benefit* means peak resident memory (RSS plus VRAM) drops
by at least 25%. Fit is the product thesis — models larger than one
consumer node — so it is measured in resident bytes, not in how small the
file on disk is. Artifact size has its own reported threshold because a
smaller download is a real but secondary good.
25% is chosen to sit well clear of ordinary run-to-run variance on a busy
developer machine while still being a benefit a user would notice. A 5%
edge would not justify owning a native runtime and a patch stack.
"""
min_decode_speedup: float = 1.25
max_ttft_ratio: float = 1.25
min_aggregate_throughput_speedup: float = 1.25
max_resident_memory_ratio: float = 0.75
max_artifact_size_ratio: float = 0.60
min_quality_exact_match_rate: float = 0.90
min_quality_mean_similarity: float = 0.97
max_failure_rate: float = 0.0
def to_dict(self) -> dict:
return asdict(self)
@dataclass(frozen=True)
class PerformanceContract:
"""A locked, versioned contract plus the baseline it was locked against."""
contract_version: int
locked_at: str
locked_by: str
plan_id: str
thresholds: ContractThresholds
baseline: Mapping[str, Any]
stop_condition: str
notes: str = ""
schema_version: int = CONTRACT_SCHEMA_VERSION
def to_dict(self) -> dict:
return {
"schema_version": self.schema_version,
"contract_version": self.contract_version,
"locked_at": self.locked_at,
"locked_by": self.locked_by,
"plan_id": self.plan_id,
"thresholds": self.thresholds.to_dict(),
"baseline": dict(self.baseline),
"stop_condition": self.stop_condition,
"notes": self.notes,
}
STOP_CONDITION = (
"Stop the native llama.cpp/GGUF track when, on the same machine and device "
"as the Transformers/safetensors reference and under this plan, no "
"performance-fit GGUF recipe delivers either a meaningful speed benefit "
"(>=25% higher single-request decode tokens/sec without a >25% worse TTFT, "
"or >=25% higher aggregate throughput under concurrency) or a meaningful fit "
"benefit (>=25% lower peak resident memory), or when the near-lossless "
"quality lane fails, which indicates a broken runtime rather than a "
"quantization trade-off."
)
def _recipe_entries(report: Mapping[str, Any]) -> dict[str, Mapping[str, Any]]:
return {entry["recipe"]["id"]: entry for entry in report["recipes"]}
def _cell(entry: Mapping[str, Any], concurrency: int) -> Mapping[str, Any] | None:
return entry["concurrency"].get(str(concurrency))
def _resident_bytes(cell: Mapping[str, Any]) -> int:
return int(cell["peak_rss_bytes"]) + int(cell["peak_vram_bytes"])
def _ratio(value: float, reference: float) -> float:
"""Ratio guarded against a zero reference, which means "not measured"."""
if reference <= 0:
return 0.0
return round(value / reference, 4)
def _canonical_sha256(value: Any) -> str:
payload = json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False)
return hashlib.sha256(payload.encode("utf-8")).hexdigest()
def _validate_report(contract: PerformanceContract, report: Mapping[str, Any]) -> None:
"""Fail closed when a report is not the experiment the contract locked."""
try:
schema_version = report["schema_version"]
evidence_class = report["evidence_class"]
plan = report["plan"]
recipes = report["recipes"]
reference_id = report["reference_recipe_id"]
host = report["host"]
except (KeyError, TypeError) as exc:
raise PerformanceContractError("benchmark report is missing required structure") from exc
if schema_version != REPORT_SCHEMA_VERSION:
raise PerformanceContractError(
f"report schema {schema_version!r} is not supported schema {REPORT_SCHEMA_VERSION}"
)
if plan.get("plan_id") != contract.plan_id:
raise PerformanceContractError(
f"report plan {plan.get('plan_id')!r} does not match locked plan {contract.plan_id!r}"
)
required_plan_sha256 = contract.baseline.get("required_plan_sha256")
measured_plan_sha256 = _canonical_sha256(plan)
if required_plan_sha256 and measured_plan_sha256 != required_plan_sha256:
raise PerformanceContractError(
f"report plan digest {measured_plan_sha256} does not match locked digest "
f"{required_plan_sha256}"
)
minimum_repeats = int(contract.baseline.get("minimum_repeats", 0))
minimum_prompts = int(contract.baseline.get("minimum_prompt_count", 0))
if int(plan.get("repeats", 0)) < minimum_repeats:
raise PerformanceContractError("report has too few repeats for the locked contract")
if len(plan.get("prompts", ())) < minimum_prompts:
raise PerformanceContractError("report has too few prompts for the locked contract")
minimum_output_tokens = int(contract.baseline.get("minimum_output_tokens", 0))
if int(plan.get("sampling", {}).get("max_output_tokens", 0)) < minimum_output_tokens:
raise PerformanceContractError("report output length is below the locked contract")
required_evidence = contract.baseline.get("required_evidence_class")
if required_evidence and evidence_class != required_evidence:
raise PerformanceContractError(
f"report evidence class {evidence_class!r} does not satisfy {required_evidence!r}"
)
if required_evidence and (
not isinstance(host, Mapping)
or any(key not in host for key in ("hostname", "platform", "python", "cpu_count"))
):
raise PerformanceContractError("report lacks measured host provenance")
if not isinstance(recipes, list) or not recipes:
raise PerformanceContractError("report contains no recipes")
recipe_ids = [entry.get("recipe", {}).get("id") for entry in recipes]
if len(set(recipe_ids)) != len(recipe_ids) or None in recipe_ids:
raise PerformanceContractError("report recipe IDs must be present and unique")
required_recipes = set(contract.baseline.get("required_recipes", ()))
missing_recipes = required_recipes - set(recipe_ids)
if missing_recipes:
raise PerformanceContractError(
f"report is missing required recipes {sorted(missing_recipes)}"
)
if reference_id not in recipe_ids:
raise PerformanceContractError("report reference recipe is absent")
levels = {int(level) for level in plan.get("concurrency_levels", ())}
required_levels = {
int(level) for level in contract.baseline.get("required_concurrency_levels", ())
}
if not required_levels.issubset(levels):
raise PerformanceContractError(
f"report concurrency {sorted(levels)} lacks required levels {sorted(required_levels)}"
)
if not plan.get("prompts"):
raise PerformanceContractError("report plan contains no prompts")
model_id = plan.get("model_id")
model_revision = plan.get("model_revision")
required_device = contract.baseline.get("required_device")
for entry in recipes:
recipe = entry.get("recipe", {})
if required_device and recipe.get("device") != required_device:
raise PerformanceContractError(
f"recipe {recipe.get('id')!r} did not run on locked device {required_device!r}"
)
if required_evidence:
if recipe.get("source_model_id") != model_id:
raise PerformanceContractError("report mixes source model IDs")
if recipe.get("source_model_revision") != model_revision:
raise PerformanceContractError("report mixes source model revisions")
digest = recipe.get("artifact_sha256", "")
if not isinstance(digest, str) or len(digest) != 64:
raise PerformanceContractError("report lacks an artifact SHA-256 digest")
if entry.get("available"):
cells = entry.get("concurrency", {})
missing_cells = required_levels - {int(level) for level in cells}
if missing_cells:
raise PerformanceContractError(
f"recipe {recipe.get('id')!r} lacks concurrency cells {sorted(missing_cells)}"
)
reference = next(entry for entry in recipes if entry["recipe"]["id"] == reference_id)
if not reference.get("available"):
raise PerformanceContractError("reference recipe is unavailable")
reference_failures = sum(
int(cell.get("failures", 0)) for cell in reference.get("concurrency", {}).values()
)
if reference_failures:
raise PerformanceContractError("reference recipe contains failed requests")
def token_counts(entry: Mapping[str, Any]) -> dict[tuple[str, int, int], list[tuple[int, int]]]:
counts: dict[tuple[str, int, int], list[tuple[int, int]]] = {}
for outcome in entry.get("outcomes", ()):
if not outcome.get("ok"):
continue
key = (
str(outcome.get("prompt_id")),
int(outcome.get("concurrency", 0)),
int(outcome.get("repeat", -1)),
)
counts.setdefault(key, []).append(
(int(outcome.get("prompt_tokens", 0)), int(outcome.get("decode_tokens", 0)))
)
return {key: sorted(values) for key, values in counts.items()}
reference_counts = token_counts(reference)
prompt_ids = {str(prompt["id"]) for prompt in plan["prompts"]}
repeats = int(plan["repeats"])
for prompt_id in prompt_ids:
for level in required_levels:
for repeat in range(repeats):
values = reference_counts.get((prompt_id, level, repeat), ())
if len(values) != level:
raise PerformanceContractError(
"reference recipe lacks complete prompt/repeat/concurrency coverage"
)
for entry in recipes:
if entry.get("available") and token_counts(entry) != reference_counts:
raise PerformanceContractError(
f"recipe {entry['recipe']['id']!r} used different prompt/decode token counts"
)
@dataclass(frozen=True)
class RecipeEvaluation:
"""How one GGUF recipe fared against the reference under the contract."""
recipe_id: str
lane: str
comparable: bool
incomparable_reason: str
speed_benefit: bool
fit_benefit: bool
quality_pass: bool | None
failures: int
measurements: Mapping[str, Any]
reasons: tuple[str, ...]
def to_dict(self) -> dict:
data = asdict(self)
data["measurements"] = dict(self.measurements)
data["reasons"] = list(self.reasons)
return data
@dataclass(frozen=True)
class ContractEvaluation:
"""The release-gate answer: a verdict plus every reason behind it."""
contract_version: int
plan_id: str
verdict: str
quality_lane_pass: bool
speed_benefit: bool
fit_benefit: bool
stop_condition_met: bool
recipes: tuple[RecipeEvaluation, ...]
rationale: tuple[str, ...]
def to_dict(self) -> dict:
return {
"contract_version": self.contract_version,
"plan_id": self.plan_id,
"verdict": self.verdict,
"quality_lane_pass": self.quality_lane_pass,
"speed_benefit": self.speed_benefit,
"fit_benefit": self.fit_benefit,
"stop_condition_met": self.stop_condition_met,
"recipes": [recipe.to_dict() for recipe in self.recipes],
"rationale": list(self.rationale),
}
def _evaluate_recipe(
entry: Mapping[str, Any],
reference: Mapping[str, Any],
drift_by_recipe: Mapping[str, Mapping[str, Any]],
thresholds: ContractThresholds,
concurrency_levels: list[int],
expected_prompt_count: int,
) -> RecipeEvaluation:
recipe = entry["recipe"]
lane = recipe["lane"]
reasons: list[str] = []
if not entry["available"]:
return RecipeEvaluation(
recipe_id=recipe["id"], lane=lane, comparable=False,
incomparable_reason=entry["unavailable_reason"] or "recipe was not measured",
speed_benefit=False, fit_benefit=False, quality_pass=None, failures=0,
measurements={}, reasons=("recipe unavailable; no benefit granted",),
)
if recipe["device"] != reference["recipe"]["device"]:
return RecipeEvaluation(
recipe_id=recipe["id"], lane=lane, comparable=False,
incomparable_reason=(
f"recipe ran on device {recipe['device']!r} but the reference ran on "
f"{reference['recipe']['device']!r}; a cross-device ratio is not a runtime result"
),
speed_benefit=False, fit_benefit=False, quality_pass=None, failures=0,
measurements={}, reasons=("cross-device comparison; no benefit granted",),
)
single = _cell(entry, 1)
reference_single = _cell(reference, 1)
measurements: dict[str, Any] = {}
speed_benefit = False
if single and reference_single:
decode_speedup = _ratio(
single["decode_tokens_per_sec"], reference_single["decode_tokens_per_sec"]
)
ttft_ratio = _ratio(single["ttft_p50_ms"], reference_single["ttft_p50_ms"])
measurements["decode_speedup"] = decode_speedup
measurements["ttft_ratio"] = ttft_ratio
single_request_win = (
decode_speedup >= thresholds.min_decode_speedup
and 0 < ttft_ratio <= thresholds.max_ttft_ratio
)
if single_request_win:
speed_benefit = lane == Lane.PERFORMANCE_FIT.value
reasons.append(
f"single-request decode {decode_speedup:.2f}x reference "
f"(>= {thresholds.min_decode_speedup:.2f}x) at TTFT ratio {ttft_ratio:.2f}"
)
else:
reasons.append(
f"no single-request speed win: decode {decode_speedup:.2f}x, TTFT {ttft_ratio:.2f}x"
)
concurrent = [level for level in concurrency_levels if level > 1]
if concurrent:
top = max(concurrent)
cell, reference_cell = _cell(entry, top), _cell(reference, top)
if cell and reference_cell:
aggregate_speedup = _ratio(
cell["aggregate_decode_tokens_per_sec"],
reference_cell["aggregate_decode_tokens_per_sec"],
)
measurements["aggregate_throughput_speedup"] = aggregate_speedup
measurements["aggregate_concurrency"] = top
if aggregate_speedup >= thresholds.min_aggregate_throughput_speedup:
speed_benefit = lane == Lane.PERFORMANCE_FIT.value
reasons.append(
f"aggregate throughput at concurrency {top} is {aggregate_speedup:.2f}x reference "
f"(>= {thresholds.min_aggregate_throughput_speedup:.2f}x)"
)
else:
reasons.append(
f"no concurrency speed win: aggregate throughput at {top} is "
f"{aggregate_speedup:.2f}x reference"
)
fit_benefit = False
if single and reference_single:
resident_ratio = _ratio(_resident_bytes(single), _resident_bytes(reference_single))
artifact_ratio = _ratio(
entry["load"]["artifact_bytes"], reference["load"]["artifact_bytes"]
)
measurements["resident_memory_ratio"] = resident_ratio
measurements["artifact_size_ratio"] = artifact_ratio
if 0 < resident_ratio <= thresholds.max_resident_memory_ratio:
fit_benefit = lane == Lane.PERFORMANCE_FIT.value
reasons.append(
f"peak resident memory is {resident_ratio:.2f}x reference "
f"(<= {thresholds.max_resident_memory_ratio:.2f}x)"
)
else:
reasons.append(f"no fit win: peak resident memory is {resident_ratio:.2f}x reference")
measurements["artifact_size_win"] = (
0 < artifact_ratio <= thresholds.max_artifact_size_ratio
)
failures = sum(cell["failures"] for cell in entry["concurrency"].values())
requests = sum(cell["requests"] for cell in entry["concurrency"].values())
failure_rate = _ratio(failures, requests) if requests else 0.0
measurements["failure_rate"] = failure_rate
if failure_rate > thresholds.max_failure_rate:
reasons.append(f"failure rate {failure_rate:.2%} exceeds the contract limit")
speed_benefit = False
fit_benefit = False
# Quality is a claim only the near-lossless lane is allowed to make. A
# quantized recipe's drift is recorded elsewhere and deliberately not read
# here: Q4 disagreeing with bf16 is the trade-off, not a failure.
quality_pass: bool | None = None
if lane == Lane.QUALITY.value:
drift = drift_by_recipe.get(recipe["id"])
if drift is None:
quality_pass = False
reasons.append("quality-lane recipe has no drift measurement against the reference")
else:
complete_coverage = drift.get("compared_prompts") == expected_prompt_count
quality_pass = (
complete_coverage
and failures == 0
and drift["exact_match_rate"] >= thresholds.min_quality_exact_match_rate
and drift["mean_similarity"] >= thresholds.min_quality_mean_similarity
)
measurements["compared_prompts"] = drift.get("compared_prompts", 0)
measurements["expected_prompts"] = expected_prompt_count
measurements["exact_match_rate"] = drift["exact_match_rate"]
measurements["mean_similarity"] = drift["mean_similarity"]
if not complete_coverage:
reasons.append(
f"quality lane compared {drift.get('compared_prompts', 0)} of "
f"{expected_prompt_count} required prompts"
)
if failures:
reasons.append("quality lane contains failed requests")
reasons.append(
f"quality lane exact-match {drift['exact_match_rate']:.2f} / similarity "
f"{drift['mean_similarity']:.3f} versus the reference "
f"({'pass' if quality_pass else 'fail'})"
)
return RecipeEvaluation(
recipe_id=recipe["id"], lane=lane, comparable=True, incomparable_reason="",
speed_benefit=speed_benefit, fit_benefit=fit_benefit, quality_pass=quality_pass,
failures=failures, measurements=measurements, reasons=tuple(reasons),
)
def evaluate_contract(
contract: PerformanceContract,
report: Mapping[str, Any],
) -> ContractEvaluation:
"""Judge a benchmark report against the locked contract.
Only performance-fit recipes can earn a speed or fit benefit; the quality
lane decides only whether the GGUF runtime is numerically sane. A GGUF
runtime that fails the quality lane is broken, and no amount of speed
redeems it, so the verdict is ``stop`` regardless of the other numbers.
"""
_validate_report(contract, report)
entries = _recipe_entries(report)
reference_id = report["reference_recipe_id"]
reference = entries.get(reference_id)
if reference is None:
raise PerformanceContractError(
f"report names reference recipe {reference_id!r}, which it does not contain"
)
drift_by_recipe = {entry["recipe_id"]: entry for entry in report["drift"]}
concurrency_levels = list(report["plan"]["concurrency_levels"])
evaluations = tuple(
_evaluate_recipe(
entry,
reference,
drift_by_recipe,
contract.thresholds,
concurrency_levels,
len(report["plan"]["prompts"]),
)
for recipe_id, entry in entries.items()
if recipe_id != reference_id
)
quality_lane = [
evaluation for evaluation in evaluations
if evaluation.lane == Lane.QUALITY.value and evaluation.comparable
]
quality_lane_pass = bool(quality_lane) and all(
evaluation.quality_pass for evaluation in quality_lane
)
performance_lane = [
evaluation for evaluation in evaluations
if evaluation.lane == Lane.PERFORMANCE_FIT.value
]
speed_benefit = any(evaluation.speed_benefit for evaluation in performance_lane)
fit_benefit = any(evaluation.fit_benefit for evaluation in performance_lane)
rationale: list[str] = []
if not quality_lane:
rationale.append(
"no comparable near-lossless GGUF recipe was measured, so the runtime's "
"numerical correctness is unproven"
)
elif not quality_lane_pass:
rationale.append(
"the near-lossless quality lane failed: the GGUF runtime disagrees with the "
"safetensors reference beyond what near-lossless weights can explain"
)
else:
rationale.append("the near-lossless quality lane passed against the safetensors reference")
rationale.append(
"a meaningful speed benefit was measured" if speed_benefit
else "no performance-fit recipe delivered a meaningful speed benefit"
)
rationale.append(
"a meaningful fit benefit was measured" if fit_benefit
else "no performance-fit recipe delivered a meaningful fit benefit"
)
stop_condition_met = not quality_lane_pass or not (speed_benefit or fit_benefit)
if stop_condition_met:
verdict = VERDICT_STOP
elif speed_benefit and fit_benefit:
verdict = VERDICT_PROMOTE
else:
verdict = VERDICT_OPTIMIZE
rationale.append(
"exactly one of speed or fit cleared the contract: the benefit is real but partial, "
"so the measured bottleneck needs a bounded optimization task before promotion"
)
return ContractEvaluation(
contract_version=contract.contract_version,
plan_id=contract.plan_id,
verdict=verdict,
quality_lane_pass=quality_lane_pass,
speed_benefit=speed_benefit,
fit_benefit=fit_benefit,
stop_condition_met=stop_condition_met,
recipes=evaluations,
rationale=tuple(rationale),
)
def parse_contract(data: Any, source: str = "<memory>") -> PerformanceContract:
"""Validate an already-decoded contract document."""
if not isinstance(data, Mapping):
raise PerformanceContractError(f"contract root in {source} must be a JSON object")
schema_version = data.get("schema_version")
if not isinstance(schema_version, int) or isinstance(schema_version, bool):
raise PerformanceContractError(f"'schema_version' in {source} must be an integer")
if schema_version != CONTRACT_SCHEMA_VERSION:
raise PerformanceContractError(
f"{source} declares contract schema version {schema_version}, but this node reads "
f"version {CONTRACT_SCHEMA_VERSION}; upgrade the node or use a supported contract"
)
for required in ("contract_version", "locked_at", "locked_by", "plan_id", "stop_condition"):
if not data.get(required):
raise PerformanceContractError(f"{source} is missing {required!r}")
raw_thresholds = data.get("thresholds")
if not isinstance(raw_thresholds, Mapping):
raise PerformanceContractError(f"'thresholds' in {source} must be a JSON object")
known = set(ContractThresholds().to_dict())
unknown = set(raw_thresholds) - known
missing = known - set(raw_thresholds)
if unknown or missing:
raise PerformanceContractError(
f"{source} threshold keys differ from v1; unknown={sorted(unknown)}, "
f"missing={sorted(missing)}"
)
thresholds = ContractThresholds(**{
key: float(value) for key, value in raw_thresholds.items()
})
contract_version = int(data["contract_version"])
if contract_version != 1 or thresholds != ContractThresholds():
raise PerformanceContractError(
f"{source} changes immutable v1 thresholds without a supported contract version"
)
if str(data["stop_condition"]) != STOP_CONDITION:
raise PerformanceContractError(
f"{source} stop condition differs from executable v1 semantics"
)
return PerformanceContract(
contract_version=contract_version,
locked_at=str(data["locked_at"]),
locked_by=str(data["locked_by"]),
plan_id=str(data["plan_id"]),
thresholds=thresholds,
baseline=dict(data.get("baseline") or {}),
stop_condition=str(data["stop_condition"]),
notes=str(data.get("notes", "")),
schema_version=schema_version,
)
def load_contract(path: Path) -> PerformanceContract:
"""Load and validate the contract at ``path``."""
try:
raw = path.read_text(encoding="utf-8")
except OSError as exc:
raise PerformanceContractError(
f"cannot read performance contract {path}: {exc.strerror or exc}"
) from exc
try:
data = json.loads(raw)
except json.JSONDecodeError as exc:
raise PerformanceContractError(
f"{path} is not valid JSON: {exc.msg} at line {exc.lineno} column {exc.colno}"
) from exc
return parse_contract(data, source=str(path))
def baseline_from_report(report: Mapping[str, Any]) -> dict[str, Any]:
"""Distil the reference numbers a later gate needs to compare against."""
entries = _recipe_entries(report)
baseline: dict[str, Any] = {
"evidence_class": report["evidence_class"],
"model_id": report["plan"]["model_id"],
"model_revision": report["plan"]["model_revision"],
"plan_sha256": _canonical_sha256(report["plan"]),
"reference_recipe_id": report["reference_recipe_id"],
"host": report["host"],
"recipes": {},
}
for recipe_id, entry in entries.items():
if not entry["available"]:
baseline["recipes"][recipe_id] = {"available": False,
"reason": entry["unavailable_reason"]}
continue
baseline["recipes"][recipe_id] = {
"available": True,
"lane": entry["recipe"]["lane"],
"device": entry["recipe"]["device"],
"artifact_bytes": entry["load"]["artifact_bytes"],
"concurrency": {
level: {
"ttft_p50_ms": cell["ttft_p50_ms"],
"ttft_p95_ms": cell["ttft_p95_ms"],
"latency_p50_ms": cell["latency_p50_ms"],
"latency_p95_ms": cell["latency_p95_ms"],
"prefill_tokens_per_sec": cell["prefill_tokens_per_sec"],
"decode_tokens_per_sec": cell["decode_tokens_per_sec"],
"aggregate_decode_tokens_per_sec": cell["aggregate_decode_tokens_per_sec"],
"peak_rss_bytes": cell["peak_rss_bytes"],
"peak_vram_bytes": cell["peak_vram_bytes"],
"failures": cell["failures"],
}
for level, cell in entry["concurrency"].items()
},
}
return baseline

674
packages/node/meshnet_node/recipe_benchmark.py Normal file
View File

@@ -0,0 +1,674 @@
"""Controlled safetensors-versus-GGUF recipe benchmark.
This is a *model recipe* benchmark, unlike
:mod:`meshnet_node.route_session_benchmark`, which is a transport harness. It
answers one question: on one machine, with one model revision and one fixed
workload, what do the Transformers/safetensors recipe and the whole-model
llama.cpp/GGUF recipes actually cost in speed, memory, fit, and output drift?
Two ideas keep the answer honest.
**Lanes.** A recipe belongs to exactly one :class:`Lane`. The quality lane
holds near-lossless recipes (bf16/f16 weights) whose outputs may legitimately be
compared for numerical agreement. The performance-fit lane holds quantized
recipes (Q8_0, Q4_K_M, ...). Quantized recipes are judged on speed, memory, and
artifact size only; their drift is *reported* but never read as evidence that
Q4 and bf16 are numerically equivalent, because they are not. The lane is a
property of the recipe, so nothing downstream can quietly cross the boundary.
**Drivers.** The measurement core here is pure and runtime-free: it drives a
:class:`RecipeDriver` and computes metrics. Real runtimes live in
:mod:`meshnet_node.recipe_drivers` and are imported only on demand, which keeps
the default test suite deterministic, GPU-free and model-download-free while the
same code path produces the real evidence.
"""
from __future__ import annotations
import argparse
import json
import statistics
import threading
import time
from concurrent.futures import ThreadPoolExecutor
from dataclasses import asdict, dataclass, field
from difflib import SequenceMatcher
from enum import Enum
from pathlib import Path
from typing import Any, Protocol, Sequence
# Layout of the report document produced by :func:`build_report`.
REPORT_SCHEMA_VERSION = 1
class Lane(str, Enum):
"""Why a recipe is being measured at all.
``QUALITY`` recipes carry near-lossless weights, so comparing their output
with the reference recipe is a meaningful correctness signal.
``PERFORMANCE_FIT`` recipes carry quantized weights: they exist to be faster
or to fit, and their drift is descriptive, never a pass/fail equivalence
claim.
"""
QUALITY = "quality"
PERFORMANCE_FIT = "performance-fit"
class BenchmarkError(RuntimeError):
"""Raised when a benchmark cannot be run as specified."""
@dataclass(frozen=True)
class SamplingPolicy:
"""The sampling policy every recipe must be given, identically.
Greedy by default: sampling noise would otherwise be indistinguishable from
quantization drift, and the whole point of the quality lane is to tell those
two apart.
"""
temperature: float = 0.0
top_p: float = 1.0
top_k: int = 1
seed: int = 1234
max_output_tokens: int = 64
def to_dict(self) -> dict:
return asdict(self)
@dataclass(frozen=True)
class PromptSpec:
"""One fixed prompt, tagged with the context length it is meant to exercise."""
id: str
text: str
context_class: str = "short"
def to_dict(self) -> dict:
return asdict(self)
@dataclass(frozen=True)
class BenchmarkPlan:
"""The controlled variables: identical for every recipe in a report.
A plan is the experiment. If two recipes were measured under different
plans, their numbers are not comparable and the report must not pretend they
are, so the plan is recorded once at the top of the document rather than
per-recipe.
"""
plan_id: str
model_id: str
model_revision: str
prompts: tuple[PromptSpec, ...]
sampling: SamplingPolicy = SamplingPolicy()
concurrency_levels: tuple[int, ...] = (1, 4)
repeats: int = 1
warmup_requests: int = 1
def __post_init__(self) -> None:
if not self.prompts:
raise BenchmarkError("a benchmark plan needs at least one prompt")
if not self.concurrency_levels or any(level < 1 for level in self.concurrency_levels):
raise BenchmarkError("concurrency levels must all be >= 1")
if self.repeats < 1:
raise BenchmarkError("repeats must be >= 1")
if 1 not in self.concurrency_levels or 4 not in self.concurrency_levels:
raise BenchmarkError("a controlled baseline must include concurrency levels 1 and 4")
def to_dict(self) -> dict:
return {
"plan_id": self.plan_id,
"model_id": self.model_id,
"model_revision": self.model_revision,
"prompts": [prompt.to_dict() for prompt in self.prompts],
"sampling": self.sampling.to_dict(),
"concurrency_levels": list(self.concurrency_levels),
"repeats": self.repeats,
"warmup_requests": self.warmup_requests,
}
@dataclass(frozen=True)
class RecipeSpec:
"""One runtime recipe under test.
``is_reference`` marks the single recipe every other recipe's output drift is
measured against — the current Transformers/safetensors route, which
decision Gate 8 keeps as the correctness backend.
"""
id: str
runtime: str
weight_format: str
weight_quantization: str
lane: Lane
device: str
artifact_path: str = ""
source_model_id: str = ""
source_model_revision: str = ""
artifact_sha256: str = ""
is_reference: bool = False
notes: str = ""
def to_dict(self) -> dict:
data = asdict(self)
data["lane"] = self.lane.value
return data
@dataclass(frozen=True)
class LoadStats:
"""What loading the recipe cost, before any token is generated."""
artifact_bytes: int
load_ms: float
rss_bytes: int = 0
vram_bytes: int = 0
backend_detail: str = ""
def to_dict(self) -> dict:
return asdict(self)
@dataclass(frozen=True)
class GenerationSample:
"""One completed generation as reported by a driver.
``prefill_ms``/``decode_ms`` are the runtime's own split where it exposes one
(llama.cpp does); drivers that cannot split honestly report ``prefill_ms`` as
the time to the first token. ``queue_wait_ms`` separates time spent waiting
for a runtime slot from time spent computing, so a concurrency-4 TTFT is not
silently read as a slower prefill.
"""
text: str
prompt_tokens: int
decode_tokens: int
ttft_ms: float
prefill_ms: float
decode_ms: float
total_ms: float
queue_wait_ms: float = 0.0
class RecipeDriver(Protocol):
"""The seam every runtime implements; the measurement core knows nothing else."""
def load(self) -> LoadStats:
"""Load the artifact and return its cost."""
def generate(self, prompt: str, sampling: SamplingPolicy) -> GenerationSample:
"""Run one complete generation under the given sampling policy."""
def memory_probe(self) -> tuple[int, int]:
"""Return ``(rss_bytes, vram_bytes)`` observed right now."""
def close(self) -> None:
"""Release the runtime."""
@dataclass(frozen=True)
class RequestOutcome:
"""One request attempt, successful or not.
A failure is a first-class result, not an exception that aborts the run: a
recipe that cannot sustain concurrency 4 has told us something, and the
report must carry it rather than lose it.
"""
recipe_id: str
concurrency: int
prompt_id: str
repeat: int
ok: bool
latency_ms: float
ttft_ms: float = 0.0
prefill_ms: float = 0.0
decode_ms: float = 0.0
queue_wait_ms: float = 0.0
prompt_tokens: int = 0
decode_tokens: int = 0
text: str = ""
error: str = ""
def to_dict(self) -> dict:
return asdict(self)
def _percentile(values: Sequence[float], percentile: float) -> float:
"""Nearest-rank percentile; 0.0 for an empty sample."""
ordered = sorted(values)
if not ordered:
return 0.0
rank = max(1, -(-len(ordered) * percentile // 100))
return round(ordered[int(rank) - 1], 4)
def _mean(values: Sequence[float]) -> float:
return round(statistics.fmean(values), 4) if values else 0.0
@dataclass(frozen=True)
class ConcurrencyMetrics:
"""Aggregate metrics for one recipe at one concurrency level."""
concurrency: int
requests: int
failures: int
wall_ms: float
ttft_p50_ms: float
ttft_p95_ms: float
latency_p50_ms: float
latency_p95_ms: float
prefill_tokens_per_sec: float
decode_tokens_per_sec: float
aggregate_decode_tokens_per_sec: float
peak_rss_bytes: int
peak_vram_bytes: int
failure_reasons: tuple[str, ...] = ()
def to_dict(self) -> dict:
data = asdict(self)
data["failure_reasons"] = list(self.failure_reasons)
return data
def summarize_concurrency(
outcomes: Sequence[RequestOutcome],
*,
concurrency: int,
wall_ms: float,
peak_rss_bytes: int,
peak_vram_bytes: int,
) -> ConcurrencyMetrics:
"""Aggregate one recipe/concurrency cell.
Per-request rates are averaged over successful requests; aggregate
throughput is total decoded tokens over the wall clock of the whole cell,
which is the only figure that credits a runtime for overlapping work.
"""
ok = [outcome for outcome in outcomes if outcome.ok]
failures = [outcome for outcome in outcomes if not outcome.ok]
decode_tokens = sum(outcome.decode_tokens for outcome in ok)
prefill_rates = [
outcome.prompt_tokens / (outcome.prefill_ms / 1000)
for outcome in ok
if outcome.prefill_ms > 0 and outcome.prompt_tokens
]
decode_rates = [
outcome.decode_tokens / (outcome.decode_ms / 1000)
for outcome in ok
if outcome.decode_ms > 0 and outcome.decode_tokens
]
return ConcurrencyMetrics(
concurrency=concurrency,
requests=len(outcomes),
failures=len(failures),
wall_ms=round(wall_ms, 4),
ttft_p50_ms=_percentile([outcome.ttft_ms for outcome in ok], 50),
ttft_p95_ms=_percentile([outcome.ttft_ms for outcome in ok], 95),
latency_p50_ms=_percentile([outcome.latency_ms for outcome in ok], 50),
latency_p95_ms=_percentile([outcome.latency_ms for outcome in ok], 95),
prefill_tokens_per_sec=_mean(prefill_rates),
decode_tokens_per_sec=_mean(decode_rates),
aggregate_decode_tokens_per_sec=round(decode_tokens / max(1e-6, wall_ms / 1000), 4),
peak_rss_bytes=peak_rss_bytes,
peak_vram_bytes=peak_vram_bytes,
failure_reasons=tuple(sorted({outcome.error for outcome in failures if outcome.error})),
)
@dataclass
class RecipeMeasurement:
"""Everything measured for one recipe across every concurrency level."""
recipe: RecipeSpec
load: LoadStats
metrics: dict[int, ConcurrencyMetrics] = field(default_factory=dict)
outcomes: list[RequestOutcome] = field(default_factory=list)
unavailable_reason: str = ""
@property
def available(self) -> bool:
return not self.unavailable_reason
def outputs_by_prompt(self) -> dict[str, str]:
"""First successful output per prompt, at the lowest concurrency measured.
Drift is a property of the recipe, not of load: concurrency must not
change greedy output, so the least-contended sample is the fair one.
"""
best: dict[str, tuple[int, str]] = {}
for outcome in self.outcomes:
if not outcome.ok:
continue
seen = best.get(outcome.prompt_id)
if seen is None or outcome.concurrency < seen[0]:
best[outcome.prompt_id] = (outcome.concurrency, outcome.text)
return {prompt_id: text for prompt_id, (_, text) in best.items()}
def to_dict(self) -> dict:
return {
"recipe": self.recipe.to_dict(),
"available": self.available,
"unavailable_reason": self.unavailable_reason,
"load": self.load.to_dict(),
"concurrency": {
str(level): metrics.to_dict() for level, metrics in sorted(self.metrics.items())
},
"outcomes": [outcome.to_dict() for outcome in self.outcomes],
}
@dataclass(frozen=True)
class DriftReport:
"""Output drift of one recipe against the reference recipe.
``advisory`` is true for every performance-fit recipe: the number is
published, but a Q4 recipe disagreeing with bf16 is expected behaviour, not a
defect, and no gate may read it as one.
"""
recipe_id: str
lane: Lane
reference_id: str
compared_prompts: int
exact_match_rate: float
mean_similarity: float
advisory: bool
per_prompt: tuple[dict[str, Any], ...] = ()
def to_dict(self) -> dict:
return {
"recipe_id": self.recipe_id,
"lane": self.lane.value,
"reference_id": self.reference_id,
"compared_prompts": self.compared_prompts,
"exact_match_rate": self.exact_match_rate,
"mean_similarity": self.mean_similarity,
"advisory": self.advisory,
"per_prompt": list(self.per_prompt),
}
def _first_divergence(left: str, right: str) -> int:
"""Index of the first differing character, or -1 when the strings agree."""
for index, (a, b) in enumerate(zip(left, right)):
if a != b:
return index
return -1 if len(left) == len(right) else min(len(left), len(right))
def compute_drift(
measurement: RecipeMeasurement,
reference: RecipeMeasurement,
) -> DriftReport:
"""Compare one recipe's greedy outputs with the reference recipe's."""
reference_outputs = reference.outputs_by_prompt()
outputs = measurement.outputs_by_prompt()
shared = sorted(set(outputs) & set(reference_outputs))
per_prompt: list[dict[str, Any]] = []
exact = 0
similarities: list[float] = []
for prompt_id in shared:
got, want = outputs[prompt_id], reference_outputs[prompt_id]
matches = got == want
exact += matches
similarity = round(SequenceMatcher(None, want, got).ratio(), 4)
similarities.append(similarity)
per_prompt.append({
"prompt_id": prompt_id,
"exact_match": matches,
"similarity": similarity,
"first_divergence_char": _first_divergence(want, got),
"reference_text": want,
"recipe_text": got,
})
return DriftReport(
recipe_id=measurement.recipe.id,
lane=measurement.recipe.lane,
reference_id=reference.recipe.id,
compared_prompts=len(shared),
exact_match_rate=round(exact / len(shared), 4) if shared else 0.0,
mean_similarity=_mean(similarities),
advisory=measurement.recipe.lane is Lane.PERFORMANCE_FIT,
per_prompt=tuple(per_prompt),
)
class _PeakMemory:
"""Continuously sample a driver's memory while requests are in flight."""
def __init__(self, driver: RecipeDriver) -> None:
self._driver = driver
self.peak_rss = 0
self.peak_vram = 0
def sample(self) -> None:
try:
rss, vram = self._driver.memory_probe()
except Exception: # a probe must never fail a benchmark run
return
self.peak_rss = max(self.peak_rss, rss)
self.peak_vram = max(self.peak_vram, vram)
def sample_until(self, stop: threading.Event, interval_s: float = 0.01) -> None:
"""Sample until ``stop`` is set, including one final post-request probe."""
while not stop.wait(interval_s):
self.sample()
self.sample()
def _run_request(
driver: RecipeDriver,
recipe: RecipeSpec,
prompt: PromptSpec,
sampling: SamplingPolicy,
concurrency: int,
repeat: int,
memory: _PeakMemory,
) -> RequestOutcome:
started = time.monotonic()
try:
sample = driver.generate(prompt.text, sampling)
except Exception as exc: # a failed request is data, not a crashed benchmark
return RequestOutcome(
recipe_id=recipe.id,
concurrency=concurrency,
prompt_id=prompt.id,
repeat=repeat,
ok=False,
latency_ms=round((time.monotonic() - started) * 1000, 4),
error=f"{type(exc).__name__}: {exc}",
)
finally:
memory.sample()
return RequestOutcome(
recipe_id=recipe.id,
concurrency=concurrency,
prompt_id=prompt.id,
repeat=repeat,
ok=True,
latency_ms=round(sample.total_ms, 4),
ttft_ms=round(sample.ttft_ms, 4),
prefill_ms=round(sample.prefill_ms, 4),
decode_ms=round(sample.decode_ms, 4),
queue_wait_ms=round(sample.queue_wait_ms, 4),
prompt_tokens=sample.prompt_tokens,
decode_tokens=sample.decode_tokens,
text=sample.text,
)
def measure_recipe(
driver: RecipeDriver,
recipe: RecipeSpec,
plan: BenchmarkPlan,
) -> RecipeMeasurement:
"""Load one recipe and run the whole plan against it.
The driver is closed exactly once, whatever happens, so a recipe that dies at
concurrency 4 still releases its weights before the next recipe loads.
"""
load = driver.load()
measurement = RecipeMeasurement(recipe=recipe, load=load)
try:
for _ in range(plan.warmup_requests):
try:
driver.generate(plan.prompts[0].text, plan.sampling)
except Exception: # a failing warmup is reported by the real requests
break
for concurrency in plan.concurrency_levels:
# Measure each cell independently and sample while requests are in
# flight; post-request probes alone miss transient KV/workspace peaks.
memory = _PeakMemory(driver)
memory.sample()
stop_sampling = threading.Event()
sampler = threading.Thread(
target=memory.sample_until,
args=(stop_sampling,),
name=f"recipe-memory-{recipe.id}-c{concurrency}",
daemon=True,
)
requests = [
(prompt, repeat)
for repeat in range(plan.repeats)
for prompt in plan.prompts
for _ in range(concurrency)
]
started = time.monotonic()
sampler.start()
try:
with ThreadPoolExecutor(max_workers=concurrency) as pool:
outcomes = list(pool.map(
lambda item: _run_request(
driver, recipe, item[0], plan.sampling, concurrency, item[1], memory
),
requests,
))
finally:
stop_sampling.set()
sampler.join()
wall_ms = (time.monotonic() - started) * 1000
measurement.outcomes.extend(outcomes)
measurement.metrics[concurrency] = summarize_concurrency(
outcomes,
concurrency=concurrency,
wall_ms=wall_ms,
peak_rss_bytes=memory.peak_rss,
peak_vram_bytes=memory.peak_vram,
)
finally:
driver.close()
return measurement
def build_report(
plan: BenchmarkPlan,
measurements: Sequence[RecipeMeasurement],
*,
host: dict[str, Any],
evidence_class: str,
) -> dict:
"""Assemble the machine-readable benchmark document.
``evidence_class`` is one of ``synthetic``, ``local-real`` or
``multi-machine-real`` and is never inferred: a report that cannot say how it
was produced cannot be trusted by a release gate.
"""
if evidence_class not in {"synthetic", "local-real", "multi-machine-real"}:
raise BenchmarkError(f"unknown evidence class {evidence_class!r}")
references = [m for m in measurements if m.recipe.is_reference]
if len(references) != 1:
raise BenchmarkError(
f"exactly one reference recipe is required, got {len(references)}"
)
reference = references[0]
if reference.recipe.lane is not Lane.QUALITY:
raise BenchmarkError("the reference recipe must sit in the quality lane")
drift = [
compute_drift(measurement, reference).to_dict()
for measurement in measurements
if measurement is not reference and measurement.available
]
return {
"schema_version": REPORT_SCHEMA_VERSION,
"evidence_class": evidence_class,
"plan": plan.to_dict(),
"host": host,
"reference_recipe_id": reference.recipe.id,
"recipes": [measurement.to_dict() for measurement in measurements],
"drift": drift,
}
def format_summary(report: dict) -> str:
"""Render the human-readable companion to the JSON artifact."""
plan = report["plan"]
lines = [
f"Recipe benchmark {plan['plan_id']} ({report['evidence_class']})",
f"model {plan['model_id']}@{plan['model_revision']}",
]
for entry in report["recipes"]:
recipe = entry["recipe"]
if not entry["available"]:
lines.append(f"{recipe['id']:38} UNAVAILABLE: {entry['unavailable_reason']}")
continue
artifact_gb = entry["load"]["artifact_bytes"] / 1e9
for level, metrics in entry["concurrency"].items():
lines.append(
f"{recipe['id']:38} [{recipe['lane']:16}] c={level:>2} "
f"ttft p50/p95 {metrics['ttft_p50_ms']:8.1f}/{metrics['ttft_p95_ms']:8.1f} ms; "
f"prefill {metrics['prefill_tokens_per_sec']:7.1f} tok/s; "
f"decode {metrics['decode_tokens_per_sec']:6.1f} tok/s; "
f"aggregate {metrics['aggregate_decode_tokens_per_sec']:7.1f} tok/s; "
f"rss {metrics['peak_rss_bytes'] / 1e9:5.2f} GB; "
f"vram {metrics['peak_vram_bytes'] / 1e9:5.2f} GB; "
f"artifact {artifact_gb:5.2f} GB; failures {metrics['failures']}"
)
for entry in report["drift"]:
tag = "advisory" if entry["advisory"] else "gated"
lines.append(
f"drift {entry['recipe_id']:32} vs {entry['reference_id']:28} "
f"exact {entry['exact_match_rate']:.2f}; similarity {entry['mean_similarity']:.3f} ({tag})"
)
return "\n".join(lines)
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(
description="Run the controlled safetensors-versus-GGUF recipe benchmark"
)
parser.add_argument("--config", type=Path, required=True, help="benchmark configuration JSON")
parser.add_argument("--json-out", type=Path, help="write the JSON report to this path")
parser.add_argument("--summary-out", type=Path, help="write the text summary to this path")
args = parser.parse_args(argv)
from .recipe_drivers import run_configured_benchmark # heavy runtimes: import on demand
report = run_configured_benchmark(json.loads(args.config.read_text(encoding="utf-8")))
summary = format_summary(report)
if args.json_out:
args.json_out.write_text(json.dumps(report, indent=2, sort_keys=True) + "\n", encoding="utf-8")
if args.summary_out:
args.summary_out.write_text(summary + "\n", encoding="utf-8")
print(summary)
return 0
if __name__ == "__main__": # pragma: no cover - CLI entry point
raise SystemExit(main())

664
packages/node/meshnet_node/recipe_drivers.py Normal file
View File

@@ -0,0 +1,664 @@
"""Real runtime drivers for the recipe benchmark.
This module is the only place that imports torch, transformers, or spawns a
llama.cpp server, and :mod:`meshnet_node.recipe_benchmark` imports it lazily.
That keeps the default test suite deterministic, GPU-free and download-free
while the real evidence runs through exactly the same measurement core.
Fairness is the whole point of a baseline, so both drivers are held to the same
rules:
* They are handed a **pre-formatted prompt string**. Neither applies a chat
template, because a template applied twice — or differently — by two runtimes
would show up as a speed and drift difference that has nothing to do with the
runtime.
* They are given the **same CPU thread budget**, so the comparison measures
kernels rather than how many cores each runtime felt entitled to take.
* They report the runtime's **own prefill/decode split** where it has one, and
say so honestly where it does not.
"""
from __future__ import annotations
import hashlib
import hmac
import json
import os
import platform
import re
import socket
import subprocess
import sys
import tempfile
import time
import urllib.error
import urllib.request
from pathlib import Path
from typing import Any, Mapping
from .recipe_benchmark import (
BenchmarkError,
BenchmarkPlan,
GenerationSample,
Lane,
LoadStats,
PromptSpec,
RecipeSpec,
SamplingPolicy,
build_report,
measure_recipe,
)
REAL_INFERENCE_ENV = "MESHNET_ENABLE_REAL_INFERENCE_TESTS"
def real_inference_enabled() -> bool:
"""Real runtimes stay off unless the operator opts in explicitly."""
return os.environ.get(REAL_INFERENCE_ENV) == "1"
def require_real_inference() -> None:
if not real_inference_enabled():
raise BenchmarkError(
f"real model execution is opt-in: set {REAL_INFERENCE_ENV}=1 to run this benchmark"
)
def _process_rss(pid: int | None = None) -> int:
"""Resident bytes for a process and its children, or 0 when unobservable."""
try:
import psutil
except ImportError:
return 0
try:
process = psutil.Process(pid) if pid else psutil.Process()
total = process.memory_info().rss
for child in process.children(recursive=True):
try:
total += child.memory_info().rss
except psutil.Error:
continue
return int(total)
except Exception:
return 0
def _directory_bytes(path: Path) -> int:
if path.is_file():
return path.stat().st_size
return sum(entry.stat().st_size for entry in path.rglob("*") if entry.is_file())
def _artifact_sha256(path: Path) -> str:
"""Hash an artifact file or a deterministic directory content manifest.
A file uses the ordinary SHA-256 digest. A directory hashes each sorted
relative path, resolved file size, and file bytes, so tokenizer/config drift
cannot hide behind a weight-only digest.
"""
digest = hashlib.sha256()
if path.is_file():
entries = [(None, path)]
else:
entries = [
(entry.relative_to(path).as_posix(), entry)
for entry in sorted(path.rglob("*"))
if entry.is_file()
]
if not entries:
raise BenchmarkError(f"artifact directory is empty: {path}")
for relative, entry in entries:
if relative is not None:
encoded = relative.encode("utf-8")
digest.update(len(encoded).to_bytes(8, "big"))
digest.update(encoded)
digest.update(entry.stat().st_size.to_bytes(8, "big"))
with entry.open("rb") as stream:
while chunk := stream.read(8 * 1024 * 1024):
digest.update(chunk)
return digest.hexdigest()
def _host_manifest() -> dict[str, Any]:
"""Capture non-secret host facts with the report rather than trusting prose."""
manifest: dict[str, Any] = {
"hostname": socket.gethostname(),
"platform": platform.platform(),
"python": sys.version.split()[0],
"cpu_count": os.cpu_count(),
}
try:
import torch
import transformers
manifest["torch_version"] = torch.__version__
manifest["transformers_version"] = transformers.__version__
manifest["cuda_available"] = bool(torch.cuda.is_available())
if torch.cuda.is_available():
manifest["accelerator_name"] = torch.cuda.get_device_name(0)
manifest["accelerator_runtime"] = getattr(torch.version, "cuda", None) or getattr(
torch.version, "hip", None
)
except ImportError:
manifest["torch_version"] = None
return manifest
def _validate_config(config: Mapping[str, Any]) -> None:
"""Reject comparisons that mix models, artifacts, devices, or budgets."""
try:
plan = config["plan"]
root = Path(config["artifact_storage_root"]).resolve(strict=True)
recipes = config["recipes"]
except (KeyError, TypeError, OSError) as exc:
raise BenchmarkError(
"benchmark config needs an existing artifact_storage_root, plan, and recipes"
) from exc
if not root.is_absolute() or root == Path("/home") or Path("/home") in root.parents:
raise BenchmarkError("model artifacts must use configured mounted-drive storage, never /home")
if not isinstance(recipes, list) or not recipes:
raise BenchmarkError("benchmark config needs at least one recipe")
sampling = plan.get("sampling", {})
if (
float(sampling.get("temperature", 0.0)) != 0.0
or int(sampling.get("top_k", 1)) != 1
or float(sampling.get("top_p", 1.0)) != 1.0
):
raise BenchmarkError("the quality comparison requires greedy sampling")
if len(plan.get("prompts", ())) < 3 or int(plan.get("repeats", 0)) < 3:
raise BenchmarkError("contract-grade evidence requires at least 3 prompts and 3 repeats")
if int(plan.get("warmup_requests", 0)) < 1:
raise BenchmarkError("contract-grade evidence requires at least one warmup")
if int(sampling.get("max_output_tokens", 0)) < 32:
raise BenchmarkError("contract-grade evidence requires at least 32 output tokens")
thread_budgets: set[int] = set()
max_concurrency = max(int(level) for level in plan.get("concurrency_levels", (1, 4)))
for spec in recipes:
if spec.get("source_model_id") != plan.get("model_id"):
raise BenchmarkError("every recipe must declare the plan's exact source_model_id")
if spec.get("source_model_revision") != plan.get("model_revision"):
raise BenchmarkError("every recipe must declare the plan's exact source_model_revision")
digest = spec.get("artifact_sha256", "")
if not isinstance(digest, str) or re.fullmatch(r"[0-9a-f]{64}", digest) is None:
raise BenchmarkError("every recipe must declare a lowercase SHA-256 artifact digest")
artifact = Path(spec.get("artifact_path", "")).resolve(strict=True)
if artifact != root and root not in artifact.parents:
raise BenchmarkError("every model artifact must be beneath artifact_storage_root")
actual_digest = _artifact_sha256(artifact)
if not hmac.compare_digest(digest, actual_digest):
raise BenchmarkError(
f"artifact digest mismatch for {spec.get('id', '<unknown>')}: "
f"declared {digest}, measured {actual_digest}"
)
driver = spec.get("driver")
if not isinstance(driver, Mapping):
raise BenchmarkError("every recipe needs a driver object")
kind = driver.get("type")
if kind == "transformers":
driver_artifact = Path(driver.get("model_path", "")).resolve(strict=True)
elif kind == "llama-cpp-server":
driver_artifact = Path(driver.get("gguf_path", "")).resolve(strict=True)
binary = Path(driver.get("binary", "")).resolve(strict=True)
binary_digest = driver.get("binary_sha256", "")
if (
not isinstance(binary_digest, str)
or re.fullmatch(r"[0-9a-f]{64}", binary_digest) is None
or not hmac.compare_digest(binary_digest, _artifact_sha256(binary))
):
raise BenchmarkError("llama.cpp binary SHA-256 mismatch")
if int(driver.get("n_parallel", max_concurrency)) < max_concurrency:
raise BenchmarkError("llama.cpp parallel slots must cover maximum concurrency")
if driver.get("device", "cpu") != "cpu" or int(driver.get("n_gpu_layers", 0)) != 0:
raise BenchmarkError(
"v1 benchmark supports CPU-only llama.cpp until process VRAM is measurable"
)
else:
raise BenchmarkError(f"unknown driver type {kind!r}")
if driver_artifact != artifact:
raise BenchmarkError("driver artifact path must match the hashed recipe artifact")
if driver.get("device", "cpu") != spec.get("device"):
raise BenchmarkError("recipe and driver must declare the same device")
thread_budgets.add(int(driver.get("threads", 8)))
if len(thread_budgets) != 1:
raise BenchmarkError("every recipe must use the same CPU thread budget")
class TransformersDriver:
"""The current Transformers/safetensors recipe: the correctness reference.
Generation is a hand-written prefill-then-decode loop rather than
``model.generate`` because the benchmark needs the two phases separated: one
forward over the prompt gives an exact prefill time and TTFT, and the cached
single-token steps that follow give an exact decode rate. ``generate`` would
hand back one blended number.
"""
def __init__(
self,
model_path: str,
*,
device: str = "cpu",
dtype: str = "bfloat16",
threads: int = 8,
) -> None:
self.model_path = Path(model_path)
self.device = device
self.dtype = dtype
self.threads = threads
self._model: Any = None
self._tokenizer: Any = None
self._torch: Any = None
self._rss_baseline = 0
def load(self) -> LoadStats:
self._rss_baseline = _process_rss()
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
self._torch = torch
torch.set_num_threads(self.threads)
torch.manual_seed(0)
started = time.monotonic()
self._tokenizer = AutoTokenizer.from_pretrained(
str(self.model_path), local_files_only=True
)
self._model = AutoModelForCausalLM.from_pretrained(
str(self.model_path),
dtype=getattr(torch, self.dtype),
local_files_only=True,
)
self._model.to(self.device)
self._model.eval()
load_ms = (time.monotonic() - started) * 1000
return LoadStats(
artifact_bytes=_directory_bytes(self.model_path),
load_ms=round(load_ms, 4),
rss_bytes=max(0, _process_rss() - self._rss_baseline),
vram_bytes=self._vram_bytes(),
backend_detail=(
f"torch {torch.__version__}; dtype {self.dtype}; "
f"device {self.device}; intra-op threads {self.threads}"
),
)
def _vram_bytes(self) -> int:
torch = self._torch
if torch is None or self.device == "cpu":
return 0
try:
if torch.cuda.is_available():
return int(torch.cuda.max_memory_allocated())
except Exception:
return 0
return 0
def generate(self, prompt: str, sampling: SamplingPolicy) -> GenerationSample:
if self._model is None:
raise BenchmarkError("TransformersDriver.generate called before load()")
torch = self._torch
# add_special_tokens=False: the plan owns the prompt format, and the
# llama.cpp recipe is given the identical string.
encoded = self._tokenizer(prompt, return_tensors="pt", add_special_tokens=False)
input_ids = encoded["input_ids"].to(self.device)
prompt_tokens = int(input_ids.shape[-1])
eos_ids = {self._tokenizer.eos_token_id} | set(
getattr(self._model.generation_config, "eos_token_id", None) or []
if isinstance(getattr(self._model.generation_config, "eos_token_id", None), list)
else []
)
eos_ids.discard(None)
started = time.monotonic()
with torch.inference_mode():
outputs = self._model(input_ids=input_ids, use_cache=True)
past = outputs.past_key_values
next_id = self._select(outputs.logits[:, -1, :], sampling)
ttft_ms = (time.monotonic() - started) * 1000
token_ids = [int(next_id.item())]
decode_started = time.monotonic()
while len(token_ids) < sampling.max_output_tokens and token_ids[-1] not in eos_ids:
outputs = self._model(
input_ids=next_id.view(1, 1), past_key_values=past, use_cache=True
)
past = outputs.past_key_values
next_id = self._select(outputs.logits[:, -1, :], sampling)
token_ids.append(int(next_id.item()))
decode_ms = (time.monotonic() - decode_started) * 1000
total_ms = (time.monotonic() - started) * 1000
emitted = [token for token in token_ids if token not in eos_ids]
return GenerationSample(
text=self._tokenizer.decode(emitted, skip_special_tokens=True),
prompt_tokens=prompt_tokens,
# The first token is produced by the prefill forward, so the decode
# rate must not be credited with it.
decode_tokens=max(0, len(token_ids) - 1),
ttft_ms=ttft_ms,
prefill_ms=ttft_ms,
decode_ms=decode_ms,
total_ms=total_ms,
)
def _select(self, logits: Any, sampling: SamplingPolicy) -> Any:
if sampling.temperature > 0:
raise BenchmarkError(
"this benchmark is greedy-only: sampling noise is indistinguishable from "
"quantization drift, which is precisely what the quality lane must isolate"
)
return logits.argmax(dim=-1)
def memory_probe(self) -> tuple[int, int]:
return max(0, _process_rss() - self._rss_baseline), self._vram_bytes()
def close(self) -> None:
self._model = None
self._tokenizer = None
if self._torch is not None:
import gc
gc.collect()
def _free_port() -> int:
with socket.socket() as probe:
probe.bind(("127.0.0.1", 0))
return int(probe.getsockname()[1])
class LlamaCppServerDriver:
"""The whole-model llama.cpp/GGUF recipe, driven through ``llama-server``.
``llama-server`` is used rather than an in-process binding because it is the
shape llama.cpp is actually deployed in and the only one that offers
continuous batching across parallel slots — which is the runtime property
this project cares about most. It also reports its own prefill/decode
timings per request, so the decode rate is the runtime's own number and not
an inference drawn from a client-side stopwatch.
"""
def __init__(
self,
binary: str,
gguf_path: str,
*,
binary_sha256: str,
device: str = "cpu",
threads: int = 8,
n_parallel: int = 4,
context_per_slot: int = 1024,
n_gpu_layers: int = 0,
startup_timeout_s: float = 120.0,
) -> None:
self.binary = Path(binary)
self.binary_sha256 = binary_sha256
self.gguf_path = Path(gguf_path)
self.device = device
self.threads = threads
self.n_parallel = n_parallel
self.context_per_slot = context_per_slot
self.n_gpu_layers = n_gpu_layers
self.startup_timeout_s = startup_timeout_s
self._process: subprocess.Popen | None = None
self._port = 0
self._log: Any = None
@property
def _url(self) -> str:
return f"http://127.0.0.1:{self._port}"
def _log_excerpt(self) -> str:
if self._log is None:
return ""
try:
self._log.flush()
self._log.seek(0)
return self._log.read()[-4096:].decode("utf-8", errors="replace").strip()
except Exception:
return ""
def load(self) -> LoadStats:
if not self.binary.exists():
raise BenchmarkError(f"llama-server binary not found at {self.binary}")
if not self.gguf_path.exists():
raise BenchmarkError(f"GGUF artifact not found at {self.gguf_path}")
measured_binary_sha256 = _artifact_sha256(self.binary)
if not hmac.compare_digest(self.binary_sha256, measured_binary_sha256):
raise BenchmarkError("llama-server binary changed after config validation")
version = " | ".join(
subprocess.run(
[str(self.binary), "--version"],
check=True,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
text=True,
timeout=10,
).stdout.strip().splitlines()
)
self._port = _free_port()
command = [
str(self.binary),
"--model", str(self.gguf_path),
"--host", "127.0.0.1",
"--port", str(self._port),
"--threads", str(self.threads),
"--parallel", str(self.n_parallel),
# Every slot must hold a whole request, so the pool is sized for the
# worst case rather than letting llama.cpp silently truncate context.
"--ctx-size", str(self.context_per_slot * self.n_parallel),
"--n-gpu-layers", str(self.n_gpu_layers),
"--no-webui",
]
started = time.monotonic()
self._log = tempfile.TemporaryFile(mode="w+b")
self._process = subprocess.Popen(
command, stdout=self._log, stderr=subprocess.STDOUT
)
self._await_health(started)
load_ms = (time.monotonic() - started) * 1000
return LoadStats(
artifact_bytes=self.gguf_path.stat().st_size,
load_ms=round(load_ms, 4),
rss_bytes=_process_rss(self._process.pid),
vram_bytes=0,
backend_detail=(
f"{version}; binary sha256 {measured_binary_sha256}; "
f"threads {self.threads}; parallel slots {self.n_parallel}; "
f"ctx/slot {self.context_per_slot}; gpu layers {self.n_gpu_layers}"
),
)
def _await_health(self, started: float) -> None:
while time.monotonic() - started < self.startup_timeout_s:
if self._process is not None and self._process.poll() is not None:
raise BenchmarkError(
f"llama-server exited with code {self._process.returncode} during startup; "
f"log tail: {self._log_excerpt()}"
)
try:
with urllib.request.urlopen(f"{self._url}/health", timeout=2) as response:
if response.status == 200:
return
except (urllib.error.URLError, OSError):
time.sleep(0.25)
raise BenchmarkError(
f"llama-server did not become healthy within {self.startup_timeout_s:.0f}s; "
f"log tail: {self._log_excerpt()}"
)
def generate(self, prompt: str, sampling: SamplingPolicy) -> GenerationSample:
if self._process is None:
raise BenchmarkError("LlamaCppServerDriver.generate called before load()")
if sampling.temperature > 0:
raise BenchmarkError("this benchmark is greedy-only; see TransformersDriver._select")
body = json.dumps({
"prompt": prompt,
"n_predict": sampling.max_output_tokens,
"temperature": 0.0,
"top_k": 1,
"top_p": 1.0,
"seed": sampling.seed,
# Prompt cache reuse across repeats would measure the cache, not the
# prefill, and the safetensors recipe has no equivalent.
"cache_prompt": False,
"stream": True,
}).encode()
request = urllib.request.Request(
f"{self._url}/completion", data=body,
headers={"Content-Type": "application/json"}, method="POST",
)
started = time.monotonic()
chunks: list[str] = []
timings: Mapping[str, Any] = {}
with urllib.request.urlopen(request, timeout=600) as response:
for raw in response:
line = raw.decode("utf-8").strip()
if not line.startswith("data:"):
continue
payload = json.loads(line[len("data:"):].strip())
content = payload.get("content", "")
chunks.append(content)
if payload.get("stop"):
timings = payload.get("timings") or {}
total_ms = (time.monotonic() - started) * 1000
if not timings:
raise BenchmarkError("llama-server returned no timings; cannot report an honest split")
prefill_ms = float(timings.get("prompt_ms", 0.0))
decode_ms = float(timings.get("predicted_ms", 0.0))
return GenerationSample(
text="".join(chunks),
prompt_tokens=int(timings.get("prompt_n", 0)),
# llama.cpp starts predicted_ms after sampling the first token while
# predicted_n includes it. Exclude that token to match the
# Transformers inter-token decode metric.
decode_tokens=max(0, int(timings.get("predicted_n", 0)) - 1),
# Use the runtime's prompt/first-token timing, matching the
# in-process Transformers boundary. HTTP/SSE and slot delay remain
# represented by total latency and queue_wait_ms.
ttft_ms=prefill_ms,
prefill_ms=prefill_ms,
decode_ms=decode_ms,
total_ms=total_ms,
# Whatever the wall clock saw but the runtime did not attribute to
# compute is time this request spent waiting for a slot.
queue_wait_ms=max(0.0, total_ms - prefill_ms - decode_ms),
)
def memory_probe(self) -> tuple[int, int]:
if self._process is None:
return 0, 0
return _process_rss(self._process.pid), 0
def close(self) -> None:
if self._process is not None:
if self._process.poll() is None:
self._process.terminate()
try:
self._process.wait(timeout=20)
except subprocess.TimeoutExpired:
self._process.kill()
self._process.wait(timeout=10)
self._process = None
if self._log is not None:
self._log.close()
self._log = None
def build_driver(spec: Mapping[str, Any], plan: BenchmarkPlan) -> RecipeDriverBundle:
"""Construct the driver named by a recipe's ``driver`` block."""
driver_spec = dict(spec["driver"])
kind = driver_spec.pop("type")
if kind == "transformers":
return TransformersDriver(**driver_spec)
if kind == "llama-cpp-server":
driver_spec.setdefault("n_parallel", max(plan.concurrency_levels))
return LlamaCppServerDriver(**driver_spec)
raise BenchmarkError(f"unknown driver type {kind!r}")
RecipeDriverBundle = Any # a RecipeDriver; named for readability at the call site
def _plan_from_config(config: Mapping[str, Any]) -> BenchmarkPlan:
raw = config["plan"]
return BenchmarkPlan(
plan_id=raw["plan_id"],
model_id=raw["model_id"],
model_revision=raw["model_revision"],
prompts=tuple(PromptSpec(**prompt) for prompt in raw["prompts"]),
sampling=SamplingPolicy(**raw.get("sampling", {})),
concurrency_levels=tuple(raw.get("concurrency_levels", (1, 4))),
repeats=int(raw.get("repeats", 1)),
warmup_requests=int(raw.get("warmup_requests", 1)),
)
def _recipe_from_config(spec: Mapping[str, Any]) -> RecipeSpec:
return RecipeSpec(
id=spec["id"],
runtime=spec["runtime"],
weight_format=spec["weight_format"],
weight_quantization=spec["weight_quantization"],
lane=Lane(spec["lane"]),
device=spec["device"],
artifact_path=spec.get("artifact_path", ""),
source_model_id=spec.get("source_model_id", ""),
source_model_revision=spec.get("source_model_revision", ""),
artifact_sha256=spec.get("artifact_sha256", ""),
is_reference=bool(spec.get("is_reference", False)),
notes=spec.get("notes", ""),
)
def run_configured_benchmark(config: Mapping[str, Any]) -> dict:
"""Run every recipe in ``config`` against one shared plan and return the report.
A recipe whose runtime cannot start is recorded as unavailable with the real
reason rather than dropped: a report that silently omits the recipe that
crashed would read as a clean result.
"""
require_real_inference()
_validate_config(config)
plan = _plan_from_config(config)
from .recipe_benchmark import RecipeMeasurement # local import keeps the seam obvious
measurements = []
for spec in config["recipes"]:
recipe = _recipe_from_config(spec)
driver = None
try:
driver = build_driver(spec, plan)
measurements.append(measure_recipe(driver, recipe, plan))
except Exception as exc:
measurements.append(RecipeMeasurement(
recipe=recipe,
load=LoadStats(artifact_bytes=0, load_ms=0.0),
unavailable_reason=f"{type(exc).__name__}: {exc}",
))
finally:
if driver is not None:
driver.close()
return build_report(
plan,
measurements,
host={**dict(config.get("host", {})), **_host_manifest()},
evidence_class=config.get("evidence_class", "local-real"),
)

74
packages/node/native/CMakeLists.txt Normal file
View File

@@ -0,0 +1,74 @@
# Native Shard protocol: C++ schema generation, build wiring and conformance test.
#
# The C++ stubs are generated into the build tree at configure time — they are
# never committed. A C++ consumer already needs a toolchain, so committing
# generated C++ would only create a second copy of the schema that can rot.
#
# gRPC C++ is optional here on purpose. The conformance test only needs message
# types, so the schema can be verified on a machine that has protobuf but not
# the gRPC C++ stack. When gRPC *is* found, the service stubs are generated too
# and exported as `shard_runtime_grpc` for the worker (DGR-008) to link.
#
# Build:
# cmake -S packages/node/native -B build/native -DCMAKE_PREFIX_PATH=<protobuf-install>
# cmake --build build/native -j
# ctest --test-dir build/native --output-on-failure
#
# `scripts/bootstrap_native_toolchain.sh` builds a protobuf install from source
# when the system has none.
cmake_minimum_required(VERSION 3.24)
project(meshnet_shard_protocol CXX)
set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
find_package(protobuf CONFIG REQUIRED)
find_package(gRPC CONFIG QUIET)
set(SHARD_PROTO "${CMAKE_CURRENT_SOURCE_DIR}/proto/shard_runtime.proto")
# Message types: always available.
add_library(shard_runtime_proto STATIC "${SHARD_PROTO}")
target_link_libraries(shard_runtime_proto PUBLIC protobuf::libprotobuf)
target_include_directories(shard_runtime_proto PUBLIC "${CMAKE_CURRENT_BINARY_DIR}")
protobuf_generate(
TARGET shard_runtime_proto
LANGUAGE cpp
IMPORT_DIRS "${CMAKE_CURRENT_SOURCE_DIR}/proto"
PROTOC_OUT_DIR "${CMAKE_CURRENT_BINARY_DIR}"
)
# Service stubs: only when the gRPC C++ stack is present.
if(gRPC_FOUND)
add_library(shard_runtime_grpc STATIC "${SHARD_PROTO}")
target_link_libraries(shard_runtime_grpc PUBLIC shard_runtime_proto gRPC::grpc++)
target_include_directories(shard_runtime_grpc PUBLIC "${CMAKE_CURRENT_BINARY_DIR}")
protobuf_generate(
TARGET shard_runtime_grpc
LANGUAGE grpc
GENERATE_EXTENSIONS .grpc.pb.h .grpc.pb.cc
PLUGIN "protoc-gen-grpc=$<TARGET_FILE:gRPC::grpc_cpp_plugin>"
IMPORT_DIRS "${CMAKE_CURRENT_SOURCE_DIR}/proto"
PROTOC_OUT_DIR "${CMAKE_CURRENT_BINARY_DIR}"
)
message(STATUS "gRPC C++ found: building ShardRuntime service stubs")
else()
message(STATUS "gRPC C++ not found: building message types only "
"(sufficient for the conformance test)")
endif()
enable_testing()
add_executable(shard_protocol_conformance tests/test_shard_protocol_conformance.cpp)
target_link_libraries(shard_protocol_conformance PRIVATE shard_runtime_proto)
# The test asserts against the same committed vectors the Python test uses, and
# writes its own re-serialization back out so Python can prove the two languages
# agree byte-for-byte.
add_test(
NAME shard_protocol_conformance
COMMAND shard_protocol_conformance
"${CMAKE_CURRENT_SOURCE_DIR}/testdata"
"${CMAKE_CURRENT_BINARY_DIR}"
)

69
packages/node/native/README.md Normal file
View File

@@ -0,0 +1,69 @@
# Native Shard protocol
`proto/shard_runtime.proto` is the semantic contract between a Meshnet node and
a Shard worker: Protocol Buffers over gRPC/HTTP2 (ADR-0020). It is the source of
truth. The Python and C++ types are generated from it; neither is the contract.
## What lives here
| Path | Purpose |
|---|---|
| `proto/shard_runtime.proto` | The schema: capability, health, session stream, release, cancel |
| `testdata/*.binpb` | Committed conformance vectors both languages assert against |
| `tests/test_shard_protocol_conformance.cpp` | C++ conformance test |
| `CMakeLists.txt` | C++ generation, build wiring, and `ctest` registration |
The Python stubs are generated into
`packages/node/meshnet_node/native_protocol/generated/` and are committed, so
installing a node needs no protoc. The C++ stubs are generated into the build
tree and are never committed — a C++ consumer already has a toolchain, and a
committed copy could only rot.
## Regenerating
```bash
pip install grpcio-tools==1.82.1 # bundles protoc; no system protoc needed
python scripts/generate_native_protocol.py # rewrite the Python stubs
python scripts/generate_native_protocol.py --check # fail if they drifted
python scripts/generate_protocol_goldens.py --check # fail if the vectors drifted
```
Both `--check` modes run in CI via `tests/test_native_shard_protocol.py`, so a
schema edit that is not accompanied by regenerated output fails the suite rather
than shipping stubs that disagree with the schema they claim to implement.
## Building and running the C++ conformance test
If the machine has no protobuf C++ toolchain:
```bash
scripts/bootstrap_native_toolchain.sh build/native-toolchain
```
Then:
```bash
cmake -S packages/node/native -B build/native \
-DCMAKE_PREFIX_PATH="$PWD/build/native-toolchain"
cmake --build build/native -j
ctest --test-dir build/native --output-on-failure
```
gRPC C++ is optional: without it, CMake builds the message types only, which is
all the conformance test needs. When gRPC C++ *is* found, the `ShardRuntime`
service stubs are built too and exported as `shard_runtime_grpc` for the worker
(DGR-008) to link.
## How the cross-language check actually proves something
Two codecs that each round-trip their own output prove only that each is
self-consistent. Instead:
1. Python builds the canonical message and commits its bytes to `testdata/`.
2. The C++ test parses *those* bytes, asserts every field, independently
recomputes the CRC32C from the polynomial, and re-serializes to
`cpp_roundtrip.binpb` in the build tree.
3. `test_cpp_and_python_agree_byte_for_byte` compares that file to the golden.
Byte equality across the two implementations is the claim; anything less is two
parallel test suites that can drift apart.

592
packages/node/native/proto/shard_runtime.proto Normal file
View File

@@ -0,0 +1,592 @@
// The native Shard data plane: Protocol Buffers over gRPC/HTTP2 (ADR-0020).
//
// This schema — not any Python or C++ type — is the semantic contract between a
// Meshnet node and a Shard worker. Direct hops speak gRPC. When direct
// connectivity is unavailable the existing relay carries these exact serialized
// frames as opaque binary, which is why every deadline, cancellation and
// identity field is carried *in the schema* rather than left to gRPC metadata:
// a relayed frame has no HTTP/2 context to inherit them from.
//
// Meshnet remains the only control plane. Nothing here selects routes, prices
// work, or authenticates peers; the worker executes a layer range it has been
// admitted for and reports what it did.
syntax = "proto3";
package meshnet.shard.v1;
option cc_enable_arenas = true;
option go_package = "github.com/meshnet/shard/v1;shardv1";
// ---------------------------------------------------------------------------
// Versioning
// ---------------------------------------------------------------------------
// Wire-compatibility generation of this schema. A worker rejects a peer whose
// SCHEMA_VERSION it cannot satisfy instead of guessing at field meaning.
//
// This is deliberately NOT the `X-Meshnet-Wire` version of the legacy HTTP
// activation format (currently "2", see ADR-0008). The native protocol is a
// separate contract with its own generation counter and starts at 1.
enum SchemaVersion {
SCHEMA_VERSION_UNSPECIFIED = 0;
SCHEMA_VERSION_1 = 1;
}
// ---------------------------------------------------------------------------
// Tensor bundle — the public activation boundary
// ---------------------------------------------------------------------------
// Element type of a tensor payload.
//
// bfloat16 is the canonical activation dtype at every Shard boundary regardless
// of the weight quantization a node uses internally (ADR-0008): weights may be
// NF4 or INT8, compute upcasts, and the boundary stays bfloat16. The other
// values exist for auxiliary tensors (position ids, attention masks) and for
// architectures whose boundary is not a plain hidden state.
enum DType {
DTYPE_UNSPECIFIED = 0;
DTYPE_BFLOAT16 = 1;
DTYPE_FLOAT16 = 2;
DTYPE_FLOAT32 = 3;
DTYPE_INT32 = 4;
DTYPE_INT64 = 5;
DTYPE_UINT8 = 6;
DTYPE_INT8 = 7;
DTYPE_BOOL = 8;
}
// Byte order of a tensor payload. Little-endian is canonical on the wire; the
// field is explicit so a big-endian peer is rejected loudly rather than
// silently reading byte-swapped activations.
enum ByteOrder {
BYTE_ORDER_UNSPECIFIED = 0;
BYTE_ORDER_LITTLE_ENDIAN = 1;
BYTE_ORDER_BIG_ENDIAN = 2;
}
// Payload compression. Mirrors the policy-driven zstd decision the existing
// activation seam already makes (`activation_compression.py`): compression is a
// transport optimisation and NONE is always a legal choice for any payload.
enum Compression {
COMPRESSION_UNSPECIFIED = 0;
COMPRESSION_NONE = 1;
COMPRESSION_ZSTD = 2;
}
// Integrity algorithm covering a payload.
enum ChecksumAlgorithm {
CHECKSUM_ALGORITHM_UNSPECIFIED = 0;
CHECKSUM_ALGORITHM_NONE = 1;
CHECKSUM_ALGORITHM_CRC32C = 2;
}
// An integrity check over the *uncompressed* canonical payload bytes.
//
// Checksumming the uncompressed bytes (not the compressed frame) means the
// value is stable whether a hop compressed the payload or not, so a relay may
// re-frame or a peer may decline compression without invalidating it.
message Checksum {
ChecksumAlgorithm algorithm = 1;
// Big-endian encoding of the checksum value.
bytes value = 2;
}
// One slice of a tensor's wire payload.
//
// Fragments bound individual payload pieces and make coverage independently
// verifiable. The negotiated `FlowControl.max_chunk_bytes` bounds the enclosing
// serialized stream message; `byte_offset` proves the pieces tile the tensor
// exactly — no hole, no overlap — instead of trusting arrival order.
//
// Offsets and payloads describe the *wire* body, which is the compressed frame
// when the parent tensor declares a compression. A zstd frame is not decodable
// per fragment, so a receiver reassembles first and decompresses once. The
// uncompressed size is not repeated here: `NamedTensor.total_bytes` is the
// single source of truth, and a second copy could only ever disagree with it.
message TensorFragment {
uint32 fragment_index = 1;
uint32 fragment_count = 2;
// Offset of this fragment within the tensor's wire body.
uint64 byte_offset = 3;
// Fragment bytes, compressed iff the parent tensor declares a compression.
bytes payload = 4;
reserved 5;
reserved "uncompressed_size";
}
// One named tensor at an architecture boundary.
message NamedTensor {
// Architecture-defined boundary name, e.g. "hidden_states", "position_ids".
// A boundary may need more than one tensor, which is why the payload is a
// named bundle rather than a bare buffer (ADR-0020).
string name = 1;
repeated int64 shape = 2;
DType dtype = 3;
ByteOrder byte_order = 4;
// Total uncompressed payload size. A receiver sizes its buffer from this
// before reading fragments and refuses a bundle that exceeds its budget.
uint64 total_bytes = 5;
Compression compression = 6;
// Over the uncompressed payload; see Checksum.
Checksum checksum = 7;
// Ordered fragments. A tensor small enough for one message carries exactly
// one fragment with fragment_count == 1.
repeated TensorFragment fragments = 8;
}
// A versioned named-tensor bundle: the public activation boundary payload.
message TensorBundle {
// Generation of the bundle layout itself, independent of SchemaVersion so a
// boundary payload can evolve without a whole-protocol version bump.
uint32 bundle_version = 1;
repeated NamedTensor tensors = 2;
}
// ---------------------------------------------------------------------------
// Identity: what work this is, for which route, against which artifact
// ---------------------------------------------------------------------------
// Exact identity of the executable thing a Shard runs.
//
// Both fingerprints must match end to end across a route. Two nodes agreeing on
// a model name but not on quantization, kernel or tail-norm placement would
// produce silently wrong tokens, so identity is a digest, not a label.
message Fingerprint {
// Digest of the exact Model Artifact (weights + config), e.g. the GGUF hash.
string model_artifact_digest = 1;
// Digest of the exact runtime recipe (backend, quantization, kernels, dtype).
string runtime_recipe_digest = 2;
// Human-readable recipe identity, mirroring the capability report an admitted
// node already registers with (ADR-0023). Labels are for diagnosis; the
// digests above are what a peer actually compares.
string recipe_id = 3;
string recipe_version = 4;
string catalogue_version = 5;
}
// The contiguous transformer layer range a Shard owns.
message ShardRange {
// Registered range, inclusive start, exclusive end.
uint32 start_layer = 1;
uint32 end_layer = 2;
// The layer this Shard must actually begin at for this route (ADR-0012).
//
// Shard ranges may overlap; the Tracker resolves the overlap when it builds
// the route and every hop is told where the previous hop stopped. Executing
// from `start_layer` when `effective_start_layer` is higher would re-apply
// layers already applied to the incoming activation and silently corrupt the
// result. A worker executes [effective_start_layer, end_layer).
uint32 effective_start_layer = 3;
}
// Which part of generation a message belongs to.
enum Phase {
PHASE_UNSPECIFIED = 0;
PHASE_PREFILL = 1;
PHASE_DECODE = 2;
PHASE_RELEASE = 3;
PHASE_CANCEL = 4;
}
// Token span carried by one chunk.
message PositionSpan {
// Absolute position of the first token in this chunk within the sequence.
uint64 first_position = 1;
// Number of token positions in this chunk. A decode step carries 1.
uint32 token_count = 2;
}
// Bounded chunking for a prefill.
//
// A long prompt is split into token-aligned chunks before the first forward
// pass (ADR-0008) so peak transfer per boundary stays bounded regardless of
// prompt length. Splits never fall mid-token.
message ChunkInfo {
uint32 chunk_index = 1;
uint32 chunk_count = 2;
// True on the final chunk of a prefill. A receiver that has not seen a final
// chunk knows the prefill is still incomplete.
bool final_chunk = 3;
}
// How the sender expects the receiver's Hot KV State to be positioned.
enum CacheMode {
CACHE_MODE_UNSPECIFIED = 0;
// No session state; the payload carries everything needed. Always a legal
// fallback and the recovery path after a miss.
CACHE_MODE_STATELESS = 1;
// Establish fresh session state for this Shard's layer range.
CACHE_MODE_PREFILL = 2;
// Continue existing session state. `expected_past_len` must match exactly.
CACHE_MODE_DECODE = 3;
}
// The sender's expectation about receiver-side cache state (ADR-0022).
//
// A mismatch is always a declared cache miss, never a silent stateless forward:
// running a single-token decode payload without the matching cache would emit
// plausible garbage. The receiver answers a miss with ShardError.CACHE_MISS and
// the head re-prefills the whole sequence under the same Route Session ID.
message CacheExpectation {
CacheMode mode = 1;
// Decode only: the number of tokens the receiver's session cache must already
// hold for this Shard's layer range.
uint64 expected_past_len = 2;
}
// What the receiver's cache actually did.
message CacheResult {
CacheMode mode = 1;
// Tokens held for this Shard's range after the step completed.
uint64 past_len = 2;
// True when session state was reused rather than rebuilt.
bool cache_hit = 3;
}
// Everything required to interpret one unit of work, independent of transport.
//
// Carried on every request and response because a relayed frame arrives as
// opaque binary with no gRPC metadata, no deadline and no channel identity.
message Envelope {
SchemaVersion schema_version = 1;
// Unique id for this unit of work; also the unit of billing attribution.
string work_id = 2;
// The Route Session this work belongs to. Together with `route_epoch` it maps
// to exactly one isolated llama sequence / bounded context (ADR-0020).
string route_session_id = 3;
// Bumped by the control plane whenever the route changes. A worker refuses
// work from a stale epoch rather than mixing it into live session state.
uint64 route_epoch = 4;
Fingerprint fingerprint = 5;
ShardRange shard_range = 6;
Phase phase = 7;
PositionSpan position = 8;
// Monotonic step within (route_session_id, route_epoch).
//
// Idempotency key: a retried or duplicated step carries the same value and
// must be acknowledged, not re-applied. Re-applying a decode step would
// advance the KV cache twice and desynchronise the route.
uint64 idempotency_step = 9;
CacheExpectation cache_expectation = 10;
// Absolute deadline. Carried in-band so a relayed frame keeps its deadline;
// on a direct hop it is set consistently with the gRPC deadline.
int64 deadline_unix_nanos = 11;
// Chunking, when this message is part of a bounded prefill.
ChunkInfo chunk = 12;
}
// ---------------------------------------------------------------------------
// Structured errors
// ---------------------------------------------------------------------------
// Why a unit of work could not be executed as asked.
//
// These are semantic outcomes, distinct from transport failure. They travel as
// a ShardStatus on the stream (and as google.rpc.Status details on unary RPCs)
// so a relayed frame carries the same diagnosis a direct gRPC hop would.
enum ErrorCode {
ERROR_CODE_UNSPECIFIED = 0;
// Peer cannot satisfy the requested SchemaVersion.
ERROR_CODE_SCHEMA_UNSUPPORTED = 1;
// Model artifact or runtime recipe digest differs from this worker's.
ERROR_CODE_FINGERPRINT_MISMATCH = 2;
// Work arrived for a route epoch the worker has already moved past.
ERROR_CODE_EPOCH_STALE = 3;
// Requested layer range is not the range this worker serves.
ERROR_CODE_SHARD_RANGE_MISMATCH = 4;
// Session state absent or positioned differently than expected (ADR-0022).
// The head recovers with one full re-prefill; this is not a fatal error.
ERROR_CODE_CACHE_MISS = 5;
// Admission budget (weights, KV, scratch, queue depth) would be exceeded.
ERROR_CODE_RESOURCE_EXHAUSTED = 6;
// Payload failed its checksum, or fragments did not cover the tensor.
ERROR_CODE_PAYLOAD_CORRUPT = 7;
// Work was cancelled by the control plane or the peer.
ERROR_CODE_CANCELLED = 8;
// Deadline in the envelope had already passed when work was dequeued.
ERROR_CODE_DEADLINE_EXCEEDED = 9;
// Sender exceeded its granted flow-control credit.
ERROR_CODE_FLOW_CONTROL_VIOLATION = 10;
// The worker failed while executing; detail is sanitized.
ERROR_CODE_INTERNAL = 11;
}
message ShardError {
ErrorCode code = 1;
// Sanitized, operator-facing explanation. Never a raw exception, file path or
// credential (ADR-0023 sanitization rule).
string detail = 2;
// True when the same work may be retried unchanged. A CACHE_MISS is not
// retryable unchanged — the head must re-prefill first.
bool retryable = 3;
// Set on CACHE_MISS so the head knows where the receiver actually is.
uint64 actual_past_len = 4;
}
// ---------------------------------------------------------------------------
// Flow control
// ---------------------------------------------------------------------------
// Application-level credit, layered on top of HTTP/2 flow control.
//
// HTTP/2 bounds bytes in flight; it does not bound how much *work* a worker has
// queued, and a relayed frame gets no HTTP/2 window at all. Credits bound the
// number of un-acked chunks a sender may have outstanding, which is what keeps
// worker queues and KV pressure bounded and lets prefill avoid starving decode.
message FlowControl {
// Additional chunks the receiver is willing to accept beyond those already
// granted. Purely additive: a grant never reduces outstanding credit.
uint32 credits_granted = 1;
// Hard ceiling on un-acked chunks, independent of granted credit.
uint32 max_inflight_chunks = 2;
// Hard ceiling on the serialized size of one chunk message.
uint64 max_chunk_bytes = 3;
// Hard ceiling on token positions per prefill chunk.
uint32 max_prefill_chunk_tokens = 4;
}
// ---------------------------------------------------------------------------
// Session stream messages
// ---------------------------------------------------------------------------
// Opens one long-lived bidirectional stream for one Route Session Activation
// Seam. The handshake settles version, identity and the initial flow-control
// window before any activation is sent, so an incompatible peer fails at open
// rather than mid-generation.
message SessionOpen {
// Highest schema version the initiator supports; the acceptor replies with
// the version actually negotiated.
SchemaVersion schema_version = 1;
string route_session_id = 2;
uint64 route_epoch = 3;
Fingerprint fingerprint = 4;
ShardRange shard_range = 5;
// Flow-control limits the initiator can honour. The acceptor replies with the
// limits that actually apply.
FlowControl proposed_flow_control = 6;
// Compressions the initiator can decode. NONE is implicitly always supported.
repeated Compression accepted_compression = 7;
}
message SessionAccepted {
// The version both peers will use for the life of this stream.
SchemaVersion schema_version = 1;
string route_session_id = 2;
uint64 route_epoch = 3;
// Authoritative limits. The initiator must not exceed these.
FlowControl flow_control = 4;
repeated Compression accepted_compression = 5;
// Fingerprint the worker actually serves, so a mismatch is visible at open.
Fingerprint fingerprint = 6;
}
// One unit of activation work: a bounded prefill chunk or a decode step.
message ActivationChunk {
Envelope envelope = 1;
TensorBundle bundle = 2;
}
// The small decode fast path.
//
// A decode step is one token: the envelope's identity fields are already fixed
// for the life of the stream, so repeating them per token is pure overhead on
// the hottest path. A DecodeStep carries only what changes — the step, the
// position, and one tensor — and inherits the rest from the SessionOpen
// handshake. A peer may always fall back to ActivationChunk with PHASE_DECODE.
message DecodeStep {
// Idempotency step within the session; also orders the stream.
uint64 idempotency_step = 1;
// Absolute position of this token.
uint64 position = 2;
// Tokens the receiver's cache must already hold. A mismatch is a CACHE_MISS.
uint64 expected_past_len = 3;
// The single boundary tensor for this token, typically [1, 1, hidden].
NamedTensor tensor = 4;
// Work id for attribution; the session/epoch/fingerprint/range are inherited.
string work_id = 5;
int64 deadline_unix_nanos = 6;
}
// Drop session state for a Route Session. Bounded memory does not depend on
// this arriving — workers also evict by TTL and LRU (ADR-0022) — but an
// explicit release returns KV immediately instead of holding it for the TTL.
message ReleaseSignal {
string route_session_id = 1;
uint64 route_epoch = 2;
string work_id = 3;
}
// Abandon in-flight work.
//
// Cancellation must remain possible when the stream is wedged behind flow
// control, which is why Cancel also exists as a unary RPC on a fresh call.
message CancelSignal {
string route_session_id = 1;
uint64 route_epoch = 2;
// Cancel one unit of work; empty cancels every in-flight unit for the session.
string work_id = 3;
string reason = 4;
}
// Acknowledges a chunk or decode step, releasing its flow-control credit and
// recording where the receiver's cache now stands.
message Ack {
string work_id = 1;
uint64 idempotency_step = 2;
CacheResult cache_result = 3;
// True when the step was recognised as an already-applied duplicate and was
// therefore acknowledged without being re-applied.
bool duplicate = 4;
// Observed execution time, for Generation Telemetry.
uint64 execution_nanos = 5;
}
// A terminal outcome for one unit of work, or for the session.
message ShardStatus {
string work_id = 1;
string route_session_id = 2;
uint64 idempotency_step = 3;
ShardError error = 4;
// True when the stream itself is finished and no further work will be served.
bool terminal = 5;
}
message SessionRequest {
oneof kind {
SessionOpen open = 1;
ActivationChunk chunk = 2;
DecodeStep decode = 3;
FlowControl flow_control = 4;
ReleaseSignal release = 5;
CancelSignal cancel = 6;
}
}
message SessionResponse {
oneof kind {
SessionAccepted accepted = 1;
// Result activations forwarded toward the next Shard, or to the head.
ActivationChunk chunk = 2;
Ack ack = 3;
FlowControl flow_control = 4;
ShardStatus status = 5;
}
}
// ---------------------------------------------------------------------------
// Capability and health
// ---------------------------------------------------------------------------
message CapabilityRequest {
SchemaVersion schema_version = 1;
}
// What this worker can actually execute, proven by a bounded real forward
// (ADR-0023). Detected hardware is not a capability; only a validated recipe is.
message CapabilityReport {
SchemaVersion schema_version = 1;
Fingerprint fingerprint = 2;
ShardRange shard_range = 3;
// Backend/device identity, e.g. "rocm:gfx1151", "cpu:avx512".
string backend = 4;
string device = 5;
// True only when a bounded real forward has passed for exactly this
// (artifact, range, recipe, device). Never set from hardware detection alone.
bool validated = 6;
// Sanitized reason when `validated` is false.
string detail = 7;
// Admission budgets this worker will enforce.
uint64 max_concurrent_sessions = 8;
uint64 max_context_tokens = 9;
FlowControl flow_control = 10;
repeated Compression accepted_compression = 11;
repeated SchemaVersion supported_schema_versions = 12;
int64 validated_at_unix_nanos = 13;
}
message HealthRequest {
SchemaVersion schema_version = 1;
}
enum ServingState {
SERVING_STATE_UNSPECIFIED = 0;
// Accepting new Route Sessions.
SERVING_STATE_SERVING = 1;
// Alive but refusing new sessions; existing sessions continue draining.
SERVING_STATE_DRAINING = 2;
// Alive but not routable — e.g. registered-but-dark pending certification.
SERVING_STATE_NOT_SERVING = 3;
}
// Live load, for backpressure and Generation Telemetry.
message HealthReport {
SchemaVersion schema_version = 1;
ServingState state = 2;
uint32 active_sessions = 3;
uint32 queued_chunks = 4;
// Occupancy of the continuous decode batch.
uint32 batch_occupancy = 5;
// Fraction of the KV budget in use, 0..1.
float kv_pressure = 6;
uint64 resident_bytes = 7;
string detail = 8;
}
message ReleaseRequest {
SchemaVersion schema_version = 1;
string route_session_id = 2;
uint64 route_epoch = 3;
}
message ReleaseResponse {
// True when session state existed and was dropped; false when there was
// nothing to drop, which is a success, not an error — release is idempotent.
bool released = 1;
ShardError error = 2;
}
message CancelRequest {
SchemaVersion schema_version = 1;
string route_session_id = 2;
uint64 route_epoch = 3;
// Empty cancels every in-flight unit for the session.
string work_id = 4;
string reason = 5;
}
message CancelResponse {
uint32 cancelled_work_items = 1;
ShardError error = 2;
}
// ---------------------------------------------------------------------------
// Service
// ---------------------------------------------------------------------------
service ShardRuntime {
// What this worker can execute. Read before a route is built.
rpc GetCapability(CapabilityRequest) returns (CapabilityReport);
// Live load and serving state.
rpc Health(HealthRequest) returns (HealthReport);
// One long-lived bidirectional stream per Route Session Activation Seam.
//
// The stream opens with SessionOpen/SessionAccepted, then carries bounded
// prefill chunks and decode steps in both directions for the life of the
// session. Per-token channel creation is a non-goal: the handshake cost is
// paid once and the hot path carries only what changes.
rpc Session(stream SessionRequest) returns (stream SessionResponse);
// Drop session state out of band. Idempotent.
rpc Release(ReleaseRequest) returns (ReleaseResponse);
// Cancel out of band, on a fresh call.
//
// In-band CancelSignal is preferred, but a sender that is blocked on flow
// control cannot write one — a cancel that can only travel down a wedged
// stream is not a cancel. This RPC always has a path to the worker.
rpc Cancel(CancelRequest) returns (CancelResponse);
}

2
packages/node/native/testdata/capability_report_golden.binpb vendored Normal file
View File

@@ -0,0 +1,2 @@
F
sha256:1f0c9d2esha256:ab77e410llama-gguf-q4km-rocm"3* 2026.07.1 "rocm*gfx11510@H€@R €€€ €Zbh€€Ð<E282AC>éθý

BIN
packages/node/native/testdata/session_request_golden.binpb vendored Normal file
View File

Binary file not shown.

337
packages/node/native/tests/test_shard_protocol_conformance.cpp Normal file
View File

@@ -0,0 +1,337 @@
// C++ conformance test for the native Shard protocol.
//
// This test does not check that C++ can round-trip its own output — that would
// only prove C++ is self-consistent. It parses the *Python-produced* committed
// vectors, asserts every field the protocol promises to carry, independently
// recomputes the CRC32C over the reassembled tensor, and re-serializes the
// message back out for the Python test to compare byte-for-byte. Only that
// closes the loop: both languages agree on the same bytes and the same meaning.
//
// Run via ctest; see packages/node/native/CMakeLists.txt.
#include "shard_runtime.pb.h"
#include <cstdint>
#include <cstdio>
#include <filesystem>
#include <fstream>
#include <iostream>
#include <sstream>
#include <string>
#include <vector>
// `pb` is already taken by protobuf's own generated headers.
namespace sp = meshnet::shard::v1;
namespace {
int g_failures = 0;
#define CHECK(cond) \
do { \
if (!(cond)) { \
std::cerr << "FAIL " << __FILE__ << ":" << __LINE__ << ": " << #cond \
<< "\n"; \
++g_failures; \
} \
} while (0)
#define CHECK_EQ(actual, expected) \
do { \
auto &&a_ = (actual); \
auto &&e_ = (expected); \
if (!(a_ == e_)) { \
std::cerr << "FAIL " << __FILE__ << ":" << __LINE__ << ": " << #actual \
<< " == " << #expected << "\n actual: " << a_ \
<< "\n expected: " << e_ << "\n"; \
++g_failures; \
} \
} while (0)
// Independent CRC32C (Castagnoli). Written from the polynomial rather than
// shared with the Python side on purpose: a checksum that both languages
// compute with the *same* code proves nothing about interoperability.
uint32_t Crc32c(const std::string &data) {
static uint32_t table[256];
static bool built = false;
if (!built) {
for (uint32_t i = 0; i < 256; ++i) {
uint32_t c = i;
for (int k = 0; k < 8; ++k) {
c = (c & 1) ? (c >> 1) ^ 0x82F63B78u : (c >> 1);
}
table[i] = c;
}
built = true;
}
uint32_t crc = 0xFFFFFFFFu;
for (unsigned char byte : data) {
crc = (crc >> 8) ^ table[(crc ^ byte) & 0xFF];
}
return crc ^ 0xFFFFFFFFu;
}
std::string ReadFile(const std::filesystem::path &path) {
std::ifstream in(path, std::ios::binary);
if (!in) {
std::cerr << "FAIL cannot read " << path << "\n";
++g_failures;
return {};
}
std::ostringstream buffer;
buffer << in.rdbuf();
return buffer.str();
}
// Reassemble a tensor's fragments and validate coverage, exactly as a worker
// must before it feeds the bytes to a forward pass.
std::string ReassembleUncompressed(const sp::NamedTensor &tensor) {
std::string body;
std::vector<const sp::TensorFragment *> fragments;
for (const auto &fragment : tensor.fragments()) {
fragments.push_back(&fragment);
}
CHECK(!fragments.empty());
for (uint32_t index = 0; index < fragments.size(); ++index) {
const sp::TensorFragment *found = nullptr;
for (const auto *fragment : fragments) {
if (fragment->fragment_index() == index) {
found = fragment;
break;
}
}
CHECK(found != nullptr);
if (found == nullptr) {
return {};
}
CHECK_EQ(found->fragment_count(),
static_cast<uint32_t>(fragments.size()));
// Offsets must tile the wire body exactly: no hole, no overlap.
CHECK_EQ(found->byte_offset(), static_cast<uint64_t>(body.size()));
body.append(found->payload());
}
return body;
}
void CheckFingerprint(const sp::Fingerprint &fingerprint) {
CHECK_EQ(fingerprint.model_artifact_digest(), std::string("sha256:1f0c9d2e"));
CHECK_EQ(fingerprint.runtime_recipe_digest(), std::string("sha256:ab77e410"));
CHECK_EQ(fingerprint.recipe_id(), std::string("llama-gguf-q4km-rocm"));
CHECK_EQ(fingerprint.recipe_version(), std::string("3"));
CHECK_EQ(fingerprint.catalogue_version(), std::string("2026.07.1"));
}
// The canonical session request, as produced by Python.
void TestSessionRequestVector(const std::string &bytes) {
sp::SessionRequest request;
CHECK(request.ParseFromString(bytes));
CHECK(request.kind_case() == sp::SessionRequest::kChunk);
const sp::ActivationChunk &chunk = request.chunk();
const sp::Envelope &envelope = chunk.envelope();
// Every field the protocol promises to carry (acceptance criterion 4).
CHECK_EQ(envelope.schema_version(), sp::SCHEMA_VERSION_1);
CHECK_EQ(envelope.work_id(), std::string("work-7f3a"));
CHECK_EQ(envelope.route_session_id(), std::string("rs-2b91"));
CHECK_EQ(envelope.route_epoch(), 7u);
CheckFingerprint(envelope.fingerprint());
CHECK_EQ(envelope.shard_range().start_layer(), 12u);
CHECK_EQ(envelope.shard_range().end_layer(), 24u);
// Overlap-safe effective start (ADR-0012): a worker that begins at
// start_layer instead would re-apply layers 12..15.
CHECK_EQ(envelope.shard_range().effective_start_layer(), 16u);
CHECK_EQ(envelope.phase(), sp::PHASE_PREFILL);
CHECK_EQ(envelope.position().first_position(), 256u);
CHECK_EQ(envelope.position().token_count(), 128u);
CHECK_EQ(envelope.idempotency_step(), 42u);
CHECK_EQ(envelope.cache_expectation().mode(), sp::CACHE_MODE_PREFILL);
CHECK_EQ(envelope.cache_expectation().expected_past_len(), 256u);
CHECK_EQ(envelope.deadline_unix_nanos(), 1800000000000000000LL);
CHECK_EQ(envelope.chunk().chunk_index(), 1u);
CHECK_EQ(envelope.chunk().chunk_count(), 3u);
CHECK_EQ(envelope.chunk().final_chunk(), false);
// The versioned named-tensor bundle (acceptance criterion 5).
const sp::TensorBundle &bundle = chunk.bundle();
CHECK_EQ(bundle.bundle_version(), 1u);
CHECK_EQ(bundle.tensors_size(), 1);
const sp::NamedTensor &tensor = bundle.tensors(0);
CHECK_EQ(tensor.name(), std::string("hidden_states"));
CHECK_EQ(tensor.shape_size(), 3);
CHECK_EQ(tensor.shape(0), 1);
CHECK_EQ(tensor.shape(1), 128);
CHECK_EQ(tensor.shape(2), 8);
CHECK_EQ(tensor.dtype(), sp::DTYPE_BFLOAT16);
CHECK_EQ(tensor.byte_order(), sp::BYTE_ORDER_LITTLE_ENDIAN);
CHECK_EQ(tensor.total_bytes(), 1u * 128u * 8u * 2u);
CHECK_EQ(tensor.compression(), sp::COMPRESSION_NONE);
// Multi-fragment on purpose: reassembly is what a worker actually does.
CHECK(tensor.fragments_size() > 1);
const std::string payload = ReassembleUncompressed(tensor);
CHECK_EQ(payload.size(), static_cast<size_t>(tensor.total_bytes()));
// The payload Python generated, recomputed here from its rule.
std::string expected;
expected.reserve(payload.size());
for (size_t i = 0; i < payload.size(); ++i) {
expected.push_back(static_cast<char>((i * 7 + 11) % 256));
}
CHECK(payload == expected);
// Checksum: computed by Python, verified by an independent C++ CRC32C.
CHECK_EQ(tensor.checksum().algorithm(), sp::CHECKSUM_ALGORITHM_CRC32C);
const std::string &checksum = tensor.checksum().value();
CHECK_EQ(checksum.size(), 4u);
if (checksum.size() == 4) {
const uint32_t actual = Crc32c(payload);
const uint32_t declared =
(static_cast<uint32_t>(static_cast<unsigned char>(checksum[0])) << 24) |
(static_cast<uint32_t>(static_cast<unsigned char>(checksum[1])) << 16) |
(static_cast<uint32_t>(static_cast<unsigned char>(checksum[2])) << 8) |
static_cast<uint32_t>(static_cast<unsigned char>(checksum[3]));
CHECK_EQ(actual, declared);
}
}
void TestCapabilityReportVector(const std::string &bytes) {
sp::CapabilityReport report;
CHECK(report.ParseFromString(bytes));
CHECK_EQ(report.schema_version(), sp::SCHEMA_VERSION_1);
CheckFingerprint(report.fingerprint());
CHECK_EQ(report.backend(), std::string("rocm"));
CHECK_EQ(report.device(), std::string("gfx1151"));
CHECK_EQ(report.validated(), true);
CHECK_EQ(report.max_concurrent_sessions(), 4u);
CHECK_EQ(report.max_context_tokens(), 8192u);
CHECK_EQ(report.flow_control().max_prefill_chunk_tokens(), 128u);
CHECK_EQ(report.flow_control().max_chunk_bytes(), 4u * 1024u * 1024u);
CHECK_EQ(report.accepted_compression_size(), 2);
CHECK_EQ(report.supported_schema_versions_size(), 1);
}
// Forward compatibility: a field this build has never heard of must survive a
// parse/serialize cycle untouched. Without this, an old Shard silently strips
// fields a newer peer depends on as it forwards activations down the route.
void TestUnknownFieldsArePreserved(const std::string &bytes) {
// Field 9999, wire type 0 (varint), value 12345 — not in this schema.
std::string forward = bytes;
forward.push_back(static_cast<char>(0xB8)); // tag: (9999 << 3) | 0
forward.push_back(static_cast<char>(0xE0));
forward.push_back(static_cast<char>(0x04));
forward.push_back(static_cast<char>(0xB9)); // value: 12345
forward.push_back(static_cast<char>(0x60));
sp::SessionRequest request;
CHECK(request.ParseFromString(forward));
// Known fields still parse.
CHECK_EQ(request.chunk().envelope().work_id(), std::string("work-7f3a"));
// And the unknown field is retained rather than dropped.
CHECK(!request.GetReflection()->GetUnknownFields(request).empty());
std::string reserialized;
CHECK(request.SerializeToString(&reserialized));
CHECK_EQ(reserialized.size(), forward.size());
sp::SessionRequest reparsed;
CHECK(reparsed.ParseFromString(reserialized));
CHECK(!reparsed.GetReflection()->GetUnknownFields(reparsed).empty());
}
// Backward compatibility: a message from a peer that sets none of the optional
// groups must still parse, yielding proto3 defaults rather than an error.
void TestSparseMessageParses() {
sp::SessionRequest minimal;
minimal.mutable_chunk()->mutable_envelope()->set_work_id("w");
std::string bytes;
CHECK(minimal.SerializeToString(&bytes));
sp::SessionRequest parsed;
CHECK(parsed.ParseFromString(bytes));
CHECK_EQ(parsed.chunk().envelope().work_id(), std::string("w"));
CHECK_EQ(parsed.chunk().envelope().route_epoch(), 0u);
CHECK_EQ(parsed.chunk().envelope().phase(), sp::PHASE_UNSPECIFIED);
CHECK_EQ(parsed.chunk().bundle().tensors_size(), 0);
}
// The decode fast path must stay small: repeating the full envelope per token
// is pure overhead on the hottest path in the system.
void TestDecodeFastPathIsSmall() {
sp::SessionRequest decode;
sp::DecodeStep *step = decode.mutable_decode();
step->set_idempotency_step(9);
step->set_position(1024);
step->set_expected_past_len(1024);
step->set_work_id("work-7f3a");
sp::NamedTensor *tensor = step->mutable_tensor();
tensor->set_name("hidden_states");
tensor->add_shape(1);
tensor->add_shape(1);
tensor->add_shape(8);
tensor->set_dtype(sp::DTYPE_BFLOAT16);
tensor->set_byte_order(sp::BYTE_ORDER_LITTLE_ENDIAN);
tensor->set_total_bytes(16);
tensor->set_compression(sp::COMPRESSION_NONE);
sp::TensorFragment *fragment = tensor->add_fragments();
fragment->set_fragment_index(0);
fragment->set_fragment_count(1);
fragment->set_byte_offset(0);
fragment->set_payload(std::string(16, '\x01'));
std::string bytes;
CHECK(decode.SerializeToString(&bytes));
// Payload is 16 bytes; the framing around it must stay well under the
// envelope-carrying prefill path.
CHECK(bytes.size() < 96);
sp::SessionRequest parsed;
CHECK(parsed.ParseFromString(bytes));
CHECK(parsed.kind_case() == sp::SessionRequest::kDecode);
CHECK_EQ(parsed.decode().position(), 1024u);
CHECK_EQ(parsed.decode().tensor().total_bytes(), 16u);
}
} // namespace
int main(int argc, char **argv) {
GOOGLE_PROTOBUF_VERIFY_VERSION;
if (argc < 3) {
std::cerr << "usage: " << argv[0] << " <testdata-dir> <output-dir>\n";
return 2;
}
const std::filesystem::path testdata(argv[1]);
const std::filesystem::path output(argv[2]);
const std::string session_bytes =
ReadFile(testdata / "session_request_golden.binpb");
const std::string capability_bytes =
ReadFile(testdata / "capability_report_golden.binpb");
TestSessionRequestVector(session_bytes);
TestCapabilityReportVector(capability_bytes);
TestUnknownFieldsArePreserved(session_bytes);
TestSparseMessageParses();
TestDecodeFastPathIsSmall();
// Re-serialize the canonical message from the C++ object model and hand it
// back for Python to compare against the golden bytes. If the two languages
// disagreed about any field's encoding, this file would differ.
sp::SessionRequest request;
if (request.ParseFromString(session_bytes)) {
std::string reserialized;
if (request.SerializeToString(&reserialized)) {
std::ofstream out(output / "cpp_roundtrip.binpb", std::ios::binary);
out.write(reserialized.data(),
static_cast<std::streamsize>(reserialized.size()));
}
}
if (g_failures != 0) {
std::cerr << g_failures << " check(s) failed\n";
return 1;
}
std::cout << "all C++ conformance checks passed\n";
return 0;
}

10
packages/node/pyproject.toml
View File

@@ -13,6 +13,11 @@ dependencies = [
"huggingface-hub>=0.20",
"accelerate>=0.28",
"bitsandbytes>=0.43",
# Keep these floors aligned with the committed grpcio-tools 1.82.1 output.
# Older runtimes satisfy broad constraints but fail while importing the
# generated modules before a node can report a useful protocol error.
"grpcio>=1.82.1",
"protobuf>=7.35.0",
"rich>=13",
"safetensors>=0.4",
"torch>=2.1",
@@ -23,6 +28,11 @@ dependencies = [
"kernels>=0.11.1,<0.16",
]
[project.optional-dependencies]
# Regenerating the native Shard protocol stubs. Not needed to run a node: the
# generated modules are committed, and `grpcio-tools` bundles its own protoc.
proto = ["grpcio-tools==1.82.1"]
[project.scripts]
meshnet-node = "meshnet_node.cli:main"

66
scripts/bootstrap_native_toolchain.sh Normal file
View File

@@ -0,0 +1,66 @@
#!/usr/bin/env bash
# Build a protobuf C++ toolchain for the native Shard protocol.
#
# The Python side needs nothing beyond `pip install grpcio-tools` — it bundles
# protoc. The C++ side needs libprotobuf headers and a protoc binary, and a
# machine that has neither (no protobuf-devel, no cmake, no system protoc) can
# still get a working one from source with this script. It is the exact recipe
# DGR-002 used to build and run the C++ conformance test.
#
# gRPC C++ is deliberately NOT built here. The conformance test only needs
# message types, so verifying the schema does not require the whole gRPC stack.
# The worker (DGR-008) will need gRPC C++ and should extend this script then.
#
# Usage:
# scripts/bootstrap_native_toolchain.sh [install-prefix]
#
# Then:
# cmake -S packages/node/native -B build/native -DCMAKE_PREFIX_PATH=<prefix>
# cmake --build build/native -j
# ctest --test-dir build/native --output-on-failure
set -euo pipefail
PREFIX="${1:-${PWD}/build/native-toolchain}"
WORK="$(mktemp -d)"
trap 'rm -rf "${WORK}"' EXIT
# Pinned: the C++ runtime a generated stub is compiled against must be a version
# that stub is allowed to use, so these are exact, not floating.
PROTOBUF_VERSION="33.1"
ABSEIL_VERSION="20250814.1"
command -v cmake >/dev/null || {
echo "cmake is required (pip install cmake==4.4.0)" >&2
exit 1
}
echo "--- fetching protobuf ${PROTOBUF_VERSION} and abseil ${ABSEIL_VERSION}"
cd "${WORK}"
curl -sfL -o protobuf.tar.gz \
"https://github.com/protocolbuffers/protobuf/releases/download/v${PROTOBUF_VERSION}/protobuf-${PROTOBUF_VERSION}.tar.gz"
tar xzf protobuf.tar.gz
# The protobuf release tarball ships utf8_range but not abseil, and its default
# CMake provider expects abseil as a submodule, so vendor it into place.
curl -sfL -o abseil.tar.gz \
"https://github.com/abseil/abseil-cpp/releases/download/${ABSEIL_VERSION}/abseil-cpp-${ABSEIL_VERSION}.tar.gz"
tar xzf abseil.tar.gz
rm -rf "protobuf-${PROTOBUF_VERSION}/third_party/abseil-cpp"
mv "abseil-cpp-${ABSEIL_VERSION}" "protobuf-${PROTOBUF_VERSION}/third_party/abseil-cpp"
echo "--- building protobuf into ${PREFIX}"
cmake -S "protobuf-${PROTOBUF_VERSION}" -B build \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_INSTALL_PREFIX="${PREFIX}" \
-DCMAKE_POSITION_INDEPENDENT_CODE=ON \
-Dprotobuf_ABSL_PROVIDER=module \
-Dprotobuf_BUILD_TESTS=OFF \
-Dprotobuf_BUILD_SHARED_LIBS=OFF \
-DABSL_PROPAGATE_CXX_STD=ON
cmake --build build -j"$(nproc)"
cmake --install build
echo "--- done"
"${PREFIX}/bin/protoc" --version
echo "configure the protocol build with: -DCMAKE_PREFIX_PATH=${PREFIX}"

125
scripts/generate_native_protocol.py Normal file
View File

@@ -0,0 +1,125 @@
#!/usr/bin/env python3
"""Generate the Python Shard-protocol stubs from `shard_runtime.proto`.
The `.proto` file is the contract; the Python modules under
`meshnet_node/native_protocol/generated/` are build output that happens to be
committed. They are committed so that installing the node does not require a
protoc toolchain, and `--check` exists so a committed stub can never silently
drift from the schema it claims to implement.
Usage::
python scripts/generate_native_protocol.py # regenerate in place
python scripts/generate_native_protocol.py --check # fail if out of date
C++ stubs are *not* generated here. They are build artifacts produced by CMake
(`packages/node/native/CMakeLists.txt`) into the build tree, because a C++ build
already requires a toolchain and nothing is gained by committing them.
"""
from __future__ import annotations
import argparse
import pathlib
import shutil
import subprocess
import sys
import tempfile
REPO_ROOT = pathlib.Path(__file__).resolve().parent.parent
PROTO_DIR = REPO_ROOT / "packages/node/native/proto"
PROTO_FILE = PROTO_DIR / "shard_runtime.proto"
OUT_DIR = REPO_ROOT / "packages/node/meshnet_node/native_protocol/generated"
# Regenerating with a different protoc emits different gencode headers, so the
# generator version is part of the contract and `--check` would catch a drift.
REQUIRED_GRPCIO_TOOLS = "1.82.1"
_HEADER = "# Generated by scripts/generate_native_protocol.py. Do not edit.\n"
def _generate(into: pathlib.Path) -> None:
"""Run protoc, writing generated modules into `into`."""
try:
from grpc_tools import protoc
except ImportError: # pragma: no cover - exercised only without the toolchain
sys.exit(
"grpc_tools is required to generate stubs:\n"
f" pip install grpcio-tools=={REQUIRED_GRPCIO_TOOLS}"
)
into.mkdir(parents=True, exist_ok=True)
# grpc_tools bundles protoc and the well-known types, so generation needs no
# system protoc and produces identical output on any machine.
well_known = pathlib.Path(protoc.__file__).parent / "_proto"
args = [
"protoc",
f"--proto_path={PROTO_DIR}",
f"--proto_path={well_known}",
f"--python_out={into}",
f"--pyi_out={into}",
f"--grpc_python_out={into}",
str(PROTO_FILE),
]
if protoc.main(args) != 0:
sys.exit("protoc failed")
# protoc emits `import shard_runtime_pb2` — a bare top-level import that only
# resolves if the generated directory happens to be on sys.path. Rewrite it
# to a relative import so the package works as an installed package.
grpc_module = into / "shard_runtime_pb2_grpc.py"
text = grpc_module.read_text()
text = text.replace(
"import shard_runtime_pb2 as shard__runtime__pb2",
"from . import shard_runtime_pb2 as shard__runtime__pb2",
)
grpc_module.write_text(text)
(into / "__init__.py").write_text(
_HEADER + '"""Generated protobuf/gRPC stubs for the native Shard protocol."""\n'
)
def _tracked_files(directory: pathlib.Path) -> dict[str, bytes]:
return {
path.name: path.read_bytes()
for path in sorted(directory.iterdir())
if path.is_file() and path.suffix in {".py", ".pyi"}
}
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument(
"--check",
action="store_true",
help="verify the committed stubs match the .proto instead of rewriting them",
)
args = parser.parse_args()
if args.check:
with tempfile.TemporaryDirectory() as tmp:
fresh = pathlib.Path(tmp) / "generated"
_generate(fresh)
if not OUT_DIR.is_dir():
print(f"generated stubs are missing: {OUT_DIR}", file=sys.stderr)
return 1
if _tracked_files(fresh) != _tracked_files(OUT_DIR):
print(
"generated stubs are out of date with shard_runtime.proto.\n"
"Run: python scripts/generate_native_protocol.py",
file=sys.stderr,
)
return 1
print("generated stubs are up to date")
return 0
if OUT_DIR.exists():
shutil.rmtree(OUT_DIR)
_generate(OUT_DIR)
print(f"wrote {OUT_DIR.relative_to(REPO_ROOT)}")
return 0
if __name__ == "__main__":
raise SystemExit(main())

71
scripts/generate_protocol_goldens.py Normal file
View File

@@ -0,0 +1,71 @@
#!/usr/bin/env python3
"""Write the committed cross-language conformance vectors.
The bytes under `packages/node/native/testdata/` are the reference both the
Python and the C++ conformance tests assert against. They are committed so the
C++ test can run without a Python step, and `--check` exists so they can never
drift from the schema unnoticed: if a schema edit changes the canonical
message's encoding, `--check` fails and the change has to be acknowledged.
Usage::
python scripts/generate_protocol_goldens.py # rewrite vectors
python scripts/generate_protocol_goldens.py --check # fail if stale
"""
from __future__ import annotations
import argparse
import pathlib
import sys
REPO_ROOT = pathlib.Path(__file__).resolve().parent.parent
sys.path.insert(0, str(REPO_ROOT / "packages/node"))
from meshnet_node.native_protocol import conformance # noqa: E402
def _vectors() -> dict[str, bytes]:
return {
conformance.GOLDEN_SESSION_REQUEST: conformance.serialize(
conformance.canonical_session_request()
),
conformance.GOLDEN_CAPABILITY_REPORT: conformance.serialize(
conformance.canonical_capability_report()
),
}
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--check", action="store_true")
args = parser.parse_args()
out_dir = conformance.TESTDATA_DIR
out_dir.mkdir(parents=True, exist_ok=True)
stale = []
for name, payload in _vectors().items():
path = out_dir / name
if args.check:
if not path.is_file() or path.read_bytes() != payload:
stale.append(name)
continue
path.write_bytes(payload)
print(f"wrote {path.relative_to(REPO_ROOT)} ({len(payload)} bytes)")
if stale:
print(
"conformance vectors are stale: " + ", ".join(stale) + "\n"
"The canonical message no longer encodes to the committed bytes. If "
"that is intended, run: python scripts/generate_protocol_goldens.py",
file=sys.stderr,
)
return 1
if args.check:
print("conformance vectors are up to date")
return 0
if __name__ == "__main__":
raise SystemExit(main())

577
tests/test_native_shard_protocol.py Normal file
View File

@@ -0,0 +1,577 @@
"""Conformance tests for the native Shard protocol (ADR-0020, DGR-002).
Three layers are tested, and they are not the same thing:
1. The *schema* — asserted against the descriptor, not against the Python
helpers. If a field the protocol promises to carry were dropped from the
`.proto`, a test that only exercised the codec would still pass.
2. The *codec* — that a payload which is corrupt, short, holed, or byte-swapped
is rejected rather than fed to a forward pass.
3. *Compatibility* — that an old build preserves fields a newer peer added, and
that the committed cross-language vectors still encode as promised.
"""
from __future__ import annotations
import pathlib
import subprocess
import sys
import pytest
pytest.importorskip("google.protobuf", reason="protobuf runtime is required")
from google.protobuf import descriptor_pb2
from meshnet_node.activation_compression import CompressionPolicy
from meshnet_node.native_protocol import (
DEFAULT_MAX_CHUNK_BYTES,
DEFAULT_MAX_FRAGMENTS_PER_TENSOR,
DEFAULT_MAX_PREFILL_CHUNK_TOKENS,
DEFAULT_MAX_TENSORS_PER_BUNDLE,
HIDDEN_STATES,
PayloadCorrupt,
ProtocolError,
checksum_of,
decode_bundle,
decode_tensor,
default_flow_control,
encode_bundle,
encode_tensor,
negotiate_flow_control,
pb,
plan_prefill_chunks,
validate_session_message_size,
)
from meshnet_node.native_protocol import conformance
REPO_ROOT = pathlib.Path(__file__).resolve().parent.parent
CPP_ROUNDTRIP = REPO_ROOT / "build/native" / conformance.CPP_ROUNDTRIP
# --- The schema itself ------------------------------------------------------
def test_service_exposes_capability_health_session_release_and_cancel():
service = pb.DESCRIPTOR.services_by_name["ShardRuntime"]
assert set(service.methods_by_name) == {
"GetCapability",
"Health",
"Session",
"Release",
"Cancel",
}
def test_session_is_one_long_lived_bidirectional_stream():
session = pb.DESCRIPTOR.services_by_name["ShardRuntime"].methods_by_name["Session"]
assert session.client_streaming, "the seam must stream requests"
assert session.server_streaming, "the seam must stream responses"
# Cancellation must not have to travel down a stream that flow control has
# wedged, so it also exists as its own unary call.
cancel = pb.DESCRIPTOR.services_by_name["ShardRuntime"].methods_by_name["Cancel"]
assert not cancel.client_streaming and not cancel.server_streaming
def test_envelope_carries_every_field_the_protocol_promises():
# Asserted against the descriptor: this is the acceptance criterion, and it
# must fail if the .proto drops a field, not merely if the codec stops
# setting one.
fields = set(pb.Envelope.DESCRIPTOR.fields_by_name)
assert {
"schema_version",
"work_id",
"route_session_id",
"route_epoch",
"fingerprint",
"shard_range",
"phase",
"position",
"idempotency_step",
"cache_expectation",
"deadline_unix_nanos",
"chunk",
} <= fields
assert {"model_artifact_digest", "runtime_recipe_digest"} <= set(
pb.Fingerprint.DESCRIPTOR.fields_by_name
)
# Overlap-safe start (ADR-0012) is a distinct field from the registered one.
assert {"start_layer", "end_layer", "effective_start_layer"} <= set(
pb.ShardRange.DESCRIPTOR.fields_by_name
)
assert {"mode", "expected_past_len"} <= set(
pb.CacheExpectation.DESCRIPTOR.fields_by_name
)
def test_named_tensor_bundle_is_versioned_and_fully_described():
assert "bundle_version" in pb.TensorBundle.DESCRIPTOR.fields_by_name
assert {"name", "shape", "dtype", "byte_order", "compression", "checksum", "fragments"} <= set(
pb.NamedTensor.DESCRIPTOR.fields_by_name
)
def test_phases_and_error_codes_cover_the_lifecycle():
assert {"PHASE_PREFILL", "PHASE_DECODE", "PHASE_RELEASE", "PHASE_CANCEL"} <= set(
pb.Phase.keys()
)
# A cache miss is a first-class, recoverable outcome (ADR-0022), not a crash.
assert {
"ERROR_CODE_CACHE_MISS",
"ERROR_CODE_FINGERPRINT_MISMATCH",
"ERROR_CODE_EPOCH_STALE",
"ERROR_CODE_SCHEMA_UNSUPPORTED",
"ERROR_CODE_DEADLINE_EXCEEDED",
"ERROR_CODE_FLOW_CONTROL_VIOLATION",
} <= set(pb.ErrorCode.keys())
# --- Tensor bundle round-trip ----------------------------------------------
def test_tensor_round_trips_through_fragments():
payload = bytes(range(256)) * 4
tensor = encode_tensor(
HIDDEN_STATES, payload, [1, 64, 8], pb.DTYPE_BFLOAT16, max_fragment_bytes=100
)
assert len(tensor.fragments) > 1, "the bound must actually split the payload"
assert tensor.total_bytes == len(payload)
assert decode_tensor(tensor) == payload
def test_bundle_round_trips_multiple_named_tensors():
# An architecture boundary may need more than one tensor; that is why the
# payload is a named bundle rather than a bare buffer.
hidden = encode_tensor(HIDDEN_STATES, b"\x01\x02" * 8, [1, 8, 1], pb.DTYPE_BFLOAT16)
positions = encode_tensor(
"position_ids", (7).to_bytes(4, "little") * 8, [8], pb.DTYPE_INT32
)
bundle = encode_bundle([hidden, positions])
restored = decode_bundle(pb.TensorBundle.FromString(bundle.SerializeToString()))
assert restored == {
HIDDEN_STATES: b"\x01\x02" * 8,
"position_ids": (7).to_bytes(4, "little") * 8,
}
def test_compressed_tensor_round_trips_and_keeps_its_uncompressed_checksum():
pytest.importorskip("zstandard")
# Highly compressible, and over the policy's minimum input size.
payload = b"\x00" * 65536
always = CompressionPolicy(min_input_bytes=0, min_savings_bytes=0, min_savings_ratio=0.0)
tensor = encode_tensor(
HIDDEN_STATES, payload, [1, 4096, 8], pb.DTYPE_BFLOAT16, policy=always
)
assert tensor.compression == pb.COMPRESSION_ZSTD
assert sum(len(f.payload) for f in tensor.fragments) < len(payload)
# The checksum covers the uncompressed bytes, so it stays valid whether or
# not a hop chose to compress.
assert tensor.checksum == checksum_of(payload)
assert decode_tensor(tensor) == payload
# --- The codec refuses what it cannot account for --------------------------
def test_corrupt_payload_is_rejected_by_checksum():
tensor = encode_tensor(HIDDEN_STATES, b"\xaa" * 32, [1, 16, 1], pb.DTYPE_BFLOAT16)
# Flip one byte, as a lossy relay or a bad NIC would.
tensor.fragments[0].payload = b"\xab" + tensor.fragments[0].payload[1:]
with pytest.raises(PayloadCorrupt, match="CRC32C"):
decode_tensor(tensor)
def test_missing_fragment_is_rejected_rather_than_silently_truncated():
tensor = encode_tensor(
HIDDEN_STATES, b"\xaa" * 64, [1, 32, 1], pb.DTYPE_BFLOAT16, max_fragment_bytes=16
)
del tensor.fragments[1]
with pytest.raises(PayloadCorrupt):
decode_tensor(tensor)
def test_fragment_hole_is_rejected():
tensor = encode_tensor(
HIDDEN_STATES, b"\xaa" * 64, [1, 32, 1], pb.DTYPE_BFLOAT16, max_fragment_bytes=16
)
# A gap in coverage: offsets no longer tile the body.
tensor.fragments[2].byte_offset += 4
with pytest.raises(PayloadCorrupt, match="expected"):
decode_tensor(tensor)
def test_shape_that_disagrees_with_payload_is_rejected_at_encode():
with pytest.raises(ProtocolError, match="carries"):
encode_tensor(HIDDEN_STATES, b"\x01\x02", [1, 8, 8], pb.DTYPE_BFLOAT16)
def test_shape_that_disagrees_with_declared_bytes_is_rejected_at_decode():
tensor = encode_tensor(HIDDEN_STATES, b"\xaa" * 32, [1, 16, 1], pb.DTYPE_BFLOAT16)
# A peer claiming a larger tensor than its bytes describe.
tensor.shape[1] = 32
with pytest.raises(PayloadCorrupt, match="implies"):
decode_tensor(tensor)
def test_big_endian_tensor_is_rejected_loudly():
tensor = encode_tensor(HIDDEN_STATES, b"\xaa" * 32, [1, 16, 1], pb.DTYPE_BFLOAT16)
tensor.byte_order = pb.BYTE_ORDER_BIG_ENDIAN
# Byte-swapped activations would be plausible-looking garbage, so this is an
# error rather than a best-effort read.
with pytest.raises(ProtocolError, match="big-endian"):
decode_tensor(tensor)
def test_bundle_from_a_newer_layout_is_refused():
bundle = encode_bundle([])
bundle.bundle_version = 99
with pytest.raises(ProtocolError, match="not supported"):
decode_bundle(bundle)
def test_bundle_with_unspecified_version_is_refused():
with pytest.raises(ProtocolError, match="version 0"):
decode_bundle(pb.TensorBundle())
def test_tensor_with_unspecified_compression_is_refused():
tensor = encode_tensor(HIDDEN_STATES, b"\xaa" * 32, [1, 16, 1], pb.DTYPE_BFLOAT16)
tensor.compression = pb.COMPRESSION_UNSPECIFIED
with pytest.raises(ProtocolError, match="unspecified"):
decode_tensor(tensor)
def test_tensor_with_unspecified_checksum_is_refused():
tensor = encode_tensor(HIDDEN_STATES, b"\xaa" * 32, [1, 16, 1], pb.DTYPE_BFLOAT16)
tensor.checksum.algorithm = pb.CHECKSUM_ALGORITHM_UNSPECIFIED
with pytest.raises(ProtocolError, match="unspecified"):
decode_tensor(tensor)
def test_declared_tensor_size_cannot_exceed_negotiated_chunk_bound():
tensor = pb.NamedTensor(
name=HIDDEN_STATES,
shape=[1],
dtype=pb.DTYPE_UINT8,
byte_order=pb.BYTE_ORDER_LITTLE_ENDIAN,
total_bytes=DEFAULT_MAX_CHUNK_BYTES + 1,
compression=pb.COMPRESSION_NONE,
)
with pytest.raises(ProtocolError, match="negotiated chunk bound"):
decode_tensor(tensor)
def test_stricter_session_negotiation_overrides_global_defaults():
tensor = encode_tensor(
HIDDEN_STATES,
b"\xaa" * 32,
[32],
pb.DTYPE_UINT8,
max_fragment_bytes=32,
)
with pytest.raises(ProtocolError, match="16-byte negotiated"):
decode_tensor(tensor, max_chunk_bytes=16, max_fragment_bytes=16)
with pytest.raises(ProtocolError, match="16-byte"):
decode_bundle(
encode_bundle([tensor]),
max_chunk_bytes=16,
max_fragment_bytes=16,
)
def test_fragment_cannot_exceed_protocol_fragment_bound():
tensor = encode_tensor(
HIDDEN_STATES,
b"\xaa" * (1024 * 1024 + 1),
[1024 * 1024 + 1],
pb.DTYPE_UINT8,
max_fragment_bytes=1024 * 1024 + 1,
)
with pytest.raises(PayloadCorrupt, match="fragment larger"):
decode_tensor(tensor)
def test_fragment_count_is_bounded_before_sorting_or_allocation():
tensor = encode_tensor(
HIDDEN_STATES,
b"\xaa" * (DEFAULT_MAX_FRAGMENTS_PER_TENSOR + 1),
[DEFAULT_MAX_FRAGMENTS_PER_TENSOR + 1],
pb.DTYPE_UINT8,
max_fragment_bytes=1,
max_fragments=DEFAULT_MAX_FRAGMENTS_PER_TENSOR + 1,
)
with pytest.raises(PayloadCorrupt, match="fragments.*exceeding"):
decode_tensor(tensor)
def test_tensor_count_is_bounded_before_bundle_decoding():
tensors = [
pb.NamedTensor(name=f"t-{index}")
for index in range(DEFAULT_MAX_TENSORS_PER_BUNDLE + 1)
]
with pytest.raises(ProtocolError, match="tensors.*exceeding"):
encode_bundle(tensors)
@pytest.mark.parametrize(
("shape", "match"),
[
([1] * 9, "rank"),
([-1], "dimension outside"),
([1 << 31], "dimension outside"),
],
)
def test_shape_rank_and_dimensions_are_bounded_before_payload_work(shape, match):
with pytest.raises(ProtocolError, match=match):
encode_tensor(HIDDEN_STATES, b"\x00", shape, pb.DTYPE_UINT8)
def test_complete_session_message_includes_envelope_in_byte_bound():
small = pb.SessionRequest(chunk=pb.ActivationChunk(bundle=encode_bundle([])))
assert validate_session_message_size(small, max_chunk_bytes=64) == small.ByteSize()
oversized = pb.SessionRequest(
chunk=pb.ActivationChunk(
envelope=pb.Envelope(work_id="w" * 128),
bundle=encode_bundle([]),
)
)
with pytest.raises(ProtocolError, match="serialized session message"):
validate_session_message_size(oversized, max_chunk_bytes=64)
def test_bundle_serialized_size_is_bounded():
payload = b"\xaa" * (DEFAULT_MAX_CHUNK_BYTES // 2)
first = encode_tensor("first", payload, [len(payload)], pb.DTYPE_UINT8)
second = encode_tensor("second", payload, [len(payload)], pb.DTYPE_UINT8)
with pytest.raises(ProtocolError, match="serialized tensor bundle"):
encode_bundle([first, second])
def test_compressed_tensor_cannot_expand_past_declared_size():
pytest.importorskip("zstandard")
payload = b"\x00" * 65536
always = CompressionPolicy(min_input_bytes=0, min_savings_bytes=0, min_savings_ratio=0.0)
tensor = encode_tensor(
HIDDEN_STATES, payload, [len(payload)], pb.DTYPE_UINT8, policy=always
)
assert tensor.compression == pb.COMPRESSION_ZSTD
# Simulate a hostile frame that advertises a small output while expanding to
# the original 64 KiB. Shape and total_bytes agree, so only the bounded
# decompressor protects the receiver.
tensor.shape[:] = [1024]
tensor.total_bytes = 1024
with pytest.raises(PayloadCorrupt, match="bounded zstd"):
decode_tensor(tensor)
def test_generated_runtime_floors_match_committed_stubs():
metadata = (REPO_ROOT / "packages/node/pyproject.toml").read_text()
assert '"grpcio>=1.82.1"' in metadata
assert '"protobuf>=7.35.0"' in metadata
# --- Bounded prefill chunking and the decode fast path ----------------------
def test_prefill_is_split_into_bounded_token_aligned_chunks():
chunks = plan_prefill_chunks(2048)
assert all(c.token_count <= DEFAULT_MAX_PREFILL_CHUNK_TOKENS for c in chunks)
assert sum(c.token_count for c in chunks) == 2048
# Contiguous, token-aligned, and no split falls mid-token.
assert [c.first_position for c in chunks] == [
i * DEFAULT_MAX_PREFILL_CHUNK_TOKENS for i in range(len(chunks))
]
assert [c.final_chunk for c in chunks] == [False] * (len(chunks) - 1) + [True]
def test_final_prefill_chunk_carries_the_remainder():
chunks = plan_prefill_chunks(300, max_tokens=128)
assert [c.token_count for c in chunks] == [128, 128, 44]
assert chunks[-1].chunk_info().final_chunk
assert chunks[0].position().first_position == 0
assert chunks[-1].position().first_position == 256
def test_empty_prefill_is_refused():
with pytest.raises(ProtocolError):
plan_prefill_chunks(0)
def test_decode_fast_path_is_much_smaller_than_a_full_envelope_chunk():
hidden = b"\x01\x02" * 8 # one token, hidden=8, bfloat16
tensor = encode_tensor(HIDDEN_STATES, hidden, [1, 1, 8], pb.DTYPE_BFLOAT16)
fast = pb.SessionRequest(
decode=pb.DecodeStep(
idempotency_step=9,
position=1024,
expected_past_len=1024,
tensor=tensor,
work_id="work-7f3a",
)
)
# The same single token carried the long way, repeating identity that the
# handshake already fixed for the life of the stream.
full = pb.SessionRequest(
chunk=pb.ActivationChunk(
envelope=conformance.canonical_session_request().chunk.envelope,
bundle=encode_bundle([tensor]),
)
)
assert len(fast.SerializeToString()) * 2 < len(full.SerializeToString())
assert decode_tensor(fast.decode.tensor) == hidden
def test_flow_control_defaults_bound_the_queue_and_the_message():
limits = default_flow_control()
assert limits.max_prefill_chunk_tokens == DEFAULT_MAX_PREFILL_CHUNK_TOKENS
assert limits.max_chunk_bytes == DEFAULT_MAX_CHUNK_BYTES
assert limits.max_inflight_chunks > 0
def test_flow_control_negotiation_takes_the_strictest_bound():
proposed = pb.FlowControl(
credits_granted=64, max_inflight_chunks=64, max_chunk_bytes=64 << 20,
max_prefill_chunk_tokens=1024,
)
settled = negotiate_flow_control(proposed, default_flow_control())
# A sender cannot talk a worker into unbounded queues by proposing a large
# window: neither peer can raise the other's ceiling.
assert settled.max_inflight_chunks == default_flow_control().max_inflight_chunks
assert settled.credits_granted <= settled.max_inflight_chunks
assert settled.max_chunk_bytes == DEFAULT_MAX_CHUNK_BYTES
assert settled.max_prefill_chunk_tokens == DEFAULT_MAX_PREFILL_CHUNK_TOKENS
def test_initial_credits_never_exceed_negotiated_inflight_limit():
proposed = pb.FlowControl(credits_granted=64, max_inflight_chunks=64)
limits = pb.FlowControl(credits_granted=64, max_inflight_chunks=8)
settled = negotiate_flow_control(proposed, limits)
assert settled.credits_granted == 8
assert settled.max_inflight_chunks == 8
# --- Compatibility ----------------------------------------------------------
def test_committed_vectors_still_encode_as_promised():
# The C++ test asserts against these exact bytes. If a schema change alters
# the canonical encoding, it must be acknowledged by regenerating them.
golden = (conformance.TESTDATA_DIR / conformance.GOLDEN_SESSION_REQUEST).read_bytes()
assert conformance.serialize(conformance.canonical_session_request()) == golden
report = (conformance.TESTDATA_DIR / conformance.GOLDEN_CAPABILITY_REPORT).read_bytes()
assert conformance.serialize(conformance.canonical_capability_report()) == report
def test_golden_session_request_round_trips_with_every_field_intact():
golden = (conformance.TESTDATA_DIR / conformance.GOLDEN_SESSION_REQUEST).read_bytes()
request = pb.SessionRequest.FromString(golden)
envelope = request.chunk.envelope
assert envelope.work_id == conformance.WORK_ID
assert envelope.route_session_id == conformance.ROUTE_SESSION_ID
assert envelope.route_epoch == conformance.ROUTE_EPOCH
assert envelope.idempotency_step == conformance.IDEMPOTENCY_STEP
assert envelope.shard_range.effective_start_layer == conformance.EFFECTIVE_START_LAYER
assert envelope.phase == pb.PHASE_PREFILL
assert envelope.cache_expectation.mode == pb.CACHE_MODE_PREFILL
assert envelope.deadline_unix_nanos == conformance.DEADLINE_UNIX_NANOS
assert decode_bundle(request.chunk.bundle) == {
HIDDEN_STATES: conformance.canonical_payload()
}
assert request.SerializeToString(deterministic=True) == golden
def test_unknown_fields_from_a_newer_peer_survive_a_forwarding_hop():
# A Shard forwards activations onward. If it silently dropped fields a newer
# peer added, it would corrupt a route it is merely a waypoint on.
golden = (conformance.TESTDATA_DIR / conformance.GOLDEN_SESSION_REQUEST).read_bytes()
future_field = b"\xb8\xe0\x04\xb9\x60" # field 9999, varint 12345
request = pb.SessionRequest.FromString(golden + future_field)
assert request.chunk.envelope.work_id == conformance.WORK_ID
assert request.SerializeToString() == golden + future_field
def test_a_message_missing_newer_field_groups_still_parses():
sparse = pb.SessionRequest(chunk=pb.ActivationChunk(envelope=pb.Envelope(work_id="w")))
parsed = pb.SessionRequest.FromString(sparse.SerializeToString())
assert parsed.chunk.envelope.work_id == "w"
assert parsed.chunk.envelope.route_epoch == 0
assert parsed.chunk.envelope.phase == pb.PHASE_UNSPECIFIED
def test_retired_fragment_field_stays_reserved():
# `uncompressed_size` (field 5) was removed because NamedTensor.total_bytes
# is the single source of truth. The number stays reserved so it can never
# be recycled for a different meaning — a recycled number is the one schema
# change that old and new peers cannot detect, because the bytes still parse.
descriptor = descriptor_pb2.DescriptorProto()
pb.TensorFragment.DESCRIPTOR.CopyToProto(descriptor)
assert 5 not in {field.number for field in descriptor.field}
assert any(r.start <= 5 < r.end for r in descriptor.reserved_range)
assert "uncompressed_size" in descriptor.reserved_name
def test_a_peer_still_sending_the_retired_field_does_not_corrupt_the_tensor():
# An older peer that still sets field 5 must be parsed, not rejected: the
# value lands in unknown fields and the payload is unaffected.
tensor = encode_tensor(HIDDEN_STATES, b"\xaa" * 32, [1, 16, 1], pb.DTYPE_BFLOAT16)
wire = tensor.SerializeToString()
fragment = pb.TensorFragment.FromString(tensor.fragments[0].SerializeToString() + b"\x28\x20")
assert fragment.payload == tensor.fragments[0].payload
assert decode_tensor(pb.NamedTensor.FromString(wire)) == b"\xaa" * 32
def test_generated_python_stubs_match_the_proto():
pytest.importorskip("grpc_tools", reason="protoc toolchain is required to verify")
result = subprocess.run(
[sys.executable, str(REPO_ROOT / "scripts/generate_native_protocol.py"), "--check"],
capture_output=True,
text=True,
)
assert result.returncode == 0, result.stdout + result.stderr
@pytest.mark.skipif(
not CPP_ROUNDTRIP.is_file(),
reason="build the C++ conformance test to check cross-language agreement",
)
def test_cpp_and_python_agree_byte_for_byte():
# Written by the C++ conformance test: it parsed the golden bytes into its
# own object model and serialized them back. Byte equality means both
# languages encode every field of this schema identically.
golden = (conformance.TESTDATA_DIR / conformance.GOLDEN_SESSION_REQUEST).read_bytes()
assert CPP_ROUNDTRIP.read_bytes() == golden

564
tests/test_recipe_benchmark.py Normal file
View File

@@ -0,0 +1,564 @@
"""The recipe benchmark's measurement core, driven by a scripted fake runtime.
These tests never load a model, touch a GPU, or open a socket: the core is
deliberately runtime-free so the arithmetic and the lane rules can be pinned
down exactly, and the real drivers only have to be honest about what they
report.
"""
from __future__ import annotations
import copy
import time
from pathlib import Path
import pytest
from meshnet_node.performance_contract import (
STOP_CONDITION,
ContractThresholds,
PerformanceContract,
PerformanceContractError,
_canonical_sha256,
evaluate_contract,
parse_contract,
)
from meshnet_node.recipe_drivers import (
_artifact_sha256,
_validate_config,
require_real_inference,
)
from meshnet_node.recipe_benchmark import (
BenchmarkError,
BenchmarkPlan,
GenerationSample,
Lane,
LoadStats,
PromptSpec,
RecipeSpec,
SamplingPolicy,
build_report,
compute_drift,
measure_recipe,
summarize_concurrency,
RequestOutcome,
)
PROMPTS = (
PromptSpec(id="short", text="Say hello.", context_class="short"),
PromptSpec(id="long", text="Summarize the following. " * 40, context_class="long"),
)
def plan(**overrides) -> BenchmarkPlan:
defaults = dict(
plan_id="test-plan",
model_id="test/model",
model_revision="revision-1",
prompts=PROMPTS,
sampling=SamplingPolicy(max_output_tokens=8),
concurrency_levels=(1, 4),
repeats=1,
warmup_requests=0,
)
defaults.update(overrides)
return BenchmarkPlan(**defaults)
class FakeDriver:
"""A runtime with fixed timings, so every metric below has one right answer."""
def __init__(
self,
*,
decode_ms_per_token: float = 10.0,
prefill_ms: float = 100.0,
artifact_bytes: int = 1_000_000,
rss_bytes: int = 4_000_000,
vram_bytes: int = 0,
texts: dict[str, str] | None = None,
fail_at_concurrency: int | None = None,
decode_tokens: int = 8,
generation_delay_s: float = 0.0,
) -> None:
self.decode_ms_per_token = decode_ms_per_token
self.prefill_ms = prefill_ms
self.artifact_bytes = artifact_bytes
self.rss_bytes = rss_bytes
self.vram_bytes = vram_bytes
self.texts = texts or {}
self.fail_at_concurrency = fail_at_concurrency
self.decode_tokens = decode_tokens
self.generation_delay_s = generation_delay_s
self.in_flight = 0
self.max_in_flight = 0
self.loads = 0
self.closes = 0
self.generations = 0
def load(self) -> LoadStats:
self.loads += 1
return LoadStats(
artifact_bytes=self.artifact_bytes, load_ms=50.0,
rss_bytes=self.rss_bytes, vram_bytes=self.vram_bytes,
)
def generate(self, prompt: str, sampling: SamplingPolicy) -> GenerationSample:
self.in_flight += 1
self.max_in_flight = max(self.max_in_flight, self.in_flight)
try:
if self.generation_delay_s:
time.sleep(self.generation_delay_s)
if self.fail_at_concurrency and self.in_flight >= self.fail_at_concurrency:
raise RuntimeError("slot exhausted")
self.generations += 1
decode_ms = self.decode_ms_per_token * self.decode_tokens
return GenerationSample(
text=self.texts.get(prompt, "hello world"),
prompt_tokens=10,
decode_tokens=self.decode_tokens,
ttft_ms=self.prefill_ms,
prefill_ms=self.prefill_ms,
decode_ms=decode_ms,
total_ms=self.prefill_ms + decode_ms,
)
finally:
self.in_flight -= 1
def memory_probe(self) -> tuple[int, int]:
return self.rss_bytes, self.vram_bytes
def close(self) -> None:
self.closes += 1
def recipe(recipe_id: str, lane: Lane, *, reference: bool = False, device: str = "cpu") -> RecipeSpec:
return RecipeSpec(
id=recipe_id, runtime="fake", weight_format="fake", weight_quantization="bf16",
lane=lane, device=device, is_reference=reference,
)
def test_plan_rejects_an_experiment_it_cannot_run():
with pytest.raises(BenchmarkError):
plan(prompts=())
with pytest.raises(BenchmarkError):
plan(concurrency_levels=(0,))
with pytest.raises(BenchmarkError):
plan(repeats=0)
def test_measure_runs_every_prompt_at_every_concurrency_level():
driver = FakeDriver()
measurement = measure_recipe(driver, recipe("r", Lane.QUALITY, reference=True), plan())
# 2 prompts x (1 + 4) requests-per-level.
assert len(measurement.outcomes) == 2 * 1 + 2 * 4
assert sorted(measurement.metrics) == [1, 4]
assert driver.loads == 1
assert driver.closes == 1
assert measurement.available
def test_concurrency_level_actually_overlaps_requests():
driver = FakeDriver(decode_ms_per_token=5.0, generation_delay_s=0.02)
measure_recipe(driver, recipe("r", Lane.QUALITY, reference=True), plan(concurrency_levels=(1, 4)))
assert driver.max_in_flight > 1, "concurrency 4 must run requests in parallel, not serially"
def test_driver_is_closed_even_when_every_request_fails():
driver = FakeDriver(fail_at_concurrency=1)
measurement = measure_recipe(driver, recipe("r", Lane.QUALITY, reference=True), plan())
assert driver.closes == 1
assert all(not outcome.ok for outcome in measurement.outcomes)
assert measurement.metrics[1].failures == 2
assert measurement.metrics[1].failure_reasons == ("RuntimeError: slot exhausted",)
def test_failed_requests_are_reported_not_raised():
driver = FakeDriver(fail_at_concurrency=4, generation_delay_s=0.02)
measurement = measure_recipe(driver, recipe("r", Lane.QUALITY, reference=True), plan())
assert measurement.metrics[1].failures == 0
assert measurement.metrics[4].failures > 0
assert measurement.metrics[4].requests == 8
def test_summary_arithmetic_is_exact():
outcomes = [
RequestOutcome(
recipe_id="r", concurrency=2, prompt_id="p", repeat=0, ok=True,
latency_ms=200.0, ttft_ms=100.0, prefill_ms=100.0, decode_ms=100.0,
prompt_tokens=10, decode_tokens=10,
),
RequestOutcome(
recipe_id="r", concurrency=2, prompt_id="p", repeat=1, ok=True,
latency_ms=400.0, ttft_ms=200.0, prefill_ms=200.0, decode_ms=200.0,
prompt_tokens=10, decode_tokens=10,
),
]
metrics = summarize_concurrency(
outcomes, concurrency=2, wall_ms=1000.0, peak_rss_bytes=7, peak_vram_bytes=9
)
assert metrics.latency_p50_ms == 200.0
assert metrics.latency_p95_ms == 400.0
# 10 tok / 0.1 s = 100 tok/s and 10 tok / 0.2 s = 50 tok/s, averaged.
assert metrics.decode_tokens_per_sec == 75.0
# 20 decoded tokens over a 1 s wall clock, regardless of per-request rates.
assert metrics.aggregate_decode_tokens_per_sec == 20.0
assert (metrics.peak_rss_bytes, metrics.peak_vram_bytes) == (7, 9)
def test_aggregate_throughput_credits_overlap_but_per_request_rate_does_not():
"""Two runtimes with identical per-request speed must be told apart by overlap."""
serial = summarize_concurrency(
[
RequestOutcome(recipe_id="s", concurrency=4, prompt_id="p", repeat=i, ok=True,
latency_ms=100.0, decode_ms=100.0, decode_tokens=10)
for i in range(4)
],
concurrency=4, wall_ms=400.0, peak_rss_bytes=0, peak_vram_bytes=0,
)
batched = summarize_concurrency(
[
RequestOutcome(recipe_id="b", concurrency=4, prompt_id="p", repeat=i, ok=True,
latency_ms=100.0, decode_ms=100.0, decode_tokens=10)
for i in range(4)
],
concurrency=4, wall_ms=100.0, peak_rss_bytes=0, peak_vram_bytes=0,
)
assert serial.decode_tokens_per_sec == batched.decode_tokens_per_sec == 100.0
assert serial.aggregate_decode_tokens_per_sec == 100.0
assert batched.aggregate_decode_tokens_per_sec == 400.0
def test_drift_against_the_reference_is_exact_for_an_identical_runtime():
texts = {prompt.text: f"answer for {prompt.id}" for prompt in PROMPTS}
reference = measure_recipe(
FakeDriver(texts=texts), recipe("ref", Lane.QUALITY, reference=True), plan()
)
twin = measure_recipe(FakeDriver(texts=texts), recipe("twin", Lane.QUALITY), plan())
drift = compute_drift(twin, reference)
assert drift.compared_prompts == 2
assert drift.exact_match_rate == 1.0
assert drift.mean_similarity == 1.0
assert drift.advisory is False
def test_quantized_drift_is_advisory_and_never_an_equivalence_claim():
reference = measure_recipe(
FakeDriver(texts={prompt.text: "the capital is Paris" for prompt in PROMPTS}),
recipe("ref", Lane.QUALITY, reference=True), plan(),
)
quantized = measure_recipe(
FakeDriver(texts={prompt.text: "the capital is Lyon" for prompt in PROMPTS}),
recipe("q4", Lane.PERFORMANCE_FIT), plan(),
)
drift = compute_drift(quantized, reference)
assert drift.advisory is True, "a quantized recipe's drift must be advisory"
assert drift.exact_match_rate == 0.0
assert 0.0 < drift.mean_similarity < 1.0
assert drift.per_prompt[0]["first_divergence_char"] > 0
def test_report_needs_exactly_one_quality_lane_reference():
measurement = measure_recipe(FakeDriver(), recipe("a", Lane.QUALITY, reference=True), plan())
second = measure_recipe(FakeDriver(), recipe("b", Lane.QUALITY, reference=True), plan())
quantized = measure_recipe(FakeDriver(), recipe("q", Lane.PERFORMANCE_FIT), plan())
with pytest.raises(BenchmarkError, match="exactly one reference"):
build_report(plan(), [measurement, second], host={}, evidence_class="synthetic")
with pytest.raises(BenchmarkError, match="exactly one reference"):
build_report(plan(), [quantized], host={}, evidence_class="synthetic")
def test_reference_recipe_may_not_be_quantized():
quantized_reference = measure_recipe(
FakeDriver(), recipe("q", Lane.PERFORMANCE_FIT, reference=True), plan()
)
with pytest.raises(BenchmarkError, match="quality lane"):
build_report(plan(), [quantized_reference], host={}, evidence_class="synthetic")
def test_report_must_declare_how_it_was_produced():
measurement = measure_recipe(FakeDriver(), recipe("a", Lane.QUALITY, reference=True), plan())
with pytest.raises(BenchmarkError, match="evidence class"):
build_report(plan(), [measurement], host={}, evidence_class="probably-real")
def test_report_carries_every_metric_the_contract_reads():
reference = measure_recipe(FakeDriver(), recipe("ref", Lane.QUALITY, reference=True), plan())
quantized = measure_recipe(
FakeDriver(decode_ms_per_token=4.0, artifact_bytes=400_000, rss_bytes=1_000_000),
recipe("q4", Lane.PERFORMANCE_FIT), plan(),
)
report = build_report(
plan(), [reference, quantized], host={"cpu": "test"}, evidence_class="synthetic"
)
assert report["schema_version"] == 1
assert report["reference_recipe_id"] == "ref"
entry = next(e for e in report["recipes"] if e["recipe"]["id"] == "q4")
cell = entry["concurrency"]["1"]
for metric in (
"ttft_p50_ms", "ttft_p95_ms", "latency_p50_ms", "latency_p95_ms",
"prefill_tokens_per_sec", "decode_tokens_per_sec", "aggregate_decode_tokens_per_sec",
"peak_rss_bytes", "peak_vram_bytes", "failures",
):
assert metric in cell, f"the contract reads {metric}, so the report must carry it"
assert entry["load"]["artifact_bytes"] == 400_000
assert [d["recipe_id"] for d in report["drift"]] == ["q4"]
def test_unavailable_recipes_are_recorded_rather_than_dropped():
from meshnet_node.recipe_benchmark import RecipeMeasurement
reference = measure_recipe(FakeDriver(), recipe("ref", Lane.QUALITY, reference=True), plan())
missing = RecipeMeasurement(
recipe=recipe("q4", Lane.PERFORMANCE_FIT),
load=LoadStats(artifact_bytes=0, load_ms=0.0),
unavailable_reason="BenchmarkError: GGUF artifact not found",
)
report = build_report(plan(), [reference, missing], host={}, evidence_class="synthetic")
entry = next(e for e in report["recipes"] if e["recipe"]["id"] == "q4")
assert entry["available"] is False
assert "not found" in entry["unavailable_reason"]
assert report["drift"] == [], "an unmeasured recipe has no drift to report"
def test_contract_requires_a_quality_lane_then_allows_quantized_fit_benefit():
texts = {prompt.text: "same greedy answer" for prompt in PROMPTS}
reference = measure_recipe(
FakeDriver(texts=texts, rss_bytes=4_000_000), recipe("safetensors", Lane.QUALITY, reference=True), plan()
)
quality = measure_recipe(
FakeDriver(texts=texts), recipe("gguf-f16", Lane.QUALITY), plan()
)
q4 = measure_recipe(
FakeDriver(texts={prompt.text: "different quantized answer" for prompt in PROMPTS},
rss_bytes=1_000_000, decode_ms_per_token=20.0),
recipe("gguf-q4", Lane.PERFORMANCE_FIT), plan()
)
report = build_report(plan(), [reference, quality, q4], host={}, evidence_class="synthetic")
contract = PerformanceContract(
contract_version=1, locked_at="2026-07-13T00:00:00Z", locked_by="test",
plan_id="test-plan", thresholds=ContractThresholds(), baseline={}, stop_condition="test",
)
evaluation = evaluate_contract(contract, report)
assert evaluation.quality_lane_pass is True
assert evaluation.fit_benefit is True
assert evaluation.verdict == "optimize"
def test_peak_memory_is_sampled_while_requests_are_in_flight():
class TransientMemoryDriver(FakeDriver):
def memory_probe(self) -> tuple[int, int]:
return (99_000_000 if self.in_flight else 1_000_000), 0
measurement = measure_recipe(
TransientMemoryDriver(generation_delay_s=0.05),
recipe("transient", Lane.QUALITY, reference=True),
plan(warmup_requests=0),
)
assert measurement.metrics[1].peak_rss_bytes == 99_000_000
assert measurement.metrics[4].peak_rss_bytes == 99_000_000
def test_real_inference_requires_explicit_opt_in(monkeypatch):
monkeypatch.delenv("MESHNET_ENABLE_REAL_INFERENCE_TESTS", raising=False)
with pytest.raises(BenchmarkError, match="opt-in"):
require_real_inference()
def test_config_rejects_an_artifact_digest_mismatch(tmp_path: Path):
artifact = tmp_path / "model.gguf"
artifact.write_bytes(b"real model bytes")
config = {
"artifact_storage_root": str(tmp_path),
"plan": {
"model_id": "test/model",
"model_revision": "revision-1",
"prompts": [
{"id": "p1", "text": "one"},
{"id": "p2", "text": "two"},
{"id": "p3", "text": "three"},
],
"concurrency_levels": [1, 4],
"repeats": 3,
"warmup_requests": 1,
"sampling": {
"temperature": 0.0,
"top_k": 1,
"top_p": 1.0,
"max_output_tokens": 32,
},
},
"recipes": [{
"id": "recipe",
"source_model_id": "test/model",
"source_model_revision": "revision-1",
"artifact_path": str(artifact),
"artifact_sha256": _artifact_sha256(artifact),
"device": "cpu",
"driver": {
"type": "llama-cpp-server",
"binary": str(artifact),
"binary_sha256": _artifact_sha256(artifact),
"gguf_path": str(artifact),
"device": "cpu",
"threads": 8,
"n_parallel": 4,
},
}],
}
_validate_config(config)
config["recipes"][0]["artifact_sha256"] = "0" * 64
with pytest.raises(BenchmarkError, match="digest mismatch"):
_validate_config(config)
config["recipes"][0]["artifact_sha256"] = _artifact_sha256(artifact)
config["recipes"][0]["driver"]["binary_sha256"] = "0" * 64
with pytest.raises(BenchmarkError, match="binary SHA-256 mismatch"):
_validate_config(config)
config["recipes"][0]["driver"]["binary_sha256"] = _artifact_sha256(artifact)
config["recipes"][0]["driver"]["n_gpu_layers"] = 1
with pytest.raises(BenchmarkError, match="CPU-only"):
_validate_config(config)
def _lock_real_report(report: dict) -> PerformanceContract:
report["evidence_class"] = "local-real"
report["host"] = {
"hostname": "test-host",
"platform": "test-platform",
"python": "3.12",
"cpu_count": 8,
}
for entry in report["recipes"]:
entry["recipe"]["source_model_id"] = report["plan"]["model_id"]
entry["recipe"]["source_model_revision"] = report["plan"]["model_revision"]
entry["recipe"]["artifact_sha256"] = "a" * 64
return PerformanceContract(
contract_version=1,
locked_at="2026-07-13T00:00:00Z",
locked_by="test",
plan_id=report["plan"]["plan_id"],
thresholds=ContractThresholds(),
baseline={
"required_evidence_class": "local-real",
"required_recipes": [entry["recipe"]["id"] for entry in report["recipes"]],
"required_concurrency_levels": [1, 4],
},
stop_condition="test",
)
def test_locked_contract_rejects_synthetic_evidence():
reference = measure_recipe(
FakeDriver(), recipe("ref", Lane.QUALITY, reference=True), plan()
)
quality = measure_recipe(FakeDriver(), recipe("quality", Lane.QUALITY), plan())
q4 = measure_recipe(FakeDriver(), recipe("q4", Lane.PERFORMANCE_FIT), plan())
report = build_report(
plan(), [reference, quality, q4], host={}, evidence_class="synthetic"
)
contract = _lock_real_report(report)
report["evidence_class"] = "synthetic"
with pytest.raises(PerformanceContractError, match="evidence class"):
evaluate_contract(contract, report)
def test_quality_lane_requires_every_prompt_to_be_compared():
texts = {prompt.text: "same greedy answer" for prompt in PROMPTS}
reference = measure_recipe(
FakeDriver(texts=texts), recipe("ref", Lane.QUALITY, reference=True), plan()
)
quality = measure_recipe(
FakeDriver(texts=texts), recipe("quality", Lane.QUALITY), plan()
)
q4 = measure_recipe(
FakeDriver(texts=texts, rss_bytes=1_000_000),
recipe("q4", Lane.PERFORMANCE_FIT),
plan(),
)
report = build_report(
plan(), [reference, quality, q4], host={}, evidence_class="synthetic"
)
contract = _lock_real_report(report)
incomplete = copy.deepcopy(report)
quality_drift = next(
drift for drift in incomplete["drift"] if drift["recipe_id"] == "quality"
)
quality_drift["compared_prompts"] = 1
quality_drift["per_prompt"] = quality_drift["per_prompt"][:1]
evaluation = evaluate_contract(contract, incomplete)
assert evaluation.quality_lane_pass is False
assert evaluation.verdict == "stop"
def test_locked_contract_rejects_changed_plan_and_token_counts():
texts = {prompt.text: "same greedy answer" for prompt in PROMPTS}
reference = measure_recipe(
FakeDriver(texts=texts), recipe("ref", Lane.QUALITY, reference=True), plan()
)
quality = measure_recipe(
FakeDriver(texts=texts), recipe("quality", Lane.QUALITY), plan()
)
q4 = measure_recipe(
FakeDriver(texts=texts), recipe("q4", Lane.PERFORMANCE_FIT), plan()
)
report = build_report(
plan(), [reference, quality, q4], host={}, evidence_class="synthetic"
)
contract = _lock_real_report(report)
contract.baseline["required_plan_sha256"] = _canonical_sha256(report["plan"])
changed_plan = copy.deepcopy(report)
changed_plan["plan"]["prompts"][0]["text"] = "changed after locking"
with pytest.raises(PerformanceContractError, match="plan digest"):
evaluate_contract(contract, changed_plan)
changed_tokens = copy.deepcopy(report)
quality_entry = next(
entry for entry in changed_tokens["recipes"] if entry["recipe"]["id"] == "quality"
)
quality_entry["outcomes"][0]["decode_tokens"] += 1
with pytest.raises(PerformanceContractError, match="different prompt/decode token counts"):
evaluate_contract(contract, changed_tokens)
def test_v1_contract_thresholds_and_stop_condition_are_immutable():
raw = PerformanceContract(
contract_version=1,
locked_at="2026-07-13T00:00:00Z",
locked_by="test",
plan_id="test-plan",
thresholds=ContractThresholds(),
baseline={},
stop_condition=STOP_CONDITION,
).to_dict()
parse_contract(raw)
changed_threshold = copy.deepcopy(raw)
changed_threshold["thresholds"]["min_decode_speedup"] = 1.01
with pytest.raises(PerformanceContractError, match="immutable v1 thresholds"):
parse_contract(changed_threshold)
changed_stop = copy.deepcopy(raw)
changed_stop["stop_condition"] = "promote everything"
with pytest.raises(PerformanceContractError, match="stop condition differs"):
parse_contract(changed_stop)