neuron-tai/.scratch/distributed-gguf-runtime/evidence/DGR-001

DGR-001 — performance contract baseline

Files changed

• packages/node/meshnet_node/performance_contract.py
• tests/test_performance_contract.py
• .scratch/distributed-gguf-runtime/issues/01-lock-the-safetensors-versus-gguf-performance-contract.md
• .scratch/distributed-gguf-runtime/evidence/DGR-001/performance-contract.json

What this slice does

• Locks the DGR-001 benchmark contract in code.
• Pins the architecture-aligned baseline to DeepSeek-V2-Lite-Chat (deepseek2).
• Uses the same model on both sides of the comparison:
  • safetensors: deepseek-ai/DeepSeek-V2-Lite-Chat in BF16
  • GGUF: second-state/DeepSeek-V2-Lite-Chat-GGUF in Q2_K
• Exposes a machine-readable JSON contract with:
  • benchmark lanes for transformers safetensors and llama.cpp GGUF on CPU and GPU
  • concurrency levels 1 and 4
  • the required metrics list
  • an explicit stop condition for “no meaningful speed or fit benefit”
• Adds a deterministic stub benchmark report so the contract now has an executable report shape end to end.

Recent benchmark runner slice

The runner currently uses a deterministic stub backend to exercise the comparison matrix without downloading a model. It emits:

• .scratch/distributed-gguf-runtime/evidence/DGR-001/performance-contract.json
• .scratch/distributed-gguf-runtime/evidence/DGR-001/stub-benchmark-report.json

The report includes per-device comparisons for:

• transformers-safetensors-cpu vs llama-cpp-gguf-cpu
• transformers-safetensors-gpu vs llama-cpp-gguf-gpu

and records the memory metric (rss_bytes on CPU, vram_bytes on GPU), decode speedup, artifact ratio, and output drift.

Live endpoint CLI wiring

The contract CLI can now drive the live endpoint runner. Passing one --live-endpoint LANE_ID=URL mapping per contract lane (plus --live-benchmark-out) invokes run_real_model_endpoint_benchmark against already-running OpenAI-compatible servers and writes the report using the same schema as the stub:

PYTHONPATH=packages/node python -m meshnet_node.performance_contract \
  --live-endpoint transformers-safetensors-cpu=http://127.0.0.1:8001 \
  --live-endpoint llama-cpp-gguf-cpu=http://127.0.0.1:8002 \
  --live-endpoint transformers-safetensors-gpu=http://127.0.0.1:8003 \
  --live-endpoint llama-cpp-gguf-gpu=http://127.0.0.1:8004 \
  --live-benchmark-out .scratch/distributed-gguf-runtime/evidence/DGR-001/live-benchmark-report.json

--live-model overrides the model name sent in requests (defaults to the contract's safetensors repo). Without any --live-endpoint flags the CLI behaves exactly as before: it writes the contract JSON and, with --benchmark-out, the deterministic stub report.

Exact commands and real results

Targeted tests

PYTHONPATH=packages/node pytest -q tests/test_performance_contract.py tests/test_route_session_benchmark.py

Result: 19 passed in 0.11s

Contract artifact generation

PYTHONPATH=packages/node python -m meshnet_node.performance_contract --json-out .scratch/distributed-gguf-runtime/evidence/DGR-001/performance-contract.json

Result: wrote .scratch/distributed-gguf-runtime/evidence/DGR-001/performance-contract.json

Python compile check

python -m compileall packages/node/meshnet_node/performance_contract.py tests/test_performance_contract.py

Result: passed

Public relay smoke benchmark (2026-07-15)

A real streamed request was run through the public tracker — not by connecting directly to the private node address:

https://meshnet.2.d-popov.com/v1/chat/completions
  -> wss://meshnet.2.d-popov.com/ws
  -> wss://meshnet.2.d-popov.com/rpc/7j77FsPY1evV8tuf-7000
  -> local CUDA node, Qwen/Qwen2.5-0.5B-Instruct layers 0-23

The local public-tracker node had an expired proof and a wedged HTTP server. A graceful restart refreshed its CUDA capability proof in 336 ms, restored admitted/routable status, and reconnected its relay endpoint.

Measured streaming results after recovery:

metric	result
warm-up TTFT	420.80 ms
warm-up elapsed	610.23 ms
p50 TTFT (3 runs)	288.26 ms
p50 elapsed (3 runs)	363.20 ms
tracker-recorded relay throughput	58.18-65.25 tok/s
HTTP status	200 for all runs

The tracker recorded relay: true and the local node ID 7j77FsPY-b32476219492 for each completion. Full redacted evidence is in public-relay-smoke-benchmark.json.

The other connected node is still alive but not routable because its capability proof is stale. It must revalidate before a multi-node shard/relay test can run.

Limitations

• This slice still uses a deterministic stub backend for the core comparison matrix.
• It now also includes a live endpoint runner, reachable from the CLI via --live-endpoint/--live-benchmark-out, that fans out one OpenAI-compatible request per lane when the caller provides endpoints; the CLI does not start those servers.
• It does not download or run a real model from within the repo.
• Real safetensors vs GGUF execution, TTFT/prefill/decode measurements, RSS/VRAM capture, and output-drift comparison are still to be implemented against the contract.

Compatibility notes

• The contract stays on the DeepSeek2 family to remain close to the DeepSeek-V4-Flash end goal.
• A smaller non-DeepSeek model can still be used later for loader-plumbing smoke tests, but it does not replace this baseline.
• Model artifacts must stay on the mounted drive and not under /home.

Dependent-story handoff

Next implementation work should attach to this contract and add the live benchmark runner that actually compares:

1. current Transformers/safetensors recipe
2. whole-model llama.cpp GGUF recipe

using the same model architecture/revision and the same prompt/context/concurrency settings.