100 lines
4.5 KiB
Markdown
100 lines
4.5 KiB
Markdown
# DGR-001 — performance contract baseline
|
|
|
|
## Files changed
|
|
|
|
- `packages/node/meshnet_node/performance_contract.py`
|
|
- `tests/test_performance_contract.py`
|
|
- `.scratch/distributed-gguf-runtime/issues/01-lock-the-safetensors-versus-gguf-performance-contract.md`
|
|
- `.scratch/distributed-gguf-runtime/evidence/DGR-001/performance-contract.json`
|
|
|
|
## What this slice does
|
|
|
|
- Locks the DGR-001 benchmark contract in code.
|
|
- Pins the architecture-aligned baseline to **DeepSeek-V2-Lite-Chat** (`deepseek2`).
|
|
- Uses the same model on both sides of the comparison:
|
|
- **safetensors:** `deepseek-ai/DeepSeek-V2-Lite-Chat` in **BF16**
|
|
- **GGUF:** `second-state/DeepSeek-V2-Lite-Chat-GGUF` in **Q2_K**
|
|
- Exposes a machine-readable JSON contract with:
|
|
- benchmark lanes for `transformers` safetensors and `llama.cpp` GGUF on **CPU** and **GPU**
|
|
- concurrency levels `1` and `4`
|
|
- the required metrics list
|
|
- an explicit stop condition for “no meaningful speed or fit benefit”
|
|
- Adds a deterministic stub benchmark report so the contract now has an executable report shape end to end.
|
|
|
|
## Recent benchmark runner slice
|
|
|
|
The runner currently uses a deterministic stub backend to exercise the comparison matrix without downloading a model. It emits:
|
|
|
|
- `.scratch/distributed-gguf-runtime/evidence/DGR-001/performance-contract.json`
|
|
- `.scratch/distributed-gguf-runtime/evidence/DGR-001/stub-benchmark-report.json`
|
|
|
|
The report includes per-device comparisons for:
|
|
|
|
- `transformers-safetensors-cpu` vs `llama-cpp-gguf-cpu`
|
|
- `transformers-safetensors-gpu` vs `llama-cpp-gguf-gpu`
|
|
|
|
and records the memory metric (`rss_bytes` on CPU, `vram_bytes` on GPU), decode speedup, artifact ratio, and output drift.
|
|
|
|
## Live endpoint CLI wiring
|
|
|
|
The contract CLI can now drive the live endpoint runner. Passing one `--live-endpoint LANE_ID=URL` mapping per contract lane (plus `--live-benchmark-out`) invokes `run_real_model_endpoint_benchmark` against already-running OpenAI-compatible servers and writes the report using the same schema as the stub:
|
|
|
|
```bash
|
|
PYTHONPATH=packages/node python -m meshnet_node.performance_contract \
|
|
--live-endpoint transformers-safetensors-cpu=http://127.0.0.1:8001 \
|
|
--live-endpoint llama-cpp-gguf-cpu=http://127.0.0.1:8002 \
|
|
--live-endpoint transformers-safetensors-gpu=http://127.0.0.1:8003 \
|
|
--live-endpoint llama-cpp-gguf-gpu=http://127.0.0.1:8004 \
|
|
--live-benchmark-out .scratch/distributed-gguf-runtime/evidence/DGR-001/live-benchmark-report.json
|
|
```
|
|
|
|
`--live-model` overrides the model name sent in requests (defaults to the contract's safetensors repo). Without any `--live-endpoint` flags the CLI behaves exactly as before: it writes the contract JSON and, with `--benchmark-out`, the deterministic stub report.
|
|
|
|
## Exact commands and real results
|
|
|
|
### Targeted tests
|
|
|
|
```bash
|
|
PYTHONPATH=packages/node pytest -q tests/test_performance_contract.py tests/test_route_session_benchmark.py
|
|
```
|
|
|
|
Result: `19 passed in 0.11s`
|
|
|
|
### Contract artifact generation
|
|
|
|
```bash
|
|
PYTHONPATH=packages/node python -m meshnet_node.performance_contract --json-out .scratch/distributed-gguf-runtime/evidence/DGR-001/performance-contract.json
|
|
```
|
|
|
|
Result: wrote `.scratch/distributed-gguf-runtime/evidence/DGR-001/performance-contract.json`
|
|
|
|
### Python compile check
|
|
|
|
```bash
|
|
python -m compileall packages/node/meshnet_node/performance_contract.py tests/test_performance_contract.py
|
|
```
|
|
|
|
Result: passed
|
|
|
|
## Limitations
|
|
|
|
- This slice still uses a deterministic stub backend for the core comparison matrix.
|
|
- It now also includes a live endpoint runner, reachable from the CLI via `--live-endpoint`/`--live-benchmark-out`, that fans out one OpenAI-compatible request per lane when the caller provides endpoints; the CLI does not start those servers.
|
|
- It does **not** download or run a real model from within the repo.
|
|
- Real safetensors vs GGUF execution, TTFT/prefill/decode measurements, RSS/VRAM capture, and output-drift comparison are still to be implemented against the contract.
|
|
|
|
## Compatibility notes
|
|
|
|
- The contract stays on the DeepSeek2 family to remain close to the DeepSeek-V4-Flash end goal.
|
|
- A smaller non-DeepSeek model can still be used later for loader-plumbing smoke tests, but it does not replace this baseline.
|
|
- Model artifacts must stay on the mounted drive and not under `/home`.
|
|
|
|
## Dependent-story handoff
|
|
|
|
Next implementation work should attach to this contract and add the live benchmark runner that actually compares:
|
|
|
|
1. current Transformers/safetensors recipe
|
|
2. whole-model llama.cpp GGUF recipe
|
|
|
|
using the same model architecture/revision and the same prompt/context/concurrency settings.
|