feat: add DGR-001 performance contract
This commit is contained in:
66
.scratch/distributed-gguf-runtime/evidence/DGR-001/README.md
Normal file
66
.scratch/distributed-gguf-runtime/evidence/DGR-001/README.md
Normal file
@@ -0,0 +1,66 @@
|
||||
# DGR-001 — performance contract baseline
|
||||
|
||||
## Files changed
|
||||
|
||||
- `packages/node/meshnet_node/performance_contract.py`
|
||||
- `tests/test_performance_contract.py`
|
||||
- `.scratch/distributed-gguf-runtime/issues/01-lock-the-safetensors-versus-gguf-performance-contract.md`
|
||||
- `.scratch/distributed-gguf-runtime/evidence/DGR-001/performance-contract.json`
|
||||
|
||||
## What this slice does
|
||||
|
||||
- Locks the DGR-001 benchmark contract in code.
|
||||
- Pins the architecture-aligned baseline to **DeepSeek-V2-Lite-Chat** (`deepseek2`).
|
||||
- Uses the smallest DeepSeek-family GGUF target selected for this story: **Q2_K** via `second-state/DeepSeek-V2-Lite-Chat-GGUF`.
|
||||
- Exposes a machine-readable JSON contract with:
|
||||
- benchmark lanes for `transformers` safetensors and `llama.cpp` GGUF
|
||||
- concurrency levels `1` and `4`
|
||||
- the required metrics list
|
||||
- an explicit stop condition for “no meaningful speed or fit benefit”
|
||||
|
||||
## Exact commands and real results
|
||||
|
||||
### Targeted tests
|
||||
|
||||
```bash
|
||||
pytest -q tests/test_performance_contract.py tests/test_route_session_benchmark.py
|
||||
```
|
||||
|
||||
Result: `9 passed in 0.14s`
|
||||
|
||||
### Contract artifact generation
|
||||
|
||||
```bash
|
||||
PYTHONPATH=packages/node python -m meshnet_node.performance_contract --json-out .scratch/distributed-gguf-runtime/evidence/DGR-001/performance-contract.json
|
||||
```
|
||||
|
||||
Result: wrote `.scratch/distributed-gguf-runtime/evidence/DGR-001/performance-contract.json`
|
||||
|
||||
### Python compile check
|
||||
|
||||
```bash
|
||||
python -m compileall packages/node/meshnet_node/performance_contract.py tests/test_performance_contract.py
|
||||
```
|
||||
|
||||
Result: passed
|
||||
|
||||
## Limitations
|
||||
|
||||
- This slice captures the DGR-001 contract and baseline selection only.
|
||||
- It does **not** download or run a real model yet.
|
||||
- Real safetensors vs GGUF execution, TTFT/prefill/decode measurements, RSS/VRAM capture, and output-drift comparison are still to be implemented against the contract.
|
||||
|
||||
## Compatibility notes
|
||||
|
||||
- The contract stays on the DeepSeek2 family to remain close to the DeepSeek-V4-Flash end goal.
|
||||
- A smaller non-DeepSeek model can still be used later for loader-plumbing smoke tests, but it does not replace this baseline.
|
||||
- Model artifacts must stay on the mounted drive and not under `/home`.
|
||||
|
||||
## Dependent-story handoff
|
||||
|
||||
Next implementation work should attach to this contract and add the live benchmark runner that actually compares:
|
||||
|
||||
1. current Transformers/safetensors recipe
|
||||
2. whole-model llama.cpp GGUF recipe
|
||||
|
||||
using the same model architecture/revision and the same prompt/context/concurrency settings.
|
||||
@@ -0,0 +1,51 @@
|
||||
{
|
||||
"benchmark_lanes": [
|
||||
{
|
||||
"concurrency_levels": [
|
||||
1,
|
||||
4
|
||||
],
|
||||
"id": "transformers-safetensors",
|
||||
"recipe": "current safetensors recipe",
|
||||
"runtime": "transformers"
|
||||
},
|
||||
{
|
||||
"concurrency_levels": [
|
||||
1,
|
||||
4
|
||||
],
|
||||
"id": "llama-cpp-gguf",
|
||||
"recipe": "whole-model GGUF recipe",
|
||||
"runtime": "llama.cpp"
|
||||
}
|
||||
],
|
||||
"metrics": [
|
||||
"ttft_ms",
|
||||
"prefill_tok_per_sec",
|
||||
"decode_tok_per_sec",
|
||||
"p50_latency_ms",
|
||||
"p95_latency_ms",
|
||||
"aggregate_throughput_tok_per_sec",
|
||||
"rss_bytes",
|
||||
"vram_bytes",
|
||||
"artifact_bytes",
|
||||
"failure_count",
|
||||
"output_drift"
|
||||
],
|
||||
"model_target": {
|
||||
"architecture": "deepseek2",
|
||||
"gguf_quant": "Q2_K",
|
||||
"gguf_repo": "second-state/DeepSeek-V2-Lite-Chat-GGUF",
|
||||
"gguf_size_gb": 6.43,
|
||||
"name": "DeepSeek-V2-Lite-Chat",
|
||||
"rationale": "Smallest DeepSeek-family benchmark anchor that still points toward DeepSeek-V4-Flash; keeps the runtime on the DeepSeek2 path instead of falling back to a tiny but architecture-mismatched smoke model.",
|
||||
"safetensors_repo": "deepseek-ai/DeepSeek-V2-Lite-Chat"
|
||||
},
|
||||
"notes": [
|
||||
"Real model execution stays opt-in and must keep model artifacts on the mounted drive.",
|
||||
"Use the tiny fallback only for loader plumbing smoke tests; it does not replace the architecture-aligned baseline."
|
||||
],
|
||||
"schema_version": 1,
|
||||
"stop_condition": "Stop if GGUF does not provide a meaningful speed or fit benefit over the safetensors baseline for the chosen DeepSeek-family model target.",
|
||||
"story_id": "DGR-001"
|
||||
}
|
||||
Reference in New Issue
Block a user