# DGR-001 — performance contract baseline

## Files changed

- `packages/node/meshnet_node/performance_contract.py`
- `tests/test_performance_contract.py`
- `.scratch/distributed-gguf-runtime/issues/01-lock-the-safetensors-versus-gguf-performance-contract.md`
- `.scratch/distributed-gguf-runtime/evidence/DGR-001/performance-contract.json`

## What this slice does

- Locks the DGR-001 benchmark contract in code.
- Pins the architecture-aligned baseline to **DeepSeek-V2-Lite-Chat** (`deepseek2`).
- Uses the same model on both sides of the comparison:
  - **safetensors:** `deepseek-ai/DeepSeek-V2-Lite-Chat` in **BF16**
  - **GGUF:** `second-state/DeepSeek-V2-Lite-Chat-GGUF` in **Q2_K**
- Exposes a machine-readable JSON contract with:
  - benchmark lanes for `transformers` safetensors and `llama.cpp` GGUF on **CPU** and **GPU**
  - concurrency levels `1` and `4`
  - the required metrics list
  - an explicit stop condition for “no meaningful speed or fit benefit”
- Adds a deterministic stub benchmark report so the contract now has an executable report shape end to end.

## Recent benchmark runner slice

The runner currently uses a deterministic stub backend to exercise the comparison matrix without downloading a model. It emits:

- `.scratch/distributed-gguf-runtime/evidence/DGR-001/performance-contract.json`
- `.scratch/distributed-gguf-runtime/evidence/DGR-001/stub-benchmark-report.json`

The report includes per-device comparisons for:

- `transformers-safetensors-cpu` vs `llama-cpp-gguf-cpu`
- `transformers-safetensors-gpu` vs `llama-cpp-gguf-gpu`

and records the memory metric (`rss_bytes` on CPU, `vram_bytes` on GPU), decode speedup, artifact ratio, and output drift.

## Exact commands and real results

### Targeted tests

```bash
pytest -q tests/test_performance_contract.py tests/test_route_session_benchmark.py
```

Result: `9 passed in 0.14s`

### Contract artifact generation

```bash
PYTHONPATH=packages/node python -m meshnet_node.performance_contract --json-out .scratch/distributed-gguf-runtime/evidence/DGR-001/performance-contract.json
```

Result: wrote `.scratch/distributed-gguf-runtime/evidence/DGR-001/performance-contract.json`

### Python compile check

```bash
python -m compileall packages/node/meshnet_node/performance_contract.py tests/test_performance_contract.py
```

Result: passed

## Limitations

- This slice captures the DGR-001 contract and baseline selection only.
- It does **not** download or run a real model yet.
- Real safetensors vs GGUF execution, TTFT/prefill/decode measurements, RSS/VRAM capture, and output-drift comparison are still to be implemented against the contract.

## Compatibility notes

- The contract stays on the DeepSeek2 family to remain close to the DeepSeek-V4-Flash end goal.
- A smaller non-DeepSeek model can still be used later for loader-plumbing smoke tests, but it does not replace this baseline.
- Model artifacts must stay on the mounted drive and not under `/home`.

## Dependent-story handoff

Next implementation work should attach to this contract and add the live benchmark runner that actually compares:

1. current Transformers/safetensors recipe
2. whole-model llama.cpp GGUF recipe

using the same model architecture/revision and the same prompt/context/concurrency settings.