3.8 KiB
3.8 KiB
01 — Lock the safetensors-versus-GGUF performance contract
Status: ready-for-agent
Mandatory fresh-session context
- Read RALPH-CONTEXT.md completely before changing code.
- This issue is
DGR-001in prd.json. - Read the evidence README for every dependency listed below.
- Inspect current code and
git status; historical text and previous agent claims are not evidence.
Description
As a runtime engineer, I need a controlled baseline so that GGUF work proceeds from measured speed, memory, and quality rather than reputation.
Baseline model target
Use the same model on both sides of the comparison, with the closest practical low-footprint precision pair:
- safetensors:
deepseek-ai/DeepSeek-V2-Lite-Chatin BF16 - GGUF:
second-state/DeepSeek-V2-Lite-Chat-GGUFin Q2_K (~6.5GB)
Keep the benchmark matrix explicit for CPU and GPU runs. Reserve smaller non-DeepSeek fallback models only for loader plumbing smoke tests if needed; they do not count as the DGR-001 architecture-aligned baseline.
Expected durable outputs
- Benchmark harness and deterministic tests
- evidence/DGR-001/performance-contract.json
- Raw and summarized safetensors/GGUF benchmark evidence
Acceptance criteria
- Benchmark the same model architecture/revision, machine, prompts, context lengths, output lengths, sampling policy, and concurrency across the current Transformers/safetensors recipe and whole-model llama.cpp recipes.
- Separate correctness/quality lanes from quantized performance/fit lanes instead of claiming BF16 and Q4 are numerically equivalent.
- Report TTFT, prefill tok/s, decode tok/s, p50/p95 latency, aggregate throughput, RSS, VRAM, artifact size, failures, and output drift in machine-readable JSON.
- Add concurrency levels 1 and 4 where memory permits.
- Write a versioned performance contract consumed by later release gates, including an explicit stop condition when llama.cpp/GGUF has no meaningful speed or fit benefit.
- Targeted pytest tests pass
- python -m compileall packages tests passes for Python changes
- git diff --check passes
- Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
- Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
- Real-model execution is opt-in through MESHNET_ENABLE_REAL_INFERENCE_TESTS=1 and records exact artifact/runtime/hardware evidence
- Model artifacts remain on the configured mounted-drive storage and never under /home
- Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
- Read and verify every dependency evidence README before relying on dependency behavior
- Preserve all pre-existing working-tree changes and stage only files belonging to this story
- Write .scratch/distributed-gguf-runtime/evidence/DGR-001/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
- Update only this story issue to Status: done after every acceptance criterion and quality gate passes
Dependency handoff
- None. This story may start immediately.
Finish contract
- Create the task evidence directory and durable handoff required above.
- Preserve real failures and blockers; never fabricate benchmark, model, test or hardware output.
- Change this issue to
Status: doneonly after all criteria pass. - Emit
<promise>COMPLETE</promise>only after the evidence handoff exists.