3.3 KiB
3.3 KiB
01 — Lock the safetensors-versus-GGUF performance contract
Status: ready-for-agent
Mandatory fresh-session context
- Read RALPH-CONTEXT.md completely before changing code.
- This issue is
DGR-001in prd.json. - Read the evidence README for every dependency listed below.
- Inspect current code and
git status; historical text and previous agent claims are not evidence.
Description
As a runtime engineer, I need a controlled baseline so that GGUF work proceeds from measured speed, memory, and quality rather than reputation.
Expected durable outputs
- Benchmark harness and deterministic tests
- evidence/DGR-001/performance-contract.json
- Raw and summarized safetensors/GGUF benchmark evidence
Acceptance criteria
- Benchmark the same model architecture/revision, machine, prompts, context lengths, output lengths, sampling policy, and concurrency across the current Transformers/safetensors recipe and whole-model llama.cpp recipes.
- Separate correctness/quality lanes from quantized performance/fit lanes instead of claiming BF16 and Q4 are numerically equivalent.
- Report TTFT, prefill tok/s, decode tok/s, p50/p95 latency, aggregate throughput, RSS, VRAM, artifact size, failures, and output drift in machine-readable JSON.
- Add concurrency levels 1 and 4 where memory permits.
- Write a versioned performance contract consumed by later release gates, including an explicit stop condition when llama.cpp/GGUF has no meaningful speed or fit benefit.
- Targeted pytest tests pass
- python -m compileall packages tests passes for Python changes
- git diff --check passes
- Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
- Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
- Real-model execution is opt-in through MESHNET_ENABLE_REAL_INFERENCE_TESTS=1 and records exact artifact/runtime/hardware evidence
- Model artifacts remain on the configured mounted-drive storage and never under /home
- Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
- Read and verify every dependency evidence README before relying on dependency behavior
- Preserve all pre-existing working-tree changes and stage only files belonging to this story
- Write .scratch/distributed-gguf-runtime/evidence/DGR-001/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
- Update only this story issue to Status: done after every acceptance criterion and quality gate passes
Dependency handoff
- None. This story may start immediately.
Finish contract
- Create the task evidence directory and durable handoff required above.
- Preserve real failures and blockers; never fabricate benchmark, model, test or hardware output.
- Change this issue to
Status: doneonly after all criteria pass. - Emit
<promise>COMPLETE</promise>only after the evidence handoff exists.