feat: compare safetensors and gguf on cpu and gpu

This commit is contained in:
Dobromir Popov
2026-07-14 18:45:12 +03:00
parent c7554ef7d8
commit 5b33bf8b99
5 changed files with 88 additions and 9 deletions

View File

@@ -15,7 +15,12 @@ As a runtime engineer, I need a controlled baseline so that GGUF work proceeds f
## Baseline model target
Use the smallest *DeepSeek-family* GGUF that still points toward DeepSeek-V4-Flash. Current choice: **DeepSeek-V2-Lite GGUF Q2_K** (~6.5GB, `deepseek2` architecture). Reserve smaller non-DeepSeek fallback models only for loader plumbing smoke tests if needed; they do not count as the DGR-001 architecture-aligned baseline.
Use the same model on both sides of the comparison, with the closest practical low-footprint precision pair:
- **safetensors:** `deepseek-ai/DeepSeek-V2-Lite-Chat` in **BF16**
- **GGUF:** `second-state/DeepSeek-V2-Lite-Chat-GGUF` in **Q2_K** (~6.5GB)
Keep the benchmark matrix explicit for **CPU** and **GPU** runs. Reserve smaller non-DeepSeek fallback models only for loader plumbing smoke tests if needed; they do not count as the DGR-001 architecture-aligned baseline.
## Expected durable outputs