docs: retarget gguf epic to DeepSeek-V4-Flash

This commit is contained in:
Dobromir Popov
2026-07-14 16:24:39 +03:00
parent 29351d6217
commit f102be1098

View File

@@ -1,9 +1,16 @@
# US-042 — GGUF/llama.cpp node backend
Status: planned
Priority: High (whole-model GGUF shortcut; distributed path in [ADR-0024](../adr/0024-distributed-gguf-runtime.md))
Priority: High (unlocks DeepSeek-V4-Flash on volunteer hardware — the pool's core value)
Stage: Draft design
## Goal
Run **DeepSeek-V4-Flash** as the first real large-model target on volunteer
hardware via GGUF/llama.cpp. This epic is no longer GLM-oriented: the initial
objective is to prove that DeepSeek-V4-Flash can load and serve correctly on
consumer/unified-memory nodes, then expand from there.
## Context
The node backend is transformers-only (`model_backend.py`
@@ -35,17 +42,7 @@ to it (single-hop route). Smallest step, no cross-node activation work, and
already useful: Strix Halo 128 GB serves DeepSeek-V4-Flash IQ3_XXS (114 GB)
via llama.cpp Vulkan today.
Recommended sequencing: **C first** (US-042), then **ADR-0024 benchmark gate** (DGR-001), then distributed native worker (DGR-002+). Direction B (llama.cpp RPC) is rejected per ADR-0024.
## Runtime sequencing
| Stage | Track | Delivers |
|---|---|---|
| **C — Whole-model GGUF** | US-042 (this issue) | Single-hop llama.cpp, billing, relay streaming |
| **0 — Benchmark gate** | ADR-0024 DGR-001 | Safetensors vs GGUF measured contract |
| **1 — Distributed GGUF** | ADR-0024 `.scratch/distributed-gguf-runtime/` | gRPC C++ worker, layer-range GGUF |
Phase C uses the existing tracker hop path (whole model, one node). ADR-0024 direction A (layer-range GGUF + activations) merges into the native worker track after the benchmark gate — not in parallel with phase C on the same backend without an integration plan.
Recommended sequencing: C first (small, real value), then A/B investigation.
## Also in scope