Compare commits
1 Commits
ralph/dist
...
ralph/deep
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
f102be1098 |
@@ -1,9 +1,16 @@
|
|||||||
# US-042 — GGUF/llama.cpp node backend
|
# US-042 — GGUF/llama.cpp node backend
|
||||||
|
|
||||||
Status: planned
|
Status: planned
|
||||||
Priority: High (whole-model GGUF shortcut; distributed path in [ADR-0024](../adr/0024-distributed-gguf-runtime.md))
|
Priority: High (unlocks DeepSeek-V4-Flash on volunteer hardware — the pool's core value)
|
||||||
Stage: Draft design
|
Stage: Draft design
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
|
||||||
|
Run **DeepSeek-V4-Flash** as the first real large-model target on volunteer
|
||||||
|
hardware via GGUF/llama.cpp. This epic is no longer GLM-oriented: the initial
|
||||||
|
objective is to prove that DeepSeek-V4-Flash can load and serve correctly on
|
||||||
|
consumer/unified-memory nodes, then expand from there.
|
||||||
|
|
||||||
## Context
|
## Context
|
||||||
|
|
||||||
The node backend is transformers-only (`model_backend.py` →
|
The node backend is transformers-only (`model_backend.py` →
|
||||||
@@ -35,17 +42,7 @@ to it (single-hop route). Smallest step, no cross-node activation work, and
|
|||||||
already useful: Strix Halo 128 GB serves DeepSeek-V4-Flash IQ3_XXS (114 GB)
|
already useful: Strix Halo 128 GB serves DeepSeek-V4-Flash IQ3_XXS (114 GB)
|
||||||
via llama.cpp Vulkan today.
|
via llama.cpp Vulkan today.
|
||||||
|
|
||||||
Recommended sequencing: **C first** (US-042), then **ADR-0024 benchmark gate** (DGR-001), then distributed native worker (DGR-002+). Direction B (llama.cpp RPC) is rejected per ADR-0024.
|
Recommended sequencing: C first (small, real value), then A/B investigation.
|
||||||
|
|
||||||
## Runtime sequencing
|
|
||||||
|
|
||||||
| Stage | Track | Delivers |
|
|
||||||
|---|---|---|
|
|
||||||
| **C — Whole-model GGUF** | US-042 (this issue) | Single-hop llama.cpp, billing, relay streaming |
|
|
||||||
| **0 — Benchmark gate** | ADR-0024 DGR-001 | Safetensors vs GGUF measured contract |
|
|
||||||
| **1 — Distributed GGUF** | ADR-0024 `.scratch/distributed-gguf-runtime/` | gRPC C++ worker, layer-range GGUF |
|
|
||||||
|
|
||||||
Phase C uses the existing tracker hop path (whole model, one node). ADR-0024 direction A (layer-range GGUF + activations) merges into the native worker track after the benchmark gate — not in parallel with phase C on the same backend without an integration plan.
|
|
||||||
|
|
||||||
## Also in scope
|
## Also in scope
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user