3.4 KiB
US-042 — GGUF/llama.cpp node backend
Status: planned Priority: High (whole-model GGUF shortcut; distributed path in ADR-0024) Stage: Draft design
Context
The node backend is transformers-only (model_backend.py →
AutoModelForCausalLM). For DeepSeek-V4-Flash (158B MoE, official weights FP8
160 GB) the only quantizations that run on consumer hardware are GGUF
(IQ2 87 GB → Q4_K_M-XL 175 GB) — llama.cpp format. The transformers-compatible
quants (FP8, NVFP4, GPTQ W4A16) all need datacenter GPUs. Volunteer machines —
including our own Strix Halo boxes (128 GB and 80 GB unified memory, GPU via
Vulkan/ROCm, no FP8 support on RDNA3.5) — run these models today only under
llama.cpp.
Design directions to evaluate (design-it-twice)
A. llama.cpp as a per-node shard executor. Node loads a layer range of a GGUF via llama-cpp-python; our existing hop protocol (X-Meshnet-Route, activations over HTTP/relay) moves hidden states between nodes. Requires llama.cpp partial-layer loading and activation import/export — investigate feasibility first; this is the riskiest unknown.
B. llama.cpp RPC mode under tracker orchestration. llama.cpp ships a native RPC backend that splits one model across machines. The tracker would provision/route to an llama.cpp RPC cluster rather than our own hop pipeline. Less code, but bypasses our billing/telemetry hop instrumentation and relay NAT path — needs a story for both.
C. Whole-model GGUF nodes (no sharding). A node with enough memory serves a full GGUF (e.g. IQ2/IQ3 on a 128 GB box); the tracker routes whole requests to it (single-hop route). Smallest step, no cross-node activation work, and already useful: Strix Halo 128 GB serves DeepSeek-V4-Flash IQ3_XXS (114 GB) via llama.cpp Vulkan today.
Recommended sequencing: C first (US-042), then ADR-0024 benchmark gate (DGR-001), then distributed native worker (DGR-002+). Direction B (llama.cpp RPC) is rejected per ADR-0024.
Runtime sequencing
| Stage | Track | Delivers |
|---|---|---|
| C — Whole-model GGUF | US-042 (this issue) | Single-hop llama.cpp, billing, relay streaming |
| 0 — Benchmark gate | ADR-0024 DGR-001 | Safetensors vs GGUF measured contract |
| 1 — Distributed GGUF | ADR-0024 .scratch/distributed-gguf-runtime/ |
gRPC C++ worker, layer-range GGUF |
Phase C uses the existing tracker hop path (whole model, one node). ADR-0024 direction A (layer-range GGUF + activations) merges into the native worker track after the benchmark gate — not in parallel with phase C on the same backend without an integration plan.
Also in scope
- Model catalog: allow GGUF entries with quant selection; feature
DeepSeek-V4-FlashIQ4_XS/UD-Q4_K_XL as a curated/featured entry once at least direction C works (a featured model nobody can load is an anti-feature) - Hardware detection: recognize Strix Halo/unified-memory APUs and Vulkan
(
hardware.pycurrently reports "CPU mode" on these boxes) MESHNET_DOWNLOAD_DIR/--download-dirapplies to GGUF files as well
Acceptance criteria (phase C)
- A node with
--gguf <repo-or-path> --quant IQ3_XXSserves/v1/chat/completionsvia llama.cpp with GPU offload where available - Tracker treats it as a full-coverage node (single-hop routes, billing works)
- Streamed responses work through the tracker proxy and the relay (US-036)
python -m pytestpasses from repo root (llama.cpp behind an optional extra)