new tasks, model pricing, auto quantisation, etc...

docs/issues/42-gguf-llamacpp-node-backend.md

@@ -0,0 +1,55 @@
+# US-042 — GGUF/llama.cpp node backend
+
+Status: planned
+Priority: High (unlocks big MoE models on volunteer hardware — the pool's core value)
+Stage: Draft design
+
+## Context
+
+The node backend is transformers-only (`model_backend.py` →
+`AutoModelForCausalLM`). For DeepSeek-V4-Flash (158B MoE, official weights FP8
+160 GB) the only quantizations that run on consumer hardware are GGUF
+(IQ2 87 GB → Q4_K_M-XL 175 GB) — llama.cpp format. The transformers-compatible
+quants (FP8, NVFP4, GPTQ W4A16) all need datacenter GPUs. Volunteer machines —
+including our own Strix Halo boxes (128 GB and 80 GB unified memory, GPU via
+Vulkan/ROCm, no FP8 support on RDNA3.5) — run these models today only under
+llama.cpp.
+
+## Design directions to evaluate (design-it-twice)
+
+**A. llama.cpp as a per-node shard executor.** Node loads a *layer range* of a
+GGUF via llama-cpp-python; our existing hop protocol (X-Meshnet-Route,
+activations over HTTP/relay) moves hidden states between nodes. Requires
+llama.cpp partial-layer loading and activation import/export — investigate
+feasibility first; this is the riskiest unknown.
+
+**B. llama.cpp RPC mode under tracker orchestration.** llama.cpp ships a
+native RPC backend that splits one model across machines. The tracker would
+provision/route to an llama.cpp RPC cluster rather than our own hop pipeline.
+Less code, but bypasses our billing/telemetry hop instrumentation and relay
+NAT path — needs a story for both.
+
+**C. Whole-model GGUF nodes (no sharding).** A node with enough memory serves
+a full GGUF (e.g. IQ2/IQ3 on a 128 GB box); the tracker routes whole requests
+to it (single-hop route). Smallest step, no cross-node activation work, and
+already useful: Strix Halo 128 GB serves DeepSeek-V4-Flash IQ3_XXS (114 GB)
+via llama.cpp Vulkan today.
+
+Recommended sequencing: C first (small, real value), then A/B investigation.
+
+## Also in scope
+
+- Model catalog: allow GGUF entries with quant selection; feature
+  `DeepSeek-V4-Flash` IQ4_XS/UD-Q4_K_XL as a curated/featured entry once at
+  least direction C works (a featured model nobody can load is an anti-feature)
+- Hardware detection: recognize Strix Halo/unified-memory APUs and Vulkan
+  (`hardware.py` currently reports "CPU mode" on these boxes)
+- `MESHNET_DOWNLOAD_DIR`/`--download-dir` applies to GGUF files as well
+
+## Acceptance criteria (phase C)
+
+- A node with `--gguf <repo-or-path> --quant IQ3_XXS` serves
+  `/v1/chat/completions` via llama.cpp with GPU offload where available
+- Tracker treats it as a full-coverage node (single-hop routes, billing works)
+- Streamed responses work through the tracker proxy and the relay (US-036)
+- `python -m pytest` passes from repo root (llama.cpp behind an optional extra)