neuron-tai/.scratch/distributed-gguf-runtime/issues/05-local-llamacpp-gguf-backend.md

05 — Local llama.cpp/GGUF backend

Status: ready-for-agent

What to build

Add a local full-model GGUF backend so a node that can hold a GGUF model can serve it through the existing node API. This is the immediate CPU-performance path and the baseline for later distributed llama.cpp work.

Acceptance criteria

• A node can start with backend llama.cpp or gguf for a local full-model GGUF artifact.
• The node can answer an OpenAI-compatible chat completion through the existing API.
• Startup and registration clearly report backend, quantization/artifact metadata, context cap, and local model path.
• The PyTorch backend remains unchanged and selectable.
• A smoke test or script validates backend wiring with a small GGUF model or a stubbed llama.cpp process.
• A benchmark command can compare local PyTorch CPU and local GGUF CPU for the same small supported model when both are available.

Blocked by

None - can start immediately.