neuron-tai/.scratch/distributed-gguf-runtime/issues/01-local-llamacpp-gguf-backend.md

01 — Local llama.cpp/GGUF backend

Status: ready-for-agent

Goal

Add a local full-model llama.cpp/GGUF backend to the node so a machine that can hold a GGUF model can serve it through the existing OpenAI-compatible node API.

Scope

• Add backend selection for llama.cpp / GGUF.
• Support launching or calling llama-server first; direct libllama bindings may come later.
• Register model metadata and hardware profile with tracker.
• Preserve current PyTorch path.
• Add a local benchmark comparing PyTorch CPU vs llama.cpp/GGUF for the same supported small model.

Non-Goals

• No distributed GGUF route yet.
• No partial layer loading yet.
• No torrent seeding yet.

Acceptance

• A local GGUF model can answer /v1/chat/completions.
• Startup output clearly says backend=llama.cpp.
• Node registration includes backend and artifact metadata.
• Test or smoke script verifies the backend wiring without requiring a huge model.