21 lines
980 B
Markdown
21 lines
980 B
Markdown
# 05 — Local llama.cpp/GGUF backend
|
|
|
|
Status: ready-for-agent
|
|
|
|
## What to build
|
|
|
|
Add a local full-model GGUF backend so a node that can hold a GGUF model can serve it through the existing node API. This is the immediate CPU-performance path and the baseline for later distributed llama.cpp work.
|
|
|
|
## Acceptance criteria
|
|
|
|
- [ ] A node can start with backend `llama.cpp` or `gguf` for a local full-model GGUF artifact.
|
|
- [ ] The node can answer an OpenAI-compatible chat completion through the existing API.
|
|
- [ ] Startup and registration clearly report backend, quantization/artifact metadata, context cap, and local model path.
|
|
- [ ] The PyTorch backend remains unchanged and selectable.
|
|
- [ ] A smoke test or script validates backend wiring with a small GGUF model or a stubbed llama.cpp process.
|
|
- [ ] A benchmark command can compare local PyTorch CPU and local GGUF CPU for the same small supported model when both are available.
|
|
|
|
## Blocked by
|
|
|
|
None - can start immediately.
|