30 lines
955 B
Markdown
30 lines
955 B
Markdown
# 01 — Local llama.cpp/GGUF backend
|
|
|
|
Status: ready-for-agent
|
|
|
|
## Goal
|
|
|
|
Add a local full-model llama.cpp/GGUF backend to the node so a machine that can hold a GGUF model can serve it through the existing OpenAI-compatible node API.
|
|
|
|
## Scope
|
|
|
|
- Add backend selection for `llama.cpp` / GGUF.
|
|
- Support launching or calling `llama-server` first; direct `libllama` bindings may come later.
|
|
- Register model metadata and hardware profile with tracker.
|
|
- Preserve current PyTorch path.
|
|
- Add a local benchmark comparing PyTorch CPU vs llama.cpp/GGUF for the same supported small model.
|
|
|
|
## Non-Goals
|
|
|
|
- No distributed GGUF route yet.
|
|
- No partial layer loading yet.
|
|
- No torrent seeding yet.
|
|
|
|
## Acceptance
|
|
|
|
- A local GGUF model can answer `/v1/chat/completions`.
|
|
- Startup output clearly says backend=`llama.cpp`.
|
|
- Node registration includes backend and artifact metadata.
|
|
- Test or smoke script verifies the backend wiring without requiring a huge model.
|
|
|