1.7 KiB
PyTorch over llama.cpp for the inference engine
Runtime direction update (2026-07-13): PyTorch/safetensors remains the current production backend and correctness reference. A benchmark-gated native GGUF path is defined in ADR-0024; it does not replace this ADR until release gates pass.
We started with llama.cpp RPC as the distributed backend (following kyuz0/amd-strix-halo-toolboxes), but switched to PyTorch with a Petals-style shard pipeline. llama.cpp RPC requires the primary node to load the full model and distribute weights over the network at every session start — for a 70B model that's ~70GB over LAN per launch, making tracker-driven node rebalancing prohibitively expensive. PyTorch/Petals lets each node load its shard independently from local disk; only activations (~8KB per layer boundary per token) cross the network at inference time. PyTorch also has same-day support for new model architectures, training support (required for the planned torrent-style fine-tuning feature), and is the engine Petals itself uses for this exact use case.
Considered Options
- llama.cpp RPC: single binary, great quantized/CPU inference, no training support, full weights transferred over network on every session, day-0 model support lags by weeks
- PyTorch + Petals-style: nodes own their shards on disk, only activations transferred at inference, full training support, immediate new model support via HuggingFace
Consequences
The existing scripts/run_distributed_llama.py script (llama.cpp-based) is superseded. llama.cpp may still be used as an optional single-node inference backend on leaf nodes that don't participate in training.