15 lines
1.7 KiB
Markdown
15 lines
1.7 KiB
Markdown
# PyTorch over llama.cpp for the inference engine
|
|
|
|
> **Runtime direction update (2026-07-13):** PyTorch/safetensors remains the current production backend and correctness reference. A benchmark-gated native GGUF path is defined in [ADR-0024](0024-distributed-gguf-runtime.md); it does not replace this ADR until release gates pass.
|
|
|
|
We started with llama.cpp RPC as the distributed backend (following kyuz0/amd-strix-halo-toolboxes), but switched to PyTorch with a Petals-style shard pipeline. llama.cpp RPC requires the primary node to load the full model and distribute weights over the network at every session start — for a 70B model that's ~70GB over LAN per launch, making tracker-driven node rebalancing prohibitively expensive. PyTorch/Petals lets each node load its shard independently from local disk; only activations (~8KB per layer boundary per token) cross the network at inference time. PyTorch also has same-day support for new model architectures, training support (required for the planned torrent-style fine-tuning feature), and is the engine Petals itself uses for this exact use case.
|
|
|
|
## Considered Options
|
|
|
|
- **llama.cpp RPC**: single binary, great quantized/CPU inference, no training support, full weights transferred over network on every session, day-0 model support lags by weeks
|
|
- **PyTorch + Petals-style**: nodes own their shards on disk, only activations transferred at inference, full training support, immediate new model support via HuggingFace
|
|
|
|
## Consequences
|
|
|
|
The existing `scripts/run_distributed_llama.py` script (llama.cpp-based) is superseded. llama.cpp may still be used as an optional single-node inference backend on leaf nodes that don't participate in training.
|