# PyTorch over llama.cpp for the inference engine

We started with llama.cpp RPC as the distributed backend (following kyuz0/amd-strix-halo-toolboxes), but switched to PyTorch with a Petals-style shard pipeline. llama.cpp RPC requires the primary node to load the full model and distribute weights over the network at every session start — for a 70B model that's ~70GB over LAN per launch, making tracker-driven node rebalancing prohibitively expensive. PyTorch/Petals lets each node load its shard independently from local disk; only activations (~8KB per layer boundary per token) cross the network at inference time. PyTorch also has same-day support for new model architectures, training support (required for the planned torrent-style fine-tuning feature), and is the engine Petals itself uses for this exact use case.

## Considered Options

- **llama.cpp RPC**: single binary, great quantized/CPU inference, no training support, full weights transferred over network on every session, day-0 model support lags by weeks
- **PyTorch + Petals-style**: nodes own their shards on disk, only activations transferred at inference, full training support, immediate new model support via HuggingFace

## Consequences

The existing `scripts/run_distributed_llama.py` script (llama.cpp-based) is superseded. llama.cpp may still be used as an optional single-node inference backend on leaf nodes that don't participate in training.