popov/neuron-tai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

documentation revision

Browse Source
This commit is contained in:
Dobromir Popov
2026-07-13 18:14:21 +02:00
parent 180a7674e6
commit 95245be512
43 changed files with 78 additions and 57 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
docs/adr/0001-pytorch-over-llama-cpp.md
View File

@@ -1,5 +1,7 @@
# PyTorch over llama.cpp for the inference engine
> **Runtime direction update (2026-07-13):** PyTorch/safetensors remains the current production backend and correctness reference. A benchmark-gated native GGUF path is defined in [ADR-0024](0024-distributed-gguf-runtime.md); it does not replace this ADR until release gates pass.
We started with llama.cpp RPC as the distributed backend (following kyuz0/amd-strix-halo-toolboxes), but switched to PyTorch with a Petals-style shard pipeline. llama.cpp RPC requires the primary node to load the full model and distribute weights over the network at every session start — for a 70B model that's ~70GB over LAN per launch, making tracker-driven node rebalancing prohibitively expensive. PyTorch/Petals lets each node load its shard independently from local disk; only activations (~8KB per layer boundary per token) cross the network at inference time. PyTorch also has same-day support for new model architectures, training support (required for the planned torrent-style fine-tuning feature), and is the engine Petals itself uses for this exact use case.
## Considered Options