234 lines
9.9 KiB
Markdown
234 lines
9.9 KiB
Markdown
# Prior Art: Distributed Large-Model Inference
|
|
|
|
> **Superseded as the current source audit.** Use [`docs/research/distributed-gguf-landscape.md`](../../docs/research/distributed-gguf-landscape.md), [`distributed-gguf-github-followup.md`](../../docs/research/distributed-gguf-github-followup.md), and [`vllm-distributed-gguf-assessment.md`](../../docs/research/vllm-distributed-gguf-assessment.md). This file remains as early historical research.
|
|
|
|
This note captures what existing projects appear to solve and what remains specific to this platform.
|
|
|
|
## Petals
|
|
|
|
Source: <https://github.com/bigscience-workshop/petals>
|
|
|
|
Petals is the closest conceptual match for public volunteer inference. Its README describes running large models "BitTorrent-style", where a user loads a model through a `transformers`-like API and connects to a distributed network that hosts model layers. It explicitly supports seeing hidden states and using PyTorch/Transformers flexibility. The public README also notes privacy limitations: data is processed by other people in the public swarm, and sensitive use should run in a private swarm.
|
|
|
|
What it solves:
|
|
|
|
- public swarm of layer-serving peers
|
|
- hidden-state exposure
|
|
- route-like execution over model blocks
|
|
- private swarm option
|
|
- PyTorch/Transformers integration
|
|
|
|
What it does not directly solve for us:
|
|
|
|
- GGUF/llama.cpp artifact path
|
|
- torrent artifact storage tied to node rewards
|
|
- our billing/fraud/reputation model
|
|
- our OpenAI-compatible tracker/node route model
|
|
- a production path for GLM-5.2/DSA GGUF
|
|
|
|
Design import:
|
|
|
|
- Keep a PyTorch route as a reference implementation and validation harness.
|
|
- Preserve hidden-state seam semantics.
|
|
- Treat privacy as an explicit swarm property.
|
|
|
|
## exo
|
|
|
|
Source: <https://github.com/exo-explore/exo>
|
|
|
|
exo connects local devices into an AI cluster. Its README emphasizes automatic device discovery, topology-aware model splitting, tensor parallelism, MLX support, RDMA over Thunderbolt, and multiple API compatibilities. It is strongest for colocated owned devices, especially Apple Silicon / MLX clusters.
|
|
|
|
What it solves:
|
|
|
|
- automatic local cluster discovery
|
|
- topology-aware splitting
|
|
- tensor parallelism
|
|
- OpenAI/Ollama/Claude API compatibility
|
|
- model placement previews
|
|
- cluster dashboard
|
|
|
|
What it does not directly solve for us:
|
|
|
|
- untrusted internet volunteer network
|
|
- reward, fraud, and reputation
|
|
- torrent artifact distribution
|
|
- Linux GPU maturity is stated as still under development in the README
|
|
- GGUF/llama.cpp route protocol
|
|
|
|
Design import:
|
|
|
|
- Add placement previews before committing a route.
|
|
- Model prefill/decode separately in benchmarks.
|
|
- Use topology-aware routing, not just layer coverage.
|
|
|
|
## Distributed Llama / dllama
|
|
|
|
Source: <https://github.com/b4rtaz/distributed-llama>
|
|
|
|
Distributed Llama connects home devices into a cluster for CPU/GPU inference. Its README describes tensor parallelism, Ethernet synchronization, Linux/macOS/Windows support, ARM and x86 AVX2 optimization, and a root/worker architecture. The root node loads the model and forwards weights/state to workers. Known limitations include only `2^n` nodes and a maximum node count equal to the model's number of KV heads.
|
|
|
|
What it solves:
|
|
|
|
- practical cross-platform home-device cluster
|
|
- tensor-parallel synchronization
|
|
- root/worker process model
|
|
- custom model format and conversion path
|
|
|
|
What it does not directly solve for us:
|
|
|
|
- arbitrary volunteer joins/leaves
|
|
- independent shard ownership from local/torrent disk
|
|
- layer-range routing with tracker-managed marketplace
|
|
- public network fraud/billing
|
|
- GGUF as the native published artifact
|
|
|
|
Design import:
|
|
|
|
- KV-head constraints matter for tensor-parallel designs.
|
|
- A root node that distributes weights is unacceptable for our torrent-first marketplace; nodes must independently acquire artifacts.
|
|
|
|
## prima.cpp
|
|
|
|
Sources:
|
|
|
|
- <https://github.com/Lizonghang/prima.cpp>
|
|
- <https://arxiv.org/abs/2504.08791>
|
|
|
|
prima.cpp is a distributed llama.cpp implementation for low-resource home clusters. The README highlights mmap-based low memory pressure, piped-ring parallelism with prefetching, heterogeneity-aware workload distribution, automatic weak-device removal, GGUF quantization support, speculative decoding, dynamic batching, and support for Llama/Qwen/DeepSeek-class models. Its commands require each rank to point at the same GGUF file, and the README shows ring communication across ranks.
|
|
|
|
What it solves:
|
|
|
|
- llama.cpp-derived GGUF distributed execution
|
|
- heterogeneous device scheduling
|
|
- low memory pressure via mmap/page cache behavior
|
|
- disk prefetch as a first-class performance dimension
|
|
- ring communication for home clusters
|
|
- GGUF quantization support
|
|
|
|
What it does not directly solve for us:
|
|
|
|
- public volunteer marketplace
|
|
- torrent artifact discovery and seeding economics
|
|
- tracker-injected route over internet/NAT/relay
|
|
- per-node independent shard selection and rewards
|
|
- GLM-5.2 support is not established from the README
|
|
|
|
Design import:
|
|
|
|
- Study mmap and prefetching before inventing partial GGUF loading.
|
|
- Include disk speed and memory pressure in routing.
|
|
- Heterogeneity-aware scheduling is mandatory.
|
|
- Weak nodes should be excluded from a route if they slow the whole decode path.
|
|
|
|
## llama.cpp / GGUF
|
|
|
|
Sources:
|
|
|
|
- <https://github.com/ggml-org/llama.cpp>
|
|
- <https://raw.githubusercontent.com/ggml-org/llama.cpp/master/tools/gguf-split/README.md>
|
|
- <https://raw.githubusercontent.com/ggml-org/llama.cpp/master/ggml/CMakeLists.txt>
|
|
|
|
llama.cpp is the performance runtime we want for GGUF. It supports local GGUF loading, many CPU/GPU backends, OpenAI-compatible serving, quantization, and `gguf-split` can split or merge GGUF files by max size or tensor count. The ggml build options include many hardware backends and RPC support.
|
|
|
|
What it solves:
|
|
|
|
- mature CPU/GPU local inference
|
|
- GGUF ecosystem
|
|
- quantized weights
|
|
- local OpenAI-compatible server
|
|
- split/merge tooling for artifact distribution
|
|
|
|
What it does not solve by itself:
|
|
|
|
- torrent distribution and reward model
|
|
- per-session distributed route over arbitrary nodes
|
|
- public-node trust/fraud model
|
|
- stable API for arbitrary layer-boundary hidden-state I/O, if not already exposed
|
|
|
|
Design import:
|
|
|
|
- Use llama.cpp locally before attempting distributed GGUF.
|
|
- Collaborate upstream on layer-range execution and KV ownership APIs.
|
|
- Keep GGUF split for artifacts, not as the only execution-shard definition.
|
|
|
|
## GLM-5.2
|
|
|
|
Sources:
|
|
|
|
- <https://huggingface.co/zai-org/GLM-5.2>
|
|
- <https://huggingface.co/zai-org/GLM-5.2/blob/main/config.json>
|
|
|
|
GLM-5.2 is MIT licensed, 753B parameters, and advertises a 1M-token context. The config uses `glm_moe_dsa`, 78 layers, `hidden_size=6144`, `kv_lora_rank=512`, `qk_head_dim=256`, `qk_nope_head_dim=192`, `qk_rope_head_dim=64`, `v_head_dim=256`, and `max_position_embeddings=1048576`. The model card states IndexShare reduces per-token FLOPs at 1M context.
|
|
|
|
Design import:
|
|
|
|
- DSA/MLA-style compressed KV makes 128K feasible.
|
|
- Tracker should not need to understand DSA internals; backend should expose cache budget and compatibility metadata.
|
|
- GLM-5.2 is a later target after generic distributed KV works.
|
|
|
|
## DeepSeek-V4-Flash
|
|
|
|
Sources:
|
|
|
|
- <https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash>
|
|
- <https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/main/config.json>
|
|
|
|
DeepSeek-V4-Flash is MIT licensed and published as `deepseek-ai/DeepSeek-V4-Flash` on Hugging Face. The model card describes DeepSeek-V4-Flash as a 284B-parameter MoE model with 13B activated parameters and a 1M-token context. Hugging Face tags it as `deepseek_v4`, Transformers, Safetensors, and FP8. The repository lists 46 safetensor shards and around 160 GB total size.
|
|
|
|
Config highlights:
|
|
|
|
- `model_type=deepseek_v4`
|
|
- `hidden_size=4096`
|
|
- `num_hidden_layers=43`
|
|
- `num_attention_heads=64`
|
|
- `num_key_value_heads=1`
|
|
- `n_routed_experts=256`
|
|
- `num_experts_per_tok=6`
|
|
- `q_lora_rank=1024`
|
|
- `o_lora_rank=1024`
|
|
- `qk_rope_head_dim=64`
|
|
- `sliding_window=128`
|
|
- `max_position_embeddings=1048576`
|
|
- `expert_dtype=fp4`
|
|
- FP8 quantization metadata
|
|
|
|
Design import:
|
|
|
|
- Good first serious large-model target after the protocol smoke test because it is much smaller than 1.6T-class models while still validating MoE, compressed attention/cache behavior, and large-context routing.
|
|
- Not the first protocol smoke model. Use a smaller, boring, llama.cpp-supported GGUF model first so route/session/KV bugs are isolated from DeepSeek-specific architecture support.
|
|
- The support audit must verify the available local runtime path: PyTorch/Transformers, vLLM/SGLang, and any GGUF/llama.cpp quantization route.
|
|
|
|
## Ornith-1.0-397B
|
|
|
|
Sources:
|
|
|
|
- <https://huggingface.co/deepreinforce-ai/Ornith-1.0-397B>
|
|
- <https://huggingface.co/inferencerlabs/Ornith-1.0-397B-MLX-Q9>
|
|
|
|
Ornith-1.0-397B is MIT licensed, Qwen3.5-MoE based, with 397B MoE scale. Its base config shows 60 layers and a hybrid pattern where full attention appears every fourth layer, with other layers using linear attention. The MLX Q9 quantized variant is around 447 GB and reports high-quality Q9 behavior in its model card.
|
|
|
|
Design import:
|
|
|
|
- Hybrid attention can make large models more tractable than dense full-attention assumptions.
|
|
- Model-specific cache accounting is required; "params" alone is not enough to route.
|
|
|
|
## Synthesis
|
|
|
|
The prior art strongly supports the direction, but no project exactly matches the target product:
|
|
|
|
- Petals proves volunteer layer-serving is useful.
|
|
- exo proves UX/topology-aware local clusters matter.
|
|
- Distributed Llama proves CPU home clusters can cooperate but also shows root/worker constraints.
|
|
- prima.cpp proves llama.cpp/GGUF distribution across low-resource devices is plausible and that disk/mmap scheduling matters.
|
|
- llama.cpp/GGUF is the ecosystem to collaborate with for runtime performance.
|
|
- DeepSeek-V4-Flash is a plausible first serious large-model target after a small protocol smoke model.
|
|
|
|
The platform-specific work remains:
|
|
|
|
- torrent/content-addressed model artifact marketplace
|
|
- tracker-owned route selection and billing
|
|
- per-shard local KV sessions
|
|
- relay/NAT support
|
|
- fraud/reputation/audit
|
|
- OpenAI-compatible public gateway
|