# Prior Art: Distributed Large-Model Inference

> **Superseded as the current source audit.** Use [`docs/research/distributed-gguf-landscape.md`](../../docs/research/distributed-gguf-landscape.md), [`distributed-gguf-github-followup.md`](../../docs/research/distributed-gguf-github-followup.md), and [`vllm-distributed-gguf-assessment.md`](../../docs/research/vllm-distributed-gguf-assessment.md). This file remains as early historical research.

This note captures what existing projects appear to solve and what remains specific to this platform.

## Petals

Source: <https://github.com/bigscience-workshop/petals>

Petals is the closest conceptual match for public volunteer inference. Its README describes running large models "BitTorrent-style", where a user loads a model through a `transformers`-like API and connects to a distributed network that hosts model layers. It explicitly supports seeing hidden states and using PyTorch/Transformers flexibility. The public README also notes privacy limitations: data is processed by other people in the public swarm, and sensitive use should run in a private swarm.

What it solves:

- public swarm of layer-serving peers
- hidden-state exposure
- route-like execution over model blocks
- private swarm option
- PyTorch/Transformers integration

What it does not directly solve for us:

- GGUF/llama.cpp artifact path
- torrent artifact storage tied to node rewards
- our billing/fraud/reputation model
- our OpenAI-compatible tracker/node route model
- a production path for GLM-5.2/DSA GGUF

Design import:

- Keep a PyTorch route as a reference implementation and validation harness.
- Preserve hidden-state seam semantics.
- Treat privacy as an explicit swarm property.

## exo

Source: <https://github.com/exo-explore/exo>

exo connects local devices into an AI cluster. Its README emphasizes automatic device discovery, topology-aware model splitting, tensor parallelism, MLX support, RDMA over Thunderbolt, and multiple API compatibilities. It is strongest for colocated owned devices, especially Apple Silicon / MLX clusters.

What it solves:

- automatic local cluster discovery
- topology-aware splitting
- tensor parallelism
- OpenAI/Ollama/Claude API compatibility
- model placement previews
- cluster dashboard

What it does not directly solve for us:

- untrusted internet volunteer network
- reward, fraud, and reputation
- torrent artifact distribution
- Linux GPU maturity is stated as still under development in the README
- GGUF/llama.cpp route protocol

Design import:

- Add placement previews before committing a route.
- Model prefill/decode separately in benchmarks.
- Use topology-aware routing, not just layer coverage.

## Distributed Llama / dllama

Source: <https://github.com/b4rtaz/distributed-llama>

Distributed Llama connects home devices into a cluster for CPU/GPU inference. Its README describes tensor parallelism, Ethernet synchronization, Linux/macOS/Windows support, ARM and x86 AVX2 optimization, and a root/worker architecture. The root node loads the model and forwards weights/state to workers. Known limitations include only `2^n` nodes and a maximum node count equal to the model's number of KV heads.

What it solves:

- practical cross-platform home-device cluster
- tensor-parallel synchronization
- root/worker process model
- custom model format and conversion path

What it does not directly solve for us:

- arbitrary volunteer joins/leaves
- independent shard ownership from local/torrent disk
- layer-range routing with tracker-managed marketplace
- public network fraud/billing
- GGUF as the native published artifact

Design import:

- KV-head constraints matter for tensor-parallel designs.
- A root node that distributes weights is unacceptable for our torrent-first marketplace; nodes must independently acquire artifacts.

## prima.cpp

Sources:

- <https://github.com/Lizonghang/prima.cpp>
- <https://arxiv.org/abs/2504.08791>

prima.cpp is a distributed llama.cpp implementation for low-resource home clusters. The README highlights mmap-based low memory pressure, piped-ring parallelism with prefetching, heterogeneity-aware workload distribution, automatic weak-device removal, GGUF quantization support, speculative decoding, dynamic batching, and support for Llama/Qwen/DeepSeek-class models. Its commands require each rank to point at the same GGUF file, and the README shows ring communication across ranks.

What it solves:

- llama.cpp-derived GGUF distributed execution
- heterogeneous device scheduling
- low memory pressure via mmap/page cache behavior
- disk prefetch as a first-class performance dimension
- ring communication for home clusters
- GGUF quantization support

What it does not directly solve for us:

- public volunteer marketplace
- torrent artifact discovery and seeding economics
- tracker-injected route over internet/NAT/relay
- per-node independent shard selection and rewards
- GLM-5.2 support is not established from the README

Design import:

- Study mmap and prefetching before inventing partial GGUF loading.
- Include disk speed and memory pressure in routing.
- Heterogeneity-aware scheduling is mandatory.
- Weak nodes should be excluded from a route if they slow the whole decode path.

## llama.cpp / GGUF

Sources:

- <https://github.com/ggml-org/llama.cpp>
- <https://raw.githubusercontent.com/ggml-org/llama.cpp/master/tools/gguf-split/README.md>
- <https://raw.githubusercontent.com/ggml-org/llama.cpp/master/ggml/CMakeLists.txt>

llama.cpp is the performance runtime we want for GGUF. It supports local GGUF loading, many CPU/GPU backends, OpenAI-compatible serving, quantization, and `gguf-split` can split or merge GGUF files by max size or tensor count. The ggml build options include many hardware backends and RPC support.

What it solves:

- mature CPU/GPU local inference
- GGUF ecosystem
- quantized weights
- local OpenAI-compatible server
- split/merge tooling for artifact distribution

What it does not solve by itself:

- torrent distribution and reward model
- per-session distributed route over arbitrary nodes
- public-node trust/fraud model
- stable API for arbitrary layer-boundary hidden-state I/O, if not already exposed

Design import:

- Use llama.cpp locally before attempting distributed GGUF.
- Collaborate upstream on layer-range execution and KV ownership APIs.
- Keep GGUF split for artifacts, not as the only execution-shard definition.

## GLM-5.2

Sources:

- <https://huggingface.co/zai-org/GLM-5.2>
- <https://huggingface.co/zai-org/GLM-5.2/blob/main/config.json>

GLM-5.2 is MIT licensed, 753B parameters, and advertises a 1M-token context. The config uses `glm_moe_dsa`, 78 layers, `hidden_size=6144`, `kv_lora_rank=512`, `qk_head_dim=256`, `qk_nope_head_dim=192`, `qk_rope_head_dim=64`, `v_head_dim=256`, and `max_position_embeddings=1048576`. The model card states IndexShare reduces per-token FLOPs at 1M context.

Design import:

- DSA/MLA-style compressed KV makes 128K feasible.
- Tracker should not need to understand DSA internals; backend should expose cache budget and compatibility metadata.
- GLM-5.2 is a later target after generic distributed KV works.

## DeepSeek-V4-Flash

Sources:

- <https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash>
- <https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/main/config.json>

DeepSeek-V4-Flash is MIT licensed and published as `deepseek-ai/DeepSeek-V4-Flash` on Hugging Face. The model card describes DeepSeek-V4-Flash as a 284B-parameter MoE model with 13B activated parameters and a 1M-token context. Hugging Face tags it as `deepseek_v4`, Transformers, Safetensors, and FP8. The repository lists 46 safetensor shards and around 160 GB total size.

Config highlights:

- `model_type=deepseek_v4`
- `hidden_size=4096`
- `num_hidden_layers=43`
- `num_attention_heads=64`
- `num_key_value_heads=1`
- `n_routed_experts=256`
- `num_experts_per_tok=6`
- `q_lora_rank=1024`
- `o_lora_rank=1024`
- `qk_rope_head_dim=64`
- `sliding_window=128`
- `max_position_embeddings=1048576`
- `expert_dtype=fp4`
- FP8 quantization metadata

Design import:

- Good first serious large-model target after the protocol smoke test because it is much smaller than 1.6T-class models while still validating MoE, compressed attention/cache behavior, and large-context routing.
- Not the first protocol smoke model. Use a smaller, boring, llama.cpp-supported GGUF model first so route/session/KV bugs are isolated from DeepSeek-specific architecture support.
- The support audit must verify the available local runtime path: PyTorch/Transformers, vLLM/SGLang, and any GGUF/llama.cpp quantization route.

## Ornith-1.0-397B

Sources:

- <https://huggingface.co/deepreinforce-ai/Ornith-1.0-397B>
- <https://huggingface.co/inferencerlabs/Ornith-1.0-397B-MLX-Q9>

Ornith-1.0-397B is MIT licensed, Qwen3.5-MoE based, with 397B MoE scale. Its base config shows 60 layers and a hybrid pattern where full attention appears every fourth layer, with other layers using linear attention. The MLX Q9 quantized variant is around 447 GB and reports high-quality Q9 behavior in its model card.

Design import:

- Hybrid attention can make large models more tractable than dense full-attention assumptions.
- Model-specific cache accounting is required; "params" alone is not enough to route.

## Synthesis

The prior art strongly supports the direction, but no project exactly matches the target product:

- Petals proves volunteer layer-serving is useful.
- exo proves UX/topology-aware local clusters matter.
- Distributed Llama proves CPU home clusters can cooperate but also shows root/worker constraints.
- prima.cpp proves llama.cpp/GGUF distribution across low-resource devices is plausible and that disk/mmap scheduling matters.
- llama.cpp/GGUF is the ecosystem to collaborate with for runtime performance.
- DeepSeek-V4-Flash is a plausible first serious large-model target after a small protocol smoke model.

The platform-specific work remains:

- torrent/content-addressed model artifact marketplace
- tracker-owned route selection and billing
- per-shard local KV sessions
- relay/NAT support
- fraud/reputation/audit
- OpenAI-compatible public gateway
