# Prior Art: Distributed Large-Model Inference This note captures what existing projects appear to solve and what remains specific to this platform. ## Petals Source: Petals is the closest conceptual match for public volunteer inference. Its README describes running large models "BitTorrent-style", where a user loads a model through a `transformers`-like API and connects to a distributed network that hosts model layers. It explicitly supports seeing hidden states and using PyTorch/Transformers flexibility. The public README also notes privacy limitations: data is processed by other people in the public swarm, and sensitive use should run in a private swarm. What it solves: - public swarm of layer-serving peers - hidden-state exposure - route-like execution over model blocks - private swarm option - PyTorch/Transformers integration What it does not directly solve for us: - GGUF/llama.cpp artifact path - torrent artifact storage tied to node rewards - our billing/fraud/reputation model - our OpenAI-compatible tracker/node route model - a production path for GLM-5.2/DSA GGUF Design import: - Keep a PyTorch route as a reference implementation and validation harness. - Preserve hidden-state seam semantics. - Treat privacy as an explicit swarm property. ## exo Source: exo connects local devices into an AI cluster. Its README emphasizes automatic device discovery, topology-aware model splitting, tensor parallelism, MLX support, RDMA over Thunderbolt, and multiple API compatibilities. It is strongest for colocated owned devices, especially Apple Silicon / MLX clusters. What it solves: - automatic local cluster discovery - topology-aware splitting - tensor parallelism - OpenAI/Ollama/Claude API compatibility - model placement previews - cluster dashboard What it does not directly solve for us: - untrusted internet volunteer network - reward, fraud, and reputation - torrent artifact distribution - Linux GPU maturity is stated as still under development in the README - GGUF/llama.cpp route protocol Design import: - Add placement previews before committing a route. - Model prefill/decode separately in benchmarks. - Use topology-aware routing, not just layer coverage. ## Distributed Llama / dllama Source: Distributed Llama connects home devices into a cluster for CPU/GPU inference. Its README describes tensor parallelism, Ethernet synchronization, Linux/macOS/Windows support, ARM and x86 AVX2 optimization, and a root/worker architecture. The root node loads the model and forwards weights/state to workers. Known limitations include only `2^n` nodes and a maximum node count equal to the model's number of KV heads. What it solves: - practical cross-platform home-device cluster - tensor-parallel synchronization - root/worker process model - custom model format and conversion path What it does not directly solve for us: - arbitrary volunteer joins/leaves - independent shard ownership from local/torrent disk - layer-range routing with tracker-managed marketplace - public network fraud/billing - GGUF as the native published artifact Design import: - KV-head constraints matter for tensor-parallel designs. - A root node that distributes weights is unacceptable for our torrent-first marketplace; nodes must independently acquire artifacts. ## prima.cpp Sources: - - prima.cpp is a distributed llama.cpp implementation for low-resource home clusters. The README highlights mmap-based low memory pressure, piped-ring parallelism with prefetching, heterogeneity-aware workload distribution, automatic weak-device removal, GGUF quantization support, speculative decoding, dynamic batching, and support for Llama/Qwen/DeepSeek-class models. Its commands require each rank to point at the same GGUF file, and the README shows ring communication across ranks. What it solves: - llama.cpp-derived GGUF distributed execution - heterogeneous device scheduling - low memory pressure via mmap/page cache behavior - disk prefetch as a first-class performance dimension - ring communication for home clusters - GGUF quantization support What it does not directly solve for us: - public volunteer marketplace - torrent artifact discovery and seeding economics - tracker-injected route over internet/NAT/relay - per-node independent shard selection and rewards - GLM-5.2 support is not established from the README Design import: - Study mmap and prefetching before inventing partial GGUF loading. - Include disk speed and memory pressure in routing. - Heterogeneity-aware scheduling is mandatory. - Weak nodes should be excluded from a route if they slow the whole decode path. ## llama.cpp / GGUF Sources: - - - llama.cpp is the performance runtime we want for GGUF. It supports local GGUF loading, many CPU/GPU backends, OpenAI-compatible serving, quantization, and `gguf-split` can split or merge GGUF files by max size or tensor count. The ggml build options include many hardware backends and RPC support. What it solves: - mature CPU/GPU local inference - GGUF ecosystem - quantized weights - local OpenAI-compatible server - split/merge tooling for artifact distribution What it does not solve by itself: - torrent distribution and reward model - per-session distributed route over arbitrary nodes - public-node trust/fraud model - stable API for arbitrary layer-boundary hidden-state I/O, if not already exposed Design import: - Use llama.cpp locally before attempting distributed GGUF. - Collaborate upstream on layer-range execution and KV ownership APIs. - Keep GGUF split for artifacts, not as the only execution-shard definition. ## GLM-5.2 Sources: - - GLM-5.2 is MIT licensed, 753B parameters, and advertises a 1M-token context. The config uses `glm_moe_dsa`, 78 layers, `hidden_size=6144`, `kv_lora_rank=512`, `qk_head_dim=256`, `qk_nope_head_dim=192`, `qk_rope_head_dim=64`, `v_head_dim=256`, and `max_position_embeddings=1048576`. The model card states IndexShare reduces per-token FLOPs at 1M context. Design import: - DSA/MLA-style compressed KV makes 128K feasible. - Tracker should not need to understand DSA internals; backend should expose cache budget and compatibility metadata. - GLM-5.2 is a later target after generic distributed KV works. ## DeepSeek-V4-Flash Sources: - - DeepSeek-V4-Flash is MIT licensed and published as `deepseek-ai/DeepSeek-V4-Flash` on Hugging Face. The model card describes DeepSeek-V4-Flash as a 284B-parameter MoE model with 13B activated parameters and a 1M-token context. Hugging Face tags it as `deepseek_v4`, Transformers, Safetensors, and FP8. The repository lists 46 safetensor shards and around 160 GB total size. Config highlights: - `model_type=deepseek_v4` - `hidden_size=4096` - `num_hidden_layers=43` - `num_attention_heads=64` - `num_key_value_heads=1` - `n_routed_experts=256` - `num_experts_per_tok=6` - `q_lora_rank=1024` - `o_lora_rank=1024` - `qk_rope_head_dim=64` - `sliding_window=128` - `max_position_embeddings=1048576` - `expert_dtype=fp4` - FP8 quantization metadata Design import: - Good first serious large-model target after the protocol smoke test because it is much smaller than 1.6T-class models while still validating MoE, compressed attention/cache behavior, and large-context routing. - Not the first protocol smoke model. Use a smaller, boring, llama.cpp-supported GGUF model first so route/session/KV bugs are isolated from DeepSeek-specific architecture support. - The support audit must verify the available local runtime path: PyTorch/Transformers, vLLM/SGLang, and any GGUF/llama.cpp quantization route. ## Ornith-1.0-397B Sources: - - Ornith-1.0-397B is MIT licensed, Qwen3.5-MoE based, with 397B MoE scale. Its base config shows 60 layers and a hybrid pattern where full attention appears every fourth layer, with other layers using linear attention. The MLX Q9 quantized variant is around 447 GB and reports high-quality Q9 behavior in its model card. Design import: - Hybrid attention can make large models more tractable than dense full-attention assumptions. - Model-specific cache accounting is required; "params" alone is not enough to route. ## Synthesis The prior art strongly supports the direction, but no project exactly matches the target product: - Petals proves volunteer layer-serving is useful. - exo proves UX/topology-aware local clusters matter. - Distributed Llama proves CPU home clusters can cooperate but also shows root/worker constraints. - prima.cpp proves llama.cpp/GGUF distribution across low-resource devices is plausible and that disk/mmap scheduling matters. - llama.cpp/GGUF is the ecosystem to collaborate with for runtime performance. - DeepSeek-V4-Flash is a plausible first serious large-model target after a small protocol smoke model. The platform-specific work remains: - torrent/content-addressed model artifact marketplace - tracker-owned route selection and billing - per-shard local KV sessions - relay/NAT support - fraud/reputation/audit - OpenAI-compatible public gateway