neuron-tai/.scratch/distributed-gguf-runtime/research-prior-art.md

Prior Art: Distributed Large-Model Inference

This note captures what existing projects appear to solve and what remains specific to this platform.

Petals

Source: https://github.com/bigscience-workshop/petals

Petals is the closest conceptual match for public volunteer inference. Its README describes running large models "BitTorrent-style", where a user loads a model through a transformers-like API and connects to a distributed network that hosts model layers. It explicitly supports seeing hidden states and using PyTorch/Transformers flexibility. The public README also notes privacy limitations: data is processed by other people in the public swarm, and sensitive use should run in a private swarm.

What it solves:

• public swarm of layer-serving peers
• hidden-state exposure
• route-like execution over model blocks
• private swarm option
• PyTorch/Transformers integration

What it does not directly solve for us:

• GGUF/llama.cpp artifact path
• torrent artifact storage tied to node rewards
• our billing/fraud/reputation model
• our OpenAI-compatible tracker/node route model
• a production path for GLM-5.2/DSA GGUF

Design import:

• Keep a PyTorch route as a reference implementation and validation harness.
• Preserve hidden-state seam semantics.
• Treat privacy as an explicit swarm property.

exo

Source: https://github.com/exo-explore/exo

exo connects local devices into an AI cluster. Its README emphasizes automatic device discovery, topology-aware model splitting, tensor parallelism, MLX support, RDMA over Thunderbolt, and multiple API compatibilities. It is strongest for colocated owned devices, especially Apple Silicon / MLX clusters.

What it solves:

• automatic local cluster discovery
• topology-aware splitting
• tensor parallelism
• OpenAI/Ollama/Claude API compatibility
• model placement previews
• cluster dashboard

What it does not directly solve for us:

• untrusted internet volunteer network
• reward, fraud, and reputation
• torrent artifact distribution
• Linux GPU maturity is stated as still under development in the README
• GGUF/llama.cpp route protocol

Design import:

• Add placement previews before committing a route.
• Model prefill/decode separately in benchmarks.
• Use topology-aware routing, not just layer coverage.

Distributed Llama / dllama

Source: https://github.com/b4rtaz/distributed-llama

Distributed Llama connects home devices into a cluster for CPU/GPU inference. Its README describes tensor parallelism, Ethernet synchronization, Linux/macOS/Windows support, ARM and x86 AVX2 optimization, and a root/worker architecture. The root node loads the model and forwards weights/state to workers. Known limitations include only 2^n nodes and a maximum node count equal to the model's number of KV heads.

What it solves:

• practical cross-platform home-device cluster
• tensor-parallel synchronization
• root/worker process model
• custom model format and conversion path

What it does not directly solve for us:

• arbitrary volunteer joins/leaves
• independent shard ownership from local/torrent disk
• layer-range routing with tracker-managed marketplace
• public network fraud/billing
• GGUF as the native published artifact

Design import:

• KV-head constraints matter for tensor-parallel designs.
• A root node that distributes weights is unacceptable for our torrent-first marketplace; nodes must independently acquire artifacts.

prima.cpp

Sources:

• https://github.com/Lizonghang/prima.cpp
• https://arxiv.org/abs/2504.08791

prima.cpp is a distributed llama.cpp implementation for low-resource home clusters. The README highlights mmap-based low memory pressure, piped-ring parallelism with prefetching, heterogeneity-aware workload distribution, automatic weak-device removal, GGUF quantization support, speculative decoding, dynamic batching, and support for Llama/Qwen/DeepSeek-class models. Its commands require each rank to point at the same GGUF file, and the README shows ring communication across ranks.

What it solves:

• llama.cpp-derived GGUF distributed execution
• heterogeneous device scheduling
• low memory pressure via mmap/page cache behavior
• disk prefetch as a first-class performance dimension
• ring communication for home clusters
• GGUF quantization support

What it does not directly solve for us:

• public volunteer marketplace
• torrent artifact discovery and seeding economics
• tracker-injected route over internet/NAT/relay
• per-node independent shard selection and rewards
• GLM-5.2 support is not established from the README

Design import:

• Study mmap and prefetching before inventing partial GGUF loading.
• Include disk speed and memory pressure in routing.
• Heterogeneity-aware scheduling is mandatory.
• Weak nodes should be excluded from a route if they slow the whole decode path.

llama.cpp / GGUF

Sources:

• https://github.com/ggml-org/llama.cpp
• https://raw.githubusercontent.com/ggml-org/llama.cpp/master/tools/gguf-split/README.md
• https://raw.githubusercontent.com/ggml-org/llama.cpp/master/ggml/CMakeLists.txt

llama.cpp is the performance runtime we want for GGUF. It supports local GGUF loading, many CPU/GPU backends, OpenAI-compatible serving, quantization, and gguf-split can split or merge GGUF files by max size or tensor count. The ggml build options include many hardware backends and RPC support.

What it solves:

• mature CPU/GPU local inference
• GGUF ecosystem
• quantized weights
• local OpenAI-compatible server
• split/merge tooling for artifact distribution

What it does not solve by itself:

• torrent distribution and reward model
• per-session distributed route over arbitrary nodes
• public-node trust/fraud model
• stable API for arbitrary layer-boundary hidden-state I/O, if not already exposed

Design import:

• Use llama.cpp locally before attempting distributed GGUF.
• Collaborate upstream on layer-range execution and KV ownership APIs.
• Keep GGUF split for artifacts, not as the only execution-shard definition.

GLM-5.2

Sources:

• https://huggingface.co/zai-org/GLM-5.2
• https://huggingface.co/zai-org/GLM-5.2/blob/main/config.json

GLM-5.2 is MIT licensed, 753B parameters, and advertises a 1M-token context. The config uses glm_moe_dsa, 78 layers, hidden_size=6144, kv_lora_rank=512, qk_head_dim=256, qk_nope_head_dim=192, qk_rope_head_dim=64, v_head_dim=256, and max_position_embeddings=1048576. The model card states IndexShare reduces per-token FLOPs at 1M context.

Design import:

• DSA/MLA-style compressed KV makes 128K feasible.
• Tracker should not need to understand DSA internals; backend should expose cache budget and compatibility metadata.
• GLM-5.2 is a later target after generic distributed KV works.

DeepSeek-V4-Flash

Sources:

• https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash
• https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/main/config.json

DeepSeek-V4-Flash is MIT licensed and published as deepseek-ai/DeepSeek-V4-Flash on Hugging Face. The model card describes DeepSeek-V4-Flash as a 284B-parameter MoE model with 13B activated parameters and a 1M-token context. Hugging Face tags it as deepseek_v4, Transformers, Safetensors, and FP8. The repository lists 46 safetensor shards and around 160 GB total size.

Config highlights:

• model_type=deepseek_v4
• hidden_size=4096
• num_hidden_layers=43
• num_attention_heads=64
• num_key_value_heads=1
• n_routed_experts=256
• num_experts_per_tok=6
• q_lora_rank=1024
• o_lora_rank=1024
• qk_rope_head_dim=64
• sliding_window=128
• max_position_embeddings=1048576
• expert_dtype=fp4
• FP8 quantization metadata

Design import:

• Good first serious large-model target after the protocol smoke test because it is much smaller than 1.6T-class models while still validating MoE, compressed attention/cache behavior, and large-context routing.
• Not the first protocol smoke model. Use a smaller, boring, llama.cpp-supported GGUF model first so route/session/KV bugs are isolated from DeepSeek-specific architecture support.
• The support audit must verify the available local runtime path: PyTorch/Transformers, vLLM/SGLang, and any GGUF/llama.cpp quantization route.

Ornith-1.0-397B

Sources:

• https://huggingface.co/deepreinforce-ai/Ornith-1.0-397B
• https://huggingface.co/inferencerlabs/Ornith-1.0-397B-MLX-Q9

Ornith-1.0-397B is MIT licensed, Qwen3.5-MoE based, with 397B MoE scale. Its base config shows 60 layers and a hybrid pattern where full attention appears every fourth layer, with other layers using linear attention. The MLX Q9 quantized variant is around 447 GB and reports high-quality Q9 behavior in its model card.

Design import:

• Hybrid attention can make large models more tractable than dense full-attention assumptions.
• Model-specific cache accounting is required; "params" alone is not enough to route.

Synthesis

The prior art strongly supports the direction, but no project exactly matches the target product:

• Petals proves volunteer layer-serving is useful.
• exo proves UX/topology-aware local clusters matter.
• Distributed Llama proves CPU home clusters can cooperate but also shows root/worker constraints.
• prima.cpp proves llama.cpp/GGUF distribution across low-resource devices is plausible and that disk/mmap scheduling matters.
• llama.cpp/GGUF is the ecosystem to collaborate with for runtime performance.
• DeepSeek-V4-Flash is a plausible first serious large-model target after a small protocol smoke model.

The platform-specific work remains:

• torrent/content-addressed model artifact marketplace
• tracker-owned route selection and billing
• per-shard local KV sessions
• relay/NAT support
• fraud/reputation/audit
• OpenAI-compatible public gateway