popov/neuron-tai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: define distributed GGUF runtime plan

Browse Source
This commit is contained in:
Dobromir Popov
2026-07-13 15:09:27 +03:00
parent b5fa7245df
commit 4cae4a6c5c
42 changed files with 4913 additions and 691 deletions
Download Patch File Download Diff File Expand all files Collapse all files

803
docs/research/distributed-gguf-github-followup.md Normal file
View File

@@ -0,0 +1,803 @@
# Distributed GGUF GitHub follow-up: GPUStack, Nakshatra, LiGGUF, and additional candidates
Status: Source audit complete
Last updated: 2026-07-13
## 1. Why this follow-up exists
This document evaluates additional claims and repositories found after the initial distributed-GGUF landscape report:
- The GPUStack 0.4 multi-worker GGUF tutorial.
- The claim that llama.cpp is the base of most practical GGUF distribution.
- Nakshatra's patched llama.cpp layer workers.
- LiGGUF's SARA distributed example.
- Chameleon and Continuum.
- Additional GitHub searches for sub-GGUF, layer-range, activation-chain, and llama.cpp RPC implementations.
It supplements [the main distributed-GGUF landscape report](distributed-gguf-landscape.md). The dedicated [vLLM assessment](vllm-distributed-gguf-assessment.md) remains separate.
## 2. Corrected terminology
The original research summary was directionally correct but combined several different forms of parallelism.
### 2.1 Layer or pipeline parallelism
Whole contiguous transformer-layer ranges are assigned to stages:
```text
tokens
-> layers 0..N
-> boundary residual
-> layers N..M
-> logits
```
This is the closest match to neuron-tai's tracker-selected route.
### 2.2 Tensor parallelism
Operations inside every layer are divided across ranks:
- Attention heads.
- Matrix rows or columns.
- FFN channels.
- Experts.
Ranks exchange collectives or partial reductions inside each transformer layer. LiGGUF SARA is an example.
### 2.3 Local multi-device placement
llama.cpp's `n_gpu_layers` and `tensor_split` choose how one coordinator places work across devices. This is local offload unless some devices are llama.cpp RPC devices.
### 2.4 llama.cpp RPC
llama.cpp RPC exposes a remote GGML backend/device to the coordinator. It is real cross-machine inference, but the remote server is not an independent layer/session worker. GPUStack 0.4 and llama-box used this mechanism.
### 2.5 Quantization
Q2_K, Q4_K_M, Q8_0, and related GGUF types reduce weight storage and memory. They do not define:
- Activation dtype.
- Compute dtype.
- KV-cache dtype.
- Parallelism topology.
- Session or route semantics.
## 3. Audited source snapshots
```text
GPUStack current main
244eb0da57add11d1ce07c70f31c1a15ae65ae0d
GPUStack v0.4.1
dbf71dd16cec1f42896139c3b82380cb1fd06a10
llama-box
4d068484fe198a30f8ca6d6d23d9890fbd8eee8c
Nakshatra
0c16119713396ec6052400f3eb049c5e7a66cd94
LiGGUF
2b5ac66ca37f36600aa5101b4237e74f3becb7c4
Chameleon
96fbd96a9f67d29d12292d3373c88996aba65f84
Continuum
dd976df36079d75244719a23956e1c9e2dcddc27
```
## 4. GPUStack 0.4 and llama-box
### 4.1 Verdict
The [GPUStack 0.4 tutorial](https://docs.gpustack.ai/0.4/tutorials/performing-distributed-inference-across-workers/) describes a real open-source deployment path. It is not a closed-source native layer-shard engine.
GPUStack 0.4 orchestrated llama-box, which embedded llama.cpp/ggml RPC. The execution shape was:
```text
GPUStack scheduler
-> select main worker and remote GPU devices
-> start llama-box RPC server on each selected remote GPU
-> launch llama-box on main worker
--rpc remote-a,remote-b,...
--tensor-split remote-vram...,local-vram...
-> coordinator opens the full GGUF
-> llama.cpp places tensors and graph operations on local and RPC devices
```
This is strong real-world evidence for llama.cpp RPC. It is not the independent layer-worker topology neuron-tai needs.
### 4.2 Scheduler and resource estimation
GPUStack detects GGUF models, estimates memory using its parser/calculator, marks distributable models, selects a main worker plus remote GPU devices, and persists their resource claims.
Evidence from GPUStack v0.4.1:
- `gpustack/scheduler/scheduler.py:147-205`.
- `gpustack/scheduler/scheduler.py:328-377`.
- `gpustack/scheduler/scheduler.py:429-449`.
- `gpustack/policies/utils.py:35-47`.
- `gpustack/policies/scorers/placement_scorer.py:184-196`.
This control-plane work is reusable conceptually:
- Parse exact GGUF requirements before placement.
- Allocate memory claims per device.
- Include remote-device allocations in global scheduling.
- Reject incompatible backend/runtime versions.
### 4.3 RPC server lifecycle
GPUStack workers periodically start one llama-box RPC process per GPU and publish its port in worker status.
Evidence:
- `gpustack/worker/worker.py:153-163`.
- `gpustack/worker/worker_manager.py:140-200`.
- `gpustack/worker/collector.py:66-80`.
- `gpustack/worker/rpc_server.py:31-76`.
The launched command uses:
```text
--rpc-server-host 0.0.0.0
--rpc-server-port <port>
--rpc-server-main-gpu 0
```
GPU selection is enforced through the vendor-specific visible-device environment.
### 4.4 Main server launch
The main worker obtains remote RPC addresses and remote VRAM claims, then launches llama-box with:
```text
--rpc <host:port,...>
--tensor-split <remote MiB...,local MiB...>
```
Evidence:
- `gpustack/worker/backends/llama_box.py:33-95`.
- `gpustack/worker/backends/llama_box.py:165-181`.
This confirms that GPUStack's worker list did not become a route of independently callable layer stages. llama-box remained the single model owner and request server.
### 4.5 llama-box RPC behavior
llama-box's RPC server serializes GGML buffers, tensors, and graph-compute requests. It exposes remote backend memory and supports optional tensor caching.
Evidence:
- `llama-box/rpcserver.hpp:74-213`.
- `llama-box/rpcserver.hpp:215-226`.
- `llama-box/rpcserver.hpp:374-410`.
- `llama-box/rpcserver.hpp:413-447`.
The RPC server is a low-level remote device. It does not own:
- A source GGUF identity.
- A tracker layer-range lease.
- A route session or route epoch.
- Independent tokenization/head/tail semantics.
- A project-compatible activation endpoint.
- Per-request node work receipts.
### 4.6 Real operational evidence
Three GPUStack issue reports are useful:
1. [Issue 1233](https://github.com/gpustack/gpustack/issues/1233) records a two-Mac llama-box RPC deployment and a maintainer reproduction command. Maintainers reported roughly 5-6 token/s for a large DeepSeek-R1 GGUF on two M2 Ultra systems.
2. [Issue 756](https://github.com/gpustack/gpustack/issues/756) records a crash when main and RPC llama-box versions differed and a remote Metal backend did not support an operation.
3. [Issue 1269](https://github.com/gpustack/gpustack/issues/1269) includes a real ten-device `--rpc`/`--tensor-split` command and an official maintainer statement that GPUStack 2.0 deprecated llama-box and no longer supports distributed GGUF inference.
These reports prove practical use while also demonstrating:
- Exact runtime/version compatibility is mandatory.
- Every remote backend must support every placed graph operation.
- Remote-memory estimates can still fail.
- Coordinator interruption can leak or strand remote resources.
- Cross-machine operation may be slower than expected.
### 4.7 Current status
The audited current GPUStack main snapshot retains legacy GGUF RPC-placement calculations, deprecated `rpc_servers` schema fields, migration code, and fixtures. It no longer has the llama-box backend or role-specific per-GPU RPC process launcher needed to turn those placement records into a working GGUF data plane. Current GGUF defaults to a custom backend, and GPUStack's supported multi-node matrix lists vLLM, SGLang, and MindIE rather than custom/GGUF.
Evidence from current GPUStack main:
- `gpustack/policies/candidate_selectors/gguf_resource_fit_selector.py:459-473`.
- `gpustack/policies/candidate_selectors/gguf_resource_fit_selector.py:1969-1997`.
- `gpustack/schemas/workers.py:198-202`.
- `docs/migration.md:157-167`.
- `gpustack/schemas/models.py:59-65`.
- `gpustack/schemas/models.py:266-275`.
- `gpustack/schemas/models.py:873-880`.
- `gpustack/worker/backends/custom.py:152-181`.
- `docs/faq.md:9-21`.
The retained selector is legacy residue and compatibility evidence, not a supported runnable distributed-GGUF backend.
Decision:
- Use GPUStack v0.4 as a llama.cpp RPC baseline and scheduling reference.
- Do not treat current GPUStack as a maintained distributed-GGUF backend.
- Do not reuse llama.cpp RPC as the volunteer network trust boundary.
## 5. Nakshatra
### 5.1 Verdict
Nakshatra is the closest implementation found to neuron-tai's native distributed-GGUF target.
It independently implements the same core seam selected in the initial report:
- Patched llama.cpp.
- Local layer-only GGUF artifacts.
- Contiguous first/middle/last workers.
- Residual activation transport.
- Worker-local llama.cpp KV.
- A long-lived C++ daemon supervised by Python.
- Dynamic placement and recovery above the worker.
It changes the implementation strategy from “write the first spike from scratch” to “reproduce, collaborate, reuse, and harden.”
It is not a wholesale drop-in because its control plane, artifact model, concurrency, identity, accounting, and architecture coverage differ from Meshnet.
### 5.2 Sub-GGUF construction
`partial_gguf.py` creates a derivative GGUF per layer range.
It preserves source metadata, filters `blk.N.*` tensors to `[start,end)`, keeps embeddings only for the head unless tied output needs them, and keeps output tensors only for the tail.
Evidence:
- `experiments/v0.0/partial_gguf.py:59-115`.
- `experiments/v0.0/partial_gguf.py:117-143`.
- `experiments/v0.0/partial_gguf.py:184-201`.
It writes:
```text
nakshatra.layer_range_start
nakshatra.layer_range_end
nakshatra.has_token_embd
nakshatra.has_lm_head
```
This gives workers real local ownership and prevents inference-time weight transfer.
Meshnet differences:
- Meshnet prefers one exact source artifact hash plus a range/recipe identity.
- Derived sub-GGUF files need their own hash and a signed binding to the source artifact and range.
- Rewriting a large GGUF for every placement is expensive and duplicates storage.
- A range-aware mmap loader from one shared artifact remains preferable long term.
Sub-GGUFs are still acceptable for a first integration spike because they already work.
### 5.3 llama.cpp patch mechanics
Nakshatra's patch series:
- Adds range and endpoint fields to the model.
- Reads namespaced partial-model metadata.
- Allows unowned tensors to be absent.
- Creates/loads only owned layer tensors.
- Iterates only the selected layer interval.
- Returns the unnormalized residual stream from non-tail stages.
- Applies final norm and LM head only on the tail.
Evidence:
- `experiments/v0.0/m4_patches/llama-model.h.patch:1-17`.
- `experiments/v0.0/m4_patches/llama-model.cpp.patch:1-73`.
- `experiments/v0.0/m4_patches/llama-model-loader.cpp.patch:1-11`.
- `experiments/v0.0/m4_patches/llama-graph.cpp.patch:1-24`.
- `experiments/v0.0/m4_patches/models_llama.cpp.patch:1-70`.
This is direct proof that the small-fork direction is feasible. The current patch is architecture-specific and built against older llama.cpp revisions. It should be rebased rather than copied blindly.
### 5.4 Worker shape
Each Python gRPC worker supervises one long-lived C++ daemon. The daemon owns the patched llama model/context and communicates with Python through a framed stdin/stdout or shared-memory protocol.
Evidence:
- `scripts/worker.py:1-17`.
- `experiments/v0.0/worker_daemon.cpp:1-52`.
- `experiments/v0.0/worker_daemon.cpp:166-267`.
- `experiments/v0.0/worker_daemon.cpp:282-396`.
The daemon accepts:
- Token IDs for a head stage.
- F32 boundary activations for middle/tail stages.
- `start_pos` and `keep_kv` controls.
It returns:
- F32 residual activations for non-tail stages.
- Greedy token IDs on the tail.
- Optional all-position top tokens for speculative verification.
The implementation also contains optional blockwise int8 activation transport:
- `experiments/v0.0/worker_daemon.cpp:105-149`.
This process boundary is close to the selected Meshnet worker shape and avoids exposing llama.cpp internal ABI to Python.
### 5.5 Protobuf and route semantics
Nakshatra's protobuf includes:
- Protocol/backend/model/range capability reporting.
- A source-model hash field.
- Head/tail ownership flags.
- Stable `session_id` and idempotency `step_id`.
- Prefix/KV position metadata.
- Token, hidden-state, logits, and error variants.
- Server-to-server next-hop chains.
- KV truncation.
- Sleep/wake lifecycle operations.
Evidence:
- `proto/nakshatra.proto:1-55`.
- `proto/nakshatra.proto:57-93`.
- `proto/nakshatra.proto:95-131`.
Meshnet's worker contract still needs additional fields:
- Route epoch.
- Effective overlap-safe start layer.
- Exact source and derived artifact hashes.
- Architecture/runtime/quantization/activation recipe.
- Request/work/accounting identity.
- Payload checksum and compression recipe.
- Cache expectation and explicit cache-miss response.
- Cancellation and lease identity.
The implementation should sit behind a Meshnet-owned protocol rather than adopting the protobuf as a permanent public API.
### 5.6 KV ownership and concurrency gap
The daemon supports:
- Cold prefill.
- Incremental decode.
- KV truncation after speculative rejection.
- Sleep/wake and model reload.
Evidence:
- `experiments/v0.0/worker_daemon.cpp:344-384`.
- `experiments/v0.0/worker_daemon.cpp:416-463`.
- `experiments/v0.0/worker_daemon.cpp:479-500`.
However, the audited worker creates one llama context with two sequence slots:
```text
sequence 0: serving/verification
sequence 1: EAGLE scratch
```
The v0.5 design explicitly states that the daemon still has one logical serving session and ships with `n_concurrent_sessions = 1`:
- `docs/v0.5-design-lock.md:97-111`.
The Python idempotency cache deduplicates `(session_id,step_id)` outputs. It does not isolate independent llama KV state for multiple concurrent route sessions.
Meshnet integration must add:
```text
(route_session_id, route_epoch)
-> llama_seq_id or isolated context
```
and verify concurrent prefill/decode, release, eviction, stale epoch rejection, and cache misses.
### 5.7 Real acceptance evidence
Nakshatra records a two-physical-machine CPU-only test over Tailscale:
- Head worker: layers `[0,14)`.
- Tail worker: layers `[14,28)`.
- Prompt: “The capital of France is”.
- Distributed first token: `12366`, “ Paris”.
- Matched localhost, single-process chain, and whole-model llama.cpp reference.
Evidence:
- `experiments/v0.0/m6_findings.md:1-42`.
This proves real independent layer execution and cross-machine activation transport. It does not prove GPU heterogeneity because both machines used CPU for the acceptance run.
The repository also records a four-worker Llama-3.3-70B chain using Macs with Metal and one CPU stage:
- Streaming completed around 0.21 token/s.
- Multi-hop server push completed around 0.19 token/s.
- Push was slower because compute dominated network time.
- First generated token was “Paris”.
- Alternate-worker replay completed with expected post-splice divergence.
Evidence:
- `docs/v0.5-design-lock.md:145-175`.
- `docs/v0.5-design-lock.md:258-277`.
The five-machine ROCm + Metal cross-vendor acceptance remained pending in the audited snapshot. Claims of fully validated ROCm+Metal execution should therefore remain qualified.
### 5.8 Failure and numerical behavior
Nakshatra supports:
- Persistent streaming RPCs.
- Server-to-server activation push.
- Fallback from failed push to client relay.
- Full-history replay after stream failure.
- Alternate workers per range.
- Drift-class-aware recovery design.
Evidence:
- `scripts/client.py:163-269`.
- `scripts/client.py:842-850`.
- `docs/v0.5-design-lock.md:258-277`.
- `docs/v1.0-fault-tolerance.md:93-103`.
The project correctly acknowledges that replay on another backend can numerically diverge. For Meshnet, the safe default remains:
1. Exact recipe and same drift class for in-session replacement.
2. Otherwise restart from token zero on a newly consistent route.
3. Never silently import incompatible KV or continue with an unvalidated mixed recipe.
### 5.9 Security and work receipts
The worker contains optional:
- TLS and SPKI pinning.
- Ed25519 request authentication.
- Admission and sandbox hooks.
- Audit logging.
- An experimental encrypted fabric.
Evidence:
- `scripts/worker.py:49-117`.
- `scripts/worker.py:218-319`.
The receipt implementation explicitly documents that:
- Output hash and structural consistency are independently checkable.
- Model identity is only asserted because the live model hash is a zero stub.
- Participation is coordinator-asserted because worker signatures are not populated.
Evidence:
- `scripts/receipt.py:1-27`.
- `scripts/receipt.py:34-47`.
- `scripts/receipt.py:50-97`.
- `scripts/receipt.py:100-160`.
This is not sufficient for per-node rewards. Meshnet must bind receipts to:
```text
request/work ID
route session and epoch
source artifact hash
layer range and effective start
input/output activation digests
positions/token count
runtime recipe
measured compute
worker identity/signature
```
### 5.10 Reproducibility and integration defects
The audited repository is not a self-contained llama.cpp fork or reproducible worker build:
- It ships patch files but no pinned llama.cpp source tree or submodule.
- The README recommends llama.cpp commit `c46583b` “or close enough”.
- The daemon is copied manually into an external llama.cpp checkout and its CMake target is added manually.
- The documented copy recipe omits `shm_ring.hpp`, which the daemon includes.
- The historical two-machine run used different llama.cpp builds on the two hosts.
Evidence:
- `README.md:41-77`.
- `experiments/v0.0/worker_daemon.cpp:54-57`.
- `experiments/v0.0/m6_findings.md:35-42`.
The live daemon also trusts operator-supplied range and endpoint mode rather than returning authoritative metadata from the loaded sub-GGUF. Its INFO response reports a generic full range and both endpoint flags, while Python advertises CLI values:
- `experiments/v0.0/worker_daemon.cpp:387-392`.
- `experiments/v0.0/worker_daemon.cpp:466-475`.
- `scripts/worker.py:1141-1153`.
Additional integration gaps:
- Normal activations are little-endian F32 even though the protobuf comment says FP16 is the default; int8 activation mode is environment-global rather than negotiated.
- The coordinator requires full-GGUF access through `llama-cpp-python` for tokenization.
- Tail sampling is hard-coded greedy top-1 rather than returning general logits.
- Many tests use fake daemons; no CI job builds patched llama.cpp, generates slices, starts two real workers, and tests prefill/decode/failure.
- Nakshatra runtime dependencies are absent from the inherited Petals package metadata.
Evidence:
- `proto/nakshatra.proto:11-12`.
- `scripts/worker.py:340-343`.
- `scripts/worker.py:1141-1151`.
- `scripts/client.py:129-139`.
- `scripts/client.py:719-722`.
- `experiments/v0.0/worker_daemon.cpp:617-641`.
- `tests/test_worker_eagle_sleep_rpc.py:1-31`.
- `setup.cfg:1-16`.
- `setup.cfg:29-72`.
These defects make Nakshatra a strong source donor and independent feasibility proof, but not the repository to fork as the production base.
### 5.11 Reuse decision
Do not recreate Nakshatra's working Llama-family layer patch and daemon from scratch without first attempting upstream collaboration.
Preferred plan:
1. Reproduce its two-worker path locally.
2. Rebase its patch against the exact current llama.cpp commit already audited.
3. Compare the patch with the proposed `llama_model_load_range` and boundary-output design.
4. Borrow or jointly maintain narrow patch concepts and tests, but keep a project-owned standalone worker and small pinned llama.cpp fork rather than forking the Petals-derived repository wholesale.
5. Replace or adapt its Python control plane with existing Meshnet tracker/session/relay/billing infrastructure.
6. Add multi-session KV isolation, exact recipe identity, route epochs, cancellation, signed work receipts, and architecture certification.
7. Upstream generic llama.cpp range-loading/boundary hooks where maintainers will accept them.
Classification:
```text
Primary source donor and collaboration candidate
Do not adopt or fork the repository wholesale
```
## 6. LiGGUF SARA
### 6.1 Verdict
LiGGUF contains a real experimental networked distributed implementation. It is not layer pipeline parallelism.
Its SARA implementation performs tensor-parallel activation reduction across every transformer layer.
### 6.2 Mechanism
Every process mmaps and parses the complete GGUF and constructs pointers for all transformer blocks:
- `cpp/ligguf_distrib.cpp:323-485`.
Rank assignments split:
- KV heads.
- Query heads derived from KV heads.
- FFN channel blocks.
Evidence:
- `cpp/ligguf_distrib.cpp:120-124`.
- `cpp/ligguf_distrib.cpp:657-679`.
Each rank allocates KV for its attention-head slice across every layer:
- `cpp/ligguf_distrib.cpp:681-707`.
For every transformer layer, the master:
1. Normalizes the residual.
2. Broadcasts a Q8_0 activation to all workers.
3. Every rank computes its attention partial.
4. Workers return Q8_0 full-width partials.
5. The master sums partials into the residual.
6. Repeats the same process for the FFN.
Evidence:
- `cpp/ligguf_distrib.cpp:709-767`.
- `cpp/ligguf_distrib.cpp:892-903`.
- `cpp/ligguf_distrib.cpp:957-974`.
- `cpp/ligguf_distrib.cpp:1025-1066`.
### 6.3 Strengths
- Very compact and readable.
- Direct GGUF mmap.
- Real raw-TCP master/worker implementation.
- Q8_0 activation and partial transport.
- Local KV per attention-head shard.
- Useful reference for tensor-parallel reductions on CPU-heavy edge systems.
### 6.4 Mismatch
- Every worker needs the complete GGUF.
- Every worker participates in every layer.
- Two network reduction phases occur per transformer layer.
- Static rank/world topology.
- One master connection and one generation at a time.
- Sequential worker receive loop.
- No model/session/request/range identity.
- No authentication, checksums, recovery, cancellation, or accounting.
- Custom inference core has much narrower model/kernel coverage than llama.cpp.
Evidence:
- `cpp/ligguf_distrib.cpp:769-974`.
- `cpp/ligguf_distrib.cpp:1025-1046`.
- `cpp/ligguf_distrib.cpp:1138-1241`.
Decision: retain as a source donor for compact Q8 activation transport, tensor-partition tests, and reduction benchmarking. Do not use it as the native Meshnet layer worker.
## 7. Chameleon
Audited snapshot:
```text
megeezy/Chameleon
commit 96fbd96a9f67d29d12292d3373c88996aba65f84
```
Chameleon is a whole-model lifecycle and routing system:
- Coordinator selects a model and worker.
- Python worker loads a complete llama-cpp-python, vLLM, Transformers, or ExLlamaV2 backend.
- The model executes, can remain warm, and is later unloaded.
It does not split one model's layers or tensors across workers. Its README also labels it design phase.
Decision: exclude from the distributed-GGUF implementation list. Whole-model load/unload and warm-cache ideas may be relevant to proxy backends only.
## 8. Continuum
Audited snapshot:
```text
CambrianTech/continuum
commit dd976df36079d75244719a23956e1c9e2dcddc27
```
Continuum has:
- A local GGUF loader.
- A local inference backend.
- Mesh/federation infrastructure.
- Roadmap language about dividing models across nodes.
No executable distributed GGUF layer or tensor data plane was found in this snapshot. There is no layer-range protocol, boundary-activation transport, distributed KV ownership, or distributed GGUF test corresponding to the roadmap statement.
Decision: exclude until source and real execution evidence exist.
## 9. GitHub search conclusion
The expanded searches used terms including:
- `distributed GGUF`.
- `sub-GGUF`.
- `layer range` with llama.cpp.
- `result_partial_hidden`.
- `activation chain`.
- llama.cpp RPC orchestration.
- GGUF tensor parallelism.
The working implementation families found are:
| Family | Projects | What is real |
|---|---|---|
| Coordinator-owned remote devices | llama.cpp RPC, historical GPUStack/llama-box, LocalAI integrations | Full GGML graph controlled by one coordinator |
| Independent GGUF layer pipeline | Nakshatra, prima.cpp | Local layer ownership, residual boundaries, local KV |
| Custom Rust layer pipeline | `llama-gguf` | gRPC layers, but coordinator-streamed weights and weak session semantics |
| Tensor parallel GGUF | LiGGUF SARA | Head/FFN partitions and per-layer partial reductions |
| Static/custom non-GGUF tensor engines | distributed-llama/dllama and similar | Useful algorithms, different artifacts/runtime |
| Whole-model routing | Chameleon, Ollama, most LocalAI use, current GPUStack backends | Independent complete models, not one split model |
| Roadmap-only mesh claims | Continuum and several search hits | No executable distributed GGUF data plane found |
No additional mature project was found that already combines:
- Arbitrary tracker-selected layer intervals.
- Exact local GGUF artifact ownership.
- Heterogeneous volunteer nodes.
- Concurrent route-session KV.
- Dynamic route epochs and recovery.
- Relay and cancellation.
- Exact runtime/activation compatibility.
- Worker-authenticated accounting.
Nakshatra is the closest and materially reduces the amount of new inference-engine work required.
## 10. Revised architecture decision
The selected runtime remains llama.cpp/GGML, but the implementation source priority changes:
```text
Existing Meshnet tracker, relay, sessions, capability admission, and billing
|
Meshnet-owned stable shard protocol
|
project-owned C++ worker borrowing narrow Nakshatra concepts/tests
|
narrow, pinned, architecture-certified llama.cpp patch
|
GGUF loader, quant kernels, KV, CPU/GPU backends
```
This is not an architectural pivot away from the initial decision. Nakshatra is an external implementation of nearly the same missing seam.
## 11. Revised implementation sequence
### 11.1 Reproduction and patch comparison
- Reproduce Nakshatra's two local workers with a small dense Llama-family GGUF.
- Verify disjoint sub-GGUF weight ownership.
- Verify boundary residual parity with whole-model llama.cpp.
- Rebase against the pinned current llama.cpp commit.
- Compare pre-sliced GGUF loading with range-aware loading from one source artifact.
### 11.2 Meshnet protocol adapter
- Keep the project-owned llama.cpp worker behind a supervised executable.
- Translate Meshnet request/session/route metadata into worker calls.
- Preserve BF16 or versioned named-tensor bundles on the public network boundary.
- Add exact source/slice/runtime recipe checks.
- Reject stale route epochs and incompatible caches.
### 11.3 Multi-session KV
- Map each route session/epoch to an isolated `llama_seq_id` or context.
- Test concurrent prefill and decode.
- Add bounded release, TTL, LRU, and cache-miss semantics.
- Verify only owned layers allocate KV.
### 11.4 Real heterogeneous route
- CPU plus AMD HIP/ROCm first on the available machine.
- Add CUDA, Vulkan, and Metal as certified lanes when hardware is available.
- Measure numerical drift and define compatibility classes.
- Require a clean from-token-zero restart when exact-recipe recovery is unavailable.
### 11.5 Trustworthy accounting
- Worker signs work receipts.
- Receipt binds route, request, artifact, range, activation digests, positions, and runtime recipe.
- Tracker validates structural consistency and completion evidence before rewards.
### 11.6 Architecture expansion
- Dense Llama-family first.
- Explicit Qwen3/Qwen3-MoE adapter next.
- Fail closed for all unvalidated architectures.
## 12. Local validation performed
The source audit included limited executable verification without downloading model artifacts:
```text
LiGGUF distributed target:
make ligguf-cpp-distrib
PASS — g++ produced ligguf-cpp-distrib
Nakshatra dependency-free focused tests:
47 passed in 0.42s
```
The Nakshatra subset covered receipt validation, wire-version behavior, wire handshake behavior, and topology ordering.
The networked idempotency integration test was not collected because this audit environment does not have the optional `grpcio` package installed. This is an environment dependency blocker, not a test assertion failure. No real Nakshatra model inference was run locally during this research; the real-model conclusions remain tied to the source, recorded experiment evidence, and the future reproduction gate in section 11.
## 13. Final conclusions
1. The user's central practical observation is correct: llama.cpp is the base of most real-world GGUF distribution found.
2. GPUStack 0.4 was open source and genuinely distributed, but through llama-box/llama.cpp RPC rather than independent shards.
3. GPUStack 2.0 removed distributed GGUF support, so the old tutorial must be treated as historical.
4. Nakshatra is the most important new source. It has already implemented and exercised the narrow llama.cpp layer-worker design.
5. Nakshatra should be approached for narrow collaboration and mined for source/tests, but its repository should not be adopted or forked wholesale.
6. Nakshatra still needs Meshnet-specific hardening: exact identity, concurrent KV, route epochs, cancellation, accounting, and architecture certification.
7. LiGGUF SARA is real distributed GGUF tensor parallelism, but every worker loads the whole model and communicates twice per layer.
8. Chameleon is whole-model routing; Continuum's distributed GGUF is roadmap-only in the audited snapshot.
9. No drop-in project yet satisfies the complete tracker-routed volunteer-network contract.
10. The implementation risk is now lower because the hardest first proof—partial llama.cpp layer loading plus boundary execution—has independent working source and cross-machine evidence.

829
docs/research/distributed-gguf-landscape.md Normal file
View File

@@ -0,0 +1,829 @@
# Distributed GGUF inference: existing projects, source audits, and implementation direction
Status: Research complete; architecture direction selected
Last updated: 2026-07-13
## 1. Purpose
This document records the research behind native distributed GGUF inference for the neuron-tai network. It is intentionally not a design for proxying an already-running whole-model `llama-server`, Ollama, vLLM, or another OpenAI-compatible runtime. Whole-model proxying is useful as a correctness/performance baseline, but it does not solve the network's central problem.
The required product is a tracker-routed distributed GGUF data plane in which heterogeneous nodes independently execute model layer ranges, exchange boundary activations, retain cache state for their own layers, and receive credit for work they actually perform.
### 1.1 Required contract
A satisfactory implementation must support:
- Native GGUF model artifacts and llama.cpp-compatible quantizations.
- Independently loaded, executable layer ranges on each node.
- Tracker-selected routes rather than one static rank topology.
- Heterogeneous CPU, CUDA, HIP/ROCm, Vulkan, Metal, and other certified lanes.
- A versioned hidden-state activation boundary between nodes.
- Local per-shard KV or recurrent state keyed by route session.
- Prefill and decode phases with bounded wire and compute cost.
- Overlap-safe execution using the tracker's effective `start_layer` semantics.
- Explicit cache miss, eviction, route epoch, and recovery behavior.
- Dynamic health and route failure handling.
- Per-node work telemetry and accounting.
- Model-agnostic infrastructure with architecture-specific certification.
### 1.2 Non-goals
- Reimplementing GGUF parsing, quantization kernels, tokenization, or mature CPU/GPU kernels from scratch.
- Treating a whole-model node as the distributed solution.
- Treating stock llama.cpp RPC as a safe volunteer-network protocol.
- Claiming every GGUF architecture works without a bounded real distributed validation.
- Mixing Transformers and GGUF shards in one route without an explicit compatible activation contract.
### 1.3 Parallelism taxonomy
These mechanisms must not be conflated:
- **Layer or pipeline parallelism** assigns whole contiguous transformer-layer ranges to different stages and transports boundary activations between them. This is the closest match to neuron-tai's tracker-routed shard contract.
- **Tensor parallelism** partitions operations or tensors within each layer, such as attention heads, matrix rows/columns, or experts. It usually requires all ranks for every layer and collective or reduction traffic inside every layer.
- **Local multi-device offload** places tensors or layers across devices controlled by one process. llama.cpp's `n_gpu_layers` and `tensor_split` belong here unless RPC devices are included.
- **llama.cpp RPC offload** exposes remote GGML devices to a coordinator-owned graph. It is cross-machine, but remote processes are devices rather than independent model/session workers.
- **Whole-model replication** runs one complete model per server and load-balances requests. It increases throughput and availability but does not allow one oversized model to span workers.
GGUF weight quantization such as Q4_K_M or Q8_0 reduces storage and memory pressure. It does not define the activation dtype, compute dtype, KV-cache dtype, or distributed topology.
## 2. Existing neuron-tai infrastructure
The repository already contains most of the control plane needed by a GGUF shard worker.
### 2.1 Backend and execution path
There is no formal backend interface yet. Production code uses the concrete `TorchModelShard`, while several call sites rely on duck typing.
Relevant source:
- `packages/node/meshnet_node/model_backend.py:72-223` — result types, session cache, and concrete backend.
- `packages/node/meshnet_node/model_backend.py:226-345` — Torch/Hugging Face construction.
- `packages/node/meshnet_node/model_backend.py:347-566` — effective head, middle, tail, and whole-model backend methods.
- `packages/node/meshnet_node/model_backend.py:730-747` — Torch factory.
- `packages/node/meshnet_node/torch_server.py:302-317` — injected backend usage.
- `packages/node/meshnet_node/torch_server.py:717-766` — whole-model generation fast path.
- `packages/node/meshnet_node/torch_server.py:1464-1514` — server construction.
- `packages/node/meshnet_node/torch_server.py:1638-1659` — factory hard-wired to `load_torch_shard`.
The current effective distributed backend contract includes:
- `model_id`, `shard_start`, `shard_end`, `total_layers`, `is_head`, `is_tail`, and `device`.
- `encode_prompt`, `encode_next_token`, `forward_bytes`, and `decode_tail_token`.
- `eos_token_ids` and `release_session`.
- Whole-model generation and token-count helpers.
A GGUF design should separate a generic backend/process lifecycle interface from an optional activation-shard interface rather than making every backend pretend to be `TorchModelShard`.
### 2.2 Distributed activation wire
The current data plane already provides a reusable envelope:
- Binary `POST /forward` requests.
- `X-Meshnet-Session` route-session identity.
- `X-Meshnet-Cache: prefill | decode`.
- `X-Meshnet-Past-Len` cache consistency check.
- `X-Meshnet-Start-Layer` overlap-safe execution.
- Explicit tensor shape, dtype, position IDs, attention mask, chunk, and compression metadata.
- Direct and relay downstream clients owned by the generation handler.
- HTTP 409 cache-miss responses and re-prefill recovery.
Relevant source:
- `packages/node/meshnet_node/server.py:13-76` — wire contract and validation.
- `packages/node/meshnet_node/torch_server.py:497-635` — binary handler.
- `packages/node/meshnet_node/torch_server.py:974-1288` — route parsing and hop execution.
- `packages/tracker/meshnet_tracker/server.py:3635-3781` — route planning and effective start layers.
- `docs/adr/0008-binary-activation-wire-format.md`.
- `docs/adr/0012-start-layer-overlapping-shards.md`.
The current payload semantics are still Torch/Hugging Face-specific:
- Boundary dtype is fixed to BF16.
- Tensor conversion imports Torch.
- Attention masks and position IDs use Torch-oriented serialization.
- Tail-local decode reconstructs a Torch tensor.
The HTTP envelope is reusable; backend-neutral activation semantics need to be made explicit.
### 2.3 Local shard cache
The existing Transformers path already has the desired session behavior:
- One stable UUID per distributed generation.
- Prefill establishes state on every shard.
- Decode forwards only the new-token activation.
- Every shard stores only its own layer state.
- Cache lookup checks sequence length and effective start layer.
- Cache miss, restart, or route mismatch returns HTTP 409.
- The head re-prefills accumulated tokens after a miss.
- TTL and LRU bound local memory.
Relevant source:
- `packages/node/meshnet_node/model_backend.py:102-193`.
- `packages/node/meshnet_node/model_backend.py:334-345`.
- `packages/node/meshnet_node/model_backend.py:595-662`.
- `packages/node/meshnet_node/torch_server.py:806-944`.
- `docs/adr/0022-sharded-per-node-kv-cache.md`.
A GGUF worker should preserve this product-level contract while mapping route sessions to llama.cpp sequence IDs or isolated contexts.
### 2.4 Tracker and capability admission
The tracker already supports:
- Model and layer-range registration.
- Capability reports and fail-closed admission.
- Heterogeneous route candidates.
- Greedy interval coverage.
- Effective start-layer injection.
- Throughput/load/reputation-informed selection.
- Dynamic health and route statistics.
- Accounting and work attribution.
Relevant source:
- `packages/tracker/meshnet_tracker/server.py:592-942`.
- `packages/tracker/meshnet_tracker/server.py:3635-3781`.
- `packages/tracker/meshnet_tracker/server.py:4425-4595`.
- `packages/tracker/meshnet_tracker/server.py:6200-6285`.
- `docs/adr/0021-dynamic-statistical-routing.md`.
- `docs/adr/0023-model-agnostic-node-capability-admission.md`.
GGUF gaps include:
- The closed `bfloat16`, `int8`, `nf4` precision vocabulary.
- No first-class route kind or activation compatibility key.
- No GGUF artifact descriptor.
- No llama.cpp/Vulkan/HIP execution recipe.
- No model-aware GGUF memory estimator.
- No real llama.cpp doctor benchmark.
## 3. Ecosystem survey
No mature project was found that combines native GGUF, arbitrary tracker-selected layer ranges, heterogeneous cross-machine execution, independent local shard cache, dynamic route sessions, recovery, and per-node accounting.
A later GitHub follow-up found that [Nakshatra](https://github.com/fthrvi/nakshatra) already implements the narrow patched-llama.cpp Llama-family layer-worker seam and should be treated as the primary source donor and collaboration candidate. Its repository is not self-contained or production-ready enough to adopt as the runtime base. Historical GPUStack/llama-box and LiGGUF SARA were also audited in [the GitHub follow-up](distributed-gguf-github-followup.md). This reduces implementation risk but does not provide a drop-in Meshnet runtime.
### 3.1 Candidate summary
| Project | Native GGUF | Distributed form | Direct fit | Decision |
|---|---:|---|---|---|
| [llama.cpp](https://github.com/ggml-org/llama.cpp) | Yes | Device offload/RPC, local multi-GPU | Best kernel, wrong stock topology | Use as primary kernel through a small pinned fork |
| [Nakshatra](https://github.com/fthrvi/nakshatra) | Yes, via sub-GGUFs and patched llama.cpp | Independent contiguous layer workers over gRPC | Closest implementation found | Primary source donor/collaboration candidate; do not adopt its repository wholesale |
| [llama-gguf](https://github.com/Lexmata/llama-gguf) | Yes | gRPC layer pipeline | Architecturally close, immature custom runtime | Source donor only |
| [prima.cpp](https://github.com/OpenCPIL/prima.cpp) | Yes | Static piped-ring layer windows | Proves partial loading and local KV | Source donor only |
| [LiGGUF](https://github.com/matrixsmaster/ligguf) | Yes | SARA head/FFN tensor sharding and activation reduction | Every rank loads the full model; static master/workers | Compact tensor-parallel source donor only |
| [mistral.rs](https://github.com/EricLBuehler/mistral.rs) | Yes | NCCL TP and multi-machine ring | Static/homogeneous distributed assumptions | Evaluate as optional homogeneous backend |
| [vLLM](https://github.com/vllm-project/vllm) | Experimental/plugin | TP, PP, DP, EP via Ray/Torch distributed | Production clusters, not volunteer layer routes | Whole-model/managed-cluster lane and design donor; see [vLLM assessment](vllm-distributed-gguf-assessment.md) |
| [distributed-llama](https://github.com/b4rtaz/distributed-llama) | No, custom format | Root/worker tensor parallelism | Power-of-two/static topology | Ideas only |
| [Petals](https://github.com/bigscience-workshop/petals) | No | Internet layer pipeline | Strong cache/session semantics | Conceptual donor |
| [exo](https://github.com/exo-explore/exo) | No native path | MLX tensor/pipeline sharding | MLX-centric and platform-asymmetric | Placement/cache ideas only |
| [LocalAI](https://github.com/mudler/LocalAI) | Via llama.cpp | RPC workers and request routing | Control-plane overlap | Lifecycle ideas only |
| [GPUStack](https://github.com/gpustack/gpustack) 0.4-0.7 | Via llama-box/RPC | Scheduler + primary/RPC workers | Proven RPC orchestration, not layer workers | Historical reference; GGUF distribution removed in GPUStack 2.0 |
| [llama-box](https://github.com/gpustack/llama-box) | Yes | llama.cpp RPC | Archived | Historical source donor only |
| [Chameleon](https://github.com/megeezy/Chameleon) | Via whole-model backends | Whole-model worker lifecycle/routing | No single-model sharding | Exclude from distributed-GGUF candidates |
| [Continuum](https://github.com/CambrianTech/continuum) | Local loader/backend | Mesh orchestration; tensor distribution is roadmap text | No distributed GGUF execution path found | Exclude until executable evidence exists |
| Ollama | Yes | Local multi-device; no cross-host model split | Whole-model only | Proxy baseline only |
| MLC LLM | No native GGUF | Compiled local/multi-GPU | Different artifact/runtime | Separate recipe at most |
### 3.2 llama.cpp RPC
Stock RPC exposes remote GGML devices to one coordinator. Current upstream documentation says it can distribute weights and KV across local and remote devices and can cache remote tensors with `ggml-rpc-server -c`. This makes the original claim in `docs/adr/0001-pytorch-over-llama-cpp.md`—that every launch must always resend all weights—outdated.
RPC still does not provide:
- Independently loaded local GGUF layer workers.
- Tracker-selected per-request routes.
- A hidden-state HTTP endpoint.
- Route-session-owned shard cache.
- Per-worker tracker accounting.
- A safe untrusted volunteer-node boundary.
Upstream explicitly calls RPC proof-of-concept, fragile, and insecure. It is not the target architecture.
### 3.3 Petals
Petals remains the strongest conceptual reference for:
- Hosting independent transformer block ranges.
- Per-server cache state.
- Session-based inference.
- Rebuilding cache after route failure.
- Capacity-aware block placement.
- Public/private swarm control.
It is PyTorch/Transformers-based, not GGUF, and is no longer active enough to adopt as the runtime. The existing neuron-tai Transformers path already implements many of its core semantics.
## 4. Source audit: prima.cpp
Audited source snapshot:
```text
OpenCPIL/prima.cpp
commit 6f9b7c40962d777d1726456b4359340d932bef12
```
### 4.1 Decision
Do not fork prima.cpp wholesale. Extract partial-loading, local-KV, graph-boundary, and placement ideas into a small fork of current llama.cpp.
prima.cpp is a full invasive llama.cpp fork. Its exact upstream ancestry could not be established from the available repository history. Distributed changes are embedded in public structures, `src/llama.cpp`, common CLI code, and the old server.
### 4.2 Layer ownership
prima.cpp adds static rank topology fields:
- `n_world`.
- `rank`.
- Fixed `n_layer_window[32]` arrays.
- Cycle count and fixed network addresses/ports.
Evidence:
- `include/llama.h:292-353`.
- `src/llama.cpp:2597-2636`.
- `common/arg.cpp:680-785`.
Layer ownership is a repeated cyclic window, not an arbitrary tracker-provided `[start,end)` route:
- `src/llama.cpp:3838-3883`.
- `src/llama.cpp:16941-16951`.
This cannot directly represent dynamic tracker routes.
### 4.3 Partial local GGUF loading
This is prima.cpp's strongest reusable contribution.
- Rank zero creates token embedding, final norm, and output tensors.
- Transformer tensors are created only when the layer belongs to the process.
- Global layer IDs are compacted into local storage.
- Unneeded GGUF tensors are erased before mapping/loading.
- Only selected file regions are mapped and loaded.
Evidence:
- `src/llama.cpp:7500-7619`.
- `src/llama.cpp:9358-9515`.
This proves a llama.cpp-derived process can independently load only the GGUF tensors required by its layer range.
### 4.4 Graph boundaries
prima.cpp introduces explicit intermediate input/output tensors and skips unowned layers:
- `src/llama.cpp:10793-10813`.
- `src/llama.cpp:11000-11117`.
- `src/llama.cpp:16953-17010`.
The distributed graph path is hard-asserted to Llama or Qwen2 despite inherited architecture enums. Architecture support is therefore narrow.
### 4.5 Activation transport
Transport is raw ZeroMQ multipart PUSH/PULL. Messages include native shapes and raw contiguous F32 activation bytes.
Evidence:
- `src/llama.cpp:18031-18077`.
- `src/llama.cpp:18542-18563`.
It has no protocol version, model/session/route identity, layer range, dtype field, checksum, authentication, compression, or robust payload validation. It is incompatible with the existing Meshnet BF16 route-session protocol.
### 4.6 Local KV
Each process has one local llama.cpp context and allocates KV only for layers assigned to that process.
Evidence:
- `src/llama.cpp:3889-3969`.
- `src/llama.cpp:18268-18269`.
This is valuable physical behavior, but cache operations are coupled to one static ring and server slot numbering. There is no route session, route epoch, model fingerprint, range identity, lease, or idempotency contract.
### 4.7 Fault model and backends
Runtime sends and receives are blocking. Socket failures can terminate or stall a process. There is no runtime route repair, KV replay, deduplication, or accounting.
The inherited tree contains many backends, but the distributed profiler and placement solver primarily model CPU, CUDA, and Metal. Project documentation excludes AMD/Vulkan distributed support.
Evidence:
- `src/llama.cpp:18031-18077`.
- `src/llama.cpp:20492-20769`.
- `common/profiler.h:329-405`.
- `common/common.cpp:850-1183`.
- `README.md:362-368`.
### 4.8 Tests
No meaningful distributed regression suite was found for layer windows, transport, cache propagation, heterogeneous placement, session isolation, or failure behavior.
### 4.9 Reusable parts
Port or use as design references:
- Selective endpoint/head/tail tensor creation.
- Selected-layer GGUF mapping and loading.
- Global-to-local layer indexing.
- Per-layer KV filtering and allocation.
- Boundary residual input/output.
- Placement cost ideas and page prefetching.
Do not reuse:
- Static ring topology.
- Fixed rank arrays.
- ZeroMQ wire protocol.
- Server patches.
- Cache-command propagation.
- Failure semantics.
## 5. Source audit: llama-gguf
Audited source snapshot:
```text
Lexmata/llama-gguf
commit 6e9f194206450080d47101c6f88a80f604ce69be
```
### 5.1 Decision
Do not adopt `llama-gguf` as the inference runtime or distributed data plane. Use it as a source donor for protobuf organization, health/capability messages, explicit ranges, and synthetic multi-process tests.
### 5.2 Coordinator owns and streams the model
The coordinator loads and builds the entire GGUF model, then serializes layer tensors and streams them to shards over gRPC.
Evidence:
- `src/distributed/coordinator.rs:44-53`.
- `src/distributed/coordinator.rs:160-168`.
- `src/distributed/coordinator.rs:196-299`.
Workers explicitly do not open local GGUF files:
- `src/distributed/shard.rs:53-59`.
This conflicts with independently owned local artifacts, startup efficiency, and tracker-controlled model distribution.
### 5.3 Explicit ranges but simple coverage validation
The cluster config supports explicit half-open ranges or even auto-partitioning:
- `src/distributed/config.rs:65-92`.
- `src/distributed/config.rs:168-220`.
Manual validation checks total assigned layer count but does not prove sorted, gap-free, non-overlapping coverage. It is a static startup configuration, not a dynamic per-request route.
### 5.4 Shard-local KV, but only one global sequence
A worker allocates one KV cache for its local layer count:
- `src/distributed/shard.rs:30-44`.
- `src/distributed/shard.rs:261-286`.
- `src/model/mod.rs:62-117`.
There is no cache map or route-session identity. `ResetKvCache` takes an empty request and resets the one global cache:
- `src/distributed/shard.rs:447-458`.
- `proto/distributed.proto:15-16`.
- `proto/distributed.proto:95-103`.
Concurrent sessions, cache epochs, stale requests, and per-request eviction are unsupported.
### 5.5 Forward protocol
`ForwardRequest` contains only hidden state, position, and sequence length:
- `proto/distributed.proto:77-93`.
It has no model fingerprint, route/session/request identity, layer range, phase, chunk, expected cache length, accounting identity, idempotency key, or route epoch.
The tensor envelope does include shape, dtype, little-endian bytes, and a name:
- `proto/distributed.proto:36-46`.
- `src/distributed/tensor_transfer.rs:10-115`.
The DType enum can represent BF16, but the distributed execution path and tests use F32 hidden states and F32 KV.
### 5.6 Token-by-token serialized execution
The distributed model loops over tokens one at a time and sends each hidden vector through each shard. The complete pipeline is protected by one mutex.
Evidence:
- `src/distributed/model.rs:21-39`.
- `src/distributed/model.rs:86-147`.
- `src/distributed/pipeline.rs:45-113`.
This prevents efficient batched prefill and safe session multiplexing.
### 5.7 Architecture claims versus shard reconstruction
The repository has a large architecture enum, but the distributed shard reconstructs a simple dense layer consisting of RMSNorm, ordinary attention, and dense gated FFN.
Evidence:
- `src/model/architecture.rs:5-155`.
- `src/distributed/shard.rs:154-223`.
The worker cannot reconstruct many architecture-specific layer forms such as MoE, hybrid/recurrent layers, shared cache, architecture-specific norms, or specialized projections. Qwen3 MoE support cannot be inferred from the enum list.
### 5.8 Backends
Shard backend selection tries CUDA, Metal, DX12, Vulkan, then CPU, but the source notes GPU model-weight preloading is skipped because weights arrive as gRPC tensors. The per-operation backend path is not equivalent to llama.cpp's mature graph scheduling.
Evidence:
- `src/distributed/shard.rs:53-115`.
- `src/backend/mod.rs:25-260`.
### 5.9 Fault recovery
The repository has health monitoring and can reconnect, reconfigure, and resend layers:
- `src/distributed/fault.rs:83-360`.
It does not restore active KV, route position, partial prefill, session ownership, or accounting state. Reloading a process is not active-generation recovery.
### 5.10 Tests
Distributed integration tests start local CPU gRPC shards with synthetic dense F32 weights. They verify configure, load, forward, reset, and a two-shard pipeline.
The file explicitly states that no real GGUF is loaded:
- `tests/distributed_integration_test.rs:1-7`.
Missing evidence includes real quantized GGUF parity, heterogeneous devices, BF16 boundaries, batched prefill, concurrent sessions, worker loss, route epochs, local artifact loading, and cache recovery.
### 5.11 Reusable parts
Potential donors:
- Protobuf tensor shape/dtype/data envelope.
- Explicit half-open layer ranges.
- Health and capabilities messages.
- Synthetic multi-process shard test harness.
- Separation of configure, load, forward, reset, and health methods.
Do not reuse:
- Coordinator-side weight streaming.
- Custom inference engine as the production kernel.
- Single global KV cache.
- Token-by-token pipeline.
- Dense-only layer reconstruction.
- Fault-reload semantics.
- The all-reduce implementation, which currently echoes its input at `src/distributed/shard.rs:522-541`.
## 6. Source audit: current llama.cpp
Audited source snapshot:
```text
ggml-org/llama.cpp
commit 91c631b21d6e5d09e9c6659efdf6baeef5a44ddb
```
### 6.1 Decision
The smallest credible implementation is a pinned, architecture-certified layer-worker fork of current llama.cpp. A fully generic all-architecture patch is not credible as the first implementation.
### 6.2 Layer-range API
Prefer a new range loader that isolates ABI churn:
```c
llama_model * llama_model_load_range(
const char * path,
llama_model_params params,
int32_t il_start,
int32_t il_end);
```
Relevant source:
- `include/llama.h:295-331`.
- `src/llama-model.cpp:2300-2319`.
Adding fields directly to the by-value public parameter structure changes ABI. A project-owned API around internal model state is safer for a pinned worker executable.
### 6.3 Partial weight loading
Current loader behavior already makes a small patch possible:
- Architectures create/register tensors.
- Backend buffers are allocated only for created tensor contexts.
- Mmap ranges derive from created tensors.
- `done_getting_tensors(partial=true)` exists.
- Data loading only processes tensors present in created contexts.
Relevant source:
- `src/llama-model.cpp:1229-1647`.
- `src/llama-model-loader.cpp:1054-1572`.
For an initial Llama-family adapter:
- Create token embedding only if `il_start == 0`.
- Create final norm/output only if `il_end == n_layer`.
- Create repeating tensors only for `[il_start, il_end)`.
- Finish model loading with partial mode.
Relevant source:
- `src/models/llama.cpp:34-92`.
- `src/llama-model.cpp:1490`.
### 6.4 Residual input
Current input machinery supports F32 supplied embeddings:
- `src/llama-graph.h:120-150`.
- `src/llama-graph.cpp:66-121`.
- `src/llama-graph.cpp:2151-2229`.
A middle shard needs a dedicated residual input that bypasses token lookup, embedding scaling, LoRA, and padding behavior while still accepting position and sequence IDs for RoPE/KV.
### 6.5 Ranged graph execution
The initial Llama graph loop is in:
- `src/models/llama.cpp:98-247`.
It must:
- Iterate only `[il_start, il_end)`.
- Preserve all boundary rows for intermediate shards.
- Avoid final-token-only row pruning except on the actual tail.
- Run final norm and LM head only on the actual tail.
### 6.6 Boundary output
The network boundary is the unnormalized residual stream after the final owned transformer layer, not llama.cpp's final embedding tensor.
Add a dedicated graph result such as `t_boundary` in:
- `src/llama-graph.h:791-865`.
- `src/llama-graph.cpp:1190-1251`.
- `src/llama-context.cpp:2206-2231`.
Existing `llama-ext` layer-input extraction is useful precedent but is explicitly staging/WIP and only populated by some architectures.
### 6.7 Shard-local KV
The standard KV cache already supports a layer filter and compact global-to-local mapping:
- `src/llama-kv-cache.cpp:64-100`.
- `src/llama-kv-cache.cpp:163-248`.
- `src/llama-kv-cache.cpp:1210-1327`.
- `src/llama-model.cpp:2152-2266`.
For a standard Llama adapter, the filter is conceptually:
```cpp
return il >= il_start && il < il_end;
```
Do not apply this blindly to hybrid, recurrent, shared-cache, MLA, or other architecture-specific memory implementations.
### 6.8 Route-session mapping
Existing sequence APIs support:
- Sequence removal, copy, keep, and position operations.
- Sequence state get/set.
- Partial/on-device state flags.
Relevant source:
- `include/llama.h:720-913`.
- `src/llama-context.cpp:2900-2968`.
- `src/llama-context.cpp:3978-4024`.
A shard worker can map `(route_session_id, route_epoch)` to a stable `llama_seq_id` or one isolated context. Because the model contains only filtered layers, sequence state naturally represents shard-local cache.
### 6.9 Scheduler
No scheduler rewrite should be required. The current graph scheduler can allocate a new residual input and boundary output if they are registered correctly.
Relevant source:
- `src/llama-context.cpp:1286-1355`.
- `src/llama-context.cpp:2324-2459`.
### 6.10 Architecture coupling
A fully generic minimal patch is not plausible:
- The audited tree has roughly 136 model implementation files.
- More than one hundred contain architecture-specific layer loops.
- Architectures differ in cache, residual, attention, MoE, recurrent, multimodal, and output behavior.
Generic infrastructure is appropriate for:
- Range validation.
- Tensor filtering.
- Residual input/output plumbing.
- Standard KV filtering.
- C ABI and worker protocol.
Each supported architecture still needs a reviewed range-aware adapter and real distributed validation. Unsupported architectures must fail closed.
## 7. Selected architecture
```text
Existing tracker and accounting
|
Existing versioned activation/relay protocol
|
Project-owned standalone C++ GGUF shard worker
(borrowing or jointly maintaining narrow Nakshatra patch concepts where practical)
|
Pinned llama.cpp fork with small architecture-specific patch series
|
GGUF loader, model graphs, quant kernels, KV, CPU/GPU backends
```
Nakshatra's working patch and daemon should be reproduced, rebased, and used as source/test evidence before equivalent code is independently designed. Collaboration on narrow upstreamable llama.cpp hooks is preferable to duplicate patch families, but the project should keep its own small pinned fork and standalone worker rather than fork Nakshatra's whole Petals-derived repository. The project-owned boundary remains necessary because Nakshatra's build, control plane, protobuf, sub-GGUF artifact model, single-session daemon, and accounting semantics do not satisfy the full Meshnet contract.
### 7.1 Responsibility split
Python/node agent remains responsible for:
- Tracker registration and heartbeat.
- Capability proof and admission.
- Artifact distribution and verification.
- Route selection and effective start layers.
- Activation transport and relay.
- Route sessions, epochs, cache-miss recovery, and process supervision.
- Work telemetry and accounting.
The C++ worker is responsible for:
- Loading exactly one model/range/recipe.
- Token embedding on a head shard.
- Boundary activation input on middle/tail shards.
- Executing only owned layers.
- Returning boundary activations on non-tail shards.
- Final norm/logits/sampling or token output on a tail shard.
- Local KV/recurrent state by mapped sequence.
- Exporting health, cache, load, compute, and memory metrics.
### 7.2 Stable worker boundary
Do not expose `ggml_tensor *`, scheduler objects, or llama.cpp structs to Python. Prefer a supervised executable with a small project-owned API, for example:
```text
load_layer_range
prefill_tokens
prefill_activation
decode_token
decode_activation
get_boundary
get_logits_or_token
release_session
export_session
import_session
health
metrics
```
The wire carries at least:
```text
protocol version
request/work ID
route session ID
route epoch
model/artifact fingerprint
layer begin/end
effective start layer
prefill/decode phase
token start/count
hidden shape and dtype
payload length/checksum
```
### 7.3 Battle-proven transport and concurrency
The implementation program chooses gRPC over HTTP/2 with Protocol Buffers for the standalone Python/C++ Shard data plane rather than inventing a raw socket protocol.
- One long-lived bidirectional stream serves one Route Session Activation Seam.
- HTTP/2 supplies connection reuse and flow control; gRPC supplies deadlines, cancellation, status, TLS hooks, and generated Python/C++ schemas.
- Large prefill tensors are split into bounded frames; decode uses a small fast path.
- Existing relay/WebSocket infrastructure may transport the same protobuf frames as opaque binary when direct gRPC reachability is unavailable.
- OpenAI-facing HTTP/SSE and existing Tracker APIs remain unchanged.
The public payload is a versioned named-tensor bundle, not one anonymous activation, because architecture boundaries such as vLLM's Llama/Qwen3-MoE pipeline stages can require both `hidden_states` and `residual`.
Concurrency is implemented with local llama.cpp sequences or bounded contexts mapped from `(Route Session ID, route epoch)`. Compatible active decode steps are continuously batched inside each node. This adapts the useful vLLM scheduling concept without importing vLLM's Torch process groups, PagedAttention allocator, or static distributed executor.
Tensor/expert parallel collectives remain confined to a future trusted composite-node or managed-cluster provider. The public volunteer primitive remains contiguous layer Shards.
The benchmark-gated execution plan and Ralph backlog live in [the active distributed-GGUF feature](../../.scratch/distributed-gguf-runtime/README.md).
## 8. Implementation spikes and acceptance gates
### 8.1 Spike 1: reproduce and rebase Nakshatra
Before writing a separate worker, reproduce Nakshatra's dense Llama-family two-worker path and rebase its narrow patch onto the exact current llama.cpp commit selected by this project.
Run two local workers with disjoint sub-GGUF ranges, then compare sub-GGUF loading with a range-aware loader over one shared source artifact.
Acceptance:
- The audited Nakshatra test is independently reproducible rather than accepted from repository claims.
- Each process maps only its assigned tensors.
- Each process allocates KV only for assigned layers.
- Head/middle/tail module ownership is correct.
- Boundary residuals produce bounded numerical error against whole-model llama.cpp.
- The patch rebases cleanly onto the pinned current llama.cpp baseline.
- Reusable worker code is isolated from Nakshatra's Petals-derived and project-specific control plane.
- Upstream collaboration or shared maintenance is evaluated before duplicating the patch family.
### 8.2 Spike 2: Meshnet protocol and multi-session KV
Place the rebased/Nakshatra-derived worker behind Meshnet's route/session protocol.
Acceptance:
- Multiple route sessions remain isolated through separate `llama_seq_id` values or contexts.
- Duplicate requests are idempotent and stale route epochs are rejected.
- Exact source/slice artifact hashes and runtime recipes are checked.
- Cache miss, release, TTL/LRU eviction, cancellation, and re-prefill are bounded.
- Killing a worker produces a bounded failure, not a deadlock.
- Work receipts bind worker identity, request, route, range, artifact, and activation evidence.
### 8.3 Spike 3: heterogeneous two-machine route
Use a tracker-selected two-machine route with real CPU/GPU execution.
Measure:
- Cold and warm model load.
- Mapped tensor and KV memory per process.
- Prefill and decode throughput.
- Activation bytes and boundary latency.
- Whole-model parity.
- Process/node failure behavior.
- Per-node completed work and compute time.
Synthetic tests remain unit coverage, not distributed validation.
### 8.4 Spike 4: Qwen3 MoE adapter
Qwen3 30B-A3B requires an explicit architecture adapter. It must review:
- Expert/router tensor ownership.
- Top-k routing and MoE graph behavior.
- Architecture-specific normalization and Q/K normalization.
- Layer-local expert loading.
- KV type and layout.
- Boundary residual placement.
- Supported HIP/Vulkan/CPU paths.
The adapter should reuse current llama.cpp's Qwen3 graph rather than reconstructing a simplified layer as `llama-gguf` does.
### 8.5 Architecture certification
Each architecture recipe should declare:
- Range-aware graph implementation and version.
- Head/tail module rules.
- Cache kind and state support.
- Boundary dtype/layout.
- Supported backends.
- Required runtime commit.
- Real distributed validation evidence.
Capability admission keeps unsupported combinations registered-but-dark.
## 9. Principal risks
1. **Upstream churn:** llama.cpp model/graph/cache internals change frequently. Keep a small ordered patch series and pinned revisions.
2. **Architecture coupling:** every architecture adapter needs review and live proof.
3. **Cache locality:** dynamic rerouting requires replay/checkpoint semantics, not socket retry.
4. **Boundary compatibility:** shape, dtype, residual point, positions, and cache semantics must match exactly.
5. **Artifact identity:** exact GGUF/model/tokenizer/runtime fingerprints must be part of capability proof.
6. **Heterogeneous bottlenecks:** the slowest seam can dominate; placement must use measured end-to-end cost.
7. **Accounting integrity:** completed-work receipts need route and worker identity and must not trust self-reported latency alone.
8. **Security:** the worker accepts structured activations, not arbitrary GGML graphs or executable recipes.
9. **Memory estimates:** weight quantization and KV/state type must both be represented.
10. **Testing:** every supported lane needs real CPU/GPU tracker-routed acceptance evidence.
## 10. Final conclusion
Distributed GGUF should not be built from zero, but no existing project can be adopted as-is.
The reusable composition is:
- llama.cpp for mature GGUF parsing, architecture graphs, quantized kernels, backend scheduling, and KV/state APIs.
- prima.cpp for proof and reference implementations of selective local GGUF loading, local layer KV, graph boundaries, and placement ideas.
- `llama-gguf` for explicit range/protobuf/health/test-organization ideas.
- Petals and the existing neuron-tai Transformers path for route-session, cache-locality, and recovery semantics.
- The existing tracker, relay, capability, and accounting systems as the authoritative control plane.
The selected path is a small pinned llama.cpp layer-worker fork with generic shard infrastructure and explicitly certified architecture adapters.

657
docs/research/vllm-distributed-gguf-assessment.md Normal file
View File

@@ -0,0 +1,657 @@
# vLLM assessment for distributed GGUF inference
Status: Source and documentation assessment
Last updated: 2026-07-13
## 1. Question
Can vLLM or its GGUF plugin be reused to implement neuron-tai's primary distributed GGUF data plane?
The required data plane is not merely a whole-model serving cluster. It requires independently loaded GGUF layer ranges on heterogeneous nodes, tracker-selected per-request routes, a versioned activation boundary, local route-session cache, dynamic failure handling, relay support, and per-node work attribution.
This assessment also identifies vLLM components and concepts that remain useful even if its distributed runtime is not adopted.
## 2. Sources and snapshots
Source snapshots audited:
```text
vllm-project/vllm
commit 107a03ba63e005ff03424fed9c4e6cf551b98bb2
vllm-project/vllm-gguf-plugin
commit ad209df10bb1856ba53b6663745a831eb7eb09cc
```
Current official documentation reviewed:
- [GGUF](https://docs.vllm.ai/en/latest/features/quantization/gguf/).
- [Parallelism and Scaling](https://docs.vllm.ai/en/stable/serving/parallelism_scaling/).
- [Disaggregated Prefilling](https://docs.vllm.ai/en/latest/features/disagg_prefill/).
- [Data Parallel Deployment](https://docs.vllm.ai/en/latest/serving/data_parallel_deployment/).
- [Installation and supported platforms](https://docs.vllm.ai/en/latest/getting_started/installation/).
- [RFC #39583: Migrate bitsandbytes and GGUF quantization support to an out-of-tree plugin](https://github.com/vllm-project/vllm/issues/39583).
Both source projects are Apache-2.0 licensed.
## 3. Executive decision
### 3.1 Primary distributed GGUF runtime
Do **not** use vLLM as the primary tracker-routed heterogeneous GGUF layer-worker runtime.
Reasons:
- Current GGUF support is an experimental out-of-tree quantization plugin.
- The plugin converts GGUF tensors into Torch parameters and uses vLLM's PyTorch model implementations; it is not a lightweight llama.cpp/GGML runtime.
- The plugin builds CUDA/HIP Torch extensions and does not provide CPU, Vulkan, or Metal GGUF execution.
- vLLM pipeline parallelism assumes one static Torch distributed world with rank/process-group semantics.
- Pipeline workers communicate with neighboring ranks through Torch distributed or a trusted TCP/device communicator, not the Meshnet HTTP/relay/session protocol.
- Requests are scheduled as one synchronized engine, not independently routed through tracker-selected workers.
- Failure of a pipeline rank is a distributed-engine failure, not a route-local cache miss that can be recovered by rebuilding a route.
- Multi-node guidance requires identical environments and private trusted networking.
- External vLLM load balancing is across complete data-parallel model replicas, not model-layer workers.
### 3.2 Supported secondary roles
vLLM remains useful in three narrower roles:
1. **Whole-model node backend:** a node proxy can register an independent vLLM server for models/hardware that fit on that deployment. This is an execution recipe, not the core distributed GGUF feature.
2. **Static managed-cluster backend:** a separately managed, trusted vLLM TP/PP cluster can be exposed as one logical node. The tracker accounts at cluster boundary unless a trusted internal accounting bridge is added.
3. **Source/design donor:** architecture-aware pipeline boundaries, local layer construction, PagedAttention scheduling, KV connector lifecycles, and telemetry are valuable references.
### 3.3 GGUF kernel reuse
Do not import the vLLM GGUF plugin into the selected llama.cpp shard worker.
The plugin's GGUF kernels are tightly coupled to:
- PyTorch parameters and custom operators.
- vLLM quantization interfaces.
- Triton/CUDA/HIP execution.
- vLLM's model and tensor-parallel parameter layouts.
- vLLM release internals and monkey-patched plugin registration.
llama.cpp already provides broader GGUF quant coverage, CPU and edge backends, mmap loading, and the desired C/C++ runtime boundary.
## 4. Current GGUF support in vLLM
### 4.1 Support moved out of core
Official documentation states that GGUF support is:
- Highly experimental.
- Under-optimized.
- Potentially incompatible with other features.
- Now provided by the separate `vllm-gguf-plugin`.
The migration RFC gives the maintenance rationale. A vLLM maintainer estimated GGUF usage at roughly 0.1%, described poor performance relative to llama.cpp for batch-size-one consumer-GPU workloads, and identified a large special-case burden: legacy weight-loader branches, model-specific mappings, and roughly 6,000 lines of GGUF CUDA kernels. The out-of-tree plugin preserves availability while allowing vLLM core to prioritize its native production quantization paths. This is evidence that GGUF is a compatibility lane in vLLM, not its primary runtime focus.
At the audited snapshot, the plugin is itself young: version `0.0.4`, with its repository created during the 2026 migration. This is not a reason to reject it as a whole-model backend, but it raises compatibility and maintenance risk for using it as foundational distributed infrastructure.
The plugin registers itself through the `vllm.general_plugins` entry point:
- `vllm-gguf-plugin/pyproject.toml:17-18`.
- `vllm-gguf-plugin/vllm_gguf_plugin/plugin.py:109-127`.
It patches vLLM engine argument creation and speculative-model probing at runtime:
- `vllm-gguf-plugin/vllm_gguf_plugin/plugin.py:51-97`.
This demonstrates close coupling to vLLM internals and increases upgrade risk.
### 4.2 Dependencies and hardware assumptions
The plugin requires:
- `gguf>=0.17.0`.
- `vllm`.
- `torch>=2.9`.
- CUDA or ROCm toolkit for normal installation.
Evidence:
- `vllm-gguf-plugin/pyproject.toml:1-18`.
- `vllm-gguf-plugin/README.md:7-13`.
Its compiled extension is created through `torch.utils.cpp_extension.CUDAExtension`. ROCm is handled by compiling the same extension through HIP-compatible tooling:
- `vllm-gguf-plugin/setup.py:28-65`.
This gives a CUDA/ROCm lane, not the CPU/Vulkan/Metal hardware coverage expected from llama.cpp.
### 4.3 Model/config/tokenizer dependency
GGUF is treated as a weight source for a vLLM/Hugging Face model implementation.
The plugin determines model configuration from, in order:
- Explicit `hf_config_path`.
- A non-GGUF tokenizer path.
- The remote GGUF repository.
- The local GGUF parent directory.
Evidence:
- `vllm-gguf-plugin/vllm_gguf_plugin/plugin.py:34-48`.
- `vllm-gguf-plugin/vllm_gguf_plugin/plugin.py:51-79`.
Official docs recommend using the base Hugging Face tokenizer because converting a tokenizer from GGUF is slow and unstable. If Hugging Face cannot derive the architecture config, users must supply a compatible config path.
This is different from llama.cpp, which treats GGUF metadata, tensors, tokenizer, and architecture implementation as one native runtime artifact.
### 4.4 Artifact resolution
The plugin accepts:
- A local GGUF file.
- A local directory plus quantization type.
- A remote `repo_id:quant_type`.
- A remote `repo_id/filename.gguf`.
Evidence:
- `vllm-gguf-plugin/vllm_gguf_plugin/loader.py:41-67`.
- `vllm-gguf-plugin/vllm_gguf_plugin/weight_utils.py:18-70`.
It supports split GGUF artifacts named like `-00001-of-00004.gguf`:
- `vllm-gguf-plugin/vllm_gguf_plugin/weights_adapter/default.py:248-264`.
This is file sharding, not model-layer network sharding.
### 4.5 GGUF-to-vLLM name mapping
The default adapter:
1. Maps a GGUF architecture to Hugging Face tensor names through the Python `gguf` package.
2. Instantiates a Hugging Face model on the `meta` device to obtain its state-dict names.
3. Creates architecture-specific special mappings for MoE and other variants.
4. Maps those Hugging Face names into vLLM's packed parameter layout.
Evidence:
- `vllm-gguf-plugin/vllm_gguf_plugin/weights_adapter/default.py:42-231`.
- Qwen2/Qwen3 MoE mappings: `vllm-gguf-plugin/vllm_gguf_plugin/weights_adapter/default.py:81-98`.
- vLLM packed Llama mappings: `vllm/model_executor/models/llama.py:344-354`.
- vLLM packed Qwen3-MoE mappings: `vllm/model_executor/models/qwen3_moe.py:432-446`.
This provides broad model integration by relying on vLLM and Transformers model classes. It does not expose a generic independently executable GGUF layer object.
### 4.6 Loading behavior
`GGUFModelLoader` initializes an ordinary vLLM model on the target Torch device and calls its `load_weights` method with tensors yielded by the adapter:
- `vllm-gguf-plugin/vllm_gguf_plugin/loader.py:78-104`.
The GGUF iterator:
- Opens each GGUF file through `gguf.GGUFReader`.
- Enumerates every tensor.
- Converts NumPy-backed data to a Torch tensor.
- Emits an extra quantization-type parameter for quantized weights.
Evidence:
- `vllm-gguf-plugin/vllm_gguf_plugin/weight_utils.py:73-135`.
Pipeline stages instantiate only their repeating local layers, so vLLM's `AutoWeightsLoader` can skip parameters represented by `PPMissingLayer`. However, the plugin still enumerates the artifact through Python on each worker. This is not the same as selectively registering only a layer range and mmaping only its file regions in llama.cpp.
### 4.7 Quantization types and kernels
The plugin declares support for:
- Unquantized F32, F16, BF16.
- Q4_0, Q4_1, Q5_0, Q5_1, Q8_0, Q8_1.
- Q2_K, Q3_K, Q4_K, Q5_K, Q6_K.
- IQ1_M, IQ1_S, IQ2_XXS, IQ2_XS, IQ2_S, IQ3_XXS, IQ3_S, IQ4_XS, IQ4_NL.
Evidence:
- `vllm-gguf-plugin/vllm_gguf_plugin/quantization/utils.py:46-75`.
Linear execution chooses among:
- Native unquantized matrix multiplication.
- Quantized matrix-vector kernels for small token counts.
- Quantized matrix-matrix kernels for larger token counts.
- Full dequantization followed by Torch matrix multiplication as fallback.
Evidence:
- `vllm-gguf-plugin/vllm_gguf_plugin/quantization/linear.py:34-57`.
- `vllm-gguf-plugin/vllm_gguf_plugin/quantization/linear.py:79-247`.
There are separate Triton and compiled CUDA/HIP implementations for GEMM, dequantization, embedding, and fused MoE.
These kernels are useful to the vLLM ecosystem but are not portable drop-in components for a llama.cpp/GGML worker.
### 4.8 Test evidence
Plugin tests compare GGUF output against unquantized vLLM output for small models including Qwen2.5, Qwen3 dense, Phi3, GPT-2, StableLM, Gemma3, and OLMoE:
- `vllm-gguf-plugin/tests/test_gguf_generation.py:22-79`.
- `vllm-gguf-plugin/tests/test_gguf_generation.py:177-226`.
There is a two-GPU tensor-parallel GGUF test for Qwen3-0.6B:
- `vllm-gguf-plugin/tests/test_gguf_generation.py:229-283`.
The core vLLM plugin test also covers Qwen3 dense and OLMoE at TP=1 and TP=2:
- `tests/plugins_tests/gguf/test_gguf_plugin_generate.py:27-132`.
No source test was found for:
- GGUF pipeline parallelism.
- Multi-node GGUF PP.
- Mixed hardware.
- Qwen3 30B-A3B GGUF.
- Tracker-selected ranges.
- Independent route sessions or dynamic route recovery.
Kernel tests exercise CUDA tensors and broad quant types. One variable-length batching kernel test is explicitly skipped because the current CUDA kernel does not support that case:
- `vllm-gguf-plugin/tests/test_kernels.py:260-310`.
## 5. vLLM pipeline parallelism
### 5.1 What it gets right
vLLM has a mature architecture-aware pipeline abstraction:
- Model implementations construct only repeating layers assigned to the local PP rank.
- Missing layers are represented with `PPMissingLayer` placeholders.
- First-stage token embeddings and last-stage norm/head can be conditionally owned.
- Non-tail stages return named intermediate tensor bundles.
- The scheduler and model runner overlap stage communication and execution.
- Uneven layer-count splits are supported.
This is substantially more mature than the static dense-layer reconstruction in `llama-gguf`.
### 5.2 Static stage partitioning
`make_layers` derives `(start_layer, end_layer)` from PP rank and PP world size, creates real modules only for that interval, and fills all other positions with `PPMissingLayer`:
- `vllm/model_executor/models/utils.py:674-719`.
The default partition is approximately even. A static `VLLM_PP_LAYER_PARTITION` environment variable can override the number of layers per rank:
- `vllm/distributed/utils.py:127-172`.
This can represent uneven contiguous ranges, but it remains a launch-time rank partition:
- One ordered process group.
- One fixed world size.
- One static set of adjacent ranks.
- No per-request tracker route.
- No arbitrary start/end assignment independent of rank.
- No overlapping shard ranges or Meshnet effective-start semantics.
### 5.3 Llama stage ownership
For Llama:
- The first rank owns token embedding.
- A tied output rank may also own embedding.
- Only the local repeating layers are instantiated.
- Only the last rank owns final RMSNorm.
- Only the last rank owns LM head and logits processor.
Evidence:
- `vllm/model_executor/models/llama.py:356-395`.
- `vllm/model_executor/models/llama.py:400-439`.
- `vllm/model_executor/models/llama.py:466-503`.
This is a good design reference for explicit head, middle, and tail ownership.
### 5.4 Intermediate tensor bundles
`IntermediateTensors` is a dictionary of named Torch tensors:
- `vllm/sequence.py:10-62`.
For Llama and Qwen3-MoE, the PP boundary includes both:
```text
hidden_states
residual
```
Evidence:
- `vllm/model_executor/models/llama.py:393-395`.
- `vllm/model_executor/models/llama.py:408-433`.
- `vllm/model_executor/models/qwen3_moe.py:478-518`.
This is an important architectural lesson. Depending on where the boundary is placed, an exact pipeline stage may require more than one tensor. Meshnet's backend-neutral activation envelope should support a named tensor bundle or deliberately choose a canonical boundary after residual fusion.
A universal single `hidden_states` tensor contract may silently break architecture parity.
### 5.5 Qwen3-MoE stage behavior
The audited Qwen3-MoE implementation supports:
- Q/K normalization.
- Routed experts.
- Shared experts.
- Expert/tensor parallel groups.
- Stage-local repeating layers.
- `hidden_states` plus `residual` PP boundaries.
Evidence:
- `vllm/model_executor/models/qwen3_moe.py:130-251`.
- `vllm/model_executor/models/qwen3_moe.py:254-354`.
- `vllm/model_executor/models/qwen3_moe.py:357-429`.
- `vllm/model_executor/models/qwen3_moe.py:432-524`.
However, in this source snapshot, Qwen3-MoE constructs embedding, final norm, and LM head without the first/last-rank guards used by Llama:
- Embedding: `vllm/model_executor/models/qwen3_moe.py:463-471`.
- Final norm: `vllm/model_executor/models/qwen3_moe.py:477`.
- LM head/logits: `vllm/model_executor/models/qwen3_moe.py:566-589`.
Repeating layer weights remain stage-local, but endpoint parameters appear replicated. This source observation should be validated in a real PP load before using vLLM memory behavior as a reference for Qwen3 30B-A3B.
### 5.6 Stage communication
The worker receives intermediate tensor dictionaries from the previous PP rank, executes the local model, and asynchronously sends output tensors to the next rank:
- `vllm/v1/worker/gpu_worker.py:1045-1088`.
The PP group:
- Sends metadata through a CPU process group.
- Sends tensors through Torch distributed device or CPU groups.
- Defaults destination to the next static rank.
- Defaults source to the previous static rank.
- Can optimize TP-group slices with all-gather reconstruction.
Evidence:
- `vllm/distributed/parallel_state.py:960-1053`.
- `vllm/distributed/parallel_state.py:1055-1149`.
A stateless coordinator variant uses a trusted TCP store and optional device communicator:
- `vllm/distributed/stateless_coordinator.py:300-356`.
This is not a versioned application protocol. It has no Meshnet route session, route epoch, model fingerprint, layer range, request work receipt, relay semantics, authentication, or HTTP cache-miss contract.
### 5.7 Static distributed world
Official vLLM deployment guidance recommends:
- Tensor parallelism within a multi-GPU node.
- Pipeline parallelism across nodes.
- Ray or multiprocessing as the distributed executor.
- Identical model paths, packages, and execution environments across all nodes.
- High-speed networks such as InfiniBand for cross-node TP.
- Private networking because distributed traffic is unencrypted and may permit code execution if exposed.
These assumptions are appropriate for one trusted managed cluster, not an open heterogeneous volunteer route.
### 5.8 Cache and scheduling semantics
Each PP stage owns attention modules only for its local layers, so its physical KV is stage-local. However, request scheduling and block assignment are coordinated by one vLLM engine. Cache identity is internal request/block state, not an externally routable `(route_session, route_epoch, model, range)` contract.
A pipeline rank cannot independently accept an arbitrary activation from a tracker and safely infer whether it has the required local cache without building a new application protocol around it.
### 5.9 Failure behavior
vLLM's Ray layer can restart actors and Ray Serve can provide deployment-level fault tolerance, but a live PP engine depends on its process group and synchronized ranks. A lost stage invalidates active distributed execution and local cache for that stage.
No source path was found that dynamically replaces one PP rank during an active generation and resumes from another worker's independently reconstructed local cache. This is fundamentally different from Meshnet's route cache-miss and re-prefill recovery model.
## 6. Other vLLM distributed modes
### 6.1 Tensor parallelism
TP splits tensors and performs collectives within most transformer layers. It assumes tightly coordinated ranks and fast communication. It is a poor fit for independent heterogeneous internet nodes but can be valuable inside one trusted node or managed cluster.
GGUF plugin evidence currently includes a two-GPU TP test. That does not validate PP or volunteer routing.
### 6.2 Expert parallelism
Qwen3-MoE and other MoE models can partition experts across EP ranks. Every forward requires routing and collective coordination across the expert group:
- `vllm/model_executor/models/qwen3_moe.py:130-251`.
This is useful inside a high-speed managed cluster. It should not be confused with assigning whole transformer-layer ranges to independent volunteer nodes.
### 6.3 Data parallelism
vLLM DP replicates complete model weights across independent engine ranks. External load balancing can route HTTP requests to separate complete vLLM deployments, and each replica has an independent KV cache.
This maps cleanly to a whole-model node proxy or managed serving lane, not distributed single-model execution.
### 6.4 Disaggregated prefill
Disaggregated prefill runs separate complete vLLM instances for prefill and decode and transfers KV/results through connector implementations. Official documentation explicitly says it does not improve throughput; it separates TTFT and inter-token-latency tuning.
It is not pipeline layer sharding.
The connector interface is still a useful source of lifecycle ideas:
- Scheduler and worker roles.
- Request-finished ownership transfer.
- Asynchronous save/load completion.
- Per-layer KV save/load hooks.
- Block IDs and invalid-block reporting.
- Connector metrics and events.
- Preemption handling before blocks are overwritten.
Evidence:
- `vllm/distributed/kv_transfer/kv_connector/v1/base.py:1-41`.
- `vllm/distributed/kv_transfer/kv_connector/v1/base.py:124-229`.
- `vllm/distributed/kv_transfer/kv_connector/v1/base.py:251-300`.
- `vllm/v1/worker/kv_connector_model_runner_mixin.py:33-113`.
The API is explicitly experimental and subject to change:
- `vllm/distributed/kv_transfer/kv_connector/v1/base.py:184-200`.
Importing it as a dependency would tightly couple Meshnet cache semantics to vLLM. Reusing its lifecycle concepts in the project-owned cache protocol is safer.
## 7. Hardware portability
vLLM supports multiple platform families through separate builds and platform implementations, including CUDA, ROCm, CPU, XPU, and TPU, with additional ecosystem integrations.
This does **not** imply that one model-parallel vLLM world can mix arbitrary platform types. PP/TP communication and model workers share:
- Torch distributed process groups.
- A platform-selected device type and communicator.
- Compatible tensor dtypes and model implementations.
- Identical package/runtime environments.
- Synchronization and collective assumptions.
The GGUF plugin itself requires CUDA or ROCm and uses a CUDAExtension/HIP build. It does not provide the CPU/Vulkan/Metal GGUF lane needed for broad volunteer hardware.
Therefore vLLM is multi-platform across separate deployments, not demonstrated as heterogeneous within one distributed model replica.
## 8. Security and trust boundary
Official multi-node documentation warns that distributed traffic is unencrypted and can expose code-execution risk. It recommends a private network.
This alone disqualifies direct exposure of vLLM's Ray/Torch distributed plane to untrusted network participants.
A safe Meshnet node can still supervise vLLM behind a local proxy. The external network sees only the project-owned authenticated and bounded protocol.
## 9. Reuse matrix
| vLLM component | Direct code reuse | Design reuse | Decision |
|---|---:|---:|---|
| GGUF Python loader | No | Limited | Whole-model vLLM recipe only |
| GGUF CUDA/HIP/Triton kernels | No for llama.cpp worker | Performance reference | Keep in vLLM lane |
| `make_layers` / `PPMissingLayer` | No | Yes | Reference for local stage construction |
| `IntermediateTensors` | No | Strong yes | Add named tensor-bundle concept to activation ABI |
| Torch PP communicator | No | Limited | Static trusted cluster only |
| PagedAttention/block manager | No | Yes | Cache budgeting and metrics concepts |
| KV connector lifecycle | No | Strong yes | Adapt lifecycle semantics, not dependency |
| Disaggregated prefill connectors | No | Yes | Reference for async KV transfer/checkpointing |
| DP external load balancing | Via HTTP proxy | Yes | Whole-model vLLM node lane |
| Ray executor | No for volunteer mesh | Limited | Managed cluster backend only |
| Qwen3-MoE model graph | No in C++ worker | Strong yes | Cross-check llama.cpp adapter behavior |
| Request/KV telemetry | Possibly via proxy | Yes | Map to tracker metrics where trustworthy |
## 10. Implications for native GGUF design
### 10.1 Activation ABI must support tensor bundles
The first GGUF design assumed one BF16 boundary residual. vLLM shows that architecture implementations may naturally expose more than one state tensor, such as:
```text
hidden_states
residual
```
Other architectures may need recurrent state, cross-attention state, or auxiliary routing data.
The backend-neutral envelope should support:
```text
bundle schema/version
named tensors
each tensor's shape, dtype, byte order, compression and checksum
architecture recipe and boundary point
```
For each certified architecture, either:
- Define a canonical fused single-residual boundary, or
- Define an exact named bundle schema.
### 10.2 Separate model artifact from execution recipe
A GGUF artifact fingerprint is insufficient by itself. Capability proof should include:
- Artifact hash.
- Runtime family and version.
- Architecture adapter/recipe version.
- Boundary schema version.
- Backend and kernel lane.
- Quantization types actually supported.
This prevents a vLLM GGUF route from being mixed with a llama.cpp GGUF route merely because both opened the same file.
### 10.3 Keep whole-model vLLM as a distinct route kind
A vLLM server can be registered as:
```text
route_kind = whole_model
runtime = vllm
artifact/model identity
quantization = gguf | awq | gptq | fp8 | ...
```
It should not claim activation-shard compatibility unless an explicit compatible worker protocol exists.
### 10.4 Managed vLLM cluster as one logical node
A trusted vLLM TP/PP/EP deployment can register as one complete provider. Internal ranks remain invisible to tracker route construction; the cluster owns its own static world, cache, and failure behavior.
### 10.5 Reuse the KV-transfer envelope and lifecycle concepts
vLLM's most portable contribution is not PagedAttention code but its control-plane treatment of external KV movement:
- Protocol versions change when schema, wire format, or memory layout changes.
- Transfer IDs are distinct from engine request IDs.
- Metadata carries producer/consumer identity, request, block IDs, topology, block size/layout, and backend details.
- Compatibility fingerprints include model shape, dtype, KV heads/layers, cache dtype, backend, and allocator mode.
- Send, receive, completion, abort, abort acknowledgement, and failed-block states are explicit.
- Failed or unavailable external blocks become a cache miss and local-prefill fallback.
- Cache ownership can be leased until asynchronous transfer completion before blocks are released.
Evidence:
- `vllm/distributed/kv_transfer/kv_connector/v1/nixl/metadata.py:28-43`.
- `vllm/distributed/kv_transfer/kv_connector/v1/nixl/metadata.py:46-76`.
- `vllm/distributed/kv_transfer/kv_connector/v1/nixl/metadata.py:79-139`.
- `vllm/distributed/kv_transfer/kv_connector/v1/nixl/metadata.py:142-189`.
- `vllm/distributed/kv_transfer/kv_connector/v1/base.py:453-527`.
- `vllm/distributed/kv_transfer/kv_connector/v1/base.py:547-566`.
- `vllm/v1/kv_offload/tiering/p2p/session/protocol.py:167-247`.
- `vllm/v1/kv_offload/tiering/p2p/manager.py:180-202`.
These small metadata/state-machine patterns can be reimplemented in Meshnet's Python control plane with fields such as:
```text
schema_version
request_id
kv_transfer_id
producer_worker_id
consumer_worker_id
model/cache compatibility fingerprint
block or token range
dtype and layout
route/generation epoch
transfer state
```
Do not import vLLM's connector ABCs, block allocator, PagedAttention tensors, or worker classes. They are coupled to vLLM's scheduler, PyTorch/Triton layout, and static runtime.
## 11. Optional validation spikes
These spikes are optional and do not replace the llama.cpp layer-worker plan.
### 11.1 Whole-model vLLM GGUF baseline
On compatible CUDA or ROCm hardware:
- Install a pinned vLLM and plugin pair.
- Run a small Qwen3 dense GGUF.
- Compare load time, memory, prefill, decode, and output parity with llama.cpp.
- Verify whether the target Radeon lane builds the plugin successfully.
This establishes a baseline and potential whole-model recipe.
### 11.2 Static two-stage vLLM PP experiment
Using a non-GGUF or small supported GGUF model:
- Run PP=2 in a trusted local environment.
- Record each stage's loaded parameter names and memory.
- Capture actual `IntermediateTensors` names, shapes, and dtypes.
- Validate local KV allocation by stage.
- Kill one stage and document engine/request recovery behavior.
This validates design assumptions but does not make vLLM suitable for volunteer routing.
### 11.3 Qwen3 30B-A3B compatibility probe
Before claiming a vLLM GGUF whole-model lane:
- Load the exact target GGUF and base tokenizer/config.
- Verify fused MoE quantization type support.
- Test TP=1 and the intended TP/EP configuration.
- Compare deterministic output to llama.cpp.
- Measure expert memory, endpoint replication, and ROCm behavior.
No current test in the audited repositories establishes this target combination.
## 12. Final conclusion
vLLM is a strong production serving engine and contains the most mature architecture-aware PyTorch pipeline implementation found in this research. Its stage-local model construction and intermediate tensor bundles are valuable design references.
It is not the correct primary runtime for neuron-tai's distributed GGUF layer mesh because its GGUF path is experimental and GPU-oriented, while its distributed execution assumes one static, trusted, synchronized Torch world.
The architecture decision remains:
```text
Primary distributed GGUF layer mesh:
small pinned llama.cpp layer-worker fork
+ project-owned activation/session protocol
+ existing tracker, relay and accounting
Secondary whole-model serving lane:
local proxy to vLLM/Ollama/llama-server/etc.
Optional managed-cluster lane:
one trusted vLLM TP/PP cluster represented as one logical provider
```
The strongest new lesson from vLLM is that the distributed activation ABI should be architecture-aware and support named tensor bundles rather than assuming every model can be split at a single anonymous hidden-state tensor.