neuron-tai/docs/research/vllm-distributed-gguf-assessment.md

# vLLM assessment for distributed GGUF inference

Status: Source and documentation assessment
Last updated: 2026-07-13

## 1. Question

Can vLLM or its GGUF plugin be reused to implement neuron-tai's primary distributed GGUF data plane?

The required data plane is not merely a whole-model serving cluster. It requires independently loaded GGUF layer ranges on heterogeneous nodes, tracker-selected per-request routes, a versioned activation boundary, local route-session cache, dynamic failure handling, relay support, and per-node work attribution.

This assessment also identifies vLLM components and concepts that remain useful even if its distributed runtime is not adopted.

## 2. Sources and snapshots

Source snapshots audited:

```text
vllm-project/vllm
commit 107a03ba63e005ff03424fed9c4e6cf551b98bb2

vllm-project/vllm-gguf-plugin
commit ad209df10bb1856ba53b6663745a831eb7eb09cc
```

Current official documentation reviewed:

- [GGUF](https://docs.vllm.ai/en/latest/features/quantization/gguf/).
- [Parallelism and Scaling](https://docs.vllm.ai/en/stable/serving/parallelism_scaling/).
- [Disaggregated Prefilling](https://docs.vllm.ai/en/latest/features/disagg_prefill/).
- [Data Parallel Deployment](https://docs.vllm.ai/en/latest/serving/data_parallel_deployment/).
- [Installation and supported platforms](https://docs.vllm.ai/en/latest/getting_started/installation/).
- [RFC #39583: Migrate bitsandbytes and GGUF quantization support to an out-of-tree plugin](https://github.com/vllm-project/vllm/issues/39583).

Both source projects are Apache-2.0 licensed.

## 3. Executive decision

### 3.1 Primary distributed GGUF runtime

Do **not** use vLLM as the primary tracker-routed heterogeneous GGUF layer-worker runtime.

Reasons:

- Current GGUF support is an experimental out-of-tree quantization plugin.
- The plugin converts GGUF tensors into Torch parameters and uses vLLM's PyTorch model implementations; it is not a lightweight llama.cpp/GGML runtime.
- The plugin builds CUDA/HIP Torch extensions and does not provide CPU, Vulkan, or Metal GGUF execution.
- vLLM pipeline parallelism assumes one static Torch distributed world with rank/process-group semantics.
- Pipeline workers communicate with neighboring ranks through Torch distributed or a trusted TCP/device communicator, not the Meshnet HTTP/relay/session protocol.
- Requests are scheduled as one synchronized engine, not independently routed through tracker-selected workers.
- Failure of a pipeline rank is a distributed-engine failure, not a route-local cache miss that can be recovered by rebuilding a route.
- Multi-node guidance requires identical environments and private trusted networking.
- External vLLM load balancing is across complete data-parallel model replicas, not model-layer workers.

### 3.2 Supported secondary roles

vLLM remains useful in three narrower roles:

1. **Whole-model node backend:** a node proxy can register an independent vLLM server for models/hardware that fit on that deployment. This is an execution recipe, not the core distributed GGUF feature.
2. **Static managed-cluster backend:** a separately managed, trusted vLLM TP/PP cluster can be exposed as one logical node. The tracker accounts at cluster boundary unless a trusted internal accounting bridge is added.
3. **Source/design donor:** architecture-aware pipeline boundaries, local layer construction, PagedAttention scheduling, KV connector lifecycles, and telemetry are valuable references.

### 3.3 GGUF kernel reuse

Do not import the vLLM GGUF plugin into the selected llama.cpp shard worker.

The plugin's GGUF kernels are tightly coupled to:

- PyTorch parameters and custom operators.
- vLLM quantization interfaces.
- Triton/CUDA/HIP execution.
- vLLM's model and tensor-parallel parameter layouts.
- vLLM release internals and monkey-patched plugin registration.

llama.cpp already provides broader GGUF quant coverage, CPU and edge backends, mmap loading, and the desired C/C++ runtime boundary.

## 4. Current GGUF support in vLLM

### 4.1 Support moved out of core

Official documentation states that GGUF support is:

- Highly experimental.
- Under-optimized.
- Potentially incompatible with other features.
- Now provided by the separate `vllm-gguf-plugin`.

The migration RFC gives the maintenance rationale. A vLLM maintainer estimated GGUF usage at roughly 0.1%, described poor performance relative to llama.cpp for batch-size-one consumer-GPU workloads, and identified a large special-case burden: legacy weight-loader branches, model-specific mappings, and roughly 6,000 lines of GGUF CUDA kernels. The out-of-tree plugin preserves availability while allowing vLLM core to prioritize its native production quantization paths. This is evidence that GGUF is a compatibility lane in vLLM, not its primary runtime focus.

At the audited snapshot, the plugin is itself young: version `0.0.4`, with its repository created during the 2026 migration. This is not a reason to reject it as a whole-model backend, but it raises compatibility and maintenance risk for using it as foundational distributed infrastructure.

The plugin registers itself through the `vllm.general_plugins` entry point:

- `vllm-gguf-plugin/pyproject.toml:17-18`.
- `vllm-gguf-plugin/vllm_gguf_plugin/plugin.py:109-127`.

It patches vLLM engine argument creation and speculative-model probing at runtime:

- `vllm-gguf-plugin/vllm_gguf_plugin/plugin.py:51-97`.

This demonstrates close coupling to vLLM internals and increases upgrade risk.

### 4.2 Dependencies and hardware assumptions

The plugin requires:

- `gguf>=0.17.0`.
- `vllm`.
- `torch>=2.9`.
- CUDA or ROCm toolkit for normal installation.

Evidence:

- `vllm-gguf-plugin/pyproject.toml:1-18`.
- `vllm-gguf-plugin/README.md:7-13`.

Its compiled extension is created through `torch.utils.cpp_extension.CUDAExtension`. ROCm is handled by compiling the same extension through HIP-compatible tooling:

- `vllm-gguf-plugin/setup.py:28-65`.

This gives a CUDA/ROCm lane, not the CPU/Vulkan/Metal hardware coverage expected from llama.cpp.

### 4.3 Model/config/tokenizer dependency

GGUF is treated as a weight source for a vLLM/Hugging Face model implementation.

The plugin determines model configuration from, in order:

- Explicit `hf_config_path`.
- A non-GGUF tokenizer path.
- The remote GGUF repository.
- The local GGUF parent directory.

Evidence:

- `vllm-gguf-plugin/vllm_gguf_plugin/plugin.py:34-48`.
- `vllm-gguf-plugin/vllm_gguf_plugin/plugin.py:51-79`.

Official docs recommend using the base Hugging Face tokenizer because converting a tokenizer from GGUF is slow and unstable. If Hugging Face cannot derive the architecture config, users must supply a compatible config path.

This is different from llama.cpp, which treats GGUF metadata, tensors, tokenizer, and architecture implementation as one native runtime artifact.

### 4.4 Artifact resolution

The plugin accepts:

- A local GGUF file.
- A local directory plus quantization type.
- A remote `repo_id:quant_type`.
- A remote `repo_id/filename.gguf`.

Evidence:

- `vllm-gguf-plugin/vllm_gguf_plugin/loader.py:41-67`.
- `vllm-gguf-plugin/vllm_gguf_plugin/weight_utils.py:18-70`.

It supports split GGUF artifacts named like `-00001-of-00004.gguf`:

- `vllm-gguf-plugin/vllm_gguf_plugin/weights_adapter/default.py:248-264`.

This is file sharding, not model-layer network sharding.

### 4.5 GGUF-to-vLLM name mapping

The default adapter:

1. Maps a GGUF architecture to Hugging Face tensor names through the Python `gguf` package.
2. Instantiates a Hugging Face model on the `meta` device to obtain its state-dict names.
3. Creates architecture-specific special mappings for MoE and other variants.
4. Maps those Hugging Face names into vLLM's packed parameter layout.

Evidence:

- `vllm-gguf-plugin/vllm_gguf_plugin/weights_adapter/default.py:42-231`.
- Qwen2/Qwen3 MoE mappings: `vllm-gguf-plugin/vllm_gguf_plugin/weights_adapter/default.py:81-98`.
- vLLM packed Llama mappings: `vllm/model_executor/models/llama.py:344-354`.
- vLLM packed Qwen3-MoE mappings: `vllm/model_executor/models/qwen3_moe.py:432-446`.

This provides broad model integration by relying on vLLM and Transformers model classes. It does not expose a generic independently executable GGUF layer object.

### 4.6 Loading behavior

`GGUFModelLoader` initializes an ordinary vLLM model on the target Torch device and calls its `load_weights` method with tensors yielded by the adapter:

- `vllm-gguf-plugin/vllm_gguf_plugin/loader.py:78-104`.

The GGUF iterator:

- Opens each GGUF file through `gguf.GGUFReader`.
- Enumerates every tensor.
- Converts NumPy-backed data to a Torch tensor.
- Emits an extra quantization-type parameter for quantized weights.

Evidence:

- `vllm-gguf-plugin/vllm_gguf_plugin/weight_utils.py:73-135`.

Pipeline stages instantiate only their repeating local layers, so vLLM's `AutoWeightsLoader` can skip parameters represented by `PPMissingLayer`. However, the plugin still enumerates the artifact through Python on each worker. This is not the same as selectively registering only a layer range and mmaping only its file regions in llama.cpp.

### 4.7 Quantization types and kernels

The plugin declares support for:

- Unquantized F32, F16, BF16.
- Q4_0, Q4_1, Q5_0, Q5_1, Q8_0, Q8_1.
- Q2_K, Q3_K, Q4_K, Q5_K, Q6_K.
- IQ1_M, IQ1_S, IQ2_XXS, IQ2_XS, IQ2_S, IQ3_XXS, IQ3_S, IQ4_XS, IQ4_NL.

Evidence:

- `vllm-gguf-plugin/vllm_gguf_plugin/quantization/utils.py:46-75`.

Linear execution chooses among:

- Native unquantized matrix multiplication.
- Quantized matrix-vector kernels for small token counts.
- Quantized matrix-matrix kernels for larger token counts.
- Full dequantization followed by Torch matrix multiplication as fallback.

Evidence:

- `vllm-gguf-plugin/vllm_gguf_plugin/quantization/linear.py:34-57`.
- `vllm-gguf-plugin/vllm_gguf_plugin/quantization/linear.py:79-247`.

There are separate Triton and compiled CUDA/HIP implementations for GEMM, dequantization, embedding, and fused MoE.

These kernels are useful to the vLLM ecosystem but are not portable drop-in components for a llama.cpp/GGML worker.

### 4.8 Test evidence

Plugin tests compare GGUF output against unquantized vLLM output for small models including Qwen2.5, Qwen3 dense, Phi3, GPT-2, StableLM, Gemma3, and OLMoE:

- `vllm-gguf-plugin/tests/test_gguf_generation.py:22-79`.
- `vllm-gguf-plugin/tests/test_gguf_generation.py:177-226`.

There is a two-GPU tensor-parallel GGUF test for Qwen3-0.6B:

- `vllm-gguf-plugin/tests/test_gguf_generation.py:229-283`.

The core vLLM plugin test also covers Qwen3 dense and OLMoE at TP=1 and TP=2:

- `tests/plugins_tests/gguf/test_gguf_plugin_generate.py:27-132`.

No source test was found for:

- GGUF pipeline parallelism.
- Multi-node GGUF PP.
- Mixed hardware.
- Qwen3 30B-A3B GGUF.
- Tracker-selected ranges.
- Independent route sessions or dynamic route recovery.

Kernel tests exercise CUDA tensors and broad quant types. One variable-length batching kernel test is explicitly skipped because the current CUDA kernel does not support that case:

- `vllm-gguf-plugin/tests/test_kernels.py:260-310`.

## 5. vLLM pipeline parallelism

### 5.1 What it gets right

vLLM has a mature architecture-aware pipeline abstraction:

- Model implementations construct only repeating layers assigned to the local PP rank.
- Missing layers are represented with `PPMissingLayer` placeholders.
- First-stage token embeddings and last-stage norm/head can be conditionally owned.
- Non-tail stages return named intermediate tensor bundles.
- The scheduler and model runner overlap stage communication and execution.
- Uneven layer-count splits are supported.

This is substantially more mature than the static dense-layer reconstruction in `llama-gguf`.

### 5.2 Static stage partitioning

`make_layers` derives `(start_layer, end_layer)` from PP rank and PP world size, creates real modules only for that interval, and fills all other positions with `PPMissingLayer`:

- `vllm/model_executor/models/utils.py:674-719`.

The default partition is approximately even. A static `VLLM_PP_LAYER_PARTITION` environment variable can override the number of layers per rank:

- `vllm/distributed/utils.py:127-172`.

This can represent uneven contiguous ranges, but it remains a launch-time rank partition:

- One ordered process group.
- One fixed world size.
- One static set of adjacent ranks.
- No per-request tracker route.
- No arbitrary start/end assignment independent of rank.
- No overlapping shard ranges or Meshnet effective-start semantics.

### 5.3 Llama stage ownership

For Llama:

- The first rank owns token embedding.
- A tied output rank may also own embedding.
- Only the local repeating layers are instantiated.
- Only the last rank owns final RMSNorm.
- Only the last rank owns LM head and logits processor.

Evidence:

- `vllm/model_executor/models/llama.py:356-395`.
- `vllm/model_executor/models/llama.py:400-439`.
- `vllm/model_executor/models/llama.py:466-503`.

This is a good design reference for explicit head, middle, and tail ownership.

### 5.4 Intermediate tensor bundles

`IntermediateTensors` is a dictionary of named Torch tensors:

- `vllm/sequence.py:10-62`.

For Llama and Qwen3-MoE, the PP boundary includes both:

```text
hidden_states
residual
```

Evidence:

- `vllm/model_executor/models/llama.py:393-395`.
- `vllm/model_executor/models/llama.py:408-433`.
- `vllm/model_executor/models/qwen3_moe.py:478-518`.

This is an important architectural lesson. Depending on where the boundary is placed, an exact pipeline stage may require more than one tensor. Meshnet's backend-neutral activation envelope should support a named tensor bundle or deliberately choose a canonical boundary after residual fusion.

A universal single `hidden_states` tensor contract may silently break architecture parity.

### 5.5 Qwen3-MoE stage behavior

The audited Qwen3-MoE implementation supports:

- Q/K normalization.
- Routed experts.
- Shared experts.
- Expert/tensor parallel groups.
- Stage-local repeating layers.
- `hidden_states` plus `residual` PP boundaries.

Evidence:

- `vllm/model_executor/models/qwen3_moe.py:130-251`.
- `vllm/model_executor/models/qwen3_moe.py:254-354`.
- `vllm/model_executor/models/qwen3_moe.py:357-429`.
- `vllm/model_executor/models/qwen3_moe.py:432-524`.

However, in this source snapshot, Qwen3-MoE constructs embedding, final norm, and LM head without the first/last-rank guards used by Llama:

- Embedding: `vllm/model_executor/models/qwen3_moe.py:463-471`.
- Final norm: `vllm/model_executor/models/qwen3_moe.py:477`.
- LM head/logits: `vllm/model_executor/models/qwen3_moe.py:566-589`.

Repeating layer weights remain stage-local, but endpoint parameters appear replicated. This source observation should be validated in a real PP load before using vLLM memory behavior as a reference for Qwen3 30B-A3B.

### 5.6 Stage communication

The worker receives intermediate tensor dictionaries from the previous PP rank, executes the local model, and asynchronously sends output tensors to the next rank:

- `vllm/v1/worker/gpu_worker.py:1045-1088`.

The PP group:

- Sends metadata through a CPU process group.
- Sends tensors through Torch distributed device or CPU groups.
- Defaults destination to the next static rank.
- Defaults source to the previous static rank.
- Can optimize TP-group slices with all-gather reconstruction.

Evidence:

- `vllm/distributed/parallel_state.py:960-1053`.
- `vllm/distributed/parallel_state.py:1055-1149`.

A stateless coordinator variant uses a trusted TCP store and optional device communicator:

- `vllm/distributed/stateless_coordinator.py:300-356`.

This is not a versioned application protocol. It has no Meshnet route session, route epoch, model fingerprint, layer range, request work receipt, relay semantics, authentication, or HTTP cache-miss contract.

### 5.7 Static distributed world

Official vLLM deployment guidance recommends:

- Tensor parallelism within a multi-GPU node.
- Pipeline parallelism across nodes.
- Ray or multiprocessing as the distributed executor.
- Identical model paths, packages, and execution environments across all nodes.
- High-speed networks such as InfiniBand for cross-node TP.
- Private networking because distributed traffic is unencrypted and may permit code execution if exposed.

These assumptions are appropriate for one trusted managed cluster, not an open heterogeneous volunteer route.

### 5.8 Cache and scheduling semantics

Each PP stage owns attention modules only for its local layers, so its physical KV is stage-local. However, request scheduling and block assignment are coordinated by one vLLM engine. Cache identity is internal request/block state, not an externally routable `(route_session, route_epoch, model, range)` contract.

A pipeline rank cannot independently accept an arbitrary activation from a tracker and safely infer whether it has the required local cache without building a new application protocol around it.

### 5.9 Failure behavior

vLLM's Ray layer can restart actors and Ray Serve can provide deployment-level fault tolerance, but a live PP engine depends on its process group and synchronized ranks. A lost stage invalidates active distributed execution and local cache for that stage.

No source path was found that dynamically replaces one PP rank during an active generation and resumes from another worker's independently reconstructed local cache. This is fundamentally different from Meshnet's route cache-miss and re-prefill recovery model.

## 6. Other vLLM distributed modes

### 6.1 Tensor parallelism

TP splits tensors and performs collectives within most transformer layers. It assumes tightly coordinated ranks and fast communication. It is a poor fit for independent heterogeneous internet nodes but can be valuable inside one trusted node or managed cluster.

GGUF plugin evidence currently includes a two-GPU TP test. That does not validate PP or volunteer routing.

### 6.2 Expert parallelism

Qwen3-MoE and other MoE models can partition experts across EP ranks. Every forward requires routing and collective coordination across the expert group:

- `vllm/model_executor/models/qwen3_moe.py:130-251`.

This is useful inside a high-speed managed cluster. It should not be confused with assigning whole transformer-layer ranges to independent volunteer nodes.

### 6.3 Data parallelism

vLLM DP replicates complete model weights across independent engine ranks. External load balancing can route HTTP requests to separate complete vLLM deployments, and each replica has an independent KV cache.

This maps cleanly to a whole-model node proxy or managed serving lane, not distributed single-model execution.

### 6.4 Disaggregated prefill

Disaggregated prefill runs separate complete vLLM instances for prefill and decode and transfers KV/results through connector implementations. Official documentation explicitly says it does not improve throughput; it separates TTFT and inter-token-latency tuning.

It is not pipeline layer sharding.

The connector interface is still a useful source of lifecycle ideas:

- Scheduler and worker roles.
- Request-finished ownership transfer.
- Asynchronous save/load completion.
- Per-layer KV save/load hooks.
- Block IDs and invalid-block reporting.
- Connector metrics and events.
- Preemption handling before blocks are overwritten.

Evidence:

- `vllm/distributed/kv_transfer/kv_connector/v1/base.py:1-41`.
- `vllm/distributed/kv_transfer/kv_connector/v1/base.py:124-229`.
- `vllm/distributed/kv_transfer/kv_connector/v1/base.py:251-300`.
- `vllm/v1/worker/kv_connector_model_runner_mixin.py:33-113`.

The API is explicitly experimental and subject to change:

- `vllm/distributed/kv_transfer/kv_connector/v1/base.py:184-200`.

Importing it as a dependency would tightly couple Meshnet cache semantics to vLLM. Reusing its lifecycle concepts in the project-owned cache protocol is safer.

## 7. Hardware portability

vLLM supports multiple platform families through separate builds and platform implementations, including CUDA, ROCm, CPU, XPU, and TPU, with additional ecosystem integrations.

This does **not** imply that one model-parallel vLLM world can mix arbitrary platform types. PP/TP communication and model workers share:

- Torch distributed process groups.
- A platform-selected device type and communicator.
- Compatible tensor dtypes and model implementations.
- Identical package/runtime environments.
- Synchronization and collective assumptions.

The GGUF plugin itself requires CUDA or ROCm and uses a CUDAExtension/HIP build. It does not provide the CPU/Vulkan/Metal GGUF lane needed for broad volunteer hardware.

Therefore vLLM is multi-platform across separate deployments, not demonstrated as heterogeneous within one distributed model replica.

## 8. Security and trust boundary

Official multi-node documentation warns that distributed traffic is unencrypted and can expose code-execution risk. It recommends a private network.

This alone disqualifies direct exposure of vLLM's Ray/Torch distributed plane to untrusted network participants.

A safe Meshnet node can still supervise vLLM behind a local proxy. The external network sees only the project-owned authenticated and bounded protocol.

## 9. Reuse matrix

| vLLM component | Direct code reuse | Design reuse | Decision |
|---|---:|---:|---|
| GGUF Python loader | No | Limited | Whole-model vLLM recipe only |
| GGUF CUDA/HIP/Triton kernels | No for llama.cpp worker | Performance reference | Keep in vLLM lane |
| `make_layers` / `PPMissingLayer` | No | Yes | Reference for local stage construction |
| `IntermediateTensors` | No | Strong yes | Add named tensor-bundle concept to activation ABI |
| Torch PP communicator | No | Limited | Static trusted cluster only |
| PagedAttention/block manager | No | Yes | Cache budgeting and metrics concepts |
| KV connector lifecycle | No | Strong yes | Adapt lifecycle semantics, not dependency |
| Disaggregated prefill connectors | No | Yes | Reference for async KV transfer/checkpointing |
| DP external load balancing | Via HTTP proxy | Yes | Whole-model vLLM node lane |
| Ray executor | No for volunteer mesh | Limited | Managed cluster backend only |
| Qwen3-MoE model graph | No in C++ worker | Strong yes | Cross-check llama.cpp adapter behavior |
| Request/KV telemetry | Possibly via proxy | Yes | Map to tracker metrics where trustworthy |

## 10. Implications for native GGUF design

### 10.1 Activation ABI must support tensor bundles

The first GGUF design assumed one BF16 boundary residual. vLLM shows that architecture implementations may naturally expose more than one state tensor, such as:

```text
hidden_states
residual
```

Other architectures may need recurrent state, cross-attention state, or auxiliary routing data.

The backend-neutral envelope should support:

```text
bundle schema/version
named tensors
each tensor's shape, dtype, byte order, compression and checksum
architecture recipe and boundary point
```

For each certified architecture, either:

- Define a canonical fused single-residual boundary, or
- Define an exact named bundle schema.

### 10.2 Separate model artifact from execution recipe

A GGUF artifact fingerprint is insufficient by itself. Capability proof should include:

- Artifact hash.
- Runtime family and version.
- Architecture adapter/recipe version.
- Boundary schema version.
- Backend and kernel lane.
- Quantization types actually supported.

This prevents a vLLM GGUF route from being mixed with a llama.cpp GGUF route merely because both opened the same file.

### 10.3 Keep whole-model vLLM as a distinct route kind

A vLLM server can be registered as:

```text
route_kind = whole_model
runtime = vllm
artifact/model identity
quantization = gguf | awq | gptq | fp8 | ...
```

It should not claim activation-shard compatibility unless an explicit compatible worker protocol exists.

### 10.4 Managed vLLM cluster as one logical node

A trusted vLLM TP/PP/EP deployment can register as one complete provider. Internal ranks remain invisible to tracker route construction; the cluster owns its own static world, cache, and failure behavior.

### 10.5 Reuse the KV-transfer envelope and lifecycle concepts

vLLM's most portable contribution is not PagedAttention code but its control-plane treatment of external KV movement:

- Protocol versions change when schema, wire format, or memory layout changes.
- Transfer IDs are distinct from engine request IDs.
- Metadata carries producer/consumer identity, request, block IDs, topology, block size/layout, and backend details.
- Compatibility fingerprints include model shape, dtype, KV heads/layers, cache dtype, backend, and allocator mode.
- Send, receive, completion, abort, abort acknowledgement, and failed-block states are explicit.
- Failed or unavailable external blocks become a cache miss and local-prefill fallback.
- Cache ownership can be leased until asynchronous transfer completion before blocks are released.

Evidence:

- `vllm/distributed/kv_transfer/kv_connector/v1/nixl/metadata.py:28-43`.
- `vllm/distributed/kv_transfer/kv_connector/v1/nixl/metadata.py:46-76`.
- `vllm/distributed/kv_transfer/kv_connector/v1/nixl/metadata.py:79-139`.
- `vllm/distributed/kv_transfer/kv_connector/v1/nixl/metadata.py:142-189`.
- `vllm/distributed/kv_transfer/kv_connector/v1/base.py:453-527`.
- `vllm/distributed/kv_transfer/kv_connector/v1/base.py:547-566`.
- `vllm/v1/kv_offload/tiering/p2p/session/protocol.py:167-247`.
- `vllm/v1/kv_offload/tiering/p2p/manager.py:180-202`.

These small metadata/state-machine patterns can be reimplemented in Meshnet's Python control plane with fields such as:

```text
schema_version
request_id
kv_transfer_id
producer_worker_id
consumer_worker_id
model/cache compatibility fingerprint
block or token range
dtype and layout
route/generation epoch
transfer state
```

Do not import vLLM's connector ABCs, block allocator, PagedAttention tensors, or worker classes. They are coupled to vLLM's scheduler, PyTorch/Triton layout, and static runtime.

## 11. Optional validation spikes

These spikes are optional and do not replace the llama.cpp layer-worker plan.

### 11.1 Whole-model vLLM GGUF baseline

On compatible CUDA or ROCm hardware:

- Install a pinned vLLM and plugin pair.
- Run a small Qwen3 dense GGUF.
- Compare load time, memory, prefill, decode, and output parity with llama.cpp.
- Verify whether the target Radeon lane builds the plugin successfully.

This establishes a baseline and potential whole-model recipe.

### 11.2 Static two-stage vLLM PP experiment

Using a non-GGUF or small supported GGUF model:

- Run PP=2 in a trusted local environment.
- Record each stage's loaded parameter names and memory.
- Capture actual `IntermediateTensors` names, shapes, and dtypes.
- Validate local KV allocation by stage.
- Kill one stage and document engine/request recovery behavior.

This validates design assumptions but does not make vLLM suitable for volunteer routing.

### 11.3 Qwen3 30B-A3B compatibility probe

Before claiming a vLLM GGUF whole-model lane:

- Load the exact target GGUF and base tokenizer/config.
- Verify fused MoE quantization type support.
- Test TP=1 and the intended TP/EP configuration.
- Compare deterministic output to llama.cpp.
- Measure expert memory, endpoint replication, and ROCm behavior.

No current test in the audited repositories establishes this target combination.

## 12. Final conclusion

vLLM is a strong production serving engine and contains the most mature architecture-aware PyTorch pipeline implementation found in this research. Its stage-local model construction and intermediate tensor bundles are valuable design references.

It is not the correct primary runtime for neuron-tai's distributed GGUF layer mesh because its GGUF path is experimental and GPU-oriented, while its distributed execution assumes one static, trusted, synchronized Torch world.

The architecture decision remains:

```text
Primary distributed GGUF layer mesh:
    small pinned llama.cpp layer-worker fork
    + project-owned activation/session protocol
    + existing tracker, relay and accounting

Secondary whole-model serving lane:
    local proxy to vLLM/Ollama/llama-server/etc.

Optional managed-cluster lane:
    one trusted vLLM TP/PP cluster represented as one logical provider
```

The strongest new lesson from vLLM is that the distributed activation ABI should be architecture-aware and support named tensor bundles rather than assuming every model can be split at a single anonymous hidden-state tensor.