popov/neuron-tai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
e2f3ae32b86102c5d26528eb7fd2b8aafcae07c0
neuron-tai/docs/research/distributed-gguf-github-followup.md
Dobromir Popov 4cae4a6c5c docs: define distributed GGUF runtime plan
2026-07-13 15:09:27 +03:00

804 lines
29 KiB
Markdown
Raw Blame History

# Distributed GGUF GitHub follow-up: GPUStack, Nakshatra, LiGGUF, and additional candidates
Status: Source audit complete
Last updated: 2026-07-13
## 1. Why this follow-up exists
This document evaluates additional claims and repositories found after the initial distributed-GGUF landscape report:
- The GPUStack 0.4 multi-worker GGUF tutorial.
- The claim that llama.cpp is the base of most practical GGUF distribution.
- Nakshatra's patched llama.cpp layer workers.
- LiGGUF's SARA distributed example.
- Chameleon and Continuum.
- Additional GitHub searches for sub-GGUF, layer-range, activation-chain, and llama.cpp RPC implementations.
It supplements [the main distributed-GGUF landscape report](distributed-gguf-landscape.md). The dedicated [vLLM assessment](vllm-distributed-gguf-assessment.md) remains separate.
## 2. Corrected terminology
The original research summary was directionally correct but combined several different forms of parallelism.
### 2.1 Layer or pipeline parallelism
Whole contiguous transformer-layer ranges are assigned to stages:
```text
tokens
-> layers 0..N
-> boundary residual
-> layers N..M
-> logits
```
This is the closest match to neuron-tai's tracker-selected route.
### 2.2 Tensor parallelism
Operations inside every layer are divided across ranks:
- Attention heads.
- Matrix rows or columns.
- FFN channels.
- Experts.
Ranks exchange collectives or partial reductions inside each transformer layer. LiGGUF SARA is an example.
### 2.3 Local multi-device placement
llama.cpp's `n_gpu_layers` and `tensor_split` choose how one coordinator places work across devices. This is local offload unless some devices are llama.cpp RPC devices.
### 2.4 llama.cpp RPC
llama.cpp RPC exposes a remote GGML backend/device to the coordinator. It is real cross-machine inference, but the remote server is not an independent layer/session worker. GPUStack 0.4 and llama-box used this mechanism.
### 2.5 Quantization
Q2_K, Q4_K_M, Q8_0, and related GGUF types reduce weight storage and memory. They do not define:
- Activation dtype.
- Compute dtype.
- KV-cache dtype.
- Parallelism topology.
- Session or route semantics.
## 3. Audited source snapshots
```text
GPUStack current main
244eb0da57add11d1ce07c70f31c1a15ae65ae0d
GPUStack v0.4.1
dbf71dd16cec1f42896139c3b82380cb1fd06a10
llama-box
4d068484fe198a30f8ca6d6d23d9890fbd8eee8c
Nakshatra
0c16119713396ec6052400f3eb049c5e7a66cd94
LiGGUF
2b5ac66ca37f36600aa5101b4237e74f3becb7c4
Chameleon
96fbd96a9f67d29d12292d3373c88996aba65f84
Continuum
dd976df36079d75244719a23956e1c9e2dcddc27
```
## 4. GPUStack 0.4 and llama-box
### 4.1 Verdict
The [GPUStack 0.4 tutorial](https://docs.gpustack.ai/0.4/tutorials/performing-distributed-inference-across-workers/) describes a real open-source deployment path. It is not a closed-source native layer-shard engine.
GPUStack 0.4 orchestrated llama-box, which embedded llama.cpp/ggml RPC. The execution shape was:
```text
GPUStack scheduler
-> select main worker and remote GPU devices
-> start llama-box RPC server on each selected remote GPU
-> launch llama-box on main worker
--rpc remote-a,remote-b,...
--tensor-split remote-vram...,local-vram...
-> coordinator opens the full GGUF
-> llama.cpp places tensors and graph operations on local and RPC devices
```
This is strong real-world evidence for llama.cpp RPC. It is not the independent layer-worker topology neuron-tai needs.
### 4.2 Scheduler and resource estimation
GPUStack detects GGUF models, estimates memory using its parser/calculator, marks distributable models, selects a main worker plus remote GPU devices, and persists their resource claims.
Evidence from GPUStack v0.4.1:
- `gpustack/scheduler/scheduler.py:147-205`.
- `gpustack/scheduler/scheduler.py:328-377`.
- `gpustack/scheduler/scheduler.py:429-449`.
- `gpustack/policies/utils.py:35-47`.
- `gpustack/policies/scorers/placement_scorer.py:184-196`.
This control-plane work is reusable conceptually:
- Parse exact GGUF requirements before placement.
- Allocate memory claims per device.
- Include remote-device allocations in global scheduling.
- Reject incompatible backend/runtime versions.
### 4.3 RPC server lifecycle
GPUStack workers periodically start one llama-box RPC process per GPU and publish its port in worker status.
Evidence:
- `gpustack/worker/worker.py:153-163`.
- `gpustack/worker/worker_manager.py:140-200`.
- `gpustack/worker/collector.py:66-80`.
- `gpustack/worker/rpc_server.py:31-76`.
The launched command uses:
```text
--rpc-server-host 0.0.0.0
--rpc-server-port <port>
--rpc-server-main-gpu 0
```
GPU selection is enforced through the vendor-specific visible-device environment.
### 4.4 Main server launch
The main worker obtains remote RPC addresses and remote VRAM claims, then launches llama-box with:
```text
--rpc <host:port,...>
--tensor-split <remote MiB...,local MiB...>
```
Evidence:
- `gpustack/worker/backends/llama_box.py:33-95`.
- `gpustack/worker/backends/llama_box.py:165-181`.
This confirms that GPUStack's worker list did not become a route of independently callable layer stages. llama-box remained the single model owner and request server.
### 4.5 llama-box RPC behavior
llama-box's RPC server serializes GGML buffers, tensors, and graph-compute requests. It exposes remote backend memory and supports optional tensor caching.
Evidence:
- `llama-box/rpcserver.hpp:74-213`.
- `llama-box/rpcserver.hpp:215-226`.
- `llama-box/rpcserver.hpp:374-410`.
- `llama-box/rpcserver.hpp:413-447`.
The RPC server is a low-level remote device. It does not own:
- A source GGUF identity.
- A tracker layer-range lease.
- A route session or route epoch.
- Independent tokenization/head/tail semantics.
- A project-compatible activation endpoint.
- Per-request node work receipts.
### 4.6 Real operational evidence
Three GPUStack issue reports are useful:
1. [Issue 1233](https://github.com/gpustack/gpustack/issues/1233) records a two-Mac llama-box RPC deployment and a maintainer reproduction command. Maintainers reported roughly 5-6 token/s for a large DeepSeek-R1 GGUF on two M2 Ultra systems.
2. [Issue 756](https://github.com/gpustack/gpustack/issues/756) records a crash when main and RPC llama-box versions differed and a remote Metal backend did not support an operation.
3. [Issue 1269](https://github.com/gpustack/gpustack/issues/1269) includes a real ten-device `--rpc`/`--tensor-split` command and an official maintainer statement that GPUStack 2.0 deprecated llama-box and no longer supports distributed GGUF inference.
These reports prove practical use while also demonstrating:
- Exact runtime/version compatibility is mandatory.
- Every remote backend must support every placed graph operation.
- Remote-memory estimates can still fail.
- Coordinator interruption can leak or strand remote resources.
- Cross-machine operation may be slower than expected.
### 4.7 Current status
The audited current GPUStack main snapshot retains legacy GGUF RPC-placement calculations, deprecated `rpc_servers` schema fields, migration code, and fixtures. It no longer has the llama-box backend or role-specific per-GPU RPC process launcher needed to turn those placement records into a working GGUF data plane. Current GGUF defaults to a custom backend, and GPUStack's supported multi-node matrix lists vLLM, SGLang, and MindIE rather than custom/GGUF.
Evidence from current GPUStack main:
- `gpustack/policies/candidate_selectors/gguf_resource_fit_selector.py:459-473`.
- `gpustack/policies/candidate_selectors/gguf_resource_fit_selector.py:1969-1997`.
- `gpustack/schemas/workers.py:198-202`.
- `docs/migration.md:157-167`.
- `gpustack/schemas/models.py:59-65`.
- `gpustack/schemas/models.py:266-275`.
- `gpustack/schemas/models.py:873-880`.
- `gpustack/worker/backends/custom.py:152-181`.
- `docs/faq.md:9-21`.
The retained selector is legacy residue and compatibility evidence, not a supported runnable distributed-GGUF backend.
Decision:
- Use GPUStack v0.4 as a llama.cpp RPC baseline and scheduling reference.
- Do not treat current GPUStack as a maintained distributed-GGUF backend.
- Do not reuse llama.cpp RPC as the volunteer network trust boundary.
## 5. Nakshatra
### 5.1 Verdict
Nakshatra is the closest implementation found to neuron-tai's native distributed-GGUF target.
It independently implements the same core seam selected in the initial report:
- Patched llama.cpp.
- Local layer-only GGUF artifacts.
- Contiguous first/middle/last workers.
- Residual activation transport.
- Worker-local llama.cpp KV.
- A long-lived C++ daemon supervised by Python.
- Dynamic placement and recovery above the worker.
It changes the implementation strategy from “write the first spike from scratch” to “reproduce, collaborate, reuse, and harden.”
It is not a wholesale drop-in because its control plane, artifact model, concurrency, identity, accounting, and architecture coverage differ from Meshnet.
### 5.2 Sub-GGUF construction
`partial_gguf.py` creates a derivative GGUF per layer range.
It preserves source metadata, filters `blk.N.*` tensors to `[start,end)`, keeps embeddings only for the head unless tied output needs them, and keeps output tensors only for the tail.
Evidence:
- `experiments/v0.0/partial_gguf.py:59-115`.
- `experiments/v0.0/partial_gguf.py:117-143`.
- `experiments/v0.0/partial_gguf.py:184-201`.
It writes:
```text
nakshatra.layer_range_start
nakshatra.layer_range_end
nakshatra.has_token_embd
nakshatra.has_lm_head
```
This gives workers real local ownership and prevents inference-time weight transfer.
Meshnet differences:
- Meshnet prefers one exact source artifact hash plus a range/recipe identity.
- Derived sub-GGUF files need their own hash and a signed binding to the source artifact and range.
- Rewriting a large GGUF for every placement is expensive and duplicates storage.
- A range-aware mmap loader from one shared artifact remains preferable long term.
Sub-GGUFs are still acceptable for a first integration spike because they already work.
### 5.3 llama.cpp patch mechanics
Nakshatra's patch series:
- Adds range and endpoint fields to the model.
- Reads namespaced partial-model metadata.
- Allows unowned tensors to be absent.
- Creates/loads only owned layer tensors.
- Iterates only the selected layer interval.
- Returns the unnormalized residual stream from non-tail stages.
- Applies final norm and LM head only on the tail.
Evidence:
- `experiments/v0.0/m4_patches/llama-model.h.patch:1-17`.
- `experiments/v0.0/m4_patches/llama-model.cpp.patch:1-73`.
- `experiments/v0.0/m4_patches/llama-model-loader.cpp.patch:1-11`.
- `experiments/v0.0/m4_patches/llama-graph.cpp.patch:1-24`.
- `experiments/v0.0/m4_patches/models_llama.cpp.patch:1-70`.
This is direct proof that the small-fork direction is feasible. The current patch is architecture-specific and built against older llama.cpp revisions. It should be rebased rather than copied blindly.
### 5.4 Worker shape
Each Python gRPC worker supervises one long-lived C++ daemon. The daemon owns the patched llama model/context and communicates with Python through a framed stdin/stdout or shared-memory protocol.
Evidence:
- `scripts/worker.py:1-17`.
- `experiments/v0.0/worker_daemon.cpp:1-52`.
- `experiments/v0.0/worker_daemon.cpp:166-267`.
- `experiments/v0.0/worker_daemon.cpp:282-396`.
The daemon accepts:
- Token IDs for a head stage.
- F32 boundary activations for middle/tail stages.
- `start_pos` and `keep_kv` controls.
It returns:
- F32 residual activations for non-tail stages.
- Greedy token IDs on the tail.
- Optional all-position top tokens for speculative verification.
The implementation also contains optional blockwise int8 activation transport:
- `experiments/v0.0/worker_daemon.cpp:105-149`.
This process boundary is close to the selected Meshnet worker shape and avoids exposing llama.cpp internal ABI to Python.
### 5.5 Protobuf and route semantics
Nakshatra's protobuf includes:
- Protocol/backend/model/range capability reporting.
- A source-model hash field.
- Head/tail ownership flags.
- Stable `session_id` and idempotency `step_id`.
- Prefix/KV position metadata.
- Token, hidden-state, logits, and error variants.
- Server-to-server next-hop chains.
- KV truncation.
- Sleep/wake lifecycle operations.
Evidence:
- `proto/nakshatra.proto:1-55`.
- `proto/nakshatra.proto:57-93`.
- `proto/nakshatra.proto:95-131`.
Meshnet's worker contract still needs additional fields:
- Route epoch.
- Effective overlap-safe start layer.
- Exact source and derived artifact hashes.
- Architecture/runtime/quantization/activation recipe.
- Request/work/accounting identity.
- Payload checksum and compression recipe.
- Cache expectation and explicit cache-miss response.
- Cancellation and lease identity.
The implementation should sit behind a Meshnet-owned protocol rather than adopting the protobuf as a permanent public API.
### 5.6 KV ownership and concurrency gap
The daemon supports:
- Cold prefill.
- Incremental decode.
- KV truncation after speculative rejection.
- Sleep/wake and model reload.
Evidence:
- `experiments/v0.0/worker_daemon.cpp:344-384`.
- `experiments/v0.0/worker_daemon.cpp:416-463`.
- `experiments/v0.0/worker_daemon.cpp:479-500`.
However, the audited worker creates one llama context with two sequence slots:
```text
sequence 0: serving/verification
sequence 1: EAGLE scratch
```
The v0.5 design explicitly states that the daemon still has one logical serving session and ships with `n_concurrent_sessions = 1`:
- `docs/v0.5-design-lock.md:97-111`.
The Python idempotency cache deduplicates `(session_id,step_id)` outputs. It does not isolate independent llama KV state for multiple concurrent route sessions.
Meshnet integration must add:
```text
(route_session_id, route_epoch)
-> llama_seq_id or isolated context
```
and verify concurrent prefill/decode, release, eviction, stale epoch rejection, and cache misses.
### 5.7 Real acceptance evidence
Nakshatra records a two-physical-machine CPU-only test over Tailscale:
- Head worker: layers `[0,14)`.
- Tail worker: layers `[14,28)`.
- Prompt: “The capital of France is”.
- Distributed first token: `12366`, “ Paris”.
- Matched localhost, single-process chain, and whole-model llama.cpp reference.
Evidence:
- `experiments/v0.0/m6_findings.md:1-42`.
This proves real independent layer execution and cross-machine activation transport. It does not prove GPU heterogeneity because both machines used CPU for the acceptance run.
The repository also records a four-worker Llama-3.3-70B chain using Macs with Metal and one CPU stage:
- Streaming completed around 0.21 token/s.
- Multi-hop server push completed around 0.19 token/s.
- Push was slower because compute dominated network time.
- First generated token was “Paris”.
- Alternate-worker replay completed with expected post-splice divergence.
Evidence:
- `docs/v0.5-design-lock.md:145-175`.
- `docs/v0.5-design-lock.md:258-277`.
The five-machine ROCm + Metal cross-vendor acceptance remained pending in the audited snapshot. Claims of fully validated ROCm+Metal execution should therefore remain qualified.
### 5.8 Failure and numerical behavior
Nakshatra supports:
- Persistent streaming RPCs.
- Server-to-server activation push.
- Fallback from failed push to client relay.
- Full-history replay after stream failure.
- Alternate workers per range.
- Drift-class-aware recovery design.
Evidence:
- `scripts/client.py:163-269`.
- `scripts/client.py:842-850`.
- `docs/v0.5-design-lock.md:258-277`.
- `docs/v1.0-fault-tolerance.md:93-103`.
The project correctly acknowledges that replay on another backend can numerically diverge. For Meshnet, the safe default remains:
1. Exact recipe and same drift class for in-session replacement.
2. Otherwise restart from token zero on a newly consistent route.
3. Never silently import incompatible KV or continue with an unvalidated mixed recipe.
### 5.9 Security and work receipts
The worker contains optional:
- TLS and SPKI pinning.
- Ed25519 request authentication.
- Admission and sandbox hooks.
- Audit logging.
- An experimental encrypted fabric.
Evidence:
- `scripts/worker.py:49-117`.
- `scripts/worker.py:218-319`.
The receipt implementation explicitly documents that:
- Output hash and structural consistency are independently checkable.
- Model identity is only asserted because the live model hash is a zero stub.
- Participation is coordinator-asserted because worker signatures are not populated.
Evidence:
- `scripts/receipt.py:1-27`.
- `scripts/receipt.py:34-47`.
- `scripts/receipt.py:50-97`.
- `scripts/receipt.py:100-160`.
This is not sufficient for per-node rewards. Meshnet must bind receipts to:
```text
request/work ID
route session and epoch
source artifact hash
layer range and effective start
input/output activation digests
positions/token count
runtime recipe
measured compute
worker identity/signature
```
### 5.10 Reproducibility and integration defects
The audited repository is not a self-contained llama.cpp fork or reproducible worker build:
- It ships patch files but no pinned llama.cpp source tree or submodule.
- The README recommends llama.cpp commit `c46583b` “or close enough”.
- The daemon is copied manually into an external llama.cpp checkout and its CMake target is added manually.
- The documented copy recipe omits `shm_ring.hpp`, which the daemon includes.
- The historical two-machine run used different llama.cpp builds on the two hosts.
Evidence:
- `README.md:41-77`.
- `experiments/v0.0/worker_daemon.cpp:54-57`.
- `experiments/v0.0/m6_findings.md:35-42`.
The live daemon also trusts operator-supplied range and endpoint mode rather than returning authoritative metadata from the loaded sub-GGUF. Its INFO response reports a generic full range and both endpoint flags, while Python advertises CLI values:
- `experiments/v0.0/worker_daemon.cpp:387-392`.
- `experiments/v0.0/worker_daemon.cpp:466-475`.
- `scripts/worker.py:1141-1153`.
Additional integration gaps:
- Normal activations are little-endian F32 even though the protobuf comment says FP16 is the default; int8 activation mode is environment-global rather than negotiated.
- The coordinator requires full-GGUF access through `llama-cpp-python` for tokenization.
- Tail sampling is hard-coded greedy top-1 rather than returning general logits.
- Many tests use fake daemons; no CI job builds patched llama.cpp, generates slices, starts two real workers, and tests prefill/decode/failure.
- Nakshatra runtime dependencies are absent from the inherited Petals package metadata.
Evidence:
- `proto/nakshatra.proto:11-12`.
- `scripts/worker.py:340-343`.
- `scripts/worker.py:1141-1151`.
- `scripts/client.py:129-139`.
- `scripts/client.py:719-722`.
- `experiments/v0.0/worker_daemon.cpp:617-641`.
- `tests/test_worker_eagle_sleep_rpc.py:1-31`.
- `setup.cfg:1-16`.
- `setup.cfg:29-72`.
These defects make Nakshatra a strong source donor and independent feasibility proof, but not the repository to fork as the production base.
### 5.11 Reuse decision
Do not recreate Nakshatra's working Llama-family layer patch and daemon from scratch without first attempting upstream collaboration.
Preferred plan:
1. Reproduce its two-worker path locally.
2. Rebase its patch against the exact current llama.cpp commit already audited.
3. Compare the patch with the proposed `llama_model_load_range` and boundary-output design.
4. Borrow or jointly maintain narrow patch concepts and tests, but keep a project-owned standalone worker and small pinned llama.cpp fork rather than forking the Petals-derived repository wholesale.
5. Replace or adapt its Python control plane with existing Meshnet tracker/session/relay/billing infrastructure.
6. Add multi-session KV isolation, exact recipe identity, route epochs, cancellation, signed work receipts, and architecture certification.
7. Upstream generic llama.cpp range-loading/boundary hooks where maintainers will accept them.
Classification:
```text
Primary source donor and collaboration candidate
Do not adopt or fork the repository wholesale
```
## 6. LiGGUF SARA
### 6.1 Verdict
LiGGUF contains a real experimental networked distributed implementation. It is not layer pipeline parallelism.
Its SARA implementation performs tensor-parallel activation reduction across every transformer layer.
### 6.2 Mechanism
Every process mmaps and parses the complete GGUF and constructs pointers for all transformer blocks:
- `cpp/ligguf_distrib.cpp:323-485`.
Rank assignments split:
- KV heads.
- Query heads derived from KV heads.
- FFN channel blocks.
Evidence:
- `cpp/ligguf_distrib.cpp:120-124`.
- `cpp/ligguf_distrib.cpp:657-679`.
Each rank allocates KV for its attention-head slice across every layer:
- `cpp/ligguf_distrib.cpp:681-707`.
For every transformer layer, the master:
1. Normalizes the residual.
2. Broadcasts a Q8_0 activation to all workers.
3. Every rank computes its attention partial.
4. Workers return Q8_0 full-width partials.
5. The master sums partials into the residual.
6. Repeats the same process for the FFN.
Evidence:
- `cpp/ligguf_distrib.cpp:709-767`.
- `cpp/ligguf_distrib.cpp:892-903`.
- `cpp/ligguf_distrib.cpp:957-974`.
- `cpp/ligguf_distrib.cpp:1025-1066`.
### 6.3 Strengths
- Very compact and readable.
- Direct GGUF mmap.
- Real raw-TCP master/worker implementation.
- Q8_0 activation and partial transport.
- Local KV per attention-head shard.
- Useful reference for tensor-parallel reductions on CPU-heavy edge systems.
### 6.4 Mismatch
- Every worker needs the complete GGUF.
- Every worker participates in every layer.
- Two network reduction phases occur per transformer layer.
- Static rank/world topology.
- One master connection and one generation at a time.
- Sequential worker receive loop.
- No model/session/request/range identity.
- No authentication, checksums, recovery, cancellation, or accounting.
- Custom inference core has much narrower model/kernel coverage than llama.cpp.
Evidence:
- `cpp/ligguf_distrib.cpp:769-974`.
- `cpp/ligguf_distrib.cpp:1025-1046`.
- `cpp/ligguf_distrib.cpp:1138-1241`.
Decision: retain as a source donor for compact Q8 activation transport, tensor-partition tests, and reduction benchmarking. Do not use it as the native Meshnet layer worker.
## 7. Chameleon
Audited snapshot:
```text
megeezy/Chameleon
commit 96fbd96a9f67d29d12292d3373c88996aba65f84
```
Chameleon is a whole-model lifecycle and routing system:
- Coordinator selects a model and worker.
- Python worker loads a complete llama-cpp-python, vLLM, Transformers, or ExLlamaV2 backend.
- The model executes, can remain warm, and is later unloaded.
It does not split one model's layers or tensors across workers. Its README also labels it design phase.
Decision: exclude from the distributed-GGUF implementation list. Whole-model load/unload and warm-cache ideas may be relevant to proxy backends only.
## 8. Continuum
Audited snapshot:
```text
CambrianTech/continuum
commit dd976df36079d75244719a23956e1c9e2dcddc27
```
Continuum has:
- A local GGUF loader.
- A local inference backend.
- Mesh/federation infrastructure.
- Roadmap language about dividing models across nodes.
No executable distributed GGUF layer or tensor data plane was found in this snapshot. There is no layer-range protocol, boundary-activation transport, distributed KV ownership, or distributed GGUF test corresponding to the roadmap statement.
Decision: exclude until source and real execution evidence exist.
## 9. GitHub search conclusion
The expanded searches used terms including:
- `distributed GGUF`.
- `sub-GGUF`.
- `layer range` with llama.cpp.
- `result_partial_hidden`.
- `activation chain`.
- llama.cpp RPC orchestration.
- GGUF tensor parallelism.
The working implementation families found are:
| Family | Projects | What is real |
|---|---|---|
| Coordinator-owned remote devices | llama.cpp RPC, historical GPUStack/llama-box, LocalAI integrations | Full GGML graph controlled by one coordinator |
| Independent GGUF layer pipeline | Nakshatra, prima.cpp | Local layer ownership, residual boundaries, local KV |
| Custom Rust layer pipeline | `llama-gguf` | gRPC layers, but coordinator-streamed weights and weak session semantics |
| Tensor parallel GGUF | LiGGUF SARA | Head/FFN partitions and per-layer partial reductions |
| Static/custom non-GGUF tensor engines | distributed-llama/dllama and similar | Useful algorithms, different artifacts/runtime |
| Whole-model routing | Chameleon, Ollama, most LocalAI use, current GPUStack backends | Independent complete models, not one split model |
| Roadmap-only mesh claims | Continuum and several search hits | No executable distributed GGUF data plane found |
No additional mature project was found that already combines:
- Arbitrary tracker-selected layer intervals.
- Exact local GGUF artifact ownership.
- Heterogeneous volunteer nodes.
- Concurrent route-session KV.
- Dynamic route epochs and recovery.
- Relay and cancellation.
- Exact runtime/activation compatibility.
- Worker-authenticated accounting.
Nakshatra is the closest and materially reduces the amount of new inference-engine work required.
## 10. Revised architecture decision
The selected runtime remains llama.cpp/GGML, but the implementation source priority changes:
```text
Existing Meshnet tracker, relay, sessions, capability admission, and billing
|
Meshnet-owned stable shard protocol
|
project-owned C++ worker borrowing narrow Nakshatra concepts/tests
|
narrow, pinned, architecture-certified llama.cpp patch
|
GGUF loader, quant kernels, KV, CPU/GPU backends
```
This is not an architectural pivot away from the initial decision. Nakshatra is an external implementation of nearly the same missing seam.
## 11. Revised implementation sequence
### 11.1 Reproduction and patch comparison
- Reproduce Nakshatra's two local workers with a small dense Llama-family GGUF.
- Verify disjoint sub-GGUF weight ownership.
- Verify boundary residual parity with whole-model llama.cpp.
- Rebase against the pinned current llama.cpp commit.
- Compare pre-sliced GGUF loading with range-aware loading from one source artifact.
### 11.2 Meshnet protocol adapter
- Keep the project-owned llama.cpp worker behind a supervised executable.
- Translate Meshnet request/session/route metadata into worker calls.
- Preserve BF16 or versioned named-tensor bundles on the public network boundary.
- Add exact source/slice/runtime recipe checks.
- Reject stale route epochs and incompatible caches.
### 11.3 Multi-session KV
- Map each route session/epoch to an isolated `llama_seq_id` or context.
- Test concurrent prefill and decode.
- Add bounded release, TTL, LRU, and cache-miss semantics.
- Verify only owned layers allocate KV.
### 11.4 Real heterogeneous route
- CPU plus AMD HIP/ROCm first on the available machine.
- Add CUDA, Vulkan, and Metal as certified lanes when hardware is available.
- Measure numerical drift and define compatibility classes.
- Require a clean from-token-zero restart when exact-recipe recovery is unavailable.
### 11.5 Trustworthy accounting
- Worker signs work receipts.
- Receipt binds route, request, artifact, range, activation digests, positions, and runtime recipe.
- Tracker validates structural consistency and completion evidence before rewards.
### 11.6 Architecture expansion
- Dense Llama-family first.
- Explicit Qwen3/Qwen3-MoE adapter next.
- Fail closed for all unvalidated architectures.
## 12. Local validation performed
The source audit included limited executable verification without downloading model artifacts:
```text
LiGGUF distributed target:
make ligguf-cpp-distrib
PASS — g++ produced ligguf-cpp-distrib
Nakshatra dependency-free focused tests:
47 passed in 0.42s
```
The Nakshatra subset covered receipt validation, wire-version behavior, wire handshake behavior, and topology ordering.
The networked idempotency integration test was not collected because this audit environment does not have the optional `grpcio` package installed. This is an environment dependency blocker, not a test assertion failure. No real Nakshatra model inference was run locally during this research; the real-model conclusions remain tied to the source, recorded experiment evidence, and the future reproduction gate in section 11.
## 13. Final conclusions
1. The user's central practical observation is correct: llama.cpp is the base of most real-world GGUF distribution found.
2. GPUStack 0.4 was open source and genuinely distributed, but through llama-box/llama.cpp RPC rather than independent shards.
3. GPUStack 2.0 removed distributed GGUF support, so the old tutorial must be treated as historical.
4. Nakshatra is the most important new source. It has already implemented and exercised the narrow llama.cpp layer-worker design.
5. Nakshatra should be approached for narrow collaboration and mined for source/tests, but its repository should not be adopted or forked wholesale.
6. Nakshatra still needs Meshnet-specific hardening: exact identity, concurrent KV, route epochs, cancellation, accounting, and architecture certification.
7. LiGGUF SARA is real distributed GGUF tensor parallelism, but every worker loads the whole model and communicates twice per layer.
8. Chameleon is whole-model routing; Continuum's distributed GGUF is roadmap-only in the audited snapshot.
9. No drop-in project yet satisfies the complete tracker-routed volunteer-network contract.
10. The implementation risk is now lower because the hardest first proof—partial llama.cpp layer loading plus boundary execution—has independent working source and cross-machine evidence.
Reference in New Issue View Git Blame Copy Permalink