docs: define distributed GGUF runtime plan

2026-07-13 15:09:27 +03:00
parent b5fa7245df
commit 4cae4a6c5c
42 changed files with 4913 additions and 691 deletions
--- a/docs/research/distributed-gguf-landscape.md
+++ b/docs/research/distributed-gguf-landscape.md
@@ -0,0 +1,829 @@
+# Distributed GGUF inference: existing projects, source audits, and implementation direction
+
+Status: Research complete; architecture direction selected
+Last updated: 2026-07-13
+
+## 1. Purpose
+
+This document records the research behind native distributed GGUF inference for the neuron-tai network. It is intentionally not a design for proxying an already-running whole-model `llama-server`, Ollama, vLLM, or another OpenAI-compatible runtime. Whole-model proxying is useful as a correctness/performance baseline, but it does not solve the network's central problem.
+
+The required product is a tracker-routed distributed GGUF data plane in which heterogeneous nodes independently execute model layer ranges, exchange boundary activations, retain cache state for their own layers, and receive credit for work they actually perform.
+
+### 1.1 Required contract
+
+A satisfactory implementation must support:
+
+- Native GGUF model artifacts and llama.cpp-compatible quantizations.
+- Independently loaded, executable layer ranges on each node.
+- Tracker-selected routes rather than one static rank topology.
+- Heterogeneous CPU, CUDA, HIP/ROCm, Vulkan, Metal, and other certified lanes.
+- A versioned hidden-state activation boundary between nodes.
+- Local per-shard KV or recurrent state keyed by route session.
+- Prefill and decode phases with bounded wire and compute cost.
+- Overlap-safe execution using the tracker's effective `start_layer` semantics.
+- Explicit cache miss, eviction, route epoch, and recovery behavior.
+- Dynamic health and route failure handling.
+- Per-node work telemetry and accounting.
+- Model-agnostic infrastructure with architecture-specific certification.
+
+### 1.2 Non-goals
+
+- Reimplementing GGUF parsing, quantization kernels, tokenization, or mature CPU/GPU kernels from scratch.
+- Treating a whole-model node as the distributed solution.
+- Treating stock llama.cpp RPC as a safe volunteer-network protocol.
+- Claiming every GGUF architecture works without a bounded real distributed validation.
+- Mixing Transformers and GGUF shards in one route without an explicit compatible activation contract.
+
+### 1.3 Parallelism taxonomy
+
+These mechanisms must not be conflated:
+
+- **Layer or pipeline parallelism** assigns whole contiguous transformer-layer ranges to different stages and transports boundary activations between them. This is the closest match to neuron-tai's tracker-routed shard contract.
+- **Tensor parallelism** partitions operations or tensors within each layer, such as attention heads, matrix rows/columns, or experts. It usually requires all ranks for every layer and collective or reduction traffic inside every layer.
+- **Local multi-device offload** places tensors or layers across devices controlled by one process. llama.cpp's `n_gpu_layers` and `tensor_split` belong here unless RPC devices are included.
+- **llama.cpp RPC offload** exposes remote GGML devices to a coordinator-owned graph. It is cross-machine, but remote processes are devices rather than independent model/session workers.
+- **Whole-model replication** runs one complete model per server and load-balances requests. It increases throughput and availability but does not allow one oversized model to span workers.
+
+GGUF weight quantization such as Q4_K_M or Q8_0 reduces storage and memory pressure. It does not define the activation dtype, compute dtype, KV-cache dtype, or distributed topology.
+
+## 2. Existing neuron-tai infrastructure
+
+The repository already contains most of the control plane needed by a GGUF shard worker.
+
+### 2.1 Backend and execution path
+
+There is no formal backend interface yet. Production code uses the concrete `TorchModelShard`, while several call sites rely on duck typing.
+
+Relevant source:
+
+- `packages/node/meshnet_node/model_backend.py:72-223` — result types, session cache, and concrete backend.
+- `packages/node/meshnet_node/model_backend.py:226-345` — Torch/Hugging Face construction.
+- `packages/node/meshnet_node/model_backend.py:347-566` — effective head, middle, tail, and whole-model backend methods.
+- `packages/node/meshnet_node/model_backend.py:730-747` — Torch factory.
+- `packages/node/meshnet_node/torch_server.py:302-317` — injected backend usage.
+- `packages/node/meshnet_node/torch_server.py:717-766` — whole-model generation fast path.
+- `packages/node/meshnet_node/torch_server.py:1464-1514` — server construction.
+- `packages/node/meshnet_node/torch_server.py:1638-1659` — factory hard-wired to `load_torch_shard`.
+
+The current effective distributed backend contract includes:
+
+- `model_id`, `shard_start`, `shard_end`, `total_layers`, `is_head`, `is_tail`, and `device`.
+- `encode_prompt`, `encode_next_token`, `forward_bytes`, and `decode_tail_token`.
+- `eos_token_ids` and `release_session`.
+- Whole-model generation and token-count helpers.
+
+A GGUF design should separate a generic backend/process lifecycle interface from an optional activation-shard interface rather than making every backend pretend to be `TorchModelShard`.
+
+### 2.2 Distributed activation wire
+
+The current data plane already provides a reusable envelope:
+
+- Binary `POST /forward` requests.
+- `X-Meshnet-Session` route-session identity.
+- `X-Meshnet-Cache: prefill | decode`.
+- `X-Meshnet-Past-Len` cache consistency check.
+- `X-Meshnet-Start-Layer` overlap-safe execution.
+- Explicit tensor shape, dtype, position IDs, attention mask, chunk, and compression metadata.
+- Direct and relay downstream clients owned by the generation handler.
+- HTTP 409 cache-miss responses and re-prefill recovery.
+
+Relevant source:
+
+- `packages/node/meshnet_node/server.py:13-76` — wire contract and validation.
+- `packages/node/meshnet_node/torch_server.py:497-635` — binary handler.
+- `packages/node/meshnet_node/torch_server.py:974-1288` — route parsing and hop execution.
+- `packages/tracker/meshnet_tracker/server.py:3635-3781` — route planning and effective start layers.
+- `docs/adr/0008-binary-activation-wire-format.md`.
+- `docs/adr/0012-start-layer-overlapping-shards.md`.
+
+The current payload semantics are still Torch/Hugging Face-specific:
+
+- Boundary dtype is fixed to BF16.
+- Tensor conversion imports Torch.
+- Attention masks and position IDs use Torch-oriented serialization.
+- Tail-local decode reconstructs a Torch tensor.
+
+The HTTP envelope is reusable; backend-neutral activation semantics need to be made explicit.
+
+### 2.3 Local shard cache
+
+The existing Transformers path already has the desired session behavior:
+
+- One stable UUID per distributed generation.
+- Prefill establishes state on every shard.
+- Decode forwards only the new-token activation.
+- Every shard stores only its own layer state.
+- Cache lookup checks sequence length and effective start layer.
+- Cache miss, restart, or route mismatch returns HTTP 409.
+- The head re-prefills accumulated tokens after a miss.
+- TTL and LRU bound local memory.
+
+Relevant source:
+
+- `packages/node/meshnet_node/model_backend.py:102-193`.
+- `packages/node/meshnet_node/model_backend.py:334-345`.
+- `packages/node/meshnet_node/model_backend.py:595-662`.
+- `packages/node/meshnet_node/torch_server.py:806-944`.
+- `docs/adr/0022-sharded-per-node-kv-cache.md`.
+
+A GGUF worker should preserve this product-level contract while mapping route sessions to llama.cpp sequence IDs or isolated contexts.
+
+### 2.4 Tracker and capability admission
+
+The tracker already supports:
+
+- Model and layer-range registration.
+- Capability reports and fail-closed admission.
+- Heterogeneous route candidates.
+- Greedy interval coverage.
+- Effective start-layer injection.
+- Throughput/load/reputation-informed selection.
+- Dynamic health and route statistics.
+- Accounting and work attribution.
+
+Relevant source:
+
+- `packages/tracker/meshnet_tracker/server.py:592-942`.
+- `packages/tracker/meshnet_tracker/server.py:3635-3781`.
+- `packages/tracker/meshnet_tracker/server.py:4425-4595`.
+- `packages/tracker/meshnet_tracker/server.py:6200-6285`.
+- `docs/adr/0021-dynamic-statistical-routing.md`.
+- `docs/adr/0023-model-agnostic-node-capability-admission.md`.
+
+GGUF gaps include:
+
+- The closed `bfloat16`, `int8`, `nf4` precision vocabulary.
+- No first-class route kind or activation compatibility key.
+- No GGUF artifact descriptor.
+- No llama.cpp/Vulkan/HIP execution recipe.
+- No model-aware GGUF memory estimator.
+- No real llama.cpp doctor benchmark.
+
+## 3. Ecosystem survey
+
+No mature project was found that combines native GGUF, arbitrary tracker-selected layer ranges, heterogeneous cross-machine execution, independent local shard cache, dynamic route sessions, recovery, and per-node accounting.
+
+A later GitHub follow-up found that [Nakshatra](https://github.com/fthrvi/nakshatra) already implements the narrow patched-llama.cpp Llama-family layer-worker seam and should be treated as the primary source donor and collaboration candidate. Its repository is not self-contained or production-ready enough to adopt as the runtime base. Historical GPUStack/llama-box and LiGGUF SARA were also audited in [the GitHub follow-up](distributed-gguf-github-followup.md). This reduces implementation risk but does not provide a drop-in Meshnet runtime.
+
+### 3.1 Candidate summary
+
+| Project | Native GGUF | Distributed form | Direct fit | Decision |
+|---|---:|---|---|---|
+| [llama.cpp](https://github.com/ggml-org/llama.cpp) | Yes | Device offload/RPC, local multi-GPU | Best kernel, wrong stock topology | Use as primary kernel through a small pinned fork |
+| [Nakshatra](https://github.com/fthrvi/nakshatra) | Yes, via sub-GGUFs and patched llama.cpp | Independent contiguous layer workers over gRPC | Closest implementation found | Primary source donor/collaboration candidate; do not adopt its repository wholesale |
+| [llama-gguf](https://github.com/Lexmata/llama-gguf) | Yes | gRPC layer pipeline | Architecturally close, immature custom runtime | Source donor only |
+| [prima.cpp](https://github.com/OpenCPIL/prima.cpp) | Yes | Static piped-ring layer windows | Proves partial loading and local KV | Source donor only |
+| [LiGGUF](https://github.com/matrixsmaster/ligguf) | Yes | SARA head/FFN tensor sharding and activation reduction | Every rank loads the full model; static master/workers | Compact tensor-parallel source donor only |
+| [mistral.rs](https://github.com/EricLBuehler/mistral.rs) | Yes | NCCL TP and multi-machine ring | Static/homogeneous distributed assumptions | Evaluate as optional homogeneous backend |
+| [vLLM](https://github.com/vllm-project/vllm) | Experimental/plugin | TP, PP, DP, EP via Ray/Torch distributed | Production clusters, not volunteer layer routes | Whole-model/managed-cluster lane and design donor; see [vLLM assessment](vllm-distributed-gguf-assessment.md) |
+| [distributed-llama](https://github.com/b4rtaz/distributed-llama) | No, custom format | Root/worker tensor parallelism | Power-of-two/static topology | Ideas only |
+| [Petals](https://github.com/bigscience-workshop/petals) | No | Internet layer pipeline | Strong cache/session semantics | Conceptual donor |
+| [exo](https://github.com/exo-explore/exo) | No native path | MLX tensor/pipeline sharding | MLX-centric and platform-asymmetric | Placement/cache ideas only |
+| [LocalAI](https://github.com/mudler/LocalAI) | Via llama.cpp | RPC workers and request routing | Control-plane overlap | Lifecycle ideas only |
+| [GPUStack](https://github.com/gpustack/gpustack) 0.4-0.7 | Via llama-box/RPC | Scheduler + primary/RPC workers | Proven RPC orchestration, not layer workers | Historical reference; GGUF distribution removed in GPUStack 2.0 |
+| [llama-box](https://github.com/gpustack/llama-box) | Yes | llama.cpp RPC | Archived | Historical source donor only |
+| [Chameleon](https://github.com/megeezy/Chameleon) | Via whole-model backends | Whole-model worker lifecycle/routing | No single-model sharding | Exclude from distributed-GGUF candidates |
+| [Continuum](https://github.com/CambrianTech/continuum) | Local loader/backend | Mesh orchestration; tensor distribution is roadmap text | No distributed GGUF execution path found | Exclude until executable evidence exists |
+| Ollama | Yes | Local multi-device; no cross-host model split | Whole-model only | Proxy baseline only |
+| MLC LLM | No native GGUF | Compiled local/multi-GPU | Different artifact/runtime | Separate recipe at most |
+
+### 3.2 llama.cpp RPC
+
+Stock RPC exposes remote GGML devices to one coordinator. Current upstream documentation says it can distribute weights and KV across local and remote devices and can cache remote tensors with `ggml-rpc-server -c`. This makes the original claim in `docs/adr/0001-pytorch-over-llama-cpp.md`—that every launch must always resend all weights—outdated.
+
+RPC still does not provide:
+
+- Independently loaded local GGUF layer workers.
+- Tracker-selected per-request routes.
+- A hidden-state HTTP endpoint.
+- Route-session-owned shard cache.
+- Per-worker tracker accounting.
+- A safe untrusted volunteer-node boundary.
+
+Upstream explicitly calls RPC proof-of-concept, fragile, and insecure. It is not the target architecture.
+
+### 3.3 Petals
+
+Petals remains the strongest conceptual reference for:
+
+- Hosting independent transformer block ranges.
+- Per-server cache state.
+- Session-based inference.
+- Rebuilding cache after route failure.
+- Capacity-aware block placement.
+- Public/private swarm control.
+
+It is PyTorch/Transformers-based, not GGUF, and is no longer active enough to adopt as the runtime. The existing neuron-tai Transformers path already implements many of its core semantics.
+
+## 4. Source audit: prima.cpp
+
+Audited source snapshot:
+
+```text
+OpenCPIL/prima.cpp
+commit 6f9b7c40962d777d1726456b4359340d932bef12
+```
+
+### 4.1 Decision
+
+Do not fork prima.cpp wholesale. Extract partial-loading, local-KV, graph-boundary, and placement ideas into a small fork of current llama.cpp.
+
+prima.cpp is a full invasive llama.cpp fork. Its exact upstream ancestry could not be established from the available repository history. Distributed changes are embedded in public structures, `src/llama.cpp`, common CLI code, and the old server.
+
+### 4.2 Layer ownership
+
+prima.cpp adds static rank topology fields:
+
+- `n_world`.
+- `rank`.
+- Fixed `n_layer_window[32]` arrays.
+- Cycle count and fixed network addresses/ports.
+
+Evidence:
+
+- `include/llama.h:292-353`.
+- `src/llama.cpp:2597-2636`.
+- `common/arg.cpp:680-785`.
+
+Layer ownership is a repeated cyclic window, not an arbitrary tracker-provided `[start,end)` route:
+
+- `src/llama.cpp:3838-3883`.
+- `src/llama.cpp:16941-16951`.
+
+This cannot directly represent dynamic tracker routes.
+
+### 4.3 Partial local GGUF loading
+
+This is prima.cpp's strongest reusable contribution.
+
+- Rank zero creates token embedding, final norm, and output tensors.
+- Transformer tensors are created only when the layer belongs to the process.
+- Global layer IDs are compacted into local storage.
+- Unneeded GGUF tensors are erased before mapping/loading.
+- Only selected file regions are mapped and loaded.
+
+Evidence:
+
+- `src/llama.cpp:7500-7619`.
+- `src/llama.cpp:9358-9515`.
+
+This proves a llama.cpp-derived process can independently load only the GGUF tensors required by its layer range.
+
+### 4.4 Graph boundaries
+
+prima.cpp introduces explicit intermediate input/output tensors and skips unowned layers:
+
+- `src/llama.cpp:10793-10813`.
+- `src/llama.cpp:11000-11117`.
+- `src/llama.cpp:16953-17010`.
+
+The distributed graph path is hard-asserted to Llama or Qwen2 despite inherited architecture enums. Architecture support is therefore narrow.
+
+### 4.5 Activation transport
+
+Transport is raw ZeroMQ multipart PUSH/PULL. Messages include native shapes and raw contiguous F32 activation bytes.
+
+Evidence:
+
+- `src/llama.cpp:18031-18077`.
+- `src/llama.cpp:18542-18563`.
+
+It has no protocol version, model/session/route identity, layer range, dtype field, checksum, authentication, compression, or robust payload validation. It is incompatible with the existing Meshnet BF16 route-session protocol.
+
+### 4.6 Local KV
+
+Each process has one local llama.cpp context and allocates KV only for layers assigned to that process.
+
+Evidence:
+
+- `src/llama.cpp:3889-3969`.
+- `src/llama.cpp:18268-18269`.
+
+This is valuable physical behavior, but cache operations are coupled to one static ring and server slot numbering. There is no route session, route epoch, model fingerprint, range identity, lease, or idempotency contract.
+
+### 4.7 Fault model and backends
+
+Runtime sends and receives are blocking. Socket failures can terminate or stall a process. There is no runtime route repair, KV replay, deduplication, or accounting.
+
+The inherited tree contains many backends, but the distributed profiler and placement solver primarily model CPU, CUDA, and Metal. Project documentation excludes AMD/Vulkan distributed support.
+
+Evidence:
+
+- `src/llama.cpp:18031-18077`.
+- `src/llama.cpp:20492-20769`.
+- `common/profiler.h:329-405`.
+- `common/common.cpp:850-1183`.
+- `README.md:362-368`.
+
+### 4.8 Tests
+
+No meaningful distributed regression suite was found for layer windows, transport, cache propagation, heterogeneous placement, session isolation, or failure behavior.
+
+### 4.9 Reusable parts
+
+Port or use as design references:
+
+- Selective endpoint/head/tail tensor creation.
+- Selected-layer GGUF mapping and loading.
+- Global-to-local layer indexing.
+- Per-layer KV filtering and allocation.
+- Boundary residual input/output.
+- Placement cost ideas and page prefetching.
+
+Do not reuse:
+
+- Static ring topology.
+- Fixed rank arrays.
+- ZeroMQ wire protocol.
+- Server patches.
+- Cache-command propagation.
+- Failure semantics.
+
+## 5. Source audit: llama-gguf
+
+Audited source snapshot:
+
+```text
+Lexmata/llama-gguf
+commit 6e9f194206450080d47101c6f88a80f604ce69be
+```
+
+### 5.1 Decision
+
+Do not adopt `llama-gguf` as the inference runtime or distributed data plane. Use it as a source donor for protobuf organization, health/capability messages, explicit ranges, and synthetic multi-process tests.
+
+### 5.2 Coordinator owns and streams the model
+
+The coordinator loads and builds the entire GGUF model, then serializes layer tensors and streams them to shards over gRPC.
+
+Evidence:
+
+- `src/distributed/coordinator.rs:44-53`.
+- `src/distributed/coordinator.rs:160-168`.
+- `src/distributed/coordinator.rs:196-299`.
+
+Workers explicitly do not open local GGUF files:
+
+- `src/distributed/shard.rs:53-59`.
+
+This conflicts with independently owned local artifacts, startup efficiency, and tracker-controlled model distribution.
+
+### 5.3 Explicit ranges but simple coverage validation
+
+The cluster config supports explicit half-open ranges or even auto-partitioning:
+
+- `src/distributed/config.rs:65-92`.
+- `src/distributed/config.rs:168-220`.
+
+Manual validation checks total assigned layer count but does not prove sorted, gap-free, non-overlapping coverage. It is a static startup configuration, not a dynamic per-request route.
+
+### 5.4 Shard-local KV, but only one global sequence
+
+A worker allocates one KV cache for its local layer count:
+
+- `src/distributed/shard.rs:30-44`.
+- `src/distributed/shard.rs:261-286`.
+- `src/model/mod.rs:62-117`.
+
+There is no cache map or route-session identity. `ResetKvCache` takes an empty request and resets the one global cache:
+
+- `src/distributed/shard.rs:447-458`.
+- `proto/distributed.proto:15-16`.
+- `proto/distributed.proto:95-103`.
+
+Concurrent sessions, cache epochs, stale requests, and per-request eviction are unsupported.
+
+### 5.5 Forward protocol
+
+`ForwardRequest` contains only hidden state, position, and sequence length:
+
+- `proto/distributed.proto:77-93`.
+
+It has no model fingerprint, route/session/request identity, layer range, phase, chunk, expected cache length, accounting identity, idempotency key, or route epoch.
+
+The tensor envelope does include shape, dtype, little-endian bytes, and a name:
+
+- `proto/distributed.proto:36-46`.
+- `src/distributed/tensor_transfer.rs:10-115`.
+
+The DType enum can represent BF16, but the distributed execution path and tests use F32 hidden states and F32 KV.
+
+### 5.6 Token-by-token serialized execution
+
+The distributed model loops over tokens one at a time and sends each hidden vector through each shard. The complete pipeline is protected by one mutex.
+
+Evidence:
+
+- `src/distributed/model.rs:21-39`.
+- `src/distributed/model.rs:86-147`.
+- `src/distributed/pipeline.rs:45-113`.
+
+This prevents efficient batched prefill and safe session multiplexing.
+
+### 5.7 Architecture claims versus shard reconstruction
+
+The repository has a large architecture enum, but the distributed shard reconstructs a simple dense layer consisting of RMSNorm, ordinary attention, and dense gated FFN.
+
+Evidence:
+
+- `src/model/architecture.rs:5-155`.
+- `src/distributed/shard.rs:154-223`.
+
+The worker cannot reconstruct many architecture-specific layer forms such as MoE, hybrid/recurrent layers, shared cache, architecture-specific norms, or specialized projections. Qwen3 MoE support cannot be inferred from the enum list.
+
+### 5.8 Backends
+
+Shard backend selection tries CUDA, Metal, DX12, Vulkan, then CPU, but the source notes GPU model-weight preloading is skipped because weights arrive as gRPC tensors. The per-operation backend path is not equivalent to llama.cpp's mature graph scheduling.
+
+Evidence:
+
+- `src/distributed/shard.rs:53-115`.
+- `src/backend/mod.rs:25-260`.
+
+### 5.9 Fault recovery
+
+The repository has health monitoring and can reconnect, reconfigure, and resend layers:
+
+- `src/distributed/fault.rs:83-360`.
+
+It does not restore active KV, route position, partial prefill, session ownership, or accounting state. Reloading a process is not active-generation recovery.
+
+### 5.10 Tests
+
+Distributed integration tests start local CPU gRPC shards with synthetic dense F32 weights. They verify configure, load, forward, reset, and a two-shard pipeline.
+
+The file explicitly states that no real GGUF is loaded:
+
+- `tests/distributed_integration_test.rs:1-7`.
+
+Missing evidence includes real quantized GGUF parity, heterogeneous devices, BF16 boundaries, batched prefill, concurrent sessions, worker loss, route epochs, local artifact loading, and cache recovery.
+
+### 5.11 Reusable parts
+
+Potential donors:
+
+- Protobuf tensor shape/dtype/data envelope.
+- Explicit half-open layer ranges.
+- Health and capabilities messages.
+- Synthetic multi-process shard test harness.
+- Separation of configure, load, forward, reset, and health methods.
+
+Do not reuse:
+
+- Coordinator-side weight streaming.
+- Custom inference engine as the production kernel.
+- Single global KV cache.
+- Token-by-token pipeline.
+- Dense-only layer reconstruction.
+- Fault-reload semantics.
+- The all-reduce implementation, which currently echoes its input at `src/distributed/shard.rs:522-541`.
+
+## 6. Source audit: current llama.cpp
+
+Audited source snapshot:
+
+```text
+ggml-org/llama.cpp
+commit 91c631b21d6e5d09e9c6659efdf6baeef5a44ddb
+```
+
+### 6.1 Decision
+
+The smallest credible implementation is a pinned, architecture-certified layer-worker fork of current llama.cpp. A fully generic all-architecture patch is not credible as the first implementation.
+
+### 6.2 Layer-range API
+
+Prefer a new range loader that isolates ABI churn:
+
+```c
+llama_model * llama_model_load_range(
+    const char * path,
+    llama_model_params params,
+    int32_t il_start,
+    int32_t il_end);
+```
+
+Relevant source:
+
+- `include/llama.h:295-331`.
+- `src/llama-model.cpp:2300-2319`.
+
+Adding fields directly to the by-value public parameter structure changes ABI. A project-owned API around internal model state is safer for a pinned worker executable.
+
+### 6.3 Partial weight loading
+
+Current loader behavior already makes a small patch possible:
+
+- Architectures create/register tensors.
+- Backend buffers are allocated only for created tensor contexts.
+- Mmap ranges derive from created tensors.
+- `done_getting_tensors(partial=true)` exists.
+- Data loading only processes tensors present in created contexts.
+
+Relevant source:
+
+- `src/llama-model.cpp:1229-1647`.
+- `src/llama-model-loader.cpp:1054-1572`.
+
+For an initial Llama-family adapter:
+
+- Create token embedding only if `il_start == 0`.
+- Create final norm/output only if `il_end == n_layer`.
+- Create repeating tensors only for `[il_start, il_end)`.
+- Finish model loading with partial mode.
+
+Relevant source:
+
+- `src/models/llama.cpp:34-92`.
+- `src/llama-model.cpp:1490`.
+
+### 6.4 Residual input
+
+Current input machinery supports F32 supplied embeddings:
+
+- `src/llama-graph.h:120-150`.
+- `src/llama-graph.cpp:66-121`.
+- `src/llama-graph.cpp:2151-2229`.
+
+A middle shard needs a dedicated residual input that bypasses token lookup, embedding scaling, LoRA, and padding behavior while still accepting position and sequence IDs for RoPE/KV.
+
+### 6.5 Ranged graph execution
+
+The initial Llama graph loop is in:
+
+- `src/models/llama.cpp:98-247`.
+
+It must:
+
+- Iterate only `[il_start, il_end)`.
+- Preserve all boundary rows for intermediate shards.
+- Avoid final-token-only row pruning except on the actual tail.
+- Run final norm and LM head only on the actual tail.
+
+### 6.6 Boundary output
+
+The network boundary is the unnormalized residual stream after the final owned transformer layer, not llama.cpp's final embedding tensor.
+
+Add a dedicated graph result such as `t_boundary` in:
+
+- `src/llama-graph.h:791-865`.
+- `src/llama-graph.cpp:1190-1251`.
+- `src/llama-context.cpp:2206-2231`.
+
+Existing `llama-ext` layer-input extraction is useful precedent but is explicitly staging/WIP and only populated by some architectures.
+
+### 6.7 Shard-local KV
+
+The standard KV cache already supports a layer filter and compact global-to-local mapping:
+
+- `src/llama-kv-cache.cpp:64-100`.
+- `src/llama-kv-cache.cpp:163-248`.
+- `src/llama-kv-cache.cpp:1210-1327`.
+- `src/llama-model.cpp:2152-2266`.
+
+For a standard Llama adapter, the filter is conceptually:
+
+```cpp
+return il >= il_start && il < il_end;
+```
+
+Do not apply this blindly to hybrid, recurrent, shared-cache, MLA, or other architecture-specific memory implementations.
+
+### 6.8 Route-session mapping
+
+Existing sequence APIs support:
+
+- Sequence removal, copy, keep, and position operations.
+- Sequence state get/set.
+- Partial/on-device state flags.
+
+Relevant source:
+
+- `include/llama.h:720-913`.
+- `src/llama-context.cpp:2900-2968`.
+- `src/llama-context.cpp:3978-4024`.
+
+A shard worker can map `(route_session_id, route_epoch)` to a stable `llama_seq_id` or one isolated context. Because the model contains only filtered layers, sequence state naturally represents shard-local cache.
+
+### 6.9 Scheduler
+
+No scheduler rewrite should be required. The current graph scheduler can allocate a new residual input and boundary output if they are registered correctly.
+
+Relevant source:
+
+- `src/llama-context.cpp:1286-1355`.
+- `src/llama-context.cpp:2324-2459`.
+
+### 6.10 Architecture coupling
+
+A fully generic minimal patch is not plausible:
+
+- The audited tree has roughly 136 model implementation files.
+- More than one hundred contain architecture-specific layer loops.
+- Architectures differ in cache, residual, attention, MoE, recurrent, multimodal, and output behavior.
+
+Generic infrastructure is appropriate for:
+
+- Range validation.
+- Tensor filtering.
+- Residual input/output plumbing.
+- Standard KV filtering.
+- C ABI and worker protocol.
+
+Each supported architecture still needs a reviewed range-aware adapter and real distributed validation. Unsupported architectures must fail closed.
+
+## 7. Selected architecture
+
+```text
+Existing tracker and accounting
+              |
+Existing versioned activation/relay protocol
+              |
+Project-owned standalone C++ GGUF shard worker
+(borrowing or jointly maintaining narrow Nakshatra patch concepts where practical)
+              |
+Pinned llama.cpp fork with small architecture-specific patch series
+              |
+GGUF loader, model graphs, quant kernels, KV, CPU/GPU backends
+```
+
+Nakshatra's working patch and daemon should be reproduced, rebased, and used as source/test evidence before equivalent code is independently designed. Collaboration on narrow upstreamable llama.cpp hooks is preferable to duplicate patch families, but the project should keep its own small pinned fork and standalone worker rather than fork Nakshatra's whole Petals-derived repository. The project-owned boundary remains necessary because Nakshatra's build, control plane, protobuf, sub-GGUF artifact model, single-session daemon, and accounting semantics do not satisfy the full Meshnet contract.
+
+### 7.1 Responsibility split
+
+Python/node agent remains responsible for:
+
+- Tracker registration and heartbeat.
+- Capability proof and admission.
+- Artifact distribution and verification.
+- Route selection and effective start layers.
+- Activation transport and relay.
+- Route sessions, epochs, cache-miss recovery, and process supervision.
+- Work telemetry and accounting.
+
+The C++ worker is responsible for:
+
+- Loading exactly one model/range/recipe.
+- Token embedding on a head shard.
+- Boundary activation input on middle/tail shards.
+- Executing only owned layers.
+- Returning boundary activations on non-tail shards.
+- Final norm/logits/sampling or token output on a tail shard.
+- Local KV/recurrent state by mapped sequence.
+- Exporting health, cache, load, compute, and memory metrics.
+
+### 7.2 Stable worker boundary
+
+Do not expose `ggml_tensor *`, scheduler objects, or llama.cpp structs to Python. Prefer a supervised executable with a small project-owned API, for example:
+
+```text
+load_layer_range
+prefill_tokens
+prefill_activation
+decode_token
+decode_activation
+get_boundary
+get_logits_or_token
+release_session
+export_session
+import_session
+health
+metrics
+```
+
+The wire carries at least:
+
+```text
+protocol version
+request/work ID
+route session ID
+route epoch
+model/artifact fingerprint
+layer begin/end
+effective start layer
+prefill/decode phase
+token start/count
+hidden shape and dtype
+payload length/checksum
+```
+
+### 7.3 Battle-proven transport and concurrency
+
+The implementation program chooses gRPC over HTTP/2 with Protocol Buffers for the standalone Python/C++ Shard data plane rather than inventing a raw socket protocol.
+
+- One long-lived bidirectional stream serves one Route Session Activation Seam.
+- HTTP/2 supplies connection reuse and flow control; gRPC supplies deadlines, cancellation, status, TLS hooks, and generated Python/C++ schemas.
+- Large prefill tensors are split into bounded frames; decode uses a small fast path.
+- Existing relay/WebSocket infrastructure may transport the same protobuf frames as opaque binary when direct gRPC reachability is unavailable.
+- OpenAI-facing HTTP/SSE and existing Tracker APIs remain unchanged.
+
+The public payload is a versioned named-tensor bundle, not one anonymous activation, because architecture boundaries such as vLLM's Llama/Qwen3-MoE pipeline stages can require both `hidden_states` and `residual`.
+
+Concurrency is implemented with local llama.cpp sequences or bounded contexts mapped from `(Route Session ID, route epoch)`. Compatible active decode steps are continuously batched inside each node. This adapts the useful vLLM scheduling concept without importing vLLM's Torch process groups, PagedAttention allocator, or static distributed executor.
+
+Tensor/expert parallel collectives remain confined to a future trusted composite-node or managed-cluster provider. The public volunteer primitive remains contiguous layer Shards.
+
+The benchmark-gated execution plan and Ralph backlog live in [the active distributed-GGUF feature](../../.scratch/distributed-gguf-runtime/README.md).
+
+## 8. Implementation spikes and acceptance gates
+
+### 8.1 Spike 1: reproduce and rebase Nakshatra
+
+Before writing a separate worker, reproduce Nakshatra's dense Llama-family two-worker path and rebase its narrow patch onto the exact current llama.cpp commit selected by this project.
+
+Run two local workers with disjoint sub-GGUF ranges, then compare sub-GGUF loading with a range-aware loader over one shared source artifact.
+
+Acceptance:
+
+- The audited Nakshatra test is independently reproducible rather than accepted from repository claims.
+- Each process maps only its assigned tensors.
+- Each process allocates KV only for assigned layers.
+- Head/middle/tail module ownership is correct.
+- Boundary residuals produce bounded numerical error against whole-model llama.cpp.
+- The patch rebases cleanly onto the pinned current llama.cpp baseline.
+- Reusable worker code is isolated from Nakshatra's Petals-derived and project-specific control plane.
+- Upstream collaboration or shared maintenance is evaluated before duplicating the patch family.
+
+### 8.2 Spike 2: Meshnet protocol and multi-session KV
+
+Place the rebased/Nakshatra-derived worker behind Meshnet's route/session protocol.
+
+Acceptance:
+
+- Multiple route sessions remain isolated through separate `llama_seq_id` values or contexts.
+- Duplicate requests are idempotent and stale route epochs are rejected.
+- Exact source/slice artifact hashes and runtime recipes are checked.
+- Cache miss, release, TTL/LRU eviction, cancellation, and re-prefill are bounded.
+- Killing a worker produces a bounded failure, not a deadlock.
+- Work receipts bind worker identity, request, route, range, artifact, and activation evidence.
+
+### 8.3 Spike 3: heterogeneous two-machine route
+
+Use a tracker-selected two-machine route with real CPU/GPU execution.
+
+Measure:
+
+- Cold and warm model load.
+- Mapped tensor and KV memory per process.
+- Prefill and decode throughput.
+- Activation bytes and boundary latency.
+- Whole-model parity.
+- Process/node failure behavior.
+- Per-node completed work and compute time.
+
+Synthetic tests remain unit coverage, not distributed validation.
+
+### 8.4 Spike 4: Qwen3 MoE adapter
+
+Qwen3 30B-A3B requires an explicit architecture adapter. It must review:
+
+- Expert/router tensor ownership.
+- Top-k routing and MoE graph behavior.
+- Architecture-specific normalization and Q/K normalization.
+- Layer-local expert loading.
+- KV type and layout.
+- Boundary residual placement.
+- Supported HIP/Vulkan/CPU paths.
+
+The adapter should reuse current llama.cpp's Qwen3 graph rather than reconstructing a simplified layer as `llama-gguf` does.
+
+### 8.5 Architecture certification
+
+Each architecture recipe should declare:
+
+- Range-aware graph implementation and version.
+- Head/tail module rules.
+- Cache kind and state support.
+- Boundary dtype/layout.
+- Supported backends.
+- Required runtime commit.
+- Real distributed validation evidence.
+
+Capability admission keeps unsupported combinations registered-but-dark.
+
+## 9. Principal risks
+
+1. **Upstream churn:** llama.cpp model/graph/cache internals change frequently. Keep a small ordered patch series and pinned revisions.
+2. **Architecture coupling:** every architecture adapter needs review and live proof.
+3. **Cache locality:** dynamic rerouting requires replay/checkpoint semantics, not socket retry.
+4. **Boundary compatibility:** shape, dtype, residual point, positions, and cache semantics must match exactly.
+5. **Artifact identity:** exact GGUF/model/tokenizer/runtime fingerprints must be part of capability proof.
+6. **Heterogeneous bottlenecks:** the slowest seam can dominate; placement must use measured end-to-end cost.
+7. **Accounting integrity:** completed-work receipts need route and worker identity and must not trust self-reported latency alone.
+8. **Security:** the worker accepts structured activations, not arbitrary GGML graphs or executable recipes.
+9. **Memory estimates:** weight quantization and KV/state type must both be represented.
+10. **Testing:** every supported lane needs real CPU/GPU tracker-routed acceptance evidence.
+
+## 10. Final conclusion
+
+Distributed GGUF should not be built from zero, but no existing project can be adopted as-is.
+
+The reusable composition is:
+
+- llama.cpp for mature GGUF parsing, architecture graphs, quantized kernels, backend scheduling, and KV/state APIs.
+- prima.cpp for proof and reference implementations of selective local GGUF loading, local layer KV, graph boundaries, and placement ideas.
+- `llama-gguf` for explicit range/protobuf/health/test-organization ideas.
+- Petals and the existing neuron-tai Transformers path for route-session, cache-locality, and recovery semantics.
+- The existing tracker, relay, capability, and accounting systems as the authoritative control plane.
+
+The selected path is a small pinned llama.cpp layer-worker fork with generic shard infrastructure and explicitly certified architecture adapters.