popov/neuron-tai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: define distributed GGUF runtime plan

Browse Source
This commit is contained in:
Dobromir Popov
2026-07-13 15:09:27 +03:00
parent b5fa7245df
commit 4cae4a6c5c
42 changed files with 4913 additions and 691 deletions
Download Patch File Download Diff File Expand all files Collapse all files

389
.scratch/distributed-gguf-runtime/architecture.md
View File

@@ -1,274 +1,259 @@
# Distributed GGUF Runtime Architecture
# Performant Concurrent Distributed GGUF Architecture
## Product Stance
Status: current target architecture
Last updated: 2026-07-13
The platform optimizes for access to high-quality models, not lowest latency. Latency is acceptable if the user can run models that are otherwise unavailable to them. The hard context limit for the first serious distributed runtime should be **128K tokens**. Longer context usually means the product is compensating for missing task decomposition, retrieval, or workspace summarization.
## Product invariant
## Current State
The system exists to serve high-quality models that exceed one consumer node's memory while retaining useful interactive speed and aggregate concurrency. A feature that only produces a distributed demo but is slower, globally serialized, or impossible to operate on consumer hardware is not complete.
The current node has two materially different inference paths:
## Existing control plane
- **Full local PyTorch model**: calls Hugging Face `model.generate()`, so Transformers owns autoregressive decode and local KV cache.
- **Distributed PyTorch route**: bypasses `model.generate()`, calls individual layers with `use_cache=False`, and recomputes the full growing sequence for every generated token.
Meshnet remains the only public control plane:
Current distributed data flow:
- Tracker registration, Coverage Map, route scoring and assignment.
- Contiguous Shards and overlap-safe effective starts.
- Stable Route Sessions and route epochs.
- Local per-Shard Hot KV State in the reference backend.
- Direct/relay transport, cancellation and backpressure.
- Generation Telemetry, billing, validation and per-node attribution.
- Model-agnostic capability admission.
No external engine replaces these responsibilities.
## Runtime topology
```text
client request
-> head node formats prompt
-> for each output token:
head tokenizes full current text
head runs early layers over all tokens
head sends full activation [batch, sequence, hidden] to next node
middle nodes run their layers over all tokens
tail returns one decoded token string
head appends token to text
OpenAI-compatible client
|
Gateway / Tracker Node
|
ordered Inference Route
|
+-- head Shard: tokenizer/embedding + early layers
| local weights and Hot KV State
|
+-- middle Shard(s): architecture boundary + owned layers
| local weights and Hot KV State
|
+-- tail Shard: final layers + norm/head/sampling
local weights and Hot KV State
```
This is correct for small demos but not viable for large models. For GLM-5.2, a single 128K seam activation is roughly:
Weights never move in the per-request hot path. Every node opens and verifies its local Model Artifact before becoming routable.
## Primary execution substrate
```text
128K tokens * hidden_size 6144 * 2 bytes ~= 1.5 GiB per hop
project-owned C++ Shard worker
|
small exact-commit llama.cpp patch stack
|
GGUF mmap, quantized kernels, architecture graphs,
KV/sequence operations, CPU/CUDA/HIP/Vulkan/Metal backends
```
Sending that every output token is the bottleneck.
The patch stack adds only the missing local execution seam:
## Target State
1. Range-aware tensor registration/loading.
2. Endpoint-specific embedding and final head ownership.
3. Architecture-defined intermediate input.
4. Architecture-defined pre-tail boundary output.
5. Layer-filtered KV and external session mapping.
Target distributed data flow:
The worker owns protocol translation and process lifecycle. llama.cpp never receives Tracker, relay, billing or volunteer-network code.
## Shard data plane
Use Protocol Buffers and gRPC over HTTP/2.
### Service shape
- Unary capability and health.
- Bidirectional Route Session stream.
- Explicit release and cancellation.
- Metrics suitable for capability admission and route scoring.
### Session stream
One long-lived stream represents one Route Session Activation Seam. It amortizes connection setup and inherits HTTP/2 flow control. Every message carries enough identity to reject stale or incompatible work.
```text
client request
-> tracker selects route and pins session
-> head node creates session_id
-> prefill:
prompt is chunked
each shard computes its layer range
each shard appends local KV/state for its own layers
activations cross only layer seams
-> decode loop:
head sends one new token / one-step hidden state
each shard reads local KV/state for session_id
each shard appends one step to local KV/state
only one-step activation crosses seams
tail returns logits/token
schema version
request/work id
Route Session id
route epoch
Model Artifact hash
runtime recipe fingerprint
Shard begin/end and effective start
prefill/decode/release/cancel phase
position and token range
idempotency step id
cache expectation/result
named tensor bundle
compression/checksum
```
The KV cache remains local to the node that computed it. It is not sent to the next node and not read from a remote cache server during every decode step.
Prefill tensors are split into bounded ordered frames. Decode messages carry one-step architecture boundary bundles and remain small.
## Client Feedback
Direct nodes use gRPC. Nodes requiring the existing relay carry the same protobuf frames as opaque binary through the relay session. This preserves one semantic protocol instead of maintaining separate direct and relay payload contracts.
Streaming responses are desirable when the backend and client transport support them. The product should stream token deltas when possible, and it must always provide realtime Generation Telemetry while the route is working.
## Architecture boundary
The fallback behavior is a non-streaming final answer plus live telemetry. That fallback is acceptable for early route proofs or models/backends that cannot expose clean token deltas yet, but the preferred client experience is streamed output plus telemetry.
Minimum client-visible telemetry:
- route/session accepted
- selected model and quantization
- prefill phase started/completed
- decode phase started
- generated token count
- rolling tokens per second
- route health or retry/failure reason
- estimated billing units when available
Implementation options:
- Server-Sent Events or WebSocket for realtime progress
- polling endpoint for simple clients
- OpenAI-compatible streaming for clients that require token deltas
This means "no token streaming" is acceptable only as a fallback. "Silent wait for minutes" is not acceptable.
## Artifact Plane
Artifact distribution is separate from execution.
The public boundary is a versioned named-tensor bundle:
```text
model publisher
-> produces model manifest
-> creates GGUF / safetensors / tokenizer artifacts
-> content-addresses every file/chunk
-> publishes torrent/magnet + HTTP fallback metadata
node
-> chooses model/layer range
-> downloads needed files/chunks
-> verifies hash
-> advertises availability to tracker
bundle schema/version
architecture adapter and boundary point
named tensors
per-tensor shape, dtype and byte order
payload fragments
compression/checksum
```
Required manifest fields:
Dense Llama may use one residual tensor. Other adapters may require more. vLLM's Llama and Qwen3-MoE PP paths demonstrate a boundary with both `hidden_states` and `residual`; therefore the generic protocol must not assume one anonymous tensor.
- model id and version
- upstream source repo and revision
- license
- architecture name
- tokenizer files and hashes
- quantization
- tensor-to-layer map
- file/chunk hashes
- optional GGUF split files
- supported runtime backends
- context cap
- KV/cache format descriptor
Only the head owns token embedding. Only the tail owns final normalization, LM head and sampling. Middle Shards exchange the architecture-defined pre-tail boundary, not final normalized embeddings.
## Execution Plane
The tracker selects routes using layer coverage and observed performance:
## Hot KV State and concurrency
```text
route = [
head node: embeddings + layers 0..k
middle nodes: contiguous layer ranges
tail node: final layers + norm + lm_head
]
(Route Session id, route epoch)
-> local llama sequence or bounded context
-> KV for owned layers only
-> lease, memory accounting and lifecycle
```
Route selection inputs:
Required operations:
- model id/version/quantization
- layer coverage
- node hardware
- measured prefill throughput
- measured decode throughput
- queue depth
- latency to neighboring nodes
- cache warmth for the requested prefix/session
- reliability/reputation
- Prefill append.
- Decode append.
- Truncate after rejected speculative positions if later enabled.
- Explicit release.
- TTL/LRU eviction.
- Cache-miss response.
- Stale-epoch rejection.
The route is sticky for the request/session. A new route means either a fresh prefill or restoring compatible KV snapshots.
A node must not clear global KV on a new stream or serialize all requests behind one logical serving sequence.
## KV Cache Ownership
## Continuous batching
KV/state ownership is by layer range:
Autoregressive dependencies remain sequential inside one Route Session. Aggregate throughput comes from batching compatible decode steps across active sessions:
```text
session_id = request scoped id
node A owns layers 0..15 KV for session_id
node B owns layers 16..31 KV for session_id
node C owns layers 32..77 KV for session_id
time 0: session A token 1 + session B token 8 + session C token 3
-> one llama batch for this Shard
time 1: next ready positions from active sessions
-> next llama batch
```
The tracker does not own hot KV. It may know which nodes hold active KV for session accounting and failure handling.
The node scheduler:
Cache servers may store:
- Admits work against weight, KV, scratch and queue budgets.
- Keeps per-session token positions and outputs separate.
- Prevents long prefill from starving decode.
- Applies bounded backpressure.
- Reports active sessions, queue depth, batch occupancy, KV pressure and throughput.
- prompt-prefix snapshots
- session checkpoints for retry
- cold reusable context blocks
- audit samples
The initial deterministic gate is four concurrent sessions on a small model without cross-talk. Hardware-specific limits are measured and advertised through capability admission.
Cache servers must not be in the per-token hot loop unless colocated with the compute node.
## Parallelism boundaries
## 128K KV Budget
| Mechanism | First-runtime use |
|---|---|
| Layer/pipeline parallelism | Public Inference Route across contiguous Shards |
| Continuous batching | Inside every node across active Route Sessions |
| Data parallelism | Multiple complete routes for independent requests |
| Tensor parallelism | Deferred to a trusted composite node/managed cluster |
| Expert parallelism | Deferred to a trusted composite node/managed cluster |
| Disaggregated prefill | Deferred until core route performance passes |
| Speculative decoding | Deferred optimization |
GLM-5.2 compressed DSA/MLA-style estimate from config:
Public WAN tensor/expert collectives are rejected for the first runtime because their per-layer communication and static rank assumptions conflict with heterogeneous volunteer nodes.
```text
layers = 78
kv_lora_rank = 512
qk_rope_head_dim = 64
dtype = bf16 = 2 bytes
context = 128K
## Optional providers
per_token ~= 78 * (512 + 64) * 2 = 89,856 bytes ~= 87.75 KiB
128K total ~= 10.7 GiB
per layer ~= 137 MiB
```
### Transformers/safetensors
This is feasible when sharded:
Remains:
| Layer count | Approx active KV at 128K |
|---:|---:|
| 1 | 137 MiB |
| 10 | 1.37 GiB |
| 20 | 2.75 GiB |
| 78 | 10.7 GiB |
- Correctness/reference backend.
- Fallback for unsupported architectures.
- Baseline for performance and output quality.
The exact runtime value depends on implementation and cache quantization, but the order of magnitude is acceptable.
### vLLM
## Protocol Sketch
May run unmodified as a complete model or managed TP/PP/EP cluster represented as one logical provider. Its internal ranks are not independently routed or rewarded.
### Prefill
Borrow only concepts such as named bundles, continuous batching, typed compatibility fingerprints, explicit transfer lifecycle and load telemetry.
```http
POST /v1/sessions/{session_id}/prefill
Content-Type: application/octet-stream
X-Meshnet-Model: zai-org/GLM-5.2
X-Meshnet-Route-Id: ...
X-Meshnet-Token-Range: 0-2047
X-Meshnet-Shape: 1,2048,6144
X-Meshnet-Dtype: bfloat16
### Whole-model llama.cpp
<activation bytes>
```
Provides a local proxy backend, correctness oracle and performance baseline. It is not the native distributed milestone.
The receiver:
## Artifact and recipe compatibility
- validates route/session
- runs assigned layer range for that chunk
- appends local KV/state
- forwards resulting activation to next hop
A routable recipe identifies separately:
### Decode
- Source Model Artifact hash and optional derivative/slice hash.
- Architecture and adapter version.
- Tokenizer revision and vocabulary.
- Weight quantization.
- Activation interchange dtype/schema.
- Backend compute dtype and backend implementation.
- KV dtype/layout.
- RoPE/context parameters.
- llama.cpp commit and project patch version.
- Shard range and endpoint ownership.
```http
POST /v1/sessions/{session_id}/decode-step
Content-Type: application/octet-stream
X-Meshnet-Model: zai-org/GLM-5.2
X-Meshnet-Position: 131072
X-Meshnet-Shape: 1,1,6144
X-Meshnet-Dtype: bfloat16
Compatibility fails closed. Similar quantization labels or model names are not enough.
<one-step activation bytes>
```
## Admission and failure
The receiver:
A recipe becomes routable only after a real local and distributed forward passes. Synthetic tests remain unit coverage.
- loads local KV/state by `session_id`
- runs one decode step for assigned layers
- appends one token position to local KV/state
- forwards one-step activation
Alpha failure behavior:
## GGUF / llama.cpp Integration
- Deadline or node loss cancels the Route Session.
- Every node releases KV and queued buffers.
- Uncertain mutations are not replayed silently.
- Retry starts from token zero on a newly compatible route.
- No cross-node KV import is trusted until a later signed/compatible snapshot protocol exists.
The target llama.cpp integration needs more than `llama-server`.
## Performance release contract
Required capabilities:
Before native development proceeds, compare the current Transformers/safetensors backend with whole-model llama.cpp under controlled model/hardware/quality lanes.
- load full GGUF locally for immediate single-node performance
- optionally load only selected tensors/layers
- execute a layer range against inbound hidden states
- expose outbound hidden states at a boundary
- own per-session KV/state for only the loaded layer range
- support prefill chunks and decode-step calls
- expose model-specific cache metadata for DSA/MLA without requiring the tracker to understand tensor internals
Final release compares distributed GGUF with distributed safetensors using thresholds locked before seeing final results.
If llama.cpp cannot expose these as stable APIs today, the collaboration target is an upstream extension rather than a long-lived fork.
Required measurements:
## Failure Model
- TTFT.
- Prefill and decode tokens/sec.
- Aggregate concurrency throughput.
- p50/p95 latency.
- Seam bytes and latency.
- Queue/batch occupancy.
- RSS, VRAM and KV pressure.
- Output-quality drift.
- Cancellation/failure cleanup.
Alpha behavior:
The GGUF path ships only if it is faster at acceptable quality or enables a larger otherwise-unroutable model at useful measured speed.
- Route node drops during prefill: fail request and retry from scratch.
- Route node drops during decode: fail request unless a recent KV snapshot exists.
- Tracker restart: active sessions may be lost; completed billing records persist.
- Node restart: local hot KV is lost.
## Implementation sequence
Later behavior:
1. Lock benchmark/performance contract.
2. Define gRPC/protobuf and exact recipe identity.
3. Pin llama.cpp and create the minimal patch stack.
4. Implement dense-Llama range loading and boundary parity.
5. Implement concurrent local KV.
6. Build and integrate the standalone worker.
7. Pass local two-process real-model acceptance.
8. Pass real heterogeneous two-machine acceptance.
9. Add continuous batching and failure hardening.
10. Enforce the GGUF-versus-safetensors release gate.
11. Add Qwen3/Qwen3-MoE as a separately certified adapter.
12. Prepare narrow upstream collaboration patches/tests.
- periodic KV snapshots for long sessions
- prefix cache reuse across requests
- route repair when a semantically equivalent node has the same model/layer range and compatible cache snapshot
## Security And Trust
Activation/KV data can reveal user prompts. Public volunteer routes are not private. For sensitive workloads:
- use private swarms
- allow paid trusted nodes
- encrypt transport
- avoid storing hot KV on untrusted shared cache servers
- sample outputs for fraud/audit as already planned in alpha hardening
See [the Ralph backlog](prd.json) and [implementation strategy](implementation-strategy.md).