248 lines
11 KiB
Markdown
248 lines
11 KiB
Markdown
# Focused implementation strategy: performant concurrent distributed inference
|
|
|
|
Status: Accepted planning direction
|
|
Last updated: 2026-07-13
|
|
|
|
## Product objective
|
|
|
|
Enable clients to run top open models that do not fit on one consumer machine by combining independently owned model Shards into performant, concurrent Inference Routes.
|
|
|
|
The alpha-release target is the exact `zai-org/GLM-5.2` model, pinned by revision and served with `reasoning_effort=max`, using the smallest published Unsloth `UD-IQ1_S` GGUF across physical consumer machines. See [GLM-5.2-MAX-ALPHA-ROADMAP.md](GLM-5.2-MAX-ALPHA-ROADMAP.md). Dense Llama remains a cheap structural fixture; Qwen expansion is post-alpha.
|
|
|
|
The project is not trying to reproduce every vLLM feature or support every inference engine. It is optimizing for:
|
|
|
|
1. Models larger than one node's RAM/VRAM.
|
|
2. Useful interactive decode speed on consumer CPU, AMD, NVIDIA, Vulkan, and mixed routes where certified.
|
|
3. Multiple concurrent Route Sessions without cache corruption or global serialization.
|
|
4. A lean runtime with one control plane and one primary GGUF engine.
|
|
5. Measured improvement over the existing Transformers/safetensors implementation.
|
|
|
|
## Current reality
|
|
|
|
The existing project already owns the differentiating distributed control plane:
|
|
|
|
- Tracker-selected contiguous Shards.
|
|
- Stable Route Sessions.
|
|
- Local per-Shard Hot KV State in the Transformers reference backend.
|
|
- Binary Activation Seams.
|
|
- Relay/direct routing, cancellation, telemetry, billing, and capability admission.
|
|
- Persistent relay and direct transport optimizations.
|
|
|
|
The missing production path is a native GGUF execution worker that can load and execute only an assigned layer range while retaining local Hot KV State for concurrent Route Sessions.
|
|
|
|
Whole-model llama.cpp, vLLM, and existing Transformers serving remain baselines or optional route kinds. They are not substitutes for native distributed Shards.
|
|
|
|
## Performance hypothesis—not an assumption
|
|
|
|
GGUF itself is a format. Performance comes from llama.cpp/GGML's quantized kernels, memory layout, mmap, backend scheduling, and reduced working set.
|
|
|
|
Quantized GGUF may be faster or may merely fit a larger model. Comparisons against safetensors must report both speed and quality because BF16 safetensors and Q4/Q8 GGUF are not numerically equivalent.
|
|
|
|
Before expensive native work, establish controlled lanes. DGR-001 remains immutable; DGR-017 adds a target-specific fit and semantics contract without rewriting DGR-001 evidence:
|
|
|
|
- Same model architecture and upstream revision.
|
|
- Same machine, prompt set, context, output length, sampling policy, and concurrency.
|
|
- Transformers/safetensors BF16 or the current production recipe.
|
|
- llama.cpp GGUF F16/BF16 or Q8 correctness lane where available.
|
|
- Q4_K_M or selected production quantization performance/fit lane.
|
|
- TTFT, prefill tok/s, decode tok/s, p50/p95 latency, RSS, VRAM, artifact size, energy where available, and output-quality drift.
|
|
|
|
The program proceeds only if llama.cpp/GGUF provides at least one meaningful advantage recorded in a machine-readable performance contract:
|
|
|
|
- Better decode or aggregate throughput at acceptable quality; or
|
|
- Materially lower memory that makes the target model routable while preserving useful throughput.
|
|
|
|
## Parallelism we will use
|
|
|
|
### Public Inference Route: layer/pipeline parallelism
|
|
|
|
Each node independently executes one contiguous Shard. Activations cross seams; weights and Hot KV State remain local.
|
|
|
|
This is the only public cross-machine model-parallel primitive in the first runtime.
|
|
|
|
### Per-node continuous batching
|
|
|
|
Autoregressive tokens remain sequential within one generation. Throughput comes from batching decode steps from multiple active Route Sessions inside each node using llama.cpp batches and sequence IDs or bounded context pools.
|
|
|
|
This is essential. A worker that globally serializes sessions is not production-ready.
|
|
|
|
### Multiple complete routes: data parallelism
|
|
|
|
The Tracker may select multiple complete routes for independent requests. This increases network throughput and availability without requiring collectives between routes.
|
|
|
|
### Trusted composite node: optional tensor/expert parallelism
|
|
|
|
Tensor parallelism and expert parallelism require frequent collectives and tight compatibility. They may be used later inside one operator-controlled composite node or managed cluster exposed as one logical provider. They are not public WAN routing primitives.
|
|
|
|
### Deferred mechanisms
|
|
|
|
- Disaggregated prefill and KV transfer.
|
|
- Speculative decoding.
|
|
- Cross-route prefix snapshots.
|
|
- Route repair with KV migration.
|
|
- Public tensor/expert parallel collectives.
|
|
|
|
They remain out of the critical path until the native layer route passes performance and concurrency gates.
|
|
|
|
## Reuse decisions
|
|
|
|
### llama.cpp/GGML: primary runtime substrate
|
|
|
|
Reuse:
|
|
|
|
- GGUF parsing and mmap.
|
|
- Quantized kernels.
|
|
- CPU, CUDA, HIP/ROCm, Vulkan, Metal, and other supported backends.
|
|
- Tokenizer and model architecture implementations.
|
|
- KV and sequence operations.
|
|
- Backend scheduler and graph execution.
|
|
|
|
Maintain a small exact-commit fork only for the missing local seam:
|
|
|
|
- Range-aware tensor ownership/loading.
|
|
- Architecture-defined boundary input/output.
|
|
- Intermediate boundary output without tail normalization.
|
|
- Layer-filtered KV and sequence mapping.
|
|
|
|
Keep networking, Tracker logic, billing, and public protocol outside llama.cpp. Upstream generic hooks where possible.
|
|
|
|
### vLLM: concepts and optional managed backend
|
|
|
|
Use unmodified vLLM only as:
|
|
|
|
- A whole-model node backend.
|
|
- A managed TP/PP/EP cluster represented as one logical provider.
|
|
- A performance/correctness baseline.
|
|
|
|
Adapt concepts, not runtime code:
|
|
|
|
- Named intermediate tensor bundles.
|
|
- Continuous batching and request-owner maps.
|
|
- Versioned KV-transfer compatibility fingerprints.
|
|
- Explicit send/receive/abort/failure lifecycle.
|
|
- Load telemetry and unbiased route selection.
|
|
|
|
Do not fork vLLM for public Shards and do not transplant PagedAttention, Torch process groups, or GGUF-plugin kernels into the llama.cpp worker.
|
|
|
|
### Nakshatra, prima.cpp, llama-gguf, LiGGUF, GPUStack
|
|
|
|
Use as source and test donors only:
|
|
|
|
- Nakshatra: partial-GGUF patches, daemon concepts, replay cases.
|
|
- prima.cpp: selected tensor ownership and local-layer KV evidence.
|
|
- llama-gguf: small protocol and integration-test patterns.
|
|
- LiGGUF: Q8 activation transport and tensor-reduction reference.
|
|
- historical GPUStack: resource preflight and role-oriented placement.
|
|
|
|
Do not adopt or fork their repositories wholesale.
|
|
|
|
### Mesh-LLM GLM branch: focused test/patch donor only
|
|
|
|
Use its GLM-5.2 branch to study DSA, IndexShare, stage-local KV, and sideband tests. Do not import its scheduler, discovery/control plane, package manager, or broad llama.cpp patch stack. Every adopted idea must be independently understood, minimized, attributed, and tested against our exact pin.
|
|
|
|
## Battle-proven transport decision
|
|
|
|
Use gRPC over HTTP/2 with Protocol Buffers for the native C++ Shard worker protocol.
|
|
|
|
Why:
|
|
|
|
- Mature Python and C++ implementations.
|
|
- Bidirectional streaming.
|
|
- HTTP/2 flow control and connection reuse.
|
|
- Deadlines, cancellation, status codes, TLS, authentication interceptors, and generated schemas.
|
|
- Avoids inventing a socket protocol.
|
|
|
|
Scope boundary:
|
|
|
|
- OpenAI-compatible client/Gateway APIs remain HTTP/SSE.
|
|
- Tracker/control APIs remain existing project interfaces.
|
|
- One long-lived bidirectional gRPC stream serves one Route Session Activation Seam.
|
|
- Existing relay/WebSocket infrastructure may carry the same versioned protobuf frames as opaque binary when direct gRPC reachability is unavailable.
|
|
- Large prefill tensors are chunked into bounded frames; decode bundles stay small.
|
|
- No QUIC/WebRTC/custom transport in this milestone.
|
|
|
|
The public boundary uses a versioned named-tensor bundle rather than one anonymous tensor because architecture boundaries can require more than `hidden_states`.
|
|
|
|
Minimum identity:
|
|
|
|
```text
|
|
schema version
|
|
request/work id
|
|
Route Session id and route epoch
|
|
Model Artifact and runtime recipe fingerprint
|
|
Shard range and effective start
|
|
phase: prefill/decode/release/cancel
|
|
position/token range
|
|
named tensors with shape/dtype/byte order
|
|
compression and checksum
|
|
idempotency step id
|
|
cache expectation/result
|
|
```
|
|
|
|
## Concurrency model
|
|
|
|
A native worker must not use one global serving sequence or one lock around all model execution.
|
|
|
|
Required ownership:
|
|
|
|
```text
|
|
(Route Session id, route epoch)
|
|
-> local sequence/context
|
|
-> Shard-local Hot KV State
|
|
-> bounded lease and memory accounting
|
|
```
|
|
|
|
The node scheduler:
|
|
|
|
- Admits sessions against model memory and KV budget.
|
|
- Forms compatible decode batches from active sessions.
|
|
- Preserves per-session position and route order.
|
|
- Applies bounded queues and backpressure.
|
|
- Cancels/releases independently.
|
|
- Reports queue, batch, KV, prefill, decode, and seam telemetry.
|
|
|
|
Initial deterministic gate: at least four concurrent sessions on a small certified model with no token/KV cross-talk. Final concurrency targets are hardware/recipe-specific and recorded by capability admission rather than hardcoded globally.
|
|
|
|
## Stage gates
|
|
|
|
### Gate A: performance hypothesis
|
|
|
|
Controlled safetensors-versus-GGUF benchmark produces a signed/reproducible report and locks thresholds. Stop native work if there is no meaningful speed or fit benefit.
|
|
|
|
### Gate B: local range parity
|
|
|
|
Two local processes own disjoint GGUF ranges and match whole-model llama.cpp within the certified numerical tolerance for prefill and greedy decode.
|
|
|
|
### Gate C: concurrent KV
|
|
|
|
Multiple Route Sessions prefill/decode concurrently with isolated local KV, bounded memory, cancellation, and release.
|
|
|
|
### Gate D: real distributed route
|
|
|
|
Two physical machines execute one model that uses both Shards. Synthetic activation tests do not satisfy this gate.
|
|
|
|
### Gate E: consumer-hardware performance
|
|
|
|
On certified consumer hardware, the GGUF route beats the current distributed safetensors route under the locked performance contract or enables a larger otherwise-unroutable model at useful measured speed.
|
|
|
|
### Gate F: exact GLM-5.2 alpha target
|
|
|
|
After the generic dense fixture proves range and boundary mechanics, certify explicit GLM-5.2 MoE, MLA KV, DSA, IndexShare, and NextN policy. Alpha requires the exact `UD-IQ1_S` target across physical consumer nodes, native Max-mode semantics, locked parity/usefulness/performance thresholds, and bounded failure cleanup. Qwen3/Qwen3-MoE is later architecture expansion.
|
|
|
|
## Scope discipline
|
|
|
|
The following do not block the first production candidate:
|
|
|
|
- New cryptocurrency/economics work.
|
|
- New artifact P2P protocol.
|
|
- QUIC or WebRTC.
|
|
- vLLM fork.
|
|
- Whole-repository Nakshatra/prima adoption.
|
|
- Every GGUF architecture.
|
|
- Automatic route repair.
|
|
- Prefix snapshot migration.
|
|
- Speculative decoding.
|
|
- A large-model marketing demo before small-model parity and concurrency pass.
|
|
|
|
Every optimization must preserve output contract, session isolation, cancellation, resource cleanup, capability admission, and per-node attribution.
|