docs: define implementation-ready distributed GGUF roadmap
This commit is contained in:
Dobromir Popov
@@ -1,270 +1,40 @@
|
||||
# Distributed GGUF Decision Framework
|
||||
|
||||
> **Superseded for active implementation decisions.** The grill was resolved on 2026-07-13. Use [implementation-strategy.md](implementation-strategy.md), [architecture.md](architecture.md), [ADR-0024](../../docs/adr/0024-distributed-gguf-runtime.md), and [prd.json](prd.json). This file remains as historical decision rationale.
|
||||
|
||||
This framework is for grilling open decisions. It keeps decisions tied to project vocabulary and implementation gates instead of vague "distributed inference" language.
|
||||
|
||||
## Core Vocabulary
|
||||
|
||||
Use the existing domain terms this way:
|
||||
|
||||
- **Shard**: contiguous transformer layer range. This is the compute, routing, cache, and reward unit.
|
||||
- **Shard Swarm**: storage/download group for artifacts needed by a shard.
|
||||
- **Inference Route**: ordered node sequence that covers all layers for one request.
|
||||
- **Route Session**: one active request bound to one inference route and stable session id.
|
||||
- **Hot KV State**: live per-shard cache held by the route node during a route session.
|
||||
- **Prefix Snapshot**: persisted route-session state used for reuse or failover, not the hot decode path.
|
||||
- **Artifact Manifest**: canonical mapping from model artifacts to semantic model parts and runtime support.
|
||||
- **Generation Telemetry**: realtime progress for a route session, including phase and tokens/sec, independent of whether token deltas are streamed.
|
||||
|
||||
## The Five Planes
|
||||
|
||||
### 1. Control Plane
|
||||
|
||||
Owner: Tracker.
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- node registry
|
||||
- coverage map
|
||||
- route selection
|
||||
- rebalance directives
|
||||
- route-session creation
|
||||
- health and telemetry
|
||||
- client-visible Generation Telemetry
|
||||
- billing/audit records
|
||||
|
||||
Must not do:
|
||||
|
||||
- serve hot KV during every token
|
||||
- become the only place model artifacts can be fetched
|
||||
|
||||
### 2. Artifact Plane
|
||||
|
||||
Owner: Shard Swarms, local node storage, optional CDN/bootstrap mirrors.
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- GGUF/safetensors/tokenizer download
|
||||
- content-addressed verification
|
||||
- local artifact inventory
|
||||
- artifact-to-layer mapping
|
||||
- cache eviction
|
||||
|
||||
Must not do:
|
||||
|
||||
- define execution order by file split alone
|
||||
- imply that a downloaded file chunk equals a Shard
|
||||
|
||||
### 3. Execution Plane
|
||||
|
||||
Owner: active Inference Route.
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- chunked prefill
|
||||
- one-step decode
|
||||
- hidden-state transfer across activation seams
|
||||
- start-layer handling for overlapping shards
|
||||
- backpressure
|
||||
|
||||
Must not do:
|
||||
|
||||
- resend full context activations during decode
|
||||
- require cross-node tensor parallel all-reduce for public v1
|
||||
|
||||
### 4. Session State Plane
|
||||
|
||||
Owner: route nodes for hot KV; cache servers only for snapshots.
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- per-shard local KV ownership
|
||||
- cache allocation and eviction
|
||||
- cache ABI compatibility
|
||||
- session close/release
|
||||
- optional prefix snapshots
|
||||
|
||||
Must not do:
|
||||
|
||||
- centralize hot KV in a remote service
|
||||
- let a replacement node continue from incompatible state
|
||||
|
||||
### 5. Economics And Trust Plane
|
||||
|
||||
Owner: tracker plus settlement/validation components.
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- distinguish storage/seeding work from inference work
|
||||
- account for prefill and decode separately
|
||||
- record route participation
|
||||
- sample validation events
|
||||
- slash proven fraud
|
||||
|
||||
Must not do:
|
||||
|
||||
- pay a node for merely holding files as if it generated tokens
|
||||
- hide public-swarm privacy limits from clients
|
||||
|
||||
## Hard Invariants
|
||||
|
||||
These are the framework rules unless we deliberately write a new ADR:
|
||||
|
||||
1. Public-network Shards are contiguous layer ranges.
|
||||
2. Hot KV State is local to the node serving that Shard in that Route Session.
|
||||
3. Artifact distribution and route execution are separate systems.
|
||||
4. Decode seam payload must be `O(hidden_size)`.
|
||||
5. Prefill may be `O(sequence_length * hidden_size)`, but only in bounded chunks.
|
||||
6. The tracker chooses routes; nodes do not negotiate route topology peer-to-peer.
|
||||
7. Model/backend-specific cache internals stay behind backend capability reports.
|
||||
8. PyTorch remains the correctness/reference backend while llama.cpp/GGUF becomes the performance backend.
|
||||
9. Streaming responses are preferred when feasible; Generation Telemetry is always required.
|
||||
|
||||
## Resolved Gates
|
||||
|
||||
### Gate 1: Public Shard Semantics
|
||||
|
||||
Decision: public-network Shards are contiguous transformer layer ranges. Tensor-parallel or ring-style execution is allowed only inside one trusted node, one colocated pod, or a future composite node abstraction.
|
||||
|
||||
Rationale:
|
||||
|
||||
- Layer ranges match the existing `Shard`, `Coverage Map`, `Inference Route`, billing, and fraud vocabulary.
|
||||
- Public volunteer nodes should not require cross-node all-reduce or tight per-layer synchronization in v1.
|
||||
- Existing projects such as prima.cpp and Distributed Llama can still inform local-cluster/backend execution without becoming the public routing primitive.
|
||||
|
||||
Consequences:
|
||||
|
||||
- Artifact Manifests must map files/tensors to semantic layer ranges.
|
||||
- Route selection remains ordered layer coverage.
|
||||
- Rewards can be attributed to layer-range work.
|
||||
- Hot KV State is naturally owned by the node serving that layer range for the Route Session.
|
||||
|
||||
### Gate 2: Hot KV Strategy
|
||||
|
||||
Decision: v1 rejects centralized hot KV. Hot KV State is local to the node serving the relevant Shard in the active Route Session. Cache servers may store Prefix Snapshots for reuse, retry, or failover, but they are not in the per-token decode path.
|
||||
|
||||
Rationale:
|
||||
|
||||
- Decode is the tight loop; adding remote cache I/O there makes latency and bandwidth worse at the worst point.
|
||||
- Local KV naturally follows layer-range Shard ownership.
|
||||
- Centralized hot KV increases privacy exposure and creates consistency problems.
|
||||
- Prefix Snapshots preserve the useful part of central storage without making it mandatory for every generated token.
|
||||
|
||||
Consequences:
|
||||
|
||||
- Route Session must be sticky.
|
||||
- Failover is limited in alpha unless a compatible Prefix Snapshot exists.
|
||||
- Cache servers are optimization infrastructure, not required runtime infrastructure.
|
||||
- Route repair requires compatible model revision, layer range, backend cache ABI, and snapshot position.
|
||||
|
||||
### Gate 3: First Runtime Proof
|
||||
|
||||
Decision: prove distributed Route Session and Hot KV State semantics in the existing PyTorch route before modifying llama.cpp/GGUF.
|
||||
|
||||
Rationale:
|
||||
|
||||
- PyTorch exposes model internals and cache objects more directly, so it is the fastest way to validate the distributed protocol.
|
||||
- The current distributed PyTorch route already has the right high-level shape but disables cache and recomputes full prompts.
|
||||
- Fixing that path gives us a reference implementation for correctness tests, telemetry, session lifecycle, and wire protocol behavior.
|
||||
- llama.cpp/GGUF should receive a clear target ABI rather than becoming both the protocol experiment and the performance backend at once.
|
||||
|
||||
Consequences:
|
||||
|
||||
- Issue 02 precedes issue 05.
|
||||
- llama.cpp collaboration has a concrete target ABI.
|
||||
- The PyTorch route remains the architecture-coverage/reference backend even after GGUF becomes the preferred performance path.
|
||||
- The first success metric is eliminating full-prompt recompute in distributed decode.
|
||||
|
||||
### Gate 3A: Client Feedback During Latency
|
||||
|
||||
Decision: streaming responses are preferred when feasible, and realtime Generation Telemetry is required regardless of streaming support.
|
||||
|
||||
Rationale:
|
||||
|
||||
- The product optimizes for access to large capable models, so some latency is acceptable.
|
||||
- Users still need confidence that the route is alive and roughly how fast it is generating.
|
||||
- Streaming token deltas give the best user experience when the backend exposes them cleanly.
|
||||
- Tokens/sec remains useful during prefill, queueing, and any backend that cannot stream token deltas.
|
||||
|
||||
Consequences:
|
||||
|
||||
- The gateway should stream token deltas through an OpenAI-compatible response when possible.
|
||||
- The gateway must expose progress through SSE, WebSocket, or polling.
|
||||
- The final answer can be delivered after completion only as a fallback.
|
||||
- Telemetry must include route phase, generated token count, and rolling tokens/sec.
|
||||
- Non-streaming clients still need realtime telemetry.
|
||||
|
||||
### Gate 4: llama.cpp Collaboration Shape
|
||||
|
||||
Decision: target upstreamable `libllama`/ggml hooks instead of planning around a permanent fork.
|
||||
|
||||
Rationale:
|
||||
|
||||
- llama.cpp changes quickly across model support, quantization, kernels, and hardware backends.
|
||||
- A permanent fork would become expensive to maintain and would lag upstream improvements.
|
||||
- A short-lived prototype branch is acceptable if it proves the API and makes upstream collaboration concrete.
|
||||
- Keeping tracker/routing logic outside llama.cpp makes the upstream ask smaller and cleaner.
|
||||
|
||||
Consequences:
|
||||
|
||||
- Need a minimal reproducible localhost demo before asking upstream to carry the design.
|
||||
- Need to separate "what llama.cpp should expose" from "what our tracker does".
|
||||
- Desired upstream surface is layer-range execution, hidden-state boundary I/O, partial loading/introspection, and per-session KV ownership.
|
||||
- If upstream rejects the shape, we revisit whether to carry a narrow adapter fork or keep GGUF distributed execution as experimental.
|
||||
|
||||
### Gate 5: First Model Target
|
||||
|
||||
Decision: use a two-tier model target. Use a small, boring, llama.cpp-supported GGUF model for the first protocol smoke test. Use `deepseek-ai/DeepSeek-V4-Flash` as the first serious large-model target. Keep GLM-5.2 and Ornith as later support audits.
|
||||
|
||||
Rationale:
|
||||
|
||||
- The first protocol proof should isolate route/session/KV bugs from model-architecture bugs.
|
||||
- DeepSeek-V4-Flash is a strong first serious target because it is much smaller than 1.6T-class models while still being large enough to validate the product thesis.
|
||||
- DeepSeek-V4-Flash still has architecture-specific risks, so it should not be the first smoke test.
|
||||
- GLM-5.2 and Ornith remain valuable targets, but they add DSA/MLA/hybrid attention uncertainty.
|
||||
|
||||
Consequences:
|
||||
|
||||
- 128K cache accounting can be modeled now.
|
||||
- The first "real" target-model audit is DeepSeek-V4-Flash support in PyTorch, vLLM/SGLang, and any available GGUF/llama.cpp quantization path.
|
||||
- Production support waits for backend capability reports and exact cache ABI support.
|
||||
|
||||
### Gate 6: Failure Semantics
|
||||
|
||||
Decision: alpha fails Route Sessions on route-node loss instead of attempting automatic route repair.
|
||||
|
||||
Rationale:
|
||||
|
||||
- Route repair requires compatible Prefix Snapshots, cache ABI checks, replacement-node selection, billing correction, and client stream/error recovery.
|
||||
- Local Hot KV State means a replacement node cannot continue unless it has compatible state at the same position.
|
||||
- Fail-fast keeps the first implementation correct while the session/KV protocol is still being proven.
|
||||
|
||||
Consequences:
|
||||
|
||||
- Better observability and explicit errors are required.
|
||||
- Snapshotting becomes a later feature, not a blocker for first inference.
|
||||
- Generation Telemetry must report the last known phase and failure reason.
|
||||
- Client or gateway retry starts a new Route Session from scratch.
|
||||
|
||||
### Gate 7: Transport
|
||||
|
||||
Decision: keep binary HTTP for v1 activation transfer instead of jumping immediately to QUIC, WebRTC, or a custom transport.
|
||||
|
||||
Rationale:
|
||||
|
||||
- ADR-0008 already defines binary activation bodies with HTTP headers.
|
||||
- HTTP keeps the first implementation debuggable with the existing server stack and tooling.
|
||||
- The core risk is route/session/KV correctness, not transport optimization.
|
||||
- QUIC/WebRTC can be introduced later behind the same activation protocol once semantics are proven.
|
||||
|
||||
Consequences:
|
||||
|
||||
- Focus benchmark work on payload shape, chunking, and cache behavior first.
|
||||
- QUIC/WebRTC can be introduced as an optimization behind the same activation protocol.
|
||||
- v1 implementation can reuse the current HTTP routing, relay, and observability infrastructure.
|
||||
- Transport abstraction should be kept narrow enough that HTTP can be replaced later without changing backend cache semantics.
|
||||
|
||||
## Grilling Progress
|
||||
|
||||
Gates 1, 2, 3, 3A, 4, 5, 6, and 7 are resolved. The remaining work is to convert the resolved framework into implementation-ready issue briefs and prototype milestones.
|
||||
# Distributed GGUF Runtime decision framework
|
||||
|
||||
> **Specification status:** planning artifacts only. No distributed GGUF runtime is implemented by this materialization, no story has completion credit, and legacy files remain for the DGR-017 audit. `prd.json` is authoritative.
|
||||
|
||||
## Decision order
|
||||
|
||||
1. DGR-019 locks comparable lanes and thresholds before results.
|
||||
2. DGR-020 runs safetensors and whole-model llama.cpp only, then returns `go`, `optimize baseline`, or `stop`.
|
||||
3. Dense and V4 work must prove parity, independent per-stage execution, local-state isolation, bounded failure, and measured resources.
|
||||
4. DGR-054 returns `alpha`, `optimize measured bottleneck`, or `stop`; MTP is explicitly off.
|
||||
5. Post-alpha optimizations must be selected from profiles, not assumptions.
|
||||
6. DGR-070 returns `beta`, `targeted optimization`, or `stop/rollback`, and requires MTP and the exact certified hardware/recipe matrix.
|
||||
|
||||
## Interpretation rules
|
||||
|
||||
- Quant/model-fit gains are separate from runtime/kernel/transport gains.
|
||||
- Fixture, real-model, real-hardware, and release evidence are never interchangeable.
|
||||
- 2–4 and 10+ stages are certification scenarios only.
|
||||
- Existing routing policy is certified, not redesigned.
|
||||
- Build success is not hardware certification; dark lanes remain unroutable.
|
||||
- Route loss uses cache miss and re-prefill/restart, never WAN cache migration.
|
||||
|
||||
## Locked scope
|
||||
|
||||
- Existing Meshnet Tracker routing, load balancing, billing, telemetry, relay, and provider semantics are backend-agnostic and are **not redesigned**. GGUF contributes exact compatibility, range/capacity, queue/load, seam-cost, health/reliability, and certification inputs only.
|
||||
- The data plane is a standalone project-owned C++ Shard worker with gRPC/Protobuf and a project-owned `ShardEngine` boundary.
|
||||
- llama.cpp is fetched at one exact commit into an ignored workspace from an in-repo manifest, then a numbered minimal patch stack is applied. There is no submodule, vendored tree, or permanent-fork dependency.
|
||||
- llama.cpp owns DeepSeek V4 graphs, mHC, MoE, attention, hash routing, and kernels. Meshnet adds only range-ownership hooks, typed boundary/local-state adapters, worker integration, and parity/certification.
|
||||
- Quantization and placement are dynamic recipe inputs. The 2–4 and 10+ stage layouts are certification scenarios, never product constants.
|
||||
- Per-shard Hot KV and V4 CSA/HCA/SWA/indexer/compressor state remain local and keyed by route session/epoch. The WAN seam carries the typed mHC 4×4096 residual boundary, positions, token-ID sideband where required, and schema/cache expectations—not per-layer caches.
|
||||
- Route changes use cache miss plus re-prefill/restart. There is no WAN KV or V4 auxiliary-cache migration.
|
||||
- CPU/CUDA/ROCm/Vulkan/Metal compile lanes are planned; only exact real-hardware-certified backend/model/recipe lanes may be advertised.
|
||||
- Alpha requires correctness and the pre-locked useful-speed gate. MTP is reserved and off for alpha; its ownership contract, implementation, and benchmark are required before beta.
|
||||
|
||||
## Target identities
|
||||
|
||||
- DeepSeek V4 official target SHA: `60d8d70770c6776ff598c94bb586a859a38244f1`.
|
||||
- llama.cpp V4 support lineage began at PR 24162 / merge `8c146a8366304c871efc26057cc90370ccf58dad`; DGR-027 later pins one exact validated current commit.
|
||||
- V4 scope: 43 main layers plus MTP; mHC 4×4096 boundary; 256 routed + 1 shared experts with six routed active; token IDs required for the first three hash-routed layers.
|
||||
- Exact split-GGUF artifacts are provisioned to mounted-drive storage with a complete hashed manifest and resumable verification; no model artifact may be placed under `/home`.
|
||||
|
||||
Reference in New Issue
Block a user