neuron-tai/.scratch/distributed-gguf-runtime/decision-framework.md

Distributed GGUF Decision Framework

Superseded for active implementation decisions. The grill was resolved on 2026-07-13. Use implementation-strategy.md, architecture.md, ADR-0020, and prd.json. This file remains as historical decision rationale.

This framework is for grilling open decisions. It keeps decisions tied to project vocabulary and implementation gates instead of vague "distributed inference" language.

Core Vocabulary

Use the existing domain terms this way:

• Shard: contiguous transformer layer range. This is the compute, routing, cache, and reward unit.
• Shard Swarm: storage/download group for artifacts needed by a shard.
• Inference Route: ordered node sequence that covers all layers for one request.
• Route Session: one active request bound to one inference route and stable session id.
• Hot KV State: live per-shard cache held by the route node during a route session.
• Prefix Snapshot: persisted route-session state used for reuse or failover, not the hot decode path.
• Artifact Manifest: canonical mapping from model artifacts to semantic model parts and runtime support.
• Generation Telemetry: realtime progress for a route session, including phase and tokens/sec, independent of whether token deltas are streamed.

The Five Planes

1. Control Plane

Owner: Tracker.

Responsibilities:

• node registry
• coverage map
• route selection
• rebalance directives
• route-session creation
• health and telemetry
• client-visible Generation Telemetry
• billing/audit records

Must not do:

• serve hot KV during every token
• become the only place model artifacts can be fetched

2. Artifact Plane

Owner: Shard Swarms, local node storage, optional CDN/bootstrap mirrors.

Responsibilities:

• GGUF/safetensors/tokenizer download
• content-addressed verification
• local artifact inventory
• artifact-to-layer mapping
• cache eviction

Must not do:

• define execution order by file split alone
• imply that a downloaded file chunk equals a Shard

3. Execution Plane

Owner: active Inference Route.

Responsibilities:

• chunked prefill
• one-step decode
• hidden-state transfer across activation seams
• start-layer handling for overlapping shards
• backpressure

Must not do:

• resend full context activations during decode
• require cross-node tensor parallel all-reduce for public v1

4. Session State Plane

Owner: route nodes for hot KV; cache servers only for snapshots.

Responsibilities:

• per-shard local KV ownership
• cache allocation and eviction
• cache ABI compatibility
• session close/release
• optional prefix snapshots

Must not do:

• centralize hot KV in a remote service
• let a replacement node continue from incompatible state

5. Economics And Trust Plane

Owner: tracker plus settlement/validation components.

Responsibilities:

• distinguish storage/seeding work from inference work
• account for prefill and decode separately
• record route participation
• sample validation events
• slash proven fraud

Must not do:

• pay a node for merely holding files as if it generated tokens
• hide public-swarm privacy limits from clients

Hard Invariants

These are the framework rules unless we deliberately write a new ADR:

1. Public-network Shards are contiguous layer ranges.
2. Hot KV State is local to the node serving that Shard in that Route Session.
3. Artifact distribution and route execution are separate systems.
4. Decode seam payload must be O(hidden_size).
5. Prefill may be O(sequence_length * hidden_size), but only in bounded chunks.
6. The tracker chooses routes; nodes do not negotiate route topology peer-to-peer.
7. Model/backend-specific cache internals stay behind backend capability reports.
8. PyTorch remains the correctness/reference backend while llama.cpp/GGUF becomes the performance backend.
9. Streaming responses are preferred when feasible; Generation Telemetry is always required.

Resolved Gates

Gate 1: Public Shard Semantics

Decision: public-network Shards are contiguous transformer layer ranges. Tensor-parallel or ring-style execution is allowed only inside one trusted node, one colocated pod, or a future composite node abstraction.

Rationale:

• Layer ranges match the existing Shard, Coverage Map, Inference Route, billing, and fraud vocabulary.
• Public volunteer nodes should not require cross-node all-reduce or tight per-layer synchronization in v1.
• Existing projects such as prima.cpp and Distributed Llama can still inform local-cluster/backend execution without becoming the public routing primitive.

Consequences:

• Artifact Manifests must map files/tensors to semantic layer ranges.
• Route selection remains ordered layer coverage.
• Rewards can be attributed to layer-range work.
• Hot KV State is naturally owned by the node serving that layer range for the Route Session.

Gate 2: Hot KV Strategy

Decision: v1 rejects centralized hot KV. Hot KV State is local to the node serving the relevant Shard in the active Route Session. Cache servers may store Prefix Snapshots for reuse, retry, or failover, but they are not in the per-token decode path.

Rationale:

• Decode is the tight loop; adding remote cache I/O there makes latency and bandwidth worse at the worst point.
• Local KV naturally follows layer-range Shard ownership.
• Centralized hot KV increases privacy exposure and creates consistency problems.
• Prefix Snapshots preserve the useful part of central storage without making it mandatory for every generated token.

Consequences:

• Route Session must be sticky.
• Failover is limited in alpha unless a compatible Prefix Snapshot exists.
• Cache servers are optimization infrastructure, not required runtime infrastructure.
• Route repair requires compatible model revision, layer range, backend cache ABI, and snapshot position.

Gate 3: First Runtime Proof

Decision: prove distributed Route Session and Hot KV State semantics in the existing PyTorch route before modifying llama.cpp/GGUF.

Rationale:

• PyTorch exposes model internals and cache objects more directly, so it is the fastest way to validate the distributed protocol.
• The current distributed PyTorch route already has the right high-level shape but disables cache and recomputes full prompts.
• Fixing that path gives us a reference implementation for correctness tests, telemetry, session lifecycle, and wire protocol behavior.
• llama.cpp/GGUF should receive a clear target ABI rather than becoming both the protocol experiment and the performance backend at once.

Consequences:

• Issue 02 precedes issue 05.
• llama.cpp collaboration has a concrete target ABI.
• The PyTorch route remains the architecture-coverage/reference backend even after GGUF becomes the preferred performance path.
• The first success metric is eliminating full-prompt recompute in distributed decode.

Gate 3A: Client Feedback During Latency

Decision: streaming responses are preferred when feasible, and realtime Generation Telemetry is required regardless of streaming support.

Rationale:

• The product optimizes for access to large capable models, so some latency is acceptable.
• Users still need confidence that the route is alive and roughly how fast it is generating.
• Streaming token deltas give the best user experience when the backend exposes them cleanly.
• Tokens/sec remains useful during prefill, queueing, and any backend that cannot stream token deltas.

Consequences:

• The gateway should stream token deltas through an OpenAI-compatible response when possible.
• The gateway must expose progress through SSE, WebSocket, or polling.
• The final answer can be delivered after completion only as a fallback.
• Telemetry must include route phase, generated token count, and rolling tokens/sec.
• Non-streaming clients still need realtime telemetry.

Gate 4: llama.cpp Collaboration Shape

Decision: target upstreamable libllama/ggml hooks instead of planning around a permanent fork.

Rationale:

• llama.cpp changes quickly across model support, quantization, kernels, and hardware backends.
• A permanent fork would become expensive to maintain and would lag upstream improvements.
• A short-lived prototype branch is acceptable if it proves the API and makes upstream collaboration concrete.
• Keeping tracker/routing logic outside llama.cpp makes the upstream ask smaller and cleaner.

Consequences:

• Need a minimal reproducible localhost demo before asking upstream to carry the design.
• Need to separate "what llama.cpp should expose" from "what our tracker does".
• Desired upstream surface is layer-range execution, hidden-state boundary I/O, partial loading/introspection, and per-session KV ownership.
• If upstream rejects the shape, we revisit whether to carry a narrow adapter fork or keep GGUF distributed execution as experimental.

Gate 5: First Model Target

Decision: use a two-tier model target. Use a small, boring, llama.cpp-supported GGUF model for the first protocol smoke test. Use deepseek-ai/DeepSeek-V4-Flash as the first serious large-model target. Keep GLM-5.2 and Ornith as later support audits.

Rationale:

• The first protocol proof should isolate route/session/KV bugs from model-architecture bugs.
• DeepSeek-V4-Flash is a strong first serious target because it is much smaller than 1.6T-class models while still being large enough to validate the product thesis.
• DeepSeek-V4-Flash still has architecture-specific risks, so it should not be the first smoke test.
• GLM-5.2 and Ornith remain valuable targets, but they add DSA/MLA/hybrid attention uncertainty.

Consequences:

• 128K cache accounting can be modeled now.
• The first "real" target-model audit is DeepSeek-V4-Flash support in PyTorch, vLLM/SGLang, and any available GGUF/llama.cpp quantization path.
• Production support waits for backend capability reports and exact cache ABI support.

Gate 6: Failure Semantics

Decision: alpha fails Route Sessions on route-node loss instead of attempting automatic route repair.

Rationale:

• Route repair requires compatible Prefix Snapshots, cache ABI checks, replacement-node selection, billing correction, and client stream/error recovery.
• Local Hot KV State means a replacement node cannot continue unless it has compatible state at the same position.
• Fail-fast keeps the first implementation correct while the session/KV protocol is still being proven.

Consequences:

• Better observability and explicit errors are required.
• Snapshotting becomes a later feature, not a blocker for first inference.
• Generation Telemetry must report the last known phase and failure reason.
• Client or gateway retry starts a new Route Session from scratch.

Gate 7: Transport

Decision: keep binary HTTP for v1 activation transfer instead of jumping immediately to QUIC, WebRTC, or a custom transport.

Rationale:

• ADR-0008 already defines binary activation bodies with HTTP headers.
• HTTP keeps the first implementation debuggable with the existing server stack and tooling.
• The core risk is route/session/KV correctness, not transport optimization.
• QUIC/WebRTC can be introduced later behind the same activation protocol once semantics are proven.

Consequences:

• Focus benchmark work on payload shape, chunking, and cache behavior first.
• QUIC/WebRTC can be introduced as an optimization behind the same activation protocol.
• v1 implementation can reuse the current HTTP routing, relay, and observability infrastructure.
• Transport abstraction should be kept narrow enough that HTTP can be replaced later without changing backend cache semantics.

Grilling Progress

Gates 1, 2, 3, 3A, 4, 5, 6, and 7 are resolved. The remaining work is to convert the resolved framework into implementation-ready issue briefs and prototype milestones.