neuron-tai/.scratch/distributed-gguf-runtime/technical-challenges.md

Distributed GGUF Technical Challenge Register

Historical challenge register. Route Session, binary activation, local Hot KV State, and transport performance work have advanced since this file was written. Current implementation gates live in PRD.md, implementation-strategy.md, architecture.md, and prd.json. Preserve this file for detailed risk context; do not treat its “current constraint” section as live system state.

This document focuses on the engineering problems that decide whether the distributed GGUF path is viable. The important distinction is:

• Model artifacts move like torrents.
• Inference state moves like a pipeline.
• Hot KV state does not move unless we are explicitly checkpointing or repairing a route.

Current Constraint

The existing full local PyTorch path lets Transformers own generation and local KV cache.

The existing distributed PyTorch path does not. It manually calls shard layers with cache disabled and recomputes the whole growing prompt for every generated token. It passes hidden activations across shard boundaries, not KV cache, but those activations currently include the full sequence on every decode step.

For a 128K context and hidden_size=6144, one bfloat16 activation crossing one shard boundary is roughly:

131072 tokens * 6144 hidden * 2 bytes = 1.5 GiB

That is acceptable once during chunked prefill only if chunked and streamed. It is not acceptable once per generated token.

Challenge 1: Decode Must Be O(1) Per Token Across Each Seam

Problem:

During decode, sending [batch, sequence, hidden] over the network scales with context length. At 128K, the network dominates everything.

Solution:

Split execution into explicit prefill and decode-step phases.

• Prefill accepts prompt chunks and builds local cache on every shard.
• Decode-step accepts exactly one new token or one-step activation.
• Every shard reads its own hot KV state, appends one position, and forwards a one-step activation.

Acceptance test:

• A two-node route prefills a 4K prompt once.
• The next 100 generated tokens do not resend a 4K activation.
• Decode seam payload is proportional to hidden_size, not context_length * hidden_size.

Challenge 2: Stable Route Session State

Problem:

KV cache only works if every hop agrees that multiple calls belong to the same route session. A fresh request id per hop or per token destroys cache locality.

Solution:

Introduce a route-session lifecycle.

create route session
  -> tracker pins inference route
  -> head node assigns session_id and route_id
  -> every hop allocates local cache for its layer range
  -> prefill chunks append cache
  -> decode steps append cache
  -> close session releases cache

Minimum state key:

session_id
route_id
model_preset
model_revision
backend_id
cache_abi
layer_start
layer_end
position

Acceptance test:

• A node can report active sessions and cache bytes by session.
• Closing a session frees the per-shard cache.
• Replaying a decode-step with the wrong route/session fails before model execution.

Challenge 3: KV Cache Ownership

Problem:

A centralized KV cache sounds attractive for failover, but it puts remote storage in the tightest loop of generation. It also creates privacy and consistency problems.

Solution:

Hot KV state is owned by the node that owns the shard for that route session.

Node A: layers 0..15 hot KV for session S
Node B: layers 16..31 hot KV for session S
Node C: layers 32..77 hot KV for session S

The tracker may know where active KV lives, but it does not serve it during decode.

Cache servers may store:

• prefix snapshots
• failover checkpoints
• audit samples
• cold reusable context blocks

They must not be required for every generated token.

Acceptance test:

• Killing a cache server does not affect an active decode route.
• Killing a route node fails the route in alpha unless a compatible snapshot exists.

Challenge 4: Prefill Is Still Large

Problem:

Even with correct decode, prefill can move a lot of data. A 128K prompt cannot be sent as one activation blob through many shard boundaries.

Solution:

Use the existing binary activation direction from ADR-0008:

• bfloat16 activation body
• shape/dtype/session metadata in headers
• zstd level 1 optional compression
• chunked prefill
• backpressure between hops

For large contexts, prefill should stream in chunks such as 128, 256, or 512 tokens. The right chunk size is a benchmark output, not a constant baked into the domain model.

Acceptance test:

• Peak per-hop prefill memory is bounded by chunk size.
• A slow downstream node applies backpressure instead of letting upstream buffer the whole prompt.

Challenge 5: GGUF Artifact Splits Are Not Execution Shards

Problem:

GGUF split files can divide model data by size or tensor count. That is useful for storage and transfer, but it is not the same as a network Shard. A Shard in this project is a contiguous layer range with reward, route, and cache meaning.

Solution:

Define an artifact manifest that maps storage chunks to semantic model parts.

Required concepts:

artifact_id
model_preset
upstream_repo
upstream_revision
license
runtime_backend
quantization
context_cap
file_hashes
piece_hashes
tensor_to_layer_map
layer_to_artifact_map
tokenizer_artifacts
cache_descriptor

The Shard Swarm seeds artifacts. The Inference Route executes shards.

Acceptance test:

• Given layers 16..31, a node can compute the exact artifact pieces it must download and verify.
• Given a local artifact directory, the node can prove which layer ranges it can serve.

Challenge 6: llama.cpp Is Optimized For Whole-Graph Execution

Problem:

llama-server is excellent for local inference, but a distributed route needs lower-level capabilities:

• load selected layers/tensors or mmap them without full materialization
• accept hidden states from a previous shard
• execute only a layer range
• emit hidden states at a boundary
• own KV/state for only that layer range
• report cache layout and memory requirements

Those are not the normal public serving abstractions.

Solution:

Stage the llama.cpp path instead of jumping directly to internet-scale distributed GGUF.

1. Use llama.cpp as a full local GGUF backend for immediate CPU performance.
2. Build a localhost layer-boundary prototype on a simple supported GGUF model.
3. Identify the minimal libllama/ggml hooks needed for layer-range execution.
4. Collaborate upstream on a stable extension rather than carrying a long-lived fork.

Acceptance test:

• Process A runs layers 0..k, exports hidden states.
• Process B imports those hidden states, runs k+1..n, and produces logits close to full single-process execution.
• Both processes maintain only their own cache state.

Challenge 7: Model Architectures Are Not Uniform

Problem:

Dense Llama-style attention, MoE, MLA/DSA, and hybrid linear/full attention do not have the same cache shape, routing cost, or layer cost.

GLM-5.2 uses compressed DSA/MLA-style state. Ornith uses a hybrid attention pattern. Parameter count alone is a poor routing metric.

Solution:

Keep model-specific cache internals inside the backend. The tracker should route based on backend-advertised capabilities and measured telemetry, not on hardcoded tensor formulas.

Backend capability report:

model_arch
supported_runtime
supports_prefill
supports_decode_step
supports_layer_range
supports_partial_artifacts
cache_abi
cache_bytes_per_token_estimate
prefill_tokens_per_second
decode_tokens_per_second
active_memory_floor

Acceptance test:

• The tracker can reject a route because one node lacks the required cache ABI.
• A model support audit can say "artifact available, local full inference works, distributed layer-boundary unsupported" without ambiguity.

Challenge 8: Tensor Parallelism Is Not The Same Product

Problem:

Projects like Distributed Llama and prima.cpp lean toward local-cluster tensor/ring parallelism. That can work on a trusted LAN, but it usually requires tight synchronization every layer. On a public internet volunteer route, that becomes fragile and hard to reward.

Solution:

For the public network, make a Shard a contiguous layer range. Tensor parallelism can exist inside one node, one trusted colocated pod, or one future "composite node", but not as the first public routing primitive.

Acceptance test:

• A public route can be represented as ordered layer coverage.
• Billing can attribute work to layer ranges.
• No cross-node all-reduce is required on every layer for v1.

Challenge 9: Heterogeneity And Stragglers

Problem:

A route is only as fast as its slowest hop during decode. A weak node holding a bottleneck shard can make a 1.6T model technically available but unusable.

Solution:

Route selection must use measured telemetry, not static declarations.

Metrics:

• prefill throughput
• decode throughput
• queue depth
• disk read rate
• memory pressure
• network latency to neighbors
• route failure rate
• cache warmth

The tracker should prefer complete routes that avoid weak nodes, and the rebalancer should increase redundancy for bottleneck layer ranges.

Acceptance test:

• A slow node is removed from a candidate route even if it has the needed layer range.
• The coverage map can show "covered but under-provisioned" separately from "coverage gap".

Challenge 10: Reliability And Failover

Problem:

If hot KV is local, route repair is not free. A replacement node cannot continue decoding unless it has compatible cache state.

Solution:

Alpha behavior should be simple:

• route failure during prefill: fail and retry from scratch
• route failure during decode: fail unless compatible snapshot exists
• tracker restart: active sessions may be lost
• node restart: local hot KV is lost
• client-visible telemetry reports the last known phase and failure reason

Later behavior:

• periodic prefix snapshots
• snapshot generation ids
• cache ABI compatibility checks
• route repair only when the replacement node has the same model revision, layer range, backend cache ABI, and snapshot position

Acceptance test:

• Failures produce explicit route-session errors.
• No node silently continues from missing or incompatible cache state.

Challenge 11: Privacy, Fraud, And Audit

Problem:

Hidden activations and KV state can leak information. Public volunteer inference is not private by default. Also, a node can return bad activations while still appearing available.

Solution:

Separate product modes:

• public swarm: low privacy, broad access, audited
• private swarm: trusted nodes, stronger privacy expectation
• paid trusted route: selected nodes with stronger guarantees

Use existing validation-event and slash-proof concepts for audit, but adapt them to distributed routes:

• record model preset, route, node wallets, prompt metadata, output, and sampling seed
• sample full-route replays where feasible
• compare output/logits within model-specific tolerance

Acceptance test:

• A client can choose public or private route policy.
• A validation event contains enough information to reproduce route membership and observed output.

Challenge 12: Economics Must Not Reward The Wrong Bottleneck

Problem:

Layer count, parameter count, active MoE experts, cache memory, disk serving, and network transfer are different costs. A naive equal split across nodes will be wrong.

Solution:

Start with simple compute accounting:

node_reward_weight =
  owned_layer_work
  * prefill_tokens
  + owned_layer_work
  * decode_tokens

Then refine with:

• measured throughput
• active MoE cost
• storage/seeding contribution
• cache memory reservation
• reliability

Keep artifact seeding rewards separate from inference rewards until fraud and metering are clear.

Acceptance test:

• A node that only seeds artifacts is not paid as if it executed inference.
• A node that executes a heavier shard can earn more than a node executing a light shard.

Challenge 13: Long Requests Need Streaming Or Realtime Feedback

Problem:

Large distributed routes may spend meaningful time in artifact loading, prefill, queueing, or slow decode. The product can tolerate latency, but users should not wait blindly.

Solution:

Streaming token deltas is preferred when the backend and client transport support it. Generation Telemetry is required regardless of whether token deltas are streamed.

Minimum telemetry:

session_id
route_id
model_preset
phase = queued | loading | prefill | decode | finalizing | failed
prefill_tokens_done
prefill_tokens_total
generated_tokens
rolling_tokens_per_second
average_tokens_per_second
active_route_nodes
failure_reason

The gateway may expose token deltas and telemetry through Server-Sent Events or WebSocket. Simple clients may use a polling endpoint for telemetry and receive the final answer only when complete.

Acceptance test:

• A client can show live progress before the first output token is available.
• During decode, the user sees streamed token deltas when supported.
• During decode, the user sees rolling tokens/sec even if output text is not streamed.
• A failed route returns a final error and the last known phase/reason.

Engineering Order

1. Fix distributed PyTorch cache semantics first. This proves the route-session model without llama.cpp internals.
2. Add local full-model llama.cpp/GGUF serving for immediate CPU improvement.
3. Add Generation Telemetry for route sessions so long requests are observable.
4. Preserve binary HTTP activation transfer while splitting prefill/decode and measuring payload sizes.
5. Add artifact manifest and Shard Swarm metadata.
6. Prototype llama.cpp layer-boundary execution locally.
7. Network the GGUF route only after the cache/session protocol works.
8. Audit DeepSeek-V4-Flash as the first serious large-model target.
9. Audit GLM-5.2 and Ornith support after simpler GGUF models pass the route test.