neuron-tai/.scratch/distributed-gguf-runtime/issues/06-networked-distributed-gguf-route.md

06 — Networked distributed GGUF route

Status: pending

Depends on: 01, 03, 04, 05

Goal

Run a GGUF-backed model over a real multi-node route using the tracker-selected route and per-shard local KV.

Scope

• Extend node backend registry with GGUF layer ranges.
• Add route selection for GGUF nodes.
• Use the prefill/decode protocol.
• Track route health and queue depth.
• Bill by layer work and token work.

Acceptance

• Two physical machines can serve one model route.
• Node dropout during alpha fails request cleanly.
• Tracker metrics show prefill TPS, decode TPS, seam latency, and cache memory.