neuron-tai/.scratch/distributed-gguf-runtime/README.md

Performant concurrent distributed GGUF runtime

Status: active benchmark-gated implementation program.

Objective

Serve the exact pinned GLM-5.2 UD-IQ1_S artifact in reasoning_effort=max mode across consumer machines with useful measured performance. Dense Llama is a structural fixture; the real multi-node GLM target is the alpha release gate.

See GLM-5.2 Max distributed alpha roadmap for the target identity, minimum hardware, immutable acceptance matrix, and revised execution order.

Critical path

Meshnet control plane
  -> versioned gRPC/Protobuf Shard protocol
  -> project-owned standalone C++ worker
  -> small pinned llama.cpp patch stack

Transformers/safetensors remains the correctness baseline. vLLM remains an optional complete managed provider and a design donor; it is not forked into the public mesh.

Planning artifacts

• Mandatory Ralph context — read first in every fresh iteration
• Task evidence contract
• Implementation strategy
• Current architecture
• PRD
• Ralph backlog
• ADR-0020
• Milestones
• Issues
• Distributed GGUF research
• GitHub follow-up
• vLLM assessment

Ralph execution

Use supervised one-story iterations for this high-risk runtime:

ralph-tui run \
  --prd .scratch/distributed-gguf-runtime/prd.json \
  --agent claude --model opus \
  --iterations 1 --no-tui --no-setup --verify

Inspect the diff, run the story gates, and commit one verified story before the next iteration. Real-model stories require the explicit environment gate and mounted-drive model storage.