49 lines
2.0 KiB
Markdown
49 lines
2.0 KiB
Markdown
# Performant concurrent distributed GGUF runtime
|
||
|
||
Status: active benchmark-gated implementation program.
|
||
|
||
## Objective
|
||
|
||
Serve the exact pinned GLM-5.2 `UD-IQ1_S` artifact in `reasoning_effort=max` mode across consumer machines with useful measured performance. Dense Llama is a structural fixture; the real multi-node GLM target is the alpha release gate.
|
||
|
||
See **[GLM-5.2 Max distributed alpha roadmap](GLM-5.2-MAX-ALPHA-ROADMAP.md)** for the target identity, minimum hardware, immutable acceptance matrix, and revised execution order. The 224-GiB figure is an experimental hard-fit floor; recommended topology is 5×64 GiB or 3×96/128 GiB after the required per-node reserve.
|
||
|
||
## Critical path
|
||
|
||
```text
|
||
Meshnet control plane
|
||
-> versioned gRPC/Protobuf Shard protocol
|
||
-> project-owned standalone C++ worker
|
||
-> small pinned llama.cpp patch stack
|
||
```
|
||
|
||
Transformers/safetensors remains the correctness baseline. vLLM remains an optional complete managed provider and a design donor; it is not forked into the public mesh.
|
||
|
||
## Planning artifacts
|
||
|
||
- **[Mandatory Ralph context](RALPH-CONTEXT.md)** — read first in every fresh iteration
|
||
- [Task evidence contract](evidence/README.md)
|
||
- [Implementation strategy](implementation-strategy.md)
|
||
- [Current architecture](architecture.md)
|
||
- [PRD](PRD.md)
|
||
- [Ralph backlog](prd.json)
|
||
- [ADR-0020](ADR-0020-distributed-gguf-runtime.md)
|
||
- [Milestones](milestones.md)
|
||
- [Issues](issues/)
|
||
- [Distributed GGUF research](../../docs/research/distributed-gguf-landscape.md)
|
||
- [GitHub follow-up](../../docs/research/distributed-gguf-github-followup.md)
|
||
- [vLLM assessment](../../docs/research/vllm-distributed-gguf-assessment.md)
|
||
|
||
## Ralph execution
|
||
|
||
Use supervised one-story iterations for this high-risk runtime:
|
||
|
||
```bash
|
||
ralph-tui run \
|
||
--prd .scratch/distributed-gguf-runtime/prd.json \
|
||
--agent claude --model opus \
|
||
--iterations 1 --no-tui --no-setup --verify
|
||
```
|
||
|
||
Inspect the diff, run the story gates, and commit one verified story before the next iteration. Real-model stories require the explicit environment gate and mounted-drive model storage.
|