Files
neuron-tai/.scratch/distributed-gguf-runtime/issues/20-pass-real-distributed-glm-5-2-max-alpha.md
2026-07-13 22:48:26 +03:00

63 lines
5.1 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# 20 — Pass real distributed GLM-5.2 Max alpha acceptance
Status: ready-for-agent
## Mandatory fresh-session context
- Read [RALPH-CONTEXT.md](../RALPH-CONTEXT.md), [GLM-5.2-MAX-ALPHA-ROADMAP.md](../GLM-5.2-MAX-ALPHA-ROADMAP.md), and this issue completely before changing code.
- This issue is `DGR-020` in [prd.json](../prd.json).
- Read and verify every dependency evidence README and the immutable DGR-017 contract.
- Inspect live source, hardware, network, storage, and `git status`; synthetic or layer-reduced evidence cannot satisfy this story.
## Description
As an alpha user, I need the exact lowest-quant GLM-5.2 model to run in Max reasoning mode across consumer machines through Meshnet so that the project has proven its end goal on real hardware.
## Expected durable outputs
- Exact multi-node hardware/network/storage/runtime manifest
- Signed target identity, ownership, parity, quality, performance, and reliability reports
- Raw worker/tracker/API logs, metrics, prompts, outputs, and cleanup evidence
- Explicit immutable `alpha` or `stop` verdict
- `evidence/DGR-020/README.md`
## Acceptance criteria
- [ ] Use the exact DGR-017 `UD-IQ1_S` artifact and certified DGR-018/DGR-019 runtime recipe with all source and patch hashes verified.
- [ ] Use at least two physical consumer machines; each reserves at least `max(20% of physically usable memory, 8 GiB)` outside weight-plus-Q8-KV placement, no node can place the complete recipe, unified memory is counted once, and measured peak use stays within physical memory without swap/overcommit.
- [ ] Treat 5×64 GiB or 3×96/128 GiB as the recommended topology; arithmetic-minimum 4×64 or 2×128 qualifies only with exact contiguous placement and measured reserve evidence.
- [ ] Use a same-switch wired network with at least 2.5 GbE; record 10 GbE as recommended and measure hop RTT/serialization/queue latency.
- [ ] Tracker selects disjoint contiguous Shards whose exact required tensor inventory has complete union and no unintended overlap.
- [ ] Every stage reports real CPU/GPU compute, owned tensor bytes/layers, local KV, backend, queue, and seam telemetry; synthetic, passthrough, or zero-layer workers fail acceptance.
- [ ] Runtime evidence proves native GLM MoE/shared expert, DSA lightning indexer/sparse attention, and IndexShare Full/Shared paths are active; dense fallback fails acceptance.
- [ ] OpenAI-compatible API applies and records `reasoning_effort=max`, stable model ID, finish reason, and token usage.
- [ ] Configure 16,384 context, concurrency 1, and Q8_0 MLA/indexer KV; complete the fixed 4,096-token prefill lane and generate at least 512 output tokens or valid natural EOS after at least 128 tokens.
- [ ] Production seam meets at least 0.90 greedy token agreement and 0.999 mean compared-state/logit cosine similarity against DGR-018 on the fixed corpus, with no non-finite or malformed tensors.
- [ ] Fixed coding, structured tool-call/JSON, and multi-step reasoning sentinels produce parseable relevant outputs retained for human review.
- [ ] After warm-up, fixed Max lane median decode is at least 0.5 token/s, 4,096-token-prompt TTFT is at most 10 minutes, and no unexplained stall exceeds 60 seconds without progress telemetry.
- [ ] Record per-stage/seam p50/p95 latency, bytes, compute time, peak RSS/VRAM, KV pressure, network properties, errors, and total energy when available.
- [ ] Two consecutive cold starts load, generate, release, and exit without leaked processes, mapped weights, queues, or KV leases.
- [ ] Cancellation in prefill and decode releases all stages; one worker loss aborts the route; retry starts from token zero on a newly compatible route; stale epochs and duplicate steps fail closed.
- [ ] Model artifacts/caches remain on configured mounted-drive storage and never under `/home`; secret scan passes.
- [ ] Run targeted/full deterministic tests and clean native rebuild in addition to target execution; `git diff --check` passes.
- [ ] Preserve all raw failures and do not weaken DGR-017 thresholds after results.
- [ ] Write a signed `alpha` verdict only if every criterion passes; otherwise write `stop` with the measured bottleneck and next narrow action.
- [ ] Write `evidence/DGR-020/README.md` with exact commands, manifests, raw-result paths, outputs, limitations, and post-alpha handoff.
- [ ] Update only this story issue to `Status: done` after the immutable verdict and all evidence exist.
## Dependency handoff
- `DGR-007`, `DGR-008`, `DGR-009`, `DGR-011`, `DGR-013`, `DGR-017`, `DGR-018`, and `DGR-019` must have `passes: true`; read and verify their evidence READMEs.
- DGR-012 continuous batching and DGR-014 final comparative release gate are post-alpha and do not block one target session.
## Finish contract
- If required physical nodes/storage are unavailable, write `evidence/DGR-020/BLOCKED.md`; do not substitute a smaller model, API, synthetic worker, or single host.
- Preserve real failures and blockers; never fabricate target output or telemetry.
- Emit `<promise>COMPLETE</promise>` only after the signed verdict exists.
## References
- [GLM-5.2 Max alpha roadmap](../GLM-5.2-MAX-ALPHA-ROADMAP.md)
- [Ralph execution context](../RALPH-CONTEXT.md)