popov/neuron-tai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: harden GLM alpha resource and protocol gates

Browse Source
This commit is contained in:
Dobromir Popov
2026-07-13 22:48:26 +03:00
parent 5ebce15d7a
commit 9580ed643e
11 changed files with 86 additions and 48 deletions
Download Patch File Download Diff File Expand all files Collapse all files

85
.scratch/distributed-gguf-runtime/GLM-5.2-MAX-ALPHA-ROADMAP.md
View File

@@ -28,11 +28,13 @@ The shortest safe path is not “support every GGUF architecture.” Dense Llama
|---|---|
| Official repository | `zai-org/GLM-5.2` |
| Official revision observed 2026-07-13 | `b4734de4facf877f85769a911abafc5283eab3d9` |
| License | MIT |
| Model-weight license | MIT |
| Official code/documentation license | Apache-2.0 |
| Architecture | `glm_moe_dsa` / `GlmMoeDsaForCausalLM` |
| Total parameters | 744B |
| Active parameters | approximately 40B/token |
| Transformer layers | 78 main layers plus one shared NextN/MTP layer in the artifact |
| Official architecture label | 744B total / approximately 40B active per token |
| Exact stored checkpoint tensors | 753,329,940,480 parameters |
| Transformer layers | 78 backbone layers plus one shared NextN/MTP layer in the artifact |
| Layer types | first 3 dense; remaining 75 sparse MoE |
| Routed experts | 256 |
| Experts selected | 8 routed experts plus shared expert path |
| Hidden width | 6,144 |
@@ -88,33 +90,39 @@ Not required for alpha:
### 2.1 Weight and runtime memory
The smallest artifact occupies 201.832 GiB before KV, scratch buffers, backend workspaces, process memory, and the operating system. The alpha planner uses **224 GiB aggregate usable model memory** as the initial conservative admission floor and records actual peak memory. “Usable” means memory available after per-node OS/runtime reserve, not installed capacity.
The smallest artifact occupies 201.832 GiB before KV, DSA indexer state, scratch buffers, backend workspaces, process memory, and the operating system. **224 GiB aggregate runtime-accessible memory is only the experimental hard-fit floor**, consistent with Unsloth's approximate 223 GB one-bit requirement. It is not a conservative operational envelope.
Unified memory is counted once. An integrated GPU's reported VRAM must not be added again to the same physical system RAM.
For admission, each node reserves:
| Consumer-node class | Reserve per node | Usable per node | Minimum nodes for 224 GiB | Position |
```text
max(20% of physically usable memory, 8 GiB)
```
The remainder is the combined weight-plus-KV placement budget. Actual peak scratch is measured by backend/context and can force one extra node. Unified memory is counted once: integrated-GPU “VRAM” must not be added again to the same physical system RAM.
| Physical usable tier | Minimum reserve | Weight + KV placement budget | IQ1_S 16K arithmetic minimum | Operational position |
|---:|---:|---:|---:|---|
| 32 GiB | 6 GiB | 26 GiB | 9 | technically possible; operationally poor |
| 48 GiB | 8 GiB | 40 GiB | 6 | possible |
| 64 GiB | 8 GiB | 56 GiB | 4 | minimum practical route |
| 96 GiB | 12 GiB | 84 GiB | 3 | preferred |
| 128 GiB unified/system | 16 GiB | 112 GiB | 2 | minimum high-memory route; narrow headroom |
| 32 GiB | 8.0 GiB | 24.0 GiB | 9 nodes | use 10 if attempted; latency-heavy |
| 48 GiB | 9.6 GiB | 38.4 GiB | 6 nodes | possible; latency-heavy |
| 64 GiB | 12.8 GiB | 51.2 GiB | 4 nodes | hard minimum; **5 recommended** |
| 96 GiB | 19.2 GiB | 76.8 GiB | 3 nodes | recommended |
| 128 GiB unified/system | 25.6 GiB | 102.4 GiB | 2 nodes | arithmetic hard minimum; **3 recommended** |
The planner must use exact tensor byte ownership, not equal percentages. Embeddings, final head, shared experts, indexer tensors, quant block alignment, and backend workspace can make equal layer counts unequal in memory.
The planner must use exact tensor byte ownership, not equal percentages. Embeddings, final head, dense versus MoE layers, shared experts, indexer tensors, quant block alignment, KV distribution, and backend workspace make equal layer counts unequal in memory.
Recommended first target route: **three 96/128-GiB-class physical machines** or **four 64-GiB-class machines**, each with at least 1 GbE and mounted model storage. Two 128-GiB machines are a valuable minimum-fit trial but not the only certification topology.
Recommended first target route: **three 96/128-GiB-class physical machines** or **five 64-GiB-class machines**, on the same wired switch with mounted model storage. Four 64-GiB or two 128-GiB machines are fit probes only and qualify solely if exact placement and measured peak-memory evidence retain the required reserve with no swap/overcommit.
### 2.2 KV cache
GLM-5.2 MLA stores approximately 576 latent/rope elements per token per main layer. Across 78 layers:
GLM-5.2 MLA caches 576 latent/rope values per token per backbone layer. Correct DSA also caches 128-dimensional indexer keys: ideally only for the 21 Full indexer layers, while the current experimental implementation may allocate them across all 78 layers. Alpha locks **Q8_0 KV** for quality and budgets the conservative current-implementation layout.
| Context | F16 KV, aggregate | Approx. Q4_1 KV, aggregate |
|---:|---:|---:|
| 16,384 | 1.37 GiB | 0.43 GiB |
| 131,072 | 10.97 GiB | 3.43 GiB |
| 1,048,576 | 87.75 GiB | 27.42 GiB |
| Context × concurrency | MLA-only Q8 | Optimized DSA Q8 | Conservative current-DSA Q8 | Conservative current-DSA F16 |
|---:|---:|---:|---:|---:|
| 16,384 × 1 | 0.73 GiB | 0.77 GiB | **0.89 GiB** | 1.68 GiB |
| 131,072 × 1 | 5.83 GiB | 6.18 GiB | **7.12 GiB** | 13.41 GiB |
| 1,048,576 × 1 | 46.62 GiB | 49.41 GiB | **56.98 GiB** | 107.25 GiB |
These are planning estimates, not admission truth. The runtime must report measured allocated/resident KV by Shard and recipe. Alpha configures a 16,384-token window, uses one session, and records the exact KV dtype. Longer contexts are later certification gates.
These are planning estimates, not admission truth. The runtime must report measured allocated/resident MLA and indexer cache by Shard. Alpha configures a 16,384-token window, Q8_0 KV, and one session. Longer contexts and lower-bit KV are separate quality/resource certification gates.
### 2.3 Activation seams and network
@@ -127,7 +135,7 @@ A BF16 hidden-state boundary is 6,144 elements = 12,288 bytes/token before frami
If a Shard boundary splits an IndexShare producer/consumer group, a 2,048-entry int32 top-k sideband can add up to 8 KiB/query before framing. The route planner should prefer boundaries that preserve complete IndexShare ownership groups. The protocol must still support and validate the named sideband because memory fit may force an internal group split.
Bandwidth is manageable on 1 GbE; sequential seam latency and compute dominate decode. Alpha records per-seam bytes, p50/p95 transfer latency, retries, and checksum failures. No speed claim is inferred from link speed alone.
Decode bandwidth is small, but every generated token crosses all seams serially, so node count and per-hop latency dominate. Alpha requires a same-switch wired route: **2.5 GbE minimum and 10 GbE recommended**, with measured one-way/RTT, serialization, and queue latency. A 1 GbE route may be retained as fit-only evidence but is not the recommended alpha topology. Alpha records per-seam bytes, p50/p95 transfer latency, retries, and checksum failures; no speed claim is inferred from link rate alone.
### 2.4 Storage
@@ -153,12 +161,13 @@ Deterministic source-bound layer packages are a follow-up optimization. If neede
1. Exact GLM target/artifact manifest and memory-fit planner.
2. A current llama.cpp pin proven to load and generate with the exact `UD-IQ1_S` artifact.
3. A narrow decision on native GLM-5.2 DSA/IndexShare support. As observed 2026-07-13, llama.cpp issue #24730 remains open and community GGUF execution may use incomplete or compatibility paths.
4. Correct range-owned GGUF loading and memory proof.
5. GLM-specific boundary/KV/IndexShare semantics.
6. Standalone native worker and Meshnet integration.
7. Real target hardware route with no node individually able to admit the whole model.
8. Locked target parity, usefulness, speed, failure, and cleanup evidence.
3. A narrow decision on native GLM-5.2 DSA/IndexShare support. As observed 2026-07-13, merged llama.cpp PR #24770 loads GLM-5.2 through a dense-MLA compatibility path, while full IndexShare/DSA PR #25407 remains open and its generic sparse path can be slower than dense fallback. Generic CPU lightning-indexer support is merged; backend coverage remains uneven.
4. A decode protocol amendment. `ActivationChunk` carries `TensorBundle`, but the current `DecodeStep` fast path carries only one `NamedTensor`; it cannot transport a hidden state plus GLM top-k sideband. Tail token/logit and sampling behavior also needs an explicit typed result contract.
5. Correct range-owned GGUF loading and memory proof.
6. GLM-specific boundary/KV/IndexShare semantics.
7. Standalone native worker and Meshnet integration.
8. Real target hardware route with no node individually able to admit the whole model.
9. Locked target parity, usefulness, speed, failure, and cleanup evidence.
### Donor policy
@@ -204,7 +213,7 @@ Select a current exact upstream commit only after testing its stock GLM behavior
### DGR-018: certify whole-model GLM-5.2 runtime semantics
On a 224+ GiB usable reference host or temporary equivalent:
On a 256-GiB-class reference host with at least 224 GiB runtime-accessible memory after OS reservation, or a measured equivalent:
1. verify all six `UD-IQ1_S` shards;
2. load with a stock pinned runtime and capture tensor/metadata warnings;
@@ -223,7 +232,7 @@ Keep dense Llama as a cheap structural fixture. Implement authoritative owned-te
### DGR-006: architecture-defined boundary
Implement named boundary bundles, F32 correctness lane, bounded fragmentation, and optional sidebands. Dense fixture parity proves the seam mechanism, not GLM certification.
Implement named boundary bundles, F32 correctness lane, bounded fragmentation, and optional sidebands. Amend the decode fast path so it carries a versioned `TensorBundle` rather than one `NamedTensor`, while preserving a small one-tensor encoding. Define an explicit typed tail result for logits/token output and bind sampling/chat-template parameters to the recipe/request. Regenerate Python/C++ schema code and compatibility goldens. Dense fixture parity proves the seam mechanism, not GLM certification.
Exit: two local processes can execute a small dense model with correct range ownership and boundary parity.
@@ -288,8 +297,10 @@ These thresholds are set before target execution.
- All six source GGUF sizes and LFS SHA-256 values verify.
- Every route node reports owned tensor names/bytes, layer range, endpoint role, backend, KV recipe, and patch fingerprint.
- Union of owned tensors equals the certified runtime-required tensor inventory; unintended overlap is zero.
- No node's admitted usable memory can hold the complete recipe.
- Aggregate peak RSS/VRAM stays within admitted budgets with no swap-driven success claim.
- No node's weight-plus-KV placement budget can hold the complete recipe.
- Every node reserves at least `max(20% of physically usable memory, 8 GiB)` outside weight-plus-KV placement; measured peak scratch must remain inside that reserve.
- Aggregate peak RSS/VRAM stays within physical budgets with no swap, overcommit, mmap-only, or double-counted unified-memory success claim.
- Arithmetic-minimum topologies require exact contiguous tensor placement evidence; recommended alpha topology is 5×64 GiB or 3×96/128 GiB.
- Unified RAM/VRAM is not double-counted.
### 5.2 Semantic correctness
@@ -302,8 +313,9 @@ These thresholds are set before target execution.
### 5.3 End-to-end target run
- Context configured to 16,384 tokens.
- Context configured to 16,384 tokens with Q8_0 MLA/indexer KV.
- Fixed 4,096-token prompt lane completes prefill.
- Route uses a same-switch wired network; 2.5 GbE is the alpha minimum and 10 GbE is recommended.
- One Max-mode request generates at least 512 output tokens or reaches a valid natural EOS after at least 128 tokens.
- Fixed coding, structured tool-call/JSON, and multi-step reasoning sentinels produce parseable, relevant outputs; raw prompts and outputs are retained for review.
- OpenAI-compatible response includes stable model ID, finish reason, and token usage.
@@ -337,7 +349,7 @@ The next unattended work should run in this order:
2. DGR-003 — exact recipe identity.
3. DGR-004 — current llama.cpp pin and minimal patch harness.
4. Run in parallel:
- DGR-018 — whole-model oracle on 224+ GiB usable memory.
- DGR-018 — whole-model oracle on a 256-GiB-class host with at least 224 GiB runtime-accessible memory.
- DGR-005 and DGR-006 — generic range/boundary seam on local small fixtures.
5. DGR-019 — GLM semantics and parity after both parallel lanes pass.
6. DGR-007 through DGR-011 — native worker and real transport route.
@@ -351,11 +363,16 @@ The first external hardware blocker is DGR-018, but DGR-005/DGR-006 proceed loca
Authoritative or primary:
- Official model card and config: <https://huggingface.co/zai-org/GLM-5.2>
- Official release/architecture blog: <https://z.ai/blog/glm-5.2>
- Official code/documentation repository: <https://github.com/zai-org/GLM-5>
- Official source revision API: <https://huggingface.co/api/models/zai-org/GLM-5.2>
- Official GLM-5 technical report: <https://arxiv.org/abs/2602.15763>
- Unsloth GGUF repository: <https://huggingface.co/unsloth/GLM-5.2-GGUF>
- Unsloth local-run/quantization guide: <https://unsloth.ai/docs/models/glm-5.2>
- llama.cpp GLM-5.2 support issue: <https://github.com/ggml-org/llama.cpp/issues/24730>
- llama.cpp merged dense-MLA compatibility loader: <https://github.com/ggml-org/llama.cpp/pull/24770>
- llama.cpp open GLM-5.2 DSA/IndexShare implementation: <https://github.com/ggml-org/llama.cpp/pull/25407>
- llama.cpp merged generic CPU lightning indexer: <https://github.com/ggml-org/llama.cpp/pull/24231>
- llama.cpp 1M-context discussion: <https://github.com/ggml-org/llama.cpp/discussions/24622>
- IndexCache/IndexShare paper: <https://arxiv.org/abs/2603.12201>