neuron-tai/.scratch/distributed-gguf-runtime/GLM-5.2-MAX-ALPHA-ROADMAP.md

# GLM-5.2 Max distributed alpha roadmap

Status: proposed executable epic target
Last updated: 2026-07-13

## Executive decision

The alpha-release target is the exact open-weight model `zai-org/GLM-5.2` served with `reasoning_effort=max` from the smallest published Unsloth GGUF recipe, `UD-IQ1_S`, across multiple consumer machines.

“Max” is a reasoning mode selected by the chat template/API request. It is not a separate model checkpoint.

Alpha is earned only when the real target model:

1. cannot fit within any one participating node's admitted memory;
2. loads as contiguous layer Shards on at least two physical consumer machines;
3. performs real GLM-5.2 MoE + DSA + IndexShare computation on every selected node;
4. responds through the existing OpenAI-compatible Meshnet API with `reasoning_effort=max`;
5. passes locked parity, usefulness, performance, telemetry, cancellation, and cleanup checks; and
6. stores all model artifacts on configured mounted-drive storage, never under `/home`.

The shortest safe path is not “support every GGUF architecture.” Dense Llama remains a small structural fixture. GLM-5.2 moves onto the critical path immediately after exact recipe identity and the pinned llama.cpp boundary. Qwen expansion, 1M context certification, MTP/speculative decoding, broad concurrency, and automatic route repair are post-alpha.

## 1. Exact target contract

### 1.1 Source model

| Field | Locked target |
|---|---|
| Official repository | `zai-org/GLM-5.2` |
| Official revision observed 2026-07-13 | `b4734de4facf877f85769a911abafc5283eab3d9` |
| License | MIT |
| Architecture | `glm_moe_dsa` / `GlmMoeDsaForCausalLM` |
| Total parameters | 744B |
| Active parameters | approximately 40B/token |
| Transformer layers | 78 main layers plus one shared NextN/MTP layer in the artifact |
| Routed experts | 256 |
| Experts selected | 8 routed experts plus shared expert path |
| Hidden width | 6,144 |
| Attention | MLA under DSA, lightning indexer top-k 2,048 |
| IndexShare | indexer roles are encoded by `indexer_types`; consumers reuse prior Full-layer indices |
| Architectural maximum context | 1,048,576 tokens |
| Alpha reasoning mode | `reasoning_effort=max` |

The runtime must derive these values from the pinned artifact and fail closed on contradictory metadata. Marketing names are not compatibility identity.

### 1.2 Alpha GGUF artifact

| Field | Locked target |
|---|---|
| GGUF repository | `unsloth/GLM-5.2-GGUF` |
| GGUF revision observed 2026-07-13 | `abc55e72527792c6e77069c99b4cb7de16fa9f23` |
| Quantization | `UD-IQ1_S` |
| Files | six GGUF shards |
| Exact published bytes | 216,715,360,960 bytes |
| Binary GiB | 201.832 GiB |
| Published quality indication | about 76.2% top-1 agreement with the high-precision reference on Unsloth's quantization analysis |
| Mounted storage rule | configured mounted drive only; never `/home` |

Before downloading 216.7 GB, DGR-017 must generate a checked-in target manifest containing repository revisions, expected filenames, byte sizes, and resolved LFS SHA-256 values. Download is resumable and verified before route admission.

`UD-IQ1_M` (228,492,966,624 bytes / 212.801 GiB) is the first diagnostic fallback if `UD-IQ1_S` exposes a runtime or quality defect. It does not satisfy the explicit “lowest quantization” alpha target unless the target contract is changed by human review.

### 1.3 Runtime semantics required for alpha

Required:

- GGUF parsing and quantized kernels from one exact llama.cpp pin.
- Correct GLM-5.2 MoE routing, selected experts, and shared expert.
- Correct compressed MLA KV cache for locally owned layers.
- Native DSA lightning indexer and sparse attention.
- Correct IndexShare Full/Shared role execution from artifact metadata.
- Range-owned contiguous transformer layers; each owned layer keeps all of its experts local.
- Head-only embeddings and tail-only final norm/output head/sampling.
- Architecture-defined activation boundary and optional DSA index sideband.
- `reasoning_effort=max` chat-template behavior through the public API.
- F32 seam correctness lane and a separately certified production activation dtype.

Not required for alpha:

- MTP/speculative decoding. The trailing NextN tensors may be loaded or explicitly excluded according to a certified recipe, but cannot be silently misinterpreted.
- Full 1,048,576-token context.
- Continuous batching beyond one target session.
- Public-WAN tensor or expert parallel collectives.
- Automatic mid-generation repair or KV migration.
- Every CPU/GPU backend combination.

## 2. Minimum resource envelope

### 2.1 Weight and runtime memory

The smallest artifact occupies 201.832 GiB before KV, scratch buffers, backend workspaces, process memory, and the operating system. The alpha planner uses **224 GiB aggregate usable model memory** as the initial conservative admission floor and records actual peak memory. “Usable” means memory available after per-node OS/runtime reserve, not installed capacity.

Unified memory is counted once. An integrated GPU's reported VRAM must not be added again to the same physical system RAM.

| Consumer-node class | Reserve per node | Usable per node | Minimum nodes for 224 GiB | Position |
|---:|---:|---:|---:|---|
| 32 GiB | 6 GiB | 26 GiB | 9 | technically possible; operationally poor |
| 48 GiB | 8 GiB | 40 GiB | 6 | possible |
| 64 GiB | 8 GiB | 56 GiB | 4 | minimum practical route |
| 96 GiB | 12 GiB | 84 GiB | 3 | preferred |
| 128 GiB unified/system | 16 GiB | 112 GiB | 2 | minimum high-memory route; narrow headroom |

The planner must use exact tensor byte ownership, not equal percentages. Embeddings, final head, shared experts, indexer tensors, quant block alignment, and backend workspace can make equal layer counts unequal in memory.

Recommended first target route: **three 96/128-GiB-class physical machines** or **four 64-GiB-class machines**, each with at least 1 GbE and mounted model storage. Two 128-GiB machines are a valuable minimum-fit trial but not the only certification topology.

### 2.2 KV cache

GLM-5.2 MLA stores approximately 576 latent/rope elements per token per main layer. Across 78 layers:

| Context | F16 KV, aggregate | Approx. Q4_1 KV, aggregate |
|---:|---:|---:|
| 16,384 | 1.37 GiB | 0.43 GiB |
| 131,072 | 10.97 GiB | 3.43 GiB |
| 1,048,576 | 87.75 GiB | 27.42 GiB |

These are planning estimates, not admission truth. The runtime must report measured allocated/resident KV by Shard and recipe. Alpha configures a 16,384-token window, uses one session, and records the exact KV dtype. Longer contexts are later certification gates.

### 2.3 Activation seams and network

A BF16 hidden-state boundary is 6,144 elements = 12,288 bytes/token before framing.

- A 16,384-token prefill sends about 192 MiB per seam.
- One decode token sends about 12 KiB per seam.
- A 512-token decode sends about 6 MiB per seam.
- Four nodes imply three serial seams.

If a Shard boundary splits an IndexShare producer/consumer group, a 2,048-entry int32 top-k sideband can add up to 8 KiB/query before framing. The route planner should prefer boundaries that preserve complete IndexShare ownership groups. The protocol must still support and validate the named sideband because memory fit may force an internal group split.

Bandwidth is manageable on 1 GbE; sequential seam latency and compute dominate decode. Alpha records per-seam bytes, p50/p95 transfer latency, retries, and checksum failures. No speed claim is inferred from link speed alone.

### 2.4 Storage

The shortest alpha path allows every node to hold the complete six-file source GGUF while mapping/allocating only owned tensors. This minimizes artifact-transformation risk but costs 216.7 GB disk per node.

Deterministic source-bound layer packages are a follow-up optimization. If needed before target fit, every package must retain:

- source repository/revision and source file hashes;
- exact owned tensor names, layer range, and endpoint role;
- tokenizer/config identity;
- deterministic package hash; and
- proof that composing all packages matches the source tensor inventory.

## 3. Current state and critical gaps

### Completed foundation

- DGR-001: immutable CPU contract plus separate signed ROCm diagnostic. CPU v1 remains `stop`; the GPU diagnostic establishes a viable fit/performance investigation lane but does not rewrite CPU evidence.
- DGR-002: versioned backend-neutral gRPC/Protobuf Shard protocol with bounded fragments and compatibility checks.
- Existing Meshnet: Tracker, contiguous Shards, Route Sessions/epochs, relay/direct transport, local Hot KV semantics in the reference backend, cancellation, telemetry, billing, and model-agnostic admission.

### Missing before target alpha

1. Exact GLM target/artifact manifest and memory-fit planner.
2. A current llama.cpp pin proven to load and generate with the exact `UD-IQ1_S` artifact.
3. A narrow decision on native GLM-5.2 DSA/IndexShare support. As observed 2026-07-13, llama.cpp issue #24730 remains open and community GGUF execution may use incomplete or compatibility paths.
4. Correct range-owned GGUF loading and memory proof.
5. GLM-specific boundary/KV/IndexShare semantics.
6. Standalone native worker and Meshnet integration.
7. Real target hardware route with no node individually able to admit the whole model.
8. Locked target parity, usefulness, speed, failure, and cleanup evidence.

### Donor policy

`Mesh-LLM/mesh-llm` is a high-value test and patch donor. Its live GLM branch was observed with 261 llama.cpp patches, 167 named for GLM/DSA/MTP-related work. That is evidence of the problem's depth, not an acceptable maintained fork boundary.

Audit and selectively reproduce the smallest independently understood pieces for:

- GLM DSA graph semantics;
- lightning indexer and sparse-attention tests;
- IndexShare metadata/Full/Shared validation;
- top-k sideband shape and lifecycle;
- stage-local KV filtering; and
- target parity/performance fixtures.

Do not import Mesh-LLM routing, discovery, scheduler, public mesh, package manager, or full patch stack. Keep Meshnet as the sole control plane and collaborate narrowly upstream with llama.cpp/Mesh-LLM maintainers where practical.

## 4. Revised roadmap

## Phase 0 — lock the target before implementation

### DGR-017: lock GLM-5.2 Max target and alpha contract

Deliver:

- machine-readable target manifest for official and GGUF revisions;
- exact `UD-IQ1_S` file/size/hash inventory;
- architecture/config/chat-template snapshot;
- memory/KV/network planner with unified-memory de-duplication;
- immutable alpha acceptance thresholds from section 5; and
- current upstream/donor status report.

Exit: target identity and alpha requirements are reviewable without downloading the model.

## Phase 1 — establish a correct whole-model oracle

### DGR-003: exact runtime recipe identity

Extend the existing generic identity with GLM fields: DSA/IndexShare metadata, adapter version, reasoning template revision, activation bundle schema, KV dtype/layout, llama.cpp pin/patch hash, and target artifact manifest hash.

### DGR-004: reproducible llama.cpp pin and narrow patch boundary

Select a current exact upstream commit only after testing its stock GLM behavior. Add clean fetch/apply/build checks. Record every donor patch and whether it is adopted, rewritten, rejected, or waiting upstream.

### DGR-018: certify whole-model GLM-5.2 runtime semantics

On a 224+ GiB usable reference host or temporary equivalent:

1. verify all six `UD-IQ1_S` shards;
2. load with a stock pinned runtime and capture tensor/metadata warnings;
3. prove whether DSA, IndexShare, shared expert, and Max template are actually active;
4. add the minimum correctness patches/tests required;
5. run deterministic prefill/decode and fixed Max-mode sentinel prompts; and
6. sign the oracle recipe, output, telemetry, and limitations.

Exit: one whole-model oracle exists for the same artifact/runtime semantics the distributed path will implement. “It emits text” is insufficient.

## Phase 2 — build the generic local seam using small fixtures

### DGR-005: range-owned GGUF tensors

Keep dense Llama as a cheap structural fixture. Implement authoritative owned-tensor registration/loading, head/tail ownership, and measured resident-memory scaling. Design tensor classification so GLM adds explicit rules rather than unchecked name substitution.

### DGR-006: architecture-defined boundary

Implement named boundary bundles, F32 correctness lane, bounded fragmentation, and optional sidebands. Dense fixture parity proves the seam mechanism, not GLM certification.

Exit: two local processes can execute a small dense model with correct range ownership and boundary parity.

## Phase 3 — add GLM-5.2 as the product adapter

### DGR-019: implement and certify GLM-5.2 range/DSA/IndexShare semantics

Deliver explicit support for:

- 78 main layers and endpoint tensor ownership;
- 256-expert MoE routing/top-8 and shared expert;
- compressed MLA KV by owned layers;
- DSA lightning indexer and sparse attention;
- IndexShare metadata, Full producer, Shared consumer, and sideband behavior;
- NextN/MTP tensor policy with MTP disabled or enabled explicitly;
- shard-boundary planner aware of IndexShare ownership groups; and
- whole-model versus two-stage parity against DGR-018.

Exit: a same-host two-stage target run matches the locked oracle tolerance with real GLM computation in both stages. If the full target cannot fit on one host for this check, use a layer-reduced GLM architecture fixture for graph parity and defer full-artifact output parity to DGR-020; label the distinction explicitly.

## Phase 4 — worker, KV, and Meshnet route

Execute existing stories with GLM requirements included:

1. DGR-007 — isolated local Hot KV keyed by `(Route Session, epoch)`, including DSA/IndexShare state.
2. DGR-008 — standalone C++ gRPC worker.
3. DGR-009 — Meshnet backend, capability, relay/direct, cancellation, and telemetry integration.
4. DGR-010 — small-model local two-process acceptance.
5. DGR-011 — real two-physical-machine route and heterogeneous fail-closed behavior.
6. DGR-013 subset required by alpha — node loss, cancellation, stale epoch, restart, and memory/KV cleanup.

Continuous batching (DGR-012) is deliberately not an alpha dependency. The first target release supports one admitted GLM route session; concurrency follows after target correctness and fit.

## Phase 5 — target alpha gate

### DGR-020: pass real distributed GLM-5.2 Max alpha acceptance

Use at least two physical machines and enough aggregate usable memory to meet the locked target planner. No participating node may individually admit the complete target. All stages must report real compute and exact tensor ownership.

Run the complete acceptance matrix in section 5, preserve raw logs/metrics/output, sign the evidence, and publish an explicit `alpha` or `stop` verdict. Thresholds cannot be weakened after results are known.

## Phase 6 — post-alpha hardening

After DGR-020 passes:

1. DGR-012 — 1/2/4-session continuous batching and bounded admission.
2. DGR-014 — final distributed GGUF versus reference-route performance decision.
3. 32K, 128K, 200K, then 1M context certification with quantized KV.
4. MTP/speculative decoding.
5. Deterministic range packages to remove full-artifact replication.
6. Additional backend compatibility classes and route topologies.
7. DGR-016 — narrow upstream collaboration package, split by independently reviewable llama.cpp changes.
8. DGR-015 — Qwen3/Qwen3-MoE only as later architecture expansion, not as the GLM alpha target.

## 5. Locked alpha acceptance matrix

These thresholds are set before target execution.

### 5.1 Identity and fit

- Exact official and GGUF repository revisions match the target manifest.
- All six source GGUF sizes and LFS SHA-256 values verify.
- Every route node reports owned tensor names/bytes, layer range, endpoint role, backend, KV recipe, and patch fingerprint.
- Union of owned tensors equals the certified runtime-required tensor inventory; unintended overlap is zero.
- No node's admitted usable memory can hold the complete recipe.
- Aggregate peak RSS/VRAM stays within admitted budgets with no swap-driven success claim.
- Unified RAM/VRAM is not double-counted.

### 5.2 Semantic correctness

- Logs and graph tests prove GLM MoE/shared-expert, DSA lightning indexer, sparse attention, and IndexShare Full/Shared paths are active; dense-attention compatibility fallback cannot satisfy alpha.
- `reasoning_effort=max` is observable in the rendered template/API recipe.
- F32 same-backend seam fixture: 32 greedy decode tokens exactly match the whole-model oracle and activation tolerance is locked by DGR-006.
- Production seam on the fixed prompt corpus: greedy token agreement is at least 0.90 and mean compared-state/logit cosine similarity is at least 0.999 versus DGR-018, with no malformed or non-finite tensors.
- Incompatible artifact, tokenizer, adapter, DSA metadata, boundary, activation, KV, backend class, or runtime patch fingerprints fail closed.

### 5.3 End-to-end target run

- Context configured to 16,384 tokens.
- Fixed 4,096-token prompt lane completes prefill.
- One Max-mode request generates at least 512 output tokens or reaches a valid natural EOS after at least 128 tokens.
- Fixed coding, structured tool-call/JSON, and multi-step reasoning sentinels produce parseable, relevant outputs; raw prompts and outputs are retained for review.
- OpenAI-compatible response includes stable model ID, finish reason, and token usage.

### 5.4 Minimum useful performance

On the declared minimum alpha topology after one warm-up:

- median decode throughput is at least 0.5 generated token/s for the fixed Max-mode lane;
- 4,096-token-prompt TTFT is at most 10 minutes;
- no unexplained stall exceeds 60 seconds without progress telemetry;
- per-stage compute, queue, KV, seam bytes/latency, RSS/VRAM, and backend timing are present; and
- results are labeled by hardware/topology and are not generalized to other consumer systems.

If output quality passes but the speed floor fails, verdict is `stop` for alpha and the evidence selects the next optimization target. It is not relabeled as success merely because the model loaded.

### 5.5 Reliability and security

- Two consecutive cold starts load, generate, release, and exit cleanly.
- Cancellation during prefill and decode releases every stage's queued buffers and KV lease.
- One worker loss aborts the route; alpha retries only from token zero on a new compatible route.
- Stale epochs and duplicate step IDs are rejected.
- Artifact paths stay outside `/home`; logs contain no secrets or unrestricted prompt payloads.
- Synthetic workers and layer-reduced fixtures are labeled unit/integration coverage and cannot satisfy target alpha.

## 6. First execution order

The next unattended work should run in this order:

1. DGR-017 — target contract, manifest, planner, and upstream status.
2. DGR-003 — exact recipe identity.
3. DGR-004 — current llama.cpp pin and minimal patch harness.
4. Run in parallel:
   - DGR-018 — whole-model oracle on 224+ GiB usable memory.
   - DGR-005 and DGR-006 — generic range/boundary seam on local small fixtures.
5. DGR-019 — GLM semantics and parity after both parallel lanes pass.
6. DGR-007 through DGR-011 — native worker and real transport route.
7. Required DGR-013 failure subset.
8. DGR-020 — real target alpha verdict.

The first external hardware blocker is DGR-018, but DGR-005/DGR-006 proceed locally while that host is sourced. Do not download the full model until DGR-017's exact manifest and storage preflight pass.

## 7. Sources checked on 2026-07-13

Authoritative or primary:

- Official model card and config: <https://huggingface.co/zai-org/GLM-5.2>
- Official source revision API: <https://huggingface.co/api/models/zai-org/GLM-5.2>
- Official GLM-5 technical report: <https://arxiv.org/abs/2602.15763>
- Unsloth GGUF repository: <https://huggingface.co/unsloth/GLM-5.2-GGUF>
- Unsloth local-run/quantization guide: <https://unsloth.ai/docs/models/glm-5.2>
- llama.cpp GLM-5.2 support issue: <https://github.com/ggml-org/llama.cpp/issues/24730>
- llama.cpp 1M-context discussion: <https://github.com/ggml-org/llama.cpp/discussions/24622>
- IndexCache/IndexShare paper: <https://arxiv.org/abs/2603.12201>

Donor/current implementation evidence:

- Mesh-LLM repository: <https://github.com/Mesh-LLM/mesh-llm>
- Mesh-LLM GLM branch noted by llama.cpp collaborator in issue #24730: `feat/jianyang-glm-52`

Web/repository observations are pinned by date and must be refreshed in DGR-017 before implementation because upstream support is moving quickly.