popov/neuron-tai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
7364ed673144873ec4341169a3e1d2bd1308f64a
neuron-tai/.scratch/distributed-gguf-runtime/GLM-5.2-MAX-ALPHA-ROADMAP.md
Dobromir Popov 9580ed643e docs: harden GLM alpha resource and protocol gates
2026-07-13 22:48:26 +03:00

24 KiB
Raw Blame History

GLM-5.2 Max distributed alpha roadmap

Status: proposed executable epic target Last updated: 2026-07-13

Executive decision

The alpha-release target is the exact open-weight model zai-org/GLM-5.2 served with reasoning_effort=max from the smallest published Unsloth GGUF recipe, UD-IQ1_S, across multiple consumer machines.

“Max” is a reasoning mode selected by the chat template/API request. It is not a separate model checkpoint.

Alpha is earned only when the real target model:

  1. cannot fit within any one participating node's admitted memory;
  2. loads as contiguous layer Shards on at least two physical consumer machines;
  3. performs real GLM-5.2 MoE + DSA + IndexShare computation on every selected node;
  4. responds through the existing OpenAI-compatible Meshnet API with reasoning_effort=max;
  5. passes locked parity, usefulness, performance, telemetry, cancellation, and cleanup checks; and
  6. stores all model artifacts on configured mounted-drive storage, never under /home.

The shortest safe path is not “support every GGUF architecture.” Dense Llama remains a small structural fixture. GLM-5.2 moves onto the critical path immediately after exact recipe identity and the pinned llama.cpp boundary. Qwen expansion, 1M context certification, MTP/speculative decoding, broad concurrency, and automatic route repair are post-alpha.

1. Exact target contract

1.1 Source model

Field Locked target
Official repository zai-org/GLM-5.2
Official revision observed 2026-07-13 b4734de4facf877f85769a911abafc5283eab3d9
Model-weight license MIT
Official code/documentation license Apache-2.0
Architecture glm_moe_dsa / GlmMoeDsaForCausalLM
Official architecture label 744B total / approximately 40B active per token
Exact stored checkpoint tensors 753,329,940,480 parameters
Transformer layers 78 backbone layers plus one shared NextN/MTP layer in the artifact
Layer types first 3 dense; remaining 75 sparse MoE
Routed experts 256
Experts selected 8 routed experts plus shared expert path
Hidden width 6,144
Attention MLA under DSA, lightning indexer top-k 2,048
IndexShare indexer roles are encoded by indexer_types; consumers reuse prior Full-layer indices
Architectural maximum context 1,048,576 tokens
Alpha reasoning mode reasoning_effort=max

The runtime must derive these values from the pinned artifact and fail closed on contradictory metadata. Marketing names are not compatibility identity.

1.2 Alpha GGUF artifact

Field Locked target
GGUF repository unsloth/GLM-5.2-GGUF
GGUF revision observed 2026-07-13 abc55e72527792c6e77069c99b4cb7de16fa9f23
Quantization UD-IQ1_S
Files six GGUF shards
Exact published bytes 216,715,360,960 bytes
Binary GiB 201.832 GiB
Published quality indication about 76.2% top-1 agreement with the high-precision reference on Unsloth's quantization analysis
Mounted storage rule configured mounted drive only; never /home

Before downloading 216.7 GB, DGR-017 must generate a checked-in target manifest containing repository revisions, expected filenames, byte sizes, and resolved LFS SHA-256 values. Download is resumable and verified before route admission.

UD-IQ1_M (228,492,966,624 bytes / 212.801 GiB) is the first diagnostic fallback if UD-IQ1_S exposes a runtime or quality defect. It does not satisfy the explicit “lowest quantization” alpha target unless the target contract is changed by human review.

1.3 Runtime semantics required for alpha

Required:

  • GGUF parsing and quantized kernels from one exact llama.cpp pin.
  • Correct GLM-5.2 MoE routing, selected experts, and shared expert.
  • Correct compressed MLA KV cache for locally owned layers.
  • Native DSA lightning indexer and sparse attention.
  • Correct IndexShare Full/Shared role execution from artifact metadata.
  • Range-owned contiguous transformer layers; each owned layer keeps all of its experts local.
  • Head-only embeddings and tail-only final norm/output head/sampling.
  • Architecture-defined activation boundary and optional DSA index sideband.
  • reasoning_effort=max chat-template behavior through the public API.
  • F32 seam correctness lane and a separately certified production activation dtype.

Not required for alpha:

  • MTP/speculative decoding. The trailing NextN tensors may be loaded or explicitly excluded according to a certified recipe, but cannot be silently misinterpreted.
  • Full 1,048,576-token context.
  • Continuous batching beyond one target session.
  • Public-WAN tensor or expert parallel collectives.
  • Automatic mid-generation repair or KV migration.
  • Every CPU/GPU backend combination.

2. Minimum resource envelope

2.1 Weight and runtime memory

The smallest artifact occupies 201.832 GiB before KV, DSA indexer state, scratch buffers, backend workspaces, process memory, and the operating system. 224 GiB aggregate runtime-accessible memory is only the experimental hard-fit floor, consistent with Unsloth's approximate 223 GB one-bit requirement. It is not a conservative operational envelope.

For admission, each node reserves:

max(20% of physically usable memory, 8 GiB)

The remainder is the combined weight-plus-KV placement budget. Actual peak scratch is measured by backend/context and can force one extra node. Unified memory is counted once: integrated-GPU “VRAM” must not be added again to the same physical system RAM.

Physical usable tier Minimum reserve Weight + KV placement budget IQ1_S 16K arithmetic minimum Operational position
32 GiB 8.0 GiB 24.0 GiB 9 nodes use 10 if attempted; latency-heavy
48 GiB 9.6 GiB 38.4 GiB 6 nodes possible; latency-heavy
64 GiB 12.8 GiB 51.2 GiB 4 nodes hard minimum; 5 recommended
96 GiB 19.2 GiB 76.8 GiB 3 nodes recommended
128 GiB unified/system 25.6 GiB 102.4 GiB 2 nodes arithmetic hard minimum; 3 recommended

The planner must use exact tensor byte ownership, not equal percentages. Embeddings, final head, dense versus MoE layers, shared experts, indexer tensors, quant block alignment, KV distribution, and backend workspace make equal layer counts unequal in memory.

Recommended first target route: three 96/128-GiB-class physical machines or five 64-GiB-class machines, on the same wired switch with mounted model storage. Four 64-GiB or two 128-GiB machines are fit probes only and qualify solely if exact placement and measured peak-memory evidence retain the required reserve with no swap/overcommit.

2.2 KV cache

GLM-5.2 MLA caches 576 latent/rope values per token per backbone layer. Correct DSA also caches 128-dimensional indexer keys: ideally only for the 21 Full indexer layers, while the current experimental implementation may allocate them across all 78 layers. Alpha locks Q8_0 KV for quality and budgets the conservative current-implementation layout.

Context × concurrency MLA-only Q8 Optimized DSA Q8 Conservative current-DSA Q8 Conservative current-DSA F16
16,384 × 1 0.73 GiB 0.77 GiB 0.89 GiB 1.68 GiB
131,072 × 1 5.83 GiB 6.18 GiB 7.12 GiB 13.41 GiB
1,048,576 × 1 46.62 GiB 49.41 GiB 56.98 GiB 107.25 GiB

These are planning estimates, not admission truth. The runtime must report measured allocated/resident MLA and indexer cache by Shard. Alpha configures a 16,384-token window, Q8_0 KV, and one session. Longer contexts and lower-bit KV are separate quality/resource certification gates.

2.3 Activation seams and network

A BF16 hidden-state boundary is 6,144 elements = 12,288 bytes/token before framing.

  • A 16,384-token prefill sends about 192 MiB per seam.
  • One decode token sends about 12 KiB per seam.
  • A 512-token decode sends about 6 MiB per seam.
  • Four nodes imply three serial seams.

If a Shard boundary splits an IndexShare producer/consumer group, a 2,048-entry int32 top-k sideband can add up to 8 KiB/query before framing. The route planner should prefer boundaries that preserve complete IndexShare ownership groups. The protocol must still support and validate the named sideband because memory fit may force an internal group split.

Decode bandwidth is small, but every generated token crosses all seams serially, so node count and per-hop latency dominate. Alpha requires a same-switch wired route: 2.5 GbE minimum and 10 GbE recommended, with measured one-way/RTT, serialization, and queue latency. A 1 GbE route may be retained as fit-only evidence but is not the recommended alpha topology. Alpha records per-seam bytes, p50/p95 transfer latency, retries, and checksum failures; no speed claim is inferred from link rate alone.

2.4 Storage

The shortest alpha path allows every node to hold the complete six-file source GGUF while mapping/allocating only owned tensors. This minimizes artifact-transformation risk but costs 216.7 GB disk per node.

Deterministic source-bound layer packages are a follow-up optimization. If needed before target fit, every package must retain:

  • source repository/revision and source file hashes;
  • exact owned tensor names, layer range, and endpoint role;
  • tokenizer/config identity;
  • deterministic package hash; and
  • proof that composing all packages matches the source tensor inventory.

3. Current state and critical gaps

Completed foundation

  • DGR-001: immutable CPU contract plus separate signed ROCm diagnostic. CPU v1 remains stop; the GPU diagnostic establishes a viable fit/performance investigation lane but does not rewrite CPU evidence.
  • DGR-002: versioned backend-neutral gRPC/Protobuf Shard protocol with bounded fragments and compatibility checks.
  • Existing Meshnet: Tracker, contiguous Shards, Route Sessions/epochs, relay/direct transport, local Hot KV semantics in the reference backend, cancellation, telemetry, billing, and model-agnostic admission.

Missing before target alpha

  1. Exact GLM target/artifact manifest and memory-fit planner.
  2. A current llama.cpp pin proven to load and generate with the exact UD-IQ1_S artifact.
  3. A narrow decision on native GLM-5.2 DSA/IndexShare support. As observed 2026-07-13, merged llama.cpp PR #24770 loads GLM-5.2 through a dense-MLA compatibility path, while full IndexShare/DSA PR #25407 remains open and its generic sparse path can be slower than dense fallback. Generic CPU lightning-indexer support is merged; backend coverage remains uneven.
  4. A decode protocol amendment. ActivationChunk carries TensorBundle, but the current DecodeStep fast path carries only one NamedTensor; it cannot transport a hidden state plus GLM top-k sideband. Tail token/logit and sampling behavior also needs an explicit typed result contract.
  5. Correct range-owned GGUF loading and memory proof.
  6. GLM-specific boundary/KV/IndexShare semantics.
  7. Standalone native worker and Meshnet integration.
  8. Real target hardware route with no node individually able to admit the whole model.
  9. Locked target parity, usefulness, speed, failure, and cleanup evidence.

Donor policy

Mesh-LLM/mesh-llm is a high-value test and patch donor. Its live GLM branch was observed with 261 llama.cpp patches, 167 named for GLM/DSA/MTP-related work. That is evidence of the problem's depth, not an acceptable maintained fork boundary.

Audit and selectively reproduce the smallest independently understood pieces for:

  • GLM DSA graph semantics;
  • lightning indexer and sparse-attention tests;
  • IndexShare metadata/Full/Shared validation;
  • top-k sideband shape and lifecycle;
  • stage-local KV filtering; and
  • target parity/performance fixtures.

Do not import Mesh-LLM routing, discovery, scheduler, public mesh, package manager, or full patch stack. Keep Meshnet as the sole control plane and collaborate narrowly upstream with llama.cpp/Mesh-LLM maintainers where practical.

4. Revised roadmap

Phase 0 — lock the target before implementation

DGR-017: lock GLM-5.2 Max target and alpha contract

Deliver:

  • machine-readable target manifest for official and GGUF revisions;
  • exact UD-IQ1_S file/size/hash inventory;
  • architecture/config/chat-template snapshot;
  • memory/KV/network planner with unified-memory de-duplication;
  • immutable alpha acceptance thresholds from section 5; and
  • current upstream/donor status report.

Exit: target identity and alpha requirements are reviewable without downloading the model.

Phase 1 — establish a correct whole-model oracle

DGR-003: exact runtime recipe identity

Extend the existing generic identity with GLM fields: DSA/IndexShare metadata, adapter version, reasoning template revision, activation bundle schema, KV dtype/layout, llama.cpp pin/patch hash, and target artifact manifest hash.

DGR-004: reproducible llama.cpp pin and narrow patch boundary

Select a current exact upstream commit only after testing its stock GLM behavior. Add clean fetch/apply/build checks. Record every donor patch and whether it is adopted, rewritten, rejected, or waiting upstream.

DGR-018: certify whole-model GLM-5.2 runtime semantics

On a 256-GiB-class reference host with at least 224 GiB runtime-accessible memory after OS reservation, or a measured equivalent:

  1. verify all six UD-IQ1_S shards;
  2. load with a stock pinned runtime and capture tensor/metadata warnings;
  3. prove whether DSA, IndexShare, shared expert, and Max template are actually active;
  4. add the minimum correctness patches/tests required;
  5. run deterministic prefill/decode and fixed Max-mode sentinel prompts; and
  6. sign the oracle recipe, output, telemetry, and limitations.

Exit: one whole-model oracle exists for the same artifact/runtime semantics the distributed path will implement. “It emits text” is insufficient.

Phase 2 — build the generic local seam using small fixtures

DGR-005: range-owned GGUF tensors

Keep dense Llama as a cheap structural fixture. Implement authoritative owned-tensor registration/loading, head/tail ownership, and measured resident-memory scaling. Design tensor classification so GLM adds explicit rules rather than unchecked name substitution.

DGR-006: architecture-defined boundary

Implement named boundary bundles, F32 correctness lane, bounded fragmentation, and optional sidebands. Amend the decode fast path so it carries a versioned TensorBundle rather than one NamedTensor, while preserving a small one-tensor encoding. Define an explicit typed tail result for logits/token output and bind sampling/chat-template parameters to the recipe/request. Regenerate Python/C++ schema code and compatibility goldens. Dense fixture parity proves the seam mechanism, not GLM certification.

Exit: two local processes can execute a small dense model with correct range ownership and boundary parity.

Phase 3 — add GLM-5.2 as the product adapter

DGR-019: implement and certify GLM-5.2 range/DSA/IndexShare semantics

Deliver explicit support for:

  • 78 main layers and endpoint tensor ownership;
  • 256-expert MoE routing/top-8 and shared expert;
  • compressed MLA KV by owned layers;
  • DSA lightning indexer and sparse attention;
  • IndexShare metadata, Full producer, Shared consumer, and sideband behavior;
  • NextN/MTP tensor policy with MTP disabled or enabled explicitly;
  • shard-boundary planner aware of IndexShare ownership groups; and
  • whole-model versus two-stage parity against DGR-018.

Exit: a same-host two-stage target run matches the locked oracle tolerance with real GLM computation in both stages. If the full target cannot fit on one host for this check, use a layer-reduced GLM architecture fixture for graph parity and defer full-artifact output parity to DGR-020; label the distinction explicitly.

Phase 4 — worker, KV, and Meshnet route

Execute existing stories with GLM requirements included:

  1. DGR-007 — isolated local Hot KV keyed by (Route Session, epoch), including DSA/IndexShare state.
  2. DGR-008 — standalone C++ gRPC worker.
  3. DGR-009 — Meshnet backend, capability, relay/direct, cancellation, and telemetry integration.
  4. DGR-010 — small-model local two-process acceptance.
  5. DGR-011 — real two-physical-machine route and heterogeneous fail-closed behavior.
  6. DGR-013 subset required by alpha — node loss, cancellation, stale epoch, restart, and memory/KV cleanup.

Continuous batching (DGR-012) is deliberately not an alpha dependency. The first target release supports one admitted GLM route session; concurrency follows after target correctness and fit.

Phase 5 — target alpha gate

DGR-020: pass real distributed GLM-5.2 Max alpha acceptance

Use at least two physical machines and enough aggregate usable memory to meet the locked target planner. No participating node may individually admit the complete target. All stages must report real compute and exact tensor ownership.

Run the complete acceptance matrix in section 5, preserve raw logs/metrics/output, sign the evidence, and publish an explicit alpha or stop verdict. Thresholds cannot be weakened after results are known.

Phase 6 — post-alpha hardening

After DGR-020 passes:

  1. DGR-012 — 1/2/4-session continuous batching and bounded admission.
  2. DGR-014 — final distributed GGUF versus reference-route performance decision.
  3. 32K, 128K, 200K, then 1M context certification with quantized KV.
  4. MTP/speculative decoding.
  5. Deterministic range packages to remove full-artifact replication.
  6. Additional backend compatibility classes and route topologies.
  7. DGR-016 — narrow upstream collaboration package, split by independently reviewable llama.cpp changes.
  8. DGR-015 — Qwen3/Qwen3-MoE only as later architecture expansion, not as the GLM alpha target.

5. Locked alpha acceptance matrix

These thresholds are set before target execution.

5.1 Identity and fit

  • Exact official and GGUF repository revisions match the target manifest.
  • All six source GGUF sizes and LFS SHA-256 values verify.
  • Every route node reports owned tensor names/bytes, layer range, endpoint role, backend, KV recipe, and patch fingerprint.
  • Union of owned tensors equals the certified runtime-required tensor inventory; unintended overlap is zero.
  • No node's weight-plus-KV placement budget can hold the complete recipe.
  • Every node reserves at least max(20% of physically usable memory, 8 GiB) outside weight-plus-KV placement; measured peak scratch must remain inside that reserve.
  • Aggregate peak RSS/VRAM stays within physical budgets with no swap, overcommit, mmap-only, or double-counted unified-memory success claim.
  • Arithmetic-minimum topologies require exact contiguous tensor placement evidence; recommended alpha topology is 5×64 GiB or 3×96/128 GiB.
  • Unified RAM/VRAM is not double-counted.

5.2 Semantic correctness

  • Logs and graph tests prove GLM MoE/shared-expert, DSA lightning indexer, sparse attention, and IndexShare Full/Shared paths are active; dense-attention compatibility fallback cannot satisfy alpha.
  • reasoning_effort=max is observable in the rendered template/API recipe.
  • F32 same-backend seam fixture: 32 greedy decode tokens exactly match the whole-model oracle and activation tolerance is locked by DGR-006.
  • Production seam on the fixed prompt corpus: greedy token agreement is at least 0.90 and mean compared-state/logit cosine similarity is at least 0.999 versus DGR-018, with no malformed or non-finite tensors.
  • Incompatible artifact, tokenizer, adapter, DSA metadata, boundary, activation, KV, backend class, or runtime patch fingerprints fail closed.

5.3 End-to-end target run

  • Context configured to 16,384 tokens with Q8_0 MLA/indexer KV.
  • Fixed 4,096-token prompt lane completes prefill.
  • Route uses a same-switch wired network; 2.5 GbE is the alpha minimum and 10 GbE is recommended.
  • One Max-mode request generates at least 512 output tokens or reaches a valid natural EOS after at least 128 tokens.
  • Fixed coding, structured tool-call/JSON, and multi-step reasoning sentinels produce parseable, relevant outputs; raw prompts and outputs are retained for review.
  • OpenAI-compatible response includes stable model ID, finish reason, and token usage.

5.4 Minimum useful performance

On the declared minimum alpha topology after one warm-up:

  • median decode throughput is at least 0.5 generated token/s for the fixed Max-mode lane;
  • 4,096-token-prompt TTFT is at most 10 minutes;
  • no unexplained stall exceeds 60 seconds without progress telemetry;
  • per-stage compute, queue, KV, seam bytes/latency, RSS/VRAM, and backend timing are present; and
  • results are labeled by hardware/topology and are not generalized to other consumer systems.

If output quality passes but the speed floor fails, verdict is stop for alpha and the evidence selects the next optimization target. It is not relabeled as success merely because the model loaded.

5.5 Reliability and security

  • Two consecutive cold starts load, generate, release, and exit cleanly.
  • Cancellation during prefill and decode releases every stage's queued buffers and KV lease.
  • One worker loss aborts the route; alpha retries only from token zero on a new compatible route.
  • Stale epochs and duplicate step IDs are rejected.
  • Artifact paths stay outside /home; logs contain no secrets or unrestricted prompt payloads.
  • Synthetic workers and layer-reduced fixtures are labeled unit/integration coverage and cannot satisfy target alpha.

6. First execution order

The next unattended work should run in this order:

  1. DGR-017 — target contract, manifest, planner, and upstream status.
  2. DGR-003 — exact recipe identity.
  3. DGR-004 — current llama.cpp pin and minimal patch harness.
  4. Run in parallel:
    • DGR-018 — whole-model oracle on a 256-GiB-class host with at least 224 GiB runtime-accessible memory.
    • DGR-005 and DGR-006 — generic range/boundary seam on local small fixtures.
  5. DGR-019 — GLM semantics and parity after both parallel lanes pass.
  6. DGR-007 through DGR-011 — native worker and real transport route.
  7. Required DGR-013 failure subset.
  8. DGR-020 — real target alpha verdict.

The first external hardware blocker is DGR-018, but DGR-005/DGR-006 proceed locally while that host is sourced. Do not download the full model until DGR-017's exact manifest and storage preflight pass.

7. Sources checked on 2026-07-13

Authoritative or primary:

Donor/current implementation evidence:

Web/repository observations are pinned by date and must be refreshed in DGR-017 before implementation because upstream support is moving quickly.

Reference in New Issue View Git Blame Copy Permalink