docs: target GLM-5.2 Max for distributed alpha
This commit is contained in:
Dobromir Popov
367
.scratch/distributed-gguf-runtime/GLM-5.2-MAX-ALPHA-ROADMAP.md
Normal file
367
.scratch/distributed-gguf-runtime/GLM-5.2-MAX-ALPHA-ROADMAP.md
Normal file
@@ -0,0 +1,367 @@
|
||||
# GLM-5.2 Max distributed alpha roadmap
|
||||
|
||||
Status: proposed executable epic target
|
||||
Last updated: 2026-07-13
|
||||
|
||||
## Executive decision
|
||||
|
||||
The alpha-release target is the exact open-weight model `zai-org/GLM-5.2` served with `reasoning_effort=max` from the smallest published Unsloth GGUF recipe, `UD-IQ1_S`, across multiple consumer machines.
|
||||
|
||||
“Max” is a reasoning mode selected by the chat template/API request. It is not a separate model checkpoint.
|
||||
|
||||
Alpha is earned only when the real target model:
|
||||
|
||||
1. cannot fit within any one participating node's admitted memory;
|
||||
2. loads as contiguous layer Shards on at least two physical consumer machines;
|
||||
3. performs real GLM-5.2 MoE + DSA + IndexShare computation on every selected node;
|
||||
4. responds through the existing OpenAI-compatible Meshnet API with `reasoning_effort=max`;
|
||||
5. passes locked parity, usefulness, performance, telemetry, cancellation, and cleanup checks; and
|
||||
6. stores all model artifacts on configured mounted-drive storage, never under `/home`.
|
||||
|
||||
The shortest safe path is not “support every GGUF architecture.” Dense Llama remains a small structural fixture. GLM-5.2 moves onto the critical path immediately after exact recipe identity and the pinned llama.cpp boundary. Qwen expansion, 1M context certification, MTP/speculative decoding, broad concurrency, and automatic route repair are post-alpha.
|
||||
|
||||
## 1. Exact target contract
|
||||
|
||||
### 1.1 Source model
|
||||
|
||||
| Field | Locked target |
|
||||
|---|---|
|
||||
| Official repository | `zai-org/GLM-5.2` |
|
||||
| Official revision observed 2026-07-13 | `b4734de4facf877f85769a911abafc5283eab3d9` |
|
||||
| License | MIT |
|
||||
| Architecture | `glm_moe_dsa` / `GlmMoeDsaForCausalLM` |
|
||||
| Total parameters | 744B |
|
||||
| Active parameters | approximately 40B/token |
|
||||
| Transformer layers | 78 main layers plus one shared NextN/MTP layer in the artifact |
|
||||
| Routed experts | 256 |
|
||||
| Experts selected | 8 routed experts plus shared expert path |
|
||||
| Hidden width | 6,144 |
|
||||
| Attention | MLA under DSA, lightning indexer top-k 2,048 |
|
||||
| IndexShare | indexer roles are encoded by `indexer_types`; consumers reuse prior Full-layer indices |
|
||||
| Architectural maximum context | 1,048,576 tokens |
|
||||
| Alpha reasoning mode | `reasoning_effort=max` |
|
||||
|
||||
The runtime must derive these values from the pinned artifact and fail closed on contradictory metadata. Marketing names are not compatibility identity.
|
||||
|
||||
### 1.2 Alpha GGUF artifact
|
||||
|
||||
| Field | Locked target |
|
||||
|---|---|
|
||||
| GGUF repository | `unsloth/GLM-5.2-GGUF` |
|
||||
| GGUF revision observed 2026-07-13 | `abc55e72527792c6e77069c99b4cb7de16fa9f23` |
|
||||
| Quantization | `UD-IQ1_S` |
|
||||
| Files | six GGUF shards |
|
||||
| Exact published bytes | 216,715,360,960 bytes |
|
||||
| Binary GiB | 201.832 GiB |
|
||||
| Published quality indication | about 76.2% top-1 agreement with the high-precision reference on Unsloth's quantization analysis |
|
||||
| Mounted storage rule | configured mounted drive only; never `/home` |
|
||||
|
||||
Before downloading 216.7 GB, DGR-017 must generate a checked-in target manifest containing repository revisions, expected filenames, byte sizes, and resolved LFS SHA-256 values. Download is resumable and verified before route admission.
|
||||
|
||||
`UD-IQ1_M` (228,492,966,624 bytes / 212.801 GiB) is the first diagnostic fallback if `UD-IQ1_S` exposes a runtime or quality defect. It does not satisfy the explicit “lowest quantization” alpha target unless the target contract is changed by human review.
|
||||
|
||||
### 1.3 Runtime semantics required for alpha
|
||||
|
||||
Required:
|
||||
|
||||
- GGUF parsing and quantized kernels from one exact llama.cpp pin.
|
||||
- Correct GLM-5.2 MoE routing, selected experts, and shared expert.
|
||||
- Correct compressed MLA KV cache for locally owned layers.
|
||||
- Native DSA lightning indexer and sparse attention.
|
||||
- Correct IndexShare Full/Shared role execution from artifact metadata.
|
||||
- Range-owned contiguous transformer layers; each owned layer keeps all of its experts local.
|
||||
- Head-only embeddings and tail-only final norm/output head/sampling.
|
||||
- Architecture-defined activation boundary and optional DSA index sideband.
|
||||
- `reasoning_effort=max` chat-template behavior through the public API.
|
||||
- F32 seam correctness lane and a separately certified production activation dtype.
|
||||
|
||||
Not required for alpha:
|
||||
|
||||
- MTP/speculative decoding. The trailing NextN tensors may be loaded or explicitly excluded according to a certified recipe, but cannot be silently misinterpreted.
|
||||
- Full 1,048,576-token context.
|
||||
- Continuous batching beyond one target session.
|
||||
- Public-WAN tensor or expert parallel collectives.
|
||||
- Automatic mid-generation repair or KV migration.
|
||||
- Every CPU/GPU backend combination.
|
||||
|
||||
## 2. Minimum resource envelope
|
||||
|
||||
### 2.1 Weight and runtime memory
|
||||
|
||||
The smallest artifact occupies 201.832 GiB before KV, scratch buffers, backend workspaces, process memory, and the operating system. The alpha planner uses **224 GiB aggregate usable model memory** as the initial conservative admission floor and records actual peak memory. “Usable” means memory available after per-node OS/runtime reserve, not installed capacity.
|
||||
|
||||
Unified memory is counted once. An integrated GPU's reported VRAM must not be added again to the same physical system RAM.
|
||||
|
||||
| Consumer-node class | Reserve per node | Usable per node | Minimum nodes for 224 GiB | Position |
|
||||
|---:|---:|---:|---:|---|
|
||||
| 32 GiB | 6 GiB | 26 GiB | 9 | technically possible; operationally poor |
|
||||
| 48 GiB | 8 GiB | 40 GiB | 6 | possible |
|
||||
| 64 GiB | 8 GiB | 56 GiB | 4 | minimum practical route |
|
||||
| 96 GiB | 12 GiB | 84 GiB | 3 | preferred |
|
||||
| 128 GiB unified/system | 16 GiB | 112 GiB | 2 | minimum high-memory route; narrow headroom |
|
||||
|
||||
The planner must use exact tensor byte ownership, not equal percentages. Embeddings, final head, shared experts, indexer tensors, quant block alignment, and backend workspace can make equal layer counts unequal in memory.
|
||||
|
||||
Recommended first target route: **three 96/128-GiB-class physical machines** or **four 64-GiB-class machines**, each with at least 1 GbE and mounted model storage. Two 128-GiB machines are a valuable minimum-fit trial but not the only certification topology.
|
||||
|
||||
### 2.2 KV cache
|
||||
|
||||
GLM-5.2 MLA stores approximately 576 latent/rope elements per token per main layer. Across 78 layers:
|
||||
|
||||
| Context | F16 KV, aggregate | Approx. Q4_1 KV, aggregate |
|
||||
|---:|---:|---:|
|
||||
| 16,384 | 1.37 GiB | 0.43 GiB |
|
||||
| 131,072 | 10.97 GiB | 3.43 GiB |
|
||||
| 1,048,576 | 87.75 GiB | 27.42 GiB |
|
||||
|
||||
These are planning estimates, not admission truth. The runtime must report measured allocated/resident KV by Shard and recipe. Alpha configures a 16,384-token window, uses one session, and records the exact KV dtype. Longer contexts are later certification gates.
|
||||
|
||||
### 2.3 Activation seams and network
|
||||
|
||||
A BF16 hidden-state boundary is 6,144 elements = 12,288 bytes/token before framing.
|
||||
|
||||
- A 16,384-token prefill sends about 192 MiB per seam.
|
||||
- One decode token sends about 12 KiB per seam.
|
||||
- A 512-token decode sends about 6 MiB per seam.
|
||||
- Four nodes imply three serial seams.
|
||||
|
||||
If a Shard boundary splits an IndexShare producer/consumer group, a 2,048-entry int32 top-k sideband can add up to 8 KiB/query before framing. The route planner should prefer boundaries that preserve complete IndexShare ownership groups. The protocol must still support and validate the named sideband because memory fit may force an internal group split.
|
||||
|
||||
Bandwidth is manageable on 1 GbE; sequential seam latency and compute dominate decode. Alpha records per-seam bytes, p50/p95 transfer latency, retries, and checksum failures. No speed claim is inferred from link speed alone.
|
||||
|
||||
### 2.4 Storage
|
||||
|
||||
The shortest alpha path allows every node to hold the complete six-file source GGUF while mapping/allocating only owned tensors. This minimizes artifact-transformation risk but costs 216.7 GB disk per node.
|
||||
|
||||
Deterministic source-bound layer packages are a follow-up optimization. If needed before target fit, every package must retain:
|
||||
|
||||
- source repository/revision and source file hashes;
|
||||
- exact owned tensor names, layer range, and endpoint role;
|
||||
- tokenizer/config identity;
|
||||
- deterministic package hash; and
|
||||
- proof that composing all packages matches the source tensor inventory.
|
||||
|
||||
## 3. Current state and critical gaps
|
||||
|
||||
### Completed foundation
|
||||
|
||||
- DGR-001: immutable CPU contract plus separate signed ROCm diagnostic. CPU v1 remains `stop`; the GPU diagnostic establishes a viable fit/performance investigation lane but does not rewrite CPU evidence.
|
||||
- DGR-002: versioned backend-neutral gRPC/Protobuf Shard protocol with bounded fragments and compatibility checks.
|
||||
- Existing Meshnet: Tracker, contiguous Shards, Route Sessions/epochs, relay/direct transport, local Hot KV semantics in the reference backend, cancellation, telemetry, billing, and model-agnostic admission.
|
||||
|
||||
### Missing before target alpha
|
||||
|
||||
1. Exact GLM target/artifact manifest and memory-fit planner.
|
||||
2. A current llama.cpp pin proven to load and generate with the exact `UD-IQ1_S` artifact.
|
||||
3. A narrow decision on native GLM-5.2 DSA/IndexShare support. As observed 2026-07-13, llama.cpp issue #24730 remains open and community GGUF execution may use incomplete or compatibility paths.
|
||||
4. Correct range-owned GGUF loading and memory proof.
|
||||
5. GLM-specific boundary/KV/IndexShare semantics.
|
||||
6. Standalone native worker and Meshnet integration.
|
||||
7. Real target hardware route with no node individually able to admit the whole model.
|
||||
8. Locked target parity, usefulness, speed, failure, and cleanup evidence.
|
||||
|
||||
### Donor policy
|
||||
|
||||
`Mesh-LLM/mesh-llm` is a high-value test and patch donor. Its live GLM branch was observed with 261 llama.cpp patches, 167 named for GLM/DSA/MTP-related work. That is evidence of the problem's depth, not an acceptable maintained fork boundary.
|
||||
|
||||
Audit and selectively reproduce the smallest independently understood pieces for:
|
||||
|
||||
- GLM DSA graph semantics;
|
||||
- lightning indexer and sparse-attention tests;
|
||||
- IndexShare metadata/Full/Shared validation;
|
||||
- top-k sideband shape and lifecycle;
|
||||
- stage-local KV filtering; and
|
||||
- target parity/performance fixtures.
|
||||
|
||||
Do not import Mesh-LLM routing, discovery, scheduler, public mesh, package manager, or full patch stack. Keep Meshnet as the sole control plane and collaborate narrowly upstream with llama.cpp/Mesh-LLM maintainers where practical.
|
||||
|
||||
## 4. Revised roadmap
|
||||
|
||||
## Phase 0 — lock the target before implementation
|
||||
|
||||
### DGR-017: lock GLM-5.2 Max target and alpha contract
|
||||
|
||||
Deliver:
|
||||
|
||||
- machine-readable target manifest for official and GGUF revisions;
|
||||
- exact `UD-IQ1_S` file/size/hash inventory;
|
||||
- architecture/config/chat-template snapshot;
|
||||
- memory/KV/network planner with unified-memory de-duplication;
|
||||
- immutable alpha acceptance thresholds from section 5; and
|
||||
- current upstream/donor status report.
|
||||
|
||||
Exit: target identity and alpha requirements are reviewable without downloading the model.
|
||||
|
||||
## Phase 1 — establish a correct whole-model oracle
|
||||
|
||||
### DGR-003: exact runtime recipe identity
|
||||
|
||||
Extend the existing generic identity with GLM fields: DSA/IndexShare metadata, adapter version, reasoning template revision, activation bundle schema, KV dtype/layout, llama.cpp pin/patch hash, and target artifact manifest hash.
|
||||
|
||||
### DGR-004: reproducible llama.cpp pin and narrow patch boundary
|
||||
|
||||
Select a current exact upstream commit only after testing its stock GLM behavior. Add clean fetch/apply/build checks. Record every donor patch and whether it is adopted, rewritten, rejected, or waiting upstream.
|
||||
|
||||
### DGR-018: certify whole-model GLM-5.2 runtime semantics
|
||||
|
||||
On a 224+ GiB usable reference host or temporary equivalent:
|
||||
|
||||
1. verify all six `UD-IQ1_S` shards;
|
||||
2. load with a stock pinned runtime and capture tensor/metadata warnings;
|
||||
3. prove whether DSA, IndexShare, shared expert, and Max template are actually active;
|
||||
4. add the minimum correctness patches/tests required;
|
||||
5. run deterministic prefill/decode and fixed Max-mode sentinel prompts; and
|
||||
6. sign the oracle recipe, output, telemetry, and limitations.
|
||||
|
||||
Exit: one whole-model oracle exists for the same artifact/runtime semantics the distributed path will implement. “It emits text” is insufficient.
|
||||
|
||||
## Phase 2 — build the generic local seam using small fixtures
|
||||
|
||||
### DGR-005: range-owned GGUF tensors
|
||||
|
||||
Keep dense Llama as a cheap structural fixture. Implement authoritative owned-tensor registration/loading, head/tail ownership, and measured resident-memory scaling. Design tensor classification so GLM adds explicit rules rather than unchecked name substitution.
|
||||
|
||||
### DGR-006: architecture-defined boundary
|
||||
|
||||
Implement named boundary bundles, F32 correctness lane, bounded fragmentation, and optional sidebands. Dense fixture parity proves the seam mechanism, not GLM certification.
|
||||
|
||||
Exit: two local processes can execute a small dense model with correct range ownership and boundary parity.
|
||||
|
||||
## Phase 3 — add GLM-5.2 as the product adapter
|
||||
|
||||
### DGR-019: implement and certify GLM-5.2 range/DSA/IndexShare semantics
|
||||
|
||||
Deliver explicit support for:
|
||||
|
||||
- 78 main layers and endpoint tensor ownership;
|
||||
- 256-expert MoE routing/top-8 and shared expert;
|
||||
- compressed MLA KV by owned layers;
|
||||
- DSA lightning indexer and sparse attention;
|
||||
- IndexShare metadata, Full producer, Shared consumer, and sideband behavior;
|
||||
- NextN/MTP tensor policy with MTP disabled or enabled explicitly;
|
||||
- shard-boundary planner aware of IndexShare ownership groups; and
|
||||
- whole-model versus two-stage parity against DGR-018.
|
||||
|
||||
Exit: a same-host two-stage target run matches the locked oracle tolerance with real GLM computation in both stages. If the full target cannot fit on one host for this check, use a layer-reduced GLM architecture fixture for graph parity and defer full-artifact output parity to DGR-020; label the distinction explicitly.
|
||||
|
||||
## Phase 4 — worker, KV, and Meshnet route
|
||||
|
||||
Execute existing stories with GLM requirements included:
|
||||
|
||||
1. DGR-007 — isolated local Hot KV keyed by `(Route Session, epoch)`, including DSA/IndexShare state.
|
||||
2. DGR-008 — standalone C++ gRPC worker.
|
||||
3. DGR-009 — Meshnet backend, capability, relay/direct, cancellation, and telemetry integration.
|
||||
4. DGR-010 — small-model local two-process acceptance.
|
||||
5. DGR-011 — real two-physical-machine route and heterogeneous fail-closed behavior.
|
||||
6. DGR-013 subset required by alpha — node loss, cancellation, stale epoch, restart, and memory/KV cleanup.
|
||||
|
||||
Continuous batching (DGR-012) is deliberately not an alpha dependency. The first target release supports one admitted GLM route session; concurrency follows after target correctness and fit.
|
||||
|
||||
## Phase 5 — target alpha gate
|
||||
|
||||
### DGR-020: pass real distributed GLM-5.2 Max alpha acceptance
|
||||
|
||||
Use at least two physical machines and enough aggregate usable memory to meet the locked target planner. No participating node may individually admit the complete target. All stages must report real compute and exact tensor ownership.
|
||||
|
||||
Run the complete acceptance matrix in section 5, preserve raw logs/metrics/output, sign the evidence, and publish an explicit `alpha` or `stop` verdict. Thresholds cannot be weakened after results are known.
|
||||
|
||||
## Phase 6 — post-alpha hardening
|
||||
|
||||
After DGR-020 passes:
|
||||
|
||||
1. DGR-012 — 1/2/4-session continuous batching and bounded admission.
|
||||
2. DGR-014 — final distributed GGUF versus reference-route performance decision.
|
||||
3. 32K, 128K, 200K, then 1M context certification with quantized KV.
|
||||
4. MTP/speculative decoding.
|
||||
5. Deterministic range packages to remove full-artifact replication.
|
||||
6. Additional backend compatibility classes and route topologies.
|
||||
7. DGR-016 — narrow upstream collaboration package, split by independently reviewable llama.cpp changes.
|
||||
8. DGR-015 — Qwen3/Qwen3-MoE only as later architecture expansion, not as the GLM alpha target.
|
||||
|
||||
## 5. Locked alpha acceptance matrix
|
||||
|
||||
These thresholds are set before target execution.
|
||||
|
||||
### 5.1 Identity and fit
|
||||
|
||||
- Exact official and GGUF repository revisions match the target manifest.
|
||||
- All six source GGUF sizes and LFS SHA-256 values verify.
|
||||
- Every route node reports owned tensor names/bytes, layer range, endpoint role, backend, KV recipe, and patch fingerprint.
|
||||
- Union of owned tensors equals the certified runtime-required tensor inventory; unintended overlap is zero.
|
||||
- No node's admitted usable memory can hold the complete recipe.
|
||||
- Aggregate peak RSS/VRAM stays within admitted budgets with no swap-driven success claim.
|
||||
- Unified RAM/VRAM is not double-counted.
|
||||
|
||||
### 5.2 Semantic correctness
|
||||
|
||||
- Logs and graph tests prove GLM MoE/shared-expert, DSA lightning indexer, sparse attention, and IndexShare Full/Shared paths are active; dense-attention compatibility fallback cannot satisfy alpha.
|
||||
- `reasoning_effort=max` is observable in the rendered template/API recipe.
|
||||
- F32 same-backend seam fixture: 32 greedy decode tokens exactly match the whole-model oracle and activation tolerance is locked by DGR-006.
|
||||
- Production seam on the fixed prompt corpus: greedy token agreement is at least 0.90 and mean compared-state/logit cosine similarity is at least 0.999 versus DGR-018, with no malformed or non-finite tensors.
|
||||
- Incompatible artifact, tokenizer, adapter, DSA metadata, boundary, activation, KV, backend class, or runtime patch fingerprints fail closed.
|
||||
|
||||
### 5.3 End-to-end target run
|
||||
|
||||
- Context configured to 16,384 tokens.
|
||||
- Fixed 4,096-token prompt lane completes prefill.
|
||||
- One Max-mode request generates at least 512 output tokens or reaches a valid natural EOS after at least 128 tokens.
|
||||
- Fixed coding, structured tool-call/JSON, and multi-step reasoning sentinels produce parseable, relevant outputs; raw prompts and outputs are retained for review.
|
||||
- OpenAI-compatible response includes stable model ID, finish reason, and token usage.
|
||||
|
||||
### 5.4 Minimum useful performance
|
||||
|
||||
On the declared minimum alpha topology after one warm-up:
|
||||
|
||||
- median decode throughput is at least 0.5 generated token/s for the fixed Max-mode lane;
|
||||
- 4,096-token-prompt TTFT is at most 10 minutes;
|
||||
- no unexplained stall exceeds 60 seconds without progress telemetry;
|
||||
- per-stage compute, queue, KV, seam bytes/latency, RSS/VRAM, and backend timing are present; and
|
||||
- results are labeled by hardware/topology and are not generalized to other consumer systems.
|
||||
|
||||
If output quality passes but the speed floor fails, verdict is `stop` for alpha and the evidence selects the next optimization target. It is not relabeled as success merely because the model loaded.
|
||||
|
||||
### 5.5 Reliability and security
|
||||
|
||||
- Two consecutive cold starts load, generate, release, and exit cleanly.
|
||||
- Cancellation during prefill and decode releases every stage's queued buffers and KV lease.
|
||||
- One worker loss aborts the route; alpha retries only from token zero on a new compatible route.
|
||||
- Stale epochs and duplicate step IDs are rejected.
|
||||
- Artifact paths stay outside `/home`; logs contain no secrets or unrestricted prompt payloads.
|
||||
- Synthetic workers and layer-reduced fixtures are labeled unit/integration coverage and cannot satisfy target alpha.
|
||||
|
||||
## 6. First execution order
|
||||
|
||||
The next unattended work should run in this order:
|
||||
|
||||
1. DGR-017 — target contract, manifest, planner, and upstream status.
|
||||
2. DGR-003 — exact recipe identity.
|
||||
3. DGR-004 — current llama.cpp pin and minimal patch harness.
|
||||
4. Run in parallel:
|
||||
- DGR-018 — whole-model oracle on 224+ GiB usable memory.
|
||||
- DGR-005 and DGR-006 — generic range/boundary seam on local small fixtures.
|
||||
5. DGR-019 — GLM semantics and parity after both parallel lanes pass.
|
||||
6. DGR-007 through DGR-011 — native worker and real transport route.
|
||||
7. Required DGR-013 failure subset.
|
||||
8. DGR-020 — real target alpha verdict.
|
||||
|
||||
The first external hardware blocker is DGR-018, but DGR-005/DGR-006 proceed locally while that host is sourced. Do not download the full model until DGR-017's exact manifest and storage preflight pass.
|
||||
|
||||
## 7. Sources checked on 2026-07-13
|
||||
|
||||
Authoritative or primary:
|
||||
|
||||
- Official model card and config: <https://huggingface.co/zai-org/GLM-5.2>
|
||||
- Official source revision API: <https://huggingface.co/api/models/zai-org/GLM-5.2>
|
||||
- Official GLM-5 technical report: <https://arxiv.org/abs/2602.15763>
|
||||
- Unsloth GGUF repository: <https://huggingface.co/unsloth/GLM-5.2-GGUF>
|
||||
- Unsloth local-run/quantization guide: <https://unsloth.ai/docs/models/glm-5.2>
|
||||
- llama.cpp GLM-5.2 support issue: <https://github.com/ggml-org/llama.cpp/issues/24730>
|
||||
- llama.cpp 1M-context discussion: <https://github.com/ggml-org/llama.cpp/discussions/24622>
|
||||
- IndexCache/IndexShare paper: <https://arxiv.org/abs/2603.12201>
|
||||
|
||||
Donor/current implementation evidence:
|
||||
|
||||
- Mesh-LLM repository: <https://github.com/Mesh-LLM/mesh-llm>
|
||||
- Mesh-LLM GLM branch noted by llama.cpp collaborator in issue #24730: `feat/jianyang-glm-52`
|
||||
|
||||
Web/repository observations are pinned by date and must be refreshed in DGR-017 before implementation because upstream support is moving quickly.
|
||||
@@ -6,6 +6,8 @@ Build one lean native GGUF execution path that lets an Inference Route combine c
|
||||
|
||||
The program is benchmark-gated. GGUF is not assumed faster merely because it is quantized or uses a different file format. The first story compares the current Transformers/safetensors backend against whole-model llama.cpp on controlled model/hardware/quality lanes and locks a performance contract. Native distributed work proceeds only when GGUF provides a meaningful speed or fit benefit.
|
||||
|
||||
The alpha target is now exact: `zai-org/GLM-5.2` at a pinned revision, the lowest published Unsloth `UD-IQ1_S` GGUF, and `reasoning_effort=max`, distributed across physical consumer machines. See [GLM-5.2-MAX-ALPHA-ROADMAP.md](GLM-5.2-MAX-ALPHA-ROADMAP.md). Dense Llama is a cheap structural fixture; Qwen is post-alpha expansion.
|
||||
|
||||
## Goals
|
||||
|
||||
- Execute one GGUF model across independently addressable contiguous Shards.
|
||||
@@ -15,6 +17,7 @@ The program is benchmark-gated. GGUF is not assumed faster merely because it is
|
||||
- Beat the current distributed safetensors route under a controlled performance contract or enable a larger otherwise-unroutable model at useful measured speed.
|
||||
- Keep the critical path to Meshnet plus a small pinned llama.cpp fork and standalone C++ worker.
|
||||
- Produce narrow upstream collaboration material for llama.cpp without placing Meshnet networking or economics inside upstream.
|
||||
- Pass an immutable GLM-5.2 Max `alpha`/`stop` gate with native MoE, DSA, IndexShare, parity, usefulness, speed, failure, and cleanup evidence.
|
||||
|
||||
## Quality Gates
|
||||
|
||||
@@ -213,6 +216,34 @@ Before a story is marked complete, run the full deterministic `pytest -q` suite
|
||||
- [ ] Preserve one scoped commit/patch per concern against the exact upstream pin.
|
||||
- [ ] Produce an outreach document suitable for Georgi/llama.cpp maintainers; actual sending remains a human action.
|
||||
|
||||
### DGR-017: Lock the GLM-5.2 Max target and alpha contract
|
||||
**Description:** Pin exact official/GGUF revisions, `UD-IQ1_S` files and hashes, Max-mode semantics, resource accounting, and immutable target thresholds before implementation results exist.
|
||||
|
||||
**Acceptance Criteria:**
|
||||
- [ ] Produce machine-readable target, resource, upstream-status, and immutable acceptance contracts without downloading full weights.
|
||||
- [ ] Count unified RAM/VRAM once and test revision/shard/byte/threshold mutation failures.
|
||||
|
||||
### DGR-018: Certify whole-model GLM-5.2 runtime semantics
|
||||
**Description:** Establish a 224+ GiB usable-memory oracle for the exact target artifact before distributed parity work.
|
||||
|
||||
**Acceptance Criteria:**
|
||||
- [ ] Verify the complete artifact and prove native MoE/shared expert, DSA, IndexShare, KV, NextN policy, and Max-template behavior.
|
||||
- [ ] Dense/replicated compatibility fallback cannot become the oracle merely because it emits text.
|
||||
|
||||
### DGR-019: Implement and certify GLM-5.2 range, DSA, and IndexShare semantics
|
||||
**Description:** Add explicit target-model tensor, graph, boundary, sideband, and local-KV ownership after the generic dense seam.
|
||||
|
||||
**Acceptance Criteria:**
|
||||
- [ ] Preserve MoE/shared expert, DSA lightning indexer/sparse attention, and IndexShare Full/Shared semantics across contiguous Shards.
|
||||
- [ ] Pass locked fixture/target parity and measured per-Shard memory ownership; never claim full-target parity from a reduced fixture.
|
||||
|
||||
### DGR-020: Pass real distributed GLM-5.2 Max alpha acceptance
|
||||
**Description:** Run the exact lowest-quant target through Meshnet on enough physical consumer nodes that no node can admit the whole recipe.
|
||||
|
||||
**Acceptance Criteria:**
|
||||
- [ ] Pass the immutable identity, semantic, parity, Max-mode usefulness, 0.5 token/s, TTFT, reliability, and mounted-storage gates.
|
||||
- [ ] Preserve signed raw evidence and emit `alpha` only if every target criterion passes; otherwise emit `stop`.
|
||||
|
||||
## Functional Requirements
|
||||
|
||||
1. The public distributed primitive is an ordered Inference Route of contiguous Shards.
|
||||
@@ -234,7 +265,7 @@ Before a story is marked complete, run the full deterministic `pytest -q` suite
|
||||
- QUIC, WebRTC, or a custom socket protocol.
|
||||
- Automatic KV migration or mid-generation route repair in the first release.
|
||||
- Speculative decoding or disaggregated prefill before the core release gate.
|
||||
- Supporting every GGUF architecture before dense Llama and Qwen3-family certification.
|
||||
- Supporting every GGUF architecture before the exact GLM-5.2 target; Qwen3-family certification is post-alpha.
|
||||
- A marketing-scale model demo that bypasses parity, concurrency, admission, or performance gates.
|
||||
|
||||
## Success Metrics
|
||||
@@ -247,6 +278,6 @@ Before a story is marked complete, run the full deterministic `pytest -q` suite
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Exact benchmark model and quantization lanes are selected by DGR-001 from currently supported, legally redistributable artifacts.
|
||||
- DGR-001 remains immutable. DGR-017 locks the exact GLM-5.2/`UD-IQ1_S` target and alpha thresholds without rewriting DGR-001 evidence.
|
||||
- Final hardware-specific concurrency and useful-speed thresholds are locked by measured baselines rather than guessed globally.
|
||||
- Upstream llama.cpp acceptance is desirable but not a prerequisite for the first narrow pinned fork.
|
||||
|
||||
@@ -9,7 +9,7 @@ Before changing code, every Ralph agent must:
|
||||
|
||||
1. Read this file completely.
|
||||
2. Read the selected issue under `.scratch/distributed-gguf-runtime/issues/`.
|
||||
3. Read `.scratch/distributed-gguf-runtime/ADR-0020-distributed-gguf-runtime.md` and the relevant part of `architecture.md`.
|
||||
3. Read `.scratch/distributed-gguf-runtime/GLM-5.2-MAX-ALPHA-ROADMAP.md`, `.scratch/distributed-gguf-runtime/ADR-0020-distributed-gguf-runtime.md`, and the relevant part of `architecture.md`.
|
||||
4. Read `.claude/memory/MEMORY.md` and root `CONTEXT.md` for current project vocabulary and constraints.
|
||||
5. Inspect the current implementation and tests; do not assume historical scratch text describes live code.
|
||||
6. Read the evidence/handoff directories for every declared dependency.
|
||||
@@ -35,6 +35,8 @@ If interrupted after code changes, record every changed file, command result and
|
||||
|
||||
Build performant, concurrent distributed inference that combines consumer machines to serve top open models that exceed one node's RAM/VRAM.
|
||||
|
||||
The alpha target is the exact pinned GLM-5.2 `UD-IQ1_S` artifact served with `reasoning_effort=max` across physical consumer machines. Dense Llama is a structural fixture. Synthetic workers, dense-attention compatibility fallback, a smaller model, or a single host cannot satisfy target alpha. The immutable target contract and resource envelope are in `GLM-5.2-MAX-ALPHA-ROADMAP.md`.
|
||||
|
||||
A distributed demo is not success. The product must provide:
|
||||
|
||||
- Useful measured prefill and decode speed.
|
||||
@@ -84,6 +86,8 @@ Do not introduce another scheduler/control plane from vLLM, Nakshatra, prima.cpp
|
||||
13. Unsupported architectures/backends remain registered-but-dark until real certification passes.
|
||||
14. Alpha failure retries from token zero; unverified KV is never migrated silently.
|
||||
15. Model artifacts must remain on mounted-drive storage and never under `/home`.
|
||||
16. Unified system RAM and integrated-GPU memory are one physical pool and must never be double-counted for admission.
|
||||
17. Alpha requires native GLM-5.2 MoE, DSA, and IndexShare semantics; MTP/speculative decoding and 1M-context certification are post-alpha.
|
||||
|
||||
Changing one of these requires an explicit ADR update and human review, not an incidental story implementation.
|
||||
|
||||
@@ -154,7 +158,7 @@ The project patch stack is limited to:
|
||||
|
||||
Do not place Meshnet routing, transport, billing or authentication inside llama.cpp. Keep patches numbered, scoped, pinned and upstreamable.
|
||||
|
||||
Dense Llama-family is first. Qwen3/Qwen3-MoE is a separate adapter after the dense release gate. Do not generalize through unchecked tensor-name substitutions.
|
||||
Dense Llama is first only as a cheap range/boundary fixture. GLM-5.2 is the explicit product adapter and alpha target immediately afterward. Qwen3/Qwen3-MoE is post-alpha. Do not generalize through unchecked tensor-name substitutions.
|
||||
|
||||
## Existing code seams to inspect first
|
||||
|
||||
@@ -222,6 +226,10 @@ Generated protobuf/C++ build outputs belong in build directories unless packagin
|
||||
| DGR-014 | immutable final comparison against DGR-001 thresholds and ship/stop recommendation; `evidence/DGR-014/` |
|
||||
| DGR-015 | Qwen3-family adapter, architecture-specific parity/admission/performance evidence; `evidence/DGR-015/` |
|
||||
| DGR-016 | narrow upstream patches/tests, design note and human-ready outreach package; `evidence/DGR-016/` |
|
||||
| DGR-017 | exact GLM-5.2/GGUF target manifest, resource planner, immutable alpha contract and upstream status; `evidence/DGR-017/` |
|
||||
| DGR-018 | verified whole-model `UD-IQ1_S` oracle with native GLM semantic evidence; `evidence/DGR-018/` |
|
||||
| DGR-019 | explicit range-owned GLM MoE/MLA/DSA/IndexShare adapter, fixtures and parity; `evidence/DGR-019/` |
|
||||
| DGR-020 | real multi-node GLM-5.2 Max target evidence and immutable `alpha`/`stop` verdict; `evidence/DGR-020/` |
|
||||
|
||||
## Dependency handoff rule
|
||||
|
||||
|
||||
@@ -4,7 +4,9 @@ Status: active benchmark-gated implementation program.
|
||||
|
||||
## Objective
|
||||
|
||||
Serve top open models across consumer machines with useful performance and concurrent Route Sessions while keeping the runtime lean.
|
||||
Serve the exact pinned GLM-5.2 `UD-IQ1_S` artifact in `reasoning_effort=max` mode across consumer machines with useful measured performance. Dense Llama is a structural fixture; the real multi-node GLM target is the alpha release gate.
|
||||
|
||||
See **[GLM-5.2 Max distributed alpha roadmap](GLM-5.2-MAX-ALPHA-ROADMAP.md)** for the target identity, minimum hardware, immutable acceptance matrix, and revised execution order.
|
||||
|
||||
## Critical path
|
||||
|
||||
|
||||
@@ -7,6 +7,8 @@ Last updated: 2026-07-13
|
||||
|
||||
The system exists to serve high-quality models that exceed one consumer node's memory while retaining useful interactive speed and aggregate concurrency. A feature that only produces a distributed demo but is slower, globally serialized, or impossible to operate on consumer hardware is not complete.
|
||||
|
||||
The alpha target is the exact pinned GLM-5.2 `UD-IQ1_S` artifact in `reasoning_effort=max` mode. Its target-specific architecture/resource/acceptance contract is [GLM-5.2-MAX-ALPHA-ROADMAP.md](GLM-5.2-MAX-ALPHA-ROADMAP.md). Dense Llama is a structural fixture, not the product target.
|
||||
|
||||
## Existing control plane
|
||||
|
||||
Meshnet remains the only public control plane:
|
||||
@@ -113,6 +115,8 @@ compression/checksum
|
||||
|
||||
Dense Llama may use one residual tensor. Other adapters may require more. vLLM's Llama and Qwen3-MoE PP paths demonstrate a boundary with both `hidden_states` and `residual`; therefore the generic protocol must not assume one anonymous tensor.
|
||||
|
||||
GLM-5.2 normally exchanges a 6,144-element hidden state. If a memory-balanced Shard boundary splits an IndexShare Full producer from Shared consumers, the bundle also carries the typed top-k index sideband. The planner prefers boundaries that keep an IndexShare ownership group local, but the protocol validates the sideband rather than assuming it never crosses a seam.
|
||||
|
||||
Only the head owns token embedding. Only the tail owns final normalization, LM head and sampling. Middle Shards exchange the architecture-defined pre-tail boundary, not final normalized embeddings.
|
||||
|
||||
## Hot KV State and concurrency
|
||||
@@ -243,17 +247,16 @@ The GGUF path ships only if it is faster at acceptable quality or enables a larg
|
||||
|
||||
## Implementation sequence
|
||||
|
||||
1. Lock benchmark/performance contract.
|
||||
2. Define gRPC/protobuf and exact recipe identity.
|
||||
3. Pin llama.cpp and create the minimal patch stack.
|
||||
4. Implement dense-Llama range loading and boundary parity.
|
||||
5. Implement concurrent local KV.
|
||||
6. Build and integrate the standalone worker.
|
||||
7. Pass local two-process real-model acceptance.
|
||||
8. Pass real heterogeneous two-machine acceptance.
|
||||
9. Add continuous batching and failure hardening.
|
||||
10. Enforce the GGUF-versus-safetensors release gate.
|
||||
11. Add Qwen3/Qwen3-MoE as a separately certified adapter.
|
||||
12. Prepare narrow upstream collaboration patches/tests.
|
||||
1. Preserve completed DGR-001 performance and DGR-002 protocol contracts.
|
||||
2. DGR-017 locks exact GLM-5.2 Max artifact, resource, and alpha acceptance identity.
|
||||
3. Define exact recipe identity and pin one reproducible llama.cpp boundary.
|
||||
4. Run two lanes in parallel: DGR-018 establishes the whole-model `UD-IQ1_S` oracle on 224+ GiB usable memory, while DGR-005/DGR-006 implement range loading and named boundary parity with a cheap dense fixture.
|
||||
5. DGR-019 adds explicit GLM-5.2 MoE/MLA/DSA/IndexShare semantics after both lanes pass.
|
||||
6. Implement local KV; build and integrate the standalone worker.
|
||||
7. Pass local two-process and real two-physical-machine execution.
|
||||
8. Harden cancellation, node loss, restart, and cleanup required by alpha.
|
||||
9. DGR-020 executes the exact multi-node target and emits immutable `alpha` or `stop`.
|
||||
10. Post-alpha: continuous batching, final comparison, longer context, MTP, and package optimization.
|
||||
11. Prepare narrow upstream patches/tests; add Qwen as later architecture expansion.
|
||||
|
||||
See [the Ralph backlog](prd.json) and [implementation strategy](implementation-strategy.md).
|
||||
|
||||
@@ -7,6 +7,8 @@ Last updated: 2026-07-13
|
||||
|
||||
Enable clients to run top open models that do not fit on one consumer machine by combining independently owned model Shards into performant, concurrent Inference Routes.
|
||||
|
||||
The alpha-release target is the exact `zai-org/GLM-5.2` model, pinned by revision and served with `reasoning_effort=max`, using the smallest published Unsloth `UD-IQ1_S` GGUF across physical consumer machines. See [GLM-5.2-MAX-ALPHA-ROADMAP.md](GLM-5.2-MAX-ALPHA-ROADMAP.md). Dense Llama remains a cheap structural fixture; Qwen expansion is post-alpha.
|
||||
|
||||
The project is not trying to reproduce every vLLM feature or support every inference engine. It is optimizing for:
|
||||
|
||||
1. Models larger than one node's RAM/VRAM.
|
||||
@@ -36,7 +38,7 @@ GGUF itself is a format. Performance comes from llama.cpp/GGML's quantized kerne
|
||||
|
||||
Quantized GGUF may be faster or may merely fit a larger model. Comparisons against safetensors must report both speed and quality because BF16 safetensors and Q4/Q8 GGUF are not numerically equivalent.
|
||||
|
||||
Before expensive native work, establish controlled lanes:
|
||||
Before expensive native work, establish controlled lanes. DGR-001 remains immutable; DGR-017 adds a target-specific fit and semantics contract without rewriting DGR-001 evidence:
|
||||
|
||||
- Same model architecture and upstream revision.
|
||||
- Same machine, prompt set, context, output length, sampling policy, and concurrency.
|
||||
@@ -134,6 +136,10 @@ Use as source and test donors only:
|
||||
|
||||
Do not adopt or fork their repositories wholesale.
|
||||
|
||||
### Mesh-LLM GLM branch: focused test/patch donor only
|
||||
|
||||
Use its GLM-5.2 branch to study DSA, IndexShare, stage-local KV, and sideband tests. Do not import its scheduler, discovery/control plane, package manager, or broad llama.cpp patch stack. Every adopted idea must be independently understood, minimized, attributed, and tested against our exact pin.
|
||||
|
||||
## Battle-proven transport decision
|
||||
|
||||
Use gRPC over HTTP/2 with Protocol Buffers for the native C++ Shard worker protocol.
|
||||
@@ -219,9 +225,9 @@ Two physical machines execute one model that uses both Shards. Synthetic activat
|
||||
|
||||
On certified consumer hardware, the GGUF route beats the current distributed safetensors route under the locked performance contract or enables a larger otherwise-unroutable model at useful measured speed.
|
||||
|
||||
### Gate F: architecture expansion
|
||||
### Gate F: exact GLM-5.2 alpha target
|
||||
|
||||
Only after dense Llama-family gates pass, add an explicit Qwen3/Qwen3-MoE adapter and certify it independently.
|
||||
After the generic dense fixture proves range and boundary mechanics, certify explicit GLM-5.2 MoE, MLA KV, DSA, IndexShare, and NextN policy. Alpha requires the exact `UD-IQ1_S` target across physical consumer nodes, native Max-mode semantics, locked parity/usefulness/performance thresholds, and bounded failure cleanup. Qwen3/Qwen3-MoE is later architecture expansion.
|
||||
|
||||
## Scope discipline
|
||||
|
||||
|
||||
@@ -39,7 +39,7 @@ As the Tracker, I need exact compatibility identity so that only numerically and
|
||||
|
||||
## Dependency handoff
|
||||
|
||||
- `DGR-002` must have `passes: true`; read `../evidence/DGR-002/README.md` and verify its referenced files/commands.
|
||||
- `DGR-002` and `DGR-017` must have `passes: true`; read both evidence READMEs and verify their referenced files/commands.
|
||||
|
||||
## Finish contract
|
||||
|
||||
|
||||
@@ -43,7 +43,7 @@ As a maintainer, I need a small auditable fork boundary so that upstream updates
|
||||
|
||||
## Dependency handoff
|
||||
|
||||
- `DGR-001` must have `passes: true`; read `../evidence/DGR-001/README.md` and verify its referenced files/commands.
|
||||
- `DGR-001` and `DGR-017` must have `passes: true`; read both evidence READMEs and verify their referenced files/commands.
|
||||
|
||||
## Finish contract
|
||||
|
||||
|
||||
@@ -42,7 +42,7 @@ As a client, I need concurrent Route Sessions to retain independent per-Shard ca
|
||||
|
||||
## Dependency handoff
|
||||
|
||||
- `DGR-006` must have `passes: true`; read `../evidence/DGR-006/README.md` and verify its referenced files/commands.
|
||||
- `DGR-006` and `DGR-019` must have `passes: true`; read both evidence READMEs and verify their referenced files/commands.
|
||||
|
||||
## Finish contract
|
||||
|
||||
|
||||
@@ -44,7 +44,7 @@ As a client, I need failures to be bounded and explicit so that distributed spee
|
||||
|
||||
- `DGR-008` must have `passes: true`; read `../evidence/DGR-008/README.md` and verify its referenced files/commands.
|
||||
- `DGR-009` must have `passes: true`; read `../evidence/DGR-009/README.md` and verify its referenced files/commands.
|
||||
- `DGR-012` must have `passes: true`; read `../evidence/DGR-012/README.md` and verify its referenced files/commands.
|
||||
- DGR-012 continuous batching is post-alpha; this story must harden single-session failure/cancellation first and later extend the same invariants to batched sessions.
|
||||
|
||||
## Finish contract
|
||||
|
||||
|
||||
@@ -0,0 +1,60 @@
|
||||
# 17 — Lock the GLM-5.2 Max target and alpha contract
|
||||
|
||||
Status: ready-for-agent
|
||||
|
||||
## Mandatory fresh-session context
|
||||
|
||||
- Read [RALPH-CONTEXT.md](../RALPH-CONTEXT.md), [GLM-5.2-MAX-ALPHA-ROADMAP.md](../GLM-5.2-MAX-ALPHA-ROADMAP.md), and this issue completely before changing code.
|
||||
- This issue is `DGR-017` in [prd.json](../prd.json).
|
||||
- Read the evidence README for every dependency listed below.
|
||||
- Inspect current code, upstream sources, and `git status`; historical text and previous agent claims are not evidence.
|
||||
|
||||
## Description
|
||||
|
||||
As a release owner, I need one exact GLM-5.2 Max target and immutable alpha contract so that later agents cannot swap artifacts, runtime semantics, hardware accounting, or thresholds after seeing results.
|
||||
|
||||
## Expected durable outputs
|
||||
|
||||
- Machine-readable official/GGUF target manifest with revisions, filenames, sizes, and resolved hashes
|
||||
- Architecture/config/chat-template snapshot and source links
|
||||
- Deterministic memory/KV/network planner with unified-memory de-duplication
|
||||
- Current llama.cpp and donor-support status report
|
||||
- Immutable machine-readable alpha acceptance contract
|
||||
- `evidence/DGR-017/README.md`
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- [ ] Pin `zai-org/GLM-5.2` and `unsloth/GLM-5.2-GGUF` by exact observed repository revisions and identify `UD-IQ1_S` as the alpha quantization.
|
||||
- [ ] Resolve all six `UD-IQ1_S` filenames, exact byte sizes, LFS SHA-256 values, aggregate decimal GB/GiB, license, and source URLs without downloading the weight payloads.
|
||||
- [ ] Snapshot and hash architecture-critical config/tokenizer/chat-template metadata, including main/NextN layer counts, hidden width, experts/top-k, DSA top-k, IndexShare roles, context maximum, and `reasoning_effort=max` behavior.
|
||||
- [ ] Generate deterministic minimum-node calculations from exact artifact bytes, configurable per-node reserve, KV dtype/context, scratch reserve, and endpoint/tensor imbalance.
|
||||
- [ ] Count unified system RAM/integrated-GPU memory once and reject additive RAM+VRAM claims for one shared pool.
|
||||
- [ ] Lock the identity, semantic correctness, target-run, performance, reliability, and mounted-storage criteria from the roadmap in a machine-readable contract before full target execution.
|
||||
- [ ] Refresh and record current upstream llama.cpp GLM-5.2/DSA/IndexShare/MTP support and focused donor candidates; do not adopt a broad donor fork or scheduler.
|
||||
- [ ] Add tests that reject changed revisions, missing shards, inconsistent aggregate bytes, duplicate unified-memory accounting, and post-result threshold mutation.
|
||||
- [ ] Targeted pytest tests pass.
|
||||
- [ ] `python -m compileall packages tests` passes for Python changes.
|
||||
- [ ] `git diff --check` passes.
|
||||
- [ ] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free.
|
||||
- [ ] Full deterministic `pytest -q` passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction.
|
||||
- [ ] Read and verify every dependency evidence README before relying on dependency behavior.
|
||||
- [ ] Preserve all pre-existing working-tree changes and stage only files belonging to this story.
|
||||
- [ ] Write `evidence/DGR-017/README.md` with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff.
|
||||
- [ ] Update only this story issue to `Status: done` after every acceptance criterion and quality gate passes.
|
||||
|
||||
## Dependency handoff
|
||||
|
||||
- `DGR-001` and `DGR-002` must have `passes: true`; read their evidence READMEs and verify referenced files/commands.
|
||||
|
||||
## Finish contract
|
||||
|
||||
- Create the task evidence directory and durable handoff required above.
|
||||
- Preserve real failures and blockers; never fabricate source, benchmark, model, test, or hardware output.
|
||||
- Change this issue to `Status: done` only after all criteria pass.
|
||||
- Emit `<promise>COMPLETE</promise>` only after the evidence handoff exists.
|
||||
|
||||
## References
|
||||
|
||||
- [GLM-5.2 Max alpha roadmap](../GLM-5.2-MAX-ALPHA-ROADMAP.md)
|
||||
- [Ralph execution context](../RALPH-CONTEXT.md)
|
||||
- [Current architecture](../architecture.md)
|
||||
@@ -0,0 +1,57 @@
|
||||
# 18 — Certify whole-model GLM-5.2 runtime semantics
|
||||
|
||||
Status: ready-for-agent
|
||||
|
||||
## Mandatory fresh-session context
|
||||
|
||||
- Read [RALPH-CONTEXT.md](../RALPH-CONTEXT.md), [GLM-5.2-MAX-ALPHA-ROADMAP.md](../GLM-5.2-MAX-ALPHA-ROADMAP.md), and this issue completely before changing code.
|
||||
- This issue is `DGR-018` in [prd.json](../prd.json).
|
||||
- Read and verify every dependency evidence README.
|
||||
- Inspect current upstream behavior and `git status`; community claims that a model “works” are not correctness evidence.
|
||||
|
||||
## Description
|
||||
|
||||
As a runtime maintainer, I need a certified whole-model oracle for the exact lowest-quant GLM-5.2 artifact so that distributed parity is measured against correct MoE, DSA, IndexShare, cache, and Max-template semantics.
|
||||
|
||||
## Expected durable outputs
|
||||
|
||||
- Verified target artifact on mounted storage
|
||||
- Stock-pin baseline and warning/tensor inventory
|
||||
- Focused GLM runtime correctness tests and minimum patch series, if required
|
||||
- Signed whole-model oracle recipe, deterministic outputs, metrics, and limitations
|
||||
- `evidence/DGR-018/README.md`
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- [ ] Preflight at least 224 GiB usable memory and approximately 250 GB free mounted storage; abort before download if storage would resolve under `/home` or requirements are unmet.
|
||||
- [ ] Download/resume and verify all six DGR-017 `UD-IQ1_S` files against exact sizes and LFS SHA-256 values.
|
||||
- [ ] Load the target with the unmodified DGR-004 pin first and retain full logs, tensor warnings, peak memory, context/KV allocation, TTFT, and decode timing.
|
||||
- [ ] Prove from graph/runtime evidence whether 256-expert MoE/top-8/shared expert, DSA lightning indexer/sparse attention, IndexShare Full/Shared reuse, and `reasoning_effort=max` are active.
|
||||
- [ ] Dense-attention or replicated-indexer compatibility fallback is labeled incomplete and cannot become the oracle solely because it emits plausible text.
|
||||
- [ ] Audit focused upstream/Mesh-LLM donor tests and patches; adopt or rewrite only independently understood minimum changes and record provenance/rejection rationale.
|
||||
- [ ] Handle the trailing NextN/MTP tensors explicitly; MTP may remain disabled for alpha, but ignored tensors and layer-count behavior must be understood and tested.
|
||||
- [ ] Run deterministic prefill/greedy decode and fixed Max-mode coding, structured-output/tool-call, and reasoning sentinels; retain raw prompts, outputs, token IDs, and compared state/logit evidence.
|
||||
- [ ] Bind the oracle to exact artifact, tokenizer/template, adapter, backend class, compute/activation/KV dtypes, context/RoPE, llama.cpp commit, and patch-series hash.
|
||||
- [ ] Produce a signed `pass` or `stop` semantic-runtime verdict; no performance threshold is weakened after execution.
|
||||
- [ ] Targeted native/unit tests and clean pinned rebuild pass.
|
||||
- [ ] `git diff --check` passes.
|
||||
- [ ] Default tests remain model-download-free/GPU-free; full-target execution is opt-in and never runs in default CI.
|
||||
- [ ] Model artifacts and caches remain on configured mounted-drive storage and never under `/home`.
|
||||
- [ ] Preserve all pre-existing working-tree changes and stage only files belonging to this story.
|
||||
- [ ] Write `evidence/DGR-018/README.md` with exact hardware, storage, source revisions, files, commands, raw-result paths, real results, limitations, and handoff.
|
||||
- [ ] Update only this story issue to `Status: done` after every acceptance criterion passes.
|
||||
|
||||
## Dependency handoff
|
||||
|
||||
- `DGR-003`, `DGR-004`, and `DGR-017` must have `passes: true`; read and verify their evidence READMEs.
|
||||
|
||||
## Finish contract
|
||||
|
||||
- If 224+ GiB usable reference hardware is unavailable, write `evidence/DGR-018/BLOCKED.md` with exact preflight output; do not substitute a smaller model.
|
||||
- Preserve real failures and blockers; never fabricate output.
|
||||
- Emit `<promise>COMPLETE</promise>` only after the signed oracle evidence exists.
|
||||
|
||||
## References
|
||||
|
||||
- [GLM-5.2 Max alpha roadmap](../GLM-5.2-MAX-ALPHA-ROADMAP.md)
|
||||
- [llama.cpp GLM-5.2 issue](https://github.com/ggml-org/llama.cpp/issues/24730)
|
||||
@@ -0,0 +1,58 @@
|
||||
# 19 — Implement and certify GLM-5.2 range, DSA, and IndexShare semantics
|
||||
|
||||
Status: ready-for-agent
|
||||
|
||||
## Mandatory fresh-session context
|
||||
|
||||
- Read [RALPH-CONTEXT.md](../RALPH-CONTEXT.md), [GLM-5.2-MAX-ALPHA-ROADMAP.md](../GLM-5.2-MAX-ALPHA-ROADMAP.md), and this issue completely before changing code.
|
||||
- This issue is `DGR-019` in [prd.json](../prd.json).
|
||||
- Read and verify every dependency evidence README.
|
||||
- Inspect current source and `git status`; do not generalize through tensor-name substitution.
|
||||
|
||||
## Description
|
||||
|
||||
As a target-model operator, I need explicit range-owned GLM-5.2 semantics so that contiguous consumer Shards preserve the whole-model MoE, DSA, IndexShare, and local-KV computation.
|
||||
|
||||
## Expected durable outputs
|
||||
|
||||
- Explicit GLM-5.2 architecture adapter and tensor-ownership rules
|
||||
- IndexShare-aware Shard planner and named DSA sideband protocol mapping
|
||||
- GLM architecture fixture and target parity evidence
|
||||
- Memory/tensor/KV ownership report
|
||||
- `evidence/DGR-019/README.md`
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- [ ] Implement explicit ownership for 78 main layers, head embedding, tail norm/output head, routed/shared experts, DSA/indexer tensors, and the certified NextN/MTP policy.
|
||||
- [ ] Each contiguous layer owner keeps all 256 experts and the shared-expert path for its layers local; no public cross-machine expert collectives are introduced.
|
||||
- [ ] Define and fixture-test compressed MLA KV ownership for locally owned layers, keyed by Route Session/epoch, for DGR-007 to implement.
|
||||
- [ ] Implement native DSA lightning indexer/top-2,048 and sparse-attention graph behavior matching DGR-018.
|
||||
- [ ] Parse and validate artifact `indexer_types`; implement Full producer and Shared consumer behavior without fabricated/duplicated indexer tensors.
|
||||
- [ ] Prefer Shard boundaries that preserve IndexShare ownership groups; when memory fit forces a split, carry a typed, bounded, validated top-k sideband in the DGR-006 named bundle.
|
||||
- [ ] Reject missing, stale, wrong-width, wrong-position, or Shared-before-Full sideband/index state.
|
||||
- [ ] Demonstrate mapped/resident weights and allocated KV scale with exact owned tensors/layers rather than the full model.
|
||||
- [ ] Same-host two-stage F32 seam fixture produces 32 exact greedy tokens against the DGR-018 semantics; production seam meets the pre-locked token/similarity thresholds.
|
||||
- [ ] If full-target same-host execution cannot fit, use a layer-reduced GLM architecture fixture only for graph parity and defer full-artifact parity to DGR-020; label it non-target evidence.
|
||||
- [ ] Add deterministic fixture tests for MoE route/shared expert, DSA Full/Shared groups, internal group split, endpoint ownership, KV filtering, malformed metadata, and NextN policy.
|
||||
- [ ] Targeted pytest/CTest/native tests pass; pinned patch stack applies and rebuilds cleanly.
|
||||
- [ ] `python -m compileall packages tests` and `git diff --check` pass.
|
||||
- [ ] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free.
|
||||
- [ ] Full deterministic `pytest -q` passes, or exact pre-existing unrelated failures are recorded.
|
||||
- [ ] Preserve all pre-existing working-tree changes and stage only files belonging to this story.
|
||||
- [ ] Write `evidence/DGR-019/README.md` with files, commands, real results, raw evidence, limitations, and dependent-story handoff.
|
||||
- [ ] Update only this story issue to `Status: done` after every acceptance criterion passes.
|
||||
|
||||
## Dependency handoff
|
||||
|
||||
- `DGR-005`, `DGR-006`, and `DGR-018` must have `passes: true`; read and verify their evidence READMEs.
|
||||
- DGR-007 integration may be completed in parallel only after the adapter's cache contract is explicit.
|
||||
|
||||
## Finish contract
|
||||
|
||||
- Preserve real failures and blockers; never claim target parity from the architecture fixture.
|
||||
- Emit `<promise>COMPLETE</promise>` only after evidence exists.
|
||||
|
||||
## References
|
||||
|
||||
- [GLM-5.2 Max alpha roadmap](../GLM-5.2-MAX-ALPHA-ROADMAP.md)
|
||||
- [Current architecture](../architecture.md)
|
||||
@@ -0,0 +1,60 @@
|
||||
# 20 — Pass real distributed GLM-5.2 Max alpha acceptance
|
||||
|
||||
Status: ready-for-agent
|
||||
|
||||
## Mandatory fresh-session context
|
||||
|
||||
- Read [RALPH-CONTEXT.md](../RALPH-CONTEXT.md), [GLM-5.2-MAX-ALPHA-ROADMAP.md](../GLM-5.2-MAX-ALPHA-ROADMAP.md), and this issue completely before changing code.
|
||||
- This issue is `DGR-020` in [prd.json](../prd.json).
|
||||
- Read and verify every dependency evidence README and the immutable DGR-017 contract.
|
||||
- Inspect live source, hardware, network, storage, and `git status`; synthetic or layer-reduced evidence cannot satisfy this story.
|
||||
|
||||
## Description
|
||||
|
||||
As an alpha user, I need the exact lowest-quant GLM-5.2 model to run in Max reasoning mode across consumer machines through Meshnet so that the project has proven its end goal on real hardware.
|
||||
|
||||
## Expected durable outputs
|
||||
|
||||
- Exact multi-node hardware/network/storage/runtime manifest
|
||||
- Signed target identity, ownership, parity, quality, performance, and reliability reports
|
||||
- Raw worker/tracker/API logs, metrics, prompts, outputs, and cleanup evidence
|
||||
- Explicit immutable `alpha` or `stop` verdict
|
||||
- `evidence/DGR-020/README.md`
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- [ ] Use the exact DGR-017 `UD-IQ1_S` artifact and certified DGR-018/DGR-019 runtime recipe with all source and patch hashes verified.
|
||||
- [ ] Use at least two physical consumer machines with at least 224 GiB aggregate usable admitted memory; no participating node may individually admit the complete recipe and unified memory is counted once.
|
||||
- [ ] Tracker selects disjoint contiguous Shards whose exact required tensor inventory has complete union and no unintended overlap.
|
||||
- [ ] Every stage reports real CPU/GPU compute, owned tensor bytes/layers, local KV, backend, queue, and seam telemetry; synthetic, passthrough, or zero-layer workers fail acceptance.
|
||||
- [ ] Runtime evidence proves native GLM MoE/shared expert, DSA lightning indexer/sparse attention, and IndexShare Full/Shared paths are active; dense fallback fails acceptance.
|
||||
- [ ] OpenAI-compatible API applies and records `reasoning_effort=max`, stable model ID, finish reason, and token usage.
|
||||
- [ ] Configure 16,384 context; complete the fixed 4,096-token prefill lane and generate at least 512 output tokens or valid natural EOS after at least 128 tokens.
|
||||
- [ ] Production seam meets at least 0.90 greedy token agreement and 0.999 mean compared-state/logit cosine similarity against DGR-018 on the fixed corpus, with no non-finite or malformed tensors.
|
||||
- [ ] Fixed coding, structured tool-call/JSON, and multi-step reasoning sentinels produce parseable relevant outputs retained for human review.
|
||||
- [ ] After warm-up, fixed Max lane median decode is at least 0.5 token/s, 4,096-token-prompt TTFT is at most 10 minutes, and no unexplained stall exceeds 60 seconds without progress telemetry.
|
||||
- [ ] Record per-stage/seam p50/p95 latency, bytes, compute time, peak RSS/VRAM, KV pressure, network properties, errors, and total energy when available.
|
||||
- [ ] Two consecutive cold starts load, generate, release, and exit without leaked processes, mapped weights, queues, or KV leases.
|
||||
- [ ] Cancellation in prefill and decode releases all stages; one worker loss aborts the route; retry starts from token zero on a newly compatible route; stale epochs and duplicate steps fail closed.
|
||||
- [ ] Model artifacts/caches remain on configured mounted-drive storage and never under `/home`; secret scan passes.
|
||||
- [ ] Run targeted/full deterministic tests and clean native rebuild in addition to target execution; `git diff --check` passes.
|
||||
- [ ] Preserve all raw failures and do not weaken DGR-017 thresholds after results.
|
||||
- [ ] Write a signed `alpha` verdict only if every criterion passes; otherwise write `stop` with the measured bottleneck and next narrow action.
|
||||
- [ ] Write `evidence/DGR-020/README.md` with exact commands, manifests, raw-result paths, outputs, limitations, and post-alpha handoff.
|
||||
- [ ] Update only this story issue to `Status: done` after the immutable verdict and all evidence exist.
|
||||
|
||||
## Dependency handoff
|
||||
|
||||
- `DGR-007`, `DGR-008`, `DGR-009`, `DGR-011`, `DGR-013`, `DGR-017`, `DGR-018`, and `DGR-019` must have `passes: true`; read and verify their evidence READMEs.
|
||||
- DGR-012 continuous batching and DGR-014 final comparative release gate are post-alpha and do not block one target session.
|
||||
|
||||
## Finish contract
|
||||
|
||||
- If required physical nodes/storage are unavailable, write `evidence/DGR-020/BLOCKED.md`; do not substitute a smaller model, API, synthetic worker, or single host.
|
||||
- Preserve real failures and blockers; never fabricate target output or telemetry.
|
||||
- Emit `<promise>COMPLETE</promise>` only after the signed verdict exists.
|
||||
|
||||
## References
|
||||
|
||||
- [GLM-5.2 Max alpha roadmap](../GLM-5.2-MAX-ALPHA-ROADMAP.md)
|
||||
- [Ralph execution context](../RALPH-CONTEXT.md)
|
||||
@@ -1,35 +1,47 @@
|
||||
# Distributed GGUF runtime milestones
|
||||
|
||||
## Gate A — measured runtime value
|
||||
The exact alpha target and immutable acceptance gates are defined in [GLM-5.2-MAX-ALPHA-ROADMAP.md](GLM-5.2-MAX-ALPHA-ROADMAP.md).
|
||||
|
||||
- DGR-001 locks the safetensors-versus-GGUF performance/fit/quality contract.
|
||||
- DGR-002 can proceed independently and defines the battle-proven backend-neutral wire protocol.
|
||||
- DGR-003 builds exact recipe identity on DGR-002.
|
||||
- Expensive native llama.cpp work remains gated by DGR-001.
|
||||
## Completed foundation
|
||||
|
||||
- DGR-001 locks the safetensors-versus-GGUF performance/fit/quality contract. Its immutable CPU v1 verdict remains `stop`; separate ROCm evidence supports a fit-oriented investigation without rewriting CPU evidence.
|
||||
- DGR-002 defines the backend-neutral gRPC/Protobuf Shard protocol.
|
||||
|
||||
## Gate A — exact GLM-5.2 target and oracle
|
||||
|
||||
- DGR-017 locks official/GGUF revisions, `UD-IQ1_S`, hashes, resource accounting, Max-mode semantics, and alpha thresholds.
|
||||
- DGR-003 builds exact recipe identity on DGR-002 and DGR-017.
|
||||
- DGR-004 creates the reproducible pinned llama.cpp boundary after stock GLM behavior is measured.
|
||||
- DGR-018 certifies one whole-model `UD-IQ1_S` oracle with real MoE, DSA, IndexShare, KV, and `reasoning_effort=max` semantics.
|
||||
|
||||
## Gate B — minimal native execution seam
|
||||
|
||||
- DGR-004 creates the reproducible pinned fork boundary.
|
||||
- DGR-005 implements dense-Llama range ownership.
|
||||
- DGR-006 proves architecture-defined boundary parity.
|
||||
- DGR-005 implements range-owned tensors using dense Llama as a cheap structural fixture.
|
||||
- DGR-006 proves the generic named activation boundary and F32 correctness lane.
|
||||
- DGR-019 implements explicit GLM-5.2 MoE/MLA/DSA/IndexShare/NextN range semantics and parity.
|
||||
|
||||
## Gate C — concurrent production worker
|
||||
## Gate C — native Meshnet route
|
||||
|
||||
- DGR-007 isolates concurrent Hot KV State.
|
||||
- DGR-007 isolates Shard-local Hot KV State.
|
||||
- DGR-008 exposes the native worker over gRPC.
|
||||
- DGR-009 integrates the worker without replacing Meshnet's control plane.
|
||||
- DGR-010 passes local real-model two-process acceptance.
|
||||
- DGR-011 passes real two-physical-machine execution.
|
||||
- DGR-013 supplies the cancellation/node-loss/restart/cleanup subset required by alpha.
|
||||
|
||||
## Gate D — real consumer-hardware route
|
||||
## Gate D — GLM-5.2 Max alpha verdict
|
||||
|
||||
- DGR-020 runs the exact `UD-IQ1_S` target across enough physical consumer nodes that no one node can admit the whole recipe.
|
||||
- It produces an immutable `alpha` or `stop` verdict from target identity, native GLM semantics, parity, Max-mode usefulness, minimum speed, telemetry, failure, and cleanup evidence.
|
||||
- Synthetic workers, layer-reduced fixtures, dense-attention fallbacks, and single-host execution cannot satisfy this gate.
|
||||
|
||||
## Gate E — post-alpha product hardening
|
||||
|
||||
- DGR-011 passes two-physical-machine execution.
|
||||
- DGR-012 adds continuous batching and bounded admission.
|
||||
- DGR-013 hardens failure and cancellation.
|
||||
|
||||
## Gate E — product release decision
|
||||
|
||||
- DGR-014 compares distributed GGUF against the current distributed safetensors route under locked thresholds.
|
||||
- DGR-015 adds Qwen3/Qwen3-MoE only after the dense runtime passes.
|
||||
- DGR-014 compares distributed GGUF against the current reference route under locked thresholds.
|
||||
- Longer contexts progress through 32K/128K/200K before 1M certification.
|
||||
- MTP/speculative decoding and source-bound layer packages remain measured optimizations.
|
||||
- DGR-016 prepares narrow upstream llama.cpp collaboration material.
|
||||
- DGR-015 adds Qwen3/Qwen3-MoE only as later architecture expansion.
|
||||
|
||||
No later gate may be claimed from synthetic workers or documentation-only evidence.
|
||||
No later gate may be claimed from synthetic workers or documentation-only evidence. Model artifacts remain on mounted-drive storage and never under `/home`.
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"name": "Performant Concurrent Distributed GGUF Runtime",
|
||||
"branchName": "ralph/performant-concurrent-distributed-gguf",
|
||||
"description": "Benchmark-gated native llama.cpp/GGUF Shards with gRPC streaming, concurrent local KV, continuous batching, real heterogeneous acceptance, and a measured release gate against Transformers/safetensors.",
|
||||
"description": "Build a performant concurrent distributed GGUF runtime on Meshnet. Alpha is gated by exact GLM-5.2 UD-IQ1_S execution in reasoning_effort=max mode across consumer machines; synthetic or dense-fallback evidence cannot satisfy the target.",
|
||||
"userStories": [
|
||||
{
|
||||
"id": "DGR-001",
|
||||
@@ -26,7 +26,7 @@
|
||||
"Write .scratch/distributed-gguf-runtime/evidence/DGR-001/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff",
|
||||
"Update only this story issue to Status: done after every acceptance criterion and quality gate passes"
|
||||
],
|
||||
"priority": 2,
|
||||
"priority": 1,
|
||||
"passes": true,
|
||||
"notes": "Source issue: .scratch/distributed-gguf-runtime/issues/01-lock-the-safetensors-versus-gguf-performance-contract.md",
|
||||
"dependsOn": []
|
||||
@@ -53,7 +53,7 @@
|
||||
"Write .scratch/distributed-gguf-runtime/evidence/DGR-002/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff",
|
||||
"Update only this story issue to Status: done after every acceptance criterion and quality gate passes"
|
||||
],
|
||||
"priority": 1,
|
||||
"priority": 2,
|
||||
"passes": true,
|
||||
"notes": "Source issue: .scratch/distributed-gguf-runtime/issues/02-adopt-the-versioned-grpc-shard-protocol.md",
|
||||
"dependsOn": []
|
||||
@@ -79,11 +79,12 @@
|
||||
"Write .scratch/distributed-gguf-runtime/evidence/DGR-003/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff",
|
||||
"Update only this story issue to Status: done after every acceptance criterion and quality gate passes"
|
||||
],
|
||||
"priority": 3,
|
||||
"priority": 4,
|
||||
"passes": false,
|
||||
"notes": "Source issue: .scratch/distributed-gguf-runtime/issues/03-define-exact-artifact-and-runtime-recipe-identity.md",
|
||||
"dependsOn": [
|
||||
"DGR-002"
|
||||
"DGR-002",
|
||||
"DGR-017"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -110,11 +111,12 @@
|
||||
"Write .scratch/distributed-gguf-runtime/evidence/DGR-004/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff",
|
||||
"Update only this story issue to Status: done after every acceptance criterion and quality gate passes"
|
||||
],
|
||||
"priority": 4,
|
||||
"priority": 5,
|
||||
"passes": false,
|
||||
"notes": "Source issue: .scratch/distributed-gguf-runtime/issues/04-create-the-reproducible-pinned-llama-cpp-patch-stack.md",
|
||||
"dependsOn": [
|
||||
"DGR-001"
|
||||
"DGR-001",
|
||||
"DGR-017"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -140,7 +142,7 @@
|
||||
"Write .scratch/distributed-gguf-runtime/evidence/DGR-005/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff",
|
||||
"Update only this story issue to Status: done after every acceptance criterion and quality gate passes"
|
||||
],
|
||||
"priority": 5,
|
||||
"priority": 6,
|
||||
"passes": false,
|
||||
"notes": "Source issue: .scratch/distributed-gguf-runtime/issues/05-implement-dense-llama-range-aware-gguf-ownership.md",
|
||||
"dependsOn": [
|
||||
@@ -172,7 +174,7 @@
|
||||
"Write .scratch/distributed-gguf-runtime/evidence/DGR-006/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff",
|
||||
"Update only this story issue to Status: done after every acceptance criterion and quality gate passes"
|
||||
],
|
||||
"priority": 6,
|
||||
"priority": 7,
|
||||
"passes": false,
|
||||
"notes": "Source issue: .scratch/distributed-gguf-runtime/issues/06-implement-architecture-defined-boundary-input-output.md",
|
||||
"dependsOn": [
|
||||
@@ -204,11 +206,12 @@
|
||||
"Write .scratch/distributed-gguf-runtime/evidence/DGR-007/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff",
|
||||
"Update only this story issue to Status: done after every acceptance criterion and quality gate passes"
|
||||
],
|
||||
"priority": 7,
|
||||
"priority": 10,
|
||||
"passes": false,
|
||||
"notes": "Source issue: .scratch/distributed-gguf-runtime/issues/07-add-isolated-concurrent-local-hot-kv-state.md",
|
||||
"dependsOn": [
|
||||
"DGR-006"
|
||||
"DGR-006",
|
||||
"DGR-019"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -235,7 +238,7 @@
|
||||
"Write .scratch/distributed-gguf-runtime/evidence/DGR-008/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff",
|
||||
"Update only this story issue to Status: done after every acceptance criterion and quality gate passes"
|
||||
],
|
||||
"priority": 8,
|
||||
"priority": 11,
|
||||
"passes": false,
|
||||
"notes": "Source issue: .scratch/distributed-gguf-runtime/issues/08-build-the-standalone-c-grpc-shard-worker.md",
|
||||
"dependsOn": [
|
||||
@@ -270,7 +273,7 @@
|
||||
"Write .scratch/distributed-gguf-runtime/evidence/DGR-009/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff",
|
||||
"Update only this story issue to Status: done after every acceptance criterion and quality gate passes"
|
||||
],
|
||||
"priority": 9,
|
||||
"priority": 12,
|
||||
"passes": false,
|
||||
"notes": "Source issue: .scratch/distributed-gguf-runtime/issues/09-integrate-the-native-worker-with-meshnet.md",
|
||||
"dependsOn": [
|
||||
@@ -304,7 +307,7 @@
|
||||
"Write .scratch/distributed-gguf-runtime/evidence/DGR-010/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff",
|
||||
"Update only this story issue to Status: done after every acceptance criterion and quality gate passes"
|
||||
],
|
||||
"priority": 10,
|
||||
"priority": 13,
|
||||
"passes": false,
|
||||
"notes": "Source issue: .scratch/distributed-gguf-runtime/issues/10-pass-local-real-model-two-process-acceptance.md",
|
||||
"dependsOn": [
|
||||
@@ -337,7 +340,7 @@
|
||||
"Write .scratch/distributed-gguf-runtime/evidence/DGR-011/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff",
|
||||
"Update only this story issue to Status: done after every acceptance criterion and quality gate passes"
|
||||
],
|
||||
"priority": 11,
|
||||
"priority": 14,
|
||||
"passes": false,
|
||||
"notes": "Source issue: .scratch/distributed-gguf-runtime/issues/11-pass-a-real-heterogeneous-two-machine-route.md",
|
||||
"dependsOn": [
|
||||
@@ -368,7 +371,7 @@
|
||||
"Write .scratch/distributed-gguf-runtime/evidence/DGR-012/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff",
|
||||
"Update only this story issue to Status: done after every acceptance criterion and quality gate passes"
|
||||
],
|
||||
"priority": 12,
|
||||
"priority": 17,
|
||||
"passes": false,
|
||||
"notes": "Source issue: .scratch/distributed-gguf-runtime/issues/12-implement-continuous-batching-and-bounded-admission.md",
|
||||
"dependsOn": [
|
||||
@@ -401,13 +404,12 @@
|
||||
"Write .scratch/distributed-gguf-runtime/evidence/DGR-013/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff",
|
||||
"Update only this story issue to Status: done after every acceptance criterion and quality gate passes"
|
||||
],
|
||||
"priority": 13,
|
||||
"priority": 15,
|
||||
"passes": false,
|
||||
"notes": "Source issue: .scratch/distributed-gguf-runtime/issues/13-harden-failure-cancellation-and-restart-semantics.md",
|
||||
"dependsOn": [
|
||||
"DGR-008",
|
||||
"DGR-009",
|
||||
"DGR-012"
|
||||
"DGR-009"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -435,7 +437,7 @@
|
||||
"Write .scratch/distributed-gguf-runtime/evidence/DGR-014/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff",
|
||||
"Update only this story issue to Status: done after every acceptance criterion and quality gate passes"
|
||||
],
|
||||
"priority": 14,
|
||||
"priority": 18,
|
||||
"passes": false,
|
||||
"notes": "Source issue: .scratch/distributed-gguf-runtime/issues/14-enforce-the-gguf-versus-safetensors-release-gate.md",
|
||||
"dependsOn": [
|
||||
@@ -470,7 +472,7 @@
|
||||
"Write .scratch/distributed-gguf-runtime/evidence/DGR-015/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff",
|
||||
"Update only this story issue to Status: done after every acceptance criterion and quality gate passes"
|
||||
],
|
||||
"priority": 15,
|
||||
"priority": 20,
|
||||
"passes": false,
|
||||
"notes": "Source issue: .scratch/distributed-gguf-runtime/issues/15-add-and-certify-a-qwen3-qwen3-moe-adapter.md",
|
||||
"dependsOn": [
|
||||
@@ -500,12 +502,147 @@
|
||||
"Write .scratch/distributed-gguf-runtime/evidence/DGR-016/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff",
|
||||
"Update only this story issue to Status: done after every acceptance criterion and quality gate passes"
|
||||
],
|
||||
"priority": 16,
|
||||
"priority": 19,
|
||||
"passes": false,
|
||||
"notes": "Source issue: .scratch/distributed-gguf-runtime/issues/16-produce-the-upstream-llama-cpp-collaboration-package.md",
|
||||
"dependsOn": [
|
||||
"DGR-010"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "DGR-017",
|
||||
"title": "Lock the GLM-5.2 Max target and alpha contract",
|
||||
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md`, `.scratch/distributed-gguf-runtime/GLM-5.2-MAX-ALPHA-ROADMAP.md`, and `.scratch/distributed-gguf-runtime/issues/17-lock-glm-5-2-max-target-and-alpha-contract.md` completely before coding. Read and verify every dependency evidence README.\n\nAs a release owner, I need one exact GLM-5.2 Max target and immutable alpha contract so that later agents cannot swap artifacts, runtime semantics, hardware accounting, or thresholds after seeing results.",
|
||||
"acceptanceCriteria": [
|
||||
"Pin `zai-org/GLM-5.2` and `unsloth/GLM-5.2-GGUF` by exact observed repository revisions and identify `UD-IQ1_S` as the alpha quantization.",
|
||||
"Resolve all six `UD-IQ1_S` filenames, exact byte sizes, LFS SHA-256 values, aggregate decimal GB/GiB, license, and source URLs without downloading the weight payloads.",
|
||||
"Snapshot and hash architecture-critical config/tokenizer/chat-template metadata, including main/NextN layer counts, hidden width, experts/top-k, DSA top-k, IndexShare roles, context maximum, and `reasoning_effort=max` behavior.",
|
||||
"Generate deterministic minimum-node calculations from exact artifact bytes, configurable per-node reserve, KV dtype/context, scratch reserve, and endpoint/tensor imbalance.",
|
||||
"Count unified system RAM/integrated-GPU memory once and reject additive RAM+VRAM claims for one shared pool.",
|
||||
"Lock the identity, semantic correctness, target-run, performance, reliability, and mounted-storage criteria from the roadmap in a machine-readable contract before full target execution.",
|
||||
"Refresh and record current upstream llama.cpp GLM-5.2/DSA/IndexShare/MTP support and focused donor candidates; do not adopt a broad donor fork or scheduler.",
|
||||
"Add tests that reject changed revisions, missing shards, inconsistent aggregate bytes, duplicate unified-memory accounting, and post-result threshold mutation.",
|
||||
"Targeted pytest tests pass.",
|
||||
"`python -m compileall packages tests` passes for Python changes.",
|
||||
"`git diff --check` passes.",
|
||||
"Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free.",
|
||||
"Full deterministic `pytest -q` passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction.",
|
||||
"Read and verify every dependency evidence README before relying on dependency behavior.",
|
||||
"Preserve all pre-existing working-tree changes and stage only files belonging to this story.",
|
||||
"Write `evidence/DGR-017/README.md` with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff.",
|
||||
"Update only this story issue to `Status: done` after every acceptance criterion and quality gate passes."
|
||||
],
|
||||
"priority": 3,
|
||||
"passes": false,
|
||||
"notes": "Source issue: .scratch/distributed-gguf-runtime/issues/17-lock-glm-5-2-max-target-and-alpha-contract.md",
|
||||
"dependsOn": [
|
||||
"DGR-001",
|
||||
"DGR-002"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "DGR-018",
|
||||
"title": "Certify whole-model GLM-5.2 runtime semantics",
|
||||
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md`, `.scratch/distributed-gguf-runtime/GLM-5.2-MAX-ALPHA-ROADMAP.md`, and `.scratch/distributed-gguf-runtime/issues/18-certify-whole-model-glm-5-2-runtime-semantics.md` completely before coding. Read and verify every dependency evidence README.\n\nAs a runtime maintainer, I need a certified whole-model oracle for the exact lowest-quant GLM-5.2 artifact so that distributed parity is measured against correct MoE, DSA, IndexShare, cache, and Max-template semantics.",
|
||||
"acceptanceCriteria": [
|
||||
"Preflight at least 224 GiB usable memory and approximately 250 GB free mounted storage; abort before download if storage would resolve under `/home` or requirements are unmet.",
|
||||
"Download/resume and verify all six DGR-017 `UD-IQ1_S` files against exact sizes and LFS SHA-256 values.",
|
||||
"Load the target with the unmodified DGR-004 pin first and retain full logs, tensor warnings, peak memory, context/KV allocation, TTFT, and decode timing.",
|
||||
"Prove from graph/runtime evidence whether 256-expert MoE/top-8/shared expert, DSA lightning indexer/sparse attention, IndexShare Full/Shared reuse, and `reasoning_effort=max` are active.",
|
||||
"Dense-attention or replicated-indexer compatibility fallback is labeled incomplete and cannot become the oracle solely because it emits plausible text.",
|
||||
"Audit focused upstream/Mesh-LLM donor tests and patches; adopt or rewrite only independently understood minimum changes and record provenance/rejection rationale.",
|
||||
"Handle the trailing NextN/MTP tensors explicitly; MTP may remain disabled for alpha, but ignored tensors and layer-count behavior must be understood and tested.",
|
||||
"Run deterministic prefill/greedy decode and fixed Max-mode coding, structured-output/tool-call, and reasoning sentinels; retain raw prompts, outputs, token IDs, and compared state/logit evidence.",
|
||||
"Bind the oracle to exact artifact, tokenizer/template, adapter, backend class, compute/activation/KV dtypes, context/RoPE, llama.cpp commit, and patch-series hash.",
|
||||
"Produce a signed `pass` or `stop` semantic-runtime verdict; no performance threshold is weakened after execution.",
|
||||
"Targeted native/unit tests and clean pinned rebuild pass.",
|
||||
"`git diff --check` passes.",
|
||||
"Default tests remain model-download-free/GPU-free; full-target execution is opt-in and never runs in default CI.",
|
||||
"Model artifacts and caches remain on configured mounted-drive storage and never under `/home`.",
|
||||
"Preserve all pre-existing working-tree changes and stage only files belonging to this story.",
|
||||
"Write `evidence/DGR-018/README.md` with exact hardware, storage, source revisions, files, commands, raw-result paths, real results, limitations, and handoff.",
|
||||
"Update only this story issue to `Status: done` after every acceptance criterion passes."
|
||||
],
|
||||
"priority": 8,
|
||||
"passes": false,
|
||||
"notes": "Source issue: .scratch/distributed-gguf-runtime/issues/18-certify-whole-model-glm-5-2-runtime-semantics.md",
|
||||
"dependsOn": [
|
||||
"DGR-003",
|
||||
"DGR-004",
|
||||
"DGR-017"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "DGR-019",
|
||||
"title": "Implement and certify GLM-5.2 range, DSA, and IndexShare semantics",
|
||||
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md`, `.scratch/distributed-gguf-runtime/GLM-5.2-MAX-ALPHA-ROADMAP.md`, and `.scratch/distributed-gguf-runtime/issues/19-implement-and-certify-glm-5-2-range-dsa-indexshare.md` completely before coding. Read and verify every dependency evidence README.\n\nAs a target-model operator, I need explicit range-owned GLM-5.2 semantics so that contiguous consumer Shards preserve the whole-model MoE, DSA, IndexShare, and local-KV computation.",
|
||||
"acceptanceCriteria": [
|
||||
"Implement explicit ownership for 78 main layers, head embedding, tail norm/output head, routed/shared experts, DSA/indexer tensors, and the certified NextN/MTP policy.",
|
||||
"Each contiguous layer owner keeps all 256 experts and the shared-expert path for its layers local; no public cross-machine expert collectives are introduced.",
|
||||
"Define and fixture-test compressed MLA KV ownership for locally owned layers, keyed by Route Session/epoch, for DGR-007 to implement.",
|
||||
"Implement native DSA lightning indexer/top-2,048 and sparse-attention graph behavior matching DGR-018.",
|
||||
"Parse and validate artifact `indexer_types`; implement Full producer and Shared consumer behavior without fabricated/duplicated indexer tensors.",
|
||||
"Prefer Shard boundaries that preserve IndexShare ownership groups; when memory fit forces a split, carry a typed, bounded, validated top-k sideband in the DGR-006 named bundle.",
|
||||
"Reject missing, stale, wrong-width, wrong-position, or Shared-before-Full sideband/index state.",
|
||||
"Demonstrate mapped/resident weights and allocated KV scale with exact owned tensors/layers rather than the full model.",
|
||||
"Same-host two-stage F32 seam fixture produces 32 exact greedy tokens against the DGR-018 semantics; production seam meets the pre-locked token/similarity thresholds.",
|
||||
"If full-target same-host execution cannot fit, use a layer-reduced GLM architecture fixture only for graph parity and defer full-artifact parity to DGR-020; label it non-target evidence.",
|
||||
"Add deterministic fixture tests for MoE route/shared expert, DSA Full/Shared groups, internal group split, endpoint ownership, KV filtering, malformed metadata, and NextN policy.",
|
||||
"Targeted pytest/CTest/native tests pass; pinned patch stack applies and rebuilds cleanly.",
|
||||
"`python -m compileall packages tests` and `git diff --check` pass.",
|
||||
"Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free.",
|
||||
"Full deterministic `pytest -q` passes, or exact pre-existing unrelated failures are recorded.",
|
||||
"Preserve all pre-existing working-tree changes and stage only files belonging to this story.",
|
||||
"Write `evidence/DGR-019/README.md` with files, commands, real results, raw evidence, limitations, and dependent-story handoff.",
|
||||
"Update only this story issue to `Status: done` after every acceptance criterion passes."
|
||||
],
|
||||
"priority": 9,
|
||||
"passes": false,
|
||||
"notes": "Source issue: .scratch/distributed-gguf-runtime/issues/19-implement-and-certify-glm-5-2-range-dsa-indexshare.md",
|
||||
"dependsOn": [
|
||||
"DGR-005",
|
||||
"DGR-006",
|
||||
"DGR-018"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "DGR-020",
|
||||
"title": "Pass real distributed GLM-5.2 Max alpha acceptance",
|
||||
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md`, `.scratch/distributed-gguf-runtime/GLM-5.2-MAX-ALPHA-ROADMAP.md`, and `.scratch/distributed-gguf-runtime/issues/20-pass-real-distributed-glm-5-2-max-alpha.md` completely before coding. Read and verify every dependency evidence README.\n\nAs an alpha user, I need the exact lowest-quant GLM-5.2 model to run in Max reasoning mode across consumer machines through Meshnet so that the project has proven its end goal on real hardware.",
|
||||
"acceptanceCriteria": [
|
||||
"Use the exact DGR-017 `UD-IQ1_S` artifact and certified DGR-018/DGR-019 runtime recipe with all source and patch hashes verified.",
|
||||
"Use at least two physical consumer machines with at least 224 GiB aggregate usable admitted memory; no participating node may individually admit the complete recipe and unified memory is counted once.",
|
||||
"Tracker selects disjoint contiguous Shards whose exact required tensor inventory has complete union and no unintended overlap.",
|
||||
"Every stage reports real CPU/GPU compute, owned tensor bytes/layers, local KV, backend, queue, and seam telemetry; synthetic, passthrough, or zero-layer workers fail acceptance.",
|
||||
"Runtime evidence proves native GLM MoE/shared expert, DSA lightning indexer/sparse attention, and IndexShare Full/Shared paths are active; dense fallback fails acceptance.",
|
||||
"OpenAI-compatible API applies and records `reasoning_effort=max`, stable model ID, finish reason, and token usage.",
|
||||
"Configure 16,384 context; complete the fixed 4,096-token prefill lane and generate at least 512 output tokens or valid natural EOS after at least 128 tokens.",
|
||||
"Production seam meets at least 0.90 greedy token agreement and 0.999 mean compared-state/logit cosine similarity against DGR-018 on the fixed corpus, with no non-finite or malformed tensors.",
|
||||
"Fixed coding, structured tool-call/JSON, and multi-step reasoning sentinels produce parseable relevant outputs retained for human review.",
|
||||
"After warm-up, fixed Max lane median decode is at least 0.5 token/s, 4,096-token-prompt TTFT is at most 10 minutes, and no unexplained stall exceeds 60 seconds without progress telemetry.",
|
||||
"Record per-stage/seam p50/p95 latency, bytes, compute time, peak RSS/VRAM, KV pressure, network properties, errors, and total energy when available.",
|
||||
"Two consecutive cold starts load, generate, release, and exit without leaked processes, mapped weights, queues, or KV leases.",
|
||||
"Cancellation in prefill and decode releases all stages; one worker loss aborts the route; retry starts from token zero on a newly compatible route; stale epochs and duplicate steps fail closed.",
|
||||
"Model artifacts/caches remain on configured mounted-drive storage and never under `/home`; secret scan passes.",
|
||||
"Run targeted/full deterministic tests and clean native rebuild in addition to target execution; `git diff --check` passes.",
|
||||
"Preserve all raw failures and do not weaken DGR-017 thresholds after results.",
|
||||
"Write a signed `alpha` verdict only if every criterion passes; otherwise write `stop` with the measured bottleneck and next narrow action.",
|
||||
"Write `evidence/DGR-020/README.md` with exact commands, manifests, raw-result paths, outputs, limitations, and post-alpha handoff.",
|
||||
"Update only this story issue to `Status: done` after the immutable verdict and all evidence exist."
|
||||
],
|
||||
"priority": 16,
|
||||
"passes": false,
|
||||
"notes": "Source issue: .scratch/distributed-gguf-runtime/issues/20-pass-real-distributed-glm-5-2-max-alpha.md",
|
||||
"dependsOn": [
|
||||
"DGR-007",
|
||||
"DGR-008",
|
||||
"DGR-009",
|
||||
"DGR-011",
|
||||
"DGR-013",
|
||||
"DGR-017",
|
||||
"DGR-018",
|
||||
"DGR-019"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user