popov/neuron-tai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: target GLM-5.2 Max for distributed alpha

Browse Source
This commit is contained in:
Dobromir Popov
2026-07-13 22:32:14 +03:00
parent ef2a9e67e8
commit 5ebce15d7a
16 changed files with 866 additions and 65 deletions
Download Patch File Download Diff File Expand all files Collapse all files

27
.scratch/distributed-gguf-runtime/architecture.md
View File

@@ -7,6 +7,8 @@ Last updated: 2026-07-13
The system exists to serve high-quality models that exceed one consumer node's memory while retaining useful interactive speed and aggregate concurrency. A feature that only produces a distributed demo but is slower, globally serialized, or impossible to operate on consumer hardware is not complete.
The alpha target is the exact pinned GLM-5.2 `UD-IQ1_S` artifact in `reasoning_effort=max` mode. Its target-specific architecture/resource/acceptance contract is [GLM-5.2-MAX-ALPHA-ROADMAP.md](GLM-5.2-MAX-ALPHA-ROADMAP.md). Dense Llama is a structural fixture, not the product target.
## Existing control plane
Meshnet remains the only public control plane:
@@ -113,6 +115,8 @@ compression/checksum
Dense Llama may use one residual tensor. Other adapters may require more. vLLM's Llama and Qwen3-MoE PP paths demonstrate a boundary with both `hidden_states` and `residual`; therefore the generic protocol must not assume one anonymous tensor.
GLM-5.2 normally exchanges a 6,144-element hidden state. If a memory-balanced Shard boundary splits an IndexShare Full producer from Shared consumers, the bundle also carries the typed top-k index sideband. The planner prefers boundaries that keep an IndexShare ownership group local, but the protocol validates the sideband rather than assuming it never crosses a seam.
Only the head owns token embedding. Only the tail owns final normalization, LM head and sampling. Middle Shards exchange the architecture-defined pre-tail boundary, not final normalized embeddings.
## Hot KV State and concurrency
@@ -243,17 +247,16 @@ The GGUF path ships only if it is faster at acceptable quality or enables a larg
## Implementation sequence
1. Lock benchmark/performance contract.
2. Define gRPC/protobuf and exact recipe identity.
3. Pin llama.cpp and create the minimal patch stack.
4. Implement dense-Llama range loading and boundary parity.
5. Implement concurrent local KV.
6. Build and integrate the standalone worker.
7. Pass local two-process real-model acceptance.
8. Pass real heterogeneous two-machine acceptance.
9. Add continuous batching and failure hardening.
10. Enforce the GGUF-versus-safetensors release gate.
11. Add Qwen3/Qwen3-MoE as a separately certified adapter.
12. Prepare narrow upstream collaboration patches/tests.
1. Preserve completed DGR-001 performance and DGR-002 protocol contracts.
2. DGR-017 locks exact GLM-5.2 Max artifact, resource, and alpha acceptance identity.
3. Define exact recipe identity and pin one reproducible llama.cpp boundary.
4. Run two lanes in parallel: DGR-018 establishes the whole-model `UD-IQ1_S` oracle on 224+ GiB usable memory, while DGR-005/DGR-006 implement range loading and named boundary parity with a cheap dense fixture.
5. DGR-019 adds explicit GLM-5.2 MoE/MLA/DSA/IndexShare semantics after both lanes pass.
6. Implement local KV; build and integrate the standalone worker.
7. Pass local two-process and real two-physical-machine execution.
8. Harden cancellation, node loss, restart, and cleanup required by alpha.
9. DGR-020 executes the exact multi-node target and emits immutable `alpha` or `stop`.
10. Post-alpha: continuous batching, final comparison, longer context, MTP, and package optimization.
11. Prepare narrow upstream patches/tests; add Qwen as later architecture expansion.
See [the Ralph backlog](prd.json) and [implementation strategy](implementation-strategy.md).