chore: triage maintenance review and close completed stories

2026-07-14 14:33:09 +03:00
parent 64f83d4392
commit cae7c2b171
17 changed files with 412 additions and 16 deletions
--- a/.scratch/distributed-gguf-runtime/evidence/DGR-018/BLOCKED.md
+++ b/.scratch/distributed-gguf-runtime/evidence/DGR-018/BLOCKED.md
@@ -0,0 +1,53 @@
+# DGR-018 — BLOCKED: no 256-GiB-class oracle host
+
+Recorded: 2026-07-14 (MAINT-003). Preflight scripts preserved at commit
+`a0f28b5` ("chore: preserve DGR-018 preflight scripts (postponed)").
+
+## Blocker
+
+DGR-018 requires a 256-GiB-class host with at least **224 GiB
+runtime-accessible memory** (the DGR-017 experimental hard-fit floor for the
+whole-model `UD-IQ1_S` oracle) and **250 GB free storage** on one filesystem
+outside `/home` (216.715 GB artifact plus resume/temp headroom). The available
+development host fails both gates, so the whole-model oracle cannot be
+established. Per the issue's finish contract, no smaller model may be
+substituted.
+
+DGR-019 (needs the DGR-018 oracle for parity certification) and DGR-020
+(needs DGR-018 and DGR-019, plus enough physical consumer nodes that no single
+node admits the whole recipe) are blocked transitively.
+
+## Exact preflight output
+
+Command (offline; resolves everything from the pinned target manifest and
+never contacts the network):
+
+```
+$ python scripts/glm_whole_model_preflight.py
+target: UD-IQ1_S 216.715 GB, 6 shards @ abc55e725277
+[FAIL] storage: need >= 250 GB free on one filesystem outside ['/home']; observed no eligible filesystem
+[FAIL] memory: need >= 224 GiB runtime-accessible memory (DGR-017 experimental hard-fit floor); observed 124.9 GiB MemTotal
+destination: NONE — no filesystem outside ['/home'] has 250 GB free
+  - /run/media/popov/DATA (ext4): 74.2 GB free
+  - / (ext4): 51.1 GB free
+  - /run/media/popov/Windows (fuseblk): 26.0 GB free
+  - /run/media/popov/d (fuseblk): 5.1 GB free
+verdict: fail
+$ echo $?
+1
+```
+
+Host: Linux 7.0.14-101.fc43.x86_64 x86_64, `MemTotal: 130997376 kB`
+(124.9 GiB). The full machine-readable report (including the ordered
+download/verify plan against revision `abc55e72527792c6e77069c99b4cb7de16fa9f23`,
+manifest SHA-256 `0b6aed04479d204902bb64c0203f1a46cab26a47b378ecccf85237b63f6c1962`)
+is in [preflight.json](preflight.json).
+
+## How to resume
+
+1. On a qualifying host, run `python scripts/glm_whole_model_preflight.py`
+   (optionally `--dest DIR`); it must exit 0 with `verdict: pass`.
+2. Download shards in the preflight's ordered plan; verify each with
+   `python scripts/verify_glm_shards.py` before the next transfer starts.
+3. Proceed with the DGR-018 issue
+   (`.scratch/distributed-gguf-runtime/issues/18-certify-whole-model-glm-5-2-runtime-semantics.md`).
--- a/.scratch/distributed-gguf-runtime/evidence/DGR-018/preflight.json
+++ b/.scratch/distributed-gguf-runtime/evidence/DGR-018/preflight.json
@@ -0,0 +1,140 @@
+{
+  "generated_by": "scripts/glm_whole_model_preflight.py",
+  "target": {
+    "gguf_repo_id": "unsloth/GLM-5.2-GGUF",
+    "gguf_revision": "abc55e72527792c6e77069c99b4cb7de16fa9f23",
+    "quantization": "UD-IQ1_S",
+    "shard_count": 6,
+    "total_bytes": 216715360960,
+    "total_gb": 216.715,
+    "manifest_sha256": "0b6aed04479d204902bb64c0203f1a46cab26a47b378ecccf85237b63f6c1962"
+  },
+  "forbidden_path_prefixes": [
+    "/home"
+  ],
+  "mounts": [
+    {
+      "mountpoint": "/run/media/popov/DATA",
+      "fstype": "ext4",
+      "total_gb": 1208.8,
+      "free_gb": 74.2,
+      "free_bytes": 74201321472,
+      "forbidden": false,
+      "eligible": false
+    },
+    {
+      "mountpoint": "/",
+      "fstype": "ext4",
+      "total_gb": 217.7,
+      "free_gb": 51.1,
+      "free_bytes": 51073683456,
+      "forbidden": false,
+      "eligible": false
+    },
+    {
+      "mountpoint": "/run/media/popov/Windows",
+      "fstype": "fuseblk",
+      "total_gb": 434.9,
+      "free_gb": 26.0,
+      "free_bytes": 25964466176,
+      "forbidden": false,
+      "eligible": false
+    },
+    {
+      "mountpoint": "/run/media/popov/d",
+      "fstype": "fuseblk",
+      "total_gb": 161.1,
+      "free_gb": 5.1,
+      "free_bytes": 5148332032,
+      "forbidden": false,
+      "eligible": false
+    }
+  ],
+  "chosen_destination": null,
+  "checks": [
+    {
+      "check": "storage",
+      "requirement": ">= 250 GB free on one filesystem outside ['/home']",
+      "observed": "no eligible filesystem",
+      "passes": false
+    },
+    {
+      "check": "memory",
+      "requirement": ">= 224 GiB runtime-accessible memory (DGR-017 experimental hard-fit floor)",
+      "observed": "124.9 GiB MemTotal",
+      "passes": false,
+      "waived": false
+    }
+  ],
+  "download_authorized": false,
+  "storage_only": false,
+  "download_plan": [
+    {
+      "step": 1,
+      "shard_index": 1,
+      "path": "UD-IQ1_S/GLM-5.2-UD-IQ1_S-00001-of-00006.gguf",
+      "size_bytes": 9423744,
+      "size_gb": 0.009,
+      "sha256": "46b6148389219ae45167cb8124fbb18ef7d432daf619b4faf9e06ea80d3f4777",
+      "url": "https://huggingface.co/unsloth/GLM-5.2-GGUF/resolve/abc55e72527792c6e77069c99b4cb7de16fa9f23/UD-IQ1_S/GLM-5.2-UD-IQ1_S-00001-of-00006.gguf",
+      "download_command": "curl -L -C - --fail -o \"$GLM_DEST/UD-IQ1_S/GLM-5.2-UD-IQ1_S-00001-of-00006.gguf\" \"https://huggingface.co/unsloth/GLM-5.2-GGUF/resolve/abc55e72527792c6e77069c99b4cb7de16fa9f23/UD-IQ1_S/GLM-5.2-UD-IQ1_S-00001-of-00006.gguf\"",
+      "verify_command": "python scripts/verify_glm_shards.py --model-dir \"$GLM_DEST\" --shard 1"
+    },
+    {
+      "step": 2,
+      "shard_index": 6,
+      "path": "UD-IQ1_S/GLM-5.2-UD-IQ1_S-00006-of-00006.gguf",
+      "size_bytes": 19171063136,
+      "size_gb": 19.171,
+      "sha256": "3b767f55df64e0432d52fcf1a14eb47a1ef3bbc91339e2ae220f38602237d7d7",
+      "url": "https://huggingface.co/unsloth/GLM-5.2-GGUF/resolve/abc55e72527792c6e77069c99b4cb7de16fa9f23/UD-IQ1_S/GLM-5.2-UD-IQ1_S-00006-of-00006.gguf",
+      "download_command": "curl -L -C - --fail -o \"$GLM_DEST/UD-IQ1_S/GLM-5.2-UD-IQ1_S-00006-of-00006.gguf\" \"https://huggingface.co/unsloth/GLM-5.2-GGUF/resolve/abc55e72527792c6e77069c99b4cb7de16fa9f23/UD-IQ1_S/GLM-5.2-UD-IQ1_S-00006-of-00006.gguf\"",
+      "verify_command": "python scripts/verify_glm_shards.py --model-dir \"$GLM_DEST\" --shard 6"
+    },
+    {
+      "step": 3,
+      "shard_index": 2,
+      "path": "UD-IQ1_S/GLM-5.2-UD-IQ1_S-00002-of-00006.gguf",
+      "size_bytes": 49208128256,
+      "size_gb": 49.208,
+      "sha256": "f2180207285e04fcaa5b8c53ba6e77ad5cc58666b6e7c6b04a5eded3fe8bef09",
+      "url": "https://huggingface.co/unsloth/GLM-5.2-GGUF/resolve/abc55e72527792c6e77069c99b4cb7de16fa9f23/UD-IQ1_S/GLM-5.2-UD-IQ1_S-00002-of-00006.gguf",
+      "download_command": "curl -L -C - --fail -o \"$GLM_DEST/UD-IQ1_S/GLM-5.2-UD-IQ1_S-00002-of-00006.gguf\" \"https://huggingface.co/unsloth/GLM-5.2-GGUF/resolve/abc55e72527792c6e77069c99b4cb7de16fa9f23/UD-IQ1_S/GLM-5.2-UD-IQ1_S-00002-of-00006.gguf\"",
+      "verify_command": "python scripts/verify_glm_shards.py --model-dir \"$GLM_DEST\" --shard 2"
+    },
+    {
+      "step": 4,
+      "shard_index": 3,
+      "path": "UD-IQ1_S/GLM-5.2-UD-IQ1_S-00003-of-00006.gguf",
+      "size_bytes": 49684417024,
+      "size_gb": 49.684,
+      "sha256": "b1c0c5a302cc8d5d9ea0bcd4467c01db72c26839f820f7e882079582ea0a8d2b",
+      "url": "https://huggingface.co/unsloth/GLM-5.2-GGUF/resolve/abc55e72527792c6e77069c99b4cb7de16fa9f23/UD-IQ1_S/GLM-5.2-UD-IQ1_S-00003-of-00006.gguf",
+      "download_command": "curl -L -C - --fail -o \"$GLM_DEST/UD-IQ1_S/GLM-5.2-UD-IQ1_S-00003-of-00006.gguf\" \"https://huggingface.co/unsloth/GLM-5.2-GGUF/resolve/abc55e72527792c6e77069c99b4cb7de16fa9f23/UD-IQ1_S/GLM-5.2-UD-IQ1_S-00003-of-00006.gguf\"",
+      "verify_command": "python scripts/verify_glm_shards.py --model-dir \"$GLM_DEST\" --shard 3"
+    },
+    {
+      "step": 5,
+      "shard_index": 4,
+      "path": "UD-IQ1_S/GLM-5.2-UD-IQ1_S-00004-of-00006.gguf",
+      "size_bytes": 49396052864,
+      "size_gb": 49.396,
+      "sha256": "a6a42da6975e29f89866dcde2956e9e50e6ea26635fb5063b74f3973f4f863b6",
+      "url": "https://huggingface.co/unsloth/GLM-5.2-GGUF/resolve/abc55e72527792c6e77069c99b4cb7de16fa9f23/UD-IQ1_S/GLM-5.2-UD-IQ1_S-00004-of-00006.gguf",
+      "download_command": "curl -L -C - --fail -o \"$GLM_DEST/UD-IQ1_S/GLM-5.2-UD-IQ1_S-00004-of-00006.gguf\" \"https://huggingface.co/unsloth/GLM-5.2-GGUF/resolve/abc55e72527792c6e77069c99b4cb7de16fa9f23/UD-IQ1_S/GLM-5.2-UD-IQ1_S-00004-of-00006.gguf\"",
+      "verify_command": "python scripts/verify_glm_shards.py --model-dir \"$GLM_DEST\" --shard 4"
+    },
+    {
+      "step": 6,
+      "shard_index": 5,
+      "path": "UD-IQ1_S/GLM-5.2-UD-IQ1_S-00005-of-00006.gguf",
+      "size_bytes": 49246275936,
+      "size_gb": 49.246,
+      "sha256": "a4a9851a50db533f21ef824e5d8038f04e6782e7d602d18e5fdd6643f68ccccb",
+      "url": "https://huggingface.co/unsloth/GLM-5.2-GGUF/resolve/abc55e72527792c6e77069c99b4cb7de16fa9f23/UD-IQ1_S/GLM-5.2-UD-IQ1_S-00005-of-00006.gguf",
+      "download_command": "curl -L -C - --fail -o \"$GLM_DEST/UD-IQ1_S/GLM-5.2-UD-IQ1_S-00005-of-00006.gguf\" \"https://huggingface.co/unsloth/GLM-5.2-GGUF/resolve/abc55e72527792c6e77069c99b4cb7de16fa9f23/UD-IQ1_S/GLM-5.2-UD-IQ1_S-00005-of-00006.gguf\"",
+      "verify_command": "python scripts/verify_glm_shards.py --model-dir \"$GLM_DEST\" --shard 5"
+    }
+  ],
+  "verdict": "fail"
+}
--- a/.scratch/distributed-gguf-runtime/evidence/MAINTENANCE-HANDOFF.md
+++ b/.scratch/distributed-gguf-runtime/evidence/MAINTENANCE-HANDOFF.md
@@ -0,0 +1,62 @@
+# Maintenance review handoff — distributed GGUF runtime
+
+Date: 2026-07-14
+Scope: close the maintenance review, preserve the hard blockers, and hand off the remaining implementation work to the next model.
+
+## What is complete
+
+- Completed stories are now recorded in `docs/issues/distributed-gguf-runtime/`.
+- The PRD and milestone docs were updated to reflect the closed set and the blocked set.
+- The DGR-018 preflight scripts were preserved at commit `a0f28b5`.
+- The current feature line has delivered DGR-001 through DGR-006 and DGR-017.
+
+## Hard / unsolved issues for later
+
+### 1) DGR-018 requires hardware we do not have
+
+DGR-018 is blocked because the whole-model GLM-5.2 UD-IQ1_S oracle requires:
+
+- a **256-GiB-class host**,
+- at least **224 GiB runtime-accessible memory**,
+- at least **250 GB free storage on one filesystem outside `/home`**.
+
+The current development host reports only **124.9 GiB MemTotal** and has no eligible filesystem with 250 GB free.
+The authoritative blocker evidence is in `evidence/DGR-018/BLOCKED.md` and `evidence/DGR-018/preflight.json`.
+
+### 2) DGR-019 and DGR-020 are transitively blocked
+
+- **DGR-019** needs the DGR-018 oracle for parity certification.
+- **DGR-020** needs DGR-018 and DGR-019, plus enough physical consumer nodes that no single node can admit the whole recipe.
+
+No smaller model may be substituted for these stories.
+
+### 3) The remainder of the graph stays blocked unless replanned
+
+The current graph makes **DGR-007 depend on DGR-019**, which means:
+
+- DGR-007 through DGR-016 are also blocked transitively.
+- Unblocking the dense pipeline without the 256-GiB host would require an explicit replanning decision to relax the DGR-007 → DGR-019 dependency.
+- That replanning decision has **not** been made.
+
+### 4) Maintenance-only tasks should stay separate from feature implementation
+
+The review uncovered that the codebase now has a clean closed-story split, but further work should avoid mixing:
+
+- maintenance cleanup,
+- blocked-hardware preparation,
+- and actual distributed GLM implementation.
+
+The next model should treat the maintenance pass as closed and only pick up real implementation work that is not hardware-blocked.
+
+## Recommended next move
+
+Use the next model to continue on the **non-blocked implementation queue** only.
+Priority candidates are whatever is still actionable without the GLM oracle host; if a story depends on DGR-018, keep it deferred.
+
+## Reference files
+
+- `docs/issues/distributed-gguf-runtime/README.md`
+- `.scratch/distributed-gguf-runtime/PRD.md`
+- `.scratch/distributed-gguf-runtime/milestones.md`
+- `.scratch/distributed-gguf-runtime/evidence/DGR-018/BLOCKED.md`
+- `.scratch/distributed-gguf-runtime/evidence/DGR-018/preflight.json`