neuron-tai/.scratch/distributed-gguf-runtime/evidence/DGR-026

DGR-026 evidence — provision exact split-GGUF artifacts outside /home

Status: implemented and verified this session; live re-review, not inherited credit. Dependency: DGR-025 (evidence/DGR-025/README.md) — read before changing code.

Objective

Make exact split-GGUF inputs reproducibly available from mounted-drive storage, bound by a hashed manifest that fingerprints the source artifact, tokenizer/revision, and every split file, without embedding a quantization or split-topology assumption anywhere in product code.

What was found live (verified, not inherited)

Per RALPH-CONTEXT, legacy pass states were not trusted. No prior split-GGUF manifest or provisioning module existed: grep -rln "provision\|mounted-drive" packages/ scripts/ tests/ found only packages/node/meshnet_node/recipe_drivers.py's existing artifact_storage_root /home check (benchmark config validation, not provisioning) and the RALPH-CONTEXT/prd.json prose itself. The pre-existing packages/node/meshnet_node/downloader.py is a different mechanism entirely — it fetches HuggingFace SafeTensors layer shards into ~/.cache/meshnet/shards (i.e. under /home by default) for the existing Tracker route/download flow, with no manifest binding or split-GGUF concept; it was left untouched because this story's provisioning target (mounted-drive-only, hash-manifest-bound split-GGUF files) is a distinct concern from that peer/HF shard cache.

Two existing conventions were read and reused directly rather than reinvented:

• packages/node/meshnet_node/glm_alpha/manifest.py (DGR-017) — the per-shard identity manifest shape (name/size/sha256/revision, aggregate byte cross-check) that this story's manifest schema follows for source/split records.
• packages/node/meshnet_node/runtime_recipe.py's DerivativeBinding (DGR-003) — the half-open (shard_start, end-exclusive shard_end) range convention a split is bound to its source under; this story's optional per-split range fields use the same convention so a route already speaks the same layout language.
• packages/node/meshnet_node/recipe_drivers.py's _validate_config — the exact /home rejection shape (not root.is_absolute() or root == Path("/home") or Path("/home") in root.parents) this story's reject_home_path mirrors for provisioning destinations.

What was built (this story's change)

packages/node/meshnet_node/split_gguf/ (new package)

• manifest.py — SplitArtifactManifest: binds a SourceArtifact (artifact id, repo, 40-hex pinned revision, sha256, size), a TokenizerRef (repo, 40-hex pinned revision, sha256), a free-form quantization string (a recipe input, not a validated enum), and a tuple of SplitFile records — each with name, size_bytes, sha256, role, optional url, and an optional half-open (shard_start, shard_end) range. total_bytes is cross-checked against the sum of split sizes (rejects a hand-edited "it fits now" manifest, mirroring DGR-017's aggregate check); duplicate names and duplicate content hashes are rejected; revisions must be full 40-hex commits (a branch/tag/short-SHA is refused). Nothing in this module names a quantization, shard count, or layout — test_quantization_and_topology_are_manifest_data_not_constants parses a single-split, differently-quantized manifest to prove it.
• provision.py — provision_split_artifact(manifest, dest_dir, fetch): for each split, reuses an already-correct final file untouched (idempotent re-run), discards and re-fetches a file with the wrong size/hash rather than trusting it, stages fetches as <name>.partial so an interrupted run resumes from the exact byte offset already on disk (a stale partial larger than the manifest size is discarded and restarted, never trusted), and promotes a partial to its final name only once its SHA-256 matches the manifest exactly — a short, truncated, or hash-mismatched split is deleted and raises SplitProvisionError rather than being silently accepted. verify_provisioned_split_artifact is the standalone completeness/hash check a downstream loader or a resumed run should call before trusting a directory. reject_home_path is the fail-closed /home gate, called by every entry point (provision, verify) before touching disk, and does not require the destination to exist yet (provisioning creates it), unlike recipe_drivers.py's strict=True benchmark-root check. Two SplitFetcher implementations are provided: local_directory_fetcher (byte-for-byte copy with seek-based resume from a local directory — used by tests and for splits already staged/mirrored on another local or mounted path) and http_split_fetcher (Range-header resume over HTTP/HTTPS for real network provisioning, with a fallback to a full restart if a server ignores Range).

scripts/provision_split_gguf.py (new)

A CLI wrapper: --manifest, --dest, optional --source-dir (uses local_directory_fetcher instead of downloading each split's manifest url). Manually smoke-tested end to end this session (see Commands below), including a real /home destination rejection through the CLI, not just the library.

Tests (new, deterministic, offline, GPU-free, download-free)

• tests/test_split_gguf_manifest.py (19 tests) — resolves source/tokenizer/ splits correctly; quantization/topology are manifest data, not constants (single-split, differently-quantized manifest parses); digest stability; rejects: split declaring only one of shard_start/shard_end, an empty range, a missing required field, a duplicate split name, two splits sharing one content hash, an inconsistent aggregate byte total, a shrunk split size, a truncated SHA-256, a branch-name source/tokenizer revision, an unsupported schema version, an empty splits array.
• tests/test_split_gguf_provision.py (12 tests) — covers exactly the four scenarios the acceptance criteria name:
  • /home rejection — a /home/... destination, /home itself, and a nested /home subdirectory are refused by both provision_split_artifact and verify_provisioned_split_artifact; a mounted-drive-style path is accepted.
  • Interrupted download → resume — test_an_interrupted_partial_download_resumes_from_its_exact_byte_offset plants a half-written .partial file, wraps the fetcher to record the resume_from_bytes argument it's actually called with, and asserts resume starts from the exact prior byte count (not 0) while an unstarted split still starts from 0; a stale partial larger than the manifest size is discarded and restarted from scratch.
  • Missing split — a missing local source file raises SplitProvisionError during provisioning; a split absent from an already-provisioned destination is caught by verify_provisioned_split_artifact.
  • Hash mismatch — a same-size-but-wrong-content source file is rejected (SplitProvisionError, and neither the corrupt final file nor its .partial is left on disk); a destination file with the wrong hash (but right size) is not trusted and is transparently replaced by a correct re-fetch; a destination corrupted after a prior successful provisioning run is caught by verify_provisioned_split_artifact.
  • Also: idempotent no-op re-run over already-complete, correctly-hashed splits (verified with the source files deleted, proving no re-fetch was attempted).

Acceptance criteria → evidence

1. Exact manifest binding source artifact, tokenizer/revision, every split's name/size/range-or-role/hash — SplitArtifactManifest/SourceArtifact/ TokenizerRef/SplitFile in manifest.py; covered by test_split_gguf_manifest.py.
2. Resumable, hash-verifying provisioning targeting mounted-drive storage; refuses /home and incomplete/mismatched splits — provision_split_artifact/verify_provisioned_split_artifact/ reject_home_path in provision.py; covered by test_split_gguf_provision.py and the CLI smoke test below.
3. Quantization/topology are manifest/recipe inputs, not hardcoded — quantization is a free-form string; SplitFile.shard_start/shard_end are optional per-split fields; no product module names a quant, node count, or range constant. Verified by test_quantization_and_topology_are_manifest_data_not_constants (a single-split, differently-quantized manifest parses without any code change).
4. Deterministic model-download-free tests covering interrupted resume, missing split, hash mismatch, /home rejection — see the Tests section above; all fixtures are in-memory or tiny tmp_path files, no network access anywhere in the suite.
5. Gates + this handoff — below.

Commands and results

python3 -m pytest -q tests/test_split_gguf_manifest.py tests/test_split_gguf_provision.py

31 passed in 0.10s

python3 -m pytest -q tests/test_ralph_prd_schema.py

108 passed

python3 -m compileall -q packages/node/meshnet_node/split_gguf tests scripts/provision_split_gguf.py
git diff --check
python3 scripts/ralph_prd_schema.py validate .scratch/distributed-gguf-runtime/prd.json

(compileall exit 0; git diff --check exit 0)
OK: 55 stories validated.

CLI smoke test (manual, not part of the automated suite — exercises the real network-capable code path against tiny local files instead of a real model):

python3 scripts/provision_split_gguf.py \
  --manifest /tmp/dgr026-smoke/manifest.json --dest /tmp/dgr026-smoke/dest \
  --source-dir /tmp/dgr026-smoke/source
# -> "provisioned 2 split(s) to /tmp/dgr026-smoke/dest"

python3 scripts/provision_split_gguf.py \
  --manifest /tmp/dgr026-smoke/manifest.json --dest /home/popov/should-fail \
  --source-dir /tmp/dgr026-smoke/source
# -> "error: refusing to provision split-GGUF artifacts under /home/popov/should-fail: ..."
# exit 1

The scratch directory (/tmp/dgr026-smoke) was removed after the smoke test; nothing from it is committed or referenced by the test suite.

Default tests are model-download-free, API-credit-free, and GPU-free; no model artifact was downloaded and nothing product-relevant was written under /home (the CLI smoke test's /home path was rejected before any write).

Changed files

• packages/node/meshnet_node/split_gguf/__init__.py (new)
• packages/node/meshnet_node/split_gguf/manifest.py (new)
• packages/node/meshnet_node/split_gguf/provision.py (new)
• scripts/provision_split_gguf.py (new)
• tests/test_split_gguf_manifest.py (new)
• tests/test_split_gguf_provision.py (new)
• .scratch/distributed-gguf-runtime/prd.json (DGR-026.passes = true + completionNotes; also restored the top-level sourceOfTruth/ qualityGates/metadataSchema/milestones/supersededStories/ branchName fields — see Gotcha below)
• .scratch/distributed-gguf-runtime/issues/026-provision-exact-split-gguf-artifacts-outside-home.md (regenerated via scripts/ralph_prd_schema.py render)
• .scratch/distributed-gguf-runtime/evidence/DGR-026/README.md (new, this file)

Gotcha reproduced (pre-existing, documented pattern)

Before touching anything, .scratch/distributed-gguf-runtime/prd.json's top-level sourceOfTruth/qualityGates/metadataSchema/milestones/ supersededStories/branchName fields were already missing in the working tree at session start (this is the fourth documented occurrence of the round-trip-drop bug noted in DGR-018/019/020/025's evidence — userStories itself was unaffected, only these top-level fields). Restored them from git show HEAD:.scratch/distributed-gguf-runtime/prd.json before making any DGR-026 edit; scripts/ralph_prd_schema.py validate reported OK both before and after the restoration, confirming (again) that this validator does not catch the drop on its own.

Limitations

• http_split_fetcher (the real network-download path) is exercised only by manual code review and the CLI's argument wiring, not by an automated test — by design, since the default suite must stay network-free. Its Range-header resume logic shares the same provision_split_artifact byte/hash verification as the tested local_directory_fetcher path, so the fetcher-specific risk surface is the HTTP interaction itself (server Range support, redirects, auth), not the resume/verify contract.
• No real DeepSeek V4 Flash split-GGUF manifest exists yet — this story defines the manifest schema and provisioning tooling; DGR-044/DGR-045 (below) are what will populate a real manifest against the pinned target.
• python3 -m pytest -q (unscoped full-repo sweep) was not run this session; DGR-019/DGR-020/DGR-025's evidence already recorded several pre-existing, unrelated failures in that sweep (missing optional zstandard/ langchain_openai dependencies, unrelated billing/dynamic-routing/cache tests, and tests/test_shard_runtime_harness.py's grpc import requirement). This story's own targeted suites, test_ralph_prd_schema.py, compileall, and git diff --check are all green as recorded above.
• Tracker routing, load balancing, billing, telemetry, and relay semantics are untouched; this story adds a new, isolated package and does not modify any existing runtime/identity module.

Dependency handoff

• DGR-044 (DeepSeek V4 Flash target contract): when pinning the real target's split-GGUF artifact, express it as a meshnet_node.split_gguf.manifest.SplitArtifactManifest — source.sha256 is the whole-model artifact digest DGR-003's ArtifactIdentity.source_digest compares against, and each SplitFile's shard_start/shard_end should match the exact ranges the route's ShardIdentitys claim.
• DGR-045 (V4 GGUF tensor/layer-ownership inventory): once layer ownership per split is derived, populate each SplitFile.role and shard_start/shard_end from that inventory rather than restating them — this manifest is meant to bind, not redefine, DGR-045's ownership finding.
• Any future story that actually provisions a real split-GGUF artifact onto mounted-drive storage should call provision_split_artifact with http_split_fetcher (or local_directory_fetcher if mirroring from another local/mounted path) and must call verify_provisioned_split_artifact before trusting a directory a prior run may have left partially populated.