neuron-tai/.scratch/distributed-gguf-runtime/evidence/DGR-026/README.md

# DGR-026 evidence — provision exact split-GGUF artifacts outside `/home`

**Status:** implemented and verified this session; live re-review, not inherited credit.
**Dependency:** DGR-025 (`evidence/DGR-025/README.md`) — read before changing code.

## Objective

Make exact split-GGUF inputs reproducibly available from mounted-drive
storage, bound by a hashed manifest that fingerprints the source artifact,
tokenizer/revision, and every split file, without embedding a quantization or
split-topology assumption anywhere in product code.

## What was found live (verified, not inherited)

Per RALPH-CONTEXT, legacy pass states were not trusted. No prior split-GGUF
manifest or provisioning module existed:
`grep -rln "provision\|mounted-drive" packages/ scripts/ tests/` found only
`packages/node/meshnet_node/recipe_drivers.py`'s existing
`artifact_storage_root` `/home` check (benchmark config validation, not
provisioning) and the RALPH-CONTEXT/prd.json prose itself. The pre-existing
`packages/node/meshnet_node/downloader.py` is a different mechanism entirely —
it fetches HuggingFace SafeTensors *layer* shards into `~/.cache/meshnet/shards`
(i.e. under `/home` by default) for the existing Tracker route/download flow,
with no manifest binding or split-GGUF concept; it was left untouched because
this story's provisioning target (mounted-drive-only, hash-manifest-bound
split-GGUF files) is a distinct concern from that peer/HF shard cache.

Two existing conventions were read and reused directly rather than
reinvented:

- `packages/node/meshnet_node/glm_alpha/manifest.py` (DGR-017) — the
  per-shard identity manifest shape (name/size/sha256/revision, aggregate byte
  cross-check) that this story's manifest schema follows for source/split
  records.
- `packages/node/meshnet_node/runtime_recipe.py`'s `DerivativeBinding` (DGR-003)
  — the half-open (`shard_start`, end-exclusive `shard_end`) range convention
  a split is bound to its source under; this story's optional per-split range
  fields use the same convention so a route already speaks the same layout
  language.
- `packages/node/meshnet_node/recipe_drivers.py`'s `_validate_config` — the
  exact `/home` rejection shape (`not root.is_absolute() or root ==
  Path("/home") or Path("/home") in root.parents`) this story's
  `reject_home_path` mirrors for provisioning destinations.

## What was built (this story's change)

### `packages/node/meshnet_node/split_gguf/` (new package)

- **`manifest.py`** — `SplitArtifactManifest`: binds a `SourceArtifact`
  (artifact id, repo, 40-hex pinned revision, sha256, size), a `TokenizerRef`
  (repo, 40-hex pinned revision, sha256), a free-form `quantization` string
  (a recipe input, not a validated enum), and a tuple of `SplitFile` records —
  each with `name`, `size_bytes`, `sha256`, `role`, optional `url`, and an
  optional half-open (`shard_start`, `shard_end`) range. `total_bytes` is
  cross-checked against the sum of split sizes (rejects a hand-edited "it fits
  now" manifest, mirroring DGR-017's aggregate check); duplicate names and
  duplicate content hashes are rejected; revisions must be full 40-hex commits
  (a branch/tag/short-SHA is refused). Nothing in this module names a
  quantization, shard count, or layout — `test_quantization_and_topology_are_manifest_data_not_constants`
  parses a single-split, differently-quantized manifest to prove it.
- **`provision.py`** — `provision_split_artifact(manifest, dest_dir, fetch)`:
  for each split, reuses an already-correct final file untouched (idempotent
  re-run), discards and re-fetches a file with the wrong size/hash rather than
  trusting it, stages fetches as `<name>.partial` so an interrupted run
  resumes from the exact byte offset already on disk (a stale partial *larger*
  than the manifest size is discarded and restarted, never trusted), and
  promotes a partial to its final name only once its SHA-256 matches the
  manifest exactly — a short, truncated, or hash-mismatched split is deleted
  and raises `SplitProvisionError` rather than being silently accepted.
  `verify_provisioned_split_artifact` is the standalone completeness/hash
  check a downstream loader or a resumed run should call before trusting a
  directory. `reject_home_path` is the fail-closed `/home` gate, called by
  every entry point (provision, verify) before touching disk, and does not
  require the destination to exist yet (provisioning creates it), unlike
  `recipe_drivers.py`'s `strict=True` benchmark-root check. Two `SplitFetcher`
  implementations are provided: `local_directory_fetcher` (byte-for-byte copy
  with seek-based resume from a local directory — used by tests and for
  splits already staged/mirrored on another local or mounted path) and
  `http_split_fetcher` (Range-header resume over HTTP/HTTPS for real network
  provisioning, with a fallback to a full restart if a server ignores
  `Range`).

### `scripts/provision_split_gguf.py` (new)

A CLI wrapper: `--manifest`, `--dest`, optional `--source-dir` (uses
`local_directory_fetcher` instead of downloading each split's manifest `url`).
Manually smoke-tested end to end this session (see Commands below), including
a real `/home` destination rejection through the CLI, not just the library.

### Tests (new, deterministic, offline, GPU-free, download-free)

- `tests/test_split_gguf_manifest.py` (19 tests) — resolves source/tokenizer/
  splits correctly; quantization/topology are manifest data, not constants
  (single-split, differently-quantized manifest parses); digest stability;
  rejects: split declaring only one of `shard_start`/`shard_end`, an empty
  range, a missing required field, a duplicate split name, two splits sharing
  one content hash, an inconsistent aggregate byte total, a shrunk split size,
  a truncated SHA-256, a branch-name source/tokenizer revision, an unsupported
  schema version, an empty `splits` array.
- `tests/test_split_gguf_provision.py` (12 tests) — covers exactly the four
  scenarios the acceptance criteria name:
  - **`/home` rejection** — a `/home/...` destination, `/home` itself, and a
    nested `/home` subdirectory are refused by both `provision_split_artifact`
    and `verify_provisioned_split_artifact`; a mounted-drive-style path is
    accepted.
  - **Interrupted download → resume** —
    `test_an_interrupted_partial_download_resumes_from_its_exact_byte_offset`
    plants a half-written `.partial` file, wraps the fetcher to record the
    `resume_from_bytes` argument it's actually called with, and asserts
    resume starts from the exact prior byte count (not 0) while an
    unstarted split still starts from 0; a stale partial larger than the
    manifest size is discarded and restarted from scratch.
  - **Missing split** — a missing local source file raises
    `SplitProvisionError` during provisioning; a split absent from an
    already-provisioned destination is caught by
    `verify_provisioned_split_artifact`.
  - **Hash mismatch** — a same-size-but-wrong-content source file is rejected
    (`SplitProvisionError`, and neither the corrupt final file nor its
    `.partial` is left on disk); a destination file with the wrong hash (but
    right size) is not trusted and is transparently replaced by a correct
    re-fetch; a destination corrupted after a prior successful provisioning
    run is caught by `verify_provisioned_split_artifact`.
  - Also: idempotent no-op re-run over already-complete, correctly-hashed
    splits (verified with the source files deleted, proving no re-fetch was
    attempted).

## Acceptance criteria → evidence

1. **Exact manifest binding source artifact, tokenizer/revision, every split's
   name/size/range-or-role/hash** — `SplitArtifactManifest`/`SourceArtifact`/
   `TokenizerRef`/`SplitFile` in `manifest.py`; covered by
   `test_split_gguf_manifest.py`.
2. **Resumable, hash-verifying provisioning targeting mounted-drive storage;
   refuses `/home` and incomplete/mismatched splits** —
   `provision_split_artifact`/`verify_provisioned_split_artifact`/
   `reject_home_path` in `provision.py`; covered by
   `test_split_gguf_provision.py` and the CLI smoke test below.
3. **Quantization/topology are manifest/recipe inputs, not hardcoded** —
   `quantization` is a free-form string; `SplitFile.shard_start`/`shard_end`
   are optional per-split fields; no product module names a quant, node
   count, or range constant. Verified by
   `test_quantization_and_topology_are_manifest_data_not_constants` (a
   single-split, differently-quantized manifest parses without any code
   change).
4. **Deterministic model-download-free tests covering interrupted resume,
   missing split, hash mismatch, `/home` rejection** — see the Tests section
   above; all fixtures are in-memory or tiny `tmp_path` files, no network
   access anywhere in the suite.
5. **Gates + this handoff** — below.

## Commands and results

```bash
python3 -m pytest -q tests/test_split_gguf_manifest.py tests/test_split_gguf_provision.py
```
```text
31 passed in 0.10s
```

```bash
python3 -m pytest -q tests/test_ralph_prd_schema.py
```
```text
108 passed
```

```bash
python3 -m compileall -q packages/node/meshnet_node/split_gguf tests scripts/provision_split_gguf.py
git diff --check
python3 scripts/ralph_prd_schema.py validate .scratch/distributed-gguf-runtime/prd.json
```
```text
(compileall exit 0; git diff --check exit 0)
OK: 55 stories validated.
```

CLI smoke test (manual, not part of the automated suite — exercises the real
network-capable code path against tiny local files instead of a real model):

```bash
python3 scripts/provision_split_gguf.py \
  --manifest /tmp/dgr026-smoke/manifest.json --dest /tmp/dgr026-smoke/dest \
  --source-dir /tmp/dgr026-smoke/source
# -> "provisioned 2 split(s) to /tmp/dgr026-smoke/dest"

python3 scripts/provision_split_gguf.py \
  --manifest /tmp/dgr026-smoke/manifest.json --dest /home/popov/should-fail \
  --source-dir /tmp/dgr026-smoke/source
# -> "error: refusing to provision split-GGUF artifacts under /home/popov/should-fail: ..."
# exit 1
```

The scratch directory (`/tmp/dgr026-smoke`) was removed after the smoke test;
nothing from it is committed or referenced by the test suite.

Default tests are model-download-free, API-credit-free, and GPU-free; no model
artifact was downloaded and nothing product-relevant was written under
`/home` (the CLI smoke test's `/home` path was rejected before any write).

## Changed files

- `packages/node/meshnet_node/split_gguf/__init__.py` (new)
- `packages/node/meshnet_node/split_gguf/manifest.py` (new)
- `packages/node/meshnet_node/split_gguf/provision.py` (new)
- `scripts/provision_split_gguf.py` (new)
- `tests/test_split_gguf_manifest.py` (new)
- `tests/test_split_gguf_provision.py` (new)
- `.scratch/distributed-gguf-runtime/prd.json` (`DGR-026.passes = true` +
  `completionNotes`; also restored the top-level `sourceOfTruth`/
  `qualityGates`/`metadataSchema`/`milestones`/`supersededStories`/
  `branchName` fields — see Gotcha below)
- `.scratch/distributed-gguf-runtime/issues/026-provision-exact-split-gguf-artifacts-outside-home.md`
  (regenerated via `scripts/ralph_prd_schema.py render`)
- `.scratch/distributed-gguf-runtime/evidence/DGR-026/README.md` (new, this file)

## Gotcha reproduced (pre-existing, documented pattern)

Before touching anything, `.scratch/distributed-gguf-runtime/prd.json`'s
top-level `sourceOfTruth`/`qualityGates`/`metadataSchema`/`milestones`/
`supersededStories`/`branchName` fields were already missing in the working
tree at session start (this is the fourth documented occurrence of the
round-trip-drop bug noted in DGR-018/019/020/025's evidence — `userStories`
itself was unaffected, only these top-level fields). Restored them from
`git show HEAD:.scratch/distributed-gguf-runtime/prd.json` before making any
DGR-026 edit; `scripts/ralph_prd_schema.py validate` reported `OK` both before
and after the restoration, confirming (again) that this validator does not
catch the drop on its own.

## Limitations

- `http_split_fetcher` (the real network-download path) is exercised only by
  manual code review and the CLI's argument wiring, not by an automated test —
  by design, since the default suite must stay network-free. Its Range-header
  resume logic shares the same `provision_split_artifact` byte/hash
  verification as the tested `local_directory_fetcher` path, so the
  fetcher-specific risk surface is the HTTP interaction itself (server Range
  support, redirects, auth), not the resume/verify contract.
- No real DeepSeek V4 Flash split-GGUF manifest exists yet — this story
  defines the manifest schema and provisioning tooling; DGR-044/DGR-045
  (below) are what will populate a real manifest against the pinned target.
- `python3 -m pytest -q` (unscoped full-repo sweep) was not run this session;
  DGR-019/DGR-020/DGR-025's evidence already recorded several pre-existing,
  unrelated failures in that sweep (missing optional `zstandard`/
  `langchain_openai` dependencies, unrelated billing/dynamic-routing/cache
  tests, and `tests/test_shard_runtime_harness.py`'s `grpc` import
  requirement). This story's own targeted suites, `test_ralph_prd_schema.py`,
  `compileall`, and `git diff --check` are all green as recorded above.
- Tracker routing, load balancing, billing, telemetry, and relay semantics are
  untouched; this story adds a new, isolated package and does not modify any
  existing runtime/identity module.

## Dependency handoff

- **DGR-044** (DeepSeek V4 Flash target contract): when pinning the real
  target's split-GGUF artifact, express it as a
  `meshnet_node.split_gguf.manifest.SplitArtifactManifest` — `source.sha256`
  is the whole-model artifact digest DGR-003's `ArtifactIdentity.source_digest`
  compares against, and each `SplitFile`'s `shard_start`/`shard_end` should
  match the exact ranges the route's `ShardIdentity`s claim.
- **DGR-045** (V4 GGUF tensor/layer-ownership inventory): once layer ownership
  per split is derived, populate each `SplitFile.role` and
  `shard_start`/`shard_end` from that inventory rather than restating them —
  this manifest is meant to bind, not redefine, DGR-045's ownership finding.
- Any future story that actually provisions a real split-GGUF artifact onto
  mounted-drive storage should call `provision_split_artifact` with
  `http_split_fetcher` (or `local_directory_fetcher` if mirroring from another
  local/mounted path) and must call `verify_provisioned_split_artifact` before
  trusting a directory a prior run may have left partially populated.