N-th fix od model DL

This commit is contained in:
Dobromir Popov
2026-07-06 23:41:06 +03:00
parent 4bfdc814e2
commit d83224a62f
5 changed files with 224 additions and 18 deletions

View File

@@ -82,3 +82,34 @@ sites), falling back to `ram_mb`.
`test_mining_cli` case, present on clean master).
- [ ] Live two-machine retest: Windows node downloads only from tracker at
LAN speed and is assigned a RAM-sized shard.
## Round 3 (2026-07-06, after live retest showed mid-stream RST)
Live retest: RAM sizing worked (layers 036) and the failure finally printed —
`ConnectionResetError(10054)` ~70 s into the tar stream. Local reproduction
cleared the tracker: it streams the full 72 GB tar at ~900 MB/s, survives a
3-minute slow reader, and logs aborts in one line. The RST comes from the
network path (Windows laptop, likely WiFi + firewall/AV) — and a 72 GB
single-TCP-stream tar is inherently fragile there.
Fix: per-file downloads (design principle: nodes must be able to fetch any
missing shard or the complete model from the tracker alone — no hard HF
dependency):
- Tracker: `/v1/model-files/download?...&file=<rel>` streams one file with
`Content-Length` (rel must be in the requested shard/full set; traversal
rejected). `model_sources` now advertises `full_files` and a `file_sizes`
manifest.
- Node: `_download_source_files` fetches per file into
`<shard>.partial/`, retries each file 3×, verifies against
`Content-Length`, and reuses already-complete files (hardlink from the
existing shard) via the size manifest — so restarts and drops cost at most
one file. Tar stream remains the fallback for old trackers
(detected via Content-Type) and sources without a file list.
- `_full_model_sources` passes `full_files` through, so full-snapshot
downloads for the torch path get the same robustness.
Verified live against a local tracker: 14.7 GB shard in 7.6 s per-file;
re-run over a complete shard instant; corrupt + deleted file recovered in
1.5 s re-fetching only those two. 114 tests pass (node_startup +
tracker_routing).