N-th fix od model DL
This commit is contained in:
@@ -82,3 +82,34 @@ sites), falling back to `ram_mb`.
|
||||
`test_mining_cli` case, present on clean master).
|
||||
- [ ] Live two-machine retest: Windows node downloads only from tracker at
|
||||
LAN speed and is assigned a RAM-sized shard.
|
||||
|
||||
## Round 3 (2026-07-06, after live retest showed mid-stream RST)
|
||||
|
||||
Live retest: RAM sizing worked (layers 0–36) and the failure finally printed —
|
||||
`ConnectionResetError(10054)` ~70 s into the tar stream. Local reproduction
|
||||
cleared the tracker: it streams the full 72 GB tar at ~900 MB/s, survives a
|
||||
3-minute slow reader, and logs aborts in one line. The RST comes from the
|
||||
network path (Windows laptop, likely WiFi + firewall/AV) — and a 72 GB
|
||||
single-TCP-stream tar is inherently fragile there.
|
||||
|
||||
Fix: per-file downloads (design principle: nodes must be able to fetch any
|
||||
missing shard or the complete model from the tracker alone — no hard HF
|
||||
dependency):
|
||||
|
||||
- Tracker: `/v1/model-files/download?...&file=<rel>` streams one file with
|
||||
`Content-Length` (rel must be in the requested shard/full set; traversal
|
||||
rejected). `model_sources` now advertises `full_files` and a `file_sizes`
|
||||
manifest.
|
||||
- Node: `_download_source_files` fetches per file into
|
||||
`<shard>.partial/`, retries each file 3×, verifies against
|
||||
`Content-Length`, and reuses already-complete files (hardlink from the
|
||||
existing shard) via the size manifest — so restarts and drops cost at most
|
||||
one file. Tar stream remains the fallback for old trackers
|
||||
(detected via Content-Type) and sources without a file list.
|
||||
- `_full_model_sources` passes `full_files` through, so full-snapshot
|
||||
downloads for the torch path get the same robustness.
|
||||
|
||||
Verified live against a local tracker: 14.7 GB shard in 7.6 s per-file;
|
||||
re-run over a complete shard instant; corrupt + deleted file recovered in
|
||||
1.5 s re-fetching only those two. 114 tests pass (node_startup +
|
||||
tracker_routing).
|
||||
|
||||
Reference in New Issue
Block a user