neuron-tai/docs/issues/47-model-source-download-visibility.md

US-047 — Tracker-first model downloads: visibility, sane timeouts, RAM-based sizing

Status: in progress Priority: High (follow-up to US-044/US-046; blocks usable LAN downloads)

Context

Reported 2026-07-06 (Windows CPU node, 79.2 GB RAM, --tracker http://192.168.0.179:8080 --model Qwen3.6-35B-A3B):

1. Startup prints (auto-join unavailable: HTTP Error 503) even though the user explicitly named a model. The auto-join query (/v1/network/assign) never sends the requested model, so a fresh tracker + a caller too small for the recommended preset 503s (expected per US-046) — but the whole auto-join step is pointless when the user already picked a model: the /v1/nodes/assign?model=… call right after it succeeds (assigned layers 0–2 with tracker model_sources).
2. The tracker-vs-HuggingFace race then starts, but only HuggingFace shows progress (hf tqdm bars). The tracker tar download prints nothing and swallows every failure (except Exception: return None), so the node appears to download only from slow HF; the user killed it. Tracker-side log showed the tar stream reset mid-archive.add — with no way to tell whether the client timed out or the user aborted.
3. _download_model_source inherits peer_timeout (2.0 s) as its urlopen socket timeout. Any 2 s read stall during a multi-GB tar stream silently kills the tracker source and leaves HF as the only contender.
4. Every client abort spams the tracker console with a full BrokenPipeError/ConnectionResetError traceback from socketserver.

Fix

1. startup.py: skip the network auto-join query entirely when a model was explicitly requested (model set and not "stub-model"); path 3b (/v1/nodes/assign?model=…) is the authoritative one there.
2. downloader.py: model-source downloads get their own timeout constant (30 s socket timeout) instead of the 2 s peer-probe timeout. Peer shard downloads keep 2 s — they run sequentially before the race, and a dead peer must not hang startup for 30 s; the race is concurrent so a slow source costs nothing.
3. downloader.py: progress + failure visibility for the race — _download_model_source prints received bytes every 512 MB and prints the exception when a source fails, so "downloads only from HF" can never happen silently again.
4. Tracker _handle_model_files_download: catch BrokenPipeError/ConnectionResetError around the tar stream and log a single line instead of a traceback.

Design revision (2026-07-06, after live retest)

The race is gone. User decision: HuggingFace is used only when the model is not available from a tracker/peer source, or when --tracker-source-disabled is passed. Sources are tried sequentially with progress + failure output; HF (layer-filtered via the source file list, else the remote index) is the fallback.

Second live finding: the node was assigned only layers 0–2 of 40 on a 79 GB box. Cause: CPU-mode nodes still report the detected-but-unusable GPU's vram_mb (RTX 4060 → 8192), and shard sizing used VRAM whenever it was > 0 (8 GB × 0.8 ≈ 6.5 GB ≈ 3 layers). Fixed on both sides: the node now sends assignment_vram_mb (0 unless CUDA is actually usable) to /v1/nodes/assign, and the tracker only trusts vram_mb when device=cuda (all three sizing sites), falling back to ram_mb.

Acceptance criteria

• Node started with an explicit --model never queries /v1/network/assign and never prints auto-join unavailable.
• Tracker/peer model source is preferred outright; HF is contacted only when no source is advertised, every source fails, or --tracker-source-disabled is passed (flag on both CLI parsers, plumbed through config and run_startup).
• Tracker-source downloads print progress every 512 MB and print the exception + URL on failure; nothing fails silently.
• A ≥2 s read stall no longer aborts a tracker model-source download (30 s socket timeout).
• Client disconnect during /v1/model-files/download logs one line on the tracker, no traceback.
• CPU node with big RAM gets a RAM-sized shard: /v1/nodes/assign and both /v1/network/assign sizing paths ignore VRAM unless device=cuda.
• pytest tests/test_node_startup.py tests/test_tracker_routing.py passes (139/140; the one failure is the pre-existing port-dependent test_mining_cli case, present on clean master).
• Live two-machine retest: Windows node downloads only from tracker at LAN speed and is assigned a RAM-sized shard.

Round 3 (2026-07-06, after live retest showed mid-stream RST)

Live retest: RAM sizing worked (layers 0–36) and the failure finally printed — ConnectionResetError(10054) ~70 s into the tar stream. Local reproduction cleared the tracker: it streams the full 72 GB tar at ~900 MB/s, survives a 3-minute slow reader, and logs aborts in one line. The RST comes from the network path (Windows laptop, likely WiFi + firewall/AV) — and a 72 GB single-TCP-stream tar is inherently fragile there.

Fix: per-file downloads (design principle: nodes must be able to fetch any missing shard or the complete model from the tracker alone — no hard HF dependency):

• Tracker: /v1/model-files/download?...&file=<rel> streams one file with Content-Length (rel must be in the requested shard/full set; traversal rejected). model_sources now advertises full_files and a file_sizes manifest.
• Node: _download_source_files fetches per file into <shard>.partial/, retries each file 3×, verifies against Content-Length, and reuses already-complete files (hardlink from the existing shard) via the size manifest — so restarts and drops cost at most one file. Tar stream remains the fallback for old trackers (detected via Content-Type) and sources without a file list.
• _full_model_sources passes full_files through, so full-snapshot downloads for the torch path get the same robustness.

Verified live against a local tracker: 14.7 GB shard in 7.6 s per-file; re-run over a complete shard instant; corrupt + deleted file recovered in 1.5 s re-fetching only those two. 114 tests pass (node_startup + tracker_routing).