neuron-tai/docs/issues/47-model-source-download-visibility.md

US-047 — Tracker-first model downloads: visibility, sane timeouts, RAM-based sizing

Status: in progress Priority: High (follow-up to US-044/US-046; blocks usable LAN downloads)

Context

Reported 2026-07-06 (Windows CPU node, 79.2 GB RAM, --tracker http://192.168.0.179:8080 --model Qwen3.6-35B-A3B):

1. Startup prints (auto-join unavailable: HTTP Error 503) even though the user explicitly named a model. The auto-join query (/v1/network/assign) never sends the requested model, so a fresh tracker + a caller too small for the recommended preset 503s (expected per US-046) — but the whole auto-join step is pointless when the user already picked a model: the /v1/nodes/assign?model=… call right after it succeeds (assigned layers 0–2 with tracker model_sources).
2. The tracker-vs-HuggingFace race then starts, but only HuggingFace shows progress (hf tqdm bars). The tracker tar download prints nothing and swallows every failure (except Exception: return None), so the node appears to download only from slow HF; the user killed it. Tracker-side log showed the tar stream reset mid-archive.add — with no way to tell whether the client timed out or the user aborted.
3. _download_model_source inherits peer_timeout (2.0 s) as its urlopen socket timeout. Any 2 s read stall during a multi-GB tar stream silently kills the tracker source and leaves HF as the only contender.
4. Every client abort spams the tracker console with a full BrokenPipeError/ConnectionResetError traceback from socketserver.

Fix

1. startup.py: skip the network auto-join query entirely when a model was explicitly requested (model set and not "stub-model"); path 3b (/v1/nodes/assign?model=…) is the authoritative one there.
2. downloader.py: model-source downloads get their own timeout constant (30 s socket timeout) instead of the 2 s peer-probe timeout. Peer shard downloads keep 2 s — they run sequentially before the race, and a dead peer must not hang startup for 30 s; the race is concurrent so a slow source costs nothing.
3. downloader.py: progress + failure visibility for the race — _download_model_source prints received bytes every 512 MB and prints the exception when a source fails, so "downloads only from HF" can never happen silently again.
4. Tracker _handle_model_files_download: catch BrokenPipeError/ConnectionResetError around the tar stream and log a single line instead of a traceback.

Design revision (2026-07-06, after live retest)

The race is gone. User decision: HuggingFace is used only when the model is not available from a tracker/peer source, or when --tracker-source-disabled is passed. Sources are tried sequentially with progress + failure output; HF (layer-filtered via the source file list, else the remote index) is the fallback.

Second live finding: the node was assigned only layers 0–2 of 40 on a 79 GB box. Cause: CPU-mode nodes still report the detected-but-unusable GPU's vram_mb (RTX 4060 → 8192), and shard sizing used VRAM whenever it was > 0 (8 GB × 0.8 ≈ 6.5 GB ≈ 3 layers). Fixed on both sides: the node now sends assignment_vram_mb (0 unless CUDA is actually usable) to /v1/nodes/assign, and the tracker only trusts vram_mb when device=cuda (all three sizing sites), falling back to ram_mb.

Acceptance criteria

• Node started with an explicit --model never queries /v1/network/assign and never prints auto-join unavailable.
• Tracker/peer model source is preferred outright; HF is contacted only when no source is advertised, every source fails, or --tracker-source-disabled is passed (flag on both CLI parsers, plumbed through config and run_startup).
• Tracker-source downloads print progress every 512 MB and print the exception + URL on failure; nothing fails silently.
• A ≥2 s read stall no longer aborts a tracker model-source download (30 s socket timeout).
• Client disconnect during /v1/model-files/download logs one line on the tracker, no traceback.
• CPU node with big RAM gets a RAM-sized shard: /v1/nodes/assign and both /v1/network/assign sizing paths ignore VRAM unless device=cuda.
• pytest tests/test_node_startup.py tests/test_tracker_routing.py passes (139/140; the one failure is the pre-existing port-dependent test_mining_cli case, present on clean master).
• Live two-machine retest: Windows node downloads only from tracker at LAN speed and is assigned a RAM-sized shard.