116 lines
6.1 KiB
Markdown
116 lines
6.1 KiB
Markdown
# US-047 — Tracker-first model downloads: visibility, sane timeouts, RAM-based sizing
|
||
|
||
Status: in progress
|
||
Priority: High (follow-up to US-044/US-046; blocks usable LAN downloads)
|
||
|
||
## Context
|
||
|
||
Reported 2026-07-06 (Windows CPU node, 79.2 GB RAM, `--tracker
|
||
http://192.168.0.179:8080 --model Qwen3.6-35B-A3B`):
|
||
|
||
1. Startup prints `(auto-join unavailable: HTTP Error 503)` even though the
|
||
user explicitly named a model. The auto-join query (`/v1/network/assign`)
|
||
never sends the requested model, so a fresh tracker + a caller too small
|
||
for the *recommended* preset 503s (expected per US-046) — but the whole
|
||
auto-join step is pointless when the user already picked a model: the
|
||
`/v1/nodes/assign?model=…` call right after it succeeds (assigned layers
|
||
0–2 with tracker `model_sources`).
|
||
2. The tracker-vs-HuggingFace race then starts, but only HuggingFace shows
|
||
progress (hf tqdm bars). The tracker tar download prints nothing and
|
||
swallows every failure (`except Exception: return None`), so the node
|
||
*appears* to download only from slow HF; the user killed it. Tracker-side
|
||
log showed the tar stream reset mid-`archive.add` — with no way to tell
|
||
whether the client timed out or the user aborted.
|
||
3. `_download_model_source` inherits `peer_timeout` (2.0 s) as its urlopen
|
||
socket timeout. Any 2 s read stall during a multi-GB tar stream silently
|
||
kills the tracker source and leaves HF as the only contender.
|
||
4. Every client abort spams the tracker console with a full
|
||
`BrokenPipeError`/`ConnectionResetError` traceback from `socketserver`.
|
||
|
||
## Fix
|
||
|
||
1. `startup.py`: skip the network auto-join query entirely when a model was
|
||
explicitly requested (`model` set and not `"stub-model"`); path 3b
|
||
(`/v1/nodes/assign?model=…`) is the authoritative one there.
|
||
2. `downloader.py`: model-source downloads get their own timeout constant
|
||
(30 s socket timeout) instead of the 2 s peer-probe timeout. Peer shard
|
||
downloads keep 2 s — they run sequentially before the race, and a dead
|
||
peer must not hang startup for 30 s; the race is concurrent so a slow
|
||
source costs nothing.
|
||
3. `downloader.py`: progress + failure visibility for the race —
|
||
`_download_model_source` prints received bytes every 512 MB and prints
|
||
the exception when a source fails, so "downloads only from HF" can never
|
||
happen silently again.
|
||
4. Tracker `_handle_model_files_download`: catch
|
||
`BrokenPipeError`/`ConnectionResetError` around the tar stream and log a
|
||
single line instead of a traceback.
|
||
|
||
## Design revision (2026-07-06, after live retest)
|
||
|
||
The race is gone. User decision: **HuggingFace is used only when the model is
|
||
not available from a tracker/peer source, or when `--tracker-source-disabled`
|
||
is passed.** Sources are tried sequentially with progress + failure output;
|
||
HF (layer-filtered via the source file list, else the remote index) is the
|
||
fallback.
|
||
|
||
Second live finding: the node was assigned only layers 0–2 of 40 on a 79 GB
|
||
box. Cause: CPU-mode nodes still report the detected-but-unusable GPU's
|
||
`vram_mb` (RTX 4060 → 8192), and shard sizing used VRAM whenever it was > 0
|
||
(8 GB × 0.8 ≈ 6.5 GB ≈ 3 layers). Fixed on both sides: the node now sends
|
||
`assignment_vram_mb` (0 unless CUDA is actually usable) to `/v1/nodes/assign`,
|
||
and the tracker only trusts `vram_mb` when `device=cuda` (all three sizing
|
||
sites), falling back to `ram_mb`.
|
||
|
||
## Acceptance criteria
|
||
|
||
- [x] Node started with an explicit `--model` never queries
|
||
`/v1/network/assign` and never prints `auto-join unavailable`.
|
||
- [x] Tracker/peer model source is preferred outright; HF is contacted only
|
||
when no source is advertised, every source fails, or
|
||
`--tracker-source-disabled` is passed (flag on both CLI parsers, plumbed
|
||
through config and `run_startup`).
|
||
- [x] Tracker-source downloads print progress every 512 MB and print the
|
||
exception + URL on failure; nothing fails silently.
|
||
- [x] A ≥2 s read stall no longer aborts a tracker model-source download
|
||
(30 s socket timeout).
|
||
- [x] Client disconnect during `/v1/model-files/download` logs one line on
|
||
the tracker, no traceback.
|
||
- [x] CPU node with big RAM gets a RAM-sized shard: `/v1/nodes/assign` and
|
||
both `/v1/network/assign` sizing paths ignore VRAM unless `device=cuda`.
|
||
- [x] `pytest tests/test_node_startup.py tests/test_tracker_routing.py`
|
||
passes (139/140; the one failure is the pre-existing port-dependent
|
||
`test_mining_cli` case, present on clean master).
|
||
- [ ] Live two-machine retest: Windows node downloads only from tracker at
|
||
LAN speed and is assigned a RAM-sized shard.
|
||
|
||
## Round 3 (2026-07-06, after live retest showed mid-stream RST)
|
||
|
||
Live retest: RAM sizing worked (layers 0–36) and the failure finally printed —
|
||
`ConnectionResetError(10054)` ~70 s into the tar stream. Local reproduction
|
||
cleared the tracker: it streams the full 72 GB tar at ~900 MB/s, survives a
|
||
3-minute slow reader, and logs aborts in one line. The RST comes from the
|
||
network path (Windows laptop, likely WiFi + firewall/AV) — and a 72 GB
|
||
single-TCP-stream tar is inherently fragile there.
|
||
|
||
Fix: per-file downloads (design principle: nodes must be able to fetch any
|
||
missing shard or the complete model from the tracker alone — no hard HF
|
||
dependency):
|
||
|
||
- Tracker: `/v1/model-files/download?...&file=<rel>` streams one file with
|
||
`Content-Length` (rel must be in the requested shard/full set; traversal
|
||
rejected). `model_sources` now advertises `full_files` and a `file_sizes`
|
||
manifest.
|
||
- Node: `_download_source_files` fetches per file into
|
||
`<shard>.partial/`, retries each file 3×, verifies against
|
||
`Content-Length`, and reuses already-complete files (hardlink from the
|
||
existing shard) via the size manifest — so restarts and drops cost at most
|
||
one file. Tar stream remains the fallback for old trackers
|
||
(detected via Content-Type) and sources without a file list.
|
||
- `_full_model_sources` passes `full_files` through, so full-snapshot
|
||
downloads for the torch path get the same robustness.
|
||
|
||
Verified live against a local tracker: 14.7 GB shard in 7.6 s per-file;
|
||
re-run over a complete shard instant; corrupt + deleted file recovered in
|
||
1.5 s re-fetching only those two. 114 tests pass (node_startup +
|
||
tracker_routing).
|