5-th DL fix
This commit is contained in:
@@ -1,4 +1,4 @@
|
||||
# US-047 — Model-source race: visibility, sane timeouts, quiet tracker aborts
|
||||
# US-047 — Tracker-first model downloads: visibility, sane timeouts, RAM-based sizing
|
||||
|
||||
Status: in progress
|
||||
Priority: High (follow-up to US-044/US-046; blocks usable LAN downloads)
|
||||
@@ -45,14 +45,40 @@ http://192.168.0.179:8080 --model Qwen3.6-35B-A3B`):
|
||||
`BrokenPipeError`/`ConnectionResetError` around the tar stream and log a
|
||||
single line instead of a traceback.
|
||||
|
||||
## Design revision (2026-07-06, after live retest)
|
||||
|
||||
The race is gone. User decision: **HuggingFace is used only when the model is
|
||||
not available from a tracker/peer source, or when `--tracker-source-disabled`
|
||||
is passed.** Sources are tried sequentially with progress + failure output;
|
||||
HF (layer-filtered via the source file list, else the remote index) is the
|
||||
fallback.
|
||||
|
||||
Second live finding: the node was assigned only layers 0–2 of 40 on a 79 GB
|
||||
box. Cause: CPU-mode nodes still report the detected-but-unusable GPU's
|
||||
`vram_mb` (RTX 4060 → 8192), and shard sizing used VRAM whenever it was > 0
|
||||
(8 GB × 0.8 ≈ 6.5 GB ≈ 3 layers). Fixed on both sides: the node now sends
|
||||
`assignment_vram_mb` (0 unless CUDA is actually usable) to `/v1/nodes/assign`,
|
||||
and the tracker only trusts `vram_mb` when `device=cuda` (all three sizing
|
||||
sites), falling back to `ram_mb`.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- [ ] Node started with an explicit `--model` never queries
|
||||
- [x] Node started with an explicit `--model` never queries
|
||||
`/v1/network/assign` and never prints `auto-join unavailable`.
|
||||
- [ ] During the race, tracker-source progress lines appear alongside HF
|
||||
tqdm output; a failing tracker source prints its exception.
|
||||
- [ ] A ≥2 s read stall no longer aborts a tracker model-source download
|
||||
- [x] Tracker/peer model source is preferred outright; HF is contacted only
|
||||
when no source is advertised, every source fails, or
|
||||
`--tracker-source-disabled` is passed (flag on both CLI parsers, plumbed
|
||||
through config and `run_startup`).
|
||||
- [x] Tracker-source downloads print progress every 512 MB and print the
|
||||
exception + URL on failure; nothing fails silently.
|
||||
- [x] A ≥2 s read stall no longer aborts a tracker model-source download
|
||||
(30 s socket timeout).
|
||||
- [ ] Client disconnect during `/v1/model-files/download` logs one line on
|
||||
- [x] Client disconnect during `/v1/model-files/download` logs one line on
|
||||
the tracker, no traceback.
|
||||
- [ ] `python -m pytest` passes from repo root.
|
||||
- [x] CPU node with big RAM gets a RAM-sized shard: `/v1/nodes/assign` and
|
||||
both `/v1/network/assign` sizing paths ignore VRAM unless `device=cuda`.
|
||||
- [x] `pytest tests/test_node_startup.py tests/test_tracker_routing.py`
|
||||
passes (139/140; the one failure is the pre-existing port-dependent
|
||||
`test_mining_cli` case, present on clean master).
|
||||
- [ ] Live two-machine retest: Windows node downloads only from tracker at
|
||||
LAN speed and is assigned a RAM-sized shard.
|
||||
|
||||
Reference in New Issue
Block a user