Start background re-registration when the tracker is unreachable at startup so nodes do not stay permanently unregistered.
Co-authored-by: Cursor <cursoragent@cursor.com>
Unify --model and --model-id so catalog names use the tracker path, and allow --shard-start/--shard-end with --model instead of requiring --model-id.
Co-authored-by: Cursor <cursoragent@cursor.com>
- cli.py: change default --quantization from bfloat16 to int8; saves
~50% VRAM/RAM for new nodes that don't specify a quantization
- startup.py: display memory budget and GPU info in GB (e.g. 124.9 GB RAM)
instead of MB; show remaining headroom after full model load
- test_tracker_routing.py: add test_shard_heal_cycle_surviving_node_covers_dead_peers_gap
— end-to-end proof that:
1. tracker purges expired node A and queues LOAD_SHARD for node B
2. node B receives directive on next heartbeat
3. TorchNodeServer.apply_tracker_directives hot-swaps the backend
4. node B re-registers covering the full model; coverage gap closed
Test runs in <1s with monkeypatched _load_backend (no GPU needed)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Nodes running inside WSL2 or Docker have a virtual network interface whose IP
(172.16-31.x.x) is not reachable from physical machines on the LAN. The UDP
socket probe for outbound IP returns this virtual IP, which gets registered with
the tracker — causing downstream pipeline hops to time out with "urlopen error
timed out".
_warn_virtual_network_ip() now prints a clear multi-line warning at startup
when the auto-detected advertise IP falls in 172.16.0.0/12, including the
fix: --advertise-host <LAN-ip>.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
/v1/network/assign now scores models by (demand_rpm + 1) × (coverage_deficit + 0.01)
so high-traffic, under-covered models are preferred when assigning new nodes.
Response includes price_per_token: 0.0 (reserved for future pricing protocol).
--memory MB flag added to node CLI to override autodetected VRAM budget for
shard assignment without changing hardware detection for inference.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Node now sends cumulative stats in heartbeat body:
total_requests, failed_requests, queue_depth, uptime_seconds, status
Stats are tracked thread-safely in _TorchHTTPServer; buffered locally during
outage streak and flushed on next successful heartbeat.
Tracker stores stats on _NodeEntry (total_requests, failed_requests,
queue_depth, uptime_seconds, status, heartbeats_expected/received) and
returns new_assignment in heartbeat response when pending_new_assignment is set.
Node logs received new_assignment (hot-reload wired in US-026).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Default 30s replaces the hardcoded 5s. Wired through TorchNodeServer →
_TorchHTTPServer → _get_remaining_route. Available on both the legacy
`start` subcommand and the default wizard path.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Wrap wfile.write in _TrackerHandler._send_json with except BrokenPipeError
- Replace uuid4 node IDs with deterministic SHA-256 hash of endpoint+model+shards
so nodes keep the same ID on re-registration after tracker restart
- /v1/models now lists HF-repo models (not just preset models)
- /v1/coverage/{model} now resolves HF repos, not just preset names
- /v1/route response includes node_id alongside endpoint
- startup.py exposes tracker_node_id on node object and prints it in dashboard
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
socket.getfqdn() returns *.localdomain names that other machines on
the same LAN (especially cross-OS) cannot resolve via DNS. When the
node is bound to 0.0.0.0 and --advertise-host is not given, probe the
outbound IP by connecting a UDP socket toward the tracker — this picks
the correct interface IP without sending any data.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Previous logic caught 404 on heartbeat and re-registered, but the re-
registration failed silently if the tracker wasn't fully ready yet.
New approach: after any connection-refused streak, the heartbeat loop
switches to re-registration mode and keeps retrying until the tracker
accepts the registration (instead of sending heartbeats into a fresh
registry that doesn't know this node). Falls back to the 404 path for
the case where a node is purged without a full tracker restart.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When the tracker restarts, nodes' registrations are lost. The heartbeat loop
was catching the 404 and printing a warning but never re-registering, leaving
the node permanently invisible until manually restarted.
Fix: on HTTP 404 heartbeat response, the loop re-posts the original
registration payload to /v1/nodes/register and updates the node_id and
heartbeat URL for subsequent beats. This also handles tracker expiry races.
The register_payload is now passed into _start_heartbeat so the thread has
everything needed for re-registration without reaching back into run_startup.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Tracker /v1/route now resolves HF model nodes (by hf_repo or short name)
in addition to preset models, using the same greedy interval-cover logic.
This allows distributed inference routing across two nodes each holding
half the model.
Endpoint dedup: re-registering the same endpoint atomically replaces the
old entry so stale registrations don't accumulate across node restarts.
Purge logging: tracker now prints when a node expires due to missed
heartbeats so operators can see dead nodes being removed.
Timing fix: heartbeat timeout raised from 30s to 90s (3 missed beats);
node heartbeat interval lowered from 30s to 20s to maintain margin.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Tracker /v1/network/assign now accepts an optional `hf_repo` query param
to restrict assignment to a specific model, and returns `gap_found: bool`
so callers know whether they received a real gap vs a redundancy slot.
Node startup with --model-id (no explicit shard args) now queries the
tracker first for an uncovered gap for that model before defaulting to
full coverage (0..n-1). This means a second node with --model-id will
serve only the missing layers, not the whole model again.
Auto-join fallback (no --model-id) now prints why it fell through
instead of silently switching to stub-model.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Tracker now prints a line when a node registers and on every heartbeat
received. Node prints its assigned node_id after successful registration
and starts a daemon heartbeat thread (30s interval) that logs each send.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
