- cli.py: change default --quantization from bfloat16 to int8; saves
~50% VRAM/RAM for new nodes that don't specify a quantization
- startup.py: display memory budget and GPU info in GB (e.g. 124.9 GB RAM)
instead of MB; show remaining headroom after full model load
- test_tracker_routing.py: add test_shard_heal_cycle_surviving_node_covers_dead_peers_gap
— end-to-end proof that:
1. tracker purges expired node A and queues LOAD_SHARD for node B
2. node B receives directive on next heartbeat
3. TorchNodeServer.apply_tracker_directives hot-swaps the backend
4. node B re-registers covering the full model; coverage gap closed
Test runs in <1s with monkeypatched _load_backend (no GPU needed)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Replace stale "only this works" PowerShell comment with accurate relay
one-liner that works from behind NAT without --advertise-host
- Expand "Public tracker + WSS relay" into a full architecture section:
nginx proxy paths, start commands, relay hop sequence, Base64 encoding
- Add WSL2 two-node relay test scenario with curl examples
- Document startup output when relay connects (Relay connected line)
- Remove "Start the relay node" stub; fold into the main relay section
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Nodes behind NAT (5G, WSL2, home routers) can now participate in
distributed pipeline inference over the internet via the relay server.
- torch_server: add module-level _relay_hop() that opens a WebSocket
to relay.../rpc/{peer_id}, sends the binary activation with
body_base64 encoding, and returns (status, headers, body)
- torch_server: _get_remaining_route returns list[dict] (was list[tuple])
preserving relay_addr from injected X-Meshnet-Route header and
from /v1/route slow-path node info
- torch_server: _run_downstream_pipeline dispatches via _relay_hop
when hop has relay_addr; falls back to direct HTTP on relay error
- tracker server: downstream_hops dicts include relay_addr when node
has one registered, so head node knows how to reach each peer
- relay_bridge: binary bodies (bfloat16 activations) use body_base64;
response preserves all X-Meshnet-* headers
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Nodes running inside WSL2 or Docker have a virtual network interface whose IP
(172.16-31.x.x) is not reachable from physical machines on the LAN. The UDP
socket probe for outbound IP returns this virtual IP, which gets registered with
the tracker — causing downstream pipeline hops to time out with "urlopen error
timed out".
_warn_virtual_network_ip() now prints a clear multi-line warning at startup
when the auto-detected advertise IP falls in 172.16.0.0/12, including the
fix: --advertise-host <LAN-ip>.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
_select_route now prefers nodes with higher effective throughput
(benchmark_tokens_per_sec / (queue_depth + 1)) when multiple nodes
cover the same interval, breaking the tie after max-reach selection.
TorchNodeServer.route_timeout property added for external inspection.
Tests added:
- _select_route with overlapping shards (A:0-22, B:20-24) → correct X-Meshnet-Start-Layer protocol
- _select_route with gap → clear error message
- Fast node beats slow node when shards are equal
- Busy fast node vs idle slow node (queue_depth factor)
- Two-StubNodeServer integration via TrackerServer
- route_timeout property exposed on TorchNodeServer
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
/v1/network/assign now scores models by (demand_rpm + 1) × (coverage_deficit + 0.01)
so high-traffic, under-covered models are preferred when assigning new nodes.
Response includes price_per_token: 0.0 (reserved for future pricing protocol).
--memory MB flag added to node CLI to override autodetected VRAM budget for
shard assignment without changing hardware detection for inference.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds _RollingCounter and _StatsCollector to the tracker: three rolling windows
(hour=60×1min, day=24×1hr, month=30×1day) track request RPMs per model.
GET /v1/stats returns combined local + peer stats. POST /v1/stats/gossip lets
trackers push their local slice for additive merge in hive mode.
SQLite persistence via --stats-db flag; stats survive tracker restarts.
Records a stat for each proxied /v1/chat/completions request.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Coverage endpoint now skips node purge so dead nodes remain visible.
Each band in /v1/coverage/<model> includes a `nodes` array with per-node
health: alive status, last_seen_seconds_ago, heartbeat_success_rate,
inference_success_rate, queue_depth, uptime_seconds, and endpoint.
Updated tests to assert new band structure.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Node now sends cumulative stats in heartbeat body:
total_requests, failed_requests, queue_depth, uptime_seconds, status
Stats are tracked thread-safely in _TorchHTTPServer; buffered locally during
outage streak and flushed on next successful heartbeat.
Tracker stores stats on _NodeEntry (total_requests, failed_requests,
queue_depth, uptime_seconds, status, heartbeats_expected/received) and
returns new_assignment in heartbeat response when pending_new_assignment is set.
Node logs received new_assignment (hot-reload wired in US-026).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When _select_route picks two nodes with overlapping registrations (e.g.
A:0-22 and B:20-24), the tracker now injects start_layer per hop so B
executes only layers 23-24, not 20-24.
- model_backend: forward_bytes + _run_layers accept start_layer offset;
clamped to shard_start to prevent out-of-bounds indexing
- torch_server: _handle_binary_forward reads X-Meshnet-Start-Layer header;
_run_downstream_pipeline sends it per hop; route is now list[tuple[str,int]]
- server: proxy injects {endpoint, start_layer} objects in X-Meshnet-Route;
/v1/route response includes start_layer per node in the nodes list
- test: fake backends accept start_layer=None kwarg
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Default 30s replaces the hardcoded 5s. Wired through TorchNodeServer →
_TorchHTTPServer → _get_remaining_route. Available on both the legacy
`start` subcommand and the default wizard path.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Merges worktree-feat+us-016 into master. Combined both sets of _NodeEntry
fields: hf_repo/num_layers (HEAD) and relay_addr/cert_fingerprint/peer_id
(US-017). Kept HEAD's auto-shard tracker query and shard_label formatting.
US-016: mining-style startup CLI with live ASCII dashboard, hardware
detection, auto-shard range detection from model layer count, tracker
network-assign integration for gap-filling.
US-017: P2P gossip protocol, NAT-traversal relay node, TLS peer
authentication via cert fingerprint.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Wrap wfile.write in _TrackerHandler._send_json with except BrokenPipeError
- Replace uuid4 node IDs with deterministic SHA-256 hash of endpoint+model+shards
so nodes keep the same ID on re-registration after tracker restart
- /v1/models now lists HF-repo models (not just preset models)
- /v1/coverage/{model} now resolves HF repos, not just preset names
- /v1/route response includes node_id alongside endpoint
- startup.py exposes tracker_node_id on node object and prints it in dashboard
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When the tracker proxies /v1/chat/completions to a first-shard node it
already holds the full registry picture. It now resolves the downstream
route inline via _select_route, strips the proxied node, and sends the
result as X-Meshnet-Route header alongside the request body.
The first-shard node reads this header in _get_remaining_route and
returns it directly, skipping the second tracker HTTP call entirely.
Falls back to the tracker query transparently when the header is absent
(direct node-to-node calls or older tracker versions).
Reduces per-inference tracker round-trips from 2 to 1.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The tracker is now the single entrypoint for inference. Clients POST to
the tracker's /v1/chat/completions and it forwards to a live tracker-mode
(first-shard) node for the requested model, applying round-robin load
balancing across multiple first-shard nodes when available.
Streaming (text/event-stream) is relayed chunk-by-chunk. Non-streaming
responses are buffered and forwarded. BrokenPipeError on client disconnect
is silenced. Upstream errors relay the original HTTP status and body.
Also adds GET /v1/health → {"status": "ok"} on the tracker.
Usage:
curl -s http://tracker:8081/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"Qwen/Qwen2.5-0.5B-Instruct","messages":[...]}'
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
BrokenPipeError on wfile.write is harmless — the client disconnected
before the response arrived. Suppress it instead of printing a full
traceback to stderr.
Raise the downstream pipeline timeout from 10s to 120s so the Windows
node (or any caller) waits long enough for CPU-mode inference to
complete before giving up.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
socket.getfqdn() returns *.localdomain names that other machines on
the same LAN (especially cross-OS) cannot resolve via DNS. When the
node is bound to 0.0.0.0 and --advertise-host is not given, probe the
outbound IP by connecting a UDP socket toward the tracker — this picks
the correct interface IP without sending any data.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
