- cli.py: change default --quantization from bfloat16 to int8; saves
~50% VRAM/RAM for new nodes that don't specify a quantization
- startup.py: display memory budget and GPU info in GB (e.g. 124.9 GB RAM)
instead of MB; show remaining headroom after full model load
- test_tracker_routing.py: add test_shard_heal_cycle_surviving_node_covers_dead_peers_gap
— end-to-end proof that:
1. tracker purges expired node A and queues LOAD_SHARD for node B
2. node B receives directive on next heartbeat
3. TorchNodeServer.apply_tracker_directives hot-swaps the backend
4. node B re-registers covering the full model; coverage gap closed
Test runs in <1s with monkeypatched _load_backend (no GPU needed)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Nodes behind NAT (5G, WSL2, home routers) can now participate in
distributed pipeline inference over the internet via the relay server.
- torch_server: add module-level _relay_hop() that opens a WebSocket
to relay.../rpc/{peer_id}, sends the binary activation with
body_base64 encoding, and returns (status, headers, body)
- torch_server: _get_remaining_route returns list[dict] (was list[tuple])
preserving relay_addr from injected X-Meshnet-Route header and
from /v1/route slow-path node info
- torch_server: _run_downstream_pipeline dispatches via _relay_hop
when hop has relay_addr; falls back to direct HTTP on relay error
- tracker server: downstream_hops dicts include relay_addr when node
has one registered, so head node knows how to reach each peer
- relay_bridge: binary bodies (bfloat16 activations) use body_base64;
response preserves all X-Meshnet-* headers
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Nodes running inside WSL2 or Docker have a virtual network interface whose IP
(172.16-31.x.x) is not reachable from physical machines on the LAN. The UDP
socket probe for outbound IP returns this virtual IP, which gets registered with
the tracker — causing downstream pipeline hops to time out with "urlopen error
timed out".
_warn_virtual_network_ip() now prints a clear multi-line warning at startup
when the auto-detected advertise IP falls in 172.16.0.0/12, including the
fix: --advertise-host <LAN-ip>.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
_select_route now prefers nodes with higher effective throughput
(benchmark_tokens_per_sec / (queue_depth + 1)) when multiple nodes
cover the same interval, breaking the tie after max-reach selection.
TorchNodeServer.route_timeout property added for external inspection.
Tests added:
- _select_route with overlapping shards (A:0-22, B:20-24) → correct X-Meshnet-Start-Layer protocol
- _select_route with gap → clear error message
- Fast node beats slow node when shards are equal
- Busy fast node vs idle slow node (queue_depth factor)
- Two-StubNodeServer integration via TrackerServer
- route_timeout property exposed on TorchNodeServer
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
/v1/network/assign now scores models by (demand_rpm + 1) × (coverage_deficit + 0.01)
so high-traffic, under-covered models are preferred when assigning new nodes.
Response includes price_per_token: 0.0 (reserved for future pricing protocol).
--memory MB flag added to node CLI to override autodetected VRAM budget for
shard assignment without changing hardware detection for inference.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Node now sends cumulative stats in heartbeat body:
total_requests, failed_requests, queue_depth, uptime_seconds, status
Stats are tracked thread-safely in _TorchHTTPServer; buffered locally during
outage streak and flushed on next successful heartbeat.
Tracker stores stats on _NodeEntry (total_requests, failed_requests,
queue_depth, uptime_seconds, status, heartbeats_expected/received) and
returns new_assignment in heartbeat response when pending_new_assignment is set.
Node logs received new_assignment (hot-reload wired in US-026).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When _select_route picks two nodes with overlapping registrations (e.g.
A:0-22 and B:20-24), the tracker now injects start_layer per hop so B
executes only layers 23-24, not 20-24.
- model_backend: forward_bytes + _run_layers accept start_layer offset;
clamped to shard_start to prevent out-of-bounds indexing
- torch_server: _handle_binary_forward reads X-Meshnet-Start-Layer header;
_run_downstream_pipeline sends it per hop; route is now list[tuple[str,int]]
- server: proxy injects {endpoint, start_layer} objects in X-Meshnet-Route;
/v1/route response includes start_layer per node in the nodes list
- test: fake backends accept start_layer=None kwarg
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Default 30s replaces the hardcoded 5s. Wired through TorchNodeServer →
_TorchHTTPServer → _get_remaining_route. Available on both the legacy
`start` subcommand and the default wizard path.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Wrap wfile.write in _TrackerHandler._send_json with except BrokenPipeError
- Replace uuid4 node IDs with deterministic SHA-256 hash of endpoint+model+shards
so nodes keep the same ID on re-registration after tracker restart
- /v1/models now lists HF-repo models (not just preset models)
- /v1/coverage/{model} now resolves HF repos, not just preset names
- /v1/route response includes node_id alongside endpoint
- startup.py exposes tracker_node_id on node object and prints it in dashboard
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When the tracker proxies /v1/chat/completions to a first-shard node it
already holds the full registry picture. It now resolves the downstream
route inline via _select_route, strips the proxied node, and sends the
result as X-Meshnet-Route header alongside the request body.
The first-shard node reads this header in _get_remaining_route and
returns it directly, skipping the second tracker HTTP call entirely.
Falls back to the tracker query transparently when the header is absent
(direct node-to-node calls or older tracker versions).
Reduces per-inference tracker round-trips from 2 to 1.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
BrokenPipeError on wfile.write is harmless — the client disconnected
before the response arrived. Suppress it instead of printing a full
traceback to stderr.
Raise the downstream pipeline timeout from 10s to 120s so the Windows
node (or any caller) waits long enough for CPU-mode inference to
complete before giving up.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
socket.getfqdn() returns *.localdomain names that other machines on
the same LAN (especially cross-OS) cannot resolve via DNS. When the
node is bound to 0.0.0.0 and --advertise-host is not given, probe the
outbound IP by connecting a UDP socket toward the tracker — this picks
the correct interface IP without sending any data.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Previous logic caught 404 on heartbeat and re-registered, but the re-
registration failed silently if the tracker wasn't fully ready yet.
New approach: after any connection-refused streak, the heartbeat loop
switches to re-registration mode and keeps retrying until the tracker
accepts the registration (instead of sending heartbeats into a fresh
registry that doesn't know this node). Falls back to the 404 path for
the case where a node is purged without a full tracker restart.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When the tracker restarts, nodes' registrations are lost. The heartbeat loop
was catching the 404 and printing a warning but never re-registering, leaving
the node permanently invisible until manually restarted.
Fix: on HTTP 404 heartbeat response, the loop re-posts the original
registration payload to /v1/nodes/register and updates the node_id and
heartbeat URL for subsequent beats. This also handles tracker expiry races.
The register_payload is now passed into _start_heartbeat so the thread has
everything needed for re-registration without reaching back into run_startup.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Route lookup was using the client-provided model name ("qwen2.5-0.5b") but
the tracker registers nodes under their full hf_repo ("Qwen/Qwen2.5-0.5B-Instruct").
This caused a 404 on /v1/route and the non-tail node fell back to the
"no downstream route available" error message.
Fix: _get_remaining_route now uses server.backend.model_id (the actual hf_repo)
for the tracker query. Skips self by port matching rather than blind route[0] drop.
Also prints a warning when route lookup fails so the cause is visible.
Distributed generation was also only producing 1 token (single greedy argmax
in decode_tail). Replaced with an autoregressive loop: head node encodes the
growing sequence and forwards to the downstream shard each step, collecting
one token per iteration up to max_tokens or EOS.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Tracker /v1/route now resolves HF model nodes (by hf_repo or short name)
in addition to preset models, using the same greedy interval-cover logic.
This allows distributed inference routing across two nodes each holding
half the model.
Endpoint dedup: re-registering the same endpoint atomically replaces the
old entry so stale registrations don't accumulate across node restarts.
Purge logging: tracker now prints when a node expires due to missed
heartbeats so operators can see dead nodes being removed.
Timing fix: heartbeat timeout raised from 30s to 90s (3 missed beats);
node heartbeat interval lowered from 30s to 20s to maintain margin.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Tracker /v1/network/assign now accepts an optional `hf_repo` query param
to restrict assignment to a specific model, and returns `gap_found: bool`
so callers know whether they received a real gap vs a redundancy slot.
Node startup with --model-id (no explicit shard args) now queries the
tracker first for an uncovered gap for that model before defaulting to
full coverage (0..n-1). This means a second node with --model-id will
serve only the missing layers, not the whole model again.
Auto-join fallback (no --model-id) now prints why it fell through
instead of silently switching to stub-model.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Tracker now prints a line when a node registers and on every heartbeat
received. Node prints its assigned node_id after successful registration
and starts a daemon heartbeat thread (30s interval) that logs each send.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Tracker:
- _NodeEntry gains hf_repo + num_layers fields (parsed from register body)
- GET /v1/network/assign — finds the biggest uncovered shard gap across
registered HF-model nodes; returns {hf_repo, shard_start, shard_end, num_layers}
- Returns 503 when no HF-model nodes are registered yet
Node startup:
- When model_id is set: registers with tracker including hf_repo + num_layers
so other nodes can auto-join this model
- When model_id is empty/None: queries /v1/network/assign, gets assigned the
missing layers, loads TorchNodeServer with the assigned shard automatically
- Fixes empty-string model_id leaking from DEFAULTS (treats "" same as None)
Usage: `meshnet-node start --tracker http://localhost:8080 --quantization bfloat16`
Node discovers what to serve and joins the network without any model flags.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Single-node mode now uses HF model.generate() instead of one-shot
decode_tail(), giving correct multi-token output with KV cache.
model_backend.py:
- generate_text(messages, max_new_tokens, temperature, top_p) — full
autoregressive generation via model.generate() with chat template
- generate_text_streaming() — yields token strings via TextIteratorStreamer
- _encode_messages() — applies chat template (tokenize=False then tokenize),
falls back to joining user messages; avoids BatchEncoding issues
torch_server.py:
- _handle_chat_completions: fast path when backend is head+tail — calls
generate_text() or generate_text_streaming() directly instead of the
single-token encode_prompt+decode_tail pipeline
- _stream_openai_response: new SSE streaming handler for token iterators
- Parses max_tokens, temperature, top_p from request body
- Distributed path (partial shards) unchanged
Verified: streaming and non-streaming both work with Qwen2.5-0.5B-Instruct.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
model_backend.py was using Python-style exclusive end (layers[start:end])
while all callers (CLI, tests, QUICKSTART) use inclusive 0-based indexing.
Result: 24-layer model with shard_end=23 ran only 23 layers and never
set is_tail=True, so decode_tail() was never called and responses were empty.
- is_tail: == total_layers → >= total_layers - 1
- _run_layers: layers[start:end] → layers[start:end+1]
- Validation: > total_layers → >= total_layers (was also wrong)
Inference confirmed: Qwen2.5-0.5B-Instruct now returns real LLM output.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Tracker: add GET /v1/tracker-nodes/<model> returning nodes registered
with tracker_mode=true whose shard_start matches the model's first layer
- Node: StubNodeServer and TorchNodeServer accept tracker_mode/tracker_url;
when tracker_mode=True (or auto-detected via shard_start==0 for Torch),
/v1/chat/completions is served alongside /forward
- TorchNodeServer: full pipeline implementation — encode_prompt → route
selection via tracker → binary forward through remaining hops → decode
- Gateway: _handle_chat_completions checks _get_tracker_nodes() first and
proxies round-robin to tracker-nodes; falls back to existing direct
pipeline when none found (preserves all US-005 backward compat)
- CLI: --tracker-mode and --tracker-url flags added to meshnet-node start
- Test: two stub tracker-nodes + two mid-shard nodes for gpt2; 10 requests;
round-robin 5/5 split verified; all OpenAI-format responses validated
- All 78 tests pass
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
