Route lookup was using the client-provided model name ("qwen2.5-0.5b") but
the tracker registers nodes under their full hf_repo ("Qwen/Qwen2.5-0.5B-Instruct").
This caused a 404 on /v1/route and the non-tail node fell back to the
"no downstream route available" error message.
Fix: _get_remaining_route now uses server.backend.model_id (the actual hf_repo)
for the tracker query. Skips self by port matching rather than blind route[0] drop.
Also prints a warning when route lookup fails so the cause is visible.
Distributed generation was also only producing 1 token (single greedy argmax
in decode_tail). Replaced with an autoregressive loop: head node encodes the
growing sequence and forwards to the downstream shard each step, collecting
one token per iteration up to max_tokens or EOS.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Tracker /v1/route now resolves HF model nodes (by hf_repo or short name)
in addition to preset models, using the same greedy interval-cover logic.
This allows distributed inference routing across two nodes each holding
half the model.
Endpoint dedup: re-registering the same endpoint atomically replaces the
old entry so stale registrations don't accumulate across node restarts.
Purge logging: tracker now prints when a node expires due to missed
heartbeats so operators can see dead nodes being removed.
Timing fix: heartbeat timeout raised from 30s to 90s (3 missed beats);
node heartbeat interval lowered from 30s to 20s to maintain margin.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Tracker /v1/network/assign now accepts an optional `hf_repo` query param
to restrict assignment to a specific model, and returns `gap_found: bool`
so callers know whether they received a real gap vs a redundancy slot.
Node startup with --model-id (no explicit shard args) now queries the
tracker first for an uncovered gap for that model before defaulting to
full coverage (0..n-1). This means a second node with --model-id will
serve only the missing layers, not the whole model again.
Auto-join fallback (no --model-id) now prints why it fell through
instead of silently switching to stub-model.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Tracker now prints a line when a node registers and on every heartbeat
received. Node prints its assigned node_id after successful registration
and starts a daemon heartbeat thread (30s interval) that logs each send.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Tracker:
- _NodeEntry gains hf_repo + num_layers fields (parsed from register body)
- GET /v1/network/assign — finds the biggest uncovered shard gap across
registered HF-model nodes; returns {hf_repo, shard_start, shard_end, num_layers}
- Returns 503 when no HF-model nodes are registered yet
Node startup:
- When model_id is set: registers with tracker including hf_repo + num_layers
so other nodes can auto-join this model
- When model_id is empty/None: queries /v1/network/assign, gets assigned the
missing layers, loads TorchNodeServer with the assigned shard automatically
- Fixes empty-string model_id leaking from DEFAULTS (treats "" same as None)
Usage: `meshnet-node start --tracker http://localhost:8080 --quantization bfloat16`
Node discovers what to serve and joins the network without any model flags.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Single-node mode now uses HF model.generate() instead of one-shot
decode_tail(), giving correct multi-token output with KV cache.
model_backend.py:
- generate_text(messages, max_new_tokens, temperature, top_p) — full
autoregressive generation via model.generate() with chat template
- generate_text_streaming() — yields token strings via TextIteratorStreamer
- _encode_messages() — applies chat template (tokenize=False then tokenize),
falls back to joining user messages; avoids BatchEncoding issues
torch_server.py:
- _handle_chat_completions: fast path when backend is head+tail — calls
generate_text() or generate_text_streaming() directly instead of the
single-token encode_prompt+decode_tail pipeline
- _stream_openai_response: new SSE streaming handler for token iterators
- Parses max_tokens, temperature, top_p from request body
- Distributed path (partial shards) unchanged
Verified: streaming and non-streaming both work with Qwen2.5-0.5B-Instruct.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
model_backend.py was using Python-style exclusive end (layers[start:end])
while all callers (CLI, tests, QUICKSTART) use inclusive 0-based indexing.
Result: 24-layer model with shard_end=23 ran only 23 layers and never
set is_tail=True, so decode_tail() was never called and responses were empty.
- is_tail: == total_layers → >= total_layers - 1
- _run_layers: layers[start:end] → layers[start:end+1]
- Validation: > total_layers → >= total_layers (was also wrong)
Inference confirmed: Qwen2.5-0.5B-Instruct now returns real LLM output.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Layer count is now fetched from the curated catalog (zero network calls
for known models) or via AutoConfig.from_pretrained() (~1 KB config.json
only) when model_id is given without --shard-start/--shard-end.
- model_catalog: add detect_num_layers(), two small Qwen models at top
- startup: _detect_num_layers() helper; shard range auto-derived
- wizard: show detected layer count for custom HF repos
- tests: 3 new tests for auto-shard; fix catalog-order assumptions
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- `meshnet-node` with no args runs interactive setup wizard on first run,
then starts directly on subsequent runs using saved config
- Wizard auto-detects all GPUs/VRAM, shows curated model list with per-quant
VRAM requirements, marks models that exceed available VRAM as incompatible,
offers HuggingFace Hub browse as escape hatch
- Persistent config saved to ~/.config/meshnet/config.json (0o600)
- Live rich dashboard (tokens/sec EMA, VRAM, requests, peers, uptime) with
automatic plain-text fallback when stdout is not a TTY (WSL2/SSH/CI)
- All wizard values overridable via CLI flags; --reset-config re-runs wizard
- `meshnet-node models` lists curated models; `--browse` fetches HF Hub top-20
- `meshnet-node config` prints saved config
- `meshnet-node start ...` preserved for backward compatibility
- 19 new tests; 97 passed, 1 skipped (no regressions)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Tracker: add GET /v1/tracker-nodes/<model> returning nodes registered
with tracker_mode=true whose shard_start matches the model's first layer
- Node: StubNodeServer and TorchNodeServer accept tracker_mode/tracker_url;
when tracker_mode=True (or auto-detected via shard_start==0 for Torch),
/v1/chat/completions is served alongside /forward
- TorchNodeServer: full pipeline implementation — encode_prompt → route
selection via tracker → binary forward through remaining hops → decode
- Gateway: _handle_chat_completions checks _get_tracker_nodes() first and
proxies round-robin to tracker-nodes; falls back to existing direct
pipeline when none found (preserves all US-005 backward compat)
- CLI: --tracker-mode and --tracker-url flags added to meshnet-node start
- Test: two stub tracker-nodes + two mid-shard nodes for gpt2; 10 requests;
round-robin 5/5 split verified; all OpenAI-format responses validated
- All 78 tests pass
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Recovered from interrupted Codex session (ended 13:02, changes uncommitted).
All 74 tests pass.
- Tracker accepts vram_bytes, ram_bytes, quantizations[], benchmark_tokens_per_sec
in node registration payload
- GET /v1/coverage/<model_preset> returns [{start_layer, end_layer, node_count}]
- Coverage-first bin-packing: fills gaps before adding redundancy
- Speed-weighted assignment: faster nodes get wider shard ranges
- LOAD_SHARD/DROP_SHARD rebalance directives delivered via heartbeat responses
- Model is unroutable when any layer range has node_count=0
- model_presets.json config for bytes_per_layer at each quantization level
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- GET /v1/health, GET /v1/models, POST /v1/chat/completions (streaming + non-streaming)
- OpenAI SDK, LangChain ChatOpenAI, and SSE streaming integration tests
- Tracker-backed GET /v1/models endpoint
- OpenAI-format errors for unavailable model (503) and pipeline failures
- Malformed JSON body handled with 400 instead of crash
- Test deps (openai, langchain-openai) declared in root pyproject dev extras
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
