Single-node mode now uses HF model.generate() instead of one-shot
decode_tail(), giving correct multi-token output with KV cache.
model_backend.py:
- generate_text(messages, max_new_tokens, temperature, top_p) — full
autoregressive generation via model.generate() with chat template
- generate_text_streaming() — yields token strings via TextIteratorStreamer
- _encode_messages() — applies chat template (tokenize=False then tokenize),
falls back to joining user messages; avoids BatchEncoding issues
torch_server.py:
- _handle_chat_completions: fast path when backend is head+tail — calls
generate_text() or generate_text_streaming() directly instead of the
single-token encode_prompt+decode_tail pipeline
- _stream_openai_response: new SSE streaming handler for token iterators
- Parses max_tokens, temperature, top_p from request body
- Distributed path (partial shards) unchanged
Verified: streaming and non-streaming both work with Qwen2.5-0.5B-Instruct.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
model_backend.py was using Python-style exclusive end (layers[start:end])
while all callers (CLI, tests, QUICKSTART) use inclusive 0-based indexing.
Result: 24-layer model with shard_end=23 ran only 23 layers and never
set is_tail=True, so decode_tail() was never called and responses were empty.
- is_tail: == total_layers → >= total_layers - 1
- _run_layers: layers[start:end] → layers[start:end+1]
- Validation: > total_layers → >= total_layers (was also wrong)
Inference confirmed: Qwen2.5-0.5B-Instruct now returns real LLM output.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
