Dobromir Popov 607d49f5b0 fix: proper autoregressive inference with streaming support ...

Single-node mode now uses HF model.generate() instead of one-shot
decode_tail(), giving correct multi-token output with KV cache.

model_backend.py:
- generate_text(messages, max_new_tokens, temperature, top_p) — full
  autoregressive generation via model.generate() with chat template
- generate_text_streaming() — yields token strings via TextIteratorStreamer
- _encode_messages() — applies chat template (tokenize=False then tokenize),
  falls back to joining user messages; avoids BatchEncoding issues

torch_server.py:
- _handle_chat_completions: fast path when backend is head+tail — calls
  generate_text() or generate_text_streaming() directly instead of the
  single-token encode_prompt+decode_tail pipeline
- _stream_openai_response: new SSE streaming handler for token iterators
- Parses max_tokens, temperature, top_p from request body
- Distributed path (partial shards) unchanged

Verified: streaming and non-streaming both work with Qwen2.5-0.5B-Instruct.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

2026-06-29 18:46:51 +03:00

17 KiB

Raw Blame History

View Raw