62 lines
2.7 KiB
Markdown
62 lines
2.7 KiB
Markdown
# Qwen3.6-27B recommended deployment
|
|
|
|
## Goal
|
|
|
|
Offer the pinned `Qwen/Qwen3.6-27B` BF16 artifact as a recommended,
|
|
text-only chat model. A valid chat request proves model demand. When the
|
|
shared tracker pool has sufficient eligible capacity, the tracker deploys the
|
|
model rather than waiting for an operator to request a load.
|
|
|
|
## Catalog and artifacts
|
|
|
|
- Canonical artifact: `Qwen/Qwen3.6-27B` at
|
|
`6a9e13bd6fc8f0983b9b99948120bc37f49c13e9`.
|
|
- Tracker/peer artifact sources remain preferred over Hugging Face; this is a
|
|
system rule, not model-specific behavior.
|
|
- BF16 is canonical. Nodes may load BF16, INT8, or NF4 from the canonical
|
|
shard according to their declared capability.
|
|
- The model is text-only for this feature. Image inputs are rejected rather
|
|
than implicitly advertised as supported.
|
|
|
|
## Quantization contract
|
|
|
|
- Chat accepts an optional `quantization` field: `bfloat16`, `int8`, or
|
|
`nf4`. Omission means `bfloat16`.
|
|
- A request is a minimum-quality constraint: BF16 uses BF16 only; INT8 can
|
|
use INT8 or BF16; NF4 can use NF4, INT8, or BF16.
|
|
- A selectable variant requires complete end-to-end coverage using only
|
|
qualifying shards. Mixed qualifying precision is valid.
|
|
- The UI defaults to the highest complete coverage, lists unavailable variants
|
|
as non-selectable, and gives them a small coverage-vote control.
|
|
|
|
## Demand and placement
|
|
|
|
- The first valid model request queues initial tracker-managed placement when
|
|
sufficient pooled capability exists. Until complete coverage exists, return
|
|
retryable `503 model_loading` with coverage metadata.
|
|
- Demand and node supply are hive-wide; a leader makes each assignment and
|
|
broadcasts it to redundant trackers.
|
|
- A request for an unavailable quantization is retained as demand. Votes are
|
|
weaker when another Qwen quantization is already usable.
|
|
- If deployment replaces existing complete coverage, rolling demand for the
|
|
requested variant must exceed the displaced variant by more than 1.5x.
|
|
The multiplier and rolling window are tracker configuration.
|
|
- Managed replacements require at least three complete copies beforehand and
|
|
must leave two. Managed placement has a configurable cooldown.
|
|
|
|
## Node ownership
|
|
|
|
- A startup-assigned `(model, shard range, quantization)` is pinned and never
|
|
changed by the tracker.
|
|
- Spare capacity on a pinned node, and all capacity on a model-less node, is
|
|
available for tracker-managed assignments.
|
|
- Tracker-added assignments are explicitly marked managed and may be moved or
|
|
removed by the tracker under the safety policy. Runtime UI controls are a
|
|
later feature.
|
|
|
|
## Pricing
|
|
|
|
Use exact-model online provider pricing. Preserve the last verified price if
|
|
the provider lookup fails; use a model-specific development fallback only when
|
|
there is no verified price.
|