61 lines
3.0 KiB
Markdown
61 lines
3.0 KiB
Markdown
# Qwen3.6-27B recommended deployment
|
|
|
|
## Goal
|
|
|
|
Offer the pinned `Qwen/Qwen3.6-27B` BF16 artifact as a recommended,
|
|
text-only chat model. A valid chat request proves model demand. When the
|
|
shared tracker pool has sufficient eligible capacity, the tracker deploys the
|
|
model rather than waiting for an operator to request a load.
|
|
|
|
## Catalog and artifacts
|
|
|
|
- Canonical artifact: `Qwen/Qwen3.6-27B` at
|
|
`6a9e13bd6fc8f0983b9b99948120bc37f49c13e9`.
|
|
- Tracker/peer artifact sources remain preferred over Hugging Face; this is a
|
|
system rule, not model-specific behavior.
|
|
- BF16 is canonical. Nodes may load BF16, INT8, or NF4 from the canonical
|
|
shard according to their declared capability.
|
|
- The model is text-only for this feature. Image inputs are rejected rather
|
|
than implicitly advertised as supported.
|
|
|
|
## Quantization contract
|
|
|
|
- Chat accepts an optional `quantization` field: `bfloat16`, `int8`, or
|
|
`nf4`. Omission means `bfloat16`.
|
|
- A request is a minimum-quality constraint: BF16 uses BF16 only; INT8 can
|
|
use INT8 or BF16; NF4 can use NF4, INT8, or BF16.
|
|
- A selectable variant requires complete end-to-end coverage using only
|
|
qualifying shards. Mixed qualifying precision is valid.
|
|
- The UI defaults to the highest complete coverage, lists unavailable variants
|
|
as non-selectable, and gives them a small coverage-vote control.
|
|
|
|
## Demand and placement
|
|
|
|
- The first valid model request queues initial tracker-managed placement when
|
|
sufficient pooled capability exists. Until complete coverage exists, return
|
|
retryable `503 model_loading` with coverage metadata.
|
|
- Demand and node supply are hive-wide; a leader makes each assignment and
|
|
broadcasts it to redundant trackers.
|
|
- A request for an unavailable quantization is retained as demand. Votes are
|
|
weaker when another Qwen quantization is already usable.
|
|
- If deployment replaces existing complete coverage, rolling demand for the
|
|
requested variant must exceed the displaced variant by more than 1.5x.
|
|
The multiplier and rolling window are tracker configuration.
|
|
- Managed replacements require at least three complete copies beforehand and
|
|
must leave two. Managed placement has a configurable cooldown.
|
|
|
|
## Node ownership
|
|
|
|
Reconciled with [ADR-0026](../../docs/adr/0026-node-assignment-ownership-and-managed-placement.md) and NCA (ADR-0023):
|
|
|
|
- A **startup-assigned** `(model, shard range, quantization)` from explicit `--model` or accepted bootstrap assign is **pinned** until the operator restarts.
|
|
- **Tracker-managed** assignments (this feature) use only **spare capacity** — model-less nodes or (future, US-048) unused shard slots — and are marked `managed: true`.
|
|
- The tracker may move or remove managed assignments under the safety policy below; it must not retarget a pinned serving assignment to satisfy demand.
|
|
- Every assignment, pinned or managed, must pass NCA `doctor` before becoming routable when admission is enabled.
|
|
|
|
## Pricing
|
|
|
|
Use exact-model online provider pricing. Preserve the last verified price if
|
|
the provider lookup fails; use a model-specific development fallback only when
|
|
there is no verified price.
|