quantizations

QUICKSTART.md

@@ -13,6 +13,56 @@ support depends on the exact AMD GPU/APU, kernel, driver, and ROCm runtime.
 |------|-------------------|---------|-------|
 | Smoke tests, small splits | `Qwen/Qwen2.5-0.5B-Instruct` | same | 24 layers, ~1 GB BF16, no gating — default for new setups |
 | Alpha / production target | `qwen3.6-35b-a3b` | `unsloth/Qwen3.6-35B-A3B` | 40 layers, ~72 GB BF16, hybrid linear-attention MoE; aliases include `Qwen3.6-35B-A3B`, `Qwen/Qwen3.6-35B-A3B` |
+| Recommended dense model | `qwen3.6-27b` | `Qwen/Qwen3.6-27B` | 64 layers, ~56 GB BF16, text-only; canonical revision is pinned by the tracker |
+
+---
+
+## Models
+
+The tracker advertises recommended models through `GET /v1/models`; the chat
+dashboard shows the same catalog. `Qwen/Qwen3.6-27B` is recommended even before
+it has complete coverage, so users and operators can see that the network intends
+to serve it.
+
+```bash
+curl -s http://localhost:8080/v1/models | python -m json.tool
+```
+
+Each model reports its shard coverage and its selectable quantizations. A
+quantization is selectable only when the tracker can build a complete route from
+shards at that precision or higher:
+
+- `bfloat16` requires BF16 for every shard.
+- `int8` may combine INT8 and BF16 shards.
+- `nf4` may combine NF4, INT8, and BF16 shards.
+
+The chat UI chooses the highest complete precision by default. Uncovered variants
+remain visible as coverage requests: use the small **Vote for coverage** control
+to register demand without sending an unusable chat request.
+
+Clients can request a minimum precision explicitly. Omit `quantization` to
+request BF16:
+
+```bash
+curl http://localhost:8080/v1/chat/completions \
+  -H 'Content-Type: application/json' \
+  -d '{
+    "model": "qwen3.6-27b",
+    "quantization": "int8",
+    "messages": [{"role": "user", "content": "Explain sharded inference."}]
+  }'
+```
+
+The first valid request for a recommended model is demand proof. When the shared
+tracker pool has enough spare eligible capacity, it queues a tracker-managed
+initial deployment. Until a complete route exists, the request returns retryable
+`503 model_loading` rather than silently lowering precision.
+
+Nodes may serve BF16, INT8, or NF4 according to their declared capability. The
+official BF16 snapshot remains the canonical Qwen source; tracker/peer sources
+are preferred and Hugging Face is the fallback. A node's startup-selected
+`(model, shard range, quantization)` is pinned, while spare capacity can be used
+for tracker-managed work.
 
 ---