quantizations
This commit is contained in:
61
.scratch/qwen3.6-27b-demand-placement/PRD.md
Normal file
61
.scratch/qwen3.6-27b-demand-placement/PRD.md
Normal file
@@ -0,0 +1,61 @@
|
|||||||
|
# Qwen3.6-27B recommended deployment
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
|
||||||
|
Offer the pinned `Qwen/Qwen3.6-27B` BF16 artifact as a recommended,
|
||||||
|
text-only chat model. A valid chat request proves model demand. When the
|
||||||
|
shared tracker pool has sufficient eligible capacity, the tracker deploys the
|
||||||
|
model rather than waiting for an operator to request a load.
|
||||||
|
|
||||||
|
## Catalog and artifacts
|
||||||
|
|
||||||
|
- Canonical artifact: `Qwen/Qwen3.6-27B` at
|
||||||
|
`6a9e13bd6fc8f0983b9b99948120bc37f49c13e9`.
|
||||||
|
- Tracker/peer artifact sources remain preferred over Hugging Face; this is a
|
||||||
|
system rule, not model-specific behavior.
|
||||||
|
- BF16 is canonical. Nodes may load BF16, INT8, or NF4 from the canonical
|
||||||
|
shard according to their declared capability.
|
||||||
|
- The model is text-only for this feature. Image inputs are rejected rather
|
||||||
|
than implicitly advertised as supported.
|
||||||
|
|
||||||
|
## Quantization contract
|
||||||
|
|
||||||
|
- Chat accepts an optional `quantization` field: `bfloat16`, `int8`, or
|
||||||
|
`nf4`. Omission means `bfloat16`.
|
||||||
|
- A request is a minimum-quality constraint: BF16 uses BF16 only; INT8 can
|
||||||
|
use INT8 or BF16; NF4 can use NF4, INT8, or BF16.
|
||||||
|
- A selectable variant requires complete end-to-end coverage using only
|
||||||
|
qualifying shards. Mixed qualifying precision is valid.
|
||||||
|
- The UI defaults to the highest complete coverage, lists unavailable variants
|
||||||
|
as non-selectable, and gives them a small coverage-vote control.
|
||||||
|
|
||||||
|
## Demand and placement
|
||||||
|
|
||||||
|
- The first valid model request queues initial tracker-managed placement when
|
||||||
|
sufficient pooled capability exists. Until complete coverage exists, return
|
||||||
|
retryable `503 model_loading` with coverage metadata.
|
||||||
|
- Demand and node supply are hive-wide; a leader makes each assignment and
|
||||||
|
broadcasts it to redundant trackers.
|
||||||
|
- A request for an unavailable quantization is retained as demand. Votes are
|
||||||
|
weaker when another Qwen quantization is already usable.
|
||||||
|
- If deployment replaces existing complete coverage, rolling demand for the
|
||||||
|
requested variant must exceed the displaced variant by more than 1.5x.
|
||||||
|
The multiplier and rolling window are tracker configuration.
|
||||||
|
- Managed replacements require at least three complete copies beforehand and
|
||||||
|
must leave two. Managed placement has a configurable cooldown.
|
||||||
|
|
||||||
|
## Node ownership
|
||||||
|
|
||||||
|
- A startup-assigned `(model, shard range, quantization)` is pinned and never
|
||||||
|
changed by the tracker.
|
||||||
|
- Spare capacity on a pinned node, and all capacity on a model-less node, is
|
||||||
|
available for tracker-managed assignments.
|
||||||
|
- Tracker-added assignments are explicitly marked managed and may be moved or
|
||||||
|
removed by the tracker under the safety policy. Runtime UI controls are a
|
||||||
|
later feature.
|
||||||
|
|
||||||
|
## Pricing
|
||||||
|
|
||||||
|
Use exact-model online provider pricing. Preserve the last verified price if
|
||||||
|
the provider lookup fails; use a model-specific development fallback only when
|
||||||
|
there is no verified price.
|
||||||
@@ -13,6 +13,56 @@ support depends on the exact AMD GPU/APU, kernel, driver, and ROCm runtime.
|
|||||||
|------|-------------------|---------|-------|
|
|------|-------------------|---------|-------|
|
||||||
| Smoke tests, small splits | `Qwen/Qwen2.5-0.5B-Instruct` | same | 24 layers, ~1 GB BF16, no gating — default for new setups |
|
| Smoke tests, small splits | `Qwen/Qwen2.5-0.5B-Instruct` | same | 24 layers, ~1 GB BF16, no gating — default for new setups |
|
||||||
| Alpha / production target | `qwen3.6-35b-a3b` | `unsloth/Qwen3.6-35B-A3B` | 40 layers, ~72 GB BF16, hybrid linear-attention MoE; aliases include `Qwen3.6-35B-A3B`, `Qwen/Qwen3.6-35B-A3B` |
|
| Alpha / production target | `qwen3.6-35b-a3b` | `unsloth/Qwen3.6-35B-A3B` | 40 layers, ~72 GB BF16, hybrid linear-attention MoE; aliases include `Qwen3.6-35B-A3B`, `Qwen/Qwen3.6-35B-A3B` |
|
||||||
|
| Recommended dense model | `qwen3.6-27b` | `Qwen/Qwen3.6-27B` | 64 layers, ~56 GB BF16, text-only; canonical revision is pinned by the tracker |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Models
|
||||||
|
|
||||||
|
The tracker advertises recommended models through `GET /v1/models`; the chat
|
||||||
|
dashboard shows the same catalog. `Qwen/Qwen3.6-27B` is recommended even before
|
||||||
|
it has complete coverage, so users and operators can see that the network intends
|
||||||
|
to serve it.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s http://localhost:8080/v1/models | python -m json.tool
|
||||||
|
```
|
||||||
|
|
||||||
|
Each model reports its shard coverage and its selectable quantizations. A
|
||||||
|
quantization is selectable only when the tracker can build a complete route from
|
||||||
|
shards at that precision or higher:
|
||||||
|
|
||||||
|
- `bfloat16` requires BF16 for every shard.
|
||||||
|
- `int8` may combine INT8 and BF16 shards.
|
||||||
|
- `nf4` may combine NF4, INT8, and BF16 shards.
|
||||||
|
|
||||||
|
The chat UI chooses the highest complete precision by default. Uncovered variants
|
||||||
|
remain visible as coverage requests: use the small **Vote for coverage** control
|
||||||
|
to register demand without sending an unusable chat request.
|
||||||
|
|
||||||
|
Clients can request a minimum precision explicitly. Omit `quantization` to
|
||||||
|
request BF16:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl http://localhost:8080/v1/chat/completions \
|
||||||
|
-H 'Content-Type: application/json' \
|
||||||
|
-d '{
|
||||||
|
"model": "qwen3.6-27b",
|
||||||
|
"quantization": "int8",
|
||||||
|
"messages": [{"role": "user", "content": "Explain sharded inference."}]
|
||||||
|
}'
|
||||||
|
```
|
||||||
|
|
||||||
|
The first valid request for a recommended model is demand proof. When the shared
|
||||||
|
tracker pool has enough spare eligible capacity, it queues a tracker-managed
|
||||||
|
initial deployment. Until a complete route exists, the request returns retryable
|
||||||
|
`503 model_loading` rather than silently lowering precision.
|
||||||
|
|
||||||
|
Nodes may serve BF16, INT8, or NF4 according to their declared capability. The
|
||||||
|
official BF16 snapshot remains the canonical Qwen source; tracker/peer sources
|
||||||
|
are preferred and Hugging Face is the fallback. A node's startup-selected
|
||||||
|
`(model, shard range, quantization)` is pinned, while spare capacity can be used
|
||||||
|
for tracker-managed work.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
@@ -22,14 +22,21 @@
|
|||||||
--tracker http://192.168.0.179:8081 `
|
--tracker http://192.168.0.179:8081 `
|
||||||
--model Qwen/Qwen2.5-0.5B-Instruct `
|
--model Qwen/Qwen2.5-0.5B-Instruct `
|
||||||
--advertise-host 192.168.0.20
|
--advertise-host 192.168.0.20
|
||||||
|
# trackers:
|
||||||
|
https://meshnet.2.d-popov.com
|
||||||
|
https://ai.neuron.d-popov.com
|
||||||
|
|
||||||
|
# Models
|
||||||
|
qwen3.6-35b-a3b Qwen/Qwen2.5-0.5B-Instruct Qwen3.6-27B
|
||||||
|
|
||||||
qwen3.6-35b-a3b Qwen/Qwen2.5-0.5B-Instruct
|
|
||||||
# linux
|
# linux
|
||||||
HF_HOME=/run/media/popov/d/DEV/models .venv/bin/meshnet-node start --model-id Qwen/Qwen2.5-0.5B-Instruct --shard-start 0 --shard-end 21 --quantization bfloat16 --tracker http://localhost:8081
|
HF_HOME=/run/media/popov/d/DEV/models .venv/bin/meshnet-node start --model-id Qwen/Qwen2.5-0.5B-Instruct --shard-start 0 --shard-end 21 --quantization bfloat16 --tracker http://localhost:8081
|
||||||
HF_HOME=/run/media/popov/d/DEV/models .venv-rocm/bin/meshnet-node start --tracker https://meshnet.2.d-popov.com --model qwen3.6-35b-a3b --shard-start 10
|
HF_HOME=/run/media/popov/d/DEV/models .venv-rocm/bin/meshnet-node start --tracker https://meshnet.2.d-popov.com --model qwen3.6-35b-a3b --shard-start 10
|
||||||
meshnet-node start --tracker http://192.168.0.179:8080 --model Qwen/Qwen2.5-0.5B-Instruct --shard-start 0 --shard-end 20
|
meshnet-node start --tracker http://192.168.0.179:8080 --model Qwen/Qwen2.5-0.5B-Instruct --shard-start 0 --shard-end 20
|
||||||
|
|
||||||
.venv-rocm/bin/meshnet-node start --tracker https://meshnet.2.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct --shard-start 10
|
.venv-rocm/bin/meshnet-node start --tracker https://meshnet.2.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct --shard-start 10
|
||||||
|
.venv-rocm/bin/meshnet-node start --tracker https://meshnet.2.d-popov.com --model Qwen3.6-27B
|
||||||
|
|
||||||
meshnet-node start --tracker https://meshnet.2.d-popov.com --model qwen3.6-35b-a3b --cpu
|
meshnet-node start --tracker https://meshnet.2.d-popov.com --model qwen3.6-35b-a3b --cpu
|
||||||
|
|
||||||
meshnet-node start --tracker https://meshnet.2.d-popov.com --model qwen3.6-35b-a3b --shard-start 0 --shard-end 21 --node-name gpu-head
|
meshnet-node start --tracker https://meshnet.2.d-popov.com --model qwen3.6-35b-a3b --shard-start 0 --shard-end 21 --node-name gpu-head
|
||||||
|
|||||||
@@ -264,6 +264,8 @@
|
|||||||
<div class="chat-toolbar">
|
<div class="chat-toolbar">
|
||||||
<label>Model
|
<label>Model
|
||||||
<select id="chat-model" onchange="selectChatModel(this.value)"></select>
|
<select id="chat-model" onchange="selectChatModel(this.value)"></select>
|
||||||
|
<select id="chat-quantization" onchange="selectChatQuantization(this.value)"></select>
|
||||||
|
<button type="button" id="vote-coverage" onclick="voteForCoverage()" style="display:none">Vote for coverage</button>
|
||||||
</label>
|
</label>
|
||||||
<button type="button" id="request-model-load" style="display:none" onclick="requestSelectedModelLoad()">Load on available node</button>
|
<button type="button" id="request-model-load" style="display:none" onclick="requestSelectedModelLoad()">Load on available node</button>
|
||||||
<div id="chat-status" class="chat-status">select a model to start</div>
|
<div id="chat-status" class="chat-status">select a model to start</div>
|
||||||
@@ -1268,6 +1270,7 @@ let chatBusy = false;
|
|||||||
let chatSessions = [];
|
let chatSessions = [];
|
||||||
let activeChatSessionId = "";
|
let activeChatSessionId = "";
|
||||||
let selectedChatModel = localStorage.getItem("meshnet_chat_model") || "";
|
let selectedChatModel = localStorage.getItem("meshnet_chat_model") || "";
|
||||||
|
let selectedChatQuantization = localStorage.getItem("meshnet_chat_quantization") || "";
|
||||||
const CHAT_SESSIONS_KEY = "meshnet_chat_sessions_v1";
|
const CHAT_SESSIONS_KEY = "meshnet_chat_sessions_v1";
|
||||||
const CHAT_ACTIVE_SESSION_KEY = "meshnet_chat_active_session_v1";
|
const CHAT_ACTIVE_SESSION_KEY = "meshnet_chat_active_session_v1";
|
||||||
const CHAT_SESSIONS_LIMIT = 50;
|
const CHAT_SESSIONS_LIMIT = 50;
|
||||||
@@ -1628,6 +1631,36 @@ function renderChatModels(force) {
|
|||||||
return `<option value="${esc(model.id)}"${model.id === selectedChatModel ? " selected" : ""}>${esc(label)}</option>`;
|
return `<option value="${esc(model.id)}"${model.id === selectedChatModel ? " selected" : ""}>${esc(label)}</option>`;
|
||||||
}).join("");
|
}).join("");
|
||||||
select.value = selectedChatModel;
|
select.value = selectedChatModel;
|
||||||
|
renderChatQuantizations();
|
||||||
|
}
|
||||||
|
|
||||||
|
function selectedChatModelEntry() {
|
||||||
|
return availableModels.find(model => model.id === selectedChatModel) || null;
|
||||||
|
}
|
||||||
|
|
||||||
|
function renderChatQuantizations() {
|
||||||
|
const select = $("chat-quantization");
|
||||||
|
const vote = $("vote-coverage");
|
||||||
|
const model = selectedChatModelEntry();
|
||||||
|
if (!select || !vote) return;
|
||||||
|
const variants = (model && model.quantizations) || [];
|
||||||
|
const selectable = variants.filter(variant => variant.selectable);
|
||||||
|
if (!selectable.length) {
|
||||||
|
select.innerHTML = '<option value="">no complete coverage</option>';
|
||||||
|
select.disabled = true;
|
||||||
|
vote.style.display = model ? "" : "none";
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
const preferred = selectable.find(variant => variant.id === selectedChatQuantization)
|
||||||
|
|| selectable[0];
|
||||||
|
selectedChatQuantization = preferred.id;
|
||||||
|
localStorage.setItem("meshnet_chat_quantization", selectedChatQuantization);
|
||||||
|
select.innerHTML = selectable.map(variant =>
|
||||||
|
`<option value="${esc(variant.id)}">${esc(variant.id)}</option>`
|
||||||
|
).join("");
|
||||||
|
select.value = selectedChatQuantization;
|
||||||
|
select.disabled = false;
|
||||||
|
vote.style.display = variants.some(variant => !variant.selectable) ? "" : "none";
|
||||||
}
|
}
|
||||||
|
|
||||||
function bindChatModelSelect() {
|
function bindChatModelSelect() {
|
||||||
@@ -1649,6 +1682,24 @@ function selectChatModel(value) {
|
|||||||
saveChatSessionsStore();
|
saveChatSessionsStore();
|
||||||
renderChatSessionList();
|
renderChatSessionList();
|
||||||
}
|
}
|
||||||
|
renderChatQuantizations();
|
||||||
|
}
|
||||||
|
|
||||||
|
function selectChatQuantization(value) {
|
||||||
|
selectedChatQuantization = value || "";
|
||||||
|
localStorage.setItem("meshnet_chat_quantization", selectedChatQuantization);
|
||||||
|
}
|
||||||
|
|
||||||
|
async function voteForCoverage() {
|
||||||
|
const model = selectedChatModelEntry();
|
||||||
|
if (!model) return;
|
||||||
|
const unavailable = (model.quantizations || []).find(variant => !variant.selectable);
|
||||||
|
if (!unavailable) return;
|
||||||
|
const result = await apiCall("/v1/models/vote", "POST", {
|
||||||
|
model: model.id,
|
||||||
|
quantization: unavailable.id,
|
||||||
|
});
|
||||||
|
renderChatStatus(result.ok ? `vote recorded for ${unavailable.id} coverage` : "coverage vote failed");
|
||||||
}
|
}
|
||||||
|
|
||||||
async function requestSelectedModelLoad() {
|
async function requestSelectedModelLoad() {
|
||||||
@@ -2049,6 +2100,7 @@ async function sendChat() {
|
|||||||
{ role: "user", content: prompt },
|
{ role: "user", content: prompt },
|
||||||
],
|
],
|
||||||
stream: true,
|
stream: true,
|
||||||
|
quantization: selectedChatQuantization || "bfloat16",
|
||||||
max_tokens: 15120,
|
max_tokens: 15120,
|
||||||
};
|
};
|
||||||
chatBusy = true;
|
chatBusy = true;
|
||||||
@@ -2213,6 +2265,7 @@ function applyAvailableModels(models, map) {
|
|||||||
aliases: model.aliases || [],
|
aliases: model.aliases || [],
|
||||||
coverage: model.shard_coverage_percentage,
|
coverage: model.shard_coverage_percentage,
|
||||||
servedCopies: model.served_model_copies,
|
servedCopies: model.served_model_copies,
|
||||||
|
quantizations: model.quantizations || [],
|
||||||
};
|
};
|
||||||
entry.servedCopies = modelServedCopiesFromMap(lastNetworkMap, entry);
|
entry.servedCopies = modelServedCopiesFromMap(lastNetworkMap, entry);
|
||||||
return entry;
|
return entry;
|
||||||
|
|||||||
@@ -107,6 +107,51 @@
|
|||||||
},
|
},
|
||||||
"input_price_per_1k_tokens": 0.00012,
|
"input_price_per_1k_tokens": 0.00012,
|
||||||
"output_price_per_1k_tokens": 0.00076
|
"output_price_per_1k_tokens": 0.00076
|
||||||
|
},
|
||||||
|
"qwen3.6-27b": {
|
||||||
|
"layers_start": 0,
|
||||||
|
"layers_end": 63,
|
||||||
|
"hf_repo": "Qwen/Qwen3.6-27B",
|
||||||
|
"revision": "6a9e13bd6fc8f0983b9b99948120bc37f49c13e9",
|
||||||
|
"aliases": [
|
||||||
|
"qwen3.6-27b",
|
||||||
|
"Qwen3.6-27B",
|
||||||
|
"Qwen/Qwen3.6-27B"
|
||||||
|
],
|
||||||
|
"recommended": true,
|
||||||
|
"deployment_status": "recommended",
|
||||||
|
"price_per_1k_tokens": 0.0006,
|
||||||
|
"hf_aliases": [
|
||||||
|
"qwen/qwen3.6-27b"
|
||||||
|
],
|
||||||
|
"hf_verified_match_note": "Pinned to the official Qwen/Qwen3.6-27B BF16 revision. Live comparable-provider pricing refreshes this fallback when available; the last verified rate is retained when the provider lookup is unavailable.",
|
||||||
|
"required_model_bytes": 55576522126,
|
||||||
|
"download_size_bytes": 55576522126,
|
||||||
|
"native_quantization": "bfloat16",
|
||||||
|
"canonical_audit_dtype": "bfloat16",
|
||||||
|
"canonical_audit_quantization": "bfloat16",
|
||||||
|
"bytes_per_layer": {
|
||||||
|
"bfloat16": 868383159,
|
||||||
|
"int8": 434191580,
|
||||||
|
"nf4": 217095790
|
||||||
|
},
|
||||||
|
"metadata": {
|
||||||
|
"architecture": "Qwen3.5 hybrid linear-attention transformer",
|
||||||
|
"total_parameters": "27B",
|
||||||
|
"num_layers": 64,
|
||||||
|
"context_length": 262144,
|
||||||
|
"native_quantization": "bfloat16",
|
||||||
|
"download_size_gb": 52,
|
||||||
|
"recommended_short_name": "qwen3.6-27b",
|
||||||
|
"text_only": true,
|
||||||
|
"recommended_engines": [
|
||||||
|
"Transformers",
|
||||||
|
"vLLM",
|
||||||
|
"SGLang"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"input_price_per_1k_tokens": 0.0002,
|
||||||
|
"output_price_per_1k_tokens": 0.001
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -970,6 +970,59 @@ def _preset_bytes_per_layer(preset: dict) -> dict[str, int]:
|
|||||||
return {"bfloat16": 30 * 1024 * 1024}
|
return {"bfloat16": 30 * 1024 * 1024}
|
||||||
|
|
||||||
|
|
||||||
|
_QUANTIZATION_QUALITY = {"nf4": 0, "int8": 1, "bfloat16": 2}
|
||||||
|
|
||||||
|
|
||||||
|
def _normalize_quantization(value: object) -> str | None:
|
||||||
|
"""Normalize public precision names to the node/backend vocabulary."""
|
||||||
|
if not isinstance(value, str):
|
||||||
|
return None
|
||||||
|
normalized = value.strip().lower()
|
||||||
|
if normalized == "bf16":
|
||||||
|
normalized = "bfloat16"
|
||||||
|
return normalized if normalized in _QUANTIZATION_QUALITY else None
|
||||||
|
|
||||||
|
|
||||||
|
def _quantization_satisfies(available: object, requested: object) -> bool:
|
||||||
|
"""Return whether a shard meets a request's minimum precision."""
|
||||||
|
available_normalized = _normalize_quantization(available)
|
||||||
|
requested_normalized = _normalize_quantization(requested)
|
||||||
|
return (
|
||||||
|
available_normalized is not None
|
||||||
|
and requested_normalized is not None
|
||||||
|
and _QUANTIZATION_QUALITY[available_normalized]
|
||||||
|
>= _QUANTIZATION_QUALITY[requested_normalized]
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _available_quantizations(
|
||||||
|
nodes: list["_NodeEntry"],
|
||||||
|
required_start: int,
|
||||||
|
required_end: int,
|
||||||
|
) -> list[dict]:
|
||||||
|
"""Summarize selectable minimum-precision variants for a model.
|
||||||
|
|
||||||
|
A request may use higher-precision shards, so each row counts coverage from
|
||||||
|
every node whose active quantization satisfies that row's minimum.
|
||||||
|
"""
|
||||||
|
rows = []
|
||||||
|
for quantization in ("bfloat16", "int8", "nf4"):
|
||||||
|
qualifying = [
|
||||||
|
node for node in nodes
|
||||||
|
if _quantization_satisfies(node.quantization, quantization)
|
||||||
|
]
|
||||||
|
coverage = _coverage_percentage(qualifying, required_start, required_end)
|
||||||
|
rows.append({
|
||||||
|
"id": quantization,
|
||||||
|
"shard_coverage_percentage": coverage,
|
||||||
|
"served_model_copies": _served_model_copies(
|
||||||
|
qualifying, required_start, required_end,
|
||||||
|
),
|
||||||
|
"selectable": coverage >= 100.0,
|
||||||
|
})
|
||||||
|
return rows
|
||||||
|
|
||||||
|
|
||||||
def _node_quantization(node: _NodeEntry, preset: dict) -> str:
|
def _node_quantization(node: _NodeEntry, preset: dict) -> str:
|
||||||
bytes_per_layer = _preset_bytes_per_layer(preset)
|
bytes_per_layer = _preset_bytes_per_layer(preset)
|
||||||
if node.quantization in bytes_per_layer:
|
if node.quantization in bytes_per_layer:
|
||||||
@@ -1311,6 +1364,105 @@ def _request_model_load_locked(server: "_TrackerHTTPServer", model_key: str) ->
|
|||||||
return None
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _preferred_node_quantization(
|
||||||
|
node: _NodeEntry,
|
||||||
|
preset: dict,
|
||||||
|
requested: str,
|
||||||
|
) -> str | None:
|
||||||
|
"""Choose the highest supported precision that satisfies a request."""
|
||||||
|
supported = {
|
||||||
|
normalized
|
||||||
|
for value in ([node.quantization] if node.quantization else []) + list(node.quantizations)
|
||||||
|
if (normalized := _normalize_quantization(value)) in _preset_bytes_per_layer(preset)
|
||||||
|
}
|
||||||
|
qualifying = [
|
||||||
|
value for value in supported if _quantization_satisfies(value, requested)
|
||||||
|
]
|
||||||
|
return max(qualifying, key=lambda value: _QUANTIZATION_QUALITY[value]) if qualifying else None
|
||||||
|
|
||||||
|
|
||||||
|
def _queue_initial_model_placement_locked(
|
||||||
|
server: "_TrackerHTTPServer",
|
||||||
|
model_key: str,
|
||||||
|
requested_quantization: str,
|
||||||
|
) -> list[dict]:
|
||||||
|
"""Queue a complete first copy on spare tracker-managed capacity.
|
||||||
|
|
||||||
|
This intentionally never replaces an existing assignment. Replacement is
|
||||||
|
a later demand-balancing action, while the first chat request may use only
|
||||||
|
genuinely spare slots and memory.
|
||||||
|
"""
|
||||||
|
resolved_name, preset = _resolve_model_preset(server.model_presets, model_key)
|
||||||
|
if preset is None or not preset.get("recommended") or not preset.get("hf_repo"):
|
||||||
|
return []
|
||||||
|
required_start, required_end = _preset_layer_bounds(preset)
|
||||||
|
qualifying = [
|
||||||
|
node for node in server.registry.values()
|
||||||
|
if _node_matches_preset(node, resolved_name, preset) # type: ignore[arg-type]
|
||||||
|
and _quantization_satisfies(node.quantization, requested_quantization)
|
||||||
|
]
|
||||||
|
if _coverage_percentage(qualifying, required_start, required_end) >= 100.0:
|
||||||
|
return []
|
||||||
|
|
||||||
|
cursor = required_start
|
||||||
|
queued: list[dict] = []
|
||||||
|
for host in _memory_pool_map(server)["hosts"]:
|
||||||
|
if cursor > required_end or host["spare_slots"] <= 0:
|
||||||
|
break
|
||||||
|
host_nodes = [
|
||||||
|
server.registry[item["node_id"]]
|
||||||
|
for item in host["loaded"]
|
||||||
|
if item["node_id"] in server.registry
|
||||||
|
]
|
||||||
|
candidates = [
|
||||||
|
node for node in host_nodes
|
||||||
|
if node.status == "ready" and node.pending_new_assignment is None
|
||||||
|
and _preferred_node_quantization(node, preset, requested_quantization) is not None
|
||||||
|
]
|
||||||
|
if not candidates:
|
||||||
|
continue
|
||||||
|
anchor = max(candidates, key=lambda node: node.benchmark_tokens_per_sec)
|
||||||
|
quantization = _preferred_node_quantization(anchor, preset, requested_quantization)
|
||||||
|
if quantization is None:
|
||||||
|
continue
|
||||||
|
layer_bytes = _preset_bytes_per_layer(preset)[quantization]
|
||||||
|
capacity = int(host["memory_spare_bytes"] // layer_bytes)
|
||||||
|
if capacity <= 0:
|
||||||
|
continue
|
||||||
|
end = min(required_end, cursor + capacity - 1)
|
||||||
|
directive = _add_shard_directive(
|
||||||
|
anchor, str(preset["hf_repo"]), cursor, end, quantization,
|
||||||
|
)
|
||||||
|
anchor.pending_new_assignment = directive
|
||||||
|
anchor.pending_directives.append(directive)
|
||||||
|
queued.append({
|
||||||
|
"node_id": anchor.node_id,
|
||||||
|
"shard_start": cursor,
|
||||||
|
"shard_end": end,
|
||||||
|
"quantization": quantization,
|
||||||
|
})
|
||||||
|
cursor = end + 1
|
||||||
|
if cursor <= required_end:
|
||||||
|
# The directives are not safe to issue without a complete route. Undo
|
||||||
|
# only the directives created above; unrelated pending work is intact.
|
||||||
|
for placement in queued:
|
||||||
|
node = server.registry.get(placement["node_id"])
|
||||||
|
if node is None:
|
||||||
|
continue
|
||||||
|
node.pending_directives = [
|
||||||
|
directive for directive in node.pending_directives
|
||||||
|
if directive is not node.pending_new_assignment
|
||||||
|
]
|
||||||
|
node.pending_new_assignment = None
|
||||||
|
return []
|
||||||
|
_tracker_log(
|
||||||
|
server, "info", "initial demanded model placement queued",
|
||||||
|
model=resolved_name, hf_repo=preset["hf_repo"],
|
||||||
|
requested_quantization=requested_quantization, placements=queued,
|
||||||
|
)
|
||||||
|
return queued
|
||||||
|
|
||||||
|
|
||||||
def _deployment_summary(nodes: list[_NodeEntry], preset: dict | None) -> dict:
|
def _deployment_summary(nodes: list[_NodeEntry], preset: dict | None) -> dict:
|
||||||
if preset is None:
|
if preset is None:
|
||||||
return {"recommended": False}
|
return {"recommended": False}
|
||||||
@@ -2715,6 +2867,9 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
|
|||||||
if self.path == "/v1/models/load":
|
if self.path == "/v1/models/load":
|
||||||
self._handle_model_load_request()
|
self._handle_model_load_request()
|
||||||
return
|
return
|
||||||
|
if self.path == "/v1/models/vote":
|
||||||
|
self._handle_model_coverage_vote()
|
||||||
|
return
|
||||||
if self.path == "/v1/calibration/toploc/run":
|
if self.path == "/v1/calibration/toploc/run":
|
||||||
self._handle_toploc_calibration_run()
|
self._handle_toploc_calibration_run()
|
||||||
return
|
return
|
||||||
@@ -2842,6 +2997,11 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
|
|||||||
required_start,
|
required_start,
|
||||||
required_end,
|
required_end,
|
||||||
)
|
)
|
||||||
|
quantizations = _available_quantizations(
|
||||||
|
model_nodes,
|
||||||
|
required_start,
|
||||||
|
required_end,
|
||||||
|
)
|
||||||
aliases = [name]
|
aliases = [name]
|
||||||
hf_repo = preset.get("hf_repo")
|
hf_repo = preset.get("hf_repo")
|
||||||
if hf_repo and hf_repo not in aliases:
|
if hf_repo and hf_repo not in aliases:
|
||||||
@@ -2856,12 +3016,14 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
|
|||||||
"owned_by": "meshnet",
|
"owned_by": "meshnet",
|
||||||
"name": name,
|
"name": name,
|
||||||
"hf_repo": hf_repo,
|
"hf_repo": hf_repo,
|
||||||
|
"revision": preset.get("revision"),
|
||||||
"aliases": aliases,
|
"aliases": aliases,
|
||||||
"metadata": dict(preset.get("metadata") or _model_metadata_from_nodes(model_nodes)),
|
"metadata": dict(preset.get("metadata") or _model_metadata_from_nodes(model_nodes)),
|
||||||
"recommended": bool(preset.get("recommended", False)),
|
"recommended": bool(preset.get("recommended", False)),
|
||||||
"deployment": _deployment_summary(alive, preset),
|
"deployment": _deployment_summary(alive, preset),
|
||||||
"shard_coverage_percentage": coverage,
|
"shard_coverage_percentage": coverage,
|
||||||
"served_model_copies": served_copies,
|
"served_model_copies": served_copies,
|
||||||
|
"quantizations": quantizations,
|
||||||
})
|
})
|
||||||
seen_ids.add(name)
|
seen_ids.add(name)
|
||||||
# Note: the preset's hf_repo is deliberately NOT added to seen_ids —
|
# Note: the preset's hf_repo is deliberately NOT added to seen_ids —
|
||||||
@@ -3023,6 +3185,7 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
|
|||||||
{
|
{
|
||||||
"id": name,
|
"id": name,
|
||||||
"hf_repo": preset.get("hf_repo"),
|
"hf_repo": preset.get("hf_repo"),
|
||||||
|
"revision": preset.get("revision"),
|
||||||
"aliases": list(preset.get("aliases", []) or []),
|
"aliases": list(preset.get("aliases", []) or []),
|
||||||
"metadata": dict(preset.get("metadata") or {}),
|
"metadata": dict(preset.get("metadata") or {}),
|
||||||
"deployment": _deployment_summary(nodes, preset),
|
"deployment": _deployment_summary(nodes, preset),
|
||||||
@@ -3077,6 +3240,17 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
|
|||||||
|
|
||||||
model: str = body.get("model", "")
|
model: str = body.get("model", "")
|
||||||
is_stream: bool = bool(body.get("stream", False))
|
is_stream: bool = bool(body.get("stream", False))
|
||||||
|
requested_quantization = _normalize_quantization(
|
||||||
|
body.get("quantization", "bfloat16")
|
||||||
|
)
|
||||||
|
if requested_quantization is None:
|
||||||
|
self._send_json(400, {"error": {
|
||||||
|
"message": "quantization must be one of bfloat16, int8, or nf4",
|
||||||
|
"type": "invalid_request_error",
|
||||||
|
"code": "invalid_quantization",
|
||||||
|
}})
|
||||||
|
return
|
||||||
|
body["quantization"] = requested_quantization
|
||||||
|
|
||||||
if model and server.stats is not None:
|
if model and server.stats is not None:
|
||||||
server.stats.record_request(model)
|
server.stats.record_request(model)
|
||||||
@@ -3170,6 +3344,16 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
|
|||||||
pinned_nodes = [server.registry[nid] for nid in pinned_ids]
|
pinned_nodes = [server.registry[nid] for nid in pinned_ids]
|
||||||
|
|
||||||
if pinned_nodes is not None:
|
if pinned_nodes is not None:
|
||||||
|
if any(
|
||||||
|
not _quantization_satisfies(node.quantization, requested_quantization)
|
||||||
|
for node in pinned_nodes
|
||||||
|
):
|
||||||
|
self._send_json(409, {"error": {
|
||||||
|
"message": "pinned route does not satisfy requested quantization",
|
||||||
|
"type": "invalid_request_error",
|
||||||
|
"code": "route_quantization_unavailable",
|
||||||
|
}})
|
||||||
|
return
|
||||||
node = pinned_nodes[0]
|
node = pinned_nodes[0]
|
||||||
else:
|
else:
|
||||||
# Find a live tracker-mode node for this model
|
# Find a live tracker-mode node for this model
|
||||||
@@ -3177,7 +3361,9 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
|
|||||||
self._purge_expired_nodes()
|
self._purge_expired_nodes()
|
||||||
candidates = [
|
candidates = [
|
||||||
n for n in server.registry.values()
|
n for n in server.registry.values()
|
||||||
if n.tracker_mode and _node_matches_model(n, model)
|
if n.tracker_mode
|
||||||
|
and _node_matches_model(n, model)
|
||||||
|
and _quantization_satisfies(n.quantization, requested_quantization)
|
||||||
]
|
]
|
||||||
|
|
||||||
if not candidates:
|
if not candidates:
|
||||||
@@ -3185,11 +3371,48 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
|
|||||||
with server.lock:
|
with server.lock:
|
||||||
candidates = [
|
candidates = [
|
||||||
n for n in server.registry.values()
|
n for n in server.registry.values()
|
||||||
if n.shard_start == 0 and _node_matches_model(n, model)
|
if n.shard_start == 0
|
||||||
|
and _node_matches_model(n, model)
|
||||||
|
and _quantization_satisfies(n.quantization, requested_quantization)
|
||||||
]
|
]
|
||||||
|
|
||||||
if not candidates:
|
if not candidates:
|
||||||
with server.lock:
|
with server.lock:
|
||||||
|
resolved_name, preset = _resolve_model_preset(server.model_presets, model)
|
||||||
|
placements = (
|
||||||
|
_queue_initial_model_placement_locked(
|
||||||
|
server,
|
||||||
|
resolved_name or model,
|
||||||
|
requested_quantization,
|
||||||
|
)
|
||||||
|
if preset is not None and preset.get("recommended")
|
||||||
|
else []
|
||||||
|
)
|
||||||
|
if preset is not None:
|
||||||
|
required_start, required_end = _preset_layer_bounds(preset)
|
||||||
|
qualifying = [
|
||||||
|
n for n in server.registry.values()
|
||||||
|
if _node_matches_preset(n, resolved_name, preset)
|
||||||
|
and _quantization_satisfies(
|
||||||
|
n.quantization, requested_quantization,
|
||||||
|
)
|
||||||
|
]
|
||||||
|
coverage = _coverage_percentage(
|
||||||
|
qualifying, required_start, required_end,
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
coverage = 0.0
|
||||||
|
if placements:
|
||||||
|
self._send_json(503, {"error": {
|
||||||
|
"message": f"model {model!r} is being deployed",
|
||||||
|
"type": "service_unavailable",
|
||||||
|
"code": "model_loading",
|
||||||
|
"retry_after_seconds": 30,
|
||||||
|
"requested_quantization": requested_quantization,
|
||||||
|
"shard_coverage_percentage": coverage,
|
||||||
|
"placements": placements,
|
||||||
|
}})
|
||||||
|
return
|
||||||
registered = [
|
registered = [
|
||||||
{
|
{
|
||||||
"node_id": n.node_id,
|
"node_id": n.node_id,
|
||||||
@@ -3235,12 +3458,14 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
|
|||||||
if _node_matches_preset(n, route_model, preset)
|
if _node_matches_preset(n, route_model, preset)
|
||||||
and n.shard_start is not None
|
and n.shard_start is not None
|
||||||
and n.shard_end is not None
|
and n.shard_end is not None
|
||||||
|
and _quantization_satisfies(n.quantization, requested_quantization)
|
||||||
]
|
]
|
||||||
else:
|
else:
|
||||||
all_nodes = [
|
all_nodes = [
|
||||||
n for n in server.registry.values()
|
n for n in server.registry.values()
|
||||||
if _node_matches_model(n, route_model)
|
if _node_matches_model(n, route_model)
|
||||||
and n.shard_start is not None and n.num_layers is not None
|
and n.shard_start is not None and n.num_layers is not None
|
||||||
|
and _quantization_satisfies(n.quantization, requested_quantization)
|
||||||
]
|
]
|
||||||
rs, re = 0, (max((n.num_layers for n in all_nodes), default=1) - 1)
|
rs, re = 0, (max((n.num_layers for n in all_nodes), default=1) - 1)
|
||||||
if pinned_nodes is not None:
|
if pinned_nodes is not None:
|
||||||
@@ -4364,6 +4589,27 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
|
|||||||
return
|
return
|
||||||
self._send_json(202, {"status": "queued", "assignment": assignment})
|
self._send_json(202, {"status": "queued", "assignment": assignment})
|
||||||
|
|
||||||
|
def _handle_model_coverage_vote(self):
|
||||||
|
"""Record a rolling wish-list signal for an unavailable precision."""
|
||||||
|
server: _TrackerHTTPServer = self.server # type: ignore[assignment]
|
||||||
|
body = self._read_json_body()
|
||||||
|
if body is None:
|
||||||
|
return
|
||||||
|
model = body.get("model")
|
||||||
|
quantization = _normalize_quantization(body.get("quantization"))
|
||||||
|
if not isinstance(model, str) or not model.strip() or quantization is None:
|
||||||
|
self._send_json(400, {"error": "model and valid quantization are required"})
|
||||||
|
return
|
||||||
|
resolved_name, preset = _resolve_model_preset(server.model_presets, model)
|
||||||
|
if preset is None:
|
||||||
|
self._send_json(404, {"error": "unknown model preset"})
|
||||||
|
return
|
||||||
|
vote_key = f"wishlist:{resolved_name}:{quantization}"
|
||||||
|
if server.stats is not None:
|
||||||
|
server.stats.record_request(vote_key)
|
||||||
|
_tracker_log(server, "info", "model coverage vote", model=resolved_name, quantization=quantization)
|
||||||
|
self._send_json(202, {"status": "recorded", "model": resolved_name, "quantization": quantization})
|
||||||
|
|
||||||
def _handle_stats(self):
|
def _handle_stats(self):
|
||||||
server: _TrackerHTTPServer = self.server # type: ignore[assignment]
|
server: _TrackerHTTPServer = self.server # type: ignore[assignment]
|
||||||
if server.stats is None:
|
if server.stats is None:
|
||||||
|
|||||||
@@ -17,6 +17,7 @@ from meshnet_tracker.auth import sign_hive_request
|
|||||||
from meshnet_tracker.server import (
|
from meshnet_tracker.server import (
|
||||||
TrackerServer,
|
TrackerServer,
|
||||||
_NodeEntry,
|
_NodeEntry,
|
||||||
|
_available_quantizations,
|
||||||
_memory_pool_map,
|
_memory_pool_map,
|
||||||
_registration_ban_error,
|
_registration_ban_error,
|
||||||
_scale_demanded_models_locked,
|
_scale_demanded_models_locked,
|
||||||
@@ -130,6 +131,48 @@ def test_tracker_lists_recommended_kimi_before_nodes_register():
|
|||||||
assert network_map["recommended_models"][0]["id"] == "kimi-k2.7"
|
assert network_map["recommended_models"][0]["id"] == "kimi-k2.7"
|
||||||
|
|
||||||
|
|
||||||
|
def test_tracker_lists_recommended_qwen_27b_before_nodes_register():
|
||||||
|
"""Recommended Qwen 27B is advertised with its pinned canonical artifact.
|
||||||
|
|
||||||
|
Tags: http, routing, tracker
|
||||||
|
"""
|
||||||
|
tracker = TrackerServer()
|
||||||
|
port = tracker.start()
|
||||||
|
url = f"http://127.0.0.1:{port}"
|
||||||
|
try:
|
||||||
|
models = _get_json(f"{url}/v1/models")
|
||||||
|
network_map = _get_json(f"{url}/v1/network/map")
|
||||||
|
finally:
|
||||||
|
tracker.stop()
|
||||||
|
|
||||||
|
qwen = next(model for model in models["data"] if model["id"] == "qwen3.6-27b")
|
||||||
|
assert qwen["hf_repo"] == "Qwen/Qwen3.6-27B"
|
||||||
|
assert qwen["revision"] == "6a9e13bd6fc8f0983b9b99948120bc37f49c13e9"
|
||||||
|
assert qwen["metadata"]["num_layers"] == 64
|
||||||
|
assert qwen["metadata"]["text_only"] is True
|
||||||
|
assert qwen["deployment"]["recommended"] is True
|
||||||
|
advertised = next(model for model in network_map["recommended_models"] if model["id"] == "qwen3.6-27b")
|
||||||
|
assert advertised["hf_repo"] == "Qwen/Qwen3.6-27B"
|
||||||
|
|
||||||
|
|
||||||
|
def test_quantization_coverage_uses_higher_precision_shards():
|
||||||
|
"""A minimum-precision route may upgrade individual shards."""
|
||||||
|
bf16 = _NodeEntry(
|
||||||
|
"bf16", "http://node-a", 0, 0, "qwen3.6-27b", None, {}, None, 1.0,
|
||||||
|
quantization="bfloat16",
|
||||||
|
)
|
||||||
|
int8 = _NodeEntry(
|
||||||
|
"int8", "http://node-b", 1, 1, "qwen3.6-27b", None, {}, None, 1.0,
|
||||||
|
quantization="int8",
|
||||||
|
)
|
||||||
|
|
||||||
|
variants = {entry["id"]: entry for entry in _available_quantizations([bf16, int8], 0, 1)}
|
||||||
|
|
||||||
|
assert variants["bfloat16"]["selectable"] is False
|
||||||
|
assert variants["int8"]["selectable"] is True
|
||||||
|
assert variants["nf4"]["selectable"] is True
|
||||||
|
|
||||||
|
|
||||||
def test_network_map_exposes_pool_size_and_speed_summary():
|
def test_network_map_exposes_pool_size_and_speed_summary():
|
||||||
"Network map exposes pool size and speed summary\n\nTags: http, routing, tracker"
|
"Network map exposes pool size and speed summary\n\nTags: http, routing, tracker"
|
||||||
tracker = TrackerServer()
|
tracker = TrackerServer()
|
||||||
|
|||||||
Reference in New Issue
Block a user