feat: harden node placement and partial model loading

This commit is contained in:
Dobromir Popov
2026-07-13 21:58:08 +02:00
parent a6bcc69288
commit 5d87e81bc9
21 changed files with 497 additions and 55 deletions

View File

@@ -123,3 +123,11 @@ Rejected. gRPC/HTTP2 already provides mature streaming, flow control, deadlines,
4. Real two-machine execution using both Shards.
5. End-to-end performance/fit advantage over the current distributed route.
6. Separate Qwen3-family architecture certification.
## Relationship to US-042 (whole-model GGUF shortcut)
[US-042](../issues/42-gguf-llamacpp-node-backend.md) **phase C** ships first: a node with enough RAM serves a **full** GGUF via llama.cpp on a single-hop Inference Route using the existing HTTP activation seam and PyTorch-era tracker integration. That is intentionally small and does not require this ADR's gRPC worker or llama.cpp patch stack.
This ADR's track starts only after **DGR-001** (controlled safetensors-vs-GGUF benchmark) shows a meaningful speed or fit benefit. Then implement the native worker (DGR-002+) — which subsumes US-042 direction A (layer-range GGUF + boundary tensors) if the benchmark warrants it.
Do not run US-042 phase C and DGR-008+ in parallel on the same node backend without an explicit integration plan; phase C uses llama-cpp-python (or equivalent) whole-model path; ADR-0024 uses the standalone C++ worker.

View File

@@ -0,0 +1,51 @@
# ADR-0026: Node assignment ownership — pinned startup vs managed demand placement
## Status: Accepted
## Context
Three features define how a node gets its `(model, shard range, recipe/quantization)`:
1. **ADR-0011 / US-013** — tracker suggests a gap from coverage map on startup or auto-join.
2. **Node capability admission (ADR-0023 / NCA)** — a node must pass `doctor` + real forward before becoming routable; startup-assigned work is validated, not blindly trusted.
3. **Qwen demand placement** (`.scratch/qwen3.6-27b-demand-placement/`) — tracker deploys a model when chat demand appears and spare capacity exists.
These looked contradictory: NCA and the Qwen PRD both say startup assignments are "pinned," while demand placement wants the tracker to assign models dynamically.
## Decision
### Three assignment tiers
| Tier | How it is created | Mutable by tracker? | Admission |
|---|---|---|---|
| **Operator-initiated** | Node starts with explicit `--model` / shard flags | **No** — pinned until operator restarts or explicitly reloads | Must pass NCA `doctor` before routable |
| **Network bootstrap** | `/v1/network/assign` or `/v1/nodes/assign` on first join (ADR-0011) | **No** for the active loaded shard — treated as operator-equivalent once accepted at startup | Must pass NCA before routable |
| **Tracker-managed** | Demand-driven placement (Qwen PRD) on spare capacity | **Yes** — marked `managed: true`; subject to cooldown / safety policy | Must pass NCA for the new assignment before routable |
### Spare capacity rule (unifies NCA + Qwen)
- A nodes **active** `(model, shard, recipe)` from startup is **pinned** — the tracker does not silently retarget a serving node to a different model.
- **Spare capacity** — memory/slots not holding the pinned assignment, or a node registered without a model — may receive **tracker-managed** assignments to satisfy demand.
- Until multi-shard runtime exists (US-048), “spare capacity” effectively means **model-less nodes** or nodes explicitly registered for managed placement; do not overload a single-shard node with a second assignment.
### Demand placement interaction
- First chat request for an unrouted model queues **demand**; leader tracker may assign **managed** nodes only when eligible spare capacity exists (Qwen PRD).
- Until complete coverage + validated recipes exist, return retryable `503 model_loading` with coverage metadata.
- Managed assignments must not evict pinned assignments on other nodes without the Qwen safety policy (≥3 copies, 1.5× demand multiplier, cooldown).
### NCA is not optional for any tier
Regardless of assignment source, registration carries **validated capability** only after `doctor` succeeds. The tracker excludes nodes with absent, stale, or failed capability reports (ADR-0023).
## Consequences
- NCA and Qwen demand placement are complementary: NCA gates *quality*; demand placement gates *where new coverage comes from*.
- US-048 (multi-shard slots) extends spare capacity — until then, demand placement primarily targets nodes that join without `--model`.
- Rebalance / dropout relocation (US-013, US-048) applies to **coverage gaps**, not retroactive retargeting of pinned nodes for demand convenience.
## Verification
- NCA tests: unvalidated nodes never routed.
- Demand-placement tests (when implemented): managed flag set; pinned nodes unchanged.
- Documented in Qwen scratch PRD and NCA README cross-links.