feat: harden node placement and partial model loading

2026-07-13 21:58:08 +02:00
parent a6bcc69288
commit 5d87e81bc9
21 changed files with 497 additions and 55 deletions
--- a/docs/PRD.md
+++ b/docs/PRD.md
@@ -1,4 +1,4 @@
-Status: done (base program US-001…US-035 complete; friends-test arc US-036…US-047 in `docs/prd.json`. Payment/settlement superseded by ADR-0015; fraud by ADR-0018.)
+Status: done (US-001…US-035 complete; friends-test arc US-036…US-049 in `docs/prd.json`; US-048/050 tracked. See ADRs 0015–0018, 0023, 0025–0026.)

 # Distributed Inference Network — PRD

@@ -8,10 +8,12 @@ Running large language models requires expensive dedicated hardware that most pe

 ## Solution

-A volunteer GPU network where anyone can share their GPU by running a single command and immediately start earning tokens. Nodes each load a shard of a large model; a tracker routes inference requests through the optimal chain of nodes whose shards collectively cover all layers. Developers access the network through an OpenAI-compatible API — a one-line change from any existing LLM integration. Clients pay in **USDT** (alpha: devnet mock-USDT; production: mainnet USDT). Node operators earn USDT payouts from the custodial treasury (ADR-0015); the TAI reward token (ADR-0002) remains deferred. Everything is auto-configured: GPU detection, shard download, wallet creation, and network registration happen automatically on first start.
+A volunteer GPU network where anyone can share their GPU by running a single command and immediately start earning **USDT**. Nodes each load a shard of a large model; a tracker routes inference requests through the optimal chain of nodes whose shards collectively cover all layers. Developers access the network through an OpenAI-compatible API — a one-line change from any existing LLM integration. Clients pay in **USDT** (alpha: devnet mock-USDT; production: mainnet USDT). Node operators earn USDT payouts from the custodial treasury (ADR-0015); the TAI reward token (ADR-0002) remains deferred. Everything is auto-configured: GPU detection, shard download, wallet creation, and network registration happen automatically on first start.

 ## User Stories

+> **Status (2026-07-13):** Stories below are the original product intent. **Shipped behavior** is in [Implementation Decisions](#implementation-decisions) and ADRs 0015–0018, 0023, 0025–0026. Superseded lines are marked inline.
+
 ### Node Operator

 1. As a node operator, I want to install the node client with a single command (`pip install meshnet-node`), so that I can start contributing without reading documentation.
@@ -21,10 +23,10 @@ A volunteer GPU network where anyone can share their GPU by running a single com
 5. As a node operator, I want my assigned shard to download automatically from HuggingFace on first start, so that I don't have to manually find or download model weights.
 6. As a node operator, I want to seed my shard to other nodes via P2P once I have it, so that new nodes with the same shard assignment don't need to download from HuggingFace.
 7. As a node operator, I want the node client to register with the tracker automatically and begin serving inference requests, so that I start earning as soon as setup is complete.
-8. As a node operator, I want to see my current node score, shard assignment, and token earnings in the terminal, so that I can verify my node is contributing correctly.
-9. As a node operator, I want to stake tokens before serving paid inference, so that I have skin in the game and the network can trust my outputs.
+8. As a node operator, I want to see my current node score, shard assignment, and USDT earnings in the terminal, so that I can verify my node is contributing correctly.
+9. As a node operator, I want to serve paid inference without upfront stake deposits, with my accrued USDT pending balance as fraud collateral and probation as the anti-sybil cost, so that onboarding stays frictionless. *(Supersedes stake-before-serving; ADR-0015/0018.)*
 10. As a node operator, I want my first N jobs to run without earning (probationary period), so that the network can establish trust before paying me.
-11. As a node operator, I want to be notified immediately if my stake is slashed due to a fraud detection event, so that I can investigate and fix the issue.
+11. As a node operator, I want to be notified when my pending balance is forfeited due to a failed audit, so that I can investigate and fix the issue. *(Supersedes stake slash; ADR-0018 forfeiture.)*
 12. As a node operator, I want to receive a strike and a warning before being banned, so that accidental failures don't immediately end my participation.
 13. As a node operator, I want to be automatically reassigned to a different shard when the tracker determines another shard is more in demand, so that my hardware is always optimally used.
 14. As a node operator, I want the node client to reconnect automatically if the tracker is temporarily unavailable, so that transient network issues don't stop me from earning.
@@ -34,8 +36,8 @@ A volunteer GPU network where anyone can share their GPU by running a single com
 ### Client Developer

 17. As a client developer, I want to send `POST /v1/chat/completions` requests to the gateway in the same format as the OpenAI API, so that I can switch to the network with a one-line code change.
-18. As a client developer, I want to authenticate with an API key funded by SOL or USDC, so that I never need to acquire or hold our native token.
-19. As a client developer, I want to top up my API key balance by sending SOL or USDC to a Solana address, so that payment is simple and familiar.
+18. As a client developer, I want to authenticate with an API key funded by USDT, so that I never need to acquire or hold our native token. *(ADR-0015.)*
+19. As a client developer, I want to top up my API key balance by sending USDT to the treasury Solana address, so that payment is simple and familiar. *(ADR-0015; wallet binding US-039/041.)*
 20. As a client developer, I want to see a per-request cost estimate before sending a request, so that I can budget inference costs accurately.
 21. As a client developer, I want to receive streaming responses (`text/event-stream`) in OpenAI-compatible format, so that I can build low-latency user experiences.
 22. As a client developer, I want `GET /v1/models` to return the list of available model presets on the network, so that I know what I can request.
@@ -46,15 +48,15 @@ A volunteer GPU network where anyone can share their GPU by running a single com

 ### End User (via a client app)

-27. As an end user, I want to buy SOL on any exchange and use it to pay for inference, so that I don't need to understand blockchain technology to use the service.
+27. As an end user, I want to buy USDT on an exchange and use it to pay for inference via Solana, so that I don't need deep crypto knowledge to use the service. *(Clients pay USDT; SOL is only for network fees if they self-custody.)*
 28. As an end user, I want responses of equivalent quality to centralised providers, so that I don't have to trade quality for cost savings.
 29. As an end user, I want low latency on first token, so that conversational applications feel responsive.

 ### Validator

-30. As a validator, I want to automatically re-run a random sample (~5%) of completed inference requests on a reference node, so that I can detect nodes returning fraudulent outputs.
-31. As a validator, I want to submit a fraud proof on-chain when a node's output diverges beyond tolerance, so that the slash event is recorded trustlessly.
-32. As a validator, I want to earn a reward for each successful fraud detection, so that there is an economic incentive to run validation.
+30. As a validator, I want to automatically re-run a random sample (~5%) of completed inference requests on a reference node with TOPLOC activation verification, so that I can detect nodes returning fraudulent outputs. *(ADR-0018.)*
+31. As a validator, I want the tracker to record forfeiture and strikes when an audit fails, so that penalties are applied consistently. *(Supersedes on-chain fraud proof in alpha; ADR-0018.)*
+32. As a validator, I want economic incentive to run validation, so that fraud detection is not purely altruistic. *(Validator reward share deferred; forfeiture to protocol cut today.)*

 ### Network (tracker / system)

@@ -62,7 +64,7 @@ A volunteer GPU network where anyone can share their GPU by running a single com
 34. As the tracker, I want to rebalance shard assignments across nodes when demand for a model preset changes, so that the network always covers the most-requested models.
 35. As the tracker, I want to instruct a node to download a new shard when no other node covers it, so that model preset coverage is maintained automatically.
 36. As the tracker, I want to exclude banned wallets from route selection, so that fraudulent nodes cannot serve paid inference.
-37. As the tracker, I want to read stake, slash, strike, and ban state exclusively from Solana smart contracts, so that I cannot manipulate payouts even with full control of the routing layer.
+37. As the tracker, I want strike and ban state persisted in the registry and enforced on route selection, so that fraudulent wallets cannot serve paid inference. *(Supersedes on-chain-only stake/slash registry; ADR-0018; on-chain deferred per ADR-0007/0015.)*
 38. As the network, I want new model presets to be addable by submitting a HuggingFace model ID and shard count, so that the set of available models can grow without code changes.

 ## Implementation Decisions
@@ -73,7 +75,7 @@ The codebase is organized as a Python monorepo with the following top-level pack
 - `packages/gateway` — OpenAI-compatible HTTP gateway and route orchestration
 - `packages/tracker` — centralized tracker service (node registry, scoring, route selection)
 - `packages/sdk` — `meshnet` Python SDK wrapping gateway + wallet controls
- `packages/contracts` — Solana L2 smart contracts (stake, slash, strike, ban, settlement)
+- `packages/contracts` — Solana adapter boundary (custodial USDT treasury, local registry prototype)
 - `packages/p2p` — P2P gossip layer and shard swarm seeding

 ### Inference engine (ADR-0001; native GGUF path ADR-0024)
@@ -98,7 +100,7 @@ Clients pre-fund an API key with USDT. The tracker meters each request against t
 Validators re-run ~5% of completed requests with TOPLOC activation verification. Caught cheaters forfeit pending balance and receive strikes; three strikes bans the wallet. Probation (first N unpaid jobs) remains the anti-sybil re-entry cost.

 ### Tracker architecture (ADR-0004)
-Centralized tracker service (HTTP + WebSocket) for fast routing. Nodes gossip state via a lightweight P2P layer so the node client can discover routes during tracker outages. Solana is the authoritative source of truth for all incentive-relevant state.
+Centralized tracker service (HTTP + WebSocket) for fast routing. Nodes gossip state via a lightweight P2P layer so the node client can discover routes during tracker outages. **Alpha:** strike/ban/forfeiture state lives in the tracker registry (ADR-0018); USDT settlement via custodial treasury (ADR-0015). On-chain programs deferred (ADR-0007).

 ### Shard distribution (ADR-0005)
 Shards are identified by `(model_preset, shard_index)`. On assignment, the node downloads the shard layers from HuggingFace using `huggingface_hub`. Once downloaded, the node joins the P2P shard swarm and seeds to other nodes requesting the same shard. Popular shards propagate entirely via P2P; cold shards fall back to HuggingFace.
@@ -115,9 +117,9 @@ The gateway exposes OpenAI-compatible endpoints (`/v1/chat/completions`, `/v1/mo
 **Per-component seams:**
 - **Tracker**: given a set of registered nodes with known shard coverage and node scores, assert `select_route(model_preset)` returns an optimal ordered list of node endpoints.
 - **Node shard serving**: given an activation tensor for the node's layer range, assert the output tensor shape and dtype are correct.
- **Fraud detection**: given a validator that re-runs a known-bad node response, assert a slash transaction is submitted on-chain with correct attribution.
+- **Fraud detection**: given a validator that re-runs a known-bad node response, assert strike/forfeiture state updates with correct attribution (ADR-0018; on-chain slash deferred).
 - **Shard swarm**: given a node that has a shard, assert a second node with the same assignment downloads it via P2P rather than HuggingFace.
- **Payment settlement**: given a completed inference session with known compute attribution, assert token balances change by the expected amounts after epoch settlement.
+- **Payment settlement**: given a completed inference session with known compute attribution, assert USDT ledger balances change by the expected amounts after epoch settlement (ADR-0015).

 ## Out of Scope

@@ -135,4 +137,4 @@ The gateway exposes OpenAI-compatible endpoints (`/v1/chat/completions`, `/v1/mo
 - The `meshnet-node` CLI is the primary viral growth vector. Every friction point in the install/start sequence costs node operators. The startup sequence must complete without any manual configuration on a machine with a CUDA-capable GPU.
 - The name "meshnet" is a working name. The actual package and token names are TBD.
 - The Solana L2 chain selection (vs Base/Arbitrum) is not yet finalised — both are cheap, EVM-compatible fallbacks. The contracts package should abstract chain-specific details.
- The probationary period length (N free jobs) and slash amounts are economic parameters that will need tuning once the network has real usage data. Hardcode sensible defaults; make them on-chain governable.
+- The probationary period length (N free jobs) and forfeiture amounts are economic parameters that will need tuning once the network has real usage data. Hardcode sensible defaults; governance TBD (ADR-0018).
--- a/docs/adr/0024-distributed-gguf-runtime.md
+++ b/docs/adr/0024-distributed-gguf-runtime.md
@@ -123,3 +123,11 @@ Rejected. gRPC/HTTP2 already provides mature streaming, flow control, deadlines,
 4. Real two-machine execution using both Shards.
 5. End-to-end performance/fit advantage over the current distributed route.
 6. Separate Qwen3-family architecture certification.
+
+## Relationship to US-042 (whole-model GGUF shortcut)
+
+[US-042](../issues/42-gguf-llamacpp-node-backend.md) **phase C** ships first: a node with enough RAM serves a **full** GGUF via llama.cpp on a single-hop Inference Route using the existing HTTP activation seam and PyTorch-era tracker integration. That is intentionally small and does not require this ADR's gRPC worker or llama.cpp patch stack.
+
+This ADR's track starts only after **DGR-001** (controlled safetensors-vs-GGUF benchmark) shows a meaningful speed or fit benefit. Then implement the native worker (DGR-002+) — which subsumes US-042 direction A (layer-range GGUF + boundary tensors) if the benchmark warrants it.
+
+Do not run US-042 phase C and DGR-008+ in parallel on the same node backend without an explicit integration plan; phase C uses llama-cpp-python (or equivalent) whole-model path; ADR-0024 uses the standalone C++ worker.
--- a/docs/adr/0026-node-assignment-ownership-and-managed-placement.md
+++ b/docs/adr/0026-node-assignment-ownership-and-managed-placement.md
@@ -0,0 +1,51 @@
+# ADR-0026: Node assignment ownership — pinned startup vs managed demand placement
+
+## Status: Accepted
+
+## Context
+
+Three features define how a node gets its `(model, shard range, recipe/quantization)`:
+
+1. **ADR-0011 / US-013** — tracker suggests a gap from coverage map on startup or auto-join.
+2. **Node capability admission (ADR-0023 / NCA)** — a node must pass `doctor` + real forward before becoming routable; startup-assigned work is validated, not blindly trusted.
+3. **Qwen demand placement** (`.scratch/qwen3.6-27b-demand-placement/`) — tracker deploys a model when chat demand appears and spare capacity exists.
+
+These looked contradictory: NCA and the Qwen PRD both say startup assignments are "pinned," while demand placement wants the tracker to assign models dynamically.
+
+## Decision
+
+### Three assignment tiers
+
+| Tier | How it is created | Mutable by tracker? | Admission |
+|---|---|---|---|
+| **Operator-initiated** | Node starts with explicit `--model` / shard flags | **No** — pinned until operator restarts or explicitly reloads | Must pass NCA `doctor` before routable |
+| **Network bootstrap** | `/v1/network/assign` or `/v1/nodes/assign` on first join (ADR-0011) | **No** for the active loaded shard — treated as operator-equivalent once accepted at startup | Must pass NCA before routable |
+| **Tracker-managed** | Demand-driven placement (Qwen PRD) on spare capacity | **Yes** — marked `managed: true`; subject to cooldown / safety policy | Must pass NCA for the new assignment before routable |
+
+### Spare capacity rule (unifies NCA + Qwen)
+
+- A node’s **active** `(model, shard, recipe)` from startup is **pinned** — the tracker does not silently retarget a serving node to a different model.
+- **Spare capacity** — memory/slots not holding the pinned assignment, or a node registered without a model — may receive **tracker-managed** assignments to satisfy demand.
+- Until multi-shard runtime exists (US-048), “spare capacity” effectively means **model-less nodes** or nodes explicitly registered for managed placement; do not overload a single-shard node with a second assignment.
+
+### Demand placement interaction
+
+- First chat request for an unrouted model queues **demand**; leader tracker may assign **managed** nodes only when eligible spare capacity exists (Qwen PRD).
+- Until complete coverage + validated recipes exist, return retryable `503 model_loading` with coverage metadata.
+- Managed assignments must not evict pinned assignments on other nodes without the Qwen safety policy (≥3 copies, 1.5× demand multiplier, cooldown).
+
+### NCA is not optional for any tier
+
+Regardless of assignment source, registration carries **validated capability** only after `doctor` succeeds. The tracker excludes nodes with absent, stale, or failed capability reports (ADR-0023).
+
+## Consequences
+
+- NCA and Qwen demand placement are complementary: NCA gates *quality*; demand placement gates *where new coverage comes from*.
+- US-048 (multi-shard slots) extends spare capacity — until then, demand placement primarily targets nodes that join without `--model`.
+- Rebalance / dropout relocation (US-013, US-048) applies to **coverage gaps**, not retroactive retargeting of pinned nodes for demand convenience.
+
+## Verification
+
+- NCA tests: unvalidated nodes never routed.
+- Demand-placement tests (when implemented): managed flag set; pinned nodes unchanged.
+- Documented in Qwen scratch PRD and NCA README cross-links.
--- a/docs/issues/20-memory-budget-shard-slots-and-dropout-relocation.md
+++ b/docs/issues/20-memory-budget-shard-slots-and-dropout-relocation.md
@@ -0,0 +1,10 @@
+Status: superseded
+
+# Superseded — renumbered to US-048
+
+This issue slot was a duplicate of tracker-node-hardening (US-020). Memory budget / shard slots / dropout relocation work lives at:
+
+- **Issue:** [48-memory-budget-shard-slots-and-dropout-relocation.md](./48-memory-budget-shard-slots-and-dropout-relocation.md)
+- **PRD:** `docs/prd.json` → `US-048`
+
+Do not implement from this file.
--- a/docs/issues/42-gguf-llamacpp-node-backend.md
+++ b/docs/issues/42-gguf-llamacpp-node-backend.md
@@ -35,7 +35,17 @@ to it (single-hop route). Smallest step, no cross-node activation work, and
 already useful: Strix Halo 128 GB serves DeepSeek-V4-Flash IQ3_XXS (114 GB)
 via llama.cpp Vulkan today.

-Recommended sequencing: C first (small, real value), then A/B investigation.
+Recommended sequencing: **C first** (US-042), then **ADR-0024 benchmark gate** (DGR-001), then distributed native worker (DGR-002+). Direction B (llama.cpp RPC) is rejected per ADR-0024.
+
+## Runtime sequencing
+
+| Stage | Track | Delivers |
+|---|---|---|
+| **C — Whole-model GGUF** | US-042 (this issue) | Single-hop llama.cpp, billing, relay streaming |
+| **0 — Benchmark gate** | ADR-0024 DGR-001 | Safetensors vs GGUF measured contract |
+| **1 — Distributed GGUF** | ADR-0024 `.scratch/distributed-gguf-runtime/` | gRPC C++ worker, layer-range GGUF |
+
+Phase C uses the existing tracker hop path (whole model, one node). ADR-0024 direction A (layer-range GGUF + activations) merges into the native worker track after the benchmark gate — not in parallel with phase C on the same backend without an integration plan.

 ## Also in scope

--- a/docs/issues/44-tracker-shard-source-partial-download.md
+++ b/docs/issues/44-tracker-shard-source-partial-download.md
@@ -86,10 +86,10 @@ What exists already (build on it, don't duplicate):
 - [ ] Two-machine test: machine A (tracker + node, holds full snapshot) serves
  layers 0–k; machine B joins with no model and receives **only** the files
  for its assigned range from A — nothing fetched from HF
- [ ] Machine B's resident memory scales with its shard size, not model size
- [ ] Checksums verified end-to-end; corrupted transfer falls back cleanly
+- [x] Machine B's resident memory scales with its shard size, not model size
+- [x] Checksums verified end-to-end; corrupted transfer falls back cleanly
 - [x] Single-node/full-model flows unchanged
- [ ] `python -m pytest` passes from repo root
+- [x] `python -m pytest` passes from repo root

 ## Implementation notes

@@ -98,6 +98,13 @@ What exists already (build on it, don't duplicate):
  `full_url`; HuggingFace remains fallback-only, and when it is used the node
  computes `allow_patterns` from the repo's remote SafeTensors index so it
  stays layer-filtered even without tracker-cached files. Remaining hard half
-  is true partial model materialization: the backend can prefer a downloaded
+  is partial model materialization: the backend can prefer a downloaded
  local model directory, but Transformers still needs a `meta`-device load
  path that materializes only assigned layers.
+- 2026-07-13: Partial LOAD implemented. `_load_partial_model_from_snapshot` builds
+  on `meta` via `init_empty_weights`, materializes only layer-scoped checkpoint
+  tensors, and finalizes device placement without copying unmaterialized meta
+  weights (`_finalize_active_shard_modules_on_device`). Tests cover memory
+  scaling (`test_partial_snapshot_resident_weight_numel_scales_with_shard`)
+  and real-torch meta-vs-materialized counts. Remaining: live two-machine LAN
+  verification.
--- a/docs/issues/49-mainnet-usdt-cutover-two-wallet-pilot.md
+++ b/docs/issues/49-mainnet-usdt-cutover-two-wallet-pilot.md
@@ -75,7 +75,7 @@ meshnet-tracker start \
 - [ ] `--starting-credit 0` — no free inference credit
 - [ ] Treasury keypair not committed to git; file mode 600
 - [ ] Plan multisig migration before large float ([alpha runbook](../../.scratch/alpha-hardening/runbooks/02-treasury-key-rotation.md) intent)
- [ ] Issue **21** (TOPLOC calibration) before production audit thresholds on untrusted nodes
+- [ ] Issue **21** (TOPLOC calibration) before production audit thresholds on untrusted nodes — runbook: [04-toploc-calibration-run](../../.scratch/alpha-hardening/runbooks/04-toploc-calibration-run.md)

 ## Cost estimate (this pilot)

--- a/docs/issues/50-qwen3.6-27b-demand-placement.md
+++ b/docs/issues/50-qwen3.6-27b-demand-placement.md
@@ -0,0 +1,15 @@
+Status: in-design
+
+# US-050 — Qwen3.6-27B demand-driven managed placement
+
+> Full spec: [.scratch/qwen3.6-27b-demand-placement/PRD.md](../../.scratch/qwen3.6-27b-demand-placement/PRD.md)  
+> Assignment rules: [ADR-0026](../adr/0026-node-assignment-ownership-and-managed-placement.md)  
+> Admission: [ADR-0023](../adr/0023-model-agnostic-node-capability-admission.md)
+
+## Summary
+
+Deploy `Qwen/Qwen3.6-27B` when chat demand appears and **spare** fleet capacity exists. Startup `--model` assignments stay **pinned**; tracker-managed loads fill gaps on model-less or (future US-048) unused slot capacity only.
+
+## Acceptance criteria
+
+See scratch PRD and `docs/prd.json` US-050.
--- a/docs/prd.json
+++ b/docs/prd.json
@@ -1,6 +1,6 @@
 {
  "name": "Distributed Inference Network",
-  "description": "Distributed inference network: base program US-001…US-035 complete; post-alpha friends-test and LAN-serving arc US-036…US-047 tracked here. Active scratch features (alpha hardening, NCA, distributed GGUF) have separate prd.json files under .scratch/.",
+  "description": "Distributed inference network: base program US-001…US-035 complete; friends-test arc US-036…US-049; capacity/placement US-048/050. Scratch features (alpha hardening AH-001…AH-025, NCA, distributed GGUF, Qwen demand) have separate prd.json files under .scratch/.",
  "branchName": "ralph/distributed-inference-network",
  "userStories": [
    {
@@ -851,11 +851,12 @@
        "python -m pytest passes from repo root"
      ],
      "priority": 37,
-      "status": "open",
+      "status": "done",
      "notes": "Source issue: docs/issues/37-relay-bridge-concurrency.md. Critical for public friends-test; blocks concurrent head + hop on same node.",
      "dependsOn": [
        "US-036"
-      ]
+      ],
+      "completionNotes": "ThreadPoolExecutor dispatch in relay_bridge.py; per-frame WS send lock; test_relay_bridge_serves_concurrent_requests; --relay-concurrency CLI flag sets MESHNET_RELAY_CONCURRENCY."
    },
    {
      "id": "US-038",
@@ -890,12 +891,13 @@
        "python -m pytest passes from repo root"
      ],
      "priority": 39,
-      "status": "open",
+      "status": "done",
      "notes": "Source issue: docs/issues/39-caller-credit-account-keys.md. Critical for friends-test inference.",
      "dependsOn": [
        "US-031",
        "US-035"
-      ]
+      ],
+      "completionNotes": "Caller credit granted once per account on first API key via deterministic event id; tests/test_accounts.py covers grant, revoke, and invented-bearer rejection."
    },
    {
      "id": "US-040",
@@ -908,11 +910,12 @@
        "python -m pytest passes from repo root"
      ],
      "priority": 40,
-      "status": "open",
+      "status": "done",
      "notes": "Source issue: docs/issues/40-devnet-dashboard-topup.md. Mainnet deployments set --devnet-topup 0.",
      "dependsOn": [
        "US-039"
-      ]
+      ],
+      "completionNotes": "POST /v1/account/topup with session auth and flag gating; tests/test_accounts.py covers flag off/on, own-account credit, and cross-account 403."
    },
    {
      "id": "US-041",
@@ -946,7 +949,7 @@
      ],
      "priority": 42,
      "status": "in-design",
-      "notes": "Source issue: docs/issues/42-gguf-llamacpp-node-backend.md. Distributed native path: ADR-0024. Sequencing: phase C before A/B investigation.",
+      "notes": "Source issue: docs/issues/42-gguf-llamacpp-node-backend.md. Phase C before ADR-0024 distributed worker; see runtime sequencing in issue file.",
      "dependsOn": [
        "US-036"
      ]
@@ -984,12 +987,12 @@
      ],
      "priority": 44,
      "status": "in-progress",
-      "notes": "Source issue: docs/issues/44-tracker-shard-source-partial-download.md. Download path largely implemented 2026-07-06; partial LOAD (meta-device materialization) and two-machine acceptance remain.",
+      "notes": "Source issue: docs/issues/44-tracker-shard-source-partial-download.md. Download path and partial LOAD implemented; live two-machine LAN verification remains.",
      "dependsOn": [
        "US-004",
        "US-012"
      ],
-      "completionNotes": "Tracker models-dir indexing, layer-scoped tar stream, HF allow_patterns client-side from remote index, per-file download API with retries, symlink dereference in tar writers. Remaining: true partial model load and live two-machine verification."
+      "completionNotes": "Tracker models-dir indexing, layer-scoped tar stream, HF allow_patterns client-side from remote index, per-file download API with retries, symlink dereference in tar writers. Partial LOAD via init_empty_weights + layer-scoped safetensors materialization; memory-scaling and checksum fallback tests pass. Remaining: live two-machine test (machine B receives only assigned files from A, no HF)."
    },
    {
      "id": "US-045",
@@ -1072,10 +1075,46 @@
        "US-033",
        "US-039"
      ]
+    },
+    {
+      "id": "US-048",
+      "title": "48 — Memory budget, shard slots, and dropout relocation hardening",
+      "description": "Harden the capacity contract around US-013 coverage-first assignment: enforce memory budget and max_loaded_shards in rebalance/dropout relocation, and decide whether one node may host multiple concurrent shard backends or max_loaded_shards remains metadata until runtime support lands.",
+      "acceptanceCriteria": [
+        "Assignment/rebalance never exceeds memory budget or max_loaded_shards",
+        "Dropout test restores full coverage without violating capacity limits",
+        "CLI --memory and --max-shards reflected in registration payload",
+        "python -m pytest tests/test_tracker_routing.py tests/test_node_startup.py passes"
+      ],
+      "priority": 48,
+      "status": "open",
+      "notes": "Source issue: docs/issues/48-memory-budget-shard-slots-and-dropout-relocation.md. Renumbered from duplicate slot 20. Enables spare shard slots for ADR-0026 managed placement.",
+      "dependsOn": [
+        "US-013"
+      ]
+    },
+    {
+      "id": "US-050",
+      "title": "50 — Qwen3.6-27B demand-driven managed placement",
+      "description": "Offer pinned Qwen/Qwen3.6-27B as a recommended text-only chat model. Valid chat requests prove demand; when spare fleet capacity exists, the tracker assigns managed nodes to reach complete coverage. Pinned startup assignments remain immutable per ADR-0026; NCA admission required before routable.",
+      "acceptanceCriteria": [
+        "First valid request for an uncovered variant queues demand and returns 503 model_loading until complete validated coverage exists",
+        "Managed assignments use only spare capacity and carry managed: true",
+        "Pinned startup assignments are never silently retargeted",
+        "Optional quantization field (bfloat16/int8/nf4) with coverage-vote UI semantics per scratch PRD",
+        "python -m pytest passes from repo root"
+      ],
+      "priority": 50,
+      "status": "in-design",
+      "notes": "Source: .scratch/qwen3.6-27b-demand-placement/PRD.md and docs/issues/50-qwen3.6-27b-demand-placement.md. Reconciled with ADR-0026 and ADR-0023.",
+      "dependsOn": [
+        "US-035",
+        "US-048"
+      ]
    }
  ],
  "metadata": {
-    "updatedAt": "2026-07-13T16:40:00.000Z",
+    "updatedAt": "2026-07-13T17:00:00.000Z",
    "statusVocabulary": {
      "open": "Not started",
      "in-design": "Decisions pending before implementation can begin",