Retry tracker registration when initial connect fails.

Start background re-registration when the tracker is unreachable at startup so nodes do not stay permanently unregistered.

Co-authored-by: Cursor <cursoragent@cursor.com>

.claude/memory/project-status.md

@@ -42,3 +42,7 @@ Historical handoff note: `/mnt/c/Users/popov/Downloads/neuron-tai-alpha-handoff-
 - Verification: downloader/startup targeted subset passes (`pytest tests/test_node_startup.py -k "download_shard or same_shard"`). Full `tests/test_node_startup.py` has 46 passed and 4 unrelated Windows chmod/path separator failures.
 - Live Windows confirmation: `meshnet-node start --tracker http://192.168.0.179:8080 --model Qwen3.6-35B-A3B` reuses `F:\_STORAGE\models\qwen3.6-35b-a3b`, prints `Cached at`, registers, and reaches ready as node `5gMLrmyB-26b1f8a4204a`.
 - Follow-up fix: preset-model startup now starts the heartbeat thread after registration; without this, the node appeared briefly on the dashboard and was purged on first inference/route after heartbeat expiry. Tracker dashboard now has a "Console output" panel backed by `/v1/console` for node register/expiry, routing failures, and proxy events.
+- Qwen3.6-35B-A3B reserve-based split is expected: an 79 GB CPU node may be assigned layers 0-36, and a second node fills 37-39. Do not "fix" this by bypassing the 20% assignment reserve unless the shard-planning policy changes.
+- Route hardening: tracker chat proxy and `/v1/route` diagnostics now use alias-aware preset node matching for split Qwen3.6 routes; dashboard derives grouped inference history from proxy route/complete console events and shows observed TPS after completion.
+- Live proxy hardening: model lookup trims outer whitespace before alias matching (`qwen3.6-35b-a3b ` resolves), and tracker route logs/dashboard queue depth combine heartbeat queue with tracker-local proxy in-flight counts so Postman-style bursts no longer show every selected route as queue `0`.
+- Split-shard streaming hardening: Qwen3.6-style distributed generation now emits SSE chunks token-by-token from the head node instead of buffering all generated text until completion. Tracker direct/relay stream proxy logs `proxy progress` with live tokens/TPS, dashboard Inference history shows currently processing requests with live TPS/tokens/queue, and relay stream completion no longer references an undefined `session_id`.

.gitignore

@@ -19,3 +19,4 @@ dist/
 !.env.testnet
 .rocm-local/*
 billing.sqlite
+.pytest-tmp/*

.scratch/distributed-gguf-runtime/PRD.md

@@ -0,0 +1,83 @@
+# PRD: Distributed GGUF Runtime
+
+## Summary
+
+Build a distributed inference runtime that can serve large, quality-first open models by combining torrent-style model artifact distribution with sticky multi-node Inference Routes and per-shard local Hot KV State.
+
+The first runtime proof uses the existing PyTorch route because it exposes model internals and cache semantics more directly. GGUF/llama.cpp becomes the performance path after the route-session contract is proven.
+
+## Goals
+
+- Eliminate full-prompt recompute in distributed decode.
+- Keep decode activation seams proportional to `hidden_size`, not `context_length * hidden_size`.
+- Keep Hot KV State local to the node serving the relevant Shard.
+- Stream token deltas when feasible and always expose Generation Telemetry.
+- Add a local full-model GGUF backend for immediate CPU performance wins.
+- Define Model Artifact manifests so nodes can verify, seed, and advertise artifacts without depending on Hugging Face at request time.
+- Prototype an upstreamable llama.cpp/libllama layer-boundary API.
+- Use DeepSeek-V4-Flash as the first serious large-model target after smaller protocol smoke tests.
+
+## Non-Goals
+
+- No centralized hot KV cache in the per-token decode path.
+- No automatic route repair in alpha.
+- No permanent llama.cpp fork as the intended architecture.
+- No GLM-5.2 or Ornith first; they remain follow-up support audits.
+- No transport rewrite to QUIC/WebRTC before route/session semantics are proven.
+
+## Resolved Decisions
+
+- Public-network Shards are contiguous transformer layer ranges.
+- Tensor/ring parallelism belongs inside one trusted node, one colocated pod, or a future composite node abstraction.
+- Hot KV State is local to route nodes; Prefix Snapshots are optional cold recovery/reuse artifacts.
+- PyTorch distributed KV/session semantics are proven before llama.cpp distributed execution.
+- Streaming responses are preferred; Generation Telemetry is mandatory.
+- llama.cpp/GGUF work targets upstreamable `libllama`/ggml hooks.
+- Alpha fails Route Sessions on route-node loss.
+- v1 activation transfer stays on binary HTTP.
+
+## Target User Experience
+
+A client sends an OpenAI-compatible request. The Gateway or Tracker Node accepts the request, creates a Route Session, and streams token deltas when supported. The client receives live Generation Telemetry for route phase, prefill progress, generated token count, rolling tokens/sec, route health, and failure reason.
+
+If a route node drops in alpha, the request fails clearly. A retry starts a new Route Session from scratch.
+
+## Runtime Shape
+
+```text
+client request
+  -> Gateway / Tracker Node creates Route Session
+  -> Tracker selects sticky Inference Route
+  -> prefill:
+       prompt chunks move through Shards
+       each node appends local Hot KV State
+  -> decode:
+       one-step activation moves through Shards
+       each node reads/appends local Hot KV State
+       tail returns token/logits
+  -> client receives streamed token deltas where possible
+  -> Generation Telemetry continues until complete or failed
+```
+
+## Milestones
+
+| Milestone | Outcome | Issues |
+|---|---|---|
+| M1 — Session protocol proof | Stub route has stable Route Sessions, prefill/decode split, telemetry, and streaming contract | 01, 02, 03 |
+| M2 — PyTorch reference route | Distributed PyTorch decode uses local per-shard cache and stops full-prompt recompute | 04 |
+| M3 — Local GGUF performance path | Single-node GGUF backend serves through the node API and reports backend metadata | 05 |
+| M4 — Artifact plane | Model Artifact manifest supports verification, layer mapping, and node advertisement | 06 |
+| M5 — llama.cpp collaboration proof | Localhost layer-boundary prototype identifies upstreamable llama.cpp/libllama API | 07 |
+| M6 — Networked GGUF route | Multi-node GGUF route uses the resolved protocol and fails cleanly on node loss | 08 |
+| M7 — First large model | DeepSeek-V4-Flash support path is audited and converted into follow-up runtime tasks | 09 |
+
+## Acceptance Criteria
+
+- A two-node route can prefill once and decode without resending full prompt activations.
+- Decode seam payload is one token/hidden-state step after prefill.
+- Route Session telemetry is visible before first token and during decode.
+- Streaming token deltas work where the backend supports them.
+- Route-node loss produces a structured alpha failure and does not attempt unsafe repair.
+- A local GGUF model can serve via the node API.
+- A Model Artifact manifest can prove which Shards a node can serve.
+- DeepSeek-V4-Flash has a written support recommendation: PyTorch, vLLM/SGLang, llama.cpp/GGUF, or blocked.

.scratch/distributed-gguf-runtime/README.md

@@ -15,7 +15,9 @@ This scratch supersedes the old assumption in [ADR-0001](../../docs/adr/0001-pyt
 | [decision-framework.md](./decision-framework.md) | Grilling framework for open decisions and recommended answers |
 | [research-prior-art.md](./research-prior-art.md) | Prior-art notes for Petals, exo, Distributed Llama, prima.cpp, llama.cpp, DeepSeek-V4-Flash, GLM-5.2, and Ornith |
 | [ADR-0020-distributed-gguf-runtime.md](./ADR-0020-distributed-gguf-runtime.md) | Draft decision record for the GGUF/llama.cpp distributed runtime |
-| [issues/](./issues/) | Implementation slices in dependency order |
+| [PRD.md](./PRD.md) | Product/runtime requirements and acceptance criteria |
+| [milestones.md](./milestones.md) | Dependency-ordered implementation milestones |
+| [issues/](./issues/) | Implementation-ready tracer-bullet issue briefs |
 
 ## Decision Summary
 
@@ -40,15 +42,18 @@ Adopt a hybrid runtime:
 
 ## Recommended Order
 
-1. Local llama.cpp/GGUF backend for full-model serving.
-2. Stable distributed session ID and per-shard KV cache in the existing PyTorch path.
-3. Binary prefill/decode protocol split: chunked prefill, one-step decode.
-4. Route-session Generation Telemetry and streaming response support where feasible.
-5. GGUF artifact manifest and torrent seeding.
-6. llama.cpp layer-boundary prototype on localhost.
-7. Networked distributed GGUF route.
-8. DeepSeek-V4-Flash as first serious large-model target.
-9. GLM-5.2 / DSA / MLA and Ornith support once runtime support is confirmed.
+See [milestones.md](./milestones.md) for the full dependency map.
+
+1. [01 — Route Session lifecycle](./issues/01-route-session-lifecycle.md)
+2. [02 — Prefill/decode binary HTTP protocol](./issues/02-prefill-decode-binary-http.md)
+3. [03 — Generation Telemetry and streaming response contract](./issues/03-generation-telemetry-and-streaming.md)
+4. [04 — PyTorch distributed KV reference route](./issues/04-pytorch-distributed-kv-reference.md)
+5. [05 — Local llama.cpp/GGUF backend](./issues/05-local-llamacpp-gguf-backend.md)
+6. [06 — Model Artifact manifest and Shard advertisement](./issues/06-model-artifact-manifest.md)
+7. [07 — llama.cpp layer-boundary prototype](./issues/07-llamacpp-layer-boundary-prototype.md)
+8. [08 — Networked distributed GGUF route](./issues/08-networked-distributed-gguf-route.md)
+9. [09 — DeepSeek-V4-Flash support audit](./issues/09-deepseek-v4-flash-support-audit.md)
+10. [10 — GLM-5.2 and Ornith follow-up support audit](./issues/10-glm52-ornith-followup-audit.md)
 
 ## Open Questions
 

.scratch/distributed-gguf-runtime/issues/01-local-llamacpp-gguf-backend.md

@@ -1,29 +0,0 @@
-# 01 — Local llama.cpp/GGUF backend
-
-Status: ready-for-agent
-
-## Goal
-
-Add a local full-model llama.cpp/GGUF backend to the node so a machine that can hold a GGUF model can serve it through the existing OpenAI-compatible node API.
-
-## Scope
-
-- Add backend selection for `llama.cpp` / GGUF.
-- Support launching or calling `llama-server` first; direct `libllama` bindings may come later.
-- Register model metadata and hardware profile with tracker.
-- Preserve current PyTorch path.
-- Add a local benchmark comparing PyTorch CPU vs llama.cpp/GGUF for the same supported small model.
-
-## Non-Goals
-
-- No distributed GGUF route yet.
-- No partial layer loading yet.
-- No torrent seeding yet.
-
-## Acceptance
-
-- A local GGUF model can answer `/v1/chat/completions`.
-- Startup output clearly says backend=`llama.cpp`.
-- Node registration includes backend and artifact metadata.
-- Test or smoke script verifies the backend wiring without requiring a huge model.
-

.scratch/distributed-gguf-runtime/issues/01-route-session-lifecycle.md

@@ -0,0 +1,19 @@
+# 01 — Route Session lifecycle
+
+Status: ready-for-agent
+
+## What to build
+
+Add the narrowest end-to-end Route Session lifecycle that can be used by distributed inference routes: create a session, bind it to a selected Inference Route, expose status, and close it cleanly. This slice does not need real model cache yet; it proves stable session identity across the control plane and activation plane.
+
+## Acceptance criteria
+
+- [ ] A request can create a Route Session with a stable `session_id`, `route_id`, model preset, backend id, and route membership.
+- [ ] Every downstream activation request carries the same session identity and fails clearly if the session or route id does not match.
+- [ ] Session status reports phase, route nodes, model preset, backend id, created time, and last activity time.
+- [ ] Closing a session releases all registered per-session state.
+- [ ] Tests cover create, status, close, stale-session rejection, and wrong-route rejection.
+
+## Blocked by
+
+None - can start immediately.

.scratch/distributed-gguf-runtime/issues/02-prefill-decode-binary-http.md

@@ -0,0 +1,20 @@
+# 02 — Prefill/decode binary HTTP protocol
+
+Status: ready-for-agent
+
+## What to build
+
+Split the activation protocol into explicit prefill and decode-step calls using the existing binary HTTP direction from ADR-0008. The completed slice should work against a stub backend so payload shape, route/session headers, relay preservation, and failure behavior are testable before real KV cache work begins.
+
+## Acceptance criteria
+
+- [ ] Prefill accepts chunked binary activations with route/session metadata and forwards them through the selected route.
+- [ ] Decode-step accepts a one-step binary activation and forwards a one-step activation through the selected route.
+- [ ] Decode-step payload size is independent of prompt length in protocol tests.
+- [ ] Relay forwarding preserves route/session headers, shape, dtype, position, and wire version.
+- [ ] Legacy `/forward` either remains as a compatibility wrapper or fails with a clear wire-version error.
+- [ ] Tests cover prefill chunking, decode-step shape validation, relay preservation, and malformed header rejection.
+
+## Blocked by
+
+- 01 — Route Session lifecycle.

.scratch/distributed-gguf-runtime/issues/02-stable-session-and-distributed-kv-pytorch.md

@@ -1,33 +0,0 @@
-# 02 — Stable session and distributed KV in PyTorch path
-
-Status: ready-for-agent
-
-## Goal
-
-Fix the existing distributed PyTorch path so it does not recompute the full growing prompt for every output token.
-
-## Scope
-
-- Introduce stable `session_id` for one request/session.
-- Add per-node session cache keyed by `session_id`.
-- Split `/forward` semantics into prefill and decode-step.
-- Use model cache objects / `past_key_values` where supported.
-- Keep hot KV local to each shard node.
-- Add cleanup/TTL for abandoned sessions.
-
-## Current Problem
-
-The current distributed path:
-
-- calls `encode_prompt(current_text)` for every generated token
-- sends full-sequence activations through the route
-- calls layers with `use_cache=False`
-- creates a fresh UUID inside `_run_downstream_pipeline()`
-
-## Acceptance
-
-- Decode seam payload is one token / one hidden state after prefill.
-- Per-shard cache grows locally with generated tokens.
-- Regression test proves layer calls use cache after prefill.
-- Fallback error is explicit for models whose manual cache API is unsupported.
-

.scratch/distributed-gguf-runtime/issues/03-generation-telemetry-and-streaming.md

@@ -0,0 +1,21 @@
+# 03 — Generation Telemetry and streaming response contract
+
+Status: ready-for-agent
+
+## What to build
+
+Expose realtime Generation Telemetry for active Route Sessions and stream token deltas when the serving path can produce them. This slice should make long distributed requests observable before real large-model work begins.
+
+## Acceptance criteria
+
+- [ ] A client can observe route-session phase changes: queued, loading, prefill, decode, finalizing, completed, failed.
+- [ ] Telemetry includes prefill progress, generated token count, rolling tokens/sec, average tokens/sec, active route nodes, and failure reason.
+- [ ] Telemetry is available before the first output token.
+- [ ] A streaming response can include token deltas while telemetry remains available.
+- [ ] A non-streaming fallback still exposes telemetry until final answer or failure.
+- [ ] Route-node failure reports the last known phase and reason.
+- [ ] Tests cover telemetry updates, streaming token deltas, non-streaming fallback, and structured failure closeout.
+
+## Blocked by
+
+- 01 — Route Session lifecycle.

.scratch/distributed-gguf-runtime/issues/03-prefill-decode-wire-protocol.md

@@ -1,29 +0,0 @@
-# 03 — Prefill/decode wire protocol
-
-Status: ready-for-agent
-
-## Goal
-
-Define and implement the activation protocol needed by both PyTorch and future GGUF backends.
-
-## Scope
-
-- Add route/session lifecycle headers.
-- Separate `prefill` from `decode-step`.
-- Keep binary bfloat16 activation bodies.
-- Preserve relay compatibility.
-- Add route id and model artifact hash validation.
-
-## Draft Endpoints
-
-- `POST /sessions/{session_id}/prefill`
-- `POST /sessions/{session_id}/decode-step`
-- `DELETE /sessions/{session_id}`
-- `GET /sessions/{session_id}/status`
-
-## Acceptance
-
-- Old `/forward` remains temporarily or fails with clear version message.
-- Tests cover relay preservation of session headers.
-- Decode-step payload is independent of prompt length.
-

.scratch/distributed-gguf-runtime/issues/04-model-artifact-manifest-and-torrent.md

@@ -1,24 +0,0 @@
-# 04 — Model artifact manifest and torrent distribution
-
-Status: ready-for-agent
-
-## Goal
-
-Represent model artifacts independently from runtime routes so nodes can seed, verify, and advertise model files without relying on Hugging Face at runtime.
-
-## Scope
-
-- Define manifest schema.
-- Include file/chunk hashes.
-- Include tensor/layer map where available.
-- Include tokenizer and chat template hashes.
-- Include backend compatibility.
-- Add torrent/magnet URI fields and HTTP fallback URLs.
-- Extend node registration with artifact availability.
-
-## Acceptance
-
-- A model can be registered from a manifest without contacting Hugging Face.
-- Tracker can show coverage by artifact and layer range.
-- Node refuses to advertise corrupt artifacts.
-

.scratch/distributed-gguf-runtime/issues/04-pytorch-distributed-kv-reference.md

@@ -0,0 +1,23 @@
+# 04 — PyTorch distributed KV reference route
+
+Status: ready-for-agent
+
+## What to build
+
+Fix the existing distributed PyTorch route so it uses the Route Session and prefill/decode protocol to keep Hot KV State local to each Shard node. The visible behavior is that prefill processes the prompt once, and decode no longer recomputes or resends the full growing prompt for every token.
+
+## Acceptance criteria
+
+- [ ] Distributed PyTorch prefill stores per-session cache/state on each Shard node.
+- [ ] Distributed PyTorch decode-step reads and appends local per-shard cache/state.
+- [ ] Decode activation seam payload is one token/hidden-state step after prefill.
+- [ ] The old full-growing-prompt decode loop is not used for models that support the reference cache path.
+- [ ] Unsupported model/cache APIs fail with an explicit backend capability error.
+- [ ] Session close or TTL cleanup releases per-shard cache.
+- [ ] Regression tests prove decode does not call the full prompt encoder for every generated token.
+
+## Blocked by
+
+- 01 — Route Session lifecycle.
+- 02 — Prefill/decode binary HTTP protocol.
+- 03 — Generation Telemetry and streaming response contract.

.scratch/distributed-gguf-runtime/issues/05-llamacpp-layer-boundary-prototype.md

@@ -1,32 +0,0 @@
-# 05 — llama.cpp layer-boundary prototype
-
-Status: ready-for-human
-
-## Goal
-
-Prototype whether llama.cpp can execute only a selected layer range and accept/return hidden activations at model layer boundaries.
-
-## Scope
-
-- Start with a small model already supported by llama.cpp.
-- Run two local processes: head and tail.
-- Head owns embeddings + early layers.
-- Tail owns later layers + norm/lm_head.
-- Prefill once, then decode using local per-process KV.
-
-## Collaboration Point
-
-This is the best place to collaborate with Georgi/upstream llama.cpp. The desired upstream API shape:
-
-- load layer range or mmap full GGUF but execute layer range
-- run prefill chunk from inbound hidden states
-- run decode step from inbound hidden state
-- expose per-session KV/state handles
-- report cache memory budget
-
-## Acceptance
-
-- Localhost two-process decode does not recompute full prompt per token.
-- Seam payload after prefill is one hidden state per token.
-- No long-lived fork-only hooks unless upstream path is infeasible.
-

.scratch/distributed-gguf-runtime/issues/05-local-llamacpp-gguf-backend.md

@@ -0,0 +1,20 @@
+# 05 — Local llama.cpp/GGUF backend
+
+Status: ready-for-agent
+
+## What to build
+
+Add a local full-model GGUF backend so a node that can hold a GGUF model can serve it through the existing node API. This is the immediate CPU-performance path and the baseline for later distributed llama.cpp work.
+
+## Acceptance criteria
+
+- [ ] A node can start with backend `llama.cpp` or `gguf` for a local full-model GGUF artifact.
+- [ ] The node can answer an OpenAI-compatible chat completion through the existing API.
+- [ ] Startup and registration clearly report backend, quantization/artifact metadata, context cap, and local model path.
+- [ ] The PyTorch backend remains unchanged and selectable.
+- [ ] A smoke test or script validates backend wiring with a small GGUF model or a stubbed llama.cpp process.
+- [ ] A benchmark command can compare local PyTorch CPU and local GGUF CPU for the same small supported model when both are available.
+
+## Blocked by
+
+None - can start immediately.

.scratch/distributed-gguf-runtime/issues/06-model-artifact-manifest.md

@@ -0,0 +1,20 @@
+# 06 — Model Artifact manifest and Shard advertisement
+
+Status: ready-for-agent
+
+## What to build
+
+Introduce a Model Artifact manifest that separates storage distribution from route execution. A node should be able to verify local model files, determine which Shards it can serve, and advertise artifact/layer availability to the Tracker without contacting Hugging Face at request time.
+
+## Acceptance criteria
+
+- [ ] Manifest records model preset, upstream revision, license, backend support, quantization, context cap, tokenizer artifacts, file hashes, piece hashes, and tensor/layer mapping where available.
+- [ ] A node can verify local artifacts against the manifest and reject corrupt or incomplete artifacts.
+- [ ] A node can derive advertised Shard ranges from the manifest and local files.
+- [ ] Tracker registration can include artifact id, backend id, Shard range, and verification status.
+- [ ] Tracker coverage can distinguish model-layer coverage from artifact availability.
+- [ ] Tests cover valid manifest registration, corrupt artifact rejection, and missing layer/tensor metadata.
+
+## Blocked by
+
+- 01 — Route Session lifecycle.

.scratch/distributed-gguf-runtime/issues/06-networked-distributed-gguf-route.md

@@ -1,24 +0,0 @@
-# 06 — Networked distributed GGUF route
-
-Status: pending
-
-Depends on: 01, 03, 04, 05
-
-## Goal
-
-Run a GGUF-backed model over a real multi-node route using the tracker-selected route and per-shard local KV.
-
-## Scope
-
-- Extend node backend registry with GGUF layer ranges.
-- Add route selection for GGUF nodes.
-- Use the prefill/decode protocol.
-- Track route health and queue depth.
-- Bill by layer work and token work.
-
-## Acceptance
-
-- Two physical machines can serve one model route.
-- Node dropout during alpha fails request cleanly.
-- Tracker metrics show prefill TPS, decode TPS, seam latency, and cache memory.
-

.scratch/distributed-gguf-runtime/issues/07-glm52-ornith-support-audit.md

@@ -1,29 +0,0 @@
-# 07 — Large-model support audit
-
-Status: pending
-
-Depends on: 01, 05
-
-## Goal
-
-Determine which large target models can run through the distributed path and what upstream runtime work remains.
-
-The first serious large-model target is `deepseek-ai/DeepSeek-V4-Flash`. GLM-5.2 and Ornith remain follow-up targets.
-
-## Scope
-
-- Verify PyTorch/Transformers load semantics for DeepSeek-V4-Flash.
-- Verify vLLM/SGLang serving support for DeepSeek-V4-Flash.
-- Verify whether a GGUF/llama.cpp quantization path exists for DeepSeek-V4-Flash.
-- Estimate artifact size and 128K KV/cache memory by layer range for DeepSeek-V4-Flash.
-- Verify llama.cpp/GGUF support for `glm_moe_dsa`.
-- Verify cache accounting for GLM-5.2 DSA/MLA.
-- Verify Ornith/Qwen3.5-MoE hybrid attention support.
-- Identify smallest viable quantization for quality-first use.
-
-## Acceptance
-
-- Written compatibility matrix.
-- Clear "supported now / upstream needed / not viable" status per model.
-- DeepSeek-V4-Flash has a recommended first-runtime path: PyTorch, vLLM/SGLang, llama.cpp/GGUF, or blocked.
-- Runtime blockers converted into issues or upstream collaboration notes.

.scratch/distributed-gguf-runtime/issues/07-llamacpp-layer-boundary-prototype.md

@@ -0,0 +1,25 @@
+# 07 — llama.cpp layer-boundary prototype
+
+Status: ready-for-human
+
+## What to build
+
+Build a local prototype that proves whether llama.cpp/libllama can support the platform's distributed execution contract: execute a selected layer range, accept inbound hidden states, emit outbound hidden states, and own per-session cache for only the loaded/served range.
+
+This is the collaboration package for upstream llama.cpp. The target is an upstreamable API shape, not a permanent fork.
+
+## Acceptance criteria
+
+- [ ] A small llama.cpp-supported GGUF model can be split into a two-process localhost head/tail prototype.
+- [ ] The head process runs embeddings and early layers, then emits hidden states at an Activation Seam.
+- [ ] The tail process accepts hidden states, runs later layers plus output head, and produces logits/tokens comparable to single-process execution.
+- [ ] Prefill is performed once and decode-step seam payload is one hidden-state step per generated token.
+- [ ] Each process owns only its own per-session cache/state.
+- [ ] The prototype records the minimum upstream API needed for layer-range execution, hidden-state I/O, partial loading/introspection, and per-session KV ownership.
+- [ ] If upstream support is unavailable, the issue ends with a concrete recommendation: upstream proposal, narrow adapter fork, or keep GGUF distribution local-only for now.
+
+## Blocked by
+
+- 02 — Prefill/decode binary HTTP protocol.
+- 05 — Local llama.cpp/GGUF backend.
+- 06 — Model Artifact manifest and Shard advertisement.

.scratch/distributed-gguf-runtime/issues/08-generation-telemetry.md

@@ -1,28 +0,0 @@
-# Issue 08: Route-Session Generation Telemetry
-
-## Goal
-
-Expose realtime progress for long-running distributed inference requests. This is required whether or not token output is streamed.
-
-## Background
-
-Streaming token deltas is the preferred client experience when the backend and transport support it. Users still need realtime confidence that the route is alive and useful speed feedback during prefill, queueing, and any non-streaming fallback path.
-
-## Scope
-
-- Define a route-session telemetry schema.
-- Track phase: queued, loading, prefill, decode, finalizing, failed.
-- Track prefill token progress.
-- Track generated token count.
-- Track rolling and average tokens/sec.
-- Track active route nodes and failure reason.
-- Expose telemetry by SSE, WebSocket, or polling.
-- Ensure telemetry can coexist with streamed token deltas.
-
-## Acceptance Criteria
-
-- A client can display live route progress before the first output token is available.
-- During decode, the client sees rolling tokens/sec.
-- A streaming response can include token deltas and telemetry.
-- A non-streaming fallback still provides progress telemetry until final answer or failure.
-- Route failures include the last known phase and reason.

.scratch/distributed-gguf-runtime/issues/08-networked-distributed-gguf-route.md

@@ -0,0 +1,24 @@
+# 08 — Networked distributed GGUF route
+
+Status: pending
+
+## What to build
+
+Run a GGUF-backed model over a real multi-node Inference Route using the resolved Route Session, binary HTTP prefill/decode protocol, local Hot KV State, Generation Telemetry, and alpha fail-fast behavior.
+
+## Acceptance criteria
+
+- [ ] Two machines can form one GGUF-backed Inference Route over contiguous Shards.
+- [ ] Prefill builds local per-shard cache/state and decode-step uses one-step seam payloads.
+- [ ] The client receives streamed token deltas when supported by the GGUF path.
+- [ ] The client receives Generation Telemetry for phase, generated tokens, tokens/sec, route health, and failure reason.
+- [ ] Route-node loss fails the Route Session cleanly; no automatic repair is attempted in alpha.
+- [ ] Tracker metrics show prefill tokens/sec, decode tokens/sec, seam latency, queue depth, and cache memory by node.
+- [ ] Billing/audit records identify route membership and layer/token work for the completed or failed session.
+
+## Blocked by
+
+- 03 — Generation Telemetry and streaming response contract.
+- 04 — PyTorch distributed KV reference route.
+- 06 — Model Artifact manifest and Shard advertisement.
+- 07 — llama.cpp layer-boundary prototype.

.scratch/distributed-gguf-runtime/issues/09-deepseek-v4-flash-support-audit.md

@@ -0,0 +1,21 @@
+# 09 — DeepSeek-V4-Flash support audit
+
+Status: ready-for-agent
+
+## What to build
+
+Audit `deepseek-ai/DeepSeek-V4-Flash` as the first serious large-model target after the small GGUF protocol smoke test. The output is a compatibility matrix and a recommended runtime path, not full production support.
+
+## Acceptance criteria
+
+- [ ] Verify current PyTorch/Transformers load and generation semantics for DeepSeek-V4-Flash from primary model documentation.
+- [ ] Verify vLLM and SGLang support status from primary runtime documentation or release notes.
+- [ ] Verify whether a GGUF/llama.cpp quantization path exists or would need upstream work.
+- [ ] Estimate artifact size, active parameter behavior, and 128K cache memory by Shard range.
+- [ ] Identify required backend capability flags for the Tracker.
+- [ ] Produce a compatibility matrix: PyTorch, vLLM, SGLang, llama.cpp/GGUF.
+- [ ] End with one recommendation: first runtime path, blocked pending upstream, or defer.
+
+## Blocked by
+
+None - can start immediately.

.scratch/distributed-gguf-runtime/issues/09-streaming-response.md

@@ -1,24 +0,0 @@
-# Issue 09: Streaming Response Support
-
-## Goal
-
-Stream generated token deltas to clients when the backend and transport support it, while preserving Generation Telemetry as an independent progress channel.
-
-## Background
-
-The preferred client experience is streamed output plus live tokens/sec feedback. Some early route proofs or backend integrations may only support a final response, so telemetry remains mandatory even when token deltas are unavailable.
-
-## Scope
-
-- Define an OpenAI-compatible streaming response shape.
-- Decide whether token deltas and telemetry travel over the same SSE stream or separate channels.
-- Preserve non-streaming final-response mode for simple clients.
-- Ensure prefill progress is visible before first token delta.
-- Ensure route failures close streams with a structured error and last known telemetry.
-
-## Acceptance Criteria
-
-- A client can request streamed token deltas.
-- A client can receive Generation Telemetry before and during streamed decode.
-- Non-streaming clients still receive telemetry through the route-session telemetry endpoint.
-- Stream failure includes session id, phase, and failure reason.

.scratch/distributed-gguf-runtime/issues/10-glm52-ornith-followup-audit.md

@@ -0,0 +1,20 @@
+# 10 — GLM-5.2 and Ornith follow-up support audit
+
+Status: pending
+
+## What to build
+
+Audit GLM-5.2 and Ornith after the smaller protocol smoke path and DeepSeek-V4-Flash audit. The output is a follow-up compatibility matrix focused on architecture/runtime blockers: DSA/MLA, hybrid attention, cache accounting, and GGUF/llama.cpp support.
+
+## Acceptance criteria
+
+- [ ] Verify GLM-5.2 PyTorch/Transformers serving requirements and cache semantics from primary model documentation.
+- [ ] Verify llama.cpp/GGUF support status for `glm_moe_dsa` or equivalent architecture support.
+- [ ] Verify Ornith/Qwen3.5-MoE and hybrid attention support status in the candidate runtimes.
+- [ ] Estimate artifact size and 128K cache memory by Shard range for each model.
+- [ ] Identify smallest quality-preserving quantization worth testing.
+- [ ] Convert each runtime blocker into a follow-up issue or upstream collaboration note.
+
+## Blocked by
+
+- 09 — DeepSeek-V4-Flash support audit.

.scratch/distributed-gguf-runtime/milestones.md

@@ -0,0 +1,32 @@
+# Distributed GGUF Runtime Milestones
+
+## Proposed Breakdown
+
+| Order | Issue | Title | Blocked by | User-visible proof |
+|---:|---|---|---|---|
+| 1 | [01](./issues/01-route-session-lifecycle.md) | Route Session lifecycle | None | Stable route/session status and cleanup |
+| 2 | [02](./issues/02-prefill-decode-binary-http.md) | Prefill/decode binary HTTP protocol | 01 | Stub route proves prefill chunks and one-step decode payloads |
+| 3 | [03](./issues/03-generation-telemetry-and-streaming.md) | Generation Telemetry and streaming response contract | 01 | Client sees route progress and streamed deltas when available |
+| 4 | [04](./issues/04-pytorch-distributed-kv-reference.md) | PyTorch distributed KV reference route | 01, 02, 03 | Distributed PyTorch decode stops full-prompt recompute |
+| 5 | [05](./issues/05-local-llamacpp-gguf-backend.md) | Local llama.cpp/GGUF backend | None | Local GGUF model serves through node API |
+| 6 | [06](./issues/06-model-artifact-manifest.md) | Model Artifact manifest and Shard advertisement | 01 | Node verifies artifacts and advertises serveable Shards |
+| 7 | [07](./issues/07-llamacpp-layer-boundary-prototype.md) | llama.cpp layer-boundary prototype | 02, 05, 06 | Local two-process GGUF route identifies upstream API |
+| 8 | [08](./issues/08-networked-distributed-gguf-route.md) | Networked distributed GGUF route | 03, 04, 06, 07 | Two machines serve one GGUF route with telemetry |
+| 9 | [09](./issues/09-deepseek-v4-flash-support-audit.md) | DeepSeek-V4-Flash support audit | None | Runtime recommendation for first serious large model |
+| 10 | [10](./issues/10-glm52-ornith-followup-audit.md) | GLM-5.2 and Ornith follow-up support audit | 09 | Follow-up compatibility matrix and upstream blockers |
+
+## First Three To Implement
+
+1. **01 — Route Session lifecycle**: makes every later cache, telemetry, and route decision concrete.
+2. **02 — Prefill/decode binary HTTP protocol**: proves the payload shape and route/session headers before model internals.
+3. **03 — Generation Telemetry and streaming response contract**: gives every later long-running route a visible user experience and failure surface.
+
+## Parallel Work
+
+- **05 — Local llama.cpp/GGUF backend** can run in parallel with 01–03 because it is a full-model local backend.
+- **09 — DeepSeek-V4-Flash support audit** can run in parallel because it is research/compatibility work.
+
+## Human-Gated Work
+
+- **07 — llama.cpp layer-boundary prototype** is the collaboration point with Georgi/upstream llama.cpp.
+- **08 — Networked distributed GGUF route** should wait until the PyTorch reference route proves the cache/session contract.

QUICKSTART.md

@@ -98,6 +98,7 @@ Nodes can then join with either the LAN tracker URL or the public URL:
 ```bash
 .venv/bin/meshnet-node start --tracker http://192.168.0.179:8080 --model Qwen/Qwen2.5-0.5B-Instruct
 .venv/bin/meshnet-node start --tracker https://ai.neuron.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct
+.venv/bin/meshnet-node start --tracker https://ai.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct
 ```
 
 ### Windows / WSL2

docs/issues/44-tracker-shard-source-partial-download.md

@@ -75,7 +75,7 @@ What exists already (build on it, don't duplicate):
 - [x] Node downloader keeps exact-shard peers first, then races tracker model
   sources against a HuggingFace `snapshot_download(..., allow_patterns=...)`
   subset download, using the first successful source.
-- [ ] When no tracker model source is available at all, the HuggingFace
+- [x] When no tracker model source is available at all, the HuggingFace
   fallback still computes `allow_patterns` from the repo's own
   `model.safetensors.index.json` (fetched directly, not via the tracker) —
   it never silently downloads the full model just because the tracker has
@@ -95,7 +95,9 @@ What exists already (build on it, don't duplicate):
 
 - 2026-07-06: Added the tracker/node download path. For immediate Qwen3.6-35B
   LAN testing, real PyTorch nodes fetch the full snapshot from the tracker via
-  `full_url` and race HuggingFace as fallback. Remaining hard half is true
-  partial model materialization: the backend can prefer a downloaded local
-  model directory, but Transformers still needs a `meta`-device load path that
-  materializes only assigned layers.
+  `full_url`; HuggingFace remains fallback-only, and when it is used the node
+  computes `allow_patterns` from the repo's remote SafeTensors index so it
+  stays layer-filtered even without tracker-cached files. Remaining hard half
+  is true partial model materialization: the backend can prefer a downloaded
+  local model directory, but Transformers still needs a `meta`-device load
+  path that materializes only assigned layers.

packages/node/meshnet_node/cli.py

@@ -52,7 +52,7 @@ def _run_node(cfg: dict) -> None:
         node = run_startup(
             tracker_url=cfg["tracker_url"],
             port=cfg.get("port", 7000),
-            model=cfg.get("model_name") or "stub-model",
+            model=cfg.get("model_name") or None,
             model_id=cfg.get("model_hf_repo") or None,
             shard_start=cfg.get("shard_start"),
             shard_end=cfg.get("shard_end"),
@@ -90,6 +90,19 @@ def _run_node(cfg: dict) -> None:
         )
 
 
+def _resolve_model_flags(
+    model: str | None,
+    model_id: str | None,
+) -> tuple[str | None, str | None]:
+    """Return (model_name, hf_repo_or_none) from --model / --model-id flags."""
+    explicit = model_id or model
+    if not explicit:
+        return None, None
+    if "/" in explicit:
+        return explicit.split("/")[-1], explicit
+    return explicit, None
+
+
 def _first_available_port(host: str, start: int = 7000, attempts: int = 100) -> int:
     """Return the first TCP port bindable on host, starting at start."""
     bind_host = "" if host == "0.0.0.0" else host
@@ -122,9 +135,10 @@ def _cmd_default(args) -> int:
 
     # Apply CLI overrides on top of saved config
     overrides: dict = {}
-    if args.model:
-        overrides["model_hf_repo"] = args.model
-        overrides["model_name"] = args.model.split("/")[-1]
+    model_name, hf_repo = _resolve_model_flags(args.model, getattr(args, "model_id", None))
+    if model_name is not None:
+        overrides["model_name"] = model_name
+        overrides["model_hf_repo"] = hf_repo or ""
     if args.quantization:
         overrides["quantization"] = args.quantization
     if args.download_dir:
@@ -215,16 +229,15 @@ def _cmd_start(args) -> int:
     if args.tracker:
         cfg["tracker_url"] = args.tracker
     cfg["port"] = args.port if args.port is not None else _first_available_port(args.host)
-    model = args.model or cfg.get("model_hf_repo") or cfg.get("model_name") or "stub-model"
-    if args.model_id is None and "/" in model:
-        cfg["model_hf_repo"] = model
-        cfg["model_name"] = model.split("/")[-1]
-    else:
-        cfg["model_name"] = model
+    model_name, hf_repo = _resolve_model_flags(
+        args.model or cfg.get("model_hf_repo") or cfg.get("model_name") or None,
+        args.model_id,
+    )
+    if model_name is not None:
+        cfg["model_name"] = model_name
+        cfg["model_hf_repo"] = hf_repo or ""
     cfg["quantization"] = args.quantization
     cfg["host"] = args.host
-    if args.model_id:
-        cfg["model_hf_repo"] = args.model_id
     if args.shard_start is not None:
         cfg["shard_start"] = args.shard_start
     if args.shard_end is not None:
@@ -242,7 +255,7 @@ def _cmd_start(args) -> int:
             tracker_url=cfg["tracker_url"],
             port=cfg["port"],
             model=cfg["model_name"],
-            model_id=cfg.get("model_hf_repo"),
+            model_id=cfg.get("model_hf_repo") or None,
             shard_start=cfg.get("shard_start"),
             shard_end=cfg.get("shard_end"),
             quantization=cfg["quantization"].replace("bf16", "bfloat16"),
@@ -288,7 +301,8 @@ def main() -> None:
     )
 
     # Flags that apply to the no-subcommand (default) path
-    parser.add_argument("--model", metavar="HF_REPO", help="HuggingFace repo ID to serve")
+    parser.add_argument("--model", metavar="MODEL", help="Model name or HuggingFace repo ID to serve")
+    parser.add_argument("--model-id", metavar="MODEL", help="Alias for --model (catalog name or HuggingFace repo)")
     parser.add_argument("--quantization", "-q", choices=["bf16", "int8", "nf4", "bfloat16"],
                         help="Quantization level")
     parser.add_argument("--download-dir", metavar="PATH", help="Model download directory")
@@ -329,8 +343,8 @@ def main() -> None:
     start_cmd = subparsers.add_parser("start", help="Start node (legacy flags)")
     start_cmd.add_argument("--tracker")
     start_cmd.add_argument("--port", type=int)
-    start_cmd.add_argument("--model")
-    start_cmd.add_argument("--model-id", help="HuggingFace repo ID")
+    start_cmd.add_argument("--model", help="Model name or HuggingFace repo ID")
+    start_cmd.add_argument("--model-id", help="Alias for --model (catalog name or HuggingFace repo)")
     start_cmd.add_argument("--shard-start", type=int)
     start_cmd.add_argument("--shard-end", type=int)
     start_cmd.add_argument("--quantization", choices=["auto", "bfloat16", "int8", "nf4", "bf16"], default="auto")

packages/node/meshnet_node/downloader.py

@@ -1,4 +1,4 @@
-"""Shard downloader — fetches model shards from peers or HuggingFace Hub.
+"""Shard downloader — fetches model files from peers, tracker sources, or HuggingFace.
 
 Cache layout: ~/.cache/meshnet/shards/<model>/
 

packages/node/meshnet_node/model_backend.py

@@ -4,6 +4,7 @@ from __future__ import annotations
 
 import base64
 from dataclasses import dataclass
+import json
 from pathlib import Path
 from typing import Any, Literal
 
@@ -22,6 +23,10 @@ class InsufficientVRAMError(ModelBackendError):
     """Raised when a requested shard cannot fit in available CUDA memory."""
 
 
+class PartialModelLoadUnsupported(ModelBackendError):
+    """Raised when a shard cannot be materialized from a local snapshot subset."""
+
+
 @dataclass(frozen=True)
 class TensorPayload:
     body: bytes
@@ -94,20 +99,39 @@ class TorchModelShard:
             None if load_source != model_id else cache_dir,
         )
         try:
-            load_kwargs = {
-                "device_map": "auto" if uses_quantized_weights else None,
-                "dtype": dtype,
-                "low_cpu_mem_usage": True,
-                "cache_dir": str(cache_dir) if cache_dir is not None and load_source == model_id else None,
-            }
-            if quant_config is not None:
-                load_kwargs["quantization_config"] = quant_config
-            self.model = AutoModelForCausalLM.from_pretrained(
+            total_layers_hint = _total_layers_for_local_snapshot(AutoConfig, load_source)
+            if _should_partial_materialize_shard(
                 load_source,
-                **load_kwargs,
-            )
-            if not uses_quantized_weights:
-                self.model.to(self.device)
+                shard_start,
+                shard_end,
+                total_layers_hint=total_layers_hint,
+                uses_quantized_weights=uses_quantized_weights,
+            ):
+                self.model = _load_partial_model_from_snapshot(
+                    AutoConfig,
+                    AutoModelForCausalLM,
+                    torch,
+                    load_source,
+                    shard_start,
+                    shard_end,
+                    dtype,
+                    self.device,
+                )
+            else:
+                load_kwargs = {
+                    "device_map": "auto" if uses_quantized_weights else None,
+                    "dtype": dtype,
+                    "low_cpu_mem_usage": True,
+                    "cache_dir": str(cache_dir) if cache_dir is not None and load_source == model_id else None,
+                }
+                if quant_config is not None:
+                    load_kwargs["quantization_config"] = quant_config
+                self.model = AutoModelForCausalLM.from_pretrained(
+                    load_source,
+                    **load_kwargs,
+                )
+                if not uses_quantized_weights:
+                    self.model.to(self.device)
         except Exception as exc:
             if _looks_like_oom(exc):
                 raise InsufficientVRAMError(
@@ -357,6 +381,135 @@ def load_torch_shard(
     return TorchModelShard(model_id, shard_start, shard_end, quantization, cache_dir)
 
 
+def _total_layers_for_local_snapshot(auto_config: Any, load_source: str) -> int | None:
+    snapshot_dir = Path(load_source)
+    if not (snapshot_dir / "config.json").exists():
+        return None
+    from .model_catalog import layers_from_config
+
+    try:
+        cfg = auto_config.from_pretrained(str(snapshot_dir))
+    except Exception:
+        return None
+    return layers_from_config(cfg)
+
+
+def _should_partial_materialize_shard(
+    load_source: str,
+    shard_start: int,
+    shard_end: int,
+    *,
+    total_layers_hint: int | None,
+    uses_quantized_weights: bool,
+) -> bool:
+    if uses_quantized_weights:
+        return False
+    snapshot_dir = Path(load_source)
+    if not snapshot_dir.exists() or not (snapshot_dir / "config.json").exists():
+        return False
+    if not (snapshot_dir / "model.safetensors.index.json").exists():
+        return False
+    if total_layers_hint is None:
+        return False
+    return not (shard_start == 0 and shard_end >= total_layers_hint - 1)
+
+
+def _load_partial_model_from_snapshot(
+    auto_config: Any,
+    auto_model_for_causal_lm: Any,
+    torch: Any,
+    load_source: str,
+    shard_start: int,
+    shard_end: int,
+    dtype: Any,
+    device: Any,
+    *,
+    init_empty_weights_fn: Any | None = None,
+    set_tensor_fn: Any | None = None,
+    safe_open_fn: Any | None = None,
+) -> Any:
+    from .model_catalog import layers_from_config
+    from .safetensors_selection import (
+        INDEX_FILENAME,
+        select_tensor_names_for_layers_from_index,
+    )
+
+    if init_empty_weights_fn is None:
+        from accelerate import init_empty_weights as init_empty_weights_fn
+    if set_tensor_fn is None:
+        from accelerate.utils import set_module_tensor_to_device as set_tensor_fn
+    if safe_open_fn is None:
+        from safetensors import safe_open as safe_open_fn
+
+    snapshot_dir = Path(load_source)
+    cfg = auto_config.from_pretrained(str(snapshot_dir))
+    total_layers = layers_from_config(cfg)
+    if total_layers is None:
+        raise PartialModelLoadUnsupported(
+            f"could not determine num_hidden_layers for local snapshot {snapshot_dir}"
+        )
+    if shard_end >= total_layers:
+        raise ValueError(
+            f"shard_end {shard_end} exceeds last layer index {total_layers - 1}"
+        )
+
+    index_path = snapshot_dir / INDEX_FILENAME
+    try:
+        index = json.loads(index_path.read_text(encoding="utf-8"))
+    except FileNotFoundError as exc:
+        raise PartialModelLoadUnsupported(
+            f"missing SafeTensors index for partial load: {index_path}"
+        ) from exc
+    weight_map = index.get("weight_map")
+    if not isinstance(weight_map, dict):
+        raise PartialModelLoadUnsupported(f"{INDEX_FILENAME} must contain a weight_map object")
+
+    tensor_names = select_tensor_names_for_layers_from_index(
+        weight_map,
+        shard_start,
+        shard_end,
+        total_layers=total_layers,
+    )
+    if not tensor_names:
+        raise PartialModelLoadUnsupported(
+            f"no checkpoint tensors matched layers {shard_start}-{shard_end} in {snapshot_dir}"
+        )
+
+    with init_empty_weights_fn():
+        model = auto_model_for_causal_lm.from_config(cfg, torch_dtype=dtype)
+    tie_weights = getattr(model, "tie_weights", None)
+    if callable(tie_weights):
+        tie_weights()
+
+    tensors_by_file: dict[str, list[str]] = {}
+    for tensor_name in sorted(tensor_names):
+        rel_file = weight_map.get(tensor_name)
+        if not isinstance(rel_file, str):
+            continue
+        tensors_by_file.setdefault(rel_file, []).append(tensor_name)
+
+    for rel_file, names in tensors_by_file.items():
+        checkpoint_file = snapshot_dir / rel_file
+        if not checkpoint_file.exists():
+            raise PartialModelLoadUnsupported(
+                f"checkpoint file advertised in {INDEX_FILENAME} is missing: {checkpoint_file}"
+            )
+        with safe_open_fn(str(checkpoint_file), framework="pt", device="cpu") as handle:
+            for tensor_name in names:
+                set_tensor_fn(
+                    model,
+                    tensor_name,
+                    device,
+                    value=handle.get_tensor(tensor_name),
+                    dtype=dtype,
+                )
+
+    for module in _active_modules_for_shard(model, shard_start, shard_end):
+        if hasattr(module, "to"):
+            module.to(device)
+    return model
+
+
 def _model_load_plan(
     auto_config: Any,
     model_id: str,
@@ -442,6 +595,37 @@ def _position_embeddings(model: Any) -> Any | None:
     return None
 
 
+def _rotary_embedding_module(model: Any) -> Any | None:
+    if hasattr(model, "model") and hasattr(model.model, "rotary_emb"):
+        return model.model.rotary_emb
+    if hasattr(model, "transformer") and hasattr(model.transformer, "rotary_emb"):
+        return model.transformer.rotary_emb
+    return None
+
+
+def _active_modules_for_shard(model: Any, shard_start: int, shard_end: int) -> list[Any]:
+    active: list[Any] = []
+
+    def add(module: Any | None) -> None:
+        if module is None:
+            return
+        if any(existing is module for existing in active):
+            return
+        active.append(module)
+
+    if shard_start == 0:
+        add(_embed_tokens(model))
+        add(_position_embeddings(model))
+    add(_rotary_embedding_module(model))
+    for layer in _model_layers(model)[shard_start:shard_end + 1]:
+        add(layer)
+    total_layers = len(_model_layers(model))
+    if shard_end >= total_layers - 1:
+        add(_final_norm(model))
+        add(getattr(model, "lm_head", None))
+    return active
+
+
 def _final_norm(model: Any) -> Any | None:
     if hasattr(model, "model") and hasattr(model.model, "norm"):
         return model.model.norm
@@ -485,11 +669,7 @@ def _rotary_position_embeddings(model: Any, hidden_states: Any, position_ids: An
     """Return model-level rotary embeddings required by newer HF decoder layers."""
     if position_ids is None:
         return None
-    rotary = None
-    if hasattr(model, "model") and hasattr(model.model, "rotary_emb"):
-        rotary = model.model.rotary_emb
-    elif hasattr(model, "transformer") and hasattr(model.transformer, "rotary_emb"):
-        rotary = model.transformer.rotary_emb
+    rotary = _rotary_embedding_module(model)
     if rotary is None:
         return None
     return rotary(hidden_states, position_ids)

packages/node/meshnet_node/safetensors_selection.py

@@ -118,6 +118,23 @@ def select_files_for_layers_from_index(
     return selected
 
 
+def select_tensor_names_for_layers_from_index(
+    weight_map: dict[str, str],
+    start_layer: int,
+    end_layer: int,
+    *,
+    total_layers: int | None = None,
+) -> set[str]:
+    """Pure variant that returns checkpoint tensor names instead of file paths."""
+    selected: set[str] = set()
+    for tensor_name, rel_file in weight_map.items():
+        if not isinstance(tensor_name, str) or not isinstance(rel_file, str):
+            continue
+        if _tensor_belongs_to_range(tensor_name, start_layer, end_layer, total_layers):
+            selected.add(tensor_name)
+    return selected
+
+
 def _tensor_belongs_to_range(
     tensor_name: str,
     start_layer: int,

packages/node/meshnet_node/startup.py

@@ -193,22 +193,33 @@ def _configure_torch_threads(
     return active
 
 
-def _max_assignable_layers(memory_mb: int, total_layers: int | None) -> int:
+def _max_assignable_layers(
+    memory_mb: int,
+    total_layers: int | None,
+    bytes_per_layer: int | None = None,
+) -> int:
     if total_layers is None or total_layers <= 0 or memory_mb <= 0:
         return 0
     budget_bytes = memory_mb * 1024 * 1024
-    return min(total_layers, int((budget_bytes * 0.8) // _DEFAULT_BYTES_PER_LAYER))
+    layer_bytes = bytes_per_layer or _DEFAULT_BYTES_PER_LAYER
+    return min(total_layers, int((budget_bytes * 0.8) // layer_bytes))
 
 
-def _shard_budget_line(memory_mb: int, memory_source: str, total_layers: int | None, quantization: str) -> str:
+def _shard_budget_line(
+    memory_mb: int,
+    memory_source: str,
+    total_layers: int | None,
+    quantization: str,
+    bytes_per_layer: int | None = None,
+) -> str:
     memory_gb = memory_mb / 1024
     gb_str = f"{memory_gb:.1f} GB"
     budget_quantization = "bfloat16" if quantization == "auto" else quantization
     if total_layers is None or total_layers <= 0:
         return f"Memory budget: {gb_str} {memory_source}; shard budget: unknown model layer count"
-    max_layers = _max_assignable_layers(memory_mb, total_layers)
+    max_layers = _max_assignable_layers(memory_mb, total_layers, bytes_per_layer=bytes_per_layer)
     # Remaining capacity after one full model load (rough estimate)
-    shard_bytes = max_layers * _DEFAULT_BYTES_PER_LAYER
+    shard_bytes = max_layers * (bytes_per_layer or _DEFAULT_BYTES_PER_LAYER)
     remaining_gb = (memory_mb * 1024 * 1024 - shard_bytes) / (1024 ** 3)
     remaining_str = f"; {remaining_gb:.1f} GB remaining after full load" if remaining_gb > 1 else ""
     return (
@@ -218,6 +229,23 @@ def _shard_budget_line(memory_mb: int, memory_source: str, total_layers: int | N
     )
 
 
+def _assignment_bytes_per_layer(assignment: dict, quantization: str) -> int | None:
+    bytes_per_layer = assignment.get("bytes_per_layer")
+    if isinstance(bytes_per_layer, int) and bytes_per_layer > 0:
+        return bytes_per_layer
+    if not isinstance(bytes_per_layer, dict):
+        return None
+    keys = [quantization, "bfloat16", "bf16", "int8", "nf4"]
+    for key in keys:
+        value = bytes_per_layer.get(key)
+        if isinstance(value, int) and value > 0:
+            return value
+    for value in bytes_per_layer.values():
+        if isinstance(value, int) and value > 0:
+            return value
+    return None
+
+
 def _post_json(url: str, payload: dict, timeout: float = 10.0) -> dict:
     data = json.dumps(payload).encode()
     req = urllib.request.Request(
@@ -303,6 +331,9 @@ def _attach_relay_bridge(node: StubNodeServer | TorchNodeServer, bridge: RelayHt
     node.stop = _stop_with_bridge  # type: ignore[method-assign]
 
 
+_PENDING_NODE_ID = "pending"
+
+
 def _start_heartbeat(
     tracker_url: str,
     node_id: str,
@@ -340,10 +371,33 @@ def _start_heartbeat(
         try:
             resp = _post_json(f"{tracker_url}/v1/nodes/register", register_payload)
             node_id = resp.get("node_id", node_id)
+            if node_ref is not None:
+                setattr(node_ref, "tracker_node_id", node_id)
             return True
         except Exception:
             return False
 
+    def _register_additional_assignment(applied: dict) -> None:
+        model_id = str(applied.get("model") or register_payload.get("hf_repo") or register_payload.get("model"))
+        extra_payload = {
+            **register_payload,
+            "model": model_id.split("/")[-1],
+            "hf_repo": model_id if "/" in model_id else register_payload.get("hf_repo"),
+            "shard_start": applied["shard_start"],
+            "shard_end": applied["shard_end"],
+            "quantization": applied.get("quantization", register_payload.get("quantization")),
+            "tracker_mode": bool(applied.get("tracker_mode", False)),
+            "managed_assignment": True,
+        }
+        try:
+            reg_resp = _post_json(f"{tracker_url}/v1/nodes/register", extra_payload)
+            print(
+                f"  [node] registered additional model — node ID: {reg_resp.get('node_id')}",
+                flush=True,
+            )
+        except Exception as exc:
+            print(f"  [node] WARNING: additional model registration failed: {exc}", flush=True)
+
     def _apply_directives(directives: list[dict]) -> None:
         if not directives:
             return
@@ -356,6 +410,9 @@ def _start_heartbeat(
             print(f"  [node] WARNING: failed to apply tracker directives: {exc}", flush=True)
             return
         if applied:
+            if applied.get("action") == "ADD_SHARD":
+                _register_additional_assignment(applied)
+                return
             model_id = applied.get("model", register_payload.get("hf_repo") or register_payload.get("model"))
             register_payload["model"] = str(model_id).split("/")[-1]
             register_payload["hf_repo"] = model_id
@@ -367,7 +424,7 @@ def _start_heartbeat(
     def _loop() -> None:
         nonlocal node_id
         hb_url = f"{tracker_url}/v1/nodes/{node_id}/heartbeat"
-        outage_streak = 0  # consecutive intervals where tracker was unreachable
+        outage_streak = 1 if node_id == _PENDING_NODE_ID else 0
 
         while True:
             time.sleep(interval)
@@ -395,11 +452,12 @@ def _start_heartbeat(
                 new_asgn = resp.get("new_assignment")
                 if new_asgn:
                     print(
-                        f"  [node] tracker reassignment received: "
-                        f"model={new_asgn.get('model')!r} "
+                        f"  [node] tracker assignment received: "
+                        f"action={new_asgn.get('action')!r} model={new_asgn.get('model')!r} "
                         f"shards={new_asgn.get('shard_start')}-{new_asgn.get('shard_end')}",
                         flush=True,
                     )
+                    _apply_directives([new_asgn])
             except urllib.error.HTTPError as exc:
                 if exc.code == 404:
                     # Node was purged (e.g. long gap before restart noticed) — re-register now.
@@ -421,6 +479,34 @@ def _start_heartbeat(
     return t
 
 
+def _register_with_tracker(
+    tracker_url: str,
+    reg_payload: dict,
+    node: Any,
+    start_time: float,
+) -> str | None:
+    """Register with the tracker, or start background retries when it is unreachable."""
+    try:
+        reg_resp = _post_json(f"{tracker_url}/v1/nodes/register", reg_payload)
+        tracker_node_id = str(reg_resp.get("node_id") or "?")
+        setattr(node, "tracker_node_id", tracker_node_id)
+        print(f"  Registered with tracker — node ID: {tracker_node_id}", flush=True)
+        _start_heartbeat(tracker_url, tracker_node_id, reg_payload, node_ref=node, start_time=start_time)
+        return tracker_node_id
+    except Exception as exc:
+        setattr(node, "tracker_node_id", None)
+        print(f"  Warning: tracker registration failed: {exc}", flush=True)
+        print("  [node] will retry registration in the background", flush=True)
+        _start_heartbeat(
+            tracker_url,
+            _PENDING_NODE_ID,
+            reg_payload,
+            node_ref=node,
+            start_time=start_time,
+        )
+        return None
+
+
 def _warn_virtual_network_ip(ip: str | None) -> None:
     """Print a warning when the auto-detected advertise IP is in a known virtual-network range.
 
@@ -454,7 +540,7 @@ def _warn_virtual_network_ip(ip: str | None) -> None:
 def run_startup(
     tracker_url: str,
     port: int = 0,
-    model: str = "stub-model",
+    model: str | None = None,
     model_id: str | None = None,
     shard_start: int | None = None,
     shard_end: int | None = None,
@@ -580,8 +666,11 @@ def run_startup(
     if probationary_line is not None:
         print(f"  {probationary_line}", flush=True)
 
+    pinned_shard_start = shard_start
+    pinned_shard_end = shard_end
+    user_pinned_shard = pinned_shard_start is not None or pinned_shard_end is not None
+
     if model_id:  # treat "" the same as None — no explicit model given
-        user_pinned_shard = shard_start is not None or shard_end is not None
         full_sources: list[dict] = []
         # Auto-detect shard range from model config if not explicitly provided
         if shard_start is None or shard_end is None:
@@ -640,6 +729,7 @@ def run_startup(
             route_timeout=route_timeout,
             cache_dir=cache_dir,
             debug=debug,
+            max_loaded_shards=max_loaded_shards,
         )
         _node_start_time = time.monotonic()
         actual_port = node.start()
@@ -692,16 +782,9 @@ def run_startup(
             **registration_capabilities,
             **relay_fields,
         }
-        tracker_node_id: str | None = None
-        try:
-            reg_resp = _post_json(f"{tracker_url}/v1/nodes/register", reg_payload)
-            tracker_node_id = str(reg_resp.get("node_id") or "?")
-            setattr(node, "tracker_node_id", tracker_node_id)
-            print(f"  Registered with tracker — node ID: {tracker_node_id}", flush=True)
-            _start_heartbeat(tracker_url, tracker_node_id, reg_payload, node_ref=node, start_time=_node_start_time)
-        except Exception as exc:
-            setattr(node, "tracker_node_id", None)
-            print(f"  Warning: tracker registration failed: {exc}", flush=True)
+        tracker_node_id = _register_with_tracker(
+            tracker_url, reg_payload, node, _node_start_time,
+        )
 
         print(
             f"\n{'=' * 32}\n"
@@ -719,16 +802,17 @@ def run_startup(
             flush=True,
         )
         return node
-    if shard_start is not None or shard_end is not None:
-        raise ValueError("--shard-start / --shard-end require --model-id")
+    if user_pinned_shard and not model:
+        raise ValueError("--shard-start / --shard-end require --model")
 
     # 3a. Auto-join: query tracker for network-wide HF model assignment.
     # Skipped when the user explicitly requested a model — the shard-assignment
     # query below (/v1/nodes/assign?model=…) is authoritative there, and a fresh
     # tracker would otherwise print a scary 503 for the model-less auto-join.
     net_assignment: dict = {}
-    if model and model != "stub-model":
-        print(f"Model {model!r} requested explicitly — skipping network auto-join.", flush=True)
+    if model_id or (model and model != "stub-model"):
+        if model:
+            print(f"Model {model!r} requested explicitly — skipping network auto-join.", flush=True)
     else:
         print("Querying tracker for network assignment...", flush=True)
         assign_qs = urllib.parse.urlencode({"device": device, "vram_mb": assignment_vram_mb, "ram_mb": ram_mb})
@@ -739,17 +823,25 @@ def run_startup(
     assigned_hf_repo: str | None = net_assignment.get("hf_repo")
     _gap_found: bool = bool(net_assignment.get("gap_found", False))
 
-    if assigned_hf_repo and _gap_found:
+    if assigned_hf_repo:
         assigned_shard_start: int = net_assignment["shard_start"]
         assigned_shard_end: int = net_assignment["shard_end"]
         assigned_num_layers: int = net_assignment["num_layers"]
         assigned_model_sources: list[dict] = net_assignment.get("model_sources", [])
-        print(
-            f"  Assigned: {assigned_hf_repo} "
-            f"layers {assigned_shard_start}–{assigned_shard_end} "
-            f"(of {assigned_num_layers})",
-            flush=True,
-        )
+        if _gap_found:
+            print(
+                f"  Assigned gap: {assigned_hf_repo} "
+                f"layers {assigned_shard_start}–{assigned_shard_end} "
+                f"(of {assigned_num_layers})",
+                flush=True,
+            )
+        else:
+            print(
+                f"  Assigned redundant copy: {assigned_hf_repo} "
+                f"layers {assigned_shard_start}–{assigned_shard_end} "
+                f"(of {assigned_num_layers})",
+                flush=True,
+            )
         full_sources = [] if tracker_source_disabled else _full_model_sources(assigned_model_sources)
         if full_sources:
             print("Downloading assigned model snapshot...", flush=True)
@@ -773,6 +865,7 @@ def run_startup(
             route_timeout=route_timeout,
             cache_dir=cache_dir,
             debug=debug,
+            max_loaded_shards=max_loaded_shards,
         )
         _node_start_time = time.monotonic()
         actual_port = node.start()
@@ -817,16 +910,9 @@ def run_startup(
             **registration_capabilities,
             **relay_fields,
         }
-        tracker_node_id = None
-        try:
-            reg_resp = _post_json(f"{tracker_url}/v1/nodes/register", auto_reg_payload)
-            tracker_node_id = str(reg_resp.get("node_id") or "?")
-            setattr(node, "tracker_node_id", tracker_node_id)
-            print(f"  Registered with tracker — node ID: {tracker_node_id}", flush=True)
-            _start_heartbeat(tracker_url, tracker_node_id, auto_reg_payload, node_ref=node, start_time=_node_start_time)
-        except Exception as exc:
-            setattr(node, "tracker_node_id", None)
-            print(f"  Warning: tracker registration failed: {exc}", flush=True)
+        tracker_node_id = _register_with_tracker(
+            tracker_url, auto_reg_payload, node, _node_start_time,
+        )
         shard_count = assigned_shard_end - assigned_shard_start + 1
         print(
             f"\n{'=' * 32}\n"
@@ -846,10 +932,16 @@ def run_startup(
         )
         return node
 
-    # 3b. Shard assignment from tracker (stub-model / preset-based path)
+    if not assigned_hf_repo and model is None:
+        raise RuntimeError(
+            "Tracker did not assign a model. Join a network that already serves one, "
+            "or start with --model <HF_REPO>."
+        )
+
+    # 3b. Stub preset path (tests / explicit stub-model) or named preset models.
     print("Querying tracker for shard assignment...", flush=True)
     assign_qs = urllib.parse.urlencode({
-        "model": model,
+        "model": model or "stub-model",
         "device": device,
         # CPU-mode nodes must be sized by RAM: a detected-but-unusable GPU's
         # VRAM would otherwise cap the shard (e.g. 8 GB VRAM → 3 layers on a
@@ -863,13 +955,25 @@ def run_startup(
         print(f"  ERROR: Cannot reach tracker at {tracker_url}: {exc}", file=sys.stderr, flush=True)
         raise
 
-    shard_start: int = assignment["shard_start"]
-    shard_end: int = assignment["shard_end"]
+    shard_start = assignment["shard_start"]
+    shard_end = assignment["shard_end"]
+    if user_pinned_shard:
+        if pinned_shard_start is not None:
+            shard_start = pinned_shard_start
+        if pinned_shard_end is not None:
+            shard_end = pinned_shard_end
     assigned_model: str = assignment.get("model", model)
     hf_repo: str | None = assignment.get("hf_repo")
     peers: list[dict] = assignment.get("peers", [])
     model_sources: list[dict] = [] if tracker_source_disabled else assignment.get("model_sources", [])
-    print(f"  Shard: layers {shard_start}-{shard_end} of {assigned_model}", flush=True)
+    assignment_bytes_per_layer = _assignment_bytes_per_layer(assignment, quantization)
+    if user_pinned_shard:
+        print(
+            f"  Shard: layers {shard_start}-{shard_end} of {assigned_model} (pinned)",
+            flush=True,
+        )
+    else:
+        print(f"  Shard: layers {shard_start}-{shard_end} of {assigned_model}", flush=True)
 
     # 4. Download shard
     print("Downloading shard...", flush=True)
@@ -931,6 +1035,7 @@ def run_startup(
         "hardware_profile": hw,
         "wallet_address": address,
         "score": 1.0,
+        "managed_assignment": not user_pinned_shard,
         **registration_capabilities,
         **relay_fields,
     }
@@ -955,7 +1060,7 @@ def run_startup(
         f"meshnet-node ready\n"
         f"  Wallet:   {address}\n"
         f"  Shard:    layers {shard_start}-{shard_end} ({assigned_model})\n"
-        f"  {_shard_budget_line(memory_budget_mb, memory_budget_source, assignment.get('model_layers_end', shard_end) + 1, quantization)}\n"
+        f"  {_shard_budget_line(memory_budget_mb, memory_budget_source, assignment.get('model_layers_end', shard_end) + 1, quantization, bytes_per_layer=assignment_bytes_per_layer)}\n"
         f"  Endpoint: {endpoint}\n"
         f"  Node ID:  {node_id}\n"
         f"  Hardware: {hw_str}\n"

packages/node/meshnet_node/torch_server.py

@@ -75,20 +75,39 @@ class _TorchHTTPServer(http.server.HTTPServer):
         tracker_url: str | None = None,
         route_timeout: float = 30.0,
         debug: bool = False,
+        max_loaded_shards: int = 1,
     ):
         super().__init__(addr, handler)
         self.backend = backend
+        self.backends: dict[str, TorchModelShard] = {backend.model_id: backend}
         self.received_activations = False
         self.forward_chunk_count = 0
         self.tracker_mode = tracker_mode
         self.tracker_url = tracker_url
         self.route_timeout = route_timeout
         self.debug = debug
+        self.max_loaded_shards = max(1, max_loaded_shards)
         self.total_requests: int = 0
         self.failed_requests: int = 0
         self.queue_depth: int = 0
         self._stats_lock = threading.Lock()
 
+    def resolve_backend(self, model_name: str | None) -> TorchModelShard | None:
+        if not model_name:
+            return self.backend
+        wanted = model_name.strip().lower()
+        for key, shard_backend in self.backends.items():
+            key_l = key.lower()
+            if key_l == wanted or key_l.rsplit("/", 1)[-1] == wanted:
+                return shard_backend
+        return self.backend
+
+    def chat_enabled(self) -> bool:
+        return any(
+            shard_backend.is_head
+            for shard_backend in self.backends.values()
+        )
+
 
 class _TorchHandler(http.server.BaseHTTPRequestHandler):
     def log_message(self, fmt, *args):  # noqa: suppress request logs in tests
@@ -100,7 +119,7 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
             self._handle_forward()
         elif self.path == "/v1/infer":
             self._handle_infer()
-        elif self.path == "/v1/chat/completions" and server.tracker_mode:
+        elif self.path == "/v1/chat/completions" and server.chat_enabled():
             self._handle_chat_completions()
         else:
             self.send_response(404)
@@ -284,22 +303,26 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
             messages = []
         stream = bool(body.get("stream", False))
         model_name = str(body.get("model", ""))
+        backend = server.resolve_backend(model_name)
+        if backend is None or not backend.is_head:
+            self._send_json(400, {"error": "model not loaded on this node"})
+            return
         max_tokens = int(body.get("max_tokens") or body.get("max_new_tokens") or 256)
         temperature = float(body.get("temperature") or 1.0)
         top_p = float(body.get("top_p") or 1.0)
 
         # Fast path: this node owns the complete model — use HF generate() with KV cache.
         # Avoids the single-token-per-forward-pass limitation of the distributed path.
-        if server.backend.is_head and server.backend.is_tail:
+        if backend.is_head and backend.is_tail:
             try:
                 if stream:
                     self._stream_openai_response(
-                        server.backend.generate_text_streaming(messages, max_tokens, temperature, top_p),
+                        backend.generate_text_streaming(messages, max_tokens, temperature, top_p),
                         model_name,
                     )
                 else:
-                    text = server.backend.generate_text(messages, max_tokens, temperature, top_p)
-                    self._send_openai_response(text, model_name, False, messages)
+                    text = backend.generate_text(messages, max_tokens, temperature, top_p)
+                    self._send_openai_response(text, model_name, False, messages, backend=backend)
             except Exception as exc:
                 self._record_failed_request()
                 self._send_json(500, {"error": f"generation failed: {exc}"})
@@ -309,7 +332,7 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
         # We do N single-step forward passes (no cross-node KV cache), which is slow
         # but correct. Each step: head encodes current sequence → forwards through route
         # → tail returns the next token string → append → repeat.
-        remaining_route = self._get_remaining_route(model_name)
+        remaining_route = self._get_remaining_route(model_name, backend=backend)
         print(
             f"  [node] chat route model={model_name!r} max_tokens={max_tokens} "
             f"downstream={remaining_route}",
@@ -318,11 +341,10 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
         if not remaining_route:
             self._send_openai_response(
                 "error: no downstream route — check tracker connectivity",
-                model_name, False, messages,
+                model_name, False, messages, backend=backend,
             )
             return
 
-        backend = server.backend
         # Format with chat template so the model knows it's in assistant mode.
         try:
             if hasattr(backend.tokenizer, "apply_chat_template"):
@@ -342,13 +364,17 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
         generated: list[str] = []
         current_text = prompt_text
 
+        stream_emit = None
+        if stream:
+            stream_emit = self._start_openai_stream(model_name)
+
         for _ in range(max_tokens):
             try:
                 payload = backend.encode_prompt(current_text)
             except Exception as exc:
                 print(f"  [node] distributed encode error: {exc}", flush=True)
                 break
-            token_str = self._run_downstream_pipeline(payload, remaining_route)
+            token_str = self._run_downstream_pipeline(payload, remaining_route, backend=backend)
             if not token_str:
                 break
             # Stop on error responses or EOS.
@@ -357,12 +383,17 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
             if eos_token and token_str == eos_token:
                 break
             generated.append(token_str)
+            if stream_emit is not None:
+                stream_emit(token_str)
             current_text = current_text + token_str
 
         result_text = "".join(generated)
-        self._send_openai_response(result_text, model_name, stream, messages)
+        if stream_emit is not None:
+            stream_emit(None)
+            return
+        self._send_openai_response(result_text, model_name, stream, messages, backend=backend)
 
-    def _get_remaining_route(self, model: str) -> list[dict]:
+    def _get_remaining_route(self, model: str, *, backend: TorchModelShard | None = None) -> list[dict]:
         """Return downstream hops as dicts with endpoint, start_layer, and optional relay_addr.
 
         Fast path reads X-Meshnet-Route header injected by the tracker.
@@ -395,9 +426,10 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
 
         # Slow path: query the tracker (direct node-to-node calls, or tracker didn't inject).
         server: _TorchHTTPServer = self.server  # type: ignore[assignment]
+        active_backend = backend or server.backend
         if server.tracker_url is None:
             return []
-        route_model = getattr(server.backend, "model_id", None) or model
+        route_model = getattr(active_backend, "model_id", None) or model
         try:
             url = f"{server.tracker_url}/v1/route?model={urllib.parse.quote(route_model)}"
             with urllib.request.urlopen(url, timeout=server.route_timeout) as r:
@@ -424,18 +456,19 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
             print(f"  [node] WARNING: route lookup failed for {route_model!r}: {exc}", flush=True)
             return []
 
-    def _run_downstream_pipeline(self, payload: object, route: list[dict]) -> str:
+    def _run_downstream_pipeline(self, payload: object, route: list[dict], *, backend: TorchModelShard | None = None) -> str:
         server: _TorchHTTPServer = self.server  # type: ignore[assignment]
+        active_backend = backend or server.backend
         if not route:
             # Partial shard at tail: decode the activation from the previous node.
             # Full single-node (head+tail) is handled before entering this method.
-            if server.backend.is_tail:
+            if active_backend.is_tail:
                 try:
-                    tensor = server.backend.torch.frombuffer(
+                    tensor = active_backend.torch.frombuffer(
                         bytearray(payload.body),  # type: ignore[union-attr]
-                        dtype=server.backend.torch.bfloat16,
-                    ).reshape(payload.shape).to(server.backend.device)  # type: ignore[union-attr]
-                    return server.backend.decode_tail(tensor)
+                        dtype=active_backend.torch.bfloat16,
+                    ).reshape(payload.shape).to(active_backend.device)  # type: ignore[union-attr]
+                    return active_backend.decode_tail(tensor)
                 except Exception as exc:
                     return f"decode error: {exc}"
             return "no downstream route available for non-tail shard"
@@ -526,6 +559,15 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
 
     def _stream_openai_response(self, token_iter, model: str) -> None:
         """Stream tokens from an iterator as SSE chunks."""
+        emit = self._start_openai_stream(model)
+        for token_text in token_iter:
+            if not token_text:
+                continue
+            emit(token_text)
+        emit(None)
+
+    def _start_openai_stream(self, model: str):
+        """Open an OpenAI-compatible SSE response and return a token emitter."""
         chunk_id = "chatcmpl-node"
         created = int(time.time())
         self.send_response(200)
@@ -537,7 +579,7 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
             try:
                 self.wfile.write(f"data: {data}\n\n".encode())
                 self.wfile.flush()
-            except BrokenPipeError:
+            except (BrokenPipeError, ConnectionResetError):
                 pass
 
         _emit(json.dumps({
@@ -545,24 +587,27 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
             "model": model,
             "choices": [{"index": 0, "delta": {"role": "assistant", "content": ""}, "finish_reason": None}],
         }))
-        for token_text in token_iter:
-            if not token_text:
-                continue
+
+        def emit_token(token_text: str | None) -> None:
+            if token_text is None:
+                _emit(json.dumps({
+                    "id": chunk_id, "object": "chat.completion.chunk", "created": created,
+                    "model": model,
+                    "choices": [{"index": 0, "delta": {}, "finish_reason": "stop"}],
+                }))
+                try:
+                    self.wfile.write(b"data: [DONE]\n\n")
+                    self.wfile.flush()
+                except (BrokenPipeError, ConnectionResetError):
+                    pass
+                return
             _emit(json.dumps({
                 "id": chunk_id, "object": "chat.completion.chunk", "created": created,
                 "model": model,
                 "choices": [{"index": 0, "delta": {"content": token_text}, "finish_reason": None}],
             }))
-        _emit(json.dumps({
-            "id": chunk_id, "object": "chat.completion.chunk", "created": created,
-            "model": model,
-            "choices": [{"index": 0, "delta": {}, "finish_reason": "stop"}],
-        }))
-        try:
-            self.wfile.write(b"data: [DONE]\n\n")
-            self.wfile.flush()
-        except BrokenPipeError:
-            pass
+
+        return emit_token
 
     def _send_openai_response(
         self,
@@ -570,11 +615,13 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
         model: str,
         stream: bool,
         messages: list[dict] | None = None,
+        backend: TorchModelShard | None = None,
     ) -> None:
         chunk_id = "chatcmpl-node"
         created = int(time.time())
+        active_backend = backend or self.server.backend  # type: ignore[attr-defined]
         if not stream:
-            usage = _usage_for_response(self.server.backend, messages or [], text)  # type: ignore[attr-defined]
+            usage = _usage_for_response(active_backend, messages or [], text)
             self._send_json(200, {
                 "id": chunk_id,
                 "object": "chat.completion",
@@ -685,9 +732,11 @@ class TorchNodeServer:
         route_timeout: float = 30.0,
         cache_dir: Path | None = None,
         debug: bool = False,
+        max_loaded_shards: int = 1,
     ) -> None:
         self._host = host
         self._requested_port = port
+        self._max_loaded_shards = max(1, max_loaded_shards)
         self._backend = backend or _load_backend(
             model_id,
             shard_start,
@@ -695,6 +744,7 @@ class TorchNodeServer:
             quantization,
             cache_dir,
         )
+        self._backends: dict[str, TorchModelShard] = {self._backend.model_id: self._backend}
         # Auto-detect tracker mode: enabled when shard_start == 0 or explicitly set
         self._tracker_mode = tracker_mode if tracker_mode is not None else (shard_start == 0)
         self._tracker_url = tracker_url
@@ -733,41 +783,64 @@ class TorchNodeServer:
     def queue_depth(self) -> int:
         return self._server.queue_depth if self._server is not None else 0
 
+    @property
+    def loaded_model_ids(self) -> list[str]:
+        return list(self._backends.keys())
+
     def apply_tracker_directives(self, directives: list[dict]) -> dict | None:
-        """Apply tracker LOAD_SHARD directives by hot-swapping the loaded backend."""
+        """Apply tracker shard directives (LOAD_SHARD replace, ADD_SHARD load-more)."""
+        add_directive = next(
+            (directive for directive in reversed(directives) if directive.get("action") == "ADD_SHARD"),
+            None,
+        )
         load_directive = next(
             (directive for directive in reversed(directives) if directive.get("action") == "LOAD_SHARD"),
             None,
         )
-        if load_directive is None:
+        directive = add_directive or load_directive
+        if directive is None:
             return None
-        shard_start = int(load_directive["shard_start"])
-        shard_end = int(load_directive["shard_end"])
-        quantization = str(load_directive.get("quantization") or self._backend.quantization)
-        model_id = str(load_directive.get("model") or self._backend.model_id)
+        shard_start = int(directive["shard_start"])
+        shard_end = int(directive["shard_end"])
+        quantization = str(directive.get("quantization") or self._backend.quantization)
+        model_id = str(directive.get("model") or self._backend.model_id)
+        replacing = directive.get("action") == "LOAD_SHARD"
+        if not replacing and len(self._backends) >= self._max_loaded_shards:
+            print(
+                f"  [node] WARNING: ignoring ADD_SHARD for {model_id!r} — "
+                f"loaded {len(self._backends)}/{self._max_loaded_shards} slots full",
+                flush=True,
+            )
+            return None
+        action_label = "reassigned" if replacing else "additional"
         print(
-            f"  [node] loading reassigned shard: {model_id} layers {shard_start}-{shard_end}",
+            f"  [node] loading {action_label} shard: {model_id} layers {shard_start}-{shard_end}",
             flush=True,
         )
         try:
             new_backend = _load_backend(model_id, shard_start, shard_end, quantization, self._cache_dir)
         except TypeError:
             new_backend = _load_backend(model_id, shard_start, shard_end, quantization)
-        self._backend = new_backend
-        self._tracker_mode = shard_start == 0
-        if self._server is not None:
-            self._server.backend = new_backend
-            self._server.tracker_mode = self._tracker_mode
+        self._backends[model_id] = new_backend
+        if replacing or shard_start == 0:
+            self._backend = new_backend
+            self._tracker_mode = shard_start == 0
         print(
-            f"  [node] loaded reassigned shard: {model_id} layers {shard_start}-{shard_end}",
+            f"  [node] loaded {action_label} shard: {model_id} layers {shard_start}-{shard_end}",
             flush=True,
         )
+        if self._server is not None:
+            self._server.backends = dict(self._backends)
+            if replacing or shard_start == 0:
+                self._server.backend = new_backend
+                self._server.tracker_mode = self._tracker_mode
         return {
+            "action": directive.get("action"),
             "model": model_id,
             "shard_start": shard_start,
             "shard_end": shard_end,
             "quantization": quantization,
-            "tracker_mode": self._tracker_mode,
+            "tracker_mode": shard_start == 0,
         }
 
     def start(self) -> int:
@@ -781,7 +854,9 @@ class TorchNodeServer:
             self._tracker_url,
             self._route_timeout,
             self._debug,
+            self._max_loaded_shards,
         )
+        self._server.backends = dict(self._backends)
         self.port = self._server.server_address[1]
         self._thread = threading.Thread(target=self._server.serve_forever, daemon=True)
         self._thread.start()

packages/tracker/meshnet_tracker/billing.py

@@ -453,13 +453,13 @@ class BillingLedger:
         with self._lock:
             return self._node_pending.get(wallet, 0.0)
 
-    def usage_for(self, api_keys: list[str], *, recent_limit: int = 20) -> dict:
+    def usage_for(self, api_keys: list[str], *, recent_limit: int | None = None) -> dict:
         """Aggregate charge history for a set of API keys (dashboard view)."""
         keys = set(api_keys)
         requests = 0
         total_tokens = 0
         total_cost = 0.0
-        recent: list[dict] = []
+        records: list[dict] = []
         with self._lock:
             for event in self._event_log:
                 if event.get("type") != "charge" or event.get("api_key") not in keys:
@@ -467,18 +467,20 @@ class BillingLedger:
                 requests += 1
                 total_tokens += int(event.get("total_tokens", 0))
                 total_cost += float(event.get("cost", 0.0))
-                recent.append({
+                records.append({
                     "api_key": event["api_key"],
                     "model": event.get("model"),
                     "total_tokens": event.get("total_tokens", 0),
                     "cost": event.get("cost", 0.0),
                     "ts": event.get("ts", 0.0),
                 })
+        recent = records[-recent_limit:] if recent_limit is not None else records
         return {
             "requests": requests,
             "total_tokens": total_tokens,
             "total_cost": total_cost,
-            "recent": recent[-recent_limit:],
+            "records": records,
+            "recent": recent,
         }
 
     def snapshot(self) -> dict:

packages/tracker/meshnet_tracker/dashboard.html

@@ -39,23 +39,56 @@
   .form-row { display:flex; gap:8px; }
   .form-row button { white-space:nowrap; }
   .error-msg { color:var(--bad); font-size:12px; min-height:16px; }
-  .keybox { word-break:break-all; background:var(--bg); border:1px solid var(--border);
+  .keybox { display:flex; flex-wrap:wrap; align-items:center; gap:6px;
+          position:relative;
+          word-break:break-all; background:var(--bg); border:1px solid var(--border);
           border-radius:6px; padding:4px 8px; margin:4px 0; font-size:11px; }
-  .tabs { display:flex; gap:10px; margin-bottom:8px; }
-  .tabs a { color:var(--dim); cursor:pointer; }
-  .tabs a.active { color:var(--accent); border-bottom:1px solid var(--accent); }
-  .wide { grid-column:1 / -1; }
-  .console {
-    background:var(--bg); border:1px solid var(--border); border-radius:6px;
-    min-height:160px; max-height:280px; overflow:auto; padding:7px 9px;
-    white-space:pre-wrap; word-break:break-word; font-size:11px;
-  }
-  .console-line { padding:1px 0; border-bottom:1px solid #161b22; }
-  .console-time { color:var(--dim); }
-  .console-level-info { color:var(--accent); }
-  .console-level-warn { color:var(--warn); }
-  .console-level-error { color:var(--bad); }
-</style>
+  .key-text { cursor:text; flex:1 1 auto; min-width:12rem; }
+  .copy-tooltip {
+    position:absolute; right:8px; top:-26px;
+    background:var(--panel); border:1px solid var(--border); color:var(--ok);
+    padding:2px 8px; border-radius:4px; font-size:11px;
+    pointer-events:none; z-index:1; white-space:nowrap;
+  }
+  .tabs { display:flex; gap:10px; margin-bottom:8px; }
+  .tabs a { color:var(--dim); cursor:pointer; }
+  .tabs a.active { color:var(--accent); border-bottom:1px solid var(--accent); }
+  .dashboard-tabs { display:flex; gap:10px; padding:10px 20px 0; border-bottom:1px solid var(--border); }
+  .dashboard-tabs button { border:0; border-bottom:1px solid transparent; border-radius:0;
+          background:transparent; color:var(--dim); padding:5px 0 8px; }
+  .dashboard-tabs button.active { color:var(--accent); border-bottom-color:var(--accent); }
+  .wide { grid-column:1 / -1; }
+  section[hidden] { display:none !important; }
+  .chat-shell { display:grid; grid-template-columns:minmax(0, 1.35fr) minmax(320px, 0.65fr); gap:12px; }
+  .chat-pane { display:flex; flex-direction:column; gap:10px; min-width:0; }
+  .chat-panel { background:var(--bg); border:1px solid var(--border); border-radius:6px; padding:10px; }
+  .chat-controls { display:flex; gap:10px; align-items:end; flex-wrap:wrap; }
+  .chat-controls label { display:flex; flex-direction:column; gap:4px; color:var(--dim); }
+  .chat-controls select { min-width:220px; }
+  .chat-history { display:flex; flex-direction:column; gap:8px; min-height:220px; max-height:420px; overflow:auto; }
+  .chat-message { border:1px solid #21262d; border-radius:6px; padding:8px 10px; background:#10151d; }
+  .chat-role { color:var(--dim); font-size:11px; text-transform:uppercase; letter-spacing:.06em; margin-bottom:4px; }
+  .chat-role-user { color:var(--accent); }
+  .chat-role-assistant { color:var(--ok); }
+  .chat-role-error { color:var(--bad); }
+  .chat-compose { display:flex; flex-direction:column; gap:8px; }
+  .chat-compose textarea { min-height:112px; resize:vertical; width:100%; }
+  .chat-status { color:var(--dim); font-size:12px; }
+  .console {
+    background:var(--bg); border:1px solid var(--border); border-radius:6px;
+    min-height:160px; max-height:280px; overflow:auto; padding:7px 9px;
+    white-space:pre-wrap; word-break:break-word; font-size:11px;
+  }
+  .console-line { padding:1px 0; border-bottom:1px solid #161b22; }
+  .console-time { color:var(--dim); }
+  .console-level-info { color:var(--accent); }
+  .console-level-warn { color:var(--warn); }
+  .console-level-error { color:var(--bad); }
+  .status-pending { color:var(--warn); }
+  .status-processing { color:var(--accent); }
+  .status-failed { color:var(--bad); }
+  .status-complete { color:var(--ok); }
+</style>
 </head>
 <body>
 <header>
@@ -63,19 +96,53 @@
   <span class="meta" id="self-url"></span>
   <span class="meta" id="refreshed"></span>
 </header>
+<nav class="dashboard-tabs" aria-label="Dashboard sections">
+  <button id="tab-overview" class="active" onclick="switchDashboardTab('overview')">Overview</button>
+  <button id="tab-chat" onclick="switchDashboardTab('chat')">Chat</button>
+  <button id="tab-billing" style="display:none" onclick="switchDashboardTab('billing')">Billing</button>
+  <button id="tab-admin" style="display:none" onclick="switchDashboardTab('admin')">Admin</button>
+</nav>
 <main>
-  <section id="account-section"><h2>Account</h2><div id="account">loading…</div></section>
-  <section id="admin-section" style="display:none"><h2>All accounts (admin)</h2><div id="admin" class="empty"></div></section>
-  <section><h2>Tracker hive</h2><div id="hive" class="empty">loading…</div></section>
-  <section><h2>Nodes &amp; coverage</h2><div id="nodes" class="empty">loading…</div></section>
-  <section><h2>Client balances</h2><div id="clients" class="empty">loading…</div></section>
-  <section><h2>Node pending payouts</h2><div id="pending" class="empty">loading…</div></section>
-  <section><h2>Settlement history</h2><div id="settlements" class="empty">loading…</div></section>
-  <section><h2>Strikes / bans / forfeitures</h2><div id="fraud" class="empty">loading…</div></section>
-  <section><h2>Model usage (RPM)</h2><div id="stats" class="empty">loading…</div></section>
-  <section><h2>Node throughput</h2><div id="throughput" class="empty">loading…</div></section>
-  <section class="wide"><h2>Console output</h2><div id="console" class="console empty">loading…</div></section>
-</main>
+  <section data-tab="overview" id="account-section"><h2>Account</h2><div id="account">loading…</div></section>
+  <section data-tab="overview"><h2>Tracker hive</h2><div id="hive" class="empty">loading…</div></section>
+  <section data-tab="overview"><h2>Nodes &amp; coverage</h2><div id="nodes" class="empty">loading…</div></section>
+  <section data-tab="overview"><h2>Model usage (RPM)</h2><div id="stats" class="empty">loading…</div></section>
+  <section data-tab="overview" class="wide"><h2>Call wall</h2><div id="call-wall" class="empty">loading...</div></section>
+  <section data-tab="chat" class="wide">
+    <h2>Chat / inference</h2>
+    <div class="chat-shell">
+      <div class="chat-pane">
+        <div class="chat-panel chat-controls">
+          <label>Model
+            <select id="chat-model" onchange="selectChatModel(this.value)"></select>
+          </label>
+          <button class="small" onclick="clearChatHistory()">clear history</button>
+        </div>
+        <div class="chat-panel chat-compose">
+          <textarea id="chat-prompt" placeholder="Ask a question or describe the task"></textarea>
+          <div class="form-row">
+            <button onclick="sendChat()" id="chat-send">Send</button>
+          </div>
+        </div>
+      </div>
+      <div class="chat-pane">
+        <div class="chat-panel">
+          <div id="chat-status" class="chat-status">select a model to start</div>
+          <div id="chat-history" class="chat-history empty">no messages yet</div>
+        </div>
+      </div>
+    </div>
+  </section>
+  <section data-tab="billing" data-logged-in-only><h2>Usage summary</h2><div id="usage-summary" class="empty">login required</div></section>
+  <section data-tab="billing" data-logged-in-only><h2>Node throughput</h2><div id="node-throughput" class="empty">login required</div></section>
+  <section data-tab="billing"><h2>Request history</h2><div id="billing-usage" class="empty">login required</div></section>
+  <section data-tab="billing" data-admin-only><h2>Node pending payouts</h2><div id="pending" class="empty">admin login required</div></section>
+  <section data-tab="billing" data-admin-only><h2>Settlement history</h2><div id="settlements" class="empty">admin login required</div></section>
+  <section data-tab="admin" id="admin-section"><h2>All accounts (admin)</h2><div id="admin" class="empty"></div></section>
+  <section data-tab="admin" data-admin-only><h2>Strikes / bans / forfeitures</h2><div id="fraud" class="empty">admin login required</div></section>
+  <section data-tab="admin"><h2>Client balances</h2><div id="clients" class="empty">admin login required</div></section>
+  <section data-tab="admin" class="wide"><h2>Console output</h2><div id="console" class="console empty">admin login required</div></section>
+</main>
 <script>
 "use strict";
 const $ = id => document.getElementById(id);
@@ -83,6 +150,7 @@ const esc = s => String(s).replace(/[&<>"]/g,
   c => ({"&":"&amp;","<":"&lt;",">":"&gt;",'"':"&quot;"}[c]));
 const usdt = v => (Math.round(v * 1e6) / 1e6).toFixed(6);
 const tps = v => (v === null || v === undefined) ? "?" : (Math.round(v * 10) / 10).toFixed(1);
+const copies = v => (v === null || v === undefined) ? "?" : Number(v).toFixed(2);
 const short = (s, n=14) => { s = String(s); return s.length > n ? s.slice(0, 6) + "…" + s.slice(-5) : s; };
 
 async function fetchJson(path) {
@@ -128,16 +196,17 @@ function renderNodes(map) {
   }
   let html = "";
   for (const [model, group] of Object.entries(byModel)) {
-    html += `<div><b>${esc(model)}</b> <span class="dim">(${group.length} node${group.length===1?"":"s"})</span></div>`;
-    html += table(["node", "shard", "tps (1h)", "queue", "health"], group.map(n => {
+    const supply = group.find(n => n.model_supply && n.model_supply.served_model_copies !== undefined);
+    const served = supply && supply.model_supply && supply.model_supply.served_model_copies;
+    html += `<div><b>${esc(model)}</b> <span class="dim">(${group.length} node${group.length===1?"":"s"} · ${esc(copies(served))} served)</span></div>`;
+    html += table(["node", "shard", "tps (1h)", "queue", "served"], group.map(n => {
       const modelStats = (n.throughput && (n.throughput[n.hf_repo] || n.throughput[n.model])) || {};
       return [
       esc(short(n.node_id || "?")),
       esc(`${n.shard_start ?? "?"}-${n.shard_end ?? "?"}`),
       `<span class="num">${esc(tps(modelStats.tokens_per_sec_last_hour))}</span>`,
       esc(String((n.stats && n.stats.queue_depth) ?? 0)),
-      (n.stats && (n.stats.alive === false || n.stats.healthy === false))
-        ? '<span class="bad">down</span>' : '<span class="ok">up</span>',
+      `<span class="num">${esc(copies(n.model_supply && n.model_supply.served_model_copies))}</span>`,
     ]; }));
   }
   $("nodes").innerHTML = html;
@@ -208,7 +277,7 @@ function renderStats(stats) {
   $("stats").innerHTML = table(["model", "rpm (1h)", "rpm (24h)", "rpm (30d)"], rows);
 }
 
-function renderThroughput(stats) {
+function renderThroughputHtml(stats) {
   const nodes = (stats && stats.nodes) || {};
   const rows = [];
   for (const [nodeId, nodeStats] of Object.entries(nodes)) {
@@ -221,32 +290,406 @@ function renderThroughput(stats) {
       ]);
     }
   }
-  $("throughput").innerHTML = table(["node", "model", "tps (1h)", "samples"], rows);
-}
-
-function renderConsole(data) {
-  const events = (data && data.events) || [];
-  if (!events.length) {
-    $("console").innerHTML = '<div class="empty">no console events</div>';
-    return;
-  }
-  $("console").innerHTML = events.slice(-120).map(e => {
-    const level = String(e.level || "info");
-    const cls = level === "error" ? "console-level-error" : level === "warn" ? "console-level-warn" : "console-level-info";
-    const fields = e.fields && Object.keys(e.fields).length ? " " + JSON.stringify(e.fields) : "";
-    return `<div class="console-line"><span class="console-time">${new Date((e.ts || 0) * 1000).toLocaleTimeString()}</span> ` +
-      `<span class="${cls}">${esc(level.toUpperCase())}</span> ${esc(e.message || "")}${esc(fields)}</div>`;
-  }).join("");
-}
+  if (!rows.length) return '<div class="empty">no throughput samples yet</div>';
+  return table(["node", "model", "tps (1h)", "samples"], rows);
+}
+
+function hiveThroughputSummary(stats) {
+  const nodes = (stats && stats.nodes) || {};
+  let totalTps = 0;
+  let samples = 0;
+  for (const nodeStats of Object.values(nodes)) {
+    for (const s of Object.values((nodeStats && nodeStats.models) || {})) {
+      const t = Number(s.tokens_per_sec_last_hour);
+      if (Number.isFinite(t)) totalTps += t;
+      samples += Number(s.sample_count_last_hour || 0);
+    }
+  }
+  return { totalTps, samples };
+}
+
+function buildCallWallStates(events) {
+  const byId = new Map();
+  for (const e of events) {
+    const f = e.fields || {};
+    const id = f.request_id;
+    if (!id) continue;
+    let rec = byId.get(id);
+    if (!rec) {
+      rec = { id, events: [] };
+      byId.set(id, rec);
+    }
+    rec.events.push(e);
+    const msg = e.message;
+    if (msg === "proxy route selected") {
+      rec.status = "pending";
+      rec.started = e.ts;
+      rec.model = f.model || f.route_model || "?";
+      rec.route = f.route || f.nodes;
+      rec.nodes = f.nodes;
+      rec.stream = f.stream;
+    } else if (msg === "proxy via relay" || msg === "proxy connected") {
+      rec.status = "processing";
+      if (!rec.started) rec.started = e.ts;
+      rec.model = rec.model || f.model || f.route_model || "?";
+    } else if (msg === "proxy progress") {
+      rec.status = "processing";
+      rec.model = rec.model || f.model || f.route_model || "?";
+      rec.tokens = f.tokens;
+      rec.tps = f.tokens_per_sec;
+      rec.elapsed = f.elapsed_seconds;
+      rec.stream = f.stream;
+    } else if (msg === "relay proxy failed, trying direct") {
+      rec.status = "processing";
+      rec.warn = "relay failed, trying direct";
+    } else if (msg === "proxy complete") {
+      rec.status = "complete";
+      rec.model = rec.model || f.model || f.route_model || "?";
+      rec.tokens = f.tokens;
+      rec.tps = f.tokens_per_sec;
+      rec.elapsed = f.elapsed_seconds;
+      rec.stream = f.stream;
+      rec.terminal = e;
+    } else if (msg === "proxy failed" || msg === "direct proxy failed after relay") {
+      rec.status = "failed";
+      rec.model = rec.model || f.model || f.route_model || "?";
+      rec.error = f.error || msg;
+      rec.terminal = e;
+    }
+  }
+  return byId;
+}
+
+function callWallAgeSeconds(rec, nowSec) {
+  const start = rec.started || (rec.events[0] && rec.events[0].ts) || nowSec;
+  return Math.max(0, nowSec - start);
+}
+
+function callWallMaxQueue(rec) {
+  const nodes = rec.nodes || [];
+  const nodeQueues = Array.isArray(nodes) ? nodes.map(n => Number(n.queue_depth || 0)) : [];
+  return nodeQueues.length ? Math.max(...nodeQueues) : 0;
+}
+
+function renderCallWall(consoleData, stats) {
+  const events = (consoleData && consoleData.events) || [];
+  const nowSec = Date.now() / 1000;
+  const states = buildCallWallStates(events);
+  const active = [];
+  const terminal = [];
+  for (const rec of states.values()) {
+    if (rec.status === "pending" || rec.status === "processing") active.push(rec);
+    else if (rec.status === "complete" || rec.status === "failed") terminal.push(rec);
+  }
+  active.sort((a, b) => (a.started || 0) - (b.started || 0));
+  terminal.sort((a, b) => (b.terminal && b.terminal.ts) - (a.terminal && a.terminal.ts));
+
+  const hive = hiveThroughputSummary(stats);
+  const pending = active.filter(r => r.status === "pending").length;
+  const processing = active.filter(r => r.status === "processing").length;
+  const failedRecent = terminal.filter(r => r.status === "failed").length;
+  let queuedEstimate = 0;
+  for (const rec of active) queuedEstimate += Math.max(0, callWallMaxQueue(rec) - 1);
+
+  let html =
+    `<div class="dim" style="margin-bottom:6px">` +
+    `hive tps (1h): <b>${esc(tps(hive.totalTps))}</b> · samples: <b>${hive.samples}</b> · ` +
+    `active: <span class="status-processing">${processing}</span> processing · ` +
+    `<span class="status-pending">${pending}</span> pending` +
+    (queuedEstimate ? ` · queued estimate: <b>${queuedEstimate}</b>` : "") +
+    (failedRecent ? ` · <span class="status-failed">${failedRecent} recent failures</span>` : "") +
+    `</div>`;
+
+  if (active.length) {
+    html += table(["status", "age", "model", "request", "live tps", "tokens", "queue", "route / note"], active.map(rec => {
+      const statusCls = rec.status === "pending" ? "status-pending" : "status-processing";
+      const note = rec.warn || (rec.route ? short(String(rec.route), 28) : "");
+      return [
+        `<span class="${statusCls}">${esc(rec.status)}</span>`,
+        `<span class="num">${esc(callWallAgeSeconds(rec, nowSec).toFixed(1))}s</span>`,
+        esc(short(rec.model || "?", 28)),
+        esc(short(rec.id, 18)),
+        `<span class="num">${esc(tps(rec.tps))}</span>`,
+        `<span class="num">${esc(String(rec.tokens ?? "—"))}</span>`,
+        `<span class="num">${esc(String(callWallMaxQueue(rec)))}</span>`,
+        esc(note),
+      ];
+    }));
+  } else {
+    html += '<div class="empty">no in-flight requests</div>';
+  }
+
+  const historyRows = terminal.slice(0, 40).map(rec => {
+    const e = rec.terminal || {};
+    const f = e.fields || {};
+    const statusCls = rec.status === "failed" ? "status-failed" : "status-complete";
+    const detail = rec.status === "failed"
+      ? esc(short(rec.error || "?", 40))
+      : (f.stream ? "stream" : "json");
+    return [
+      new Date((e.ts || 0) * 1000).toLocaleTimeString(),
+      `<span class="${statusCls}">${esc(rec.status)}</span>`,
+      esc(short(rec.model || "?", 28)),
+      esc(short(rec.id, 18)),
+      `<span class="num">${esc(tps(rec.tps ?? f.tokens_per_sec))}</span>`,
+      `<span class="num">${esc(String(rec.tokens ?? f.tokens ?? "?"))}</span>`,
+      `<span class="num">${esc(String(rec.elapsed ?? f.elapsed_seconds ?? "?"))}</span>`,
+      detail,
+    ];
+  });
+  html += '<div style="margin-top:8px"><b class="dim">recent completed / failed</b></div>';
+  html += historyRows.length
+    ? table(["time", "status", "model", "request", "tps", "tokens", "sec", "detail"], historyRows)
+    : '<div class="empty">no completed requests yet</div>';
+  $("call-wall").innerHTML = html;
+}
+
+function startOfLocalDay(tsSec) {
+  const d = new Date(tsSec * 1000);
+  d.setHours(0, 0, 0, 0);
+  return d.getTime() / 1000;
+}
+
+function formatUsageDayLabel(tsSec) {
+  return new Date(tsSec * 1000).toLocaleDateString();
+}
+
+function summarizeUsageBuckets(records) {
+  const now = Date.now() / 1000;
+  const todayStart = startOfLocalDay(now);
+  const daySec = 86400;
+  const empty = () => ({ requests: 0, tokens: 0, cost: 0 });
+  const daily = [0, 1, 2].map(offset => ({
+    label: offset === 0 ? "Today" : offset === 1 ? "Yesterday" : formatUsageDayLabel(todayStart - offset * daySec),
+    ...empty(),
+  }));
+  const last7 = { label: "Last 7 days", ...empty() };
+  const last30 = { label: "Last 30 days", ...empty() };
+  const total = { label: "All time", ...empty() };
+
+  for (const u of records) {
+    const ts = Number(u.ts || 0);
+    const tokens = Number(u.total_tokens || 0);
+    const cost = Number(u.cost || 0);
+    total.requests += 1;
+    total.tokens += tokens;
+    total.cost += cost;
+    if (ts >= now - 30 * daySec) {
+      last30.requests += 1;
+      last30.tokens += tokens;
+      last30.cost += cost;
+    }
+    if (ts >= now - 7 * daySec) {
+      last7.requests += 1;
+      last7.tokens += tokens;
+      last7.cost += cost;
+    }
+    for (let offset = 0; offset < 3; offset++) {
+      const start = todayStart - offset * daySec;
+      const end = start + daySec;
+      if (ts >= start && ts < end) {
+        daily[offset].requests += 1;
+        daily[offset].tokens += tokens;
+        daily[offset].cost += cost;
+      }
+    }
+  }
+  return [...daily, last7, last30, total];
+}
+
+function renderUsageSummary(records) {
+  const el = $("usage-summary");
+  if (!el) return;
+  if (!sessionToken) {
+    el.innerHTML = '<div class="empty">login required</div>';
+    return;
+  }
+  if (!records.length) {
+    el.innerHTML = '<div class="empty">no billed requests yet</div>';
+    return;
+  }
+  const rows = summarizeUsageBuckets(records).map(b => [
+    esc(b.label),
+    `<span class="num">${b.requests}</span>`,
+    `<span class="num">${esc(String(b.tokens))}</span>`,
+    `<span class="num">${usdt(b.cost)}</span>`,
+  ]);
+  el.innerHTML =
+    '<div class="dim" style="margin-bottom:6px">per-request detail on Request history below</div>' +
+    table(["period", "requests", "tokens", "cost (USDT)"], rows);
+}
+
+function renderNodeThroughput(stats) {
+  const el = $("node-throughput");
+  if (!el) return;
+  if (!sessionToken) {
+    el.innerHTML = '<div class="empty">login required</div>';
+    return;
+  }
+  el.innerHTML = renderThroughputHtml(stats);
+}
+
+function renderBillingUsage(records) {
+  const el = $("billing-usage");
+  if (!el) return;
+  if (!sessionToken) {
+    el.innerHTML = '<div class="empty">login required</div>';
+    return;
+  }
+  if (!records.length) {
+    el.innerHTML = '<div class="empty">no billed requests yet</div>';
+    return;
+  }
+  const rows = records.slice().reverse().map(u => [
+    new Date((u.ts || 0) * 1000).toLocaleString(),
+    esc(short(u.model || "?", 28)),
+    esc(short(u.api_key || "?", 14)),
+    `<span class="num">${esc(String(u.total_tokens))}</span>`,
+    `<span class="num">${usdt(u.cost)}</span>`,
+  ]);
+  el.innerHTML = `<div class="dim" style="margin-bottom:6px">${records.length} request${records.length === 1 ? "" : "s"}</div>` +
+    table(["time", "model", "api key", "tokens", "cost (USDT)"], rows);
+}
+
+function renderConsole(data) {
+  const events = (data && data.events) || [];
+  if (!events.length) {
+    $("console").innerHTML = '<div class="empty">no console events</div>';
+    return;
+  }
+  $("console").innerHTML = events.slice(-120).map(e => {
+    const level = String(e.level || "info");
+    const cls = level === "error" ? "console-level-error" : level === "warn" ? "console-level-warn" : "console-level-info";
+    const fields = e.fields && Object.keys(e.fields).length ? " " + JSON.stringify(e.fields) : "";
+    return `<div class="console-line"><span class="console-time">${new Date((e.ts || 0) * 1000).toLocaleTimeString()}</span> ` +
+      `<span class="${cls}">${esc(level.toUpperCase())}</span> ${esc(e.message || "")}${esc(fields)}</div>`;
+  }).join("");
+}
 
 // ---- account panel (registration / login / balance / usage / API keys) ----
 
 let sessionToken = localStorage.getItem("meshnet_session") || null;
 let authTab = "login";
+let dashboardTab = "overview";
+let isAdmin = false;
+let isLoggedIn = false;
+let accountApiKeys = [];
+let accountUsageRecords = [];
+let lastStats = null;
+let availableModels = [];
+let chatHistory = [];
+let chatBusy = false;
+let selectedChatModel = localStorage.getItem("meshnet_chat_model") || "";
 
-async function apiCall(path, method, body) {
+function switchDashboardTab(name) {
+  if (name === "admin" && !isAdmin) name = "overview";
+  if (name === "billing" && !isLoggedIn) name = "overview";
+  dashboardTab = name;
+  updateSectionVisibility();
+  for (const tabName of ["overview", "chat", "billing", "admin"]) {
+    const button = $("tab-" + tabName);
+    if (button) button.classList.toggle("active", tabName === dashboardTab);
+  }
+}
+
+function updateSectionVisibility() {
+  for (const section of document.querySelectorAll("main section[data-tab]")) {
+    const onTab = section.dataset.tab === dashboardTab;
+    const adminOnly = section.hasAttribute("data-admin-only");
+    const loggedInOnly = section.hasAttribute("data-logged-in-only");
+    section.hidden = !onTab || (adminOnly && !isAdmin) || (loggedInOnly && !isLoggedIn);
+  }
+}
+
+function renderChatStatus(text) {
+  $("chat-status").textContent = text;
+}
+
+function renderChatHistory() {
+  const history = $("chat-history");
+  if (!chatHistory.length) {
+    history.classList.add("empty");
+    history.innerHTML = "no messages yet";
+    return;
+  }
+  history.classList.remove("empty");
+  history.innerHTML = chatHistory.map(msg => {
+    const roleClass = msg.role === "user" ? "chat-role-user" : msg.role === "assistant" ? "chat-role-assistant" : "chat-role-error";
+    const label = msg.role === "user" ? "user" : msg.role === "assistant" ? "assistant" : "error";
+    const meta = msg.model ? ` <span class="dim">· ${esc(short(msg.model, 24))}</span>` : "";
+    return `<div class="chat-message"><div class="chat-role ${roleClass}">${label}${meta}</div><div>${esc(msg.content)}</div></div>`;
+  }).join("");
+  history.scrollTop = history.scrollHeight;
+}
+
+function renderChatModels() {
+  const select = $("chat-model");
+  if (!select) return;
+  const models = availableModels.slice();
+  if (!models.length) {
+    select.innerHTML = '<option value="">no models available</option>';
+    select.disabled = true;
+    return;
+  }
+  select.disabled = false;
+  const preferred = models.find(m => m.id === selectedChatModel)
+    || models[0];
+  selectedChatModel = preferred.id;
+  localStorage.setItem("meshnet_chat_model", selectedChatModel);
+  select.innerHTML = models.map(model => {
+    const label = model.name && model.name !== model.id
+      ? `${model.name} (${model.id})`
+      : model.id;
+    const suffix = model.recommended ? " [recommended]" : "";
+    return `<option value="${esc(model.id)}"${model.id === selectedChatModel ? " selected" : ""}>${esc(label + suffix)}</option>`;
+  }).join("");
+  select.value = selectedChatModel;
+}
+
+function selectChatModel(value) {
+  selectedChatModel = value || "";
+  localStorage.setItem("meshnet_chat_model", selectedChatModel);
+}
+
+function clearChatHistory() {
+  chatHistory = [];
+  renderChatHistory();
+  renderChatStatus("history cleared");
+}
+
+function chatAuthToken() {
+  if (accountApiKeys.length) return accountApiKeys[0];
+  return null;
+}
+
+function setAdminMode(enabled) {
+  isAdmin = enabled;
+  $("tab-admin").style.display = enabled ? "" : "none";
+  if (!enabled && dashboardTab === "admin") {
+    switchDashboardTab("overview");
+  } else {
+    updateSectionVisibility();
+  }
+}
+
+function setLoggedInMode(enabled) {
+  isLoggedIn = enabled;
+  $("tab-billing").style.display = enabled ? "" : "none";
+  if (!enabled) {
+    accountUsageRecords = [];
+    renderBillingUsage([]);
+    renderUsageSummary([]);
+    renderNodeThroughput(null);
+    if (dashboardTab === "billing") switchDashboardTab("overview");
+  } else {
+    updateSectionVisibility();
+  }
+}
+
+async function apiCall(path, method, body, bearerToken) {
   const headers = { "Content-Type": "application/json" };
-  if (sessionToken) headers["Authorization"] = "Bearer " + sessionToken;
+  const token = bearerToken === undefined ? sessionToken : bearerToken;
+  if (token) headers["Authorization"] = "Bearer " + token;
   try {
     const r = await fetch(path, {
       method: method || "GET",
@@ -282,7 +725,10 @@ function renderAuthForms(errorMsg) {
   $("account").innerHTML =
     `<div class="tabs">${tab("login", "Log in")}${tab("register", "Register")}</div>` +
     form + `<div class="error-msg">${errorMsg ? esc(errorMsg) : ""}</div>`;
-  $("admin-section").style.display = "none";
+  accountApiKeys = [];
+  renderChatAuthHint();
+  setLoggedInMode(false);
+  setAdminMode(false);
 }
 
 function switchAuthTab(name) { authTab = name; renderAuthForms(); }
@@ -330,15 +776,82 @@ async function topupKey(key) {
   await renderAccountPanel();
 }
 
+const COPY_TOOLTIP_MS = 2000;
+
+function showCopiedTooltip(anchor) {
+  const box = (anchor && anchor.closest && anchor.closest(".keybox")) || anchor;
+  if (!box) return;
+  const existing = box.querySelector(".copy-tooltip");
+  if (existing) existing.remove();
+  const tip = document.createElement("span");
+  tip.className = "copy-tooltip";
+  tip.textContent = "Copied!";
+  tip.setAttribute("role", "status");
+  box.appendChild(tip);
+  setTimeout(() => tip.remove(), COPY_TOOLTIP_MS);
+}
+
+async function copyApiKeyText(text, anchor) {
+  try {
+    await navigator.clipboard.writeText(text);
+  } catch {
+    const ta = document.createElement("textarea");
+    ta.value = text;
+    ta.style.position = "fixed";
+    ta.style.left = "-9999px";
+    document.body.appendChild(ta);
+    ta.select();
+    try { document.execCommand("copy"); } catch { /* ignore */ }
+    document.body.removeChild(ta);
+  }
+  if (anchor) showCopiedTooltip(anchor);
+}
+
+function selectApiKeyText(el) {
+  const range = document.createRange();
+  range.selectNodeContents(el);
+  const sel = window.getSelection();
+  if (!sel) return;
+  sel.removeAllRanges();
+  sel.addRange(range);
+}
+
+function copyApiKeyFromTextEl(el) {
+  const key = el.dataset.key || el.textContent || "";
+  return copyApiKeyText(key, el);
+}
+
+function copyApiKeyFromButton(button) {
+  const el = button.closest(".keybox") && button.closest(".keybox").querySelector(".key-text");
+  const key = (el && el.dataset.key) || "";
+  return copyApiKeyText(key, button);
+}
+
+function renderChatAuthHint() {
+  if (chatAuthToken()) {
+    renderChatStatus("ready to send with your active API key");
+  } else if (sessionToken) {
+    renderChatStatus("create an API key in Account to use chat on a billing-enabled tracker");
+  } else {
+    renderChatStatus("log in if this tracker requires an API key");
+  }
+}
+
 async function renderAccountPanel() {
   const r = await apiCall("/v1/account");
   if (r.status === 404) {  // accounts disabled on this tracker
     $("account-section").style.display = "none";
-    $("admin-section").style.display = "none";
+    accountApiKeys = [];
+    accountUsageRecords = [];
+    renderChatAuthHint();
+    setLoggedInMode(false);
+    setAdminMode(false);
     return;
   }
   if (!r.ok) { setSession(null); renderAuthForms(); return; }
   const { account, api_keys, balances, total_balance, usage, topup_amount } = r.data;
+  accountApiKeys = Array.isArray(api_keys) ? api_keys.slice() : [];
+  accountUsageRecords = (usage && (usage.records || usage.recent)) || [];
   const who = account.email || account.wallet || account.account_id;
   let html =
     `<div><b>${esc(who)}</b> <span class="pill">${esc(account.role)}</span> ` +
@@ -350,8 +863,10 @@ async function renderAccountPanel() {
     '<button class="small" onclick="createKey()">+ new key</button></div>';
   if (api_keys.length) {
     for (const key of api_keys) {
-      html += `<div class="keybox">${esc(key)}` +
-        ` <span class="dim">(${usdt(balances[key] ?? 0)} USDT)</span>` +
+      html += `<div class="keybox">` +
+        `<span class="key-text" data-key="${esc(key)}" onclick="selectApiKeyText(this)" ondblclick="copyApiKeyFromTextEl(this)">${esc(key)}</span>` +
+        `<span class="dim">(${usdt(balances[key] ?? 0)} USDT)</span>` +
+        `<button class="small" type="button" onclick="copyApiKeyFromButton(this)">copy</button>` +
         (topup_amount > 0
           ? ` <button class="small" onclick="topupKey('${esc(key)}')">+${usdt(topup_amount)} (devnet)</button>`
           : "") +
@@ -360,24 +875,74 @@ async function renderAccountPanel() {
   } else {
     html += '<div class="empty">no active keys</div>';
   }
-  if (usage.recent && usage.recent.length) {
-    html += '<div style="margin-top:6px"><b class="dim">recent usage</b></div>' +
-      table(["time", "model", "tokens", "cost"], usage.recent.slice().reverse().map(u => [
-        new Date(u.ts * 1000).toLocaleTimeString(),
-        esc(short(u.model || "?", 24)),
-        `<span class="num">${esc(String(u.total_tokens))}</span>`,
-        `<span class="num">${usdt(u.cost)}</span>`,
-      ]));
-  }
   $("account").innerHTML = html;
+  renderUsageSummary(accountUsageRecords);
+  renderNodeThroughput(lastStats);
+  renderBillingUsage(accountUsageRecords);
+  renderChatAuthHint();
+  renderChatModels();
+  renderChatHistory();
+  setLoggedInMode(true);
+  setAdminMode(account.role === "admin");
   if (account.role === "admin") await renderAdminPanel();
-  else $("admin-section").style.display = "none";
+}
+
+async function sendChat() {
+  const promptEl = $("chat-prompt");
+  const prompt = promptEl.value.trim();
+  if (!prompt || chatBusy) return;
+  if (!selectedChatModel) {
+    renderChatStatus("select a model first");
+    return;
+  }
+  const bearerToken = chatAuthToken();
+  const body = {
+    model: selectedChatModel,
+    messages: [
+      ...chatHistory
+        .filter(msg => msg.role === "user" || msg.role === "assistant")
+        .map(msg => ({ role: msg.role, content: msg.content })),
+      { role: "user", content: prompt },
+    ],
+    stream: false,
+    max_tokens: 256,
+  };
+  chatBusy = true;
+  $("chat-send").disabled = true;
+  promptEl.value = "";
+  chatHistory.push({ role: "user", content: prompt, model: selectedChatModel });
+  renderChatHistory();
+  renderChatStatus("sending request…");
+  const r = await apiCall("/v1/chat/completions", "POST", body, bearerToken);
+  chatBusy = false;
+  $("chat-send").disabled = false;
+  if (!r.ok) {
+    const error = r.data && r.data.error
+      ? (typeof r.data.error === "string" ? r.data.error : r.data.error.message || "request failed")
+      : "request failed";
+    chatHistory.push({ role: "error", content: error, model: selectedChatModel });
+    renderChatHistory();
+    renderChatStatus(error);
+    promptEl.focus();
+    return;
+  }
+  const reply = (r.data && r.data.choices && r.data.choices[0] && r.data.choices[0].message && r.data.choices[0].message.content) || "";
+  const usage = r.data && r.data.usage;
+  chatHistory.push({
+    role: "assistant",
+    content: reply || "(empty response)",
+    model: selectedChatModel,
+  });
+  renderChatHistory();
+  renderChatStatus(usage
+    ? `done: ${usage.total_tokens ?? "?"} tokens`
+    : "done");
+  promptEl.focus();
 }
 
 async function renderAdminPanel() {
   const r = await apiCall("/v1/admin/accounts");
-  if (!r.ok) { $("admin-section").style.display = "none"; return; }
-  $("admin-section").style.display = "";
+  if (!r.ok) { setAdminMode(false); return; }
   const rows = (r.data.accounts || []).map(a => {
     const balance = Object.values(a.balances || {}).reduce((x, y) => x + y, 0);
     return [
@@ -393,27 +958,44 @@ async function renderAdminPanel() {
 
 async function refresh() {
   $("self-url").textContent = location.host;
-  const [raft, map, summary, settlements, wallets, stats, consoleData] = await Promise.all([
-    fetchJson("/v1/raft/status"),
-    fetchJson("/v1/network/map"),
-    fetchJson("/v1/billing/summary"),
-    fetchJson("/v1/billing/settlements"),
-    fetchJson("/v1/registry/wallets"),
-    fetchJson("/v1/stats"),
-    fetchJson("/v1/console"),
-  ]);
+  const [raft, map, stats, models, consoleData, adminData] = await Promise.all([
+    fetchJson("/v1/raft/status"),
+    fetchJson("/v1/network/map"),
+    fetchJson("/v1/stats"),
+    fetchJson("/v1/models"),
+    fetchJson("/v1/console"),
+    isAdmin ? Promise.all([
+      fetchJson("/v1/billing/summary"),
+      fetchJson("/v1/billing/settlements"),
+      fetchJson("/v1/registry/wallets"),
+    ]) : Promise.resolve([null, null, null]),
+  ]);
+  const [summary, settlements, wallets] = adminData;
+  lastStats = stats;
+  availableModels = ((models && models.data) || []).map(model => ({
+    id: model.id,
+    name: model.name || model.id,
+    recommended: Boolean(model.recommended),
+    aliases: model.aliases || [],
+  })).filter(model => model.id);
   renderHive(raft);
   renderNodes(map);
   renderBilling(summary);
   renderSettlements(settlements);
   renderFraud(wallets, summary);
-  renderStats(stats);
-  renderThroughput(stats);
-  renderConsole(consoleData);
+  renderStats(stats);
+  renderCallWall(consoleData, stats);
+  renderConsole(consoleData);
+  renderNodeThroughput(stats);
+  renderChatModels();
+  renderChatHistory();
   $("refreshed").textContent = "refreshed " + new Date().toLocaleTimeString();
 }
 refresh();
 renderAccountPanel();
+renderChatModels();
+renderChatHistory();
+renderChatAuthHint();
 setInterval(refresh, 4000);
 setInterval(() => { if (sessionToken) renderAccountPanel(); }, 8000);
 </script>

tests/test_dashboard.py

@@ -1,6 +1,7 @@
 """US-035: tracker web dashboard — served from any tracker, embedded asset."""
 
 import json
+import time
 import urllib.request
 
 from meshnet_contracts import LocalSolanaContracts
@@ -11,7 +12,9 @@ from meshnet_tracker.server import TrackerServer
 PANELS = [
     "Tracker hive", "Nodes & coverage", "Client balances",
     "Node pending payouts", "Settlement history",
-    "Strikes / bans / forfeitures", "Model usage", "Node throughput",
+    "Strikes / bans / forfeitures", "Model usage", "Call wall",
+    "Usage summary", "Node throughput", "Request history",
+    "Chat / inference",
     "Console output",
 ]
 
@@ -91,3 +94,45 @@ def test_console_endpoint_exposes_tracker_events():
         tracker.stop()
 
     assert any(event["message"] == "node registered" for event in data["events"])
+
+
+def test_console_node_lifecycle_events_include_model_health():
+    tracker = TrackerServer(heartbeat_timeout=0.05)
+    port = tracker.start()
+    try:
+        body = json.dumps({
+            "endpoint": "http://127.0.0.1:9002",
+            "model": "console-health-test",
+            "hf_repo": "example/console-health-test",
+            "num_layers": 4,
+            "shard_start": 0,
+            "shard_end": 1,
+            "hardware_profile": {},
+        }).encode()
+        req = urllib.request.Request(
+            f"http://127.0.0.1:{port}/v1/nodes/register",
+            data=body,
+            headers={"Content-Type": "application/json"},
+            method="POST",
+        )
+        urllib.request.urlopen(req).read()
+
+        registered = json.loads(urllib.request.urlopen(f"http://127.0.0.1:{port}/v1/console").read())
+        registered_event = next(
+            event for event in registered["events"]
+            if event["message"] == "node registered"
+        )
+        assert registered_event["fields"]["model_health"]["served_model_copies"] == 0.5
+        assert registered_event["fields"]["model_health"]["coverage_percentage"] == 50.0
+
+        time.sleep(0.06)
+        urllib.request.urlopen(f"http://127.0.0.1:{port}/v1/network/map").read()
+        expired = json.loads(urllib.request.urlopen(f"http://127.0.0.1:{port}/v1/console").read())
+        expired_event = next(
+            event for event in expired["events"]
+            if event["message"] == "node expired"
+        )
+        assert expired_event["fields"]["model_health"]["served_model_copies"] == 0.0
+        assert expired_event["fields"]["model_health"]["coverage_percentage"] == 0.0
+    finally:
+        tracker.stop()

tests/test_mining_cli.py

@@ -388,6 +388,107 @@ def test_legacy_start_treats_repo_model_as_model_id(monkeypatch):
     assert captured["model_id"] == "Qwen/Qwen2.5-0.5B-Instruct"
 
 
+def test_legacy_start_catalog_model_with_pinned_shards(monkeypatch):
+    """Catalog model names accept --shard-start/--shard-end without --model-id."""
+    from meshnet_node.cli import main
+
+    captured = {}
+
+    def fake_run_startup(*args, **kwargs):
+        captured.update(kwargs)
+        class _FakeNode:
+            chat_completion_count = 0
+            def stop(self): pass
+        return _FakeNode()
+
+    monkeypatch.setattr(sys, "argv", [
+        "meshnet-node", "start",
+        "--tracker", "http://192.168.0.179:8080",
+        "--model", "Qwen3.6-35B-A3B",
+        "--shard-start", "0",
+        "--shard-end", "44",
+        "--port", "0",
+    ])
+
+    with patch("meshnet_node.startup.run_startup", side_effect=fake_run_startup):
+        with patch("time.sleep", side_effect=KeyboardInterrupt):
+            try:
+                main()
+            except SystemExit as exc:
+                assert exc.code == 0
+
+    assert captured["model"] == "Qwen3.6-35B-A3B"
+    assert captured["model_id"] is None
+    assert captured["shard_start"] == 0
+    assert captured["shard_end"] == 44
+
+
+def test_legacy_start_model_id_alias_for_catalog_name(monkeypatch):
+    """--model-id with a catalog name routes through the tracker preset path."""
+    from meshnet_node.cli import main
+
+    captured = {}
+
+    def fake_run_startup(*args, **kwargs):
+        captured.update(kwargs)
+        class _FakeNode:
+            chat_completion_count = 0
+            def stop(self): pass
+        return _FakeNode()
+
+    monkeypatch.setattr(sys, "argv", [
+        "meshnet-node", "start",
+        "--tracker", "http://192.168.0.179:8080",
+        "--model-id", "Qwen3.6-35B-A3B",
+        "--port", "0",
+    ])
+
+    with patch("meshnet_node.startup.run_startup", side_effect=fake_run_startup):
+        with patch("time.sleep", side_effect=KeyboardInterrupt):
+            try:
+                main()
+            except SystemExit as exc:
+                assert exc.code == 0
+
+    assert captured["model"] == "Qwen3.6-35B-A3B"
+    assert captured["model_id"] is None
+
+
+def test_legacy_start_hf_repo_with_pinned_shards(monkeypatch):
+    """HF repo --model with pinned shards still enters the torch startup path."""
+    from meshnet_node.cli import main
+
+    captured = {}
+
+    def fake_run_startup(*args, **kwargs):
+        captured.update(kwargs)
+        class _FakeNode:
+            chat_completion_count = 0
+            def stop(self): pass
+        return _FakeNode()
+
+    monkeypatch.setattr(sys, "argv", [
+        "meshnet-node", "start",
+        "--tracker", "http://192.168.0.179:8081",
+        "--model", "Qwen/Qwen2.5-0.5B-Instruct",
+        "--shard-start", "12",
+        "--shard-end", "23",
+        "--port", "0",
+    ])
+
+    with patch("meshnet_node.startup.run_startup", side_effect=fake_run_startup):
+        with patch("time.sleep", side_effect=KeyboardInterrupt):
+            try:
+                main()
+            except SystemExit as exc:
+                assert exc.code == 0
+
+    assert captured["model"] == "Qwen2.5-0.5B-Instruct"
+    assert captured["model_id"] == "Qwen/Qwen2.5-0.5B-Instruct"
+    assert captured["shard_start"] == 12
+    assert captured["shard_end"] == 23
+
+
 def test_legacy_start_falls_back_to_env_tracker_and_model(monkeypatch):
     """`meshnet-node start` uses env defaults when tracker/model flags are omitted."""
     import importlib

tests/test_node_startup.py

@@ -5,8 +5,10 @@ import io
 import os
 import sys
 import tarfile
+import threading
 import time
 import types
+import urllib.error
 import urllib.request
 from pathlib import Path
 
@@ -566,6 +568,78 @@ def test_download_shard_prefers_tracker_model_source_over_huggingface(
     assert hf_calls == []
 
 
+def test_download_shard_prefers_tracker_full_model_source_over_huggingface(
+    tmp_path,
+    monkeypatch,
+):
+    """A tracker-advertised full snapshot is sufficient on its own — HF is never contacted."""
+    contents = {
+        "config.json": b"{}",
+        "weights-a.safetensors": b"tracker-a",
+        "weights-b.safetensors": b"tracker-b",
+    }
+
+    class FakeFileResponse:
+        def __init__(self, payload: bytes):
+            self._payload = io.BytesIO(payload)
+            self._length = len(payload)
+
+        def __enter__(self):
+            return self
+
+        def __exit__(self, exc_type, exc, tb):
+            return False
+
+        def getheader(self, name: str):
+            if name == "Content-Length":
+                return str(self._length)
+            if name == "Content-Type":
+                return "application/octet-stream"
+            return None
+
+        def read(self, size: int = -1) -> bytes:
+            return self._payload.read(size)
+
+    def fake_urlopen(url, *args, **kwargs):
+        query = urllib.parse.parse_qs(urllib.parse.urlparse(url).query)
+        rel = query.get("file", [None])[0]
+        assert rel in contents, f"unexpected per-file request: {url}"
+        return FakeFileResponse(contents[rel])
+
+    monkeypatch.setattr(urllib.request, "urlopen", fake_urlopen)
+    hf_calls = []
+
+    def fake_snapshot_download(**kwargs):
+        hf_calls.append(kwargs)
+        raise AssertionError("HuggingFace should not be contacted when tracker full_files are available")
+
+    monkeypatch.setitem(
+        sys.modules,
+        "huggingface_hub",
+        types.SimpleNamespace(snapshot_download=fake_snapshot_download),
+    )
+
+    shard_dir = download_shard(
+        "tiny-llama",
+        0,
+        3,
+        cache_dir=tmp_path / "cache",
+        hf_repo="org/tiny-llama-shards",
+        model_sources=[{
+            "type": "tracker-full",
+            "url": "http://tracker/v1/model-files/download?model=tiny-llama&full=1",
+            "files": ["config.json", "weights-a.safetensors", "weights-b.safetensors"],
+            "full_files": ["config.json", "weights-a.safetensors", "weights-b.safetensors"],
+        }],
+        progress=False,
+    )
+
+    assert (shard_dir / "config.json").read_text() == "{}"
+    assert (shard_dir / "weights-a.safetensors").read_text() == "tracker-a"
+    assert (shard_dir / "weights-b.safetensors").read_text() == "tracker-b"
+    assert hf_calls == []
+
+
 def test_download_shard_falls_back_to_huggingface_when_tracker_source_fails(
     tmp_path,
     monkeypatch,
@@ -1359,6 +1433,84 @@ def test_later_node_auto_joins_existing_public_hf_model_with_only_tracker_url(
     assert route_resp["route"] == ["http://203.0.113.20:8001", "http://203.0.113.21:8002"]
 
 
+def test_later_node_auto_joins_redundant_copy_when_model_is_fully_covered(
+    tmp_path,
+    monkeypatch,
+):
+    """Model-less joins should load the served HF model even when gap_found=false."""
+    import meshnet_node.startup as startup_mod
+
+    captured = {}
+
+    class FakeBackend:
+        total_layers = 24
+
+    class FakeTorchNodeServer:
+        def __init__(self, **kwargs):
+            captured.update(kwargs)
+            self.backend = FakeBackend()
+            self.port = None
+            self.chat_completion_count = 0
+            self.total_requests = 0
+            self.failed_requests = 0
+            self.queue_depth = 0
+
+        def start(self):
+            self.port = 8003
+            return self.port
+
+        def stop(self):
+            pass
+
+    monkeypatch.setattr(
+        startup_mod,
+        "detect_hardware",
+        lambda: {"device": "cpu", "gpu_name": None, "vram_mb": 0},
+    )
+    monkeypatch.setattr(startup_mod, "TorchNodeServer", FakeTorchNodeServer)
+
+    tracker = TrackerServer()
+    tracker_port = tracker.start()
+    tracker_url = f"http://127.0.0.1:{tracker_port}"
+    try:
+        for endpoint, shard_start, shard_end in (
+            ("http://203.0.113.30:8001", 0, 11),
+            ("http://203.0.113.31:8001", 12, 23),
+        ):
+            data = json.dumps({
+                "endpoint": endpoint,
+                "model": "Qwen2.5-0.5B-Instruct",
+                "hf_repo": "Qwen/Qwen2.5-0.5B-Instruct",
+                "num_layers": 24,
+                "shard_start": shard_start,
+                "shard_end": shard_end,
+                "tracker_mode": shard_start == 0,
+                "hardware_profile": {},
+                "score": 1.0,
+            }).encode()
+            req = urllib.request.Request(
+                f"{tracker_url}/v1/nodes/register",
+                data=data,
+                headers={"Content-Type": "application/json"},
+                method="POST",
+            )
+            with urllib.request.urlopen(req) as resp:
+                resp.read()
+
+        node = run_startup(
+            tracker_url=tracker_url,
+            advertise_host="203.0.113.32",
+            wallet_path=tmp_path / "wallet.json",
+        )
+        try:
+            assert captured["model_id"] == "Qwen/Qwen2.5-0.5B-Instruct"
+            assert captured["shard_start"] == 0
+        finally:
+            node.stop()
+    finally:
+        tracker.stop()
+
+
 # ---------------------------------------------------------------------------
 # Full startup integration test
 # ---------------------------------------------------------------------------
@@ -1453,6 +1605,120 @@ def test_preset_model_startup_starts_heartbeat(tmp_path, monkeypatch):
         tracker.stop()
 
 
+def test_preset_model_startup_honors_pinned_shard_range(tmp_path, monkeypatch):
+    """Explicit --shard-start/--shard-end override tracker auto-assignment."""
+    import meshnet_node.startup as startup_mod
+
+    monkeypatch.setattr(
+        startup_mod,
+        "detect_hardware",
+        lambda: {"device": "cpu", "gpu_name": None, "vram_mb": 0, "ram_mb": 16 * 1024},
+    )
+    heartbeat_calls = []
+    monkeypatch.setattr(
+        startup_mod,
+        "_start_heartbeat",
+        lambda *args, **kwargs: heartbeat_calls.append((args, kwargs)),
+    )
+
+    tracker = TrackerServer(model_presets={"stub-model": {"layers_start": 0, "layers_end": 15}})
+    tracker_port = tracker.start()
+    tracker_url = f"http://127.0.0.1:{tracker_port}"
+    try:
+        node = run_startup(
+            tracker_url=tracker_url,
+            model="stub-model",
+            shard_start=0,
+            shard_end=5,
+            wallet_path=tmp_path / "wallet.json",
+            cache_dir=tmp_path / "shards",
+        )
+        try:
+            assert len(heartbeat_calls) == 1
+            args, kwargs = heartbeat_calls[0]
+            reg_payload = args[2]
+            assert reg_payload["shard_start"] == 0
+            assert reg_payload["shard_end"] == 5
+            assert reg_payload["managed_assignment"] is False
+        finally:
+            node.stop()
+    finally:
+        tracker.stop()
+
+
+def test_torch_startup_retries_registration_when_tracker_unreachable(
+    tmp_path,
+    monkeypatch,
+):
+    """Failed initial registration should start background retry, not stay unregistered."""
+    import meshnet_node.startup as startup_mod
+
+    class FakeBackend:
+        total_layers = 24
+
+    class FakeTorchNodeServer:
+        def __init__(self, **kwargs):
+            self.backend = FakeBackend()
+            self.port = None
+            self.chat_completion_count = 0
+            self.tracker_node_id = None
+
+        def start(self):
+            self.port = 7000
+            return self.port
+
+        def stop(self):
+            pass
+
+    monkeypatch.setattr(
+        startup_mod,
+        "detect_hardware",
+        lambda: {"device": "cuda", "gpu_name": "Test GPU", "vram_mb": 8192, "ram_mb": 16 * 1024},
+    )
+    monkeypatch.setattr(startup_mod, "TorchNodeServer", FakeTorchNodeServer)
+    monkeypatch.setattr(
+        startup_mod,
+        "_detect_num_layers",
+        lambda *_args, **_kwargs: 24,
+    )
+
+    heartbeat_calls = []
+    monkeypatch.setattr(
+        startup_mod,
+        "_start_heartbeat",
+        lambda *args, **kwargs: heartbeat_calls.append((args, kwargs)) or threading.Thread(),
+    )
+
+    register_calls = {"count": 0}
+
+    def flaky_register(url, payload):
+        register_calls["count"] += 1
+        raise urllib.error.URLError("connection refused")
+
+    monkeypatch.setattr(startup_mod, "_post_json", flaky_register)
+
+    tracker = TrackerServer()
+    tracker_port = tracker.start()
+    tracker_url = f"http://127.0.0.1:{tracker_port}"
+    try:
+        node = run_startup(
+            tracker_url=tracker_url,
+            model_id="Qwen/Qwen2.5-0.5B-Instruct",
+            wallet_path=tmp_path / "wallet.json",
+        )
+        try:
+            assert register_calls["count"] == 1
+            assert node.tracker_node_id is None
+            assert len(heartbeat_calls) == 1
+            args, kwargs = heartbeat_calls[0]
+            assert args[1] == startup_mod._PENDING_NODE_ID
+            assert kwargs["node_ref"] is node
+        finally:
+            node.stop()
+    finally:
+        tracker.stop()
+
+
 def test_real_model_startup_registers_downloaded_inventory_without_checksum(
     tmp_path,
     monkeypatch,

tests/test_real_model_backend.py

@@ -4,6 +4,8 @@ import json
 import os
 from pathlib import Path
 import sys
+import threading
+import time
 import types
 import urllib.request
 
@@ -11,8 +13,12 @@ import pytest
 
 from meshnet_node.model_backend import (
     InsufficientVRAMError,
+    PartialModelLoadUnsupported,
     TensorPayload,
+    TorchModelShard,
     _call_layer,
+    _load_partial_model_from_snapshot,
+    _should_partial_materialize_shard,
     _decoder_attention_mask,
     _int_tensor_header,
     build_quantization_config,
@@ -94,7 +100,7 @@ class _FakePipelineHeadBackend(_FakeBackend):
     tokenizer = _FakeChatTokenizer()
 
     def encode_prompt(self, prompt: str) -> TensorPayload:
-        assert prompt == "debug prompt"
+        assert prompt.startswith("debug prompt")
         return TensorPayload(
             body=b"\x00" * (1 * 6 * 8 * 2),
             shape=[1, 6, 8],
@@ -113,6 +119,19 @@ class _FakePipelineTailBackend(_FakeTailBackend):
         return " token"
 
 
+class _BlockingStreamingTailBackend(_FakeTailBackend):
+    def __init__(self, second_token_release: threading.Event) -> None:
+        self._release = second_token_release
+        self.calls = 0
+
+    def forward_bytes(self, body, shape, attention_mask_header, position_ids_header, start_layer=None):
+        self.calls += 1
+        if self.calls == 1:
+            return " first"
+        self._release.wait(timeout=3.0)
+        return " second"
+
+
 def test_quantization_flag_validation():
     assert validate_quantization("bfloat16") == "bfloat16"
     assert validate_quantization("int8") == "int8"
@@ -299,6 +318,56 @@ def test_pipeline_hop_logs_are_enabled_with_debug(capsys):
     assert "  [node] pipeline hop 0 returned text=' token'" in out
 
 
+def test_split_shard_chat_streams_each_generated_token_incrementally():
+    release_second = threading.Event()
+    head = TorchNodeServer(backend=_FakePipelineHeadBackend(), tracker_mode=True)
+    tail = TorchNodeServer(backend=_BlockingStreamingTailBackend(release_second))
+    head_port = head.start()
+    tail_port = tail.start()
+    response = None
+    try:
+        payload = json.dumps({
+            "model": "fake-model",
+            "messages": [{"role": "user", "content": "hello"}],
+            "stream": True,
+            "max_tokens": 2,
+        }).encode()
+        req = urllib.request.Request(
+            f"http://127.0.0.1:{head_port}/v1/chat/completions",
+            data=payload,
+            headers={
+                "Content-Type": "application/json",
+                "X-Meshnet-Route": json.dumps([
+                    {"endpoint": f"http://127.0.0.1:{tail_port}", "start_layer": 22},
+                ]),
+            },
+            method="POST",
+        )
+        response = urllib.request.urlopen(req, timeout=5)
+
+        first_token_line = ""
+        deadline = time.time() + 2.0
+        while time.time() < deadline:
+            line = response.readline().decode()
+            if '"content": " first"' in line:
+                first_token_line = line
+                break
+
+        assert first_token_line
+        assert not release_second.is_set()
+        release_second.set()
+        rest = response.read().decode()
+    finally:
+        release_second.set()
+        if response is not None:
+            response.close()
+        head.stop()
+        tail.stop()
+
+    assert '"content": " second"' in rest
+    assert "data: [DONE]" in rest
+
+
 def test_int_tensor_header_serializes_torch_tensors():
     torch = pytest.importorskip("torch")
 
@@ -334,6 +403,295 @@ def test_call_layer_passes_rotary_position_embeddings():
     ) == "hidden"
 
 
+def test_partial_materialize_guard_requires_local_non_full_non_quantized_snapshot(tmp_path):
+    snapshot_dir = tmp_path / "snapshot"
+    snapshot_dir.mkdir()
+    (snapshot_dir / "config.json").write_text("{}")
+    (snapshot_dir / "model.safetensors.index.json").write_text('{"weight_map": {}}')
+
+    assert _should_partial_materialize_shard(
+        str(snapshot_dir),
+        4,
+        7,
+        total_layers_hint=40,
+        uses_quantized_weights=False,
+    ) is True
+    assert _should_partial_materialize_shard(
+        str(snapshot_dir),
+        0,
+        39,
+        total_layers_hint=40,
+        uses_quantized_weights=False,
+    ) is False
+    assert _should_partial_materialize_shard(
+        str(snapshot_dir),
+        4,
+        7,
+        total_layers_hint=40,
+        uses_quantized_weights=True,
+    ) is False
+    assert _should_partial_materialize_shard(
+        "repo/model",
+        4,
+        7,
+        total_layers_hint=40,
+        uses_quantized_weights=False,
+    ) is False
+
+
+def test_partial_snapshot_loader_materializes_only_assigned_tensors(tmp_path):
+    snapshot_dir = tmp_path / "snapshot"
+    snapshot_dir.mkdir()
+    (snapshot_dir / "config.json").write_text("{}")
+    (snapshot_dir / "model.safetensors.index.json").write_text(json.dumps({
+        "weight_map": {
+            "model.embed_tokens.weight": "shard-1.safetensors",
+            "model.layers.0.self_attn.q_proj.weight": "shard-1.safetensors",
+            "model.layers.1.self_attn.q_proj.weight": "shard-2.safetensors",
+            "model.layers.2.self_attn.q_proj.weight": "shard-3.safetensors",
+            "model.norm.weight": "shard-3.safetensors",
+            "lm_head.weight": "shard-3.safetensors",
+        }
+    }))
+    for rel in ("shard-1.safetensors", "shard-2.safetensors", "shard-3.safetensors"):
+        (snapshot_dir / rel).write_bytes(b"stub")
+
+    class FakeModule:
+        def __init__(self, name):
+            self.name = name
+            self.to_calls = []
+
+        def to(self, device):
+            self.to_calls.append(device)
+            return self
+
+    class FakeModel:
+        def __init__(self):
+            self.model = types.SimpleNamespace(
+                embed_tokens=FakeModule("embed"),
+                layers=[FakeModule("layer0"), FakeModule("layer1"), FakeModule("layer2")],
+                rotary_emb=FakeModule("rotary"),
+                norm=FakeModule("norm"),
+            )
+            self.lm_head = FakeModule("lm_head")
+            self.tie_weights_called = 0
+
+        def tie_weights(self):
+            self.tie_weights_called += 1
+
+    class AutoConfigStub:
+        @staticmethod
+        def from_pretrained(model_id):
+            assert model_id == str(snapshot_dir)
+            return types.SimpleNamespace(num_hidden_layers=3)
+
+    class AutoModelStub:
+        @staticmethod
+        def from_config(cfg, torch_dtype=None):
+            assert cfg.num_hidden_layers == 3
+            assert torch_dtype == "bf16"
+            return FakeModel()
+
+    class EmptyWeights:
+        def __init__(self):
+            self.entered = 0
+            self.exited = 0
+
+        def __call__(self):
+            return self
+
+        def __enter__(self):
+            self.entered += 1
+            return None
+
+        def __exit__(self, exc_type, exc, tb):
+            self.exited += 1
+            return False
+
+    init_empty_weights = EmptyWeights()
+    set_calls = []
+
+    def fake_set_tensor(module, tensor_name, device, value=None, dtype=None):
+        set_calls.append((tensor_name, device, value, dtype))
+
+    tensors = {
+        "shard-1.safetensors": {
+            "model.embed_tokens.weight": "embed",
+            "model.layers.0.self_attn.q_proj.weight": "layer0",
+        },
+        "shard-2.safetensors": {
+            "model.layers.1.self_attn.q_proj.weight": "layer1",
+        },
+        "shard-3.safetensors": {
+            "model.layers.2.self_attn.q_proj.weight": "layer2",
+            "model.norm.weight": "norm",
+            "lm_head.weight": "lm_head",
+        },
+    }
+
+    class FakeSafeOpen:
+        def __init__(self, filename, framework, device):
+            assert framework == "pt"
+            assert device == "cpu"
+            self.filename = Path(filename).name
+
+        def __enter__(self):
+            return self
+
+        def __exit__(self, exc_type, exc, tb):
+            return False
+
+        def get_tensor(self, tensor_name):
+            return tensors[self.filename][tensor_name]
+
+    model = _load_partial_model_from_snapshot(
+        AutoConfigStub,
+        AutoModelStub,
+        types.SimpleNamespace(),
+        str(snapshot_dir),
+        1,
+        1,
+        "bf16",
+        "cpu:0",
+        init_empty_weights_fn=init_empty_weights,
+        set_tensor_fn=fake_set_tensor,
+        safe_open_fn=FakeSafeOpen,
+    )
+
+    assert init_empty_weights.entered == 1
+    assert init_empty_weights.exited == 1
+    assert model.tie_weights_called == 1
+    assert [call[0] for call in set_calls] == ["model.layers.1.self_attn.q_proj.weight"]
+    assert model.model.layers[1].to_calls == ["cpu:0"]
+    assert model.model.layers[0].to_calls == []
+    assert model.model.layers[2].to_calls == []
+    assert model.model.embed_tokens.to_calls == []
+    assert model.model.norm.to_calls == []
+    assert model.lm_head.to_calls == []
+    assert model.model.rotary_emb.to_calls == ["cpu:0"]
+
+
+def test_partial_snapshot_loader_requires_known_layer_count(tmp_path):
+    snapshot_dir = tmp_path / "snapshot"
+    snapshot_dir.mkdir()
+    (snapshot_dir / "config.json").write_text("{}")
+    (snapshot_dir / "model.safetensors.index.json").write_text(json.dumps({
+        "weight_map": {"model.layers.0.self_attn.q_proj.weight": "shard.safetensors"}
+    }))
+    (snapshot_dir / "shard.safetensors").write_bytes(b"stub")
+
+    class AutoConfigStub:
+        @staticmethod
+        def from_pretrained(model_id):
+            return types.SimpleNamespace()
+
+    class AutoModelStub:
+        @staticmethod
+        def from_config(cfg, torch_dtype=None):
+            raise AssertionError("from_config should not run without a known layer count")
+
+    class UnusedContext:
+        def __enter__(self):
+            return None
+
+        def __exit__(self, exc_type, exc, tb):
+            return False
+
+    with pytest.raises(PartialModelLoadUnsupported, match="num_hidden_layers"):
+        _load_partial_model_from_snapshot(
+            AutoConfigStub,
+            AutoModelStub,
+            types.SimpleNamespace(),
+            str(snapshot_dir),
+            0,
+            0,
+            "bf16",
+            "cpu:0",
+            init_empty_weights_fn=lambda: UnusedContext(),
+            set_tensor_fn=lambda *args, **kwargs: None,
+            safe_open_fn=lambda *args, **kwargs: None,
+        )
+
+
+def test_torch_model_shard_prefers_partial_loader_for_local_snapshot(tmp_path, monkeypatch):
+    import meshnet_node.model_backend as backend
+
+    snapshot_dir = tmp_path / "snapshot"
+    snapshot_dir.mkdir()
+    (snapshot_dir / "config.json").write_text("{}")
+    (snapshot_dir / "model.safetensors.index.json").write_text('{"weight_map": {}}')
+
+    class FakeModel:
+        def __init__(self):
+            self.model = types.SimpleNamespace(
+                layers=[object(), object(), object()],
+                embed_tokens=object(),
+            )
+            self.config = types.SimpleNamespace(hidden_size=8)
+            self.eval_called = 0
+
+        def eval(self):
+            self.eval_called += 1
+
+    fake_model = FakeModel()
+    partial_calls = []
+
+    class AutoConfigStub:
+        @staticmethod
+        def from_pretrained(model_id, cache_dir=None):
+            return types.SimpleNamespace(num_hidden_layers=3, text_config=types.SimpleNamespace(dtype="torch.bfloat16"))
+
+    class AutoModelStub:
+        @staticmethod
+        def from_pretrained(*args, **kwargs):
+            raise AssertionError("full model load should not run for partial local shards")
+
+    class AutoTokenizerStub:
+        @staticmethod
+        def from_pretrained(model_id, cache_dir=None):
+            assert model_id == str(snapshot_dir)
+            return types.SimpleNamespace()
+
+    monkeypatch.setitem(
+        sys.modules,
+        "torch",
+        types.SimpleNamespace(
+            cuda=types.SimpleNamespace(is_available=lambda: False),
+            device=lambda value: value,
+            bfloat16="bf16",
+        ),
+    )
+    monkeypatch.setitem(
+        sys.modules,
+        "transformers",
+        types.SimpleNamespace(
+            AutoConfig=AutoConfigStub,
+            AutoModelForCausalLM=AutoModelStub,
+            AutoTokenizer=AutoTokenizerStub,
+        ),
+    )
+    monkeypatch.setattr(
+        backend,
+        "_load_partial_model_from_snapshot",
+        lambda *args, **kwargs: partial_calls.append((args, kwargs)) or fake_model,
+    )
+
+    shard = TorchModelShard(
+        "repo/model",
+        1,
+        1,
+        quantization="auto",
+        cache_dir=snapshot_dir,
+    )
+
+    assert len(partial_calls) == 1
+    assert shard.model is fake_model
+    assert fake_model.eval_called == 1
+    assert shard.total_layers == 3
+    assert shard.is_head is False
+    assert shard.is_tail is False
+
+
 @pytest.mark.integration
 def test_two_node_gpt2_completion_is_deterministic():
     if os.environ.get("CI"):

tests/test_tracker_routing.py

@@ -13,7 +13,13 @@ from meshnet_gateway.server import GatewayServer, _banned_route_wallet
 from meshnet_node.server import StubNodeServer
 from meshnet_contracts import LocalSolanaContracts
 from meshnet_tracker.auth import sign_hive_request
-from meshnet_tracker.server import TrackerServer, _NodeEntry, _registration_ban_error
+from meshnet_tracker.server import (
+    TrackerServer,
+    _NodeEntry,
+    _memory_pool_map,
+    _registration_ban_error,
+    _scale_demanded_models_locked,
+)
 
 _TEST_HIVE_SECRET = "test-hive-secret"
 
@@ -162,6 +168,59 @@ def test_network_map_exposes_pool_size_and_speed_summary():
     assert pool["total_effective_throughput"] == 10.0
 
 
+def test_network_map_exposes_served_model_copy_count():
+    tracker = TrackerServer()
+    port = tracker.start()
+    url = f"http://127.0.0.1:{port}"
+    try:
+        _post_json(
+            f"{url}/v1/nodes/register",
+            {
+                "endpoint": "http://127.0.0.1:7201",
+                "model": "copy-count-test",
+                "hf_repo": "example/copy-count-test",
+                "num_layers": 37,
+                "shard_start": 0,
+                "shard_end": 21,
+                "hardware_profile": {},
+            },
+        )
+        network_map = _get_json(f"{url}/v1/network/map")
+        assert network_map["nodes"][0]["model_supply"]["served_model_copies"] == 0.59
+
+        _post_json(
+            f"{url}/v1/nodes/register",
+            {
+                "endpoint": "http://127.0.0.1:7202",
+                "model": "copy-count-test",
+                "hf_repo": "example/copy-count-test",
+                "num_layers": 37,
+                "shard_start": 22,
+                "shard_end": 36,
+                "hardware_profile": {},
+            },
+        )
+        network_map = _get_json(f"{url}/v1/network/map")
+        assert network_map["nodes"][0]["model_supply"]["served_model_copies"] == 1.0
+
+        _post_json(
+            f"{url}/v1/nodes/register",
+            {
+                "endpoint": "http://127.0.0.1:7203",
+                "model": "copy-count-test",
+                "hf_repo": "example/copy-count-test",
+                "num_layers": 37,
+                "shard_start": 0,
+                "shard_end": 36,
+                "hardware_profile": {},
+            },
+        )
+        network_map = _get_json(f"{url}/v1/network/map")
+        assert network_map["nodes"][0]["model_supply"]["served_model_copies"] == 2.0
+    finally:
+        tracker.stop()
+
+
 def test_recommended_kimi_becomes_deployable_when_pool_is_large_enough():
     tracker = TrackerServer()
     port = tracker.start()
@@ -271,6 +330,177 @@ def test_tracker_serves_health_while_proxy_request_is_in_flight():
         slow_thread.join(timeout=1.0)
 
 
+def test_tracker_route_log_counts_proxy_inflight_requests():
+    entered = threading.Event()
+    release = threading.Event()
+
+    class SlowChatHandler(http.server.BaseHTTPRequestHandler):
+        def log_message(self, fmt, *args):
+            pass
+
+        def do_POST(self):
+            if self.path != "/v1/chat/completions":
+                self.send_response(404)
+                self.end_headers()
+                return
+            length = int(self.headers.get("Content-Length", 0))
+            self.rfile.read(length)
+            entered.set()
+            release.wait(timeout=3.0)
+            body = json.dumps({"choices": [{"message": {"content": "ok"}}]}).encode()
+            self.send_response(200)
+            self.send_header("Content-Type", "application/json")
+            self.send_header("Content-Length", str(len(body)))
+            self.end_headers()
+            self.wfile.write(body)
+
+    slow_node = http.server.HTTPServer(("127.0.0.1", 0), SlowChatHandler)
+    slow_thread = threading.Thread(target=slow_node.serve_forever, daemon=True)
+    slow_thread.start()
+    tracker = TrackerServer(heartbeat_timeout=60.0)
+    tracker_port = tracker.start()
+    errors = []
+    try:
+        _post_json(
+            f"http://127.0.0.1:{tracker_port}/v1/nodes/register",
+            {"endpoint": f"http://127.0.0.1:{slow_node.server_address[1]}",
+             "model": "burst-model", "num_layers": 1,
+             "shard_start": 0, "shard_end": 0,
+             "hardware_profile": {}, "score": 1.0},
+        )
+
+        def call_proxy():
+            try:
+                _post_json(
+                    f"http://127.0.0.1:{tracker_port}/v1/chat/completions",
+                    {"model": "burst-model", "messages": [{"role": "user", "content": "hi"}]},
+                )
+            except Exception as exc:
+                errors.append(exc)
+
+        first = threading.Thread(target=call_proxy)
+        second = threading.Thread(target=call_proxy)
+        first.start()
+        assert entered.wait(timeout=2.0)
+        second.start()
+
+        selected_events = []
+        deadline = time.time() + 2.0
+        while time.time() < deadline:
+            console = _get_json(f"http://127.0.0.1:{tracker_port}/v1/console")
+            selected_events = [
+                event for event in console["events"]
+                if event["message"] == "proxy route selected"
+            ]
+            if len(selected_events) >= 2:
+                break
+            time.sleep(0.05)
+
+        assert len(selected_events) >= 2
+        second_nodes = selected_events[-1]["fields"]["nodes"]
+        assert second_nodes[0]["queue_depth"] == 2
+        assert second_nodes[0]["proxy_inflight"] == 2
+    finally:
+        release.set()
+        first.join(timeout=3.0)
+        second.join(timeout=3.0)
+        tracker.stop()
+        slow_node.shutdown()
+        slow_node.server_close()
+        slow_thread.join(timeout=1.0)
+
+    assert not first.is_alive()
+    assert not second.is_alive()
+    assert not errors
+
+
+def test_tracker_logs_stream_progress_before_request_completes():
+    chunk_sent = threading.Event()
+    release = threading.Event()
+
+    class StreamingChatHandler(http.server.BaseHTTPRequestHandler):
+        def log_message(self, fmt, *args):
+            pass
+
+        def do_POST(self):
+            if self.path != "/v1/chat/completions":
+                self.send_response(404)
+                self.end_headers()
+                return
+            self.rfile.read(int(self.headers.get("Content-Length", 0)))
+            self.send_response(200)
+            self.send_header("Content-Type", "text/event-stream; charset=utf-8")
+            self.end_headers()
+            payload = json.dumps({
+                "choices": [{"delta": {"content": "hello world"}}],
+            }).encode()
+            self.wfile.write(b"data: " + payload + b"\n\n")
+            self.wfile.flush()
+            chunk_sent.set()
+            release.wait(timeout=3.0)
+            self.wfile.write(b"data: [DONE]\n\n")
+            self.wfile.flush()
+
+    node = http.server.HTTPServer(("127.0.0.1", 0), StreamingChatHandler)
+    node_thread = threading.Thread(target=node.serve_forever, daemon=True)
+    node_thread.start()
+    tracker = TrackerServer(heartbeat_timeout=60.0)
+    tracker_port = tracker.start()
+    response = None
+    try:
+        _post_json(
+            f"http://127.0.0.1:{tracker_port}/v1/nodes/register",
+            {"endpoint": f"http://127.0.0.1:{node.server_address[1]}",
+             "model": "stream-progress-model", "num_layers": 1,
+             "shard_start": 0, "shard_end": 0,
+             "hardware_profile": {}, "score": 1.0},
+        )
+        req = urllib.request.Request(
+            f"http://127.0.0.1:{tracker_port}/v1/chat/completions",
+            data=json.dumps({
+                "model": "stream-progress-model",
+                "stream": True,
+                "messages": [{"role": "user", "content": "hi"}],
+            }).encode(),
+            headers={"Content-Type": "application/json"},
+            method="POST",
+        )
+        response = urllib.request.urlopen(req, timeout=3.0)
+        first_line = response.readline()
+        assert first_line.startswith(b"data:")
+        assert chunk_sent.wait(timeout=1.0)
+
+        progress_events = []
+        deadline = time.time() + 2.0
+        while time.time() < deadline:
+            console = _get_json(f"http://127.0.0.1:{tracker_port}/v1/console")
+            progress_events = [
+                event for event in console["events"]
+                if event["message"] == "proxy progress"
+            ]
+            if progress_events:
+                break
+            time.sleep(0.05)
+
+        assert progress_events
+        fields = progress_events[-1]["fields"]
+        assert fields["tokens"] == 2
+        assert fields["tokens_per_sec"] > 0
+        active = [
+            event for event in console["events"]
+            if event["message"] == "proxy route selected"
+        ]
+        assert active
+    finally:
+        release.set()
+        if response is not None:
+            response.close()
+        tracker.stop()
+        node.shutdown()
+        node.server_close()
+        node_thread.join(timeout=1.0)
+
+
 def test_tracker_routes_hf_model_alias_from_quickstart():
     """The documented qwen2.5-0.5b alias resolves a full HF repo registration."""
     tracker = TrackerServer()
@@ -355,6 +585,229 @@ def test_tracker_proxy_accepts_hf_model_alias_from_quickstart():
     assert response["choices"][0]["message"]["content"] == "56"
 
 
+def test_tracker_proxy_routes_split_preset_nodes_by_alias():
+    """The chat proxy must build routes from alias-matched preset nodes."""
+
+    class ChatHandler(http.server.BaseHTTPRequestHandler):
+        def log_message(self, fmt, *args):
+            pass
+
+        def do_POST(self):
+            if self.path != "/v1/chat/completions":
+                self.send_response(404)
+                self.end_headers()
+                return
+            length = int(self.headers.get("Content-Length", 0))
+            self.rfile.read(length)
+            route_header = self.headers.get("X-Meshnet-Route", "[]")
+            body = json.dumps({
+                "choices": [{"message": {"content": route_header}}],
+                "usage": {"prompt_tokens": 1, "completion_tokens": 1},
+            }).encode()
+            self.send_response(200)
+            self.send_header("Content-Type", "application/json")
+            self.send_header("Content-Length", str(len(body)))
+            self.end_headers()
+            self.wfile.write(body)
+
+    head = http.server.HTTPServer(("127.0.0.1", 0), ChatHandler)
+    head_thread = threading.Thread(target=head.serve_forever, daemon=True)
+    head_thread.start()
+    tracker = TrackerServer(model_presets={
+        "qwen3.6-35b-a3b": {
+            "layers_start": 0,
+            "layers_end": 39,
+            "hf_repo": "unsloth/Qwen3.6-35B-A3B",
+            "aliases": ["Qwen3.6-35B-A3B"],
+        }
+    })
+    tracker_port = tracker.start()
+    try:
+        _post_json(
+            f"http://127.0.0.1:{tracker_port}/v1/nodes/register",
+            {"endpoint": f"http://127.0.0.1:{head.server_address[1]}",
+             "model": "qwen3.6-35b-a3b",
+             "hf_repo": "unsloth/Qwen3.6-35B-A3B",
+             "num_layers": 40,
+             "shard_start": 0,
+             "shard_end": 21,
+             "tracker_mode": True,
+             "hardware_profile": {},
+             "score": 1.0},
+        )
+        _post_json(
+            f"http://127.0.0.1:{tracker_port}/v1/nodes/register",
+            {"endpoint": "http://127.0.0.1:9199",
+             "model": "Qwen3.6-35B-A3B",
+             "hf_repo": "unsloth/Qwen3.6-35B-A3B",
+             "num_layers": 40,
+             "shard_start": 22,
+             "shard_end": 39,
+             "tracker_mode": True,
+             "hardware_profile": {},
+             "score": 1.0},
+        )
+
+        response = _post_json(
+            f"http://127.0.0.1:{tracker_port}/v1/chat/completions",
+            {"model": "Qwen3.6-35B-A3B",
+             "messages": [{"role": "user", "content": "hi"}]},
+        )
+        console = _get_json(f"http://127.0.0.1:{tracker_port}/v1/console")
+    finally:
+        tracker.stop()
+        head.shutdown()
+        head.server_close()
+        head_thread.join(timeout=1.0)
+
+    route = json.loads(response["choices"][0]["message"]["content"])
+    assert route == [{"endpoint": "http://127.0.0.1:9199", "start_layer": 22}]
+    complete = [event for event in console["events"] if event["message"] == "proxy complete"][-1]
+    assert complete["fields"]["tokens"] == 2
+    assert complete["fields"]["tokens_per_sec"] >= 0
+
+
+def test_tracker_route_endpoint_routes_split_preset_nodes_by_alias():
+    tracker = TrackerServer(model_presets={
+        "qwen3.6-35b-a3b": {
+            "layers_start": 0,
+            "layers_end": 39,
+            "hf_repo": "unsloth/Qwen3.6-35B-A3B",
+            "aliases": ["Qwen3.6-35B-A3B"],
+        }
+    })
+    tracker_port = tracker.start()
+    try:
+        _post_json(
+            f"http://127.0.0.1:{tracker_port}/v1/nodes/register",
+            {"endpoint": "http://127.0.0.1:9101",
+             "model": "qwen3.6-35b-a3b",
+             "hf_repo": "unsloth/Qwen3.6-35B-A3B",
+             "num_layers": 40,
+             "shard_start": 0,
+             "shard_end": 21,
+             "tracker_mode": True,
+             "hardware_profile": {},
+             "score": 1.0},
+        )
+        _post_json(
+            f"http://127.0.0.1:{tracker_port}/v1/nodes/register",
+            {"endpoint": "http://127.0.0.1:9102",
+             "model": "Qwen3.6-35B-A3B",
+             "hf_repo": "unsloth/Qwen3.6-35B-A3B",
+             "num_layers": 40,
+             "shard_start": 22,
+             "shard_end": 39,
+             "tracker_mode": True,
+             "hardware_profile": {},
+             "score": 1.0},
+        )
+
+        response = _get_json(
+            f"http://127.0.0.1:{tracker_port}/v1/route?model=Qwen3.6-35B-A3B"
+        )
+    finally:
+        tracker.stop()
+
+    assert response["route"] == ["http://127.0.0.1:9101", "http://127.0.0.1:9102"]
+    assert [node["start_layer"] for node in response["nodes"]] == [0, 22]
+
+
+def test_tracker_route_endpoint_ignores_model_case_and_outer_whitespace():
+    tracker = TrackerServer(model_presets={
+        "qwen3.6-35b-a3b": {
+            "layers_start": 0,
+            "layers_end": 39,
+            "hf_repo": "unsloth/Qwen3.6-35B-A3B",
+            "aliases": ["Qwen3.6-35B-A3B"],
+        }
+    })
+    tracker_port = tracker.start()
+    try:
+        _post_json(
+            f"http://127.0.0.1:{tracker_port}/v1/nodes/register",
+            {"endpoint": "http://127.0.0.1:9101",
+             "model": "qwen3.6-35b-a3b",
+             "hf_repo": "unsloth/Qwen3.6-35B-A3B",
+             "num_layers": 40,
+             "shard_start": 0,
+             "shard_end": 39,
+             "tracker_mode": True,
+             "hardware_profile": {},
+             "score": 1.0},
+        )
+
+        response = _get_json(
+            f"http://127.0.0.1:{tracker_port}/v1/route?model=%20Qwen3.6-35B-A3B%20"
+        )
+    finally:
+        tracker.stop()
+
+    assert response["route"] == ["http://127.0.0.1:9101"]
+
+
+def test_tracker_proxy_ignores_model_case_and_outer_whitespace():
+    class ChatHandler(http.server.BaseHTTPRequestHandler):
+        def log_message(self, fmt, *args):
+            pass
+
+        def do_POST(self):
+            if self.path != "/v1/chat/completions":
+                self.send_response(404)
+                self.end_headers()
+                return
+            length = int(self.headers.get("Content-Length", 0))
+            request_body = json.loads(self.rfile.read(length) or b"{}")
+            body = json.dumps({
+                "model": request_body["model"],
+                "choices": [{"message": {"content": "ok"}}],
+            }).encode()
+            self.send_response(200)
+            self.send_header("Content-Type", "application/json")
+            self.send_header("Content-Length", str(len(body)))
+            self.end_headers()
+            self.wfile.write(body)
+
+    node = http.server.HTTPServer(("127.0.0.1", 0), ChatHandler)
+    node_thread = threading.Thread(target=node.serve_forever, daemon=True)
+    node_thread.start()
+    tracker = TrackerServer(model_presets={
+        "qwen3.6-35b-a3b": {
+            "layers_start": 0,
+            "layers_end": 39,
+            "hf_repo": "unsloth/Qwen3.6-35B-A3B",
+            "aliases": ["Qwen3.6-35B-A3B"],
+        }
+    })
+    tracker_port = tracker.start()
+    try:
+        _post_json(
+            f"http://127.0.0.1:{tracker_port}/v1/nodes/register",
+            {"endpoint": f"http://127.0.0.1:{node.server_address[1]}",
+             "model": "qwen3.6-35b-a3b",
+             "hf_repo": "unsloth/Qwen3.6-35B-A3B",
+             "num_layers": 40,
+             "shard_start": 0,
+             "shard_end": 39,
+             "tracker_mode": True,
+             "hardware_profile": {},
+             "score": 1.0},
+        )
+
+        response = _post_json(
+            f"http://127.0.0.1:{tracker_port}/v1/chat/completions",
+            {"model": "Qwen3.6-35B-A3B ",
+             "messages": [{"role": "user", "content": "hi"}]},
+        )
+    finally:
+        tracker.stop()
+        node.shutdown()
+        node.server_close()
+        node_thread.join(timeout=1.0)
+
+    assert response["choices"][0]["message"]["content"] == "ok"
+
+
 def test_tracker_registration_node_id_includes_wallet_prefix_and_stable_suffix():
     tracker = TrackerServer()
     tracker_port = tracker.start()
@@ -667,12 +1120,14 @@ def test_tracker_speed_is_primary_when_both_nodes_can_cover_gap():
              "benchmark_tokens_per_sec": 3.0, "hardware_profile": {}, "score": 1.0},
         )
 
-        route_resp = _get_json(f"http://127.0.0.1:{tracker_port}/v1/route?model=tiny-model")
+        net = _get_json(f"http://127.0.0.1:{tracker_port}/v1/network/map")
         widths = {
             node["endpoint"]: node["shard_end"] - node["shard_start"] + 1
-            for node in route_resp["nodes"]
+            for node in net["nodes"]
         }
-        assert widths["http://127.0.0.1:9012"] > widths["http://127.0.0.1:9011"]
+        assert widths["http://127.0.0.1:9011"] == 12
+        assert widths["http://127.0.0.1:9012"] == 12
+        assert net["nodes"][0]["model_supply"]["served_model_copies"] == 2.0
     finally:
         tracker.stop()
 
@@ -700,7 +1155,8 @@ def test_tracker_registration_directive_is_not_replayed_on_heartbeat():
         tracker.stop()
 
 
-def test_tracker_reassignment_emits_drop_before_load():
+def test_tracker_pool_join_adds_redundant_copy_without_splitting_incumbent():
+    """A second managed node with capacity for the full model keeps the first copy intact."""
     tracker = TrackerServer(model_presets={
         "tiny-model": {
             "total_layers": 4,
@@ -709,21 +1165,69 @@ def test_tracker_reassignment_emits_drop_before_load():
     })
     tracker_port = tracker.start()
     try:
-        slow = _post_json(
+        first = _post_json(
             f"http://127.0.0.1:{tracker_port}/v1/nodes/register",
             {"endpoint": "http://127.0.0.1:9015", "model": "tiny-model",
              "vram_bytes": 10_000, "ram_bytes": 10_000, "quantizations": ["bfloat16"],
              "benchmark_tokens_per_sec": 1.0, "hardware_profile": {}, "score": 1.0},
         )
-        _post_json(
+        second = _post_json(
             f"http://127.0.0.1:{tracker_port}/v1/nodes/register",
             {"endpoint": "http://127.0.0.1:9016", "model": "tiny-model",
              "vram_bytes": 10_000, "ram_bytes": 10_000, "quantizations": ["bfloat16"],
              "benchmark_tokens_per_sec": 3.0, "hardware_profile": {}, "score": 1.0},
         )
 
-        hb = _post_json(f"http://127.0.0.1:{tracker_port}/v1/nodes/{slow['node_id']}/heartbeat", {})
-        assert [directive["action"] for directive in hb["directives"]] == ["DROP_SHARD", "LOAD_SHARD"]
+        net = _get_json(f"http://127.0.0.1:{tracker_port}/v1/network/map")
+        widths = {
+            node["endpoint"]: node["shard_end"] - node["shard_start"] + 1
+            for node in net["nodes"]
+        }
+        assert widths["http://127.0.0.1:9015"] == 4
+        assert widths["http://127.0.0.1:9016"] == 4
+        assert net["nodes"][0]["model_supply"]["served_model_copies"] == 2.0
+
+        hb = _post_json(f"http://127.0.0.1:{tracker_port}/v1/nodes/{first['node_id']}/heartbeat", {})
+        assert hb.get("directives", []) == []
+    finally:
+        tracker.stop()
+
+
+def test_tracker_explicit_full_copy_join_preserves_existing_serving_node():
+    """--model style joins with explicit shards add redundancy instead of reshuffling."""
+    tracker = TrackerServer(heartbeat_timeout=10.0)
+    tracker_port = tracker.start()
+    try:
+        base_reg = {
+            "model": "Qwen2.5-0.5B-Instruct",
+            "hf_repo": "Qwen/Qwen2.5-0.5B-Instruct",
+            "num_layers": 24,
+            "shard_start": 0,
+            "shard_end": 23,
+            "managed_assignment": True,
+            "vram_bytes": 2_000_000_000,
+            "hardware_profile": {},
+            "score": 1.0,
+        }
+        first = _post_json(
+            f"http://127.0.0.1:{tracker_port}/v1/nodes/register",
+            {**base_reg, "endpoint": "http://127.0.0.1:9201"},
+        )
+        second = _post_json(
+            f"http://127.0.0.1:{tracker_port}/v1/nodes/register",
+            {**base_reg, "endpoint": "http://127.0.0.1:9202"},
+        )
+
+        coverage = _get_json(
+            f"http://127.0.0.1:{tracker_port}/v1/network/map"
+        )
+        assert coverage["nodes"][0]["model_supply"]["served_model_copies"] == 2.0
+
+        hb = _post_json(
+            f"http://127.0.0.1:{tracker_port}/v1/nodes/{first['node_id']}/heartbeat", {}
+        )
+        assert hb.get("directives", []) == []
+        assert second["node_id"] in tracker._registry
     finally:
         tracker.stop()
 
@@ -761,22 +1265,20 @@ def test_tracker_faster_node_receives_wider_range_when_capacity_ties():
         _post_json(
             f"http://127.0.0.1:{tracker_port}/v1/nodes/register",
             {"endpoint": "http://127.0.0.1:9001", "model": "tiny-model",
-             "vram_bytes": 20_000, "ram_bytes": 10_000, "quantizations": ["bfloat16"],
+             "vram_bytes": 5_000, "ram_bytes": 10_000, "quantizations": ["bfloat16"],
              "benchmark_tokens_per_sec": 1.0, "hardware_profile": {}, "score": 1.0},
         )
         _post_json(
             f"http://127.0.0.1:{tracker_port}/v1/nodes/register",
             {"endpoint": "http://127.0.0.1:9002", "model": "tiny-model",
-             "vram_bytes": 20_000, "ram_bytes": 10_000, "quantizations": ["bfloat16"],
+             "vram_bytes": 5_000, "ram_bytes": 10_000, "quantizations": ["bfloat16"],
              "benchmark_tokens_per_sec": 2.0, "hardware_profile": {}, "score": 1.0},
         )
 
-        route_resp = _get_json(f"http://127.0.0.1:{tracker_port}/v1/route?model=tiny-model")
-        widths = {
-            node["endpoint"]: node["shard_end"] - node["shard_start"] + 1
-            for node in route_resp["nodes"]
-        }
-        assert widths["http://127.0.0.1:9002"] > widths["http://127.0.0.1:9001"]
+        net = _get_json(f"http://127.0.0.1:{tracker_port}/v1/network/map")
+        heads = [node for node in net["nodes"] if node["shard_start"] == 0]
+        assert len(heads) == 1
+        assert heads[0]["endpoint"] == "http://127.0.0.1:9002"
     finally:
         tracker.stop()
 
@@ -1844,6 +2346,7 @@ def test_torch_node_applies_tracker_load_shard_directive(monkeypatch):
 
     assert loaded == [("Qwen/Qwen2.5-0.5B-Instruct", 0, 23, "bfloat16")]
     assert applied == {
+        "action": "LOAD_SHARD",
         "model": "Qwen/Qwen2.5-0.5B-Instruct",
         "shard_start": 0,
         "shard_end": 23,
@@ -1960,3 +2463,105 @@ def test_shard_heal_cycle_surviving_node_covers_dead_peers_gap(monkeypatch):
         )
     finally:
         tracker.stop()
+
+
+def test_network_map_exposes_memory_pool():
+    tracker = TrackerServer(model_presets={
+        "tiny-model": {
+            "total_layers": 8,
+            "bytes_per_layer": {"bfloat16": 1_000},
+            "hf_repo": "org/TinyModel",
+        },
+    })
+    tracker_port = tracker.start()
+    try:
+        _post_json(
+            f"http://127.0.0.1:{tracker_port}/v1/nodes/register",
+            {"endpoint": "http://127.0.0.1:9050", "model": "tiny-model",
+             "hf_repo": "org/TinyModel", "num_layers": 8,
+             "shard_start": 0, "shard_end": 7, "max_loaded_shards": 2,
+             "vram_bytes": 20_000, "ram_bytes": 20_000, "quantizations": ["bfloat16"],
+             "benchmark_tokens_per_sec": 1.0, "hardware_profile": {}, "score": 1.0},
+        )
+        net = _get_json(f"http://127.0.0.1:{tracker_port}/v1/network/map")
+        pool = net["memory_pool"]
+        assert pool["total_spare_slots"] == 1
+        assert pool["hosts"][0]["loaded_slots"] == 1
+        assert pool["hosts"][0]["max_loaded_shards"] == 2
+        assert pool["hosts"][0]["memory_spare_bytes"] > 0
+    finally:
+        tracker.stop()
+
+
+def test_same_endpoint_can_register_multiple_models():
+    tracker = TrackerServer()
+    tracker_port = tracker.start()
+    try:
+        base = {
+            "endpoint": "http://127.0.0.1:9055",
+            "num_layers": 24,
+            "shard_start": 0,
+            "shard_end": 23,
+            "hardware_profile": {},
+            "score": 1.0,
+            "max_loaded_shards": 2,
+            "vram_bytes": 50_000_000,
+            "ram_bytes": 50_000_000,
+        }
+        first = _post_json(
+            f"http://127.0.0.1:{tracker_port}/v1/nodes/register",
+            {**base, "model": "Qwen2.5-0.5B-Instruct", "hf_repo": "Qwen/Qwen2.5-0.5B-Instruct"},
+        )
+        second = _post_json(
+            f"http://127.0.0.1:{tracker_port}/v1/nodes/register",
+            {**base, "model": "OtherModel", "hf_repo": "org/OtherModel"},
+        )
+        assert first["node_id"] != second["node_id"]
+        assert len(tracker._registry) == 2
+    finally:
+        tracker.stop()
+
+
+def test_scale_demanded_models_queues_add_shard_on_spare_host():
+    tracker = TrackerServer(model_presets={
+        "model-a": {
+            "total_layers": 4,
+            "bytes_per_layer": {"bfloat16": 1_000},
+            "hf_repo": "org/ModelA",
+        },
+        "model-b": {
+            "total_layers": 4,
+            "bytes_per_layer": {"bfloat16": 1_000},
+            "hf_repo": "org/ModelB",
+        },
+    })
+    tracker_port = tracker.start()
+    try:
+        reg_b = _post_json(
+            f"http://127.0.0.1:{tracker_port}/v1/nodes/register",
+            {"endpoint": "http://127.0.0.1:9061", "model": "model-b",
+             "hf_repo": "org/ModelB", "num_layers": 4,
+             "shard_start": 0, "shard_end": 3, "max_loaded_shards": 2,
+             "vram_bytes": 20_000, "ram_bytes": 20_000, "quantizations": ["bfloat16"],
+             "benchmark_tokens_per_sec": 1.0, "hardware_profile": {}, "score": 1.0},
+        )
+        _post_json(
+            f"http://127.0.0.1:{tracker_port}/v1/nodes/register",
+            {"endpoint": "http://127.0.0.1:9060", "model": "model-a",
+             "hf_repo": "org/ModelA", "num_layers": 4,
+             "shard_start": 0, "shard_end": 3, "max_loaded_shards": 1,
+             "vram_bytes": 20_000, "ram_bytes": 20_000, "quantizations": ["bfloat16"],
+             "benchmark_tokens_per_sec": 1.0, "hardware_profile": {}, "score": 1.0},
+        )
+        assert tracker._stats is not None
+        for _ in range(400):
+            tracker._stats.record_request("org/ModelA")
+        with tracker._lock:
+            _scale_demanded_models_locked(tracker._server)  # type: ignore[arg-type]
+        node_b = tracker._registry[reg_b["node_id"]]
+        assignment = node_b.pending_new_assignment
+        assert assignment is not None
+        assert assignment["action"] == "ADD_SHARD"
+        assert assignment["model"] == "org/ModelA"
+    finally:
+        tracker.stop()