Merge branch 'master' of https://git.d-popov.com/popov/neuron-tai

.scratch/distributed-gguf-runtime/ADR-0020-distributed-gguf-runtime.md

@@ -0,0 +1,177 @@
+# ADR-0020: Distributed GGUF/llama.cpp Runtime With Per-Shard Local KV
+
+Status: Proposed
+
+## Context
+
+The project currently uses PyTorch/Transformers for real model shards. That decision was captured in ADR-0001 because llama.cpp RPC at the time required the primary node to load the full model and distribute weights to workers, which conflicted with the desired model where nodes independently hold shards.
+
+We now want to serve very large open models, including GLM-5.2 and Ornith-class MoE models, over a torrent-like inference marketplace. CPU and mixed consumer hardware matter. LM Studio and llama.cpp demonstrate much better CPU/GGUF performance than our current PyTorch CPU path. The user also has a personal relationship with Georgi Gerganov, making upstream collaboration plausible.
+
+The current distributed PyTorch path is not yet production-grade: it recomputes the full growing sequence for every output token and disables KV cache inside manual layer calls. It sends hidden activations across seams, not KV, but those activations currently cover the full sequence every decode step.
+
+## Decision
+
+Adopt a distributed GGUF/llama.cpp runtime track while keeping PyTorch as the reference and fast-architecture backend.
+
+The runtime model is:
+
+- GGUF/model artifacts are distributed through torrent/content-addressed storage.
+- Nodes independently acquire and verify artifacts; no root node streams model weights to workers at session start.
+- Tracker chooses a sticky route covering all layers.
+- Each node owns hot KV/state for the layers it executes.
+- Prefill sends chunked activations through the route and builds local per-shard KV.
+- Decode sends one-step activations through the route and appends local KV at every shard.
+- Cache/CDN servers store cold artifacts and optional prefix/session snapshots, not hot per-token KV.
+- Context is capped at 128K for the first serious product path.
+
+## Technical Framework
+
+The design separates five planes:
+
+- **Control plane**: tracker registry, coverage map, route selection, session lifecycle, telemetry, billing, and audit.
+- **Artifact plane**: Shard Swarms, GGUF/safetensors/tokenizer files, manifests, hashes, and local node storage.
+- **Execution plane**: active Inference Route, chunked prefill, one-step decode, and hidden-state movement across activation seams.
+- **Session state plane**: per-shard Hot KV State on route nodes, plus optional Prefix Snapshots outside the hot loop.
+- **Economics/trust plane**: reward accounting, validation events, slash proofs, public/private route policy.
+
+Hard invariants:
+
+1. Public-network Shards are contiguous layer ranges.
+2. Hot KV State is local to the node serving that Shard in that Route Session.
+3. Artifact distribution and route execution are separate systems.
+4. Decode seam payload must be `O(hidden_size)`.
+5. Prefill may be `O(sequence_length * hidden_size)`, but only in bounded chunks.
+6. The tracker chooses routes; nodes do not negotiate route topology peer-to-peer.
+7. Model/backend-specific cache internals stay behind backend capability reports.
+8. PyTorch remains the correctness/reference backend while llama.cpp/GGUF becomes the performance backend.
+9. Streaming responses are preferred when feasible; Generation Telemetry is always required.
+
+The full challenge register is in [technical-challenges.md](./technical-challenges.md). The open decision gates are in [decision-framework.md](./decision-framework.md).
+
+Resolved gate:
+
+- Public-network Shards are layer ranges. Tensor-parallel/ring execution belongs inside a trusted node, colocated pod, or future composite node abstraction, not as the v1 public routing primitive.
+- Hot KV State is local to each route node for the Shard it serves. Cache servers may store Prefix Snapshots, but they are not part of the per-token decode path.
+- Distributed Route Session and Hot KV State semantics will be proven in the PyTorch route before llama.cpp/GGUF is extended for layer-boundary execution.
+- Streaming responses are preferred when feasible. Realtime Generation Telemetry is required so clients can see phase, generated token count, and tokens/sec even during prefill or non-streaming fallback paths.
+- llama.cpp/GGUF work targets upstreamable `libllama`/ggml hooks. A prototype fork is acceptable for exploration, but a permanent fork is not the plan.
+- Model targeting is two-tiered: use a small llama.cpp-supported GGUF model for the first protocol smoke test, then use `deepseek-ai/DeepSeek-V4-Flash` as the first serious large-model target. GLM-5.2 and Ornith remain later support audits.
+- Alpha fails Route Sessions on route-node loss instead of attempting automatic route repair. Repair requires compatible Prefix Snapshots and is a later capability.
+- v1 activation transfer stays on binary HTTP as defined by ADR-0008. QUIC/WebRTC/custom transport can be introduced later behind the same activation protocol.
+
+## Non-Goals
+
+- Do not put remote cache servers in the per-token hot KV path.
+- Do not require every node to hold the full model.
+- Do not fork llama.cpp long-term if upstream APIs can support the needed layer-boundary hooks.
+- Do not target GLM-5.2 or Ornith first; prove the route/KV protocol on a simpler well-supported GGUF model, then target DeepSeek-V4-Flash as the first serious large model.
+
+## Options Considered
+
+### A. Keep PyTorch-only distributed inference
+
+Pros:
+
+- Easy access to new Hugging Face architectures.
+- Transformers has mature single-process KV semantics.
+- Existing code already loads shards.
+
+Cons:
+
+- CPU inference is much slower than llama.cpp/GGUF.
+- Current distributed path bypasses `generate()` and disables cache.
+- Quantized GGUF ecosystem and LM Studio users are outside the runtime.
+
+### B. Use llama.cpp only as a full local model backend
+
+Pros:
+
+- Quick performance win for nodes with enough RAM/VRAM.
+- Minimal coordination with distributed protocol.
+
+Cons:
+
+- Does not unlock 397B/753B-class models for ordinary nodes.
+- Does not solve marketplace layer routing.
+
+### C. Distributed GGUF with per-shard local KV (chosen)
+
+Pros:
+
+- Aligns with torrent artifact distribution.
+- Avoids root streaming weights to workers.
+- Uses llama.cpp/GGUF performance where supported.
+- Compatible with public node rewards by layer/work contribution.
+- Scales KV memory by layer range.
+
+Cons:
+
+- Requires new runtime APIs around layer-boundary hidden states and per-session KV.
+- Requires model-specific cache metadata for DSA/MLA/hybrid attention.
+- Harder to debug than single-process `generate()`.
+
+### D. Centralized KV cache servers
+
+Pros:
+
+- Easier apparent session failover.
+- Central accounting of active cache.
+
+Cons:
+
+- Puts remote storage in the per-token hot path.
+- Adds bandwidth and latency at the worst possible point.
+- Creates consistency and privacy problems.
+
+Rejected for hot decode. Accepted only for cold prefix snapshots and failover checkpoints.
+
+## Consequences
+
+- ADR-0001 should eventually be amended: PyTorch remains valid, but llama.cpp/GGUF becomes a first-class backend.
+- The activation protocol must split prefill and decode explicitly.
+- Session IDs must be stable across the full request. The current fresh UUID-per-hop-call behavior must change.
+- Backends must report cache budget and cache compatibility.
+- Tracker route selection must include disk, memory pressure, cache warmth, and network latency.
+- Billing can be based on layer work, prefill tokens, decode tokens, and observed route participation.
+- Client UX should stream token deltas when feasible and must include route-session progress telemetry even when token deltas are not streamed.
+
+## Required Runtime Capabilities
+
+PyTorch path:
+
+- manual layer calls with `past_key_values` / model-specific cache object
+- per-shard session cache store
+- prefill chunk append
+- decode step append
+- stable session lifecycle endpoints
+
+llama.cpp/GGUF path:
+
+- full local GGUF serving
+- layer/tensor map extraction from GGUF
+- optional partial layer loading or mmap-backed selected execution
+- inbound hidden-state execution from arbitrary start layer
+- outbound hidden-state return at stop layer
+- per-session KV ownership for loaded layers
+- cache budget/compatibility introspection
+- GLM-5.2 DSA support when upstream/runtime supports it
+
+## Implementation Plan
+
+1. Add full-model `LlamaCppBackend` using `llama-server` or `libllama`.
+2. Implement distributed KV in the PyTorch path to prove semantics.
+3. Add session lifecycle and prefill/decode wire protocol.
+4. Add model artifact manifest and torrent seeding metadata.
+5. Prototype localhost two-process llama.cpp layer boundary execution.
+6. Generalize to network route.
+7. Bring in GLM-5.2/Ornith once backend support and cache accounting are verified.
+
+## Acceptance Criteria
+
+- A two-node localhost route can prefill once and decode N tokens without recomputing the full prompt.
+- Seam payload during decode is `O(hidden_size)`, not `O(sequence_length * hidden_size)`.
+- Per-node KV memory grows with owned layer count and context length.
+- Route loss during alpha fails cleanly with explicit reason.
+- Full local GGUF backend outperforms PyTorch CPU on a supported model.
+- Artifact manifest can identify exactly which files/chunks a node must seed for its advertised layer range.

.scratch/distributed-gguf-runtime/README.md

@@ -0,0 +1,58 @@
+# Distributed GGUF runtime — planning index
+
+Status: draft scratch package.
+
+Goal: make the node network capable of serving large, high-quality open models by distributing GGUF/model artifacts over a torrent-style swarm while executing inference over a sticky multi-node route with per-shard local KV cache.
+
+This scratch supersedes the old assumption in [ADR-0001](../../docs/adr/0001-pytorch-over-llama-cpp.md) that llama.cpp is only a single-node leaf backend. That assumption was correct for the original llama.cpp RPC shape, but the target is now different: torrent-distributed GGUF artifacts plus an explicit route/KV protocol owned by this platform, ideally developed in collaboration with upstream llama.cpp.
+
+## Artifacts
+
+| Path | Purpose |
+|---|---|
+| [architecture.md](./architecture.md) | Proposed runtime architecture, data flow, session state, and failure model |
+| [technical-challenges.md](./technical-challenges.md) | Detailed challenge/solution register with acceptance tests |
+| [decision-framework.md](./decision-framework.md) | Grilling framework for open decisions and recommended answers |
+| [research-prior-art.md](./research-prior-art.md) | Prior-art notes for Petals, exo, Distributed Llama, prima.cpp, llama.cpp, DeepSeek-V4-Flash, GLM-5.2, and Ornith |
+| [ADR-0020-distributed-gguf-runtime.md](./ADR-0020-distributed-gguf-runtime.md) | Draft decision record for the GGUF/llama.cpp distributed runtime |
+| [issues/](./issues/) | Implementation slices in dependency order |
+
+## Decision Summary
+
+Adopt a hybrid runtime:
+
+- **Weights and artifacts**: distributed by torrent / content-addressed storage / optional CDN.
+- **Hot KV cache**: local to the node that owns the corresponding layer range.
+- **Prefix snapshots**: optionally persisted to cache servers for reuse, retry, and failover.
+- **Active route**: sticky for one request/session.
+- **Context cap**: 128K hard product limit for large models unless explicitly revised.
+- **Backends**: keep PyTorch for fast model-architecture coverage and validation; add llama.cpp/GGUF as the performance path for supported models.
+- **Client feedback**: stream token deltas when feasible; always expose Generation Telemetry.
+- **First serious target model**: DeepSeek-V4-Flash after a smaller GGUF protocol smoke test.
+
+## What We Learned
+
+- Our current full-model PyTorch path uses Transformers `generate()` and gets local KV cache.
+- Our current distributed PyTorch path disables cache and recomputes the full growing sequence per token.
+- The seam today carries hidden activations, not KV cache; at 128K this becomes impossible for serious models if repeated every decode token.
+- The missing capability is not "send KV across the network"; it is **stable per-session local KV cache per shard**.
+- GGUF distribution is solved enough at the artifact layer, but GGUF/llama.cpp needs explicit layer-boundary execution APIs for our route model.
+
+## Recommended Order
+
+1. Local llama.cpp/GGUF backend for full-model serving.
+2. Stable distributed session ID and per-shard KV cache in the existing PyTorch path.
+3. Binary prefill/decode protocol split: chunked prefill, one-step decode.
+4. Route-session Generation Telemetry and streaming response support where feasible.
+5. GGUF artifact manifest and torrent seeding.
+6. llama.cpp layer-boundary prototype on localhost.
+7. Networked distributed GGUF route.
+8. DeepSeek-V4-Flash as first serious large-model target.
+9. GLM-5.2 / DSA / MLA and Ornith support once runtime support is confirmed.
+
+## Open Questions
+
+- Does upstream llama.cpp already expose enough internal API for arbitrary layer-range execution and hidden-state boundary I/O, or do we need an extension?
+- Can GGUF split metadata be made layer/tensor semantic enough for torrent placement and partial loading?
+- What is the minimum protocol needed for compressed KV formats such as GLM-5.2 DSA/MLA without exposing model-specific internals to the tracker?
+- How much reliability do we need in alpha: fail request on route loss, or support route repair with KV snapshots?

.scratch/distributed-gguf-runtime/architecture.md

@@ -0,0 +1,274 @@
+# Distributed GGUF Runtime Architecture
+
+## Product Stance
+
+The platform optimizes for access to high-quality models, not lowest latency. Latency is acceptable if the user can run models that are otherwise unavailable to them. The hard context limit for the first serious distributed runtime should be **128K tokens**. Longer context usually means the product is compensating for missing task decomposition, retrieval, or workspace summarization.
+
+## Current State
+
+The current node has two materially different inference paths:
+
+- **Full local PyTorch model**: calls Hugging Face `model.generate()`, so Transformers owns autoregressive decode and local KV cache.
+- **Distributed PyTorch route**: bypasses `model.generate()`, calls individual layers with `use_cache=False`, and recomputes the full growing sequence for every generated token.
+
+Current distributed data flow:
+
+```text
+client request
+  -> head node formats prompt
+  -> for each output token:
+       head tokenizes full current text
+       head runs early layers over all tokens
+       head sends full activation [batch, sequence, hidden] to next node
+       middle nodes run their layers over all tokens
+       tail returns one decoded token string
+       head appends token to text
+```
+
+This is correct for small demos but not viable for large models. For GLM-5.2, a single 128K seam activation is roughly:
+
+```text
+128K tokens * hidden_size 6144 * 2 bytes ~= 1.5 GiB per hop
+```
+
+Sending that every output token is the bottleneck.
+
+## Target State
+
+Target distributed data flow:
+
+```text
+client request
+  -> tracker selects route and pins session
+  -> head node creates session_id
+  -> prefill:
+       prompt is chunked
+       each shard computes its layer range
+       each shard appends local KV/state for its own layers
+       activations cross only layer seams
+  -> decode loop:
+       head sends one new token / one-step hidden state
+       each shard reads local KV/state for session_id
+       each shard appends one step to local KV/state
+       only one-step activation crosses seams
+       tail returns logits/token
+```
+
+The KV cache remains local to the node that computed it. It is not sent to the next node and not read from a remote cache server during every decode step.
+
+## Client Feedback
+
+Streaming responses are desirable when the backend and client transport support them. The product should stream token deltas when possible, and it must always provide realtime Generation Telemetry while the route is working.
+
+The fallback behavior is a non-streaming final answer plus live telemetry. That fallback is acceptable for early route proofs or models/backends that cannot expose clean token deltas yet, but the preferred client experience is streamed output plus telemetry.
+
+Minimum client-visible telemetry:
+
+- route/session accepted
+- selected model and quantization
+- prefill phase started/completed
+- decode phase started
+- generated token count
+- rolling tokens per second
+- route health or retry/failure reason
+- estimated billing units when available
+
+Implementation options:
+
+- Server-Sent Events or WebSocket for realtime progress
+- polling endpoint for simple clients
+- OpenAI-compatible streaming for clients that require token deltas
+
+This means "no token streaming" is acceptable only as a fallback. "Silent wait for minutes" is not acceptable.
+
+## Artifact Plane
+
+Artifact distribution is separate from execution.
+
+```text
+model publisher
+  -> produces model manifest
+  -> creates GGUF / safetensors / tokenizer artifacts
+  -> content-addresses every file/chunk
+  -> publishes torrent/magnet + HTTP fallback metadata
+
+node
+  -> chooses model/layer range
+  -> downloads needed files/chunks
+  -> verifies hash
+  -> advertises availability to tracker
+```
+
+Required manifest fields:
+
+- model id and version
+- upstream source repo and revision
+- license
+- architecture name
+- tokenizer files and hashes
+- quantization
+- tensor-to-layer map
+- file/chunk hashes
+- optional GGUF split files
+- supported runtime backends
+- context cap
+- KV/cache format descriptor
+
+## Execution Plane
+
+The tracker selects routes using layer coverage and observed performance:
+
+```text
+route = [
+  head node: embeddings + layers 0..k
+  middle nodes: contiguous layer ranges
+  tail node: final layers + norm + lm_head
+]
+```
+
+Route selection inputs:
+
+- model id/version/quantization
+- layer coverage
+- node hardware
+- measured prefill throughput
+- measured decode throughput
+- queue depth
+- latency to neighboring nodes
+- cache warmth for the requested prefix/session
+- reliability/reputation
+
+The route is sticky for the request/session. A new route means either a fresh prefill or restoring compatible KV snapshots.
+
+## KV Cache Ownership
+
+KV/state ownership is by layer range:
+
+```text
+session_id = request scoped id
+node A owns layers 0..15 KV for session_id
+node B owns layers 16..31 KV for session_id
+node C owns layers 32..77 KV for session_id
+```
+
+The tracker does not own hot KV. It may know which nodes hold active KV for session accounting and failure handling.
+
+Cache servers may store:
+
+- prompt-prefix snapshots
+- session checkpoints for retry
+- cold reusable context blocks
+- audit samples
+
+Cache servers must not be in the per-token hot loop unless colocated with the compute node.
+
+## 128K KV Budget
+
+GLM-5.2 compressed DSA/MLA-style estimate from config:
+
+```text
+layers = 78
+kv_lora_rank = 512
+qk_rope_head_dim = 64
+dtype = bf16 = 2 bytes
+context = 128K
+
+per_token ~= 78 * (512 + 64) * 2 = 89,856 bytes ~= 87.75 KiB
+128K total ~= 10.7 GiB
+per layer ~= 137 MiB
+```
+
+This is feasible when sharded:
+
+| Layer count | Approx active KV at 128K |
+|---:|---:|
+| 1 | 137 MiB |
+| 10 | 1.37 GiB |
+| 20 | 2.75 GiB |
+| 78 | 10.7 GiB |
+
+The exact runtime value depends on implementation and cache quantization, but the order of magnitude is acceptable.
+
+## Protocol Sketch
+
+### Prefill
+
+```http
+POST /v1/sessions/{session_id}/prefill
+Content-Type: application/octet-stream
+X-Meshnet-Model: zai-org/GLM-5.2
+X-Meshnet-Route-Id: ...
+X-Meshnet-Token-Range: 0-2047
+X-Meshnet-Shape: 1,2048,6144
+X-Meshnet-Dtype: bfloat16
+
+<activation bytes>
+```
+
+The receiver:
+
+- validates route/session
+- runs assigned layer range for that chunk
+- appends local KV/state
+- forwards resulting activation to next hop
+
+### Decode
+
+```http
+POST /v1/sessions/{session_id}/decode-step
+Content-Type: application/octet-stream
+X-Meshnet-Model: zai-org/GLM-5.2
+X-Meshnet-Position: 131072
+X-Meshnet-Shape: 1,1,6144
+X-Meshnet-Dtype: bfloat16
+
+<one-step activation bytes>
+```
+
+The receiver:
+
+- loads local KV/state by `session_id`
+- runs one decode step for assigned layers
+- appends one token position to local KV/state
+- forwards one-step activation
+
+## GGUF / llama.cpp Integration
+
+The target llama.cpp integration needs more than `llama-server`.
+
+Required capabilities:
+
+- load full GGUF locally for immediate single-node performance
+- optionally load only selected tensors/layers
+- execute a layer range against inbound hidden states
+- expose outbound hidden states at a boundary
+- own per-session KV/state for only the loaded layer range
+- support prefill chunks and decode-step calls
+- expose model-specific cache metadata for DSA/MLA without requiring the tracker to understand tensor internals
+
+If llama.cpp cannot expose these as stable APIs today, the collaboration target is an upstream extension rather than a long-lived fork.
+
+## Failure Model
+
+Alpha behavior:
+
+- Route node drops during prefill: fail request and retry from scratch.
+- Route node drops during decode: fail request unless a recent KV snapshot exists.
+- Tracker restart: active sessions may be lost; completed billing records persist.
+- Node restart: local hot KV is lost.
+
+Later behavior:
+
+- periodic KV snapshots for long sessions
+- prefix cache reuse across requests
+- route repair when a semantically equivalent node has the same model/layer range and compatible cache snapshot
+
+## Security And Trust
+
+Activation/KV data can reveal user prompts. Public volunteer routes are not private. For sensitive workloads:
+
+- use private swarms
+- allow paid trusted nodes
+- encrypt transport
+- avoid storing hot KV on untrusted shared cache servers
+- sample outputs for fraud/audit as already planned in alpha hardening

.scratch/distributed-gguf-runtime/decision-framework.md

@@ -0,0 +1,268 @@
+# Distributed GGUF Decision Framework
+
+This framework is for grilling open decisions. It keeps decisions tied to project vocabulary and implementation gates instead of vague "distributed inference" language.
+
+## Core Vocabulary
+
+Use the existing domain terms this way:
+
+- **Shard**: contiguous transformer layer range. This is the compute, routing, cache, and reward unit.
+- **Shard Swarm**: storage/download group for artifacts needed by a shard.
+- **Inference Route**: ordered node sequence that covers all layers for one request.
+- **Route Session**: one active request bound to one inference route and stable session id.
+- **Hot KV State**: live per-shard cache held by the route node during a route session.
+- **Prefix Snapshot**: persisted route-session state used for reuse or failover, not the hot decode path.
+- **Artifact Manifest**: canonical mapping from model artifacts to semantic model parts and runtime support.
+- **Generation Telemetry**: realtime progress for a route session, including phase and tokens/sec, independent of whether token deltas are streamed.
+
+## The Five Planes
+
+### 1. Control Plane
+
+Owner: Tracker.
+
+Responsibilities:
+
+- node registry
+- coverage map
+- route selection
+- rebalance directives
+- route-session creation
+- health and telemetry
+- client-visible Generation Telemetry
+- billing/audit records
+
+Must not do:
+
+- serve hot KV during every token
+- become the only place model artifacts can be fetched
+
+### 2. Artifact Plane
+
+Owner: Shard Swarms, local node storage, optional CDN/bootstrap mirrors.
+
+Responsibilities:
+
+- GGUF/safetensors/tokenizer download
+- content-addressed verification
+- local artifact inventory
+- artifact-to-layer mapping
+- cache eviction
+
+Must not do:
+
+- define execution order by file split alone
+- imply that a downloaded file chunk equals a Shard
+
+### 3. Execution Plane
+
+Owner: active Inference Route.
+
+Responsibilities:
+
+- chunked prefill
+- one-step decode
+- hidden-state transfer across activation seams
+- start-layer handling for overlapping shards
+- backpressure
+
+Must not do:
+
+- resend full context activations during decode
+- require cross-node tensor parallel all-reduce for public v1
+
+### 4. Session State Plane
+
+Owner: route nodes for hot KV; cache servers only for snapshots.
+
+Responsibilities:
+
+- per-shard local KV ownership
+- cache allocation and eviction
+- cache ABI compatibility
+- session close/release
+- optional prefix snapshots
+
+Must not do:
+
+- centralize hot KV in a remote service
+- let a replacement node continue from incompatible state
+
+### 5. Economics And Trust Plane
+
+Owner: tracker plus settlement/validation components.
+
+Responsibilities:
+
+- distinguish storage/seeding work from inference work
+- account for prefill and decode separately
+- record route participation
+- sample validation events
+- slash proven fraud
+
+Must not do:
+
+- pay a node for merely holding files as if it generated tokens
+- hide public-swarm privacy limits from clients
+
+## Hard Invariants
+
+These are the framework rules unless we deliberately write a new ADR:
+
+1. Public-network Shards are contiguous layer ranges.
+2. Hot KV State is local to the node serving that Shard in that Route Session.
+3. Artifact distribution and route execution are separate systems.
+4. Decode seam payload must be `O(hidden_size)`.
+5. Prefill may be `O(sequence_length * hidden_size)`, but only in bounded chunks.
+6. The tracker chooses routes; nodes do not negotiate route topology peer-to-peer.
+7. Model/backend-specific cache internals stay behind backend capability reports.
+8. PyTorch remains the correctness/reference backend while llama.cpp/GGUF becomes the performance backend.
+9. Streaming responses are preferred when feasible; Generation Telemetry is always required.
+
+## Resolved Gates
+
+### Gate 1: Public Shard Semantics
+
+Decision: public-network Shards are contiguous transformer layer ranges. Tensor-parallel or ring-style execution is allowed only inside one trusted node, one colocated pod, or a future composite node abstraction.
+
+Rationale:
+
+- Layer ranges match the existing `Shard`, `Coverage Map`, `Inference Route`, billing, and fraud vocabulary.
+- Public volunteer nodes should not require cross-node all-reduce or tight per-layer synchronization in v1.
+- Existing projects such as prima.cpp and Distributed Llama can still inform local-cluster/backend execution without becoming the public routing primitive.
+
+Consequences:
+
+- Artifact Manifests must map files/tensors to semantic layer ranges.
+- Route selection remains ordered layer coverage.
+- Rewards can be attributed to layer-range work.
+- Hot KV State is naturally owned by the node serving that layer range for the Route Session.
+
+### Gate 2: Hot KV Strategy
+
+Decision: v1 rejects centralized hot KV. Hot KV State is local to the node serving the relevant Shard in the active Route Session. Cache servers may store Prefix Snapshots for reuse, retry, or failover, but they are not in the per-token decode path.
+
+Rationale:
+
+- Decode is the tight loop; adding remote cache I/O there makes latency and bandwidth worse at the worst point.
+- Local KV naturally follows layer-range Shard ownership.
+- Centralized hot KV increases privacy exposure and creates consistency problems.
+- Prefix Snapshots preserve the useful part of central storage without making it mandatory for every generated token.
+
+Consequences:
+
+- Route Session must be sticky.
+- Failover is limited in alpha unless a compatible Prefix Snapshot exists.
+- Cache servers are optimization infrastructure, not required runtime infrastructure.
+- Route repair requires compatible model revision, layer range, backend cache ABI, and snapshot position.
+
+### Gate 3: First Runtime Proof
+
+Decision: prove distributed Route Session and Hot KV State semantics in the existing PyTorch route before modifying llama.cpp/GGUF.
+
+Rationale:
+
+- PyTorch exposes model internals and cache objects more directly, so it is the fastest way to validate the distributed protocol.
+- The current distributed PyTorch route already has the right high-level shape but disables cache and recomputes full prompts.
+- Fixing that path gives us a reference implementation for correctness tests, telemetry, session lifecycle, and wire protocol behavior.
+- llama.cpp/GGUF should receive a clear target ABI rather than becoming both the protocol experiment and the performance backend at once.
+
+Consequences:
+
+- Issue 02 precedes issue 05.
+- llama.cpp collaboration has a concrete target ABI.
+- The PyTorch route remains the architecture-coverage/reference backend even after GGUF becomes the preferred performance path.
+- The first success metric is eliminating full-prompt recompute in distributed decode.
+
+### Gate 3A: Client Feedback During Latency
+
+Decision: streaming responses are preferred when feasible, and realtime Generation Telemetry is required regardless of streaming support.
+
+Rationale:
+
+- The product optimizes for access to large capable models, so some latency is acceptable.
+- Users still need confidence that the route is alive and roughly how fast it is generating.
+- Streaming token deltas give the best user experience when the backend exposes them cleanly.
+- Tokens/sec remains useful during prefill, queueing, and any backend that cannot stream token deltas.
+
+Consequences:
+
+- The gateway should stream token deltas through an OpenAI-compatible response when possible.
+- The gateway must expose progress through SSE, WebSocket, or polling.
+- The final answer can be delivered after completion only as a fallback.
+- Telemetry must include route phase, generated token count, and rolling tokens/sec.
+- Non-streaming clients still need realtime telemetry.
+
+### Gate 4: llama.cpp Collaboration Shape
+
+Decision: target upstreamable `libllama`/ggml hooks instead of planning around a permanent fork.
+
+Rationale:
+
+- llama.cpp changes quickly across model support, quantization, kernels, and hardware backends.
+- A permanent fork would become expensive to maintain and would lag upstream improvements.
+- A short-lived prototype branch is acceptable if it proves the API and makes upstream collaboration concrete.
+- Keeping tracker/routing logic outside llama.cpp makes the upstream ask smaller and cleaner.
+
+Consequences:
+
+- Need a minimal reproducible localhost demo before asking upstream to carry the design.
+- Need to separate "what llama.cpp should expose" from "what our tracker does".
+- Desired upstream surface is layer-range execution, hidden-state boundary I/O, partial loading/introspection, and per-session KV ownership.
+- If upstream rejects the shape, we revisit whether to carry a narrow adapter fork or keep GGUF distributed execution as experimental.
+
+### Gate 5: First Model Target
+
+Decision: use a two-tier model target. Use a small, boring, llama.cpp-supported GGUF model for the first protocol smoke test. Use `deepseek-ai/DeepSeek-V4-Flash` as the first serious large-model target. Keep GLM-5.2 and Ornith as later support audits.
+
+Rationale:
+
+- The first protocol proof should isolate route/session/KV bugs from model-architecture bugs.
+- DeepSeek-V4-Flash is a strong first serious target because it is much smaller than 1.6T-class models while still being large enough to validate the product thesis.
+- DeepSeek-V4-Flash still has architecture-specific risks, so it should not be the first smoke test.
+- GLM-5.2 and Ornith remain valuable targets, but they add DSA/MLA/hybrid attention uncertainty.
+
+Consequences:
+
+- 128K cache accounting can be modeled now.
+- The first "real" target-model audit is DeepSeek-V4-Flash support in PyTorch, vLLM/SGLang, and any available GGUF/llama.cpp quantization path.
+- Production support waits for backend capability reports and exact cache ABI support.
+
+### Gate 6: Failure Semantics
+
+Decision: alpha fails Route Sessions on route-node loss instead of attempting automatic route repair.
+
+Rationale:
+
+- Route repair requires compatible Prefix Snapshots, cache ABI checks, replacement-node selection, billing correction, and client stream/error recovery.
+- Local Hot KV State means a replacement node cannot continue unless it has compatible state at the same position.
+- Fail-fast keeps the first implementation correct while the session/KV protocol is still being proven.
+
+Consequences:
+
+- Better observability and explicit errors are required.
+- Snapshotting becomes a later feature, not a blocker for first inference.
+- Generation Telemetry must report the last known phase and failure reason.
+- Client or gateway retry starts a new Route Session from scratch.
+
+### Gate 7: Transport
+
+Decision: keep binary HTTP for v1 activation transfer instead of jumping immediately to QUIC, WebRTC, or a custom transport.
+
+Rationale:
+
+- ADR-0008 already defines binary activation bodies with HTTP headers.
+- HTTP keeps the first implementation debuggable with the existing server stack and tooling.
+- The core risk is route/session/KV correctness, not transport optimization.
+- QUIC/WebRTC can be introduced later behind the same activation protocol once semantics are proven.
+
+Consequences:
+
+- Focus benchmark work on payload shape, chunking, and cache behavior first.
+- QUIC/WebRTC can be introduced as an optimization behind the same activation protocol.
+- v1 implementation can reuse the current HTTP routing, relay, and observability infrastructure.
+- Transport abstraction should be kept narrow enough that HTTP can be replaced later without changing backend cache semantics.
+
+## Grilling Progress
+
+Gates 1, 2, 3, 3A, 4, 5, 6, and 7 are resolved. The remaining work is to convert the resolved framework into implementation-ready issue briefs and prototype milestones.

.scratch/distributed-gguf-runtime/issues/01-local-llamacpp-gguf-backend.md

@@ -0,0 +1,29 @@
+# 01 — Local llama.cpp/GGUF backend
+
+Status: ready-for-agent
+
+## Goal
+
+Add a local full-model llama.cpp/GGUF backend to the node so a machine that can hold a GGUF model can serve it through the existing OpenAI-compatible node API.
+
+## Scope
+
+- Add backend selection for `llama.cpp` / GGUF.
+- Support launching or calling `llama-server` first; direct `libllama` bindings may come later.
+- Register model metadata and hardware profile with tracker.
+- Preserve current PyTorch path.
+- Add a local benchmark comparing PyTorch CPU vs llama.cpp/GGUF for the same supported small model.
+
+## Non-Goals
+
+- No distributed GGUF route yet.
+- No partial layer loading yet.
+- No torrent seeding yet.
+
+## Acceptance
+
+- A local GGUF model can answer `/v1/chat/completions`.
+- Startup output clearly says backend=`llama.cpp`.
+- Node registration includes backend and artifact metadata.
+- Test or smoke script verifies the backend wiring without requiring a huge model.
+

.scratch/distributed-gguf-runtime/issues/02-stable-session-and-distributed-kv-pytorch.md

@@ -0,0 +1,33 @@
+# 02 — Stable session and distributed KV in PyTorch path
+
+Status: ready-for-agent
+
+## Goal
+
+Fix the existing distributed PyTorch path so it does not recompute the full growing prompt for every output token.
+
+## Scope
+
+- Introduce stable `session_id` for one request/session.
+- Add per-node session cache keyed by `session_id`.
+- Split `/forward` semantics into prefill and decode-step.
+- Use model cache objects / `past_key_values` where supported.
+- Keep hot KV local to each shard node.
+- Add cleanup/TTL for abandoned sessions.
+
+## Current Problem
+
+The current distributed path:
+
+- calls `encode_prompt(current_text)` for every generated token
+- sends full-sequence activations through the route
+- calls layers with `use_cache=False`
+- creates a fresh UUID inside `_run_downstream_pipeline()`
+
+## Acceptance
+
+- Decode seam payload is one token / one hidden state after prefill.
+- Per-shard cache grows locally with generated tokens.
+- Regression test proves layer calls use cache after prefill.
+- Fallback error is explicit for models whose manual cache API is unsupported.
+

.scratch/distributed-gguf-runtime/issues/03-prefill-decode-wire-protocol.md

@@ -0,0 +1,29 @@
+# 03 — Prefill/decode wire protocol
+
+Status: ready-for-agent
+
+## Goal
+
+Define and implement the activation protocol needed by both PyTorch and future GGUF backends.
+
+## Scope
+
+- Add route/session lifecycle headers.
+- Separate `prefill` from `decode-step`.
+- Keep binary bfloat16 activation bodies.
+- Preserve relay compatibility.
+- Add route id and model artifact hash validation.
+
+## Draft Endpoints
+
+- `POST /sessions/{session_id}/prefill`
+- `POST /sessions/{session_id}/decode-step`
+- `DELETE /sessions/{session_id}`
+- `GET /sessions/{session_id}/status`
+
+## Acceptance
+
+- Old `/forward` remains temporarily or fails with clear version message.
+- Tests cover relay preservation of session headers.
+- Decode-step payload is independent of prompt length.
+

.scratch/distributed-gguf-runtime/issues/04-model-artifact-manifest-and-torrent.md

@@ -0,0 +1,24 @@
+# 04 — Model artifact manifest and torrent distribution
+
+Status: ready-for-agent
+
+## Goal
+
+Represent model artifacts independently from runtime routes so nodes can seed, verify, and advertise model files without relying on Hugging Face at runtime.
+
+## Scope
+
+- Define manifest schema.
+- Include file/chunk hashes.
+- Include tensor/layer map where available.
+- Include tokenizer and chat template hashes.
+- Include backend compatibility.
+- Add torrent/magnet URI fields and HTTP fallback URLs.
+- Extend node registration with artifact availability.
+
+## Acceptance
+
+- A model can be registered from a manifest without contacting Hugging Face.
+- Tracker can show coverage by artifact and layer range.
+- Node refuses to advertise corrupt artifacts.
+

.scratch/distributed-gguf-runtime/issues/05-llamacpp-layer-boundary-prototype.md

@@ -0,0 +1,32 @@
+# 05 — llama.cpp layer-boundary prototype
+
+Status: ready-for-human
+
+## Goal
+
+Prototype whether llama.cpp can execute only a selected layer range and accept/return hidden activations at model layer boundaries.
+
+## Scope
+
+- Start with a small model already supported by llama.cpp.
+- Run two local processes: head and tail.
+- Head owns embeddings + early layers.
+- Tail owns later layers + norm/lm_head.
+- Prefill once, then decode using local per-process KV.
+
+## Collaboration Point
+
+This is the best place to collaborate with Georgi/upstream llama.cpp. The desired upstream API shape:
+
+- load layer range or mmap full GGUF but execute layer range
+- run prefill chunk from inbound hidden states
+- run decode step from inbound hidden state
+- expose per-session KV/state handles
+- report cache memory budget
+
+## Acceptance
+
+- Localhost two-process decode does not recompute full prompt per token.
+- Seam payload after prefill is one hidden state per token.
+- No long-lived fork-only hooks unless upstream path is infeasible.
+

.scratch/distributed-gguf-runtime/issues/06-networked-distributed-gguf-route.md

@@ -0,0 +1,24 @@
+# 06 — Networked distributed GGUF route
+
+Status: pending
+
+Depends on: 01, 03, 04, 05
+
+## Goal
+
+Run a GGUF-backed model over a real multi-node route using the tracker-selected route and per-shard local KV.
+
+## Scope
+
+- Extend node backend registry with GGUF layer ranges.
+- Add route selection for GGUF nodes.
+- Use the prefill/decode protocol.
+- Track route health and queue depth.
+- Bill by layer work and token work.
+
+## Acceptance
+
+- Two physical machines can serve one model route.
+- Node dropout during alpha fails request cleanly.
+- Tracker metrics show prefill TPS, decode TPS, seam latency, and cache memory.
+

.scratch/distributed-gguf-runtime/issues/07-glm52-ornith-support-audit.md

@@ -0,0 +1,29 @@
+# 07 — Large-model support audit
+
+Status: pending
+
+Depends on: 01, 05
+
+## Goal
+
+Determine which large target models can run through the distributed path and what upstream runtime work remains.
+
+The first serious large-model target is `deepseek-ai/DeepSeek-V4-Flash`. GLM-5.2 and Ornith remain follow-up targets.
+
+## Scope
+
+- Verify PyTorch/Transformers load semantics for DeepSeek-V4-Flash.
+- Verify vLLM/SGLang serving support for DeepSeek-V4-Flash.
+- Verify whether a GGUF/llama.cpp quantization path exists for DeepSeek-V4-Flash.
+- Estimate artifact size and 128K KV/cache memory by layer range for DeepSeek-V4-Flash.
+- Verify llama.cpp/GGUF support for `glm_moe_dsa`.
+- Verify cache accounting for GLM-5.2 DSA/MLA.
+- Verify Ornith/Qwen3.5-MoE hybrid attention support.
+- Identify smallest viable quantization for quality-first use.
+
+## Acceptance
+
+- Written compatibility matrix.
+- Clear "supported now / upstream needed / not viable" status per model.
+- DeepSeek-V4-Flash has a recommended first-runtime path: PyTorch, vLLM/SGLang, llama.cpp/GGUF, or blocked.
+- Runtime blockers converted into issues or upstream collaboration notes.

.scratch/distributed-gguf-runtime/issues/08-generation-telemetry.md

@@ -0,0 +1,28 @@
+# Issue 08: Route-Session Generation Telemetry
+
+## Goal
+
+Expose realtime progress for long-running distributed inference requests. This is required whether or not token output is streamed.
+
+## Background
+
+Streaming token deltas is the preferred client experience when the backend and transport support it. Users still need realtime confidence that the route is alive and useful speed feedback during prefill, queueing, and any non-streaming fallback path.
+
+## Scope
+
+- Define a route-session telemetry schema.
+- Track phase: queued, loading, prefill, decode, finalizing, failed.
+- Track prefill token progress.
+- Track generated token count.
+- Track rolling and average tokens/sec.
+- Track active route nodes and failure reason.
+- Expose telemetry by SSE, WebSocket, or polling.
+- Ensure telemetry can coexist with streamed token deltas.
+
+## Acceptance Criteria
+
+- A client can display live route progress before the first output token is available.
+- During decode, the client sees rolling tokens/sec.
+- A streaming response can include token deltas and telemetry.
+- A non-streaming fallback still provides progress telemetry until final answer or failure.
+- Route failures include the last known phase and reason.

.scratch/distributed-gguf-runtime/issues/09-streaming-response.md

@@ -0,0 +1,24 @@
+# Issue 09: Streaming Response Support
+
+## Goal
+
+Stream generated token deltas to clients when the backend and transport support it, while preserving Generation Telemetry as an independent progress channel.
+
+## Background
+
+The preferred client experience is streamed output plus live tokens/sec feedback. Some early route proofs or backend integrations may only support a final response, so telemetry remains mandatory even when token deltas are unavailable.
+
+## Scope
+
+- Define an OpenAI-compatible streaming response shape.
+- Decide whether token deltas and telemetry travel over the same SSE stream or separate channels.
+- Preserve non-streaming final-response mode for simple clients.
+- Ensure prefill progress is visible before first token delta.
+- Ensure route failures close streams with a structured error and last known telemetry.
+
+## Acceptance Criteria
+
+- A client can request streamed token deltas.
+- A client can receive Generation Telemetry before and during streamed decode.
+- Non-streaming clients still receive telemetry through the route-session telemetry endpoint.
+- Stream failure includes session id, phase, and failure reason.

.scratch/distributed-gguf-runtime/research-prior-art.md

@@ -0,0 +1,231 @@
+# Prior Art: Distributed Large-Model Inference
+
+This note captures what existing projects appear to solve and what remains specific to this platform.
+
+## Petals
+
+Source: <https://github.com/bigscience-workshop/petals>
+
+Petals is the closest conceptual match for public volunteer inference. Its README describes running large models "BitTorrent-style", where a user loads a model through a `transformers`-like API and connects to a distributed network that hosts model layers. It explicitly supports seeing hidden states and using PyTorch/Transformers flexibility. The public README also notes privacy limitations: data is processed by other people in the public swarm, and sensitive use should run in a private swarm.
+
+What it solves:
+
+- public swarm of layer-serving peers
+- hidden-state exposure
+- route-like execution over model blocks
+- private swarm option
+- PyTorch/Transformers integration
+
+What it does not directly solve for us:
+
+- GGUF/llama.cpp artifact path
+- torrent artifact storage tied to node rewards
+- our billing/fraud/reputation model
+- our OpenAI-compatible tracker/node route model
+- a production path for GLM-5.2/DSA GGUF
+
+Design import:
+
+- Keep a PyTorch route as a reference implementation and validation harness.
+- Preserve hidden-state seam semantics.
+- Treat privacy as an explicit swarm property.
+
+## exo
+
+Source: <https://github.com/exo-explore/exo>
+
+exo connects local devices into an AI cluster. Its README emphasizes automatic device discovery, topology-aware model splitting, tensor parallelism, MLX support, RDMA over Thunderbolt, and multiple API compatibilities. It is strongest for colocated owned devices, especially Apple Silicon / MLX clusters.
+
+What it solves:
+
+- automatic local cluster discovery
+- topology-aware splitting
+- tensor parallelism
+- OpenAI/Ollama/Claude API compatibility
+- model placement previews
+- cluster dashboard
+
+What it does not directly solve for us:
+
+- untrusted internet volunteer network
+- reward, fraud, and reputation
+- torrent artifact distribution
+- Linux GPU maturity is stated as still under development in the README
+- GGUF/llama.cpp route protocol
+
+Design import:
+
+- Add placement previews before committing a route.
+- Model prefill/decode separately in benchmarks.
+- Use topology-aware routing, not just layer coverage.
+
+## Distributed Llama / dllama
+
+Source: <https://github.com/b4rtaz/distributed-llama>
+
+Distributed Llama connects home devices into a cluster for CPU/GPU inference. Its README describes tensor parallelism, Ethernet synchronization, Linux/macOS/Windows support, ARM and x86 AVX2 optimization, and a root/worker architecture. The root node loads the model and forwards weights/state to workers. Known limitations include only `2^n` nodes and a maximum node count equal to the model's number of KV heads.
+
+What it solves:
+
+- practical cross-platform home-device cluster
+- tensor-parallel synchronization
+- root/worker process model
+- custom model format and conversion path
+
+What it does not directly solve for us:
+
+- arbitrary volunteer joins/leaves
+- independent shard ownership from local/torrent disk
+- layer-range routing with tracker-managed marketplace
+- public network fraud/billing
+- GGUF as the native published artifact
+
+Design import:
+
+- KV-head constraints matter for tensor-parallel designs.
+- A root node that distributes weights is unacceptable for our torrent-first marketplace; nodes must independently acquire artifacts.
+
+## prima.cpp
+
+Sources:
+
+- <https://github.com/Lizonghang/prima.cpp>
+- <https://arxiv.org/abs/2504.08791>
+
+prima.cpp is a distributed llama.cpp implementation for low-resource home clusters. The README highlights mmap-based low memory pressure, piped-ring parallelism with prefetching, heterogeneity-aware workload distribution, automatic weak-device removal, GGUF quantization support, speculative decoding, dynamic batching, and support for Llama/Qwen/DeepSeek-class models. Its commands require each rank to point at the same GGUF file, and the README shows ring communication across ranks.
+
+What it solves:
+
+- llama.cpp-derived GGUF distributed execution
+- heterogeneous device scheduling
+- low memory pressure via mmap/page cache behavior
+- disk prefetch as a first-class performance dimension
+- ring communication for home clusters
+- GGUF quantization support
+
+What it does not directly solve for us:
+
+- public volunteer marketplace
+- torrent artifact discovery and seeding economics
+- tracker-injected route over internet/NAT/relay
+- per-node independent shard selection and rewards
+- GLM-5.2 support is not established from the README
+
+Design import:
+
+- Study mmap and prefetching before inventing partial GGUF loading.
+- Include disk speed and memory pressure in routing.
+- Heterogeneity-aware scheduling is mandatory.
+- Weak nodes should be excluded from a route if they slow the whole decode path.
+
+## llama.cpp / GGUF
+
+Sources:
+
+- <https://github.com/ggml-org/llama.cpp>
+- <https://raw.githubusercontent.com/ggml-org/llama.cpp/master/tools/gguf-split/README.md>
+- <https://raw.githubusercontent.com/ggml-org/llama.cpp/master/ggml/CMakeLists.txt>
+
+llama.cpp is the performance runtime we want for GGUF. It supports local GGUF loading, many CPU/GPU backends, OpenAI-compatible serving, quantization, and `gguf-split` can split or merge GGUF files by max size or tensor count. The ggml build options include many hardware backends and RPC support.
+
+What it solves:
+
+- mature CPU/GPU local inference
+- GGUF ecosystem
+- quantized weights
+- local OpenAI-compatible server
+- split/merge tooling for artifact distribution
+
+What it does not solve by itself:
+
+- torrent distribution and reward model
+- per-session distributed route over arbitrary nodes
+- public-node trust/fraud model
+- stable API for arbitrary layer-boundary hidden-state I/O, if not already exposed
+
+Design import:
+
+- Use llama.cpp locally before attempting distributed GGUF.
+- Collaborate upstream on layer-range execution and KV ownership APIs.
+- Keep GGUF split for artifacts, not as the only execution-shard definition.
+
+## GLM-5.2
+
+Sources:
+
+- <https://huggingface.co/zai-org/GLM-5.2>
+- <https://huggingface.co/zai-org/GLM-5.2/blob/main/config.json>
+
+GLM-5.2 is MIT licensed, 753B parameters, and advertises a 1M-token context. The config uses `glm_moe_dsa`, 78 layers, `hidden_size=6144`, `kv_lora_rank=512`, `qk_head_dim=256`, `qk_nope_head_dim=192`, `qk_rope_head_dim=64`, `v_head_dim=256`, and `max_position_embeddings=1048576`. The model card states IndexShare reduces per-token FLOPs at 1M context.
+
+Design import:
+
+- DSA/MLA-style compressed KV makes 128K feasible.
+- Tracker should not need to understand DSA internals; backend should expose cache budget and compatibility metadata.
+- GLM-5.2 is a later target after generic distributed KV works.
+
+## DeepSeek-V4-Flash
+
+Sources:
+
+- <https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash>
+- <https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/main/config.json>
+
+DeepSeek-V4-Flash is MIT licensed and published as `deepseek-ai/DeepSeek-V4-Flash` on Hugging Face. The model card describes DeepSeek-V4-Flash as a 284B-parameter MoE model with 13B activated parameters and a 1M-token context. Hugging Face tags it as `deepseek_v4`, Transformers, Safetensors, and FP8. The repository lists 46 safetensor shards and around 160 GB total size.
+
+Config highlights:
+
+- `model_type=deepseek_v4`
+- `hidden_size=4096`
+- `num_hidden_layers=43`
+- `num_attention_heads=64`
+- `num_key_value_heads=1`
+- `n_routed_experts=256`
+- `num_experts_per_tok=6`
+- `q_lora_rank=1024`
+- `o_lora_rank=1024`
+- `qk_rope_head_dim=64`
+- `sliding_window=128`
+- `max_position_embeddings=1048576`
+- `expert_dtype=fp4`
+- FP8 quantization metadata
+
+Design import:
+
+- Good first serious large-model target after the protocol smoke test because it is much smaller than 1.6T-class models while still validating MoE, compressed attention/cache behavior, and large-context routing.
+- Not the first protocol smoke model. Use a smaller, boring, llama.cpp-supported GGUF model first so route/session/KV bugs are isolated from DeepSeek-specific architecture support.
+- The support audit must verify the available local runtime path: PyTorch/Transformers, vLLM/SGLang, and any GGUF/llama.cpp quantization route.
+
+## Ornith-1.0-397B
+
+Sources:
+
+- <https://huggingface.co/deepreinforce-ai/Ornith-1.0-397B>
+- <https://huggingface.co/inferencerlabs/Ornith-1.0-397B-MLX-Q9>
+
+Ornith-1.0-397B is MIT licensed, Qwen3.5-MoE based, with 397B MoE scale. Its base config shows 60 layers and a hybrid pattern where full attention appears every fourth layer, with other layers using linear attention. The MLX Q9 quantized variant is around 447 GB and reports high-quality Q9 behavior in its model card.
+
+Design import:
+
+- Hybrid attention can make large models more tractable than dense full-attention assumptions.
+- Model-specific cache accounting is required; "params" alone is not enough to route.
+
+## Synthesis
+
+The prior art strongly supports the direction, but no project exactly matches the target product:
+
+- Petals proves volunteer layer-serving is useful.
+- exo proves UX/topology-aware local clusters matter.
+- Distributed Llama proves CPU home clusters can cooperate but also shows root/worker constraints.
+- prima.cpp proves llama.cpp/GGUF distribution across low-resource devices is plausible and that disk/mmap scheduling matters.
+- llama.cpp/GGUF is the ecosystem to collaborate with for runtime performance.
+- DeepSeek-V4-Flash is a plausible first serious large-model target after a small protocol smoke model.
+
+The platform-specific work remains:
+
+- torrent/content-addressed model artifact marketplace
+- tracker-owned route selection and billing
+- per-shard local KV sessions
+- relay/NAT support
+- fraud/reputation/audit
+- OpenAI-compatible public gateway

.scratch/distributed-gguf-runtime/technical-challenges.md

@@ -0,0 +1,412 @@
+# Distributed GGUF Technical Challenge Register
+
+This document focuses on the engineering problems that decide whether the distributed GGUF path is viable. The important distinction is:
+
+- **Model artifacts move like torrents.**
+- **Inference state moves like a pipeline.**
+- **Hot KV state does not move unless we are explicitly checkpointing or repairing a route.**
+
+## Current Constraint
+
+The existing full local PyTorch path lets Transformers own generation and local KV cache.
+
+The existing distributed PyTorch path does not. It manually calls shard layers with cache disabled and recomputes the whole growing prompt for every generated token. It passes hidden activations across shard boundaries, not KV cache, but those activations currently include the full sequence on every decode step.
+
+For a 128K context and `hidden_size=6144`, one bfloat16 activation crossing one shard boundary is roughly:
+
+```text
+131072 tokens * 6144 hidden * 2 bytes = 1.5 GiB
+```
+
+That is acceptable once during chunked prefill only if chunked and streamed. It is not acceptable once per generated token.
+
+## Challenge 1: Decode Must Be O(1) Per Token Across Each Seam
+
+Problem:
+
+During decode, sending `[batch, sequence, hidden]` over the network scales with context length. At 128K, the network dominates everything.
+
+Solution:
+
+Split execution into explicit **prefill** and **decode-step** phases.
+
+- Prefill accepts prompt chunks and builds local cache on every shard.
+- Decode-step accepts exactly one new token or one-step activation.
+- Every shard reads its own hot KV state, appends one position, and forwards a one-step activation.
+
+Acceptance test:
+
+- A two-node route prefills a 4K prompt once.
+- The next 100 generated tokens do not resend a 4K activation.
+- Decode seam payload is proportional to `hidden_size`, not `context_length * hidden_size`.
+
+## Challenge 2: Stable Route Session State
+
+Problem:
+
+KV cache only works if every hop agrees that multiple calls belong to the same route session. A fresh request id per hop or per token destroys cache locality.
+
+Solution:
+
+Introduce a route-session lifecycle.
+
+```text
+create route session
+  -> tracker pins inference route
+  -> head node assigns session_id and route_id
+  -> every hop allocates local cache for its layer range
+  -> prefill chunks append cache
+  -> decode steps append cache
+  -> close session releases cache
+```
+
+Minimum state key:
+
+```text
+session_id
+route_id
+model_preset
+model_revision
+backend_id
+cache_abi
+layer_start
+layer_end
+position
+```
+
+Acceptance test:
+
+- A node can report active sessions and cache bytes by session.
+- Closing a session frees the per-shard cache.
+- Replaying a decode-step with the wrong route/session fails before model execution.
+
+## Challenge 3: KV Cache Ownership
+
+Problem:
+
+A centralized KV cache sounds attractive for failover, but it puts remote storage in the tightest loop of generation. It also creates privacy and consistency problems.
+
+Solution:
+
+Hot KV state is owned by the node that owns the shard for that route session.
+
+```text
+Node A: layers 0..15 hot KV for session S
+Node B: layers 16..31 hot KV for session S
+Node C: layers 32..77 hot KV for session S
+```
+
+The tracker may know where active KV lives, but it does not serve it during decode.
+
+Cache servers may store:
+
+- prefix snapshots
+- failover checkpoints
+- audit samples
+- cold reusable context blocks
+
+They must not be required for every generated token.
+
+Acceptance test:
+
+- Killing a cache server does not affect an active decode route.
+- Killing a route node fails the route in alpha unless a compatible snapshot exists.
+
+## Challenge 4: Prefill Is Still Large
+
+Problem:
+
+Even with correct decode, prefill can move a lot of data. A 128K prompt cannot be sent as one activation blob through many shard boundaries.
+
+Solution:
+
+Use the existing binary activation direction from ADR-0008:
+
+- bfloat16 activation body
+- shape/dtype/session metadata in headers
+- zstd level 1 optional compression
+- chunked prefill
+- backpressure between hops
+
+For large contexts, prefill should stream in chunks such as 128, 256, or 512 tokens. The right chunk size is a benchmark output, not a constant baked into the domain model.
+
+Acceptance test:
+
+- Peak per-hop prefill memory is bounded by chunk size.
+- A slow downstream node applies backpressure instead of letting upstream buffer the whole prompt.
+
+## Challenge 5: GGUF Artifact Splits Are Not Execution Shards
+
+Problem:
+
+GGUF split files can divide model data by size or tensor count. That is useful for storage and transfer, but it is not the same as a network Shard. A Shard in this project is a contiguous layer range with reward, route, and cache meaning.
+
+Solution:
+
+Define an artifact manifest that maps storage chunks to semantic model parts.
+
+Required concepts:
+
+```text
+artifact_id
+model_preset
+upstream_repo
+upstream_revision
+license
+runtime_backend
+quantization
+context_cap
+file_hashes
+piece_hashes
+tensor_to_layer_map
+layer_to_artifact_map
+tokenizer_artifacts
+cache_descriptor
+```
+
+The Shard Swarm seeds artifacts. The Inference Route executes shards.
+
+Acceptance test:
+
+- Given `layers 16..31`, a node can compute the exact artifact pieces it must download and verify.
+- Given a local artifact directory, the node can prove which layer ranges it can serve.
+
+## Challenge 6: llama.cpp Is Optimized For Whole-Graph Execution
+
+Problem:
+
+`llama-server` is excellent for local inference, but a distributed route needs lower-level capabilities:
+
+- load selected layers/tensors or mmap them without full materialization
+- accept hidden states from a previous shard
+- execute only a layer range
+- emit hidden states at a boundary
+- own KV/state for only that layer range
+- report cache layout and memory requirements
+
+Those are not the normal public serving abstractions.
+
+Solution:
+
+Stage the llama.cpp path instead of jumping directly to internet-scale distributed GGUF.
+
+1. Use llama.cpp as a full local GGUF backend for immediate CPU performance.
+2. Build a localhost layer-boundary prototype on a simple supported GGUF model.
+3. Identify the minimal `libllama`/ggml hooks needed for layer-range execution.
+4. Collaborate upstream on a stable extension rather than carrying a long-lived fork.
+
+Acceptance test:
+
+- Process A runs layers `0..k`, exports hidden states.
+- Process B imports those hidden states, runs `k+1..n`, and produces logits close to full single-process execution.
+- Both processes maintain only their own cache state.
+
+## Challenge 7: Model Architectures Are Not Uniform
+
+Problem:
+
+Dense Llama-style attention, MoE, MLA/DSA, and hybrid linear/full attention do not have the same cache shape, routing cost, or layer cost.
+
+GLM-5.2 uses compressed DSA/MLA-style state. Ornith uses a hybrid attention pattern. Parameter count alone is a poor routing metric.
+
+Solution:
+
+Keep model-specific cache internals inside the backend. The tracker should route based on backend-advertised capabilities and measured telemetry, not on hardcoded tensor formulas.
+
+Backend capability report:
+
+```text
+model_arch
+supported_runtime
+supports_prefill
+supports_decode_step
+supports_layer_range
+supports_partial_artifacts
+cache_abi
+cache_bytes_per_token_estimate
+prefill_tokens_per_second
+decode_tokens_per_second
+active_memory_floor
+```
+
+Acceptance test:
+
+- The tracker can reject a route because one node lacks the required cache ABI.
+- A model support audit can say "artifact available, local full inference works, distributed layer-boundary unsupported" without ambiguity.
+
+## Challenge 8: Tensor Parallelism Is Not The Same Product
+
+Problem:
+
+Projects like Distributed Llama and prima.cpp lean toward local-cluster tensor/ring parallelism. That can work on a trusted LAN, but it usually requires tight synchronization every layer. On a public internet volunteer route, that becomes fragile and hard to reward.
+
+Solution:
+
+For the public network, make a Shard a contiguous layer range. Tensor parallelism can exist inside one node, one trusted colocated pod, or one future "composite node", but not as the first public routing primitive.
+
+Acceptance test:
+
+- A public route can be represented as ordered layer coverage.
+- Billing can attribute work to layer ranges.
+- No cross-node all-reduce is required on every layer for v1.
+
+## Challenge 9: Heterogeneity And Stragglers
+
+Problem:
+
+A route is only as fast as its slowest hop during decode. A weak node holding a bottleneck shard can make a 1.6T model technically available but unusable.
+
+Solution:
+
+Route selection must use measured telemetry, not static declarations.
+
+Metrics:
+
+- prefill throughput
+- decode throughput
+- queue depth
+- disk read rate
+- memory pressure
+- network latency to neighbors
+- route failure rate
+- cache warmth
+
+The tracker should prefer complete routes that avoid weak nodes, and the rebalancer should increase redundancy for bottleneck layer ranges.
+
+Acceptance test:
+
+- A slow node is removed from a candidate route even if it has the needed layer range.
+- The coverage map can show "covered but under-provisioned" separately from "coverage gap".
+
+## Challenge 10: Reliability And Failover
+
+Problem:
+
+If hot KV is local, route repair is not free. A replacement node cannot continue decoding unless it has compatible cache state.
+
+Solution:
+
+Alpha behavior should be simple:
+
+- route failure during prefill: fail and retry from scratch
+- route failure during decode: fail unless compatible snapshot exists
+- tracker restart: active sessions may be lost
+- node restart: local hot KV is lost
+- client-visible telemetry reports the last known phase and failure reason
+
+Later behavior:
+
+- periodic prefix snapshots
+- snapshot generation ids
+- cache ABI compatibility checks
+- route repair only when the replacement node has the same model revision, layer range, backend cache ABI, and snapshot position
+
+Acceptance test:
+
+- Failures produce explicit route-session errors.
+- No node silently continues from missing or incompatible cache state.
+
+## Challenge 11: Privacy, Fraud, And Audit
+
+Problem:
+
+Hidden activations and KV state can leak information. Public volunteer inference is not private by default. Also, a node can return bad activations while still appearing available.
+
+Solution:
+
+Separate product modes:
+
+- public swarm: low privacy, broad access, audited
+- private swarm: trusted nodes, stronger privacy expectation
+- paid trusted route: selected nodes with stronger guarantees
+
+Use existing validation-event and slash-proof concepts for audit, but adapt them to distributed routes:
+
+- record model preset, route, node wallets, prompt metadata, output, and sampling seed
+- sample full-route replays where feasible
+- compare output/logits within model-specific tolerance
+
+Acceptance test:
+
+- A client can choose public or private route policy.
+- A validation event contains enough information to reproduce route membership and observed output.
+
+## Challenge 12: Economics Must Not Reward The Wrong Bottleneck
+
+Problem:
+
+Layer count, parameter count, active MoE experts, cache memory, disk serving, and network transfer are different costs. A naive equal split across nodes will be wrong.
+
+Solution:
+
+Start with simple compute accounting:
+
+```text
+node_reward_weight =
+  owned_layer_work
+  * prefill_tokens
+  + owned_layer_work
+  * decode_tokens
+```
+
+Then refine with:
+
+- measured throughput
+- active MoE cost
+- storage/seeding contribution
+- cache memory reservation
+- reliability
+
+Keep artifact seeding rewards separate from inference rewards until fraud and metering are clear.
+
+Acceptance test:
+
+- A node that only seeds artifacts is not paid as if it executed inference.
+- A node that executes a heavier shard can earn more than a node executing a light shard.
+
+## Challenge 13: Long Requests Need Streaming Or Realtime Feedback
+
+Problem:
+
+Large distributed routes may spend meaningful time in artifact loading, prefill, queueing, or slow decode. The product can tolerate latency, but users should not wait blindly.
+
+Solution:
+
+Streaming token deltas is preferred when the backend and client transport support it. Generation Telemetry is required regardless of whether token deltas are streamed.
+
+Minimum telemetry:
+
+```text
+session_id
+route_id
+model_preset
+phase = queued | loading | prefill | decode | finalizing | failed
+prefill_tokens_done
+prefill_tokens_total
+generated_tokens
+rolling_tokens_per_second
+average_tokens_per_second
+active_route_nodes
+failure_reason
+```
+
+The gateway may expose token deltas and telemetry through Server-Sent Events or WebSocket. Simple clients may use a polling endpoint for telemetry and receive the final answer only when complete.
+
+Acceptance test:
+
+- A client can show live progress before the first output token is available.
+- During decode, the user sees streamed token deltas when supported.
+- During decode, the user sees rolling tokens/sec even if output text is not streamed.
+- A failed route returns a final error and the last known phase/reason.
+
+## Engineering Order
+
+1. Fix distributed PyTorch cache semantics first. This proves the route-session model without llama.cpp internals.
+2. Add local full-model llama.cpp/GGUF serving for immediate CPU improvement.
+3. Add Generation Telemetry for route sessions so long requests are observable.
+4. Preserve binary HTTP activation transfer while splitting prefill/decode and measuring payload sizes.
+5. Add artifact manifest and Shard Swarm metadata.
+6. Prototype llama.cpp layer-boundary execution locally.
+7. Network the GGUF route only after the cache/session protocol works.
+8. Audit DeepSeek-V4-Flash as the first serious large-model target.
+9. Audit GLM-5.2 and Ornith support after simpler GGUF models pass the route test.

CONTEXT.md

@@ -18,23 +18,51 @@ _Avoid_: partition, slice, chunk, segment
 The P2P group of nodes that collectively seed and download a specific shard. One swarm exists per shard.
 _Avoid_: torrent, cluster, pool
 
-**Inference Route**:
-An ordered sequence of nodes whose shards together cover all layers of a model. The tracker selects the optimal route per request.
-_Avoid_: pipeline, chain, path
-
-**Gateway**:
-The network entry point that accepts client requests (OpenAI-compatible HTTP), selects an inference route from the tracker, and streams results back.
-_Avoid_: proxy, relay, orchestrator, primary
-
-### Tracker
+**Inference Route**:
+An ordered sequence of nodes whose shards together cover all layers of a model. The tracker selects the optimal route per request.
+_Avoid_: pipeline, chain, path
+
+**Route Session**:
+An active inference request bound to one Inference Route and one stable session id while the request is being served.
+_Avoid_: conversation, job, token stream
+
+**Activation Seam**:
+The boundary between two adjacent shard executions where hidden states pass from one node to the next.
+_Avoid_: handoff, layer gap, boundary hop
+
+**Hot KV State**:
+The live attention/cache state a node holds for its own shard during a Route Session.
+_Avoid_: centralized KV cache, global cache, remote cache
+
+**Prefix Snapshot**:
+A persisted copy of reusable Route Session state for a prompt prefix, used for reuse, retry, or failover.
+_Avoid_: hot cache, CDN cache, active KV
+
+**Model Artifact**:
+A versioned model file or tokenizer file that nodes download, verify, and keep locally to serve a Model Preset.
+_Avoid_: model blob, weights dump, asset
+
+**Artifact Manifest**:
+The canonical record that identifies the Model Artifacts, their integrity checks, and the model parts they support.
+_Avoid_: torrent file, metadata JSON, download list
+
+**Gateway**:
+The network entry point that accepts client requests (OpenAI-compatible HTTP), selects an inference route from the tracker, and streams results and progress to the client when possible.
+_Avoid_: proxy, relay, orchestrator, primary
+
+**Generation Telemetry**:
+Realtime progress information for an active Route Session, including phase, generated token count, and tokens-per-second speed.
+_Avoid_: logs, debug output
+
+### Tracker
 
 **Tracker**:
 The coordinator service that maintains the node registry, scores nodes by throughput/latency, and assigns inference routes. Runs as a centralized service with a P2P gossip fallback.
 _Avoid_: coordinator, scheduler, director
 
-**Tracker Node**:
-A node that serves at least the first-layer shard (`layers[0..k]`) for a model and acts as the inference entry point for that model. Tracker nodes own the tokenizer and `embed_tokens`, receive client requests directly, select the onward route from the coverage map, and stream results back. Any node advertising a new model to the network becomes its tracker node.
-_Avoid_: primary node, master node, gateway node
+**Tracker Node**:
+A node that serves at least the first-layer shard (`layers[0..k]`) for a model and acts as the inference entry point for that model. Tracker nodes own the tokenizer and `embed_tokens`, receive client requests directly, select the onward route from the coverage map, and stream results and progress when possible. Any node advertising a new model to the network becomes its tracker node.
+_Avoid_: primary node, master node, gateway node
 
 **Coverage Map**:
 The tracker's per-model mapping of layer ranges to node counts: `[(start_layer, end_layer, node_count), ...]`. A layer range with `node_count=0` is a coverage gap — the model is unroutable until the gap is filled. Coverage-first bin-packing fills all gaps before adding redundancy.

packages/node/meshnet_node/cli.py

@@ -66,6 +66,8 @@ def _run_node(cfg: dict) -> None:
             max_loaded_shards=int(cfg.get("max_loaded_shards", 1)),
             debug=bool(cfg.get("debug", False)),
             tracker_source_disabled=bool(cfg.get("tracker_source_disabled", False)),
+            torch_threads=cfg.get("torch_threads"),
+            torch_interop_threads=cfg.get("torch_interop_threads"),
         )
     except Exception as exc:
         print(f"\nERROR: {exc}", file=sys.stderr, flush=True)
@@ -151,6 +153,10 @@ def _cmd_default(args) -> int:
         overrides["debug"] = True
     if getattr(args, "tracker_source_disabled", False):
         overrides["tracker_source_disabled"] = True
+    if getattr(args, "torch_threads", None) is not None:
+        overrides["torch_threads"] = args.torch_threads
+    if getattr(args, "torch_interop_threads", None) is not None:
+        overrides["torch_interop_threads"] = args.torch_interop_threads
 
     if overrides:
         cfg = merge_cli_overrides(cfg, **overrides)
@@ -249,6 +255,8 @@ def _cmd_start(args) -> int:
             max_loaded_shards=getattr(args, "max_shards", 1),
             debug=getattr(args, "debug", False),
             tracker_source_disabled=getattr(args, "tracker_source_disabled", False),
+            torch_threads=getattr(args, "torch_threads", None),
+            torch_interop_threads=getattr(args, "torch_interop_threads", None),
         )
     except Exception as exc:
         print(f"ERROR: {exc}", file=sys.stderr, flush=True)
@@ -299,6 +307,10 @@ def main() -> None:
                         help="Override autodetected VRAM/RAM budget in MB used for shard assignment")
     parser.add_argument("--max-shards", type=int, metavar="N", default=None,
                         help="Maximum shard slots this node advertises to the tracker (default 1)")
+    parser.add_argument("--torch-threads", type=int, metavar="N",
+                        help="Set PyTorch intra-op CPU worker threads")
+    parser.add_argument("--torch-interop-threads", type=int, metavar="N",
+                        help="Set PyTorch inter-op CPU worker threads")
     parser.add_argument("--debug", action="store_true", help="Enable verbose node debug logging")
     parser.add_argument("--no-tui", action="store_true", help="Plain-text output (no rich dashboard)")
     parser.add_argument("--compact", action="store_true", help="Single-line status output")
@@ -334,6 +346,10 @@ def main() -> None:
                            help="Override autodetected VRAM/RAM budget in MB used for shard assignment")
     start_cmd.add_argument("--max-shards", type=int, default=1, metavar="N",
                            help="Maximum shard slots this node advertises to the tracker (default 1)")
+    start_cmd.add_argument("--torch-threads", type=int, metavar="N",
+                           help="Set PyTorch intra-op CPU worker threads")
+    start_cmd.add_argument("--torch-interop-threads", type=int, metavar="N",
+                           help="Set PyTorch inter-op CPU worker threads")
     start_cmd.add_argument("--debug", action="store_true", help="Enable verbose node debug logging")
     start_cmd.add_argument("--tracker-source-disabled", action="store_true",
                            help="Skip tracker/peer model-file sources and download from HuggingFace directly")

packages/node/meshnet_node/startup.py

@@ -3,6 +3,7 @@
 from __future__ import annotations
 
 import json
+import os
 import socket
 import sys
 import threading
@@ -139,6 +140,59 @@ def _hardware_label(device: str, gpu_name: str | None = None) -> str:
     return "CPU"
 
 
+def _positive_int(value: int | str | None, name: str) -> int | None:
+    if value is None or value == "":
+        return None
+    try:
+        parsed = int(value)
+    except (TypeError, ValueError) as exc:
+        raise ValueError(f"{name} must be a positive integer") from exc
+    if parsed < 1:
+        raise ValueError(f"{name} must be a positive integer")
+    return parsed
+
+
+def _configure_torch_threads(
+    torch_threads: int | None = None,
+    torch_interop_threads: int | None = None,
+) -> dict[str, int]:
+    """Apply PyTorch CPU thread settings before model load/benchmark."""
+    intra_threads = _positive_int(
+        torch_threads if torch_threads is not None else os.environ.get("MESHNET_TORCH_THREADS"),
+        "--torch-threads",
+    )
+    interop_threads = _positive_int(
+        torch_interop_threads
+        if torch_interop_threads is not None
+        else os.environ.get("MESHNET_TORCH_INTEROP_THREADS"),
+        "--torch-interop-threads",
+    )
+
+    if intra_threads is not None:
+        os.environ.setdefault("OMP_NUM_THREADS", str(intra_threads))
+        os.environ.setdefault("MKL_NUM_THREADS", str(intra_threads))
+    try:
+        import torch
+    except ModuleNotFoundError:
+        return {}
+
+    if intra_threads is not None:
+        torch.set_num_threads(intra_threads)
+    if interop_threads is not None:
+        torch.set_num_interop_threads(interop_threads)
+
+    active: dict[str, int] = {}
+    try:
+        active["torch_threads"] = int(torch.get_num_threads())
+    except Exception:
+        pass
+    try:
+        active["torch_interop_threads"] = int(torch.get_num_interop_threads())
+    except Exception:
+        pass
+    return active
+
+
 def _max_assignable_layers(memory_mb: int, total_layers: int | None) -> int:
     if total_layers is None or total_layers <= 0 or memory_mb <= 0:
         return 0
@@ -415,6 +469,8 @@ def run_startup(
     max_loaded_shards: int = 1,
     debug: bool = False,
     tracker_source_disabled: bool = False,
+    torch_threads: int | None = None,
+    torch_interop_threads: int | None = None,
 ) -> StubNodeServer | TorchNodeServer:
     """Execute the full startup sequence and return a running node server.
 
@@ -455,6 +511,12 @@ def run_startup(
 
     print("Detecting hardware...", flush=True)
     hw = detect_hardware()
+    torch_thread_config = _configure_torch_threads(torch_threads, torch_interop_threads)
+    if torch_thread_config:
+        hw.update(torch_thread_config)
+        intra = torch_thread_config.get("torch_threads", "?")
+        interop = torch_thread_config.get("torch_interop_threads", "?")
+        print(f"  PyTorch threads: intra-op={intra}, inter-op={interop}", flush=True)
     device: str = hw["device"]
     gpu_name: str | None = hw.get("gpu_name")
     vram_mb: int = hw.get("vram_mb", 0)

tests/test_node_startup.py

@@ -15,6 +15,7 @@ import pytest
 from meshnet_node.downloader import download_shard, write_shard_archive
 from meshnet_node.hardware import detect_hardware, benchmark_throughput
 from meshnet_node.startup import (
+    _configure_torch_threads,
     _hardware_label,
     _infer_relay_url_from_tracker,
     _memory_budget,
@@ -155,11 +156,35 @@ def test_benchmark_throughput_fallback_on_bad_device():
     assert result == 1.0
 
 
+def test_configure_torch_threads_applies_explicit_settings(monkeypatch):
+    """Node startup can tune PyTorch CPU thread pools before loading a model."""
+    calls: dict[str, int] = {}
+
+    fake_torch = types.SimpleNamespace(
+        set_num_threads=lambda value: calls.update({"threads": value}),
+        set_num_interop_threads=lambda value: calls.update({"interop_threads": value}),
+        get_num_threads=lambda: calls["threads"],
+        get_num_interop_threads=lambda: calls["interop_threads"],
+    )
+
+    monkeypatch.setitem(sys.modules, "torch", fake_torch)
+    monkeypatch.delenv("OMP_NUM_THREADS", raising=False)
+    monkeypatch.delenv("MKL_NUM_THREADS", raising=False)
+
+    active = _configure_torch_threads(torch_threads=12, torch_interop_threads=2)
+
+    assert calls == {"threads": 12, "interop_threads": 2}
+    assert os.environ["OMP_NUM_THREADS"] == "12"
+    assert os.environ["MKL_NUM_THREADS"] == "12"
+    assert active == {"torch_threads": 12, "torch_interop_threads": 2}
+
+
 def test_benchmark_throughput_is_registered_in_payload(monkeypatch, tmp_path):
     """benchmark_tokens_per_sec from the benchmark is included in the tracker registration."""
     import meshnet_node.startup as startup_mod
 
     captured: dict = {}
+    thread_calls: dict[str, int] = {}
 
     class FakeNode:
         backend = None
@@ -177,6 +202,16 @@ def test_benchmark_throughput_is_registered_in_payload(monkeypatch, tmp_path):
     monkeypatch.setattr(startup_mod, "detect_hardware",
                         lambda: {"device": "cpu", "gpu_name": None, "vram_mb": 0, "ram_mb": 16384})
     monkeypatch.setattr(startup_mod, "benchmark_throughput_checked", lambda _device: (42.5, True, None))
+    monkeypatch.setitem(
+        sys.modules,
+        "torch",
+        types.SimpleNamespace(
+            set_num_threads=lambda value: thread_calls.update({"threads": value}),
+            set_num_interop_threads=lambda value: thread_calls.update({"interop_threads": value}),
+            get_num_threads=lambda: thread_calls["threads"],
+            get_num_interop_threads=lambda: thread_calls["interop_threads"],
+        ),
+    )
     monkeypatch.setattr(startup_mod, "TorchNodeServer", lambda **_kw: FakeNode())
     monkeypatch.setattr(startup_mod, "_detect_num_layers", lambda _model_id: 24)
     monkeypatch.setattr(startup_mod, "RelayHttpBridge", None)
@@ -192,10 +227,14 @@ def test_benchmark_throughput_is_registered_in_payload(monkeypatch, tmp_path):
         shard_start=0,
         shard_end=23,
         wallet_path=tmp_path / "wallet.json",
+        torch_threads=8,
+        torch_interop_threads=1,
     )
     node.stop()
 
     assert captured.get("benchmark_tokens_per_sec") == 42.5
+    assert captured["hardware_profile"]["torch_threads"] == 8
+    assert captured["hardware_profile"]["torch_interop_threads"] == 1
     assert captured["hardware_profile"]["benchmark_device"] == "cpu"
     assert captured["hardware_profile"]["benchmark_ok"] is True