popov/neuron-tai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: define distributed GGUF runtime plan

Browse Source
This commit is contained in:
Dobromir Popov
2026-07-13 15:09:27 +03:00
parent b5fa7245df
commit 4cae4a6c5c
42 changed files with 4913 additions and 691 deletions
Download Patch File Download Diff File Expand all files Collapse all files

202
.scratch/distributed-gguf-runtime/ADR-0020-distributed-gguf-runtime.md
View File

@@ -1,177 +1,121 @@
# ADR-0020: Distributed GGUF/llama.cpp Runtime With Per-Shard Local KV
# ADR-0020: Lean Native Distributed GGUF Runtime
Status: Proposed
Status: Accepted
Date: 2026-07-13
## Context
The project currently uses PyTorch/Transformers for real model shards. That decision was captured in ADR-0001 because llama.cpp RPC at the time required the primary node to load the full model and distribute weights to workers, which conflicted with the desired model where nodes independently hold shards.
The project currently uses Transformers/safetensors as its real model execution backend. This provides broad architecture coverage and a correctness reference, but reported and observed consumer CPU/GPU inference performance motivates evaluating llama.cpp/GGML and quantized GGUF.
We now want to serve very large open models, including GLM-5.2 and Ornith-class MoE models, over a torrent-like inference marketplace. CPU and mixed consumer hardware matter. LM Studio and llama.cpp demonstrate much better CPU/GGUF performance than our current PyTorch CPU path. The user also has a personal relationship with Georgi Gerganov, making upstream collaboration plausible.
The product objective is not merely local GGUF serving. It is performant concurrent inference for top open models whose weights do not fit on one consumer node. The project already owns the Tracker, Inference Route, Route Session, Activation Seam, local Hot KV State, relay/direct transport, cancellation, telemetry, billing, and capability admission.
The current distributed PyTorch path is not yet production-grade: it recomputes the full growing sequence for every output token and disables KV cache inside manual layer calls. It sends hidden activations across seams, not KV, but those activations currently cover the full sequence every decode step.
Research audited llama.cpp RPC, GPUStack/llama-box, Nakshatra, prima.cpp, llama-gguf, LiGGUF, vLLM and its GGUF plugin, Petals, exo, and related projects. No repository provides the complete public-network contract. llama.cpp is the strongest GGUF execution substrate. vLLM has mature managed-cluster parallelism and scheduling concepts but its PP/TP/EP runtime assumes a static trusted distributed world and is unsuitable as the public Shard runtime.
The project must remain lean and avoid combining several half-integrated inference control planes.
## Decision
Adopt a distributed GGUF/llama.cpp runtime track while keeping PyTorch as the reference and fast-architecture backend.
### Primary native runtime
The runtime model is:
Use llama.cpp/GGML through one standalone C++ Shard worker and a small exact-commit patch stack.
- GGUF/model artifacts are distributed through torrent/content-addressed storage.
- Nodes independently acquire and verify artifacts; no root node streams model weights to workers at session start.
- Tracker chooses a sticky route covering all layers.
- Each node owns hot KV/state for the layers it executes.
- Prefill sends chunked activations through the route and builds local per-shard KV.
- Decode sends one-step activations through the route and appends local KV at every shard.
- Cache/CDN servers store cold artifacts and optional prefix/session snapshots, not hot per-token KV.
- Context is capped at 128K for the first serious product path.
The patch scope is limited to:
## Technical Framework
- Range-aware GGUF tensor ownership/loading.
- Architecture-defined intermediate boundary input/output.
- Intermediate output before tail normalization/head.
- Layer-filtered KV and external session-to-sequence mapping.
The design separates five planes:
Meshnet networking, routing, admission, billing, telemetry, and work evidence stay outside llama.cpp.
- **Control plane**: tracker registry, coverage map, route selection, session lifecycle, telemetry, billing, and audit.
- **Artifact plane**: Shard Swarms, GGUF/safetensors/tokenizer files, manifests, hashes, and local node storage.
- **Execution plane**: active Inference Route, chunked prefill, one-step decode, and hidden-state movement across activation seams.
- **Session state plane**: per-shard Hot KV State on route nodes, plus optional Prefix Snapshots outside the hot loop.
- **Economics/trust plane**: reward accounting, validation events, slash proofs, public/private route policy.
Nakshatra, prima.cpp, llama-gguf, LiGGUF, and historical GPUStack are source/test donors only. Their repositories are not runtime dependencies.
Hard invariants:
### Distributed parallelism
1. Public-network Shards are contiguous layer ranges.
2. Hot KV State is local to the node serving that Shard in that Route Session.
3. Artifact distribution and route execution are separate systems.
4. Decode seam payload must be `O(hidden_size)`.
5. Prefill may be `O(sequence_length * hidden_size)`, but only in bounded chunks.
6. The tracker chooses routes; nodes do not negotiate route topology peer-to-peer.
7. Model/backend-specific cache internals stay behind backend capability reports.
8. PyTorch remains the correctness/reference backend while llama.cpp/GGUF becomes the performance backend.
9. Streaming responses are preferred when feasible; Generation Telemetry is always required.
The first public-network primitive is layer/pipeline parallelism through contiguous Shards in an Inference Route.
The full challenge register is in [technical-challenges.md](./technical-challenges.md). The open decision gates are in [decision-framework.md](./decision-framework.md).
Per-node continuous batching combines decode steps from compatible active Route Sessions. Multiple complete routes provide data parallelism.
Resolved gate:
Tensor and expert parallel collectives may later operate inside one trusted composite node or managed cluster represented as one provider. They are not public WAN routing primitives.
- Public-network Shards are layer ranges. Tensor-parallel/ring execution belongs inside a trusted node, colocated pod, or future composite node abstraction, not as the v1 public routing primitive.
- Hot KV State is local to each route node for the Shard it serves. Cache servers may store Prefix Snapshots, but they are not part of the per-token decode path.
- Distributed Route Session and Hot KV State semantics will be proven in the PyTorch route before llama.cpp/GGUF is extended for layer-boundary execution.
- Streaming responses are preferred when feasible. Realtime Generation Telemetry is required so clients can see phase, generated token count, and tokens/sec even during prefill or non-streaming fallback paths.
- llama.cpp/GGUF work targets upstreamable `libllama`/ggml hooks. A prototype fork is acceptable for exploration, but a permanent fork is not the plan.
- Model targeting is two-tiered: use a small llama.cpp-supported GGUF model for the first protocol smoke test, then use `deepseek-ai/DeepSeek-V4-Flash` as the first serious large-model target. GLM-5.2 and Ornith remain later support audits.
- Alpha fails Route Sessions on route-node loss instead of attempting automatic route repair. Repair requires compatible Prefix Snapshots and is a later capability.
- v1 activation transfer stays on binary HTTP as defined by ADR-0008. QUIC/WebRTC/custom transport can be introduced later behind the same activation protocol.
### Transport
## Non-Goals
Use gRPC over HTTP/2 with Protocol Buffers for the native Python/C++ Shard data plane.
- Do not put remote cache servers in the per-token hot KV path.
- Do not require every node to hold the full model.
- Do not fork llama.cpp long-term if upstream APIs can support the needed layer-boundary hooks.
- Do not target GLM-5.2 or Ornith first; prove the route/KV protocol on a simpler well-supported GGUF model, then target DeepSeek-V4-Flash as the first serious large model.
- One long-lived bidirectional stream per Route Session Activation Seam.
- Deadlines, cancellation, flow control, TLS/authentication hooks, structured status, and generated schemas.
- Bounded chunks for prefill and a small decode fast path.
- Existing relay infrastructure may carry the same versioned protobuf frames as opaque binary when direct connectivity is unavailable.
- OpenAI client APIs remain HTTP/SSE; existing Tracker APIs remain unchanged.
## Options Considered
The boundary payload is a versioned named-tensor bundle because architecture boundaries may require more than one tensor.
### A. Keep PyTorch-only distributed inference
### vLLM
Pros:
Do not fork vLLM for public distributed Shards and do not transplant PagedAttention, Torch process groups, or the vLLM GGUF plugin into the llama.cpp worker.
- Easy access to new Hugging Face architectures.
- Transformers has mature single-process KV semantics.
- Existing code already loads shards.
Allow unmodified vLLM as an optional whole-model backend or managed TP/PP/EP cluster represented as one logical provider.
Cons:
Adapt only small control-plane concepts:
- CPU inference is much slower than llama.cpp/GGUF.
- Current distributed path bypasses `generate()` and disables cache.
- Quantized GGUF ecosystem and LM Studio users are outside the runtime.
- Named intermediate bundles.
- Continuous batching and request ownership.
- Versioned cache-transfer compatibility fingerprints.
- Explicit transfer failure/abort lifecycle.
- Load telemetry and fair tie-breaking.
### B. Use llama.cpp only as a full local model backend
### Benchmark gate
Pros:
GGUF performance is a hypothesis. Before expensive native work, compare the current Transformers/safetensors recipe with whole-model llama.cpp on controlled model, hardware, prompt, context, output, sampling, concurrency, memory, and quality lanes.
- Quick performance win for nodes with enough RAM/VRAM.
- Minimal coordination with distributed protocol.
Later distributed release gates use thresholds locked before implementation results are known. The native track stops if llama.cpp/GGUF offers neither a meaningful performance benefit nor a meaningful model-fit benefit at useful speed.
Cons:
### Concurrency
- Does not unlock 397B/753B-class models for ordinary nodes.
- Does not solve marketplace layer routing.
A native worker must isolate `(Route Session ID, route epoch)` through a llama sequence or bounded context and must not serialize all generations behind one global serving sequence.
### C. Distributed GGUF with per-shard local KV (chosen)
The node admits sessions against weight/KV/scratch budgets, batches compatible decode steps, prevents prefill starvation, applies backpressure, and exposes queue/batch/KV telemetry.
Pros:
### Architecture certification
- Aligns with torrent artifact distribution.
- Avoids root streaming weights to workers.
- Uses llama.cpp/GGUF performance where supported.
- Compatible with public node rewards by layer/work contribution.
- Scales KV memory by layer range.
Dense Llama-family is first. Qwen3/Qwen3-MoE is a separate explicit adapter. Every architecture/backend/recipe remains registered-but-dark until a real distributed forward, parity test, concurrency test, and capability admission pass.
Cons:
## Alternatives rejected
- Requires new runtime APIs around layer-boundary hidden states and per-session KV.
- Requires model-specific cache metadata for DSA/MLA/hybrid attention.
- Harder to debug than single-process `generate()`.
### Fork vLLM for the public mesh
### D. Centralized KV cache servers
Rejected because extracting its PP/TP/EP stages requires replacing static process groups, rank lifecycle, scheduler, request ownership, cache layout, failure behavior, and hardware assumptions. This would create a large difficult fork while discarding much of vLLM's core architecture.
Pros:
### llama.cpp RPC as the public protocol
- Easier apparent session failover.
- Central accounting of active cache.
Rejected because it exposes coordinator-owned raw GGML devices, not independent Shards. Its trust, security, failure, cache, and per-node accounting model is unsuitable for arbitrary volunteer nodes.
Cons:
### Adopt Nakshatra or prima.cpp wholesale
- Puts remote storage in the per-token hot path.
- Adds bandwidth and latency at the worst possible point.
- Creates consistency and privacy problems.
Rejected because their repositories, build reproducibility, session/concurrency semantics, architecture coverage, protocol identity, and control planes do not satisfy the project contract. Their partial-loading and boundary work remains valuable evidence.
Rejected for hot decode. Accepted only for cold prefix snapshots and failover checkpoints.
### Build a custom GGUF engine
Rejected because llama.cpp already provides the parser, kernels, architecture graphs, KV, tokenizer, and heterogeneous backends. Reimplementing these would spread effort and increase correctness risk.
### Invent a custom transport
Rejected. gRPC/HTTP2 already provides mature streaming, flow control, deadlines, cancellation, TLS, and cross-language schema generation.
## Consequences
- ADR-0001 should eventually be amended: PyTorch remains valid, but llama.cpp/GGUF becomes a first-class backend.
- The activation protocol must split prefill and decode explicitly.
- Session IDs must be stable across the full request. The current fresh UUID-per-hop-call behavior must change.
- Backends must report cache budget and cache compatibility.
- Tracker route selection must include disk, memory pressure, cache warmth, and network latency.
- Billing can be based on layer work, prefill tokens, decode tokens, and observed route participation.
- Client UX should stream token deltas when feasible and must include route-session progress telemetry even when token deltas are not streamed.
- The critical path contains Meshnet, one standalone worker, and one small pinned llama.cpp patch stack.
- Transformers/safetensors remains the correctness reference and fallback for unsupported architectures.
- Whole-model llama.cpp and vLLM managed clusters remain useful optional provider types.
- The first milestone emphasizes controlled benchmark, parity, concurrent KV, and real two-machine evidence rather than a large-model demo.
- Upstream collaboration with llama.cpp targets generic local hooks only; the project remains able to ship a narrow pinned fork if upstream acceptance takes time.
- QUIC, public tensor parallelism, disaggregated prefill, speculative decode, route repair, and KV migration remain deferred until the core route passes release gates.
## Required Runtime Capabilities
## Verification gates
PyTorch path:
- manual layer calls with `past_key_values` / model-specific cache object
- per-shard session cache store
- prefill chunk append
- decode step append
- stable session lifecycle endpoints
llama.cpp/GGUF path:
- full local GGUF serving
- layer/tensor map extraction from GGUF
- optional partial layer loading or mmap-backed selected execution
- inbound hidden-state execution from arbitrary start layer
- outbound hidden-state return at stop layer
- per-session KV ownership for loaded layers
- cache budget/compatibility introspection
- GLM-5.2 DSA support when upstream/runtime supports it
## Implementation Plan
1. Add full-model `LlamaCppBackend` using `llama-server` or `libllama`.
2. Implement distributed KV in the PyTorch path to prove semantics.
3. Add session lifecycle and prefill/decode wire protocol.
4. Add model artifact manifest and torrent seeding metadata.
5. Prototype localhost two-process llama.cpp layer boundary execution.
6. Generalize to network route.
7. Bring in GLM-5.2/Ornith once backend support and cache accounting are verified.
## Acceptance Criteria
- A two-node localhost route can prefill once and decode N tokens without recomputing the full prompt.
- Seam payload during decode is `O(hidden_size)`, not `O(sequence_length * hidden_size)`.
- Per-node KV memory grows with owned layer count and context length.
- Route loss during alpha fails cleanly with explicit reason.
- Full local GGUF backend outperforms PyTorch CPU on a supported model.
- Artifact manifest can identify exactly which files/chunks a node must seed for its advertised layer range.
1. Controlled safetensors-versus-GGUF performance contract.
2. Two-process local range parity.
3. Four-session concurrent KV isolation.
4. Real two-machine execution using both Shards.
5. End-to-end performance/fit advantage over the current distributed route.
6. Separate Qwen3-family architecture certification.

309
.scratch/distributed-gguf-runtime/PRD.md
View File

@@ -1,83 +1,252 @@
# PRD: Distributed GGUF Runtime
# PRD: Performant Concurrent Distributed GGUF Runtime
## Summary
## Overview
Build a distributed inference runtime that can serve large, quality-first open models by combining torrent-style model artifact distribution with sticky multi-node Inference Routes and per-shard local Hot KV State.
Build one lean native GGUF execution path that lets an Inference Route combine consumer machines to serve models larger than any one node can hold. Reuse the existing Meshnet control plane and llama.cpp/GGML execution engine. Adopt gRPC/HTTP2 and Protocol Buffers for the native Shard worker data plane rather than inventing a transport.
The first runtime proof uses the existing PyTorch route because it exposes model internals and cache semantics more directly. GGUF/llama.cpp becomes the performance path after the route-session contract is proven.
The program is benchmark-gated. GGUF is not assumed faster merely because it is quantized or uses a different file format. The first story compares the current Transformers/safetensors backend against whole-model llama.cpp on controlled model/hardware/quality lanes and locks a performance contract. Native distributed work proceeds only when GGUF provides a meaningful speed or fit benefit.
## Goals
- Eliminate full-prompt recompute in distributed decode.
- Keep decode activation seams proportional to `hidden_size`, not `context_length * hidden_size`.
- Keep Hot KV State local to the node serving the relevant Shard.
- Stream token deltas when feasible and always expose Generation Telemetry.
- Add a local full-model GGUF backend for immediate CPU performance wins.
- Define Model Artifact manifests so nodes can verify, seed, and advertise artifacts without depending on Hugging Face at request time.
- Prototype an upstreamable llama.cpp/libllama layer-boundary API.
- Use DeepSeek-V4-Flash as the first serious large-model target after smaller protocol smoke tests.
- Execute one GGUF model across independently addressable contiguous Shards.
- Retain Hot KV State locally for each Shard and isolate concurrent Route Sessions.
- Batch compatible decode steps across active sessions for aggregate throughput.
- Use consumer CPU, AMD, NVIDIA, Vulkan, Metal, and mixed routes only where a real certified forward passes.
- Beat the current distributed safetensors route under a controlled performance contract or enable a larger otherwise-unroutable model at useful measured speed.
- Keep the critical path to Meshnet plus a small pinned llama.cpp fork and standalone C++ worker.
- Produce narrow upstream collaboration material for llama.cpp without placing Meshnet networking or economics inside upstream.
## Quality Gates
Every story must:
- Run its targeted `pytest` tests.
- Run `python -m compileall packages tests` for Python changes.
- Run `git diff --check`.
- Keep default tests deterministic, model-download-free, API-credit-free, and GPU-free.
- Preserve existing Transformers/safetensors behavior unless the story explicitly changes a versioned compatibility contract.
Stories touching the native worker must also:
- Build the pinned C++ target with CMake.
- Run focused C++/protocol tests through CTest or the documented equivalent.
- Verify the llama.cpp patch stack applies cleanly to the exact pinned commit.
Real-model/hardware stories must:
- Require `MESHNET_ENABLE_REAL_INFERENCE_TESTS=1`.
- Use the machine-specific mounted-drive model path and the certified runtime environment; never place model artifacts under `/home`.
- Record exact model revision, artifact hash, runtime recipe, hardware, driver/backend, commands, raw JSON metrics, and output-quality result.
- Label synthetic tests as unit coverage rather than distributed acceptance.
Before a story is marked complete, run the full deterministic `pytest -q` suite or record the exact pre-existing unrelated failure with a clean-tree reproduction.
## User Stories
### DGR-001: Lock the safetensors-versus-GGUF performance contract
**Description:** As a runtime engineer, I need a controlled baseline so that GGUF work proceeds from measured speed, memory, and quality rather than reputation.
**Acceptance Criteria:**
- [ ] Benchmark the same model architecture/revision, machine, prompts, context lengths, output lengths, sampling policy, and concurrency across the current Transformers/safetensors recipe and whole-model llama.cpp recipes.
- [ ] Separate correctness/quality lanes from quantized performance/fit lanes instead of claiming BF16 and Q4 are numerically equivalent.
- [ ] Report TTFT, prefill tok/s, decode tok/s, p50/p95 latency, aggregate throughput, RSS, VRAM, artifact size, failures, and output drift in machine-readable JSON.
- [ ] Add concurrency levels 1 and 4 where memory permits.
- [ ] Write a versioned performance contract consumed by later release gates, including an explicit stop condition when llama.cpp/GGUF has no meaningful speed or fit benefit.
### DGR-002: Adopt the versioned gRPC Shard protocol
**Description:** As a node developer, I need a battle-proven streaming protocol so that Python and C++ Shards communicate without a custom socket protocol.
**Acceptance Criteria:**
- [ ] Add a Protocol Buffers schema for capability, health, session stream, release, and cancellation operations.
- [ ] Define one long-lived bidirectional gRPC stream per Route Session Activation Seam with deadlines, cancellation, flow control, and structured errors.
- [ ] Define bounded chunking for prefill and a small decode fast path.
- [ ] Carry schema version, request/work ID, Route Session ID, route epoch, artifact/recipe fingerprint, Shard range/effective start, phase, position, idempotency step, cache expectation, compression, and checksum.
- [ ] Define a versioned named-tensor bundle with per-tensor name, shape, dtype, byte order, and payload fragments.
- [ ] Add generated-schema round-trip and compatibility tests in Python and C++.
### DGR-003: Define exact Artifact and runtime recipe identity
**Description:** As the Tracker, I need exact compatibility identity so that only numerically and operationally compatible Shards form an Inference Route.
**Acceptance Criteria:**
- [ ] Separate weight quantization, activation dtype, compute dtype, KV dtype/layout, tokenizer revision, architecture adapter, backend, and runtime version.
- [ ] Bind derivative or split artifacts to an exact source Model Artifact hash and Shard range.
- [ ] Produce a stable compatibility fingerprint used by capability admission and the gRPC handshake.
- [ ] Fail closed on mismatched artifact, tokenizer, architecture, range, boundary schema, activation recipe, or cache layout.
- [ ] Keep unsupported recipes registered-but-dark until a real distributed forward certifies them.
### DGR-004: Create the reproducible pinned llama.cpp patch stack
**Description:** As a maintainer, I need a small auditable fork boundary so that upstream updates do not turn the runtime into an unmaintainable stitched codebase.
**Acceptance Criteria:**
- [ ] Pin one exact llama.cpp commit through a reproducible source dependency mechanism.
- [ ] Store a numbered minimal patch stack separately from Meshnet networking code.
- [ ] Add a build script that applies/checks patches and builds the standalone worker without manual source copying.
- [ ] Record upstream file/ABI assumptions and fail clearly when the pin changes.
- [ ] Preserve upstream license and attribution notices.
- [ ] Add a clean rebuild smoke test that does not download a model.
### DGR-005: Implement dense-Llama range-aware GGUF ownership
**Description:** As a node, I need to map only my assigned dense-Llama Shard so that aggregate consumer memory can hold a model larger than one node.
**Acceptance Criteria:**
- [ ] Register and allocate only `blk.N.*` tensors in the assigned range.
- [ ] Load embeddings only for the head and final norm/LM head only for the tail, including tied embeddings.
- [ ] Prefer range-aware mapping from one exact source GGUF; if derivative sub-GGUFs are used temporarily, verify source/slice hashes and avoid claiming final artifact semantics.
- [ ] Report authoritative loaded range and endpoint ownership from the model, not operator CLI claims.
- [ ] Demonstrate mapped/resident memory scales with owned tensors rather than full model size.
### DGR-006: Implement architecture-defined boundary input/output
**Description:** As a Shard, I need to consume and emit the correct transformer boundary state so that disjoint processes reproduce whole-model execution.
**Acceptance Criteria:**
- [ ] Head accepts token IDs and owns token embedding.
- [ ] Middle/tail bypass token embedding and accept the named boundary bundle.
- [ ] Non-tail emits the unnormalized architecture-defined residual/boundary before final norm/head and before tail-only row pruning.
- [ ] Tail emits logits or token output through an explicit sampling contract.
- [ ] Dense-Llama whole-model versus two-range prefill and greedy-decode parity passes the documented tolerance.
- [ ] The adapter interface fails closed for uncertified architectures.
### DGR-007: Add isolated concurrent local Hot KV State
**Description:** As a client, I need concurrent Route Sessions to retain independent per-Shard cache so that one request cannot clear or corrupt another.
**Acceptance Criteria:**
- [ ] Map `(Route Session ID, route epoch)` to an isolated llama sequence or bounded context.
- [ ] Allocate KV only for owned layers.
- [ ] Support prefill append, decode append, truncate, release, TTL/LRU eviction, and explicit cache-miss response.
- [ ] Reject stale epochs and incompatible cache recipes.
- [ ] At least four concurrent sessions on a small model complete without token or KV cross-talk.
- [ ] Cancellation/release of one session leaves other sessions intact and memory returns to the configured budget.
### DGR-008: Build the standalone C++ gRPC Shard worker
**Description:** As a node runtime, I need one supervised native process so that llama.cpp internals remain behind a stable project-owned protocol.
**Acceptance Criteria:**
- [ ] Worker exposes capability, health, session stream, release, cancellation, and metrics services from DGR-002.
- [ ] Worker loads one exact Artifact/recipe/Shard identity and refuses mismatched requests.
- [ ] Streaming path enforces bounded messages, flow control, deadlines, idempotency, and independent session cancellation.
- [ ] Worker does not expose raw llama.cpp RPC or arbitrary GGML graph execution.
- [ ] Graceful shutdown releases sessions; crash behavior is bounded and observable.
- [ ] Python integration tests run against a fake model mode without model downloads.
### DGR-009: Integrate the native worker with Meshnet
**Description:** As the existing node service, I need a GGUF Shard backend adapter so that the Tracker, relay, billing, telemetry, and capability admission remain the sole control plane.
**Acceptance Criteria:**
- [ ] Implement the existing model-backend surface without changing Transformers behavior.
- [ ] Registration carries exact validated GGUF recipe, Shard, backend and concurrency/KV capacity.
- [ ] Tracker forms only complete compatible routes and keeps uncertified recipes dark.
- [ ] Direct routes use gRPC streams; relayed routes carry the same versioned protobuf frames as opaque binary through the existing relay seam.
- [ ] Existing request/work IDs, cancellation, Generation Telemetry, billing, and per-node attribution remain correlated.
- [ ] No vLLM, Nakshatra, prima.cpp, or custom-engine control plane becomes a core dependency.
### DGR-010: Pass local real-model two-process acceptance
**Description:** As a release engineer, I need real local distributed parity before involving network variability.
**Acceptance Criteria:**
- [ ] Two local worker processes open disjoint dense-Llama ranges from the certified Artifact.
- [ ] Prefill and at least 32 greedy decode tokens match whole-model llama.cpp within the certified tolerance.
- [ ] Each worker retains only its own tensors and Hot KV State.
- [ ] Four concurrent Route Sessions pass isolation and cleanup checks.
- [ ] Report TTFT, prefill/decode throughput, seam bytes/latency, worker RSS/VRAM, KV memory, batch size, and queue time.
- [ ] Killing one worker produces a bounded structured failure rather than a deadlock.
### DGR-011: Pass a real heterogeneous two-machine route
**Description:** As a consumer-hardware operator, I need two physical machines to execute one GGUF model so that the distributed claim is real.
**Acceptance Criteria:**
- [ ] Tracker selects two physical nodes with disjoint Shards and one exact certified recipe/compatibility class.
- [ ] Actual CPU/GPU execution occurs on both nodes; synthetic workers do not satisfy acceptance.
- [ ] Prefill/decode, concurrent-session isolation, telemetry, cancellation, and cleanup pass over the real transport/relay path.
- [ ] Exact hardware, network, backend, model hash, route, commands, and raw metrics are recorded.
- [ ] A model or recipe larger than one participating node's admitted memory is exercised when available.
- [ ] Output drift is measured and incompatible mixed backends fail closed.
### DGR-012: Implement continuous batching and bounded admission
**Description:** As a node operator, I need active sessions batched safely so that concurrency increases aggregate throughput rather than serializing every request.
**Acceptance Criteria:**
- [ ] Node scheduler admits sessions against weight, KV, scratch, and queue budgets.
- [ ] Compatible decode steps from multiple sessions form llama.cpp batches while preserving per-session positions and outputs.
- [ ] Prefill does not starve decode; scheduling policy and bounds are explicit.
- [ ] Backpressure prevents unbounded queued activations or KV growth.
- [ ] Capability telemetry reports active sessions, queue depth, batch occupancy, KV pressure, prefill/decode rates, and rejected admissions.
- [ ] Concurrency 1/2/4/8 benchmark identifies saturation and shows no cross-session corruption.
### DGR-013: Harden failure, cancellation, and restart semantics
**Description:** As a client, I need failures to be bounded and explicit so that distributed speed does not come with hanging or corrupted generations.
**Acceptance Criteria:**
- [ ] Deadlines and heartbeat/health loss terminate blocked stream operations.
- [ ] Cancellation propagates across every Shard and releases local KV and queued buffers.
- [ ] Duplicate steps are idempotent; uncertain mutations are never replayed silently.
- [ ] Alpha failover restarts from token zero on a newly compatible route rather than importing unverified KV.
- [ ] Worker death, stream reset, malformed bundle, stale epoch, and cache miss tests pass.
- [ ] Billing/work records distinguish completed, cancelled, failed, and unverified work.
### DGR-014: Enforce the GGUF-versus-safetensors release gate
**Description:** As the product owner, I need an end-to-end comparison so that the native runtime ships only if it advances model access or performance.
**Acceptance Criteria:**
- [ ] Run current distributed safetensors and distributed GGUF routes on the same certified model/hardware/network scenario where technically comparable.
- [ ] Report quality, TTFT, prefill/decode throughput, aggregate concurrency throughput, p95 latency, seam cost, memory, KV pressure, failures, and cleanup.
- [ ] Evaluate against the DGR-001 performance contract without changing thresholds after seeing results.
- [ ] Ship recommendation is one of: promote GGUF, optimize a measured bottleneck with a new bounded task, or stop the native track.
- [ ] Results clearly separate quantization gains from transport/runtime gains.
### DGR-015: Add and certify a Qwen3/Qwen3-MoE adapter
**Description:** As a client seeking top models, I need a separately certified MoE-capable architecture after the dense runtime proves stable.
**Acceptance Criteria:**
- [ ] Implement explicit tensor ownership, router/top-k, expert/shared-expert, Q/K normalization, boundary bundle, and cache semantics for the selected Qwen3 family recipe.
- [ ] Do not reuse the dense-Llama adapter through unchecked name substitutions.
- [ ] Whole-model versus distributed prefill/decode parity passes the architecture-specific tolerance.
- [ ] Expert memory ownership and communication are measured.
- [ ] Real consumer-hardware acceptance and capability admission pass before the recipe becomes routable.
### DGR-016: Produce the upstream llama.cpp collaboration package
**Description:** As a maintainer, I need narrow upstreamable proposals so that our patch burden can shrink without asking llama.cpp to own Meshnet networking.
**Acceptance Criteria:**
- [ ] Separate generic llama.cpp hooks from Meshnet protocol/control-plane code.
- [ ] Prepare minimal reproducible examples and tests for range-aware loading, boundary input/output, and layer-filtered KV.
- [ ] Compare the proposal with Nakshatra and prima.cpp evidence and explain why the API is generally useful.
- [ ] Preserve one scoped commit/patch per concern against the exact upstream pin.
- [ ] Produce an outreach document suitable for Georgi/llama.cpp maintainers; actual sending remains a human action.
## Functional Requirements
1. The public distributed primitive is an ordered Inference Route of contiguous Shards.
2. The native runtime uses llama.cpp/GGML; vLLM remains optional as a complete managed provider.
3. Native worker communication uses gRPC/HTTP2 and Protocol Buffers with one stable stream per Route Session Activation Seam.
4. Artifact identity, runtime recipe, boundary schema, activation dtype and cache layout must match exactly before routing.
5. Hot KV State remains local to the node serving the Shard.
6. Multiple Route Sessions must execute concurrently without shared-cache corruption.
7. Nodes batch compatible active decode steps and enforce bounded admission/backpressure.
8. Unsupported architectures and hardware recipes remain non-routable until real certification passes.
9. Default tests never download models or require GPUs; real tests are explicit and preserve artifacts off `/home`.
10. The release decision is based on measured performance, fit, quality, concurrency, and reliability relative to the safetensors baseline.
## Non-Goals
- No centralized hot KV cache in the per-token decode path.
- No automatic route repair in alpha.
- No permanent llama.cpp fork as the intended architecture.
- No GLM-5.2 or Ornith first; they remain follow-up support audits.
- No transport rewrite to QUIC/WebRTC before route/session semantics are proven.
- Forking vLLM or importing its PagedAttention/Torch distributed runtime.
- Adopting Nakshatra, prima.cpp, llama-gguf, LiGGUF, or GPUStack as the control plane.
- Public WAN tensor/expert parallel collectives.
- QUIC, WebRTC, or a custom socket protocol.
- Automatic KV migration or mid-generation route repair in the first release.
- Speculative decoding or disaggregated prefill before the core release gate.
- Supporting every GGUF architecture before dense Llama and Qwen3-family certification.
- A marketing-scale model demo that bypasses parity, concurrency, admission, or performance gates.
## Resolved Decisions
## Success Metrics
- Public-network Shards are contiguous transformer layer ranges.
- Tensor/ring parallelism belongs inside one trusted node, one colocated pod, or a future composite node abstraction.
- Hot KV State is local to route nodes; Prefix Snapshots are optional cold recovery/reuse artifacts.
- PyTorch distributed KV/session semantics are proven before llama.cpp distributed execution.
- Streaming responses are preferred; Generation Telemetry is mandatory.
- llama.cpp/GGUF work targets upstreamable `libllama`/ggml hooks.
- Alpha fails Route Sessions on route-node loss.
- v1 activation transfer stays on binary HTTP.
- A real model larger than one admitted node can execute across consumer machines when suitable hardware/artifacts are available.
- Four or more concurrent sessions complete without cross-talk; hardware-specific saturation is measured.
- Distributed GGUF passes the locked performance/fit contract against the existing safetensors route.
- Worker and Tracker recover all resources after completion, cancellation, malformed input, and node failure.
- The critical runtime remains Meshnet plus one standalone worker and a small auditable llama.cpp patch stack.
## Target User Experience
## Open Questions
A client sends an OpenAI-compatible request. The Gateway or Tracker Node accepts the request, creates a Route Session, and streams token deltas when supported. The client receives live Generation Telemetry for route phase, prefill progress, generated token count, rolling tokens/sec, route health, and failure reason.
If a route node drops in alpha, the request fails clearly. A retry starts a new Route Session from scratch.
## Runtime Shape
```text
client request
-> Gateway / Tracker Node creates Route Session
-> Tracker selects sticky Inference Route
-> prefill:
prompt chunks move through Shards
each node appends local Hot KV State
-> decode:
one-step activation moves through Shards
each node reads/appends local Hot KV State
tail returns token/logits
-> client receives streamed token deltas where possible
-> Generation Telemetry continues until complete or failed
```
## Milestones
| Milestone | Outcome | Issues |
|---|---|---|
| M1 — Session protocol proof | Stub route has stable Route Sessions, prefill/decode split, telemetry, and streaming contract | 01, 02, 03 |
| M2 — PyTorch reference route | Distributed PyTorch decode uses local per-shard cache and stops full-prompt recompute | 04 |
| M3 — Local GGUF performance path | Single-node GGUF backend serves through the node API and reports backend metadata | 05 |
| M4 — Artifact plane | Model Artifact manifest supports verification, layer mapping, and node advertisement | 06 |
| M5 — llama.cpp collaboration proof | Localhost layer-boundary prototype identifies upstreamable llama.cpp/libllama API | 07 |
| M6 — Networked GGUF route | Multi-node GGUF route uses the resolved protocol and fails cleanly on node loss | 08 |
| M7 — First large model | DeepSeek-V4-Flash support path is audited and converted into follow-up runtime tasks | 09 |
## Acceptance Criteria
- A two-node route can prefill once and decode without resending full prompt activations.
- Decode seam payload is one token/hidden-state step after prefill.
- Route Session telemetry is visible before first token and during decode.
- Streaming token deltas work where the backend supports them.
- Route-node loss produces a structured alpha failure and does not attempt unsafe repair.
- A local GGUF model can serve via the node API.
- A Model Artifact manifest can prove which Shards a node can serve.
- DeepSeek-V4-Flash has a written support recommendation: PyTorch, vLLM/SGLang, llama.cpp/GGUF, or blocked.
- Exact benchmark model and quantization lanes are selected by DGR-001 from currently supported, legally redistributable artifacts.
- Final hardware-specific concurrency and useful-speed thresholds are locked by measured baselines rather than guessed globally.
- Upstream llama.cpp acceptance is desirable but not a prerequisite for the first narrow pinned fork.

309
.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md Normal file
View File

@@ -0,0 +1,309 @@
# Ralph execution context: Performant Concurrent Distributed GGUF Runtime
Status: authoritative context for every fresh Ralph iteration
Last updated: 2026-07-13
## Mandatory startup sequence
Before changing code, every Ralph agent must:
1. Read this file completely.
2. Read the selected issue under `.scratch/distributed-gguf-runtime/issues/`.
3. Read `.scratch/distributed-gguf-runtime/ADR-0020-distributed-gguf-runtime.md` and the relevant part of `architecture.md`.
4. Read `.claude/memory/MEMORY.md` and root `CONTEXT.md` for current project vocabulary and constraints.
5. Inspect the current implementation and tests; do not assume historical scratch text describes live code.
6. Read the evidence/handoff directories for every declared dependency.
7. Inspect `git status` and preserve all pre-existing working-tree changes.
A fresh Ralph iteration has no conversational memory. These files are the context contract.
## Story sizing and interruption rule
Each story is intended to fit one focused Ralph context. Before implementation, estimate whether every acceptance criterion can be completed and verified in the current iteration.
If the story is too large, an external dependency is unavailable, or the context/provider limit prevents completion:
- Do not weaken criteria.
- Do not mark the issue done or set `passes: true`.
- Avoid leaving an unverified cross-cutting partial implementation when a smaller safe spike is possible.
- Write `evidence/<TASK-ID>/DECOMPOSITION.md` or `BLOCKED.md` with the exact blocker, current verified state, proposed child stories, dependency graph and rollback/continuation instructions.
- Stop for supervised review.
If interrupted after code changes, record every changed file, command result and unresolved invariant so the next fresh loop can verify rather than guess.
## Product objective
Build performant, concurrent distributed inference that combines consumer machines to serve top open models that exceed one node's RAM/VRAM.
A distributed demo is not success. The product must provide:
- Useful measured prefill and decode speed.
- Multiple concurrent Route Sessions.
- No KV/token cross-talk.
- Bounded memory, queues, cancellation and failures.
- Real execution on every participating node.
- A model-fit or performance advantage over the current Transformers/safetensors route.
## Critical-path architecture
```text
Existing Meshnet control plane
|
Versioned Protobuf over gRPC/HTTP2
|
Project-owned standalone C++ Shard worker
|
Small exact-commit llama.cpp patch stack
```
Meshnet remains the only control plane and owns:
- Tracker registration, Coverage Map, route selection and route epochs.
- Route Sessions and Activation Seams.
- Direct/relay routing.
- Capability admission.
- Cancellation, Generation Telemetry and backpressure.
- Billing, validation and per-node work attribution.
Do not introduce another scheduler/control plane from vLLM, Nakshatra, prima.cpp, llama-gguf, GPUStack or another project.
## Runtime decisions that are not open
1. Public-network Shards are contiguous transformer layer ranges.
2. llama.cpp/GGML is the native GGUF execution substrate.
3. The project owns a small standalone worker and a narrow pinned llama.cpp patch stack.
4. The native Shard protocol is Protocol Buffers over gRPC/HTTP2.
5. One long-lived bidirectional stream serves one Route Session Activation Seam.
6. The public activation boundary is a versioned named-tensor bundle.
7. Hot KV State remains local to the node serving the Shard.
8. `(Route Session ID, route epoch)` maps to an isolated llama sequence or bounded context.
9. Concurrency uses continuous batching of compatible active sessions inside each node.
10. Transformers/safetensors remains the correctness and performance baseline.
11. vLLM may be an optional complete managed provider and concept donor; it is not forked into public Shards.
12. Tensor/expert collectives are deferred to a trusted composite provider, not public WAN routes.
13. Unsupported architectures/backends remain registered-but-dark until real certification passes.
14. Alpha failure retries from token zero; unverified KV is never migrated silently.
15. Model artifacts must remain on mounted-drive storage and never under `/home`.
Changing one of these requires an explicit ADR update and human review, not an incidental story implementation.
## Performance discipline
GGUF performance is a hypothesis. Never write “GGUF is faster” without measurements.
DGR-001 locks controlled benchmark lanes and thresholds. DGR-014 enforces the final distributed comparison.
Always distinguish:
- Weight quantization from activation/compute/KV dtype.
- Runtime/kernel gains from quantization/model-fit gains.
- Single-request latency from aggregate concurrency throughput.
- Synthetic unit coverage from real distributed acceptance.
Required metrics where applicable:
```text
TTFT
prefill tokens/sec
decode tokens/sec
aggregate throughput
p50/p95 latency
seam bytes and latency
queue and batch occupancy
RSS and VRAM
KV pressure
output-quality drift
failures and cleanup
```
Do not weaken or move performance thresholds after seeing implementation results.
## Transport discipline
Do not invent a raw TCP protocol, new WebSocket protocol, QUIC layer or bespoke binary control format.
The `.proto` schema is the semantic contract. Direct transport uses gRPC. Existing relay infrastructure may carry the same serialized protobuf frames as opaque binary.
Protocol requirements:
- Schema/version negotiation.
- Request/work ID.
- Route Session ID and route epoch.
- Exact Model Artifact/runtime recipe fingerprint.
- Shard range and effective overlap-safe start.
- Prefill/decode/release/cancel phases.
- Position/token range and idempotency step.
- Named tensors with shape, dtype, byte order and bounded fragments.
- Compression/checksum.
- Cache expectation/result.
- Deadlines, cancellation, flow control and structured status.
Avoid per-token channel creation and unbounded unary payloads. Generated code and build tooling must be reproducible; do not require manual copying.
## Native runtime discipline
Reuse llama.cpp for GGUF, mmap, kernels, architecture graphs, tokenizer, KV, sequences and heterogeneous backends.
The project patch stack is limited to:
- Range-aware tensor registration/loading.
- Endpoint-specific embedding/final head ownership.
- Architecture-defined intermediate input/output.
- Intermediate output before final norm/head.
- Layer-filtered KV and session mapping.
Do not place Meshnet routing, transport, billing or authentication inside llama.cpp. Keep patches numbered, scoped, pinned and upstreamable.
Dense Llama-family is first. Qwen3/Qwen3-MoE is a separate adapter after the dense release gate. Do not generalize through unchecked tensor-name substitutions.
## Existing code seams to inspect first
- `packages/node/meshnet_node/model_backend.py` — backend abstraction.
- `packages/node/meshnet_node/torch_server.py` — reference ranged execution and session behavior.
- `packages/node/meshnet_node/activation_compression.py` — current activation framing/compression.
- `packages/node/meshnet_node/route_session_benchmark.py` — existing benchmark infrastructure.
- `packages/tracker/meshnet_tracker/server.py` — registration, route and proxy behavior.
- `packages/tracker/meshnet_tracker/capability.py` — fail-closed capability admission.
- `tests/test_real_model_backend.py` — real backend coverage.
- `tests/test_tracker_routing.py` — route/session behavior.
- `tests/test_tracker_capability_admission.py` — recipe admission.
- `tests/test_route_session_benchmark.py` and `tests/test_manual_route_benchmark.py` — benchmark patterns.
- `docs/adr/0008-binary-activation-wire-format.md` — existing wire compatibility.
- `docs/adr/0012-start-layer-overlapping-shards.md` — effective start semantics.
- `docs/adr/0022-sharded-per-node-kv-cache.md` — Hot KV State contract.
- `docs/adr/0023-model-agnostic-node-capability-admission.md` — certification/admission.
Do not edit generated `build/`, `__pycache__`, egg-info, Ralph logs or unrelated scratch features.
## Planned source layout
Use these paths unless current code inspection proves a better project-consistent location. If changed, document the reason in task evidence.
```text
packages/node/native/
proto/shard_runtime.proto
cmake/
llama/
UPSTREAM_COMMIT
patches/
gguf_worker/
tests/
packages/node/meshnet_node/
native_protocol/
gguf_backend.py
runtime_recipe.py
.scratch/distributed-gguf-runtime/evidence/<TASK-ID>/
README.md
commands.txt
results.json or other machine-readable evidence
```
Generated protobuf/C++ build outputs belong in build directories unless packaging explicitly requires checked-in generated Python modules. The story must document the generation command and version.
## Story output map
| Story | Required durable outputs |
|---|---|
| DGR-001 | benchmark harness/tests; `evidence/DGR-001/performance-contract.json`; raw/summary benchmark evidence |
| DGR-002 | `packages/node/native/proto/shard_runtime.proto`; reproducible Python/C++ generation/build wiring; protocol round-trip/compatibility tests; `evidence/DGR-002/` |
| DGR-003 | exact runtime-recipe/fingerprint implementation and admission tests; `evidence/DGR-003/` |
| DGR-004 | exact upstream pin, numbered patch series, reproducible fetch/apply/build smoke; `evidence/DGR-004/` |
| DGR-005 | dense-Llama range ownership loader and memory evidence; `evidence/DGR-005/` |
| DGR-006 | architecture boundary adapter/parity tests and results; `evidence/DGR-006/` |
| DGR-007 | concurrent session/KV manager, isolation/cleanup tests; `evidence/DGR-007/` |
| DGR-008 | standalone C++ gRPC worker, fake-model integration tests, lifecycle evidence; `evidence/DGR-008/` |
| DGR-009 | Meshnet backend/registration/relay integration and tests; `evidence/DGR-009/` |
| DGR-010 | real local two-process commands, raw metrics and parity report; `evidence/DGR-010/` |
| DGR-011 | two-machine configuration, commands, hardware/network manifest and raw results; `evidence/DGR-011/` |
| DGR-012 | continuous scheduler/admission implementation and 1/2/4/8 concurrency report; `evidence/DGR-012/` |
| DGR-013 | failure/cancel/restart test matrix and resource-cleanup evidence; `evidence/DGR-013/` |
| DGR-014 | immutable final comparison against DGR-001 thresholds and ship/stop recommendation; `evidence/DGR-014/` |
| DGR-015 | Qwen3-family adapter, architecture-specific parity/admission/performance evidence; `evidence/DGR-015/` |
| DGR-016 | narrow upstream patches/tests, design note and human-ready outreach package; `evidence/DGR-016/` |
## Dependency handoff rule
For every dependency listed by Ralph:
1. Confirm its `passes` state in `prd.json`.
2. Read `.scratch/distributed-gguf-runtime/evidence/<DEPENDENCY-ID>/README.md`.
3. Verify referenced source paths and commands still exist.
4. Do not repeat completed work unless verification exposes a concrete defect.
5. If dependency evidence is missing or contradictory, stop and repair the dependency instead of guessing.
## Testing and hardware rules
Default tests must be deterministic, GPU-free, model-download-free and API-credit-free.
Real model tests require:
```text
MESHNET_ENABLE_REAL_INFERENCE_TESTS=1
```
On this machine:
- Use `.venv-rocm` for real Radeon 8060S ROCm execution.
- The default Python 3.14 `.venv` is unsuitable for real ROCm inference.
- Resolve model storage through the machine-specific `.env.<hostname>` configuration.
- Never download model artifacts under `/home`.
- Real acceptance must exercise actual Tracker-routed CPU/GPU computation; synthetic workers are only unit tests.
Record exact:
- Model/revision and Artifact hash.
- Quantization and runtime recipe.
- Host/hardware/backend/driver.
- Commands and environment names without secrets.
- Raw output and metrics.
- Whether the evidence is synthetic, local-real, or multi-machine-real.
## Worktree and commit discipline
This repository may contain pre-existing changes from research or another feature.
- Inspect `git status` before editing.
- Never reset, checkout over, stash, delete or reformat unrelated changes.
- Stage only files belonging to the selected story.
- Exclude `.ralph-tui`, iteration logs, caches, generated builds, FUSE artifacts and unrelated scratch work.
- Keep one scoped commit per completed story when the supervising loop requests commits.
- Do not modify `passes` for another story.
## Mandatory finish/handoff sequence
Before emitting `<promise>COMPLETE</promise>`:
1. Verify every acceptance criterion with real command output or file evidence.
2. Run story-specific gates and repository quality gates.
3. Write `.scratch/distributed-gguf-runtime/evidence/<TASK-ID>/README.md` containing:
- Summary of changes.
- Exact files changed.
- Commands run and their real results.
- Performance/correctness evidence.
- Known limitations and deferred work.
- Compatibility or migration notes.
- Clear handoff for dependent stories.
4. Save machine-readable evidence beside it when the story produces metrics or schemas.
5. Update the source issue status to `done` only after all gates pass.
6. Preserve failures honestly. Never fabricate model, benchmark, test or hardware output.
## Authoritative references
Active decisions:
- `.scratch/distributed-gguf-runtime/README.md`
- `.scratch/distributed-gguf-runtime/implementation-strategy.md`
- `.scratch/distributed-gguf-runtime/architecture.md`
- `.scratch/distributed-gguf-runtime/ADR-0020-distributed-gguf-runtime.md`
- `.scratch/distributed-gguf-runtime/PRD.md`
- `.scratch/distributed-gguf-runtime/prd.json`
Source research:
- `docs/research/distributed-gguf-landscape.md`
- `docs/research/distributed-gguf-github-followup.md`
- `docs/research/vllm-distributed-gguf-assessment.md`
If historical notes conflict with these files, the active decisions above win.

85
.scratch/distributed-gguf-runtime/README.md
View File

@@ -1,63 +1,46 @@
# Distributed GGUF runtime — planning index
# Performant concurrent distributed GGUF runtime
Status: draft scratch package.
Status: active benchmark-gated implementation program.
Goal: make the node network capable of serving large, high-quality open models by distributing GGUF/model artifacts over a torrent-style swarm while executing inference over a sticky multi-node route with per-shard local KV cache.
## Objective
This scratch supersedes the old assumption in [ADR-0001](../../docs/adr/0001-pytorch-over-llama-cpp.md) that llama.cpp is only a single-node leaf backend. That assumption was correct for the original llama.cpp RPC shape, but the target is now different: torrent-distributed GGUF artifacts plus an explicit route/KV protocol owned by this platform, ideally developed in collaboration with upstream llama.cpp.
Serve top open models across consumer machines with useful performance and concurrent Route Sessions while keeping the runtime lean.
## Artifacts
## Critical path
| Path | Purpose |
|---|---|
| [architecture.md](./architecture.md) | Proposed runtime architecture, data flow, session state, and failure model |
| [technical-challenges.md](./technical-challenges.md) | Detailed challenge/solution register with acceptance tests |
| [decision-framework.md](./decision-framework.md) | Grilling framework for open decisions and recommended answers |
| [research-prior-art.md](./research-prior-art.md) | Prior-art notes for Petals, exo, Distributed Llama, prima.cpp, llama.cpp, DeepSeek-V4-Flash, GLM-5.2, and Ornith |
| [ADR-0020-distributed-gguf-runtime.md](./ADR-0020-distributed-gguf-runtime.md) | Draft decision record for the GGUF/llama.cpp distributed runtime |
| [PRD.md](./PRD.md) | Product/runtime requirements and acceptance criteria |
| [milestones.md](./milestones.md) | Dependency-ordered implementation milestones |
| [issues/](./issues/) | Implementation-ready tracer-bullet issue briefs |
```text
Meshnet control plane
-> versioned gRPC/Protobuf Shard protocol
-> project-owned standalone C++ worker
-> small pinned llama.cpp patch stack
```
## Decision Summary
Transformers/safetensors remains the correctness baseline. vLLM remains an optional complete managed provider and a design donor; it is not forked into the public mesh.
Adopt a hybrid runtime:
## Planning artifacts
- **Weights and artifacts**: distributed by torrent / content-addressed storage / optional CDN.
- **Hot KV cache**: local to the node that owns the corresponding layer range.
- **Prefix snapshots**: optionally persisted to cache servers for reuse, retry, and failover.
- **Active route**: sticky for one request/session.
- **Context cap**: 128K hard product limit for large models unless explicitly revised.
- **Backends**: keep PyTorch for fast model-architecture coverage and validation; add llama.cpp/GGUF as the performance path for supported models.
- **Client feedback**: stream token deltas when feasible; always expose Generation Telemetry.
- **First serious target model**: DeepSeek-V4-Flash after a smaller GGUF protocol smoke test.
- **[Mandatory Ralph context](RALPH-CONTEXT.md)** — read first in every fresh iteration
- [Task evidence contract](evidence/README.md)
- [Implementation strategy](implementation-strategy.md)
- [Current architecture](architecture.md)
- [PRD](PRD.md)
- [Ralph backlog](prd.json)
- [ADR-0020](ADR-0020-distributed-gguf-runtime.md)
- [Milestones](milestones.md)
- [Issues](issues/)
- [Distributed GGUF research](../../docs/research/distributed-gguf-landscape.md)
- [GitHub follow-up](../../docs/research/distributed-gguf-github-followup.md)
- [vLLM assessment](../../docs/research/vllm-distributed-gguf-assessment.md)
## What We Learned
## Ralph execution
- Our current full-model PyTorch path uses Transformers `generate()` and gets local KV cache.
- Our current distributed PyTorch path disables cache and recomputes the full growing sequence per token.
- The seam today carries hidden activations, not KV cache; at 128K this becomes impossible for serious models if repeated every decode token.
- The missing capability is not "send KV across the network"; it is **stable per-session local KV cache per shard**.
- GGUF distribution is solved enough at the artifact layer, but GGUF/llama.cpp needs explicit layer-boundary execution APIs for our route model.
Use supervised one-story iterations for this high-risk runtime:
## Recommended Order
```bash
ralph-tui run \
--prd .scratch/distributed-gguf-runtime/prd.json \
--agent claude --model opus \
--iterations 1 --no-tui --no-setup --verify
```
See [milestones.md](./milestones.md) for the full dependency map.
1. [01 — Route Session lifecycle](./issues/01-route-session-lifecycle.md)
2. [02 — Prefill/decode binary HTTP protocol](./issues/02-prefill-decode-binary-http.md)
3. [03 — Generation Telemetry and streaming response contract](./issues/03-generation-telemetry-and-streaming.md)
4. [04 — PyTorch distributed KV reference route](./issues/04-pytorch-distributed-kv-reference.md)
5. [05 — Local llama.cpp/GGUF backend](./issues/05-local-llamacpp-gguf-backend.md)
6. [06 — Model Artifact manifest and Shard advertisement](./issues/06-model-artifact-manifest.md)
7. [07 — llama.cpp layer-boundary prototype](./issues/07-llamacpp-layer-boundary-prototype.md)
8. [08 — Networked distributed GGUF route](./issues/08-networked-distributed-gguf-route.md)
9. [09 — DeepSeek-V4-Flash support audit](./issues/09-deepseek-v4-flash-support-audit.md)
10. [10 — GLM-5.2 and Ornith follow-up support audit](./issues/10-glm52-ornith-followup-audit.md)
## Open Questions
- Does upstream llama.cpp already expose enough internal API for arbitrary layer-range execution and hidden-state boundary I/O, or do we need an extension?
- Can GGUF split metadata be made layer/tensor semantic enough for torrent placement and partial loading?
- What is the minimum protocol needed for compressed KV formats such as GLM-5.2 DSA/MLA without exposing model-specific internals to the tracker?
- How much reliability do we need in alpha: fail request on route loss, or support route repair with KV snapshots?
Inspect the diff, run the story gates, and commit one verified story before the next iteration. Real-model stories require the explicit environment gate and mounted-drive model storage.

389
.scratch/distributed-gguf-runtime/architecture.md
View File

@@ -1,274 +1,259 @@
# Distributed GGUF Runtime Architecture
# Performant Concurrent Distributed GGUF Architecture
## Product Stance
Status: current target architecture
Last updated: 2026-07-13
The platform optimizes for access to high-quality models, not lowest latency. Latency is acceptable if the user can run models that are otherwise unavailable to them. The hard context limit for the first serious distributed runtime should be **128K tokens**. Longer context usually means the product is compensating for missing task decomposition, retrieval, or workspace summarization.
## Product invariant
## Current State
The system exists to serve high-quality models that exceed one consumer node's memory while retaining useful interactive speed and aggregate concurrency. A feature that only produces a distributed demo but is slower, globally serialized, or impossible to operate on consumer hardware is not complete.
The current node has two materially different inference paths:
## Existing control plane
- **Full local PyTorch model**: calls Hugging Face `model.generate()`, so Transformers owns autoregressive decode and local KV cache.
- **Distributed PyTorch route**: bypasses `model.generate()`, calls individual layers with `use_cache=False`, and recomputes the full growing sequence for every generated token.
Meshnet remains the only public control plane:
Current distributed data flow:
- Tracker registration, Coverage Map, route scoring and assignment.
- Contiguous Shards and overlap-safe effective starts.
- Stable Route Sessions and route epochs.
- Local per-Shard Hot KV State in the reference backend.
- Direct/relay transport, cancellation and backpressure.
- Generation Telemetry, billing, validation and per-node attribution.
- Model-agnostic capability admission.
No external engine replaces these responsibilities.
## Runtime topology
```text
client request
-> head node formats prompt
-> for each output token:
head tokenizes full current text
head runs early layers over all tokens
head sends full activation [batch, sequence, hidden] to next node
middle nodes run their layers over all tokens
tail returns one decoded token string
head appends token to text
OpenAI-compatible client
|
Gateway / Tracker Node
|
ordered Inference Route
|
+-- head Shard: tokenizer/embedding + early layers
| local weights and Hot KV State
|
+-- middle Shard(s): architecture boundary + owned layers
| local weights and Hot KV State
|
+-- tail Shard: final layers + norm/head/sampling
local weights and Hot KV State
```
This is correct for small demos but not viable for large models. For GLM-5.2, a single 128K seam activation is roughly:
Weights never move in the per-request hot path. Every node opens and verifies its local Model Artifact before becoming routable.
## Primary execution substrate
```text
128K tokens * hidden_size 6144 * 2 bytes ~= 1.5 GiB per hop
project-owned C++ Shard worker
|
small exact-commit llama.cpp patch stack
|
GGUF mmap, quantized kernels, architecture graphs,
KV/sequence operations, CPU/CUDA/HIP/Vulkan/Metal backends
```
Sending that every output token is the bottleneck.
The patch stack adds only the missing local execution seam:
## Target State
1. Range-aware tensor registration/loading.
2. Endpoint-specific embedding and final head ownership.
3. Architecture-defined intermediate input.
4. Architecture-defined pre-tail boundary output.
5. Layer-filtered KV and external session mapping.
Target distributed data flow:
The worker owns protocol translation and process lifecycle. llama.cpp never receives Tracker, relay, billing or volunteer-network code.
## Shard data plane
Use Protocol Buffers and gRPC over HTTP/2.
### Service shape
- Unary capability and health.
- Bidirectional Route Session stream.
- Explicit release and cancellation.
- Metrics suitable for capability admission and route scoring.
### Session stream
One long-lived stream represents one Route Session Activation Seam. It amortizes connection setup and inherits HTTP/2 flow control. Every message carries enough identity to reject stale or incompatible work.
```text
client request
-> tracker selects route and pins session
-> head node creates session_id
-> prefill:
prompt is chunked
each shard computes its layer range
each shard appends local KV/state for its own layers
activations cross only layer seams
-> decode loop:
head sends one new token / one-step hidden state
each shard reads local KV/state for session_id
each shard appends one step to local KV/state
only one-step activation crosses seams
tail returns logits/token
schema version
request/work id
Route Session id
route epoch
Model Artifact hash
runtime recipe fingerprint
Shard begin/end and effective start
prefill/decode/release/cancel phase
position and token range
idempotency step id
cache expectation/result
named tensor bundle
compression/checksum
```
The KV cache remains local to the node that computed it. It is not sent to the next node and not read from a remote cache server during every decode step.
Prefill tensors are split into bounded ordered frames. Decode messages carry one-step architecture boundary bundles and remain small.
## Client Feedback
Direct nodes use gRPC. Nodes requiring the existing relay carry the same protobuf frames as opaque binary through the relay session. This preserves one semantic protocol instead of maintaining separate direct and relay payload contracts.
Streaming responses are desirable when the backend and client transport support them. The product should stream token deltas when possible, and it must always provide realtime Generation Telemetry while the route is working.
## Architecture boundary
The fallback behavior is a non-streaming final answer plus live telemetry. That fallback is acceptable for early route proofs or models/backends that cannot expose clean token deltas yet, but the preferred client experience is streamed output plus telemetry.
Minimum client-visible telemetry:
- route/session accepted
- selected model and quantization
- prefill phase started/completed
- decode phase started
- generated token count
- rolling tokens per second
- route health or retry/failure reason
- estimated billing units when available
Implementation options:
- Server-Sent Events or WebSocket for realtime progress
- polling endpoint for simple clients
- OpenAI-compatible streaming for clients that require token deltas
This means "no token streaming" is acceptable only as a fallback. "Silent wait for minutes" is not acceptable.
## Artifact Plane
Artifact distribution is separate from execution.
The public boundary is a versioned named-tensor bundle:
```text
model publisher
-> produces model manifest
-> creates GGUF / safetensors / tokenizer artifacts
-> content-addresses every file/chunk
-> publishes torrent/magnet + HTTP fallback metadata
node
-> chooses model/layer range
-> downloads needed files/chunks
-> verifies hash
-> advertises availability to tracker
bundle schema/version
architecture adapter and boundary point
named tensors
per-tensor shape, dtype and byte order
payload fragments
compression/checksum
```
Required manifest fields:
Dense Llama may use one residual tensor. Other adapters may require more. vLLM's Llama and Qwen3-MoE PP paths demonstrate a boundary with both `hidden_states` and `residual`; therefore the generic protocol must not assume one anonymous tensor.
- model id and version
- upstream source repo and revision
- license
- architecture name
- tokenizer files and hashes
- quantization
- tensor-to-layer map
- file/chunk hashes
- optional GGUF split files
- supported runtime backends
- context cap
- KV/cache format descriptor
Only the head owns token embedding. Only the tail owns final normalization, LM head and sampling. Middle Shards exchange the architecture-defined pre-tail boundary, not final normalized embeddings.
## Execution Plane
The tracker selects routes using layer coverage and observed performance:
## Hot KV State and concurrency
```text
route = [
head node: embeddings + layers 0..k
middle nodes: contiguous layer ranges
tail node: final layers + norm + lm_head
]
(Route Session id, route epoch)
-> local llama sequence or bounded context
-> KV for owned layers only
-> lease, memory accounting and lifecycle
```
Route selection inputs:
Required operations:
- model id/version/quantization
- layer coverage
- node hardware
- measured prefill throughput
- measured decode throughput
- queue depth
- latency to neighboring nodes
- cache warmth for the requested prefix/session
- reliability/reputation
- Prefill append.
- Decode append.
- Truncate after rejected speculative positions if later enabled.
- Explicit release.
- TTL/LRU eviction.
- Cache-miss response.
- Stale-epoch rejection.
The route is sticky for the request/session. A new route means either a fresh prefill or restoring compatible KV snapshots.
A node must not clear global KV on a new stream or serialize all requests behind one logical serving sequence.
## KV Cache Ownership
## Continuous batching
KV/state ownership is by layer range:
Autoregressive dependencies remain sequential inside one Route Session. Aggregate throughput comes from batching compatible decode steps across active sessions:
```text
session_id = request scoped id
node A owns layers 0..15 KV for session_id
node B owns layers 16..31 KV for session_id
node C owns layers 32..77 KV for session_id
time 0: session A token 1 + session B token 8 + session C token 3
-> one llama batch for this Shard
time 1: next ready positions from active sessions
-> next llama batch
```
The tracker does not own hot KV. It may know which nodes hold active KV for session accounting and failure handling.
The node scheduler:
Cache servers may store:
- Admits work against weight, KV, scratch and queue budgets.
- Keeps per-session token positions and outputs separate.
- Prevents long prefill from starving decode.
- Applies bounded backpressure.
- Reports active sessions, queue depth, batch occupancy, KV pressure and throughput.
- prompt-prefix snapshots
- session checkpoints for retry
- cold reusable context blocks
- audit samples
The initial deterministic gate is four concurrent sessions on a small model without cross-talk. Hardware-specific limits are measured and advertised through capability admission.
Cache servers must not be in the per-token hot loop unless colocated with the compute node.
## Parallelism boundaries
## 128K KV Budget
| Mechanism | First-runtime use |
|---|---|
| Layer/pipeline parallelism | Public Inference Route across contiguous Shards |
| Continuous batching | Inside every node across active Route Sessions |
| Data parallelism | Multiple complete routes for independent requests |
| Tensor parallelism | Deferred to a trusted composite node/managed cluster |
| Expert parallelism | Deferred to a trusted composite node/managed cluster |
| Disaggregated prefill | Deferred until core route performance passes |
| Speculative decoding | Deferred optimization |
GLM-5.2 compressed DSA/MLA-style estimate from config:
Public WAN tensor/expert collectives are rejected for the first runtime because their per-layer communication and static rank assumptions conflict with heterogeneous volunteer nodes.
```text
layers = 78
kv_lora_rank = 512
qk_rope_head_dim = 64
dtype = bf16 = 2 bytes
context = 128K
## Optional providers
per_token ~= 78 * (512 + 64) * 2 = 89,856 bytes ~= 87.75 KiB
128K total ~= 10.7 GiB
per layer ~= 137 MiB
```
### Transformers/safetensors
This is feasible when sharded:
Remains:
| Layer count | Approx active KV at 128K |
|---:|---:|
| 1 | 137 MiB |
| 10 | 1.37 GiB |
| 20 | 2.75 GiB |
| 78 | 10.7 GiB |
- Correctness/reference backend.
- Fallback for unsupported architectures.
- Baseline for performance and output quality.
The exact runtime value depends on implementation and cache quantization, but the order of magnitude is acceptable.
### vLLM
## Protocol Sketch
May run unmodified as a complete model or managed TP/PP/EP cluster represented as one logical provider. Its internal ranks are not independently routed or rewarded.
### Prefill
Borrow only concepts such as named bundles, continuous batching, typed compatibility fingerprints, explicit transfer lifecycle and load telemetry.
```http
POST /v1/sessions/{session_id}/prefill
Content-Type: application/octet-stream
X-Meshnet-Model: zai-org/GLM-5.2
X-Meshnet-Route-Id: ...
X-Meshnet-Token-Range: 0-2047
X-Meshnet-Shape: 1,2048,6144
X-Meshnet-Dtype: bfloat16
### Whole-model llama.cpp
<activation bytes>
```
Provides a local proxy backend, correctness oracle and performance baseline. It is not the native distributed milestone.
The receiver:
## Artifact and recipe compatibility
- validates route/session
- runs assigned layer range for that chunk
- appends local KV/state
- forwards resulting activation to next hop
A routable recipe identifies separately:
### Decode
- Source Model Artifact hash and optional derivative/slice hash.
- Architecture and adapter version.
- Tokenizer revision and vocabulary.
- Weight quantization.
- Activation interchange dtype/schema.
- Backend compute dtype and backend implementation.
- KV dtype/layout.
- RoPE/context parameters.
- llama.cpp commit and project patch version.
- Shard range and endpoint ownership.
```http
POST /v1/sessions/{session_id}/decode-step
Content-Type: application/octet-stream
X-Meshnet-Model: zai-org/GLM-5.2
X-Meshnet-Position: 131072
X-Meshnet-Shape: 1,1,6144
X-Meshnet-Dtype: bfloat16
Compatibility fails closed. Similar quantization labels or model names are not enough.
<one-step activation bytes>
```
## Admission and failure
The receiver:
A recipe becomes routable only after a real local and distributed forward passes. Synthetic tests remain unit coverage.
- loads local KV/state by `session_id`
- runs one decode step for assigned layers
- appends one token position to local KV/state
- forwards one-step activation
Alpha failure behavior:
## GGUF / llama.cpp Integration
- Deadline or node loss cancels the Route Session.
- Every node releases KV and queued buffers.
- Uncertain mutations are not replayed silently.
- Retry starts from token zero on a newly compatible route.
- No cross-node KV import is trusted until a later signed/compatible snapshot protocol exists.
The target llama.cpp integration needs more than `llama-server`.
## Performance release contract
Required capabilities:
Before native development proceeds, compare the current Transformers/safetensors backend with whole-model llama.cpp under controlled model/hardware/quality lanes.
- load full GGUF locally for immediate single-node performance
- optionally load only selected tensors/layers
- execute a layer range against inbound hidden states
- expose outbound hidden states at a boundary
- own per-session KV/state for only the loaded layer range
- support prefill chunks and decode-step calls
- expose model-specific cache metadata for DSA/MLA without requiring the tracker to understand tensor internals
Final release compares distributed GGUF with distributed safetensors using thresholds locked before seeing final results.
If llama.cpp cannot expose these as stable APIs today, the collaboration target is an upstream extension rather than a long-lived fork.
Required measurements:
## Failure Model
- TTFT.
- Prefill and decode tokens/sec.
- Aggregate concurrency throughput.
- p50/p95 latency.
- Seam bytes and latency.
- Queue/batch occupancy.
- RSS, VRAM and KV pressure.
- Output-quality drift.
- Cancellation/failure cleanup.
Alpha behavior:
The GGUF path ships only if it is faster at acceptable quality or enables a larger otherwise-unroutable model at useful measured speed.
- Route node drops during prefill: fail request and retry from scratch.
- Route node drops during decode: fail request unless a recent KV snapshot exists.
- Tracker restart: active sessions may be lost; completed billing records persist.
- Node restart: local hot KV is lost.
## Implementation sequence
Later behavior:
1. Lock benchmark/performance contract.
2. Define gRPC/protobuf and exact recipe identity.
3. Pin llama.cpp and create the minimal patch stack.
4. Implement dense-Llama range loading and boundary parity.
5. Implement concurrent local KV.
6. Build and integrate the standalone worker.
7. Pass local two-process real-model acceptance.
8. Pass real heterogeneous two-machine acceptance.
9. Add continuous batching and failure hardening.
10. Enforce the GGUF-versus-safetensors release gate.
11. Add Qwen3/Qwen3-MoE as a separately certified adapter.
12. Prepare narrow upstream collaboration patches/tests.
- periodic KV snapshots for long sessions
- prefix cache reuse across requests
- route repair when a semantically equivalent node has the same model/layer range and compatible cache snapshot
## Security And Trust
Activation/KV data can reveal user prompts. Public volunteer routes are not private. For sensitive workloads:
- use private swarms
- allow paid trusted nodes
- encrypt transport
- avoid storing hot KV on untrusted shared cache servers
- sample outputs for fraud/audit as already planned in alpha hardening
See [the Ralph backlog](prd.json) and [implementation strategy](implementation-strategy.md).

2
.scratch/distributed-gguf-runtime/decision-framework.md
View File

@@ -1,5 +1,7 @@
# Distributed GGUF Decision Framework
> **Superseded for active implementation decisions.** The grill was resolved on 2026-07-13. Use [implementation-strategy.md](implementation-strategy.md), [architecture.md](architecture.md), [ADR-0020](ADR-0020-distributed-gguf-runtime.md), and [prd.json](prd.json). This file remains as historical decision rationale.
This framework is for grilling open decisions. It keeps decisions tied to project vocabulary and implementation gates instead of vague "distributed inference" language.
## Core Vocabulary

15
.scratch/distributed-gguf-runtime/evidence/README.md Normal file
View File

@@ -0,0 +1,15 @@
# Ralph task evidence
Each completed story creates `evidence/<TASK-ID>/README.md`. Fresh dependent iterations must read it before coding.
Required README sections:
1. Summary and acceptance decision.
2. Exact files changed.
3. Commands run and real exit/results.
4. Correctness, performance and hardware evidence classification.
5. Known limitations and deferred work.
6. Compatibility/migration notes.
7. Explicit handoff for each dependent story.
Store raw machine-readable metrics, manifests and protocol artifacts beside the README. Never store secrets, model weights, build outputs or Ralph iteration logs here.

241
.scratch/distributed-gguf-runtime/implementation-strategy.md Normal file
View File

@@ -0,0 +1,241 @@
# Focused implementation strategy: performant concurrent distributed inference
Status: Accepted planning direction
Last updated: 2026-07-13
## Product objective
Enable clients to run top open models that do not fit on one consumer machine by combining independently owned model Shards into performant, concurrent Inference Routes.
The project is not trying to reproduce every vLLM feature or support every inference engine. It is optimizing for:
1. Models larger than one node's RAM/VRAM.
2. Useful interactive decode speed on consumer CPU, AMD, NVIDIA, Vulkan, and mixed routes where certified.
3. Multiple concurrent Route Sessions without cache corruption or global serialization.
4. A lean runtime with one control plane and one primary GGUF engine.
5. Measured improvement over the existing Transformers/safetensors implementation.
## Current reality
The existing project already owns the differentiating distributed control plane:
- Tracker-selected contiguous Shards.
- Stable Route Sessions.
- Local per-Shard Hot KV State in the Transformers reference backend.
- Binary Activation Seams.
- Relay/direct routing, cancellation, telemetry, billing, and capability admission.
- Persistent relay and direct transport optimizations.
The missing production path is a native GGUF execution worker that can load and execute only an assigned layer range while retaining local Hot KV State for concurrent Route Sessions.
Whole-model llama.cpp, vLLM, and existing Transformers serving remain baselines or optional route kinds. They are not substitutes for native distributed Shards.
## Performance hypothesis—not an assumption
GGUF itself is a format. Performance comes from llama.cpp/GGML's quantized kernels, memory layout, mmap, backend scheduling, and reduced working set.
Quantized GGUF may be faster or may merely fit a larger model. Comparisons against safetensors must report both speed and quality because BF16 safetensors and Q4/Q8 GGUF are not numerically equivalent.
Before expensive native work, establish controlled lanes:
- Same model architecture and upstream revision.
- Same machine, prompt set, context, output length, sampling policy, and concurrency.
- Transformers/safetensors BF16 or the current production recipe.
- llama.cpp GGUF F16/BF16 or Q8 correctness lane where available.
- Q4_K_M or selected production quantization performance/fit lane.
- TTFT, prefill tok/s, decode tok/s, p50/p95 latency, RSS, VRAM, artifact size, energy where available, and output-quality drift.
The program proceeds only if llama.cpp/GGUF provides at least one meaningful advantage recorded in a machine-readable performance contract:
- Better decode or aggregate throughput at acceptable quality; or
- Materially lower memory that makes the target model routable while preserving useful throughput.
## Parallelism we will use
### Public Inference Route: layer/pipeline parallelism
Each node independently executes one contiguous Shard. Activations cross seams; weights and Hot KV State remain local.
This is the only public cross-machine model-parallel primitive in the first runtime.
### Per-node continuous batching
Autoregressive tokens remain sequential within one generation. Throughput comes from batching decode steps from multiple active Route Sessions inside each node using llama.cpp batches and sequence IDs or bounded context pools.
This is essential. A worker that globally serializes sessions is not production-ready.
### Multiple complete routes: data parallelism
The Tracker may select multiple complete routes for independent requests. This increases network throughput and availability without requiring collectives between routes.
### Trusted composite node: optional tensor/expert parallelism
Tensor parallelism and expert parallelism require frequent collectives and tight compatibility. They may be used later inside one operator-controlled composite node or managed cluster exposed as one logical provider. They are not public WAN routing primitives.
### Deferred mechanisms
- Disaggregated prefill and KV transfer.
- Speculative decoding.
- Cross-route prefix snapshots.
- Route repair with KV migration.
- Public tensor/expert parallel collectives.
They remain out of the critical path until the native layer route passes performance and concurrency gates.
## Reuse decisions
### llama.cpp/GGML: primary runtime substrate
Reuse:
- GGUF parsing and mmap.
- Quantized kernels.
- CPU, CUDA, HIP/ROCm, Vulkan, Metal, and other supported backends.
- Tokenizer and model architecture implementations.
- KV and sequence operations.
- Backend scheduler and graph execution.
Maintain a small exact-commit fork only for the missing local seam:
- Range-aware tensor ownership/loading.
- Architecture-defined boundary input/output.
- Intermediate boundary output without tail normalization.
- Layer-filtered KV and sequence mapping.
Keep networking, Tracker logic, billing, and public protocol outside llama.cpp. Upstream generic hooks where possible.
### vLLM: concepts and optional managed backend
Use unmodified vLLM only as:
- A whole-model node backend.
- A managed TP/PP/EP cluster represented as one logical provider.
- A performance/correctness baseline.
Adapt concepts, not runtime code:
- Named intermediate tensor bundles.
- Continuous batching and request-owner maps.
- Versioned KV-transfer compatibility fingerprints.
- Explicit send/receive/abort/failure lifecycle.
- Load telemetry and unbiased route selection.
Do not fork vLLM for public Shards and do not transplant PagedAttention, Torch process groups, or GGUF-plugin kernels into the llama.cpp worker.
### Nakshatra, prima.cpp, llama-gguf, LiGGUF, GPUStack
Use as source and test donors only:
- Nakshatra: partial-GGUF patches, daemon concepts, replay cases.
- prima.cpp: selected tensor ownership and local-layer KV evidence.
- llama-gguf: small protocol and integration-test patterns.
- LiGGUF: Q8 activation transport and tensor-reduction reference.
- historical GPUStack: resource preflight and role-oriented placement.
Do not adopt or fork their repositories wholesale.
## Battle-proven transport decision
Use gRPC over HTTP/2 with Protocol Buffers for the native C++ Shard worker protocol.
Why:
- Mature Python and C++ implementations.
- Bidirectional streaming.
- HTTP/2 flow control and connection reuse.
- Deadlines, cancellation, status codes, TLS, authentication interceptors, and generated schemas.
- Avoids inventing a socket protocol.
Scope boundary:
- OpenAI-compatible client/Gateway APIs remain HTTP/SSE.
- Tracker/control APIs remain existing project interfaces.
- One long-lived bidirectional gRPC stream serves one Route Session Activation Seam.
- Existing relay/WebSocket infrastructure may carry the same versioned protobuf frames as opaque binary when direct gRPC reachability is unavailable.
- Large prefill tensors are chunked into bounded frames; decode bundles stay small.
- No QUIC/WebRTC/custom transport in this milestone.
The public boundary uses a versioned named-tensor bundle rather than one anonymous tensor because architecture boundaries can require more than `hidden_states`.
Minimum identity:
```text
schema version
request/work id
Route Session id and route epoch
Model Artifact and runtime recipe fingerprint
Shard range and effective start
phase: prefill/decode/release/cancel
position/token range
named tensors with shape/dtype/byte order
compression and checksum
idempotency step id
cache expectation/result
```
## Concurrency model
A native worker must not use one global serving sequence or one lock around all model execution.
Required ownership:
```text
(Route Session id, route epoch)
-> local sequence/context
-> Shard-local Hot KV State
-> bounded lease and memory accounting
```
The node scheduler:
- Admits sessions against model memory and KV budget.
- Forms compatible decode batches from active sessions.
- Preserves per-session position and route order.
- Applies bounded queues and backpressure.
- Cancels/releases independently.
- Reports queue, batch, KV, prefill, decode, and seam telemetry.
Initial deterministic gate: at least four concurrent sessions on a small certified model with no token/KV cross-talk. Final concurrency targets are hardware/recipe-specific and recorded by capability admission rather than hardcoded globally.
## Stage gates
### Gate A: performance hypothesis
Controlled safetensors-versus-GGUF benchmark produces a signed/reproducible report and locks thresholds. Stop native work if there is no meaningful speed or fit benefit.
### Gate B: local range parity
Two local processes own disjoint GGUF ranges and match whole-model llama.cpp within the certified numerical tolerance for prefill and greedy decode.
### Gate C: concurrent KV
Multiple Route Sessions prefill/decode concurrently with isolated local KV, bounded memory, cancellation, and release.
### Gate D: real distributed route
Two physical machines execute one model that uses both Shards. Synthetic activation tests do not satisfy this gate.
### Gate E: consumer-hardware performance
On certified consumer hardware, the GGUF route beats the current distributed safetensors route under the locked performance contract or enables a larger otherwise-unroutable model at useful measured speed.
### Gate F: architecture expansion
Only after dense Llama-family gates pass, add an explicit Qwen3/Qwen3-MoE adapter and certify it independently.
## Scope discipline
The following do not block the first production candidate:
- New cryptocurrency/economics work.
- New artifact P2P protocol.
- QUIC or WebRTC.
- vLLM fork.
- Whole-repository Nakshatra/prima adoption.
- Every GGUF architecture.
- Automatic route repair.
- Prefix snapshot migration.
- Speculative decoding.
- A large-model marketing demo before small-model parity and concurrency pass.
Every optimization must preserve output contract, session isolation, cancellation, resource cleanup, capability admission, and per-node attribution.

59
.scratch/distributed-gguf-runtime/issues/01-lock-the-safetensors-versus-gguf-performance-contract.md Normal file
View File

@@ -0,0 +1,59 @@
# 01 — Lock the safetensors-versus-GGUF performance contract
Status: ready-for-agent
## Mandatory fresh-session context
- Read [RALPH-CONTEXT.md](../RALPH-CONTEXT.md) completely before changing code.
- This issue is `DGR-001` in [prd.json](../prd.json).
- Read the evidence README for every dependency listed below.
- Inspect current code and `git status`; historical text and previous agent claims are not evidence.
## Description
As a runtime engineer, I need a controlled baseline so that GGUF work proceeds from measured speed, memory, and quality rather than reputation.
## Expected durable outputs
- Benchmark harness and deterministic tests
- evidence/DGR-001/performance-contract.json
- Raw and summarized safetensors/GGUF benchmark evidence
## Acceptance criteria
- [ ] Benchmark the same model architecture/revision, machine, prompts, context lengths, output lengths, sampling policy, and concurrency across the current Transformers/safetensors recipe and whole-model llama.cpp recipes.
- [ ] Separate correctness/quality lanes from quantized performance/fit lanes instead of claiming BF16 and Q4 are numerically equivalent.
- [ ] Report TTFT, prefill tok/s, decode tok/s, p50/p95 latency, aggregate throughput, RSS, VRAM, artifact size, failures, and output drift in machine-readable JSON.
- [ ] Add concurrency levels 1 and 4 where memory permits.
- [ ] Write a versioned performance contract consumed by later release gates, including an explicit stop condition when llama.cpp/GGUF has no meaningful speed or fit benefit.
- [ ] Targeted pytest tests pass
- [ ] python -m compileall packages tests passes for Python changes
- [ ] git diff --check passes
- [ ] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
- [ ] Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
- [ ] Real-model execution is opt-in through MESHNET_ENABLE_REAL_INFERENCE_TESTS=1 and records exact artifact/runtime/hardware evidence
- [ ] Model artifacts remain on the configured mounted-drive storage and never under /home
- [ ] Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
- [ ] Read and verify every dependency evidence README before relying on dependency behavior
- [ ] Preserve all pre-existing working-tree changes and stage only files belonging to this story
- [ ] Write .scratch/distributed-gguf-runtime/evidence/DGR-001/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
- [ ] Update only this story issue to Status: done after every acceptance criterion and quality gate passes
## Dependency handoff
- None. This story may start immediately.
## Finish contract
- Create the task evidence directory and durable handoff required above.
- Preserve real failures and blockers; never fabricate benchmark, model, test or hardware output.
- Change this issue to `Status: done` only after all criteria pass.
- Emit `<promise>COMPLETE</promise>` only after the evidence handoff exists.
## References
- [Ralph execution context](../RALPH-CONTEXT.md)
- [PRD](../PRD.md)
- [Implementation strategy](../implementation-strategy.md)
- [Current architecture](../architecture.md)
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)

19
.scratch/distributed-gguf-runtime/issues/01-route-session-lifecycle.md
View File

@@ -1,19 +0,0 @@
# 01 — Route Session lifecycle
Status: ready-for-agent
## What to build
Add the narrowest end-to-end Route Session lifecycle that can be used by distributed inference routes: create a session, bind it to a selected Inference Route, expose status, and close it cleanly. This slice does not need real model cache yet; it proves stable session identity across the control plane and activation plane.
## Acceptance criteria
- [ ] A request can create a Route Session with a stable `session_id`, `route_id`, model preset, backend id, and route membership.
- [ ] Every downstream activation request carries the same session identity and fails clearly if the session or route id does not match.
- [ ] Session status reports phase, route nodes, model preset, backend id, created time, and last activity time.
- [ ] Closing a session releases all registered per-session state.
- [ ] Tests cover create, status, close, stale-session rejection, and wrong-route rejection.
## Blocked by
None - can start immediately.

59
.scratch/distributed-gguf-runtime/issues/02-adopt-the-versioned-grpc-shard-protocol.md Normal file
View File

@@ -0,0 +1,59 @@
# 02 — Adopt the versioned gRPC Shard protocol
Status: ready-for-agent
## Mandatory fresh-session context
- Read [RALPH-CONTEXT.md](../RALPH-CONTEXT.md) completely before changing code.
- This issue is `DGR-002` in [prd.json](../prd.json).
- Read the evidence README for every dependency listed below.
- Inspect current code and `git status`; historical text and previous agent claims are not evidence.
## Description
As a node developer, I need a battle-proven streaming protocol so that Python and C++ Shards communicate without a custom socket protocol.
## Expected durable outputs
- packages/node/native/proto/shard_runtime.proto
- Reproducible Python/C++ schema generation and build wiring
- Protocol round-trip and compatibility tests
- evidence/DGR-002/README.md
## Acceptance criteria
- [ ] Add a Protocol Buffers schema for capability, health, session stream, release, and cancellation operations.
- [ ] Define one long-lived bidirectional gRPC stream per Route Session Activation Seam with deadlines, cancellation, flow control, and structured errors.
- [ ] Define bounded chunking for prefill and a small decode fast path.
- [ ] Carry schema version, request/work ID, Route Session ID, route epoch, artifact/recipe fingerprint, Shard range/effective start, phase, position, idempotency step, cache expectation, compression, and checksum.
- [ ] Define a versioned named-tensor bundle with per-tensor name, shape, dtype, byte order, and payload fragments.
- [ ] Add generated-schema round-trip and compatibility tests in Python and C++.
- [ ] Targeted pytest tests pass
- [ ] python -m compileall packages tests passes for Python changes
- [ ] git diff --check passes
- [ ] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
- [ ] Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
- [ ] Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
- [ ] Read and verify every dependency evidence README before relying on dependency behavior
- [ ] Preserve all pre-existing working-tree changes and stage only files belonging to this story
- [ ] Write .scratch/distributed-gguf-runtime/evidence/DGR-002/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
- [ ] Update only this story issue to Status: done after every acceptance criterion and quality gate passes
## Dependency handoff
- None. This story may start immediately.
## Finish contract
- Create the task evidence directory and durable handoff required above.
- Preserve real failures and blockers; never fabricate benchmark, model, test or hardware output.
- Change this issue to `Status: done` only after all criteria pass.
- Emit `<promise>COMPLETE</promise>` only after the evidence handoff exists.
## References
- [Ralph execution context](../RALPH-CONTEXT.md)
- [PRD](../PRD.md)
- [Implementation strategy](../implementation-strategy.md)
- [Current architecture](../architecture.md)
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)

20
.scratch/distributed-gguf-runtime/issues/02-prefill-decode-binary-http.md
View File

@@ -1,20 +0,0 @@
# 02 — Prefill/decode binary HTTP protocol
Status: ready-for-agent
## What to build
Split the activation protocol into explicit prefill and decode-step calls using the existing binary HTTP direction from ADR-0008. The completed slice should work against a stub backend so payload shape, route/session headers, relay preservation, and failure behavior are testable before real KV cache work begins.
## Acceptance criteria
- [ ] Prefill accepts chunked binary activations with route/session metadata and forwards them through the selected route.
- [ ] Decode-step accepts a one-step binary activation and forwards a one-step activation through the selected route.
- [ ] Decode-step payload size is independent of prompt length in protocol tests.
- [ ] Relay forwarding preserves route/session headers, shape, dtype, position, and wire version.
- [ ] Legacy `/forward` either remains as a compatibility wrapper or fails with a clear wire-version error.
- [ ] Tests cover prefill chunking, decode-step shape validation, relay preservation, and malformed header rejection.
## Blocked by
- 01 — Route Session lifecycle.

57
.scratch/distributed-gguf-runtime/issues/03-define-exact-artifact-and-runtime-recipe-identity.md Normal file
View File

@@ -0,0 +1,57 @@
# 03 — Define exact Artifact and runtime recipe identity
Status: ready-for-agent
## Mandatory fresh-session context
- Read [RALPH-CONTEXT.md](../RALPH-CONTEXT.md) completely before changing code.
- This issue is `DGR-003` in [prd.json](../prd.json).
- Read the evidence README for every dependency listed below.
- Inspect current code and `git status`; historical text and previous agent claims are not evidence.
## Description
As the Tracker, I need exact compatibility identity so that only numerically and operationally compatible Shards form an Inference Route.
## Expected durable outputs
- Exact runtime recipe/fingerprint implementation
- Tracker/node fail-closed admission tests
- evidence/DGR-003/README.md
## Acceptance criteria
- [ ] Separate weight quantization, activation dtype, compute dtype, KV dtype/layout, tokenizer revision, architecture adapter, backend, and runtime version.
- [ ] Bind derivative or split artifacts to an exact source Model Artifact hash and Shard range.
- [ ] Produce a stable compatibility fingerprint used by capability admission and the gRPC handshake.
- [ ] Fail closed on mismatched artifact, tokenizer, architecture, range, boundary schema, activation recipe, or cache layout.
- [ ] Keep unsupported recipes registered-but-dark until a real distributed forward certifies them.
- [ ] Targeted pytest tests pass
- [ ] python -m compileall packages tests passes for Python changes
- [ ] git diff --check passes
- [ ] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
- [ ] Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
- [ ] Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
- [ ] Read and verify every dependency evidence README before relying on dependency behavior
- [ ] Preserve all pre-existing working-tree changes and stage only files belonging to this story
- [ ] Write .scratch/distributed-gguf-runtime/evidence/DGR-003/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
- [ ] Update only this story issue to Status: done after every acceptance criterion and quality gate passes
## Dependency handoff
- `DGR-002` must have `passes: true`; read `../evidence/DGR-002/README.md` and verify its referenced files/commands.
## Finish contract
- Create the task evidence directory and durable handoff required above.
- Preserve real failures and blockers; never fabricate benchmark, model, test or hardware output.
- Change this issue to `Status: done` only after all criteria pass.
- Emit `<promise>COMPLETE</promise>` only after the evidence handoff exists.
## References
- [Ralph execution context](../RALPH-CONTEXT.md)
- [PRD](../PRD.md)
- [Implementation strategy](../implementation-strategy.md)
- [Current architecture](../architecture.md)
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)

21
.scratch/distributed-gguf-runtime/issues/03-generation-telemetry-and-streaming.md
View File

@@ -1,21 +0,0 @@
# 03 — Generation Telemetry and streaming response contract
Status: ready-for-agent
## What to build
Expose realtime Generation Telemetry for active Route Sessions and stream token deltas when the serving path can produce them. This slice should make long distributed requests observable before real large-model work begins.
## Acceptance criteria
- [ ] A client can observe route-session phase changes: queued, loading, prefill, decode, finalizing, completed, failed.
- [ ] Telemetry includes prefill progress, generated token count, rolling tokens/sec, average tokens/sec, active route nodes, and failure reason.
- [ ] Telemetry is available before the first output token.
- [ ] A streaming response can include token deltas while telemetry remains available.
- [ ] A non-streaming fallback still exposes telemetry until final answer or failure.
- [ ] Route-node failure reports the last known phase and reason.
- [ ] Tests cover telemetry updates, streaming token deltas, non-streaming fallback, and structured failure closeout.
## Blocked by
- 01 — Route Session lifecycle.

61
.scratch/distributed-gguf-runtime/issues/04-create-the-reproducible-pinned-llama-cpp-patch-stack.md Normal file
View File

@@ -0,0 +1,61 @@
# 04 — Create the reproducible pinned llama.cpp patch stack
Status: ready-for-agent
## Mandatory fresh-session context
- Read [RALPH-CONTEXT.md](../RALPH-CONTEXT.md) completely before changing code.
- This issue is `DGR-004` in [prd.json](../prd.json).
- Read the evidence README for every dependency listed below.
- Inspect current code and `git status`; historical text and previous agent claims are not evidence.
## Description
As a maintainer, I need a small auditable fork boundary so that upstream updates do not turn the runtime into an unmaintainable stitched codebase.
## Expected durable outputs
- Exact llama.cpp upstream pin
- Numbered minimal patch stack
- Reproducible fetch/apply/build smoke
- evidence/DGR-004/README.md
## Acceptance criteria
- [ ] Pin one exact llama.cpp commit through a reproducible source dependency mechanism.
- [ ] Store a numbered minimal patch stack separately from Meshnet networking code.
- [ ] Add a build script that applies/checks patches and builds the standalone worker without manual source copying.
- [ ] Record upstream file/ABI assumptions and fail clearly when the pin changes.
- [ ] Preserve upstream license and attribution notices.
- [ ] Add a clean rebuild smoke test that does not download a model.
- [ ] Targeted pytest tests pass
- [ ] python -m compileall packages tests passes for Python changes
- [ ] git diff --check passes
- [ ] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
- [ ] Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
- [ ] Pinned native C++ target builds and focused CTest/protocol tests pass where native code is touched
- [ ] llama.cpp patch stack applies cleanly to the exact pinned commit where patch code is touched
- [ ] Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
- [ ] Read and verify every dependency evidence README before relying on dependency behavior
- [ ] Preserve all pre-existing working-tree changes and stage only files belonging to this story
- [ ] Write .scratch/distributed-gguf-runtime/evidence/DGR-004/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
- [ ] Update only this story issue to Status: done after every acceptance criterion and quality gate passes
## Dependency handoff
- `DGR-001` must have `passes: true`; read `../evidence/DGR-001/README.md` and verify its referenced files/commands.
## Finish contract
- Create the task evidence directory and durable handoff required above.
- Preserve real failures and blockers; never fabricate benchmark, model, test or hardware output.
- Change this issue to `Status: done` only after all criteria pass.
- Emit `<promise>COMPLETE</promise>` only after the evidence handoff exists.
## References
- [Ralph execution context](../RALPH-CONTEXT.md)
- [PRD](../PRD.md)
- [Implementation strategy](../implementation-strategy.md)
- [Current architecture](../architecture.md)
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)

23
.scratch/distributed-gguf-runtime/issues/04-pytorch-distributed-kv-reference.md
View File

@@ -1,23 +0,0 @@
# 04 — PyTorch distributed KV reference route
Status: ready-for-agent
## What to build
Fix the existing distributed PyTorch route so it uses the Route Session and prefill/decode protocol to keep Hot KV State local to each Shard node. The visible behavior is that prefill processes the prompt once, and decode no longer recomputes or resends the full growing prompt for every token.
## Acceptance criteria
- [ ] Distributed PyTorch prefill stores per-session cache/state on each Shard node.
- [ ] Distributed PyTorch decode-step reads and appends local per-shard cache/state.
- [ ] Decode activation seam payload is one token/hidden-state step after prefill.
- [ ] The old full-growing-prompt decode loop is not used for models that support the reference cache path.
- [ ] Unsupported model/cache APIs fail with an explicit backend capability error.
- [ ] Session close or TTL cleanup releases per-shard cache.
- [ ] Regression tests prove decode does not call the full prompt encoder for every generated token.
## Blocked by
- 01 — Route Session lifecycle.
- 02 — Prefill/decode binary HTTP protocol.
- 03 — Generation Telemetry and streaming response contract.

61
.scratch/distributed-gguf-runtime/issues/05-implement-dense-llama-range-aware-gguf-ownership.md Normal file
View File

@@ -0,0 +1,61 @@
# 05 — Implement dense-Llama range-aware GGUF ownership
Status: ready-for-agent
## Mandatory fresh-session context
- Read [RALPH-CONTEXT.md](../RALPH-CONTEXT.md) completely before changing code.
- This issue is `DGR-005` in [prd.json](../prd.json).
- Read the evidence README for every dependency listed below.
- Inspect current code and `git status`; historical text and previous agent claims are not evidence.
## Description
As a node, I need to map only my assigned dense-Llama Shard so that aggregate consumer memory can hold a model larger than one node.
## Expected durable outputs
- Dense-Llama range-aware ownership implementation
- Authoritative loaded-range introspection
- Mapped/resident memory evidence
- evidence/DGR-005/README.md
## Acceptance criteria
- [ ] Register and allocate only `blk.N.*` tensors in the assigned range.
- [ ] Load embeddings only for the head and final norm/LM head only for the tail, including tied embeddings.
- [ ] Prefer range-aware mapping from one exact source GGUF; if derivative sub-GGUFs are used temporarily, verify source/slice hashes and avoid claiming final artifact semantics.
- [ ] Report authoritative loaded range and endpoint ownership from the model, not operator CLI claims.
- [ ] Demonstrate mapped/resident memory scales with owned tensors rather than full model size.
- [ ] Targeted pytest tests pass
- [ ] python -m compileall packages tests passes for Python changes
- [ ] git diff --check passes
- [ ] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
- [ ] Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
- [ ] Pinned native C++ target builds and focused CTest/protocol tests pass where native code is touched
- [ ] llama.cpp patch stack applies cleanly to the exact pinned commit where patch code is touched
- [ ] Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
- [ ] Read and verify every dependency evidence README before relying on dependency behavior
- [ ] Preserve all pre-existing working-tree changes and stage only files belonging to this story
- [ ] Write .scratch/distributed-gguf-runtime/evidence/DGR-005/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
- [ ] Update only this story issue to Status: done after every acceptance criterion and quality gate passes
## Dependency handoff
- `DGR-003` must have `passes: true`; read `../evidence/DGR-003/README.md` and verify its referenced files/commands.
- `DGR-004` must have `passes: true`; read `../evidence/DGR-004/README.md` and verify its referenced files/commands.
## Finish contract
- Create the task evidence directory and durable handoff required above.
- Preserve real failures and blockers; never fabricate benchmark, model, test or hardware output.
- Change this issue to `Status: done` only after all criteria pass.
- Emit `<promise>COMPLETE</promise>` only after the evidence handoff exists.
## References
- [Ralph execution context](../RALPH-CONTEXT.md)
- [PRD](../PRD.md)
- [Implementation strategy](../implementation-strategy.md)
- [Current architecture](../architecture.md)
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)

20
.scratch/distributed-gguf-runtime/issues/05-local-llamacpp-gguf-backend.md
View File

@@ -1,20 +0,0 @@
# 05 — Local llama.cpp/GGUF backend
Status: ready-for-agent
## What to build
Add a local full-model GGUF backend so a node that can hold a GGUF model can serve it through the existing node API. This is the immediate CPU-performance path and the baseline for later distributed llama.cpp work.
## Acceptance criteria
- [ ] A node can start with backend `llama.cpp` or `gguf` for a local full-model GGUF artifact.
- [ ] The node can answer an OpenAI-compatible chat completion through the existing API.
- [ ] Startup and registration clearly report backend, quantization/artifact metadata, context cap, and local model path.
- [ ] The PyTorch backend remains unchanged and selectable.
- [ ] A smoke test or script validates backend wiring with a small GGUF model or a stubbed llama.cpp process.
- [ ] A benchmark command can compare local PyTorch CPU and local GGUF CPU for the same small supported model when both are available.
## Blocked by
None - can start immediately.

61
.scratch/distributed-gguf-runtime/issues/06-implement-architecture-defined-boundary-input-output.md Normal file
View File

@@ -0,0 +1,61 @@
# 06 — Implement architecture-defined boundary input/output
Status: ready-for-agent
## Mandatory fresh-session context
- Read [RALPH-CONTEXT.md](../RALPH-CONTEXT.md) completely before changing code.
- This issue is `DGR-006` in [prd.json](../prd.json).
- Read the evidence README for every dependency listed below.
- Inspect current code and `git status`; historical text and previous agent claims are not evidence.
## Description
As a Shard, I need to consume and emit the correct transformer boundary state so that disjoint processes reproduce whole-model execution.
## Expected durable outputs
- Architecture boundary adapter
- Whole-model/two-range parity tests and results
- evidence/DGR-006/README.md
## Acceptance criteria
- [ ] Head accepts token IDs and owns token embedding.
- [ ] Middle/tail bypass token embedding and accept the named boundary bundle.
- [ ] Non-tail emits the unnormalized architecture-defined residual/boundary before final norm/head and before tail-only row pruning.
- [ ] Tail emits logits or token output through an explicit sampling contract.
- [ ] Dense-Llama whole-model versus two-range prefill and greedy-decode parity passes the documented tolerance.
- [ ] The adapter interface fails closed for uncertified architectures.
- [ ] Targeted pytest tests pass
- [ ] python -m compileall packages tests passes for Python changes
- [ ] git diff --check passes
- [ ] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
- [ ] Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
- [ ] Pinned native C++ target builds and focused CTest/protocol tests pass where native code is touched
- [ ] llama.cpp patch stack applies cleanly to the exact pinned commit where patch code is touched
- [ ] Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
- [ ] Read and verify every dependency evidence README before relying on dependency behavior
- [ ] Preserve all pre-existing working-tree changes and stage only files belonging to this story
- [ ] Write .scratch/distributed-gguf-runtime/evidence/DGR-006/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
- [ ] Update only this story issue to Status: done after every acceptance criterion and quality gate passes
## Dependency handoff
- `DGR-002` must have `passes: true`; read `../evidence/DGR-002/README.md` and verify its referenced files/commands.
- `DGR-005` must have `passes: true`; read `../evidence/DGR-005/README.md` and verify its referenced files/commands.
## Finish contract
- Create the task evidence directory and durable handoff required above.
- Preserve real failures and blockers; never fabricate benchmark, model, test or hardware output.
- Change this issue to `Status: done` only after all criteria pass.
- Emit `<promise>COMPLETE</promise>` only after the evidence handoff exists.
## References
- [Ralph execution context](../RALPH-CONTEXT.md)
- [PRD](../PRD.md)
- [Implementation strategy](../implementation-strategy.md)
- [Current architecture](../architecture.md)
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)

20
.scratch/distributed-gguf-runtime/issues/06-model-artifact-manifest.md
View File

@@ -1,20 +0,0 @@
# 06 — Model Artifact manifest and Shard advertisement
Status: ready-for-agent
## What to build
Introduce a Model Artifact manifest that separates storage distribution from route execution. A node should be able to verify local model files, determine which Shards it can serve, and advertise artifact/layer availability to the Tracker without contacting Hugging Face at request time.
## Acceptance criteria
- [ ] Manifest records model preset, upstream revision, license, backend support, quantization, context cap, tokenizer artifacts, file hashes, piece hashes, and tensor/layer mapping where available.
- [ ] A node can verify local artifacts against the manifest and reject corrupt or incomplete artifacts.
- [ ] A node can derive advertised Shard ranges from the manifest and local files.
- [ ] Tracker registration can include artifact id, backend id, Shard range, and verification status.
- [ ] Tracker coverage can distinguish model-layer coverage from artifact availability.
- [ ] Tests cover valid manifest registration, corrupt artifact rejection, and missing layer/tensor metadata.
## Blocked by
- 01 — Route Session lifecycle.

60
.scratch/distributed-gguf-runtime/issues/07-add-isolated-concurrent-local-hot-kv-state.md Normal file
View File

@@ -0,0 +1,60 @@
# 07 — Add isolated concurrent local Hot KV State
Status: ready-for-agent
## Mandatory fresh-session context
- Read [RALPH-CONTEXT.md](../RALPH-CONTEXT.md) completely before changing code.
- This issue is `DGR-007` in [prd.json](../prd.json).
- Read the evidence README for every dependency listed below.
- Inspect current code and `git status`; historical text and previous agent claims are not evidence.
## Description
As a client, I need concurrent Route Sessions to retain independent per-Shard cache so that one request cannot clear or corrupt another.
## Expected durable outputs
- Concurrent local KV/session manager
- Isolation, eviction, cancellation and cleanup tests
- evidence/DGR-007/README.md
## Acceptance criteria
- [ ] Map `(Route Session ID, route epoch)` to an isolated llama sequence or bounded context.
- [ ] Allocate KV only for owned layers.
- [ ] Support prefill append, decode append, truncate, release, TTL/LRU eviction, and explicit cache-miss response.
- [ ] Reject stale epochs and incompatible cache recipes.
- [ ] At least four concurrent sessions on a small model complete without token or KV cross-talk.
- [ ] Cancellation/release of one session leaves other sessions intact and memory returns to the configured budget.
- [ ] Targeted pytest tests pass
- [ ] python -m compileall packages tests passes for Python changes
- [ ] git diff --check passes
- [ ] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
- [ ] Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
- [ ] Pinned native C++ target builds and focused CTest/protocol tests pass where native code is touched
- [ ] llama.cpp patch stack applies cleanly to the exact pinned commit where patch code is touched
- [ ] Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
- [ ] Read and verify every dependency evidence README before relying on dependency behavior
- [ ] Preserve all pre-existing working-tree changes and stage only files belonging to this story
- [ ] Write .scratch/distributed-gguf-runtime/evidence/DGR-007/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
- [ ] Update only this story issue to Status: done after every acceptance criterion and quality gate passes
## Dependency handoff
- `DGR-006` must have `passes: true`; read `../evidence/DGR-006/README.md` and verify its referenced files/commands.
## Finish contract
- Create the task evidence directory and durable handoff required above.
- Preserve real failures and blockers; never fabricate benchmark, model, test or hardware output.
- Change this issue to `Status: done` only after all criteria pass.
- Emit `<promise>COMPLETE</promise>` only after the evidence handoff exists.
## References
- [Ralph execution context](../RALPH-CONTEXT.md)
- [PRD](../PRD.md)
- [Implementation strategy](../implementation-strategy.md)
- [Current architecture](../architecture.md)
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)

25
.scratch/distributed-gguf-runtime/issues/07-llamacpp-layer-boundary-prototype.md
View File

@@ -1,25 +0,0 @@
# 07 — llama.cpp layer-boundary prototype
Status: ready-for-human
## What to build
Build a local prototype that proves whether llama.cpp/libllama can support the platform's distributed execution contract: execute a selected layer range, accept inbound hidden states, emit outbound hidden states, and own per-session cache for only the loaded/served range.
This is the collaboration package for upstream llama.cpp. The target is an upstreamable API shape, not a permanent fork.
## Acceptance criteria
- [ ] A small llama.cpp-supported GGUF model can be split into a two-process localhost head/tail prototype.
- [ ] The head process runs embeddings and early layers, then emits hidden states at an Activation Seam.
- [ ] The tail process accepts hidden states, runs later layers plus output head, and produces logits/tokens comparable to single-process execution.
- [ ] Prefill is performed once and decode-step seam payload is one hidden-state step per generated token.
- [ ] Each process owns only its own per-session cache/state.
- [ ] The prototype records the minimum upstream API needed for layer-range execution, hidden-state I/O, partial loading/introspection, and per-session KV ownership.
- [ ] If upstream support is unavailable, the issue ends with a concrete recommendation: upstream proposal, narrow adapter fork, or keep GGUF distribution local-only for now.
## Blocked by
- 02 — Prefill/decode binary HTTP protocol.
- 05 — Local llama.cpp/GGUF backend.
- 06 — Model Artifact manifest and Shard advertisement.

65
.scratch/distributed-gguf-runtime/issues/08-build-the-standalone-c-grpc-shard-worker.md Normal file
View File

@@ -0,0 +1,65 @@
# 08 — Build the standalone C++ gRPC Shard worker
Status: ready-for-agent
## Mandatory fresh-session context
- Read [RALPH-CONTEXT.md](../RALPH-CONTEXT.md) completely before changing code.
- This issue is `DGR-008` in [prd.json](../prd.json).
- Read the evidence README for every dependency listed below.
- Inspect current code and `git status`; historical text and previous agent claims are not evidence.
## Description
As a node runtime, I need one supervised native process so that llama.cpp internals remain behind a stable project-owned protocol.
## Expected durable outputs
- Standalone C++ gRPC worker
- Fake-model Python/C++ integration tests
- Lifecycle and bounded-failure evidence
- evidence/DGR-008/README.md
## Acceptance criteria
- [ ] Worker exposes capability, health, session stream, release, cancellation, and metrics services from DGR-002.
- [ ] Worker loads one exact Artifact/recipe/Shard identity and refuses mismatched requests.
- [ ] Streaming path enforces bounded messages, flow control, deadlines, idempotency, and independent session cancellation.
- [ ] Worker does not expose raw llama.cpp RPC or arbitrary GGML graph execution.
- [ ] Graceful shutdown releases sessions; crash behavior is bounded and observable.
- [ ] Python integration tests run against a fake model mode without model downloads.
- [ ] Targeted pytest tests pass
- [ ] python -m compileall packages tests passes for Python changes
- [ ] git diff --check passes
- [ ] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
- [ ] Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
- [ ] Pinned native C++ target builds and focused CTest/protocol tests pass where native code is touched
- [ ] llama.cpp patch stack applies cleanly to the exact pinned commit where patch code is touched
- [ ] Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
- [ ] Read and verify every dependency evidence README before relying on dependency behavior
- [ ] Preserve all pre-existing working-tree changes and stage only files belonging to this story
- [ ] Write .scratch/distributed-gguf-runtime/evidence/DGR-008/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
- [ ] Update only this story issue to Status: done after every acceptance criterion and quality gate passes
## Dependency handoff
- `DGR-002` must have `passes: true`; read `../evidence/DGR-002/README.md` and verify its referenced files/commands.
- `DGR-003` must have `passes: true`; read `../evidence/DGR-003/README.md` and verify its referenced files/commands.
- `DGR-004` must have `passes: true`; read `../evidence/DGR-004/README.md` and verify its referenced files/commands.
- `DGR-006` must have `passes: true`; read `../evidence/DGR-006/README.md` and verify its referenced files/commands.
- `DGR-007` must have `passes: true`; read `../evidence/DGR-007/README.md` and verify its referenced files/commands.
## Finish contract
- Create the task evidence directory and durable handoff required above.
- Preserve real failures and blockers; never fabricate benchmark, model, test or hardware output.
- Change this issue to `Status: done` only after all criteria pass.
- Emit `<promise>COMPLETE</promise>` only after the evidence handoff exists.
## References
- [Ralph execution context](../RALPH-CONTEXT.md)
- [PRD](../PRD.md)
- [Implementation strategy](../implementation-strategy.md)
- [Current architecture](../architecture.md)
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)

24
.scratch/distributed-gguf-runtime/issues/08-networked-distributed-gguf-route.md
View File

@@ -1,24 +0,0 @@
# 08 — Networked distributed GGUF route
Status: pending
## What to build
Run a GGUF-backed model over a real multi-node Inference Route using the resolved Route Session, binary HTTP prefill/decode protocol, local Hot KV State, Generation Telemetry, and alpha fail-fast behavior.
## Acceptance criteria
- [ ] Two machines can form one GGUF-backed Inference Route over contiguous Shards.
- [ ] Prefill builds local per-shard cache/state and decode-step uses one-step seam payloads.
- [ ] The client receives streamed token deltas when supported by the GGUF path.
- [ ] The client receives Generation Telemetry for phase, generated tokens, tokens/sec, route health, and failure reason.
- [ ] Route-node loss fails the Route Session cleanly; no automatic repair is attempted in alpha.
- [ ] Tracker metrics show prefill tokens/sec, decode tokens/sec, seam latency, queue depth, and cache memory by node.
- [ ] Billing/audit records identify route membership and layer/token work for the completed or failed session.
## Blocked by
- 03 — Generation Telemetry and streaming response contract.
- 04 — PyTorch distributed KV reference route.
- 06 — Model Artifact manifest and Shard advertisement.
- 07 — llama.cpp layer-boundary prototype.

21
.scratch/distributed-gguf-runtime/issues/09-deepseek-v4-flash-support-audit.md
View File

@@ -1,21 +0,0 @@
# 09 — DeepSeek-V4-Flash support audit
Status: ready-for-agent
## What to build
Audit `deepseek-ai/DeepSeek-V4-Flash` as the first serious large-model target after the small GGUF protocol smoke test. The output is a compatibility matrix and a recommended runtime path, not full production support.
## Acceptance criteria
- [ ] Verify current PyTorch/Transformers load and generation semantics for DeepSeek-V4-Flash from primary model documentation.
- [ ] Verify vLLM and SGLang support status from primary runtime documentation or release notes.
- [ ] Verify whether a GGUF/llama.cpp quantization path exists or would need upstream work.
- [ ] Estimate artifact size, active parameter behavior, and 128K cache memory by Shard range.
- [ ] Identify required backend capability flags for the Tracker.
- [ ] Produce a compatibility matrix: PyTorch, vLLM, SGLang, llama.cpp/GGUF.
- [ ] End with one recommendation: first runtime path, blocked pending upstream, or defer.
## Blocked by
None - can start immediately.

61
.scratch/distributed-gguf-runtime/issues/09-integrate-the-native-worker-with-meshnet.md Normal file
View File

@@ -0,0 +1,61 @@
# 09 — Integrate the native worker with Meshnet
Status: ready-for-agent
## Mandatory fresh-session context
- Read [RALPH-CONTEXT.md](../RALPH-CONTEXT.md) completely before changing code.
- This issue is `DGR-009` in [prd.json](../prd.json).
- Read the evidence README for every dependency listed below.
- Inspect current code and `git status`; historical text and previous agent claims are not evidence.
## Description
As the existing node service, I need a GGUF Shard backend adapter so that the Tracker, relay, billing, telemetry, and capability admission remain the sole control plane.
## Expected durable outputs
- Meshnet GGUF backend adapter
- Registration, routing, relay, telemetry and billing tests
- evidence/DGR-009/README.md
## Acceptance criteria
- [ ] Implement the existing model-backend surface without changing Transformers behavior.
- [ ] Registration carries exact validated GGUF recipe, Shard, backend and concurrency/KV capacity.
- [ ] Tracker forms only complete compatible routes and keeps uncertified recipes dark.
- [ ] Direct routes use gRPC streams; relayed routes carry the same versioned protobuf frames as opaque binary through the existing relay seam.
- [ ] Existing request/work IDs, cancellation, Generation Telemetry, billing, and per-node attribution remain correlated.
- [ ] No vLLM, Nakshatra, prima.cpp, or custom-engine control plane becomes a core dependency.
- [ ] Targeted pytest tests pass
- [ ] python -m compileall packages tests passes for Python changes
- [ ] git diff --check passes
- [ ] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
- [ ] Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
- [ ] Pinned native C++ target builds and focused CTest/protocol tests pass where native code is touched
- [ ] llama.cpp patch stack applies cleanly to the exact pinned commit where patch code is touched
- [ ] Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
- [ ] Read and verify every dependency evidence README before relying on dependency behavior
- [ ] Preserve all pre-existing working-tree changes and stage only files belonging to this story
- [ ] Write .scratch/distributed-gguf-runtime/evidence/DGR-009/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
- [ ] Update only this story issue to Status: done after every acceptance criterion and quality gate passes
## Dependency handoff
- `DGR-003` must have `passes: true`; read `../evidence/DGR-003/README.md` and verify its referenced files/commands.
- `DGR-008` must have `passes: true`; read `../evidence/DGR-008/README.md` and verify its referenced files/commands.
## Finish contract
- Create the task evidence directory and durable handoff required above.
- Preserve real failures and blockers; never fabricate benchmark, model, test or hardware output.
- Change this issue to `Status: done` only after all criteria pass.
- Emit `<promise>COMPLETE</promise>` only after the evidence handoff exists.
## References
- [Ralph execution context](../RALPH-CONTEXT.md)
- [PRD](../PRD.md)
- [Implementation strategy](../implementation-strategy.md)
- [Current architecture](../architecture.md)
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)

20
.scratch/distributed-gguf-runtime/issues/10-glm52-ornith-followup-audit.md
View File

@@ -1,20 +0,0 @@
# 10 — GLM-5.2 and Ornith follow-up support audit
Status: pending
## What to build
Audit GLM-5.2 and Ornith after the smaller protocol smoke path and DeepSeek-V4-Flash audit. The output is a follow-up compatibility matrix focused on architecture/runtime blockers: DSA/MLA, hybrid attention, cache accounting, and GGUF/llama.cpp support.
## Acceptance criteria
- [ ] Verify GLM-5.2 PyTorch/Transformers serving requirements and cache semantics from primary model documentation.
- [ ] Verify llama.cpp/GGUF support status for `glm_moe_dsa` or equivalent architecture support.
- [ ] Verify Ornith/Qwen3.5-MoE and hybrid attention support status in the candidate runtimes.
- [ ] Estimate artifact size and 128K cache memory by Shard range for each model.
- [ ] Identify smallest quality-preserving quantization worth testing.
- [ ] Convert each runtime blocker into a follow-up issue or upstream collaboration note.
## Blocked by
- 09 — DeepSeek-V4-Flash support audit.

62
.scratch/distributed-gguf-runtime/issues/10-pass-local-real-model-two-process-acceptance.md Normal file
View File

@@ -0,0 +1,62 @@
# 10 — Pass local real-model two-process acceptance
Status: ready-for-agent
## Mandatory fresh-session context
- Read [RALPH-CONTEXT.md](../RALPH-CONTEXT.md) completely before changing code.
- This issue is `DGR-010` in [prd.json](../prd.json).
- Read the evidence README for every dependency listed below.
- Inspect current code and `git status`; historical text and previous agent claims are not evidence.
## Description
As a release engineer, I need real local distributed parity before involving network variability.
## Expected durable outputs
- Real local two-process commands and configuration
- Raw parity, memory and performance results
- evidence/DGR-010/README.md
## Acceptance criteria
- [ ] Two local worker processes open disjoint dense-Llama ranges from the certified Artifact.
- [ ] Prefill and at least 32 greedy decode tokens match whole-model llama.cpp within the certified tolerance.
- [ ] Each worker retains only its own tensors and Hot KV State.
- [ ] Four concurrent Route Sessions pass isolation and cleanup checks.
- [ ] Report TTFT, prefill/decode throughput, seam bytes/latency, worker RSS/VRAM, KV memory, batch size, and queue time.
- [ ] Killing one worker produces a bounded structured failure rather than a deadlock.
- [ ] Targeted pytest tests pass
- [ ] python -m compileall packages tests passes for Python changes
- [ ] git diff --check passes
- [ ] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
- [ ] Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
- [ ] Pinned native C++ target builds and focused CTest/protocol tests pass where native code is touched
- [ ] llama.cpp patch stack applies cleanly to the exact pinned commit where patch code is touched
- [ ] Real-model execution is opt-in through MESHNET_ENABLE_REAL_INFERENCE_TESTS=1 and records exact artifact/runtime/hardware evidence
- [ ] Model artifacts remain on the configured mounted-drive storage and never under /home
- [ ] Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
- [ ] Read and verify every dependency evidence README before relying on dependency behavior
- [ ] Preserve all pre-existing working-tree changes and stage only files belonging to this story
- [ ] Write .scratch/distributed-gguf-runtime/evidence/DGR-010/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
- [ ] Update only this story issue to Status: done after every acceptance criterion and quality gate passes
## Dependency handoff
- `DGR-009` must have `passes: true`; read `../evidence/DGR-009/README.md` and verify its referenced files/commands.
## Finish contract
- Create the task evidence directory and durable handoff required above.
- Preserve real failures and blockers; never fabricate benchmark, model, test or hardware output.
- Change this issue to `Status: done` only after all criteria pass.
- Emit `<promise>COMPLETE</promise>` only after the evidence handoff exists.
## References
- [Ralph execution context](../RALPH-CONTEXT.md)
- [PRD](../PRD.md)
- [Implementation strategy](../implementation-strategy.md)
- [Current architecture](../architecture.md)
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)

62
.scratch/distributed-gguf-runtime/issues/11-pass-a-real-heterogeneous-two-machine-route.md Normal file
View File

@@ -0,0 +1,62 @@
# 11 — Pass a real heterogeneous two-machine route
Status: ready-for-agent
## Mandatory fresh-session context
- Read [RALPH-CONTEXT.md](../RALPH-CONTEXT.md) completely before changing code.
- This issue is `DGR-011` in [prd.json](../prd.json).
- Read the evidence README for every dependency listed below.
- Inspect current code and `git status`; historical text and previous agent claims are not evidence.
## Description
As a consumer-hardware operator, I need two physical machines to execute one GGUF model so that the distributed claim is real.
## Expected durable outputs
- Two-machine hardware/network/runtime manifest
- Raw real-route metrics and output evidence
- evidence/DGR-011/README.md
## Acceptance criteria
- [ ] Tracker selects two physical nodes with disjoint Shards and one exact certified recipe/compatibility class.
- [ ] Actual CPU/GPU execution occurs on both nodes; synthetic workers do not satisfy acceptance.
- [ ] Prefill/decode, concurrent-session isolation, telemetry, cancellation, and cleanup pass over the real transport/relay path.
- [ ] Exact hardware, network, backend, model hash, route, commands, and raw metrics are recorded.
- [ ] A model or recipe larger than one participating node's admitted memory is exercised when available.
- [ ] Output drift is measured and incompatible mixed backends fail closed.
- [ ] Targeted pytest tests pass
- [ ] python -m compileall packages tests passes for Python changes
- [ ] git diff --check passes
- [ ] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
- [ ] Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
- [ ] Pinned native C++ target builds and focused CTest/protocol tests pass where native code is touched
- [ ] llama.cpp patch stack applies cleanly to the exact pinned commit where patch code is touched
- [ ] Real-model execution is opt-in through MESHNET_ENABLE_REAL_INFERENCE_TESTS=1 and records exact artifact/runtime/hardware evidence
- [ ] Model artifacts remain on the configured mounted-drive storage and never under /home
- [ ] Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
- [ ] Read and verify every dependency evidence README before relying on dependency behavior
- [ ] Preserve all pre-existing working-tree changes and stage only files belonging to this story
- [ ] Write .scratch/distributed-gguf-runtime/evidence/DGR-011/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
- [ ] Update only this story issue to Status: done after every acceptance criterion and quality gate passes
## Dependency handoff
- `DGR-010` must have `passes: true`; read `../evidence/DGR-010/README.md` and verify its referenced files/commands.
## Finish contract
- Create the task evidence directory and durable handoff required above.
- Preserve real failures and blockers; never fabricate benchmark, model, test or hardware output.
- Change this issue to `Status: done` only after all criteria pass.
- Emit `<promise>COMPLETE</promise>` only after the evidence handoff exists.
## References
- [Ralph execution context](../RALPH-CONTEXT.md)
- [PRD](../PRD.md)
- [Implementation strategy](../implementation-strategy.md)
- [Current architecture](../architecture.md)
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)

63
.scratch/distributed-gguf-runtime/issues/12-implement-continuous-batching-and-bounded-admission.md Normal file
View File

@@ -0,0 +1,63 @@
# 12 — Implement continuous batching and bounded admission
Status: ready-for-agent
## Mandatory fresh-session context
- Read [RALPH-CONTEXT.md](../RALPH-CONTEXT.md) completely before changing code.
- This issue is `DGR-012` in [prd.json](../prd.json).
- Read the evidence README for every dependency listed below.
- Inspect current code and `git status`; historical text and previous agent claims are not evidence.
## Description
As a node operator, I need active sessions batched safely so that concurrency increases aggregate throughput rather than serializing every request.
## Expected durable outputs
- Continuous batching/admission scheduler
- Concurrency 1/2/4/8 report
- Queue, batch and KV-pressure evidence
- evidence/DGR-012/README.md
## Acceptance criteria
- [ ] Node scheduler admits sessions against weight, KV, scratch, and queue budgets.
- [ ] Compatible decode steps from multiple sessions form llama.cpp batches while preserving per-session positions and outputs.
- [ ] Prefill does not starve decode; scheduling policy and bounds are explicit.
- [ ] Backpressure prevents unbounded queued activations or KV growth.
- [ ] Capability telemetry reports active sessions, queue depth, batch occupancy, KV pressure, prefill/decode rates, and rejected admissions.
- [ ] Concurrency 1/2/4/8 benchmark identifies saturation and shows no cross-session corruption.
- [ ] Targeted pytest tests pass
- [ ] python -m compileall packages tests passes for Python changes
- [ ] git diff --check passes
- [ ] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
- [ ] Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
- [ ] Pinned native C++ target builds and focused CTest/protocol tests pass where native code is touched
- [ ] llama.cpp patch stack applies cleanly to the exact pinned commit where patch code is touched
- [ ] Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
- [ ] Read and verify every dependency evidence README before relying on dependency behavior
- [ ] Preserve all pre-existing working-tree changes and stage only files belonging to this story
- [ ] Write .scratch/distributed-gguf-runtime/evidence/DGR-012/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
- [ ] Update only this story issue to Status: done after every acceptance criterion and quality gate passes
## Dependency handoff
- `DGR-007` must have `passes: true`; read `../evidence/DGR-007/README.md` and verify its referenced files/commands.
- `DGR-009` must have `passes: true`; read `../evidence/DGR-009/README.md` and verify its referenced files/commands.
- `DGR-010` must have `passes: true`; read `../evidence/DGR-010/README.md` and verify its referenced files/commands.
## Finish contract
- Create the task evidence directory and durable handoff required above.
- Preserve real failures and blockers; never fabricate benchmark, model, test or hardware output.
- Change this issue to `Status: done` only after all criteria pass.
- Emit `<promise>COMPLETE</promise>` only after the evidence handoff exists.
## References
- [Ralph execution context](../RALPH-CONTEXT.md)
- [PRD](../PRD.md)
- [Implementation strategy](../implementation-strategy.md)
- [Current architecture](../architecture.md)
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)

62
.scratch/distributed-gguf-runtime/issues/13-harden-failure-cancellation-and-restart-semantics.md Normal file
View File

@@ -0,0 +1,62 @@
# 13 — Harden failure, cancellation, and restart semantics
Status: ready-for-agent
## Mandatory fresh-session context
- Read [RALPH-CONTEXT.md](../RALPH-CONTEXT.md) completely before changing code.
- This issue is `DGR-013` in [prd.json](../prd.json).
- Read the evidence README for every dependency listed below.
- Inspect current code and `git status`; historical text and previous agent claims are not evidence.
## Description
As a client, I need failures to be bounded and explicit so that distributed speed does not come with hanging or corrupted generations.
## Expected durable outputs
- Failure/cancel/restart test matrix
- Resource cleanup and billing-state evidence
- evidence/DGR-013/README.md
## Acceptance criteria
- [ ] Deadlines and heartbeat/health loss terminate blocked stream operations.
- [ ] Cancellation propagates across every Shard and releases local KV and queued buffers.
- [ ] Duplicate steps are idempotent; uncertain mutations are never replayed silently.
- [ ] Alpha failover restarts from token zero on a newly compatible route rather than importing unverified KV.
- [ ] Worker death, stream reset, malformed bundle, stale epoch, and cache miss tests pass.
- [ ] Billing/work records distinguish completed, cancelled, failed, and unverified work.
- [ ] Targeted pytest tests pass
- [ ] python -m compileall packages tests passes for Python changes
- [ ] git diff --check passes
- [ ] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
- [ ] Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
- [ ] Pinned native C++ target builds and focused CTest/protocol tests pass where native code is touched
- [ ] llama.cpp patch stack applies cleanly to the exact pinned commit where patch code is touched
- [ ] Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
- [ ] Read and verify every dependency evidence README before relying on dependency behavior
- [ ] Preserve all pre-existing working-tree changes and stage only files belonging to this story
- [ ] Write .scratch/distributed-gguf-runtime/evidence/DGR-013/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
- [ ] Update only this story issue to Status: done after every acceptance criterion and quality gate passes
## Dependency handoff
- `DGR-008` must have `passes: true`; read `../evidence/DGR-008/README.md` and verify its referenced files/commands.
- `DGR-009` must have `passes: true`; read `../evidence/DGR-009/README.md` and verify its referenced files/commands.
- `DGR-012` must have `passes: true`; read `../evidence/DGR-012/README.md` and verify its referenced files/commands.
## Finish contract
- Create the task evidence directory and durable handoff required above.
- Preserve real failures and blockers; never fabricate benchmark, model, test or hardware output.
- Change this issue to `Status: done` only after all criteria pass.
- Emit `<promise>COMPLETE</promise>` only after the evidence handoff exists.
## References
- [Ralph execution context](../RALPH-CONTEXT.md)
- [PRD](../PRD.md)
- [Implementation strategy](../implementation-strategy.md)
- [Current architecture](../architecture.md)
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)

65
.scratch/distributed-gguf-runtime/issues/14-enforce-the-gguf-versus-safetensors-release-gate.md Normal file
View File

@@ -0,0 +1,65 @@
# 14 — Enforce the GGUF-versus-safetensors release gate
Status: ready-for-agent
## Mandatory fresh-session context
- Read [RALPH-CONTEXT.md](../RALPH-CONTEXT.md) completely before changing code.
- This issue is `DGR-014` in [prd.json](../prd.json).
- Read the evidence README for every dependency listed below.
- Inspect current code and `git status`; historical text and previous agent claims are not evidence.
## Description
As the product owner, I need an end-to-end comparison so that the native runtime ships only if it advances model access or performance.
## Expected durable outputs
- Immutable comparison against DGR-001 thresholds
- Machine-readable final report
- Ship/optimize/stop recommendation
- evidence/DGR-014/README.md
## Acceptance criteria
- [ ] Run current distributed safetensors and distributed GGUF routes on the same certified model/hardware/network scenario where technically comparable.
- [ ] Report quality, TTFT, prefill/decode throughput, aggregate concurrency throughput, p95 latency, seam cost, memory, KV pressure, failures, and cleanup.
- [ ] Evaluate against the DGR-001 performance contract without changing thresholds after seeing results.
- [ ] Ship recommendation is one of: promote GGUF, optimize a measured bottleneck with a new bounded task, or stop the native track.
- [ ] Results clearly separate quantization gains from transport/runtime gains.
- [ ] Targeted pytest tests pass
- [ ] python -m compileall packages tests passes for Python changes
- [ ] git diff --check passes
- [ ] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
- [ ] Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
- [ ] Pinned native C++ target builds and focused CTest/protocol tests pass where native code is touched
- [ ] llama.cpp patch stack applies cleanly to the exact pinned commit where patch code is touched
- [ ] Real-model execution is opt-in through MESHNET_ENABLE_REAL_INFERENCE_TESTS=1 and records exact artifact/runtime/hardware evidence
- [ ] Model artifacts remain on the configured mounted-drive storage and never under /home
- [ ] Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
- [ ] Read and verify every dependency evidence README before relying on dependency behavior
- [ ] Preserve all pre-existing working-tree changes and stage only files belonging to this story
- [ ] Write .scratch/distributed-gguf-runtime/evidence/DGR-014/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
- [ ] Update only this story issue to Status: done after every acceptance criterion and quality gate passes
## Dependency handoff
- `DGR-001` must have `passes: true`; read `../evidence/DGR-001/README.md` and verify its referenced files/commands.
- `DGR-011` must have `passes: true`; read `../evidence/DGR-011/README.md` and verify its referenced files/commands.
- `DGR-012` must have `passes: true`; read `../evidence/DGR-012/README.md` and verify its referenced files/commands.
- `DGR-013` must have `passes: true`; read `../evidence/DGR-013/README.md` and verify its referenced files/commands.
## Finish contract
- Create the task evidence directory and durable handoff required above.
- Preserve real failures and blockers; never fabricate benchmark, model, test or hardware output.
- Change this issue to `Status: done` only after all criteria pass.
- Emit `<promise>COMPLETE</promise>` only after the evidence handoff exists.
## References
- [Ralph execution context](../RALPH-CONTEXT.md)
- [PRD](../PRD.md)
- [Implementation strategy](../implementation-strategy.md)
- [Current architecture](../architecture.md)
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)

61
.scratch/distributed-gguf-runtime/issues/15-add-and-certify-a-qwen3-qwen3-moe-adapter.md Normal file
View File

@@ -0,0 +1,61 @@
# 15 — Add and certify a Qwen3/Qwen3-MoE adapter
Status: ready-for-agent
## Mandatory fresh-session context
- Read [RALPH-CONTEXT.md](../RALPH-CONTEXT.md) completely before changing code.
- This issue is `DGR-015` in [prd.json](../prd.json).
- Read the evidence README for every dependency listed below.
- Inspect current code and `git status`; historical text and previous agent claims are not evidence.
## Description
As a client seeking top models, I need a separately certified MoE-capable architecture after the dense runtime proves stable.
## Expected durable outputs
- Qwen3-family architecture adapter
- Architecture-specific parity/admission/performance results
- evidence/DGR-015/README.md
## Acceptance criteria
- [ ] Implement explicit tensor ownership, router/top-k, expert/shared-expert, Q/K normalization, boundary bundle, and cache semantics for the selected Qwen3 family recipe.
- [ ] Do not reuse the dense-Llama adapter through unchecked name substitutions.
- [ ] Whole-model versus distributed prefill/decode parity passes the architecture-specific tolerance.
- [ ] Expert memory ownership and communication are measured.
- [ ] Real consumer-hardware acceptance and capability admission pass before the recipe becomes routable.
- [ ] Targeted pytest tests pass
- [ ] python -m compileall packages tests passes for Python changes
- [ ] git diff --check passes
- [ ] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
- [ ] Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
- [ ] Pinned native C++ target builds and focused CTest/protocol tests pass where native code is touched
- [ ] llama.cpp patch stack applies cleanly to the exact pinned commit where patch code is touched
- [ ] Real-model execution is opt-in through MESHNET_ENABLE_REAL_INFERENCE_TESTS=1 and records exact artifact/runtime/hardware evidence
- [ ] Model artifacts remain on the configured mounted-drive storage and never under /home
- [ ] Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
- [ ] Read and verify every dependency evidence README before relying on dependency behavior
- [ ] Preserve all pre-existing working-tree changes and stage only files belonging to this story
- [ ] Write .scratch/distributed-gguf-runtime/evidence/DGR-015/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
- [ ] Update only this story issue to Status: done after every acceptance criterion and quality gate passes
## Dependency handoff
- `DGR-014` must have `passes: true`; read `../evidence/DGR-014/README.md` and verify its referenced files/commands.
## Finish contract
- Create the task evidence directory and durable handoff required above.
- Preserve real failures and blockers; never fabricate benchmark, model, test or hardware output.
- Change this issue to `Status: done` only after all criteria pass.
- Emit `<promise>COMPLETE</promise>` only after the evidence handoff exists.
## References
- [Ralph execution context](../RALPH-CONTEXT.md)
- [PRD](../PRD.md)
- [Implementation strategy](../implementation-strategy.md)
- [Current architecture](../architecture.md)
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)

60
.scratch/distributed-gguf-runtime/issues/16-produce-the-upstream-llama-cpp-collaboration-package.md Normal file
View File

@@ -0,0 +1,60 @@
# 16 — Produce the upstream llama.cpp collaboration package
Status: ready-for-agent
## Mandatory fresh-session context
- Read [RALPH-CONTEXT.md](../RALPH-CONTEXT.md) completely before changing code.
- This issue is `DGR-016` in [prd.json](../prd.json).
- Read the evidence README for every dependency listed below.
- Inspect current code and `git status`; historical text and previous agent claims are not evidence.
## Description
As a maintainer, I need narrow upstreamable proposals so that our patch burden can shrink without asking llama.cpp to own Meshnet networking.
## Expected durable outputs
- Narrow upstream patches/tests
- Generic API design note
- Human-ready llama.cpp outreach package
- evidence/DGR-016/README.md
## Acceptance criteria
- [ ] Separate generic llama.cpp hooks from Meshnet protocol/control-plane code.
- [ ] Prepare minimal reproducible examples and tests for range-aware loading, boundary input/output, and layer-filtered KV.
- [ ] Compare the proposal with Nakshatra and prima.cpp evidence and explain why the API is generally useful.
- [ ] Preserve one scoped commit/patch per concern against the exact upstream pin.
- [ ] Produce an outreach document suitable for Georgi/llama.cpp maintainers; actual sending remains a human action.
- [ ] Targeted pytest tests pass
- [ ] python -m compileall packages tests passes for Python changes
- [ ] git diff --check passes
- [ ] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
- [ ] Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
- [ ] Pinned native C++ target builds and focused CTest/protocol tests pass where native code is touched
- [ ] llama.cpp patch stack applies cleanly to the exact pinned commit where patch code is touched
- [ ] Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
- [ ] Read and verify every dependency evidence README before relying on dependency behavior
- [ ] Preserve all pre-existing working-tree changes and stage only files belonging to this story
- [ ] Write .scratch/distributed-gguf-runtime/evidence/DGR-016/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
- [ ] Update only this story issue to Status: done after every acceptance criterion and quality gate passes
## Dependency handoff
- `DGR-010` must have `passes: true`; read `../evidence/DGR-010/README.md` and verify its referenced files/commands.
## Finish contract
- Create the task evidence directory and durable handoff required above.
- Preserve real failures and blockers; never fabricate benchmark, model, test or hardware output.
- Change this issue to `Status: done` only after all criteria pass.
- Emit `<promise>COMPLETE</promise>` only after the evidence handoff exists.
## References
- [Ralph execution context](../RALPH-CONTEXT.md)
- [PRD](../PRD.md)
- [Implementation strategy](../implementation-strategy.md)
- [Current architecture](../architecture.md)
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)

51
.scratch/distributed-gguf-runtime/milestones.md
View File

@@ -1,32 +1,35 @@
# Distributed GGUF Runtime Milestones
# Distributed GGUF runtime milestones
## Proposed Breakdown
## Gate A — measured runtime value
| Order | Issue | Title | Blocked by | User-visible proof |
|---:|---|---|---|---|
| 1 | [01](./issues/01-route-session-lifecycle.md) | Route Session lifecycle | None | Stable route/session status and cleanup |
| 2 | [02](./issues/02-prefill-decode-binary-http.md) | Prefill/decode binary HTTP protocol | 01 | Stub route proves prefill chunks and one-step decode payloads |
| 3 | [03](./issues/03-generation-telemetry-and-streaming.md) | Generation Telemetry and streaming response contract | 01 | Client sees route progress and streamed deltas when available |
| 4 | [04](./issues/04-pytorch-distributed-kv-reference.md) | PyTorch distributed KV reference route | 01, 02, 03 | Distributed PyTorch decode stops full-prompt recompute |
| 5 | [05](./issues/05-local-llamacpp-gguf-backend.md) | Local llama.cpp/GGUF backend | None | Local GGUF model serves through node API |
| 6 | [06](./issues/06-model-artifact-manifest.md) | Model Artifact manifest and Shard advertisement | 01 | Node verifies artifacts and advertises serveable Shards |
| 7 | [07](./issues/07-llamacpp-layer-boundary-prototype.md) | llama.cpp layer-boundary prototype | 02, 05, 06 | Local two-process GGUF route identifies upstream API |
| 8 | [08](./issues/08-networked-distributed-gguf-route.md) | Networked distributed GGUF route | 03, 04, 06, 07 | Two machines serve one GGUF route with telemetry |
| 9 | [09](./issues/09-deepseek-v4-flash-support-audit.md) | DeepSeek-V4-Flash support audit | None | Runtime recommendation for first serious large model |
| 10 | [10](./issues/10-glm52-ornith-followup-audit.md) | GLM-5.2 and Ornith follow-up support audit | 09 | Follow-up compatibility matrix and upstream blockers |
- DGR-001 locks the safetensors-versus-GGUF performance/fit/quality contract.
- DGR-002 can proceed independently and defines the battle-proven backend-neutral wire protocol.
- DGR-003 builds exact recipe identity on DGR-002.
- Expensive native llama.cpp work remains gated by DGR-001.
## First Three To Implement
## Gate B — minimal native execution seam
1. **01 — Route Session lifecycle**: makes every later cache, telemetry, and route decision concrete.
2. **02 — Prefill/decode binary HTTP protocol**: proves the payload shape and route/session headers before model internals.
3. **03 — Generation Telemetry and streaming response contract**: gives every later long-running route a visible user experience and failure surface.
- DGR-004 creates the reproducible pinned fork boundary.
- DGR-005 implements dense-Llama range ownership.
- DGR-006 proves architecture-defined boundary parity.
## Parallel Work
## Gate C — concurrent production worker
- **05 — Local llama.cpp/GGUF backend** can run in parallel with 01–03 because it is a full-model local backend.
- **09 — DeepSeek-V4-Flash support audit** can run in parallel because it is research/compatibility work.
- DGR-007 isolates concurrent Hot KV State.
- DGR-008 exposes the native worker over gRPC.
- DGR-009 integrates the worker without replacing Meshnet's control plane.
- DGR-010 passes local real-model two-process acceptance.
## Human-Gated Work
## Gate D — real consumer-hardware route
- **07 — llama.cpp layer-boundary prototype** is the collaboration point with Georgi/upstream llama.cpp.
- **08 — Networked distributed GGUF route** should wait until the PyTorch reference route proves the cache/session contract.
- DGR-011 passes two-physical-machine execution.
- DGR-012 adds continuous batching and bounded admission.
- DGR-013 hardens failure and cancellation.
## Gate E — product release decision
- DGR-014 compares distributed GGUF against the current distributed safetensors route under locked thresholds.
- DGR-015 adds Qwen3/Qwen3-MoE only after the dense runtime passes.
- DGR-016 prepares narrow upstream llama.cpp collaboration material.
No later gate may be claimed from synthetic workers or documentation-only evidence.

511
.scratch/distributed-gguf-runtime/prd.json Normal file
View File

@@ -0,0 +1,511 @@
{
"name": "Performant Concurrent Distributed GGUF Runtime",
"branchName": "ralph/performant-concurrent-distributed-gguf",
"description": "Benchmark-gated native llama.cpp/GGUF Shards with gRPC streaming, concurrent local KV, continuous batching, real heterogeneous acceptance, and a measured release gate against Transformers/safetensors.",
"userStories": [
{
"id": "DGR-001",
"title": "Lock the safetensors-versus-GGUF performance contract",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/01-lock-the-safetensors-versus-gguf-performance-contract.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs a runtime engineer, I need a controlled baseline so that GGUF work proceeds from measured speed, memory, and quality rather than reputation.",
"acceptanceCriteria": [
"Benchmark the same model architecture/revision, machine, prompts, context lengths, output lengths, sampling policy, and concurrency across the current Transformers/safetensors recipe and whole-model llama.cpp recipes.",
"Separate correctness/quality lanes from quantized performance/fit lanes instead of claiming BF16 and Q4 are numerically equivalent.",
"Report TTFT, prefill tok/s, decode tok/s, p50/p95 latency, aggregate throughput, RSS, VRAM, artifact size, failures, and output drift in machine-readable JSON.",
"Add concurrency levels 1 and 4 where memory permits.",
"Write a versioned performance contract consumed by later release gates, including an explicit stop condition when llama.cpp/GGUF has no meaningful speed or fit benefit.",
"Targeted pytest tests pass",
"python -m compileall packages tests passes for Python changes",
"git diff --check passes",
"Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free",
"Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction",
"Real-model execution is opt-in through MESHNET_ENABLE_REAL_INFERENCE_TESTS=1 and records exact artifact/runtime/hardware evidence",
"Model artifacts remain on the configured mounted-drive storage and never under /home",
"Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code",
"Read and verify every dependency evidence README before relying on dependency behavior",
"Preserve all pre-existing working-tree changes and stage only files belonging to this story",
"Write .scratch/distributed-gguf-runtime/evidence/DGR-001/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff",
"Update only this story issue to Status: done after every acceptance criterion and quality gate passes"
],
"priority": 2,
"passes": false,
"notes": "Source issue: .scratch/distributed-gguf-runtime/issues/01-lock-the-safetensors-versus-gguf-performance-contract.md",
"dependsOn": []
},
{
"id": "DGR-002",
"title": "Adopt the versioned gRPC Shard protocol",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/02-adopt-the-versioned-grpc-shard-protocol.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs a node developer, I need a battle-proven streaming protocol so that Python and C++ Shards communicate without a custom socket protocol.",
"acceptanceCriteria": [
"Add a Protocol Buffers schema for capability, health, session stream, release, and cancellation operations.",
"Define one long-lived bidirectional gRPC stream per Route Session Activation Seam with deadlines, cancellation, flow control, and structured errors.",
"Define bounded chunking for prefill and a small decode fast path.",
"Carry schema version, request/work ID, Route Session ID, route epoch, artifact/recipe fingerprint, Shard range/effective start, phase, position, idempotency step, cache expectation, compression, and checksum.",
"Define a versioned named-tensor bundle with per-tensor name, shape, dtype, byte order, and payload fragments.",
"Add generated-schema round-trip and compatibility tests in Python and C++.",
"Targeted pytest tests pass",
"python -m compileall packages tests passes for Python changes",
"git diff --check passes",
"Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free",
"Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction",
"Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code",
"Read and verify every dependency evidence README before relying on dependency behavior",
"Preserve all pre-existing working-tree changes and stage only files belonging to this story",
"Write .scratch/distributed-gguf-runtime/evidence/DGR-002/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff",
"Update only this story issue to Status: done after every acceptance criterion and quality gate passes"
],
"priority": 1,
"passes": false,
"notes": "Source issue: .scratch/distributed-gguf-runtime/issues/02-adopt-the-versioned-grpc-shard-protocol.md",
"dependsOn": []
},
{
"id": "DGR-003",
"title": "Define exact Artifact and runtime recipe identity",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/03-define-exact-artifact-and-runtime-recipe-identity.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs the Tracker, I need exact compatibility identity so that only numerically and operationally compatible Shards form an Inference Route.",
"acceptanceCriteria": [
"Separate weight quantization, activation dtype, compute dtype, KV dtype/layout, tokenizer revision, architecture adapter, backend, and runtime version.",
"Bind derivative or split artifacts to an exact source Model Artifact hash and Shard range.",
"Produce a stable compatibility fingerprint used by capability admission and the gRPC handshake.",
"Fail closed on mismatched artifact, tokenizer, architecture, range, boundary schema, activation recipe, or cache layout.",
"Keep unsupported recipes registered-but-dark until a real distributed forward certifies them.",
"Targeted pytest tests pass",
"python -m compileall packages tests passes for Python changes",
"git diff --check passes",
"Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free",
"Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction",
"Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code",
"Read and verify every dependency evidence README before relying on dependency behavior",
"Preserve all pre-existing working-tree changes and stage only files belonging to this story",
"Write .scratch/distributed-gguf-runtime/evidence/DGR-003/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff",
"Update only this story issue to Status: done after every acceptance criterion and quality gate passes"
],
"priority": 3,
"passes": false,
"notes": "Source issue: .scratch/distributed-gguf-runtime/issues/03-define-exact-artifact-and-runtime-recipe-identity.md",
"dependsOn": [
"DGR-002"
]
},
{
"id": "DGR-004",
"title": "Create the reproducible pinned llama.cpp patch stack",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/04-create-the-reproducible-pinned-llama-cpp-patch-stack.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs a maintainer, I need a small auditable fork boundary so that upstream updates do not turn the runtime into an unmaintainable stitched codebase.",
"acceptanceCriteria": [
"Pin one exact llama.cpp commit through a reproducible source dependency mechanism.",
"Store a numbered minimal patch stack separately from Meshnet networking code.",
"Add a build script that applies/checks patches and builds the standalone worker without manual source copying.",
"Record upstream file/ABI assumptions and fail clearly when the pin changes.",
"Preserve upstream license and attribution notices.",
"Add a clean rebuild smoke test that does not download a model.",
"Targeted pytest tests pass",
"python -m compileall packages tests passes for Python changes",
"git diff --check passes",
"Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free",
"Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction",
"Pinned native C++ target builds and focused CTest/protocol tests pass where native code is touched",
"llama.cpp patch stack applies cleanly to the exact pinned commit where patch code is touched",
"Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code",
"Read and verify every dependency evidence README before relying on dependency behavior",
"Preserve all pre-existing working-tree changes and stage only files belonging to this story",
"Write .scratch/distributed-gguf-runtime/evidence/DGR-004/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff",
"Update only this story issue to Status: done after every acceptance criterion and quality gate passes"
],
"priority": 4,
"passes": false,
"notes": "Source issue: .scratch/distributed-gguf-runtime/issues/04-create-the-reproducible-pinned-llama-cpp-patch-stack.md",
"dependsOn": [
"DGR-001"
]
},
{
"id": "DGR-005",
"title": "Implement dense-Llama range-aware GGUF ownership",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/05-implement-dense-llama-range-aware-gguf-ownership.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs a node, I need to map only my assigned dense-Llama Shard so that aggregate consumer memory can hold a model larger than one node.",
"acceptanceCriteria": [
"Register and allocate only `blk.N.*` tensors in the assigned range.",
"Load embeddings only for the head and final norm/LM head only for the tail, including tied embeddings.",
"Prefer range-aware mapping from one exact source GGUF; if derivative sub-GGUFs are used temporarily, verify source/slice hashes and avoid claiming final artifact semantics.",
"Report authoritative loaded range and endpoint ownership from the model, not operator CLI claims.",
"Demonstrate mapped/resident memory scales with owned tensors rather than full model size.",
"Targeted pytest tests pass",
"python -m compileall packages tests passes for Python changes",
"git diff --check passes",
"Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free",
"Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction",
"Pinned native C++ target builds and focused CTest/protocol tests pass where native code is touched",
"llama.cpp patch stack applies cleanly to the exact pinned commit where patch code is touched",
"Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code",
"Read and verify every dependency evidence README before relying on dependency behavior",
"Preserve all pre-existing working-tree changes and stage only files belonging to this story",
"Write .scratch/distributed-gguf-runtime/evidence/DGR-005/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff",
"Update only this story issue to Status: done after every acceptance criterion and quality gate passes"
],
"priority": 5,
"passes": false,
"notes": "Source issue: .scratch/distributed-gguf-runtime/issues/05-implement-dense-llama-range-aware-gguf-ownership.md",
"dependsOn": [
"DGR-003",
"DGR-004"
]
},
{
"id": "DGR-006",
"title": "Implement architecture-defined boundary input/output",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/06-implement-architecture-defined-boundary-input-output.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs a Shard, I need to consume and emit the correct transformer boundary state so that disjoint processes reproduce whole-model execution.",
"acceptanceCriteria": [
"Head accepts token IDs and owns token embedding.",
"Middle/tail bypass token embedding and accept the named boundary bundle.",
"Non-tail emits the unnormalized architecture-defined residual/boundary before final norm/head and before tail-only row pruning.",
"Tail emits logits or token output through an explicit sampling contract.",
"Dense-Llama whole-model versus two-range prefill and greedy-decode parity passes the documented tolerance.",
"The adapter interface fails closed for uncertified architectures.",
"Targeted pytest tests pass",
"python -m compileall packages tests passes for Python changes",
"git diff --check passes",
"Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free",
"Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction",
"Pinned native C++ target builds and focused CTest/protocol tests pass where native code is touched",
"llama.cpp patch stack applies cleanly to the exact pinned commit where patch code is touched",
"Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code",
"Read and verify every dependency evidence README before relying on dependency behavior",
"Preserve all pre-existing working-tree changes and stage only files belonging to this story",
"Write .scratch/distributed-gguf-runtime/evidence/DGR-006/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff",
"Update only this story issue to Status: done after every acceptance criterion and quality gate passes"
],
"priority": 6,
"passes": false,
"notes": "Source issue: .scratch/distributed-gguf-runtime/issues/06-implement-architecture-defined-boundary-input-output.md",
"dependsOn": [
"DGR-002",
"DGR-005"
]
},
{
"id": "DGR-007",
"title": "Add isolated concurrent local Hot KV State",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/07-add-isolated-concurrent-local-hot-kv-state.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs a client, I need concurrent Route Sessions to retain independent per-Shard cache so that one request cannot clear or corrupt another.",
"acceptanceCriteria": [
"Map `(Route Session ID, route epoch)` to an isolated llama sequence or bounded context.",
"Allocate KV only for owned layers.",
"Support prefill append, decode append, truncate, release, TTL/LRU eviction, and explicit cache-miss response.",
"Reject stale epochs and incompatible cache recipes.",
"At least four concurrent sessions on a small model complete without token or KV cross-talk.",
"Cancellation/release of one session leaves other sessions intact and memory returns to the configured budget.",
"Targeted pytest tests pass",
"python -m compileall packages tests passes for Python changes",
"git diff --check passes",
"Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free",
"Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction",
"Pinned native C++ target builds and focused CTest/protocol tests pass where native code is touched",
"llama.cpp patch stack applies cleanly to the exact pinned commit where patch code is touched",
"Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code",
"Read and verify every dependency evidence README before relying on dependency behavior",
"Preserve all pre-existing working-tree changes and stage only files belonging to this story",
"Write .scratch/distributed-gguf-runtime/evidence/DGR-007/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff",
"Update only this story issue to Status: done after every acceptance criterion and quality gate passes"
],
"priority": 7,
"passes": false,
"notes": "Source issue: .scratch/distributed-gguf-runtime/issues/07-add-isolated-concurrent-local-hot-kv-state.md",
"dependsOn": [
"DGR-006"
]
},
{
"id": "DGR-008",
"title": "Build the standalone C++ gRPC Shard worker",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/08-build-the-standalone-c-grpc-shard-worker.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs a node runtime, I need one supervised native process so that llama.cpp internals remain behind a stable project-owned protocol.",
"acceptanceCriteria": [
"Worker exposes capability, health, session stream, release, cancellation, and metrics services from DGR-002.",
"Worker loads one exact Artifact/recipe/Shard identity and refuses mismatched requests.",
"Streaming path enforces bounded messages, flow control, deadlines, idempotency, and independent session cancellation.",
"Worker does not expose raw llama.cpp RPC or arbitrary GGML graph execution.",
"Graceful shutdown releases sessions; crash behavior is bounded and observable.",
"Python integration tests run against a fake model mode without model downloads.",
"Targeted pytest tests pass",
"python -m compileall packages tests passes for Python changes",
"git diff --check passes",
"Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free",
"Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction",
"Pinned native C++ target builds and focused CTest/protocol tests pass where native code is touched",
"llama.cpp patch stack applies cleanly to the exact pinned commit where patch code is touched",
"Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code",
"Read and verify every dependency evidence README before relying on dependency behavior",
"Preserve all pre-existing working-tree changes and stage only files belonging to this story",
"Write .scratch/distributed-gguf-runtime/evidence/DGR-008/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff",
"Update only this story issue to Status: done after every acceptance criterion and quality gate passes"
],
"priority": 8,
"passes": false,
"notes": "Source issue: .scratch/distributed-gguf-runtime/issues/08-build-the-standalone-c-grpc-shard-worker.md",
"dependsOn": [
"DGR-002",
"DGR-003",
"DGR-004",
"DGR-006",
"DGR-007"
]
},
{
"id": "DGR-009",
"title": "Integrate the native worker with Meshnet",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/09-integrate-the-native-worker-with-meshnet.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs the existing node service, I need a GGUF Shard backend adapter so that the Tracker, relay, billing, telemetry, and capability admission remain the sole control plane.",
"acceptanceCriteria": [
"Implement the existing model-backend surface without changing Transformers behavior.",
"Registration carries exact validated GGUF recipe, Shard, backend and concurrency/KV capacity.",
"Tracker forms only complete compatible routes and keeps uncertified recipes dark.",
"Direct routes use gRPC streams; relayed routes carry the same versioned protobuf frames as opaque binary through the existing relay seam.",
"Existing request/work IDs, cancellation, Generation Telemetry, billing, and per-node attribution remain correlated.",
"No vLLM, Nakshatra, prima.cpp, or custom-engine control plane becomes a core dependency.",
"Targeted pytest tests pass",
"python -m compileall packages tests passes for Python changes",
"git diff --check passes",
"Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free",
"Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction",
"Pinned native C++ target builds and focused CTest/protocol tests pass where native code is touched",
"llama.cpp patch stack applies cleanly to the exact pinned commit where patch code is touched",
"Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code",
"Read and verify every dependency evidence README before relying on dependency behavior",
"Preserve all pre-existing working-tree changes and stage only files belonging to this story",
"Write .scratch/distributed-gguf-runtime/evidence/DGR-009/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff",
"Update only this story issue to Status: done after every acceptance criterion and quality gate passes"
],
"priority": 9,
"passes": false,
"notes": "Source issue: .scratch/distributed-gguf-runtime/issues/09-integrate-the-native-worker-with-meshnet.md",
"dependsOn": [
"DGR-003",
"DGR-008"
]
},
{
"id": "DGR-010",
"title": "Pass local real-model two-process acceptance",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/10-pass-local-real-model-two-process-acceptance.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs a release engineer, I need real local distributed parity before involving network variability.",
"acceptanceCriteria": [
"Two local worker processes open disjoint dense-Llama ranges from the certified Artifact.",
"Prefill and at least 32 greedy decode tokens match whole-model llama.cpp within the certified tolerance.",
"Each worker retains only its own tensors and Hot KV State.",
"Four concurrent Route Sessions pass isolation and cleanup checks.",
"Report TTFT, prefill/decode throughput, seam bytes/latency, worker RSS/VRAM, KV memory, batch size, and queue time.",
"Killing one worker produces a bounded structured failure rather than a deadlock.",
"Targeted pytest tests pass",
"python -m compileall packages tests passes for Python changes",
"git diff --check passes",
"Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free",
"Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction",
"Pinned native C++ target builds and focused CTest/protocol tests pass where native code is touched",
"llama.cpp patch stack applies cleanly to the exact pinned commit where patch code is touched",
"Real-model execution is opt-in through MESHNET_ENABLE_REAL_INFERENCE_TESTS=1 and records exact artifact/runtime/hardware evidence",
"Model artifacts remain on the configured mounted-drive storage and never under /home",
"Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code",
"Read and verify every dependency evidence README before relying on dependency behavior",
"Preserve all pre-existing working-tree changes and stage only files belonging to this story",
"Write .scratch/distributed-gguf-runtime/evidence/DGR-010/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff",
"Update only this story issue to Status: done after every acceptance criterion and quality gate passes"
],
"priority": 10,
"passes": false,
"notes": "Source issue: .scratch/distributed-gguf-runtime/issues/10-pass-local-real-model-two-process-acceptance.md",
"dependsOn": [
"DGR-009"
]
},
{
"id": "DGR-011",
"title": "Pass a real heterogeneous two-machine route",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/11-pass-a-real-heterogeneous-two-machine-route.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs a consumer-hardware operator, I need two physical machines to execute one GGUF model so that the distributed claim is real.",
"acceptanceCriteria": [
"Tracker selects two physical nodes with disjoint Shards and one exact certified recipe/compatibility class.",
"Actual CPU/GPU execution occurs on both nodes; synthetic workers do not satisfy acceptance.",
"Prefill/decode, concurrent-session isolation, telemetry, cancellation, and cleanup pass over the real transport/relay path.",
"Exact hardware, network, backend, model hash, route, commands, and raw metrics are recorded.",
"A model or recipe larger than one participating node's admitted memory is exercised when available.",
"Output drift is measured and incompatible mixed backends fail closed.",
"Targeted pytest tests pass",
"python -m compileall packages tests passes for Python changes",
"git diff --check passes",
"Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free",
"Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction",
"Pinned native C++ target builds and focused CTest/protocol tests pass where native code is touched",
"llama.cpp patch stack applies cleanly to the exact pinned commit where patch code is touched",
"Real-model execution is opt-in through MESHNET_ENABLE_REAL_INFERENCE_TESTS=1 and records exact artifact/runtime/hardware evidence",
"Model artifacts remain on the configured mounted-drive storage and never under /home",
"Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code",
"Read and verify every dependency evidence README before relying on dependency behavior",
"Preserve all pre-existing working-tree changes and stage only files belonging to this story",
"Write .scratch/distributed-gguf-runtime/evidence/DGR-011/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff",
"Update only this story issue to Status: done after every acceptance criterion and quality gate passes"
],
"priority": 11,
"passes": false,
"notes": "Source issue: .scratch/distributed-gguf-runtime/issues/11-pass-a-real-heterogeneous-two-machine-route.md",
"dependsOn": [
"DGR-010"
]
},
{
"id": "DGR-012",
"title": "Implement continuous batching and bounded admission",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/12-implement-continuous-batching-and-bounded-admission.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs a node operator, I need active sessions batched safely so that concurrency increases aggregate throughput rather than serializing every request.",
"acceptanceCriteria": [
"Node scheduler admits sessions against weight, KV, scratch, and queue budgets.",
"Compatible decode steps from multiple sessions form llama.cpp batches while preserving per-session positions and outputs.",
"Prefill does not starve decode; scheduling policy and bounds are explicit.",
"Backpressure prevents unbounded queued activations or KV growth.",
"Capability telemetry reports active sessions, queue depth, batch occupancy, KV pressure, prefill/decode rates, and rejected admissions.",
"Concurrency 1/2/4/8 benchmark identifies saturation and shows no cross-session corruption.",
"Targeted pytest tests pass",
"python -m compileall packages tests passes for Python changes",
"git diff --check passes",
"Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free",
"Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction",
"Pinned native C++ target builds and focused CTest/protocol tests pass where native code is touched",
"llama.cpp patch stack applies cleanly to the exact pinned commit where patch code is touched",
"Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code",
"Read and verify every dependency evidence README before relying on dependency behavior",
"Preserve all pre-existing working-tree changes and stage only files belonging to this story",
"Write .scratch/distributed-gguf-runtime/evidence/DGR-012/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff",
"Update only this story issue to Status: done after every acceptance criterion and quality gate passes"
],
"priority": 12,
"passes": false,
"notes": "Source issue: .scratch/distributed-gguf-runtime/issues/12-implement-continuous-batching-and-bounded-admission.md",
"dependsOn": [
"DGR-007",
"DGR-009",
"DGR-010"
]
},
{
"id": "DGR-013",
"title": "Harden failure, cancellation, and restart semantics",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/13-harden-failure-cancellation-and-restart-semantics.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs a client, I need failures to be bounded and explicit so that distributed speed does not come with hanging or corrupted generations.",
"acceptanceCriteria": [
"Deadlines and heartbeat/health loss terminate blocked stream operations.",
"Cancellation propagates across every Shard and releases local KV and queued buffers.",
"Duplicate steps are idempotent; uncertain mutations are never replayed silently.",
"Alpha failover restarts from token zero on a newly compatible route rather than importing unverified KV.",
"Worker death, stream reset, malformed bundle, stale epoch, and cache miss tests pass.",
"Billing/work records distinguish completed, cancelled, failed, and unverified work.",
"Targeted pytest tests pass",
"python -m compileall packages tests passes for Python changes",
"git diff --check passes",
"Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free",
"Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction",
"Pinned native C++ target builds and focused CTest/protocol tests pass where native code is touched",
"llama.cpp patch stack applies cleanly to the exact pinned commit where patch code is touched",
"Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code",
"Read and verify every dependency evidence README before relying on dependency behavior",
"Preserve all pre-existing working-tree changes and stage only files belonging to this story",
"Write .scratch/distributed-gguf-runtime/evidence/DGR-013/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff",
"Update only this story issue to Status: done after every acceptance criterion and quality gate passes"
],
"priority": 13,
"passes": false,
"notes": "Source issue: .scratch/distributed-gguf-runtime/issues/13-harden-failure-cancellation-and-restart-semantics.md",
"dependsOn": [
"DGR-008",
"DGR-009",
"DGR-012"
]
},
{
"id": "DGR-014",
"title": "Enforce the GGUF-versus-safetensors release gate",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/14-enforce-the-gguf-versus-safetensors-release-gate.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs the product owner, I need an end-to-end comparison so that the native runtime ships only if it advances model access or performance.",
"acceptanceCriteria": [
"Run current distributed safetensors and distributed GGUF routes on the same certified model/hardware/network scenario where technically comparable.",
"Report quality, TTFT, prefill/decode throughput, aggregate concurrency throughput, p95 latency, seam cost, memory, KV pressure, failures, and cleanup.",
"Evaluate against the DGR-001 performance contract without changing thresholds after seeing results.",
"Ship recommendation is one of: promote GGUF, optimize a measured bottleneck with a new bounded task, or stop the native track.",
"Results clearly separate quantization gains from transport/runtime gains.",
"Targeted pytest tests pass",
"python -m compileall packages tests passes for Python changes",
"git diff --check passes",
"Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free",
"Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction",
"Pinned native C++ target builds and focused CTest/protocol tests pass where native code is touched",
"llama.cpp patch stack applies cleanly to the exact pinned commit where patch code is touched",
"Real-model execution is opt-in through MESHNET_ENABLE_REAL_INFERENCE_TESTS=1 and records exact artifact/runtime/hardware evidence",
"Model artifacts remain on the configured mounted-drive storage and never under /home",
"Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code",
"Read and verify every dependency evidence README before relying on dependency behavior",
"Preserve all pre-existing working-tree changes and stage only files belonging to this story",
"Write .scratch/distributed-gguf-runtime/evidence/DGR-014/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff",
"Update only this story issue to Status: done after every acceptance criterion and quality gate passes"
],
"priority": 14,
"passes": false,
"notes": "Source issue: .scratch/distributed-gguf-runtime/issues/14-enforce-the-gguf-versus-safetensors-release-gate.md",
"dependsOn": [
"DGR-001",
"DGR-011",
"DGR-012",
"DGR-013"
]
},
{
"id": "DGR-015",
"title": "Add and certify a Qwen3/Qwen3-MoE adapter",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/15-add-and-certify-a-qwen3-qwen3-moe-adapter.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs a client seeking top models, I need a separately certified MoE-capable architecture after the dense runtime proves stable.",
"acceptanceCriteria": [
"Implement explicit tensor ownership, router/top-k, expert/shared-expert, Q/K normalization, boundary bundle, and cache semantics for the selected Qwen3 family recipe.",
"Do not reuse the dense-Llama adapter through unchecked name substitutions.",
"Whole-model versus distributed prefill/decode parity passes the architecture-specific tolerance.",
"Expert memory ownership and communication are measured.",
"Real consumer-hardware acceptance and capability admission pass before the recipe becomes routable.",
"Targeted pytest tests pass",
"python -m compileall packages tests passes for Python changes",
"git diff --check passes",
"Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free",
"Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction",
"Pinned native C++ target builds and focused CTest/protocol tests pass where native code is touched",
"llama.cpp patch stack applies cleanly to the exact pinned commit where patch code is touched",
"Real-model execution is opt-in through MESHNET_ENABLE_REAL_INFERENCE_TESTS=1 and records exact artifact/runtime/hardware evidence",
"Model artifacts remain on the configured mounted-drive storage and never under /home",
"Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code",
"Read and verify every dependency evidence README before relying on dependency behavior",
"Preserve all pre-existing working-tree changes and stage only files belonging to this story",
"Write .scratch/distributed-gguf-runtime/evidence/DGR-015/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff",
"Update only this story issue to Status: done after every acceptance criterion and quality gate passes"
],
"priority": 15,
"passes": false,
"notes": "Source issue: .scratch/distributed-gguf-runtime/issues/15-add-and-certify-a-qwen3-qwen3-moe-adapter.md",
"dependsOn": [
"DGR-014"
]
},
{
"id": "DGR-016",
"title": "Produce the upstream llama.cpp collaboration package",
"description": "MANDATORY FRESH-SESSION CONTEXT: Read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md` and `.scratch/distributed-gguf-runtime/issues/16-produce-the-upstream-llama-cpp-collaboration-package.md` completely before coding. Read the evidence handoff for every dependency. The global goal is performant concurrent inference for models larger than one consumer node, using Meshnet as the sole control plane, gRPC/Protobuf as the Shard protocol, and a small pinned llama.cpp worker\u2014not a stitched collection of runtimes.\n\nAs a maintainer, I need narrow upstreamable proposals so that our patch burden can shrink without asking llama.cpp to own Meshnet networking.",
"acceptanceCriteria": [
"Separate generic llama.cpp hooks from Meshnet protocol/control-plane code.",
"Prepare minimal reproducible examples and tests for range-aware loading, boundary input/output, and layer-filtered KV.",
"Compare the proposal with Nakshatra and prima.cpp evidence and explain why the API is generally useful.",
"Preserve one scoped commit/patch per concern against the exact upstream pin.",
"Produce an outreach document suitable for Georgi/llama.cpp maintainers; actual sending remains a human action.",
"Targeted pytest tests pass",
"python -m compileall packages tests passes for Python changes",
"git diff --check passes",
"Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free",
"Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction",
"Pinned native C++ target builds and focused CTest/protocol tests pass where native code is touched",
"llama.cpp patch stack applies cleanly to the exact pinned commit where patch code is touched",
"Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code",
"Read and verify every dependency evidence README before relying on dependency behavior",
"Preserve all pre-existing working-tree changes and stage only files belonging to this story",
"Write .scratch/distributed-gguf-runtime/evidence/DGR-016/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff",
"Update only this story issue to Status: done after every acceptance criterion and quality gate passes"
],
"priority": 16,
"passes": false,
"notes": "Source issue: .scratch/distributed-gguf-runtime/issues/16-produce-the-upstream-llama-cpp-collaboration-package.md",
"dependsOn": [
"DGR-010"
]
}
]
}

2
.scratch/distributed-gguf-runtime/research-prior-art.md
View File

@@ -1,5 +1,7 @@
# Prior Art: Distributed Large-Model Inference
> **Superseded as the current source audit.** Use [`docs/research/distributed-gguf-landscape.md`](../../docs/research/distributed-gguf-landscape.md), [`distributed-gguf-github-followup.md`](../../docs/research/distributed-gguf-github-followup.md), and [`vllm-distributed-gguf-assessment.md`](../../docs/research/vllm-distributed-gguf-assessment.md). This file remains as early historical research.
This note captures what existing projects appear to solve and what remains specific to this platform.
## Petals

2
.scratch/distributed-gguf-runtime/technical-challenges.md
View File

@@ -1,5 +1,7 @@
# Distributed GGUF Technical Challenge Register
> **Historical challenge register.** Route Session, binary activation, local Hot KV State, and transport performance work have advanced since this file was written. Current implementation gates live in [PRD.md](PRD.md), [implementation-strategy.md](implementation-strategy.md), [architecture.md](architecture.md), and [prd.json](prd.json). Preserve this file for detailed risk context; do not treat its “current constraint” section as live system state.
This document focuses on the engineering problems that decide whether the distributed GGUF path is viable. The important distinction is:
- **Model artifacts move like torrents.**