popov/neuron-tai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: define distributed GGUF runtime plan

Browse Source
This commit is contained in:
Dobromir Popov
2026-07-13 15:09:27 +03:00
parent b5fa7245df
commit 4cae4a6c5c
42 changed files with 4913 additions and 691 deletions
Download Patch File Download Diff File Expand all files Collapse all files

59
.scratch/distributed-gguf-runtime/issues/01-lock-the-safetensors-versus-gguf-performance-contract.md Normal file
View File

@@ -0,0 +1,59 @@
# 01 — Lock the safetensors-versus-GGUF performance contract
Status: ready-for-agent
## Mandatory fresh-session context
- Read [RALPH-CONTEXT.md](../RALPH-CONTEXT.md) completely before changing code.
- This issue is `DGR-001` in [prd.json](../prd.json).
- Read the evidence README for every dependency listed below.
- Inspect current code and `git status`; historical text and previous agent claims are not evidence.
## Description
As a runtime engineer, I need a controlled baseline so that GGUF work proceeds from measured speed, memory, and quality rather than reputation.
## Expected durable outputs
- Benchmark harness and deterministic tests
- evidence/DGR-001/performance-contract.json
- Raw and summarized safetensors/GGUF benchmark evidence
## Acceptance criteria
- [ ] Benchmark the same model architecture/revision, machine, prompts, context lengths, output lengths, sampling policy, and concurrency across the current Transformers/safetensors recipe and whole-model llama.cpp recipes.
- [ ] Separate correctness/quality lanes from quantized performance/fit lanes instead of claiming BF16 and Q4 are numerically equivalent.
- [ ] Report TTFT, prefill tok/s, decode tok/s, p50/p95 latency, aggregate throughput, RSS, VRAM, artifact size, failures, and output drift in machine-readable JSON.
- [ ] Add concurrency levels 1 and 4 where memory permits.
- [ ] Write a versioned performance contract consumed by later release gates, including an explicit stop condition when llama.cpp/GGUF has no meaningful speed or fit benefit.
- [ ] Targeted pytest tests pass
- [ ] python -m compileall packages tests passes for Python changes
- [ ] git diff --check passes
- [ ] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
- [ ] Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
- [ ] Real-model execution is opt-in through MESHNET_ENABLE_REAL_INFERENCE_TESTS=1 and records exact artifact/runtime/hardware evidence
- [ ] Model artifacts remain on the configured mounted-drive storage and never under /home
- [ ] Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
- [ ] Read and verify every dependency evidence README before relying on dependency behavior
- [ ] Preserve all pre-existing working-tree changes and stage only files belonging to this story
- [ ] Write .scratch/distributed-gguf-runtime/evidence/DGR-001/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
- [ ] Update only this story issue to Status: done after every acceptance criterion and quality gate passes
## Dependency handoff
- None. This story may start immediately.
## Finish contract
- Create the task evidence directory and durable handoff required above.
- Preserve real failures and blockers; never fabricate benchmark, model, test or hardware output.
- Change this issue to `Status: done` only after all criteria pass.
- Emit `<promise>COMPLETE</promise>` only after the evidence handoff exists.
## References
- [Ralph execution context](../RALPH-CONTEXT.md)
- [PRD](../PRD.md)
- [Implementation strategy](../implementation-strategy.md)
- [Current architecture](../architecture.md)
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)

19
.scratch/distributed-gguf-runtime/issues/01-route-session-lifecycle.md
View File

@@ -1,19 +0,0 @@
# 01 — Route Session lifecycle
Status: ready-for-agent
## What to build
Add the narrowest end-to-end Route Session lifecycle that can be used by distributed inference routes: create a session, bind it to a selected Inference Route, expose status, and close it cleanly. This slice does not need real model cache yet; it proves stable session identity across the control plane and activation plane.
## Acceptance criteria
- [ ] A request can create a Route Session with a stable `session_id`, `route_id`, model preset, backend id, and route membership.
- [ ] Every downstream activation request carries the same session identity and fails clearly if the session or route id does not match.
- [ ] Session status reports phase, route nodes, model preset, backend id, created time, and last activity time.
- [ ] Closing a session releases all registered per-session state.
- [ ] Tests cover create, status, close, stale-session rejection, and wrong-route rejection.
## Blocked by
None - can start immediately.

59
.scratch/distributed-gguf-runtime/issues/02-adopt-the-versioned-grpc-shard-protocol.md Normal file
View File

@@ -0,0 +1,59 @@
# 02 — Adopt the versioned gRPC Shard protocol
Status: ready-for-agent
## Mandatory fresh-session context
- Read [RALPH-CONTEXT.md](../RALPH-CONTEXT.md) completely before changing code.
- This issue is `DGR-002` in [prd.json](../prd.json).
- Read the evidence README for every dependency listed below.
- Inspect current code and `git status`; historical text and previous agent claims are not evidence.
## Description
As a node developer, I need a battle-proven streaming protocol so that Python and C++ Shards communicate without a custom socket protocol.
## Expected durable outputs
- packages/node/native/proto/shard_runtime.proto
- Reproducible Python/C++ schema generation and build wiring
- Protocol round-trip and compatibility tests
- evidence/DGR-002/README.md
## Acceptance criteria
- [ ] Add a Protocol Buffers schema for capability, health, session stream, release, and cancellation operations.
- [ ] Define one long-lived bidirectional gRPC stream per Route Session Activation Seam with deadlines, cancellation, flow control, and structured errors.
- [ ] Define bounded chunking for prefill and a small decode fast path.
- [ ] Carry schema version, request/work ID, Route Session ID, route epoch, artifact/recipe fingerprint, Shard range/effective start, phase, position, idempotency step, cache expectation, compression, and checksum.
- [ ] Define a versioned named-tensor bundle with per-tensor name, shape, dtype, byte order, and payload fragments.
- [ ] Add generated-schema round-trip and compatibility tests in Python and C++.
- [ ] Targeted pytest tests pass
- [ ] python -m compileall packages tests passes for Python changes
- [ ] git diff --check passes
- [ ] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
- [ ] Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
- [ ] Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
- [ ] Read and verify every dependency evidence README before relying on dependency behavior
- [ ] Preserve all pre-existing working-tree changes and stage only files belonging to this story
- [ ] Write .scratch/distributed-gguf-runtime/evidence/DGR-002/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
- [ ] Update only this story issue to Status: done after every acceptance criterion and quality gate passes
## Dependency handoff
- None. This story may start immediately.
## Finish contract
- Create the task evidence directory and durable handoff required above.
- Preserve real failures and blockers; never fabricate benchmark, model, test or hardware output.
- Change this issue to `Status: done` only after all criteria pass.
- Emit `<promise>COMPLETE</promise>` only after the evidence handoff exists.
## References
- [Ralph execution context](../RALPH-CONTEXT.md)
- [PRD](../PRD.md)
- [Implementation strategy](../implementation-strategy.md)
- [Current architecture](../architecture.md)
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)

20
.scratch/distributed-gguf-runtime/issues/02-prefill-decode-binary-http.md
View File

@@ -1,20 +0,0 @@
# 02 — Prefill/decode binary HTTP protocol
Status: ready-for-agent
## What to build
Split the activation protocol into explicit prefill and decode-step calls using the existing binary HTTP direction from ADR-0008. The completed slice should work against a stub backend so payload shape, route/session headers, relay preservation, and failure behavior are testable before real KV cache work begins.
## Acceptance criteria
- [ ] Prefill accepts chunked binary activations with route/session metadata and forwards them through the selected route.
- [ ] Decode-step accepts a one-step binary activation and forwards a one-step activation through the selected route.
- [ ] Decode-step payload size is independent of prompt length in protocol tests.
- [ ] Relay forwarding preserves route/session headers, shape, dtype, position, and wire version.
- [ ] Legacy `/forward` either remains as a compatibility wrapper or fails with a clear wire-version error.
- [ ] Tests cover prefill chunking, decode-step shape validation, relay preservation, and malformed header rejection.
## Blocked by
- 01 — Route Session lifecycle.

57
.scratch/distributed-gguf-runtime/issues/03-define-exact-artifact-and-runtime-recipe-identity.md Normal file
View File

@@ -0,0 +1,57 @@
# 03 — Define exact Artifact and runtime recipe identity
Status: ready-for-agent
## Mandatory fresh-session context
- Read [RALPH-CONTEXT.md](../RALPH-CONTEXT.md) completely before changing code.
- This issue is `DGR-003` in [prd.json](../prd.json).
- Read the evidence README for every dependency listed below.
- Inspect current code and `git status`; historical text and previous agent claims are not evidence.
## Description
As the Tracker, I need exact compatibility identity so that only numerically and operationally compatible Shards form an Inference Route.
## Expected durable outputs
- Exact runtime recipe/fingerprint implementation
- Tracker/node fail-closed admission tests
- evidence/DGR-003/README.md
## Acceptance criteria
- [ ] Separate weight quantization, activation dtype, compute dtype, KV dtype/layout, tokenizer revision, architecture adapter, backend, and runtime version.
- [ ] Bind derivative or split artifacts to an exact source Model Artifact hash and Shard range.
- [ ] Produce a stable compatibility fingerprint used by capability admission and the gRPC handshake.
- [ ] Fail closed on mismatched artifact, tokenizer, architecture, range, boundary schema, activation recipe, or cache layout.
- [ ] Keep unsupported recipes registered-but-dark until a real distributed forward certifies them.
- [ ] Targeted pytest tests pass
- [ ] python -m compileall packages tests passes for Python changes
- [ ] git diff --check passes
- [ ] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
- [ ] Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
- [ ] Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
- [ ] Read and verify every dependency evidence README before relying on dependency behavior
- [ ] Preserve all pre-existing working-tree changes and stage only files belonging to this story
- [ ] Write .scratch/distributed-gguf-runtime/evidence/DGR-003/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
- [ ] Update only this story issue to Status: done after every acceptance criterion and quality gate passes
## Dependency handoff
- `DGR-002` must have `passes: true`; read `../evidence/DGR-002/README.md` and verify its referenced files/commands.
## Finish contract
- Create the task evidence directory and durable handoff required above.
- Preserve real failures and blockers; never fabricate benchmark, model, test or hardware output.
- Change this issue to `Status: done` only after all criteria pass.
- Emit `<promise>COMPLETE</promise>` only after the evidence handoff exists.
## References
- [Ralph execution context](../RALPH-CONTEXT.md)
- [PRD](../PRD.md)
- [Implementation strategy](../implementation-strategy.md)
- [Current architecture](../architecture.md)
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)

21
.scratch/distributed-gguf-runtime/issues/03-generation-telemetry-and-streaming.md
View File

@@ -1,21 +0,0 @@
# 03 — Generation Telemetry and streaming response contract
Status: ready-for-agent
## What to build
Expose realtime Generation Telemetry for active Route Sessions and stream token deltas when the serving path can produce them. This slice should make long distributed requests observable before real large-model work begins.
## Acceptance criteria
- [ ] A client can observe route-session phase changes: queued, loading, prefill, decode, finalizing, completed, failed.
- [ ] Telemetry includes prefill progress, generated token count, rolling tokens/sec, average tokens/sec, active route nodes, and failure reason.
- [ ] Telemetry is available before the first output token.
- [ ] A streaming response can include token deltas while telemetry remains available.
- [ ] A non-streaming fallback still exposes telemetry until final answer or failure.
- [ ] Route-node failure reports the last known phase and reason.
- [ ] Tests cover telemetry updates, streaming token deltas, non-streaming fallback, and structured failure closeout.
## Blocked by
- 01 — Route Session lifecycle.

61
.scratch/distributed-gguf-runtime/issues/04-create-the-reproducible-pinned-llama-cpp-patch-stack.md Normal file
View File

@@ -0,0 +1,61 @@
# 04 — Create the reproducible pinned llama.cpp patch stack
Status: ready-for-agent
## Mandatory fresh-session context
- Read [RALPH-CONTEXT.md](../RALPH-CONTEXT.md) completely before changing code.
- This issue is `DGR-004` in [prd.json](../prd.json).
- Read the evidence README for every dependency listed below.
- Inspect current code and `git status`; historical text and previous agent claims are not evidence.
## Description
As a maintainer, I need a small auditable fork boundary so that upstream updates do not turn the runtime into an unmaintainable stitched codebase.
## Expected durable outputs
- Exact llama.cpp upstream pin
- Numbered minimal patch stack
- Reproducible fetch/apply/build smoke
- evidence/DGR-004/README.md
## Acceptance criteria
- [ ] Pin one exact llama.cpp commit through a reproducible source dependency mechanism.
- [ ] Store a numbered minimal patch stack separately from Meshnet networking code.
- [ ] Add a build script that applies/checks patches and builds the standalone worker without manual source copying.
- [ ] Record upstream file/ABI assumptions and fail clearly when the pin changes.
- [ ] Preserve upstream license and attribution notices.
- [ ] Add a clean rebuild smoke test that does not download a model.
- [ ] Targeted pytest tests pass
- [ ] python -m compileall packages tests passes for Python changes
- [ ] git diff --check passes
- [ ] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
- [ ] Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
- [ ] Pinned native C++ target builds and focused CTest/protocol tests pass where native code is touched
- [ ] llama.cpp patch stack applies cleanly to the exact pinned commit where patch code is touched
- [ ] Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
- [ ] Read and verify every dependency evidence README before relying on dependency behavior
- [ ] Preserve all pre-existing working-tree changes and stage only files belonging to this story
- [ ] Write .scratch/distributed-gguf-runtime/evidence/DGR-004/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
- [ ] Update only this story issue to Status: done after every acceptance criterion and quality gate passes
## Dependency handoff
- `DGR-001` must have `passes: true`; read `../evidence/DGR-001/README.md` and verify its referenced files/commands.
## Finish contract
- Create the task evidence directory and durable handoff required above.
- Preserve real failures and blockers; never fabricate benchmark, model, test or hardware output.
- Change this issue to `Status: done` only after all criteria pass.
- Emit `<promise>COMPLETE</promise>` only after the evidence handoff exists.
## References
- [Ralph execution context](../RALPH-CONTEXT.md)
- [PRD](../PRD.md)
- [Implementation strategy](../implementation-strategy.md)
- [Current architecture](../architecture.md)
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)

23
.scratch/distributed-gguf-runtime/issues/04-pytorch-distributed-kv-reference.md
View File

@@ -1,23 +0,0 @@
# 04 — PyTorch distributed KV reference route
Status: ready-for-agent
## What to build
Fix the existing distributed PyTorch route so it uses the Route Session and prefill/decode protocol to keep Hot KV State local to each Shard node. The visible behavior is that prefill processes the prompt once, and decode no longer recomputes or resends the full growing prompt for every token.
## Acceptance criteria
- [ ] Distributed PyTorch prefill stores per-session cache/state on each Shard node.
- [ ] Distributed PyTorch decode-step reads and appends local per-shard cache/state.
- [ ] Decode activation seam payload is one token/hidden-state step after prefill.
- [ ] The old full-growing-prompt decode loop is not used for models that support the reference cache path.
- [ ] Unsupported model/cache APIs fail with an explicit backend capability error.
- [ ] Session close or TTL cleanup releases per-shard cache.
- [ ] Regression tests prove decode does not call the full prompt encoder for every generated token.
## Blocked by
- 01 — Route Session lifecycle.
- 02 — Prefill/decode binary HTTP protocol.
- 03 — Generation Telemetry and streaming response contract.

61
.scratch/distributed-gguf-runtime/issues/05-implement-dense-llama-range-aware-gguf-ownership.md Normal file
View File

@@ -0,0 +1,61 @@
# 05 — Implement dense-Llama range-aware GGUF ownership
Status: ready-for-agent
## Mandatory fresh-session context
- Read [RALPH-CONTEXT.md](../RALPH-CONTEXT.md) completely before changing code.
- This issue is `DGR-005` in [prd.json](../prd.json).
- Read the evidence README for every dependency listed below.
- Inspect current code and `git status`; historical text and previous agent claims are not evidence.
## Description
As a node, I need to map only my assigned dense-Llama Shard so that aggregate consumer memory can hold a model larger than one node.
## Expected durable outputs
- Dense-Llama range-aware ownership implementation
- Authoritative loaded-range introspection
- Mapped/resident memory evidence
- evidence/DGR-005/README.md
## Acceptance criteria
- [ ] Register and allocate only `blk.N.*` tensors in the assigned range.
- [ ] Load embeddings only for the head and final norm/LM head only for the tail, including tied embeddings.
- [ ] Prefer range-aware mapping from one exact source GGUF; if derivative sub-GGUFs are used temporarily, verify source/slice hashes and avoid claiming final artifact semantics.
- [ ] Report authoritative loaded range and endpoint ownership from the model, not operator CLI claims.
- [ ] Demonstrate mapped/resident memory scales with owned tensors rather than full model size.
- [ ] Targeted pytest tests pass
- [ ] python -m compileall packages tests passes for Python changes
- [ ] git diff --check passes
- [ ] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
- [ ] Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
- [ ] Pinned native C++ target builds and focused CTest/protocol tests pass where native code is touched
- [ ] llama.cpp patch stack applies cleanly to the exact pinned commit where patch code is touched
- [ ] Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
- [ ] Read and verify every dependency evidence README before relying on dependency behavior
- [ ] Preserve all pre-existing working-tree changes and stage only files belonging to this story
- [ ] Write .scratch/distributed-gguf-runtime/evidence/DGR-005/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
- [ ] Update only this story issue to Status: done after every acceptance criterion and quality gate passes
## Dependency handoff
- `DGR-003` must have `passes: true`; read `../evidence/DGR-003/README.md` and verify its referenced files/commands.
- `DGR-004` must have `passes: true`; read `../evidence/DGR-004/README.md` and verify its referenced files/commands.
## Finish contract
- Create the task evidence directory and durable handoff required above.
- Preserve real failures and blockers; never fabricate benchmark, model, test or hardware output.
- Change this issue to `Status: done` only after all criteria pass.
- Emit `<promise>COMPLETE</promise>` only after the evidence handoff exists.
## References
- [Ralph execution context](../RALPH-CONTEXT.md)
- [PRD](../PRD.md)
- [Implementation strategy](../implementation-strategy.md)
- [Current architecture](../architecture.md)
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)

20
.scratch/distributed-gguf-runtime/issues/05-local-llamacpp-gguf-backend.md
View File

@@ -1,20 +0,0 @@
# 05 — Local llama.cpp/GGUF backend
Status: ready-for-agent
## What to build
Add a local full-model GGUF backend so a node that can hold a GGUF model can serve it through the existing node API. This is the immediate CPU-performance path and the baseline for later distributed llama.cpp work.
## Acceptance criteria
- [ ] A node can start with backend `llama.cpp` or `gguf` for a local full-model GGUF artifact.
- [ ] The node can answer an OpenAI-compatible chat completion through the existing API.
- [ ] Startup and registration clearly report backend, quantization/artifact metadata, context cap, and local model path.
- [ ] The PyTorch backend remains unchanged and selectable.
- [ ] A smoke test or script validates backend wiring with a small GGUF model or a stubbed llama.cpp process.
- [ ] A benchmark command can compare local PyTorch CPU and local GGUF CPU for the same small supported model when both are available.
## Blocked by
None - can start immediately.

61
.scratch/distributed-gguf-runtime/issues/06-implement-architecture-defined-boundary-input-output.md Normal file
View File

@@ -0,0 +1,61 @@
# 06 — Implement architecture-defined boundary input/output
Status: ready-for-agent
## Mandatory fresh-session context
- Read [RALPH-CONTEXT.md](../RALPH-CONTEXT.md) completely before changing code.
- This issue is `DGR-006` in [prd.json](../prd.json).
- Read the evidence README for every dependency listed below.
- Inspect current code and `git status`; historical text and previous agent claims are not evidence.
## Description
As a Shard, I need to consume and emit the correct transformer boundary state so that disjoint processes reproduce whole-model execution.
## Expected durable outputs
- Architecture boundary adapter
- Whole-model/two-range parity tests and results
- evidence/DGR-006/README.md
## Acceptance criteria
- [ ] Head accepts token IDs and owns token embedding.
- [ ] Middle/tail bypass token embedding and accept the named boundary bundle.
- [ ] Non-tail emits the unnormalized architecture-defined residual/boundary before final norm/head and before tail-only row pruning.
- [ ] Tail emits logits or token output through an explicit sampling contract.
- [ ] Dense-Llama whole-model versus two-range prefill and greedy-decode parity passes the documented tolerance.
- [ ] The adapter interface fails closed for uncertified architectures.
- [ ] Targeted pytest tests pass
- [ ] python -m compileall packages tests passes for Python changes
- [ ] git diff --check passes
- [ ] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
- [ ] Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
- [ ] Pinned native C++ target builds and focused CTest/protocol tests pass where native code is touched
- [ ] llama.cpp patch stack applies cleanly to the exact pinned commit where patch code is touched
- [ ] Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
- [ ] Read and verify every dependency evidence README before relying on dependency behavior
- [ ] Preserve all pre-existing working-tree changes and stage only files belonging to this story
- [ ] Write .scratch/distributed-gguf-runtime/evidence/DGR-006/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
- [ ] Update only this story issue to Status: done after every acceptance criterion and quality gate passes
## Dependency handoff
- `DGR-002` must have `passes: true`; read `../evidence/DGR-002/README.md` and verify its referenced files/commands.
- `DGR-005` must have `passes: true`; read `../evidence/DGR-005/README.md` and verify its referenced files/commands.
## Finish contract
- Create the task evidence directory and durable handoff required above.
- Preserve real failures and blockers; never fabricate benchmark, model, test or hardware output.
- Change this issue to `Status: done` only after all criteria pass.
- Emit `<promise>COMPLETE</promise>` only after the evidence handoff exists.
## References
- [Ralph execution context](../RALPH-CONTEXT.md)
- [PRD](../PRD.md)
- [Implementation strategy](../implementation-strategy.md)
- [Current architecture](../architecture.md)
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)

20
.scratch/distributed-gguf-runtime/issues/06-model-artifact-manifest.md
View File

@@ -1,20 +0,0 @@
# 06 — Model Artifact manifest and Shard advertisement
Status: ready-for-agent
## What to build
Introduce a Model Artifact manifest that separates storage distribution from route execution. A node should be able to verify local model files, determine which Shards it can serve, and advertise artifact/layer availability to the Tracker without contacting Hugging Face at request time.
## Acceptance criteria
- [ ] Manifest records model preset, upstream revision, license, backend support, quantization, context cap, tokenizer artifacts, file hashes, piece hashes, and tensor/layer mapping where available.
- [ ] A node can verify local artifacts against the manifest and reject corrupt or incomplete artifacts.
- [ ] A node can derive advertised Shard ranges from the manifest and local files.
- [ ] Tracker registration can include artifact id, backend id, Shard range, and verification status.
- [ ] Tracker coverage can distinguish model-layer coverage from artifact availability.
- [ ] Tests cover valid manifest registration, corrupt artifact rejection, and missing layer/tensor metadata.
## Blocked by
- 01 — Route Session lifecycle.

60
.scratch/distributed-gguf-runtime/issues/07-add-isolated-concurrent-local-hot-kv-state.md Normal file
View File

@@ -0,0 +1,60 @@
# 07 — Add isolated concurrent local Hot KV State
Status: ready-for-agent
## Mandatory fresh-session context
- Read [RALPH-CONTEXT.md](../RALPH-CONTEXT.md) completely before changing code.
- This issue is `DGR-007` in [prd.json](../prd.json).
- Read the evidence README for every dependency listed below.
- Inspect current code and `git status`; historical text and previous agent claims are not evidence.
## Description
As a client, I need concurrent Route Sessions to retain independent per-Shard cache so that one request cannot clear or corrupt another.
## Expected durable outputs
- Concurrent local KV/session manager
- Isolation, eviction, cancellation and cleanup tests
- evidence/DGR-007/README.md
## Acceptance criteria
- [ ] Map `(Route Session ID, route epoch)` to an isolated llama sequence or bounded context.
- [ ] Allocate KV only for owned layers.
- [ ] Support prefill append, decode append, truncate, release, TTL/LRU eviction, and explicit cache-miss response.
- [ ] Reject stale epochs and incompatible cache recipes.
- [ ] At least four concurrent sessions on a small model complete without token or KV cross-talk.
- [ ] Cancellation/release of one session leaves other sessions intact and memory returns to the configured budget.
- [ ] Targeted pytest tests pass
- [ ] python -m compileall packages tests passes for Python changes
- [ ] git diff --check passes
- [ ] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
- [ ] Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
- [ ] Pinned native C++ target builds and focused CTest/protocol tests pass where native code is touched
- [ ] llama.cpp patch stack applies cleanly to the exact pinned commit where patch code is touched
- [ ] Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
- [ ] Read and verify every dependency evidence README before relying on dependency behavior
- [ ] Preserve all pre-existing working-tree changes and stage only files belonging to this story
- [ ] Write .scratch/distributed-gguf-runtime/evidence/DGR-007/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
- [ ] Update only this story issue to Status: done after every acceptance criterion and quality gate passes
## Dependency handoff
- `DGR-006` must have `passes: true`; read `../evidence/DGR-006/README.md` and verify its referenced files/commands.
## Finish contract
- Create the task evidence directory and durable handoff required above.
- Preserve real failures and blockers; never fabricate benchmark, model, test or hardware output.
- Change this issue to `Status: done` only after all criteria pass.
- Emit `<promise>COMPLETE</promise>` only after the evidence handoff exists.
## References
- [Ralph execution context](../RALPH-CONTEXT.md)
- [PRD](../PRD.md)
- [Implementation strategy](../implementation-strategy.md)
- [Current architecture](../architecture.md)
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)

25
.scratch/distributed-gguf-runtime/issues/07-llamacpp-layer-boundary-prototype.md
View File

@@ -1,25 +0,0 @@
# 07 — llama.cpp layer-boundary prototype
Status: ready-for-human
## What to build
Build a local prototype that proves whether llama.cpp/libllama can support the platform's distributed execution contract: execute a selected layer range, accept inbound hidden states, emit outbound hidden states, and own per-session cache for only the loaded/served range.
This is the collaboration package for upstream llama.cpp. The target is an upstreamable API shape, not a permanent fork.
## Acceptance criteria
- [ ] A small llama.cpp-supported GGUF model can be split into a two-process localhost head/tail prototype.
- [ ] The head process runs embeddings and early layers, then emits hidden states at an Activation Seam.
- [ ] The tail process accepts hidden states, runs later layers plus output head, and produces logits/tokens comparable to single-process execution.
- [ ] Prefill is performed once and decode-step seam payload is one hidden-state step per generated token.
- [ ] Each process owns only its own per-session cache/state.
- [ ] The prototype records the minimum upstream API needed for layer-range execution, hidden-state I/O, partial loading/introspection, and per-session KV ownership.
- [ ] If upstream support is unavailable, the issue ends with a concrete recommendation: upstream proposal, narrow adapter fork, or keep GGUF distribution local-only for now.
## Blocked by
- 02 — Prefill/decode binary HTTP protocol.
- 05 — Local llama.cpp/GGUF backend.
- 06 — Model Artifact manifest and Shard advertisement.

65
.scratch/distributed-gguf-runtime/issues/08-build-the-standalone-c-grpc-shard-worker.md Normal file
View File

@@ -0,0 +1,65 @@
# 08 — Build the standalone C++ gRPC Shard worker
Status: ready-for-agent
## Mandatory fresh-session context
- Read [RALPH-CONTEXT.md](../RALPH-CONTEXT.md) completely before changing code.
- This issue is `DGR-008` in [prd.json](../prd.json).
- Read the evidence README for every dependency listed below.
- Inspect current code and `git status`; historical text and previous agent claims are not evidence.
## Description
As a node runtime, I need one supervised native process so that llama.cpp internals remain behind a stable project-owned protocol.
## Expected durable outputs
- Standalone C++ gRPC worker
- Fake-model Python/C++ integration tests
- Lifecycle and bounded-failure evidence
- evidence/DGR-008/README.md
## Acceptance criteria
- [ ] Worker exposes capability, health, session stream, release, cancellation, and metrics services from DGR-002.
- [ ] Worker loads one exact Artifact/recipe/Shard identity and refuses mismatched requests.
- [ ] Streaming path enforces bounded messages, flow control, deadlines, idempotency, and independent session cancellation.
- [ ] Worker does not expose raw llama.cpp RPC or arbitrary GGML graph execution.
- [ ] Graceful shutdown releases sessions; crash behavior is bounded and observable.
- [ ] Python integration tests run against a fake model mode without model downloads.
- [ ] Targeted pytest tests pass
- [ ] python -m compileall packages tests passes for Python changes
- [ ] git diff --check passes
- [ ] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
- [ ] Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
- [ ] Pinned native C++ target builds and focused CTest/protocol tests pass where native code is touched
- [ ] llama.cpp patch stack applies cleanly to the exact pinned commit where patch code is touched
- [ ] Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
- [ ] Read and verify every dependency evidence README before relying on dependency behavior
- [ ] Preserve all pre-existing working-tree changes and stage only files belonging to this story
- [ ] Write .scratch/distributed-gguf-runtime/evidence/DGR-008/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
- [ ] Update only this story issue to Status: done after every acceptance criterion and quality gate passes
## Dependency handoff
- `DGR-002` must have `passes: true`; read `../evidence/DGR-002/README.md` and verify its referenced files/commands.
- `DGR-003` must have `passes: true`; read `../evidence/DGR-003/README.md` and verify its referenced files/commands.
- `DGR-004` must have `passes: true`; read `../evidence/DGR-004/README.md` and verify its referenced files/commands.
- `DGR-006` must have `passes: true`; read `../evidence/DGR-006/README.md` and verify its referenced files/commands.
- `DGR-007` must have `passes: true`; read `../evidence/DGR-007/README.md` and verify its referenced files/commands.
## Finish contract
- Create the task evidence directory and durable handoff required above.
- Preserve real failures and blockers; never fabricate benchmark, model, test or hardware output.
- Change this issue to `Status: done` only after all criteria pass.
- Emit `<promise>COMPLETE</promise>` only after the evidence handoff exists.
## References
- [Ralph execution context](../RALPH-CONTEXT.md)
- [PRD](../PRD.md)
- [Implementation strategy](../implementation-strategy.md)
- [Current architecture](../architecture.md)
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)

24
.scratch/distributed-gguf-runtime/issues/08-networked-distributed-gguf-route.md
View File

@@ -1,24 +0,0 @@
# 08 — Networked distributed GGUF route
Status: pending
## What to build
Run a GGUF-backed model over a real multi-node Inference Route using the resolved Route Session, binary HTTP prefill/decode protocol, local Hot KV State, Generation Telemetry, and alpha fail-fast behavior.
## Acceptance criteria
- [ ] Two machines can form one GGUF-backed Inference Route over contiguous Shards.
- [ ] Prefill builds local per-shard cache/state and decode-step uses one-step seam payloads.
- [ ] The client receives streamed token deltas when supported by the GGUF path.
- [ ] The client receives Generation Telemetry for phase, generated tokens, tokens/sec, route health, and failure reason.
- [ ] Route-node loss fails the Route Session cleanly; no automatic repair is attempted in alpha.
- [ ] Tracker metrics show prefill tokens/sec, decode tokens/sec, seam latency, queue depth, and cache memory by node.
- [ ] Billing/audit records identify route membership and layer/token work for the completed or failed session.
## Blocked by
- 03 — Generation Telemetry and streaming response contract.
- 04 — PyTorch distributed KV reference route.
- 06 — Model Artifact manifest and Shard advertisement.
- 07 — llama.cpp layer-boundary prototype.

21
.scratch/distributed-gguf-runtime/issues/09-deepseek-v4-flash-support-audit.md
View File

@@ -1,21 +0,0 @@
# 09 — DeepSeek-V4-Flash support audit
Status: ready-for-agent
## What to build
Audit `deepseek-ai/DeepSeek-V4-Flash` as the first serious large-model target after the small GGUF protocol smoke test. The output is a compatibility matrix and a recommended runtime path, not full production support.
## Acceptance criteria
- [ ] Verify current PyTorch/Transformers load and generation semantics for DeepSeek-V4-Flash from primary model documentation.
- [ ] Verify vLLM and SGLang support status from primary runtime documentation or release notes.
- [ ] Verify whether a GGUF/llama.cpp quantization path exists or would need upstream work.
- [ ] Estimate artifact size, active parameter behavior, and 128K cache memory by Shard range.
- [ ] Identify required backend capability flags for the Tracker.
- [ ] Produce a compatibility matrix: PyTorch, vLLM, SGLang, llama.cpp/GGUF.
- [ ] End with one recommendation: first runtime path, blocked pending upstream, or defer.
## Blocked by
None - can start immediately.

61
.scratch/distributed-gguf-runtime/issues/09-integrate-the-native-worker-with-meshnet.md Normal file
View File

@@ -0,0 +1,61 @@
# 09 — Integrate the native worker with Meshnet
Status: ready-for-agent
## Mandatory fresh-session context
- Read [RALPH-CONTEXT.md](../RALPH-CONTEXT.md) completely before changing code.
- This issue is `DGR-009` in [prd.json](../prd.json).
- Read the evidence README for every dependency listed below.
- Inspect current code and `git status`; historical text and previous agent claims are not evidence.
## Description
As the existing node service, I need a GGUF Shard backend adapter so that the Tracker, relay, billing, telemetry, and capability admission remain the sole control plane.
## Expected durable outputs
- Meshnet GGUF backend adapter
- Registration, routing, relay, telemetry and billing tests
- evidence/DGR-009/README.md
## Acceptance criteria
- [ ] Implement the existing model-backend surface without changing Transformers behavior.
- [ ] Registration carries exact validated GGUF recipe, Shard, backend and concurrency/KV capacity.
- [ ] Tracker forms only complete compatible routes and keeps uncertified recipes dark.
- [ ] Direct routes use gRPC streams; relayed routes carry the same versioned protobuf frames as opaque binary through the existing relay seam.
- [ ] Existing request/work IDs, cancellation, Generation Telemetry, billing, and per-node attribution remain correlated.
- [ ] No vLLM, Nakshatra, prima.cpp, or custom-engine control plane becomes a core dependency.
- [ ] Targeted pytest tests pass
- [ ] python -m compileall packages tests passes for Python changes
- [ ] git diff --check passes
- [ ] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
- [ ] Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
- [ ] Pinned native C++ target builds and focused CTest/protocol tests pass where native code is touched
- [ ] llama.cpp patch stack applies cleanly to the exact pinned commit where patch code is touched
- [ ] Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
- [ ] Read and verify every dependency evidence README before relying on dependency behavior
- [ ] Preserve all pre-existing working-tree changes and stage only files belonging to this story
- [ ] Write .scratch/distributed-gguf-runtime/evidence/DGR-009/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
- [ ] Update only this story issue to Status: done after every acceptance criterion and quality gate passes
## Dependency handoff
- `DGR-003` must have `passes: true`; read `../evidence/DGR-003/README.md` and verify its referenced files/commands.
- `DGR-008` must have `passes: true`; read `../evidence/DGR-008/README.md` and verify its referenced files/commands.
## Finish contract
- Create the task evidence directory and durable handoff required above.
- Preserve real failures and blockers; never fabricate benchmark, model, test or hardware output.
- Change this issue to `Status: done` only after all criteria pass.
- Emit `<promise>COMPLETE</promise>` only after the evidence handoff exists.
## References
- [Ralph execution context](../RALPH-CONTEXT.md)
- [PRD](../PRD.md)
- [Implementation strategy](../implementation-strategy.md)
- [Current architecture](../architecture.md)
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)

20
.scratch/distributed-gguf-runtime/issues/10-glm52-ornith-followup-audit.md
View File

@@ -1,20 +0,0 @@
# 10 — GLM-5.2 and Ornith follow-up support audit
Status: pending
## What to build
Audit GLM-5.2 and Ornith after the smaller protocol smoke path and DeepSeek-V4-Flash audit. The output is a follow-up compatibility matrix focused on architecture/runtime blockers: DSA/MLA, hybrid attention, cache accounting, and GGUF/llama.cpp support.
## Acceptance criteria
- [ ] Verify GLM-5.2 PyTorch/Transformers serving requirements and cache semantics from primary model documentation.
- [ ] Verify llama.cpp/GGUF support status for `glm_moe_dsa` or equivalent architecture support.
- [ ] Verify Ornith/Qwen3.5-MoE and hybrid attention support status in the candidate runtimes.
- [ ] Estimate artifact size and 128K cache memory by Shard range for each model.
- [ ] Identify smallest quality-preserving quantization worth testing.
- [ ] Convert each runtime blocker into a follow-up issue or upstream collaboration note.
## Blocked by
- 09 — DeepSeek-V4-Flash support audit.

62
.scratch/distributed-gguf-runtime/issues/10-pass-local-real-model-two-process-acceptance.md Normal file
View File

@@ -0,0 +1,62 @@
# 10 — Pass local real-model two-process acceptance
Status: ready-for-agent
## Mandatory fresh-session context
- Read [RALPH-CONTEXT.md](../RALPH-CONTEXT.md) completely before changing code.
- This issue is `DGR-010` in [prd.json](../prd.json).
- Read the evidence README for every dependency listed below.
- Inspect current code and `git status`; historical text and previous agent claims are not evidence.
## Description
As a release engineer, I need real local distributed parity before involving network variability.
## Expected durable outputs
- Real local two-process commands and configuration
- Raw parity, memory and performance results
- evidence/DGR-010/README.md
## Acceptance criteria
- [ ] Two local worker processes open disjoint dense-Llama ranges from the certified Artifact.
- [ ] Prefill and at least 32 greedy decode tokens match whole-model llama.cpp within the certified tolerance.
- [ ] Each worker retains only its own tensors and Hot KV State.
- [ ] Four concurrent Route Sessions pass isolation and cleanup checks.
- [ ] Report TTFT, prefill/decode throughput, seam bytes/latency, worker RSS/VRAM, KV memory, batch size, and queue time.
- [ ] Killing one worker produces a bounded structured failure rather than a deadlock.
- [ ] Targeted pytest tests pass
- [ ] python -m compileall packages tests passes for Python changes
- [ ] git diff --check passes
- [ ] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
- [ ] Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
- [ ] Pinned native C++ target builds and focused CTest/protocol tests pass where native code is touched
- [ ] llama.cpp patch stack applies cleanly to the exact pinned commit where patch code is touched
- [ ] Real-model execution is opt-in through MESHNET_ENABLE_REAL_INFERENCE_TESTS=1 and records exact artifact/runtime/hardware evidence
- [ ] Model artifacts remain on the configured mounted-drive storage and never under /home
- [ ] Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
- [ ] Read and verify every dependency evidence README before relying on dependency behavior
- [ ] Preserve all pre-existing working-tree changes and stage only files belonging to this story
- [ ] Write .scratch/distributed-gguf-runtime/evidence/DGR-010/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
- [ ] Update only this story issue to Status: done after every acceptance criterion and quality gate passes
## Dependency handoff
- `DGR-009` must have `passes: true`; read `../evidence/DGR-009/README.md` and verify its referenced files/commands.
## Finish contract
- Create the task evidence directory and durable handoff required above.
- Preserve real failures and blockers; never fabricate benchmark, model, test or hardware output.
- Change this issue to `Status: done` only after all criteria pass.
- Emit `<promise>COMPLETE</promise>` only after the evidence handoff exists.
## References
- [Ralph execution context](../RALPH-CONTEXT.md)
- [PRD](../PRD.md)
- [Implementation strategy](../implementation-strategy.md)
- [Current architecture](../architecture.md)
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)

62
.scratch/distributed-gguf-runtime/issues/11-pass-a-real-heterogeneous-two-machine-route.md Normal file
View File

@@ -0,0 +1,62 @@
# 11 — Pass a real heterogeneous two-machine route
Status: ready-for-agent
## Mandatory fresh-session context
- Read [RALPH-CONTEXT.md](../RALPH-CONTEXT.md) completely before changing code.
- This issue is `DGR-011` in [prd.json](../prd.json).
- Read the evidence README for every dependency listed below.
- Inspect current code and `git status`; historical text and previous agent claims are not evidence.
## Description
As a consumer-hardware operator, I need two physical machines to execute one GGUF model so that the distributed claim is real.
## Expected durable outputs
- Two-machine hardware/network/runtime manifest
- Raw real-route metrics and output evidence
- evidence/DGR-011/README.md
## Acceptance criteria
- [ ] Tracker selects two physical nodes with disjoint Shards and one exact certified recipe/compatibility class.
- [ ] Actual CPU/GPU execution occurs on both nodes; synthetic workers do not satisfy acceptance.
- [ ] Prefill/decode, concurrent-session isolation, telemetry, cancellation, and cleanup pass over the real transport/relay path.
- [ ] Exact hardware, network, backend, model hash, route, commands, and raw metrics are recorded.
- [ ] A model or recipe larger than one participating node's admitted memory is exercised when available.
- [ ] Output drift is measured and incompatible mixed backends fail closed.
- [ ] Targeted pytest tests pass
- [ ] python -m compileall packages tests passes for Python changes
- [ ] git diff --check passes
- [ ] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
- [ ] Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
- [ ] Pinned native C++ target builds and focused CTest/protocol tests pass where native code is touched
- [ ] llama.cpp patch stack applies cleanly to the exact pinned commit where patch code is touched
- [ ] Real-model execution is opt-in through MESHNET_ENABLE_REAL_INFERENCE_TESTS=1 and records exact artifact/runtime/hardware evidence
- [ ] Model artifacts remain on the configured mounted-drive storage and never under /home
- [ ] Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
- [ ] Read and verify every dependency evidence README before relying on dependency behavior
- [ ] Preserve all pre-existing working-tree changes and stage only files belonging to this story
- [ ] Write .scratch/distributed-gguf-runtime/evidence/DGR-011/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
- [ ] Update only this story issue to Status: done after every acceptance criterion and quality gate passes
## Dependency handoff
- `DGR-010` must have `passes: true`; read `../evidence/DGR-010/README.md` and verify its referenced files/commands.
## Finish contract
- Create the task evidence directory and durable handoff required above.
- Preserve real failures and blockers; never fabricate benchmark, model, test or hardware output.
- Change this issue to `Status: done` only after all criteria pass.
- Emit `<promise>COMPLETE</promise>` only after the evidence handoff exists.
## References
- [Ralph execution context](../RALPH-CONTEXT.md)
- [PRD](../PRD.md)
- [Implementation strategy](../implementation-strategy.md)
- [Current architecture](../architecture.md)
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)

63
.scratch/distributed-gguf-runtime/issues/12-implement-continuous-batching-and-bounded-admission.md Normal file
View File

@@ -0,0 +1,63 @@
# 12 — Implement continuous batching and bounded admission
Status: ready-for-agent
## Mandatory fresh-session context
- Read [RALPH-CONTEXT.md](../RALPH-CONTEXT.md) completely before changing code.
- This issue is `DGR-012` in [prd.json](../prd.json).
- Read the evidence README for every dependency listed below.
- Inspect current code and `git status`; historical text and previous agent claims are not evidence.
## Description
As a node operator, I need active sessions batched safely so that concurrency increases aggregate throughput rather than serializing every request.
## Expected durable outputs
- Continuous batching/admission scheduler
- Concurrency 1/2/4/8 report
- Queue, batch and KV-pressure evidence
- evidence/DGR-012/README.md
## Acceptance criteria
- [ ] Node scheduler admits sessions against weight, KV, scratch, and queue budgets.
- [ ] Compatible decode steps from multiple sessions form llama.cpp batches while preserving per-session positions and outputs.
- [ ] Prefill does not starve decode; scheduling policy and bounds are explicit.
- [ ] Backpressure prevents unbounded queued activations or KV growth.
- [ ] Capability telemetry reports active sessions, queue depth, batch occupancy, KV pressure, prefill/decode rates, and rejected admissions.
- [ ] Concurrency 1/2/4/8 benchmark identifies saturation and shows no cross-session corruption.
- [ ] Targeted pytest tests pass
- [ ] python -m compileall packages tests passes for Python changes
- [ ] git diff --check passes
- [ ] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
- [ ] Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
- [ ] Pinned native C++ target builds and focused CTest/protocol tests pass where native code is touched
- [ ] llama.cpp patch stack applies cleanly to the exact pinned commit where patch code is touched
- [ ] Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
- [ ] Read and verify every dependency evidence README before relying on dependency behavior
- [ ] Preserve all pre-existing working-tree changes and stage only files belonging to this story
- [ ] Write .scratch/distributed-gguf-runtime/evidence/DGR-012/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
- [ ] Update only this story issue to Status: done after every acceptance criterion and quality gate passes
## Dependency handoff
- `DGR-007` must have `passes: true`; read `../evidence/DGR-007/README.md` and verify its referenced files/commands.
- `DGR-009` must have `passes: true`; read `../evidence/DGR-009/README.md` and verify its referenced files/commands.
- `DGR-010` must have `passes: true`; read `../evidence/DGR-010/README.md` and verify its referenced files/commands.
## Finish contract
- Create the task evidence directory and durable handoff required above.
- Preserve real failures and blockers; never fabricate benchmark, model, test or hardware output.
- Change this issue to `Status: done` only after all criteria pass.
- Emit `<promise>COMPLETE</promise>` only after the evidence handoff exists.
## References
- [Ralph execution context](../RALPH-CONTEXT.md)
- [PRD](../PRD.md)
- [Implementation strategy](../implementation-strategy.md)
- [Current architecture](../architecture.md)
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)

62
.scratch/distributed-gguf-runtime/issues/13-harden-failure-cancellation-and-restart-semantics.md Normal file
View File

@@ -0,0 +1,62 @@
# 13 — Harden failure, cancellation, and restart semantics
Status: ready-for-agent
## Mandatory fresh-session context
- Read [RALPH-CONTEXT.md](../RALPH-CONTEXT.md) completely before changing code.
- This issue is `DGR-013` in [prd.json](../prd.json).
- Read the evidence README for every dependency listed below.
- Inspect current code and `git status`; historical text and previous agent claims are not evidence.
## Description
As a client, I need failures to be bounded and explicit so that distributed speed does not come with hanging or corrupted generations.
## Expected durable outputs
- Failure/cancel/restart test matrix
- Resource cleanup and billing-state evidence
- evidence/DGR-013/README.md
## Acceptance criteria
- [ ] Deadlines and heartbeat/health loss terminate blocked stream operations.
- [ ] Cancellation propagates across every Shard and releases local KV and queued buffers.
- [ ] Duplicate steps are idempotent; uncertain mutations are never replayed silently.
- [ ] Alpha failover restarts from token zero on a newly compatible route rather than importing unverified KV.
- [ ] Worker death, stream reset, malformed bundle, stale epoch, and cache miss tests pass.
- [ ] Billing/work records distinguish completed, cancelled, failed, and unverified work.
- [ ] Targeted pytest tests pass
- [ ] python -m compileall packages tests passes for Python changes
- [ ] git diff --check passes
- [ ] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
- [ ] Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
- [ ] Pinned native C++ target builds and focused CTest/protocol tests pass where native code is touched
- [ ] llama.cpp patch stack applies cleanly to the exact pinned commit where patch code is touched
- [ ] Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
- [ ] Read and verify every dependency evidence README before relying on dependency behavior
- [ ] Preserve all pre-existing working-tree changes and stage only files belonging to this story
- [ ] Write .scratch/distributed-gguf-runtime/evidence/DGR-013/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
- [ ] Update only this story issue to Status: done after every acceptance criterion and quality gate passes
## Dependency handoff
- `DGR-008` must have `passes: true`; read `../evidence/DGR-008/README.md` and verify its referenced files/commands.
- `DGR-009` must have `passes: true`; read `../evidence/DGR-009/README.md` and verify its referenced files/commands.
- `DGR-012` must have `passes: true`; read `../evidence/DGR-012/README.md` and verify its referenced files/commands.
## Finish contract
- Create the task evidence directory and durable handoff required above.
- Preserve real failures and blockers; never fabricate benchmark, model, test or hardware output.
- Change this issue to `Status: done` only after all criteria pass.
- Emit `<promise>COMPLETE</promise>` only after the evidence handoff exists.
## References
- [Ralph execution context](../RALPH-CONTEXT.md)
- [PRD](../PRD.md)
- [Implementation strategy](../implementation-strategy.md)
- [Current architecture](../architecture.md)
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)

65
.scratch/distributed-gguf-runtime/issues/14-enforce-the-gguf-versus-safetensors-release-gate.md Normal file
View File

@@ -0,0 +1,65 @@
# 14 — Enforce the GGUF-versus-safetensors release gate
Status: ready-for-agent
## Mandatory fresh-session context
- Read [RALPH-CONTEXT.md](../RALPH-CONTEXT.md) completely before changing code.
- This issue is `DGR-014` in [prd.json](../prd.json).
- Read the evidence README for every dependency listed below.
- Inspect current code and `git status`; historical text and previous agent claims are not evidence.
## Description
As the product owner, I need an end-to-end comparison so that the native runtime ships only if it advances model access or performance.
## Expected durable outputs
- Immutable comparison against DGR-001 thresholds
- Machine-readable final report
- Ship/optimize/stop recommendation
- evidence/DGR-014/README.md
## Acceptance criteria
- [ ] Run current distributed safetensors and distributed GGUF routes on the same certified model/hardware/network scenario where technically comparable.
- [ ] Report quality, TTFT, prefill/decode throughput, aggregate concurrency throughput, p95 latency, seam cost, memory, KV pressure, failures, and cleanup.
- [ ] Evaluate against the DGR-001 performance contract without changing thresholds after seeing results.
- [ ] Ship recommendation is one of: promote GGUF, optimize a measured bottleneck with a new bounded task, or stop the native track.
- [ ] Results clearly separate quantization gains from transport/runtime gains.
- [ ] Targeted pytest tests pass
- [ ] python -m compileall packages tests passes for Python changes
- [ ] git diff --check passes
- [ ] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
- [ ] Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
- [ ] Pinned native C++ target builds and focused CTest/protocol tests pass where native code is touched
- [ ] llama.cpp patch stack applies cleanly to the exact pinned commit where patch code is touched
- [ ] Real-model execution is opt-in through MESHNET_ENABLE_REAL_INFERENCE_TESTS=1 and records exact artifact/runtime/hardware evidence
- [ ] Model artifacts remain on the configured mounted-drive storage and never under /home
- [ ] Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
- [ ] Read and verify every dependency evidence README before relying on dependency behavior
- [ ] Preserve all pre-existing working-tree changes and stage only files belonging to this story
- [ ] Write .scratch/distributed-gguf-runtime/evidence/DGR-014/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
- [ ] Update only this story issue to Status: done after every acceptance criterion and quality gate passes
## Dependency handoff
- `DGR-001` must have `passes: true`; read `../evidence/DGR-001/README.md` and verify its referenced files/commands.
- `DGR-011` must have `passes: true`; read `../evidence/DGR-011/README.md` and verify its referenced files/commands.
- `DGR-012` must have `passes: true`; read `../evidence/DGR-012/README.md` and verify its referenced files/commands.
- `DGR-013` must have `passes: true`; read `../evidence/DGR-013/README.md` and verify its referenced files/commands.
## Finish contract
- Create the task evidence directory and durable handoff required above.
- Preserve real failures and blockers; never fabricate benchmark, model, test or hardware output.
- Change this issue to `Status: done` only after all criteria pass.
- Emit `<promise>COMPLETE</promise>` only after the evidence handoff exists.
## References
- [Ralph execution context](../RALPH-CONTEXT.md)
- [PRD](../PRD.md)
- [Implementation strategy](../implementation-strategy.md)
- [Current architecture](../architecture.md)
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)

61
.scratch/distributed-gguf-runtime/issues/15-add-and-certify-a-qwen3-qwen3-moe-adapter.md Normal file
View File

@@ -0,0 +1,61 @@
# 15 — Add and certify a Qwen3/Qwen3-MoE adapter
Status: ready-for-agent
## Mandatory fresh-session context
- Read [RALPH-CONTEXT.md](../RALPH-CONTEXT.md) completely before changing code.
- This issue is `DGR-015` in [prd.json](../prd.json).
- Read the evidence README for every dependency listed below.
- Inspect current code and `git status`; historical text and previous agent claims are not evidence.
## Description
As a client seeking top models, I need a separately certified MoE-capable architecture after the dense runtime proves stable.
## Expected durable outputs
- Qwen3-family architecture adapter
- Architecture-specific parity/admission/performance results
- evidence/DGR-015/README.md
## Acceptance criteria
- [ ] Implement explicit tensor ownership, router/top-k, expert/shared-expert, Q/K normalization, boundary bundle, and cache semantics for the selected Qwen3 family recipe.
- [ ] Do not reuse the dense-Llama adapter through unchecked name substitutions.
- [ ] Whole-model versus distributed prefill/decode parity passes the architecture-specific tolerance.
- [ ] Expert memory ownership and communication are measured.
- [ ] Real consumer-hardware acceptance and capability admission pass before the recipe becomes routable.
- [ ] Targeted pytest tests pass
- [ ] python -m compileall packages tests passes for Python changes
- [ ] git diff --check passes
- [ ] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
- [ ] Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
- [ ] Pinned native C++ target builds and focused CTest/protocol tests pass where native code is touched
- [ ] llama.cpp patch stack applies cleanly to the exact pinned commit where patch code is touched
- [ ] Real-model execution is opt-in through MESHNET_ENABLE_REAL_INFERENCE_TESTS=1 and records exact artifact/runtime/hardware evidence
- [ ] Model artifacts remain on the configured mounted-drive storage and never under /home
- [ ] Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
- [ ] Read and verify every dependency evidence README before relying on dependency behavior
- [ ] Preserve all pre-existing working-tree changes and stage only files belonging to this story
- [ ] Write .scratch/distributed-gguf-runtime/evidence/DGR-015/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
- [ ] Update only this story issue to Status: done after every acceptance criterion and quality gate passes
## Dependency handoff
- `DGR-014` must have `passes: true`; read `../evidence/DGR-014/README.md` and verify its referenced files/commands.
## Finish contract
- Create the task evidence directory and durable handoff required above.
- Preserve real failures and blockers; never fabricate benchmark, model, test or hardware output.
- Change this issue to `Status: done` only after all criteria pass.
- Emit `<promise>COMPLETE</promise>` only after the evidence handoff exists.
## References
- [Ralph execution context](../RALPH-CONTEXT.md)
- [PRD](../PRD.md)
- [Implementation strategy](../implementation-strategy.md)
- [Current architecture](../architecture.md)
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)

60
.scratch/distributed-gguf-runtime/issues/16-produce-the-upstream-llama-cpp-collaboration-package.md Normal file
View File

@@ -0,0 +1,60 @@
# 16 — Produce the upstream llama.cpp collaboration package
Status: ready-for-agent
## Mandatory fresh-session context
- Read [RALPH-CONTEXT.md](../RALPH-CONTEXT.md) completely before changing code.
- This issue is `DGR-016` in [prd.json](../prd.json).
- Read the evidence README for every dependency listed below.
- Inspect current code and `git status`; historical text and previous agent claims are not evidence.
## Description
As a maintainer, I need narrow upstreamable proposals so that our patch burden can shrink without asking llama.cpp to own Meshnet networking.
## Expected durable outputs
- Narrow upstream patches/tests
- Generic API design note
- Human-ready llama.cpp outreach package
- evidence/DGR-016/README.md
## Acceptance criteria
- [ ] Separate generic llama.cpp hooks from Meshnet protocol/control-plane code.
- [ ] Prepare minimal reproducible examples and tests for range-aware loading, boundary input/output, and layer-filtered KV.
- [ ] Compare the proposal with Nakshatra and prima.cpp evidence and explain why the API is generally useful.
- [ ] Preserve one scoped commit/patch per concern against the exact upstream pin.
- [ ] Produce an outreach document suitable for Georgi/llama.cpp maintainers; actual sending remains a human action.
- [ ] Targeted pytest tests pass
- [ ] python -m compileall packages tests passes for Python changes
- [ ] git diff --check passes
- [ ] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
- [ ] Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
- [ ] Pinned native C++ target builds and focused CTest/protocol tests pass where native code is touched
- [ ] llama.cpp patch stack applies cleanly to the exact pinned commit where patch code is touched
- [ ] Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
- [ ] Read and verify every dependency evidence README before relying on dependency behavior
- [ ] Preserve all pre-existing working-tree changes and stage only files belonging to this story
- [ ] Write .scratch/distributed-gguf-runtime/evidence/DGR-016/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
- [ ] Update only this story issue to Status: done after every acceptance criterion and quality gate passes
## Dependency handoff
- `DGR-010` must have `passes: true`; read `../evidence/DGR-010/README.md` and verify its referenced files/commands.
## Finish contract
- Create the task evidence directory and durable handoff required above.
- Preserve real failures and blockers; never fabricate benchmark, model, test or hardware output.
- Change this issue to `Status: done` only after all criteria pass.
- Emit `<promise>COMPLETE</promise>` only after the evidence handoff exists.
## References
- [Ralph execution context](../RALPH-CONTEXT.md)
- [PRD](../PRD.md)
- [Implementation strategy](../implementation-strategy.md)
- [Current architecture](../architecture.md)
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)