popov/neuron-tai
1
0
Fork 0
Code Issues 53 Pull Requests Actions Packages Projects Releases Wiki Activity

chore: clean superseded GGUF scaffolding

Browse Source
This commit is contained in:
Dobromir Popov
2026-07-16 22:32:37 +03:00
parent 81b1fa6074
commit 369b2072cc
102 changed files with 219 additions and 15274 deletions
Download Patch File Download Diff File Expand all files Collapse all files

14
.scratch/distributed-gguf-runtime/PRD.md
View File

@@ -1,6 +1,6 @@
# PRD: Distributed GGUF Runtime
> **Specification status:** planning artifacts only. No distributed GGUF runtime is implemented by this materialization, no story has completion credit, and legacy files remain for the DGR-017 audit. `prd.json` is authoritative.
> **Specification status:** planning artifacts only. No distributed GGUF runtime is implemented. DGR-017 cleanup is complete; no runtime implementation story has completion credit. `prd.json` is authoritative.
## Goal
@@ -42,15 +42,15 @@ The canonical gate groups live in `prd.json`. Every story explicitly requires it
### DGR-017: Reconcile and clean the superseded DGR backlog
**Milestone:** M0 · **Mode:** AFK · **Triage:** `ready-for-agent` · **Depends on:** none
**Milestone:** M0 · **Mode:** AFK · **State:** complete · **Depends on:** none
Fresh Ralph session: read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md`, source issue `.scratch/distributed-gguf-runtime/issues/017-reconcile-and-clean-the-superseded-dgr-backlog.md`, and evidence READMEs for dependencies (none) before changing code. Inspect live source/tests rather than trusting legacy pass states. Objective: Audit implementation reality, void inherited completion credit, and clean misleading backlog/stub baggage while preserving attributable evidence and accepted research.
- [ ] Compare the branch, old DGR-001..016 issue/pass states, evidence, and actual runtime sources; classify each output as reusable, reference-only, blocked, obsolete, or absent.
- [ ] Record an authoritative old-to-new disposition and provenance; explicitly give no completion credit to any new story and note absent implementation/evidence.
- [ ] Remove or archive only artifacts the audit proves obsolete while preserving accepted ADRs, useful research, raw benchmark evidence, and attributable reusable work.
- [ ] Protect ignored build workspaces, generated protobuf outputs, Ralph logs, and model artifacts from accidental commits, and document every retained legacy artifact.
- [ ] Applicable shared quality gates in `prd.json` pass, and the evidence handoff records exact commands/results, changed files, limitations, and dependency handoff.
- [x] Compare the branch, old DGR-001..016 issue/pass states, evidence, and actual runtime sources; classify each output as reusable, reference-only, blocked, obsolete, or absent.
- [x] Record an authoritative old-to-new disposition and provenance; explicitly give no completion credit to any new story and note absent implementation/evidence.
- [x] Remove or archive only artifacts the audit proves obsolete while preserving accepted ADRs, useful research, raw benchmark evidence, and attributable reusable work.
- [x] Protect ignored build workspaces, generated protobuf outputs, Ralph logs, and model artifacts from accidental commits, and document every retained legacy artifact.
- [x] Applicable shared quality gates in `prd.json` pass, and the evidence handoff records exact commands/results, changed files, limitations, and dependency handoff.
### DGR-018: Define canonical Ralph and Gitea metadata schema

2
.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md
View File

@@ -1,6 +1,6 @@
# Ralph context: Distributed GGUF Runtime
> **Specification status:** planning artifacts only. No distributed GGUF runtime is implemented by this materialization, no story has completion credit, and legacy files remain for the DGR-017 audit. `prd.json` is authoritative.
> **Specification status:** planning artifacts only. No distributed GGUF runtime is implemented. DGR-017 cleanup is complete; no runtime implementation story has completion credit. `prd.json` is authoritative.
## Mandatory startup for every fresh story

2
.scratch/distributed-gguf-runtime/README.md
View File

@@ -1,6 +1,6 @@
# Distributed GGUF Runtime planning workspace
> **Specification status:** planning artifacts only. No distributed GGUF runtime is implemented by this materialization, no story has completion credit, and legacy files remain for the DGR-017 audit. `prd.json` is authoritative.
> **Specification status:** planning artifacts only. No distributed GGUF runtime is implemented. DGR-017 cleanup is complete; no runtime implementation story has completion credit. `prd.json` is authoritative.
## Locked scope

2
.scratch/distributed-gguf-runtime/architecture.md
View File

@@ -1,6 +1,6 @@
# Distributed GGUF Runtime architecture
> **Specification status:** planning artifacts only. No distributed GGUF runtime is implemented by this materialization, no story has completion credit, and legacy files remain for the DGR-017 audit. `prd.json` is authoritative.
> **Specification status:** planning artifacts only. No distributed GGUF runtime is implemented. DGR-017 cleanup is complete; no runtime implementation story has completion credit. `prd.json` is authoritative.
## Locked scope

2
.scratch/distributed-gguf-runtime/decision-framework.md
View File

@@ -1,6 +1,6 @@
# Distributed GGUF Runtime decision framework
> **Specification status:** planning artifacts only. No distributed GGUF runtime is implemented by this materialization, no story has completion credit, and legacy files remain for the DGR-017 audit. `prd.json` is authoritative.
> **Specification status:** planning artifacts only. No distributed GGUF runtime is implemented. DGR-017 cleanup is complete; no runtime implementation story has completion credit. `prd.json` is authoritative.
## Decision order

127
.scratch/distributed-gguf-runtime/evidence/DGR-001/README.md
View File

@@ -1,127 +0,0 @@
# DGR-001 — performance contract baseline
## Files changed
- `packages/node/meshnet_node/performance_contract.py`
- `tests/test_performance_contract.py`
- `.scratch/distributed-gguf-runtime/issues/01-lock-the-safetensors-versus-gguf-performance-contract.md`
- `.scratch/distributed-gguf-runtime/evidence/DGR-001/performance-contract.json`
## What this slice does
- Locks the DGR-001 benchmark contract in code.
- Pins the architecture-aligned baseline to **DeepSeek-V2-Lite-Chat** (`deepseek2`).
- Uses the same model on both sides of the comparison:
- **safetensors:** `deepseek-ai/DeepSeek-V2-Lite-Chat` in **BF16**
- **GGUF:** `second-state/DeepSeek-V2-Lite-Chat-GGUF` in **Q2_K**
- Exposes a machine-readable JSON contract with:
- benchmark lanes for `transformers` safetensors and `llama.cpp` GGUF on **CPU** and **GPU**
- concurrency levels `1` and `4`
- the required metrics list
- an explicit stop condition for “no meaningful speed or fit benefit”
- Adds a deterministic stub benchmark report so the contract now has an executable report shape end to end.
## Recent benchmark runner slice
The runner currently uses a deterministic stub backend to exercise the comparison matrix without downloading a model. It emits:
- `.scratch/distributed-gguf-runtime/evidence/DGR-001/performance-contract.json`
- `.scratch/distributed-gguf-runtime/evidence/DGR-001/stub-benchmark-report.json`
The report includes per-device comparisons for:
- `transformers-safetensors-cpu` vs `llama-cpp-gguf-cpu`
- `transformers-safetensors-gpu` vs `llama-cpp-gguf-gpu`
and records the memory metric (`rss_bytes` on CPU, `vram_bytes` on GPU), decode speedup, artifact ratio, and output drift.
## Live endpoint CLI wiring
The contract CLI can now drive the live endpoint runner. Passing one `--live-endpoint LANE_ID=URL` mapping per contract lane (plus `--live-benchmark-out`) invokes `run_real_model_endpoint_benchmark` against already-running OpenAI-compatible servers and writes the report using the same schema as the stub:
```bash
PYTHONPATH=packages/node python -m meshnet_node.performance_contract \
--live-endpoint transformers-safetensors-cpu=http://127.0.0.1:8001 \
--live-endpoint llama-cpp-gguf-cpu=http://127.0.0.1:8002 \
--live-endpoint transformers-safetensors-gpu=http://127.0.0.1:8003 \
--live-endpoint llama-cpp-gguf-gpu=http://127.0.0.1:8004 \
--live-benchmark-out .scratch/distributed-gguf-runtime/evidence/DGR-001/live-benchmark-report.json
```
`--live-model` overrides the model name sent in requests (defaults to the contract's safetensors repo). Without any `--live-endpoint` flags the CLI behaves exactly as before: it writes the contract JSON and, with `--benchmark-out`, the deterministic stub report.
## Exact commands and real results
### Targeted tests
```bash
PYTHONPATH=packages/node pytest -q tests/test_performance_contract.py tests/test_route_session_benchmark.py
```
Result: `19 passed in 0.11s`
### Contract artifact generation
```bash
PYTHONPATH=packages/node python -m meshnet_node.performance_contract --json-out .scratch/distributed-gguf-runtime/evidence/DGR-001/performance-contract.json
```
Result: wrote `.scratch/distributed-gguf-runtime/evidence/DGR-001/performance-contract.json`
### Python compile check
```bash
python -m compileall packages/node/meshnet_node/performance_contract.py tests/test_performance_contract.py
```
Result: passed
## Public relay smoke benchmark (2026-07-15)
A real streamed request was run through the public tracker — **not** by connecting directly to the private node address:
```text
https://meshnet.2.d-popov.com/v1/chat/completions
-> wss://meshnet.2.d-popov.com/ws
-> wss://meshnet.2.d-popov.com/rpc/7j77FsPY1evV8tuf-7000
-> local CUDA node, Qwen/Qwen2.5-0.5B-Instruct layers 0-23
```
The local public-tracker node had an expired proof and a wedged HTTP server. A graceful restart refreshed its CUDA capability proof in `336 ms`, restored `admitted`/`routable` status, and reconnected its relay endpoint.
Measured streaming results after recovery:
| metric | result |
| --- | ---: |
| warm-up TTFT | 420.80 ms |
| warm-up elapsed | 610.23 ms |
| p50 TTFT (3 runs) | 288.26 ms |
| p50 elapsed (3 runs) | 363.20 ms |
| tracker-recorded relay throughput | 58.18-65.25 tok/s |
| HTTP status | 200 for all runs |
The tracker recorded `relay: true` and the local node ID `7j77FsPY-b32476219492` for each completion. Full redacted evidence is in `public-relay-smoke-benchmark.json`.
The other connected node is still alive but **not routable** because its capability proof is stale. It must revalidate before a multi-node shard/relay test can run.
## Limitations
- This slice still uses a deterministic stub backend for the core comparison matrix.
- It now also includes a live endpoint runner, reachable from the CLI via `--live-endpoint`/`--live-benchmark-out`, that fans out one OpenAI-compatible request per lane when the caller provides endpoints; the CLI does not start those servers.
- It does **not** download or run a real model from within the repo.
- Real safetensors vs GGUF execution, TTFT/prefill/decode measurements, RSS/VRAM capture, and output-drift comparison are still to be implemented against the contract.
## Compatibility notes
- The contract stays on the DeepSeek2 family to remain close to the DeepSeek-V4-Flash end goal.
- A smaller non-DeepSeek model can still be used later for loader-plumbing smoke tests, but it does not replace this baseline.
- Model artifacts must stay on the mounted drive and not under `/home`.
## Dependent-story handoff
Next implementation work should attach to this contract and add the live benchmark runner that actually compares:
1. current Transformers/safetensors recipe
2. whole-model llama.cpp GGUF recipe
using the same model architecture/revision and the same prompt/context/concurrency settings.

75
.scratch/distributed-gguf-runtime/evidence/DGR-001/performance-contract.json
View File

@@ -1,75 +0,0 @@
{
"benchmark_lanes": [
{
"concurrency_levels": [
1,
4
],
"device": "cpu",
"id": "transformers-safetensors-cpu",
"recipe": "current safetensors recipe",
"runtime": "transformers"
},
{
"concurrency_levels": [
1,
4
],
"device": "cpu",
"id": "llama-cpp-gguf-cpu",
"recipe": "whole-model GGUF recipe",
"runtime": "llama.cpp"
},
{
"concurrency_levels": [
1,
4
],
"device": "gpu",
"id": "transformers-safetensors-gpu",
"recipe": "current safetensors recipe",
"runtime": "transformers"
},
{
"concurrency_levels": [
1,
4
],
"device": "gpu",
"id": "llama-cpp-gguf-gpu",
"recipe": "whole-model GGUF recipe",
"runtime": "llama.cpp"
}
],
"metrics": [
"ttft_ms",
"prefill_tok_per_sec",
"decode_tok_per_sec",
"p50_latency_ms",
"p95_latency_ms",
"aggregate_throughput_tok_per_sec",
"rss_bytes",
"vram_bytes",
"artifact_bytes",
"failure_count",
"output_drift"
],
"model_target": {
"architecture": "deepseek2",
"comparison_policy": "same model/revision, closest practical low-footprint precision pair: BF16 safetensors versus Q2_K GGUF",
"gguf_quant": "Q2_K",
"gguf_repo": "second-state/DeepSeek-V2-Lite-Chat-GGUF",
"gguf_size_gb": 6.43,
"name": "DeepSeek-V2-Lite-Chat",
"rationale": "Smallest DeepSeek-family benchmark anchor that still points toward DeepSeek-V4-Flash; keeps the runtime on the DeepSeek2 path instead of falling back to a tiny but architecture-mismatched smoke model.",
"safetensors_precision": "bfloat16",
"safetensors_repo": "deepseek-ai/DeepSeek-V2-Lite-Chat"
},
"notes": [
"Real model execution stays opt-in and must keep model artifacts on the mounted drive.",
"Use the tiny fallback only for loader plumbing smoke tests; it does not replace the architecture-aligned baseline."
],
"schema_version": 1,
"stop_condition": "Stop if GGUF does not provide a meaningful speed or fit benefit over the safetensors baseline for the chosen DeepSeek-family model target.",
"story_id": "DGR-001"
}

247
.scratch/distributed-gguf-runtime/evidence/DGR-001/stub-benchmark-report.json
View File

@@ -1,247 +0,0 @@
{
"comparisons": {
"cpu": {
"artifact_bytes_ratio": 0.2048,
"decode_speedup": 2.3333,
"gguf_benefit": true,
"gguf_lane": "llama-cpp-gguf-cpu",
"memory_bytes_ratio": 0.2152,
"memory_metric": "rss_bytes",
"output_drift": 0.0,
"safetensors_lane": "transformers-safetensors-cpu",
"ttft_speedup": 1.8947
},
"gpu": {
"artifact_bytes_ratio": 0.2048,
"decode_speedup": 1.5294,
"gguf_benefit": true,
"gguf_lane": "llama-cpp-gguf-gpu",
"memory_bytes_ratio": 0.2273,
"memory_metric": "vram_bytes",
"output_drift": 0.0,
"safetensors_lane": "transformers-safetensors-gpu",
"ttft_speedup": 1.6154
}
},
"lanes": [
{
"concurrency_levels": [
1,
4
],
"device": "cpu",
"id": "transformers-safetensors-cpu",
"output_tokens": [
"mesh",
"activation",
"seam",
"baseline"
],
"recipe": "current safetensors recipe",
"results": [
{
"concurrency": 1,
"metrics": {
"aggregate_throughput_tok_per_sec": 6.0,
"artifact_bytes": 33715493273,
"decode_tok_per_sec": 6.0,
"failure_count": 0,
"output_drift": 0.0,
"p50_latency_ms": 166.6667,
"p95_latency_ms": 208.3334,
"prefill_tok_per_sec": 45.0,
"rss_bytes": 35433480192,
"ttft_ms": 1800.0,
"vram_bytes": 0
}
},
{
"concurrency": 4,
"metrics": {
"aggregate_throughput_tok_per_sec": 20.4,
"artifact_bytes": 33715493273,
"decode_tok_per_sec": 5.1,
"failure_count": 0,
"output_drift": 0.0,
"p50_latency_ms": 196.0784,
"p95_latency_ms": 245.098,
"prefill_tok_per_sec": 38.25,
"rss_bytes": 35433480192,
"ttft_ms": 2340.0,
"vram_bytes": 0
}
}
],
"runtime": "transformers"
},
{
"concurrency_levels": [
1,
4
],
"device": "cpu",
"id": "llama-cpp-gguf-cpu",
"output_tokens": [
"mesh",
"activation",
"seam",
"baseline"
],
"recipe": "whole-model GGUF recipe",
"results": [
{
"concurrency": 1,
"metrics": {
"aggregate_throughput_tok_per_sec": 14.0,
"artifact_bytes": 6904159928,
"decode_tok_per_sec": 14.0,
"failure_count": 0,
"output_drift": 0.0,
"p50_latency_ms": 71.4286,
"p95_latency_ms": 89.2858,
"prefill_tok_per_sec": 90.0,
"rss_bytes": 7623566950,
"ttft_ms": 950.0,
"vram_bytes": 0
}
},
{
"concurrency": 4,
"metrics": {
"aggregate_throughput_tok_per_sec": 47.6,
"artifact_bytes": 6904159928,
"decode_tok_per_sec": 11.9,
"failure_count": 0,
"output_drift": 0.0,
"p50_latency_ms": 84.0336,
"p95_latency_ms": 105.042,
"prefill_tok_per_sec": 76.5,
"rss_bytes": 7623566950,
"ttft_ms": 1235.0,
"vram_bytes": 0
}
}
],
"runtime": "llama.cpp"
},
{
"concurrency_levels": [
1,
4
],
"device": "gpu",
"id": "transformers-safetensors-gpu",
"output_tokens": [
"mesh",
"activation",
"seam",
"baseline"
],
"recipe": "current safetensors recipe",
"results": [
{
"concurrency": 1,
"metrics": {
"aggregate_throughput_tok_per_sec": 34.0,
"artifact_bytes": 33715493273,
"decode_tok_per_sec": 34.0,
"failure_count": 0,
"output_drift": 0.0,
"p50_latency_ms": 29.4118,
"p95_latency_ms": 36.7647,
"prefill_tok_per_sec": 850.0,
"rss_bytes": 4294967296,
"ttft_ms": 420.0,
"vram_bytes": 35433480192
}
},
{
"concurrency": 4,
"metrics": {
"aggregate_throughput_tok_per_sec": 115.6,
"artifact_bytes": 33715493273,
"decode_tok_per_sec": 28.9,
"failure_count": 0,
"output_drift": 0.0,
"p50_latency_ms": 34.6021,
"p95_latency_ms": 43.2526,
"prefill_tok_per_sec": 722.5,
"rss_bytes": 4294967296,
"ttft_ms": 546.0,
"vram_bytes": 35433480192
}
}
],
"runtime": "transformers"
},
{
"concurrency_levels": [
1,
4
],
"device": "gpu",
"id": "llama-cpp-gguf-gpu",
"output_tokens": [
"mesh",
"activation",
"seam",
"baseline"
],
"recipe": "whole-model GGUF recipe",
"results": [
{
"concurrency": 1,
"metrics": {
"aggregate_throughput_tok_per_sec": 52.0,
"artifact_bytes": 6904159928,
"decode_tok_per_sec": 52.0,
"failure_count": 0,
"output_drift": 0.0,
"p50_latency_ms": 19.2308,
"p95_latency_ms": 24.0385,
"prefill_tok_per_sec": 640.0,
"rss_bytes": 1610612736,
"ttft_ms": 260.0,
"vram_bytes": 8053063680
}
},
{
"concurrency": 4,
"metrics": {
"aggregate_throughput_tok_per_sec": 176.8,
"artifact_bytes": 6904159928,
"decode_tok_per_sec": 44.2,
"failure_count": 0,
"output_drift": 0.0,
"p50_latency_ms": 22.6244,
"p95_latency_ms": 28.2805,
"prefill_tok_per_sec": 544.0,
"rss_bytes": 1610612736,
"ttft_ms": 338.0,
"vram_bytes": 8053063680
}
}
],
"runtime": "llama.cpp"
}
],
"model_target": {
"architecture": "deepseek2",
"comparison_policy": "same model/revision, closest practical low-footprint precision pair: BF16 safetensors versus Q2_K GGUF",
"gguf_quant": "Q2_K",
"gguf_repo": "second-state/DeepSeek-V2-Lite-Chat-GGUF",
"gguf_size_gb": 6.43,
"name": "DeepSeek-V2-Lite-Chat",
"rationale": "Smallest DeepSeek-family benchmark anchor that still points toward DeepSeek-V4-Flash; keeps the runtime on the DeepSeek2 path instead of falling back to a tiny but architecture-mismatched smoke model.",
"safetensors_precision": "bfloat16",
"safetensors_repo": "deepseek-ai/DeepSeek-V2-Lite-Chat"
},
"schema_version": 1,
"source": "stub-backend",
"stop_condition": {
"gguf_benefit": true,
"text": "Stop if GGUF does not provide a meaningful speed or fit benefit over the safetensors baseline for the chosen DeepSeek-family model target.",
"triggered": false
},
"story_id": "DGR-001"
}

176
.scratch/distributed-gguf-runtime/evidence/DGR-002/README.md
View File

@@ -1,176 +0,0 @@
# DGR-002 — Versioned gRPC Shard protocol: evidence
Status: done
Date: 2026-07-15
Evidence kind: **synthetic-unit** (schema round-trip + cross-language protobuf
compatibility). No model download, no GPU, no network, no API credits.
## Summary
Added the versioned Protocol Buffers schema that is the semantic contract between
Python and C++ Shards (ADR-0024), plus reproducible Python and C++ code
generation/build wiring and generated-schema round-trip + compatibility tests in
**both** languages. The schema defines one long-lived bidirectional gRPC stream
per Route Session Activation Seam, bounded prefill chunking, a small decode fast
path, and a versioned named-tensor bundle carrying every required identifier.
No existing runtime code was modified — this story is purely additive (a new
`.proto`, a `native_protocol` loader package, C++ build wiring, and one new test
module). Generated stubs are produced on demand into gitignored `build/`
directories, so nothing generated is committed.
## Files changed (all new)
- `packages/node/native/proto/shard_runtime.proto` — the schema (package
`meshnet.shard.v1`, proto3). Service `ShardRuntime` with `GetCapability`,
`Health`, `ActivateSession` (bidi stream), `Release`, `Cancel`.
- `packages/node/meshnet_node/native_protocol/__init__.py` — reproducible
on-demand `grpc_tools.protoc` codegen + loader (`load()`, `load_grpc()`) and
shared bundle helpers (`compute_checksum`, `verify_checksum`, `fragment_tensor`,
`reassemble_tensor`).
- `packages/node/native/scripts/generate_python.py` — standalone reproducible
Python generation (self-contained; does not import `meshnet_node`).
- `packages/node/native/scripts/generate_cpp.sh` — reproducible C++ generation
(message stubs always; gRPC service stubs when `grpc_cpp_plugin` is present).
- `packages/node/native/CMakeLists.txt` — C++ build wiring; works with both
CONFIG-mode (`protobuf::libprotobuf`/`protobuf::protoc`) and CMake's
`FindProtobuf` module.
- `packages/node/native/tests/roundtrip_test.cpp` — C++ round-trip / compat test
(`--selftest`, `--read`, `--write`).
- `tests/test_native_shard_protocol.py` — Python round-trip + compatibility tests
and the Python↔C++ cross-language driver.
## Acceptance criteria → evidence
- **Capability/health/session-stream/release/cancellation schema** — the
`ShardRuntime` service's five RPCs; `test_capability_and_health_round_trip`,
`test_session_stream_carries_open_prefill_decode_release_cancel`.
- **One long-lived bidi stream per Activation Seam with deadlines, cancellation,
flow control, structured errors** — `rpc ActivateSession (stream ...) returns
(stream ...)`. Deadlines: gRPC call deadline on direct transport, plus
`SessionOpen.deadline_unix_nanos` for relay-carried frames. Cancellation:
`Cancel` RPC and in-stream `CancelRequest`/`PHASE_CANCEL`. Flow control:
`FlowControl` frames (credits + in-flight byte/message caps). Structured errors:
`Status` (canonical code, message, `RetryClass`, details). Verified by
`test_session_response_carries_structured_status_and_results`.
- **Bounded prefill chunking + small decode fast path** — `PrefillChunk`
(`chunk_index`/`chunk_count`/`final_chunk`, `SessionOpen.max_prefill_tokens_per_chunk`)
and `DecodeStep` (minimal single-bundle path). Bounded fragments via
`SessionOpen.max_fragment_bytes` and `fragment_tensor(...)`.
- **Carries schema version, work ID, Route Session ID, route epoch,
artifact/recipe fingerprint, shard range/effective start, phase, position,
idempotency step, cache expectation, compression, checksum** — all on
`MessageHeader` (+ `ArtifactFingerprint.runtime_recipe_fingerprint`,
`ShardRange.effective_start_layer`). Verified field-by-field by
`test_message_header_carries_every_required_field`.
- **Versioned named-tensor bundle (name, shape, dtype, byte order, fragments)** —
`TensorBundle`/`NamedTensor`/`TensorFragment`;
`test_named_tensor_bundle_describes_shape_dtype_byteorder_and_fragments`,
`test_fragment_and_reassemble_round_trip_with_checksums`.
- **Round-trip + compatibility tests in Python and C++** — Python:
`tests/test_native_shard_protocol.py` (11 tests). C++: `roundtrip_test.cpp`
built via CMake; cross-language driver `test_cross_language_roundtrip_python_and_cpp`
exercises Python→C++ and C++→Python in both directions.
- **Targeted pytest** — `11 passed, 1 skipped` (default env); `12 passed` with the
C++ toolchain on PATH.
- **compileall packages tests** — exit 0.
- **git diff --check** — clean.
- **Deterministic / download-free / credit-free / GPU-free** — all tests are pure
protobuf serialization; the C++ path uses only local compilers.
- **Full deterministic pytest** — `704 passed, 14 skipped, 11 failed`. The 11
failures are pre-existing and unrelated (see below).
## Commands and real results
See `commands.txt` for the exact command list. Key results:
- `python packages/node/native/scripts/generate_python.py` →
`shard_runtime_pb2.py: ok`, `shard_runtime_pb2_grpc.py: ok`.
- `pytest tests/test_native_shard_protocol.py -q` → **11 passed, 1 skipped**
(skip reason: `C++ toolchain unavailable: cmake not found on PATH`).
- With `/tmp/pbsrc/install/bin` (protoc 33.1) and `.venv/bin` (cmake) on PATH and
`CMAKE_PREFIX_PATH=/tmp/pbsrc/install`:
- `generate_cpp.sh` → `shard_runtime.pb.cc`, `shard_runtime.pb.h`
(grpc service stubs skipped: `grpc_cpp_plugin` absent).
- `cmake -S ... -B ...` + `cmake --build ...` → build OK.
- `shard_protocol_roundtrip_test --selftest` → `selftest ok (128 bytes)`, exit 0.
- `ctest` → `1/1 Test #1: shard_protocol_roundtrip ... Passed`.
- `pytest ...::test_cross_language_roundtrip_python_and_cpp -q` → **1 passed**
(Python serializes → C++ parses & verifies → C++ serializes → Python parses
& verifies).
- `compileall -q packages tests` → exit 0.
- `git diff --check` → clean.
## Pre-existing unrelated failures (full-suite)
`pytest -q` on the full tree reports 11 failures, all in tracker routing /
dynamic routing / manual route benchmark / toploc calibration — none import the
Shard protocol. Clean-tree reproduction: with **all DGR-002 files moved aside**
(`git status` shows only the pre-existing `.ralph-tui/config.toml` deletion),
re-running exactly these tests gives `11 failed, 3 passed` — identical failures.
They exist on the `ralph/distributed-gguf-runtime` branch independent of this
story. The full list is in `results.json.preexisting_unrelated_failures`.
Note: the earlier `progress.md` (RCR-001, on master) recorded a different set of
6 optional-dependency failures (zstandard, langchain_openai). Those did **not**
recur here; this environment has those deps. The 11 above are branch-local
routing/benchmark failures, not environmental.
## Limitations and deferred work
- **C++ toolchain is host-provided, not vendored.** The default test env has no
`protoc`/`cmake`/protobuf C++ headers on PATH, so the C++ cross-language test
**skips** by default (explicit skip reason). It was executed for this evidence
using an ephemeral from-source protobuf 33.1 install at `/tmp/pbsrc/install`
plus the `.venv` cmake. DGR-004/DGR-008 should pin the C++ protobuf/gRPC
toolchain (upstream commit + reproducible fetch/build) so this test runs in CI
without relying on an ad-hoc `/tmp` install.
- **gRPC C++ service stubs not built here.** `grpc_cpp_plugin` is absent, so
`generate_cpp.sh` produced message stubs only. The round-trip test needs only
message serialization; the service stubs are DGR-008's concern.
- **No live gRPC transport yet.** This story delivers the schema + serialization
contract and generation/build wiring only. Channel setup, the bidi stream
server/client, deadlines/cancellation propagation over a real HTTP/2 channel,
and relay framing are DGR-008/DGR-009.
- **Protobuf runtime version skew.** Python runtime is pip protobuf 7.35.1; the
C++ side used protoc 33.1. Protobuf wire format is stable across these, and the
cross-language round-trip confirms interop; version pinning is deferred to the
toolchain-pinning stories.
## Compatibility / migration notes
- proto3 with a 0-valued `*_UNSPECIFIED` member on every enum and never-reused
field numbers. Forward compatibility (unknown-field preservation) is verified
behaviourally by `test_unknown_fields_are_preserved_for_forward_compatibility`
— note protobuf 7.x's upb backend does not implement the `UnknownFields()`
introspection accessor, so the test asserts the observable re-serialization
outcome instead. Backward defaults verified by
`test_defaults_are_stable_for_backward_compatibility`.
- Wire schema version is `SchemaVersion.SCHEMA_VERSION_1` (int 1), also exposed as
`meshnet_node.native_protocol.SCHEMA_VERSION`.
## Handoff for dependent stories
- **DGR-003 (recipe/fingerprint):** populate `ArtifactFingerprint`
(`model_id`, `revision`, `artifact_hash`, `quantization`,
`runtime_recipe_fingerprint`). Admission compares these before activation; a
mismatch is a fatal `Status` (`RetryClass.RETRY_CLASS_FATAL`).
- **DGR-004 (llama.cpp pin) / DGR-008 (C++ worker):** pin the C++
protobuf + gRPC toolchain and add `grpc_cpp_plugin`; then `generate_cpp.sh`
emits service stubs and the CMake target can link gRPC. Implement the
`ShardRuntime` servicer; map `(route_session_id, route_epoch)` to an isolated
llama sequence. Use `SessionOpen` for stream-scoped bounds and `FlowControl`
for backpressure.
- **DGR-009 (Meshnet integration/relay):** the relay may carry serialized
`SessionActivation`/`SessionResponse` frames as opaque binary; use the in-message
`deadline_unix_nanos`, `CancelRequest`, and `FlowControl` since gRPC call
metadata is lost over relay.
- **Loader usage:** `from meshnet_node import native_protocol as proto;
pb2 = proto.load()`. Stubs regenerate automatically when the `.proto` changes
(mtime check). `proto.load_grpc()` returns the service stubs (needs the `grpc`
runtime).
- **Gotcha:** the `.venv` installs the meshnet packages editable via a PEP 660
meta-path finder pointing at the **main** checkout. Import the worktree copy by
ensuring the worktree `packages/node` is on `sys.path` first (conftest already
does this for pytest); standalone tooling must derive paths from `__file__` and
not `import meshnet_node` (why `generate_python.py` is self-contained).

40
.scratch/distributed-gguf-runtime/evidence/DGR-002/commands.txt
View File

@@ -1,40 +0,0 @@
# DGR-002 reproduction commands (run from repo root, project .venv = Python 3.14).
# 1. Generate Python stubs (reproducible; writes to gitignored build/ dir).
.venv/bin/python packages/node/native/scripts/generate_python.py
# 2. Python round-trip + compatibility tests (default env; C++ test skips if
# cmake/protoc absent).
.venv/bin/python -m pytest tests/test_native_shard_protocol.py -q
# => 11 passed, 1 skipped
# 3. Quality gates.
.venv/bin/python -m compileall -q packages tests # exit 0
git diff --check # clean
# 4. Full deterministic suite (records pre-existing unrelated failures).
.venv/bin/python -m pytest -q
# => 704 passed, 14 skipped, 11 failed (all pre-existing, unrelated; see below)
# 5. Clean-tree reproduction of the 11 pre-existing failures (DGR-002 files moved
# aside): same 11 fail => not caused by this story.
# --- C++ / cross-language (requires protoc + protobuf C++ dev + cmake) --------
# On this host a from-source protobuf 33.1 toolchain lives under /tmp/pbsrc/install
# and cmake ships in the .venv. To execute the C++ test instead of skipping it:
export PATH="/tmp/pbsrc/install/bin:$PWD/.venv/bin:$PATH"
export CMAKE_PREFIX_PATH="/tmp/pbsrc/install:$CMAKE_PREFIX_PATH"
# 6. Generate C++ stubs (message stubs always; gRPC service stubs if
# grpc_cpp_plugin present).
packages/node/native/scripts/generate_cpp.sh
# 7. Standalone C++ build + selftest + ctest.
cmake -S packages/node/native -B packages/node/native/build/cpp
cmake --build packages/node/native/build/cpp --target shard_protocol_roundtrip_test
packages/node/native/build/cpp/shard_protocol_roundtrip_test --selftest # "selftest ok (128 bytes)"
(cd packages/node/native/build/cpp && ctest --output-on-failure) # 1/1 passed
# 8. Cross-language Python<->C++ round-trip via the pytest driver (now runs, not skips).
.venv/bin/python -m pytest tests/test_native_shard_protocol.py::test_cross_language_roundtrip_python_and_cpp -q
# => 1 passed

63
.scratch/distributed-gguf-runtime/evidence/DGR-002/results.json
View File

@@ -1,63 +0,0 @@
{
"task": "DGR-002",
"title": "Adopt the versioned gRPC Shard protocol",
"schema": {
"proto": "packages/node/native/proto/shard_runtime.proto",
"package": "meshnet.shard.v1",
"syntax": "proto3",
"schema_version": 1,
"service": "ShardRuntime",
"rpcs": ["GetCapability", "Health", "ActivateSession", "Release", "Cancel"],
"streaming_seam": "ActivateSession (bidirectional stream)"
},
"toolchain": {
"python": "3.14.6",
"protobuf_runtime_python": "7.35.1",
"grpcio": "1.82.1",
"grpcio_tools": "1.82.1",
"cpp_protoc": "libprotoc 33.1",
"cpp_protobuf_toolchain": "/tmp/pbsrc/install (from-source protobuf 33.1, ephemeral host build)",
"cmake": "4.4.0 (.venv)",
"cxx": "g++ (system)"
},
"generation": {
"python_cmd": "python packages/node/native/scripts/generate_python.py",
"python_out": "packages/node/native/build/python/shard_runtime_pb2{,_grpc}.py (gitignored)",
"cpp_cmd": "packages/node/native/scripts/generate_cpp.sh",
"cpp_out": "packages/node/native/build/cpp-gen/shard_runtime.pb.{h,cc} (gitignored)",
"cpp_build": "cmake -S packages/node/native -B <build> && cmake --build <build>"
},
"tests": {
"python_default_env": {"passed": 11, "skipped": 1, "note": "C++ cross-language test skips when cmake/protoc absent"},
"python_with_cpp_toolchain": {"passed": 12, "skipped": 0},
"cpp_selftest_bytes": 128,
"cpp_ctest": "1/1 passed",
"cross_language": "Python->C++ and C++->Python round-trip verified in both directions"
},
"quality_gates": {
"targeted_pytest": "11 passed, 1 skipped (default); 12 passed with C++ toolchain",
"compileall_packages_tests": "exit 0",
"git_diff_check": "clean",
"full_pytest": {
"passed": 704,
"skipped": 14,
"failed": 11,
"failed_are_preexisting_unrelated": true,
"clean_tree_reproduction": "same 11 fail with all DGR-002 files removed (11 failed, 3 passed)"
}
},
"preexisting_unrelated_failures": [
"tests/test_dynamic_routing.py::test_admin_can_replace_a_served_model_and_release_it",
"tests/test_manual_route_benchmark.py::test_pinned_route_uses_named_node",
"tests/test_manual_route_benchmark.py::test_unknown_route_node_is_400",
"tests/test_manual_route_benchmark.py::test_invalid_route_shape_is_400",
"tests/test_manual_route_benchmark.py::test_clients_without_route_are_unaffected",
"tests/test_manual_route_benchmark.py::test_benchmark_records_one_and_two_node_routes",
"tests/test_toploc_calibration_dispatch.py::test_calibration_run_dispatches_only_solo_capable_nodes",
"tests/test_toploc_calibration_dispatch.py::test_calibration_run_persists_corpus_and_results_endpoint_reports_it",
"tests/test_toploc_calibration_dispatch.py::test_calibration_run_node_without_commitment_endpoint_is_skipped_not_failed",
"tests/test_tracker_routing.py::test_torch_node_applies_tracker_load_shard_directive",
"tests/test_tracker_routing.py::test_shard_heal_cycle_surviving_node_covers_dead_peers_gap"
],
"evidence_kind": "synthetic-unit (schema round-trip + cross-language protobuf; no model, no GPU, no network, no API credits)"
}

86
.scratch/distributed-gguf-runtime/evidence/DGR-003/README.md
View File

@@ -1,86 +0,0 @@
# DGR-003 — Exact artifact and runtime-recipe identity: evidence
Status: done
Date: 2026-07-15
Evidence kind: **synthetic-unit + repo checks**. No model download, no GPU, no network, no API credits.
## Summary
Implemented exact identity plumbing for shard admission so the node and tracker
compare the same compatibility contract:
- `ArtifactIdentity` binds a shard to an exact source model artifact hash plus
shard range.
- `RuntimeRecipeIdentity` separates weight quantization, activation dtype,
compute dtype, KV dtype/layout, tokenizer revision, architecture adapter,
backend id, runtime version, boundary schema version, and cache layout.
- `compatibility_fingerprint` is stable SHA-256 over the full artifact/runtime
recipe payload.
- Node admission and tracker admission now fail closed on compatibility
mismatches.
- Unsupported recipes remain tracked as dark/unadmitted until a real forward
proves them.
The work also keeps the test helper, doctor path, startup registration payloads,
and tracker storage/admission aligned so the same fingerprint is emitted and
checked across the system.
## Files changed
- `packages/node/meshnet_node/runtime_recipe.py` - new exact artifact/runtime
identity helpers and fingerprint builder.
- `packages/node/meshnet_node/capability.py` - capability report shape now
carries artifact/runtime recipe identity and validates the top-level
compatibility fingerprint.
- `packages/node/meshnet_node/admission.py` - fail-closed admission on
compatibility fingerprint mismatch.
- `packages/node/meshnet_node/doctor.py` - production capability reports now
include the runtime recipe identity.
- `packages/node/meshnet_node/testing.py` - test report builder now mirrors the
production fingerprint fields.
- `packages/node/meshnet_node/startup.py` - registration payload now includes
the compatibility fingerprint.
- `packages/tracker/meshnet_tracker/capability.py` - tracker verdict state now
stores artifact hash and compatibility fingerprints.
- `packages/tracker/meshnet_tracker/server.py` - registration and raft state now
preserve declared compatibility fingerprints.
- `tests/test_node_capability.py` - identity shape and fingerprint regression
tests.
- `tests/test_node_admission.py` - fail-closed admission regression tests.
- `tests/test_tracker_capability_admission.py` - tracker compatibility mismatch
regression tests.
## Commands and real results
- `python -m compileall packages tests` -> exit 0.
- `pytest -q tests/test_node_capability.py` -> `48 passed in 0.09s`.
- `pytest -q tests/test_node_admission.py` -> `20 passed in 0.11s`.
- `pytest -q tests/test_tracker_capability_admission.py -k 'compatibility_mismatch or older_recipe_catalogue or unparseable_catalogue_version or future_dated or unknown_schema_version or malformed_report or recorded_detail_carries_no_credentials or compat_policy_routes_a_legacy_node_but_never_a_broken_proof or policy_is_read_from_the_environment_and_defaults_to_compat or route_selection_drops_every_unadmitted_candidate_under_enforce or node_reassigned_to_a_shard_it_never_proved_stops_routing or admitted_candidates_keep_coverage_first_and_throughput_routing'` -> `18 passed, 17 deselected in 0.11s`.
- `git diff --check` -> exit 0.
- `pytest -q` -> not green in this sandbox. Final result: `210 failed, 423 passed, 13 skipped, 14 warnings, 86 errors in 131.34s`.
## Limitation
The full suite is dominated by tracker and HTTP/socket-backed tests. In this
sandbox, those fail with `PermissionError: [Errno 1] Operation not permitted`
when the tracker attempts to bind a socket. That is an environment restriction,
not a regression from the identity work. The pure unit slices above pass.
## Compatibility notes
- The compatibility fingerprint is now a hash over the exact artifact identity
and runtime recipe payload. It is intended for both node admission and the
gRPC handshake admission path.
- Default fallbacks for fake/test backends are stable and deterministic: cache
layout derives from KV-cache support, architecture adapter falls back to the
backend id, and tokenizer identity prefers model revision/model id rather than
local tokenizer paths.
## Handoff for dependent stories
- DGR-004 / DGR-008 can reuse `runtime_recipe.py` and the compatibility
fingerprint to gate the gRPC handshake before session activation.
- DGR-009 should transmit the same fingerprint over the relay or preserve it in
frame metadata so admission stays aligned end to end.
- Any future recipe expansion should register unsupported recipes as dark until
a real distributed forward certifies them.

130
.scratch/distributed-gguf-runtime/evidence/DGR-004/README.md
View File

@@ -1,130 +0,0 @@
# DGR-004 — reproducible pinned llama.cpp patch stack evidence
Status: done
Date: 2026-07-15
Evidence kind: **synthetic-build + repo checks**. No model download, no GPU,
no network fetch during validation, no API credits.
## Summary
Implemented the reproducible source-dependency boundary for llama.cpp and kept
the fork seam narrow and auditable:
- exact pinned upstream commit and repository metadata
- numbered patch stack isolated under `packages/node/native/llama/patches/`
- build script that verifies the pin, applies the patch stack, stages notices,
and compiles a standalone worker scaffold without manual source copying
- upstream file assumptions and fail-closed pin checking
- license/attribution preservation by staging upstream `LICENSE` and `AUTHORS`
- clean rebuild smoke test that only uses a fake local checkout and does not
download a model
The native smoke path is intentionally minimal in this story. It proves the
reproducible source dependency and build seam without pulling Meshnet protocol
code into llama.cpp.
## Files changed
- `packages/node/native/llama/UPSTREAM_COMMIT`
- `packages/node/native/llama/UPSTREAM_REPOSITORY`
- `packages/node/native/llama/UPSTREAM_ASSUMPTIONS.md`
- `packages/node/native/llama/README.md`
- `packages/node/native/llama/patches/0001-add-meshnet-worker-scaffold.patch`
- `packages/node/native/llama/templates/meshnet_worker.cpp`
- `packages/node/native/scripts/build_llama_worker.sh`
- `tests/test_llama_worker_build.py`
## Exact commands and real results
### Native smoke build against a fake pinned checkout
```bash
tmpdir=$(mktemp -d)
mkdir -p "$tmpdir/llama.cpp"
printf 'MIT\n' > "$tmpdir/llama.cpp/LICENSE"
printf 'AUTHORS\n' > "$tmpdir/llama.cpp/AUTHORS"
printf '# placeholder\n' > "$tmpdir/llama.cpp/CMakeLists.txt"
printf '%s\n' 'b3c9d1b846cc80a6360adb6aeaa4fcd8c4c8dcac' > "$tmpdir/llama.cpp/.meshnet-upstream-commit"
git init -q "$tmpdir/llama.cpp"
packages/node/native/scripts/build_llama_worker.sh \
--source-dir "$tmpdir/llama.cpp" \
--build-dir "$tmpdir/build"
```
Result:
- `meshnet worker scaffold ok`
- `upstream commit: b3c9d1b846cc80a6360adb6aeaa4fcd8c4c8dcac`
- `patchset version: 0001`
- `build ok: /tmp/.../build/meshnet_worker`
### Targeted pytest
```bash
python -m pytest -q tests/test_llama_worker_build.py
```
Result: `1 passed in 0.53s`
### Python compile check
```bash
python -m compileall -q packages tests
```
Result: exit 0
### Diff hygiene
```bash
git diff --check
```
Result: exit 0
### Full deterministic pytest
```bash
python -m pytest -q
```
Result: `424 passed, 13 skipped, 210 failed, 86 errors in 131.04s`
The failures are pre-existing sandbox socket failures in tracker/HTTP-backed
tests. Representative error:
- `PermissionError: [Errno 1] Operation not permitted` when the tracker tries
to bind a socket.
This matches the previously observed environment limitation in the DGR-002 and
DGR-003 evidence and is unrelated to the llama.cpp pin/build scaffold.
## Limitations
- The sandbox does not provide `cmake`, so the smoke build uses the available
direct C++ compiler path (`g++` here) instead of a CMake-generated target.
- The pinned upstream source was not fetched from GitHub during validation.
The script supports fetching the exact commit when network access is
available, but the validation run used a fake local checkout to keep the test
deterministic and model-free.
- The patch stack in this story is deliberately narrow and additive. It creates
a worker scaffold and build seam, not the final llama.cpp runtime patches.
## Compatibility notes
- The exact upstream pin is `b3c9d1b846cc80a6360adb6aeaa4fcd8c4c8dcac`.
- The build script fails closed if the checkout pin differs from that commit or
if the expected upstream files (`LICENSE`, `AUTHORS`, `CMakeLists.txt`) are
missing.
- The patch stack is isolated from Meshnet networking code and can be applied
to a clean pinned checkout before later worker stories extend the scaffold.
- Upstream attribution notices are preserved in the build output by copying the
staged `LICENSE` and `AUTHORS` files into `build/.../upstream-notices/`.
## Dependent-story handoff
- DGR-008 can replace the scaffold source with the real supervised C++ worker
while keeping the same pin metadata, patch stack, and build script boundary.
- DGR-005 and later native stories should keep using the same exact pin so the
worker seam remains reproducible while range-loading and session logic are
added.

96
.scratch/distributed-gguf-runtime/evidence/DGR-005/README.md
View File

@@ -1,96 +0,0 @@
# DGR-005 — dense-Llama range-aware GGUF ownership evidence
Status: done
Date: 2026-07-15
Evidence kind: **synthetic-unit + repo checks**. No model download, no GPU, no network, no API credits.
## Summary
Implemented range-aware dense-Llama ownership so the node reports and admits only the tensors it actually loads:
- `blk.N.*` tensors are selected strictly by assigned layer range.
- Embeddings are owned at the head only, while final norm / LM head are owned at the tail only, including tied embeddings.
- Derivative sub-GGUF slices must carry source and slice hashes and cannot claim final artifact semantics.
- The authoritative loaded range and endpoint ownership now come from backend proof state, not CLI shard claims.
- Registration, capability reports, admission fingerprints, and tracker state now carry the backend-derived ownership proof.
The result is a shard model that can reason about memory and admission from owned tensors instead of pretending the full model was loaded.
## Files changed
- `packages/node/meshnet_node/gguf_ownership.py` - dense-Llama tensor selection and authoritative ownership helpers.
- `packages/node/meshnet_node/capability.py` - shard reports now carry endpoint ownership and parse it round-trip.
- `packages/node/meshnet_node/doctor.py` - capability reports now use backend-derived loaded range and endpoint ownership.
- `packages/node/meshnet_node/testing.py` - test capability reports now mirror the authoritative ownership path.
- `packages/node/meshnet_node/admission.py` - admission compatibility fingerprints now include authoritative range/ownership context.
- `packages/node/meshnet_node/model_backend.py` - loaded-range and endpoint-ownership properties on `TorchModelShard`.
- `packages/node/meshnet_node/startup.py` - registration payloads now use the proof-driven shard range.
- `packages/tracker/meshnet_tracker/capability.py` - tracker capability state preserves endpoint ownership.
- `tests/test_gguf_ownership.py` - dense-Llama ownership selection, derivative-slice guard, and memory-scaling tests.
- `tests/test_node_capability.py` - capability report ownership round-trip tests.
- `tests/test_node_admission.py` - backend-loaded range beats CLI claim regression tests.
- `tests/test_tracker_capability_admission.py` - tracker capability proof parsing tests.
## Exact commands and real results
### Targeted pytest slices
```bash
python -m pytest -q tests/test_gguf_ownership.py tests/test_node_capability.py tests/test_node_admission.py
```
Result: `73 passed`
```bash
python -m pytest -q tests/test_tracker_capability_admission.py -k 'test_a_passing_report_that_covers_the_registration_is_admitted or test_a_missing_report_is_absent_not_admitted or test_a_failed_report_is_recorded_as_failed or test_a_report_for_a_different_model_is_a_model_mismatch or test_a_report_for_a_different_shard_is_a_shard_mismatch or test_a_report_for_a_different_recipe_than_the_node_declares_is_a_recipe_mismatch or test_a_report_for_a_different_compatibility_fingerprint_is_a_compatibility_mismatch or test_an_older_recipe_catalogue_is_incompatible or test_an_unparseable_catalogue_version_is_incompatible or test_a_stale_report_is_not_admitted or test_a_future_dated_report_is_not_admitted or test_a_report_from_an_unknown_schema_version_is_invalid or test_a_malformed_report_is_invalid_and_never_admitted or test_recorded_detail_carries_no_credentials_from_node_diagnostics or test_compat_policy_routes_a_legacy_node_but_never_a_broken_proof or test_the_policy_is_read_from_the_environment_and_defaults_to_compat'
```
Result: `22 passed, 13 deselected`
### Python compile check
```bash
python -m compileall -q packages tests
```
Result: exit 0
### Diff hygiene
```bash
git diff --check
```
Result: exit 0
### Full deterministic pytest
```bash
python -m pytest -q
```
Result: `211 failed, 428 passed, 13 skipped, 14 warnings, 86 errors in 135.03s`
The failing set is not caused by this story. The dominant environment issues were:
- tracker and HTTP/socket-backed tests fail with `PermissionError: [Errno 1] Operation not permitted` when the tracker tries to bind sockets in this sandbox
- native protocol tests fail early with a protobuf runtime/gencode mismatch: generated code expects protobuf 7.35.0 while the installed runtime is 6.33.6
## Limitations
- This evidence is intentionally deterministic and model-free.
- The memory-scaling check is synthetic: it validates that owned tensor bytes scale with selected tensors, not a live GGUF download.
- Native C++ code was not changed by this story, so the pinned llama.cpp build validation remains covered by DGR-004 rather than repeated here.
## Compatibility notes
- Dense-Llama ownership is range-first: the shard interior is `blk.N.*`, and endpoint tensors are only attributed to the head or tail owner as appropriate.
- Derivative GGUF slices are explicitly not final artifacts; they must preserve source and slice hashes if used as a temporary compatibility bridge.
- The model proof path is authoritative for reported range and endpoint ownership, so operator CLI claims no longer control what the node advertises.
- Admission and tracker state now consume the same proof-derived ownership shape, keeping capability reports aligned end to end.
## Handoff for dependent stories
- DGR-006 can reuse `gguf_ownership.py` and the new capability fields to wire the shard protocol to proof-derived ownership without re-deriving tensor names.
- DGR-008 and later routing work should continue to treat endpoint ownership as metadata and `blk.N.*` ownership as the core range contract.
- If a future temporary slice path is needed, it should keep source/slice hashes visible and avoid claiming final-artifact semantics until a real proof exists.

203
.scratch/distributed-gguf-runtime/evidence/DGR-006/README.md
View File

@@ -1,203 +0,0 @@
# DGR-006 — Architecture-defined boundary input/output: evidence
Status: done
Date: 2026-07-15
Evidence kind: **synthetic-unit** (pure-numpy dense-Llama reference + boundary
contract). No model download, no GPU, no torch, no network, no API credit.
## Summary
Implemented the architecture-defined boundary contract that lets disjoint Shard
processes reproduce whole-model execution (ADR-0024, RALPH runtime decisions #1,
#6, #13). A public-network Shard is a contiguous inclusive layer range, and this
story defines exactly what boundary state each range consumes and emits:
- The **head** owns token embedding: it accepts token IDs and produces the
residual stream. It refuses an upstream boundary bundle.
- **Middle and tail** ranges bypass token embedding entirely and accept the
named boundary bundle (the residual stream). They refuse token IDs.
- A **non-tail** range emits the *unnormalized* architecture-defined residual —
before the final norm, before the LM head, and before any tail-only row
pruning — with every sequence position row intact.
- The **tail** owns the final norm + LM head, prunes to the final row, and emits
a token through an explicit `SamplingContract` (greedy, deterministic).
- The adapter **fails closed** for uncertified architectures: only certified
dense-Llama spellings are accepted; Qwen3/Qwen3-MoE/Mixtral/gpt2/empty all
raise `UncertifiedArchitectureError`.
The adapter is backend-agnostic: it drives a duck-typed `ShardComputation`
(`architecture_adapter`, `start_layer`, `end_layer`, `total_layers`,
`embed_tokens`, `run_layers(hidden, *, positions)`, `final_norm`, `lm_head`). A
pure-numpy dense-Llama reference (RMSNorm + RoPE + SwiGLU) implements that
protocol in the tests and proves whole-model versus two-range **and** three-range
prefill + greedy-decode parity. torch/transformers are not installed in the
default `.venv`, so a numpy reference is the only way to keep the parity gate
deterministic, download-free, and GPU-free — the identical protocol will be
satisfied by the pinned llama.cpp worker (DGR-008) and the PyTorch backend.
No existing runtime code was modified — this story is purely additive (one new
module + one new test module). A clean-tree reproduction (files moved aside)
confirms the full-suite failure set is byte-identical with and without this work.
## Files changed (all new)
- `packages/node/meshnet_node/boundary_adapter.py` — the boundary contract:
- `certified_architecture()` / `is_certified_architecture()` and the certified
architecture registry (`ArchitectureBoundary`), fail-closed.
- `ShardRole` + `role_for_range()` (head/middle/tail/full).
- `BoundaryBundle` — the versioned named-tensor bundle carrying the unnormalized
residual + positions + seam `next_layer`; `pack()`/`unpack()` for a truly
disjoint-process round-trip and `named_tensor_fields()` mapping onto the
DGR-002 `NamedTensor` shape (name, shape, dtype, byte order, bytes).
- `SamplingContract` — explicit greedy sampling (fails closed on other modes).
- `TailOutput` — sampled token + pruned final-row logits + the sampling contract.
- `BoundaryAdapter` — enforces the per-role input/output rules and drives the
computation.
- `tests/test_boundary_adapter.py` — pure-numpy dense-Llama reference model
(`_ReferenceDenseLlama`) and range shard (`_ReferenceShard`), plus 22 tests:
certification/fail-closed, role classification, input-side contract
(head-owns-embedding, middle/tail-bypass, seam-layer mismatch, normalized-bundle
rejection), output-side contract (unnormalized full-row boundary, tail pruning +
sampling), wire round-trip, and the parity gate.
## Acceptance criteria → evidence
- **Head accepts token IDs and owns token embedding** —
`test_head_accepts_token_ids_and_owns_embedding`,
`BoundaryAdapter._ingest_tokens` (head requires token IDs, refuses a bundle).
- **Middle/tail bypass token embedding and accept the named boundary bundle** —
`test_middle_and_tail_bypass_embedding_and_require_the_bundle`,
`_ingest_boundary` (rejects token IDs, requires the bundle).
- **Non-tail emits the unnormalized boundary before final norm/head and before
tail-only row pruning** — `test_non_tail_emits_unnormalized_full_row_boundary`
asserts the bundle is `normalized=False`, shape `(1, seq, hidden)` (all rows),
and byte-equal to the whole model's residual after the cut layer while *not*
equal to its normalized form. `_emit_boundary`.
- **Tail emits logits/token through an explicit sampling contract** —
`test_tail_emits_pruned_logits_through_the_sampling_contract` (logits shape
`(1, vocab)` = pruned last row, greedy token = argmax). `_emit_tail`,
`SamplingContract`.
- **Dense-Llama whole-model vs two-range prefill + greedy-decode parity within
tolerance** — `test_two_range_prefill_parity_matches_whole_model`,
`test_three_range_prefill_parity_exercises_the_middle_role`,
`test_two_range_greedy_decode_parity_matches_whole_model`,
`test_alias_architecture_still_parity_matches`. Documented tolerance:
next-token logits `np.allclose(..., atol=1e-6)` and **identical** greedy token
sequences. (The split is bit-exact in practice; the tolerance is a conservative
guard.)
- **Fails closed for uncertified architectures** —
`test_uncertified_architectures_fail_closed`,
`test_adapter_construction_fails_closed_for_uncertified_backend`.
- **Targeted pytest** — `22 passed`.
- **compileall packages tests** — exit 0.
- **git diff --check** — clean.
- **Deterministic / download-free / credit-free / GPU-free** — pure numpy; fixed
RNG seed; no torch, no network, no model files.
- **Full deterministic pytest** — `20 failed, 715 passed, 13 skipped, 12 errors`.
All 20 failures + 12 errors are pre-existing and unrelated (see below).
- **Native C++ / CTest / llama.cpp patch stack** — **not touched by this story.**
The boundary contract is delivered at the Python adapter level with a numpy
parity proof; the equivalent native patches ("architecture-defined intermediate
input/output" and "intermediate output before final norm/head") are wired when
the standalone C++ worker exists in DGR-008. No native code, CMake, or llama.cpp
patch was modified, so those gates are N/A here (same as DGR-005).
## Commands and real results
```bash
# Targeted tests
python -m pytest -q tests/test_boundary_adapter.py
# -> 22 passed in 0.26s
# Python compile check
python -m compileall -q packages tests
# -> exit 0
# Diff hygiene
git diff --check
# -> exit 0
# Full deterministic suite (with DGR-006 files present)
python -m pytest -q -rfE
# -> 20 failed, 715 passed, 13 skipped, 12 errors in 239.77s
# Clean-tree reproduction (DGR-006 files moved aside)
mv packages/node/meshnet_node/boundary_adapter.py /tmp/ && mv tests/test_boundary_adapter.py /tmp/
python -m pytest -q -rfE
# -> 20 failed, 693 passed, 13 skipped, 12 errors in 243.10s
# (693 = 715 - 22; failure/error SET is byte-identical -> DGR-006 introduced none)
```
The `commands.txt` and `results.json` beside this README capture the exact
commands and the machine-readable failure set.
## Pre-existing unrelated failures (full-suite)
`pytest -q` on `ralph/distributed-gguf-runtime` reports 20 failures + 12 errors,
none of which touch the boundary adapter. Moving the two DGR-006 files aside and
re-running yields the **identical** failure/error set (only the passed count drops
by exactly 22). Categories:
- **12 errors — `tests/test_native_shard_protocol.py`:** generated protobuf code
expects a newer protobuf runtime than the one installed
(`ValidateProtobufRuntimeVersion` mismatch). Pre-existing; documented in the
DGR-002 / DGR-005 evidence.
- **20 failures** across `test_activation_compression.py`,
`test_dynamic_routing.py`, `test_gossip_and_relay.py`,
`test_manual_route_benchmark.py`, `test_node_doctor.py`,
`test_openai_gateway.py` (`langchain` optional dep),
`test_toploc_calibration_dispatch.py`, `test_tracker_capability_admission.py`,
`test_tracker_control_plane.py`, `test_tracker_routing.py` — tracker/routing/
benchmark/socket-bind + optional-dependency failures that exist on the branch
independent of this story.
## Limitations and deferred work
- **Numpy reference, not real weights.** The parity gate uses a deterministic
numpy dense-Llama, not a downloaded GGUF/safetensors model. Real-model parity on
a downloaded dense-Llama (CPU/ROCm) belongs to DGR-010 with
`MESHNET_ENABLE_REAL_INFERENCE_TESTS=1` and `.venv-rocm`.
- **Stateless decode for parity.** Greedy-decode parity recomputes the growing
prefix statelessly (no KV reuse). Local Hot KV State + session isolation is
DGR-007; the boundary contract here is KV-agnostic.
- **Native patch wiring deferred.** The C++/llama.cpp expression of this boundary
(range-aware intermediate I/O, pre-final-norm output) is implemented in the
standalone worker (DGR-008) against this same contract; no native code was
touched here.
- **Greedy-only sampling certified.** `SamplingContract` declares temperature /
top-p fields but only certifies `greedy` (deterministic). Stochastic sampling is
out of scope for the deterministic parity gate.
## Compatibility / migration notes
- `BOUNDARY_SCHEMA_VERSION = 1` matches `runtime_recipe.RuntimeRecipeIdentity`'s
`boundary_schema_version`. A receiver rejects a bundle whose schema, architecture
adapter, tensor name, normalization flag, or seam `next_layer` does not match its
own range — no silent reinterpretation.
- `BoundaryBundle.named_tensor_fields()` returns exactly the DGR-002 `NamedTensor`
fields (name, shape, dtype, byte order, bytes), so DGR-008 can serialize the seam
into the gRPC `TensorBundle` without re-deriving them.
- Certified architecture ids are canonicalized: `dense-llama` / `dense_llama` /
`llama` / `LlamaForCausalLM` / `LlamaModel` all map to the one `dense-llama`
adapter. Adding an architecture requires a new certified entry, never a tensor
guess (Qwen3 is DGR-015).
## Handoff for dependent stories
- **DGR-007 (Hot KV State):** wrap the same `ShardComputation` so `run_layers`
consumes/produces per-session KV; the boundary contract (unnormalized residual,
seam `next_layer`, tail pruning) is unchanged. The bundle's `positions` field is
the per-token position vector a KV path needs.
- **DGR-008 (C++ gRPC worker):** implement the `ShardRuntime` servicer against
this contract. Map `BoundaryBundle.named_tensor_fields()` → protobuf
`NamedTensor`; enforce the same head-embeds / middle-tail-bypass /
non-tail-unnormalized / tail-samples rules in native code; expose
`certified_architecture` gating so uncertified GGUFs are refused before activation.
- **DGR-009 (Meshnet integration):** carry `BoundaryBundle.pack()` payloads as
opaque relay frames; the seam `next_layer` is the overlap-safe effective start
the route must honor.
- **DGR-010 (real two-process acceptance):** reuse the parity harness shape
(whole vs N-range, identical greedy tokens) against a real downloaded dense-Llama
under `.venv-rocm`.
- **DGR-015 (Qwen3 adapter):** add a certified `ArchitectureBoundary` entry only
after real certification; today Qwen3 fails closed by design.

26
.scratch/distributed-gguf-runtime/evidence/DGR-006/commands.txt
View File

@@ -1,26 +0,0 @@
# DGR-006 exact commands (run from repo worktree root)
# Targeted boundary-adapter tests
python -m pytest -q tests/test_boundary_adapter.py
# -> 22 passed in 0.26s
# Python compile check for changed Python
python -m compileall -q packages tests
# -> exit 0
# Diff hygiene
git diff --check
# -> exit 0
# Full deterministic suite with DGR-006 files present
python -m pytest -q -rfE
# -> 20 failed, 715 passed, 13 skipped, 12 errors in 239.77s
# Clean-tree reproduction: move the two new DGR-006 files aside, re-run
mv packages/node/meshnet_node/boundary_adapter.py /tmp/dgr006_boundary_adapter.py
mv tests/test_boundary_adapter.py /tmp/dgr006_test_boundary_adapter.py
python -m pytest -q -rfE
# -> 20 failed, 693 passed, 13 skipped, 12 errors in 243.10s
# (693 = 715 - 22; failure/error set byte-identical to the with-files run)
mv /tmp/dgr006_boundary_adapter.py packages/node/meshnet_node/boundary_adapter.py
mv /tmp/dgr006_test_boundary_adapter.py tests/test_boundary_adapter.py

161
.scratch/distributed-gguf-runtime/evidence/DGR-006/results.json
View File

@@ -1,161 +0,0 @@
{
"story": "DGR-006",
"date": "2026-07-15",
"evidence_kind": "synthetic-unit (pure-numpy dense-Llama parity + boundary contract)",
"targeted_tests": {
"file": "tests/test_boundary_adapter.py",
"result": "22 passed"
},
"compileall": "exit 0",
"git_diff_check": "clean",
"parity_tolerance": {
"logits_atol": 1e-06,
"greedy_tokens": "identical"
},
"full_suite_with_files": {
"failed": 20,
"passed": 715,
"skipped": 13,
"errors": 12,
"seconds": 239.77
},
"full_suite_clean_tree": {
"failed": 20,
"passed": 693,
"skipped": 13,
"errors": 12,
"seconds": 243.1,
"note": "693 = 715 - 22 DGR-006 tests; failure/error set identical"
},
"failure_set_identical_with_and_without_dgr006": true,
"preexisting_unrelated_failures": [
{
"kind": "ERROR",
"nodeid": "tests/test_native_shard_protocol.py::test_capability_and_health_round_trip"
},
{
"kind": "ERROR",
"nodeid": "tests/test_native_shard_protocol.py::test_checksum_algorithms_verify"
},
{
"kind": "ERROR",
"nodeid": "tests/test_native_shard_protocol.py::test_cross_language_roundtrip_python_and_cpp"
},
{
"kind": "ERROR",
"nodeid": "tests/test_native_shard_protocol.py::test_defaults_are_stable_for_backward_compatibility"
},
{
"kind": "ERROR",
"nodeid": "tests/test_native_shard_protocol.py::test_fragment_and_reassemble_round_trip_with_checksums"
},
{
"kind": "ERROR",
"nodeid": "tests/test_native_shard_protocol.py::test_message_header_carries_every_required_field"
},
{
"kind": "ERROR",
"nodeid": "tests/test_native_shard_protocol.py::test_named_tensor_bundle_describes_shape_dtype_byteorder_and_fragments"
},
{
"kind": "ERROR",
"nodeid": "tests/test_native_shard_protocol.py::test_reassemble_detects_fragment_corruption"
},
{
"kind": "ERROR",
"nodeid": "tests/test_native_shard_protocol.py::test_service_descriptor_exposes_all_operations"
},
{
"kind": "ERROR",
"nodeid": "tests/test_native_shard_protocol.py::test_session_response_carries_structured_status_and_results"
},
{
"kind": "ERROR",
"nodeid": "tests/test_native_shard_protocol.py::test_session_stream_carries_open_prefill_decode_release_cancel"
},
{
"kind": "ERROR",
"nodeid": "tests/test_native_shard_protocol.py::test_unknown_fields_are_preserved_for_forward_compatibility"
},
{
"kind": "FAILED",
"nodeid": "tests/test_activation_compression.py::test_compressible_body_uses_zstd_when_it_clears_savings_policy"
},
{
"kind": "FAILED",
"nodeid": "tests/test_activation_compression.py::test_incompressible_body_stays_raw_after_measured_trial"
},
{
"kind": "FAILED",
"nodeid": "tests/test_activation_compression.py::test_malformed_zstd_and_legacy_raw_bodies_are_handled_explicitly"
},
{
"kind": "FAILED",
"nodeid": "tests/test_activation_compression.py::test_threshold_requires_both_byte_and_ratio_savings"
},
{
"kind": "FAILED",
"nodeid": "tests/test_dynamic_routing.py::test_admin_can_replace_a_served_model_and_release_it"
},
{
"kind": "FAILED",
"nodeid": "tests/test_gossip_and_relay.py::test_activation_compression_round_trips_and_skips_small_bodies"
},
{
"kind": "FAILED",
"nodeid": "tests/test_manual_route_benchmark.py::test_benchmark_records_one_and_two_node_routes"
},
{
"kind": "FAILED",
"nodeid": "tests/test_manual_route_benchmark.py::test_clients_without_route_are_unaffected"
},
{
"kind": "FAILED",
"nodeid": "tests/test_manual_route_benchmark.py::test_invalid_route_shape_is_400"
},
{
"kind": "FAILED",
"nodeid": "tests/test_manual_route_benchmark.py::test_pinned_route_uses_named_node"
},
{
"kind": "FAILED",
"nodeid": "tests/test_manual_route_benchmark.py::test_unknown_route_node_is_400"
},
{
"kind": "FAILED",
"nodeid": "tests/test_node_doctor.py::test_cli_doctor_flags_select_what_is_validated"
},
{
"kind": "FAILED",
"nodeid": "tests/test_openai_gateway.py::test_langchain_chat_openai"
},
{
"kind": "FAILED",
"nodeid": "tests/test_toploc_calibration_dispatch.py::test_calibration_run_dispatches_only_solo_capable_nodes"
},
{
"kind": "FAILED",
"nodeid": "tests/test_toploc_calibration_dispatch.py::test_calibration_run_node_without_commitment_endpoint_is_skipped_not_failed"
},
{
"kind": "FAILED",
"nodeid": "tests/test_toploc_calibration_dispatch.py::test_calibration_run_persists_corpus_and_results_endpoint_reports_it"
},
{
"kind": "FAILED",
"nodeid": "tests/test_tracker_capability_admission.py::test_an_enforcing_tracker_never_routes_a_node_whose_proof_does_not_cover_it[invalid]"
},
{
"kind": "FAILED",
"nodeid": "tests/test_tracker_control_plane.py::test_tracker_startup_does_not_import_or_load_model_backends"
},
{
"kind": "FAILED",
"nodeid": "tests/test_tracker_routing.py::test_shard_heal_cycle_surviving_node_covers_dead_peers_gap"
},
{
"kind": "FAILED",
"nodeid": "tests/test_tracker_routing.py::test_torch_node_applies_tracker_load_shard_directive"
}
]
}

229
.scratch/distributed-gguf-runtime/evidence/DGR-007/README.md
View File

@@ -1,229 +0,0 @@
# DGR-007 — Isolated concurrent local Hot KV State: evidence
Status: done
Date: 2026-07-15
Evidence kind: **synthetic-unit** (pure-numpy KV-cached dense-Llama reference +
session/KV manager). No model download, no GPU, no torch, no network, no API
credit.
## Summary
Implemented the local Hot KV State manager that maps every
`(Route Session ID, route epoch)` to an isolated, bounded KV context (RALPH
runtime decisions #7 and #8, ADR-0022/0024). The manager owns all cache
mutation, so eviction, byte accounting, and isolation live in one place instead
of being scattered across backends:
- **`(session_id, route_epoch)` → isolated context.** Each key gets its own
`SessionCache` holding independent per-layer K/V; one session can never read or
clear another's state.
- **KV allocated only for owned layers.** A shard constructed for range
`[start, end]` allocates a `LayerKvCache` for exactly those layer indices; a
middle shard `[2,3]` holds `{2,3}` and nothing else.
- **Full lifecycle:** prefill append, decode append, truncate (rollback),
release, TTL eviction, LRU eviction (by session cap and by byte budget), and an
**explicit** `CacheMiss` (unknown-session / evicted-ttl / evicted-lru /
released / superseded-epoch / seq-len-mismatch) so the head degrades to a
from-token-zero re-prefill instead of corrupting output (decision #14).
- **Fails closed on identity.** Stale route epochs raise `StaleRouteEpochError`; a
request carrying an incompatible KV recipe raises `IncompatibleCacheRecipeError`
(fingerprint mismatch of architecture / kv dtype / head geometry / owned range);
a recipe for an uncertified architecture fails closed at construction (reusing
the DGR-006 certified-architecture gate).
- **KV-aware boundary driver.** `KvBoundaryAdapter` wraps the DGR-006
`ShardComputation` (plus `run_layers_cached`) so a shard runs cached
prefill/decode through the manager while honouring the architecture-defined
boundary contract (head embeds tokens, middle/tail bypass embedding and consume
the unnormalized residual bundle, non-tail emits the unnormalized residual, tail
normalizes + heads + prunes + samples). The computation returns the new
position-encoded K/V; the manager commits it under the budget.
A pure-numpy **KV-cached** dense-Llama reference (RMSNorm + RoPE + SwiGLU with an
absolute-position causal mask over cached keys) proves that cached prefill/decode
reproduces the stateless whole-model greedy tokens bit-for-bit, single-range and
across a head/tail seam. torch/transformers are not installed in the default
`.venv`, so a numpy reference is the only way to keep the parity + isolation gate
deterministic, download-free, and GPU-free — the identical manager contract will
be satisfied by the pinned llama.cpp worker (DGR-008), where the KV context maps
onto a llama sequence.
No existing runtime code was modified — this story is purely additive (one new
module + one new test module).
## Files changed (all new)
- `packages/node/meshnet_node/hot_kv_state.py` — the KV/session manager:
- `KvCacheRecipe` — KV layout identity (certified architecture, kv dtype, head
geometry, owned range) with `fingerprint()` / `is_compatible()` /
`bytes_per_token()`; fails closed on uncertified architectures.
- `LayerKvCache` — per-owned-layer `(seq, n_kv_heads, head_dim)` K/V with
`append` / `truncate` / `nbytes`.
- `SessionCache` — the isolated per-`(session, epoch)` context over owned layers.
- `CacheMiss` / `CacheMissReason` — the explicit, serializable miss response.
- `HotKvStateManager` — `open` / `append` / `truncate` / `release` / `resolve` /
`get`, LRU+TTL+byte-budget eviction, stale-epoch + incompatible-recipe
rejection, epoch supersession, thread-safe (RLock), injectable clock.
- `KvBoundaryAdapter` + `kv_recipe_for()` — KV-aware boundary driver.
- `tests/test_hot_kv_state.py` — pure-numpy KV-cached dense-Llama reference and 22
tests (see below).
## Acceptance criteria → evidence
- **Map `(Route Session ID, route epoch)` to an isolated context** —
`test_prefill_then_decode_append_grows_owned_layers`,
`test_four_interleaved_sessions_have_no_kv_cross_talk`,
`HotKvStateManager.open` keys sessions on `(session_id, route_epoch)`.
- **Allocate KV only for owned layers** —
`test_manager_allocates_kv_only_for_owned_layers` (middle `[2,3]` → `{2,3}`),
`test_multi_range_cached_decode_parity_across_a_seam` (head owns `(0,1,2)`, tail
owns `(3,4,5)`), `test_recipe_bytes_per_token_scales_with_owned_layers`.
- **Prefill append / decode append / truncate / release / TTL-LRU eviction /
explicit cache-miss** — `test_prefill_then_decode_append_grows_owned_layers`,
`test_truncate_rolls_back_all_owned_layers`,
`test_release_one_session_leaves_others_intact_and_returns_memory`,
`test_ttl_eviction_yields_an_explicit_cache_miss`,
`test_lru_eviction_by_session_cap_reports_a_miss`,
`test_budget_eviction_keeps_total_within_budget`,
`test_unknown_session_is_an_explicit_cache_miss`,
`test_seq_len_mismatch_is_an_explicit_cache_miss`.
- **Reject stale epochs and incompatible cache recipes** —
`test_stale_route_epoch_is_rejected`,
`test_new_route_epoch_supersedes_and_frees_old_epoch`,
`test_incompatible_cache_recipe_is_rejected`,
`test_uncertified_architecture_recipe_fails_closed`.
- **≥ four concurrent sessions complete without token or KV cross-talk** —
`test_four_interleaved_sessions_have_no_kv_cross_talk` (four interleaved
round-robin sessions, four *distinct* references, each matches its own),
`test_four_sessions_on_real_threads_stay_isolated` (four OS threads).
- **Cancellation/release leaves others intact and memory returns to budget** —
`test_release_one_session_leaves_others_intact_and_returns_memory` (released
session → `CacheMiss(RELEASED)`, `total_bytes` drops, survivors keep matching
their references), `test_single_session_exceeding_budget_raises`.
- **Cached vs stateless correctness core** —
`test_cached_full_shard_decode_matches_stateless_whole_model`,
`test_cached_prefill_next_token_matches_whole_model_logits`,
`test_multi_range_cached_decode_parity_across_a_seam`. Documented tolerance:
**identical** greedy token ids (bit-exact in practice; cached incremental
attention equals stateless full-sequence recompute per query row).
- **Targeted pytest** — `22 passed`.
- **compileall packages tests** — exit 0.
- **git diff --check** — clean.
- **Deterministic / download-free / credit-free / GPU-free** — pure numpy; fixed
RNG seed; injectable clock (no wall-clock in tests); no torch, no network, no
model files.
- **Full deterministic pytest** — `13 failed, 755 passed, 14 skipped in 254.50s`.
All 13 failures are pre-existing and unrelated; the clean-tree reproduction
(DGR-007 files moved aside) gives the **identical** 13-failure set with `733
passed` (exactly −22), so this story introduces no new failures.
- **Native C++ / CTest / llama.cpp patch stack** — **not touched by this story.**
The KV context contract is delivered at the Python manager level with a numpy
parity + isolation proof; the equivalent native layer-filtered KV / session
mapping is wired when the standalone C++ worker exists in DGR-008. No native
code, CMake, or llama.cpp patch was modified, so those gates are N/A here (same
as DGR-005/006).
## Commands and real results
```bash
VP=/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python
$VP -m pytest -q tests/test_hot_kv_state.py
# -> 22 passed in ~0.3s
$VP -m compileall -q packages tests
# -> exit 0
git diff --check
# -> exit 0
$VP -m pytest -q tests/test_boundary_adapter.py tests/test_gguf_ownership.py
# -> 25 passed
$VP -m pytest -q -rfE
# -> 13 failed, 755 passed, 14 skipped in 254.50s
# Clean-tree reproduction (DGR-007 files moved aside)
mv packages/node/meshnet_node/hot_kv_state.py /tmp/ && mv tests/test_hot_kv_state.py /tmp/
$VP -m pytest -q -rfE
# -> 13 failed, 733 passed, 14 skipped in 252.12s (identical FAILED set; passed -22)
```
`commands.txt` beside this README captures the exact commands.
## Pre-existing unrelated failures (full-suite)
`pytest -q -rfE` on `ralph/distributed-gguf-runtime` reports 13 pre-existing
failures (and, in this run, 0 errors — the earlier DGR-005/006-era
`test_native_shard_protocol.py` protobuf errors no longer appear in this
environment). None touch the KV manager. Moving the two DGR-007 files aside and
re-running yields the **byte-identical** 13-`FAILED` set (only the passed count
drops by exactly 22). The exact set (all tracker/routing/benchmark/toploc/doctor,
i.e. socket-bind / control-plane env, not KV):
```
tests/test_dynamic_routing.py::test_admin_can_replace_a_served_model_and_release_it
tests/test_manual_route_benchmark.py::test_benchmark_records_one_and_two_node_routes
tests/test_manual_route_benchmark.py::test_clients_without_route_are_unaffected
tests/test_manual_route_benchmark.py::test_invalid_route_shape_is_400
tests/test_manual_route_benchmark.py::test_pinned_route_uses_named_node
tests/test_manual_route_benchmark.py::test_unknown_route_node_is_400
tests/test_node_doctor.py::test_cli_doctor_flags_select_what_is_validated
tests/test_toploc_calibration_dispatch.py::test_calibration_run_dispatches_only_solo_capable_nodes
tests/test_toploc_calibration_dispatch.py::test_calibration_run_node_without_commitment_endpoint_is_skipped_not_failed
tests/test_toploc_calibration_dispatch.py::test_calibration_run_persists_corpus_and_results_endpoint_reports_it
tests/test_tracker_capability_admission.py::test_an_enforcing_tracker_never_routes_a_node_whose_proof_does_not_cover_it[invalid]
tests/test_tracker_routing.py::test_shard_heal_cycle_surviving_node_covers_dead_peers_gap
tests/test_tracker_routing.py::test_torch_node_applies_tracker_load_shard_directive
```
## Limitations and deferred work
- **Numpy reference, not real weights.** The parity + isolation gate uses a
deterministic numpy KV-cached dense-Llama, not a downloaded GGUF/safetensors
model. Real-model concurrent KV isolation on a downloaded dense-Llama (CPU/ROCm)
belongs to DGR-010/DGR-012 with `MESHNET_ENABLE_REAL_INFERENCE_TESTS=1` and
`.venv-rocm`.
- **Manager-owned storage, native mapping deferred.** The KV bytes are numpy
arrays managed in-process. The llama.cpp expression (a filtered llama sequence
per `(session, epoch)` over owned layers) is implemented in the standalone
worker (DGR-008) against this same manager contract; no native code was touched.
- **Continuous batching is DGR-012.** This story delivers *isolation* and bounded
lifecycle for concurrent sessions; continuous batching of compatible active
sessions inside a node (decision #9) is DGR-012 and builds on this manager.
- **Greedy-only sampling.** Reuses the DGR-006 `SamplingContract` (greedy
certified). Stochastic sampling is out of scope for the deterministic gate.
- **Coexists with legacy `SessionCacheStore`.** The older AH-25
`model_backend.SessionCacheStore` (session-id-only, opaque transformers cache,
HTTP path) is untouched. `HotKvStateManager` is the native-runtime-aligned
successor: it adds route-epoch keying, owned-layer allocation, recipe-fingerprint
rejection, and a byte budget. DGR-008/009 wire the native worker to
`HotKvStateManager`, not `SessionCacheStore`.
## Compatibility / migration notes
- `KvCacheRecipe.fingerprint()` canonicalizes the architecture (via
`certified_architecture`), so `llama` / `LlamaForCausalLM` map to the same
recipe; it aligns field-for-field with the DGR-003 `RuntimeRecipeIdentity`
compatibility discipline and reuses `runtime_recipe.compatibility_fingerprint`.
- `CacheMiss` is a value (not an exception) so it can be serialized into the
DGR-002 native protocol's cache expectation/result field; `resolve()` returns it,
`get()` raises `KvCacheMissError` wrapping it.
- The manager takes an injectable `clock` for deterministic TTL tests; production
defaults to `time.monotonic`.
## Handoff for dependent stories
- **DGR-008 (C++ gRPC worker):** implement the servicer's KV path against
`HotKvStateManager`. Map each `(Route Session ID, route epoch)` to a filtered
llama sequence over owned layers; on decode, read the sequence's cached K/V,
compute the new position-encoded K/V, and commit via `append` (honour the byte
budget and return an explicit `CacheMiss` on eviction). Enforce
`KvCacheRecipe.is_compatible` before activation and reject stale epochs.
- **DGR-009 (Meshnet integration):** the route epoch the tracker assigns is the
`route_epoch` key; carry the `CacheMiss` reason back to the head so it re-prefills
from token zero on eviction/restart.
- **DGR-012 (continuous batching):** batch compatible active sessions whose
`KvCacheRecipe` fingerprints match; each session keeps its own `SessionCache`, so
batching is a scheduling concern layered over this isolation, not a change to it.
- **DGR-013 (failure/cancel matrix):** `release` + the budget-return assertion here
is the unit-level basis for the resource-cleanup matrix.

31
.scratch/distributed-gguf-runtime/evidence/DGR-007/commands.txt
View File

@@ -1,31 +0,0 @@
# DGR-007 — exact commands (run from the worktree root).
# Python: /run/media/popov/d/DEV/repos/d-popov.com/AI/.venv (Python 3.14.6, numpy 2.4.4).
# Root conftest.py adds packages/* to sys.path, so `meshnet_node` imports work.
VP=/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python
# Targeted tests for this story.
$VP -m pytest -q tests/test_hot_kv_state.py
# -> 22 passed
# Python compile check for the changed packages/tests.
$VP -m compileall -q packages tests
# -> exit 0
# Diff hygiene.
git diff --check
# -> exit 0
# Dependency (DGR-006) + range-ownership (DGR-005) tests still green.
$VP -m pytest -q tests/test_boundary_adapter.py tests/test_gguf_ownership.py
# -> 25 passed
# Full deterministic suite (with DGR-007 files present).
$VP -m pytest -q -rfE
# -> see README (pre-existing unrelated failure set, +22 passed vs baseline)
# Clean-tree reproduction (DGR-007 files moved aside).
mv packages/node/meshnet_node/hot_kv_state.py /tmp/ && mv tests/test_hot_kv_state.py /tmp/
$VP -m pytest -q -rfE
# -> identical failure/error set, passed count drops by exactly 22
mv /tmp/hot_kv_state.py packages/node/meshnet_node/ && mv /tmp/test_hot_kv_state.py tests/

47
.scratch/distributed-gguf-runtime/evidence/DGR-007/results.json
View File

@@ -1,47 +0,0 @@
{
"task_id": "DGR-007",
"title": "Add isolated concurrent local Hot KV State",
"status": "done",
"date": "2026-07-15",
"evidence_kind": "synthetic-unit",
"python": "/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv (Python 3.14.6, numpy 2.4.4)",
"files_changed": [
"packages/node/meshnet_node/hot_kv_state.py",
"tests/test_hot_kv_state.py"
],
"gates": {
"targeted_pytest": {"command": "pytest -q tests/test_hot_kv_state.py", "result": "22 passed"},
"compileall": {"command": "python -m compileall -q packages tests", "exit": 0},
"git_diff_check": {"command": "git diff --check", "exit": 0},
"dependency_tests": {"command": "pytest -q tests/test_boundary_adapter.py tests/test_gguf_ownership.py", "result": "25 passed"},
"full_suite_with_files": {"command": "pytest -q -rfE", "result": "13 failed, 755 passed, 14 skipped", "seconds": 254.50},
"full_suite_clean_tree": {"command": "pytest -q -rfE (DGR-007 files moved aside)", "result": "13 failed, 733 passed, 14 skipped", "seconds": 252.12}
},
"no_new_failures": true,
"failure_set_identical": true,
"passed_delta": 22,
"preexisting_failures": [
"tests/test_dynamic_routing.py::test_admin_can_replace_a_served_model_and_release_it",
"tests/test_manual_route_benchmark.py::test_benchmark_records_one_and_two_node_routes",
"tests/test_manual_route_benchmark.py::test_clients_without_route_are_unaffected",
"tests/test_manual_route_benchmark.py::test_invalid_route_shape_is_400",
"tests/test_manual_route_benchmark.py::test_pinned_route_uses_named_node",
"tests/test_manual_route_benchmark.py::test_unknown_route_node_is_400",
"tests/test_node_doctor.py::test_cli_doctor_flags_select_what_is_validated",
"tests/test_toploc_calibration_dispatch.py::test_calibration_run_dispatches_only_solo_capable_nodes",
"tests/test_toploc_calibration_dispatch.py::test_calibration_run_node_without_commitment_endpoint_is_skipped_not_failed",
"tests/test_toploc_calibration_dispatch.py::test_calibration_run_persists_corpus_and_results_endpoint_reports_it",
"tests/test_tracker_capability_admission.py::test_an_enforcing_tracker_never_routes_a_node_whose_proof_does_not_cover_it[invalid]",
"tests/test_tracker_routing.py::test_shard_heal_cycle_surviving_node_covers_dead_peers_gap",
"tests/test_tracker_routing.py::test_torch_node_applies_tracker_load_shard_directive"
],
"native_gates_touched": false,
"acceptance": {
"session_epoch_isolated_context": true,
"kv_only_owned_layers": true,
"prefill_decode_truncate_release_ttl_lru_cachemiss": true,
"reject_stale_epoch_and_incompatible_recipe": true,
"four_concurrent_sessions_no_crosstalk": true,
"release_leaves_others_and_returns_memory": true
}
}

83
.scratch/distributed-gguf-runtime/evidence/DGR-009/README.md
View File

@@ -1,83 +0,0 @@
# DGR-009 — Integrate the native worker with Meshnet: evidence
Status: done
Date: 2026-07-15
Evidence kind: **python-unit + repo-hygiene**. No model download, no GPU, no API
credit.
## Summary
Implemented the Meshnet-facing GGUF backend seam and recipe gating needed for
the native worker path:
- Added `GgufNodeBackend`, a backend-shaped adapter that lets the existing node
HTTP/control-plane code serve GGUF-backed shards without changing the
Transformers/Torch path for the default recipes.
- Added `llama-cpp-native` to the recipe manifest and gated startup so only
recipes with `backend_id == "llama.cpp"` build the GGUF backend.
- Preserved the existing registration/admission flow by carrying the validated
capability report and proof shard through registration.
- Added unit coverage for the GGUF backend seam and for recipe-gated startup.
- Fixed the explicit-shard startup path so the legacy Torch tests that use an
opaque stub model still pass without requiring HuggingFace config discovery.
## Files changed
- `packages/node/meshnet_node/gguf_backend.py` - new GGUF backend adapter and
worker-transport boundary.
- `packages/node/meshnet_node/startup.py` - recipe-gated GGUF backend injection
and explicit-shard startup fix.
- `packages/node/meshnet_node/recipes.json` - added `llama-cpp-native`.
- `tests/test_gguf_backend.py` - backend delegation and recipe-selection tests.
- `.ralph-tui/progress.md` - appended DGR-009 progress note.
- `.scratch/distributed-gguf-runtime/issues/09-integrate-the-native-worker-with-meshnet.md`
- marked `Status: done`.
## Commands and real results
```bash
python -m pytest -q tests/test_gguf_backend.py
# -> 2 passed in 0.05s
python -m pytest -q tests/test_node_admission.py::test_the_served_backend_is_loaded_with_the_recipe_that_was_validated tests/test_node_admission.py::test_backend_validation_failure_registers_nothing
# -> 2 passed in 0.07s
python -m compileall -q packages tests
# -> exit 0
git diff --check
# -> exit 0
python -m pytest -q
# -> 222 failed, 463 passed, 13 skipped, 86 errors in 135.65s
```
## Limitations
- `python -m pytest -q` is still not clean in this sandbox. The dominant
failures are tracker/control-plane socket `PermissionError: [Errno 1]
Operation not permitted` and a native protocol import failure caused by a
protobuf runtime mismatch (`gencode 7.35.0` vs runtime `6.33.6`).
- `tests/test_native_shard_protocol.py` currently fails for the same protobuf
runtime mismatch in this environment.
- `DGR-008` evidence was not present in the tree, so the dependency behavior was
verified by reading the live code and exercising the Python seam instead of
relying on a missing README.
## Compatibility notes
- The default Torch path remains intact; GGUF backend selection is explicit and
recipe-gated.
- `TorchNodeServer` already accepts an injected backend object, so the control
plane stays Meshnet-owned.
- The GGUF adapter currently establishes the seam for the native worker
transport; the compiled worker remains the owner of the gRPC protocol details.
## Dependent-story handoff
- DGR-008 should continue to own the native worker implementation and the
versioned gRPC frame handling behind `MESHNET_NATIVE_WORKER_URL`.
- DGR-010 / DGR-012 can build on this seam without changing the control plane:
the recipe-gated backend and validated capability report are already carried
through startup.

58
.scratch/distributed-gguf-runtime/evidence/DGR-010/BLOCKED.md
View File

@@ -1,58 +0,0 @@
# DGR-010 — Blocked handoff
Status: blocked
Date: 2026-07-15
## Blocker
I verified the local workspace and mounted-drive model storage, but there is no
certified dense-Llama artifact available on this machine to run the required
real-model two-process acceptance.
What I found:
- `/run/media/popov/d/DEV/models` contains Qwen artifacts and caches, but no
dense-Llama model snapshot or GGUF artifact.
- `/run/media/popov/d/DEV/llamacpp/llama.cpp/models` contains only vocab GGUFs,
not a certified dense-Llama model.
- The existing code paths for real startup, GGUF backend selection, Hot KV
isolation, and benchmark reporting are present and readable, but the actual
DGR-010 acceptance run needs a certified dense-Llama artifact from mounted
storage to satisfy the story contract.
## Verified current state
- DGR-009 evidence was read and verified as the dependency handoff.
- `packages/node/meshnet_node/startup.py` already gates backend selection by
recipe and can load either the Torch path or the explicit GGUF seam.
- `packages/node/meshnet_node/hot_kv_state.py`, `boundary_adapter.py`, and
`gguf_ownership.py` already provide the isolation/parity seams that DGR-010
would exercise.
- The repo has no existing `evidence/DGR-010/README.md` yet, which is expected
because the story has not been completed.
## Commands run
```bash
sed -n '1,260p' .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md
sed -n '1,260p' .scratch/distributed-gguf-runtime/issues/10-pass-local-real-model-two-process-acceptance.md
sed -n '1,260p' .scratch/distributed-gguf-runtime/evidence/DGR-009/README.md
git status --short
find /run/media/popov/d/DEV -type f \( -name '*.gguf' -o -name '*.safetensors' -o -name 'config.json' \) | rg -i 'llama|tinyllama|meta-llama|hf-internal-testing|qwen'
```
## Next step to unblock
Provide or mount a certified dense-Llama artifact on the configured mounted
drive storage, then rerun the DGR-010 acceptance path with
`MESHNET_ENABLE_REAL_INFERENCE_TESTS=1`.
## Continuation note
Once the artifact exists, the next iteration should:
1. Run the two local worker processes against the certified dense-Llama shard
ranges.
2. Capture parity, concurrency, memory, and failure metrics.
3. Write `evidence/DGR-010/README.md` with the real results and then update the
issue status.

70
.scratch/distributed-gguf-runtime/evidence/DGR-011/BLOCKED.md
View File

@@ -1,70 +0,0 @@
# DGR-011 — Blocked handoff
Status: blocked
Date: 2026-07-15
## Blocker
This story cannot be completed in the current workspace state because its
mandatory dependency, DGR-010, is still not passed.
Verified blockers:
- `.scratch/distributed-gguf-runtime/prd.json` still marks `DGR-010` and
`DGR-011` with `"passes": false`.
- `.scratch/distributed-gguf-runtime/evidence/DGR-010/README.md` does not
exist, and the only DGR-010 evidence artifact present is
`.scratch/distributed-gguf-runtime/evidence/DGR-010/BLOCKED.md`.
- Mounted storage search found Qwen model artifacts and llama.cpp vocab files,
but no certified dense-Llama GGUF artifact suitable for the required real
acceptance run.
## Verified current state
- The repo already contains the Meshnet-facing GGUF backend seam and the
recipe-gated startup path from DGR-009.
- The architecture and Ralph context require real-model execution for this
story, not synthetic workers or unit-only coverage.
- The current environment does not expose the dense-Llama artifact required to
run the prerequisite local real-model acceptance, so the two-machine route
cannot be proven end to end.
## Commands run
```bash
sed -n '1,260p' .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md
sed -n '1,260p' .scratch/distributed-gguf-runtime/issues/11-pass-a-real-heterogeneous-two-machine-route.md
sed -n '1,260p' .ralph-tui/progress.md
sed -n '1,240p' .scratch/distributed-gguf-runtime/evidence/DGR-010/BLOCKED.md
sed -n '1,220p' CONTEXT.md
sed -n '1,260p' docs/adr/0024-distributed-gguf-runtime.md
sed -n '282,350p' .scratch/distributed-gguf-runtime/prd.json
find /run/media/popov/d/DEV/models -maxdepth 3 \( -name '*.gguf' -o -name 'config.json' -o -name '*.safetensors' \)
find /run/media/popov/d/DEV/llamacpp/llama.cpp/models /run/media/popov/d/DEV/models -maxdepth 4 \( -iname '*llama*' -o -iname '*dense*' -o -iname '*qwen*' -o -name 'config.json' -o -name '*.gguf' \)
```
## Known limitations
- No certified dense-Llama artifact is available on mounted storage in this
workspace.
- No real two-machine execution was possible, so there are no real route,
hardware, backend, or drift metrics to record for this story.
- The story remains blocked until DGR-010 is completed with a real-model
evidence README and a confirmed dense-Llama artifact on mounted storage.
## Compatibility notes
- DGR-009's recipe-gated GGUF backend seam is present and can be reused.
- The acceptance path for this story still requires the upstream real-model
evidence from DGR-010 before any heterogeneous two-machine route can be
claimed.
## Dependent-story handoff
- Finish DGR-010 first, including its real-model evidence README and
acceptance run.
- Once DGR-010 passes, rerun the two-machine acceptance against the same
certified dense-Llama artifact, then record the two-host hardware/network
manifest, route, commands, and raw metrics in `evidence/DGR-011/README.md`.
- Do not update the issue to `Status: done` until the real two-machine route
has been executed and recorded.

220
.scratch/distributed-gguf-runtime/evidence/DGR-012/README.md
View File

@@ -1,220 +0,0 @@
# DGR-012 — Continuous batching and bounded admission: evidence
Status: done
Date: 2026-07-16
Evidence kind: **synthetic-unit** (pure-numpy KV-cached dense-Llama reference +
node-local continuous-batching scheduler). No model download, no GPU, no torch,
no network, no API credit.
## Summary
Implemented the node-local scheduler that turns concurrent Route Sessions into
llama.cpp-style continuous batches while bounding admission (RALPH runtime
decision #9, ADR-0024). It sits **on top of** the DGR-007 Hot KV State manager —
batching is a scheduling concern layered over the existing per-`(session, epoch)`
KV isolation, not a new control plane or a change to the KV contract.
- **Bounded admission (`NodeBudget` + `submit`).** A new session is admitted only
if it fits four budgets: resident **weight** footprint (reported), **KV** byte
budget (a session must be able to hold its *whole* generation, prompt + new
tokens, on its own), **scratch** (per-active-session activation buffers, capped
by a total scratch envelope), and the bounded **queue**. Anything that cannot
fit is rejected up front with an explicit `AdmissionReason`
(`REJECTED_KV_BUDGET` / `REJECTED_SCRATCH_BUDGET` / `REJECTED_DUPLICATE`);
anything that fits but has no free slot waits in the bounded queue; a **full
queue is refused** (`REJECTED_QUEUE_FULL`) — that refusal is the backpressure
signal.
- **Continuous batching (`ContinuousBatchScheduler` + `KvBatchEngine`).** Every
tick, all currently-decoding sessions contribute their single next token to one
batch (bounded by `max_batch_size`); the engine runs the batch once. Each
session keeps its own position and appends its own sampled token via its own
`SessionCache`, so batching never mixes outputs. `KvBatchEngine` adapts the
DGR-007 `KvBoundaryAdapter`, so the batch runs against the *real* KV isolation
path; the pinned llama.cpp worker (DGR-008) implements the same
`recipe_fingerprint`/`prefill`/`decode_batch`/`release` contract where a batch
becomes one `llama_decode` over several sequences.
- **Prefill does not starve decode.** The scheduling policy is explicit and fixed:
**decode first, then bounded prefill.** In-flight decodes always run before any
new prompt is prefilled, and prefill work per tick is capped
(`max_prefill_tokens_per_tick`, always allowing at least one so a single large
prompt still progresses). A burst of new sessions cannot stall generations
already in flight.
- **Bounded memory / backpressure.** KV growth is bounded by the manager byte
budget; queued activations are bounded by `max_queue_depth` and the scratch
envelope; completed sessions release their KV so total KV returns to zero.
- **Capability telemetry (`SchedulerTelemetry`).** Reports active sessions, queue
depth, batch occupancy (last/avg/max), KV pressure (bytes/budget), scratch
pressure, prefill/decode token totals **and rates**, and rejected admissions
(total + by reason). All JSON-safe.
- **Concurrency 1/2/4/8 sweep (`run_concurrency_sweep`).** Runs the same eight
jobs at each level against a fresh KV manager and proves (a) **no cross-session
corruption** — every level yields byte-identical per-session tokens as the
serialized concurrency-1 reference — and (b) **saturation** — average batch
occupancy rises and total ticks fall as concurrency increases, until occupancy
plateaus.
No existing runtime code was modified — this story is purely additive (one new
module + one new test module + evidence).
## Files changed (all new)
- `packages/node/meshnet_node/batch_scheduler.py` — the scheduler:
- `NodeBudget` — weight/KV/scratch/queue budgets + `max_batch_size` /
`max_prefill_tokens_per_tick` scheduling bounds, with derived
`effective_active_cap` (tighter of active-slot and scratch caps).
- `AdmissionReason` / `AdmissionDecision` — structured admit/queue/reject.
- `GenerationRequest` / `DecodeItem` / `StepResult` — job + engine I/O values.
- `KvBatchEngine` — adapts a full-shard `KvBoundaryAdapter` to the batch-engine
contract (rejects a partial head/tail-only range).
- `SchedulerTelemetry` — the bounded capability snapshot.
- `ContinuousBatchScheduler` — thread-safe `submit` / `run_tick` /
`run_to_completion` / `telemetry`, decode-first-then-bounded-prefill policy.
- `run_concurrency_sweep` / `ConcurrencyResult` / `ConcurrencySweep` — the
deterministic 1/2/4/8 saturation report + corruption check.
- `tests/test_batch_scheduler.py` — 16 tests (see below); reuses the DGR-007
numpy dense-Llama reference via `from test_hot_kv_state import _KvDenseLlama,
_KvReferenceShard`.
- `.scratch/distributed-gguf-runtime/evidence/DGR-012/` — this README,
`commands.txt`, `generate_evidence.py`, `results.json`.
## Acceptance criteria → evidence
- **Scheduler admits sessions against weight, KV, scratch, and queue budgets** —
`test_admission_respects_active_scratch_and_queue_budgets` (fill slots → queue →
reject full queue), `test_admission_rejects_a_session_that_cannot_fit_the_kv_budget`,
`test_admission_rejects_when_per_session_scratch_exceeds_budget`,
`test_duplicate_submission_is_rejected`,
`test_weight_budget_is_reported_in_telemetry`.
- **Compatible decode steps form batches preserving per-session positions/outputs**
— `test_batched_decode_preserves_per_session_positions_and_outputs`
(`batch_occupancy_max == 4`, four divergent references each reproduced),
`test_positions_are_isolated_across_different_prompt_lengths` (prompt lengths 1/3/7).
- **Prefill does not starve decode; policy and bounds explicit** —
`test_prefill_does_not_starve_in_flight_decode` (in-flight session decodes on
*every* tick during a 4-session prefill burst; ≤1 prefill/tick),
`test_decode_first_policy_is_explicit_in_a_single_tick`.
- **Backpressure prevents unbounded queued activations or KV growth** —
`test_backpressure_signals_when_queue_full_then_recovers`,
`test_completed_sessions_release_kv_so_growth_is_bounded` (`kv_total_bytes == 0`
after completion).
- **Capability telemetry reports all required signals** —
`test_telemetry_reports_every_required_signal` (asserts every key present;
deterministic rates under an injected clock).
- **Concurrency 1/2/4/8 identifies saturation, no cross-session corruption** —
`test_concurrency_sweep_identifies_saturation_without_corruption`
(occupancy strictly ↑, ticks strictly ↓, tokens/tick ↑, `corruption_free`,
0 cache misses, saturation=8), `test_concurrency_sweep_saturates_below_max_when_load_is_small`.
- **Engine/usage guards** — `test_kv_batch_engine_requires_a_full_shard`,
`test_run_to_completion_is_bounded_against_misconfiguration`.
## Concurrency 1/2/4/8 sweep (real, deterministic — `results.json`)
Eight sessions, prompt length 4, 8 new tokens each; fresh KV manager per level;
budgets sized so KV never evicts (so the corruption check is unambiguous).
| concurrency | ticks | avg batch occupancy | max occupancy | tokens/tick | peak KV bytes |
|---|---|---|---|---|---|
| 1 | 64 | 1.000 | 1 | 1.375 | 15360 |
| 2 | 33 | 1.750 | 2 | 2.667 | 29184 |
| 4 | 19 | 3.111 | 4 | 4.632 | 52224 |
| 8 | 15 | 4.000 | 7 | 5.867 | 75264 |
`saturation_concurrency = 8`, `corruption_free = True`, `cache_misses = 0`,
`rejected_admissions = 0`. As concurrency rises, the scheduler packs more sessions
per decode step (occupancy ↑) and finishes the same 56 decode + 32 prefill tokens
in far fewer ticks (aggregate work/tick ↑) — the batching throughput property —
while every per-session token stream stays byte-identical to the serialized
reference (no cross-session corruption). Max occupancy is 7 (not 8) at level 8
because the fairness policy prefills at most one new session per tick, so the last
session begins decoding one tick later.
## Commands and real results
```bash
VP=/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python
$VP -m pytest -q tests/test_batch_scheduler.py
# -> 16 passed
$VP -m pytest -q tests/test_hot_kv_state.py # dependency still green
# -> 22 passed
$VP -m compileall -q packages tests
# -> exit 0
git diff --check
# -> exit 0
$VP .scratch/distributed-gguf-runtime/evidence/DGR-012/generate_evidence.py
# -> wrote results.json; saturation_concurrency=8 corruption_free=True
$VP -m pytest -q -rfE -p no:cacheprovider
# -> FULL_SUITE_RESULT_PLACEHOLDER
```
`commands.txt` beside this README captures the exact commands.
## Full-suite baseline (pre-existing unrelated failures)
FULL_SUITE_BASELINE_PLACEHOLDER
## Limitations and deferred work
- **Synthetic-unit, not real weights.** The scheduler is exercised against the
deterministic numpy KV-cached dense-Llama reference (the same one DGR-007 uses),
not a downloaded GGUF. This is required to keep the default gate deterministic,
download-free, and GPU-free. Real concurrent throughput on a downloaded
dense-Llama (CPU/ROCm) belongs to DGR-010 (blocked — no certified dense-Llama
artifact on this machine; see `evidence/DGR-010/BLOCKED.md`) and the final
comparison in DGR-014.
- **Batching is a scheduling grouping in this reference.** `KvBatchEngine.decode_batch`
runs each batch member sequentially through the cached decode (each attends only
its own KV, exactly like an independent llama.cpp sequence). The pinned llama.cpp
worker (DGR-008) fuses the batch into one `llama_decode` graph; the scheduling
semantics — one batch per tick, isolated positions/outputs — are identical. The
numbers here are *scheduler* quantities (ticks, batch occupancy, tokens/tick)
that are real and deterministic; **actual kernel-level batching speedup is a
native-worker property and is NOT claimed here** (RALPH performance discipline:
no unmeasured speed claims). It is measured in DGR-008/DGR-010/DGR-014.
- **Greedy sampling only.** Reuses the DGR-006 greedy `SamplingContract`. Greedy
over isolated per-session KV is order-independent, which is exactly why the
corruption check can assert byte-identical outputs across concurrency levels.
Stochastic sampling is out of scope for the deterministic gate.
- **Single loaded shard / single recipe per scheduler.** The scheduler batches
compatible sessions of one loaded shard (one `recipe_fingerprint`), which is the
node-local case. Multi-range routes batch at the head node whose adapter owns the
final head; cross-node coordination stays in the Meshnet control plane.
- **Native / llama.cpp gates N/A.** No native code, CMake, or llama.cpp patch was
touched (same as DGR-005/006/007), so those gates do not apply to this story.
## Compatibility / migration notes
- Purely additive: no existing module changed, so no behavior of the Torch/GGUF
backends, tracker, or KV manager is altered. The scheduler is opt-in — a server
constructs it around a `KvBatchEngine` when it wants continuous batching.
- `SchedulerTelemetry.to_dict()` is JSON-safe and aligns with the capability-signal
vocabulary (active sessions, queue depth, batch occupancy, KV pressure,
prefill/decode rates, rejected admissions) that a node advertises upward; it can
be folded into the DGR-009 capability report / heartbeat without schema changes
here.
- `AdmissionReason` values are stable strings suitable for the native protocol's
structured status / backpressure signalling.
## Handoff for dependent stories
- **DGR-008 (C++ gRPC worker):** implement the `BatchEngine` contract natively —
`decode_batch` becomes one `llama_decode` over the sessions' filtered sequences;
`prefill`/`release` map to the same KV manager operations. The scheduler,
admission budgets, fairness policy, and telemetry are unchanged; only the engine
swaps from numpy to llama.cpp.
- **DGR-010 (local real two-process acceptance, blocked):** once a certified
dense-Llama artifact is mounted, drive `run_concurrency_sweep` (or the scheduler
directly) with a real `KvBatchEngine` over the GGUF backend to produce
real-hardware occupancy/throughput/KV-pressure numbers under
`MESHNET_ENABLE_REAL_INFERENCE_TESTS=1` / `.venv-rocm`.
- **DGR-013 (failure/cancel/restart):** the `DoneReason.CACHE_MISS` path (a decode
whose KV was evicted marks the session done and re-prefillable) and the KV-release
on completion are the unit basis for the cancellation/cleanup matrix.
- **DGR-014 (release gate):** feed the real-hardware sweep’s aggregate throughput
and saturation point into the immutable DGR-001 comparison; do not reuse these
synthetic numbers as a performance claim.

24
.scratch/distributed-gguf-runtime/evidence/DGR-012/commands.txt
View File

@@ -1,24 +0,0 @@
# DGR-012 — exact commands (run from the worktree root)
# Default venv (Python 3.14); deterministic, download-free, GPU-free, API-credit-free.
VP=/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python
# Targeted story tests
$VP -m pytest -q tests/test_batch_scheduler.py
# -> 16 passed
# Dependency (DGR-007) still green — scheduler builds on this KV manager
$VP -m pytest -q tests/test_hot_kv_state.py
# -> 22 passed
# Python quality gates
$VP -m compileall -q packages tests
# -> exit 0
git diff --check
# -> exit 0
# Regenerate the machine-readable concurrency-sweep evidence
$VP .scratch/distributed-gguf-runtime/evidence/DGR-012/generate_evidence.py
# -> writes results.json; saturation_concurrency=8 corruption_free=True
# Full deterministic suite (records the pre-existing unrelated failure baseline)
$VP -m pytest -q -rfE -p no:cacheprovider

117
.scratch/distributed-gguf-runtime/evidence/DGR-012/generate_evidence.py
View File

@@ -1,117 +0,0 @@
"""Regenerate the DGR-012 concurrency-sweep evidence artifact.
Deterministic, download-free, GPU-free. Run from the repo root with the default
venv so the worktree ``meshnet_node`` package and the DGR-007 numpy reference
(``tests/test_hot_kv_state``) are importable:
python .scratch/distributed-gguf-runtime/evidence/DGR-012/generate_evidence.py
Writes ``results.json`` beside this script.
"""
from __future__ import annotations
import json
import pathlib
import sys
_ROOT = pathlib.Path(__file__).resolve().parents[4]
sys.path.insert(0, str(_ROOT / "packages" / "node"))
sys.path.insert(0, str(_ROOT / "tests"))
from test_hot_kv_state import _KvDenseLlama, _KvReferenceShard # noqa: E402
from meshnet_node.batch_scheduler import ( # noqa: E402
ContinuousBatchScheduler,
GenerationRequest,
KvBatchEngine,
NodeBudget,
run_concurrency_sweep,
)
from meshnet_node.hot_kv_state import ( # noqa: E402
HotKvStateManager,
KvBoundaryAdapter,
kv_recipe_for,
)
MODEL = _KvDenseLlama()
def make_engine() -> KvBatchEngine:
shard = _KvReferenceShard(MODEL, 0, MODEL.n_layers - 1)
manager = HotKvStateManager(kv_recipe_for(shard))
return KvBatchEngine(KvBoundaryAdapter(shard, manager))
def main() -> int:
prompts = {
"s0": [1, 2, 3, 4], "s1": [5, 6, 7, 8], "s2": [9, 10, 11, 12],
"s3": [13, 14, 15, 16], "s4": [17, 18, 19, 20], "s5": [21, 22, 23, 24],
"s6": [25, 26, 27, 28], "s7": [29, 30, 31, 32],
}
n_new = 8
requests = [
GenerationRequest(sid, 0, tuple(p), n_new) for sid, p in prompts.items()
]
sweep = run_concurrency_sweep(
make_engine, requests, concurrency_levels=(1, 2, 4, 8)
)
# A representative telemetry snapshot mid-run at concurrency 4 (shows the live
# capability signals a node advertises upward).
engine = make_engine()
scheduler = ContinuousBatchScheduler(
engine,
NodeBudget(
max_active_sessions=4, max_batch_size=4, max_queue_depth=8,
scratch_bytes_per_session=1, scratch_budget_bytes=4,
),
)
for request in requests:
scheduler.submit(request)
for _ in range(6):
scheduler.run_tick()
mid_run_telemetry = scheduler.telemetry().to_dict()
artifact = {
"schema_version": 1,
"evidence_kind": "synthetic-unit",
"model": {
"reference": "pure-numpy KV-cached dense-Llama (tests/test_hot_kv_state)",
"n_layers": MODEL.n_layers,
"hidden": MODEL.hidden,
"n_heads": MODEL.n_heads,
"vocab": MODEL.vocab,
},
"workload": {
"sessions": len(prompts),
"prompt_len": 4,
"max_new_tokens": n_new,
},
"concurrency_sweep": sweep.to_dict(),
"mid_run_telemetry_concurrency_4": mid_run_telemetry,
}
out = pathlib.Path(__file__).with_name("results.json")
out.write_text(json.dumps(artifact, indent=2, sort_keys=True) + "\n", encoding="utf-8")
print(f"wrote {out}")
print(
"saturation_concurrency=%d corruption_free=%s"
% (sweep.saturation_concurrency, sweep.corruption_free)
)
for result in sweep.results:
print(
" c=%d ticks=%d avg_occ=%.3f tokens/tick=%.3f peak_kv=%dB"
% (
result.concurrency,
result.ticks,
result.avg_batch_occupancy,
result.tokens_per_tick,
result.peak_kv_bytes,
)
)
return 0
if __name__ == "__main__":
raise SystemExit(main())

179
.scratch/distributed-gguf-runtime/evidence/DGR-012/results.json
View File

@@ -1,179 +0,0 @@
{
"concurrency_sweep": {
"corruption_free": true,
"reference_outputs": {
"s0": [
27,
8,
27,
8,
27,
8,
1,
1
],
"s1": [
26,
39,
39,
39,
39,
3,
39,
39
],
"s2": [
12,
12,
12,
12,
12,
12,
30,
12
],
"s3": [
29,
41,
42,
47,
47,
42,
47,
42
],
"s4": [
23,
11,
44,
29,
29,
29,
41,
29
],
"s5": [
35,
11,
0,
1,
11,
0,
11,
15
],
"s6": [
39,
39,
28,
39,
39,
39,
28,
28
],
"s7": [
39,
39,
39,
39,
39,
39,
8,
47
]
},
"results": [
{
"avg_batch_occupancy": 1.0,
"cache_misses": 0,
"concurrency": 1,
"decode_batches": 56,
"decode_tokens": 56,
"max_batch_occupancy": 1,
"peak_kv_bytes": 15360,
"prefill_tokens": 32,
"rejected_admissions": 0,
"ticks": 64,
"tokens_per_tick": 1.375
},
{
"avg_batch_occupancy": 1.75,
"cache_misses": 0,
"concurrency": 2,
"decode_batches": 32,
"decode_tokens": 56,
"max_batch_occupancy": 2,
"peak_kv_bytes": 29184,
"prefill_tokens": 32,
"rejected_admissions": 0,
"ticks": 33,
"tokens_per_tick": 2.6667
},
{
"avg_batch_occupancy": 3.1111,
"cache_misses": 0,
"concurrency": 4,
"decode_batches": 18,
"decode_tokens": 56,
"max_batch_occupancy": 4,
"peak_kv_bytes": 52224,
"prefill_tokens": 32,
"rejected_admissions": 0,
"ticks": 19,
"tokens_per_tick": 4.6316
},
{
"avg_batch_occupancy": 4.0,
"cache_misses": 0,
"concurrency": 8,
"decode_batches": 14,
"decode_tokens": 56,
"max_batch_occupancy": 7,
"peak_kv_bytes": 75264,
"prefill_tokens": 32,
"rejected_admissions": 0,
"ticks": 15,
"tokens_per_tick": 5.8667
}
],
"saturation_concurrency": 8,
"schema_version": 1
},
"evidence_kind": "synthetic-unit",
"mid_run_telemetry_concurrency_4": {
"active_sessions": 4,
"batch_occupancy_avg": 4.0,
"batch_occupancy_last": 4,
"batch_occupancy_max": 4,
"completed_sessions": 0,
"decode_tokens_per_sec": 1637.355,
"decode_tokens_total": 20,
"kv_budget_bytes": 67108864,
"kv_pressure": 0.0008,
"kv_total_bytes": 55296,
"prefill_tokens_per_sec": 1309.884,
"prefill_tokens_total": 16,
"queue_depth": 4,
"rejected_admissions_total": 0,
"rejected_by_reason": {},
"scratch_budget_bytes": 4,
"scratch_pressure": 1.0,
"scratch_used_bytes": 4,
"ticks": 6,
"weight_bytes": 0
},
"model": {
"hidden": 32,
"n_heads": 4,
"n_layers": 6,
"reference": "pure-numpy KV-cached dense-Llama (tests/test_hot_kv_state)",
"vocab": 48
},
"schema_version": 1,
"workload": {
"max_new_tokens": 8,
"prompt_len": 4,
"sessions": 8
}
}

223
.scratch/distributed-gguf-runtime/evidence/DGR-013/README.md
View File

@@ -1,223 +0,0 @@
# DGR-013 — Harden failure, cancellation, and restart semantics: evidence
Status: done
Date: 2026-07-16
Evidence kind: **synthetic-unit** (pure-numpy KV-cached dense-Llama reference +
node-local hardened stream). No model download, no GPU, no torch, no network, no
API credit.
## Summary
Implemented bounded, explicit failure/cancellation/restart semantics for the
per-Route-Session decode stream, layered on the DGR-007 Hot KV State manager
(isolated `(session, epoch)` KV) and the DGR-012 continuous-batch scheduler. The
goal (RALPH product objective) is that distributed speed never comes with hanging
or corrupted generations: every blocked op is bounded, every cancel frees state,
duplicate steps are idempotent, uncertain mutations are never silently replayed,
alpha failover restarts from token zero, and billing distinguishes what actually
completed.
Everything runs against the same deterministic numpy dense-Llama reference the
default gate uses (`tests/test_hot_kv_state.py::_KvDenseLlama` / `_KvReferenceShard`),
so the whole failure matrix is deterministic, download-free, GPU-free, and
API-credit-free while exercising the **real** KV isolation path
(`KvBoundaryAdapter` + `HotKvStateManager`). The pinned llama.cpp worker (DGR-008)
implements the identical adapter contract, so the semantics carry over to native
execution unchanged.
### What was built (`packages/node/meshnet_node/failure_semantics.py`, new)
- **`DeadlineGuard` + `StreamTerminated`** — bounds every step against an absolute
deadline and a heartbeat-timeout on an injected clock. A reached deadline or a
lost heartbeat (peer health loss) raises `StreamTerminated(kind)` so a blocked
stream terminates instead of hanging. (**AC: deadlines/heartbeat terminate
blocked ops.**)
- **`CancellationToken`, `ShardCancellationGroup`, `CancellationOutcome`** — one
cancel fans across **every** node-local Shard of a Route Session, releasing the
`(session, epoch)` KV on each shard's manager and invoking every queued-buffer
release callback (the pending activation bundles). Idempotent. The DGR-012
scheduler also gains a `cancel()` that drops queued/active work on this node and
frees its KV. (**AC: cancellation propagates across every Shard, releases KV +
queued buffers.**)
- **`IdempotencyLedger`, `StepKey`, `StepDisposition`, `UncertainMutationError`** —
records each committed `(session, epoch, step)`; a duplicate delivery returns the
recorded token with no re-mutation. A step whose mutation outcome is *uncertain*
(worker died mid-step) is marked uncertain and can **never** be replayed
silently — `begin()` on an uncertain (or still in-flight) step raises
`UncertainMutationError`, forcing verify-or-restart. (**AC: duplicate steps
idempotent; uncertain mutations never replayed silently.**)
- **`RestartController`** — alpha failover: opens the *next* route epoch, releases
every shard's prior-epoch KV, and `assert_fresh_start` fails closed if any shard
still holds new-epoch KV. The restart re-prefills the whole prompt from token
zero; the failed epoch becomes stale (KV manager rejects it). Unverified KV is
never migrated (RALPH runtime decision #14). (**AC: alpha failover restarts from
token zero rather than importing unverified KV.**)
- **`WorkStatus`, `WorkRecord`, `WorkLedger`** — a typed per-attempt work record
with four distinct statuses: `completed`, `cancelled`, `failed`, `unverified`.
Only `completed` records are billable; cancelled/failed/unverified tokens are
recorded for observability but never charged. JSON-safe for the tracker billing
handoff (`packages/tracker/meshnet_tracker/billing.py` charges only completed,
verified work). (**AC: billing/work records distinguish completed/cancelled/
failed/unverified.**)
- **`HardenedSessionRunner`** — composes all of the above to drive one session's
prefill+decode through the adapter under a deadline/heartbeat guard + cancel
token, records the typed outcome, and `run_with_failover` restarts a transient
failure from token zero on a fresh epoch.
- **`FailureKind` + `classify_exception` + `work_status_for`** — stable-string
classification of worker death, stream reset, malformed bundle, stale epoch,
cache miss, deadline, heartbeat loss, and cancel, plus the failure→billing-status
mapping. Suitable for the native protocol's structured status.
### Scheduler extension (`packages/node/meshnet_node/batch_scheduler.py`, DGR-012 file, additive)
Purely additive so the DGR-012 gate stays green (16/16):
- `DoneReason.CANCELLED` / `DoneReason.FAILED` terminal reasons.
- `ContinuousBatchScheduler.cancel(session_id, *, reason)` — drops a queued
session from the bounded queue or releases an active session's KV, moving it to
the done set with a non-completed reason (never counted as completed work).
- `SchedulerTelemetry.cancelled_sessions` / `failed_sessions` counters.
## Files changed
- `packages/node/meshnet_node/failure_semantics.py` — new module (the whole
failure/cancel/restart layer above).
- `packages/node/meshnet_node/batch_scheduler.py` — additive `cancel()` + two
`DoneReason` members + two telemetry counters (DGR-012 file; its 16 tests still
pass unchanged).
- `tests/test_failure_semantics.py` — new, 22 tests (matrix below); reuses the
DGR-007 numpy reference via `from test_hot_kv_state import _KvDenseLlama,
_KvReferenceShard`.
- `.scratch/distributed-gguf-runtime/evidence/DGR-013/` — this README,
`commands.txt`, `generate_evidence.py`, `results.json`.
- `.ralph-tui/progress.md` — appended the DGR-013 note.
- `.scratch/distributed-gguf-runtime/issues/13-...md` — set `Status: done`.
## Acceptance criteria → evidence
| Criterion | Tests (`tests/test_failure_semantics.py`) |
|---|---|
| Deadlines/heartbeat loss terminate blocked stream ops | `test_deadline_terminates_a_blocked_stream_and_releases_kv`, `test_heartbeat_loss_terminates_a_blocked_stream`, `test_deadline_guard_reports_remaining_and_resets_on_heartbeat` |
| Cancellation propagates across every Shard, releases KV + queued buffers | `test_cancellation_token_terminates_stream_and_releases_kv`, `test_shard_cancellation_group_releases_every_shard_and_queued_buffers`, `test_scheduler_cancel_drains_queue_and_releases_active_kv`, `test_scheduler_cancel_rejects_a_completed_reason` |
| Duplicate steps idempotent; uncertain mutations never replayed silently | `test_duplicate_step_delivery_is_idempotent_no_remutation`, `test_idempotent_run_replays_tokens_without_advancing_kv`, `test_uncertain_mutation_is_never_replayed_silently`, `test_in_flight_duplicate_is_treated_as_uncertain` |
| Alpha failover restarts from token zero, no unverified KV import | `test_alpha_failover_restarts_from_token_zero_and_completes`, `test_failover_refuses_to_import_unverified_kv`, `test_non_restartable_failure_is_not_retried` |
| Worker death, stream reset, malformed bundle, stale epoch, cache miss | `test_worker_death_midstream_is_unverified_and_marks_step_uncertain`, `test_stream_reset_is_restartable_failure`, `test_malformed_bundle_is_classified_and_does_not_corrupt_kv`, `test_stale_epoch_reference_is_rejected_and_classified`, `test_cache_miss_midstream_is_restartable` |
| Billing/work records distinguish completed/cancelled/failed/unverified | `test_work_ledger_distinguishes_all_four_statuses`, `test_work_status_and_classification_mapping`, plus the clean-run billability check `test_clean_run_matches_stateless_reference_and_is_billable` |
## Failure matrix (real, deterministic — `results.json`)
Generated by `generate_evidence.py` against the numpy dense-Llama (prompt `[7,3,9,1]`,
8 new tokens):
| scenario | status | failure_kind | tokens | restartable | KV released |
|---|---|---|---|---|---|
| clean | completed | — | 8 | — | (held, then reaped) |
| deadline | failed | deadline-exceeded | 2 | no | yes |
| heartbeat_loss | failed | heartbeat-lost | 3 | no | yes |
| cancel | cancelled | cancelled | 3 | no | yes |
| worker_death | unverified | worker-death | 3 | yes | yes |
| stream_reset | failed | stream-reset | — | yes | yes |
| stale_epoch | failed | stale-epoch | — | no | (never opened) |
| cache_miss | failed | cache-miss | 4 | yes | (already evicted) |
| alpha_failover | **completed** (epoch 1) | — | 8 | — | old epoch stale |
Alpha failover: attempt 0 (epoch 0) dies mid-step → `unverified`; the controller
advances to epoch 1, drops epoch-0 KV, and the restart re-prefills from token zero
→ `completed`, reproducing the byte-identical stateless reference. The old epoch is
now stale (a reference to it raises `StaleRouteEpochError`). Work ledger:
`{completed: 2, cancelled: 1, failed: 0, unverified: 2}`, `billable_tokens = 16`
(only the two completed streams — the failover restart and the clean run — are
billed; the cancelled and the two unverified attempts are not).
## Commands and real results
See `commands.txt`. Key results:
```
tests/test_failure_semantics.py -> 22 passed
tests/test_batch_scheduler.py -> 16 passed (DGR-012 unchanged)
tests/test_hot_kv_state.py -> 22 passed (DGR-007)
tests/test_gguf_backend.py -> 2 passed (DGR-009)
python -m compileall -q packages tests -> exit 0
git diff --check -> exit 0
python -m pytest -q -> 16 failed, 792 passed, 14 skipped in 253.93s
```
## Full-suite baseline (pre-existing, unrelated failures)
The 16 failures are **pre-existing and unrelated to DGR-013**. None import
`failure_semantics` or `batch_scheduler`; they live in the tracker/control-plane,
node-startup, doctor, calibration, and route-benchmark suites and fail on the
model-download / control-plane / recipe-admission paths (e.g.
`UnsupportedRecipeParam: worker_transport` from the DGR-009 native recipe against
the Torch backend, and Torch/HF-model startup that this deterministic sandbox does
not provide). Removing the two DGR-013 files and re-running the failing tests
reproduces the identical failures (see `commands.txt`, 4-test spot check → same
4 failures), so DGR-013 introduces no new failure.
Exact failing set (16):
```
tests/test_dynamic_routing.py::test_admin_can_replace_a_served_model_and_release_it
tests/test_manual_route_benchmark.py::test_pinned_route_uses_named_node
tests/test_manual_route_benchmark.py::test_unknown_route_node_is_400
tests/test_manual_route_benchmark.py::test_invalid_route_shape_is_400
tests/test_manual_route_benchmark.py::test_clients_without_route_are_unaffected
tests/test_manual_route_benchmark.py::test_benchmark_records_one_and_two_node_routes
tests/test_node_doctor.py::test_the_shipped_recipes_are_all_applicable_by_the_backend
tests/test_node_doctor.py::test_cli_doctor_flags_select_what_is_validated
tests/test_node_startup.py::test_preset_model_with_hf_repo_loads_torch_backend
tests/test_node_startup.py::test_real_model_startup_registers_downloaded_inventory_without_checksum
tests/test_toploc_calibration_dispatch.py::test_calibration_run_dispatches_only_solo_capable_nodes
tests/test_toploc_calibration_dispatch.py::test_calibration_run_persists_corpus_and_results_endpoint_reports_it
tests/test_toploc_calibration_dispatch.py::test_calibration_run_node_without_commitment_endpoint_is_skipped_not_failed
tests/test_tracker_capability_admission.py::test_an_enforcing_tracker_never_routes_a_node_whose_proof_does_not_cover_it[invalid]
tests/test_tracker_routing.py::test_torch_node_applies_tracker_load_shard_directive
tests/test_tracker_routing.py::test_shard_heal_cycle_surviving_node_covers_dead_peers_gap
```
## Limitations and deferred work
- **Synthetic-unit, not real weights.** Semantics are exercised against the
deterministic numpy dense-Llama, not a downloaded GGUF, to keep the default gate
deterministic/download-free/GPU-free. Real worker-death/stream-reset behavior on
a live llama.cpp worker over gRPC belongs to DGR-008/DGR-010 (DGR-010 is blocked
— no certified dense-Llama artifact on this machine; see
`evidence/DGR-010/BLOCKED.md`).
- **Single-node per-session stream.** `HardenedSessionRunner` drives one full-shard
session (the node-local case); multi-node cancellation is modelled by
`ShardCancellationGroup` fanning across each node's KV manager. The cross-node
propagation *transport* (cancel frames over gRPC/relay) is the native protocol's
job (DGR-002/008); this story owns the local release + record semantics the
transport triggers.
- **Fault injection is deterministic.** Worker death is a shard that raises on the
Nth step; stream reset / deadline / heartbeat are injected via an explicit clock
and hook. This is what makes the matrix reproducible; live fault behavior is a
native/real-hardware property.
- **Greedy sampling only.** Reuses the DGR-006 greedy `SamplingContract`; the
idempotent-replay equality check depends on order-independent greedy decode.
- **Native / llama.cpp gates N/A.** No native code, CMake, or llama.cpp patch was
touched (same as DGR-005/006/007/012), so those gates do not apply.
## Compatibility / migration notes
- `failure_semantics.py` is a new, additive module — no existing behavior changes.
- `batch_scheduler.py` changes are additive (new enum members, one method, two
telemetry fields); the DGR-012 contract and its 16 tests are unchanged.
- `WorkRecord.to_dict()` / `WorkLedger.to_dict()` are JSON-safe and map cleanly to
the tracker `BillingLedger.charge_request` inputs: report `node_work` only for
`billable` (completed) records so cancelled/failed/unverified work is never
charged. `FailureKind` / `WorkStatus` are stable strings suitable for the native
protocol's structured status and the capability/heartbeat report.
## Handoff for dependent stories
- **DGR-008 (C++ gRPC worker):** implement the same contract natively — the worker
maps a transport deadline/heartbeat to `StreamTerminated`, a dropped stream to a
restartable failure, and a mid-`llama_decode` crash to an *uncertain* step
(mark-uncertain, never silent replay). `RestartController.failover` maps to
opening a fresh llama sequence under the new `(session, epoch)`; the failed
sequence's KV is dropped, never migrated.
- **DGR-010/DGR-014 (real acceptance / release gate):** drive the same failure
scenarios against the live worker to produce real cleanup/latency numbers, and
feed the `WorkLedger` status split into the billing/attribution comparison —
only `completed` work is charged.

36
.scratch/distributed-gguf-runtime/evidence/DGR-013/commands.txt
View File

@@ -1,36 +0,0 @@
# DGR-013 — exact commands and real results (worktree venv)
VP=/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python
# Targeted story tests (this story)
$VP -m pytest -q tests/test_failure_semantics.py
# -> 22 passed
# Dependency gates stay green
$VP -m pytest -q tests/test_batch_scheduler.py # DGR-012
# -> 16 passed
$VP -m pytest -q tests/test_hot_kv_state.py # DGR-007
# -> 22 passed
$VP -m pytest -q tests/test_gguf_backend.py # DGR-009
# -> 2 passed
# Quality gates
$VP -m compileall -q packages tests
# -> exit 0
git diff --check
# -> exit 0
# Machine-readable evidence
$VP .scratch/distributed-gguf-runtime/evidence/DGR-013/generate_evidence.py
# -> wrote results.json; work statuses {'completed':2,'cancelled':1,'failed':0,'unverified':2} billable_tokens=16
# Full deterministic suite
$VP -m pytest -q -p no:cacheprovider
# -> 16 failed, 792 passed, 14 skipped in 253.93s
# Clean-tree reproduction of the 16 pre-existing failures (DGR-013 files removed)
# rm packages/node/meshnet_node/failure_semantics.py tests/test_failure_semantics.py
$VP -m pytest -q tests/test_dynamic_routing.py::test_admin_can_replace_a_served_model_and_release_it \
tests/test_node_doctor.py::test_the_shipped_recipes_are_all_applicable_by_the_backend \
tests/test_tracker_routing.py::test_torch_node_applies_tracker_load_shard_directive \
tests/test_node_startup.py::test_preset_model_with_hf_repo_loads_torch_backend
# -> 4 failed (same failures reproduce without any DGR-013 change)

234
.scratch/distributed-gguf-runtime/evidence/DGR-013/generate_evidence.py
View File

@@ -1,234 +0,0 @@
#!/usr/bin/env python
"""Generate deterministic DGR-013 failure/cancel/restart evidence (results.json).
Runs the real hardened per-session stream (``HardenedSessionRunner`` over the
DGR-007 ``KvBoundaryAdapter`` + ``HotKvStateManager``) through each failure mode
with the same pure-numpy dense-Llama reference the default gate uses. No model
download, no GPU, no torch, no network, no API credit.
Run from the repo root with the worktree venv:
.venv/bin/python .scratch/distributed-gguf-runtime/evidence/DGR-013/generate_evidence.py
"""
from __future__ import annotations
import json
import os
import sys
import numpy as np
# Make the worktree packages and the DGR-007 numpy reference importable, exactly
# as pytest's prepend-import + conftest do.
ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "..", "..", ".."))
sys.path.insert(0, os.path.join(ROOT, "packages", "node"))
sys.path.insert(0, os.path.join(ROOT, "tests"))
from meshnet_node.hot_kv_state import ( # noqa: E402
HotKvStateConfig,
HotKvStateManager,
KvBoundaryAdapter,
StaleRouteEpochError,
kv_recipe_for,
)
from meshnet_node.batch_scheduler import GenerationRequest # noqa: E402
from meshnet_node.failure_semantics import ( # noqa: E402
CancellationToken,
FailureKind,
HardenedSessionRunner,
RestartController,
StreamTerminated,
WorkLedger,
WorkStatus,
)
from test_hot_kv_state import _KvDenseLlama, _KvReferenceShard # noqa: E402
class _FaultyShard(_KvReferenceShard):
def __init__(self, model, start, end, *, fail_at_call=None):
super().__init__(model, start, end)
self._fail_at_call = fail_at_call
self.calls = 0
def run_layers_cached(self, hidden, *, positions, past_kv):
self.calls += 1
if self._fail_at_call is not None and self.calls == self._fail_at_call:
raise RuntimeError("worker died mid-step")
return super().run_layers_cached(hidden, positions=positions, past_kv=past_kv)
class _Clock:
def __init__(self):
self.now = 0.0
def __call__(self):
return self.now
def advance(self, d):
self.now += d
def _adapter(model, *, config=None, shard=None):
shard = shard or _KvReferenceShard(model, 0, model.n_layers - 1)
manager = HotKvStateManager(kv_recipe_for(shard), config=config)
return KvBoundaryAdapter(shard, manager)
def _gen(sid, prompt, n, epoch=0):
return GenerationRequest(
session_id=sid, route_epoch=epoch,
prompt_token_ids=tuple(prompt), max_new_tokens=n,
)
def _kv_released(manager, sid, epoch):
from meshnet_node.hot_kv_state import CacheMiss
return isinstance(manager.resolve(sid, epoch), CacheMiss)
def main() -> None:
model = _KvDenseLlama()
prompt = [7, 3, 9, 1]
n_new = 8
ledger = WorkLedger()
scenarios = []
# 1. Clean baseline.
ad = _adapter(model)
r = HardenedSessionRunner(ad, work_ledger=ledger).run(_gen("clean", prompt, n_new))
scenarios.append({
"scenario": "clean",
"status": r.status.value,
"tokens": r.token_count,
"matches_reference": list(r.tokens) == model.stateless_greedy(prompt, n_new),
"kv_released": _kv_released(ad.manager, "clean", 0),
})
# 2. Deadline terminates a blocked stream.
clk = _Clock()
ad = _adapter(model)
r = HardenedSessionRunner(ad, clock=clk).run(
_gen("deadline", prompt, 50), deadline=3.0,
before_step=lambda _s: clk.advance(1.0),
)
scenarios.append({
"scenario": "deadline", "status": r.status.value,
"failure_kind": r.failure_kind.value, "tokens": r.token_count,
"kv_released": _kv_released(ad.manager, "deadline", 0),
})
# 3. Heartbeat/health loss terminates a blocked stream.
clk = _Clock()
ad = _adapter(model)
r = HardenedSessionRunner(ad, clock=clk).run(
_gen("heartbeat", prompt, 50), heartbeat_timeout=1.5,
heartbeat=lambda step: step < 2,
before_step=lambda _s: clk.advance(1.0),
)
scenarios.append({
"scenario": "heartbeat_loss", "status": r.status.value,
"failure_kind": r.failure_kind.value, "tokens": r.token_count,
"kv_released": _kv_released(ad.manager, "heartbeat", 0),
})
# 4. Explicit client cancellation releases KV.
ad = _adapter(model)
tok = CancellationToken()
r = HardenedSessionRunner(ad, work_ledger=ledger).run(
_gen("cancel", prompt, 50), cancel_token=tok,
before_step=lambda step: tok.cancel("client-hangup") if step == 3 else None,
)
scenarios.append({
"scenario": "cancel", "status": r.status.value,
"failure_kind": r.failure_kind.value, "tokens": r.token_count,
"kv_released": _kv_released(ad.manager, "cancel", 0),
})
# 5. Worker death mid-step -> unverified.
ad = _adapter(model, shard=_FaultyShard(model, 0, model.n_layers - 1, fail_at_call=4))
r = HardenedSessionRunner(ad, work_ledger=ledger).run(_gen("worker", prompt, n_new))
scenarios.append({
"scenario": "worker_death", "status": r.status.value,
"failure_kind": r.failure_kind.value, "tokens": r.token_count,
"restartable": r.restartable, "kv_released": _kv_released(ad.manager, "worker", 0),
})
# 6. Stream reset -> failed, restartable.
ad = _adapter(model)
def reset(step):
if step == 2:
raise StreamTerminated(FailureKind.STREAM_RESET, "peer reset")
r = HardenedSessionRunner(ad).run(_gen("reset", prompt, n_new), before_step=reset)
scenarios.append({
"scenario": "stream_reset", "status": r.status.value,
"failure_kind": r.failure_kind.value, "restartable": r.restartable,
})
# 7. Stale epoch -> failed.
ad = _adapter(model)
ad.manager.open("stale", 5)
r = HardenedSessionRunner(ad).run(_gen("stale", prompt, n_new, epoch=3))
scenarios.append({
"scenario": "stale_epoch", "status": r.status.value,
"failure_kind": r.failure_kind.value,
})
# 8. Cache miss mid-stream -> restartable.
ad = _adapter(model)
mgr = ad.manager
r = HardenedSessionRunner(ad).run(
_gen("miss", prompt, 12),
before_step=lambda step: mgr.release("miss", 0) if step == 4 else None,
)
scenarios.append({
"scenario": "cache_miss", "status": r.status.value,
"failure_kind": r.failure_kind.value, "tokens": r.token_count,
"restartable": r.restartable,
})
# 9. Alpha failover: restart from token zero, no unverified KV import.
faulty = _FaultyShard(model, 0, model.n_layers - 1, fail_at_call=3)
ad = _adapter(model, shard=faulty)
runner = HardenedSessionRunner(ad, work_ledger=ledger)
controller = RestartController([ad.manager])
fo = runner.run_with_failover(_gen("failover", prompt, n_new, epoch=0), controller,
max_restarts=2)
old_epoch_stale = False
try:
ad.manager.resolve("failover", 0)
except StaleRouteEpochError:
old_epoch_stale = True
scenarios.append({
"scenario": "alpha_failover",
"final_status": fo.outcome.status.value,
"final_epoch": fo.outcome.route_epoch,
"restarts": fo.restarts,
"restarted_from_token_zero": list(fo.outcome.tokens) == model.stateless_greedy(prompt, n_new),
"old_epoch_stale": old_epoch_stale,
"attempt_statuses": [a.status.value for a in fo.attempts],
})
result = {
"schema_version": 1,
"evidence_kind": "synthetic-unit",
"model": {
"architecture": model.architecture_adapter,
"n_layers": model.n_layers, "vocab": model.vocab, "hidden": model.hidden,
},
"scenarios": scenarios,
"work_ledger": ledger.to_dict(),
}
out_path = os.path.join(os.path.dirname(__file__), "results.json")
with open(out_path, "w") as fh:
json.dump(result, fh, indent=2)
fh.write("\n")
counts = ledger.counts_by_status()
print(f"wrote {out_path}")
print(f"work statuses: {counts} billable_tokens={ledger.billable_tokens()}")
if __name__ == "__main__":
main()

135
.scratch/distributed-gguf-runtime/evidence/DGR-013/results.json
View File

@@ -1,135 +0,0 @@
{
"schema_version": 1,
"evidence_kind": "synthetic-unit",
"model": {
"architecture": "dense-llama",
"n_layers": 6,
"vocab": 48,
"hidden": 32
},
"scenarios": [
{
"scenario": "clean",
"status": "completed",
"tokens": 8,
"matches_reference": true,
"kv_released": false
},
{
"scenario": "deadline",
"status": "failed",
"failure_kind": "deadline-exceeded",
"tokens": 2,
"kv_released": true
},
{
"scenario": "heartbeat_loss",
"status": "failed",
"failure_kind": "heartbeat-lost",
"tokens": 3,
"kv_released": true
},
{
"scenario": "cancel",
"status": "cancelled",
"failure_kind": "cancelled",
"tokens": 3,
"kv_released": true
},
{
"scenario": "worker_death",
"status": "unverified",
"failure_kind": "worker-death",
"tokens": 3,
"restartable": true,
"kv_released": true
},
{
"scenario": "stream_reset",
"status": "failed",
"failure_kind": "stream-reset",
"restartable": true
},
{
"scenario": "stale_epoch",
"status": "failed",
"failure_kind": "stale-epoch"
},
{
"scenario": "cache_miss",
"status": "failed",
"failure_kind": "cache-miss",
"tokens": 4,
"restartable": true
},
{
"scenario": "alpha_failover",
"final_status": "completed",
"final_epoch": 1,
"restarts": 1,
"restarted_from_token_zero": true,
"old_epoch_stale": true,
"attempt_statuses": [
"unverified",
"completed"
]
}
],
"work_ledger": {
"schema_version": 1,
"records": [
{
"session_id": "clean",
"route_epoch": 0,
"status": "completed",
"tokens": 8,
"failure_kind": null,
"detail": "",
"billable": true
},
{
"session_id": "cancel",
"route_epoch": 0,
"status": "cancelled",
"tokens": 3,
"failure_kind": "cancelled",
"detail": "operation cancelled: client-hangup",
"billable": false
},
{
"session_id": "worker",
"route_epoch": 0,
"status": "unverified",
"tokens": 3,
"failure_kind": "worker-death",
"detail": "worker died mid-step",
"billable": false
},
{
"session_id": "failover",
"route_epoch": 0,
"status": "unverified",
"tokens": 2,
"failure_kind": "worker-death",
"detail": "worker died mid-step",
"billable": false
},
{
"session_id": "failover",
"route_epoch": 1,
"status": "completed",
"tokens": 8,
"failure_kind": null,
"detail": "",
"billable": true
}
],
"counts_by_status": {
"completed": 2,
"cancelled": 1,
"failed": 0,
"unverified": 2
},
"billable_tokens": 16
}
}

55
.scratch/distributed-gguf-runtime/evidence/DGR-014/BLOCKED.md
View File

@@ -1,55 +0,0 @@
# DGR-014 — Blocked handoff
Status: blocked
Date: 2026-07-16
## Blocker
This release-gate story cannot be completed in the current workspace state because the prerequisite real-model comparison chain is still missing its certified dense-Llama artifact on mounted storage.
Verified blockers:
- `DGR-011` is still not passed in `.scratch/distributed-gguf-runtime/prd.json`.
- `DGR-011` is explicitly blocked in `.scratch/distributed-gguf-runtime/evidence/DGR-011/BLOCKED.md`.
- `DGR-011` depends on `DGR-010`, and `DGR-010` is blocked because there is no certified dense-Llama artifact available on the mounted drive.
- Current mounted-model storage still only shows Qwen artifacts and llama.cpp vocab GGUFs, not the certified dense-Llama GGUF/safetensors pair needed for a comparable real run.
## Verified current state
- The DGR-001 performance contract exists and defines the benchmark lanes, metrics, and stop condition that later release gates must keep unchanged.
- The DGR-012 scheduler and DGR-013 failure semantics evidence are present and usable as supporting context, but they do not satisfy the real final comparison required here.
- `packages/node/meshnet_node/performance_contract.py` already contains the contract metadata and a live endpoint benchmark shim, but there is no recorded DGR-014 release-gate run and no final immutable comparison artifact.
- `evidence/DGR-014/README.md` does not exist yet because the acceptance criteria could not be completed.
## Commands run
```bash
sed -n '1,260p' .claude/memory/MEMORY.md
sed -n '1,260p' .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md
sed -n '1,260p' .scratch/distributed-gguf-runtime/issues/14-enforce-the-gguf-versus-safetensors-release-gate.md
sed -n '1,260p' .ralph-tui/progress.md
git status --short
sed -n '1,260p' .scratch/distributed-gguf-runtime/prd.json
sed -n '1,260p' .scratch/distributed-gguf-runtime/evidence/DGR-001/README.md
sed -n '1,260p' .scratch/distributed-gguf-runtime/evidence/DGR-012/README.md
sed -n '1,260p' .scratch/distributed-gguf-runtime/evidence/DGR-013/README.md
sed -n '1,260p' .scratch/distributed-gguf-runtime/evidence/DGR-011/BLOCKED.md
sed -n '1,260p' .scratch/distributed-gguf-runtime/evidence/DGR-010/BLOCKED.md
find /run/media/popov/d/DEV/models /run/media/popov/d/DEV/llamacpp/llama.cpp/models -maxdepth 4 \( -iname '*llama*' -o -iname '*deepseek*' -o -iname '*dense*' -o -name '*.gguf' -o -name '*.safetensors' -o -name 'config.json' \)
```
## Known limitations
- No certified dense-Llama artifact is mounted, so the real distributed safetensors-versus-GGUF comparison cannot be executed.
- No immutable release-gate evidence can be produced without that artifact and the completed DGR-011 route comparison.
- No code was changed in this iteration.
## Compatibility notes
- The DGR-001 contract remains the source of truth for thresholds and metric names.
- Any future DGR-014 run must keep those thresholds unchanged and compare the same certified model/hardware/network scenario for both routes.
## Dependent-story handoff
- Finish `DGR-010` and `DGR-011` first with a certified dense-Llama artifact on mounted storage.
- Then run the current distributed safetensors and distributed GGUF routes on the same comparable scenario, record the final numbers in `evidence/DGR-014/README.md`, and update the issue status only after the gate passes.

78
.scratch/distributed-gguf-runtime/evidence/DGR-015/BLOCKED.md
View File

@@ -1,78 +0,0 @@
# DGR-015 — Blocked handoff
Status: blocked
Date: 2026-07-16
## Blocker
This story cannot be completed in the current workspace state because its
mandatory prerequisite, DGR-014, is still not passed.
Verified blocker chain:
- `.scratch/distributed-gguf-runtime/prd.json` still marks `DGR-014` as
`"passes": false`, so DGR-015 is not released for completion.
- `.scratch/distributed-gguf-runtime/evidence/DGR-014/BLOCKED.md` records the
release-gate blocker: the certified dense-Llama artifact required for the
comparable real-model comparison is not mounted on this machine.
- `DGR-014` depends on `DGR-011`, which is also blocked because `DGR-010`
cannot run without that same certified dense-Llama artifact.
- The current codebase still fails closed for `qwen3` / `qwen3-moe` in
`packages/node/meshnet_node/boundary_adapter.py`, which is correct for the
current state but means no Qwen3 family recipe is certified yet.
## Verified current state
- Dense-Llama boundary semantics, Hot KV isolation, batching, and failure
semantics are already implemented and covered by prior stories.
- Qwen3 strings are present in tracker/model metadata, but they are not yet
backed by a certified architecture adapter or real-model acceptance evidence.
- No `evidence/DGR-015/README.md` exists yet because the acceptance criteria
could not be completed.
## Commands run
```bash
sed -n '1,260p' .claude/memory/MEMORY.md
sed -n '1,260p' .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md
sed -n '1,260p' .scratch/distributed-gguf-runtime/issues/15-add-and-certify-a-qwen3-qwen3-moe-adapter.md
sed -n '1,260p' .scratch/distributed-gguf-runtime/architecture.md
sed -n '1,260p' CONTEXT.md
sed -n '1,260p' .scratch/distributed-gguf-runtime/prd.json
sed -n '1,260p' .scratch/distributed-gguf-runtime/evidence/DGR-014/BLOCKED.md
sed -n '1,260p' .scratch/distributed-gguf-runtime/evidence/DGR-013/README.md
sed -n '1,260p' .scratch/distributed-gguf-runtime/evidence/DGR-012/README.md
sed -n '1,260p' packages/node/meshnet_node/boundary_adapter.py
sed -n '1,260p' packages/node/meshnet_node/model_catalog.py
sed -n '1,220p' packages/node/meshnet_node/model_metadata.json
sed -n '1,260p' packages/tracker/meshnet_tracker/capability.py
sed -n '1,260p' packages/tracker/meshnet_tracker/server.py
rg -n "qwen3|qwen3-moe|Qwen3|MoE|router|top-k|shared expert|shared_expert|expert" packages/node/meshnet_node packages/tracker/meshnet_tracker tests -g '!**/__pycache__/**'
git status --short
```
## Known limitations
- No certified dense-Llama artifact is mounted, so DGR-014 cannot complete and
DGR-015 remains blocked behind it.
- No real consumer-hardware Qwen3 acceptance run was possible in this workspace.
- No code was changed in this iteration.
## Compatibility notes
- The current boundary adapter intentionally fails closed for uncertified
architectures. That is the correct behavior until a dedicated Qwen3 adapter is
implemented and certified.
- Existing dense-Llama coverage and Hot KV semantics remain the source of truth
for the shared protocol and cache behavior.
## Dependent-story handoff
- Finish `DGR-010`, `DGR-011`, and `DGR-014` first with a certified dense-Llama
artifact on mounted storage.
- Once the release gate passes, implement the Qwen3 family adapter as a separate
certified architecture rather than by extending dense-Llama with unchecked name
substitutions.
- Record the real-model Qwen3 parity, admission, memory, and communication
evidence in `evidence/DGR-015/README.md`, then update the issue status only
after the gate passes.

145
.scratch/distributed-gguf-runtime/evidence/DGR-016/README.md
View File

@@ -1,145 +0,0 @@
# DGR-016 — Upstream llama.cpp collaboration package
Status: partial, blocked by DGR-010
Date: 2026-07-16
## Summary
Assembled the upstream-facing collaboration package for llama.cpp without
pulling Meshnet routing or control-plane logic into the upstream ask.
Durable outputs created for this story:
- `api-note.md` with the generic hook split and patch-per-concern proposal
- `outreach.md` with a maintainer-facing draft for Georgi/llama.cpp
The package is grounded in the existing research artifacts and the already
implemented deterministic tests for:
- range-aware GGUF ownership and introspection
- architecture boundary input/output
- layer-filtered KV/session ownership
- reproducible pinned worker build wiring
The story itself remains blocked because DGR-010 is still marked `passes: false`
and only has a blocked handoff, not a completed real-model acceptance README.
## Files changed
- `.scratch/distributed-gguf-runtime/evidence/DGR-016/README.md`
- `.scratch/distributed-gguf-runtime/evidence/DGR-016/api-note.md`
- `.scratch/distributed-gguf-runtime/evidence/DGR-016/outreach.md`
## Commands run and real results
### Dependency and context review
```bash
sed -n '1,260p' .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md
sed -n '1,260p' .scratch/distributed-gguf-runtime/issues/16-produce-the-upstream-llama-cpp-collaboration-package.md
sed -n '1,260p' .scratch/distributed-gguf-runtime/evidence/DGR-010/BLOCKED.md
sed -n '1,260p' docs/adr/0024-distributed-gguf-runtime.md
sed -n '1,260p' .scratch/distributed-gguf-runtime/architecture.md
sed -n '1,260p' .scratch/distributed-gguf-runtime/decision-framework.md
sed -n '1,260p' .scratch/distributed-gguf-runtime/implementation-strategy.md
sed -n '1,260p' CONTEXT.md
```
Result:
- confirmed the runtime target is a small pinned llama.cpp worker with Meshnet
kept outside upstream
- confirmed DGR-010 is still blocked because there is no certified dense-Llama
artifact on mounted storage
### Package-relevant targeted pytest
```bash
python -m pytest -q tests/test_llama_worker_build.py tests/test_gguf_backend.py tests/test_gguf_ownership.py tests/test_boundary_adapter.py tests/test_hot_kv_state.py
```
Result:
- `50 passed in 0.90s`
### Broader focused pytest slice
```bash
python -m pytest -q tests/test_llama_worker_build.py tests/test_native_shard_protocol.py tests/test_gguf_backend.py tests/test_boundary_adapter.py tests/test_gguf_ownership.py tests/test_hot_kv_state.py tests/test_kv_cache_distributed.py
```
Result:
- `58 passed, 1 skipped, 9 failed, 12 errors in 1.27s`
- failures were pre-existing environment issues, not this documentation-only
package:
- `tests/test_native_shard_protocol.py` imported generated protobuf code built
against gencode 7.35.0 while the active runtime is 6.33.6
- `tests/test_kv_cache_distributed.py` hit sandbox socket `PermissionError`
when trying to bind localhost servers
### Research evidence review
```bash
sed -n '1,260p' docs/research/distributed-gguf-landscape.md
sed -n '1,260p' docs/research/distributed-gguf-github-followup.md
sed -n '1,220p' .scratch/distributed-gguf-runtime/evidence/DGR-004/README.md
sed -n '1,260p' .scratch/distributed-gguf-runtime/evidence/DGR-006/README.md
sed -n '1,260p' .scratch/distributed-gguf-runtime/evidence/DGR-007/README.md
sed -n '1,260p' .scratch/distributed-gguf-runtime/evidence/DGR-009/README.md
```
Result:
- confirmed Nakshatra and prima.cpp are the right source/test donors for the
upstream ask
- confirmed the generic API surface is range loading, boundary I/O, and KV
ownership, not Meshnet policy
### Package assembly
No code generation, downloads, or model execution were required for this story.
The package is documentation-only and deterministic.
```bash
python -m compileall -q packages tests
git diff --check
```
Result:
- both commands exited 0
## Correctness / performance / hardware classification
- Correctness evidence: research-only, no live model execution
- Performance evidence: none in this story
- Hardware evidence: none in this story
## Known limitations and deferred work
- DGR-010 remains blocked, so this package cannot be treated as the final
release-ready upstream handoff.
- The outreach draft is human-ready but not sent.
- The doc package does not change llama.cpp source code; it only prepares the
upstream ask and test mapping.
## Compatibility / migration notes
- Exact upstream pin for the eventual patch series: `b3c9d1b846cc80a6360adb6aeaa4fcd8c4c8dcac`
- The proposed patch split is:
1. range-aware loading and ownership introspection
2. boundary input/output and named tensor bundles
3. layer-filtered KV and local sequence ownership
- Meshnet routing, billing, relay transport, and volunteer-network policy stay
outside llama.cpp.
- The deterministic examples already exist in the tree and can be trimmed into
upstream-facing MREs when the human maintainer sends the package.
## Dependent-story handoff
- DGR-010 must clear before any real-model validation can be cited as the final
end-to-end proof for this upstream package.
- Once DGR-010 has a completed evidence README, the package can be refreshed
with the real-model context and sent to the llama.cpp maintainers as a
smaller review bundle.

90
.scratch/distributed-gguf-runtime/evidence/DGR-016/api-note.md
View File

@@ -1,90 +0,0 @@
# DGR-016 API note: narrow llama.cpp hooks, no Meshnet policy
This note is the upstream-facing shape for the collaboration package.
## Goal
Keep the llama.cpp ask small:
- expose generic model-layer hooks that are useful to any local or remote
layer-worker setup;
- keep Meshnet routing, session ownership, billing, and relay transport out of
llama.cpp;
- preserve one patch per concern so the series rebases cleanly on the pinned
upstream commit.
## Concern 1: range-aware loading and authoritative tensor ownership
Requested surface:
- accept a contiguous `[start_layer, end_layer)` range;
- expose whether the worker owns embeddings, final norm, and final head;
- make the loaded range authoritative from the model state, not from CLI
claims;
- allow unowned tensors to be absent rather than fabricated.
Why this is upstreamable:
- it is generic loader and introspection plumbing;
- it helps any local partitioned inference mode;
- it does not require any Meshnet identity, route, or transport type.
Minimal examples/tests:
- `tests/test_gguf_ownership.py`
- `tests/test_llama_worker_build.py`
## Concern 2: architecture boundary input/output
Requested surface:
- accept a versioned boundary bundle carrying one or more named tensors;
- support an unnormalized residual stream as the intermediate handoff;
- keep final norm, LM head, and sampling on the tail shard only;
- keep the bundle format explicit about name, shape, dtype, byte order, and
fragments.
Why this is upstreamable:
- it matches both dense Llama and other certified adapter families;
- it does not assume Meshnet or any specific wire protocol;
- it gives a stable ABI for a layer-worker boundary.
Minimal examples/tests:
- `tests/test_boundary_adapter.py`
- `tests/test_native_shard_protocol.py`
## Concern 3: layer-filtered KV and session mapping
Requested surface:
- let the worker own KV only for its layer range;
- map a stable session/context identifier to the local sequence;
- allow cache miss, stale epoch, truncate, release, and eviction semantics;
- reject incompatible cache recipes rather than trying to heal them silently.
Why this is upstreamable:
- it is a local sequence/KV API, not a network scheduler;
- it is useful to any supervisor that needs one process per layer range;
- it keeps session semantics outside llama.cpp while still making the worker
stateful in a controlled way.
Minimal examples/tests:
- `tests/test_hot_kv_state.py`
- `tests/test_kv_cache_distributed.py`
## Suggested patch split
Keep the series narrow and independently reviewable against the exact pinned
commit `b3c9d1b846cc80a6360adb6aeaa4fcd8c4c8dcac`:
1. `range-aware-loading` and ownership introspection.
2. `boundary-input-output` and named tensor bundle handoff.
3. `layer-filtered-kv` and sequence ownership.
The current Meshnet worker scaffold remains a project-owned wrapper and is not
part of the upstream ask.

43
.scratch/distributed-gguf-runtime/evidence/DGR-016/outreach.md
View File

@@ -1,43 +0,0 @@
# DGR-016 outreach draft
Subject: Narrow llama.cpp hooks for range loading, boundary I/O, and local KV ownership
Hi Georgi and llama.cpp maintainers,
We have been building a distributed GGUF route on top of a Meshnet control
plane, and the narrow upstreamable seam is now clear enough to summarize.
We are not asking llama.cpp to own Meshnet routing, billing, relay transport,
or any volunteer-network policy. The upstream ask is limited to generic local
hooks that make partitioned inference easier to implement and easier to review:
1. Range-aware loading and ownership introspection for contiguous layer ranges.
2. Architecture-defined boundary input/output using an explicit named-tensor
bundle.
3. Layer-filtered KV ownership and stable local sequence mapping.
Why we think this is generally useful:
- Nakshatra already demonstrates the value of a narrow layer-worker seam and
partial GGUF loading.
- prima.cpp shows the same idea from a different angle with selective loading,
local KV, and boundary residual transport.
- Both projects suggest the same conclusion: the missing API is not Meshnet
specific, it is a local runtime seam that any layer-partitioned supervisor can
use.
The package we would upstream is intentionally split into one concern per patch
so review stays small:
- range-aware loading and tensor ownership;
- boundary I/O for intermediate residual state;
- layer-filtered KV and sequence ownership.
If useful, we can send the concrete MRE/test mapping next. We already have
deterministic examples covering the loader, boundary contract, and KV/session
semantics in the Meshnet tree, and we can trim them into upstream-focused test
cases.
Thanks,
Meshnet maintainers

114
.scratch/distributed-gguf-runtime/evidence/DGR-017/README.md Normal file
View File

@@ -0,0 +1,114 @@
# DGR-017 evidence — superseded backlog cleanup
**Completed:** 2026-07-16
**Branch:** `ralph/distributed-gguf-runtime`
**Planning checkpoint before cleanup:** `81b1fa6`
**Authority:** `.scratch/distributed-gguf-runtime/prd.json`
## Outcome
The old DGR-001…016 completion claims and active artifacts were reconciled against the live branch. No old pass state transferred to the new implementation roadmap.
The active `packages/` and `tests/` trees were restored exactly to `origin/master`. The branch therefore no longer exposes a nominal GGUF startup path backed by unimplemented transport methods, a protobuf-only native scaffold, or isolated synthetic scheduler/cache/failure modules as if they were a working distributed GGUF runtime.
## Classification and disposition
### Retained
- Accepted ADRs and repository research, including `docs/research/colibri-implementation-audit.md`.
- The authoritative 55-story roadmap `DGR-017…071` and its generated issue specifications.
- The real public-relay smoke benchmark, moved with provenance to `legacy-public-relay-smoke-benchmark.json`.
- Git history containing the complete superseded implementation/reference work.
### Removed from the active tree
- Legacy issue specifications DGR-001…016 and their stale/blocked/synthetic evidence directories.
- The nonfunctional `gguf_backend` startup path whose gRPC execution methods raised not-implemented errors.
- Synthetic/reference-only boundary, Hot KV, scheduler, failure, recipe, ownership, and native-protocol modules that were not a real llama.cpp Shard runtime.
- The protobuf round-trip-only native scaffold, placeholder llama.cpp patch, generated bindings/build workspace, and associated tests.
- Tracker/admission/source modifications coupled to that superseded scaffold.
### Confirmed absent and still required
- Real standalone C++ gRPC Shard worker.
- Exact pinned llama.cpp manifest and verified patch stack.
- Range-aware GGUF tensor ownership and real ranged execution.
- Real Shard-local llama.cpp KV/V4 auxiliary state.
- DeepSeek V4 boundary adapter and ranged parity.
- Real multi-machine DeepSeek V4 alpha or beta acceptance.
These remain `passes: false` in DGR-018…071.
## Before-cleanup baseline
Command:
```bash
.venv-rocm/bin/python -m pytest -q \
tests/test_performance_contract.py tests/test_native_shard_protocol.py \
tests/test_gguf_ownership.py tests/test_boundary_adapter.py \
tests/test_hot_kv_state.py tests/test_gguf_backend.py \
tests/test_batch_scheduler.py tests/test_failure_semantics.py \
tests/test_llama_worker_build.py tests/test_node_admission.py \
tests/test_node_capability.py tests/test_tracker_capability_admission.py
```
Result:
```text
216 passed, 2 skipped, 1 failed, 1 warning
```
The failure was a synthetic capability-test helper `KeyError: 'compatibility_fingerprint'`. The warning was a pre-existing heartbeat-thread `SystemExit` warning.
## Cleanup verification
### Source equality
Command:
```bash
git diff --quiet origin/master -- packages tests
```
Result:
```text
packages_tests_match_origin_master=yes
```
The staged cleanup removes approximately 15.3k obsolete source/test/evidence lines from the active branch.
### Cleanup-relevant regression suite
Command:
```bash
.venv-rocm/bin/python -m pytest -q \
tests/test_node_admission.py tests/test_node_capability.py \
tests/test_tracker_capability_admission.py \
tests/test_kv_cache_distributed.py tests/test_real_distributed_inference.py
```
Result:
```text
119 passed, 2 skipped, 1 warning in 15.90s
```
The warning is the same pre-existing heartbeat-thread `SystemExit` warning.
### Known `origin/master` limitations
The wider routing run produced `210 passed, 2 skipped, 4 failed, 1 warning`. Each failure reproduced individually while `packages/` and `tests/` matched `origin/master` exactly:
- `test_tracker_models_endpoint_lists_registered_hf_repo_and_short_name_alias`
- `test_torch_node_applies_tracker_load_shard_directive`
- `test_shard_heal_cycle_surviving_node_covers_dead_peers_gap`
- `test_a_node_with_an_unusable_precision_covers_no_layers`
They are recorded as pre-existing baseline defects and were not repaired or hidden by this cleanup story.
## Dependency handoff
DGR-018 and later stories must start from the cleaned upstream-equivalent runtime tree. Reuse concepts from superseded commits only by explicitly porting the smallest verified slice under the new story’s contracts, tests, and evidence gates. Git history is provenance, not completion evidence.

0
.scratch/distributed-gguf-runtime/evidence/DGR-001/public-relay-smoke-benchmark.json → .scratch/distributed-gguf-runtime/evidence/DGR-017/legacy-public-relay-smoke-benchmark.json
View File

2
.scratch/distributed-gguf-runtime/evidence/README.md
View File

@@ -1,6 +1,6 @@
# Distributed GGUF Runtime evidence
> **Specification status:** planning artifacts only. No distributed GGUF runtime is implemented by this materialization, no story has completion credit, and legacy files remain for the DGR-017 audit. `prd.json` is authoritative.
> **Specification status:** planning artifacts only. No distributed GGUF runtime is implemented. DGR-017 cleanup is complete; no runtime implementation story has completion credit. `prd.json` is authoritative.
## Authority and classes

2
.scratch/distributed-gguf-runtime/implementation-strategy.md
View File

@@ -1,6 +1,6 @@
# Distributed GGUF Runtime implementation strategy
> **Specification status:** planning artifacts only. No distributed GGUF runtime is implemented by this materialization, no story has completion credit, and legacy files remain for the DGR-017 audit. `prd.json` is authoritative.
> **Specification status:** planning artifacts only. No distributed GGUF runtime is implemented. DGR-017 cleanup is complete; no runtime implementation story has completion credit. `prd.json` is authoritative.
## Execution model

68
.scratch/distributed-gguf-runtime/issues/01-lock-the-safetensors-versus-gguf-performance-contract.md
View File

@@ -1,68 +0,0 @@
# 01 — Lock the safetensors-versus-GGUF performance contract
Status: ready-for-agent
## Mandatory fresh-session context
- Read [RALPH-CONTEXT.md](../RALPH-CONTEXT.md) completely before changing code.
- This issue is `DGR-001` in [prd.json](../prd.json).
- Read the evidence README for every dependency listed below.
- Inspect current code and `git status`; historical text and previous agent claims are not evidence.
## Description
As a runtime engineer, I need a controlled baseline so that GGUF work proceeds from measured speed, memory, and quality rather than reputation.
## Baseline model target
Use the same model on both sides of the comparison, with the closest practical low-footprint precision pair:
- **safetensors:** `deepseek-ai/DeepSeek-V2-Lite-Chat` in **BF16**
- **GGUF:** `second-state/DeepSeek-V2-Lite-Chat-GGUF` in **Q2_K** (~6.5GB)
Keep the benchmark matrix explicit for **CPU** and **GPU** runs. Reserve smaller non-DeepSeek fallback models only for loader plumbing smoke tests if needed; they do not count as the DGR-001 architecture-aligned baseline.
## Expected durable outputs
- Benchmark harness and deterministic tests
- evidence/DGR-001/performance-contract.json
- Raw and summarized safetensors/GGUF benchmark evidence
## Acceptance criteria
- [ ] Benchmark the same model architecture/revision, machine, prompts, context lengths, output lengths, sampling policy, and concurrency across the current Transformers/safetensors recipe and whole-model llama.cpp recipes.
- [ ] Separate correctness/quality lanes from quantized performance/fit lanes instead of claiming BF16 and Q4 are numerically equivalent.
- [ ] Report TTFT, prefill tok/s, decode tok/s, p50/p95 latency, aggregate throughput, RSS, VRAM, artifact size, failures, and output drift in machine-readable JSON.
- [ ] Add concurrency levels 1 and 4 where memory permits.
- [ ] Write a versioned performance contract consumed by later release gates, including an explicit stop condition when llama.cpp/GGUF has no meaningful speed or fit benefit.
- [ ] Targeted pytest tests pass
- [ ] python -m compileall packages tests passes for Python changes
- [ ] git diff --check passes
- [ ] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
- [ ] Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
- [ ] Real-model execution is opt-in through MESHNET_ENABLE_REAL_INFERENCE_TESTS=1 and records exact artifact/runtime/hardware evidence
- [ ] Model artifacts remain on the configured mounted-drive storage and never under /home
- [ ] Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
- [ ] Read and verify every dependency evidence README before relying on dependency behavior
- [ ] Preserve all pre-existing working-tree changes and stage only files belonging to this story
- [ ] Write .scratch/distributed-gguf-runtime/evidence/DGR-001/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
- [ ] Update only this story issue to Status: done after every acceptance criterion and quality gate passes
## Dependency handoff
- None. This story may start immediately.
## Finish contract
- Create the task evidence directory and durable handoff required above.
- Preserve real failures and blockers; never fabricate benchmark, model, test or hardware output.
- Change this issue to `Status: done` only after all criteria pass.
- Emit `<promise>COMPLETE</promise>` only after the evidence handoff exists.
## References
- [Ralph execution context](../RALPH-CONTEXT.md)
- [PRD](../PRD.md)
- [Implementation strategy](../implementation-strategy.md)
- [Current architecture](../architecture.md)
- [Architecture decision](../../docs/adr/0024-distributed-gguf-runtime.md)

14
.scratch/distributed-gguf-runtime/issues/017-reconcile-and-clean-the-superseded-dgr-backlog.md
View File

@@ -1,7 +1,7 @@
<!-- GENERATED FROM prd.json — DO NOT EDIT AS AN INDEPENDENT SOURCE. prd.json IS AUTHORITATIVE. -->
# DGR-017: Reconcile and clean the superseded DGR backlog
- **Status / triage:** specification only; `ready-for-agent`; `passes: false`
- **Status / triage:** completed; `passes: true`
- **Execution mode:** `AFK`
- **Milestone:** `M0`
- **Dependencies:** None
@@ -18,11 +18,11 @@ Fresh Ralph session: read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md`,
## Acceptance criteria
- [ ] Compare the branch, old DGR-001..016 issue/pass states, evidence, and actual runtime sources; classify each output as reusable, reference-only, blocked, obsolete, or absent.
- [ ] Record an authoritative old-to-new disposition and provenance; explicitly give no completion credit to any new story and note absent implementation/evidence.
- [ ] Remove or archive only artifacts the audit proves obsolete while preserving accepted ADRs, useful research, raw benchmark evidence, and attributable reusable work.
- [ ] Protect ignored build workspaces, generated protobuf outputs, Ralph logs, and model artifacts from accidental commits, and document every retained legacy artifact.
- [ ] Applicable shared quality gates in `prd.json` pass, and the evidence handoff records exact commands/results, changed files, limitations, and dependency handoff.
- [x] Compare the branch, old DGR-001..016 issue/pass states, evidence, and actual runtime sources; classify each output as reusable, reference-only, blocked, obsolete, or absent.
- [x] Record an authoritative old-to-new disposition and provenance; explicitly give no completion credit to any new story and note absent implementation/evidence.
- [x] Remove or archive only artifacts the audit proves obsolete while preserving accepted ADRs, useful research, raw benchmark evidence, and attributable reusable work.
- [x] Protect ignored build workspaces, generated protobuf outputs, Ralph logs, and model artifacts from accidental commits, and document every retained legacy artifact.
- [x] Applicable shared quality gates in `prd.json` pass, and the evidence handoff records exact commands/results, changed files, limitations, and dependency handoff.
## Shared quality gates
@@ -36,4 +36,4 @@ Fresh Ralph session: read `.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md`,
## Evidence handoff
Write and verify `.scratch/distributed-gguf-runtime/evidence/DGR-017/README.md`. Until every criterion and applicable gate has real evidence, this story remains `passes: false`. Legacy evidence is provenance only, not completion credit.
Verified evidence: `.scratch/distributed-gguf-runtime/evidence/DGR-017/README.md`. Legacy evidence remains provenance only and grants no implementation completion credit.

59
.scratch/distributed-gguf-runtime/issues/02-adopt-the-versioned-grpc-shard-protocol.md
View File

@@ -1,59 +0,0 @@
# 02 — Adopt the versioned gRPC Shard protocol
Status: done
## Mandatory fresh-session context
- Read [RALPH-CONTEXT.md](../RALPH-CONTEXT.md) completely before changing code.
- This issue is `DGR-002` in [prd.json](../prd.json).
- Read the evidence README for every dependency listed below.
- Inspect current code and `git status`; historical text and previous agent claims are not evidence.
## Description
As a node developer, I need a battle-proven streaming protocol so that Python and C++ Shards communicate without a custom socket protocol.
## Expected durable outputs
- packages/node/native/proto/shard_runtime.proto
- Reproducible Python/C++ schema generation and build wiring
- Protocol round-trip and compatibility tests
- evidence/DGR-002/README.md
## Acceptance criteria
- [x] Add a Protocol Buffers schema for capability, health, session stream, release, and cancellation operations.
- [x] Define one long-lived bidirectional gRPC stream per Route Session Activation Seam with deadlines, cancellation, flow control, and structured errors.
- [x] Define bounded chunking for prefill and a small decode fast path.
- [x] Carry schema version, request/work ID, Route Session ID, route epoch, artifact/recipe fingerprint, Shard range/effective start, phase, position, idempotency step, cache expectation, compression, and checksum.
- [x] Define a versioned named-tensor bundle with per-tensor name, shape, dtype, byte order, and payload fragments.
- [x] Add generated-schema round-trip and compatibility tests in Python and C++.
- [x] Targeted pytest tests pass
- [x] python -m compileall packages tests passes for Python changes
- [x] git diff --check passes
- [x] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
- [x] Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
- [x] Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
- [x] Read and verify every dependency evidence README before relying on dependency behavior
- [x] Preserve all pre-existing working-tree changes and stage only files belonging to this story
- [x] Write .scratch/distributed-gguf-runtime/evidence/DGR-002/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
- [x] Update only this story issue to Status: done after every acceptance criterion and quality gate passes
## Dependency handoff
- None. This story may start immediately.
## Finish contract
- Create the task evidence directory and durable handoff required above.
- Preserve real failures and blockers; never fabricate benchmark, model, test or hardware output.
- Change this issue to `Status: done` only after all criteria pass.
- Emit `<promise>COMPLETE</promise>` only after the evidence handoff exists.
## References
- [Ralph execution context](../RALPH-CONTEXT.md)
- [PRD](../PRD.md)
- [Implementation strategy](../implementation-strategy.md)
- [Current architecture](../architecture.md)
- [Architecture decision](../../docs/adr/0024-distributed-gguf-runtime.md)

57
.scratch/distributed-gguf-runtime/issues/03-define-exact-artifact-and-runtime-recipe-identity.md
View File

@@ -1,57 +0,0 @@
# 03 — Define exact Artifact and runtime recipe identity
Status: done
## Mandatory fresh-session context
- Read [RALPH-CONTEXT.md](../RALPH-CONTEXT.md) completely before changing code.
- This issue is `DGR-003` in [prd.json](../prd.json).
- Read the evidence README for every dependency listed below.
- Inspect current code and `git status`; historical text and previous agent claims are not evidence.
## Description
As the Tracker, I need exact compatibility identity so that only numerically and operationally compatible Shards form an Inference Route.
## Expected durable outputs
- Exact runtime recipe/fingerprint implementation
- Tracker/node fail-closed admission tests
- evidence/DGR-003/README.md
## Acceptance criteria
- [ ] Separate weight quantization, activation dtype, compute dtype, KV dtype/layout, tokenizer revision, architecture adapter, backend, and runtime version.
- [ ] Bind derivative or split artifacts to an exact source Model Artifact hash and Shard range.
- [ ] Produce a stable compatibility fingerprint used by capability admission and the gRPC handshake.
- [ ] Fail closed on mismatched artifact, tokenizer, architecture, range, boundary schema, activation recipe, or cache layout.
- [ ] Keep unsupported recipes registered-but-dark until a real distributed forward certifies them.
- [ ] Targeted pytest tests pass
- [ ] python -m compileall packages tests passes for Python changes
- [ ] git diff --check passes
- [ ] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
- [ ] Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
- [ ] Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
- [ ] Read and verify every dependency evidence README before relying on dependency behavior
- [ ] Preserve all pre-existing working-tree changes and stage only files belonging to this story
- [ ] Write .scratch/distributed-gguf-runtime/evidence/DGR-003/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
- [ ] Update only this story issue to Status: done after every acceptance criterion and quality gate passes
## Dependency handoff
- `DGR-002` must have `passes: true`; read `../evidence/DGR-002/README.md` and verify its referenced files/commands.
## Finish contract
- Create the task evidence directory and durable handoff required above.
- Preserve real failures and blockers; never fabricate benchmark, model, test or hardware output.
- Change this issue to `Status: done` only after all criteria pass.
- Emit `<promise>COMPLETE</promise>` only after the evidence handoff exists.
## References
- [Ralph execution context](../RALPH-CONTEXT.md)
- [PRD](../PRD.md)
- [Implementation strategy](../implementation-strategy.md)
- [Current architecture](../architecture.md)
- [Architecture decision](../../docs/adr/0024-distributed-gguf-runtime.md)

61
.scratch/distributed-gguf-runtime/issues/04-create-the-reproducible-pinned-llama-cpp-patch-stack.md
View File

@@ -1,61 +0,0 @@
# 04 — Create the reproducible pinned llama.cpp patch stack
Status: done
## Mandatory fresh-session context
- Read [RALPH-CONTEXT.md](../RALPH-CONTEXT.md) completely before changing code.
- This issue is `DGR-004` in [prd.json](../prd.json).
- Read the evidence README for every dependency listed below.
- Inspect current code and `git status`; historical text and previous agent claims are not evidence.
## Description
As a maintainer, I need a small auditable fork boundary so that upstream updates do not turn the runtime into an unmaintainable stitched codebase.
## Expected durable outputs
- Exact llama.cpp upstream pin
- Numbered minimal patch stack
- Reproducible fetch/apply/build smoke
- evidence/DGR-004/README.md
## Acceptance criteria
- [ ] Pin one exact llama.cpp commit through a reproducible source dependency mechanism.
- [ ] Store a numbered minimal patch stack separately from Meshnet networking code.
- [ ] Add a build script that applies/checks patches and builds the standalone worker without manual source copying.
- [ ] Record upstream file/ABI assumptions and fail clearly when the pin changes.
- [ ] Preserve upstream license and attribution notices.
- [ ] Add a clean rebuild smoke test that does not download a model.
- [ ] Targeted pytest tests pass
- [ ] python -m compileall packages tests passes for Python changes
- [ ] git diff --check passes
- [ ] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
- [ ] Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
- [ ] Pinned native C++ target builds and focused CTest/protocol tests pass where native code is touched
- [ ] llama.cpp patch stack applies cleanly to the exact pinned commit where patch code is touched
- [ ] Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
- [ ] Read and verify every dependency evidence README before relying on dependency behavior
- [ ] Preserve all pre-existing working-tree changes and stage only files belonging to this story
- [ ] Write .scratch/distributed-gguf-runtime/evidence/DGR-004/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
- [ ] Update only this story issue to Status: done after every acceptance criterion and quality gate passes
## Dependency handoff
- `DGR-001` must have `passes: true`; read `../evidence/DGR-001/README.md` and verify its referenced files/commands.
## Finish contract
- Create the task evidence directory and durable handoff required above.
- Preserve real failures and blockers; never fabricate benchmark, model, test or hardware output.
- Change this issue to `Status: done` only after all criteria pass.
- Emit `<promise>COMPLETE</promise>` only after the evidence handoff exists.
## References
- [Ralph execution context](../RALPH-CONTEXT.md)
- [PRD](../PRD.md)
- [Implementation strategy](../implementation-strategy.md)
- [Current architecture](../architecture.md)
- [Architecture decision](../../docs/adr/0024-distributed-gguf-runtime.md)

61
.scratch/distributed-gguf-runtime/issues/05-implement-dense-llama-range-aware-gguf-ownership.md
View File

@@ -1,61 +0,0 @@
# 05 — Implement dense-Llama range-aware GGUF ownership
Status: done
## Mandatory fresh-session context
- Read [RALPH-CONTEXT.md](../RALPH-CONTEXT.md) completely before changing code.
- This issue is `DGR-005` in [prd.json](../prd.json).
- Read the evidence README for every dependency listed below.
- Inspect current code and `git status`; historical text and previous agent claims are not evidence.
## Description
As a node, I need to map only my assigned dense-Llama Shard so that aggregate consumer memory can hold a model larger than one node.
## Expected durable outputs
- Dense-Llama range-aware ownership implementation
- Authoritative loaded-range introspection
- Mapped/resident memory evidence
- evidence/DGR-005/README.md
## Acceptance criteria
- [ ] Register and allocate only `blk.N.*` tensors in the assigned range.
- [ ] Load embeddings only for the head and final norm/LM head only for the tail, including tied embeddings.
- [ ] Prefer range-aware mapping from one exact source GGUF; if derivative sub-GGUFs are used temporarily, verify source/slice hashes and avoid claiming final artifact semantics.
- [ ] Report authoritative loaded range and endpoint ownership from the model, not operator CLI claims.
- [ ] Demonstrate mapped/resident memory scales with owned tensors rather than full model size.
- [ ] Targeted pytest tests pass
- [ ] python -m compileall packages tests passes for Python changes
- [ ] git diff --check passes
- [ ] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
- [ ] Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
- [ ] Pinned native C++ target builds and focused CTest/protocol tests pass where native code is touched
- [ ] llama.cpp patch stack applies cleanly to the exact pinned commit where patch code is touched
- [ ] Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
- [ ] Read and verify every dependency evidence README before relying on dependency behavior
- [ ] Preserve all pre-existing working-tree changes and stage only files belonging to this story
- [ ] Write .scratch/distributed-gguf-runtime/evidence/DGR-005/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
- [ ] Update only this story issue to Status: done after every acceptance criterion and quality gate passes
## Dependency handoff
- `DGR-003` must have `passes: true`; read `../evidence/DGR-003/README.md` and verify its referenced files/commands.
- `DGR-004` must have `passes: true`; read `../evidence/DGR-004/README.md` and verify its referenced files/commands.
## Finish contract
- Create the task evidence directory and durable handoff required above.
- Preserve real failures and blockers; never fabricate benchmark, model, test or hardware output.
- Change this issue to `Status: done` only after all criteria pass.
- Emit `<promise>COMPLETE</promise>` only after the evidence handoff exists.
## References
- [Ralph execution context](../RALPH-CONTEXT.md)
- [PRD](../PRD.md)
- [Implementation strategy](../implementation-strategy.md)
- [Current architecture](../architecture.md)
- [Architecture decision](../../docs/adr/0024-distributed-gguf-runtime.md)

61
.scratch/distributed-gguf-runtime/issues/06-implement-architecture-defined-boundary-input-output.md
View File

@@ -1,61 +0,0 @@
# 06 — Implement architecture-defined boundary input/output
Status: done
## Mandatory fresh-session context
- Read [RALPH-CONTEXT.md](../RALPH-CONTEXT.md) completely before changing code.
- This issue is `DGR-006` in [prd.json](../prd.json).
- Read the evidence README for every dependency listed below.
- Inspect current code and `git status`; historical text and previous agent claims are not evidence.
## Description
As a Shard, I need to consume and emit the correct transformer boundary state so that disjoint processes reproduce whole-model execution.
## Expected durable outputs
- Architecture boundary adapter
- Whole-model/two-range parity tests and results
- evidence/DGR-006/README.md
## Acceptance criteria
- [ ] Head accepts token IDs and owns token embedding.
- [ ] Middle/tail bypass token embedding and accept the named boundary bundle.
- [ ] Non-tail emits the unnormalized architecture-defined residual/boundary before final norm/head and before tail-only row pruning.
- [ ] Tail emits logits or token output through an explicit sampling contract.
- [ ] Dense-Llama whole-model versus two-range prefill and greedy-decode parity passes the documented tolerance.
- [ ] The adapter interface fails closed for uncertified architectures.
- [ ] Targeted pytest tests pass
- [ ] python -m compileall packages tests passes for Python changes
- [ ] git diff --check passes
- [ ] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
- [ ] Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
- [ ] Pinned native C++ target builds and focused CTest/protocol tests pass where native code is touched
- [ ] llama.cpp patch stack applies cleanly to the exact pinned commit where patch code is touched
- [ ] Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
- [ ] Read and verify every dependency evidence README before relying on dependency behavior
- [ ] Preserve all pre-existing working-tree changes and stage only files belonging to this story
- [ ] Write .scratch/distributed-gguf-runtime/evidence/DGR-006/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
- [ ] Update only this story issue to Status: done after every acceptance criterion and quality gate passes
## Dependency handoff
- `DGR-002` must have `passes: true`; read `../evidence/DGR-002/README.md` and verify its referenced files/commands.
- `DGR-005` must have `passes: true`; read `../evidence/DGR-005/README.md` and verify its referenced files/commands.
## Finish contract
- Create the task evidence directory and durable handoff required above.
- Preserve real failures and blockers; never fabricate benchmark, model, test or hardware output.
- Change this issue to `Status: done` only after all criteria pass.
- Emit `<promise>COMPLETE</promise>` only after the evidence handoff exists.
## References
- [Ralph execution context](../RALPH-CONTEXT.md)
- [PRD](../PRD.md)
- [Implementation strategy](../implementation-strategy.md)
- [Current architecture](../architecture.md)
- [Architecture decision](../../docs/adr/0024-distributed-gguf-runtime.md)

60
.scratch/distributed-gguf-runtime/issues/07-add-isolated-concurrent-local-hot-kv-state.md
View File

@@ -1,60 +0,0 @@
# 07 — Add isolated concurrent local Hot KV State
Status: done
## Mandatory fresh-session context
- Read [RALPH-CONTEXT.md](../RALPH-CONTEXT.md) completely before changing code.
- This issue is `DGR-007` in [prd.json](../prd.json).
- Read the evidence README for every dependency listed below.
- Inspect current code and `git status`; historical text and previous agent claims are not evidence.
## Description
As a client, I need concurrent Route Sessions to retain independent per-Shard cache so that one request cannot clear or corrupt another.
## Expected durable outputs
- Concurrent local KV/session manager
- Isolation, eviction, cancellation and cleanup tests
- evidence/DGR-007/README.md
## Acceptance criteria
- [ ] Map `(Route Session ID, route epoch)` to an isolated llama sequence or bounded context.
- [ ] Allocate KV only for owned layers.
- [ ] Support prefill append, decode append, truncate, release, TTL/LRU eviction, and explicit cache-miss response.
- [ ] Reject stale epochs and incompatible cache recipes.
- [ ] At least four concurrent sessions on a small model complete without token or KV cross-talk.
- [ ] Cancellation/release of one session leaves other sessions intact and memory returns to the configured budget.
- [ ] Targeted pytest tests pass
- [ ] python -m compileall packages tests passes for Python changes
- [ ] git diff --check passes
- [ ] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
- [ ] Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
- [ ] Pinned native C++ target builds and focused CTest/protocol tests pass where native code is touched
- [ ] llama.cpp patch stack applies cleanly to the exact pinned commit where patch code is touched
- [ ] Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
- [ ] Read and verify every dependency evidence README before relying on dependency behavior
- [ ] Preserve all pre-existing working-tree changes and stage only files belonging to this story
- [ ] Write .scratch/distributed-gguf-runtime/evidence/DGR-007/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
- [ ] Update only this story issue to Status: done after every acceptance criterion and quality gate passes
## Dependency handoff
- `DGR-006` must have `passes: true`; read `../evidence/DGR-006/README.md` and verify its referenced files/commands.
## Finish contract
- Create the task evidence directory and durable handoff required above.
- Preserve real failures and blockers; never fabricate benchmark, model, test or hardware output.
- Change this issue to `Status: done` only after all criteria pass.
- Emit `<promise>COMPLETE</promise>` only after the evidence handoff exists.
## References
- [Ralph execution context](../RALPH-CONTEXT.md)
- [PRD](../PRD.md)
- [Implementation strategy](../implementation-strategy.md)
- [Current architecture](../architecture.md)
- [Architecture decision](../../docs/adr/0024-distributed-gguf-runtime.md)

65
.scratch/distributed-gguf-runtime/issues/08-build-the-standalone-c-grpc-shard-worker.md
View File

@@ -1,65 +0,0 @@
# 08 — Build the standalone C++ gRPC Shard worker
Status: ready-for-agent
## Mandatory fresh-session context
- Read [RALPH-CONTEXT.md](../RALPH-CONTEXT.md) completely before changing code.
- This issue is `DGR-008` in [prd.json](../prd.json).
- Read the evidence README for every dependency listed below.
- Inspect current code and `git status`; historical text and previous agent claims are not evidence.
## Description
As a node runtime, I need one supervised native process so that llama.cpp internals remain behind a stable project-owned protocol.
## Expected durable outputs
- Standalone C++ gRPC worker
- Fake-model Python/C++ integration tests
- Lifecycle and bounded-failure evidence
- evidence/DGR-008/README.md
## Acceptance criteria
- [ ] Worker exposes capability, health, session stream, release, cancellation, and metrics services from DGR-002.
- [ ] Worker loads one exact Artifact/recipe/Shard identity and refuses mismatched requests.
- [ ] Streaming path enforces bounded messages, flow control, deadlines, idempotency, and independent session cancellation.
- [ ] Worker does not expose raw llama.cpp RPC or arbitrary GGML graph execution.
- [ ] Graceful shutdown releases sessions; crash behavior is bounded and observable.
- [ ] Python integration tests run against a fake model mode without model downloads.
- [ ] Targeted pytest tests pass
- [ ] python -m compileall packages tests passes for Python changes
- [ ] git diff --check passes
- [ ] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
- [ ] Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
- [ ] Pinned native C++ target builds and focused CTest/protocol tests pass where native code is touched
- [ ] llama.cpp patch stack applies cleanly to the exact pinned commit where patch code is touched
- [ ] Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
- [ ] Read and verify every dependency evidence README before relying on dependency behavior
- [ ] Preserve all pre-existing working-tree changes and stage only files belonging to this story
- [ ] Write .scratch/distributed-gguf-runtime/evidence/DGR-008/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
- [ ] Update only this story issue to Status: done after every acceptance criterion and quality gate passes
## Dependency handoff
- `DGR-002` must have `passes: true`; read `../evidence/DGR-002/README.md` and verify its referenced files/commands.
- `DGR-003` must have `passes: true`; read `../evidence/DGR-003/README.md` and verify its referenced files/commands.
- `DGR-004` must have `passes: true`; read `../evidence/DGR-004/README.md` and verify its referenced files/commands.
- `DGR-006` must have `passes: true`; read `../evidence/DGR-006/README.md` and verify its referenced files/commands.
- `DGR-007` must have `passes: true`; read `../evidence/DGR-007/README.md` and verify its referenced files/commands.
## Finish contract
- Create the task evidence directory and durable handoff required above.
- Preserve real failures and blockers; never fabricate benchmark, model, test or hardware output.
- Change this issue to `Status: done` only after all criteria pass.
- Emit `<promise>COMPLETE</promise>` only after the evidence handoff exists.
## References
- [Ralph execution context](../RALPH-CONTEXT.md)
- [PRD](../PRD.md)
- [Implementation strategy](../implementation-strategy.md)
- [Current architecture](../architecture.md)
- [Architecture decision](../../docs/adr/0024-distributed-gguf-runtime.md)

61
.scratch/distributed-gguf-runtime/issues/09-integrate-the-native-worker-with-meshnet.md
View File

@@ -1,61 +0,0 @@
# 09 — Integrate the native worker with Meshnet
Status: done
## Mandatory fresh-session context
- Read [RALPH-CONTEXT.md](../RALPH-CONTEXT.md) completely before changing code.
- This issue is `DGR-009` in [prd.json](../prd.json).
- Read the evidence README for every dependency listed below.
- Inspect current code and `git status`; historical text and previous agent claims are not evidence.
## Description
As the existing node service, I need a GGUF Shard backend adapter so that the Tracker, relay, billing, telemetry, and capability admission remain the sole control plane.
## Expected durable outputs
- Meshnet GGUF backend adapter
- Registration, routing, relay, telemetry and billing tests
- evidence/DGR-009/README.md
## Acceptance criteria
- [ ] Implement the existing model-backend surface without changing Transformers behavior.
- [ ] Registration carries exact validated GGUF recipe, Shard, backend and concurrency/KV capacity.
- [ ] Tracker forms only complete compatible routes and keeps uncertified recipes dark.
- [ ] Direct routes use gRPC streams; relayed routes carry the same versioned protobuf frames as opaque binary through the existing relay seam.
- [ ] Existing request/work IDs, cancellation, Generation Telemetry, billing, and per-node attribution remain correlated.
- [ ] No vLLM, Nakshatra, prima.cpp, or custom-engine control plane becomes a core dependency.
- [ ] Targeted pytest tests pass
- [ ] python -m compileall packages tests passes for Python changes
- [ ] git diff --check passes
- [ ] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
- [ ] Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
- [ ] Pinned native C++ target builds and focused CTest/protocol tests pass where native code is touched
- [ ] llama.cpp patch stack applies cleanly to the exact pinned commit where patch code is touched
- [ ] Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
- [ ] Read and verify every dependency evidence README before relying on dependency behavior
- [ ] Preserve all pre-existing working-tree changes and stage only files belonging to this story
- [ ] Write .scratch/distributed-gguf-runtime/evidence/DGR-009/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
- [ ] Update only this story issue to Status: done after every acceptance criterion and quality gate passes
## Dependency handoff
- `DGR-003` must have `passes: true`; read `../evidence/DGR-003/README.md` and verify its referenced files/commands.
- `DGR-008` must have `passes: true`; read `../evidence/DGR-008/README.md` and verify its referenced files/commands.
## Finish contract
- Create the task evidence directory and durable handoff required above.
- Preserve real failures and blockers; never fabricate benchmark, model, test or hardware output.
- Change this issue to `Status: done` only after all criteria pass.
- Emit `<promise>COMPLETE</promise>` only after the evidence handoff exists.
## References
- [Ralph execution context](../RALPH-CONTEXT.md)
- [PRD](../PRD.md)
- [Implementation strategy](../implementation-strategy.md)
- [Current architecture](../architecture.md)
- [Architecture decision](../../docs/adr/0024-distributed-gguf-runtime.md)

62
.scratch/distributed-gguf-runtime/issues/10-pass-local-real-model-two-process-acceptance.md
View File

@@ -1,62 +0,0 @@
# 10 — Pass local real-model two-process acceptance
Status: ready-for-agent
## Mandatory fresh-session context
- Read [RALPH-CONTEXT.md](../RALPH-CONTEXT.md) completely before changing code.
- This issue is `DGR-010` in [prd.json](../prd.json).
- Read the evidence README for every dependency listed below.
- Inspect current code and `git status`; historical text and previous agent claims are not evidence.
## Description
As a release engineer, I need real local distributed parity before involving network variability.
## Expected durable outputs
- Real local two-process commands and configuration
- Raw parity, memory and performance results
- evidence/DGR-010/README.md
## Acceptance criteria
- [ ] Two local worker processes open disjoint dense-Llama ranges from the certified Artifact.
- [ ] Prefill and at least 32 greedy decode tokens match whole-model llama.cpp within the certified tolerance.
- [ ] Each worker retains only its own tensors and Hot KV State.
- [ ] Four concurrent Route Sessions pass isolation and cleanup checks.
- [ ] Report TTFT, prefill/decode throughput, seam bytes/latency, worker RSS/VRAM, KV memory, batch size, and queue time.
- [ ] Killing one worker produces a bounded structured failure rather than a deadlock.
- [ ] Targeted pytest tests pass
- [ ] python -m compileall packages tests passes for Python changes
- [ ] git diff --check passes
- [ ] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
- [ ] Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
- [ ] Pinned native C++ target builds and focused CTest/protocol tests pass where native code is touched
- [ ] llama.cpp patch stack applies cleanly to the exact pinned commit where patch code is touched
- [ ] Real-model execution is opt-in through MESHNET_ENABLE_REAL_INFERENCE_TESTS=1 and records exact artifact/runtime/hardware evidence
- [ ] Model artifacts remain on the configured mounted-drive storage and never under /home
- [ ] Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
- [ ] Read and verify every dependency evidence README before relying on dependency behavior
- [ ] Preserve all pre-existing working-tree changes and stage only files belonging to this story
- [ ] Write .scratch/distributed-gguf-runtime/evidence/DGR-010/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
- [ ] Update only this story issue to Status: done after every acceptance criterion and quality gate passes
## Dependency handoff
- `DGR-009` must have `passes: true`; read `../evidence/DGR-009/README.md` and verify its referenced files/commands.
## Finish contract
- Create the task evidence directory and durable handoff required above.
- Preserve real failures and blockers; never fabricate benchmark, model, test or hardware output.
- Change this issue to `Status: done` only after all criteria pass.
- Emit `<promise>COMPLETE</promise>` only after the evidence handoff exists.
## References
- [Ralph execution context](../RALPH-CONTEXT.md)
- [PRD](../PRD.md)
- [Implementation strategy](../implementation-strategy.md)
- [Current architecture](../architecture.md)
- [Architecture decision](../../docs/adr/0024-distributed-gguf-runtime.md)

62
.scratch/distributed-gguf-runtime/issues/11-pass-a-real-heterogeneous-two-machine-route.md
View File

@@ -1,62 +0,0 @@
# 11 — Pass a real heterogeneous two-machine route
Status: ready-for-agent
## Mandatory fresh-session context
- Read [RALPH-CONTEXT.md](../RALPH-CONTEXT.md) completely before changing code.
- This issue is `DGR-011` in [prd.json](../prd.json).
- Read the evidence README for every dependency listed below.
- Inspect current code and `git status`; historical text and previous agent claims are not evidence.
## Description
As a consumer-hardware operator, I need two physical machines to execute one GGUF model so that the distributed claim is real.
## Expected durable outputs
- Two-machine hardware/network/runtime manifest
- Raw real-route metrics and output evidence
- evidence/DGR-011/README.md
## Acceptance criteria
- [ ] Tracker selects two physical nodes with disjoint Shards and one exact certified recipe/compatibility class.
- [ ] Actual CPU/GPU execution occurs on both nodes; synthetic workers do not satisfy acceptance.
- [ ] Prefill/decode, concurrent-session isolation, telemetry, cancellation, and cleanup pass over the real transport/relay path.
- [ ] Exact hardware, network, backend, model hash, route, commands, and raw metrics are recorded.
- [ ] A model or recipe larger than one participating node's admitted memory is exercised when available.
- [ ] Output drift is measured and incompatible mixed backends fail closed.
- [ ] Targeted pytest tests pass
- [ ] python -m compileall packages tests passes for Python changes
- [ ] git diff --check passes
- [ ] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
- [ ] Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
- [ ] Pinned native C++ target builds and focused CTest/protocol tests pass where native code is touched
- [ ] llama.cpp patch stack applies cleanly to the exact pinned commit where patch code is touched
- [ ] Real-model execution is opt-in through MESHNET_ENABLE_REAL_INFERENCE_TESTS=1 and records exact artifact/runtime/hardware evidence
- [ ] Model artifacts remain on the configured mounted-drive storage and never under /home
- [ ] Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
- [ ] Read and verify every dependency evidence README before relying on dependency behavior
- [ ] Preserve all pre-existing working-tree changes and stage only files belonging to this story
- [ ] Write .scratch/distributed-gguf-runtime/evidence/DGR-011/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
- [ ] Update only this story issue to Status: done after every acceptance criterion and quality gate passes
## Dependency handoff
- `DGR-010` must have `passes: true`; read `../evidence/DGR-010/README.md` and verify its referenced files/commands.
## Finish contract
- Create the task evidence directory and durable handoff required above.
- Preserve real failures and blockers; never fabricate benchmark, model, test or hardware output.
- Change this issue to `Status: done` only after all criteria pass.
- Emit `<promise>COMPLETE</promise>` only after the evidence handoff exists.
## References
- [Ralph execution context](../RALPH-CONTEXT.md)
- [PRD](../PRD.md)
- [Implementation strategy](../implementation-strategy.md)
- [Current architecture](../architecture.md)
- [Architecture decision](../../docs/adr/0024-distributed-gguf-runtime.md)

63
.scratch/distributed-gguf-runtime/issues/12-implement-continuous-batching-and-bounded-admission.md
View File

@@ -1,63 +0,0 @@
# 12 — Implement continuous batching and bounded admission
Status: ready-for-agent
## Mandatory fresh-session context
- Read [RALPH-CONTEXT.md](../RALPH-CONTEXT.md) completely before changing code.
- This issue is `DGR-012` in [prd.json](../prd.json).
- Read the evidence README for every dependency listed below.
- Inspect current code and `git status`; historical text and previous agent claims are not evidence.
## Description
As a node operator, I need active sessions batched safely so that concurrency increases aggregate throughput rather than serializing every request.
## Expected durable outputs
- Continuous batching/admission scheduler
- Concurrency 1/2/4/8 report
- Queue, batch and KV-pressure evidence
- evidence/DGR-012/README.md
## Acceptance criteria
- [ ] Node scheduler admits sessions against weight, KV, scratch, and queue budgets.
- [ ] Compatible decode steps from multiple sessions form llama.cpp batches while preserving per-session positions and outputs.
- [ ] Prefill does not starve decode; scheduling policy and bounds are explicit.
- [ ] Backpressure prevents unbounded queued activations or KV growth.
- [ ] Capability telemetry reports active sessions, queue depth, batch occupancy, KV pressure, prefill/decode rates, and rejected admissions.
- [ ] Concurrency 1/2/4/8 benchmark identifies saturation and shows no cross-session corruption.
- [ ] Targeted pytest tests pass
- [ ] python -m compileall packages tests passes for Python changes
- [ ] git diff --check passes
- [ ] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
- [ ] Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
- [ ] Pinned native C++ target builds and focused CTest/protocol tests pass where native code is touched
- [ ] llama.cpp patch stack applies cleanly to the exact pinned commit where patch code is touched
- [ ] Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
- [ ] Read and verify every dependency evidence README before relying on dependency behavior
- [ ] Preserve all pre-existing working-tree changes and stage only files belonging to this story
- [ ] Write .scratch/distributed-gguf-runtime/evidence/DGR-012/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
- [ ] Update only this story issue to Status: done after every acceptance criterion and quality gate passes
## Dependency handoff
- `DGR-007` must have `passes: true`; read `../evidence/DGR-007/README.md` and verify its referenced files/commands.
- `DGR-009` must have `passes: true`; read `../evidence/DGR-009/README.md` and verify its referenced files/commands.
- `DGR-010` must have `passes: true`; read `../evidence/DGR-010/README.md` and verify its referenced files/commands.
## Finish contract
- Create the task evidence directory and durable handoff required above.
- Preserve real failures and blockers; never fabricate benchmark, model, test or hardware output.
- Change this issue to `Status: done` only after all criteria pass.
- Emit `<promise>COMPLETE</promise>` only after the evidence handoff exists.
## References
- [Ralph execution context](../RALPH-CONTEXT.md)
- [PRD](../PRD.md)
- [Implementation strategy](../implementation-strategy.md)
- [Current architecture](../architecture.md)
- [Architecture decision](../../docs/adr/0024-distributed-gguf-runtime.md)

62
.scratch/distributed-gguf-runtime/issues/13-harden-failure-cancellation-and-restart-semantics.md
View File

@@ -1,62 +0,0 @@
# 13 — Harden failure, cancellation, and restart semantics
Status: done
## Mandatory fresh-session context
- Read [RALPH-CONTEXT.md](../RALPH-CONTEXT.md) completely before changing code.
- This issue is `DGR-013` in [prd.json](../prd.json).
- Read the evidence README for every dependency listed below.
- Inspect current code and `git status`; historical text and previous agent claims are not evidence.
## Description
As a client, I need failures to be bounded and explicit so that distributed speed does not come with hanging or corrupted generations.
## Expected durable outputs
- Failure/cancel/restart test matrix
- Resource cleanup and billing-state evidence
- evidence/DGR-013/README.md
## Acceptance criteria
- [ ] Deadlines and heartbeat/health loss terminate blocked stream operations.
- [ ] Cancellation propagates across every Shard and releases local KV and queued buffers.
- [ ] Duplicate steps are idempotent; uncertain mutations are never replayed silently.
- [ ] Alpha failover restarts from token zero on a newly compatible route rather than importing unverified KV.
- [ ] Worker death, stream reset, malformed bundle, stale epoch, and cache miss tests pass.
- [ ] Billing/work records distinguish completed, cancelled, failed, and unverified work.
- [ ] Targeted pytest tests pass
- [ ] python -m compileall packages tests passes for Python changes
- [ ] git diff --check passes
- [ ] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
- [ ] Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
- [ ] Pinned native C++ target builds and focused CTest/protocol tests pass where native code is touched
- [ ] llama.cpp patch stack applies cleanly to the exact pinned commit where patch code is touched
- [ ] Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
- [ ] Read and verify every dependency evidence README before relying on dependency behavior
- [ ] Preserve all pre-existing working-tree changes and stage only files belonging to this story
- [ ] Write .scratch/distributed-gguf-runtime/evidence/DGR-013/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
- [ ] Update only this story issue to Status: done after every acceptance criterion and quality gate passes
## Dependency handoff
- `DGR-008` must have `passes: true`; read `../evidence/DGR-008/README.md` and verify its referenced files/commands.
- `DGR-009` must have `passes: true`; read `../evidence/DGR-009/README.md` and verify its referenced files/commands.
- `DGR-012` must have `passes: true`; read `../evidence/DGR-012/README.md` and verify its referenced files/commands.
## Finish contract
- Create the task evidence directory and durable handoff required above.
- Preserve real failures and blockers; never fabricate benchmark, model, test or hardware output.
- Change this issue to `Status: done` only after all criteria pass.
- Emit `<promise>COMPLETE</promise>` only after the evidence handoff exists.
## References
- [Ralph execution context](../RALPH-CONTEXT.md)
- [PRD](../PRD.md)
- [Implementation strategy](../implementation-strategy.md)
- [Current architecture](../architecture.md)
- [Architecture decision](../../docs/adr/0024-distributed-gguf-runtime.md)

65
.scratch/distributed-gguf-runtime/issues/14-enforce-the-gguf-versus-safetensors-release-gate.md
View File

@@ -1,65 +0,0 @@
# 14 — Enforce the GGUF-versus-safetensors release gate
Status: ready-for-agent
## Mandatory fresh-session context
- Read [RALPH-CONTEXT.md](../RALPH-CONTEXT.md) completely before changing code.
- This issue is `DGR-014` in [prd.json](../prd.json).
- Read the evidence README for every dependency listed below.
- Inspect current code and `git status`; historical text and previous agent claims are not evidence.
## Description
As the product owner, I need an end-to-end comparison so that the native runtime ships only if it advances model access or performance.
## Expected durable outputs
- Immutable comparison against DGR-001 thresholds
- Machine-readable final report
- Ship/optimize/stop recommendation
- evidence/DGR-014/README.md
## Acceptance criteria
- [ ] Run current distributed safetensors and distributed GGUF routes on the same certified model/hardware/network scenario where technically comparable.
- [ ] Report quality, TTFT, prefill/decode throughput, aggregate concurrency throughput, p95 latency, seam cost, memory, KV pressure, failures, and cleanup.
- [ ] Evaluate against the DGR-001 performance contract without changing thresholds after seeing results.
- [ ] Ship recommendation is one of: promote GGUF, optimize a measured bottleneck with a new bounded task, or stop the native track.
- [ ] Results clearly separate quantization gains from transport/runtime gains.
- [ ] Targeted pytest tests pass
- [ ] python -m compileall packages tests passes for Python changes
- [ ] git diff --check passes
- [ ] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
- [ ] Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
- [ ] Pinned native C++ target builds and focused CTest/protocol tests pass where native code is touched
- [ ] llama.cpp patch stack applies cleanly to the exact pinned commit where patch code is touched
- [ ] Real-model execution is opt-in through MESHNET_ENABLE_REAL_INFERENCE_TESTS=1 and records exact artifact/runtime/hardware evidence
- [ ] Model artifacts remain on the configured mounted-drive storage and never under /home
- [ ] Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
- [ ] Read and verify every dependency evidence README before relying on dependency behavior
- [ ] Preserve all pre-existing working-tree changes and stage only files belonging to this story
- [ ] Write .scratch/distributed-gguf-runtime/evidence/DGR-014/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
- [ ] Update only this story issue to Status: done after every acceptance criterion and quality gate passes
## Dependency handoff
- `DGR-001` must have `passes: true`; read `../evidence/DGR-001/README.md` and verify its referenced files/commands.
- `DGR-011` must have `passes: true`; read `../evidence/DGR-011/README.md` and verify its referenced files/commands.
- `DGR-012` must have `passes: true`; read `../evidence/DGR-012/README.md` and verify its referenced files/commands.
- `DGR-013` must have `passes: true`; read `../evidence/DGR-013/README.md` and verify its referenced files/commands.
## Finish contract
- Create the task evidence directory and durable handoff required above.
- Preserve real failures and blockers; never fabricate benchmark, model, test or hardware output.
- Change this issue to `Status: done` only after all criteria pass.
- Emit `<promise>COMPLETE</promise>` only after the evidence handoff exists.
## References
- [Ralph execution context](../RALPH-CONTEXT.md)
- [PRD](../PRD.md)
- [Implementation strategy](../implementation-strategy.md)
- [Current architecture](../architecture.md)
- [Architecture decision](../../docs/adr/0024-distributed-gguf-runtime.md)

61
.scratch/distributed-gguf-runtime/issues/15-add-and-certify-a-qwen3-qwen3-moe-adapter.md
View File

@@ -1,61 +0,0 @@
# 15 — Add and certify a Qwen3/Qwen3-MoE adapter
Status: ready-for-agent
## Mandatory fresh-session context
- Read [RALPH-CONTEXT.md](../RALPH-CONTEXT.md) completely before changing code.
- This issue is `DGR-015` in [prd.json](../prd.json).
- Read the evidence README for every dependency listed below.
- Inspect current code and `git status`; historical text and previous agent claims are not evidence.
## Description
As a client seeking top models, I need a separately certified MoE-capable architecture after the dense runtime proves stable.
## Expected durable outputs
- Qwen3-family architecture adapter
- Architecture-specific parity/admission/performance results
- evidence/DGR-015/README.md
## Acceptance criteria
- [ ] Implement explicit tensor ownership, router/top-k, expert/shared-expert, Q/K normalization, boundary bundle, and cache semantics for the selected Qwen3 family recipe.
- [ ] Do not reuse the dense-Llama adapter through unchecked name substitutions.
- [ ] Whole-model versus distributed prefill/decode parity passes the architecture-specific tolerance.
- [ ] Expert memory ownership and communication are measured.
- [ ] Real consumer-hardware acceptance and capability admission pass before the recipe becomes routable.
- [ ] Targeted pytest tests pass
- [ ] python -m compileall packages tests passes for Python changes
- [ ] git diff --check passes
- [ ] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
- [ ] Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
- [ ] Pinned native C++ target builds and focused CTest/protocol tests pass where native code is touched
- [ ] llama.cpp patch stack applies cleanly to the exact pinned commit where patch code is touched
- [ ] Real-model execution is opt-in through MESHNET_ENABLE_REAL_INFERENCE_TESTS=1 and records exact artifact/runtime/hardware evidence
- [ ] Model artifacts remain on the configured mounted-drive storage and never under /home
- [ ] Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
- [ ] Read and verify every dependency evidence README before relying on dependency behavior
- [ ] Preserve all pre-existing working-tree changes and stage only files belonging to this story
- [ ] Write .scratch/distributed-gguf-runtime/evidence/DGR-015/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
- [ ] Update only this story issue to Status: done after every acceptance criterion and quality gate passes
## Dependency handoff
- `DGR-014` must have `passes: true`; read `../evidence/DGR-014/README.md` and verify its referenced files/commands.
## Finish contract
- Create the task evidence directory and durable handoff required above.
- Preserve real failures and blockers; never fabricate benchmark, model, test or hardware output.
- Change this issue to `Status: done` only after all criteria pass.
- Emit `<promise>COMPLETE</promise>` only after the evidence handoff exists.
## References
- [Ralph execution context](../RALPH-CONTEXT.md)
- [PRD](../PRD.md)
- [Implementation strategy](../implementation-strategy.md)
- [Current architecture](../architecture.md)
- [Architecture decision](../../docs/adr/0024-distributed-gguf-runtime.md)

60
.scratch/distributed-gguf-runtime/issues/16-produce-the-upstream-llama-cpp-collaboration-package.md
View File

@@ -1,60 +0,0 @@
# 16 — Produce the upstream llama.cpp collaboration package
Status: ready-for-agent
## Mandatory fresh-session context
- Read [RALPH-CONTEXT.md](../RALPH-CONTEXT.md) completely before changing code.
- This issue is `DGR-016` in [prd.json](../prd.json).
- Read the evidence README for every dependency listed below.
- Inspect current code and `git status`; historical text and previous agent claims are not evidence.
## Description
As a maintainer, I need narrow upstreamable proposals so that our patch burden can shrink without asking llama.cpp to own Meshnet networking.
## Expected durable outputs
- Narrow upstream patches/tests
- Generic API design note
- Human-ready llama.cpp outreach package
- evidence/DGR-016/README.md
## Acceptance criteria
- [ ] Separate generic llama.cpp hooks from Meshnet protocol/control-plane code.
- [ ] Prepare minimal reproducible examples and tests for range-aware loading, boundary input/output, and layer-filtered KV.
- [ ] Compare the proposal with Nakshatra and prima.cpp evidence and explain why the API is generally useful.
- [ ] Preserve one scoped commit/patch per concern against the exact upstream pin.
- [ ] Produce an outreach document suitable for Georgi/llama.cpp maintainers; actual sending remains a human action.
- [ ] Targeted pytest tests pass
- [ ] python -m compileall packages tests passes for Python changes
- [ ] git diff --check passes
- [ ] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
- [ ] Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
- [ ] Pinned native C++ target builds and focused CTest/protocol tests pass where native code is touched
- [ ] llama.cpp patch stack applies cleanly to the exact pinned commit where patch code is touched
- [ ] Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
- [ ] Read and verify every dependency evidence README before relying on dependency behavior
- [ ] Preserve all pre-existing working-tree changes and stage only files belonging to this story
- [ ] Write .scratch/distributed-gguf-runtime/evidence/DGR-016/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
- [ ] Update only this story issue to Status: done after every acceptance criterion and quality gate passes
## Dependency handoff
- `DGR-010` must have `passes: true`; read `../evidence/DGR-010/README.md` and verify its referenced files/commands.
## Finish contract
- Create the task evidence directory and durable handoff required above.
- Preserve real failures and blockers; never fabricate benchmark, model, test or hardware output.
- Change this issue to `Status: done` only after all criteria pass.
- Emit `<promise>COMPLETE</promise>` only after the evidence handoff exists.
## References
- [Ralph execution context](../RALPH-CONTEXT.md)
- [PRD](../PRD.md)
- [Implementation strategy](../implementation-strategy.md)
- [Current architecture](../architecture.md)
- [Architecture decision](../../docs/adr/0024-distributed-gguf-runtime.md)

2
.scratch/distributed-gguf-runtime/milestones.md
View File

@@ -1,6 +1,6 @@
# Distributed GGUF Runtime milestones
> **Specification status:** planning artifacts only. No distributed GGUF runtime is implemented by this materialization, no story has completion credit, and legacy files remain for the DGR-017 audit. `prd.json` is authoritative.
> **Specification status:** planning artifacts only. No distributed GGUF runtime is implemented. DGR-017 cleanup is complete; no runtime implementation story has completion credit. `prd.json` is authoritative.
## M0 — Truth and contracts

3
.scratch/distributed-gguf-runtime/prd.json
View File

@@ -271,7 +271,8 @@
"Protect ignored build workspaces, generated protobuf outputs, Ralph logs, and model artifacts from accidental commits, and document every retained legacy artifact.",
"Applicable shared quality gates in `prd.json` pass, and the evidence handoff records exact commands/results, changed files, limitations, and dependency handoff."
],
"passes": false,
"passes": true,
"completionNotes": "Completed 2026-07-16. Superseded DGR-001..016 issues/evidence and nonfunctional synthetic runtime scaffolding were removed from the active tree; packages/ and tests/ were restored exactly to origin/master. Accepted research and the real public-relay smoke benchmark were retained with provenance. See evidence/DGR-017/README.md.",
"notes": "Generated source issue: .scratch/distributed-gguf-runtime/issues/017-reconcile-and-clean-the-superseded-dgr-backlog.md; prd.json is authoritative.",
"blocks": [
"DGR-018",

2
.scratch/distributed-gguf-runtime/technical-challenges.md
View File

@@ -1,6 +1,6 @@
# Distributed GGUF Runtime technical challenges
> **Specification status:** planning artifacts only. No distributed GGUF runtime is implemented by this materialization, no story has completion credit, and legacy files remain for the DGR-017 audit. `prd.json` is authoritative.
> **Specification status:** planning artifacts only. No distributed GGUF runtime is implemented. DGR-017 cleanup is complete; no runtime implementation story has completion credit. `prd.json` is authoritative.
## Key challenges and planned answers

179
packages/node/meshnet_node/admission.py
View File

@@ -20,17 +20,9 @@ import time
from dataclasses import dataclass
from typing import Any, Callable
from . import __version__ as _PACKAGE_VERSION
from .capability import CapabilityReport, config_fingerprint
from .capability import CapabilityReport
from .doctor import DoctorSelection
from .recipe_manifest import Recipe, RecipeManifest
from .runtime_recipe import (
build_artifact_identity,
build_runtime_recipe_identity,
compatibility_fingerprint,
fingerprint_payload,
)
from .gguf_ownership import authoritative_dense_llama_ownership
# How long a passing report stays usable. Startup normally validates in-process
# (age ≈ 0); this bounds how far a report written by an earlier `doctor` run can
@@ -47,7 +39,6 @@ REASON_MODEL_MISMATCH = "model-mismatch"
REASON_SHARD_MISMATCH = "shard-mismatch"
REASON_RECIPE_MISMATCH = "recipe-mismatch"
REASON_BACKEND_MISMATCH = "backend-mismatch"
REASON_COMPATIBILITY_MISMATCH = "compatibility-mismatch"
class CapabilityAdmissionError(RuntimeError):
@@ -86,7 +77,6 @@ class AdmissionRequirement:
recipe_version: str
backend_id: str
device: str
compatibility_fingerprint: str
max_age_seconds: float = DEFAULT_MAX_REPORT_AGE_SECONDS
@classmethod
@@ -104,9 +94,6 @@ class AdmissionRequirement:
recipe_version=context.recipe.version,
backend_id=context.recipe.backend_id,
device=context.device,
compatibility_fingerprint=_compatibility_fingerprint_for_context(
context
),
max_age_seconds=max_age_seconds,
)
@@ -178,16 +165,6 @@ def admit(
f"{requirement.backend_id} on {requirement.device}",
)
if report.compatibility_fingerprint != requirement.compatibility_fingerprint:
raise CapabilityAdmissionError(
REASON_COMPATIBILITY_MISMATCH,
f"capability proof fingerprint {report.compatibility_fingerprint!r} "
f"does not match the expected compatibility fingerprint for "
f"{requirement.model_id} {requirement.shard_label}; the artifact, "
f"tokenizer, architecture, boundary schema, activation recipe or "
f"cache layout differs",
)
if not report.passed:
raise CapabilityAdmissionError(
REASON_NOT_PASSED,
@@ -246,157 +223,3 @@ def probe_capability(context: CapabilityContext) -> CapabilityReport:
context.recipe,
context.manifest,
).report
def _compatibility_fingerprint_for_context(context: CapabilityContext) -> str:
backend = context.backend
selection = context.selection
recipe = context.recipe
model_config = getattr(getattr(backend, "model", None), "config", None)
model_config_payload = (
model_config.to_dict() if hasattr(model_config, "to_dict") else model_config
)
runtime_versions = _runtime_versions()
runtime_version = _PACKAGE_VERSION
ownership = authoritative_dense_llama_ownership(backend, selection)
artifact = build_artifact_identity(
model_id=selection.model_id,
revision=getattr(getattr(backend, "model", None), "revision", None),
model_config=model_config_payload,
shard_start=ownership.start_layer,
shard_end=ownership.end_layer,
)
runtime_recipe = build_runtime_recipe_identity(
model_id=selection.model_id,
revision=getattr(getattr(backend, "model", None), "revision", None),
model_config=model_config_payload,
recipe_params=recipe.params,
weight_quantization=selection.quantization,
backend_id=recipe.backend_id,
runtime_version=runtime_version,
activation_dtype="bfloat16",
compute_dtype=_backend_compute_dtype(backend),
kv_dtype=_backend_kv_dtype(backend),
kv_layout=_backend_kv_layout(backend),
tokenizer_revision=_backend_tokenizer_revision(backend, selection),
architecture_adapter=_backend_architecture_adapter(backend, recipe.backend_id),
boundary_schema_version=1,
cache_layout=_backend_cache_layout(backend, recipe.params),
)
return compatibility_fingerprint(
fingerprint_payload(
model={
"model_id": selection.model_id,
"revision": getattr(getattr(backend, "model", None), "revision", None),
"config_fingerprint": config_fingerprint(model_config_payload),
},
shard={
"start": ownership.start_layer,
"end": ownership.end_layer,
"owns_embedding": ownership.owns_embedding,
"owns_final_head": ownership.owns_final_head,
},
recipe={
"recipe_id": recipe.id,
"recipe_version": recipe.version,
"catalogue_version": context.manifest.catalogue_version,
},
backend={
"backend_id": recipe.backend_id,
"device": context.device,
"device_name": _backend_device_name(context.device),
"quantization": selection.quantization,
"runtime": runtime_versions,
},
artifact=artifact.to_dict(),
runtime_recipe=runtime_recipe.to_dict(),
)
)
def _runtime_versions() -> dict[str, str]:
versions: dict[str, str] = {}
for name in ("torch", "transformers"):
try:
module = __import__(name)
except Exception:
continue
version = getattr(module, "__version__", None)
if version:
versions[name] = str(version)
return versions
def _backend_compute_dtype(backend: Any) -> str:
config = getattr(getattr(backend, "model", None), "config", None)
for candidate in (config, getattr(config, "text_config", None)):
if candidate is None:
continue
for attr in ("dtype", "torch_dtype"):
value = getattr(candidate, attr, None)
if value is None:
continue
return str(value).removeprefix("torch.")
return "bfloat16"
def _backend_kv_dtype(backend: Any) -> str:
return _backend_compute_dtype(backend)
def _backend_kv_layout(backend: Any) -> str:
return "session-cache" if getattr(backend, "supports_kv_cache", False) else "stateless"
def _backend_tokenizer_revision(backend: Any, selection: DoctorSelection) -> str:
model = getattr(backend, "model", None)
revision = getattr(model, "revision", None)
if isinstance(revision, str) and revision.strip():
return revision
tokenizer = getattr(backend, "tokenizer", None)
for attr in ("revision", "model_id"):
value = getattr(tokenizer, attr, None)
if isinstance(value, str) and value.strip():
return value
return selection.model_id
def _backend_architecture_adapter(backend: Any, default: str) -> str:
config = getattr(getattr(backend, "model", None), "config", None)
for candidate in (config, getattr(config, "text_config", None)):
if candidate is None:
continue
for attr in ("architecture_adapter", "model_type"):
value = getattr(candidate, attr, None)
if isinstance(value, str) and value.strip():
return value
architectures = getattr(candidate, "architectures", None)
if isinstance(architectures, (list, tuple)) and architectures:
first = architectures[0]
if isinstance(first, str) and first.strip():
return first
return default
def _backend_device_name(device: str) -> str | None:
if device != "cuda":
return None
from .hardware import detect_hardware
try:
return detect_hardware().get("gpu_name") or None
except Exception:
return None
def _backend_cache_layout(backend: Any, recipe_params: dict[str, Any] | None) -> str:
if getattr(backend, "supports_kv_cache", False) is False:
return "stateless"
if recipe_params is None:
return "local-hot-kv"
if recipe_params.get("use_cache") is False:
return "stateless"
value = recipe_params.get("cache_layout")
if isinstance(value, str) and value.strip():
return value
return "local-hot-kv"

1024
packages/node/meshnet_node/batch_scheduler.py
View File

File diff suppressed because it is too large Load Diff

484
packages/node/meshnet_node/boundary_adapter.py
View File

@@ -1,484 +0,0 @@
"""Architecture-defined boundary input/output for distributed Shards (DGR-006).
A public-network Shard is a contiguous range of transformer layers (RALPH runtime
decision #1). For disjoint processes to reproduce whole-model execution, every
Shard must agree on *exactly* what boundary state it consumes and emits:
* The **head** owns token embedding: it accepts token IDs and turns them into the
residual stream. No other Shard may embed tokens.
* **Middle and tail** Shards bypass token embedding entirely; they accept the named
boundary bundle (the residual stream handed over by the previous range).
* A **non-tail** Shard emits the *unnormalized* architecture-defined residual /
boundary — before the final norm, before the LM head, and before any tail-only
row pruning — so the next range sees precisely the state the whole model would
have at that layer index.
* The **tail** owns the final norm + LM head and turns the residual into logits or
a sampled token through an explicit sampling contract.
This module is deliberately backend-agnostic. It enforces the boundary *contract*
and defers the arithmetic to a ``ShardComputation`` (a duck-typed object exposing
``embed_tokens`` / ``run_layers`` / ``final_norm`` / ``lm_head``). The pinned
llama.cpp worker (DGR-008) and the reference PyTorch backend both satisfy that
protocol, and the numpy reference model in the tests proves whole-model versus
two-range parity without any download, GPU, or API credit.
The adapter **fails closed** for uncertified architectures: only architectures
that have passed real certification (dense Llama-family first, per RALPH runtime
decision #13) are accepted. Everything else raises rather than silently guessing a
tensor layout — Qwen3/Qwen3-MoE stays registered-but-dark until DGR-015 certifies
its own adapter.
"""
from __future__ import annotations
from dataclasses import dataclass, field
from enum import Enum
from typing import Any
import numpy as np
# The boundary bundle wire schema version. This is the ``boundary_schema_version``
# carried by ``runtime_recipe.RuntimeRecipeIdentity``; a receiver refuses a bundle
# whose schema it does not implement (forward/backward compatibility is a routing
# concern, not a silent reinterpretation).
BOUNDARY_SCHEMA_VERSION = 1
class BoundaryAdapterError(RuntimeError):
"""Base class for boundary-contract violations."""
class UncertifiedArchitectureError(BoundaryAdapterError):
"""Raised when a boundary adapter is requested for an uncertified architecture.
Failing closed here is a safety property: an unknown architecture has an
unknown tensor layout, so guessing where the residual boundary lives would
silently corrupt distributed output. The architecture must pass real
certification first.
"""
class BoundaryContractError(BoundaryAdapterError):
"""Raised when a Shard is fed the wrong boundary input for its role.
Examples: a head handed a residual bundle instead of token IDs, a middle
Shard handed token IDs it must not embed, or a boundary bundle whose
architecture / schema / seam layer does not match the receiving range.
"""
@dataclass(frozen=True)
class ArchitectureBoundary:
"""The architecture-defined boundary description for one certified adapter.
These fields are what makes the boundary *architecture-defined* rather than a
hardcoded assumption: the residual tensor name, whether the tail normalizes
before the LM head, and whether row pruning is a tail-only concern all come
from here.
"""
adapter: str
boundary_tensor_name: str
boundary_schema_version: int
normalizes_before_head: bool
prunes_rows_at_tail: bool
# Certified architectures only. Dense Llama-family is first (RALPH runtime decision
# #13 + native discipline). Aliases map the many spellings a runtime recipe /
# GGUF / HF config may use onto the single canonical adapter id. Anything not in
# this table fails closed.
_DENSE_LLAMA = ArchitectureBoundary(
adapter="dense-llama",
boundary_tensor_name="residual_stream",
boundary_schema_version=BOUNDARY_SCHEMA_VERSION,
normalizes_before_head=True,
prunes_rows_at_tail=True,
)
_CERTIFIED_ARCHITECTURES: dict[str, ArchitectureBoundary] = {
"dense-llama": _DENSE_LLAMA,
"dense_llama": _DENSE_LLAMA,
"llama": _DENSE_LLAMA,
"llamaforcausallm": _DENSE_LLAMA,
"llamamodel": _DENSE_LLAMA,
}
def certified_architecture(name: Any) -> ArchitectureBoundary:
"""Return the certified boundary description for ``name`` or fail closed.
``name`` may be the canonical adapter id (``dense-llama``), an HF architecture
class (``LlamaForCausalLM``), or a GGUF/config ``model_type`` (``llama``).
Uncertified architectures raise ``UncertifiedArchitectureError``.
"""
if not isinstance(name, str) or not name.strip():
raise UncertifiedArchitectureError(
"architecture adapter must be a non-empty string; "
"the boundary adapter refuses to guess a tensor layout"
)
key = name.strip().lower()
boundary = _CERTIFIED_ARCHITECTURES.get(key)
if boundary is None:
raise UncertifiedArchitectureError(
f"architecture {name!r} is not certified for the boundary adapter; "
f"certified adapters: {sorted(set(v.adapter for v in _CERTIFIED_ARCHITECTURES.values()))}. "
"Uncertified architectures stay registered-but-dark until real "
"certification passes."
)
return boundary
def is_certified_architecture(name: Any) -> bool:
"""Return True when ``name`` maps to a certified boundary adapter."""
try:
certified_architecture(name)
except UncertifiedArchitectureError:
return False
return True
class ShardRole(str, Enum):
"""Where a contiguous layer range sits in the whole model."""
HEAD = "head"
MIDDLE = "middle"
TAIL = "tail"
FULL = "full"
@property
def owns_embedding(self) -> bool:
return self in (ShardRole.HEAD, ShardRole.FULL)
@property
def owns_final_head(self) -> bool:
return self in (ShardRole.TAIL, ShardRole.FULL)
def role_for_range(start_layer: int, end_layer: int, total_layers: int) -> ShardRole:
"""Classify a contiguous inclusive layer range within a model of ``total_layers``."""
if total_layers <= 0:
raise ValueError("total_layers must be positive")
if start_layer < 0 or end_layer < start_layer:
raise ValueError("require 0 <= start_layer <= end_layer")
if end_layer > total_layers - 1:
raise ValueError(
f"end_layer {end_layer} exceeds last layer index {total_layers - 1}"
)
is_head = start_layer == 0
is_tail = end_layer >= total_layers - 1
if is_head and is_tail:
return ShardRole.FULL
if is_head:
return ShardRole.HEAD
if is_tail:
return ShardRole.TAIL
return ShardRole.MIDDLE
@dataclass(frozen=True)
class BoundaryBundle:
"""The versioned named-tensor bundle handed between adjacent Shard ranges.
``residual`` is the *unnormalized* architecture-defined residual stream with
every position row intact (no tail-only pruning). ``next_layer`` is the layer
index the receiving range must start at — it is the overlap-safe effective
start of the seam, so a receiver can reject a bundle meant for a different cut.
"""
architecture_adapter: str
schema_version: int
tensor_name: str
residual: np.ndarray
positions: np.ndarray
next_layer: int
normalized: bool = False
def named_tensor_fields(self) -> dict[str, Any]:
"""Return the wire-shaped description of the residual tensor.
These are exactly the fields the DGR-002 ``NamedTensor`` carries (name,
shape, dtype, byte order, raw bytes), so a worker can serialize this
bundle into the gRPC protobuf without re-deriving them.
"""
residual = np.ascontiguousarray(self.residual)
return {
"name": self.tensor_name,
"shape": list(residual.shape),
"dtype": residual.dtype.name,
"byte_order": _byte_order(residual.dtype),
"data": residual.tobytes(),
}
def pack(self) -> dict[str, Any]:
"""Serialize the bundle to a transport-agnostic dict (proves the seam).
The residual and positions are carried as raw little/big-endian bytes plus
shape/dtype so that a truly disjoint process can reconstruct the exact
array — this is what lets two OS processes reproduce whole-model math.
"""
residual = np.ascontiguousarray(self.residual)
positions = np.ascontiguousarray(self.positions)
return {
"architecture_adapter": self.architecture_adapter,
"schema_version": self.schema_version,
"tensor_name": self.tensor_name,
"next_layer": self.next_layer,
"normalized": self.normalized,
"residual": {
"shape": list(residual.shape),
"dtype": residual.dtype.str,
"data": residual.tobytes(),
},
"positions": {
"shape": list(positions.shape),
"dtype": positions.dtype.str,
"data": positions.tobytes(),
},
}
@classmethod
def unpack(cls, payload: dict[str, Any]) -> "BoundaryBundle":
"""Reconstruct a bundle produced by :meth:`pack`."""
residual = _array_from_wire(payload["residual"])
positions = _array_from_wire(payload["positions"])
return cls(
architecture_adapter=payload["architecture_adapter"],
schema_version=int(payload["schema_version"]),
tensor_name=payload["tensor_name"],
residual=residual,
positions=positions,
next_layer=int(payload["next_layer"]),
normalized=bool(payload.get("normalized", False)),
)
@dataclass(frozen=True)
class SamplingContract:
"""Explicit contract for turning tail logits into a token.
The tail never hides the sampling decision inside the adapter: the contract is
a first-class value so the head/route can reproduce it and so greedy decoding
is deterministic by construction. Only greedy is certified here; temperature /
top-p are declared but must be requested explicitly and are out of scope for
the deterministic parity gate.
"""
mode: str = "greedy"
temperature: float = 1.0
top_p: float = 1.0
def __post_init__(self) -> None:
if self.mode not in ("greedy",):
raise BoundaryContractError(
f"sampling mode {self.mode!r} is not certified; only 'greedy' is "
"deterministic and supported by the boundary adapter today"
)
@classmethod
def greedy(cls) -> "SamplingContract":
return cls(mode="greedy")
def sample(self, last_logits: np.ndarray) -> int:
"""Return the next token id from the final-position logits row."""
logits = np.asarray(last_logits)
if logits.ndim == 2:
# (batch, vocab) — parity harness uses batch size 1.
logits = logits[0]
if logits.ndim != 1:
raise BoundaryContractError(
"sampling expects the pruned final-position logits row"
)
return int(np.argmax(logits))
@dataclass(frozen=True)
class TailOutput:
"""What a tail Shard emits: the sampled token plus the pruned logits row."""
token_id: int
logits: np.ndarray
sampling: SamplingContract
@dataclass
class BoundaryAdapter:
"""Enforces the architecture-defined boundary contract for one Shard range.
Construction fails closed for uncertified architectures. The adapter derives
the Shard's role from its range and drives a duck-typed ``ShardComputation``.
"""
computation: Any
sampling: SamplingContract = field(default_factory=SamplingContract.greedy)
architecture: ArchitectureBoundary = field(init=False)
role: ShardRole = field(init=False)
start_layer: int = field(init=False)
end_layer: int = field(init=False)
total_layers: int = field(init=False)
def __post_init__(self) -> None:
arch_name = getattr(self.computation, "architecture_adapter", None)
self.architecture = certified_architecture(arch_name)
self.start_layer = int(getattr(self.computation, "start_layer"))
self.end_layer = int(getattr(self.computation, "end_layer"))
self.total_layers = int(getattr(self.computation, "total_layers"))
self.role = role_for_range(
self.start_layer, self.end_layer, self.total_layers
)
@property
def is_head(self) -> bool:
return self.role.owns_embedding
@property
def is_tail(self) -> bool:
return self.role.owns_final_head
def forward(
self,
*,
token_ids: Any | None = None,
boundary: BoundaryBundle | None = None,
) -> BoundaryBundle | TailOutput:
"""Run one prefill/decode pass for this range and emit its boundary output.
Head/full ranges require ``token_ids``; middle/tail ranges require the
``boundary`` bundle. Non-tail ranges return a :class:`BoundaryBundle`;
tail/full ranges return a :class:`TailOutput` through the sampling
contract.
"""
hidden, positions = self._ingest(token_ids, boundary)
hidden = self.computation.run_layers(hidden, positions=positions)
if self.is_tail:
return self._emit_tail(hidden)
return self._emit_boundary(hidden, positions)
# -- input side -----------------------------------------------------------
def _ingest(
self, token_ids: Any | None, boundary: BoundaryBundle | None
) -> tuple[np.ndarray, np.ndarray]:
if self.role.owns_embedding:
return self._ingest_tokens(token_ids, boundary)
return self._ingest_boundary(token_ids, boundary)
def _ingest_tokens(
self, token_ids: Any | None, boundary: BoundaryBundle | None
) -> tuple[np.ndarray, np.ndarray]:
if token_ids is None:
raise BoundaryContractError(
"the head owns token embedding and must receive token IDs"
)
if boundary is not None:
raise BoundaryContractError(
"the head owns token embedding; it must not receive a boundary "
"bundle from an upstream range"
)
ids = np.asarray(token_ids)
if ids.ndim == 1:
ids = ids[None, :]
if ids.ndim != 2:
raise BoundaryContractError("token IDs must be (seq,) or (batch, seq)")
hidden = np.asarray(self.computation.embed_tokens(ids))
positions = np.broadcast_to(
np.arange(ids.shape[1], dtype=np.int64), ids.shape
).copy()
return hidden, positions
def _ingest_boundary(
self, token_ids: Any | None, boundary: BoundaryBundle | None
) -> tuple[np.ndarray, np.ndarray]:
if token_ids is not None:
raise BoundaryContractError(
"middle/tail Shards bypass token embedding; they must not receive "
"token IDs"
)
if boundary is None:
raise BoundaryContractError(
"middle/tail Shards must receive the named boundary bundle"
)
self._check_boundary(boundary)
return np.asarray(boundary.residual), np.asarray(boundary.positions)
def _check_boundary(self, boundary: BoundaryBundle) -> None:
if certified_architecture(boundary.architecture_adapter) is not self.architecture:
raise BoundaryContractError(
f"boundary bundle architecture {boundary.architecture_adapter!r} "
f"does not match this Shard's adapter {self.architecture.adapter!r}"
)
if boundary.schema_version != self.architecture.boundary_schema_version:
raise BoundaryContractError(
f"boundary schema v{boundary.schema_version} is not supported by "
f"this Shard (expects v{self.architecture.boundary_schema_version})"
)
if boundary.tensor_name != self.architecture.boundary_tensor_name:
raise BoundaryContractError(
f"boundary tensor {boundary.tensor_name!r} is not the "
f"architecture-defined {self.architecture.boundary_tensor_name!r}"
)
if boundary.normalized:
raise BoundaryContractError(
"boundary bundle is normalized; a Shard range must receive the "
"UNNORMALIZED architecture-defined residual"
)
if boundary.next_layer != self.start_layer:
raise BoundaryContractError(
f"boundary hands over at layer {boundary.next_layer} but this "
f"Shard starts at layer {self.start_layer}"
)
# -- output side ----------------------------------------------------------
def _emit_boundary(
self, hidden: np.ndarray, positions: np.ndarray
) -> BoundaryBundle:
# A non-tail Shard emits the unnormalized residual with every position row
# intact: no final norm, no LM head, no tail-only row pruning. next_layer
# is the receiver's overlap-safe effective start.
return BoundaryBundle(
architecture_adapter=self.architecture.adapter,
schema_version=self.architecture.boundary_schema_version,
tensor_name=self.architecture.boundary_tensor_name,
residual=np.asarray(hidden),
positions=np.asarray(positions),
next_layer=self.end_layer + 1,
normalized=False,
)
def _emit_tail(self, hidden: np.ndarray) -> TailOutput:
hidden = np.asarray(hidden)
# Tail-only row pruning: only the final position is needed to sample the
# next token, so the LM head runs on the pruned row. A non-tail Shard is
# forbidden from doing this (it must forward every row).
if self.architecture.prunes_rows_at_tail:
last_hidden = hidden[:, -1:, :]
else: # pragma: no cover - no certified architecture takes this path yet
last_hidden = hidden
if self.architecture.normalizes_before_head:
last_hidden = np.asarray(self.computation.final_norm(last_hidden))
logits = np.asarray(self.computation.lm_head(last_hidden))
last_logits = logits[:, -1, :]
token_id = self.sampling.sample(last_logits)
return TailOutput(
token_id=token_id, logits=last_logits, sampling=self.sampling
)
def _byte_order(dtype: np.dtype) -> str:
order = dtype.byteorder
if order == "<":
return "little"
if order == ">":
return "big"
# '=' native, '|' not applicable (single byte)
import sys
return sys.byteorder if order in ("=", "|") else "little"
def _array_from_wire(field_payload: dict[str, Any]) -> np.ndarray:
array = np.frombuffer(
field_payload["data"], dtype=np.dtype(field_payload["dtype"])
)
return array.reshape(field_payload["shape"]).copy()

160
packages/node/meshnet_node/capability.py
View File

@@ -20,16 +20,6 @@ import time
from dataclasses import dataclass, field
from typing import Any, Mapping
from . import __version__ as _PACKAGE_VERSION
from .runtime_recipe import (
ArtifactIdentity,
RuntimeRecipeIdentity,
build_artifact_identity,
build_runtime_recipe_identity,
compatibility_fingerprint,
fingerprint_payload,
)
# Layout of the serialized report. Bump when the JSON shape changes.
CAPABILITY_SCHEMA_VERSION = 1
@@ -182,14 +172,6 @@ def _optional_text(value: Any, field_name: str) -> str | None:
return _require_text(value, field_name)
def _optional_bool(value: Any, field_name: str) -> bool:
if value is None:
return False
if isinstance(value, bool):
return value
raise CapabilityReportError(f"{field_name!r} must be a boolean")
def _require_int(value: Any, field_name: str, minimum: int) -> int:
if isinstance(value, bool) or not isinstance(value, int):
raise CapabilityReportError(f"{field_name!r} must be an integer")
@@ -236,8 +218,6 @@ class ShardRange:
start: int
end: int
owns_embedding: bool = False
owns_final_head: bool = False
def __post_init__(self) -> None:
_require_int(self.start, "shard.start", 0)
@@ -246,18 +226,9 @@ class ShardRange:
raise CapabilityReportError(
f"'shard.end' ({self.end}) must be >= 'shard.start' ({self.start})"
)
if not isinstance(self.owns_embedding, bool):
raise CapabilityReportError("'shard.owns_embedding' must be a boolean")
if not isinstance(self.owns_final_head, bool):
raise CapabilityReportError("'shard.owns_final_head' must be a boolean")
def to_dict(self) -> dict:
return {
"start": self.start,
"end": self.end,
"owns_embedding": self.owns_embedding,
"owns_final_head": self.owns_final_head,
}
return {"start": self.start, "end": self.end}
@classmethod
def from_dict(cls, data: Any) -> ShardRange:
@@ -265,12 +236,6 @@ class ShardRange:
return cls(
start=_require_int(doc.get("start"), "shard.start", 0),
end=_require_int(doc.get("end"), "shard.end", 0),
owns_embedding=_optional_bool(
doc.get("owns_embedding"), "shard.owns_embedding"
),
owns_final_head=_optional_bool(
doc.get("owns_final_head"), "shard.owns_final_head"
),
)
@@ -371,8 +336,6 @@ class CapabilityReport:
shard: ShardRange
recipe: RecipeIdentity
backend: BackendIdentity
artifact: ArtifactIdentity
runtime_recipe: RuntimeRecipeIdentity
status: str
validated_at: float
duration_ms: int
@@ -413,20 +376,6 @@ class CapabilityReport:
self.backend.device,
)
@property
def compatibility_fingerprint(self) -> str:
"""Stable compatibility digest over the full routable identity."""
return compatibility_fingerprint(
fingerprint_payload(
model=self.model.to_dict(),
shard=self.shard.to_dict(),
recipe=self.recipe.to_dict(),
backend=self.backend.to_dict(),
artifact=self.artifact.to_dict(),
runtime_recipe=self.runtime_recipe.to_dict(),
)
)
def age_seconds(self, now: float | None = None) -> float:
return max(0.0, (time.time() if now is None else now) - self.validated_at)
@@ -437,9 +386,6 @@ class CapabilityReport:
"shard": self.shard.to_dict(),
"recipe": self.recipe.to_dict(),
"backend": self.backend.to_dict(),
"artifact": self.artifact.to_dict(),
"runtime_recipe": self.runtime_recipe.to_dict(),
"compatibility_fingerprint": self.compatibility_fingerprint,
"status": self.status,
"validated_at": self.validated_at,
"duration_ms": self.duration_ms,
@@ -452,9 +398,6 @@ class CapabilityReport:
@classmethod
def from_dict(cls, data: Any) -> CapabilityReport:
doc = _as_mapping(data, "report")
declared_compatibility_fingerprint = _optional_text(
doc.get("compatibility_fingerprint"), "compatibility_fingerprint"
)
if "schema_version" not in doc:
raise CapabilityReportError(
@@ -474,13 +417,7 @@ class CapabilityReport:
):
raise CapabilityReportError("'validated_at' must be a Unix timestamp")
try:
artifact = ArtifactIdentity.from_dict(doc.get("artifact"))
runtime_recipe = RuntimeRecipeIdentity.from_dict(doc.get("runtime_recipe"))
except ValueError as exc:
raise CapabilityReportError(str(exc)) from exc
report = cls(
return cls(
schema_version=schema_version,
model=ModelIdentity.from_dict(doc.get("model")),
shard=ShardRange.from_dict(doc.get("shard")),
@@ -490,18 +427,7 @@ class CapabilityReport:
validated_at=float(validated_at),
duration_ms=_require_int(doc.get("duration_ms"), "duration_ms", 0),
diagnostics=sanitize_diagnostics(doc.get("diagnostics")),
artifact=artifact,
runtime_recipe=runtime_recipe,
)
if (
declared_compatibility_fingerprint is not None
and report.compatibility_fingerprint != declared_compatibility_fingerprint
):
raise CapabilityReportError(
"report declares a compatibility fingerprint that does not match "
"its artifact/runtime recipe"
)
return report
@classmethod
def from_json(cls, text: str) -> CapabilityReport:
@@ -532,19 +458,6 @@ def build_capability_report(
device_name: str | None = None,
quantization: str | None = None,
runtime: Mapping[str, str] | None = None,
artifact_hash: str | None = None,
runtime_recipe: RuntimeRecipeIdentity | None = None,
owns_embedding: bool = False,
owns_final_head: bool = False,
activation_dtype: Any = None,
compute_dtype: Any = None,
kv_dtype: Any = None,
kv_layout: str | None = None,
tokenizer_revision: str | None = None,
architecture_adapter: str | None = None,
boundary_schema_version: int = 1,
cache_layout: str | None = None,
recipe_params: Mapping[str, Any] | None = None,
diagnostics: Any = None,
validated_at: float | None = None,
environ: Mapping[str, str] | None = None,
@@ -555,62 +468,25 @@ def build_capability_report(
or an already-computed ``sha256:…`` string. `validated_at` defaults to now,
so callers that need determinism pass it explicitly.
"""
model_identity = ModelIdentity(
model_id=model_id,
revision=revision,
config_fingerprint=config_fingerprint(model_config),
)
shard = ShardRange(
start=shard_start,
end=shard_end,
owns_embedding=owns_embedding,
owns_final_head=owns_final_head,
)
recipe_identity = RecipeIdentity(
recipe_id=recipe_id,
recipe_version=recipe_version,
catalogue_version=catalogue_version,
)
backend_identity = BackendIdentity(
backend_id=backend_id,
device=device,
device_name=device_name,
quantization=quantization,
runtime=dict(runtime or {}),
)
artifact = build_artifact_identity(
model_id=model_id,
revision=revision,
model_config=model_config,
artifact_hash=artifact_hash,
shard_start=shard_start,
shard_end=shard_end,
)
if runtime_recipe is None:
runtime_recipe = build_runtime_recipe_identity(
return CapabilityReport(
model=ModelIdentity(
model_id=model_id,
revision=revision,
model_config=model_config,
recipe_params=recipe_params,
weight_quantization=quantization or "unknown",
config_fingerprint=config_fingerprint(model_config),
),
shard=ShardRange(start=shard_start, end=shard_end),
recipe=RecipeIdentity(
recipe_id=recipe_id,
recipe_version=recipe_version,
catalogue_version=catalogue_version,
),
backend=BackendIdentity(
backend_id=backend_id,
runtime_version=_PACKAGE_VERSION,
activation_dtype=activation_dtype,
compute_dtype=compute_dtype,
kv_dtype=kv_dtype,
kv_layout=kv_layout,
tokenizer_revision=tokenizer_revision,
architecture_adapter=architecture_adapter,
boundary_schema_version=boundary_schema_version,
cache_layout=cache_layout,
)
return CapabilityReport(
model=model_identity,
shard=shard,
recipe=recipe_identity,
backend=backend_identity,
artifact=artifact,
runtime_recipe=runtime_recipe,
device=device,
device_name=device_name,
quantization=quantization,
runtime=dict(runtime or {}),
),
status=status,
validated_at=time.time() if validated_at is None else validated_at,
duration_ms=duration_ms,

87
packages/node/meshnet_node/doctor.py
View File

@@ -36,8 +36,6 @@ from .capability import (
CapabilityReport,
build_capability_report,
)
from . import __version__ as _PACKAGE_VERSION
from .runtime_recipe import build_runtime_recipe_identity
from .recipe_manifest import (
DEFAULT_RECIPE_ID,
Recipe,
@@ -45,7 +43,6 @@ from .recipe_manifest import (
RecipeManifestError,
load_recipe_manifest,
)
from .gguf_ownership import authoritative_dense_llama_ownership
# The probe is deliberately tiny: enough tokens to drive every layer in the
# shard once, small enough that `doctor` costs seconds beyond the model load.
@@ -467,28 +464,10 @@ def _validate_recipe(
duration_ms = int((time.monotonic() - started) * 1000)
device = _backend_device(backend, selection)
ownership = authoritative_dense_llama_ownership(backend, selection)
runtime_recipe = build_runtime_recipe_identity(
model_id=selection.model_id,
revision=getattr(getattr(backend, "model", None), "revision", None),
model_config=_model_config(backend),
recipe_params=recipe.params,
weight_quantization=selection.quantization,
backend_id=recipe.backend_id,
runtime_version=_PACKAGE_VERSION,
activation_dtype="bfloat16",
compute_dtype=_backend_compute_dtype(backend),
kv_dtype=_backend_kv_dtype(backend),
kv_layout=_backend_kv_layout(backend),
tokenizer_revision=_backend_tokenizer_revision(backend, selection),
architecture_adapter=_backend_architecture_adapter(backend, recipe.backend_id),
boundary_schema_version=1,
cache_layout=_backend_cache_layout(backend, recipe.params),
)
report = build_capability_report(
model_id=selection.model_id,
shard_start=ownership.start_layer,
shard_end=ownership.end_layer,
shard_start=selection.shard_start,
shard_end=selection.shard_end,
recipe_id=recipe.id,
recipe_version=recipe.version,
catalogue_version=manifest.catalogue_version,
@@ -498,9 +477,6 @@ def _validate_recipe(
quantization=selection.quantization,
runtime=_runtime_versions(),
model_config=_model_config(backend),
runtime_recipe=runtime_recipe,
owns_embedding=ownership.owns_embedding,
owns_final_head=ownership.owns_final_head,
status=STATUS_FAILED if category else STATUS_PASSED,
duration_ms=duration_ms,
diagnostics=[d for d in diagnostics if d] or None,
@@ -592,65 +568,6 @@ def _runtime_versions() -> dict[str, str]:
return versions
def _backend_compute_dtype(backend: Any) -> str:
config = getattr(getattr(backend, "model", None), "config", None)
for candidate in (config, getattr(config, "text_config", None)):
if candidate is None:
continue
for attr in ("dtype", "torch_dtype"):
value = getattr(candidate, attr, None)
if value is None:
continue
return str(value).removeprefix("torch.")
return "bfloat16"
def _backend_kv_dtype(backend: Any) -> str:
return _backend_compute_dtype(backend)
def _backend_kv_layout(backend: Any) -> str:
return "session-cache" if getattr(backend, "supports_kv_cache", False) else "stateless"
def _backend_tokenizer_revision(backend: Any, selection: DoctorSelection) -> str:
model = getattr(backend, "model", None)
revision = getattr(model, "revision", None)
if isinstance(revision, str) and revision.strip():
return revision
return selection.model_id
def _backend_architecture_adapter(backend: Any, default: str) -> str:
config = getattr(getattr(backend, "model", None), "config", None)
for candidate in (config, getattr(config, "text_config", None)):
if candidate is None:
continue
for attr in ("architecture_adapter", "model_type"):
value = getattr(candidate, attr, None)
if isinstance(value, str) and value.strip():
return value
architectures = getattr(candidate, "architectures", None)
if isinstance(architectures, (list, tuple)) and architectures:
first = architectures[0]
if isinstance(first, str) and first.strip():
return first
return default
def _backend_cache_layout(backend: Any, recipe_params: Mapping[str, Any] | None) -> str:
if getattr(backend, "supports_kv_cache", False) is False:
return "stateless"
if recipe_params is None:
return "local-hot-kv"
if recipe_params.get("use_cache") is False:
return "stateless"
value = recipe_params.get("cache_layout")
if isinstance(value, str) and value.strip():
return value
return "local-hot-kv"
# --- output -----------------------------------------------------------------
DEFAULT_REPORT_FILENAME = "capability.json"

893
packages/node/meshnet_node/failure_semantics.py
View File

@@ -1,893 +0,0 @@
"""Bounded failure, cancellation, and restart semantics for Shard streams (DGR-013).
Distributed speed must not come with hanging or corrupted generations. This module
hardens the per-Route-Session decode stream that runs over the DGR-007 Hot KV State
manager (isolated ``(session, epoch)`` KV) and the DGR-012 continuous-batch
scheduler. It is deliberately backend-agnostic and pure-Python: it drives the same
``KvBoundaryAdapter`` the default deterministic gate uses, so the whole matrix stays
download-free, GPU-free, and API-credit-free while exercising the *real* KV
isolation path (the pinned llama.cpp worker, DGR-008, implements the identical
adapter contract).
The guarantees, mapped to the story's acceptance criteria:
* **Deadlines and heartbeat/health loss terminate blocked stream operations.**
:class:`DeadlineGuard` bounds every step against an absolute deadline and a
heartbeat-timeout; when either is breached it raises :class:`StreamTerminated`
so a blocked stream never hangs.
* **Cancellation propagates across every Shard and releases local KV and queued
buffers.** :class:`ShardCancellationGroup` fans a single cancel across every
node-local KV manager serving a Route Session and releases queued activation
buffers; the DGR-012 scheduler's :meth:`~meshnet_node.batch_scheduler.
ContinuousBatchScheduler.cancel` drops queued/active work on this node.
* **Duplicate steps are idempotent; uncertain mutations are never replayed
silently.** :class:`IdempotencyLedger` records each committed
``(session, epoch, step)`` and returns the recorded token for a duplicate
delivery instead of re-running it. A step whose outcome is *uncertain* (the
worker died mid-mutation) is marked uncertain and can never be silently
replayed — a replay attempt raises :class:`UncertainMutationError`, forcing an
explicit verify-or-restart.
* **Alpha failover restarts from token zero on a newly compatible route rather
than importing unverified KV.** :class:`RestartController` opens a *new* route
epoch, releases every shard's prior-epoch KV, and the restart re-prefills the
whole prompt from token zero. The old epoch becomes stale (rejected by the KV
manager); unverified KV is never migrated (RALPH runtime decision #14).
* **Billing/work records distinguish completed, cancelled, failed, and unverified
work.** :class:`WorkLedger` records a typed :class:`WorkRecord` per attempt;
only :attr:`WorkStatus.COMPLETED` records are billable, so cancelled, failed,
and uncertain (unverified) work is accounted but never charged.
:class:`HardenedSessionRunner` composes these into one drivable stream: it runs a
single session's prefill+decode through the adapter under a deadline/heartbeat
guard and a cancellation token, records the typed work outcome, and — via
:meth:`HardenedSessionRunner.run_with_failover` — restarts a transient failure
from token zero on a fresh epoch.
"""
from __future__ import annotations
import threading
import time
from dataclasses import dataclass, field, replace
from enum import Enum
from typing import Any, Callable, Mapping, Sequence
from meshnet_node.batch_scheduler import DoneReason, GenerationRequest
from meshnet_node.boundary_adapter import BoundaryContractError, TailOutput
from meshnet_node.hot_kv_state import (
CacheMiss,
HotKvStateManager,
IncompatibleCacheRecipeError,
KvBoundaryAdapter,
KvCacheMissError,
StaleRouteEpochError,
)
class FailureSemanticsError(RuntimeError):
"""Base class for failure/cancellation/restart errors."""
# --------------------------------------------------------------------------- #
# Typed outcomes: failure kinds and billing/work statuses.
# --------------------------------------------------------------------------- #
class FailureKind(str, Enum):
"""Why a stream step failed. Stable strings for the protocol's structured status."""
# Bounded termination of a blocked op.
DEADLINE_EXCEEDED = "deadline-exceeded"
HEARTBEAT_LOST = "heartbeat-lost"
# Transport / worker loss (transient — a restart from token zero may succeed).
WORKER_DEATH = "worker-death"
STREAM_RESET = "stream-reset"
# Protocol violations (deterministic — a restart would fail identically).
MALFORMED_BUNDLE = "malformed-bundle"
STALE_EPOCH = "stale-epoch"
INCOMPATIBLE_RECIPE = "incompatible-recipe"
# KV state expected by the caller is gone; re-prefill from token zero.
CACHE_MISS = "cache-miss"
# Explicit client cancellation.
CANCELLED = "cancelled"
# Failure kinds that a from-token-zero restart on a fresh route may recover from.
# A protocol violation or an explicit bound (deadline/cancel) is NOT restartable —
# retrying it would hang or fail identically, so we surface it instead.
_RESTARTABLE = frozenset(
{
FailureKind.WORKER_DEATH,
FailureKind.STREAM_RESET,
FailureKind.CACHE_MISS,
}
)
# Failure kinds whose mutation outcome is *uncertain* — the KV may or may not have
# advanced, so the confirmed work is billed as UNVERIFIED and never replayed
# silently. Only an *unexpected* error raised while a step was executing is
# uncertain (mapped to WORKER_DEATH). A stream reset, deadline, or cache miss
# detected at a step boundary is certain: nothing committed for that step.
_UNCERTAIN = frozenset({FailureKind.WORKER_DEATH})
class WorkStatus(str, Enum):
"""The billing-relevant outcome class of a unit of work (AC: billing records).
Only :attr:`COMPLETED` work is billable. Cancelled, failed, and unverified
work is recorded distinctly so a client is never charged for a generation that
hung, was cancelled, or whose mutations could not be verified.
"""
COMPLETED = "completed"
CANCELLED = "cancelled"
FAILED = "failed"
UNVERIFIED = "unverified"
def work_status_for(kind: FailureKind) -> WorkStatus:
"""Map a terminal failure kind to its billing/work status."""
if kind is FailureKind.CANCELLED:
return WorkStatus.CANCELLED
if kind in _UNCERTAIN:
return WorkStatus.UNVERIFIED
return WorkStatus.FAILED
def classify_exception(exc: BaseException) -> FailureKind:
"""Classify a raised error into a :class:`FailureKind`.
Protocol violations map to their specific kind; a :class:`StreamTerminated`
carries its own kind; any *unexpected* error is treated as worker death
(an uncertain, transient loss), never silently ignored.
"""
if isinstance(exc, StreamTerminated):
return exc.kind
if isinstance(exc, OperationCancelled):
return FailureKind.CANCELLED
if isinstance(exc, StaleRouteEpochError):
return FailureKind.STALE_EPOCH
if isinstance(exc, IncompatibleCacheRecipeError):
return FailureKind.INCOMPATIBLE_RECIPE
if isinstance(exc, BoundaryContractError):
return FailureKind.MALFORMED_BUNDLE
if isinstance(exc, KvCacheMissError):
return FailureKind.CACHE_MISS
return FailureKind.WORKER_DEATH
# --------------------------------------------------------------------------- #
# Deadlines and heartbeat/health loss.
# --------------------------------------------------------------------------- #
class StreamTerminated(FailureSemanticsError):
"""A blocked stream op was terminated by a deadline or heartbeat/health loss."""
def __init__(self, kind: FailureKind, detail: str = "") -> None:
self.kind = kind
self.detail = detail
suffix = f": {detail}" if detail else ""
super().__init__(f"stream terminated ({kind.value}){suffix}")
class OperationCancelled(FailureSemanticsError):
"""Raised when a step observes its :class:`CancellationToken` is cancelled."""
def __init__(self, reason: str = "client-cancel") -> None:
self.reason = reason
super().__init__(f"operation cancelled: {reason}")
@dataclass
class DeadlineGuard:
"""Bounds a blocked stream op against an absolute deadline and heartbeat loss.
``deadline`` is an absolute time on ``clock``'s scale (``None`` disables it).
``heartbeat_timeout`` is the maximum tolerated gap since the last observed
heartbeat; when the peer stops sending heartbeats (its health is lost) the gap
grows past the timeout and :meth:`check` raises rather than blocking forever.
Both bounds are checked with an injected ``clock`` so the matrix is
deterministic.
"""
deadline: float | None = None
heartbeat_timeout: float | None = None
clock: Callable[[], float] = time.monotonic
_last_heartbeat: float = field(default=0.0, init=False)
_started: bool = field(default=False, init=False)
def __post_init__(self) -> None:
if self.heartbeat_timeout is not None and self.heartbeat_timeout <= 0:
raise FailureSemanticsError("heartbeat_timeout must be positive")
def start(self) -> None:
self._last_heartbeat = self.clock()
self._started = True
def heartbeat(self) -> None:
"""Record that the peer is alive (resets the heartbeat gap)."""
self._last_heartbeat = self.clock()
def check(self) -> None:
"""Raise :class:`StreamTerminated` if the deadline or heartbeat lapsed."""
if not self._started:
self.start()
now = self.clock()
if self.deadline is not None and now >= self.deadline:
raise StreamTerminated(
FailureKind.DEADLINE_EXCEEDED,
f"deadline {self.deadline} reached at {now}",
)
if self.heartbeat_timeout is not None:
gap = now - self._last_heartbeat
if gap > self.heartbeat_timeout:
raise StreamTerminated(
FailureKind.HEARTBEAT_LOST,
f"no heartbeat for {gap} > {self.heartbeat_timeout}",
)
def remaining(self) -> float | None:
if self.deadline is None:
return None
return self.deadline - self.clock()
# --------------------------------------------------------------------------- #
# Cancellation that propagates across shards and releases KV + queued buffers.
# --------------------------------------------------------------------------- #
class CancellationToken:
"""A thread-safe one-shot cancellation flag shared by a Route Session's steps."""
def __init__(self) -> None:
self._cancelled = False
self._reason = ""
self._lock = threading.Lock()
def cancel(self, reason: str = "client-cancel") -> None:
with self._lock:
if not self._cancelled:
self._cancelled = True
self._reason = reason
@property
def cancelled(self) -> bool:
with self._lock:
return self._cancelled
@property
def reason(self) -> str:
with self._lock:
return self._reason
def raise_if_cancelled(self) -> None:
with self._lock:
if self._cancelled:
raise OperationCancelled(self._reason)
@dataclass(frozen=True)
class CancellationOutcome:
"""What a :meth:`ShardCancellationGroup.cancel` released (for observability)."""
session_id: str
route_epoch: int
shards_released: int
buffers_released: int
def to_dict(self) -> dict:
return {
"session_id": self.session_id,
"route_epoch": self.route_epoch,
"shards_released": self.shards_released,
"buffers_released": self.buffers_released,
}
class ShardCancellationGroup:
"""Fan one cancellation across every node-local Shard of a Route Session.
A Route Session spans a chain of Shards, each with its own local Hot KV State
manager (KV is never migrated between nodes). Cancelling the session must free
*all* of that state: this group releases the ``(session, epoch)`` KV on every
registered manager and invokes every registered queued-buffer release callback
(the pending activation bundles a node holds for the session). Release is
idempotent, so cancelling twice is safe.
"""
def __init__(self, session_id: str, route_epoch: int) -> None:
if not isinstance(session_id, str) or not session_id.strip():
raise FailureSemanticsError("session_id must be a non-empty string")
self.session_id = session_id
self.route_epoch = int(route_epoch)
self._managers: list[HotKvStateManager] = []
self._buffers: list[Callable[[], None]] = []
self._lock = threading.Lock()
self._cancelled = False
def add_shard(self, manager: HotKvStateManager) -> "ShardCancellationGroup":
with self._lock:
self._managers.append(manager)
return self
def add_queued_buffer(
self, release: Callable[[], None]
) -> "ShardCancellationGroup":
"""Register a queued activation buffer's release callback."""
with self._lock:
self._buffers.append(release)
return self
@property
def cancelled(self) -> bool:
with self._lock:
return self._cancelled
def cancel(self) -> CancellationOutcome:
"""Release every shard's KV and every queued buffer for this session."""
with self._lock:
managers = list(self._managers)
buffers = list(self._buffers)
self._buffers.clear()
self._cancelled = True
shards_released = 0
for manager in managers:
if manager.release(self.session_id, self.route_epoch):
shards_released += 1
buffers_released = 0
for release in buffers:
release()
buffers_released += 1
return CancellationOutcome(
session_id=self.session_id,
route_epoch=self.route_epoch,
shards_released=shards_released,
buffers_released=buffers_released,
)
# --------------------------------------------------------------------------- #
# Idempotency: duplicate steps are no-ops; uncertain mutations never replay.
# --------------------------------------------------------------------------- #
class StepPhase(str, Enum):
IN_FLIGHT = "in-flight"
COMMITTED = "committed"
UNCERTAIN = "uncertain"
class UncertainMutationError(FailureSemanticsError):
"""Raised when a caller tries to replay a step whose outcome is uncertain.
A step is uncertain when its mutation may or may not have been applied (worker
death / stream reset mid-append). Replaying it silently could double-apply KV
or bill unverified work, so the ledger refuses: the caller must verify against
the actual KV length or restart from token zero on a fresh epoch instead.
"""
@dataclass(frozen=True)
class StepKey:
"""Identity of one idempotent stream step within a route epoch."""
session_id: str
route_epoch: int
step_index: int
@dataclass(frozen=True)
class StepDisposition:
"""What :meth:`IdempotencyLedger.begin` decided for a step."""
fresh: bool
token: int | None = None
@property
def duplicate(self) -> bool:
return not self.fresh
class IdempotencyLedger:
"""Records committed/uncertain stream steps so duplicates never re-mutate.
Keyed by ``(session, epoch, step_index)`` — the protocol's idempotency step.
* :meth:`begin` on a *fresh* key marks it in-flight and returns "execute".
* :meth:`begin` on a *committed* key returns the recorded token so a duplicate
delivery is a no-op (idempotent replay).
* :meth:`begin` on an *in-flight* or *uncertain* key raises
:class:`UncertainMutationError` — a concurrent duplicate or a replay of an
unverified mutation is never silently applied.
"""
def __init__(self) -> None:
self._phase: dict[StepKey, StepPhase] = {}
self._token: dict[StepKey, int] = {}
self._lock = threading.Lock()
def begin(self, key: StepKey) -> StepDisposition:
with self._lock:
phase = self._phase.get(key)
if phase is None:
self._phase[key] = StepPhase.IN_FLIGHT
return StepDisposition(fresh=True)
if phase is StepPhase.COMMITTED:
return StepDisposition(fresh=False, token=self._token[key])
# IN_FLIGHT (concurrent duplicate) or UNCERTAIN (post-crash replay):
# both are unverified and must not be silently re-applied.
raise UncertainMutationError(
f"step {key.step_index} for session {key.session_id[:8]} epoch "
f"{key.route_epoch} is {phase.value}; refusing silent replay"
)
def commit(self, key: StepKey, token: int) -> None:
with self._lock:
self._phase[key] = StepPhase.COMMITTED
self._token[key] = int(token)
def mark_uncertain(self, key: StepKey, detail: str = "") -> None:
with self._lock:
# A committed step is verified; never downgrade it.
if self._phase.get(key) is StepPhase.COMMITTED:
return
self._phase[key] = StepPhase.UNCERTAIN
def phase_of(self, key: StepKey) -> StepPhase | None:
with self._lock:
return self._phase.get(key)
def committed_token(self, key: StepKey) -> int | None:
with self._lock:
return self._token.get(key)
def has_uncertain(self) -> bool:
with self._lock:
return any(p is StepPhase.UNCERTAIN for p in self._phase.values())
# --------------------------------------------------------------------------- #
# Restart / alpha failover: from token zero on a fresh compatible route.
# --------------------------------------------------------------------------- #
class RestartController:
"""Alpha failover that restarts from token zero, never importing prior KV.
RALPH runtime decision #14: when the alpha (the head owning embedding + final
head) fails, the route retries from token zero; unverified KV is never
migrated. :meth:`failover` opens the *next* route epoch and releases every
node-local shard's prior-epoch KV, so the restart begins with empty caches. The
KV manager then treats the failed epoch as stale (a later reference to it is
rejected), which is what keeps a half-computed cache from being reused.
"""
def __init__(self, managers: Sequence[HotKvStateManager]) -> None:
self._managers = list(managers)
def failover(self, session_id: str, failed_epoch: int) -> int:
"""Advance to a fresh epoch and drop the failed epoch's KV on every shard."""
new_epoch = int(failed_epoch) + 1
for manager in self._managers:
manager.release(session_id, failed_epoch)
return new_epoch
def assert_fresh_start(self, session_id: str, new_epoch: int) -> None:
"""Verify no shard carries KV for the new epoch (a true token-zero restart).
Any residual KV under the new epoch would be unverified imported state;
this fails closed so a restart can never silently attend over it.
"""
for manager in self._managers:
result = manager.resolve(session_id, new_epoch)
if not isinstance(result, CacheMiss):
raise FailureSemanticsError(
f"restart epoch {new_epoch} for session {session_id[:8]} is not "
"empty; refusing to import unverified KV"
)
# --------------------------------------------------------------------------- #
# Billing / work records.
# --------------------------------------------------------------------------- #
@dataclass(frozen=True)
class WorkRecord:
"""A typed unit of served work, distinguishing what may be billed.
``tokens`` counts only *committed* generated tokens. Only a
:attr:`WorkStatus.COMPLETED` record is billable; cancelled/failed/unverified
records carry their confirmed token count for observability but are excluded
from billing so uncompleted or unverified work is never charged.
"""
session_id: str
route_epoch: int
status: WorkStatus
tokens: int
failure_kind: FailureKind | None = None
detail: str = ""
@property
def billable(self) -> bool:
return self.status is WorkStatus.COMPLETED
def to_dict(self) -> dict:
return {
"session_id": self.session_id,
"route_epoch": self.route_epoch,
"status": self.status.value,
"tokens": self.tokens,
"failure_kind": self.failure_kind.value if self.failure_kind else None,
"detail": self.detail,
"billable": self.billable,
}
class WorkLedger:
"""Append-only ledger of :class:`WorkRecord`, split by billing status."""
def __init__(self) -> None:
self._records: list[WorkRecord] = []
self._lock = threading.Lock()
def record(self, record: WorkRecord) -> WorkRecord:
with self._lock:
self._records.append(record)
return record
def records(self) -> list[WorkRecord]:
with self._lock:
return list(self._records)
def records_for(self, session_id: str) -> list[WorkRecord]:
with self._lock:
return [r for r in self._records if r.session_id == session_id]
def billable_records(self) -> list[WorkRecord]:
with self._lock:
return [r for r in self._records if r.billable]
def billable_tokens(self) -> int:
"""Total tokens that may be charged (completed work only)."""
with self._lock:
return sum(r.tokens for r in self._records if r.billable)
def counts_by_status(self) -> dict[str, int]:
counts: dict[str, int] = {s.value: 0 for s in WorkStatus}
with self._lock:
for record in self._records:
counts[record.status.value] += 1
return counts
def to_dict(self) -> dict:
with self._lock:
records = [r.to_dict() for r in self._records]
counts: dict[str, int] = {s.value: 0 for s in WorkStatus}
for record in records:
counts[record["status"]] += 1
return {
"schema_version": 1,
"records": records,
"counts_by_status": counts,
"billable_tokens": sum(r["tokens"] for r in records if r["billable"]),
}
# --------------------------------------------------------------------------- #
# The hardened single-session stream runner.
# --------------------------------------------------------------------------- #
@dataclass(frozen=True)
class RunOutcome:
"""The typed result of one hardened generation attempt."""
session_id: str
route_epoch: int
status: WorkStatus
tokens: tuple[int, ...]
failure_kind: FailureKind | None
detail: str
@property
def completed(self) -> bool:
return self.status is WorkStatus.COMPLETED
@property
def token_count(self) -> int:
return len(self.tokens)
@property
def restartable(self) -> bool:
return self.failure_kind in _RESTARTABLE
def work_record(self) -> WorkRecord:
return WorkRecord(
session_id=self.session_id,
route_epoch=self.route_epoch,
status=self.status,
tokens=len(self.tokens),
failure_kind=self.failure_kind,
detail=self.detail,
)
@dataclass(frozen=True)
class FailoverResult:
"""The result of a run that may have restarted from token zero after a failure."""
outcome: RunOutcome
attempts: tuple[RunOutcome, ...]
restarts: int
@property
def completed(self) -> bool:
return self.outcome.completed
def to_dict(self) -> dict:
return {
"final_status": self.outcome.status.value,
"final_epoch": self.outcome.route_epoch,
"restarts": self.restarts,
"attempts": [
{
"route_epoch": a.route_epoch,
"status": a.status.value,
"failure_kind": a.failure_kind.value if a.failure_kind else None,
"tokens": a.token_count,
}
for a in self.attempts
],
}
class HardenedSessionRunner:
"""Drive one Route Session's decode stream with bounded failure semantics.
The runner owns a single full-shard :class:`KvBoundaryAdapter` (head **and**
tail, so a step samples a token) and threads every DGR-013 guarantee through a
step loop:
* every step is bounded by a :class:`DeadlineGuard` and can observe a
:class:`CancellationToken`;
* every step is idempotent through an :class:`IdempotencyLedger` (a duplicate
returns the recorded token; an uncertain mutation is never replayed);
* any failure releases this session's KV (cancellation) and is recorded as a
typed :class:`WorkRecord` in the :class:`WorkLedger`;
* :meth:`run_with_failover` restarts a transient failure from token zero on a
fresh epoch via a :class:`RestartController`.
"""
def __init__(
self,
adapter: KvBoundaryAdapter,
*,
clock: Callable[[], float] | None = None,
work_ledger: WorkLedger | None = None,
idempotency: IdempotencyLedger | None = None,
) -> None:
if not (adapter.is_head and adapter.is_tail):
raise FailureSemanticsError(
"HardenedSessionRunner needs a full (head+tail) shard so decode "
"steps sample tokens; got a partial range "
f"(head={adapter.is_head} tail={adapter.is_tail})"
)
self._adapter = adapter
self._manager: HotKvStateManager = adapter.manager
self._clock = clock or time.monotonic
self.work_ledger = work_ledger or WorkLedger()
self.idempotency = idempotency or IdempotencyLedger()
# -- single attempt -------------------------------------------------------
def run(
self,
request: GenerationRequest,
*,
deadline: float | None = None,
heartbeat_timeout: float | None = None,
cancel_token: CancellationToken | None = None,
heartbeat: Callable[[int], bool] | None = None,
before_step: Callable[[int], None] | None = None,
) -> RunOutcome:
"""Run one attempt of ``request``; record and return a typed outcome.
``deadline`` (absolute, on the injected clock) and ``heartbeat_timeout``
bound blocked steps. ``cancel_token`` lets a client cancel mid-stream.
``heartbeat(step)`` returns ``True`` when a heartbeat was heard before that
step (resetting the health timer); ``before_step(step)`` is a fault-
injection / clock-advance hook run before each step and may raise
:class:`StreamTerminated` (e.g. a stream reset) or
:class:`OperationCancelled`.
"""
sid = request.session_id
epoch = request.route_epoch
guard = DeadlineGuard(
deadline=deadline,
heartbeat_timeout=heartbeat_timeout,
clock=self._clock,
)
guard.start()
tokens: list[int] = []
current_key: StepKey | None = None
try:
# step 0 is the prefill (emits the first token); steps 1..N are decodes.
for step_index in range(request.max_new_tokens):
# before_step is the fault-injection / clock-advance hook and may
# itself terminate the step (stream reset, cancel); run it first so
# a fault it raises takes effect on this step, then re-check the
# bounds it may have advanced (deadline / heartbeat / cancel).
if before_step is not None:
before_step(step_index)
if cancel_token is not None:
cancel_token.raise_if_cancelled()
if heartbeat is not None and heartbeat(step_index):
guard.heartbeat()
guard.check()
current_key = StepKey(sid, epoch, step_index)
disposition = self.idempotency.begin(current_key)
if disposition.duplicate:
# Idempotent replay: reuse the recorded token, do not re-mutate.
assert disposition.token is not None
tokens.append(disposition.token)
continue
token = self._execute_step(request, step_index, tokens)
if isinstance(token, CacheMiss):
# The expected KV was gone; the append never started, so this is
# a certain (not uncertain) miss — restartable from token zero.
return self._finish_failure(
request,
tokens,
FailureKind.CACHE_MISS,
str(token),
cancel_token,
)
self.idempotency.commit(current_key, token)
tokens.append(token)
except (StreamTerminated, OperationCancelled) as exc:
return self._finish_failure(
request, tokens, classify_exception(exc), str(exc), cancel_token
)
except (
BoundaryContractError,
StaleRouteEpochError,
IncompatibleCacheRecipeError,
KvCacheMissError,
) as exc:
# Deterministic protocol/state errors, all validated before any KV
# append committed — certain, not uncertain.
return self._finish_failure(
request, tokens, classify_exception(exc), str(exc), cancel_token
)
except UncertainMutationError as exc:
# A replay of an unverified step reached the ledger — never silent.
return self._finish_failure(
request, tokens, FailureKind.WORKER_DEATH, str(exc), cancel_token
)
except Exception as exc: # noqa: BLE001 - unexpected == worker death
# An unexpected error mid-step may have left the KV half-mutated; mark
# the step uncertain so it can never be silently replayed, then fail
# closed as unverified work.
if current_key is not None:
self.idempotency.mark_uncertain(current_key, str(exc))
return self._finish_failure(
request, tokens, FailureKind.WORKER_DEATH, str(exc), cancel_token
)
return self._finish_completed(request, tokens)
def _execute_step(
self, request: GenerationRequest, step_index: int, tokens: list[int]
) -> int | CacheMiss:
sid = request.session_id
epoch = request.route_epoch
if step_index == 0:
out = self._adapter.prefill(
sid, epoch, token_ids=list(request.prompt_token_ids)
)
else:
# expected_seq_len defends the KV layer against a desynchronised decode:
# prompt positions plus the tokens already committed this run.
expected = request.prompt_len + (step_index - 1)
out = self._adapter.decode(
sid,
epoch,
token_ids=[tokens[-1]],
expected_seq_len=expected,
)
if isinstance(out, CacheMiss):
return out
if not isinstance(out, TailOutput):
raise FailureSemanticsError(
"full-shard step did not yield a sampled token; got "
f"{type(out).__name__}"
)
return int(out.token_id)
# -- failover across restarts --------------------------------------------
def run_with_failover(
self,
request: GenerationRequest,
controller: RestartController,
*,
max_restarts: int = 3,
**run_kwargs: Any,
) -> FailoverResult:
"""Run ``request``, restarting a transient failure from token zero.
On a restartable failure (worker death, stream reset, cache miss) the
controller advances to a fresh epoch and drops the failed epoch's KV; the
next attempt re-prefills the whole prompt from token zero. A deterministic
failure (deadline, cancel, malformed bundle, stale epoch) is returned as-is
— retrying it would hang or fail identically. Per-attempt fault-injection
hooks (``before_step`` / ``heartbeat``) are only applied to the *first*
attempt so a restart runs clean.
"""
if max_restarts < 0:
raise FailureSemanticsError("max_restarts must be >= 0")
epoch = request.route_epoch
attempts: list[RunOutcome] = []
first_kwargs = run_kwargs
for attempt in range(max_restarts + 1):
attempt_request = replace(request, route_epoch=epoch)
kwargs = first_kwargs if attempt == 0 else {}
outcome = self.run(attempt_request, **kwargs)
attempts.append(outcome)
if outcome.completed or not outcome.restartable or attempt == max_restarts:
return FailoverResult(
outcome=outcome, attempts=tuple(attempts), restarts=attempt
)
# Alpha failover: fresh epoch, drop prior-epoch KV on every shard, and
# verify the new epoch starts empty (no unverified KV import).
epoch = controller.failover(request.session_id, epoch)
controller.assert_fresh_start(request.session_id, epoch)
# Unreachable: the loop always returns, but keep the type-checker happy.
raise FailureSemanticsError("run_with_failover exhausted without returning")
# -- outcome bookkeeping --------------------------------------------------
def _finish_completed(
self, request: GenerationRequest, tokens: list[int]
) -> RunOutcome:
outcome = RunOutcome(
session_id=request.session_id,
route_epoch=request.route_epoch,
status=WorkStatus.COMPLETED,
tokens=tuple(tokens),
failure_kind=None,
detail="",
)
self.work_ledger.record(outcome.work_record())
return outcome
def _finish_failure(
self,
request: GenerationRequest,
tokens: list[int],
kind: FailureKind,
detail: str,
cancel_token: CancellationToken | None,
) -> RunOutcome:
# Cancellation semantics: release this session's local KV so a failed or
# cancelled stream never leaks its cache. release() is idempotent.
self._manager.release(request.session_id, request.route_epoch)
if cancel_token is not None and kind is not FailureKind.CANCELLED:
# Ensure downstream shards sharing the token also stop.
cancel_token.cancel(kind.value)
outcome = RunOutcome(
session_id=request.session_id,
route_epoch=request.route_epoch,
status=work_status_for(kind),
tokens=tuple(tokens),
failure_kind=kind,
detail=detail,
)
self.work_ledger.record(outcome.work_record())
return outcome

423
packages/node/meshnet_node/gguf_backend.py
View File

@@ -1,423 +0,0 @@
"""Native llama.cpp/GGUF backend adapter for Meshnet node startup.
This module keeps the node-side GGUF seam separate from the Torch-backed
reference path. The public object intentionally looks like the existing
``TorchModelShard`` surface so ``TorchNodeServer`` can serve it without changing
the HTTP/control-plane code that already correlates request ids, telemetry and
billing.
The transport layer is intentionally explicit:
* direct worker calls are expected to use the versioned gRPC Shard protocol
from :mod:`meshnet_node.native_protocol`;
* the backend itself stays transport-agnostic and delegates to a worker
transport object with the same method surface as the existing node backend.
The default factory is strict: if no worker endpoint is configured, it fails
closed rather than silently pretending the native worker exists.
"""
from __future__ import annotations
import os
from dataclasses import dataclass, field
from types import SimpleNamespace
from typing import Any, Protocol, runtime_checkable
from .model_backend import (
MissingModelDependencyError,
ModelBackendError,
TailTokenResult,
TensorPayload,
)
_BACKEND_ID = "llama.cpp"
@runtime_checkable
class NativeWorkerTransport(Protocol):
"""Backend-shaped transport for the supervised native worker."""
def encode_prompt(
self,
prompt: str,
session_id: str | None = None,
) -> TensorPayload | TailTokenResult | str: ...
def encode_next_token(
self,
token_id: int,
session_id: str,
) -> TensorPayload | TailTokenResult | str: ...
def forward_bytes(
self,
body: bytes,
shape: list[int],
attention_mask_header: str | None,
position_ids_header: str | None,
*,
start_layer: int | None = None,
session_id: str | None = None,
cache_mode: str | None = None,
past_len: int | None = None,
) -> TensorPayload | TailTokenResult | str: ...
def decode_tail_token(self, hidden_states: Any) -> TailTokenResult: ...
def generate_text(
self,
messages: list[dict],
max_new_tokens: int = 5120,
temperature: float = 1.0,
top_p: float = 1.0,
) -> str: ...
def generate_text_streaming(
self,
messages: list[dict],
max_new_tokens: int = 5120,
temperature: float = 1.0,
top_p: float = 1.0,
): ...
def count_prompt_tokens(self, messages: list[dict]) -> int: ...
def count_text_tokens(self, text: str) -> int: ...
def eos_token_ids(self) -> list[int]: ...
def release_session(self, session_id: str) -> None: ...
@dataclass(frozen=True)
class _NativeModelConfig:
"""Enough model metadata for admission and capability reporting."""
model_type: str = "llama"
architecture_adapter: str = "dense-llama"
num_hidden_layers: int = 1
torch_dtype: str = "bfloat16"
def to_dict(self) -> dict[str, Any]:
return {
"model_type": self.model_type,
"architecture_adapter": self.architecture_adapter,
"num_hidden_layers": self.num_hidden_layers,
"torch_dtype": self.torch_dtype,
}
@dataclass
class GgufNodeBackend:
"""GGUF shard backend shaped like ``TorchModelShard``.
The adapter keeps the Meshnet-facing surface stable while the actual model
execution is delegated to a worker transport. The backend carries the exact
model, shard and runtime metadata required for admission and registration.
"""
model_id: str
shard_start: int
shard_end: int
quantization: str = "bfloat16"
transport: NativeWorkerTransport | None = None
total_layers: int | None = None
model_revision: str | None = None
loaded_tensor_names: tuple[str, ...] = ()
device_type: str = "cpu"
supports_kv_cache: bool = True
worker_url: str | None = None
architecture_adapter: str = "dense-llama"
tokenizer_revision: str | None = None
runtime_recipe_fingerprint: str | None = None
_model: SimpleNamespace = field(init=False, repr=False)
_tokenizer: SimpleNamespace = field(init=False, repr=False)
is_head: bool = field(init=False)
is_tail: bool = field(init=False)
loaded_shard_start: int = field(init=False)
loaded_shard_end: int = field(init=False)
owns_embedding: bool = field(init=False)
owns_final_head: bool = field(init=False)
backend_id = _BACKEND_ID
def __post_init__(self) -> None:
if self.shard_start < 0 or self.shard_end < self.shard_start:
raise ValueError("shard_start must be <= shard_end and non-negative")
total_layers = self.total_layers or (self.shard_end + 1)
object.__setattr__(
self,
"total_layers",
int(total_layers),
)
object.__setattr__(
self,
"_model",
SimpleNamespace(
revision=self.model_revision or self.model_id,
config=_NativeModelConfig(
num_hidden_layers=int(total_layers),
torch_dtype=self.quantization,
),
),
)
object.__setattr__(
self,
"_tokenizer",
SimpleNamespace(
model_id=self.model_id,
revision=self.tokenizer_revision or self.model_revision or self.model_id,
eos_token="",
eos_token_id=[],
),
)
object.__setattr__(self, "is_head", self.shard_start == 0)
object.__setattr__(self, "is_tail", self.shard_end >= int(total_layers) - 1)
object.__setattr__(self, "loaded_shard_start", self.shard_start)
object.__setattr__(self, "loaded_shard_end", self.shard_end)
object.__setattr__(self, "owns_embedding", self.is_head)
object.__setattr__(self, "owns_final_head", self.is_tail)
if not self.loaded_tensor_names:
object.__setattr__(
self,
"loaded_tensor_names",
self._default_tensor_inventory(),
)
@property
def model(self) -> Any:
return self._model
@property
def tokenizer(self) -> Any:
return self._tokenizer
@property
def device(self) -> SimpleNamespace:
return SimpleNamespace(type=self.device_type)
@property
def shard_range(self) -> tuple[int, int]:
return self.shard_start, self.shard_end
def encode_prompt(self, prompt: str, session_id: str | None = None) -> TensorPayload | TailTokenResult | str:
return self._transport().encode_prompt(prompt, session_id=session_id)
def encode_next_token(self, token_id: int, session_id: str) -> TensorPayload | TailTokenResult | str:
return self._transport().encode_next_token(token_id, session_id)
def forward_bytes(
self,
body: bytes,
shape: list[int],
attention_mask_header: str | None,
position_ids_header: str | None,
start_layer: int | None = None,
session_id: str | None = None,
cache_mode: str | None = None,
past_len: int | None = None,
) -> TensorPayload | TailTokenResult | str:
return self._transport().forward_bytes(
body,
shape,
attention_mask_header,
position_ids_header,
start_layer=start_layer,
session_id=session_id,
cache_mode=cache_mode,
past_len=past_len,
)
def decode_tail(self, hidden_states: Any) -> str:
return self.decode_tail_token(hidden_states).text
def decode_tail_token(self, hidden_states: Any) -> TailTokenResult:
return self._transport().decode_tail_token(hidden_states)
def generate_text(
self,
messages: list[dict],
max_new_tokens: int = 5120,
temperature: float = 1.0,
top_p: float = 1.0,
) -> str:
return self._transport().generate_text(messages, max_new_tokens, temperature, top_p)
def generate_text_streaming(
self,
messages: list[dict],
max_new_tokens: int = 5120,
temperature: float = 1.0,
top_p: float = 1.0,
):
yield from self._transport().generate_text_streaming(messages, max_new_tokens, temperature, top_p)
def count_prompt_tokens(self, messages: list[dict]) -> int:
return self._transport().count_prompt_tokens(messages)
def count_text_tokens(self, text: str) -> int:
return self._transport().count_text_tokens(text)
def eos_token_ids(self) -> list[int]:
return self._transport().eos_token_ids()
def release_session(self, session_id: str) -> None:
self._transport().release_session(session_id)
def _transport(self) -> NativeWorkerTransport:
if self.transport is None:
raise MissingModelDependencyError(
"native GGUF backend needs a worker transport; set MESHNET_NATIVE_WORKER_URL "
"or inject a test transport"
)
return self.transport
def _default_tensor_inventory(self) -> tuple[str, ...]:
tensor_names = [f"blk.{layer}.weight" for layer in range(self.shard_start, self.shard_end + 1)]
if self.is_head:
tensor_names.append("token_embd.weight")
if self.is_tail:
tensor_names.extend(["output_norm.weight", "output.weight"])
return tuple(tensor_names)
class GrpcNativeWorkerTransport:
"""Transport that speaks the versioned gRPC worker protocol.
The transport is intentionally conservative: it provides the unary service
hooks and carries the protocol metadata, but it does not guess at worker
behavior beyond what the compiled protobuf schema already describes.
"""
def __init__(self, worker_url: str, *, timeout: float = 30.0) -> None:
self.worker_url = worker_url
self.timeout = timeout
self._grpc = None
self._channel = None
self._stub = None
def _ensure_stub(self) -> Any:
if self._stub is not None:
return self._stub
try:
import grpc # type: ignore[import]
except ImportError as exc: # pragma: no cover - environment dependent
raise MissingModelDependencyError(
"grpc is required for the native GGUF worker transport"
) from exc
from . import native_protocol
grpc_mod = native_protocol.load_grpc()
self._grpc = grpc
self._channel = grpc.insecure_channel(self.worker_url)
self._stub = grpc_mod.ShardRuntimeStub(self._channel)
return self._stub
def encode_prompt(self, prompt: str, session_id: str | None = None) -> TensorPayload | TailTokenResult | str:
raise ModelBackendError(
"gRPC transport is present, but prompt-to-activation translation is provided "
"by the backend wrapper so it can keep worker framing and tokenizer state aligned"
)
def encode_next_token(self, token_id: int, session_id: str) -> TensorPayload | TailTokenResult | str:
raise ModelBackendError(
"gRPC transport is present, but decode translation is provided by the backend wrapper"
)
def forward_bytes(
self,
body: bytes,
shape: list[int],
attention_mask_header: str | None,
position_ids_header: str | None,
*,
start_layer: int | None = None,
session_id: str | None = None,
cache_mode: str | None = None,
past_len: int | None = None,
) -> TensorPayload | TailTokenResult | str:
raise ModelBackendError(
"gRPC transport is present, but activation streaming is handled by the backend wrapper"
)
def decode_tail_token(self, hidden_states: Any) -> TailTokenResult:
raise ModelBackendError("tail decoding is handled by the backend wrapper")
def generate_text(
self,
messages: list[dict],
max_new_tokens: int = 5120,
temperature: float = 1.0,
top_p: float = 1.0,
) -> str:
raise ModelBackendError("text generation is handled by the backend wrapper")
def generate_text_streaming(
self,
messages: list[dict],
max_new_tokens: int = 5120,
temperature: float = 1.0,
top_p: float = 1.0,
):
raise ModelBackendError("streaming generation is handled by the backend wrapper")
def count_prompt_tokens(self, messages: list[dict]) -> int:
return sum(1 for message in messages if isinstance(message, dict))
def count_text_tokens(self, text: str) -> int:
return len(text.split()) or (1 if text else 0)
def eos_token_ids(self) -> list[int]:
return []
def release_session(self, session_id: str) -> None:
stub = self._ensure_stub()
from . import native_protocol
pb2 = native_protocol.load()
stub.Release(pb2.ReleaseRequest(reason="release from adapter"))
def build_gguf_backend(
*,
model_id: str,
shard_start: int,
shard_end: int,
quantization: str = "bfloat16",
transport: NativeWorkerTransport | None = None,
worker_url: str | None = None,
total_layers: int | None = None,
model_revision: str | None = None,
loaded_tensor_names: tuple[str, ...] = (),
device_type: str = "cpu",
architecture_adapter: str = "dense-llama",
tokenizer_revision: str | None = None,
runtime_recipe_fingerprint: str | None = None,
supports_kv_cache: bool = True,
) -> GgufNodeBackend:
"""Construct a native-worker-backed GGUF node backend."""
if transport is None:
worker_url = worker_url or os.environ.get("MESHNET_NATIVE_WORKER_URL")
if not worker_url:
raise MissingModelDependencyError(
"set MESHNET_NATIVE_WORKER_URL to the local gRPC worker endpoint "
"or inject a fake transport in tests"
)
transport = GrpcNativeWorkerTransport(worker_url)
return GgufNodeBackend(
model_id=model_id,
shard_start=shard_start,
shard_end=shard_end,
quantization=quantization,
transport=transport,
total_layers=total_layers,
model_revision=model_revision,
loaded_tensor_names=loaded_tensor_names,
device_type=device_type,
supports_kv_cache=supports_kv_cache,
worker_url=worker_url,
architecture_adapter=architecture_adapter,
tokenizer_revision=tokenizer_revision,
runtime_recipe_fingerprint=runtime_recipe_fingerprint,
)

287
packages/node/meshnet_node/gguf_ownership.py
View File

@@ -1,287 +0,0 @@
"""Dense-Llama GGUF ownership helpers.
This module keeps two related concerns together:
* selecting the tensors a dense-Llama GGUF shard is allowed to own; and
* inferring the authoritative loaded range / endpoint ownership from the
tensors the model actually exposes.
The first is used by the range-aware loader seam. The second is used by the
doctor/admission/reporting path so the tracker sees what the model loaded, not
what a CLI flag claimed.
"""
from __future__ import annotations
import re
from dataclasses import dataclass
from typing import Any, Iterable, Mapping
_BLOCK_RE = re.compile(r"^blk\.(\d+)\.")
_HEAD_TENSOR_NAMES = {
"token_embd.weight",
"token_embd.bias",
"tok_embeddings.weight",
"tok_embeddings.bias",
"embed_tokens.weight",
"embed_tokens.bias",
}
_TAIL_TENSOR_NAMES = {
"output_norm.weight",
"output_norm.bias",
"output.weight",
"output.bias",
"lm_head.weight",
"lm_head.bias",
}
@dataclass(frozen=True)
class DenseLlamaShardOwnership:
"""Authoritative ownership for one dense-Llama shard."""
start_layer: int
end_layer: int
owns_embedding: bool
owns_final_head: bool
tensor_names: tuple[str, ...] = ()
source_artifact_hash: str | None = None
slice_artifact_hash: str | None = None
derivative_slice: bool = False
final_artifact_semantics: bool = True
def __post_init__(self) -> None:
if self.start_layer < 0:
raise ValueError("start_layer must be non-negative")
if self.end_layer < self.start_layer:
raise ValueError("end_layer must be >= start_layer")
if self.derivative_slice:
if not self.source_artifact_hash or not self.slice_artifact_hash:
raise ValueError(
"temporary derivative sub-GGUFs must carry source and slice hashes"
)
if self.final_artifact_semantics:
raise ValueError(
"temporary derivative sub-GGUFs must not be claimed as final artifacts"
)
@property
def range(self) -> tuple[int, int]:
return self.start_layer, self.end_layer
def to_dict(self) -> dict[str, Any]:
return {
"start_layer": self.start_layer,
"end_layer": self.end_layer,
"owns_embedding": self.owns_embedding,
"owns_final_head": self.owns_final_head,
"tensor_names": list(self.tensor_names),
"source_artifact_hash": self.source_artifact_hash,
"slice_artifact_hash": self.slice_artifact_hash,
"derivative_slice": self.derivative_slice,
"final_artifact_semantics": self.final_artifact_semantics,
}
def select_dense_llama_tensor_names(
tensor_names: Iterable[str],
start_layer: int,
end_layer: int,
*,
total_layers: int | None = None,
) -> set[str]:
"""Return the dense-Llama GGUF tensor names owned by an inclusive range."""
if start_layer < 0:
raise ValueError("start_layer must be non-negative")
if end_layer < start_layer:
raise ValueError("end_layer must be greater than or equal to start_layer")
selected: set[str] = set()
for tensor_name in tensor_names:
if _tensor_belongs_to_range(tensor_name, start_layer, end_layer, total_layers):
selected.add(tensor_name)
return selected
def infer_dense_llama_ownership(
tensor_names: Iterable[str],
*,
total_layers: int | None = None,
source_artifact_hash: str | None = None,
slice_artifact_hash: str | None = None,
derivative_slice: bool = False,
final_artifact_semantics: bool = True,
) -> DenseLlamaShardOwnership:
"""Infer authoritative loaded range and endpoint ownership from tensors."""
names = tuple(str(name) for name in tensor_names if isinstance(name, str))
if not names:
raise ValueError("tensor inventory is empty")
block_layers = sorted(
{
layer
for name in names
if (layer := _layer_index(name)) is not None
}
)
if not block_layers:
raise ValueError("tensor inventory does not contain any blk.N.* tensors")
selected = tuple(sorted(names))
return DenseLlamaShardOwnership(
start_layer=block_layers[0],
end_layer=block_layers[-1],
owns_embedding=any(_is_head_tensor(name) for name in names),
owns_final_head=any(
_is_tail_tensor(name, total_layers=total_layers, loaded_end=block_layers[-1])
for name in names
),
tensor_names=selected,
source_artifact_hash=source_artifact_hash,
slice_artifact_hash=slice_artifact_hash,
derivative_slice=derivative_slice,
final_artifact_semantics=final_artifact_semantics,
)
def authoritative_dense_llama_ownership(
backend: Any,
selection: Any | None = None,
) -> DenseLlamaShardOwnership:
"""Return the most authoritative dense-Llama ownership the backend exposes."""
tensor_names = _tensor_names_from_backend(backend)
if tensor_names:
try:
return infer_dense_llama_ownership(
tensor_names,
total_layers=_backend_total_layers(backend, selection),
)
except ValueError:
pass
start, end = _backend_loaded_bounds(backend, selection)
return DenseLlamaShardOwnership(
start_layer=start,
end_layer=end,
owns_embedding=_backend_owns_embedding(backend, start),
owns_final_head=_backend_owns_final_head(backend, end),
)
def _backend_loaded_bounds(backend: Any, selection: Any | None) -> tuple[int, int]:
start = getattr(backend, "loaded_shard_start", None)
end = getattr(backend, "loaded_shard_end", None)
if start is None:
start = getattr(backend, "shard_start", None)
if end is None:
end = getattr(backend, "shard_end", None)
if start is None or end is None:
if selection is None:
raise ValueError("backend does not expose a loaded shard range")
start = getattr(selection, "shard_start")
end = getattr(selection, "shard_end")
return int(start), int(end)
def _backend_owns_embedding(backend: Any, start: int) -> bool:
value = getattr(backend, "owns_embedding", None)
if value is None:
value = getattr(backend, "is_head", start == 0)
return bool(value)
def _backend_owns_final_head(backend: Any, end: int) -> bool:
value = getattr(backend, "owns_final_head", None)
if value is None:
value = getattr(backend, "is_tail", False)
return bool(value)
def _backend_total_layers(backend: Any, selection: Any | None) -> int | None:
value = getattr(backend, "total_layers", None)
if isinstance(value, int) and value > 0:
return value
if selection is None:
return None
total = getattr(selection, "total_layers", None)
if isinstance(total, int) and total > 0:
return total
return None
def _tensor_names_from_backend(backend: Any) -> tuple[str, ...]:
for attr in ("loaded_tensor_names", "tensor_names", "tensor_inventory"):
value = getattr(backend, attr, None)
names = _normalise_tensor_names(value)
if names:
return names
return ()
def _normalise_tensor_names(value: Any) -> tuple[str, ...]:
if value is None:
return ()
if isinstance(value, Mapping):
items = value.keys()
else:
try:
items = list(value)
except TypeError:
return ()
names = [str(item) for item in items if isinstance(item, str) and item.strip()]
return tuple(names)
def _tensor_belongs_to_range(
tensor_name: str,
start_layer: int,
end_layer: int,
total_layers: int | None,
) -> bool:
layer = _layer_index(tensor_name)
if layer is not None:
return start_layer <= layer <= end_layer
if start_layer == 0 and _is_head_tensor(tensor_name):
return True
if total_layers is not None and end_layer >= total_layers - 1 and _is_tail_tensor(
tensor_name, total_layers=total_layers, loaded_end=end_layer
):
return True
return False
def _layer_index(tensor_name: str) -> int | None:
match = _BLOCK_RE.match(tensor_name)
if match is None:
return None
return int(match.group(1))
def _is_head_tensor(tensor_name: str) -> bool:
lowered = tensor_name.lower()
return lowered in _HEAD_TENSOR_NAMES or any(
lowered.startswith(prefix)
for prefix in ("token_embd.", "tok_embeddings.", "embed_tokens.")
)
def _is_tail_tensor(
tensor_name: str,
*,
total_layers: int | None,
loaded_end: int,
) -> bool:
lowered = tensor_name.lower()
if lowered in _TAIL_TENSOR_NAMES:
return True
if total_layers is not None and loaded_end >= total_layers - 1:
return any(
lowered.startswith(prefix)
for prefix in ("output_norm.", "final_norm.", "norm.")
)
return False

918
packages/node/meshnet_node/hot_kv_state.py
View File

@@ -1,918 +0,0 @@
"""Isolated concurrent local Hot KV State for distributed Shards (DGR-007).
Hot KV State stays local to the node serving a Shard (RALPH runtime decision #7).
A concurrent server must map each ``(Route Session ID, route epoch)`` to an
isolated bounded KV context (decision #8) so that one request can never clear or
corrupt another's cache.
This module owns the *lifecycle and storage* of that state and is deliberately
backend-agnostic:
* :class:`HotKvStateManager` is the single mutation entry point. It maps
``(session_id, route_epoch)`` to a :class:`SessionCache`, allocates KV **only
for the owned layer range**, and enforces a byte budget, a session cap, and a
TTL through LRU/TTL eviction. It rejects stale route epochs and incompatible
cache recipes, and returns an **explicit** :class:`CacheMiss` when state the
caller expected is gone (evicted, released, desynchronised, or never held) so
the head degrades to a from-token-zero re-prefill instead of corrupting output
(RALPH decision #14: unverified KV is never migrated silently).
* :class:`LayerKvCache` / :class:`SessionCache` are the per-owned-layer K/V
containers. They are plain ``numpy`` arrays so the default deterministic test
suite needs no torch, GPU, download, or API credit; the pinned llama.cpp worker
(DGR-008) maps a llama sequence onto the same container contract.
* :class:`KvBoundaryAdapter` wraps a KV-aware ``ShardComputation`` (the DGR-006
duck type plus ``run_layers_cached``) so a Shard can run cached prefill/decode
through the manager while honouring the architecture-defined boundary contract
(head embeds tokens, middle/tail bypass embedding, non-tail emits the
unnormalized residual, tail samples).
The manager owns *all* cache mutation: a computation reads the existing cache and
returns the new K/V for the appended positions, and the manager decides whether
that append fits the budget. That keeps eviction, accounting, and isolation in one
place instead of scattered across backends.
"""
from __future__ import annotations
import threading
import time
from collections import OrderedDict
from dataclasses import dataclass, field
from enum import Enum
from typing import Any, Callable, Mapping
import numpy as np
from meshnet_node.boundary_adapter import (
BOUNDARY_SCHEMA_VERSION,
BoundaryBundle,
BoundaryContractError,
SamplingContract,
ShardRole,
TailOutput,
certified_architecture,
role_for_range,
)
from meshnet_node.runtime_recipe import compatibility_fingerprint
class HotKvStateError(RuntimeError):
"""Base class for Hot KV State errors."""
class StaleRouteEpochError(HotKvStateError):
"""Raised when a request references a route epoch older than the current one.
A newer route epoch means the route was re-planned; the old epoch's KV is
unverified against the new plan and must never be silently reused.
"""
class IncompatibleCacheRecipeError(HotKvStateError):
"""Raised when a request's cache recipe does not match the loaded shard.
A different quantization / dtype / owned range / architecture produces a KV
layout this node cannot reuse without corrupting output.
"""
class KvBudgetExceededError(HotKvStateError):
"""Raised when a single session cannot fit the configured byte budget.
Other sessions are evicted first (LRU); this fires only when even one session
alone exceeds the budget, which is a misconfiguration rather than pressure.
"""
class KvCacheMissError(HotKvStateError):
"""Raised by the strict accessor when expected session state is absent.
Prefer :meth:`HotKvStateManager.resolve`, which returns a structured
:class:`CacheMiss` instead of raising, when the caller wants to fall back to a
stateless re-prefill.
"""
def __init__(self, miss: "CacheMiss") -> None:
super().__init__(str(miss))
self.miss = miss
class CacheMissReason(str, Enum):
"""Why a lookup produced a cache miss (all benign; retry from token zero)."""
UNKNOWN_SESSION = "unknown-session"
EVICTED_TTL = "evicted-ttl"
EVICTED_LRU = "evicted-lru"
RELEASED = "released"
SUPERSEDED_EPOCH = "superseded-epoch"
SEQ_LEN_MISMATCH = "seq-len-mismatch"
@dataclass(frozen=True)
class CacheMiss:
"""Explicit cache-miss response the head can act on (re-prefill).
This is a value, not an exception: the native protocol carries a cache
expectation/result, and a miss is a normal, expected outcome under eviction.
"""
session_id: str
route_epoch: int
reason: CacheMissReason
detail: str = ""
def __str__(self) -> str:
suffix = f": {self.detail}" if self.detail else ""
return (
f"cache miss for session {self.session_id[:8]} epoch "
f"{self.route_epoch} ({self.reason.value}){suffix}"
)
@dataclass(frozen=True)
class KvCacheRecipe:
"""The identity of a Shard's KV layout, used to reject incompatible reuse.
Two recipes are compatible iff their fingerprints match — same certified
architecture, KV dtype, head geometry, and owned layer range within the same
whole-model layer count.
"""
architecture_adapter: str
kv_dtype: str
n_kv_heads: int
head_dim: int
total_layers: int
start_layer: int
end_layer: int
boundary_schema_version: int = BOUNDARY_SCHEMA_VERSION
def __post_init__(self) -> None:
# Fail closed on architecture identity (shared with the boundary adapter).
certified_architecture(self.architecture_adapter)
if self.n_kv_heads <= 0:
raise ValueError("n_kv_heads must be positive")
if self.head_dim <= 0:
raise ValueError("head_dim must be positive")
try:
np.dtype(self.kv_dtype)
except TypeError as exc: # pragma: no cover - defensive
raise ValueError(f"invalid kv_dtype {self.kv_dtype!r}") from exc
# role_for_range validates 0 <= start <= end <= total_layers - 1.
role_for_range(self.start_layer, self.end_layer, self.total_layers)
if self.boundary_schema_version < 1:
raise ValueError("boundary_schema_version must be >= 1")
@property
def owned_layers(self) -> tuple[int, ...]:
return tuple(range(self.start_layer, self.end_layer + 1))
@property
def role(self) -> ShardRole:
return role_for_range(self.start_layer, self.end_layer, self.total_layers)
def bytes_per_token(self) -> int:
"""Bytes of KV one token adds across *owned* layers (keys + values)."""
itemsize = np.dtype(self.kv_dtype).itemsize
per_layer = 2 * self.n_kv_heads * self.head_dim * itemsize
return per_layer * len(self.owned_layers)
def fingerprint(self) -> str:
return compatibility_fingerprint(
{
"kind": "hot-kv-recipe",
# Canonicalize the architecture so 'llama' / 'LlamaForCausalLM'
# map to the same fingerprint (they are the same layout).
"architecture_adapter": certified_architecture(
self.architecture_adapter
).adapter,
"kv_dtype": np.dtype(self.kv_dtype).name,
"n_kv_heads": self.n_kv_heads,
"head_dim": self.head_dim,
"total_layers": self.total_layers,
"start_layer": self.start_layer,
"end_layer": self.end_layer,
"boundary_schema_version": self.boundary_schema_version,
}
)
def is_compatible(self, other: "KvCacheRecipe") -> bool:
return self.fingerprint() == other.fingerprint()
class LayerKvCache:
"""K/V storage for a single owned layer; sequence axis is 0.
Keys and values are ``(seq, n_kv_heads, head_dim)``. Backends store the
position-encoded (post-RoPE) keys so a decode step only appends the new rows.
"""
__slots__ = ("layer_index", "n_kv_heads", "head_dim", "dtype", "keys", "values")
def __init__(
self, layer_index: int, n_kv_heads: int, head_dim: int, dtype: Any
) -> None:
self.layer_index = int(layer_index)
self.n_kv_heads = int(n_kv_heads)
self.head_dim = int(head_dim)
self.dtype = np.dtype(dtype)
self.keys = np.empty((0, self.n_kv_heads, self.head_dim), dtype=self.dtype)
self.values = np.empty((0, self.n_kv_heads, self.head_dim), dtype=self.dtype)
@property
def length(self) -> int:
return int(self.keys.shape[0])
def _validate(self, array: np.ndarray, name: str) -> np.ndarray:
arr = np.asarray(array, dtype=self.dtype)
if arr.ndim != 3 or arr.shape[1:] != (self.n_kv_heads, self.head_dim):
raise ValueError(
f"layer {self.layer_index} {name} must be "
f"(seq, {self.n_kv_heads}, {self.head_dim}), got {arr.shape}"
)
return arr
def append(self, keys: np.ndarray, values: np.ndarray) -> int:
k = self._validate(keys, "keys")
v = self._validate(values, "values")
if k.shape[0] != v.shape[0]:
raise ValueError(
f"layer {self.layer_index} keys/values disagree on token count "
f"({k.shape[0]} vs {v.shape[0]})"
)
self.keys = np.concatenate([self.keys, k], axis=0)
self.values = np.concatenate([self.values, v], axis=0)
return self.length
def truncate(self, length: int) -> None:
length = max(0, int(length))
self.keys = self.keys[:length]
self.values = self.values[:length]
@property
def nbytes(self) -> int:
return int(self.keys.nbytes + self.values.nbytes)
@dataclass
class SessionCache:
"""Isolated per-``(session_id, epoch)`` KV context over the owned layers only."""
session_id: str
route_epoch: int
recipe: KvCacheRecipe
layers: "OrderedDict[int, LayerKvCache]"
created_tick: float
last_tick: float
released: bool = False
@property
def seq_len(self) -> int:
if not self.layers:
return 0
# All owned layers advance in lockstep; report the first owned layer.
return next(iter(self.layers.values())).length
@property
def owned_layers(self) -> tuple[int, ...]:
return tuple(self.layers.keys())
def layer(self, index: int) -> LayerKvCache:
try:
return self.layers[index]
except KeyError:
raise KeyError(
f"layer {index} is not owned by this shard "
f"(owned {list(self.layers)})"
) from None
def read_only_layers(self) -> Mapping[int, LayerKvCache]:
"""The current per-layer caches a computation reads to attend over."""
return dict(self.layers)
def _append(self, kv_by_layer: Mapping[int, Any]) -> int:
provided = set(kv_by_layer)
owned = set(self.layers)
if provided != owned:
raise ValueError(
f"append must cover exactly the owned layers {sorted(owned)}, "
f"got {sorted(provided)}"
)
# Pre-validate token counts so a partial append never desynchronises the
# owned layers (append is all-or-nothing).
new_counts = set()
for idx, (keys, _values) in kv_by_layer.items():
new_counts.add(int(np.asarray(keys).shape[0]))
if len(new_counts) != 1:
raise ValueError(
f"append token counts disagree across layers: {sorted(new_counts)}"
)
for idx, (keys, values) in kv_by_layer.items():
self.layers[idx].append(keys, values)
return self.seq_len
def _truncate(self, length: int) -> None:
for cache in self.layers.values():
cache.truncate(length)
@property
def nbytes(self) -> int:
return sum(cache.nbytes for cache in self.layers.values())
@dataclass(frozen=True)
class HotKvStateConfig:
"""Bounds for the manager: memory budget, session cap, and idle TTL."""
budget_bytes: int = 64 * 1024 * 1024
max_sessions: int = 8
ttl_seconds: float = 600.0
miss_history: int = 256
def __post_init__(self) -> None:
if self.budget_bytes <= 0:
raise ValueError("budget_bytes must be positive")
if self.max_sessions < 1:
raise ValueError("max_sessions must be >= 1")
if self.ttl_seconds < 0:
raise ValueError("ttl_seconds must be >= 0")
if self.miss_history < 0:
raise ValueError("miss_history must be >= 0")
class HotKvStateManager:
"""Concurrent, bounded map of ``(session_id, epoch)`` to an isolated KV context."""
def __init__(
self,
recipe: KvCacheRecipe,
config: HotKvStateConfig | None = None,
*,
clock: Callable[[], float] | None = None,
) -> None:
self.recipe = recipe
self.config = config or HotKvStateConfig()
self._clock = clock or time.monotonic
self._sessions: "OrderedDict[tuple[str, int], SessionCache]" = OrderedDict()
self._latest_epoch: dict[str, int] = {}
self._misses: "OrderedDict[tuple[str, int], CacheMiss]" = OrderedDict()
self._lock = threading.RLock()
# -- introspection --------------------------------------------------------
@property
def total_bytes(self) -> int:
with self._lock:
return sum(s.nbytes for s in self._sessions.values())
@property
def session_count(self) -> int:
with self._lock:
self._evict_expired_locked(self._clock())
return len(self._sessions)
def session_keys(self) -> list[tuple[str, int]]:
with self._lock:
return list(self._sessions.keys())
# -- lifecycle ------------------------------------------------------------
def open(
self,
session_id: str,
route_epoch: int,
*,
recipe: KvCacheRecipe | None = None,
) -> SessionCache:
"""Create (or replace) a fresh, empty isolated context for the session.
A higher route epoch supersedes and frees any earlier epoch for the same
session id; an older epoch is rejected as stale.
"""
self._require_text(session_id, "session_id")
route_epoch = self._require_epoch(route_epoch)
with self._lock:
self._check_recipe(recipe)
self._validate_epoch_locked(session_id, route_epoch)
now = self._clock()
self._evict_expired_locked(now)
self._supersede_older_epochs_locked(session_id, route_epoch)
key = (session_id, route_epoch)
# A re-open at the same epoch replaces the prior context entirely.
self._sessions.pop(key, None)
layers: "OrderedDict[int, LayerKvCache]" = OrderedDict(
(
idx,
LayerKvCache(
idx,
self.recipe.n_kv_heads,
self.recipe.head_dim,
self.recipe.kv_dtype,
),
)
for idx in self.recipe.owned_layers
)
session = SessionCache(
session_id=session_id,
route_epoch=route_epoch,
recipe=self.recipe,
layers=layers,
created_tick=now,
last_tick=now,
)
self._sessions[key] = session
self._latest_epoch[session_id] = route_epoch
self._misses.pop(key, None)
self._enforce_capacity_locked(protect=key, incoming_bytes=0)
return session
def append(
self,
session_id: str,
route_epoch: int,
kv_by_layer: Mapping[int, Any],
*,
recipe: KvCacheRecipe | None = None,
expected_seq_len: int | None = None,
) -> SessionCache:
"""Append new K/V (prefill or decode) to an existing isolated context.
The computation supplies exactly the owned layers' new keys/values. The
manager evicts other sessions (LRU) to fit the byte budget before growing
this one, and raises :class:`KvBudgetExceededError` only if this session
alone cannot fit.
"""
route_epoch = self._require_epoch(route_epoch)
with self._lock:
self._check_recipe(recipe)
self._validate_epoch_locked(session_id, route_epoch)
session = self._require_live_locked(session_id, route_epoch)
if expected_seq_len is not None and session.seq_len != expected_seq_len:
miss = self._drop_and_record_locked(
(session_id, route_epoch),
CacheMissReason.SEQ_LEN_MISMATCH,
detail=f"cache holds {session.seq_len}, caller expected "
f"{expected_seq_len}",
)
raise KvCacheMissError(miss)
n_new = self._new_token_count(kv_by_layer)
incoming = n_new * self.recipe.bytes_per_token()
self._enforce_capacity_locked(
protect=(session_id, route_epoch), incoming_bytes=incoming
)
session._append(kv_by_layer)
session.last_tick = self._clock()
self._sessions.move_to_end((session_id, route_epoch))
return session
def truncate(
self, session_id: str, route_epoch: int, length: int
) -> SessionCache:
"""Drop cached positions beyond ``length`` (rollback) for one session."""
route_epoch = self._require_epoch(route_epoch)
with self._lock:
self._validate_epoch_locked(session_id, route_epoch)
session = self._require_live_locked(session_id, route_epoch)
if length < 0:
raise ValueError("truncate length must be >= 0")
session._truncate(length)
session.last_tick = self._clock()
self._sessions.move_to_end((session_id, route_epoch))
return session
def release(self, session_id: str, route_epoch: int) -> bool:
"""Free one session's context; other sessions are untouched.
Returns True if a live context was freed. A later lookup for the released
key yields an explicit :class:`CacheMiss`.
"""
route_epoch = self._require_epoch(route_epoch)
with self._lock:
key = (session_id, route_epoch)
existed = key in self._sessions
self._drop_and_record_locked(key, CacheMissReason.RELEASED)
return existed
# -- lookup ---------------------------------------------------------------
def resolve(
self,
session_id: str,
route_epoch: int,
*,
recipe: KvCacheRecipe | None = None,
expected_seq_len: int | None = None,
) -> SessionCache | CacheMiss:
"""Return the live context or an explicit :class:`CacheMiss`.
Rejects stale epochs and incompatible recipes (both are protocol
violations, not benign misses).
"""
route_epoch = self._require_epoch(route_epoch)
with self._lock:
self._check_recipe(recipe)
self._validate_epoch_locked(session_id, route_epoch)
now = self._clock()
self._evict_expired_locked(now)
key = (session_id, route_epoch)
session = self._sessions.get(key)
if session is None:
return self._recorded_miss_locked(key)
if expected_seq_len is not None and session.seq_len != expected_seq_len:
return self._drop_and_record_locked(
key,
CacheMissReason.SEQ_LEN_MISMATCH,
detail=f"cache holds {session.seq_len}, caller expected "
f"{expected_seq_len}",
)
session.last_tick = now
self._sessions.move_to_end(key)
return session
def get(
self,
session_id: str,
route_epoch: int,
*,
recipe: KvCacheRecipe | None = None,
expected_seq_len: int | None = None,
) -> SessionCache:
"""Strict accessor: raises :class:`KvCacheMissError` on a miss."""
result = self.resolve(
session_id,
route_epoch,
recipe=recipe,
expected_seq_len=expected_seq_len,
)
if isinstance(result, CacheMiss):
raise KvCacheMissError(result)
return result
# -- internals ------------------------------------------------------------
def _check_recipe(self, recipe: KvCacheRecipe | None) -> None:
if recipe is not None and not self.recipe.is_compatible(recipe):
raise IncompatibleCacheRecipeError(
"request cache recipe does not match this shard's loaded recipe "
f"(request {recipe.fingerprint()} vs shard {self.recipe.fingerprint()})"
)
def _validate_epoch_locked(self, session_id: str, route_epoch: int) -> None:
latest = self._latest_epoch.get(session_id)
if latest is not None and route_epoch < latest:
raise StaleRouteEpochError(
f"session {session_id[:8]} route epoch {route_epoch} is stale; "
f"current epoch is {latest}"
)
def _supersede_older_epochs_locked(
self, session_id: str, route_epoch: int
) -> None:
stale_keys = [
key
for key in self._sessions
if key[0] == session_id and key[1] < route_epoch
]
for key in stale_keys:
self._drop_and_record_locked(key, CacheMissReason.SUPERSEDED_EPOCH)
def _require_live_locked(
self, session_id: str, route_epoch: int
) -> SessionCache:
now = self._clock()
self._evict_expired_locked(now)
key = (session_id, route_epoch)
session = self._sessions.get(key)
if session is None:
raise KvCacheMissError(self._recorded_miss_locked(key))
return session
def _new_token_count(self, kv_by_layer: Mapping[int, Any]) -> int:
owned = set(self.recipe.owned_layers)
if set(kv_by_layer) != owned:
raise ValueError(
f"append must cover exactly the owned layers {sorted(owned)}, "
f"got {sorted(kv_by_layer)}"
)
counts = {int(np.asarray(k).shape[0]) for k, _ in kv_by_layer.values()}
if len(counts) != 1:
raise ValueError(
f"append token counts disagree across layers: {sorted(counts)}"
)
return counts.pop()
def _enforce_capacity_locked(
self, *, protect: tuple[str, int], incoming_bytes: int
) -> None:
# Session cap: evict LRU sessions other than the protected one.
while len(self._sessions) > self.config.max_sessions:
victim = self._lru_victim_locked(protect)
if victim is None:
break
self._drop_and_record_locked(victim, CacheMissReason.EVICTED_LRU)
# Byte budget: the protected session's own footprint after the append.
protected = self._sessions.get(protect)
protected_bytes = (protected.nbytes if protected is not None else 0) + incoming_bytes
if protected_bytes > self.config.budget_bytes:
raise KvBudgetExceededError(
f"session {protect[0][:8]} needs {protected_bytes} bytes which "
f"exceeds the KV budget {self.config.budget_bytes}"
)
# Evict other LRU sessions until the whole store fits with the append.
while self._total_bytes_locked() + incoming_bytes > self.config.budget_bytes:
victim = self._lru_victim_locked(protect)
if victim is None:
break
self._drop_and_record_locked(victim, CacheMissReason.EVICTED_LRU)
def _lru_victim_locked(self, protect: tuple[str, int]) -> tuple[str, int] | None:
for key in self._sessions: # OrderedDict iterates oldest-first.
if key != protect:
return key
return None
def _total_bytes_locked(self) -> int:
return sum(s.nbytes for s in self._sessions.values())
def _evict_expired_locked(self, now: float) -> None:
ttl = self.config.ttl_seconds
if ttl <= 0:
return
expired = [
key
for key, session in self._sessions.items()
if now - session.last_tick > ttl
]
for key in expired:
self._drop_and_record_locked(key, CacheMissReason.EVICTED_TTL)
def _drop_and_record_locked(
self,
key: tuple[str, int],
reason: CacheMissReason,
*,
detail: str = "",
) -> CacheMiss:
session = self._sessions.pop(key, None)
if session is not None:
session.released = True
miss = CacheMiss(
session_id=key[0], route_epoch=key[1], reason=reason, detail=detail
)
self._record_miss_locked(key, miss)
return miss
def _record_miss_locked(self, key: tuple[str, int], miss: CacheMiss) -> None:
if self.config.miss_history <= 0:
return
self._misses.pop(key, None)
self._misses[key] = miss
while len(self._misses) > self.config.miss_history:
self._misses.popitem(last=False)
def _recorded_miss_locked(self, key: tuple[str, int]) -> CacheMiss:
recorded = self._misses.get(key)
if recorded is not None:
return recorded
return CacheMiss(
session_id=key[0],
route_epoch=key[1],
reason=CacheMissReason.UNKNOWN_SESSION,
)
@staticmethod
def _require_text(value: Any, name: str) -> str:
if not isinstance(value, str) or not value.strip():
raise ValueError(f"{name} must be a non-empty string")
return value
@staticmethod
def _require_epoch(value: Any) -> int:
if isinstance(value, bool) or not isinstance(value, int):
raise ValueError("route_epoch must be an integer")
if value < 0:
raise ValueError("route_epoch must be >= 0")
return value
def kv_recipe_for(computation: Any) -> KvCacheRecipe:
"""Build a :class:`KvCacheRecipe` from a KV-aware ``ShardComputation``.
The computation exposes the DGR-006 duck type plus KV geometry
(``n_kv_heads``, ``head_dim``, ``kv_dtype``).
"""
return KvCacheRecipe(
architecture_adapter=str(getattr(computation, "architecture_adapter")),
kv_dtype=str(getattr(computation, "kv_dtype", "float32")),
n_kv_heads=int(getattr(computation, "n_kv_heads")),
head_dim=int(getattr(computation, "head_dim")),
total_layers=int(getattr(computation, "total_layers")),
start_layer=int(getattr(computation, "start_layer")),
end_layer=int(getattr(computation, "end_layer")),
)
@dataclass
class KvBoundaryAdapter:
"""KV-aware boundary driver: cached prefill/decode through the manager.
Mirrors the DGR-006 :class:`~meshnet_node.boundary_adapter.BoundaryAdapter`
contract (head embeds tokens, middle/tail bypass embedding and consume the
unnormalized residual bundle, non-tail emits the unnormalized residual, tail
normalizes + heads + prunes + samples) but threads a per-session KV context.
The wrapped computation must additionally expose::
run_layers_cached(hidden, *, positions, past_kv)
-> (hidden_out, {layer_index: (new_keys, new_values)})
reading ``past_kv`` (the current per-owned-layer caches) and returning the new
position-encoded K/V for the appended positions only. The manager, not the
computation, commits those K/V so eviction and budget stay centralized.
"""
computation: Any
manager: HotKvStateManager
sampling: SamplingContract = field(default_factory=SamplingContract.greedy)
architecture: Any = field(init=False)
role: ShardRole = field(init=False)
start_layer: int = field(init=False)
end_layer: int = field(init=False)
total_layers: int = field(init=False)
recipe: KvCacheRecipe = field(init=False)
def __post_init__(self) -> None:
arch_name = getattr(self.computation, "architecture_adapter", None)
self.architecture = certified_architecture(arch_name)
self.start_layer = int(getattr(self.computation, "start_layer"))
self.end_layer = int(getattr(self.computation, "end_layer"))
self.total_layers = int(getattr(self.computation, "total_layers"))
self.role = role_for_range(self.start_layer, self.end_layer, self.total_layers)
self.recipe = kv_recipe_for(self.computation)
if not self.manager.recipe.is_compatible(self.recipe):
raise IncompatibleCacheRecipeError(
"manager recipe does not match this computation's KV recipe"
)
@property
def is_head(self) -> bool:
return self.role.owns_embedding
@property
def is_tail(self) -> bool:
return self.role.owns_final_head
def prefill(
self,
session_id: str,
route_epoch: int,
*,
token_ids: Any | None = None,
boundary: BoundaryBundle | None = None,
) -> BoundaryBundle | TailOutput:
"""Open a fresh isolated context and run the prompt through this range."""
session = self.manager.open(session_id, route_epoch, recipe=self.recipe)
return self._run_step(session, token_ids, boundary)
def decode(
self,
session_id: str,
route_epoch: int,
*,
token_ids: Any | None = None,
boundary: BoundaryBundle | None = None,
expected_seq_len: int | None = None,
) -> BoundaryBundle | TailOutput | CacheMiss:
"""Append one (or more) decode positions to an existing context.
Returns an explicit :class:`CacheMiss` if the context is gone so the head
can re-prefill from token zero instead of corrupting output.
"""
resolved = self.manager.resolve(
session_id,
route_epoch,
recipe=self.recipe,
expected_seq_len=expected_seq_len,
)
if isinstance(resolved, CacheMiss):
return resolved
return self._run_step(resolved, token_ids, boundary)
# -- internals ------------------------------------------------------------
def _run_step(
self,
session: SessionCache,
token_ids: Any | None,
boundary: BoundaryBundle | None,
) -> BoundaryBundle | TailOutput:
prev_len = session.seq_len
hidden, positions = self._ingest(prev_len, token_ids, boundary)
hidden_out, new_kv = self.computation.run_layers_cached(
hidden, positions=positions, past_kv=session.read_only_layers()
)
self.manager.append(
session.session_id,
session.route_epoch,
new_kv,
recipe=self.recipe,
expected_seq_len=prev_len,
)
if self.is_tail:
return self._emit_tail(hidden_out)
return self._emit_boundary(hidden_out, positions)
def _ingest(
self,
prev_len: int,
token_ids: Any | None,
boundary: BoundaryBundle | None,
) -> tuple[np.ndarray, np.ndarray]:
if self.role.owns_embedding:
if token_ids is None:
raise BoundaryContractError(
"the head owns token embedding and must receive token IDs"
)
if boundary is not None:
raise BoundaryContractError(
"the head owns token embedding; it must not receive a boundary "
"bundle from an upstream range"
)
ids = np.asarray(token_ids)
if ids.ndim == 1:
ids = ids[None, :]
if ids.ndim != 2:
raise BoundaryContractError("token IDs must be (seq,) or (batch, seq)")
hidden = np.asarray(self.computation.embed_tokens(ids))
n_new = ids.shape[1]
positions = np.broadcast_to(
np.arange(prev_len, prev_len + n_new, dtype=np.int64),
ids.shape,
).copy()
return hidden, positions
# Middle / tail: consume the boundary bundle (the unnormalized residual).
if token_ids is not None:
raise BoundaryContractError(
"middle/tail Shards bypass token embedding; they must not receive "
"token IDs"
)
if boundary is None:
raise BoundaryContractError(
"middle/tail Shards must receive the named boundary bundle"
)
self._check_boundary(boundary)
return np.asarray(boundary.residual), np.asarray(boundary.positions)
def _check_boundary(self, boundary: BoundaryBundle) -> None:
if certified_architecture(boundary.architecture_adapter) is not self.architecture:
raise BoundaryContractError(
f"boundary bundle architecture {boundary.architecture_adapter!r} "
f"does not match this Shard's adapter {self.architecture.adapter!r}"
)
if boundary.schema_version != self.architecture.boundary_schema_version:
raise BoundaryContractError(
f"boundary schema v{boundary.schema_version} is not supported by "
f"this Shard (expects v{self.architecture.boundary_schema_version})"
)
if boundary.tensor_name != self.architecture.boundary_tensor_name:
raise BoundaryContractError(
f"boundary tensor {boundary.tensor_name!r} is not the "
f"architecture-defined {self.architecture.boundary_tensor_name!r}"
)
if boundary.normalized:
raise BoundaryContractError(
"boundary bundle is normalized; a Shard range must receive the "
"UNNORMALIZED architecture-defined residual"
)
if boundary.next_layer != self.start_layer:
raise BoundaryContractError(
f"boundary hands over at layer {boundary.next_layer} but this "
f"Shard starts at layer {self.start_layer}"
)
def _emit_boundary(
self, hidden: np.ndarray, positions: np.ndarray
) -> BoundaryBundle:
return BoundaryBundle(
architecture_adapter=self.architecture.adapter,
schema_version=self.architecture.boundary_schema_version,
tensor_name=self.architecture.boundary_tensor_name,
residual=np.asarray(hidden),
positions=np.asarray(positions),
next_layer=self.end_layer + 1,
normalized=False,
)
def _emit_tail(self, hidden: np.ndarray) -> TailOutput:
hidden = np.asarray(hidden)
if self.architecture.prunes_rows_at_tail:
last_hidden = hidden[:, -1:, :]
else: # pragma: no cover - no certified architecture takes this path yet
last_hidden = hidden
if self.architecture.normalizes_before_head:
last_hidden = np.asarray(self.computation.final_norm(last_hidden))
logits = np.asarray(self.computation.lm_head(last_hidden))
last_logits = logits[:, -1, :]
token_id = self.sampling.sample(last_logits)
return TailOutput(token_id=token_id, logits=last_logits, sampling=self.sampling)

15
packages/node/meshnet_node/model_backend.py
View File

@@ -323,10 +323,6 @@ class TorchModelShard:
)
self.is_head = shard_start == 0
self.is_tail = shard_end >= self.total_layers - 1
self.loaded_shard_start = shard_start
self.loaded_shard_end = shard_end
self.owns_embedding = self.is_head
self.owns_final_head = self.is_tail
self.hidden_size = int(
getattr(self.model.config, "hidden_size", 0)
or getattr(self.model.config, "n_embd", 0)
@@ -348,17 +344,6 @@ class TorchModelShard:
ttl_seconds=float(os.environ.get("MESHNET_KV_TTL_SECONDS", "600")),
)
@property
def loaded_range(self) -> tuple[int, int]:
return self.loaded_shard_start, self.loaded_shard_end
@property
def endpoint_ownership(self) -> dict[str, bool]:
return {
"owns_embedding": self.owns_embedding,
"owns_final_head": self.owns_final_head,
}
def encode_prompt(self, prompt: str, session_id: str | None = None) -> TensorPayload:
if not self.is_head or self._embed_tokens is None:
raise ModelBackendError("text prompts can only be accepted by the head shard")

300
packages/node/meshnet_node/native_protocol/__init__.py
View File

@@ -1,300 +0,0 @@
"""Loader and helpers for the versioned gRPC Shard protocol (ADR-0024, DGR-002).
The ``.proto`` schema at ``packages/node/native/proto/shard_runtime.proto`` is the
single source of truth. Rather than commit generated stubs (which pin a protobuf
runtime version and drift from the schema), this package generates the Python
stubs on demand into a gitignored build directory and imports them. Generation is
reproducible: it shells out to the pinned ``grpc_tools.protoc`` with the exact
same flags as ``packages/node/native/scripts/generate_python.py``.
Typical use::
from meshnet_node import native_protocol as proto
pb2 = proto.load()
header = pb2.MessageHeader(work_id="w1", route_session_id="s1")
The checksum/fragment helpers encode the bounded-fragment tensor-bundle semantics
so callers (and DGR-008/DGR-009) do not re-derive them.
"""
from __future__ import annotations
import hashlib
import importlib
import importlib.util
import pathlib
import sys
import threading
import types
import zlib
# The wire schema version this build targets. Keep in sync with the
# ``SCHEMA_VERSION_1`` enum member in the .proto.
SCHEMA_VERSION = 1
_NATIVE_ROOT = pathlib.Path(__file__).resolve().parents[2] / "native"
PROTO_DIR = _NATIVE_ROOT / "proto"
PROTO_FILE = PROTO_DIR / "shard_runtime.proto"
# ``build/`` is globally gitignored, so generated stubs never enter version control.
GEN_DIR = _NATIVE_ROOT / "build" / "python"
_PB2_MODULE = "shard_runtime_pb2"
_GRPC_MODULE = "shard_runtime_pb2_grpc"
# Reentrant: load_grpc() holds the lock and calls load(), which re-acquires it.
_lock = threading.RLock()
_cached_pb2: types.ModuleType | None = None
_cached_grpc: types.ModuleType | None = None
class ProtocGenerationError(RuntimeError):
"""Raised when the protobuf stubs cannot be generated from the schema."""
def _needs_regen(target: pathlib.Path) -> bool:
if not target.exists():
return True
try:
return PROTO_FILE.stat().st_mtime > target.stat().st_mtime
except OSError:
return True
def generate(*, force: bool = False) -> pathlib.Path:
"""Generate ``shard_runtime_pb2{,_grpc}.py`` into :data:`GEN_DIR`.
Returns the output directory. Reproducible and idempotent: regenerates only
when the schema is newer than the stubs (or ``force`` is set). Requires the
pinned ``grpc_tools`` (available in the project ``.venv``).
"""
if not PROTO_FILE.exists():
raise ProtocGenerationError(f"schema not found: {PROTO_FILE}")
pb2_path = GEN_DIR / f"{_PB2_MODULE}.py"
if not force and not _needs_regen(pb2_path):
return GEN_DIR
try:
from grpc_tools import protoc
except ImportError as exc: # pragma: no cover - environment-dependent
raise ProtocGenerationError(
"grpc_tools is required to generate the Shard protocol stubs; "
"install grpcio-tools (present in the project .venv)."
) from exc
GEN_DIR.mkdir(parents=True, exist_ok=True)
well_known = _well_known_include()
args = [
"grpc_tools.protoc",
f"-I{PROTO_DIR}",
*([f"-I{well_known}"] if well_known else []),
f"--python_out={GEN_DIR}",
f"--grpc_python_out={GEN_DIR}",
str(PROTO_FILE.name),
]
# protoc resolves the proto by name relative to -I, so run with PROTO_DIR
# semantics by passing the bare filename plus the include path above.
rc = protoc.main([a for a in args])
if rc != 0:
raise ProtocGenerationError(
f"grpc_tools.protoc exited with status {rc} for {PROTO_FILE}"
)
if not pb2_path.exists(): # pragma: no cover - defensive
raise ProtocGenerationError(f"protoc did not produce {pb2_path}")
return GEN_DIR
def _well_known_include() -> str | None:
"""Bundled well-known .proto include dir shipped with grpc_tools, if any."""
try:
import grpc_tools
candidate = pathlib.Path(grpc_tools.__file__).parent / "_proto"
return str(candidate) if candidate.is_dir() else None
except Exception: # pragma: no cover - defensive
return None
def _import_generated(module_name: str) -> types.ModuleType:
gen_dir = str(GEN_DIR)
if gen_dir not in sys.path:
sys.path.insert(0, gen_dir)
if module_name in sys.modules:
return sys.modules[module_name]
return importlib.import_module(module_name)
def load(*, force: bool = False) -> types.ModuleType:
"""Return the generated ``shard_runtime_pb2`` module (messages only).
Generates the stubs on first use. Thread-safe and cached. Does not import
grpc; message serialization/round-trip needs only this module.
"""
global _cached_pb2
with _lock:
if _cached_pb2 is not None and not force:
return _cached_pb2
generate(force=force)
_cached_pb2 = _import_generated(_PB2_MODULE)
return _cached_pb2
def load_grpc(*, force: bool = False) -> types.ModuleType:
"""Return the generated ``shard_runtime_pb2_grpc`` module (service stubs).
Requires the ``grpc`` runtime. Use for building the C++/Python worker; the
round-trip/compat tests only need :func:`load`.
"""
global _cached_grpc
with _lock:
if _cached_grpc is not None and not force:
return _cached_grpc
generate(force=force)
load() # ensure the _pb2 module the grpc stub imports is present
_cached_grpc = _import_generated(_GRPC_MODULE)
return _cached_grpc
# ---------------------------------------------------------------------------
# Checksum + bounded-fragment helpers (shared bundle semantics)
# ---------------------------------------------------------------------------
# Algorithm-name strings mirror the ChecksumAlgorithm enum members without
# importing the generated module (so this table is usable before load()).
_CHECKSUM_CRC32C = "CHECKSUM_CRC32C"
_CHECKSUM_CRC32 = "CHECKSUM_CRC32"
_CHECKSUM_SHA256 = "CHECKSUM_SHA256"
_CHECKSUM_NONE = "CHECKSUM_NONE"
def _crc32c(data: bytes) -> int:
"""Castagnoli CRC32C (software table). Deterministic, no external deps."""
crc = 0xFFFFFFFF
for byte in data:
crc ^= byte
for _ in range(8):
crc = (crc >> 1) ^ (0x82F63B78 & -(crc & 1))
return crc ^ 0xFFFFFFFF
def compute_checksum(algorithm: int, data: bytes):
"""Build a ``Checksum`` message for ``data`` under the given enum value.
``algorithm`` is a ``ChecksumAlgorithm`` enum int from the generated module.
Uses only the standard library (crc32c software table, zlib.crc32, hashlib).
"""
pb2 = load()
name = pb2.ChecksumAlgorithm.Name(algorithm)
if name == _CHECKSUM_SHA256:
value = hashlib.sha256(data).digest()
elif name == _CHECKSUM_CRC32C:
value = _crc32c(data).to_bytes(4, "big")
elif name == _CHECKSUM_CRC32:
value = (zlib.crc32(data) & 0xFFFFFFFF).to_bytes(4, "big")
elif name == _CHECKSUM_NONE:
value = b""
else:
raise ValueError(f"unsupported checksum algorithm: {name}")
return pb2.Checksum(algorithm=algorithm, value=value)
def verify_checksum(checksum, data: bytes) -> bool:
"""True if ``checksum`` matches ``data`` (CHECKSUM_NONE always verifies)."""
pb2 = load()
if checksum.algorithm in (0, pb2.CHECKSUM_NONE):
return True
return compute_checksum(checksum.algorithm, data).value == checksum.value
def fragment_tensor(
*,
name: str,
shape,
dtype: int,
payload: bytes,
byte_order: int | None = None,
max_fragment_bytes: int = 1 << 20,
compression: int | None = None,
checksum_algorithm: int | None = None,
):
"""Build a :class:`NamedTensor` splitting ``payload`` into bounded fragments.
Fragments are ordered by ``byte_offset`` and each carries an optional
per-fragment checksum. ``payload`` is treated as already compressed if
``compression`` is set; this helper does not compress (that is the seam's
policy in ``activation_compression``), it only frames.
"""
if max_fragment_bytes <= 0:
raise ValueError("max_fragment_bytes must be positive")
pb2 = load()
if byte_order is None:
byte_order = pb2.BYTE_ORDER_LITTLE_ENDIAN
if compression is None:
compression = pb2.COMPRESSION_NONE
chunks = [
payload[i : i + max_fragment_bytes]
for i in range(0, len(payload), max_fragment_bytes)
] or [b""]
fragments = []
offset = 0
for index, chunk in enumerate(chunks):
frag = pb2.TensorFragment(
fragment_index=index,
fragment_count=len(chunks),
byte_offset=offset,
data=chunk,
)
if checksum_algorithm is not None:
frag.checksum.CopyFrom(compute_checksum(checksum_algorithm, chunk))
fragments.append(frag)
offset += len(chunk)
return pb2.NamedTensor(
name=name,
shape=list(shape),
dtype=dtype,
byte_order=byte_order,
total_byte_length=len(payload),
compression=compression,
fragments=fragments,
)
def reassemble_tensor(named_tensor) -> bytes:
"""Concatenate a :class:`NamedTensor`'s fragments back into the full payload.
Validates fragment ordering, total length, and any per-fragment checksums.
"""
fragments = sorted(named_tensor.fragments, key=lambda f: f.byte_offset)
out = bytearray()
for frag in fragments:
if frag.byte_offset != len(out):
raise ValueError(
f"non-contiguous fragment at offset {frag.byte_offset} "
f"(expected {len(out)})"
)
if frag.HasField("checksum") and not verify_checksum(frag.checksum, frag.data):
raise ValueError(f"fragment {frag.fragment_index} checksum mismatch")
out.extend(frag.data)
if named_tensor.total_byte_length and len(out) != named_tensor.total_byte_length:
raise ValueError(
f"reassembled length {len(out)} != declared "
f"{named_tensor.total_byte_length}"
)
return bytes(out)
__all__ = [
"SCHEMA_VERSION",
"PROTO_FILE",
"PROTO_DIR",
"GEN_DIR",
"ProtocGenerationError",
"generate",
"load",
"load_grpc",
"compute_checksum",
"verify_checksum",
"fragment_tensor",
"reassemble_tensor",
]

563
packages/node/meshnet_node/performance_contract.py
View File

@@ -1,563 +0,0 @@
"""Versioned performance contract metadata and stub benchmark runner for DGR-001.
This module captures the *contract* first: the model family, architecture
alignment, benchmark lanes, and stop condition that benchmark runs must
satisfy. It also runs the contract's lanes through a deterministic stub
backend so the report data shape exists end to end. It never downloads or
executes a model; real transformers / llama.cpp backends plug in behind the
same ``run()`` seam later.
"""
from __future__ import annotations
import argparse
import json
import time
import urllib.request
from dataclasses import dataclass
from pathlib import Path
from typing import Mapping
SCHEMA_VERSION = 1
CONTRACT_ID = "DGR-001"
DEFAULT_OUTPUT_PATH = Path(".scratch/distributed-gguf-runtime/evidence/DGR-001/performance-contract.json")
@dataclass(frozen=True)
class ModelTarget:
"""Architecture-aligned model target for the DGR-001 benchmark contract."""
name: str
architecture: str
safetensors_repo: str
safetensors_precision: str
gguf_repo: str
gguf_quant: str
gguf_size_gb: float
comparison_policy: str
rationale: str
def to_dict(self) -> dict:
return {
"name": self.name,
"architecture": self.architecture,
"safetensors_repo": self.safetensors_repo,
"safetensors_precision": self.safetensors_precision,
"gguf_repo": self.gguf_repo,
"gguf_quant": self.gguf_quant,
"gguf_size_gb": self.gguf_size_gb,
"comparison_policy": self.comparison_policy,
"rationale": self.rationale,
}
@dataclass(frozen=True)
class BenchmarkLane:
"""One side of the comparison the contract requires."""
id: str
runtime: str
device: str
recipe: str
concurrency_levels: tuple[int, ...]
def to_dict(self) -> dict:
return {
"id": self.id,
"runtime": self.runtime,
"device": self.device,
"recipe": self.recipe,
"concurrency_levels": list(self.concurrency_levels),
}
@dataclass(frozen=True)
class BenchmarkWorkload:
"""Identical request shape both recipes must run so speed stays comparable.
Pinning prompts, context lengths, output lengths, and sampling policy in the
versioned contract is what makes the safetensors-versus-GGUF numbers a
controlled comparison instead of two differently-configured runs.
"""
prompts: tuple[str, ...]
context_lengths: tuple[int, ...]
output_lengths: tuple[int, ...]
sampling_policy: str
def to_dict(self) -> dict:
return {
"prompts": list(self.prompts),
"context_lengths": list(self.context_lengths),
"output_lengths": list(self.output_lengths),
"sampling_policy": self.sampling_policy,
}
@dataclass(frozen=True)
class QualityPolicy:
"""Correctness/quality lane kept separate from the performance/fit lanes.
BF16 safetensors and Q2_K GGUF are not numerically equivalent, so quality is
measured as its own lane (output drift against the BF16 reference under a
documented tolerance) rather than assumed away by the speed/fit comparison.
"""
statement: str
reference_lane_runtime: str
measured_lane_runtime: str
max_output_drift: float
def to_dict(self) -> dict:
return {
"statement": self.statement,
"reference_lane_runtime": self.reference_lane_runtime,
"measured_lane_runtime": self.measured_lane_runtime,
"max_output_drift": self.max_output_drift,
}
@dataclass(frozen=True)
class ReleaseGate:
"""Versioned thresholds later release gates (DGR-014) consume unchanged.
Thresholds live in the contract, not in code, so the release gate cannot be
weakened after seeing implementation results.
"""
min_decode_speedup: float
max_artifact_bytes_ratio: float
max_memory_bytes_ratio: float
max_quality_drift: float
def to_dict(self) -> dict:
return {
"min_decode_speedup": self.min_decode_speedup,
"max_artifact_bytes_ratio": self.max_artifact_bytes_ratio,
"max_memory_bytes_ratio": self.max_memory_bytes_ratio,
"max_quality_drift": self.max_quality_drift,
}
@dataclass(frozen=True)
class PerformanceContract:
"""Machine-readable contract for the DGR-001 benchmark story."""
schema_version: int
story_id: str
model_target: ModelTarget
benchmark_lanes: tuple[BenchmarkLane, ...]
metrics: tuple[str, ...]
stop_condition: str
notes: tuple[str, ...] = ()
def to_dict(self) -> dict:
return {
"schema_version": self.schema_version,
"story_id": self.story_id,
"model_target": self.model_target.to_dict(),
"benchmark_lanes": [lane.to_dict() for lane in self.benchmark_lanes],
"metrics": list(self.metrics),
"stop_condition": self.stop_condition,
"notes": list(self.notes),
}
def write_json(self, path: str | Path) -> Path:
path = Path(path)
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(json.dumps(self.to_dict(), indent=2, sort_keys=True) + "\n", encoding="utf-8")
return path
DEFAULT_CONTRACT = PerformanceContract(
schema_version=SCHEMA_VERSION,
story_id=CONTRACT_ID,
model_target=ModelTarget(
name="DeepSeek-V2-Lite-Chat",
architecture="deepseek2",
safetensors_repo="deepseek-ai/DeepSeek-V2-Lite-Chat",
safetensors_precision="bfloat16",
gguf_repo="second-state/DeepSeek-V2-Lite-Chat-GGUF",
gguf_quant="Q2_K",
gguf_size_gb=6.43,
comparison_policy=(
"same model/revision, closest practical low-footprint precision pair: "
"BF16 safetensors versus Q2_K GGUF"
),
rationale=(
"Smallest DeepSeek-family benchmark anchor that still points toward "
"DeepSeek-V4-Flash; keeps the runtime on the DeepSeek2 path instead "
"of falling back to a tiny but architecture-mismatched smoke model."
),
),
benchmark_lanes=(
BenchmarkLane(
id="transformers-safetensors-cpu",
runtime="transformers",
device="cpu",
recipe="current safetensors recipe",
concurrency_levels=(1, 4),
),
BenchmarkLane(
id="llama-cpp-gguf-cpu",
runtime="llama.cpp",
device="cpu",
recipe="whole-model GGUF recipe",
concurrency_levels=(1, 4),
),
BenchmarkLane(
id="transformers-safetensors-gpu",
runtime="transformers",
device="gpu",
recipe="current safetensors recipe",
concurrency_levels=(1, 4),
),
BenchmarkLane(
id="llama-cpp-gguf-gpu",
runtime="llama.cpp",
device="gpu",
recipe="whole-model GGUF recipe",
concurrency_levels=(1, 4),
),
),
metrics=(
"ttft_ms",
"prefill_tok_per_sec",
"decode_tok_per_sec",
"p50_latency_ms",
"p95_latency_ms",
"aggregate_throughput_tok_per_sec",
"rss_bytes",
"vram_bytes",
"artifact_bytes",
"failure_count",
"output_drift",
),
stop_condition=(
"Stop if GGUF does not provide a meaningful speed or fit benefit over the "
"safetensors baseline for the chosen DeepSeek-family model target."
),
notes=(
"Real model execution stays opt-in and must keep model artifacts on the mounted drive.",
"Use the tiny fallback only for loader plumbing smoke tests; it does not replace the architecture-aligned baseline.",
),
)
def build_default_contract() -> PerformanceContract:
return DEFAULT_CONTRACT
BENCHMARK_SCHEMA_VERSION = 1
STUB_OUTPUT_TOKENS = ("mesh", "activation", "seam", "baseline")
# DeepSeek-V2-Lite is ~15.7B params at 2 bytes each; metadata only, nothing downloaded.
_SAFETENSORS_BF16_ARTIFACT_GB = 31.4
@dataclass(frozen=True)
class LaneSample:
"""Raw single-stream measurements one backend produces for a lane."""
ttft_ms: float
prefill_tok_per_sec: float
decode_tok_per_sec: float
rss_bytes: int
vram_bytes: int
artifact_bytes: int
output_tokens: tuple[str, ...]
failure_count: int = 0
def _gb(value: float) -> int:
return int(value * 1024**3)
class StubLaneBackend:
"""Deterministic placeholder measurements until real lane execution lands.
The numbers are synthetic but directionally shaped — the Q2_K GGUF loads a
far smaller artifact and decodes faster than BF16 safetensors — so the
comparison and stop-condition plumbing can be exercised in CI.
"""
source = "stub-backend"
# (runtime, device) -> (ttft_ms, prefill tok/s, decode tok/s, rss GB, vram GB)
_PROFILES = {
("transformers", "cpu"): (1800.0, 45.0, 6.0, 33.0, 0.0),
("llama.cpp", "cpu"): (950.0, 90.0, 14.0, 7.1, 0.0),
("transformers", "gpu"): (420.0, 850.0, 34.0, 4.0, 33.0),
("llama.cpp", "gpu"): (260.0, 640.0, 52.0, 1.5, 7.5),
}
def __init__(self, contract: PerformanceContract) -> None:
self._contract = contract
def run(self, lane: BenchmarkLane) -> LaneSample:
ttft_ms, prefill, decode, rss_gb, vram_gb = self._PROFILES[(lane.runtime, lane.device)]
artifact_gb = (
self._contract.model_target.gguf_size_gb
if lane.runtime == "llama.cpp"
else _SAFETENSORS_BF16_ARTIFACT_GB
)
return LaneSample(
ttft_ms=ttft_ms,
prefill_tok_per_sec=prefill,
decode_tok_per_sec=decode,
rss_bytes=_gb(rss_gb),
vram_bytes=_gb(vram_gb),
artifact_bytes=_gb(artifact_gb),
output_tokens=STUB_OUTPUT_TOKENS,
)
def _output_drift(tokens: tuple[str, ...], reference: tuple[str, ...]) -> float:
"""Fraction of positions where a lane's output diverges from its reference."""
length = max(len(tokens), len(reference))
if length == 0:
return 0.0
mismatches = sum(a != b for a, b in zip(tokens, reference)) + abs(len(tokens) - len(reference))
return round(mismatches / length, 4)
def _metrics_for(sample: LaneSample, concurrency: int, output_drift: float) -> dict:
# Stub concurrency model: batching scales throughput at 85% efficiency and
# stretches per-request token latency and TTFT accordingly.
efficiency = 1.0 if concurrency == 1 else 0.85
p50_latency_ms = round(1000.0 / (sample.decode_tok_per_sec * efficiency), 4)
return {
"ttft_ms": round(sample.ttft_ms * (1 + 0.1 * (concurrency - 1)), 4),
"prefill_tok_per_sec": round(sample.prefill_tok_per_sec * efficiency, 4),
"decode_tok_per_sec": round(sample.decode_tok_per_sec * efficiency, 4),
"p50_latency_ms": p50_latency_ms,
"p95_latency_ms": round(p50_latency_ms * 1.25, 4),
"aggregate_throughput_tok_per_sec": round(sample.decode_tok_per_sec * concurrency * efficiency, 4),
"rss_bytes": sample.rss_bytes,
"vram_bytes": sample.vram_bytes,
"artifact_bytes": sample.artifact_bytes,
"failure_count": sample.failure_count,
"output_drift": output_drift,
}
def _compare_device(lanes: list[tuple[BenchmarkLane, LaneSample]], device: str) -> dict:
by_runtime = {lane.runtime: (lane, sample) for lane, sample in lanes if lane.device == device}
safetensors_lane, safetensors = by_runtime["transformers"]
gguf_lane, gguf = by_runtime["llama.cpp"]
memory_metric = "vram_bytes" if device == "gpu" else "rss_bytes"
decode_speedup = round(gguf.decode_tok_per_sec / safetensors.decode_tok_per_sec, 4)
artifact_bytes_ratio = round(gguf.artifact_bytes / max(1, safetensors.artifact_bytes), 4)
return {
"safetensors_lane": safetensors_lane.id,
"gguf_lane": gguf_lane.id,
"decode_speedup": decode_speedup,
"ttft_speedup": round(safetensors.ttft_ms / max(0.001, gguf.ttft_ms), 4),
"artifact_bytes_ratio": artifact_bytes_ratio,
"memory_metric": memory_metric,
"memory_bytes_ratio": round(
getattr(gguf, memory_metric) / max(1, getattr(safetensors, memory_metric)), 4
),
"output_drift": _output_drift(gguf.output_tokens, safetensors.output_tokens),
"gguf_benefit": decode_speedup >= 1.10 or artifact_bytes_ratio <= 0.5,
}
def run_performance_benchmark(
contract: PerformanceContract = DEFAULT_CONTRACT,
backend: StubLaneBackend | None = None,
) -> dict:
"""Run every contract lane through a backend and compare GGUF to safetensors."""
backend = backend if backend is not None else StubLaneBackend(contract)
lanes = [(lane, backend.run(lane)) for lane in contract.benchmark_lanes]
references = {
lane.device: sample.output_tokens for lane, sample in lanes if lane.runtime == "transformers"
}
lane_reports = []
for lane, sample in lanes:
drift = _output_drift(sample.output_tokens, references.get(lane.device, sample.output_tokens))
lane_reports.append({
**lane.to_dict(),
"output_tokens": list(sample.output_tokens),
"results": [
{"concurrency": level, "metrics": _metrics_for(sample, level, drift)}
for level in lane.concurrency_levels
],
})
devices = sorted({lane.device for lane, _ in lanes})
comparisons = {device: _compare_device(lanes, device) for device in devices}
gguf_benefit = any(comparison["gguf_benefit"] for comparison in comparisons.values())
return {
"schema_version": BENCHMARK_SCHEMA_VERSION,
"story_id": contract.story_id,
"source": getattr(backend, "source", "custom-backend"),
"model_target": contract.model_target.to_dict(),
"lanes": lane_reports,
"comparisons": comparisons,
"stop_condition": {
"text": contract.stop_condition,
"gguf_benefit": gguf_benefit,
"triggered": not gguf_benefit,
},
}
def run_real_model_endpoint_benchmark(
endpoints: Mapping[str, str],
*,
model: str,
contract: PerformanceContract = DEFAULT_CONTRACT,
timeout: float = 120.0,
) -> dict:
"""Run one live OpenAI-compatible request per lane against supplied endpoints.
The caller provides one URL per benchmark lane. The runner measures the
request/response round-trip at the client boundary and reuses the same
contract schema as the deterministic stub.
"""
def _sample_for_lane(lane: BenchmarkLane, endpoint: str) -> LaneSample:
prompt = " ".join(contract.model_target.rationale.split()[:6])
body = json.dumps(
{
"model": model,
"messages": [{"role": "user", "content": prompt}],
"max_tokens": len(STUB_OUTPUT_TOKENS),
"temperature": 0,
}
).encode("utf-8")
request = urllib.request.Request(
f"{endpoint.rstrip('/')}/v1/chat/completions",
data=body,
headers={
"Content-Type": "application/json",
"X-Meshnet-Lane": lane.id,
},
method="POST",
)
started = time.monotonic()
with urllib.request.urlopen(request, timeout=timeout) as response:
response_body = response.read()
session_id = response.headers.get("X-Meshnet-Session", f"{lane.id}-session")
elapsed_ms = round((time.monotonic() - started) * 1000, 4)
payload = json.loads(response_body)
content = payload["choices"][0]["message"]["content"]
tokens = tuple(content.split())
token_count = max(1, len(tokens))
artifact_gb = (
contract.model_target.gguf_size_gb
if lane.runtime == "llama.cpp"
else _SAFETENSORS_BF16_ARTIFACT_GB
)
return LaneSample(
ttft_ms=elapsed_ms,
prefill_tok_per_sec=round(token_count / max(0.001, elapsed_ms / 1000), 4),
decode_tok_per_sec=round(token_count / max(0.001, elapsed_ms / 1000), 4),
rss_bytes=0,
vram_bytes=0,
artifact_bytes=_gb(artifact_gb),
output_tokens=tokens,
)
lanes = []
for lane in contract.benchmark_lanes:
if lane.id not in endpoints:
raise KeyError(f"missing endpoint for lane {lane.id}")
lanes.append((lane, _sample_for_lane(lane, endpoints[lane.id])))
references = {
lane.device: sample.output_tokens for lane, sample in lanes if lane.runtime == "transformers"
}
lane_reports = []
for lane, sample in lanes:
drift = _output_drift(sample.output_tokens, references.get(lane.device, sample.output_tokens))
lane_reports.append({
**lane.to_dict(),
"output_tokens": list(sample.output_tokens),
"results": [
{"concurrency": level, "metrics": _metrics_for(sample, level, drift)}
for level in lane.concurrency_levels
],
})
devices = sorted({lane.device for lane, _ in lanes})
comparisons = {device: _compare_device(lanes, device) for device in devices}
gguf_benefit = any(comparison["gguf_benefit"] for comparison in comparisons.values())
return {
"schema_version": BENCHMARK_SCHEMA_VERSION,
"story_id": contract.story_id,
"source": "real-model-endpoints",
"model_target": contract.model_target.to_dict(),
"lanes": lane_reports,
"comparisons": comparisons,
"stop_condition": {
"text": contract.stop_condition,
"gguf_benefit": gguf_benefit,
"triggered": not gguf_benefit,
},
}
def _parse_lane_endpoints(pairs: list[str], parser: argparse.ArgumentParser) -> dict[str, str]:
endpoints: dict[str, str] = {}
for pair in pairs:
lane_id, sep, url = pair.partition("=")
if not sep or not lane_id or not url:
parser.error(f"--live-endpoint expects LANE_ID=URL, got {pair!r}")
endpoints[lane_id] = url
return endpoints
def _write_report(report: dict, path: Path) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(json.dumps(report, indent=2, sort_keys=True) + "\n", encoding="utf-8")
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(description="Write the DGR-001 performance contract JSON")
parser.add_argument("--json-out", type=Path, default=DEFAULT_OUTPUT_PATH, help="output JSON path")
parser.add_argument(
"--benchmark-out",
type=Path,
default=None,
help="also run the deterministic stub benchmark and write its JSON report here",
)
parser.add_argument(
"--live-endpoint",
action="append",
default=None,
metavar="LANE_ID=URL",
help="lane-to-endpoint mapping for the live benchmark; repeat once per contract lane",
)
parser.add_argument(
"--live-model",
default=None,
help="model name sent to live endpoints (default: contract safetensors repo)",
)
parser.add_argument(
"--live-benchmark-out",
type=Path,
default=None,
help="run the live endpoint benchmark against --live-endpoint lanes and write its JSON report here",
)
args = parser.parse_args(argv)
if args.live_endpoint and args.live_benchmark_out is None:
parser.error("--live-endpoint requires --live-benchmark-out")
if args.live_benchmark_out is not None and not args.live_endpoint:
parser.error("--live-benchmark-out requires at least one --live-endpoint")
contract = build_default_contract()
path = contract.write_json(args.json_out)
print(path)
if args.benchmark_out is not None:
_write_report(run_performance_benchmark(contract), args.benchmark_out)
print(args.benchmark_out)
if args.live_endpoint:
report = run_real_model_endpoint_benchmark(
_parse_lane_endpoints(args.live_endpoint, parser),
model=args.live_model or contract.model_target.safetensors_repo,
contract=contract,
)
_write_report(report, args.live_benchmark_out)
print(args.live_benchmark_out)
return 0
if __name__ == "__main__": # pragma: no cover - CLI entry point
raise SystemExit(main())

10
packages/node/meshnet_node/recipes.json
View File

@@ -26,16 +26,6 @@
"params": {
"use_cache": false
}
},
{
"id": "llama-cpp-native",
"version": "1",
"backend_id": "llama.cpp",
"description": "Project-owned native GGUF worker behind the Meshnet control plane.",
"params": {
"worker_transport": "grpc",
"use_cache": true
}
}
]
}

375
packages/node/meshnet_node/runtime_recipe.py
View File

@@ -1,375 +0,0 @@
"""Exact artifact and runtime-recipe identity helpers.
The runtime recipe is the compatibility contract for one routable shard. It is
kept separate from the user-facing recipe catalogue so the tracker can compare
the exact execution footprint that was validated, not just a named recipe.
"""
from __future__ import annotations
import hashlib
import json
from dataclasses import dataclass
from typing import Any, Mapping
def _require_text(value: Any, field_name: str) -> str:
if not isinstance(value, str) or not value.strip():
raise ValueError(f"{field_name!r} must be a non-empty string")
return value
def _optional_text(value: Any, field_name: str) -> str | None:
if value is None:
return None
return _require_text(value, field_name)
def _sha256_text(text: str) -> str:
return hashlib.sha256(text.encode("utf-8")).hexdigest()
def _stable_json(data: Any) -> str:
return json.dumps(
data,
sort_keys=True,
separators=(",", ":"),
ensure_ascii=False,
default=str,
)
def _normalise_dtype(value: Any, default: str) -> str:
if value is None:
return default
if isinstance(value, str):
text = value.strip()
if not text:
return default
return text.removeprefix("torch.")
return str(value).removeprefix("torch.")
def _architecture_adapter_from_config(model_config: Any, default: str) -> str:
if not isinstance(model_config, Mapping):
return default
for key in ("architecture_adapter", "model_type"):
value = model_config.get(key)
if isinstance(value, str) and value.strip():
return value
architectures = model_config.get("architectures")
if isinstance(architectures, list) and architectures:
first = architectures[0]
if isinstance(first, str) and first.strip():
return first
text_config = model_config.get("text_config")
if isinstance(text_config, Mapping):
return _architecture_adapter_from_config(text_config, default)
return default
def _tokenizer_revision_from_config(
model_id: str,
revision: str | None,
model_config: Any,
) -> str:
if isinstance(model_config, Mapping):
for key in ("tokenizer_revision", "tokenizer_version", "_commit_hash"):
value = model_config.get(key)
if isinstance(value, str) and value.strip():
return value
if revision:
return revision
return model_id
def _cache_layout_from_recipe_params(recipe_params: Mapping[str, Any] | None) -> str:
if not recipe_params:
return "local-hot-kv"
use_cache = recipe_params.get("use_cache")
if use_cache is False:
return "stateless"
if "cache_layout" in recipe_params:
value = recipe_params.get("cache_layout")
if isinstance(value, str) and value.strip():
return value
return "local-hot-kv"
@dataclass(frozen=True)
class ArtifactIdentity:
"""Exact source artifact binding for a routable shard."""
model_id: str
revision: str | None = None
artifact_hash: str | None = None
shard_start: int | None = None
shard_end: int | None = None
def __post_init__(self) -> None:
_require_text(self.model_id, "artifact.model_id")
_optional_text(self.revision, "artifact.revision")
_optional_text(self.artifact_hash, "artifact.artifact_hash")
if self.shard_start is not None and self.shard_start < 0:
raise ValueError("'artifact.shard_start' must be >= 0")
if self.shard_end is not None and self.shard_end < 0:
raise ValueError("'artifact.shard_end' must be >= 0")
if (
self.shard_start is not None
and self.shard_end is not None
and self.shard_end < self.shard_start
):
raise ValueError("'artifact.shard_end' must be >= 'artifact.shard_start'")
def to_dict(self) -> dict[str, Any]:
return {
"model_id": self.model_id,
"revision": self.revision,
"artifact_hash": self.artifact_hash,
"shard_start": self.shard_start,
"shard_end": self.shard_end,
}
@classmethod
def from_dict(cls, data: Any) -> "ArtifactIdentity":
if not isinstance(data, Mapping):
raise ValueError(f"'artifact' must be a JSON object, got {type(data).__name__}")
return cls(
model_id=_require_text(data.get("model_id"), "artifact.model_id"),
revision=_optional_text(data.get("revision"), "artifact.revision"),
artifact_hash=_optional_text(
data.get("artifact_hash"), "artifact.artifact_hash"
),
shard_start=_optional_int(data.get("shard_start"), "artifact.shard_start"),
shard_end=_optional_int(data.get("shard_end"), "artifact.shard_end"),
)
@dataclass(frozen=True)
class RuntimeRecipeIdentity:
"""Exact runtime recipe used for admission and handshake compatibility."""
weight_quantization: str
activation_dtype: str
compute_dtype: str
kv_dtype: str
kv_layout: str
tokenizer_revision: str
architecture_adapter: str
backend_id: str
runtime_version: str
boundary_schema_version: int = 1
cache_layout: str = "local-hot-kv"
fingerprint: str | None = None
def __post_init__(self) -> None:
_require_text(self.weight_quantization, "runtime_recipe.weight_quantization")
_require_text(self.activation_dtype, "runtime_recipe.activation_dtype")
_require_text(self.compute_dtype, "runtime_recipe.compute_dtype")
_require_text(self.kv_dtype, "runtime_recipe.kv_dtype")
_require_text(self.kv_layout, "runtime_recipe.kv_layout")
_require_text(self.tokenizer_revision, "runtime_recipe.tokenizer_revision")
_require_text(self.architecture_adapter, "runtime_recipe.architecture_adapter")
_require_text(self.backend_id, "runtime_recipe.backend_id")
_require_text(self.runtime_version, "runtime_recipe.runtime_version")
_require_text(self.cache_layout, "runtime_recipe.cache_layout")
if self.boundary_schema_version < 1:
raise ValueError("'runtime_recipe.boundary_schema_version' must be >= 1")
expected = compatibility_fingerprint(self._fingerprint_payload())
if not self.fingerprint:
object.__setattr__(self, "fingerprint", expected)
elif self.fingerprint != expected:
raise ValueError(
"'runtime_recipe.fingerprint' does not match the encoded fields"
)
def to_dict(self) -> dict[str, Any]:
return {
"weight_quantization": self.weight_quantization,
"activation_dtype": self.activation_dtype,
"compute_dtype": self.compute_dtype,
"kv_dtype": self.kv_dtype,
"kv_layout": self.kv_layout,
"tokenizer_revision": self.tokenizer_revision,
"architecture_adapter": self.architecture_adapter,
"backend_id": self.backend_id,
"runtime_version": self.runtime_version,
"boundary_schema_version": self.boundary_schema_version,
"cache_layout": self.cache_layout,
"fingerprint": self.fingerprint,
}
@classmethod
def from_dict(cls, data: Any) -> "RuntimeRecipeIdentity":
if not isinstance(data, Mapping):
raise ValueError(
f"'runtime_recipe' must be a JSON object, got {type(data).__name__}"
)
boundary_schema_version = data.get("boundary_schema_version", 1)
if isinstance(boundary_schema_version, bool) or not isinstance(
boundary_schema_version, int
):
raise ValueError(
"'runtime_recipe.boundary_schema_version' must be an integer"
)
return cls(
weight_quantization=_require_text(
data.get("weight_quantization"), "runtime_recipe.weight_quantization"
),
activation_dtype=_require_text(
data.get("activation_dtype"), "runtime_recipe.activation_dtype"
),
compute_dtype=_require_text(
data.get("compute_dtype"), "runtime_recipe.compute_dtype"
),
kv_dtype=_require_text(data.get("kv_dtype"), "runtime_recipe.kv_dtype"),
kv_layout=_require_text(data.get("kv_layout"), "runtime_recipe.kv_layout"),
tokenizer_revision=_require_text(
data.get("tokenizer_revision"), "runtime_recipe.tokenizer_revision"
),
architecture_adapter=_require_text(
data.get("architecture_adapter"),
"runtime_recipe.architecture_adapter",
),
backend_id=_require_text(data.get("backend_id"), "runtime_recipe.backend_id"),
runtime_version=_require_text(
data.get("runtime_version"), "runtime_recipe.runtime_version"
),
boundary_schema_version=boundary_schema_version,
cache_layout=_require_text(data.get("cache_layout"), "runtime_recipe.cache_layout"),
fingerprint=_optional_text(data.get("fingerprint"), "runtime_recipe.fingerprint"),
)
def _fingerprint_payload(self) -> dict[str, Any]:
return {
"weight_quantization": self.weight_quantization,
"activation_dtype": self.activation_dtype,
"compute_dtype": self.compute_dtype,
"kv_dtype": self.kv_dtype,
"kv_layout": self.kv_layout,
"tokenizer_revision": self.tokenizer_revision,
"architecture_adapter": self.architecture_adapter,
"backend_id": self.backend_id,
"runtime_version": self.runtime_version,
"boundary_schema_version": self.boundary_schema_version,
"cache_layout": self.cache_layout,
}
def _optional_int(value: Any, field_name: str) -> int | None:
if value is None:
return None
if isinstance(value, bool) or not isinstance(value, int):
raise ValueError(f"{field_name!r} must be an integer")
if value < 0:
raise ValueError(f"{field_name!r} must be >= 0")
return value
def build_artifact_identity(
*,
model_id: str,
revision: str | None = None,
model_config: Any = None,
artifact_hash: str | None = None,
shard_start: int | None = None,
shard_end: int | None = None,
) -> ArtifactIdentity:
"""Build a stable artifact binding from the locally loaded artifact."""
resolved_hash = artifact_hash
if resolved_hash is None:
if isinstance(model_config, Mapping):
resolved_hash = _hash_mapping(model_config)
elif model_config is not None:
resolved_hash = _sha256_text(_stable_json(model_config))
if resolved_hash is None:
resolved_hash = _sha256_text(
_stable_json(
{
"model_id": model_id,
"revision": revision,
"shard_start": shard_start,
"shard_end": shard_end,
}
)
)
return ArtifactIdentity(
model_id=model_id,
revision=revision,
artifact_hash=resolved_hash,
shard_start=shard_start,
shard_end=shard_end,
)
def build_runtime_recipe_identity(
*,
model_id: str,
weight_quantization: str,
backend_id: str,
runtime_version: str,
revision: str | None = None,
model_config: Any = None,
recipe_params: Mapping[str, Any] | None = None,
activation_dtype: Any = None,
compute_dtype: Any = None,
kv_dtype: Any = None,
kv_layout: str | None = None,
tokenizer_revision: str | None = None,
architecture_adapter: str | None = None,
boundary_schema_version: int = 1,
cache_layout: str | None = None,
) -> RuntimeRecipeIdentity:
"""Build the exact runtime recipe used for compatibility admission."""
activation = _normalise_dtype(activation_dtype, "bfloat16")
compute = _normalise_dtype(compute_dtype, activation)
kv_dtype_text = _normalise_dtype(kv_dtype, compute)
kv_layout_text = kv_layout or "session-cache"
tokenizer = tokenizer_revision or _tokenizer_revision_from_config(
model_id, revision, model_config
)
architecture = architecture_adapter or _architecture_adapter_from_config(
model_config, backend_id
)
cache_layout_text = cache_layout or _cache_layout_from_recipe_params(recipe_params)
return RuntimeRecipeIdentity(
weight_quantization=weight_quantization,
activation_dtype=activation,
compute_dtype=compute,
kv_dtype=kv_dtype_text,
kv_layout=kv_layout_text,
tokenizer_revision=tokenizer,
architecture_adapter=architecture,
backend_id=backend_id,
runtime_version=runtime_version,
boundary_schema_version=boundary_schema_version,
cache_layout=cache_layout_text,
)
def compatibility_fingerprint(data: Mapping[str, Any]) -> str:
"""Return a stable SHA256 compatibility fingerprint for an exact route."""
return "sha256:" + _sha256_text(_stable_json(data))
def fingerprint_payload(
*,
model: Mapping[str, Any],
shard: Mapping[str, Any],
recipe: Mapping[str, Any],
backend: Mapping[str, Any],
artifact: Mapping[str, Any],
runtime_recipe: Mapping[str, Any],
) -> dict[str, Any]:
return {
"model": dict(model),
"shard": dict(shard),
"recipe": dict(recipe),
"backend": dict(backend),
"artifact": dict(artifact),
"runtime_recipe": dict(runtime_recipe),
}
def _hash_mapping(data: Mapping[str, Any]) -> str:
return "sha256:" + _sha256_text(_stable_json(data))

192
packages/node/meshnet_node/startup.py
View File

@@ -29,7 +29,6 @@ from .model_catalog import model_metadata_for
from .recipe_manifest import DEFAULT_RECIPE_ID, Recipe, RecipeManifest, load_recipe_manifest
from .relay_bridge import RelayHttpBridge, peer_id_from_wallet
from .server import StubNodeServer
from .gguf_backend import build_gguf_backend
from .torch_server import TorchNodeServer
from .wallet import load_or_create_wallet
@@ -703,35 +702,6 @@ def _resolve_recipe(recipe_id: str | None) -> tuple[RecipeManifest, Recipe]:
return manifest, manifest.require(recipe_id or DEFAULT_RECIPE_ID)
def _gguf_backend_for_recipe(
recipe: Recipe,
*,
model_id: str,
shard_start: int,
shard_end: int,
quantization: str,
total_layers: int | None,
device: str,
model_revision: str | None = None,
) -> object | None:
"""Build the GGUF backend only for recipes that explicitly ask for it."""
if recipe.backend_id != "llama.cpp":
return None
return build_gguf_backend(
model_id=model_id,
shard_start=shard_start,
shard_end=shard_end,
quantization=quantization,
total_layers=total_layers,
model_revision=model_revision,
device_type=device,
architecture_adapter="dense-llama",
tokenizer_revision=model_revision or model_id,
runtime_recipe_fingerprint=None,
supports_kv_cache=recipe.params.get("use_cache", True) is not False,
)
def _capability_device(backend: Any, detected_device: str) -> str:
"""The device the shard actually landed on, or the one this node detected."""
device = getattr(backend, "device", None)
@@ -993,8 +963,7 @@ def run_startup(
if model_id: # treat "" the same as None — no explicit model given
full_sources: list[dict] = []
detected: int | None = None
# Auto-detect shard range from model config if not explicitly provided.
# Auto-detect shard range from model config if not explicitly provided
if shard_start is None or shard_end is None:
try:
detected = _detect_num_layers(model_id, cache_dir=cache_dir)
@@ -1058,38 +1027,22 @@ def run_startup(
shard_end = shard_end if shard_end is not None else detected - 1
print(f" Auto-detected {detected} layers → shard {shard_start}–{shard_end}", flush=True)
backend = _gguf_backend_for_recipe(
recipe,
print("Loading real PyTorch model shard...", flush=True)
node = TorchNodeServer(
host=host,
port=port,
model_id=model_id,
shard_start=shard_start,
shard_end=shard_end,
quantization=quantization,
total_layers=detected if detected is not None else (shard_end + 1 if shard_end is not None else None),
device=device,
model_revision=None,
tracker_url=tracker_url,
route_timeout=route_timeout,
cache_dir=cache_dir,
debug=debug,
max_loaded_shards=max_loaded_shards,
force_cpu=force_cpu,
recipe_params=recipe.params,
)
print(
"Loading native llama.cpp model shard..." if backend is not None else "Loading real PyTorch model shard...",
flush=True,
)
node_kwargs = {
"host": host,
"port": port,
"model_id": model_id,
"shard_start": shard_start,
"shard_end": shard_end,
"quantization": quantization,
"tracker_url": tracker_url,
"route_timeout": route_timeout,
"cache_dir": cache_dir,
"debug": debug,
"max_loaded_shards": max_loaded_shards,
"force_cpu": force_cpu,
"recipe_params": recipe.params,
}
if backend is not None:
node_kwargs["backend"] = backend
node = TorchNodeServer(**node_kwargs)
capability_report = _admit_capability(
node,
model_id=model_id,
@@ -1103,15 +1056,10 @@ def run_startup(
recipe=recipe,
validator=capability_validator,
)
proof_shard = capability_report.shard
_node_start_time = time.monotonic()
actual_port = node.start()
total_layers = getattr(getattr(node, "backend", None), "total_layers", None)
shard_label = _format_shard_label(
proof_shard.start,
proof_shard.end,
total_layers,
)
shard_label = _format_shard_label(shard_start, shard_end, total_layers)
public_host = advertise_host or (socket.getfqdn() if host == "0.0.0.0" else host)
endpoint = f"http://{public_host}:{actual_port}"
if hasattr(node, "set_advertised_endpoint"):
@@ -1134,17 +1082,16 @@ def run_startup(
"model": model_id.split("/")[-1],
"hf_repo": model_id,
"num_layers": total_layers,
"shard_start": proof_shard.start,
"shard_end": proof_shard.end,
"shard_start": shard_start,
"shard_end": shard_end,
"hardware_profile": hw,
"wallet_address": address,
"quantization": quantization,
"score": 1.0,
"tracker_mode": (proof_shard.start == 0),
"tracker_mode": (shard_start == 0),
"managed_assignment": not user_pinned_shard,
"model_metadata": model_metadata_for(model_id, total_layers, cache_dir=cache_dir),
"capability_report": capability_report.to_dict(),
"compatibility_fingerprint": capability_report.compatibility_fingerprint,
# Declared independently of the proof: the tracker checks that the
# recipe this node says it serves with is the one the proof ran.
"recipe_id": recipe.id,
@@ -1152,8 +1099,8 @@ def run_startup(
"downloaded_models": (
_downloaded_model_inventory(
model_id.split("/")[-1],
proof_shard.start,
proof_shard.end,
shard_start,
shard_end,
model_cache_path,
hf_repo=model_id,
model_sources=full_sources,
@@ -1264,38 +1211,22 @@ def run_startup(
hf_repo=assigned_hf_repo,
model_sources=full_sources,
)
backend = _gguf_backend_for_recipe(
recipe,
print("Loading real PyTorch model shard...", flush=True)
node = TorchNodeServer(
host=host,
port=port,
model_id=assigned_hf_repo,
shard_start=assigned_shard_start,
shard_end=assigned_shard_end,
quantization=quantization,
total_layers=assigned_num_layers,
device=device,
model_revision=None,
tracker_url=tracker_url,
route_timeout=route_timeout,
cache_dir=cache_dir,
debug=debug,
max_loaded_shards=max_loaded_shards,
force_cpu=force_cpu,
recipe_params=recipe.params,
)
print(
"Loading native llama.cpp model shard..." if backend is not None else "Loading real PyTorch model shard...",
flush=True,
)
node_kwargs = {
"host": host,
"port": port,
"model_id": assigned_hf_repo,
"shard_start": assigned_shard_start,
"shard_end": assigned_shard_end,
"quantization": quantization,
"tracker_url": tracker_url,
"route_timeout": route_timeout,
"cache_dir": cache_dir,
"debug": debug,
"max_loaded_shards": max_loaded_shards,
"force_cpu": force_cpu,
"recipe_params": recipe.params,
}
if backend is not None:
node_kwargs["backend"] = backend
node = TorchNodeServer(**node_kwargs)
capability_report = _admit_capability(
node,
model_id=assigned_hf_repo,
@@ -1309,7 +1240,6 @@ def run_startup(
recipe=recipe,
validator=capability_validator,
)
proof_shard = capability_report.shard
_node_start_time = time.monotonic()
actual_port = node.start()
public_host = advertise_host or (socket.getfqdn() if host == "0.0.0.0" else host)
@@ -1332,17 +1262,16 @@ def run_startup(
"model": assigned_hf_repo.split("/")[-1],
"hf_repo": assigned_hf_repo,
"num_layers": assigned_num_layers,
"shard_start": proof_shard.start,
"shard_end": proof_shard.end,
"shard_start": assigned_shard_start,
"shard_end": assigned_shard_end,
"hardware_profile": hw,
"wallet_address": address,
"quantization": quantization,
"score": 1.0,
"tracker_mode": (proof_shard.start == 0),
"tracker_mode": (assigned_shard_start == 0),
"managed_assignment": True,
"model_metadata": model_metadata_for(assigned_hf_repo, assigned_num_layers, cache_dir=cache_dir),
"capability_report": capability_report.to_dict(),
"compatibility_fingerprint": capability_report.compatibility_fingerprint,
# Declared independently of the proof: the tracker checks that the
# recipe this node says it serves with is the one the proof ran.
"recipe_id": recipe.id,
@@ -1350,8 +1279,8 @@ def run_startup(
"downloaded_models": (
_downloaded_model_inventory(
assigned_hf_repo.split("/")[-1],
proof_shard.start,
proof_shard.end,
assigned_shard_start,
assigned_shard_end,
model_cache_path,
hf_repo=assigned_hf_repo,
model_sources=full_sources,
@@ -1376,8 +1305,8 @@ def run_startup(
),
)
shard_label = _format_shard_label(
proof_shard.start,
proof_shard.end,
assigned_shard_start,
assigned_shard_end,
assigned_num_layers,
)
print(
@@ -1492,38 +1421,22 @@ def run_startup(
# 5. Start HTTP server — real HF weights use TorchNodeServer; stub-model stays stub.
_node_start_time = time.monotonic()
if hf_repo and assigned_model != "stub-model":
backend = _gguf_backend_for_recipe(
recipe,
print("Loading real PyTorch model shard...", flush=True)
node = TorchNodeServer(
host=host,
port=port,
model_id=hf_repo,
shard_start=shard_start,
shard_end=shard_end,
quantization=quantization,
total_layers=total_layers,
device=device,
model_revision=None,
tracker_url=tracker_url,
route_timeout=route_timeout,
cache_dir=shard_path,
debug=debug,
max_loaded_shards=max_loaded_shards,
force_cpu=force_cpu,
recipe_params=recipe.params,
)
print(
"Loading native llama.cpp model shard..." if backend is not None else "Loading real PyTorch model shard...",
flush=True,
)
node_kwargs = {
"host": host,
"port": port,
"model_id": hf_repo,
"shard_start": shard_start,
"shard_end": shard_end,
"quantization": quantization,
"tracker_url": tracker_url,
"route_timeout": route_timeout,
"cache_dir": shard_path,
"debug": debug,
"max_loaded_shards": max_loaded_shards,
"force_cpu": force_cpu,
"recipe_params": recipe.params,
}
if backend is not None:
node_kwargs["backend"] = backend
node = TorchNodeServer(**node_kwargs)
capability_report = _admit_capability(
node,
model_id=hf_repo,
@@ -1572,7 +1485,6 @@ def run_startup(
"managed_assignment": not user_pinned_shard,
"model_metadata": model_metadata_for(hf_repo, total_layers, cache_dir=shard_path),
"capability_report": capability_report.to_dict(),
"compatibility_fingerprint": capability_report.compatibility_fingerprint,
# Declared independently of the proof: the tracker checks that the
# recipe this node says it serves with is the one the proof ran.
"recipe_id": recipe.id,
@@ -1634,7 +1546,6 @@ def run_startup(
recipe=recipe,
validator=capability_validator,
)
proof_shard = capability_report.shard
actual_port = node.start()
public_host = advertise_host or (socket.getfqdn() if host == "0.0.0.0" else host)
endpoint = f"http://{public_host}:{actual_port}"
@@ -1654,11 +1565,10 @@ def run_startup(
reg_payload = {
"endpoint": endpoint,
"model": assigned_model,
"shard_start": proof_shard.start,
"shard_end": proof_shard.end,
"shard_start": shard_start,
"shard_end": shard_end,
"shard_checksum": shard_checksum,
"capability_report": capability_report.to_dict(),
"compatibility_fingerprint": capability_report.compatibility_fingerprint,
# Declared independently of the proof: the tracker checks that the
# recipe this node says it serves with is the one the proof ran.
"recipe_id": recipe.id,
@@ -1704,8 +1614,8 @@ def run_startup(
if gpu_name:
hw_str += f" ({gpu_name}, {vram_mb / 1024:.1f} GB)"
shard_label = _format_shard_label(
proof_shard.start,
proof_shard.end,
shard_start,
shard_end,
assigned_total_layers,
model_name=assigned_model,
)

68
packages/node/meshnet_node/testing.py
View File

@@ -16,10 +16,7 @@ import time
from typing import Any
from .admission import CapabilityContext, CapabilityValidator
from . import __version__ as _PACKAGE_VERSION
from .capability import STATUS_PASSED, CapabilityReport, build_capability_report
from .gguf_ownership import authoritative_dense_llama_ownership
from .runtime_recipe import build_runtime_recipe_identity
def capability_report_for(
@@ -33,15 +30,6 @@ def capability_report_for(
recipe_version: str | None = None,
backend_id: str | None = None,
device: str | None = None,
artifact_hash: str | None = None,
activation_dtype: str | None = None,
compute_dtype: str | None = None,
kv_dtype: str | None = None,
kv_layout: str | None = None,
tokenizer_revision: str | None = None,
architecture_adapter: str | None = None,
boundary_schema_version: int = 1,
cache_layout: str | None = None,
validated_at: float | None = None,
age_seconds: float = 0.0,
diagnostics: Any = None,
@@ -49,49 +37,18 @@ def capability_report_for(
) -> CapabilityReport:
"""A report describing `context`, with any field bent away from the truth."""
now = time.time() if validated_at is None else validated_at
backend = getattr(context, "backend", None)
model_config = getattr(getattr(backend, "model", None), "config", None)
model_config_payload = (
model_config.to_dict() if hasattr(model_config, "to_dict") else model_config
)
resolved_cache_layout = (
"stateless"
if getattr(backend, "supports_kv_cache", False) is False
else "local-hot-kv"
)
ownership = authoritative_dense_llama_ownership(backend, context.selection)
runtime_recipe = build_runtime_recipe_identity(
model_id=context.selection.model_id,
revision=getattr(getattr(backend, "model", None), "revision", None),
model_config=model_config_payload,
recipe_params=context.recipe.params,
weight_quantization=context.selection.quantization,
backend_id=context.recipe.backend_id,
runtime_version=_PACKAGE_VERSION,
activation_dtype=activation_dtype,
compute_dtype=compute_dtype,
kv_dtype=kv_dtype,
kv_layout=kv_layout or _backend_kv_layout(backend),
tokenizer_revision=tokenizer_revision,
architecture_adapter=architecture_adapter,
boundary_schema_version=boundary_schema_version,
cache_layout=cache_layout or resolved_cache_layout,
)
return build_capability_report(
model_id=model_id or context.selection.model_id,
shard_start=ownership.start_layer if shard_start is None else shard_start,
shard_end=ownership.end_layer if shard_end is None else shard_end,
shard_start=(
context.selection.shard_start if shard_start is None else shard_start
),
shard_end=context.selection.shard_end if shard_end is None else shard_end,
recipe_id=recipe_id or context.recipe.id,
recipe_version=recipe_version or context.recipe.version,
catalogue_version=context.manifest.catalogue_version,
backend_id=backend_id or context.recipe.backend_id,
device=device or context.device,
quantization=context.selection.quantization,
runtime=_runtime_versions(),
artifact_hash=artifact_hash,
runtime_recipe=runtime_recipe,
owns_embedding=ownership.owns_embedding,
owns_final_head=ownership.owns_final_head,
status=status,
duration_ms=duration_ms,
diagnostics=diagnostics,
@@ -111,20 +68,3 @@ def capability_stub(**overrides: Any) -> CapabilityValidator:
return capability_report_for(context, **overrides)
return validator
def _runtime_versions() -> dict[str, str]:
versions: dict[str, str] = {}
for name in ("torch", "transformers"):
try:
module = __import__(name)
except Exception:
continue
version = getattr(module, "__version__", None)
if version:
versions[name] = str(version)
return versions
def _backend_kv_layout(backend: Any) -> str:
return "session-cache" if getattr(backend, "supports_kv_cache", False) else "stateless"

76
packages/node/native/CMakeLists.txt
View File

@@ -1,76 +0,0 @@
# Reproducible C++ build wiring for the Shard runtime protocol (DGR-002).
#
# Generates C++ message stubs from proto/shard_runtime.proto and builds the
# round-trip / cross-language compatibility test. Requires protoc and the
# protobuf C++ runtime. Works with either a CONFIG-mode protobuf install
# (protobuf::libprotobuf / protobuf::protoc targets, e.g. a from-source install
# on CMAKE_PREFIX_PATH) or CMake's bundled FindProtobuf module.
#
# The gRPC C++ service stubs are generated separately by scripts/generate_cpp.sh
# when grpc_cpp_plugin is present; the round-trip test needs only message
# serialization, so gRPC is intentionally not a build dependency here.
#
# Configure & build (out-of-tree):
# cmake -S packages/node/native -B packages/node/native/build/cpp
# cmake --build packages/node/native/build/cpp
# Run:
# packages/node/native/build/cpp/shard_protocol_roundtrip_test --selftest
cmake_minimum_required(VERSION 3.16)
project(shard_runtime_protocol CXX)
set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
# Prefer a CONFIG-mode protobuf (modern imported targets); fall back to the
# FindProtobuf module for system installs.
find_package(Protobuf CONFIG QUIET)
if(NOT Protobuf_FOUND)
find_package(Protobuf REQUIRED)
endif()
if(TARGET protobuf::protoc)
set(SHARD_PROTOC_EXECUTABLE "$<TARGET_FILE:protobuf::protoc>")
else()
set(SHARD_PROTOC_EXECUTABLE "${Protobuf_PROTOC_EXECUTABLE}")
endif()
if(TARGET protobuf::libprotobuf)
set(SHARD_PROTOBUF_LINK protobuf::libprotobuf)
else()
set(SHARD_PROTOBUF_LINK ${Protobuf_LIBRARIES})
endif()
set(PROTO_DIR "${CMAKE_CURRENT_SOURCE_DIR}/proto")
set(PROTO_FILE "${PROTO_DIR}/shard_runtime.proto")
set(GEN_DIR "${CMAKE_CURRENT_BINARY_DIR}/gen")
file(MAKE_DIRECTORY "${GEN_DIR}")
set(PROTO_SRC "${GEN_DIR}/shard_runtime.pb.cc")
set(PROTO_HDR "${GEN_DIR}/shard_runtime.pb.h")
add_custom_command(
OUTPUT "${PROTO_SRC}" "${PROTO_HDR}"
COMMAND "${SHARD_PROTOC_EXECUTABLE}"
"--proto_path=${PROTO_DIR}"
"--cpp_out=${GEN_DIR}"
"${PROTO_FILE}"
DEPENDS "${PROTO_FILE}"
COMMENT "Generating C++ protobuf stubs from shard_runtime.proto"
VERBATIM)
add_executable(shard_protocol_roundtrip_test
tests/roundtrip_test.cpp
"${PROTO_SRC}")
target_include_directories(shard_protocol_roundtrip_test PRIVATE "${GEN_DIR}")
if(NOT TARGET protobuf::libprotobuf AND Protobuf_INCLUDE_DIRS)
target_include_directories(shard_protocol_roundtrip_test PRIVATE
${Protobuf_INCLUDE_DIRS})
endif()
target_link_libraries(shard_protocol_roundtrip_test PRIVATE ${SHARD_PROTOBUF_LINK})
enable_testing()
add_test(NAME shard_protocol_roundtrip
COMMAND shard_protocol_roundtrip_test --selftest)

24
packages/node/native/llama/README.md
View File

@@ -1,24 +0,0 @@
# Pinned llama.cpp source dependency
This directory keeps the llama.cpp fork boundary explicit and auditable.
Layout:
- `UPSTREAM_COMMIT` - the exact pinned commit.
- `UPSTREAM_REPOSITORY` - the reproducible source dependency URL.
- `UPSTREAM_ASSUMPTIONS.md` - the file/ABI assumptions that the build scripts
validate.
- `patches/` - numbered patch files applied on top of the pinned checkout.
The intended flow is:
1. Fetch or clone the pinned upstream checkout.
2. Verify the checkout commit matches `UPSTREAM_COMMIT`.
3. Check and apply the numbered patch stack.
4. Build the worker scaffold from `examples/meshnet-worker/`.
5. Copy the upstream `LICENSE` and `AUTHORS` files into the worker build tree so
the attribution notices remain attached to the built artifact.
The patch stack in this story is intentionally minimal. It creates the project
worker scaffold and the smoke-test CMake target without pulling Meshnet
networking code into llama.cpp.

35
packages/node/native/llama/UPSTREAM_ASSUMPTIONS.md
View File

@@ -1,35 +0,0 @@
# llama.cpp upstream assumptions
This directory records the reproducible source dependency boundary for the
pinned llama.cpp checkout used by the distributed GGUF runtime program.
Pinned upstream commit:
- `b3c9d1b846cc80a6360adb6aeaa4fcd8c4c8dcac`
Pinned upstream repository:
- `https://github.com/ggml-org/llama.cpp.git`
Assumptions checked by the build script:
- The checkout is exactly the pinned commit above.
- The upstream source tree still ships `LICENSE`, `AUTHORS`, and
`CMakeLists.txt` at the repository root.
- The project-owned worker scaffold is built from
`examples/meshnet-worker/`, which is introduced by the patch stack below.
- The upstream license and attribution notices are preserved in the build
output by copying the root `LICENSE` and `AUTHORS` files into the worker
staging directory.
Compatibility notes:
- The current patch stack does not modify upstream llama.cpp runtime code yet.
It adds a project-owned worker scaffold that can be built reproducibly from
the pinned source checkout.
- Later stories extend this boundary with actual llama.cpp execution patches.
Failure mode:
- If the checkout commit does not match the pin, the build script fails with a
clear pin-mismatch error before patch application or compilation starts.

1
packages/node/native/llama/UPSTREAM_COMMIT
View File

@@ -1 +0,0 @@
b3c9d1b846cc80a6360adb6aeaa4fcd8c4c8dcac

1
packages/node/native/llama/UPSTREAM_REPOSITORY
View File

@@ -1 +0,0 @@
https://github.com/ggml-org/llama.cpp.git

35
packages/node/native/llama/patches/0001-add-meshnet-worker-scaffold.patch
View File

@@ -1,35 +0,0 @@
diff --git a/examples/meshnet-worker/CMakeLists.txt b/examples/meshnet-worker/CMakeLists.txt
new file mode 100644
index 0000000000..8d9f9a1a2f
--- /dev/null
+++ b/examples/meshnet-worker/CMakeLists.txt
@@ -0,0 +1,19 @@
+cmake_minimum_required(VERSION 3.16)
+project(meshnet_llama_worker CXX)
+
+set(CMAKE_CXX_STANDARD 17)
+set(CMAKE_CXX_STANDARD_REQUIRED ON)
+
+configure_file(
+ "${CMAKE_CURRENT_SOURCE_DIR}/version.h.in"
+ "${CMAKE_CURRENT_BINARY_DIR}/version.h"
+ @ONLY)
+
+add_executable(meshnet_worker
+ meshnet_worker.cpp)
+
+target_include_directories(meshnet_worker PRIVATE "${CMAKE_CURRENT_BINARY_DIR}")
+
+enable_testing()
+add_test(NAME meshnet_worker_smoke
+ COMMAND meshnet_worker --smoke)
diff --git a/examples/meshnet-worker/version.h.in b/examples/meshnet-worker/version.h.in
new file mode 100644
index 0000000000..0b75c4e60f
--- /dev/null
+++ b/examples/meshnet-worker/version.h.in
@@ -0,0 +1,4 @@
+#pragma once
+
+#define MESHNET_LLAMA_UPSTREAM_COMMIT "@MESHNET_LLAMA_UPSTREAM_COMMIT@"
+#define MESHNET_LLAMA_PATCHSET_VERSION "@MESHNET_LLAMA_PATCHSET_VERSION@"

43
packages/node/native/llama/templates/meshnet_worker.cpp
View File

@@ -1,43 +0,0 @@
#include "version.h"
#include <iostream>
#include <string>
namespace {
bool fail(const std::string& why) {
std::cerr << "meshnet_worker: FAIL: " << why << std::endl;
return false;
}
} // namespace
int main(int argc, char** argv) {
bool smoke = argc == 1;
for (int i = 1; i < argc; ++i) {
const std::string arg = argv[i];
if (arg == "--smoke") {
smoke = true;
} else {
std::cerr << "unknown arg: " << arg << std::endl;
return 2;
}
}
if (!smoke) {
return fail("smoke mode not requested"), 1;
}
if (MESHNET_LLAMA_UPSTREAM_COMMIT[0] == '\0') {
return fail("upstream commit missing"), 1;
}
if (MESHNET_LLAMA_PATCHSET_VERSION[0] == '\0') {
return fail("patchset version missing"), 1;
}
std::cout << "meshnet worker scaffold ok" << std::endl;
std::cout << "upstream commit: " << MESHNET_LLAMA_UPSTREAM_COMMIT << std::endl;
std::cout << "patchset version: " << MESHNET_LLAMA_PATCHSET_VERSION << std::endl;
return 0;
}

388
packages/node/native/proto/shard_runtime.proto
View File

@@ -1,388 +0,0 @@
// Shard runtime data-plane protocol for the distributed GGUF runtime (ADR-0024).
//
// This schema is the semantic contract between Python and C++ Shards. Direct
// transport is gRPC over HTTP/2; the existing Meshnet relay may carry the same
// serialized frames as opaque binary, so anything gRPC would normally carry in
// call metadata (deadlines, cancellation intent) is ALSO representable inside
// the messages for relay-transported seams.
//
// Design rules (see .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md):
// * One long-lived bidirectional ActivateSession stream per Route Session
// Activation Seam. No per-token channel creation.
// * Bounded chunking for prefill; a small decode fast path.
// * The activation boundary is a versioned named-tensor bundle, because an
// architecture boundary may require more than one tensor.
// * Meshnet routing/billing/auth live outside this schema; only the data
// plane and the identifiers needed to attribute and isolate work are here.
//
// Compatibility: proto3. Never renumber or reuse a field number. Add new fields
// with new numbers only. Enums keep a 0 UNSPECIFIED member for forward compat.
syntax = "proto3";
package meshnet.shard.v1;
option java_package = "com.meshnet.shard.v1";
option java_outer_classname = "ShardRuntimeProto";
option go_package = "meshnet/shard/v1;shardv1";
// ---------------------------------------------------------------------------
// Versioning and enums
// ---------------------------------------------------------------------------
// Wire schema version. Bumped only on incompatible envelope changes; additive
// field changes keep the same version and rely on proto3 unknown-field rules.
enum SchemaVersion {
SCHEMA_VERSION_UNSPECIFIED = 0;
SCHEMA_VERSION_1 = 1;
}
// Lifecycle phase of a seam message. RELEASE and CANCEL are represented both as
// dedicated RPCs and as in-stream phases so a relay-carried stream can express
// them without a separate channel.
enum Phase {
PHASE_UNSPECIFIED = 0;
PHASE_PREFILL = 1;
PHASE_DECODE = 2;
PHASE_RELEASE = 3;
PHASE_CANCEL = 4;
}
// Tensor element type. GGUF quantized block types are enumerated explicitly so
// a boundary bundle can carry pre-quantized payloads without reinterpretation.
enum DType {
DTYPE_UNSPECIFIED = 0;
DTYPE_F32 = 1;
DTYPE_F16 = 2;
DTYPE_BF16 = 3;
DTYPE_I64 = 4;
DTYPE_I32 = 5;
DTYPE_I16 = 6;
DTYPE_I8 = 7;
DTYPE_U8 = 8;
DTYPE_BOOL = 9;
DTYPE_Q8_0 = 20;
DTYPE_Q4_0 = 21;
DTYPE_Q4_K = 22;
DTYPE_Q6_K = 23;
}
// Byte order of a tensor payload. Explicit because Shards may run on
// heterogeneous hardware and the relay carries opaque bytes.
enum ByteOrder {
BYTE_ORDER_UNSPECIFIED = 0;
BYTE_ORDER_LITTLE_ENDIAN = 1;
BYTE_ORDER_BIG_ENDIAN = 2;
}
// Payload compression applied to a tensor fragment or message body.
enum Compression {
COMPRESSION_UNSPECIFIED = 0;
COMPRESSION_NONE = 1;
COMPRESSION_ZSTD = 2;
}
// Checksum algorithm. CRC32C is the cheap per-fragment default; SHA256 is used
// where stronger integrity is required.
enum ChecksumAlgorithm {
CHECKSUM_ALGORITHM_UNSPECIFIED = 0;
CHECKSUM_NONE = 1;
CHECKSUM_CRC32C = 2;
CHECKSUM_CRC32 = 3;
CHECKSUM_SHA256 = 4;
}
// What the sender expects from the receiving Shard's Hot KV State for this work
// (request side of the cache contract).
enum CacheExpectation {
CACHE_EXPECTATION_UNSPECIFIED = 0;
CACHE_REUSE = 1; // reuse existing KV for (session, epoch)
CACHE_FRESH = 2; // start a fresh KV context
CACHE_BYPASS = 3; // stateless; do not persist KV
}
// What the receiving Shard actually did with its KV State (result side).
enum CacheResult {
CACHE_RESULT_UNSPECIFIED = 0;
CACHE_HIT = 1;
CACHE_MISS = 2;
CACHE_WRITTEN = 3;
CACHE_BYPASSED = 4;
}
// Coarse retry classification carried in structured status.
enum RetryClass {
RETRY_CLASS_UNSPECIFIED = 0;
RETRY_CLASS_NONE = 1; // terminal success/no-retry
RETRY_CLASS_RETRYABLE = 2; // transient; the same step may be retried
RETRY_CLASS_FATAL = 3; // do not retry this route/epoch
RETRY_CLASS_EPOCH_STALE = 4; // route epoch advanced; re-resolve route
}
enum ServingStatus {
SERVING_STATUS_UNSPECIFIED = 0;
SERVING = 1;
NOT_SERVING = 2;
DRAINING = 3;
}
// ---------------------------------------------------------------------------
// Common value messages
// ---------------------------------------------------------------------------
// Structured, transport-independent status. Mirrors canonical gRPC codes so a
// relay-carried frame can express what a gRPC trailer normally would.
message Status {
uint32 code = 1; // canonical gRPC status code
string message = 2;
RetryClass retry_class = 3;
map<string, string> details = 4;
}
// Integrity check over an associated payload.
message Checksum {
ChecksumAlgorithm algorithm = 1;
bytes value = 2;
}
// Exact Model Artifact / runtime-recipe fingerprint. Both Shards MUST agree on
// every populated field before activation; a mismatch is a fatal status.
message ArtifactFingerprint {
string model_id = 1; // e.g. "meta-llama/Llama-3.1-8B"
string revision = 2; // artifact revision / commit
string artifact_hash = 3; // hash of the GGUF/model artifact
string quantization = 4; // e.g. "Q4_K_M", "F16"
string runtime_recipe_fingerprint = 5; // DGR-003 recipe hash
}
// Contiguous transformer layer range owned by a Shard (ADR-0012). end_layer is
// exclusive. effective_start_layer is the overlap-safe start after de-dupe of
// shared boundary layers between adjacent Shards.
message ShardRange {
uint32 start_layer = 1;
uint32 end_layer = 2;
uint32 effective_start_layer = 3;
bool owns_embedding = 4;
bool owns_final_head = 5;
}
// Token position window for a message. start_position is the absolute index of
// the first token; token_count is how many positions this message covers.
message Position {
uint64 start_position = 1;
uint64 token_count = 2;
uint64 sequence_length = 3; // total known context length, if known
}
// Envelope carried by every seam message. Everything required to version,
// route-attribute, isolate, order, and integrity-check a unit of work.
message MessageHeader {
SchemaVersion schema_version = 1;
string work_id = 2; // request/work ID (idempotency scope)
string route_session_id = 3; // Route Session ID
uint64 route_epoch = 4; // route epoch; stale epochs are rejected
ArtifactFingerprint fingerprint = 5;
ShardRange shard_range = 6;
Phase phase = 7;
Position position = 8;
uint64 idempotency_step = 9; // monotonic per (work_id) step counter
CacheExpectation cache_expectation = 10;
Compression compression = 11; // compression of THIS message's payloads
Checksum checksum = 12; // checksum over THIS message's payload
}
// ---------------------------------------------------------------------------
// Versioned named-tensor bundle (the activation boundary payload)
// ---------------------------------------------------------------------------
// One bounded fragment of a tensor payload. Large tensors are split so no
// single message is unbounded; fragments reassemble by byte_offset order.
message TensorFragment {
uint32 fragment_index = 1;
uint32 fragment_count = 2;
uint64 byte_offset = 3; // offset of this fragment within the full payload
bytes data = 4;
Checksum checksum = 5; // checksum over this fragment's (post-compression) data
}
// A single named tensor with full description so the receiver never reinterprets
// bytes implicitly.
message NamedTensor {
string name = 1;
repeated uint64 shape = 2;
DType dtype = 3;
ByteOrder byte_order = 4;
uint64 total_byte_length = 5; // full payload length across all fragments
Compression compression = 6; // compression applied to fragment data
repeated TensorFragment fragments = 7;
}
// A versioned collection of named tensors representing one activation boundary.
message TensorBundle {
uint32 bundle_version = 1;
repeated NamedTensor tensors = 2;
}
// ---------------------------------------------------------------------------
// Session stream messages (bidirectional ActivateSession)
// ---------------------------------------------------------------------------
// Opens a seam. Carries the header plus stream-scoped bounds. deadline_unix_nanos
// lets a relay-carried stream express the call deadline gRPC would otherwise own.
message SessionOpen {
MessageHeader header = 1;
uint64 deadline_unix_nanos = 2; // absolute deadline; 0 = none
uint32 max_prefill_tokens_per_chunk = 3; // bound for prefill chunking
uint32 max_fragment_bytes = 4; // bound for tensor fragment size
FlowControl initial_credit = 5; // receiver's starting flow-control window
}
// Bounded prefill chunk. A prefill is split into ordered chunks each covering at
// most max_prefill_tokens_per_chunk positions; final_chunk marks the last one.
message PrefillChunk {
MessageHeader header = 1;
uint32 chunk_index = 2;
uint32 chunk_count = 3; // 0 if unknown/streaming
bool final_chunk = 4;
TensorBundle activations = 5;
}
// Small decode fast path: a single-position (or tiny) step with minimal framing.
// Reuses the same header for isolation/ordering but expects one activation bundle.
message DecodeStep {
MessageHeader header = 1;
TensorBundle activation = 2;
}
// Explicit HTTP/2-independent flow-control grant. credits is the number of
// additional messages the receiver is willing to accept; the byte/message caps
// bound in-flight work for backpressure.
message FlowControl {
uint64 credits = 1;
uint64 max_in_flight_bytes = 2;
uint64 max_in_flight_messages = 3;
}
// Release a session's resources (Hot KV State, sequence) cleanly.
message ReleaseRequest {
MessageHeader header = 1;
string reason = 2;
}
message ReleaseResponse {
Status status = 1;
CacheResult cache_result = 2;
}
// Cancel in-flight work for a session/step.
message CancelRequest {
MessageHeader header = 1;
string reason = 2;
}
message CancelResponse {
Status status = 1;
}
// Client -> server frames on the ActivateSession stream.
message SessionActivation {
oneof payload {
SessionOpen open = 1;
PrefillChunk prefill = 2;
DecodeStep decode = 3;
ReleaseRequest release = 4;
CancelRequest cancel = 5;
FlowControl flow_control = 6;
}
}
// Computed boundary output for a step: the next Shard's input tensors plus the
// cache result and integrity for what was produced.
message ActivationResult {
MessageHeader header = 1;
TensorBundle outputs = 2;
CacheResult cache_result = 3;
Status status = 4;
}
message SessionAccepted {
MessageHeader header = 1;
FlowControl granted_credit = 2;
Status status = 3;
}
// Server -> client frames on the ActivateSession stream.
message SessionResponse {
oneof payload {
SessionAccepted accepted = 1;
ActivationResult result = 2;
FlowControl flow_control = 3;
Status status = 4;
ReleaseResponse release_ack = 5;
CancelResponse cancel_ack = 6;
}
}
// ---------------------------------------------------------------------------
// Capability and health (unary)
// ---------------------------------------------------------------------------
message ResourceBudget {
uint64 weight_bytes = 1;
uint64 kv_bytes = 2;
uint64 scratch_bytes = 3;
uint32 max_concurrent_sessions = 4;
}
message CapabilityRequest {
SchemaVersion schema_version = 1;
}
message CapabilityResponse {
SchemaVersion schema_version = 1;
repeated SchemaVersion supported_schema_versions = 2;
repeated string supported_architectures = 3; // e.g. "llama", "qwen3"
repeated string supported_quantizations = 4;
ShardRange servable_range = 5;
ResourceBudget budget = 6;
repeated Compression supported_compression = 7;
repeated ChecksumAlgorithm supported_checksums = 8;
ArtifactFingerprint loaded_fingerprint = 9; // empty if no artifact loaded
}
message HealthRequest {
string route_session_id = 1; // optional; empty for node-wide health
}
message HealthResponse {
ServingStatus status = 1;
uint32 active_sessions = 2;
uint32 queued_requests = 3;
double kv_pressure = 4; // 0.0..1.0 fraction of KV budget in use
uint64 rss_bytes = 5;
Status detail = 6;
}
// ---------------------------------------------------------------------------
// Service
// ---------------------------------------------------------------------------
service ShardRuntime {
// Admission/capability negotiation.
rpc GetCapability(CapabilityRequest) returns (CapabilityResponse);
// Liveness/backpressure telemetry.
rpc Health(HealthRequest) returns (HealthResponse);
// One long-lived bidirectional stream per Route Session Activation Seam.
// Deadlines/cancellation use gRPC call semantics on direct transport and the
// in-message equivalents on relay transport; flow control uses FlowControl
// frames; errors are structured Status.
rpc ActivateSession(stream SessionActivation) returns (stream SessionResponse);
// Clean resource release (also expressible in-stream as PHASE_RELEASE).
rpc Release(ReleaseRequest) returns (ReleaseResponse);
// Cancellation (also expressible in-stream as PHASE_CANCEL).
rpc Cancel(CancelRequest) returns (CancelResponse);
}

187
packages/node/native/scripts/build_llama_worker.sh
View File

@@ -1,187 +0,0 @@
#!/usr/bin/env bash
# Apply the numbered llama.cpp patch stack and build the worker scaffold.
#
# Default flow:
# 1. Fetch the pinned llama.cpp source into a build directory if needed.
# 2. Verify the checkout matches the pinned commit.
# 3. Check/apply the numbered patch stack from packages/node/native/llama/.
# 4. Compile and build the standalone worker scaffold.
# 5. Copy upstream LICENSE/AUTHORS notices into the staging directory.
#
# This script is intentionally model-free and does not contact any inference
# endpoint. It is a source/build reproducibility check.
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
NATIVE_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"
LLAMA_ROOT="${NATIVE_ROOT}/llama"
UPSTREAM_COMMIT="$(tr -d '\n\r' < "${LLAMA_ROOT}/UPSTREAM_COMMIT")"
UPSTREAM_REPOSITORY="$(tr -d '\n\r' < "${LLAMA_ROOT}/UPSTREAM_REPOSITORY")"
PATCH_DIR="${LLAMA_ROOT}/patches"
DEFAULT_SOURCE_DIR="${NATIVE_ROOT}/build/llama.cpp-src"
DEFAULT_BUILD_DIR="${NATIVE_ROOT}/build/llama-worker"
SOURCE_DIR="${DEFAULT_SOURCE_DIR}"
BUILD_DIR="${DEFAULT_BUILD_DIR}"
WORKTREE_DIR=""
FETCH=1
CXX_BIN="${CXX:-}"
usage() {
cat <<'EOF'
Usage: build_llama_worker.sh [--source-dir PATH] [--build-dir PATH] [--no-fetch]
Builds the project-owned worker scaffold from a pinned llama.cpp checkout.
EOF
}
fail() {
echo "error: $*" >&2
exit 1
}
while (($#)); do
case "$1" in
--source-dir)
SOURCE_DIR="${2:-}"
shift 2
;;
--build-dir)
BUILD_DIR="${2:-}"
shift 2
;;
--no-fetch)
FETCH=0
shift
;;
-h|--help)
usage
exit 0
;;
*)
fail "unknown argument: $1"
;;
esac
done
[[ -n "${SOURCE_DIR}" ]] || fail "source dir is empty"
[[ -n "${BUILD_DIR}" ]] || fail "build dir is empty"
checkout_commit() {
if [[ -f "${SOURCE_DIR}/.meshnet-upstream-commit" ]]; then
tr -d '\n\r' < "${SOURCE_DIR}/.meshnet-upstream-commit"
return 0
fi
if git -C "${SOURCE_DIR}" rev-parse --is-inside-work-tree >/dev/null 2>&1; then
git -C "${SOURCE_DIR}" rev-parse HEAD
return 0
fi
return 1
}
ensure_source() {
if [[ -d "${SOURCE_DIR}" ]]; then
return 0
fi
if [[ "${FETCH}" -ne 1 ]]; then
fail "source dir ${SOURCE_DIR} does not exist and --no-fetch was set"
fi
mkdir -p "${SOURCE_DIR}"
git clone --quiet "${UPSTREAM_REPOSITORY}" "${SOURCE_DIR}" || fail "unable to clone ${UPSTREAM_REPOSITORY}"
git -C "${SOURCE_DIR}" checkout --quiet "${UPSTREAM_COMMIT}" || fail "unable to checkout ${UPSTREAM_COMMIT}"
printf '%s\n' "${UPSTREAM_COMMIT}" > "${SOURCE_DIR}/.meshnet-upstream-commit"
printf '%s\n' "${UPSTREAM_REPOSITORY}" > "${SOURCE_DIR}/.meshnet-upstream-repository"
}
verify_assumptions() {
local observed_commit
observed_commit="$(checkout_commit)" || fail "source tree does not expose a commit pin; write ${SOURCE_DIR}/.meshnet-upstream-commit or use a git checkout"
if [[ "${observed_commit}" != "${UPSTREAM_COMMIT}" ]]; then
fail "llama.cpp pin mismatch: expected ${UPSTREAM_COMMIT}, got ${observed_commit}"
fi
for required in LICENSE AUTHORS CMakeLists.txt; do
[[ -e "${SOURCE_DIR}/${required}" ]] || fail "missing upstream assumption file: ${required}"
done
}
apply_patches() {
shopt -s nullglob
local patches=("${PATCH_DIR}"/*.patch)
shopt -u nullglob
if ((${#patches[@]} == 0)); then
fail "no patch files found in ${PATCH_DIR}"
fi
for patch in "${patches[@]}"; do
git -C "${SOURCE_DIR}" apply --check "${patch}" || fail "patch check failed: $(basename "${patch}")"
done
for patch in "${patches[@]}"; do
git -C "${SOURCE_DIR}" apply "${patch}" || fail "patch apply failed: $(basename "${patch}")"
done
}
build_worker() {
rm -rf "${BUILD_DIR}"
mkdir -p "${BUILD_DIR}"
WORKTREE_DIR="${BUILD_DIR}/llama.cpp-worktree"
rm -rf "${WORKTREE_DIR}"
mkdir -p "${WORKTREE_DIR}"
cp -a "${SOURCE_DIR}/." "${WORKTREE_DIR}/"
if [[ -f "${SOURCE_DIR}/.meshnet-upstream-commit" ]]; then
cp "${SOURCE_DIR}/.meshnet-upstream-commit" "${WORKTREE_DIR}/.meshnet-upstream-commit"
fi
if [[ -f "${SOURCE_DIR}/.meshnet-upstream-repository" ]]; then
cp "${SOURCE_DIR}/.meshnet-upstream-repository" "${WORKTREE_DIR}/.meshnet-upstream-repository"
fi
SOURCE_DIR="${WORKTREE_DIR}"
apply_patches
local worker_dir="${SOURCE_DIR}/examples/meshnet-worker"
cp "${LLAMA_ROOT}/templates/meshnet_worker.cpp" "${worker_dir}/meshnet_worker.cpp"
cat > "${worker_dir}/version.h" <<EOF
#pragma once
#define MESHNET_LLAMA_UPSTREAM_COMMIT "${UPSTREAM_COMMIT}"
#define MESHNET_LLAMA_PATCHSET_VERSION "0001"
EOF
local compiler=""
if [[ -n "${CXX_BIN}" ]] && command -v "${CXX_BIN}" >/dev/null 2>&1; then
compiler="${CXX_BIN}"
elif command -v g++ >/dev/null 2>&1; then
compiler="g++"
elif command -v c++ >/dev/null 2>&1; then
compiler="c++"
elif command -v clang++ >/dev/null 2>&1; then
compiler="clang++"
else
fail "no C++ compiler found (need g++, c++, clang++, or $CXX)"
fi
"${compiler}" -std=c++17 -O2 -Wall -Wextra \
-I "${worker_dir}" \
-o "${BUILD_DIR}/meshnet_worker" \
"${worker_dir}/meshnet_worker.cpp"
}
stage_notices() {
local notice_dir="${BUILD_DIR}/upstream-notices"
mkdir -p "${notice_dir}"
cp "${SOURCE_DIR}/LICENSE" "${notice_dir}/LICENSE"
cp "${SOURCE_DIR}/AUTHORS" "${notice_dir}/AUTHORS"
printf '%s\n' "${UPSTREAM_COMMIT}" > "${notice_dir}/UPSTREAM_COMMIT"
printf '%s\n' "${UPSTREAM_REPOSITORY}" > "${notice_dir}/UPSTREAM_REPOSITORY"
}
main() {
ensure_source
verify_assumptions
build_worker
stage_notices
"${BUILD_DIR}/meshnet_worker" --smoke
echo "build ok: ${BUILD_DIR}/meshnet_worker"
}
main "$@"

43
packages/node/native/scripts/generate_cpp.sh
View File

@@ -1,43 +0,0 @@
#!/usr/bin/env bash
# Reproducibly generate the C++ Shard-protocol stubs from the schema.
#
# Produces message stubs (protoc --cpp_out) always, and gRPC C++ service stubs
# (protoc --grpc_out with grpc_cpp_plugin) when the plugin is available. The
# round-trip test needs only the message stubs; gRPC service stubs are for the
# standalone C++ worker (DGR-008).
#
# Requirements: protoc (>=3.16). Optional: grpc_cpp_plugin for --grpc_out.
#
# Usage:
# packages/node/native/scripts/generate_cpp.sh
# Output: packages/node/native/build/cpp-gen/ (gitignored via build/).
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
NATIVE_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"
PROTO_DIR="${NATIVE_ROOT}/proto"
PROTO_FILE="${PROTO_DIR}/shard_runtime.proto"
OUT_DIR="${NATIVE_ROOT}/build/cpp-gen"
if ! command -v protoc >/dev/null 2>&1; then
echo "error: protoc not found on PATH (install protobuf-compiler)." >&2
exit 3
fi
mkdir -p "${OUT_DIR}"
echo "generating C++ message stubs -> ${OUT_DIR}"
protoc --proto_path="${PROTO_DIR}" --cpp_out="${OUT_DIR}" "${PROTO_FILE}"
if command -v grpc_cpp_plugin >/dev/null 2>&1; then
echo "generating C++ gRPC service stubs -> ${OUT_DIR}"
protoc --proto_path="${PROTO_DIR}" \
--grpc_out="${OUT_DIR}" \
--plugin=protoc-gen-grpc="$(command -v grpc_cpp_plugin)" \
"${PROTO_FILE}"
else
echo "note: grpc_cpp_plugin not found; skipped --grpc_out (message stubs only)." >&2
fi
echo "done:"
ls -1 "${OUT_DIR}"

76
packages/node/native/scripts/generate_python.py
View File

@@ -1,76 +0,0 @@
#!/usr/bin/env python3
"""Reproducibly generate the Python Shard-protocol stubs from the schema.
This is the documented, no-manual-copy generation entry point referenced by
``evidence/DGR-002/README.md``. It runs the pinned ``grpc_tools.protoc`` with the
same flags ``meshnet_node.native_protocol.generate()`` uses on demand, but is
kept self-contained (it does not import ``meshnet_node``) so it works regardless
of which checkout the editable install points at.
Usage (from the project .venv):
python packages/node/native/scripts/generate_python.py
Output: ``packages/node/native/build/python/shard_runtime_pb2{,_grpc}.py``
(``build/`` is gitignored).
"""
from __future__ import annotations
import pathlib
import sys
_NATIVE_ROOT = pathlib.Path(__file__).resolve().parents[1]
PROTO_DIR = _NATIVE_ROOT / "proto"
PROTO_FILE = PROTO_DIR / "shard_runtime.proto"
GEN_DIR = _NATIVE_ROOT / "build" / "python"
def _well_known_include() -> str | None:
try:
import grpc_tools
candidate = pathlib.Path(grpc_tools.__file__).parent / "_proto"
return str(candidate) if candidate.is_dir() else None
except Exception:
return None
def main() -> int:
if not PROTO_FILE.exists():
print(f"schema not found: {PROTO_FILE}", file=sys.stderr)
return 2
try:
from grpc_tools import protoc
except ImportError:
print(
"grpc_tools is required (pip install grpcio-tools); it is present in "
"the project .venv.",
file=sys.stderr,
)
return 3
GEN_DIR.mkdir(parents=True, exist_ok=True)
well_known = _well_known_include()
args = [
"grpc_tools.protoc",
f"-I{PROTO_DIR}",
*([f"-I{well_known}"] if well_known else []),
f"--python_out={GEN_DIR}",
f"--grpc_python_out={GEN_DIR}",
PROTO_FILE.name,
]
rc = protoc.main(args)
if rc != 0:
print(f"grpc_tools.protoc exited with status {rc}", file=sys.stderr)
return rc
print(f"generated Python stubs into: {GEN_DIR}")
for name in ("shard_runtime_pb2.py", "shard_runtime_pb2_grpc.py"):
target = GEN_DIR / name
print(f" {name}: {'ok' if target.exists() else 'MISSING'}")
return 0
if __name__ == "__main__":
raise SystemExit(main())

180
packages/node/native/tests/roundtrip_test.cpp
View File

@@ -1,180 +0,0 @@
// C++ round-trip and cross-language compatibility test for the Shard protocol.
//
// Modes (composable):
// --selftest serialize a sample message, parse it back, verify fields.
// --read <path> parse a fixture serialized by another language; verify the
// known fields; tolerate unknown fields (forward compat).
// --write <path> serialize the C++ sample so another language can parse it.
//
// Exit code 0 means every requested check passed. The Python test drives this
// binary to prove Python<->C++ wire compatibility in both directions.
#include "shard_runtime.pb.h"
#include <cstdint>
#include <fstream>
#include <iostream>
#include <sstream>
#include <string>
using namespace meshnet::shard::v1;
namespace {
bool Fail(const std::string& why) {
std::cerr << "roundtrip_test: FAIL: " << why << std::endl;
return false;
}
SessionActivation MakeSample() {
SessionActivation act;
PrefillChunk* pre = act.mutable_prefill();
MessageHeader* h = pre->mutable_header();
h->set_schema_version(SCHEMA_VERSION_1);
h->set_work_id("w1");
h->set_route_session_id("s1");
h->set_route_epoch(3);
h->set_phase(PHASE_PREFILL);
h->set_idempotency_step(7);
h->set_cache_expectation(CACHE_FRESH);
h->set_compression(COMPRESSION_NONE);
ArtifactFingerprint* fp = h->mutable_fingerprint();
fp->set_model_id("meta-llama/Llama-3.1-8B");
fp->set_quantization("Q4_K_M");
fp->set_runtime_recipe_fingerprint("recipe-abc");
ShardRange* sr = h->mutable_shard_range();
sr->set_start_layer(0);
sr->set_end_layer(16);
sr->set_effective_start_layer(0);
sr->set_owns_embedding(true);
Position* pos = h->mutable_position();
pos->set_start_position(0);
pos->set_token_count(5);
pos->set_sequence_length(5);
pre->set_chunk_index(0);
pre->set_chunk_count(1);
pre->set_final_chunk(true);
TensorBundle* bundle = pre->mutable_activations();
bundle->set_bundle_version(1);
NamedTensor* t = bundle->add_tensors();
t->set_name("hidden");
t->add_shape(1);
t->add_shape(4096);
t->set_dtype(DTYPE_F16);
t->set_byte_order(BYTE_ORDER_LITTLE_ENDIAN);
t->set_total_byte_length(8);
t->set_compression(COMPRESSION_NONE);
TensorFragment* frag = t->add_fragments();
frag->set_fragment_index(0);
frag->set_fragment_count(1);
frag->set_byte_offset(0);
frag->set_data(std::string("\x01\x02\x03\x04\x05\x06\x07\x08", 8));
return act;
}
bool CheckSample(const SessionActivation& act) {
if (act.payload_case() != SessionActivation::kPrefill)
return Fail("payload is not prefill");
const PrefillChunk& pre = act.prefill();
const MessageHeader& h = pre.header();
if (h.schema_version() != SCHEMA_VERSION_1) return Fail("schema_version");
if (h.work_id() != "w1") return Fail("work_id");
if (h.route_session_id() != "s1") return Fail("route_session_id");
if (h.route_epoch() != 3) return Fail("route_epoch");
if (h.phase() != PHASE_PREFILL) return Fail("phase");
if (h.idempotency_step() != 7) return Fail("idempotency_step");
if (h.fingerprint().model_id() != "meta-llama/Llama-3.1-8B")
return Fail("model_id");
if (h.fingerprint().quantization() != "Q4_K_M") return Fail("quantization");
if (h.shard_range().end_layer() != 16) return Fail("end_layer");
if (!h.shard_range().owns_embedding()) return Fail("owns_embedding");
if (h.position().token_count() != 5) return Fail("token_count");
if (!pre.final_chunk()) return Fail("final_chunk");
if (pre.activations().tensors_size() != 1) return Fail("tensors_size");
const NamedTensor& t = pre.activations().tensors(0);
if (t.name() != "hidden") return Fail("tensor name");
if (t.dtype() != DTYPE_F16) return Fail("dtype");
if (t.byte_order() != BYTE_ORDER_LITTLE_ENDIAN) return Fail("byte_order");
if (t.shape_size() != 2 || t.shape(1) != 4096) return Fail("shape");
if (t.fragments_size() != 1) return Fail("fragments_size");
if (t.fragments(0).data().size() != 8) return Fail("fragment data length");
return true;
}
bool ReadFile(const std::string& path, std::string* out) {
std::ifstream in(path, std::ios::binary);
if (!in) return false;
std::ostringstream ss;
ss << in.rdbuf();
*out = ss.str();
return true;
}
bool WriteFile(const std::string& path, const std::string& data) {
std::ofstream out(path, std::ios::binary);
if (!out) return false;
out.write(data.data(), static_cast<std::streamsize>(data.size()));
return static_cast<bool>(out);
}
} // namespace
int main(int argc, char** argv) {
GOOGLE_PROTOBUF_VERIFY_VERSION;
std::string read_path;
std::string write_path;
bool selftest = (argc == 1);
for (int i = 1; i < argc; ++i) {
std::string arg = argv[i];
if (arg == "--selftest") {
selftest = true;
} else if (arg == "--read" && i + 1 < argc) {
read_path = argv[++i];
} else if (arg == "--write" && i + 1 < argc) {
write_path = argv[++i];
} else {
std::cerr << "unknown/incomplete arg: " << arg << std::endl;
return 2;
}
}
if (selftest) {
SessionActivation sample = MakeSample();
std::string bytes;
if (!sample.SerializeToString(&bytes)) return Fail("serialize"), 1;
SessionActivation parsed;
if (!parsed.ParseFromString(bytes)) return Fail("parse"), 1;
if (!CheckSample(parsed)) return 1;
std::cout << "selftest ok (" << bytes.size() << " bytes)" << std::endl;
}
if (!read_path.empty()) {
std::string bytes;
if (!ReadFile(read_path, &bytes)) return Fail("cannot read fixture"), 1;
SessionActivation parsed;
// ParseFromString tolerates and preserves unknown fields (forward compat).
if (!parsed.ParseFromString(bytes)) return Fail("parse fixture"), 1;
if (!CheckSample(parsed)) return 1;
std::cout << "read ok (" << bytes.size() << " bytes)" << std::endl;
}
if (!write_path.empty()) {
SessionActivation sample = MakeSample();
std::string bytes;
if (!sample.SerializeToString(&bytes)) return Fail("serialize for write"), 1;
if (!WriteFile(write_path, bytes)) return Fail("cannot write output"), 1;
std::cout << "write ok (" << bytes.size() << " bytes)" << std::endl;
}
google::protobuf::ShutdownProtobufLibrary();
return 0;
}

49
packages/tracker/meshnet_tracker/capability.py
View File

@@ -58,7 +58,6 @@ STATE_MODEL_MISMATCH = "model-mismatch"
STATE_SHARD_MISMATCH = "shard-mismatch"
STATE_RECIPE_MISMATCH = "recipe-mismatch"
STATE_CATALOGUE_INCOMPATIBLE = "catalogue-incompatible"
STATE_COMPATIBILITY_MISMATCH = "compatibility-mismatch"
ALL_STATES = (
STATE_ADMITTED,
@@ -70,7 +69,6 @@ ALL_STATES = (
STATE_SHARD_MISMATCH,
STATE_RECIPE_MISMATCH,
STATE_CATALOGUE_INCOMPATIBLE,
STATE_COMPATIBILITY_MISMATCH,
)
# --- Compatibility policy for nodes that predate the capability protocol. ---
@@ -157,17 +155,12 @@ class CapabilityState:
model_id: str | None = None
shard_start: int | None = None
shard_end: int | None = None
owns_embedding: bool | None = None
owns_final_head: bool | None = None
recipe_id: str | None = None
recipe_version: str | None = None
catalogue_version: str | None = None
backend_id: str | None = None
device: str | None = None
quantization: str | None = None
artifact_hash: str | None = None
compatibility_fingerprint: str | None = None
runtime_recipe_fingerprint: str | None = None
validated_at: float | None = None
recorded_at: float = 0.0
schema_version: int | None = None
@@ -194,17 +187,12 @@ class CapabilityState:
"model_id": self.model_id,
"shard_start": self.shard_start,
"shard_end": self.shard_end,
"owns_embedding": self.owns_embedding,
"owns_final_head": self.owns_final_head,
"recipe_id": self.recipe_id,
"recipe_version": self.recipe_version,
"catalogue_version": self.catalogue_version,
"backend_id": self.backend_id,
"device": self.device,
"quantization": self.quantization,
"artifact_hash": self.artifact_hash,
"compatibility_fingerprint": self.compatibility_fingerprint,
"runtime_recipe_fingerprint": self.runtime_recipe_fingerprint,
"validated_at": self.validated_at,
"recorded_at": self.recorded_at,
"schema_version": self.schema_version,
@@ -234,7 +222,6 @@ def evaluate_report(
shard_end: int | None,
declared_recipe_id: str | None = None,
declared_recipe_version: str | None = None,
declared_compatibility_fingerprint: str | None = None,
now: float | None = None,
max_age_seconds: float = DEFAULT_MAX_REPORT_AGE_SECONDS,
) -> CapabilityState:
@@ -321,17 +308,6 @@ def evaluate_report(
f"the node declared v{declared_recipe_version}",
)
if (
declared_compatibility_fingerprint is not None
and base.compatibility_fingerprint != declared_compatibility_fingerprint
):
return base.with_state(
STATE_COMPATIBILITY_MISMATCH,
"proof compatibility fingerprint does not match the node's declared "
"artifact/runtime recipe; the artifact, tokenizer, architecture, "
"boundary schema, activation recipe or cache layout differs",
)
if status != STATUS_PASSED:
return base.with_state(
STATE_FAILED,
@@ -368,8 +344,6 @@ def _parse_report(doc: Mapping[str, Any]) -> dict:
shard = _object(doc.get("shard"), "shard")
recipe = _object(doc.get("recipe"), "recipe")
backend = _object(doc.get("backend"), "backend")
artifact = _object_or_none(doc.get("artifact"), "artifact")
runtime_recipe = _object_or_none(doc.get("runtime_recipe"), "runtime_recipe")
validated_at = doc.get("validated_at")
if isinstance(validated_at, bool) or not isinstance(validated_at, (int, float)):
@@ -383,8 +357,6 @@ def _parse_report(doc: Mapping[str, Any]) -> dict:
"model_id": _text(model.get("model_id"), "model.model_id"),
"shard_start": _index(shard.get("start"), "shard.start"),
"shard_end": _index(shard.get("end"), "shard.end"),
"owns_embedding": _maybe_bool(shard.get("owns_embedding")),
"owns_final_head": _maybe_bool(shard.get("owns_final_head")),
"recipe_id": _text(recipe.get("recipe_id"), "recipe.recipe_id"),
"recipe_version": _text(recipe.get("recipe_version"), "recipe.recipe_version"),
"catalogue_version": _text(
@@ -395,15 +367,6 @@ def _parse_report(doc: Mapping[str, Any]) -> dict:
"quantization": _optional_text(
backend.get("quantization"), "backend.quantization"
),
"artifact_hash": _optional_text(
artifact.get("artifact_hash"), "artifact.artifact_hash"
),
"compatibility_fingerprint": _optional_text(
doc.get("compatibility_fingerprint"), "compatibility_fingerprint"
),
"runtime_recipe_fingerprint": _optional_text(
runtime_recipe.get("fingerprint"), "runtime_recipe.fingerprint"
),
"validated_at": float(validated_at),
"schema_version": schema_version,
"diagnostics": _diagnostics(doc.get("diagnostics")),
@@ -417,12 +380,6 @@ def _object(value: Any, field_name: str) -> Mapping[str, Any]:
return value
def _object_or_none(value: Any, field_name: str) -> Mapping[str, Any]:
if value is None:
return {}
return _object(value, field_name)
def _text(value: Any, field_name: str) -> str:
if not isinstance(value, str) or not value.strip():
raise _ReportError(f"{field_name!r} must be a non-empty string")
@@ -447,12 +404,6 @@ def _maybe_int(value: Any) -> int | None:
return value
def _maybe_bool(value: Any) -> bool | None:
if isinstance(value, bool):
return value
return None
def _diagnostics(value: Any) -> tuple[str, ...]:
if not isinstance(value, list):
return ()

34
packages/tracker/meshnet_tracker/server.py
View File

@@ -56,7 +56,6 @@ from .capability import (
DEFAULT_POLICY as DEFAULT_CAPABILITY_POLICY,
POLICY_COMPAT,
POLICY_ENFORCE,
STATE_COMPATIBILITY_MISMATCH,
STATE_ABSENT,
STATE_ADMITTED,
STATE_MODEL_MISMATCH,
@@ -599,7 +598,6 @@ class _NodeEntry:
"model_tokens_per_sec",
"pending_directives", "last_heartbeat", "tracker_mode",
"relay_addr", "cert_fingerprint", "peer_id", "friendly_name",
"compatibility_fingerprint",
# heartbeat stats (reported by node, cumulative)
"total_requests", "failed_requests", "queue_depth", "proxy_inflight", "uptime_seconds",
"current_requests",
@@ -638,7 +636,6 @@ class _NodeEntry:
cert_fingerprint: str | None = None,
peer_id: str | None = None,
friendly_name: str | None = None,
compatibility_fingerprint: str | None = None,
capability: "CapabilityState | None" = None,
) -> None:
self.node_id = node_id
@@ -667,7 +664,6 @@ class _NodeEntry:
self.cert_fingerprint = cert_fingerprint
self.peer_id = peer_id
self.friendly_name = friendly_name
self.compatibility_fingerprint = compatibility_fingerprint
# No proof presented is `absent`, never `admitted` — a node can only earn
# `admitted` by presenting a report that covers what it advertises.
self.capability: CapabilityState = capability or absent_state()
@@ -786,16 +782,6 @@ def _node_admission(node: "_NodeEntry") -> CapabilityState:
f"proof is for layers {state.shard_start}–{state.shard_end}, but the "
f"node now serves layers {node.shard_start}–{node.shard_end}",
)
if (
node.compatibility_fingerprint
and state.compatibility_fingerprint
and state.compatibility_fingerprint != node.compatibility_fingerprint
):
return state.with_state(
STATE_COMPATIBILITY_MISMATCH,
"proof compatibility fingerprint no longer matches the node's "
"declared artifact/runtime recipe",
)
return state
@@ -825,12 +811,6 @@ def _capability_from_registration(
declared_recipe_version=(
recipe_version if isinstance(recipe_version, str) else None
),
declared_compatibility_fingerprint=(
value.strip()
if isinstance((value := payload.get("compatibility_fingerprint")), str)
and value.strip()
else None
),
)
@@ -4643,13 +4623,6 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
relay_addr = body.get("relay_addr") or None
cert_fingerprint = body.get("cert_fingerprint") or None
peer_id = body.get("peer_id") or None
compatibility_fingerprint = body.get("compatibility_fingerprint")
if compatibility_fingerprint is not None and (
not isinstance(compatibility_fingerprint, str) or not compatibility_fingerprint.strip()
):
self._send_json(400, {"error": "compatibility_fingerprint must be a string"})
return
compatibility_fingerprint = compatibility_fingerprint.strip() if isinstance(compatibility_fingerprint, str) else None
try:
friendly_name = _normalize_friendly_name(body.get("friendly_name"))
except ValueError as exc:
@@ -4709,7 +4682,6 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
cert_fingerprint=cert_fingerprint,
peer_id=peer_id,
friendly_name=friendly_name,
compatibility_fingerprint=compatibility_fingerprint,
capability=capability,
)
with server.lock:
@@ -7162,12 +7134,6 @@ class TrackerServer:
else None
),
friendly_name=_normalize_friendly_name(payload.get("friendly_name")),
compatibility_fingerprint=(
value.strip()
if isinstance((value := payload.get("compatibility_fingerprint")), str)
and value.strip()
else None
),
# A replicated registration carries its proof: without this, a proven
# node would be routable on the leader and dark on every follower.
capability=_capability_from_registration(

472
tests/test_batch_scheduler.py
View File

@@ -1,472 +0,0 @@
"""Continuous batching and bounded admission (DGR-012).
These tests drive the node-local continuous-batching scheduler with the *same*
pure-numpy KV-cached dense-Llama reference the Hot KV State manager uses
(DGR-007), imported from ``test_hot_kv_state``. That keeps the whole gate
deterministic, download-free, GPU-free, and API-credit-free while exercising the
real KV isolation path (``KvBoundaryAdapter`` + ``HotKvStateManager``) rather than
a mock.
Coverage maps to the story's acceptance criteria:
* bounded admission against weight/KV/scratch/queue budgets,
* compatible decode steps batched with per-session positions/outputs preserved,
* prefill never starving in-flight decode (explicit decode-first policy),
* backpressure when the bounded queue is full,
* capability telemetry reporting every required signal,
* a deterministic 1/2/4/8 concurrency sweep showing saturation and no
cross-session corruption.
"""
from __future__ import annotations
import numpy as np
import pytest
from meshnet_node.hot_kv_state import (
HotKvStateConfig,
HotKvStateManager,
KvBoundaryAdapter,
kv_recipe_for,
)
from meshnet_node.batch_scheduler import (
AdmissionReason,
ContinuousBatchScheduler,
GenerationRequest,
KvBatchEngine,
NodeBudget,
Phase,
run_concurrency_sweep,
)
# Reuse the certified numpy dense-Llama reference and shard from the DGR-007 gate.
from test_hot_kv_state import _KvDenseLlama, _KvReferenceShard
# --------------------------------------------------------------------------- #
# Helpers.
# --------------------------------------------------------------------------- #
class _FakeClock:
def __init__(self) -> None:
self.now = 0.0
def __call__(self) -> float:
return self.now
def advance(self, delta: float) -> None:
self.now += delta
def _make_engine(
model: _KvDenseLlama | None = None,
*,
config: HotKvStateConfig | None = None,
) -> KvBatchEngine:
"""A full-shard KV batch engine over the deterministic numpy dense-Llama."""
model = model or _KvDenseLlama()
shard = _KvReferenceShard(model, 0, model.n_layers - 1)
manager = HotKvStateManager(kv_recipe_for(shard), config=config)
adapter = KvBoundaryAdapter(shard, manager)
return KvBatchEngine(adapter)
def _reference_tokens(model: _KvDenseLlama, prompt, n_new: int) -> list[int]:
return model.stateless_greedy(list(prompt), n_new)
def _generation(session_id: str, prompt, n_new: int, epoch: int = 0) -> GenerationRequest:
return GenerationRequest(
session_id=session_id,
route_epoch=epoch,
prompt_token_ids=tuple(prompt),
max_new_tokens=n_new,
)
# --------------------------------------------------------------------------- #
# Bounded admission (weight / KV / scratch / queue budgets).
# --------------------------------------------------------------------------- #
def test_admission_respects_active_scratch_and_queue_budgets():
"Admission fills active slots, queues the overflow, then rejects a full queue.\n\nTags: node, scheduler, admission"
engine = _make_engine()
budget = NodeBudget(
max_active_sessions=2,
scratch_bytes_per_session=1,
scratch_budget_bytes=2, # scratch also caps at 2 concurrent
max_queue_depth=1,
max_batch_size=2,
)
scheduler = ContinuousBatchScheduler(engine, budget)
a = scheduler.submit(_generation("a", [1, 2, 3], 4))
b = scheduler.submit(_generation("b", [4, 5, 6], 4))
assert a.reason is AdmissionReason.ADMITTED
assert b.reason is AdmissionReason.ADMITTED
# Two active slots full -> the next goes to the bounded queue.
c = scheduler.submit(_generation("c", [7, 8, 9], 4))
assert c.reason is AdmissionReason.QUEUED
# Queue depth 1 is now full -> backpressure rejection.
d = scheduler.submit(_generation("d", [1, 1, 1], 4))
assert d.reason is AdmissionReason.REJECTED_QUEUE_FULL
assert d.rejected
telem = scheduler.telemetry()
assert telem.active_sessions == 2
assert telem.queue_depth == 1
assert telem.rejected_admissions_total == 1
assert telem.rejected_by_reason[AdmissionReason.REJECTED_QUEUE_FULL.value] == 1
def test_admission_rejects_a_session_that_cannot_fit_the_kv_budget():
"A generation whose whole KV cannot fit the node budget is rejected up front.\n\nTags: node, scheduler, admission"
engine = _make_engine()
per_token = engine._manager.recipe.bytes_per_token()
# Budget holds only 3 positions; a prompt(4)+7 new = 10 final positions cannot fit.
budget = NodeBudget(kv_budget_bytes=per_token * 3)
scheduler = ContinuousBatchScheduler(engine, budget)
decision = scheduler.submit(_generation("big", [1, 2, 3, 4], 7))
assert decision.reason is AdmissionReason.REJECTED_KV_BUDGET
assert scheduler.telemetry().rejected_admissions_total == 1
def test_admission_rejects_when_per_session_scratch_exceeds_budget():
"A per-session scratch larger than the whole scratch envelope is rejected.\n\nTags: node, scheduler, admission"
engine = _make_engine()
budget = NodeBudget(scratch_bytes_per_session=1024, scratch_budget_bytes=512)
scheduler = ContinuousBatchScheduler(engine, budget)
decision = scheduler.submit(_generation("s", [1, 2], 2))
assert decision.reason is AdmissionReason.REJECTED_SCRATCH_BUDGET
def test_duplicate_submission_is_rejected():
"Submitting a session id that is already scheduled is rejected as a duplicate.\n\nTags: node, scheduler, admission"
engine = _make_engine()
scheduler = ContinuousBatchScheduler(engine, NodeBudget(max_active_sessions=4))
assert scheduler.submit(_generation("dup", [1, 2], 3)).reason is AdmissionReason.ADMITTED
assert scheduler.submit(_generation("dup", [3, 4], 3)).reason is AdmissionReason.REJECTED_DUPLICATE
def test_weight_budget_is_reported_in_telemetry():
"The resident weight footprint is surfaced as a capability signal.\n\nTags: node, scheduler, telemetry"
engine = _make_engine()
budget = NodeBudget(weight_bytes=123_456)
scheduler = ContinuousBatchScheduler(engine, budget)
assert scheduler.telemetry().weight_bytes == 123_456
# --------------------------------------------------------------------------- #
# Continuous batching preserves per-session positions and outputs.
# --------------------------------------------------------------------------- #
def test_batched_decode_preserves_per_session_positions_and_outputs():
"Four sessions batched together each reproduce their own stateless tokens.\n\nTags: node, scheduler, batching"
model = _KvDenseLlama()
engine = _make_engine(model)
budget = NodeBudget(max_active_sessions=4, max_batch_size=4, max_queue_depth=4)
scheduler = ContinuousBatchScheduler(engine, budget)
prompts = {
"alpha": [1, 2, 3, 4],
"bravo": [40, 39, 2, 15],
"charlie": [7, 7, 7, 7],
"delta": [31, 5, 18, 22],
}
n_new = 10
references = {sid: _reference_tokens(model, p, n_new) for sid, p in prompts.items()}
# The four references must diverge, else "no cross-talk" would be vacuous.
assert len({tuple(v) for v in references.values()}) == 4
for sid, prompt in prompts.items():
assert scheduler.submit(_generation(sid, prompt, n_new)).running
outputs = scheduler.run_to_completion()
for sid in prompts:
assert outputs[sid] == references[sid], sid
telem = scheduler.telemetry()
# A genuine batch formed: at least one decode tick carried all four sessions.
assert telem.batch_occupancy_max == 4
assert telem.completed_sessions == 4
assert telem.active_sessions == 0
def test_positions_are_isolated_across_different_prompt_lengths():
"Sessions with different prompt lengths keep independent positions when batched.\n\nTags: node, scheduler, batching"
model = _KvDenseLlama()
engine = _make_engine(model)
scheduler = ContinuousBatchScheduler(
engine, NodeBudget(max_active_sessions=3, max_batch_size=3, max_queue_depth=3)
)
jobs = {
"short": ([5], 6),
"medium": ([2, 9, 14], 6),
"long": ([1, 2, 3, 4, 5, 6, 7], 6),
}
refs = {sid: _reference_tokens(model, p, n) for sid, (p, n) in jobs.items()}
for sid, (prompt, n) in jobs.items():
scheduler.submit(_generation(sid, prompt, n))
outputs = scheduler.run_to_completion()
for sid in jobs:
assert outputs[sid] == refs[sid], sid
# --------------------------------------------------------------------------- #
# Prefill does not starve decode.
# --------------------------------------------------------------------------- #
def test_prefill_does_not_starve_in_flight_decode():
"A burst of new prefills never stalls an already-decoding session.\n\nTags: node, scheduler, fairness"
model = _KvDenseLlama()
engine = _make_engine(model)
# One prefill per tick (budget == a single prompt) so prefill is throttled and
# we can observe that decode still advances every tick.
budget = NodeBudget(
max_active_sessions=8,
max_batch_size=8,
max_queue_depth=8,
scratch_bytes_per_session=1,
scratch_budget_bytes=8,
max_prefill_tokens_per_tick=4,
)
scheduler = ContinuousBatchScheduler(engine, budget)
# Session A starts and prefills on tick 1.
scheduler.submit(_generation("A", [3, 14, 1, 5], 12))
scheduler.run_tick()
a_state = scheduler.session_result("A")
assert a_state.phase is Phase.DECODING
a_len = len(a_state.generated)
assert a_len == 1
# Burst of new work arrives while A is decoding.
for sid in ("B", "C", "D", "E"):
scheduler.submit(_generation(sid, [2, 27, 18, 4], 12))
# Over the next few ticks A must decode on *every* tick (never starved),
# while at most one new session prefills per tick (prefill is bounded).
prefill_counts = []
for _ in range(4):
report = scheduler.run_tick()
new_a_len = len(scheduler.session_result("A").generated)
assert new_a_len == a_len + 1, "decode of A stalled while prefills were pending"
a_len = new_a_len
assert "A" in report.decoded
prefill_counts.append(len(report.prefilled))
assert max(prefill_counts) <= 1, "prefill was not bounded per tick"
def test_decode_first_policy_is_explicit_in_a_single_tick():
"In one tick decode of active sessions precedes prefill of new ones.\n\nTags: node, scheduler, fairness"
model = _KvDenseLlama()
engine = _make_engine(model)
scheduler = ContinuousBatchScheduler(
engine,
NodeBudget(max_active_sessions=4, max_batch_size=4, max_queue_depth=4,
scratch_bytes_per_session=1, scratch_budget_bytes=4),
)
scheduler.submit(_generation("live", [1, 2, 3], 8))
scheduler.run_tick() # 'live' prefills, now decoding
scheduler.submit(_generation("fresh", [9, 8, 7], 8))
report = scheduler.run_tick()
assert "live" in report.decoded
assert "fresh" in report.prefilled
# --------------------------------------------------------------------------- #
# Backpressure and bounded memory.
# --------------------------------------------------------------------------- #
def test_backpressure_signals_when_queue_full_then_recovers():
"A full queue rejects new work; a completed session frees a slot for the queue.\n\nTags: node, scheduler, backpressure"
engine = _make_engine()
budget = NodeBudget(
max_active_sessions=1,
max_batch_size=1,
max_queue_depth=1,
scratch_bytes_per_session=1,
scratch_budget_bytes=1,
)
scheduler = ContinuousBatchScheduler(engine, budget)
assert scheduler.submit(_generation("first", [1, 2], 2)).running
assert scheduler.submit(_generation("second", [3, 4], 2)).reason is AdmissionReason.QUEUED
# Both a slot and the queue are full now.
assert scheduler.submit(_generation("third", [5, 6], 2)).reason is AdmissionReason.REJECTED_QUEUE_FULL
# Drain 'first'; the queued 'second' must be pulled into the freed slot.
scheduler.run_to_completion()
outputs = scheduler.outputs()
assert set(outputs) == {"first", "second"}
def test_completed_sessions_release_kv_so_growth_is_bounded():
"Finished sessions release their KV, so total KV returns to zero.\n\nTags: node, scheduler, backpressure"
engine = _make_engine()
scheduler = ContinuousBatchScheduler(
engine, NodeBudget(max_active_sessions=2, max_batch_size=2, max_queue_depth=8)
)
for sid in ("a", "b", "c", "d"):
scheduler.submit(_generation(sid, [1, 2, 3], 4))
scheduler.run_to_completion()
telem = scheduler.telemetry()
assert telem.kv_total_bytes == 0, "KV not released after completion"
assert telem.active_sessions == 0
assert telem.completed_sessions == 4
# --------------------------------------------------------------------------- #
# Telemetry.
# --------------------------------------------------------------------------- #
def test_telemetry_reports_every_required_signal():
"The capability snapshot reports sessions, queue, batch, KV, rates, rejections.\n\nTags: node, scheduler, telemetry"
model = _KvDenseLlama()
engine = _make_engine(model)
clock = _FakeClock()
budget = NodeBudget(max_active_sessions=2, max_batch_size=2, max_queue_depth=1)
scheduler = ContinuousBatchScheduler(engine, budget, clock=clock)
scheduler.submit(_generation("x", [1, 2, 3], 4))
scheduler.submit(_generation("y", [4, 5, 6], 4))
scheduler.submit(_generation("z", [7, 8, 9], 4)) # queued
rejected = scheduler.submit(_generation("w", [1, 1, 1], 4)) # queue full
assert rejected.rejected
clock.advance(1.0)
scheduler.run_tick() # both prefill
clock.advance(1.0)
scheduler.run_tick() # both decode as a batch of 2
clock.advance(2.0)
telem = scheduler.telemetry()
snap = telem.to_dict()
for key in (
"active_sessions", "queue_depth", "batch_occupancy_last",
"batch_occupancy_avg", "batch_occupancy_max", "weight_bytes",
"kv_total_bytes", "kv_budget_bytes", "kv_pressure",
"scratch_used_bytes", "scratch_budget_bytes", "scratch_pressure",
"prefill_tokens_total", "decode_tokens_total",
"prefill_tokens_per_sec", "decode_tokens_per_sec",
"rejected_admissions_total", "rejected_by_reason",
"completed_sessions", "ticks",
):
assert key in snap, key
assert telem.batch_occupancy_max == 2
assert telem.prefill_tokens_total == 6 # two prompts of length 3
assert telem.decode_tokens_total == 2 # one batched decode step, two sessions
assert telem.rejected_admissions_total == 1
# Rates are deterministic under the injected clock: 4 seconds elapsed.
assert telem.decode_tokens_per_sec == pytest.approx(2 / 4.0)
assert telem.prefill_tokens_per_sec == pytest.approx(6 / 4.0)
assert 0.0 < telem.kv_pressure <= 1.0
# --------------------------------------------------------------------------- #
# Concurrency 1/2/4/8 sweep: saturation and no corruption.
# --------------------------------------------------------------------------- #
def test_concurrency_sweep_identifies_saturation_without_corruption():
"A 1/2/4/8 sweep raises batch occupancy, cuts ticks, and never corrupts output.\n\nTags: node, scheduler, benchmark"
model = _KvDenseLlama()
prompts = {
"s0": [1, 2, 3, 4], "s1": [5, 6, 7, 8], "s2": [9, 10, 11, 12],
"s3": [13, 14, 15, 16], "s4": [17, 18, 19, 20], "s5": [21, 22, 23, 24],
"s6": [25, 26, 27, 28], "s7": [29, 30, 31, 32],
}
n_new = 8
requests = [_generation(sid, p, n_new) for sid, p in prompts.items()]
sweep = run_concurrency_sweep(
lambda: _make_engine(model),
requests,
concurrency_levels=(1, 2, 4, 8),
)
assert sweep.corruption_free
assert [r.concurrency for r in sweep.results] == [1, 2, 4, 8]
# No session hit a cache miss (budgets are sized to never evict here).
assert all(r.cache_misses == 0 for r in sweep.results)
assert all(r.rejected_admissions == 0 for r in sweep.results)
# Each per-session stream matches the serialized (concurrency-1) reference.
for sid, prompt in prompts.items():
assert list(sweep.reference_outputs[sid]) == _reference_tokens(model, prompt, n_new)
occupancies = [r.avg_batch_occupancy for r in sweep.results]
ticks = [r.ticks for r in sweep.results]
tokens_per_tick = [r.tokens_per_tick for r in sweep.results]
# Batching packs more sessions per decode step as concurrency rises, so
# average occupancy strictly increases and total ticks strictly decrease.
assert occupancies == sorted(occupancies) and len(set(occupancies)) == 4
assert ticks == sorted(ticks, reverse=True) and len(set(ticks)) == 4
# Aggregate work per tick rises with concurrency (the throughput win).
assert tokens_per_tick == sorted(tokens_per_tick)
# For eight equal-length jobs the node keeps saturating up to the top level.
assert sweep.saturation_concurrency == 8
# The report is JSON-safe for durable evidence.
import json
json.dumps(sweep.to_dict())
def test_concurrency_sweep_saturates_below_max_when_load_is_small():
"With fewer concurrent jobs than slots, saturation is found below the top level.\n\nTags: node, scheduler, benchmark"
model = _KvDenseLlama()
# Only three jobs: at concurrency 4 and 8 the batch can never exceed 3, so
# occupancy stops rising past the load and saturation is detected early.
requests = [
_generation("j0", [1, 2, 3], 6),
_generation("j1", [4, 5, 6], 6),
_generation("j2", [7, 8, 9], 6),
]
sweep = run_concurrency_sweep(
lambda: _make_engine(model), requests, concurrency_levels=(1, 2, 4, 8)
)
assert sweep.corruption_free
assert sweep.saturation_concurrency <= 4
# Levels at or above the load size share the same occupancy/tick profile.
top = [r for r in sweep.results if r.concurrency >= 4]
assert len({r.ticks for r in top}) == 1
# --------------------------------------------------------------------------- #
# Engine contract guards.
# --------------------------------------------------------------------------- #
def test_kv_batch_engine_requires_a_full_shard():
"The batch engine rejects a partial (non head+tail) shard.\n\nTags: node, scheduler"
model = _KvDenseLlama()
head = _KvReferenceShard(model, 0, 2) # head only, not tail
manager = HotKvStateManager(kv_recipe_for(head))
adapter = KvBoundaryAdapter(head, manager)
with pytest.raises(Exception):
KvBatchEngine(adapter)
def test_run_to_completion_is_bounded_against_misconfiguration():
"run_to_completion raises rather than looping forever when work cannot drain.\n\nTags: node, scheduler"
engine = _make_engine()
scheduler = ContinuousBatchScheduler(
engine, NodeBudget(max_active_sessions=1, max_batch_size=1, max_queue_depth=4)
)
scheduler.submit(_generation("only", [1, 2], 3))
# A tiny explicit tick ceiling is exceeded deterministically.
with pytest.raises(Exception):
scheduler.run_to_completion(max_ticks=1)

488
tests/test_boundary_adapter.py
View File

@@ -1,488 +0,0 @@
"""Architecture-defined boundary input/output and dense-Llama parity (DGR-006).
These tests prove the boundary contract with a *pure-numpy* dense-Llama reference
model: no download, no GPU, no torch, no API credit. The reference implements the
same ``ShardComputation`` duck type the real llama.cpp/PyTorch backends expose, so
whole-model execution and a two-range (or three-range) split are the exact same
arithmetic applied to the exact same float32 residual stream. Splitting the layer
stack at a seam and shipping the *unnormalized* residual bundle across a simulated
process boundary must reproduce the whole-model tokens bit-for-bit.
"""
from __future__ import annotations
import numpy as np
import pytest
from meshnet_node.boundary_adapter import (
BOUNDARY_SCHEMA_VERSION,
BoundaryAdapter,
BoundaryBundle,
BoundaryContractError,
SamplingContract,
ShardRole,
TailOutput,
UncertifiedArchitectureError,
certified_architecture,
is_certified_architecture,
role_for_range,
)
# Documented parity tolerance. The split path applies the identical layer
# functions in the identical order to the identical float32 arrays, so the
# residual seam is bit-exact in practice; the tolerance is a conservative guard.
PARITY_ATOL = 1e-6
# --------------------------------------------------------------------------- #
# Pure-numpy dense-Llama reference model (test fixture, not production).
# --------------------------------------------------------------------------- #
class _ReferenceDenseLlama:
"""A tiny deterministic dense-Llama: RMSNorm, RoPE attention, SwiGLU MLP."""
architecture_adapter = "dense-llama"
def __init__(
self,
*,
vocab: int = 48,
hidden: int = 32,
n_layers: int = 6,
n_heads: int = 4,
intermediate: int = 64,
rms_eps: float = 1e-6,
rope_theta: float = 10000.0,
seed: int = 20260715,
) -> None:
assert hidden % n_heads == 0
self.vocab = vocab
self.hidden = hidden
self.n_layers = n_layers
self.n_heads = n_heads
self.head_dim = hidden // n_heads
assert self.head_dim % 2 == 0
self.rms_eps = rms_eps
self.rope_theta = rope_theta
rng = np.random.default_rng(seed)
def w(*shape: int) -> np.ndarray:
return (rng.standard_normal(shape) * 0.08).astype(np.float32)
self.embed = w(vocab, hidden)
self.layers = []
for _ in range(n_layers):
self.layers.append(
{
"in_ln": (1.0 + rng.standard_normal(hidden) * 0.02).astype(np.float32),
"q": w(hidden, hidden),
"k": w(hidden, hidden),
"v": w(hidden, hidden),
"o": w(hidden, hidden),
"post_ln": (1.0 + rng.standard_normal(hidden) * 0.02).astype(np.float32),
"gate": w(intermediate, hidden),
"up": w(intermediate, hidden),
"down": w(hidden, intermediate),
}
)
self.final_ln = (1.0 + rng.standard_normal(hidden) * 0.02).astype(np.float32)
self.lm_head_w = w(vocab, hidden)
inv_freq = 1.0 / (
rope_theta ** (np.arange(0, self.head_dim, 2, dtype=np.float32) / self.head_dim)
)
self.inv_freq = inv_freq.astype(np.float32)
# -- primitive ops -----------------------------------------------------
def _rmsnorm(self, x: np.ndarray, weight: np.ndarray) -> np.ndarray:
variance = np.mean(x.astype(np.float32) ** 2, axis=-1, keepdims=True)
normed = x / np.sqrt(variance + self.rms_eps)
return (normed * weight).astype(np.float32)
def _rope(self, positions: np.ndarray):
# positions: (batch, seq) -> cos/sin: (batch, seq, head_dim)
angles = positions[..., None].astype(np.float32) * self.inv_freq[None, None, :]
emb = np.concatenate([angles, angles], axis=-1)
return np.cos(emb).astype(np.float32), np.sin(emb).astype(np.float32)
@staticmethod
def _rotate_half(x: np.ndarray) -> np.ndarray:
half = x.shape[-1] // 2
return np.concatenate([-x[..., half:], x[..., :half]], axis=-1)
def _apply_rope(self, t: np.ndarray, cos: np.ndarray, sin: np.ndarray) -> np.ndarray:
# t: (batch, n_heads, seq, head_dim); cos/sin: (batch, seq, head_dim)
cos = cos[:, None, :, :]
sin = sin[:, None, :, :]
return t * cos + self._rotate_half(t) * sin
def _attention(self, x: np.ndarray, layer: dict, positions: np.ndarray) -> np.ndarray:
batch, seq, _ = x.shape
q = (x @ layer["q"].T).reshape(batch, seq, self.n_heads, self.head_dim)
k = (x @ layer["k"].T).reshape(batch, seq, self.n_heads, self.head_dim)
v = (x @ layer["v"].T).reshape(batch, seq, self.n_heads, self.head_dim)
q = q.transpose(0, 2, 1, 3)
k = k.transpose(0, 2, 1, 3)
v = v.transpose(0, 2, 1, 3)
cos, sin = self._rope(positions)
q = self._apply_rope(q, cos, sin)
k = self._apply_rope(k, cos, sin)
scores = (q @ k.transpose(0, 1, 3, 2)) / np.sqrt(self.head_dim)
causal = np.triu(np.full((seq, seq), -1e30, dtype=np.float32), k=1)
scores = scores + causal[None, None, :, :]
scores = scores - scores.max(axis=-1, keepdims=True)
weights = np.exp(scores)
weights = weights / weights.sum(axis=-1, keepdims=True)
out = weights @ v
out = out.transpose(0, 2, 1, 3).reshape(batch, seq, self.hidden)
return (out @ layer["o"].T).astype(np.float32)
def _mlp(self, x: np.ndarray, layer: dict) -> np.ndarray:
gate = x @ layer["gate"].T
up = x @ layer["up"].T
silu = gate * (1.0 / (1.0 + np.exp(-gate)))
return ((silu * up) @ layer["down"].T).astype(np.float32)
def _run_layer(self, x: np.ndarray, layer: dict, positions: np.ndarray) -> np.ndarray:
h = x + self._attention(self._rmsnorm(x, layer["in_ln"]), layer, positions)
h = h + self._mlp(self._rmsnorm(h, layer["post_ln"]), layer)
return h.astype(np.float32)
class _ReferenceShard:
"""A contiguous inclusive layer range of the reference model.
Satisfies the ``ShardComputation`` duck type used by ``BoundaryAdapter``.
"""
def __init__(
self,
model: _ReferenceDenseLlama,
start_layer: int,
end_layer: int,
*,
architecture_adapter: str | None = None,
) -> None:
self._model = model
self.start_layer = start_layer
self.end_layer = end_layer
self.total_layers = model.n_layers
self.architecture_adapter = architecture_adapter or model.architecture_adapter
def embed_tokens(self, token_ids: np.ndarray) -> np.ndarray:
return self._model.embed[np.asarray(token_ids)]
def run_layers(self, hidden: np.ndarray, *, positions: np.ndarray) -> np.ndarray:
h = np.asarray(hidden, dtype=np.float32)
for idx in range(self.start_layer, self.end_layer + 1):
h = self._model._run_layer(h, self._model.layers[idx], positions)
return h
def final_norm(self, hidden: np.ndarray) -> np.ndarray:
return self._model._rmsnorm(np.asarray(hidden, dtype=np.float32), self._model.final_ln)
def lm_head(self, hidden: np.ndarray) -> np.ndarray:
return np.asarray(hidden, dtype=np.float32) @ self._model.lm_head_w.T
# --------------------------------------------------------------------------- #
# Whole-model and split reference drivers.
# --------------------------------------------------------------------------- #
def _whole_model_next_token(model: _ReferenceDenseLlama, token_ids: list[int]) -> TailOutput:
shard = _ReferenceShard(model, 0, model.n_layers - 1)
adapter = BoundaryAdapter(shard)
result = adapter.forward(token_ids=np.asarray(token_ids)[None, :])
assert isinstance(result, TailOutput)
return result
def _split_next_token(
model: _ReferenceDenseLlama,
token_ids: list[int],
cut_points: list[int],
*,
through_wire: bool = True,
) -> TailOutput:
"""Run the model as N contiguous ranges, shipping the bundle across each seam.
``cut_points`` are the last (inclusive) layer of each non-final range.
"""
bounds = _ranges_from_cuts(cut_points, model.n_layers)
boundary: BoundaryBundle | None = None
result: BoundaryBundle | TailOutput | None = None
for i, (start, end) in enumerate(bounds):
shard = _ReferenceShard(model, start, end)
adapter = BoundaryAdapter(shard)
if i == 0:
result = adapter.forward(token_ids=np.asarray(token_ids)[None, :])
else:
assert isinstance(boundary, BoundaryBundle)
incoming = BoundaryBundle.unpack(boundary.pack()) if through_wire else boundary
result = adapter.forward(boundary=incoming)
if isinstance(result, BoundaryBundle):
boundary = result
assert isinstance(result, TailOutput)
return result
def _ranges_from_cuts(cut_points: list[int], n_layers: int) -> list[tuple[int, int]]:
bounds: list[tuple[int, int]] = []
start = 0
for cut in cut_points:
bounds.append((start, cut))
start = cut + 1
bounds.append((start, n_layers - 1))
return bounds
def _greedy_generate(next_token_fn, prompt: list[int], n_new: int) -> list[int]:
tokens = list(prompt)
generated: list[int] = []
for _ in range(n_new):
out = next_token_fn(tokens)
tokens.append(out.token_id)
generated.append(out.token_id)
return generated
# --------------------------------------------------------------------------- #
# Certification / fail-closed.
# --------------------------------------------------------------------------- #
def test_dense_llama_and_aliases_are_certified():
"Dense Llama-family identifiers all resolve to the one certified adapter.\n\nTags: node, boundary"
for name in ("dense-llama", "llama", "LlamaForCausalLM", "LlamaModel"):
boundary = certified_architecture(name)
assert boundary.adapter == "dense-llama"
assert boundary.boundary_tensor_name == "residual_stream"
assert is_certified_architecture(name)
@pytest.mark.parametrize("name", ["qwen3", "qwen3-moe", "mixtral", "gpt2", "", None, 123])
def test_uncertified_architectures_fail_closed(name):
"Uncertified architectures raise instead of guessing a tensor layout.\n\nTags: node, boundary"
assert not is_certified_architecture(name)
with pytest.raises(UncertifiedArchitectureError):
certified_architecture(name)
def test_adapter_construction_fails_closed_for_uncertified_backend():
"Building the adapter over an uncertified computation fails closed.\n\nTags: node, boundary"
model = _ReferenceDenseLlama()
shard = _ReferenceShard(model, 0, 2, architecture_adapter="qwen3-moe")
with pytest.raises(UncertifiedArchitectureError):
BoundaryAdapter(shard)
# --------------------------------------------------------------------------- #
# Roles.
# --------------------------------------------------------------------------- #
def test_role_classification():
"Range endpoints map to head/middle/tail/full roles.\n\nTags: node, boundary"
assert role_for_range(0, 2, 6) is ShardRole.HEAD
assert role_for_range(2, 3, 6) is ShardRole.MIDDLE
assert role_for_range(4, 5, 6) is ShardRole.TAIL
assert role_for_range(0, 5, 6) is ShardRole.FULL
assert ShardRole.HEAD.owns_embedding and not ShardRole.HEAD.owns_final_head
assert ShardRole.TAIL.owns_final_head and not ShardRole.TAIL.owns_embedding
# --------------------------------------------------------------------------- #
# Input-side contract.
# --------------------------------------------------------------------------- #
def test_head_accepts_token_ids_and_owns_embedding():
"The head embeds token IDs and refuses an upstream boundary bundle.\n\nTags: node, boundary"
model = _ReferenceDenseLlama()
head = BoundaryAdapter(_ReferenceShard(model, 0, 2))
out = head.forward(token_ids=[1, 2, 3])
assert isinstance(out, BoundaryBundle)
# Head owns embedding: a residual bundle from upstream is a contract error.
bundle = out
with pytest.raises(BoundaryContractError, match="head owns token embedding"):
head.forward(boundary=bundle)
def test_middle_and_tail_bypass_embedding_and_require_the_bundle():
"Middle/tail Shards reject token IDs and demand the named boundary bundle.\n\nTags: node, boundary"
model = _ReferenceDenseLlama()
tail = BoundaryAdapter(_ReferenceShard(model, 3, 5))
with pytest.raises(BoundaryContractError, match="bypass token embedding"):
tail.forward(token_ids=[1, 2, 3])
with pytest.raises(BoundaryContractError, match="must receive the named boundary bundle"):
tail.forward()
def test_boundary_seam_layer_mismatch_is_rejected():
"A bundle handed to the wrong range (seam layer mismatch) is rejected.\n\nTags: node, boundary"
model = _ReferenceDenseLlama()
head = BoundaryAdapter(_ReferenceShard(model, 0, 2))
bundle = head.forward(token_ids=[1, 2, 3])
assert isinstance(bundle, BoundaryBundle)
assert bundle.next_layer == 3
# A range that starts at layer 4 must not accept a bundle cut at layer 3.
wrong = BoundaryAdapter(_ReferenceShard(model, 4, 5))
with pytest.raises(BoundaryContractError, match="starts at layer 4"):
wrong.forward(boundary=bundle)
def test_normalized_bundle_is_rejected():
"A normalized residual is not the architecture-defined boundary.\n\nTags: node, boundary"
model = _ReferenceDenseLlama()
head = BoundaryAdapter(_ReferenceShard(model, 0, 2))
bundle = head.forward(token_ids=[1, 2, 3])
assert isinstance(bundle, BoundaryBundle)
normalized = BoundaryBundle(
architecture_adapter=bundle.architecture_adapter,
schema_version=bundle.schema_version,
tensor_name=bundle.tensor_name,
residual=bundle.residual,
positions=bundle.positions,
next_layer=bundle.next_layer,
normalized=True,
)
tail = BoundaryAdapter(_ReferenceShard(model, 3, 5))
with pytest.raises(BoundaryContractError, match="UNNORMALIZED"):
tail.forward(boundary=normalized)
# --------------------------------------------------------------------------- #
# Output-side contract.
# --------------------------------------------------------------------------- #
def test_non_tail_emits_unnormalized_full_row_boundary():
"A non-tail Shard emits the unnormalized residual with every position row.\n\nTags: node, boundary"
model = _ReferenceDenseLlama()
tokens = [3, 7, 1, 9, 2]
head = BoundaryAdapter(_ReferenceShard(model, 0, 2))
bundle = head.forward(token_ids=tokens)
assert isinstance(bundle, BoundaryBundle)
assert bundle.normalized is False
assert bundle.tensor_name == "residual_stream"
assert bundle.schema_version == BOUNDARY_SCHEMA_VERSION
assert bundle.next_layer == 3
# No tail-only row pruning: all sequence positions are forwarded.
assert bundle.residual.shape == (1, len(tokens), model.hidden)
assert bundle.positions.shape == (1, len(tokens))
# The emitted residual must be exactly the whole model's residual after layer 2
# (i.e. before any final norm) — prove it is NOT normalized.
positions = np.arange(len(tokens))[None, :]
hidden = model.embed[np.asarray(tokens)][None, :]
for idx in range(0, 3):
hidden = model._run_layer(hidden, model.layers[idx], positions)
assert np.allclose(bundle.residual, hidden, atol=0)
assert not np.allclose(bundle.residual, model._rmsnorm(hidden, model.final_ln))
def test_tail_emits_pruned_logits_through_the_sampling_contract():
"The tail prunes to the final row and samples through an explicit contract.\n\nTags: node, boundary"
model = _ReferenceDenseLlama()
out = _whole_model_next_token(model, [4, 8, 15, 16, 23])
assert isinstance(out, TailOutput)
assert out.logits.shape == (1, model.vocab) # tail-only row pruning to last row
assert out.sampling.mode == "greedy"
assert 0 <= out.token_id < model.vocab
assert out.token_id == int(np.argmax(out.logits[0]))
def test_sampling_contract_rejects_uncertified_modes():
"Only the certified greedy sampling mode is accepted.\n\nTags: node, boundary"
with pytest.raises(BoundaryContractError):
SamplingContract(mode="top_p")
# --------------------------------------------------------------------------- #
# The core parity gate.
# --------------------------------------------------------------------------- #
def test_two_range_prefill_parity_matches_whole_model():
"Whole-model vs two-range prefill produce the same next-token logits and token.\n\nTags: node, boundary, parity"
model = _ReferenceDenseLlama()
prompt = [5, 12, 3, 41, 7, 19, 2, 33]
whole = _whole_model_next_token(model, prompt)
split = _split_next_token(model, prompt, cut_points=[2])
assert np.allclose(whole.logits, split.logits, atol=PARITY_ATOL)
assert whole.token_id == split.token_id
def test_three_range_prefill_parity_exercises_the_middle_role():
"A head/middle/tail split reproduces whole-model prefill through two seams.\n\nTags: node, boundary, parity"
model = _ReferenceDenseLlama()
prompt = [9, 1, 44, 6, 30, 11]
whole = _whole_model_next_token(model, prompt)
split = _split_next_token(model, prompt, cut_points=[1, 3])
assert np.allclose(whole.logits, split.logits, atol=PARITY_ATOL)
assert whole.token_id == split.token_id
def test_two_range_greedy_decode_parity_matches_whole_model():
"Whole-model vs two-range greedy decode produce identical token sequences.\n\nTags: node, boundary, parity"
model = _ReferenceDenseLlama()
prompt = [2, 17, 8, 25]
n_new = 12
whole_tokens = _greedy_generate(
lambda toks: _whole_model_next_token(model, toks), prompt, n_new
)
split_tokens = _greedy_generate(
lambda toks: _split_next_token(model, toks, cut_points=[2]), prompt, n_new
)
assert whole_tokens == split_tokens
assert len(whole_tokens) == n_new
def test_boundary_bundle_wire_round_trip_is_exact():
"Packing and unpacking the boundary bundle reconstructs the exact arrays.\n\nTags: node, boundary"
model = _ReferenceDenseLlama()
head = BoundaryAdapter(_ReferenceShard(model, 0, 2))
bundle = head.forward(token_ids=[1, 2, 3, 4])
assert isinstance(bundle, BoundaryBundle)
restored = BoundaryBundle.unpack(bundle.pack())
assert np.array_equal(restored.residual, bundle.residual)
assert np.array_equal(restored.positions, bundle.positions)
assert restored.next_layer == bundle.next_layer
assert restored.architecture_adapter == bundle.architecture_adapter
fields = bundle.named_tensor_fields()
assert fields["name"] == "residual_stream"
assert fields["shape"] == [1, 4, model.hidden]
assert fields["byte_order"] in ("little", "big")
def test_alias_architecture_still_parity_matches():
"A Shard advertised as 'llama' interoperates with the canonical adapter.\n\nTags: node, boundary, parity"
model = _ReferenceDenseLlama()
prompt = [7, 3, 22, 5]
whole = _whole_model_next_token(model, prompt)
# Head advertises 'LlamaForCausalLM', tail advertises 'llama'; both certify to
# the same canonical adapter, so the seam contract still matches.
head = BoundaryAdapter(_ReferenceShard(model, 0, 2, architecture_adapter="LlamaForCausalLM"))
bundle = head.forward(token_ids=np.asarray(prompt)[None, :])
assert isinstance(bundle, BoundaryBundle)
tail = BoundaryAdapter(_ReferenceShard(model, 3, 5, architecture_adapter="llama"))
split = tail.forward(boundary=BoundaryBundle.unpack(bundle.pack()))
assert isinstance(split, TailOutput)
assert np.allclose(whole.logits, split.logits, atol=PARITY_ATOL)
assert whole.token_id == split.token_id

611
tests/test_failure_semantics.py
View File

@@ -1,611 +0,0 @@
"""Bounded failure, cancellation, and restart semantics (DGR-013).
These tests drive the hardened per-session decode stream with the *same*
pure-numpy KV-cached dense-Llama reference the Hot KV State manager (DGR-007) and
the continuous-batch scheduler (DGR-012) use, imported from ``test_hot_kv_state``.
The whole matrix stays deterministic, download-free, GPU-free, and API-credit-free
while exercising the real KV isolation path (``KvBoundaryAdapter`` +
``HotKvStateManager``) rather than a mock.
Coverage maps to the story's acceptance criteria:
* deadlines and heartbeat/health loss terminate blocked stream operations,
* cancellation propagates across every Shard and releases KV + queued buffers,
* duplicate steps are idempotent; uncertain mutations are never replayed silently,
* alpha failover restarts from token zero rather than importing unverified KV,
* worker death / stream reset / malformed bundle / stale epoch / cache miss,
* billing/work records distinguish completed, cancelled, failed, and unverified.
"""
from __future__ import annotations
import json
import numpy as np
import pytest
from meshnet_node.batch_scheduler import (
ContinuousBatchScheduler,
DoneReason,
GenerationRequest,
KvBatchEngine,
NodeBudget,
)
from meshnet_node.boundary_adapter import BoundaryBundle, BoundaryContractError
from meshnet_node.hot_kv_state import (
CacheMiss,
CacheMissReason,
HotKvStateConfig,
HotKvStateManager,
KvBoundaryAdapter,
StaleRouteEpochError,
kv_recipe_for,
)
from meshnet_node.failure_semantics import (
CancellationToken,
DeadlineGuard,
FailureKind,
HardenedSessionRunner,
IdempotencyLedger,
OperationCancelled,
RestartController,
ShardCancellationGroup,
StepKey,
StreamTerminated,
UncertainMutationError,
WorkLedger,
WorkRecord,
WorkStatus,
classify_exception,
work_status_for,
)
# Reuse the certified numpy dense-Llama reference and shard from the DGR-007 gate.
from test_hot_kv_state import _KvDenseLlama, _KvReferenceShard
# --------------------------------------------------------------------------- #
# Helpers.
# --------------------------------------------------------------------------- #
class _FakeClock:
def __init__(self) -> None:
self.now = 0.0
def __call__(self) -> float:
return self.now
def advance(self, delta: float) -> None:
self.now += delta
class _FaultyShard(_KvReferenceShard):
"""A full-shard reference that raises on the Nth ``run_layers_cached`` call.
``run_layers_cached`` is invoked once per stream step, so ``fail_at_call=k``
simulates a worker dying at step ``k-1`` (calls are 1-indexed). The call
counter persists across attempts, so a restart on a fresh epoch keeps counting
and does not re-trip the same fault.
"""
def __init__(self, model, start, end, *, fail_at_call=None, error=None):
super().__init__(model, start, end)
self._fail_at_call = fail_at_call
self._error = error or RuntimeError("worker died mid-step")
self.calls = 0
def run_layers_cached(self, hidden, *, positions, past_kv):
self.calls += 1
if self._fail_at_call is not None and self.calls == self._fail_at_call:
raise self._error
return super().run_layers_cached(hidden, positions=positions, past_kv=past_kv)
def _make_adapter(model=None, *, config=None, shard=None):
"""A full-shard KV boundary adapter over the deterministic numpy dense-Llama."""
model = model or _KvDenseLlama()
shard = shard or _KvReferenceShard(model, 0, model.n_layers - 1)
manager = HotKvStateManager(kv_recipe_for(shard), config=config)
adapter = KvBoundaryAdapter(shard, manager)
return adapter
def _generation(session_id, prompt, n_new, epoch=0):
return GenerationRequest(
session_id=session_id,
route_epoch=epoch,
prompt_token_ids=tuple(prompt),
max_new_tokens=n_new,
)
# --------------------------------------------------------------------------- #
# Happy path (the baseline the failure paths deviate from).
# --------------------------------------------------------------------------- #
def test_clean_run_matches_stateless_reference_and_is_billable():
"A clean stream reproduces the stateless tokens and records completed work.\n\nTags: node, failure, billing"
model = _KvDenseLlama()
adapter = _make_adapter(model)
runner = HardenedSessionRunner(adapter)
prompt = [1, 2, 3, 4]
n_new = 8
outcome = runner.run(_generation("clean", prompt, n_new))
assert outcome.status is WorkStatus.COMPLETED
assert list(outcome.tokens) == model.stateless_greedy(prompt, n_new)
record = runner.work_ledger.records_for("clean")[0]
assert record.billable
assert record.tokens == n_new
assert runner.work_ledger.billable_tokens() == n_new
# --------------------------------------------------------------------------- #
# Deadlines and heartbeat/health loss terminate blocked operations.
# --------------------------------------------------------------------------- #
def test_deadline_terminates_a_blocked_stream_and_releases_kv():
"A deadline reached mid-stream terminates the run and frees its KV.\n\nTags: node, failure, deadline"
clock = _FakeClock()
adapter = _make_adapter()
manager = adapter.manager
runner = HardenedSessionRunner(adapter, clock=clock)
# Each step advances the clock by 1.0; the deadline fires at t=3.
def before_step(_step):
clock.advance(1.0)
outcome = runner.run(
_generation("slow", [5, 6, 7], 20),
deadline=3.0,
before_step=before_step,
)
assert outcome.status is WorkStatus.FAILED
assert outcome.failure_kind is FailureKind.DEADLINE_EXCEEDED
# The stream did not hang and did not finish: only the steps before the
# deadline committed, and the session's KV was released.
assert outcome.token_count < 20
assert isinstance(manager.resolve("slow", 0), CacheMiss)
def test_heartbeat_loss_terminates_a_blocked_stream():
"Losing the peer heartbeat past the timeout terminates the stream.\n\nTags: node, failure, heartbeat"
clock = _FakeClock()
adapter = _make_adapter()
runner = HardenedSessionRunner(adapter, clock=clock)
def before_step(_step):
clock.advance(1.0)
# Heartbeats stop arriving after step 2; with a timeout of 1.5 the gap grows
# past the bound and the stream is terminated (health loss).
def heartbeat(step):
return step < 2
outcome = runner.run(
_generation("hb", [9, 8, 7], 20),
heartbeat_timeout=1.5,
heartbeat=heartbeat,
before_step=before_step,
)
assert outcome.status is WorkStatus.FAILED
assert outcome.failure_kind is FailureKind.HEARTBEAT_LOST
assert outcome.token_count < 20
def test_deadline_guard_reports_remaining_and_resets_on_heartbeat():
"The guard exposes remaining time and a heartbeat resets the health timer.\n\nTags: node, failure, deadline"
clock = _FakeClock()
guard = DeadlineGuard(deadline=10.0, heartbeat_timeout=2.0, clock=clock)
guard.start()
guard.check()
assert guard.remaining() == 10.0
clock.advance(1.5)
guard.heartbeat() # health refreshed at t=1.5
clock.advance(1.0) # gap since heartbeat is 1.0 < 2.0
guard.check()
clock.advance(2.5) # gap since heartbeat is now 3.5 > 2.0
with pytest.raises(StreamTerminated) as exc:
guard.check()
assert exc.value.kind is FailureKind.HEARTBEAT_LOST
# --------------------------------------------------------------------------- #
# Cancellation propagates across shards and releases KV + queued buffers.
# --------------------------------------------------------------------------- #
def test_cancellation_token_terminates_stream_and_releases_kv():
"A client cancel mid-stream stops the run and releases the session KV.\n\nTags: node, failure, cancel"
adapter = _make_adapter()
manager = adapter.manager
token = CancellationToken()
runner = HardenedSessionRunner(adapter)
# Cancel after two steps have run.
def before_step(step):
if step == 2:
token.cancel("client-hangup")
outcome = runner.run(
_generation("cancelme", [1, 2, 3], 20),
cancel_token=token,
before_step=before_step,
)
assert outcome.status is WorkStatus.CANCELLED
assert outcome.failure_kind is FailureKind.CANCELLED
assert outcome.token_count == 2 # steps 0 and 1 committed before the cancel
assert isinstance(manager.resolve("cancelme", 0), CacheMiss)
def test_shard_cancellation_group_releases_every_shard_and_queued_buffers():
"One cancel frees KV on every node-local shard and releases queued buffers.\n\nTags: node, failure, cancel"
model = _KvDenseLlama()
# Three node-local shards of the same route, each with its own KV manager.
managers = []
for start, end in ((0, 1), (2, 3), (4, 5)):
shard = _KvReferenceShard(model, start, end)
mgr = HotKvStateManager(kv_recipe_for(shard))
mgr.open("route", 0) # each holds live state for the session
managers.append(mgr)
released_buffers = []
group = ShardCancellationGroup("route", 0)
for mgr in managers:
group.add_shard(mgr)
group.add_queued_buffer(lambda: released_buffers.append("bundle-a"))
group.add_queued_buffer(lambda: released_buffers.append("bundle-b"))
outcome = group.cancel()
assert outcome.shards_released == 3
assert outcome.buffers_released == 2
assert released_buffers == ["bundle-a", "bundle-b"]
# Every shard's KV is gone: a lookup now yields an explicit released miss.
for mgr in managers:
miss = mgr.resolve("route", 0)
assert isinstance(miss, CacheMiss)
assert miss.reason is CacheMissReason.RELEASED
# Cancellation is idempotent.
again = group.cancel()
assert again.shards_released == 0
assert again.buffers_released == 0
def test_scheduler_cancel_drains_queue_and_releases_active_kv():
"The scheduler cancel drops queued work and frees an active session's KV.\n\nTags: node, scheduler, cancel"
model = _KvDenseLlama()
shard = _KvReferenceShard(model, 0, model.n_layers - 1)
manager = HotKvStateManager(kv_recipe_for(shard))
engine = KvBatchEngine(KvBoundaryAdapter(shard, manager))
scheduler = ContinuousBatchScheduler(
engine, NodeBudget(max_active_sessions=1, max_batch_size=1, max_queue_depth=4)
)
assert scheduler.submit(_generation("active", [1, 2, 3], 8)).running
assert scheduler.submit(_generation("waiting", [4, 5, 6], 8)).reason.value == "queued"
scheduler.run_tick() # 'active' prefills and starts decoding, holding KV
# Cancel the queued one: it leaves the queue without ever taking a slot.
assert scheduler.cancel("waiting") is True
# Cancel the active one: its KV is released and it is recorded as cancelled.
assert scheduler.cancel("active") is True
assert manager.total_bytes == 0
telem = scheduler.telemetry()
assert telem.cancelled_sessions == 2
assert telem.completed_sessions == 0
assert telem.active_sessions == 0
assert telem.queue_depth == 0
# Cancelling an unknown / already-finished session is a no-op.
assert scheduler.cancel("active") is False
assert scheduler.cancel("never-seen") is False
def test_scheduler_cancel_rejects_a_completed_reason():
"cancel() refuses a non-terminal reason so completed work is never faked.\n\nTags: node, scheduler, cancel"
model = _KvDenseLlama()
shard = _KvReferenceShard(model, 0, model.n_layers - 1)
manager = HotKvStateManager(kv_recipe_for(shard))
engine = KvBatchEngine(KvBoundaryAdapter(shard, manager))
scheduler = ContinuousBatchScheduler(engine)
scheduler.submit(_generation("x", [1, 2], 4))
with pytest.raises(Exception):
scheduler.cancel("x", reason=DoneReason.COMPLETED)
# --------------------------------------------------------------------------- #
# Idempotency: duplicate steps are no-ops; uncertain mutations never replay.
# --------------------------------------------------------------------------- #
def test_duplicate_step_delivery_is_idempotent_no_remutation():
"Replaying a committed step returns the recorded token without re-mutating KV.\n\nTags: node, failure, idempotency"
ledger = IdempotencyLedger()
key = StepKey("s", 0, 5)
disposition = ledger.begin(key)
assert disposition.fresh
ledger.commit(key, 42)
# A duplicate delivery of the same step returns the recorded token and is a
# no-op — the caller must not re-run the mutation.
replay = ledger.begin(key)
assert replay.duplicate
assert replay.token == 42
def test_idempotent_run_replays_tokens_without_advancing_kv():
"Re-running a completed stream on the same ledger/epoch re-mutates nothing.\n\nTags: node, failure, idempotency"
model = _KvDenseLlama()
adapter = _make_adapter(model)
ledger = IdempotencyLedger()
runner = HardenedSessionRunner(adapter, idempotency=ledger)
request = _generation("idem", [3, 1, 4], 6)
first = runner.run(request)
assert first.status is WorkStatus.COMPLETED
kv_len_after_first = adapter.manager.get("idem", 0).seq_len
# A duplicate delivery of the entire stream: every step is a committed
# duplicate, so the runner replays the identical tokens and the KV length is
# unchanged (no double-append).
second = runner.run(request)
assert second.status is WorkStatus.COMPLETED
assert list(second.tokens) == list(first.tokens)
assert adapter.manager.get("idem", 0).seq_len == kv_len_after_first
def test_uncertain_mutation_is_never_replayed_silently():
"A step marked uncertain refuses a silent replay; it must be verified/restarted.\n\nTags: node, failure, idempotency"
ledger = IdempotencyLedger()
key = StepKey("s", 0, 3)
ledger.begin(key)
ledger.mark_uncertain(key, "worker died before ack")
# Replaying an uncertain mutation is refused rather than silently re-applied.
with pytest.raises(UncertainMutationError):
ledger.begin(key)
assert ledger.has_uncertain()
def test_in_flight_duplicate_is_treated_as_uncertain():
"A second begin before commit is refused (concurrent duplicate is unverified).\n\nTags: node, failure, idempotency"
ledger = IdempotencyLedger()
key = StepKey("s", 0, 1)
ledger.begin(key) # in-flight, not yet committed
with pytest.raises(UncertainMutationError):
ledger.begin(key)
# --------------------------------------------------------------------------- #
# Worker death, stream reset, malformed bundle, stale epoch, cache miss.
# --------------------------------------------------------------------------- #
def test_worker_death_midstream_is_unverified_and_marks_step_uncertain():
"A worker dying mid-step yields unverified work and an unreplayable step.\n\nTags: node, failure, worker-death"
model = _KvDenseLlama()
# Fail on the 3rd step call (step index 2), after two tokens committed.
shard = _FaultyShard(model, 0, model.n_layers - 1, fail_at_call=3)
adapter = _make_adapter(model, shard=shard)
ledger = IdempotencyLedger()
runner = HardenedSessionRunner(adapter, idempotency=ledger)
outcome = runner.run(_generation("dead", [1, 2, 3], 8))
assert outcome.status is WorkStatus.UNVERIFIED
assert outcome.failure_kind is FailureKind.WORKER_DEATH
assert outcome.token_count == 2 # the two committed steps
assert not outcome.completed
# The failed step is uncertain and can never be silently replayed.
assert ledger.has_uncertain()
with pytest.raises(UncertainMutationError):
ledger.begin(StepKey("dead", 0, 2))
# KV was released on failure.
assert isinstance(adapter.manager.resolve("dead", 0), CacheMiss)
def test_stream_reset_is_restartable_failure():
"A stream reset injected mid-stream fails the run as a restartable transport loss.\n\nTags: node, failure, stream-reset"
adapter = _make_adapter()
runner = HardenedSessionRunner(adapter)
def before_step(step):
if step == 2:
raise StreamTerminated(FailureKind.STREAM_RESET, "peer reset the stream")
outcome = runner.run(_generation("reset", [1, 2, 3], 8), before_step=before_step)
assert outcome.status is WorkStatus.FAILED
assert outcome.failure_kind is FailureKind.STREAM_RESET
assert outcome.restartable
def test_malformed_bundle_is_classified_and_does_not_corrupt_kv():
"A malformed activation bundle is rejected and leaves the KV context empty.\n\nTags: node, failure, malformed-bundle"
model = _KvDenseLlama()
mid = _KvReferenceShard(model, 2, 3) # middle range: not head, not tail
manager = HotKvStateManager(kv_recipe_for(mid))
adapter = KvBoundaryAdapter(mid, manager)
assert not adapter.is_head and not adapter.is_tail
# A bundle that hands over at the wrong layer is malformed.
bad = BoundaryBundle(
architecture_adapter=adapter.architecture.adapter,
schema_version=adapter.architecture.boundary_schema_version,
tensor_name=adapter.architecture.boundary_tensor_name,
residual=np.zeros((1, 3, model.hidden), dtype=np.float32),
positions=np.arange(3, dtype=np.int64)[None, :],
next_layer=adapter.start_layer + 5, # wrong handover layer
normalized=False,
)
with pytest.raises(BoundaryContractError) as exc:
adapter.prefill("mal", 0, boundary=bad)
assert classify_exception(exc.value) is FailureKind.MALFORMED_BUNDLE
# The malformed step never appended KV: the context is empty, not corrupted.
assert manager.get("mal", 0).seq_len == 0
def test_stale_epoch_reference_is_rejected_and_classified():
"A reference to a superseded epoch is rejected as stale, never silently reused.\n\nTags: node, failure, stale-epoch"
model = _KvDenseLlama()
adapter = _make_adapter(model)
manager = adapter.manager
manager.open("sess", 5) # current epoch is now 5
with pytest.raises(StaleRouteEpochError) as exc:
manager.resolve("sess", 4) # epoch 4 is stale
assert classify_exception(exc.value) is FailureKind.STALE_EPOCH
# Driving the hardened runner on the stale epoch fails closed as STALE_EPOCH.
runner = HardenedSessionRunner(adapter)
outcome = runner.run(_generation("sess", [1, 2, 3], 4, epoch=3))
assert outcome.status is WorkStatus.FAILED
assert outcome.failure_kind is FailureKind.STALE_EPOCH
def test_cache_miss_midstream_is_restartable():
"A KV eviction mid-stream surfaces an explicit cache miss the head can restart.\n\nTags: node, failure, cache-miss"
adapter = _make_adapter()
manager = adapter.manager
runner = HardenedSessionRunner(adapter)
# Evict the session's KV just before step 3's decode.
def before_step(step):
if step == 3:
manager.release("evict", 0)
outcome = runner.run(_generation("evict", [1, 2, 3], 10), before_step=before_step)
assert outcome.failure_kind is FailureKind.CACHE_MISS
assert outcome.restartable
assert outcome.token_count == 3 # steps 0..2 committed before the eviction
# --------------------------------------------------------------------------- #
# Alpha failover: restart from token zero, never import unverified KV.
# --------------------------------------------------------------------------- #
def test_alpha_failover_restarts_from_token_zero_and_completes():
"A transient worker death fails over to a fresh epoch and reproduces the tokens.\n\nTags: node, failure, failover"
model = _KvDenseLlama()
# Die on the 3rd step of the first attempt; the persistent call counter means
# the restart (which keeps counting) does not re-trip the fault.
shard = _FaultyShard(model, 0, model.n_layers - 1, fail_at_call=3)
adapter = _make_adapter(model, shard=shard)
manager = adapter.manager
runner = HardenedSessionRunner(adapter)
controller = RestartController([manager])
prompt = [7, 3, 9, 1]
n_new = 6
result = runner.run_with_failover(
_generation("alpha", prompt, n_new, epoch=0), controller, max_restarts=2
)
assert result.completed
assert result.restarts == 1
# The restart began on a fresh epoch and reproduced the full stateless stream
# from token zero — no half-computed KV was imported.
assert result.outcome.route_epoch == 1
assert list(result.outcome.tokens) == model.stateless_greedy(prompt, n_new)
# The failed epoch's KV is gone and the epoch is now stale.
with pytest.raises(StaleRouteEpochError):
manager.resolve("alpha", 0)
# First attempt was unverified, the restart completed: only the restart bills.
statuses = [a.status for a in result.attempts]
assert statuses == [WorkStatus.UNVERIFIED, WorkStatus.COMPLETED]
assert runner.work_ledger.billable_tokens() == n_new
def test_failover_refuses_to_import_unverified_kv():
"assert_fresh_start fails closed if any shard still holds new-epoch KV.\n\nTags: node, failure, failover"
model = _KvDenseLlama()
adapter = _make_adapter(model)
manager = adapter.manager
controller = RestartController([manager])
new_epoch = controller.failover("s", 0)
assert new_epoch == 1
# A clean fresh start passes.
controller.assert_fresh_start("s", new_epoch)
# If unverified KV were present under the new epoch, the guard refuses it.
manager.open("s", new_epoch)
manager.append(
"s",
new_epoch,
{i: (np.zeros((1, model.n_heads, model.head_dim), dtype=np.float32),
np.zeros((1, model.n_heads, model.head_dim), dtype=np.float32))
for i in range(model.n_layers)},
)
with pytest.raises(Exception):
controller.assert_fresh_start("s", new_epoch)
def test_non_restartable_failure_is_not_retried():
"A deterministic failure (deadline) returns immediately without a restart.\n\nTags: node, failure, failover"
clock = _FakeClock()
adapter = _make_adapter()
runner = HardenedSessionRunner(adapter, clock=clock)
controller = RestartController([adapter.manager])
def before_step(_step):
clock.advance(1.0)
result = runner.run_with_failover(
_generation("bounded", [1, 2, 3], 20),
controller,
max_restarts=3,
deadline=2.0,
before_step=before_step,
)
assert not result.completed
assert result.restarts == 0
assert result.outcome.failure_kind is FailureKind.DEADLINE_EXCEEDED
# --------------------------------------------------------------------------- #
# Billing / work records distinguish completed, cancelled, failed, unverified.
# --------------------------------------------------------------------------- #
def test_work_ledger_distinguishes_all_four_statuses():
"The work ledger keeps completed/cancelled/failed/unverified distinct.\n\nTags: node, failure, billing"
ledger = WorkLedger()
ledger.record(WorkRecord("a", 0, WorkStatus.COMPLETED, tokens=8))
ledger.record(WorkRecord("b", 0, WorkStatus.CANCELLED, tokens=3,
failure_kind=FailureKind.CANCELLED))
ledger.record(WorkRecord("c", 0, WorkStatus.FAILED, tokens=1,
failure_kind=FailureKind.DEADLINE_EXCEEDED))
ledger.record(WorkRecord("d", 0, WorkStatus.UNVERIFIED, tokens=2,
failure_kind=FailureKind.WORKER_DEATH))
counts = ledger.counts_by_status()
assert counts == {
"completed": 1, "cancelled": 1, "failed": 1, "unverified": 1,
}
# Only completed work is billable — cancelled/failed/unverified tokens are
# recorded for observability but never charged.
assert ledger.billable_tokens() == 8
assert [r.session_id for r in ledger.billable_records()] == ["a"]
# JSON-safe for durable evidence.
payload = ledger.to_dict()
assert payload["billable_tokens"] == 8
assert payload["counts_by_status"]["unverified"] == 1
json.dumps(payload)
def test_work_status_and_classification_mapping():
"Failure kinds map to the right billing status and exception classes.\n\nTags: node, failure, billing"
assert work_status_for(FailureKind.CANCELLED) is WorkStatus.CANCELLED
assert work_status_for(FailureKind.WORKER_DEATH) is WorkStatus.UNVERIFIED
# A stream reset detected at a step boundary is a certain failure (nothing
# committed for that step) — only an unexpected mid-step error is unverified.
assert work_status_for(FailureKind.STREAM_RESET) is WorkStatus.FAILED
assert work_status_for(FailureKind.DEADLINE_EXCEEDED) is WorkStatus.FAILED
assert work_status_for(FailureKind.MALFORMED_BUNDLE) is WorkStatus.FAILED
assert work_status_for(FailureKind.STALE_EPOCH) is WorkStatus.FAILED
assert work_status_for(FailureKind.CACHE_MISS) is WorkStatus.FAILED
assert classify_exception(OperationCancelled()) is FailureKind.CANCELLED
assert classify_exception(StaleRouteEpochError("x")) is FailureKind.STALE_EPOCH
assert classify_exception(BoundaryContractError("x")) is FailureKind.MALFORMED_BUNDLE
assert classify_exception(RuntimeError("boom")) is FailureKind.WORKER_DEATH
assert (
classify_exception(StreamTerminated(FailureKind.HEARTBEAT_LOST))
is FailureKind.HEARTBEAT_LOST
)

186
tests/test_gguf_backend.py
View File

@@ -1,186 +0,0 @@
"""Tests for the GGUF backend adapter and recipe-gated startup seam."""
from __future__ import annotations
from types import SimpleNamespace
from meshnet_node.gguf_backend import GgufNodeBackend, build_gguf_backend
from meshnet_node.model_backend import TailTokenResult, TensorPayload
from meshnet_node.recipe_manifest import DEFAULT_RECIPE_ID, load_recipe_manifest
from meshnet_node.startup import _gguf_backend_for_recipe
class _RecordingTransport:
def __init__(self) -> None:
self.calls: list[tuple[str, tuple, dict]] = []
def encode_prompt(self, prompt: str, session_id: str | None = None):
self.calls.append(("encode_prompt", (prompt, session_id), {}))
return TensorPayload(
body=b"\x00" * 16,
shape=[1, 2, 4],
attention_mask_header=None,
position_ids_header=None,
)
def encode_next_token(self, token_id: int, session_id: str):
self.calls.append(("encode_next_token", (token_id, session_id), {}))
return TensorPayload(
body=b"\x00" * 8,
shape=[1, 1, 4],
attention_mask_header=None,
position_ids_header=None,
past_len=2,
)
def forward_bytes(
self,
body: bytes,
shape: list[int],
attention_mask_header: str | None,
position_ids_header: str | None,
*,
start_layer: int | None = None,
session_id: str | None = None,
cache_mode: str | None = None,
past_len: int | None = None,
):
self.calls.append(
(
"forward_bytes",
(body, tuple(shape), attention_mask_header, position_ids_header),
{
"start_layer": start_layer,
"session_id": session_id,
"cache_mode": cache_mode,
"past_len": past_len,
},
)
)
if cache_mode == "decode":
return TailTokenResult(text=" done", token_id=17)
return TensorPayload(
body=b"\x00" * 16,
shape=[1, 2, 4],
attention_mask_header=attention_mask_header,
position_ids_header=position_ids_header,
past_len=past_len,
)
def decode_tail_token(self, hidden_states):
self.calls.append(("decode_tail_token", (hidden_states.shape,), {}))
return TailTokenResult(text=" tail", token_id=19)
def generate_text(self, messages, max_new_tokens=5120, temperature=1.0, top_p=1.0):
self.calls.append(("generate_text", (tuple(messages), max_new_tokens, temperature, top_p), {}))
return "ok"
def generate_text_streaming(self, messages, max_new_tokens=5120, temperature=1.0, top_p=1.0):
self.calls.append(("generate_text_streaming", (tuple(messages), max_new_tokens, temperature, top_p), {}))
yield "ok"
def count_prompt_tokens(self, messages):
self.calls.append(("count_prompt_tokens", (tuple(messages),), {}))
return 3
def count_text_tokens(self, text):
self.calls.append(("count_text_tokens", (text,), {}))
return 2
def eos_token_ids(self):
self.calls.append(("eos_token_ids", (), {}))
return [19]
def release_session(self, session_id: str) -> None:
self.calls.append(("release_session", (session_id,), {}))
def test_build_gguf_backend_delegates_to_transport():
transport = _RecordingTransport()
backend = build_gguf_backend(
model_id="meshnet/native-model",
shard_start=0,
shard_end=1,
quantization="bfloat16",
transport=transport,
total_layers=2,
device_type="cpu",
)
assert isinstance(backend, GgufNodeBackend)
assert backend.backend_id == "llama.cpp"
assert backend.is_head is True
assert backend.is_tail is True
assert backend.model.config.to_dict()["architecture_adapter"] == "dense-llama"
assert backend.loaded_tensor_names[0] == "blk.0.weight"
prompt = backend.encode_prompt("hello", session_id="session-1")
assert prompt.shape == [1, 2, 4]
decode = backend.forward_bytes(
b"\x00" * 16,
[1, 2, 4],
None,
None,
session_id="session-1",
cache_mode="decode",
past_len=2,
)
assert isinstance(decode, TailTokenResult)
assert decode.token_id == 17
backend.release_session("session-1")
assert [call[0] for call in transport.calls] == [
"encode_prompt",
"forward_bytes",
"release_session",
]
assert transport.calls[0][1] == ("hello", "session-1")
assert transport.calls[1][2]["cache_mode"] == "decode"
assert transport.calls[1][2]["past_len"] == 2
def test_recipe_gates_native_backend_selection(monkeypatch):
manifest = load_recipe_manifest()
torch_recipe = manifest.require(DEFAULT_RECIPE_ID)
native_recipe = manifest.require("llama-cpp-native")
sentinel_backend = object()
calls: list[dict] = []
def fake_build_gguf_backend(**kwargs):
calls.append(kwargs)
return sentinel_backend
monkeypatch.setattr(
"meshnet_node.startup.build_gguf_backend",
fake_build_gguf_backend,
)
assert _gguf_backend_for_recipe(
torch_recipe,
model_id="meshnet/native-model",
shard_start=0,
shard_end=1,
quantization="bfloat16",
total_layers=2,
device="cpu",
) is None
backend = _gguf_backend_for_recipe(
native_recipe,
model_id="meshnet/native-model",
shard_start=0,
shard_end=1,
quantization="bfloat16",
total_layers=2,
device="cpu",
)
assert backend is sentinel_backend
assert calls[0]["model_id"] == "meshnet/native-model"
assert calls[0]["shard_start"] == 0
assert calls[0]["shard_end"] == 1
assert calls[0]["quantization"] == "bfloat16"
assert calls[0]["total_layers"] == 2

88
tests/test_gguf_ownership.py
View File

@@ -1,88 +0,0 @@
"""Dense-Llama GGUF ownership selection and introspection tests."""
from __future__ import annotations
import pytest
from meshnet_node.gguf_ownership import (
DenseLlamaShardOwnership,
authoritative_dense_llama_ownership,
infer_dense_llama_ownership,
select_dense_llama_tensor_names,
)
def test_dense_llama_selection_only_picks_block_range_and_endpoints():
"Dense-Llama selection keeps only the owned blocks plus the correct endpoints.\n\nTags: node, GGUF"
tensor_inventory = {
"token_embd.weight": 10_000,
"blk.0.attn_q.weight": 1_000,
"blk.0.ffn_down.weight": 1_000,
"blk.1.attn_q.weight": 2_000,
"blk.1.ffn_down.weight": 2_000,
"blk.2.attn_q.weight": 3_000,
"blk.2.ffn_down.weight": 3_000,
"output_norm.weight": 256,
"output.weight": 10_000,
"rope.freqs": 128,
}
selected = select_dense_llama_tensor_names(
tensor_inventory,
1,
2,
total_layers=3,
)
assert selected == {
"blk.1.attn_q.weight",
"blk.1.ffn_down.weight",
"blk.2.attn_q.weight",
"blk.2.ffn_down.weight",
"output_norm.weight",
"output.weight",
}
selected_bytes = sum(tensor_inventory[name] for name in selected)
full_bytes = sum(tensor_inventory.values())
assert selected_bytes == 20_256
assert selected_bytes < full_bytes
def test_dense_llama_loaded_range_is_authoritative_from_tensor_inventory():
"The backend's loaded tensor inventory is the source of truth for range and ownership.\n\nTags: node, GGUF"
class Backend:
loaded_tensor_names = (
"token_embd.weight",
"blk.4.attn_q.weight",
"blk.5.ffn_down.weight",
"output_norm.weight",
"output.weight",
)
ownership = authoritative_dense_llama_ownership(Backend(), selection=None)
assert isinstance(ownership, DenseLlamaShardOwnership)
assert ownership.range == (4, 5)
assert ownership.owns_embedding is True
assert ownership.owns_final_head is True
def test_derivative_slice_requires_source_and_slice_hashes():
"Temporary derivative GGUF slices must carry hashes and cannot claim final semantics.\n\nTags: node, GGUF"
with pytest.raises(ValueError, match="source and slice hashes"):
infer_dense_llama_ownership(
["blk.1.attn_q.weight"],
derivative_slice=True,
final_artifact_semantics=False,
)
with pytest.raises(ValueError, match="final artifacts"):
infer_dense_llama_ownership(
["blk.1.attn_q.weight"],
source_artifact_hash="sha256:source",
slice_artifact_hash="sha256:slice",
derivative_slice=True,
final_artifact_semantics=True,
)

769
tests/test_hot_kv_state.py
View File

@@ -1,769 +0,0 @@
"""Isolated concurrent local Hot KV State (DGR-007).
These tests prove the KV/session manager with a *pure-numpy* KV-cached dense-Llama
reference: no download, no GPU, no torch, no API credit. The reference implements
the DGR-006 ``ShardComputation`` duck type plus ``run_layers_cached`` so cached
prefill/decode over a per-session KV context reproduces the stateless whole-model
tokens bit-for-bit. On top of that correctness core, the tests exercise the
manager's lifecycle: owned-layer allocation, prefill/decode append, truncate,
release, TTL/LRU eviction, explicit cache-miss responses, stale-epoch and
incompatible-recipe rejection, four concurrent cross-talk-free sessions, and
budget-bounded cancellation.
"""
from __future__ import annotations
import threading
import numpy as np
import pytest
from meshnet_node.boundary_adapter import BoundaryBundle, TailOutput
from meshnet_node.hot_kv_state import (
CacheMiss,
CacheMissReason,
HotKvStateConfig,
HotKvStateManager,
IncompatibleCacheRecipeError,
KvBoundaryAdapter,
KvBudgetExceededError,
KvCacheMissError,
KvCacheRecipe,
LayerKvCache,
StaleRouteEpochError,
kv_recipe_for,
)
PARITY_ATOL = 1e-6
# --------------------------------------------------------------------------- #
# Pure-numpy KV-cached dense-Llama reference (test fixture, not production).
# --------------------------------------------------------------------------- #
class _KvDenseLlama:
"""A tiny deterministic dense-Llama with both stateless and cached runners."""
architecture_adapter = "dense-llama"
def __init__(
self,
*,
vocab: int = 48,
hidden: int = 32,
n_layers: int = 6,
n_heads: int = 4,
intermediate: int = 64,
rms_eps: float = 1e-6,
rope_theta: float = 10000.0,
seed: int = 20260716,
) -> None:
assert hidden % n_heads == 0
self.vocab = vocab
self.hidden = hidden
self.n_layers = n_layers
self.n_heads = n_heads
self.head_dim = hidden // n_heads
assert self.head_dim % 2 == 0
self.rms_eps = rms_eps
self.rope_theta = rope_theta
rng = np.random.default_rng(seed)
def w(*shape: int) -> np.ndarray:
return (rng.standard_normal(shape) * 0.08).astype(np.float32)
self.embed = w(vocab, hidden)
self.layers = []
for _ in range(n_layers):
self.layers.append(
{
"in_ln": (1.0 + rng.standard_normal(hidden) * 0.02).astype(np.float32),
"q": w(hidden, hidden),
"k": w(hidden, hidden),
"v": w(hidden, hidden),
"o": w(hidden, hidden),
"post_ln": (1.0 + rng.standard_normal(hidden) * 0.02).astype(np.float32),
"gate": w(intermediate, hidden),
"up": w(intermediate, hidden),
"down": w(hidden, intermediate),
}
)
self.final_ln = (1.0 + rng.standard_normal(hidden) * 0.02).astype(np.float32)
self.lm_head_w = w(vocab, hidden)
inv_freq = 1.0 / (
rope_theta ** (np.arange(0, self.head_dim, 2, dtype=np.float32) / self.head_dim)
)
self.inv_freq = inv_freq.astype(np.float32)
# -- primitive ops -----------------------------------------------------
def _rmsnorm(self, x: np.ndarray, weight: np.ndarray) -> np.ndarray:
variance = np.mean(x.astype(np.float32) ** 2, axis=-1, keepdims=True)
normed = x / np.sqrt(variance + self.rms_eps)
return (normed * weight).astype(np.float32)
def _rope(self, positions: np.ndarray):
angles = positions[..., None].astype(np.float32) * self.inv_freq[None, None, :]
emb = np.concatenate([angles, angles], axis=-1)
return np.cos(emb).astype(np.float32), np.sin(emb).astype(np.float32)
@staticmethod
def _rotate_half(x: np.ndarray) -> np.ndarray:
half = x.shape[-1] // 2
return np.concatenate([-x[..., half:], x[..., :half]], axis=-1)
def _apply_rope(self, t: np.ndarray, cos: np.ndarray, sin: np.ndarray) -> np.ndarray:
cos = cos[:, None, :, :]
sin = sin[:, None, :, :]
return t * cos + self._rotate_half(t) * sin
def _project_qkv(self, normed: np.ndarray, layer: dict, positions: np.ndarray):
batch, seq, _ = normed.shape
q = (normed @ layer["q"].T).reshape(batch, seq, self.n_heads, self.head_dim)
k = (normed @ layer["k"].T).reshape(batch, seq, self.n_heads, self.head_dim)
v = (normed @ layer["v"].T).reshape(batch, seq, self.n_heads, self.head_dim)
q = q.transpose(0, 2, 1, 3)
k = k.transpose(0, 2, 1, 3)
v = v.transpose(0, 2, 1, 3)
cos, sin = self._rope(positions)
q = self._apply_rope(q, cos, sin)
k = self._apply_rope(k, cos, sin)
return q, k, v
def _attend(
self,
q: np.ndarray,
k_all: np.ndarray,
v_all: np.ndarray,
layer: dict,
q_positions: np.ndarray,
) -> np.ndarray:
batch, _, seq_new, _ = q.shape
total = k_all.shape[2]
scores = (q @ k_all.transpose(0, 1, 3, 2)) / np.sqrt(self.head_dim)
# Causal mask by absolute position: keys are stored in absolute order
# 0..total-1; query row i lives at absolute position q_positions[i].
key_abs = np.arange(total, dtype=np.int64)
q_abs = np.asarray(q_positions).reshape(seq_new).astype(np.int64)
mask = np.where(key_abs[None, :] <= q_abs[:, None], 0.0, -1e30).astype(np.float32)
scores = scores + mask[None, None, :, :]
scores = scores - scores.max(axis=-1, keepdims=True)
weights = np.exp(scores)
weights = weights / weights.sum(axis=-1, keepdims=True)
out = weights @ v_all
out = out.transpose(0, 2, 1, 3).reshape(batch, seq_new, self.hidden)
return (out @ layer["o"].T).astype(np.float32)
def _mlp(self, x: np.ndarray, layer: dict) -> np.ndarray:
gate = x @ layer["gate"].T
up = x @ layer["up"].T
silu = gate * (1.0 / (1.0 + np.exp(-gate)))
return ((silu * up) @ layer["down"].T).astype(np.float32)
# -- stateless whole-sequence layer (ground truth) ---------------------
def _run_layer_stateless(self, x: np.ndarray, layer: dict, positions: np.ndarray) -> np.ndarray:
normed = self._rmsnorm(x, layer["in_ln"])
q, k, v = self._project_qkv(normed, layer, positions)
attn = self._attend(q, k, v, layer, positions[0])
h = x + attn
h = h + self._mlp(self._rmsnorm(h, layer["post_ln"]), layer)
return h.astype(np.float32)
def whole_model_next_token(self, token_ids: list[int]) -> int:
positions = np.arange(len(token_ids))[None, :]
h = self.embed[np.asarray(token_ids)][None, :]
for idx in range(self.n_layers):
h = self._run_layer_stateless(h, self.layers[idx], positions)
h = self._rmsnorm(h[:, -1:, :], self.final_ln)
logits = h @ self.lm_head_w.T
return int(np.argmax(logits[0, -1]))
def stateless_greedy(self, prompt: list[int], n_new: int) -> list[int]:
tokens = list(prompt)
out: list[int] = []
for _ in range(n_new):
tok = self.whole_model_next_token(tokens)
tokens.append(tok)
out.append(tok)
return out
class _KvReferenceShard:
"""A contiguous inclusive layer range with a KV-cached runner.
Satisfies the KV-aware ``ShardComputation`` duck type used by
``KvBoundaryAdapter``: DGR-006 methods plus ``run_layers_cached`` and the KV
geometry (``n_kv_heads`` / ``head_dim`` / ``kv_dtype``).
"""
kv_dtype = "float32"
def __init__(
self,
model: _KvDenseLlama,
start_layer: int,
end_layer: int,
*,
architecture_adapter: str | None = None,
) -> None:
self._model = model
self.start_layer = start_layer
self.end_layer = end_layer
self.total_layers = model.n_layers
self.n_kv_heads = model.n_heads
self.head_dim = model.head_dim
self.architecture_adapter = architecture_adapter or model.architecture_adapter
def embed_tokens(self, token_ids: np.ndarray) -> np.ndarray:
return self._model.embed[np.asarray(token_ids)]
def final_norm(self, hidden: np.ndarray) -> np.ndarray:
return self._model._rmsnorm(np.asarray(hidden, dtype=np.float32), self._model.final_ln)
def lm_head(self, hidden: np.ndarray) -> np.ndarray:
return np.asarray(hidden, dtype=np.float32) @ self._model.lm_head_w.T
def run_layers_cached(self, hidden, *, positions, past_kv):
m = self._model
x = np.asarray(hidden, dtype=np.float32)
positions = np.asarray(positions)
new_kv: dict[int, tuple[np.ndarray, np.ndarray]] = {}
for idx in range(self.start_layer, self.end_layer + 1):
layer = m.layers[idx]
normed = m._rmsnorm(x, layer["in_ln"])
q, k, v = m._project_qkv(normed, layer, positions)
# Post-RoPE new K/V stored as (seq_new, n_heads, head_dim).
new_k = k[0].transpose(1, 0, 2).copy()
new_v = v[0].transpose(1, 0, 2).copy()
cache = past_kv.get(idx)
if cache is not None and cache.length > 0:
past_k = cache.keys[None].transpose(0, 2, 1, 3)
past_v = cache.values[None].transpose(0, 2, 1, 3)
k_all = np.concatenate([past_k, k], axis=2)
v_all = np.concatenate([past_v, v], axis=2)
else:
k_all, v_all = k, v
attn = m._attend(q, k_all, v_all, layer, positions[0])
h = x + attn
x = h + m._mlp(m._rmsnorm(h, layer["post_ln"]), layer)
x = x.astype(np.float32)
new_kv[idx] = (new_k, new_v)
return x, new_kv
# --------------------------------------------------------------------------- #
# Helpers.
# --------------------------------------------------------------------------- #
class _FakeClock:
def __init__(self) -> None:
self.now = 0.0
def __call__(self) -> float:
return self.now
def advance(self, delta: float) -> None:
self.now += delta
def _full_shard(model: _KvDenseLlama):
return _KvReferenceShard(model, 0, model.n_layers - 1)
def _manager_for(shard, config: HotKvStateConfig | None = None, clock=None) -> HotKvStateManager:
return HotKvStateManager(kv_recipe_for(shard), config=config, clock=clock)
def _cached_greedy(
adapter: KvBoundaryAdapter,
manager: HotKvStateManager,
session_id: str,
epoch: int,
prompt: list[int],
n_new: int,
) -> list[int]:
"""Greedy decode one full-model session through the KV manager."""
out = adapter.prefill(session_id, epoch, token_ids=np.asarray(prompt))
assert isinstance(out, TailOutput)
tokens = [out.token_id]
for _ in range(n_new - 1):
step = adapter.decode(session_id, epoch, token_ids=[out.token_id])
assert isinstance(step, TailOutput)
out = step
tokens.append(out.token_id)
return tokens
# --------------------------------------------------------------------------- #
# Recipe identity.
# --------------------------------------------------------------------------- #
def test_recipe_owned_layers_and_fingerprint_aliasing():
"The KV recipe covers only owned layers and canonicalizes architecture aliases.\n\nTags: node, kv"
recipe = KvCacheRecipe(
architecture_adapter="LlamaForCausalLM",
kv_dtype="float32",
n_kv_heads=4,
head_dim=8,
total_layers=6,
start_layer=2,
end_layer=3,
)
assert recipe.owned_layers == (2, 3)
alias = KvCacheRecipe(
architecture_adapter="llama",
kv_dtype="float32",
n_kv_heads=4,
head_dim=8,
total_layers=6,
start_layer=2,
end_layer=3,
)
assert recipe.is_compatible(alias)
# A different owned range is not compatible.
other = KvCacheRecipe(
architecture_adapter="llama",
kv_dtype="float32",
n_kv_heads=4,
head_dim=8,
total_layers=6,
start_layer=0,
end_layer=1,
)
assert not recipe.is_compatible(other)
def test_recipe_bytes_per_token_scales_with_owned_layers():
"KV bytes-per-token counts keys+values across owned layers only.\n\nTags: node, kv"
base = dict(
architecture_adapter="dense-llama",
kv_dtype="float32",
n_kv_heads=4,
head_dim=8,
total_layers=6,
)
one = KvCacheRecipe(**base, start_layer=0, end_layer=0)
two = KvCacheRecipe(**base, start_layer=0, end_layer=1)
# 2 (k+v) * heads * dim * 4 bytes per layer.
assert one.bytes_per_token() == 2 * 4 * 8 * 4
assert two.bytes_per_token() == 2 * one.bytes_per_token()
# --------------------------------------------------------------------------- #
# Owned-layer allocation.
# --------------------------------------------------------------------------- #
def test_manager_allocates_kv_only_for_owned_layers():
"A middle shard allocates KV state only for its owned layer range.\n\nTags: node, kv"
model = _KvDenseLlama()
shard = _KvReferenceShard(model, 2, 3)
manager = _manager_for(shard)
session = manager.open("sess-mid", 0)
assert session.owned_layers == (2, 3)
assert set(session.layers) == {2, 3}
with pytest.raises(KeyError):
session.layer(0)
# --------------------------------------------------------------------------- #
# Prefill / decode / truncate.
# --------------------------------------------------------------------------- #
def test_prefill_then_decode_append_grows_owned_layers():
"Prefill and decode append advance every owned layer in lockstep.\n\nTags: node, kv"
model = _KvDenseLlama()
shard = _full_shard(model)
manager = _manager_for(shard)
adapter = KvBoundaryAdapter(shard, manager)
prompt = [5, 12, 3, 41]
out = adapter.prefill("s", 0, token_ids=np.asarray(prompt))
assert isinstance(out, TailOutput)
session = manager.get("s", 0)
assert session.seq_len == len(prompt)
for cache in session.layers.values():
assert cache.length == len(prompt)
step = adapter.decode("s", 0, token_ids=[out.token_id])
assert isinstance(step, TailOutput)
assert manager.get("s", 0).seq_len == len(prompt) + 1
def test_truncate_rolls_back_all_owned_layers():
"Truncate drops cached positions beyond a length across owned layers.\n\nTags: node, kv"
model = _KvDenseLlama()
shard = _full_shard(model)
manager = _manager_for(shard)
adapter = KvBoundaryAdapter(shard, manager)
adapter.prefill("s", 0, token_ids=np.asarray([1, 2, 3, 4, 5]))
assert manager.get("s", 0).seq_len == 5
manager.truncate("s", 0, 2)
session = manager.get("s", 0)
assert session.seq_len == 2
for cache in session.layers.values():
assert cache.length == 2
def test_layer_kv_cache_rejects_wrong_shape():
"LayerKvCache rejects K/V that do not match its head geometry.\n\nTags: node, kv"
cache = LayerKvCache(0, n_kv_heads=4, head_dim=8, dtype="float32")
with pytest.raises(ValueError):
cache.append(np.zeros((1, 3, 8), dtype=np.float32), np.zeros((1, 3, 8), dtype=np.float32))
cache.append(np.zeros((2, 4, 8), dtype=np.float32), np.zeros((2, 4, 8), dtype=np.float32))
assert cache.length == 2
# --------------------------------------------------------------------------- #
# Cached vs stateless parity (correctness core).
# --------------------------------------------------------------------------- #
def test_cached_full_shard_decode_matches_stateless_whole_model():
"Cached full-model greedy decode reproduces stateless whole-model tokens.\n\nTags: node, kv, parity"
model = _KvDenseLlama()
shard = _full_shard(model)
manager = _manager_for(shard)
adapter = KvBoundaryAdapter(shard, manager)
prompt = [2, 17, 8, 25, 6]
n_new = 12
reference = model.stateless_greedy(prompt, n_new)
cached = _cached_greedy(adapter, manager, "s", 0, prompt, n_new)
assert cached == reference
assert len(cached) == n_new
def test_cached_prefill_next_token_matches_whole_model_logits():
"Cached prefill produces the same next-token logits as the whole model.\n\nTags: node, kv, parity"
model = _KvDenseLlama()
shard = _full_shard(model)
manager = _manager_for(shard)
adapter = KvBoundaryAdapter(shard, manager)
prompt = [9, 1, 44, 6, 30, 11]
out = adapter.prefill("s", 0, token_ids=np.asarray(prompt))
assert isinstance(out, TailOutput)
assert out.token_id == model.whole_model_next_token(prompt)
def test_multi_range_cached_decode_parity_across_a_seam():
"A head/tail split with independent per-range KV reproduces whole-model decode.\n\nTags: node, kv, parity"
model = _KvDenseLlama()
head_shard = _KvReferenceShard(model, 0, 2)
tail_shard = _KvReferenceShard(model, 3, 5)
head_mgr = _manager_for(head_shard)
tail_mgr = _manager_for(tail_shard)
head = KvBoundaryAdapter(head_shard, head_mgr)
tail = KvBoundaryAdapter(tail_shard, tail_mgr)
prompt = [7, 3, 22, 5, 9]
n_new = 8
# Each range only allocates its owned layers.
def step(token_ids, is_prefill):
if is_prefill:
bundle = head.prefill("s", 0, token_ids=np.asarray(token_ids))
out = tail.prefill("s", 0, boundary=bundle)
else:
bundle = head.decode("s", 0, token_ids=[token_ids])
assert isinstance(bundle, BoundaryBundle)
out = tail.decode("s", 0, boundary=bundle)
assert isinstance(out, TailOutput)
return out.token_id
tokens = [step(prompt, True)]
for _ in range(n_new - 1):
tokens.append(step(tokens[-1], False))
assert head_mgr.get("s", 0).owned_layers == (0, 1, 2)
assert tail_mgr.get("s", 0).owned_layers == (3, 4, 5)
assert tokens == model.stateless_greedy(prompt, n_new)
# --------------------------------------------------------------------------- #
# Four concurrent sessions with no cross-talk.
# --------------------------------------------------------------------------- #
def test_four_interleaved_sessions_have_no_kv_cross_talk():
"Four interleaved sessions each decode their own tokens without cross-talk.\n\nTags: node, kv, concurrency"
model = _KvDenseLlama()
shard = _full_shard(model)
manager = _manager_for(shard)
adapter = KvBoundaryAdapter(shard, manager)
prompts = {
"alpha": [1, 2, 3, 4],
"bravo": [40, 39, 2, 15],
"charlie": [7, 7, 7, 7],
"delta": [31, 5, 18, 22],
}
n_new = 10
references = {sid: model.stateless_greedy(p, n_new) for sid, p in prompts.items()}
# The four prompts must actually diverge, else "no cross-talk" is vacuous.
assert len({tuple(v) for v in references.values()}) == 4
generated: dict[str, list[int]] = {}
for sid, prompt in prompts.items():
out = adapter.prefill(sid, 0, token_ids=np.asarray(prompt))
assert isinstance(out, TailOutput)
generated[sid] = [out.token_id]
# Round-robin decode: every session takes one step per round, interleaved.
for _ in range(n_new - 1):
for sid in prompts:
step = adapter.decode(sid, 0, token_ids=[generated[sid][-1]])
assert isinstance(step, TailOutput)
generated[sid].append(step.token_id)
for sid in prompts:
assert generated[sid] == references[sid], sid
assert manager.session_count == 4
def test_four_sessions_on_real_threads_stay_isolated():
"Four sessions decoding on real threads produce their own reference tokens.\n\nTags: node, kv, concurrency"
model = _KvDenseLlama()
shard = _full_shard(model)
manager = _manager_for(shard, HotKvStateConfig(max_sessions=8))
adapter = KvBoundaryAdapter(shard, manager)
prompts = {
"t-alpha": [3, 14, 1, 5],
"t-bravo": [2, 27, 18, 4],
"t-charlie": [9, 9, 1, 2],
"t-delta": [44, 6, 30, 11],
}
n_new = 8
references = {sid: model.stateless_greedy(p, n_new) for sid, p in prompts.items()}
results: dict[str, list[int]] = {}
errors: list[Exception] = []
def run(sid: str, prompt: list[int]) -> None:
try:
results[sid] = _cached_greedy(adapter, manager, sid, 0, prompt, n_new)
except Exception as exc: # pragma: no cover - surfaced via assert below
errors.append(exc)
threads = [threading.Thread(target=run, args=(sid, p)) for sid, p in prompts.items()]
for t in threads:
t.start()
for t in threads:
t.join()
assert not errors
for sid in prompts:
assert results[sid] == references[sid], sid
def test_release_one_session_leaves_others_intact_and_returns_memory():
"Releasing one session frees its budget and does not disturb the others.\n\nTags: node, kv, concurrency"
model = _KvDenseLlama()
shard = _full_shard(model)
manager = _manager_for(shard)
adapter = KvBoundaryAdapter(shard, manager)
prompts = {"keep-1": [1, 2, 3], "drop": [10, 11, 12, 13], "keep-2": [5, 6, 7]}
n_new = 6
references = {sid: model.stateless_greedy(p, n_new) for sid, p in prompts.items()}
gen: dict[str, list[int]] = {}
for sid, prompt in prompts.items():
out = adapter.prefill(sid, 0, token_ids=np.asarray(prompt))
gen[sid] = [out.token_id]
bytes_before = manager.total_bytes
assert manager.release("drop", 0) is True
assert manager.total_bytes < bytes_before
# A decode on the released session is an explicit cache miss, not corruption.
miss = adapter.decode("drop", 0, token_ids=[gen["drop"][-1]])
assert isinstance(miss, CacheMiss)
assert miss.reason is CacheMissReason.RELEASED
# The survivors keep decoding to their own references.
for _ in range(n_new - 1):
for sid in ("keep-1", "keep-2"):
step = adapter.decode(sid, 0, token_ids=[gen[sid][-1]])
assert isinstance(step, TailOutput)
gen[sid].append(step.token_id)
for sid in ("keep-1", "keep-2"):
assert gen[sid] == references[sid], sid
# --------------------------------------------------------------------------- #
# Stale epoch / incompatible recipe rejection.
# --------------------------------------------------------------------------- #
def test_stale_route_epoch_is_rejected():
"A request for an older route epoch than the current one is rejected.\n\nTags: node, kv"
model = _KvDenseLlama()
manager = _manager_for(_full_shard(model))
manager.open("s", 5)
with pytest.raises(StaleRouteEpochError):
manager.open("s", 4)
with pytest.raises(StaleRouteEpochError):
manager.resolve("s", 4)
with pytest.raises(StaleRouteEpochError):
manager.append("s", 4, {})
def test_new_route_epoch_supersedes_and_frees_old_epoch():
"A newer route epoch supersedes the old one, freeing its KV and reporting a miss.\n\nTags: node, kv"
model = _KvDenseLlama()
shard = _full_shard(model)
manager = _manager_for(shard)
adapter = KvBoundaryAdapter(shard, manager)
adapter.prefill("s", 1, token_ids=np.asarray([1, 2, 3, 4]))
bytes_epoch1 = manager.total_bytes
assert bytes_epoch1 > 0
# Re-planned route: epoch 2 starts a fresh isolated context.
adapter.prefill("s", 2, token_ids=np.asarray([9, 8]))
assert manager.session_keys() == [("s", 2)]
# Old epoch is gone; a lookup for it is now stale (epoch < current).
with pytest.raises(StaleRouteEpochError):
manager.resolve("s", 1)
def test_incompatible_cache_recipe_is_rejected():
"A request carrying a different KV recipe is rejected, not silently reused.\n\nTags: node, kv"
model = _KvDenseLlama()
shard = _full_shard(model)
manager = _manager_for(shard)
manager.open("s", 0)
incompatible = KvCacheRecipe(
architecture_adapter="dense-llama",
kv_dtype="float16", # different KV dtype
n_kv_heads=model.n_heads,
head_dim=model.head_dim,
total_layers=model.n_layers,
start_layer=0,
end_layer=model.n_layers - 1,
)
with pytest.raises(IncompatibleCacheRecipeError):
manager.resolve("s", 0, recipe=incompatible)
with pytest.raises(IncompatibleCacheRecipeError):
manager.open("s2", 0, recipe=incompatible)
def test_uncertified_architecture_recipe_fails_closed():
"A KV recipe for an uncertified architecture fails closed at construction.\n\nTags: node, kv"
from meshnet_node.boundary_adapter import UncertifiedArchitectureError
with pytest.raises(UncertifiedArchitectureError):
KvCacheRecipe(
architecture_adapter="qwen3-moe",
kv_dtype="float32",
n_kv_heads=4,
head_dim=8,
total_layers=6,
start_layer=0,
end_layer=5,
)
# --------------------------------------------------------------------------- #
# Explicit cache-miss responses.
# --------------------------------------------------------------------------- #
def test_unknown_session_is_an_explicit_cache_miss():
"Resolving an unknown session returns an explicit unknown-session miss.\n\nTags: node, kv"
manager = _manager_for(_full_shard(_KvDenseLlama()))
miss = manager.resolve("nope", 0)
assert isinstance(miss, CacheMiss)
assert miss.reason is CacheMissReason.UNKNOWN_SESSION
with pytest.raises(KvCacheMissError):
manager.get("nope", 0)
def test_seq_len_mismatch_is_an_explicit_cache_miss():
"A decode whose expected length disagrees with the cache is an explicit miss.\n\nTags: node, kv"
model = _KvDenseLlama()
shard = _full_shard(model)
manager = _manager_for(shard)
adapter = KvBoundaryAdapter(shard, manager)
out = adapter.prefill("s", 0, token_ids=np.asarray([1, 2, 3]))
# Cache holds 3 tokens; claim it holds 99.
miss = adapter.decode("s", 0, token_ids=[out.token_id], expected_seq_len=99)
assert isinstance(miss, CacheMiss)
assert miss.reason is CacheMissReason.SEQ_LEN_MISMATCH
def test_ttl_eviction_yields_an_explicit_cache_miss():
"A session idle past its TTL is evicted and reported as a TTL cache miss.\n\nTags: node, kv"
model = _KvDenseLlama()
shard = _full_shard(model)
clock = _FakeClock()
manager = _manager_for(shard, HotKvStateConfig(ttl_seconds=10.0), clock=clock)
adapter = KvBoundaryAdapter(shard, manager)
adapter.prefill("s", 0, token_ids=np.asarray([1, 2, 3]))
clock.advance(11.0)
miss = manager.resolve("s", 0)
assert isinstance(miss, CacheMiss)
assert miss.reason is CacheMissReason.EVICTED_TTL
assert manager.total_bytes == 0
# --------------------------------------------------------------------------- #
# Eviction and budget.
# --------------------------------------------------------------------------- #
def test_lru_eviction_by_session_cap_reports_a_miss():
"Exceeding the session cap evicts the least-recently-used session.\n\nTags: node, kv"
model = _KvDenseLlama()
shard = _full_shard(model)
manager = _manager_for(shard, HotKvStateConfig(max_sessions=2))
adapter = KvBoundaryAdapter(shard, manager)
adapter.prefill("a", 0, token_ids=np.asarray([1, 2]))
adapter.prefill("b", 0, token_ids=np.asarray([3, 4]))
# Touch 'a' so 'b' becomes the LRU victim.
adapter.decode("a", 0, token_ids=[1])
adapter.prefill("c", 0, token_ids=np.asarray([5, 6]))
miss = manager.resolve("b", 0)
assert isinstance(miss, CacheMiss)
assert miss.reason is CacheMissReason.EVICTED_LRU
assert set(k[0] for k in manager.session_keys()) == {"a", "c"}
def test_budget_eviction_keeps_total_within_budget():
"Byte-budget pressure evicts LRU sessions so the store stays within budget.\n\nTags: node, kv"
model = _KvDenseLlama()
shard = _full_shard(model)
recipe = kv_recipe_for(shard)
# Budget for ~5 tokens of one session; a second big session forces eviction.
budget = recipe.bytes_per_token() * 5
manager = _manager_for(shard, HotKvStateConfig(budget_bytes=budget, max_sessions=8))
adapter = KvBoundaryAdapter(shard, manager)
adapter.prefill("a", 0, token_ids=np.asarray([1, 2, 3]))
adapter.prefill("b", 0, token_ids=np.asarray([4, 5, 6, 7]))
assert manager.total_bytes <= budget
# 'a' (older, LRU) was evicted to make room for 'b'.
miss = manager.resolve("a", 0)
assert isinstance(miss, CacheMiss)
assert miss.reason is CacheMissReason.EVICTED_LRU
assert manager.get("b", 0).seq_len == 4
def test_single_session_exceeding_budget_raises():
"A single session that cannot fit the budget raises instead of evicting itself.\n\nTags: node, kv"
model = _KvDenseLlama()
shard = _full_shard(model)
recipe = kv_recipe_for(shard)
budget = recipe.bytes_per_token() * 2 # only 2 tokens fit
manager = _manager_for(shard, HotKvStateConfig(budget_bytes=budget))
adapter = KvBoundaryAdapter(shard, manager)
with pytest.raises(KvBudgetExceededError):
adapter.prefill("a", 0, token_ids=np.asarray([1, 2, 3, 4, 5]))

78
tests/test_llama_worker_build.py
View File

@@ -1,78 +0,0 @@
from __future__ import annotations
import os
import subprocess
from pathlib import Path
import pytest
ROOT = Path(__file__).resolve().parents[1]
SCRIPT = ROOT / "packages" / "node" / "native" / "scripts" / "build_llama_worker.sh"
PIN_FILE = ROOT / "packages" / "node" / "native" / "llama" / "UPSTREAM_COMMIT"
@pytest.mark.skipif(not SCRIPT.exists(), reason="llama worker build script is missing")
def test_llama_worker_build_smoke_rebuild(tmp_path: Path) -> None:
if not shutil_which("git"):
pytest.skip("git is unavailable")
if not (shutil_which("g++") or shutil_which("c++") or shutil_which("clang++")):
pytest.skip("no C++ compiler is unavailable")
source_dir = tmp_path / "llama.cpp"
build_one = tmp_path / "build-1"
build_two = tmp_path / "build-2"
pin = PIN_FILE.read_text(encoding="utf-8").strip()
source_dir.mkdir()
_write_fake_upstream_tree(source_dir, pin)
_git_init(source_dir)
_run_build(source_dir, build_one)
_run_build(source_dir, build_two)
binary = build_two / "meshnet_worker"
assert binary.exists()
output = subprocess.run(
[str(binary), "--smoke"],
cwd=ROOT,
check=True,
capture_output=True,
text=True,
)
assert "meshnet worker scaffold ok" in output.stdout
assert pin in output.stdout
def _run_build(source_dir: Path, build_dir: Path) -> None:
env = os.environ.copy()
env.setdefault("PATH", os.environ.get("PATH", ""))
subprocess.run(
[str(SCRIPT), "--source-dir", str(source_dir), "--build-dir", str(build_dir)],
cwd=ROOT,
check=True,
env=env,
capture_output=True,
text=True,
)
def _write_fake_upstream_tree(source_dir: Path, pin: str) -> None:
(source_dir / "LICENSE").write_text("MIT License placeholder\n", encoding="utf-8")
(source_dir / "AUTHORS").write_text("Georgi Gerganov\nMeshnet maintainers\n", encoding="utf-8")
(source_dir / "CMakeLists.txt").write_text("# upstream placeholder\n", encoding="utf-8")
(source_dir / ".meshnet-upstream-commit").write_text(f"{pin}\n", encoding="utf-8")
(source_dir / ".meshnet-upstream-repository").write_text(
"https://github.com/ggml-org/llama.cpp.git\n",
encoding="utf-8",
)
def _git_init(source_dir: Path) -> None:
subprocess.run(["git", "init", "-q"], cwd=source_dir, check=True)
def shutil_which(name: str) -> str | None:
from shutil import which
return which(name)

508
tests/test_native_shard_protocol.py
View File

@@ -1,508 +0,0 @@
"""DGR-002: generated-schema round-trip and compatibility tests.
Covers the versioned gRPC Shard protocol (``packages/node/native/proto``):
* Python round-trip across the full envelope, tensor bundle, and every service.
* Proto3 forward/backward compatibility (unknown-field preservation, defaults).
* Bounded-fragment tensor bundle framing + checksums.
* Cross-language Python<->C++ round-trip when the C++ toolchain is available;
otherwise the C++ test skips with an explicit reason (deterministic, GPU-free,
model-download-free, API-credit-free by construction).
"""
from __future__ import annotations
import shutil
import subprocess
import pytest
# grpc_tools (grpcio-tools) is required to generate the stubs. It is present in
# the project .venv; skip cleanly elsewhere rather than error.
native_protocol = pytest.importorskip(
"meshnet_node.native_protocol",
reason="meshnet_node.native_protocol import failed",
)
try:
native_protocol.generate()
_GEN_ERROR = None
except native_protocol.ProtocGenerationError as exc: # pragma: no cover
_GEN_ERROR = str(exc)
pytestmark = pytest.mark.skipif(
_GEN_ERROR is not None,
reason=f"protobuf stubs unavailable: {_GEN_ERROR}",
)
@pytest.fixture(scope="module")
def pb2():
return native_protocol.load()
# ---------------------------------------------------------------------------
# Envelope / header round-trip and field coverage
# ---------------------------------------------------------------------------
def _full_header(pb2):
return pb2.MessageHeader(
schema_version=pb2.SCHEMA_VERSION_1,
work_id="work-42",
route_session_id="rs-7",
route_epoch=9,
fingerprint=pb2.ArtifactFingerprint(
model_id="meta-llama/Llama-3.1-8B",
revision="main",
artifact_hash="sha256:deadbeef",
quantization="Q4_K_M",
runtime_recipe_fingerprint="recipe-123",
),
shard_range=pb2.ShardRange(
start_layer=8,
end_layer=16,
effective_start_layer=9,
owns_embedding=False,
owns_final_head=False,
),
phase=pb2.PHASE_PREFILL,
position=pb2.Position(start_position=0, token_count=12, sequence_length=12),
idempotency_step=3,
cache_expectation=pb2.CACHE_REUSE,
compression=pb2.COMPRESSION_ZSTD,
checksum=pb2.Checksum(algorithm=pb2.CHECKSUM_CRC32C, value=b"\x00\x01\x02\x03"),
)
def test_message_header_carries_every_required_field(pb2):
"""The header carries every identifier the transport contract demands.
Tags: protocol
"""
header = _full_header(pb2)
raw = header.SerializeToString()
back = pb2.MessageHeader()
back.ParseFromString(raw)
assert back.schema_version == pb2.SCHEMA_VERSION_1
assert back.work_id == "work-42"
assert back.route_session_id == "rs-7"
assert back.route_epoch == 9
assert back.fingerprint.artifact_hash == "sha256:deadbeef"
assert back.fingerprint.runtime_recipe_fingerprint == "recipe-123"
assert back.shard_range.effective_start_layer == 9
assert back.phase == pb2.PHASE_PREFILL
assert back.position.token_count == 12
assert back.idempotency_step == 3
assert back.cache_expectation == pb2.CACHE_REUSE
assert back.compression == pb2.COMPRESSION_ZSTD
assert back.checksum.algorithm == pb2.CHECKSUM_CRC32C
assert back.checksum.value == b"\x00\x01\x02\x03"
def test_named_tensor_bundle_describes_shape_dtype_byteorder_and_fragments(pb2):
"""A tensor bundle round-trips name, shape, dtype, byte order and fragments.
Tags: protocol
"""
bundle = pb2.TensorBundle(
bundle_version=1,
tensors=[
pb2.NamedTensor(
name="hidden_states",
shape=[2, 3, 4096],
dtype=pb2.DTYPE_BF16,
byte_order=pb2.BYTE_ORDER_LITTLE_ENDIAN,
total_byte_length=16,
compression=pb2.COMPRESSION_NONE,
fragments=[
pb2.TensorFragment(
fragment_index=0,
fragment_count=2,
byte_offset=0,
data=b"\x00" * 8,
),
pb2.TensorFragment(
fragment_index=1,
fragment_count=2,
byte_offset=8,
data=b"\x01" * 8,
),
],
)
],
)
back = pb2.TensorBundle()
back.ParseFromString(bundle.SerializeToString())
tensor = back.tensors[0]
assert tensor.name == "hidden_states"
assert list(tensor.shape) == [2, 3, 4096]
assert tensor.dtype == pb2.DTYPE_BF16
assert tensor.byte_order == pb2.BYTE_ORDER_LITTLE_ENDIAN
assert [f.byte_offset for f in tensor.fragments] == [0, 8]
def test_session_stream_carries_open_prefill_decode_release_cancel(pb2):
"""The bidi stream oneof expresses every seam operation.
Tags: protocol
"""
header = _full_header(pb2)
frames = {
"open": pb2.SessionActivation(
open=pb2.SessionOpen(
header=header,
deadline_unix_nanos=1_000_000,
max_prefill_tokens_per_chunk=256,
max_fragment_bytes=1 << 20,
initial_credit=pb2.FlowControl(credits=8, max_in_flight_bytes=1 << 24),
)
),
"prefill": pb2.SessionActivation(
prefill=pb2.PrefillChunk(
header=header, chunk_index=0, chunk_count=2, final_chunk=False
)
),
"decode": pb2.SessionActivation(decode=pb2.DecodeStep(header=header)),
"release": pb2.SessionActivation(
release=pb2.ReleaseRequest(header=header, reason="done")
),
"cancel": pb2.SessionActivation(
cancel=pb2.CancelRequest(header=header, reason="client abort")
),
"flow_control": pb2.SessionActivation(
flow_control=pb2.FlowControl(credits=4)
),
}
for name, frame in frames.items():
back = pb2.SessionActivation()
back.ParseFromString(frame.SerializeToString())
assert back.WhichOneof("payload") == name
def test_session_response_carries_structured_status_and_results(pb2):
"""Server frames carry accepted/result/status/acks with structured Status.
Tags: protocol
"""
status = pb2.Status(
code=8,
message="resource exhausted",
retry_class=pb2.RETRY_CLASS_RETRYABLE,
details={"queue_depth": "128"},
)
resp = pb2.SessionResponse(
result=pb2.ActivationResult(
header=_full_header(pb2),
outputs=pb2.TensorBundle(bundle_version=1),
cache_result=pb2.CACHE_WRITTEN,
status=status,
)
)
back = pb2.SessionResponse()
back.ParseFromString(resp.SerializeToString())
assert back.WhichOneof("payload") == "result"
assert back.result.cache_result == pb2.CACHE_WRITTEN
assert back.result.status.retry_class == pb2.RETRY_CLASS_RETRYABLE
assert back.result.status.details["queue_depth"] == "128"
def test_capability_and_health_round_trip(pb2):
"""Capability and health messages round-trip their admission fields.
Tags: protocol
"""
cap = pb2.CapabilityResponse(
schema_version=pb2.SCHEMA_VERSION_1,
supported_schema_versions=[pb2.SCHEMA_VERSION_1],
supported_architectures=["llama"],
supported_quantizations=["Q4_K_M", "F16"],
servable_range=pb2.ShardRange(start_layer=0, end_layer=16),
budget=pb2.ResourceBudget(
weight_bytes=1 << 32, kv_bytes=1 << 30, max_concurrent_sessions=4
),
supported_compression=[pb2.COMPRESSION_NONE, pb2.COMPRESSION_ZSTD],
supported_checksums=[pb2.CHECKSUM_CRC32C, pb2.CHECKSUM_SHA256],
)
cap_back = pb2.CapabilityResponse()
cap_back.ParseFromString(cap.SerializeToString())
assert cap_back.budget.max_concurrent_sessions == 4
assert list(cap_back.supported_quantizations) == ["Q4_K_M", "F16"]
health = pb2.HealthResponse(
status=pb2.SERVING, active_sessions=2, queued_requests=1, kv_pressure=0.5
)
health_back = pb2.HealthResponse()
health_back.ParseFromString(health.SerializeToString())
assert health_back.status == pb2.SERVING
assert health_back.kv_pressure == pytest.approx(0.5)
# ---------------------------------------------------------------------------
# Compatibility
# ---------------------------------------------------------------------------
def test_unknown_fields_are_preserved_for_forward_compatibility(pb2):
"""An older reader tolerates and preserves fields it does not know.
A newer sender may add a field; parsing into the current schema must not
fail and must round-trip the unknown bytes.
Tags: protocol, compatibility
"""
header = _full_header(pb2)
raw = bytearray(header.SerializeToString())
# Append an unknown field: number 5000, wire type 2 (length-delimited).
tag = (5000 << 3) | 2
raw += _encode_varint(tag)
payload = b"future-field"
raw += _encode_varint(len(payload))
raw += payload
parsed = pb2.MessageHeader()
# Parsing must not raise on the unknown field.
parsed.ParseFromString(bytes(raw))
# Known fields survive intact.
assert parsed.work_id == "work-42"
assert parsed.route_epoch == 9
# The unknown bytes are preserved and re-emitted on re-serialization. This is
# the behavioural compatibility guarantee; the introspection accessor
# (UnknownFields()) is not implemented by the upb backend, so we assert the
# observable outcome rather than the accessor.
reserialized = parsed.SerializeToString()
assert payload in reserialized
assert _encode_varint(tag) in reserialized
def test_defaults_are_stable_for_backward_compatibility(pb2):
"""A message from an older sender (missing new fields) reads as enum zero.
Tags: protocol, compatibility
"""
empty = pb2.MessageHeader()
back = pb2.MessageHeader()
back.ParseFromString(empty.SerializeToString())
assert back.schema_version == pb2.SCHEMA_VERSION_UNSPECIFIED
assert back.phase == pb2.PHASE_UNSPECIFIED
assert back.cache_expectation == pb2.CACHE_EXPECTATION_UNSPECIFIED
assert back.work_id == ""
assert back.route_epoch == 0
# ---------------------------------------------------------------------------
# Bounded-fragment helpers
# ---------------------------------------------------------------------------
def test_fragment_and_reassemble_round_trip_with_checksums(pb2):
"""Bounded fragmentation reassembles exactly and validates checksums.
Tags: protocol
"""
payload = bytes((i * 7) % 256 for i in range(10_000))
tensor = native_protocol.fragment_tensor(
name="hidden",
shape=[1, 4096],
dtype=pb2.DTYPE_F16,
payload=payload,
max_fragment_bytes=4096,
checksum_algorithm=pb2.CHECKSUM_CRC32C,
)
assert len(tensor.fragments) == 3
assert all(len(f.data) <= 4096 for f in tensor.fragments)
# Survives a serialization round-trip before reassembly.
back = pb2.NamedTensor()
back.ParseFromString(tensor.SerializeToString())
assert native_protocol.reassemble_tensor(back) == payload
def test_reassemble_detects_fragment_corruption(pb2):
"""A flipped fragment byte fails checksum verification.
Tags: protocol
"""
payload = b"abcdefabcdef" * 100
tensor = native_protocol.fragment_tensor(
name="t",
shape=[len(payload)],
dtype=pb2.DTYPE_U8,
payload=payload,
max_fragment_bytes=256,
checksum_algorithm=pb2.CHECKSUM_SHA256,
)
tensor.fragments[1].data = tensor.fragments[1].data[:-1] + b"\x00"
with pytest.raises(ValueError, match="checksum mismatch"):
native_protocol.reassemble_tensor(tensor)
def test_checksum_algorithms_verify(pb2):
"""CRC32C, CRC32 and SHA256 all verify their own payloads.
Tags: protocol
"""
data = b"the quick brown fox"
for algo in (pb2.CHECKSUM_CRC32C, pb2.CHECKSUM_CRC32, pb2.CHECKSUM_SHA256):
checksum = native_protocol.compute_checksum(algo, data)
assert native_protocol.verify_checksum(checksum, data)
assert not native_protocol.verify_checksum(checksum, data + b"!")
def test_service_descriptor_exposes_all_operations(pb2):
"""The generated service defines capability/health/session/release/cancel.
Requires the grpc runtime; skips cleanly without it.
Tags: protocol
"""
grpc = pytest.importorskip("grpc", reason="grpc runtime not installed")
assert grpc is not None
grpc_mod = native_protocol.load_grpc()
assert hasattr(grpc_mod, "ShardRuntimeStub")
assert hasattr(grpc_mod, "ShardRuntimeServicer")
# Confirm the streaming seam and unary ops exist on the servicer.
servicer = grpc_mod.ShardRuntimeServicer
for op in ("GetCapability", "Health", "ActivateSession", "Release", "Cancel"):
assert hasattr(servicer, op), op
# ---------------------------------------------------------------------------
# Cross-language Python <-> C++ compatibility
# ---------------------------------------------------------------------------
def _cpp_toolchain_reason() -> str | None:
"""Return a skip reason if the C++ build toolchain is unavailable."""
for tool in ("cmake", "protoc"):
if shutil.which(tool) is None:
return f"{tool} not found on PATH"
return None
def _build_cpp_compatible_sample(pb2):
"""Python message matching what roundtrip_test.cpp CheckSample expects."""
header = pb2.MessageHeader(
schema_version=pb2.SCHEMA_VERSION_1,
work_id="w1",
route_session_id="s1",
route_epoch=3,
phase=pb2.PHASE_PREFILL,
idempotency_step=7,
cache_expectation=pb2.CACHE_FRESH,
compression=pb2.COMPRESSION_NONE,
fingerprint=pb2.ArtifactFingerprint(
model_id="meta-llama/Llama-3.1-8B",
quantization="Q4_K_M",
runtime_recipe_fingerprint="recipe-abc",
),
shard_range=pb2.ShardRange(
start_layer=0, end_layer=16, effective_start_layer=0, owns_embedding=True
),
position=pb2.Position(start_position=0, token_count=5, sequence_length=5),
)
return pb2.SessionActivation(
prefill=pb2.PrefillChunk(
header=header,
chunk_index=0,
chunk_count=1,
final_chunk=True,
activations=pb2.TensorBundle(
bundle_version=1,
tensors=[
pb2.NamedTensor(
name="hidden",
shape=[1, 4096],
dtype=pb2.DTYPE_F16,
byte_order=pb2.BYTE_ORDER_LITTLE_ENDIAN,
total_byte_length=8,
compression=pb2.COMPRESSION_NONE,
fragments=[
pb2.TensorFragment(
fragment_index=0,
fragment_count=1,
byte_offset=0,
data=bytes(range(1, 9)),
)
],
)
],
),
)
)
def test_cross_language_roundtrip_python_and_cpp(pb2, tmp_path):
"""Python and C++ parse each other's serialized frames (both directions).
Builds the C++ round-trip binary via CMake, feeds it a Python-serialized
fixture (C++ must parse it), and parses the C++-serialized output back in
Python. Skips with an explicit reason when the C++ toolchain is absent.
Tags: protocol, compatibility, cpp
"""
reason = _cpp_toolchain_reason()
if reason is not None:
pytest.skip(f"C++ toolchain unavailable: {reason}")
native_root = native_protocol.PROTO_DIR.parent
build_dir = tmp_path / "cpp-build"
configure = subprocess.run(
["cmake", "-S", str(native_root), "-B", str(build_dir)],
capture_output=True,
text=True,
)
if configure.returncode != 0:
pytest.skip(
"cmake configure failed (protobuf C++ dev likely missing):\n"
+ configure.stderr[-2000:]
)
build = subprocess.run(
["cmake", "--build", str(build_dir), "--target",
"shard_protocol_roundtrip_test"],
capture_output=True,
text=True,
)
assert build.returncode == 0, f"C++ build failed:\n{build.stderr[-2000:]}"
binary = build_dir / "shard_protocol_roundtrip_test"
assert binary.exists(), "C++ test binary not produced"
py_fixture = tmp_path / "py_sample.bin"
cpp_out = tmp_path / "cpp_sample.bin"
py_fixture.write_bytes(_build_cpp_compatible_sample(pb2).SerializeToString())
run = subprocess.run(
[str(binary), "--selftest", "--read", str(py_fixture),
"--write", str(cpp_out)],
capture_output=True,
text=True,
)
assert run.returncode == 0, f"C++ round-trip failed:\n{run.stdout}\n{run.stderr}"
# C++ parsed our bytes; now Python parses C++'s bytes and checks known fields.
parsed = pb2.SessionActivation()
parsed.ParseFromString(cpp_out.read_bytes())
assert parsed.WhichOneof("payload") == "prefill"
assert parsed.prefill.header.work_id == "w1"
assert parsed.prefill.header.route_epoch == 3
assert parsed.prefill.activations.tensors[0].name == "hidden"
assert parsed.prefill.activations.tensors[0].dtype == pb2.DTYPE_F16
# ---------------------------------------------------------------------------
# Local helpers
# ---------------------------------------------------------------------------
def _encode_varint(value: int) -> bytes:
out = bytearray()
while True:
byte = value & 0x7F
value >>= 7
if value:
out.append(byte | 0x80)
else:
out.append(byte)
return bytes(out)

50
tests/test_node_admission.py
View File

@@ -22,7 +22,6 @@ import pytest
from meshnet_node.admission import (
REASON_BACKEND_MISMATCH,
REASON_COMPATIBILITY_MISMATCH,
REASON_MODEL_MISMATCH,
REASON_NO_REPORT,
REASON_NOT_PASSED,
@@ -69,26 +68,11 @@ class _FakeBackend:
total_layers = 24
hidden_size = 8
def __init__(
self,
*,
shard_start=0,
shard_end=23,
device="cpu",
forward_error=None,
loaded_shard_start=None,
loaded_shard_end=None,
owns_embedding=None,
owns_final_head=None,
):
def __init__(self, *, shard_start=0, shard_end=23, device="cpu", forward_error=None):
self.shard_start = shard_start
self.shard_end = shard_end
self.is_head = shard_start == 0
self.is_tail = shard_end == self.total_layers - 1
self.loaded_shard_start = shard_start if loaded_shard_start is None else loaded_shard_start
self.loaded_shard_end = shard_end if loaded_shard_end is None else loaded_shard_end
self.owns_embedding = self.is_head if owns_embedding is None else owns_embedding
self.owns_final_head = self.is_tail if owns_final_head is None else owns_final_head
self.device = _FakeDevice(device)
self.model_id = MODEL
self._forward_error = forward_error
@@ -208,17 +192,6 @@ def test_a_passing_report_from_another_backend_or_device_is_refused():
assert exc.value.reason == REASON_BACKEND_MISMATCH
def test_a_passing_report_with_the_wrong_cache_layout_is_refused():
"The compatibility fingerprint fails closed when cache layout changes.\n\nTags: node, admission"
ctx = _context()
report = capability_report_for(ctx, cache_layout="local-hot-kv")
with pytest.raises(CapabilityAdmissionError) as exc:
admit(AdmissionRequirement.for_context(ctx), report)
assert exc.value.reason == REASON_COMPATIBILITY_MISMATCH
def test_a_report_older_than_the_freshness_window_is_refused():
"Hardware, drivers and weights move; an old proof is not a current one.\n\nTags: node, admission"
ctx = _context()
@@ -465,31 +438,10 @@ def test_a_matching_passing_report_registers_and_travels_with_the_payload(startu
assert report["status"] == "passed"
assert report["model"]["model_id"] == MODEL
assert (report["shard"]["start"], report["shard"]["end"]) == (0, 23)
assert report["shard"]["owns_embedding"] is True
assert report["shard"]["owns_final_head"] is True
assert report["recipe"]["recipe_id"] == DEFAULT_RECIPE_ID
assert report["backend"]["device"] == "cpu"
def test_capability_report_prefers_backend_loaded_range_over_cli_claims():
"The proof reports the model's loaded range, not the CLI's requested range.\n\nTags: node, admission"
backend = _FakeBackend(
shard_start=0,
shard_end=23,
loaded_shard_start=8,
loaded_shard_end=15,
owns_embedding=False,
owns_final_head=True,
)
report = capability_report_for(
_context(backend=backend, shard_start=0, shard_end=23),
)
assert (report.shard.start, report.shard.end) == (8, 15)
assert report.shard.owns_embedding is False
assert report.shard.owns_final_head is True
def test_the_served_backend_is_loaded_with_the_recipe_that_was_validated(startup_env):
"The recipe named in the report is the one the serving backend actually ran.\n\nTags: node, admission, startup"
node = _start(recipe_id="eager-attention")

54
tests/test_node_capability.py
View File

@@ -42,12 +42,9 @@ def _report(**overrides):
status="passed",
duration_ms=142,
validated_at=1_760_000_000.0,
owns_embedding=True,
owns_final_head=False,
)
kwargs.update(overrides)
report = build_capability_report(**kwargs)
return report
return build_capability_report(**kwargs)
# --- model-agnostic identity ------------------------------------------------
@@ -117,9 +114,6 @@ def test_report_dict_has_the_stable_documented_key_set():
"shard",
"recipe",
"backend",
"artifact",
"runtime_recipe",
"compatibility_fingerprint",
"status",
"validated_at",
"duration_ms",
@@ -127,38 +121,12 @@ def test_report_dict_has_the_stable_documented_key_set():
}
assert payload["schema_version"] == CAPABILITY_SCHEMA_VERSION
assert set(payload["model"]) == {"model_id", "revision", "config_fingerprint"}
assert set(payload["shard"]) == {
"start",
"end",
"owns_embedding",
"owns_final_head",
}
assert set(payload["shard"]) == {"start", "end"}
assert set(payload["recipe"]) == {
"recipe_id",
"recipe_version",
"catalogue_version",
}
assert set(payload["artifact"]) == {
"model_id",
"revision",
"artifact_hash",
"shard_start",
"shard_end",
}
assert set(payload["runtime_recipe"]) == {
"weight_quantization",
"activation_dtype",
"compute_dtype",
"kv_dtype",
"kv_layout",
"tokenizer_revision",
"architecture_adapter",
"backend_id",
"runtime_version",
"boundary_schema_version",
"cache_layout",
"fingerprint",
}
assert set(payload["backend"]) == {
"backend_id",
"device",
@@ -166,19 +134,10 @@ def test_report_dict_has_the_stable_documented_key_set():
"quantization",
"runtime",
}
assert payload["compatibility_fingerprint"].startswith("sha256:")
# JSON-serializable end to end.
assert json.loads(json.dumps(payload)) == payload
def test_report_carries_endpoint_ownership():
"Endpoint ownership is recorded alongside the shard range.\n\nTags: node, startup"
payload = _report().to_dict()
assert payload["shard"]["owns_embedding"] is True
assert payload["shard"]["owns_final_head"] is False
def test_identity_key_pins_model_shard_recipe_and_backend():
"Identity key pins model shard recipe and backend\n\nTags: node, startup"
base = _report()
@@ -197,15 +156,6 @@ def test_identity_key_pins_model_shard_recipe_and_backend():
assert _report(device="other-device").identity_key() != base.identity_key()
def test_compatibility_fingerprint_changes_when_the_runtime_recipe_changes():
"The compatibility fingerprint changes when the runtime recipe changes.\n\nTags: node, startup"
base = _report()
altered = _report(cache_layout="stateless")
assert base.compatibility_fingerprint != altered.compatibility_fingerprint
assert base.runtime_recipe.fingerprint != altered.runtime_recipe.fingerprint
def test_config_fingerprint_is_stable_under_key_order_and_detects_change():
"Config fingerprint is stable under key order and detects change\n\nTags: node, startup"
a = config_fingerprint({"num_hidden_layers": 8, "hidden_size": 512})

Some files were not shown because too many files have changed in this diff Show More