chore: clean superseded GGUF scaffolding
This commit is contained in:
Dobromir Popov
@@ -1,127 +0,0 @@
|
||||
# DGR-001 — performance contract baseline
|
||||
|
||||
## Files changed
|
||||
|
||||
- `packages/node/meshnet_node/performance_contract.py`
|
||||
- `tests/test_performance_contract.py`
|
||||
- `.scratch/distributed-gguf-runtime/issues/01-lock-the-safetensors-versus-gguf-performance-contract.md`
|
||||
- `.scratch/distributed-gguf-runtime/evidence/DGR-001/performance-contract.json`
|
||||
|
||||
## What this slice does
|
||||
|
||||
- Locks the DGR-001 benchmark contract in code.
|
||||
- Pins the architecture-aligned baseline to **DeepSeek-V2-Lite-Chat** (`deepseek2`).
|
||||
- Uses the same model on both sides of the comparison:
|
||||
- **safetensors:** `deepseek-ai/DeepSeek-V2-Lite-Chat` in **BF16**
|
||||
- **GGUF:** `second-state/DeepSeek-V2-Lite-Chat-GGUF` in **Q2_K**
|
||||
- Exposes a machine-readable JSON contract with:
|
||||
- benchmark lanes for `transformers` safetensors and `llama.cpp` GGUF on **CPU** and **GPU**
|
||||
- concurrency levels `1` and `4`
|
||||
- the required metrics list
|
||||
- an explicit stop condition for “no meaningful speed or fit benefit”
|
||||
- Adds a deterministic stub benchmark report so the contract now has an executable report shape end to end.
|
||||
|
||||
## Recent benchmark runner slice
|
||||
|
||||
The runner currently uses a deterministic stub backend to exercise the comparison matrix without downloading a model. It emits:
|
||||
|
||||
- `.scratch/distributed-gguf-runtime/evidence/DGR-001/performance-contract.json`
|
||||
- `.scratch/distributed-gguf-runtime/evidence/DGR-001/stub-benchmark-report.json`
|
||||
|
||||
The report includes per-device comparisons for:
|
||||
|
||||
- `transformers-safetensors-cpu` vs `llama-cpp-gguf-cpu`
|
||||
- `transformers-safetensors-gpu` vs `llama-cpp-gguf-gpu`
|
||||
|
||||
and records the memory metric (`rss_bytes` on CPU, `vram_bytes` on GPU), decode speedup, artifact ratio, and output drift.
|
||||
|
||||
## Live endpoint CLI wiring
|
||||
|
||||
The contract CLI can now drive the live endpoint runner. Passing one `--live-endpoint LANE_ID=URL` mapping per contract lane (plus `--live-benchmark-out`) invokes `run_real_model_endpoint_benchmark` against already-running OpenAI-compatible servers and writes the report using the same schema as the stub:
|
||||
|
||||
```bash
|
||||
PYTHONPATH=packages/node python -m meshnet_node.performance_contract \
|
||||
--live-endpoint transformers-safetensors-cpu=http://127.0.0.1:8001 \
|
||||
--live-endpoint llama-cpp-gguf-cpu=http://127.0.0.1:8002 \
|
||||
--live-endpoint transformers-safetensors-gpu=http://127.0.0.1:8003 \
|
||||
--live-endpoint llama-cpp-gguf-gpu=http://127.0.0.1:8004 \
|
||||
--live-benchmark-out .scratch/distributed-gguf-runtime/evidence/DGR-001/live-benchmark-report.json
|
||||
```
|
||||
|
||||
`--live-model` overrides the model name sent in requests (defaults to the contract's safetensors repo). Without any `--live-endpoint` flags the CLI behaves exactly as before: it writes the contract JSON and, with `--benchmark-out`, the deterministic stub report.
|
||||
|
||||
## Exact commands and real results
|
||||
|
||||
### Targeted tests
|
||||
|
||||
```bash
|
||||
PYTHONPATH=packages/node pytest -q tests/test_performance_contract.py tests/test_route_session_benchmark.py
|
||||
```
|
||||
|
||||
Result: `19 passed in 0.11s`
|
||||
|
||||
### Contract artifact generation
|
||||
|
||||
```bash
|
||||
PYTHONPATH=packages/node python -m meshnet_node.performance_contract --json-out .scratch/distributed-gguf-runtime/evidence/DGR-001/performance-contract.json
|
||||
```
|
||||
|
||||
Result: wrote `.scratch/distributed-gguf-runtime/evidence/DGR-001/performance-contract.json`
|
||||
|
||||
### Python compile check
|
||||
|
||||
```bash
|
||||
python -m compileall packages/node/meshnet_node/performance_contract.py tests/test_performance_contract.py
|
||||
```
|
||||
|
||||
Result: passed
|
||||
|
||||
## Public relay smoke benchmark (2026-07-15)
|
||||
|
||||
A real streamed request was run through the public tracker — **not** by connecting directly to the private node address:
|
||||
|
||||
```text
|
||||
https://meshnet.2.d-popov.com/v1/chat/completions
|
||||
-> wss://meshnet.2.d-popov.com/ws
|
||||
-> wss://meshnet.2.d-popov.com/rpc/7j77FsPY1evV8tuf-7000
|
||||
-> local CUDA node, Qwen/Qwen2.5-0.5B-Instruct layers 0-23
|
||||
```
|
||||
|
||||
The local public-tracker node had an expired proof and a wedged HTTP server. A graceful restart refreshed its CUDA capability proof in `336 ms`, restored `admitted`/`routable` status, and reconnected its relay endpoint.
|
||||
|
||||
Measured streaming results after recovery:
|
||||
|
||||
| metric | result |
|
||||
| --- | ---: |
|
||||
| warm-up TTFT | 420.80 ms |
|
||||
| warm-up elapsed | 610.23 ms |
|
||||
| p50 TTFT (3 runs) | 288.26 ms |
|
||||
| p50 elapsed (3 runs) | 363.20 ms |
|
||||
| tracker-recorded relay throughput | 58.18-65.25 tok/s |
|
||||
| HTTP status | 200 for all runs |
|
||||
|
||||
The tracker recorded `relay: true` and the local node ID `7j77FsPY-b32476219492` for each completion. Full redacted evidence is in `public-relay-smoke-benchmark.json`.
|
||||
|
||||
The other connected node is still alive but **not routable** because its capability proof is stale. It must revalidate before a multi-node shard/relay test can run.
|
||||
|
||||
## Limitations
|
||||
|
||||
- This slice still uses a deterministic stub backend for the core comparison matrix.
|
||||
- It now also includes a live endpoint runner, reachable from the CLI via `--live-endpoint`/`--live-benchmark-out`, that fans out one OpenAI-compatible request per lane when the caller provides endpoints; the CLI does not start those servers.
|
||||
- It does **not** download or run a real model from within the repo.
|
||||
- Real safetensors vs GGUF execution, TTFT/prefill/decode measurements, RSS/VRAM capture, and output-drift comparison are still to be implemented against the contract.
|
||||
|
||||
## Compatibility notes
|
||||
|
||||
- The contract stays on the DeepSeek2 family to remain close to the DeepSeek-V4-Flash end goal.
|
||||
- A smaller non-DeepSeek model can still be used later for loader-plumbing smoke tests, but it does not replace this baseline.
|
||||
- Model artifacts must stay on the mounted drive and not under `/home`.
|
||||
|
||||
## Dependent-story handoff
|
||||
|
||||
Next implementation work should attach to this contract and add the live benchmark runner that actually compares:
|
||||
|
||||
1. current Transformers/safetensors recipe
|
||||
2. whole-model llama.cpp GGUF recipe
|
||||
|
||||
using the same model architecture/revision and the same prompt/context/concurrency settings.
|
||||
@@ -1,75 +0,0 @@
|
||||
{
|
||||
"benchmark_lanes": [
|
||||
{
|
||||
"concurrency_levels": [
|
||||
1,
|
||||
4
|
||||
],
|
||||
"device": "cpu",
|
||||
"id": "transformers-safetensors-cpu",
|
||||
"recipe": "current safetensors recipe",
|
||||
"runtime": "transformers"
|
||||
},
|
||||
{
|
||||
"concurrency_levels": [
|
||||
1,
|
||||
4
|
||||
],
|
||||
"device": "cpu",
|
||||
"id": "llama-cpp-gguf-cpu",
|
||||
"recipe": "whole-model GGUF recipe",
|
||||
"runtime": "llama.cpp"
|
||||
},
|
||||
{
|
||||
"concurrency_levels": [
|
||||
1,
|
||||
4
|
||||
],
|
||||
"device": "gpu",
|
||||
"id": "transformers-safetensors-gpu",
|
||||
"recipe": "current safetensors recipe",
|
||||
"runtime": "transformers"
|
||||
},
|
||||
{
|
||||
"concurrency_levels": [
|
||||
1,
|
||||
4
|
||||
],
|
||||
"device": "gpu",
|
||||
"id": "llama-cpp-gguf-gpu",
|
||||
"recipe": "whole-model GGUF recipe",
|
||||
"runtime": "llama.cpp"
|
||||
}
|
||||
],
|
||||
"metrics": [
|
||||
"ttft_ms",
|
||||
"prefill_tok_per_sec",
|
||||
"decode_tok_per_sec",
|
||||
"p50_latency_ms",
|
||||
"p95_latency_ms",
|
||||
"aggregate_throughput_tok_per_sec",
|
||||
"rss_bytes",
|
||||
"vram_bytes",
|
||||
"artifact_bytes",
|
||||
"failure_count",
|
||||
"output_drift"
|
||||
],
|
||||
"model_target": {
|
||||
"architecture": "deepseek2",
|
||||
"comparison_policy": "same model/revision, closest practical low-footprint precision pair: BF16 safetensors versus Q2_K GGUF",
|
||||
"gguf_quant": "Q2_K",
|
||||
"gguf_repo": "second-state/DeepSeek-V2-Lite-Chat-GGUF",
|
||||
"gguf_size_gb": 6.43,
|
||||
"name": "DeepSeek-V2-Lite-Chat",
|
||||
"rationale": "Smallest DeepSeek-family benchmark anchor that still points toward DeepSeek-V4-Flash; keeps the runtime on the DeepSeek2 path instead of falling back to a tiny but architecture-mismatched smoke model.",
|
||||
"safetensors_precision": "bfloat16",
|
||||
"safetensors_repo": "deepseek-ai/DeepSeek-V2-Lite-Chat"
|
||||
},
|
||||
"notes": [
|
||||
"Real model execution stays opt-in and must keep model artifacts on the mounted drive.",
|
||||
"Use the tiny fallback only for loader plumbing smoke tests; it does not replace the architecture-aligned baseline."
|
||||
],
|
||||
"schema_version": 1,
|
||||
"stop_condition": "Stop if GGUF does not provide a meaningful speed or fit benefit over the safetensors baseline for the chosen DeepSeek-family model target.",
|
||||
"story_id": "DGR-001"
|
||||
}
|
||||
@@ -1,247 +0,0 @@
|
||||
{
|
||||
"comparisons": {
|
||||
"cpu": {
|
||||
"artifact_bytes_ratio": 0.2048,
|
||||
"decode_speedup": 2.3333,
|
||||
"gguf_benefit": true,
|
||||
"gguf_lane": "llama-cpp-gguf-cpu",
|
||||
"memory_bytes_ratio": 0.2152,
|
||||
"memory_metric": "rss_bytes",
|
||||
"output_drift": 0.0,
|
||||
"safetensors_lane": "transformers-safetensors-cpu",
|
||||
"ttft_speedup": 1.8947
|
||||
},
|
||||
"gpu": {
|
||||
"artifact_bytes_ratio": 0.2048,
|
||||
"decode_speedup": 1.5294,
|
||||
"gguf_benefit": true,
|
||||
"gguf_lane": "llama-cpp-gguf-gpu",
|
||||
"memory_bytes_ratio": 0.2273,
|
||||
"memory_metric": "vram_bytes",
|
||||
"output_drift": 0.0,
|
||||
"safetensors_lane": "transformers-safetensors-gpu",
|
||||
"ttft_speedup": 1.6154
|
||||
}
|
||||
},
|
||||
"lanes": [
|
||||
{
|
||||
"concurrency_levels": [
|
||||
1,
|
||||
4
|
||||
],
|
||||
"device": "cpu",
|
||||
"id": "transformers-safetensors-cpu",
|
||||
"output_tokens": [
|
||||
"mesh",
|
||||
"activation",
|
||||
"seam",
|
||||
"baseline"
|
||||
],
|
||||
"recipe": "current safetensors recipe",
|
||||
"results": [
|
||||
{
|
||||
"concurrency": 1,
|
||||
"metrics": {
|
||||
"aggregate_throughput_tok_per_sec": 6.0,
|
||||
"artifact_bytes": 33715493273,
|
||||
"decode_tok_per_sec": 6.0,
|
||||
"failure_count": 0,
|
||||
"output_drift": 0.0,
|
||||
"p50_latency_ms": 166.6667,
|
||||
"p95_latency_ms": 208.3334,
|
||||
"prefill_tok_per_sec": 45.0,
|
||||
"rss_bytes": 35433480192,
|
||||
"ttft_ms": 1800.0,
|
||||
"vram_bytes": 0
|
||||
}
|
||||
},
|
||||
{
|
||||
"concurrency": 4,
|
||||
"metrics": {
|
||||
"aggregate_throughput_tok_per_sec": 20.4,
|
||||
"artifact_bytes": 33715493273,
|
||||
"decode_tok_per_sec": 5.1,
|
||||
"failure_count": 0,
|
||||
"output_drift": 0.0,
|
||||
"p50_latency_ms": 196.0784,
|
||||
"p95_latency_ms": 245.098,
|
||||
"prefill_tok_per_sec": 38.25,
|
||||
"rss_bytes": 35433480192,
|
||||
"ttft_ms": 2340.0,
|
||||
"vram_bytes": 0
|
||||
}
|
||||
}
|
||||
],
|
||||
"runtime": "transformers"
|
||||
},
|
||||
{
|
||||
"concurrency_levels": [
|
||||
1,
|
||||
4
|
||||
],
|
||||
"device": "cpu",
|
||||
"id": "llama-cpp-gguf-cpu",
|
||||
"output_tokens": [
|
||||
"mesh",
|
||||
"activation",
|
||||
"seam",
|
||||
"baseline"
|
||||
],
|
||||
"recipe": "whole-model GGUF recipe",
|
||||
"results": [
|
||||
{
|
||||
"concurrency": 1,
|
||||
"metrics": {
|
||||
"aggregate_throughput_tok_per_sec": 14.0,
|
||||
"artifact_bytes": 6904159928,
|
||||
"decode_tok_per_sec": 14.0,
|
||||
"failure_count": 0,
|
||||
"output_drift": 0.0,
|
||||
"p50_latency_ms": 71.4286,
|
||||
"p95_latency_ms": 89.2858,
|
||||
"prefill_tok_per_sec": 90.0,
|
||||
"rss_bytes": 7623566950,
|
||||
"ttft_ms": 950.0,
|
||||
"vram_bytes": 0
|
||||
}
|
||||
},
|
||||
{
|
||||
"concurrency": 4,
|
||||
"metrics": {
|
||||
"aggregate_throughput_tok_per_sec": 47.6,
|
||||
"artifact_bytes": 6904159928,
|
||||
"decode_tok_per_sec": 11.9,
|
||||
"failure_count": 0,
|
||||
"output_drift": 0.0,
|
||||
"p50_latency_ms": 84.0336,
|
||||
"p95_latency_ms": 105.042,
|
||||
"prefill_tok_per_sec": 76.5,
|
||||
"rss_bytes": 7623566950,
|
||||
"ttft_ms": 1235.0,
|
||||
"vram_bytes": 0
|
||||
}
|
||||
}
|
||||
],
|
||||
"runtime": "llama.cpp"
|
||||
},
|
||||
{
|
||||
"concurrency_levels": [
|
||||
1,
|
||||
4
|
||||
],
|
||||
"device": "gpu",
|
||||
"id": "transformers-safetensors-gpu",
|
||||
"output_tokens": [
|
||||
"mesh",
|
||||
"activation",
|
||||
"seam",
|
||||
"baseline"
|
||||
],
|
||||
"recipe": "current safetensors recipe",
|
||||
"results": [
|
||||
{
|
||||
"concurrency": 1,
|
||||
"metrics": {
|
||||
"aggregate_throughput_tok_per_sec": 34.0,
|
||||
"artifact_bytes": 33715493273,
|
||||
"decode_tok_per_sec": 34.0,
|
||||
"failure_count": 0,
|
||||
"output_drift": 0.0,
|
||||
"p50_latency_ms": 29.4118,
|
||||
"p95_latency_ms": 36.7647,
|
||||
"prefill_tok_per_sec": 850.0,
|
||||
"rss_bytes": 4294967296,
|
||||
"ttft_ms": 420.0,
|
||||
"vram_bytes": 35433480192
|
||||
}
|
||||
},
|
||||
{
|
||||
"concurrency": 4,
|
||||
"metrics": {
|
||||
"aggregate_throughput_tok_per_sec": 115.6,
|
||||
"artifact_bytes": 33715493273,
|
||||
"decode_tok_per_sec": 28.9,
|
||||
"failure_count": 0,
|
||||
"output_drift": 0.0,
|
||||
"p50_latency_ms": 34.6021,
|
||||
"p95_latency_ms": 43.2526,
|
||||
"prefill_tok_per_sec": 722.5,
|
||||
"rss_bytes": 4294967296,
|
||||
"ttft_ms": 546.0,
|
||||
"vram_bytes": 35433480192
|
||||
}
|
||||
}
|
||||
],
|
||||
"runtime": "transformers"
|
||||
},
|
||||
{
|
||||
"concurrency_levels": [
|
||||
1,
|
||||
4
|
||||
],
|
||||
"device": "gpu",
|
||||
"id": "llama-cpp-gguf-gpu",
|
||||
"output_tokens": [
|
||||
"mesh",
|
||||
"activation",
|
||||
"seam",
|
||||
"baseline"
|
||||
],
|
||||
"recipe": "whole-model GGUF recipe",
|
||||
"results": [
|
||||
{
|
||||
"concurrency": 1,
|
||||
"metrics": {
|
||||
"aggregate_throughput_tok_per_sec": 52.0,
|
||||
"artifact_bytes": 6904159928,
|
||||
"decode_tok_per_sec": 52.0,
|
||||
"failure_count": 0,
|
||||
"output_drift": 0.0,
|
||||
"p50_latency_ms": 19.2308,
|
||||
"p95_latency_ms": 24.0385,
|
||||
"prefill_tok_per_sec": 640.0,
|
||||
"rss_bytes": 1610612736,
|
||||
"ttft_ms": 260.0,
|
||||
"vram_bytes": 8053063680
|
||||
}
|
||||
},
|
||||
{
|
||||
"concurrency": 4,
|
||||
"metrics": {
|
||||
"aggregate_throughput_tok_per_sec": 176.8,
|
||||
"artifact_bytes": 6904159928,
|
||||
"decode_tok_per_sec": 44.2,
|
||||
"failure_count": 0,
|
||||
"output_drift": 0.0,
|
||||
"p50_latency_ms": 22.6244,
|
||||
"p95_latency_ms": 28.2805,
|
||||
"prefill_tok_per_sec": 544.0,
|
||||
"rss_bytes": 1610612736,
|
||||
"ttft_ms": 338.0,
|
||||
"vram_bytes": 8053063680
|
||||
}
|
||||
}
|
||||
],
|
||||
"runtime": "llama.cpp"
|
||||
}
|
||||
],
|
||||
"model_target": {
|
||||
"architecture": "deepseek2",
|
||||
"comparison_policy": "same model/revision, closest practical low-footprint precision pair: BF16 safetensors versus Q2_K GGUF",
|
||||
"gguf_quant": "Q2_K",
|
||||
"gguf_repo": "second-state/DeepSeek-V2-Lite-Chat-GGUF",
|
||||
"gguf_size_gb": 6.43,
|
||||
"name": "DeepSeek-V2-Lite-Chat",
|
||||
"rationale": "Smallest DeepSeek-family benchmark anchor that still points toward DeepSeek-V4-Flash; keeps the runtime on the DeepSeek2 path instead of falling back to a tiny but architecture-mismatched smoke model.",
|
||||
"safetensors_precision": "bfloat16",
|
||||
"safetensors_repo": "deepseek-ai/DeepSeek-V2-Lite-Chat"
|
||||
},
|
||||
"schema_version": 1,
|
||||
"source": "stub-backend",
|
||||
"stop_condition": {
|
||||
"gguf_benefit": true,
|
||||
"text": "Stop if GGUF does not provide a meaningful speed or fit benefit over the safetensors baseline for the chosen DeepSeek-family model target.",
|
||||
"triggered": false
|
||||
},
|
||||
"story_id": "DGR-001"
|
||||
}
|
||||
@@ -1,176 +0,0 @@
|
||||
# DGR-002 — Versioned gRPC Shard protocol: evidence
|
||||
|
||||
Status: done
|
||||
Date: 2026-07-15
|
||||
Evidence kind: **synthetic-unit** (schema round-trip + cross-language protobuf
|
||||
compatibility). No model download, no GPU, no network, no API credits.
|
||||
|
||||
## Summary
|
||||
|
||||
Added the versioned Protocol Buffers schema that is the semantic contract between
|
||||
Python and C++ Shards (ADR-0024), plus reproducible Python and C++ code
|
||||
generation/build wiring and generated-schema round-trip + compatibility tests in
|
||||
**both** languages. The schema defines one long-lived bidirectional gRPC stream
|
||||
per Route Session Activation Seam, bounded prefill chunking, a small decode fast
|
||||
path, and a versioned named-tensor bundle carrying every required identifier.
|
||||
|
||||
No existing runtime code was modified — this story is purely additive (a new
|
||||
`.proto`, a `native_protocol` loader package, C++ build wiring, and one new test
|
||||
module). Generated stubs are produced on demand into gitignored `build/`
|
||||
directories, so nothing generated is committed.
|
||||
|
||||
## Files changed (all new)
|
||||
|
||||
- `packages/node/native/proto/shard_runtime.proto` — the schema (package
|
||||
`meshnet.shard.v1`, proto3). Service `ShardRuntime` with `GetCapability`,
|
||||
`Health`, `ActivateSession` (bidi stream), `Release`, `Cancel`.
|
||||
- `packages/node/meshnet_node/native_protocol/__init__.py` — reproducible
|
||||
on-demand `grpc_tools.protoc` codegen + loader (`load()`, `load_grpc()`) and
|
||||
shared bundle helpers (`compute_checksum`, `verify_checksum`, `fragment_tensor`,
|
||||
`reassemble_tensor`).
|
||||
- `packages/node/native/scripts/generate_python.py` — standalone reproducible
|
||||
Python generation (self-contained; does not import `meshnet_node`).
|
||||
- `packages/node/native/scripts/generate_cpp.sh` — reproducible C++ generation
|
||||
(message stubs always; gRPC service stubs when `grpc_cpp_plugin` is present).
|
||||
- `packages/node/native/CMakeLists.txt` — C++ build wiring; works with both
|
||||
CONFIG-mode (`protobuf::libprotobuf`/`protobuf::protoc`) and CMake's
|
||||
`FindProtobuf` module.
|
||||
- `packages/node/native/tests/roundtrip_test.cpp` — C++ round-trip / compat test
|
||||
(`--selftest`, `--read`, `--write`).
|
||||
- `tests/test_native_shard_protocol.py` — Python round-trip + compatibility tests
|
||||
and the Python↔C++ cross-language driver.
|
||||
|
||||
## Acceptance criteria → evidence
|
||||
|
||||
- **Capability/health/session-stream/release/cancellation schema** — the
|
||||
`ShardRuntime` service's five RPCs; `test_capability_and_health_round_trip`,
|
||||
`test_session_stream_carries_open_prefill_decode_release_cancel`.
|
||||
- **One long-lived bidi stream per Activation Seam with deadlines, cancellation,
|
||||
flow control, structured errors** — `rpc ActivateSession (stream ...) returns
|
||||
(stream ...)`. Deadlines: gRPC call deadline on direct transport, plus
|
||||
`SessionOpen.deadline_unix_nanos` for relay-carried frames. Cancellation:
|
||||
`Cancel` RPC and in-stream `CancelRequest`/`PHASE_CANCEL`. Flow control:
|
||||
`FlowControl` frames (credits + in-flight byte/message caps). Structured errors:
|
||||
`Status` (canonical code, message, `RetryClass`, details). Verified by
|
||||
`test_session_response_carries_structured_status_and_results`.
|
||||
- **Bounded prefill chunking + small decode fast path** — `PrefillChunk`
|
||||
(`chunk_index`/`chunk_count`/`final_chunk`, `SessionOpen.max_prefill_tokens_per_chunk`)
|
||||
and `DecodeStep` (minimal single-bundle path). Bounded fragments via
|
||||
`SessionOpen.max_fragment_bytes` and `fragment_tensor(...)`.
|
||||
- **Carries schema version, work ID, Route Session ID, route epoch,
|
||||
artifact/recipe fingerprint, shard range/effective start, phase, position,
|
||||
idempotency step, cache expectation, compression, checksum** — all on
|
||||
`MessageHeader` (+ `ArtifactFingerprint.runtime_recipe_fingerprint`,
|
||||
`ShardRange.effective_start_layer`). Verified field-by-field by
|
||||
`test_message_header_carries_every_required_field`.
|
||||
- **Versioned named-tensor bundle (name, shape, dtype, byte order, fragments)** —
|
||||
`TensorBundle`/`NamedTensor`/`TensorFragment`;
|
||||
`test_named_tensor_bundle_describes_shape_dtype_byteorder_and_fragments`,
|
||||
`test_fragment_and_reassemble_round_trip_with_checksums`.
|
||||
- **Round-trip + compatibility tests in Python and C++** — Python:
|
||||
`tests/test_native_shard_protocol.py` (11 tests). C++: `roundtrip_test.cpp`
|
||||
built via CMake; cross-language driver `test_cross_language_roundtrip_python_and_cpp`
|
||||
exercises Python→C++ and C++→Python in both directions.
|
||||
- **Targeted pytest** — `11 passed, 1 skipped` (default env); `12 passed` with the
|
||||
C++ toolchain on PATH.
|
||||
- **compileall packages tests** — exit 0.
|
||||
- **git diff --check** — clean.
|
||||
- **Deterministic / download-free / credit-free / GPU-free** — all tests are pure
|
||||
protobuf serialization; the C++ path uses only local compilers.
|
||||
- **Full deterministic pytest** — `704 passed, 14 skipped, 11 failed`. The 11
|
||||
failures are pre-existing and unrelated (see below).
|
||||
|
||||
## Commands and real results
|
||||
|
||||
See `commands.txt` for the exact command list. Key results:
|
||||
|
||||
- `python packages/node/native/scripts/generate_python.py` →
|
||||
`shard_runtime_pb2.py: ok`, `shard_runtime_pb2_grpc.py: ok`.
|
||||
- `pytest tests/test_native_shard_protocol.py -q` → **11 passed, 1 skipped**
|
||||
(skip reason: `C++ toolchain unavailable: cmake not found on PATH`).
|
||||
- With `/tmp/pbsrc/install/bin` (protoc 33.1) and `.venv/bin` (cmake) on PATH and
|
||||
`CMAKE_PREFIX_PATH=/tmp/pbsrc/install`:
|
||||
- `generate_cpp.sh` → `shard_runtime.pb.cc`, `shard_runtime.pb.h`
|
||||
(grpc service stubs skipped: `grpc_cpp_plugin` absent).
|
||||
- `cmake -S ... -B ...` + `cmake --build ...` → build OK.
|
||||
- `shard_protocol_roundtrip_test --selftest` → `selftest ok (128 bytes)`, exit 0.
|
||||
- `ctest` → `1/1 Test #1: shard_protocol_roundtrip ... Passed`.
|
||||
- `pytest ...::test_cross_language_roundtrip_python_and_cpp -q` → **1 passed**
|
||||
(Python serializes → C++ parses & verifies → C++ serializes → Python parses
|
||||
& verifies).
|
||||
- `compileall -q packages tests` → exit 0.
|
||||
- `git diff --check` → clean.
|
||||
|
||||
## Pre-existing unrelated failures (full-suite)
|
||||
|
||||
`pytest -q` on the full tree reports 11 failures, all in tracker routing /
|
||||
dynamic routing / manual route benchmark / toploc calibration — none import the
|
||||
Shard protocol. Clean-tree reproduction: with **all DGR-002 files moved aside**
|
||||
(`git status` shows only the pre-existing `.ralph-tui/config.toml` deletion),
|
||||
re-running exactly these tests gives `11 failed, 3 passed` — identical failures.
|
||||
They exist on the `ralph/distributed-gguf-runtime` branch independent of this
|
||||
story. The full list is in `results.json.preexisting_unrelated_failures`.
|
||||
|
||||
Note: the earlier `progress.md` (RCR-001, on master) recorded a different set of
|
||||
6 optional-dependency failures (zstandard, langchain_openai). Those did **not**
|
||||
recur here; this environment has those deps. The 11 above are branch-local
|
||||
routing/benchmark failures, not environmental.
|
||||
|
||||
## Limitations and deferred work
|
||||
|
||||
- **C++ toolchain is host-provided, not vendored.** The default test env has no
|
||||
`protoc`/`cmake`/protobuf C++ headers on PATH, so the C++ cross-language test
|
||||
**skips** by default (explicit skip reason). It was executed for this evidence
|
||||
using an ephemeral from-source protobuf 33.1 install at `/tmp/pbsrc/install`
|
||||
plus the `.venv` cmake. DGR-004/DGR-008 should pin the C++ protobuf/gRPC
|
||||
toolchain (upstream commit + reproducible fetch/build) so this test runs in CI
|
||||
without relying on an ad-hoc `/tmp` install.
|
||||
- **gRPC C++ service stubs not built here.** `grpc_cpp_plugin` is absent, so
|
||||
`generate_cpp.sh` produced message stubs only. The round-trip test needs only
|
||||
message serialization; the service stubs are DGR-008's concern.
|
||||
- **No live gRPC transport yet.** This story delivers the schema + serialization
|
||||
contract and generation/build wiring only. Channel setup, the bidi stream
|
||||
server/client, deadlines/cancellation propagation over a real HTTP/2 channel,
|
||||
and relay framing are DGR-008/DGR-009.
|
||||
- **Protobuf runtime version skew.** Python runtime is pip protobuf 7.35.1; the
|
||||
C++ side used protoc 33.1. Protobuf wire format is stable across these, and the
|
||||
cross-language round-trip confirms interop; version pinning is deferred to the
|
||||
toolchain-pinning stories.
|
||||
|
||||
## Compatibility / migration notes
|
||||
|
||||
- proto3 with a 0-valued `*_UNSPECIFIED` member on every enum and never-reused
|
||||
field numbers. Forward compatibility (unknown-field preservation) is verified
|
||||
behaviourally by `test_unknown_fields_are_preserved_for_forward_compatibility`
|
||||
— note protobuf 7.x's upb backend does not implement the `UnknownFields()`
|
||||
introspection accessor, so the test asserts the observable re-serialization
|
||||
outcome instead. Backward defaults verified by
|
||||
`test_defaults_are_stable_for_backward_compatibility`.
|
||||
- Wire schema version is `SchemaVersion.SCHEMA_VERSION_1` (int 1), also exposed as
|
||||
`meshnet_node.native_protocol.SCHEMA_VERSION`.
|
||||
|
||||
## Handoff for dependent stories
|
||||
|
||||
- **DGR-003 (recipe/fingerprint):** populate `ArtifactFingerprint`
|
||||
(`model_id`, `revision`, `artifact_hash`, `quantization`,
|
||||
`runtime_recipe_fingerprint`). Admission compares these before activation; a
|
||||
mismatch is a fatal `Status` (`RetryClass.RETRY_CLASS_FATAL`).
|
||||
- **DGR-004 (llama.cpp pin) / DGR-008 (C++ worker):** pin the C++
|
||||
protobuf + gRPC toolchain and add `grpc_cpp_plugin`; then `generate_cpp.sh`
|
||||
emits service stubs and the CMake target can link gRPC. Implement the
|
||||
`ShardRuntime` servicer; map `(route_session_id, route_epoch)` to an isolated
|
||||
llama sequence. Use `SessionOpen` for stream-scoped bounds and `FlowControl`
|
||||
for backpressure.
|
||||
- **DGR-009 (Meshnet integration/relay):** the relay may carry serialized
|
||||
`SessionActivation`/`SessionResponse` frames as opaque binary; use the in-message
|
||||
`deadline_unix_nanos`, `CancelRequest`, and `FlowControl` since gRPC call
|
||||
metadata is lost over relay.
|
||||
- **Loader usage:** `from meshnet_node import native_protocol as proto;
|
||||
pb2 = proto.load()`. Stubs regenerate automatically when the `.proto` changes
|
||||
(mtime check). `proto.load_grpc()` returns the service stubs (needs the `grpc`
|
||||
runtime).
|
||||
- **Gotcha:** the `.venv` installs the meshnet packages editable via a PEP 660
|
||||
meta-path finder pointing at the **main** checkout. Import the worktree copy by
|
||||
ensuring the worktree `packages/node` is on `sys.path` first (conftest already
|
||||
does this for pytest); standalone tooling must derive paths from `__file__` and
|
||||
not `import meshnet_node` (why `generate_python.py` is self-contained).
|
||||
@@ -1,40 +0,0 @@
|
||||
# DGR-002 reproduction commands (run from repo root, project .venv = Python 3.14).
|
||||
|
||||
# 1. Generate Python stubs (reproducible; writes to gitignored build/ dir).
|
||||
.venv/bin/python packages/node/native/scripts/generate_python.py
|
||||
|
||||
# 2. Python round-trip + compatibility tests (default env; C++ test skips if
|
||||
# cmake/protoc absent).
|
||||
.venv/bin/python -m pytest tests/test_native_shard_protocol.py -q
|
||||
# => 11 passed, 1 skipped
|
||||
|
||||
# 3. Quality gates.
|
||||
.venv/bin/python -m compileall -q packages tests # exit 0
|
||||
git diff --check # clean
|
||||
|
||||
# 4. Full deterministic suite (records pre-existing unrelated failures).
|
||||
.venv/bin/python -m pytest -q
|
||||
# => 704 passed, 14 skipped, 11 failed (all pre-existing, unrelated; see below)
|
||||
|
||||
# 5. Clean-tree reproduction of the 11 pre-existing failures (DGR-002 files moved
|
||||
# aside): same 11 fail => not caused by this story.
|
||||
|
||||
# --- C++ / cross-language (requires protoc + protobuf C++ dev + cmake) --------
|
||||
# On this host a from-source protobuf 33.1 toolchain lives under /tmp/pbsrc/install
|
||||
# and cmake ships in the .venv. To execute the C++ test instead of skipping it:
|
||||
export PATH="/tmp/pbsrc/install/bin:$PWD/.venv/bin:$PATH"
|
||||
export CMAKE_PREFIX_PATH="/tmp/pbsrc/install:$CMAKE_PREFIX_PATH"
|
||||
|
||||
# 6. Generate C++ stubs (message stubs always; gRPC service stubs if
|
||||
# grpc_cpp_plugin present).
|
||||
packages/node/native/scripts/generate_cpp.sh
|
||||
|
||||
# 7. Standalone C++ build + selftest + ctest.
|
||||
cmake -S packages/node/native -B packages/node/native/build/cpp
|
||||
cmake --build packages/node/native/build/cpp --target shard_protocol_roundtrip_test
|
||||
packages/node/native/build/cpp/shard_protocol_roundtrip_test --selftest # "selftest ok (128 bytes)"
|
||||
(cd packages/node/native/build/cpp && ctest --output-on-failure) # 1/1 passed
|
||||
|
||||
# 8. Cross-language Python<->C++ round-trip via the pytest driver (now runs, not skips).
|
||||
.venv/bin/python -m pytest tests/test_native_shard_protocol.py::test_cross_language_roundtrip_python_and_cpp -q
|
||||
# => 1 passed
|
||||
@@ -1,63 +0,0 @@
|
||||
{
|
||||
"task": "DGR-002",
|
||||
"title": "Adopt the versioned gRPC Shard protocol",
|
||||
"schema": {
|
||||
"proto": "packages/node/native/proto/shard_runtime.proto",
|
||||
"package": "meshnet.shard.v1",
|
||||
"syntax": "proto3",
|
||||
"schema_version": 1,
|
||||
"service": "ShardRuntime",
|
||||
"rpcs": ["GetCapability", "Health", "ActivateSession", "Release", "Cancel"],
|
||||
"streaming_seam": "ActivateSession (bidirectional stream)"
|
||||
},
|
||||
"toolchain": {
|
||||
"python": "3.14.6",
|
||||
"protobuf_runtime_python": "7.35.1",
|
||||
"grpcio": "1.82.1",
|
||||
"grpcio_tools": "1.82.1",
|
||||
"cpp_protoc": "libprotoc 33.1",
|
||||
"cpp_protobuf_toolchain": "/tmp/pbsrc/install (from-source protobuf 33.1, ephemeral host build)",
|
||||
"cmake": "4.4.0 (.venv)",
|
||||
"cxx": "g++ (system)"
|
||||
},
|
||||
"generation": {
|
||||
"python_cmd": "python packages/node/native/scripts/generate_python.py",
|
||||
"python_out": "packages/node/native/build/python/shard_runtime_pb2{,_grpc}.py (gitignored)",
|
||||
"cpp_cmd": "packages/node/native/scripts/generate_cpp.sh",
|
||||
"cpp_out": "packages/node/native/build/cpp-gen/shard_runtime.pb.{h,cc} (gitignored)",
|
||||
"cpp_build": "cmake -S packages/node/native -B <build> && cmake --build <build>"
|
||||
},
|
||||
"tests": {
|
||||
"python_default_env": {"passed": 11, "skipped": 1, "note": "C++ cross-language test skips when cmake/protoc absent"},
|
||||
"python_with_cpp_toolchain": {"passed": 12, "skipped": 0},
|
||||
"cpp_selftest_bytes": 128,
|
||||
"cpp_ctest": "1/1 passed",
|
||||
"cross_language": "Python->C++ and C++->Python round-trip verified in both directions"
|
||||
},
|
||||
"quality_gates": {
|
||||
"targeted_pytest": "11 passed, 1 skipped (default); 12 passed with C++ toolchain",
|
||||
"compileall_packages_tests": "exit 0",
|
||||
"git_diff_check": "clean",
|
||||
"full_pytest": {
|
||||
"passed": 704,
|
||||
"skipped": 14,
|
||||
"failed": 11,
|
||||
"failed_are_preexisting_unrelated": true,
|
||||
"clean_tree_reproduction": "same 11 fail with all DGR-002 files removed (11 failed, 3 passed)"
|
||||
}
|
||||
},
|
||||
"preexisting_unrelated_failures": [
|
||||
"tests/test_dynamic_routing.py::test_admin_can_replace_a_served_model_and_release_it",
|
||||
"tests/test_manual_route_benchmark.py::test_pinned_route_uses_named_node",
|
||||
"tests/test_manual_route_benchmark.py::test_unknown_route_node_is_400",
|
||||
"tests/test_manual_route_benchmark.py::test_invalid_route_shape_is_400",
|
||||
"tests/test_manual_route_benchmark.py::test_clients_without_route_are_unaffected",
|
||||
"tests/test_manual_route_benchmark.py::test_benchmark_records_one_and_two_node_routes",
|
||||
"tests/test_toploc_calibration_dispatch.py::test_calibration_run_dispatches_only_solo_capable_nodes",
|
||||
"tests/test_toploc_calibration_dispatch.py::test_calibration_run_persists_corpus_and_results_endpoint_reports_it",
|
||||
"tests/test_toploc_calibration_dispatch.py::test_calibration_run_node_without_commitment_endpoint_is_skipped_not_failed",
|
||||
"tests/test_tracker_routing.py::test_torch_node_applies_tracker_load_shard_directive",
|
||||
"tests/test_tracker_routing.py::test_shard_heal_cycle_surviving_node_covers_dead_peers_gap"
|
||||
],
|
||||
"evidence_kind": "synthetic-unit (schema round-trip + cross-language protobuf; no model, no GPU, no network, no API credits)"
|
||||
}
|
||||
@@ -1,86 +0,0 @@
|
||||
# DGR-003 — Exact artifact and runtime-recipe identity: evidence
|
||||
|
||||
Status: done
|
||||
Date: 2026-07-15
|
||||
Evidence kind: **synthetic-unit + repo checks**. No model download, no GPU, no network, no API credits.
|
||||
|
||||
## Summary
|
||||
|
||||
Implemented exact identity plumbing for shard admission so the node and tracker
|
||||
compare the same compatibility contract:
|
||||
|
||||
- `ArtifactIdentity` binds a shard to an exact source model artifact hash plus
|
||||
shard range.
|
||||
- `RuntimeRecipeIdentity` separates weight quantization, activation dtype,
|
||||
compute dtype, KV dtype/layout, tokenizer revision, architecture adapter,
|
||||
backend id, runtime version, boundary schema version, and cache layout.
|
||||
- `compatibility_fingerprint` is stable SHA-256 over the full artifact/runtime
|
||||
recipe payload.
|
||||
- Node admission and tracker admission now fail closed on compatibility
|
||||
mismatches.
|
||||
- Unsupported recipes remain tracked as dark/unadmitted until a real forward
|
||||
proves them.
|
||||
|
||||
The work also keeps the test helper, doctor path, startup registration payloads,
|
||||
and tracker storage/admission aligned so the same fingerprint is emitted and
|
||||
checked across the system.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `packages/node/meshnet_node/runtime_recipe.py` - new exact artifact/runtime
|
||||
identity helpers and fingerprint builder.
|
||||
- `packages/node/meshnet_node/capability.py` - capability report shape now
|
||||
carries artifact/runtime recipe identity and validates the top-level
|
||||
compatibility fingerprint.
|
||||
- `packages/node/meshnet_node/admission.py` - fail-closed admission on
|
||||
compatibility fingerprint mismatch.
|
||||
- `packages/node/meshnet_node/doctor.py` - production capability reports now
|
||||
include the runtime recipe identity.
|
||||
- `packages/node/meshnet_node/testing.py` - test report builder now mirrors the
|
||||
production fingerprint fields.
|
||||
- `packages/node/meshnet_node/startup.py` - registration payload now includes
|
||||
the compatibility fingerprint.
|
||||
- `packages/tracker/meshnet_tracker/capability.py` - tracker verdict state now
|
||||
stores artifact hash and compatibility fingerprints.
|
||||
- `packages/tracker/meshnet_tracker/server.py` - registration and raft state now
|
||||
preserve declared compatibility fingerprints.
|
||||
- `tests/test_node_capability.py` - identity shape and fingerprint regression
|
||||
tests.
|
||||
- `tests/test_node_admission.py` - fail-closed admission regression tests.
|
||||
- `tests/test_tracker_capability_admission.py` - tracker compatibility mismatch
|
||||
regression tests.
|
||||
|
||||
## Commands and real results
|
||||
|
||||
- `python -m compileall packages tests` -> exit 0.
|
||||
- `pytest -q tests/test_node_capability.py` -> `48 passed in 0.09s`.
|
||||
- `pytest -q tests/test_node_admission.py` -> `20 passed in 0.11s`.
|
||||
- `pytest -q tests/test_tracker_capability_admission.py -k 'compatibility_mismatch or older_recipe_catalogue or unparseable_catalogue_version or future_dated or unknown_schema_version or malformed_report or recorded_detail_carries_no_credentials or compat_policy_routes_a_legacy_node_but_never_a_broken_proof or policy_is_read_from_the_environment_and_defaults_to_compat or route_selection_drops_every_unadmitted_candidate_under_enforce or node_reassigned_to_a_shard_it_never_proved_stops_routing or admitted_candidates_keep_coverage_first_and_throughput_routing'` -> `18 passed, 17 deselected in 0.11s`.
|
||||
- `git diff --check` -> exit 0.
|
||||
- `pytest -q` -> not green in this sandbox. Final result: `210 failed, 423 passed, 13 skipped, 14 warnings, 86 errors in 131.34s`.
|
||||
|
||||
## Limitation
|
||||
|
||||
The full suite is dominated by tracker and HTTP/socket-backed tests. In this
|
||||
sandbox, those fail with `PermissionError: [Errno 1] Operation not permitted`
|
||||
when the tracker attempts to bind a socket. That is an environment restriction,
|
||||
not a regression from the identity work. The pure unit slices above pass.
|
||||
|
||||
## Compatibility notes
|
||||
|
||||
- The compatibility fingerprint is now a hash over the exact artifact identity
|
||||
and runtime recipe payload. It is intended for both node admission and the
|
||||
gRPC handshake admission path.
|
||||
- Default fallbacks for fake/test backends are stable and deterministic: cache
|
||||
layout derives from KV-cache support, architecture adapter falls back to the
|
||||
backend id, and tokenizer identity prefers model revision/model id rather than
|
||||
local tokenizer paths.
|
||||
|
||||
## Handoff for dependent stories
|
||||
|
||||
- DGR-004 / DGR-008 can reuse `runtime_recipe.py` and the compatibility
|
||||
fingerprint to gate the gRPC handshake before session activation.
|
||||
- DGR-009 should transmit the same fingerprint over the relay or preserve it in
|
||||
frame metadata so admission stays aligned end to end.
|
||||
- Any future recipe expansion should register unsupported recipes as dark until
|
||||
a real distributed forward certifies them.
|
||||
@@ -1,130 +0,0 @@
|
||||
# DGR-004 — reproducible pinned llama.cpp patch stack evidence
|
||||
|
||||
Status: done
|
||||
Date: 2026-07-15
|
||||
Evidence kind: **synthetic-build + repo checks**. No model download, no GPU,
|
||||
no network fetch during validation, no API credits.
|
||||
|
||||
## Summary
|
||||
|
||||
Implemented the reproducible source-dependency boundary for llama.cpp and kept
|
||||
the fork seam narrow and auditable:
|
||||
|
||||
- exact pinned upstream commit and repository metadata
|
||||
- numbered patch stack isolated under `packages/node/native/llama/patches/`
|
||||
- build script that verifies the pin, applies the patch stack, stages notices,
|
||||
and compiles a standalone worker scaffold without manual source copying
|
||||
- upstream file assumptions and fail-closed pin checking
|
||||
- license/attribution preservation by staging upstream `LICENSE` and `AUTHORS`
|
||||
- clean rebuild smoke test that only uses a fake local checkout and does not
|
||||
download a model
|
||||
|
||||
The native smoke path is intentionally minimal in this story. It proves the
|
||||
reproducible source dependency and build seam without pulling Meshnet protocol
|
||||
code into llama.cpp.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `packages/node/native/llama/UPSTREAM_COMMIT`
|
||||
- `packages/node/native/llama/UPSTREAM_REPOSITORY`
|
||||
- `packages/node/native/llama/UPSTREAM_ASSUMPTIONS.md`
|
||||
- `packages/node/native/llama/README.md`
|
||||
- `packages/node/native/llama/patches/0001-add-meshnet-worker-scaffold.patch`
|
||||
- `packages/node/native/llama/templates/meshnet_worker.cpp`
|
||||
- `packages/node/native/scripts/build_llama_worker.sh`
|
||||
- `tests/test_llama_worker_build.py`
|
||||
|
||||
## Exact commands and real results
|
||||
|
||||
### Native smoke build against a fake pinned checkout
|
||||
|
||||
```bash
|
||||
tmpdir=$(mktemp -d)
|
||||
mkdir -p "$tmpdir/llama.cpp"
|
||||
printf 'MIT\n' > "$tmpdir/llama.cpp/LICENSE"
|
||||
printf 'AUTHORS\n' > "$tmpdir/llama.cpp/AUTHORS"
|
||||
printf '# placeholder\n' > "$tmpdir/llama.cpp/CMakeLists.txt"
|
||||
printf '%s\n' 'b3c9d1b846cc80a6360adb6aeaa4fcd8c4c8dcac' > "$tmpdir/llama.cpp/.meshnet-upstream-commit"
|
||||
git init -q "$tmpdir/llama.cpp"
|
||||
packages/node/native/scripts/build_llama_worker.sh \
|
||||
--source-dir "$tmpdir/llama.cpp" \
|
||||
--build-dir "$tmpdir/build"
|
||||
```
|
||||
|
||||
Result:
|
||||
|
||||
- `meshnet worker scaffold ok`
|
||||
- `upstream commit: b3c9d1b846cc80a6360adb6aeaa4fcd8c4c8dcac`
|
||||
- `patchset version: 0001`
|
||||
- `build ok: /tmp/.../build/meshnet_worker`
|
||||
|
||||
### Targeted pytest
|
||||
|
||||
```bash
|
||||
python -m pytest -q tests/test_llama_worker_build.py
|
||||
```
|
||||
|
||||
Result: `1 passed in 0.53s`
|
||||
|
||||
### Python compile check
|
||||
|
||||
```bash
|
||||
python -m compileall -q packages tests
|
||||
```
|
||||
|
||||
Result: exit 0
|
||||
|
||||
### Diff hygiene
|
||||
|
||||
```bash
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Result: exit 0
|
||||
|
||||
### Full deterministic pytest
|
||||
|
||||
```bash
|
||||
python -m pytest -q
|
||||
```
|
||||
|
||||
Result: `424 passed, 13 skipped, 210 failed, 86 errors in 131.04s`
|
||||
|
||||
The failures are pre-existing sandbox socket failures in tracker/HTTP-backed
|
||||
tests. Representative error:
|
||||
|
||||
- `PermissionError: [Errno 1] Operation not permitted` when the tracker tries
|
||||
to bind a socket.
|
||||
|
||||
This matches the previously observed environment limitation in the DGR-002 and
|
||||
DGR-003 evidence and is unrelated to the llama.cpp pin/build scaffold.
|
||||
|
||||
## Limitations
|
||||
|
||||
- The sandbox does not provide `cmake`, so the smoke build uses the available
|
||||
direct C++ compiler path (`g++` here) instead of a CMake-generated target.
|
||||
- The pinned upstream source was not fetched from GitHub during validation.
|
||||
The script supports fetching the exact commit when network access is
|
||||
available, but the validation run used a fake local checkout to keep the test
|
||||
deterministic and model-free.
|
||||
- The patch stack in this story is deliberately narrow and additive. It creates
|
||||
a worker scaffold and build seam, not the final llama.cpp runtime patches.
|
||||
|
||||
## Compatibility notes
|
||||
|
||||
- The exact upstream pin is `b3c9d1b846cc80a6360adb6aeaa4fcd8c4c8dcac`.
|
||||
- The build script fails closed if the checkout pin differs from that commit or
|
||||
if the expected upstream files (`LICENSE`, `AUTHORS`, `CMakeLists.txt`) are
|
||||
missing.
|
||||
- The patch stack is isolated from Meshnet networking code and can be applied
|
||||
to a clean pinned checkout before later worker stories extend the scaffold.
|
||||
- Upstream attribution notices are preserved in the build output by copying the
|
||||
staged `LICENSE` and `AUTHORS` files into `build/.../upstream-notices/`.
|
||||
|
||||
## Dependent-story handoff
|
||||
|
||||
- DGR-008 can replace the scaffold source with the real supervised C++ worker
|
||||
while keeping the same pin metadata, patch stack, and build script boundary.
|
||||
- DGR-005 and later native stories should keep using the same exact pin so the
|
||||
worker seam remains reproducible while range-loading and session logic are
|
||||
added.
|
||||
@@ -1,96 +0,0 @@
|
||||
# DGR-005 — dense-Llama range-aware GGUF ownership evidence
|
||||
|
||||
Status: done
|
||||
Date: 2026-07-15
|
||||
Evidence kind: **synthetic-unit + repo checks**. No model download, no GPU, no network, no API credits.
|
||||
|
||||
## Summary
|
||||
|
||||
Implemented range-aware dense-Llama ownership so the node reports and admits only the tensors it actually loads:
|
||||
|
||||
- `blk.N.*` tensors are selected strictly by assigned layer range.
|
||||
- Embeddings are owned at the head only, while final norm / LM head are owned at the tail only, including tied embeddings.
|
||||
- Derivative sub-GGUF slices must carry source and slice hashes and cannot claim final artifact semantics.
|
||||
- The authoritative loaded range and endpoint ownership now come from backend proof state, not CLI shard claims.
|
||||
- Registration, capability reports, admission fingerprints, and tracker state now carry the backend-derived ownership proof.
|
||||
|
||||
The result is a shard model that can reason about memory and admission from owned tensors instead of pretending the full model was loaded.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `packages/node/meshnet_node/gguf_ownership.py` - dense-Llama tensor selection and authoritative ownership helpers.
|
||||
- `packages/node/meshnet_node/capability.py` - shard reports now carry endpoint ownership and parse it round-trip.
|
||||
- `packages/node/meshnet_node/doctor.py` - capability reports now use backend-derived loaded range and endpoint ownership.
|
||||
- `packages/node/meshnet_node/testing.py` - test capability reports now mirror the authoritative ownership path.
|
||||
- `packages/node/meshnet_node/admission.py` - admission compatibility fingerprints now include authoritative range/ownership context.
|
||||
- `packages/node/meshnet_node/model_backend.py` - loaded-range and endpoint-ownership properties on `TorchModelShard`.
|
||||
- `packages/node/meshnet_node/startup.py` - registration payloads now use the proof-driven shard range.
|
||||
- `packages/tracker/meshnet_tracker/capability.py` - tracker capability state preserves endpoint ownership.
|
||||
- `tests/test_gguf_ownership.py` - dense-Llama ownership selection, derivative-slice guard, and memory-scaling tests.
|
||||
- `tests/test_node_capability.py` - capability report ownership round-trip tests.
|
||||
- `tests/test_node_admission.py` - backend-loaded range beats CLI claim regression tests.
|
||||
- `tests/test_tracker_capability_admission.py` - tracker capability proof parsing tests.
|
||||
|
||||
## Exact commands and real results
|
||||
|
||||
### Targeted pytest slices
|
||||
|
||||
```bash
|
||||
python -m pytest -q tests/test_gguf_ownership.py tests/test_node_capability.py tests/test_node_admission.py
|
||||
```
|
||||
|
||||
Result: `73 passed`
|
||||
|
||||
```bash
|
||||
python -m pytest -q tests/test_tracker_capability_admission.py -k 'test_a_passing_report_that_covers_the_registration_is_admitted or test_a_missing_report_is_absent_not_admitted or test_a_failed_report_is_recorded_as_failed or test_a_report_for_a_different_model_is_a_model_mismatch or test_a_report_for_a_different_shard_is_a_shard_mismatch or test_a_report_for_a_different_recipe_than_the_node_declares_is_a_recipe_mismatch or test_a_report_for_a_different_compatibility_fingerprint_is_a_compatibility_mismatch or test_an_older_recipe_catalogue_is_incompatible or test_an_unparseable_catalogue_version_is_incompatible or test_a_stale_report_is_not_admitted or test_a_future_dated_report_is_not_admitted or test_a_report_from_an_unknown_schema_version_is_invalid or test_a_malformed_report_is_invalid_and_never_admitted or test_recorded_detail_carries_no_credentials_from_node_diagnostics or test_compat_policy_routes_a_legacy_node_but_never_a_broken_proof or test_the_policy_is_read_from_the_environment_and_defaults_to_compat'
|
||||
```
|
||||
|
||||
Result: `22 passed, 13 deselected`
|
||||
|
||||
### Python compile check
|
||||
|
||||
```bash
|
||||
python -m compileall -q packages tests
|
||||
```
|
||||
|
||||
Result: exit 0
|
||||
|
||||
### Diff hygiene
|
||||
|
||||
```bash
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Result: exit 0
|
||||
|
||||
### Full deterministic pytest
|
||||
|
||||
```bash
|
||||
python -m pytest -q
|
||||
```
|
||||
|
||||
Result: `211 failed, 428 passed, 13 skipped, 14 warnings, 86 errors in 135.03s`
|
||||
|
||||
The failing set is not caused by this story. The dominant environment issues were:
|
||||
|
||||
- tracker and HTTP/socket-backed tests fail with `PermissionError: [Errno 1] Operation not permitted` when the tracker tries to bind sockets in this sandbox
|
||||
- native protocol tests fail early with a protobuf runtime/gencode mismatch: generated code expects protobuf 7.35.0 while the installed runtime is 6.33.6
|
||||
|
||||
## Limitations
|
||||
|
||||
- This evidence is intentionally deterministic and model-free.
|
||||
- The memory-scaling check is synthetic: it validates that owned tensor bytes scale with selected tensors, not a live GGUF download.
|
||||
- Native C++ code was not changed by this story, so the pinned llama.cpp build validation remains covered by DGR-004 rather than repeated here.
|
||||
|
||||
## Compatibility notes
|
||||
|
||||
- Dense-Llama ownership is range-first: the shard interior is `blk.N.*`, and endpoint tensors are only attributed to the head or tail owner as appropriate.
|
||||
- Derivative GGUF slices are explicitly not final artifacts; they must preserve source and slice hashes if used as a temporary compatibility bridge.
|
||||
- The model proof path is authoritative for reported range and endpoint ownership, so operator CLI claims no longer control what the node advertises.
|
||||
- Admission and tracker state now consume the same proof-derived ownership shape, keeping capability reports aligned end to end.
|
||||
|
||||
## Handoff for dependent stories
|
||||
|
||||
- DGR-006 can reuse `gguf_ownership.py` and the new capability fields to wire the shard protocol to proof-derived ownership without re-deriving tensor names.
|
||||
- DGR-008 and later routing work should continue to treat endpoint ownership as metadata and `blk.N.*` ownership as the core range contract.
|
||||
- If a future temporary slice path is needed, it should keep source/slice hashes visible and avoid claiming final-artifact semantics until a real proof exists.
|
||||
@@ -1,203 +0,0 @@
|
||||
# DGR-006 — Architecture-defined boundary input/output: evidence
|
||||
|
||||
Status: done
|
||||
Date: 2026-07-15
|
||||
Evidence kind: **synthetic-unit** (pure-numpy dense-Llama reference + boundary
|
||||
contract). No model download, no GPU, no torch, no network, no API credit.
|
||||
|
||||
## Summary
|
||||
|
||||
Implemented the architecture-defined boundary contract that lets disjoint Shard
|
||||
processes reproduce whole-model execution (ADR-0024, RALPH runtime decisions #1,
|
||||
#6, #13). A public-network Shard is a contiguous inclusive layer range, and this
|
||||
story defines exactly what boundary state each range consumes and emits:
|
||||
|
||||
- The **head** owns token embedding: it accepts token IDs and produces the
|
||||
residual stream. It refuses an upstream boundary bundle.
|
||||
- **Middle and tail** ranges bypass token embedding entirely and accept the
|
||||
named boundary bundle (the residual stream). They refuse token IDs.
|
||||
- A **non-tail** range emits the *unnormalized* architecture-defined residual —
|
||||
before the final norm, before the LM head, and before any tail-only row
|
||||
pruning — with every sequence position row intact.
|
||||
- The **tail** owns the final norm + LM head, prunes to the final row, and emits
|
||||
a token through an explicit `SamplingContract` (greedy, deterministic).
|
||||
- The adapter **fails closed** for uncertified architectures: only certified
|
||||
dense-Llama spellings are accepted; Qwen3/Qwen3-MoE/Mixtral/gpt2/empty all
|
||||
raise `UncertifiedArchitectureError`.
|
||||
|
||||
The adapter is backend-agnostic: it drives a duck-typed `ShardComputation`
|
||||
(`architecture_adapter`, `start_layer`, `end_layer`, `total_layers`,
|
||||
`embed_tokens`, `run_layers(hidden, *, positions)`, `final_norm`, `lm_head`). A
|
||||
pure-numpy dense-Llama reference (RMSNorm + RoPE + SwiGLU) implements that
|
||||
protocol in the tests and proves whole-model versus two-range **and** three-range
|
||||
prefill + greedy-decode parity. torch/transformers are not installed in the
|
||||
default `.venv`, so a numpy reference is the only way to keep the parity gate
|
||||
deterministic, download-free, and GPU-free — the identical protocol will be
|
||||
satisfied by the pinned llama.cpp worker (DGR-008) and the PyTorch backend.
|
||||
|
||||
No existing runtime code was modified — this story is purely additive (one new
|
||||
module + one new test module). A clean-tree reproduction (files moved aside)
|
||||
confirms the full-suite failure set is byte-identical with and without this work.
|
||||
|
||||
## Files changed (all new)
|
||||
|
||||
- `packages/node/meshnet_node/boundary_adapter.py` — the boundary contract:
|
||||
- `certified_architecture()` / `is_certified_architecture()` and the certified
|
||||
architecture registry (`ArchitectureBoundary`), fail-closed.
|
||||
- `ShardRole` + `role_for_range()` (head/middle/tail/full).
|
||||
- `BoundaryBundle` — the versioned named-tensor bundle carrying the unnormalized
|
||||
residual + positions + seam `next_layer`; `pack()`/`unpack()` for a truly
|
||||
disjoint-process round-trip and `named_tensor_fields()` mapping onto the
|
||||
DGR-002 `NamedTensor` shape (name, shape, dtype, byte order, bytes).
|
||||
- `SamplingContract` — explicit greedy sampling (fails closed on other modes).
|
||||
- `TailOutput` — sampled token + pruned final-row logits + the sampling contract.
|
||||
- `BoundaryAdapter` — enforces the per-role input/output rules and drives the
|
||||
computation.
|
||||
- `tests/test_boundary_adapter.py` — pure-numpy dense-Llama reference model
|
||||
(`_ReferenceDenseLlama`) and range shard (`_ReferenceShard`), plus 22 tests:
|
||||
certification/fail-closed, role classification, input-side contract
|
||||
(head-owns-embedding, middle/tail-bypass, seam-layer mismatch, normalized-bundle
|
||||
rejection), output-side contract (unnormalized full-row boundary, tail pruning +
|
||||
sampling), wire round-trip, and the parity gate.
|
||||
|
||||
## Acceptance criteria → evidence
|
||||
|
||||
- **Head accepts token IDs and owns token embedding** —
|
||||
`test_head_accepts_token_ids_and_owns_embedding`,
|
||||
`BoundaryAdapter._ingest_tokens` (head requires token IDs, refuses a bundle).
|
||||
- **Middle/tail bypass token embedding and accept the named boundary bundle** —
|
||||
`test_middle_and_tail_bypass_embedding_and_require_the_bundle`,
|
||||
`_ingest_boundary` (rejects token IDs, requires the bundle).
|
||||
- **Non-tail emits the unnormalized boundary before final norm/head and before
|
||||
tail-only row pruning** — `test_non_tail_emits_unnormalized_full_row_boundary`
|
||||
asserts the bundle is `normalized=False`, shape `(1, seq, hidden)` (all rows),
|
||||
and byte-equal to the whole model's residual after the cut layer while *not*
|
||||
equal to its normalized form. `_emit_boundary`.
|
||||
- **Tail emits logits/token through an explicit sampling contract** —
|
||||
`test_tail_emits_pruned_logits_through_the_sampling_contract` (logits shape
|
||||
`(1, vocab)` = pruned last row, greedy token = argmax). `_emit_tail`,
|
||||
`SamplingContract`.
|
||||
- **Dense-Llama whole-model vs two-range prefill + greedy-decode parity within
|
||||
tolerance** — `test_two_range_prefill_parity_matches_whole_model`,
|
||||
`test_three_range_prefill_parity_exercises_the_middle_role`,
|
||||
`test_two_range_greedy_decode_parity_matches_whole_model`,
|
||||
`test_alias_architecture_still_parity_matches`. Documented tolerance:
|
||||
next-token logits `np.allclose(..., atol=1e-6)` and **identical** greedy token
|
||||
sequences. (The split is bit-exact in practice; the tolerance is a conservative
|
||||
guard.)
|
||||
- **Fails closed for uncertified architectures** —
|
||||
`test_uncertified_architectures_fail_closed`,
|
||||
`test_adapter_construction_fails_closed_for_uncertified_backend`.
|
||||
- **Targeted pytest** — `22 passed`.
|
||||
- **compileall packages tests** — exit 0.
|
||||
- **git diff --check** — clean.
|
||||
- **Deterministic / download-free / credit-free / GPU-free** — pure numpy; fixed
|
||||
RNG seed; no torch, no network, no model files.
|
||||
- **Full deterministic pytest** — `20 failed, 715 passed, 13 skipped, 12 errors`.
|
||||
All 20 failures + 12 errors are pre-existing and unrelated (see below).
|
||||
- **Native C++ / CTest / llama.cpp patch stack** — **not touched by this story.**
|
||||
The boundary contract is delivered at the Python adapter level with a numpy
|
||||
parity proof; the equivalent native patches ("architecture-defined intermediate
|
||||
input/output" and "intermediate output before final norm/head") are wired when
|
||||
the standalone C++ worker exists in DGR-008. No native code, CMake, or llama.cpp
|
||||
patch was modified, so those gates are N/A here (same as DGR-005).
|
||||
|
||||
## Commands and real results
|
||||
|
||||
```bash
|
||||
# Targeted tests
|
||||
python -m pytest -q tests/test_boundary_adapter.py
|
||||
# -> 22 passed in 0.26s
|
||||
|
||||
# Python compile check
|
||||
python -m compileall -q packages tests
|
||||
# -> exit 0
|
||||
|
||||
# Diff hygiene
|
||||
git diff --check
|
||||
# -> exit 0
|
||||
|
||||
# Full deterministic suite (with DGR-006 files present)
|
||||
python -m pytest -q -rfE
|
||||
# -> 20 failed, 715 passed, 13 skipped, 12 errors in 239.77s
|
||||
|
||||
# Clean-tree reproduction (DGR-006 files moved aside)
|
||||
mv packages/node/meshnet_node/boundary_adapter.py /tmp/ && mv tests/test_boundary_adapter.py /tmp/
|
||||
python -m pytest -q -rfE
|
||||
# -> 20 failed, 693 passed, 13 skipped, 12 errors in 243.10s
|
||||
# (693 = 715 - 22; failure/error SET is byte-identical -> DGR-006 introduced none)
|
||||
```
|
||||
|
||||
The `commands.txt` and `results.json` beside this README capture the exact
|
||||
commands and the machine-readable failure set.
|
||||
|
||||
## Pre-existing unrelated failures (full-suite)
|
||||
|
||||
`pytest -q` on `ralph/distributed-gguf-runtime` reports 20 failures + 12 errors,
|
||||
none of which touch the boundary adapter. Moving the two DGR-006 files aside and
|
||||
re-running yields the **identical** failure/error set (only the passed count drops
|
||||
by exactly 22). Categories:
|
||||
|
||||
- **12 errors — `tests/test_native_shard_protocol.py`:** generated protobuf code
|
||||
expects a newer protobuf runtime than the one installed
|
||||
(`ValidateProtobufRuntimeVersion` mismatch). Pre-existing; documented in the
|
||||
DGR-002 / DGR-005 evidence.
|
||||
- **20 failures** across `test_activation_compression.py`,
|
||||
`test_dynamic_routing.py`, `test_gossip_and_relay.py`,
|
||||
`test_manual_route_benchmark.py`, `test_node_doctor.py`,
|
||||
`test_openai_gateway.py` (`langchain` optional dep),
|
||||
`test_toploc_calibration_dispatch.py`, `test_tracker_capability_admission.py`,
|
||||
`test_tracker_control_plane.py`, `test_tracker_routing.py` — tracker/routing/
|
||||
benchmark/socket-bind + optional-dependency failures that exist on the branch
|
||||
independent of this story.
|
||||
|
||||
## Limitations and deferred work
|
||||
|
||||
- **Numpy reference, not real weights.** The parity gate uses a deterministic
|
||||
numpy dense-Llama, not a downloaded GGUF/safetensors model. Real-model parity on
|
||||
a downloaded dense-Llama (CPU/ROCm) belongs to DGR-010 with
|
||||
`MESHNET_ENABLE_REAL_INFERENCE_TESTS=1` and `.venv-rocm`.
|
||||
- **Stateless decode for parity.** Greedy-decode parity recomputes the growing
|
||||
prefix statelessly (no KV reuse). Local Hot KV State + session isolation is
|
||||
DGR-007; the boundary contract here is KV-agnostic.
|
||||
- **Native patch wiring deferred.** The C++/llama.cpp expression of this boundary
|
||||
(range-aware intermediate I/O, pre-final-norm output) is implemented in the
|
||||
standalone worker (DGR-008) against this same contract; no native code was
|
||||
touched here.
|
||||
- **Greedy-only sampling certified.** `SamplingContract` declares temperature /
|
||||
top-p fields but only certifies `greedy` (deterministic). Stochastic sampling is
|
||||
out of scope for the deterministic parity gate.
|
||||
|
||||
## Compatibility / migration notes
|
||||
|
||||
- `BOUNDARY_SCHEMA_VERSION = 1` matches `runtime_recipe.RuntimeRecipeIdentity`'s
|
||||
`boundary_schema_version`. A receiver rejects a bundle whose schema, architecture
|
||||
adapter, tensor name, normalization flag, or seam `next_layer` does not match its
|
||||
own range — no silent reinterpretation.
|
||||
- `BoundaryBundle.named_tensor_fields()` returns exactly the DGR-002 `NamedTensor`
|
||||
fields (name, shape, dtype, byte order, bytes), so DGR-008 can serialize the seam
|
||||
into the gRPC `TensorBundle` without re-deriving them.
|
||||
- Certified architecture ids are canonicalized: `dense-llama` / `dense_llama` /
|
||||
`llama` / `LlamaForCausalLM` / `LlamaModel` all map to the one `dense-llama`
|
||||
adapter. Adding an architecture requires a new certified entry, never a tensor
|
||||
guess (Qwen3 is DGR-015).
|
||||
|
||||
## Handoff for dependent stories
|
||||
|
||||
- **DGR-007 (Hot KV State):** wrap the same `ShardComputation` so `run_layers`
|
||||
consumes/produces per-session KV; the boundary contract (unnormalized residual,
|
||||
seam `next_layer`, tail pruning) is unchanged. The bundle's `positions` field is
|
||||
the per-token position vector a KV path needs.
|
||||
- **DGR-008 (C++ gRPC worker):** implement the `ShardRuntime` servicer against
|
||||
this contract. Map `BoundaryBundle.named_tensor_fields()` → protobuf
|
||||
`NamedTensor`; enforce the same head-embeds / middle-tail-bypass /
|
||||
non-tail-unnormalized / tail-samples rules in native code; expose
|
||||
`certified_architecture` gating so uncertified GGUFs are refused before activation.
|
||||
- **DGR-009 (Meshnet integration):** carry `BoundaryBundle.pack()` payloads as
|
||||
opaque relay frames; the seam `next_layer` is the overlap-safe effective start
|
||||
the route must honor.
|
||||
- **DGR-010 (real two-process acceptance):** reuse the parity harness shape
|
||||
(whole vs N-range, identical greedy tokens) against a real downloaded dense-Llama
|
||||
under `.venv-rocm`.
|
||||
- **DGR-015 (Qwen3 adapter):** add a certified `ArchitectureBoundary` entry only
|
||||
after real certification; today Qwen3 fails closed by design.
|
||||
@@ -1,26 +0,0 @@
|
||||
# DGR-006 exact commands (run from repo worktree root)
|
||||
|
||||
# Targeted boundary-adapter tests
|
||||
python -m pytest -q tests/test_boundary_adapter.py
|
||||
# -> 22 passed in 0.26s
|
||||
|
||||
# Python compile check for changed Python
|
||||
python -m compileall -q packages tests
|
||||
# -> exit 0
|
||||
|
||||
# Diff hygiene
|
||||
git diff --check
|
||||
# -> exit 0
|
||||
|
||||
# Full deterministic suite with DGR-006 files present
|
||||
python -m pytest -q -rfE
|
||||
# -> 20 failed, 715 passed, 13 skipped, 12 errors in 239.77s
|
||||
|
||||
# Clean-tree reproduction: move the two new DGR-006 files aside, re-run
|
||||
mv packages/node/meshnet_node/boundary_adapter.py /tmp/dgr006_boundary_adapter.py
|
||||
mv tests/test_boundary_adapter.py /tmp/dgr006_test_boundary_adapter.py
|
||||
python -m pytest -q -rfE
|
||||
# -> 20 failed, 693 passed, 13 skipped, 12 errors in 243.10s
|
||||
# (693 = 715 - 22; failure/error set byte-identical to the with-files run)
|
||||
mv /tmp/dgr006_boundary_adapter.py packages/node/meshnet_node/boundary_adapter.py
|
||||
mv /tmp/dgr006_test_boundary_adapter.py tests/test_boundary_adapter.py
|
||||
@@ -1,161 +0,0 @@
|
||||
{
|
||||
"story": "DGR-006",
|
||||
"date": "2026-07-15",
|
||||
"evidence_kind": "synthetic-unit (pure-numpy dense-Llama parity + boundary contract)",
|
||||
"targeted_tests": {
|
||||
"file": "tests/test_boundary_adapter.py",
|
||||
"result": "22 passed"
|
||||
},
|
||||
"compileall": "exit 0",
|
||||
"git_diff_check": "clean",
|
||||
"parity_tolerance": {
|
||||
"logits_atol": 1e-06,
|
||||
"greedy_tokens": "identical"
|
||||
},
|
||||
"full_suite_with_files": {
|
||||
"failed": 20,
|
||||
"passed": 715,
|
||||
"skipped": 13,
|
||||
"errors": 12,
|
||||
"seconds": 239.77
|
||||
},
|
||||
"full_suite_clean_tree": {
|
||||
"failed": 20,
|
||||
"passed": 693,
|
||||
"skipped": 13,
|
||||
"errors": 12,
|
||||
"seconds": 243.1,
|
||||
"note": "693 = 715 - 22 DGR-006 tests; failure/error set identical"
|
||||
},
|
||||
"failure_set_identical_with_and_without_dgr006": true,
|
||||
"preexisting_unrelated_failures": [
|
||||
{
|
||||
"kind": "ERROR",
|
||||
"nodeid": "tests/test_native_shard_protocol.py::test_capability_and_health_round_trip"
|
||||
},
|
||||
{
|
||||
"kind": "ERROR",
|
||||
"nodeid": "tests/test_native_shard_protocol.py::test_checksum_algorithms_verify"
|
||||
},
|
||||
{
|
||||
"kind": "ERROR",
|
||||
"nodeid": "tests/test_native_shard_protocol.py::test_cross_language_roundtrip_python_and_cpp"
|
||||
},
|
||||
{
|
||||
"kind": "ERROR",
|
||||
"nodeid": "tests/test_native_shard_protocol.py::test_defaults_are_stable_for_backward_compatibility"
|
||||
},
|
||||
{
|
||||
"kind": "ERROR",
|
||||
"nodeid": "tests/test_native_shard_protocol.py::test_fragment_and_reassemble_round_trip_with_checksums"
|
||||
},
|
||||
{
|
||||
"kind": "ERROR",
|
||||
"nodeid": "tests/test_native_shard_protocol.py::test_message_header_carries_every_required_field"
|
||||
},
|
||||
{
|
||||
"kind": "ERROR",
|
||||
"nodeid": "tests/test_native_shard_protocol.py::test_named_tensor_bundle_describes_shape_dtype_byteorder_and_fragments"
|
||||
},
|
||||
{
|
||||
"kind": "ERROR",
|
||||
"nodeid": "tests/test_native_shard_protocol.py::test_reassemble_detects_fragment_corruption"
|
||||
},
|
||||
{
|
||||
"kind": "ERROR",
|
||||
"nodeid": "tests/test_native_shard_protocol.py::test_service_descriptor_exposes_all_operations"
|
||||
},
|
||||
{
|
||||
"kind": "ERROR",
|
||||
"nodeid": "tests/test_native_shard_protocol.py::test_session_response_carries_structured_status_and_results"
|
||||
},
|
||||
{
|
||||
"kind": "ERROR",
|
||||
"nodeid": "tests/test_native_shard_protocol.py::test_session_stream_carries_open_prefill_decode_release_cancel"
|
||||
},
|
||||
{
|
||||
"kind": "ERROR",
|
||||
"nodeid": "tests/test_native_shard_protocol.py::test_unknown_fields_are_preserved_for_forward_compatibility"
|
||||
},
|
||||
{
|
||||
"kind": "FAILED",
|
||||
"nodeid": "tests/test_activation_compression.py::test_compressible_body_uses_zstd_when_it_clears_savings_policy"
|
||||
},
|
||||
{
|
||||
"kind": "FAILED",
|
||||
"nodeid": "tests/test_activation_compression.py::test_incompressible_body_stays_raw_after_measured_trial"
|
||||
},
|
||||
{
|
||||
"kind": "FAILED",
|
||||
"nodeid": "tests/test_activation_compression.py::test_malformed_zstd_and_legacy_raw_bodies_are_handled_explicitly"
|
||||
},
|
||||
{
|
||||
"kind": "FAILED",
|
||||
"nodeid": "tests/test_activation_compression.py::test_threshold_requires_both_byte_and_ratio_savings"
|
||||
},
|
||||
{
|
||||
"kind": "FAILED",
|
||||
"nodeid": "tests/test_dynamic_routing.py::test_admin_can_replace_a_served_model_and_release_it"
|
||||
},
|
||||
{
|
||||
"kind": "FAILED",
|
||||
"nodeid": "tests/test_gossip_and_relay.py::test_activation_compression_round_trips_and_skips_small_bodies"
|
||||
},
|
||||
{
|
||||
"kind": "FAILED",
|
||||
"nodeid": "tests/test_manual_route_benchmark.py::test_benchmark_records_one_and_two_node_routes"
|
||||
},
|
||||
{
|
||||
"kind": "FAILED",
|
||||
"nodeid": "tests/test_manual_route_benchmark.py::test_clients_without_route_are_unaffected"
|
||||
},
|
||||
{
|
||||
"kind": "FAILED",
|
||||
"nodeid": "tests/test_manual_route_benchmark.py::test_invalid_route_shape_is_400"
|
||||
},
|
||||
{
|
||||
"kind": "FAILED",
|
||||
"nodeid": "tests/test_manual_route_benchmark.py::test_pinned_route_uses_named_node"
|
||||
},
|
||||
{
|
||||
"kind": "FAILED",
|
||||
"nodeid": "tests/test_manual_route_benchmark.py::test_unknown_route_node_is_400"
|
||||
},
|
||||
{
|
||||
"kind": "FAILED",
|
||||
"nodeid": "tests/test_node_doctor.py::test_cli_doctor_flags_select_what_is_validated"
|
||||
},
|
||||
{
|
||||
"kind": "FAILED",
|
||||
"nodeid": "tests/test_openai_gateway.py::test_langchain_chat_openai"
|
||||
},
|
||||
{
|
||||
"kind": "FAILED",
|
||||
"nodeid": "tests/test_toploc_calibration_dispatch.py::test_calibration_run_dispatches_only_solo_capable_nodes"
|
||||
},
|
||||
{
|
||||
"kind": "FAILED",
|
||||
"nodeid": "tests/test_toploc_calibration_dispatch.py::test_calibration_run_node_without_commitment_endpoint_is_skipped_not_failed"
|
||||
},
|
||||
{
|
||||
"kind": "FAILED",
|
||||
"nodeid": "tests/test_toploc_calibration_dispatch.py::test_calibration_run_persists_corpus_and_results_endpoint_reports_it"
|
||||
},
|
||||
{
|
||||
"kind": "FAILED",
|
||||
"nodeid": "tests/test_tracker_capability_admission.py::test_an_enforcing_tracker_never_routes_a_node_whose_proof_does_not_cover_it[invalid]"
|
||||
},
|
||||
{
|
||||
"kind": "FAILED",
|
||||
"nodeid": "tests/test_tracker_control_plane.py::test_tracker_startup_does_not_import_or_load_model_backends"
|
||||
},
|
||||
{
|
||||
"kind": "FAILED",
|
||||
"nodeid": "tests/test_tracker_routing.py::test_shard_heal_cycle_surviving_node_covers_dead_peers_gap"
|
||||
},
|
||||
{
|
||||
"kind": "FAILED",
|
||||
"nodeid": "tests/test_tracker_routing.py::test_torch_node_applies_tracker_load_shard_directive"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -1,229 +0,0 @@
|
||||
# DGR-007 — Isolated concurrent local Hot KV State: evidence
|
||||
|
||||
Status: done
|
||||
Date: 2026-07-15
|
||||
Evidence kind: **synthetic-unit** (pure-numpy KV-cached dense-Llama reference +
|
||||
session/KV manager). No model download, no GPU, no torch, no network, no API
|
||||
credit.
|
||||
|
||||
## Summary
|
||||
|
||||
Implemented the local Hot KV State manager that maps every
|
||||
`(Route Session ID, route epoch)` to an isolated, bounded KV context (RALPH
|
||||
runtime decisions #7 and #8, ADR-0022/0024). The manager owns all cache
|
||||
mutation, so eviction, byte accounting, and isolation live in one place instead
|
||||
of being scattered across backends:
|
||||
|
||||
- **`(session_id, route_epoch)` → isolated context.** Each key gets its own
|
||||
`SessionCache` holding independent per-layer K/V; one session can never read or
|
||||
clear another's state.
|
||||
- **KV allocated only for owned layers.** A shard constructed for range
|
||||
`[start, end]` allocates a `LayerKvCache` for exactly those layer indices; a
|
||||
middle shard `[2,3]` holds `{2,3}` and nothing else.
|
||||
- **Full lifecycle:** prefill append, decode append, truncate (rollback),
|
||||
release, TTL eviction, LRU eviction (by session cap and by byte budget), and an
|
||||
**explicit** `CacheMiss` (unknown-session / evicted-ttl / evicted-lru /
|
||||
released / superseded-epoch / seq-len-mismatch) so the head degrades to a
|
||||
from-token-zero re-prefill instead of corrupting output (decision #14).
|
||||
- **Fails closed on identity.** Stale route epochs raise `StaleRouteEpochError`; a
|
||||
request carrying an incompatible KV recipe raises `IncompatibleCacheRecipeError`
|
||||
(fingerprint mismatch of architecture / kv dtype / head geometry / owned range);
|
||||
a recipe for an uncertified architecture fails closed at construction (reusing
|
||||
the DGR-006 certified-architecture gate).
|
||||
- **KV-aware boundary driver.** `KvBoundaryAdapter` wraps the DGR-006
|
||||
`ShardComputation` (plus `run_layers_cached`) so a shard runs cached
|
||||
prefill/decode through the manager while honouring the architecture-defined
|
||||
boundary contract (head embeds tokens, middle/tail bypass embedding and consume
|
||||
the unnormalized residual bundle, non-tail emits the unnormalized residual, tail
|
||||
normalizes + heads + prunes + samples). The computation returns the new
|
||||
position-encoded K/V; the manager commits it under the budget.
|
||||
|
||||
A pure-numpy **KV-cached** dense-Llama reference (RMSNorm + RoPE + SwiGLU with an
|
||||
absolute-position causal mask over cached keys) proves that cached prefill/decode
|
||||
reproduces the stateless whole-model greedy tokens bit-for-bit, single-range and
|
||||
across a head/tail seam. torch/transformers are not installed in the default
|
||||
`.venv`, so a numpy reference is the only way to keep the parity + isolation gate
|
||||
deterministic, download-free, and GPU-free — the identical manager contract will
|
||||
be satisfied by the pinned llama.cpp worker (DGR-008), where the KV context maps
|
||||
onto a llama sequence.
|
||||
|
||||
No existing runtime code was modified — this story is purely additive (one new
|
||||
module + one new test module).
|
||||
|
||||
## Files changed (all new)
|
||||
|
||||
- `packages/node/meshnet_node/hot_kv_state.py` — the KV/session manager:
|
||||
- `KvCacheRecipe` — KV layout identity (certified architecture, kv dtype, head
|
||||
geometry, owned range) with `fingerprint()` / `is_compatible()` /
|
||||
`bytes_per_token()`; fails closed on uncertified architectures.
|
||||
- `LayerKvCache` — per-owned-layer `(seq, n_kv_heads, head_dim)` K/V with
|
||||
`append` / `truncate` / `nbytes`.
|
||||
- `SessionCache` — the isolated per-`(session, epoch)` context over owned layers.
|
||||
- `CacheMiss` / `CacheMissReason` — the explicit, serializable miss response.
|
||||
- `HotKvStateManager` — `open` / `append` / `truncate` / `release` / `resolve` /
|
||||
`get`, LRU+TTL+byte-budget eviction, stale-epoch + incompatible-recipe
|
||||
rejection, epoch supersession, thread-safe (RLock), injectable clock.
|
||||
- `KvBoundaryAdapter` + `kv_recipe_for()` — KV-aware boundary driver.
|
||||
- `tests/test_hot_kv_state.py` — pure-numpy KV-cached dense-Llama reference and 22
|
||||
tests (see below).
|
||||
|
||||
## Acceptance criteria → evidence
|
||||
|
||||
- **Map `(Route Session ID, route epoch)` to an isolated context** —
|
||||
`test_prefill_then_decode_append_grows_owned_layers`,
|
||||
`test_four_interleaved_sessions_have_no_kv_cross_talk`,
|
||||
`HotKvStateManager.open` keys sessions on `(session_id, route_epoch)`.
|
||||
- **Allocate KV only for owned layers** —
|
||||
`test_manager_allocates_kv_only_for_owned_layers` (middle `[2,3]` → `{2,3}`),
|
||||
`test_multi_range_cached_decode_parity_across_a_seam` (head owns `(0,1,2)`, tail
|
||||
owns `(3,4,5)`), `test_recipe_bytes_per_token_scales_with_owned_layers`.
|
||||
- **Prefill append / decode append / truncate / release / TTL-LRU eviction /
|
||||
explicit cache-miss** — `test_prefill_then_decode_append_grows_owned_layers`,
|
||||
`test_truncate_rolls_back_all_owned_layers`,
|
||||
`test_release_one_session_leaves_others_intact_and_returns_memory`,
|
||||
`test_ttl_eviction_yields_an_explicit_cache_miss`,
|
||||
`test_lru_eviction_by_session_cap_reports_a_miss`,
|
||||
`test_budget_eviction_keeps_total_within_budget`,
|
||||
`test_unknown_session_is_an_explicit_cache_miss`,
|
||||
`test_seq_len_mismatch_is_an_explicit_cache_miss`.
|
||||
- **Reject stale epochs and incompatible cache recipes** —
|
||||
`test_stale_route_epoch_is_rejected`,
|
||||
`test_new_route_epoch_supersedes_and_frees_old_epoch`,
|
||||
`test_incompatible_cache_recipe_is_rejected`,
|
||||
`test_uncertified_architecture_recipe_fails_closed`.
|
||||
- **≥ four concurrent sessions complete without token or KV cross-talk** —
|
||||
`test_four_interleaved_sessions_have_no_kv_cross_talk` (four interleaved
|
||||
round-robin sessions, four *distinct* references, each matches its own),
|
||||
`test_four_sessions_on_real_threads_stay_isolated` (four OS threads).
|
||||
- **Cancellation/release leaves others intact and memory returns to budget** —
|
||||
`test_release_one_session_leaves_others_intact_and_returns_memory` (released
|
||||
session → `CacheMiss(RELEASED)`, `total_bytes` drops, survivors keep matching
|
||||
their references), `test_single_session_exceeding_budget_raises`.
|
||||
- **Cached vs stateless correctness core** —
|
||||
`test_cached_full_shard_decode_matches_stateless_whole_model`,
|
||||
`test_cached_prefill_next_token_matches_whole_model_logits`,
|
||||
`test_multi_range_cached_decode_parity_across_a_seam`. Documented tolerance:
|
||||
**identical** greedy token ids (bit-exact in practice; cached incremental
|
||||
attention equals stateless full-sequence recompute per query row).
|
||||
- **Targeted pytest** — `22 passed`.
|
||||
- **compileall packages tests** — exit 0.
|
||||
- **git diff --check** — clean.
|
||||
- **Deterministic / download-free / credit-free / GPU-free** — pure numpy; fixed
|
||||
RNG seed; injectable clock (no wall-clock in tests); no torch, no network, no
|
||||
model files.
|
||||
- **Full deterministic pytest** — `13 failed, 755 passed, 14 skipped in 254.50s`.
|
||||
All 13 failures are pre-existing and unrelated; the clean-tree reproduction
|
||||
(DGR-007 files moved aside) gives the **identical** 13-failure set with `733
|
||||
passed` (exactly −22), so this story introduces no new failures.
|
||||
- **Native C++ / CTest / llama.cpp patch stack** — **not touched by this story.**
|
||||
The KV context contract is delivered at the Python manager level with a numpy
|
||||
parity + isolation proof; the equivalent native layer-filtered KV / session
|
||||
mapping is wired when the standalone C++ worker exists in DGR-008. No native
|
||||
code, CMake, or llama.cpp patch was modified, so those gates are N/A here (same
|
||||
as DGR-005/006).
|
||||
|
||||
## Commands and real results
|
||||
|
||||
```bash
|
||||
VP=/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python
|
||||
|
||||
$VP -m pytest -q tests/test_hot_kv_state.py
|
||||
# -> 22 passed in ~0.3s
|
||||
|
||||
$VP -m compileall -q packages tests
|
||||
# -> exit 0
|
||||
|
||||
git diff --check
|
||||
# -> exit 0
|
||||
|
||||
$VP -m pytest -q tests/test_boundary_adapter.py tests/test_gguf_ownership.py
|
||||
# -> 25 passed
|
||||
|
||||
$VP -m pytest -q -rfE
|
||||
# -> 13 failed, 755 passed, 14 skipped in 254.50s
|
||||
|
||||
# Clean-tree reproduction (DGR-007 files moved aside)
|
||||
mv packages/node/meshnet_node/hot_kv_state.py /tmp/ && mv tests/test_hot_kv_state.py /tmp/
|
||||
$VP -m pytest -q -rfE
|
||||
# -> 13 failed, 733 passed, 14 skipped in 252.12s (identical FAILED set; passed -22)
|
||||
```
|
||||
|
||||
`commands.txt` beside this README captures the exact commands.
|
||||
|
||||
## Pre-existing unrelated failures (full-suite)
|
||||
|
||||
`pytest -q -rfE` on `ralph/distributed-gguf-runtime` reports 13 pre-existing
|
||||
failures (and, in this run, 0 errors — the earlier DGR-005/006-era
|
||||
`test_native_shard_protocol.py` protobuf errors no longer appear in this
|
||||
environment). None touch the KV manager. Moving the two DGR-007 files aside and
|
||||
re-running yields the **byte-identical** 13-`FAILED` set (only the passed count
|
||||
drops by exactly 22). The exact set (all tracker/routing/benchmark/toploc/doctor,
|
||||
i.e. socket-bind / control-plane env, not KV):
|
||||
|
||||
```
|
||||
tests/test_dynamic_routing.py::test_admin_can_replace_a_served_model_and_release_it
|
||||
tests/test_manual_route_benchmark.py::test_benchmark_records_one_and_two_node_routes
|
||||
tests/test_manual_route_benchmark.py::test_clients_without_route_are_unaffected
|
||||
tests/test_manual_route_benchmark.py::test_invalid_route_shape_is_400
|
||||
tests/test_manual_route_benchmark.py::test_pinned_route_uses_named_node
|
||||
tests/test_manual_route_benchmark.py::test_unknown_route_node_is_400
|
||||
tests/test_node_doctor.py::test_cli_doctor_flags_select_what_is_validated
|
||||
tests/test_toploc_calibration_dispatch.py::test_calibration_run_dispatches_only_solo_capable_nodes
|
||||
tests/test_toploc_calibration_dispatch.py::test_calibration_run_node_without_commitment_endpoint_is_skipped_not_failed
|
||||
tests/test_toploc_calibration_dispatch.py::test_calibration_run_persists_corpus_and_results_endpoint_reports_it
|
||||
tests/test_tracker_capability_admission.py::test_an_enforcing_tracker_never_routes_a_node_whose_proof_does_not_cover_it[invalid]
|
||||
tests/test_tracker_routing.py::test_shard_heal_cycle_surviving_node_covers_dead_peers_gap
|
||||
tests/test_tracker_routing.py::test_torch_node_applies_tracker_load_shard_directive
|
||||
```
|
||||
|
||||
## Limitations and deferred work
|
||||
|
||||
- **Numpy reference, not real weights.** The parity + isolation gate uses a
|
||||
deterministic numpy KV-cached dense-Llama, not a downloaded GGUF/safetensors
|
||||
model. Real-model concurrent KV isolation on a downloaded dense-Llama (CPU/ROCm)
|
||||
belongs to DGR-010/DGR-012 with `MESHNET_ENABLE_REAL_INFERENCE_TESTS=1` and
|
||||
`.venv-rocm`.
|
||||
- **Manager-owned storage, native mapping deferred.** The KV bytes are numpy
|
||||
arrays managed in-process. The llama.cpp expression (a filtered llama sequence
|
||||
per `(session, epoch)` over owned layers) is implemented in the standalone
|
||||
worker (DGR-008) against this same manager contract; no native code was touched.
|
||||
- **Continuous batching is DGR-012.** This story delivers *isolation* and bounded
|
||||
lifecycle for concurrent sessions; continuous batching of compatible active
|
||||
sessions inside a node (decision #9) is DGR-012 and builds on this manager.
|
||||
- **Greedy-only sampling.** Reuses the DGR-006 `SamplingContract` (greedy
|
||||
certified). Stochastic sampling is out of scope for the deterministic gate.
|
||||
- **Coexists with legacy `SessionCacheStore`.** The older AH-25
|
||||
`model_backend.SessionCacheStore` (session-id-only, opaque transformers cache,
|
||||
HTTP path) is untouched. `HotKvStateManager` is the native-runtime-aligned
|
||||
successor: it adds route-epoch keying, owned-layer allocation, recipe-fingerprint
|
||||
rejection, and a byte budget. DGR-008/009 wire the native worker to
|
||||
`HotKvStateManager`, not `SessionCacheStore`.
|
||||
|
||||
## Compatibility / migration notes
|
||||
|
||||
- `KvCacheRecipe.fingerprint()` canonicalizes the architecture (via
|
||||
`certified_architecture`), so `llama` / `LlamaForCausalLM` map to the same
|
||||
recipe; it aligns field-for-field with the DGR-003 `RuntimeRecipeIdentity`
|
||||
compatibility discipline and reuses `runtime_recipe.compatibility_fingerprint`.
|
||||
- `CacheMiss` is a value (not an exception) so it can be serialized into the
|
||||
DGR-002 native protocol's cache expectation/result field; `resolve()` returns it,
|
||||
`get()` raises `KvCacheMissError` wrapping it.
|
||||
- The manager takes an injectable `clock` for deterministic TTL tests; production
|
||||
defaults to `time.monotonic`.
|
||||
|
||||
## Handoff for dependent stories
|
||||
|
||||
- **DGR-008 (C++ gRPC worker):** implement the servicer's KV path against
|
||||
`HotKvStateManager`. Map each `(Route Session ID, route epoch)` to a filtered
|
||||
llama sequence over owned layers; on decode, read the sequence's cached K/V,
|
||||
compute the new position-encoded K/V, and commit via `append` (honour the byte
|
||||
budget and return an explicit `CacheMiss` on eviction). Enforce
|
||||
`KvCacheRecipe.is_compatible` before activation and reject stale epochs.
|
||||
- **DGR-009 (Meshnet integration):** the route epoch the tracker assigns is the
|
||||
`route_epoch` key; carry the `CacheMiss` reason back to the head so it re-prefills
|
||||
from token zero on eviction/restart.
|
||||
- **DGR-012 (continuous batching):** batch compatible active sessions whose
|
||||
`KvCacheRecipe` fingerprints match; each session keeps its own `SessionCache`, so
|
||||
batching is a scheduling concern layered over this isolation, not a change to it.
|
||||
- **DGR-013 (failure/cancel matrix):** `release` + the budget-return assertion here
|
||||
is the unit-level basis for the resource-cleanup matrix.
|
||||
@@ -1,31 +0,0 @@
|
||||
# DGR-007 — exact commands (run from the worktree root).
|
||||
# Python: /run/media/popov/d/DEV/repos/d-popov.com/AI/.venv (Python 3.14.6, numpy 2.4.4).
|
||||
# Root conftest.py adds packages/* to sys.path, so `meshnet_node` imports work.
|
||||
|
||||
VP=/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python
|
||||
|
||||
# Targeted tests for this story.
|
||||
$VP -m pytest -q tests/test_hot_kv_state.py
|
||||
# -> 22 passed
|
||||
|
||||
# Python compile check for the changed packages/tests.
|
||||
$VP -m compileall -q packages tests
|
||||
# -> exit 0
|
||||
|
||||
# Diff hygiene.
|
||||
git diff --check
|
||||
# -> exit 0
|
||||
|
||||
# Dependency (DGR-006) + range-ownership (DGR-005) tests still green.
|
||||
$VP -m pytest -q tests/test_boundary_adapter.py tests/test_gguf_ownership.py
|
||||
# -> 25 passed
|
||||
|
||||
# Full deterministic suite (with DGR-007 files present).
|
||||
$VP -m pytest -q -rfE
|
||||
# -> see README (pre-existing unrelated failure set, +22 passed vs baseline)
|
||||
|
||||
# Clean-tree reproduction (DGR-007 files moved aside).
|
||||
mv packages/node/meshnet_node/hot_kv_state.py /tmp/ && mv tests/test_hot_kv_state.py /tmp/
|
||||
$VP -m pytest -q -rfE
|
||||
# -> identical failure/error set, passed count drops by exactly 22
|
||||
mv /tmp/hot_kv_state.py packages/node/meshnet_node/ && mv /tmp/test_hot_kv_state.py tests/
|
||||
@@ -1,47 +0,0 @@
|
||||
{
|
||||
"task_id": "DGR-007",
|
||||
"title": "Add isolated concurrent local Hot KV State",
|
||||
"status": "done",
|
||||
"date": "2026-07-15",
|
||||
"evidence_kind": "synthetic-unit",
|
||||
"python": "/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv (Python 3.14.6, numpy 2.4.4)",
|
||||
"files_changed": [
|
||||
"packages/node/meshnet_node/hot_kv_state.py",
|
||||
"tests/test_hot_kv_state.py"
|
||||
],
|
||||
"gates": {
|
||||
"targeted_pytest": {"command": "pytest -q tests/test_hot_kv_state.py", "result": "22 passed"},
|
||||
"compileall": {"command": "python -m compileall -q packages tests", "exit": 0},
|
||||
"git_diff_check": {"command": "git diff --check", "exit": 0},
|
||||
"dependency_tests": {"command": "pytest -q tests/test_boundary_adapter.py tests/test_gguf_ownership.py", "result": "25 passed"},
|
||||
"full_suite_with_files": {"command": "pytest -q -rfE", "result": "13 failed, 755 passed, 14 skipped", "seconds": 254.50},
|
||||
"full_suite_clean_tree": {"command": "pytest -q -rfE (DGR-007 files moved aside)", "result": "13 failed, 733 passed, 14 skipped", "seconds": 252.12}
|
||||
},
|
||||
"no_new_failures": true,
|
||||
"failure_set_identical": true,
|
||||
"passed_delta": 22,
|
||||
"preexisting_failures": [
|
||||
"tests/test_dynamic_routing.py::test_admin_can_replace_a_served_model_and_release_it",
|
||||
"tests/test_manual_route_benchmark.py::test_benchmark_records_one_and_two_node_routes",
|
||||
"tests/test_manual_route_benchmark.py::test_clients_without_route_are_unaffected",
|
||||
"tests/test_manual_route_benchmark.py::test_invalid_route_shape_is_400",
|
||||
"tests/test_manual_route_benchmark.py::test_pinned_route_uses_named_node",
|
||||
"tests/test_manual_route_benchmark.py::test_unknown_route_node_is_400",
|
||||
"tests/test_node_doctor.py::test_cli_doctor_flags_select_what_is_validated",
|
||||
"tests/test_toploc_calibration_dispatch.py::test_calibration_run_dispatches_only_solo_capable_nodes",
|
||||
"tests/test_toploc_calibration_dispatch.py::test_calibration_run_node_without_commitment_endpoint_is_skipped_not_failed",
|
||||
"tests/test_toploc_calibration_dispatch.py::test_calibration_run_persists_corpus_and_results_endpoint_reports_it",
|
||||
"tests/test_tracker_capability_admission.py::test_an_enforcing_tracker_never_routes_a_node_whose_proof_does_not_cover_it[invalid]",
|
||||
"tests/test_tracker_routing.py::test_shard_heal_cycle_surviving_node_covers_dead_peers_gap",
|
||||
"tests/test_tracker_routing.py::test_torch_node_applies_tracker_load_shard_directive"
|
||||
],
|
||||
"native_gates_touched": false,
|
||||
"acceptance": {
|
||||
"session_epoch_isolated_context": true,
|
||||
"kv_only_owned_layers": true,
|
||||
"prefill_decode_truncate_release_ttl_lru_cachemiss": true,
|
||||
"reject_stale_epoch_and_incompatible_recipe": true,
|
||||
"four_concurrent_sessions_no_crosstalk": true,
|
||||
"release_leaves_others_and_returns_memory": true
|
||||
}
|
||||
}
|
||||
@@ -1,83 +0,0 @@
|
||||
# DGR-009 — Integrate the native worker with Meshnet: evidence
|
||||
|
||||
Status: done
|
||||
Date: 2026-07-15
|
||||
Evidence kind: **python-unit + repo-hygiene**. No model download, no GPU, no API
|
||||
credit.
|
||||
|
||||
## Summary
|
||||
|
||||
Implemented the Meshnet-facing GGUF backend seam and recipe gating needed for
|
||||
the native worker path:
|
||||
|
||||
- Added `GgufNodeBackend`, a backend-shaped adapter that lets the existing node
|
||||
HTTP/control-plane code serve GGUF-backed shards without changing the
|
||||
Transformers/Torch path for the default recipes.
|
||||
- Added `llama-cpp-native` to the recipe manifest and gated startup so only
|
||||
recipes with `backend_id == "llama.cpp"` build the GGUF backend.
|
||||
- Preserved the existing registration/admission flow by carrying the validated
|
||||
capability report and proof shard through registration.
|
||||
- Added unit coverage for the GGUF backend seam and for recipe-gated startup.
|
||||
- Fixed the explicit-shard startup path so the legacy Torch tests that use an
|
||||
opaque stub model still pass without requiring HuggingFace config discovery.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `packages/node/meshnet_node/gguf_backend.py` - new GGUF backend adapter and
|
||||
worker-transport boundary.
|
||||
- `packages/node/meshnet_node/startup.py` - recipe-gated GGUF backend injection
|
||||
and explicit-shard startup fix.
|
||||
- `packages/node/meshnet_node/recipes.json` - added `llama-cpp-native`.
|
||||
- `tests/test_gguf_backend.py` - backend delegation and recipe-selection tests.
|
||||
- `.ralph-tui/progress.md` - appended DGR-009 progress note.
|
||||
- `.scratch/distributed-gguf-runtime/issues/09-integrate-the-native-worker-with-meshnet.md`
|
||||
- marked `Status: done`.
|
||||
|
||||
## Commands and real results
|
||||
|
||||
```bash
|
||||
python -m pytest -q tests/test_gguf_backend.py
|
||||
# -> 2 passed in 0.05s
|
||||
|
||||
python -m pytest -q tests/test_node_admission.py::test_the_served_backend_is_loaded_with_the_recipe_that_was_validated tests/test_node_admission.py::test_backend_validation_failure_registers_nothing
|
||||
# -> 2 passed in 0.07s
|
||||
|
||||
python -m compileall -q packages tests
|
||||
# -> exit 0
|
||||
|
||||
git diff --check
|
||||
# -> exit 0
|
||||
|
||||
python -m pytest -q
|
||||
# -> 222 failed, 463 passed, 13 skipped, 86 errors in 135.65s
|
||||
```
|
||||
|
||||
## Limitations
|
||||
|
||||
- `python -m pytest -q` is still not clean in this sandbox. The dominant
|
||||
failures are tracker/control-plane socket `PermissionError: [Errno 1]
|
||||
Operation not permitted` and a native protocol import failure caused by a
|
||||
protobuf runtime mismatch (`gencode 7.35.0` vs runtime `6.33.6`).
|
||||
- `tests/test_native_shard_protocol.py` currently fails for the same protobuf
|
||||
runtime mismatch in this environment.
|
||||
- `DGR-008` evidence was not present in the tree, so the dependency behavior was
|
||||
verified by reading the live code and exercising the Python seam instead of
|
||||
relying on a missing README.
|
||||
|
||||
## Compatibility notes
|
||||
|
||||
- The default Torch path remains intact; GGUF backend selection is explicit and
|
||||
recipe-gated.
|
||||
- `TorchNodeServer` already accepts an injected backend object, so the control
|
||||
plane stays Meshnet-owned.
|
||||
- The GGUF adapter currently establishes the seam for the native worker
|
||||
transport; the compiled worker remains the owner of the gRPC protocol details.
|
||||
|
||||
## Dependent-story handoff
|
||||
|
||||
- DGR-008 should continue to own the native worker implementation and the
|
||||
versioned gRPC frame handling behind `MESHNET_NATIVE_WORKER_URL`.
|
||||
- DGR-010 / DGR-012 can build on this seam without changing the control plane:
|
||||
the recipe-gated backend and validated capability report are already carried
|
||||
through startup.
|
||||
|
||||
@@ -1,58 +0,0 @@
|
||||
# DGR-010 — Blocked handoff
|
||||
|
||||
Status: blocked
|
||||
Date: 2026-07-15
|
||||
|
||||
## Blocker
|
||||
|
||||
I verified the local workspace and mounted-drive model storage, but there is no
|
||||
certified dense-Llama artifact available on this machine to run the required
|
||||
real-model two-process acceptance.
|
||||
|
||||
What I found:
|
||||
|
||||
- `/run/media/popov/d/DEV/models` contains Qwen artifacts and caches, but no
|
||||
dense-Llama model snapshot or GGUF artifact.
|
||||
- `/run/media/popov/d/DEV/llamacpp/llama.cpp/models` contains only vocab GGUFs,
|
||||
not a certified dense-Llama model.
|
||||
- The existing code paths for real startup, GGUF backend selection, Hot KV
|
||||
isolation, and benchmark reporting are present and readable, but the actual
|
||||
DGR-010 acceptance run needs a certified dense-Llama artifact from mounted
|
||||
storage to satisfy the story contract.
|
||||
|
||||
## Verified current state
|
||||
|
||||
- DGR-009 evidence was read and verified as the dependency handoff.
|
||||
- `packages/node/meshnet_node/startup.py` already gates backend selection by
|
||||
recipe and can load either the Torch path or the explicit GGUF seam.
|
||||
- `packages/node/meshnet_node/hot_kv_state.py`, `boundary_adapter.py`, and
|
||||
`gguf_ownership.py` already provide the isolation/parity seams that DGR-010
|
||||
would exercise.
|
||||
- The repo has no existing `evidence/DGR-010/README.md` yet, which is expected
|
||||
because the story has not been completed.
|
||||
|
||||
## Commands run
|
||||
|
||||
```bash
|
||||
sed -n '1,260p' .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md
|
||||
sed -n '1,260p' .scratch/distributed-gguf-runtime/issues/10-pass-local-real-model-two-process-acceptance.md
|
||||
sed -n '1,260p' .scratch/distributed-gguf-runtime/evidence/DGR-009/README.md
|
||||
git status --short
|
||||
find /run/media/popov/d/DEV -type f \( -name '*.gguf' -o -name '*.safetensors' -o -name 'config.json' \) | rg -i 'llama|tinyllama|meta-llama|hf-internal-testing|qwen'
|
||||
```
|
||||
|
||||
## Next step to unblock
|
||||
|
||||
Provide or mount a certified dense-Llama artifact on the configured mounted
|
||||
drive storage, then rerun the DGR-010 acceptance path with
|
||||
`MESHNET_ENABLE_REAL_INFERENCE_TESTS=1`.
|
||||
|
||||
## Continuation note
|
||||
|
||||
Once the artifact exists, the next iteration should:
|
||||
|
||||
1. Run the two local worker processes against the certified dense-Llama shard
|
||||
ranges.
|
||||
2. Capture parity, concurrency, memory, and failure metrics.
|
||||
3. Write `evidence/DGR-010/README.md` with the real results and then update the
|
||||
issue status.
|
||||
@@ -1,70 +0,0 @@
|
||||
# DGR-011 — Blocked handoff
|
||||
|
||||
Status: blocked
|
||||
Date: 2026-07-15
|
||||
|
||||
## Blocker
|
||||
|
||||
This story cannot be completed in the current workspace state because its
|
||||
mandatory dependency, DGR-010, is still not passed.
|
||||
|
||||
Verified blockers:
|
||||
|
||||
- `.scratch/distributed-gguf-runtime/prd.json` still marks `DGR-010` and
|
||||
`DGR-011` with `"passes": false`.
|
||||
- `.scratch/distributed-gguf-runtime/evidence/DGR-010/README.md` does not
|
||||
exist, and the only DGR-010 evidence artifact present is
|
||||
`.scratch/distributed-gguf-runtime/evidence/DGR-010/BLOCKED.md`.
|
||||
- Mounted storage search found Qwen model artifacts and llama.cpp vocab files,
|
||||
but no certified dense-Llama GGUF artifact suitable for the required real
|
||||
acceptance run.
|
||||
|
||||
## Verified current state
|
||||
|
||||
- The repo already contains the Meshnet-facing GGUF backend seam and the
|
||||
recipe-gated startup path from DGR-009.
|
||||
- The architecture and Ralph context require real-model execution for this
|
||||
story, not synthetic workers or unit-only coverage.
|
||||
- The current environment does not expose the dense-Llama artifact required to
|
||||
run the prerequisite local real-model acceptance, so the two-machine route
|
||||
cannot be proven end to end.
|
||||
|
||||
## Commands run
|
||||
|
||||
```bash
|
||||
sed -n '1,260p' .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md
|
||||
sed -n '1,260p' .scratch/distributed-gguf-runtime/issues/11-pass-a-real-heterogeneous-two-machine-route.md
|
||||
sed -n '1,260p' .ralph-tui/progress.md
|
||||
sed -n '1,240p' .scratch/distributed-gguf-runtime/evidence/DGR-010/BLOCKED.md
|
||||
sed -n '1,220p' CONTEXT.md
|
||||
sed -n '1,260p' docs/adr/0024-distributed-gguf-runtime.md
|
||||
sed -n '282,350p' .scratch/distributed-gguf-runtime/prd.json
|
||||
find /run/media/popov/d/DEV/models -maxdepth 3 \( -name '*.gguf' -o -name 'config.json' -o -name '*.safetensors' \)
|
||||
find /run/media/popov/d/DEV/llamacpp/llama.cpp/models /run/media/popov/d/DEV/models -maxdepth 4 \( -iname '*llama*' -o -iname '*dense*' -o -iname '*qwen*' -o -name 'config.json' -o -name '*.gguf' \)
|
||||
```
|
||||
|
||||
## Known limitations
|
||||
|
||||
- No certified dense-Llama artifact is available on mounted storage in this
|
||||
workspace.
|
||||
- No real two-machine execution was possible, so there are no real route,
|
||||
hardware, backend, or drift metrics to record for this story.
|
||||
- The story remains blocked until DGR-010 is completed with a real-model
|
||||
evidence README and a confirmed dense-Llama artifact on mounted storage.
|
||||
|
||||
## Compatibility notes
|
||||
|
||||
- DGR-009's recipe-gated GGUF backend seam is present and can be reused.
|
||||
- The acceptance path for this story still requires the upstream real-model
|
||||
evidence from DGR-010 before any heterogeneous two-machine route can be
|
||||
claimed.
|
||||
|
||||
## Dependent-story handoff
|
||||
|
||||
- Finish DGR-010 first, including its real-model evidence README and
|
||||
acceptance run.
|
||||
- Once DGR-010 passes, rerun the two-machine acceptance against the same
|
||||
certified dense-Llama artifact, then record the two-host hardware/network
|
||||
manifest, route, commands, and raw metrics in `evidence/DGR-011/README.md`.
|
||||
- Do not update the issue to `Status: done` until the real two-machine route
|
||||
has been executed and recorded.
|
||||
@@ -1,220 +0,0 @@
|
||||
# DGR-012 — Continuous batching and bounded admission: evidence
|
||||
|
||||
Status: done
|
||||
Date: 2026-07-16
|
||||
Evidence kind: **synthetic-unit** (pure-numpy KV-cached dense-Llama reference +
|
||||
node-local continuous-batching scheduler). No model download, no GPU, no torch,
|
||||
no network, no API credit.
|
||||
|
||||
## Summary
|
||||
|
||||
Implemented the node-local scheduler that turns concurrent Route Sessions into
|
||||
llama.cpp-style continuous batches while bounding admission (RALPH runtime
|
||||
decision #9, ADR-0024). It sits **on top of** the DGR-007 Hot KV State manager —
|
||||
batching is a scheduling concern layered over the existing per-`(session, epoch)`
|
||||
KV isolation, not a new control plane or a change to the KV contract.
|
||||
|
||||
- **Bounded admission (`NodeBudget` + `submit`).** A new session is admitted only
|
||||
if it fits four budgets: resident **weight** footprint (reported), **KV** byte
|
||||
budget (a session must be able to hold its *whole* generation, prompt + new
|
||||
tokens, on its own), **scratch** (per-active-session activation buffers, capped
|
||||
by a total scratch envelope), and the bounded **queue**. Anything that cannot
|
||||
fit is rejected up front with an explicit `AdmissionReason`
|
||||
(`REJECTED_KV_BUDGET` / `REJECTED_SCRATCH_BUDGET` / `REJECTED_DUPLICATE`);
|
||||
anything that fits but has no free slot waits in the bounded queue; a **full
|
||||
queue is refused** (`REJECTED_QUEUE_FULL`) — that refusal is the backpressure
|
||||
signal.
|
||||
- **Continuous batching (`ContinuousBatchScheduler` + `KvBatchEngine`).** Every
|
||||
tick, all currently-decoding sessions contribute their single next token to one
|
||||
batch (bounded by `max_batch_size`); the engine runs the batch once. Each
|
||||
session keeps its own position and appends its own sampled token via its own
|
||||
`SessionCache`, so batching never mixes outputs. `KvBatchEngine` adapts the
|
||||
DGR-007 `KvBoundaryAdapter`, so the batch runs against the *real* KV isolation
|
||||
path; the pinned llama.cpp worker (DGR-008) implements the same
|
||||
`recipe_fingerprint`/`prefill`/`decode_batch`/`release` contract where a batch
|
||||
becomes one `llama_decode` over several sequences.
|
||||
- **Prefill does not starve decode.** The scheduling policy is explicit and fixed:
|
||||
**decode first, then bounded prefill.** In-flight decodes always run before any
|
||||
new prompt is prefilled, and prefill work per tick is capped
|
||||
(`max_prefill_tokens_per_tick`, always allowing at least one so a single large
|
||||
prompt still progresses). A burst of new sessions cannot stall generations
|
||||
already in flight.
|
||||
- **Bounded memory / backpressure.** KV growth is bounded by the manager byte
|
||||
budget; queued activations are bounded by `max_queue_depth` and the scratch
|
||||
envelope; completed sessions release their KV so total KV returns to zero.
|
||||
- **Capability telemetry (`SchedulerTelemetry`).** Reports active sessions, queue
|
||||
depth, batch occupancy (last/avg/max), KV pressure (bytes/budget), scratch
|
||||
pressure, prefill/decode token totals **and rates**, and rejected admissions
|
||||
(total + by reason). All JSON-safe.
|
||||
- **Concurrency 1/2/4/8 sweep (`run_concurrency_sweep`).** Runs the same eight
|
||||
jobs at each level against a fresh KV manager and proves (a) **no cross-session
|
||||
corruption** — every level yields byte-identical per-session tokens as the
|
||||
serialized concurrency-1 reference — and (b) **saturation** — average batch
|
||||
occupancy rises and total ticks fall as concurrency increases, until occupancy
|
||||
plateaus.
|
||||
|
||||
No existing runtime code was modified — this story is purely additive (one new
|
||||
module + one new test module + evidence).
|
||||
|
||||
## Files changed (all new)
|
||||
|
||||
- `packages/node/meshnet_node/batch_scheduler.py` — the scheduler:
|
||||
- `NodeBudget` — weight/KV/scratch/queue budgets + `max_batch_size` /
|
||||
`max_prefill_tokens_per_tick` scheduling bounds, with derived
|
||||
`effective_active_cap` (tighter of active-slot and scratch caps).
|
||||
- `AdmissionReason` / `AdmissionDecision` — structured admit/queue/reject.
|
||||
- `GenerationRequest` / `DecodeItem` / `StepResult` — job + engine I/O values.
|
||||
- `KvBatchEngine` — adapts a full-shard `KvBoundaryAdapter` to the batch-engine
|
||||
contract (rejects a partial head/tail-only range).
|
||||
- `SchedulerTelemetry` — the bounded capability snapshot.
|
||||
- `ContinuousBatchScheduler` — thread-safe `submit` / `run_tick` /
|
||||
`run_to_completion` / `telemetry`, decode-first-then-bounded-prefill policy.
|
||||
- `run_concurrency_sweep` / `ConcurrencyResult` / `ConcurrencySweep` — the
|
||||
deterministic 1/2/4/8 saturation report + corruption check.
|
||||
- `tests/test_batch_scheduler.py` — 16 tests (see below); reuses the DGR-007
|
||||
numpy dense-Llama reference via `from test_hot_kv_state import _KvDenseLlama,
|
||||
_KvReferenceShard`.
|
||||
- `.scratch/distributed-gguf-runtime/evidence/DGR-012/` — this README,
|
||||
`commands.txt`, `generate_evidence.py`, `results.json`.
|
||||
|
||||
## Acceptance criteria → evidence
|
||||
|
||||
- **Scheduler admits sessions against weight, KV, scratch, and queue budgets** —
|
||||
`test_admission_respects_active_scratch_and_queue_budgets` (fill slots → queue →
|
||||
reject full queue), `test_admission_rejects_a_session_that_cannot_fit_the_kv_budget`,
|
||||
`test_admission_rejects_when_per_session_scratch_exceeds_budget`,
|
||||
`test_duplicate_submission_is_rejected`,
|
||||
`test_weight_budget_is_reported_in_telemetry`.
|
||||
- **Compatible decode steps form batches preserving per-session positions/outputs**
|
||||
— `test_batched_decode_preserves_per_session_positions_and_outputs`
|
||||
(`batch_occupancy_max == 4`, four divergent references each reproduced),
|
||||
`test_positions_are_isolated_across_different_prompt_lengths` (prompt lengths 1/3/7).
|
||||
- **Prefill does not starve decode; policy and bounds explicit** —
|
||||
`test_prefill_does_not_starve_in_flight_decode` (in-flight session decodes on
|
||||
*every* tick during a 4-session prefill burst; ≤1 prefill/tick),
|
||||
`test_decode_first_policy_is_explicit_in_a_single_tick`.
|
||||
- **Backpressure prevents unbounded queued activations or KV growth** —
|
||||
`test_backpressure_signals_when_queue_full_then_recovers`,
|
||||
`test_completed_sessions_release_kv_so_growth_is_bounded` (`kv_total_bytes == 0`
|
||||
after completion).
|
||||
- **Capability telemetry reports all required signals** —
|
||||
`test_telemetry_reports_every_required_signal` (asserts every key present;
|
||||
deterministic rates under an injected clock).
|
||||
- **Concurrency 1/2/4/8 identifies saturation, no cross-session corruption** —
|
||||
`test_concurrency_sweep_identifies_saturation_without_corruption`
|
||||
(occupancy strictly ↑, ticks strictly ↓, tokens/tick ↑, `corruption_free`,
|
||||
0 cache misses, saturation=8), `test_concurrency_sweep_saturates_below_max_when_load_is_small`.
|
||||
- **Engine/usage guards** — `test_kv_batch_engine_requires_a_full_shard`,
|
||||
`test_run_to_completion_is_bounded_against_misconfiguration`.
|
||||
|
||||
## Concurrency 1/2/4/8 sweep (real, deterministic — `results.json`)
|
||||
|
||||
Eight sessions, prompt length 4, 8 new tokens each; fresh KV manager per level;
|
||||
budgets sized so KV never evicts (so the corruption check is unambiguous).
|
||||
|
||||
| concurrency | ticks | avg batch occupancy | max occupancy | tokens/tick | peak KV bytes |
|
||||
|---|---|---|---|---|---|
|
||||
| 1 | 64 | 1.000 | 1 | 1.375 | 15360 |
|
||||
| 2 | 33 | 1.750 | 2 | 2.667 | 29184 |
|
||||
| 4 | 19 | 3.111 | 4 | 4.632 | 52224 |
|
||||
| 8 | 15 | 4.000 | 7 | 5.867 | 75264 |
|
||||
|
||||
`saturation_concurrency = 8`, `corruption_free = True`, `cache_misses = 0`,
|
||||
`rejected_admissions = 0`. As concurrency rises, the scheduler packs more sessions
|
||||
per decode step (occupancy ↑) and finishes the same 56 decode + 32 prefill tokens
|
||||
in far fewer ticks (aggregate work/tick ↑) — the batching throughput property —
|
||||
while every per-session token stream stays byte-identical to the serialized
|
||||
reference (no cross-session corruption). Max occupancy is 7 (not 8) at level 8
|
||||
because the fairness policy prefills at most one new session per tick, so the last
|
||||
session begins decoding one tick later.
|
||||
|
||||
## Commands and real results
|
||||
|
||||
```bash
|
||||
VP=/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python
|
||||
|
||||
$VP -m pytest -q tests/test_batch_scheduler.py
|
||||
# -> 16 passed
|
||||
|
||||
$VP -m pytest -q tests/test_hot_kv_state.py # dependency still green
|
||||
# -> 22 passed
|
||||
|
||||
$VP -m compileall -q packages tests
|
||||
# -> exit 0
|
||||
|
||||
git diff --check
|
||||
# -> exit 0
|
||||
|
||||
$VP .scratch/distributed-gguf-runtime/evidence/DGR-012/generate_evidence.py
|
||||
# -> wrote results.json; saturation_concurrency=8 corruption_free=True
|
||||
|
||||
$VP -m pytest -q -rfE -p no:cacheprovider
|
||||
# -> FULL_SUITE_RESULT_PLACEHOLDER
|
||||
```
|
||||
|
||||
`commands.txt` beside this README captures the exact commands.
|
||||
|
||||
## Full-suite baseline (pre-existing unrelated failures)
|
||||
|
||||
FULL_SUITE_BASELINE_PLACEHOLDER
|
||||
|
||||
## Limitations and deferred work
|
||||
|
||||
- **Synthetic-unit, not real weights.** The scheduler is exercised against the
|
||||
deterministic numpy KV-cached dense-Llama reference (the same one DGR-007 uses),
|
||||
not a downloaded GGUF. This is required to keep the default gate deterministic,
|
||||
download-free, and GPU-free. Real concurrent throughput on a downloaded
|
||||
dense-Llama (CPU/ROCm) belongs to DGR-010 (blocked — no certified dense-Llama
|
||||
artifact on this machine; see `evidence/DGR-010/BLOCKED.md`) and the final
|
||||
comparison in DGR-014.
|
||||
- **Batching is a scheduling grouping in this reference.** `KvBatchEngine.decode_batch`
|
||||
runs each batch member sequentially through the cached decode (each attends only
|
||||
its own KV, exactly like an independent llama.cpp sequence). The pinned llama.cpp
|
||||
worker (DGR-008) fuses the batch into one `llama_decode` graph; the scheduling
|
||||
semantics — one batch per tick, isolated positions/outputs — are identical. The
|
||||
numbers here are *scheduler* quantities (ticks, batch occupancy, tokens/tick)
|
||||
that are real and deterministic; **actual kernel-level batching speedup is a
|
||||
native-worker property and is NOT claimed here** (RALPH performance discipline:
|
||||
no unmeasured speed claims). It is measured in DGR-008/DGR-010/DGR-014.
|
||||
- **Greedy sampling only.** Reuses the DGR-006 greedy `SamplingContract`. Greedy
|
||||
over isolated per-session KV is order-independent, which is exactly why the
|
||||
corruption check can assert byte-identical outputs across concurrency levels.
|
||||
Stochastic sampling is out of scope for the deterministic gate.
|
||||
- **Single loaded shard / single recipe per scheduler.** The scheduler batches
|
||||
compatible sessions of one loaded shard (one `recipe_fingerprint`), which is the
|
||||
node-local case. Multi-range routes batch at the head node whose adapter owns the
|
||||
final head; cross-node coordination stays in the Meshnet control plane.
|
||||
- **Native / llama.cpp gates N/A.** No native code, CMake, or llama.cpp patch was
|
||||
touched (same as DGR-005/006/007), so those gates do not apply to this story.
|
||||
|
||||
## Compatibility / migration notes
|
||||
|
||||
- Purely additive: no existing module changed, so no behavior of the Torch/GGUF
|
||||
backends, tracker, or KV manager is altered. The scheduler is opt-in — a server
|
||||
constructs it around a `KvBatchEngine` when it wants continuous batching.
|
||||
- `SchedulerTelemetry.to_dict()` is JSON-safe and aligns with the capability-signal
|
||||
vocabulary (active sessions, queue depth, batch occupancy, KV pressure,
|
||||
prefill/decode rates, rejected admissions) that a node advertises upward; it can
|
||||
be folded into the DGR-009 capability report / heartbeat without schema changes
|
||||
here.
|
||||
- `AdmissionReason` values are stable strings suitable for the native protocol's
|
||||
structured status / backpressure signalling.
|
||||
|
||||
## Handoff for dependent stories
|
||||
|
||||
- **DGR-008 (C++ gRPC worker):** implement the `BatchEngine` contract natively —
|
||||
`decode_batch` becomes one `llama_decode` over the sessions' filtered sequences;
|
||||
`prefill`/`release` map to the same KV manager operations. The scheduler,
|
||||
admission budgets, fairness policy, and telemetry are unchanged; only the engine
|
||||
swaps from numpy to llama.cpp.
|
||||
- **DGR-010 (local real two-process acceptance, blocked):** once a certified
|
||||
dense-Llama artifact is mounted, drive `run_concurrency_sweep` (or the scheduler
|
||||
directly) with a real `KvBatchEngine` over the GGUF backend to produce
|
||||
real-hardware occupancy/throughput/KV-pressure numbers under
|
||||
`MESHNET_ENABLE_REAL_INFERENCE_TESTS=1` / `.venv-rocm`.
|
||||
- **DGR-013 (failure/cancel/restart):** the `DoneReason.CACHE_MISS` path (a decode
|
||||
whose KV was evicted marks the session done and re-prefillable) and the KV-release
|
||||
on completion are the unit basis for the cancellation/cleanup matrix.
|
||||
- **DGR-014 (release gate):** feed the real-hardware sweep’s aggregate throughput
|
||||
and saturation point into the immutable DGR-001 comparison; do not reuse these
|
||||
synthetic numbers as a performance claim.
|
||||
@@ -1,24 +0,0 @@
|
||||
# DGR-012 — exact commands (run from the worktree root)
|
||||
# Default venv (Python 3.14); deterministic, download-free, GPU-free, API-credit-free.
|
||||
VP=/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python
|
||||
|
||||
# Targeted story tests
|
||||
$VP -m pytest -q tests/test_batch_scheduler.py
|
||||
# -> 16 passed
|
||||
|
||||
# Dependency (DGR-007) still green — scheduler builds on this KV manager
|
||||
$VP -m pytest -q tests/test_hot_kv_state.py
|
||||
# -> 22 passed
|
||||
|
||||
# Python quality gates
|
||||
$VP -m compileall -q packages tests
|
||||
# -> exit 0
|
||||
git diff --check
|
||||
# -> exit 0
|
||||
|
||||
# Regenerate the machine-readable concurrency-sweep evidence
|
||||
$VP .scratch/distributed-gguf-runtime/evidence/DGR-012/generate_evidence.py
|
||||
# -> writes results.json; saturation_concurrency=8 corruption_free=True
|
||||
|
||||
# Full deterministic suite (records the pre-existing unrelated failure baseline)
|
||||
$VP -m pytest -q -rfE -p no:cacheprovider
|
||||
@@ -1,117 +0,0 @@
|
||||
"""Regenerate the DGR-012 concurrency-sweep evidence artifact.
|
||||
|
||||
Deterministic, download-free, GPU-free. Run from the repo root with the default
|
||||
venv so the worktree ``meshnet_node`` package and the DGR-007 numpy reference
|
||||
(``tests/test_hot_kv_state``) are importable:
|
||||
|
||||
python .scratch/distributed-gguf-runtime/evidence/DGR-012/generate_evidence.py
|
||||
|
||||
Writes ``results.json`` beside this script.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import pathlib
|
||||
import sys
|
||||
|
||||
_ROOT = pathlib.Path(__file__).resolve().parents[4]
|
||||
sys.path.insert(0, str(_ROOT / "packages" / "node"))
|
||||
sys.path.insert(0, str(_ROOT / "tests"))
|
||||
|
||||
from test_hot_kv_state import _KvDenseLlama, _KvReferenceShard # noqa: E402
|
||||
|
||||
from meshnet_node.batch_scheduler import ( # noqa: E402
|
||||
ContinuousBatchScheduler,
|
||||
GenerationRequest,
|
||||
KvBatchEngine,
|
||||
NodeBudget,
|
||||
run_concurrency_sweep,
|
||||
)
|
||||
from meshnet_node.hot_kv_state import ( # noqa: E402
|
||||
HotKvStateManager,
|
||||
KvBoundaryAdapter,
|
||||
kv_recipe_for,
|
||||
)
|
||||
|
||||
MODEL = _KvDenseLlama()
|
||||
|
||||
|
||||
def make_engine() -> KvBatchEngine:
|
||||
shard = _KvReferenceShard(MODEL, 0, MODEL.n_layers - 1)
|
||||
manager = HotKvStateManager(kv_recipe_for(shard))
|
||||
return KvBatchEngine(KvBoundaryAdapter(shard, manager))
|
||||
|
||||
|
||||
def main() -> int:
|
||||
prompts = {
|
||||
"s0": [1, 2, 3, 4], "s1": [5, 6, 7, 8], "s2": [9, 10, 11, 12],
|
||||
"s3": [13, 14, 15, 16], "s4": [17, 18, 19, 20], "s5": [21, 22, 23, 24],
|
||||
"s6": [25, 26, 27, 28], "s7": [29, 30, 31, 32],
|
||||
}
|
||||
n_new = 8
|
||||
requests = [
|
||||
GenerationRequest(sid, 0, tuple(p), n_new) for sid, p in prompts.items()
|
||||
]
|
||||
sweep = run_concurrency_sweep(
|
||||
make_engine, requests, concurrency_levels=(1, 2, 4, 8)
|
||||
)
|
||||
|
||||
# A representative telemetry snapshot mid-run at concurrency 4 (shows the live
|
||||
# capability signals a node advertises upward).
|
||||
engine = make_engine()
|
||||
scheduler = ContinuousBatchScheduler(
|
||||
engine,
|
||||
NodeBudget(
|
||||
max_active_sessions=4, max_batch_size=4, max_queue_depth=8,
|
||||
scratch_bytes_per_session=1, scratch_budget_bytes=4,
|
||||
),
|
||||
)
|
||||
for request in requests:
|
||||
scheduler.submit(request)
|
||||
for _ in range(6):
|
||||
scheduler.run_tick()
|
||||
mid_run_telemetry = scheduler.telemetry().to_dict()
|
||||
|
||||
artifact = {
|
||||
"schema_version": 1,
|
||||
"evidence_kind": "synthetic-unit",
|
||||
"model": {
|
||||
"reference": "pure-numpy KV-cached dense-Llama (tests/test_hot_kv_state)",
|
||||
"n_layers": MODEL.n_layers,
|
||||
"hidden": MODEL.hidden,
|
||||
"n_heads": MODEL.n_heads,
|
||||
"vocab": MODEL.vocab,
|
||||
},
|
||||
"workload": {
|
||||
"sessions": len(prompts),
|
||||
"prompt_len": 4,
|
||||
"max_new_tokens": n_new,
|
||||
},
|
||||
"concurrency_sweep": sweep.to_dict(),
|
||||
"mid_run_telemetry_concurrency_4": mid_run_telemetry,
|
||||
}
|
||||
|
||||
out = pathlib.Path(__file__).with_name("results.json")
|
||||
out.write_text(json.dumps(artifact, indent=2, sort_keys=True) + "\n", encoding="utf-8")
|
||||
print(f"wrote {out}")
|
||||
print(
|
||||
"saturation_concurrency=%d corruption_free=%s"
|
||||
% (sweep.saturation_concurrency, sweep.corruption_free)
|
||||
)
|
||||
for result in sweep.results:
|
||||
print(
|
||||
" c=%d ticks=%d avg_occ=%.3f tokens/tick=%.3f peak_kv=%dB"
|
||||
% (
|
||||
result.concurrency,
|
||||
result.ticks,
|
||||
result.avg_batch_occupancy,
|
||||
result.tokens_per_tick,
|
||||
result.peak_kv_bytes,
|
||||
)
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -1,179 +0,0 @@
|
||||
{
|
||||
"concurrency_sweep": {
|
||||
"corruption_free": true,
|
||||
"reference_outputs": {
|
||||
"s0": [
|
||||
27,
|
||||
8,
|
||||
27,
|
||||
8,
|
||||
27,
|
||||
8,
|
||||
1,
|
||||
1
|
||||
],
|
||||
"s1": [
|
||||
26,
|
||||
39,
|
||||
39,
|
||||
39,
|
||||
39,
|
||||
3,
|
||||
39,
|
||||
39
|
||||
],
|
||||
"s2": [
|
||||
12,
|
||||
12,
|
||||
12,
|
||||
12,
|
||||
12,
|
||||
12,
|
||||
30,
|
||||
12
|
||||
],
|
||||
"s3": [
|
||||
29,
|
||||
41,
|
||||
42,
|
||||
47,
|
||||
47,
|
||||
42,
|
||||
47,
|
||||
42
|
||||
],
|
||||
"s4": [
|
||||
23,
|
||||
11,
|
||||
44,
|
||||
29,
|
||||
29,
|
||||
29,
|
||||
41,
|
||||
29
|
||||
],
|
||||
"s5": [
|
||||
35,
|
||||
11,
|
||||
0,
|
||||
1,
|
||||
11,
|
||||
0,
|
||||
11,
|
||||
15
|
||||
],
|
||||
"s6": [
|
||||
39,
|
||||
39,
|
||||
28,
|
||||
39,
|
||||
39,
|
||||
39,
|
||||
28,
|
||||
28
|
||||
],
|
||||
"s7": [
|
||||
39,
|
||||
39,
|
||||
39,
|
||||
39,
|
||||
39,
|
||||
39,
|
||||
8,
|
||||
47
|
||||
]
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"avg_batch_occupancy": 1.0,
|
||||
"cache_misses": 0,
|
||||
"concurrency": 1,
|
||||
"decode_batches": 56,
|
||||
"decode_tokens": 56,
|
||||
"max_batch_occupancy": 1,
|
||||
"peak_kv_bytes": 15360,
|
||||
"prefill_tokens": 32,
|
||||
"rejected_admissions": 0,
|
||||
"ticks": 64,
|
||||
"tokens_per_tick": 1.375
|
||||
},
|
||||
{
|
||||
"avg_batch_occupancy": 1.75,
|
||||
"cache_misses": 0,
|
||||
"concurrency": 2,
|
||||
"decode_batches": 32,
|
||||
"decode_tokens": 56,
|
||||
"max_batch_occupancy": 2,
|
||||
"peak_kv_bytes": 29184,
|
||||
"prefill_tokens": 32,
|
||||
"rejected_admissions": 0,
|
||||
"ticks": 33,
|
||||
"tokens_per_tick": 2.6667
|
||||
},
|
||||
{
|
||||
"avg_batch_occupancy": 3.1111,
|
||||
"cache_misses": 0,
|
||||
"concurrency": 4,
|
||||
"decode_batches": 18,
|
||||
"decode_tokens": 56,
|
||||
"max_batch_occupancy": 4,
|
||||
"peak_kv_bytes": 52224,
|
||||
"prefill_tokens": 32,
|
||||
"rejected_admissions": 0,
|
||||
"ticks": 19,
|
||||
"tokens_per_tick": 4.6316
|
||||
},
|
||||
{
|
||||
"avg_batch_occupancy": 4.0,
|
||||
"cache_misses": 0,
|
||||
"concurrency": 8,
|
||||
"decode_batches": 14,
|
||||
"decode_tokens": 56,
|
||||
"max_batch_occupancy": 7,
|
||||
"peak_kv_bytes": 75264,
|
||||
"prefill_tokens": 32,
|
||||
"rejected_admissions": 0,
|
||||
"ticks": 15,
|
||||
"tokens_per_tick": 5.8667
|
||||
}
|
||||
],
|
||||
"saturation_concurrency": 8,
|
||||
"schema_version": 1
|
||||
},
|
||||
"evidence_kind": "synthetic-unit",
|
||||
"mid_run_telemetry_concurrency_4": {
|
||||
"active_sessions": 4,
|
||||
"batch_occupancy_avg": 4.0,
|
||||
"batch_occupancy_last": 4,
|
||||
"batch_occupancy_max": 4,
|
||||
"completed_sessions": 0,
|
||||
"decode_tokens_per_sec": 1637.355,
|
||||
"decode_tokens_total": 20,
|
||||
"kv_budget_bytes": 67108864,
|
||||
"kv_pressure": 0.0008,
|
||||
"kv_total_bytes": 55296,
|
||||
"prefill_tokens_per_sec": 1309.884,
|
||||
"prefill_tokens_total": 16,
|
||||
"queue_depth": 4,
|
||||
"rejected_admissions_total": 0,
|
||||
"rejected_by_reason": {},
|
||||
"scratch_budget_bytes": 4,
|
||||
"scratch_pressure": 1.0,
|
||||
"scratch_used_bytes": 4,
|
||||
"ticks": 6,
|
||||
"weight_bytes": 0
|
||||
},
|
||||
"model": {
|
||||
"hidden": 32,
|
||||
"n_heads": 4,
|
||||
"n_layers": 6,
|
||||
"reference": "pure-numpy KV-cached dense-Llama (tests/test_hot_kv_state)",
|
||||
"vocab": 48
|
||||
},
|
||||
"schema_version": 1,
|
||||
"workload": {
|
||||
"max_new_tokens": 8,
|
||||
"prompt_len": 4,
|
||||
"sessions": 8
|
||||
}
|
||||
}
|
||||
@@ -1,223 +0,0 @@
|
||||
# DGR-013 — Harden failure, cancellation, and restart semantics: evidence
|
||||
|
||||
Status: done
|
||||
Date: 2026-07-16
|
||||
Evidence kind: **synthetic-unit** (pure-numpy KV-cached dense-Llama reference +
|
||||
node-local hardened stream). No model download, no GPU, no torch, no network, no
|
||||
API credit.
|
||||
|
||||
## Summary
|
||||
|
||||
Implemented bounded, explicit failure/cancellation/restart semantics for the
|
||||
per-Route-Session decode stream, layered on the DGR-007 Hot KV State manager
|
||||
(isolated `(session, epoch)` KV) and the DGR-012 continuous-batch scheduler. The
|
||||
goal (RALPH product objective) is that distributed speed never comes with hanging
|
||||
or corrupted generations: every blocked op is bounded, every cancel frees state,
|
||||
duplicate steps are idempotent, uncertain mutations are never silently replayed,
|
||||
alpha failover restarts from token zero, and billing distinguishes what actually
|
||||
completed.
|
||||
|
||||
Everything runs against the same deterministic numpy dense-Llama reference the
|
||||
default gate uses (`tests/test_hot_kv_state.py::_KvDenseLlama` / `_KvReferenceShard`),
|
||||
so the whole failure matrix is deterministic, download-free, GPU-free, and
|
||||
API-credit-free while exercising the **real** KV isolation path
|
||||
(`KvBoundaryAdapter` + `HotKvStateManager`). The pinned llama.cpp worker (DGR-008)
|
||||
implements the identical adapter contract, so the semantics carry over to native
|
||||
execution unchanged.
|
||||
|
||||
### What was built (`packages/node/meshnet_node/failure_semantics.py`, new)
|
||||
|
||||
- **`DeadlineGuard` + `StreamTerminated`** — bounds every step against an absolute
|
||||
deadline and a heartbeat-timeout on an injected clock. A reached deadline or a
|
||||
lost heartbeat (peer health loss) raises `StreamTerminated(kind)` so a blocked
|
||||
stream terminates instead of hanging. (**AC: deadlines/heartbeat terminate
|
||||
blocked ops.**)
|
||||
- **`CancellationToken`, `ShardCancellationGroup`, `CancellationOutcome`** — one
|
||||
cancel fans across **every** node-local Shard of a Route Session, releasing the
|
||||
`(session, epoch)` KV on each shard's manager and invoking every queued-buffer
|
||||
release callback (the pending activation bundles). Idempotent. The DGR-012
|
||||
scheduler also gains a `cancel()` that drops queued/active work on this node and
|
||||
frees its KV. (**AC: cancellation propagates across every Shard, releases KV +
|
||||
queued buffers.**)
|
||||
- **`IdempotencyLedger`, `StepKey`, `StepDisposition`, `UncertainMutationError`** —
|
||||
records each committed `(session, epoch, step)`; a duplicate delivery returns the
|
||||
recorded token with no re-mutation. A step whose mutation outcome is *uncertain*
|
||||
(worker died mid-step) is marked uncertain and can **never** be replayed
|
||||
silently — `begin()` on an uncertain (or still in-flight) step raises
|
||||
`UncertainMutationError`, forcing verify-or-restart. (**AC: duplicate steps
|
||||
idempotent; uncertain mutations never replayed silently.**)
|
||||
- **`RestartController`** — alpha failover: opens the *next* route epoch, releases
|
||||
every shard's prior-epoch KV, and `assert_fresh_start` fails closed if any shard
|
||||
still holds new-epoch KV. The restart re-prefills the whole prompt from token
|
||||
zero; the failed epoch becomes stale (KV manager rejects it). Unverified KV is
|
||||
never migrated (RALPH runtime decision #14). (**AC: alpha failover restarts from
|
||||
token zero rather than importing unverified KV.**)
|
||||
- **`WorkStatus`, `WorkRecord`, `WorkLedger`** — a typed per-attempt work record
|
||||
with four distinct statuses: `completed`, `cancelled`, `failed`, `unverified`.
|
||||
Only `completed` records are billable; cancelled/failed/unverified tokens are
|
||||
recorded for observability but never charged. JSON-safe for the tracker billing
|
||||
handoff (`packages/tracker/meshnet_tracker/billing.py` charges only completed,
|
||||
verified work). (**AC: billing/work records distinguish completed/cancelled/
|
||||
failed/unverified.**)
|
||||
- **`HardenedSessionRunner`** — composes all of the above to drive one session's
|
||||
prefill+decode through the adapter under a deadline/heartbeat guard + cancel
|
||||
token, records the typed outcome, and `run_with_failover` restarts a transient
|
||||
failure from token zero on a fresh epoch.
|
||||
- **`FailureKind` + `classify_exception` + `work_status_for`** — stable-string
|
||||
classification of worker death, stream reset, malformed bundle, stale epoch,
|
||||
cache miss, deadline, heartbeat loss, and cancel, plus the failure→billing-status
|
||||
mapping. Suitable for the native protocol's structured status.
|
||||
|
||||
### Scheduler extension (`packages/node/meshnet_node/batch_scheduler.py`, DGR-012 file, additive)
|
||||
|
||||
Purely additive so the DGR-012 gate stays green (16/16):
|
||||
- `DoneReason.CANCELLED` / `DoneReason.FAILED` terminal reasons.
|
||||
- `ContinuousBatchScheduler.cancel(session_id, *, reason)` — drops a queued
|
||||
session from the bounded queue or releases an active session's KV, moving it to
|
||||
the done set with a non-completed reason (never counted as completed work).
|
||||
- `SchedulerTelemetry.cancelled_sessions` / `failed_sessions` counters.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `packages/node/meshnet_node/failure_semantics.py` — new module (the whole
|
||||
failure/cancel/restart layer above).
|
||||
- `packages/node/meshnet_node/batch_scheduler.py` — additive `cancel()` + two
|
||||
`DoneReason` members + two telemetry counters (DGR-012 file; its 16 tests still
|
||||
pass unchanged).
|
||||
- `tests/test_failure_semantics.py` — new, 22 tests (matrix below); reuses the
|
||||
DGR-007 numpy reference via `from test_hot_kv_state import _KvDenseLlama,
|
||||
_KvReferenceShard`.
|
||||
- `.scratch/distributed-gguf-runtime/evidence/DGR-013/` — this README,
|
||||
`commands.txt`, `generate_evidence.py`, `results.json`.
|
||||
- `.ralph-tui/progress.md` — appended the DGR-013 note.
|
||||
- `.scratch/distributed-gguf-runtime/issues/13-...md` — set `Status: done`.
|
||||
|
||||
## Acceptance criteria → evidence
|
||||
|
||||
| Criterion | Tests (`tests/test_failure_semantics.py`) |
|
||||
|---|---|
|
||||
| Deadlines/heartbeat loss terminate blocked stream ops | `test_deadline_terminates_a_blocked_stream_and_releases_kv`, `test_heartbeat_loss_terminates_a_blocked_stream`, `test_deadline_guard_reports_remaining_and_resets_on_heartbeat` |
|
||||
| Cancellation propagates across every Shard, releases KV + queued buffers | `test_cancellation_token_terminates_stream_and_releases_kv`, `test_shard_cancellation_group_releases_every_shard_and_queued_buffers`, `test_scheduler_cancel_drains_queue_and_releases_active_kv`, `test_scheduler_cancel_rejects_a_completed_reason` |
|
||||
| Duplicate steps idempotent; uncertain mutations never replayed silently | `test_duplicate_step_delivery_is_idempotent_no_remutation`, `test_idempotent_run_replays_tokens_without_advancing_kv`, `test_uncertain_mutation_is_never_replayed_silently`, `test_in_flight_duplicate_is_treated_as_uncertain` |
|
||||
| Alpha failover restarts from token zero, no unverified KV import | `test_alpha_failover_restarts_from_token_zero_and_completes`, `test_failover_refuses_to_import_unverified_kv`, `test_non_restartable_failure_is_not_retried` |
|
||||
| Worker death, stream reset, malformed bundle, stale epoch, cache miss | `test_worker_death_midstream_is_unverified_and_marks_step_uncertain`, `test_stream_reset_is_restartable_failure`, `test_malformed_bundle_is_classified_and_does_not_corrupt_kv`, `test_stale_epoch_reference_is_rejected_and_classified`, `test_cache_miss_midstream_is_restartable` |
|
||||
| Billing/work records distinguish completed/cancelled/failed/unverified | `test_work_ledger_distinguishes_all_four_statuses`, `test_work_status_and_classification_mapping`, plus the clean-run billability check `test_clean_run_matches_stateless_reference_and_is_billable` |
|
||||
|
||||
## Failure matrix (real, deterministic — `results.json`)
|
||||
|
||||
Generated by `generate_evidence.py` against the numpy dense-Llama (prompt `[7,3,9,1]`,
|
||||
8 new tokens):
|
||||
|
||||
| scenario | status | failure_kind | tokens | restartable | KV released |
|
||||
|---|---|---|---|---|---|
|
||||
| clean | completed | — | 8 | — | (held, then reaped) |
|
||||
| deadline | failed | deadline-exceeded | 2 | no | yes |
|
||||
| heartbeat_loss | failed | heartbeat-lost | 3 | no | yes |
|
||||
| cancel | cancelled | cancelled | 3 | no | yes |
|
||||
| worker_death | unverified | worker-death | 3 | yes | yes |
|
||||
| stream_reset | failed | stream-reset | — | yes | yes |
|
||||
| stale_epoch | failed | stale-epoch | — | no | (never opened) |
|
||||
| cache_miss | failed | cache-miss | 4 | yes | (already evicted) |
|
||||
| alpha_failover | **completed** (epoch 1) | — | 8 | — | old epoch stale |
|
||||
|
||||
Alpha failover: attempt 0 (epoch 0) dies mid-step → `unverified`; the controller
|
||||
advances to epoch 1, drops epoch-0 KV, and the restart re-prefills from token zero
|
||||
→ `completed`, reproducing the byte-identical stateless reference. The old epoch is
|
||||
now stale (a reference to it raises `StaleRouteEpochError`). Work ledger:
|
||||
`{completed: 2, cancelled: 1, failed: 0, unverified: 2}`, `billable_tokens = 16`
|
||||
(only the two completed streams — the failover restart and the clean run — are
|
||||
billed; the cancelled and the two unverified attempts are not).
|
||||
|
||||
## Commands and real results
|
||||
|
||||
See `commands.txt`. Key results:
|
||||
|
||||
```
|
||||
tests/test_failure_semantics.py -> 22 passed
|
||||
tests/test_batch_scheduler.py -> 16 passed (DGR-012 unchanged)
|
||||
tests/test_hot_kv_state.py -> 22 passed (DGR-007)
|
||||
tests/test_gguf_backend.py -> 2 passed (DGR-009)
|
||||
python -m compileall -q packages tests -> exit 0
|
||||
git diff --check -> exit 0
|
||||
python -m pytest -q -> 16 failed, 792 passed, 14 skipped in 253.93s
|
||||
```
|
||||
|
||||
## Full-suite baseline (pre-existing, unrelated failures)
|
||||
|
||||
The 16 failures are **pre-existing and unrelated to DGR-013**. None import
|
||||
`failure_semantics` or `batch_scheduler`; they live in the tracker/control-plane,
|
||||
node-startup, doctor, calibration, and route-benchmark suites and fail on the
|
||||
model-download / control-plane / recipe-admission paths (e.g.
|
||||
`UnsupportedRecipeParam: worker_transport` from the DGR-009 native recipe against
|
||||
the Torch backend, and Torch/HF-model startup that this deterministic sandbox does
|
||||
not provide). Removing the two DGR-013 files and re-running the failing tests
|
||||
reproduces the identical failures (see `commands.txt`, 4-test spot check → same
|
||||
4 failures), so DGR-013 introduces no new failure.
|
||||
|
||||
Exact failing set (16):
|
||||
|
||||
```
|
||||
tests/test_dynamic_routing.py::test_admin_can_replace_a_served_model_and_release_it
|
||||
tests/test_manual_route_benchmark.py::test_pinned_route_uses_named_node
|
||||
tests/test_manual_route_benchmark.py::test_unknown_route_node_is_400
|
||||
tests/test_manual_route_benchmark.py::test_invalid_route_shape_is_400
|
||||
tests/test_manual_route_benchmark.py::test_clients_without_route_are_unaffected
|
||||
tests/test_manual_route_benchmark.py::test_benchmark_records_one_and_two_node_routes
|
||||
tests/test_node_doctor.py::test_the_shipped_recipes_are_all_applicable_by_the_backend
|
||||
tests/test_node_doctor.py::test_cli_doctor_flags_select_what_is_validated
|
||||
tests/test_node_startup.py::test_preset_model_with_hf_repo_loads_torch_backend
|
||||
tests/test_node_startup.py::test_real_model_startup_registers_downloaded_inventory_without_checksum
|
||||
tests/test_toploc_calibration_dispatch.py::test_calibration_run_dispatches_only_solo_capable_nodes
|
||||
tests/test_toploc_calibration_dispatch.py::test_calibration_run_persists_corpus_and_results_endpoint_reports_it
|
||||
tests/test_toploc_calibration_dispatch.py::test_calibration_run_node_without_commitment_endpoint_is_skipped_not_failed
|
||||
tests/test_tracker_capability_admission.py::test_an_enforcing_tracker_never_routes_a_node_whose_proof_does_not_cover_it[invalid]
|
||||
tests/test_tracker_routing.py::test_torch_node_applies_tracker_load_shard_directive
|
||||
tests/test_tracker_routing.py::test_shard_heal_cycle_surviving_node_covers_dead_peers_gap
|
||||
```
|
||||
|
||||
## Limitations and deferred work
|
||||
|
||||
- **Synthetic-unit, not real weights.** Semantics are exercised against the
|
||||
deterministic numpy dense-Llama, not a downloaded GGUF, to keep the default gate
|
||||
deterministic/download-free/GPU-free. Real worker-death/stream-reset behavior on
|
||||
a live llama.cpp worker over gRPC belongs to DGR-008/DGR-010 (DGR-010 is blocked
|
||||
— no certified dense-Llama artifact on this machine; see
|
||||
`evidence/DGR-010/BLOCKED.md`).
|
||||
- **Single-node per-session stream.** `HardenedSessionRunner` drives one full-shard
|
||||
session (the node-local case); multi-node cancellation is modelled by
|
||||
`ShardCancellationGroup` fanning across each node's KV manager. The cross-node
|
||||
propagation *transport* (cancel frames over gRPC/relay) is the native protocol's
|
||||
job (DGR-002/008); this story owns the local release + record semantics the
|
||||
transport triggers.
|
||||
- **Fault injection is deterministic.** Worker death is a shard that raises on the
|
||||
Nth step; stream reset / deadline / heartbeat are injected via an explicit clock
|
||||
and hook. This is what makes the matrix reproducible; live fault behavior is a
|
||||
native/real-hardware property.
|
||||
- **Greedy sampling only.** Reuses the DGR-006 greedy `SamplingContract`; the
|
||||
idempotent-replay equality check depends on order-independent greedy decode.
|
||||
- **Native / llama.cpp gates N/A.** No native code, CMake, or llama.cpp patch was
|
||||
touched (same as DGR-005/006/007/012), so those gates do not apply.
|
||||
|
||||
## Compatibility / migration notes
|
||||
|
||||
- `failure_semantics.py` is a new, additive module — no existing behavior changes.
|
||||
- `batch_scheduler.py` changes are additive (new enum members, one method, two
|
||||
telemetry fields); the DGR-012 contract and its 16 tests are unchanged.
|
||||
- `WorkRecord.to_dict()` / `WorkLedger.to_dict()` are JSON-safe and map cleanly to
|
||||
the tracker `BillingLedger.charge_request` inputs: report `node_work` only for
|
||||
`billable` (completed) records so cancelled/failed/unverified work is never
|
||||
charged. `FailureKind` / `WorkStatus` are stable strings suitable for the native
|
||||
protocol's structured status and the capability/heartbeat report.
|
||||
|
||||
## Handoff for dependent stories
|
||||
|
||||
- **DGR-008 (C++ gRPC worker):** implement the same contract natively — the worker
|
||||
maps a transport deadline/heartbeat to `StreamTerminated`, a dropped stream to a
|
||||
restartable failure, and a mid-`llama_decode` crash to an *uncertain* step
|
||||
(mark-uncertain, never silent replay). `RestartController.failover` maps to
|
||||
opening a fresh llama sequence under the new `(session, epoch)`; the failed
|
||||
sequence's KV is dropped, never migrated.
|
||||
- **DGR-010/DGR-014 (real acceptance / release gate):** drive the same failure
|
||||
scenarios against the live worker to produce real cleanup/latency numbers, and
|
||||
feed the `WorkLedger` status split into the billing/attribution comparison —
|
||||
only `completed` work is charged.
|
||||
@@ -1,36 +0,0 @@
|
||||
# DGR-013 — exact commands and real results (worktree venv)
|
||||
VP=/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python
|
||||
|
||||
# Targeted story tests (this story)
|
||||
$VP -m pytest -q tests/test_failure_semantics.py
|
||||
# -> 22 passed
|
||||
|
||||
# Dependency gates stay green
|
||||
$VP -m pytest -q tests/test_batch_scheduler.py # DGR-012
|
||||
# -> 16 passed
|
||||
$VP -m pytest -q tests/test_hot_kv_state.py # DGR-007
|
||||
# -> 22 passed
|
||||
$VP -m pytest -q tests/test_gguf_backend.py # DGR-009
|
||||
# -> 2 passed
|
||||
|
||||
# Quality gates
|
||||
$VP -m compileall -q packages tests
|
||||
# -> exit 0
|
||||
git diff --check
|
||||
# -> exit 0
|
||||
|
||||
# Machine-readable evidence
|
||||
$VP .scratch/distributed-gguf-runtime/evidence/DGR-013/generate_evidence.py
|
||||
# -> wrote results.json; work statuses {'completed':2,'cancelled':1,'failed':0,'unverified':2} billable_tokens=16
|
||||
|
||||
# Full deterministic suite
|
||||
$VP -m pytest -q -p no:cacheprovider
|
||||
# -> 16 failed, 792 passed, 14 skipped in 253.93s
|
||||
|
||||
# Clean-tree reproduction of the 16 pre-existing failures (DGR-013 files removed)
|
||||
# rm packages/node/meshnet_node/failure_semantics.py tests/test_failure_semantics.py
|
||||
$VP -m pytest -q tests/test_dynamic_routing.py::test_admin_can_replace_a_served_model_and_release_it \
|
||||
tests/test_node_doctor.py::test_the_shipped_recipes_are_all_applicable_by_the_backend \
|
||||
tests/test_tracker_routing.py::test_torch_node_applies_tracker_load_shard_directive \
|
||||
tests/test_node_startup.py::test_preset_model_with_hf_repo_loads_torch_backend
|
||||
# -> 4 failed (same failures reproduce without any DGR-013 change)
|
||||
@@ -1,234 +0,0 @@
|
||||
#!/usr/bin/env python
|
||||
"""Generate deterministic DGR-013 failure/cancel/restart evidence (results.json).
|
||||
|
||||
Runs the real hardened per-session stream (``HardenedSessionRunner`` over the
|
||||
DGR-007 ``KvBoundaryAdapter`` + ``HotKvStateManager``) through each failure mode
|
||||
with the same pure-numpy dense-Llama reference the default gate uses. No model
|
||||
download, no GPU, no torch, no network, no API credit.
|
||||
|
||||
Run from the repo root with the worktree venv:
|
||||
|
||||
.venv/bin/python .scratch/distributed-gguf-runtime/evidence/DGR-013/generate_evidence.py
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
import numpy as np
|
||||
|
||||
# Make the worktree packages and the DGR-007 numpy reference importable, exactly
|
||||
# as pytest's prepend-import + conftest do.
|
||||
ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "..", "..", ".."))
|
||||
sys.path.insert(0, os.path.join(ROOT, "packages", "node"))
|
||||
sys.path.insert(0, os.path.join(ROOT, "tests"))
|
||||
|
||||
from meshnet_node.hot_kv_state import ( # noqa: E402
|
||||
HotKvStateConfig,
|
||||
HotKvStateManager,
|
||||
KvBoundaryAdapter,
|
||||
StaleRouteEpochError,
|
||||
kv_recipe_for,
|
||||
)
|
||||
from meshnet_node.batch_scheduler import GenerationRequest # noqa: E402
|
||||
from meshnet_node.failure_semantics import ( # noqa: E402
|
||||
CancellationToken,
|
||||
FailureKind,
|
||||
HardenedSessionRunner,
|
||||
RestartController,
|
||||
StreamTerminated,
|
||||
WorkLedger,
|
||||
WorkStatus,
|
||||
)
|
||||
|
||||
from test_hot_kv_state import _KvDenseLlama, _KvReferenceShard # noqa: E402
|
||||
|
||||
|
||||
class _FaultyShard(_KvReferenceShard):
|
||||
def __init__(self, model, start, end, *, fail_at_call=None):
|
||||
super().__init__(model, start, end)
|
||||
self._fail_at_call = fail_at_call
|
||||
self.calls = 0
|
||||
|
||||
def run_layers_cached(self, hidden, *, positions, past_kv):
|
||||
self.calls += 1
|
||||
if self._fail_at_call is not None and self.calls == self._fail_at_call:
|
||||
raise RuntimeError("worker died mid-step")
|
||||
return super().run_layers_cached(hidden, positions=positions, past_kv=past_kv)
|
||||
|
||||
|
||||
class _Clock:
|
||||
def __init__(self):
|
||||
self.now = 0.0
|
||||
|
||||
def __call__(self):
|
||||
return self.now
|
||||
|
||||
def advance(self, d):
|
||||
self.now += d
|
||||
|
||||
|
||||
def _adapter(model, *, config=None, shard=None):
|
||||
shard = shard or _KvReferenceShard(model, 0, model.n_layers - 1)
|
||||
manager = HotKvStateManager(kv_recipe_for(shard), config=config)
|
||||
return KvBoundaryAdapter(shard, manager)
|
||||
|
||||
|
||||
def _gen(sid, prompt, n, epoch=0):
|
||||
return GenerationRequest(
|
||||
session_id=sid, route_epoch=epoch,
|
||||
prompt_token_ids=tuple(prompt), max_new_tokens=n,
|
||||
)
|
||||
|
||||
|
||||
def _kv_released(manager, sid, epoch):
|
||||
from meshnet_node.hot_kv_state import CacheMiss
|
||||
return isinstance(manager.resolve(sid, epoch), CacheMiss)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
model = _KvDenseLlama()
|
||||
prompt = [7, 3, 9, 1]
|
||||
n_new = 8
|
||||
ledger = WorkLedger()
|
||||
scenarios = []
|
||||
|
||||
# 1. Clean baseline.
|
||||
ad = _adapter(model)
|
||||
r = HardenedSessionRunner(ad, work_ledger=ledger).run(_gen("clean", prompt, n_new))
|
||||
scenarios.append({
|
||||
"scenario": "clean",
|
||||
"status": r.status.value,
|
||||
"tokens": r.token_count,
|
||||
"matches_reference": list(r.tokens) == model.stateless_greedy(prompt, n_new),
|
||||
"kv_released": _kv_released(ad.manager, "clean", 0),
|
||||
})
|
||||
|
||||
# 2. Deadline terminates a blocked stream.
|
||||
clk = _Clock()
|
||||
ad = _adapter(model)
|
||||
r = HardenedSessionRunner(ad, clock=clk).run(
|
||||
_gen("deadline", prompt, 50), deadline=3.0,
|
||||
before_step=lambda _s: clk.advance(1.0),
|
||||
)
|
||||
scenarios.append({
|
||||
"scenario": "deadline", "status": r.status.value,
|
||||
"failure_kind": r.failure_kind.value, "tokens": r.token_count,
|
||||
"kv_released": _kv_released(ad.manager, "deadline", 0),
|
||||
})
|
||||
|
||||
# 3. Heartbeat/health loss terminates a blocked stream.
|
||||
clk = _Clock()
|
||||
ad = _adapter(model)
|
||||
r = HardenedSessionRunner(ad, clock=clk).run(
|
||||
_gen("heartbeat", prompt, 50), heartbeat_timeout=1.5,
|
||||
heartbeat=lambda step: step < 2,
|
||||
before_step=lambda _s: clk.advance(1.0),
|
||||
)
|
||||
scenarios.append({
|
||||
"scenario": "heartbeat_loss", "status": r.status.value,
|
||||
"failure_kind": r.failure_kind.value, "tokens": r.token_count,
|
||||
"kv_released": _kv_released(ad.manager, "heartbeat", 0),
|
||||
})
|
||||
|
||||
# 4. Explicit client cancellation releases KV.
|
||||
ad = _adapter(model)
|
||||
tok = CancellationToken()
|
||||
r = HardenedSessionRunner(ad, work_ledger=ledger).run(
|
||||
_gen("cancel", prompt, 50), cancel_token=tok,
|
||||
before_step=lambda step: tok.cancel("client-hangup") if step == 3 else None,
|
||||
)
|
||||
scenarios.append({
|
||||
"scenario": "cancel", "status": r.status.value,
|
||||
"failure_kind": r.failure_kind.value, "tokens": r.token_count,
|
||||
"kv_released": _kv_released(ad.manager, "cancel", 0),
|
||||
})
|
||||
|
||||
# 5. Worker death mid-step -> unverified.
|
||||
ad = _adapter(model, shard=_FaultyShard(model, 0, model.n_layers - 1, fail_at_call=4))
|
||||
r = HardenedSessionRunner(ad, work_ledger=ledger).run(_gen("worker", prompt, n_new))
|
||||
scenarios.append({
|
||||
"scenario": "worker_death", "status": r.status.value,
|
||||
"failure_kind": r.failure_kind.value, "tokens": r.token_count,
|
||||
"restartable": r.restartable, "kv_released": _kv_released(ad.manager, "worker", 0),
|
||||
})
|
||||
|
||||
# 6. Stream reset -> failed, restartable.
|
||||
ad = _adapter(model)
|
||||
def reset(step):
|
||||
if step == 2:
|
||||
raise StreamTerminated(FailureKind.STREAM_RESET, "peer reset")
|
||||
r = HardenedSessionRunner(ad).run(_gen("reset", prompt, n_new), before_step=reset)
|
||||
scenarios.append({
|
||||
"scenario": "stream_reset", "status": r.status.value,
|
||||
"failure_kind": r.failure_kind.value, "restartable": r.restartable,
|
||||
})
|
||||
|
||||
# 7. Stale epoch -> failed.
|
||||
ad = _adapter(model)
|
||||
ad.manager.open("stale", 5)
|
||||
r = HardenedSessionRunner(ad).run(_gen("stale", prompt, n_new, epoch=3))
|
||||
scenarios.append({
|
||||
"scenario": "stale_epoch", "status": r.status.value,
|
||||
"failure_kind": r.failure_kind.value,
|
||||
})
|
||||
|
||||
# 8. Cache miss mid-stream -> restartable.
|
||||
ad = _adapter(model)
|
||||
mgr = ad.manager
|
||||
r = HardenedSessionRunner(ad).run(
|
||||
_gen("miss", prompt, 12),
|
||||
before_step=lambda step: mgr.release("miss", 0) if step == 4 else None,
|
||||
)
|
||||
scenarios.append({
|
||||
"scenario": "cache_miss", "status": r.status.value,
|
||||
"failure_kind": r.failure_kind.value, "tokens": r.token_count,
|
||||
"restartable": r.restartable,
|
||||
})
|
||||
|
||||
# 9. Alpha failover: restart from token zero, no unverified KV import.
|
||||
faulty = _FaultyShard(model, 0, model.n_layers - 1, fail_at_call=3)
|
||||
ad = _adapter(model, shard=faulty)
|
||||
runner = HardenedSessionRunner(ad, work_ledger=ledger)
|
||||
controller = RestartController([ad.manager])
|
||||
fo = runner.run_with_failover(_gen("failover", prompt, n_new, epoch=0), controller,
|
||||
max_restarts=2)
|
||||
old_epoch_stale = False
|
||||
try:
|
||||
ad.manager.resolve("failover", 0)
|
||||
except StaleRouteEpochError:
|
||||
old_epoch_stale = True
|
||||
scenarios.append({
|
||||
"scenario": "alpha_failover",
|
||||
"final_status": fo.outcome.status.value,
|
||||
"final_epoch": fo.outcome.route_epoch,
|
||||
"restarts": fo.restarts,
|
||||
"restarted_from_token_zero": list(fo.outcome.tokens) == model.stateless_greedy(prompt, n_new),
|
||||
"old_epoch_stale": old_epoch_stale,
|
||||
"attempt_statuses": [a.status.value for a in fo.attempts],
|
||||
})
|
||||
|
||||
result = {
|
||||
"schema_version": 1,
|
||||
"evidence_kind": "synthetic-unit",
|
||||
"model": {
|
||||
"architecture": model.architecture_adapter,
|
||||
"n_layers": model.n_layers, "vocab": model.vocab, "hidden": model.hidden,
|
||||
},
|
||||
"scenarios": scenarios,
|
||||
"work_ledger": ledger.to_dict(),
|
||||
}
|
||||
|
||||
out_path = os.path.join(os.path.dirname(__file__), "results.json")
|
||||
with open(out_path, "w") as fh:
|
||||
json.dump(result, fh, indent=2)
|
||||
fh.write("\n")
|
||||
counts = ledger.counts_by_status()
|
||||
print(f"wrote {out_path}")
|
||||
print(f"work statuses: {counts} billable_tokens={ledger.billable_tokens()}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,135 +0,0 @@
|
||||
{
|
||||
"schema_version": 1,
|
||||
"evidence_kind": "synthetic-unit",
|
||||
"model": {
|
||||
"architecture": "dense-llama",
|
||||
"n_layers": 6,
|
||||
"vocab": 48,
|
||||
"hidden": 32
|
||||
},
|
||||
"scenarios": [
|
||||
{
|
||||
"scenario": "clean",
|
||||
"status": "completed",
|
||||
"tokens": 8,
|
||||
"matches_reference": true,
|
||||
"kv_released": false
|
||||
},
|
||||
{
|
||||
"scenario": "deadline",
|
||||
"status": "failed",
|
||||
"failure_kind": "deadline-exceeded",
|
||||
"tokens": 2,
|
||||
"kv_released": true
|
||||
},
|
||||
{
|
||||
"scenario": "heartbeat_loss",
|
||||
"status": "failed",
|
||||
"failure_kind": "heartbeat-lost",
|
||||
"tokens": 3,
|
||||
"kv_released": true
|
||||
},
|
||||
{
|
||||
"scenario": "cancel",
|
||||
"status": "cancelled",
|
||||
"failure_kind": "cancelled",
|
||||
"tokens": 3,
|
||||
"kv_released": true
|
||||
},
|
||||
{
|
||||
"scenario": "worker_death",
|
||||
"status": "unverified",
|
||||
"failure_kind": "worker-death",
|
||||
"tokens": 3,
|
||||
"restartable": true,
|
||||
"kv_released": true
|
||||
},
|
||||
{
|
||||
"scenario": "stream_reset",
|
||||
"status": "failed",
|
||||
"failure_kind": "stream-reset",
|
||||
"restartable": true
|
||||
},
|
||||
{
|
||||
"scenario": "stale_epoch",
|
||||
"status": "failed",
|
||||
"failure_kind": "stale-epoch"
|
||||
},
|
||||
{
|
||||
"scenario": "cache_miss",
|
||||
"status": "failed",
|
||||
"failure_kind": "cache-miss",
|
||||
"tokens": 4,
|
||||
"restartable": true
|
||||
},
|
||||
{
|
||||
"scenario": "alpha_failover",
|
||||
"final_status": "completed",
|
||||
"final_epoch": 1,
|
||||
"restarts": 1,
|
||||
"restarted_from_token_zero": true,
|
||||
"old_epoch_stale": true,
|
||||
"attempt_statuses": [
|
||||
"unverified",
|
||||
"completed"
|
||||
]
|
||||
}
|
||||
],
|
||||
"work_ledger": {
|
||||
"schema_version": 1,
|
||||
"records": [
|
||||
{
|
||||
"session_id": "clean",
|
||||
"route_epoch": 0,
|
||||
"status": "completed",
|
||||
"tokens": 8,
|
||||
"failure_kind": null,
|
||||
"detail": "",
|
||||
"billable": true
|
||||
},
|
||||
{
|
||||
"session_id": "cancel",
|
||||
"route_epoch": 0,
|
||||
"status": "cancelled",
|
||||
"tokens": 3,
|
||||
"failure_kind": "cancelled",
|
||||
"detail": "operation cancelled: client-hangup",
|
||||
"billable": false
|
||||
},
|
||||
{
|
||||
"session_id": "worker",
|
||||
"route_epoch": 0,
|
||||
"status": "unverified",
|
||||
"tokens": 3,
|
||||
"failure_kind": "worker-death",
|
||||
"detail": "worker died mid-step",
|
||||
"billable": false
|
||||
},
|
||||
{
|
||||
"session_id": "failover",
|
||||
"route_epoch": 0,
|
||||
"status": "unverified",
|
||||
"tokens": 2,
|
||||
"failure_kind": "worker-death",
|
||||
"detail": "worker died mid-step",
|
||||
"billable": false
|
||||
},
|
||||
{
|
||||
"session_id": "failover",
|
||||
"route_epoch": 1,
|
||||
"status": "completed",
|
||||
"tokens": 8,
|
||||
"failure_kind": null,
|
||||
"detail": "",
|
||||
"billable": true
|
||||
}
|
||||
],
|
||||
"counts_by_status": {
|
||||
"completed": 2,
|
||||
"cancelled": 1,
|
||||
"failed": 0,
|
||||
"unverified": 2
|
||||
},
|
||||
"billable_tokens": 16
|
||||
}
|
||||
}
|
||||
@@ -1,55 +0,0 @@
|
||||
# DGR-014 — Blocked handoff
|
||||
|
||||
Status: blocked
|
||||
Date: 2026-07-16
|
||||
|
||||
## Blocker
|
||||
|
||||
This release-gate story cannot be completed in the current workspace state because the prerequisite real-model comparison chain is still missing its certified dense-Llama artifact on mounted storage.
|
||||
|
||||
Verified blockers:
|
||||
|
||||
- `DGR-011` is still not passed in `.scratch/distributed-gguf-runtime/prd.json`.
|
||||
- `DGR-011` is explicitly blocked in `.scratch/distributed-gguf-runtime/evidence/DGR-011/BLOCKED.md`.
|
||||
- `DGR-011` depends on `DGR-010`, and `DGR-010` is blocked because there is no certified dense-Llama artifact available on the mounted drive.
|
||||
- Current mounted-model storage still only shows Qwen artifacts and llama.cpp vocab GGUFs, not the certified dense-Llama GGUF/safetensors pair needed for a comparable real run.
|
||||
|
||||
## Verified current state
|
||||
|
||||
- The DGR-001 performance contract exists and defines the benchmark lanes, metrics, and stop condition that later release gates must keep unchanged.
|
||||
- The DGR-012 scheduler and DGR-013 failure semantics evidence are present and usable as supporting context, but they do not satisfy the real final comparison required here.
|
||||
- `packages/node/meshnet_node/performance_contract.py` already contains the contract metadata and a live endpoint benchmark shim, but there is no recorded DGR-014 release-gate run and no final immutable comparison artifact.
|
||||
- `evidence/DGR-014/README.md` does not exist yet because the acceptance criteria could not be completed.
|
||||
|
||||
## Commands run
|
||||
|
||||
```bash
|
||||
sed -n '1,260p' .claude/memory/MEMORY.md
|
||||
sed -n '1,260p' .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md
|
||||
sed -n '1,260p' .scratch/distributed-gguf-runtime/issues/14-enforce-the-gguf-versus-safetensors-release-gate.md
|
||||
sed -n '1,260p' .ralph-tui/progress.md
|
||||
git status --short
|
||||
sed -n '1,260p' .scratch/distributed-gguf-runtime/prd.json
|
||||
sed -n '1,260p' .scratch/distributed-gguf-runtime/evidence/DGR-001/README.md
|
||||
sed -n '1,260p' .scratch/distributed-gguf-runtime/evidence/DGR-012/README.md
|
||||
sed -n '1,260p' .scratch/distributed-gguf-runtime/evidence/DGR-013/README.md
|
||||
sed -n '1,260p' .scratch/distributed-gguf-runtime/evidence/DGR-011/BLOCKED.md
|
||||
sed -n '1,260p' .scratch/distributed-gguf-runtime/evidence/DGR-010/BLOCKED.md
|
||||
find /run/media/popov/d/DEV/models /run/media/popov/d/DEV/llamacpp/llama.cpp/models -maxdepth 4 \( -iname '*llama*' -o -iname '*deepseek*' -o -iname '*dense*' -o -name '*.gguf' -o -name '*.safetensors' -o -name 'config.json' \)
|
||||
```
|
||||
|
||||
## Known limitations
|
||||
|
||||
- No certified dense-Llama artifact is mounted, so the real distributed safetensors-versus-GGUF comparison cannot be executed.
|
||||
- No immutable release-gate evidence can be produced without that artifact and the completed DGR-011 route comparison.
|
||||
- No code was changed in this iteration.
|
||||
|
||||
## Compatibility notes
|
||||
|
||||
- The DGR-001 contract remains the source of truth for thresholds and metric names.
|
||||
- Any future DGR-014 run must keep those thresholds unchanged and compare the same certified model/hardware/network scenario for both routes.
|
||||
|
||||
## Dependent-story handoff
|
||||
|
||||
- Finish `DGR-010` and `DGR-011` first with a certified dense-Llama artifact on mounted storage.
|
||||
- Then run the current distributed safetensors and distributed GGUF routes on the same comparable scenario, record the final numbers in `evidence/DGR-014/README.md`, and update the issue status only after the gate passes.
|
||||
@@ -1,78 +0,0 @@
|
||||
# DGR-015 — Blocked handoff
|
||||
|
||||
Status: blocked
|
||||
Date: 2026-07-16
|
||||
|
||||
## Blocker
|
||||
|
||||
This story cannot be completed in the current workspace state because its
|
||||
mandatory prerequisite, DGR-014, is still not passed.
|
||||
|
||||
Verified blocker chain:
|
||||
|
||||
- `.scratch/distributed-gguf-runtime/prd.json` still marks `DGR-014` as
|
||||
`"passes": false`, so DGR-015 is not released for completion.
|
||||
- `.scratch/distributed-gguf-runtime/evidence/DGR-014/BLOCKED.md` records the
|
||||
release-gate blocker: the certified dense-Llama artifact required for the
|
||||
comparable real-model comparison is not mounted on this machine.
|
||||
- `DGR-014` depends on `DGR-011`, which is also blocked because `DGR-010`
|
||||
cannot run without that same certified dense-Llama artifact.
|
||||
- The current codebase still fails closed for `qwen3` / `qwen3-moe` in
|
||||
`packages/node/meshnet_node/boundary_adapter.py`, which is correct for the
|
||||
current state but means no Qwen3 family recipe is certified yet.
|
||||
|
||||
## Verified current state
|
||||
|
||||
- Dense-Llama boundary semantics, Hot KV isolation, batching, and failure
|
||||
semantics are already implemented and covered by prior stories.
|
||||
- Qwen3 strings are present in tracker/model metadata, but they are not yet
|
||||
backed by a certified architecture adapter or real-model acceptance evidence.
|
||||
- No `evidence/DGR-015/README.md` exists yet because the acceptance criteria
|
||||
could not be completed.
|
||||
|
||||
## Commands run
|
||||
|
||||
```bash
|
||||
sed -n '1,260p' .claude/memory/MEMORY.md
|
||||
sed -n '1,260p' .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md
|
||||
sed -n '1,260p' .scratch/distributed-gguf-runtime/issues/15-add-and-certify-a-qwen3-qwen3-moe-adapter.md
|
||||
sed -n '1,260p' .scratch/distributed-gguf-runtime/architecture.md
|
||||
sed -n '1,260p' CONTEXT.md
|
||||
sed -n '1,260p' .scratch/distributed-gguf-runtime/prd.json
|
||||
sed -n '1,260p' .scratch/distributed-gguf-runtime/evidence/DGR-014/BLOCKED.md
|
||||
sed -n '1,260p' .scratch/distributed-gguf-runtime/evidence/DGR-013/README.md
|
||||
sed -n '1,260p' .scratch/distributed-gguf-runtime/evidence/DGR-012/README.md
|
||||
sed -n '1,260p' packages/node/meshnet_node/boundary_adapter.py
|
||||
sed -n '1,260p' packages/node/meshnet_node/model_catalog.py
|
||||
sed -n '1,220p' packages/node/meshnet_node/model_metadata.json
|
||||
sed -n '1,260p' packages/tracker/meshnet_tracker/capability.py
|
||||
sed -n '1,260p' packages/tracker/meshnet_tracker/server.py
|
||||
rg -n "qwen3|qwen3-moe|Qwen3|MoE|router|top-k|shared expert|shared_expert|expert" packages/node/meshnet_node packages/tracker/meshnet_tracker tests -g '!**/__pycache__/**'
|
||||
git status --short
|
||||
```
|
||||
|
||||
## Known limitations
|
||||
|
||||
- No certified dense-Llama artifact is mounted, so DGR-014 cannot complete and
|
||||
DGR-015 remains blocked behind it.
|
||||
- No real consumer-hardware Qwen3 acceptance run was possible in this workspace.
|
||||
- No code was changed in this iteration.
|
||||
|
||||
## Compatibility notes
|
||||
|
||||
- The current boundary adapter intentionally fails closed for uncertified
|
||||
architectures. That is the correct behavior until a dedicated Qwen3 adapter is
|
||||
implemented and certified.
|
||||
- Existing dense-Llama coverage and Hot KV semantics remain the source of truth
|
||||
for the shared protocol and cache behavior.
|
||||
|
||||
## Dependent-story handoff
|
||||
|
||||
- Finish `DGR-010`, `DGR-011`, and `DGR-014` first with a certified dense-Llama
|
||||
artifact on mounted storage.
|
||||
- Once the release gate passes, implement the Qwen3 family adapter as a separate
|
||||
certified architecture rather than by extending dense-Llama with unchecked name
|
||||
substitutions.
|
||||
- Record the real-model Qwen3 parity, admission, memory, and communication
|
||||
evidence in `evidence/DGR-015/README.md`, then update the issue status only
|
||||
after the gate passes.
|
||||
@@ -1,145 +0,0 @@
|
||||
# DGR-016 — Upstream llama.cpp collaboration package
|
||||
|
||||
Status: partial, blocked by DGR-010
|
||||
Date: 2026-07-16
|
||||
|
||||
## Summary
|
||||
|
||||
Assembled the upstream-facing collaboration package for llama.cpp without
|
||||
pulling Meshnet routing or control-plane logic into the upstream ask.
|
||||
|
||||
Durable outputs created for this story:
|
||||
|
||||
- `api-note.md` with the generic hook split and patch-per-concern proposal
|
||||
- `outreach.md` with a maintainer-facing draft for Georgi/llama.cpp
|
||||
|
||||
The package is grounded in the existing research artifacts and the already
|
||||
implemented deterministic tests for:
|
||||
|
||||
- range-aware GGUF ownership and introspection
|
||||
- architecture boundary input/output
|
||||
- layer-filtered KV/session ownership
|
||||
- reproducible pinned worker build wiring
|
||||
|
||||
The story itself remains blocked because DGR-010 is still marked `passes: false`
|
||||
and only has a blocked handoff, not a completed real-model acceptance README.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `.scratch/distributed-gguf-runtime/evidence/DGR-016/README.md`
|
||||
- `.scratch/distributed-gguf-runtime/evidence/DGR-016/api-note.md`
|
||||
- `.scratch/distributed-gguf-runtime/evidence/DGR-016/outreach.md`
|
||||
|
||||
## Commands run and real results
|
||||
|
||||
### Dependency and context review
|
||||
|
||||
```bash
|
||||
sed -n '1,260p' .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md
|
||||
sed -n '1,260p' .scratch/distributed-gguf-runtime/issues/16-produce-the-upstream-llama-cpp-collaboration-package.md
|
||||
sed -n '1,260p' .scratch/distributed-gguf-runtime/evidence/DGR-010/BLOCKED.md
|
||||
sed -n '1,260p' docs/adr/0024-distributed-gguf-runtime.md
|
||||
sed -n '1,260p' .scratch/distributed-gguf-runtime/architecture.md
|
||||
sed -n '1,260p' .scratch/distributed-gguf-runtime/decision-framework.md
|
||||
sed -n '1,260p' .scratch/distributed-gguf-runtime/implementation-strategy.md
|
||||
sed -n '1,260p' CONTEXT.md
|
||||
```
|
||||
|
||||
Result:
|
||||
|
||||
- confirmed the runtime target is a small pinned llama.cpp worker with Meshnet
|
||||
kept outside upstream
|
||||
- confirmed DGR-010 is still blocked because there is no certified dense-Llama
|
||||
artifact on mounted storage
|
||||
|
||||
### Package-relevant targeted pytest
|
||||
|
||||
```bash
|
||||
python -m pytest -q tests/test_llama_worker_build.py tests/test_gguf_backend.py tests/test_gguf_ownership.py tests/test_boundary_adapter.py tests/test_hot_kv_state.py
|
||||
```
|
||||
|
||||
Result:
|
||||
|
||||
- `50 passed in 0.90s`
|
||||
|
||||
### Broader focused pytest slice
|
||||
|
||||
```bash
|
||||
python -m pytest -q tests/test_llama_worker_build.py tests/test_native_shard_protocol.py tests/test_gguf_backend.py tests/test_boundary_adapter.py tests/test_gguf_ownership.py tests/test_hot_kv_state.py tests/test_kv_cache_distributed.py
|
||||
```
|
||||
|
||||
Result:
|
||||
|
||||
- `58 passed, 1 skipped, 9 failed, 12 errors in 1.27s`
|
||||
- failures were pre-existing environment issues, not this documentation-only
|
||||
package:
|
||||
- `tests/test_native_shard_protocol.py` imported generated protobuf code built
|
||||
against gencode 7.35.0 while the active runtime is 6.33.6
|
||||
- `tests/test_kv_cache_distributed.py` hit sandbox socket `PermissionError`
|
||||
when trying to bind localhost servers
|
||||
|
||||
### Research evidence review
|
||||
|
||||
```bash
|
||||
sed -n '1,260p' docs/research/distributed-gguf-landscape.md
|
||||
sed -n '1,260p' docs/research/distributed-gguf-github-followup.md
|
||||
sed -n '1,220p' .scratch/distributed-gguf-runtime/evidence/DGR-004/README.md
|
||||
sed -n '1,260p' .scratch/distributed-gguf-runtime/evidence/DGR-006/README.md
|
||||
sed -n '1,260p' .scratch/distributed-gguf-runtime/evidence/DGR-007/README.md
|
||||
sed -n '1,260p' .scratch/distributed-gguf-runtime/evidence/DGR-009/README.md
|
||||
```
|
||||
|
||||
Result:
|
||||
|
||||
- confirmed Nakshatra and prima.cpp are the right source/test donors for the
|
||||
upstream ask
|
||||
- confirmed the generic API surface is range loading, boundary I/O, and KV
|
||||
ownership, not Meshnet policy
|
||||
|
||||
### Package assembly
|
||||
|
||||
No code generation, downloads, or model execution were required for this story.
|
||||
The package is documentation-only and deterministic.
|
||||
|
||||
```bash
|
||||
python -m compileall -q packages tests
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Result:
|
||||
|
||||
- both commands exited 0
|
||||
|
||||
## Correctness / performance / hardware classification
|
||||
|
||||
- Correctness evidence: research-only, no live model execution
|
||||
- Performance evidence: none in this story
|
||||
- Hardware evidence: none in this story
|
||||
|
||||
## Known limitations and deferred work
|
||||
|
||||
- DGR-010 remains blocked, so this package cannot be treated as the final
|
||||
release-ready upstream handoff.
|
||||
- The outreach draft is human-ready but not sent.
|
||||
- The doc package does not change llama.cpp source code; it only prepares the
|
||||
upstream ask and test mapping.
|
||||
|
||||
## Compatibility / migration notes
|
||||
|
||||
- Exact upstream pin for the eventual patch series: `b3c9d1b846cc80a6360adb6aeaa4fcd8c4c8dcac`
|
||||
- The proposed patch split is:
|
||||
1. range-aware loading and ownership introspection
|
||||
2. boundary input/output and named tensor bundles
|
||||
3. layer-filtered KV and local sequence ownership
|
||||
- Meshnet routing, billing, relay transport, and volunteer-network policy stay
|
||||
outside llama.cpp.
|
||||
- The deterministic examples already exist in the tree and can be trimmed into
|
||||
upstream-facing MREs when the human maintainer sends the package.
|
||||
|
||||
## Dependent-story handoff
|
||||
|
||||
- DGR-010 must clear before any real-model validation can be cited as the final
|
||||
end-to-end proof for this upstream package.
|
||||
- Once DGR-010 has a completed evidence README, the package can be refreshed
|
||||
with the real-model context and sent to the llama.cpp maintainers as a
|
||||
smaller review bundle.
|
||||
@@ -1,90 +0,0 @@
|
||||
# DGR-016 API note: narrow llama.cpp hooks, no Meshnet policy
|
||||
|
||||
This note is the upstream-facing shape for the collaboration package.
|
||||
|
||||
## Goal
|
||||
|
||||
Keep the llama.cpp ask small:
|
||||
|
||||
- expose generic model-layer hooks that are useful to any local or remote
|
||||
layer-worker setup;
|
||||
- keep Meshnet routing, session ownership, billing, and relay transport out of
|
||||
llama.cpp;
|
||||
- preserve one patch per concern so the series rebases cleanly on the pinned
|
||||
upstream commit.
|
||||
|
||||
## Concern 1: range-aware loading and authoritative tensor ownership
|
||||
|
||||
Requested surface:
|
||||
|
||||
- accept a contiguous `[start_layer, end_layer)` range;
|
||||
- expose whether the worker owns embeddings, final norm, and final head;
|
||||
- make the loaded range authoritative from the model state, not from CLI
|
||||
claims;
|
||||
- allow unowned tensors to be absent rather than fabricated.
|
||||
|
||||
Why this is upstreamable:
|
||||
|
||||
- it is generic loader and introspection plumbing;
|
||||
- it helps any local partitioned inference mode;
|
||||
- it does not require any Meshnet identity, route, or transport type.
|
||||
|
||||
Minimal examples/tests:
|
||||
|
||||
- `tests/test_gguf_ownership.py`
|
||||
- `tests/test_llama_worker_build.py`
|
||||
|
||||
## Concern 2: architecture boundary input/output
|
||||
|
||||
Requested surface:
|
||||
|
||||
- accept a versioned boundary bundle carrying one or more named tensors;
|
||||
- support an unnormalized residual stream as the intermediate handoff;
|
||||
- keep final norm, LM head, and sampling on the tail shard only;
|
||||
- keep the bundle format explicit about name, shape, dtype, byte order, and
|
||||
fragments.
|
||||
|
||||
Why this is upstreamable:
|
||||
|
||||
- it matches both dense Llama and other certified adapter families;
|
||||
- it does not assume Meshnet or any specific wire protocol;
|
||||
- it gives a stable ABI for a layer-worker boundary.
|
||||
|
||||
Minimal examples/tests:
|
||||
|
||||
- `tests/test_boundary_adapter.py`
|
||||
- `tests/test_native_shard_protocol.py`
|
||||
|
||||
## Concern 3: layer-filtered KV and session mapping
|
||||
|
||||
Requested surface:
|
||||
|
||||
- let the worker own KV only for its layer range;
|
||||
- map a stable session/context identifier to the local sequence;
|
||||
- allow cache miss, stale epoch, truncate, release, and eviction semantics;
|
||||
- reject incompatible cache recipes rather than trying to heal them silently.
|
||||
|
||||
Why this is upstreamable:
|
||||
|
||||
- it is a local sequence/KV API, not a network scheduler;
|
||||
- it is useful to any supervisor that needs one process per layer range;
|
||||
- it keeps session semantics outside llama.cpp while still making the worker
|
||||
stateful in a controlled way.
|
||||
|
||||
Minimal examples/tests:
|
||||
|
||||
- `tests/test_hot_kv_state.py`
|
||||
- `tests/test_kv_cache_distributed.py`
|
||||
|
||||
## Suggested patch split
|
||||
|
||||
Keep the series narrow and independently reviewable against the exact pinned
|
||||
commit `b3c9d1b846cc80a6360adb6aeaa4fcd8c4c8dcac`:
|
||||
|
||||
1. `range-aware-loading` and ownership introspection.
|
||||
2. `boundary-input-output` and named tensor bundle handoff.
|
||||
3. `layer-filtered-kv` and sequence ownership.
|
||||
|
||||
The current Meshnet worker scaffold remains a project-owned wrapper and is not
|
||||
part of the upstream ask.
|
||||
|
||||
@@ -1,43 +0,0 @@
|
||||
# DGR-016 outreach draft
|
||||
|
||||
Subject: Narrow llama.cpp hooks for range loading, boundary I/O, and local KV ownership
|
||||
|
||||
Hi Georgi and llama.cpp maintainers,
|
||||
|
||||
We have been building a distributed GGUF route on top of a Meshnet control
|
||||
plane, and the narrow upstreamable seam is now clear enough to summarize.
|
||||
|
||||
We are not asking llama.cpp to own Meshnet routing, billing, relay transport,
|
||||
or any volunteer-network policy. The upstream ask is limited to generic local
|
||||
hooks that make partitioned inference easier to implement and easier to review:
|
||||
|
||||
1. Range-aware loading and ownership introspection for contiguous layer ranges.
|
||||
2. Architecture-defined boundary input/output using an explicit named-tensor
|
||||
bundle.
|
||||
3. Layer-filtered KV ownership and stable local sequence mapping.
|
||||
|
||||
Why we think this is generally useful:
|
||||
|
||||
- Nakshatra already demonstrates the value of a narrow layer-worker seam and
|
||||
partial GGUF loading.
|
||||
- prima.cpp shows the same idea from a different angle with selective loading,
|
||||
local KV, and boundary residual transport.
|
||||
- Both projects suggest the same conclusion: the missing API is not Meshnet
|
||||
specific, it is a local runtime seam that any layer-partitioned supervisor can
|
||||
use.
|
||||
|
||||
The package we would upstream is intentionally split into one concern per patch
|
||||
so review stays small:
|
||||
|
||||
- range-aware loading and tensor ownership;
|
||||
- boundary I/O for intermediate residual state;
|
||||
- layer-filtered KV and sequence ownership.
|
||||
|
||||
If useful, we can send the concrete MRE/test mapping next. We already have
|
||||
deterministic examples covering the loader, boundary contract, and KV/session
|
||||
semantics in the Meshnet tree, and we can trim them into upstream-focused test
|
||||
cases.
|
||||
|
||||
Thanks,
|
||||
Meshnet maintainers
|
||||
|
||||
114
.scratch/distributed-gguf-runtime/evidence/DGR-017/README.md
Normal file
114
.scratch/distributed-gguf-runtime/evidence/DGR-017/README.md
Normal file
@@ -0,0 +1,114 @@
|
||||
# DGR-017 evidence — superseded backlog cleanup
|
||||
|
||||
**Completed:** 2026-07-16
|
||||
**Branch:** `ralph/distributed-gguf-runtime`
|
||||
**Planning checkpoint before cleanup:** `81b1fa6`
|
||||
**Authority:** `.scratch/distributed-gguf-runtime/prd.json`
|
||||
|
||||
## Outcome
|
||||
|
||||
The old DGR-001…016 completion claims and active artifacts were reconciled against the live branch. No old pass state transferred to the new implementation roadmap.
|
||||
|
||||
The active `packages/` and `tests/` trees were restored exactly to `origin/master`. The branch therefore no longer exposes a nominal GGUF startup path backed by unimplemented transport methods, a protobuf-only native scaffold, or isolated synthetic scheduler/cache/failure modules as if they were a working distributed GGUF runtime.
|
||||
|
||||
## Classification and disposition
|
||||
|
||||
### Retained
|
||||
|
||||
- Accepted ADRs and repository research, including `docs/research/colibri-implementation-audit.md`.
|
||||
- The authoritative 55-story roadmap `DGR-017…071` and its generated issue specifications.
|
||||
- The real public-relay smoke benchmark, moved with provenance to `legacy-public-relay-smoke-benchmark.json`.
|
||||
- Git history containing the complete superseded implementation/reference work.
|
||||
|
||||
### Removed from the active tree
|
||||
|
||||
- Legacy issue specifications DGR-001…016 and their stale/blocked/synthetic evidence directories.
|
||||
- The nonfunctional `gguf_backend` startup path whose gRPC execution methods raised not-implemented errors.
|
||||
- Synthetic/reference-only boundary, Hot KV, scheduler, failure, recipe, ownership, and native-protocol modules that were not a real llama.cpp Shard runtime.
|
||||
- The protobuf round-trip-only native scaffold, placeholder llama.cpp patch, generated bindings/build workspace, and associated tests.
|
||||
- Tracker/admission/source modifications coupled to that superseded scaffold.
|
||||
|
||||
### Confirmed absent and still required
|
||||
|
||||
- Real standalone C++ gRPC Shard worker.
|
||||
- Exact pinned llama.cpp manifest and verified patch stack.
|
||||
- Range-aware GGUF tensor ownership and real ranged execution.
|
||||
- Real Shard-local llama.cpp KV/V4 auxiliary state.
|
||||
- DeepSeek V4 boundary adapter and ranged parity.
|
||||
- Real multi-machine DeepSeek V4 alpha or beta acceptance.
|
||||
|
||||
These remain `passes: false` in DGR-018…071.
|
||||
|
||||
## Before-cleanup baseline
|
||||
|
||||
Command:
|
||||
|
||||
```bash
|
||||
.venv-rocm/bin/python -m pytest -q \
|
||||
tests/test_performance_contract.py tests/test_native_shard_protocol.py \
|
||||
tests/test_gguf_ownership.py tests/test_boundary_adapter.py \
|
||||
tests/test_hot_kv_state.py tests/test_gguf_backend.py \
|
||||
tests/test_batch_scheduler.py tests/test_failure_semantics.py \
|
||||
tests/test_llama_worker_build.py tests/test_node_admission.py \
|
||||
tests/test_node_capability.py tests/test_tracker_capability_admission.py
|
||||
```
|
||||
|
||||
Result:
|
||||
|
||||
```text
|
||||
216 passed, 2 skipped, 1 failed, 1 warning
|
||||
```
|
||||
|
||||
The failure was a synthetic capability-test helper `KeyError: 'compatibility_fingerprint'`. The warning was a pre-existing heartbeat-thread `SystemExit` warning.
|
||||
|
||||
## Cleanup verification
|
||||
|
||||
### Source equality
|
||||
|
||||
Command:
|
||||
|
||||
```bash
|
||||
git diff --quiet origin/master -- packages tests
|
||||
```
|
||||
|
||||
Result:
|
||||
|
||||
```text
|
||||
packages_tests_match_origin_master=yes
|
||||
```
|
||||
|
||||
The staged cleanup removes approximately 15.3k obsolete source/test/evidence lines from the active branch.
|
||||
|
||||
### Cleanup-relevant regression suite
|
||||
|
||||
Command:
|
||||
|
||||
```bash
|
||||
.venv-rocm/bin/python -m pytest -q \
|
||||
tests/test_node_admission.py tests/test_node_capability.py \
|
||||
tests/test_tracker_capability_admission.py \
|
||||
tests/test_kv_cache_distributed.py tests/test_real_distributed_inference.py
|
||||
```
|
||||
|
||||
Result:
|
||||
|
||||
```text
|
||||
119 passed, 2 skipped, 1 warning in 15.90s
|
||||
```
|
||||
|
||||
The warning is the same pre-existing heartbeat-thread `SystemExit` warning.
|
||||
|
||||
### Known `origin/master` limitations
|
||||
|
||||
The wider routing run produced `210 passed, 2 skipped, 4 failed, 1 warning`. Each failure reproduced individually while `packages/` and `tests/` matched `origin/master` exactly:
|
||||
|
||||
- `test_tracker_models_endpoint_lists_registered_hf_repo_and_short_name_alias`
|
||||
- `test_torch_node_applies_tracker_load_shard_directive`
|
||||
- `test_shard_heal_cycle_surviving_node_covers_dead_peers_gap`
|
||||
- `test_a_node_with_an_unusable_precision_covers_no_layers`
|
||||
|
||||
They are recorded as pre-existing baseline defects and were not repaired or hidden by this cleanup story.
|
||||
|
||||
## Dependency handoff
|
||||
|
||||
DGR-018 and later stories must start from the cleaned upstream-equivalent runtime tree. Reuse concepts from superseded commits only by explicitly porting the smallest verified slice under the new story’s contracts, tests, and evidence gates. Git history is provenance, not completion evidence.
|
||||
@@ -1,6 +1,6 @@
|
||||
# Distributed GGUF Runtime evidence
|
||||
|
||||
> **Specification status:** planning artifacts only. No distributed GGUF runtime is implemented by this materialization, no story has completion credit, and legacy files remain for the DGR-017 audit. `prd.json` is authoritative.
|
||||
> **Specification status:** planning artifacts only. No distributed GGUF runtime is implemented. DGR-017 cleanup is complete; no runtime implementation story has completion credit. `prd.json` is authoritative.
|
||||
|
||||
## Authority and classes
|
||||
|
||||
|
||||
Reference in New Issue
Block a user