Compare commits
1 Commits
ralph/dist
...
ralph/deep
| Author | SHA1 | Date | |
|---|---|---|---|
|
f102be1098 |
@@ -8,15 +8,6 @@ metadata:
|
|||||||
|
|
||||||
# Project Status (2026-07-13)
|
# Project Status (2026-07-13)
|
||||||
|
|
||||||
## Selected-node model placement (2026-07-14)
|
|
||||||
|
|
||||||
- Admin Model placement now opens a node selector for load and release; the control-plane accepts optional `node_id` and targets only that registry assignment. Multi-model serving remains supported through `ADD_SHARD` and `max_loaded_shards`.
|
|
||||||
- Total node pool resource values are rendered from `/v1/network/map`'s `node.capacity` contract. Route selection remains assignment/capability/throughput/queue based; capacity is used for placement and falls back to tracker defaults only if a node truly omits it.
|
|
||||||
|
|
||||||
## Distributed inference performance (2026-07-14)
|
|
||||||
|
|
||||||
`DIP-001` is done in `.scratch/distributed-inference-performance/`: the deterministic two-node Route Session stub benchmark covers direct/relay plus cached/stateless prefill and decode. Its JSON and concise summary explicitly attribute model execution, activation encode/decode, compression, connection setup, relay queueing, local HTTP forwarding, and end-to-end seam latency. `PYTHONPATH=packages/node pytest -q tests/test_route_session_benchmark.py` passed (7); the fixture assertion checks output-token identity and connection attempts.
|
|
||||||
|
|
||||||
> Doc reconciliation 2026-07-13: `docs/prd.json` tracks US-001…US-050 (048 memory budget, 049 mainnet pilot, 050 Qwen demand placement). ADRs 0025–0026 added (TAI phase B/C, assignment ownership).
|
> Doc reconciliation 2026-07-13: `docs/prd.json` tracks US-001…US-050 (048 memory budget, 049 mainnet pilot, 050 Qwen demand placement). ADRs 0025–0026 added (TAI phase B/C, assignment ownership).
|
||||||
|
|
||||||
All 35 user stories in docs/prd.json are done (35/35), including the reward-system arc US-030…US-035 completed 2026-07-02:
|
All 35 user stories in docs/prd.json are done (35/35), including the reward-system arc US-030…US-035 completed 2026-07-02:
|
||||||
|
|||||||
@@ -12,10 +12,4 @@ Provide an opt-in, admin-only tracker Dashboard Testing tab that dynamically dis
|
|||||||
- One active run.
|
- One active run.
|
||||||
- Real inference stays separately environment-gated and excluded from default suites.
|
- Real inference stays separately environment-gated and excluded from default suites.
|
||||||
|
|
||||||
## Operator workflow
|
|
||||||
|
|
||||||
See [`docs/dev/dashboard-test-runner.md`](../../docs/dev/dashboard-test-runner.md)
|
|
||||||
for launch configuration, default safe suites vs the gated real-inference suite,
|
|
||||||
and required environment variables.
|
|
||||||
|
|
||||||
See `prd.json` for executable Ralph user stories and acceptance criteria.
|
See `prd.json` for executable Ralph user stories and acceptance criteria.
|
||||||
|
|||||||
@@ -51,16 +51,15 @@
|
|||||||
"uv run pytest tests/test_dashboard.py tests/test_dynamic_routing.py -q passes."
|
"uv run pytest tests/test_dashboard.py tests/test_dynamic_routing.py -q passes."
|
||||||
],
|
],
|
||||||
"priority": 3,
|
"priority": 3,
|
||||||
"passes": true,
|
"passes": false,
|
||||||
"notes": "Do not reintroduce --enable-test-runner without implementing its CLI argument in US-001.",
|
"notes": "Do not reintroduce --enable-test-runner without implementing its CLI argument in US-001.",
|
||||||
"dependsOn": [
|
"dependsOn": [
|
||||||
"US-001",
|
"US-001",
|
||||||
"US-002"
|
"US-002"
|
||||||
],
|
]
|
||||||
"completionNotes": "Completed by agent"
|
|
||||||
}
|
}
|
||||||
],
|
],
|
||||||
"metadata": {
|
"metadata": {
|
||||||
"updatedAt": "2026-07-12T01:58:06.286Z"
|
"updatedAt": "2026-07-11T17:02:30.520Z"
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -1,127 +0,0 @@
|
|||||||
# DGR-001 — performance contract baseline
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `packages/node/meshnet_node/performance_contract.py`
|
|
||||||
- `tests/test_performance_contract.py`
|
|
||||||
- `.scratch/distributed-gguf-runtime/issues/01-lock-the-safetensors-versus-gguf-performance-contract.md`
|
|
||||||
- `.scratch/distributed-gguf-runtime/evidence/DGR-001/performance-contract.json`
|
|
||||||
|
|
||||||
## What this slice does
|
|
||||||
|
|
||||||
- Locks the DGR-001 benchmark contract in code.
|
|
||||||
- Pins the architecture-aligned baseline to **DeepSeek-V2-Lite-Chat** (`deepseek2`).
|
|
||||||
- Uses the same model on both sides of the comparison:
|
|
||||||
- **safetensors:** `deepseek-ai/DeepSeek-V2-Lite-Chat` in **BF16**
|
|
||||||
- **GGUF:** `second-state/DeepSeek-V2-Lite-Chat-GGUF` in **Q2_K**
|
|
||||||
- Exposes a machine-readable JSON contract with:
|
|
||||||
- benchmark lanes for `transformers` safetensors and `llama.cpp` GGUF on **CPU** and **GPU**
|
|
||||||
- concurrency levels `1` and `4`
|
|
||||||
- the required metrics list
|
|
||||||
- an explicit stop condition for “no meaningful speed or fit benefit”
|
|
||||||
- Adds a deterministic stub benchmark report so the contract now has an executable report shape end to end.
|
|
||||||
|
|
||||||
## Recent benchmark runner slice
|
|
||||||
|
|
||||||
The runner currently uses a deterministic stub backend to exercise the comparison matrix without downloading a model. It emits:
|
|
||||||
|
|
||||||
- `.scratch/distributed-gguf-runtime/evidence/DGR-001/performance-contract.json`
|
|
||||||
- `.scratch/distributed-gguf-runtime/evidence/DGR-001/stub-benchmark-report.json`
|
|
||||||
|
|
||||||
The report includes per-device comparisons for:
|
|
||||||
|
|
||||||
- `transformers-safetensors-cpu` vs `llama-cpp-gguf-cpu`
|
|
||||||
- `transformers-safetensors-gpu` vs `llama-cpp-gguf-gpu`
|
|
||||||
|
|
||||||
and records the memory metric (`rss_bytes` on CPU, `vram_bytes` on GPU), decode speedup, artifact ratio, and output drift.
|
|
||||||
|
|
||||||
## Live endpoint CLI wiring
|
|
||||||
|
|
||||||
The contract CLI can now drive the live endpoint runner. Passing one `--live-endpoint LANE_ID=URL` mapping per contract lane (plus `--live-benchmark-out`) invokes `run_real_model_endpoint_benchmark` against already-running OpenAI-compatible servers and writes the report using the same schema as the stub:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
PYTHONPATH=packages/node python -m meshnet_node.performance_contract \
|
|
||||||
--live-endpoint transformers-safetensors-cpu=http://127.0.0.1:8001 \
|
|
||||||
--live-endpoint llama-cpp-gguf-cpu=http://127.0.0.1:8002 \
|
|
||||||
--live-endpoint transformers-safetensors-gpu=http://127.0.0.1:8003 \
|
|
||||||
--live-endpoint llama-cpp-gguf-gpu=http://127.0.0.1:8004 \
|
|
||||||
--live-benchmark-out .scratch/distributed-gguf-runtime/evidence/DGR-001/live-benchmark-report.json
|
|
||||||
```
|
|
||||||
|
|
||||||
`--live-model` overrides the model name sent in requests (defaults to the contract's safetensors repo). Without any `--live-endpoint` flags the CLI behaves exactly as before: it writes the contract JSON and, with `--benchmark-out`, the deterministic stub report.
|
|
||||||
|
|
||||||
## Exact commands and real results
|
|
||||||
|
|
||||||
### Targeted tests
|
|
||||||
|
|
||||||
```bash
|
|
||||||
PYTHONPATH=packages/node pytest -q tests/test_performance_contract.py tests/test_route_session_benchmark.py
|
|
||||||
```
|
|
||||||
|
|
||||||
Result: `19 passed in 0.11s`
|
|
||||||
|
|
||||||
### Contract artifact generation
|
|
||||||
|
|
||||||
```bash
|
|
||||||
PYTHONPATH=packages/node python -m meshnet_node.performance_contract --json-out .scratch/distributed-gguf-runtime/evidence/DGR-001/performance-contract.json
|
|
||||||
```
|
|
||||||
|
|
||||||
Result: wrote `.scratch/distributed-gguf-runtime/evidence/DGR-001/performance-contract.json`
|
|
||||||
|
|
||||||
### Python compile check
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python -m compileall packages/node/meshnet_node/performance_contract.py tests/test_performance_contract.py
|
|
||||||
```
|
|
||||||
|
|
||||||
Result: passed
|
|
||||||
|
|
||||||
## Public relay smoke benchmark (2026-07-15)
|
|
||||||
|
|
||||||
A real streamed request was run through the public tracker — **not** by connecting directly to the private node address:
|
|
||||||
|
|
||||||
```text
|
|
||||||
https://meshnet.2.d-popov.com/v1/chat/completions
|
|
||||||
-> wss://meshnet.2.d-popov.com/ws
|
|
||||||
-> wss://meshnet.2.d-popov.com/rpc/7j77FsPY1evV8tuf-7000
|
|
||||||
-> local CUDA node, Qwen/Qwen2.5-0.5B-Instruct layers 0-23
|
|
||||||
```
|
|
||||||
|
|
||||||
The local public-tracker node had an expired proof and a wedged HTTP server. A graceful restart refreshed its CUDA capability proof in `336 ms`, restored `admitted`/`routable` status, and reconnected its relay endpoint.
|
|
||||||
|
|
||||||
Measured streaming results after recovery:
|
|
||||||
|
|
||||||
| metric | result |
|
|
||||||
| --- | ---: |
|
|
||||||
| warm-up TTFT | 420.80 ms |
|
|
||||||
| warm-up elapsed | 610.23 ms |
|
|
||||||
| p50 TTFT (3 runs) | 288.26 ms |
|
|
||||||
| p50 elapsed (3 runs) | 363.20 ms |
|
|
||||||
| tracker-recorded relay throughput | 58.18-65.25 tok/s |
|
|
||||||
| HTTP status | 200 for all runs |
|
|
||||||
|
|
||||||
The tracker recorded `relay: true` and the local node ID `7j77FsPY-b32476219492` for each completion. Full redacted evidence is in `public-relay-smoke-benchmark.json`.
|
|
||||||
|
|
||||||
The other connected node is still alive but **not routable** because its capability proof is stale. It must revalidate before a multi-node shard/relay test can run.
|
|
||||||
|
|
||||||
## Limitations
|
|
||||||
|
|
||||||
- This slice still uses a deterministic stub backend for the core comparison matrix.
|
|
||||||
- It now also includes a live endpoint runner, reachable from the CLI via `--live-endpoint`/`--live-benchmark-out`, that fans out one OpenAI-compatible request per lane when the caller provides endpoints; the CLI does not start those servers.
|
|
||||||
- It does **not** download or run a real model from within the repo.
|
|
||||||
- Real safetensors vs GGUF execution, TTFT/prefill/decode measurements, RSS/VRAM capture, and output-drift comparison are still to be implemented against the contract.
|
|
||||||
|
|
||||||
## Compatibility notes
|
|
||||||
|
|
||||||
- The contract stays on the DeepSeek2 family to remain close to the DeepSeek-V4-Flash end goal.
|
|
||||||
- A smaller non-DeepSeek model can still be used later for loader-plumbing smoke tests, but it does not replace this baseline.
|
|
||||||
- Model artifacts must stay on the mounted drive and not under `/home`.
|
|
||||||
|
|
||||||
## Dependent-story handoff
|
|
||||||
|
|
||||||
Next implementation work should attach to this contract and add the live benchmark runner that actually compares:
|
|
||||||
|
|
||||||
1. current Transformers/safetensors recipe
|
|
||||||
2. whole-model llama.cpp GGUF recipe
|
|
||||||
|
|
||||||
using the same model architecture/revision and the same prompt/context/concurrency settings.
|
|
||||||
@@ -1,75 +0,0 @@
|
|||||||
{
|
|
||||||
"benchmark_lanes": [
|
|
||||||
{
|
|
||||||
"concurrency_levels": [
|
|
||||||
1,
|
|
||||||
4
|
|
||||||
],
|
|
||||||
"device": "cpu",
|
|
||||||
"id": "transformers-safetensors-cpu",
|
|
||||||
"recipe": "current safetensors recipe",
|
|
||||||
"runtime": "transformers"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"concurrency_levels": [
|
|
||||||
1,
|
|
||||||
4
|
|
||||||
],
|
|
||||||
"device": "cpu",
|
|
||||||
"id": "llama-cpp-gguf-cpu",
|
|
||||||
"recipe": "whole-model GGUF recipe",
|
|
||||||
"runtime": "llama.cpp"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"concurrency_levels": [
|
|
||||||
1,
|
|
||||||
4
|
|
||||||
],
|
|
||||||
"device": "gpu",
|
|
||||||
"id": "transformers-safetensors-gpu",
|
|
||||||
"recipe": "current safetensors recipe",
|
|
||||||
"runtime": "transformers"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"concurrency_levels": [
|
|
||||||
1,
|
|
||||||
4
|
|
||||||
],
|
|
||||||
"device": "gpu",
|
|
||||||
"id": "llama-cpp-gguf-gpu",
|
|
||||||
"recipe": "whole-model GGUF recipe",
|
|
||||||
"runtime": "llama.cpp"
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"metrics": [
|
|
||||||
"ttft_ms",
|
|
||||||
"prefill_tok_per_sec",
|
|
||||||
"decode_tok_per_sec",
|
|
||||||
"p50_latency_ms",
|
|
||||||
"p95_latency_ms",
|
|
||||||
"aggregate_throughput_tok_per_sec",
|
|
||||||
"rss_bytes",
|
|
||||||
"vram_bytes",
|
|
||||||
"artifact_bytes",
|
|
||||||
"failure_count",
|
|
||||||
"output_drift"
|
|
||||||
],
|
|
||||||
"model_target": {
|
|
||||||
"architecture": "deepseek2",
|
|
||||||
"comparison_policy": "same model/revision, closest practical low-footprint precision pair: BF16 safetensors versus Q2_K GGUF",
|
|
||||||
"gguf_quant": "Q2_K",
|
|
||||||
"gguf_repo": "second-state/DeepSeek-V2-Lite-Chat-GGUF",
|
|
||||||
"gguf_size_gb": 6.43,
|
|
||||||
"name": "DeepSeek-V2-Lite-Chat",
|
|
||||||
"rationale": "Smallest DeepSeek-family benchmark anchor that still points toward DeepSeek-V4-Flash; keeps the runtime on the DeepSeek2 path instead of falling back to a tiny but architecture-mismatched smoke model.",
|
|
||||||
"safetensors_precision": "bfloat16",
|
|
||||||
"safetensors_repo": "deepseek-ai/DeepSeek-V2-Lite-Chat"
|
|
||||||
},
|
|
||||||
"notes": [
|
|
||||||
"Real model execution stays opt-in and must keep model artifacts on the mounted drive.",
|
|
||||||
"Use the tiny fallback only for loader plumbing smoke tests; it does not replace the architecture-aligned baseline."
|
|
||||||
],
|
|
||||||
"schema_version": 1,
|
|
||||||
"stop_condition": "Stop if GGUF does not provide a meaningful speed or fit benefit over the safetensors baseline for the chosen DeepSeek-family model target.",
|
|
||||||
"story_id": "DGR-001"
|
|
||||||
}
|
|
||||||
@@ -1,83 +0,0 @@
|
|||||||
{
|
|
||||||
"schema_version": 1,
|
|
||||||
"executed_at_utc": "2026-07-15T10:41:14Z",
|
|
||||||
"test_kind": "public-relay-single-node-streaming-smoke-benchmark",
|
|
||||||
"target": {
|
|
||||||
"public_chat_endpoint": "https://meshnet.2.d-popov.com/v1/chat/completions",
|
|
||||||
"relay_url": "wss://meshnet.2.d-popov.com/ws",
|
|
||||||
"model": "qwen2.5-0.5b-instruct",
|
|
||||||
"quantization": "bfloat16"
|
|
||||||
},
|
|
||||||
"recovery": {
|
|
||||||
"problem": "The local node's capability proof had expired and its port-7000 HTTP server had wedged with CLOSE-WAIT sockets.",
|
|
||||||
"action": "Gracefully restarted the local public-tracker meshnet-node process on port 7000.",
|
|
||||||
"startup_validation": {
|
|
||||||
"device": "cuda",
|
|
||||||
"capability_proof_ms": 336,
|
|
||||||
"node_id": "7j77FsPY-b32476219492",
|
|
||||||
"relay_addr": "wss://meshnet.2.d-popov.com/rpc/7j77FsPY1evV8tuf-7000"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"tracker_admission_after_recovery": {
|
|
||||||
"node_id": "7j77FsPY-b32476219492",
|
|
||||||
"alive": true,
|
|
||||||
"status": "ready",
|
|
||||||
"capability_state": "admitted",
|
|
||||||
"routable": true,
|
|
||||||
"route_hops": 1
|
|
||||||
},
|
|
||||||
"client_measurements": {
|
|
||||||
"warmup": {
|
|
||||||
"http_status": 200,
|
|
||||||
"ttft_ms": 420.8,
|
|
||||||
"elapsed_ms": 610.23,
|
|
||||||
"response_text": "MeshNet Relay Benchmark Passed"
|
|
||||||
},
|
|
||||||
"runs": [
|
|
||||||
{
|
|
||||||
"run": 1,
|
|
||||||
"ttft_ms": 376.04,
|
|
||||||
"elapsed_ms": 458.65,
|
|
||||||
"response_text": "relay benchmark pass"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"run": 2,
|
|
||||||
"ttft_ms": 258.33,
|
|
||||||
"elapsed_ms": 336.71,
|
|
||||||
"response_text": "relay benchmark pass"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"run": 3,
|
|
||||||
"ttft_ms": 288.26,
|
|
||||||
"elapsed_ms": 363.2,
|
|
||||||
"response_text": "relay benchmark pass"
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"p50_ttft_ms": 288.26,
|
|
||||||
"p50_elapsed_ms": 363.2
|
|
||||||
},
|
|
||||||
"tracker_relay_evidence": [
|
|
||||||
{
|
|
||||||
"status": 200,
|
|
||||||
"relay": true,
|
|
||||||
"node_id": "7j77FsPY-b32476219492",
|
|
||||||
"tokens": 11,
|
|
||||||
"elapsed_seconds": 0.1686,
|
|
||||||
"tokens_per_sec": 65.2541
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"status": 200,
|
|
||||||
"relay": true,
|
|
||||||
"node_id": "7j77FsPY-b32476219492",
|
|
||||||
"tokens": 11,
|
|
||||||
"elapsed_seconds": 0.1891,
|
|
||||||
"tokens_per_sec": 58.1799
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"scope_and_remaining_work": {
|
|
||||||
"validated": "Public HTTPS chat endpoint routed a streaming request through the tracker relay to the local CUDA node and completed with HTTP 200.",
|
|
||||||
"not_validated": "Two-node shard routing was not run because the remote node 5gMLrmyB-88f5cba044d0 still had an expired capability proof and was not routable.",
|
|
||||||
"next_gate": "Refresh the remote node capability proof, then load a multi-node-compatible assignment and repeat the benchmark through the public tracker relay."
|
|
||||||
},
|
|
||||||
"reproduction": "Use a valid bearer API key with the public /v1/chat/completions endpoint and stream a short qwen2.5-0.5b-instruct request. Do not connect directly to private node HTTP endpoints; the tracker relay is the required path."
|
|
||||||
}
|
|
||||||
@@ -1,247 +0,0 @@
|
|||||||
{
|
|
||||||
"comparisons": {
|
|
||||||
"cpu": {
|
|
||||||
"artifact_bytes_ratio": 0.2048,
|
|
||||||
"decode_speedup": 2.3333,
|
|
||||||
"gguf_benefit": true,
|
|
||||||
"gguf_lane": "llama-cpp-gguf-cpu",
|
|
||||||
"memory_bytes_ratio": 0.2152,
|
|
||||||
"memory_metric": "rss_bytes",
|
|
||||||
"output_drift": 0.0,
|
|
||||||
"safetensors_lane": "transformers-safetensors-cpu",
|
|
||||||
"ttft_speedup": 1.8947
|
|
||||||
},
|
|
||||||
"gpu": {
|
|
||||||
"artifact_bytes_ratio": 0.2048,
|
|
||||||
"decode_speedup": 1.5294,
|
|
||||||
"gguf_benefit": true,
|
|
||||||
"gguf_lane": "llama-cpp-gguf-gpu",
|
|
||||||
"memory_bytes_ratio": 0.2273,
|
|
||||||
"memory_metric": "vram_bytes",
|
|
||||||
"output_drift": 0.0,
|
|
||||||
"safetensors_lane": "transformers-safetensors-gpu",
|
|
||||||
"ttft_speedup": 1.6154
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"lanes": [
|
|
||||||
{
|
|
||||||
"concurrency_levels": [
|
|
||||||
1,
|
|
||||||
4
|
|
||||||
],
|
|
||||||
"device": "cpu",
|
|
||||||
"id": "transformers-safetensors-cpu",
|
|
||||||
"output_tokens": [
|
|
||||||
"mesh",
|
|
||||||
"activation",
|
|
||||||
"seam",
|
|
||||||
"baseline"
|
|
||||||
],
|
|
||||||
"recipe": "current safetensors recipe",
|
|
||||||
"results": [
|
|
||||||
{
|
|
||||||
"concurrency": 1,
|
|
||||||
"metrics": {
|
|
||||||
"aggregate_throughput_tok_per_sec": 6.0,
|
|
||||||
"artifact_bytes": 33715493273,
|
|
||||||
"decode_tok_per_sec": 6.0,
|
|
||||||
"failure_count": 0,
|
|
||||||
"output_drift": 0.0,
|
|
||||||
"p50_latency_ms": 166.6667,
|
|
||||||
"p95_latency_ms": 208.3334,
|
|
||||||
"prefill_tok_per_sec": 45.0,
|
|
||||||
"rss_bytes": 35433480192,
|
|
||||||
"ttft_ms": 1800.0,
|
|
||||||
"vram_bytes": 0
|
|
||||||
}
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"concurrency": 4,
|
|
||||||
"metrics": {
|
|
||||||
"aggregate_throughput_tok_per_sec": 20.4,
|
|
||||||
"artifact_bytes": 33715493273,
|
|
||||||
"decode_tok_per_sec": 5.1,
|
|
||||||
"failure_count": 0,
|
|
||||||
"output_drift": 0.0,
|
|
||||||
"p50_latency_ms": 196.0784,
|
|
||||||
"p95_latency_ms": 245.098,
|
|
||||||
"prefill_tok_per_sec": 38.25,
|
|
||||||
"rss_bytes": 35433480192,
|
|
||||||
"ttft_ms": 2340.0,
|
|
||||||
"vram_bytes": 0
|
|
||||||
}
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"runtime": "transformers"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"concurrency_levels": [
|
|
||||||
1,
|
|
||||||
4
|
|
||||||
],
|
|
||||||
"device": "cpu",
|
|
||||||
"id": "llama-cpp-gguf-cpu",
|
|
||||||
"output_tokens": [
|
|
||||||
"mesh",
|
|
||||||
"activation",
|
|
||||||
"seam",
|
|
||||||
"baseline"
|
|
||||||
],
|
|
||||||
"recipe": "whole-model GGUF recipe",
|
|
||||||
"results": [
|
|
||||||
{
|
|
||||||
"concurrency": 1,
|
|
||||||
"metrics": {
|
|
||||||
"aggregate_throughput_tok_per_sec": 14.0,
|
|
||||||
"artifact_bytes": 6904159928,
|
|
||||||
"decode_tok_per_sec": 14.0,
|
|
||||||
"failure_count": 0,
|
|
||||||
"output_drift": 0.0,
|
|
||||||
"p50_latency_ms": 71.4286,
|
|
||||||
"p95_latency_ms": 89.2858,
|
|
||||||
"prefill_tok_per_sec": 90.0,
|
|
||||||
"rss_bytes": 7623566950,
|
|
||||||
"ttft_ms": 950.0,
|
|
||||||
"vram_bytes": 0
|
|
||||||
}
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"concurrency": 4,
|
|
||||||
"metrics": {
|
|
||||||
"aggregate_throughput_tok_per_sec": 47.6,
|
|
||||||
"artifact_bytes": 6904159928,
|
|
||||||
"decode_tok_per_sec": 11.9,
|
|
||||||
"failure_count": 0,
|
|
||||||
"output_drift": 0.0,
|
|
||||||
"p50_latency_ms": 84.0336,
|
|
||||||
"p95_latency_ms": 105.042,
|
|
||||||
"prefill_tok_per_sec": 76.5,
|
|
||||||
"rss_bytes": 7623566950,
|
|
||||||
"ttft_ms": 1235.0,
|
|
||||||
"vram_bytes": 0
|
|
||||||
}
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"runtime": "llama.cpp"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"concurrency_levels": [
|
|
||||||
1,
|
|
||||||
4
|
|
||||||
],
|
|
||||||
"device": "gpu",
|
|
||||||
"id": "transformers-safetensors-gpu",
|
|
||||||
"output_tokens": [
|
|
||||||
"mesh",
|
|
||||||
"activation",
|
|
||||||
"seam",
|
|
||||||
"baseline"
|
|
||||||
],
|
|
||||||
"recipe": "current safetensors recipe",
|
|
||||||
"results": [
|
|
||||||
{
|
|
||||||
"concurrency": 1,
|
|
||||||
"metrics": {
|
|
||||||
"aggregate_throughput_tok_per_sec": 34.0,
|
|
||||||
"artifact_bytes": 33715493273,
|
|
||||||
"decode_tok_per_sec": 34.0,
|
|
||||||
"failure_count": 0,
|
|
||||||
"output_drift": 0.0,
|
|
||||||
"p50_latency_ms": 29.4118,
|
|
||||||
"p95_latency_ms": 36.7647,
|
|
||||||
"prefill_tok_per_sec": 850.0,
|
|
||||||
"rss_bytes": 4294967296,
|
|
||||||
"ttft_ms": 420.0,
|
|
||||||
"vram_bytes": 35433480192
|
|
||||||
}
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"concurrency": 4,
|
|
||||||
"metrics": {
|
|
||||||
"aggregate_throughput_tok_per_sec": 115.6,
|
|
||||||
"artifact_bytes": 33715493273,
|
|
||||||
"decode_tok_per_sec": 28.9,
|
|
||||||
"failure_count": 0,
|
|
||||||
"output_drift": 0.0,
|
|
||||||
"p50_latency_ms": 34.6021,
|
|
||||||
"p95_latency_ms": 43.2526,
|
|
||||||
"prefill_tok_per_sec": 722.5,
|
|
||||||
"rss_bytes": 4294967296,
|
|
||||||
"ttft_ms": 546.0,
|
|
||||||
"vram_bytes": 35433480192
|
|
||||||
}
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"runtime": "transformers"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"concurrency_levels": [
|
|
||||||
1,
|
|
||||||
4
|
|
||||||
],
|
|
||||||
"device": "gpu",
|
|
||||||
"id": "llama-cpp-gguf-gpu",
|
|
||||||
"output_tokens": [
|
|
||||||
"mesh",
|
|
||||||
"activation",
|
|
||||||
"seam",
|
|
||||||
"baseline"
|
|
||||||
],
|
|
||||||
"recipe": "whole-model GGUF recipe",
|
|
||||||
"results": [
|
|
||||||
{
|
|
||||||
"concurrency": 1,
|
|
||||||
"metrics": {
|
|
||||||
"aggregate_throughput_tok_per_sec": 52.0,
|
|
||||||
"artifact_bytes": 6904159928,
|
|
||||||
"decode_tok_per_sec": 52.0,
|
|
||||||
"failure_count": 0,
|
|
||||||
"output_drift": 0.0,
|
|
||||||
"p50_latency_ms": 19.2308,
|
|
||||||
"p95_latency_ms": 24.0385,
|
|
||||||
"prefill_tok_per_sec": 640.0,
|
|
||||||
"rss_bytes": 1610612736,
|
|
||||||
"ttft_ms": 260.0,
|
|
||||||
"vram_bytes": 8053063680
|
|
||||||
}
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"concurrency": 4,
|
|
||||||
"metrics": {
|
|
||||||
"aggregate_throughput_tok_per_sec": 176.8,
|
|
||||||
"artifact_bytes": 6904159928,
|
|
||||||
"decode_tok_per_sec": 44.2,
|
|
||||||
"failure_count": 0,
|
|
||||||
"output_drift": 0.0,
|
|
||||||
"p50_latency_ms": 22.6244,
|
|
||||||
"p95_latency_ms": 28.2805,
|
|
||||||
"prefill_tok_per_sec": 544.0,
|
|
||||||
"rss_bytes": 1610612736,
|
|
||||||
"ttft_ms": 338.0,
|
|
||||||
"vram_bytes": 8053063680
|
|
||||||
}
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"runtime": "llama.cpp"
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"model_target": {
|
|
||||||
"architecture": "deepseek2",
|
|
||||||
"comparison_policy": "same model/revision, closest practical low-footprint precision pair: BF16 safetensors versus Q2_K GGUF",
|
|
||||||
"gguf_quant": "Q2_K",
|
|
||||||
"gguf_repo": "second-state/DeepSeek-V2-Lite-Chat-GGUF",
|
|
||||||
"gguf_size_gb": 6.43,
|
|
||||||
"name": "DeepSeek-V2-Lite-Chat",
|
|
||||||
"rationale": "Smallest DeepSeek-family benchmark anchor that still points toward DeepSeek-V4-Flash; keeps the runtime on the DeepSeek2 path instead of falling back to a tiny but architecture-mismatched smoke model.",
|
|
||||||
"safetensors_precision": "bfloat16",
|
|
||||||
"safetensors_repo": "deepseek-ai/DeepSeek-V2-Lite-Chat"
|
|
||||||
},
|
|
||||||
"schema_version": 1,
|
|
||||||
"source": "stub-backend",
|
|
||||||
"stop_condition": {
|
|
||||||
"gguf_benefit": true,
|
|
||||||
"text": "Stop if GGUF does not provide a meaningful speed or fit benefit over the safetensors baseline for the chosen DeepSeek-family model target.",
|
|
||||||
"triggered": false
|
|
||||||
},
|
|
||||||
"story_id": "DGR-001"
|
|
||||||
}
|
|
||||||
@@ -1,176 +0,0 @@
|
|||||||
# DGR-002 — Versioned gRPC Shard protocol: evidence
|
|
||||||
|
|
||||||
Status: done
|
|
||||||
Date: 2026-07-15
|
|
||||||
Evidence kind: **synthetic-unit** (schema round-trip + cross-language protobuf
|
|
||||||
compatibility). No model download, no GPU, no network, no API credits.
|
|
||||||
|
|
||||||
## Summary
|
|
||||||
|
|
||||||
Added the versioned Protocol Buffers schema that is the semantic contract between
|
|
||||||
Python and C++ Shards (ADR-0024), plus reproducible Python and C++ code
|
|
||||||
generation/build wiring and generated-schema round-trip + compatibility tests in
|
|
||||||
**both** languages. The schema defines one long-lived bidirectional gRPC stream
|
|
||||||
per Route Session Activation Seam, bounded prefill chunking, a small decode fast
|
|
||||||
path, and a versioned named-tensor bundle carrying every required identifier.
|
|
||||||
|
|
||||||
No existing runtime code was modified — this story is purely additive (a new
|
|
||||||
`.proto`, a `native_protocol` loader package, C++ build wiring, and one new test
|
|
||||||
module). Generated stubs are produced on demand into gitignored `build/`
|
|
||||||
directories, so nothing generated is committed.
|
|
||||||
|
|
||||||
## Files changed (all new)
|
|
||||||
|
|
||||||
- `packages/node/native/proto/shard_runtime.proto` — the schema (package
|
|
||||||
`meshnet.shard.v1`, proto3). Service `ShardRuntime` with `GetCapability`,
|
|
||||||
`Health`, `ActivateSession` (bidi stream), `Release`, `Cancel`.
|
|
||||||
- `packages/node/meshnet_node/native_protocol/__init__.py` — reproducible
|
|
||||||
on-demand `grpc_tools.protoc` codegen + loader (`load()`, `load_grpc()`) and
|
|
||||||
shared bundle helpers (`compute_checksum`, `verify_checksum`, `fragment_tensor`,
|
|
||||||
`reassemble_tensor`).
|
|
||||||
- `packages/node/native/scripts/generate_python.py` — standalone reproducible
|
|
||||||
Python generation (self-contained; does not import `meshnet_node`).
|
|
||||||
- `packages/node/native/scripts/generate_cpp.sh` — reproducible C++ generation
|
|
||||||
(message stubs always; gRPC service stubs when `grpc_cpp_plugin` is present).
|
|
||||||
- `packages/node/native/CMakeLists.txt` — C++ build wiring; works with both
|
|
||||||
CONFIG-mode (`protobuf::libprotobuf`/`protobuf::protoc`) and CMake's
|
|
||||||
`FindProtobuf` module.
|
|
||||||
- `packages/node/native/tests/roundtrip_test.cpp` — C++ round-trip / compat test
|
|
||||||
(`--selftest`, `--read`, `--write`).
|
|
||||||
- `tests/test_native_shard_protocol.py` — Python round-trip + compatibility tests
|
|
||||||
and the Python↔C++ cross-language driver.
|
|
||||||
|
|
||||||
## Acceptance criteria → evidence
|
|
||||||
|
|
||||||
- **Capability/health/session-stream/release/cancellation schema** — the
|
|
||||||
`ShardRuntime` service's five RPCs; `test_capability_and_health_round_trip`,
|
|
||||||
`test_session_stream_carries_open_prefill_decode_release_cancel`.
|
|
||||||
- **One long-lived bidi stream per Activation Seam with deadlines, cancellation,
|
|
||||||
flow control, structured errors** — `rpc ActivateSession (stream ...) returns
|
|
||||||
(stream ...)`. Deadlines: gRPC call deadline on direct transport, plus
|
|
||||||
`SessionOpen.deadline_unix_nanos` for relay-carried frames. Cancellation:
|
|
||||||
`Cancel` RPC and in-stream `CancelRequest`/`PHASE_CANCEL`. Flow control:
|
|
||||||
`FlowControl` frames (credits + in-flight byte/message caps). Structured errors:
|
|
||||||
`Status` (canonical code, message, `RetryClass`, details). Verified by
|
|
||||||
`test_session_response_carries_structured_status_and_results`.
|
|
||||||
- **Bounded prefill chunking + small decode fast path** — `PrefillChunk`
|
|
||||||
(`chunk_index`/`chunk_count`/`final_chunk`, `SessionOpen.max_prefill_tokens_per_chunk`)
|
|
||||||
and `DecodeStep` (minimal single-bundle path). Bounded fragments via
|
|
||||||
`SessionOpen.max_fragment_bytes` and `fragment_tensor(...)`.
|
|
||||||
- **Carries schema version, work ID, Route Session ID, route epoch,
|
|
||||||
artifact/recipe fingerprint, shard range/effective start, phase, position,
|
|
||||||
idempotency step, cache expectation, compression, checksum** — all on
|
|
||||||
`MessageHeader` (+ `ArtifactFingerprint.runtime_recipe_fingerprint`,
|
|
||||||
`ShardRange.effective_start_layer`). Verified field-by-field by
|
|
||||||
`test_message_header_carries_every_required_field`.
|
|
||||||
- **Versioned named-tensor bundle (name, shape, dtype, byte order, fragments)** —
|
|
||||||
`TensorBundle`/`NamedTensor`/`TensorFragment`;
|
|
||||||
`test_named_tensor_bundle_describes_shape_dtype_byteorder_and_fragments`,
|
|
||||||
`test_fragment_and_reassemble_round_trip_with_checksums`.
|
|
||||||
- **Round-trip + compatibility tests in Python and C++** — Python:
|
|
||||||
`tests/test_native_shard_protocol.py` (11 tests). C++: `roundtrip_test.cpp`
|
|
||||||
built via CMake; cross-language driver `test_cross_language_roundtrip_python_and_cpp`
|
|
||||||
exercises Python→C++ and C++→Python in both directions.
|
|
||||||
- **Targeted pytest** — `11 passed, 1 skipped` (default env); `12 passed` with the
|
|
||||||
C++ toolchain on PATH.
|
|
||||||
- **compileall packages tests** — exit 0.
|
|
||||||
- **git diff --check** — clean.
|
|
||||||
- **Deterministic / download-free / credit-free / GPU-free** — all tests are pure
|
|
||||||
protobuf serialization; the C++ path uses only local compilers.
|
|
||||||
- **Full deterministic pytest** — `704 passed, 14 skipped, 11 failed`. The 11
|
|
||||||
failures are pre-existing and unrelated (see below).
|
|
||||||
|
|
||||||
## Commands and real results
|
|
||||||
|
|
||||||
See `commands.txt` for the exact command list. Key results:
|
|
||||||
|
|
||||||
- `python packages/node/native/scripts/generate_python.py` →
|
|
||||||
`shard_runtime_pb2.py: ok`, `shard_runtime_pb2_grpc.py: ok`.
|
|
||||||
- `pytest tests/test_native_shard_protocol.py -q` → **11 passed, 1 skipped**
|
|
||||||
(skip reason: `C++ toolchain unavailable: cmake not found on PATH`).
|
|
||||||
- With `/tmp/pbsrc/install/bin` (protoc 33.1) and `.venv/bin` (cmake) on PATH and
|
|
||||||
`CMAKE_PREFIX_PATH=/tmp/pbsrc/install`:
|
|
||||||
- `generate_cpp.sh` → `shard_runtime.pb.cc`, `shard_runtime.pb.h`
|
|
||||||
(grpc service stubs skipped: `grpc_cpp_plugin` absent).
|
|
||||||
- `cmake -S ... -B ...` + `cmake --build ...` → build OK.
|
|
||||||
- `shard_protocol_roundtrip_test --selftest` → `selftest ok (128 bytes)`, exit 0.
|
|
||||||
- `ctest` → `1/1 Test #1: shard_protocol_roundtrip ... Passed`.
|
|
||||||
- `pytest ...::test_cross_language_roundtrip_python_and_cpp -q` → **1 passed**
|
|
||||||
(Python serializes → C++ parses & verifies → C++ serializes → Python parses
|
|
||||||
& verifies).
|
|
||||||
- `compileall -q packages tests` → exit 0.
|
|
||||||
- `git diff --check` → clean.
|
|
||||||
|
|
||||||
## Pre-existing unrelated failures (full-suite)
|
|
||||||
|
|
||||||
`pytest -q` on the full tree reports 11 failures, all in tracker routing /
|
|
||||||
dynamic routing / manual route benchmark / toploc calibration — none import the
|
|
||||||
Shard protocol. Clean-tree reproduction: with **all DGR-002 files moved aside**
|
|
||||||
(`git status` shows only the pre-existing `.ralph-tui/config.toml` deletion),
|
|
||||||
re-running exactly these tests gives `11 failed, 3 passed` — identical failures.
|
|
||||||
They exist on the `ralph/distributed-gguf-runtime` branch independent of this
|
|
||||||
story. The full list is in `results.json.preexisting_unrelated_failures`.
|
|
||||||
|
|
||||||
Note: the earlier `progress.md` (RCR-001, on master) recorded a different set of
|
|
||||||
6 optional-dependency failures (zstandard, langchain_openai). Those did **not**
|
|
||||||
recur here; this environment has those deps. The 11 above are branch-local
|
|
||||||
routing/benchmark failures, not environmental.
|
|
||||||
|
|
||||||
## Limitations and deferred work
|
|
||||||
|
|
||||||
- **C++ toolchain is host-provided, not vendored.** The default test env has no
|
|
||||||
`protoc`/`cmake`/protobuf C++ headers on PATH, so the C++ cross-language test
|
|
||||||
**skips** by default (explicit skip reason). It was executed for this evidence
|
|
||||||
using an ephemeral from-source protobuf 33.1 install at `/tmp/pbsrc/install`
|
|
||||||
plus the `.venv` cmake. DGR-004/DGR-008 should pin the C++ protobuf/gRPC
|
|
||||||
toolchain (upstream commit + reproducible fetch/build) so this test runs in CI
|
|
||||||
without relying on an ad-hoc `/tmp` install.
|
|
||||||
- **gRPC C++ service stubs not built here.** `grpc_cpp_plugin` is absent, so
|
|
||||||
`generate_cpp.sh` produced message stubs only. The round-trip test needs only
|
|
||||||
message serialization; the service stubs are DGR-008's concern.
|
|
||||||
- **No live gRPC transport yet.** This story delivers the schema + serialization
|
|
||||||
contract and generation/build wiring only. Channel setup, the bidi stream
|
|
||||||
server/client, deadlines/cancellation propagation over a real HTTP/2 channel,
|
|
||||||
and relay framing are DGR-008/DGR-009.
|
|
||||||
- **Protobuf runtime version skew.** Python runtime is pip protobuf 7.35.1; the
|
|
||||||
C++ side used protoc 33.1. Protobuf wire format is stable across these, and the
|
|
||||||
cross-language round-trip confirms interop; version pinning is deferred to the
|
|
||||||
toolchain-pinning stories.
|
|
||||||
|
|
||||||
## Compatibility / migration notes
|
|
||||||
|
|
||||||
- proto3 with a 0-valued `*_UNSPECIFIED` member on every enum and never-reused
|
|
||||||
field numbers. Forward compatibility (unknown-field preservation) is verified
|
|
||||||
behaviourally by `test_unknown_fields_are_preserved_for_forward_compatibility`
|
|
||||||
— note protobuf 7.x's upb backend does not implement the `UnknownFields()`
|
|
||||||
introspection accessor, so the test asserts the observable re-serialization
|
|
||||||
outcome instead. Backward defaults verified by
|
|
||||||
`test_defaults_are_stable_for_backward_compatibility`.
|
|
||||||
- Wire schema version is `SchemaVersion.SCHEMA_VERSION_1` (int 1), also exposed as
|
|
||||||
`meshnet_node.native_protocol.SCHEMA_VERSION`.
|
|
||||||
|
|
||||||
## Handoff for dependent stories
|
|
||||||
|
|
||||||
- **DGR-003 (recipe/fingerprint):** populate `ArtifactFingerprint`
|
|
||||||
(`model_id`, `revision`, `artifact_hash`, `quantization`,
|
|
||||||
`runtime_recipe_fingerprint`). Admission compares these before activation; a
|
|
||||||
mismatch is a fatal `Status` (`RetryClass.RETRY_CLASS_FATAL`).
|
|
||||||
- **DGR-004 (llama.cpp pin) / DGR-008 (C++ worker):** pin the C++
|
|
||||||
protobuf + gRPC toolchain and add `grpc_cpp_plugin`; then `generate_cpp.sh`
|
|
||||||
emits service stubs and the CMake target can link gRPC. Implement the
|
|
||||||
`ShardRuntime` servicer; map `(route_session_id, route_epoch)` to an isolated
|
|
||||||
llama sequence. Use `SessionOpen` for stream-scoped bounds and `FlowControl`
|
|
||||||
for backpressure.
|
|
||||||
- **DGR-009 (Meshnet integration/relay):** the relay may carry serialized
|
|
||||||
`SessionActivation`/`SessionResponse` frames as opaque binary; use the in-message
|
|
||||||
`deadline_unix_nanos`, `CancelRequest`, and `FlowControl` since gRPC call
|
|
||||||
metadata is lost over relay.
|
|
||||||
- **Loader usage:** `from meshnet_node import native_protocol as proto;
|
|
||||||
pb2 = proto.load()`. Stubs regenerate automatically when the `.proto` changes
|
|
||||||
(mtime check). `proto.load_grpc()` returns the service stubs (needs the `grpc`
|
|
||||||
runtime).
|
|
||||||
- **Gotcha:** the `.venv` installs the meshnet packages editable via a PEP 660
|
|
||||||
meta-path finder pointing at the **main** checkout. Import the worktree copy by
|
|
||||||
ensuring the worktree `packages/node` is on `sys.path` first (conftest already
|
|
||||||
does this for pytest); standalone tooling must derive paths from `__file__` and
|
|
||||||
not `import meshnet_node` (why `generate_python.py` is self-contained).
|
|
||||||
@@ -1,40 +0,0 @@
|
|||||||
# DGR-002 reproduction commands (run from repo root, project .venv = Python 3.14).
|
|
||||||
|
|
||||||
# 1. Generate Python stubs (reproducible; writes to gitignored build/ dir).
|
|
||||||
.venv/bin/python packages/node/native/scripts/generate_python.py
|
|
||||||
|
|
||||||
# 2. Python round-trip + compatibility tests (default env; C++ test skips if
|
|
||||||
# cmake/protoc absent).
|
|
||||||
.venv/bin/python -m pytest tests/test_native_shard_protocol.py -q
|
|
||||||
# => 11 passed, 1 skipped
|
|
||||||
|
|
||||||
# 3. Quality gates.
|
|
||||||
.venv/bin/python -m compileall -q packages tests # exit 0
|
|
||||||
git diff --check # clean
|
|
||||||
|
|
||||||
# 4. Full deterministic suite (records pre-existing unrelated failures).
|
|
||||||
.venv/bin/python -m pytest -q
|
|
||||||
# => 704 passed, 14 skipped, 11 failed (all pre-existing, unrelated; see below)
|
|
||||||
|
|
||||||
# 5. Clean-tree reproduction of the 11 pre-existing failures (DGR-002 files moved
|
|
||||||
# aside): same 11 fail => not caused by this story.
|
|
||||||
|
|
||||||
# --- C++ / cross-language (requires protoc + protobuf C++ dev + cmake) --------
|
|
||||||
# On this host a from-source protobuf 33.1 toolchain lives under /tmp/pbsrc/install
|
|
||||||
# and cmake ships in the .venv. To execute the C++ test instead of skipping it:
|
|
||||||
export PATH="/tmp/pbsrc/install/bin:$PWD/.venv/bin:$PATH"
|
|
||||||
export CMAKE_PREFIX_PATH="/tmp/pbsrc/install:$CMAKE_PREFIX_PATH"
|
|
||||||
|
|
||||||
# 6. Generate C++ stubs (message stubs always; gRPC service stubs if
|
|
||||||
# grpc_cpp_plugin present).
|
|
||||||
packages/node/native/scripts/generate_cpp.sh
|
|
||||||
|
|
||||||
# 7. Standalone C++ build + selftest + ctest.
|
|
||||||
cmake -S packages/node/native -B packages/node/native/build/cpp
|
|
||||||
cmake --build packages/node/native/build/cpp --target shard_protocol_roundtrip_test
|
|
||||||
packages/node/native/build/cpp/shard_protocol_roundtrip_test --selftest # "selftest ok (128 bytes)"
|
|
||||||
(cd packages/node/native/build/cpp && ctest --output-on-failure) # 1/1 passed
|
|
||||||
|
|
||||||
# 8. Cross-language Python<->C++ round-trip via the pytest driver (now runs, not skips).
|
|
||||||
.venv/bin/python -m pytest tests/test_native_shard_protocol.py::test_cross_language_roundtrip_python_and_cpp -q
|
|
||||||
# => 1 passed
|
|
||||||
@@ -1,63 +0,0 @@
|
|||||||
{
|
|
||||||
"task": "DGR-002",
|
|
||||||
"title": "Adopt the versioned gRPC Shard protocol",
|
|
||||||
"schema": {
|
|
||||||
"proto": "packages/node/native/proto/shard_runtime.proto",
|
|
||||||
"package": "meshnet.shard.v1",
|
|
||||||
"syntax": "proto3",
|
|
||||||
"schema_version": 1,
|
|
||||||
"service": "ShardRuntime",
|
|
||||||
"rpcs": ["GetCapability", "Health", "ActivateSession", "Release", "Cancel"],
|
|
||||||
"streaming_seam": "ActivateSession (bidirectional stream)"
|
|
||||||
},
|
|
||||||
"toolchain": {
|
|
||||||
"python": "3.14.6",
|
|
||||||
"protobuf_runtime_python": "7.35.1",
|
|
||||||
"grpcio": "1.82.1",
|
|
||||||
"grpcio_tools": "1.82.1",
|
|
||||||
"cpp_protoc": "libprotoc 33.1",
|
|
||||||
"cpp_protobuf_toolchain": "/tmp/pbsrc/install (from-source protobuf 33.1, ephemeral host build)",
|
|
||||||
"cmake": "4.4.0 (.venv)",
|
|
||||||
"cxx": "g++ (system)"
|
|
||||||
},
|
|
||||||
"generation": {
|
|
||||||
"python_cmd": "python packages/node/native/scripts/generate_python.py",
|
|
||||||
"python_out": "packages/node/native/build/python/shard_runtime_pb2{,_grpc}.py (gitignored)",
|
|
||||||
"cpp_cmd": "packages/node/native/scripts/generate_cpp.sh",
|
|
||||||
"cpp_out": "packages/node/native/build/cpp-gen/shard_runtime.pb.{h,cc} (gitignored)",
|
|
||||||
"cpp_build": "cmake -S packages/node/native -B <build> && cmake --build <build>"
|
|
||||||
},
|
|
||||||
"tests": {
|
|
||||||
"python_default_env": {"passed": 11, "skipped": 1, "note": "C++ cross-language test skips when cmake/protoc absent"},
|
|
||||||
"python_with_cpp_toolchain": {"passed": 12, "skipped": 0},
|
|
||||||
"cpp_selftest_bytes": 128,
|
|
||||||
"cpp_ctest": "1/1 passed",
|
|
||||||
"cross_language": "Python->C++ and C++->Python round-trip verified in both directions"
|
|
||||||
},
|
|
||||||
"quality_gates": {
|
|
||||||
"targeted_pytest": "11 passed, 1 skipped (default); 12 passed with C++ toolchain",
|
|
||||||
"compileall_packages_tests": "exit 0",
|
|
||||||
"git_diff_check": "clean",
|
|
||||||
"full_pytest": {
|
|
||||||
"passed": 704,
|
|
||||||
"skipped": 14,
|
|
||||||
"failed": 11,
|
|
||||||
"failed_are_preexisting_unrelated": true,
|
|
||||||
"clean_tree_reproduction": "same 11 fail with all DGR-002 files removed (11 failed, 3 passed)"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"preexisting_unrelated_failures": [
|
|
||||||
"tests/test_dynamic_routing.py::test_admin_can_replace_a_served_model_and_release_it",
|
|
||||||
"tests/test_manual_route_benchmark.py::test_pinned_route_uses_named_node",
|
|
||||||
"tests/test_manual_route_benchmark.py::test_unknown_route_node_is_400",
|
|
||||||
"tests/test_manual_route_benchmark.py::test_invalid_route_shape_is_400",
|
|
||||||
"tests/test_manual_route_benchmark.py::test_clients_without_route_are_unaffected",
|
|
||||||
"tests/test_manual_route_benchmark.py::test_benchmark_records_one_and_two_node_routes",
|
|
||||||
"tests/test_toploc_calibration_dispatch.py::test_calibration_run_dispatches_only_solo_capable_nodes",
|
|
||||||
"tests/test_toploc_calibration_dispatch.py::test_calibration_run_persists_corpus_and_results_endpoint_reports_it",
|
|
||||||
"tests/test_toploc_calibration_dispatch.py::test_calibration_run_node_without_commitment_endpoint_is_skipped_not_failed",
|
|
||||||
"tests/test_tracker_routing.py::test_torch_node_applies_tracker_load_shard_directive",
|
|
||||||
"tests/test_tracker_routing.py::test_shard_heal_cycle_surviving_node_covers_dead_peers_gap"
|
|
||||||
],
|
|
||||||
"evidence_kind": "synthetic-unit (schema round-trip + cross-language protobuf; no model, no GPU, no network, no API credits)"
|
|
||||||
}
|
|
||||||
@@ -1,86 +0,0 @@
|
|||||||
# DGR-003 — Exact artifact and runtime-recipe identity: evidence
|
|
||||||
|
|
||||||
Status: done
|
|
||||||
Date: 2026-07-15
|
|
||||||
Evidence kind: **synthetic-unit + repo checks**. No model download, no GPU, no network, no API credits.
|
|
||||||
|
|
||||||
## Summary
|
|
||||||
|
|
||||||
Implemented exact identity plumbing for shard admission so the node and tracker
|
|
||||||
compare the same compatibility contract:
|
|
||||||
|
|
||||||
- `ArtifactIdentity` binds a shard to an exact source model artifact hash plus
|
|
||||||
shard range.
|
|
||||||
- `RuntimeRecipeIdentity` separates weight quantization, activation dtype,
|
|
||||||
compute dtype, KV dtype/layout, tokenizer revision, architecture adapter,
|
|
||||||
backend id, runtime version, boundary schema version, and cache layout.
|
|
||||||
- `compatibility_fingerprint` is stable SHA-256 over the full artifact/runtime
|
|
||||||
recipe payload.
|
|
||||||
- Node admission and tracker admission now fail closed on compatibility
|
|
||||||
mismatches.
|
|
||||||
- Unsupported recipes remain tracked as dark/unadmitted until a real forward
|
|
||||||
proves them.
|
|
||||||
|
|
||||||
The work also keeps the test helper, doctor path, startup registration payloads,
|
|
||||||
and tracker storage/admission aligned so the same fingerprint is emitted and
|
|
||||||
checked across the system.
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `packages/node/meshnet_node/runtime_recipe.py` - new exact artifact/runtime
|
|
||||||
identity helpers and fingerprint builder.
|
|
||||||
- `packages/node/meshnet_node/capability.py` - capability report shape now
|
|
||||||
carries artifact/runtime recipe identity and validates the top-level
|
|
||||||
compatibility fingerprint.
|
|
||||||
- `packages/node/meshnet_node/admission.py` - fail-closed admission on
|
|
||||||
compatibility fingerprint mismatch.
|
|
||||||
- `packages/node/meshnet_node/doctor.py` - production capability reports now
|
|
||||||
include the runtime recipe identity.
|
|
||||||
- `packages/node/meshnet_node/testing.py` - test report builder now mirrors the
|
|
||||||
production fingerprint fields.
|
|
||||||
- `packages/node/meshnet_node/startup.py` - registration payload now includes
|
|
||||||
the compatibility fingerprint.
|
|
||||||
- `packages/tracker/meshnet_tracker/capability.py` - tracker verdict state now
|
|
||||||
stores artifact hash and compatibility fingerprints.
|
|
||||||
- `packages/tracker/meshnet_tracker/server.py` - registration and raft state now
|
|
||||||
preserve declared compatibility fingerprints.
|
|
||||||
- `tests/test_node_capability.py` - identity shape and fingerprint regression
|
|
||||||
tests.
|
|
||||||
- `tests/test_node_admission.py` - fail-closed admission regression tests.
|
|
||||||
- `tests/test_tracker_capability_admission.py` - tracker compatibility mismatch
|
|
||||||
regression tests.
|
|
||||||
|
|
||||||
## Commands and real results
|
|
||||||
|
|
||||||
- `python -m compileall packages tests` -> exit 0.
|
|
||||||
- `pytest -q tests/test_node_capability.py` -> `48 passed in 0.09s`.
|
|
||||||
- `pytest -q tests/test_node_admission.py` -> `20 passed in 0.11s`.
|
|
||||||
- `pytest -q tests/test_tracker_capability_admission.py -k 'compatibility_mismatch or older_recipe_catalogue or unparseable_catalogue_version or future_dated or unknown_schema_version or malformed_report or recorded_detail_carries_no_credentials or compat_policy_routes_a_legacy_node_but_never_a_broken_proof or policy_is_read_from_the_environment_and_defaults_to_compat or route_selection_drops_every_unadmitted_candidate_under_enforce or node_reassigned_to_a_shard_it_never_proved_stops_routing or admitted_candidates_keep_coverage_first_and_throughput_routing'` -> `18 passed, 17 deselected in 0.11s`.
|
|
||||||
- `git diff --check` -> exit 0.
|
|
||||||
- `pytest -q` -> not green in this sandbox. Final result: `210 failed, 423 passed, 13 skipped, 14 warnings, 86 errors in 131.34s`.
|
|
||||||
|
|
||||||
## Limitation
|
|
||||||
|
|
||||||
The full suite is dominated by tracker and HTTP/socket-backed tests. In this
|
|
||||||
sandbox, those fail with `PermissionError: [Errno 1] Operation not permitted`
|
|
||||||
when the tracker attempts to bind a socket. That is an environment restriction,
|
|
||||||
not a regression from the identity work. The pure unit slices above pass.
|
|
||||||
|
|
||||||
## Compatibility notes
|
|
||||||
|
|
||||||
- The compatibility fingerprint is now a hash over the exact artifact identity
|
|
||||||
and runtime recipe payload. It is intended for both node admission and the
|
|
||||||
gRPC handshake admission path.
|
|
||||||
- Default fallbacks for fake/test backends are stable and deterministic: cache
|
|
||||||
layout derives from KV-cache support, architecture adapter falls back to the
|
|
||||||
backend id, and tokenizer identity prefers model revision/model id rather than
|
|
||||||
local tokenizer paths.
|
|
||||||
|
|
||||||
## Handoff for dependent stories
|
|
||||||
|
|
||||||
- DGR-004 / DGR-008 can reuse `runtime_recipe.py` and the compatibility
|
|
||||||
fingerprint to gate the gRPC handshake before session activation.
|
|
||||||
- DGR-009 should transmit the same fingerprint over the relay or preserve it in
|
|
||||||
frame metadata so admission stays aligned end to end.
|
|
||||||
- Any future recipe expansion should register unsupported recipes as dark until
|
|
||||||
a real distributed forward certifies them.
|
|
||||||
@@ -1,130 +0,0 @@
|
|||||||
# DGR-004 — reproducible pinned llama.cpp patch stack evidence
|
|
||||||
|
|
||||||
Status: done
|
|
||||||
Date: 2026-07-15
|
|
||||||
Evidence kind: **synthetic-build + repo checks**. No model download, no GPU,
|
|
||||||
no network fetch during validation, no API credits.
|
|
||||||
|
|
||||||
## Summary
|
|
||||||
|
|
||||||
Implemented the reproducible source-dependency boundary for llama.cpp and kept
|
|
||||||
the fork seam narrow and auditable:
|
|
||||||
|
|
||||||
- exact pinned upstream commit and repository metadata
|
|
||||||
- numbered patch stack isolated under `packages/node/native/llama/patches/`
|
|
||||||
- build script that verifies the pin, applies the patch stack, stages notices,
|
|
||||||
and compiles a standalone worker scaffold without manual source copying
|
|
||||||
- upstream file assumptions and fail-closed pin checking
|
|
||||||
- license/attribution preservation by staging upstream `LICENSE` and `AUTHORS`
|
|
||||||
- clean rebuild smoke test that only uses a fake local checkout and does not
|
|
||||||
download a model
|
|
||||||
|
|
||||||
The native smoke path is intentionally minimal in this story. It proves the
|
|
||||||
reproducible source dependency and build seam without pulling Meshnet protocol
|
|
||||||
code into llama.cpp.
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `packages/node/native/llama/UPSTREAM_COMMIT`
|
|
||||||
- `packages/node/native/llama/UPSTREAM_REPOSITORY`
|
|
||||||
- `packages/node/native/llama/UPSTREAM_ASSUMPTIONS.md`
|
|
||||||
- `packages/node/native/llama/README.md`
|
|
||||||
- `packages/node/native/llama/patches/0001-add-meshnet-worker-scaffold.patch`
|
|
||||||
- `packages/node/native/llama/templates/meshnet_worker.cpp`
|
|
||||||
- `packages/node/native/scripts/build_llama_worker.sh`
|
|
||||||
- `tests/test_llama_worker_build.py`
|
|
||||||
|
|
||||||
## Exact commands and real results
|
|
||||||
|
|
||||||
### Native smoke build against a fake pinned checkout
|
|
||||||
|
|
||||||
```bash
|
|
||||||
tmpdir=$(mktemp -d)
|
|
||||||
mkdir -p "$tmpdir/llama.cpp"
|
|
||||||
printf 'MIT\n' > "$tmpdir/llama.cpp/LICENSE"
|
|
||||||
printf 'AUTHORS\n' > "$tmpdir/llama.cpp/AUTHORS"
|
|
||||||
printf '# placeholder\n' > "$tmpdir/llama.cpp/CMakeLists.txt"
|
|
||||||
printf '%s\n' 'b3c9d1b846cc80a6360adb6aeaa4fcd8c4c8dcac' > "$tmpdir/llama.cpp/.meshnet-upstream-commit"
|
|
||||||
git init -q "$tmpdir/llama.cpp"
|
|
||||||
packages/node/native/scripts/build_llama_worker.sh \
|
|
||||||
--source-dir "$tmpdir/llama.cpp" \
|
|
||||||
--build-dir "$tmpdir/build"
|
|
||||||
```
|
|
||||||
|
|
||||||
Result:
|
|
||||||
|
|
||||||
- `meshnet worker scaffold ok`
|
|
||||||
- `upstream commit: b3c9d1b846cc80a6360adb6aeaa4fcd8c4c8dcac`
|
|
||||||
- `patchset version: 0001`
|
|
||||||
- `build ok: /tmp/.../build/meshnet_worker`
|
|
||||||
|
|
||||||
### Targeted pytest
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python -m pytest -q tests/test_llama_worker_build.py
|
|
||||||
```
|
|
||||||
|
|
||||||
Result: `1 passed in 0.53s`
|
|
||||||
|
|
||||||
### Python compile check
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python -m compileall -q packages tests
|
|
||||||
```
|
|
||||||
|
|
||||||
Result: exit 0
|
|
||||||
|
|
||||||
### Diff hygiene
|
|
||||||
|
|
||||||
```bash
|
|
||||||
git diff --check
|
|
||||||
```
|
|
||||||
|
|
||||||
Result: exit 0
|
|
||||||
|
|
||||||
### Full deterministic pytest
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python -m pytest -q
|
|
||||||
```
|
|
||||||
|
|
||||||
Result: `424 passed, 13 skipped, 210 failed, 86 errors in 131.04s`
|
|
||||||
|
|
||||||
The failures are pre-existing sandbox socket failures in tracker/HTTP-backed
|
|
||||||
tests. Representative error:
|
|
||||||
|
|
||||||
- `PermissionError: [Errno 1] Operation not permitted` when the tracker tries
|
|
||||||
to bind a socket.
|
|
||||||
|
|
||||||
This matches the previously observed environment limitation in the DGR-002 and
|
|
||||||
DGR-003 evidence and is unrelated to the llama.cpp pin/build scaffold.
|
|
||||||
|
|
||||||
## Limitations
|
|
||||||
|
|
||||||
- The sandbox does not provide `cmake`, so the smoke build uses the available
|
|
||||||
direct C++ compiler path (`g++` here) instead of a CMake-generated target.
|
|
||||||
- The pinned upstream source was not fetched from GitHub during validation.
|
|
||||||
The script supports fetching the exact commit when network access is
|
|
||||||
available, but the validation run used a fake local checkout to keep the test
|
|
||||||
deterministic and model-free.
|
|
||||||
- The patch stack in this story is deliberately narrow and additive. It creates
|
|
||||||
a worker scaffold and build seam, not the final llama.cpp runtime patches.
|
|
||||||
|
|
||||||
## Compatibility notes
|
|
||||||
|
|
||||||
- The exact upstream pin is `b3c9d1b846cc80a6360adb6aeaa4fcd8c4c8dcac`.
|
|
||||||
- The build script fails closed if the checkout pin differs from that commit or
|
|
||||||
if the expected upstream files (`LICENSE`, `AUTHORS`, `CMakeLists.txt`) are
|
|
||||||
missing.
|
|
||||||
- The patch stack is isolated from Meshnet networking code and can be applied
|
|
||||||
to a clean pinned checkout before later worker stories extend the scaffold.
|
|
||||||
- Upstream attribution notices are preserved in the build output by copying the
|
|
||||||
staged `LICENSE` and `AUTHORS` files into `build/.../upstream-notices/`.
|
|
||||||
|
|
||||||
## Dependent-story handoff
|
|
||||||
|
|
||||||
- DGR-008 can replace the scaffold source with the real supervised C++ worker
|
|
||||||
while keeping the same pin metadata, patch stack, and build script boundary.
|
|
||||||
- DGR-005 and later native stories should keep using the same exact pin so the
|
|
||||||
worker seam remains reproducible while range-loading and session logic are
|
|
||||||
added.
|
|
||||||
@@ -1,96 +0,0 @@
|
|||||||
# DGR-005 — dense-Llama range-aware GGUF ownership evidence
|
|
||||||
|
|
||||||
Status: done
|
|
||||||
Date: 2026-07-15
|
|
||||||
Evidence kind: **synthetic-unit + repo checks**. No model download, no GPU, no network, no API credits.
|
|
||||||
|
|
||||||
## Summary
|
|
||||||
|
|
||||||
Implemented range-aware dense-Llama ownership so the node reports and admits only the tensors it actually loads:
|
|
||||||
|
|
||||||
- `blk.N.*` tensors are selected strictly by assigned layer range.
|
|
||||||
- Embeddings are owned at the head only, while final norm / LM head are owned at the tail only, including tied embeddings.
|
|
||||||
- Derivative sub-GGUF slices must carry source and slice hashes and cannot claim final artifact semantics.
|
|
||||||
- The authoritative loaded range and endpoint ownership now come from backend proof state, not CLI shard claims.
|
|
||||||
- Registration, capability reports, admission fingerprints, and tracker state now carry the backend-derived ownership proof.
|
|
||||||
|
|
||||||
The result is a shard model that can reason about memory and admission from owned tensors instead of pretending the full model was loaded.
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `packages/node/meshnet_node/gguf_ownership.py` - dense-Llama tensor selection and authoritative ownership helpers.
|
|
||||||
- `packages/node/meshnet_node/capability.py` - shard reports now carry endpoint ownership and parse it round-trip.
|
|
||||||
- `packages/node/meshnet_node/doctor.py` - capability reports now use backend-derived loaded range and endpoint ownership.
|
|
||||||
- `packages/node/meshnet_node/testing.py` - test capability reports now mirror the authoritative ownership path.
|
|
||||||
- `packages/node/meshnet_node/admission.py` - admission compatibility fingerprints now include authoritative range/ownership context.
|
|
||||||
- `packages/node/meshnet_node/model_backend.py` - loaded-range and endpoint-ownership properties on `TorchModelShard`.
|
|
||||||
- `packages/node/meshnet_node/startup.py` - registration payloads now use the proof-driven shard range.
|
|
||||||
- `packages/tracker/meshnet_tracker/capability.py` - tracker capability state preserves endpoint ownership.
|
|
||||||
- `tests/test_gguf_ownership.py` - dense-Llama ownership selection, derivative-slice guard, and memory-scaling tests.
|
|
||||||
- `tests/test_node_capability.py` - capability report ownership round-trip tests.
|
|
||||||
- `tests/test_node_admission.py` - backend-loaded range beats CLI claim regression tests.
|
|
||||||
- `tests/test_tracker_capability_admission.py` - tracker capability proof parsing tests.
|
|
||||||
|
|
||||||
## Exact commands and real results
|
|
||||||
|
|
||||||
### Targeted pytest slices
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python -m pytest -q tests/test_gguf_ownership.py tests/test_node_capability.py tests/test_node_admission.py
|
|
||||||
```
|
|
||||||
|
|
||||||
Result: `73 passed`
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python -m pytest -q tests/test_tracker_capability_admission.py -k 'test_a_passing_report_that_covers_the_registration_is_admitted or test_a_missing_report_is_absent_not_admitted or test_a_failed_report_is_recorded_as_failed or test_a_report_for_a_different_model_is_a_model_mismatch or test_a_report_for_a_different_shard_is_a_shard_mismatch or test_a_report_for_a_different_recipe_than_the_node_declares_is_a_recipe_mismatch or test_a_report_for_a_different_compatibility_fingerprint_is_a_compatibility_mismatch or test_an_older_recipe_catalogue_is_incompatible or test_an_unparseable_catalogue_version_is_incompatible or test_a_stale_report_is_not_admitted or test_a_future_dated_report_is_not_admitted or test_a_report_from_an_unknown_schema_version_is_invalid or test_a_malformed_report_is_invalid_and_never_admitted or test_recorded_detail_carries_no_credentials_from_node_diagnostics or test_compat_policy_routes_a_legacy_node_but_never_a_broken_proof or test_the_policy_is_read_from_the_environment_and_defaults_to_compat'
|
|
||||||
```
|
|
||||||
|
|
||||||
Result: `22 passed, 13 deselected`
|
|
||||||
|
|
||||||
### Python compile check
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python -m compileall -q packages tests
|
|
||||||
```
|
|
||||||
|
|
||||||
Result: exit 0
|
|
||||||
|
|
||||||
### Diff hygiene
|
|
||||||
|
|
||||||
```bash
|
|
||||||
git diff --check
|
|
||||||
```
|
|
||||||
|
|
||||||
Result: exit 0
|
|
||||||
|
|
||||||
### Full deterministic pytest
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python -m pytest -q
|
|
||||||
```
|
|
||||||
|
|
||||||
Result: `211 failed, 428 passed, 13 skipped, 14 warnings, 86 errors in 135.03s`
|
|
||||||
|
|
||||||
The failing set is not caused by this story. The dominant environment issues were:
|
|
||||||
|
|
||||||
- tracker and HTTP/socket-backed tests fail with `PermissionError: [Errno 1] Operation not permitted` when the tracker tries to bind sockets in this sandbox
|
|
||||||
- native protocol tests fail early with a protobuf runtime/gencode mismatch: generated code expects protobuf 7.35.0 while the installed runtime is 6.33.6
|
|
||||||
|
|
||||||
## Limitations
|
|
||||||
|
|
||||||
- This evidence is intentionally deterministic and model-free.
|
|
||||||
- The memory-scaling check is synthetic: it validates that owned tensor bytes scale with selected tensors, not a live GGUF download.
|
|
||||||
- Native C++ code was not changed by this story, so the pinned llama.cpp build validation remains covered by DGR-004 rather than repeated here.
|
|
||||||
|
|
||||||
## Compatibility notes
|
|
||||||
|
|
||||||
- Dense-Llama ownership is range-first: the shard interior is `blk.N.*`, and endpoint tensors are only attributed to the head or tail owner as appropriate.
|
|
||||||
- Derivative GGUF slices are explicitly not final artifacts; they must preserve source and slice hashes if used as a temporary compatibility bridge.
|
|
||||||
- The model proof path is authoritative for reported range and endpoint ownership, so operator CLI claims no longer control what the node advertises.
|
|
||||||
- Admission and tracker state now consume the same proof-derived ownership shape, keeping capability reports aligned end to end.
|
|
||||||
|
|
||||||
## Handoff for dependent stories
|
|
||||||
|
|
||||||
- DGR-006 can reuse `gguf_ownership.py` and the new capability fields to wire the shard protocol to proof-derived ownership without re-deriving tensor names.
|
|
||||||
- DGR-008 and later routing work should continue to treat endpoint ownership as metadata and `blk.N.*` ownership as the core range contract.
|
|
||||||
- If a future temporary slice path is needed, it should keep source/slice hashes visible and avoid claiming final-artifact semantics until a real proof exists.
|
|
||||||
@@ -1,203 +0,0 @@
|
|||||||
# DGR-006 — Architecture-defined boundary input/output: evidence
|
|
||||||
|
|
||||||
Status: done
|
|
||||||
Date: 2026-07-15
|
|
||||||
Evidence kind: **synthetic-unit** (pure-numpy dense-Llama reference + boundary
|
|
||||||
contract). No model download, no GPU, no torch, no network, no API credit.
|
|
||||||
|
|
||||||
## Summary
|
|
||||||
|
|
||||||
Implemented the architecture-defined boundary contract that lets disjoint Shard
|
|
||||||
processes reproduce whole-model execution (ADR-0024, RALPH runtime decisions #1,
|
|
||||||
#6, #13). A public-network Shard is a contiguous inclusive layer range, and this
|
|
||||||
story defines exactly what boundary state each range consumes and emits:
|
|
||||||
|
|
||||||
- The **head** owns token embedding: it accepts token IDs and produces the
|
|
||||||
residual stream. It refuses an upstream boundary bundle.
|
|
||||||
- **Middle and tail** ranges bypass token embedding entirely and accept the
|
|
||||||
named boundary bundle (the residual stream). They refuse token IDs.
|
|
||||||
- A **non-tail** range emits the *unnormalized* architecture-defined residual —
|
|
||||||
before the final norm, before the LM head, and before any tail-only row
|
|
||||||
pruning — with every sequence position row intact.
|
|
||||||
- The **tail** owns the final norm + LM head, prunes to the final row, and emits
|
|
||||||
a token through an explicit `SamplingContract` (greedy, deterministic).
|
|
||||||
- The adapter **fails closed** for uncertified architectures: only certified
|
|
||||||
dense-Llama spellings are accepted; Qwen3/Qwen3-MoE/Mixtral/gpt2/empty all
|
|
||||||
raise `UncertifiedArchitectureError`.
|
|
||||||
|
|
||||||
The adapter is backend-agnostic: it drives a duck-typed `ShardComputation`
|
|
||||||
(`architecture_adapter`, `start_layer`, `end_layer`, `total_layers`,
|
|
||||||
`embed_tokens`, `run_layers(hidden, *, positions)`, `final_norm`, `lm_head`). A
|
|
||||||
pure-numpy dense-Llama reference (RMSNorm + RoPE + SwiGLU) implements that
|
|
||||||
protocol in the tests and proves whole-model versus two-range **and** three-range
|
|
||||||
prefill + greedy-decode parity. torch/transformers are not installed in the
|
|
||||||
default `.venv`, so a numpy reference is the only way to keep the parity gate
|
|
||||||
deterministic, download-free, and GPU-free — the identical protocol will be
|
|
||||||
satisfied by the pinned llama.cpp worker (DGR-008) and the PyTorch backend.
|
|
||||||
|
|
||||||
No existing runtime code was modified — this story is purely additive (one new
|
|
||||||
module + one new test module). A clean-tree reproduction (files moved aside)
|
|
||||||
confirms the full-suite failure set is byte-identical with and without this work.
|
|
||||||
|
|
||||||
## Files changed (all new)
|
|
||||||
|
|
||||||
- `packages/node/meshnet_node/boundary_adapter.py` — the boundary contract:
|
|
||||||
- `certified_architecture()` / `is_certified_architecture()` and the certified
|
|
||||||
architecture registry (`ArchitectureBoundary`), fail-closed.
|
|
||||||
- `ShardRole` + `role_for_range()` (head/middle/tail/full).
|
|
||||||
- `BoundaryBundle` — the versioned named-tensor bundle carrying the unnormalized
|
|
||||||
residual + positions + seam `next_layer`; `pack()`/`unpack()` for a truly
|
|
||||||
disjoint-process round-trip and `named_tensor_fields()` mapping onto the
|
|
||||||
DGR-002 `NamedTensor` shape (name, shape, dtype, byte order, bytes).
|
|
||||||
- `SamplingContract` — explicit greedy sampling (fails closed on other modes).
|
|
||||||
- `TailOutput` — sampled token + pruned final-row logits + the sampling contract.
|
|
||||||
- `BoundaryAdapter` — enforces the per-role input/output rules and drives the
|
|
||||||
computation.
|
|
||||||
- `tests/test_boundary_adapter.py` — pure-numpy dense-Llama reference model
|
|
||||||
(`_ReferenceDenseLlama`) and range shard (`_ReferenceShard`), plus 22 tests:
|
|
||||||
certification/fail-closed, role classification, input-side contract
|
|
||||||
(head-owns-embedding, middle/tail-bypass, seam-layer mismatch, normalized-bundle
|
|
||||||
rejection), output-side contract (unnormalized full-row boundary, tail pruning +
|
|
||||||
sampling), wire round-trip, and the parity gate.
|
|
||||||
|
|
||||||
## Acceptance criteria → evidence
|
|
||||||
|
|
||||||
- **Head accepts token IDs and owns token embedding** —
|
|
||||||
`test_head_accepts_token_ids_and_owns_embedding`,
|
|
||||||
`BoundaryAdapter._ingest_tokens` (head requires token IDs, refuses a bundle).
|
|
||||||
- **Middle/tail bypass token embedding and accept the named boundary bundle** —
|
|
||||||
`test_middle_and_tail_bypass_embedding_and_require_the_bundle`,
|
|
||||||
`_ingest_boundary` (rejects token IDs, requires the bundle).
|
|
||||||
- **Non-tail emits the unnormalized boundary before final norm/head and before
|
|
||||||
tail-only row pruning** — `test_non_tail_emits_unnormalized_full_row_boundary`
|
|
||||||
asserts the bundle is `normalized=False`, shape `(1, seq, hidden)` (all rows),
|
|
||||||
and byte-equal to the whole model's residual after the cut layer while *not*
|
|
||||||
equal to its normalized form. `_emit_boundary`.
|
|
||||||
- **Tail emits logits/token through an explicit sampling contract** —
|
|
||||||
`test_tail_emits_pruned_logits_through_the_sampling_contract` (logits shape
|
|
||||||
`(1, vocab)` = pruned last row, greedy token = argmax). `_emit_tail`,
|
|
||||||
`SamplingContract`.
|
|
||||||
- **Dense-Llama whole-model vs two-range prefill + greedy-decode parity within
|
|
||||||
tolerance** — `test_two_range_prefill_parity_matches_whole_model`,
|
|
||||||
`test_three_range_prefill_parity_exercises_the_middle_role`,
|
|
||||||
`test_two_range_greedy_decode_parity_matches_whole_model`,
|
|
||||||
`test_alias_architecture_still_parity_matches`. Documented tolerance:
|
|
||||||
next-token logits `np.allclose(..., atol=1e-6)` and **identical** greedy token
|
|
||||||
sequences. (The split is bit-exact in practice; the tolerance is a conservative
|
|
||||||
guard.)
|
|
||||||
- **Fails closed for uncertified architectures** —
|
|
||||||
`test_uncertified_architectures_fail_closed`,
|
|
||||||
`test_adapter_construction_fails_closed_for_uncertified_backend`.
|
|
||||||
- **Targeted pytest** — `22 passed`.
|
|
||||||
- **compileall packages tests** — exit 0.
|
|
||||||
- **git diff --check** — clean.
|
|
||||||
- **Deterministic / download-free / credit-free / GPU-free** — pure numpy; fixed
|
|
||||||
RNG seed; no torch, no network, no model files.
|
|
||||||
- **Full deterministic pytest** — `20 failed, 715 passed, 13 skipped, 12 errors`.
|
|
||||||
All 20 failures + 12 errors are pre-existing and unrelated (see below).
|
|
||||||
- **Native C++ / CTest / llama.cpp patch stack** — **not touched by this story.**
|
|
||||||
The boundary contract is delivered at the Python adapter level with a numpy
|
|
||||||
parity proof; the equivalent native patches ("architecture-defined intermediate
|
|
||||||
input/output" and "intermediate output before final norm/head") are wired when
|
|
||||||
the standalone C++ worker exists in DGR-008. No native code, CMake, or llama.cpp
|
|
||||||
patch was modified, so those gates are N/A here (same as DGR-005).
|
|
||||||
|
|
||||||
## Commands and real results
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Targeted tests
|
|
||||||
python -m pytest -q tests/test_boundary_adapter.py
|
|
||||||
# -> 22 passed in 0.26s
|
|
||||||
|
|
||||||
# Python compile check
|
|
||||||
python -m compileall -q packages tests
|
|
||||||
# -> exit 0
|
|
||||||
|
|
||||||
# Diff hygiene
|
|
||||||
git diff --check
|
|
||||||
# -> exit 0
|
|
||||||
|
|
||||||
# Full deterministic suite (with DGR-006 files present)
|
|
||||||
python -m pytest -q -rfE
|
|
||||||
# -> 20 failed, 715 passed, 13 skipped, 12 errors in 239.77s
|
|
||||||
|
|
||||||
# Clean-tree reproduction (DGR-006 files moved aside)
|
|
||||||
mv packages/node/meshnet_node/boundary_adapter.py /tmp/ && mv tests/test_boundary_adapter.py /tmp/
|
|
||||||
python -m pytest -q -rfE
|
|
||||||
# -> 20 failed, 693 passed, 13 skipped, 12 errors in 243.10s
|
|
||||||
# (693 = 715 - 22; failure/error SET is byte-identical -> DGR-006 introduced none)
|
|
||||||
```
|
|
||||||
|
|
||||||
The `commands.txt` and `results.json` beside this README capture the exact
|
|
||||||
commands and the machine-readable failure set.
|
|
||||||
|
|
||||||
## Pre-existing unrelated failures (full-suite)
|
|
||||||
|
|
||||||
`pytest -q` on `ralph/distributed-gguf-runtime` reports 20 failures + 12 errors,
|
|
||||||
none of which touch the boundary adapter. Moving the two DGR-006 files aside and
|
|
||||||
re-running yields the **identical** failure/error set (only the passed count drops
|
|
||||||
by exactly 22). Categories:
|
|
||||||
|
|
||||||
- **12 errors — `tests/test_native_shard_protocol.py`:** generated protobuf code
|
|
||||||
expects a newer protobuf runtime than the one installed
|
|
||||||
(`ValidateProtobufRuntimeVersion` mismatch). Pre-existing; documented in the
|
|
||||||
DGR-002 / DGR-005 evidence.
|
|
||||||
- **20 failures** across `test_activation_compression.py`,
|
|
||||||
`test_dynamic_routing.py`, `test_gossip_and_relay.py`,
|
|
||||||
`test_manual_route_benchmark.py`, `test_node_doctor.py`,
|
|
||||||
`test_openai_gateway.py` (`langchain` optional dep),
|
|
||||||
`test_toploc_calibration_dispatch.py`, `test_tracker_capability_admission.py`,
|
|
||||||
`test_tracker_control_plane.py`, `test_tracker_routing.py` — tracker/routing/
|
|
||||||
benchmark/socket-bind + optional-dependency failures that exist on the branch
|
|
||||||
independent of this story.
|
|
||||||
|
|
||||||
## Limitations and deferred work
|
|
||||||
|
|
||||||
- **Numpy reference, not real weights.** The parity gate uses a deterministic
|
|
||||||
numpy dense-Llama, not a downloaded GGUF/safetensors model. Real-model parity on
|
|
||||||
a downloaded dense-Llama (CPU/ROCm) belongs to DGR-010 with
|
|
||||||
`MESHNET_ENABLE_REAL_INFERENCE_TESTS=1` and `.venv-rocm`.
|
|
||||||
- **Stateless decode for parity.** Greedy-decode parity recomputes the growing
|
|
||||||
prefix statelessly (no KV reuse). Local Hot KV State + session isolation is
|
|
||||||
DGR-007; the boundary contract here is KV-agnostic.
|
|
||||||
- **Native patch wiring deferred.** The C++/llama.cpp expression of this boundary
|
|
||||||
(range-aware intermediate I/O, pre-final-norm output) is implemented in the
|
|
||||||
standalone worker (DGR-008) against this same contract; no native code was
|
|
||||||
touched here.
|
|
||||||
- **Greedy-only sampling certified.** `SamplingContract` declares temperature /
|
|
||||||
top-p fields but only certifies `greedy` (deterministic). Stochastic sampling is
|
|
||||||
out of scope for the deterministic parity gate.
|
|
||||||
|
|
||||||
## Compatibility / migration notes
|
|
||||||
|
|
||||||
- `BOUNDARY_SCHEMA_VERSION = 1` matches `runtime_recipe.RuntimeRecipeIdentity`'s
|
|
||||||
`boundary_schema_version`. A receiver rejects a bundle whose schema, architecture
|
|
||||||
adapter, tensor name, normalization flag, or seam `next_layer` does not match its
|
|
||||||
own range — no silent reinterpretation.
|
|
||||||
- `BoundaryBundle.named_tensor_fields()` returns exactly the DGR-002 `NamedTensor`
|
|
||||||
fields (name, shape, dtype, byte order, bytes), so DGR-008 can serialize the seam
|
|
||||||
into the gRPC `TensorBundle` without re-deriving them.
|
|
||||||
- Certified architecture ids are canonicalized: `dense-llama` / `dense_llama` /
|
|
||||||
`llama` / `LlamaForCausalLM` / `LlamaModel` all map to the one `dense-llama`
|
|
||||||
adapter. Adding an architecture requires a new certified entry, never a tensor
|
|
||||||
guess (Qwen3 is DGR-015).
|
|
||||||
|
|
||||||
## Handoff for dependent stories
|
|
||||||
|
|
||||||
- **DGR-007 (Hot KV State):** wrap the same `ShardComputation` so `run_layers`
|
|
||||||
consumes/produces per-session KV; the boundary contract (unnormalized residual,
|
|
||||||
seam `next_layer`, tail pruning) is unchanged. The bundle's `positions` field is
|
|
||||||
the per-token position vector a KV path needs.
|
|
||||||
- **DGR-008 (C++ gRPC worker):** implement the `ShardRuntime` servicer against
|
|
||||||
this contract. Map `BoundaryBundle.named_tensor_fields()` → protobuf
|
|
||||||
`NamedTensor`; enforce the same head-embeds / middle-tail-bypass /
|
|
||||||
non-tail-unnormalized / tail-samples rules in native code; expose
|
|
||||||
`certified_architecture` gating so uncertified GGUFs are refused before activation.
|
|
||||||
- **DGR-009 (Meshnet integration):** carry `BoundaryBundle.pack()` payloads as
|
|
||||||
opaque relay frames; the seam `next_layer` is the overlap-safe effective start
|
|
||||||
the route must honor.
|
|
||||||
- **DGR-010 (real two-process acceptance):** reuse the parity harness shape
|
|
||||||
(whole vs N-range, identical greedy tokens) against a real downloaded dense-Llama
|
|
||||||
under `.venv-rocm`.
|
|
||||||
- **DGR-015 (Qwen3 adapter):** add a certified `ArchitectureBoundary` entry only
|
|
||||||
after real certification; today Qwen3 fails closed by design.
|
|
||||||
@@ -1,26 +0,0 @@
|
|||||||
# DGR-006 exact commands (run from repo worktree root)
|
|
||||||
|
|
||||||
# Targeted boundary-adapter tests
|
|
||||||
python -m pytest -q tests/test_boundary_adapter.py
|
|
||||||
# -> 22 passed in 0.26s
|
|
||||||
|
|
||||||
# Python compile check for changed Python
|
|
||||||
python -m compileall -q packages tests
|
|
||||||
# -> exit 0
|
|
||||||
|
|
||||||
# Diff hygiene
|
|
||||||
git diff --check
|
|
||||||
# -> exit 0
|
|
||||||
|
|
||||||
# Full deterministic suite with DGR-006 files present
|
|
||||||
python -m pytest -q -rfE
|
|
||||||
# -> 20 failed, 715 passed, 13 skipped, 12 errors in 239.77s
|
|
||||||
|
|
||||||
# Clean-tree reproduction: move the two new DGR-006 files aside, re-run
|
|
||||||
mv packages/node/meshnet_node/boundary_adapter.py /tmp/dgr006_boundary_adapter.py
|
|
||||||
mv tests/test_boundary_adapter.py /tmp/dgr006_test_boundary_adapter.py
|
|
||||||
python -m pytest -q -rfE
|
|
||||||
# -> 20 failed, 693 passed, 13 skipped, 12 errors in 243.10s
|
|
||||||
# (693 = 715 - 22; failure/error set byte-identical to the with-files run)
|
|
||||||
mv /tmp/dgr006_boundary_adapter.py packages/node/meshnet_node/boundary_adapter.py
|
|
||||||
mv /tmp/dgr006_test_boundary_adapter.py tests/test_boundary_adapter.py
|
|
||||||
@@ -1,161 +0,0 @@
|
|||||||
{
|
|
||||||
"story": "DGR-006",
|
|
||||||
"date": "2026-07-15",
|
|
||||||
"evidence_kind": "synthetic-unit (pure-numpy dense-Llama parity + boundary contract)",
|
|
||||||
"targeted_tests": {
|
|
||||||
"file": "tests/test_boundary_adapter.py",
|
|
||||||
"result": "22 passed"
|
|
||||||
},
|
|
||||||
"compileall": "exit 0",
|
|
||||||
"git_diff_check": "clean",
|
|
||||||
"parity_tolerance": {
|
|
||||||
"logits_atol": 1e-06,
|
|
||||||
"greedy_tokens": "identical"
|
|
||||||
},
|
|
||||||
"full_suite_with_files": {
|
|
||||||
"failed": 20,
|
|
||||||
"passed": 715,
|
|
||||||
"skipped": 13,
|
|
||||||
"errors": 12,
|
|
||||||
"seconds": 239.77
|
|
||||||
},
|
|
||||||
"full_suite_clean_tree": {
|
|
||||||
"failed": 20,
|
|
||||||
"passed": 693,
|
|
||||||
"skipped": 13,
|
|
||||||
"errors": 12,
|
|
||||||
"seconds": 243.1,
|
|
||||||
"note": "693 = 715 - 22 DGR-006 tests; failure/error set identical"
|
|
||||||
},
|
|
||||||
"failure_set_identical_with_and_without_dgr006": true,
|
|
||||||
"preexisting_unrelated_failures": [
|
|
||||||
{
|
|
||||||
"kind": "ERROR",
|
|
||||||
"nodeid": "tests/test_native_shard_protocol.py::test_capability_and_health_round_trip"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"kind": "ERROR",
|
|
||||||
"nodeid": "tests/test_native_shard_protocol.py::test_checksum_algorithms_verify"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"kind": "ERROR",
|
|
||||||
"nodeid": "tests/test_native_shard_protocol.py::test_cross_language_roundtrip_python_and_cpp"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"kind": "ERROR",
|
|
||||||
"nodeid": "tests/test_native_shard_protocol.py::test_defaults_are_stable_for_backward_compatibility"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"kind": "ERROR",
|
|
||||||
"nodeid": "tests/test_native_shard_protocol.py::test_fragment_and_reassemble_round_trip_with_checksums"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"kind": "ERROR",
|
|
||||||
"nodeid": "tests/test_native_shard_protocol.py::test_message_header_carries_every_required_field"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"kind": "ERROR",
|
|
||||||
"nodeid": "tests/test_native_shard_protocol.py::test_named_tensor_bundle_describes_shape_dtype_byteorder_and_fragments"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"kind": "ERROR",
|
|
||||||
"nodeid": "tests/test_native_shard_protocol.py::test_reassemble_detects_fragment_corruption"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"kind": "ERROR",
|
|
||||||
"nodeid": "tests/test_native_shard_protocol.py::test_service_descriptor_exposes_all_operations"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"kind": "ERROR",
|
|
||||||
"nodeid": "tests/test_native_shard_protocol.py::test_session_response_carries_structured_status_and_results"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"kind": "ERROR",
|
|
||||||
"nodeid": "tests/test_native_shard_protocol.py::test_session_stream_carries_open_prefill_decode_release_cancel"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"kind": "ERROR",
|
|
||||||
"nodeid": "tests/test_native_shard_protocol.py::test_unknown_fields_are_preserved_for_forward_compatibility"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"kind": "FAILED",
|
|
||||||
"nodeid": "tests/test_activation_compression.py::test_compressible_body_uses_zstd_when_it_clears_savings_policy"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"kind": "FAILED",
|
|
||||||
"nodeid": "tests/test_activation_compression.py::test_incompressible_body_stays_raw_after_measured_trial"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"kind": "FAILED",
|
|
||||||
"nodeid": "tests/test_activation_compression.py::test_malformed_zstd_and_legacy_raw_bodies_are_handled_explicitly"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"kind": "FAILED",
|
|
||||||
"nodeid": "tests/test_activation_compression.py::test_threshold_requires_both_byte_and_ratio_savings"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"kind": "FAILED",
|
|
||||||
"nodeid": "tests/test_dynamic_routing.py::test_admin_can_replace_a_served_model_and_release_it"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"kind": "FAILED",
|
|
||||||
"nodeid": "tests/test_gossip_and_relay.py::test_activation_compression_round_trips_and_skips_small_bodies"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"kind": "FAILED",
|
|
||||||
"nodeid": "tests/test_manual_route_benchmark.py::test_benchmark_records_one_and_two_node_routes"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"kind": "FAILED",
|
|
||||||
"nodeid": "tests/test_manual_route_benchmark.py::test_clients_without_route_are_unaffected"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"kind": "FAILED",
|
|
||||||
"nodeid": "tests/test_manual_route_benchmark.py::test_invalid_route_shape_is_400"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"kind": "FAILED",
|
|
||||||
"nodeid": "tests/test_manual_route_benchmark.py::test_pinned_route_uses_named_node"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"kind": "FAILED",
|
|
||||||
"nodeid": "tests/test_manual_route_benchmark.py::test_unknown_route_node_is_400"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"kind": "FAILED",
|
|
||||||
"nodeid": "tests/test_node_doctor.py::test_cli_doctor_flags_select_what_is_validated"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"kind": "FAILED",
|
|
||||||
"nodeid": "tests/test_openai_gateway.py::test_langchain_chat_openai"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"kind": "FAILED",
|
|
||||||
"nodeid": "tests/test_toploc_calibration_dispatch.py::test_calibration_run_dispatches_only_solo_capable_nodes"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"kind": "FAILED",
|
|
||||||
"nodeid": "tests/test_toploc_calibration_dispatch.py::test_calibration_run_node_without_commitment_endpoint_is_skipped_not_failed"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"kind": "FAILED",
|
|
||||||
"nodeid": "tests/test_toploc_calibration_dispatch.py::test_calibration_run_persists_corpus_and_results_endpoint_reports_it"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"kind": "FAILED",
|
|
||||||
"nodeid": "tests/test_tracker_capability_admission.py::test_an_enforcing_tracker_never_routes_a_node_whose_proof_does_not_cover_it[invalid]"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"kind": "FAILED",
|
|
||||||
"nodeid": "tests/test_tracker_control_plane.py::test_tracker_startup_does_not_import_or_load_model_backends"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"kind": "FAILED",
|
|
||||||
"nodeid": "tests/test_tracker_routing.py::test_shard_heal_cycle_surviving_node_covers_dead_peers_gap"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"kind": "FAILED",
|
|
||||||
"nodeid": "tests/test_tracker_routing.py::test_torch_node_applies_tracker_load_shard_directive"
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
@@ -1,229 +0,0 @@
|
|||||||
# DGR-007 — Isolated concurrent local Hot KV State: evidence
|
|
||||||
|
|
||||||
Status: done
|
|
||||||
Date: 2026-07-15
|
|
||||||
Evidence kind: **synthetic-unit** (pure-numpy KV-cached dense-Llama reference +
|
|
||||||
session/KV manager). No model download, no GPU, no torch, no network, no API
|
|
||||||
credit.
|
|
||||||
|
|
||||||
## Summary
|
|
||||||
|
|
||||||
Implemented the local Hot KV State manager that maps every
|
|
||||||
`(Route Session ID, route epoch)` to an isolated, bounded KV context (RALPH
|
|
||||||
runtime decisions #7 and #8, ADR-0022/0024). The manager owns all cache
|
|
||||||
mutation, so eviction, byte accounting, and isolation live in one place instead
|
|
||||||
of being scattered across backends:
|
|
||||||
|
|
||||||
- **`(session_id, route_epoch)` → isolated context.** Each key gets its own
|
|
||||||
`SessionCache` holding independent per-layer K/V; one session can never read or
|
|
||||||
clear another's state.
|
|
||||||
- **KV allocated only for owned layers.** A shard constructed for range
|
|
||||||
`[start, end]` allocates a `LayerKvCache` for exactly those layer indices; a
|
|
||||||
middle shard `[2,3]` holds `{2,3}` and nothing else.
|
|
||||||
- **Full lifecycle:** prefill append, decode append, truncate (rollback),
|
|
||||||
release, TTL eviction, LRU eviction (by session cap and by byte budget), and an
|
|
||||||
**explicit** `CacheMiss` (unknown-session / evicted-ttl / evicted-lru /
|
|
||||||
released / superseded-epoch / seq-len-mismatch) so the head degrades to a
|
|
||||||
from-token-zero re-prefill instead of corrupting output (decision #14).
|
|
||||||
- **Fails closed on identity.** Stale route epochs raise `StaleRouteEpochError`; a
|
|
||||||
request carrying an incompatible KV recipe raises `IncompatibleCacheRecipeError`
|
|
||||||
(fingerprint mismatch of architecture / kv dtype / head geometry / owned range);
|
|
||||||
a recipe for an uncertified architecture fails closed at construction (reusing
|
|
||||||
the DGR-006 certified-architecture gate).
|
|
||||||
- **KV-aware boundary driver.** `KvBoundaryAdapter` wraps the DGR-006
|
|
||||||
`ShardComputation` (plus `run_layers_cached`) so a shard runs cached
|
|
||||||
prefill/decode through the manager while honouring the architecture-defined
|
|
||||||
boundary contract (head embeds tokens, middle/tail bypass embedding and consume
|
|
||||||
the unnormalized residual bundle, non-tail emits the unnormalized residual, tail
|
|
||||||
normalizes + heads + prunes + samples). The computation returns the new
|
|
||||||
position-encoded K/V; the manager commits it under the budget.
|
|
||||||
|
|
||||||
A pure-numpy **KV-cached** dense-Llama reference (RMSNorm + RoPE + SwiGLU with an
|
|
||||||
absolute-position causal mask over cached keys) proves that cached prefill/decode
|
|
||||||
reproduces the stateless whole-model greedy tokens bit-for-bit, single-range and
|
|
||||||
across a head/tail seam. torch/transformers are not installed in the default
|
|
||||||
`.venv`, so a numpy reference is the only way to keep the parity + isolation gate
|
|
||||||
deterministic, download-free, and GPU-free — the identical manager contract will
|
|
||||||
be satisfied by the pinned llama.cpp worker (DGR-008), where the KV context maps
|
|
||||||
onto a llama sequence.
|
|
||||||
|
|
||||||
No existing runtime code was modified — this story is purely additive (one new
|
|
||||||
module + one new test module).
|
|
||||||
|
|
||||||
## Files changed (all new)
|
|
||||||
|
|
||||||
- `packages/node/meshnet_node/hot_kv_state.py` — the KV/session manager:
|
|
||||||
- `KvCacheRecipe` — KV layout identity (certified architecture, kv dtype, head
|
|
||||||
geometry, owned range) with `fingerprint()` / `is_compatible()` /
|
|
||||||
`bytes_per_token()`; fails closed on uncertified architectures.
|
|
||||||
- `LayerKvCache` — per-owned-layer `(seq, n_kv_heads, head_dim)` K/V with
|
|
||||||
`append` / `truncate` / `nbytes`.
|
|
||||||
- `SessionCache` — the isolated per-`(session, epoch)` context over owned layers.
|
|
||||||
- `CacheMiss` / `CacheMissReason` — the explicit, serializable miss response.
|
|
||||||
- `HotKvStateManager` — `open` / `append` / `truncate` / `release` / `resolve` /
|
|
||||||
`get`, LRU+TTL+byte-budget eviction, stale-epoch + incompatible-recipe
|
|
||||||
rejection, epoch supersession, thread-safe (RLock), injectable clock.
|
|
||||||
- `KvBoundaryAdapter` + `kv_recipe_for()` — KV-aware boundary driver.
|
|
||||||
- `tests/test_hot_kv_state.py` — pure-numpy KV-cached dense-Llama reference and 22
|
|
||||||
tests (see below).
|
|
||||||
|
|
||||||
## Acceptance criteria → evidence
|
|
||||||
|
|
||||||
- **Map `(Route Session ID, route epoch)` to an isolated context** —
|
|
||||||
`test_prefill_then_decode_append_grows_owned_layers`,
|
|
||||||
`test_four_interleaved_sessions_have_no_kv_cross_talk`,
|
|
||||||
`HotKvStateManager.open` keys sessions on `(session_id, route_epoch)`.
|
|
||||||
- **Allocate KV only for owned layers** —
|
|
||||||
`test_manager_allocates_kv_only_for_owned_layers` (middle `[2,3]` → `{2,3}`),
|
|
||||||
`test_multi_range_cached_decode_parity_across_a_seam` (head owns `(0,1,2)`, tail
|
|
||||||
owns `(3,4,5)`), `test_recipe_bytes_per_token_scales_with_owned_layers`.
|
|
||||||
- **Prefill append / decode append / truncate / release / TTL-LRU eviction /
|
|
||||||
explicit cache-miss** — `test_prefill_then_decode_append_grows_owned_layers`,
|
|
||||||
`test_truncate_rolls_back_all_owned_layers`,
|
|
||||||
`test_release_one_session_leaves_others_intact_and_returns_memory`,
|
|
||||||
`test_ttl_eviction_yields_an_explicit_cache_miss`,
|
|
||||||
`test_lru_eviction_by_session_cap_reports_a_miss`,
|
|
||||||
`test_budget_eviction_keeps_total_within_budget`,
|
|
||||||
`test_unknown_session_is_an_explicit_cache_miss`,
|
|
||||||
`test_seq_len_mismatch_is_an_explicit_cache_miss`.
|
|
||||||
- **Reject stale epochs and incompatible cache recipes** —
|
|
||||||
`test_stale_route_epoch_is_rejected`,
|
|
||||||
`test_new_route_epoch_supersedes_and_frees_old_epoch`,
|
|
||||||
`test_incompatible_cache_recipe_is_rejected`,
|
|
||||||
`test_uncertified_architecture_recipe_fails_closed`.
|
|
||||||
- **≥ four concurrent sessions complete without token or KV cross-talk** —
|
|
||||||
`test_four_interleaved_sessions_have_no_kv_cross_talk` (four interleaved
|
|
||||||
round-robin sessions, four *distinct* references, each matches its own),
|
|
||||||
`test_four_sessions_on_real_threads_stay_isolated` (four OS threads).
|
|
||||||
- **Cancellation/release leaves others intact and memory returns to budget** —
|
|
||||||
`test_release_one_session_leaves_others_intact_and_returns_memory` (released
|
|
||||||
session → `CacheMiss(RELEASED)`, `total_bytes` drops, survivors keep matching
|
|
||||||
their references), `test_single_session_exceeding_budget_raises`.
|
|
||||||
- **Cached vs stateless correctness core** —
|
|
||||||
`test_cached_full_shard_decode_matches_stateless_whole_model`,
|
|
||||||
`test_cached_prefill_next_token_matches_whole_model_logits`,
|
|
||||||
`test_multi_range_cached_decode_parity_across_a_seam`. Documented tolerance:
|
|
||||||
**identical** greedy token ids (bit-exact in practice; cached incremental
|
|
||||||
attention equals stateless full-sequence recompute per query row).
|
|
||||||
- **Targeted pytest** — `22 passed`.
|
|
||||||
- **compileall packages tests** — exit 0.
|
|
||||||
- **git diff --check** — clean.
|
|
||||||
- **Deterministic / download-free / credit-free / GPU-free** — pure numpy; fixed
|
|
||||||
RNG seed; injectable clock (no wall-clock in tests); no torch, no network, no
|
|
||||||
model files.
|
|
||||||
- **Full deterministic pytest** — `13 failed, 755 passed, 14 skipped in 254.50s`.
|
|
||||||
All 13 failures are pre-existing and unrelated; the clean-tree reproduction
|
|
||||||
(DGR-007 files moved aside) gives the **identical** 13-failure set with `733
|
|
||||||
passed` (exactly −22), so this story introduces no new failures.
|
|
||||||
- **Native C++ / CTest / llama.cpp patch stack** — **not touched by this story.**
|
|
||||||
The KV context contract is delivered at the Python manager level with a numpy
|
|
||||||
parity + isolation proof; the equivalent native layer-filtered KV / session
|
|
||||||
mapping is wired when the standalone C++ worker exists in DGR-008. No native
|
|
||||||
code, CMake, or llama.cpp patch was modified, so those gates are N/A here (same
|
|
||||||
as DGR-005/006).
|
|
||||||
|
|
||||||
## Commands and real results
|
|
||||||
|
|
||||||
```bash
|
|
||||||
VP=/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python
|
|
||||||
|
|
||||||
$VP -m pytest -q tests/test_hot_kv_state.py
|
|
||||||
# -> 22 passed in ~0.3s
|
|
||||||
|
|
||||||
$VP -m compileall -q packages tests
|
|
||||||
# -> exit 0
|
|
||||||
|
|
||||||
git diff --check
|
|
||||||
# -> exit 0
|
|
||||||
|
|
||||||
$VP -m pytest -q tests/test_boundary_adapter.py tests/test_gguf_ownership.py
|
|
||||||
# -> 25 passed
|
|
||||||
|
|
||||||
$VP -m pytest -q -rfE
|
|
||||||
# -> 13 failed, 755 passed, 14 skipped in 254.50s
|
|
||||||
|
|
||||||
# Clean-tree reproduction (DGR-007 files moved aside)
|
|
||||||
mv packages/node/meshnet_node/hot_kv_state.py /tmp/ && mv tests/test_hot_kv_state.py /tmp/
|
|
||||||
$VP -m pytest -q -rfE
|
|
||||||
# -> 13 failed, 733 passed, 14 skipped in 252.12s (identical FAILED set; passed -22)
|
|
||||||
```
|
|
||||||
|
|
||||||
`commands.txt` beside this README captures the exact commands.
|
|
||||||
|
|
||||||
## Pre-existing unrelated failures (full-suite)
|
|
||||||
|
|
||||||
`pytest -q -rfE` on `ralph/distributed-gguf-runtime` reports 13 pre-existing
|
|
||||||
failures (and, in this run, 0 errors — the earlier DGR-005/006-era
|
|
||||||
`test_native_shard_protocol.py` protobuf errors no longer appear in this
|
|
||||||
environment). None touch the KV manager. Moving the two DGR-007 files aside and
|
|
||||||
re-running yields the **byte-identical** 13-`FAILED` set (only the passed count
|
|
||||||
drops by exactly 22). The exact set (all tracker/routing/benchmark/toploc/doctor,
|
|
||||||
i.e. socket-bind / control-plane env, not KV):
|
|
||||||
|
|
||||||
```
|
|
||||||
tests/test_dynamic_routing.py::test_admin_can_replace_a_served_model_and_release_it
|
|
||||||
tests/test_manual_route_benchmark.py::test_benchmark_records_one_and_two_node_routes
|
|
||||||
tests/test_manual_route_benchmark.py::test_clients_without_route_are_unaffected
|
|
||||||
tests/test_manual_route_benchmark.py::test_invalid_route_shape_is_400
|
|
||||||
tests/test_manual_route_benchmark.py::test_pinned_route_uses_named_node
|
|
||||||
tests/test_manual_route_benchmark.py::test_unknown_route_node_is_400
|
|
||||||
tests/test_node_doctor.py::test_cli_doctor_flags_select_what_is_validated
|
|
||||||
tests/test_toploc_calibration_dispatch.py::test_calibration_run_dispatches_only_solo_capable_nodes
|
|
||||||
tests/test_toploc_calibration_dispatch.py::test_calibration_run_node_without_commitment_endpoint_is_skipped_not_failed
|
|
||||||
tests/test_toploc_calibration_dispatch.py::test_calibration_run_persists_corpus_and_results_endpoint_reports_it
|
|
||||||
tests/test_tracker_capability_admission.py::test_an_enforcing_tracker_never_routes_a_node_whose_proof_does_not_cover_it[invalid]
|
|
||||||
tests/test_tracker_routing.py::test_shard_heal_cycle_surviving_node_covers_dead_peers_gap
|
|
||||||
tests/test_tracker_routing.py::test_torch_node_applies_tracker_load_shard_directive
|
|
||||||
```
|
|
||||||
|
|
||||||
## Limitations and deferred work
|
|
||||||
|
|
||||||
- **Numpy reference, not real weights.** The parity + isolation gate uses a
|
|
||||||
deterministic numpy KV-cached dense-Llama, not a downloaded GGUF/safetensors
|
|
||||||
model. Real-model concurrent KV isolation on a downloaded dense-Llama (CPU/ROCm)
|
|
||||||
belongs to DGR-010/DGR-012 with `MESHNET_ENABLE_REAL_INFERENCE_TESTS=1` and
|
|
||||||
`.venv-rocm`.
|
|
||||||
- **Manager-owned storage, native mapping deferred.** The KV bytes are numpy
|
|
||||||
arrays managed in-process. The llama.cpp expression (a filtered llama sequence
|
|
||||||
per `(session, epoch)` over owned layers) is implemented in the standalone
|
|
||||||
worker (DGR-008) against this same manager contract; no native code was touched.
|
|
||||||
- **Continuous batching is DGR-012.** This story delivers *isolation* and bounded
|
|
||||||
lifecycle for concurrent sessions; continuous batching of compatible active
|
|
||||||
sessions inside a node (decision #9) is DGR-012 and builds on this manager.
|
|
||||||
- **Greedy-only sampling.** Reuses the DGR-006 `SamplingContract` (greedy
|
|
||||||
certified). Stochastic sampling is out of scope for the deterministic gate.
|
|
||||||
- **Coexists with legacy `SessionCacheStore`.** The older AH-25
|
|
||||||
`model_backend.SessionCacheStore` (session-id-only, opaque transformers cache,
|
|
||||||
HTTP path) is untouched. `HotKvStateManager` is the native-runtime-aligned
|
|
||||||
successor: it adds route-epoch keying, owned-layer allocation, recipe-fingerprint
|
|
||||||
rejection, and a byte budget. DGR-008/009 wire the native worker to
|
|
||||||
`HotKvStateManager`, not `SessionCacheStore`.
|
|
||||||
|
|
||||||
## Compatibility / migration notes
|
|
||||||
|
|
||||||
- `KvCacheRecipe.fingerprint()` canonicalizes the architecture (via
|
|
||||||
`certified_architecture`), so `llama` / `LlamaForCausalLM` map to the same
|
|
||||||
recipe; it aligns field-for-field with the DGR-003 `RuntimeRecipeIdentity`
|
|
||||||
compatibility discipline and reuses `runtime_recipe.compatibility_fingerprint`.
|
|
||||||
- `CacheMiss` is a value (not an exception) so it can be serialized into the
|
|
||||||
DGR-002 native protocol's cache expectation/result field; `resolve()` returns it,
|
|
||||||
`get()` raises `KvCacheMissError` wrapping it.
|
|
||||||
- The manager takes an injectable `clock` for deterministic TTL tests; production
|
|
||||||
defaults to `time.monotonic`.
|
|
||||||
|
|
||||||
## Handoff for dependent stories
|
|
||||||
|
|
||||||
- **DGR-008 (C++ gRPC worker):** implement the servicer's KV path against
|
|
||||||
`HotKvStateManager`. Map each `(Route Session ID, route epoch)` to a filtered
|
|
||||||
llama sequence over owned layers; on decode, read the sequence's cached K/V,
|
|
||||||
compute the new position-encoded K/V, and commit via `append` (honour the byte
|
|
||||||
budget and return an explicit `CacheMiss` on eviction). Enforce
|
|
||||||
`KvCacheRecipe.is_compatible` before activation and reject stale epochs.
|
|
||||||
- **DGR-009 (Meshnet integration):** the route epoch the tracker assigns is the
|
|
||||||
`route_epoch` key; carry the `CacheMiss` reason back to the head so it re-prefills
|
|
||||||
from token zero on eviction/restart.
|
|
||||||
- **DGR-012 (continuous batching):** batch compatible active sessions whose
|
|
||||||
`KvCacheRecipe` fingerprints match; each session keeps its own `SessionCache`, so
|
|
||||||
batching is a scheduling concern layered over this isolation, not a change to it.
|
|
||||||
- **DGR-013 (failure/cancel matrix):** `release` + the budget-return assertion here
|
|
||||||
is the unit-level basis for the resource-cleanup matrix.
|
|
||||||
@@ -1,31 +0,0 @@
|
|||||||
# DGR-007 — exact commands (run from the worktree root).
|
|
||||||
# Python: /run/media/popov/d/DEV/repos/d-popov.com/AI/.venv (Python 3.14.6, numpy 2.4.4).
|
|
||||||
# Root conftest.py adds packages/* to sys.path, so `meshnet_node` imports work.
|
|
||||||
|
|
||||||
VP=/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv/bin/python
|
|
||||||
|
|
||||||
# Targeted tests for this story.
|
|
||||||
$VP -m pytest -q tests/test_hot_kv_state.py
|
|
||||||
# -> 22 passed
|
|
||||||
|
|
||||||
# Python compile check for the changed packages/tests.
|
|
||||||
$VP -m compileall -q packages tests
|
|
||||||
# -> exit 0
|
|
||||||
|
|
||||||
# Diff hygiene.
|
|
||||||
git diff --check
|
|
||||||
# -> exit 0
|
|
||||||
|
|
||||||
# Dependency (DGR-006) + range-ownership (DGR-005) tests still green.
|
|
||||||
$VP -m pytest -q tests/test_boundary_adapter.py tests/test_gguf_ownership.py
|
|
||||||
# -> 25 passed
|
|
||||||
|
|
||||||
# Full deterministic suite (with DGR-007 files present).
|
|
||||||
$VP -m pytest -q -rfE
|
|
||||||
# -> see README (pre-existing unrelated failure set, +22 passed vs baseline)
|
|
||||||
|
|
||||||
# Clean-tree reproduction (DGR-007 files moved aside).
|
|
||||||
mv packages/node/meshnet_node/hot_kv_state.py /tmp/ && mv tests/test_hot_kv_state.py /tmp/
|
|
||||||
$VP -m pytest -q -rfE
|
|
||||||
# -> identical failure/error set, passed count drops by exactly 22
|
|
||||||
mv /tmp/hot_kv_state.py packages/node/meshnet_node/ && mv /tmp/test_hot_kv_state.py tests/
|
|
||||||
@@ -1,47 +0,0 @@
|
|||||||
{
|
|
||||||
"task_id": "DGR-007",
|
|
||||||
"title": "Add isolated concurrent local Hot KV State",
|
|
||||||
"status": "done",
|
|
||||||
"date": "2026-07-15",
|
|
||||||
"evidence_kind": "synthetic-unit",
|
|
||||||
"python": "/run/media/popov/d/DEV/repos/d-popov.com/AI/.venv (Python 3.14.6, numpy 2.4.4)",
|
|
||||||
"files_changed": [
|
|
||||||
"packages/node/meshnet_node/hot_kv_state.py",
|
|
||||||
"tests/test_hot_kv_state.py"
|
|
||||||
],
|
|
||||||
"gates": {
|
|
||||||
"targeted_pytest": {"command": "pytest -q tests/test_hot_kv_state.py", "result": "22 passed"},
|
|
||||||
"compileall": {"command": "python -m compileall -q packages tests", "exit": 0},
|
|
||||||
"git_diff_check": {"command": "git diff --check", "exit": 0},
|
|
||||||
"dependency_tests": {"command": "pytest -q tests/test_boundary_adapter.py tests/test_gguf_ownership.py", "result": "25 passed"},
|
|
||||||
"full_suite_with_files": {"command": "pytest -q -rfE", "result": "13 failed, 755 passed, 14 skipped", "seconds": 254.50},
|
|
||||||
"full_suite_clean_tree": {"command": "pytest -q -rfE (DGR-007 files moved aside)", "result": "13 failed, 733 passed, 14 skipped", "seconds": 252.12}
|
|
||||||
},
|
|
||||||
"no_new_failures": true,
|
|
||||||
"failure_set_identical": true,
|
|
||||||
"passed_delta": 22,
|
|
||||||
"preexisting_failures": [
|
|
||||||
"tests/test_dynamic_routing.py::test_admin_can_replace_a_served_model_and_release_it",
|
|
||||||
"tests/test_manual_route_benchmark.py::test_benchmark_records_one_and_two_node_routes",
|
|
||||||
"tests/test_manual_route_benchmark.py::test_clients_without_route_are_unaffected",
|
|
||||||
"tests/test_manual_route_benchmark.py::test_invalid_route_shape_is_400",
|
|
||||||
"tests/test_manual_route_benchmark.py::test_pinned_route_uses_named_node",
|
|
||||||
"tests/test_manual_route_benchmark.py::test_unknown_route_node_is_400",
|
|
||||||
"tests/test_node_doctor.py::test_cli_doctor_flags_select_what_is_validated",
|
|
||||||
"tests/test_toploc_calibration_dispatch.py::test_calibration_run_dispatches_only_solo_capable_nodes",
|
|
||||||
"tests/test_toploc_calibration_dispatch.py::test_calibration_run_node_without_commitment_endpoint_is_skipped_not_failed",
|
|
||||||
"tests/test_toploc_calibration_dispatch.py::test_calibration_run_persists_corpus_and_results_endpoint_reports_it",
|
|
||||||
"tests/test_tracker_capability_admission.py::test_an_enforcing_tracker_never_routes_a_node_whose_proof_does_not_cover_it[invalid]",
|
|
||||||
"tests/test_tracker_routing.py::test_shard_heal_cycle_surviving_node_covers_dead_peers_gap",
|
|
||||||
"tests/test_tracker_routing.py::test_torch_node_applies_tracker_load_shard_directive"
|
|
||||||
],
|
|
||||||
"native_gates_touched": false,
|
|
||||||
"acceptance": {
|
|
||||||
"session_epoch_isolated_context": true,
|
|
||||||
"kv_only_owned_layers": true,
|
|
||||||
"prefill_decode_truncate_release_ttl_lru_cachemiss": true,
|
|
||||||
"reject_stale_epoch_and_incompatible_recipe": true,
|
|
||||||
"four_concurrent_sessions_no_crosstalk": true,
|
|
||||||
"release_leaves_others_and_returns_memory": true
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -1,83 +0,0 @@
|
|||||||
# DGR-009 — Integrate the native worker with Meshnet: evidence
|
|
||||||
|
|
||||||
Status: done
|
|
||||||
Date: 2026-07-15
|
|
||||||
Evidence kind: **python-unit + repo-hygiene**. No model download, no GPU, no API
|
|
||||||
credit.
|
|
||||||
|
|
||||||
## Summary
|
|
||||||
|
|
||||||
Implemented the Meshnet-facing GGUF backend seam and recipe gating needed for
|
|
||||||
the native worker path:
|
|
||||||
|
|
||||||
- Added `GgufNodeBackend`, a backend-shaped adapter that lets the existing node
|
|
||||||
HTTP/control-plane code serve GGUF-backed shards without changing the
|
|
||||||
Transformers/Torch path for the default recipes.
|
|
||||||
- Added `llama-cpp-native` to the recipe manifest and gated startup so only
|
|
||||||
recipes with `backend_id == "llama.cpp"` build the GGUF backend.
|
|
||||||
- Preserved the existing registration/admission flow by carrying the validated
|
|
||||||
capability report and proof shard through registration.
|
|
||||||
- Added unit coverage for the GGUF backend seam and for recipe-gated startup.
|
|
||||||
- Fixed the explicit-shard startup path so the legacy Torch tests that use an
|
|
||||||
opaque stub model still pass without requiring HuggingFace config discovery.
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `packages/node/meshnet_node/gguf_backend.py` - new GGUF backend adapter and
|
|
||||||
worker-transport boundary.
|
|
||||||
- `packages/node/meshnet_node/startup.py` - recipe-gated GGUF backend injection
|
|
||||||
and explicit-shard startup fix.
|
|
||||||
- `packages/node/meshnet_node/recipes.json` - added `llama-cpp-native`.
|
|
||||||
- `tests/test_gguf_backend.py` - backend delegation and recipe-selection tests.
|
|
||||||
- `.ralph-tui/progress.md` - appended DGR-009 progress note.
|
|
||||||
- `.scratch/distributed-gguf-runtime/issues/09-integrate-the-native-worker-with-meshnet.md`
|
|
||||||
- marked `Status: done`.
|
|
||||||
|
|
||||||
## Commands and real results
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python -m pytest -q tests/test_gguf_backend.py
|
|
||||||
# -> 2 passed in 0.05s
|
|
||||||
|
|
||||||
python -m pytest -q tests/test_node_admission.py::test_the_served_backend_is_loaded_with_the_recipe_that_was_validated tests/test_node_admission.py::test_backend_validation_failure_registers_nothing
|
|
||||||
# -> 2 passed in 0.07s
|
|
||||||
|
|
||||||
python -m compileall -q packages tests
|
|
||||||
# -> exit 0
|
|
||||||
|
|
||||||
git diff --check
|
|
||||||
# -> exit 0
|
|
||||||
|
|
||||||
python -m pytest -q
|
|
||||||
# -> 222 failed, 463 passed, 13 skipped, 86 errors in 135.65s
|
|
||||||
```
|
|
||||||
|
|
||||||
## Limitations
|
|
||||||
|
|
||||||
- `python -m pytest -q` is still not clean in this sandbox. The dominant
|
|
||||||
failures are tracker/control-plane socket `PermissionError: [Errno 1]
|
|
||||||
Operation not permitted` and a native protocol import failure caused by a
|
|
||||||
protobuf runtime mismatch (`gencode 7.35.0` vs runtime `6.33.6`).
|
|
||||||
- `tests/test_native_shard_protocol.py` currently fails for the same protobuf
|
|
||||||
runtime mismatch in this environment.
|
|
||||||
- `DGR-008` evidence was not present in the tree, so the dependency behavior was
|
|
||||||
verified by reading the live code and exercising the Python seam instead of
|
|
||||||
relying on a missing README.
|
|
||||||
|
|
||||||
## Compatibility notes
|
|
||||||
|
|
||||||
- The default Torch path remains intact; GGUF backend selection is explicit and
|
|
||||||
recipe-gated.
|
|
||||||
- `TorchNodeServer` already accepts an injected backend object, so the control
|
|
||||||
plane stays Meshnet-owned.
|
|
||||||
- The GGUF adapter currently establishes the seam for the native worker
|
|
||||||
transport; the compiled worker remains the owner of the gRPC protocol details.
|
|
||||||
|
|
||||||
## Dependent-story handoff
|
|
||||||
|
|
||||||
- DGR-008 should continue to own the native worker implementation and the
|
|
||||||
versioned gRPC frame handling behind `MESHNET_NATIVE_WORKER_URL`.
|
|
||||||
- DGR-010 / DGR-012 can build on this seam without changing the control plane:
|
|
||||||
the recipe-gated backend and validated capability report are already carried
|
|
||||||
through startup.
|
|
||||||
|
|
||||||
@@ -1,58 +0,0 @@
|
|||||||
# DGR-010 — Blocked handoff
|
|
||||||
|
|
||||||
Status: blocked
|
|
||||||
Date: 2026-07-15
|
|
||||||
|
|
||||||
## Blocker
|
|
||||||
|
|
||||||
I verified the local workspace and mounted-drive model storage, but there is no
|
|
||||||
certified dense-Llama artifact available on this machine to run the required
|
|
||||||
real-model two-process acceptance.
|
|
||||||
|
|
||||||
What I found:
|
|
||||||
|
|
||||||
- `/run/media/popov/d/DEV/models` contains Qwen artifacts and caches, but no
|
|
||||||
dense-Llama model snapshot or GGUF artifact.
|
|
||||||
- `/run/media/popov/d/DEV/llamacpp/llama.cpp/models` contains only vocab GGUFs,
|
|
||||||
not a certified dense-Llama model.
|
|
||||||
- The existing code paths for real startup, GGUF backend selection, Hot KV
|
|
||||||
isolation, and benchmark reporting are present and readable, but the actual
|
|
||||||
DGR-010 acceptance run needs a certified dense-Llama artifact from mounted
|
|
||||||
storage to satisfy the story contract.
|
|
||||||
|
|
||||||
## Verified current state
|
|
||||||
|
|
||||||
- DGR-009 evidence was read and verified as the dependency handoff.
|
|
||||||
- `packages/node/meshnet_node/startup.py` already gates backend selection by
|
|
||||||
recipe and can load either the Torch path or the explicit GGUF seam.
|
|
||||||
- `packages/node/meshnet_node/hot_kv_state.py`, `boundary_adapter.py`, and
|
|
||||||
`gguf_ownership.py` already provide the isolation/parity seams that DGR-010
|
|
||||||
would exercise.
|
|
||||||
- The repo has no existing `evidence/DGR-010/README.md` yet, which is expected
|
|
||||||
because the story has not been completed.
|
|
||||||
|
|
||||||
## Commands run
|
|
||||||
|
|
||||||
```bash
|
|
||||||
sed -n '1,260p' .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md
|
|
||||||
sed -n '1,260p' .scratch/distributed-gguf-runtime/issues/10-pass-local-real-model-two-process-acceptance.md
|
|
||||||
sed -n '1,260p' .scratch/distributed-gguf-runtime/evidence/DGR-009/README.md
|
|
||||||
git status --short
|
|
||||||
find /run/media/popov/d/DEV -type f \( -name '*.gguf' -o -name '*.safetensors' -o -name 'config.json' \) | rg -i 'llama|tinyllama|meta-llama|hf-internal-testing|qwen'
|
|
||||||
```
|
|
||||||
|
|
||||||
## Next step to unblock
|
|
||||||
|
|
||||||
Provide or mount a certified dense-Llama artifact on the configured mounted
|
|
||||||
drive storage, then rerun the DGR-010 acceptance path with
|
|
||||||
`MESHNET_ENABLE_REAL_INFERENCE_TESTS=1`.
|
|
||||||
|
|
||||||
## Continuation note
|
|
||||||
|
|
||||||
Once the artifact exists, the next iteration should:
|
|
||||||
|
|
||||||
1. Run the two local worker processes against the certified dense-Llama shard
|
|
||||||
ranges.
|
|
||||||
2. Capture parity, concurrency, memory, and failure metrics.
|
|
||||||
3. Write `evidence/DGR-010/README.md` with the real results and then update the
|
|
||||||
issue status.
|
|
||||||
@@ -1,70 +0,0 @@
|
|||||||
# DGR-011 — Blocked handoff
|
|
||||||
|
|
||||||
Status: blocked
|
|
||||||
Date: 2026-07-15
|
|
||||||
|
|
||||||
## Blocker
|
|
||||||
|
|
||||||
This story cannot be completed in the current workspace state because its
|
|
||||||
mandatory dependency, DGR-010, is still not passed.
|
|
||||||
|
|
||||||
Verified blockers:
|
|
||||||
|
|
||||||
- `.scratch/distributed-gguf-runtime/prd.json` still marks `DGR-010` and
|
|
||||||
`DGR-011` with `"passes": false`.
|
|
||||||
- `.scratch/distributed-gguf-runtime/evidence/DGR-010/README.md` does not
|
|
||||||
exist, and the only DGR-010 evidence artifact present is
|
|
||||||
`.scratch/distributed-gguf-runtime/evidence/DGR-010/BLOCKED.md`.
|
|
||||||
- Mounted storage search found Qwen model artifacts and llama.cpp vocab files,
|
|
||||||
but no certified dense-Llama GGUF artifact suitable for the required real
|
|
||||||
acceptance run.
|
|
||||||
|
|
||||||
## Verified current state
|
|
||||||
|
|
||||||
- The repo already contains the Meshnet-facing GGUF backend seam and the
|
|
||||||
recipe-gated startup path from DGR-009.
|
|
||||||
- The architecture and Ralph context require real-model execution for this
|
|
||||||
story, not synthetic workers or unit-only coverage.
|
|
||||||
- The current environment does not expose the dense-Llama artifact required to
|
|
||||||
run the prerequisite local real-model acceptance, so the two-machine route
|
|
||||||
cannot be proven end to end.
|
|
||||||
|
|
||||||
## Commands run
|
|
||||||
|
|
||||||
```bash
|
|
||||||
sed -n '1,260p' .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md
|
|
||||||
sed -n '1,260p' .scratch/distributed-gguf-runtime/issues/11-pass-a-real-heterogeneous-two-machine-route.md
|
|
||||||
sed -n '1,260p' .ralph-tui/progress.md
|
|
||||||
sed -n '1,240p' .scratch/distributed-gguf-runtime/evidence/DGR-010/BLOCKED.md
|
|
||||||
sed -n '1,220p' CONTEXT.md
|
|
||||||
sed -n '1,260p' docs/adr/0024-distributed-gguf-runtime.md
|
|
||||||
sed -n '282,350p' .scratch/distributed-gguf-runtime/prd.json
|
|
||||||
find /run/media/popov/d/DEV/models -maxdepth 3 \( -name '*.gguf' -o -name 'config.json' -o -name '*.safetensors' \)
|
|
||||||
find /run/media/popov/d/DEV/llamacpp/llama.cpp/models /run/media/popov/d/DEV/models -maxdepth 4 \( -iname '*llama*' -o -iname '*dense*' -o -iname '*qwen*' -o -name 'config.json' -o -name '*.gguf' \)
|
|
||||||
```
|
|
||||||
|
|
||||||
## Known limitations
|
|
||||||
|
|
||||||
- No certified dense-Llama artifact is available on mounted storage in this
|
|
||||||
workspace.
|
|
||||||
- No real two-machine execution was possible, so there are no real route,
|
|
||||||
hardware, backend, or drift metrics to record for this story.
|
|
||||||
- The story remains blocked until DGR-010 is completed with a real-model
|
|
||||||
evidence README and a confirmed dense-Llama artifact on mounted storage.
|
|
||||||
|
|
||||||
## Compatibility notes
|
|
||||||
|
|
||||||
- DGR-009's recipe-gated GGUF backend seam is present and can be reused.
|
|
||||||
- The acceptance path for this story still requires the upstream real-model
|
|
||||||
evidence from DGR-010 before any heterogeneous two-machine route can be
|
|
||||||
claimed.
|
|
||||||
|
|
||||||
## Dependent-story handoff
|
|
||||||
|
|
||||||
- Finish DGR-010 first, including its real-model evidence README and
|
|
||||||
acceptance run.
|
|
||||||
- Once DGR-010 passes, rerun the two-machine acceptance against the same
|
|
||||||
certified dense-Llama artifact, then record the two-host hardware/network
|
|
||||||
manifest, route, commands, and raw metrics in `evidence/DGR-011/README.md`.
|
|
||||||
- Do not update the issue to `Status: done` until the real two-machine route
|
|
||||||
has been executed and recorded.
|
|
||||||
@@ -13,15 +13,6 @@ Status: ready-for-agent
|
|||||||
|
|
||||||
As a runtime engineer, I need a controlled baseline so that GGUF work proceeds from measured speed, memory, and quality rather than reputation.
|
As a runtime engineer, I need a controlled baseline so that GGUF work proceeds from measured speed, memory, and quality rather than reputation.
|
||||||
|
|
||||||
## Baseline model target
|
|
||||||
|
|
||||||
Use the same model on both sides of the comparison, with the closest practical low-footprint precision pair:
|
|
||||||
|
|
||||||
- **safetensors:** `deepseek-ai/DeepSeek-V2-Lite-Chat` in **BF16**
|
|
||||||
- **GGUF:** `second-state/DeepSeek-V2-Lite-Chat-GGUF` in **Q2_K** (~6.5GB)
|
|
||||||
|
|
||||||
Keep the benchmark matrix explicit for **CPU** and **GPU** runs. Reserve smaller non-DeepSeek fallback models only for loader plumbing smoke tests if needed; they do not count as the DGR-001 architecture-aligned baseline.
|
|
||||||
|
|
||||||
## Expected durable outputs
|
## Expected durable outputs
|
||||||
|
|
||||||
- Benchmark harness and deterministic tests
|
- Benchmark harness and deterministic tests
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# 02 — Adopt the versioned gRPC Shard protocol
|
# 02 — Adopt the versioned gRPC Shard protocol
|
||||||
|
|
||||||
Status: done
|
Status: ready-for-agent
|
||||||
|
|
||||||
## Mandatory fresh-session context
|
## Mandatory fresh-session context
|
||||||
|
|
||||||
@@ -22,22 +22,22 @@ As a node developer, I need a battle-proven streaming protocol so that Python an
|
|||||||
|
|
||||||
## Acceptance criteria
|
## Acceptance criteria
|
||||||
|
|
||||||
- [x] Add a Protocol Buffers schema for capability, health, session stream, release, and cancellation operations.
|
- [ ] Add a Protocol Buffers schema for capability, health, session stream, release, and cancellation operations.
|
||||||
- [x] Define one long-lived bidirectional gRPC stream per Route Session Activation Seam with deadlines, cancellation, flow control, and structured errors.
|
- [ ] Define one long-lived bidirectional gRPC stream per Route Session Activation Seam with deadlines, cancellation, flow control, and structured errors.
|
||||||
- [x] Define bounded chunking for prefill and a small decode fast path.
|
- [ ] Define bounded chunking for prefill and a small decode fast path.
|
||||||
- [x] Carry schema version, request/work ID, Route Session ID, route epoch, artifact/recipe fingerprint, Shard range/effective start, phase, position, idempotency step, cache expectation, compression, and checksum.
|
- [ ] Carry schema version, request/work ID, Route Session ID, route epoch, artifact/recipe fingerprint, Shard range/effective start, phase, position, idempotency step, cache expectation, compression, and checksum.
|
||||||
- [x] Define a versioned named-tensor bundle with per-tensor name, shape, dtype, byte order, and payload fragments.
|
- [ ] Define a versioned named-tensor bundle with per-tensor name, shape, dtype, byte order, and payload fragments.
|
||||||
- [x] Add generated-schema round-trip and compatibility tests in Python and C++.
|
- [ ] Add generated-schema round-trip and compatibility tests in Python and C++.
|
||||||
- [x] Targeted pytest tests pass
|
- [ ] Targeted pytest tests pass
|
||||||
- [x] python -m compileall packages tests passes for Python changes
|
- [ ] python -m compileall packages tests passes for Python changes
|
||||||
- [x] git diff --check passes
|
- [ ] git diff --check passes
|
||||||
- [x] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
|
- [ ] Default tests remain deterministic, model-download-free, API-credit-free, and GPU-free
|
||||||
- [x] Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
|
- [ ] Full deterministic pytest -q passes, or the exact pre-existing unrelated failure is recorded with a clean-tree reproduction
|
||||||
- [x] Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
|
- [ ] Read .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md and this story issue completely before changing code
|
||||||
- [x] Read and verify every dependency evidence README before relying on dependency behavior
|
- [ ] Read and verify every dependency evidence README before relying on dependency behavior
|
||||||
- [x] Preserve all pre-existing working-tree changes and stage only files belonging to this story
|
- [ ] Preserve all pre-existing working-tree changes and stage only files belonging to this story
|
||||||
- [x] Write .scratch/distributed-gguf-runtime/evidence/DGR-002/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
|
- [ ] Write .scratch/distributed-gguf-runtime/evidence/DGR-002/README.md with files changed, exact commands and real results, limitations, compatibility notes, and dependent-story handoff
|
||||||
- [x] Update only this story issue to Status: done after every acceptance criterion and quality gate passes
|
- [ ] Update only this story issue to Status: done after every acceptance criterion and quality gate passes
|
||||||
|
|
||||||
## Dependency handoff
|
## Dependency handoff
|
||||||
|
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# 03 — Define exact Artifact and runtime recipe identity
|
# 03 — Define exact Artifact and runtime recipe identity
|
||||||
|
|
||||||
Status: done
|
Status: ready-for-agent
|
||||||
|
|
||||||
## Mandatory fresh-session context
|
## Mandatory fresh-session context
|
||||||
|
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# 04 — Create the reproducible pinned llama.cpp patch stack
|
# 04 — Create the reproducible pinned llama.cpp patch stack
|
||||||
|
|
||||||
Status: done
|
Status: ready-for-agent
|
||||||
|
|
||||||
## Mandatory fresh-session context
|
## Mandatory fresh-session context
|
||||||
|
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# 05 — Implement dense-Llama range-aware GGUF ownership
|
# 05 — Implement dense-Llama range-aware GGUF ownership
|
||||||
|
|
||||||
Status: done
|
Status: ready-for-agent
|
||||||
|
|
||||||
## Mandatory fresh-session context
|
## Mandatory fresh-session context
|
||||||
|
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# 06 — Implement architecture-defined boundary input/output
|
# 06 — Implement architecture-defined boundary input/output
|
||||||
|
|
||||||
Status: done
|
Status: ready-for-agent
|
||||||
|
|
||||||
## Mandatory fresh-session context
|
## Mandatory fresh-session context
|
||||||
|
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# 07 — Add isolated concurrent local Hot KV State
|
# 07 — Add isolated concurrent local Hot KV State
|
||||||
|
|
||||||
Status: done
|
Status: ready-for-agent
|
||||||
|
|
||||||
## Mandatory fresh-session context
|
## Mandatory fresh-session context
|
||||||
|
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# 09 — Integrate the native worker with Meshnet
|
# 09 — Integrate the native worker with Meshnet
|
||||||
|
|
||||||
Status: done
|
Status: ready-for-agent
|
||||||
|
|
||||||
## Mandatory fresh-session context
|
## Mandatory fresh-session context
|
||||||
|
|
||||||
|
|||||||
@@ -54,7 +54,7 @@
|
|||||||
"Update only this story issue to Status: done after every acceptance criterion and quality gate passes"
|
"Update only this story issue to Status: done after every acceptance criterion and quality gate passes"
|
||||||
],
|
],
|
||||||
"priority": 1,
|
"priority": 1,
|
||||||
"passes": true,
|
"passes": false,
|
||||||
"notes": "Source issue: .scratch/distributed-gguf-runtime/issues/02-adopt-the-versioned-grpc-shard-protocol.md",
|
"notes": "Source issue: .scratch/distributed-gguf-runtime/issues/02-adopt-the-versioned-grpc-shard-protocol.md",
|
||||||
"dependsOn": []
|
"dependsOn": []
|
||||||
},
|
},
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
Status: done (2026-07-14)
|
Status: ready-for-agent
|
||||||
|
|
||||||
# 01 — Baseline and profiling harness
|
# 01 — Baseline and profiling harness
|
||||||
|
|
||||||
@@ -12,15 +12,16 @@ sizes and connection counts without requiring a real model or external host.
|
|||||||
|
|
||||||
## Acceptance criteria
|
## Acceptance criteria
|
||||||
|
|
||||||
- [x] The harness runs a fixed prompt and fixed generated-token count through a
|
- [ ] The harness runs a fixed prompt and fixed generated-token count through a
|
||||||
two-node route in direct and relay modes.
|
two-node route in direct and relay modes.
|
||||||
- [x] It reports p50/p95 per-token latency, per-hop latency, payload bytes,
|
- [ ] It reports p50/p95 per-token latency, per-hop latency, payload bytes,
|
||||||
compression ratio, connection attempts, and queue wait.
|
compression ratio, connection attempts, and queue wait.
|
||||||
- [x] It distinguishes prefill from decode and cached from stateless mode.
|
- [ ] It distinguishes prefill from decode and cached from stateless mode.
|
||||||
- [x] It emits machine-readable JSON suitable for CI artifacts and a concise
|
- [ ] It emits machine-readable JSON suitable for CI artifacts and a concise
|
||||||
human-readable summary.
|
human-readable summary.
|
||||||
- [x] A test fixture can assert connection attempts and output token identity.
|
- [ ] A test fixture can assert connection attempts and output token identity.
|
||||||
|
|
||||||
## Blocked by
|
## Blocked by
|
||||||
|
|
||||||
None - completed. Verified with `PYTHONPATH=packages/node pytest -q tests/test_route_session_benchmark.py` (7 passed).
|
None - can start immediately.
|
||||||
|
|
||||||
|
|||||||
@@ -15,10 +15,9 @@
|
|||||||
"Can assert connection count and output token identity"
|
"Can assert connection count and output token identity"
|
||||||
],
|
],
|
||||||
"priority": 1,
|
"priority": 1,
|
||||||
"passes": true,
|
"passes": false,
|
||||||
"notes": "Source issue: .scratch/distributed-inference-performance/issues/01-baseline-profiling-harness.md",
|
"notes": "Source issue: .scratch/distributed-inference-performance/issues/01-baseline-profiling-harness.md",
|
||||||
"dependsOn": [],
|
"dependsOn": []
|
||||||
"completionNotes": "Completed by agent"
|
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"id": "DIP-002",
|
"id": "DIP-002",
|
||||||
@@ -32,12 +31,9 @@
|
|||||||
"Tests cover binary, JSON, timeout, disconnect, cancellation, and cleanup"
|
"Tests cover binary, JSON, timeout, disconnect, cancellation, and cleanup"
|
||||||
],
|
],
|
||||||
"priority": 2,
|
"priority": 2,
|
||||||
"passes": true,
|
"passes": false,
|
||||||
"notes": "Source issue: .scratch/distributed-inference-performance/issues/02-relay-session-compatibility.md",
|
"notes": "Source issue: .scratch/distributed-inference-performance/issues/02-relay-session-compatibility.md",
|
||||||
"dependsOn": [
|
"dependsOn": ["DIP-001"]
|
||||||
"DIP-001"
|
|
||||||
],
|
|
||||||
"completionNotes": "Completed by agent"
|
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"id": "DIP-003",
|
"id": "DIP-003",
|
||||||
@@ -51,12 +47,9 @@
|
|||||||
"Benchmark shows healthy-session connection count independent of token count"
|
"Benchmark shows healthy-session connection count independent of token count"
|
||||||
],
|
],
|
||||||
"priority": 3,
|
"priority": 3,
|
||||||
"passes": true,
|
"passes": false,
|
||||||
"notes": "Source issue: .scratch/distributed-inference-performance/issues/03-http-keepalive.md",
|
"notes": "Source issue: .scratch/distributed-inference-performance/issues/03-http-keepalive.md",
|
||||||
"dependsOn": [
|
"dependsOn": ["DIP-001"]
|
||||||
"DIP-001"
|
|
||||||
],
|
|
||||||
"completionNotes": "Completed by agent"
|
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"id": "DIP-004",
|
"id": "DIP-004",
|
||||||
@@ -70,12 +63,9 @@
|
|||||||
"Tests verify cadence and cleanup"
|
"Tests verify cadence and cleanup"
|
||||||
],
|
],
|
||||||
"priority": 4,
|
"priority": 4,
|
||||||
"passes": true,
|
"passes": false,
|
||||||
"notes": "Source issue: .scratch/distributed-inference-performance/issues/04-seam-telemetry.md",
|
"notes": "Source issue: .scratch/distributed-inference-performance/issues/04-seam-telemetry.md",
|
||||||
"dependsOn": [
|
"dependsOn": ["DIP-001"]
|
||||||
"DIP-001"
|
|
||||||
],
|
|
||||||
"completionNotes": "Completed by agent"
|
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"id": "DIP-005",
|
"id": "DIP-005",
|
||||||
@@ -89,12 +79,9 @@
|
|||||||
"Tests cover compressible, incompressible, threshold, malformed, and legacy bodies"
|
"Tests cover compressible, incompressible, threshold, malformed, and legacy bodies"
|
||||||
],
|
],
|
||||||
"priority": 5,
|
"priority": 5,
|
||||||
"passes": true,
|
"passes": false,
|
||||||
"notes": "Source issue: .scratch/distributed-inference-performance/issues/05-adaptive-compression.md",
|
"notes": "Source issue: .scratch/distributed-inference-performance/issues/05-adaptive-compression.md",
|
||||||
"dependsOn": [
|
"dependsOn": ["DIP-001"]
|
||||||
"DIP-001"
|
|
||||||
],
|
|
||||||
"completionNotes": "Completed by agent"
|
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"id": "DIP-006",
|
"id": "DIP-006",
|
||||||
@@ -108,12 +95,9 @@
|
|||||||
"Wire and token-output regression tests pass"
|
"Wire and token-output regression tests pass"
|
||||||
],
|
],
|
||||||
"priority": 6,
|
"priority": 6,
|
||||||
"passes": true,
|
"passes": false,
|
||||||
"notes": "Source issue: .scratch/distributed-inference-performance/issues/06-activation-framing-copies.md",
|
"notes": "Source issue: .scratch/distributed-inference-performance/issues/06-activation-framing-copies.md",
|
||||||
"dependsOn": [
|
"dependsOn": ["DIP-001"]
|
||||||
"DIP-001"
|
|
||||||
],
|
|
||||||
"completionNotes": "Completed by agent"
|
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"id": "DIP-007",
|
"id": "DIP-007",
|
||||||
@@ -127,13 +111,9 @@
|
|||||||
"Tests cover chunking, slow consumers, failure, and legacy peers"
|
"Tests cover chunking, slow consumers, failure, and legacy peers"
|
||||||
],
|
],
|
||||||
"priority": 7,
|
"priority": 7,
|
||||||
"passes": true,
|
"passes": false,
|
||||||
"notes": "Source issue: .scratch/distributed-inference-performance/issues/07-prefill-backpressure.md",
|
"notes": "Source issue: .scratch/distributed-inference-performance/issues/07-prefill-backpressure.md",
|
||||||
"dependsOn": [
|
"dependsOn": ["DIP-001", "DIP-004"]
|
||||||
"DIP-001",
|
|
||||||
"DIP-004"
|
|
||||||
],
|
|
||||||
"completionNotes": "Completed by agent"
|
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"id": "DIP-008",
|
"id": "DIP-008",
|
||||||
@@ -147,20 +127,9 @@
|
|||||||
"Gate verifies token identity, session stability, and resource cleanup"
|
"Gate verifies token identity, session stability, and resource cleanup"
|
||||||
],
|
],
|
||||||
"priority": 8,
|
"priority": 8,
|
||||||
"passes": true,
|
"passes": false,
|
||||||
"notes": "Source issue: .scratch/distributed-inference-performance/issues/08-end-to-end-performance-gate.md",
|
"notes": "Source issue: .scratch/distributed-inference-performance/issues/08-end-to-end-performance-gate.md",
|
||||||
"dependsOn": [
|
"dependsOn": ["DIP-002", "DIP-003", "DIP-004", "DIP-005", "DIP-006", "DIP-007"]
|
||||||
"DIP-002",
|
|
||||||
"DIP-003",
|
|
||||||
"DIP-004",
|
|
||||||
"DIP-005",
|
|
||||||
"DIP-006",
|
|
||||||
"DIP-007"
|
|
||||||
],
|
|
||||||
"completionNotes": "Completed by agent"
|
|
||||||
}
|
}
|
||||||
],
|
]
|
||||||
"metadata": {
|
|
||||||
"updatedAt": "2026-07-12T02:35:28.752Z"
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
@@ -35,12 +35,11 @@
|
|||||||
"Full pytest passes or an exact unrelated blocker is recorded"
|
"Full pytest passes or an exact unrelated blocker is recorded"
|
||||||
],
|
],
|
||||||
"priority": 2,
|
"priority": 2,
|
||||||
"passes": true,
|
"passes": false,
|
||||||
"notes": "Source issue: .scratch/node-capability-admission/issues/02-doctor-real-forward.md",
|
"notes": "Source issue: .scratch/node-capability-admission/issues/02-doctor-real-forward.md",
|
||||||
"dependsOn": [
|
"dependsOn": [
|
||||||
"NCA-001"
|
"NCA-001"
|
||||||
],
|
]
|
||||||
"completionNotes": "Completed by agent"
|
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"id": "NCA-003",
|
"id": "NCA-003",
|
||||||
@@ -55,13 +54,12 @@
|
|||||||
"Full pytest passes or an exact unrelated blocker is recorded"
|
"Full pytest passes or an exact unrelated blocker is recorded"
|
||||||
],
|
],
|
||||||
"priority": 3,
|
"priority": 3,
|
||||||
"passes": true,
|
"passes": false,
|
||||||
"notes": "Source issue: .scratch/node-capability-admission/issues/03-fail-closed-startup-admission.md",
|
"notes": "Source issue: .scratch/node-capability-admission/issues/03-fail-closed-startup-admission.md",
|
||||||
"dependsOn": [
|
"dependsOn": [
|
||||||
"NCA-001",
|
"NCA-001",
|
||||||
"NCA-002"
|
"NCA-002"
|
||||||
],
|
]
|
||||||
"completionNotes": "Completed by agent"
|
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"id": "NCA-004",
|
"id": "NCA-004",
|
||||||
@@ -78,13 +76,12 @@
|
|||||||
"Full pytest passes or an exact unrelated blocker is recorded"
|
"Full pytest passes or an exact unrelated blocker is recorded"
|
||||||
],
|
],
|
||||||
"priority": 4,
|
"priority": 4,
|
||||||
"passes": true,
|
"passes": false,
|
||||||
"notes": "Source issue: .scratch/node-capability-admission/issues/04-tracker-validated-capability-routing.md",
|
"notes": "Source issue: .scratch/node-capability-admission/issues/04-tracker-validated-capability-routing.md",
|
||||||
"dependsOn": [
|
"dependsOn": [
|
||||||
"NCA-001",
|
"NCA-001",
|
||||||
"NCA-003"
|
"NCA-003"
|
||||||
],
|
]
|
||||||
"completionNotes": "Completed by agent"
|
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"id": "NCA-005",
|
"id": "NCA-005",
|
||||||
@@ -99,16 +96,15 @@
|
|||||||
"Full pytest passes or an exact unrelated blocker is recorded"
|
"Full pytest passes or an exact unrelated blocker is recorded"
|
||||||
],
|
],
|
||||||
"priority": 5,
|
"priority": 5,
|
||||||
"passes": true,
|
"passes": false,
|
||||||
"notes": "Source issue: .scratch/node-capability-admission/issues/05-docs-hardware-lane-contract.md",
|
"notes": "Source issue: .scratch/node-capability-admission/issues/05-docs-hardware-lane-contract.md",
|
||||||
"dependsOn": [
|
"dependsOn": [
|
||||||
"NCA-002",
|
"NCA-002",
|
||||||
"NCA-004"
|
"NCA-004"
|
||||||
],
|
]
|
||||||
"completionNotes": "Completed by agent"
|
|
||||||
}
|
}
|
||||||
],
|
],
|
||||||
"metadata": {
|
"metadata": {
|
||||||
"updatedAt": "2026-07-12T01:54:03.030Z"
|
"updatedAt": "2026-07-11T19:16:52.768Z"
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -16,9 +16,12 @@
|
|||||||
|
|
||||||
|
|
||||||
.\.venv\Scripts\meshnet-node.exe start http://192.168.0.179:8081 --model-id Qwen/Qwen2.5-0.5B-Instruct --advertise-host 192.168.0.20
|
.\.venv\Scripts\meshnet-node.exe start http://192.168.0.179:8081 --model-id Qwen/Qwen2.5-0.5B-Instruct --advertise-host 192.168.0.20
|
||||||
.\.venv\Scripts\meshnet-node.exe start --tracker http://ai.neuron.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct --advertise-host 192.168.0.20
|
.\.venv\Scripts\meshnet-node.exe start --tracker http://ai.neuron.d-popov.com --model-id Qwen/Qwen2.5-0.5B-Instruct --advertise-host 192.168.0.20
|
||||||
|
|
||||||
we .\.venv\Scripts\meshnet-node.exe start --tracker http://192.168.0.179:8081 --model Qwen/Qwen2.5-0.5B-Instruct
|
we .\.venv\Scripts\meshnet-node.exe start `
|
||||||
|
--tracker http://192.168.0.179:8081 `
|
||||||
|
--model Qwen/Qwen2.5-0.5B-Instruct `
|
||||||
|
--advertise-host 192.168.0.20
|
||||||
# trackers:
|
# trackers:
|
||||||
https://meshnet.2.d-popov.com
|
https://meshnet.2.d-popov.com
|
||||||
https://ai.neuron.d-popov.com
|
https://ai.neuron.d-popov.com
|
||||||
|
|||||||
@@ -20,17 +20,9 @@ import time
|
|||||||
from dataclasses import dataclass
|
from dataclasses import dataclass
|
||||||
from typing import Any, Callable
|
from typing import Any, Callable
|
||||||
|
|
||||||
from . import __version__ as _PACKAGE_VERSION
|
from .capability import CapabilityReport
|
||||||
from .capability import CapabilityReport, config_fingerprint
|
|
||||||
from .doctor import DoctorSelection
|
from .doctor import DoctorSelection
|
||||||
from .recipe_manifest import Recipe, RecipeManifest
|
from .recipe_manifest import Recipe, RecipeManifest
|
||||||
from .runtime_recipe import (
|
|
||||||
build_artifact_identity,
|
|
||||||
build_runtime_recipe_identity,
|
|
||||||
compatibility_fingerprint,
|
|
||||||
fingerprint_payload,
|
|
||||||
)
|
|
||||||
from .gguf_ownership import authoritative_dense_llama_ownership
|
|
||||||
|
|
||||||
# How long a passing report stays usable. Startup normally validates in-process
|
# How long a passing report stays usable. Startup normally validates in-process
|
||||||
# (age ≈ 0); this bounds how far a report written by an earlier `doctor` run can
|
# (age ≈ 0); this bounds how far a report written by an earlier `doctor` run can
|
||||||
@@ -47,7 +39,6 @@ REASON_MODEL_MISMATCH = "model-mismatch"
|
|||||||
REASON_SHARD_MISMATCH = "shard-mismatch"
|
REASON_SHARD_MISMATCH = "shard-mismatch"
|
||||||
REASON_RECIPE_MISMATCH = "recipe-mismatch"
|
REASON_RECIPE_MISMATCH = "recipe-mismatch"
|
||||||
REASON_BACKEND_MISMATCH = "backend-mismatch"
|
REASON_BACKEND_MISMATCH = "backend-mismatch"
|
||||||
REASON_COMPATIBILITY_MISMATCH = "compatibility-mismatch"
|
|
||||||
|
|
||||||
|
|
||||||
class CapabilityAdmissionError(RuntimeError):
|
class CapabilityAdmissionError(RuntimeError):
|
||||||
@@ -86,7 +77,6 @@ class AdmissionRequirement:
|
|||||||
recipe_version: str
|
recipe_version: str
|
||||||
backend_id: str
|
backend_id: str
|
||||||
device: str
|
device: str
|
||||||
compatibility_fingerprint: str
|
|
||||||
max_age_seconds: float = DEFAULT_MAX_REPORT_AGE_SECONDS
|
max_age_seconds: float = DEFAULT_MAX_REPORT_AGE_SECONDS
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
@@ -104,9 +94,6 @@ class AdmissionRequirement:
|
|||||||
recipe_version=context.recipe.version,
|
recipe_version=context.recipe.version,
|
||||||
backend_id=context.recipe.backend_id,
|
backend_id=context.recipe.backend_id,
|
||||||
device=context.device,
|
device=context.device,
|
||||||
compatibility_fingerprint=_compatibility_fingerprint_for_context(
|
|
||||||
context
|
|
||||||
),
|
|
||||||
max_age_seconds=max_age_seconds,
|
max_age_seconds=max_age_seconds,
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -178,16 +165,6 @@ def admit(
|
|||||||
f"{requirement.backend_id} on {requirement.device}",
|
f"{requirement.backend_id} on {requirement.device}",
|
||||||
)
|
)
|
||||||
|
|
||||||
if report.compatibility_fingerprint != requirement.compatibility_fingerprint:
|
|
||||||
raise CapabilityAdmissionError(
|
|
||||||
REASON_COMPATIBILITY_MISMATCH,
|
|
||||||
f"capability proof fingerprint {report.compatibility_fingerprint!r} "
|
|
||||||
f"does not match the expected compatibility fingerprint for "
|
|
||||||
f"{requirement.model_id} {requirement.shard_label}; the artifact, "
|
|
||||||
f"tokenizer, architecture, boundary schema, activation recipe or "
|
|
||||||
f"cache layout differs",
|
|
||||||
)
|
|
||||||
|
|
||||||
if not report.passed:
|
if not report.passed:
|
||||||
raise CapabilityAdmissionError(
|
raise CapabilityAdmissionError(
|
||||||
REASON_NOT_PASSED,
|
REASON_NOT_PASSED,
|
||||||
@@ -246,157 +223,3 @@ def probe_capability(context: CapabilityContext) -> CapabilityReport:
|
|||||||
context.recipe,
|
context.recipe,
|
||||||
context.manifest,
|
context.manifest,
|
||||||
).report
|
).report
|
||||||
|
|
||||||
|
|
||||||
def _compatibility_fingerprint_for_context(context: CapabilityContext) -> str:
|
|
||||||
backend = context.backend
|
|
||||||
selection = context.selection
|
|
||||||
recipe = context.recipe
|
|
||||||
model_config = getattr(getattr(backend, "model", None), "config", None)
|
|
||||||
model_config_payload = (
|
|
||||||
model_config.to_dict() if hasattr(model_config, "to_dict") else model_config
|
|
||||||
)
|
|
||||||
runtime_versions = _runtime_versions()
|
|
||||||
runtime_version = _PACKAGE_VERSION
|
|
||||||
ownership = authoritative_dense_llama_ownership(backend, selection)
|
|
||||||
artifact = build_artifact_identity(
|
|
||||||
model_id=selection.model_id,
|
|
||||||
revision=getattr(getattr(backend, "model", None), "revision", None),
|
|
||||||
model_config=model_config_payload,
|
|
||||||
shard_start=ownership.start_layer,
|
|
||||||
shard_end=ownership.end_layer,
|
|
||||||
)
|
|
||||||
runtime_recipe = build_runtime_recipe_identity(
|
|
||||||
model_id=selection.model_id,
|
|
||||||
revision=getattr(getattr(backend, "model", None), "revision", None),
|
|
||||||
model_config=model_config_payload,
|
|
||||||
recipe_params=recipe.params,
|
|
||||||
weight_quantization=selection.quantization,
|
|
||||||
backend_id=recipe.backend_id,
|
|
||||||
runtime_version=runtime_version,
|
|
||||||
activation_dtype="bfloat16",
|
|
||||||
compute_dtype=_backend_compute_dtype(backend),
|
|
||||||
kv_dtype=_backend_kv_dtype(backend),
|
|
||||||
kv_layout=_backend_kv_layout(backend),
|
|
||||||
tokenizer_revision=_backend_tokenizer_revision(backend, selection),
|
|
||||||
architecture_adapter=_backend_architecture_adapter(backend, recipe.backend_id),
|
|
||||||
boundary_schema_version=1,
|
|
||||||
cache_layout=_backend_cache_layout(backend, recipe.params),
|
|
||||||
)
|
|
||||||
return compatibility_fingerprint(
|
|
||||||
fingerprint_payload(
|
|
||||||
model={
|
|
||||||
"model_id": selection.model_id,
|
|
||||||
"revision": getattr(getattr(backend, "model", None), "revision", None),
|
|
||||||
"config_fingerprint": config_fingerprint(model_config_payload),
|
|
||||||
},
|
|
||||||
shard={
|
|
||||||
"start": ownership.start_layer,
|
|
||||||
"end": ownership.end_layer,
|
|
||||||
"owns_embedding": ownership.owns_embedding,
|
|
||||||
"owns_final_head": ownership.owns_final_head,
|
|
||||||
},
|
|
||||||
recipe={
|
|
||||||
"recipe_id": recipe.id,
|
|
||||||
"recipe_version": recipe.version,
|
|
||||||
"catalogue_version": context.manifest.catalogue_version,
|
|
||||||
},
|
|
||||||
backend={
|
|
||||||
"backend_id": recipe.backend_id,
|
|
||||||
"device": context.device,
|
|
||||||
"device_name": _backend_device_name(context.device),
|
|
||||||
"quantization": selection.quantization,
|
|
||||||
"runtime": runtime_versions,
|
|
||||||
},
|
|
||||||
artifact=artifact.to_dict(),
|
|
||||||
runtime_recipe=runtime_recipe.to_dict(),
|
|
||||||
)
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _runtime_versions() -> dict[str, str]:
|
|
||||||
versions: dict[str, str] = {}
|
|
||||||
for name in ("torch", "transformers"):
|
|
||||||
try:
|
|
||||||
module = __import__(name)
|
|
||||||
except Exception:
|
|
||||||
continue
|
|
||||||
version = getattr(module, "__version__", None)
|
|
||||||
if version:
|
|
||||||
versions[name] = str(version)
|
|
||||||
return versions
|
|
||||||
|
|
||||||
|
|
||||||
def _backend_compute_dtype(backend: Any) -> str:
|
|
||||||
config = getattr(getattr(backend, "model", None), "config", None)
|
|
||||||
for candidate in (config, getattr(config, "text_config", None)):
|
|
||||||
if candidate is None:
|
|
||||||
continue
|
|
||||||
for attr in ("dtype", "torch_dtype"):
|
|
||||||
value = getattr(candidate, attr, None)
|
|
||||||
if value is None:
|
|
||||||
continue
|
|
||||||
return str(value).removeprefix("torch.")
|
|
||||||
return "bfloat16"
|
|
||||||
|
|
||||||
|
|
||||||
def _backend_kv_dtype(backend: Any) -> str:
|
|
||||||
return _backend_compute_dtype(backend)
|
|
||||||
|
|
||||||
|
|
||||||
def _backend_kv_layout(backend: Any) -> str:
|
|
||||||
return "session-cache" if getattr(backend, "supports_kv_cache", False) else "stateless"
|
|
||||||
|
|
||||||
|
|
||||||
def _backend_tokenizer_revision(backend: Any, selection: DoctorSelection) -> str:
|
|
||||||
model = getattr(backend, "model", None)
|
|
||||||
revision = getattr(model, "revision", None)
|
|
||||||
if isinstance(revision, str) and revision.strip():
|
|
||||||
return revision
|
|
||||||
tokenizer = getattr(backend, "tokenizer", None)
|
|
||||||
for attr in ("revision", "model_id"):
|
|
||||||
value = getattr(tokenizer, attr, None)
|
|
||||||
if isinstance(value, str) and value.strip():
|
|
||||||
return value
|
|
||||||
return selection.model_id
|
|
||||||
|
|
||||||
|
|
||||||
def _backend_architecture_adapter(backend: Any, default: str) -> str:
|
|
||||||
config = getattr(getattr(backend, "model", None), "config", None)
|
|
||||||
for candidate in (config, getattr(config, "text_config", None)):
|
|
||||||
if candidate is None:
|
|
||||||
continue
|
|
||||||
for attr in ("architecture_adapter", "model_type"):
|
|
||||||
value = getattr(candidate, attr, None)
|
|
||||||
if isinstance(value, str) and value.strip():
|
|
||||||
return value
|
|
||||||
architectures = getattr(candidate, "architectures", None)
|
|
||||||
if isinstance(architectures, (list, tuple)) and architectures:
|
|
||||||
first = architectures[0]
|
|
||||||
if isinstance(first, str) and first.strip():
|
|
||||||
return first
|
|
||||||
return default
|
|
||||||
|
|
||||||
|
|
||||||
def _backend_device_name(device: str) -> str | None:
|
|
||||||
if device != "cuda":
|
|
||||||
return None
|
|
||||||
from .hardware import detect_hardware
|
|
||||||
|
|
||||||
try:
|
|
||||||
return detect_hardware().get("gpu_name") or None
|
|
||||||
except Exception:
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
def _backend_cache_layout(backend: Any, recipe_params: dict[str, Any] | None) -> str:
|
|
||||||
if getattr(backend, "supports_kv_cache", False) is False:
|
|
||||||
return "stateless"
|
|
||||||
if recipe_params is None:
|
|
||||||
return "local-hot-kv"
|
|
||||||
if recipe_params.get("use_cache") is False:
|
|
||||||
return "stateless"
|
|
||||||
value = recipe_params.get("cache_layout")
|
|
||||||
if isinstance(value, str) and value.strip():
|
|
||||||
return value
|
|
||||||
return "local-hot-kv"
|
|
||||||
|
|||||||
@@ -1,484 +0,0 @@
|
|||||||
"""Architecture-defined boundary input/output for distributed Shards (DGR-006).
|
|
||||||
|
|
||||||
A public-network Shard is a contiguous range of transformer layers (RALPH runtime
|
|
||||||
decision #1). For disjoint processes to reproduce whole-model execution, every
|
|
||||||
Shard must agree on *exactly* what boundary state it consumes and emits:
|
|
||||||
|
|
||||||
* The **head** owns token embedding: it accepts token IDs and turns them into the
|
|
||||||
residual stream. No other Shard may embed tokens.
|
|
||||||
* **Middle and tail** Shards bypass token embedding entirely; they accept the named
|
|
||||||
boundary bundle (the residual stream handed over by the previous range).
|
|
||||||
* A **non-tail** Shard emits the *unnormalized* architecture-defined residual /
|
|
||||||
boundary — before the final norm, before the LM head, and before any tail-only
|
|
||||||
row pruning — so the next range sees precisely the state the whole model would
|
|
||||||
have at that layer index.
|
|
||||||
* The **tail** owns the final norm + LM head and turns the residual into logits or
|
|
||||||
a sampled token through an explicit sampling contract.
|
|
||||||
|
|
||||||
This module is deliberately backend-agnostic. It enforces the boundary *contract*
|
|
||||||
and defers the arithmetic to a ``ShardComputation`` (a duck-typed object exposing
|
|
||||||
``embed_tokens`` / ``run_layers`` / ``final_norm`` / ``lm_head``). The pinned
|
|
||||||
llama.cpp worker (DGR-008) and the reference PyTorch backend both satisfy that
|
|
||||||
protocol, and the numpy reference model in the tests proves whole-model versus
|
|
||||||
two-range parity without any download, GPU, or API credit.
|
|
||||||
|
|
||||||
The adapter **fails closed** for uncertified architectures: only architectures
|
|
||||||
that have passed real certification (dense Llama-family first, per RALPH runtime
|
|
||||||
decision #13) are accepted. Everything else raises rather than silently guessing a
|
|
||||||
tensor layout — Qwen3/Qwen3-MoE stays registered-but-dark until DGR-015 certifies
|
|
||||||
its own adapter.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from dataclasses import dataclass, field
|
|
||||||
from enum import Enum
|
|
||||||
from typing import Any
|
|
||||||
|
|
||||||
import numpy as np
|
|
||||||
|
|
||||||
# The boundary bundle wire schema version. This is the ``boundary_schema_version``
|
|
||||||
# carried by ``runtime_recipe.RuntimeRecipeIdentity``; a receiver refuses a bundle
|
|
||||||
# whose schema it does not implement (forward/backward compatibility is a routing
|
|
||||||
# concern, not a silent reinterpretation).
|
|
||||||
BOUNDARY_SCHEMA_VERSION = 1
|
|
||||||
|
|
||||||
|
|
||||||
class BoundaryAdapterError(RuntimeError):
|
|
||||||
"""Base class for boundary-contract violations."""
|
|
||||||
|
|
||||||
|
|
||||||
class UncertifiedArchitectureError(BoundaryAdapterError):
|
|
||||||
"""Raised when a boundary adapter is requested for an uncertified architecture.
|
|
||||||
|
|
||||||
Failing closed here is a safety property: an unknown architecture has an
|
|
||||||
unknown tensor layout, so guessing where the residual boundary lives would
|
|
||||||
silently corrupt distributed output. The architecture must pass real
|
|
||||||
certification first.
|
|
||||||
"""
|
|
||||||
|
|
||||||
|
|
||||||
class BoundaryContractError(BoundaryAdapterError):
|
|
||||||
"""Raised when a Shard is fed the wrong boundary input for its role.
|
|
||||||
|
|
||||||
Examples: a head handed a residual bundle instead of token IDs, a middle
|
|
||||||
Shard handed token IDs it must not embed, or a boundary bundle whose
|
|
||||||
architecture / schema / seam layer does not match the receiving range.
|
|
||||||
"""
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class ArchitectureBoundary:
|
|
||||||
"""The architecture-defined boundary description for one certified adapter.
|
|
||||||
|
|
||||||
These fields are what makes the boundary *architecture-defined* rather than a
|
|
||||||
hardcoded assumption: the residual tensor name, whether the tail normalizes
|
|
||||||
before the LM head, and whether row pruning is a tail-only concern all come
|
|
||||||
from here.
|
|
||||||
"""
|
|
||||||
|
|
||||||
adapter: str
|
|
||||||
boundary_tensor_name: str
|
|
||||||
boundary_schema_version: int
|
|
||||||
normalizes_before_head: bool
|
|
||||||
prunes_rows_at_tail: bool
|
|
||||||
|
|
||||||
|
|
||||||
# Certified architectures only. Dense Llama-family is first (RALPH runtime decision
|
|
||||||
# #13 + native discipline). Aliases map the many spellings a runtime recipe /
|
|
||||||
# GGUF / HF config may use onto the single canonical adapter id. Anything not in
|
|
||||||
# this table fails closed.
|
|
||||||
_DENSE_LLAMA = ArchitectureBoundary(
|
|
||||||
adapter="dense-llama",
|
|
||||||
boundary_tensor_name="residual_stream",
|
|
||||||
boundary_schema_version=BOUNDARY_SCHEMA_VERSION,
|
|
||||||
normalizes_before_head=True,
|
|
||||||
prunes_rows_at_tail=True,
|
|
||||||
)
|
|
||||||
|
|
||||||
_CERTIFIED_ARCHITECTURES: dict[str, ArchitectureBoundary] = {
|
|
||||||
"dense-llama": _DENSE_LLAMA,
|
|
||||||
"dense_llama": _DENSE_LLAMA,
|
|
||||||
"llama": _DENSE_LLAMA,
|
|
||||||
"llamaforcausallm": _DENSE_LLAMA,
|
|
||||||
"llamamodel": _DENSE_LLAMA,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def certified_architecture(name: Any) -> ArchitectureBoundary:
|
|
||||||
"""Return the certified boundary description for ``name`` or fail closed.
|
|
||||||
|
|
||||||
``name`` may be the canonical adapter id (``dense-llama``), an HF architecture
|
|
||||||
class (``LlamaForCausalLM``), or a GGUF/config ``model_type`` (``llama``).
|
|
||||||
Uncertified architectures raise ``UncertifiedArchitectureError``.
|
|
||||||
"""
|
|
||||||
if not isinstance(name, str) or not name.strip():
|
|
||||||
raise UncertifiedArchitectureError(
|
|
||||||
"architecture adapter must be a non-empty string; "
|
|
||||||
"the boundary adapter refuses to guess a tensor layout"
|
|
||||||
)
|
|
||||||
key = name.strip().lower()
|
|
||||||
boundary = _CERTIFIED_ARCHITECTURES.get(key)
|
|
||||||
if boundary is None:
|
|
||||||
raise UncertifiedArchitectureError(
|
|
||||||
f"architecture {name!r} is not certified for the boundary adapter; "
|
|
||||||
f"certified adapters: {sorted(set(v.adapter for v in _CERTIFIED_ARCHITECTURES.values()))}. "
|
|
||||||
"Uncertified architectures stay registered-but-dark until real "
|
|
||||||
"certification passes."
|
|
||||||
)
|
|
||||||
return boundary
|
|
||||||
|
|
||||||
|
|
||||||
def is_certified_architecture(name: Any) -> bool:
|
|
||||||
"""Return True when ``name`` maps to a certified boundary adapter."""
|
|
||||||
try:
|
|
||||||
certified_architecture(name)
|
|
||||||
except UncertifiedArchitectureError:
|
|
||||||
return False
|
|
||||||
return True
|
|
||||||
|
|
||||||
|
|
||||||
class ShardRole(str, Enum):
|
|
||||||
"""Where a contiguous layer range sits in the whole model."""
|
|
||||||
|
|
||||||
HEAD = "head"
|
|
||||||
MIDDLE = "middle"
|
|
||||||
TAIL = "tail"
|
|
||||||
FULL = "full"
|
|
||||||
|
|
||||||
@property
|
|
||||||
def owns_embedding(self) -> bool:
|
|
||||||
return self in (ShardRole.HEAD, ShardRole.FULL)
|
|
||||||
|
|
||||||
@property
|
|
||||||
def owns_final_head(self) -> bool:
|
|
||||||
return self in (ShardRole.TAIL, ShardRole.FULL)
|
|
||||||
|
|
||||||
|
|
||||||
def role_for_range(start_layer: int, end_layer: int, total_layers: int) -> ShardRole:
|
|
||||||
"""Classify a contiguous inclusive layer range within a model of ``total_layers``."""
|
|
||||||
if total_layers <= 0:
|
|
||||||
raise ValueError("total_layers must be positive")
|
|
||||||
if start_layer < 0 or end_layer < start_layer:
|
|
||||||
raise ValueError("require 0 <= start_layer <= end_layer")
|
|
||||||
if end_layer > total_layers - 1:
|
|
||||||
raise ValueError(
|
|
||||||
f"end_layer {end_layer} exceeds last layer index {total_layers - 1}"
|
|
||||||
)
|
|
||||||
is_head = start_layer == 0
|
|
||||||
is_tail = end_layer >= total_layers - 1
|
|
||||||
if is_head and is_tail:
|
|
||||||
return ShardRole.FULL
|
|
||||||
if is_head:
|
|
||||||
return ShardRole.HEAD
|
|
||||||
if is_tail:
|
|
||||||
return ShardRole.TAIL
|
|
||||||
return ShardRole.MIDDLE
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class BoundaryBundle:
|
|
||||||
"""The versioned named-tensor bundle handed between adjacent Shard ranges.
|
|
||||||
|
|
||||||
``residual`` is the *unnormalized* architecture-defined residual stream with
|
|
||||||
every position row intact (no tail-only pruning). ``next_layer`` is the layer
|
|
||||||
index the receiving range must start at — it is the overlap-safe effective
|
|
||||||
start of the seam, so a receiver can reject a bundle meant for a different cut.
|
|
||||||
"""
|
|
||||||
|
|
||||||
architecture_adapter: str
|
|
||||||
schema_version: int
|
|
||||||
tensor_name: str
|
|
||||||
residual: np.ndarray
|
|
||||||
positions: np.ndarray
|
|
||||||
next_layer: int
|
|
||||||
normalized: bool = False
|
|
||||||
|
|
||||||
def named_tensor_fields(self) -> dict[str, Any]:
|
|
||||||
"""Return the wire-shaped description of the residual tensor.
|
|
||||||
|
|
||||||
These are exactly the fields the DGR-002 ``NamedTensor`` carries (name,
|
|
||||||
shape, dtype, byte order, raw bytes), so a worker can serialize this
|
|
||||||
bundle into the gRPC protobuf without re-deriving them.
|
|
||||||
"""
|
|
||||||
residual = np.ascontiguousarray(self.residual)
|
|
||||||
return {
|
|
||||||
"name": self.tensor_name,
|
|
||||||
"shape": list(residual.shape),
|
|
||||||
"dtype": residual.dtype.name,
|
|
||||||
"byte_order": _byte_order(residual.dtype),
|
|
||||||
"data": residual.tobytes(),
|
|
||||||
}
|
|
||||||
|
|
||||||
def pack(self) -> dict[str, Any]:
|
|
||||||
"""Serialize the bundle to a transport-agnostic dict (proves the seam).
|
|
||||||
|
|
||||||
The residual and positions are carried as raw little/big-endian bytes plus
|
|
||||||
shape/dtype so that a truly disjoint process can reconstruct the exact
|
|
||||||
array — this is what lets two OS processes reproduce whole-model math.
|
|
||||||
"""
|
|
||||||
residual = np.ascontiguousarray(self.residual)
|
|
||||||
positions = np.ascontiguousarray(self.positions)
|
|
||||||
return {
|
|
||||||
"architecture_adapter": self.architecture_adapter,
|
|
||||||
"schema_version": self.schema_version,
|
|
||||||
"tensor_name": self.tensor_name,
|
|
||||||
"next_layer": self.next_layer,
|
|
||||||
"normalized": self.normalized,
|
|
||||||
"residual": {
|
|
||||||
"shape": list(residual.shape),
|
|
||||||
"dtype": residual.dtype.str,
|
|
||||||
"data": residual.tobytes(),
|
|
||||||
},
|
|
||||||
"positions": {
|
|
||||||
"shape": list(positions.shape),
|
|
||||||
"dtype": positions.dtype.str,
|
|
||||||
"data": positions.tobytes(),
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
@classmethod
|
|
||||||
def unpack(cls, payload: dict[str, Any]) -> "BoundaryBundle":
|
|
||||||
"""Reconstruct a bundle produced by :meth:`pack`."""
|
|
||||||
residual = _array_from_wire(payload["residual"])
|
|
||||||
positions = _array_from_wire(payload["positions"])
|
|
||||||
return cls(
|
|
||||||
architecture_adapter=payload["architecture_adapter"],
|
|
||||||
schema_version=int(payload["schema_version"]),
|
|
||||||
tensor_name=payload["tensor_name"],
|
|
||||||
residual=residual,
|
|
||||||
positions=positions,
|
|
||||||
next_layer=int(payload["next_layer"]),
|
|
||||||
normalized=bool(payload.get("normalized", False)),
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class SamplingContract:
|
|
||||||
"""Explicit contract for turning tail logits into a token.
|
|
||||||
|
|
||||||
The tail never hides the sampling decision inside the adapter: the contract is
|
|
||||||
a first-class value so the head/route can reproduce it and so greedy decoding
|
|
||||||
is deterministic by construction. Only greedy is certified here; temperature /
|
|
||||||
top-p are declared but must be requested explicitly and are out of scope for
|
|
||||||
the deterministic parity gate.
|
|
||||||
"""
|
|
||||||
|
|
||||||
mode: str = "greedy"
|
|
||||||
temperature: float = 1.0
|
|
||||||
top_p: float = 1.0
|
|
||||||
|
|
||||||
def __post_init__(self) -> None:
|
|
||||||
if self.mode not in ("greedy",):
|
|
||||||
raise BoundaryContractError(
|
|
||||||
f"sampling mode {self.mode!r} is not certified; only 'greedy' is "
|
|
||||||
"deterministic and supported by the boundary adapter today"
|
|
||||||
)
|
|
||||||
|
|
||||||
@classmethod
|
|
||||||
def greedy(cls) -> "SamplingContract":
|
|
||||||
return cls(mode="greedy")
|
|
||||||
|
|
||||||
def sample(self, last_logits: np.ndarray) -> int:
|
|
||||||
"""Return the next token id from the final-position logits row."""
|
|
||||||
logits = np.asarray(last_logits)
|
|
||||||
if logits.ndim == 2:
|
|
||||||
# (batch, vocab) — parity harness uses batch size 1.
|
|
||||||
logits = logits[0]
|
|
||||||
if logits.ndim != 1:
|
|
||||||
raise BoundaryContractError(
|
|
||||||
"sampling expects the pruned final-position logits row"
|
|
||||||
)
|
|
||||||
return int(np.argmax(logits))
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class TailOutput:
|
|
||||||
"""What a tail Shard emits: the sampled token plus the pruned logits row."""
|
|
||||||
|
|
||||||
token_id: int
|
|
||||||
logits: np.ndarray
|
|
||||||
sampling: SamplingContract
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class BoundaryAdapter:
|
|
||||||
"""Enforces the architecture-defined boundary contract for one Shard range.
|
|
||||||
|
|
||||||
Construction fails closed for uncertified architectures. The adapter derives
|
|
||||||
the Shard's role from its range and drives a duck-typed ``ShardComputation``.
|
|
||||||
"""
|
|
||||||
|
|
||||||
computation: Any
|
|
||||||
sampling: SamplingContract = field(default_factory=SamplingContract.greedy)
|
|
||||||
architecture: ArchitectureBoundary = field(init=False)
|
|
||||||
role: ShardRole = field(init=False)
|
|
||||||
start_layer: int = field(init=False)
|
|
||||||
end_layer: int = field(init=False)
|
|
||||||
total_layers: int = field(init=False)
|
|
||||||
|
|
||||||
def __post_init__(self) -> None:
|
|
||||||
arch_name = getattr(self.computation, "architecture_adapter", None)
|
|
||||||
self.architecture = certified_architecture(arch_name)
|
|
||||||
self.start_layer = int(getattr(self.computation, "start_layer"))
|
|
||||||
self.end_layer = int(getattr(self.computation, "end_layer"))
|
|
||||||
self.total_layers = int(getattr(self.computation, "total_layers"))
|
|
||||||
self.role = role_for_range(
|
|
||||||
self.start_layer, self.end_layer, self.total_layers
|
|
||||||
)
|
|
||||||
|
|
||||||
@property
|
|
||||||
def is_head(self) -> bool:
|
|
||||||
return self.role.owns_embedding
|
|
||||||
|
|
||||||
@property
|
|
||||||
def is_tail(self) -> bool:
|
|
||||||
return self.role.owns_final_head
|
|
||||||
|
|
||||||
def forward(
|
|
||||||
self,
|
|
||||||
*,
|
|
||||||
token_ids: Any | None = None,
|
|
||||||
boundary: BoundaryBundle | None = None,
|
|
||||||
) -> BoundaryBundle | TailOutput:
|
|
||||||
"""Run one prefill/decode pass for this range and emit its boundary output.
|
|
||||||
|
|
||||||
Head/full ranges require ``token_ids``; middle/tail ranges require the
|
|
||||||
``boundary`` bundle. Non-tail ranges return a :class:`BoundaryBundle`;
|
|
||||||
tail/full ranges return a :class:`TailOutput` through the sampling
|
|
||||||
contract.
|
|
||||||
"""
|
|
||||||
hidden, positions = self._ingest(token_ids, boundary)
|
|
||||||
hidden = self.computation.run_layers(hidden, positions=positions)
|
|
||||||
if self.is_tail:
|
|
||||||
return self._emit_tail(hidden)
|
|
||||||
return self._emit_boundary(hidden, positions)
|
|
||||||
|
|
||||||
# -- input side -----------------------------------------------------------
|
|
||||||
|
|
||||||
def _ingest(
|
|
||||||
self, token_ids: Any | None, boundary: BoundaryBundle | None
|
|
||||||
) -> tuple[np.ndarray, np.ndarray]:
|
|
||||||
if self.role.owns_embedding:
|
|
||||||
return self._ingest_tokens(token_ids, boundary)
|
|
||||||
return self._ingest_boundary(token_ids, boundary)
|
|
||||||
|
|
||||||
def _ingest_tokens(
|
|
||||||
self, token_ids: Any | None, boundary: BoundaryBundle | None
|
|
||||||
) -> tuple[np.ndarray, np.ndarray]:
|
|
||||||
if token_ids is None:
|
|
||||||
raise BoundaryContractError(
|
|
||||||
"the head owns token embedding and must receive token IDs"
|
|
||||||
)
|
|
||||||
if boundary is not None:
|
|
||||||
raise BoundaryContractError(
|
|
||||||
"the head owns token embedding; it must not receive a boundary "
|
|
||||||
"bundle from an upstream range"
|
|
||||||
)
|
|
||||||
ids = np.asarray(token_ids)
|
|
||||||
if ids.ndim == 1:
|
|
||||||
ids = ids[None, :]
|
|
||||||
if ids.ndim != 2:
|
|
||||||
raise BoundaryContractError("token IDs must be (seq,) or (batch, seq)")
|
|
||||||
hidden = np.asarray(self.computation.embed_tokens(ids))
|
|
||||||
positions = np.broadcast_to(
|
|
||||||
np.arange(ids.shape[1], dtype=np.int64), ids.shape
|
|
||||||
).copy()
|
|
||||||
return hidden, positions
|
|
||||||
|
|
||||||
def _ingest_boundary(
|
|
||||||
self, token_ids: Any | None, boundary: BoundaryBundle | None
|
|
||||||
) -> tuple[np.ndarray, np.ndarray]:
|
|
||||||
if token_ids is not None:
|
|
||||||
raise BoundaryContractError(
|
|
||||||
"middle/tail Shards bypass token embedding; they must not receive "
|
|
||||||
"token IDs"
|
|
||||||
)
|
|
||||||
if boundary is None:
|
|
||||||
raise BoundaryContractError(
|
|
||||||
"middle/tail Shards must receive the named boundary bundle"
|
|
||||||
)
|
|
||||||
self._check_boundary(boundary)
|
|
||||||
return np.asarray(boundary.residual), np.asarray(boundary.positions)
|
|
||||||
|
|
||||||
def _check_boundary(self, boundary: BoundaryBundle) -> None:
|
|
||||||
if certified_architecture(boundary.architecture_adapter) is not self.architecture:
|
|
||||||
raise BoundaryContractError(
|
|
||||||
f"boundary bundle architecture {boundary.architecture_adapter!r} "
|
|
||||||
f"does not match this Shard's adapter {self.architecture.adapter!r}"
|
|
||||||
)
|
|
||||||
if boundary.schema_version != self.architecture.boundary_schema_version:
|
|
||||||
raise BoundaryContractError(
|
|
||||||
f"boundary schema v{boundary.schema_version} is not supported by "
|
|
||||||
f"this Shard (expects v{self.architecture.boundary_schema_version})"
|
|
||||||
)
|
|
||||||
if boundary.tensor_name != self.architecture.boundary_tensor_name:
|
|
||||||
raise BoundaryContractError(
|
|
||||||
f"boundary tensor {boundary.tensor_name!r} is not the "
|
|
||||||
f"architecture-defined {self.architecture.boundary_tensor_name!r}"
|
|
||||||
)
|
|
||||||
if boundary.normalized:
|
|
||||||
raise BoundaryContractError(
|
|
||||||
"boundary bundle is normalized; a Shard range must receive the "
|
|
||||||
"UNNORMALIZED architecture-defined residual"
|
|
||||||
)
|
|
||||||
if boundary.next_layer != self.start_layer:
|
|
||||||
raise BoundaryContractError(
|
|
||||||
f"boundary hands over at layer {boundary.next_layer} but this "
|
|
||||||
f"Shard starts at layer {self.start_layer}"
|
|
||||||
)
|
|
||||||
|
|
||||||
# -- output side ----------------------------------------------------------
|
|
||||||
|
|
||||||
def _emit_boundary(
|
|
||||||
self, hidden: np.ndarray, positions: np.ndarray
|
|
||||||
) -> BoundaryBundle:
|
|
||||||
# A non-tail Shard emits the unnormalized residual with every position row
|
|
||||||
# intact: no final norm, no LM head, no tail-only row pruning. next_layer
|
|
||||||
# is the receiver's overlap-safe effective start.
|
|
||||||
return BoundaryBundle(
|
|
||||||
architecture_adapter=self.architecture.adapter,
|
|
||||||
schema_version=self.architecture.boundary_schema_version,
|
|
||||||
tensor_name=self.architecture.boundary_tensor_name,
|
|
||||||
residual=np.asarray(hidden),
|
|
||||||
positions=np.asarray(positions),
|
|
||||||
next_layer=self.end_layer + 1,
|
|
||||||
normalized=False,
|
|
||||||
)
|
|
||||||
|
|
||||||
def _emit_tail(self, hidden: np.ndarray) -> TailOutput:
|
|
||||||
hidden = np.asarray(hidden)
|
|
||||||
# Tail-only row pruning: only the final position is needed to sample the
|
|
||||||
# next token, so the LM head runs on the pruned row. A non-tail Shard is
|
|
||||||
# forbidden from doing this (it must forward every row).
|
|
||||||
if self.architecture.prunes_rows_at_tail:
|
|
||||||
last_hidden = hidden[:, -1:, :]
|
|
||||||
else: # pragma: no cover - no certified architecture takes this path yet
|
|
||||||
last_hidden = hidden
|
|
||||||
if self.architecture.normalizes_before_head:
|
|
||||||
last_hidden = np.asarray(self.computation.final_norm(last_hidden))
|
|
||||||
logits = np.asarray(self.computation.lm_head(last_hidden))
|
|
||||||
last_logits = logits[:, -1, :]
|
|
||||||
token_id = self.sampling.sample(last_logits)
|
|
||||||
return TailOutput(
|
|
||||||
token_id=token_id, logits=last_logits, sampling=self.sampling
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _byte_order(dtype: np.dtype) -> str:
|
|
||||||
order = dtype.byteorder
|
|
||||||
if order == "<":
|
|
||||||
return "little"
|
|
||||||
if order == ">":
|
|
||||||
return "big"
|
|
||||||
# '=' native, '|' not applicable (single byte)
|
|
||||||
import sys
|
|
||||||
|
|
||||||
return sys.byteorder if order in ("=", "|") else "little"
|
|
||||||
|
|
||||||
|
|
||||||
def _array_from_wire(field_payload: dict[str, Any]) -> np.ndarray:
|
|
||||||
array = np.frombuffer(
|
|
||||||
field_payload["data"], dtype=np.dtype(field_payload["dtype"])
|
|
||||||
)
|
|
||||||
return array.reshape(field_payload["shape"]).copy()
|
|
||||||
@@ -20,16 +20,6 @@ import time
|
|||||||
from dataclasses import dataclass, field
|
from dataclasses import dataclass, field
|
||||||
from typing import Any, Mapping
|
from typing import Any, Mapping
|
||||||
|
|
||||||
from . import __version__ as _PACKAGE_VERSION
|
|
||||||
from .runtime_recipe import (
|
|
||||||
ArtifactIdentity,
|
|
||||||
RuntimeRecipeIdentity,
|
|
||||||
build_artifact_identity,
|
|
||||||
build_runtime_recipe_identity,
|
|
||||||
compatibility_fingerprint,
|
|
||||||
fingerprint_payload,
|
|
||||||
)
|
|
||||||
|
|
||||||
# Layout of the serialized report. Bump when the JSON shape changes.
|
# Layout of the serialized report. Bump when the JSON shape changes.
|
||||||
CAPABILITY_SCHEMA_VERSION = 1
|
CAPABILITY_SCHEMA_VERSION = 1
|
||||||
|
|
||||||
@@ -182,14 +172,6 @@ def _optional_text(value: Any, field_name: str) -> str | None:
|
|||||||
return _require_text(value, field_name)
|
return _require_text(value, field_name)
|
||||||
|
|
||||||
|
|
||||||
def _optional_bool(value: Any, field_name: str) -> bool:
|
|
||||||
if value is None:
|
|
||||||
return False
|
|
||||||
if isinstance(value, bool):
|
|
||||||
return value
|
|
||||||
raise CapabilityReportError(f"{field_name!r} must be a boolean")
|
|
||||||
|
|
||||||
|
|
||||||
def _require_int(value: Any, field_name: str, minimum: int) -> int:
|
def _require_int(value: Any, field_name: str, minimum: int) -> int:
|
||||||
if isinstance(value, bool) or not isinstance(value, int):
|
if isinstance(value, bool) or not isinstance(value, int):
|
||||||
raise CapabilityReportError(f"{field_name!r} must be an integer")
|
raise CapabilityReportError(f"{field_name!r} must be an integer")
|
||||||
@@ -236,8 +218,6 @@ class ShardRange:
|
|||||||
|
|
||||||
start: int
|
start: int
|
||||||
end: int
|
end: int
|
||||||
owns_embedding: bool = False
|
|
||||||
owns_final_head: bool = False
|
|
||||||
|
|
||||||
def __post_init__(self) -> None:
|
def __post_init__(self) -> None:
|
||||||
_require_int(self.start, "shard.start", 0)
|
_require_int(self.start, "shard.start", 0)
|
||||||
@@ -246,18 +226,9 @@ class ShardRange:
|
|||||||
raise CapabilityReportError(
|
raise CapabilityReportError(
|
||||||
f"'shard.end' ({self.end}) must be >= 'shard.start' ({self.start})"
|
f"'shard.end' ({self.end}) must be >= 'shard.start' ({self.start})"
|
||||||
)
|
)
|
||||||
if not isinstance(self.owns_embedding, bool):
|
|
||||||
raise CapabilityReportError("'shard.owns_embedding' must be a boolean")
|
|
||||||
if not isinstance(self.owns_final_head, bool):
|
|
||||||
raise CapabilityReportError("'shard.owns_final_head' must be a boolean")
|
|
||||||
|
|
||||||
def to_dict(self) -> dict:
|
def to_dict(self) -> dict:
|
||||||
return {
|
return {"start": self.start, "end": self.end}
|
||||||
"start": self.start,
|
|
||||||
"end": self.end,
|
|
||||||
"owns_embedding": self.owns_embedding,
|
|
||||||
"owns_final_head": self.owns_final_head,
|
|
||||||
}
|
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def from_dict(cls, data: Any) -> ShardRange:
|
def from_dict(cls, data: Any) -> ShardRange:
|
||||||
@@ -265,12 +236,6 @@ class ShardRange:
|
|||||||
return cls(
|
return cls(
|
||||||
start=_require_int(doc.get("start"), "shard.start", 0),
|
start=_require_int(doc.get("start"), "shard.start", 0),
|
||||||
end=_require_int(doc.get("end"), "shard.end", 0),
|
end=_require_int(doc.get("end"), "shard.end", 0),
|
||||||
owns_embedding=_optional_bool(
|
|
||||||
doc.get("owns_embedding"), "shard.owns_embedding"
|
|
||||||
),
|
|
||||||
owns_final_head=_optional_bool(
|
|
||||||
doc.get("owns_final_head"), "shard.owns_final_head"
|
|
||||||
),
|
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -371,8 +336,6 @@ class CapabilityReport:
|
|||||||
shard: ShardRange
|
shard: ShardRange
|
||||||
recipe: RecipeIdentity
|
recipe: RecipeIdentity
|
||||||
backend: BackendIdentity
|
backend: BackendIdentity
|
||||||
artifact: ArtifactIdentity
|
|
||||||
runtime_recipe: RuntimeRecipeIdentity
|
|
||||||
status: str
|
status: str
|
||||||
validated_at: float
|
validated_at: float
|
||||||
duration_ms: int
|
duration_ms: int
|
||||||
@@ -413,20 +376,6 @@ class CapabilityReport:
|
|||||||
self.backend.device,
|
self.backend.device,
|
||||||
)
|
)
|
||||||
|
|
||||||
@property
|
|
||||||
def compatibility_fingerprint(self) -> str:
|
|
||||||
"""Stable compatibility digest over the full routable identity."""
|
|
||||||
return compatibility_fingerprint(
|
|
||||||
fingerprint_payload(
|
|
||||||
model=self.model.to_dict(),
|
|
||||||
shard=self.shard.to_dict(),
|
|
||||||
recipe=self.recipe.to_dict(),
|
|
||||||
backend=self.backend.to_dict(),
|
|
||||||
artifact=self.artifact.to_dict(),
|
|
||||||
runtime_recipe=self.runtime_recipe.to_dict(),
|
|
||||||
)
|
|
||||||
)
|
|
||||||
|
|
||||||
def age_seconds(self, now: float | None = None) -> float:
|
def age_seconds(self, now: float | None = None) -> float:
|
||||||
return max(0.0, (time.time() if now is None else now) - self.validated_at)
|
return max(0.0, (time.time() if now is None else now) - self.validated_at)
|
||||||
|
|
||||||
@@ -437,9 +386,6 @@ class CapabilityReport:
|
|||||||
"shard": self.shard.to_dict(),
|
"shard": self.shard.to_dict(),
|
||||||
"recipe": self.recipe.to_dict(),
|
"recipe": self.recipe.to_dict(),
|
||||||
"backend": self.backend.to_dict(),
|
"backend": self.backend.to_dict(),
|
||||||
"artifact": self.artifact.to_dict(),
|
|
||||||
"runtime_recipe": self.runtime_recipe.to_dict(),
|
|
||||||
"compatibility_fingerprint": self.compatibility_fingerprint,
|
|
||||||
"status": self.status,
|
"status": self.status,
|
||||||
"validated_at": self.validated_at,
|
"validated_at": self.validated_at,
|
||||||
"duration_ms": self.duration_ms,
|
"duration_ms": self.duration_ms,
|
||||||
@@ -452,9 +398,6 @@ class CapabilityReport:
|
|||||||
@classmethod
|
@classmethod
|
||||||
def from_dict(cls, data: Any) -> CapabilityReport:
|
def from_dict(cls, data: Any) -> CapabilityReport:
|
||||||
doc = _as_mapping(data, "report")
|
doc = _as_mapping(data, "report")
|
||||||
declared_compatibility_fingerprint = _optional_text(
|
|
||||||
doc.get("compatibility_fingerprint"), "compatibility_fingerprint"
|
|
||||||
)
|
|
||||||
|
|
||||||
if "schema_version" not in doc:
|
if "schema_version" not in doc:
|
||||||
raise CapabilityReportError(
|
raise CapabilityReportError(
|
||||||
@@ -474,13 +417,7 @@ class CapabilityReport:
|
|||||||
):
|
):
|
||||||
raise CapabilityReportError("'validated_at' must be a Unix timestamp")
|
raise CapabilityReportError("'validated_at' must be a Unix timestamp")
|
||||||
|
|
||||||
try:
|
return cls(
|
||||||
artifact = ArtifactIdentity.from_dict(doc.get("artifact"))
|
|
||||||
runtime_recipe = RuntimeRecipeIdentity.from_dict(doc.get("runtime_recipe"))
|
|
||||||
except ValueError as exc:
|
|
||||||
raise CapabilityReportError(str(exc)) from exc
|
|
||||||
|
|
||||||
report = cls(
|
|
||||||
schema_version=schema_version,
|
schema_version=schema_version,
|
||||||
model=ModelIdentity.from_dict(doc.get("model")),
|
model=ModelIdentity.from_dict(doc.get("model")),
|
||||||
shard=ShardRange.from_dict(doc.get("shard")),
|
shard=ShardRange.from_dict(doc.get("shard")),
|
||||||
@@ -490,18 +427,7 @@ class CapabilityReport:
|
|||||||
validated_at=float(validated_at),
|
validated_at=float(validated_at),
|
||||||
duration_ms=_require_int(doc.get("duration_ms"), "duration_ms", 0),
|
duration_ms=_require_int(doc.get("duration_ms"), "duration_ms", 0),
|
||||||
diagnostics=sanitize_diagnostics(doc.get("diagnostics")),
|
diagnostics=sanitize_diagnostics(doc.get("diagnostics")),
|
||||||
artifact=artifact,
|
|
||||||
runtime_recipe=runtime_recipe,
|
|
||||||
)
|
)
|
||||||
if (
|
|
||||||
declared_compatibility_fingerprint is not None
|
|
||||||
and report.compatibility_fingerprint != declared_compatibility_fingerprint
|
|
||||||
):
|
|
||||||
raise CapabilityReportError(
|
|
||||||
"report declares a compatibility fingerprint that does not match "
|
|
||||||
"its artifact/runtime recipe"
|
|
||||||
)
|
|
||||||
return report
|
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def from_json(cls, text: str) -> CapabilityReport:
|
def from_json(cls, text: str) -> CapabilityReport:
|
||||||
@@ -532,19 +458,6 @@ def build_capability_report(
|
|||||||
device_name: str | None = None,
|
device_name: str | None = None,
|
||||||
quantization: str | None = None,
|
quantization: str | None = None,
|
||||||
runtime: Mapping[str, str] | None = None,
|
runtime: Mapping[str, str] | None = None,
|
||||||
artifact_hash: str | None = None,
|
|
||||||
runtime_recipe: RuntimeRecipeIdentity | None = None,
|
|
||||||
owns_embedding: bool = False,
|
|
||||||
owns_final_head: bool = False,
|
|
||||||
activation_dtype: Any = None,
|
|
||||||
compute_dtype: Any = None,
|
|
||||||
kv_dtype: Any = None,
|
|
||||||
kv_layout: str | None = None,
|
|
||||||
tokenizer_revision: str | None = None,
|
|
||||||
architecture_adapter: str | None = None,
|
|
||||||
boundary_schema_version: int = 1,
|
|
||||||
cache_layout: str | None = None,
|
|
||||||
recipe_params: Mapping[str, Any] | None = None,
|
|
||||||
diagnostics: Any = None,
|
diagnostics: Any = None,
|
||||||
validated_at: float | None = None,
|
validated_at: float | None = None,
|
||||||
environ: Mapping[str, str] | None = None,
|
environ: Mapping[str, str] | None = None,
|
||||||
@@ -555,62 +468,25 @@ def build_capability_report(
|
|||||||
or an already-computed ``sha256:…`` string. `validated_at` defaults to now,
|
or an already-computed ``sha256:…`` string. `validated_at` defaults to now,
|
||||||
so callers that need determinism pass it explicitly.
|
so callers that need determinism pass it explicitly.
|
||||||
"""
|
"""
|
||||||
model_identity = ModelIdentity(
|
return CapabilityReport(
|
||||||
model_id=model_id,
|
model=ModelIdentity(
|
||||||
revision=revision,
|
|
||||||
config_fingerprint=config_fingerprint(model_config),
|
|
||||||
)
|
|
||||||
shard = ShardRange(
|
|
||||||
start=shard_start,
|
|
||||||
end=shard_end,
|
|
||||||
owns_embedding=owns_embedding,
|
|
||||||
owns_final_head=owns_final_head,
|
|
||||||
)
|
|
||||||
recipe_identity = RecipeIdentity(
|
|
||||||
recipe_id=recipe_id,
|
|
||||||
recipe_version=recipe_version,
|
|
||||||
catalogue_version=catalogue_version,
|
|
||||||
)
|
|
||||||
backend_identity = BackendIdentity(
|
|
||||||
backend_id=backend_id,
|
|
||||||
device=device,
|
|
||||||
device_name=device_name,
|
|
||||||
quantization=quantization,
|
|
||||||
runtime=dict(runtime or {}),
|
|
||||||
)
|
|
||||||
artifact = build_artifact_identity(
|
|
||||||
model_id=model_id,
|
|
||||||
revision=revision,
|
|
||||||
model_config=model_config,
|
|
||||||
artifact_hash=artifact_hash,
|
|
||||||
shard_start=shard_start,
|
|
||||||
shard_end=shard_end,
|
|
||||||
)
|
|
||||||
if runtime_recipe is None:
|
|
||||||
runtime_recipe = build_runtime_recipe_identity(
|
|
||||||
model_id=model_id,
|
model_id=model_id,
|
||||||
revision=revision,
|
revision=revision,
|
||||||
model_config=model_config,
|
config_fingerprint=config_fingerprint(model_config),
|
||||||
recipe_params=recipe_params,
|
),
|
||||||
weight_quantization=quantization or "unknown",
|
shard=ShardRange(start=shard_start, end=shard_end),
|
||||||
|
recipe=RecipeIdentity(
|
||||||
|
recipe_id=recipe_id,
|
||||||
|
recipe_version=recipe_version,
|
||||||
|
catalogue_version=catalogue_version,
|
||||||
|
),
|
||||||
|
backend=BackendIdentity(
|
||||||
backend_id=backend_id,
|
backend_id=backend_id,
|
||||||
runtime_version=_PACKAGE_VERSION,
|
device=device,
|
||||||
activation_dtype=activation_dtype,
|
device_name=device_name,
|
||||||
compute_dtype=compute_dtype,
|
quantization=quantization,
|
||||||
kv_dtype=kv_dtype,
|
runtime=dict(runtime or {}),
|
||||||
kv_layout=kv_layout,
|
),
|
||||||
tokenizer_revision=tokenizer_revision,
|
|
||||||
architecture_adapter=architecture_adapter,
|
|
||||||
boundary_schema_version=boundary_schema_version,
|
|
||||||
cache_layout=cache_layout,
|
|
||||||
)
|
|
||||||
return CapabilityReport(
|
|
||||||
model=model_identity,
|
|
||||||
shard=shard,
|
|
||||||
recipe=recipe_identity,
|
|
||||||
backend=backend_identity,
|
|
||||||
artifact=artifact,
|
|
||||||
runtime_recipe=runtime_recipe,
|
|
||||||
status=status,
|
status=status,
|
||||||
validated_at=time.time() if validated_at is None else validated_at,
|
validated_at=time.time() if validated_at is None else validated_at,
|
||||||
duration_ms=duration_ms,
|
duration_ms=duration_ms,
|
||||||
|
|||||||
@@ -36,8 +36,6 @@ from .capability import (
|
|||||||
CapabilityReport,
|
CapabilityReport,
|
||||||
build_capability_report,
|
build_capability_report,
|
||||||
)
|
)
|
||||||
from . import __version__ as _PACKAGE_VERSION
|
|
||||||
from .runtime_recipe import build_runtime_recipe_identity
|
|
||||||
from .recipe_manifest import (
|
from .recipe_manifest import (
|
||||||
DEFAULT_RECIPE_ID,
|
DEFAULT_RECIPE_ID,
|
||||||
Recipe,
|
Recipe,
|
||||||
@@ -45,7 +43,6 @@ from .recipe_manifest import (
|
|||||||
RecipeManifestError,
|
RecipeManifestError,
|
||||||
load_recipe_manifest,
|
load_recipe_manifest,
|
||||||
)
|
)
|
||||||
from .gguf_ownership import authoritative_dense_llama_ownership
|
|
||||||
|
|
||||||
# The probe is deliberately tiny: enough tokens to drive every layer in the
|
# The probe is deliberately tiny: enough tokens to drive every layer in the
|
||||||
# shard once, small enough that `doctor` costs seconds beyond the model load.
|
# shard once, small enough that `doctor` costs seconds beyond the model load.
|
||||||
@@ -467,28 +464,10 @@ def _validate_recipe(
|
|||||||
duration_ms = int((time.monotonic() - started) * 1000)
|
duration_ms = int((time.monotonic() - started) * 1000)
|
||||||
|
|
||||||
device = _backend_device(backend, selection)
|
device = _backend_device(backend, selection)
|
||||||
ownership = authoritative_dense_llama_ownership(backend, selection)
|
|
||||||
runtime_recipe = build_runtime_recipe_identity(
|
|
||||||
model_id=selection.model_id,
|
|
||||||
revision=getattr(getattr(backend, "model", None), "revision", None),
|
|
||||||
model_config=_model_config(backend),
|
|
||||||
recipe_params=recipe.params,
|
|
||||||
weight_quantization=selection.quantization,
|
|
||||||
backend_id=recipe.backend_id,
|
|
||||||
runtime_version=_PACKAGE_VERSION,
|
|
||||||
activation_dtype="bfloat16",
|
|
||||||
compute_dtype=_backend_compute_dtype(backend),
|
|
||||||
kv_dtype=_backend_kv_dtype(backend),
|
|
||||||
kv_layout=_backend_kv_layout(backend),
|
|
||||||
tokenizer_revision=_backend_tokenizer_revision(backend, selection),
|
|
||||||
architecture_adapter=_backend_architecture_adapter(backend, recipe.backend_id),
|
|
||||||
boundary_schema_version=1,
|
|
||||||
cache_layout=_backend_cache_layout(backend, recipe.params),
|
|
||||||
)
|
|
||||||
report = build_capability_report(
|
report = build_capability_report(
|
||||||
model_id=selection.model_id,
|
model_id=selection.model_id,
|
||||||
shard_start=ownership.start_layer,
|
shard_start=selection.shard_start,
|
||||||
shard_end=ownership.end_layer,
|
shard_end=selection.shard_end,
|
||||||
recipe_id=recipe.id,
|
recipe_id=recipe.id,
|
||||||
recipe_version=recipe.version,
|
recipe_version=recipe.version,
|
||||||
catalogue_version=manifest.catalogue_version,
|
catalogue_version=manifest.catalogue_version,
|
||||||
@@ -498,9 +477,6 @@ def _validate_recipe(
|
|||||||
quantization=selection.quantization,
|
quantization=selection.quantization,
|
||||||
runtime=_runtime_versions(),
|
runtime=_runtime_versions(),
|
||||||
model_config=_model_config(backend),
|
model_config=_model_config(backend),
|
||||||
runtime_recipe=runtime_recipe,
|
|
||||||
owns_embedding=ownership.owns_embedding,
|
|
||||||
owns_final_head=ownership.owns_final_head,
|
|
||||||
status=STATUS_FAILED if category else STATUS_PASSED,
|
status=STATUS_FAILED if category else STATUS_PASSED,
|
||||||
duration_ms=duration_ms,
|
duration_ms=duration_ms,
|
||||||
diagnostics=[d for d in diagnostics if d] or None,
|
diagnostics=[d for d in diagnostics if d] or None,
|
||||||
@@ -592,65 +568,6 @@ def _runtime_versions() -> dict[str, str]:
|
|||||||
return versions
|
return versions
|
||||||
|
|
||||||
|
|
||||||
def _backend_compute_dtype(backend: Any) -> str:
|
|
||||||
config = getattr(getattr(backend, "model", None), "config", None)
|
|
||||||
for candidate in (config, getattr(config, "text_config", None)):
|
|
||||||
if candidate is None:
|
|
||||||
continue
|
|
||||||
for attr in ("dtype", "torch_dtype"):
|
|
||||||
value = getattr(candidate, attr, None)
|
|
||||||
if value is None:
|
|
||||||
continue
|
|
||||||
return str(value).removeprefix("torch.")
|
|
||||||
return "bfloat16"
|
|
||||||
|
|
||||||
|
|
||||||
def _backend_kv_dtype(backend: Any) -> str:
|
|
||||||
return _backend_compute_dtype(backend)
|
|
||||||
|
|
||||||
|
|
||||||
def _backend_kv_layout(backend: Any) -> str:
|
|
||||||
return "session-cache" if getattr(backend, "supports_kv_cache", False) else "stateless"
|
|
||||||
|
|
||||||
|
|
||||||
def _backend_tokenizer_revision(backend: Any, selection: DoctorSelection) -> str:
|
|
||||||
model = getattr(backend, "model", None)
|
|
||||||
revision = getattr(model, "revision", None)
|
|
||||||
if isinstance(revision, str) and revision.strip():
|
|
||||||
return revision
|
|
||||||
return selection.model_id
|
|
||||||
|
|
||||||
|
|
||||||
def _backend_architecture_adapter(backend: Any, default: str) -> str:
|
|
||||||
config = getattr(getattr(backend, "model", None), "config", None)
|
|
||||||
for candidate in (config, getattr(config, "text_config", None)):
|
|
||||||
if candidate is None:
|
|
||||||
continue
|
|
||||||
for attr in ("architecture_adapter", "model_type"):
|
|
||||||
value = getattr(candidate, attr, None)
|
|
||||||
if isinstance(value, str) and value.strip():
|
|
||||||
return value
|
|
||||||
architectures = getattr(candidate, "architectures", None)
|
|
||||||
if isinstance(architectures, (list, tuple)) and architectures:
|
|
||||||
first = architectures[0]
|
|
||||||
if isinstance(first, str) and first.strip():
|
|
||||||
return first
|
|
||||||
return default
|
|
||||||
|
|
||||||
|
|
||||||
def _backend_cache_layout(backend: Any, recipe_params: Mapping[str, Any] | None) -> str:
|
|
||||||
if getattr(backend, "supports_kv_cache", False) is False:
|
|
||||||
return "stateless"
|
|
||||||
if recipe_params is None:
|
|
||||||
return "local-hot-kv"
|
|
||||||
if recipe_params.get("use_cache") is False:
|
|
||||||
return "stateless"
|
|
||||||
value = recipe_params.get("cache_layout")
|
|
||||||
if isinstance(value, str) and value.strip():
|
|
||||||
return value
|
|
||||||
return "local-hot-kv"
|
|
||||||
|
|
||||||
|
|
||||||
# --- output -----------------------------------------------------------------
|
# --- output -----------------------------------------------------------------
|
||||||
|
|
||||||
DEFAULT_REPORT_FILENAME = "capability.json"
|
DEFAULT_REPORT_FILENAME = "capability.json"
|
||||||
|
|||||||
@@ -1,423 +0,0 @@
|
|||||||
"""Native llama.cpp/GGUF backend adapter for Meshnet node startup.
|
|
||||||
|
|
||||||
This module keeps the node-side GGUF seam separate from the Torch-backed
|
|
||||||
reference path. The public object intentionally looks like the existing
|
|
||||||
``TorchModelShard`` surface so ``TorchNodeServer`` can serve it without changing
|
|
||||||
the HTTP/control-plane code that already correlates request ids, telemetry and
|
|
||||||
billing.
|
|
||||||
|
|
||||||
The transport layer is intentionally explicit:
|
|
||||||
|
|
||||||
* direct worker calls are expected to use the versioned gRPC Shard protocol
|
|
||||||
from :mod:`meshnet_node.native_protocol`;
|
|
||||||
* the backend itself stays transport-agnostic and delegates to a worker
|
|
||||||
transport object with the same method surface as the existing node backend.
|
|
||||||
|
|
||||||
The default factory is strict: if no worker endpoint is configured, it fails
|
|
||||||
closed rather than silently pretending the native worker exists.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import os
|
|
||||||
from dataclasses import dataclass, field
|
|
||||||
from types import SimpleNamespace
|
|
||||||
from typing import Any, Protocol, runtime_checkable
|
|
||||||
|
|
||||||
from .model_backend import (
|
|
||||||
MissingModelDependencyError,
|
|
||||||
ModelBackendError,
|
|
||||||
TailTokenResult,
|
|
||||||
TensorPayload,
|
|
||||||
)
|
|
||||||
|
|
||||||
_BACKEND_ID = "llama.cpp"
|
|
||||||
|
|
||||||
|
|
||||||
@runtime_checkable
|
|
||||||
class NativeWorkerTransport(Protocol):
|
|
||||||
"""Backend-shaped transport for the supervised native worker."""
|
|
||||||
|
|
||||||
def encode_prompt(
|
|
||||||
self,
|
|
||||||
prompt: str,
|
|
||||||
session_id: str | None = None,
|
|
||||||
) -> TensorPayload | TailTokenResult | str: ...
|
|
||||||
|
|
||||||
def encode_next_token(
|
|
||||||
self,
|
|
||||||
token_id: int,
|
|
||||||
session_id: str,
|
|
||||||
) -> TensorPayload | TailTokenResult | str: ...
|
|
||||||
|
|
||||||
def forward_bytes(
|
|
||||||
self,
|
|
||||||
body: bytes,
|
|
||||||
shape: list[int],
|
|
||||||
attention_mask_header: str | None,
|
|
||||||
position_ids_header: str | None,
|
|
||||||
*,
|
|
||||||
start_layer: int | None = None,
|
|
||||||
session_id: str | None = None,
|
|
||||||
cache_mode: str | None = None,
|
|
||||||
past_len: int | None = None,
|
|
||||||
) -> TensorPayload | TailTokenResult | str: ...
|
|
||||||
|
|
||||||
def decode_tail_token(self, hidden_states: Any) -> TailTokenResult: ...
|
|
||||||
|
|
||||||
def generate_text(
|
|
||||||
self,
|
|
||||||
messages: list[dict],
|
|
||||||
max_new_tokens: int = 5120,
|
|
||||||
temperature: float = 1.0,
|
|
||||||
top_p: float = 1.0,
|
|
||||||
) -> str: ...
|
|
||||||
|
|
||||||
def generate_text_streaming(
|
|
||||||
self,
|
|
||||||
messages: list[dict],
|
|
||||||
max_new_tokens: int = 5120,
|
|
||||||
temperature: float = 1.0,
|
|
||||||
top_p: float = 1.0,
|
|
||||||
): ...
|
|
||||||
|
|
||||||
def count_prompt_tokens(self, messages: list[dict]) -> int: ...
|
|
||||||
|
|
||||||
def count_text_tokens(self, text: str) -> int: ...
|
|
||||||
|
|
||||||
def eos_token_ids(self) -> list[int]: ...
|
|
||||||
|
|
||||||
def release_session(self, session_id: str) -> None: ...
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class _NativeModelConfig:
|
|
||||||
"""Enough model metadata for admission and capability reporting."""
|
|
||||||
|
|
||||||
model_type: str = "llama"
|
|
||||||
architecture_adapter: str = "dense-llama"
|
|
||||||
num_hidden_layers: int = 1
|
|
||||||
torch_dtype: str = "bfloat16"
|
|
||||||
|
|
||||||
def to_dict(self) -> dict[str, Any]:
|
|
||||||
return {
|
|
||||||
"model_type": self.model_type,
|
|
||||||
"architecture_adapter": self.architecture_adapter,
|
|
||||||
"num_hidden_layers": self.num_hidden_layers,
|
|
||||||
"torch_dtype": self.torch_dtype,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class GgufNodeBackend:
|
|
||||||
"""GGUF shard backend shaped like ``TorchModelShard``.
|
|
||||||
|
|
||||||
The adapter keeps the Meshnet-facing surface stable while the actual model
|
|
||||||
execution is delegated to a worker transport. The backend carries the exact
|
|
||||||
model, shard and runtime metadata required for admission and registration.
|
|
||||||
"""
|
|
||||||
|
|
||||||
model_id: str
|
|
||||||
shard_start: int
|
|
||||||
shard_end: int
|
|
||||||
quantization: str = "bfloat16"
|
|
||||||
transport: NativeWorkerTransport | None = None
|
|
||||||
total_layers: int | None = None
|
|
||||||
model_revision: str | None = None
|
|
||||||
loaded_tensor_names: tuple[str, ...] = ()
|
|
||||||
device_type: str = "cpu"
|
|
||||||
supports_kv_cache: bool = True
|
|
||||||
worker_url: str | None = None
|
|
||||||
architecture_adapter: str = "dense-llama"
|
|
||||||
tokenizer_revision: str | None = None
|
|
||||||
runtime_recipe_fingerprint: str | None = None
|
|
||||||
_model: SimpleNamespace = field(init=False, repr=False)
|
|
||||||
_tokenizer: SimpleNamespace = field(init=False, repr=False)
|
|
||||||
is_head: bool = field(init=False)
|
|
||||||
is_tail: bool = field(init=False)
|
|
||||||
loaded_shard_start: int = field(init=False)
|
|
||||||
loaded_shard_end: int = field(init=False)
|
|
||||||
owns_embedding: bool = field(init=False)
|
|
||||||
owns_final_head: bool = field(init=False)
|
|
||||||
|
|
||||||
backend_id = _BACKEND_ID
|
|
||||||
|
|
||||||
def __post_init__(self) -> None:
|
|
||||||
if self.shard_start < 0 or self.shard_end < self.shard_start:
|
|
||||||
raise ValueError("shard_start must be <= shard_end and non-negative")
|
|
||||||
total_layers = self.total_layers or (self.shard_end + 1)
|
|
||||||
object.__setattr__(
|
|
||||||
self,
|
|
||||||
"total_layers",
|
|
||||||
int(total_layers),
|
|
||||||
)
|
|
||||||
object.__setattr__(
|
|
||||||
self,
|
|
||||||
"_model",
|
|
||||||
SimpleNamespace(
|
|
||||||
revision=self.model_revision or self.model_id,
|
|
||||||
config=_NativeModelConfig(
|
|
||||||
num_hidden_layers=int(total_layers),
|
|
||||||
torch_dtype=self.quantization,
|
|
||||||
),
|
|
||||||
),
|
|
||||||
)
|
|
||||||
object.__setattr__(
|
|
||||||
self,
|
|
||||||
"_tokenizer",
|
|
||||||
SimpleNamespace(
|
|
||||||
model_id=self.model_id,
|
|
||||||
revision=self.tokenizer_revision or self.model_revision or self.model_id,
|
|
||||||
eos_token="",
|
|
||||||
eos_token_id=[],
|
|
||||||
),
|
|
||||||
)
|
|
||||||
object.__setattr__(self, "is_head", self.shard_start == 0)
|
|
||||||
object.__setattr__(self, "is_tail", self.shard_end >= int(total_layers) - 1)
|
|
||||||
object.__setattr__(self, "loaded_shard_start", self.shard_start)
|
|
||||||
object.__setattr__(self, "loaded_shard_end", self.shard_end)
|
|
||||||
object.__setattr__(self, "owns_embedding", self.is_head)
|
|
||||||
object.__setattr__(self, "owns_final_head", self.is_tail)
|
|
||||||
if not self.loaded_tensor_names:
|
|
||||||
object.__setattr__(
|
|
||||||
self,
|
|
||||||
"loaded_tensor_names",
|
|
||||||
self._default_tensor_inventory(),
|
|
||||||
)
|
|
||||||
|
|
||||||
@property
|
|
||||||
def model(self) -> Any:
|
|
||||||
return self._model
|
|
||||||
|
|
||||||
@property
|
|
||||||
def tokenizer(self) -> Any:
|
|
||||||
return self._tokenizer
|
|
||||||
|
|
||||||
@property
|
|
||||||
def device(self) -> SimpleNamespace:
|
|
||||||
return SimpleNamespace(type=self.device_type)
|
|
||||||
|
|
||||||
@property
|
|
||||||
def shard_range(self) -> tuple[int, int]:
|
|
||||||
return self.shard_start, self.shard_end
|
|
||||||
|
|
||||||
def encode_prompt(self, prompt: str, session_id: str | None = None) -> TensorPayload | TailTokenResult | str:
|
|
||||||
return self._transport().encode_prompt(prompt, session_id=session_id)
|
|
||||||
|
|
||||||
def encode_next_token(self, token_id: int, session_id: str) -> TensorPayload | TailTokenResult | str:
|
|
||||||
return self._transport().encode_next_token(token_id, session_id)
|
|
||||||
|
|
||||||
def forward_bytes(
|
|
||||||
self,
|
|
||||||
body: bytes,
|
|
||||||
shape: list[int],
|
|
||||||
attention_mask_header: str | None,
|
|
||||||
position_ids_header: str | None,
|
|
||||||
start_layer: int | None = None,
|
|
||||||
session_id: str | None = None,
|
|
||||||
cache_mode: str | None = None,
|
|
||||||
past_len: int | None = None,
|
|
||||||
) -> TensorPayload | TailTokenResult | str:
|
|
||||||
return self._transport().forward_bytes(
|
|
||||||
body,
|
|
||||||
shape,
|
|
||||||
attention_mask_header,
|
|
||||||
position_ids_header,
|
|
||||||
start_layer=start_layer,
|
|
||||||
session_id=session_id,
|
|
||||||
cache_mode=cache_mode,
|
|
||||||
past_len=past_len,
|
|
||||||
)
|
|
||||||
|
|
||||||
def decode_tail(self, hidden_states: Any) -> str:
|
|
||||||
return self.decode_tail_token(hidden_states).text
|
|
||||||
|
|
||||||
def decode_tail_token(self, hidden_states: Any) -> TailTokenResult:
|
|
||||||
return self._transport().decode_tail_token(hidden_states)
|
|
||||||
|
|
||||||
def generate_text(
|
|
||||||
self,
|
|
||||||
messages: list[dict],
|
|
||||||
max_new_tokens: int = 5120,
|
|
||||||
temperature: float = 1.0,
|
|
||||||
top_p: float = 1.0,
|
|
||||||
) -> str:
|
|
||||||
return self._transport().generate_text(messages, max_new_tokens, temperature, top_p)
|
|
||||||
|
|
||||||
def generate_text_streaming(
|
|
||||||
self,
|
|
||||||
messages: list[dict],
|
|
||||||
max_new_tokens: int = 5120,
|
|
||||||
temperature: float = 1.0,
|
|
||||||
top_p: float = 1.0,
|
|
||||||
):
|
|
||||||
yield from self._transport().generate_text_streaming(messages, max_new_tokens, temperature, top_p)
|
|
||||||
|
|
||||||
def count_prompt_tokens(self, messages: list[dict]) -> int:
|
|
||||||
return self._transport().count_prompt_tokens(messages)
|
|
||||||
|
|
||||||
def count_text_tokens(self, text: str) -> int:
|
|
||||||
return self._transport().count_text_tokens(text)
|
|
||||||
|
|
||||||
def eos_token_ids(self) -> list[int]:
|
|
||||||
return self._transport().eos_token_ids()
|
|
||||||
|
|
||||||
def release_session(self, session_id: str) -> None:
|
|
||||||
self._transport().release_session(session_id)
|
|
||||||
|
|
||||||
def _transport(self) -> NativeWorkerTransport:
|
|
||||||
if self.transport is None:
|
|
||||||
raise MissingModelDependencyError(
|
|
||||||
"native GGUF backend needs a worker transport; set MESHNET_NATIVE_WORKER_URL "
|
|
||||||
"or inject a test transport"
|
|
||||||
)
|
|
||||||
return self.transport
|
|
||||||
|
|
||||||
def _default_tensor_inventory(self) -> tuple[str, ...]:
|
|
||||||
tensor_names = [f"blk.{layer}.weight" for layer in range(self.shard_start, self.shard_end + 1)]
|
|
||||||
if self.is_head:
|
|
||||||
tensor_names.append("token_embd.weight")
|
|
||||||
if self.is_tail:
|
|
||||||
tensor_names.extend(["output_norm.weight", "output.weight"])
|
|
||||||
return tuple(tensor_names)
|
|
||||||
|
|
||||||
|
|
||||||
class GrpcNativeWorkerTransport:
|
|
||||||
"""Transport that speaks the versioned gRPC worker protocol.
|
|
||||||
|
|
||||||
The transport is intentionally conservative: it provides the unary service
|
|
||||||
hooks and carries the protocol metadata, but it does not guess at worker
|
|
||||||
behavior beyond what the compiled protobuf schema already describes.
|
|
||||||
"""
|
|
||||||
|
|
||||||
def __init__(self, worker_url: str, *, timeout: float = 30.0) -> None:
|
|
||||||
self.worker_url = worker_url
|
|
||||||
self.timeout = timeout
|
|
||||||
self._grpc = None
|
|
||||||
self._channel = None
|
|
||||||
self._stub = None
|
|
||||||
|
|
||||||
def _ensure_stub(self) -> Any:
|
|
||||||
if self._stub is not None:
|
|
||||||
return self._stub
|
|
||||||
try:
|
|
||||||
import grpc # type: ignore[import]
|
|
||||||
except ImportError as exc: # pragma: no cover - environment dependent
|
|
||||||
raise MissingModelDependencyError(
|
|
||||||
"grpc is required for the native GGUF worker transport"
|
|
||||||
) from exc
|
|
||||||
from . import native_protocol
|
|
||||||
|
|
||||||
grpc_mod = native_protocol.load_grpc()
|
|
||||||
self._grpc = grpc
|
|
||||||
self._channel = grpc.insecure_channel(self.worker_url)
|
|
||||||
self._stub = grpc_mod.ShardRuntimeStub(self._channel)
|
|
||||||
return self._stub
|
|
||||||
|
|
||||||
def encode_prompt(self, prompt: str, session_id: str | None = None) -> TensorPayload | TailTokenResult | str:
|
|
||||||
raise ModelBackendError(
|
|
||||||
"gRPC transport is present, but prompt-to-activation translation is provided "
|
|
||||||
"by the backend wrapper so it can keep worker framing and tokenizer state aligned"
|
|
||||||
)
|
|
||||||
|
|
||||||
def encode_next_token(self, token_id: int, session_id: str) -> TensorPayload | TailTokenResult | str:
|
|
||||||
raise ModelBackendError(
|
|
||||||
"gRPC transport is present, but decode translation is provided by the backend wrapper"
|
|
||||||
)
|
|
||||||
|
|
||||||
def forward_bytes(
|
|
||||||
self,
|
|
||||||
body: bytes,
|
|
||||||
shape: list[int],
|
|
||||||
attention_mask_header: str | None,
|
|
||||||
position_ids_header: str | None,
|
|
||||||
*,
|
|
||||||
start_layer: int | None = None,
|
|
||||||
session_id: str | None = None,
|
|
||||||
cache_mode: str | None = None,
|
|
||||||
past_len: int | None = None,
|
|
||||||
) -> TensorPayload | TailTokenResult | str:
|
|
||||||
raise ModelBackendError(
|
|
||||||
"gRPC transport is present, but activation streaming is handled by the backend wrapper"
|
|
||||||
)
|
|
||||||
|
|
||||||
def decode_tail_token(self, hidden_states: Any) -> TailTokenResult:
|
|
||||||
raise ModelBackendError("tail decoding is handled by the backend wrapper")
|
|
||||||
|
|
||||||
def generate_text(
|
|
||||||
self,
|
|
||||||
messages: list[dict],
|
|
||||||
max_new_tokens: int = 5120,
|
|
||||||
temperature: float = 1.0,
|
|
||||||
top_p: float = 1.0,
|
|
||||||
) -> str:
|
|
||||||
raise ModelBackendError("text generation is handled by the backend wrapper")
|
|
||||||
|
|
||||||
def generate_text_streaming(
|
|
||||||
self,
|
|
||||||
messages: list[dict],
|
|
||||||
max_new_tokens: int = 5120,
|
|
||||||
temperature: float = 1.0,
|
|
||||||
top_p: float = 1.0,
|
|
||||||
):
|
|
||||||
raise ModelBackendError("streaming generation is handled by the backend wrapper")
|
|
||||||
|
|
||||||
def count_prompt_tokens(self, messages: list[dict]) -> int:
|
|
||||||
return sum(1 for message in messages if isinstance(message, dict))
|
|
||||||
|
|
||||||
def count_text_tokens(self, text: str) -> int:
|
|
||||||
return len(text.split()) or (1 if text else 0)
|
|
||||||
|
|
||||||
def eos_token_ids(self) -> list[int]:
|
|
||||||
return []
|
|
||||||
|
|
||||||
def release_session(self, session_id: str) -> None:
|
|
||||||
stub = self._ensure_stub()
|
|
||||||
from . import native_protocol
|
|
||||||
|
|
||||||
pb2 = native_protocol.load()
|
|
||||||
stub.Release(pb2.ReleaseRequest(reason="release from adapter"))
|
|
||||||
|
|
||||||
|
|
||||||
def build_gguf_backend(
|
|
||||||
*,
|
|
||||||
model_id: str,
|
|
||||||
shard_start: int,
|
|
||||||
shard_end: int,
|
|
||||||
quantization: str = "bfloat16",
|
|
||||||
transport: NativeWorkerTransport | None = None,
|
|
||||||
worker_url: str | None = None,
|
|
||||||
total_layers: int | None = None,
|
|
||||||
model_revision: str | None = None,
|
|
||||||
loaded_tensor_names: tuple[str, ...] = (),
|
|
||||||
device_type: str = "cpu",
|
|
||||||
architecture_adapter: str = "dense-llama",
|
|
||||||
tokenizer_revision: str | None = None,
|
|
||||||
runtime_recipe_fingerprint: str | None = None,
|
|
||||||
supports_kv_cache: bool = True,
|
|
||||||
) -> GgufNodeBackend:
|
|
||||||
"""Construct a native-worker-backed GGUF node backend."""
|
|
||||||
if transport is None:
|
|
||||||
worker_url = worker_url or os.environ.get("MESHNET_NATIVE_WORKER_URL")
|
|
||||||
if not worker_url:
|
|
||||||
raise MissingModelDependencyError(
|
|
||||||
"set MESHNET_NATIVE_WORKER_URL to the local gRPC worker endpoint "
|
|
||||||
"or inject a fake transport in tests"
|
|
||||||
)
|
|
||||||
transport = GrpcNativeWorkerTransport(worker_url)
|
|
||||||
return GgufNodeBackend(
|
|
||||||
model_id=model_id,
|
|
||||||
shard_start=shard_start,
|
|
||||||
shard_end=shard_end,
|
|
||||||
quantization=quantization,
|
|
||||||
transport=transport,
|
|
||||||
total_layers=total_layers,
|
|
||||||
model_revision=model_revision,
|
|
||||||
loaded_tensor_names=loaded_tensor_names,
|
|
||||||
device_type=device_type,
|
|
||||||
supports_kv_cache=supports_kv_cache,
|
|
||||||
worker_url=worker_url,
|
|
||||||
architecture_adapter=architecture_adapter,
|
|
||||||
tokenizer_revision=tokenizer_revision,
|
|
||||||
runtime_recipe_fingerprint=runtime_recipe_fingerprint,
|
|
||||||
)
|
|
||||||
@@ -1,287 +0,0 @@
|
|||||||
"""Dense-Llama GGUF ownership helpers.
|
|
||||||
|
|
||||||
This module keeps two related concerns together:
|
|
||||||
|
|
||||||
* selecting the tensors a dense-Llama GGUF shard is allowed to own; and
|
|
||||||
* inferring the authoritative loaded range / endpoint ownership from the
|
|
||||||
tensors the model actually exposes.
|
|
||||||
|
|
||||||
The first is used by the range-aware loader seam. The second is used by the
|
|
||||||
doctor/admission/reporting path so the tracker sees what the model loaded, not
|
|
||||||
what a CLI flag claimed.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import re
|
|
||||||
from dataclasses import dataclass
|
|
||||||
from typing import Any, Iterable, Mapping
|
|
||||||
|
|
||||||
_BLOCK_RE = re.compile(r"^blk\.(\d+)\.")
|
|
||||||
|
|
||||||
_HEAD_TENSOR_NAMES = {
|
|
||||||
"token_embd.weight",
|
|
||||||
"token_embd.bias",
|
|
||||||
"tok_embeddings.weight",
|
|
||||||
"tok_embeddings.bias",
|
|
||||||
"embed_tokens.weight",
|
|
||||||
"embed_tokens.bias",
|
|
||||||
}
|
|
||||||
|
|
||||||
_TAIL_TENSOR_NAMES = {
|
|
||||||
"output_norm.weight",
|
|
||||||
"output_norm.bias",
|
|
||||||
"output.weight",
|
|
||||||
"output.bias",
|
|
||||||
"lm_head.weight",
|
|
||||||
"lm_head.bias",
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class DenseLlamaShardOwnership:
|
|
||||||
"""Authoritative ownership for one dense-Llama shard."""
|
|
||||||
|
|
||||||
start_layer: int
|
|
||||||
end_layer: int
|
|
||||||
owns_embedding: bool
|
|
||||||
owns_final_head: bool
|
|
||||||
tensor_names: tuple[str, ...] = ()
|
|
||||||
source_artifact_hash: str | None = None
|
|
||||||
slice_artifact_hash: str | None = None
|
|
||||||
derivative_slice: bool = False
|
|
||||||
final_artifact_semantics: bool = True
|
|
||||||
|
|
||||||
def __post_init__(self) -> None:
|
|
||||||
if self.start_layer < 0:
|
|
||||||
raise ValueError("start_layer must be non-negative")
|
|
||||||
if self.end_layer < self.start_layer:
|
|
||||||
raise ValueError("end_layer must be >= start_layer")
|
|
||||||
if self.derivative_slice:
|
|
||||||
if not self.source_artifact_hash or not self.slice_artifact_hash:
|
|
||||||
raise ValueError(
|
|
||||||
"temporary derivative sub-GGUFs must carry source and slice hashes"
|
|
||||||
)
|
|
||||||
if self.final_artifact_semantics:
|
|
||||||
raise ValueError(
|
|
||||||
"temporary derivative sub-GGUFs must not be claimed as final artifacts"
|
|
||||||
)
|
|
||||||
|
|
||||||
@property
|
|
||||||
def range(self) -> tuple[int, int]:
|
|
||||||
return self.start_layer, self.end_layer
|
|
||||||
|
|
||||||
def to_dict(self) -> dict[str, Any]:
|
|
||||||
return {
|
|
||||||
"start_layer": self.start_layer,
|
|
||||||
"end_layer": self.end_layer,
|
|
||||||
"owns_embedding": self.owns_embedding,
|
|
||||||
"owns_final_head": self.owns_final_head,
|
|
||||||
"tensor_names": list(self.tensor_names),
|
|
||||||
"source_artifact_hash": self.source_artifact_hash,
|
|
||||||
"slice_artifact_hash": self.slice_artifact_hash,
|
|
||||||
"derivative_slice": self.derivative_slice,
|
|
||||||
"final_artifact_semantics": self.final_artifact_semantics,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def select_dense_llama_tensor_names(
|
|
||||||
tensor_names: Iterable[str],
|
|
||||||
start_layer: int,
|
|
||||||
end_layer: int,
|
|
||||||
*,
|
|
||||||
total_layers: int | None = None,
|
|
||||||
) -> set[str]:
|
|
||||||
"""Return the dense-Llama GGUF tensor names owned by an inclusive range."""
|
|
||||||
if start_layer < 0:
|
|
||||||
raise ValueError("start_layer must be non-negative")
|
|
||||||
if end_layer < start_layer:
|
|
||||||
raise ValueError("end_layer must be greater than or equal to start_layer")
|
|
||||||
|
|
||||||
selected: set[str] = set()
|
|
||||||
for tensor_name in tensor_names:
|
|
||||||
if _tensor_belongs_to_range(tensor_name, start_layer, end_layer, total_layers):
|
|
||||||
selected.add(tensor_name)
|
|
||||||
return selected
|
|
||||||
|
|
||||||
|
|
||||||
def infer_dense_llama_ownership(
|
|
||||||
tensor_names: Iterable[str],
|
|
||||||
*,
|
|
||||||
total_layers: int | None = None,
|
|
||||||
source_artifact_hash: str | None = None,
|
|
||||||
slice_artifact_hash: str | None = None,
|
|
||||||
derivative_slice: bool = False,
|
|
||||||
final_artifact_semantics: bool = True,
|
|
||||||
) -> DenseLlamaShardOwnership:
|
|
||||||
"""Infer authoritative loaded range and endpoint ownership from tensors."""
|
|
||||||
names = tuple(str(name) for name in tensor_names if isinstance(name, str))
|
|
||||||
if not names:
|
|
||||||
raise ValueError("tensor inventory is empty")
|
|
||||||
|
|
||||||
block_layers = sorted(
|
|
||||||
{
|
|
||||||
layer
|
|
||||||
for name in names
|
|
||||||
if (layer := _layer_index(name)) is not None
|
|
||||||
}
|
|
||||||
)
|
|
||||||
if not block_layers:
|
|
||||||
raise ValueError("tensor inventory does not contain any blk.N.* tensors")
|
|
||||||
|
|
||||||
selected = tuple(sorted(names))
|
|
||||||
return DenseLlamaShardOwnership(
|
|
||||||
start_layer=block_layers[0],
|
|
||||||
end_layer=block_layers[-1],
|
|
||||||
owns_embedding=any(_is_head_tensor(name) for name in names),
|
|
||||||
owns_final_head=any(
|
|
||||||
_is_tail_tensor(name, total_layers=total_layers, loaded_end=block_layers[-1])
|
|
||||||
for name in names
|
|
||||||
),
|
|
||||||
tensor_names=selected,
|
|
||||||
source_artifact_hash=source_artifact_hash,
|
|
||||||
slice_artifact_hash=slice_artifact_hash,
|
|
||||||
derivative_slice=derivative_slice,
|
|
||||||
final_artifact_semantics=final_artifact_semantics,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def authoritative_dense_llama_ownership(
|
|
||||||
backend: Any,
|
|
||||||
selection: Any | None = None,
|
|
||||||
) -> DenseLlamaShardOwnership:
|
|
||||||
"""Return the most authoritative dense-Llama ownership the backend exposes."""
|
|
||||||
tensor_names = _tensor_names_from_backend(backend)
|
|
||||||
if tensor_names:
|
|
||||||
try:
|
|
||||||
return infer_dense_llama_ownership(
|
|
||||||
tensor_names,
|
|
||||||
total_layers=_backend_total_layers(backend, selection),
|
|
||||||
)
|
|
||||||
except ValueError:
|
|
||||||
pass
|
|
||||||
|
|
||||||
start, end = _backend_loaded_bounds(backend, selection)
|
|
||||||
return DenseLlamaShardOwnership(
|
|
||||||
start_layer=start,
|
|
||||||
end_layer=end,
|
|
||||||
owns_embedding=_backend_owns_embedding(backend, start),
|
|
||||||
owns_final_head=_backend_owns_final_head(backend, end),
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _backend_loaded_bounds(backend: Any, selection: Any | None) -> tuple[int, int]:
|
|
||||||
start = getattr(backend, "loaded_shard_start", None)
|
|
||||||
end = getattr(backend, "loaded_shard_end", None)
|
|
||||||
if start is None:
|
|
||||||
start = getattr(backend, "shard_start", None)
|
|
||||||
if end is None:
|
|
||||||
end = getattr(backend, "shard_end", None)
|
|
||||||
if start is None or end is None:
|
|
||||||
if selection is None:
|
|
||||||
raise ValueError("backend does not expose a loaded shard range")
|
|
||||||
start = getattr(selection, "shard_start")
|
|
||||||
end = getattr(selection, "shard_end")
|
|
||||||
return int(start), int(end)
|
|
||||||
|
|
||||||
|
|
||||||
def _backend_owns_embedding(backend: Any, start: int) -> bool:
|
|
||||||
value = getattr(backend, "owns_embedding", None)
|
|
||||||
if value is None:
|
|
||||||
value = getattr(backend, "is_head", start == 0)
|
|
||||||
return bool(value)
|
|
||||||
|
|
||||||
|
|
||||||
def _backend_owns_final_head(backend: Any, end: int) -> bool:
|
|
||||||
value = getattr(backend, "owns_final_head", None)
|
|
||||||
if value is None:
|
|
||||||
value = getattr(backend, "is_tail", False)
|
|
||||||
return bool(value)
|
|
||||||
|
|
||||||
|
|
||||||
def _backend_total_layers(backend: Any, selection: Any | None) -> int | None:
|
|
||||||
value = getattr(backend, "total_layers", None)
|
|
||||||
if isinstance(value, int) and value > 0:
|
|
||||||
return value
|
|
||||||
if selection is None:
|
|
||||||
return None
|
|
||||||
total = getattr(selection, "total_layers", None)
|
|
||||||
if isinstance(total, int) and total > 0:
|
|
||||||
return total
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
def _tensor_names_from_backend(backend: Any) -> tuple[str, ...]:
|
|
||||||
for attr in ("loaded_tensor_names", "tensor_names", "tensor_inventory"):
|
|
||||||
value = getattr(backend, attr, None)
|
|
||||||
names = _normalise_tensor_names(value)
|
|
||||||
if names:
|
|
||||||
return names
|
|
||||||
return ()
|
|
||||||
|
|
||||||
|
|
||||||
def _normalise_tensor_names(value: Any) -> tuple[str, ...]:
|
|
||||||
if value is None:
|
|
||||||
return ()
|
|
||||||
if isinstance(value, Mapping):
|
|
||||||
items = value.keys()
|
|
||||||
else:
|
|
||||||
try:
|
|
||||||
items = list(value)
|
|
||||||
except TypeError:
|
|
||||||
return ()
|
|
||||||
names = [str(item) for item in items if isinstance(item, str) and item.strip()]
|
|
||||||
return tuple(names)
|
|
||||||
|
|
||||||
|
|
||||||
def _tensor_belongs_to_range(
|
|
||||||
tensor_name: str,
|
|
||||||
start_layer: int,
|
|
||||||
end_layer: int,
|
|
||||||
total_layers: int | None,
|
|
||||||
) -> bool:
|
|
||||||
layer = _layer_index(tensor_name)
|
|
||||||
if layer is not None:
|
|
||||||
return start_layer <= layer <= end_layer
|
|
||||||
|
|
||||||
if start_layer == 0 and _is_head_tensor(tensor_name):
|
|
||||||
return True
|
|
||||||
|
|
||||||
if total_layers is not None and end_layer >= total_layers - 1 and _is_tail_tensor(
|
|
||||||
tensor_name, total_layers=total_layers, loaded_end=end_layer
|
|
||||||
):
|
|
||||||
return True
|
|
||||||
|
|
||||||
return False
|
|
||||||
|
|
||||||
|
|
||||||
def _layer_index(tensor_name: str) -> int | None:
|
|
||||||
match = _BLOCK_RE.match(tensor_name)
|
|
||||||
if match is None:
|
|
||||||
return None
|
|
||||||
return int(match.group(1))
|
|
||||||
|
|
||||||
|
|
||||||
def _is_head_tensor(tensor_name: str) -> bool:
|
|
||||||
lowered = tensor_name.lower()
|
|
||||||
return lowered in _HEAD_TENSOR_NAMES or any(
|
|
||||||
lowered.startswith(prefix)
|
|
||||||
for prefix in ("token_embd.", "tok_embeddings.", "embed_tokens.")
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _is_tail_tensor(
|
|
||||||
tensor_name: str,
|
|
||||||
*,
|
|
||||||
total_layers: int | None,
|
|
||||||
loaded_end: int,
|
|
||||||
) -> bool:
|
|
||||||
lowered = tensor_name.lower()
|
|
||||||
if lowered in _TAIL_TENSOR_NAMES:
|
|
||||||
return True
|
|
||||||
if total_layers is not None and loaded_end >= total_layers - 1:
|
|
||||||
return any(
|
|
||||||
lowered.startswith(prefix)
|
|
||||||
for prefix in ("output_norm.", "final_norm.", "norm.")
|
|
||||||
)
|
|
||||||
return False
|
|
||||||
@@ -2,7 +2,6 @@
|
|||||||
|
|
||||||
import json
|
import json
|
||||||
import os
|
import os
|
||||||
import shutil
|
|
||||||
import subprocess
|
import subprocess
|
||||||
import time
|
import time
|
||||||
|
|
||||||
@@ -184,17 +183,6 @@ def with_forced_cpu(hw: dict) -> dict:
|
|||||||
return forced
|
return forced
|
||||||
|
|
||||||
|
|
||||||
def _with_model_drive(profile: dict) -> dict:
|
|
||||||
"""Attach free space for the default model cache drive to tracker diagnostics."""
|
|
||||||
try:
|
|
||||||
cache_root = os.path.expanduser("~/.cache/meshnet/shards")
|
|
||||||
profile["model_drive_free_bytes"] = shutil.disk_usage(os.path.expanduser("~")).free
|
|
||||||
profile["model_drive_path"] = cache_root
|
|
||||||
except OSError:
|
|
||||||
pass
|
|
||||||
return profile
|
|
||||||
|
|
||||||
|
|
||||||
def detect_hardware() -> dict:
|
def detect_hardware() -> dict:
|
||||||
"""Detect GPU model and available VRAM. Returns hardware profile dict."""
|
"""Detect GPU model and available VRAM. Returns hardware profile dict."""
|
||||||
ram_mb = _detect_ram_mb()
|
ram_mb = _detect_ram_mb()
|
||||||
@@ -220,23 +208,23 @@ def detect_hardware() -> dict:
|
|||||||
}
|
}
|
||||||
if torch_gpu is not None and torch_gpu.get("gcn_arch"):
|
if torch_gpu is not None and torch_gpu.get("gcn_arch"):
|
||||||
profile["gcn_arch"] = torch_gpu["gcn_arch"]
|
profile["gcn_arch"] = torch_gpu["gcn_arch"]
|
||||||
return _with_model_drive(profile)
|
return profile
|
||||||
except ImportError:
|
except ImportError:
|
||||||
pass
|
pass
|
||||||
|
|
||||||
torch_inventory = _gpu_inventory_profile(torch_gpu, ram_mb)
|
torch_inventory = _gpu_inventory_profile(torch_gpu, ram_mb)
|
||||||
if torch_inventory is not None:
|
if torch_inventory is not None:
|
||||||
return _with_model_drive(torch_inventory)
|
return torch_inventory
|
||||||
|
|
||||||
nvidia_gpu = _gpu_inventory_profile(_detect_nvidia_smi_gpu_memory(), ram_mb)
|
nvidia_gpu = _gpu_inventory_profile(_detect_nvidia_smi_gpu_memory(), ram_mb)
|
||||||
if nvidia_gpu is not None:
|
if nvidia_gpu is not None:
|
||||||
return _with_model_drive(nvidia_gpu)
|
return nvidia_gpu
|
||||||
|
|
||||||
windows_gpu = _gpu_inventory_profile(_detect_windows_gpu_memory(), ram_mb)
|
windows_gpu = _gpu_inventory_profile(_detect_windows_gpu_memory(), ram_mb)
|
||||||
if windows_gpu is not None:
|
if windows_gpu is not None:
|
||||||
return _with_model_drive(windows_gpu)
|
return windows_gpu
|
||||||
|
|
||||||
return _with_model_drive({
|
return {
|
||||||
"device": "cpu",
|
"device": "cpu",
|
||||||
"gpu_name": None,
|
"gpu_name": None,
|
||||||
"vram_mb": 0,
|
"vram_mb": 0,
|
||||||
@@ -244,7 +232,7 @@ def detect_hardware() -> dict:
|
|||||||
"shared_vram_mb": 0,
|
"shared_vram_mb": 0,
|
||||||
"ram_mb": ram_mb,
|
"ram_mb": ram_mb,
|
||||||
"cuda_available": False,
|
"cuda_available": False,
|
||||||
})
|
}
|
||||||
|
|
||||||
|
|
||||||
def benchmark_throughput_checked(device_str: str = "cpu") -> tuple[float, bool, str | None]:
|
def benchmark_throughput_checked(device_str: str = "cpu") -> tuple[float, bool, str | None]:
|
||||||
|
|||||||
@@ -1,918 +0,0 @@
|
|||||||
"""Isolated concurrent local Hot KV State for distributed Shards (DGR-007).
|
|
||||||
|
|
||||||
Hot KV State stays local to the node serving a Shard (RALPH runtime decision #7).
|
|
||||||
A concurrent server must map each ``(Route Session ID, route epoch)`` to an
|
|
||||||
isolated bounded KV context (decision #8) so that one request can never clear or
|
|
||||||
corrupt another's cache.
|
|
||||||
|
|
||||||
This module owns the *lifecycle and storage* of that state and is deliberately
|
|
||||||
backend-agnostic:
|
|
||||||
|
|
||||||
* :class:`HotKvStateManager` is the single mutation entry point. It maps
|
|
||||||
``(session_id, route_epoch)`` to a :class:`SessionCache`, allocates KV **only
|
|
||||||
for the owned layer range**, and enforces a byte budget, a session cap, and a
|
|
||||||
TTL through LRU/TTL eviction. It rejects stale route epochs and incompatible
|
|
||||||
cache recipes, and returns an **explicit** :class:`CacheMiss` when state the
|
|
||||||
caller expected is gone (evicted, released, desynchronised, or never held) so
|
|
||||||
the head degrades to a from-token-zero re-prefill instead of corrupting output
|
|
||||||
(RALPH decision #14: unverified KV is never migrated silently).
|
|
||||||
* :class:`LayerKvCache` / :class:`SessionCache` are the per-owned-layer K/V
|
|
||||||
containers. They are plain ``numpy`` arrays so the default deterministic test
|
|
||||||
suite needs no torch, GPU, download, or API credit; the pinned llama.cpp worker
|
|
||||||
(DGR-008) maps a llama sequence onto the same container contract.
|
|
||||||
* :class:`KvBoundaryAdapter` wraps a KV-aware ``ShardComputation`` (the DGR-006
|
|
||||||
duck type plus ``run_layers_cached``) so a Shard can run cached prefill/decode
|
|
||||||
through the manager while honouring the architecture-defined boundary contract
|
|
||||||
(head embeds tokens, middle/tail bypass embedding, non-tail emits the
|
|
||||||
unnormalized residual, tail samples).
|
|
||||||
|
|
||||||
The manager owns *all* cache mutation: a computation reads the existing cache and
|
|
||||||
returns the new K/V for the appended positions, and the manager decides whether
|
|
||||||
that append fits the budget. That keeps eviction, accounting, and isolation in one
|
|
||||||
place instead of scattered across backends.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import threading
|
|
||||||
import time
|
|
||||||
from collections import OrderedDict
|
|
||||||
from dataclasses import dataclass, field
|
|
||||||
from enum import Enum
|
|
||||||
from typing import Any, Callable, Mapping
|
|
||||||
|
|
||||||
import numpy as np
|
|
||||||
|
|
||||||
from meshnet_node.boundary_adapter import (
|
|
||||||
BOUNDARY_SCHEMA_VERSION,
|
|
||||||
BoundaryBundle,
|
|
||||||
BoundaryContractError,
|
|
||||||
SamplingContract,
|
|
||||||
ShardRole,
|
|
||||||
TailOutput,
|
|
||||||
certified_architecture,
|
|
||||||
role_for_range,
|
|
||||||
)
|
|
||||||
from meshnet_node.runtime_recipe import compatibility_fingerprint
|
|
||||||
|
|
||||||
|
|
||||||
class HotKvStateError(RuntimeError):
|
|
||||||
"""Base class for Hot KV State errors."""
|
|
||||||
|
|
||||||
|
|
||||||
class StaleRouteEpochError(HotKvStateError):
|
|
||||||
"""Raised when a request references a route epoch older than the current one.
|
|
||||||
|
|
||||||
A newer route epoch means the route was re-planned; the old epoch's KV is
|
|
||||||
unverified against the new plan and must never be silently reused.
|
|
||||||
"""
|
|
||||||
|
|
||||||
|
|
||||||
class IncompatibleCacheRecipeError(HotKvStateError):
|
|
||||||
"""Raised when a request's cache recipe does not match the loaded shard.
|
|
||||||
|
|
||||||
A different quantization / dtype / owned range / architecture produces a KV
|
|
||||||
layout this node cannot reuse without corrupting output.
|
|
||||||
"""
|
|
||||||
|
|
||||||
|
|
||||||
class KvBudgetExceededError(HotKvStateError):
|
|
||||||
"""Raised when a single session cannot fit the configured byte budget.
|
|
||||||
|
|
||||||
Other sessions are evicted first (LRU); this fires only when even one session
|
|
||||||
alone exceeds the budget, which is a misconfiguration rather than pressure.
|
|
||||||
"""
|
|
||||||
|
|
||||||
|
|
||||||
class KvCacheMissError(HotKvStateError):
|
|
||||||
"""Raised by the strict accessor when expected session state is absent.
|
|
||||||
|
|
||||||
Prefer :meth:`HotKvStateManager.resolve`, which returns a structured
|
|
||||||
:class:`CacheMiss` instead of raising, when the caller wants to fall back to a
|
|
||||||
stateless re-prefill.
|
|
||||||
"""
|
|
||||||
|
|
||||||
def __init__(self, miss: "CacheMiss") -> None:
|
|
||||||
super().__init__(str(miss))
|
|
||||||
self.miss = miss
|
|
||||||
|
|
||||||
|
|
||||||
class CacheMissReason(str, Enum):
|
|
||||||
"""Why a lookup produced a cache miss (all benign; retry from token zero)."""
|
|
||||||
|
|
||||||
UNKNOWN_SESSION = "unknown-session"
|
|
||||||
EVICTED_TTL = "evicted-ttl"
|
|
||||||
EVICTED_LRU = "evicted-lru"
|
|
||||||
RELEASED = "released"
|
|
||||||
SUPERSEDED_EPOCH = "superseded-epoch"
|
|
||||||
SEQ_LEN_MISMATCH = "seq-len-mismatch"
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class CacheMiss:
|
|
||||||
"""Explicit cache-miss response the head can act on (re-prefill).
|
|
||||||
|
|
||||||
This is a value, not an exception: the native protocol carries a cache
|
|
||||||
expectation/result, and a miss is a normal, expected outcome under eviction.
|
|
||||||
"""
|
|
||||||
|
|
||||||
session_id: str
|
|
||||||
route_epoch: int
|
|
||||||
reason: CacheMissReason
|
|
||||||
detail: str = ""
|
|
||||||
|
|
||||||
def __str__(self) -> str:
|
|
||||||
suffix = f": {self.detail}" if self.detail else ""
|
|
||||||
return (
|
|
||||||
f"cache miss for session {self.session_id[:8]} epoch "
|
|
||||||
f"{self.route_epoch} ({self.reason.value}){suffix}"
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class KvCacheRecipe:
|
|
||||||
"""The identity of a Shard's KV layout, used to reject incompatible reuse.
|
|
||||||
|
|
||||||
Two recipes are compatible iff their fingerprints match — same certified
|
|
||||||
architecture, KV dtype, head geometry, and owned layer range within the same
|
|
||||||
whole-model layer count.
|
|
||||||
"""
|
|
||||||
|
|
||||||
architecture_adapter: str
|
|
||||||
kv_dtype: str
|
|
||||||
n_kv_heads: int
|
|
||||||
head_dim: int
|
|
||||||
total_layers: int
|
|
||||||
start_layer: int
|
|
||||||
end_layer: int
|
|
||||||
boundary_schema_version: int = BOUNDARY_SCHEMA_VERSION
|
|
||||||
|
|
||||||
def __post_init__(self) -> None:
|
|
||||||
# Fail closed on architecture identity (shared with the boundary adapter).
|
|
||||||
certified_architecture(self.architecture_adapter)
|
|
||||||
if self.n_kv_heads <= 0:
|
|
||||||
raise ValueError("n_kv_heads must be positive")
|
|
||||||
if self.head_dim <= 0:
|
|
||||||
raise ValueError("head_dim must be positive")
|
|
||||||
try:
|
|
||||||
np.dtype(self.kv_dtype)
|
|
||||||
except TypeError as exc: # pragma: no cover - defensive
|
|
||||||
raise ValueError(f"invalid kv_dtype {self.kv_dtype!r}") from exc
|
|
||||||
# role_for_range validates 0 <= start <= end <= total_layers - 1.
|
|
||||||
role_for_range(self.start_layer, self.end_layer, self.total_layers)
|
|
||||||
if self.boundary_schema_version < 1:
|
|
||||||
raise ValueError("boundary_schema_version must be >= 1")
|
|
||||||
|
|
||||||
@property
|
|
||||||
def owned_layers(self) -> tuple[int, ...]:
|
|
||||||
return tuple(range(self.start_layer, self.end_layer + 1))
|
|
||||||
|
|
||||||
@property
|
|
||||||
def role(self) -> ShardRole:
|
|
||||||
return role_for_range(self.start_layer, self.end_layer, self.total_layers)
|
|
||||||
|
|
||||||
def bytes_per_token(self) -> int:
|
|
||||||
"""Bytes of KV one token adds across *owned* layers (keys + values)."""
|
|
||||||
itemsize = np.dtype(self.kv_dtype).itemsize
|
|
||||||
per_layer = 2 * self.n_kv_heads * self.head_dim * itemsize
|
|
||||||
return per_layer * len(self.owned_layers)
|
|
||||||
|
|
||||||
def fingerprint(self) -> str:
|
|
||||||
return compatibility_fingerprint(
|
|
||||||
{
|
|
||||||
"kind": "hot-kv-recipe",
|
|
||||||
# Canonicalize the architecture so 'llama' / 'LlamaForCausalLM'
|
|
||||||
# map to the same fingerprint (they are the same layout).
|
|
||||||
"architecture_adapter": certified_architecture(
|
|
||||||
self.architecture_adapter
|
|
||||||
).adapter,
|
|
||||||
"kv_dtype": np.dtype(self.kv_dtype).name,
|
|
||||||
"n_kv_heads": self.n_kv_heads,
|
|
||||||
"head_dim": self.head_dim,
|
|
||||||
"total_layers": self.total_layers,
|
|
||||||
"start_layer": self.start_layer,
|
|
||||||
"end_layer": self.end_layer,
|
|
||||||
"boundary_schema_version": self.boundary_schema_version,
|
|
||||||
}
|
|
||||||
)
|
|
||||||
|
|
||||||
def is_compatible(self, other: "KvCacheRecipe") -> bool:
|
|
||||||
return self.fingerprint() == other.fingerprint()
|
|
||||||
|
|
||||||
|
|
||||||
class LayerKvCache:
|
|
||||||
"""K/V storage for a single owned layer; sequence axis is 0.
|
|
||||||
|
|
||||||
Keys and values are ``(seq, n_kv_heads, head_dim)``. Backends store the
|
|
||||||
position-encoded (post-RoPE) keys so a decode step only appends the new rows.
|
|
||||||
"""
|
|
||||||
|
|
||||||
__slots__ = ("layer_index", "n_kv_heads", "head_dim", "dtype", "keys", "values")
|
|
||||||
|
|
||||||
def __init__(
|
|
||||||
self, layer_index: int, n_kv_heads: int, head_dim: int, dtype: Any
|
|
||||||
) -> None:
|
|
||||||
self.layer_index = int(layer_index)
|
|
||||||
self.n_kv_heads = int(n_kv_heads)
|
|
||||||
self.head_dim = int(head_dim)
|
|
||||||
self.dtype = np.dtype(dtype)
|
|
||||||
self.keys = np.empty((0, self.n_kv_heads, self.head_dim), dtype=self.dtype)
|
|
||||||
self.values = np.empty((0, self.n_kv_heads, self.head_dim), dtype=self.dtype)
|
|
||||||
|
|
||||||
@property
|
|
||||||
def length(self) -> int:
|
|
||||||
return int(self.keys.shape[0])
|
|
||||||
|
|
||||||
def _validate(self, array: np.ndarray, name: str) -> np.ndarray:
|
|
||||||
arr = np.asarray(array, dtype=self.dtype)
|
|
||||||
if arr.ndim != 3 or arr.shape[1:] != (self.n_kv_heads, self.head_dim):
|
|
||||||
raise ValueError(
|
|
||||||
f"layer {self.layer_index} {name} must be "
|
|
||||||
f"(seq, {self.n_kv_heads}, {self.head_dim}), got {arr.shape}"
|
|
||||||
)
|
|
||||||
return arr
|
|
||||||
|
|
||||||
def append(self, keys: np.ndarray, values: np.ndarray) -> int:
|
|
||||||
k = self._validate(keys, "keys")
|
|
||||||
v = self._validate(values, "values")
|
|
||||||
if k.shape[0] != v.shape[0]:
|
|
||||||
raise ValueError(
|
|
||||||
f"layer {self.layer_index} keys/values disagree on token count "
|
|
||||||
f"({k.shape[0]} vs {v.shape[0]})"
|
|
||||||
)
|
|
||||||
self.keys = np.concatenate([self.keys, k], axis=0)
|
|
||||||
self.values = np.concatenate([self.values, v], axis=0)
|
|
||||||
return self.length
|
|
||||||
|
|
||||||
def truncate(self, length: int) -> None:
|
|
||||||
length = max(0, int(length))
|
|
||||||
self.keys = self.keys[:length]
|
|
||||||
self.values = self.values[:length]
|
|
||||||
|
|
||||||
@property
|
|
||||||
def nbytes(self) -> int:
|
|
||||||
return int(self.keys.nbytes + self.values.nbytes)
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class SessionCache:
|
|
||||||
"""Isolated per-``(session_id, epoch)`` KV context over the owned layers only."""
|
|
||||||
|
|
||||||
session_id: str
|
|
||||||
route_epoch: int
|
|
||||||
recipe: KvCacheRecipe
|
|
||||||
layers: "OrderedDict[int, LayerKvCache]"
|
|
||||||
created_tick: float
|
|
||||||
last_tick: float
|
|
||||||
released: bool = False
|
|
||||||
|
|
||||||
@property
|
|
||||||
def seq_len(self) -> int:
|
|
||||||
if not self.layers:
|
|
||||||
return 0
|
|
||||||
# All owned layers advance in lockstep; report the first owned layer.
|
|
||||||
return next(iter(self.layers.values())).length
|
|
||||||
|
|
||||||
@property
|
|
||||||
def owned_layers(self) -> tuple[int, ...]:
|
|
||||||
return tuple(self.layers.keys())
|
|
||||||
|
|
||||||
def layer(self, index: int) -> LayerKvCache:
|
|
||||||
try:
|
|
||||||
return self.layers[index]
|
|
||||||
except KeyError:
|
|
||||||
raise KeyError(
|
|
||||||
f"layer {index} is not owned by this shard "
|
|
||||||
f"(owned {list(self.layers)})"
|
|
||||||
) from None
|
|
||||||
|
|
||||||
def read_only_layers(self) -> Mapping[int, LayerKvCache]:
|
|
||||||
"""The current per-layer caches a computation reads to attend over."""
|
|
||||||
return dict(self.layers)
|
|
||||||
|
|
||||||
def _append(self, kv_by_layer: Mapping[int, Any]) -> int:
|
|
||||||
provided = set(kv_by_layer)
|
|
||||||
owned = set(self.layers)
|
|
||||||
if provided != owned:
|
|
||||||
raise ValueError(
|
|
||||||
f"append must cover exactly the owned layers {sorted(owned)}, "
|
|
||||||
f"got {sorted(provided)}"
|
|
||||||
)
|
|
||||||
# Pre-validate token counts so a partial append never desynchronises the
|
|
||||||
# owned layers (append is all-or-nothing).
|
|
||||||
new_counts = set()
|
|
||||||
for idx, (keys, _values) in kv_by_layer.items():
|
|
||||||
new_counts.add(int(np.asarray(keys).shape[0]))
|
|
||||||
if len(new_counts) != 1:
|
|
||||||
raise ValueError(
|
|
||||||
f"append token counts disagree across layers: {sorted(new_counts)}"
|
|
||||||
)
|
|
||||||
for idx, (keys, values) in kv_by_layer.items():
|
|
||||||
self.layers[idx].append(keys, values)
|
|
||||||
return self.seq_len
|
|
||||||
|
|
||||||
def _truncate(self, length: int) -> None:
|
|
||||||
for cache in self.layers.values():
|
|
||||||
cache.truncate(length)
|
|
||||||
|
|
||||||
@property
|
|
||||||
def nbytes(self) -> int:
|
|
||||||
return sum(cache.nbytes for cache in self.layers.values())
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class HotKvStateConfig:
|
|
||||||
"""Bounds for the manager: memory budget, session cap, and idle TTL."""
|
|
||||||
|
|
||||||
budget_bytes: int = 64 * 1024 * 1024
|
|
||||||
max_sessions: int = 8
|
|
||||||
ttl_seconds: float = 600.0
|
|
||||||
miss_history: int = 256
|
|
||||||
|
|
||||||
def __post_init__(self) -> None:
|
|
||||||
if self.budget_bytes <= 0:
|
|
||||||
raise ValueError("budget_bytes must be positive")
|
|
||||||
if self.max_sessions < 1:
|
|
||||||
raise ValueError("max_sessions must be >= 1")
|
|
||||||
if self.ttl_seconds < 0:
|
|
||||||
raise ValueError("ttl_seconds must be >= 0")
|
|
||||||
if self.miss_history < 0:
|
|
||||||
raise ValueError("miss_history must be >= 0")
|
|
||||||
|
|
||||||
|
|
||||||
class HotKvStateManager:
|
|
||||||
"""Concurrent, bounded map of ``(session_id, epoch)`` to an isolated KV context."""
|
|
||||||
|
|
||||||
def __init__(
|
|
||||||
self,
|
|
||||||
recipe: KvCacheRecipe,
|
|
||||||
config: HotKvStateConfig | None = None,
|
|
||||||
*,
|
|
||||||
clock: Callable[[], float] | None = None,
|
|
||||||
) -> None:
|
|
||||||
self.recipe = recipe
|
|
||||||
self.config = config or HotKvStateConfig()
|
|
||||||
self._clock = clock or time.monotonic
|
|
||||||
self._sessions: "OrderedDict[tuple[str, int], SessionCache]" = OrderedDict()
|
|
||||||
self._latest_epoch: dict[str, int] = {}
|
|
||||||
self._misses: "OrderedDict[tuple[str, int], CacheMiss]" = OrderedDict()
|
|
||||||
self._lock = threading.RLock()
|
|
||||||
|
|
||||||
# -- introspection --------------------------------------------------------
|
|
||||||
|
|
||||||
@property
|
|
||||||
def total_bytes(self) -> int:
|
|
||||||
with self._lock:
|
|
||||||
return sum(s.nbytes for s in self._sessions.values())
|
|
||||||
|
|
||||||
@property
|
|
||||||
def session_count(self) -> int:
|
|
||||||
with self._lock:
|
|
||||||
self._evict_expired_locked(self._clock())
|
|
||||||
return len(self._sessions)
|
|
||||||
|
|
||||||
def session_keys(self) -> list[tuple[str, int]]:
|
|
||||||
with self._lock:
|
|
||||||
return list(self._sessions.keys())
|
|
||||||
|
|
||||||
# -- lifecycle ------------------------------------------------------------
|
|
||||||
|
|
||||||
def open(
|
|
||||||
self,
|
|
||||||
session_id: str,
|
|
||||||
route_epoch: int,
|
|
||||||
*,
|
|
||||||
recipe: KvCacheRecipe | None = None,
|
|
||||||
) -> SessionCache:
|
|
||||||
"""Create (or replace) a fresh, empty isolated context for the session.
|
|
||||||
|
|
||||||
A higher route epoch supersedes and frees any earlier epoch for the same
|
|
||||||
session id; an older epoch is rejected as stale.
|
|
||||||
"""
|
|
||||||
self._require_text(session_id, "session_id")
|
|
||||||
route_epoch = self._require_epoch(route_epoch)
|
|
||||||
with self._lock:
|
|
||||||
self._check_recipe(recipe)
|
|
||||||
self._validate_epoch_locked(session_id, route_epoch)
|
|
||||||
now = self._clock()
|
|
||||||
self._evict_expired_locked(now)
|
|
||||||
self._supersede_older_epochs_locked(session_id, route_epoch)
|
|
||||||
key = (session_id, route_epoch)
|
|
||||||
# A re-open at the same epoch replaces the prior context entirely.
|
|
||||||
self._sessions.pop(key, None)
|
|
||||||
layers: "OrderedDict[int, LayerKvCache]" = OrderedDict(
|
|
||||||
(
|
|
||||||
idx,
|
|
||||||
LayerKvCache(
|
|
||||||
idx,
|
|
||||||
self.recipe.n_kv_heads,
|
|
||||||
self.recipe.head_dim,
|
|
||||||
self.recipe.kv_dtype,
|
|
||||||
),
|
|
||||||
)
|
|
||||||
for idx in self.recipe.owned_layers
|
|
||||||
)
|
|
||||||
session = SessionCache(
|
|
||||||
session_id=session_id,
|
|
||||||
route_epoch=route_epoch,
|
|
||||||
recipe=self.recipe,
|
|
||||||
layers=layers,
|
|
||||||
created_tick=now,
|
|
||||||
last_tick=now,
|
|
||||||
)
|
|
||||||
self._sessions[key] = session
|
|
||||||
self._latest_epoch[session_id] = route_epoch
|
|
||||||
self._misses.pop(key, None)
|
|
||||||
self._enforce_capacity_locked(protect=key, incoming_bytes=0)
|
|
||||||
return session
|
|
||||||
|
|
||||||
def append(
|
|
||||||
self,
|
|
||||||
session_id: str,
|
|
||||||
route_epoch: int,
|
|
||||||
kv_by_layer: Mapping[int, Any],
|
|
||||||
*,
|
|
||||||
recipe: KvCacheRecipe | None = None,
|
|
||||||
expected_seq_len: int | None = None,
|
|
||||||
) -> SessionCache:
|
|
||||||
"""Append new K/V (prefill or decode) to an existing isolated context.
|
|
||||||
|
|
||||||
The computation supplies exactly the owned layers' new keys/values. The
|
|
||||||
manager evicts other sessions (LRU) to fit the byte budget before growing
|
|
||||||
this one, and raises :class:`KvBudgetExceededError` only if this session
|
|
||||||
alone cannot fit.
|
|
||||||
"""
|
|
||||||
route_epoch = self._require_epoch(route_epoch)
|
|
||||||
with self._lock:
|
|
||||||
self._check_recipe(recipe)
|
|
||||||
self._validate_epoch_locked(session_id, route_epoch)
|
|
||||||
session = self._require_live_locked(session_id, route_epoch)
|
|
||||||
if expected_seq_len is not None and session.seq_len != expected_seq_len:
|
|
||||||
miss = self._drop_and_record_locked(
|
|
||||||
(session_id, route_epoch),
|
|
||||||
CacheMissReason.SEQ_LEN_MISMATCH,
|
|
||||||
detail=f"cache holds {session.seq_len}, caller expected "
|
|
||||||
f"{expected_seq_len}",
|
|
||||||
)
|
|
||||||
raise KvCacheMissError(miss)
|
|
||||||
n_new = self._new_token_count(kv_by_layer)
|
|
||||||
incoming = n_new * self.recipe.bytes_per_token()
|
|
||||||
self._enforce_capacity_locked(
|
|
||||||
protect=(session_id, route_epoch), incoming_bytes=incoming
|
|
||||||
)
|
|
||||||
session._append(kv_by_layer)
|
|
||||||
session.last_tick = self._clock()
|
|
||||||
self._sessions.move_to_end((session_id, route_epoch))
|
|
||||||
return session
|
|
||||||
|
|
||||||
def truncate(
|
|
||||||
self, session_id: str, route_epoch: int, length: int
|
|
||||||
) -> SessionCache:
|
|
||||||
"""Drop cached positions beyond ``length`` (rollback) for one session."""
|
|
||||||
route_epoch = self._require_epoch(route_epoch)
|
|
||||||
with self._lock:
|
|
||||||
self._validate_epoch_locked(session_id, route_epoch)
|
|
||||||
session = self._require_live_locked(session_id, route_epoch)
|
|
||||||
if length < 0:
|
|
||||||
raise ValueError("truncate length must be >= 0")
|
|
||||||
session._truncate(length)
|
|
||||||
session.last_tick = self._clock()
|
|
||||||
self._sessions.move_to_end((session_id, route_epoch))
|
|
||||||
return session
|
|
||||||
|
|
||||||
def release(self, session_id: str, route_epoch: int) -> bool:
|
|
||||||
"""Free one session's context; other sessions are untouched.
|
|
||||||
|
|
||||||
Returns True if a live context was freed. A later lookup for the released
|
|
||||||
key yields an explicit :class:`CacheMiss`.
|
|
||||||
"""
|
|
||||||
route_epoch = self._require_epoch(route_epoch)
|
|
||||||
with self._lock:
|
|
||||||
key = (session_id, route_epoch)
|
|
||||||
existed = key in self._sessions
|
|
||||||
self._drop_and_record_locked(key, CacheMissReason.RELEASED)
|
|
||||||
return existed
|
|
||||||
|
|
||||||
# -- lookup ---------------------------------------------------------------
|
|
||||||
|
|
||||||
def resolve(
|
|
||||||
self,
|
|
||||||
session_id: str,
|
|
||||||
route_epoch: int,
|
|
||||||
*,
|
|
||||||
recipe: KvCacheRecipe | None = None,
|
|
||||||
expected_seq_len: int | None = None,
|
|
||||||
) -> SessionCache | CacheMiss:
|
|
||||||
"""Return the live context or an explicit :class:`CacheMiss`.
|
|
||||||
|
|
||||||
Rejects stale epochs and incompatible recipes (both are protocol
|
|
||||||
violations, not benign misses).
|
|
||||||
"""
|
|
||||||
route_epoch = self._require_epoch(route_epoch)
|
|
||||||
with self._lock:
|
|
||||||
self._check_recipe(recipe)
|
|
||||||
self._validate_epoch_locked(session_id, route_epoch)
|
|
||||||
now = self._clock()
|
|
||||||
self._evict_expired_locked(now)
|
|
||||||
key = (session_id, route_epoch)
|
|
||||||
session = self._sessions.get(key)
|
|
||||||
if session is None:
|
|
||||||
return self._recorded_miss_locked(key)
|
|
||||||
if expected_seq_len is not None and session.seq_len != expected_seq_len:
|
|
||||||
return self._drop_and_record_locked(
|
|
||||||
key,
|
|
||||||
CacheMissReason.SEQ_LEN_MISMATCH,
|
|
||||||
detail=f"cache holds {session.seq_len}, caller expected "
|
|
||||||
f"{expected_seq_len}",
|
|
||||||
)
|
|
||||||
session.last_tick = now
|
|
||||||
self._sessions.move_to_end(key)
|
|
||||||
return session
|
|
||||||
|
|
||||||
def get(
|
|
||||||
self,
|
|
||||||
session_id: str,
|
|
||||||
route_epoch: int,
|
|
||||||
*,
|
|
||||||
recipe: KvCacheRecipe | None = None,
|
|
||||||
expected_seq_len: int | None = None,
|
|
||||||
) -> SessionCache:
|
|
||||||
"""Strict accessor: raises :class:`KvCacheMissError` on a miss."""
|
|
||||||
result = self.resolve(
|
|
||||||
session_id,
|
|
||||||
route_epoch,
|
|
||||||
recipe=recipe,
|
|
||||||
expected_seq_len=expected_seq_len,
|
|
||||||
)
|
|
||||||
if isinstance(result, CacheMiss):
|
|
||||||
raise KvCacheMissError(result)
|
|
||||||
return result
|
|
||||||
|
|
||||||
# -- internals ------------------------------------------------------------
|
|
||||||
|
|
||||||
def _check_recipe(self, recipe: KvCacheRecipe | None) -> None:
|
|
||||||
if recipe is not None and not self.recipe.is_compatible(recipe):
|
|
||||||
raise IncompatibleCacheRecipeError(
|
|
||||||
"request cache recipe does not match this shard's loaded recipe "
|
|
||||||
f"(request {recipe.fingerprint()} vs shard {self.recipe.fingerprint()})"
|
|
||||||
)
|
|
||||||
|
|
||||||
def _validate_epoch_locked(self, session_id: str, route_epoch: int) -> None:
|
|
||||||
latest = self._latest_epoch.get(session_id)
|
|
||||||
if latest is not None and route_epoch < latest:
|
|
||||||
raise StaleRouteEpochError(
|
|
||||||
f"session {session_id[:8]} route epoch {route_epoch} is stale; "
|
|
||||||
f"current epoch is {latest}"
|
|
||||||
)
|
|
||||||
|
|
||||||
def _supersede_older_epochs_locked(
|
|
||||||
self, session_id: str, route_epoch: int
|
|
||||||
) -> None:
|
|
||||||
stale_keys = [
|
|
||||||
key
|
|
||||||
for key in self._sessions
|
|
||||||
if key[0] == session_id and key[1] < route_epoch
|
|
||||||
]
|
|
||||||
for key in stale_keys:
|
|
||||||
self._drop_and_record_locked(key, CacheMissReason.SUPERSEDED_EPOCH)
|
|
||||||
|
|
||||||
def _require_live_locked(
|
|
||||||
self, session_id: str, route_epoch: int
|
|
||||||
) -> SessionCache:
|
|
||||||
now = self._clock()
|
|
||||||
self._evict_expired_locked(now)
|
|
||||||
key = (session_id, route_epoch)
|
|
||||||
session = self._sessions.get(key)
|
|
||||||
if session is None:
|
|
||||||
raise KvCacheMissError(self._recorded_miss_locked(key))
|
|
||||||
return session
|
|
||||||
|
|
||||||
def _new_token_count(self, kv_by_layer: Mapping[int, Any]) -> int:
|
|
||||||
owned = set(self.recipe.owned_layers)
|
|
||||||
if set(kv_by_layer) != owned:
|
|
||||||
raise ValueError(
|
|
||||||
f"append must cover exactly the owned layers {sorted(owned)}, "
|
|
||||||
f"got {sorted(kv_by_layer)}"
|
|
||||||
)
|
|
||||||
counts = {int(np.asarray(k).shape[0]) for k, _ in kv_by_layer.values()}
|
|
||||||
if len(counts) != 1:
|
|
||||||
raise ValueError(
|
|
||||||
f"append token counts disagree across layers: {sorted(counts)}"
|
|
||||||
)
|
|
||||||
return counts.pop()
|
|
||||||
|
|
||||||
def _enforce_capacity_locked(
|
|
||||||
self, *, protect: tuple[str, int], incoming_bytes: int
|
|
||||||
) -> None:
|
|
||||||
# Session cap: evict LRU sessions other than the protected one.
|
|
||||||
while len(self._sessions) > self.config.max_sessions:
|
|
||||||
victim = self._lru_victim_locked(protect)
|
|
||||||
if victim is None:
|
|
||||||
break
|
|
||||||
self._drop_and_record_locked(victim, CacheMissReason.EVICTED_LRU)
|
|
||||||
|
|
||||||
# Byte budget: the protected session's own footprint after the append.
|
|
||||||
protected = self._sessions.get(protect)
|
|
||||||
protected_bytes = (protected.nbytes if protected is not None else 0) + incoming_bytes
|
|
||||||
if protected_bytes > self.config.budget_bytes:
|
|
||||||
raise KvBudgetExceededError(
|
|
||||||
f"session {protect[0][:8]} needs {protected_bytes} bytes which "
|
|
||||||
f"exceeds the KV budget {self.config.budget_bytes}"
|
|
||||||
)
|
|
||||||
# Evict other LRU sessions until the whole store fits with the append.
|
|
||||||
while self._total_bytes_locked() + incoming_bytes > self.config.budget_bytes:
|
|
||||||
victim = self._lru_victim_locked(protect)
|
|
||||||
if victim is None:
|
|
||||||
break
|
|
||||||
self._drop_and_record_locked(victim, CacheMissReason.EVICTED_LRU)
|
|
||||||
|
|
||||||
def _lru_victim_locked(self, protect: tuple[str, int]) -> tuple[str, int] | None:
|
|
||||||
for key in self._sessions: # OrderedDict iterates oldest-first.
|
|
||||||
if key != protect:
|
|
||||||
return key
|
|
||||||
return None
|
|
||||||
|
|
||||||
def _total_bytes_locked(self) -> int:
|
|
||||||
return sum(s.nbytes for s in self._sessions.values())
|
|
||||||
|
|
||||||
def _evict_expired_locked(self, now: float) -> None:
|
|
||||||
ttl = self.config.ttl_seconds
|
|
||||||
if ttl <= 0:
|
|
||||||
return
|
|
||||||
expired = [
|
|
||||||
key
|
|
||||||
for key, session in self._sessions.items()
|
|
||||||
if now - session.last_tick > ttl
|
|
||||||
]
|
|
||||||
for key in expired:
|
|
||||||
self._drop_and_record_locked(key, CacheMissReason.EVICTED_TTL)
|
|
||||||
|
|
||||||
def _drop_and_record_locked(
|
|
||||||
self,
|
|
||||||
key: tuple[str, int],
|
|
||||||
reason: CacheMissReason,
|
|
||||||
*,
|
|
||||||
detail: str = "",
|
|
||||||
) -> CacheMiss:
|
|
||||||
session = self._sessions.pop(key, None)
|
|
||||||
if session is not None:
|
|
||||||
session.released = True
|
|
||||||
miss = CacheMiss(
|
|
||||||
session_id=key[0], route_epoch=key[1], reason=reason, detail=detail
|
|
||||||
)
|
|
||||||
self._record_miss_locked(key, miss)
|
|
||||||
return miss
|
|
||||||
|
|
||||||
def _record_miss_locked(self, key: tuple[str, int], miss: CacheMiss) -> None:
|
|
||||||
if self.config.miss_history <= 0:
|
|
||||||
return
|
|
||||||
self._misses.pop(key, None)
|
|
||||||
self._misses[key] = miss
|
|
||||||
while len(self._misses) > self.config.miss_history:
|
|
||||||
self._misses.popitem(last=False)
|
|
||||||
|
|
||||||
def _recorded_miss_locked(self, key: tuple[str, int]) -> CacheMiss:
|
|
||||||
recorded = self._misses.get(key)
|
|
||||||
if recorded is not None:
|
|
||||||
return recorded
|
|
||||||
return CacheMiss(
|
|
||||||
session_id=key[0],
|
|
||||||
route_epoch=key[1],
|
|
||||||
reason=CacheMissReason.UNKNOWN_SESSION,
|
|
||||||
)
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _require_text(value: Any, name: str) -> str:
|
|
||||||
if not isinstance(value, str) or not value.strip():
|
|
||||||
raise ValueError(f"{name} must be a non-empty string")
|
|
||||||
return value
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _require_epoch(value: Any) -> int:
|
|
||||||
if isinstance(value, bool) or not isinstance(value, int):
|
|
||||||
raise ValueError("route_epoch must be an integer")
|
|
||||||
if value < 0:
|
|
||||||
raise ValueError("route_epoch must be >= 0")
|
|
||||||
return value
|
|
||||||
|
|
||||||
|
|
||||||
def kv_recipe_for(computation: Any) -> KvCacheRecipe:
|
|
||||||
"""Build a :class:`KvCacheRecipe` from a KV-aware ``ShardComputation``.
|
|
||||||
|
|
||||||
The computation exposes the DGR-006 duck type plus KV geometry
|
|
||||||
(``n_kv_heads``, ``head_dim``, ``kv_dtype``).
|
|
||||||
"""
|
|
||||||
return KvCacheRecipe(
|
|
||||||
architecture_adapter=str(getattr(computation, "architecture_adapter")),
|
|
||||||
kv_dtype=str(getattr(computation, "kv_dtype", "float32")),
|
|
||||||
n_kv_heads=int(getattr(computation, "n_kv_heads")),
|
|
||||||
head_dim=int(getattr(computation, "head_dim")),
|
|
||||||
total_layers=int(getattr(computation, "total_layers")),
|
|
||||||
start_layer=int(getattr(computation, "start_layer")),
|
|
||||||
end_layer=int(getattr(computation, "end_layer")),
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class KvBoundaryAdapter:
|
|
||||||
"""KV-aware boundary driver: cached prefill/decode through the manager.
|
|
||||||
|
|
||||||
Mirrors the DGR-006 :class:`~meshnet_node.boundary_adapter.BoundaryAdapter`
|
|
||||||
contract (head embeds tokens, middle/tail bypass embedding and consume the
|
|
||||||
unnormalized residual bundle, non-tail emits the unnormalized residual, tail
|
|
||||||
normalizes + heads + prunes + samples) but threads a per-session KV context.
|
|
||||||
|
|
||||||
The wrapped computation must additionally expose::
|
|
||||||
|
|
||||||
run_layers_cached(hidden, *, positions, past_kv)
|
|
||||||
-> (hidden_out, {layer_index: (new_keys, new_values)})
|
|
||||||
|
|
||||||
reading ``past_kv`` (the current per-owned-layer caches) and returning the new
|
|
||||||
position-encoded K/V for the appended positions only. The manager, not the
|
|
||||||
computation, commits those K/V so eviction and budget stay centralized.
|
|
||||||
"""
|
|
||||||
|
|
||||||
computation: Any
|
|
||||||
manager: HotKvStateManager
|
|
||||||
sampling: SamplingContract = field(default_factory=SamplingContract.greedy)
|
|
||||||
architecture: Any = field(init=False)
|
|
||||||
role: ShardRole = field(init=False)
|
|
||||||
start_layer: int = field(init=False)
|
|
||||||
end_layer: int = field(init=False)
|
|
||||||
total_layers: int = field(init=False)
|
|
||||||
recipe: KvCacheRecipe = field(init=False)
|
|
||||||
|
|
||||||
def __post_init__(self) -> None:
|
|
||||||
arch_name = getattr(self.computation, "architecture_adapter", None)
|
|
||||||
self.architecture = certified_architecture(arch_name)
|
|
||||||
self.start_layer = int(getattr(self.computation, "start_layer"))
|
|
||||||
self.end_layer = int(getattr(self.computation, "end_layer"))
|
|
||||||
self.total_layers = int(getattr(self.computation, "total_layers"))
|
|
||||||
self.role = role_for_range(self.start_layer, self.end_layer, self.total_layers)
|
|
||||||
self.recipe = kv_recipe_for(self.computation)
|
|
||||||
if not self.manager.recipe.is_compatible(self.recipe):
|
|
||||||
raise IncompatibleCacheRecipeError(
|
|
||||||
"manager recipe does not match this computation's KV recipe"
|
|
||||||
)
|
|
||||||
|
|
||||||
@property
|
|
||||||
def is_head(self) -> bool:
|
|
||||||
return self.role.owns_embedding
|
|
||||||
|
|
||||||
@property
|
|
||||||
def is_tail(self) -> bool:
|
|
||||||
return self.role.owns_final_head
|
|
||||||
|
|
||||||
def prefill(
|
|
||||||
self,
|
|
||||||
session_id: str,
|
|
||||||
route_epoch: int,
|
|
||||||
*,
|
|
||||||
token_ids: Any | None = None,
|
|
||||||
boundary: BoundaryBundle | None = None,
|
|
||||||
) -> BoundaryBundle | TailOutput:
|
|
||||||
"""Open a fresh isolated context and run the prompt through this range."""
|
|
||||||
session = self.manager.open(session_id, route_epoch, recipe=self.recipe)
|
|
||||||
return self._run_step(session, token_ids, boundary)
|
|
||||||
|
|
||||||
def decode(
|
|
||||||
self,
|
|
||||||
session_id: str,
|
|
||||||
route_epoch: int,
|
|
||||||
*,
|
|
||||||
token_ids: Any | None = None,
|
|
||||||
boundary: BoundaryBundle | None = None,
|
|
||||||
expected_seq_len: int | None = None,
|
|
||||||
) -> BoundaryBundle | TailOutput | CacheMiss:
|
|
||||||
"""Append one (or more) decode positions to an existing context.
|
|
||||||
|
|
||||||
Returns an explicit :class:`CacheMiss` if the context is gone so the head
|
|
||||||
can re-prefill from token zero instead of corrupting output.
|
|
||||||
"""
|
|
||||||
resolved = self.manager.resolve(
|
|
||||||
session_id,
|
|
||||||
route_epoch,
|
|
||||||
recipe=self.recipe,
|
|
||||||
expected_seq_len=expected_seq_len,
|
|
||||||
)
|
|
||||||
if isinstance(resolved, CacheMiss):
|
|
||||||
return resolved
|
|
||||||
return self._run_step(resolved, token_ids, boundary)
|
|
||||||
|
|
||||||
# -- internals ------------------------------------------------------------
|
|
||||||
|
|
||||||
def _run_step(
|
|
||||||
self,
|
|
||||||
session: SessionCache,
|
|
||||||
token_ids: Any | None,
|
|
||||||
boundary: BoundaryBundle | None,
|
|
||||||
) -> BoundaryBundle | TailOutput:
|
|
||||||
prev_len = session.seq_len
|
|
||||||
hidden, positions = self._ingest(prev_len, token_ids, boundary)
|
|
||||||
hidden_out, new_kv = self.computation.run_layers_cached(
|
|
||||||
hidden, positions=positions, past_kv=session.read_only_layers()
|
|
||||||
)
|
|
||||||
self.manager.append(
|
|
||||||
session.session_id,
|
|
||||||
session.route_epoch,
|
|
||||||
new_kv,
|
|
||||||
recipe=self.recipe,
|
|
||||||
expected_seq_len=prev_len,
|
|
||||||
)
|
|
||||||
if self.is_tail:
|
|
||||||
return self._emit_tail(hidden_out)
|
|
||||||
return self._emit_boundary(hidden_out, positions)
|
|
||||||
|
|
||||||
def _ingest(
|
|
||||||
self,
|
|
||||||
prev_len: int,
|
|
||||||
token_ids: Any | None,
|
|
||||||
boundary: BoundaryBundle | None,
|
|
||||||
) -> tuple[np.ndarray, np.ndarray]:
|
|
||||||
if self.role.owns_embedding:
|
|
||||||
if token_ids is None:
|
|
||||||
raise BoundaryContractError(
|
|
||||||
"the head owns token embedding and must receive token IDs"
|
|
||||||
)
|
|
||||||
if boundary is not None:
|
|
||||||
raise BoundaryContractError(
|
|
||||||
"the head owns token embedding; it must not receive a boundary "
|
|
||||||
"bundle from an upstream range"
|
|
||||||
)
|
|
||||||
ids = np.asarray(token_ids)
|
|
||||||
if ids.ndim == 1:
|
|
||||||
ids = ids[None, :]
|
|
||||||
if ids.ndim != 2:
|
|
||||||
raise BoundaryContractError("token IDs must be (seq,) or (batch, seq)")
|
|
||||||
hidden = np.asarray(self.computation.embed_tokens(ids))
|
|
||||||
n_new = ids.shape[1]
|
|
||||||
positions = np.broadcast_to(
|
|
||||||
np.arange(prev_len, prev_len + n_new, dtype=np.int64),
|
|
||||||
ids.shape,
|
|
||||||
).copy()
|
|
||||||
return hidden, positions
|
|
||||||
# Middle / tail: consume the boundary bundle (the unnormalized residual).
|
|
||||||
if token_ids is not None:
|
|
||||||
raise BoundaryContractError(
|
|
||||||
"middle/tail Shards bypass token embedding; they must not receive "
|
|
||||||
"token IDs"
|
|
||||||
)
|
|
||||||
if boundary is None:
|
|
||||||
raise BoundaryContractError(
|
|
||||||
"middle/tail Shards must receive the named boundary bundle"
|
|
||||||
)
|
|
||||||
self._check_boundary(boundary)
|
|
||||||
return np.asarray(boundary.residual), np.asarray(boundary.positions)
|
|
||||||
|
|
||||||
def _check_boundary(self, boundary: BoundaryBundle) -> None:
|
|
||||||
if certified_architecture(boundary.architecture_adapter) is not self.architecture:
|
|
||||||
raise BoundaryContractError(
|
|
||||||
f"boundary bundle architecture {boundary.architecture_adapter!r} "
|
|
||||||
f"does not match this Shard's adapter {self.architecture.adapter!r}"
|
|
||||||
)
|
|
||||||
if boundary.schema_version != self.architecture.boundary_schema_version:
|
|
||||||
raise BoundaryContractError(
|
|
||||||
f"boundary schema v{boundary.schema_version} is not supported by "
|
|
||||||
f"this Shard (expects v{self.architecture.boundary_schema_version})"
|
|
||||||
)
|
|
||||||
if boundary.tensor_name != self.architecture.boundary_tensor_name:
|
|
||||||
raise BoundaryContractError(
|
|
||||||
f"boundary tensor {boundary.tensor_name!r} is not the "
|
|
||||||
f"architecture-defined {self.architecture.boundary_tensor_name!r}"
|
|
||||||
)
|
|
||||||
if boundary.normalized:
|
|
||||||
raise BoundaryContractError(
|
|
||||||
"boundary bundle is normalized; a Shard range must receive the "
|
|
||||||
"UNNORMALIZED architecture-defined residual"
|
|
||||||
)
|
|
||||||
if boundary.next_layer != self.start_layer:
|
|
||||||
raise BoundaryContractError(
|
|
||||||
f"boundary hands over at layer {boundary.next_layer} but this "
|
|
||||||
f"Shard starts at layer {self.start_layer}"
|
|
||||||
)
|
|
||||||
|
|
||||||
def _emit_boundary(
|
|
||||||
self, hidden: np.ndarray, positions: np.ndarray
|
|
||||||
) -> BoundaryBundle:
|
|
||||||
return BoundaryBundle(
|
|
||||||
architecture_adapter=self.architecture.adapter,
|
|
||||||
schema_version=self.architecture.boundary_schema_version,
|
|
||||||
tensor_name=self.architecture.boundary_tensor_name,
|
|
||||||
residual=np.asarray(hidden),
|
|
||||||
positions=np.asarray(positions),
|
|
||||||
next_layer=self.end_layer + 1,
|
|
||||||
normalized=False,
|
|
||||||
)
|
|
||||||
|
|
||||||
def _emit_tail(self, hidden: np.ndarray) -> TailOutput:
|
|
||||||
hidden = np.asarray(hidden)
|
|
||||||
if self.architecture.prunes_rows_at_tail:
|
|
||||||
last_hidden = hidden[:, -1:, :]
|
|
||||||
else: # pragma: no cover - no certified architecture takes this path yet
|
|
||||||
last_hidden = hidden
|
|
||||||
if self.architecture.normalizes_before_head:
|
|
||||||
last_hidden = np.asarray(self.computation.final_norm(last_hidden))
|
|
||||||
logits = np.asarray(self.computation.lm_head(last_hidden))
|
|
||||||
last_logits = logits[:, -1, :]
|
|
||||||
token_id = self.sampling.sample(last_logits)
|
|
||||||
return TailOutput(token_id=token_id, logits=last_logits, sampling=self.sampling)
|
|
||||||
@@ -323,10 +323,6 @@ class TorchModelShard:
|
|||||||
)
|
)
|
||||||
self.is_head = shard_start == 0
|
self.is_head = shard_start == 0
|
||||||
self.is_tail = shard_end >= self.total_layers - 1
|
self.is_tail = shard_end >= self.total_layers - 1
|
||||||
self.loaded_shard_start = shard_start
|
|
||||||
self.loaded_shard_end = shard_end
|
|
||||||
self.owns_embedding = self.is_head
|
|
||||||
self.owns_final_head = self.is_tail
|
|
||||||
self.hidden_size = int(
|
self.hidden_size = int(
|
||||||
getattr(self.model.config, "hidden_size", 0)
|
getattr(self.model.config, "hidden_size", 0)
|
||||||
or getattr(self.model.config, "n_embd", 0)
|
or getattr(self.model.config, "n_embd", 0)
|
||||||
@@ -348,17 +344,6 @@ class TorchModelShard:
|
|||||||
ttl_seconds=float(os.environ.get("MESHNET_KV_TTL_SECONDS", "600")),
|
ttl_seconds=float(os.environ.get("MESHNET_KV_TTL_SECONDS", "600")),
|
||||||
)
|
)
|
||||||
|
|
||||||
@property
|
|
||||||
def loaded_range(self) -> tuple[int, int]:
|
|
||||||
return self.loaded_shard_start, self.loaded_shard_end
|
|
||||||
|
|
||||||
@property
|
|
||||||
def endpoint_ownership(self) -> dict[str, bool]:
|
|
||||||
return {
|
|
||||||
"owns_embedding": self.owns_embedding,
|
|
||||||
"owns_final_head": self.owns_final_head,
|
|
||||||
}
|
|
||||||
|
|
||||||
def encode_prompt(self, prompt: str, session_id: str | None = None) -> TensorPayload:
|
def encode_prompt(self, prompt: str, session_id: str | None = None) -> TensorPayload:
|
||||||
if not self.is_head or self._embed_tokens is None:
|
if not self.is_head or self._embed_tokens is None:
|
||||||
raise ModelBackendError("text prompts can only be accepted by the head shard")
|
raise ModelBackendError("text prompts can only be accepted by the head shard")
|
||||||
|
|||||||
@@ -1,300 +0,0 @@
|
|||||||
"""Loader and helpers for the versioned gRPC Shard protocol (ADR-0024, DGR-002).
|
|
||||||
|
|
||||||
The ``.proto`` schema at ``packages/node/native/proto/shard_runtime.proto`` is the
|
|
||||||
single source of truth. Rather than commit generated stubs (which pin a protobuf
|
|
||||||
runtime version and drift from the schema), this package generates the Python
|
|
||||||
stubs on demand into a gitignored build directory and imports them. Generation is
|
|
||||||
reproducible: it shells out to the pinned ``grpc_tools.protoc`` with the exact
|
|
||||||
same flags as ``packages/node/native/scripts/generate_python.py``.
|
|
||||||
|
|
||||||
Typical use::
|
|
||||||
|
|
||||||
from meshnet_node import native_protocol as proto
|
|
||||||
pb2 = proto.load()
|
|
||||||
header = pb2.MessageHeader(work_id="w1", route_session_id="s1")
|
|
||||||
|
|
||||||
The checksum/fragment helpers encode the bounded-fragment tensor-bundle semantics
|
|
||||||
so callers (and DGR-008/DGR-009) do not re-derive them.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import hashlib
|
|
||||||
import importlib
|
|
||||||
import importlib.util
|
|
||||||
import pathlib
|
|
||||||
import sys
|
|
||||||
import threading
|
|
||||||
import types
|
|
||||||
import zlib
|
|
||||||
|
|
||||||
# The wire schema version this build targets. Keep in sync with the
|
|
||||||
# ``SCHEMA_VERSION_1`` enum member in the .proto.
|
|
||||||
SCHEMA_VERSION = 1
|
|
||||||
|
|
||||||
_NATIVE_ROOT = pathlib.Path(__file__).resolve().parents[2] / "native"
|
|
||||||
PROTO_DIR = _NATIVE_ROOT / "proto"
|
|
||||||
PROTO_FILE = PROTO_DIR / "shard_runtime.proto"
|
|
||||||
# ``build/`` is globally gitignored, so generated stubs never enter version control.
|
|
||||||
GEN_DIR = _NATIVE_ROOT / "build" / "python"
|
|
||||||
|
|
||||||
_PB2_MODULE = "shard_runtime_pb2"
|
|
||||||
_GRPC_MODULE = "shard_runtime_pb2_grpc"
|
|
||||||
|
|
||||||
# Reentrant: load_grpc() holds the lock and calls load(), which re-acquires it.
|
|
||||||
_lock = threading.RLock()
|
|
||||||
_cached_pb2: types.ModuleType | None = None
|
|
||||||
_cached_grpc: types.ModuleType | None = None
|
|
||||||
|
|
||||||
|
|
||||||
class ProtocGenerationError(RuntimeError):
|
|
||||||
"""Raised when the protobuf stubs cannot be generated from the schema."""
|
|
||||||
|
|
||||||
|
|
||||||
def _needs_regen(target: pathlib.Path) -> bool:
|
|
||||||
if not target.exists():
|
|
||||||
return True
|
|
||||||
try:
|
|
||||||
return PROTO_FILE.stat().st_mtime > target.stat().st_mtime
|
|
||||||
except OSError:
|
|
||||||
return True
|
|
||||||
|
|
||||||
|
|
||||||
def generate(*, force: bool = False) -> pathlib.Path:
|
|
||||||
"""Generate ``shard_runtime_pb2{,_grpc}.py`` into :data:`GEN_DIR`.
|
|
||||||
|
|
||||||
Returns the output directory. Reproducible and idempotent: regenerates only
|
|
||||||
when the schema is newer than the stubs (or ``force`` is set). Requires the
|
|
||||||
pinned ``grpc_tools`` (available in the project ``.venv``).
|
|
||||||
"""
|
|
||||||
if not PROTO_FILE.exists():
|
|
||||||
raise ProtocGenerationError(f"schema not found: {PROTO_FILE}")
|
|
||||||
|
|
||||||
pb2_path = GEN_DIR / f"{_PB2_MODULE}.py"
|
|
||||||
if not force and not _needs_regen(pb2_path):
|
|
||||||
return GEN_DIR
|
|
||||||
|
|
||||||
try:
|
|
||||||
from grpc_tools import protoc
|
|
||||||
except ImportError as exc: # pragma: no cover - environment-dependent
|
|
||||||
raise ProtocGenerationError(
|
|
||||||
"grpc_tools is required to generate the Shard protocol stubs; "
|
|
||||||
"install grpcio-tools (present in the project .venv)."
|
|
||||||
) from exc
|
|
||||||
|
|
||||||
GEN_DIR.mkdir(parents=True, exist_ok=True)
|
|
||||||
well_known = _well_known_include()
|
|
||||||
args = [
|
|
||||||
"grpc_tools.protoc",
|
|
||||||
f"-I{PROTO_DIR}",
|
|
||||||
*([f"-I{well_known}"] if well_known else []),
|
|
||||||
f"--python_out={GEN_DIR}",
|
|
||||||
f"--grpc_python_out={GEN_DIR}",
|
|
||||||
str(PROTO_FILE.name),
|
|
||||||
]
|
|
||||||
# protoc resolves the proto by name relative to -I, so run with PROTO_DIR
|
|
||||||
# semantics by passing the bare filename plus the include path above.
|
|
||||||
rc = protoc.main([a for a in args])
|
|
||||||
if rc != 0:
|
|
||||||
raise ProtocGenerationError(
|
|
||||||
f"grpc_tools.protoc exited with status {rc} for {PROTO_FILE}"
|
|
||||||
)
|
|
||||||
if not pb2_path.exists(): # pragma: no cover - defensive
|
|
||||||
raise ProtocGenerationError(f"protoc did not produce {pb2_path}")
|
|
||||||
return GEN_DIR
|
|
||||||
|
|
||||||
|
|
||||||
def _well_known_include() -> str | None:
|
|
||||||
"""Bundled well-known .proto include dir shipped with grpc_tools, if any."""
|
|
||||||
try:
|
|
||||||
import grpc_tools
|
|
||||||
|
|
||||||
candidate = pathlib.Path(grpc_tools.__file__).parent / "_proto"
|
|
||||||
return str(candidate) if candidate.is_dir() else None
|
|
||||||
except Exception: # pragma: no cover - defensive
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
def _import_generated(module_name: str) -> types.ModuleType:
|
|
||||||
gen_dir = str(GEN_DIR)
|
|
||||||
if gen_dir not in sys.path:
|
|
||||||
sys.path.insert(0, gen_dir)
|
|
||||||
if module_name in sys.modules:
|
|
||||||
return sys.modules[module_name]
|
|
||||||
return importlib.import_module(module_name)
|
|
||||||
|
|
||||||
|
|
||||||
def load(*, force: bool = False) -> types.ModuleType:
|
|
||||||
"""Return the generated ``shard_runtime_pb2`` module (messages only).
|
|
||||||
|
|
||||||
Generates the stubs on first use. Thread-safe and cached. Does not import
|
|
||||||
grpc; message serialization/round-trip needs only this module.
|
|
||||||
"""
|
|
||||||
global _cached_pb2
|
|
||||||
with _lock:
|
|
||||||
if _cached_pb2 is not None and not force:
|
|
||||||
return _cached_pb2
|
|
||||||
generate(force=force)
|
|
||||||
_cached_pb2 = _import_generated(_PB2_MODULE)
|
|
||||||
return _cached_pb2
|
|
||||||
|
|
||||||
|
|
||||||
def load_grpc(*, force: bool = False) -> types.ModuleType:
|
|
||||||
"""Return the generated ``shard_runtime_pb2_grpc`` module (service stubs).
|
|
||||||
|
|
||||||
Requires the ``grpc`` runtime. Use for building the C++/Python worker; the
|
|
||||||
round-trip/compat tests only need :func:`load`.
|
|
||||||
"""
|
|
||||||
global _cached_grpc
|
|
||||||
with _lock:
|
|
||||||
if _cached_grpc is not None and not force:
|
|
||||||
return _cached_grpc
|
|
||||||
generate(force=force)
|
|
||||||
load() # ensure the _pb2 module the grpc stub imports is present
|
|
||||||
_cached_grpc = _import_generated(_GRPC_MODULE)
|
|
||||||
return _cached_grpc
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Checksum + bounded-fragment helpers (shared bundle semantics)
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
# Algorithm-name strings mirror the ChecksumAlgorithm enum members without
|
|
||||||
# importing the generated module (so this table is usable before load()).
|
|
||||||
_CHECKSUM_CRC32C = "CHECKSUM_CRC32C"
|
|
||||||
_CHECKSUM_CRC32 = "CHECKSUM_CRC32"
|
|
||||||
_CHECKSUM_SHA256 = "CHECKSUM_SHA256"
|
|
||||||
_CHECKSUM_NONE = "CHECKSUM_NONE"
|
|
||||||
|
|
||||||
|
|
||||||
def _crc32c(data: bytes) -> int:
|
|
||||||
"""Castagnoli CRC32C (software table). Deterministic, no external deps."""
|
|
||||||
crc = 0xFFFFFFFF
|
|
||||||
for byte in data:
|
|
||||||
crc ^= byte
|
|
||||||
for _ in range(8):
|
|
||||||
crc = (crc >> 1) ^ (0x82F63B78 & -(crc & 1))
|
|
||||||
return crc ^ 0xFFFFFFFF
|
|
||||||
|
|
||||||
|
|
||||||
def compute_checksum(algorithm: int, data: bytes):
|
|
||||||
"""Build a ``Checksum`` message for ``data`` under the given enum value.
|
|
||||||
|
|
||||||
``algorithm`` is a ``ChecksumAlgorithm`` enum int from the generated module.
|
|
||||||
Uses only the standard library (crc32c software table, zlib.crc32, hashlib).
|
|
||||||
"""
|
|
||||||
pb2 = load()
|
|
||||||
name = pb2.ChecksumAlgorithm.Name(algorithm)
|
|
||||||
if name == _CHECKSUM_SHA256:
|
|
||||||
value = hashlib.sha256(data).digest()
|
|
||||||
elif name == _CHECKSUM_CRC32C:
|
|
||||||
value = _crc32c(data).to_bytes(4, "big")
|
|
||||||
elif name == _CHECKSUM_CRC32:
|
|
||||||
value = (zlib.crc32(data) & 0xFFFFFFFF).to_bytes(4, "big")
|
|
||||||
elif name == _CHECKSUM_NONE:
|
|
||||||
value = b""
|
|
||||||
else:
|
|
||||||
raise ValueError(f"unsupported checksum algorithm: {name}")
|
|
||||||
return pb2.Checksum(algorithm=algorithm, value=value)
|
|
||||||
|
|
||||||
|
|
||||||
def verify_checksum(checksum, data: bytes) -> bool:
|
|
||||||
"""True if ``checksum`` matches ``data`` (CHECKSUM_NONE always verifies)."""
|
|
||||||
pb2 = load()
|
|
||||||
if checksum.algorithm in (0, pb2.CHECKSUM_NONE):
|
|
||||||
return True
|
|
||||||
return compute_checksum(checksum.algorithm, data).value == checksum.value
|
|
||||||
|
|
||||||
|
|
||||||
def fragment_tensor(
|
|
||||||
*,
|
|
||||||
name: str,
|
|
||||||
shape,
|
|
||||||
dtype: int,
|
|
||||||
payload: bytes,
|
|
||||||
byte_order: int | None = None,
|
|
||||||
max_fragment_bytes: int = 1 << 20,
|
|
||||||
compression: int | None = None,
|
|
||||||
checksum_algorithm: int | None = None,
|
|
||||||
):
|
|
||||||
"""Build a :class:`NamedTensor` splitting ``payload`` into bounded fragments.
|
|
||||||
|
|
||||||
Fragments are ordered by ``byte_offset`` and each carries an optional
|
|
||||||
per-fragment checksum. ``payload`` is treated as already compressed if
|
|
||||||
``compression`` is set; this helper does not compress (that is the seam's
|
|
||||||
policy in ``activation_compression``), it only frames.
|
|
||||||
"""
|
|
||||||
if max_fragment_bytes <= 0:
|
|
||||||
raise ValueError("max_fragment_bytes must be positive")
|
|
||||||
pb2 = load()
|
|
||||||
if byte_order is None:
|
|
||||||
byte_order = pb2.BYTE_ORDER_LITTLE_ENDIAN
|
|
||||||
if compression is None:
|
|
||||||
compression = pb2.COMPRESSION_NONE
|
|
||||||
|
|
||||||
chunks = [
|
|
||||||
payload[i : i + max_fragment_bytes]
|
|
||||||
for i in range(0, len(payload), max_fragment_bytes)
|
|
||||||
] or [b""]
|
|
||||||
fragments = []
|
|
||||||
offset = 0
|
|
||||||
for index, chunk in enumerate(chunks):
|
|
||||||
frag = pb2.TensorFragment(
|
|
||||||
fragment_index=index,
|
|
||||||
fragment_count=len(chunks),
|
|
||||||
byte_offset=offset,
|
|
||||||
data=chunk,
|
|
||||||
)
|
|
||||||
if checksum_algorithm is not None:
|
|
||||||
frag.checksum.CopyFrom(compute_checksum(checksum_algorithm, chunk))
|
|
||||||
fragments.append(frag)
|
|
||||||
offset += len(chunk)
|
|
||||||
return pb2.NamedTensor(
|
|
||||||
name=name,
|
|
||||||
shape=list(shape),
|
|
||||||
dtype=dtype,
|
|
||||||
byte_order=byte_order,
|
|
||||||
total_byte_length=len(payload),
|
|
||||||
compression=compression,
|
|
||||||
fragments=fragments,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def reassemble_tensor(named_tensor) -> bytes:
|
|
||||||
"""Concatenate a :class:`NamedTensor`'s fragments back into the full payload.
|
|
||||||
|
|
||||||
Validates fragment ordering, total length, and any per-fragment checksums.
|
|
||||||
"""
|
|
||||||
fragments = sorted(named_tensor.fragments, key=lambda f: f.byte_offset)
|
|
||||||
out = bytearray()
|
|
||||||
for frag in fragments:
|
|
||||||
if frag.byte_offset != len(out):
|
|
||||||
raise ValueError(
|
|
||||||
f"non-contiguous fragment at offset {frag.byte_offset} "
|
|
||||||
f"(expected {len(out)})"
|
|
||||||
)
|
|
||||||
if frag.HasField("checksum") and not verify_checksum(frag.checksum, frag.data):
|
|
||||||
raise ValueError(f"fragment {frag.fragment_index} checksum mismatch")
|
|
||||||
out.extend(frag.data)
|
|
||||||
if named_tensor.total_byte_length and len(out) != named_tensor.total_byte_length:
|
|
||||||
raise ValueError(
|
|
||||||
f"reassembled length {len(out)} != declared "
|
|
||||||
f"{named_tensor.total_byte_length}"
|
|
||||||
)
|
|
||||||
return bytes(out)
|
|
||||||
|
|
||||||
|
|
||||||
__all__ = [
|
|
||||||
"SCHEMA_VERSION",
|
|
||||||
"PROTO_FILE",
|
|
||||||
"PROTO_DIR",
|
|
||||||
"GEN_DIR",
|
|
||||||
"ProtocGenerationError",
|
|
||||||
"generate",
|
|
||||||
"load",
|
|
||||||
"load_grpc",
|
|
||||||
"compute_checksum",
|
|
||||||
"verify_checksum",
|
|
||||||
"fragment_tensor",
|
|
||||||
"reassemble_tensor",
|
|
||||||
]
|
|
||||||
@@ -1,563 +0,0 @@
|
|||||||
"""Versioned performance contract metadata and stub benchmark runner for DGR-001.
|
|
||||||
|
|
||||||
This module captures the *contract* first: the model family, architecture
|
|
||||||
alignment, benchmark lanes, and stop condition that benchmark runs must
|
|
||||||
satisfy. It also runs the contract's lanes through a deterministic stub
|
|
||||||
backend so the report data shape exists end to end. It never downloads or
|
|
||||||
executes a model; real transformers / llama.cpp backends plug in behind the
|
|
||||||
same ``run()`` seam later.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import json
|
|
||||||
import time
|
|
||||||
import urllib.request
|
|
||||||
from dataclasses import dataclass
|
|
||||||
from pathlib import Path
|
|
||||||
from typing import Mapping
|
|
||||||
|
|
||||||
SCHEMA_VERSION = 1
|
|
||||||
CONTRACT_ID = "DGR-001"
|
|
||||||
DEFAULT_OUTPUT_PATH = Path(".scratch/distributed-gguf-runtime/evidence/DGR-001/performance-contract.json")
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class ModelTarget:
|
|
||||||
"""Architecture-aligned model target for the DGR-001 benchmark contract."""
|
|
||||||
|
|
||||||
name: str
|
|
||||||
architecture: str
|
|
||||||
safetensors_repo: str
|
|
||||||
safetensors_precision: str
|
|
||||||
gguf_repo: str
|
|
||||||
gguf_quant: str
|
|
||||||
gguf_size_gb: float
|
|
||||||
comparison_policy: str
|
|
||||||
rationale: str
|
|
||||||
|
|
||||||
def to_dict(self) -> dict:
|
|
||||||
return {
|
|
||||||
"name": self.name,
|
|
||||||
"architecture": self.architecture,
|
|
||||||
"safetensors_repo": self.safetensors_repo,
|
|
||||||
"safetensors_precision": self.safetensors_precision,
|
|
||||||
"gguf_repo": self.gguf_repo,
|
|
||||||
"gguf_quant": self.gguf_quant,
|
|
||||||
"gguf_size_gb": self.gguf_size_gb,
|
|
||||||
"comparison_policy": self.comparison_policy,
|
|
||||||
"rationale": self.rationale,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class BenchmarkLane:
|
|
||||||
"""One side of the comparison the contract requires."""
|
|
||||||
|
|
||||||
id: str
|
|
||||||
runtime: str
|
|
||||||
device: str
|
|
||||||
recipe: str
|
|
||||||
concurrency_levels: tuple[int, ...]
|
|
||||||
|
|
||||||
def to_dict(self) -> dict:
|
|
||||||
return {
|
|
||||||
"id": self.id,
|
|
||||||
"runtime": self.runtime,
|
|
||||||
"device": self.device,
|
|
||||||
"recipe": self.recipe,
|
|
||||||
"concurrency_levels": list(self.concurrency_levels),
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class BenchmarkWorkload:
|
|
||||||
"""Identical request shape both recipes must run so speed stays comparable.
|
|
||||||
|
|
||||||
Pinning prompts, context lengths, output lengths, and sampling policy in the
|
|
||||||
versioned contract is what makes the safetensors-versus-GGUF numbers a
|
|
||||||
controlled comparison instead of two differently-configured runs.
|
|
||||||
"""
|
|
||||||
|
|
||||||
prompts: tuple[str, ...]
|
|
||||||
context_lengths: tuple[int, ...]
|
|
||||||
output_lengths: tuple[int, ...]
|
|
||||||
sampling_policy: str
|
|
||||||
|
|
||||||
def to_dict(self) -> dict:
|
|
||||||
return {
|
|
||||||
"prompts": list(self.prompts),
|
|
||||||
"context_lengths": list(self.context_lengths),
|
|
||||||
"output_lengths": list(self.output_lengths),
|
|
||||||
"sampling_policy": self.sampling_policy,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class QualityPolicy:
|
|
||||||
"""Correctness/quality lane kept separate from the performance/fit lanes.
|
|
||||||
|
|
||||||
BF16 safetensors and Q2_K GGUF are not numerically equivalent, so quality is
|
|
||||||
measured as its own lane (output drift against the BF16 reference under a
|
|
||||||
documented tolerance) rather than assumed away by the speed/fit comparison.
|
|
||||||
"""
|
|
||||||
|
|
||||||
statement: str
|
|
||||||
reference_lane_runtime: str
|
|
||||||
measured_lane_runtime: str
|
|
||||||
max_output_drift: float
|
|
||||||
|
|
||||||
def to_dict(self) -> dict:
|
|
||||||
return {
|
|
||||||
"statement": self.statement,
|
|
||||||
"reference_lane_runtime": self.reference_lane_runtime,
|
|
||||||
"measured_lane_runtime": self.measured_lane_runtime,
|
|
||||||
"max_output_drift": self.max_output_drift,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class ReleaseGate:
|
|
||||||
"""Versioned thresholds later release gates (DGR-014) consume unchanged.
|
|
||||||
|
|
||||||
Thresholds live in the contract, not in code, so the release gate cannot be
|
|
||||||
weakened after seeing implementation results.
|
|
||||||
"""
|
|
||||||
|
|
||||||
min_decode_speedup: float
|
|
||||||
max_artifact_bytes_ratio: float
|
|
||||||
max_memory_bytes_ratio: float
|
|
||||||
max_quality_drift: float
|
|
||||||
|
|
||||||
def to_dict(self) -> dict:
|
|
||||||
return {
|
|
||||||
"min_decode_speedup": self.min_decode_speedup,
|
|
||||||
"max_artifact_bytes_ratio": self.max_artifact_bytes_ratio,
|
|
||||||
"max_memory_bytes_ratio": self.max_memory_bytes_ratio,
|
|
||||||
"max_quality_drift": self.max_quality_drift,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class PerformanceContract:
|
|
||||||
"""Machine-readable contract for the DGR-001 benchmark story."""
|
|
||||||
|
|
||||||
schema_version: int
|
|
||||||
story_id: str
|
|
||||||
model_target: ModelTarget
|
|
||||||
benchmark_lanes: tuple[BenchmarkLane, ...]
|
|
||||||
metrics: tuple[str, ...]
|
|
||||||
stop_condition: str
|
|
||||||
notes: tuple[str, ...] = ()
|
|
||||||
|
|
||||||
def to_dict(self) -> dict:
|
|
||||||
return {
|
|
||||||
"schema_version": self.schema_version,
|
|
||||||
"story_id": self.story_id,
|
|
||||||
"model_target": self.model_target.to_dict(),
|
|
||||||
"benchmark_lanes": [lane.to_dict() for lane in self.benchmark_lanes],
|
|
||||||
"metrics": list(self.metrics),
|
|
||||||
"stop_condition": self.stop_condition,
|
|
||||||
"notes": list(self.notes),
|
|
||||||
}
|
|
||||||
|
|
||||||
def write_json(self, path: str | Path) -> Path:
|
|
||||||
path = Path(path)
|
|
||||||
path.parent.mkdir(parents=True, exist_ok=True)
|
|
||||||
path.write_text(json.dumps(self.to_dict(), indent=2, sort_keys=True) + "\n", encoding="utf-8")
|
|
||||||
return path
|
|
||||||
|
|
||||||
|
|
||||||
DEFAULT_CONTRACT = PerformanceContract(
|
|
||||||
schema_version=SCHEMA_VERSION,
|
|
||||||
story_id=CONTRACT_ID,
|
|
||||||
model_target=ModelTarget(
|
|
||||||
name="DeepSeek-V2-Lite-Chat",
|
|
||||||
architecture="deepseek2",
|
|
||||||
safetensors_repo="deepseek-ai/DeepSeek-V2-Lite-Chat",
|
|
||||||
safetensors_precision="bfloat16",
|
|
||||||
gguf_repo="second-state/DeepSeek-V2-Lite-Chat-GGUF",
|
|
||||||
gguf_quant="Q2_K",
|
|
||||||
gguf_size_gb=6.43,
|
|
||||||
comparison_policy=(
|
|
||||||
"same model/revision, closest practical low-footprint precision pair: "
|
|
||||||
"BF16 safetensors versus Q2_K GGUF"
|
|
||||||
),
|
|
||||||
rationale=(
|
|
||||||
"Smallest DeepSeek-family benchmark anchor that still points toward "
|
|
||||||
"DeepSeek-V4-Flash; keeps the runtime on the DeepSeek2 path instead "
|
|
||||||
"of falling back to a tiny but architecture-mismatched smoke model."
|
|
||||||
),
|
|
||||||
),
|
|
||||||
benchmark_lanes=(
|
|
||||||
BenchmarkLane(
|
|
||||||
id="transformers-safetensors-cpu",
|
|
||||||
runtime="transformers",
|
|
||||||
device="cpu",
|
|
||||||
recipe="current safetensors recipe",
|
|
||||||
concurrency_levels=(1, 4),
|
|
||||||
),
|
|
||||||
BenchmarkLane(
|
|
||||||
id="llama-cpp-gguf-cpu",
|
|
||||||
runtime="llama.cpp",
|
|
||||||
device="cpu",
|
|
||||||
recipe="whole-model GGUF recipe",
|
|
||||||
concurrency_levels=(1, 4),
|
|
||||||
),
|
|
||||||
BenchmarkLane(
|
|
||||||
id="transformers-safetensors-gpu",
|
|
||||||
runtime="transformers",
|
|
||||||
device="gpu",
|
|
||||||
recipe="current safetensors recipe",
|
|
||||||
concurrency_levels=(1, 4),
|
|
||||||
),
|
|
||||||
BenchmarkLane(
|
|
||||||
id="llama-cpp-gguf-gpu",
|
|
||||||
runtime="llama.cpp",
|
|
||||||
device="gpu",
|
|
||||||
recipe="whole-model GGUF recipe",
|
|
||||||
concurrency_levels=(1, 4),
|
|
||||||
),
|
|
||||||
),
|
|
||||||
metrics=(
|
|
||||||
"ttft_ms",
|
|
||||||
"prefill_tok_per_sec",
|
|
||||||
"decode_tok_per_sec",
|
|
||||||
"p50_latency_ms",
|
|
||||||
"p95_latency_ms",
|
|
||||||
"aggregate_throughput_tok_per_sec",
|
|
||||||
"rss_bytes",
|
|
||||||
"vram_bytes",
|
|
||||||
"artifact_bytes",
|
|
||||||
"failure_count",
|
|
||||||
"output_drift",
|
|
||||||
),
|
|
||||||
stop_condition=(
|
|
||||||
"Stop if GGUF does not provide a meaningful speed or fit benefit over the "
|
|
||||||
"safetensors baseline for the chosen DeepSeek-family model target."
|
|
||||||
),
|
|
||||||
notes=(
|
|
||||||
"Real model execution stays opt-in and must keep model artifacts on the mounted drive.",
|
|
||||||
"Use the tiny fallback only for loader plumbing smoke tests; it does not replace the architecture-aligned baseline.",
|
|
||||||
),
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def build_default_contract() -> PerformanceContract:
|
|
||||||
return DEFAULT_CONTRACT
|
|
||||||
|
|
||||||
|
|
||||||
BENCHMARK_SCHEMA_VERSION = 1
|
|
||||||
STUB_OUTPUT_TOKENS = ("mesh", "activation", "seam", "baseline")
|
|
||||||
# DeepSeek-V2-Lite is ~15.7B params at 2 bytes each; metadata only, nothing downloaded.
|
|
||||||
_SAFETENSORS_BF16_ARTIFACT_GB = 31.4
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class LaneSample:
|
|
||||||
"""Raw single-stream measurements one backend produces for a lane."""
|
|
||||||
|
|
||||||
ttft_ms: float
|
|
||||||
prefill_tok_per_sec: float
|
|
||||||
decode_tok_per_sec: float
|
|
||||||
rss_bytes: int
|
|
||||||
vram_bytes: int
|
|
||||||
artifact_bytes: int
|
|
||||||
output_tokens: tuple[str, ...]
|
|
||||||
failure_count: int = 0
|
|
||||||
|
|
||||||
|
|
||||||
def _gb(value: float) -> int:
|
|
||||||
return int(value * 1024**3)
|
|
||||||
|
|
||||||
|
|
||||||
class StubLaneBackend:
|
|
||||||
"""Deterministic placeholder measurements until real lane execution lands.
|
|
||||||
|
|
||||||
The numbers are synthetic but directionally shaped — the Q2_K GGUF loads a
|
|
||||||
far smaller artifact and decodes faster than BF16 safetensors — so the
|
|
||||||
comparison and stop-condition plumbing can be exercised in CI.
|
|
||||||
"""
|
|
||||||
|
|
||||||
source = "stub-backend"
|
|
||||||
|
|
||||||
# (runtime, device) -> (ttft_ms, prefill tok/s, decode tok/s, rss GB, vram GB)
|
|
||||||
_PROFILES = {
|
|
||||||
("transformers", "cpu"): (1800.0, 45.0, 6.0, 33.0, 0.0),
|
|
||||||
("llama.cpp", "cpu"): (950.0, 90.0, 14.0, 7.1, 0.0),
|
|
||||||
("transformers", "gpu"): (420.0, 850.0, 34.0, 4.0, 33.0),
|
|
||||||
("llama.cpp", "gpu"): (260.0, 640.0, 52.0, 1.5, 7.5),
|
|
||||||
}
|
|
||||||
|
|
||||||
def __init__(self, contract: PerformanceContract) -> None:
|
|
||||||
self._contract = contract
|
|
||||||
|
|
||||||
def run(self, lane: BenchmarkLane) -> LaneSample:
|
|
||||||
ttft_ms, prefill, decode, rss_gb, vram_gb = self._PROFILES[(lane.runtime, lane.device)]
|
|
||||||
artifact_gb = (
|
|
||||||
self._contract.model_target.gguf_size_gb
|
|
||||||
if lane.runtime == "llama.cpp"
|
|
||||||
else _SAFETENSORS_BF16_ARTIFACT_GB
|
|
||||||
)
|
|
||||||
return LaneSample(
|
|
||||||
ttft_ms=ttft_ms,
|
|
||||||
prefill_tok_per_sec=prefill,
|
|
||||||
decode_tok_per_sec=decode,
|
|
||||||
rss_bytes=_gb(rss_gb),
|
|
||||||
vram_bytes=_gb(vram_gb),
|
|
||||||
artifact_bytes=_gb(artifact_gb),
|
|
||||||
output_tokens=STUB_OUTPUT_TOKENS,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _output_drift(tokens: tuple[str, ...], reference: tuple[str, ...]) -> float:
|
|
||||||
"""Fraction of positions where a lane's output diverges from its reference."""
|
|
||||||
length = max(len(tokens), len(reference))
|
|
||||||
if length == 0:
|
|
||||||
return 0.0
|
|
||||||
mismatches = sum(a != b for a, b in zip(tokens, reference)) + abs(len(tokens) - len(reference))
|
|
||||||
return round(mismatches / length, 4)
|
|
||||||
|
|
||||||
|
|
||||||
def _metrics_for(sample: LaneSample, concurrency: int, output_drift: float) -> dict:
|
|
||||||
# Stub concurrency model: batching scales throughput at 85% efficiency and
|
|
||||||
# stretches per-request token latency and TTFT accordingly.
|
|
||||||
efficiency = 1.0 if concurrency == 1 else 0.85
|
|
||||||
p50_latency_ms = round(1000.0 / (sample.decode_tok_per_sec * efficiency), 4)
|
|
||||||
return {
|
|
||||||
"ttft_ms": round(sample.ttft_ms * (1 + 0.1 * (concurrency - 1)), 4),
|
|
||||||
"prefill_tok_per_sec": round(sample.prefill_tok_per_sec * efficiency, 4),
|
|
||||||
"decode_tok_per_sec": round(sample.decode_tok_per_sec * efficiency, 4),
|
|
||||||
"p50_latency_ms": p50_latency_ms,
|
|
||||||
"p95_latency_ms": round(p50_latency_ms * 1.25, 4),
|
|
||||||
"aggregate_throughput_tok_per_sec": round(sample.decode_tok_per_sec * concurrency * efficiency, 4),
|
|
||||||
"rss_bytes": sample.rss_bytes,
|
|
||||||
"vram_bytes": sample.vram_bytes,
|
|
||||||
"artifact_bytes": sample.artifact_bytes,
|
|
||||||
"failure_count": sample.failure_count,
|
|
||||||
"output_drift": output_drift,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def _compare_device(lanes: list[tuple[BenchmarkLane, LaneSample]], device: str) -> dict:
|
|
||||||
by_runtime = {lane.runtime: (lane, sample) for lane, sample in lanes if lane.device == device}
|
|
||||||
safetensors_lane, safetensors = by_runtime["transformers"]
|
|
||||||
gguf_lane, gguf = by_runtime["llama.cpp"]
|
|
||||||
memory_metric = "vram_bytes" if device == "gpu" else "rss_bytes"
|
|
||||||
decode_speedup = round(gguf.decode_tok_per_sec / safetensors.decode_tok_per_sec, 4)
|
|
||||||
artifact_bytes_ratio = round(gguf.artifact_bytes / max(1, safetensors.artifact_bytes), 4)
|
|
||||||
return {
|
|
||||||
"safetensors_lane": safetensors_lane.id,
|
|
||||||
"gguf_lane": gguf_lane.id,
|
|
||||||
"decode_speedup": decode_speedup,
|
|
||||||
"ttft_speedup": round(safetensors.ttft_ms / max(0.001, gguf.ttft_ms), 4),
|
|
||||||
"artifact_bytes_ratio": artifact_bytes_ratio,
|
|
||||||
"memory_metric": memory_metric,
|
|
||||||
"memory_bytes_ratio": round(
|
|
||||||
getattr(gguf, memory_metric) / max(1, getattr(safetensors, memory_metric)), 4
|
|
||||||
),
|
|
||||||
"output_drift": _output_drift(gguf.output_tokens, safetensors.output_tokens),
|
|
||||||
"gguf_benefit": decode_speedup >= 1.10 or artifact_bytes_ratio <= 0.5,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def run_performance_benchmark(
|
|
||||||
contract: PerformanceContract = DEFAULT_CONTRACT,
|
|
||||||
backend: StubLaneBackend | None = None,
|
|
||||||
) -> dict:
|
|
||||||
"""Run every contract lane through a backend and compare GGUF to safetensors."""
|
|
||||||
backend = backend if backend is not None else StubLaneBackend(contract)
|
|
||||||
lanes = [(lane, backend.run(lane)) for lane in contract.benchmark_lanes]
|
|
||||||
references = {
|
|
||||||
lane.device: sample.output_tokens for lane, sample in lanes if lane.runtime == "transformers"
|
|
||||||
}
|
|
||||||
lane_reports = []
|
|
||||||
for lane, sample in lanes:
|
|
||||||
drift = _output_drift(sample.output_tokens, references.get(lane.device, sample.output_tokens))
|
|
||||||
lane_reports.append({
|
|
||||||
**lane.to_dict(),
|
|
||||||
"output_tokens": list(sample.output_tokens),
|
|
||||||
"results": [
|
|
||||||
{"concurrency": level, "metrics": _metrics_for(sample, level, drift)}
|
|
||||||
for level in lane.concurrency_levels
|
|
||||||
],
|
|
||||||
})
|
|
||||||
devices = sorted({lane.device for lane, _ in lanes})
|
|
||||||
comparisons = {device: _compare_device(lanes, device) for device in devices}
|
|
||||||
gguf_benefit = any(comparison["gguf_benefit"] for comparison in comparisons.values())
|
|
||||||
return {
|
|
||||||
"schema_version": BENCHMARK_SCHEMA_VERSION,
|
|
||||||
"story_id": contract.story_id,
|
|
||||||
"source": getattr(backend, "source", "custom-backend"),
|
|
||||||
"model_target": contract.model_target.to_dict(),
|
|
||||||
"lanes": lane_reports,
|
|
||||||
"comparisons": comparisons,
|
|
||||||
"stop_condition": {
|
|
||||||
"text": contract.stop_condition,
|
|
||||||
"gguf_benefit": gguf_benefit,
|
|
||||||
"triggered": not gguf_benefit,
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def run_real_model_endpoint_benchmark(
|
|
||||||
endpoints: Mapping[str, str],
|
|
||||||
*,
|
|
||||||
model: str,
|
|
||||||
contract: PerformanceContract = DEFAULT_CONTRACT,
|
|
||||||
timeout: float = 120.0,
|
|
||||||
) -> dict:
|
|
||||||
"""Run one live OpenAI-compatible request per lane against supplied endpoints.
|
|
||||||
|
|
||||||
The caller provides one URL per benchmark lane. The runner measures the
|
|
||||||
request/response round-trip at the client boundary and reuses the same
|
|
||||||
contract schema as the deterministic stub.
|
|
||||||
"""
|
|
||||||
|
|
||||||
def _sample_for_lane(lane: BenchmarkLane, endpoint: str) -> LaneSample:
|
|
||||||
prompt = " ".join(contract.model_target.rationale.split()[:6])
|
|
||||||
body = json.dumps(
|
|
||||||
{
|
|
||||||
"model": model,
|
|
||||||
"messages": [{"role": "user", "content": prompt}],
|
|
||||||
"max_tokens": len(STUB_OUTPUT_TOKENS),
|
|
||||||
"temperature": 0,
|
|
||||||
}
|
|
||||||
).encode("utf-8")
|
|
||||||
request = urllib.request.Request(
|
|
||||||
f"{endpoint.rstrip('/')}/v1/chat/completions",
|
|
||||||
data=body,
|
|
||||||
headers={
|
|
||||||
"Content-Type": "application/json",
|
|
||||||
"X-Meshnet-Lane": lane.id,
|
|
||||||
},
|
|
||||||
method="POST",
|
|
||||||
)
|
|
||||||
started = time.monotonic()
|
|
||||||
with urllib.request.urlopen(request, timeout=timeout) as response:
|
|
||||||
response_body = response.read()
|
|
||||||
session_id = response.headers.get("X-Meshnet-Session", f"{lane.id}-session")
|
|
||||||
elapsed_ms = round((time.monotonic() - started) * 1000, 4)
|
|
||||||
payload = json.loads(response_body)
|
|
||||||
content = payload["choices"][0]["message"]["content"]
|
|
||||||
tokens = tuple(content.split())
|
|
||||||
token_count = max(1, len(tokens))
|
|
||||||
artifact_gb = (
|
|
||||||
contract.model_target.gguf_size_gb
|
|
||||||
if lane.runtime == "llama.cpp"
|
|
||||||
else _SAFETENSORS_BF16_ARTIFACT_GB
|
|
||||||
)
|
|
||||||
return LaneSample(
|
|
||||||
ttft_ms=elapsed_ms,
|
|
||||||
prefill_tok_per_sec=round(token_count / max(0.001, elapsed_ms / 1000), 4),
|
|
||||||
decode_tok_per_sec=round(token_count / max(0.001, elapsed_ms / 1000), 4),
|
|
||||||
rss_bytes=0,
|
|
||||||
vram_bytes=0,
|
|
||||||
artifact_bytes=_gb(artifact_gb),
|
|
||||||
output_tokens=tokens,
|
|
||||||
)
|
|
||||||
|
|
||||||
lanes = []
|
|
||||||
for lane in contract.benchmark_lanes:
|
|
||||||
if lane.id not in endpoints:
|
|
||||||
raise KeyError(f"missing endpoint for lane {lane.id}")
|
|
||||||
lanes.append((lane, _sample_for_lane(lane, endpoints[lane.id])))
|
|
||||||
references = {
|
|
||||||
lane.device: sample.output_tokens for lane, sample in lanes if lane.runtime == "transformers"
|
|
||||||
}
|
|
||||||
lane_reports = []
|
|
||||||
for lane, sample in lanes:
|
|
||||||
drift = _output_drift(sample.output_tokens, references.get(lane.device, sample.output_tokens))
|
|
||||||
lane_reports.append({
|
|
||||||
**lane.to_dict(),
|
|
||||||
"output_tokens": list(sample.output_tokens),
|
|
||||||
"results": [
|
|
||||||
{"concurrency": level, "metrics": _metrics_for(sample, level, drift)}
|
|
||||||
for level in lane.concurrency_levels
|
|
||||||
],
|
|
||||||
})
|
|
||||||
devices = sorted({lane.device for lane, _ in lanes})
|
|
||||||
comparisons = {device: _compare_device(lanes, device) for device in devices}
|
|
||||||
gguf_benefit = any(comparison["gguf_benefit"] for comparison in comparisons.values())
|
|
||||||
return {
|
|
||||||
"schema_version": BENCHMARK_SCHEMA_VERSION,
|
|
||||||
"story_id": contract.story_id,
|
|
||||||
"source": "real-model-endpoints",
|
|
||||||
"model_target": contract.model_target.to_dict(),
|
|
||||||
"lanes": lane_reports,
|
|
||||||
"comparisons": comparisons,
|
|
||||||
"stop_condition": {
|
|
||||||
"text": contract.stop_condition,
|
|
||||||
"gguf_benefit": gguf_benefit,
|
|
||||||
"triggered": not gguf_benefit,
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def _parse_lane_endpoints(pairs: list[str], parser: argparse.ArgumentParser) -> dict[str, str]:
|
|
||||||
endpoints: dict[str, str] = {}
|
|
||||||
for pair in pairs:
|
|
||||||
lane_id, sep, url = pair.partition("=")
|
|
||||||
if not sep or not lane_id or not url:
|
|
||||||
parser.error(f"--live-endpoint expects LANE_ID=URL, got {pair!r}")
|
|
||||||
endpoints[lane_id] = url
|
|
||||||
return endpoints
|
|
||||||
|
|
||||||
|
|
||||||
def _write_report(report: dict, path: Path) -> None:
|
|
||||||
path.parent.mkdir(parents=True, exist_ok=True)
|
|
||||||
path.write_text(json.dumps(report, indent=2, sort_keys=True) + "\n", encoding="utf-8")
|
|
||||||
|
|
||||||
|
|
||||||
def main(argv: list[str] | None = None) -> int:
|
|
||||||
parser = argparse.ArgumentParser(description="Write the DGR-001 performance contract JSON")
|
|
||||||
parser.add_argument("--json-out", type=Path, default=DEFAULT_OUTPUT_PATH, help="output JSON path")
|
|
||||||
parser.add_argument(
|
|
||||||
"--benchmark-out",
|
|
||||||
type=Path,
|
|
||||||
default=None,
|
|
||||||
help="also run the deterministic stub benchmark and write its JSON report here",
|
|
||||||
)
|
|
||||||
parser.add_argument(
|
|
||||||
"--live-endpoint",
|
|
||||||
action="append",
|
|
||||||
default=None,
|
|
||||||
metavar="LANE_ID=URL",
|
|
||||||
help="lane-to-endpoint mapping for the live benchmark; repeat once per contract lane",
|
|
||||||
)
|
|
||||||
parser.add_argument(
|
|
||||||
"--live-model",
|
|
||||||
default=None,
|
|
||||||
help="model name sent to live endpoints (default: contract safetensors repo)",
|
|
||||||
)
|
|
||||||
parser.add_argument(
|
|
||||||
"--live-benchmark-out",
|
|
||||||
type=Path,
|
|
||||||
default=None,
|
|
||||||
help="run the live endpoint benchmark against --live-endpoint lanes and write its JSON report here",
|
|
||||||
)
|
|
||||||
args = parser.parse_args(argv)
|
|
||||||
if args.live_endpoint and args.live_benchmark_out is None:
|
|
||||||
parser.error("--live-endpoint requires --live-benchmark-out")
|
|
||||||
if args.live_benchmark_out is not None and not args.live_endpoint:
|
|
||||||
parser.error("--live-benchmark-out requires at least one --live-endpoint")
|
|
||||||
contract = build_default_contract()
|
|
||||||
path = contract.write_json(args.json_out)
|
|
||||||
print(path)
|
|
||||||
if args.benchmark_out is not None:
|
|
||||||
_write_report(run_performance_benchmark(contract), args.benchmark_out)
|
|
||||||
print(args.benchmark_out)
|
|
||||||
if args.live_endpoint:
|
|
||||||
report = run_real_model_endpoint_benchmark(
|
|
||||||
_parse_lane_endpoints(args.live_endpoint, parser),
|
|
||||||
model=args.live_model or contract.model_target.safetensors_repo,
|
|
||||||
contract=contract,
|
|
||||||
)
|
|
||||||
_write_report(report, args.live_benchmark_out)
|
|
||||||
print(args.live_benchmark_out)
|
|
||||||
return 0
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__": # pragma: no cover - CLI entry point
|
|
||||||
raise SystemExit(main())
|
|
||||||
@@ -26,16 +26,6 @@
|
|||||||
"params": {
|
"params": {
|
||||||
"use_cache": false
|
"use_cache": false
|
||||||
}
|
}
|
||||||
},
|
|
||||||
{
|
|
||||||
"id": "llama-cpp-native",
|
|
||||||
"version": "1",
|
|
||||||
"backend_id": "llama.cpp",
|
|
||||||
"description": "Project-owned native GGUF worker behind the Meshnet control plane.",
|
|
||||||
"params": {
|
|
||||||
"worker_transport": "grpc",
|
|
||||||
"use_cache": true
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -44,7 +44,6 @@ class SeamSample:
|
|||||||
cache_mode: CacheMode
|
cache_mode: CacheMode
|
||||||
model_ms: float
|
model_ms: float
|
||||||
encode_ms: float
|
encode_ms: float
|
||||||
activation_decode_ms: float
|
|
||||||
framing_ms: float
|
framing_ms: float
|
||||||
metadata_ms: float
|
metadata_ms: float
|
||||||
copy_allocation_ms: float
|
copy_allocation_ms: float
|
||||||
@@ -53,7 +52,6 @@ class SeamSample:
|
|||||||
decompression_ms: float
|
decompression_ms: float
|
||||||
connection_setup_ms: float
|
connection_setup_ms: float
|
||||||
queue_wait_ms: float
|
queue_wait_ms: float
|
||||||
local_http_forwarding_ms: float
|
|
||||||
transport_ms: float
|
transport_ms: float
|
||||||
seam_latency_ms: float
|
seam_latency_ms: float
|
||||||
payload_bytes: int
|
payload_bytes: int
|
||||||
@@ -122,10 +120,6 @@ def _summary(samples: list[SeamSample]) -> dict[str, float | int]:
|
|||||||
"compression_cpu_ms": round(
|
"compression_cpu_ms": round(
|
||||||
sum(sample.compression_ms + sample.decompression_ms for sample in samples), 4
|
sum(sample.compression_ms + sample.decompression_ms for sample in samples), 4
|
||||||
),
|
),
|
||||||
"model_execution_ms": round(sum(sample.model_ms for sample in samples), 4),
|
|
||||||
"activation_encoding_ms": round(sum(sample.encode_ms for sample in samples), 4),
|
|
||||||
"activation_decoding_ms": round(sum(sample.activation_decode_ms for sample in samples), 4),
|
|
||||||
"local_http_forwarding_ms": round(sum(sample.local_http_forwarding_ms for sample in samples), 4),
|
|
||||||
"peak_buffered_bytes": max((sample.copy_allocation_bytes for sample in samples), default=0),
|
"peak_buffered_bytes": max((sample.copy_allocation_bytes for sample in samples), default=0),
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -165,7 +159,6 @@ class _StubTransport:
|
|||||||
queue_wait_ms = 0.0 if self.mode == "direct" else 0.18 + (0.05 if token_index is not None and token_index % 2 else 0.0)
|
queue_wait_ms = 0.0 if self.mode == "direct" else 0.18 + (0.05 if token_index is not None and token_index % 2 else 0.0)
|
||||||
model_ms = 1.6 if phase == "prefill" else 0.45
|
model_ms = 1.6 if phase == "prefill" else 0.45
|
||||||
encode_ms = 0.16 if phase == "prefill" else 0.06
|
encode_ms = 0.16 if phase == "prefill" else 0.06
|
||||||
activation_decode_ms = 0.055 if phase == "prefill" else 0.02
|
|
||||||
# Keep framing/metadata/copy costs explicit rather than hiding them in
|
# Keep framing/metadata/copy costs explicit rather than hiding them in
|
||||||
# serialization or transport time. The stub owns one binary frame and
|
# serialization or transport time. The stub owns one binary frame and
|
||||||
# one response body per hop; no base64 body is modeled.
|
# one response body per hop; no base64 body is modeled.
|
||||||
@@ -175,26 +168,20 @@ class _StubTransport:
|
|||||||
copy_allocation_bytes = wire_bytes + payload_bytes
|
copy_allocation_bytes = wire_bytes + payload_bytes
|
||||||
compression_ms = 0.09 if self.scenario.compression else 0.0
|
compression_ms = 0.09 if self.scenario.compression else 0.0
|
||||||
decompression_ms = 0.07 if self.scenario.compression else 0.0
|
decompression_ms = 0.07 if self.scenario.compression else 0.0
|
||||||
# Both routes finish by forwarding the decoded activation to the local
|
|
||||||
# tail-node HTTP handler; relay adds its own queue before that hop.
|
|
||||||
local_http_forwarding_ms = 0.11 if self.mode == "direct" else 0.16
|
|
||||||
transport_ms = (0.32 if self.mode == "direct" else 0.61) + wire_bytes / 100_000
|
transport_ms = (0.32 if self.mode == "direct" else 0.61) + wire_bytes / 100_000
|
||||||
seam_latency_ms = round(
|
seam_latency_ms = round(
|
||||||
model_ms + encode_ms + activation_decode_ms + framing_ms + metadata_ms + copy_allocation_ms
|
model_ms + encode_ms + framing_ms + metadata_ms + copy_allocation_ms
|
||||||
+ compression_ms + decompression_ms + connection_setup_ms + queue_wait_ms + transport_ms
|
+ compression_ms + decompression_ms + connection_setup_ms + queue_wait_ms + transport_ms,
|
||||||
+ local_http_forwarding_ms,
|
|
||||||
4,
|
4,
|
||||||
)
|
)
|
||||||
return SeamSample(
|
return SeamSample(
|
||||||
phase=phase, token_index=token_index, session_id=self.session_id,
|
phase=phase, token_index=token_index, session_id=self.session_id,
|
||||||
activation_id=f"benchmark-activation-{self._activation_count}", seam="head->tail", mode=self.mode,
|
activation_id=f"benchmark-activation-{self._activation_count}", seam="head->tail", mode=self.mode,
|
||||||
cache_mode=self.cache_mode, model_ms=model_ms, encode_ms=encode_ms,
|
cache_mode=self.cache_mode, model_ms=model_ms, encode_ms=encode_ms,
|
||||||
activation_decode_ms=activation_decode_ms,
|
|
||||||
framing_ms=framing_ms, metadata_ms=metadata_ms,
|
framing_ms=framing_ms, metadata_ms=metadata_ms,
|
||||||
copy_allocation_ms=copy_allocation_ms, copy_allocation_bytes=copy_allocation_bytes,
|
copy_allocation_ms=copy_allocation_ms, copy_allocation_bytes=copy_allocation_bytes,
|
||||||
compression_ms=compression_ms, decompression_ms=decompression_ms,
|
compression_ms=compression_ms, decompression_ms=decompression_ms,
|
||||||
connection_setup_ms=connection_setup_ms, queue_wait_ms=queue_wait_ms,
|
connection_setup_ms=connection_setup_ms, queue_wait_ms=queue_wait_ms,
|
||||||
local_http_forwarding_ms=local_http_forwarding_ms,
|
|
||||||
transport_ms=round(transport_ms, 4), seam_latency_ms=seam_latency_ms,
|
transport_ms=round(transport_ms, 4), seam_latency_ms=seam_latency_ms,
|
||||||
payload_bytes=payload_bytes, wire_bytes=wire_bytes,
|
payload_bytes=payload_bytes, wire_bytes=wire_bytes,
|
||||||
compression_ratio=round(payload_bytes / wire_bytes, 4), connection_attempted=connection_attempted,
|
compression_ratio=round(payload_bytes / wire_bytes, 4), connection_attempted=connection_attempted,
|
||||||
@@ -342,10 +329,9 @@ def run_real_model_lan_benchmark(url: str, *, model: str, timeout: float = 120.0
|
|||||||
sample = SeamSample(
|
sample = SeamSample(
|
||||||
phase="decode", token_index=0, session_id=session_id, activation_id="lan-activation-1",
|
phase="decode", token_index=0, session_id=session_id, activation_id="lan-activation-1",
|
||||||
seam="head->tail", mode="direct", cache_mode="cached", model_ms=0.0, encode_ms=0.0,
|
seam="head->tail", mode="direct", cache_mode="cached", model_ms=0.0, encode_ms=0.0,
|
||||||
activation_decode_ms=0.0,
|
|
||||||
framing_ms=0.0, metadata_ms=0.0, copy_allocation_ms=0.0, copy_allocation_bytes=0,
|
framing_ms=0.0, metadata_ms=0.0, copy_allocation_ms=0.0, copy_allocation_bytes=0,
|
||||||
compression_ms=0.0, decompression_ms=0.0, connection_setup_ms=elapsed_ms,
|
compression_ms=0.0, decompression_ms=0.0, connection_setup_ms=elapsed_ms,
|
||||||
queue_wait_ms=0.0, local_http_forwarding_ms=0.0, transport_ms=elapsed_ms, seam_latency_ms=elapsed_ms,
|
queue_wait_ms=0.0, transport_ms=elapsed_ms, seam_latency_ms=elapsed_ms,
|
||||||
payload_bytes=len(body), wire_bytes=len(body) + len(response_body), compression_ratio=1.0,
|
payload_bytes=len(body), wire_bytes=len(body) + len(response_body), compression_ratio=1.0,
|
||||||
connection_attempted=True,
|
connection_attempted=True,
|
||||||
)
|
)
|
||||||
@@ -368,10 +354,6 @@ def format_summary(report: dict) -> str:
|
|||||||
f"{decode['tokens_per_sec']:.1f} tok/s; {decode['bytes_per_token']:.0f} B/tok; "
|
f"{decode['tokens_per_sec']:.1f} tok/s; {decode['bytes_per_token']:.0f} B/tok; "
|
||||||
f"seam {seam['payload_bytes']}/{seam['wire_bytes']} B "
|
f"seam {seam['payload_bytes']}/{seam['wire_bytes']} B "
|
||||||
f"({seam['compression_ratio']:.2f}x); connections {run['connections']['attempts']}; "
|
f"({seam['compression_ratio']:.2f}x); connections {run['connections']['attempts']}; "
|
||||||
f"model/encode/decode {decode['model_execution_ms']:.2f}/"
|
|
||||||
f"{decode['activation_encoding_ms']:.2f}/{decode['activation_decoding_ms']:.2f} ms; "
|
|
||||||
f"compression {decode['compression_cpu_ms']:.2f} ms; "
|
|
||||||
f"HTTP {decode['local_http_forwarding_ms']:.2f} ms; "
|
|
||||||
f"queue p95 {decode['p95_queue_wait_ms']:.2f} ms"
|
f"queue p95 {decode['p95_queue_wait_ms']:.2f} ms"
|
||||||
)
|
)
|
||||||
return "\n".join(lines)
|
return "\n".join(lines)
|
||||||
|
|||||||
@@ -1,375 +0,0 @@
|
|||||||
"""Exact artifact and runtime-recipe identity helpers.
|
|
||||||
|
|
||||||
The runtime recipe is the compatibility contract for one routable shard. It is
|
|
||||||
kept separate from the user-facing recipe catalogue so the tracker can compare
|
|
||||||
the exact execution footprint that was validated, not just a named recipe.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import hashlib
|
|
||||||
import json
|
|
||||||
from dataclasses import dataclass
|
|
||||||
from typing import Any, Mapping
|
|
||||||
|
|
||||||
|
|
||||||
def _require_text(value: Any, field_name: str) -> str:
|
|
||||||
if not isinstance(value, str) or not value.strip():
|
|
||||||
raise ValueError(f"{field_name!r} must be a non-empty string")
|
|
||||||
return value
|
|
||||||
|
|
||||||
|
|
||||||
def _optional_text(value: Any, field_name: str) -> str | None:
|
|
||||||
if value is None:
|
|
||||||
return None
|
|
||||||
return _require_text(value, field_name)
|
|
||||||
|
|
||||||
|
|
||||||
def _sha256_text(text: str) -> str:
|
|
||||||
return hashlib.sha256(text.encode("utf-8")).hexdigest()
|
|
||||||
|
|
||||||
|
|
||||||
def _stable_json(data: Any) -> str:
|
|
||||||
return json.dumps(
|
|
||||||
data,
|
|
||||||
sort_keys=True,
|
|
||||||
separators=(",", ":"),
|
|
||||||
ensure_ascii=False,
|
|
||||||
default=str,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _normalise_dtype(value: Any, default: str) -> str:
|
|
||||||
if value is None:
|
|
||||||
return default
|
|
||||||
if isinstance(value, str):
|
|
||||||
text = value.strip()
|
|
||||||
if not text:
|
|
||||||
return default
|
|
||||||
return text.removeprefix("torch.")
|
|
||||||
return str(value).removeprefix("torch.")
|
|
||||||
|
|
||||||
|
|
||||||
def _architecture_adapter_from_config(model_config: Any, default: str) -> str:
|
|
||||||
if not isinstance(model_config, Mapping):
|
|
||||||
return default
|
|
||||||
for key in ("architecture_adapter", "model_type"):
|
|
||||||
value = model_config.get(key)
|
|
||||||
if isinstance(value, str) and value.strip():
|
|
||||||
return value
|
|
||||||
architectures = model_config.get("architectures")
|
|
||||||
if isinstance(architectures, list) and architectures:
|
|
||||||
first = architectures[0]
|
|
||||||
if isinstance(first, str) and first.strip():
|
|
||||||
return first
|
|
||||||
text_config = model_config.get("text_config")
|
|
||||||
if isinstance(text_config, Mapping):
|
|
||||||
return _architecture_adapter_from_config(text_config, default)
|
|
||||||
return default
|
|
||||||
|
|
||||||
|
|
||||||
def _tokenizer_revision_from_config(
|
|
||||||
model_id: str,
|
|
||||||
revision: str | None,
|
|
||||||
model_config: Any,
|
|
||||||
) -> str:
|
|
||||||
if isinstance(model_config, Mapping):
|
|
||||||
for key in ("tokenizer_revision", "tokenizer_version", "_commit_hash"):
|
|
||||||
value = model_config.get(key)
|
|
||||||
if isinstance(value, str) and value.strip():
|
|
||||||
return value
|
|
||||||
if revision:
|
|
||||||
return revision
|
|
||||||
return model_id
|
|
||||||
|
|
||||||
|
|
||||||
def _cache_layout_from_recipe_params(recipe_params: Mapping[str, Any] | None) -> str:
|
|
||||||
if not recipe_params:
|
|
||||||
return "local-hot-kv"
|
|
||||||
use_cache = recipe_params.get("use_cache")
|
|
||||||
if use_cache is False:
|
|
||||||
return "stateless"
|
|
||||||
if "cache_layout" in recipe_params:
|
|
||||||
value = recipe_params.get("cache_layout")
|
|
||||||
if isinstance(value, str) and value.strip():
|
|
||||||
return value
|
|
||||||
return "local-hot-kv"
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class ArtifactIdentity:
|
|
||||||
"""Exact source artifact binding for a routable shard."""
|
|
||||||
|
|
||||||
model_id: str
|
|
||||||
revision: str | None = None
|
|
||||||
artifact_hash: str | None = None
|
|
||||||
shard_start: int | None = None
|
|
||||||
shard_end: int | None = None
|
|
||||||
|
|
||||||
def __post_init__(self) -> None:
|
|
||||||
_require_text(self.model_id, "artifact.model_id")
|
|
||||||
_optional_text(self.revision, "artifact.revision")
|
|
||||||
_optional_text(self.artifact_hash, "artifact.artifact_hash")
|
|
||||||
if self.shard_start is not None and self.shard_start < 0:
|
|
||||||
raise ValueError("'artifact.shard_start' must be >= 0")
|
|
||||||
if self.shard_end is not None and self.shard_end < 0:
|
|
||||||
raise ValueError("'artifact.shard_end' must be >= 0")
|
|
||||||
if (
|
|
||||||
self.shard_start is not None
|
|
||||||
and self.shard_end is not None
|
|
||||||
and self.shard_end < self.shard_start
|
|
||||||
):
|
|
||||||
raise ValueError("'artifact.shard_end' must be >= 'artifact.shard_start'")
|
|
||||||
|
|
||||||
def to_dict(self) -> dict[str, Any]:
|
|
||||||
return {
|
|
||||||
"model_id": self.model_id,
|
|
||||||
"revision": self.revision,
|
|
||||||
"artifact_hash": self.artifact_hash,
|
|
||||||
"shard_start": self.shard_start,
|
|
||||||
"shard_end": self.shard_end,
|
|
||||||
}
|
|
||||||
|
|
||||||
@classmethod
|
|
||||||
def from_dict(cls, data: Any) -> "ArtifactIdentity":
|
|
||||||
if not isinstance(data, Mapping):
|
|
||||||
raise ValueError(f"'artifact' must be a JSON object, got {type(data).__name__}")
|
|
||||||
return cls(
|
|
||||||
model_id=_require_text(data.get("model_id"), "artifact.model_id"),
|
|
||||||
revision=_optional_text(data.get("revision"), "artifact.revision"),
|
|
||||||
artifact_hash=_optional_text(
|
|
||||||
data.get("artifact_hash"), "artifact.artifact_hash"
|
|
||||||
),
|
|
||||||
shard_start=_optional_int(data.get("shard_start"), "artifact.shard_start"),
|
|
||||||
shard_end=_optional_int(data.get("shard_end"), "artifact.shard_end"),
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class RuntimeRecipeIdentity:
|
|
||||||
"""Exact runtime recipe used for admission and handshake compatibility."""
|
|
||||||
|
|
||||||
weight_quantization: str
|
|
||||||
activation_dtype: str
|
|
||||||
compute_dtype: str
|
|
||||||
kv_dtype: str
|
|
||||||
kv_layout: str
|
|
||||||
tokenizer_revision: str
|
|
||||||
architecture_adapter: str
|
|
||||||
backend_id: str
|
|
||||||
runtime_version: str
|
|
||||||
boundary_schema_version: int = 1
|
|
||||||
cache_layout: str = "local-hot-kv"
|
|
||||||
fingerprint: str | None = None
|
|
||||||
|
|
||||||
def __post_init__(self) -> None:
|
|
||||||
_require_text(self.weight_quantization, "runtime_recipe.weight_quantization")
|
|
||||||
_require_text(self.activation_dtype, "runtime_recipe.activation_dtype")
|
|
||||||
_require_text(self.compute_dtype, "runtime_recipe.compute_dtype")
|
|
||||||
_require_text(self.kv_dtype, "runtime_recipe.kv_dtype")
|
|
||||||
_require_text(self.kv_layout, "runtime_recipe.kv_layout")
|
|
||||||
_require_text(self.tokenizer_revision, "runtime_recipe.tokenizer_revision")
|
|
||||||
_require_text(self.architecture_adapter, "runtime_recipe.architecture_adapter")
|
|
||||||
_require_text(self.backend_id, "runtime_recipe.backend_id")
|
|
||||||
_require_text(self.runtime_version, "runtime_recipe.runtime_version")
|
|
||||||
_require_text(self.cache_layout, "runtime_recipe.cache_layout")
|
|
||||||
if self.boundary_schema_version < 1:
|
|
||||||
raise ValueError("'runtime_recipe.boundary_schema_version' must be >= 1")
|
|
||||||
expected = compatibility_fingerprint(self._fingerprint_payload())
|
|
||||||
if not self.fingerprint:
|
|
||||||
object.__setattr__(self, "fingerprint", expected)
|
|
||||||
elif self.fingerprint != expected:
|
|
||||||
raise ValueError(
|
|
||||||
"'runtime_recipe.fingerprint' does not match the encoded fields"
|
|
||||||
)
|
|
||||||
|
|
||||||
def to_dict(self) -> dict[str, Any]:
|
|
||||||
return {
|
|
||||||
"weight_quantization": self.weight_quantization,
|
|
||||||
"activation_dtype": self.activation_dtype,
|
|
||||||
"compute_dtype": self.compute_dtype,
|
|
||||||
"kv_dtype": self.kv_dtype,
|
|
||||||
"kv_layout": self.kv_layout,
|
|
||||||
"tokenizer_revision": self.tokenizer_revision,
|
|
||||||
"architecture_adapter": self.architecture_adapter,
|
|
||||||
"backend_id": self.backend_id,
|
|
||||||
"runtime_version": self.runtime_version,
|
|
||||||
"boundary_schema_version": self.boundary_schema_version,
|
|
||||||
"cache_layout": self.cache_layout,
|
|
||||||
"fingerprint": self.fingerprint,
|
|
||||||
}
|
|
||||||
|
|
||||||
@classmethod
|
|
||||||
def from_dict(cls, data: Any) -> "RuntimeRecipeIdentity":
|
|
||||||
if not isinstance(data, Mapping):
|
|
||||||
raise ValueError(
|
|
||||||
f"'runtime_recipe' must be a JSON object, got {type(data).__name__}"
|
|
||||||
)
|
|
||||||
boundary_schema_version = data.get("boundary_schema_version", 1)
|
|
||||||
if isinstance(boundary_schema_version, bool) or not isinstance(
|
|
||||||
boundary_schema_version, int
|
|
||||||
):
|
|
||||||
raise ValueError(
|
|
||||||
"'runtime_recipe.boundary_schema_version' must be an integer"
|
|
||||||
)
|
|
||||||
return cls(
|
|
||||||
weight_quantization=_require_text(
|
|
||||||
data.get("weight_quantization"), "runtime_recipe.weight_quantization"
|
|
||||||
),
|
|
||||||
activation_dtype=_require_text(
|
|
||||||
data.get("activation_dtype"), "runtime_recipe.activation_dtype"
|
|
||||||
),
|
|
||||||
compute_dtype=_require_text(
|
|
||||||
data.get("compute_dtype"), "runtime_recipe.compute_dtype"
|
|
||||||
),
|
|
||||||
kv_dtype=_require_text(data.get("kv_dtype"), "runtime_recipe.kv_dtype"),
|
|
||||||
kv_layout=_require_text(data.get("kv_layout"), "runtime_recipe.kv_layout"),
|
|
||||||
tokenizer_revision=_require_text(
|
|
||||||
data.get("tokenizer_revision"), "runtime_recipe.tokenizer_revision"
|
|
||||||
),
|
|
||||||
architecture_adapter=_require_text(
|
|
||||||
data.get("architecture_adapter"),
|
|
||||||
"runtime_recipe.architecture_adapter",
|
|
||||||
),
|
|
||||||
backend_id=_require_text(data.get("backend_id"), "runtime_recipe.backend_id"),
|
|
||||||
runtime_version=_require_text(
|
|
||||||
data.get("runtime_version"), "runtime_recipe.runtime_version"
|
|
||||||
),
|
|
||||||
boundary_schema_version=boundary_schema_version,
|
|
||||||
cache_layout=_require_text(data.get("cache_layout"), "runtime_recipe.cache_layout"),
|
|
||||||
fingerprint=_optional_text(data.get("fingerprint"), "runtime_recipe.fingerprint"),
|
|
||||||
)
|
|
||||||
|
|
||||||
def _fingerprint_payload(self) -> dict[str, Any]:
|
|
||||||
return {
|
|
||||||
"weight_quantization": self.weight_quantization,
|
|
||||||
"activation_dtype": self.activation_dtype,
|
|
||||||
"compute_dtype": self.compute_dtype,
|
|
||||||
"kv_dtype": self.kv_dtype,
|
|
||||||
"kv_layout": self.kv_layout,
|
|
||||||
"tokenizer_revision": self.tokenizer_revision,
|
|
||||||
"architecture_adapter": self.architecture_adapter,
|
|
||||||
"backend_id": self.backend_id,
|
|
||||||
"runtime_version": self.runtime_version,
|
|
||||||
"boundary_schema_version": self.boundary_schema_version,
|
|
||||||
"cache_layout": self.cache_layout,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def _optional_int(value: Any, field_name: str) -> int | None:
|
|
||||||
if value is None:
|
|
||||||
return None
|
|
||||||
if isinstance(value, bool) or not isinstance(value, int):
|
|
||||||
raise ValueError(f"{field_name!r} must be an integer")
|
|
||||||
if value < 0:
|
|
||||||
raise ValueError(f"{field_name!r} must be >= 0")
|
|
||||||
return value
|
|
||||||
|
|
||||||
|
|
||||||
def build_artifact_identity(
|
|
||||||
*,
|
|
||||||
model_id: str,
|
|
||||||
revision: str | None = None,
|
|
||||||
model_config: Any = None,
|
|
||||||
artifact_hash: str | None = None,
|
|
||||||
shard_start: int | None = None,
|
|
||||||
shard_end: int | None = None,
|
|
||||||
) -> ArtifactIdentity:
|
|
||||||
"""Build a stable artifact binding from the locally loaded artifact."""
|
|
||||||
resolved_hash = artifact_hash
|
|
||||||
if resolved_hash is None:
|
|
||||||
if isinstance(model_config, Mapping):
|
|
||||||
resolved_hash = _hash_mapping(model_config)
|
|
||||||
elif model_config is not None:
|
|
||||||
resolved_hash = _sha256_text(_stable_json(model_config))
|
|
||||||
if resolved_hash is None:
|
|
||||||
resolved_hash = _sha256_text(
|
|
||||||
_stable_json(
|
|
||||||
{
|
|
||||||
"model_id": model_id,
|
|
||||||
"revision": revision,
|
|
||||||
"shard_start": shard_start,
|
|
||||||
"shard_end": shard_end,
|
|
||||||
}
|
|
||||||
)
|
|
||||||
)
|
|
||||||
return ArtifactIdentity(
|
|
||||||
model_id=model_id,
|
|
||||||
revision=revision,
|
|
||||||
artifact_hash=resolved_hash,
|
|
||||||
shard_start=shard_start,
|
|
||||||
shard_end=shard_end,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def build_runtime_recipe_identity(
|
|
||||||
*,
|
|
||||||
model_id: str,
|
|
||||||
weight_quantization: str,
|
|
||||||
backend_id: str,
|
|
||||||
runtime_version: str,
|
|
||||||
revision: str | None = None,
|
|
||||||
model_config: Any = None,
|
|
||||||
recipe_params: Mapping[str, Any] | None = None,
|
|
||||||
activation_dtype: Any = None,
|
|
||||||
compute_dtype: Any = None,
|
|
||||||
kv_dtype: Any = None,
|
|
||||||
kv_layout: str | None = None,
|
|
||||||
tokenizer_revision: str | None = None,
|
|
||||||
architecture_adapter: str | None = None,
|
|
||||||
boundary_schema_version: int = 1,
|
|
||||||
cache_layout: str | None = None,
|
|
||||||
) -> RuntimeRecipeIdentity:
|
|
||||||
"""Build the exact runtime recipe used for compatibility admission."""
|
|
||||||
activation = _normalise_dtype(activation_dtype, "bfloat16")
|
|
||||||
compute = _normalise_dtype(compute_dtype, activation)
|
|
||||||
kv_dtype_text = _normalise_dtype(kv_dtype, compute)
|
|
||||||
kv_layout_text = kv_layout or "session-cache"
|
|
||||||
tokenizer = tokenizer_revision or _tokenizer_revision_from_config(
|
|
||||||
model_id, revision, model_config
|
|
||||||
)
|
|
||||||
architecture = architecture_adapter or _architecture_adapter_from_config(
|
|
||||||
model_config, backend_id
|
|
||||||
)
|
|
||||||
cache_layout_text = cache_layout or _cache_layout_from_recipe_params(recipe_params)
|
|
||||||
return RuntimeRecipeIdentity(
|
|
||||||
weight_quantization=weight_quantization,
|
|
||||||
activation_dtype=activation,
|
|
||||||
compute_dtype=compute,
|
|
||||||
kv_dtype=kv_dtype_text,
|
|
||||||
kv_layout=kv_layout_text,
|
|
||||||
tokenizer_revision=tokenizer,
|
|
||||||
architecture_adapter=architecture,
|
|
||||||
backend_id=backend_id,
|
|
||||||
runtime_version=runtime_version,
|
|
||||||
boundary_schema_version=boundary_schema_version,
|
|
||||||
cache_layout=cache_layout_text,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def compatibility_fingerprint(data: Mapping[str, Any]) -> str:
|
|
||||||
"""Return a stable SHA256 compatibility fingerprint for an exact route."""
|
|
||||||
return "sha256:" + _sha256_text(_stable_json(data))
|
|
||||||
|
|
||||||
|
|
||||||
def fingerprint_payload(
|
|
||||||
*,
|
|
||||||
model: Mapping[str, Any],
|
|
||||||
shard: Mapping[str, Any],
|
|
||||||
recipe: Mapping[str, Any],
|
|
||||||
backend: Mapping[str, Any],
|
|
||||||
artifact: Mapping[str, Any],
|
|
||||||
runtime_recipe: Mapping[str, Any],
|
|
||||||
) -> dict[str, Any]:
|
|
||||||
return {
|
|
||||||
"model": dict(model),
|
|
||||||
"shard": dict(shard),
|
|
||||||
"recipe": dict(recipe),
|
|
||||||
"backend": dict(backend),
|
|
||||||
"artifact": dict(artifact),
|
|
||||||
"runtime_recipe": dict(runtime_recipe),
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def _hash_mapping(data: Mapping[str, Any]) -> str:
|
|
||||||
return "sha256:" + _sha256_text(_stable_json(data))
|
|
||||||
@@ -12,7 +12,7 @@ import urllib.error
|
|||||||
import urllib.parse
|
import urllib.parse
|
||||||
import urllib.request
|
import urllib.request
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Any, Callable
|
from typing import Any
|
||||||
|
|
||||||
from .admission import (
|
from .admission import (
|
||||||
AdmissionRequirement,
|
AdmissionRequirement,
|
||||||
@@ -29,7 +29,6 @@ from .model_catalog import model_metadata_for
|
|||||||
from .recipe_manifest import DEFAULT_RECIPE_ID, Recipe, RecipeManifest, load_recipe_manifest
|
from .recipe_manifest import DEFAULT_RECIPE_ID, Recipe, RecipeManifest, load_recipe_manifest
|
||||||
from .relay_bridge import RelayHttpBridge, peer_id_from_wallet
|
from .relay_bridge import RelayHttpBridge, peer_id_from_wallet
|
||||||
from .server import StubNodeServer
|
from .server import StubNodeServer
|
||||||
from .gguf_backend import build_gguf_backend
|
|
||||||
from .torch_server import TorchNodeServer
|
from .torch_server import TorchNodeServer
|
||||||
from .wallet import load_or_create_wallet
|
from .wallet import load_or_create_wallet
|
||||||
|
|
||||||
@@ -420,7 +419,6 @@ def _start_heartbeat(
|
|||||||
interval: float = _HEARTBEAT_INTERVAL_IDLE,
|
interval: float = _HEARTBEAT_INTERVAL_IDLE,
|
||||||
node_ref: Any | None = None,
|
node_ref: Any | None = None,
|
||||||
start_time: float | None = None,
|
start_time: float | None = None,
|
||||||
refresh_capability: Callable[[dict], dict | None] | None = None,
|
|
||||||
) -> threading.Thread:
|
) -> threading.Thread:
|
||||||
"""Daemon thread: sends heartbeats and re-registers automatically after tracker restarts.
|
"""Daemon thread: sends heartbeats and re-registers automatically after tracker restarts.
|
||||||
|
|
||||||
@@ -432,7 +430,6 @@ def _start_heartbeat(
|
|||||||
which is logged for now (hot-reload implemented in US-026).
|
which is logged for now (hot-reload implemented in US-026).
|
||||||
"""
|
"""
|
||||||
_start_time = start_time or time.monotonic()
|
_start_time = start_time or time.monotonic()
|
||||||
completed_directives: list[dict] = []
|
|
||||||
|
|
||||||
def _current_requests_snapshot() -> list[dict]:
|
def _current_requests_snapshot() -> list[dict]:
|
||||||
if node_ref is None:
|
if node_ref is None:
|
||||||
@@ -457,8 +454,6 @@ def _start_heartbeat(
|
|||||||
current_requests = _current_requests_snapshot()
|
current_requests = _current_requests_snapshot()
|
||||||
if current_requests:
|
if current_requests:
|
||||||
stats["current_requests"] = current_requests
|
stats["current_requests"] = current_requests
|
||||||
if completed_directives:
|
|
||||||
stats["completed_directives"] = list(completed_directives)
|
|
||||||
return stats
|
return stats
|
||||||
|
|
||||||
def _sleep_interval() -> float:
|
def _sleep_interval() -> float:
|
||||||
@@ -466,26 +461,9 @@ def _start_heartbeat(
|
|||||||
return _HEARTBEAT_INTERVAL_BUSY
|
return _HEARTBEAT_INTERVAL_BUSY
|
||||||
return interval
|
return interval
|
||||||
|
|
||||||
def _refresh_proof(payload: dict) -> None:
|
|
||||||
"""Re-prove the current shard so a re-registration never presents an aged proof.
|
|
||||||
|
|
||||||
The tracker refuses proofs older than its freshness budget: re-sending the
|
|
||||||
startup-time report after an outage would re-register the node unroutable.
|
|
||||||
"""
|
|
||||||
if refresh_capability is None or "capability_report" not in payload:
|
|
||||||
return
|
|
||||||
try:
|
|
||||||
fresh = refresh_capability(payload)
|
|
||||||
except Exception as exc:
|
|
||||||
print(f" [node] WARNING: capability re-validation failed: {exc}", flush=True)
|
|
||||||
return
|
|
||||||
if fresh:
|
|
||||||
payload["capability_report"] = fresh
|
|
||||||
|
|
||||||
def _reregister() -> bool:
|
def _reregister() -> bool:
|
||||||
nonlocal node_id
|
nonlocal node_id
|
||||||
try:
|
try:
|
||||||
_refresh_proof(register_payload)
|
|
||||||
resp = _post_json(f"{tracker_url}/v1/nodes/register", register_payload)
|
resp = _post_json(f"{tracker_url}/v1/nodes/register", register_payload)
|
||||||
node_id = resp.get("node_id", node_id)
|
node_id = resp.get("node_id", node_id)
|
||||||
if node_ref is not None:
|
if node_ref is not None:
|
||||||
@@ -507,7 +485,6 @@ def _start_heartbeat(
|
|||||||
"managed_assignment": True,
|
"managed_assignment": True,
|
||||||
}
|
}
|
||||||
try:
|
try:
|
||||||
_refresh_proof(extra_payload)
|
|
||||||
reg_resp = _post_json(f"{tracker_url}/v1/nodes/register", extra_payload)
|
reg_resp = _post_json(f"{tracker_url}/v1/nodes/register", extra_payload)
|
||||||
print(
|
print(
|
||||||
f" [node] registered additional model — node ID: {reg_resp.get('node_id')}",
|
f" [node] registered additional model — node ID: {reg_resp.get('node_id')}",
|
||||||
@@ -516,26 +493,21 @@ def _start_heartbeat(
|
|||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
print(f" [node] WARNING: additional model registration failed: {exc}", flush=True)
|
print(f" [node] WARNING: additional model registration failed: {exc}", flush=True)
|
||||||
|
|
||||||
def _apply_directives(directives: list[dict]) -> dict | None:
|
def _apply_directives(directives: list[dict]) -> None:
|
||||||
if not directives:
|
if not directives:
|
||||||
return None
|
return
|
||||||
if node_ref is None or not hasattr(node_ref, "apply_tracker_directives"):
|
if node_ref is None or not hasattr(node_ref, "apply_tracker_directives"):
|
||||||
print(f" [node] tracker directives received: {directives}", flush=True)
|
print(f" [node] tracker directives received: {directives}", flush=True)
|
||||||
return None
|
return
|
||||||
try:
|
try:
|
||||||
applied = node_ref.apply_tracker_directives(directives)
|
applied = node_ref.apply_tracker_directives(directives)
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
print(f" [node] WARNING: failed to apply tracker directives: {exc}", flush=True)
|
print(f" [node] WARNING: failed to apply tracker directives: {exc}", flush=True)
|
||||||
return None
|
return
|
||||||
if applied:
|
if applied:
|
||||||
completed_directives.append(dict(applied))
|
|
||||||
if applied.get("action") == "ADD_SHARD":
|
if applied.get("action") == "ADD_SHARD":
|
||||||
_register_additional_assignment(applied)
|
_register_additional_assignment(applied)
|
||||||
return applied
|
return
|
||||||
if applied.get("action") in {"DROP_SHARD", "DROP_ALL_SHARDS"}:
|
|
||||||
# A release has no replacement range. It is not a failed
|
|
||||||
# heartbeat and must not re-register the released assignment.
|
|
||||||
return applied
|
|
||||||
model_id = applied.get("model", register_payload.get("hf_repo") or register_payload.get("model"))
|
model_id = applied.get("model", register_payload.get("hf_repo") or register_payload.get("model"))
|
||||||
register_payload["model"] = str(model_id).split("/")[-1]
|
register_payload["model"] = str(model_id).split("/")[-1]
|
||||||
register_payload["hf_repo"] = model_id
|
register_payload["hf_repo"] = model_id
|
||||||
@@ -543,7 +515,6 @@ def _start_heartbeat(
|
|||||||
register_payload["shard_end"] = applied["shard_end"]
|
register_payload["shard_end"] = applied["shard_end"]
|
||||||
register_payload["quantization"] = applied.get("quantization", register_payload.get("quantization"))
|
register_payload["quantization"] = applied.get("quantization", register_payload.get("quantization"))
|
||||||
register_payload["tracker_mode"] = bool(applied.get("tracker_mode", False))
|
register_payload["tracker_mode"] = bool(applied.get("tracker_mode", False))
|
||||||
return applied
|
|
||||||
|
|
||||||
def _loop() -> None:
|
def _loop() -> None:
|
||||||
nonlocal node_id
|
nonlocal node_id
|
||||||
@@ -571,10 +542,7 @@ def _start_heartbeat(
|
|||||||
continue
|
continue
|
||||||
|
|
||||||
try:
|
try:
|
||||||
heartbeat = _get_stats()
|
resp = _post_json(hb_url, _get_stats())
|
||||||
resp = _post_json(hb_url, heartbeat)
|
|
||||||
if heartbeat.get("completed_directives"):
|
|
||||||
completed_directives.clear()
|
|
||||||
_apply_directives(resp.get("directives", []))
|
_apply_directives(resp.get("directives", []))
|
||||||
new_asgn = resp.get("new_assignment")
|
new_asgn = resp.get("new_assignment")
|
||||||
if new_asgn:
|
if new_asgn:
|
||||||
@@ -611,7 +579,6 @@ def _register_with_tracker(
|
|||||||
reg_payload: dict,
|
reg_payload: dict,
|
||||||
node: Any,
|
node: Any,
|
||||||
start_time: float,
|
start_time: float,
|
||||||
refresh_capability: Callable[[dict], dict | None] | None = None,
|
|
||||||
) -> str | None:
|
) -> str | None:
|
||||||
"""Register with the tracker, or start background retries when it is unreachable."""
|
"""Register with the tracker, or start background retries when it is unreachable."""
|
||||||
try:
|
try:
|
||||||
@@ -619,14 +586,7 @@ def _register_with_tracker(
|
|||||||
tracker_node_id = str(reg_resp.get("node_id") or "?")
|
tracker_node_id = str(reg_resp.get("node_id") or "?")
|
||||||
setattr(node, "tracker_node_id", tracker_node_id)
|
setattr(node, "tracker_node_id", tracker_node_id)
|
||||||
print(f" Registered with tracker — node ID: {tracker_node_id}", flush=True)
|
print(f" Registered with tracker — node ID: {tracker_node_id}", flush=True)
|
||||||
_start_heartbeat(
|
_start_heartbeat(tracker_url, tracker_node_id, reg_payload, node_ref=node, start_time=start_time)
|
||||||
tracker_url,
|
|
||||||
tracker_node_id,
|
|
||||||
reg_payload,
|
|
||||||
node_ref=node,
|
|
||||||
start_time=start_time,
|
|
||||||
refresh_capability=refresh_capability,
|
|
||||||
)
|
|
||||||
return tracker_node_id
|
return tracker_node_id
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
setattr(node, "tracker_node_id", None)
|
setattr(node, "tracker_node_id", None)
|
||||||
@@ -638,7 +598,6 @@ def _register_with_tracker(
|
|||||||
reg_payload,
|
reg_payload,
|
||||||
node_ref=node,
|
node_ref=node,
|
||||||
start_time=start_time,
|
start_time=start_time,
|
||||||
refresh_capability=refresh_capability,
|
|
||||||
)
|
)
|
||||||
return None
|
return None
|
||||||
|
|
||||||
@@ -703,35 +662,6 @@ def _resolve_recipe(recipe_id: str | None) -> tuple[RecipeManifest, Recipe]:
|
|||||||
return manifest, manifest.require(recipe_id or DEFAULT_RECIPE_ID)
|
return manifest, manifest.require(recipe_id or DEFAULT_RECIPE_ID)
|
||||||
|
|
||||||
|
|
||||||
def _gguf_backend_for_recipe(
|
|
||||||
recipe: Recipe,
|
|
||||||
*,
|
|
||||||
model_id: str,
|
|
||||||
shard_start: int,
|
|
||||||
shard_end: int,
|
|
||||||
quantization: str,
|
|
||||||
total_layers: int | None,
|
|
||||||
device: str,
|
|
||||||
model_revision: str | None = None,
|
|
||||||
) -> object | None:
|
|
||||||
"""Build the GGUF backend only for recipes that explicitly ask for it."""
|
|
||||||
if recipe.backend_id != "llama.cpp":
|
|
||||||
return None
|
|
||||||
return build_gguf_backend(
|
|
||||||
model_id=model_id,
|
|
||||||
shard_start=shard_start,
|
|
||||||
shard_end=shard_end,
|
|
||||||
quantization=quantization,
|
|
||||||
total_layers=total_layers,
|
|
||||||
model_revision=model_revision,
|
|
||||||
device_type=device,
|
|
||||||
architecture_adapter="dense-llama",
|
|
||||||
tokenizer_revision=model_revision or model_id,
|
|
||||||
runtime_recipe_fingerprint=None,
|
|
||||||
supports_kv_cache=recipe.params.get("use_cache", True) is not False,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _capability_device(backend: Any, detected_device: str) -> str:
|
def _capability_device(backend: Any, detected_device: str) -> str:
|
||||||
"""The device the shard actually landed on, or the one this node detected."""
|
"""The device the shard actually landed on, or the one this node detected."""
|
||||||
device = getattr(backend, "device", None)
|
device = getattr(backend, "device", None)
|
||||||
@@ -788,54 +718,6 @@ def _admit_capability(
|
|||||||
return report
|
return report
|
||||||
|
|
||||||
|
|
||||||
def _capability_refresher(
|
|
||||||
node: Any,
|
|
||||||
*,
|
|
||||||
manifest: RecipeManifest,
|
|
||||||
recipe: Recipe,
|
|
||||||
detected_device: str,
|
|
||||||
cache_dir: Path | None,
|
|
||||||
force_cpu: bool,
|
|
||||||
validator: CapabilityValidator | None = None,
|
|
||||||
) -> Callable[[dict], dict | None]:
|
|
||||||
"""A fresh proof for what the node serves *now*, run at re-registration time.
|
|
||||||
|
|
||||||
The startup proof ages past the tracker's freshness budget, and directives
|
|
||||||
can move the node to a shard the startup proof never covered — so every
|
|
||||||
re-registration re-proves against the currently loaded backend rather than
|
|
||||||
replaying the report captured at boot.
|
|
||||||
"""
|
|
||||||
def refresh(payload: dict) -> dict | None:
|
|
||||||
target_model = payload.get("hf_repo") or payload.get("model")
|
|
||||||
backend = None
|
|
||||||
accessor = getattr(node, "backend_for", None)
|
|
||||||
if callable(accessor) and target_model:
|
|
||||||
backend = accessor(str(target_model))
|
|
||||||
if backend is None:
|
|
||||||
backend = getattr(node, "backend", None)
|
|
||||||
if backend is None:
|
|
||||||
return None
|
|
||||||
context = CapabilityContext(
|
|
||||||
backend=backend,
|
|
||||||
selection=DoctorSelection(
|
|
||||||
model_id=str(getattr(backend, "model_id", target_model)),
|
|
||||||
shard_start=int(getattr(backend, "shard_start", 0) or 0),
|
|
||||||
shard_end=int(getattr(backend, "shard_end", 0) or 0),
|
|
||||||
quantization=str(getattr(backend, "quantization", None) or "auto"),
|
|
||||||
cache_dir=cache_dir,
|
|
||||||
force_cpu=force_cpu,
|
|
||||||
),
|
|
||||||
recipe=recipe,
|
|
||||||
manifest=manifest,
|
|
||||||
device=_capability_device(backend, detected_device),
|
|
||||||
)
|
|
||||||
report = (validator or probe_capability)(context)
|
|
||||||
setattr(node, "capability_report", report)
|
|
||||||
return report.to_dict()
|
|
||||||
|
|
||||||
return refresh
|
|
||||||
|
|
||||||
|
|
||||||
def run_startup(
|
def run_startup(
|
||||||
tracker_url: str,
|
tracker_url: str,
|
||||||
port: int = 0,
|
port: int = 0,
|
||||||
@@ -993,8 +875,7 @@ def run_startup(
|
|||||||
|
|
||||||
if model_id: # treat "" the same as None — no explicit model given
|
if model_id: # treat "" the same as None — no explicit model given
|
||||||
full_sources: list[dict] = []
|
full_sources: list[dict] = []
|
||||||
detected: int | None = None
|
# Auto-detect shard range from model config if not explicitly provided
|
||||||
# Auto-detect shard range from model config if not explicitly provided.
|
|
||||||
if shard_start is None or shard_end is None:
|
if shard_start is None or shard_end is None:
|
||||||
try:
|
try:
|
||||||
detected = _detect_num_layers(model_id, cache_dir=cache_dir)
|
detected = _detect_num_layers(model_id, cache_dir=cache_dir)
|
||||||
@@ -1058,38 +939,22 @@ def run_startup(
|
|||||||
shard_end = shard_end if shard_end is not None else detected - 1
|
shard_end = shard_end if shard_end is not None else detected - 1
|
||||||
print(f" Auto-detected {detected} layers → shard {shard_start}–{shard_end}", flush=True)
|
print(f" Auto-detected {detected} layers → shard {shard_start}–{shard_end}", flush=True)
|
||||||
|
|
||||||
backend = _gguf_backend_for_recipe(
|
print("Loading real PyTorch model shard...", flush=True)
|
||||||
recipe,
|
node = TorchNodeServer(
|
||||||
|
host=host,
|
||||||
|
port=port,
|
||||||
model_id=model_id,
|
model_id=model_id,
|
||||||
shard_start=shard_start,
|
shard_start=shard_start,
|
||||||
shard_end=shard_end,
|
shard_end=shard_end,
|
||||||
quantization=quantization,
|
quantization=quantization,
|
||||||
total_layers=detected if detected is not None else (shard_end + 1 if shard_end is not None else None),
|
tracker_url=tracker_url,
|
||||||
device=device,
|
route_timeout=route_timeout,
|
||||||
model_revision=None,
|
cache_dir=cache_dir,
|
||||||
|
debug=debug,
|
||||||
|
max_loaded_shards=max_loaded_shards,
|
||||||
|
force_cpu=force_cpu,
|
||||||
|
recipe_params=recipe.params,
|
||||||
)
|
)
|
||||||
print(
|
|
||||||
"Loading native llama.cpp model shard..." if backend is not None else "Loading real PyTorch model shard...",
|
|
||||||
flush=True,
|
|
||||||
)
|
|
||||||
node_kwargs = {
|
|
||||||
"host": host,
|
|
||||||
"port": port,
|
|
||||||
"model_id": model_id,
|
|
||||||
"shard_start": shard_start,
|
|
||||||
"shard_end": shard_end,
|
|
||||||
"quantization": quantization,
|
|
||||||
"tracker_url": tracker_url,
|
|
||||||
"route_timeout": route_timeout,
|
|
||||||
"cache_dir": cache_dir,
|
|
||||||
"debug": debug,
|
|
||||||
"max_loaded_shards": max_loaded_shards,
|
|
||||||
"force_cpu": force_cpu,
|
|
||||||
"recipe_params": recipe.params,
|
|
||||||
}
|
|
||||||
if backend is not None:
|
|
||||||
node_kwargs["backend"] = backend
|
|
||||||
node = TorchNodeServer(**node_kwargs)
|
|
||||||
capability_report = _admit_capability(
|
capability_report = _admit_capability(
|
||||||
node,
|
node,
|
||||||
model_id=model_id,
|
model_id=model_id,
|
||||||
@@ -1103,15 +968,10 @@ def run_startup(
|
|||||||
recipe=recipe,
|
recipe=recipe,
|
||||||
validator=capability_validator,
|
validator=capability_validator,
|
||||||
)
|
)
|
||||||
proof_shard = capability_report.shard
|
|
||||||
_node_start_time = time.monotonic()
|
_node_start_time = time.monotonic()
|
||||||
actual_port = node.start()
|
actual_port = node.start()
|
||||||
total_layers = getattr(getattr(node, "backend", None), "total_layers", None)
|
total_layers = getattr(getattr(node, "backend", None), "total_layers", None)
|
||||||
shard_label = _format_shard_label(
|
shard_label = _format_shard_label(shard_start, shard_end, total_layers)
|
||||||
proof_shard.start,
|
|
||||||
proof_shard.end,
|
|
||||||
total_layers,
|
|
||||||
)
|
|
||||||
public_host = advertise_host or (socket.getfqdn() if host == "0.0.0.0" else host)
|
public_host = advertise_host or (socket.getfqdn() if host == "0.0.0.0" else host)
|
||||||
endpoint = f"http://{public_host}:{actual_port}"
|
endpoint = f"http://{public_host}:{actual_port}"
|
||||||
if hasattr(node, "set_advertised_endpoint"):
|
if hasattr(node, "set_advertised_endpoint"):
|
||||||
@@ -1134,17 +994,16 @@ def run_startup(
|
|||||||
"model": model_id.split("/")[-1],
|
"model": model_id.split("/")[-1],
|
||||||
"hf_repo": model_id,
|
"hf_repo": model_id,
|
||||||
"num_layers": total_layers,
|
"num_layers": total_layers,
|
||||||
"shard_start": proof_shard.start,
|
"shard_start": shard_start,
|
||||||
"shard_end": proof_shard.end,
|
"shard_end": shard_end,
|
||||||
"hardware_profile": hw,
|
"hardware_profile": hw,
|
||||||
"wallet_address": address,
|
"wallet_address": address,
|
||||||
"quantization": quantization,
|
"quantization": quantization,
|
||||||
"score": 1.0,
|
"score": 1.0,
|
||||||
"tracker_mode": (proof_shard.start == 0),
|
"tracker_mode": (shard_start == 0),
|
||||||
"managed_assignment": not user_pinned_shard,
|
"managed_assignment": not user_pinned_shard,
|
||||||
"model_metadata": model_metadata_for(model_id, total_layers, cache_dir=cache_dir),
|
"model_metadata": model_metadata_for(model_id, total_layers, cache_dir=cache_dir),
|
||||||
"capability_report": capability_report.to_dict(),
|
"capability_report": capability_report.to_dict(),
|
||||||
"compatibility_fingerprint": capability_report.compatibility_fingerprint,
|
|
||||||
# Declared independently of the proof: the tracker checks that the
|
# Declared independently of the proof: the tracker checks that the
|
||||||
# recipe this node says it serves with is the one the proof ran.
|
# recipe this node says it serves with is the one the proof ran.
|
||||||
"recipe_id": recipe.id,
|
"recipe_id": recipe.id,
|
||||||
@@ -1152,8 +1011,8 @@ def run_startup(
|
|||||||
"downloaded_models": (
|
"downloaded_models": (
|
||||||
_downloaded_model_inventory(
|
_downloaded_model_inventory(
|
||||||
model_id.split("/")[-1],
|
model_id.split("/")[-1],
|
||||||
proof_shard.start,
|
shard_start,
|
||||||
proof_shard.end,
|
shard_end,
|
||||||
model_cache_path,
|
model_cache_path,
|
||||||
hf_repo=model_id,
|
hf_repo=model_id,
|
||||||
model_sources=full_sources,
|
model_sources=full_sources,
|
||||||
@@ -1167,15 +1026,6 @@ def run_startup(
|
|||||||
}
|
}
|
||||||
tracker_node_id = _register_with_tracker(
|
tracker_node_id = _register_with_tracker(
|
||||||
tracker_url, reg_payload, node, _node_start_time,
|
tracker_url, reg_payload, node, _node_start_time,
|
||||||
refresh_capability=_capability_refresher(
|
|
||||||
node,
|
|
||||||
manifest=manifest,
|
|
||||||
recipe=recipe,
|
|
||||||
detected_device=device,
|
|
||||||
cache_dir=cache_dir,
|
|
||||||
force_cpu=force_cpu,
|
|
||||||
validator=capability_validator,
|
|
||||||
),
|
|
||||||
)
|
)
|
||||||
|
|
||||||
print(
|
print(
|
||||||
@@ -1264,38 +1114,22 @@ def run_startup(
|
|||||||
hf_repo=assigned_hf_repo,
|
hf_repo=assigned_hf_repo,
|
||||||
model_sources=full_sources,
|
model_sources=full_sources,
|
||||||
)
|
)
|
||||||
backend = _gguf_backend_for_recipe(
|
print("Loading real PyTorch model shard...", flush=True)
|
||||||
recipe,
|
node = TorchNodeServer(
|
||||||
|
host=host,
|
||||||
|
port=port,
|
||||||
model_id=assigned_hf_repo,
|
model_id=assigned_hf_repo,
|
||||||
shard_start=assigned_shard_start,
|
shard_start=assigned_shard_start,
|
||||||
shard_end=assigned_shard_end,
|
shard_end=assigned_shard_end,
|
||||||
quantization=quantization,
|
quantization=quantization,
|
||||||
total_layers=assigned_num_layers,
|
tracker_url=tracker_url,
|
||||||
device=device,
|
route_timeout=route_timeout,
|
||||||
model_revision=None,
|
cache_dir=cache_dir,
|
||||||
|
debug=debug,
|
||||||
|
max_loaded_shards=max_loaded_shards,
|
||||||
|
force_cpu=force_cpu,
|
||||||
|
recipe_params=recipe.params,
|
||||||
)
|
)
|
||||||
print(
|
|
||||||
"Loading native llama.cpp model shard..." if backend is not None else "Loading real PyTorch model shard...",
|
|
||||||
flush=True,
|
|
||||||
)
|
|
||||||
node_kwargs = {
|
|
||||||
"host": host,
|
|
||||||
"port": port,
|
|
||||||
"model_id": assigned_hf_repo,
|
|
||||||
"shard_start": assigned_shard_start,
|
|
||||||
"shard_end": assigned_shard_end,
|
|
||||||
"quantization": quantization,
|
|
||||||
"tracker_url": tracker_url,
|
|
||||||
"route_timeout": route_timeout,
|
|
||||||
"cache_dir": cache_dir,
|
|
||||||
"debug": debug,
|
|
||||||
"max_loaded_shards": max_loaded_shards,
|
|
||||||
"force_cpu": force_cpu,
|
|
||||||
"recipe_params": recipe.params,
|
|
||||||
}
|
|
||||||
if backend is not None:
|
|
||||||
node_kwargs["backend"] = backend
|
|
||||||
node = TorchNodeServer(**node_kwargs)
|
|
||||||
capability_report = _admit_capability(
|
capability_report = _admit_capability(
|
||||||
node,
|
node,
|
||||||
model_id=assigned_hf_repo,
|
model_id=assigned_hf_repo,
|
||||||
@@ -1309,7 +1143,6 @@ def run_startup(
|
|||||||
recipe=recipe,
|
recipe=recipe,
|
||||||
validator=capability_validator,
|
validator=capability_validator,
|
||||||
)
|
)
|
||||||
proof_shard = capability_report.shard
|
|
||||||
_node_start_time = time.monotonic()
|
_node_start_time = time.monotonic()
|
||||||
actual_port = node.start()
|
actual_port = node.start()
|
||||||
public_host = advertise_host or (socket.getfqdn() if host == "0.0.0.0" else host)
|
public_host = advertise_host or (socket.getfqdn() if host == "0.0.0.0" else host)
|
||||||
@@ -1332,17 +1165,16 @@ def run_startup(
|
|||||||
"model": assigned_hf_repo.split("/")[-1],
|
"model": assigned_hf_repo.split("/")[-1],
|
||||||
"hf_repo": assigned_hf_repo,
|
"hf_repo": assigned_hf_repo,
|
||||||
"num_layers": assigned_num_layers,
|
"num_layers": assigned_num_layers,
|
||||||
"shard_start": proof_shard.start,
|
"shard_start": assigned_shard_start,
|
||||||
"shard_end": proof_shard.end,
|
"shard_end": assigned_shard_end,
|
||||||
"hardware_profile": hw,
|
"hardware_profile": hw,
|
||||||
"wallet_address": address,
|
"wallet_address": address,
|
||||||
"quantization": quantization,
|
"quantization": quantization,
|
||||||
"score": 1.0,
|
"score": 1.0,
|
||||||
"tracker_mode": (proof_shard.start == 0),
|
"tracker_mode": (assigned_shard_start == 0),
|
||||||
"managed_assignment": True,
|
"managed_assignment": True,
|
||||||
"model_metadata": model_metadata_for(assigned_hf_repo, assigned_num_layers, cache_dir=cache_dir),
|
"model_metadata": model_metadata_for(assigned_hf_repo, assigned_num_layers, cache_dir=cache_dir),
|
||||||
"capability_report": capability_report.to_dict(),
|
"capability_report": capability_report.to_dict(),
|
||||||
"compatibility_fingerprint": capability_report.compatibility_fingerprint,
|
|
||||||
# Declared independently of the proof: the tracker checks that the
|
# Declared independently of the proof: the tracker checks that the
|
||||||
# recipe this node says it serves with is the one the proof ran.
|
# recipe this node says it serves with is the one the proof ran.
|
||||||
"recipe_id": recipe.id,
|
"recipe_id": recipe.id,
|
||||||
@@ -1350,8 +1182,8 @@ def run_startup(
|
|||||||
"downloaded_models": (
|
"downloaded_models": (
|
||||||
_downloaded_model_inventory(
|
_downloaded_model_inventory(
|
||||||
assigned_hf_repo.split("/")[-1],
|
assigned_hf_repo.split("/")[-1],
|
||||||
proof_shard.start,
|
assigned_shard_start,
|
||||||
proof_shard.end,
|
assigned_shard_end,
|
||||||
model_cache_path,
|
model_cache_path,
|
||||||
hf_repo=assigned_hf_repo,
|
hf_repo=assigned_hf_repo,
|
||||||
model_sources=full_sources,
|
model_sources=full_sources,
|
||||||
@@ -1365,19 +1197,10 @@ def run_startup(
|
|||||||
}
|
}
|
||||||
tracker_node_id = _register_with_tracker(
|
tracker_node_id = _register_with_tracker(
|
||||||
tracker_url, auto_reg_payload, node, _node_start_time,
|
tracker_url, auto_reg_payload, node, _node_start_time,
|
||||||
refresh_capability=_capability_refresher(
|
|
||||||
node,
|
|
||||||
manifest=manifest,
|
|
||||||
recipe=recipe,
|
|
||||||
detected_device=device,
|
|
||||||
cache_dir=cache_dir,
|
|
||||||
force_cpu=force_cpu,
|
|
||||||
validator=capability_validator,
|
|
||||||
),
|
|
||||||
)
|
)
|
||||||
shard_label = _format_shard_label(
|
shard_label = _format_shard_label(
|
||||||
proof_shard.start,
|
assigned_shard_start,
|
||||||
proof_shard.end,
|
assigned_shard_end,
|
||||||
assigned_num_layers,
|
assigned_num_layers,
|
||||||
)
|
)
|
||||||
print(
|
print(
|
||||||
@@ -1492,38 +1315,22 @@ def run_startup(
|
|||||||
# 5. Start HTTP server — real HF weights use TorchNodeServer; stub-model stays stub.
|
# 5. Start HTTP server — real HF weights use TorchNodeServer; stub-model stays stub.
|
||||||
_node_start_time = time.monotonic()
|
_node_start_time = time.monotonic()
|
||||||
if hf_repo and assigned_model != "stub-model":
|
if hf_repo and assigned_model != "stub-model":
|
||||||
backend = _gguf_backend_for_recipe(
|
print("Loading real PyTorch model shard...", flush=True)
|
||||||
recipe,
|
node = TorchNodeServer(
|
||||||
|
host=host,
|
||||||
|
port=port,
|
||||||
model_id=hf_repo,
|
model_id=hf_repo,
|
||||||
shard_start=shard_start,
|
shard_start=shard_start,
|
||||||
shard_end=shard_end,
|
shard_end=shard_end,
|
||||||
quantization=quantization,
|
quantization=quantization,
|
||||||
total_layers=total_layers,
|
tracker_url=tracker_url,
|
||||||
device=device,
|
route_timeout=route_timeout,
|
||||||
model_revision=None,
|
cache_dir=shard_path,
|
||||||
|
debug=debug,
|
||||||
|
max_loaded_shards=max_loaded_shards,
|
||||||
|
force_cpu=force_cpu,
|
||||||
|
recipe_params=recipe.params,
|
||||||
)
|
)
|
||||||
print(
|
|
||||||
"Loading native llama.cpp model shard..." if backend is not None else "Loading real PyTorch model shard...",
|
|
||||||
flush=True,
|
|
||||||
)
|
|
||||||
node_kwargs = {
|
|
||||||
"host": host,
|
|
||||||
"port": port,
|
|
||||||
"model_id": hf_repo,
|
|
||||||
"shard_start": shard_start,
|
|
||||||
"shard_end": shard_end,
|
|
||||||
"quantization": quantization,
|
|
||||||
"tracker_url": tracker_url,
|
|
||||||
"route_timeout": route_timeout,
|
|
||||||
"cache_dir": shard_path,
|
|
||||||
"debug": debug,
|
|
||||||
"max_loaded_shards": max_loaded_shards,
|
|
||||||
"force_cpu": force_cpu,
|
|
||||||
"recipe_params": recipe.params,
|
|
||||||
}
|
|
||||||
if backend is not None:
|
|
||||||
node_kwargs["backend"] = backend
|
|
||||||
node = TorchNodeServer(**node_kwargs)
|
|
||||||
capability_report = _admit_capability(
|
capability_report = _admit_capability(
|
||||||
node,
|
node,
|
||||||
model_id=hf_repo,
|
model_id=hf_repo,
|
||||||
@@ -1572,7 +1379,6 @@ def run_startup(
|
|||||||
"managed_assignment": not user_pinned_shard,
|
"managed_assignment": not user_pinned_shard,
|
||||||
"model_metadata": model_metadata_for(hf_repo, total_layers, cache_dir=shard_path),
|
"model_metadata": model_metadata_for(hf_repo, total_layers, cache_dir=shard_path),
|
||||||
"capability_report": capability_report.to_dict(),
|
"capability_report": capability_report.to_dict(),
|
||||||
"compatibility_fingerprint": capability_report.compatibility_fingerprint,
|
|
||||||
# Declared independently of the proof: the tracker checks that the
|
# Declared independently of the proof: the tracker checks that the
|
||||||
# recipe this node says it serves with is the one the proof ran.
|
# recipe this node says it serves with is the one the proof ran.
|
||||||
"recipe_id": recipe.id,
|
"recipe_id": recipe.id,
|
||||||
@@ -1583,15 +1389,6 @@ def run_startup(
|
|||||||
}
|
}
|
||||||
tracker_node_id = _register_with_tracker(
|
tracker_node_id = _register_with_tracker(
|
||||||
tracker_url, reg_payload, node, _node_start_time,
|
tracker_url, reg_payload, node, _node_start_time,
|
||||||
refresh_capability=_capability_refresher(
|
|
||||||
node,
|
|
||||||
manifest=manifest,
|
|
||||||
recipe=recipe,
|
|
||||||
detected_device=device,
|
|
||||||
cache_dir=cache_dir,
|
|
||||||
force_cpu=force_cpu,
|
|
||||||
validator=capability_validator,
|
|
||||||
),
|
|
||||||
)
|
)
|
||||||
print(
|
print(
|
||||||
f"\n{'=' * 32}\n"
|
f"\n{'=' * 32}\n"
|
||||||
@@ -1634,7 +1431,6 @@ def run_startup(
|
|||||||
recipe=recipe,
|
recipe=recipe,
|
||||||
validator=capability_validator,
|
validator=capability_validator,
|
||||||
)
|
)
|
||||||
proof_shard = capability_report.shard
|
|
||||||
actual_port = node.start()
|
actual_port = node.start()
|
||||||
public_host = advertise_host or (socket.getfqdn() if host == "0.0.0.0" else host)
|
public_host = advertise_host or (socket.getfqdn() if host == "0.0.0.0" else host)
|
||||||
endpoint = f"http://{public_host}:{actual_port}"
|
endpoint = f"http://{public_host}:{actual_port}"
|
||||||
@@ -1654,11 +1450,10 @@ def run_startup(
|
|||||||
reg_payload = {
|
reg_payload = {
|
||||||
"endpoint": endpoint,
|
"endpoint": endpoint,
|
||||||
"model": assigned_model,
|
"model": assigned_model,
|
||||||
"shard_start": proof_shard.start,
|
"shard_start": shard_start,
|
||||||
"shard_end": proof_shard.end,
|
"shard_end": shard_end,
|
||||||
"shard_checksum": shard_checksum,
|
"shard_checksum": shard_checksum,
|
||||||
"capability_report": capability_report.to_dict(),
|
"capability_report": capability_report.to_dict(),
|
||||||
"compatibility_fingerprint": capability_report.compatibility_fingerprint,
|
|
||||||
# Declared independently of the proof: the tracker checks that the
|
# Declared independently of the proof: the tracker checks that the
|
||||||
# recipe this node says it serves with is the one the proof ran.
|
# recipe this node says it serves with is the one the proof ran.
|
||||||
"recipe_id": recipe.id,
|
"recipe_id": recipe.id,
|
||||||
@@ -1679,22 +1474,7 @@ def run_startup(
|
|||||||
)
|
)
|
||||||
node_id = str(reg_resp["node_id"])
|
node_id = str(reg_resp["node_id"])
|
||||||
setattr(node, "tracker_node_id", node_id)
|
setattr(node, "tracker_node_id", node_id)
|
||||||
_start_heartbeat(
|
_start_heartbeat(tracker_url, node_id, reg_payload, node_ref=node, start_time=_node_start_time)
|
||||||
tracker_url,
|
|
||||||
node_id,
|
|
||||||
reg_payload,
|
|
||||||
node_ref=node,
|
|
||||||
start_time=_node_start_time,
|
|
||||||
refresh_capability=_capability_refresher(
|
|
||||||
node,
|
|
||||||
manifest=manifest,
|
|
||||||
recipe=recipe,
|
|
||||||
detected_device=device,
|
|
||||||
cache_dir=shard_path,
|
|
||||||
force_cpu=force_cpu,
|
|
||||||
validator=capability_validator,
|
|
||||||
),
|
|
||||||
)
|
|
||||||
except Exception:
|
except Exception:
|
||||||
node.stop()
|
node.stop()
|
||||||
raise
|
raise
|
||||||
@@ -1704,8 +1484,8 @@ def run_startup(
|
|||||||
if gpu_name:
|
if gpu_name:
|
||||||
hw_str += f" ({gpu_name}, {vram_mb / 1024:.1f} GB)"
|
hw_str += f" ({gpu_name}, {vram_mb / 1024:.1f} GB)"
|
||||||
shard_label = _format_shard_label(
|
shard_label = _format_shard_label(
|
||||||
proof_shard.start,
|
shard_start,
|
||||||
proof_shard.end,
|
shard_end,
|
||||||
assigned_total_layers,
|
assigned_total_layers,
|
||||||
model_name=assigned_model,
|
model_name=assigned_model,
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -16,10 +16,7 @@ import time
|
|||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
from .admission import CapabilityContext, CapabilityValidator
|
from .admission import CapabilityContext, CapabilityValidator
|
||||||
from . import __version__ as _PACKAGE_VERSION
|
|
||||||
from .capability import STATUS_PASSED, CapabilityReport, build_capability_report
|
from .capability import STATUS_PASSED, CapabilityReport, build_capability_report
|
||||||
from .gguf_ownership import authoritative_dense_llama_ownership
|
|
||||||
from .runtime_recipe import build_runtime_recipe_identity
|
|
||||||
|
|
||||||
|
|
||||||
def capability_report_for(
|
def capability_report_for(
|
||||||
@@ -33,15 +30,6 @@ def capability_report_for(
|
|||||||
recipe_version: str | None = None,
|
recipe_version: str | None = None,
|
||||||
backend_id: str | None = None,
|
backend_id: str | None = None,
|
||||||
device: str | None = None,
|
device: str | None = None,
|
||||||
artifact_hash: str | None = None,
|
|
||||||
activation_dtype: str | None = None,
|
|
||||||
compute_dtype: str | None = None,
|
|
||||||
kv_dtype: str | None = None,
|
|
||||||
kv_layout: str | None = None,
|
|
||||||
tokenizer_revision: str | None = None,
|
|
||||||
architecture_adapter: str | None = None,
|
|
||||||
boundary_schema_version: int = 1,
|
|
||||||
cache_layout: str | None = None,
|
|
||||||
validated_at: float | None = None,
|
validated_at: float | None = None,
|
||||||
age_seconds: float = 0.0,
|
age_seconds: float = 0.0,
|
||||||
diagnostics: Any = None,
|
diagnostics: Any = None,
|
||||||
@@ -49,49 +37,18 @@ def capability_report_for(
|
|||||||
) -> CapabilityReport:
|
) -> CapabilityReport:
|
||||||
"""A report describing `context`, with any field bent away from the truth."""
|
"""A report describing `context`, with any field bent away from the truth."""
|
||||||
now = time.time() if validated_at is None else validated_at
|
now = time.time() if validated_at is None else validated_at
|
||||||
backend = getattr(context, "backend", None)
|
|
||||||
model_config = getattr(getattr(backend, "model", None), "config", None)
|
|
||||||
model_config_payload = (
|
|
||||||
model_config.to_dict() if hasattr(model_config, "to_dict") else model_config
|
|
||||||
)
|
|
||||||
resolved_cache_layout = (
|
|
||||||
"stateless"
|
|
||||||
if getattr(backend, "supports_kv_cache", False) is False
|
|
||||||
else "local-hot-kv"
|
|
||||||
)
|
|
||||||
ownership = authoritative_dense_llama_ownership(backend, context.selection)
|
|
||||||
runtime_recipe = build_runtime_recipe_identity(
|
|
||||||
model_id=context.selection.model_id,
|
|
||||||
revision=getattr(getattr(backend, "model", None), "revision", None),
|
|
||||||
model_config=model_config_payload,
|
|
||||||
recipe_params=context.recipe.params,
|
|
||||||
weight_quantization=context.selection.quantization,
|
|
||||||
backend_id=context.recipe.backend_id,
|
|
||||||
runtime_version=_PACKAGE_VERSION,
|
|
||||||
activation_dtype=activation_dtype,
|
|
||||||
compute_dtype=compute_dtype,
|
|
||||||
kv_dtype=kv_dtype,
|
|
||||||
kv_layout=kv_layout or _backend_kv_layout(backend),
|
|
||||||
tokenizer_revision=tokenizer_revision,
|
|
||||||
architecture_adapter=architecture_adapter,
|
|
||||||
boundary_schema_version=boundary_schema_version,
|
|
||||||
cache_layout=cache_layout or resolved_cache_layout,
|
|
||||||
)
|
|
||||||
return build_capability_report(
|
return build_capability_report(
|
||||||
model_id=model_id or context.selection.model_id,
|
model_id=model_id or context.selection.model_id,
|
||||||
shard_start=ownership.start_layer if shard_start is None else shard_start,
|
shard_start=(
|
||||||
shard_end=ownership.end_layer if shard_end is None else shard_end,
|
context.selection.shard_start if shard_start is None else shard_start
|
||||||
|
),
|
||||||
|
shard_end=context.selection.shard_end if shard_end is None else shard_end,
|
||||||
recipe_id=recipe_id or context.recipe.id,
|
recipe_id=recipe_id or context.recipe.id,
|
||||||
recipe_version=recipe_version or context.recipe.version,
|
recipe_version=recipe_version or context.recipe.version,
|
||||||
catalogue_version=context.manifest.catalogue_version,
|
catalogue_version=context.manifest.catalogue_version,
|
||||||
backend_id=backend_id or context.recipe.backend_id,
|
backend_id=backend_id or context.recipe.backend_id,
|
||||||
device=device or context.device,
|
device=device or context.device,
|
||||||
quantization=context.selection.quantization,
|
quantization=context.selection.quantization,
|
||||||
runtime=_runtime_versions(),
|
|
||||||
artifact_hash=artifact_hash,
|
|
||||||
runtime_recipe=runtime_recipe,
|
|
||||||
owns_embedding=ownership.owns_embedding,
|
|
||||||
owns_final_head=ownership.owns_final_head,
|
|
||||||
status=status,
|
status=status,
|
||||||
duration_ms=duration_ms,
|
duration_ms=duration_ms,
|
||||||
diagnostics=diagnostics,
|
diagnostics=diagnostics,
|
||||||
@@ -111,20 +68,3 @@ def capability_stub(**overrides: Any) -> CapabilityValidator:
|
|||||||
return capability_report_for(context, **overrides)
|
return capability_report_for(context, **overrides)
|
||||||
|
|
||||||
return validator
|
return validator
|
||||||
|
|
||||||
|
|
||||||
def _runtime_versions() -> dict[str, str]:
|
|
||||||
versions: dict[str, str] = {}
|
|
||||||
for name in ("torch", "transformers"):
|
|
||||||
try:
|
|
||||||
module = __import__(name)
|
|
||||||
except Exception:
|
|
||||||
continue
|
|
||||||
version = getattr(module, "__version__", None)
|
|
||||||
if version:
|
|
||||||
versions[name] = str(version)
|
|
||||||
return versions
|
|
||||||
|
|
||||||
|
|
||||||
def _backend_kv_layout(backend: Any) -> str:
|
|
||||||
return "session-cache" if getattr(backend, "supports_kv_cache", False) else "stateless"
|
|
||||||
|
|||||||
@@ -1543,52 +1543,8 @@ class TorchNodeServer:
|
|||||||
def loaded_model_ids(self) -> list[str]:
|
def loaded_model_ids(self) -> list[str]:
|
||||||
return list(self._backends.keys())
|
return list(self._backends.keys())
|
||||||
|
|
||||||
def backend_for(self, model_id: str) -> TorchModelShard | None:
|
|
||||||
"""The loaded backend serving `model_id` — full repo id or short name."""
|
|
||||||
backend = self._backends.get(model_id)
|
|
||||||
if backend is not None:
|
|
||||||
return backend
|
|
||||||
short = model_id.split("/")[-1].lower()
|
|
||||||
for key, candidate in self._backends.items():
|
|
||||||
if key.split("/")[-1].lower() == short:
|
|
||||||
return candidate
|
|
||||||
return None
|
|
||||||
|
|
||||||
def apply_tracker_directives(self, directives: list[dict]) -> dict | None:
|
def apply_tracker_directives(self, directives: list[dict]) -> dict | None:
|
||||||
"""Apply tracker shard directives (LOAD_SHARD replace, ADD_SHARD load-more)."""
|
"""Apply tracker shard directives (LOAD_SHARD replace, ADD_SHARD load-more)."""
|
||||||
drop_all_directive = next(
|
|
||||||
(directive for directive in reversed(directives) if directive.get("action") == "DROP_ALL_SHARDS"),
|
|
||||||
None,
|
|
||||||
)
|
|
||||||
if drop_all_directive is not None:
|
|
||||||
self._backends.clear()
|
|
||||||
self._backend = None
|
|
||||||
self._tracker_mode = False
|
|
||||||
if self._server is not None:
|
|
||||||
self._server.backends = {}
|
|
||||||
self._server.backend = None
|
|
||||||
self._server.tracker_mode = False
|
|
||||||
return {"action": "DROP_ALL_SHARDS"}
|
|
||||||
drop_directive = next(
|
|
||||||
(directive for directive in reversed(directives) if directive.get("action") == "DROP_SHARD"),
|
|
||||||
None,
|
|
||||||
)
|
|
||||||
if drop_directive is not None:
|
|
||||||
model_id = str(drop_directive.get("model") or "")
|
|
||||||
removed = self._backends.pop(model_id, None)
|
|
||||||
if removed is None:
|
|
||||||
return None
|
|
||||||
if self._backends:
|
|
||||||
self._backend = next(iter(self._backends.values()))
|
|
||||||
self._tracker_mode = self._backend.shard_start == 0
|
|
||||||
else:
|
|
||||||
self._backend = None
|
|
||||||
self._tracker_mode = False
|
|
||||||
if self._server is not None:
|
|
||||||
self._server.backends = dict(self._backends)
|
|
||||||
self._server.backend = self._backend
|
|
||||||
self._server.tracker_mode = self._tracker_mode
|
|
||||||
return {"action": "DROP_SHARD", "model": model_id}
|
|
||||||
add_directive = next(
|
add_directive = next(
|
||||||
(directive for directive in reversed(directives) if directive.get("action") == "ADD_SHARD"),
|
(directive for directive in reversed(directives) if directive.get("action") == "ADD_SHARD"),
|
||||||
None,
|
None,
|
||||||
@@ -1618,8 +1574,6 @@ class TorchNodeServer:
|
|||||||
flush=True,
|
flush=True,
|
||||||
)
|
)
|
||||||
try:
|
try:
|
||||||
if replacing:
|
|
||||||
self._backends.clear()
|
|
||||||
new_backend = _load_backend(model_id, shard_start, shard_end, quantization, self._cache_dir)
|
new_backend = _load_backend(model_id, shard_start, shard_end, quantization, self._cache_dir)
|
||||||
except TypeError:
|
except TypeError:
|
||||||
new_backend = _load_backend(model_id, shard_start, shard_end, quantization)
|
new_backend = _load_backend(model_id, shard_start, shard_end, quantization)
|
||||||
|
|||||||
@@ -1,76 +0,0 @@
|
|||||||
# Reproducible C++ build wiring for the Shard runtime protocol (DGR-002).
|
|
||||||
#
|
|
||||||
# Generates C++ message stubs from proto/shard_runtime.proto and builds the
|
|
||||||
# round-trip / cross-language compatibility test. Requires protoc and the
|
|
||||||
# protobuf C++ runtime. Works with either a CONFIG-mode protobuf install
|
|
||||||
# (protobuf::libprotobuf / protobuf::protoc targets, e.g. a from-source install
|
|
||||||
# on CMAKE_PREFIX_PATH) or CMake's bundled FindProtobuf module.
|
|
||||||
#
|
|
||||||
# The gRPC C++ service stubs are generated separately by scripts/generate_cpp.sh
|
|
||||||
# when grpc_cpp_plugin is present; the round-trip test needs only message
|
|
||||||
# serialization, so gRPC is intentionally not a build dependency here.
|
|
||||||
#
|
|
||||||
# Configure & build (out-of-tree):
|
|
||||||
# cmake -S packages/node/native -B packages/node/native/build/cpp
|
|
||||||
# cmake --build packages/node/native/build/cpp
|
|
||||||
# Run:
|
|
||||||
# packages/node/native/build/cpp/shard_protocol_roundtrip_test --selftest
|
|
||||||
|
|
||||||
cmake_minimum_required(VERSION 3.16)
|
|
||||||
project(shard_runtime_protocol CXX)
|
|
||||||
|
|
||||||
set(CMAKE_CXX_STANDARD 17)
|
|
||||||
set(CMAKE_CXX_STANDARD_REQUIRED ON)
|
|
||||||
|
|
||||||
# Prefer a CONFIG-mode protobuf (modern imported targets); fall back to the
|
|
||||||
# FindProtobuf module for system installs.
|
|
||||||
find_package(Protobuf CONFIG QUIET)
|
|
||||||
if(NOT Protobuf_FOUND)
|
|
||||||
find_package(Protobuf REQUIRED)
|
|
||||||
endif()
|
|
||||||
|
|
||||||
if(TARGET protobuf::protoc)
|
|
||||||
set(SHARD_PROTOC_EXECUTABLE "$<TARGET_FILE:protobuf::protoc>")
|
|
||||||
else()
|
|
||||||
set(SHARD_PROTOC_EXECUTABLE "${Protobuf_PROTOC_EXECUTABLE}")
|
|
||||||
endif()
|
|
||||||
|
|
||||||
if(TARGET protobuf::libprotobuf)
|
|
||||||
set(SHARD_PROTOBUF_LINK protobuf::libprotobuf)
|
|
||||||
else()
|
|
||||||
set(SHARD_PROTOBUF_LINK ${Protobuf_LIBRARIES})
|
|
||||||
endif()
|
|
||||||
|
|
||||||
set(PROTO_DIR "${CMAKE_CURRENT_SOURCE_DIR}/proto")
|
|
||||||
set(PROTO_FILE "${PROTO_DIR}/shard_runtime.proto")
|
|
||||||
set(GEN_DIR "${CMAKE_CURRENT_BINARY_DIR}/gen")
|
|
||||||
file(MAKE_DIRECTORY "${GEN_DIR}")
|
|
||||||
|
|
||||||
set(PROTO_SRC "${GEN_DIR}/shard_runtime.pb.cc")
|
|
||||||
set(PROTO_HDR "${GEN_DIR}/shard_runtime.pb.h")
|
|
||||||
|
|
||||||
add_custom_command(
|
|
||||||
OUTPUT "${PROTO_SRC}" "${PROTO_HDR}"
|
|
||||||
COMMAND "${SHARD_PROTOC_EXECUTABLE}"
|
|
||||||
"--proto_path=${PROTO_DIR}"
|
|
||||||
"--cpp_out=${GEN_DIR}"
|
|
||||||
"${PROTO_FILE}"
|
|
||||||
DEPENDS "${PROTO_FILE}"
|
|
||||||
COMMENT "Generating C++ protobuf stubs from shard_runtime.proto"
|
|
||||||
VERBATIM)
|
|
||||||
|
|
||||||
add_executable(shard_protocol_roundtrip_test
|
|
||||||
tests/roundtrip_test.cpp
|
|
||||||
"${PROTO_SRC}")
|
|
||||||
|
|
||||||
target_include_directories(shard_protocol_roundtrip_test PRIVATE "${GEN_DIR}")
|
|
||||||
if(NOT TARGET protobuf::libprotobuf AND Protobuf_INCLUDE_DIRS)
|
|
||||||
target_include_directories(shard_protocol_roundtrip_test PRIVATE
|
|
||||||
${Protobuf_INCLUDE_DIRS})
|
|
||||||
endif()
|
|
||||||
|
|
||||||
target_link_libraries(shard_protocol_roundtrip_test PRIVATE ${SHARD_PROTOBUF_LINK})
|
|
||||||
|
|
||||||
enable_testing()
|
|
||||||
add_test(NAME shard_protocol_roundtrip
|
|
||||||
COMMAND shard_protocol_roundtrip_test --selftest)
|
|
||||||
@@ -1,24 +0,0 @@
|
|||||||
# Pinned llama.cpp source dependency
|
|
||||||
|
|
||||||
This directory keeps the llama.cpp fork boundary explicit and auditable.
|
|
||||||
|
|
||||||
Layout:
|
|
||||||
|
|
||||||
- `UPSTREAM_COMMIT` - the exact pinned commit.
|
|
||||||
- `UPSTREAM_REPOSITORY` - the reproducible source dependency URL.
|
|
||||||
- `UPSTREAM_ASSUMPTIONS.md` - the file/ABI assumptions that the build scripts
|
|
||||||
validate.
|
|
||||||
- `patches/` - numbered patch files applied on top of the pinned checkout.
|
|
||||||
|
|
||||||
The intended flow is:
|
|
||||||
|
|
||||||
1. Fetch or clone the pinned upstream checkout.
|
|
||||||
2. Verify the checkout commit matches `UPSTREAM_COMMIT`.
|
|
||||||
3. Check and apply the numbered patch stack.
|
|
||||||
4. Build the worker scaffold from `examples/meshnet-worker/`.
|
|
||||||
5. Copy the upstream `LICENSE` and `AUTHORS` files into the worker build tree so
|
|
||||||
the attribution notices remain attached to the built artifact.
|
|
||||||
|
|
||||||
The patch stack in this story is intentionally minimal. It creates the project
|
|
||||||
worker scaffold and the smoke-test CMake target without pulling Meshnet
|
|
||||||
networking code into llama.cpp.
|
|
||||||
@@ -1,35 +0,0 @@
|
|||||||
# llama.cpp upstream assumptions
|
|
||||||
|
|
||||||
This directory records the reproducible source dependency boundary for the
|
|
||||||
pinned llama.cpp checkout used by the distributed GGUF runtime program.
|
|
||||||
|
|
||||||
Pinned upstream commit:
|
|
||||||
|
|
||||||
- `b3c9d1b846cc80a6360adb6aeaa4fcd8c4c8dcac`
|
|
||||||
|
|
||||||
Pinned upstream repository:
|
|
||||||
|
|
||||||
- `https://github.com/ggml-org/llama.cpp.git`
|
|
||||||
|
|
||||||
Assumptions checked by the build script:
|
|
||||||
|
|
||||||
- The checkout is exactly the pinned commit above.
|
|
||||||
- The upstream source tree still ships `LICENSE`, `AUTHORS`, and
|
|
||||||
`CMakeLists.txt` at the repository root.
|
|
||||||
- The project-owned worker scaffold is built from
|
|
||||||
`examples/meshnet-worker/`, which is introduced by the patch stack below.
|
|
||||||
- The upstream license and attribution notices are preserved in the build
|
|
||||||
output by copying the root `LICENSE` and `AUTHORS` files into the worker
|
|
||||||
staging directory.
|
|
||||||
|
|
||||||
Compatibility notes:
|
|
||||||
|
|
||||||
- The current patch stack does not modify upstream llama.cpp runtime code yet.
|
|
||||||
It adds a project-owned worker scaffold that can be built reproducibly from
|
|
||||||
the pinned source checkout.
|
|
||||||
- Later stories extend this boundary with actual llama.cpp execution patches.
|
|
||||||
|
|
||||||
Failure mode:
|
|
||||||
|
|
||||||
- If the checkout commit does not match the pin, the build script fails with a
|
|
||||||
clear pin-mismatch error before patch application or compilation starts.
|
|
||||||
@@ -1 +0,0 @@
|
|||||||
b3c9d1b846cc80a6360adb6aeaa4fcd8c4c8dcac
|
|
||||||
@@ -1 +0,0 @@
|
|||||||
https://github.com/ggml-org/llama.cpp.git
|
|
||||||
@@ -1,35 +0,0 @@
|
|||||||
diff --git a/examples/meshnet-worker/CMakeLists.txt b/examples/meshnet-worker/CMakeLists.txt
|
|
||||||
new file mode 100644
|
|
||||||
index 0000000000..8d9f9a1a2f
|
|
||||||
--- /dev/null
|
|
||||||
+++ b/examples/meshnet-worker/CMakeLists.txt
|
|
||||||
@@ -0,0 +1,19 @@
|
|
||||||
+cmake_minimum_required(VERSION 3.16)
|
|
||||||
+project(meshnet_llama_worker CXX)
|
|
||||||
+
|
|
||||||
+set(CMAKE_CXX_STANDARD 17)
|
|
||||||
+set(CMAKE_CXX_STANDARD_REQUIRED ON)
|
|
||||||
+
|
|
||||||
+configure_file(
|
|
||||||
+ "${CMAKE_CURRENT_SOURCE_DIR}/version.h.in"
|
|
||||||
+ "${CMAKE_CURRENT_BINARY_DIR}/version.h"
|
|
||||||
+ @ONLY)
|
|
||||||
+
|
|
||||||
+add_executable(meshnet_worker
|
|
||||||
+ meshnet_worker.cpp)
|
|
||||||
+
|
|
||||||
+target_include_directories(meshnet_worker PRIVATE "${CMAKE_CURRENT_BINARY_DIR}")
|
|
||||||
+
|
|
||||||
+enable_testing()
|
|
||||||
+add_test(NAME meshnet_worker_smoke
|
|
||||||
+ COMMAND meshnet_worker --smoke)
|
|
||||||
diff --git a/examples/meshnet-worker/version.h.in b/examples/meshnet-worker/version.h.in
|
|
||||||
new file mode 100644
|
|
||||||
index 0000000000..0b75c4e60f
|
|
||||||
--- /dev/null
|
|
||||||
+++ b/examples/meshnet-worker/version.h.in
|
|
||||||
@@ -0,0 +1,4 @@
|
|
||||||
+#pragma once
|
|
||||||
+
|
|
||||||
+#define MESHNET_LLAMA_UPSTREAM_COMMIT "@MESHNET_LLAMA_UPSTREAM_COMMIT@"
|
|
||||||
+#define MESHNET_LLAMA_PATCHSET_VERSION "@MESHNET_LLAMA_PATCHSET_VERSION@"
|
|
||||||
@@ -1,43 +0,0 @@
|
|||||||
#include "version.h"
|
|
||||||
|
|
||||||
#include <iostream>
|
|
||||||
#include <string>
|
|
||||||
|
|
||||||
namespace {
|
|
||||||
|
|
||||||
bool fail(const std::string& why) {
|
|
||||||
std::cerr << "meshnet_worker: FAIL: " << why << std::endl;
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
|
|
||||||
} // namespace
|
|
||||||
|
|
||||||
int main(int argc, char** argv) {
|
|
||||||
bool smoke = argc == 1;
|
|
||||||
|
|
||||||
for (int i = 1; i < argc; ++i) {
|
|
||||||
const std::string arg = argv[i];
|
|
||||||
if (arg == "--smoke") {
|
|
||||||
smoke = true;
|
|
||||||
} else {
|
|
||||||
std::cerr << "unknown arg: " << arg << std::endl;
|
|
||||||
return 2;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
if (!smoke) {
|
|
||||||
return fail("smoke mode not requested"), 1;
|
|
||||||
}
|
|
||||||
|
|
||||||
if (MESHNET_LLAMA_UPSTREAM_COMMIT[0] == '\0') {
|
|
||||||
return fail("upstream commit missing"), 1;
|
|
||||||
}
|
|
||||||
if (MESHNET_LLAMA_PATCHSET_VERSION[0] == '\0') {
|
|
||||||
return fail("patchset version missing"), 1;
|
|
||||||
}
|
|
||||||
|
|
||||||
std::cout << "meshnet worker scaffold ok" << std::endl;
|
|
||||||
std::cout << "upstream commit: " << MESHNET_LLAMA_UPSTREAM_COMMIT << std::endl;
|
|
||||||
std::cout << "patchset version: " << MESHNET_LLAMA_PATCHSET_VERSION << std::endl;
|
|
||||||
return 0;
|
|
||||||
}
|
|
||||||
@@ -1,388 +0,0 @@
|
|||||||
// Shard runtime data-plane protocol for the distributed GGUF runtime (ADR-0024).
|
|
||||||
//
|
|
||||||
// This schema is the semantic contract between Python and C++ Shards. Direct
|
|
||||||
// transport is gRPC over HTTP/2; the existing Meshnet relay may carry the same
|
|
||||||
// serialized frames as opaque binary, so anything gRPC would normally carry in
|
|
||||||
// call metadata (deadlines, cancellation intent) is ALSO representable inside
|
|
||||||
// the messages for relay-transported seams.
|
|
||||||
//
|
|
||||||
// Design rules (see .scratch/distributed-gguf-runtime/RALPH-CONTEXT.md):
|
|
||||||
// * One long-lived bidirectional ActivateSession stream per Route Session
|
|
||||||
// Activation Seam. No per-token channel creation.
|
|
||||||
// * Bounded chunking for prefill; a small decode fast path.
|
|
||||||
// * The activation boundary is a versioned named-tensor bundle, because an
|
|
||||||
// architecture boundary may require more than one tensor.
|
|
||||||
// * Meshnet routing/billing/auth live outside this schema; only the data
|
|
||||||
// plane and the identifiers needed to attribute and isolate work are here.
|
|
||||||
//
|
|
||||||
// Compatibility: proto3. Never renumber or reuse a field number. Add new fields
|
|
||||||
// with new numbers only. Enums keep a 0 UNSPECIFIED member for forward compat.
|
|
||||||
|
|
||||||
syntax = "proto3";
|
|
||||||
|
|
||||||
package meshnet.shard.v1;
|
|
||||||
|
|
||||||
option java_package = "com.meshnet.shard.v1";
|
|
||||||
option java_outer_classname = "ShardRuntimeProto";
|
|
||||||
option go_package = "meshnet/shard/v1;shardv1";
|
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
|
||||||
// Versioning and enums
|
|
||||||
// ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
// Wire schema version. Bumped only on incompatible envelope changes; additive
|
|
||||||
// field changes keep the same version and rely on proto3 unknown-field rules.
|
|
||||||
enum SchemaVersion {
|
|
||||||
SCHEMA_VERSION_UNSPECIFIED = 0;
|
|
||||||
SCHEMA_VERSION_1 = 1;
|
|
||||||
}
|
|
||||||
|
|
||||||
// Lifecycle phase of a seam message. RELEASE and CANCEL are represented both as
|
|
||||||
// dedicated RPCs and as in-stream phases so a relay-carried stream can express
|
|
||||||
// them without a separate channel.
|
|
||||||
enum Phase {
|
|
||||||
PHASE_UNSPECIFIED = 0;
|
|
||||||
PHASE_PREFILL = 1;
|
|
||||||
PHASE_DECODE = 2;
|
|
||||||
PHASE_RELEASE = 3;
|
|
||||||
PHASE_CANCEL = 4;
|
|
||||||
}
|
|
||||||
|
|
||||||
// Tensor element type. GGUF quantized block types are enumerated explicitly so
|
|
||||||
// a boundary bundle can carry pre-quantized payloads without reinterpretation.
|
|
||||||
enum DType {
|
|
||||||
DTYPE_UNSPECIFIED = 0;
|
|
||||||
DTYPE_F32 = 1;
|
|
||||||
DTYPE_F16 = 2;
|
|
||||||
DTYPE_BF16 = 3;
|
|
||||||
DTYPE_I64 = 4;
|
|
||||||
DTYPE_I32 = 5;
|
|
||||||
DTYPE_I16 = 6;
|
|
||||||
DTYPE_I8 = 7;
|
|
||||||
DTYPE_U8 = 8;
|
|
||||||
DTYPE_BOOL = 9;
|
|
||||||
DTYPE_Q8_0 = 20;
|
|
||||||
DTYPE_Q4_0 = 21;
|
|
||||||
DTYPE_Q4_K = 22;
|
|
||||||
DTYPE_Q6_K = 23;
|
|
||||||
}
|
|
||||||
|
|
||||||
// Byte order of a tensor payload. Explicit because Shards may run on
|
|
||||||
// heterogeneous hardware and the relay carries opaque bytes.
|
|
||||||
enum ByteOrder {
|
|
||||||
BYTE_ORDER_UNSPECIFIED = 0;
|
|
||||||
BYTE_ORDER_LITTLE_ENDIAN = 1;
|
|
||||||
BYTE_ORDER_BIG_ENDIAN = 2;
|
|
||||||
}
|
|
||||||
|
|
||||||
// Payload compression applied to a tensor fragment or message body.
|
|
||||||
enum Compression {
|
|
||||||
COMPRESSION_UNSPECIFIED = 0;
|
|
||||||
COMPRESSION_NONE = 1;
|
|
||||||
COMPRESSION_ZSTD = 2;
|
|
||||||
}
|
|
||||||
|
|
||||||
// Checksum algorithm. CRC32C is the cheap per-fragment default; SHA256 is used
|
|
||||||
// where stronger integrity is required.
|
|
||||||
enum ChecksumAlgorithm {
|
|
||||||
CHECKSUM_ALGORITHM_UNSPECIFIED = 0;
|
|
||||||
CHECKSUM_NONE = 1;
|
|
||||||
CHECKSUM_CRC32C = 2;
|
|
||||||
CHECKSUM_CRC32 = 3;
|
|
||||||
CHECKSUM_SHA256 = 4;
|
|
||||||
}
|
|
||||||
|
|
||||||
// What the sender expects from the receiving Shard's Hot KV State for this work
|
|
||||||
// (request side of the cache contract).
|
|
||||||
enum CacheExpectation {
|
|
||||||
CACHE_EXPECTATION_UNSPECIFIED = 0;
|
|
||||||
CACHE_REUSE = 1; // reuse existing KV for (session, epoch)
|
|
||||||
CACHE_FRESH = 2; // start a fresh KV context
|
|
||||||
CACHE_BYPASS = 3; // stateless; do not persist KV
|
|
||||||
}
|
|
||||||
|
|
||||||
// What the receiving Shard actually did with its KV State (result side).
|
|
||||||
enum CacheResult {
|
|
||||||
CACHE_RESULT_UNSPECIFIED = 0;
|
|
||||||
CACHE_HIT = 1;
|
|
||||||
CACHE_MISS = 2;
|
|
||||||
CACHE_WRITTEN = 3;
|
|
||||||
CACHE_BYPASSED = 4;
|
|
||||||
}
|
|
||||||
|
|
||||||
// Coarse retry classification carried in structured status.
|
|
||||||
enum RetryClass {
|
|
||||||
RETRY_CLASS_UNSPECIFIED = 0;
|
|
||||||
RETRY_CLASS_NONE = 1; // terminal success/no-retry
|
|
||||||
RETRY_CLASS_RETRYABLE = 2; // transient; the same step may be retried
|
|
||||||
RETRY_CLASS_FATAL = 3; // do not retry this route/epoch
|
|
||||||
RETRY_CLASS_EPOCH_STALE = 4; // route epoch advanced; re-resolve route
|
|
||||||
}
|
|
||||||
|
|
||||||
enum ServingStatus {
|
|
||||||
SERVING_STATUS_UNSPECIFIED = 0;
|
|
||||||
SERVING = 1;
|
|
||||||
NOT_SERVING = 2;
|
|
||||||
DRAINING = 3;
|
|
||||||
}
|
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
|
||||||
// Common value messages
|
|
||||||
// ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
// Structured, transport-independent status. Mirrors canonical gRPC codes so a
|
|
||||||
// relay-carried frame can express what a gRPC trailer normally would.
|
|
||||||
message Status {
|
|
||||||
uint32 code = 1; // canonical gRPC status code
|
|
||||||
string message = 2;
|
|
||||||
RetryClass retry_class = 3;
|
|
||||||
map<string, string> details = 4;
|
|
||||||
}
|
|
||||||
|
|
||||||
// Integrity check over an associated payload.
|
|
||||||
message Checksum {
|
|
||||||
ChecksumAlgorithm algorithm = 1;
|
|
||||||
bytes value = 2;
|
|
||||||
}
|
|
||||||
|
|
||||||
// Exact Model Artifact / runtime-recipe fingerprint. Both Shards MUST agree on
|
|
||||||
// every populated field before activation; a mismatch is a fatal status.
|
|
||||||
message ArtifactFingerprint {
|
|
||||||
string model_id = 1; // e.g. "meta-llama/Llama-3.1-8B"
|
|
||||||
string revision = 2; // artifact revision / commit
|
|
||||||
string artifact_hash = 3; // hash of the GGUF/model artifact
|
|
||||||
string quantization = 4; // e.g. "Q4_K_M", "F16"
|
|
||||||
string runtime_recipe_fingerprint = 5; // DGR-003 recipe hash
|
|
||||||
}
|
|
||||||
|
|
||||||
// Contiguous transformer layer range owned by a Shard (ADR-0012). end_layer is
|
|
||||||
// exclusive. effective_start_layer is the overlap-safe start after de-dupe of
|
|
||||||
// shared boundary layers between adjacent Shards.
|
|
||||||
message ShardRange {
|
|
||||||
uint32 start_layer = 1;
|
|
||||||
uint32 end_layer = 2;
|
|
||||||
uint32 effective_start_layer = 3;
|
|
||||||
bool owns_embedding = 4;
|
|
||||||
bool owns_final_head = 5;
|
|
||||||
}
|
|
||||||
|
|
||||||
// Token position window for a message. start_position is the absolute index of
|
|
||||||
// the first token; token_count is how many positions this message covers.
|
|
||||||
message Position {
|
|
||||||
uint64 start_position = 1;
|
|
||||||
uint64 token_count = 2;
|
|
||||||
uint64 sequence_length = 3; // total known context length, if known
|
|
||||||
}
|
|
||||||
|
|
||||||
// Envelope carried by every seam message. Everything required to version,
|
|
||||||
// route-attribute, isolate, order, and integrity-check a unit of work.
|
|
||||||
message MessageHeader {
|
|
||||||
SchemaVersion schema_version = 1;
|
|
||||||
string work_id = 2; // request/work ID (idempotency scope)
|
|
||||||
string route_session_id = 3; // Route Session ID
|
|
||||||
uint64 route_epoch = 4; // route epoch; stale epochs are rejected
|
|
||||||
ArtifactFingerprint fingerprint = 5;
|
|
||||||
ShardRange shard_range = 6;
|
|
||||||
Phase phase = 7;
|
|
||||||
Position position = 8;
|
|
||||||
uint64 idempotency_step = 9; // monotonic per (work_id) step counter
|
|
||||||
CacheExpectation cache_expectation = 10;
|
|
||||||
Compression compression = 11; // compression of THIS message's payloads
|
|
||||||
Checksum checksum = 12; // checksum over THIS message's payload
|
|
||||||
}
|
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
|
||||||
// Versioned named-tensor bundle (the activation boundary payload)
|
|
||||||
// ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
// One bounded fragment of a tensor payload. Large tensors are split so no
|
|
||||||
// single message is unbounded; fragments reassemble by byte_offset order.
|
|
||||||
message TensorFragment {
|
|
||||||
uint32 fragment_index = 1;
|
|
||||||
uint32 fragment_count = 2;
|
|
||||||
uint64 byte_offset = 3; // offset of this fragment within the full payload
|
|
||||||
bytes data = 4;
|
|
||||||
Checksum checksum = 5; // checksum over this fragment's (post-compression) data
|
|
||||||
}
|
|
||||||
|
|
||||||
// A single named tensor with full description so the receiver never reinterprets
|
|
||||||
// bytes implicitly.
|
|
||||||
message NamedTensor {
|
|
||||||
string name = 1;
|
|
||||||
repeated uint64 shape = 2;
|
|
||||||
DType dtype = 3;
|
|
||||||
ByteOrder byte_order = 4;
|
|
||||||
uint64 total_byte_length = 5; // full payload length across all fragments
|
|
||||||
Compression compression = 6; // compression applied to fragment data
|
|
||||||
repeated TensorFragment fragments = 7;
|
|
||||||
}
|
|
||||||
|
|
||||||
// A versioned collection of named tensors representing one activation boundary.
|
|
||||||
message TensorBundle {
|
|
||||||
uint32 bundle_version = 1;
|
|
||||||
repeated NamedTensor tensors = 2;
|
|
||||||
}
|
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
|
||||||
// Session stream messages (bidirectional ActivateSession)
|
|
||||||
// ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
// Opens a seam. Carries the header plus stream-scoped bounds. deadline_unix_nanos
|
|
||||||
// lets a relay-carried stream express the call deadline gRPC would otherwise own.
|
|
||||||
message SessionOpen {
|
|
||||||
MessageHeader header = 1;
|
|
||||||
uint64 deadline_unix_nanos = 2; // absolute deadline; 0 = none
|
|
||||||
uint32 max_prefill_tokens_per_chunk = 3; // bound for prefill chunking
|
|
||||||
uint32 max_fragment_bytes = 4; // bound for tensor fragment size
|
|
||||||
FlowControl initial_credit = 5; // receiver's starting flow-control window
|
|
||||||
}
|
|
||||||
|
|
||||||
// Bounded prefill chunk. A prefill is split into ordered chunks each covering at
|
|
||||||
// most max_prefill_tokens_per_chunk positions; final_chunk marks the last one.
|
|
||||||
message PrefillChunk {
|
|
||||||
MessageHeader header = 1;
|
|
||||||
uint32 chunk_index = 2;
|
|
||||||
uint32 chunk_count = 3; // 0 if unknown/streaming
|
|
||||||
bool final_chunk = 4;
|
|
||||||
TensorBundle activations = 5;
|
|
||||||
}
|
|
||||||
|
|
||||||
// Small decode fast path: a single-position (or tiny) step with minimal framing.
|
|
||||||
// Reuses the same header for isolation/ordering but expects one activation bundle.
|
|
||||||
message DecodeStep {
|
|
||||||
MessageHeader header = 1;
|
|
||||||
TensorBundle activation = 2;
|
|
||||||
}
|
|
||||||
|
|
||||||
// Explicit HTTP/2-independent flow-control grant. credits is the number of
|
|
||||||
// additional messages the receiver is willing to accept; the byte/message caps
|
|
||||||
// bound in-flight work for backpressure.
|
|
||||||
message FlowControl {
|
|
||||||
uint64 credits = 1;
|
|
||||||
uint64 max_in_flight_bytes = 2;
|
|
||||||
uint64 max_in_flight_messages = 3;
|
|
||||||
}
|
|
||||||
|
|
||||||
// Release a session's resources (Hot KV State, sequence) cleanly.
|
|
||||||
message ReleaseRequest {
|
|
||||||
MessageHeader header = 1;
|
|
||||||
string reason = 2;
|
|
||||||
}
|
|
||||||
|
|
||||||
message ReleaseResponse {
|
|
||||||
Status status = 1;
|
|
||||||
CacheResult cache_result = 2;
|
|
||||||
}
|
|
||||||
|
|
||||||
// Cancel in-flight work for a session/step.
|
|
||||||
message CancelRequest {
|
|
||||||
MessageHeader header = 1;
|
|
||||||
string reason = 2;
|
|
||||||
}
|
|
||||||
|
|
||||||
message CancelResponse {
|
|
||||||
Status status = 1;
|
|
||||||
}
|
|
||||||
|
|
||||||
// Client -> server frames on the ActivateSession stream.
|
|
||||||
message SessionActivation {
|
|
||||||
oneof payload {
|
|
||||||
SessionOpen open = 1;
|
|
||||||
PrefillChunk prefill = 2;
|
|
||||||
DecodeStep decode = 3;
|
|
||||||
ReleaseRequest release = 4;
|
|
||||||
CancelRequest cancel = 5;
|
|
||||||
FlowControl flow_control = 6;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Computed boundary output for a step: the next Shard's input tensors plus the
|
|
||||||
// cache result and integrity for what was produced.
|
|
||||||
message ActivationResult {
|
|
||||||
MessageHeader header = 1;
|
|
||||||
TensorBundle outputs = 2;
|
|
||||||
CacheResult cache_result = 3;
|
|
||||||
Status status = 4;
|
|
||||||
}
|
|
||||||
|
|
||||||
message SessionAccepted {
|
|
||||||
MessageHeader header = 1;
|
|
||||||
FlowControl granted_credit = 2;
|
|
||||||
Status status = 3;
|
|
||||||
}
|
|
||||||
|
|
||||||
// Server -> client frames on the ActivateSession stream.
|
|
||||||
message SessionResponse {
|
|
||||||
oneof payload {
|
|
||||||
SessionAccepted accepted = 1;
|
|
||||||
ActivationResult result = 2;
|
|
||||||
FlowControl flow_control = 3;
|
|
||||||
Status status = 4;
|
|
||||||
ReleaseResponse release_ack = 5;
|
|
||||||
CancelResponse cancel_ack = 6;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
|
||||||
// Capability and health (unary)
|
|
||||||
// ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
message ResourceBudget {
|
|
||||||
uint64 weight_bytes = 1;
|
|
||||||
uint64 kv_bytes = 2;
|
|
||||||
uint64 scratch_bytes = 3;
|
|
||||||
uint32 max_concurrent_sessions = 4;
|
|
||||||
}
|
|
||||||
|
|
||||||
message CapabilityRequest {
|
|
||||||
SchemaVersion schema_version = 1;
|
|
||||||
}
|
|
||||||
|
|
||||||
message CapabilityResponse {
|
|
||||||
SchemaVersion schema_version = 1;
|
|
||||||
repeated SchemaVersion supported_schema_versions = 2;
|
|
||||||
repeated string supported_architectures = 3; // e.g. "llama", "qwen3"
|
|
||||||
repeated string supported_quantizations = 4;
|
|
||||||
ShardRange servable_range = 5;
|
|
||||||
ResourceBudget budget = 6;
|
|
||||||
repeated Compression supported_compression = 7;
|
|
||||||
repeated ChecksumAlgorithm supported_checksums = 8;
|
|
||||||
ArtifactFingerprint loaded_fingerprint = 9; // empty if no artifact loaded
|
|
||||||
}
|
|
||||||
|
|
||||||
message HealthRequest {
|
|
||||||
string route_session_id = 1; // optional; empty for node-wide health
|
|
||||||
}
|
|
||||||
|
|
||||||
message HealthResponse {
|
|
||||||
ServingStatus status = 1;
|
|
||||||
uint32 active_sessions = 2;
|
|
||||||
uint32 queued_requests = 3;
|
|
||||||
double kv_pressure = 4; // 0.0..1.0 fraction of KV budget in use
|
|
||||||
uint64 rss_bytes = 5;
|
|
||||||
Status detail = 6;
|
|
||||||
}
|
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
|
||||||
// Service
|
|
||||||
// ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
service ShardRuntime {
|
|
||||||
// Admission/capability negotiation.
|
|
||||||
rpc GetCapability(CapabilityRequest) returns (CapabilityResponse);
|
|
||||||
|
|
||||||
// Liveness/backpressure telemetry.
|
|
||||||
rpc Health(HealthRequest) returns (HealthResponse);
|
|
||||||
|
|
||||||
// One long-lived bidirectional stream per Route Session Activation Seam.
|
|
||||||
// Deadlines/cancellation use gRPC call semantics on direct transport and the
|
|
||||||
// in-message equivalents on relay transport; flow control uses FlowControl
|
|
||||||
// frames; errors are structured Status.
|
|
||||||
rpc ActivateSession(stream SessionActivation) returns (stream SessionResponse);
|
|
||||||
|
|
||||||
// Clean resource release (also expressible in-stream as PHASE_RELEASE).
|
|
||||||
rpc Release(ReleaseRequest) returns (ReleaseResponse);
|
|
||||||
|
|
||||||
// Cancellation (also expressible in-stream as PHASE_CANCEL).
|
|
||||||
rpc Cancel(CancelRequest) returns (CancelResponse);
|
|
||||||
}
|
|
||||||
@@ -1,187 +0,0 @@
|
|||||||
#!/usr/bin/env bash
|
|
||||||
# Apply the numbered llama.cpp patch stack and build the worker scaffold.
|
|
||||||
#
|
|
||||||
# Default flow:
|
|
||||||
# 1. Fetch the pinned llama.cpp source into a build directory if needed.
|
|
||||||
# 2. Verify the checkout matches the pinned commit.
|
|
||||||
# 3. Check/apply the numbered patch stack from packages/node/native/llama/.
|
|
||||||
# 4. Compile and build the standalone worker scaffold.
|
|
||||||
# 5. Copy upstream LICENSE/AUTHORS notices into the staging directory.
|
|
||||||
#
|
|
||||||
# This script is intentionally model-free and does not contact any inference
|
|
||||||
# endpoint. It is a source/build reproducibility check.
|
|
||||||
set -euo pipefail
|
|
||||||
|
|
||||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
||||||
NATIVE_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"
|
|
||||||
LLAMA_ROOT="${NATIVE_ROOT}/llama"
|
|
||||||
UPSTREAM_COMMIT="$(tr -d '\n\r' < "${LLAMA_ROOT}/UPSTREAM_COMMIT")"
|
|
||||||
UPSTREAM_REPOSITORY="$(tr -d '\n\r' < "${LLAMA_ROOT}/UPSTREAM_REPOSITORY")"
|
|
||||||
PATCH_DIR="${LLAMA_ROOT}/patches"
|
|
||||||
DEFAULT_SOURCE_DIR="${NATIVE_ROOT}/build/llama.cpp-src"
|
|
||||||
DEFAULT_BUILD_DIR="${NATIVE_ROOT}/build/llama-worker"
|
|
||||||
SOURCE_DIR="${DEFAULT_SOURCE_DIR}"
|
|
||||||
BUILD_DIR="${DEFAULT_BUILD_DIR}"
|
|
||||||
WORKTREE_DIR=""
|
|
||||||
FETCH=1
|
|
||||||
CXX_BIN="${CXX:-}"
|
|
||||||
|
|
||||||
usage() {
|
|
||||||
cat <<'EOF'
|
|
||||||
Usage: build_llama_worker.sh [--source-dir PATH] [--build-dir PATH] [--no-fetch]
|
|
||||||
|
|
||||||
Builds the project-owned worker scaffold from a pinned llama.cpp checkout.
|
|
||||||
EOF
|
|
||||||
}
|
|
||||||
|
|
||||||
fail() {
|
|
||||||
echo "error: $*" >&2
|
|
||||||
exit 1
|
|
||||||
}
|
|
||||||
|
|
||||||
while (($#)); do
|
|
||||||
case "$1" in
|
|
||||||
--source-dir)
|
|
||||||
SOURCE_DIR="${2:-}"
|
|
||||||
shift 2
|
|
||||||
;;
|
|
||||||
--build-dir)
|
|
||||||
BUILD_DIR="${2:-}"
|
|
||||||
shift 2
|
|
||||||
;;
|
|
||||||
--no-fetch)
|
|
||||||
FETCH=0
|
|
||||||
shift
|
|
||||||
;;
|
|
||||||
-h|--help)
|
|
||||||
usage
|
|
||||||
exit 0
|
|
||||||
;;
|
|
||||||
*)
|
|
||||||
fail "unknown argument: $1"
|
|
||||||
;;
|
|
||||||
esac
|
|
||||||
done
|
|
||||||
|
|
||||||
[[ -n "${SOURCE_DIR}" ]] || fail "source dir is empty"
|
|
||||||
[[ -n "${BUILD_DIR}" ]] || fail "build dir is empty"
|
|
||||||
|
|
||||||
checkout_commit() {
|
|
||||||
if [[ -f "${SOURCE_DIR}/.meshnet-upstream-commit" ]]; then
|
|
||||||
tr -d '\n\r' < "${SOURCE_DIR}/.meshnet-upstream-commit"
|
|
||||||
return 0
|
|
||||||
fi
|
|
||||||
if git -C "${SOURCE_DIR}" rev-parse --is-inside-work-tree >/dev/null 2>&1; then
|
|
||||||
git -C "${SOURCE_DIR}" rev-parse HEAD
|
|
||||||
return 0
|
|
||||||
fi
|
|
||||||
return 1
|
|
||||||
}
|
|
||||||
|
|
||||||
ensure_source() {
|
|
||||||
if [[ -d "${SOURCE_DIR}" ]]; then
|
|
||||||
return 0
|
|
||||||
fi
|
|
||||||
if [[ "${FETCH}" -ne 1 ]]; then
|
|
||||||
fail "source dir ${SOURCE_DIR} does not exist and --no-fetch was set"
|
|
||||||
fi
|
|
||||||
|
|
||||||
mkdir -p "${SOURCE_DIR}"
|
|
||||||
git clone --quiet "${UPSTREAM_REPOSITORY}" "${SOURCE_DIR}" || fail "unable to clone ${UPSTREAM_REPOSITORY}"
|
|
||||||
git -C "${SOURCE_DIR}" checkout --quiet "${UPSTREAM_COMMIT}" || fail "unable to checkout ${UPSTREAM_COMMIT}"
|
|
||||||
printf '%s\n' "${UPSTREAM_COMMIT}" > "${SOURCE_DIR}/.meshnet-upstream-commit"
|
|
||||||
printf '%s\n' "${UPSTREAM_REPOSITORY}" > "${SOURCE_DIR}/.meshnet-upstream-repository"
|
|
||||||
}
|
|
||||||
|
|
||||||
verify_assumptions() {
|
|
||||||
local observed_commit
|
|
||||||
observed_commit="$(checkout_commit)" || fail "source tree does not expose a commit pin; write ${SOURCE_DIR}/.meshnet-upstream-commit or use a git checkout"
|
|
||||||
if [[ "${observed_commit}" != "${UPSTREAM_COMMIT}" ]]; then
|
|
||||||
fail "llama.cpp pin mismatch: expected ${UPSTREAM_COMMIT}, got ${observed_commit}"
|
|
||||||
fi
|
|
||||||
|
|
||||||
for required in LICENSE AUTHORS CMakeLists.txt; do
|
|
||||||
[[ -e "${SOURCE_DIR}/${required}" ]] || fail "missing upstream assumption file: ${required}"
|
|
||||||
done
|
|
||||||
}
|
|
||||||
|
|
||||||
apply_patches() {
|
|
||||||
shopt -s nullglob
|
|
||||||
local patches=("${PATCH_DIR}"/*.patch)
|
|
||||||
shopt -u nullglob
|
|
||||||
if ((${#patches[@]} == 0)); then
|
|
||||||
fail "no patch files found in ${PATCH_DIR}"
|
|
||||||
fi
|
|
||||||
|
|
||||||
for patch in "${patches[@]}"; do
|
|
||||||
git -C "${SOURCE_DIR}" apply --check "${patch}" || fail "patch check failed: $(basename "${patch}")"
|
|
||||||
done
|
|
||||||
for patch in "${patches[@]}"; do
|
|
||||||
git -C "${SOURCE_DIR}" apply "${patch}" || fail "patch apply failed: $(basename "${patch}")"
|
|
||||||
done
|
|
||||||
}
|
|
||||||
|
|
||||||
build_worker() {
|
|
||||||
rm -rf "${BUILD_DIR}"
|
|
||||||
mkdir -p "${BUILD_DIR}"
|
|
||||||
WORKTREE_DIR="${BUILD_DIR}/llama.cpp-worktree"
|
|
||||||
rm -rf "${WORKTREE_DIR}"
|
|
||||||
mkdir -p "${WORKTREE_DIR}"
|
|
||||||
cp -a "${SOURCE_DIR}/." "${WORKTREE_DIR}/"
|
|
||||||
if [[ -f "${SOURCE_DIR}/.meshnet-upstream-commit" ]]; then
|
|
||||||
cp "${SOURCE_DIR}/.meshnet-upstream-commit" "${WORKTREE_DIR}/.meshnet-upstream-commit"
|
|
||||||
fi
|
|
||||||
if [[ -f "${SOURCE_DIR}/.meshnet-upstream-repository" ]]; then
|
|
||||||
cp "${SOURCE_DIR}/.meshnet-upstream-repository" "${WORKTREE_DIR}/.meshnet-upstream-repository"
|
|
||||||
fi
|
|
||||||
|
|
||||||
SOURCE_DIR="${WORKTREE_DIR}"
|
|
||||||
apply_patches
|
|
||||||
|
|
||||||
local worker_dir="${SOURCE_DIR}/examples/meshnet-worker"
|
|
||||||
cp "${LLAMA_ROOT}/templates/meshnet_worker.cpp" "${worker_dir}/meshnet_worker.cpp"
|
|
||||||
cat > "${worker_dir}/version.h" <<EOF
|
|
||||||
#pragma once
|
|
||||||
|
|
||||||
#define MESHNET_LLAMA_UPSTREAM_COMMIT "${UPSTREAM_COMMIT}"
|
|
||||||
#define MESHNET_LLAMA_PATCHSET_VERSION "0001"
|
|
||||||
EOF
|
|
||||||
|
|
||||||
local compiler=""
|
|
||||||
if [[ -n "${CXX_BIN}" ]] && command -v "${CXX_BIN}" >/dev/null 2>&1; then
|
|
||||||
compiler="${CXX_BIN}"
|
|
||||||
elif command -v g++ >/dev/null 2>&1; then
|
|
||||||
compiler="g++"
|
|
||||||
elif command -v c++ >/dev/null 2>&1; then
|
|
||||||
compiler="c++"
|
|
||||||
elif command -v clang++ >/dev/null 2>&1; then
|
|
||||||
compiler="clang++"
|
|
||||||
else
|
|
||||||
fail "no C++ compiler found (need g++, c++, clang++, or $CXX)"
|
|
||||||
fi
|
|
||||||
|
|
||||||
"${compiler}" -std=c++17 -O2 -Wall -Wextra \
|
|
||||||
-I "${worker_dir}" \
|
|
||||||
-o "${BUILD_DIR}/meshnet_worker" \
|
|
||||||
"${worker_dir}/meshnet_worker.cpp"
|
|
||||||
}
|
|
||||||
|
|
||||||
stage_notices() {
|
|
||||||
local notice_dir="${BUILD_DIR}/upstream-notices"
|
|
||||||
mkdir -p "${notice_dir}"
|
|
||||||
cp "${SOURCE_DIR}/LICENSE" "${notice_dir}/LICENSE"
|
|
||||||
cp "${SOURCE_DIR}/AUTHORS" "${notice_dir}/AUTHORS"
|
|
||||||
printf '%s\n' "${UPSTREAM_COMMIT}" > "${notice_dir}/UPSTREAM_COMMIT"
|
|
||||||
printf '%s\n' "${UPSTREAM_REPOSITORY}" > "${notice_dir}/UPSTREAM_REPOSITORY"
|
|
||||||
}
|
|
||||||
|
|
||||||
main() {
|
|
||||||
ensure_source
|
|
||||||
verify_assumptions
|
|
||||||
build_worker
|
|
||||||
stage_notices
|
|
||||||
"${BUILD_DIR}/meshnet_worker" --smoke
|
|
||||||
echo "build ok: ${BUILD_DIR}/meshnet_worker"
|
|
||||||
}
|
|
||||||
|
|
||||||
main "$@"
|
|
||||||
@@ -1,43 +0,0 @@
|
|||||||
#!/usr/bin/env bash
|
|
||||||
# Reproducibly generate the C++ Shard-protocol stubs from the schema.
|
|
||||||
#
|
|
||||||
# Produces message stubs (protoc --cpp_out) always, and gRPC C++ service stubs
|
|
||||||
# (protoc --grpc_out with grpc_cpp_plugin) when the plugin is available. The
|
|
||||||
# round-trip test needs only the message stubs; gRPC service stubs are for the
|
|
||||||
# standalone C++ worker (DGR-008).
|
|
||||||
#
|
|
||||||
# Requirements: protoc (>=3.16). Optional: grpc_cpp_plugin for --grpc_out.
|
|
||||||
#
|
|
||||||
# Usage:
|
|
||||||
# packages/node/native/scripts/generate_cpp.sh
|
|
||||||
# Output: packages/node/native/build/cpp-gen/ (gitignored via build/).
|
|
||||||
set -euo pipefail
|
|
||||||
|
|
||||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
||||||
NATIVE_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"
|
|
||||||
PROTO_DIR="${NATIVE_ROOT}/proto"
|
|
||||||
PROTO_FILE="${PROTO_DIR}/shard_runtime.proto"
|
|
||||||
OUT_DIR="${NATIVE_ROOT}/build/cpp-gen"
|
|
||||||
|
|
||||||
if ! command -v protoc >/dev/null 2>&1; then
|
|
||||||
echo "error: protoc not found on PATH (install protobuf-compiler)." >&2
|
|
||||||
exit 3
|
|
||||||
fi
|
|
||||||
|
|
||||||
mkdir -p "${OUT_DIR}"
|
|
||||||
|
|
||||||
echo "generating C++ message stubs -> ${OUT_DIR}"
|
|
||||||
protoc --proto_path="${PROTO_DIR}" --cpp_out="${OUT_DIR}" "${PROTO_FILE}"
|
|
||||||
|
|
||||||
if command -v grpc_cpp_plugin >/dev/null 2>&1; then
|
|
||||||
echo "generating C++ gRPC service stubs -> ${OUT_DIR}"
|
|
||||||
protoc --proto_path="${PROTO_DIR}" \
|
|
||||||
--grpc_out="${OUT_DIR}" \
|
|
||||||
--plugin=protoc-gen-grpc="$(command -v grpc_cpp_plugin)" \
|
|
||||||
"${PROTO_FILE}"
|
|
||||||
else
|
|
||||||
echo "note: grpc_cpp_plugin not found; skipped --grpc_out (message stubs only)." >&2
|
|
||||||
fi
|
|
||||||
|
|
||||||
echo "done:"
|
|
||||||
ls -1 "${OUT_DIR}"
|
|
||||||
@@ -1,76 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
"""Reproducibly generate the Python Shard-protocol stubs from the schema.
|
|
||||||
|
|
||||||
This is the documented, no-manual-copy generation entry point referenced by
|
|
||||||
``evidence/DGR-002/README.md``. It runs the pinned ``grpc_tools.protoc`` with the
|
|
||||||
same flags ``meshnet_node.native_protocol.generate()`` uses on demand, but is
|
|
||||||
kept self-contained (it does not import ``meshnet_node``) so it works regardless
|
|
||||||
of which checkout the editable install points at.
|
|
||||||
|
|
||||||
Usage (from the project .venv):
|
|
||||||
|
|
||||||
python packages/node/native/scripts/generate_python.py
|
|
||||||
|
|
||||||
Output: ``packages/node/native/build/python/shard_runtime_pb2{,_grpc}.py``
|
|
||||||
(``build/`` is gitignored).
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import pathlib
|
|
||||||
import sys
|
|
||||||
|
|
||||||
_NATIVE_ROOT = pathlib.Path(__file__).resolve().parents[1]
|
|
||||||
PROTO_DIR = _NATIVE_ROOT / "proto"
|
|
||||||
PROTO_FILE = PROTO_DIR / "shard_runtime.proto"
|
|
||||||
GEN_DIR = _NATIVE_ROOT / "build" / "python"
|
|
||||||
|
|
||||||
|
|
||||||
def _well_known_include() -> str | None:
|
|
||||||
try:
|
|
||||||
import grpc_tools
|
|
||||||
|
|
||||||
candidate = pathlib.Path(grpc_tools.__file__).parent / "_proto"
|
|
||||||
return str(candidate) if candidate.is_dir() else None
|
|
||||||
except Exception:
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
def main() -> int:
|
|
||||||
if not PROTO_FILE.exists():
|
|
||||||
print(f"schema not found: {PROTO_FILE}", file=sys.stderr)
|
|
||||||
return 2
|
|
||||||
try:
|
|
||||||
from grpc_tools import protoc
|
|
||||||
except ImportError:
|
|
||||||
print(
|
|
||||||
"grpc_tools is required (pip install grpcio-tools); it is present in "
|
|
||||||
"the project .venv.",
|
|
||||||
file=sys.stderr,
|
|
||||||
)
|
|
||||||
return 3
|
|
||||||
|
|
||||||
GEN_DIR.mkdir(parents=True, exist_ok=True)
|
|
||||||
well_known = _well_known_include()
|
|
||||||
args = [
|
|
||||||
"grpc_tools.protoc",
|
|
||||||
f"-I{PROTO_DIR}",
|
|
||||||
*([f"-I{well_known}"] if well_known else []),
|
|
||||||
f"--python_out={GEN_DIR}",
|
|
||||||
f"--grpc_python_out={GEN_DIR}",
|
|
||||||
PROTO_FILE.name,
|
|
||||||
]
|
|
||||||
rc = protoc.main(args)
|
|
||||||
if rc != 0:
|
|
||||||
print(f"grpc_tools.protoc exited with status {rc}", file=sys.stderr)
|
|
||||||
return rc
|
|
||||||
|
|
||||||
print(f"generated Python stubs into: {GEN_DIR}")
|
|
||||||
for name in ("shard_runtime_pb2.py", "shard_runtime_pb2_grpc.py"):
|
|
||||||
target = GEN_DIR / name
|
|
||||||
print(f" {name}: {'ok' if target.exists() else 'MISSING'}")
|
|
||||||
return 0
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
raise SystemExit(main())
|
|
||||||
@@ -1,180 +0,0 @@
|
|||||||
// C++ round-trip and cross-language compatibility test for the Shard protocol.
|
|
||||||
//
|
|
||||||
// Modes (composable):
|
|
||||||
// --selftest serialize a sample message, parse it back, verify fields.
|
|
||||||
// --read <path> parse a fixture serialized by another language; verify the
|
|
||||||
// known fields; tolerate unknown fields (forward compat).
|
|
||||||
// --write <path> serialize the C++ sample so another language can parse it.
|
|
||||||
//
|
|
||||||
// Exit code 0 means every requested check passed. The Python test drives this
|
|
||||||
// binary to prove Python<->C++ wire compatibility in both directions.
|
|
||||||
|
|
||||||
#include "shard_runtime.pb.h"
|
|
||||||
|
|
||||||
#include <cstdint>
|
|
||||||
#include <fstream>
|
|
||||||
#include <iostream>
|
|
||||||
#include <sstream>
|
|
||||||
#include <string>
|
|
||||||
|
|
||||||
using namespace meshnet::shard::v1;
|
|
||||||
|
|
||||||
namespace {
|
|
||||||
|
|
||||||
bool Fail(const std::string& why) {
|
|
||||||
std::cerr << "roundtrip_test: FAIL: " << why << std::endl;
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
|
|
||||||
SessionActivation MakeSample() {
|
|
||||||
SessionActivation act;
|
|
||||||
PrefillChunk* pre = act.mutable_prefill();
|
|
||||||
|
|
||||||
MessageHeader* h = pre->mutable_header();
|
|
||||||
h->set_schema_version(SCHEMA_VERSION_1);
|
|
||||||
h->set_work_id("w1");
|
|
||||||
h->set_route_session_id("s1");
|
|
||||||
h->set_route_epoch(3);
|
|
||||||
h->set_phase(PHASE_PREFILL);
|
|
||||||
h->set_idempotency_step(7);
|
|
||||||
h->set_cache_expectation(CACHE_FRESH);
|
|
||||||
h->set_compression(COMPRESSION_NONE);
|
|
||||||
|
|
||||||
ArtifactFingerprint* fp = h->mutable_fingerprint();
|
|
||||||
fp->set_model_id("meta-llama/Llama-3.1-8B");
|
|
||||||
fp->set_quantization("Q4_K_M");
|
|
||||||
fp->set_runtime_recipe_fingerprint("recipe-abc");
|
|
||||||
|
|
||||||
ShardRange* sr = h->mutable_shard_range();
|
|
||||||
sr->set_start_layer(0);
|
|
||||||
sr->set_end_layer(16);
|
|
||||||
sr->set_effective_start_layer(0);
|
|
||||||
sr->set_owns_embedding(true);
|
|
||||||
|
|
||||||
Position* pos = h->mutable_position();
|
|
||||||
pos->set_start_position(0);
|
|
||||||
pos->set_token_count(5);
|
|
||||||
pos->set_sequence_length(5);
|
|
||||||
|
|
||||||
pre->set_chunk_index(0);
|
|
||||||
pre->set_chunk_count(1);
|
|
||||||
pre->set_final_chunk(true);
|
|
||||||
|
|
||||||
TensorBundle* bundle = pre->mutable_activations();
|
|
||||||
bundle->set_bundle_version(1);
|
|
||||||
NamedTensor* t = bundle->add_tensors();
|
|
||||||
t->set_name("hidden");
|
|
||||||
t->add_shape(1);
|
|
||||||
t->add_shape(4096);
|
|
||||||
t->set_dtype(DTYPE_F16);
|
|
||||||
t->set_byte_order(BYTE_ORDER_LITTLE_ENDIAN);
|
|
||||||
t->set_total_byte_length(8);
|
|
||||||
t->set_compression(COMPRESSION_NONE);
|
|
||||||
TensorFragment* frag = t->add_fragments();
|
|
||||||
frag->set_fragment_index(0);
|
|
||||||
frag->set_fragment_count(1);
|
|
||||||
frag->set_byte_offset(0);
|
|
||||||
frag->set_data(std::string("\x01\x02\x03\x04\x05\x06\x07\x08", 8));
|
|
||||||
|
|
||||||
return act;
|
|
||||||
}
|
|
||||||
|
|
||||||
bool CheckSample(const SessionActivation& act) {
|
|
||||||
if (act.payload_case() != SessionActivation::kPrefill)
|
|
||||||
return Fail("payload is not prefill");
|
|
||||||
const PrefillChunk& pre = act.prefill();
|
|
||||||
const MessageHeader& h = pre.header();
|
|
||||||
if (h.schema_version() != SCHEMA_VERSION_1) return Fail("schema_version");
|
|
||||||
if (h.work_id() != "w1") return Fail("work_id");
|
|
||||||
if (h.route_session_id() != "s1") return Fail("route_session_id");
|
|
||||||
if (h.route_epoch() != 3) return Fail("route_epoch");
|
|
||||||
if (h.phase() != PHASE_PREFILL) return Fail("phase");
|
|
||||||
if (h.idempotency_step() != 7) return Fail("idempotency_step");
|
|
||||||
if (h.fingerprint().model_id() != "meta-llama/Llama-3.1-8B")
|
|
||||||
return Fail("model_id");
|
|
||||||
if (h.fingerprint().quantization() != "Q4_K_M") return Fail("quantization");
|
|
||||||
if (h.shard_range().end_layer() != 16) return Fail("end_layer");
|
|
||||||
if (!h.shard_range().owns_embedding()) return Fail("owns_embedding");
|
|
||||||
if (h.position().token_count() != 5) return Fail("token_count");
|
|
||||||
if (!pre.final_chunk()) return Fail("final_chunk");
|
|
||||||
if (pre.activations().tensors_size() != 1) return Fail("tensors_size");
|
|
||||||
const NamedTensor& t = pre.activations().tensors(0);
|
|
||||||
if (t.name() != "hidden") return Fail("tensor name");
|
|
||||||
if (t.dtype() != DTYPE_F16) return Fail("dtype");
|
|
||||||
if (t.byte_order() != BYTE_ORDER_LITTLE_ENDIAN) return Fail("byte_order");
|
|
||||||
if (t.shape_size() != 2 || t.shape(1) != 4096) return Fail("shape");
|
|
||||||
if (t.fragments_size() != 1) return Fail("fragments_size");
|
|
||||||
if (t.fragments(0).data().size() != 8) return Fail("fragment data length");
|
|
||||||
return true;
|
|
||||||
}
|
|
||||||
|
|
||||||
bool ReadFile(const std::string& path, std::string* out) {
|
|
||||||
std::ifstream in(path, std::ios::binary);
|
|
||||||
if (!in) return false;
|
|
||||||
std::ostringstream ss;
|
|
||||||
ss << in.rdbuf();
|
|
||||||
*out = ss.str();
|
|
||||||
return true;
|
|
||||||
}
|
|
||||||
|
|
||||||
bool WriteFile(const std::string& path, const std::string& data) {
|
|
||||||
std::ofstream out(path, std::ios::binary);
|
|
||||||
if (!out) return false;
|
|
||||||
out.write(data.data(), static_cast<std::streamsize>(data.size()));
|
|
||||||
return static_cast<bool>(out);
|
|
||||||
}
|
|
||||||
|
|
||||||
} // namespace
|
|
||||||
|
|
||||||
int main(int argc, char** argv) {
|
|
||||||
GOOGLE_PROTOBUF_VERIFY_VERSION;
|
|
||||||
|
|
||||||
std::string read_path;
|
|
||||||
std::string write_path;
|
|
||||||
bool selftest = (argc == 1);
|
|
||||||
|
|
||||||
for (int i = 1; i < argc; ++i) {
|
|
||||||
std::string arg = argv[i];
|
|
||||||
if (arg == "--selftest") {
|
|
||||||
selftest = true;
|
|
||||||
} else if (arg == "--read" && i + 1 < argc) {
|
|
||||||
read_path = argv[++i];
|
|
||||||
} else if (arg == "--write" && i + 1 < argc) {
|
|
||||||
write_path = argv[++i];
|
|
||||||
} else {
|
|
||||||
std::cerr << "unknown/incomplete arg: " << arg << std::endl;
|
|
||||||
return 2;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
if (selftest) {
|
|
||||||
SessionActivation sample = MakeSample();
|
|
||||||
std::string bytes;
|
|
||||||
if (!sample.SerializeToString(&bytes)) return Fail("serialize"), 1;
|
|
||||||
SessionActivation parsed;
|
|
||||||
if (!parsed.ParseFromString(bytes)) return Fail("parse"), 1;
|
|
||||||
if (!CheckSample(parsed)) return 1;
|
|
||||||
std::cout << "selftest ok (" << bytes.size() << " bytes)" << std::endl;
|
|
||||||
}
|
|
||||||
|
|
||||||
if (!read_path.empty()) {
|
|
||||||
std::string bytes;
|
|
||||||
if (!ReadFile(read_path, &bytes)) return Fail("cannot read fixture"), 1;
|
|
||||||
SessionActivation parsed;
|
|
||||||
// ParseFromString tolerates and preserves unknown fields (forward compat).
|
|
||||||
if (!parsed.ParseFromString(bytes)) return Fail("parse fixture"), 1;
|
|
||||||
if (!CheckSample(parsed)) return 1;
|
|
||||||
std::cout << "read ok (" << bytes.size() << " bytes)" << std::endl;
|
|
||||||
}
|
|
||||||
|
|
||||||
if (!write_path.empty()) {
|
|
||||||
SessionActivation sample = MakeSample();
|
|
||||||
std::string bytes;
|
|
||||||
if (!sample.SerializeToString(&bytes)) return Fail("serialize for write"), 1;
|
|
||||||
if (!WriteFile(write_path, bytes)) return Fail("cannot write output"), 1;
|
|
||||||
std::cout << "write ok (" << bytes.size() << " bytes)" << std::endl;
|
|
||||||
}
|
|
||||||
|
|
||||||
google::protobuf::ShutdownProtobufLibrary();
|
|
||||||
return 0;
|
|
||||||
}
|
|
||||||
@@ -58,7 +58,6 @@ STATE_MODEL_MISMATCH = "model-mismatch"
|
|||||||
STATE_SHARD_MISMATCH = "shard-mismatch"
|
STATE_SHARD_MISMATCH = "shard-mismatch"
|
||||||
STATE_RECIPE_MISMATCH = "recipe-mismatch"
|
STATE_RECIPE_MISMATCH = "recipe-mismatch"
|
||||||
STATE_CATALOGUE_INCOMPATIBLE = "catalogue-incompatible"
|
STATE_CATALOGUE_INCOMPATIBLE = "catalogue-incompatible"
|
||||||
STATE_COMPATIBILITY_MISMATCH = "compatibility-mismatch"
|
|
||||||
|
|
||||||
ALL_STATES = (
|
ALL_STATES = (
|
||||||
STATE_ADMITTED,
|
STATE_ADMITTED,
|
||||||
@@ -70,7 +69,6 @@ ALL_STATES = (
|
|||||||
STATE_SHARD_MISMATCH,
|
STATE_SHARD_MISMATCH,
|
||||||
STATE_RECIPE_MISMATCH,
|
STATE_RECIPE_MISMATCH,
|
||||||
STATE_CATALOGUE_INCOMPATIBLE,
|
STATE_CATALOGUE_INCOMPATIBLE,
|
||||||
STATE_COMPATIBILITY_MISMATCH,
|
|
||||||
)
|
)
|
||||||
|
|
||||||
# --- Compatibility policy for nodes that predate the capability protocol. ---
|
# --- Compatibility policy for nodes that predate the capability protocol. ---
|
||||||
@@ -157,17 +155,12 @@ class CapabilityState:
|
|||||||
model_id: str | None = None
|
model_id: str | None = None
|
||||||
shard_start: int | None = None
|
shard_start: int | None = None
|
||||||
shard_end: int | None = None
|
shard_end: int | None = None
|
||||||
owns_embedding: bool | None = None
|
|
||||||
owns_final_head: bool | None = None
|
|
||||||
recipe_id: str | None = None
|
recipe_id: str | None = None
|
||||||
recipe_version: str | None = None
|
recipe_version: str | None = None
|
||||||
catalogue_version: str | None = None
|
catalogue_version: str | None = None
|
||||||
backend_id: str | None = None
|
backend_id: str | None = None
|
||||||
device: str | None = None
|
device: str | None = None
|
||||||
quantization: str | None = None
|
quantization: str | None = None
|
||||||
artifact_hash: str | None = None
|
|
||||||
compatibility_fingerprint: str | None = None
|
|
||||||
runtime_recipe_fingerprint: str | None = None
|
|
||||||
validated_at: float | None = None
|
validated_at: float | None = None
|
||||||
recorded_at: float = 0.0
|
recorded_at: float = 0.0
|
||||||
schema_version: int | None = None
|
schema_version: int | None = None
|
||||||
@@ -194,17 +187,12 @@ class CapabilityState:
|
|||||||
"model_id": self.model_id,
|
"model_id": self.model_id,
|
||||||
"shard_start": self.shard_start,
|
"shard_start": self.shard_start,
|
||||||
"shard_end": self.shard_end,
|
"shard_end": self.shard_end,
|
||||||
"owns_embedding": self.owns_embedding,
|
|
||||||
"owns_final_head": self.owns_final_head,
|
|
||||||
"recipe_id": self.recipe_id,
|
"recipe_id": self.recipe_id,
|
||||||
"recipe_version": self.recipe_version,
|
"recipe_version": self.recipe_version,
|
||||||
"catalogue_version": self.catalogue_version,
|
"catalogue_version": self.catalogue_version,
|
||||||
"backend_id": self.backend_id,
|
"backend_id": self.backend_id,
|
||||||
"device": self.device,
|
"device": self.device,
|
||||||
"quantization": self.quantization,
|
"quantization": self.quantization,
|
||||||
"artifact_hash": self.artifact_hash,
|
|
||||||
"compatibility_fingerprint": self.compatibility_fingerprint,
|
|
||||||
"runtime_recipe_fingerprint": self.runtime_recipe_fingerprint,
|
|
||||||
"validated_at": self.validated_at,
|
"validated_at": self.validated_at,
|
||||||
"recorded_at": self.recorded_at,
|
"recorded_at": self.recorded_at,
|
||||||
"schema_version": self.schema_version,
|
"schema_version": self.schema_version,
|
||||||
@@ -234,7 +222,6 @@ def evaluate_report(
|
|||||||
shard_end: int | None,
|
shard_end: int | None,
|
||||||
declared_recipe_id: str | None = None,
|
declared_recipe_id: str | None = None,
|
||||||
declared_recipe_version: str | None = None,
|
declared_recipe_version: str | None = None,
|
||||||
declared_compatibility_fingerprint: str | None = None,
|
|
||||||
now: float | None = None,
|
now: float | None = None,
|
||||||
max_age_seconds: float = DEFAULT_MAX_REPORT_AGE_SECONDS,
|
max_age_seconds: float = DEFAULT_MAX_REPORT_AGE_SECONDS,
|
||||||
) -> CapabilityState:
|
) -> CapabilityState:
|
||||||
@@ -321,17 +308,6 @@ def evaluate_report(
|
|||||||
f"the node declared v{declared_recipe_version}",
|
f"the node declared v{declared_recipe_version}",
|
||||||
)
|
)
|
||||||
|
|
||||||
if (
|
|
||||||
declared_compatibility_fingerprint is not None
|
|
||||||
and base.compatibility_fingerprint != declared_compatibility_fingerprint
|
|
||||||
):
|
|
||||||
return base.with_state(
|
|
||||||
STATE_COMPATIBILITY_MISMATCH,
|
|
||||||
"proof compatibility fingerprint does not match the node's declared "
|
|
||||||
"artifact/runtime recipe; the artifact, tokenizer, architecture, "
|
|
||||||
"boundary schema, activation recipe or cache layout differs",
|
|
||||||
)
|
|
||||||
|
|
||||||
if status != STATUS_PASSED:
|
if status != STATUS_PASSED:
|
||||||
return base.with_state(
|
return base.with_state(
|
||||||
STATE_FAILED,
|
STATE_FAILED,
|
||||||
@@ -368,8 +344,6 @@ def _parse_report(doc: Mapping[str, Any]) -> dict:
|
|||||||
shard = _object(doc.get("shard"), "shard")
|
shard = _object(doc.get("shard"), "shard")
|
||||||
recipe = _object(doc.get("recipe"), "recipe")
|
recipe = _object(doc.get("recipe"), "recipe")
|
||||||
backend = _object(doc.get("backend"), "backend")
|
backend = _object(doc.get("backend"), "backend")
|
||||||
artifact = _object_or_none(doc.get("artifact"), "artifact")
|
|
||||||
runtime_recipe = _object_or_none(doc.get("runtime_recipe"), "runtime_recipe")
|
|
||||||
|
|
||||||
validated_at = doc.get("validated_at")
|
validated_at = doc.get("validated_at")
|
||||||
if isinstance(validated_at, bool) or not isinstance(validated_at, (int, float)):
|
if isinstance(validated_at, bool) or not isinstance(validated_at, (int, float)):
|
||||||
@@ -383,8 +357,6 @@ def _parse_report(doc: Mapping[str, Any]) -> dict:
|
|||||||
"model_id": _text(model.get("model_id"), "model.model_id"),
|
"model_id": _text(model.get("model_id"), "model.model_id"),
|
||||||
"shard_start": _index(shard.get("start"), "shard.start"),
|
"shard_start": _index(shard.get("start"), "shard.start"),
|
||||||
"shard_end": _index(shard.get("end"), "shard.end"),
|
"shard_end": _index(shard.get("end"), "shard.end"),
|
||||||
"owns_embedding": _maybe_bool(shard.get("owns_embedding")),
|
|
||||||
"owns_final_head": _maybe_bool(shard.get("owns_final_head")),
|
|
||||||
"recipe_id": _text(recipe.get("recipe_id"), "recipe.recipe_id"),
|
"recipe_id": _text(recipe.get("recipe_id"), "recipe.recipe_id"),
|
||||||
"recipe_version": _text(recipe.get("recipe_version"), "recipe.recipe_version"),
|
"recipe_version": _text(recipe.get("recipe_version"), "recipe.recipe_version"),
|
||||||
"catalogue_version": _text(
|
"catalogue_version": _text(
|
||||||
@@ -395,15 +367,6 @@ def _parse_report(doc: Mapping[str, Any]) -> dict:
|
|||||||
"quantization": _optional_text(
|
"quantization": _optional_text(
|
||||||
backend.get("quantization"), "backend.quantization"
|
backend.get("quantization"), "backend.quantization"
|
||||||
),
|
),
|
||||||
"artifact_hash": _optional_text(
|
|
||||||
artifact.get("artifact_hash"), "artifact.artifact_hash"
|
|
||||||
),
|
|
||||||
"compatibility_fingerprint": _optional_text(
|
|
||||||
doc.get("compatibility_fingerprint"), "compatibility_fingerprint"
|
|
||||||
),
|
|
||||||
"runtime_recipe_fingerprint": _optional_text(
|
|
||||||
runtime_recipe.get("fingerprint"), "runtime_recipe.fingerprint"
|
|
||||||
),
|
|
||||||
"validated_at": float(validated_at),
|
"validated_at": float(validated_at),
|
||||||
"schema_version": schema_version,
|
"schema_version": schema_version,
|
||||||
"diagnostics": _diagnostics(doc.get("diagnostics")),
|
"diagnostics": _diagnostics(doc.get("diagnostics")),
|
||||||
@@ -417,12 +380,6 @@ def _object(value: Any, field_name: str) -> Mapping[str, Any]:
|
|||||||
return value
|
return value
|
||||||
|
|
||||||
|
|
||||||
def _object_or_none(value: Any, field_name: str) -> Mapping[str, Any]:
|
|
||||||
if value is None:
|
|
||||||
return {}
|
|
||||||
return _object(value, field_name)
|
|
||||||
|
|
||||||
|
|
||||||
def _text(value: Any, field_name: str) -> str:
|
def _text(value: Any, field_name: str) -> str:
|
||||||
if not isinstance(value, str) or not value.strip():
|
if not isinstance(value, str) or not value.strip():
|
||||||
raise _ReportError(f"{field_name!r} must be a non-empty string")
|
raise _ReportError(f"{field_name!r} must be a non-empty string")
|
||||||
@@ -447,12 +404,6 @@ def _maybe_int(value: Any) -> int | None:
|
|||||||
return value
|
return value
|
||||||
|
|
||||||
|
|
||||||
def _maybe_bool(value: Any) -> bool | None:
|
|
||||||
if isinstance(value, bool):
|
|
||||||
return value
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
def _diagnostics(value: Any) -> tuple[str, ...]:
|
def _diagnostics(value: Any) -> tuple[str, ...]:
|
||||||
if not isinstance(value, list):
|
if not isinstance(value, list):
|
||||||
return ()
|
return ()
|
||||||
|
|||||||
@@ -22,9 +22,8 @@
|
|||||||
border-bottom:1px solid var(--border); flex-shrink:0; }
|
border-bottom:1px solid var(--border); flex-shrink:0; }
|
||||||
header h1 { font-size:16px; margin:0; color:var(--accent); }
|
header h1 { font-size:16px; margin:0; color:var(--accent); }
|
||||||
header .meta { color:var(--dim); font-size:12px; }
|
header .meta { color:var(--dim); font-size:12px; }
|
||||||
main { display:grid; grid-template-columns:1fr;
|
main { display:grid; grid-template-columns:repeat(auto-fit,minmax(340px,1fr));
|
||||||
gap:14px; padding:14px 20px; }
|
gap:14px; padding:14px 20px; }
|
||||||
main > section { width:100%; min-width:0; }
|
|
||||||
body.chat-tab-active main {
|
body.chat-tab-active main {
|
||||||
flex:1; min-height:0; display:flex; flex-direction:column;
|
flex:1; min-height:0; display:flex; flex-direction:column;
|
||||||
padding:0; gap:0; overflow:hidden;
|
padding:0; gap:0; overflow:hidden;
|
||||||
@@ -44,15 +43,12 @@
|
|||||||
.empty { color:var(--dim); font-style:italic; }
|
.empty { color:var(--dim); font-style:italic; }
|
||||||
.pill { display:inline-block; padding:0 7px; border-radius:9px;
|
.pill { display:inline-block; padding:0 7px; border-radius:9px;
|
||||||
border:1px solid var(--border); font-size:11px; }
|
border:1px solid var(--border); font-size:11px; }
|
||||||
input, button, select { font:inherit; color:var(--fg); background:var(--bg);
|
input, button { font:inherit; color:var(--fg); background:var(--bg);
|
||||||
border:1px solid var(--border); border-radius:6px; padding:5px 8px; }
|
border:1px solid var(--border); border-radius:6px; padding:5px 8px; }
|
||||||
input { width:100%; margin-bottom:6px; }
|
input { width:100%; margin-bottom:6px; }
|
||||||
button { cursor:pointer; color:var(--accent); }
|
button { cursor:pointer; color:var(--accent); }
|
||||||
button:hover { border-color:var(--accent); }
|
button:hover { border-color:var(--accent); }
|
||||||
button.small { font-size:11px; padding:1px 7px; }
|
button.small { font-size:11px; padding:1px 7px; }
|
||||||
dialog { color:var(--fg); background:var(--panel); border:1px solid var(--border); border-radius:8px; min-width:min(420px,calc(100vw - 32px)); }
|
|
||||||
dialog::backdrop { background:rgba(0,0,0,.55); }
|
|
||||||
.placement-dialog-actions { display:flex; justify-content:flex-end; gap:8px; margin-top:12px; }
|
|
||||||
.form-row { display:flex; gap:8px; }
|
.form-row { display:flex; gap:8px; }
|
||||||
.form-row button { white-space:nowrap; }
|
.form-row button { white-space:nowrap; }
|
||||||
.error-msg { color:var(--bad); font-size:12px; min-height:16px; }
|
.error-msg { color:var(--bad); font-size:12px; min-height:16px; }
|
||||||
@@ -75,12 +71,6 @@
|
|||||||
background:transparent; color:var(--dim); padding:5px 0 8px; }
|
background:transparent; color:var(--dim); padding:5px 0 8px; }
|
||||||
.dashboard-tabs button.active { color:var(--accent); border-bottom-color:var(--accent); }
|
.dashboard-tabs button.active { color:var(--accent); border-bottom-color:var(--accent); }
|
||||||
.wide { grid-column:1 / -1; }
|
.wide { grid-column:1 / -1; }
|
||||||
/* Compact status cards fan out on desktop; tables remain readable at half width. */
|
|
||||||
@media (min-width:900px) {
|
|
||||||
main { grid-template-columns:repeat(4,minmax(0,1fr)); }
|
|
||||||
main > section { grid-column:span 1; }
|
|
||||||
.wide { grid-column:span 2; }
|
|
||||||
}
|
|
||||||
section[hidden] { display:none !important; }
|
section[hidden] { display:none !important; }
|
||||||
section.chat-section {
|
section.chat-section {
|
||||||
padding:0; border:0; border-radius:0; background:var(--bg); min-height:0;
|
padding:0; border:0; border-radius:0; background:var(--bg); min-height:0;
|
||||||
@@ -215,7 +205,7 @@
|
|||||||
.chat-compose button:disabled { opacity:.45; cursor:not-allowed; }
|
.chat-compose button:disabled { opacity:.45; cursor:not-allowed; }
|
||||||
.console {
|
.console {
|
||||||
background:var(--bg); border:1px solid var(--border); border-radius:6px;
|
background:var(--bg); border:1px solid var(--border); border-radius:6px;
|
||||||
min-height:160px; max-height:520px; overflow-y:auto; overflow-x:auto; padding:7px 9px;
|
min-height:160px; max-height:280px; overflow:auto; padding:7px 9px;
|
||||||
white-space:pre-wrap; word-break:break-word; font-size:11px;
|
white-space:pre-wrap; word-break:break-word; font-size:11px;
|
||||||
}
|
}
|
||||||
.console-line { padding:1px 0; border-bottom:1px solid #161b22; }
|
.console-line { padding:1px 0; border-bottom:1px solid #161b22; }
|
||||||
@@ -298,8 +288,6 @@
|
|||||||
<section data-tab="billing" data-admin-only><h2>Node pending payouts</h2><div id="pending" class="empty">admin login required</div></section>
|
<section data-tab="billing" data-admin-only><h2>Node pending payouts</h2><div id="pending" class="empty">admin login required</div></section>
|
||||||
<section data-tab="billing" data-admin-only><h2>Settlement history</h2><div id="settlements" class="empty">admin login required</div></section>
|
<section data-tab="billing" data-admin-only><h2>Settlement history</h2><div id="settlements" class="empty">admin login required</div></section>
|
||||||
<section data-tab="admin"><h2>Tracker hive</h2><div id="hive" class="empty">loading…</div></section>
|
<section data-tab="admin"><h2>Tracker hive</h2><div id="hive" class="empty">loading…</div></section>
|
||||||
<section data-tab="admin" class="wide"><h2>Model placement</h2><div id="admin-model-placement-status" class="dim">Choose a model to load or release.</div><div id="admin-model-placement" class="empty">admin login required</div></section>
|
|
||||||
<section data-tab="admin" class="wide"><h2>Total node pool</h2><div id="admin-node-pool" class="empty">admin login required</div></section>
|
|
||||||
<section data-tab="admin" id="admin-section"><h2>All accounts (admin)</h2><div id="admin" class="empty"></div></section>
|
<section data-tab="admin" id="admin-section"><h2>All accounts (admin)</h2><div id="admin" class="empty"></div></section>
|
||||||
<section data-tab="admin" data-admin-only><h2>Strikes / bans / forfeitures</h2><div id="fraud" class="empty">admin login required</div></section>
|
<section data-tab="admin" data-admin-only><h2>Strikes / bans / forfeitures</h2><div id="fraud" class="empty">admin login required</div></section>
|
||||||
<section data-tab="admin"><h2>Client balances</h2><div id="clients" class="empty">admin login required</div></section>
|
<section data-tab="admin"><h2>Client balances</h2><div id="clients" class="empty">admin login required</div></section>
|
||||||
@@ -327,16 +315,6 @@
|
|||||||
<div id="testing-log" class="console empty">no test output yet</div>
|
<div id="testing-log" class="console empty">no test output yet</div>
|
||||||
</section>
|
</section>
|
||||||
</main>
|
</main>
|
||||||
<dialog id="model-placement-dialog">
|
|
||||||
<form method="dialog">
|
|
||||||
<div id="model-placement-dialog-title"></div>
|
|
||||||
<label for="model-placement-node">Node</label>
|
|
||||||
<select id="model-placement-node"></select>
|
|
||||||
<label id="model-placement-replace" style="display:none"><input type="checkbox" id="model-placement-replace-confirm"> Unload the currently loaded model before loading this one</label>
|
|
||||||
<div id="model-placement-replace-error" class="bad" style="display:none"></div>
|
|
||||||
<div class="placement-dialog-actions"><button value="cancel">Cancel</button><button type="button" id="model-placement-confirm">Confirm</button></div>
|
|
||||||
</form>
|
|
||||||
</dialog>
|
|
||||||
<script>
|
<script>
|
||||||
"use strict";
|
"use strict";
|
||||||
const $ = id => document.getElementById(id);
|
const $ = id => document.getElementById(id);
|
||||||
@@ -1096,7 +1074,6 @@ function renderBillingUsage(records) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
let consoleClearedAt = 0;
|
let consoleClearedAt = 0;
|
||||||
const CONSOLE_MAX_LINES = 1000;
|
|
||||||
|
|
||||||
function clearConsole() {
|
function clearConsole() {
|
||||||
consoleClearedAt = Date.now() / 1000;
|
consoleClearedAt = Date.now() / 1000;
|
||||||
@@ -1110,7 +1087,7 @@ function renderConsole(data) {
|
|||||||
$("console").innerHTML = '<div class="empty">no console events</div>';
|
$("console").innerHTML = '<div class="empty">no console events</div>';
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
$("console").innerHTML = events.slice(-CONSOLE_MAX_LINES).map(e => {
|
$("console").innerHTML = events.slice(-120).map(e => {
|
||||||
const level = String(e.level || "info");
|
const level = String(e.level || "info");
|
||||||
const cls = level === "error" ? "console-level-error" : level === "warn" ? "console-level-warn" : "console-level-info";
|
const cls = level === "error" ? "console-level-error" : level === "warn" ? "console-level-warn" : "console-level-info";
|
||||||
const fields = e.fields && Object.keys(e.fields).length ? " " + JSON.stringify(e.fields) : "";
|
const fields = e.fields && Object.keys(e.fields).length ? " " + JSON.stringify(e.fields) : "";
|
||||||
@@ -1804,7 +1781,7 @@ async function requestSelectedModelLoad() {
|
|||||||
if (!selectedChatModel) return;
|
if (!selectedChatModel) return;
|
||||||
const button = $("request-model-load");
|
const button = $("request-model-load");
|
||||||
if (button) button.disabled = true;
|
if (button) button.disabled = true;
|
||||||
const result = await apiCall("/v1/models/load", "POST", { model: selectedChatModel, force: isAdmin });
|
const result = await apiCall("/v1/models/load", "POST", { model: selectedChatModel });
|
||||||
if (button) button.disabled = false;
|
if (button) button.disabled = false;
|
||||||
if (!result.ok) {
|
if (!result.ok) {
|
||||||
alert(result.data.error || "model load request failed");
|
alert(result.data.error || "model load request failed");
|
||||||
@@ -1814,136 +1791,6 @@ async function requestSelectedModelLoad() {
|
|||||||
$("chat-status").textContent = `load queued on ${short(assignment.node_id || "node")} for layers ${assignment.shard_start}-${assignment.shard_end}`;
|
$("chat-status").textContent = `load queued on ${short(assignment.node_id || "node")} for layers ${assignment.shard_start}-${assignment.shard_end}`;
|
||||||
}
|
}
|
||||||
|
|
||||||
async function requestAdminModelLoad(model, nodeId, replacing) {
|
|
||||||
const result = await apiCall("/v1/models/load", "POST", { model, node_id: nodeId, force: replacing });
|
|
||||||
if (!result.ok) return showAdminModelPlacementStatus(result.data.error || "model load request failed", true);
|
|
||||||
const assignment = result.data.assignment || {};
|
|
||||||
showAdminModelPlacementStatus(`Load queued on ${short(assignment.node_id || "node")} for ${model}.`);
|
|
||||||
await refreshActiveTab(true);
|
|
||||||
}
|
|
||||||
|
|
||||||
async function releaseAdminModel(model, nodeId) {
|
|
||||||
const result = await apiCall("/v1/models/release", "POST", { model, node_id: nodeId });
|
|
||||||
if (!result.ok) return showAdminModelPlacementStatus(result.data.error || "model release request failed", true);
|
|
||||||
showAdminModelPlacementStatus(`Release queued for ${result.data.released || 0} node(s) serving ${model}.`);
|
|
||||||
await refreshActiveTab(true);
|
|
||||||
}
|
|
||||||
|
|
||||||
async function releaseAllNodeModels(nodeId) {
|
|
||||||
if (!confirm("Unload every model from this node?")) return;
|
|
||||||
const result = await apiCall("/v1/nodes/release-all", "POST", { node_id: nodeId });
|
|
||||||
if (!result.ok) return showAdminModelPlacementStatus(result.data.error || "node unload failed", true);
|
|
||||||
showAdminModelPlacementStatus(`Unload queued for ${short(nodeId)}.`);
|
|
||||||
await refreshActiveTab(true);
|
|
||||||
}
|
|
||||||
|
|
||||||
function showAdminModelPlacementStatus(message, isError) {
|
|
||||||
const status = $("admin-model-placement-status");
|
|
||||||
status.textContent = message;
|
|
||||||
status.className = isError ? "bad" : "ok";
|
|
||||||
}
|
|
||||||
|
|
||||||
function gib(bytes) { return bytes == null ? "not reported" : `${(Number(bytes) / 1073741824).toFixed(1)} GiB`; }
|
|
||||||
|
|
||||||
function renderAdminNodePool(map) {
|
|
||||||
const groups = {};
|
|
||||||
for (const node of (map && map.nodes) || []) {
|
|
||||||
const account = node.wallet_address || "unbound account";
|
|
||||||
(groups[account] = groups[account] || []).push(node);
|
|
||||||
}
|
|
||||||
let html = "";
|
|
||||||
for (const [account, nodes] of Object.entries(groups).sort(([a], [b]) => a.localeCompare(b))) {
|
|
||||||
html += `<div style="margin-top:10px"><b>${esc(short(account, 20))}</b> <span class="dim">${nodes.length} node(s)</span></div>`;
|
|
||||||
html += table(["node", "assignment", "state / slots", "model RAM", "RAM", "GPU / VRAM", "model drive", "action"], nodes.map(node => {
|
|
||||||
const hw = node.hardware_profile || {};
|
|
||||||
const cap = node.capacity || {};
|
|
||||||
// The network map keeps reported resource capacity under `capacity`.
|
|
||||||
node.ram_bytes = cap.ram_bytes ?? node.ram_bytes;
|
|
||||||
node.vram_bytes = cap.vram_bytes ?? node.vram_bytes;
|
|
||||||
const disk = hw.model_drive_free_bytes ?? hw.model_path_free_bytes ?? hw.disk_free_bytes;
|
|
||||||
const gpu = hw.gpu_name || (hw.cuda_available ? "CUDA GPU" : "CPU only");
|
|
||||||
const row = [nodeDisplayCell(node), esc(node.hf_repo || node.model || "unassigned"),
|
|
||||||
esc(`${node.stats?.status || "?"} · ${cap.loaded_slots ?? "?"}/${cap.max_loaded_shards ?? node.max_loaded_shards ?? "?"} slots`),
|
|
||||||
esc(gib(cap.loaded_model_bytes)),
|
|
||||||
esc(gib(node.ram_bytes || (hw.ram_mb && hw.ram_mb * 1048576))),
|
|
||||||
esc(`${gpu} · ${gib(node.vram_bytes || (hw.vram_mb && hw.vram_mb * 1048576))}`), esc(gib(disk))];
|
|
||||||
return row.concat([
|
|
||||||
node.shard_start == null ? '<span class="dim">empty</span>' :
|
|
||||||
`<button class="small" data-admin-node-release="${esc(node.node_id)}">unload all</button>`,
|
|
||||||
]);
|
|
||||||
}));
|
|
||||||
}
|
|
||||||
$("admin-node-pool").innerHTML = html || '<div class="empty">no nodes registered</div>';
|
|
||||||
}
|
|
||||||
|
|
||||||
$("admin-node-pool").addEventListener("click", event => {
|
|
||||||
const unload = event.target.closest("[data-admin-node-release]");
|
|
||||||
if (unload) void releaseAllNodeModels(unload.dataset.adminNodeRelease);
|
|
||||||
});
|
|
||||||
|
|
||||||
function renderAdminModelPlacement(models, map) {
|
|
||||||
const nodes = (map && map.nodes) || [];
|
|
||||||
const rows = ((models && models.data) || []).map(model => {
|
|
||||||
const aliases = new Set([model.id, model.hf_repo, ...(model.aliases || [])].filter(Boolean));
|
|
||||||
const serving = nodes.filter(node => aliases.has(node.model) || aliases.has(node.hf_repo)).length;
|
|
||||||
const downloaded = nodes.filter(node => aliases.has(node.model) || aliases.has(node.hf_repo) ||
|
|
||||||
(node.downloaded_models || []).some(item => aliases.has(item.model) || aliases.has(item.hf_repo))).length;
|
|
||||||
const loadable = model.id !== "stub-model";
|
|
||||||
const actions = `<button class="small" data-admin-model-load="${esc(model.id)}"${loadable ? "" : " disabled"}>load</button> ` +
|
|
||||||
`<button class="small" data-admin-model-release="${esc(model.id)}"${serving ? "" : " disabled"}>release</button>`;
|
|
||||||
return [esc(model.name || model.id), String(serving), String(downloaded), actions];
|
|
||||||
});
|
|
||||||
$("admin-model-placement").innerHTML = rows.length
|
|
||||||
? table(["model", "serving nodes", "downloaded on nodes", "admin action"], rows)
|
|
||||||
: '<div class="empty">no model presets configured</div>';
|
|
||||||
}
|
|
||||||
|
|
||||||
$("admin-model-placement").addEventListener("click", event => {
|
|
||||||
const load = event.target.closest("[data-admin-model-load]");
|
|
||||||
const release = event.target.closest("[data-admin-model-release]");
|
|
||||||
if (load) void chooseModelPlacementNode("load", load.dataset.adminModelLoad);
|
|
||||||
if (release) void chooseModelPlacementNode("release", release.dataset.adminModelRelease);
|
|
||||||
});
|
|
||||||
|
|
||||||
function chooseModelPlacementNode(action, model) {
|
|
||||||
const dialog = $("model-placement-dialog");
|
|
||||||
const select = $("model-placement-node");
|
|
||||||
const targetAlias = modelAliasKey(model);
|
|
||||||
const nodes = (lastNetworkMap?.nodes || []).filter(node => action === "load" ||
|
|
||||||
modelAliasKey(node.model) === targetAlias || modelAliasKey(node.hf_repo) === targetAlias);
|
|
||||||
if (!nodes.length) return showAdminModelPlacementStatus(`No node can ${action} ${model}.`, true);
|
|
||||||
$("model-placement-dialog-title").textContent = `${action === "load" ? "Load" : "Release"} ${model} on a node`;
|
|
||||||
select.innerHTML = nodes.map(node => `<option value="${esc(node.node_id)}">${esc(short(node.friendly_name || node.node_id, 20))} — ${esc(node.hf_repo || node.model || "unassigned")}</option>`).join("");
|
|
||||||
const replace = $("model-placement-replace");
|
|
||||||
const replaceConfirm = $("model-placement-replace-confirm");
|
|
||||||
const replaceError = $("model-placement-replace-error");
|
|
||||||
const confirmButton = $("model-placement-confirm");
|
|
||||||
const selectedNode = () => nodes.find(node => node.node_id === select.value);
|
|
||||||
const updateReplacementWarning = () => {
|
|
||||||
const node = selectedNode();
|
|
||||||
const occupied = action === "load" && node && node.shard_start != null && node.shard_end != null &&
|
|
||||||
modelAliasKey(node.hf_repo || node.model) !== targetAlias;
|
|
||||||
replace.style.display = occupied ? "" : "none";
|
|
||||||
replaceConfirm.checked = false;
|
|
||||||
replaceError.style.display = "none";
|
|
||||||
};
|
|
||||||
select.onchange = updateReplacementWarning;
|
|
||||||
updateReplacementWarning();
|
|
||||||
dialog.onclose = null;
|
|
||||||
confirmButton.onclick = () => {
|
|
||||||
const replacing = replace.style.display !== "none";
|
|
||||||
if (replacing && !replaceConfirm.checked) {
|
|
||||||
replaceError.textContent = "Tick the box to confirm that this will unload the current model.";
|
|
||||||
replaceError.style.display = "";
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
dialog.close("confirm");
|
|
||||||
if (action === "load") void requestAdminModelLoad(model, select.value, replacing);
|
|
||||||
else void releaseAdminModel(model, select.value);
|
|
||||||
};
|
|
||||||
dialog.showModal();
|
|
||||||
}
|
|
||||||
|
|
||||||
function chatAuthToken() {
|
function chatAuthToken() {
|
||||||
if (accountApiKeys.length) return accountApiKeys[0];
|
if (accountApiKeys.length) return accountApiKeys[0];
|
||||||
return null;
|
return null;
|
||||||
@@ -2580,19 +2427,14 @@ async function fetchAdminTab() {
|
|||||||
fetchJson("/v1/console"),
|
fetchJson("/v1/console"),
|
||||||
fetchJson("/v1/billing/summary"),
|
fetchJson("/v1/billing/summary"),
|
||||||
fetchJson("/v1/registry/wallets"),
|
fetchJson("/v1/registry/wallets"),
|
||||||
fetchJson("/v1/models"),
|
|
||||||
fetchJson("/v1/network/map"),
|
|
||||||
];
|
];
|
||||||
if (isAdmin) fetches.push(apiCall("/v1/admin/accounts"));
|
if (isAdmin) fetches.push(apiCall("/v1/admin/accounts"));
|
||||||
const results = await Promise.all(fetches);
|
const results = await Promise.all(fetches);
|
||||||
const [raft, consoleData, summary, wallets, models, map, adminResp] = results;
|
const [raft, consoleData, summary, wallets, adminResp] = results;
|
||||||
if (map) lastNetworkMap = map;
|
|
||||||
renderIfChanged("hive", raft, renderHive);
|
renderIfChanged("hive", raft, renderHive);
|
||||||
renderIfChanged("console", consoleData, renderConsole);
|
renderIfChanged("console", consoleData, renderConsole);
|
||||||
renderIfChanged("billing-summary", summary, data => renderBilling(data));
|
renderIfChanged("billing-summary", summary, data => renderBilling(data));
|
||||||
renderIfChanged("fraud", { wallets, summary }, data => renderFraud(data.wallets, data.summary));
|
renderIfChanged("fraud", { wallets, summary }, data => renderFraud(data.wallets, data.summary));
|
||||||
renderIfChanged("admin-model-placement", { models, map }, data => renderAdminModelPlacement(data.models, data.map));
|
|
||||||
renderIfChanged("admin-node-pool", map, renderAdminNodePool);
|
|
||||||
if (adminResp && adminResp.ok) {
|
if (adminResp && adminResp.ok) {
|
||||||
renderIfChanged("admin", adminResp.data.accounts || [], accounts => {
|
renderIfChanged("admin", adminResp.data.accounts || [], accounts => {
|
||||||
const rows = accounts.map(a => {
|
const rows = accounts.map(a => {
|
||||||
|
|||||||
@@ -56,7 +56,6 @@ from .capability import (
|
|||||||
DEFAULT_POLICY as DEFAULT_CAPABILITY_POLICY,
|
DEFAULT_POLICY as DEFAULT_CAPABILITY_POLICY,
|
||||||
POLICY_COMPAT,
|
POLICY_COMPAT,
|
||||||
POLICY_ENFORCE,
|
POLICY_ENFORCE,
|
||||||
STATE_COMPATIBILITY_MISMATCH,
|
|
||||||
STATE_ABSENT,
|
STATE_ABSENT,
|
||||||
STATE_ADMITTED,
|
STATE_ADMITTED,
|
||||||
STATE_MODEL_MISMATCH,
|
STATE_MODEL_MISMATCH,
|
||||||
@@ -87,7 +86,7 @@ from .model_files import files_for_layer_range, snapshot_dir_for_repo
|
|||||||
from .raft import RaftNode
|
from .raft import RaftNode
|
||||||
|
|
||||||
|
|
||||||
_CONSOLE_LIMIT = 1000
|
_CONSOLE_LIMIT = 300
|
||||||
_PROXY_PROGRESS_LOG_INTERVAL = 5.0
|
_PROXY_PROGRESS_LOG_INTERVAL = 5.0
|
||||||
_SESSION_COOKIE_NAME = "meshnet_session"
|
_SESSION_COOKIE_NAME = "meshnet_session"
|
||||||
|
|
||||||
@@ -599,7 +598,6 @@ class _NodeEntry:
|
|||||||
"model_tokens_per_sec",
|
"model_tokens_per_sec",
|
||||||
"pending_directives", "last_heartbeat", "tracker_mode",
|
"pending_directives", "last_heartbeat", "tracker_mode",
|
||||||
"relay_addr", "cert_fingerprint", "peer_id", "friendly_name",
|
"relay_addr", "cert_fingerprint", "peer_id", "friendly_name",
|
||||||
"compatibility_fingerprint",
|
|
||||||
# heartbeat stats (reported by node, cumulative)
|
# heartbeat stats (reported by node, cumulative)
|
||||||
"total_requests", "failed_requests", "queue_depth", "proxy_inflight", "uptime_seconds",
|
"total_requests", "failed_requests", "queue_depth", "proxy_inflight", "uptime_seconds",
|
||||||
"current_requests",
|
"current_requests",
|
||||||
@@ -638,7 +636,6 @@ class _NodeEntry:
|
|||||||
cert_fingerprint: str | None = None,
|
cert_fingerprint: str | None = None,
|
||||||
peer_id: str | None = None,
|
peer_id: str | None = None,
|
||||||
friendly_name: str | None = None,
|
friendly_name: str | None = None,
|
||||||
compatibility_fingerprint: str | None = None,
|
|
||||||
capability: "CapabilityState | None" = None,
|
capability: "CapabilityState | None" = None,
|
||||||
) -> None:
|
) -> None:
|
||||||
self.node_id = node_id
|
self.node_id = node_id
|
||||||
@@ -667,7 +664,6 @@ class _NodeEntry:
|
|||||||
self.cert_fingerprint = cert_fingerprint
|
self.cert_fingerprint = cert_fingerprint
|
||||||
self.peer_id = peer_id
|
self.peer_id = peer_id
|
||||||
self.friendly_name = friendly_name
|
self.friendly_name = friendly_name
|
||||||
self.compatibility_fingerprint = compatibility_fingerprint
|
|
||||||
# No proof presented is `absent`, never `admitted` — a node can only earn
|
# No proof presented is `absent`, never `admitted` — a node can only earn
|
||||||
# `admitted` by presenting a report that covers what it advertises.
|
# `admitted` by presenting a report that covers what it advertises.
|
||||||
self.capability: CapabilityState = capability or absent_state()
|
self.capability: CapabilityState = capability or absent_state()
|
||||||
@@ -786,16 +782,6 @@ def _node_admission(node: "_NodeEntry") -> CapabilityState:
|
|||||||
f"proof is for layers {state.shard_start}–{state.shard_end}, but the "
|
f"proof is for layers {state.shard_start}–{state.shard_end}, but the "
|
||||||
f"node now serves layers {node.shard_start}–{node.shard_end}",
|
f"node now serves layers {node.shard_start}–{node.shard_end}",
|
||||||
)
|
)
|
||||||
if (
|
|
||||||
node.compatibility_fingerprint
|
|
||||||
and state.compatibility_fingerprint
|
|
||||||
and state.compatibility_fingerprint != node.compatibility_fingerprint
|
|
||||||
):
|
|
||||||
return state.with_state(
|
|
||||||
STATE_COMPATIBILITY_MISMATCH,
|
|
||||||
"proof compatibility fingerprint no longer matches the node's "
|
|
||||||
"declared artifact/runtime recipe",
|
|
||||||
)
|
|
||||||
return state
|
return state
|
||||||
|
|
||||||
|
|
||||||
@@ -825,12 +811,6 @@ def _capability_from_registration(
|
|||||||
declared_recipe_version=(
|
declared_recipe_version=(
|
||||||
recipe_version if isinstance(recipe_version, str) else None
|
recipe_version if isinstance(recipe_version, str) else None
|
||||||
),
|
),
|
||||||
declared_compatibility_fingerprint=(
|
|
||||||
value.strip()
|
|
||||||
if isinstance((value := payload.get("compatibility_fingerprint")), str)
|
|
||||||
and value.strip()
|
|
||||||
else None
|
|
||||||
),
|
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -1121,15 +1101,12 @@ def _registration_quantization(body: dict, quantizations: list[str]) -> str | No
|
|||||||
|
|
||||||
An absent field predates the protocol adding it: it means "unknown", not
|
An absent field predates the protocol adding it: it means "unknown", not
|
||||||
"unsupported", so the node keeps the best precision it advertises and stays
|
"unsupported", so the node keeps the best precision it advertises and stays
|
||||||
routable. An explicit "auto" means the same thing — the node's CLI default
|
routable. Anything the node states explicitly is taken at its word -- a null,
|
||||||
delegates the choice, it does not refuse one. Anything else the node states
|
a non-string, or an unsupported name leaves it with no usable precision and
|
||||||
explicitly is taken at its word -- a null, a non-string, or an unsupported
|
routing excludes it.
|
||||||
name leaves it with no usable precision and routing excludes it.
|
|
||||||
"""
|
"""
|
||||||
declared = body.get("quantization")
|
if "quantization" in body:
|
||||||
declared_auto = isinstance(declared, str) and declared.strip().lower() == "auto"
|
return _normalize_quantization(body["quantization"])
|
||||||
if "quantization" in body and not declared_auto:
|
|
||||||
return _normalize_quantization(declared)
|
|
||||||
supported = [
|
supported = [
|
||||||
normalized for value in quantizations
|
normalized for value in quantizations
|
||||||
if (normalized := _normalize_quantization(value)) is not None
|
if (normalized := _normalize_quantization(value)) is not None
|
||||||
@@ -1248,7 +1225,6 @@ def _node_capacity_summary(node: _NodeEntry, preset: dict | None = None) -> dict
|
|||||||
"quantization": node.quantization,
|
"quantization": node.quantization,
|
||||||
"benchmark_tokens_per_sec": node.benchmark_tokens_per_sec,
|
"benchmark_tokens_per_sec": node.benchmark_tokens_per_sec,
|
||||||
"effective_throughput": round(_effective_throughput(node), 4),
|
"effective_throughput": round(_effective_throughput(node), 4),
|
||||||
"loaded_model_bytes": _assignment_memory_bytes(node, preset),
|
|
||||||
}
|
}
|
||||||
if preset is not None:
|
if preset is not None:
|
||||||
summary["max_assignable_layers"] = _node_layer_capacity(node, preset)
|
summary["max_assignable_layers"] = _node_layer_capacity(node, preset)
|
||||||
@@ -1518,9 +1494,7 @@ def _scale_demanded_models_locked(server: "_TrackerHTTPServer") -> None:
|
|||||||
break
|
break
|
||||||
|
|
||||||
|
|
||||||
def _request_model_load_locked(
|
def _request_model_load_locked(server: "_TrackerHTTPServer", model_key: str) -> dict | None:
|
||||||
server: "_TrackerHTTPServer", model_key: str, node_id: str | None = None,
|
|
||||||
) -> dict | None:
|
|
||||||
"""Queue an explicitly requested model on the best available joined node."""
|
"""Queue an explicitly requested model on the best available joined node."""
|
||||||
resolved_name, preset = _resolve_model_preset(server.model_presets, model_key)
|
resolved_name, preset = _resolve_model_preset(server.model_presets, model_key)
|
||||||
if preset is None or not preset.get("hf_repo"):
|
if preset is None or not preset.get("hf_repo"):
|
||||||
@@ -1536,8 +1510,6 @@ def _request_model_load_locked(
|
|||||||
continue
|
continue
|
||||||
host_nodes = [server.registry[item["node_id"]] for item in host["loaded"] if item["node_id"] in server.registry]
|
host_nodes = [server.registry[item["node_id"]] for item in host["loaded"] if item["node_id"] in server.registry]
|
||||||
placeable = [node for node in host_nodes if _has_usable_quantization(node)]
|
placeable = [node for node in host_nodes if _has_usable_quantization(node)]
|
||||||
if node_id is not None:
|
|
||||||
placeable = [node for node in placeable if node.node_id == node_id]
|
|
||||||
if not placeable:
|
if not placeable:
|
||||||
continue
|
continue
|
||||||
anchor = max(placeable, key=lambda node: node.benchmark_tokens_per_sec)
|
anchor = max(placeable, key=lambda node: node.benchmark_tokens_per_sec)
|
||||||
@@ -1556,68 +1528,6 @@ def _request_model_load_locked(
|
|||||||
return None
|
return None
|
||||||
|
|
||||||
|
|
||||||
def _force_model_load_locked(
|
|
||||||
server: "_TrackerHTTPServer", model_key: str, node_id: str | None = None,
|
|
||||||
) -> dict | None:
|
|
||||||
"""Replace the fastest ready assignment after an explicit admin eviction."""
|
|
||||||
resolved_name, preset = _resolve_model_preset(server.model_presets, model_key)
|
|
||||||
if preset is None or not preset.get("hf_repo"):
|
|
||||||
return None
|
|
||||||
start, end = _preset_layer_bounds(preset)
|
|
||||||
# An explicit admin eviction is permitted to recover a stuck/loading node
|
|
||||||
# and to use the preset default precision. It must only avoid a node that
|
|
||||||
# already has another assignment in flight.
|
|
||||||
candidates = [
|
|
||||||
node for node in server.registry.values()
|
|
||||||
if node.pending_new_assignment is None
|
|
||||||
and (node_id is None or node.node_id == node_id)
|
|
||||||
]
|
|
||||||
if not candidates:
|
|
||||||
return None
|
|
||||||
node = max(candidates, key=lambda item: item.benchmark_tokens_per_sec)
|
|
||||||
shard_end = min(end, start + max(1, min(_node_layer_capacity(node, preset), end - start + 1)) - 1)
|
|
||||||
quantization = _node_quantization(node, preset)
|
|
||||||
directive = _load_directive(node, str(preset["hf_repo"]), start, shard_end, quantization)
|
|
||||||
replaced = node.hf_repo or node.model
|
|
||||||
node.model, node.hf_repo = resolved_name, str(preset["hf_repo"])
|
|
||||||
node.shard_start, node.shard_end, node.quantization = start, shard_end, quantization
|
|
||||||
node.managed_assignment, node.pending_new_assignment = True, directive
|
|
||||||
node.pending_directives.append(directive)
|
|
||||||
_tracker_log(server, "warn", "model load forced", node_id=node.node_id,
|
|
||||||
model=resolved_name, replaced_model=replaced, shard=f"{start}-{shard_end}")
|
|
||||||
return {"node_id": node.node_id, "model": resolved_name, "hf_repo": preset["hf_repo"],
|
|
||||||
"shard_start": start, "shard_end": shard_end, "replaced_model": replaced}
|
|
||||||
|
|
||||||
|
|
||||||
def _release_model_locked(
|
|
||||||
server: "_TrackerHTTPServer", model_key: str, node_id: str | None = None,
|
|
||||||
) -> int:
|
|
||||||
"""Queue DROP_SHARD for every served shard and remove it from routing immediately."""
|
|
||||||
resolved_name, preset = _resolve_model_preset(server.model_presets, model_key)
|
|
||||||
if preset is None:
|
|
||||||
return 0
|
|
||||||
released = 0
|
|
||||||
for node in server.registry.values():
|
|
||||||
if node_id is not None and node.node_id != node_id:
|
|
||||||
continue
|
|
||||||
if not _node_matches_preset(node, resolved_name, preset) or node.shard_start is None or node.shard_end is None:
|
|
||||||
continue
|
|
||||||
node.pending_directives.append(_drop_directive(node, str(preset.get("hf_repo") or resolved_name), node.shard_start, node.shard_end, node.quantization or "bfloat16"))
|
|
||||||
node.status = "loading"
|
|
||||||
released += 1
|
|
||||||
return released
|
|
||||||
|
|
||||||
|
|
||||||
def _release_all_node_models_locked(server: "_TrackerHTTPServer", node_id: str) -> int:
|
|
||||||
"""Queue removal of every loaded assignment on one node."""
|
|
||||||
node = server.registry.get(node_id)
|
|
||||||
if node is None or node.shard_start is None or node.shard_end is None:
|
|
||||||
return 0
|
|
||||||
node.pending_directives.append({"action": "DROP_ALL_SHARDS"})
|
|
||||||
node.status = "loading"
|
|
||||||
return 1
|
|
||||||
|
|
||||||
|
|
||||||
def _preferred_node_quantization(
|
def _preferred_node_quantization(
|
||||||
node: _NodeEntry,
|
node: _NodeEntry,
|
||||||
preset: dict,
|
preset: dict,
|
||||||
@@ -3133,12 +3043,6 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
|
|||||||
if self.path == "/v1/models/load":
|
if self.path == "/v1/models/load":
|
||||||
self._handle_model_load_request()
|
self._handle_model_load_request()
|
||||||
return
|
return
|
||||||
if self.path == "/v1/models/release":
|
|
||||||
self._handle_model_release_request()
|
|
||||||
return
|
|
||||||
if self.path == "/v1/nodes/release-all":
|
|
||||||
self._handle_node_release_all_request()
|
|
||||||
return
|
|
||||||
if self.path == "/v1/models/vote":
|
if self.path == "/v1/models/vote":
|
||||||
self._handle_model_coverage_vote()
|
self._handle_model_coverage_vote()
|
||||||
return
|
return
|
||||||
@@ -3266,6 +3170,8 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
|
|||||||
seen_ids: set[str] = set()
|
seen_ids: set[str] = set()
|
||||||
for name, preset in server.model_presets.items():
|
for name, preset in server.model_presets.items():
|
||||||
model_nodes = [node for node in alive if _node_matches_preset(node, name, preset)]
|
model_nodes = [node for node in alive if _node_matches_preset(node, name, preset)]
|
||||||
|
if not model_nodes and not preset.get("recommended"):
|
||||||
|
continue
|
||||||
required_start, required_end = _preset_layer_bounds(preset)
|
required_start, required_end = _preset_layer_bounds(preset)
|
||||||
coverage = _coverage_percentage(
|
coverage = _coverage_percentage(
|
||||||
model_nodes,
|
model_nodes,
|
||||||
@@ -3315,12 +3221,7 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
|
|||||||
node.hf_repo or node.model
|
node.hf_repo or node.model
|
||||||
for node in alive
|
for node in alive
|
||||||
if node.model is not None
|
if node.model is not None
|
||||||
# The same model can be registered under its HF repository while
|
and node.model not in server.model_presets
|
||||||
# the catalogue exposes its short preset id. Do not emit a second
|
|
||||||
# repo-keyed entry when either node identifier resolves to a preset.
|
|
||||||
and _resolve_model_preset(
|
|
||||||
server.model_presets, node.hf_repo or node.model,
|
|
||||||
)[1] is None
|
|
||||||
and node.shard_start is not None
|
and node.shard_start is not None
|
||||||
and node.shard_end is not None
|
and node.shard_end is not None
|
||||||
and node.num_layers is not None
|
and node.num_layers is not None
|
||||||
@@ -3421,11 +3322,6 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
|
|||||||
"endpoint": node.endpoint,
|
"endpoint": node.endpoint,
|
||||||
"relay_addr": node.relay_addr,
|
"relay_addr": node.relay_addr,
|
||||||
"peer_id": node.peer_id,
|
"peer_id": node.peer_id,
|
||||||
"wallet_address": node.wallet_address,
|
|
||||||
"hardware_profile": dict(node.hardware_profile),
|
|
||||||
"ram_bytes": node.ram_bytes,
|
|
||||||
"vram_bytes": node.vram_bytes,
|
|
||||||
"max_loaded_shards": node.max_loaded_shards,
|
|
||||||
}
|
}
|
||||||
for node in tracker_nodes
|
for node in tracker_nodes
|
||||||
],
|
],
|
||||||
@@ -3449,7 +3345,12 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
|
|||||||
memory_pool = _memory_pool_map(server)
|
memory_pool = _memory_pool_map(server)
|
||||||
|
|
||||||
def capacity_for(node: _NodeEntry) -> dict:
|
def capacity_for(node: _NodeEntry) -> dict:
|
||||||
return _node_capacity_summary(node, _preset_for_node(server, node))
|
preset = None
|
||||||
|
if node.model:
|
||||||
|
preset = server.model_presets.get(node.model)
|
||||||
|
if preset is None and node.hf_repo and node.num_layers:
|
||||||
|
preset = _hf_rebalance_preset([node])
|
||||||
|
return _node_capacity_summary(node, preset)
|
||||||
|
|
||||||
def throughput_for(node: _NodeEntry) -> dict:
|
def throughput_for(node: _NodeEntry) -> dict:
|
||||||
if server.stats is None:
|
if server.stats is None:
|
||||||
@@ -4643,13 +4544,6 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
|
|||||||
relay_addr = body.get("relay_addr") or None
|
relay_addr = body.get("relay_addr") or None
|
||||||
cert_fingerprint = body.get("cert_fingerprint") or None
|
cert_fingerprint = body.get("cert_fingerprint") or None
|
||||||
peer_id = body.get("peer_id") or None
|
peer_id = body.get("peer_id") or None
|
||||||
compatibility_fingerprint = body.get("compatibility_fingerprint")
|
|
||||||
if compatibility_fingerprint is not None and (
|
|
||||||
not isinstance(compatibility_fingerprint, str) or not compatibility_fingerprint.strip()
|
|
||||||
):
|
|
||||||
self._send_json(400, {"error": "compatibility_fingerprint must be a string"})
|
|
||||||
return
|
|
||||||
compatibility_fingerprint = compatibility_fingerprint.strip() if isinstance(compatibility_fingerprint, str) else None
|
|
||||||
try:
|
try:
|
||||||
friendly_name = _normalize_friendly_name(body.get("friendly_name"))
|
friendly_name = _normalize_friendly_name(body.get("friendly_name"))
|
||||||
except ValueError as exc:
|
except ValueError as exc:
|
||||||
@@ -4709,7 +4603,6 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
|
|||||||
cert_fingerprint=cert_fingerprint,
|
cert_fingerprint=cert_fingerprint,
|
||||||
peer_id=peer_id,
|
peer_id=peer_id,
|
||||||
friendly_name=friendly_name,
|
friendly_name=friendly_name,
|
||||||
compatibility_fingerprint=compatibility_fingerprint,
|
|
||||||
capability=capability,
|
capability=capability,
|
||||||
)
|
)
|
||||||
with server.lock:
|
with server.lock:
|
||||||
@@ -4867,20 +4760,6 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
|
|||||||
entry.uptime_seconds = float(body["uptime_seconds"])
|
entry.uptime_seconds = float(body["uptime_seconds"])
|
||||||
if "status" in body and body["status"] in ("ready", "loading"):
|
if "status" in body and body["status"] in ("ready", "loading"):
|
||||||
entry.status = body["status"]
|
entry.status = body["status"]
|
||||||
completed_directives = body.get("completed_directives", [])
|
|
||||||
if isinstance(completed_directives, list):
|
|
||||||
for directive in completed_directives:
|
|
||||||
if not isinstance(directive, dict) or directive.get("action") not in {"DROP_SHARD", "DROP_ALL_SHARDS"}:
|
|
||||||
continue
|
|
||||||
# A node has confirmed the release. Stop advertising its
|
|
||||||
# old route immediately so the dashboard and routing state
|
|
||||||
# agree with the runtime.
|
|
||||||
entry.model = "stub-model"
|
|
||||||
entry.hf_repo = None
|
|
||||||
entry.shard_start = None
|
|
||||||
entry.shard_end = None
|
|
||||||
entry.tracker_mode = False
|
|
||||||
entry.status = "ready"
|
|
||||||
if "friendly_name" in body:
|
if "friendly_name" in body:
|
||||||
try:
|
try:
|
||||||
entry.friendly_name = _normalize_friendly_name(body.get("friendly_name"))
|
entry.friendly_name = _normalize_friendly_name(body.get("friendly_name"))
|
||||||
@@ -4952,68 +4831,14 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
|
|||||||
if not isinstance(model, str) or not model.strip():
|
if not isinstance(model, str) or not model.strip():
|
||||||
self._send_json(400, {"error": "model is required"})
|
self._send_json(400, {"error": "model is required"})
|
||||||
return
|
return
|
||||||
node_id = body.get("node_id")
|
|
||||||
if node_id is not None and (not isinstance(node_id, str) or not node_id):
|
|
||||||
self._send_json(400, {"error": "node_id must be a non-empty string"})
|
|
||||||
return
|
|
||||||
_resolved_name, preset = _resolve_model_preset(server.model_presets, model)
|
|
||||||
if preset is None or str(preset.get("hf_repo") or "").strip().lower() == "stub-model":
|
|
||||||
self._send_json(400, {"error": "stub-model is a local test backend and cannot be loaded onto a node"})
|
|
||||||
return
|
|
||||||
with server.lock:
|
with server.lock:
|
||||||
self._purge_expired_nodes()
|
self._purge_expired_nodes()
|
||||||
assignment = _request_model_load_locked(server, model, node_id)
|
assignment = _request_model_load_locked(server, model)
|
||||||
if assignment is None and body.get("force") is True:
|
|
||||||
assignment = _force_model_load_locked(server, model, node_id)
|
|
||||||
if assignment is None:
|
if assignment is None:
|
||||||
self._send_json(409, {"error": "no ready joined node has an available model slot and sufficient capacity"})
|
self._send_json(409, {"error": "no ready joined node has an available model slot and sufficient capacity"})
|
||||||
return
|
return
|
||||||
self._send_json(202, {"status": "queued", "assignment": assignment})
|
self._send_json(202, {"status": "queued", "assignment": assignment})
|
||||||
|
|
||||||
def _handle_model_release_request(self):
|
|
||||||
server: _TrackerHTTPServer = self.server # type: ignore[assignment]
|
|
||||||
if not self._require_role("admin", "validator"):
|
|
||||||
return
|
|
||||||
body = self._read_json_body()
|
|
||||||
if body is None:
|
|
||||||
return
|
|
||||||
model = body.get("model")
|
|
||||||
if not isinstance(model, str) or not model.strip():
|
|
||||||
self._send_json(400, {"error": "model is required"})
|
|
||||||
return
|
|
||||||
node_id = body.get("node_id")
|
|
||||||
if node_id is not None and (not isinstance(node_id, str) or not node_id):
|
|
||||||
self._send_json(400, {"error": "node_id must be a non-empty string"})
|
|
||||||
return
|
|
||||||
with server.lock:
|
|
||||||
self._purge_expired_nodes()
|
|
||||||
released = _release_model_locked(server, model, node_id)
|
|
||||||
if not released:
|
|
||||||
self._send_json(404, {"error": "no served shards found for model"})
|
|
||||||
return
|
|
||||||
self._send_json(202, {"status": "release_queued", "released": released})
|
|
||||||
|
|
||||||
def _handle_node_release_all_request(self):
|
|
||||||
server: _TrackerHTTPServer = self.server # type: ignore[assignment]
|
|
||||||
if not self._require_role("admin", "validator"):
|
|
||||||
return
|
|
||||||
body = self._read_json_body()
|
|
||||||
if body is None:
|
|
||||||
return
|
|
||||||
node_id = body.get("node_id")
|
|
||||||
if not isinstance(node_id, str) or not node_id:
|
|
||||||
self._send_json(400, {"error": "node_id must be a non-empty string"})
|
|
||||||
return
|
|
||||||
with server.lock:
|
|
||||||
self._purge_expired_nodes()
|
|
||||||
released = _release_all_node_models_locked(server, node_id)
|
|
||||||
if not released:
|
|
||||||
self._send_json(404, {"error": "no loaded models found for node"})
|
|
||||||
return
|
|
||||||
self._send_json(202, {
|
|
||||||
"status": "release_queued", "released": released, "node_id": node_id,
|
|
||||||
})
|
|
||||||
|
|
||||||
def _handle_model_coverage_vote(self):
|
def _handle_model_coverage_vote(self):
|
||||||
"""Record a rolling wish-list signal for an unavailable precision."""
|
"""Record a rolling wish-list signal for an unavailable precision."""
|
||||||
server: _TrackerHTTPServer = self.server # type: ignore[assignment]
|
server: _TrackerHTTPServer = self.server # type: ignore[assignment]
|
||||||
@@ -7162,12 +6987,6 @@ class TrackerServer:
|
|||||||
else None
|
else None
|
||||||
),
|
),
|
||||||
friendly_name=_normalize_friendly_name(payload.get("friendly_name")),
|
friendly_name=_normalize_friendly_name(payload.get("friendly_name")),
|
||||||
compatibility_fingerprint=(
|
|
||||||
value.strip()
|
|
||||||
if isinstance((value := payload.get("compatibility_fingerprint")), str)
|
|
||||||
and value.strip()
|
|
||||||
else None
|
|
||||||
),
|
|
||||||
# A replicated registration carries its proof: without this, a proven
|
# A replicated registration carries its proof: without this, a proven
|
||||||
# node would be routable on the leader and dark on every follower.
|
# node would be routable on the leader and dark on every follower.
|
||||||
capability=_capability_from_registration(
|
capability=_capability_from_registration(
|
||||||
|
|||||||
@@ -1,488 +0,0 @@
|
|||||||
"""Architecture-defined boundary input/output and dense-Llama parity (DGR-006).
|
|
||||||
|
|
||||||
These tests prove the boundary contract with a *pure-numpy* dense-Llama reference
|
|
||||||
model: no download, no GPU, no torch, no API credit. The reference implements the
|
|
||||||
same ``ShardComputation`` duck type the real llama.cpp/PyTorch backends expose, so
|
|
||||||
whole-model execution and a two-range (or three-range) split are the exact same
|
|
||||||
arithmetic applied to the exact same float32 residual stream. Splitting the layer
|
|
||||||
stack at a seam and shipping the *unnormalized* residual bundle across a simulated
|
|
||||||
process boundary must reproduce the whole-model tokens bit-for-bit.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import numpy as np
|
|
||||||
import pytest
|
|
||||||
|
|
||||||
from meshnet_node.boundary_adapter import (
|
|
||||||
BOUNDARY_SCHEMA_VERSION,
|
|
||||||
BoundaryAdapter,
|
|
||||||
BoundaryBundle,
|
|
||||||
BoundaryContractError,
|
|
||||||
SamplingContract,
|
|
||||||
ShardRole,
|
|
||||||
TailOutput,
|
|
||||||
UncertifiedArchitectureError,
|
|
||||||
certified_architecture,
|
|
||||||
is_certified_architecture,
|
|
||||||
role_for_range,
|
|
||||||
)
|
|
||||||
|
|
||||||
# Documented parity tolerance. The split path applies the identical layer
|
|
||||||
# functions in the identical order to the identical float32 arrays, so the
|
|
||||||
# residual seam is bit-exact in practice; the tolerance is a conservative guard.
|
|
||||||
PARITY_ATOL = 1e-6
|
|
||||||
|
|
||||||
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
# Pure-numpy dense-Llama reference model (test fixture, not production).
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
|
|
||||||
|
|
||||||
class _ReferenceDenseLlama:
|
|
||||||
"""A tiny deterministic dense-Llama: RMSNorm, RoPE attention, SwiGLU MLP."""
|
|
||||||
|
|
||||||
architecture_adapter = "dense-llama"
|
|
||||||
|
|
||||||
def __init__(
|
|
||||||
self,
|
|
||||||
*,
|
|
||||||
vocab: int = 48,
|
|
||||||
hidden: int = 32,
|
|
||||||
n_layers: int = 6,
|
|
||||||
n_heads: int = 4,
|
|
||||||
intermediate: int = 64,
|
|
||||||
rms_eps: float = 1e-6,
|
|
||||||
rope_theta: float = 10000.0,
|
|
||||||
seed: int = 20260715,
|
|
||||||
) -> None:
|
|
||||||
assert hidden % n_heads == 0
|
|
||||||
self.vocab = vocab
|
|
||||||
self.hidden = hidden
|
|
||||||
self.n_layers = n_layers
|
|
||||||
self.n_heads = n_heads
|
|
||||||
self.head_dim = hidden // n_heads
|
|
||||||
assert self.head_dim % 2 == 0
|
|
||||||
self.rms_eps = rms_eps
|
|
||||||
self.rope_theta = rope_theta
|
|
||||||
|
|
||||||
rng = np.random.default_rng(seed)
|
|
||||||
|
|
||||||
def w(*shape: int) -> np.ndarray:
|
|
||||||
return (rng.standard_normal(shape) * 0.08).astype(np.float32)
|
|
||||||
|
|
||||||
self.embed = w(vocab, hidden)
|
|
||||||
self.layers = []
|
|
||||||
for _ in range(n_layers):
|
|
||||||
self.layers.append(
|
|
||||||
{
|
|
||||||
"in_ln": (1.0 + rng.standard_normal(hidden) * 0.02).astype(np.float32),
|
|
||||||
"q": w(hidden, hidden),
|
|
||||||
"k": w(hidden, hidden),
|
|
||||||
"v": w(hidden, hidden),
|
|
||||||
"o": w(hidden, hidden),
|
|
||||||
"post_ln": (1.0 + rng.standard_normal(hidden) * 0.02).astype(np.float32),
|
|
||||||
"gate": w(intermediate, hidden),
|
|
||||||
"up": w(intermediate, hidden),
|
|
||||||
"down": w(hidden, intermediate),
|
|
||||||
}
|
|
||||||
)
|
|
||||||
self.final_ln = (1.0 + rng.standard_normal(hidden) * 0.02).astype(np.float32)
|
|
||||||
self.lm_head_w = w(vocab, hidden)
|
|
||||||
|
|
||||||
inv_freq = 1.0 / (
|
|
||||||
rope_theta ** (np.arange(0, self.head_dim, 2, dtype=np.float32) / self.head_dim)
|
|
||||||
)
|
|
||||||
self.inv_freq = inv_freq.astype(np.float32)
|
|
||||||
|
|
||||||
# -- primitive ops -----------------------------------------------------
|
|
||||||
def _rmsnorm(self, x: np.ndarray, weight: np.ndarray) -> np.ndarray:
|
|
||||||
variance = np.mean(x.astype(np.float32) ** 2, axis=-1, keepdims=True)
|
|
||||||
normed = x / np.sqrt(variance + self.rms_eps)
|
|
||||||
return (normed * weight).astype(np.float32)
|
|
||||||
|
|
||||||
def _rope(self, positions: np.ndarray):
|
|
||||||
# positions: (batch, seq) -> cos/sin: (batch, seq, head_dim)
|
|
||||||
angles = positions[..., None].astype(np.float32) * self.inv_freq[None, None, :]
|
|
||||||
emb = np.concatenate([angles, angles], axis=-1)
|
|
||||||
return np.cos(emb).astype(np.float32), np.sin(emb).astype(np.float32)
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _rotate_half(x: np.ndarray) -> np.ndarray:
|
|
||||||
half = x.shape[-1] // 2
|
|
||||||
return np.concatenate([-x[..., half:], x[..., :half]], axis=-1)
|
|
||||||
|
|
||||||
def _apply_rope(self, t: np.ndarray, cos: np.ndarray, sin: np.ndarray) -> np.ndarray:
|
|
||||||
# t: (batch, n_heads, seq, head_dim); cos/sin: (batch, seq, head_dim)
|
|
||||||
cos = cos[:, None, :, :]
|
|
||||||
sin = sin[:, None, :, :]
|
|
||||||
return t * cos + self._rotate_half(t) * sin
|
|
||||||
|
|
||||||
def _attention(self, x: np.ndarray, layer: dict, positions: np.ndarray) -> np.ndarray:
|
|
||||||
batch, seq, _ = x.shape
|
|
||||||
q = (x @ layer["q"].T).reshape(batch, seq, self.n_heads, self.head_dim)
|
|
||||||
k = (x @ layer["k"].T).reshape(batch, seq, self.n_heads, self.head_dim)
|
|
||||||
v = (x @ layer["v"].T).reshape(batch, seq, self.n_heads, self.head_dim)
|
|
||||||
q = q.transpose(0, 2, 1, 3)
|
|
||||||
k = k.transpose(0, 2, 1, 3)
|
|
||||||
v = v.transpose(0, 2, 1, 3)
|
|
||||||
cos, sin = self._rope(positions)
|
|
||||||
q = self._apply_rope(q, cos, sin)
|
|
||||||
k = self._apply_rope(k, cos, sin)
|
|
||||||
scores = (q @ k.transpose(0, 1, 3, 2)) / np.sqrt(self.head_dim)
|
|
||||||
causal = np.triu(np.full((seq, seq), -1e30, dtype=np.float32), k=1)
|
|
||||||
scores = scores + causal[None, None, :, :]
|
|
||||||
scores = scores - scores.max(axis=-1, keepdims=True)
|
|
||||||
weights = np.exp(scores)
|
|
||||||
weights = weights / weights.sum(axis=-1, keepdims=True)
|
|
||||||
out = weights @ v
|
|
||||||
out = out.transpose(0, 2, 1, 3).reshape(batch, seq, self.hidden)
|
|
||||||
return (out @ layer["o"].T).astype(np.float32)
|
|
||||||
|
|
||||||
def _mlp(self, x: np.ndarray, layer: dict) -> np.ndarray:
|
|
||||||
gate = x @ layer["gate"].T
|
|
||||||
up = x @ layer["up"].T
|
|
||||||
silu = gate * (1.0 / (1.0 + np.exp(-gate)))
|
|
||||||
return ((silu * up) @ layer["down"].T).astype(np.float32)
|
|
||||||
|
|
||||||
def _run_layer(self, x: np.ndarray, layer: dict, positions: np.ndarray) -> np.ndarray:
|
|
||||||
h = x + self._attention(self._rmsnorm(x, layer["in_ln"]), layer, positions)
|
|
||||||
h = h + self._mlp(self._rmsnorm(h, layer["post_ln"]), layer)
|
|
||||||
return h.astype(np.float32)
|
|
||||||
|
|
||||||
|
|
||||||
class _ReferenceShard:
|
|
||||||
"""A contiguous inclusive layer range of the reference model.
|
|
||||||
|
|
||||||
Satisfies the ``ShardComputation`` duck type used by ``BoundaryAdapter``.
|
|
||||||
"""
|
|
||||||
|
|
||||||
def __init__(
|
|
||||||
self,
|
|
||||||
model: _ReferenceDenseLlama,
|
|
||||||
start_layer: int,
|
|
||||||
end_layer: int,
|
|
||||||
*,
|
|
||||||
architecture_adapter: str | None = None,
|
|
||||||
) -> None:
|
|
||||||
self._model = model
|
|
||||||
self.start_layer = start_layer
|
|
||||||
self.end_layer = end_layer
|
|
||||||
self.total_layers = model.n_layers
|
|
||||||
self.architecture_adapter = architecture_adapter or model.architecture_adapter
|
|
||||||
|
|
||||||
def embed_tokens(self, token_ids: np.ndarray) -> np.ndarray:
|
|
||||||
return self._model.embed[np.asarray(token_ids)]
|
|
||||||
|
|
||||||
def run_layers(self, hidden: np.ndarray, *, positions: np.ndarray) -> np.ndarray:
|
|
||||||
h = np.asarray(hidden, dtype=np.float32)
|
|
||||||
for idx in range(self.start_layer, self.end_layer + 1):
|
|
||||||
h = self._model._run_layer(h, self._model.layers[idx], positions)
|
|
||||||
return h
|
|
||||||
|
|
||||||
def final_norm(self, hidden: np.ndarray) -> np.ndarray:
|
|
||||||
return self._model._rmsnorm(np.asarray(hidden, dtype=np.float32), self._model.final_ln)
|
|
||||||
|
|
||||||
def lm_head(self, hidden: np.ndarray) -> np.ndarray:
|
|
||||||
return np.asarray(hidden, dtype=np.float32) @ self._model.lm_head_w.T
|
|
||||||
|
|
||||||
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
# Whole-model and split reference drivers.
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
|
|
||||||
|
|
||||||
def _whole_model_next_token(model: _ReferenceDenseLlama, token_ids: list[int]) -> TailOutput:
|
|
||||||
shard = _ReferenceShard(model, 0, model.n_layers - 1)
|
|
||||||
adapter = BoundaryAdapter(shard)
|
|
||||||
result = adapter.forward(token_ids=np.asarray(token_ids)[None, :])
|
|
||||||
assert isinstance(result, TailOutput)
|
|
||||||
return result
|
|
||||||
|
|
||||||
|
|
||||||
def _split_next_token(
|
|
||||||
model: _ReferenceDenseLlama,
|
|
||||||
token_ids: list[int],
|
|
||||||
cut_points: list[int],
|
|
||||||
*,
|
|
||||||
through_wire: bool = True,
|
|
||||||
) -> TailOutput:
|
|
||||||
"""Run the model as N contiguous ranges, shipping the bundle across each seam.
|
|
||||||
|
|
||||||
``cut_points`` are the last (inclusive) layer of each non-final range.
|
|
||||||
"""
|
|
||||||
bounds = _ranges_from_cuts(cut_points, model.n_layers)
|
|
||||||
boundary: BoundaryBundle | None = None
|
|
||||||
result: BoundaryBundle | TailOutput | None = None
|
|
||||||
for i, (start, end) in enumerate(bounds):
|
|
||||||
shard = _ReferenceShard(model, start, end)
|
|
||||||
adapter = BoundaryAdapter(shard)
|
|
||||||
if i == 0:
|
|
||||||
result = adapter.forward(token_ids=np.asarray(token_ids)[None, :])
|
|
||||||
else:
|
|
||||||
assert isinstance(boundary, BoundaryBundle)
|
|
||||||
incoming = BoundaryBundle.unpack(boundary.pack()) if through_wire else boundary
|
|
||||||
result = adapter.forward(boundary=incoming)
|
|
||||||
if isinstance(result, BoundaryBundle):
|
|
||||||
boundary = result
|
|
||||||
assert isinstance(result, TailOutput)
|
|
||||||
return result
|
|
||||||
|
|
||||||
|
|
||||||
def _ranges_from_cuts(cut_points: list[int], n_layers: int) -> list[tuple[int, int]]:
|
|
||||||
bounds: list[tuple[int, int]] = []
|
|
||||||
start = 0
|
|
||||||
for cut in cut_points:
|
|
||||||
bounds.append((start, cut))
|
|
||||||
start = cut + 1
|
|
||||||
bounds.append((start, n_layers - 1))
|
|
||||||
return bounds
|
|
||||||
|
|
||||||
|
|
||||||
def _greedy_generate(next_token_fn, prompt: list[int], n_new: int) -> list[int]:
|
|
||||||
tokens = list(prompt)
|
|
||||||
generated: list[int] = []
|
|
||||||
for _ in range(n_new):
|
|
||||||
out = next_token_fn(tokens)
|
|
||||||
tokens.append(out.token_id)
|
|
||||||
generated.append(out.token_id)
|
|
||||||
return generated
|
|
||||||
|
|
||||||
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
# Certification / fail-closed.
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
|
|
||||||
|
|
||||||
def test_dense_llama_and_aliases_are_certified():
|
|
||||||
"Dense Llama-family identifiers all resolve to the one certified adapter.\n\nTags: node, boundary"
|
|
||||||
for name in ("dense-llama", "llama", "LlamaForCausalLM", "LlamaModel"):
|
|
||||||
boundary = certified_architecture(name)
|
|
||||||
assert boundary.adapter == "dense-llama"
|
|
||||||
assert boundary.boundary_tensor_name == "residual_stream"
|
|
||||||
assert is_certified_architecture(name)
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.parametrize("name", ["qwen3", "qwen3-moe", "mixtral", "gpt2", "", None, 123])
|
|
||||||
def test_uncertified_architectures_fail_closed(name):
|
|
||||||
"Uncertified architectures raise instead of guessing a tensor layout.\n\nTags: node, boundary"
|
|
||||||
assert not is_certified_architecture(name)
|
|
||||||
with pytest.raises(UncertifiedArchitectureError):
|
|
||||||
certified_architecture(name)
|
|
||||||
|
|
||||||
|
|
||||||
def test_adapter_construction_fails_closed_for_uncertified_backend():
|
|
||||||
"Building the adapter over an uncertified computation fails closed.\n\nTags: node, boundary"
|
|
||||||
model = _ReferenceDenseLlama()
|
|
||||||
shard = _ReferenceShard(model, 0, 2, architecture_adapter="qwen3-moe")
|
|
||||||
with pytest.raises(UncertifiedArchitectureError):
|
|
||||||
BoundaryAdapter(shard)
|
|
||||||
|
|
||||||
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
# Roles.
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
|
|
||||||
|
|
||||||
def test_role_classification():
|
|
||||||
"Range endpoints map to head/middle/tail/full roles.\n\nTags: node, boundary"
|
|
||||||
assert role_for_range(0, 2, 6) is ShardRole.HEAD
|
|
||||||
assert role_for_range(2, 3, 6) is ShardRole.MIDDLE
|
|
||||||
assert role_for_range(4, 5, 6) is ShardRole.TAIL
|
|
||||||
assert role_for_range(0, 5, 6) is ShardRole.FULL
|
|
||||||
assert ShardRole.HEAD.owns_embedding and not ShardRole.HEAD.owns_final_head
|
|
||||||
assert ShardRole.TAIL.owns_final_head and not ShardRole.TAIL.owns_embedding
|
|
||||||
|
|
||||||
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
# Input-side contract.
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
|
|
||||||
|
|
||||||
def test_head_accepts_token_ids_and_owns_embedding():
|
|
||||||
"The head embeds token IDs and refuses an upstream boundary bundle.\n\nTags: node, boundary"
|
|
||||||
model = _ReferenceDenseLlama()
|
|
||||||
head = BoundaryAdapter(_ReferenceShard(model, 0, 2))
|
|
||||||
out = head.forward(token_ids=[1, 2, 3])
|
|
||||||
assert isinstance(out, BoundaryBundle)
|
|
||||||
|
|
||||||
# Head owns embedding: a residual bundle from upstream is a contract error.
|
|
||||||
bundle = out
|
|
||||||
with pytest.raises(BoundaryContractError, match="head owns token embedding"):
|
|
||||||
head.forward(boundary=bundle)
|
|
||||||
|
|
||||||
|
|
||||||
def test_middle_and_tail_bypass_embedding_and_require_the_bundle():
|
|
||||||
"Middle/tail Shards reject token IDs and demand the named boundary bundle.\n\nTags: node, boundary"
|
|
||||||
model = _ReferenceDenseLlama()
|
|
||||||
tail = BoundaryAdapter(_ReferenceShard(model, 3, 5))
|
|
||||||
with pytest.raises(BoundaryContractError, match="bypass token embedding"):
|
|
||||||
tail.forward(token_ids=[1, 2, 3])
|
|
||||||
with pytest.raises(BoundaryContractError, match="must receive the named boundary bundle"):
|
|
||||||
tail.forward()
|
|
||||||
|
|
||||||
|
|
||||||
def test_boundary_seam_layer_mismatch_is_rejected():
|
|
||||||
"A bundle handed to the wrong range (seam layer mismatch) is rejected.\n\nTags: node, boundary"
|
|
||||||
model = _ReferenceDenseLlama()
|
|
||||||
head = BoundaryAdapter(_ReferenceShard(model, 0, 2))
|
|
||||||
bundle = head.forward(token_ids=[1, 2, 3])
|
|
||||||
assert isinstance(bundle, BoundaryBundle)
|
|
||||||
assert bundle.next_layer == 3
|
|
||||||
|
|
||||||
# A range that starts at layer 4 must not accept a bundle cut at layer 3.
|
|
||||||
wrong = BoundaryAdapter(_ReferenceShard(model, 4, 5))
|
|
||||||
with pytest.raises(BoundaryContractError, match="starts at layer 4"):
|
|
||||||
wrong.forward(boundary=bundle)
|
|
||||||
|
|
||||||
|
|
||||||
def test_normalized_bundle_is_rejected():
|
|
||||||
"A normalized residual is not the architecture-defined boundary.\n\nTags: node, boundary"
|
|
||||||
model = _ReferenceDenseLlama()
|
|
||||||
head = BoundaryAdapter(_ReferenceShard(model, 0, 2))
|
|
||||||
bundle = head.forward(token_ids=[1, 2, 3])
|
|
||||||
assert isinstance(bundle, BoundaryBundle)
|
|
||||||
normalized = BoundaryBundle(
|
|
||||||
architecture_adapter=bundle.architecture_adapter,
|
|
||||||
schema_version=bundle.schema_version,
|
|
||||||
tensor_name=bundle.tensor_name,
|
|
||||||
residual=bundle.residual,
|
|
||||||
positions=bundle.positions,
|
|
||||||
next_layer=bundle.next_layer,
|
|
||||||
normalized=True,
|
|
||||||
)
|
|
||||||
tail = BoundaryAdapter(_ReferenceShard(model, 3, 5))
|
|
||||||
with pytest.raises(BoundaryContractError, match="UNNORMALIZED"):
|
|
||||||
tail.forward(boundary=normalized)
|
|
||||||
|
|
||||||
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
# Output-side contract.
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
|
|
||||||
|
|
||||||
def test_non_tail_emits_unnormalized_full_row_boundary():
|
|
||||||
"A non-tail Shard emits the unnormalized residual with every position row.\n\nTags: node, boundary"
|
|
||||||
model = _ReferenceDenseLlama()
|
|
||||||
tokens = [3, 7, 1, 9, 2]
|
|
||||||
head = BoundaryAdapter(_ReferenceShard(model, 0, 2))
|
|
||||||
bundle = head.forward(token_ids=tokens)
|
|
||||||
assert isinstance(bundle, BoundaryBundle)
|
|
||||||
assert bundle.normalized is False
|
|
||||||
assert bundle.tensor_name == "residual_stream"
|
|
||||||
assert bundle.schema_version == BOUNDARY_SCHEMA_VERSION
|
|
||||||
assert bundle.next_layer == 3
|
|
||||||
# No tail-only row pruning: all sequence positions are forwarded.
|
|
||||||
assert bundle.residual.shape == (1, len(tokens), model.hidden)
|
|
||||||
assert bundle.positions.shape == (1, len(tokens))
|
|
||||||
|
|
||||||
# The emitted residual must be exactly the whole model's residual after layer 2
|
|
||||||
# (i.e. before any final norm) — prove it is NOT normalized.
|
|
||||||
positions = np.arange(len(tokens))[None, :]
|
|
||||||
hidden = model.embed[np.asarray(tokens)][None, :]
|
|
||||||
for idx in range(0, 3):
|
|
||||||
hidden = model._run_layer(hidden, model.layers[idx], positions)
|
|
||||||
assert np.allclose(bundle.residual, hidden, atol=0)
|
|
||||||
assert not np.allclose(bundle.residual, model._rmsnorm(hidden, model.final_ln))
|
|
||||||
|
|
||||||
|
|
||||||
def test_tail_emits_pruned_logits_through_the_sampling_contract():
|
|
||||||
"The tail prunes to the final row and samples through an explicit contract.\n\nTags: node, boundary"
|
|
||||||
model = _ReferenceDenseLlama()
|
|
||||||
out = _whole_model_next_token(model, [4, 8, 15, 16, 23])
|
|
||||||
assert isinstance(out, TailOutput)
|
|
||||||
assert out.logits.shape == (1, model.vocab) # tail-only row pruning to last row
|
|
||||||
assert out.sampling.mode == "greedy"
|
|
||||||
assert 0 <= out.token_id < model.vocab
|
|
||||||
assert out.token_id == int(np.argmax(out.logits[0]))
|
|
||||||
|
|
||||||
|
|
||||||
def test_sampling_contract_rejects_uncertified_modes():
|
|
||||||
"Only the certified greedy sampling mode is accepted.\n\nTags: node, boundary"
|
|
||||||
with pytest.raises(BoundaryContractError):
|
|
||||||
SamplingContract(mode="top_p")
|
|
||||||
|
|
||||||
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
# The core parity gate.
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
|
|
||||||
|
|
||||||
def test_two_range_prefill_parity_matches_whole_model():
|
|
||||||
"Whole-model vs two-range prefill produce the same next-token logits and token.\n\nTags: node, boundary, parity"
|
|
||||||
model = _ReferenceDenseLlama()
|
|
||||||
prompt = [5, 12, 3, 41, 7, 19, 2, 33]
|
|
||||||
|
|
||||||
whole = _whole_model_next_token(model, prompt)
|
|
||||||
split = _split_next_token(model, prompt, cut_points=[2])
|
|
||||||
|
|
||||||
assert np.allclose(whole.logits, split.logits, atol=PARITY_ATOL)
|
|
||||||
assert whole.token_id == split.token_id
|
|
||||||
|
|
||||||
|
|
||||||
def test_three_range_prefill_parity_exercises_the_middle_role():
|
|
||||||
"A head/middle/tail split reproduces whole-model prefill through two seams.\n\nTags: node, boundary, parity"
|
|
||||||
model = _ReferenceDenseLlama()
|
|
||||||
prompt = [9, 1, 44, 6, 30, 11]
|
|
||||||
|
|
||||||
whole = _whole_model_next_token(model, prompt)
|
|
||||||
split = _split_next_token(model, prompt, cut_points=[1, 3])
|
|
||||||
|
|
||||||
assert np.allclose(whole.logits, split.logits, atol=PARITY_ATOL)
|
|
||||||
assert whole.token_id == split.token_id
|
|
||||||
|
|
||||||
|
|
||||||
def test_two_range_greedy_decode_parity_matches_whole_model():
|
|
||||||
"Whole-model vs two-range greedy decode produce identical token sequences.\n\nTags: node, boundary, parity"
|
|
||||||
model = _ReferenceDenseLlama()
|
|
||||||
prompt = [2, 17, 8, 25]
|
|
||||||
n_new = 12
|
|
||||||
|
|
||||||
whole_tokens = _greedy_generate(
|
|
||||||
lambda toks: _whole_model_next_token(model, toks), prompt, n_new
|
|
||||||
)
|
|
||||||
split_tokens = _greedy_generate(
|
|
||||||
lambda toks: _split_next_token(model, toks, cut_points=[2]), prompt, n_new
|
|
||||||
)
|
|
||||||
|
|
||||||
assert whole_tokens == split_tokens
|
|
||||||
assert len(whole_tokens) == n_new
|
|
||||||
|
|
||||||
|
|
||||||
def test_boundary_bundle_wire_round_trip_is_exact():
|
|
||||||
"Packing and unpacking the boundary bundle reconstructs the exact arrays.\n\nTags: node, boundary"
|
|
||||||
model = _ReferenceDenseLlama()
|
|
||||||
head = BoundaryAdapter(_ReferenceShard(model, 0, 2))
|
|
||||||
bundle = head.forward(token_ids=[1, 2, 3, 4])
|
|
||||||
assert isinstance(bundle, BoundaryBundle)
|
|
||||||
|
|
||||||
restored = BoundaryBundle.unpack(bundle.pack())
|
|
||||||
assert np.array_equal(restored.residual, bundle.residual)
|
|
||||||
assert np.array_equal(restored.positions, bundle.positions)
|
|
||||||
assert restored.next_layer == bundle.next_layer
|
|
||||||
assert restored.architecture_adapter == bundle.architecture_adapter
|
|
||||||
|
|
||||||
fields = bundle.named_tensor_fields()
|
|
||||||
assert fields["name"] == "residual_stream"
|
|
||||||
assert fields["shape"] == [1, 4, model.hidden]
|
|
||||||
assert fields["byte_order"] in ("little", "big")
|
|
||||||
|
|
||||||
|
|
||||||
def test_alias_architecture_still_parity_matches():
|
|
||||||
"A Shard advertised as 'llama' interoperates with the canonical adapter.\n\nTags: node, boundary, parity"
|
|
||||||
model = _ReferenceDenseLlama()
|
|
||||||
prompt = [7, 3, 22, 5]
|
|
||||||
|
|
||||||
whole = _whole_model_next_token(model, prompt)
|
|
||||||
|
|
||||||
# Head advertises 'LlamaForCausalLM', tail advertises 'llama'; both certify to
|
|
||||||
# the same canonical adapter, so the seam contract still matches.
|
|
||||||
head = BoundaryAdapter(_ReferenceShard(model, 0, 2, architecture_adapter="LlamaForCausalLM"))
|
|
||||||
bundle = head.forward(token_ids=np.asarray(prompt)[None, :])
|
|
||||||
assert isinstance(bundle, BoundaryBundle)
|
|
||||||
tail = BoundaryAdapter(_ReferenceShard(model, 3, 5, architecture_adapter="llama"))
|
|
||||||
split = tail.forward(boundary=BoundaryBundle.unpack(bundle.pack()))
|
|
||||||
assert isinstance(split, TailOutput)
|
|
||||||
|
|
||||||
assert np.allclose(whole.logits, split.logits, atol=PARITY_ATOL)
|
|
||||||
assert whole.token_id == split.token_id
|
|
||||||
@@ -39,14 +39,9 @@ def test_dashboard_served_with_all_panels():
|
|||||||
assert "resolveModelGroup" in html
|
assert "resolveModelGroup" in html
|
||||||
assert "buildModelAliasMap" in html
|
assert "buildModelAliasMap" in html
|
||||||
assert "modelAliasKey(raw)" in html
|
assert "modelAliasKey(raw)" in html
|
||||||
assert "@media (min-width:900px)" in html
|
assert "main { display:grid; grid-template-columns:repeat(auto-fit,minmax(340px,1fr));" in html
|
||||||
assert "grid-template-columns:repeat(4,minmax(0,1fr));" in html
|
|
||||||
assert ".wide { grid-column:span 2; }" in html
|
|
||||||
assert 'onclick="clearConsole()"' in html
|
assert 'onclick="clearConsole()"' in html
|
||||||
assert "let consoleClearedAt = 0;" in html
|
assert "let consoleClearedAt = 0;" in html
|
||||||
assert "max-height:520px; overflow-y:auto; overflow-x:auto;" in html
|
|
||||||
assert "const CONSOLE_MAX_LINES = 1000;" in html
|
|
||||||
assert "events.slice(-CONSOLE_MAX_LINES)" in html
|
|
||||||
finally:
|
finally:
|
||||||
tracker.stop()
|
tracker.stop()
|
||||||
|
|
||||||
@@ -105,39 +100,6 @@ def test_dashboard_allows_admin_to_request_selected_model_load():
|
|||||||
assert '$("request-model-load").style.display = enabled ? "" : "none"' in html
|
assert '$("request-model-load").style.display = enabled ? "" : "none"' in html
|
||||||
|
|
||||||
|
|
||||||
def test_dashboard_exposes_admin_model_inventory_and_release_controls():
|
|
||||||
"Admin placement controls show the full model inventory and can release capacity."
|
|
||||||
html = _dashboard_html()
|
|
||||||
|
|
||||||
assert 'id="admin-model-placement"' in html
|
|
||||||
assert "renderAdminModelPlacement" in html
|
|
||||||
assert '"/v1/models/release"' in html
|
|
||||||
assert "requestAdminModelLoad" in html
|
|
||||||
assert "releaseAdminModel" in html
|
|
||||||
assert 'data-admin-model-load=' in html
|
|
||||||
assert 'data-admin-model-release=' in html
|
|
||||||
assert "admin-model-placement-status" in html
|
|
||||||
assert 'id="admin-node-pool"' in html
|
|
||||||
assert "renderAdminNodePool" in html
|
|
||||||
assert "model drive" in html
|
|
||||||
# RAM and VRAM live in the network-map capacity object, not at node top level.
|
|
||||||
assert "node.ram_bytes = cap.ram_bytes" in html
|
|
||||||
assert "node.vram_bytes = cap.vram_bytes" in html
|
|
||||||
assert 'id="model-placement-dialog"' in html
|
|
||||||
assert "chooseModelPlacementNode" in html
|
|
||||||
assert "node_id: nodeId" in html
|
|
||||||
assert "modelAliasKey(node.model)" in html
|
|
||||||
assert 'id="model-placement-replace"' in html
|
|
||||||
assert 'id="model-placement-confirm"' in html
|
|
||||||
assert 'id="model-placement-replace-error"' in html
|
|
||||||
assert "force: replacing" in html
|
|
||||||
assert "Tick the box to confirm" in html
|
|
||||||
assert "releaseAllNodeModels" in html
|
|
||||||
assert '"/v1/nodes/release-all"' in html
|
|
||||||
assert "model RAM" in html
|
|
||||||
assert "loaded_model_bytes" in html
|
|
||||||
|
|
||||||
|
|
||||||
def test_network_map_includes_node_friendly_name():
|
def test_network_map_includes_node_friendly_name():
|
||||||
"Network map includes node friendly name\n\nTags: dashboard, http"
|
"Network map includes node friendly name\n\nTags: dashboard, http"
|
||||||
tracker = TrackerServer()
|
tracker = TrackerServer()
|
||||||
|
|||||||
@@ -355,75 +355,6 @@ def test_admin_model_load_request_queues_directive_on_joined_node():
|
|||||||
assert heartbeat["directives"][0]["model"] == "Qwen/Qwen2.5-0.5B-Instruct"
|
assert heartbeat["directives"][0]["model"] == "Qwen/Qwen2.5-0.5B-Instruct"
|
||||||
|
|
||||||
|
|
||||||
def test_admin_can_replace_a_served_model_and_release_it():
|
|
||||||
"Forced admin placement replaces a served shard; release queues DROP_SHARD."
|
|
||||||
tracker = TrackerServer(enable_billing=False, validator_service_token="test-admin")
|
|
||||||
port = tracker.start()
|
|
||||||
try:
|
|
||||||
node = _post_json(
|
|
||||||
f"http://127.0.0.1:{port}/v1/nodes/register",
|
|
||||||
{"endpoint": "http://127.0.0.1:9912", "model": "stub-model",
|
|
||||||
"shard_start": 0, "shard_end": 3, "managed_assignment": True,
|
|
||||||
"max_loaded_shards": 1, "memory_mb": 1,
|
|
||||||
"hardware_profile": {"host_id": "full-host"}},
|
|
||||||
)
|
|
||||||
headers = {"Content-Type": "application/json", "Authorization": "Bearer test-admin"}
|
|
||||||
load = urllib.request.Request(
|
|
||||||
f"http://127.0.0.1:{port}/v1/models/load",
|
|
||||||
data=json.dumps({
|
|
||||||
"model": "qwen2.5-0.5b-instruct",
|
|
||||||
"node_id": node["node_id"],
|
|
||||||
"force": True,
|
|
||||||
}).encode(),
|
|
||||||
headers=headers, method="POST")
|
|
||||||
with urllib.request.urlopen(load) as response:
|
|
||||||
assert json.loads(response.read())["assignment"]["node_id"] == node["node_id"]
|
|
||||||
heartbeat = _post_json(f"http://127.0.0.1:{port}/v1/nodes/{node['node_id']}/heartbeat", {})
|
|
||||||
_post_json(
|
|
||||||
f"http://127.0.0.1:{port}/v1/nodes/{node['node_id']}/heartbeat",
|
|
||||||
{"completed_directives": [{"action": "DROP_SHARD", "model": "Qwen/Qwen2.5-0.5B-Instruct"}]},
|
|
||||||
)
|
|
||||||
network = _get_json(f"http://127.0.0.1:{port}/v1/network/map")
|
|
||||||
assert heartbeat["directives"][0]["action"] == "LOAD_SHARD"
|
|
||||||
release = urllib.request.Request(
|
|
||||||
f"http://127.0.0.1:{port}/v1/models/release",
|
|
||||||
data=json.dumps({"model": "qwen2.5-0.5b-instruct"}).encode(), headers=headers, method="POST")
|
|
||||||
with urllib.request.urlopen(release) as response:
|
|
||||||
assert json.loads(response.read())["released"] == 1
|
|
||||||
heartbeat = _post_json(f"http://127.0.0.1:{port}/v1/nodes/{node['node_id']}/heartbeat", {})
|
|
||||||
finally:
|
|
||||||
tracker.stop()
|
|
||||||
|
|
||||||
assert heartbeat["directives"][0]["action"] == "DROP_SHARD"
|
|
||||||
released_node = next(item for item in network["nodes"] if item["node_id"] == node["node_id"])
|
|
||||||
assert released_node["shard_start"] is None
|
|
||||||
assert released_node["shard_end"] is None
|
|
||||||
|
|
||||||
|
|
||||||
def test_models_list_does_not_duplicate_a_preset_registered_by_hf_repo():
|
|
||||||
"""A preset and its canonical repository are one selectable model."""
|
|
||||||
tracker = TrackerServer(enable_billing=False)
|
|
||||||
port = tracker.start()
|
|
||||||
try:
|
|
||||||
_post_json(
|
|
||||||
f"http://127.0.0.1:{port}/v1/nodes/register",
|
|
||||||
{
|
|
||||||
"endpoint": "http://127.0.0.1:9913",
|
|
||||||
"model": "Qwen2.5-0.5B-Instruct",
|
|
||||||
"hf_repo": "Qwen/Qwen2.5-0.5B-Instruct",
|
|
||||||
"num_layers": 24,
|
|
||||||
"shard_start": 0,
|
|
||||||
"shard_end": 23,
|
|
||||||
},
|
|
||||||
)
|
|
||||||
models = _get_json(f"http://127.0.0.1:{port}/v1/models")["data"]
|
|
||||||
finally:
|
|
||||||
tracker.stop()
|
|
||||||
|
|
||||||
assert [model["id"] for model in models].count("qwen2.5-0.5b-instruct") == 1
|
|
||||||
assert not any(model["id"] == "Qwen/Qwen2.5-0.5B-Instruct" for model in models)
|
|
||||||
|
|
||||||
|
|
||||||
def test_endpoint_key_distinguishes_same_port_different_hosts():
|
def test_endpoint_key_distinguishes_same_port_different_hosts():
|
||||||
"Endpoint key distinguishes same port different hosts\n\nTags: http, performance, routing, tracker"
|
"Endpoint key distinguishes same port different hosts\n\nTags: http, performance, routing, tracker"
|
||||||
from meshnet_node.torch_server import _clamp_downstream_hops, _endpoint_key
|
from meshnet_node.torch_server import _clamp_downstream_hops, _endpoint_key
|
||||||
|
|||||||
@@ -1,186 +0,0 @@
|
|||||||
"""Tests for the GGUF backend adapter and recipe-gated startup seam."""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from types import SimpleNamespace
|
|
||||||
|
|
||||||
from meshnet_node.gguf_backend import GgufNodeBackend, build_gguf_backend
|
|
||||||
from meshnet_node.model_backend import TailTokenResult, TensorPayload
|
|
||||||
from meshnet_node.recipe_manifest import DEFAULT_RECIPE_ID, load_recipe_manifest
|
|
||||||
from meshnet_node.startup import _gguf_backend_for_recipe
|
|
||||||
|
|
||||||
|
|
||||||
class _RecordingTransport:
|
|
||||||
def __init__(self) -> None:
|
|
||||||
self.calls: list[tuple[str, tuple, dict]] = []
|
|
||||||
|
|
||||||
def encode_prompt(self, prompt: str, session_id: str | None = None):
|
|
||||||
self.calls.append(("encode_prompt", (prompt, session_id), {}))
|
|
||||||
return TensorPayload(
|
|
||||||
body=b"\x00" * 16,
|
|
||||||
shape=[1, 2, 4],
|
|
||||||
attention_mask_header=None,
|
|
||||||
position_ids_header=None,
|
|
||||||
)
|
|
||||||
|
|
||||||
def encode_next_token(self, token_id: int, session_id: str):
|
|
||||||
self.calls.append(("encode_next_token", (token_id, session_id), {}))
|
|
||||||
return TensorPayload(
|
|
||||||
body=b"\x00" * 8,
|
|
||||||
shape=[1, 1, 4],
|
|
||||||
attention_mask_header=None,
|
|
||||||
position_ids_header=None,
|
|
||||||
past_len=2,
|
|
||||||
)
|
|
||||||
|
|
||||||
def forward_bytes(
|
|
||||||
self,
|
|
||||||
body: bytes,
|
|
||||||
shape: list[int],
|
|
||||||
attention_mask_header: str | None,
|
|
||||||
position_ids_header: str | None,
|
|
||||||
*,
|
|
||||||
start_layer: int | None = None,
|
|
||||||
session_id: str | None = None,
|
|
||||||
cache_mode: str | None = None,
|
|
||||||
past_len: int | None = None,
|
|
||||||
):
|
|
||||||
self.calls.append(
|
|
||||||
(
|
|
||||||
"forward_bytes",
|
|
||||||
(body, tuple(shape), attention_mask_header, position_ids_header),
|
|
||||||
{
|
|
||||||
"start_layer": start_layer,
|
|
||||||
"session_id": session_id,
|
|
||||||
"cache_mode": cache_mode,
|
|
||||||
"past_len": past_len,
|
|
||||||
},
|
|
||||||
)
|
|
||||||
)
|
|
||||||
if cache_mode == "decode":
|
|
||||||
return TailTokenResult(text=" done", token_id=17)
|
|
||||||
return TensorPayload(
|
|
||||||
body=b"\x00" * 16,
|
|
||||||
shape=[1, 2, 4],
|
|
||||||
attention_mask_header=attention_mask_header,
|
|
||||||
position_ids_header=position_ids_header,
|
|
||||||
past_len=past_len,
|
|
||||||
)
|
|
||||||
|
|
||||||
def decode_tail_token(self, hidden_states):
|
|
||||||
self.calls.append(("decode_tail_token", (hidden_states.shape,), {}))
|
|
||||||
return TailTokenResult(text=" tail", token_id=19)
|
|
||||||
|
|
||||||
def generate_text(self, messages, max_new_tokens=5120, temperature=1.0, top_p=1.0):
|
|
||||||
self.calls.append(("generate_text", (tuple(messages), max_new_tokens, temperature, top_p), {}))
|
|
||||||
return "ok"
|
|
||||||
|
|
||||||
def generate_text_streaming(self, messages, max_new_tokens=5120, temperature=1.0, top_p=1.0):
|
|
||||||
self.calls.append(("generate_text_streaming", (tuple(messages), max_new_tokens, temperature, top_p), {}))
|
|
||||||
yield "ok"
|
|
||||||
|
|
||||||
def count_prompt_tokens(self, messages):
|
|
||||||
self.calls.append(("count_prompt_tokens", (tuple(messages),), {}))
|
|
||||||
return 3
|
|
||||||
|
|
||||||
def count_text_tokens(self, text):
|
|
||||||
self.calls.append(("count_text_tokens", (text,), {}))
|
|
||||||
return 2
|
|
||||||
|
|
||||||
def eos_token_ids(self):
|
|
||||||
self.calls.append(("eos_token_ids", (), {}))
|
|
||||||
return [19]
|
|
||||||
|
|
||||||
def release_session(self, session_id: str) -> None:
|
|
||||||
self.calls.append(("release_session", (session_id,), {}))
|
|
||||||
|
|
||||||
|
|
||||||
def test_build_gguf_backend_delegates_to_transport():
|
|
||||||
transport = _RecordingTransport()
|
|
||||||
backend = build_gguf_backend(
|
|
||||||
model_id="meshnet/native-model",
|
|
||||||
shard_start=0,
|
|
||||||
shard_end=1,
|
|
||||||
quantization="bfloat16",
|
|
||||||
transport=transport,
|
|
||||||
total_layers=2,
|
|
||||||
device_type="cpu",
|
|
||||||
)
|
|
||||||
|
|
||||||
assert isinstance(backend, GgufNodeBackend)
|
|
||||||
assert backend.backend_id == "llama.cpp"
|
|
||||||
assert backend.is_head is True
|
|
||||||
assert backend.is_tail is True
|
|
||||||
assert backend.model.config.to_dict()["architecture_adapter"] == "dense-llama"
|
|
||||||
assert backend.loaded_tensor_names[0] == "blk.0.weight"
|
|
||||||
|
|
||||||
prompt = backend.encode_prompt("hello", session_id="session-1")
|
|
||||||
assert prompt.shape == [1, 2, 4]
|
|
||||||
|
|
||||||
decode = backend.forward_bytes(
|
|
||||||
b"\x00" * 16,
|
|
||||||
[1, 2, 4],
|
|
||||||
None,
|
|
||||||
None,
|
|
||||||
session_id="session-1",
|
|
||||||
cache_mode="decode",
|
|
||||||
past_len=2,
|
|
||||||
)
|
|
||||||
assert isinstance(decode, TailTokenResult)
|
|
||||||
assert decode.token_id == 17
|
|
||||||
|
|
||||||
backend.release_session("session-1")
|
|
||||||
|
|
||||||
assert [call[0] for call in transport.calls] == [
|
|
||||||
"encode_prompt",
|
|
||||||
"forward_bytes",
|
|
||||||
"release_session",
|
|
||||||
]
|
|
||||||
assert transport.calls[0][1] == ("hello", "session-1")
|
|
||||||
assert transport.calls[1][2]["cache_mode"] == "decode"
|
|
||||||
assert transport.calls[1][2]["past_len"] == 2
|
|
||||||
|
|
||||||
|
|
||||||
def test_recipe_gates_native_backend_selection(monkeypatch):
|
|
||||||
manifest = load_recipe_manifest()
|
|
||||||
torch_recipe = manifest.require(DEFAULT_RECIPE_ID)
|
|
||||||
native_recipe = manifest.require("llama-cpp-native")
|
|
||||||
|
|
||||||
sentinel_backend = object()
|
|
||||||
calls: list[dict] = []
|
|
||||||
|
|
||||||
def fake_build_gguf_backend(**kwargs):
|
|
||||||
calls.append(kwargs)
|
|
||||||
return sentinel_backend
|
|
||||||
|
|
||||||
monkeypatch.setattr(
|
|
||||||
"meshnet_node.startup.build_gguf_backend",
|
|
||||||
fake_build_gguf_backend,
|
|
||||||
)
|
|
||||||
|
|
||||||
assert _gguf_backend_for_recipe(
|
|
||||||
torch_recipe,
|
|
||||||
model_id="meshnet/native-model",
|
|
||||||
shard_start=0,
|
|
||||||
shard_end=1,
|
|
||||||
quantization="bfloat16",
|
|
||||||
total_layers=2,
|
|
||||||
device="cpu",
|
|
||||||
) is None
|
|
||||||
|
|
||||||
backend = _gguf_backend_for_recipe(
|
|
||||||
native_recipe,
|
|
||||||
model_id="meshnet/native-model",
|
|
||||||
shard_start=0,
|
|
||||||
shard_end=1,
|
|
||||||
quantization="bfloat16",
|
|
||||||
total_layers=2,
|
|
||||||
device="cpu",
|
|
||||||
)
|
|
||||||
|
|
||||||
assert backend is sentinel_backend
|
|
||||||
assert calls[0]["model_id"] == "meshnet/native-model"
|
|
||||||
assert calls[0]["shard_start"] == 0
|
|
||||||
assert calls[0]["shard_end"] == 1
|
|
||||||
assert calls[0]["quantization"] == "bfloat16"
|
|
||||||
assert calls[0]["total_layers"] == 2
|
|
||||||
@@ -1,88 +0,0 @@
|
|||||||
"""Dense-Llama GGUF ownership selection and introspection tests."""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
|
|
||||||
from meshnet_node.gguf_ownership import (
|
|
||||||
DenseLlamaShardOwnership,
|
|
||||||
authoritative_dense_llama_ownership,
|
|
||||||
infer_dense_llama_ownership,
|
|
||||||
select_dense_llama_tensor_names,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def test_dense_llama_selection_only_picks_block_range_and_endpoints():
|
|
||||||
"Dense-Llama selection keeps only the owned blocks plus the correct endpoints.\n\nTags: node, GGUF"
|
|
||||||
tensor_inventory = {
|
|
||||||
"token_embd.weight": 10_000,
|
|
||||||
"blk.0.attn_q.weight": 1_000,
|
|
||||||
"blk.0.ffn_down.weight": 1_000,
|
|
||||||
"blk.1.attn_q.weight": 2_000,
|
|
||||||
"blk.1.ffn_down.weight": 2_000,
|
|
||||||
"blk.2.attn_q.weight": 3_000,
|
|
||||||
"blk.2.ffn_down.weight": 3_000,
|
|
||||||
"output_norm.weight": 256,
|
|
||||||
"output.weight": 10_000,
|
|
||||||
"rope.freqs": 128,
|
|
||||||
}
|
|
||||||
|
|
||||||
selected = select_dense_llama_tensor_names(
|
|
||||||
tensor_inventory,
|
|
||||||
1,
|
|
||||||
2,
|
|
||||||
total_layers=3,
|
|
||||||
)
|
|
||||||
|
|
||||||
assert selected == {
|
|
||||||
"blk.1.attn_q.weight",
|
|
||||||
"blk.1.ffn_down.weight",
|
|
||||||
"blk.2.attn_q.weight",
|
|
||||||
"blk.2.ffn_down.weight",
|
|
||||||
"output_norm.weight",
|
|
||||||
"output.weight",
|
|
||||||
}
|
|
||||||
|
|
||||||
selected_bytes = sum(tensor_inventory[name] for name in selected)
|
|
||||||
full_bytes = sum(tensor_inventory.values())
|
|
||||||
assert selected_bytes == 20_256
|
|
||||||
assert selected_bytes < full_bytes
|
|
||||||
|
|
||||||
|
|
||||||
def test_dense_llama_loaded_range_is_authoritative_from_tensor_inventory():
|
|
||||||
"The backend's loaded tensor inventory is the source of truth for range and ownership.\n\nTags: node, GGUF"
|
|
||||||
|
|
||||||
class Backend:
|
|
||||||
loaded_tensor_names = (
|
|
||||||
"token_embd.weight",
|
|
||||||
"blk.4.attn_q.weight",
|
|
||||||
"blk.5.ffn_down.weight",
|
|
||||||
"output_norm.weight",
|
|
||||||
"output.weight",
|
|
||||||
)
|
|
||||||
|
|
||||||
ownership = authoritative_dense_llama_ownership(Backend(), selection=None)
|
|
||||||
|
|
||||||
assert isinstance(ownership, DenseLlamaShardOwnership)
|
|
||||||
assert ownership.range == (4, 5)
|
|
||||||
assert ownership.owns_embedding is True
|
|
||||||
assert ownership.owns_final_head is True
|
|
||||||
|
|
||||||
|
|
||||||
def test_derivative_slice_requires_source_and_slice_hashes():
|
|
||||||
"Temporary derivative GGUF slices must carry hashes and cannot claim final semantics.\n\nTags: node, GGUF"
|
|
||||||
with pytest.raises(ValueError, match="source and slice hashes"):
|
|
||||||
infer_dense_llama_ownership(
|
|
||||||
["blk.1.attn_q.weight"],
|
|
||||||
derivative_slice=True,
|
|
||||||
final_artifact_semantics=False,
|
|
||||||
)
|
|
||||||
|
|
||||||
with pytest.raises(ValueError, match="final artifacts"):
|
|
||||||
infer_dense_llama_ownership(
|
|
||||||
["blk.1.attn_q.weight"],
|
|
||||||
source_artifact_hash="sha256:source",
|
|
||||||
slice_artifact_hash="sha256:slice",
|
|
||||||
derivative_slice=True,
|
|
||||||
final_artifact_semantics=True,
|
|
||||||
)
|
|
||||||
@@ -1,769 +0,0 @@
|
|||||||
"""Isolated concurrent local Hot KV State (DGR-007).
|
|
||||||
|
|
||||||
These tests prove the KV/session manager with a *pure-numpy* KV-cached dense-Llama
|
|
||||||
reference: no download, no GPU, no torch, no API credit. The reference implements
|
|
||||||
the DGR-006 ``ShardComputation`` duck type plus ``run_layers_cached`` so cached
|
|
||||||
prefill/decode over a per-session KV context reproduces the stateless whole-model
|
|
||||||
tokens bit-for-bit. On top of that correctness core, the tests exercise the
|
|
||||||
manager's lifecycle: owned-layer allocation, prefill/decode append, truncate,
|
|
||||||
release, TTL/LRU eviction, explicit cache-miss responses, stale-epoch and
|
|
||||||
incompatible-recipe rejection, four concurrent cross-talk-free sessions, and
|
|
||||||
budget-bounded cancellation.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import threading
|
|
||||||
|
|
||||||
import numpy as np
|
|
||||||
import pytest
|
|
||||||
|
|
||||||
from meshnet_node.boundary_adapter import BoundaryBundle, TailOutput
|
|
||||||
from meshnet_node.hot_kv_state import (
|
|
||||||
CacheMiss,
|
|
||||||
CacheMissReason,
|
|
||||||
HotKvStateConfig,
|
|
||||||
HotKvStateManager,
|
|
||||||
IncompatibleCacheRecipeError,
|
|
||||||
KvBoundaryAdapter,
|
|
||||||
KvBudgetExceededError,
|
|
||||||
KvCacheMissError,
|
|
||||||
KvCacheRecipe,
|
|
||||||
LayerKvCache,
|
|
||||||
StaleRouteEpochError,
|
|
||||||
kv_recipe_for,
|
|
||||||
)
|
|
||||||
|
|
||||||
PARITY_ATOL = 1e-6
|
|
||||||
|
|
||||||
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
# Pure-numpy KV-cached dense-Llama reference (test fixture, not production).
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
|
|
||||||
|
|
||||||
class _KvDenseLlama:
|
|
||||||
"""A tiny deterministic dense-Llama with both stateless and cached runners."""
|
|
||||||
|
|
||||||
architecture_adapter = "dense-llama"
|
|
||||||
|
|
||||||
def __init__(
|
|
||||||
self,
|
|
||||||
*,
|
|
||||||
vocab: int = 48,
|
|
||||||
hidden: int = 32,
|
|
||||||
n_layers: int = 6,
|
|
||||||
n_heads: int = 4,
|
|
||||||
intermediate: int = 64,
|
|
||||||
rms_eps: float = 1e-6,
|
|
||||||
rope_theta: float = 10000.0,
|
|
||||||
seed: int = 20260716,
|
|
||||||
) -> None:
|
|
||||||
assert hidden % n_heads == 0
|
|
||||||
self.vocab = vocab
|
|
||||||
self.hidden = hidden
|
|
||||||
self.n_layers = n_layers
|
|
||||||
self.n_heads = n_heads
|
|
||||||
self.head_dim = hidden // n_heads
|
|
||||||
assert self.head_dim % 2 == 0
|
|
||||||
self.rms_eps = rms_eps
|
|
||||||
self.rope_theta = rope_theta
|
|
||||||
|
|
||||||
rng = np.random.default_rng(seed)
|
|
||||||
|
|
||||||
def w(*shape: int) -> np.ndarray:
|
|
||||||
return (rng.standard_normal(shape) * 0.08).astype(np.float32)
|
|
||||||
|
|
||||||
self.embed = w(vocab, hidden)
|
|
||||||
self.layers = []
|
|
||||||
for _ in range(n_layers):
|
|
||||||
self.layers.append(
|
|
||||||
{
|
|
||||||
"in_ln": (1.0 + rng.standard_normal(hidden) * 0.02).astype(np.float32),
|
|
||||||
"q": w(hidden, hidden),
|
|
||||||
"k": w(hidden, hidden),
|
|
||||||
"v": w(hidden, hidden),
|
|
||||||
"o": w(hidden, hidden),
|
|
||||||
"post_ln": (1.0 + rng.standard_normal(hidden) * 0.02).astype(np.float32),
|
|
||||||
"gate": w(intermediate, hidden),
|
|
||||||
"up": w(intermediate, hidden),
|
|
||||||
"down": w(hidden, intermediate),
|
|
||||||
}
|
|
||||||
)
|
|
||||||
self.final_ln = (1.0 + rng.standard_normal(hidden) * 0.02).astype(np.float32)
|
|
||||||
self.lm_head_w = w(vocab, hidden)
|
|
||||||
|
|
||||||
inv_freq = 1.0 / (
|
|
||||||
rope_theta ** (np.arange(0, self.head_dim, 2, dtype=np.float32) / self.head_dim)
|
|
||||||
)
|
|
||||||
self.inv_freq = inv_freq.astype(np.float32)
|
|
||||||
|
|
||||||
# -- primitive ops -----------------------------------------------------
|
|
||||||
def _rmsnorm(self, x: np.ndarray, weight: np.ndarray) -> np.ndarray:
|
|
||||||
variance = np.mean(x.astype(np.float32) ** 2, axis=-1, keepdims=True)
|
|
||||||
normed = x / np.sqrt(variance + self.rms_eps)
|
|
||||||
return (normed * weight).astype(np.float32)
|
|
||||||
|
|
||||||
def _rope(self, positions: np.ndarray):
|
|
||||||
angles = positions[..., None].astype(np.float32) * self.inv_freq[None, None, :]
|
|
||||||
emb = np.concatenate([angles, angles], axis=-1)
|
|
||||||
return np.cos(emb).astype(np.float32), np.sin(emb).astype(np.float32)
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _rotate_half(x: np.ndarray) -> np.ndarray:
|
|
||||||
half = x.shape[-1] // 2
|
|
||||||
return np.concatenate([-x[..., half:], x[..., :half]], axis=-1)
|
|
||||||
|
|
||||||
def _apply_rope(self, t: np.ndarray, cos: np.ndarray, sin: np.ndarray) -> np.ndarray:
|
|
||||||
cos = cos[:, None, :, :]
|
|
||||||
sin = sin[:, None, :, :]
|
|
||||||
return t * cos + self._rotate_half(t) * sin
|
|
||||||
|
|
||||||
def _project_qkv(self, normed: np.ndarray, layer: dict, positions: np.ndarray):
|
|
||||||
batch, seq, _ = normed.shape
|
|
||||||
q = (normed @ layer["q"].T).reshape(batch, seq, self.n_heads, self.head_dim)
|
|
||||||
k = (normed @ layer["k"].T).reshape(batch, seq, self.n_heads, self.head_dim)
|
|
||||||
v = (normed @ layer["v"].T).reshape(batch, seq, self.n_heads, self.head_dim)
|
|
||||||
q = q.transpose(0, 2, 1, 3)
|
|
||||||
k = k.transpose(0, 2, 1, 3)
|
|
||||||
v = v.transpose(0, 2, 1, 3)
|
|
||||||
cos, sin = self._rope(positions)
|
|
||||||
q = self._apply_rope(q, cos, sin)
|
|
||||||
k = self._apply_rope(k, cos, sin)
|
|
||||||
return q, k, v
|
|
||||||
|
|
||||||
def _attend(
|
|
||||||
self,
|
|
||||||
q: np.ndarray,
|
|
||||||
k_all: np.ndarray,
|
|
||||||
v_all: np.ndarray,
|
|
||||||
layer: dict,
|
|
||||||
q_positions: np.ndarray,
|
|
||||||
) -> np.ndarray:
|
|
||||||
batch, _, seq_new, _ = q.shape
|
|
||||||
total = k_all.shape[2]
|
|
||||||
scores = (q @ k_all.transpose(0, 1, 3, 2)) / np.sqrt(self.head_dim)
|
|
||||||
# Causal mask by absolute position: keys are stored in absolute order
|
|
||||||
# 0..total-1; query row i lives at absolute position q_positions[i].
|
|
||||||
key_abs = np.arange(total, dtype=np.int64)
|
|
||||||
q_abs = np.asarray(q_positions).reshape(seq_new).astype(np.int64)
|
|
||||||
mask = np.where(key_abs[None, :] <= q_abs[:, None], 0.0, -1e30).astype(np.float32)
|
|
||||||
scores = scores + mask[None, None, :, :]
|
|
||||||
scores = scores - scores.max(axis=-1, keepdims=True)
|
|
||||||
weights = np.exp(scores)
|
|
||||||
weights = weights / weights.sum(axis=-1, keepdims=True)
|
|
||||||
out = weights @ v_all
|
|
||||||
out = out.transpose(0, 2, 1, 3).reshape(batch, seq_new, self.hidden)
|
|
||||||
return (out @ layer["o"].T).astype(np.float32)
|
|
||||||
|
|
||||||
def _mlp(self, x: np.ndarray, layer: dict) -> np.ndarray:
|
|
||||||
gate = x @ layer["gate"].T
|
|
||||||
up = x @ layer["up"].T
|
|
||||||
silu = gate * (1.0 / (1.0 + np.exp(-gate)))
|
|
||||||
return ((silu * up) @ layer["down"].T).astype(np.float32)
|
|
||||||
|
|
||||||
# -- stateless whole-sequence layer (ground truth) ---------------------
|
|
||||||
def _run_layer_stateless(self, x: np.ndarray, layer: dict, positions: np.ndarray) -> np.ndarray:
|
|
||||||
normed = self._rmsnorm(x, layer["in_ln"])
|
|
||||||
q, k, v = self._project_qkv(normed, layer, positions)
|
|
||||||
attn = self._attend(q, k, v, layer, positions[0])
|
|
||||||
h = x + attn
|
|
||||||
h = h + self._mlp(self._rmsnorm(h, layer["post_ln"]), layer)
|
|
||||||
return h.astype(np.float32)
|
|
||||||
|
|
||||||
def whole_model_next_token(self, token_ids: list[int]) -> int:
|
|
||||||
positions = np.arange(len(token_ids))[None, :]
|
|
||||||
h = self.embed[np.asarray(token_ids)][None, :]
|
|
||||||
for idx in range(self.n_layers):
|
|
||||||
h = self._run_layer_stateless(h, self.layers[idx], positions)
|
|
||||||
h = self._rmsnorm(h[:, -1:, :], self.final_ln)
|
|
||||||
logits = h @ self.lm_head_w.T
|
|
||||||
return int(np.argmax(logits[0, -1]))
|
|
||||||
|
|
||||||
def stateless_greedy(self, prompt: list[int], n_new: int) -> list[int]:
|
|
||||||
tokens = list(prompt)
|
|
||||||
out: list[int] = []
|
|
||||||
for _ in range(n_new):
|
|
||||||
tok = self.whole_model_next_token(tokens)
|
|
||||||
tokens.append(tok)
|
|
||||||
out.append(tok)
|
|
||||||
return out
|
|
||||||
|
|
||||||
|
|
||||||
class _KvReferenceShard:
|
|
||||||
"""A contiguous inclusive layer range with a KV-cached runner.
|
|
||||||
|
|
||||||
Satisfies the KV-aware ``ShardComputation`` duck type used by
|
|
||||||
``KvBoundaryAdapter``: DGR-006 methods plus ``run_layers_cached`` and the KV
|
|
||||||
geometry (``n_kv_heads`` / ``head_dim`` / ``kv_dtype``).
|
|
||||||
"""
|
|
||||||
|
|
||||||
kv_dtype = "float32"
|
|
||||||
|
|
||||||
def __init__(
|
|
||||||
self,
|
|
||||||
model: _KvDenseLlama,
|
|
||||||
start_layer: int,
|
|
||||||
end_layer: int,
|
|
||||||
*,
|
|
||||||
architecture_adapter: str | None = None,
|
|
||||||
) -> None:
|
|
||||||
self._model = model
|
|
||||||
self.start_layer = start_layer
|
|
||||||
self.end_layer = end_layer
|
|
||||||
self.total_layers = model.n_layers
|
|
||||||
self.n_kv_heads = model.n_heads
|
|
||||||
self.head_dim = model.head_dim
|
|
||||||
self.architecture_adapter = architecture_adapter or model.architecture_adapter
|
|
||||||
|
|
||||||
def embed_tokens(self, token_ids: np.ndarray) -> np.ndarray:
|
|
||||||
return self._model.embed[np.asarray(token_ids)]
|
|
||||||
|
|
||||||
def final_norm(self, hidden: np.ndarray) -> np.ndarray:
|
|
||||||
return self._model._rmsnorm(np.asarray(hidden, dtype=np.float32), self._model.final_ln)
|
|
||||||
|
|
||||||
def lm_head(self, hidden: np.ndarray) -> np.ndarray:
|
|
||||||
return np.asarray(hidden, dtype=np.float32) @ self._model.lm_head_w.T
|
|
||||||
|
|
||||||
def run_layers_cached(self, hidden, *, positions, past_kv):
|
|
||||||
m = self._model
|
|
||||||
x = np.asarray(hidden, dtype=np.float32)
|
|
||||||
positions = np.asarray(positions)
|
|
||||||
new_kv: dict[int, tuple[np.ndarray, np.ndarray]] = {}
|
|
||||||
for idx in range(self.start_layer, self.end_layer + 1):
|
|
||||||
layer = m.layers[idx]
|
|
||||||
normed = m._rmsnorm(x, layer["in_ln"])
|
|
||||||
q, k, v = m._project_qkv(normed, layer, positions)
|
|
||||||
# Post-RoPE new K/V stored as (seq_new, n_heads, head_dim).
|
|
||||||
new_k = k[0].transpose(1, 0, 2).copy()
|
|
||||||
new_v = v[0].transpose(1, 0, 2).copy()
|
|
||||||
cache = past_kv.get(idx)
|
|
||||||
if cache is not None and cache.length > 0:
|
|
||||||
past_k = cache.keys[None].transpose(0, 2, 1, 3)
|
|
||||||
past_v = cache.values[None].transpose(0, 2, 1, 3)
|
|
||||||
k_all = np.concatenate([past_k, k], axis=2)
|
|
||||||
v_all = np.concatenate([past_v, v], axis=2)
|
|
||||||
else:
|
|
||||||
k_all, v_all = k, v
|
|
||||||
attn = m._attend(q, k_all, v_all, layer, positions[0])
|
|
||||||
h = x + attn
|
|
||||||
x = h + m._mlp(m._rmsnorm(h, layer["post_ln"]), layer)
|
|
||||||
x = x.astype(np.float32)
|
|
||||||
new_kv[idx] = (new_k, new_v)
|
|
||||||
return x, new_kv
|
|
||||||
|
|
||||||
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
# Helpers.
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
|
|
||||||
|
|
||||||
class _FakeClock:
|
|
||||||
def __init__(self) -> None:
|
|
||||||
self.now = 0.0
|
|
||||||
|
|
||||||
def __call__(self) -> float:
|
|
||||||
return self.now
|
|
||||||
|
|
||||||
def advance(self, delta: float) -> None:
|
|
||||||
self.now += delta
|
|
||||||
|
|
||||||
|
|
||||||
def _full_shard(model: _KvDenseLlama):
|
|
||||||
return _KvReferenceShard(model, 0, model.n_layers - 1)
|
|
||||||
|
|
||||||
|
|
||||||
def _manager_for(shard, config: HotKvStateConfig | None = None, clock=None) -> HotKvStateManager:
|
|
||||||
return HotKvStateManager(kv_recipe_for(shard), config=config, clock=clock)
|
|
||||||
|
|
||||||
|
|
||||||
def _cached_greedy(
|
|
||||||
adapter: KvBoundaryAdapter,
|
|
||||||
manager: HotKvStateManager,
|
|
||||||
session_id: str,
|
|
||||||
epoch: int,
|
|
||||||
prompt: list[int],
|
|
||||||
n_new: int,
|
|
||||||
) -> list[int]:
|
|
||||||
"""Greedy decode one full-model session through the KV manager."""
|
|
||||||
out = adapter.prefill(session_id, epoch, token_ids=np.asarray(prompt))
|
|
||||||
assert isinstance(out, TailOutput)
|
|
||||||
tokens = [out.token_id]
|
|
||||||
for _ in range(n_new - 1):
|
|
||||||
step = adapter.decode(session_id, epoch, token_ids=[out.token_id])
|
|
||||||
assert isinstance(step, TailOutput)
|
|
||||||
out = step
|
|
||||||
tokens.append(out.token_id)
|
|
||||||
return tokens
|
|
||||||
|
|
||||||
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
# Recipe identity.
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
|
|
||||||
|
|
||||||
def test_recipe_owned_layers_and_fingerprint_aliasing():
|
|
||||||
"The KV recipe covers only owned layers and canonicalizes architecture aliases.\n\nTags: node, kv"
|
|
||||||
recipe = KvCacheRecipe(
|
|
||||||
architecture_adapter="LlamaForCausalLM",
|
|
||||||
kv_dtype="float32",
|
|
||||||
n_kv_heads=4,
|
|
||||||
head_dim=8,
|
|
||||||
total_layers=6,
|
|
||||||
start_layer=2,
|
|
||||||
end_layer=3,
|
|
||||||
)
|
|
||||||
assert recipe.owned_layers == (2, 3)
|
|
||||||
alias = KvCacheRecipe(
|
|
||||||
architecture_adapter="llama",
|
|
||||||
kv_dtype="float32",
|
|
||||||
n_kv_heads=4,
|
|
||||||
head_dim=8,
|
|
||||||
total_layers=6,
|
|
||||||
start_layer=2,
|
|
||||||
end_layer=3,
|
|
||||||
)
|
|
||||||
assert recipe.is_compatible(alias)
|
|
||||||
# A different owned range is not compatible.
|
|
||||||
other = KvCacheRecipe(
|
|
||||||
architecture_adapter="llama",
|
|
||||||
kv_dtype="float32",
|
|
||||||
n_kv_heads=4,
|
|
||||||
head_dim=8,
|
|
||||||
total_layers=6,
|
|
||||||
start_layer=0,
|
|
||||||
end_layer=1,
|
|
||||||
)
|
|
||||||
assert not recipe.is_compatible(other)
|
|
||||||
|
|
||||||
|
|
||||||
def test_recipe_bytes_per_token_scales_with_owned_layers():
|
|
||||||
"KV bytes-per-token counts keys+values across owned layers only.\n\nTags: node, kv"
|
|
||||||
base = dict(
|
|
||||||
architecture_adapter="dense-llama",
|
|
||||||
kv_dtype="float32",
|
|
||||||
n_kv_heads=4,
|
|
||||||
head_dim=8,
|
|
||||||
total_layers=6,
|
|
||||||
)
|
|
||||||
one = KvCacheRecipe(**base, start_layer=0, end_layer=0)
|
|
||||||
two = KvCacheRecipe(**base, start_layer=0, end_layer=1)
|
|
||||||
# 2 (k+v) * heads * dim * 4 bytes per layer.
|
|
||||||
assert one.bytes_per_token() == 2 * 4 * 8 * 4
|
|
||||||
assert two.bytes_per_token() == 2 * one.bytes_per_token()
|
|
||||||
|
|
||||||
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
# Owned-layer allocation.
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
|
|
||||||
|
|
||||||
def test_manager_allocates_kv_only_for_owned_layers():
|
|
||||||
"A middle shard allocates KV state only for its owned layer range.\n\nTags: node, kv"
|
|
||||||
model = _KvDenseLlama()
|
|
||||||
shard = _KvReferenceShard(model, 2, 3)
|
|
||||||
manager = _manager_for(shard)
|
|
||||||
session = manager.open("sess-mid", 0)
|
|
||||||
assert session.owned_layers == (2, 3)
|
|
||||||
assert set(session.layers) == {2, 3}
|
|
||||||
with pytest.raises(KeyError):
|
|
||||||
session.layer(0)
|
|
||||||
|
|
||||||
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
# Prefill / decode / truncate.
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
|
|
||||||
|
|
||||||
def test_prefill_then_decode_append_grows_owned_layers():
|
|
||||||
"Prefill and decode append advance every owned layer in lockstep.\n\nTags: node, kv"
|
|
||||||
model = _KvDenseLlama()
|
|
||||||
shard = _full_shard(model)
|
|
||||||
manager = _manager_for(shard)
|
|
||||||
adapter = KvBoundaryAdapter(shard, manager)
|
|
||||||
|
|
||||||
prompt = [5, 12, 3, 41]
|
|
||||||
out = adapter.prefill("s", 0, token_ids=np.asarray(prompt))
|
|
||||||
assert isinstance(out, TailOutput)
|
|
||||||
session = manager.get("s", 0)
|
|
||||||
assert session.seq_len == len(prompt)
|
|
||||||
for cache in session.layers.values():
|
|
||||||
assert cache.length == len(prompt)
|
|
||||||
|
|
||||||
step = adapter.decode("s", 0, token_ids=[out.token_id])
|
|
||||||
assert isinstance(step, TailOutput)
|
|
||||||
assert manager.get("s", 0).seq_len == len(prompt) + 1
|
|
||||||
|
|
||||||
|
|
||||||
def test_truncate_rolls_back_all_owned_layers():
|
|
||||||
"Truncate drops cached positions beyond a length across owned layers.\n\nTags: node, kv"
|
|
||||||
model = _KvDenseLlama()
|
|
||||||
shard = _full_shard(model)
|
|
||||||
manager = _manager_for(shard)
|
|
||||||
adapter = KvBoundaryAdapter(shard, manager)
|
|
||||||
adapter.prefill("s", 0, token_ids=np.asarray([1, 2, 3, 4, 5]))
|
|
||||||
assert manager.get("s", 0).seq_len == 5
|
|
||||||
manager.truncate("s", 0, 2)
|
|
||||||
session = manager.get("s", 0)
|
|
||||||
assert session.seq_len == 2
|
|
||||||
for cache in session.layers.values():
|
|
||||||
assert cache.length == 2
|
|
||||||
|
|
||||||
|
|
||||||
def test_layer_kv_cache_rejects_wrong_shape():
|
|
||||||
"LayerKvCache rejects K/V that do not match its head geometry.\n\nTags: node, kv"
|
|
||||||
cache = LayerKvCache(0, n_kv_heads=4, head_dim=8, dtype="float32")
|
|
||||||
with pytest.raises(ValueError):
|
|
||||||
cache.append(np.zeros((1, 3, 8), dtype=np.float32), np.zeros((1, 3, 8), dtype=np.float32))
|
|
||||||
cache.append(np.zeros((2, 4, 8), dtype=np.float32), np.zeros((2, 4, 8), dtype=np.float32))
|
|
||||||
assert cache.length == 2
|
|
||||||
|
|
||||||
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
# Cached vs stateless parity (correctness core).
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
|
|
||||||
|
|
||||||
def test_cached_full_shard_decode_matches_stateless_whole_model():
|
|
||||||
"Cached full-model greedy decode reproduces stateless whole-model tokens.\n\nTags: node, kv, parity"
|
|
||||||
model = _KvDenseLlama()
|
|
||||||
shard = _full_shard(model)
|
|
||||||
manager = _manager_for(shard)
|
|
||||||
adapter = KvBoundaryAdapter(shard, manager)
|
|
||||||
|
|
||||||
prompt = [2, 17, 8, 25, 6]
|
|
||||||
n_new = 12
|
|
||||||
reference = model.stateless_greedy(prompt, n_new)
|
|
||||||
cached = _cached_greedy(adapter, manager, "s", 0, prompt, n_new)
|
|
||||||
assert cached == reference
|
|
||||||
assert len(cached) == n_new
|
|
||||||
|
|
||||||
|
|
||||||
def test_cached_prefill_next_token_matches_whole_model_logits():
|
|
||||||
"Cached prefill produces the same next-token logits as the whole model.\n\nTags: node, kv, parity"
|
|
||||||
model = _KvDenseLlama()
|
|
||||||
shard = _full_shard(model)
|
|
||||||
manager = _manager_for(shard)
|
|
||||||
adapter = KvBoundaryAdapter(shard, manager)
|
|
||||||
|
|
||||||
prompt = [9, 1, 44, 6, 30, 11]
|
|
||||||
out = adapter.prefill("s", 0, token_ids=np.asarray(prompt))
|
|
||||||
assert isinstance(out, TailOutput)
|
|
||||||
assert out.token_id == model.whole_model_next_token(prompt)
|
|
||||||
|
|
||||||
|
|
||||||
def test_multi_range_cached_decode_parity_across_a_seam():
|
|
||||||
"A head/tail split with independent per-range KV reproduces whole-model decode.\n\nTags: node, kv, parity"
|
|
||||||
model = _KvDenseLlama()
|
|
||||||
head_shard = _KvReferenceShard(model, 0, 2)
|
|
||||||
tail_shard = _KvReferenceShard(model, 3, 5)
|
|
||||||
head_mgr = _manager_for(head_shard)
|
|
||||||
tail_mgr = _manager_for(tail_shard)
|
|
||||||
head = KvBoundaryAdapter(head_shard, head_mgr)
|
|
||||||
tail = KvBoundaryAdapter(tail_shard, tail_mgr)
|
|
||||||
|
|
||||||
prompt = [7, 3, 22, 5, 9]
|
|
||||||
n_new = 8
|
|
||||||
|
|
||||||
# Each range only allocates its owned layers.
|
|
||||||
def step(token_ids, is_prefill):
|
|
||||||
if is_prefill:
|
|
||||||
bundle = head.prefill("s", 0, token_ids=np.asarray(token_ids))
|
|
||||||
out = tail.prefill("s", 0, boundary=bundle)
|
|
||||||
else:
|
|
||||||
bundle = head.decode("s", 0, token_ids=[token_ids])
|
|
||||||
assert isinstance(bundle, BoundaryBundle)
|
|
||||||
out = tail.decode("s", 0, boundary=bundle)
|
|
||||||
assert isinstance(out, TailOutput)
|
|
||||||
return out.token_id
|
|
||||||
|
|
||||||
tokens = [step(prompt, True)]
|
|
||||||
for _ in range(n_new - 1):
|
|
||||||
tokens.append(step(tokens[-1], False))
|
|
||||||
|
|
||||||
assert head_mgr.get("s", 0).owned_layers == (0, 1, 2)
|
|
||||||
assert tail_mgr.get("s", 0).owned_layers == (3, 4, 5)
|
|
||||||
assert tokens == model.stateless_greedy(prompt, n_new)
|
|
||||||
|
|
||||||
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
# Four concurrent sessions with no cross-talk.
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
|
|
||||||
|
|
||||||
def test_four_interleaved_sessions_have_no_kv_cross_talk():
|
|
||||||
"Four interleaved sessions each decode their own tokens without cross-talk.\n\nTags: node, kv, concurrency"
|
|
||||||
model = _KvDenseLlama()
|
|
||||||
shard = _full_shard(model)
|
|
||||||
manager = _manager_for(shard)
|
|
||||||
adapter = KvBoundaryAdapter(shard, manager)
|
|
||||||
|
|
||||||
prompts = {
|
|
||||||
"alpha": [1, 2, 3, 4],
|
|
||||||
"bravo": [40, 39, 2, 15],
|
|
||||||
"charlie": [7, 7, 7, 7],
|
|
||||||
"delta": [31, 5, 18, 22],
|
|
||||||
}
|
|
||||||
n_new = 10
|
|
||||||
references = {sid: model.stateless_greedy(p, n_new) for sid, p in prompts.items()}
|
|
||||||
# The four prompts must actually diverge, else "no cross-talk" is vacuous.
|
|
||||||
assert len({tuple(v) for v in references.values()}) == 4
|
|
||||||
|
|
||||||
generated: dict[str, list[int]] = {}
|
|
||||||
for sid, prompt in prompts.items():
|
|
||||||
out = adapter.prefill(sid, 0, token_ids=np.asarray(prompt))
|
|
||||||
assert isinstance(out, TailOutput)
|
|
||||||
generated[sid] = [out.token_id]
|
|
||||||
|
|
||||||
# Round-robin decode: every session takes one step per round, interleaved.
|
|
||||||
for _ in range(n_new - 1):
|
|
||||||
for sid in prompts:
|
|
||||||
step = adapter.decode(sid, 0, token_ids=[generated[sid][-1]])
|
|
||||||
assert isinstance(step, TailOutput)
|
|
||||||
generated[sid].append(step.token_id)
|
|
||||||
|
|
||||||
for sid in prompts:
|
|
||||||
assert generated[sid] == references[sid], sid
|
|
||||||
assert manager.session_count == 4
|
|
||||||
|
|
||||||
|
|
||||||
def test_four_sessions_on_real_threads_stay_isolated():
|
|
||||||
"Four sessions decoding on real threads produce their own reference tokens.\n\nTags: node, kv, concurrency"
|
|
||||||
model = _KvDenseLlama()
|
|
||||||
shard = _full_shard(model)
|
|
||||||
manager = _manager_for(shard, HotKvStateConfig(max_sessions=8))
|
|
||||||
adapter = KvBoundaryAdapter(shard, manager)
|
|
||||||
|
|
||||||
prompts = {
|
|
||||||
"t-alpha": [3, 14, 1, 5],
|
|
||||||
"t-bravo": [2, 27, 18, 4],
|
|
||||||
"t-charlie": [9, 9, 1, 2],
|
|
||||||
"t-delta": [44, 6, 30, 11],
|
|
||||||
}
|
|
||||||
n_new = 8
|
|
||||||
references = {sid: model.stateless_greedy(p, n_new) for sid, p in prompts.items()}
|
|
||||||
results: dict[str, list[int]] = {}
|
|
||||||
errors: list[Exception] = []
|
|
||||||
|
|
||||||
def run(sid: str, prompt: list[int]) -> None:
|
|
||||||
try:
|
|
||||||
results[sid] = _cached_greedy(adapter, manager, sid, 0, prompt, n_new)
|
|
||||||
except Exception as exc: # pragma: no cover - surfaced via assert below
|
|
||||||
errors.append(exc)
|
|
||||||
|
|
||||||
threads = [threading.Thread(target=run, args=(sid, p)) for sid, p in prompts.items()]
|
|
||||||
for t in threads:
|
|
||||||
t.start()
|
|
||||||
for t in threads:
|
|
||||||
t.join()
|
|
||||||
|
|
||||||
assert not errors
|
|
||||||
for sid in prompts:
|
|
||||||
assert results[sid] == references[sid], sid
|
|
||||||
|
|
||||||
|
|
||||||
def test_release_one_session_leaves_others_intact_and_returns_memory():
|
|
||||||
"Releasing one session frees its budget and does not disturb the others.\n\nTags: node, kv, concurrency"
|
|
||||||
model = _KvDenseLlama()
|
|
||||||
shard = _full_shard(model)
|
|
||||||
manager = _manager_for(shard)
|
|
||||||
adapter = KvBoundaryAdapter(shard, manager)
|
|
||||||
|
|
||||||
prompts = {"keep-1": [1, 2, 3], "drop": [10, 11, 12, 13], "keep-2": [5, 6, 7]}
|
|
||||||
n_new = 6
|
|
||||||
references = {sid: model.stateless_greedy(p, n_new) for sid, p in prompts.items()}
|
|
||||||
|
|
||||||
gen: dict[str, list[int]] = {}
|
|
||||||
for sid, prompt in prompts.items():
|
|
||||||
out = adapter.prefill(sid, 0, token_ids=np.asarray(prompt))
|
|
||||||
gen[sid] = [out.token_id]
|
|
||||||
|
|
||||||
bytes_before = manager.total_bytes
|
|
||||||
assert manager.release("drop", 0) is True
|
|
||||||
assert manager.total_bytes < bytes_before
|
|
||||||
|
|
||||||
# A decode on the released session is an explicit cache miss, not corruption.
|
|
||||||
miss = adapter.decode("drop", 0, token_ids=[gen["drop"][-1]])
|
|
||||||
assert isinstance(miss, CacheMiss)
|
|
||||||
assert miss.reason is CacheMissReason.RELEASED
|
|
||||||
|
|
||||||
# The survivors keep decoding to their own references.
|
|
||||||
for _ in range(n_new - 1):
|
|
||||||
for sid in ("keep-1", "keep-2"):
|
|
||||||
step = adapter.decode(sid, 0, token_ids=[gen[sid][-1]])
|
|
||||||
assert isinstance(step, TailOutput)
|
|
||||||
gen[sid].append(step.token_id)
|
|
||||||
for sid in ("keep-1", "keep-2"):
|
|
||||||
assert gen[sid] == references[sid], sid
|
|
||||||
|
|
||||||
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
# Stale epoch / incompatible recipe rejection.
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
|
|
||||||
|
|
||||||
def test_stale_route_epoch_is_rejected():
|
|
||||||
"A request for an older route epoch than the current one is rejected.\n\nTags: node, kv"
|
|
||||||
model = _KvDenseLlama()
|
|
||||||
manager = _manager_for(_full_shard(model))
|
|
||||||
manager.open("s", 5)
|
|
||||||
with pytest.raises(StaleRouteEpochError):
|
|
||||||
manager.open("s", 4)
|
|
||||||
with pytest.raises(StaleRouteEpochError):
|
|
||||||
manager.resolve("s", 4)
|
|
||||||
with pytest.raises(StaleRouteEpochError):
|
|
||||||
manager.append("s", 4, {})
|
|
||||||
|
|
||||||
|
|
||||||
def test_new_route_epoch_supersedes_and_frees_old_epoch():
|
|
||||||
"A newer route epoch supersedes the old one, freeing its KV and reporting a miss.\n\nTags: node, kv"
|
|
||||||
model = _KvDenseLlama()
|
|
||||||
shard = _full_shard(model)
|
|
||||||
manager = _manager_for(shard)
|
|
||||||
adapter = KvBoundaryAdapter(shard, manager)
|
|
||||||
adapter.prefill("s", 1, token_ids=np.asarray([1, 2, 3, 4]))
|
|
||||||
bytes_epoch1 = manager.total_bytes
|
|
||||||
assert bytes_epoch1 > 0
|
|
||||||
|
|
||||||
# Re-planned route: epoch 2 starts a fresh isolated context.
|
|
||||||
adapter.prefill("s", 2, token_ids=np.asarray([9, 8]))
|
|
||||||
assert manager.session_keys() == [("s", 2)]
|
|
||||||
# Old epoch is gone; a lookup for it is now stale (epoch < current).
|
|
||||||
with pytest.raises(StaleRouteEpochError):
|
|
||||||
manager.resolve("s", 1)
|
|
||||||
|
|
||||||
|
|
||||||
def test_incompatible_cache_recipe_is_rejected():
|
|
||||||
"A request carrying a different KV recipe is rejected, not silently reused.\n\nTags: node, kv"
|
|
||||||
model = _KvDenseLlama()
|
|
||||||
shard = _full_shard(model)
|
|
||||||
manager = _manager_for(shard)
|
|
||||||
manager.open("s", 0)
|
|
||||||
|
|
||||||
incompatible = KvCacheRecipe(
|
|
||||||
architecture_adapter="dense-llama",
|
|
||||||
kv_dtype="float16", # different KV dtype
|
|
||||||
n_kv_heads=model.n_heads,
|
|
||||||
head_dim=model.head_dim,
|
|
||||||
total_layers=model.n_layers,
|
|
||||||
start_layer=0,
|
|
||||||
end_layer=model.n_layers - 1,
|
|
||||||
)
|
|
||||||
with pytest.raises(IncompatibleCacheRecipeError):
|
|
||||||
manager.resolve("s", 0, recipe=incompatible)
|
|
||||||
with pytest.raises(IncompatibleCacheRecipeError):
|
|
||||||
manager.open("s2", 0, recipe=incompatible)
|
|
||||||
|
|
||||||
|
|
||||||
def test_uncertified_architecture_recipe_fails_closed():
|
|
||||||
"A KV recipe for an uncertified architecture fails closed at construction.\n\nTags: node, kv"
|
|
||||||
from meshnet_node.boundary_adapter import UncertifiedArchitectureError
|
|
||||||
|
|
||||||
with pytest.raises(UncertifiedArchitectureError):
|
|
||||||
KvCacheRecipe(
|
|
||||||
architecture_adapter="qwen3-moe",
|
|
||||||
kv_dtype="float32",
|
|
||||||
n_kv_heads=4,
|
|
||||||
head_dim=8,
|
|
||||||
total_layers=6,
|
|
||||||
start_layer=0,
|
|
||||||
end_layer=5,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
# Explicit cache-miss responses.
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
|
|
||||||
|
|
||||||
def test_unknown_session_is_an_explicit_cache_miss():
|
|
||||||
"Resolving an unknown session returns an explicit unknown-session miss.\n\nTags: node, kv"
|
|
||||||
manager = _manager_for(_full_shard(_KvDenseLlama()))
|
|
||||||
miss = manager.resolve("nope", 0)
|
|
||||||
assert isinstance(miss, CacheMiss)
|
|
||||||
assert miss.reason is CacheMissReason.UNKNOWN_SESSION
|
|
||||||
with pytest.raises(KvCacheMissError):
|
|
||||||
manager.get("nope", 0)
|
|
||||||
|
|
||||||
|
|
||||||
def test_seq_len_mismatch_is_an_explicit_cache_miss():
|
|
||||||
"A decode whose expected length disagrees with the cache is an explicit miss.\n\nTags: node, kv"
|
|
||||||
model = _KvDenseLlama()
|
|
||||||
shard = _full_shard(model)
|
|
||||||
manager = _manager_for(shard)
|
|
||||||
adapter = KvBoundaryAdapter(shard, manager)
|
|
||||||
out = adapter.prefill("s", 0, token_ids=np.asarray([1, 2, 3]))
|
|
||||||
# Cache holds 3 tokens; claim it holds 99.
|
|
||||||
miss = adapter.decode("s", 0, token_ids=[out.token_id], expected_seq_len=99)
|
|
||||||
assert isinstance(miss, CacheMiss)
|
|
||||||
assert miss.reason is CacheMissReason.SEQ_LEN_MISMATCH
|
|
||||||
|
|
||||||
|
|
||||||
def test_ttl_eviction_yields_an_explicit_cache_miss():
|
|
||||||
"A session idle past its TTL is evicted and reported as a TTL cache miss.\n\nTags: node, kv"
|
|
||||||
model = _KvDenseLlama()
|
|
||||||
shard = _full_shard(model)
|
|
||||||
clock = _FakeClock()
|
|
||||||
manager = _manager_for(shard, HotKvStateConfig(ttl_seconds=10.0), clock=clock)
|
|
||||||
adapter = KvBoundaryAdapter(shard, manager)
|
|
||||||
adapter.prefill("s", 0, token_ids=np.asarray([1, 2, 3]))
|
|
||||||
clock.advance(11.0)
|
|
||||||
miss = manager.resolve("s", 0)
|
|
||||||
assert isinstance(miss, CacheMiss)
|
|
||||||
assert miss.reason is CacheMissReason.EVICTED_TTL
|
|
||||||
assert manager.total_bytes == 0
|
|
||||||
|
|
||||||
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
# Eviction and budget.
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
|
|
||||||
|
|
||||||
def test_lru_eviction_by_session_cap_reports_a_miss():
|
|
||||||
"Exceeding the session cap evicts the least-recently-used session.\n\nTags: node, kv"
|
|
||||||
model = _KvDenseLlama()
|
|
||||||
shard = _full_shard(model)
|
|
||||||
manager = _manager_for(shard, HotKvStateConfig(max_sessions=2))
|
|
||||||
adapter = KvBoundaryAdapter(shard, manager)
|
|
||||||
adapter.prefill("a", 0, token_ids=np.asarray([1, 2]))
|
|
||||||
adapter.prefill("b", 0, token_ids=np.asarray([3, 4]))
|
|
||||||
# Touch 'a' so 'b' becomes the LRU victim.
|
|
||||||
adapter.decode("a", 0, token_ids=[1])
|
|
||||||
adapter.prefill("c", 0, token_ids=np.asarray([5, 6]))
|
|
||||||
|
|
||||||
miss = manager.resolve("b", 0)
|
|
||||||
assert isinstance(miss, CacheMiss)
|
|
||||||
assert miss.reason is CacheMissReason.EVICTED_LRU
|
|
||||||
assert set(k[0] for k in manager.session_keys()) == {"a", "c"}
|
|
||||||
|
|
||||||
|
|
||||||
def test_budget_eviction_keeps_total_within_budget():
|
|
||||||
"Byte-budget pressure evicts LRU sessions so the store stays within budget.\n\nTags: node, kv"
|
|
||||||
model = _KvDenseLlama()
|
|
||||||
shard = _full_shard(model)
|
|
||||||
recipe = kv_recipe_for(shard)
|
|
||||||
# Budget for ~5 tokens of one session; a second big session forces eviction.
|
|
||||||
budget = recipe.bytes_per_token() * 5
|
|
||||||
manager = _manager_for(shard, HotKvStateConfig(budget_bytes=budget, max_sessions=8))
|
|
||||||
adapter = KvBoundaryAdapter(shard, manager)
|
|
||||||
|
|
||||||
adapter.prefill("a", 0, token_ids=np.asarray([1, 2, 3]))
|
|
||||||
adapter.prefill("b", 0, token_ids=np.asarray([4, 5, 6, 7]))
|
|
||||||
assert manager.total_bytes <= budget
|
|
||||||
# 'a' (older, LRU) was evicted to make room for 'b'.
|
|
||||||
miss = manager.resolve("a", 0)
|
|
||||||
assert isinstance(miss, CacheMiss)
|
|
||||||
assert miss.reason is CacheMissReason.EVICTED_LRU
|
|
||||||
assert manager.get("b", 0).seq_len == 4
|
|
||||||
|
|
||||||
|
|
||||||
def test_single_session_exceeding_budget_raises():
|
|
||||||
"A single session that cannot fit the budget raises instead of evicting itself.\n\nTags: node, kv"
|
|
||||||
model = _KvDenseLlama()
|
|
||||||
shard = _full_shard(model)
|
|
||||||
recipe = kv_recipe_for(shard)
|
|
||||||
budget = recipe.bytes_per_token() * 2 # only 2 tokens fit
|
|
||||||
manager = _manager_for(shard, HotKvStateConfig(budget_bytes=budget))
|
|
||||||
adapter = KvBoundaryAdapter(shard, manager)
|
|
||||||
with pytest.raises(KvBudgetExceededError):
|
|
||||||
adapter.prefill("a", 0, token_ids=np.asarray([1, 2, 3, 4, 5]))
|
|
||||||
@@ -1,78 +0,0 @@
|
|||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import os
|
|
||||||
import subprocess
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
|
|
||||||
|
|
||||||
ROOT = Path(__file__).resolve().parents[1]
|
|
||||||
SCRIPT = ROOT / "packages" / "node" / "native" / "scripts" / "build_llama_worker.sh"
|
|
||||||
PIN_FILE = ROOT / "packages" / "node" / "native" / "llama" / "UPSTREAM_COMMIT"
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.skipif(not SCRIPT.exists(), reason="llama worker build script is missing")
|
|
||||||
def test_llama_worker_build_smoke_rebuild(tmp_path: Path) -> None:
|
|
||||||
if not shutil_which("git"):
|
|
||||||
pytest.skip("git is unavailable")
|
|
||||||
if not (shutil_which("g++") or shutil_which("c++") or shutil_which("clang++")):
|
|
||||||
pytest.skip("no C++ compiler is unavailable")
|
|
||||||
|
|
||||||
source_dir = tmp_path / "llama.cpp"
|
|
||||||
build_one = tmp_path / "build-1"
|
|
||||||
build_two = tmp_path / "build-2"
|
|
||||||
pin = PIN_FILE.read_text(encoding="utf-8").strip()
|
|
||||||
|
|
||||||
source_dir.mkdir()
|
|
||||||
_write_fake_upstream_tree(source_dir, pin)
|
|
||||||
_git_init(source_dir)
|
|
||||||
|
|
||||||
_run_build(source_dir, build_one)
|
|
||||||
_run_build(source_dir, build_two)
|
|
||||||
|
|
||||||
binary = build_two / "meshnet_worker"
|
|
||||||
assert binary.exists()
|
|
||||||
output = subprocess.run(
|
|
||||||
[str(binary), "--smoke"],
|
|
||||||
cwd=ROOT,
|
|
||||||
check=True,
|
|
||||||
capture_output=True,
|
|
||||||
text=True,
|
|
||||||
)
|
|
||||||
assert "meshnet worker scaffold ok" in output.stdout
|
|
||||||
assert pin in output.stdout
|
|
||||||
|
|
||||||
|
|
||||||
def _run_build(source_dir: Path, build_dir: Path) -> None:
|
|
||||||
env = os.environ.copy()
|
|
||||||
env.setdefault("PATH", os.environ.get("PATH", ""))
|
|
||||||
subprocess.run(
|
|
||||||
[str(SCRIPT), "--source-dir", str(source_dir), "--build-dir", str(build_dir)],
|
|
||||||
cwd=ROOT,
|
|
||||||
check=True,
|
|
||||||
env=env,
|
|
||||||
capture_output=True,
|
|
||||||
text=True,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _write_fake_upstream_tree(source_dir: Path, pin: str) -> None:
|
|
||||||
(source_dir / "LICENSE").write_text("MIT License placeholder\n", encoding="utf-8")
|
|
||||||
(source_dir / "AUTHORS").write_text("Georgi Gerganov\nMeshnet maintainers\n", encoding="utf-8")
|
|
||||||
(source_dir / "CMakeLists.txt").write_text("# upstream placeholder\n", encoding="utf-8")
|
|
||||||
(source_dir / ".meshnet-upstream-commit").write_text(f"{pin}\n", encoding="utf-8")
|
|
||||||
(source_dir / ".meshnet-upstream-repository").write_text(
|
|
||||||
"https://github.com/ggml-org/llama.cpp.git\n",
|
|
||||||
encoding="utf-8",
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _git_init(source_dir: Path) -> None:
|
|
||||||
subprocess.run(["git", "init", "-q"], cwd=source_dir, check=True)
|
|
||||||
|
|
||||||
|
|
||||||
def shutil_which(name: str) -> str | None:
|
|
||||||
from shutil import which
|
|
||||||
|
|
||||||
return which(name)
|
|
||||||
@@ -1,508 +0,0 @@
|
|||||||
"""DGR-002: generated-schema round-trip and compatibility tests.
|
|
||||||
|
|
||||||
Covers the versioned gRPC Shard protocol (``packages/node/native/proto``):
|
|
||||||
* Python round-trip across the full envelope, tensor bundle, and every service.
|
|
||||||
* Proto3 forward/backward compatibility (unknown-field preservation, defaults).
|
|
||||||
* Bounded-fragment tensor bundle framing + checksums.
|
|
||||||
* Cross-language Python<->C++ round-trip when the C++ toolchain is available;
|
|
||||||
otherwise the C++ test skips with an explicit reason (deterministic, GPU-free,
|
|
||||||
model-download-free, API-credit-free by construction).
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import shutil
|
|
||||||
import subprocess
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
|
|
||||||
# grpc_tools (grpcio-tools) is required to generate the stubs. It is present in
|
|
||||||
# the project .venv; skip cleanly elsewhere rather than error.
|
|
||||||
native_protocol = pytest.importorskip(
|
|
||||||
"meshnet_node.native_protocol",
|
|
||||||
reason="meshnet_node.native_protocol import failed",
|
|
||||||
)
|
|
||||||
|
|
||||||
try:
|
|
||||||
native_protocol.generate()
|
|
||||||
_GEN_ERROR = None
|
|
||||||
except native_protocol.ProtocGenerationError as exc: # pragma: no cover
|
|
||||||
_GEN_ERROR = str(exc)
|
|
||||||
|
|
||||||
pytestmark = pytest.mark.skipif(
|
|
||||||
_GEN_ERROR is not None,
|
|
||||||
reason=f"protobuf stubs unavailable: {_GEN_ERROR}",
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture(scope="module")
|
|
||||||
def pb2():
|
|
||||||
return native_protocol.load()
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Envelope / header round-trip and field coverage
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def _full_header(pb2):
|
|
||||||
return pb2.MessageHeader(
|
|
||||||
schema_version=pb2.SCHEMA_VERSION_1,
|
|
||||||
work_id="work-42",
|
|
||||||
route_session_id="rs-7",
|
|
||||||
route_epoch=9,
|
|
||||||
fingerprint=pb2.ArtifactFingerprint(
|
|
||||||
model_id="meta-llama/Llama-3.1-8B",
|
|
||||||
revision="main",
|
|
||||||
artifact_hash="sha256:deadbeef",
|
|
||||||
quantization="Q4_K_M",
|
|
||||||
runtime_recipe_fingerprint="recipe-123",
|
|
||||||
),
|
|
||||||
shard_range=pb2.ShardRange(
|
|
||||||
start_layer=8,
|
|
||||||
end_layer=16,
|
|
||||||
effective_start_layer=9,
|
|
||||||
owns_embedding=False,
|
|
||||||
owns_final_head=False,
|
|
||||||
),
|
|
||||||
phase=pb2.PHASE_PREFILL,
|
|
||||||
position=pb2.Position(start_position=0, token_count=12, sequence_length=12),
|
|
||||||
idempotency_step=3,
|
|
||||||
cache_expectation=pb2.CACHE_REUSE,
|
|
||||||
compression=pb2.COMPRESSION_ZSTD,
|
|
||||||
checksum=pb2.Checksum(algorithm=pb2.CHECKSUM_CRC32C, value=b"\x00\x01\x02\x03"),
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def test_message_header_carries_every_required_field(pb2):
|
|
||||||
"""The header carries every identifier the transport contract demands.
|
|
||||||
|
|
||||||
Tags: protocol
|
|
||||||
"""
|
|
||||||
header = _full_header(pb2)
|
|
||||||
raw = header.SerializeToString()
|
|
||||||
back = pb2.MessageHeader()
|
|
||||||
back.ParseFromString(raw)
|
|
||||||
|
|
||||||
assert back.schema_version == pb2.SCHEMA_VERSION_1
|
|
||||||
assert back.work_id == "work-42"
|
|
||||||
assert back.route_session_id == "rs-7"
|
|
||||||
assert back.route_epoch == 9
|
|
||||||
assert back.fingerprint.artifact_hash == "sha256:deadbeef"
|
|
||||||
assert back.fingerprint.runtime_recipe_fingerprint == "recipe-123"
|
|
||||||
assert back.shard_range.effective_start_layer == 9
|
|
||||||
assert back.phase == pb2.PHASE_PREFILL
|
|
||||||
assert back.position.token_count == 12
|
|
||||||
assert back.idempotency_step == 3
|
|
||||||
assert back.cache_expectation == pb2.CACHE_REUSE
|
|
||||||
assert back.compression == pb2.COMPRESSION_ZSTD
|
|
||||||
assert back.checksum.algorithm == pb2.CHECKSUM_CRC32C
|
|
||||||
assert back.checksum.value == b"\x00\x01\x02\x03"
|
|
||||||
|
|
||||||
|
|
||||||
def test_named_tensor_bundle_describes_shape_dtype_byteorder_and_fragments(pb2):
|
|
||||||
"""A tensor bundle round-trips name, shape, dtype, byte order and fragments.
|
|
||||||
|
|
||||||
Tags: protocol
|
|
||||||
"""
|
|
||||||
bundle = pb2.TensorBundle(
|
|
||||||
bundle_version=1,
|
|
||||||
tensors=[
|
|
||||||
pb2.NamedTensor(
|
|
||||||
name="hidden_states",
|
|
||||||
shape=[2, 3, 4096],
|
|
||||||
dtype=pb2.DTYPE_BF16,
|
|
||||||
byte_order=pb2.BYTE_ORDER_LITTLE_ENDIAN,
|
|
||||||
total_byte_length=16,
|
|
||||||
compression=pb2.COMPRESSION_NONE,
|
|
||||||
fragments=[
|
|
||||||
pb2.TensorFragment(
|
|
||||||
fragment_index=0,
|
|
||||||
fragment_count=2,
|
|
||||||
byte_offset=0,
|
|
||||||
data=b"\x00" * 8,
|
|
||||||
),
|
|
||||||
pb2.TensorFragment(
|
|
||||||
fragment_index=1,
|
|
||||||
fragment_count=2,
|
|
||||||
byte_offset=8,
|
|
||||||
data=b"\x01" * 8,
|
|
||||||
),
|
|
||||||
],
|
|
||||||
)
|
|
||||||
],
|
|
||||||
)
|
|
||||||
back = pb2.TensorBundle()
|
|
||||||
back.ParseFromString(bundle.SerializeToString())
|
|
||||||
tensor = back.tensors[0]
|
|
||||||
assert tensor.name == "hidden_states"
|
|
||||||
assert list(tensor.shape) == [2, 3, 4096]
|
|
||||||
assert tensor.dtype == pb2.DTYPE_BF16
|
|
||||||
assert tensor.byte_order == pb2.BYTE_ORDER_LITTLE_ENDIAN
|
|
||||||
assert [f.byte_offset for f in tensor.fragments] == [0, 8]
|
|
||||||
|
|
||||||
|
|
||||||
def test_session_stream_carries_open_prefill_decode_release_cancel(pb2):
|
|
||||||
"""The bidi stream oneof expresses every seam operation.
|
|
||||||
|
|
||||||
Tags: protocol
|
|
||||||
"""
|
|
||||||
header = _full_header(pb2)
|
|
||||||
frames = {
|
|
||||||
"open": pb2.SessionActivation(
|
|
||||||
open=pb2.SessionOpen(
|
|
||||||
header=header,
|
|
||||||
deadline_unix_nanos=1_000_000,
|
|
||||||
max_prefill_tokens_per_chunk=256,
|
|
||||||
max_fragment_bytes=1 << 20,
|
|
||||||
initial_credit=pb2.FlowControl(credits=8, max_in_flight_bytes=1 << 24),
|
|
||||||
)
|
|
||||||
),
|
|
||||||
"prefill": pb2.SessionActivation(
|
|
||||||
prefill=pb2.PrefillChunk(
|
|
||||||
header=header, chunk_index=0, chunk_count=2, final_chunk=False
|
|
||||||
)
|
|
||||||
),
|
|
||||||
"decode": pb2.SessionActivation(decode=pb2.DecodeStep(header=header)),
|
|
||||||
"release": pb2.SessionActivation(
|
|
||||||
release=pb2.ReleaseRequest(header=header, reason="done")
|
|
||||||
),
|
|
||||||
"cancel": pb2.SessionActivation(
|
|
||||||
cancel=pb2.CancelRequest(header=header, reason="client abort")
|
|
||||||
),
|
|
||||||
"flow_control": pb2.SessionActivation(
|
|
||||||
flow_control=pb2.FlowControl(credits=4)
|
|
||||||
),
|
|
||||||
}
|
|
||||||
for name, frame in frames.items():
|
|
||||||
back = pb2.SessionActivation()
|
|
||||||
back.ParseFromString(frame.SerializeToString())
|
|
||||||
assert back.WhichOneof("payload") == name
|
|
||||||
|
|
||||||
|
|
||||||
def test_session_response_carries_structured_status_and_results(pb2):
|
|
||||||
"""Server frames carry accepted/result/status/acks with structured Status.
|
|
||||||
|
|
||||||
Tags: protocol
|
|
||||||
"""
|
|
||||||
status = pb2.Status(
|
|
||||||
code=8,
|
|
||||||
message="resource exhausted",
|
|
||||||
retry_class=pb2.RETRY_CLASS_RETRYABLE,
|
|
||||||
details={"queue_depth": "128"},
|
|
||||||
)
|
|
||||||
resp = pb2.SessionResponse(
|
|
||||||
result=pb2.ActivationResult(
|
|
||||||
header=_full_header(pb2),
|
|
||||||
outputs=pb2.TensorBundle(bundle_version=1),
|
|
||||||
cache_result=pb2.CACHE_WRITTEN,
|
|
||||||
status=status,
|
|
||||||
)
|
|
||||||
)
|
|
||||||
back = pb2.SessionResponse()
|
|
||||||
back.ParseFromString(resp.SerializeToString())
|
|
||||||
assert back.WhichOneof("payload") == "result"
|
|
||||||
assert back.result.cache_result == pb2.CACHE_WRITTEN
|
|
||||||
assert back.result.status.retry_class == pb2.RETRY_CLASS_RETRYABLE
|
|
||||||
assert back.result.status.details["queue_depth"] == "128"
|
|
||||||
|
|
||||||
|
|
||||||
def test_capability_and_health_round_trip(pb2):
|
|
||||||
"""Capability and health messages round-trip their admission fields.
|
|
||||||
|
|
||||||
Tags: protocol
|
|
||||||
"""
|
|
||||||
cap = pb2.CapabilityResponse(
|
|
||||||
schema_version=pb2.SCHEMA_VERSION_1,
|
|
||||||
supported_schema_versions=[pb2.SCHEMA_VERSION_1],
|
|
||||||
supported_architectures=["llama"],
|
|
||||||
supported_quantizations=["Q4_K_M", "F16"],
|
|
||||||
servable_range=pb2.ShardRange(start_layer=0, end_layer=16),
|
|
||||||
budget=pb2.ResourceBudget(
|
|
||||||
weight_bytes=1 << 32, kv_bytes=1 << 30, max_concurrent_sessions=4
|
|
||||||
),
|
|
||||||
supported_compression=[pb2.COMPRESSION_NONE, pb2.COMPRESSION_ZSTD],
|
|
||||||
supported_checksums=[pb2.CHECKSUM_CRC32C, pb2.CHECKSUM_SHA256],
|
|
||||||
)
|
|
||||||
cap_back = pb2.CapabilityResponse()
|
|
||||||
cap_back.ParseFromString(cap.SerializeToString())
|
|
||||||
assert cap_back.budget.max_concurrent_sessions == 4
|
|
||||||
assert list(cap_back.supported_quantizations) == ["Q4_K_M", "F16"]
|
|
||||||
|
|
||||||
health = pb2.HealthResponse(
|
|
||||||
status=pb2.SERVING, active_sessions=2, queued_requests=1, kv_pressure=0.5
|
|
||||||
)
|
|
||||||
health_back = pb2.HealthResponse()
|
|
||||||
health_back.ParseFromString(health.SerializeToString())
|
|
||||||
assert health_back.status == pb2.SERVING
|
|
||||||
assert health_back.kv_pressure == pytest.approx(0.5)
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Compatibility
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def test_unknown_fields_are_preserved_for_forward_compatibility(pb2):
|
|
||||||
"""An older reader tolerates and preserves fields it does not know.
|
|
||||||
|
|
||||||
A newer sender may add a field; parsing into the current schema must not
|
|
||||||
fail and must round-trip the unknown bytes.
|
|
||||||
|
|
||||||
Tags: protocol, compatibility
|
|
||||||
"""
|
|
||||||
header = _full_header(pb2)
|
|
||||||
raw = bytearray(header.SerializeToString())
|
|
||||||
# Append an unknown field: number 5000, wire type 2 (length-delimited).
|
|
||||||
tag = (5000 << 3) | 2
|
|
||||||
raw += _encode_varint(tag)
|
|
||||||
payload = b"future-field"
|
|
||||||
raw += _encode_varint(len(payload))
|
|
||||||
raw += payload
|
|
||||||
|
|
||||||
parsed = pb2.MessageHeader()
|
|
||||||
# Parsing must not raise on the unknown field.
|
|
||||||
parsed.ParseFromString(bytes(raw))
|
|
||||||
# Known fields survive intact.
|
|
||||||
assert parsed.work_id == "work-42"
|
|
||||||
assert parsed.route_epoch == 9
|
|
||||||
# The unknown bytes are preserved and re-emitted on re-serialization. This is
|
|
||||||
# the behavioural compatibility guarantee; the introspection accessor
|
|
||||||
# (UnknownFields()) is not implemented by the upb backend, so we assert the
|
|
||||||
# observable outcome rather than the accessor.
|
|
||||||
reserialized = parsed.SerializeToString()
|
|
||||||
assert payload in reserialized
|
|
||||||
assert _encode_varint(tag) in reserialized
|
|
||||||
|
|
||||||
|
|
||||||
def test_defaults_are_stable_for_backward_compatibility(pb2):
|
|
||||||
"""A message from an older sender (missing new fields) reads as enum zero.
|
|
||||||
|
|
||||||
Tags: protocol, compatibility
|
|
||||||
"""
|
|
||||||
empty = pb2.MessageHeader()
|
|
||||||
back = pb2.MessageHeader()
|
|
||||||
back.ParseFromString(empty.SerializeToString())
|
|
||||||
assert back.schema_version == pb2.SCHEMA_VERSION_UNSPECIFIED
|
|
||||||
assert back.phase == pb2.PHASE_UNSPECIFIED
|
|
||||||
assert back.cache_expectation == pb2.CACHE_EXPECTATION_UNSPECIFIED
|
|
||||||
assert back.work_id == ""
|
|
||||||
assert back.route_epoch == 0
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Bounded-fragment helpers
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def test_fragment_and_reassemble_round_trip_with_checksums(pb2):
|
|
||||||
"""Bounded fragmentation reassembles exactly and validates checksums.
|
|
||||||
|
|
||||||
Tags: protocol
|
|
||||||
"""
|
|
||||||
payload = bytes((i * 7) % 256 for i in range(10_000))
|
|
||||||
tensor = native_protocol.fragment_tensor(
|
|
||||||
name="hidden",
|
|
||||||
shape=[1, 4096],
|
|
||||||
dtype=pb2.DTYPE_F16,
|
|
||||||
payload=payload,
|
|
||||||
max_fragment_bytes=4096,
|
|
||||||
checksum_algorithm=pb2.CHECKSUM_CRC32C,
|
|
||||||
)
|
|
||||||
assert len(tensor.fragments) == 3
|
|
||||||
assert all(len(f.data) <= 4096 for f in tensor.fragments)
|
|
||||||
# Survives a serialization round-trip before reassembly.
|
|
||||||
back = pb2.NamedTensor()
|
|
||||||
back.ParseFromString(tensor.SerializeToString())
|
|
||||||
assert native_protocol.reassemble_tensor(back) == payload
|
|
||||||
|
|
||||||
|
|
||||||
def test_reassemble_detects_fragment_corruption(pb2):
|
|
||||||
"""A flipped fragment byte fails checksum verification.
|
|
||||||
|
|
||||||
Tags: protocol
|
|
||||||
"""
|
|
||||||
payload = b"abcdefabcdef" * 100
|
|
||||||
tensor = native_protocol.fragment_tensor(
|
|
||||||
name="t",
|
|
||||||
shape=[len(payload)],
|
|
||||||
dtype=pb2.DTYPE_U8,
|
|
||||||
payload=payload,
|
|
||||||
max_fragment_bytes=256,
|
|
||||||
checksum_algorithm=pb2.CHECKSUM_SHA256,
|
|
||||||
)
|
|
||||||
tensor.fragments[1].data = tensor.fragments[1].data[:-1] + b"\x00"
|
|
||||||
with pytest.raises(ValueError, match="checksum mismatch"):
|
|
||||||
native_protocol.reassemble_tensor(tensor)
|
|
||||||
|
|
||||||
|
|
||||||
def test_checksum_algorithms_verify(pb2):
|
|
||||||
"""CRC32C, CRC32 and SHA256 all verify their own payloads.
|
|
||||||
|
|
||||||
Tags: protocol
|
|
||||||
"""
|
|
||||||
data = b"the quick brown fox"
|
|
||||||
for algo in (pb2.CHECKSUM_CRC32C, pb2.CHECKSUM_CRC32, pb2.CHECKSUM_SHA256):
|
|
||||||
checksum = native_protocol.compute_checksum(algo, data)
|
|
||||||
assert native_protocol.verify_checksum(checksum, data)
|
|
||||||
assert not native_protocol.verify_checksum(checksum, data + b"!")
|
|
||||||
|
|
||||||
|
|
||||||
def test_service_descriptor_exposes_all_operations(pb2):
|
|
||||||
"""The generated service defines capability/health/session/release/cancel.
|
|
||||||
|
|
||||||
Requires the grpc runtime; skips cleanly without it.
|
|
||||||
|
|
||||||
Tags: protocol
|
|
||||||
"""
|
|
||||||
grpc = pytest.importorskip("grpc", reason="grpc runtime not installed")
|
|
||||||
assert grpc is not None
|
|
||||||
grpc_mod = native_protocol.load_grpc()
|
|
||||||
assert hasattr(grpc_mod, "ShardRuntimeStub")
|
|
||||||
assert hasattr(grpc_mod, "ShardRuntimeServicer")
|
|
||||||
# Confirm the streaming seam and unary ops exist on the servicer.
|
|
||||||
servicer = grpc_mod.ShardRuntimeServicer
|
|
||||||
for op in ("GetCapability", "Health", "ActivateSession", "Release", "Cancel"):
|
|
||||||
assert hasattr(servicer, op), op
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Cross-language Python <-> C++ compatibility
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def _cpp_toolchain_reason() -> str | None:
|
|
||||||
"""Return a skip reason if the C++ build toolchain is unavailable."""
|
|
||||||
for tool in ("cmake", "protoc"):
|
|
||||||
if shutil.which(tool) is None:
|
|
||||||
return f"{tool} not found on PATH"
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
def _build_cpp_compatible_sample(pb2):
|
|
||||||
"""Python message matching what roundtrip_test.cpp CheckSample expects."""
|
|
||||||
header = pb2.MessageHeader(
|
|
||||||
schema_version=pb2.SCHEMA_VERSION_1,
|
|
||||||
work_id="w1",
|
|
||||||
route_session_id="s1",
|
|
||||||
route_epoch=3,
|
|
||||||
phase=pb2.PHASE_PREFILL,
|
|
||||||
idempotency_step=7,
|
|
||||||
cache_expectation=pb2.CACHE_FRESH,
|
|
||||||
compression=pb2.COMPRESSION_NONE,
|
|
||||||
fingerprint=pb2.ArtifactFingerprint(
|
|
||||||
model_id="meta-llama/Llama-3.1-8B",
|
|
||||||
quantization="Q4_K_M",
|
|
||||||
runtime_recipe_fingerprint="recipe-abc",
|
|
||||||
),
|
|
||||||
shard_range=pb2.ShardRange(
|
|
||||||
start_layer=0, end_layer=16, effective_start_layer=0, owns_embedding=True
|
|
||||||
),
|
|
||||||
position=pb2.Position(start_position=0, token_count=5, sequence_length=5),
|
|
||||||
)
|
|
||||||
return pb2.SessionActivation(
|
|
||||||
prefill=pb2.PrefillChunk(
|
|
||||||
header=header,
|
|
||||||
chunk_index=0,
|
|
||||||
chunk_count=1,
|
|
||||||
final_chunk=True,
|
|
||||||
activations=pb2.TensorBundle(
|
|
||||||
bundle_version=1,
|
|
||||||
tensors=[
|
|
||||||
pb2.NamedTensor(
|
|
||||||
name="hidden",
|
|
||||||
shape=[1, 4096],
|
|
||||||
dtype=pb2.DTYPE_F16,
|
|
||||||
byte_order=pb2.BYTE_ORDER_LITTLE_ENDIAN,
|
|
||||||
total_byte_length=8,
|
|
||||||
compression=pb2.COMPRESSION_NONE,
|
|
||||||
fragments=[
|
|
||||||
pb2.TensorFragment(
|
|
||||||
fragment_index=0,
|
|
||||||
fragment_count=1,
|
|
||||||
byte_offset=0,
|
|
||||||
data=bytes(range(1, 9)),
|
|
||||||
)
|
|
||||||
],
|
|
||||||
)
|
|
||||||
],
|
|
||||||
),
|
|
||||||
)
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def test_cross_language_roundtrip_python_and_cpp(pb2, tmp_path):
|
|
||||||
"""Python and C++ parse each other's serialized frames (both directions).
|
|
||||||
|
|
||||||
Builds the C++ round-trip binary via CMake, feeds it a Python-serialized
|
|
||||||
fixture (C++ must parse it), and parses the C++-serialized output back in
|
|
||||||
Python. Skips with an explicit reason when the C++ toolchain is absent.
|
|
||||||
|
|
||||||
Tags: protocol, compatibility, cpp
|
|
||||||
"""
|
|
||||||
reason = _cpp_toolchain_reason()
|
|
||||||
if reason is not None:
|
|
||||||
pytest.skip(f"C++ toolchain unavailable: {reason}")
|
|
||||||
|
|
||||||
native_root = native_protocol.PROTO_DIR.parent
|
|
||||||
build_dir = tmp_path / "cpp-build"
|
|
||||||
|
|
||||||
configure = subprocess.run(
|
|
||||||
["cmake", "-S", str(native_root), "-B", str(build_dir)],
|
|
||||||
capture_output=True,
|
|
||||||
text=True,
|
|
||||||
)
|
|
||||||
if configure.returncode != 0:
|
|
||||||
pytest.skip(
|
|
||||||
"cmake configure failed (protobuf C++ dev likely missing):\n"
|
|
||||||
+ configure.stderr[-2000:]
|
|
||||||
)
|
|
||||||
|
|
||||||
build = subprocess.run(
|
|
||||||
["cmake", "--build", str(build_dir), "--target",
|
|
||||||
"shard_protocol_roundtrip_test"],
|
|
||||||
capture_output=True,
|
|
||||||
text=True,
|
|
||||||
)
|
|
||||||
assert build.returncode == 0, f"C++ build failed:\n{build.stderr[-2000:]}"
|
|
||||||
|
|
||||||
binary = build_dir / "shard_protocol_roundtrip_test"
|
|
||||||
assert binary.exists(), "C++ test binary not produced"
|
|
||||||
|
|
||||||
py_fixture = tmp_path / "py_sample.bin"
|
|
||||||
cpp_out = tmp_path / "cpp_sample.bin"
|
|
||||||
py_fixture.write_bytes(_build_cpp_compatible_sample(pb2).SerializeToString())
|
|
||||||
|
|
||||||
run = subprocess.run(
|
|
||||||
[str(binary), "--selftest", "--read", str(py_fixture),
|
|
||||||
"--write", str(cpp_out)],
|
|
||||||
capture_output=True,
|
|
||||||
text=True,
|
|
||||||
)
|
|
||||||
assert run.returncode == 0, f"C++ round-trip failed:\n{run.stdout}\n{run.stderr}"
|
|
||||||
|
|
||||||
# C++ parsed our bytes; now Python parses C++'s bytes and checks known fields.
|
|
||||||
parsed = pb2.SessionActivation()
|
|
||||||
parsed.ParseFromString(cpp_out.read_bytes())
|
|
||||||
assert parsed.WhichOneof("payload") == "prefill"
|
|
||||||
assert parsed.prefill.header.work_id == "w1"
|
|
||||||
assert parsed.prefill.header.route_epoch == 3
|
|
||||||
assert parsed.prefill.activations.tensors[0].name == "hidden"
|
|
||||||
assert parsed.prefill.activations.tensors[0].dtype == pb2.DTYPE_F16
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Local helpers
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def _encode_varint(value: int) -> bytes:
|
|
||||||
out = bytearray()
|
|
||||||
while True:
|
|
||||||
byte = value & 0x7F
|
|
||||||
value >>= 7
|
|
||||||
if value:
|
|
||||||
out.append(byte | 0x80)
|
|
||||||
else:
|
|
||||||
out.append(byte)
|
|
||||||
return bytes(out)
|
|
||||||
@@ -22,7 +22,6 @@ import pytest
|
|||||||
|
|
||||||
from meshnet_node.admission import (
|
from meshnet_node.admission import (
|
||||||
REASON_BACKEND_MISMATCH,
|
REASON_BACKEND_MISMATCH,
|
||||||
REASON_COMPATIBILITY_MISMATCH,
|
|
||||||
REASON_MODEL_MISMATCH,
|
REASON_MODEL_MISMATCH,
|
||||||
REASON_NO_REPORT,
|
REASON_NO_REPORT,
|
||||||
REASON_NOT_PASSED,
|
REASON_NOT_PASSED,
|
||||||
@@ -69,26 +68,11 @@ class _FakeBackend:
|
|||||||
total_layers = 24
|
total_layers = 24
|
||||||
hidden_size = 8
|
hidden_size = 8
|
||||||
|
|
||||||
def __init__(
|
def __init__(self, *, shard_start=0, shard_end=23, device="cpu", forward_error=None):
|
||||||
self,
|
|
||||||
*,
|
|
||||||
shard_start=0,
|
|
||||||
shard_end=23,
|
|
||||||
device="cpu",
|
|
||||||
forward_error=None,
|
|
||||||
loaded_shard_start=None,
|
|
||||||
loaded_shard_end=None,
|
|
||||||
owns_embedding=None,
|
|
||||||
owns_final_head=None,
|
|
||||||
):
|
|
||||||
self.shard_start = shard_start
|
self.shard_start = shard_start
|
||||||
self.shard_end = shard_end
|
self.shard_end = shard_end
|
||||||
self.is_head = shard_start == 0
|
self.is_head = shard_start == 0
|
||||||
self.is_tail = shard_end == self.total_layers - 1
|
self.is_tail = shard_end == self.total_layers - 1
|
||||||
self.loaded_shard_start = shard_start if loaded_shard_start is None else loaded_shard_start
|
|
||||||
self.loaded_shard_end = shard_end if loaded_shard_end is None else loaded_shard_end
|
|
||||||
self.owns_embedding = self.is_head if owns_embedding is None else owns_embedding
|
|
||||||
self.owns_final_head = self.is_tail if owns_final_head is None else owns_final_head
|
|
||||||
self.device = _FakeDevice(device)
|
self.device = _FakeDevice(device)
|
||||||
self.model_id = MODEL
|
self.model_id = MODEL
|
||||||
self._forward_error = forward_error
|
self._forward_error = forward_error
|
||||||
@@ -208,17 +192,6 @@ def test_a_passing_report_from_another_backend_or_device_is_refused():
|
|||||||
assert exc.value.reason == REASON_BACKEND_MISMATCH
|
assert exc.value.reason == REASON_BACKEND_MISMATCH
|
||||||
|
|
||||||
|
|
||||||
def test_a_passing_report_with_the_wrong_cache_layout_is_refused():
|
|
||||||
"The compatibility fingerprint fails closed when cache layout changes.\n\nTags: node, admission"
|
|
||||||
ctx = _context()
|
|
||||||
report = capability_report_for(ctx, cache_layout="local-hot-kv")
|
|
||||||
|
|
||||||
with pytest.raises(CapabilityAdmissionError) as exc:
|
|
||||||
admit(AdmissionRequirement.for_context(ctx), report)
|
|
||||||
|
|
||||||
assert exc.value.reason == REASON_COMPATIBILITY_MISMATCH
|
|
||||||
|
|
||||||
|
|
||||||
def test_a_report_older_than_the_freshness_window_is_refused():
|
def test_a_report_older_than_the_freshness_window_is_refused():
|
||||||
"Hardware, drivers and weights move; an old proof is not a current one.\n\nTags: node, admission"
|
"Hardware, drivers and weights move; an old proof is not a current one.\n\nTags: node, admission"
|
||||||
ctx = _context()
|
ctx = _context()
|
||||||
@@ -385,73 +358,6 @@ def test_a_stale_report_cannot_be_reused_to_register(startup_env):
|
|||||||
assert startup_env == []
|
assert startup_env == []
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Re-registration: the proof presented is fresh, never the one captured at boot
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def test_run_startup_hands_the_heartbeat_a_refresher_for_the_current_shard(startup_env, monkeypatch):
|
|
||||||
"The tracker refuses aged proofs, so the heartbeat must be able to re-prove what the node serves now.\n\nTags: node, admission, startup"
|
|
||||||
import meshnet_node.startup as startup_mod
|
|
||||||
|
|
||||||
captured: dict = {}
|
|
||||||
monkeypatch.setattr(
|
|
||||||
startup_mod, "_start_heartbeat", lambda *a, **kw: captured.update(kw)
|
|
||||||
)
|
|
||||||
|
|
||||||
_start(capability_validator=capability_stub())
|
|
||||||
|
|
||||||
refresh = captured.get("refresh_capability")
|
|
||||||
assert callable(refresh), "run_startup no longer wires a capability refresher"
|
|
||||||
fresh = refresh({"hf_repo": MODEL, "model": MODEL.split("/")[-1]})
|
|
||||||
assert fresh is not None
|
|
||||||
assert fresh["model"]["model_id"] == MODEL
|
|
||||||
assert (fresh["shard"]["start"], fresh["shard"]["end"]) == (0, 23)
|
|
||||||
assert fresh["validated_at"] > time.time() - 60
|
|
||||||
|
|
||||||
|
|
||||||
def test_a_reregistration_presents_a_refreshed_proof(monkeypatch):
|
|
||||||
"Replaying the boot-time report after an outage re-registers the node unroutable; the re-register path must present a fresh proof.\n\nTags: node, admission, startup"
|
|
||||||
import json
|
|
||||||
|
|
||||||
import meshnet_node.startup as startup_mod
|
|
||||||
|
|
||||||
model = "acme/refresh-model-7b"
|
|
||||||
boot_report = {"validated_at": 1.0, "marker": "boot"}
|
|
||||||
fresh_report = {"validated_at": 2.0, "marker": "fresh"}
|
|
||||||
posted: list[dict] = []
|
|
||||||
|
|
||||||
def _record(url, payload, timeout=10.0):
|
|
||||||
if url.endswith("/v1/nodes/register") and payload.get("hf_repo") == model:
|
|
||||||
posted.append(json.loads(json.dumps(payload)))
|
|
||||||
return {"node_id": "node-refresh"}
|
|
||||||
raise SystemExit # first heartbeat POST ends the daemon loop
|
|
||||||
|
|
||||||
monkeypatch.setattr(startup_mod, "_post_json", _record)
|
|
||||||
|
|
||||||
payload = {
|
|
||||||
"hf_repo": model,
|
|
||||||
"model": model.split("/")[-1],
|
|
||||||
"capability_report": dict(boot_report),
|
|
||||||
}
|
|
||||||
thread = startup_mod._start_heartbeat(
|
|
||||||
"http://tracker.invalid",
|
|
||||||
startup_mod._PENDING_NODE_ID, # forces a re-registration on the first tick
|
|
||||||
payload,
|
|
||||||
interval=0.02,
|
|
||||||
refresh_capability=lambda _payload: dict(fresh_report),
|
|
||||||
)
|
|
||||||
|
|
||||||
# The loop must be dead before this test returns: once monkeypatch restores
|
|
||||||
# `_post_json`, a surviving thread would re-register through whatever the
|
|
||||||
# *next* test patches in and corrupt its call counts.
|
|
||||||
thread.join(timeout=5.0)
|
|
||||||
|
|
||||||
assert not thread.is_alive(), "the heartbeat loop outlived the test"
|
|
||||||
assert posted, "the heartbeat never re-registered"
|
|
||||||
assert posted[0]["capability_report"] == fresh_report
|
|
||||||
|
|
||||||
|
|
||||||
def test_a_matching_passing_report_registers_and_travels_with_the_payload(startup_env):
|
def test_a_matching_passing_report_registers_and_travels_with_the_payload(startup_env):
|
||||||
"Registration carries the proof for exactly the model/shard/recipe it advertises.\n\nTags: node, admission, startup"
|
"Registration carries the proof for exactly the model/shard/recipe it advertises.\n\nTags: node, admission, startup"
|
||||||
node = _start() # production validator against a working fake backend
|
node = _start() # production validator against a working fake backend
|
||||||
@@ -465,31 +371,10 @@ def test_a_matching_passing_report_registers_and_travels_with_the_payload(startu
|
|||||||
assert report["status"] == "passed"
|
assert report["status"] == "passed"
|
||||||
assert report["model"]["model_id"] == MODEL
|
assert report["model"]["model_id"] == MODEL
|
||||||
assert (report["shard"]["start"], report["shard"]["end"]) == (0, 23)
|
assert (report["shard"]["start"], report["shard"]["end"]) == (0, 23)
|
||||||
assert report["shard"]["owns_embedding"] is True
|
|
||||||
assert report["shard"]["owns_final_head"] is True
|
|
||||||
assert report["recipe"]["recipe_id"] == DEFAULT_RECIPE_ID
|
assert report["recipe"]["recipe_id"] == DEFAULT_RECIPE_ID
|
||||||
assert report["backend"]["device"] == "cpu"
|
assert report["backend"]["device"] == "cpu"
|
||||||
|
|
||||||
|
|
||||||
def test_capability_report_prefers_backend_loaded_range_over_cli_claims():
|
|
||||||
"The proof reports the model's loaded range, not the CLI's requested range.\n\nTags: node, admission"
|
|
||||||
backend = _FakeBackend(
|
|
||||||
shard_start=0,
|
|
||||||
shard_end=23,
|
|
||||||
loaded_shard_start=8,
|
|
||||||
loaded_shard_end=15,
|
|
||||||
owns_embedding=False,
|
|
||||||
owns_final_head=True,
|
|
||||||
)
|
|
||||||
report = capability_report_for(
|
|
||||||
_context(backend=backend, shard_start=0, shard_end=23),
|
|
||||||
)
|
|
||||||
|
|
||||||
assert (report.shard.start, report.shard.end) == (8, 15)
|
|
||||||
assert report.shard.owns_embedding is False
|
|
||||||
assert report.shard.owns_final_head is True
|
|
||||||
|
|
||||||
|
|
||||||
def test_the_served_backend_is_loaded_with_the_recipe_that_was_validated(startup_env):
|
def test_the_served_backend_is_loaded_with_the_recipe_that_was_validated(startup_env):
|
||||||
"The recipe named in the report is the one the serving backend actually ran.\n\nTags: node, admission, startup"
|
"The recipe named in the report is the one the serving backend actually ran.\n\nTags: node, admission, startup"
|
||||||
node = _start(recipe_id="eager-attention")
|
node = _start(recipe_id="eager-attention")
|
||||||
|
|||||||
@@ -42,12 +42,9 @@ def _report(**overrides):
|
|||||||
status="passed",
|
status="passed",
|
||||||
duration_ms=142,
|
duration_ms=142,
|
||||||
validated_at=1_760_000_000.0,
|
validated_at=1_760_000_000.0,
|
||||||
owns_embedding=True,
|
|
||||||
owns_final_head=False,
|
|
||||||
)
|
)
|
||||||
kwargs.update(overrides)
|
kwargs.update(overrides)
|
||||||
report = build_capability_report(**kwargs)
|
return build_capability_report(**kwargs)
|
||||||
return report
|
|
||||||
|
|
||||||
|
|
||||||
# --- model-agnostic identity ------------------------------------------------
|
# --- model-agnostic identity ------------------------------------------------
|
||||||
@@ -117,9 +114,6 @@ def test_report_dict_has_the_stable_documented_key_set():
|
|||||||
"shard",
|
"shard",
|
||||||
"recipe",
|
"recipe",
|
||||||
"backend",
|
"backend",
|
||||||
"artifact",
|
|
||||||
"runtime_recipe",
|
|
||||||
"compatibility_fingerprint",
|
|
||||||
"status",
|
"status",
|
||||||
"validated_at",
|
"validated_at",
|
||||||
"duration_ms",
|
"duration_ms",
|
||||||
@@ -127,38 +121,12 @@ def test_report_dict_has_the_stable_documented_key_set():
|
|||||||
}
|
}
|
||||||
assert payload["schema_version"] == CAPABILITY_SCHEMA_VERSION
|
assert payload["schema_version"] == CAPABILITY_SCHEMA_VERSION
|
||||||
assert set(payload["model"]) == {"model_id", "revision", "config_fingerprint"}
|
assert set(payload["model"]) == {"model_id", "revision", "config_fingerprint"}
|
||||||
assert set(payload["shard"]) == {
|
assert set(payload["shard"]) == {"start", "end"}
|
||||||
"start",
|
|
||||||
"end",
|
|
||||||
"owns_embedding",
|
|
||||||
"owns_final_head",
|
|
||||||
}
|
|
||||||
assert set(payload["recipe"]) == {
|
assert set(payload["recipe"]) == {
|
||||||
"recipe_id",
|
"recipe_id",
|
||||||
"recipe_version",
|
"recipe_version",
|
||||||
"catalogue_version",
|
"catalogue_version",
|
||||||
}
|
}
|
||||||
assert set(payload["artifact"]) == {
|
|
||||||
"model_id",
|
|
||||||
"revision",
|
|
||||||
"artifact_hash",
|
|
||||||
"shard_start",
|
|
||||||
"shard_end",
|
|
||||||
}
|
|
||||||
assert set(payload["runtime_recipe"]) == {
|
|
||||||
"weight_quantization",
|
|
||||||
"activation_dtype",
|
|
||||||
"compute_dtype",
|
|
||||||
"kv_dtype",
|
|
||||||
"kv_layout",
|
|
||||||
"tokenizer_revision",
|
|
||||||
"architecture_adapter",
|
|
||||||
"backend_id",
|
|
||||||
"runtime_version",
|
|
||||||
"boundary_schema_version",
|
|
||||||
"cache_layout",
|
|
||||||
"fingerprint",
|
|
||||||
}
|
|
||||||
assert set(payload["backend"]) == {
|
assert set(payload["backend"]) == {
|
||||||
"backend_id",
|
"backend_id",
|
||||||
"device",
|
"device",
|
||||||
@@ -166,19 +134,10 @@ def test_report_dict_has_the_stable_documented_key_set():
|
|||||||
"quantization",
|
"quantization",
|
||||||
"runtime",
|
"runtime",
|
||||||
}
|
}
|
||||||
assert payload["compatibility_fingerprint"].startswith("sha256:")
|
|
||||||
# JSON-serializable end to end.
|
# JSON-serializable end to end.
|
||||||
assert json.loads(json.dumps(payload)) == payload
|
assert json.loads(json.dumps(payload)) == payload
|
||||||
|
|
||||||
|
|
||||||
def test_report_carries_endpoint_ownership():
|
|
||||||
"Endpoint ownership is recorded alongside the shard range.\n\nTags: node, startup"
|
|
||||||
payload = _report().to_dict()
|
|
||||||
|
|
||||||
assert payload["shard"]["owns_embedding"] is True
|
|
||||||
assert payload["shard"]["owns_final_head"] is False
|
|
||||||
|
|
||||||
|
|
||||||
def test_identity_key_pins_model_shard_recipe_and_backend():
|
def test_identity_key_pins_model_shard_recipe_and_backend():
|
||||||
"Identity key pins model shard recipe and backend\n\nTags: node, startup"
|
"Identity key pins model shard recipe and backend\n\nTags: node, startup"
|
||||||
base = _report()
|
base = _report()
|
||||||
@@ -197,15 +156,6 @@ def test_identity_key_pins_model_shard_recipe_and_backend():
|
|||||||
assert _report(device="other-device").identity_key() != base.identity_key()
|
assert _report(device="other-device").identity_key() != base.identity_key()
|
||||||
|
|
||||||
|
|
||||||
def test_compatibility_fingerprint_changes_when_the_runtime_recipe_changes():
|
|
||||||
"The compatibility fingerprint changes when the runtime recipe changes.\n\nTags: node, startup"
|
|
||||||
base = _report()
|
|
||||||
altered = _report(cache_layout="stateless")
|
|
||||||
|
|
||||||
assert base.compatibility_fingerprint != altered.compatibility_fingerprint
|
|
||||||
assert base.runtime_recipe.fingerprint != altered.runtime_recipe.fingerprint
|
|
||||||
|
|
||||||
|
|
||||||
def test_config_fingerprint_is_stable_under_key_order_and_detects_change():
|
def test_config_fingerprint_is_stable_under_key_order_and_detects_change():
|
||||||
"Config fingerprint is stable under key order and detects change\n\nTags: node, startup"
|
"Config fingerprint is stable under key order and detects change\n\nTags: node, startup"
|
||||||
a = config_fingerprint({"num_hidden_layers": 8, "hidden_size": 512})
|
a = config_fingerprint({"num_hidden_layers": 8, "hidden_size": 512})
|
||||||
|
|||||||
@@ -287,47 +287,6 @@ def test_configure_torch_threads_applies_explicit_settings(monkeypatch):
|
|||||||
assert active == {"torch_threads": 12, "torch_interop_threads": 2}
|
assert active == {"torch_threads": 12, "torch_interop_threads": 2}
|
||||||
|
|
||||||
|
|
||||||
def test_heartbeat_applies_release_without_reregistering(monkeypatch):
|
|
||||||
"""DROP_SHARD has no replacement range and must not look like an outage."""
|
|
||||||
import meshnet_node.startup as startup_mod
|
|
||||||
|
|
||||||
released = threading.Event()
|
|
||||||
requests: list[tuple[str, dict]] = []
|
|
||||||
|
|
||||||
class FakeNode:
|
|
||||||
def apply_tracker_directives(self, directives):
|
|
||||||
assert directives == [{"action": "DROP_SHARD", "model": "Qwen/Qwen2.5-0.5B-Instruct"}]
|
|
||||||
return {"action": "DROP_SHARD", "model": "Qwen/Qwen2.5-0.5B-Instruct"}
|
|
||||||
|
|
||||||
def fake_post(url, payload, timeout=10.0):
|
|
||||||
requests.append((url, dict(payload)))
|
|
||||||
released.set()
|
|
||||||
return {"directives": [{"action": "DROP_SHARD", "model": "Qwen/Qwen2.5-0.5B-Instruct"}]}
|
|
||||||
|
|
||||||
sleep_calls = 0
|
|
||||||
|
|
||||||
def one_heartbeat(_seconds):
|
|
||||||
nonlocal sleep_calls
|
|
||||||
sleep_calls += 1
|
|
||||||
if sleep_calls > 1:
|
|
||||||
raise SystemExit
|
|
||||||
|
|
||||||
monkeypatch.setattr(startup_mod, "_post_json", fake_post)
|
|
||||||
monkeypatch.setattr(startup_mod.time, "sleep", one_heartbeat)
|
|
||||||
payload = {
|
|
||||||
"model": "Qwen2.5-0.5B-Instruct",
|
|
||||||
"hf_repo": "Qwen/Qwen2.5-0.5B-Instruct",
|
|
||||||
"shard_start": 0,
|
|
||||||
"shard_end": 23,
|
|
||||||
}
|
|
||||||
startup_mod._start_heartbeat("http://tracker", "node-1", payload, interval=0, node_ref=FakeNode())
|
|
||||||
|
|
||||||
assert released.wait(1), "heartbeat did not receive the queued release"
|
|
||||||
assert len(requests) == 1, "release must not trigger a re-registration"
|
|
||||||
assert payload["shard_start"] == 0
|
|
||||||
assert payload["shard_end"] == 23
|
|
||||||
|
|
||||||
|
|
||||||
def test_benchmark_throughput_is_registered_in_payload(monkeypatch, tmp_path):
|
def test_benchmark_throughput_is_registered_in_payload(monkeypatch, tmp_path):
|
||||||
"benchmark_tokens_per_sec from the benchmark is included in the tracker registration.\n\nTags: node, performance, startup"
|
"benchmark_tokens_per_sec from the benchmark is included in the tracker registration.\n\nTags: node, performance, startup"
|
||||||
import meshnet_node.startup as startup_mod
|
import meshnet_node.startup as startup_mod
|
||||||
|
|||||||
@@ -1,286 +0,0 @@
|
|||||||
"""Tests for the DGR-001 performance contract metadata."""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import json
|
|
||||||
from unittest.mock import MagicMock, patch
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
|
|
||||||
from meshnet_node.performance_contract import (
|
|
||||||
BENCHMARK_SCHEMA_VERSION,
|
|
||||||
DEFAULT_CONTRACT,
|
|
||||||
SCHEMA_VERSION,
|
|
||||||
main,
|
|
||||||
run_performance_benchmark,
|
|
||||||
run_real_model_endpoint_benchmark,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def test_default_contract_is_architecture_aligned_and_small():
|
|
||||||
"""The baseline stays on DeepSeek2 and uses the smallest DeepSeek-family GGUF.
|
|
||||||
|
|
||||||
Tags: performance, model, gguf
|
|
||||||
"""
|
|
||||||
payload = DEFAULT_CONTRACT.to_dict()
|
|
||||||
|
|
||||||
assert payload["schema_version"] == SCHEMA_VERSION
|
|
||||||
assert payload["story_id"] == "DGR-001"
|
|
||||||
assert payload["model_target"] == {
|
|
||||||
"name": "DeepSeek-V2-Lite-Chat",
|
|
||||||
"architecture": "deepseek2",
|
|
||||||
"safetensors_repo": "deepseek-ai/DeepSeek-V2-Lite-Chat",
|
|
||||||
"safetensors_precision": "bfloat16",
|
|
||||||
"gguf_repo": "second-state/DeepSeek-V2-Lite-Chat-GGUF",
|
|
||||||
"gguf_quant": "Q2_K",
|
|
||||||
"gguf_size_gb": 6.43,
|
|
||||||
"comparison_policy": (
|
|
||||||
"same model/revision, closest practical low-footprint precision pair: "
|
|
||||||
"BF16 safetensors versus Q2_K GGUF"
|
|
||||||
),
|
|
||||||
"rationale": (
|
|
||||||
"Smallest DeepSeek-family benchmark anchor that still points toward "
|
|
||||||
"DeepSeek-V4-Flash; keeps the runtime on the DeepSeek2 path instead "
|
|
||||||
"of falling back to a tiny but architecture-mismatched smoke model."
|
|
||||||
),
|
|
||||||
}
|
|
||||||
assert payload["benchmark_lanes"] == [
|
|
||||||
{
|
|
||||||
"id": "transformers-safetensors-cpu",
|
|
||||||
"runtime": "transformers",
|
|
||||||
"device": "cpu",
|
|
||||||
"recipe": "current safetensors recipe",
|
|
||||||
"concurrency_levels": [1, 4],
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"id": "llama-cpp-gguf-cpu",
|
|
||||||
"runtime": "llama.cpp",
|
|
||||||
"device": "cpu",
|
|
||||||
"recipe": "whole-model GGUF recipe",
|
|
||||||
"concurrency_levels": [1, 4],
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"id": "transformers-safetensors-gpu",
|
|
||||||
"runtime": "transformers",
|
|
||||||
"device": "gpu",
|
|
||||||
"recipe": "current safetensors recipe",
|
|
||||||
"concurrency_levels": [1, 4],
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"id": "llama-cpp-gguf-gpu",
|
|
||||||
"runtime": "llama.cpp",
|
|
||||||
"device": "gpu",
|
|
||||||
"recipe": "whole-model GGUF recipe",
|
|
||||||
"concurrency_levels": [1, 4],
|
|
||||||
},
|
|
||||||
]
|
|
||||||
assert "ttft_ms" in payload["metrics"]
|
|
||||||
assert "output_drift" in payload["metrics"]
|
|
||||||
assert "meaningful speed or fit benefit" in payload["stop_condition"]
|
|
||||||
assert any("mounted drive" in note for note in payload["notes"])
|
|
||||||
|
|
||||||
|
|
||||||
def test_contract_cli_writes_json(tmp_path, capsys):
|
|
||||||
"""The contract can be emitted as a machine-readable artifact.
|
|
||||||
|
|
||||||
Tags: performance, artifact
|
|
||||||
"""
|
|
||||||
output = tmp_path / "performance-contract.json"
|
|
||||||
|
|
||||||
assert main(["--json-out", str(output)]) == 0
|
|
||||||
written = json.loads(output.read_text(encoding="utf-8"))
|
|
||||||
|
|
||||||
assert written == DEFAULT_CONTRACT.to_dict()
|
|
||||||
assert str(output) in capsys.readouterr().out
|
|
||||||
|
|
||||||
|
|
||||||
def test_stub_benchmark_covers_every_lane_concurrency_and_metric():
|
|
||||||
"""The runner exercises all four CPU/GPU lanes with the full metric set.
|
|
||||||
|
|
||||||
Tags: performance, benchmark, gguf
|
|
||||||
"""
|
|
||||||
report = run_performance_benchmark()
|
|
||||||
|
|
||||||
assert report["schema_version"] == BENCHMARK_SCHEMA_VERSION
|
|
||||||
assert report["story_id"] == "DGR-001"
|
|
||||||
assert report["source"] == "stub-backend"
|
|
||||||
assert report["model_target"] == DEFAULT_CONTRACT.model_target.to_dict()
|
|
||||||
assert [lane["id"] for lane in report["lanes"]] == [
|
|
||||||
lane.id for lane in DEFAULT_CONTRACT.benchmark_lanes
|
|
||||||
]
|
|
||||||
for lane in report["lanes"]:
|
|
||||||
assert [result["concurrency"] for result in lane["results"]] == [1, 4]
|
|
||||||
for result in lane["results"]:
|
|
||||||
assert set(result["metrics"]) == set(DEFAULT_CONTRACT.metrics)
|
|
||||||
assert result["metrics"]["failure_count"] == 0
|
|
||||||
assert result["metrics"]["decode_tok_per_sec"] > 0
|
|
||||||
|
|
||||||
|
|
||||||
def test_stub_benchmark_is_deterministic():
|
|
||||||
"""Two runs produce byte-identical reports; no clocks or randomness leak in.
|
|
||||||
|
|
||||||
Tags: performance, benchmark, deterministic
|
|
||||||
"""
|
|
||||||
first = run_performance_benchmark()
|
|
||||||
second = run_performance_benchmark()
|
|
||||||
|
|
||||||
assert first == second
|
|
||||||
assert json.dumps(first, sort_keys=True) == json.dumps(second, sort_keys=True)
|
|
||||||
|
|
||||||
|
|
||||||
def test_stub_benchmark_compares_gguf_against_safetensors_per_device():
|
|
||||||
"""Each device gets a GGUF-vs-safetensors comparison and a stop-condition verdict.
|
|
||||||
|
|
||||||
Tags: performance, benchmark, gguf
|
|
||||||
"""
|
|
||||||
report = run_performance_benchmark()
|
|
||||||
|
|
||||||
assert set(report["comparisons"]) == {"cpu", "gpu"}
|
|
||||||
cpu, gpu = report["comparisons"]["cpu"], report["comparisons"]["gpu"]
|
|
||||||
assert cpu["safetensors_lane"] == "transformers-safetensors-cpu"
|
|
||||||
assert cpu["gguf_lane"] == "llama-cpp-gguf-cpu"
|
|
||||||
assert cpu["memory_metric"] == "rss_bytes"
|
|
||||||
assert gpu["safetensors_lane"] == "transformers-safetensors-gpu"
|
|
||||||
assert gpu["gguf_lane"] == "llama-cpp-gguf-gpu"
|
|
||||||
assert gpu["memory_metric"] == "vram_bytes"
|
|
||||||
for comparison in (cpu, gpu):
|
|
||||||
assert comparison["decode_speedup"] > 1.0
|
|
||||||
assert comparison["artifact_bytes_ratio"] < 0.5
|
|
||||||
assert comparison["memory_bytes_ratio"] < 1.0
|
|
||||||
assert comparison["output_drift"] == 0.0
|
|
||||||
assert comparison["gguf_benefit"] is True
|
|
||||||
assert report["stop_condition"]["gguf_benefit"] is True
|
|
||||||
assert report["stop_condition"]["triggered"] is False
|
|
||||||
assert report["stop_condition"]["text"] == DEFAULT_CONTRACT.stop_condition
|
|
||||||
|
|
||||||
|
|
||||||
def test_contract_cli_writes_benchmark_report(tmp_path, capsys):
|
|
||||||
"""--benchmark-out emits the stub benchmark report next to the contract.
|
|
||||||
|
|
||||||
Tags: performance, benchmark, artifact
|
|
||||||
"""
|
|
||||||
contract_out = tmp_path / "performance-contract.json"
|
|
||||||
benchmark_out = tmp_path / "artifacts" / "stub-benchmark-report.json"
|
|
||||||
|
|
||||||
assert main(["--json-out", str(contract_out), "--benchmark-out", str(benchmark_out)]) == 0
|
|
||||||
report = json.loads(benchmark_out.read_text(encoding="utf-8"))
|
|
||||||
|
|
||||||
assert report == run_performance_benchmark()
|
|
||||||
output = capsys.readouterr().out
|
|
||||||
assert str(contract_out) in output
|
|
||||||
assert str(benchmark_out) in output
|
|
||||||
|
|
||||||
|
|
||||||
def test_real_model_endpoint_benchmark_uses_lane_specific_endpoints_and_shared_schema():
|
|
||||||
"""The live client path fans out to one endpoint per CPU/GPU lane.
|
|
||||||
|
|
||||||
Tags: performance, benchmark, live
|
|
||||||
"""
|
|
||||||
response = MagicMock()
|
|
||||||
response.read.return_value = json.dumps({"choices": [{"message": {"content": "mesh activation"}}]}).encode()
|
|
||||||
response.headers.get.return_value = "lane-session"
|
|
||||||
response.__enter__.return_value = response
|
|
||||||
|
|
||||||
endpoints = {
|
|
||||||
"transformers-safetensors-cpu": "http://cpu-safetensors",
|
|
||||||
"llama-cpp-gguf-cpu": "http://cpu-gguf",
|
|
||||||
"transformers-safetensors-gpu": "http://gpu-safetensors",
|
|
||||||
"llama-cpp-gguf-gpu": "http://gpu-gguf",
|
|
||||||
}
|
|
||||||
|
|
||||||
with patch("meshnet_node.performance_contract.urllib.request.urlopen", return_value=response) as urlopen:
|
|
||||||
report = run_real_model_endpoint_benchmark(endpoints=endpoints, model="deepseek-ai/DeepSeek-V2-Lite-Chat")
|
|
||||||
|
|
||||||
assert report["source"] == "real-model-endpoints"
|
|
||||||
assert report["model_target"] == DEFAULT_CONTRACT.model_target.to_dict()
|
|
||||||
assert set(report["comparisons"]) == {"cpu", "gpu"}
|
|
||||||
assert urlopen.call_count == len(endpoints)
|
|
||||||
called_urls = [call.args[0].full_url for call in urlopen.call_args_list]
|
|
||||||
assert called_urls == [f"{url}/v1/chat/completions" for url in endpoints.values()]
|
|
||||||
for lane in report["lanes"]:
|
|
||||||
assert lane["results"][0]["metrics"]["decode_tok_per_sec"] > 0
|
|
||||||
assert lane["results"][0]["metrics"]["ttft_ms"] > 0
|
|
||||||
assert lane["output_tokens"] == ["mesh", "activation"]
|
|
||||||
assert report["comparisons"]["cpu"]["gguf_lane"] == "llama-cpp-gguf-cpu"
|
|
||||||
assert report["comparisons"]["gpu"]["gguf_lane"] == "llama-cpp-gguf-gpu"
|
|
||||||
|
|
||||||
|
|
||||||
def test_contract_cli_runs_live_endpoint_benchmark(tmp_path, capsys):
|
|
||||||
"""--live-endpoint mappings drive the live runner and write its report.
|
|
||||||
|
|
||||||
Tags: performance, benchmark, live, artifact
|
|
||||||
"""
|
|
||||||
contract_out = tmp_path / "performance-contract.json"
|
|
||||||
live_out = tmp_path / "artifacts" / "live-benchmark-report.json"
|
|
||||||
endpoints = {
|
|
||||||
"transformers-safetensors-cpu": "http://cpu-safetensors",
|
|
||||||
"llama-cpp-gguf-cpu": "http://cpu-gguf",
|
|
||||||
"transformers-safetensors-gpu": "http://gpu-safetensors",
|
|
||||||
"llama-cpp-gguf-gpu": "http://gpu-gguf",
|
|
||||||
}
|
|
||||||
fake_report = {"schema_version": BENCHMARK_SCHEMA_VERSION, "source": "real-model-endpoints"}
|
|
||||||
argv = ["--json-out", str(contract_out), "--live-benchmark-out", str(live_out)]
|
|
||||||
for lane_id, url in endpoints.items():
|
|
||||||
argv += ["--live-endpoint", f"{lane_id}={url}"]
|
|
||||||
|
|
||||||
with patch(
|
|
||||||
"meshnet_node.performance_contract.run_real_model_endpoint_benchmark",
|
|
||||||
return_value=fake_report,
|
|
||||||
) as runner:
|
|
||||||
assert main(argv) == 0
|
|
||||||
|
|
||||||
runner.assert_called_once_with(
|
|
||||||
endpoints,
|
|
||||||
model=DEFAULT_CONTRACT.model_target.safetensors_repo,
|
|
||||||
contract=DEFAULT_CONTRACT,
|
|
||||||
)
|
|
||||||
assert json.loads(live_out.read_text(encoding="utf-8")) == fake_report
|
|
||||||
output = capsys.readouterr().out
|
|
||||||
assert str(contract_out) in output
|
|
||||||
assert str(live_out) in output
|
|
||||||
|
|
||||||
|
|
||||||
def test_contract_cli_passes_explicit_live_model(tmp_path):
|
|
||||||
"""--live-model overrides the contract's safetensors repo default.
|
|
||||||
|
|
||||||
Tags: performance, benchmark, live
|
|
||||||
"""
|
|
||||||
live_out = tmp_path / "live-benchmark-report.json"
|
|
||||||
argv = [
|
|
||||||
"--json-out", str(tmp_path / "performance-contract.json"),
|
|
||||||
"--live-benchmark-out", str(live_out),
|
|
||||||
"--live-endpoint", "transformers-safetensors-cpu=http://cpu-safetensors",
|
|
||||||
"--live-model", "local/DeepSeek-V2-Lite-Chat-Q2_K",
|
|
||||||
]
|
|
||||||
|
|
||||||
with patch(
|
|
||||||
"meshnet_node.performance_contract.run_real_model_endpoint_benchmark",
|
|
||||||
return_value={},
|
|
||||||
) as runner:
|
|
||||||
assert main(argv) == 0
|
|
||||||
|
|
||||||
assert runner.call_args.kwargs["model"] == "local/DeepSeek-V2-Lite-Chat-Q2_K"
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.parametrize(
|
|
||||||
"argv",
|
|
||||||
[
|
|
||||||
["--live-endpoint", "transformers-safetensors-cpu=http://cpu"],
|
|
||||||
["--live-benchmark-out", "live-report.json"],
|
|
||||||
[
|
|
||||||
"--live-endpoint", "not-a-mapping",
|
|
||||||
"--live-benchmark-out", "live-report.json",
|
|
||||||
],
|
|
||||||
],
|
|
||||||
ids=["endpoint-without-out", "out-without-endpoint", "malformed-mapping"],
|
|
||||||
)
|
|
||||||
def test_contract_cli_rejects_incomplete_live_arguments(tmp_path, argv, capsys):
|
|
||||||
"""Live flags must arrive as a consistent LANE_ID=URL + output-path set.
|
|
||||||
|
|
||||||
Tags: performance, benchmark, live, cli
|
|
||||||
"""
|
|
||||||
with pytest.raises(SystemExit) as excinfo:
|
|
||||||
main(["--json-out", str(tmp_path / "performance-contract.json"), *argv])
|
|
||||||
|
|
||||||
assert excinfo.value.code == 2
|
|
||||||
assert "--live-" in capsys.readouterr().err
|
|
||||||
@@ -32,18 +32,12 @@ def test_matrix_reports_direct_relay_prefill_decode_and_machine_readable_metrics
|
|||||||
assert {"p50_latency_ms", "p95_latency_ms", "payload_bytes", "compression_ratio",
|
assert {"p50_latency_ms", "p95_latency_ms", "payload_bytes", "compression_ratio",
|
||||||
"connection_attempts", "p95_queue_wait_ms"} <= set(run["phases"]["decode"])
|
"connection_attempts", "p95_queue_wait_ms"} <= set(run["phases"]["decode"])
|
||||||
sample = run["samples"][0]
|
sample = run["samples"][0]
|
||||||
assert sample["model_ms"] > 0
|
|
||||||
assert sample["encode_ms"] > 0
|
|
||||||
assert sample["activation_decode_ms"] > 0
|
|
||||||
assert sample["framing_ms"] > 0
|
assert sample["framing_ms"] > 0
|
||||||
assert sample["metadata_ms"] > 0
|
assert sample["metadata_ms"] > 0
|
||||||
assert sample["copy_allocation_ms"] > 0
|
assert sample["copy_allocation_ms"] > 0
|
||||||
assert sample["copy_allocation_bytes"] >= sample["payload_bytes"]
|
assert sample["copy_allocation_bytes"] >= sample["payload_bytes"]
|
||||||
assert sample["local_http_forwarding_ms"] > 0
|
|
||||||
assert len(run["samples"]) == 1 + len(run["output_tokens"])
|
assert len(run["samples"]) == 1 + len(run["output_tokens"])
|
||||||
assert {"tokens_per_sec", "bytes_per_token", "compression_cpu_ms", "peak_buffered_bytes",
|
assert {"tokens_per_sec", "bytes_per_token", "compression_cpu_ms", "peak_buffered_bytes"} <= set(run["phases"]["decode"])
|
||||||
"model_execution_ms", "activation_encoding_ms", "activation_decoding_ms",
|
|
||||||
"local_http_forwarding_ms"} <= set(run["phases"]["decode"])
|
|
||||||
|
|
||||||
|
|
||||||
def test_cached_sessions_reuse_one_connection_and_preserve_stub_tokens():
|
def test_cached_sessions_reuse_one_connection_and_preserve_stub_tokens():
|
||||||
@@ -80,10 +74,7 @@ def test_cli_writes_json_artifact_and_human_summary(tmp_path, capsys):
|
|||||||
report = json.loads(output.read_text())
|
report = json.loads(output.read_text())
|
||||||
assert report["schema_version"] == 1
|
assert report["schema_version"] == 1
|
||||||
assert "Route Session benchmark" in capsys.readouterr().out
|
assert "Route Session benchmark" in capsys.readouterr().out
|
||||||
summary = format_summary(report)
|
assert "relay" in format_summary(report)
|
||||||
assert "relay" in summary
|
|
||||||
assert "model/encode/decode" in summary
|
|
||||||
assert "HTTP" in summary
|
|
||||||
|
|
||||||
|
|
||||||
def test_performance_gate_checks_comparison_identity_session_and_cleanup():
|
def test_performance_gate_checks_comparison_identity_session_and_cleanup():
|
||||||
|
|||||||
@@ -5,7 +5,6 @@ Model ids here are arbitrary and made up on purpose: nothing in the admission or
|
|||||||
routing path may branch on a vendor, model or kernel name.
|
routing path may branch on a vendor, model or kernel name.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import hashlib
|
|
||||||
import json
|
import json
|
||||||
import time
|
import time
|
||||||
import urllib.error
|
import urllib.error
|
||||||
@@ -19,7 +18,6 @@ from meshnet_tracker.capability import (
|
|||||||
POLICY_ENFORCE,
|
POLICY_ENFORCE,
|
||||||
STATE_ABSENT,
|
STATE_ABSENT,
|
||||||
STATE_ADMITTED,
|
STATE_ADMITTED,
|
||||||
STATE_COMPATIBILITY_MISMATCH,
|
|
||||||
STATE_CATALOGUE_INCOMPATIBLE,
|
STATE_CATALOGUE_INCOMPATIBLE,
|
||||||
STATE_FAILED,
|
STATE_FAILED,
|
||||||
STATE_INVALID,
|
STATE_INVALID,
|
||||||
@@ -43,14 +41,6 @@ SHORT = "oracle-9b"
|
|||||||
LAYERS = 32
|
LAYERS = 32
|
||||||
|
|
||||||
|
|
||||||
def _stable_json(data: dict) -> str:
|
|
||||||
return json.dumps(data, sort_keys=True, separators=(",", ":"), ensure_ascii=False)
|
|
||||||
|
|
||||||
|
|
||||||
def _sha256_text(data: dict) -> str:
|
|
||||||
return "sha256:" + hashlib.sha256(_stable_json(data).encode("utf-8")).hexdigest()
|
|
||||||
|
|
||||||
|
|
||||||
def _post_json(url: str, payload: dict) -> dict:
|
def _post_json(url: str, payload: dict) -> dict:
|
||||||
data = json.dumps(payload).encode()
|
data = json.dumps(payload).encode()
|
||||||
req = urllib.request.Request(
|
req = urllib.request.Request(
|
||||||
@@ -70,8 +60,6 @@ def _report(
|
|||||||
model_id: str = MODEL,
|
model_id: str = MODEL,
|
||||||
start: int = 0,
|
start: int = 0,
|
||||||
end: int = 15,
|
end: int = 15,
|
||||||
owns_embedding: bool | None = None,
|
|
||||||
owns_final_head: bool | None = None,
|
|
||||||
status: str = "passed",
|
status: str = "passed",
|
||||||
validated_at: float | None = None,
|
validated_at: float | None = None,
|
||||||
recipe_id: str = "baseline",
|
recipe_id: str = "baseline",
|
||||||
@@ -82,48 +70,10 @@ def _report(
|
|||||||
diagnostics: list | None = None,
|
diagnostics: list | None = None,
|
||||||
) -> dict:
|
) -> dict:
|
||||||
"""A capability report shaped exactly as `meshnet_node.capability` emits it."""
|
"""A capability report shaped exactly as `meshnet_node.capability` emits it."""
|
||||||
if owns_embedding is None:
|
return {
|
||||||
owns_embedding = start == 0
|
|
||||||
if owns_final_head is None:
|
|
||||||
owns_final_head = end >= LAYERS - 1
|
|
||||||
artifact = {
|
|
||||||
"model_id": model_id,
|
|
||||||
"revision": None,
|
|
||||||
"artifact_hash": _sha256_text(
|
|
||||||
{
|
|
||||||
"model_id": model_id,
|
|
||||||
"shard_start": start,
|
|
||||||
"shard_end": end,
|
|
||||||
"recipe_id": recipe_id,
|
|
||||||
"recipe_version": recipe_version,
|
|
||||||
}
|
|
||||||
),
|
|
||||||
"shard_start": start,
|
|
||||||
"shard_end": end,
|
|
||||||
}
|
|
||||||
runtime_recipe = {
|
|
||||||
"weight_quantization": "bfloat16",
|
|
||||||
"activation_dtype": "bfloat16",
|
|
||||||
"compute_dtype": "bfloat16",
|
|
||||||
"kv_dtype": "bfloat16",
|
|
||||||
"kv_layout": "session-cache",
|
|
||||||
"tokenizer_revision": model_id,
|
|
||||||
"architecture_adapter": "unknown",
|
|
||||||
"backend_id": "torch-transformers",
|
|
||||||
"runtime_version": "0.1.0",
|
|
||||||
"boundary_schema_version": 1,
|
|
||||||
"cache_layout": "local-hot-kv",
|
|
||||||
}
|
|
||||||
runtime_recipe["fingerprint"] = _sha256_text(runtime_recipe)
|
|
||||||
payload = {
|
|
||||||
"schema_version": schema_version,
|
"schema_version": schema_version,
|
||||||
"model": {"model_id": model_id, "revision": None, "config_fingerprint": None},
|
"model": {"model_id": model_id, "revision": None, "config_fingerprint": None},
|
||||||
"shard": {
|
"shard": {"start": start, "end": end},
|
||||||
"start": start,
|
|
||||||
"end": end,
|
|
||||||
"owns_embedding": owns_embedding,
|
|
||||||
"owns_final_head": owns_final_head,
|
|
||||||
},
|
|
||||||
"recipe": {
|
"recipe": {
|
||||||
"recipe_id": recipe_id,
|
"recipe_id": recipe_id,
|
||||||
"recipe_version": recipe_version,
|
"recipe_version": recipe_version,
|
||||||
@@ -136,24 +86,11 @@ def _report(
|
|||||||
"quantization": "bfloat16",
|
"quantization": "bfloat16",
|
||||||
"runtime": {},
|
"runtime": {},
|
||||||
},
|
},
|
||||||
"artifact": artifact,
|
|
||||||
"runtime_recipe": runtime_recipe,
|
|
||||||
"status": status,
|
"status": status,
|
||||||
"validated_at": time.time() if validated_at is None else validated_at,
|
"validated_at": time.time() if validated_at is None else validated_at,
|
||||||
"duration_ms": 42,
|
"duration_ms": 42,
|
||||||
"diagnostics": list(diagnostics or []),
|
"diagnostics": list(diagnostics or []),
|
||||||
}
|
}
|
||||||
payload["compatibility_fingerprint"] = _sha256_text(
|
|
||||||
{
|
|
||||||
"model": payload["model"],
|
|
||||||
"shard": payload["shard"],
|
|
||||||
"recipe": payload["recipe"],
|
|
||||||
"backend": payload["backend"],
|
|
||||||
"artifact": payload["artifact"],
|
|
||||||
"runtime_recipe": payload["runtime_recipe"],
|
|
||||||
}
|
|
||||||
)
|
|
||||||
return payload
|
|
||||||
|
|
||||||
|
|
||||||
def _registration(
|
def _registration(
|
||||||
@@ -182,7 +119,6 @@ def _registration(
|
|||||||
report = _report(start=start, end=end)
|
report = _report(start=start, end=end)
|
||||||
if report is not None:
|
if report is not None:
|
||||||
payload["capability_report"] = report
|
payload["capability_report"] = report
|
||||||
payload["compatibility_fingerprint"] = report["compatibility_fingerprint"]
|
|
||||||
if recipe_id is not None:
|
if recipe_id is not None:
|
||||||
payload["recipe_id"] = recipe_id
|
payload["recipe_id"] = recipe_id
|
||||||
if recipe_version is not None:
|
if recipe_version is not None:
|
||||||
@@ -260,15 +196,6 @@ def test_a_report_for_a_different_recipe_than_the_node_declares_is_a_recipe_mism
|
|||||||
assert versioned.state == STATE_RECIPE_MISMATCH
|
assert versioned.state == STATE_RECIPE_MISMATCH
|
||||||
|
|
||||||
|
|
||||||
def test_a_report_for_a_different_compatibility_fingerprint_is_a_compatibility_mismatch():
|
|
||||||
"The exact artifact/runtime recipe fingerprint gates admission.\n\nTags: routing, tracker"
|
|
||||||
state = _evaluate(
|
|
||||||
_report(),
|
|
||||||
declared_compatibility_fingerprint="sha256:deadbeef",
|
|
||||||
)
|
|
||||||
assert state.state == STATE_COMPATIBILITY_MISMATCH
|
|
||||||
|
|
||||||
|
|
||||||
def test_an_older_recipe_catalogue_is_incompatible():
|
def test_an_older_recipe_catalogue_is_incompatible():
|
||||||
"Recipe ids from a catalogue older than the tracker's minimum cannot be matched.\n\nTags: routing, tracker"
|
"Recipe ids from a catalogue older than the tracker's minimum cannot be matched.\n\nTags: routing, tracker"
|
||||||
state = _evaluate(_report(catalogue_version="2025.01.1"))
|
state = _evaluate(_report(catalogue_version="2025.01.1"))
|
||||||
|
|||||||
@@ -2869,43 +2869,6 @@ def test_same_endpoint_can_register_multiple_models():
|
|||||||
tracker.stop()
|
tracker.stop()
|
||||||
|
|
||||||
|
|
||||||
def test_explicit_model_placement_targets_only_the_selected_node():
|
|
||||||
"An admin can add and release a model on one chosen multi-model node.\n\nTags: http, routing, tracker"
|
|
||||||
from meshnet_tracker.server import _release_model_locked, _request_model_load_locked
|
|
||||||
|
|
||||||
tracker = _tracker(model_presets={
|
|
||||||
"model-a": {"total_layers": 4, "bytes_per_layer": {"bfloat16": 1_000}, "hf_repo": "org/ModelA"},
|
|
||||||
"model-b": {"total_layers": 4, "bytes_per_layer": {"bfloat16": 1_000}, "hf_repo": "org/ModelB"},
|
|
||||||
})
|
|
||||||
tracker_port = tracker.start()
|
|
||||||
try:
|
|
||||||
registrations = []
|
|
||||||
for port in (9062, 9063):
|
|
||||||
registrations.append(_post_json(
|
|
||||||
f"http://127.0.0.1:{tracker_port}/v1/nodes/register",
|
|
||||||
{"endpoint": f"http://127.0.0.1:{port}", "model": "model-a", "hf_repo": "org/ModelA",
|
|
||||||
"num_layers": 4, "shard_start": 0, "shard_end": 3, "max_loaded_shards": 2,
|
|
||||||
"vram_bytes": 20_000, "ram_bytes": 20_000, "quantizations": ["bfloat16"],
|
|
||||||
"benchmark_tokens_per_sec": 1.0, "hardware_profile": {}, "score": 1.0},
|
|
||||||
))
|
|
||||||
selected, other = (item["node_id"] for item in registrations)
|
|
||||||
|
|
||||||
with tracker._lock:
|
|
||||||
assignment = _request_model_load_locked(tracker._server, "model-b", selected) # type: ignore[arg-type]
|
|
||||||
assert assignment is not None
|
|
||||||
assert assignment["node_id"] == selected
|
|
||||||
assert tracker._registry[selected].pending_new_assignment is not None
|
|
||||||
assert tracker._registry[other].pending_new_assignment is None
|
|
||||||
|
|
||||||
with tracker._lock:
|
|
||||||
released = _release_model_locked(tracker._server, "model-a", selected) # type: ignore[arg-type]
|
|
||||||
assert released == 1
|
|
||||||
assert len(tracker._registry[selected].pending_directives) == 2
|
|
||||||
assert tracker._registry[other].pending_directives == []
|
|
||||||
finally:
|
|
||||||
tracker.stop()
|
|
||||||
|
|
||||||
|
|
||||||
def test_scale_demanded_models_queues_add_shard_on_spare_host():
|
def test_scale_demanded_models_queues_add_shard_on_spare_host():
|
||||||
"Scale demanded models queues add shard on spare host\n\nTags: http, routing, tracker"
|
"Scale demanded models queues add shard on spare host\n\nTags: http, routing, tracker"
|
||||||
tracker = _tracker(model_presets={
|
tracker = _tracker(model_presets={
|
||||||
@@ -3042,13 +3005,6 @@ def test_a_node_declaring_an_unsupported_quantization_is_never_routed():
|
|||||||
assert status == 503
|
assert status == 503
|
||||||
|
|
||||||
|
|
||||||
def test_a_node_declaring_auto_quantization_serves_a_default_precision_request():
|
|
||||||
"'auto' is the CLI default that delegates the choice — it is not a refusal, so the node must resolve to its best advertised precision and route.\n\nTags: http, routing, tracker"
|
|
||||||
status, response = _proxy_chat_status(POLICY_COMPAT, quantization="auto")
|
|
||||||
assert status == 200
|
|
||||||
assert response["choices"][0]["message"]["content"] == "ok"
|
|
||||||
|
|
||||||
|
|
||||||
def test_a_node_declaring_a_null_quantization_is_never_routed():
|
def test_a_node_declaring_a_null_quantization_is_never_routed():
|
||||||
"An explicit null states 'no usable precision' -- only an absent field is legacy.\n\nTags: http, routing, tracker"
|
"An explicit null states 'no usable precision' -- only an absent field is legacy.\n\nTags: http, routing, tracker"
|
||||||
node = http.server.HTTPServer(("127.0.0.1", 0), _EchoChatHandler)
|
node = http.server.HTTPServer(("127.0.0.1", 0), _EchoChatHandler)
|
||||||
|
|||||||
Reference in New Issue
Block a user