Compare commits
1 Commits
ralph/dist
...
ralph/deep
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
f102be1098 |
@@ -8,10 +8,6 @@ metadata:
|
|||||||
|
|
||||||
# Project Status (2026-07-13)
|
# Project Status (2026-07-13)
|
||||||
|
|
||||||
## Distributed inference performance (2026-07-14)
|
|
||||||
|
|
||||||
`DIP-001` is done in `.scratch/distributed-inference-performance/`: the deterministic two-node Route Session stub benchmark covers direct/relay plus cached/stateless prefill and decode. Its JSON and concise summary explicitly attribute model execution, activation encode/decode, compression, connection setup, relay queueing, local HTTP forwarding, and end-to-end seam latency. `PYTHONPATH=packages/node pytest -q tests/test_route_session_benchmark.py` passed (7); the fixture assertion checks output-token identity and connection attempts.
|
|
||||||
|
|
||||||
> Doc reconciliation 2026-07-13: `docs/prd.json` tracks US-001…US-050 (048 memory budget, 049 mainnet pilot, 050 Qwen demand placement). ADRs 0025–0026 added (TAI phase B/C, assignment ownership).
|
> Doc reconciliation 2026-07-13: `docs/prd.json` tracks US-001…US-050 (048 memory budget, 049 mainnet pilot, 050 Qwen demand placement). ADRs 0025–0026 added (TAI phase B/C, assignment ownership).
|
||||||
|
|
||||||
All 35 user stories in docs/prd.json are done (35/35), including the reward-system arc US-030…US-035 completed 2026-07-02:
|
All 35 user stories in docs/prd.json are done (35/35), including the reward-system arc US-030…US-035 completed 2026-07-02:
|
||||||
|
|||||||
@@ -12,10 +12,4 @@ Provide an opt-in, admin-only tracker Dashboard Testing tab that dynamically dis
|
|||||||
- One active run.
|
- One active run.
|
||||||
- Real inference stays separately environment-gated and excluded from default suites.
|
- Real inference stays separately environment-gated and excluded from default suites.
|
||||||
|
|
||||||
## Operator workflow
|
|
||||||
|
|
||||||
See [`docs/dev/dashboard-test-runner.md`](../../docs/dev/dashboard-test-runner.md)
|
|
||||||
for launch configuration, default safe suites vs the gated real-inference suite,
|
|
||||||
and required environment variables.
|
|
||||||
|
|
||||||
See `prd.json` for executable Ralph user stories and acceptance criteria.
|
See `prd.json` for executable Ralph user stories and acceptance criteria.
|
||||||
|
|||||||
@@ -51,16 +51,15 @@
|
|||||||
"uv run pytest tests/test_dashboard.py tests/test_dynamic_routing.py -q passes."
|
"uv run pytest tests/test_dashboard.py tests/test_dynamic_routing.py -q passes."
|
||||||
],
|
],
|
||||||
"priority": 3,
|
"priority": 3,
|
||||||
"passes": true,
|
"passes": false,
|
||||||
"notes": "Do not reintroduce --enable-test-runner without implementing its CLI argument in US-001.",
|
"notes": "Do not reintroduce --enable-test-runner without implementing its CLI argument in US-001.",
|
||||||
"dependsOn": [
|
"dependsOn": [
|
||||||
"US-001",
|
"US-001",
|
||||||
"US-002"
|
"US-002"
|
||||||
],
|
]
|
||||||
"completionNotes": "Completed by agent"
|
|
||||||
}
|
}
|
||||||
],
|
],
|
||||||
"metadata": {
|
"metadata": {
|
||||||
"updatedAt": "2026-07-12T01:58:06.286Z"
|
"updatedAt": "2026-07-11T17:02:30.520Z"
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -1,84 +0,0 @@
|
|||||||
# DGR-001 — performance contract baseline
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `packages/node/meshnet_node/performance_contract.py`
|
|
||||||
- `tests/test_performance_contract.py`
|
|
||||||
- `.scratch/distributed-gguf-runtime/issues/01-lock-the-safetensors-versus-gguf-performance-contract.md`
|
|
||||||
- `.scratch/distributed-gguf-runtime/evidence/DGR-001/performance-contract.json`
|
|
||||||
|
|
||||||
## What this slice does
|
|
||||||
|
|
||||||
- Locks the DGR-001 benchmark contract in code.
|
|
||||||
- Pins the architecture-aligned baseline to **DeepSeek-V2-Lite-Chat** (`deepseek2`).
|
|
||||||
- Uses the same model on both sides of the comparison:
|
|
||||||
- **safetensors:** `deepseek-ai/DeepSeek-V2-Lite-Chat` in **BF16**
|
|
||||||
- **GGUF:** `second-state/DeepSeek-V2-Lite-Chat-GGUF` in **Q2_K**
|
|
||||||
- Exposes a machine-readable JSON contract with:
|
|
||||||
- benchmark lanes for `transformers` safetensors and `llama.cpp` GGUF on **CPU** and **GPU**
|
|
||||||
- concurrency levels `1` and `4`
|
|
||||||
- the required metrics list
|
|
||||||
- an explicit stop condition for “no meaningful speed or fit benefit”
|
|
||||||
- Adds a deterministic stub benchmark report so the contract now has an executable report shape end to end.
|
|
||||||
|
|
||||||
## Recent benchmark runner slice
|
|
||||||
|
|
||||||
The runner currently uses a deterministic stub backend to exercise the comparison matrix without downloading a model. It emits:
|
|
||||||
|
|
||||||
- `.scratch/distributed-gguf-runtime/evidence/DGR-001/performance-contract.json`
|
|
||||||
- `.scratch/distributed-gguf-runtime/evidence/DGR-001/stub-benchmark-report.json`
|
|
||||||
|
|
||||||
The report includes per-device comparisons for:
|
|
||||||
|
|
||||||
- `transformers-safetensors-cpu` vs `llama-cpp-gguf-cpu`
|
|
||||||
- `transformers-safetensors-gpu` vs `llama-cpp-gguf-gpu`
|
|
||||||
|
|
||||||
and records the memory metric (`rss_bytes` on CPU, `vram_bytes` on GPU), decode speedup, artifact ratio, and output drift.
|
|
||||||
|
|
||||||
## Exact commands and real results
|
|
||||||
|
|
||||||
### Targeted tests
|
|
||||||
|
|
||||||
```bash
|
|
||||||
pytest -q tests/test_performance_contract.py tests/test_route_session_benchmark.py
|
|
||||||
```
|
|
||||||
|
|
||||||
Result: `9 passed in 0.14s`
|
|
||||||
|
|
||||||
### Contract artifact generation
|
|
||||||
|
|
||||||
```bash
|
|
||||||
PYTHONPATH=packages/node python -m meshnet_node.performance_contract --json-out .scratch/distributed-gguf-runtime/evidence/DGR-001/performance-contract.json
|
|
||||||
```
|
|
||||||
|
|
||||||
Result: wrote `.scratch/distributed-gguf-runtime/evidence/DGR-001/performance-contract.json`
|
|
||||||
|
|
||||||
### Python compile check
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python -m compileall packages/node/meshnet_node/performance_contract.py tests/test_performance_contract.py
|
|
||||||
```
|
|
||||||
|
|
||||||
Result: passed
|
|
||||||
|
|
||||||
## Limitations
|
|
||||||
|
|
||||||
- This slice still uses a deterministic stub backend for the core comparison matrix.
|
|
||||||
- It now also includes a live endpoint runner that can fan out one OpenAI-compatible request per lane when the caller provides endpoints.
|
|
||||||
- It does **not** download or run a real model from within the repo.
|
|
||||||
- Real safetensors vs GGUF execution, TTFT/prefill/decode measurements, RSS/VRAM capture, and output-drift comparison are still to be implemented against the contract.
|
|
||||||
|
|
||||||
## Compatibility notes
|
|
||||||
|
|
||||||
- The contract stays on the DeepSeek2 family to remain close to the DeepSeek-V4-Flash end goal.
|
|
||||||
- A smaller non-DeepSeek model can still be used later for loader-plumbing smoke tests, but it does not replace this baseline.
|
|
||||||
- Model artifacts must stay on the mounted drive and not under `/home`.
|
|
||||||
|
|
||||||
## Dependent-story handoff
|
|
||||||
|
|
||||||
Next implementation work should attach to this contract and add the live benchmark runner that actually compares:
|
|
||||||
|
|
||||||
1. current Transformers/safetensors recipe
|
|
||||||
2. whole-model llama.cpp GGUF recipe
|
|
||||||
|
|
||||||
using the same model architecture/revision and the same prompt/context/concurrency settings.
|
|
||||||
@@ -1,75 +0,0 @@
|
|||||||
{
|
|
||||||
"benchmark_lanes": [
|
|
||||||
{
|
|
||||||
"concurrency_levels": [
|
|
||||||
1,
|
|
||||||
4
|
|
||||||
],
|
|
||||||
"device": "cpu",
|
|
||||||
"id": "transformers-safetensors-cpu",
|
|
||||||
"recipe": "current safetensors recipe",
|
|
||||||
"runtime": "transformers"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"concurrency_levels": [
|
|
||||||
1,
|
|
||||||
4
|
|
||||||
],
|
|
||||||
"device": "cpu",
|
|
||||||
"id": "llama-cpp-gguf-cpu",
|
|
||||||
"recipe": "whole-model GGUF recipe",
|
|
||||||
"runtime": "llama.cpp"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"concurrency_levels": [
|
|
||||||
1,
|
|
||||||
4
|
|
||||||
],
|
|
||||||
"device": "gpu",
|
|
||||||
"id": "transformers-safetensors-gpu",
|
|
||||||
"recipe": "current safetensors recipe",
|
|
||||||
"runtime": "transformers"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"concurrency_levels": [
|
|
||||||
1,
|
|
||||||
4
|
|
||||||
],
|
|
||||||
"device": "gpu",
|
|
||||||
"id": "llama-cpp-gguf-gpu",
|
|
||||||
"recipe": "whole-model GGUF recipe",
|
|
||||||
"runtime": "llama.cpp"
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"metrics": [
|
|
||||||
"ttft_ms",
|
|
||||||
"prefill_tok_per_sec",
|
|
||||||
"decode_tok_per_sec",
|
|
||||||
"p50_latency_ms",
|
|
||||||
"p95_latency_ms",
|
|
||||||
"aggregate_throughput_tok_per_sec",
|
|
||||||
"rss_bytes",
|
|
||||||
"vram_bytes",
|
|
||||||
"artifact_bytes",
|
|
||||||
"failure_count",
|
|
||||||
"output_drift"
|
|
||||||
],
|
|
||||||
"model_target": {
|
|
||||||
"architecture": "deepseek2",
|
|
||||||
"comparison_policy": "same model/revision, closest practical low-footprint precision pair: BF16 safetensors versus Q2_K GGUF",
|
|
||||||
"gguf_quant": "Q2_K",
|
|
||||||
"gguf_repo": "second-state/DeepSeek-V2-Lite-Chat-GGUF",
|
|
||||||
"gguf_size_gb": 6.43,
|
|
||||||
"name": "DeepSeek-V2-Lite-Chat",
|
|
||||||
"rationale": "Smallest DeepSeek-family benchmark anchor that still points toward DeepSeek-V4-Flash; keeps the runtime on the DeepSeek2 path instead of falling back to a tiny but architecture-mismatched smoke model.",
|
|
||||||
"safetensors_precision": "bfloat16",
|
|
||||||
"safetensors_repo": "deepseek-ai/DeepSeek-V2-Lite-Chat"
|
|
||||||
},
|
|
||||||
"notes": [
|
|
||||||
"Real model execution stays opt-in and must keep model artifacts on the mounted drive.",
|
|
||||||
"Use the tiny fallback only for loader plumbing smoke tests; it does not replace the architecture-aligned baseline."
|
|
||||||
],
|
|
||||||
"schema_version": 1,
|
|
||||||
"stop_condition": "Stop if GGUF does not provide a meaningful speed or fit benefit over the safetensors baseline for the chosen DeepSeek-family model target.",
|
|
||||||
"story_id": "DGR-001"
|
|
||||||
}
|
|
||||||
@@ -1,247 +0,0 @@
|
|||||||
{
|
|
||||||
"comparisons": {
|
|
||||||
"cpu": {
|
|
||||||
"artifact_bytes_ratio": 0.2048,
|
|
||||||
"decode_speedup": 2.3333,
|
|
||||||
"gguf_benefit": true,
|
|
||||||
"gguf_lane": "llama-cpp-gguf-cpu",
|
|
||||||
"memory_bytes_ratio": 0.2152,
|
|
||||||
"memory_metric": "rss_bytes",
|
|
||||||
"output_drift": 0.0,
|
|
||||||
"safetensors_lane": "transformers-safetensors-cpu",
|
|
||||||
"ttft_speedup": 1.8947
|
|
||||||
},
|
|
||||||
"gpu": {
|
|
||||||
"artifact_bytes_ratio": 0.2048,
|
|
||||||
"decode_speedup": 1.5294,
|
|
||||||
"gguf_benefit": true,
|
|
||||||
"gguf_lane": "llama-cpp-gguf-gpu",
|
|
||||||
"memory_bytes_ratio": 0.2273,
|
|
||||||
"memory_metric": "vram_bytes",
|
|
||||||
"output_drift": 0.0,
|
|
||||||
"safetensors_lane": "transformers-safetensors-gpu",
|
|
||||||
"ttft_speedup": 1.6154
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"lanes": [
|
|
||||||
{
|
|
||||||
"concurrency_levels": [
|
|
||||||
1,
|
|
||||||
4
|
|
||||||
],
|
|
||||||
"device": "cpu",
|
|
||||||
"id": "transformers-safetensors-cpu",
|
|
||||||
"output_tokens": [
|
|
||||||
"mesh",
|
|
||||||
"activation",
|
|
||||||
"seam",
|
|
||||||
"baseline"
|
|
||||||
],
|
|
||||||
"recipe": "current safetensors recipe",
|
|
||||||
"results": [
|
|
||||||
{
|
|
||||||
"concurrency": 1,
|
|
||||||
"metrics": {
|
|
||||||
"aggregate_throughput_tok_per_sec": 6.0,
|
|
||||||
"artifact_bytes": 33715493273,
|
|
||||||
"decode_tok_per_sec": 6.0,
|
|
||||||
"failure_count": 0,
|
|
||||||
"output_drift": 0.0,
|
|
||||||
"p50_latency_ms": 166.6667,
|
|
||||||
"p95_latency_ms": 208.3334,
|
|
||||||
"prefill_tok_per_sec": 45.0,
|
|
||||||
"rss_bytes": 35433480192,
|
|
||||||
"ttft_ms": 1800.0,
|
|
||||||
"vram_bytes": 0
|
|
||||||
}
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"concurrency": 4,
|
|
||||||
"metrics": {
|
|
||||||
"aggregate_throughput_tok_per_sec": 20.4,
|
|
||||||
"artifact_bytes": 33715493273,
|
|
||||||
"decode_tok_per_sec": 5.1,
|
|
||||||
"failure_count": 0,
|
|
||||||
"output_drift": 0.0,
|
|
||||||
"p50_latency_ms": 196.0784,
|
|
||||||
"p95_latency_ms": 245.098,
|
|
||||||
"prefill_tok_per_sec": 38.25,
|
|
||||||
"rss_bytes": 35433480192,
|
|
||||||
"ttft_ms": 2340.0,
|
|
||||||
"vram_bytes": 0
|
|
||||||
}
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"runtime": "transformers"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"concurrency_levels": [
|
|
||||||
1,
|
|
||||||
4
|
|
||||||
],
|
|
||||||
"device": "cpu",
|
|
||||||
"id": "llama-cpp-gguf-cpu",
|
|
||||||
"output_tokens": [
|
|
||||||
"mesh",
|
|
||||||
"activation",
|
|
||||||
"seam",
|
|
||||||
"baseline"
|
|
||||||
],
|
|
||||||
"recipe": "whole-model GGUF recipe",
|
|
||||||
"results": [
|
|
||||||
{
|
|
||||||
"concurrency": 1,
|
|
||||||
"metrics": {
|
|
||||||
"aggregate_throughput_tok_per_sec": 14.0,
|
|
||||||
"artifact_bytes": 6904159928,
|
|
||||||
"decode_tok_per_sec": 14.0,
|
|
||||||
"failure_count": 0,
|
|
||||||
"output_drift": 0.0,
|
|
||||||
"p50_latency_ms": 71.4286,
|
|
||||||
"p95_latency_ms": 89.2858,
|
|
||||||
"prefill_tok_per_sec": 90.0,
|
|
||||||
"rss_bytes": 7623566950,
|
|
||||||
"ttft_ms": 950.0,
|
|
||||||
"vram_bytes": 0
|
|
||||||
}
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"concurrency": 4,
|
|
||||||
"metrics": {
|
|
||||||
"aggregate_throughput_tok_per_sec": 47.6,
|
|
||||||
"artifact_bytes": 6904159928,
|
|
||||||
"decode_tok_per_sec": 11.9,
|
|
||||||
"failure_count": 0,
|
|
||||||
"output_drift": 0.0,
|
|
||||||
"p50_latency_ms": 84.0336,
|
|
||||||
"p95_latency_ms": 105.042,
|
|
||||||
"prefill_tok_per_sec": 76.5,
|
|
||||||
"rss_bytes": 7623566950,
|
|
||||||
"ttft_ms": 1235.0,
|
|
||||||
"vram_bytes": 0
|
|
||||||
}
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"runtime": "llama.cpp"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"concurrency_levels": [
|
|
||||||
1,
|
|
||||||
4
|
|
||||||
],
|
|
||||||
"device": "gpu",
|
|
||||||
"id": "transformers-safetensors-gpu",
|
|
||||||
"output_tokens": [
|
|
||||||
"mesh",
|
|
||||||
"activation",
|
|
||||||
"seam",
|
|
||||||
"baseline"
|
|
||||||
],
|
|
||||||
"recipe": "current safetensors recipe",
|
|
||||||
"results": [
|
|
||||||
{
|
|
||||||
"concurrency": 1,
|
|
||||||
"metrics": {
|
|
||||||
"aggregate_throughput_tok_per_sec": 34.0,
|
|
||||||
"artifact_bytes": 33715493273,
|
|
||||||
"decode_tok_per_sec": 34.0,
|
|
||||||
"failure_count": 0,
|
|
||||||
"output_drift": 0.0,
|
|
||||||
"p50_latency_ms": 29.4118,
|
|
||||||
"p95_latency_ms": 36.7647,
|
|
||||||
"prefill_tok_per_sec": 850.0,
|
|
||||||
"rss_bytes": 4294967296,
|
|
||||||
"ttft_ms": 420.0,
|
|
||||||
"vram_bytes": 35433480192
|
|
||||||
}
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"concurrency": 4,
|
|
||||||
"metrics": {
|
|
||||||
"aggregate_throughput_tok_per_sec": 115.6,
|
|
||||||
"artifact_bytes": 33715493273,
|
|
||||||
"decode_tok_per_sec": 28.9,
|
|
||||||
"failure_count": 0,
|
|
||||||
"output_drift": 0.0,
|
|
||||||
"p50_latency_ms": 34.6021,
|
|
||||||
"p95_latency_ms": 43.2526,
|
|
||||||
"prefill_tok_per_sec": 722.5,
|
|
||||||
"rss_bytes": 4294967296,
|
|
||||||
"ttft_ms": 546.0,
|
|
||||||
"vram_bytes": 35433480192
|
|
||||||
}
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"runtime": "transformers"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"concurrency_levels": [
|
|
||||||
1,
|
|
||||||
4
|
|
||||||
],
|
|
||||||
"device": "gpu",
|
|
||||||
"id": "llama-cpp-gguf-gpu",
|
|
||||||
"output_tokens": [
|
|
||||||
"mesh",
|
|
||||||
"activation",
|
|
||||||
"seam",
|
|
||||||
"baseline"
|
|
||||||
],
|
|
||||||
"recipe": "whole-model GGUF recipe",
|
|
||||||
"results": [
|
|
||||||
{
|
|
||||||
"concurrency": 1,
|
|
||||||
"metrics": {
|
|
||||||
"aggregate_throughput_tok_per_sec": 52.0,
|
|
||||||
"artifact_bytes": 6904159928,
|
|
||||||
"decode_tok_per_sec": 52.0,
|
|
||||||
"failure_count": 0,
|
|
||||||
"output_drift": 0.0,
|
|
||||||
"p50_latency_ms": 19.2308,
|
|
||||||
"p95_latency_ms": 24.0385,
|
|
||||||
"prefill_tok_per_sec": 640.0,
|
|
||||||
"rss_bytes": 1610612736,
|
|
||||||
"ttft_ms": 260.0,
|
|
||||||
"vram_bytes": 8053063680
|
|
||||||
}
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"concurrency": 4,
|
|
||||||
"metrics": {
|
|
||||||
"aggregate_throughput_tok_per_sec": 176.8,
|
|
||||||
"artifact_bytes": 6904159928,
|
|
||||||
"decode_tok_per_sec": 44.2,
|
|
||||||
"failure_count": 0,
|
|
||||||
"output_drift": 0.0,
|
|
||||||
"p50_latency_ms": 22.6244,
|
|
||||||
"p95_latency_ms": 28.2805,
|
|
||||||
"prefill_tok_per_sec": 544.0,
|
|
||||||
"rss_bytes": 1610612736,
|
|
||||||
"ttft_ms": 338.0,
|
|
||||||
"vram_bytes": 8053063680
|
|
||||||
}
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"runtime": "llama.cpp"
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"model_target": {
|
|
||||||
"architecture": "deepseek2",
|
|
||||||
"comparison_policy": "same model/revision, closest practical low-footprint precision pair: BF16 safetensors versus Q2_K GGUF",
|
|
||||||
"gguf_quant": "Q2_K",
|
|
||||||
"gguf_repo": "second-state/DeepSeek-V2-Lite-Chat-GGUF",
|
|
||||||
"gguf_size_gb": 6.43,
|
|
||||||
"name": "DeepSeek-V2-Lite-Chat",
|
|
||||||
"rationale": "Smallest DeepSeek-family benchmark anchor that still points toward DeepSeek-V4-Flash; keeps the runtime on the DeepSeek2 path instead of falling back to a tiny but architecture-mismatched smoke model.",
|
|
||||||
"safetensors_precision": "bfloat16",
|
|
||||||
"safetensors_repo": "deepseek-ai/DeepSeek-V2-Lite-Chat"
|
|
||||||
},
|
|
||||||
"schema_version": 1,
|
|
||||||
"source": "stub-backend",
|
|
||||||
"stop_condition": {
|
|
||||||
"gguf_benefit": true,
|
|
||||||
"text": "Stop if GGUF does not provide a meaningful speed or fit benefit over the safetensors baseline for the chosen DeepSeek-family model target.",
|
|
||||||
"triggered": false
|
|
||||||
},
|
|
||||||
"story_id": "DGR-001"
|
|
||||||
}
|
|
||||||
@@ -13,15 +13,6 @@ Status: ready-for-agent
|
|||||||
|
|
||||||
As a runtime engineer, I need a controlled baseline so that GGUF work proceeds from measured speed, memory, and quality rather than reputation.
|
As a runtime engineer, I need a controlled baseline so that GGUF work proceeds from measured speed, memory, and quality rather than reputation.
|
||||||
|
|
||||||
## Baseline model target
|
|
||||||
|
|
||||||
Use the same model on both sides of the comparison, with the closest practical low-footprint precision pair:
|
|
||||||
|
|
||||||
- **safetensors:** `deepseek-ai/DeepSeek-V2-Lite-Chat` in **BF16**
|
|
||||||
- **GGUF:** `second-state/DeepSeek-V2-Lite-Chat-GGUF` in **Q2_K** (~6.5GB)
|
|
||||||
|
|
||||||
Keep the benchmark matrix explicit for **CPU** and **GPU** runs. Reserve smaller non-DeepSeek fallback models only for loader plumbing smoke tests if needed; they do not count as the DGR-001 architecture-aligned baseline.
|
|
||||||
|
|
||||||
## Expected durable outputs
|
## Expected durable outputs
|
||||||
|
|
||||||
- Benchmark harness and deterministic tests
|
- Benchmark harness and deterministic tests
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
Status: done (2026-07-14)
|
Status: ready-for-agent
|
||||||
|
|
||||||
# 01 — Baseline and profiling harness
|
# 01 — Baseline and profiling harness
|
||||||
|
|
||||||
@@ -12,15 +12,16 @@ sizes and connection counts without requiring a real model or external host.
|
|||||||
|
|
||||||
## Acceptance criteria
|
## Acceptance criteria
|
||||||
|
|
||||||
- [x] The harness runs a fixed prompt and fixed generated-token count through a
|
- [ ] The harness runs a fixed prompt and fixed generated-token count through a
|
||||||
two-node route in direct and relay modes.
|
two-node route in direct and relay modes.
|
||||||
- [x] It reports p50/p95 per-token latency, per-hop latency, payload bytes,
|
- [ ] It reports p50/p95 per-token latency, per-hop latency, payload bytes,
|
||||||
compression ratio, connection attempts, and queue wait.
|
compression ratio, connection attempts, and queue wait.
|
||||||
- [x] It distinguishes prefill from decode and cached from stateless mode.
|
- [ ] It distinguishes prefill from decode and cached from stateless mode.
|
||||||
- [x] It emits machine-readable JSON suitable for CI artifacts and a concise
|
- [ ] It emits machine-readable JSON suitable for CI artifacts and a concise
|
||||||
human-readable summary.
|
human-readable summary.
|
||||||
- [x] A test fixture can assert connection attempts and output token identity.
|
- [ ] A test fixture can assert connection attempts and output token identity.
|
||||||
|
|
||||||
## Blocked by
|
## Blocked by
|
||||||
|
|
||||||
None - completed. Verified with `PYTHONPATH=packages/node pytest -q tests/test_route_session_benchmark.py` (7 passed).
|
None - can start immediately.
|
||||||
|
|
||||||
|
|||||||
@@ -15,10 +15,9 @@
|
|||||||
"Can assert connection count and output token identity"
|
"Can assert connection count and output token identity"
|
||||||
],
|
],
|
||||||
"priority": 1,
|
"priority": 1,
|
||||||
"passes": true,
|
"passes": false,
|
||||||
"notes": "Source issue: .scratch/distributed-inference-performance/issues/01-baseline-profiling-harness.md",
|
"notes": "Source issue: .scratch/distributed-inference-performance/issues/01-baseline-profiling-harness.md",
|
||||||
"dependsOn": [],
|
"dependsOn": []
|
||||||
"completionNotes": "Completed by agent"
|
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"id": "DIP-002",
|
"id": "DIP-002",
|
||||||
@@ -32,12 +31,9 @@
|
|||||||
"Tests cover binary, JSON, timeout, disconnect, cancellation, and cleanup"
|
"Tests cover binary, JSON, timeout, disconnect, cancellation, and cleanup"
|
||||||
],
|
],
|
||||||
"priority": 2,
|
"priority": 2,
|
||||||
"passes": true,
|
"passes": false,
|
||||||
"notes": "Source issue: .scratch/distributed-inference-performance/issues/02-relay-session-compatibility.md",
|
"notes": "Source issue: .scratch/distributed-inference-performance/issues/02-relay-session-compatibility.md",
|
||||||
"dependsOn": [
|
"dependsOn": ["DIP-001"]
|
||||||
"DIP-001"
|
|
||||||
],
|
|
||||||
"completionNotes": "Completed by agent"
|
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"id": "DIP-003",
|
"id": "DIP-003",
|
||||||
@@ -51,12 +47,9 @@
|
|||||||
"Benchmark shows healthy-session connection count independent of token count"
|
"Benchmark shows healthy-session connection count independent of token count"
|
||||||
],
|
],
|
||||||
"priority": 3,
|
"priority": 3,
|
||||||
"passes": true,
|
"passes": false,
|
||||||
"notes": "Source issue: .scratch/distributed-inference-performance/issues/03-http-keepalive.md",
|
"notes": "Source issue: .scratch/distributed-inference-performance/issues/03-http-keepalive.md",
|
||||||
"dependsOn": [
|
"dependsOn": ["DIP-001"]
|
||||||
"DIP-001"
|
|
||||||
],
|
|
||||||
"completionNotes": "Completed by agent"
|
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"id": "DIP-004",
|
"id": "DIP-004",
|
||||||
@@ -70,12 +63,9 @@
|
|||||||
"Tests verify cadence and cleanup"
|
"Tests verify cadence and cleanup"
|
||||||
],
|
],
|
||||||
"priority": 4,
|
"priority": 4,
|
||||||
"passes": true,
|
"passes": false,
|
||||||
"notes": "Source issue: .scratch/distributed-inference-performance/issues/04-seam-telemetry.md",
|
"notes": "Source issue: .scratch/distributed-inference-performance/issues/04-seam-telemetry.md",
|
||||||
"dependsOn": [
|
"dependsOn": ["DIP-001"]
|
||||||
"DIP-001"
|
|
||||||
],
|
|
||||||
"completionNotes": "Completed by agent"
|
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"id": "DIP-005",
|
"id": "DIP-005",
|
||||||
@@ -89,12 +79,9 @@
|
|||||||
"Tests cover compressible, incompressible, threshold, malformed, and legacy bodies"
|
"Tests cover compressible, incompressible, threshold, malformed, and legacy bodies"
|
||||||
],
|
],
|
||||||
"priority": 5,
|
"priority": 5,
|
||||||
"passes": true,
|
"passes": false,
|
||||||
"notes": "Source issue: .scratch/distributed-inference-performance/issues/05-adaptive-compression.md",
|
"notes": "Source issue: .scratch/distributed-inference-performance/issues/05-adaptive-compression.md",
|
||||||
"dependsOn": [
|
"dependsOn": ["DIP-001"]
|
||||||
"DIP-001"
|
|
||||||
],
|
|
||||||
"completionNotes": "Completed by agent"
|
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"id": "DIP-006",
|
"id": "DIP-006",
|
||||||
@@ -108,12 +95,9 @@
|
|||||||
"Wire and token-output regression tests pass"
|
"Wire and token-output regression tests pass"
|
||||||
],
|
],
|
||||||
"priority": 6,
|
"priority": 6,
|
||||||
"passes": true,
|
"passes": false,
|
||||||
"notes": "Source issue: .scratch/distributed-inference-performance/issues/06-activation-framing-copies.md",
|
"notes": "Source issue: .scratch/distributed-inference-performance/issues/06-activation-framing-copies.md",
|
||||||
"dependsOn": [
|
"dependsOn": ["DIP-001"]
|
||||||
"DIP-001"
|
|
||||||
],
|
|
||||||
"completionNotes": "Completed by agent"
|
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"id": "DIP-007",
|
"id": "DIP-007",
|
||||||
@@ -127,13 +111,9 @@
|
|||||||
"Tests cover chunking, slow consumers, failure, and legacy peers"
|
"Tests cover chunking, slow consumers, failure, and legacy peers"
|
||||||
],
|
],
|
||||||
"priority": 7,
|
"priority": 7,
|
||||||
"passes": true,
|
"passes": false,
|
||||||
"notes": "Source issue: .scratch/distributed-inference-performance/issues/07-prefill-backpressure.md",
|
"notes": "Source issue: .scratch/distributed-inference-performance/issues/07-prefill-backpressure.md",
|
||||||
"dependsOn": [
|
"dependsOn": ["DIP-001", "DIP-004"]
|
||||||
"DIP-001",
|
|
||||||
"DIP-004"
|
|
||||||
],
|
|
||||||
"completionNotes": "Completed by agent"
|
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"id": "DIP-008",
|
"id": "DIP-008",
|
||||||
@@ -147,20 +127,9 @@
|
|||||||
"Gate verifies token identity, session stability, and resource cleanup"
|
"Gate verifies token identity, session stability, and resource cleanup"
|
||||||
],
|
],
|
||||||
"priority": 8,
|
"priority": 8,
|
||||||
"passes": true,
|
"passes": false,
|
||||||
"notes": "Source issue: .scratch/distributed-inference-performance/issues/08-end-to-end-performance-gate.md",
|
"notes": "Source issue: .scratch/distributed-inference-performance/issues/08-end-to-end-performance-gate.md",
|
||||||
"dependsOn": [
|
"dependsOn": ["DIP-002", "DIP-003", "DIP-004", "DIP-005", "DIP-006", "DIP-007"]
|
||||||
"DIP-002",
|
|
||||||
"DIP-003",
|
|
||||||
"DIP-004",
|
|
||||||
"DIP-005",
|
|
||||||
"DIP-006",
|
|
||||||
"DIP-007"
|
|
||||||
],
|
|
||||||
"completionNotes": "Completed by agent"
|
|
||||||
}
|
}
|
||||||
],
|
]
|
||||||
"metadata": {
|
}
|
||||||
"updatedAt": "2026-07-12T02:35:28.752Z"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -35,12 +35,11 @@
|
|||||||
"Full pytest passes or an exact unrelated blocker is recorded"
|
"Full pytest passes or an exact unrelated blocker is recorded"
|
||||||
],
|
],
|
||||||
"priority": 2,
|
"priority": 2,
|
||||||
"passes": true,
|
"passes": false,
|
||||||
"notes": "Source issue: .scratch/node-capability-admission/issues/02-doctor-real-forward.md",
|
"notes": "Source issue: .scratch/node-capability-admission/issues/02-doctor-real-forward.md",
|
||||||
"dependsOn": [
|
"dependsOn": [
|
||||||
"NCA-001"
|
"NCA-001"
|
||||||
],
|
]
|
||||||
"completionNotes": "Completed by agent"
|
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"id": "NCA-003",
|
"id": "NCA-003",
|
||||||
@@ -55,13 +54,12 @@
|
|||||||
"Full pytest passes or an exact unrelated blocker is recorded"
|
"Full pytest passes or an exact unrelated blocker is recorded"
|
||||||
],
|
],
|
||||||
"priority": 3,
|
"priority": 3,
|
||||||
"passes": true,
|
"passes": false,
|
||||||
"notes": "Source issue: .scratch/node-capability-admission/issues/03-fail-closed-startup-admission.md",
|
"notes": "Source issue: .scratch/node-capability-admission/issues/03-fail-closed-startup-admission.md",
|
||||||
"dependsOn": [
|
"dependsOn": [
|
||||||
"NCA-001",
|
"NCA-001",
|
||||||
"NCA-002"
|
"NCA-002"
|
||||||
],
|
]
|
||||||
"completionNotes": "Completed by agent"
|
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"id": "NCA-004",
|
"id": "NCA-004",
|
||||||
@@ -78,13 +76,12 @@
|
|||||||
"Full pytest passes or an exact unrelated blocker is recorded"
|
"Full pytest passes or an exact unrelated blocker is recorded"
|
||||||
],
|
],
|
||||||
"priority": 4,
|
"priority": 4,
|
||||||
"passes": true,
|
"passes": false,
|
||||||
"notes": "Source issue: .scratch/node-capability-admission/issues/04-tracker-validated-capability-routing.md",
|
"notes": "Source issue: .scratch/node-capability-admission/issues/04-tracker-validated-capability-routing.md",
|
||||||
"dependsOn": [
|
"dependsOn": [
|
||||||
"NCA-001",
|
"NCA-001",
|
||||||
"NCA-003"
|
"NCA-003"
|
||||||
],
|
]
|
||||||
"completionNotes": "Completed by agent"
|
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"id": "NCA-005",
|
"id": "NCA-005",
|
||||||
@@ -99,16 +96,15 @@
|
|||||||
"Full pytest passes or an exact unrelated blocker is recorded"
|
"Full pytest passes or an exact unrelated blocker is recorded"
|
||||||
],
|
],
|
||||||
"priority": 5,
|
"priority": 5,
|
||||||
"passes": true,
|
"passes": false,
|
||||||
"notes": "Source issue: .scratch/node-capability-admission/issues/05-docs-hardware-lane-contract.md",
|
"notes": "Source issue: .scratch/node-capability-admission/issues/05-docs-hardware-lane-contract.md",
|
||||||
"dependsOn": [
|
"dependsOn": [
|
||||||
"NCA-002",
|
"NCA-002",
|
||||||
"NCA-004"
|
"NCA-004"
|
||||||
],
|
]
|
||||||
"completionNotes": "Completed by agent"
|
|
||||||
}
|
}
|
||||||
],
|
],
|
||||||
"metadata": {
|
"metadata": {
|
||||||
"updatedAt": "2026-07-12T01:54:03.030Z"
|
"updatedAt": "2026-07-11T19:16:52.768Z"
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -1,452 +0,0 @@
|
|||||||
"""Versioned performance contract metadata and stub benchmark runner for DGR-001.
|
|
||||||
|
|
||||||
This module captures the *contract* first: the model family, architecture
|
|
||||||
alignment, benchmark lanes, and stop condition that benchmark runs must
|
|
||||||
satisfy. It also runs the contract's lanes through a deterministic stub
|
|
||||||
backend so the report data shape exists end to end. It never downloads or
|
|
||||||
executes a model; real transformers / llama.cpp backends plug in behind the
|
|
||||||
same ``run()`` seam later.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import json
|
|
||||||
import time
|
|
||||||
import urllib.request
|
|
||||||
from dataclasses import dataclass
|
|
||||||
from pathlib import Path
|
|
||||||
from typing import Mapping
|
|
||||||
|
|
||||||
SCHEMA_VERSION = 1
|
|
||||||
CONTRACT_ID = "DGR-001"
|
|
||||||
DEFAULT_OUTPUT_PATH = Path(".scratch/distributed-gguf-runtime/evidence/DGR-001/performance-contract.json")
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class ModelTarget:
|
|
||||||
"""Architecture-aligned model target for the DGR-001 benchmark contract."""
|
|
||||||
|
|
||||||
name: str
|
|
||||||
architecture: str
|
|
||||||
safetensors_repo: str
|
|
||||||
safetensors_precision: str
|
|
||||||
gguf_repo: str
|
|
||||||
gguf_quant: str
|
|
||||||
gguf_size_gb: float
|
|
||||||
comparison_policy: str
|
|
||||||
rationale: str
|
|
||||||
|
|
||||||
def to_dict(self) -> dict:
|
|
||||||
return {
|
|
||||||
"name": self.name,
|
|
||||||
"architecture": self.architecture,
|
|
||||||
"safetensors_repo": self.safetensors_repo,
|
|
||||||
"safetensors_precision": self.safetensors_precision,
|
|
||||||
"gguf_repo": self.gguf_repo,
|
|
||||||
"gguf_quant": self.gguf_quant,
|
|
||||||
"gguf_size_gb": self.gguf_size_gb,
|
|
||||||
"comparison_policy": self.comparison_policy,
|
|
||||||
"rationale": self.rationale,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class BenchmarkLane:
|
|
||||||
"""One side of the comparison the contract requires."""
|
|
||||||
|
|
||||||
id: str
|
|
||||||
runtime: str
|
|
||||||
device: str
|
|
||||||
recipe: str
|
|
||||||
concurrency_levels: tuple[int, ...]
|
|
||||||
|
|
||||||
def to_dict(self) -> dict:
|
|
||||||
return {
|
|
||||||
"id": self.id,
|
|
||||||
"runtime": self.runtime,
|
|
||||||
"device": self.device,
|
|
||||||
"recipe": self.recipe,
|
|
||||||
"concurrency_levels": list(self.concurrency_levels),
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class PerformanceContract:
|
|
||||||
"""Machine-readable contract for the DGR-001 benchmark story."""
|
|
||||||
|
|
||||||
schema_version: int
|
|
||||||
story_id: str
|
|
||||||
model_target: ModelTarget
|
|
||||||
benchmark_lanes: tuple[BenchmarkLane, ...]
|
|
||||||
metrics: tuple[str, ...]
|
|
||||||
stop_condition: str
|
|
||||||
notes: tuple[str, ...] = ()
|
|
||||||
|
|
||||||
def to_dict(self) -> dict:
|
|
||||||
return {
|
|
||||||
"schema_version": self.schema_version,
|
|
||||||
"story_id": self.story_id,
|
|
||||||
"model_target": self.model_target.to_dict(),
|
|
||||||
"benchmark_lanes": [lane.to_dict() for lane in self.benchmark_lanes],
|
|
||||||
"metrics": list(self.metrics),
|
|
||||||
"stop_condition": self.stop_condition,
|
|
||||||
"notes": list(self.notes),
|
|
||||||
}
|
|
||||||
|
|
||||||
def write_json(self, path: str | Path) -> Path:
|
|
||||||
path = Path(path)
|
|
||||||
path.parent.mkdir(parents=True, exist_ok=True)
|
|
||||||
path.write_text(json.dumps(self.to_dict(), indent=2, sort_keys=True) + "\n", encoding="utf-8")
|
|
||||||
return path
|
|
||||||
|
|
||||||
|
|
||||||
DEFAULT_CONTRACT = PerformanceContract(
|
|
||||||
schema_version=SCHEMA_VERSION,
|
|
||||||
story_id=CONTRACT_ID,
|
|
||||||
model_target=ModelTarget(
|
|
||||||
name="DeepSeek-V2-Lite-Chat",
|
|
||||||
architecture="deepseek2",
|
|
||||||
safetensors_repo="deepseek-ai/DeepSeek-V2-Lite-Chat",
|
|
||||||
safetensors_precision="bfloat16",
|
|
||||||
gguf_repo="second-state/DeepSeek-V2-Lite-Chat-GGUF",
|
|
||||||
gguf_quant="Q2_K",
|
|
||||||
gguf_size_gb=6.43,
|
|
||||||
comparison_policy=(
|
|
||||||
"same model/revision, closest practical low-footprint precision pair: "
|
|
||||||
"BF16 safetensors versus Q2_K GGUF"
|
|
||||||
),
|
|
||||||
rationale=(
|
|
||||||
"Smallest DeepSeek-family benchmark anchor that still points toward "
|
|
||||||
"DeepSeek-V4-Flash; keeps the runtime on the DeepSeek2 path instead "
|
|
||||||
"of falling back to a tiny but architecture-mismatched smoke model."
|
|
||||||
),
|
|
||||||
),
|
|
||||||
benchmark_lanes=(
|
|
||||||
BenchmarkLane(
|
|
||||||
id="transformers-safetensors-cpu",
|
|
||||||
runtime="transformers",
|
|
||||||
device="cpu",
|
|
||||||
recipe="current safetensors recipe",
|
|
||||||
concurrency_levels=(1, 4),
|
|
||||||
),
|
|
||||||
BenchmarkLane(
|
|
||||||
id="llama-cpp-gguf-cpu",
|
|
||||||
runtime="llama.cpp",
|
|
||||||
device="cpu",
|
|
||||||
recipe="whole-model GGUF recipe",
|
|
||||||
concurrency_levels=(1, 4),
|
|
||||||
),
|
|
||||||
BenchmarkLane(
|
|
||||||
id="transformers-safetensors-gpu",
|
|
||||||
runtime="transformers",
|
|
||||||
device="gpu",
|
|
||||||
recipe="current safetensors recipe",
|
|
||||||
concurrency_levels=(1, 4),
|
|
||||||
),
|
|
||||||
BenchmarkLane(
|
|
||||||
id="llama-cpp-gguf-gpu",
|
|
||||||
runtime="llama.cpp",
|
|
||||||
device="gpu",
|
|
||||||
recipe="whole-model GGUF recipe",
|
|
||||||
concurrency_levels=(1, 4),
|
|
||||||
),
|
|
||||||
),
|
|
||||||
metrics=(
|
|
||||||
"ttft_ms",
|
|
||||||
"prefill_tok_per_sec",
|
|
||||||
"decode_tok_per_sec",
|
|
||||||
"p50_latency_ms",
|
|
||||||
"p95_latency_ms",
|
|
||||||
"aggregate_throughput_tok_per_sec",
|
|
||||||
"rss_bytes",
|
|
||||||
"vram_bytes",
|
|
||||||
"artifact_bytes",
|
|
||||||
"failure_count",
|
|
||||||
"output_drift",
|
|
||||||
),
|
|
||||||
stop_condition=(
|
|
||||||
"Stop if GGUF does not provide a meaningful speed or fit benefit over the "
|
|
||||||
"safetensors baseline for the chosen DeepSeek-family model target."
|
|
||||||
),
|
|
||||||
notes=(
|
|
||||||
"Real model execution stays opt-in and must keep model artifacts on the mounted drive.",
|
|
||||||
"Use the tiny fallback only for loader plumbing smoke tests; it does not replace the architecture-aligned baseline.",
|
|
||||||
),
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def build_default_contract() -> PerformanceContract:
|
|
||||||
return DEFAULT_CONTRACT
|
|
||||||
|
|
||||||
|
|
||||||
BENCHMARK_SCHEMA_VERSION = 1
|
|
||||||
STUB_OUTPUT_TOKENS = ("mesh", "activation", "seam", "baseline")
|
|
||||||
# DeepSeek-V2-Lite is ~15.7B params at 2 bytes each; metadata only, nothing downloaded.
|
|
||||||
_SAFETENSORS_BF16_ARTIFACT_GB = 31.4
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class LaneSample:
|
|
||||||
"""Raw single-stream measurements one backend produces for a lane."""
|
|
||||||
|
|
||||||
ttft_ms: float
|
|
||||||
prefill_tok_per_sec: float
|
|
||||||
decode_tok_per_sec: float
|
|
||||||
rss_bytes: int
|
|
||||||
vram_bytes: int
|
|
||||||
artifact_bytes: int
|
|
||||||
output_tokens: tuple[str, ...]
|
|
||||||
failure_count: int = 0
|
|
||||||
|
|
||||||
|
|
||||||
def _gb(value: float) -> int:
|
|
||||||
return int(value * 1024**3)
|
|
||||||
|
|
||||||
|
|
||||||
class StubLaneBackend:
|
|
||||||
"""Deterministic placeholder measurements until real lane execution lands.
|
|
||||||
|
|
||||||
The numbers are synthetic but directionally shaped — the Q2_K GGUF loads a
|
|
||||||
far smaller artifact and decodes faster than BF16 safetensors — so the
|
|
||||||
comparison and stop-condition plumbing can be exercised in CI.
|
|
||||||
"""
|
|
||||||
|
|
||||||
source = "stub-backend"
|
|
||||||
|
|
||||||
# (runtime, device) -> (ttft_ms, prefill tok/s, decode tok/s, rss GB, vram GB)
|
|
||||||
_PROFILES = {
|
|
||||||
("transformers", "cpu"): (1800.0, 45.0, 6.0, 33.0, 0.0),
|
|
||||||
("llama.cpp", "cpu"): (950.0, 90.0, 14.0, 7.1, 0.0),
|
|
||||||
("transformers", "gpu"): (420.0, 850.0, 34.0, 4.0, 33.0),
|
|
||||||
("llama.cpp", "gpu"): (260.0, 640.0, 52.0, 1.5, 7.5),
|
|
||||||
}
|
|
||||||
|
|
||||||
def __init__(self, contract: PerformanceContract) -> None:
|
|
||||||
self._contract = contract
|
|
||||||
|
|
||||||
def run(self, lane: BenchmarkLane) -> LaneSample:
|
|
||||||
ttft_ms, prefill, decode, rss_gb, vram_gb = self._PROFILES[(lane.runtime, lane.device)]
|
|
||||||
artifact_gb = (
|
|
||||||
self._contract.model_target.gguf_size_gb
|
|
||||||
if lane.runtime == "llama.cpp"
|
|
||||||
else _SAFETENSORS_BF16_ARTIFACT_GB
|
|
||||||
)
|
|
||||||
return LaneSample(
|
|
||||||
ttft_ms=ttft_ms,
|
|
||||||
prefill_tok_per_sec=prefill,
|
|
||||||
decode_tok_per_sec=decode,
|
|
||||||
rss_bytes=_gb(rss_gb),
|
|
||||||
vram_bytes=_gb(vram_gb),
|
|
||||||
artifact_bytes=_gb(artifact_gb),
|
|
||||||
output_tokens=STUB_OUTPUT_TOKENS,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _output_drift(tokens: tuple[str, ...], reference: tuple[str, ...]) -> float:
|
|
||||||
"""Fraction of positions where a lane's output diverges from its reference."""
|
|
||||||
length = max(len(tokens), len(reference))
|
|
||||||
if length == 0:
|
|
||||||
return 0.0
|
|
||||||
mismatches = sum(a != b for a, b in zip(tokens, reference)) + abs(len(tokens) - len(reference))
|
|
||||||
return round(mismatches / length, 4)
|
|
||||||
|
|
||||||
|
|
||||||
def _metrics_for(sample: LaneSample, concurrency: int, output_drift: float) -> dict:
|
|
||||||
# Stub concurrency model: batching scales throughput at 85% efficiency and
|
|
||||||
# stretches per-request token latency and TTFT accordingly.
|
|
||||||
efficiency = 1.0 if concurrency == 1 else 0.85
|
|
||||||
p50_latency_ms = round(1000.0 / (sample.decode_tok_per_sec * efficiency), 4)
|
|
||||||
return {
|
|
||||||
"ttft_ms": round(sample.ttft_ms * (1 + 0.1 * (concurrency - 1)), 4),
|
|
||||||
"prefill_tok_per_sec": round(sample.prefill_tok_per_sec * efficiency, 4),
|
|
||||||
"decode_tok_per_sec": round(sample.decode_tok_per_sec * efficiency, 4),
|
|
||||||
"p50_latency_ms": p50_latency_ms,
|
|
||||||
"p95_latency_ms": round(p50_latency_ms * 1.25, 4),
|
|
||||||
"aggregate_throughput_tok_per_sec": round(sample.decode_tok_per_sec * concurrency * efficiency, 4),
|
|
||||||
"rss_bytes": sample.rss_bytes,
|
|
||||||
"vram_bytes": sample.vram_bytes,
|
|
||||||
"artifact_bytes": sample.artifact_bytes,
|
|
||||||
"failure_count": sample.failure_count,
|
|
||||||
"output_drift": output_drift,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def _compare_device(lanes: list[tuple[BenchmarkLane, LaneSample]], device: str) -> dict:
|
|
||||||
by_runtime = {lane.runtime: (lane, sample) for lane, sample in lanes if lane.device == device}
|
|
||||||
safetensors_lane, safetensors = by_runtime["transformers"]
|
|
||||||
gguf_lane, gguf = by_runtime["llama.cpp"]
|
|
||||||
memory_metric = "vram_bytes" if device == "gpu" else "rss_bytes"
|
|
||||||
decode_speedup = round(gguf.decode_tok_per_sec / safetensors.decode_tok_per_sec, 4)
|
|
||||||
artifact_bytes_ratio = round(gguf.artifact_bytes / max(1, safetensors.artifact_bytes), 4)
|
|
||||||
return {
|
|
||||||
"safetensors_lane": safetensors_lane.id,
|
|
||||||
"gguf_lane": gguf_lane.id,
|
|
||||||
"decode_speedup": decode_speedup,
|
|
||||||
"ttft_speedup": round(safetensors.ttft_ms / max(0.001, gguf.ttft_ms), 4),
|
|
||||||
"artifact_bytes_ratio": artifact_bytes_ratio,
|
|
||||||
"memory_metric": memory_metric,
|
|
||||||
"memory_bytes_ratio": round(
|
|
||||||
getattr(gguf, memory_metric) / max(1, getattr(safetensors, memory_metric)), 4
|
|
||||||
),
|
|
||||||
"output_drift": _output_drift(gguf.output_tokens, safetensors.output_tokens),
|
|
||||||
"gguf_benefit": decode_speedup >= 1.10 or artifact_bytes_ratio <= 0.5,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def run_performance_benchmark(
|
|
||||||
contract: PerformanceContract = DEFAULT_CONTRACT,
|
|
||||||
backend: StubLaneBackend | None = None,
|
|
||||||
) -> dict:
|
|
||||||
"""Run every contract lane through a backend and compare GGUF to safetensors."""
|
|
||||||
backend = backend if backend is not None else StubLaneBackend(contract)
|
|
||||||
lanes = [(lane, backend.run(lane)) for lane in contract.benchmark_lanes]
|
|
||||||
references = {
|
|
||||||
lane.device: sample.output_tokens for lane, sample in lanes if lane.runtime == "transformers"
|
|
||||||
}
|
|
||||||
lane_reports = []
|
|
||||||
for lane, sample in lanes:
|
|
||||||
drift = _output_drift(sample.output_tokens, references.get(lane.device, sample.output_tokens))
|
|
||||||
lane_reports.append({
|
|
||||||
**lane.to_dict(),
|
|
||||||
"output_tokens": list(sample.output_tokens),
|
|
||||||
"results": [
|
|
||||||
{"concurrency": level, "metrics": _metrics_for(sample, level, drift)}
|
|
||||||
for level in lane.concurrency_levels
|
|
||||||
],
|
|
||||||
})
|
|
||||||
devices = sorted({lane.device for lane, _ in lanes})
|
|
||||||
comparisons = {device: _compare_device(lanes, device) for device in devices}
|
|
||||||
gguf_benefit = any(comparison["gguf_benefit"] for comparison in comparisons.values())
|
|
||||||
return {
|
|
||||||
"schema_version": BENCHMARK_SCHEMA_VERSION,
|
|
||||||
"story_id": contract.story_id,
|
|
||||||
"source": getattr(backend, "source", "custom-backend"),
|
|
||||||
"model_target": contract.model_target.to_dict(),
|
|
||||||
"lanes": lane_reports,
|
|
||||||
"comparisons": comparisons,
|
|
||||||
"stop_condition": {
|
|
||||||
"text": contract.stop_condition,
|
|
||||||
"gguf_benefit": gguf_benefit,
|
|
||||||
"triggered": not gguf_benefit,
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def run_real_model_endpoint_benchmark(
|
|
||||||
endpoints: Mapping[str, str],
|
|
||||||
*,
|
|
||||||
model: str,
|
|
||||||
contract: PerformanceContract = DEFAULT_CONTRACT,
|
|
||||||
timeout: float = 120.0,
|
|
||||||
) -> dict:
|
|
||||||
"""Run one live OpenAI-compatible request per lane against supplied endpoints.
|
|
||||||
|
|
||||||
The caller provides one URL per benchmark lane. The runner measures the
|
|
||||||
request/response round-trip at the client boundary and reuses the same
|
|
||||||
contract schema as the deterministic stub.
|
|
||||||
"""
|
|
||||||
|
|
||||||
def _sample_for_lane(lane: BenchmarkLane, endpoint: str) -> LaneSample:
|
|
||||||
prompt = " ".join(contract.model_target.rationale.split()[:6])
|
|
||||||
body = json.dumps(
|
|
||||||
{
|
|
||||||
"model": model,
|
|
||||||
"messages": [{"role": "user", "content": prompt}],
|
|
||||||
"max_tokens": len(STUB_OUTPUT_TOKENS),
|
|
||||||
"temperature": 0,
|
|
||||||
}
|
|
||||||
).encode("utf-8")
|
|
||||||
request = urllib.request.Request(
|
|
||||||
f"{endpoint.rstrip('/')}/v1/chat/completions",
|
|
||||||
data=body,
|
|
||||||
headers={
|
|
||||||
"Content-Type": "application/json",
|
|
||||||
"X-Meshnet-Lane": lane.id,
|
|
||||||
},
|
|
||||||
method="POST",
|
|
||||||
)
|
|
||||||
started = time.monotonic()
|
|
||||||
with urllib.request.urlopen(request, timeout=timeout) as response:
|
|
||||||
response_body = response.read()
|
|
||||||
session_id = response.headers.get("X-Meshnet-Session", f"{lane.id}-session")
|
|
||||||
elapsed_ms = round((time.monotonic() - started) * 1000, 4)
|
|
||||||
payload = json.loads(response_body)
|
|
||||||
content = payload["choices"][0]["message"]["content"]
|
|
||||||
tokens = tuple(content.split())
|
|
||||||
token_count = max(1, len(tokens))
|
|
||||||
artifact_gb = (
|
|
||||||
contract.model_target.gguf_size_gb
|
|
||||||
if lane.runtime == "llama.cpp"
|
|
||||||
else _SAFETENSORS_BF16_ARTIFACT_GB
|
|
||||||
)
|
|
||||||
return LaneSample(
|
|
||||||
ttft_ms=elapsed_ms,
|
|
||||||
prefill_tok_per_sec=round(token_count / max(0.001, elapsed_ms / 1000), 4),
|
|
||||||
decode_tok_per_sec=round(token_count / max(0.001, elapsed_ms / 1000), 4),
|
|
||||||
rss_bytes=0,
|
|
||||||
vram_bytes=0,
|
|
||||||
artifact_bytes=_gb(artifact_gb),
|
|
||||||
output_tokens=tokens,
|
|
||||||
)
|
|
||||||
|
|
||||||
lanes = []
|
|
||||||
for lane in contract.benchmark_lanes:
|
|
||||||
if lane.id not in endpoints:
|
|
||||||
raise KeyError(f"missing endpoint for lane {lane.id}")
|
|
||||||
lanes.append((lane, _sample_for_lane(lane, endpoints[lane.id])))
|
|
||||||
references = {
|
|
||||||
lane.device: sample.output_tokens for lane, sample in lanes if lane.runtime == "transformers"
|
|
||||||
}
|
|
||||||
lane_reports = []
|
|
||||||
for lane, sample in lanes:
|
|
||||||
drift = _output_drift(sample.output_tokens, references.get(lane.device, sample.output_tokens))
|
|
||||||
lane_reports.append({
|
|
||||||
**lane.to_dict(),
|
|
||||||
"output_tokens": list(sample.output_tokens),
|
|
||||||
"results": [
|
|
||||||
{"concurrency": level, "metrics": _metrics_for(sample, level, drift)}
|
|
||||||
for level in lane.concurrency_levels
|
|
||||||
],
|
|
||||||
})
|
|
||||||
devices = sorted({lane.device for lane, _ in lanes})
|
|
||||||
comparisons = {device: _compare_device(lanes, device) for device in devices}
|
|
||||||
gguf_benefit = any(comparison["gguf_benefit"] for comparison in comparisons.values())
|
|
||||||
return {
|
|
||||||
"schema_version": BENCHMARK_SCHEMA_VERSION,
|
|
||||||
"story_id": contract.story_id,
|
|
||||||
"source": "real-model-endpoints",
|
|
||||||
"model_target": contract.model_target.to_dict(),
|
|
||||||
"lanes": lane_reports,
|
|
||||||
"comparisons": comparisons,
|
|
||||||
"stop_condition": {
|
|
||||||
"text": contract.stop_condition,
|
|
||||||
"gguf_benefit": gguf_benefit,
|
|
||||||
"triggered": not gguf_benefit,
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def main(argv: list[str] | None = None) -> int:
|
|
||||||
parser = argparse.ArgumentParser(description="Write the DGR-001 performance contract JSON")
|
|
||||||
parser.add_argument("--json-out", type=Path, default=DEFAULT_OUTPUT_PATH, help="output JSON path")
|
|
||||||
parser.add_argument(
|
|
||||||
"--benchmark-out",
|
|
||||||
type=Path,
|
|
||||||
default=None,
|
|
||||||
help="also run the deterministic stub benchmark and write its JSON report here",
|
|
||||||
)
|
|
||||||
args = parser.parse_args(argv)
|
|
||||||
contract = build_default_contract()
|
|
||||||
path = contract.write_json(args.json_out)
|
|
||||||
print(path)
|
|
||||||
if args.benchmark_out is not None:
|
|
||||||
report = run_performance_benchmark(contract)
|
|
||||||
args.benchmark_out.parent.mkdir(parents=True, exist_ok=True)
|
|
||||||
args.benchmark_out.write_text(json.dumps(report, indent=2, sort_keys=True) + "\n", encoding="utf-8")
|
|
||||||
print(args.benchmark_out)
|
|
||||||
return 0
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__": # pragma: no cover - CLI entry point
|
|
||||||
raise SystemExit(main())
|
|
||||||
@@ -44,7 +44,6 @@ class SeamSample:
|
|||||||
cache_mode: CacheMode
|
cache_mode: CacheMode
|
||||||
model_ms: float
|
model_ms: float
|
||||||
encode_ms: float
|
encode_ms: float
|
||||||
activation_decode_ms: float
|
|
||||||
framing_ms: float
|
framing_ms: float
|
||||||
metadata_ms: float
|
metadata_ms: float
|
||||||
copy_allocation_ms: float
|
copy_allocation_ms: float
|
||||||
@@ -53,7 +52,6 @@ class SeamSample:
|
|||||||
decompression_ms: float
|
decompression_ms: float
|
||||||
connection_setup_ms: float
|
connection_setup_ms: float
|
||||||
queue_wait_ms: float
|
queue_wait_ms: float
|
||||||
local_http_forwarding_ms: float
|
|
||||||
transport_ms: float
|
transport_ms: float
|
||||||
seam_latency_ms: float
|
seam_latency_ms: float
|
||||||
payload_bytes: int
|
payload_bytes: int
|
||||||
@@ -122,10 +120,6 @@ def _summary(samples: list[SeamSample]) -> dict[str, float | int]:
|
|||||||
"compression_cpu_ms": round(
|
"compression_cpu_ms": round(
|
||||||
sum(sample.compression_ms + sample.decompression_ms for sample in samples), 4
|
sum(sample.compression_ms + sample.decompression_ms for sample in samples), 4
|
||||||
),
|
),
|
||||||
"model_execution_ms": round(sum(sample.model_ms for sample in samples), 4),
|
|
||||||
"activation_encoding_ms": round(sum(sample.encode_ms for sample in samples), 4),
|
|
||||||
"activation_decoding_ms": round(sum(sample.activation_decode_ms for sample in samples), 4),
|
|
||||||
"local_http_forwarding_ms": round(sum(sample.local_http_forwarding_ms for sample in samples), 4),
|
|
||||||
"peak_buffered_bytes": max((sample.copy_allocation_bytes for sample in samples), default=0),
|
"peak_buffered_bytes": max((sample.copy_allocation_bytes for sample in samples), default=0),
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -165,7 +159,6 @@ class _StubTransport:
|
|||||||
queue_wait_ms = 0.0 if self.mode == "direct" else 0.18 + (0.05 if token_index is not None and token_index % 2 else 0.0)
|
queue_wait_ms = 0.0 if self.mode == "direct" else 0.18 + (0.05 if token_index is not None and token_index % 2 else 0.0)
|
||||||
model_ms = 1.6 if phase == "prefill" else 0.45
|
model_ms = 1.6 if phase == "prefill" else 0.45
|
||||||
encode_ms = 0.16 if phase == "prefill" else 0.06
|
encode_ms = 0.16 if phase == "prefill" else 0.06
|
||||||
activation_decode_ms = 0.055 if phase == "prefill" else 0.02
|
|
||||||
# Keep framing/metadata/copy costs explicit rather than hiding them in
|
# Keep framing/metadata/copy costs explicit rather than hiding them in
|
||||||
# serialization or transport time. The stub owns one binary frame and
|
# serialization or transport time. The stub owns one binary frame and
|
||||||
# one response body per hop; no base64 body is modeled.
|
# one response body per hop; no base64 body is modeled.
|
||||||
@@ -175,26 +168,20 @@ class _StubTransport:
|
|||||||
copy_allocation_bytes = wire_bytes + payload_bytes
|
copy_allocation_bytes = wire_bytes + payload_bytes
|
||||||
compression_ms = 0.09 if self.scenario.compression else 0.0
|
compression_ms = 0.09 if self.scenario.compression else 0.0
|
||||||
decompression_ms = 0.07 if self.scenario.compression else 0.0
|
decompression_ms = 0.07 if self.scenario.compression else 0.0
|
||||||
# Both routes finish by forwarding the decoded activation to the local
|
|
||||||
# tail-node HTTP handler; relay adds its own queue before that hop.
|
|
||||||
local_http_forwarding_ms = 0.11 if self.mode == "direct" else 0.16
|
|
||||||
transport_ms = (0.32 if self.mode == "direct" else 0.61) + wire_bytes / 100_000
|
transport_ms = (0.32 if self.mode == "direct" else 0.61) + wire_bytes / 100_000
|
||||||
seam_latency_ms = round(
|
seam_latency_ms = round(
|
||||||
model_ms + encode_ms + activation_decode_ms + framing_ms + metadata_ms + copy_allocation_ms
|
model_ms + encode_ms + framing_ms + metadata_ms + copy_allocation_ms
|
||||||
+ compression_ms + decompression_ms + connection_setup_ms + queue_wait_ms + transport_ms
|
+ compression_ms + decompression_ms + connection_setup_ms + queue_wait_ms + transport_ms,
|
||||||
+ local_http_forwarding_ms,
|
|
||||||
4,
|
4,
|
||||||
)
|
)
|
||||||
return SeamSample(
|
return SeamSample(
|
||||||
phase=phase, token_index=token_index, session_id=self.session_id,
|
phase=phase, token_index=token_index, session_id=self.session_id,
|
||||||
activation_id=f"benchmark-activation-{self._activation_count}", seam="head->tail", mode=self.mode,
|
activation_id=f"benchmark-activation-{self._activation_count}", seam="head->tail", mode=self.mode,
|
||||||
cache_mode=self.cache_mode, model_ms=model_ms, encode_ms=encode_ms,
|
cache_mode=self.cache_mode, model_ms=model_ms, encode_ms=encode_ms,
|
||||||
activation_decode_ms=activation_decode_ms,
|
|
||||||
framing_ms=framing_ms, metadata_ms=metadata_ms,
|
framing_ms=framing_ms, metadata_ms=metadata_ms,
|
||||||
copy_allocation_ms=copy_allocation_ms, copy_allocation_bytes=copy_allocation_bytes,
|
copy_allocation_ms=copy_allocation_ms, copy_allocation_bytes=copy_allocation_bytes,
|
||||||
compression_ms=compression_ms, decompression_ms=decompression_ms,
|
compression_ms=compression_ms, decompression_ms=decompression_ms,
|
||||||
connection_setup_ms=connection_setup_ms, queue_wait_ms=queue_wait_ms,
|
connection_setup_ms=connection_setup_ms, queue_wait_ms=queue_wait_ms,
|
||||||
local_http_forwarding_ms=local_http_forwarding_ms,
|
|
||||||
transport_ms=round(transport_ms, 4), seam_latency_ms=seam_latency_ms,
|
transport_ms=round(transport_ms, 4), seam_latency_ms=seam_latency_ms,
|
||||||
payload_bytes=payload_bytes, wire_bytes=wire_bytes,
|
payload_bytes=payload_bytes, wire_bytes=wire_bytes,
|
||||||
compression_ratio=round(payload_bytes / wire_bytes, 4), connection_attempted=connection_attempted,
|
compression_ratio=round(payload_bytes / wire_bytes, 4), connection_attempted=connection_attempted,
|
||||||
@@ -342,10 +329,9 @@ def run_real_model_lan_benchmark(url: str, *, model: str, timeout: float = 120.0
|
|||||||
sample = SeamSample(
|
sample = SeamSample(
|
||||||
phase="decode", token_index=0, session_id=session_id, activation_id="lan-activation-1",
|
phase="decode", token_index=0, session_id=session_id, activation_id="lan-activation-1",
|
||||||
seam="head->tail", mode="direct", cache_mode="cached", model_ms=0.0, encode_ms=0.0,
|
seam="head->tail", mode="direct", cache_mode="cached", model_ms=0.0, encode_ms=0.0,
|
||||||
activation_decode_ms=0.0,
|
|
||||||
framing_ms=0.0, metadata_ms=0.0, copy_allocation_ms=0.0, copy_allocation_bytes=0,
|
framing_ms=0.0, metadata_ms=0.0, copy_allocation_ms=0.0, copy_allocation_bytes=0,
|
||||||
compression_ms=0.0, decompression_ms=0.0, connection_setup_ms=elapsed_ms,
|
compression_ms=0.0, decompression_ms=0.0, connection_setup_ms=elapsed_ms,
|
||||||
queue_wait_ms=0.0, local_http_forwarding_ms=0.0, transport_ms=elapsed_ms, seam_latency_ms=elapsed_ms,
|
queue_wait_ms=0.0, transport_ms=elapsed_ms, seam_latency_ms=elapsed_ms,
|
||||||
payload_bytes=len(body), wire_bytes=len(body) + len(response_body), compression_ratio=1.0,
|
payload_bytes=len(body), wire_bytes=len(body) + len(response_body), compression_ratio=1.0,
|
||||||
connection_attempted=True,
|
connection_attempted=True,
|
||||||
)
|
)
|
||||||
@@ -368,10 +354,6 @@ def format_summary(report: dict) -> str:
|
|||||||
f"{decode['tokens_per_sec']:.1f} tok/s; {decode['bytes_per_token']:.0f} B/tok; "
|
f"{decode['tokens_per_sec']:.1f} tok/s; {decode['bytes_per_token']:.0f} B/tok; "
|
||||||
f"seam {seam['payload_bytes']}/{seam['wire_bytes']} B "
|
f"seam {seam['payload_bytes']}/{seam['wire_bytes']} B "
|
||||||
f"({seam['compression_ratio']:.2f}x); connections {run['connections']['attempts']}; "
|
f"({seam['compression_ratio']:.2f}x); connections {run['connections']['attempts']}; "
|
||||||
f"model/encode/decode {decode['model_execution_ms']:.2f}/"
|
|
||||||
f"{decode['activation_encoding_ms']:.2f}/{decode['activation_decoding_ms']:.2f} ms; "
|
|
||||||
f"compression {decode['compression_cpu_ms']:.2f} ms; "
|
|
||||||
f"HTTP {decode['local_http_forwarding_ms']:.2f} ms; "
|
|
||||||
f"queue p95 {decode['p95_queue_wait_ms']:.2f} ms"
|
f"queue p95 {decode['p95_queue_wait_ms']:.2f} ms"
|
||||||
)
|
)
|
||||||
return "\n".join(lines)
|
return "\n".join(lines)
|
||||||
|
|||||||
@@ -1545,26 +1545,6 @@ class TorchNodeServer:
|
|||||||
|
|
||||||
def apply_tracker_directives(self, directives: list[dict]) -> dict | None:
|
def apply_tracker_directives(self, directives: list[dict]) -> dict | None:
|
||||||
"""Apply tracker shard directives (LOAD_SHARD replace, ADD_SHARD load-more)."""
|
"""Apply tracker shard directives (LOAD_SHARD replace, ADD_SHARD load-more)."""
|
||||||
drop_directive = next(
|
|
||||||
(directive for directive in reversed(directives) if directive.get("action") == "DROP_SHARD"),
|
|
||||||
None,
|
|
||||||
)
|
|
||||||
if drop_directive is not None:
|
|
||||||
model_id = str(drop_directive.get("model") or "")
|
|
||||||
removed = self._backends.pop(model_id, None)
|
|
||||||
if removed is None:
|
|
||||||
return None
|
|
||||||
if self._backends:
|
|
||||||
self._backend = next(iter(self._backends.values()))
|
|
||||||
self._tracker_mode = self._backend.shard_start == 0
|
|
||||||
else:
|
|
||||||
self._backend = None
|
|
||||||
self._tracker_mode = False
|
|
||||||
if self._server is not None:
|
|
||||||
self._server.backends = dict(self._backends)
|
|
||||||
self._server.backend = self._backend
|
|
||||||
self._server.tracker_mode = self._tracker_mode
|
|
||||||
return {"action": "DROP_SHARD", "model": model_id}
|
|
||||||
add_directive = next(
|
add_directive = next(
|
||||||
(directive for directive in reversed(directives) if directive.get("action") == "ADD_SHARD"),
|
(directive for directive in reversed(directives) if directive.get("action") == "ADD_SHARD"),
|
||||||
None,
|
None,
|
||||||
@@ -1594,8 +1574,6 @@ class TorchNodeServer:
|
|||||||
flush=True,
|
flush=True,
|
||||||
)
|
)
|
||||||
try:
|
try:
|
||||||
if replacing:
|
|
||||||
self._backends.clear()
|
|
||||||
new_backend = _load_backend(model_id, shard_start, shard_end, quantization, self._cache_dir)
|
new_backend = _load_backend(model_id, shard_start, shard_end, quantization, self._cache_dir)
|
||||||
except TypeError:
|
except TypeError:
|
||||||
new_backend = _load_backend(model_id, shard_start, shard_end, quantization)
|
new_backend = _load_backend(model_id, shard_start, shard_end, quantization)
|
||||||
|
|||||||
@@ -22,9 +22,8 @@
|
|||||||
border-bottom:1px solid var(--border); flex-shrink:0; }
|
border-bottom:1px solid var(--border); flex-shrink:0; }
|
||||||
header h1 { font-size:16px; margin:0; color:var(--accent); }
|
header h1 { font-size:16px; margin:0; color:var(--accent); }
|
||||||
header .meta { color:var(--dim); font-size:12px; }
|
header .meta { color:var(--dim); font-size:12px; }
|
||||||
main { display:grid; grid-template-columns:1fr;
|
main { display:grid; grid-template-columns:repeat(auto-fit,minmax(340px,1fr));
|
||||||
gap:14px; padding:14px 20px; }
|
gap:14px; padding:14px 20px; }
|
||||||
main > section { width:100%; min-width:0; }
|
|
||||||
body.chat-tab-active main {
|
body.chat-tab-active main {
|
||||||
flex:1; min-height:0; display:flex; flex-direction:column;
|
flex:1; min-height:0; display:flex; flex-direction:column;
|
||||||
padding:0; gap:0; overflow:hidden;
|
padding:0; gap:0; overflow:hidden;
|
||||||
@@ -72,12 +71,6 @@
|
|||||||
background:transparent; color:var(--dim); padding:5px 0 8px; }
|
background:transparent; color:var(--dim); padding:5px 0 8px; }
|
||||||
.dashboard-tabs button.active { color:var(--accent); border-bottom-color:var(--accent); }
|
.dashboard-tabs button.active { color:var(--accent); border-bottom-color:var(--accent); }
|
||||||
.wide { grid-column:1 / -1; }
|
.wide { grid-column:1 / -1; }
|
||||||
/* Compact status cards fan out on desktop; tables remain readable at half width. */
|
|
||||||
@media (min-width:900px) {
|
|
||||||
main { grid-template-columns:repeat(4,minmax(0,1fr)); }
|
|
||||||
main > section { grid-column:span 1; }
|
|
||||||
.wide { grid-column:span 2; }
|
|
||||||
}
|
|
||||||
section[hidden] { display:none !important; }
|
section[hidden] { display:none !important; }
|
||||||
section.chat-section {
|
section.chat-section {
|
||||||
padding:0; border:0; border-radius:0; background:var(--bg); min-height:0;
|
padding:0; border:0; border-radius:0; background:var(--bg); min-height:0;
|
||||||
@@ -295,7 +288,6 @@
|
|||||||
<section data-tab="billing" data-admin-only><h2>Node pending payouts</h2><div id="pending" class="empty">admin login required</div></section>
|
<section data-tab="billing" data-admin-only><h2>Node pending payouts</h2><div id="pending" class="empty">admin login required</div></section>
|
||||||
<section data-tab="billing" data-admin-only><h2>Settlement history</h2><div id="settlements" class="empty">admin login required</div></section>
|
<section data-tab="billing" data-admin-only><h2>Settlement history</h2><div id="settlements" class="empty">admin login required</div></section>
|
||||||
<section data-tab="admin"><h2>Tracker hive</h2><div id="hive" class="empty">loading…</div></section>
|
<section data-tab="admin"><h2>Tracker hive</h2><div id="hive" class="empty">loading…</div></section>
|
||||||
<section data-tab="admin" class="wide"><h2>Model placement</h2><div id="admin-model-placement-status" class="dim">Choose a model to load or release.</div><div id="admin-model-placement" class="empty">admin login required</div></section>
|
|
||||||
<section data-tab="admin" id="admin-section"><h2>All accounts (admin)</h2><div id="admin" class="empty"></div></section>
|
<section data-tab="admin" id="admin-section"><h2>All accounts (admin)</h2><div id="admin" class="empty"></div></section>
|
||||||
<section data-tab="admin" data-admin-only><h2>Strikes / bans / forfeitures</h2><div id="fraud" class="empty">admin login required</div></section>
|
<section data-tab="admin" data-admin-only><h2>Strikes / bans / forfeitures</h2><div id="fraud" class="empty">admin login required</div></section>
|
||||||
<section data-tab="admin"><h2>Client balances</h2><div id="clients" class="empty">admin login required</div></section>
|
<section data-tab="admin"><h2>Client balances</h2><div id="clients" class="empty">admin login required</div></section>
|
||||||
@@ -1799,50 +1791,6 @@ async function requestSelectedModelLoad() {
|
|||||||
$("chat-status").textContent = `load queued on ${short(assignment.node_id || "node")} for layers ${assignment.shard_start}-${assignment.shard_end}`;
|
$("chat-status").textContent = `load queued on ${short(assignment.node_id || "node")} for layers ${assignment.shard_start}-${assignment.shard_end}`;
|
||||||
}
|
}
|
||||||
|
|
||||||
async function requestAdminModelLoad(model) {
|
|
||||||
const result = await apiCall("/v1/models/load", "POST", { model, force: true });
|
|
||||||
if (!result.ok) return showAdminModelPlacementStatus(result.data.error || "model load request failed", true);
|
|
||||||
const assignment = result.data.assignment || {};
|
|
||||||
showAdminModelPlacementStatus(`Load queued on ${short(assignment.node_id || "node")} for ${model}.`);
|
|
||||||
await refreshActiveTab(true);
|
|
||||||
}
|
|
||||||
|
|
||||||
async function releaseAdminModel(model) {
|
|
||||||
const result = await apiCall("/v1/models/release", "POST", { model });
|
|
||||||
if (!result.ok) return showAdminModelPlacementStatus(result.data.error || "model release request failed", true);
|
|
||||||
showAdminModelPlacementStatus(`Release queued for ${result.data.released || 0} node(s) serving ${model}.`);
|
|
||||||
await refreshActiveTab(true);
|
|
||||||
}
|
|
||||||
|
|
||||||
function showAdminModelPlacementStatus(message, isError) {
|
|
||||||
const status = $("admin-model-placement-status");
|
|
||||||
status.textContent = message;
|
|
||||||
status.className = isError ? "bad" : "ok";
|
|
||||||
}
|
|
||||||
|
|
||||||
function renderAdminModelPlacement(models, map) {
|
|
||||||
const nodes = (map && map.nodes) || [];
|
|
||||||
const rows = ((models && models.data) || []).map(model => {
|
|
||||||
const aliases = new Set([model.id, model.hf_repo, ...(model.aliases || [])].filter(Boolean));
|
|
||||||
const serving = nodes.filter(node => aliases.has(node.model) || aliases.has(node.hf_repo)).length;
|
|
||||||
const downloaded = nodes.filter(node => aliases.has(node.model) || aliases.has(node.hf_repo) ||
|
|
||||||
(node.downloaded_models || []).some(item => aliases.has(item.model) || aliases.has(item.hf_repo))).length;
|
|
||||||
const actions = `<button class="small" data-admin-model-load="${esc(model.id)}">load</button> ` +
|
|
||||||
`<button class="small" data-admin-model-release="${esc(model.id)}"${serving ? "" : " disabled"}>release</button>`;
|
|
||||||
return [esc(model.name || model.id), String(serving), String(downloaded), actions];
|
|
||||||
});
|
|
||||||
$("admin-model-placement").innerHTML = rows.length
|
|
||||||
? table(["model", "serving nodes", "downloaded on nodes", "admin action"], rows)
|
|
||||||
: '<div class="empty">no model presets configured</div>';
|
|
||||||
}
|
|
||||||
|
|
||||||
$("admin-model-placement").addEventListener("click", event => {
|
|
||||||
const load = event.target.closest("[data-admin-model-load]");
|
|
||||||
const release = event.target.closest("[data-admin-model-release]");
|
|
||||||
if (load) void requestAdminModelLoad(load.dataset.adminModelLoad);
|
|
||||||
if (release) void releaseAdminModel(release.dataset.adminModelRelease);
|
|
||||||
});
|
|
||||||
|
|
||||||
function chatAuthToken() {
|
function chatAuthToken() {
|
||||||
if (accountApiKeys.length) return accountApiKeys[0];
|
if (accountApiKeys.length) return accountApiKeys[0];
|
||||||
return null;
|
return null;
|
||||||
@@ -2479,17 +2427,14 @@ async function fetchAdminTab() {
|
|||||||
fetchJson("/v1/console"),
|
fetchJson("/v1/console"),
|
||||||
fetchJson("/v1/billing/summary"),
|
fetchJson("/v1/billing/summary"),
|
||||||
fetchJson("/v1/registry/wallets"),
|
fetchJson("/v1/registry/wallets"),
|
||||||
fetchJson("/v1/models"),
|
|
||||||
fetchJson("/v1/network/map"),
|
|
||||||
];
|
];
|
||||||
if (isAdmin) fetches.push(apiCall("/v1/admin/accounts"));
|
if (isAdmin) fetches.push(apiCall("/v1/admin/accounts"));
|
||||||
const results = await Promise.all(fetches);
|
const results = await Promise.all(fetches);
|
||||||
const [raft, consoleData, summary, wallets, models, map, adminResp] = results;
|
const [raft, consoleData, summary, wallets, adminResp] = results;
|
||||||
renderIfChanged("hive", raft, renderHive);
|
renderIfChanged("hive", raft, renderHive);
|
||||||
renderIfChanged("console", consoleData, renderConsole);
|
renderIfChanged("console", consoleData, renderConsole);
|
||||||
renderIfChanged("billing-summary", summary, data => renderBilling(data));
|
renderIfChanged("billing-summary", summary, data => renderBilling(data));
|
||||||
renderIfChanged("fraud", { wallets, summary }, data => renderFraud(data.wallets, data.summary));
|
renderIfChanged("fraud", { wallets, summary }, data => renderFraud(data.wallets, data.summary));
|
||||||
renderIfChanged("admin-model-placement", { models, map }, data => renderAdminModelPlacement(data.models, data.map));
|
|
||||||
if (adminResp && adminResp.ok) {
|
if (adminResp && adminResp.ok) {
|
||||||
renderIfChanged("admin", adminResp.data.accounts || [], accounts => {
|
renderIfChanged("admin", adminResp.data.accounts || [], accounts => {
|
||||||
const rows = accounts.map(a => {
|
const rows = accounts.map(a => {
|
||||||
|
|||||||
@@ -1528,49 +1528,6 @@ def _request_model_load_locked(server: "_TrackerHTTPServer", model_key: str) ->
|
|||||||
return None
|
return None
|
||||||
|
|
||||||
|
|
||||||
def _force_model_load_locked(server: "_TrackerHTTPServer", model_key: str) -> dict | None:
|
|
||||||
"""Replace the fastest ready assignment after an explicit admin eviction."""
|
|
||||||
resolved_name, preset = _resolve_model_preset(server.model_presets, model_key)
|
|
||||||
if preset is None or not preset.get("hf_repo"):
|
|
||||||
return None
|
|
||||||
start, end = _preset_layer_bounds(preset)
|
|
||||||
candidates = [node for node in server.registry.values()
|
|
||||||
if node.status == "ready" and node.pending_new_assignment is None
|
|
||||||
and _has_usable_quantization(node)]
|
|
||||||
if not candidates:
|
|
||||||
return None
|
|
||||||
node = max(candidates, key=lambda item: item.benchmark_tokens_per_sec)
|
|
||||||
shard_end = min(end, start + min(_node_layer_capacity(node, preset), end - start + 1) - 1)
|
|
||||||
if shard_end < start:
|
|
||||||
return None
|
|
||||||
quantization = _node_quantization(node, preset)
|
|
||||||
directive = _load_directive(node, str(preset["hf_repo"]), start, shard_end, quantization)
|
|
||||||
replaced = node.hf_repo or node.model
|
|
||||||
node.model, node.hf_repo = resolved_name, str(preset["hf_repo"])
|
|
||||||
node.shard_start, node.shard_end, node.quantization = start, shard_end, quantization
|
|
||||||
node.managed_assignment, node.pending_new_assignment = True, directive
|
|
||||||
node.pending_directives.append(directive)
|
|
||||||
_tracker_log(server, "warn", "model load forced", node_id=node.node_id,
|
|
||||||
model=resolved_name, replaced_model=replaced, shard=f"{start}-{shard_end}")
|
|
||||||
return {"node_id": node.node_id, "model": resolved_name, "hf_repo": preset["hf_repo"],
|
|
||||||
"shard_start": start, "shard_end": shard_end, "replaced_model": replaced}
|
|
||||||
|
|
||||||
|
|
||||||
def _release_model_locked(server: "_TrackerHTTPServer", model_key: str) -> int:
|
|
||||||
"""Queue DROP_SHARD for every served shard and remove it from routing immediately."""
|
|
||||||
resolved_name, preset = _resolve_model_preset(server.model_presets, model_key)
|
|
||||||
if preset is None:
|
|
||||||
return 0
|
|
||||||
released = 0
|
|
||||||
for node in server.registry.values():
|
|
||||||
if not _node_matches_preset(node, resolved_name, preset) or node.shard_start is None or node.shard_end is None:
|
|
||||||
continue
|
|
||||||
node.pending_directives.append(_drop_directive(node, str(preset.get("hf_repo") or resolved_name), node.shard_start, node.shard_end, node.quantization or "bfloat16"))
|
|
||||||
node.status = "loading"
|
|
||||||
released += 1
|
|
||||||
return released
|
|
||||||
|
|
||||||
|
|
||||||
def _preferred_node_quantization(
|
def _preferred_node_quantization(
|
||||||
node: _NodeEntry,
|
node: _NodeEntry,
|
||||||
preset: dict,
|
preset: dict,
|
||||||
@@ -3086,9 +3043,6 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
|
|||||||
if self.path == "/v1/models/load":
|
if self.path == "/v1/models/load":
|
||||||
self._handle_model_load_request()
|
self._handle_model_load_request()
|
||||||
return
|
return
|
||||||
if self.path == "/v1/models/release":
|
|
||||||
self._handle_model_release_request()
|
|
||||||
return
|
|
||||||
if self.path == "/v1/models/vote":
|
if self.path == "/v1/models/vote":
|
||||||
self._handle_model_coverage_vote()
|
self._handle_model_coverage_vote()
|
||||||
return
|
return
|
||||||
@@ -3216,6 +3170,8 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
|
|||||||
seen_ids: set[str] = set()
|
seen_ids: set[str] = set()
|
||||||
for name, preset in server.model_presets.items():
|
for name, preset in server.model_presets.items():
|
||||||
model_nodes = [node for node in alive if _node_matches_preset(node, name, preset)]
|
model_nodes = [node for node in alive if _node_matches_preset(node, name, preset)]
|
||||||
|
if not model_nodes and not preset.get("recommended"):
|
||||||
|
continue
|
||||||
required_start, required_end = _preset_layer_bounds(preset)
|
required_start, required_end = _preset_layer_bounds(preset)
|
||||||
coverage = _coverage_percentage(
|
coverage = _coverage_percentage(
|
||||||
model_nodes,
|
model_nodes,
|
||||||
@@ -4878,32 +4834,11 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
|
|||||||
with server.lock:
|
with server.lock:
|
||||||
self._purge_expired_nodes()
|
self._purge_expired_nodes()
|
||||||
assignment = _request_model_load_locked(server, model)
|
assignment = _request_model_load_locked(server, model)
|
||||||
if assignment is None and body.get("force") is True:
|
|
||||||
assignment = _force_model_load_locked(server, model)
|
|
||||||
if assignment is None:
|
if assignment is None:
|
||||||
self._send_json(409, {"error": "no ready joined node has an available model slot and sufficient capacity"})
|
self._send_json(409, {"error": "no ready joined node has an available model slot and sufficient capacity"})
|
||||||
return
|
return
|
||||||
self._send_json(202, {"status": "queued", "assignment": assignment})
|
self._send_json(202, {"status": "queued", "assignment": assignment})
|
||||||
|
|
||||||
def _handle_model_release_request(self):
|
|
||||||
server: _TrackerHTTPServer = self.server # type: ignore[assignment]
|
|
||||||
if not self._require_role("admin", "validator"):
|
|
||||||
return
|
|
||||||
body = self._read_json_body()
|
|
||||||
if body is None:
|
|
||||||
return
|
|
||||||
model = body.get("model")
|
|
||||||
if not isinstance(model, str) or not model.strip():
|
|
||||||
self._send_json(400, {"error": "model is required"})
|
|
||||||
return
|
|
||||||
with server.lock:
|
|
||||||
self._purge_expired_nodes()
|
|
||||||
released = _release_model_locked(server, model)
|
|
||||||
if not released:
|
|
||||||
self._send_json(404, {"error": "no served shards found for model"})
|
|
||||||
return
|
|
||||||
self._send_json(202, {"status": "release_queued", "released": released})
|
|
||||||
|
|
||||||
def _handle_model_coverage_vote(self):
|
def _handle_model_coverage_vote(self):
|
||||||
"""Record a rolling wish-list signal for an unavailable precision."""
|
"""Record a rolling wish-list signal for an unavailable precision."""
|
||||||
server: _TrackerHTTPServer = self.server # type: ignore[assignment]
|
server: _TrackerHTTPServer = self.server # type: ignore[assignment]
|
||||||
|
|||||||
@@ -39,9 +39,7 @@ def test_dashboard_served_with_all_panels():
|
|||||||
assert "resolveModelGroup" in html
|
assert "resolveModelGroup" in html
|
||||||
assert "buildModelAliasMap" in html
|
assert "buildModelAliasMap" in html
|
||||||
assert "modelAliasKey(raw)" in html
|
assert "modelAliasKey(raw)" in html
|
||||||
assert "@media (min-width:900px)" in html
|
assert "main { display:grid; grid-template-columns:repeat(auto-fit,minmax(340px,1fr));" in html
|
||||||
assert "grid-template-columns:repeat(4,minmax(0,1fr));" in html
|
|
||||||
assert ".wide { grid-column:span 2; }" in html
|
|
||||||
assert 'onclick="clearConsole()"' in html
|
assert 'onclick="clearConsole()"' in html
|
||||||
assert "let consoleClearedAt = 0;" in html
|
assert "let consoleClearedAt = 0;" in html
|
||||||
finally:
|
finally:
|
||||||
@@ -102,20 +100,6 @@ def test_dashboard_allows_admin_to_request_selected_model_load():
|
|||||||
assert '$("request-model-load").style.display = enabled ? "" : "none"' in html
|
assert '$("request-model-load").style.display = enabled ? "" : "none"' in html
|
||||||
|
|
||||||
|
|
||||||
def test_dashboard_exposes_admin_model_inventory_and_release_controls():
|
|
||||||
"Admin placement controls show the full model inventory and can release capacity."
|
|
||||||
html = _dashboard_html()
|
|
||||||
|
|
||||||
assert 'id="admin-model-placement"' in html
|
|
||||||
assert "renderAdminModelPlacement" in html
|
|
||||||
assert '"/v1/models/release"' in html
|
|
||||||
assert "requestAdminModelLoad" in html
|
|
||||||
assert "releaseAdminModel" in html
|
|
||||||
assert 'data-admin-model-load=' in html
|
|
||||||
assert 'data-admin-model-release=' in html
|
|
||||||
assert "admin-model-placement-status" in html
|
|
||||||
|
|
||||||
|
|
||||||
def test_network_map_includes_node_friendly_name():
|
def test_network_map_includes_node_friendly_name():
|
||||||
"Network map includes node friendly name\n\nTags: dashboard, http"
|
"Network map includes node friendly name\n\nTags: dashboard, http"
|
||||||
tracker = TrackerServer()
|
tracker = TrackerServer()
|
||||||
|
|||||||
@@ -355,39 +355,6 @@ def test_admin_model_load_request_queues_directive_on_joined_node():
|
|||||||
assert heartbeat["directives"][0]["model"] == "Qwen/Qwen2.5-0.5B-Instruct"
|
assert heartbeat["directives"][0]["model"] == "Qwen/Qwen2.5-0.5B-Instruct"
|
||||||
|
|
||||||
|
|
||||||
def test_admin_can_replace_a_served_model_and_release_it():
|
|
||||||
"Forced admin placement replaces a served shard; release queues DROP_SHARD."
|
|
||||||
tracker = TrackerServer(enable_billing=False, validator_service_token="test-admin")
|
|
||||||
port = tracker.start()
|
|
||||||
try:
|
|
||||||
node = _post_json(
|
|
||||||
f"http://127.0.0.1:{port}/v1/nodes/register",
|
|
||||||
{"endpoint": "http://127.0.0.1:9912", "model": "stub-model",
|
|
||||||
"shard_start": 0, "shard_end": 3, "managed_assignment": True,
|
|
||||||
"max_loaded_shards": 1, "memory_mb": 1,
|
|
||||||
"hardware_profile": {"host_id": "full-host"}},
|
|
||||||
)
|
|
||||||
headers = {"Content-Type": "application/json", "Authorization": "Bearer test-admin"}
|
|
||||||
load = urllib.request.Request(
|
|
||||||
f"http://127.0.0.1:{port}/v1/models/load",
|
|
||||||
data=json.dumps({"model": "qwen2.5-0.5b-instruct", "force": True}).encode(),
|
|
||||||
headers=headers, method="POST")
|
|
||||||
with urllib.request.urlopen(load) as response:
|
|
||||||
assert json.loads(response.read())["assignment"]["node_id"] == node["node_id"]
|
|
||||||
heartbeat = _post_json(f"http://127.0.0.1:{port}/v1/nodes/{node['node_id']}/heartbeat", {})
|
|
||||||
assert heartbeat["directives"][0]["action"] == "LOAD_SHARD"
|
|
||||||
release = urllib.request.Request(
|
|
||||||
f"http://127.0.0.1:{port}/v1/models/release",
|
|
||||||
data=json.dumps({"model": "qwen2.5-0.5b-instruct"}).encode(), headers=headers, method="POST")
|
|
||||||
with urllib.request.urlopen(release) as response:
|
|
||||||
assert json.loads(response.read())["released"] == 1
|
|
||||||
heartbeat = _post_json(f"http://127.0.0.1:{port}/v1/nodes/{node['node_id']}/heartbeat", {})
|
|
||||||
finally:
|
|
||||||
tracker.stop()
|
|
||||||
|
|
||||||
assert heartbeat["directives"][0]["action"] == "DROP_SHARD"
|
|
||||||
|
|
||||||
|
|
||||||
def test_endpoint_key_distinguishes_same_port_different_hosts():
|
def test_endpoint_key_distinguishes_same_port_different_hosts():
|
||||||
"Endpoint key distinguishes same port different hosts\n\nTags: http, performance, routing, tracker"
|
"Endpoint key distinguishes same port different hosts\n\nTags: http, performance, routing, tracker"
|
||||||
from meshnet_node.torch_server import _clamp_downstream_hops, _endpoint_key
|
from meshnet_node.torch_server import _clamp_downstream_hops, _endpoint_key
|
||||||
|
|||||||
@@ -1,203 +0,0 @@
|
|||||||
"""Tests for the DGR-001 performance contract metadata."""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import json
|
|
||||||
from unittest.mock import MagicMock, patch
|
|
||||||
|
|
||||||
from meshnet_node.performance_contract import (
|
|
||||||
BENCHMARK_SCHEMA_VERSION,
|
|
||||||
DEFAULT_CONTRACT,
|
|
||||||
SCHEMA_VERSION,
|
|
||||||
main,
|
|
||||||
run_performance_benchmark,
|
|
||||||
run_real_model_endpoint_benchmark,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def test_default_contract_is_architecture_aligned_and_small():
|
|
||||||
"""The baseline stays on DeepSeek2 and uses the smallest DeepSeek-family GGUF.
|
|
||||||
|
|
||||||
Tags: performance, model, gguf
|
|
||||||
"""
|
|
||||||
payload = DEFAULT_CONTRACT.to_dict()
|
|
||||||
|
|
||||||
assert payload["schema_version"] == SCHEMA_VERSION
|
|
||||||
assert payload["story_id"] == "DGR-001"
|
|
||||||
assert payload["model_target"] == {
|
|
||||||
"name": "DeepSeek-V2-Lite-Chat",
|
|
||||||
"architecture": "deepseek2",
|
|
||||||
"safetensors_repo": "deepseek-ai/DeepSeek-V2-Lite-Chat",
|
|
||||||
"safetensors_precision": "bfloat16",
|
|
||||||
"gguf_repo": "second-state/DeepSeek-V2-Lite-Chat-GGUF",
|
|
||||||
"gguf_quant": "Q2_K",
|
|
||||||
"gguf_size_gb": 6.43,
|
|
||||||
"comparison_policy": (
|
|
||||||
"same model/revision, closest practical low-footprint precision pair: "
|
|
||||||
"BF16 safetensors versus Q2_K GGUF"
|
|
||||||
),
|
|
||||||
"rationale": (
|
|
||||||
"Smallest DeepSeek-family benchmark anchor that still points toward "
|
|
||||||
"DeepSeek-V4-Flash; keeps the runtime on the DeepSeek2 path instead "
|
|
||||||
"of falling back to a tiny but architecture-mismatched smoke model."
|
|
||||||
),
|
|
||||||
}
|
|
||||||
assert payload["benchmark_lanes"] == [
|
|
||||||
{
|
|
||||||
"id": "transformers-safetensors-cpu",
|
|
||||||
"runtime": "transformers",
|
|
||||||
"device": "cpu",
|
|
||||||
"recipe": "current safetensors recipe",
|
|
||||||
"concurrency_levels": [1, 4],
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"id": "llama-cpp-gguf-cpu",
|
|
||||||
"runtime": "llama.cpp",
|
|
||||||
"device": "cpu",
|
|
||||||
"recipe": "whole-model GGUF recipe",
|
|
||||||
"concurrency_levels": [1, 4],
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"id": "transformers-safetensors-gpu",
|
|
||||||
"runtime": "transformers",
|
|
||||||
"device": "gpu",
|
|
||||||
"recipe": "current safetensors recipe",
|
|
||||||
"concurrency_levels": [1, 4],
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"id": "llama-cpp-gguf-gpu",
|
|
||||||
"runtime": "llama.cpp",
|
|
||||||
"device": "gpu",
|
|
||||||
"recipe": "whole-model GGUF recipe",
|
|
||||||
"concurrency_levels": [1, 4],
|
|
||||||
},
|
|
||||||
]
|
|
||||||
assert "ttft_ms" in payload["metrics"]
|
|
||||||
assert "output_drift" in payload["metrics"]
|
|
||||||
assert "meaningful speed or fit benefit" in payload["stop_condition"]
|
|
||||||
assert any("mounted drive" in note for note in payload["notes"])
|
|
||||||
|
|
||||||
|
|
||||||
def test_contract_cli_writes_json(tmp_path, capsys):
|
|
||||||
"""The contract can be emitted as a machine-readable artifact.
|
|
||||||
|
|
||||||
Tags: performance, artifact
|
|
||||||
"""
|
|
||||||
output = tmp_path / "performance-contract.json"
|
|
||||||
|
|
||||||
assert main(["--json-out", str(output)]) == 0
|
|
||||||
written = json.loads(output.read_text(encoding="utf-8"))
|
|
||||||
|
|
||||||
assert written == DEFAULT_CONTRACT.to_dict()
|
|
||||||
assert str(output) in capsys.readouterr().out
|
|
||||||
|
|
||||||
|
|
||||||
def test_stub_benchmark_covers_every_lane_concurrency_and_metric():
|
|
||||||
"""The runner exercises all four CPU/GPU lanes with the full metric set.
|
|
||||||
|
|
||||||
Tags: performance, benchmark, gguf
|
|
||||||
"""
|
|
||||||
report = run_performance_benchmark()
|
|
||||||
|
|
||||||
assert report["schema_version"] == BENCHMARK_SCHEMA_VERSION
|
|
||||||
assert report["story_id"] == "DGR-001"
|
|
||||||
assert report["source"] == "stub-backend"
|
|
||||||
assert report["model_target"] == DEFAULT_CONTRACT.model_target.to_dict()
|
|
||||||
assert [lane["id"] for lane in report["lanes"]] == [
|
|
||||||
lane.id for lane in DEFAULT_CONTRACT.benchmark_lanes
|
|
||||||
]
|
|
||||||
for lane in report["lanes"]:
|
|
||||||
assert [result["concurrency"] for result in lane["results"]] == [1, 4]
|
|
||||||
for result in lane["results"]:
|
|
||||||
assert set(result["metrics"]) == set(DEFAULT_CONTRACT.metrics)
|
|
||||||
assert result["metrics"]["failure_count"] == 0
|
|
||||||
assert result["metrics"]["decode_tok_per_sec"] > 0
|
|
||||||
|
|
||||||
|
|
||||||
def test_stub_benchmark_is_deterministic():
|
|
||||||
"""Two runs produce byte-identical reports; no clocks or randomness leak in.
|
|
||||||
|
|
||||||
Tags: performance, benchmark, deterministic
|
|
||||||
"""
|
|
||||||
first = run_performance_benchmark()
|
|
||||||
second = run_performance_benchmark()
|
|
||||||
|
|
||||||
assert first == second
|
|
||||||
assert json.dumps(first, sort_keys=True) == json.dumps(second, sort_keys=True)
|
|
||||||
|
|
||||||
|
|
||||||
def test_stub_benchmark_compares_gguf_against_safetensors_per_device():
|
|
||||||
"""Each device gets a GGUF-vs-safetensors comparison and a stop-condition verdict.
|
|
||||||
|
|
||||||
Tags: performance, benchmark, gguf
|
|
||||||
"""
|
|
||||||
report = run_performance_benchmark()
|
|
||||||
|
|
||||||
assert set(report["comparisons"]) == {"cpu", "gpu"}
|
|
||||||
cpu, gpu = report["comparisons"]["cpu"], report["comparisons"]["gpu"]
|
|
||||||
assert cpu["safetensors_lane"] == "transformers-safetensors-cpu"
|
|
||||||
assert cpu["gguf_lane"] == "llama-cpp-gguf-cpu"
|
|
||||||
assert cpu["memory_metric"] == "rss_bytes"
|
|
||||||
assert gpu["safetensors_lane"] == "transformers-safetensors-gpu"
|
|
||||||
assert gpu["gguf_lane"] == "llama-cpp-gguf-gpu"
|
|
||||||
assert gpu["memory_metric"] == "vram_bytes"
|
|
||||||
for comparison in (cpu, gpu):
|
|
||||||
assert comparison["decode_speedup"] > 1.0
|
|
||||||
assert comparison["artifact_bytes_ratio"] < 0.5
|
|
||||||
assert comparison["memory_bytes_ratio"] < 1.0
|
|
||||||
assert comparison["output_drift"] == 0.0
|
|
||||||
assert comparison["gguf_benefit"] is True
|
|
||||||
assert report["stop_condition"]["gguf_benefit"] is True
|
|
||||||
assert report["stop_condition"]["triggered"] is False
|
|
||||||
assert report["stop_condition"]["text"] == DEFAULT_CONTRACT.stop_condition
|
|
||||||
|
|
||||||
|
|
||||||
def test_contract_cli_writes_benchmark_report(tmp_path, capsys):
|
|
||||||
"""--benchmark-out emits the stub benchmark report next to the contract.
|
|
||||||
|
|
||||||
Tags: performance, benchmark, artifact
|
|
||||||
"""
|
|
||||||
contract_out = tmp_path / "performance-contract.json"
|
|
||||||
benchmark_out = tmp_path / "artifacts" / "stub-benchmark-report.json"
|
|
||||||
|
|
||||||
assert main(["--json-out", str(contract_out), "--benchmark-out", str(benchmark_out)]) == 0
|
|
||||||
report = json.loads(benchmark_out.read_text(encoding="utf-8"))
|
|
||||||
|
|
||||||
assert report == run_performance_benchmark()
|
|
||||||
output = capsys.readouterr().out
|
|
||||||
assert str(contract_out) in output
|
|
||||||
assert str(benchmark_out) in output
|
|
||||||
|
|
||||||
|
|
||||||
def test_real_model_endpoint_benchmark_uses_lane_specific_endpoints_and_shared_schema():
|
|
||||||
"""The live client path fans out to one endpoint per CPU/GPU lane.
|
|
||||||
|
|
||||||
Tags: performance, benchmark, live
|
|
||||||
"""
|
|
||||||
response = MagicMock()
|
|
||||||
response.read.return_value = json.dumps({"choices": [{"message": {"content": "mesh activation"}}]}).encode()
|
|
||||||
response.headers.get.return_value = "lane-session"
|
|
||||||
response.__enter__.return_value = response
|
|
||||||
|
|
||||||
endpoints = {
|
|
||||||
"transformers-safetensors-cpu": "http://cpu-safetensors",
|
|
||||||
"llama-cpp-gguf-cpu": "http://cpu-gguf",
|
|
||||||
"transformers-safetensors-gpu": "http://gpu-safetensors",
|
|
||||||
"llama-cpp-gguf-gpu": "http://gpu-gguf",
|
|
||||||
}
|
|
||||||
|
|
||||||
with patch("meshnet_node.performance_contract.urllib.request.urlopen", return_value=response) as urlopen:
|
|
||||||
report = run_real_model_endpoint_benchmark(endpoints=endpoints, model="deepseek-ai/DeepSeek-V2-Lite-Chat")
|
|
||||||
|
|
||||||
assert report["source"] == "real-model-endpoints"
|
|
||||||
assert report["model_target"] == DEFAULT_CONTRACT.model_target.to_dict()
|
|
||||||
assert set(report["comparisons"]) == {"cpu", "gpu"}
|
|
||||||
assert urlopen.call_count == len(endpoints)
|
|
||||||
called_urls = [call.args[0].full_url for call in urlopen.call_args_list]
|
|
||||||
assert called_urls == [f"{url}/v1/chat/completions" for url in endpoints.values()]
|
|
||||||
for lane in report["lanes"]:
|
|
||||||
assert lane["results"][0]["metrics"]["decode_tok_per_sec"] > 0
|
|
||||||
assert lane["results"][0]["metrics"]["ttft_ms"] > 0
|
|
||||||
assert lane["output_tokens"] == ["mesh", "activation"]
|
|
||||||
assert report["comparisons"]["cpu"]["gguf_lane"] == "llama-cpp-gguf-cpu"
|
|
||||||
assert report["comparisons"]["gpu"]["gguf_lane"] == "llama-cpp-gguf-gpu"
|
|
||||||
@@ -32,18 +32,12 @@ def test_matrix_reports_direct_relay_prefill_decode_and_machine_readable_metrics
|
|||||||
assert {"p50_latency_ms", "p95_latency_ms", "payload_bytes", "compression_ratio",
|
assert {"p50_latency_ms", "p95_latency_ms", "payload_bytes", "compression_ratio",
|
||||||
"connection_attempts", "p95_queue_wait_ms"} <= set(run["phases"]["decode"])
|
"connection_attempts", "p95_queue_wait_ms"} <= set(run["phases"]["decode"])
|
||||||
sample = run["samples"][0]
|
sample = run["samples"][0]
|
||||||
assert sample["model_ms"] > 0
|
|
||||||
assert sample["encode_ms"] > 0
|
|
||||||
assert sample["activation_decode_ms"] > 0
|
|
||||||
assert sample["framing_ms"] > 0
|
assert sample["framing_ms"] > 0
|
||||||
assert sample["metadata_ms"] > 0
|
assert sample["metadata_ms"] > 0
|
||||||
assert sample["copy_allocation_ms"] > 0
|
assert sample["copy_allocation_ms"] > 0
|
||||||
assert sample["copy_allocation_bytes"] >= sample["payload_bytes"]
|
assert sample["copy_allocation_bytes"] >= sample["payload_bytes"]
|
||||||
assert sample["local_http_forwarding_ms"] > 0
|
|
||||||
assert len(run["samples"]) == 1 + len(run["output_tokens"])
|
assert len(run["samples"]) == 1 + len(run["output_tokens"])
|
||||||
assert {"tokens_per_sec", "bytes_per_token", "compression_cpu_ms", "peak_buffered_bytes",
|
assert {"tokens_per_sec", "bytes_per_token", "compression_cpu_ms", "peak_buffered_bytes"} <= set(run["phases"]["decode"])
|
||||||
"model_execution_ms", "activation_encoding_ms", "activation_decoding_ms",
|
|
||||||
"local_http_forwarding_ms"} <= set(run["phases"]["decode"])
|
|
||||||
|
|
||||||
|
|
||||||
def test_cached_sessions_reuse_one_connection_and_preserve_stub_tokens():
|
def test_cached_sessions_reuse_one_connection_and_preserve_stub_tokens():
|
||||||
@@ -80,10 +74,7 @@ def test_cli_writes_json_artifact_and_human_summary(tmp_path, capsys):
|
|||||||
report = json.loads(output.read_text())
|
report = json.loads(output.read_text())
|
||||||
assert report["schema_version"] == 1
|
assert report["schema_version"] == 1
|
||||||
assert "Route Session benchmark" in capsys.readouterr().out
|
assert "Route Session benchmark" in capsys.readouterr().out
|
||||||
summary = format_summary(report)
|
assert "relay" in format_summary(report)
|
||||||
assert "relay" in summary
|
|
||||||
assert "model/encode/decode" in summary
|
|
||||||
assert "HTTP" in summary
|
|
||||||
|
|
||||||
|
|
||||||
def test_performance_gate_checks_comparison_identity_session_and_cleanup():
|
def test_performance_gate_checks_comparison_identity_session_and_cleanup():
|
||||||
|
|||||||
Reference in New Issue
Block a user