popov/neuron-tai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: wire live benchmark CLI endpoints

Browse Source
This commit is contained in:
Dobromir Popov
2026-07-15 10:34:20 +03:00
parent a508768e8a
commit c035bad5b7
3 changed files with 147 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files

21
.scratch/distributed-gguf-runtime/evidence/DGR-001/README.md
View File

@@ -35,15 +35,30 @@ The report includes per-device comparisons for:
and records the memory metric (`rss_bytes` on CPU, `vram_bytes` on GPU), decode speedup, artifact ratio, and output drift.
## Live endpoint CLI wiring
The contract CLI can now drive the live endpoint runner. Passing one `--live-endpoint LANE_ID=URL` mapping per contract lane (plus `--live-benchmark-out`) invokes `run_real_model_endpoint_benchmark` against already-running OpenAI-compatible servers and writes the report using the same schema as the stub:
```bash
PYTHONPATH=packages/node python -m meshnet_node.performance_contract \
--live-endpoint transformers-safetensors-cpu=http://127.0.0.1:8001 \
--live-endpoint llama-cpp-gguf-cpu=http://127.0.0.1:8002 \
--live-endpoint transformers-safetensors-gpu=http://127.0.0.1:8003 \
--live-endpoint llama-cpp-gguf-gpu=http://127.0.0.1:8004 \
--live-benchmark-out .scratch/distributed-gguf-runtime/evidence/DGR-001/live-benchmark-report.json
```
`--live-model` overrides the model name sent in requests (defaults to the contract's safetensors repo). Without any `--live-endpoint` flags the CLI behaves exactly as before: it writes the contract JSON and, with `--benchmark-out`, the deterministic stub report.
## Exact commands and real results
### Targeted tests
```bash
pytest -q tests/test_performance_contract.py tests/test_route_session_benchmark.py
PYTHONPATH=packages/node pytest -q tests/test_performance_contract.py tests/test_route_session_benchmark.py
```
Result: `9 passed in 0.14s`
Result: `19 passed in 0.11s`
### Contract artifact generation
@@ -64,7 +79,7 @@ Result: passed
## Limitations
- This slice still uses a deterministic stub backend for the core comparison matrix.
- It now also includes a live endpoint runner that can fan out one OpenAI-compatible request per lane when the caller provides endpoints.
- It now also includes a live endpoint runner, reachable from the CLI via `--live-endpoint`/`--live-benchmark-out`, that fans out one OpenAI-compatible request per lane when the caller provides endpoints; the CLI does not start those servers.
- It does **not** download or run a real model from within the repo.
- Real safetensors vs GGUF execution, TTFT/prefill/decode measurements, RSS/VRAM capture, and output-drift comparison are still to be implemented against the contract.