94 lines
3.6 KiB
Markdown
94 lines
3.6 KiB
Markdown
# Dashboard test runner (operator workflow)
|
|
|
|
The tracker dashboard **Testing** tab can discover repository pytest targets and
|
|
run one collected test or approved suite at a time with live logs. The feature is
|
|
**disabled by default** and **admin-only**.
|
|
|
|
## Enable intentionally
|
|
|
|
Start the tracker with the test runner API enabled using either:
|
|
|
|
- VS Code launch configuration **`Tracker: local + dashboard test runner (8080)`**
|
|
(uses the project tracker runtime at `.venv-rocm/bin/python` and
|
|
`meshnet_tracker.cli`), or
|
|
- CLI flag **`--enable-test-runner`**, or
|
|
- Environment variable **`MESHNET_ENABLE_TEST_RUNNER=1`**
|
|
|
|
The default **`Tracker: local (8080)`** launch configuration does **not** enable
|
|
the test runner.
|
|
|
|
Verify the flag is available:
|
|
|
|
```bash
|
|
uv run python -m meshnet_tracker.cli --help | grep enable-test-runner
|
|
```
|
|
|
|
Log in to the dashboard as an admin account, open **Testing**, and use **Refresh
|
|
collection** before running targets.
|
|
|
|
## Child pytest interpreter
|
|
|
|
The runner spawns pytest as a subprocess without a shell. It uses
|
|
**`MESHNET_PYTHON`** when set (typically via `.env.<hostname>` loaded by
|
|
`meshnet_tracker.cli`); otherwise it falls back to the tracker process
|
|
interpreter. Point this at the venv that has dev extras and package dependencies
|
|
installed (see [test-env.md](test-env.md)).
|
|
|
|
## Default safe suites
|
|
|
|
These named suites are always available when the test runner is enabled and the
|
|
files exist in the checkout:
|
|
|
|
| Suite ID | Paths | Notes |
|
|
| --- | --- | --- |
|
|
| `suite:smoke` | `tests/test_smoke.py` | Fast sanity checks |
|
|
| `suite:dashboard` | `tests/test_dashboard.py` | Dashboard HTML/API regressions |
|
|
| `suite:routing` | `tests/test_tracker_routing.py`, `tests/test_dynamic_routing.py` | Tracker routing logic |
|
|
|
|
Collection also lists individual pytest node IDs (excluding real-inference
|
|
modules by default). You can run `suite:all` or `tag:<name>` after collection.
|
|
|
|
These suites use mocks/stubs or in-process fakes. They do **not** require live
|
|
GPU nodes, paid API credits, or a running mesh beyond the tracker itself.
|
|
|
|
## Real-inference suite (explicitly gated)
|
|
|
|
Modules matching `tests/test_real_*.py` are **never collected** and **never**
|
|
included in default suites unless you set:
|
|
|
|
```bash
|
|
export MESHNET_ENABLE_REAL_INFERENCE_TESTS=1
|
|
```
|
|
|
|
With that gate, an additional suite appears:
|
|
|
|
| Suite ID | Paths |
|
|
| --- | --- |
|
|
| `suite:real-inference` | `tests/test_real_distributed_inference.py`, `tests/test_real_model_backend.py` |
|
|
|
|
### Implications
|
|
|
|
- **`tests/test_real_distributed_inference.py`** — integration test against a
|
|
**live tracker and registered model shards**. Requires env vars such as
|
|
`MESHNET_REAL_INFERENCE_URL`, `MESHNET_REAL_INFERENCE_API_KEY`,
|
|
`MESHNET_REAL_INFERENCE_MODEL`, and `MESHNET_REAL_INFERENCE_ROUTE`. Uses real
|
|
chat completions and **consumes caller billing / API credit** on the target
|
|
tracker.
|
|
- **`tests/test_real_model_backend.py`** — loads real PyTorch model code paths;
|
|
needs **`torch`**, **`transformers`**, and related optional deps, and can
|
|
require **substantial GPU/CPU RAM** depending on which cases run.
|
|
|
|
Do not enable `MESHNET_ENABLE_REAL_INFERENCE_TESTS=1` on shared or production
|
|
trackers unless you intend to spend credits and tie up hardware.
|
|
|
|
## Safety summary
|
|
|
|
| Control | Purpose |
|
|
| --- | --- |
|
|
| Disabled by default | No test subprocess unless operator opts in |
|
|
| Admin-only API/UI | Non-admins cannot start runs |
|
|
| Fixed suite list | API cannot pass arbitrary shell commands |
|
|
| No `shell=True` | pytest argv is fixed server-side |
|
|
| One run at a time | Concurrent starts are rejected |
|
|
| Real-inference env gate | Live inference tests stay out of default collection |
|