Dashboard test runner (operator workflow)

The tracker dashboard Testing tab can discover repository pytest targets and run one collected test or approved suite at a time with live logs. The feature is disabled by default and admin-only.

Enable intentionally

Start the tracker with the test runner API enabled using either:

VS Code launch configuration Tracker: local + dashboard test runner (8080) (uses the project tracker runtime at .venv-rocm/bin/python and meshnet_tracker.cli), or
CLI flag --enable-test-runner, or
Environment variable MESHNET_ENABLE_TEST_RUNNER=1

The default Tracker: local (8080) launch configuration does not enable the test runner.

Verify the flag is available:

uv run python -m meshnet_tracker.cli --help | grep enable-test-runner

Child pytest interpreter

The runner spawns pytest as a subprocess without a shell. It uses MESHNET_PYTHON when set (typically via .env.<hostname> loaded by meshnet_tracker.cli); otherwise it falls back to the tracker process interpreter. Point this at the venv that has dev extras and package dependencies installed (see test-env.md).

Default safe suites

These named suites are always available when the test runner is enabled and the files exist in the checkout:

Suite ID	Paths	Notes
`suite:smoke`	`tests/test_smoke.py`	Fast sanity checks
`suite:dashboard`	`tests/test_dashboard.py`	Dashboard HTML/API regressions
`suite:routing`	`tests/test_tracker_routing.py`, `tests/test_dynamic_routing.py`	Tracker routing logic

Collection also lists individual pytest node IDs (excluding real-inference modules by default). You can run suite:all or tag:<name> after collection.

These suites use mocks/stubs or in-process fakes. They do not require live GPU nodes, paid API credits, or a running mesh beyond the tracker itself.

Real-inference suite (explicitly gated)

Modules matching tests/test_real_*.py are never collected and never included in default suites unless you set:

export MESHNET_ENABLE_REAL_INFERENCE_TESTS=1

With that gate, an additional suite appears:

Suite ID	Paths
`suite:real-inference`	`tests/test_real_distributed_inference.py`, `tests/test_real_model_backend.py`

Implications

tests/test_real_distributed_inference.py — integration test against a live tracker and registered model shards. Requires env vars such as MESHNET_REAL_INFERENCE_URL, MESHNET_REAL_INFERENCE_API_KEY, MESHNET_REAL_INFERENCE_MODEL, and MESHNET_REAL_INFERENCE_ROUTE. Uses real chat completions and consumes caller billing / API credit on the target tracker.
tests/test_real_model_backend.py — loads real PyTorch model code paths; needs torch, transformers, and related optional deps, and can require substantial GPU/CPU RAM depending on which cases run.

Do not enable MESHNET_ENABLE_REAL_INFERENCE_TESTS=1 on shared or production trackers unless you intend to spend credits and tie up hardware.

Safety summary

Control	Purpose
Disabled by default	No test subprocess unless operator opts in
Admin-only API/UI	Non-admins cannot start runs
Fixed suite list	API cannot pass arbitrary shell commands
No `shell=True`	pytest argv is fixed server-side
One run at a time	Concurrent starts are rejected
Real-inference env gate	Live inference tests stay out of default collection

3.6 KiB Raw Blame History