[verified] feat: complete Ralph task workstreams

docs/dev/dashboard-test-runner.md

@@ -0,0 +1,93 @@
+# Dashboard test runner (operator workflow)
+
+The tracker dashboard **Testing** tab can discover repository pytest targets and
+run one collected test or approved suite at a time with live logs. The feature is
+**disabled by default** and **admin-only**.
+
+## Enable intentionally
+
+Start the tracker with the test runner API enabled using either:
+
+- VS Code launch configuration **`Tracker: local + dashboard test runner (8080)`**
+  (uses the project tracker runtime at `.venv-rocm/bin/python` and
+  `meshnet_tracker.cli`), or
+- CLI flag **`--enable-test-runner`**, or
+- Environment variable **`MESHNET_ENABLE_TEST_RUNNER=1`**
+
+The default **`Tracker: local (8080)`** launch configuration does **not** enable
+the test runner.
+
+Verify the flag is available:
+
+```bash
+uv run python -m meshnet_tracker.cli --help | grep enable-test-runner
+```
+
+Log in to the dashboard as an admin account, open **Testing**, and use **Refresh
+collection** before running targets.
+
+## Child pytest interpreter
+
+The runner spawns pytest as a subprocess without a shell. It uses
+**`MESHNET_PYTHON`** when set (typically via `.env.<hostname>` loaded by
+`meshnet_tracker.cli`); otherwise it falls back to the tracker process
+interpreter. Point this at the venv that has dev extras and package dependencies
+installed (see [test-env.md](test-env.md)).
+
+## Default safe suites
+
+These named suites are always available when the test runner is enabled and the
+files exist in the checkout:
+
+| Suite ID | Paths | Notes |
+| --- | --- | --- |
+| `suite:smoke` | `tests/test_smoke.py` | Fast sanity checks |
+| `suite:dashboard` | `tests/test_dashboard.py` | Dashboard HTML/API regressions |
+| `suite:routing` | `tests/test_tracker_routing.py`, `tests/test_dynamic_routing.py` | Tracker routing logic |
+
+Collection also lists individual pytest node IDs (excluding real-inference
+modules by default). You can run `suite:all` or `tag:<name>` after collection.
+
+These suites use mocks/stubs or in-process fakes. They do **not** require live
+GPU nodes, paid API credits, or a running mesh beyond the tracker itself.
+
+## Real-inference suite (explicitly gated)
+
+Modules matching `tests/test_real_*.py` are **never collected** and **never**
+included in default suites unless you set:
+
+```bash
+export MESHNET_ENABLE_REAL_INFERENCE_TESTS=1
+```
+
+With that gate, an additional suite appears:
+
+| Suite ID | Paths |
+| --- | --- |
+| `suite:real-inference` | `tests/test_real_distributed_inference.py`, `tests/test_real_model_backend.py` |
+
+### Implications
+
+- **`tests/test_real_distributed_inference.py`** — integration test against a
+  **live tracker and registered model shards**. Requires env vars such as
+  `MESHNET_REAL_INFERENCE_URL`, `MESHNET_REAL_INFERENCE_API_KEY`,
+  `MESHNET_REAL_INFERENCE_MODEL`, and `MESHNET_REAL_INFERENCE_ROUTE`. Uses real
+  chat completions and **consumes caller billing / API credit** on the target
+  tracker.
+- **`tests/test_real_model_backend.py`** — loads real PyTorch model code paths;
+  needs **`torch`**, **`transformers`**, and related optional deps, and can
+  require **substantial GPU/CPU RAM** depending on which cases run.
+
+Do not enable `MESHNET_ENABLE_REAL_INFERENCE_TESTS=1` on shared or production
+trackers unless you intend to spend credits and tie up hardware.
+
+## Safety summary
+
+| Control | Purpose |
+| --- | --- |
+| Disabled by default | No test subprocess unless operator opts in |
+| Admin-only API/UI | Non-admins cannot start runs |
+| Fixed suite list | API cannot pass arbitrary shell commands |
+| No `shell=True` | pytest argv is fixed server-side |
+| One run at a time | Concurrent starts are rejected |
+| Real-inference env gate | Live inference tests stay out of default collection |