[verified] feat: complete Ralph task workstreams

docs/adr/0023-model-agnostic-node-capability-admission.md

@@ -18,6 +18,31 @@ This is incompatible with a consumer-grade node experience. A Node must never ad
 - P0 carries the version of a local recipe manifest. New executable recipes arrive only through signed Node releases in a future feature. P0 does not download executable recipes, dynamically install dependencies, install OS packages/drivers, or implement an updater.
 - A future Tracker-provided Model Artifact Manifest may be signed data only; it cannot instruct a Node to execute arbitrary code.
 
+## Tracker admission and the compatibility policy for older Nodes
+
+A Node ships its capability report with `POST /v1/nodes/register` (`capability_report`), alongside an independent declaration of the recipe it serves with (`recipe_id`, `recipe_version`). The Tracker does not re-run the forward. It decides whether the presented proof *covers what the Node advertises*, records the verdict as a sanitized state, and routes accordingly.
+
+Registration always succeeds — a Node with a bad proof is registered and visible, it is simply not routable. "Registered but dark" is a state an operator must be able to see and diagnose, so the verdict is returned in the registration response, logged, and exposed per node on `GET /v1/network/map` under `capability` (state, detail, proven model/shard/recipe/backend/device, timestamps). The detail is credential-redacted and clipped; a raw exception or token never reaches an operator view.
+
+Verdicts: `admitted`, `absent`, `invalid`, `failed`, `stale`, `model-mismatch`, `shard-mismatch`, `recipe-mismatch`, `catalogue-incompatible`. Only `admitted` is proof. The proof does not travel with a reassignment: if the Tracker later moves a Node to a range it never validated, the Node is re-verdicted `shard-mismatch` and stops routing until it re-registers with a proof for the range it now advertises.
+
+Freshness is checked when the proof is *presented*, not continuously — a long-lived Node's proof does not expire out from under it while it is heartbeating; liveness is already carried by heartbeat expiry.
+
+**Compatibility policy** (`--capability-policy`, `$MESHNET_TRACKER_CAPABILITY_POLICY`):
+
+- **`compat` (default, transitional)** — a Node that presents *no* report at all still routes, preserving pre-capability Node behaviour during the fleet rollout. Every other verdict is refused. Presenting a broken, failed, stale or mismatched proof is a stronger negative signal than presenting none, so it is never grandfathered.
+- **`enforce`** — only `admitted` routes. Absent proof is not routable, and no paid route can rest on an unproven Node.
+
+`compat` is a deprecating default: it exists to let a mixed fleet upgrade without an outage, and `enforce` becomes the default once the deployed Nodes emit reports. The policy is a single explicit switch, checked in one gate (`_admitted_nodes`) that every route path — proxy head selection, `/v1/route`, `/v1/routes`, and bandit route enumeration — passes through. The gate only ever *removes* candidates; coverage-first selection and throughput-weighted preference among the survivors are untouched, and nothing in a report can raise a Node's routing weight (performance stays measured, per ADR-0013/ADR-0021).
+
+The Tracker also refuses a report whose recipe catalogue predates `MIN_CATALOGUE_VERSION`: recipe ids from an older catalogue may since have been redefined, so the proof cannot be matched to a name reliably.
+
+## Hardware claims are evidence, not a support matrix
+
+Operator docs must distinguish three states and never collapse them: **detected hardware** (a GPU, a torch build, or an optional package is present — proves nothing), **validated recipe** (this machine ran a real forward for this model/shard/recipe/device, and there is a capability report to show for it), and **routable Node** (the Tracker admitted that proof for what the Node advertises). Each is strictly stronger than the last.
+
+Consequently no doc promises that a model, vendor, or optional kernel works universally. A concrete model appears only as a clearly-labelled example or as environment-supplied test configuration. Hardware support is claimed per *certified lane*, where a lane is certified by an opt-in `integration` doctor run whose model identity comes from CI configuration and whose retained evidence is the capability report — see `docs/dev/certified-hardware-lanes.md`. A lane certifies hardware, not models: a new Model Artifact is unproven there until doctor has run it.
+
 ## Consequences
 
 - First startup has a bounded validation cost before registration, but failures occur before traffic rather than under a paid request.

docs/dev/certified-hardware-lanes.md

@@ -0,0 +1,105 @@
+# Certified hardware lanes
+
+A **certified hardware lane** is one (hardware, torch build, OS) configuration on which we
+have *evidence* that nodes can execute real work — a self-hosted release runner that runs
+the opt-in integration doctor test and keeps the resulting capability report as the
+artifact. This document is the contract that lane runners and the release check must meet.
+
+Certification is per *lane*, and evidence is per *(lane, model, shard, recipe)*. Nothing
+here promises that an arbitrary model runs on a certified lane; it promises that the lane
+itself is real, and that the models we ran on it produced passing capability reports on a
+named date. See [ADR-0023](../adr/0023-model-agnostic-node-capability-admission.md) for the
+admission model this rests on.
+
+## What certification is not
+
+- **Not a model support matrix.** A lane certifies hardware, not models. A new model is
+  unproven on a certified lane until doctor has run it there.
+- **Not an optional-kernel promise.** An optional accelerator package importing cleanly on
+  a lane says nothing about another lane, another GPU architecture, or another model's
+  recipe. Only a passing report for that exact combination is evidence.
+- **Not a promise the node will install anything.** Lane runners are provisioned *ahead of
+  time*, by hand or by their own image build. The node under test never downloads an
+  executable recipe, installs a Python or OS package, or touches a driver. Signed node
+  updates are a deliberate follow-up feature and are out of scope here — nothing in this
+  lane contract may depend on dynamic executable-dependency installation.
+
+## The lane check
+
+Every lane runs the same environment-configured integration test. It is
+`tests/test_node_doctor.py::test_doctor_smoke_runs_a_real_forward_on_a_real_model`, marked
+`@pytest.mark.integration` and skipped unless `MESHNET_DOCTOR_MODEL` is set. It carries no
+default model: **model identity comes from the CI configuration**, so no vendor or model
+assumption can leak into the suite.
+
+```bash
+MESHNET_DOCTOR_MODEL="$LANE_MODEL" \
+MESHNET_DOCTOR_QUANTIZATION=bfloat16 \
+MESHNET_DOWNLOAD_DIR=/srv/models \
+  .venv/bin/pytest -m integration tests/test_node_doctor.py -v
+```
+
+| Variable | Required | Meaning |
+|----------|----------|---------|
+| `MESHNET_DOCTOR_MODEL` | yes — the test skips without it | Model artifact identity for this lane's run. No default. |
+| `MESHNET_DOCTOR_SHARD_START` | no (default `0`) | First layer of the shard to prove. |
+| `MESHNET_DOCTOR_SHARD_END` | no (default: whole model) | Last layer, **inclusive**. |
+| `MESHNET_DOCTOR_QUANTIZATION` | no (default `auto`) | Quantization to prove. |
+| `MESHNET_DOCTOR_CPU` | no | `1` forces CPU — use to certify a CPU lane on GPU hardware. |
+| `MESHNET_DOWNLOAD_DIR` | no | Where the artifact is cached on the runner. |
+
+The test asserts that doctor passed, that the report is `passed` with the model id it was
+asked for, that a forward actually took time (`duration_ms > 0`), and that the report
+round-trips through `CapabilityReport.from_json`. A lane where this fails is not certified,
+regardless of what `rocminfo`, `nvidia-smi` or `torch.cuda.is_available()` say.
+
+Lanes that must cover more than the default recipe run doctor directly with
+`--all-recipes`, which validates every recipe for the selection and writes a report per
+recipe:
+
+```bash
+.venv/bin/meshnet-node doctor --model "$LANE_MODEL" --all-recipes --report "$ARTIFACTS/capability.json"
+```
+
+## Expected evidence
+
+A lane run is only certified if it produces, and the release check retains:
+
+1. **The capability report(s)** — `capability.json` from the run, archived as a build
+   artifact. This is the evidence; a green checkmark without it is not.
+2. **Backend identity from the report**: device, torch/backend version, and the recipe id
+   and version that passed. This is what makes "certified on ROCm gfx1151" a checkable
+   claim rather than a slogan.
+3. **The model artifact identity and shard range** the report covers — recorded as run
+   configuration, since it came from the environment.
+4. **Failures kept, not discarded.** Doctor writes a report for a failed recipe too, and a
+   failing lane must archive it. A red lane with a `forward-failed` report is a more useful
+   release signal than a lane that was quietly skipped.
+
+A lane that *skips* (because `MESHNET_DOCTOR_MODEL` was unset) must be reported as skipped,
+never as passed. A silent skip is how an uncertified lane gets mistaken for a certified one.
+
+## Release check
+
+The default CI lane runs the normal suite and never needs a GPU, a download, or torch:
+
+```bash
+.venv/bin/pytest -m "not integration"
+```
+
+The release check additionally requires every declared certified lane to have run the
+integration doctor test green, against the model(s) configured for that lane, on the
+release commit. Adding a lane means standing up a runner and adding its configuration; it
+does not mean adding a model default to the test suite.
+
+## Adding a lane
+
+1. Provision the runner: OS, driver, and the torch build for that hardware (see the
+   platform sections in `QUICKSTART.md`). Install any optional accelerator packages the
+   lane is meant to certify.
+2. Configure `MESHNET_DOCTOR_MODEL` (and shard/quantization if the lane certifies a
+   partial shard) in the runner's CI configuration.
+3. Run the lane check. Archive the capability report.
+4. Record the lane with the evidence it produced: hardware, torch build, model, shard,
+   recipe, device, and the date. That record — not the hardware's spec sheet — is the
+   support claim.

docs/dev/dashboard-test-runner.md

@@ -0,0 +1,93 @@
+# Dashboard test runner (operator workflow)
+
+The tracker dashboard **Testing** tab can discover repository pytest targets and
+run one collected test or approved suite at a time with live logs. The feature is
+**disabled by default** and **admin-only**.
+
+## Enable intentionally
+
+Start the tracker with the test runner API enabled using either:
+
+- VS Code launch configuration **`Tracker: local + dashboard test runner (8080)`**
+  (uses the project tracker runtime at `.venv-rocm/bin/python` and
+  `meshnet_tracker.cli`), or
+- CLI flag **`--enable-test-runner`**, or
+- Environment variable **`MESHNET_ENABLE_TEST_RUNNER=1`**
+
+The default **`Tracker: local (8080)`** launch configuration does **not** enable
+the test runner.
+
+Verify the flag is available:
+
+```bash
+uv run python -m meshnet_tracker.cli --help | grep enable-test-runner
+```
+
+Log in to the dashboard as an admin account, open **Testing**, and use **Refresh
+collection** before running targets.
+
+## Child pytest interpreter
+
+The runner spawns pytest as a subprocess without a shell. It uses
+**`MESHNET_PYTHON`** when set (typically via `.env.<hostname>` loaded by
+`meshnet_tracker.cli`); otherwise it falls back to the tracker process
+interpreter. Point this at the venv that has dev extras and package dependencies
+installed (see [test-env.md](test-env.md)).
+
+## Default safe suites
+
+These named suites are always available when the test runner is enabled and the
+files exist in the checkout:
+
+| Suite ID | Paths | Notes |
+| --- | --- | --- |
+| `suite:smoke` | `tests/test_smoke.py` | Fast sanity checks |
+| `suite:dashboard` | `tests/test_dashboard.py` | Dashboard HTML/API regressions |
+| `suite:routing` | `tests/test_tracker_routing.py`, `tests/test_dynamic_routing.py` | Tracker routing logic |
+
+Collection also lists individual pytest node IDs (excluding real-inference
+modules by default). You can run `suite:all` or `tag:<name>` after collection.
+
+These suites use mocks/stubs or in-process fakes. They do **not** require live
+GPU nodes, paid API credits, or a running mesh beyond the tracker itself.
+
+## Real-inference suite (explicitly gated)
+
+Modules matching `tests/test_real_*.py` are **never collected** and **never**
+included in default suites unless you set:
+
+```bash
+export MESHNET_ENABLE_REAL_INFERENCE_TESTS=1
+```
+
+With that gate, an additional suite appears:
+
+| Suite ID | Paths |
+| --- | --- |
+| `suite:real-inference` | `tests/test_real_distributed_inference.py`, `tests/test_real_model_backend.py` |
+
+### Implications
+
+- **`tests/test_real_distributed_inference.py`** — integration test against a
+  **live tracker and registered model shards**. Requires env vars such as
+  `MESHNET_REAL_INFERENCE_URL`, `MESHNET_REAL_INFERENCE_API_KEY`,
+  `MESHNET_REAL_INFERENCE_MODEL`, and `MESHNET_REAL_INFERENCE_ROUTE`. Uses real
+  chat completions and **consumes caller billing / API credit** on the target
+  tracker.
+- **`tests/test_real_model_backend.py`** — loads real PyTorch model code paths;
+  needs **`torch`**, **`transformers`**, and related optional deps, and can
+  require **substantial GPU/CPU RAM** depending on which cases run.
+
+Do not enable `MESHNET_ENABLE_REAL_INFERENCE_TESTS=1` on shared or production
+trackers unless you intend to spend credits and tie up hardware.
+
+## Safety summary
+
+| Control | Purpose |
+| --- | --- |
+| Disabled by default | No test subprocess unless operator opts in |
+| Admin-only API/UI | Non-admins cannot start runs |
+| Fixed suite list | API cannot pass arbitrary shell commands |
+| No `shell=True` | pytest argv is fixed server-side |
+| One run at a time | Concurrent starts are rejected |
+| Real-inference env gate | Live inference tests stay out of default collection |

docs/dev/test-env.md

@@ -38,6 +38,12 @@ even without installing `packages/node`.
 .venv/bin/python -m pytest
 ```
 
+## Dashboard test runner
+
+For the opt-in tracker dashboard **Testing** tab (suites, env gates, VS Code
+launch config, and real-inference safeguards), see
+[dashboard-test-runner.md](dashboard-test-runner.md).
+
 ## Optional-dependency tests
 
 Some tests import heavyweight or optional third-party packages and guard