# Certified hardware lanes

A **certified hardware lane** is one (hardware, torch build, OS) configuration on which we
have *evidence* that nodes can execute real work — a self-hosted release runner that runs
the opt-in integration doctor test and keeps the resulting capability report as the
artifact. This document is the contract that lane runners and the release check must meet.

Certification is per *lane*, and evidence is per *(lane, model, shard, recipe)*. Nothing
here promises that an arbitrary model runs on a certified lane; it promises that the lane
itself is real, and that the models we ran on it produced passing capability reports on a
named date. See [ADR-0023](../adr/0023-model-agnostic-node-capability-admission.md) for the
admission model this rests on.

## What certification is not

- **Not a model support matrix.** A lane certifies hardware, not models. A new model is
  unproven on a certified lane until doctor has run it there.
- **Not an optional-kernel promise.** An optional accelerator package importing cleanly on
  a lane says nothing about another lane, another GPU architecture, or another model's
  recipe. Only a passing report for that exact combination is evidence.
- **Not a promise the node will install anything.** Lane runners are provisioned *ahead of
  time*, by hand or by their own image build. The node under test never downloads an
  executable recipe, installs a Python or OS package, or touches a driver. Signed node
  updates are a deliberate follow-up feature and are out of scope here — nothing in this
  lane contract may depend on dynamic executable-dependency installation.

## The lane check

Every lane runs the same environment-configured integration test. It is
`tests/test_node_doctor.py::test_doctor_smoke_runs_a_real_forward_on_a_real_model`, marked
`@pytest.mark.integration` and skipped unless `MESHNET_DOCTOR_MODEL` is set. It carries no
default model: **model identity comes from the CI configuration**, so no vendor or model
assumption can leak into the suite.

```bash
MESHNET_DOCTOR_MODEL="$LANE_MODEL" \
MESHNET_DOCTOR_QUANTIZATION=bfloat16 \
MESHNET_DOWNLOAD_DIR=/srv/models \
  .venv/bin/pytest -m integration tests/test_node_doctor.py -v
```

| Variable | Required | Meaning |
|----------|----------|---------|
| `MESHNET_DOCTOR_MODEL` | yes — the test skips without it | Model artifact identity for this lane's run. No default. |
| `MESHNET_DOCTOR_SHARD_START` | no (default `0`) | First layer of the shard to prove. |
| `MESHNET_DOCTOR_SHARD_END` | no (default: whole model) | Last layer, **inclusive**. |
| `MESHNET_DOCTOR_QUANTIZATION` | no (default `auto`) | Quantization to prove. |
| `MESHNET_DOCTOR_CPU` | no | `1` forces CPU — use to certify a CPU lane on GPU hardware. |
| `MESHNET_DOWNLOAD_DIR` | no | Where the artifact is cached on the runner. |

The test asserts that doctor passed, that the report is `passed` with the model id it was
asked for, that a forward actually took time (`duration_ms > 0`), and that the report
round-trips through `CapabilityReport.from_json`. A lane where this fails is not certified,
regardless of what `rocminfo`, `nvidia-smi` or `torch.cuda.is_available()` say.

Lanes that must cover more than the default recipe run doctor directly with
`--all-recipes`, which validates every recipe for the selection and writes a report per
recipe:

```bash
.venv/bin/meshnet-node doctor --model "$LANE_MODEL" --all-recipes --report "$ARTIFACTS/capability.json"
```

## Expected evidence

A lane run is only certified if it produces, and the release check retains:

1. **The capability report(s)** — `capability.json` from the run, archived as a build
   artifact. This is the evidence; a green checkmark without it is not.
2. **Backend identity from the report**: device, torch/backend version, and the recipe id
   and version that passed. This is what makes "certified on ROCm gfx1151" a checkable
   claim rather than a slogan.
3. **The model artifact identity and shard range** the report covers — recorded as run
   configuration, since it came from the environment.
4. **Failures kept, not discarded.** Doctor writes a report for a failed recipe too, and a
   failing lane must archive it. A red lane with a `forward-failed` report is a more useful
   release signal than a lane that was quietly skipped.

A lane that *skips* (because `MESHNET_DOCTOR_MODEL` was unset) must be reported as skipped,
never as passed. A silent skip is how an uncertified lane gets mistaken for a certified one.

## Release check

The default CI lane runs the normal suite and never needs a GPU, a download, or torch:

```bash
.venv/bin/pytest -m "not integration"
```

The release check additionally requires every declared certified lane to have run the
integration doctor test green, against the model(s) configured for that lane, on the
release commit. Adding a lane means standing up a runner and adding its configuration; it
does not mean adding a model default to the test suite.

## Adding a lane

1. Provision the runner: OS, driver, and the torch build for that hardware (see the
   platform sections in `QUICKSTART.md`). Install any optional accelerator packages the
   lane is meant to certify.
2. Configure `MESHNET_DOCTOR_MODEL` (and shard/quantization if the lane certifies a
   partial shard) in the runner's CI configuration.
3. Run the lane check. Archive the capability report.
4. Record the lane with the evidence it produced: hardware, torch build, model, shard,
   recipe, device, and the date. That record — not the hardware's spec sheet — is the
   support claim.