106 lines
5.5 KiB
Markdown
106 lines
5.5 KiB
Markdown
# Certified hardware lanes
|
|
|
|
A **certified hardware lane** is one (hardware, torch build, OS) configuration on which we
|
|
have *evidence* that nodes can execute real work — a self-hosted release runner that runs
|
|
the opt-in integration doctor test and keeps the resulting capability report as the
|
|
artifact. This document is the contract that lane runners and the release check must meet.
|
|
|
|
Certification is per *lane*, and evidence is per *(lane, model, shard, recipe)*. Nothing
|
|
here promises that an arbitrary model runs on a certified lane; it promises that the lane
|
|
itself is real, and that the models we ran on it produced passing capability reports on a
|
|
named date. See [ADR-0023](../adr/0023-model-agnostic-node-capability-admission.md) for the
|
|
admission model this rests on.
|
|
|
|
## What certification is not
|
|
|
|
- **Not a model support matrix.** A lane certifies hardware, not models. A new model is
|
|
unproven on a certified lane until doctor has run it there.
|
|
- **Not an optional-kernel promise.** An optional accelerator package importing cleanly on
|
|
a lane says nothing about another lane, another GPU architecture, or another model's
|
|
recipe. Only a passing report for that exact combination is evidence.
|
|
- **Not a promise the node will install anything.** Lane runners are provisioned *ahead of
|
|
time*, by hand or by their own image build. The node under test never downloads an
|
|
executable recipe, installs a Python or OS package, or touches a driver. Signed node
|
|
updates are a deliberate follow-up feature and are out of scope here — nothing in this
|
|
lane contract may depend on dynamic executable-dependency installation.
|
|
|
|
## The lane check
|
|
|
|
Every lane runs the same environment-configured integration test. It is
|
|
`tests/test_node_doctor.py::test_doctor_smoke_runs_a_real_forward_on_a_real_model`, marked
|
|
`@pytest.mark.integration` and skipped unless `MESHNET_DOCTOR_MODEL` is set. It carries no
|
|
default model: **model identity comes from the CI configuration**, so no vendor or model
|
|
assumption can leak into the suite.
|
|
|
|
```bash
|
|
MESHNET_DOCTOR_MODEL="$LANE_MODEL" \
|
|
MESHNET_DOCTOR_QUANTIZATION=bfloat16 \
|
|
MESHNET_DOWNLOAD_DIR=/srv/models \
|
|
.venv/bin/pytest -m integration tests/test_node_doctor.py -v
|
|
```
|
|
|
|
| Variable | Required | Meaning |
|
|
|----------|----------|---------|
|
|
| `MESHNET_DOCTOR_MODEL` | yes — the test skips without it | Model artifact identity for this lane's run. No default. |
|
|
| `MESHNET_DOCTOR_SHARD_START` | no (default `0`) | First layer of the shard to prove. |
|
|
| `MESHNET_DOCTOR_SHARD_END` | no (default: whole model) | Last layer, **inclusive**. |
|
|
| `MESHNET_DOCTOR_QUANTIZATION` | no (default `auto`) | Quantization to prove. |
|
|
| `MESHNET_DOCTOR_CPU` | no | `1` forces CPU — use to certify a CPU lane on GPU hardware. |
|
|
| `MESHNET_DOWNLOAD_DIR` | no | Where the artifact is cached on the runner. |
|
|
|
|
The test asserts that doctor passed, that the report is `passed` with the model id it was
|
|
asked for, that a forward actually took time (`duration_ms > 0`), and that the report
|
|
round-trips through `CapabilityReport.from_json`. A lane where this fails is not certified,
|
|
regardless of what `rocminfo`, `nvidia-smi` or `torch.cuda.is_available()` say.
|
|
|
|
Lanes that must cover more than the default recipe run doctor directly with
|
|
`--all-recipes`, which validates every recipe for the selection and writes a report per
|
|
recipe:
|
|
|
|
```bash
|
|
.venv/bin/meshnet-node doctor --model "$LANE_MODEL" --all-recipes --report "$ARTIFACTS/capability.json"
|
|
```
|
|
|
|
## Expected evidence
|
|
|
|
A lane run is only certified if it produces, and the release check retains:
|
|
|
|
1. **The capability report(s)** — `capability.json` from the run, archived as a build
|
|
artifact. This is the evidence; a green checkmark without it is not.
|
|
2. **Backend identity from the report**: device, torch/backend version, and the recipe id
|
|
and version that passed. This is what makes "certified on ROCm gfx1151" a checkable
|
|
claim rather than a slogan.
|
|
3. **The model artifact identity and shard range** the report covers — recorded as run
|
|
configuration, since it came from the environment.
|
|
4. **Failures kept, not discarded.** Doctor writes a report for a failed recipe too, and a
|
|
failing lane must archive it. A red lane with a `forward-failed` report is a more useful
|
|
release signal than a lane that was quietly skipped.
|
|
|
|
A lane that *skips* (because `MESHNET_DOCTOR_MODEL` was unset) must be reported as skipped,
|
|
never as passed. A silent skip is how an uncertified lane gets mistaken for a certified one.
|
|
|
|
## Release check
|
|
|
|
The default CI lane runs the normal suite and never needs a GPU, a download, or torch:
|
|
|
|
```bash
|
|
.venv/bin/pytest -m "not integration"
|
|
```
|
|
|
|
The release check additionally requires every declared certified lane to have run the
|
|
integration doctor test green, against the model(s) configured for that lane, on the
|
|
release commit. Adding a lane means standing up a runner and adding its configuration; it
|
|
does not mean adding a model default to the test suite.
|
|
|
|
## Adding a lane
|
|
|
|
1. Provision the runner: OS, driver, and the torch build for that hardware (see the
|
|
platform sections in `QUICKSTART.md`). Install any optional accelerator packages the
|
|
lane is meant to certify.
|
|
2. Configure `MESHNET_DOCTOR_MODEL` (and shard/quantization if the lane certifies a
|
|
partial shard) in the runner's CI configuration.
|
|
3. Run the lane check. Archive the capability report.
|
|
4. Record the lane with the evidence it produced: hardware, torch build, model, shard,
|
|
recipe, device, and the date. That record — not the hardware's spec sheet — is the
|
|
support claim.
|