neuron-tai/docs/dev/certified-hardware-lanes.md

Certified hardware lanes

A certified hardware lane is one (hardware, torch build, OS) configuration on which we have evidence that nodes can execute real work — a self-hosted release runner that runs the opt-in integration doctor test and keeps the resulting capability report as the artifact. This document is the contract that lane runners and the release check must meet.

Certification is per lane, and evidence is per (lane, model, shard, recipe). Nothing here promises that an arbitrary model runs on a certified lane; it promises that the lane itself is real, and that the models we ran on it produced passing capability reports on a named date. See ADR-0023 for the admission model this rests on.

What certification is not

• Not a model support matrix. A lane certifies hardware, not models. A new model is unproven on a certified lane until doctor has run it there.
• Not an optional-kernel promise. An optional accelerator package importing cleanly on a lane says nothing about another lane, another GPU architecture, or another model's recipe. Only a passing report for that exact combination is evidence.
• Not a promise the node will install anything. Lane runners are provisioned ahead of time, by hand or by their own image build. The node under test never downloads an executable recipe, installs a Python or OS package, or touches a driver. Signed node updates are a deliberate follow-up feature and are out of scope here — nothing in this lane contract may depend on dynamic executable-dependency installation.

The lane check

Every lane runs the same environment-configured integration test. It is tests/test_node_doctor.py::test_doctor_smoke_runs_a_real_forward_on_a_real_model, marked @pytest.mark.integration and skipped unless MESHNET_DOCTOR_MODEL is set. It carries no default model: model identity comes from the CI configuration, so no vendor or model assumption can leak into the suite.

MESHNET_DOCTOR_MODEL="$LANE_MODEL" \
MESHNET_DOCTOR_QUANTIZATION=bfloat16 \
MESHNET_DOWNLOAD_DIR=/srv/models \
  .venv/bin/pytest -m integration tests/test_node_doctor.py -v

Variable	Required	Meaning
MESHNET_DOCTOR_MODEL	yes — the test skips without it	Model artifact identity for this lane's run. No default.
MESHNET_DOCTOR_SHARD_START	no (default 0)	First layer of the shard to prove.
MESHNET_DOCTOR_SHARD_END	no (default: whole model)	Last layer, inclusive.
MESHNET_DOCTOR_QUANTIZATION	no (default auto)	Quantization to prove.
MESHNET_DOCTOR_CPU	no	1 forces CPU — use to certify a CPU lane on GPU hardware.
MESHNET_DOWNLOAD_DIR	no	Where the artifact is cached on the runner.

The test asserts that doctor passed, that the report is passed with the model id it was asked for, that a forward actually took time (duration_ms > 0), and that the report round-trips through CapabilityReport.from_json. A lane where this fails is not certified, regardless of what rocminfo, nvidia-smi or torch.cuda.is_available() say.

Lanes that must cover more than the default recipe run doctor directly with --all-recipes, which validates every recipe for the selection and writes a report per recipe:

.venv/bin/meshnet-node doctor --model "$LANE_MODEL" --all-recipes --report "$ARTIFACTS/capability.json"

Expected evidence

A lane run is only certified if it produces, and the release check retains:

1. The capability report(s) — capability.json from the run, archived as a build artifact. This is the evidence; a green checkmark without it is not.
2. Backend identity from the report: device, torch/backend version, and the recipe id and version that passed. This is what makes "certified on ROCm gfx1151" a checkable claim rather than a slogan.
3. The model artifact identity and shard range the report covers — recorded as run configuration, since it came from the environment.
4. Failures kept, not discarded. Doctor writes a report for a failed recipe too, and a failing lane must archive it. A red lane with a forward-failed report is a more useful release signal than a lane that was quietly skipped.

A lane that skips (because MESHNET_DOCTOR_MODEL was unset) must be reported as skipped, never as passed. A silent skip is how an uncertified lane gets mistaken for a certified one.

Release check

The default CI lane runs the normal suite and never needs a GPU, a download, or torch:

.venv/bin/pytest -m "not integration"

The release check additionally requires every declared certified lane to have run the integration doctor test green, against the model(s) configured for that lane, on the release commit. Adding a lane means standing up a runner and adding its configuration; it does not mean adding a model default to the test suite.

Adding a lane

1. Provision the runner: OS, driver, and the torch build for that hardware (see the platform sections in QUICKSTART.md). Install any optional accelerator packages the lane is meant to certify.
2. Configure MESHNET_DOCTOR_MODEL (and shard/quantization if the lane certifies a partial shard) in the runner's CI configuration.
3. Run the lane check. Archive the capability report.
4. Record the lane with the evidence it produced: hardware, torch build, model, shard, recipe, device, and the date. That record — not the hardware's spec sheet — is the support claim.