doctor docs

.scratch/node-capability-admission/PRD.md

@@ -0,0 +1,84 @@
+# PRD: Model-agnostic Node capability admission
+
+## Overview
+
+Make a Node demonstrate that it can execute the selected Model Artifact and assigned Shard before the Tracker exposes it in an Inference Route. The current flow registers a Node after a hardware inventory and synthetic Torch benchmark, but before any real model forward; optional backend/JIT failures can therefore occur on paid traffic.
+
+The solution is generic by design. It must not hardcode Qwen3.6, FLA, Triton, CUDA, ROCm, or any other model/backend name into the capability contract. Qwen3.6 may be used only by opt-in development integration tests.
+
+## Goals
+
+- Provide `meshnet-node doctor` to emit a machine-readable and human-readable capability report for the selected model/shard.
+- Require a successful real forward on the selected execution path before a Node becomes routable.
+- Track named recipes as data, allowing more than one validated implementation for one Model Preset.
+- Let the Tracker schedule only a Node/model/shard/recipe combination that the Node validated locally.
+- Preserve current generic Hugging Face model support and backward-compatible protocol behavior where possible.
+
+## Quality gates
+
+Every user story:
+
+- Runs its targeted `pytest` tests.
+- Runs full `pytest` before completion, or records the exact unrelated failure.
+- Keeps default tests deterministic, model-download-free, and GPU-free.
+
+Release/hardware CI:
+
+- Runs an `integration`-marked real-model doctor smoke test per certified hardware lane.
+- Passes the model ID, source, and expected backend through environment/configuration; no test has a Qwen-specific default.
+
+## User stories
+
+### NCA-001: Generic capability and recipe report
+
+As a Node operator, I need a model-agnostic capability report so that readiness is based on the executable model/shard/backend combination, not a generic GPU claim.
+
+### NCA-002: Doctor selected model/shard
+
+As a Node operator, I need `meshnet-node doctor` to validate the selected model/shard with a real forward before I join the network.
+
+### NCA-003: Fail-closed startup admission
+
+As a Node operator, I need startup to remain non-routable when the selected recipe fails so that a Node never accepts paid work it cannot execute.
+
+### NCA-004: Tracker validated-recipe routing gate
+
+As a client, I need the Tracker to select only validated Node capabilities so that an Inference Route does not include a Node that merely claims compatibility.
+
+### NCA-005: Model-agnostic operations and certified-lane verification
+
+As an operator and release engineer, I need clear doctor output and opt-in hardware-lane test instructions so that failures are actionable without exposing Python/JIT internals to ordinary users.
+
+## Functional requirements
+
+1. The local capability report identifies the Model Artifact by generic model ID/revision/config fingerprint, shard range, selected recipe ID/version, device/backend identity, success/failure status, diagnostics, and measured validation duration.
+2. A recipe is data, not a model-specific code branch. A model may offer multiple recipes; a recipe is valid only after its own real forward succeeds.
+3. `doctor` defaults to the selected model/shard and does not search/download/test unrelated models. `--all-recipes` is explicit.
+4. Startup must execute or consume a fresh matching validation before ready registration. A failed selected recipe exits non-zero before routable registration.
+5. The Tracker records validated capabilities and excludes invalid, absent, stale, model-mismatched, shard-mismatched, or catalogue-version-incompatible capabilities from route selection.
+6. The tracker protocol remains tolerant of old Nodes only during a documented compatibility window; old registrations are not eligible for routes requiring admission proof.
+7. The Node reports a versioned local recipe-manifest version. P0 has no remote executable recipe download, dependency installer, self-updater, driver installer, or GUI.
+
+## Non-goals
+
+- A signed Node auto-updater or dynamic executable recipe delivery.
+- Automatic installation of OS packages, compilers, drivers, or Python dependencies.
+- A native NiceHash-style desktop manager.
+- Supporting or certifying a particular model, GPU vendor, OS, or optional-kernel library.
+- Replacing the existing Model Artifact/assignment protocol.
+
+## Architecture
+
+Add a small generic capability domain object in the node package. `doctor` loads the requested generic model path through the same backend startup uses, executes a bounded real forward at the assigned Shard, and emits the report. Startup gates routable registration on the successful report. Registration carries validated capabilities; the tracker persists/exposes them and filters route candidates at the model/shard/recipe seam.
+
+The future signed-update contract is represented only by a local manifest version and generic schema in P0. A future Tracker Model Artifact Manifest may be signed data, but Node executable behavior remains supplied by signed Node releases.
+
+## Success measures
+
+- A backend failure that formerly appeared on the first `/forward` is caught by `doctor` and prevents ready registration.
+- A Tracker route never includes an unvalidated capability in deterministic tests.
+- The same implementation works for arbitrary test model identifiers supplied by fixtures/configuration, with no Qwen-specific branch.
+
+## Open follow-up
+
+Specify and build the signed Node release/update channel as a separate product feature after the capability contract has proved stable.

.scratch/node-capability-admission/README.md

@@ -0,0 +1,31 @@
+# Node capability admission — planning index
+
+**Status:** ready for supervised Ralph execution.
+
+This P0 makes a Node prove it can serve its selected Model Artifact and Shard before the Tracker treats it as routable. It is deliberately model-agnostic: Qwen3.6 is only a development integration fixture, never a hardcoded runtime target.
+
+## Locked decisions
+
+- A Node explicitly asked to serve a Model Preset fails closed when no validated recipe can execute it; it must not register as ready or accept paid inference.
+- Default validation covers the selected model/shard only. `meshnet-node doctor --all-recipes` is reserved for support and CI.
+- A Model Preset may have multiple named recipes. Each independently proves a real forward; the Tracker schedules only validated recipes while considering measured performance.
+- Compatibility schemas are generic. A future Tracker may publish signed, data-only Model Artifact Manifests, but executable recipes arrive only through signed Node releases.
+- P0 ships a local versioned recipe manifest and reports its version. It does **not** build a self-updater, download executable recipes, or install system dependencies.
+- Every story requires `pytest`; release CI additionally runs an `integration`-marked real-model doctor smoke test on each certified hardware lane.
+
+## Ralph order
+
+1. `NCA-001` generic capability/report contract
+2. `NCA-002` generic doctor command and real-forward validation
+3. `NCA-003` startup admission lifecycle and fail-closed behavior
+4. `NCA-004` tracker registration/routing enforcement
+5. `NCA-005` operator documentation and hardware-lane integration contract
+
+Run serially. Stories 3 and 4 both change registration/startup behavior and must not be executed in parallel.
+
+## Quality gates
+
+- Targeted `pytest` tests named by the issue.
+- Full `pytest` before marking a story done, or record the unrelated blocker.
+- No default test downloads a model or requires a GPU.
+- `pytest -m integration` / the real-model doctor test remains explicit and environment-gated.

.scratch/node-capability-admission/issues/01-generic-capability-report.md

@@ -0,0 +1,34 @@
+Status: ready-for-agent
+
+# 01 — Generic capability and recipe report
+
+## What to build
+
+Create a model-agnostic node capability domain object and local versioned recipe-manifest reader. It must represent a selected Model Artifact identity/revision/config fingerprint, Shard range, named recipe ID/version, device/backend identity, validation timestamp/duration, success/failure state, and sanitized diagnostics.
+
+Do not add Qwen-, FLA-, Triton-, CUDA-, ROCm-, or vendor-specific branches. A recipe is generic data; specific runtime behavior remains in the existing backend.
+
+**Code refs:**
+
+- `packages/node/meshnet_node/model_catalog.py` — existing generic HF config/model metadata helpers
+- `packages/node/meshnet_node/hardware.py` — device identity/executability inventory
+- `packages/node/meshnet_node/model_backend.py` — model/shard loading path
+- `packages/node/pyproject.toml` — package data declarations
+
+## Test-first
+
+1. Write a unit test building reports for two arbitrary fixture model IDs and asserting no model-specific normalization/branch is required.
+2. Write a manifest-version validation test: valid local manifest loads; malformed/unknown schema produces actionable non-secret diagnostics.
+3. Implement the smallest schema, serialization, and local manifest reader needed by later stories.
+
+## Acceptance criteria
+
+- [ ] Capability report has a stable JSON-serializable schema with model identity/fingerprint, shard range, recipe ID/version, backend/device identity, status, timing, and sanitized diagnostic fields
+- [ ] Generic arbitrary model IDs are preserved; no Qwen or optional-kernel name is a product default or code-path discriminator
+- [ ] Local recipe manifest has an explicit schema/catalogue version
+- [ ] Malformed manifest/report input fails locally with actionable diagnostics and never leaks environment secrets
+- [ ] Unit tests cover serialization, schema validation, and model-agnostic behavior
+
+## Blocked by
+
+None.

.scratch/node-capability-admission/issues/02-doctor-real-forward.md

@@ -0,0 +1,35 @@
+Status: ready-for-agent
+
+# 02 — Doctor selected model/shard with a bounded real forward
+
+## What to build
+
+Add `meshnet-node doctor`. By default it validates only the selected Model Artifact and Shard from flags/config. Reuse the production model-loading/backend execution path and execute a bounded real forward through the selected Shard; a generic Torch allocation or synthetic benchmark is insufficient.
+
+It emits concise human output plus capability-report JSON. Add explicit `--all-recipes` plumbing for support/CI without making ordinary startup validate unrelated/downloaded models. The default tests must inject a fake/lightweight backend; a real-model test is integration-marked and environment-gated with model identity supplied externally.
+
+**Code refs:**
+
+- `packages/node/meshnet_node/cli.py` — subcommand parser and config/flag resolution
+- `packages/node/meshnet_node/model_backend.py` — `TorchModelShard`, `encode_prompt`, `forward_bytes`
+- `packages/node/meshnet_node/torch_server.py` — production backend construction
+- `tests/test_node_startup.py`, `tests/test_real_model_backend.py` — startup/backend test patterns
+
+## Test-first
+
+1. Red: `doctor` reports generic hardware availability as ready without invoking model validation.
+2. Red: an injected backend forward failure still produces a success capability.
+3. Green: selected model/shard invokes one bounded generic forward and yields success only on completion.
+4. Add an `integration`-marked, env-gated test whose model ID/source is configurable; it has no model-specific default.
+
+## Acceptance criteria
+
+- [ ] `meshnet-node doctor` resolves the same selected model/shard/config path as startup
+- [ ] Default doctor performs a bounded real forward through the selected shard before reporting success
+- [ ] `--all-recipes` is explicit and does not change default onboarding cost
+- [ ] Failure returns non-zero, writes a failed capability report, and prints a user-actionable category without raw traceback by default
+- [ ] Unit tests require no GPU or model download; a separately marked integration smoke test is model-configurable
+
+## Blocked by
+
+`01-generic-capability-report.md`.

.scratch/node-capability-admission/issues/03-fail-closed-startup-admission.md

@@ -0,0 +1,34 @@
+Status: ready-for-agent
+
+# 03 — Fail-closed startup admission lifecycle
+
+## What to build
+
+Gate `run_startup` on a fresh, matching successful capability report before routable Tracker registration. A Node selected for a Model Preset/shard must fail closed if its recipe cannot perform the doctor forward: no ready/registered endpoint and no paid request acceptance.
+
+Keep local diagnostic behavior useful: a failed report may be persisted/exposed locally, but the Node must not advertise the failed model/shard as ready. Define a bounded freshness/match rule so a report cannot be reused for a different model revision, shard, recipe, or backend identity.
+
+**Code refs:**
+
+- `packages/node/meshnet_node/startup.py` — download/load/start/register sequence
+- `packages/node/meshnet_node/cli.py` — `start` and default startup error paths
+- `packages/node/meshnet_node/torch_server.py` — server lifecycle
+- `tests/test_node_startup.py` — fake startup and registration capture patterns
+
+## Test-first
+
+1. Red: backend validation failure still causes `/v1/nodes/register` to be called.
+2. Red: a success report for one arbitrary model/shard is reused for another.
+3. Green: matching successful validation reaches registration; failed/stale/mismatched validation exits before registration.
+
+## Acceptance criteria
+
+- [ ] Explicit selected model/shard fails closed before routable registration when validation fails
+- [ ] Startup sends only a matching successful capability report with its registration payload
+- [ ] Failed, stale, model-mismatched, shard-mismatched, recipe-mismatched, and backend-mismatched reports are rejected locally
+- [ ] Existing stub/test startup remains usable through an explicit test-safe capability path, not a production bypass
+- [ ] Tests prove the tracker receives no registration on a failed validation
+
+## Blocked by
+
+`01-generic-capability-report.md`, `02-doctor-real-forward.md`.

.scratch/node-capability-admission/issues/04-tracker-validated-capability-routing.md

@@ -0,0 +1,34 @@
+Status: ready-for-agent
+
+# 04 — Tracker validated-capability registration and routing gate
+
+## What to build
+
+Extend Node registration, tracker state, network-map visibility, and route selection so a candidate is eligible only when it presents a successful capability report matching the route Model Artifact and Shard. Treat recipe/backend/capability data as evidence, not a trusted performance assertion. Preserve legacy behavior only through an explicit, documented compatibility policy; no new paid route may rely on an absent proof once admission is enforced.
+
+**Code refs:**
+
+- `packages/tracker/meshnet_tracker/server.py` — `/v1/nodes/register`, tracker node state, route selection, network map
+- `tests/test_tracker_routing.py` — registration and route tests
+- `packages/node/meshnet_node/startup.py` — registration payload producer
+- `docs/adr/0011-auto-shard-and-network-assignment.md` — tracker-owned assignment context
+- `docs/adr/0013-rolling-stats-smart-routing.md` — performance routing context
+
+## Test-first
+
+1. Red: a node with no/failed/mismatched capability report can register as route-eligible for a model/shard.
+2. Red: route selection includes a candidate whose report is for a different arbitrary model or shard.
+3. Green: valid matching candidates route normally; network map exposes only sanitized admission status.
+
+## Acceptance criteria
+
+- [ ] Registration validates the generic capability-report schema and records sanitized capability state
+- [ ] Route selection excludes invalid, absent, failed, stale, model-mismatched, shard-mismatched, recipe-mismatched, or catalogue-version-incompatible candidates
+- [ ] Valid matching candidates retain normal coverage-first and throughput routing behavior
+- [ ] Network map/operator view exposes an actionable admission state without raw exceptions or secrets
+- [ ] Protocol compatibility policy for older Nodes is tested and documented
+- [ ] Deterministic tracker tests cover arbitrary model IDs, not a Qwen fixture
+
+## Blocked by
+
+`01-generic-capability-report.md`, `03-fail-closed-startup-admission.md`.

.scratch/node-capability-admission/issues/05-docs-hardware-lane-contract.md

@@ -0,0 +1,34 @@
+Status: ready-for-agent
+
+# 05 — Model-agnostic operator documentation and hardware-lane contract
+
+## What to build
+
+Document the capability-admission lifecycle, `doctor` usage, failure states, model-agnostic recipe semantics, and the certified hardware-lane release check. Correct setup guidance so it does not imply that an optional accelerator path is universally supported merely because a package can be installed.
+
+Use generic commands/placeholders in primary docs. Any concrete model used for development belongs in a clearly labelled optional example or environment-gated test configuration, never a support guarantee.
+
+**Code refs:**
+
+- `QUICKSTART.md` — node installation/ROCm/optional-backend guidance
+- `packages/node/meshnet_node/cli.py` — doctor user-facing output
+- `docs/adr/0023-model-agnostic-node-capability-admission.md`
+- `tests/test_node_startup.py`, `tests/test_real_model_backend.py` — integration marker conventions
+
+## Test-first / verification
+
+1. Add tests for concise doctor output/category mapping where practical.
+2. Verify documentation commands use the generic selected-model interface and explain the distinction between validated versus merely detected hardware.
+3. Add a release-CI runbook contract for an opt-in `integration` doctor run per certified hardware lane, with model identity supplied by CI configuration.
+
+## Acceptance criteria
+
+- [ ] Docs explain that readiness requires a successful real-forward capability report
+- [ ] Docs distinguish detected hardware, validated recipe, and routable Node states
+- [ ] Docs make no model/vendor/optional-kernel universal support promise
+- [ ] Certified-lane CI contract is documented, including model-configurable integration environment and expected evidence
+- [ ] Signed Node updates are listed as a follow-up; P0 is explicit that it does not dynamically install executable recipes or system dependencies
+
+## Blocked by
+
+`02-doctor-real-forward.md`, `04-tracker-validated-capability-routing.md`.

.scratch/node-capability-admission/prd.json

@@ -0,0 +1,95 @@
+{
+  "name": "Model-agnostic Node capability admission",
+  "branchName": "ralph/node-capability-admission",
+  "description": "Make a Node prove its selected Model Artifact, Shard, and execution recipe work before it becomes routable. Qwen3.6 is only an opt-in development fixture; the implementation and protocol are model-agnostic.",
+  "userStories": [
+    {
+      "id": "NCA-001",
+      "title": "Generic capability and recipe report",
+      "description": "Create a model-agnostic versioned capability report and local recipe-manifest contract without model or vendor code branches.",
+      "acceptanceCriteria": [
+        "Stable JSON-serializable report includes generic model identity/fingerprint, shard range, recipe ID/version, backend/device identity, status, timing, and sanitized diagnostics",
+        "Arbitrary model IDs are preserved without Qwen or optional-kernel code paths",
+        "Local recipe manifest has explicit schema/catalogue version",
+        "Malformed input fails with actionable, secret-safe diagnostics",
+        "Targeted pytest passes",
+        "Full pytest passes or an exact unrelated blocker is recorded"
+      ],
+      "priority": 1,
+      "passes": false,
+      "notes": "Source issue: .scratch/node-capability-admission/issues/01-generic-capability-report.md",
+      "dependsOn": []
+    },
+    {
+      "id": "NCA-002",
+      "title": "Doctor selected model/shard with a bounded real forward",
+      "description": "Add model-agnostic doctor validation using the same backend execution path as startup.",
+      "acceptanceCriteria": [
+        "meshnet-node doctor resolves the same selected model/shard/config as startup",
+        "Default doctor performs one bounded real selected-shard forward before success",
+        "All-recipes mode is explicit",
+        "Failure exits non-zero and writes actionable, non-traceback diagnostics by default",
+        "Unit tests have no GPU/download requirement; integration doctor smoke test is marker- and model-config-gated",
+        "Targeted pytest passes",
+        "Full pytest passes or an exact unrelated blocker is recorded"
+      ],
+      "priority": 2,
+      "passes": false,
+      "notes": "Source issue: .scratch/node-capability-admission/issues/02-doctor-real-forward.md",
+      "dependsOn": ["NCA-001"]
+    },
+    {
+      "id": "NCA-003",
+      "title": "Fail-closed startup admission lifecycle",
+      "description": "Prevent a selected model/shard from registering as routable unless its matching capability report passed.",
+      "acceptanceCriteria": [
+        "Failed selected-recipe validation makes startup exit before tracker registration",
+        "Only a fresh matching model/shard/recipe/backend report can accompany registration",
+        "Stub tests use an explicit test-safe capability path rather than production bypass",
+        "Tests prove tracker registration is not called after validation failure",
+        "Targeted pytest passes",
+        "Full pytest passes or an exact unrelated blocker is recorded"
+      ],
+      "priority": 3,
+      "passes": false,
+      "notes": "Source issue: .scratch/node-capability-admission/issues/03-fail-closed-startup-admission.md",
+      "dependsOn": ["NCA-001", "NCA-002"]
+    },
+    {
+      "id": "NCA-004",
+      "title": "Tracker validated-capability routing gate",
+      "description": "Persist and expose validated generic capability data, then route only matching admitted candidates.",
+      "acceptanceCriteria": [
+        "Tracker validates/records sanitized generic report data",
+        "Route selection excludes invalid, absent, failed, stale, or mismatched capabilities",
+        "Valid candidates retain coverage-first and throughput routing behavior",
+        "Network map exposes safe admission state",
+        "Older-node compatibility policy is documented and tested",
+        "Deterministic tests use arbitrary model IDs",
+        "Targeted pytest passes",
+        "Full pytest passes or an exact unrelated blocker is recorded"
+      ],
+      "priority": 4,
+      "passes": false,
+      "notes": "Source issue: .scratch/node-capability-admission/issues/04-tracker-validated-capability-routing.md",
+      "dependsOn": ["NCA-001", "NCA-003"]
+    },
+    {
+      "id": "NCA-005",
+      "title": "Model-agnostic docs and hardware-lane contract",
+      "description": "Document doctor/admission behavior and the opt-in real-model CI lane without promising model-specific support.",
+      "acceptanceCriteria": [
+        "Docs distinguish detected hardware, validated recipe, and routable Node",
+        "Docs make no universal optional-backend/model/vendor claim",
+        "Certified-lane CI contract includes environment-configured integration doctor test",
+        "Signed updater is explicitly deferred; P0 has no dynamic executable dependency installation",
+        "Targeted pytest passes",
+        "Full pytest passes or an exact unrelated blocker is recorded"
+      ],
+      "priority": 5,
+      "passes": false,
+      "notes": "Source issue: .scratch/node-capability-admission/issues/05-docs-hardware-lane-contract.md",
+      "dependsOn": ["NCA-002", "NCA-004"]
+    }
+  ]
+}