# DGR-040 evidence — node-side native worker supervision

**Date:** 2026-08-01  
**Authority:** `.scratch/distributed-gguf-runtime/prd.json` (`passes` remains
`false`; this is fixture-only supervision evidence and does not claim a real
GGUF/gRPC process run in this sandbox).

## Implemented

- Added `NativeWorkerSupervisor`, the node-side owner of one standalone native
  worker's process lifecycle. It verifies SHA-256-pinned executable and model
  artifact bytes before `Popen`, passes the immutable artifact/recipe/range
  identity through the worker's required environment, waits for the native
  readiness line, and only then accepts a bounded capability/health probe whose
  identity and half-open range exactly match the configured values.
- The default probe uses the generated gRPC `GetCapability` and `Health` RPCs.
  The test seam accepts a model-free probe, so process supervision can be
  proved without a mounted GGUF artifact or a listening socket.
- Both stdout and stderr are captured into a bounded in-memory log tail.
  `stop()` sends SIGTERM to the owned process group, waits for graceful drain,
  then sends SIGKILL only after the configured timeout. `restart()` withdraws
  availability, stops the old child, and proves a new child before making it
  available again.
- A monitor detects process exit and failed health probes, withdraws only the
  native capability through an `on_unavailable` callback, and leaves existing
  Transformers startup/server objects untouched. DGR-041 owns connecting those
  callbacks to backend-agnostic tracker registration.
- Added deterministic fake-worker tests. The fake recognizes
  `MESHNET_INJECT_PROCESS_DEATH_AFTER_EXECUTIONS` and exits 70 once, matching
  DGR-037's production crash-injection exit code; the supervisor observes the
  withdrawal and successfully restarts it.

## Changed files

- `packages/node/meshnet_node/native_worker_supervisor.py`
- `tests/test_native_worker_supervisor.py`
- `.scratch/distributed-gguf-runtime/evidence/DGR-040/README.md`
- `.ralph-tui/progress.md`

## Commands and results

```bash
PYTHONPATH=packages/node /home/popov/.hermes/hermes-agent/venv/bin/python -m pytest -q \
  tests/test_native_worker_supervisor.py tests/test_llama_shard_worker_binding.py \
  tests/test_native_shard_protocol.py
# 60 passed, 2 skipped in 0.97s

python3 -m compileall -q packages tests
# exit 0

git diff --check
# exit 0

/home/popov/.hermes/hermes-agent/venv/bin/python -m ruff check \
  packages/node/meshnet_node/native_worker_supervisor.py \
  tests/test_native_worker_supervisor.py
# All checks passed!
```

The system Python and repository `.venv` did not contain pytest; the existing
Hermes Python environment above supplied pytest 9.0.3 and grpc for the focused
checks. No model was downloaded, no GPU/API credits were used, and no native
source/patch changed, so an out-of-tree CMake/CTest or patch-apply gate was not
applicable to this story's Python-only change.

## Limitations

- The real worker requires a mounted GGUF artifact and a pinned native runtime;
  this fixture run did not exercise the default socket-based gRPC probe. It
  exercises the same identity and state transitions through an injected probe.
- Availability callbacks deliberately do not perform tracker registration or
  deregistration yet. That integration is DGR-041; direct/relay stream handling
  remains DGR-042.
- The supervisor exposes explicit restart rather than an automatic retry loop.
  Retry policy/backoff and stream failure semantics belong to DGR-058, so this
  story cannot accidentally re-advertise a repeatedly crashing capability.

## Dependency handoff

- DGR-033 supplied the readiness line and SIGTERM-clean-shutdown contract used
  here. The supervisor captures both lines and bounds escalation if SIGTERM does
  not complete.
- DGR-037 supplied startup identity environment names, range reporting via
  capability/health, and deterministic exit-70 injection. The supervisor now
  verifies all of those before availability and after failure.
- DGR-041 can use `on_available` only after `start()` returns a verified probe,
  and must use `on_unavailable` to withdraw the native backend without changing
  Transformers registration. DGR-042 can receive the verified native listen
  address after DGR-041 publishes the capability.