neuron-tai/.scratch/distributed-gguf-runtime/evidence/DGR-040

DGR-040 evidence — node-side native worker supervision

Date: 2026-08-01
Authority: .scratch/distributed-gguf-runtime/prd.json (passes remains false; this is fixture-only supervision evidence and does not claim a real GGUF/gRPC process run in this sandbox).

Implemented

• Added NativeWorkerSupervisor, the node-side owner of one standalone native worker's process lifecycle. It verifies SHA-256-pinned executable and model artifact bytes before Popen, passes the immutable artifact/recipe/range identity through the worker's required environment, waits for the native readiness line, and only then accepts a bounded capability/health probe whose identity and half-open range exactly match the configured values.
• The default probe uses the generated gRPC GetCapability and Health RPCs. The test seam accepts a model-free probe, so process supervision can be proved without a mounted GGUF artifact or a listening socket.
• Both stdout and stderr are captured into a bounded in-memory log tail. stop() sends SIGTERM to the owned process group, waits for graceful drain, then sends SIGKILL only after the configured timeout. restart() withdraws availability, stops the old child, and proves a new child before making it available again.
• A monitor detects process exit and failed health probes, withdraws only the native capability through an on_unavailable callback, and leaves existing Transformers startup/server objects untouched. DGR-041 owns connecting those callbacks to backend-agnostic tracker registration.
• Added deterministic fake-worker tests. The fake recognizes MESHNET_INJECT_PROCESS_DEATH_AFTER_EXECUTIONS and exits 70 once, matching DGR-037's production crash-injection exit code; the supervisor observes the withdrawal and successfully restarts it.

Changed files

• packages/node/meshnet_node/native_worker_supervisor.py
• tests/test_native_worker_supervisor.py
• .scratch/distributed-gguf-runtime/evidence/DGR-040/README.md
• .ralph-tui/progress.md

Commands and results

PYTHONPATH=packages/node /home/popov/.hermes/hermes-agent/venv/bin/python -m pytest -q \
  tests/test_native_worker_supervisor.py tests/test_llama_shard_worker_binding.py \
  tests/test_native_shard_protocol.py
# 60 passed, 2 skipped in 0.97s

python3 -m compileall -q packages tests
# exit 0

git diff --check
# exit 0

/home/popov/.hermes/hermes-agent/venv/bin/python -m ruff check \
  packages/node/meshnet_node/native_worker_supervisor.py \
  tests/test_native_worker_supervisor.py
# All checks passed!

The system Python and repository .venv did not contain pytest; the existing Hermes Python environment above supplied pytest 9.0.3 and grpc for the focused checks. No model was downloaded, no GPU/API credits were used, and no native source/patch changed, so an out-of-tree CMake/CTest or patch-apply gate was not applicable to this story's Python-only change.

Limitations

• The real worker requires a mounted GGUF artifact and a pinned native runtime; this fixture run did not exercise the default socket-based gRPC probe. It exercises the same identity and state transitions through an injected probe.
• Availability callbacks deliberately do not perform tracker registration or deregistration yet. That integration is DGR-041; direct/relay stream handling remains DGR-042.
• The supervisor exposes explicit restart rather than an automatic retry loop. Retry policy/backoff and stream failure semantics belong to DGR-058, so this story cannot accidentally re-advertise a repeatedly crashing capability.

Dependency handoff

• DGR-033 supplied the readiness line and SIGTERM-clean-shutdown contract used here. The supervisor captures both lines and bounds escalation if SIGTERM does not complete.
• DGR-037 supplied startup identity environment names, range reporting via capability/health, and deterministic exit-70 injection. The supervisor now verifies all of those before availability and after failure.
• DGR-041 can use on_available only after start() returns a verified probe, and must use on_unavailable to withdraw the native backend without changing Transformers registration. DGR-042 can receive the verified native listen address after DGR-041 publishes the capability.