90 lines
4.2 KiB
Markdown
90 lines
4.2 KiB
Markdown
# DGR-040 evidence — node-side native worker supervision
|
|
|
|
**Date:** 2026-08-01
|
|
**Authority:** `.scratch/distributed-gguf-runtime/prd.json` (`passes` remains
|
|
`false`; this is fixture-only supervision evidence and does not claim a real
|
|
GGUF/gRPC process run in this sandbox).
|
|
|
|
## Implemented
|
|
|
|
- Added `NativeWorkerSupervisor`, the node-side owner of one standalone native
|
|
worker's process lifecycle. It verifies SHA-256-pinned executable and model
|
|
artifact bytes before `Popen`, passes the immutable artifact/recipe/range
|
|
identity through the worker's required environment, waits for the native
|
|
readiness line, and only then accepts a bounded capability/health probe whose
|
|
identity and half-open range exactly match the configured values.
|
|
- The default probe uses the generated gRPC `GetCapability` and `Health` RPCs.
|
|
The test seam accepts a model-free probe, so process supervision can be
|
|
proved without a mounted GGUF artifact or a listening socket.
|
|
- Both stdout and stderr are captured into a bounded in-memory log tail.
|
|
`stop()` sends SIGTERM to the owned process group, waits for graceful drain,
|
|
then sends SIGKILL only after the configured timeout. `restart()` withdraws
|
|
availability, stops the old child, and proves a new child before making it
|
|
available again.
|
|
- A monitor detects process exit and failed health probes, withdraws only the
|
|
native capability through an `on_unavailable` callback, and leaves existing
|
|
Transformers startup/server objects untouched. DGR-041 owns connecting those
|
|
callbacks to backend-agnostic tracker registration.
|
|
- Added deterministic fake-worker tests. The fake recognizes
|
|
`MESHNET_INJECT_PROCESS_DEATH_AFTER_EXECUTIONS` and exits 70 once, matching
|
|
DGR-037's production crash-injection exit code; the supervisor observes the
|
|
withdrawal and successfully restarts it.
|
|
|
|
## Changed files
|
|
|
|
- `packages/node/meshnet_node/native_worker_supervisor.py`
|
|
- `tests/test_native_worker_supervisor.py`
|
|
- `.scratch/distributed-gguf-runtime/evidence/DGR-040/README.md`
|
|
- `.ralph-tui/progress.md`
|
|
|
|
## Commands and results
|
|
|
|
```bash
|
|
PYTHONPATH=packages/node /home/popov/.hermes/hermes-agent/venv/bin/python -m pytest -q \
|
|
tests/test_native_worker_supervisor.py tests/test_llama_shard_worker_binding.py \
|
|
tests/test_native_shard_protocol.py
|
|
# 60 passed, 2 skipped in 0.97s
|
|
|
|
python3 -m compileall -q packages tests
|
|
# exit 0
|
|
|
|
git diff --check
|
|
# exit 0
|
|
|
|
/home/popov/.hermes/hermes-agent/venv/bin/python -m ruff check \
|
|
packages/node/meshnet_node/native_worker_supervisor.py \
|
|
tests/test_native_worker_supervisor.py
|
|
# All checks passed!
|
|
```
|
|
|
|
The system Python and repository `.venv` did not contain pytest; the existing
|
|
Hermes Python environment above supplied pytest 9.0.3 and grpc for the focused
|
|
checks. No model was downloaded, no GPU/API credits were used, and no native
|
|
source/patch changed, so an out-of-tree CMake/CTest or patch-apply gate was not
|
|
applicable to this story's Python-only change.
|
|
|
|
## Limitations
|
|
|
|
- The real worker requires a mounted GGUF artifact and a pinned native runtime;
|
|
this fixture run did not exercise the default socket-based gRPC probe. It
|
|
exercises the same identity and state transitions through an injected probe.
|
|
- Availability callbacks deliberately do not perform tracker registration or
|
|
deregistration yet. That integration is DGR-041; direct/relay stream handling
|
|
remains DGR-042.
|
|
- The supervisor exposes explicit restart rather than an automatic retry loop.
|
|
Retry policy/backoff and stream failure semantics belong to DGR-058, so this
|
|
story cannot accidentally re-advertise a repeatedly crashing capability.
|
|
|
|
## Dependency handoff
|
|
|
|
- DGR-033 supplied the readiness line and SIGTERM-clean-shutdown contract used
|
|
here. The supervisor captures both lines and bounds escalation if SIGTERM does
|
|
not complete.
|
|
- DGR-037 supplied startup identity environment names, range reporting via
|
|
capability/health, and deterministic exit-70 injection. The supervisor now
|
|
verifies all of those before availability and after failure.
|
|
- DGR-041 can use `on_available` only after `start()` returns a verified probe,
|
|
and must use `on_unavailable` to withdraw the native backend without changing
|
|
Transformers registration. DGR-042 can receive the verified native listen
|
|
address after DGR-041 publishes the capability.
|