story: DGR-040 Add node-side native worker supervision
This commit is contained in:
89
.scratch/distributed-gguf-runtime/evidence/DGR-040/README.md
Normal file
89
.scratch/distributed-gguf-runtime/evidence/DGR-040/README.md
Normal file
@@ -0,0 +1,89 @@
|
||||
# DGR-040 evidence — node-side native worker supervision
|
||||
|
||||
**Date:** 2026-08-01
|
||||
**Authority:** `.scratch/distributed-gguf-runtime/prd.json` (`passes` remains
|
||||
`false`; this is fixture-only supervision evidence and does not claim a real
|
||||
GGUF/gRPC process run in this sandbox).
|
||||
|
||||
## Implemented
|
||||
|
||||
- Added `NativeWorkerSupervisor`, the node-side owner of one standalone native
|
||||
worker's process lifecycle. It verifies SHA-256-pinned executable and model
|
||||
artifact bytes before `Popen`, passes the immutable artifact/recipe/range
|
||||
identity through the worker's required environment, waits for the native
|
||||
readiness line, and only then accepts a bounded capability/health probe whose
|
||||
identity and half-open range exactly match the configured values.
|
||||
- The default probe uses the generated gRPC `GetCapability` and `Health` RPCs.
|
||||
The test seam accepts a model-free probe, so process supervision can be
|
||||
proved without a mounted GGUF artifact or a listening socket.
|
||||
- Both stdout and stderr are captured into a bounded in-memory log tail.
|
||||
`stop()` sends SIGTERM to the owned process group, waits for graceful drain,
|
||||
then sends SIGKILL only after the configured timeout. `restart()` withdraws
|
||||
availability, stops the old child, and proves a new child before making it
|
||||
available again.
|
||||
- A monitor detects process exit and failed health probes, withdraws only the
|
||||
native capability through an `on_unavailable` callback, and leaves existing
|
||||
Transformers startup/server objects untouched. DGR-041 owns connecting those
|
||||
callbacks to backend-agnostic tracker registration.
|
||||
- Added deterministic fake-worker tests. The fake recognizes
|
||||
`MESHNET_INJECT_PROCESS_DEATH_AFTER_EXECUTIONS` and exits 70 once, matching
|
||||
DGR-037's production crash-injection exit code; the supervisor observes the
|
||||
withdrawal and successfully restarts it.
|
||||
|
||||
## Changed files
|
||||
|
||||
- `packages/node/meshnet_node/native_worker_supervisor.py`
|
||||
- `tests/test_native_worker_supervisor.py`
|
||||
- `.scratch/distributed-gguf-runtime/evidence/DGR-040/README.md`
|
||||
- `.ralph-tui/progress.md`
|
||||
|
||||
## Commands and results
|
||||
|
||||
```bash
|
||||
PYTHONPATH=packages/node /home/popov/.hermes/hermes-agent/venv/bin/python -m pytest -q \
|
||||
tests/test_native_worker_supervisor.py tests/test_llama_shard_worker_binding.py \
|
||||
tests/test_native_shard_protocol.py
|
||||
# 60 passed, 2 skipped in 0.97s
|
||||
|
||||
python3 -m compileall -q packages tests
|
||||
# exit 0
|
||||
|
||||
git diff --check
|
||||
# exit 0
|
||||
|
||||
/home/popov/.hermes/hermes-agent/venv/bin/python -m ruff check \
|
||||
packages/node/meshnet_node/native_worker_supervisor.py \
|
||||
tests/test_native_worker_supervisor.py
|
||||
# All checks passed!
|
||||
```
|
||||
|
||||
The system Python and repository `.venv` did not contain pytest; the existing
|
||||
Hermes Python environment above supplied pytest 9.0.3 and grpc for the focused
|
||||
checks. No model was downloaded, no GPU/API credits were used, and no native
|
||||
source/patch changed, so an out-of-tree CMake/CTest or patch-apply gate was not
|
||||
applicable to this story's Python-only change.
|
||||
|
||||
## Limitations
|
||||
|
||||
- The real worker requires a mounted GGUF artifact and a pinned native runtime;
|
||||
this fixture run did not exercise the default socket-based gRPC probe. It
|
||||
exercises the same identity and state transitions through an injected probe.
|
||||
- Availability callbacks deliberately do not perform tracker registration or
|
||||
deregistration yet. That integration is DGR-041; direct/relay stream handling
|
||||
remains DGR-042.
|
||||
- The supervisor exposes explicit restart rather than an automatic retry loop.
|
||||
Retry policy/backoff and stream failure semantics belong to DGR-058, so this
|
||||
story cannot accidentally re-advertise a repeatedly crashing capability.
|
||||
|
||||
## Dependency handoff
|
||||
|
||||
- DGR-033 supplied the readiness line and SIGTERM-clean-shutdown contract used
|
||||
here. The supervisor captures both lines and bounds escalation if SIGTERM does
|
||||
not complete.
|
||||
- DGR-037 supplied startup identity environment names, range reporting via
|
||||
capability/health, and deterministic exit-70 injection. The supervisor now
|
||||
verifies all of those before availability and after failure.
|
||||
- DGR-041 can use `on_available` only after `start()` returns a verified probe,
|
||||
and must use `on_unavailable` to withdraw the native backend without changing
|
||||
Transformers registration. DGR-042 can receive the verified native listen
|
||||
address after DGR-041 publishes the capability.
|
||||
Reference in New Issue
Block a user