2b439e8a5f
Issue files (.scratch/issues/20-29): retrospective specs for all work done in the current sprint — hardening, route-timeout, start-layer protocol, heartbeat stats, availability map, rolling RPM, smart assignment, throughput routing, routing tests, relay outbound client. ADRs (docs/adr/0011-0014): 0011 — Auto-shard from memory budget and tracker network assignment 0012 — X-Meshnet-Start-Layer overlapping shard execution protocol 0013 — Rolling RPM statistics, smart assignment scoring, throughput routing 0014 — Relay outbound client for NAT/internet pipeline hops prd.json: US-020 through US-029 added, all marked done. ralph_progress.py now shows 29/29 complete (100%). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
27 lines
1.1 KiB
Markdown
27 lines
1.1 KiB
Markdown
# US-023 — Heartbeat stats payload: request counters + dynamic reassignment response
|
|
|
|
Status: done
|
|
Priority: Medium
|
|
Stage: Implemented
|
|
|
|
## Context
|
|
|
|
Node heartbeats are currently empty POSTs. The tracker has no visibility into per-node load,
|
|
making load balancing and assignment decisions blind. Heartbeats should carry cumulative stats.
|
|
The heartbeat response channel is also the natural place for the tracker to deliver reassignment
|
|
instructions without requiring a node restart.
|
|
|
|
## Acceptance criteria
|
|
|
|
- [ ] Heartbeat POST body includes: `total_requests`, `failed_requests`, `queue_depth`, `uptime_seconds`, `status`
|
|
- [ ] `TorchNodeServer` tracks the three counters with a `threading.Lock`
|
|
- [ ] Tracker stores the last heartbeat payload per node
|
|
- [ ] Heartbeat response may include `new_assignment: {model, shard_start, shard_end}`; node logs it
|
|
- [ ] Stats survive tracker outage: buffered locally, flushed on next successful heartbeat
|
|
- [ ] `python -m pytest` passes
|
|
|
|
## Notes
|
|
|
|
Hot-reload (loading a new shard without restart) is deferred to a future story. The response
|
|
field is wired so trackers can send the signal; nodes log it but don't act yet.
|