US-048 — Memory budget, shard slots, and dropout relocation hardening

Renumbered from duplicate slot 20 (which belongs to tracker-node-hardening / US-020 in docs/prd.json).

Goal

Make node capacity limits explicit and enforce them consistently when the tracker assigns, rebalances, and relocates shards after a node dropout.

This is a follow-up to US-013, not a replacement. US-013 owns the coverage-first assignment and rebalance algorithm. This issue hardens the capacity contract around that algorithm: operator memory budget, maximum loaded shard slots, and relocation behavior when one node must absorb or split ranges after another node disappears.

Context

Recent work added the first part of the contract:

meshnet-node --memory MB is registered with the tracker as vram_bytes when explicitly set.
CPU nodes without --memory keep the tracker default capacity, preserving old behavior.
meshnet-node --max-shards N is accepted and registered as max_loaded_shards.
Tracker registration validates max_loaded_shards >= 1.

The current runtime still effectively has one active backend shard per node. A node may advertise max_loaded_shards, but the tracker does not yet use multiple shard slots in bin-packing, and the node does not yet host multiple concurrently loaded shard ranges.

Scope

Make tracker rebalance logic account for max_loaded_shards as a capacity multiplier or explicit shard-slot list.
Ensure a node is never assigned more total layers than its memory budget can support across all loaded shard slots.
Decide and implement the runtime behavior for multiple loaded shards:
- either support multiple concurrently loaded shard backends on one node, or
- keep one backend active and treat max_loaded_shards as future metadata, with tracker enforcement preventing multi-range assignment for now.
On heartbeat timeout, relocate the dropped node's uncovered layer range to eligible managed nodes while respecting both memory and shard-slot limits.
Surface the effective memory budget and shard slot count in tracker/network inspection output so operators can diagnose why a node did or did not receive a range.

Non-Goals

Do not redesign the US-013 coverage-first algorithm from scratch.
Do not change relay, /ws, or /rpc behavior.
Do not change the token/reward model.
Do not require public internet verification; all behavior must be locally testable.

Acceptance Criteria

Tracker stores and exposes max_loaded_shards for registered nodes.
Assignment/rebalance never exceeds:
- assigned_layers_total <= floor((vram_bytes * 0.8) / bytes_per_layer_at_quant)
- assigned_range_count <= max_loaded_shards
A managed node with max_loaded_shards=1 only receives one active shard range.
A managed node with max_loaded_shards=2 can absorb two non-contiguous uncovered ranges only if the node runtime supports serving both; otherwise tracker must keep assigning at most one range and document max_loaded_shards as reserved.
Dropout test: register nodes covering a model, let a middle/tail node heartbeat-expire, and assert the tracker queues LOAD_SHARD directives that restore full coverage without violating memory or shard-slot limits.
CLI test: --memory and --max-shards are reflected in the registration payload.
python -m pytest tests/test_tracker_routing.py tests/test_node_startup.py passes in the project virtualenv, aside from any pre-existing platform-specific wallet permission assertion documented in the final notes.

Implementation Notes

Existing files likely involved:
- packages/node/meshnet_node/cli.py
- packages/node/meshnet_node/startup.py
- packages/node/meshnet_node/torch_server.py
- packages/tracker/meshnet_tracker/server.py
- tests/test_tracker_routing.py
- tests/test_node_startup.py
Keep backward compatibility: nodes that omit vram_bytes default to tracker defaults; nodes that omit max_loaded_shards default to 1.
Prefer a small internal representation for assigned ranges if multiple ranges become real, for example assigned_shards: list[tuple[int, int]], while preserving shard_start/shard_end in public responses for single-range nodes.

Comments

2026-06-30: Created after implementing the initial registration plumbing in commit f1e4ed6 (--memory, --max-shards, tracker validation). This issue captures the remaining end-to-end behavior so it does not conflict with US-013.
2026-06-30: Implementation decision: max_loaded_shards is currently a validated and exposed capacity field, but multi-range assignment remains reserved because TorchNodeServer serves one active backend shard. The tracker therefore emits at most one active range per node while exposing vram_bytes, ram_bytes, max_loaded_shards, quantization, throughput, and computed max_assignable_layers in inspection endpoints.
2026-07-13: Renumbered from docs/issues/20-memory-budget-… to resolve duplicate issue slot 20.

4.9 KiB Raw Blame History