From 344f432880ed32f55b79494ea5597dd15a8b6023 Mon Sep 17 00:00:00 2001 From: Dobromir Popov Date: Tue, 30 Jun 2026 21:19:47 +0200 Subject: [PATCH] tasks --- .../13-coverage-first-shard-assignment.md | 4 ++ ...dget-shard-slots-and-dropout-relocation.md | 65 +++++++++++++++++++ 2 files changed, 69 insertions(+) create mode 100644 .scratch/distributed-inference-network/issues/20-memory-budget-shard-slots-and-dropout-relocation.md diff --git a/.scratch/distributed-inference-network/issues/13-coverage-first-shard-assignment.md b/.scratch/distributed-inference-network/issues/13-coverage-first-shard-assignment.md index 5db9829..d9712ea 100644 --- a/.scratch/distributed-inference-network/issues/13-coverage-first-shard-assignment.md +++ b/.scratch/distributed-inference-network/issues/13-coverage-first-shard-assignment.md @@ -47,3 +47,7 @@ Model preset metadata (stored in tracker config) includes `bytes_per_layer` at e - Rebalance directives are delivered as responses to node heartbeat POSTs (node polls tracker every 10s anyway) — no new push channel needed - `bytes_per_layer` is read from a `model_presets.json` config file; add GPT-2 (12 layers, ~30MB/layer bfloat16) as the test preset - The current tracker `_select_route` function is extended, not replaced — backward compat with stub nodes that don't send VRAM data (default to 8GB / bfloat16 if omitted) + +## Comments + +- 2026-06-30: Follow-up capacity hardening lives in `20-memory-budget-shard-slots-and-dropout-relocation.md`. US-013 remains the base coverage-first assignment and dropout rebalance story; US-020 owns operator `--memory`, `--max-shards`, shard-slot enforcement, and relocation limit hardening so the two scopes do not conflict. diff --git a/.scratch/distributed-inference-network/issues/20-memory-budget-shard-slots-and-dropout-relocation.md b/.scratch/distributed-inference-network/issues/20-memory-budget-shard-slots-and-dropout-relocation.md new file mode 100644 index 0000000..f128193 --- /dev/null +++ b/.scratch/distributed-inference-network/issues/20-memory-budget-shard-slots-and-dropout-relocation.md @@ -0,0 +1,65 @@ +Status: ready-for-agent + +# US-020 - Memory budget, shard slots, and dropout relocation hardening + +## Goal + +Make node capacity limits explicit and enforce them consistently when the tracker assigns, rebalances, and relocates shards after a node dropout. + +This is a follow-up to US-013, not a replacement. US-013 owns the coverage-first assignment and rebalance algorithm. This issue hardens the capacity contract around that algorithm: operator memory budget, maximum loaded shard slots, and relocation behavior when one node must absorb or split ranges after another node disappears. + +## Context + +Recent work added the first part of the contract: + +- `meshnet-node --memory MB` is registered with the tracker as `vram_bytes` when explicitly set. +- CPU nodes without `--memory` keep the tracker default capacity, preserving old behavior. +- `meshnet-node --max-shards N` is accepted and registered as `max_loaded_shards`. +- Tracker registration validates `max_loaded_shards >= 1`. + +The current runtime still effectively has one active backend shard per node. A node may advertise `max_loaded_shards`, but the tracker does not yet use multiple shard slots in bin-packing, and the node does not yet host multiple concurrently loaded shard ranges. + +## Scope + +- Make tracker rebalance logic account for `max_loaded_shards` as a capacity multiplier or explicit shard-slot list. +- Ensure a node is never assigned more total layers than its memory budget can support across all loaded shard slots. +- Decide and implement the runtime behavior for multiple loaded shards: + - either support multiple concurrently loaded shard backends on one node, or + - keep one backend active and treat `max_loaded_shards` as future metadata, with tracker enforcement preventing multi-range assignment for now. +- On heartbeat timeout, relocate the dropped node's uncovered layer range to eligible managed nodes while respecting both memory and shard-slot limits. +- Surface the effective memory budget and shard slot count in tracker/network inspection output so operators can diagnose why a node did or did not receive a range. + +## Non-Goals + +- Do not redesign the US-013 coverage-first algorithm from scratch. +- Do not change relay, `/ws`, or `/rpc` behavior. +- Do not change the token/reward model. +- Do not require public internet verification; all behavior must be locally testable. + +## Acceptance Criteria + +- Tracker stores and exposes `max_loaded_shards` for registered nodes. +- Assignment/rebalance never exceeds: + - `assigned_layers_total <= floor((vram_bytes * 0.8) / bytes_per_layer_at_quant)` + - `assigned_range_count <= max_loaded_shards` +- A managed node with `max_loaded_shards=1` only receives one active shard range. +- A managed node with `max_loaded_shards=2` can absorb two non-contiguous uncovered ranges only if the node runtime supports serving both; otherwise tracker must keep assigning at most one range and document `max_loaded_shards` as reserved. +- Dropout test: register nodes covering a model, let a middle/tail node heartbeat-expire, and assert the tracker queues `LOAD_SHARD` directives that restore full coverage without violating memory or shard-slot limits. +- CLI test: `--memory` and `--max-shards` are reflected in the registration payload. +- `python -m pytest tests/test_tracker_routing.py tests/test_node_startup.py` passes in the project virtualenv, aside from any pre-existing platform-specific wallet permission assertion documented in the final notes. + +## Implementation Notes + +- Existing files likely involved: + - `packages/node/meshnet_node/cli.py` + - `packages/node/meshnet_node/startup.py` + - `packages/node/meshnet_node/torch_server.py` + - `packages/tracker/meshnet_tracker/server.py` + - `tests/test_tracker_routing.py` + - `tests/test_node_startup.py` +- Keep backward compatibility: nodes that omit `vram_bytes` default to tracker defaults; nodes that omit `max_loaded_shards` default to `1`. +- Prefer a small internal representation for assigned ranges if multiple ranges become real, for example `assigned_shards: list[tuple[int, int]]`, while preserving `shard_start`/`shard_end` in public responses for single-range nodes. + +## Comments + +- 2026-06-30: Created after implementing the initial registration plumbing in commit `f1e4ed6` (`--memory`, `--max-shards`, tracker validation). This issue captures the remaining end-to-end behavior so it does not conflict with US-013.