popov/neuron-tai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: default quantization int8, GB display, shard heal cycle test

Browse Source 
- cli.py: change default --quantization from bfloat16 to int8; saves
  ~50% VRAM/RAM for new nodes that don't specify a quantization
- startup.py: display memory budget and GPU info in GB (e.g. 124.9 GB RAM)
  instead of MB; show remaining headroom after full model load
- test_tracker_routing.py: add test_shard_heal_cycle_surviving_node_covers_dead_peers_gap
  — end-to-end proof that:
    1. tracker purges expired node A and queues LOAD_SHARD for node B
    2. node B receives directive on next heartbeat
    3. TorchNodeServer.apply_tracker_directives hot-swaps the backend
    4. node B re-registers covering the full model; coverage gap closed
  Test runs in <1s with monkeypatched _load_backend (no GPU needed)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Dobromir Popov
2026-06-30 23:08:34 +03:00
parent ff4115f611
commit b6272db93d
3 changed files with 196 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
packages/node/meshnet_node/cli.py
View File

@@ -23,7 +23,7 @@ def _run_node(cfg: dict) -> None:
model_id=cfg.get("model_hf_repo") or None,
shard_start=cfg.get("shard_start"),
shard_end=cfg.get("shard_end"),
quantization=cfg.get("quantization", "bfloat16").replace("bf16", "bfloat16"),
quantization=cfg.get("quantization", "int8").replace("bf16", "bfloat16"),
wallet_path=Path(cfg["wallet_path"]) if cfg.get("wallet_path") else None,
cache_dir=Path(cfg["download_dir"]) if cfg.get("download_dir") else None,
host=cfg.get("host", "0.0.0.0"),
@@ -278,7 +278,7 @@ def main() -> None:
start_cmd.add_argument("--model-id", help="HuggingFace repo ID")
start_cmd.add_argument("--shard-start", type=int)
start_cmd.add_argument("--shard-end", type=int)
start_cmd.add_argument("--quantization", choices=["bfloat16", "int8", "nf4", "bf16"], default="bfloat16")
start_cmd.add_argument("--quantization", choices=["bfloat16", "int8", "nf4", "bf16"], default="int8")
start_cmd.add_argument("--host", default="0.0.0.0")
start_cmd.add_argument("--advertise-host")
start_cmd.add_argument("--tracker-mode", action="store_true")